I would go for sure with the kids. 99% of the bears want nothing to do with us humans it's just being smart to plan for the 1%. You mentioned fishing. I probably wouldn't bring any of that back to camp
Seriously though, I don't want to dismiss the possible danger but odds are very, VERY slim you would have an encounter. Good luck and have fun.
Oh and as others said, get some bear spray.