What vehicle is right for me

This is for sure a consideration as more and more used diesel ones hit the market
Before jumping into a diesel, please be aware of what comes with that. Be familiar with the extra maintenance. Look up regen issues. Diesels are nice, but not magical engines. Applicable in certain situations, but in my opinion, would end up costing you more.
subaru all the way.. their cheap, reliable, and good on gas, I'll use my 12 forester for some specs here. stock a subaru will go (with good offroad driving habits) down moderately crappy trails (find freshlikesushi) add a small lift kit and your above ground clearance of a stock 10 year old landcruiser. towing capacity of the new 3.6r ob is 3k. with attention to top speed (55) a 5 year old scoob will tow 3000lbs as well (I do with mine) with upgraded springs "normal" speeds can be achieved as the limiting factor for towing on the 09-13 forester is the back suspension (and all the other models have the same suspension) approach angle on them isn't the greatest, but it can be dealt with.. and I wouldn't take it rock bouncing, but for everyday chores, and the occasional offroad jaunt in the woods mine fits the bill well. after 2.5 years of ownersihp and 60k miles of pushing it to its limits, I just had to replace the clutch due to the throw out bearing performing cold fusion to itself, and then eating the clutch fingers. the clutch friction itself was in good shape. as well as most of the mechanical's, I average 22.5 based on the mpg thing on the dash, and I don't drive for economy :) the trade off on a subaru is 4x4 prowess for mpg, and since I'm not going to moab every weekend, it was a trade off I could live with.. I did research for about 4 months before I pulled the trigger on mine, and yes.. its been great...
Okay a couple clarifications

It's 2 adults and 2 kids in car seats. (2.5 and 1 years old)

A lot of the time it will be me and 1 of the boys or in a couple years both of them. Wife will come on rare occasion

The MPGs are for empty daily driving. If it goes from 24 to 17 loaded I don't care much as it's only a couple times a year I take a trip like that. I'd be willing to go down to 20 mpgs unloaded highway real world numbers but not really any less.
You might take a look at an 05-12 Nissan Pathfinder. When we went shopping for a car for my wife we were originally thinking about a Honda Element or a Subaru crosstrek. We went to Carmax to check out all of the cute ute suvs because they usually have one of everything on the lot. I'm about 6'3" and it was clear that most compact suv's don't have much headroom, You have to recline the seat so far back it's like sitting in a sports car. The Nissan Pathfinder or Xterra weren't even on our list but we drove both and decided on the Pathfinder. It gets 20 mpg and I believe might be able to tow up to 5,000 lbs.
I think the R51 Pathfinder would fit the bill nicely. Its MPG will be at the bottom of your range. My 07 averages between 19-22 on loaded (2 adults, 2 medium dogs) trips, depending on whether we are headed up to the mountains or down the coast. I would look at the 2010-2012 R51s. Any issues were sorted out by then and they have 3.13 gears as opposed to the 3.36 I have so you should be able to squeak out a little better mileage. Mine is rated to tow 6k.

We thoroughly enjoy the space ours has for the 4 of us without cluttering up the roof and it has been more than capable for my off road excursions.

Cargurus lists several 4x4, 2012 Pathfinders w/ <50k miles for under $18k
Sounds like an outback is right for me for the most part. I kind of thought that when I wrote my original post. The only other thing I am considering is a Cherokee trail hawk used (lots of those used listed sub $24k so I should be able to fit it in the budget) cargo space is limited but with enough stuff on the roof/hitch it be fine
I've got 2 kids, and no way would I even consider a Cherokee. I would go with a Grand Cherokee, at a minimum (if you were going with a Jeep). They could probably be had for cheaper price as well, and the Pentastar V6 gets good MPG.
I would still vote a Subaru, either a Outback or Forester. You can get a new Forester for your budget. I believe they all come with X-Mode now that does pretty well off-road. Enough for what you'll be doing, I believe. I like the roof rails on the Forester much better than the Outback as well.
Not to leave this hanging but I ended up getting a 2018 GMC Canyon. Changed jobs and got a bigger vehicle allowance.

Started pulling the stickers and badges off right before that picture.