the main issues ive had have always been around turning radius. Other than that for steep climbs or descents id much rather have a long wheel base rig with larger brakes. I sometimes consider getting something smaller but for the size i really have been able to get everywhere i wanted to, it may have just taken a little more thinking (and some scratches) to get there. I wouldnt give up the payload either i have a 92 3/4 ton suburban with a 454 and it drives pretty much exactly the same completely loaded up with gear and i can haul a bunch more gear without it even feeling it. The smaller rigs are nice but when you load them up with a bunch of gear you can usually feel the difference
1994 K2500 Suburban 6.5 turbo diesel, 4l80e, 14bff, g80
2000 Ford F250, v10, 4x4
2010 Toyota Carolla S (My DD)
2013 Ford Fusion (Her DD)
1992 K2500 Suburban 454tbi, 4l80e, 14bff g80 (Sold but miss it)