Chevy/GMC 3/4 & 1 ton + IFS differential housings are very strong and aftermarket does indeed provide locker carriers, gear sets etc for em, the axles & CV joints are strong too. All thing being equal or not if one beats the snot out of any drive-train it will break eventually.
Well regardless of how strong the GM ¾ & 1-ton IFS might be, it's still far more limited for flex (leaving you spinning because it won't let your tire rise up onto that ledge, rock, or whatever). Sure, a locker helps to a good extent, but it's still no substitute for having a compliant suspension that can follow the terrain better.
For probably 95% of the people on here, this discussion likely is not of much importance (the stock IFS setup works perfectly fine). But the question was asked: What are the worries with IFS?. For the remaining 5% (or however many it is here that venture on more than the typical semi-rutted dirt road leading to a campsite), it's a very valid concern.
As the miles accrue with age, maintenance on a typical IFS tends to become more costly as well (all the control arm bushings, steering joints, idler arms/rack, CV shafts, dust boots, the unitized wheel bearings on most, etc.).