Most people can actually; for awhile most families did. The thing stopping them is largely the state. I got my land for "a song" post COVID when home prices were already insane. Near lots of jobs too. I used a rarely used 'loophole' to build it for about 1/4 the cost of what anything around me costs. It did cost me 60k, but for 30k (the price of a newish car, which you see even in many section 8 / welfare areas, so accessible even sometimes to the 'lower' class) you could easily finish out a 200 sq ft shed with a plumbed haul water cistern, minimal solar, a wood stove, and a simple waste treatment system. Then as you have more money, expand. You don't even have to have all the money at one time; many families in latin america or SEA just buy blocks as money allows and slowly build over time (masonry construction more forgiving of this).

I've even found land for <100k in places like San Francisco that is enough to drop a shipping container on, but of course the fascists won't let you, better to be homeless and shit hepatitis into the street than have an unpermitted shipping container and bury some 50 gallon drums and drain field DIY septic system.