I think there really is a market for special fruits but the problems that prevent them from being ubiquitous lead to them being extremely local. In my area, we have a specific type of strawberry that sells out as quickly as they can be harvested. They are unlike any other strawberry available. Super juicy and sweet. Bright red throughout. Extremely perishable and delicate. They are only available for about 3 weeks in June and they tend to get moldy within a few days of harvesting, so you really only see them at farmers markets and a handful of grocery stores that specialize in local produce. If you want the best ones, you need to buy them within an hour of the opening of the farmers market. By 90 minutes, even the lower quality ones are gone. You get 2 more tries the following weekends and then they are gone until next year.
Since they cannot travel or be kept in inventory, they will never be available fresh more than about 50 miles from where they are grown.
I asked someone why local village wine in the south of France was better than 75% of the bottles I'd had.
They told me "The best grapes never make it out of the village. The next best grapes never make it out of the region. The next best grapes never make it out of the country."
Wine was probably a terrible example, being a less perishable good suited for transport, but the quip always stuck with me. No idea the origin.
We (city folk) assume that the best agricultural output is exported, but... why would it be?
The price differential for quality likely isn't very large, local consumption is relatively modest compared to output, and cap that off with the freshness timer.
The best wine I've ever had was in the French countryside. The best beef I've had was in a dirty road place in the middle of cow country. The best salmon I've had was in Seattle. The best sushi I've had was behind Tsukiji (when they still did wholesale fish there).
Eat local is as much about quality as it is other things.
Since they cannot travel or be kept in inventory, they will never be available fresh more than about 50 miles from where they are grown.