You are right, this is why approaches like plenoxels are vastly faster than nerf. They combine the optimization approach of neural nets, but combine it with a simple and regular data representation of the scene.

https://arxiv.org/pdf/2112.05131.pdf