Anecdote from someone does a lot of graphics programming. (Building a molecule viewer/editor):

I've only needed matrices exactly once, when building the engine. It does a few standard transforms (model view matrices etc).

The rest is all len-3 Vecs, and unit quaternions. You can use matrices for these, but I'm on team Vec+Quaternion!