It's more about the organizational problems than the software.
QNX has the "minimal system" thing down. QNX is a POSIX microkernel for embedded applications. There's a standard distro for development purposes, but you can easily build your own, and that's what's normally done for embedded devices.
A boot image for QNX contains the QNX kernel, its associated user process "proc", and any other user space programs you'd like to have around at boot-up, such as drivers. (All QNX device drivers are user space programs.) File systems and networking are also optional programs. You can put user programs into the boot image as well. If you have a system with a disk, "diskboot" is usually included to bring up a system with more programs and daemons, but that's optional. If you're doing a car stereo or an industrial controller, you'd probably just put the application program in the boot image and avoid any elaborate startup process.
(While QNX is a neat technology, the company is totally screwed up now. In the last 12 years, they went open source, closed source, open source, and back to closed source again. They were acquired twice, once by Harmon (mostly a car stereo company) and then by RIM (the Blackberry people). The end result is that the old point of sale and industrial control customers became fed up and abandoned the system. Nevertheless, QNX is worth understanding to see what a usable microkernel system looks like.)
Well I went through some of the same stuff as Rob, but with fewer side projects, and there are several issues, and QNX was one possible route, as you point out dying. I would look at eg Genode for something to work on in that kind of model instead, open source L4 based.
The big problem is GNU, that turned Linux into a bloated userspace, with often broken development processes which are only gradually being fixed. The whole busybox is, apart from the multicall binary thing, just about writing smaller tools. The hard solution is to rewrite userspace, which got us uClibc and later the far superior Musl libc. The easy solution is to use the BSDs, which basically just provide the core system without insane bloat, and with cross compiling and the core userspace already built in.
Meh. Your bloat is my usability. It's the difference between Emacs and vi: vi is smaller, also less usable, according to me, and I'm the only one whose opinions I trust in this matter.
I loved this line: "shrinking teams get paralyzed by knowledge transfers: everybody spends all their time trying to learn what the departing engineers know".
It's very interesting to read about Rob's experiences. I had a similar (ish) experience working at a startup that was in way over its head making hardware. I got out earlier than he did (I only lasted a few months). My only advice is that if you're working at a startup and you don't have confidence in the management, definitely get out.
I also worked at a very well-run company (Laurel Networks) that also sold an appliance running embedded Linux in the early 2000s. We basically shipped a modified Red Hat 9. I was on the OS team. Our main concern was making sure no unnecessary ports were exposed or services running. We didn't have quite the same focus on creating the absolute minimum OS image size since our product included a hard disk. We had similar concerns with providing a seamless (and secure) upgrade path.
I think disk size is not that important in embedded Linux any more. If you're shipping a router that has its OS on a read-only compact flash card, you'll struggle to even find a CF card under 32 MB these days. They just don't manufacture them. So there's not much incentive to squeeze things on to a floppy disk any more. Microcontrollers still exist out there, but they don't run Linux (or usually any OS)-- usually they're just a "for" loop on bare metal.
On the other hand, minimizing the amount of code you're running is as important as ever. The more code you're running, the bigger the attack surface is. I hope that the Toybox project gains some traction out there as an option for folks that need it.
QNX has the "minimal system" thing down. QNX is a POSIX microkernel for embedded applications. There's a standard distro for development purposes, but you can easily build your own, and that's what's normally done for embedded devices.
A boot image for QNX contains the QNX kernel, its associated user process "proc", and any other user space programs you'd like to have around at boot-up, such as drivers. (All QNX device drivers are user space programs.) File systems and networking are also optional programs. You can put user programs into the boot image as well. If you have a system with a disk, "diskboot" is usually included to bring up a system with more programs and daemons, but that's optional. If you're doing a car stereo or an industrial controller, you'd probably just put the application program in the boot image and avoid any elaborate startup process.
(While QNX is a neat technology, the company is totally screwed up now. In the last 12 years, they went open source, closed source, open source, and back to closed source again. They were acquired twice, once by Harmon (mostly a car stereo company) and then by RIM (the Blackberry people). The end result is that the old point of sale and industrial control customers became fed up and abandoned the system. Nevertheless, QNX is worth understanding to see what a usable microkernel system looks like.)