We've made a couple of small engineering investments to get features that Heroku doesn't offer. For example, we built a simple auto-scaler for dyno's that consume from RabbitMQ. This helps a bit to keep our costs down and respond to massive spikes quickly.
We run our web servers, database, caching, data pipeline etc on Heroku. However, we are not _that_ dependent on Heroku. We use Docker for deployments, so we can easily reproduce what's happening and it reduces our vendor lock-in.
I missed this part. If you are okay with using managed services, why are you doing so much “undifferentiated heavy lifting”? Setting up autoscaling based on queues, managed RabbitMQ based messages (Amazon MQ), and without knowing what your data pipeline consists of, probably that too could all be done with managed services.
Don't imagine anything fancy here. This is roughly 100 lines of Python code that scales based on some tresholds. It took a day or two to develop, test and deploy.
However, we recently found a managed service to do this for us so we'll be switching to that as soon as we can.
If we'd have to spend more engineering resources on our infrastructure than the occasional simple tool, then we'd consider switching away from Heroku. The benefits still outweigh the potential engineering effort of not running on Heroku.
I missed this part. If you are okay with using managed services, why are you doing so much “undifferentiated heavy lifting”? Setting up autoscaling based on queues, managed RabbitMQ based messages (Amazon MQ), and without knowing what your data pipeline consists of, probably that too could all be done with managed services.