How are these clowns deploying stuff on a Friday, it is unbelievable to me. It is not even funny any more. It seems cloudflare is held together by marketing only. They should stop all of these stupid initiatives and keep their stack simple.
And I'm 100% sure the management responsible for this is already fueling up the ferraris to drive to their beach house. All of us make them rich and they keep on enshittifying their product out of pure hubris.
> How are these clowns deploying stuff on a Friday, it is unbelievable to me
I have stopped fighting this battle at work (not Cloudflare). Despite Friday being one of the most important days of the week for our customers, people still deploy the latest commit 10 minutes before they leave in the afternoon. Going on a weekend trip home to your family? No problem, just deploy and be offline for hours while you are traveling...
The response was that my way of thinking is "old school". Modern development is "fail fast" and that CI/CD with good tests and rollback fixes everything. Being afraid of deploys is "so last decade"... The problem is that our tests don't cover everything, it may not fail fast, and not all deploys can be rolled back quickly and the person who knows what their huge commit that touches multiple files actually does is unavailable!
We have had multiple issues with late afternoon deploys, but somehow we keep doing this. Funnily enough, I have noticed a pattern. Most experienced devs stops doing this after causing a couple of major downtimes, due to the massive backlash from customers while they are scrambling to fix the bug. So gradually they learn to deploy at less busy times and monitor the logs to be able to catch potential bugs early.
The problem is that not enough has learned this lesson because they have been lucky that their bugs have not been critical, or because they are too invested in their point of view to change. It seems that some individuals learn the hard way, but the organization has not learned or is reluctant to push for a change due to office politics. I decided to keep my head low and let things play out, as I simply no longer care as long as the management don't care either.
- Affecting both shopify.com and claude.ai, so no phased deployment
- Takes 30 minutes to remediate
If they would've just deployed to a single of their high-value customers at once, they could've spared shopify.com an hour of downtime and maybe millions in abandoned shopping carts.
If you are a monopoly, there is no incentive to do anything well. You've saturated the market, the incentive is to cut costs.
In fact, there are incentives for public failures: they'll help the politicians that you bought sell the legislation that you wrote explaining how national security requires that the taxpayer write a check to your stockholders/owners in return for nothing.
Yes, but a hotfix was already in place. They chose to deploy the "proper fix" this morning, and obviously it went wrong. Also they didn't do a phased rollout because it impacted their high-value customers such as shopify as well as claude, causing significant damages. Their procedures are not good.
"A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components."
The bug is known since several days, and the hotfix was already in place. So they worked on the "final fix" and chose to deploy it on a friday morning.
And I'm 100% sure the management responsible for this is already fueling up the ferraris to drive to their beach house. All of us make them rich and they keep on enshittifying their product out of pure hubris.