Honestly, the culture/org structure is a way bigger problem in this story than any proper noun tool.
If you’re ignoring guidance and patterns and getting mad reinventing the wheel, that’s on dev. If “ops” mandates tooling and doesn’t have any skin in the game, that’s on them. And both problems are on your leadership.
If y’all just hate each other and don’t listen or participate, then you can’t be successful. It is ironic that this is the pattern that the devops movement landed us in.
I don’t think you’re being pedantic. You’re just making a weird assumption that the radio itself is the only resource. I learned a ton from this as a kid. And I learned from Radio Shack. You stare at it, you go research, you try to fix it, you fail. Talk to someone who knows stuff. Repeat until it works or you work on a new one.
It’s really no different than how I taught myself to fix a chain or replace a spoke. Or know to use WD-40 to clean, but then apply an oil to keep stuff lubricated and protected.
With the internet, it’s a lot easier. I can look up spec sheets just googling component markings and see the sample circuits.
I’ve stared at the Linux kernel a ton. I messed with some stuff. I couldn’t write a kernel myself, but I program better from doing it and I can troubleshoot things easier knowing the components and topology.
Off the top of my head, I can fumble around and make a crappy amplifier from parts in my closet, or write a crappy FAT-like file system. I’d probably struggle a bit with a nice new bike. I think gear shifters and stuff are a lot fancier than an old 10 speed.
Yeah. There’s Jamf and similar tools. Companies often block major updates until their 100 agents all officially support it. Oh, and do cool things like not letting you change your background or whatever random settings some admin decides are good.
Yep, a lot of these policies seem to come from some random person scrolling through a list of supported options and arbitrarily making up values that are enforced on people.
One of our policies enforce that screen savers must start after 20 minutes, and it’s not possible to reduce it (I have my personal on 3 minutes). Or the fact that it will constantly reset the UI notification volume to 100% and speaker output, even though have headphones almost always.
The one thing I’ve seen are where companies have tax incentives tied to butts in seats. Usually like 0 property tax, with the government assumption that they’ll make it up in sales tax (lunch, gas, etc.) and taxes from employees that move to the town.
But honestly, I think a lot of companies are just doing this instead of layoffs or in addition to small “don’t raise eyebrows” layoffs. Raise the pain to get attrition.
The problem I’ve seen historically is when a company is founded around one project or ecosystem.
Someone like Microsoft or Google could take software like this, pay the original developer, and still see tons of ROI offering it as a canned cloud service. And to a certain degree they don’t care about the profitability of that 1 thing if it helps sell the rest of their system. Quite honestly, they won’t care about competition using it if it’s already common. People want X, they’re using it, you can offer it. You’re paying a handful of people for street cred, a guarantee it will continue to work well with your stuff, and input into direction.
Folks like RedisLabs, MongoDB, Hashicorp, etc. think they can do the same with a marketplace offering. But they’re reliant that the particular product is profitable on its own. They’re also reliant on the cloud customer being willing to establish another relationship with another vendor, even when they can automatically deploy and bill through their existing provider.
We’ve seen folks behind OSS projects hop from company to company over time and the project continues to thrive. I haven’t seen a company restrict a license and the project do so… at least that I can think of. I might be wrong.
As an old Unix guy this is exactly how I see jq: a gateway to a fantastic library of text processing tools. I see a lot of complicated things done inside the language, which is a valid approach. But I don’t need it to be a programming language itself, just a transform to meet my next command after the pipe.
If I want logic beyond that, then I skip the shell and write “real” software.
I personally find those both to be more readable and easier to fit in my head than long complex jq expressions. But that’s completely subjective and others may find the jq expression language easier to read than shell or (choose your programming language).
Your comment made me go look up jq (even more than the article did) and the first paragraph of the repo [0] feels like a secret club's secret language.
I'm very interested, but not a Linux person, do you know of any good resources for learning the Linux shell as a programming language?
I’ll say, I did shell scripting for years from copy/paste, cribbing smarter people, and reading online guides. But I didn’t really understand until I read The Unix Programming Environment by Brian Kernighan and Rob Pike.
It’s a very old book and the audience was using dumb terminals. But it made me understand why and how. I think I’ve read every Kernighan book at this point and most he was involved in because he is just so amazing and not just conveying facts, but teaching how to think idiomatically in the topic.
I also used awk for 2 decades, kind of like how I use jq now. But when I read his memoir I suddenly “got it.” What I make with it now is intentional and not just me banging on the keyboard until it works. A great middle ground for something a little sophisticated, but not worth writing a full program for.
Something else that helped me was to install a minimal distro… actually a base FreeBSD install would be great… and read the man pages for all the commands. I don’t remember the details, but I learned that things existed. I have many man pages that I look at the same options on every few months because I’m not positive I remember right. Heck, I ‘man test’ all the time still. (‘test’ and ‘[‘ are the same thing)
I also had an advantage of 2 great coworkers. They’d been working on Unix since the 80s and their feedback helped me be more efficient, clean, and avoid “useless use of cat” problems.
I also highly recommend using shellcheck. I sometimes disagree with it when I’m intentionally abusing shell behavior, but it’s a great way to train good habits and prevent bugs that only crop up with bad input, scale, etc. I get new devs to use it and it’s helped them “ramp up” quickly, with me explaining the “why” from time to time.
But yeah. The biggest problem I see is that people think there is more syntax than there really is (like my test and [ comment). And remember it’s all text, processes, and files. Except when we pretend it’s not ;).
Really love your comment, so much that I wanted to check out the books you mentioned.
After searching z-lib for "The UNIX Programming Environment", all I found was a janky and grainy PDF. Then I searched archive.org and discovered this high fidelity PDF version:
Much thanks for this. I've been struggling to grasp what's happening down in the engine room beneath me as I operate a modern Linux environment and only knew that there have been different waves of evolutions over the decades. I instead only see today's pretty deck up top, outside.
I'm only ten pages in, but that's mostly because the format and approach of this book is quickly yielding me tons of conversation fodder for ChatGPT4, where I've been endlessly asking it to clarify or point out holes in my mental model of everything I'm doing in the terminal, and still working out things such as why Backspace, Ctrl+H, Ctrl+/, Ctrl+?, Ctrl+-, and Ctrl+_ all together seem to have some overlap or differences in what's happening, depending on terminal contexts. I always had working notions of raw versus cooked sessions, the line discipline, etc., but I'm finding ways to play with the machinery with silly exercises that I otherwise couldn't have come up with.
For example, I just opened up three terminal windows:
Then, reproducing the whole "Hex" column of the manpage, in order, using my keystrokes. And, observing the multi-byte payloads of some of the other keys. I feel like a kid again :)
I’m kind of annoyed by the bait and switch of the title. It’s a play on Knuth’s classic but then turns into showing why Unix/Linux is better than Windows, etc.
As a disclaimer: I really don’t respect ESR and his work, and admire Brian Kernighan immensely. Very odd to be in a situation where those names are put side by side. Just want to call out that I do have bias on the people here. Don’t want to get into why as that’s not constructive.
I wasn't aware of the bait and switch at the time I read it, but I did really enjoy the history of how the Unix/Linux ethic came together and evolved over time. Had I heard of The Unix Programming Environment when I read it in 2014 I may have gone with that instead, as I was looking for something more along the lines of a technical handbook rather than a code of ethics.
Yeah and ESR can be revisionist in his history, projecting intention on something organic. He alienated a lot of people over time with this… and other behavior.
The book I recommended is both a handbook and a “how to think.” It applies forward to things introduced well after the book. But it also helped me understand why the Byzantine behavior of a tty is what it is.
If you are interested in the history from a first person perspective, I do recommend Kernighan’s “Unix: A History and a Memoir”. He went from originally trying to write something objective to realizing it was necessarily his personal experience. Even the culture aspect of his story has influenced how I try to foster teamwork. It was an engaging read for me.
I felt it was a good dive into "good" programs at a bit of higher level, rather than "here's how to do X". Quite a bit applies to Windows software and other software that never touches Unix as well.
Some bits are better than others, some bits haven't aged too well in the last 20 years, and it's a shame esr has since turned crazy. But I still feel the book holds up reasonably well.
"Bait and switch" certainly seems too strong of an accusation, especially for a book that's available for free.
I do agree that even pre-crazy esr was never on the level of Kernighan in any way.
So, grab yourself a Linux box (I suggest Debian), a large CSV file or JSON lines file you need to slice up, and an hour of time, and start trying out some bash one-liners on your data. Set some goals like "find the Yahoo email addresses in the data and sort by frequency" or "find error messages that look like X" or "find how many times Ben Franklin mentions his wife in his autobiography"
Here's the thing. These tools have been used since the '70s to slice, dice and filter log files, CSVs, or other semi-structured data. They can be chained together with the pipe command. Sys admins were going through 100MB logs with these tools before CPUs hit the gigahertz
These tools are blisteringly fast, and they are basically installed on every Linux machine.
>Your comment made me go look up jq (even more than the article did) and the first paragraph of the repo [0] feels like a secret club's secret language.
Or one of the most old standing widespread clubs of computing open standard language :)
"jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text."
Translation:
JQ is like a (UNIX/POSIX staple command line text-manipulation tool) but specialized for text structured in JSON format. You can use it to extract parts of a JSON document (slice), keep nodes based on some criteria (filter), transform each element in a list of structured data to get a new list with the transformed versions (map), and do that as easily as you can with the sed (basic command line text manipulation program), awk (command line text manipulation program with a full featured text-processing oriented language), grep (command line program to search for strings), and other assorted unix userland programs.
I do almost all hand tool woodworking, but not purist. My main smoothing plane is from 1910ish. My most new fangled hand tool is a Japanese Shinto rasp.
And you’re 100% right. I’ve changed my chisel and plane iron sharpening method twice in the last 12 months.
There’s oil stones, wet stones, diamond stones, and sandpaper. Plus a leather strop with pick-your-compound. I’ve used 2 different jigs on stones to get a more consistent bevel than by hand. There’s high speed grinders with a lot of pauses and cooling to not lose the temper. There’s expensive water cooled low speed grinders. Then there’s debate on the angle, microbeveling, and how much of the back you should flatten.
I’m now using a new jig from TayTools that uses a 3M Cubitron II sandpaper disk on a drill press when they get bad. I freehand on diamond and a strop after and between resetting the bevel. It’s the laziest way I’ve found so far.
Claiming any choice is best is likely to result in fisticuffs. And don’t start a conversation on workbench design or vice choices.
I dunno about rock solid. I’ve had plenty of issues forcing a failover/reboot, multiple complicated tickets open a year, etc. But we have a sh ton of them. To be fair, some are kernel bugs with connection table leaks, SNAT + UDP, etc.
Buuuut, they have by far the best support. They’re as responsive as Cisco, but every product isn’t a completely different thing, team, etc. And they work really well in a big company used to having Network Engineering as a silo. I’d only use them as physical hardware, though. As a virtual appliance, they’re too resource hungry.
Nginx or HA-Proxy are technically great for anything reasonable and when fronting a small set of applications. I prefer nginx because the config is easier to read for someone coming in behind me. But they take a modern IT structure to support because “Developers” don’t get them and “Network Engineers” don’t have a CLI.
For VMWare, NSX-V HA-Proxy and NSX-T nginx config are like someone read the HOWTO and never got into production ready deployments. They’re poorly tuned and failure recovery is sloooow. AVI looked so promising, but development slowed down and seemed to lose direction post acquisition. And that was before Broadcom. Sigh.
I'm very out of date so take my opinion with a grain of salt. The customer support I received from F5 when they acquired a telco product was about the worst support I've ever seen. Now this wasn't the general LB equipment that F5 has the reputation around, it's some specific equipment for LTE networks.
We'd get completely bogus explanations for bugs, escalate up the chain to VPs and leadership because there was an obvious training, understanding, and support for complex issues problem, and get the VPs trying to gaslight us into believing their explanations were valid. We're talking things like on our IPv4 only network, the reason we're having issues is due to bugs in the equipment receiving IPv6 packets.
So it's one of those things where I've personally been burned so hard by F5 that I'd probably to an unreasonable level look for other vendors. The only thing is, this was awhile ago, and the rumor's I've heard are that no one involved is still employed by F5.
I completely get this. I feel like every product I’ve had outside of a vendor’s wheelhouse has gone that way. We just use the BigIP gear from F5 and they’re better than the load balancers we used in the past. Thank god Cisco just abandoned that business.
I can’t imagine them supporting telco gear. The IPv6 thing has me LOLing because I just had a similar experience with a vendor where we don’t route IPv6 in that segment and even if we did, it shouldn’t break. Similarly, a vendor in a space they don’t belong that I imagine we bought because of a golf game.
A thing I dread is a product we’ve adopted being acquired… and worse, being acquired by someone extending their brand into a new area. It’s also why we often choose a big brand over a superior product. It’s not the issue of today, but when they get bought and by who. I hate that so much and not my decision, but it’s a reality.
It’s also a terrible sign if you’re dealing with a real bug and you’re stuck with a sales engineer and can’t get a product engineer directly involved.
I have a list of “thou shalt not” companies as well, and some may be similar where a few bad experiences ruined the brand for me. Some we’re still stuck with and I maaaay be looking for ways to kill that.
First, I don’t make these decisions but sometimes have influence. These opinions are my own and not my intentionally unnamed employer, and might be flat out wrong. This list is very focused on big companies at stupid scale with a lot of legacy… applied tech.
Generally my rule is “except for their very core product.” But this is full “hate everything” that pops into my mind:
RedHat won’t accept gifted patches for critical bugs in their tools that they won’t troubleshoot themselves. Getting the patch upstream means you get to use it in the next major version years later. That predates IBM. I won’t use their distribution specific tooling anymore. Outside the OS sucks worse. If I hear ActiveMQ one more time… [caveat: I probably hate every commercial Linux distro and Windows because my nonexistent beard is grayer than my age]
IBM… kind of feel sad about it, but they now suck at everything.
Oracle has good support, but they’re predatory and require an army of humans to manage inherently hodgepodge systems. Also creates an organizational unit of certified admins that can’t transition to alternatives because they’ve only memorized the product. Cisco’s the same except the predatory part and without many good alternatives for core DC gear.
CA, Symantec were awful pre-Broadcom and even worse now that they’re Broadcom’s annuity. Where products go to die.
Trellix (ex McAffee) is like the new Symantec or something.
There’s more I wish I could list for you, but can’t for various reasons.
On the other end, Satya has made MS a reasonable choice in so many things. Still a lot that sucks or is immature, but still… I didn’t think that was possible. I had to shift my mindset.
When was this? I worked with them 2009-2018, support was really top notch. We could get super technical guys on the call and even custom patches for our issues, but our usage was relatively simple. I contrast them with McAfee products we've used, now that was a complete shitshow as a product and support.
They really were older mentors to Nirvana. They and their band were about 10 years older (1 Beatles career), recruited them to Geffen, and really got them their first exposure outside Seattle. They also were largely responsible for getting Dinosaur Jr. their break.
I did not realize how old they were when I was obsessed with Daydream Nation and Dirty in high school. Kim’s just younger than my parents and they could never have been that cool. I went back into their catalog from there and followed along with them. That’s when I realized they formed the year after I was born. Blew my mind. Listening to the stuff like No New York that they emerged out of is kind of fascinating. Not my thing, but interesting historically.
Kim’s autobiography was an interesting view into the era my folks grew up in. I’ll have to read Thurston’s some time. Still bugs me they fell apart. I don’t normally care about famous people relationships, but they were like some kind punk rock institution to me. They were together since I wandered into that kind of music and nurtured so many bands I loved. But they’re just pretty normal people with pretty normal problems.
I totally agree on Sonic Nurse and Rather Ripped as well. The stuff from A Thousand Leaves until then was interesting, but not stuff I play often. Those have stood up well next to the aforementioned albums, Goo, EVOL, etc.
To me, there’s an ecosystem and a culture for a language in a situation.
The ecosystem is the tooling around the language. Debugging, performance profiling, building and dependency management are examples.
The culture is how the language is used in real life. How stable is the ecosystem?
There are lots of beautiful languages that are used horrifically. Enterprise OOP is an example for me where more than half the code is dedicated to avoiding basic principles for “what if we want to change later” (looking at you getters and setters).
And within a company or software suite, there can be subcultures of how “we” use the language. Every class needs an interface. Our C is written OO (see old GNOME).
For the ecosystem you have extremes. C where debugging and performance profiling are well defined practice, but modern dependency management doesn’t exit. The JavaScript ecosystem changes so rapidly, documentation is usually out of date, “nobody does that anymore”, etc.
And so much awful comes from trying to force a language into something the language wasn’t originally designed or even used for. Your Haskell example here. Adding functional language features and convoluted asynch to every freaking language is another. People create things much harder to learn than a new language with a mature ecosystem and culture in a domain just to avoid “learning a new language.”
In woodworking, I can do pretty much anything with a chisel. But I plane with a plane, saw with a saw, use specialty planes for rabbets, etc. Each has a learning curve, but I’ll end up with a better, more consistent result. And really, they’re all just chisels with jigs in different configurations. But I’m going to use a chisel where it’s still best, like mortising. Well, and where I’m not sure I want to invest in a more specialized tool yet and risk my wife killing me. Then I use C… I mean, a chisel.
If you’re ignoring guidance and patterns and getting mad reinventing the wheel, that’s on dev. If “ops” mandates tooling and doesn’t have any skin in the game, that’s on them. And both problems are on your leadership.
If y’all just hate each other and don’t listen or participate, then you can’t be successful. It is ironic that this is the pattern that the devops movement landed us in.