There's Too Much To Do
I just finished reading Drug Cartels Do Not Exist by Oswaldo Zavala, which has to be one of the best books I've read in the past few years. It steps through the creation of the "narco" and drug cartels not as things that exist on their own, but as an arm of state policy. Some of this will be old hat if you've read stuff by Graeber or things about the Middle East, where so often structures and instutions that we think are self evident, such as bureacracies, sectarianism, and narcos, are wholly swallowed and creations of the state1.
Meanwhile, I'm knee-deep in a network murder mystery. One of my data centers is plagued with more timeouts than a flaky internet connection. I'm also looking at issues with the Erlang scheduler, where the rise of 256 core CPUs like Bergamo is causing problems with scheduler stealing algorithms. Tokio and Go seem to have "solved" this by having various workers steal half of a queue rather than a single task at a time, but this still feels like it only kicks the can down the line, especially when Zen 5 chips are coming online with performance and efficiency cores. Maybe the scheduler stealing model is fundalmentally bad, but that only broadens the amount of space I have to search through for a solution that I like.
I've also been consumed with this Linux-is-dying idea, where conjunction of the warehouse scale computer has forced us to partition the pieces that we traditoinally considered operating systems into various ones, network attached storage being the most obvious one. Everything has grown faster, while x86 CPUs2 have largely stangated. Datacenter software cares about cooling, power, and network, the three pieces that now constrain our computing. Linux is being bypassed on most levels, most serious applications do some form of kernel bypass on the network, while other ones want to talk directly to the block device (Ceph, Crucible, etc). If it's not being bypassed, Linux has invited massive elephants into the room like eBPF, where the garguantuan LLVM monster now sits on our couches. None of these are bad things per se, but where does this leave Linux, if not a thin shim? And if there's a thin shim, might we revisit the days of small, dare I say, "micro" kernels?
This leads to my delimma: there's just too many goddamn things to do. Our towers of abstraction have grown so high, such that exploring any level is dizzying. To figure out my network issue I need to triage into the OS, into the rack, into the suite, into the network planes I'm running. To understand the little slice of Iraq I care about, I need to understand how we produce knowledge about Iraq, which means understanding how we produce knowledge in general about failed states. To see how people understand disappearances, and the formations of mourning, I need to see how people responded to state enforced disappearances in Uruguay and Argentina. Every step forward forces me to tap into a raging river of knowedlge and study that others are contributing to, which then produces this sense of vertigo.
I have become convinced that software is inseparable from the people that develop it. Cantrill's line about software as an oral tradition is not only correct, it might be too weak of an assertion. I will cautiously put forward the idea that software is not only an oral tradition, it is a set of rituals that coerce you into thinking about the world in a specific fashion. Good software engineers begin to tackle problems by building abstractions, almost to a point where they have difficulty seeing any other way to solve problems. Our rituals become our practices, which only then starts the flywheel of understanding. We as software engineers strive to develop a mechanical sympathy, to engage with an almost-communion with the machine. What do rituals in religion serve but to be repeated, which, once they become self evident, bind us to a parcitular way of seeing the world?
I get it - saying 'software is like religion' sounds like one of those half-baked 'everything is everything' ideas. But hear me out. There is something to that we must understand software not as a specific set of practices, but rather as a tradition. Just as Zavala's book reveals how 'narcos' are constructs of state policy rather than independent entities, our software practices and rituals are not merely technical choices, but constructs of our tech culture. Studying specific practices, such as programming languages, models, etc is more akin to studying a specific type of prayer, but none of these rituals and practices hold by themselves. They hold in repitition, in tension with all the other rituals, which can only be described as a tradition.
Which then brings us back to how there's too much to do. I don't think I'm going to reach a satisfactory conclusion, but I'm going to take a whack at dipping my head into the pool of knowledge and see what I come back with.