Or, Deep Work is Illegible
Or, Everything Worth Learning You'll Have to Teach Yourself
I remember quite clearly how I learned Git. Spring quarter of my second year of undergrad, I took Data Ethics, which was a required course in the Data Science major. This was a mistake. The intended path was to take Computer Science for Data Scientists after completing the introductory courses, and take Data Ethics a year later. Consequently, it assumed a lot more fundamentals experience than I actually had, and getting through it was a bit rough.
Then, halfway through the course, I got Covid. At the time, that meant being quarantined in the Stony Island dormitories (UChicago's Elba) with a few other sick students for a few weeks until we tested negative.
This was the moment I needed to very quickly learn both Git and LaTeX, neither of which I had any prior exposure to, in order to continue my coursework. Luckily, one of my fellow inmates was willing to walk me through LaTeX basics. No such luck with Git.
My first experience with this piece of fundamental software was fighting a congested headache and blearily staring at an unadorned iTerm panel (a scary black box most closely associated with hackers), trying to remember which files were already committed.
I knew three commands: git add, git commit, and git push. I was struggling to remember which files I had already committed after changes and which I hadn't, but every time I got it wrong, a bunch of errors popped up in the terminal and I panicked.
Only after I finished those assignments did I realize git status existed.
I could have discovered it and gotten a much better idea of Git's branching model if only I'd taken a deep breath, put away the assignment, and started reading Pro Git, or any tutorial, or probably even the man pages.
But this was my second year of undergrad, and I still believed in deferring five minutes of reading the manual in favor of five hours of painful, confused head-on-brickwall-action.
This all came back to me a few weeks ago, when I was talking to one of my collaborators, Vera Delfavero (the developer of BASIL, previously discussed here).
In the course of our conversation, we wound up saying "Everything worth learning you'll have to teach yourself".
On the surface, it's a very odd thing to hear from two people with advanced degrees, who are at least half-academics. It took me a while to think about what that really meant.
Vera's background is in astrophysics with an electrical engineering minor, with a decade of experience in C and Python. But she never took a general-purpose CS class. Her programming knowledge is almost all self-taught.
I also had limited exposure to CS in academia, other than the aforementioned CS for Data Scientists (aka CS for Dummies). All of my lower-level knowledge, including all my knowledge of Rust, which is most of what I use these days, is self-taught.
Does that mean university was a waste of time for either of us?
No.
Rather, regardless of what learning environment you're in—formal schooling, on-the-job, just learning on the side—you need to take personal responsibility for your own learning.
Which just about sounds like the most 'no shit' statement you could make about education. Yeah, obviously, nobody else can just cram knowledge into your brain, everyone knows that.
It's just one of those things everyone knows and most people don't really act on.
I can't for the life of me justify it now, but I had a deep-set assumption well into undergrad, a strange kind of entitlement, that all the necessary tools and information to complete an assignment must necessarily all be on the assignment page, so taking time to look up tools, read documentation, or otherwise learn anything other than the strictly prescribed content was therefore wrong in some way.
In spite of all the friction those first few days with Git, I halfway convinced myself that it was difficult and unpleasant on purpose. I'd seen enough complaining about learning Git online that I assumed forcing you to manually remember committed files was the masochistic point.
The same extended to tooling more broadly. We mostly used RStudio and Jupyter notebooks in class, and after two years of that, graduating to VSCode felt like a breath of fresh air. I can't tell you how many hours I spent manually indenting code line by line in a Jupyter cell, and how much stress this waste of time induced, yet I never even thought to take a breath and look for tools that would speed it up.
When I was in middle school, one of my teachers told me off for my terrible handwriting (which remains terrible to this day), informing me that I would need to be able to write cursive to graduate college, because I would need to write quickly in order to turn in essays on time.
Which is kind of a psychotic thing to say. I don't know if he was just making it up to get us to improve our handwriting or if he had a VERY different college experience but this was never the case for me, even in classes with handwritten essay exams (some of my favorite courses, actually).
But I maintain that a variant on this is very much true for typing and coding. Not only was I very slow just at typing code (I was a hunt-and-pecker until partway through my MS!) but editing already-written code was full of friction and frustration.
It took me years, despite explicit recommendations that it was one of the most valuable things I could do while learning to code, to try Vim. I'm typing this into a Neovim buffer right now, on a psychotic multimodal split keyboard (thanks ZSA!) which I can type on and use all my tmux shortcuts while wearing a blindfold, and the difference in quality of life is INCOMPARABLE. I wake up in the morning, sit at my computer, and just start coding because I've made the process so pleasant and frictionless. It's smoother and more immediately pleasant than playing videogames (hence why I haven't finished BG3 after 3 years).
But back then, I was subjecting myself to this weird kind of learned helplessness where I was willing to endure frustrating, stressful, inefficient conditions entirely of my own making in order to avoid actually focusing and learning to improve. The most important skill I've learned in the last ten years is to notice when I'm doing that and decisively kick myself out of that state without hesitation.
The rest of this post is about why it's socially (and psychologically) acceptable to smash your face against a brick wall.
I was reminded of this subject listening to Nik Suresh's Does a Frog Have Scorpion Nature? interview with Ed Zitron. In particular, the section around 48 minutes in, when they point out that most people just don't like to think. More specifically, people will actively avoid concentrated work and learning (e.g. reading the manual) even if it would save them time and stress on net.
It's not obvious that this should be the case, let alone why. Suresh settles on the explanation that concentration is just more intense compared to fumbling around in the terminal until everything is green. There's a grain of truth there, but it demands more explanation. The latter is certainly more time-consuming and stressful, but it's also more comfortable, because it doesn't demand that you learn anything new or revise your current knowledge.
But I think the biggest unexplored reason is that effortful but thoughtless and dubiously productive work looks and feels like productivity, which forms a defense against accusations of laziness or unproductivity. Look at how long I've been smashing my head against this brick wall, you might say. I'm putting in so much effort, what more can you ask of me?
Meanwhile, the kind of work that could actually solve the problem, often more quickly and which produces a more durable understanding of the domain, doesn't look like hard work from the outside; you might as well just be reading a random book, browsing, the internet, or spacing out.
Despite praise for 'lazy' programmers being quite widespread in software engineering, there's no social script for telling someone they're working ineffectively, and to take a break and read a book instead. Meanwhile, there's a very well-developed social script for telling (if you're a superior) or pressuring (if you're a peer) someone seemingly doing nothing to get to work, even if they just end up wasting time and energy.
This is, I think, a matter of legibility, in the James C. Scott sense.
The same point has been raised before by others, perhaps most notably Sean Goedecke in 'Seeing like a Software Company.' Though he doesn't bring it up as an example, learning tasks like 'figure out how this API works' don't tend to show up on tickets. Most organizations don't consciously make room for active, ongoing learning at all.
This dovetails with criticisms I and many others have made of Scrum-Agile in the past. In an environment where you can't get detailed, on-the-ground information about the state of work (this seems to include all medium-large organizations, even with well-organized and technically-savvy management, and anything larger than an early startup without), management will gravitate towards structures that maximize legibility at the cost of efficacy (in pathological cases, totally eclipsing the capacity to get anything done).
Velocity may be slow as long as we can measure it (or better, can fudge metrics to make it look like velocity is high) because the alternative is a chaotic abyss where your dynamic 10x engineers might be just about to ship... or they might have nothing at all.
We can argue about the merits of technical management or what executive risk tolerance should look like, but I think it's undeniable that the legible option is an immensely more comfortable place to be for almost everyone within firing distance of a given project.
This is the case for executives and managers, for whom this process produces an abundance of statistics, feedback, reports, and other wood-pulp- and slide-show-based paraphernalia. If the project goes belly-up, it all serves as a shield and evidence for whatever changes you think should be made. There's probably some Soviet euphemism about toilet paper that isn't coming to mind.
But it's also the case for most of the engineers and developers. It's nice to know there's a plan, and that if you keep your head down and participate in its rituals you can avoid taking blame onto yourself and diffuse accountability, which we know makes people less effective, but again, that's not the point.
The only person who prefers the illegibility is the rogue engineer who just starts doing things regardless of what the plan says. This person might rescue the project, or accomplish nothing at all, or doom everything by causing confusion and siphoning resources.
I've been that person a few times. Most recently, it was in my MS capstone. It's been six months since then, so I feel comfortable saying, despite liking and admiring many people on it, that it was a shitshow.
Hang on, I'm going somewhere with this.
Part of it was outside of our control. We were trying to build an LLM wrapper to turn natural language descriptions into three-dimensional animated drone light shows. We started practicing with smaller, less capable drones while we waited for the larger, more powerful drones capable of intelligent swarm navigation to arrive... which were delayed for weeks because of tariff chaos. By the time they arrived, we discovered that the model had firmware conflicts, and it took us months to reliably get even one to fly, let alone coordinate a swarm.
Those were weeks upon weeks of helplessness, waiting for someone to figure out how the drones worked (in theory it might be anyone, which diffused responsibility among everyone) ahead of actually building the description-to-animated-drone-lightshow-pipeline... which we just kinda assumed would be easier once we could pilot the drones? Yeah, no.
Around fifteen weeks into a twenty-week project, we had pretty much nothing, so I just told the team that we needed to have some kind of MVP and started building it solo. It ended up being a egui-based rasterizer that turned black-and-white images into 2-d point maps and then exported those maps to a drone light show extension in Blender. Most of the effort went into the UI so it looked and felt smooth.
In theory, there was an LLM integration, a half-completed HTTP API that could reach out to external image generators to produce the source image, but I had trouble getting the keys to work on my own account, stopped bothering, never merged to main, and I don't think anyone else ever looked at it.
By presentation time, our LLM drone project was really only gesturing vaguely at LLMs or drones in any capacity, but we still ended up doing ok because judges like visuals and we technically shipped a working library/GUI that people could theoretically import and use (pretty sure the only people who did were another team who wound up using part of the library as a component in their project. Helping them out with that component was by far the most fulfilling and energizing part of the capstone).
This isn't uncommon by any means. A few months ago, I was bemoaning this state of affairs to an engineer then at Uber, who told me his capstone project turned out much the same. Multiple teams working on computer vision and autonomous toy cars got their projects scrambled by Covid, and all their final presentations wound up being in simulations rather than with actual robots. This engineer swept the competition awards, despite knowing his results were worse than other teams', because he spent a while in Blender making an animated car model, while everyone else had pastel cubes.
You can interpret that as doom and gloom for our ability to recognize good engineering work absent jangling keys. It's partly that. But mostly, it was all a product of the fact it was a student project, and the department simply couldn't hold us back or delay our graduation on account of not actually shipping what we set out to ship. The bar was low, and we weren't the only people limbo-ing under it.
The real tragedy is that when all your student projects look like this, you're not teaching people to ship. You might well be teaching people how not to ship and get away with it.
And listening to Suresh and Goedecke, the same dynamics are ubiquitous in industry.
We had weekly three-hour in-person meetings to discuss and get work done. One or two people reliably weren't there by the end. Most everyone else had no energy or morale. What did you get done this last week? Nothing? Yeah, me neither.
It was kinda supposed to be an Agile project, but we didn't really have stakeholders outside the team that could give us feedback or demand a working prototype, which I understand is the most common kind of cargo-cult Agile.
Notionally, we were coordinating on an Azure kanban board. For a few weeks, I was the person limply asking the team to add their cards onto the board and report progress there, not expecting anyone to do it. I retroactively broke up some half-completed tasks so I could move something to the 'Done' box. Our instructor never gave us flack for it. He knew the score. In retrospect, I can't help but think that part of the issue was that logging into Azure was a pain, navigating the board was confusing, and the UI was ugly as sin, so nobody used it unless forced.
Those with energy bashed their heads against actually getting the drones to fly, and the rest of us spent our time doing work for other classes or effectively just simulating work. Working on simulations or simulatin' work, amiright fellas?
By the time we were three-quarters of the way through with nothing to show for it, I started working on the GUI not to glorify myself or out of a dedication to the project, but just because I was bored, bored out of my gourd, and just wanted to accomplish something, if it actually pushed the project forward all the better.
It worked out because it was a student project and they needed to graduate us. But the same outcome in a real company would have been delays, controlled failure, maybe some firings. Total failure to ship.
I've done work I'm proud of. I'm proud of fi-slurm-utils (I recently came across an astronomer who says he uses them all the time, and it was the proudest I'd felt in weeks). I'm proud of the work I've done on McFACTS. As noted earlier, I'm even proud of some of the raster-drone work I did for another team during the capstone, siphoning effort away from my own project to make the API they were using just a little nicer, and seeing that it made them just a bit happier.
Most of that work was some variety of solo, or at least somewhat isolated. fi-slurm-utils was developed with weekly check-ins with my mentor, and pretty much complete independence outside of that. My optimization work on McFACTS is reviewed and guided by the needs of the larger team, but isn't really collaborative. raster-drone was pure rogue work.
One of these days, I'm going to be in a job interview and get asked a question about my experiences collaborating on teams, and I'm going to either shut up, make shit up, or sound like an insufferable, antagonistic, anti-social bastard.
But nevertheless, those are the circumstances I've done my best work, and I think it's largely because those were situations in which it was impossible to just bash my head against a wall. Those were situations where I was in close contact with the reason why the work needed to be done. I knew what the success condition was, and I could tell how far I was from it, and it was impossible to ignore that distance. And if I didn't move it forward, nobody else would pick up the slack.
Suresh has a post where he mentions that he, all of his friends, and probably everyone reading the blog, have the experience of being the only person on a group project that actually does the work. Well, small samples and all that, but my n=1 is in support.
The antidote, as best as I can tell, is to locate and stay inside that set of circumstances as much as possible. Actually doing that seems quite difficult, not least because there's no standard narrative or social script for it. I'm currently taking inspiration from Suresh's results-oriented consultancy model, which does seem purpose-built to provide that kind of environment.
In other words, bashing your head against the wall is legible and socially-supported, but ineffective, while deep work is illegible and hard to distinguish from being lazy (derogatory).