Everything Google's Python team were responsible for

In a questionable strategic move, Google laid off the majority of their internal Python team a few days ago. Someone on Hacker News asked what the team had been responsible for, and team member zem relied with this fascinating comment providing detailed insight into how the team worked and indirectly how Python is used within Google.

I have aphantasia—no ability to create images in my mind, or to remember things in a visual way.

For well over half of my life I lived under the assumption that suggestions to visualize things were entirely metaphorical, and also that I was incredibly stupid in certain ways, finding things that everyone else seemed to find easy a challenge. I struggle to learn graphical interfaces, I’ve mixed up speakers and their talks at conferences, I spend a lot of time staring at icons on my phone—as far as I’m concerned every blue icon leads to the same app.

I store information as lists. For people I see a lot, I could probably draw a reasonably accurate picture of them, based on the list of data I’ve collected. However that information can never be as detailed as being able to recall an image could be. For example, perhaps you are someone I work with. I see you several times a week on video calls. I know lots of things about how you look, including that you wear glasses. However, your glasses were unremarkable to me and so I’ve not stored any particular information about them. You could dramatically change your style of glasses, I’d not notice. I’m not comparing a visual memory of you from a few days ago, I just know you wear glasses.

If I’m visiting a new place and I go for a run, I collect data points along the way to navigate back. I will not remember how the route looked. This means that I can be incredibly good at taking people somewhere that I’ve only been once, I’ve got turn by turn directions. However, if I am walking with someone and talking, or for some reason don’t actively collect information, I’m doomed. The other week I lost an entire car park, never mind my car, due to having been distracted by a message after leaving the car park and failed to collect any information about what it looked like.

I dream, but not in pictures. For example, I have a recurring dream where I’m back in a theatre, putting on pointe shoes. I know it’s a theatre because of the smell, I feel the roughness of the shoes as I put my feet into them, the creak of the leather sole as I roll through my foot. I can feel that dream as I write about it, but there’s no image involved.

When I learned about aphantasia about ten years ago, suddenly so many things made sense. It’s harder to remember certain types of things if you have no visual memory of them. I’m not an awful person because I didn’t remember that I’d met a person before, there was no way for them to look familiar to me. However, I also think it’s at the root of some of the things I’m really good at.

My lists of information, are closer to a relational database than just a set of lists. It’s no surprise to me now that I always enjoyed working with databases and could design a complex schema without needing to sketch out a diagram. I’m constantly making connections between these bits of information. Many of these connections are just amusing to me, but other times they bring up interesting paths to investigate.

I context switch very easily, I can jump between these information sets without losing my train of thought.

I can write entire articles, documents, or conference talks in my head while out for a run. It’s usually quicker for me to create content in this way than sit at a computer and think about it. I can come in from a run and type out 2,000 words, transferring what I’ve written in my head to the document.

Having discovered this about myself, I’ve found plenty of other people who experience the world in the same way, probably unsurprisingly as people with aphantasia tend to be drawn to computing and science. I find it fascinating that we are all experiencing the world so differently, and how that can so fundamentally impact the things we find easy, or difficult.

Use an llm to automagically generate meaningful git commit messages

Neat, thoroughly documented recipe by Harper Reed using my LLM CLI tool as part of a scheme for if you're feeling too lazy to write a commit message - it uses a prepare-commit-msg Git hook which runs any time you commit without a message and pipes your changes to a model along with a custom system prompt.

Last week, the internet dodged a major nation-state attack that would have had catastrophic cybersecurity repercussions worldwide. It’s a catastrophe that didn’t happen, so it won’t get much attention—but it should. There’s an important moral to the story of the attack and its discovery: The security of the global internet depends on countless obscure pieces of software written and maintained by even more obscure unpaid, distractible, and sometimes vulnerable volunteers. It’s an untenable situation, and one that is being exploited by malicious actors. Yet precious little is being done to remedy it.

Programmers dislike doing extra work. If they can find already-written code that does what they want, they’re going to use it rather than recreate the functionality. These code repositories, called libraries, are hosted on sites like GitHub. There are libraries for everything: displaying objects in 3D, spell-checking, performing complex mathematics, managing an e-commerce shopping cart, moving files around the internet—everything. Libraries are essential to modern programming; they’re the building blocks of complex software. The modularity they provide makes software projects tractable. Everything you use contains dozens of these libraries: some commercial, some open source and freely available. They are essential to the functionality of the finished software. And to its security.

You’ve likely never heard of an open-source library called XZ Utils, but it’s on hundreds of millions of computers. It’s probably on yours. It’s certainly in whatever corporate or organizational network you use. It’s a freely available library that does data compression. It’s important, in the same way that hundreds of other similar obscure libraries are important.

Many open-source libraries, like XZ Utils, are maintained by volunteers. In the case of XZ Utils, it’s one person, named Lasse Collin. He has been in charge of XZ Utils since he wrote it in 2009. And, at least in 2022, he’s had some “longterm mental health issues.” (To be clear, he is not to blame in this story. This is a systems problem.)

Beginning in at least 2021, Collin was personally targeted. We don’t know by whom, but we have account names: Jia Tan, Jigar Kumar, Dennis Ens. They’re not real names. They pressured Collin to transfer control over XZ Utils. In early 2023, they succeeded. Tan spent the year slowly incorporating a backdoor into XZ Utils: disabling systems that might discover his actions, laying the groundwork, and finally adding the complete backdoor earlier this year. On March 25, Hans Jansen—another fake name—tried to push the various Unix systems to upgrade to the new version of XZ Utils.

And everyone was poised to do so. It’s a routine update. In the span of a few weeks, it would have been part of both Debian and Red Hat Linux, which run on the vast majority of servers on the internet. But on March 29, another unpaid volunteer, Andres Freund—a real person who works for Microsoft but who was doing this in his spare time—noticed something weird about how much processing the new version of XZ Utils was doing. It’s the sort of thing that could be easily overlooked, and even more easily ignored. But for whatever reason, Freund tracked down the weirdness and discovered the backdoor.

It’s a masterful piece of work. It affects the SSH remote login protocol, basically by adding a hidden piece of functionality that requires a specific key to enable. Someone with that key can use the backdoored SSH to upload and execute an arbitrary piece of code on the target machine. SSH runs as root, so that code could have done anything. Let your imagination run wild.

This isn’t something a hacker just whips up. This backdoor is the result of a years-long engineering effort. The ways the code evades detection in source form, how it lies dormant and undetectable until activated, and its immense power and flexibility give credence to the widely held assumption that a major nation-state is behind this.

If it hadn’t been discovered, it probably would have eventually ended up on every computer and server on the internet. Though it’s unclear whether the backdoor would have affected Windows and Mac, it would have worked on Linux. Remember in 2020, when Russia planted a backdoor into SolarWinds that affected 14,000 networks? That seemed like a lot, but this would have been orders of magnitude more damaging. And again, the catastrophe was averted only because a volunteer stumbled on it. And it was possible in the first place only because the first unpaid volunteer, someone who turns out to be a national security single point of failure, was personally targeted and exploited by a foreign actor.

This is no way to run critical national infrastructure. And yet, here we are. This was an attack on our software supply chain. This attack subverted software dependencies. The SolarWinds attack targeted the update process. Other attacks target system design, development, and deployment. Such attacks are becoming increasingly common and effective, and also are increasingly the weapon of choice of nation-states.

It’s impossible to count how many of these single points of failure are in our computer systems. And there’s no way to know how many of the unpaid and unappreciated maintainers of critical software libraries are vulnerable to pressure. (Again, don’t blame them. Blame the industry that is happy to exploit their unpaid labor.) Or how many more have accidentally created exploitable vulnerabilities. How many other coercion attempts are ongoing? A dozen? A hundred? It seems impossible that the XZ Utils operation was a unique instance.

Solutions are hard. Banning open source won’t work; it’s precisely because XZ Utils is open source that an engineer discovered the problem in time. Banning software libraries won’t work, either; modern software can’t function without them. For years security engineers have been pushing something called a “software bill of materials”: an ingredients list of sorts so that when one of these packages is compromised, network owners at least know if they’re vulnerable. The industry hates this idea and has been fighting it for years, but perhaps the tide is turning.

The fundamental problem is that tech companies dislike spending extra money even more than programmers dislike doing extra work. If there’s free software out there, they are going to use it—and they’re not going to do much in-house security testing. Easier software development equals lower costs equals more profits. The market economy rewards this sort of insecurity.

We need some sustainable ways to fund open-source projects that become de facto critical infrastructure. Public shaming can help here. The Open Source Security Foundation (OSSF), founded in 2022 after another critical vulnerability in an open-source library—Log4j—was discovered, addresses this problem. The big tech companies pledged $30 million in funding after the critical Log4j supply chain vulnerability, but they never delivered. And they are still happy to make use of all this free labor and free resources, as a recent Microsoft anecdote indicates. The companies benefiting from these freely available libraries need to actually step up, and the government can force them to.

There’s a lot of tech that could be applied to this problem, if corporations were willing to spend the money. Liabilities will help. The Cybersecurity and Infrastructure Security Agency’s (CISA’s) “secure by design” initiative will help, and CISA is finally partnering with OSSF on this problem. Certainly the security of these libraries needs to be part of any broad government cybersecurity initiative.

We got extraordinarily lucky this time, but maybe we can learn from the catastrophe that didn’t happen. Like the power grid, communications network, and transportation systems, the software supply chain is critical infrastructure, part of national security, and vulnerable to foreign attack. The U.S. government needs to recognize this as a national security problem and start treating it as such.

This essay originally appeared in Lawfare.

Well — we just witnessed one of the most daring infosec capers of my career.

Here’s what we know so far: some time ago, an unknown party evidently noticed that liblzma (aka xz) — a relatively obscure open-source compression library — was a dependency of OpenSSH, a security-critical remote administration tool used to manage millions of servers around the world. This dependency existed not because of a deliberate design decision by the developers of OpenSSH, but because of a kludge added by some Linux distributions to integrate the tool with the operating system’s newfangled orchestration service, systemd.

Equipped with this knowledge about xz, the aforementioned party probably invented the persona of "Jia Tan” — a developer with no prior online footprint who materialized out of the blue in October 2021 and started making helpful contributions to the library. Up to that point, xz had a single maintainer — Lasse Collin — who was dealing with health issues and was falling behind. Shortly after the arrival of “Jia”, several apparent sock puppet accounts showed up and started pressuring Lasse to pass the baton; it seems that he relented at some point in 2023.

Since then, “Jia” diligently continued the maintenance work — culminating in February 2024 with the seamless introduction of a sophisticated, well-concealed backdoor tucked inside one of the build scripts. Full analysis of the payload is still pending, but it appears to have targeted the pre-authentication crypto functions of OpenSSH; it’s probably safe to assume that it added “master key” functionality to let the attackers access all affected servers at will.

Some time after getting the backdoor in, “Jia” — along with a new cast of sock puppet accounts — started pinging Linux distro maintainers to have the backdoored library packaged and distributed to end users. The scheme worked until Andres Freund — a PostgreSQL developer in the employ of Microsoft — reportedly decided to investigate some unexpected SSH latency caused by a minor bug in the backdoor code.

If this timeline is correct, it’s not the modus operandi of a hobbyist. In today’s world, if you have the technical chops and the patience to pull this off, you can easily land a job that would set you for life without risking any prison time. It’s true that we also have some brilliant folks with sociopathic tendencies and poor impulse control — but almost by definition, such “black hat” groups seek instant gratification and don’t plan heists years in advance. In other words, all signs point to this being a professional, for-pay operation — and it wouldn’t be surprising if it was paid for by a state actor.

With attribution up in the air, it’s still tempting to assign blame. Some pundits are pointing fingers at the supposedly exploitative relationship between Big Tech and the open source community; they claim that the lack of adequate compensation is the source of all malaise. I don’t buy this. The relationship with commercial vendors isn’t always healthy, but many major OSS projects are supported to a significant extent. Countless prominent OSS developers are on Big Tech payroll; quite a few projects receive hefty grants.

The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.

Heck, the same happens on the other side of the equation: even with Big Tech staffing and money, if you have a library that almost never needs any attention, the “ownership” of that code becomes pretty theoretical too. It’s hard to build a rewarding career on being very familiar with some boring, old dependency that’s just taken for granted by everyone else.

More fundamentally, the xz backdoor isn’t a technical problem and it probably can’t be solved with technology alone. To a large extent, it’s a counterintelligence challenge — squarely within the competencies of governments and a handful of commercial entities with ecosystem-wide surveillance capabilities. This notably includes Google and Microsoft.

In fact, here’s an interesting thought: perhaps they have known for a while. Would we be able to tell the difference between a carefully-timed disclosure — presumably engineered to conceal “methods and sources” — and a serendipitous discovery?

Subscribe now