Use an llm to automagically generate meaningful git commit messages

Neat, thoroughly documented recipe by Harper Reed using my LLM CLI tool as part of a scheme for if you're feeling too lazy to write a commit message - it uses a prepare-commit-msg Git hook which runs any time you commit without a message and pipes your changes to a model along with a custom system prompt.

Last week, the internet dodged a major nation-state attack that would have had catastrophic cybersecurity repercussions worldwide. It’s a catastrophe that didn’t happen, so it won’t get much attention—but it should. There’s an important moral to the story of the attack and its discovery: The security of the global internet depends on countless obscure pieces of software written and maintained by even more obscure unpaid, distractible, and sometimes vulnerable volunteers. It’s an untenable situation, and one that is being exploited by malicious actors. Yet precious little is being done to remedy it.

Programmers dislike doing extra work. If they can find already-written code that does what they want, they’re going to use it rather than recreate the functionality. These code repositories, called libraries, are hosted on sites like GitHub. There are libraries for everything: displaying objects in 3D, spell-checking, performing complex mathematics, managing an e-commerce shopping cart, moving files around the internet—everything. Libraries are essential to modern programming; they’re the building blocks of complex software. The modularity they provide makes software projects tractable. Everything you use contains dozens of these libraries: some commercial, some open source and freely available. They are essential to the functionality of the finished software. And to its security.

You’ve likely never heard of an open-source library called XZ Utils, but it’s on hundreds of millions of computers. It’s probably on yours. It’s certainly in whatever corporate or organizational network you use. It’s a freely available library that does data compression. It’s important, in the same way that hundreds of other similar obscure libraries are important.

Many open-source libraries, like XZ Utils, are maintained by volunteers. In the case of XZ Utils, it’s one person, named Lasse Collin. He has been in charge of XZ Utils since he wrote it in 2009. And, at least in 2022, he’s had some “longterm mental health issues.” (To be clear, he is not to blame in this story. This is a systems problem.)

Beginning in at least 2021, Collin was personally targeted. We don’t know by whom, but we have account names: Jia Tan, Jigar Kumar, Dennis Ens. They’re not real names. They pressured Collin to transfer control over XZ Utils. In early 2023, they succeeded. Tan spent the year slowly incorporating a backdoor into XZ Utils: disabling systems that might discover his actions, laying the groundwork, and finally adding the complete backdoor earlier this year. On March 25, Hans Jansen—another fake name—tried to push the various Unix systems to upgrade to the new version of XZ Utils.

And everyone was poised to do so. It’s a routine update. In the span of a few weeks, it would have been part of both Debian and Red Hat Linux, which run on the vast majority of servers on the internet. But on March 29, another unpaid volunteer, Andres Freund—a real person who works for Microsoft but who was doing this in his spare time—noticed something weird about how much processing the new version of XZ Utils was doing. It’s the sort of thing that could be easily overlooked, and even more easily ignored. But for whatever reason, Freund tracked down the weirdness and discovered the backdoor.

It’s a masterful piece of work. It affects the SSH remote login protocol, basically by adding a hidden piece of functionality that requires a specific key to enable. Someone with that key can use the backdoored SSH to upload and execute an arbitrary piece of code on the target machine. SSH runs as root, so that code could have done anything. Let your imagination run wild.

This isn’t something a hacker just whips up. This backdoor is the result of a years-long engineering effort. The ways the code evades detection in source form, how it lies dormant and undetectable until activated, and its immense power and flexibility give credence to the widely held assumption that a major nation-state is behind this.

If it hadn’t been discovered, it probably would have eventually ended up on every computer and server on the internet. Though it’s unclear whether the backdoor would have affected Windows and Mac, it would have worked on Linux. Remember in 2020, when Russia planted a backdoor into SolarWinds that affected 14,000 networks? That seemed like a lot, but this would have been orders of magnitude more damaging. And again, the catastrophe was averted only because a volunteer stumbled on it. And it was possible in the first place only because the first unpaid volunteer, someone who turns out to be a national security single point of failure, was personally targeted and exploited by a foreign actor.

This is no way to run critical national infrastructure. And yet, here we are. This was an attack on our software supply chain. This attack subverted software dependencies. The SolarWinds attack targeted the update process. Other attacks target system design, development, and deployment. Such attacks are becoming increasingly common and effective, and also are increasingly the weapon of choice of nation-states.

It’s impossible to count how many of these single points of failure are in our computer systems. And there’s no way to know how many of the unpaid and unappreciated maintainers of critical software libraries are vulnerable to pressure. (Again, don’t blame them. Blame the industry that is happy to exploit their unpaid labor.) Or how many more have accidentally created exploitable vulnerabilities. How many other coercion attempts are ongoing? A dozen? A hundred? It seems impossible that the XZ Utils operation was a unique instance.

Solutions are hard. Banning open source won’t work; it’s precisely because XZ Utils is open source that an engineer discovered the problem in time. Banning software libraries won’t work, either; modern software can’t function without them. For years security engineers have been pushing something called a “software bill of materials”: an ingredients list of sorts so that when one of these packages is compromised, network owners at least know if they’re vulnerable. The industry hates this idea and has been fighting it for years, but perhaps the tide is turning.

The fundamental problem is that tech companies dislike spending extra money even more than programmers dislike doing extra work. If there’s free software out there, they are going to use it—and they’re not going to do much in-house security testing. Easier software development equals lower costs equals more profits. The market economy rewards this sort of insecurity.

We need some sustainable ways to fund open-source projects that become de facto critical infrastructure. Public shaming can help here. The Open Source Security Foundation (OSSF), founded in 2022 after another critical vulnerability in an open-source library—Log4j—was discovered, addresses this problem. The big tech companies pledged $30 million in funding after the critical Log4j supply chain vulnerability, but they never delivered. And they are still happy to make use of all this free labor and free resources, as a recent Microsoft anecdote indicates. The companies benefiting from these freely available libraries need to actually step up, and the government can force them to.

There’s a lot of tech that could be applied to this problem, if corporations were willing to spend the money. Liabilities will help. The Cybersecurity and Infrastructure Security Agency’s (CISA’s) “secure by design” initiative will help, and CISA is finally partnering with OSSF on this problem. Certainly the security of these libraries needs to be part of any broad government cybersecurity initiative.

We got extraordinarily lucky this time, but maybe we can learn from the catastrophe that didn’t happen. Like the power grid, communications network, and transportation systems, the software supply chain is critical infrastructure, part of national security, and vulnerable to foreign attack. The U.S. government needs to recognize this as a national security problem and start treating it as such.

This essay originally appeared in Lawfare.

Well — we just witnessed one of the most daring infosec capers of my career.

Here’s what we know so far: some time ago, an unknown party evidently noticed that liblzma (aka xz) — a relatively obscure open-source compression library — was a dependency of OpenSSH, a security-critical remote administration tool used to manage millions of servers around the world. This dependency existed not because of a deliberate design decision by the developers of OpenSSH, but because of a kludge added by some Linux distributions to integrate the tool with the operating system’s newfangled orchestration service, systemd.

Equipped with this knowledge about xz, the aforementioned party probably invented the persona of "Jia Tan” — a developer with no prior online footprint who materialized out of the blue in October 2021 and started making helpful contributions to the library. Up to that point, xz had a single maintainer — Lasse Collin — who was dealing with health issues and was falling behind. Shortly after the arrival of “Jia”, several apparent sock puppet accounts showed up and started pressuring Lasse to pass the baton; it seems that he relented at some point in 2023.

Since then, “Jia” diligently continued the maintenance work — culminating in February 2024 with the seamless introduction of a sophisticated, well-concealed backdoor tucked inside one of the build scripts. Full analysis of the payload is still pending, but it appears to have targeted the pre-authentication crypto functions of OpenSSH; it’s probably safe to assume that it added “master key” functionality to let the attackers access all affected servers at will.

Some time after getting the backdoor in, “Jia” — along with a new cast of sock puppet accounts — started pinging Linux distro maintainers to have the backdoored library packaged and distributed to end users. The scheme worked until Andres Freund — a PostgreSQL developer in the employ of Microsoft — reportedly decided to investigate some unexpected SSH latency caused by a minor bug in the backdoor code.

If this timeline is correct, it’s not the modus operandi of a hobbyist. In today’s world, if you have the technical chops and the patience to pull this off, you can easily land a job that would set you for life without risking any prison time. It’s true that we also have some brilliant folks with sociopathic tendencies and poor impulse control — but almost by definition, such “black hat” groups seek instant gratification and don’t plan heists years in advance. In other words, all signs point to this being a professional, for-pay operation — and it wouldn’t be surprising if it was paid for by a state actor.

With attribution up in the air, it’s still tempting to assign blame. Some pundits are pointing fingers at the supposedly exploitative relationship between Big Tech and the open source community; they claim that the lack of adequate compensation is the source of all malaise. I don’t buy this. The relationship with commercial vendors isn’t always healthy, but many major OSS projects are supported to a significant extent. Countless prominent OSS developers are on Big Tech payroll; quite a few projects receive hefty grants.

The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.

Heck, the same happens on the other side of the equation: even with Big Tech staffing and money, if you have a library that almost never needs any attention, the “ownership” of that code becomes pretty theoretical too. It’s hard to build a rewarding career on being very familiar with some boring, old dependency that’s just taken for granted by everyone else.

More fundamentally, the xz backdoor isn’t a technical problem and it probably can’t be solved with technology alone. To a large extent, it’s a counterintelligence challenge — squarely within the competencies of governments and a handful of commercial entities with ecosystem-wide surveillance capabilities. This notably includes Google and Microsoft.

In fact, here’s an interesting thought: perhaps they have known for a while. Would we be able to tell the difference between a carefully-timed disclosure — presumably engineered to conceal “methods and sources” — and a serendipitous discovery?

Subscribe now

Okay so, we all know how the Earth ends, right? In six billion years or so, the Sun swells up into a red giant, and the Earth gets melted. Pretty straightforward.

But it turns out that /life/ on Earth will end long before that. There are reasons to think that the biosphere will collapse about a billion years from now — long enough!  But still long before the planet itself gets melted.

Why? Basically two reasons.

First, long before the Sun blows up into a red giant, it will get brighter.  You remember from grade school how the Sun is fusing hydrogen into helium to produce energy?  Well, helium is denser than hydrogen.  So over time, as helium builds up, the Sun’s core gets denser.  And — up to a point — denser means hotter, and hotter means more energy.  So over geological time the Sun very slowly gets brighter.  “Very slowly” here meaning, about 1% brighter every hundred million years.  Stegosaurus lived under a sun that was about 1% dimmer.

Okay, so in about a billion years the Sun will be around 10% brighter.  As it turns out, that’s about the maximum extra energy the biosphere can absorb.  Beyond that, we start to get a runaway greenhouse effect, and the Earth turns into another Venus.  The error bars on the models are pretty large, but it looks like, yeah, about a billion years.

(Just to be clear: this has nothing to do with climate change / global warming.  Climate change is not going to turn us into Venus, a planet with a dense poisonous atmosphere whose surface is hot enough to melt lead.  There is simply no way we can pump enough greenhouse gas into the atmosphere to do that.  Melt the ice caps, sure.  Raise sea levels catastrophically, expand deserts, cause a mass extinction, sure sure.  Render the planet uninhabitable, no.  Totally different thing.)

By a curious coincidence, there’s a completely different process that will /also/ crash Earth’s biosphere in about a billion years, give or take.  That is the expansion of the Earth’s solid core.

Again, back to grade school: remember that diagram of the Earth’s interior in your textbook?  Solid iron core in the center (“the size of the Moon, as hot as the Sun”), surrounded by a liquid “outer core” of hot molten iron, all wrapped in a “mantle” of gooey half-molten rock?  Okay, so the interior of the Earth is fantastically hot.  But that heat slowly escapes through volcanoes and earthquakes, meaning that over geological time the Earth is cooling down.  This cooling gets expressed as a slow, gradual expansion of the Earth’s solid core.  Basically the solid inner core is growing by about 1 millimeter per year — a meter per millennium, a kilometer every million years — into the liquid outer core.

(“But doesn’t that mean the solid core was smaller in the past?”  Yes it sure does!  In fact, the Earth’s solid core is a lot younger than the Earth.  For its first couple of billion years of existence, Earth probably didn’t have a solid core at all.)

Here’s the thing: the liquid outer core is where Earth’s magnetic field is generated.  The details are complex, but basically there are convection currents moving up and down in the molten iron, and that’s where the magnetic field comes from.  But once the Earth’s liquid core gets too shallow, those currents will break down.  Current models suggest that will happen around a billion years from now, give or take.

No magnetic field means no protection from various sorts of horrible radiation coming in from space.   That’s not necessarily game over for the biosphere, but it’s very bad news for anything that doesn’t live underground or underwater.

Again, the error bars are large, and we’re talking about /around/ a billion years.  So there’s still time to clean your attic, yeah?

But!  Here’s a fun complication.  I mentioned the solid iron core is growing, right?  Okay, so just how does it grow?  Well, apparently it grows like a glacier grows, from the accretion of snow.  Crystals of solid iron “snow”, forming in the liquid outer core, softly snowing down.  This paper here looks at the details of that mechanism.  What they find is that snow occurs at different places and different times throughout the liquid core:  “crystallization and the associated buoyancy flux would be strongly heterogeneous in time and space”.  Most of the time, the snow melts again before it reaches the core.  (This happens in Earth’s atmosphere, too.  Ever seen a distant cloud with rain or snow coming out of it, but disappearing before hitting the ground?  The technical term is “virga“; you might have heard them called “jellyfish clouds”.)

The fun complication is… Earth’s core probably has *weather*.  Makes sense, right?  It’s a fluid with lots of heat energy moving through it, plus also it’s rotating.  Differential density, Coriolis force, and oh yeah magnetism probably plays a role.  The Earth’s liquid core has convection cells and iron snow.  And quite possibly it also has storms, some equivalent to thunderstorms and hurricanes.  Except this is happening a couple of thousand miles beneath your feet and, you know, in a dark sea of molten iron. 

And where there’s weather there may be climate.  This sort of thing may explain why the Earth’s magnetic field wanders around, occasionally hiccups a little, and every few hundred millennia simply flips right over.  This is an active field of study right now.

But anyway!  Iron snow, beneath your feet.

💬 Join the discussion on kottke.org →

Anderson’s experiments with language-based AI predated ChatGPT – with one machine modelled on her dead husband that her friends ‘just can’t stand’

• Get our weekend culture and lifestyle email

There’s a 2013 Black Mirror episode in which a young widow played by Hayley Atwell signs up to an online service that scrapes a person’s entire digital footprint to create a virtual simulation. She soon starts chatting online with her late husband (Domhnall Gleeson), before things inevitably get Black Mirror-y.

Laurie Anderson, the American avant garde artist, musician and thinker, hasn’t seen the episode but, in the last few years, has lived a version of it: growing hopelessly hooked on an AI text generator that emulates the vocabulary and style of her own longtime partner and collaborator, Velvet Underground co-founder Lou Reed, who died in 2013.