If you’re not familiar with the Soham Parekh story, Suhail Doshi put out a PSA that Soham has been running an overemployment scam by simultaneously working at multiple startups. He did this remotely from India. TechCrunch reports that he had been sweetening the deal by taking low salary and high equity at each of the jobs. He was caught once members of the Y Combinator community started checking each others notes.

A Hacker News thread on the topic was flooded with people saying that they hired him. In summary, he excels at LeetCode-style interviews. But once it came time to work, he would give wild excuses about why he couldn’t meet deadlines. He would skip meetings, have massive delays for shipping pull requests, and get nothing done until he was eventually terminated.

I think we can do better. He repeatedly got caught by companies, and since his compensation packages were always equity-heavy — which typically vests with a 1 year cliff — he was getting underpaid per job.

Let’s break down his strategy:

• Focus on startups. These are companies that are growing, hungry to hire, and are small enough to make rapid decisions.

• Be a good candidate. Master the skill of interviewing. Convince people that you are one of the top candidates.

• Look enticing. Proactively ask for a lowball salary offer in exchange for equity.

• Add uncertainty. Claim that you are working on your US visa renewal, but raise some doubt about whether it will go through.

• Make a good first impression. Attend any in-person meetings or orientations.

• Surf the long tail. Obviously your work suffers because you’re working at 4ish places at once. The company notices eventually and fires you. Cash the paychecks and start interviewing somewhere else.

If you truly want to become overemployed in the tech industry, there are certainly better ways to do this.

First, trying to fleece startups is a bad idea. They actually need things to get done! And everyone knows everyone in a small company, and it will quickly become clear that you’re not producing any work. As companies grow, they develop more places to hide. I know several people who ended up in situations at Google where they didn’t need to do anything for 3-6 months. I’m curious what the record is. It was hard to engineer; usually it happened when a large team catastrophically failed and the organization started picking apart its remains.

Second, the output of his work could be compared against the output of other engineers in the company. He did full stack work, but startups employ tons of people who know about full stack work. So his output isn’t keeping up and anyone can load up Github and see that he is slacking.

So how can we do better? Being a consultant in a third-party integration tech like Salesforce and targeting businesses with somewhere in the ballpark of 100-1000 employees. If you do this right, you could hold at least 2 gigs at the same time while raking in retainers. Yes, this does imply that you need to become an expert in Salesforce, but hear me out. This company size is the sweet spot where they might have decided that they definitely need Salesforce, but realized that they don’t have the in-house expertise to do a Salesforce integration and had bad experiences trying to pay Salesforce contracting fees to do parts of the integration.

First, you’re not going to do the integration yourself. You’re going to be the subject matter expert on a team that is building the integration so that there is no knowledge lost when you leave the team. You are going to host training sessions and produce documentation about how things should be done, but obviously you’ll leave out enough details that there’s room for error. You’re going to figure out what engineers on the project are completely clueless with Salesforce and assign them those parts of the project. Give yourself deliverables that are small and achievable. So at each meeting everyone will be underwater and you will have your part done and have suggestions for how everyone can proceed — namely, by telling them everything you left out of the training and documents. If people ask you for help, just schedule a training 2 days into the future and invite everyone. You’re just always trying to produce artifacts, and a meeting is a good artifact.

The goal is to drag the project out while doing minimal work and giving it the illlusion of forward progress.

Towards the end of the project, make sure that everyone on the team is trying to use Agentforce. That should create some great messes that you can fix.

And now the magic: your retainer! This will require some experimentation, but you want the company to buy as many hours as possible while using as few hours as possible. You might think, “I don’t want to sell 2 days per month and then have to work that,” but don’t be afraid! You already set the precedent that Salesforce work is really slow. You won’t be doing much, and if they give you something real to do, you can sign a separate contract for the overflow work.

Your goal is to pick up as many retainers as possible and have to work as few of them as you can. When you first start doing this gig, you might have to put some work in to generate buzz. You’ve gotta jump start the consulting pipeline somehow. But you will eventually find companies that sign your retainer contracts and keep you around for a year or two — or more.

The news comes on the heels of a major announcement from Zuckerberg. On Monday, the Meta CEO sent a memo to staff introducing the company’s new superintelligence team, which will be helmed by Alexandr Wang, formerly of Scale AI, and Nat Friedman, who previously led GitHub. The list of new hires also included a number of people from OpenAI, including Shengjia Zhao, Shuchao Bi, Jiahui Yu, and Hongyu Ren. OpenAI’s chief research officer, Mark Chen, told staff that it felt like “someone has broken into our home and stolen something.”

On his brother’s podcast, Sam Altman says that the offers have approached $100 million per year, a number also reported by Wired.

Every single article on this has the same 9 facts, and I certainly don’t have anything to add. So let’s go in a different direction: what happened the last time that tech talent poaching was in the news? How did it end up?

The biggest talent poaching spree in recent memory was in the self-driving car industry in the late 2010s1.

Back then, everyone was convinced that self-driving cars were close to being a solved problem, and they were spending as much money as possible to be the industry winner by the beginning of the 2020s.

Google had a self-driving program called Chauffeur. The engineers on Chauffeur believed that their work could be worth billions, so at one point they rebelled. They threatened to leave the company together and get outside venture funding to build their own efforts and reap the rewards themselves.

So Page authorized the creation of a new kind of compensation at Google. The idea was to motivate the team with the sort of incentives they would enjoy at their own startup. Chauffeur's members would remain employees, with regular salaries. The real money came in their bonuses. Every four years, Google would determine the value of the project and pay each teammate a given percentage of the sum, based on their role. Anyone who left before the four-year mark would get nothing.

The problem, of course, is that the valuation for Chauffeur/Waymo skyrocketed. So they got huge payouts (one engineer named Anthony Levandowski2 reportedly got $120 million) and in 2016, with the next payout 3 years into the future, everyone had enough “fuck you” money that there was no reason to wait for the next payout. Many left for more immediate paydays. I heard rumors at the time that some people were just taking their $20 million payouts and retiring, but I can’t find any official sources that confirm that idea. So treat that like the unsubstantiated rumor it is!

Many of the engineers left. For example, Anthony Levandowski left and founded a commercial trucking company called Otto, which was acquired by Uber that same year. A few years later, Uber would end up selling their technology to Aurora, who is — as of 2025 — starting to run autonomous trucking routes.

But yeah, Uber was hellbent on having self-driving technology. They famously partnered with CMU to create an advanced research lab near the campus, and then proceeded to poach 50 people who were involved with CMU’s autonomous vehicles program.

By this point, everyone wanted a piece of the action. Apple poached a Waymo employee. Tesla poached Andrej Karpathy from OpenAI for their own autonomous vehicles efforts. Traditional car companies started feeling like they were falling behind and started acquiring startups, like GM acquiring Cruise for a billion dollars.

At the peak of this jostling, Google accused Uber of coordinating with Anthony Levandowski to steal a bunch of company secrets. Google had proof that Levandowski had downloaded tens of thousands of documents just before leaving Google. A civil suit and a criminal case followed. At the end of it, Waymo got $245 million in Uber stock in the civil trial, and at the end of the criminal trial Anthony Levandowski was sentenced to 18 months in prison.

This is what’s exciting about Meta poaching employees from OpenAI. All of these corporations slugging it out in the news while we get to eat some popcorn and watch. Does past history foreshadow what will happen? Once people get their $100 million payouts, will they feel any pressure to continue working at Meta? Will these companies start accusing each other of steealing industry secrets? No way, this is AI. Something weirder is going to happen. And I’m excited to see what it is.

Here’s the most notable section of the new policy:

Amount of AI Coding

If your purpose is for review or feedback, please be clear about the amount of AI coding used, and if relevant, the amount of effort put into the project, which should be reflected in the project itself.

Using AI coding tools is not a disqualification for posting. However, in order to align the effort of creating a post-worthy project with reviewing it. the subreddit will remove posts for "vibe-coded" projects with little human input. This is not because such projects are "bad", but precisely because they are so easy to put out they are no longer noteworthy.

What happened?

The subreddit has been flooded with low-effort posts recently. Here is a post that list some of the threads that people are complaining about. Since some of the links have been deleted and more may be in the future, they are…

• A “production ready” high-speed logger that couldn’t even be benchmarked because it had a memory leak.

• Something called “hands-on Go” that was an AI-slop repo.

• A terminal-based notetaking app that seems to work, but was posted to Reddit with a LLM summary.

• A monitoring tool with astroturf support. It seems to be a real tool, but again the Reddit post is obviously generated by AI.

• A web framework that looks vibe-coded.

• A “production ready” 7000-line message queue that was ostensibly implemented in a single commit.

What’s interesting is that these all have different degrees of slop. Sometimes the objection is straightforward: the damn thing was obviously AI-genrated and doesn’t work. But sometimes the objection is just that the post to Reddit was written with an LLM. It just goes to show: you can’t seperate technical and social problems1.

This adds problems to one of the most thankless jobs: Reddit moderator2.

LLM-powered chatbots have obviously existed on Reddit as long as LLMs have been available, and bots have been creating accounts and spamming reposts or tried-and-true formulas for as long as I can remember3.

It’s already unpaid, most of your work is invisible, the visible parts of your work mostly come when the community is angry, and Reddit can take your mod position at the drop of the hat. And now you need to scale your effort to catch LLM slop flooding your subreddit, while still dealing with all of your previous moderation duties.

If you’re a moderator of a subreddit like /r/Golang, how much do you care about bots in your comments? If you have a bunch of bots spewing gibberish, that’s pretty bad. But if they’re staying on topic, not making obvious errors, and following the community rules? Is it your job to play Turing Test detective? No. Past a certain quality level, it’s Reddit’s job to stop them and not yours. You’re already doing them a favor by moderating. Also, I speak from 18 years of experience that the bottom of a Reddit thread has always been a hive of scum and villainy. So just by virtue of being a long-time Redditor, mods develop some immunity for whatever trash is happening if the discussion is largely on topic and nobody is getting reported.

However, something has changed with the vibecoding era. LLMs have gotten good enough that they can generate a fully-functional website, library, anything really. They are fantastic at zero-to-one implementation. They can generate the READMEs, they can create documentation, they can write the Reddit post summarizing it. And it’s all terrible.

I like how the /r/golang moderators approached the situation. It’s clear that something needed to be done, and the moderators took a thoughtful approach. It’s also clear that LLMs are here to stay — especially for code generation — so by encouraging projects to be open about their LLM usage they are leaning into the fad instead of trying to uselessly block it.

Google’s Incident Report has a clear explanation of how the incident happened.

On May 29, 2025, a new feature was added to Service Contfrol for additional quota policy checks. This code change and binary release went through our region by region rollout, but the code path that failed was never exercised during this rollout due to needing a policy change that would trigger the code. As a safety precaution, this code change came with a red-button to turn off that particular policy serving path. The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash. Feature flags are used to gradually enable the feature region by region per project, starting with internal projects, to enable us to catch issues. If this had been flag protected, the issue would have been caught in staging.

On June 12, 2025 at ~10:45am PDT, a policy change was inserted into the regional Spanner tables that Service Control uses for policies. Given the global nature of quota management, this metadata was replicated globally within seconds. This policy data contained unintended blank fields. Service Control, then regionally exercised quota checks on policies in each regional datastore. This pulled in blank fields for this respective policy change and exercised the code path that hit the null pointer causing the binaries to go into a crash loop. This occurred globally given each regional deployment.

[…]

Within some of our larger regions, such as us-central-1, as Service Control tasks restarted, it created a herd effect on the underlying infrastructure it depends on (i.e. that Spanner table), overloading the infrastructure. Service Control did not have the appropriate randomized exponential backoff implemented to avoid this.

There has been a lot of discussion online, outlining different ways that the Google engineers must be the worst engineers that have ever walked the earth. I mean, there’s even someone to blame, right? The error handling is bad! There’s no feature flag, and the post promises that a feature flag cannot be misused and fixes problems like this! They didn’t implement randomized exponential backoff! Where are the tests?1

This is a classic blunder. In reality, you lack enough context to make a determination solely by reading the incident log. I’ve talked before about how it’s important to step back and imagine the circumstances surrounding major outages.

What do I mean? Let’s ask ourselves some questions:

• Despite huge investment, Google Cloud is far behind Microsoft Azure and Amazon Web Services in marketshare. How much pressure do you think that executives feel to turn that around relative to prioritizing code quality complaints?

• The incident report identifies this change as part of development of a new feature. Are we 100% sure that this was business as usual, or is it possible that this team, their group, or their product area have been under increased deadline pressure from executives?

• The developers chose to enable this feature via a configuration method that replicates within seconds, instead of going through full rig-by-rig rollouts with feature flags. Were there reasons they might have preferred the faster approach?

• A few days before the incident, Google solicited voluntary buyouts in other organizations within Google. The last time they did this, it preceded layoffs. Can you imagine individuals or teams feeling extra pressure from this, even in a different org? Can you imagine that pressure increasing when hearing people describe the “tough job market” in tech? Do they have reason to believe that their performance rating would be impacted if they didn’t launch these features?

• The post talks about feature flags as a silver-bullet way to avoid an outage. When I was on Google Docs from 2010 to 2015, we shipped several bugs with bad feature flag eligibility logic. Have the feature flags become idiot proof?

• The service did not have proper error handling in this code path. How rare is it for code paths in Service Control to lack proper error handling in this service? Is it normalized to expect that errors will be seen and reverted in minutes with minimal splash damage?

• Was it reasonable for the reviewer and the code author to have any context on whether there was top-level error handling in this part of the service?

• How many techniques are as important as “randomized exponential backoff” in keeping the entire system working? How common is it for these techniques to have uneven implementation coverage across the entire suite of services? As new techniques are identified as critical, are old systems upgraded to use them?

• How easy or hard was it to understand that the service did not implement exponential randomized backoff? Was it controlled by some BCL spaghetti that nobody knew how to read?

Sure, maybe it’s the case that the engineers that made the change are out of line. Maybe the executives were chill, there was no deadline pressure, they recklessly rejected feature flags even though they knew the risk, they are definitely not worried about layoffs and the job market more broadly, feature flags are impossible to misuse, the reviewer and code author went to extraordinary lengths to add a code path that did not have error handling, and specific people have been negligant and failing to implement a single specific technique despite everyone knowing they weren’t doing it.

Personally, it seems unlikely that every single one of these circumstances are true. I find it likely that there was deadline pressure2. I find it likely that they had a lot of competing priorities, and it was difficult to prioritize system modernization/hardening versus new product work. I find it likely that engineering decisions are starting to change in parts of the company, ever so slightly, even if they won’t admit it, as the spectre of layoffs looms over their head.

Are all of these things true? Are any of these things true? I have no idea. I lack the complete story because Google would never put any of it in the incident report, even if they are all relevant to why the outage happened. If an external agency — some kind of FAA for software outages — investigated the incident and identified all of the factors, they would absolutely include all of the surrounding circumstances that included things like executive pressure and job safety concerns.

But again, I know that I lack the complete story. And accordingly I’m going to give everyone the benefit of the doubt until we learn more, if we ever do.

A friend of mine solicited advice for high school students who have the goal of working in tech. I had planned to skip my second post of this week, but I liked the writing prompt. So here you go!

Background: I’m 39. I’m a staff engineer. I’ve been coding since I was 14. I currently work at Hinge, the dating app. Previously I’ve worked at Google, Etsy, and Sarnoff Corporation (now part of SRI). I went to a state school for college, and got internships/gigs at Google Summer of Code (for Boost C++) and Johnson and Johnson.

Everyone will have different experiences and different opinions, but these are mine.

Anyways, in no particular order:

The default advice is still good

If you don’t know how to program, start learning. I am so jealous of the wealth of educational resources available today.

Take a programming class as soon as you can.

In college, try getting an internship in tech. Failing that, try to get the most tech-adjacent job you can. I worked for my school’s IT department until I got real internships. Internships are part of the full-time engineer recruitment process, so do your best to excel if you land one.

You should be able to point to real technical work that you’ve done. Did you make a small game in Unity? You wrote your blog that hosts itself? You made a face recognition app by following a tutorial? That’s awesome! Make a short YouTube video that demos it that you can share with people. You’ll want some way to show that you went above and beyond your coursework, which is a sign that an employee has great potential.

Dive all the way down

Learn everything you can about core computing stacks while you still have dedicated time to learn. There is almost always a positive return on learning about core computing stacks. Learn about CPU architecture, DMA, architectures, programming language design, network protocols, operating systems. I think about these things all the time.

If you have a dream job, make sure it’s real

Do you have a specific job that you want? Then you should be able to prove that it actually exists. Is it a company? Do they hire new grads or interns? Do they only recruit interns from specific schools, or would you have a realistic shot if you applied?

You should be able to find real job postings for entry-level positions. If you can’t, then isn’t that a bad sign? Yes!

Pay attention to industry trends

Trends and fads are real even if they’re temporary. This will shape companies. It affects where funding is allocated. People will be hired and fired because of it. Companies will exist that do it. In 4 years there will be something else. There always is. Pay attention to the industry.

For example, the big trend right now is AI, and especially LLMs and agents. In 10 years it’ll be something else. If you’re in college and hoping to get an internship, you should be able to explain to an interviewer why LLMs and agents are in the news so much. Ideally you’d have some experience with them.

Increase your luck surface

Try presenting your projects at meetups. Talk to people about them. Participate in online communities. Get to know your professors and learn which ones have industry contacts. Before I graduated, I had 2 different full-time job offers because professors referred me to companies in the area. I got my job at Etsy because someone saw a talk I gave about a colorblind-simulating Twitterbot I wrote.

There are many dimensions along which you can increase your luck surface. One of the best is to attach yourself to something that’s growing. It can be an industry, it can be a company, it can be anything really. But if something is growing then you must grow along with it.

At the end of the day, the computer industry is made up of people. If you get someone excited at the prospect of hiring you, don’t you think they’d rather look at your resume than the 1200 LLM submissions they got when they posted their job on the job board?

A lot of increasing your luck surface will involve encountering roadblocks and failure. That’s okay. Try to learn from every interaction.

Be humble

The instant you stop learning is the instant that you start to become obsolete. You will need to learn and relearn in this industry. Many people have something to teach you. Figure out what it is. Get a brain dump from them. I do this all the time.

There’s a whole world outside of tech

When I remember my past, I remember the moments I experienced and not the tech that I built. Even when I think about work, I think about the moments that I had with my coworkers while we built something.

There’s an entire world outside of your computer monitor. There are a practically infinite number of things you could do. Travel. Bike. Run. Go to watch parties. Go to a trivia night. Fall in love.

If you have spare credits in college, prefer to take the class that is interesting instead of the class that is easy.

To restate an earlier point

There is no substitute for comprehension. Comprehension is not a substitute for mastery.

Programming note: If you don’t know a lot about Golang, check out the Appendix below for all the background you need.

Go recently announced that they will not pursue syntactic improvements for error checking. They managed to do this without addressing the elephant in the room, which is that LLM code generation has gotten to the point where syntactic sugar doesn’t matter as much as it used to.

[ On | No ] syntactic support for error handling

Still, no attempt to address error handling so far has gained sufficient traction. If we are honestly taking stock of where we are, we can only admit that we neither have a shared understanding of the problem, nor do we all agree that there is a problem in the first place. With this in mind, we are making the following pragmatic decision:

For the foreseeable future, the Go team will stop pursuing syntactic language changes for error handling. We will also close all open and incoming proposals that concern themselves primarily with the syntax of error handling, without further investigation.

There are also a few introspective questions that have not featured heavily in the Online Discourse, but are actually really important to hammer out.

We don’t really know how much the issue is the straightforward syntactic verbosity of error checking, versus the verbosity of good error handling: constructing errors that are a useful part of an API and meaningful to developers and end-users alike. This is something we’d like to study in greater depth.

This is a more important point that is lost in the discussion. Do errors fail to serve both end-users and developers? If you’ve used a lot of third-party libraries in Golang, you know that the answer is “lol yes.” Since Go errors are just types, you’re at the mercy of every single layer of your stack to thoughtfully provide and handle errors. This includes all of the layers that your dependencies transitively pull in. Nothing more fun to get a random error bubbling up from a library — stack trace not included — that says something like “config error” and you’re just screaming “What failed to initialize? Do I need to set another config option? Is something misspelled? What did you see?”

If you think to yourself, “why don’t they just do $myfavoritesolutiontothisproblem,” I highly recommend skimming the umbrella issue that Ian Lance Taylor assembled to gather more serious proposals into one place. Why do they need an umbrella issue? Because people keep coming up with the same proposals over and over again. They’ve been debated. They’re all missing something.

I’ve only been a professional Golang coder for 2 years, but I’ve been coding Golang since before the 1.0 release. It’s my favorite language for web servers.

I was hyped for the initial check proposal. But now, with the advent of LLM tools, it doesn’t feel that pressing.

Go is a great language for code generation. I’m not the only one who has noticed this. LLMs are excellent at automatically providing the correct error handling for the context. Do you just pass them through? Just press Tab. Do you wrap them? Just press tab.

So now you get the best of all worlds: the error handling is out in the open, and the LLM generates all of the checks. Very little effort is actually expended “writing” the code. In a world where it takes the same amount of effort to type ? as it does to type 4 lines of error handling, do you really want to waste your time with syntactic proposals?

I did wonder briefly why the Go team didn’t mention this. But pretend that you’re designing a language. It would actually be weird if you did mention it! You’re just desining a language. The tooling that exists to generate the language might not exist in 5 years. What if everyone decided that AI was a money-losing investment and LLM code generation tools started being charged at cost instead of subsidized to win marketshare? That puts you in an awkward position as a language designer. You designed your language with the assumption that tooling would go in a specific direction, and now it’s going in another direction.

So obviously, you just put out a blog post that says “we’re not going to talk about syntactic changes anymore.”

Appendix: background for non-Go programmers

Are you a programmer that doesn’t know Golang? Here’s all the context that you need!

Go has two different types of errors: panics and errors.

panic is an unchecked exception. It behaves how you might expect: the stack is automatically unwound until either the panic is handled or it reaches the top level and the application is terminated. To give webservers as an example, you might wrap all of your HTTP handlers in a function that recovers from the panic, logs the error associated with the panic, and then returns a “500: Internal Server Error” to the request.

Go intends these to be reserved for truly exceptional events, like dereferencing nil pointers.

Nothing stops you from using these as exceptions in your program. However, if you panic() for a non-fatal error, the Go community will call your code “not idiomatic.” When the Go community says “not idiomatic,” they are punishing you. This is meant to be on par with excommunication from your church, or being left alone naked on a desert island. All of this is to say: you’re only supposed to panic for truly fatal problems.

The other type of error return is the error interface. This is an extra parameter that is returned from every function. By convention, it is the last return type. These are just types that implement an interface and don’t have any special handling. There are even some fun “gotcha!”s around Go’s nil interface semantics.

By design, Go’s errors are intended to be checked with manual if checks. There is no linguistic support for this; just use the regular control flow structures available in the language. In practice, almost every check just looks like this:

a, err := somepackage.SomeFunction()
if err != nil {
    return nil, err
}

Of course, the handling can get arbitrarily complex. For example, you might examine the error type to determine whether an I/O error is fatal or maybe the request can be retried. Or you might return a wrapped error to accumulate a stack trace for yourself2.

In theory you could forget to check the errors, but there are linters that make you.

So, dear programmer that doesn’t know Go, the question facing the community is: “should errors in Go be handled like they were normal types, or should errors have extra language support?”

• They changed their config format.

• They removed support from really old Node.js versions.

• They changed defaults in their config file format.

• They changed a lot of stuff in their rules engine.

This was a hard upgrade. Since there were so many breaking changes at once, it was difficult to pinpoint a single problem. It’s not hard to find Reddit or Hacker News threads of people complaining about it, but I would just skim Yacine Kharoubi’s blog post about upgrading. The upgrade can be extensive depending on your setup.

Recently, ESLint released a retrospective on the upgrade, with the benefit of 1 year of time between the initial release and now. The retro obviously focuses on the ESLint project, but there are great takeaways even if you’re not a frontend developer.

Users are lazy

Since the upgrade was so extensive, it affected multiple classes of users:

1. People who produce ESLint rules and plugins.

2. End users that run ESLint as part of their work, which may also include rules and plugins.

ESLint bet that the plugin and rule developers would be proactive about the change, but expected the end users to lag more. But in reality, nobody was.

There are a trillion concepts that say the same thing. “Inertia.” “Users don’t read.” “Defaults are powerful.” There’s an argument that a proactive plugin developer could get involved early in the development process and provide good advice.

But in reality, who wants to develop against an alpha API? Can you imagine spending a few weeks upgrading your code to account for some hairy change, and then the implementation detail changes and you need to spend that few weeks again? I think the ESLint developers deserve a lot of credit for trying to approach the upgrade thoughtfully, but they would be wise to pay attention to this in future upgrades that affect rules and plugins.

Stay the course

I really appreciated that they stuck to their initial migration timeline.

Some suggested delaying the v9.0.0 release to give the ecosystem more time to catch up. We decided against this for several reasons:

• Users weren’t required to upgrade immediately. ESLint v8.x remained fully compatible with the ecosystem and continued receiving bug fixes, so those who didn’t want to upgrade could continue using it.

• It was unclear how long such a delay would last. How could we determine when the ecosystem had “caught up”? Should we keep v9.0.0 in limbo while only providing bug fixes for v8.x indefinitely? That didn’t seem like a viable solution.

• The @eslint/eslintrc package already offered substantial compatibility for eslintrc plugins and shareable configs with v9.0.0, addressing the most common issues we encountered.

• We had communicated the upcoming changes for 18 months, with increasing reminders as we neared the first alpha release. Although adoption was slower than expected, we saw momentum building and wanted to maintain that pace. Delaying the release could have sent the wrong message and allowed further delays to snowball.

They faced pressure to delay once the community started to popularize the idea that the upgrade was difficult. But without the launch deadline, there wouldn’t be any pressure applied to the lazy developers. Most people would just wait until they were forced to do the upgrade.

Support mitigates pain

A lot has been written about how painful this upgrade is. But first, I want to highlight how thoughtfully the ESLint team approached the upgrade. First, they had a policy of updating the migration guide in the same pull request where changes were introduced.

We also introduced a new process: the v9.0.0 migration guide was updated in the same pull request as each new feature. Previously, we wrote the guide after all features had been merged, increasing the chance of omissions. This new approach helped ensure nothing was missed.

They also had a great support story for their community. They supported version 8 and 9 side-by-side for six months. After that, they had a formalized support policy and an official commercial partner for companies that needed support for version 8 longer than ESLint could provide it. They also improved the migration tools, documentation, and front-line support over time. This allowed them to make migrations easier for people who adopted later.

Six months is not a lot of time in the grand scheme of things. Hell, Python 2 and Python 3 coexisted for a decade. But maintaining out-of-date versions is a major drain on resources, especially for major releases like this. So I commend them for defining an explicit support path.

Avoid bundling lots of breaking changes together

It’s easy to argue in favor of lumping breaking changes together. “You only go through the pain once. We don’t want to have a reputation of always breaking everything on releases. We can always support the two side-by-side for a specific period of time.” The ESLint project even had their own project-specific reasons. “Well, we can’t launch language plugins without all of these features.”

Too many breaking changes

The biggest mistake was bundling too many breaking changes into a single major release. A key example was introducing the new configuration system alongside the rule API changes, both of which were necessary steps to enable language plugins. This often made it difficult to pinpoint the cause of issues with existing plugins. Many assumed the new configuration system was to blame, creating a narrative that it was “broken” or “not ready.” In reality, the rule API changes were just as disruptive.

We were so focused on the configuration system rollout that we underestimated the impact of the rule API changes.

This is a good reminder that the project is about the users. What incentivizes them to upgrade? What do they gain out of it? Is it a slog that they are doing for our benefit? Are the benefits concrete for us and abstract for them?

Starting this newsletter was a big deal for me. It was the first serious hobby I had since my daughter was born in 2023. And even now, I only get an hour or two per day to myself. But I’ve still managed to publish one to two times per week.

I’m so grateful for all of my readers. To pay it back, this post has everything that I’ve learned about running a newsletter on minimal free time. Even if you can only cobble together a few hours a month, you can still publish one post per month.

“How do you know that I have the time to run a newsletter? You don’t know my life.” Buddy, if you have the time to argue with strangers on the internet, I don’t want to hear it.

“Why would I start a newsletter?” That’s a separate post. Right now, I’m assuming that you are motivated and just want a playbook. Anyways, here we go!

Where should I publish my work?

You have limited time. Pick an existing platform instead of trying to write your own. You have plenty of options: Substack, Ghost, Wordpress, Medium, etc. Just do some basic research. For example, Substack has good discovery features that can get you extra subscribers, but its code blocks are terrible. So if you were going to share a lot of code in your posts, you’ll want to use a different platform.

Who is my audience?

The most important thing isn’t “what,” but “who.” Pick an audience and write each post to them. I personally imagine a junior engineer when I’m writing. They understand the fundamentals. But I have experience that they don’t, so it provides a natural premise for explanation. It’s also fun for more senior people to follow along. Finance nerds subscribe to Money Stuff even though it contains a lot of really basic explanations. Similarly, technical people aren’t scared away when the target audience is more junior than them.

How should I get my topics?

News is the best story source. There’s always something new happening. There might be a slow news week, but it’ll always pick up again.

You will want to scan for stories as quickly as possible. Here’s what you’re going to do:

• Make a list of all of the subreddits, social media accounts, industry publications, etc that you can think of. Basically anything that has news on your beat.

• Whenever you start writing, open up every single one of these sources.

• Is there anything that you find interesting? Is anything funny? Is there anything that you know more than average about? These are great story opportunities.

Of course, there are plenty of things to write about besides news. You could pick a facet of a technology that you find interesting. You could tell a war story. Whatever it is, you want to keep the topic focused. Don’t write a complete guide to every frontend technology; you’ll never finish. But is there an interesting use case for HTMX that people might not realize? A cool trending GitHub project that people don’t know about yet? Those sound achievable in a few hours.

Another useful format I’ve found is “be the guy who actually reads things.” For example, for this post on Google’s illegal ad monopoly, I went through the court filing, took the time to understand the arguments in the case, and then explained why it was interesting to the reader.

How do I write a post?

Write an outline. It’s the first thing you should do. I write most of my posts across two days. On the first day, I research topics and then write an outline. The next day, I turn the outline into a post, and then schedule it for the next morning and go to bed.

There are two reasons that you should write an outline. First, moving items around a bulleted list is really easy. Rewriting a few pages of text is a waste of time. Second, I find that the outline loads the topic into my brain. When I finally sit down to write the post, it’s been rattling around my brain for about 24 hours. The writing flows much more easily.

How do I promote a post?

This is the part that I’m worst at. Some weeks I just don’t have the energy to write AND promote, and the promotion falls by the wayside. But every single subscriber has found my newsletter because I promoted it, so it’s clearly important.

First, assemble a list of possible places to post. I have tried posting on Hacker News, Reddit, Mastodon, Bluesky, and LinkedIn. Obviously there are an almost unlimited number of sites and aggregators you could post to.

Next, don’t wear out your welcome in any of them. Most places have clearly-posted rules about self promotion. Follow them. Be a good citizen. Don’t get banned. Over time, you’ll get a feel for what posts are appropriate for what places.

You should also pay attention to any promotional tools that your platform offers. I’ve gotten some subscribers from Substack’s Notes feature, even though I’m terrible at using them.

What would I do if I had more time?

I would likely have one more post a week. I’ve also played with the idea of having 3 posts per week without having more time. I’ve been mulling over the logistics of having a news roundup issue every Sunday, and moving my first weekly post to Tuesday. For a while I was dissuaded because Pragmatic Engineer has one. But my links would be different and I would have different things to say, so maybe it’d be worth it?

I would be more diligent about self-promotion. First, I would set up Client/Server social media accounts and automate posting across all of them. Same with Substack notes. I would set up extra content messages to be posted on days that I don’t publish content on the main newsletter.

I would start looking for cross-collaboration opportunities with other newsletters.

What else should I know?

Rapid fire:

• Tell people that you are doing it. Post your first post to your LinkedIn and Facebook. Mention it at lunch. Tell your friends. I feel blessed by how many people I know have told me that they like my posts, or that they think that I’m doing a good job. The people around you are a built-in source of encouragement. Use them.

• Most of your posts are going to be duds. But the posts that do blow up will be huge. I have 43 posts, and only six posts have over 1000 views. The two biggest posts have around 50,000 views each.

• Every post is a reminder to unsubscribe. You should prefer not clicking “publish” over publishing a half-baked story.

• You won’t always have a topic to write about. Sometimes it’s just a slow news week. If you want a challenge, try to find a new format and see if you’re happy with the results. Otherwise, just skip this issue.

• Ask people to subscribe. Put a “Subscribe” call-to-action somewhere in your Substack post1.

• You can use LLMs as much or as little as you want. It’s your site. For Client/Server, I write every single word in the body of the posts. My personal viewpoint is “what would be the point if a robot did my hobby?” I do use ChatGPT to help me brainstorm titles and subheads. I tried using it for proofreading and was underwhelmed.

A number of coding agents have been released since the beginning of the year. If you’re curious you can check out some of the announcements here:

• Jules is Google’s effort.

• Codex is OpenAI’s effort.

• Copilot agent mode is GitHub/Microsoft’s effort.

I tried out Jules this weekend out of curiosity. I gave it two changes:

1. Update a dependency in my Discord chatbot that my friends and I use.

2. Deprecate voting functionality in the chatbot, since I noticed that Discord finally added polls last year.

Adding the dependency went fine. I could have done it faster, but Jules also reran the tests for me. My only intervention was testing it against my test account. A promising start!

The second task went poorly. Part of this was my own inexperience with these agentic workflows; for example, I told it to “remove” the feature and it just commented everything out. After it was done, it asserted that it ran the tests. While reading the code I suspected it hadn’t actually run the tests. I asked it to run them again. It gave me a passive-aggressive remark but tried to run them. Sure enough, the tests were failing and it spun its wheels trying to fix the tests.

After 15 minutes, I started worrying that Jules wouldn’t finish during my kid’s naptime. So I fired up Visual Studio Code and started racing Jules. I didn’t have any LLM augmentation, but it was mostly deletion so I didn’t need it. 20 minutes later, I was testing my finished change. Jules was still spinning its wheels.

I don’t want to overextrapolate from this example. Jules is in beta, I’m not used to these longer-lived agentic workflows, and I also intentionally underspecified the task to see how it would handle it.

At the same time, I’m disappointed that it failed as hard as it did. It has access to the full commit history of the project. It can actually go and see how the feature was added, and what the project looked like before it was added. If the task was underspecified, then it could have told me that! But instead it just didn’t make much progress beyond its initial salvo. But I want it to behave like a human that is capable of reasoning and communicating its needs.

In that 15 minutes, I had a lot of time1 to reflect on the workflow. It required a lot of interaction. It would work for 2 or 6 or 10 minutes and then show me its progress. I needed to keep checking the tab to see whether it was done. By doing this, I learned that Jules threatens to auto-approve its own plans if you don’t respond quickly enough.

How am I supposed to get work done during that time? What engineer is most productive in the 2 and 6 and 10 minutes between their interruptions?

This leads me to my “30-minute rule” for LLM agents: in order to recoup my lost focus, I need to save at least 30 minutes over what Cursor does for me with prompting.

Why 30 minutes? Because I lose time writing a task with more exacting precision than I would for a human, then I lose time handling the questions corrections and reviews, and then I spend 10-20 minutes getting into a flow state on another task, then I go back and review what it does. Let’s give the LLM some leeway since you can run multiple. Let’s say that I break even on the agent when it can save me 30 minutes over what I did before.

Let’s pretend that Jules completed the task within the 35 minutes instead of failing. In comparison, it only took me 20 minutes — and I was alt-tabbing back to Jules constantly. I lost 15 minutes because I used the agent. In reality, I would need to run 4 Jules instances together on 4 different problems, and they all solved the problem instantly and correctly.

This also made me reflect on how the Jules LLM experience stacked up against human engineers. I like working with other engineers, and I’m happy to specify a task to the level that an engineer needs and handle their questions and review their code. Why does it feel worse with the LLM? Because my coworkers are effective engineers and they save me time. When I needed a benchmarking suite last year and gave it to a senior engineer? Man you should have seen what he wrote. It was great and saved me a ton of time. I basically told him, “we need to benchmark this thing under load. Here’s what I’ve done so far. Ideally it would have similar characteristics as our production load.” And he knocked it out of the park. Jules would have never made progress on that task.

When you’re giving a task to an engineer, you need to adjust the task to their leveling. A senior engineer might be able to convert a well-scoped business objective into an engineering project, but a junior engineer might never finish that project. So you’d break it down for them.

What kind of scope and specification does each engineering level require?

Here’s how I imagine each classic engineering level should be able to interact with scope and specification. This is an oversimplification, but it suffices here.

• Intern: They can handle a well-specified problem with minimal scope. You can scale the number of hints they get, depending on their experience.

• Junior engineer: They can handle a well-specified problem with small scope. They may also need some hints and pointers to get started.

• Senior engineer: They can handle a scoped problem that is underspecified.

• Staff engineer: Give them an unscoped and underspecified problem.

• Beyond staff: You’ve evolved beyond this. You identify the problems. You prioritize the problems. You create the problems. You are the problem.

There’s also the contractor dimension. Here are a few archetypes that I’ve run into over the years, out of contractors that can complete assignments:

• Brute-force contractor: They somehow follow the contract to the letter. What about the spirit? Only if you have a spirit clause in the contract. Is there a logical contradiction? The task is impossible? They’ll somehow torture the wording to produce a deliverable. Your codebase will be worse than it was before they started.

• Long-term relationship contractor: This is the contractor that tries to understand the business needs and guides each contract back to sanity. These are the types of contractors that eventually become full-time employees2.

• I’m trying to save your business contractor: This is the true domain expert. They attend the conference every year. They explain how your project fits into the greater ecosystem, how you’re failing, and necessary changes that you must make. This is unpopular because not everyone wants to hear what they have to say, and not everyone is talented enough to do it. But these people are worth their weight in gold.

I would be tempted to rate my interactions with Jules as at the level of an Intern, but that’s an insult to interns. First of all, you kinda want your intern bothering you every 2 or 6 or 10 minutes if they legitimately have a question. Investing in your interns pays dividends, since (a) they become more capable engineers, and (b) it is one of the best recruiting strategies imaginable. So when your intern asks you a question every 6 or 10 minutes at first, then you are simply doing your job as a host by responding to them. Your focus should be on them. They will grow and the questions will get further and further apart and you will need to direct them less and less.

On the other hand, the LLM doesn’t grow from this3. My experience with Jules was more on the lines of the “brute-force contractor,” where it was just doing everything that it could to get the project completed to the letter of what I wrote. And in this situation, it is a huge loss to have to answer questions every 6 or 10 minutes. The LLM isn’t growing from the conversation, my project isn’t moving forward.

Some outlets reported that the layoffs hit developers hard. But if you chase down the numbers this doesn’t seem right; Bloomberg is reporting that 40% of the workers laid off were software engineers. Yet in 2021 — the most recent year I could find accurate data about Microsoft’s software engineer employment counts — Microsoft said that they had over 100,000 developers and their 2021 10-K lists them as having 181,000 employees. So estimating that the company is about 55% developers, that suggests that developers were spared relative to other departments. If you have more accurate numbers, HMU!

But it’s still astonishing to see the list of developers that were laid off.

• The “faster CPython” project was all laid off including 3 core Python developers.

• A director of AI was let go.

• I previously wrote about Microsoft speeding up Typescript compilation by 10x; at least one of the people responsible was laid off.

• If you look on forums, it’s easy to find (unsubstantiated) posts stating that the people laid off were not low performers.

• Microsoft themselves said that they made cuts without regard to performance.

The situation is unbelievable from the bottom-up approach. You just look at the list and you think, “my god, they laid off the guy that improved Typescript compilation speed by 10x. He made a substantial improvement that could end up helping millions of developers. I work in this menial area of the codebase. My contributions could never hope to match his. How did this happen? Why am I still here and they are gone?”

We’re previously written about the disconnect between executives and leaf-node employees and how they view layoffs. This is because you are trying to apply top-down logic from your bottom-up perspective. In the bottom-up perspective, you are using all of the facts on the ground to make your decisions. You are actually looking at the person, their history with the company, the weight of their contributions, etc.

But let’s flip this around to the top-down view again. This is going to be an oversimplified view of organizational staffing, but it will suffice for the discussion. Companies look at roles in terms of having a seniority component (entry-level, junior, senior, staff, etc) and a role (backend, iOS, Android, ML, etc). Sometimes a large company will further specialize on domain, so you might have a Senior Backend Engineer in Payments or a Junior iOS Engineer in Adtech.

So periodically the executives need to plan for the future. They go through a bunch of planning exercises to figure out “given where we want to be in N years, what organization do we need? What engineering roles are needed to support those goals? What seniority do each of those roles require?”

So from the top down, “we’re not doing enough on AI” could translate into many shifts. Sometimes if you are misaligned with your future goals, you can fix it through attrition and headcount allocation. But if you need to change quicker, you likely need to conduct a layoff. Sometimes the management layer will do a nod towards the bottom-up view1 , but they’ve already made peace with the idea that talented people will be let go. These decisions are mostly “Ah, I see that you are a ::checks notes:: staff compiler engineer on ::checks notes:: the Typescript compiler. It turns out that we have 1 more of these than we need and it seems that you are pretty expensive, so you will now be routed into the severance pipeline.”

And that’s where the disconnect happens! From the top-down perspective it’s largely just a numbers exercise, but from the bottom down perspective everyone is just shouting “you let go of THIS PERSON? And you kept THIS PERSON, whom everyone knows is terrible? Make it make sense.”

If you’re worried about this happening to you, then what should you do? You should try to find a way to work on your company’s top priorities. And then you should do good work2. Personally, I’ve asked myself what I want to accomplish with my work. I personally want to be proud of the impact of my work. So my current philosophy is, “I want to work for a business model that I feel good about, and I want to be on a team that directly impacts it.” I know that I would have been reasonably happy on something like productivity software, but I’m much happier in my current role at a dating app. But these don’t have to be your values. You should think really hard about what you what out of this life, and what would make you proud looking back. And then you should try to find a company that does that as their business model.

At the time, I remember an anecdote told by Larry Page. He said that companies like Yahoo! used to be a punchline at Google because it would take them weeks to get something onto their homepage. Google could accomplish the same thing in a few hours, or a few days at worst. But now he was the CEO of a company where it took weeks to get something onto the homepage, and he was sure that he was the butt of some startup’s jokes.

Anyways, all of this clearly bothered Larry Page. He wanted to fix it. One of his first actions was to shutter tons of projects that didn’t make tactical or strategic sense, and focus on fewer efforts. This came with the catch phrase “more wood behind fewer arrows.” For example, they shuttered Google Buzz so that it wouldn’t distract from Google+.

And second, Larry Page emailed the whole company a ham-fisted attempt to revamp how meetings were done.

• Every meeting needed a “decision-maker.”

• Meetings should be capped at 10 people.

• Everybody in a meeting should give input or they shouldn’t be in the meeting.

• Hour-long meetings should be only 50 minutes to give the participants an opportunity to use the restroom between meetings.

They later softened some of the language by saying that these were properties of “decision-oriented meetings,” implying there were other types of meetings that someone might need to attend. But you could never shake the feeling that Larry Page had to make decisions all day long and forgot that sometimes people meet for other reasons.

Anyways, let’s focus on the fact that Larry Page wanted hour-long meetings to only be 50 minutes. This is a good thing! It gives people a chance to stretch, go to the bathroom, grab a snack, etc. During a Q/A on the changes1, someone asked him whether Google Calendar should default to 25 and 50 minutes for meeting lengths instead of 30 and 60 minutes. Larry Page said “yes.” And then someone on the Google Calendar team implemented this.

And then nothing changed. When 2:50 rolled around and your meeting was supposed to end, do you think people actually ended the meeting? Noooooo. Absolutely not! Meetings continue until the participants of the next meeting are clawing on your door like a pack of zombies.

At one point, one team in the NYC office noticed that their standups were about 10 minutes long. They didn’t want to compete with meetings that respected the half-hour boundaries. And why would they need to? Every meeting room had free slots at the last 10 minutes of every hour because people were now booking 50-minute meetings. So they did what any rational engineering team would do: they started booking their standup in the tiny 10-minute time slices that were free on the calendar of every meeting room.

I found this out when I saw them knock on the door to a meeting room by my desk. 2:50 rolls around and someone knocks on the door and says “I have the meeting room.”

The person in the room responds, “No you don’t, it’s 2:50.”

“Look again at the room’s calendar. You booked a 50-minute meeting, we have the room for the last 10 minutes of the hour for our standup.”

I could hear the muffled exasperation. “You’ve got to be joking me.”

“We have the room, sorry.”

Then everyone shuffled out of the room, looking vaguely pissed off. And who could blame them! Can you imagine if someone actually held you to this policy? You’re there stammering “it’s the default, I meant for the room to be ours for an hour” and they counter with the fact that their names are listed as the active participant? I mean, I’d personally tell them that I wasn’t going to leave the room, but surely it worked a lot?

I wish I knew the identities of these brave meeting crashers. I saw them pull this stunt twice and then ride off into the sunset, and I never got to learn what team they were on. I wish I knew their motivations. Were they true believers in the 50-minute policy? Were they bored pedants? Were they wraiths, cursed to hunt the office for available meeting rooms? I’ll never know for sure.

Adopting sudo-rs By Default in Ubuntu 25.10

The sudo-rs project is designed to be a drop in replacement for the original tool. For the vast majority of users, the upgrade should be completely transparent to their workflow. That said, sudo-rs is not a “blind” reimplementation. The developers are taking a “less is more” approach. This means that some features of the original sudo may not be reimplemented if they serve only niche, or more recently considered “outdated” practices.

This effort is part of Canonical’s north star of “oxidising” Ubuntu. They want to improve Ubuntu’s stability and resilience by replacing the most core programs with memory-safe alternatives. And obviously sudo is a great place to start. Its role is to let trusted user accounts run commands as another user — including root. A vulnerability in sudo is obviously a worst-case scenario: if an attacker could trick sudo into providing root access, then it’s game over and the system is fully compromised.

Since sudo is written in C, it has no protection against these types of problems. In fact, attacks have happened in recent memory. In 2021, Qualsis published a CVE where calling sudoedit -s '\' allows an attacker to use environment variables to get root access on any default sudo implementation. This bug had been lurking for about ten years, which is about as close to a “holy shit” moment as I usually get with security issues.

So what’s left before sudo-rs can ship by default in Ubuntu? They have a few things to finish before launch: proper internationalization support, and some attempts at reducing the binary size.

I also want to revisit a fun point from the initial announcement: they’re intentionally dropping features and breaking compatibility with regular sudo!

Naturally, they’re doing this in cases where they don’t expect that it will matter. You can see the complete list here, but it ranges from mundane stuff like “the sudoers file must be valid UTF-8,” which is fair enough in the Year of our Lord 2025. There is also stuff like “This will only work in a system with PAM, we should not rely on the sudoers file to specify things like umasks.” But then there was the one that really surprised me: they are removing sendmail support from sudo.

Wait, sudo can send email? And not just to the regular mail spool destinations inside of Linux, but it can actually fire up sendmail and send an honest-to-God email directly? This surprised me, although I’m not familiar enough with system administration. When you’re not a system administrator, it’s really easy to imagine aggroing an administrator, who would be very happy to tell you that there are obviously systems where sendmail and sudo both run, but there isn’t any type of integrated mail spooler. It’s a variation on Zawinski’s law1 for sure, and since sudo is a command runner, it can read mail given the right input command. It’s only natural that it should want to send email back.

The next day, Spotify submitted a new release of their iOS app for consideration.

Once Apple approves our update, U.S. consumers:

• Can finally see how much something costs in our app, including pricing details on subscriptions and information about promotions that will save money;

• Can click a link to purchase the subscription of choice, upgrading from a Free account to one of our Premium plans;

• Can seamlessly click the link and easily change Premium subscriptions from Individual to a Student, Duo, or Family plan;

• Can use other payment options beyond just Apple’s payment system—we provide a wider range of options on our website; and

• Going forward, this opens the door to other seamless buying opportunities that will directly benefit creators (think easy-to-purchase audiobooks)

First, let’s reflect on the absurdity of this. These were not allowed by Apple before a week ago. Spotify refused to use in-app purchases for their premium accounts. Accordingly, they could only show the text “You can’t upgrade to Premium in the app. We know, it’s not ideal” in addition to the name of your current plan. The user had to deduce that they should visit in a browser. They weren’t even allowed to say how much it cost! That’s egregious! I get a little angry just looking over Spotify’s list of what their app update does.

That’s all well and good.

However, I’m super curious about one facet of this news article: how did Spotify turn around their release in 24 hours? Corporations aren’t supposed to move quickly.

Let’s think about the naive option: that they wrote it from scratch in under 24 hours. It’s some links, it uses their pre-existing branding. How hard could it be?

But now let’s think about what would be necessary to get a project started and completed in a corporation:

• A product manager specifies behavior and goals.

• A designer produces a mockup or prototype of the change.

• A separate branding person either produces or reviews the copy in the design.

• A developer needs to implement the change.

• The change needs to be “QA”d in a number of scenarios: large system fonts, other languages, etc.

• Someone in the manager chain ensures that all of the resources are available.

• Someone in Legal needs to review to make sure you stayed within the bounds of what the injunction permits.

• Someone reaches out to the relevant backend teams to ensure that they have the capacity for you to deploy your change.

• And then someone needs to actually build and deploy the change to Apple.

And don’t forget that everyone needs to be available and working at top speed. And note the fact that many of these steps must be done serially.

Is it possible that they were able to slash through the red tape and have one or two people crank out new screens of the app that required minimal testing, minimal legal review, and Just Worked with no problems? Sure, it’s possible. But it’s unlikely.

Okay, so that option is out. What about the next option: they are using React Native or Electron or something. What if they are simply able to remove all of their iOS-specific flags within the application code and push a new update to users after some testing?

This one is trivially easy to debunk if you look at open job listings for iOS engineers at Spotify. Here are three bullet points from a listing open at the time of this writing:

• You know how to write readable, idiomatic, and maintainable Swift and are willing to follow already defined coding guidelines and workflows.

• You are experienced with a variety of iOS frameworks.

• You have a deep understanding of Cocoa design patterns and API design.

What does this mean? iOS engineers are expected to understand the iOS platform. And this means that they are writing their native applications from scratch instead of using a framework like React Native to produce the mobile apps1.

So that only leaves the final option, which is frankly the most impressive: they built this in anticipation that it would be used at the conclusion of the court battle, and they’ve been potentially maintaining it for years.

If so, this is an impressive amount of foresight for a corporation, especially a public one. The original injunction was issued three years ago. What would need to happen? Sometime between then and now, an executive mandated that this iOS paywall would be built. Between then and now, some team has been thanklessly cultivating this garden, just waiting for the moment that it would finally be ready to deploy.

It doesn’t cost that much to keep a feature on life support once it’s written. But it does cost some small amount of engineering effort! You’d expect that most pieces of the app are updated at least every few months, and some pieces would be updated multiple times a day. They were surely working in this area of the app, performing other work and maintenance. So they’ve been putting up with the annoyance for potentially years, all in exchange of being able to produce this update a week earlier than they would have otherwise.

For the curious: the hardware setup was 4 cameras on a helmet, an IMU, VR goggles, and a plastic gun. The gun was placed into the scene using computer vision. Tying all of this together was a top-of-the-line gaming laptop in a backpack.

Here is baby me, wearing the finished research prototype in Camp Pendleton:

As usual, my company overpromised our capabilities so they would win the contract, and that was the engineers’ problem. It was an 18-month-long death march.

We did have some advantages. We had research prototypes where camera pairs could be tracked through 3D space. We had a new GPU-powered stereo vision algorithm that performed well on real-world occlusions like branches and fingers. We had some code that used a “landmark database” to correct any dead-reckoning drift that had accumulated in the system. These were all required by the system.

The advantages got us like 5% of the way. We still needed working hardware. We needed all of the systems code to interop with that hardware. We still needed to identify VR goggles that could work with the system. We still needed to manage a fleet of these machines remotely during the scenario. And my boss demanded that we repurpose a pipes-and-filters library from an old project at the company, so we had to throw away all of our normal executables and rewrite everything within this framework.

About a third of the way into the project, we finally had all of the hardware and software ready for a first test run of the independent backpack system. This was just a small part of the project — it didn’t include the guns, the scenarios, etc — but it was proof of life. The hardware worked okay on the bench. The software worked okay on the bench. Let’s see what happens when we run it together! We turned everything on, strapped a researcher into the system, created a scene where a guy was standing still in front of him, and then started the scenario.

It actually worked! The researcher was slowly shuffling around1 and looking at the virtual bad guy. Then he stopped. “Something’s wrong. The FPS dropped. It’s barely working. This isn’t even showing 1 frame per second.”

We passed around the helmet and looked. The video was really choppy.

We did the most reasonable debugging step: we turned everything off and on again. The same thing happened. It worked fine for a few seconds, and then it got unusably choppy.

I opened the flap of the laptop. Heat whooshed up into my face. I took the laptop out. It was almost too hot to handle. We checked the CPU temperature and it was through the roof. Well, that’s bad news! We needed this to operate in deserts, and it wasn’t even working on a crisp fall day in New Jersey.

My boss wanted to see what happened if we prevented the laptop from doing any thermal throttling. We found a BIOS setting related to throttling. I crawled over MSDN and developer forums looking for any documentation or posts about this and found a Windows Registry setting we could try.

We started the system again. “I think it worked,” the researcher with the helmet said. And then the laptop powered down. At first it wouldn’t turn on, but eventually it sputtered back to life. The laptop got so hot that it tripped a temperature sensor and the machine powered off rather than suffer damage.

Someone said, “Just cut it open, this backpack idea is junk. Just cut as much of the fabric off as you can. Tie the laptop to the straps.”

So some Guy With A Knife materialized and cut the backpack down to the straps and we ziptied it down. Then we started the system again.

The researcher got up and shuffled around. It lasted longer than it had before. We started doing experiments, but after a couple of minutes the video got really choppy again.

I wanted to run an experiment. “If this is heat, shouldn’t it recover when it cools off? Can we just leave it on the desk for a few minutes and see if it gets better?”

We restarted the scenario, waited until it started stuttering, put it on the desk, and checked back 15 minutes later. The framerate never recovered and the computer’s temperature was fine.

One of the systems guys wondered if there was separate power-based throttling. Since the machine wasn’t plugged in, he hypothesized that either Windows or the BIOS was slowly scaling the clock speed below the performance we needed. The deadlines were pretty tight: it took just under 100 milliseconds to run the stereo algorithm, and we were taking 10 pictures per second. We did some more research and discovered (1) yes it was probably doing power throttling, and (2) we couldn’t disable it on this specific laptop. So we did the next best thing: we had the researcher carry around a huge portable battery with the laptop plugged in. The system lasted longer! But after about 10 minutes, it started stuttering again.

The hardware guys started spec’ing their own portable battery that could fit onto the backpack to prevent the power throttling. Meanwhile, my boss pulls me aside and tells me he thinks there’s a system problem and that we need to do much more intensive bench testing. He wanted me to run everything on the bench all day and try to identify the problem. In some ways, he was saying, “Please argue with everyone for an entire day.” There was only 1 working unit and everyone needed it.

And this created a huge buzz around the office! My boss asked me about the system problem whenever he saw me. People kept asking me how the system problem was going. My vice president popped his head in and asked about the systems problem. I was quite argumentative 15 years ago, and even I couldn’t argue with this. There probably was something happening in this new pipes-and-filters code I had written!

On Monday, I put our only working prototype on the bench and turned it on. I started swatting away everyone who tried to take the system for whatever they were doing. After about 10 minutes, it suddenly started exhibiting the problem. I wonder if it’s always about 10 minutes. I turn it on and wait a half hour. No stuttering. I went to lunch and came back. It’s stuttering again.

Okay, how did our pipes-and-filters code work? It was controlled from an XML file2 and specified a series of processing nodes in a graph. Each node specified a DLL and class that should be loaded into the application, and allowed the user to specify three things:

1. The node’s configuration: name, any node-specific configuration, input queue depth, output queue depth.

2. All of the data in the node’s output.

3. A list of other nodes that are the inputs to this node.

And then the framework would load all of the DLLs, connect the graph together, etc. This was a framework that was written for a previous project. My boss had been eyeing it for some time. This was finally his chance to split it into a standalone library. My boss also had some axes to grind. The elders who ran the previous project had strict rules: all pipelines should be linear — do NOT run things in parallel! — only pass along exactly what needs to be output, write very granular filters. My boss had the exact opposite philosophy: the pipeline should be maximally parallelizable, every filter should pass along every packet it got “because you’ll always need to correlate something with something else eventually, and there’s just so much churn constantly editing the configs as you develop.”

How often did data flow through the system? The cameras took pictures at 10FPS, since the laptop could only run the stereo vision GPU algorithm in about 100 milliseconds, give or take. The IMUs generated position data at 200 packets per second. There was some fanout to a few calculator modules — visual odometry and a few things that I don’t remember — and then went to the GPU and a few other modules — and then all of the data flowed into an octopus-merge sink that merged all of the position and stereo data with the camera images, and sent all of that to the video game engine for display. I’m a little sorry for the vagueness, but this was quite some time ago. The point is, we did as much fanout as possible and then merged it all back together for the final publication to the video game server.

At this point in the investigation, I have no idea where to start. So I start looking at each of the individual components.

Each node had fixed-length queues. Were these somehow filling up? I doubled their lengths. Within a half hour the system was stuttering. I tried tuning other configuration settings. No dice.

I poured over all of the code I had written. I wrote a merge filter that was a heavy abuse of the S++ STL’s map type, and I was quite proud of them. Were these somehow the problem? I added some logging to them and ran the system until the problem happened again. It didn’t seem to be the problem.

At this point, I go over everything that I’ve done with my boss. We brainstorm one or two more things it could be, but in the next few days I tested each of them and it would still seize up after 15-30 minutes.

And at that point, my boss decided that we should stop investigating it. It’s just a research prototype, we’re not shipping this to customers. We just needed to make it easy to restart a scenario remotely when it acted up. So for a few months, we just lived with this.

And then one day three months later, I was in the middle of a meeting for an unrelated project. I was speaking a sentence that had nothing to do with the project. Suddenly a moment of clarity overcame me.

The stuttering isn’t caused by a single component. The stuttering is caused by EVERYTHING. The sensor sample rate. The fanout. The fan in. The GPU algorithm. Passing every packet along.

So here’s what was happening: We had about 240 sensor samples per second: 200 IMU samples and four cameras producing 10 images each. Do you remember the rules that our elders had for using this system? Never parallelize and only pass along data that you will use? Well, it turned out those didn’t just exist to annoy my boss.

So let’s say all of these sensor messages fan out to 4 modules. Each of those 4 modules gets 240 sensor messages per second. If they then pass all of these messages to a merge, that merge has to process almost 1000 messages per second, as does every downstream module.

The downstream modules all have input queues. When the queues in this system filled up, they stopped accepting new messages and just dropped the others. We thought by setting the queue length in the thousands, we would always be guaranteed to catch up.

However, we had an extremely tight budget on our stereo vision. It took a little less than 100ms to run, and needed to process 10 frames per second. Did it really take a little less than 100ms to run? No, in practice that was the floor. It wasn’t uncommon for it to take 120 or 130 milliseconds. The laptop was doing a lot of work. RAM was filled to the gills, the CPUs were running close to max, and the GPUs were also fully saturated. Just because of system scheduling, sometimes it took even longer to finally get around to reading the GPU response.

The problem is that the octopus merge effectively waited for the stereo packet to synchronize all of the outputs of the various streams, so it had the same frame rate as the stereo module. When it would finally finish processing a frame, it would gallop through a bunch of IMU packets until it finally reached one with camera frames it hadn’t yet seen. But every delay would slowly make the queue longer, and longer, and longer. Eventually, the whole queue would be completely filled up, and the new camera frames would mostly be dropped on the floor. Suddenly, most of the images don’t actually get merged or sent downstream because they’re just never added to the queue. And the final few steps of the system — merge and send to the video game server — were very cheap, so it was never noticed that they were doing appreciably less work.

So I removed the “pass through unnecessary packets” behavior and put a system on a bench. Multiple hours later, it was still running smoothly. It even seemed to be a little more responsive than before! I finally caught up with my boss and delivered the news.

He went through all seven stages of grief. He was shocked that I even suggested this. Denial came quick. Anger came quicker. But we worked through the stages until he reached the inevitable conclusion: he couldn’t just pass through all information to each filter. We actually needed to spend a few minutes to modify the config when we needed this to happen.

What are the takeaways? Well first, having good monitoring would have made this a non-issue. However that was a bit challenging in 2009 - it was years before common monitoring tools like Prometheus and Grafana were developed. Second, this may have been the first “emergent behavior” bug that I debugged, where no individual component of the system was malfunctioning but the whole system was misbehaving. When I was 24, I was still fixated on the idea that some individual component must be misbehaving. In a way, I was the component that was misbehaving, because I was one of the technical owners of the application graph.

Do you have a war story that you’d like to share? If so, email a brief pitch to jakevoytko@gmail.com

You don’t have a few hours? You’ve come to the right place. I don’t have any qualifications — I’m not a lawyer and I never worked in adtech. But unlike many of you, I read the ruling and I’m doing my best to summarize it. If you want something authoritative, then find someone who knows what they’re talking about.

Anyways, here are my notes:

• Google was accused of having an illegal monopoly in three parts of the adtech stack: publisher ad servers, buy-side ad exchanges, and advertiser-side ad networks like Adwords. The entire complaint focuses heavily on their abuse of the buy-side ad server and exchanges, saying that they were able to unfairly exclude other exchanges from competing because of anticompetitive features in the publisher ad server.

• To describe the adtech stack, imagine that a publisher like New York Times wants to run advertisements. Their view of the ecosystem might look something like this:

  

  The NYTimes contacts an ad server (run by Google) that fires off a query when someone visits the page. The ad server contacts competing exchanges (one of which is owned by Google) to determine which exchange can provide the best bid. Each exchange itself runs an auction across all of the sell-side platforms to determine what to bid for the given request.

• The NYTimes’ ad server is supposed to make decisions that are in the best interest of the New York Times. Otherwise, why would they use it? However, Google repeatedly takes advantage of the fact that they have a monopoly on ad servers AND also runs its own exchange. They give their own ad exchange preferential treatment over competing exchanges, deliberately harming the competing exchanges.

• Google added features across the publisher platform / ad exchange ecosystem to limit competition.

  • First look: any publisher using Google’s publisher platform was required to offer Google’s exchange a buyout price. So Google’s ad exchange could buy inventory even though another ad exchange might have bid higher.

  • Open Bidding: to counteract Google’s stranglehold on the market, publishers began a practice called “header bidding” where they could offer the inventory across many ad exchanges simultaneously. Google added Open Bidding as an implementation to their own ad platform so that publishers could easily use the Google-provided header bidding instead of rolling their own. However, there was a big difference: when it was done on-platform, Google would charge non-Google ad exchanges a 5% fee.

  • Last look: After the auction ended across the exchange, Google’s ad exchange was allowed to submit a bid 1 cent higher than the winner and steal the auction.

  • Project Poirot: Google’s publisher platform simply offers non-Google exchanges much lower revenue, causing Google’s ad exchange to receive a significant boost. On average, spending on non-Google ad exchanges dropped 15%.

  • Unified pricing rules: Worried about anticompetitiveness, Google removed Last Look and introduced “Unified Pricing Rules.1” Some publishers had grown concerned about their reliance on Google. To counter this, they set their Google exchange bids much higher than their non-Google ad bids. In theory, this should have caused the Google exchanges to receive much less traffic. Unified Pricing Rules forced their Google ad exchange bids to be the same price as non-Google ad exchanges. Google allowed publishers to set higher prices for non-Google properties.

• After all of these changes, many publishers were disgruntled over their reliance on Google’s ad tech stack. They had explicitly tried to break their reliance on Google through “header bidding” that would allow them to contact multiple exchanges. But Google ended up capturing them by moving a Google-friendly header bidding into the publisher ad server. The judgement notes that Google retained 99 of their top 100 publishers because there were no realistic alternatives to Google’s publisher stack.

• Google tried to argue that the entire market is a two-sided marketplace. But the ruling notes that across the adtech industry, each of the components are considered to be their own separate lines of business and function as their own marketplaces.

• Google’s assertions that there were alternative stacks like Facebook’s or Amazon’s were not compelling because it’s not like the New York Times could exclusively advertise on Facebook or Amazon. They have inventory on their site that they need to fill and Google’s publisher platform is the only viable game in town.

• In conclusion: the court agreed with the characterization of their publisher ad server and ad exchange constituted monopolies. Additionally, because of the anticompetitive practices listed above (and some unlisted), they acted illegally and anticompetitively.

When I left Google in 2015 after almost 5 years on the Google Docs team, I asked everyone I knew the following question: “Did Google make money off of me?” I got every answer under the sun.

• “Obviously! you helped write features that are checklist items in enterprise contracts.”

• “Obviously! the product is mostly free but you are responsible for any downstream revenue it may generate in perpetuity.”

• “Obviously not! If Google Docs were a standalone business it would be wildly unprofitable.”

• “Obviously not! It’s silly to believe that Docs is the primary reason that anyone buys an enterprise contract over Microsoft.”

But to me, the most correct answer came from an engineer on the Google Sheets team. He said, “They obviously made money on you. It’s incalculably large. You participated in creating an ecosystem between Docs, Chrome, the other editors, and every future integration that Docs has. Strengthening a part of the ecosystem strengthens the whole thing, and causes everything to become more valuable as a unit.2”

The ad money was obviously the secret sauce to this ecosystem. Nobody pretended that the Docs/Drive organization could stand alone. Microsoft would have killed us before breakfast3. But Docs was granted years and years where the only objective was growth. Nobody cared if the growth was enterprise or consumer. They just wanted growth. Google Docs became huge and eventually the focus shifted towards enterprise customers.

It also enabled lots of bad behavior across the company. In truth, the ad money erased all of the risk. There was no consequence for the management chain lacking vision. There was no consequence to being an also-ran in a new industry. Many of the engineering “best practices” that I learned only make sense when money and delivery timelines don’t matter.

To back up: In June of 2023 and June of 2024, security researchers explored LLMs hallucinating package names in popular languages. This was a variant of “typosquatting”, where an attacker publishes a package with an easily-confused name and waits for people to accidentally download it.

The idea behind the slopsquatting attack is simple:

• LLMs generate code that reference nonexistant packages.

• An attacker publishes packages under these names. Maybe there is a malicious payload now. Maybe a later update will have the malicious payload.

• A developer generates code containing the fake package name.

• The package exists and seems to work. The developer accepts the changes into their project.

To prove their point, one of the researchers published a fake empty package under the name huggingface-cli.

The Register - AI hallucinates software packages and devs download them – even if potentially poisoned with malware

The result, he claims, is that huggingface-cli received more than 15,000 authentic downloads in the three months it has been available.

"In addition, we conducted a search on GitHub to determine whether this package was utilized within other companies' repositories," Lanyado said in the write-up for his experiment.

"Our findings revealed that several large companies either use or recommend this package in their repositories. For instance, instructions for installing this package can be found in the README of a repository dedicated to research conducted by Alibaba."

Alibaba did not respond to a request for comment.

Lanyado also said that there was a Hugging Face-owned project that incorporated the fake huggingface-cli, but that was removed after he alerted the biz.

And I have some sympathy for the Hugging Face employee that added a dependency to the fake package1. It’s lazy to point a finger at them. However, the modern package manager situation is unbelievable. I’m supposed to just download these third-party libraries with dozens of dependencies across dozens of authors and hope that it’s going to all work out in the end? You’ll wake up screaming every night if you think too hard about it.

Subscribe now

A fake package with a plausible name is fine. But how often do LLMs actually cause this to happen? Thanks to researchers, now we have some data.

First, LLMs hallucinate fake packages a lot.

In a recent study, researchers found that about 5.2 percent of package suggestions from commercial models didn't exist, compared to 21.7 percent from open source or openly available models.

The names they hallucinate tend to be persistent.

The recurrence appears to follow a bimodal pattern - some hallucinated names show up repeatedly when prompts are re-run, while others vanish entirely - suggesting certain prompts reliably produce the same phantom packages.

As noted by security firm Socket recently, the academic researchers who explored the subject last year found that re-running the same hallucination-triggering prompt ten times resulted in 43 percent of hallucinated packages being repeated every time and 39 percent never reappearing.

The researchers also noted that open-source models tended to hallucinate package names about 4 times as often as their commercial counterparts. So, as of September 2024 — which is when this research was published — you get what you pay for.

“But that was September, and it’s April now! The LLM world moves so fast! Surely all of the models have had time to adapt.” If you believe that then don’t check your dependencies. That’s way higher than my own personal risk tolerance. Personally, I’m going to keep reviewing the code that they generate until I’m made obsolete as a reviewer.

But take heart! People are doing this in the wild and training each other.

He also noted that recently a threat actor using the name "_Iain" published a playbook on a dark web forum detailing how to build a blockchain-based botnet using malicious npm packages.

Aboukhadijeh explained that _Iain "automated the creation of thousands of typo-squatted packages (many targeting crypto libraries) and even used ChatGPT to generate realistic-sounding variants of real package names at scale. He shared video tutorials walking others through the process, from publishing the packages to executing payloads on infected machines via a GUI. It’s a clear example of how attackers are weaponizing AI to accelerate software supply chain attacks."

I tried to find these videos, but this was futile since I don’t know anything about the dark web. The extent of my l33t haxxing was running a ton of search queries against Bing and Yandex instead of Google.

I think the only silver lining is that this guy is selling video courses. If he were wildly successful with this approach he wouldn’t need to grind out the videos to make extra money. But you know what they say about attacks: they never get worse. They only get better and better.

As a Blaze-pilled former Googler, I originally wished Bazel would become a good build system for every language. I didn’t want it to be the only build system for each language. I didn’t care if it was the best one. I just wanted to be able to easily use it everywhere. But it never reached “viable” in many situations, for reasons that I’ll get into below. I’ve come to think that Bazel is for three groups of people:

• Blaze-pilled former Googlers1.

• Companies that are large enough to staff a build infrastructure team.

• Companies with insanely slow builds or tests.

There are certainly other people who use it. They could probably live without it.

Why are its use cases so narrow? It’s because Google is a separate branch of engineering evolution. If Google hired an experienced engineer off the street today, they would think, “why is all of this so alien? Why can’t I recognize any of this?” It’s because Google is an old company by internet standards. They had to invent their ability to scale as they went along. So they invented a bunch of new engineering, and then Google ossified around the engineering. So now onboarding into Google is learning this ossified way, the Google way2.

Bazel was born from Google’s build infrastructure. It has the same virus: Bazel makes you do things the Bazel way. This is a huge problem for Bazel. The world has drifted away from Bazel’s happy place.

Bazel wants every target to be fully specified. It should have an exact set of inputs and a well-defined configuration. It should be able to clearly define all of its outputs ahead of time. And everything must be reproducible: the same inputs should always produce the same outputs. The build system also wants to touch every directory of your project. Ideally, you would define per-module BUILD files and maintain them over time.

In practice, the world has moved more aggressively towards “convention over configuration” and not worrying too much about the details. Take Go for instance: you can just run “go install” or “go get” or “go build” and it Just Works. Or building up a Next.js directory structure.

And the world has also moved towards command runners and layers of compilers. For example, a modern frontend project would not have a monolithic compiler that understood how to use every single layer of the app. You might have the Tailwind compiler that dealt with the Tailwind, then a SCSS compiler that dealt with the SCSS, then some React compilers and plugins, and throw in a “css-in-Javascript” plugin. And then all you need is to run the correct npm run command and then your build system will invoke all of these compiler layers under the hood.

Bazel has tried to adapt to this over time. For example, additions like Gazelle, the modern frontend rules, and rules_foreign_cc acknowledge that it’s difficult to build and maintain BUILD.bazel files, and additionally that most things work with external build systems that Bazel must interface with.

But yet, every time I use Bazel, I fall into some corner case that just doesn’t work, and then I’m arms-deep in someone else’s rule trying to figure out why the quoting is broken or something like that. I always hated running into some obscure command quoting problem when using a rule, or writing patch files to imported third-party libraries to make them work in the hermetic build environment, or discover that you are downloading a third-party dependency that doesn’t always have a consistent hash, or upgrading Bazel and getting a host of deprecation warnings3. I just don’t run into these problems when I don’t use Bazel.

Another common problem I’ve had: I work on OSX about 70% of the time and my Windows gaming desktop about 30% of the time. Thanks to a job I had 15 years ago, I’m comfortable developing in Windows and would rather not use WSL unless forced to. And Bazel always forces you, since many rules are powered by genrules — glorified Bash scripts — and many libraries do not include the Windows-equivalent commands. If you think “lul who uses Windows?” you may be surprised to learn that the answer is that a plurality (and almost a majority) of developers use Windows professionally.

And empirically, developers often don’t want Bazel. How do I know? I went through all of the open-source libraries on Bazel’s “Who's Using Bazel” page. Some of them are abandoned micro projects that they never should have listed. Even worse, some of those projects are abandoned micro projects that had already deleted their Bazel files by the time they were abandoned.

But I want to draw your attention to the following two projects:

• https://github.com/google/nomulus, Google’s cloud service for operating TLDs. Bazel was removed in favor of Gradle.

• https://github.com/kubernetes/kubernetes, the library that Google open-sourced which became the popular orchestration layer we all know and love. Bazel was removed in favor of Make.

Nomulus is owned by Google. Kubernetes started at Google. If anyone could figure out how to make them work, it’s these two projects. But it turns out that Bazel just isn’t a killer feature here, relative to “we have lowered the barriers for high-quality contributions to our codebase.”

In an open-source context, you should just follow open-source conventions. Nobody wants to learn the BUILD.bazel syntax. Nobody wants to know what a genrule or a custom rule is. It’s a Java project. They just want to invoke gradlew or npm run or make or pip or whatever people typically do.

But oh man, if you’re in a company that can afford to maintain Bazel? It’s incredible. Building code at Google was incredible. I can see why entire companies have spawned to provide consulting services or incremental technology improvements for Bazel. And I can see why ex-Googlers everywhere are torturing their new employers with the threat of Bazel4.

You can do some really cool things with Bazel. When I was at Etsy, the Java codebase that powered the search stack was tragically slow to work on. Builds took a while, and rerunning the test suite took forever. It noticeably impacted development speed.

An engineer — somehow not ex-FAANG — protoyped a rewrite Bazel. Of course, you need to do things The Bazel Way, so it took him a few weeks to break all of the circular dependencies that their previous build system was chill with, but Bazel absolutely COULD NOT accept. But he broke the circles one at a time, and eventually he had something that could compile.

The project was immediately greenlit when he demoed building the project and running their test suite. The build was already faster than it was before. And then he made a single change and reran the tests. Only the 2 tests that depended on the file rebuilt and reran. Instead of taking minutes to build and run the whole suite, it was over in a few seconds. After that, they went even further and used rules_k8s to quickly push containerized builds to any of their clusters. It was super cool, and I later tried it myself and had zero problems doing this pattern. It just worked.

But inevitably, I ran into problems somewhere else.

After it leaked, he posted it to X/Twitter in its entirety. The full memo is quite long, so below is the most important section.

What This Means

1. Using AI effectively is now a fundamental expectation of everyone at Shopify. It's a tool of all trades today, and will only grow in importance. Frankly, I don't think it's feasible to opt out of learning the skill of applying AI in your craft; you are welcome to try, but I want to be honest I cannot see this working out today, and definitely not tomorrow. Stagnation is almost certain, and stagnation is slow-motion failure. If you're not climbing, you're sliding.

2. AI must be part of your GSD Prototype phase. The prototype phase of any GSD project should be dominated by AI exploration. Prototypes are meant for learning and creating information. AI dramatically accelerates this process. You can learn to produce something that other team mates can look at, use, and reason about in a fraction of the time it used to take.

3. We will add AI usage questions to our performance and peer review questionnaire. Learning to use AI well is an unobvious skill. My sense is that a lot of people give up after writing a prompt and not getting the ideal thing back immediately. Learning to prompt and load context is important, and getting peers to provide feedback on how this is going will be valuable.

4. Learning is self directed, but share what you learned. You have access to as much of the cutting edge AI tools as possible. There is chat.shopify.io, which we had for years now. Developers have proxy, Copilot, Cursor, Claude code, all pre-tooled and ready to go. We’ll learn and adapt together as a team. We’ll be sharing Ws (and Ls!) with each other as we experiment with new AI capabilities, and we’ll dedicate time to AI integration in our monthly business reviews and product development cycles. Slack and Vault have lots of places where people share prompts that they developed, like #revenue-ai-use-cases and #ai-centaurs.

5. Before asking for more Headcount and resources, teams must demonstrate why they cannot get what they want done using AI. What would this area look like if autonomous AI agents were already part of the team? This question can lead to really fun discussions and projects.

I’ve previously written about gaming performance management criteria at big companies, which is where you selectively overperform on tasks that are valued by the promotion criteria, and try to avoid doing anything that is not rewarded by the promotion criteria.

Throughout the past 20 years, there have been common playbooks for gaming promotions in big companies. For example: let’s say that you’re a manager and you want to get promoted to director or higher. Career ladders normally say that as you get promoted, you will be managing progressively larger teams (or teams of teams). The naive approach is to deliver impact with your small team and then gradually get more engineers under you as you gain trust.

But that will take a long time. Let’s game the promotion criteria. Isn’t it better to find a company priority and work on some critical functionality for the success of the initiative, and use that position to argue for as much headcount as possible1? As the team grows, you can even convince some engineers under you to become managers of a few employees each. Boom! Suddenly you’re managing a team that a normally a director would. If you don’t mess up the execution, you’re a shoo-in for that promotion.

But the AI trend will require us to find new ways to game promotion criteria. So I want to help all of the Shopify employees that are now figuring out what this means. Obviously I don’t have access to your career ladder. But I have access to that memo and I’m feeling spicy. Let’s brainstorm ways that career-minded Shopify employees can game their promotion system.

First, let’s look at “AI values” that the memo calls out.

• A good Shopify employee…

  • Uses AI to get 100x the work done

  • Improves by 20-40% every year

  • Learns from other Shopify employees

  • Follows the Shopify values “Be a constant learner” and “thrive on change”

• A good team…

  • Uses AI during their “Get shit done” prototyping phase

  • Can demonstrate why AI cannot perform the work they want from a new headcount

• The performance review process…

  • Will have AI usage questions on their performance and peer-review questionnaire

  • Will explicitly involve getting feedback from peers on AI usage

Okay, first things first: make sure that an AI agent can’t replace you. Can it give engineering feedback on designs and product specs? Can it take a design and product spec and produce an engineering spec? Can it produce code from the product spec? Can it write tests for that code? Can it fix code based on comments on your PR?

Is your job safe? Great! Let’s become a 100x AI-powered engineer.

We know that sharing knowledge is important. Document what it can and cannot do. I don’t know what Shopify uses. Google Docs? Notion? Cram your findings into there and share this with your immediate team.

First, it’s clear that Tobi wants the company to use AI to rapidly prototype. Does vibe coding come naturally to you? No? Then you need to find someone who will teach you. And here’s the trick: instead of just having them show you, have them give a public demo. Be the person who asks in a Slack channel, “hey, @soandso is going to demo how they build large prototypes quickly in Cursor. DM me if you have something neat you’d like to present, and react :raised-hand: if you want an invite.”

Why would you set up a meeting? Because there are going to be peer review questions involving the use of AI. You want to remind your coworkers early-and-often that you are a 100x AI Engineer.

Now that you know how to prototype rapidly using AI, use that knowledge to show everyone that you are a 100x AI engineer. Build an “AI Solutions” site. Why? As an outsider, I noticed that Tobi referenced Slack channels as the knowledge repositories for prompting. Slack is a great place to coordinate work and shitpost, and a bad place to store persisted instititutional memory. So you’re going to have the AI spit out an internal website. Index it by outcome (prototype, debugging, etc). Your first story will be a prototyping story showing the prompts you used to build the prototyping site. Next you’re going to add the example from the @soandso talk you organized. And then finally you’re going to have an AI scrape all of the messages from that Slack channel to fill out content. Remember: we’re being a 100x engineer, so if it sounds hard you’re just going to throw the AI at it.

Go ahead and share your new prototype site in the Slack channels where people have been collecting prompts. Mention that anyone who wants to add one should reach out to you. At this point, you have the option to never think about this again. Well, you should probably present it to your team or group or whatever. Who will be writing your peer reviews for your performance review? Make sure that they are all included in that group.

Now let’s look at your current project. What are the parts that you’re ashamed of? What are the parts that kinda suck? Try to throw AI at it and see what sticks. We’re trying to 100x, so don’t waste your time if it’s not working out. If something doesn’t look promising just immediately bin it.

Do your services have the READMEs they should? Well, they do now! Give Cursor some of the important files in your service and an example README and it’ll spit out a pretty reasonable one. Just fix it up and send it out.

Point it to particularly fragile parts of your project and ask it “what are the most likely bugs in this project?” and “how can this be refactored so that it is easier to modify X?”

If anything actually works, make sure you make a big deal about how you generated it with AI. You can even mention it in the commit message or the PR description’s body. Since you’re going to get peer review feedback on the use of AI, you want everyone to know that you are an AI engineer.

If you’re used to seeing people game promotion criteria in companies, this may look different than you’re used to! In most large companies you scale your impact with the size of the team or the value of the technical contribution. But in a regime where rapid prototyping and proper value sharing is important, you’re trying to scale your impact based on how much you can share without someone saying, “stop showing us all of this junk.”

The Future of U.S. AI Leadership with CEO of Anthropic Dario Amodei

I think we’ll be there in three to six months, where AI is writing 90 percent of the code. And then in twelve months, we may be in a world where AI is writing essentially all of the code.

But the programmer still needs to specify what are the conditions of what you’re doing? What is the overall app you’re trying to make? What’s the overall design decisions? How do we collaborate with other code that’s been written? How do we have some common sense on whether this is a secure design, or an insecure design? So as long as there are these small pieces that a human programmer needs to do, the AI isn’t good at… I think human productivity will actually be enhanced.

But on the other hand, I think that eventually all those little islands will get picked off by AI systems. And then we will eventually reach the point where, you know, the AIs can do everything that humans can.

Amjad Masad at Replit took the ball and ran with it.

Amjad Masad, posted to Twitter/X on March 26th 2025

I no longer think you should learn to code.

If you click through to the Tweet and then watch the associated video, you’ll see that his position is moderated a bit. He says that if Amodei is correct and essentially all code will be AI-generated within 12 months, it would be a waste of time to learn how to code. To me this gives him some outs, like “it’s 12 months from now and wow that wasn’t true, obviously that changes my response.” But I’m going to take the Tweet as his most-recent belief since it was in response to the video.

If you’re new to the industry: don’t worry. It’s still useful to know how to code now, even by Replit’s standards. At the time of writing, Replit has 7 open engineering positions: four Software Engineers, 1 SRE, 1 Head of Product Engineering, and one Design Engineer.

The CEO of NVidia, Jensen Huang, also had a similar take but on a longer time horizon.

Don’t learn to code: Nvidia’s founder Jensen Huang advises a different career path

Over the course of the last 10 years, 15 years, almost everybody who sits on a stage like this would tell you that it is vital that your children learn computer science. [That] everybody should learn how to program. And in fact, it’s almost exactly the opposite.

It is our job to create computing technology such that nobody has to program and that the programming language is human. Everybody in the world is now a programmer.

Let’s summarize their claims in terms of the timelines:

1. Amodei doesn’t put a timeline on his prediction. But he says in the video that it’s the kind of prediction where he will be ridiculed on at least a 10-year time horizon. So this prediction is quite far into the future but quite specific, where the full economic output of software engineering jobs will be captured by AI.

2. Huang things that current children should not learn to code. So let’s say this is a 12-year prediction (i.e. you should dissuade a precocious 10-year-old from learning how to code if they were interested).

3. Masad thinks that nobody should learn to code. You can become a non-junior software engineer about 6 years after you begin learning1, so let’s say that he predicts that the net lifetime gain on learning coding will become negative within 6 years.

Yes, these are CEOs who hype AI coding for financial gain. But I’ll be fair to them. If they were 100% confident in their stated positions, how would they behave differently? I’ll assume that these arguments are in good faith and I’ll engage with the arguments directly.

First, let’s take the most aggressive timeline: Masad’s assertion that you should not learn how to code now, given that essentially all code will be written by AI in 12 months.

What does that imply? Let’s say that you’re entering a computer science program this fall. It is obviously important to get the best internship and jobs that you possibly can, regardless of whether you can code. You don’t know how the future of work will change, but “working at a world-class engineering company” is a good bet for catching the next wave. Plus the money is always nice.

• For the first 3 years of college, you’re going to study all of the regular courses. Algorithms, operating systems, etc. You learn them well. You minor in something else. But you will complete all of your assignments with generative AI. All of your teachers need to accept that you use generative AI tools on their quizzes and exams.

• During your 3rd year, you will land an internship. By then, every internship interview screen will need to accept generative AI coding tools as part of their screen.

• You will use AI agents to code your way through your internship, gaining positive feedback and a reference you can use for your job search.

• During your 4th year, you will need to interview with prospective companies using your generative coding interview skills and your reference from your internship.

• During your 5th and 6th year, you are a productive member of the organization post-graduation. You can efficiently communicate with AI agents and use generative coding to deploy, and use generated graphs to monitor the system in flight. After 2 years, you get the good news: you are being promoted from Junior Software Engineer to Software Engineer. Congratulations! You have demonstrated your economic value as a software engineer.

And now let’s couple this internship with the following: “The year is 2030. All of my company’s code is produced by AI. Nobody at my company knows a single line of code we’re running. This is the most optimal situation.”

Why does this sound so far fetched? It’s because Software Engineering has a huge “last mile” problem. This is a phrase that I’m borrowing from logistics. It’s easy to ship between shipping hubs because there is a constant flow of freight between these hubs, but the “last mile” to the house or retail store is much more difficult. You actually need to drive through traffic to that house and find where it is and park your truck somewhere and go to the door and try to collect a signature. Software engineering has its own last-mile. Once you have a spec for the code, producing the code can be mechanical. However, the process of generating a clear spec for the code is difficult, as well as determining that the code is functioning as part of a working system afterwards.

In some ways, this reminds me of articles from 10+ years ago declaring that autonomous trucking will soon be automated. But then many of the major players in the space folded or kept pushing their deadlines back. Even now the furthest along seems to be Aurora. They only operate in Texas, and their website claims that they still use vehicle operators. So the impending doom from 8 years ago appeared to be more than 10 years into the future, perhaps far more.

Let’s talk through some of the last-mile problems that software engineers face. These are the “islands” that Dario Amodei mentioned in his response. There are a bunch of them like “gaining organizational consensus,” “making decisions that benefit the business,” and “managing stakeholders” that engineers spend a lot of time doing. Current LLM technology adds extra problems that you need expertise to resolve, like hallucinations, errors summarizing, etc. But I want to focus on the islands that relate to understanding code specifically.

Island one: Security

An incorrect model of security is “if every layer of the stack simply did security correctly, then there would never be a security vulnerability.” But security problems often fall across several layers of the stack. Either every layer is properly functioning but the whole system is flawed together, or several layers are misbehaving together. An example of collective misbehavior: when I was at Etsy, we ported our infrastructure from our own servers to Google Cloud. Shortly after, we received a security bounty that an attacker could log into any account simply by pasting any large payload into the password field. This was several layers of the app failing together2:

• We had accidentally configured one of the new Google infra bits to strip headers when they were above some threshold. It was something like 64Kb.

• If the auth service received a request to / with no other information, it returned a 200.

• The backend code checked for 200 response codes to see whether the password was accepted.

There was no single broken part of the stack, it turned out that the whole stack was wrong.

• The Google infra bit needed to be changed to return a 4xx error.

• The auth service needed a tighter contract, and additionally needed to return a specific payload like {“status”: “ok”}.

• The backend code needed to check for the response code and check for the expected payloads.

Fixing only one layer doesn’t reduce the fragility of the system. The security team really needed to second guess each layer and determine its correct function. So you can’t just point an AI agent at the problem and expect it to generate a holistic solution. If you just asked it to fix the problem it would very likely just fix the Google infra bit.

And this is just a very simple example. For each security construct used by your application, it’s important to understand the actual code constructs and their properties. It’s important when the AI tries deleting that CSP or removing encryption from a cookie or checking response code instead of payload on your login form because this is the simplest way to satisfy your prompt, that you have the perspective of understanding why the check was there and what value it serves.

Island two: System design

At the moment, large software ecosystems are complex systems. They have the interesting property that they can have emergent behavior3. That means that every component of a system is functioning correctly, but when they work together they fail. This means that you can’t create a properly-functioning system by simply writing a bunch of components that work well together. It also means that you cannot predict all of the failure cases for the system.

In fact, properly maintaining the system turns into a control problem. Think of your software system as an aircraft, and your graphs and alerts as the instruments of an airplane. You are simply looking at all of the available data. And when you see the system deviate from the expected behavior, you need to react and move it back within normal operating parameters.

The AI is hampered by only having access to the code. I’m sure future systems will plug into your logs and your source code and your monitoring and boil an entire ocean to tell you that everything is happening within normal parameters. But at the moment, reading the code doesn’t tell you anything about the outside usage patterns. It doesn’t tell you what happened all of the previous times an approach caused bugs at a different company. This is where the human can own the code structure and organization, and go from a middling solution generated by the AI to the best solution for your particular domain.

Island three: Handling on-call issues

This is the flip-side of the coin to system design. When the complex system is no longer functioning correctly, you need to be alerted to this fact and then understand the problem so that you can fix it.

There aren’t singular reasons that things fail. You can’t just point an AI at an error message that is flooding your logs, and expect the AI to fix all problems without being plugged into the instrument panel on the plane. Your database is failing because the requests are hanging and then failing. This could have any cause from “the database is overloaded because your application is running too hot” to “the database is misconfigured” to “your cloud provider is struggling” to “your new query has an infinite loop.” Until the AI system is able to be plugged into the instrument panel, it won’t produce proper fixes for this, and you will need to understand the underlying code and the underlying system well enough that you can convert that call stack problem into an actionable plan. And then maybe the AI takes over and can happily generate it.

Errors happen at points in code. When you are on call, it is your responsibility to look at that code and decide what is happening. Very often, this will be in a piece of business logic. Sometimes there are several options for remediating an error: do you ignore the error? Do you catch and log the error at another level? Do you need to clean up the input source? Do you need to page the staff engineer on another team because nobody on the oncall rotation worked on that project and there’s not a cookie-cutter answer? What is the difference between “the best fix” and “what is sufficient at 3am?”

Furthermore, is your on-call rotation depending on generative AI to produce solutions to get you back on track? What if it goes down during your outage? Are you just going to wait for it to come back up? Is your CEO going to be happy with that answer?

So which CEO was the most correct?

Let’s go from the most aggressive prediction to the least aggressive.

Amjad Masad at Replit: If you were entering a computer science program today, it is irresponsible to bet your whole career on learning computer science without learning how to code. In some ways, coding is the easy part of what a software engineer does. There is a whole host of interrelated activities that all inform the code, and are all informed by the code. There is only one conclusion you can draw: it is imperative that you learn how to code. I also think that it’s important to learn how to use AI tools to accelerate this process. These tools are only going to get better, so you should learn how they can accelerate you.

Jensen Huang at NVidia: This puts it on a longer timeframe, where children should not learn to code because they would be better served learning to think. In this world, everyone becomes a programmer because the systems are so powerful that the most important thing is asking good questions and providing the system good insight and input. It’s still quite a rigid timeline though, and as we see from autonomous trucking, sometimes the devil is in the details. This prediction is plausible but at risk, given the specific timeframe.

Dario Amodei of Anthropic: His view seems like the most plausible answer to me. I agree that coding is a small piece of the value of humans in software engineering. I can also imagine that eventually these AI systems will be so powerful that you can sit them in front of an AI-accelerated product manager, and they will have access to the current codebase and the logging and the monitoring and they can fly the plane by themselves, and there will no longer be humans involved in coding. But this does seem like a long ways away. There are likely several generational improvements that need to happen before these systems can all coexist while using the amount of energy we have available on Earth.

So, yes. If you weren’t sure whether you should learn to code nowadays, take comfort. We still power a whole host of human activity surrounding the actual code. So even if the AI were generating a lot of it, humans still need to be in the loop understanding and refining the code for the foreseeable future.

Researchers Yasser Allam and Rachid Allam put out an in-depth explanation explaining the vulnerability that they had discovered. I highly recommend reading it.

They identified three different sets of vulnerabilities that were in Next.js’ handling of recursive calls. It looks like these are subrequests that are intended to be made after the middleware has executed. So Next.js adds these headers to notify the handler that the middleware can be skipped. However, the system didn’t treat these headers like they could be supplied by an untrusted user. So it blindly accepted its request to disable middleware.

1. In versions of Next.js prior to 12.2, simply passing the header “x-middleware-subrequest: pages/_middleware”, causes the middleware to be skipped and the page to be executed.

2. After version 12.2, it gets a little more complicated. Now it could either be “x-middleware-subrequest: middleware” or “x-middleware-subrequest: src/middleware” depending on whether the application placed the middleware in the first or second location.

3. In recent versions, Next.js allows calls to execute recursively up to 5 times before skipping the middleware. So now you can simply call “x-middleware-subrequest: middleware:middleware:middleware:middleware:middleware” to trick Next.js into believing that the middleware had recursed 5 times already and simply did not execute it.

This bug checks all of my boxes for “is this a fun CVE?” As someone who often works on hidden load-bearing layers of applications, I love it when hidden infrastructure layers of an application turn out to have juicy bugs. I love security vulnerabilities that are simple enough to explain in a casual blog post. And who doesn’t love a complete bypass?

Why is the impact so high?

Many news publications advertised this as an “auth bypass.” It’s a lot worse than that, but it’s also true that this can be an auth bypass. It’s common for authorization and authentication middleware to redirect to login screens when the user is not logged in, or to read the user ID from a secure cookie and provide it within the request’s context. Normally you’d expect the endpoint to try to access this user object and then break because of a null pointer exception, but it depends on how the endpoint is written.

You can imagine the appeal of auth libraries that work exclusively with middleware, especially in the frontend environment where Developer Experience is king. You can just tell users “Installation is soooo easy. Just install this middleware and it runs everywhere! Add a React login form in minutes!” And then you hope nobody notices that the Next.js docs have heavy-handed recommendations that put middleware as an optional up-front step, backstopped by architectural decisions.

For both cases, we recommend:

• Creating a Data Access Layer to centralize your authorization logic

• Using Data Transfer Objects (DTO) to only return the necessary data

• Optionally use Middleware to perform optimistic checks.

If the CVE were just an authentication or authorization bypass, it would still be a severe bug. But this can bypass all middleware. The researchers give another motivating example: is a pesky Content Security Policy getting in your way? Well, run the bypass and that CSP goes away. My own motivating example is more fun-oriented: is a middleware script enforcing a max field size and preventing you from stuffing the entire contents of the Bee Movie script into every text field in your app? Put a bypass on it.

There was another interesting bit of drama, which was that Cloudflare attempted to block requests in this format and then Cloudflare had issues with their fix. It turned out that a lot of requests had valid reasons to include the header, so there was no way for them to filter attacker traffic instead of valid use cases. So at the moment, Cloudflare’s implementation is opt-in.

The internet is held together with magic intercepting scripts. I’ve worked for several web companies, and every company has a magic and hidden implementation of critical functionality that completely abstracts all of the hard stuff from their endpoint. It turns out that you just don’t want to repeatedly write the same integration code

In practice, writing endpoints is tedious. You need to declare the route and the HTTP method used to reach the endpoint. Any authentication code needs to run. Any authorization code needs to run. If the user is not authenticated they need to get redirected to /login and if they’re not authorized they get an error. Then you need to unpack and validate the request. If there are any JWTs, gotta check those. Was this larger than the max request size? That sounds important. Let’s also make sure that exceptions get caught and logged somewhere. Maybe something with distributed tracing. And now that you’ve performed all of these steps, we can now start serving the request.

Or, hear me out: you can just write middleware that lets you handle it in every endpoint that needs it.