Something that has popped into my head more than once, given the amount of software being shipped lately, is whether and/or how we are compromising our systems in the future as we rely on agents to build faster. I don’t think this is inevitable, but I find one risk more potent than the rest. The first concern is bad code. If you’ve worked in production codebases, you’ve seen bad code. Most arguments made by people concerned about code quality are based on the premise that most of the code LLMs are trained on is bad code. I don’t buy this, among other reasons, because lots of pre-AI production code was bad, but we somehow made do and kept evolving our systems, however slowly. The next is the collapse of unmaintainable systems. The easiest way to imagine this is that bad code leads to a change that creates an irreversible mess. A code update corrupts a database, and now everything is lost. I view this more as a process problem than an agent problem. Many databases were accidentally dropped in the pre-AI world. Whether or not it was catastrophic had more to do with the surrounding engineering processes than with whether AI was used for development. The final one that I will mention is that we lose the ability to understand and reason about our systems, with or without the help of agents, because they outgrow any agent’s or person’s ability to do so. This could be the perfect storm that brings it all down for us. I definitely view this as a relevant but not new risk. Large companies have been afflicted by long-lasting, wide-ranging outages of critical systems, which also predates the broad adoption of AI. This isn’t a good thing, and I expect it to happen more with AI, but I also wouldn’t attribute the problem to AI itself. Using AI to make software still requires the articulation of intent. What are you building? You can get away without doing a lot. You don’t have to say how everything will look. The agent can fill in the gaps. You could even hand over the keys to deploying your systems to the agents. But this doesn’t happen by accident. These are decisions made by people and processes in companies and organizations. You may not like them, and you may think they are bad ones, but it’s people making the decisions nonetheless. As far as I know, the superintelligence doesn’t run the world yet. It still seems to be mostly run by people making decisions, both by working with other people and by using AI systems to help. If our technical systems fail, it will be because of decisions people made, not whether we used AI or not. We should hold decision-makers and leaders accountable for the quality and consideration of their decisions. Blaming or hating AI does nothing. AI cannot be held accountable for harm done in the world.
Codex kind of looks like opencode now
Opus 5.5 feels fast