arthritic robots will have their day.

cataclysmic in their consequences, but they're indicative of a buggy technology that's not reliably knowledgeable. It can drum up an answer quickly but it can be wrong. And it doesn't know it's wrong. And it doesn't care.

Why anyone would allow it to operate with no "guardrails" blows my mind. But, the list of human behaviors that blow my mind grows longer day by day...

Which sounds great until you realize how fallible human knowledge is, especially when you start looking at ALL of it. There is WAY more information accessible on Reddit and social media vs. either published and peer-reviewed data or legitimate journalistic sources.

I would lose my job and get sued if an IT consultant. AI will keep its job, but Anthropic is getting sued.

I suspect the actual problem here is that the instruction set provided to the model were not explicit enough about what "destructive and irreversible" actual meant. I use Claude Code all the time. There's an option that has the agent stream out its though process instead of just giving a blinking "thinking..." prompt while it goes about its business. I like to open those and follow along, especially early on in any given iteration. It's kind of wild all of the different directions it can go, the gaps it finds in what I thought were clear instructions, and the assumptions it sometimes makes to try and figure out what I meant. Claude is a junior developer. It's a very fast junior developer, but it also makes all those same junior developer mistakes really fast. It should NEVER have permissions to do anything more than you would let any other junior dev do.

let it have access to anything in production. Or letting it directly do anything with a database, even in dev. If there's a routine task you want it to do, have it generate a script, then you run that script. A lot of people having issues with AI are because they're using it carelessly (which may be partially the result of the companies hyping how it can do everything). Though runaway AI is a concern that serious, thoughtful people have had for a long time. But in this case, they needed better procedures. At my job, doing anything in production requires certain permissions and approvals and it's a whole ordeal, but for good reason.

before just unleashing the beast