Last post I talked about the thrill of watching AI write SQL faster than I could.
Here's what I didn't mention: it also broke a perfectly working stored procedure within the first week.
Not maliciously. Not obviously. It "optimized" a sproc that had been QA-tested across dozens of parameter combinations over months. The code looked cleaner. It ran faster. And the results were wrong.
Wrong results from a financial process don't announce themselves. They just quietly produce bad numbers that someone catches downstream — maybe.
I've heard other developers use moments like this as evidence that AI can't be trusted. "Be careful. It screws things up." Glass half empty. Case closed.
I went the other direction. I asked myself: why did it break?
The answer was simple. I let AI change code without telling it what "correct" looked like.
That's on me, not the AI.
So I built a framework. Nothing fancy — just discipline:
1. Preserve the original.
Keep the working code untouched. This is your baseline. Always test new against old.
2. Test date ranges at every granularity.
Full range, monthly, daily. Counts must match. If your sproc handles dates, this is where breaks hide.
3. Exercise every parameter.
Understand what each parameter does. Create test cases for every key option. Check counts. Sum key amounts. Compare old vs. new.
4. Let AI do the testing.
Here's the beautiful part — AI runs these comparisons in seconds. It presents every case tested with results. That output becomes your evidence for code review, your Jira ticket documentation, and your Zephyr test case. All at once.
When results don't match, AI tries alternative approaches. If it still can't match, it leaves the code untouched and flags it for human review. That's the right behavior — but only if you've set up the expectation.
The lesson: AI doesn't know what "correct" means unless you define it.
Think about it this way — developers complain when a business analyst doesn't give enough scope to fully understand a task. There's even a button in most ticketing systems: "Send back — more info needed."
Coaching AI is the same dynamic. You're the business analyst AND the QA team. Define the expected results. Set the boundaries. Then let it work.
Here's a revelation I had recently: if you can find the original Jira ticket from when a process was first built — the intent, the comments, the acceptance criteria — and feed that to AI along with the code? It will understand the process better than most humans who've inherited it. AI consumes context extremely fast.
The developers I hear complaining that "AI broke the code" aren't wrong about what happened. They're wrong about why. It wasn't AI's fault. They gave it a task without giving it a definition of success.
Coach it. Define the expected results. Let it test its own work. Then verify.
That's not slower. That's faster AND safer.
I'm lucky to work somewhere that mandates AI optimization but also gives us room to develop these patterns. Not every team has that — but every team needs it.