DEV Community: Gil Zilberfeld

AI vs CI: Nobody Told the Pipeline

Gil Zilberfeld — Thu, 20 Aug 2026 11:17:28 +0000

CI is so commonplace these days, we don’t often think about why it’s there. And that reasoning really matters these days, because of how we use it.

Especially when you’re writing AI agents and expect the CI system to run a regression test suite, and sound the alarm when something goes wrong.

Here’s a funny story you may not know. CI - Continuous Integration – was not originally a tool category. It’s a process - it’s how you continuously integrate pieces of code together.

Anyway, why do you need this process anyway? Because you’re scared. You’re terrified that integration broke something.

So what do you do? You look for approval. An automation system that runs all the tests all the time is our approval of choice.

That’s how CI became a tool category. Because in its heart, it’s a simple automation pipeline.

So, that automation’s holy grail is speed, right?

Wrong.

The mission was to kill “Works on my machine”

Remember when we wrote software in a cave, and we had a sticker on our computers – “Works on my machine. Don’t run it on another cave”?

Ah, the good old days.

Because the CI automation #1 OG mission was to kill “Works on my machine”. How? By creating a repeatable process. Independent of all the weird installs on my machine. Multiple versions of libraries. And different configurations. And admin hacks.

The CI builds, packages, runs and tests the software every time the same way. That way, if a test turns red, you know something bad happened, because it was green until now. And why your stomach turns, when it flakes between red and green – that’s the feeling of lost repeatability.

Yes, repeatability brings bliss. Well, it did, before AI.

Then AI showed up

If your app does not touch AI (although building with code agents, can sometimes count as “touch”), you’ll continue to feel that bliss.

But if you’re developing AI-based features, or agents, or vibe-coding, you’re in for a new experience every time you push your code (or prompts) into CI.

Things are not repeatable anymore. Models change, sometimes without warning. And always under your feet. And when a model changes, does it trigger a run?

You wish! Your alert system is malfunctioning.

And the worst part is not a regression. Although if it’s a bug in a prompt, you may not be able to fix it.

No, the worst part is that Green today doesn’t tell you what kind of Green it is. Things are no longer Working or not, they are Work-ish. But even that’s not the same work-ish every day.

So we’ve got fewer triggers, results we can’t read, and on top of it – every run means something else.

What’s the solution?

Repeatability gives us confidence. CI is the process, and tools, that give us the approval we seek, and the confidence we have comes from that repeatability.

We can’t rely on the trigger anymore, but we can initiate our own runs. And since one run is not enough, we need to run more and look at the trends. Catch drift before it ships.

What happens to quality when AI meets CI? We need to get back to the reasoning, and change how we manage quality.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

It's Almost Like You Need an Actual Programmer

Gil Zilberfeld — Thu, 13 Aug 2026 07:07:33 +0000

Vibe coding ain’t what it used to be.

Back in the old days (2 years ago), we conjured up apps with our own programming language.

Turns out, it does work, but more for prototypes. If you want an actual working production app, you need that thing from before. How do the old timers call it? Ah yes. Code.

So our builder says to the genie: “Code me this”.
And the genie does build this. Mostly.

When we’ve got a prototype and it’s working, two things happen. The first thing is management thinks they have an actual product. In our case, it’s the builder that thinks “code me this” is simply a translation layer.

But then, another thing happens – we now have to take care of our puppy. I mean the code.

As long as we don’t need to touch it, it doesn’t matter what it looks like. But then we do.

So what does our hero think? What worked before, will work again. He tells his favorite code agent – “build me my greatest feature yet”.

But then weird things start happening. Bugs that weren’t there. Multiple tries of code changes – costly ones, mind you. Performance slows down.

It’s almost like you need an actual programmer to do the work.
And you do. You know what helps developers? And coding agents?

Clean code. Readable, maintainable code. The classics are still with us.

Now, coding agents are getting better at producing code. However, they are only as good as the code they are trained on. Now think – what do you think the average level of code out there is?

Exactly. That’s what you’re getting.

So if you are on a transition from prompts to code, it may look like continuing vibing is the shortest way to more features. Yet, the first thing you need is a good platform to grow your code.

Now it’s time to take control.

Read the code.

You’ll do this first anyway. The code is the only truth you have. The prompts you used before are scattered across forty chat sessions and there were never any docs.

Just know what you’re up against. Agents drop a lot of code, and after ten files your eyes glaze over. You won’t read all of it, but try to understand the important bits.

Write down what it should do.

Ask the agent to derive documentation from the codebase. It’s fast and it beats no docs at all.

But know what you got. Those docs describe what the code does, not what it should do. If there’s a bug in there, it appears as a requirement. Review and sieve out the wrong stuff.

If you can find them, collect your prompts. They were the intent for building the app.

Review both, and drop everything that doesn’t add up. Now you have a documented system.

Now write tests. Against the requirements.

It’s tempting to generate tests for what’s already there. And for legacy apps, I recommend capturing existing behavior of undocumented knowledge built up for ten years.

But here? You’re better set. The intent is still around. Capture what the code does today and you’ve just made every mistake a requirement.

Then you can change it.

Including asking the genie to change it for you – which is the part everyone wants to skip to.

But now, with documentation and tests in place, now it’s a lot safer to make changes.

Next you need to understand the principles of Clean Code. If you don’t, you may ask the genie to change the code for you, but you’ll get more of the same. We don’t want more of that mess.

Exactly like programming. Who knows, maybe you’ll become one eventually.

So what do you say – Are you ready for clean up?

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

The New Technical Debt

Gil Zilberfeld — Thu, 13 Aug 2026 07:06:32 +0000

Is AI code debt the new technical debt? As I keep learning, generating code and examining it – it’s worse.

In the old days, we worked hard to create technical debt. Just kidding, it was easy.

Technical debt has many definitions, and origin stories. But one thing is common: We leave the code as it is, knowing it could be better. The debt is the gap of effort of making it better.

And why the gap? Because we know we’ll see that code again. And when we do, it’ll be hard to change. Better code would have made it easier.

Now we’re in the age of genies. They can write any code. They can change any code. Do we need to worry about AI code debt?

Sure we do.

AI code debt is exactly the same as technical debt – we’re leaving the code as it is, knowing it could be better. The gap is still the effort of making it better.

But this time the gap is a lot bigger. There’s a lot more code, it’s probably a lot more complex, and some of it – let’s be frank – is code we didn’t review. So the gap is a lot bigger than we guess.

But that’s a bot problem, right? The code agent will deal with all the needed changes. We don’t need to even look at the code.

Nah, you know you will. You know what code is generated, and it’s not how you would have written it. And coding agents have the same problem of making sense of complex code bases as us.

And they’ll make mistakes.

Can the code agent clean the code up by itself?

Cleaning is not just “make it readable”. It’s preparing it for more changes, reducing dependencies and isolating interfaces – all in the context of future plans. And the agent doesn’t have this context.

In fact, it will assume another context, and we get stuck with the code and the assumptions.

How do you pay down AI code debt?

Same as always: find the code you know you’ll touch again, and make that code easier to change. The problem starts when you don’t know where that code is.

Because, you didn’t write it. And you didn’t review all of it. If any.

Putting a price on technical debt before was an exercise in imagination. Now it’s pure fantasy.

How do you avoid it next time?

First know what code was generated. And for that you need to enforce smaller code generation.

If you don’t enforce it, you’ll have a lot more to review. And if that happens you won’t review it all. It’s a human thing.

Then, you can wish the genie to refactor it to take the shape you want. Genies are good at transformations. And if the genie breaks something, your tests will tell you.

You do have tests, right?

The old technical debt was based on maintenance work. Now we have bigger maintenance queued up, along with risks of unverified code. Not cool.

So, first, we need to be aware of the AI code debt – we’re creating code that will cost a lot more to maintain, than “regular” code.

Before, we thought “it’s ok, we’ll take care of that later”, and then, when the bill came it was a lot more than we thought.

Now? Expect a much bigger one.

Questions people ask about AI code debt

What is AI code debt?

AI code debt is technical debt created by generated code: you leave the code as it is, knowing it could be better, and the debt is the effort of making it better. The difference is scale. There’s more of it, it’s more complex, and some of it was never reviewed.

Is AI code debt different from technical debt?

Not in kind, only in size and visibility. The old debt was something you put there and remembered. This debt arrived while you were reading something else, so you don’t know where it is.

Can an AI agent clean up its own code debt?

Not on its own. Cleaning up means preparing code for changes you plan to make, and the agent doesn’t have that context. It will assume a different one, and you’re left with the code and the assumptions.

How do you check that an AI refactor didn’t break anything?

With tests you wrote before the refactor. Directing the genie to reshape code is fine as long as something independent tells you when it breaks.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.

You Can't Fix A Bug With A Prompt

Gil Zilberfeld — Thu, 06 Aug 2026 10:46:27 +0000

Let's talk about fixing bugs. My second favorite activity after creating them.

How do I know the bug is fixed? Well, I've been taught by wiser and more experienced people than me (way back when), that if you want to make sure the bug never shows its face again - write an automated test for it.

Ok, I admit that there were times I didn't write a test. Some fixes are so trivial, that sometimes a test is a luxury.

But even then, I was working on a hidden assumption. Just like in Fallout, code never changes. Until it does. But between those times it never changes.

I fix the bug. From this point until it actually changes by a human, or a bot or someone in-between - that code will compile, or transpile, or whatever we call "runs" the same way. A statement will execute, a condition evaluated, an exception caught - every time in the same way.

And now we come to modern programming languages: Prompts. Agents, prompts, workflows - they don't run the same way every time. What they do is run their interpretation of the request. If they run tools - they run the deterministic parts. If they run sub-agents, they run interpretations based on other interpretations.

Interpretations work mostly the same way, until they don't. When I was teaching my agent to develop in TDD, I had a couple of requests. One of them was to not create code without a test.

Which I thought was a very normal request from an agent. In fact, when I started out by telling it - we're working in TDD - I assumed it knows what TDD is. Ha.

Always remember that what LLMs know is exactly what most of the population knows. And usually the "don't create any code without a test" falls through the cracks.

So I made it official: One of the agent rules was exactly that. This was a legitimate bug fix. At least I thought so.

But it really was a suggestion. Which the agent considered, and depending on its mood, sometimes did, and sometimes didn't.

I won't go into the full solution (still in progress, if you want me to elaborate, comment), but part of it was to run a targeted coverage tool - a deterministic one, check it and stop the process if it found extra code. The TDD sequence looks like this:

Per step:
1. Write test(s).
2. `node scripts/tdd.mjs red <test-file>` — all newly added tests must fail. Pre-existing passing tests in the same file are allowed to remain passing.
3. Implement minimum to pass.
4. `node scripts/tdd.mjs targeted <test-file>` — targeted coverage + pass check.
5. `node scripts/tdd.mjs lint` — ESLint.
6. `node scripts/tdd.mjs full` — full suite + coverage.
7. Script says STOP. Human reviews.
8. Human runs `node scripts/tdd.mjs commit "message"`.
9. Human confirms next step. AI runs `/clear`.

But each step here is the suggestion. The real enforcement is done in the tdd.mjs code. Real code.

But this is just an example. The real problem is that more and more "code" is not programmed. It's interpreted. That means that bugs are a lot more flaky to catch, but also are not permanently fixable.

And don't get me started on companies switching model capabilities every other Tuesday. In the past, updating versions was a whole ceremony because we were worried something would break.

Now LLM providers do it for us without us knowing.
We can live with that. We should make sure we know.

And remember - bug fixes in prompts are not real bug fixes. They are more like wishes. Which may or may not come true.

Originally published at testingil.com.

I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.