DEV Community: Igor Potapenko

Your idempotency test probably cannot fail

Igor Potapenko — Wed, 19 Aug 2026 14:20:29 +0000

I had a test called "stays quiet when the plan is already what the event grants". It passed on every run for two weeks.

Then two identical "payment received" messages arrived on my phone, 142 milliseconds apart.

The test was not weak. It could not have failed. And that is a more interesting problem than a missing test, because a missing test at least looks like a gap — this one looked like proof.

What the code did

Paddle sends subscription.created and subscription.activated for a single purchase, and it retries anything that does not answer 200. So two deliveries for one payment is not an edge case, it is Tuesday.

I knew that. I had written the dedupe:

const { data: profile } = await supabase
  .from('profiles')
  .select('plan')
  .eq('id', userId)
  .single()

const alreadyOnPlan = profile?.plan === plan

await supabase.from('profiles').update({ plan }).eq('id', userId)

if (!alreadyOnPlan) await notifyPayment({ userId, plan, amount })

Read the current plan, compare, write, announce only if it changed. It reads correctly. It is correct, for one caller.

Two deliveries arrive. Both read free. Both see alreadyOnPlan === false. Both write basic. Both announce.

The window between the read and the write is where the whole bug lives, and it is exactly as wide as one round trip to the database.

Why the test agreed with me

Here is what I had written:

it('stays quiet when the plan is already what the event grants', async () => {
  db.setRow('profiles', { id: 'user-1', plan: 'basic' })
  await post(activation())
  expect(notifyPayment).not.toHaveBeenCalled()
})

Read it again with the bug in mind. It calls the handler once, against a row that is already in the target state. It asks: if I run this after the plan is already set, does it stay quiet?

That is a real question. It is not the question the bug is about.

A race requires two things to overlap. await post(...) runs one handler to completion before the next line executes, so the two deliveries never coexist. I could have added a second call, a third, a hundred — sequentially they would all pass, forever, while production kept sending doubles.

This is the part worth taking away. My test was not a bad test of concurrency. It was a test of something else that I had filed under concurrency, and the passing green tick was doing active harm: it told me the case was covered.

Making it fail first

The fix in the product code is small, and I will get to it. But the test had to fail before the fix, or I would have no evidence the fix did anything.

Two handlers have to overlap. In JavaScript that does not need threads — it needs the first handler to yield at an await while the second one starts:

it('announces once when two deliveries race each other', async () => {
  db.setRow('profiles', { id: 'user-1', plan: 'free' })

  await Promise.all([
    post(activation()),
    post(activation('subscription.created')),
  ])

  expect(notifyPayment).toHaveBeenCalledTimes(1)
})

Promise.all starts both, and the first one suspends at its first await — the read. The second handler runs its own read against a row nobody has written yet. That is the production interleaving, reproduced deterministically, in a unit test, with no timing hacks.

Run that against the old code and it fails: two announcements. Which is what a test is for.

The part I did not expect

It did not fail. It passed against the broken code too.

The mock database ignored the condition I was about to rely on. It recorded that an update happened and returned success; whether the row actually matched was not something it modelled. So a conditional write and an unconditional one produced identical results, and no test on earth could tell them apart.

I want to be precise about how bad this is. A missing test leaves a known hole. A test double that quietly simplifies the thing you are testing produces confident wrong answers, and it produces them in the exact area you thought you had covered. It is the same failure mode as the original bug, one level up.

So the mock had to learn the one behaviour that matters here: a conditional update is a check and a write in a single step, and the row is changed before anyone else can read it.

if (pendingWrite === 'update' && notEquals.length) {
  const current = state.rows[table]
  const blocked = notEquals.some(([col, val]) => current?.[col] === val)
  if (blocked) return { data: [], error: null }
  if (current && writeValues) state.rows[table] = { ...current, ...writeValues }
}

Now the second caller in the Promise.all sees what the first one wrote. Now the test fails on the old code and passes on the new one, which is the only property that makes a test worth keeping.

The fix

Once check-and-act has to be one step, the shape of the answer is forced. The condition moves out of the process and into the write:

async function setPlan(userId: string, plan: Plan): Promise<{ changed: boolean }> {
  const { data, error } = await supabaseAdmin
    .from('profiles')
    .update({ plan })
    .eq('id', userId)
    .neq('plan', plan)          // ← the whole fix
    .select('id')

  if (error) throw new Error(`Failed to update plan: ${error.message}`)
  return { changed: Array.isArray(data) ? data.length > 0 : Boolean(data) }
}

Postgres decides who was first. The update that actually matched a row returns it; the loser gets an empty list. Only the winner announces.

Note what did not change: both deliveries still attempt the write. That matters — if the first one's write is lost to a network failure, the second one heals it. A dedupe that skips the write on the second delivery would turn a lost message into a lost purchase.

Three questions I now ask my tests

Can this test fail? Not "does it pass" — can I write a version of the product code, plausible enough that I might have written it, that this test does not catch? If the answer is no, the test is decoration. Deleting the fix and watching the test go red takes ten seconds and is the only way to know.

Does the shape of the test match the shape of the bug? Races need overlap. Retries need repetition. Ordering bugs need the wrong order. A sequential test cannot express a race, in the same way a unit test cannot express a deployment problem — not "is unlikely to catch", cannot express.

Does my test double model the property I am relying on? I relied on a conditional write. My mock did not have conditions. Every assertion built on top of that was measuring the mock, not the code. If the fix depends on a database guarantee, the double has to implement that guarantee or the test is theatre.

Why this keeps happening

I write this product alongside an AI, and this bug is a good example of what that changes and what it does not.

The dedupe it wrote was reasonable. Read, compare, write — that is what the task sounded like, and it is what most people write by hand too. The failure needed context nobody stated: that this specific provider fans one purchase into two events and retries on top of that. Nothing in the code says so. It is knowledge about the outside world, and it is exactly the kind of thing that will not be in the diff.

The test was mine. I wrote it to feel covered, and it worked — I felt covered for two weeks, until my phone buzzed twice.

The lesson is not "review AI code more carefully". It is narrower and more useful: a passing test is a claim, and claims about concurrency made by sequential code are worth nothing. That was true before any of this, and it will be true after.

Built while working on prepair.app, an AI interviewer for practising before the real thing. If you have a favourite test that turned out to be incapable of failing, I would like to hear it.

Five bugs AI wrote for me that never threw an error

Igor Potapenko — Fri, 14 Aug 2026 13:18:44 +0000

I have been building an interview practice tool for the last two months, writing it alongside Claude. Not "generate me an app" — ordinary daily work where I set a task, look at the result, argue with it and rewrite.

Plenty has been written about how fast AI writes code. I find the other question more interesting: what does it break in a way you cannot see?

All four bugs below are real, from my own project. What they have in common is that none of them throws. No exception, no 500, no red line in the logs. Every one of them looks correct in review. You find out when a user tells you — or you never find out at all.

1. The counter that never counted

Free plan, three interviews a month. Simple logic: questions generated, counter goes up.

Roughly this:

if (!isPaid) {
  incrementInterviewsUsed(profile.userId).catch(() => {})
}
return NextResponse.json({ questions })

That reads fine. The error is swallowed deliberately, so a database hiccup does not break somebody's interview. The limit is checked above, the write happens below.

Except the write does not happen.

This is Vercel — serverless. The function returns its response, and the platform is entirely within its rights to freeze it immediately afterwards. A promise nobody awaited simply never reaches the database. Sometimes it lands, sometimes it does not. No pattern to it.

Here is how it surfaced: a user runs their fourth interview and the dashboard says "1".

The fix is one line:

if (!isPaid) {
  try { await incrementInterviewsUsed(profile.userId) }
  catch (e) { console.error('[Quota] increment failed', profile.userId, e) }
}

That is not the interesting part. The interesting part is why it survived for weeks.

I had tests on the quota. They checked that a request is rejected when used = 3. In other words they tested the read path. The broken thing was the write path — the one responsible for used ever reaching three in the first place.

The rule I took away and now keep written down: anything that enforces a limit or touches money gets tested on the write path, not on the logic surrounding it.

2. The model obeys the last thing it read

The product generates questions for a role and a seniority level. The complaint: "I picked senior and I am getting junior questions."

The prompt was assembled from blocks, roughly like this:

const prompt = `
  ${roleGuide}
  ${mixRule}                 // "60% theory, 40% practice"
  ${levelGuide[difficulty]}  // "for senior: architecture, trade-offs..."
`

And mixRule said something along the lines of "these proportions override the block above."

The level block was interpolated after it.

A model does not read a prompt as a specification with a hierarchy. It reads it as text. When one place says "this overrides the block above" and something further down says otherwise, the thing closest to the end wins.

There was a second layer to the same problem. The proportions rule historically lived inside the branch for QA roles. Twenty-one roles out of twenty-three never saw it at all. That one was not the AI's mistake — it was mine, faithfully replicated when it added new roles by following the existing pattern.

Fixed by reordering the blocks and lifting the rule out of the branch. The broader lesson: a prompt is code with no compiler. Nothing will tell you two instructions contradict each other. The only way to find it is to read the assembled prompt end to end, with your eyes, the way the model sees it.

3. Zero interviews for every paying customer

There is a B2B tier: a team, with an admin dashboard showing each member. Among other things, how many interviews each person has done.

That number came from interviews_used on the profile.

The same field from bug one. The free plan quota counter, which is deliberately never incremented for paid plans — why count something that is not limited.

The team plan is a paid plan.

So on the single screen a company pays for twenty seats to look at, every employee showed zero, permanently. The data existed — interviews were happening, scores were being saved. The dashboard was simply reading the wrong column.

This one is not a technology mistake. It is a mistake about one field meaning two different things, with nothing in the name to say so. interviews_used reads as "interviews this person has done". It actually means "how much has been drawn down from the free quota".

Everything is now counted from the sessions themselves, and the quota field stayed a quota field.

4. The schema did not exist

The worst of the four, even though it broke nothing.

The product runs on two tables — interview sessions, and per-answer feedback. I created both by hand in the Supabase dashboard at some point. Fast, convenient, works.

The migrations folder in the repository described different tables. The ones the project started with and stopped using long ago.

Nothing failed. Nothing could fail: production has the tables, the code queries them, all good. It would have diverged at exactly one moment — standing up a staging environment, or restoring the database after an incident. Then the application would go looking for tables that are not there and fail to come up.

I went a long time without noticing precisely because there are no symptoms at all. It turned up by accident while I was writing team reports and went to check which columns a table actually had.

One more detail from the same dig. The original migration had:

check (plan in ('free', 'basic', 'pro'))

The team tier came later. I fixed the constraint in the dashboard and never carried it back to the repository. On a fresh database every B2B purchase would have been rejected by Postgres.

The fix: a migration describing what actually runs, plus a test that reads every table reference out of the source and requires each name to be created in a migration. The next hand-made table fails CI.

for (const [table, files] of queriedTables()) {
  it(`creates ${table}`, () => {
    const created = new RegExp(`create table (if not exists )?${table}\\b`)
      .test(migrationSql)
    expect(created, `${table} is queried in ${files[0]} but no migration creates it`)
      .toBe(true)
  })
}

There is a nice epilogue. The first time I ran that migration against the live database it failed with cannot change return type of existing function. Two columns in my version of a function were in a different order from the real one. The application reads fields by name and does not care — but Postgres treats the set of output parameters as a row type and refuses outright. The database itself showed me how far the description had drifted from reality.

The one that got a real user

While writing this I found a fifth, and it is the one that actually cost something.

The interview screen requested its questions on every mount. Generating questions charged a free interview. So a refresh, a back-and-forward, or the installed app being relaunched quietly spent another of somebody's three.

The first stranger who ever found this product did exactly that: two sessions, forty seconds apart, zero answers in either. She left having lost two thirds of her free quota without seeing the tool work once.

Generating questions is now free, and the interview is only charged when the first answer is sent. A reload picks up where it left off.

And when I wrote the test for it, the test failed — because the database mock returned nothing from a write, while real PostgREST returns the rows it touched. The product code was right; the thing meant to verify it was wrong, and it failed in the most convincing way possible, by quietly reporting that nothing had happened.

What I take from this

What got me was not that the AI makes mistakes. Everyone does.

It was how it makes them. A person who does not understand the task writes code you can see is wrong — it is clumsy, it does not compile, it falls over on the first run. Code written by a model looks like code written by someone experienced. Sensible names, tidy structure, thoughtful error handling. And exactly where it lacks context — how a serverless function gets frozen, what a particular column has historically meant — it will confidently write something that looks reasonable and is wrong.

Three things that actually work against this, in my experience:

Test the write path, not the logic around it. My most expensive bug was in a line I wrote myself and verified myself — from the reading side.

Write tests that compare the code to the outside world. Not "this function returns the right value", but "what the migrations declare matches what the code asks for". Those catch an entire class of problem that unit tests cannot see by definition.

Check it on the running product. Four of the five above were impossible to spot in code. You only see them by opening the screen and looking at the number.

None of this is new advice. What is new is how much it matters now, when the volume of plausible-looking code you did not write yourself goes up by an order of magnitude.

The junior QA bar keeps rising — and here is what each requirement is actually for

Igor Potapenko — Tue, 11 Aug 2026 22:17:37 +0000

Twelve years ago, a route into testing could look like this: turn up at a conference, listen to a talk, walk up to the speaker afterwards and ask one good question. An offer arrives the following week.

That is not nostalgia talking. It was ordinary. There were more openings than people, the industry was growing faster than anyone could train for, and companies hired for the ability to think and planned to teach the rest on the job.

Today the same position asks for test design techniques, an understanding of HTTP, the ability to write a SQL query with a join, some idea of what XSS and SQL injection are, experience with Postman, and increasingly "automation is a plus".

The first reaction to that list is usually: is this reasonable? Let us be honest about it — some of those requirements are earned, some are not, and the difference is worth knowing.

Why the bar went up

The ratio of applicants to openings inverted. In 2013 someone with English and a willingness to learn was scarce. Now a junior posting draws hundreds of applications. The interview stopped being a conversation and became a filter, and filters need criteria — preferably formal ones that are easy to compare across candidates.

The job itself changed. A QA engineer twelve years ago mostly clicked through a UI against a script and filed bugs. Today the same junior opens devtools, watches the Network tab, goes to Postman to see what the backend actually returned, and to the database to confirm it was stored. Not because it is fashionable, but because products became distributed and half the defects no longer live on the screen.

The runway for learning got shorter. Teams are smaller and release more often. A junior used to have six quiet months growing next to a senior. Now they are expected to be useful by the second sprint, and companies try to buy that readiness at the door.

And part of it is simply requirement inflation. Once every posting says "SQL knowledge", the next one says it too, because leaving it out looks like advertising for someone weaker. That is how things a junior will never touch in the role end up in the requirements for it.

That last point matters and we will come back to it. First, the parts of the list that are there for a reason.

What each requirement is actually for

Test design techniques

These get treated as theory for the sake of theory: learn equivalence partitioning, recite it, forget it.

In practice the technique answers the one question you will be asked constantly at work: why did you test that, and why is that enough?

Without a technique the answer is "well, I clicked through the main flows". That is not an argument — it cannot be checked or challenged. With one it becomes: "the field takes 18 to 65, I tested one value from each class plus the boundaries — six tests instead of a hundred and twenty, and here is why the rest add nothing."

The second, less obvious point: techniques are how you fit into the time you have. Testing everything is never an option. Test design is a tool for cutting scope defensibly, not an academic subject.

SQL

The usual objection is "why does a tester need the database, I test the interface".

The interface shows you what the frontend rendered. It does not show you what was stored.

The classic case: a user saves their profile, the page says "saved", you move on. In the database the field is still empty, because the backend returned 200 for a request that changed nothing. A customer finds it a week later.

The second use is more frequent still — test data. You need a user with an expired subscription, three orders and one returned item. Through the UI that is half a day of clicking. As a query it is a minute.

The volume a junior actually needs is small: select with where, sorting, a join across two tables, count with group by. That is not "knowing SQL", it is four constructs.

API and HTTP

The reason here is the simplest of all. The core of a junior QA's job is not finding the defect. It is localising it.

"It does not work" is not a bug report. A developer will spend an hour establishing what you could have established in a minute: did the request go out, what came back, is this a 400 caused by your input or a 500 caused by their code.

That is where the questions about status codes, devtools and Postman come from. It is not trivia, it is being able to say whose side the problem is on. A tester who can do that saves the team hours every week, which is exactly why it is asked at the door.

There is also functionality with no interface at all. A payment provider integration, a webhook from an external service, a background job — the only way to check those is with a request.

XSS, injection and basic security

This looks like the least junior item on the list. But the reason is not depth, it is the cost of the miss.

A missed layout bug is a bug. A missed XSS is an incident: leaked data, a notification to users, sometimes a fine. The difference is not difficulty, it is consequences.

And the cheapest checks cost nothing. Paste <script>alert(1)</script> into a name field and see whether it executes. Put a single quote in a search box and see whether the query falls over with a database error. It is a five-item checklist that takes ten minutes and closes off the cheapest class of attack there is.

Nobody expects a junior to run a penetration test. They expect you not to walk past the obvious.

Environments and logs

Questions about dev, staging and production, and about how to read logs, get the same puzzled reaction. The reason is practical again: half of all "bugs" are not bugs but the wrong environment, a stale build, or a config difference.

A tester who checks which build reproduces it and what the logs say before filing files less noise. And noise in the tracker costs the team more than it looks like it does.

And what is genuinely excess

Now the honest part.

Asking a junior manual QA about the layers of the OSI model, replication in SQL Server, or the difference between SIP and PRI is not an assessment of readiness for the job. It is either a list copied from somewhere or an attempt to filter somehow when two hundred people applied.

The test for a healthy interview is simple: for every question, the interviewer can say how it will come up in the first month of work. "You will be checking data in the database" — they can. "You will be configuring replication" — not in a junior role.

If most of the questions fail that test, it tells you something about the company's hiring process rather than about you. That is a perfectly good reason not to regret the rejection.

What to do about it

The bar is not coming down; the old ratio of people to openings is not coming back. But the list of what you genuinely need is far shorter than the postings suggest.

The minimum that clears most junior interviews:

Test design — equivalence partitioning, boundaries, decision tables. With worked examples on concrete fields, not definitions.
HTTP — methods, status code classes, the Network tab, being able to say whose side an error is on.
API — Postman, GET and POST, reading a response, checking the things the UI does not show.
SQL — select, where, order by, join, count with group by. Four constructs.
Security — a five-item checklist: XSS in input fields, a quote in the search box, another user's record by id in the URL, password masking, and an error message that does not reveal whether a login exists.
Bug reports — and understanding that the preconditions matter more than the screenshot.

That is a few weeks of focused preparation, not a year of study. Most rejections happen not because the candidate has never heard of these things, but because they know them as "I have read about that" rather than "I can apply it and say why it matters".

One last thing. Every item above is worth being able to say out loud. Written down, your reasoning looks complete. Said for the first time in an interview, it comes out in a different order and half of it goes missing. That is not a knowledge problem, and reading does not fix it.

We put together Manual QA interview questions with model answers — split by junior and middle, free to read. And we built Prepair, where you answer those questions instead of reading them and get scored on each one. Three interviews a month are free.

The DevOps interview question that predicts an outage

Igor Potapenko — Thu, 06 Aug 2026 15:16:25 +0000

There is a specific way to turn a slow database into a total outage, and it fits in one line of YAML.

You point your Kubernetes liveness probe at an endpoint that checks the database. The database gets slow. The probe times out. Kubernetes concludes the container is wedged and restarts it. It restarts every replica, all at once, because they all failed for the same reason at the same time. What was a degraded service — slow, but serving — is now nothing at all, and the restarts keep the database busy enough that it cannot recover.

I have never met an engineer who did this on purpose. It happens because liveness and readiness probes look interchangeable in the documentation, and the difference only becomes obvious at three in the morning.

That is exactly why it is a good interview question.

Naming tools is not the same as knowing what to do with them

Most DevOps interviews are a vocabulary test. Do you know what Terraform is. Have you used Kubernetes. Can you list the stages of a pipeline. A candidate who has read the docs passes. A candidate who has run things in production passes identically, and you learn nothing about the difference between them.

The job is not remembering what a Deployment is. The job is deciding what to roll back first when two changes went out together and you can only undo one. It is deciding what deserves to wake someone up. It is deciding what to leave broken until morning — which is a real decision that real engineers make, and one that never appears in a tutorial.

So the useful questions are the ones with a fork in them, where a confident answer either way is fine and the reasoning is the whole point.

Five questions that actually separate people

"What is the difference between a liveness and a readiness probe?"

Everyone gets the definitions. Readiness controls whether traffic arrives; liveness controls whether the container is restarted. The follow-up is where it gets interesting: what happens if your liveness probe checks a dependency?

What you are listening for is whether the candidate sees the amplification. A readiness failure removes one pod from rotation and the rest carry on. A liveness failure kills the pod. Point liveness at something shared and you have built a mechanism that converts partial degradation into simultaneous, cluster-wide restarts. An engineer who has been burned by this answers immediately and with feeling.

"A pod is in CrashLoopBackOff. Give me your first three commands."

This one is unfakeable. There is no clever answer, only a practised one.

A good answer starts with kubectl describe pod, because the events section holds image pull failures, failed mounts, OOMKilled, and probe failures — most of the answer, before you have read a single log line. Then kubectl logs --previous, because the container currently running may be seconds old and have logged nothing useful. Then a look at the exit code, and if it smells like OOM, a comparison of the memory limit against actual usage.

What is telling is not the exact commands but the order. People who debug from experience narrow the space before they start reading. People who have only read about it start with logs and get lost.

"What is an error budget, and how would you use it in an argument with a product manager?"

The definitions part is easy: an SLI is what you measure, an SLO is the target over a window, and the error budget is the gap you are allowed to spend.

The second half of the question is the real one. An error budget is not a metric, it is a negotiating device. It converts "we should slow down and fix reliability" from an opinion — which loses to a roadmap every time — into arithmetic that both sides agreed to in advance. Candidates who have used one describe it that way. Candidates who have only read the SRE book describe the maths and stop.

"What is Terraform state, and why is it dangerous?"

The trap here is answering only the first half. State maps your configuration to real resources so Terraform knows what to create, change, or destroy.

Why it is dangerous is the part that matters. It is authoritative: lose it and Terraform no longer knows it owns your infrastructure, so it offers to build everything again. It frequently contains secrets in plaintext. And two engineers running apply at the same time without locking can corrupt it. A candidate who has actually operated Terraform gets to remote backends, encryption, versioning and state locking without being prompted, because they have felt the fear.

"You are deploying a service with a database migration. Rolling, blue-green, or canary?"

There is no correct answer, which is the point.

Rolling needs almost no spare capacity, but two versions serve traffic during the roll — so the migration has to be backward compatible whether you like it or not. Blue-green gives instant rollback and doubles your infrastructure, and the migration problem gets worse rather than better, because the two environments share one database. Canary catches problems on real users but only pays off if your observability is good enough to notice within the canary window.

Anyone who picks one instantly without mentioning the migration has not thought about it. Anyone who says "it depends" and stops has not either. The answer you want names a choice and then names what it costs.

If you are the one preparing

The pattern across all five is the same: the question sounds like it is about a tool, and it is actually about a decision.

So when you prepare, do not stop at the definition. For every concept you revise, ask yourself what it looks like when it goes wrong, what you would check first, and what you would give up to fix it. If you can only recite what something is, you will answer the first half of the question and go quiet on the second — and the second half is the one being graded.

The other thing worth saying: reading answers is not the same as saying them. Written down, your reasoning looks complete. Out loud, under mild pressure, with someone waiting, it comes out in a different order and half of it goes missing. That gap is not a knowledge problem, and revising harder does not close it. Practice does.

If you want the longer list, I keep DevOps and SRE interview questions with model answers written up, free to read.

I build Prepair — an AI interviewer that asks these questions out loud and scores the answers. Writing this list is how I work out what the product should ask.

Good luck. And check where your liveness probe is pointing.