DEV Community: Libme

When Should You Move Off Managed Hosting to Kubernetes? The Thresholds That Actually Matter

Libme — Thu, 20 Aug 2026 08:42:41 +0000

If you are trying to decide whether it is finally time to run a cluster, the honest answer is that request volume is almost never the trigger. The triggers are structural: per-tenant isolation you cannot express in your database, workloads you do not trust, a compliance boundary that requires network-level segmentation with an audit trail, or a pile of homegrown scripts that have quietly become a bad orchestrator. Before any of those, you should be able to prove you have run out of headroom on the managed platform you already pay for.

A reader on an earlier post about boring stacks made the sharpest version of this point: the real failure mode that pushes teams to Kubernetes is not high traffic, it is complex multi-tenant stateful workloads and compliance boundaries that a managed platform cannot express. They then asked the question this post exists to answer — what specific metrics and team thresholds should sit in the decision table. Here is the one I use.

Why is "we're getting too much traffic" almost never the real trigger?

Because managed platforms scale vertically and horizontally long past where most teams assume they stop. A single large instance running a well-tuned application server handles a volume of traffic that surprises people who have only ever read about scale, and every managed runtime worth using will run several identical copies of your container behind a load balancer without you learning a new config language.

Before you accept "we need to scale" as an argument, measure your actual headroom. The database is usually the real ceiling, and it is the one people misread most often — an app that falls over at 300 concurrent requests is frequently exhausting Postgres connections, not CPU:

SELECT
  (SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max_connections,
  count(*) AS in_use,
  round(
    100.0 * count(*) / (SELECT setting::int FROM pg_settings WHERE name = 'max_connections'),
    1
  ) AS pct_used
FROM pg_stat_activity;

If that comes back at 95% while your instance CPU sits at 20%, Kubernetes solves nothing for you. A connection pooler and a smaller per-process pool size solve it in an afternoon. The same applies to the app tier: if your peak CPU utilization never crosses 40% on the instance size you are paying for, you do not have a scaling problem, you have a latency or concurrency problem hiding behind one.

If you cannot point at a specific resource you have saturated after tuning, you do not have a scaling argument — you have a discomfort.

What are the structural signals that actually justify a cluster?

These are the four I treat as genuine. Each one has a test you can apply today, and each one has a boring-stack workaround you should exhaust first.

Signal	How to test it	Try this first	If that fails
Per-tenant stateful isolation	Can one tenant's data or load reach another's, in a way a customer contract forbids?	Row-level security, or a schema/database per tenant on managed Postgres	Per-tenant namespaces with resource quotas
Untrusted or arbitrary workloads	Are you executing code you did not write — customer plugins, build jobs, notebooks?	Managed sandbox/build services, hard per-job timeouts	Real pod-level isolation and admission control
Compliance boundary	Does an auditor need to see enforced network segmentation and deploy provenance?	Separate managed environments per boundary, VPC peering rules	Network policies, RBAC, signed image admission
Deploy coordination	Count the lines of custom bash that restart, health-check, and roll out services	A managed runtime with health checks and rolling deploys	Declarative orchestration

The multi-tenancy row is where most teams misdiagnose themselves, so it deserves the concrete version. "Multi-tenant stateful workload" usually means shared rows in shared tables, and that is a database problem with a database answer:

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON invoices
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

Then set the tenant once per transaction, from your connection checkout or request middleware:

BEGIN;
SET LOCAL app.tenant_id = '3f1c2a9e-0b47-4f8a-9d21-5c6e8f0a1b23';
SELECT * FROM invoices WHERE status = 'open';
COMMIT;

FORCE ROW LEVEL SECURITY matters because policies are skipped for the table owner by default, which is exactly the role most applications connect as — that omission is the single most common way an RLS setup looks correct and enforces nothing. SET LOCAL scopes the setting to the transaction so a pooled connection cannot leak one tenant's context into the next request.

If that satisfies your isolation requirement, you did not need an orchestrator, you needed twelve lines of SQL. If your requirement is instead "tenant A's batch job must never consume tenant B's CPU," you have a genuine scheduling problem and the case for Kubernetes gets real.

Isolation requirements that live in your data model are database work; isolation requirements that live in the kernel are orchestrator work.

What team size can actually carry a cluster?

The number I use is not a headcount, it is an ownership commitment: someone whose job description includes the cluster, with roughly half a day a week of unglamorous maintenance budgeted, and at least one other person who can do a node upgrade when that person is on vacation. On a team of five engineers that is 10% of your engineering capacity gone, permanently, to something your customers never see.

The recurring work is real and it does not go away with a managed control plane: node group upgrades on the provider's deprecation schedule, ingress controller and cert-manager version bumps, CNI plugin compatibility, RBAC drift, and the periodic afternoon spent discovering that a pod is pending because of a resource request nobody remembers writing. Managed offerings like EKS and GKE remove the control plane from your worry list and genuinely lower the floor, but the worker nodes, networking, and deploy pipeline are still yours.

There is also a middle tier that most decision tables skip entirely. If your only real need is "run these containers, restart them when they die, roll them out without downtime," AWS Fargate gives you container scheduling with no nodes to patch, at the cost of slower cold starts and a weaker local development story. If your workloads are HTTP services that can tolerate scale-to-zero, Google Cloud Run handles request-driven autoscaling and TLS termination without exposing you to any cluster concepts at all, though it constrains you to its request lifecycle. If you want orchestration primitives without the Kubernetes ecosystem surface area, HashiCorp Nomad schedules containers and plain binaries with a config format a new engineer can read in an afternoon, with the tradeoff of a much smaller community and fewer off-the-shelf integrations.

Adopt Kubernetes when you need what only Kubernetes gives you, not when you need what any scheduler gives you.

How do you make the switch without a big-bang migration?

Move one stateless service first, and pick the least important one you have — an internal admin tool, a metrics exporter, a webhook receiver. It should be something whose outage costs you an apology, not revenue. That first workload is how you discover the parts nobody writes tickets for: image pull secrets, DNS resolution inside the cluster, log shipping, and how you actually get a shell when something breaks.

Keep state outside the cluster for as long as you can. Managed Postgres, managed object storage, and managed queues stay exactly where they are; running your own database inside Kubernetes is a separate project with its own operator, backup, and failover story, and taking both on at once is how migrations stall for a quarter.

Set an explicit rollback condition before you start — something like "if the admin tool is not stable on the cluster within three weeks, it goes back." Migrations without a stated failure condition tend to continue on sunk cost alone.

Your first cluster workload should be chosen for how little it matters, not how well it demonstrates the platform.

FAQ

How many services do you need before Kubernetes is worth it?
There is no clean count, but the useful proxy is deploy coordination, not service count. If you are running fewer than roughly five services and your deploys are independent, a managed runtime handles it; once services must be rolled out in a specific order, share service discovery, and are held together by custom scripts you are afraid to edit, the orchestrator is doing work you are currently doing by hand.

Is Kubernetes required for SOC 2 or HIPAA compliance?
No. Neither framework names any orchestrator. What auditors ask for is enforced access control, network segmentation between environments, encryption, and evidence of change management — all of which managed platforms can satisfy with separate environments and provider-level controls. Kubernetes becomes relevant when your segmentation requirements are finer-grained than the boundaries your platform lets you draw.

Can you run a multi-tenant SaaS without Kubernetes?
Yes, and most do. Tenant isolation at the data layer with row-level security or a database per tenant covers the majority of contractual requirements. You need kernel-level isolation only when tenants can trigger workloads that compete for CPU and memory, or when you execute code the tenant supplied.

Bottom line

If you are on a managed platform and shopping for a reason to leave it, measure your headroom first — connection saturation, CPU utilization at peak, and how much of your latency is your own code. Adopt Kubernetes when you hit a structural wall: untrusted workloads, isolation that must be enforced below your application, or an audit boundary your platform cannot draw. If you need scheduling but not the full ecosystem, Fargate, Cloud Run, and Nomad are the honest middle, and each costs you flexibility in exchange for the operational load it removes. And if nobody on your team can name the person who owns node upgrades six months from now, the answer for today is still no.

Where Should a Small Team Store Its Secrets? Comparing .env Files, SOPS, Doppler, 1Password, and Vault

Libme — Wed, 19 Aug 2026 21:42:06 +0000

If your team is under about ten people and already runs on one cloud provider, the cheapest correct answer is your provider's own secret store plus OIDC in CI, and no long-lived keys anywhere. If you're spread across several providers or hand secrets to non-infra teammates, a hosted secrets manager with a CLI injector — Doppler or 1Password — pays for itself the first time a rotation doesn't take an afternoon. Encrypted files in git (SOPS) are the right answer for a narrow case: config that must version alongside code, reviewed in PRs.

What almost never works past the second engineer is passing .env files around in Slack. Here's why, concretely, and what to replace it with.

The failure mode that ends the .env era

The bug that finally forced this decision on me wasn't a breach. It was a deploy that came up healthy and served traffic against the wrong database for twenty minutes.

The shape is always the same:

// db.js — the line that costs you an incident
const url = process.env.DATABASE_URL || "postgres://localhost:5432/app_dev";

A secret gets renamed in one place and not another, or a CI job runs before the env is populated, and process.env.DATABASE_URL is undefined. The || fallback is silent by design. Nothing throws, health checks pass, and the failure only surfaces when someone notices writes going nowhere.

Two fixes, and you want both. First, fail fast at boot so an absent secret is a crash, not a fallback:

// env.js — parse once, at startup, before anything connects
import { z } from "zod";

const schema = z.object({
  DATABASE_URL: z.string().url(),
  STRIPE_SECRET_KEY: z.string().startsWith("sk_"),
  SESSION_SECRET: z.string().min(32),
});

const parsed = schema.safeParse(process.env);
if (!parsed.success) {
  console.error("Invalid environment:", parsed.error.flatten().fieldErrors);
  process.exit(1);
}

export const env = parsed.data;

Import env everywhere instead of touching process.env directly, and add a lint rule banning process.env outside that one file. Now a missing secret is a loud, immediate, unambiguous crash.

Second, remove the drift itself by having exactly one place a secret lives. That's the actual decision below.

Fail-fast env parsing is the highest-leverage twenty lines in this entire post — do it before you pick a vendor, because it turns every secrets bug from a silent wrong answer into a stack trace.

What are the real options, and what does each cost you?

Approach	Best when	Real drawback
`.env` files, shared manually	Solo, one machine, throwaway projects	No rotation story, no audit trail, leaks via Slack/backups
SOPS + age/KMS, committed to git	Config that must version with code and be PR-reviewed	Rotation = commit + redeploy; key management is on you
Cloud-native (AWS Secrets Manager, Parameter Store, GCP Secret Manager, Azure Key Vault)	Already all-in on one cloud	Clumsy for local dev; per-secret and per-API-call billing adds up at high call volume
Doppler	Multi-provider deploys, want a fast CLI + integrations	Another vendor in your boot path; SaaS-first
1Password Secrets Automation	Team already lives in 1Password; humans and machines need the same vault	Service-account model takes a beat to grasp; usage-metered
Vault / OpenBao (self-hosted)	Dynamic short-lived DB creds, strict compliance	Genuine operational burden — unseal, HA, upgrades

A few honest notes on each, since the marketing pages won't give you these.

Cloud-native stores are the default nobody regrets on cost or reliability, and AWS Systems Manager Parameter Store's standard tier in particular is the underrated option — plain-string parameters with KMS encryption, no per-secret monthly charge on the standard tier as of mid-2026 (Secrets Manager bills per secret per month plus API calls, so check your call pattern before assuming it's cheap). The pain is local development: your laptop now needs cloud credentials to boot the app, which is exactly the long-lived key you were trying to eliminate.

Doppler is the one that handles the "same secret, five environments, three deploy targets" problem without a bespoke sync script, and its CLI injects secrets as environment variables for the duration of a process:

doppler run --project api --config dev -- node server.js

Nothing lands on disk, and doppler run composes with whatever your app already expects. The drawback is real: you've added a network dependency to your startup path, and you should understand its cached-fallback behavior before you put it in front of production boots.

1Password Secrets Automation is the right pick when the same credential needs to be readable by a person during an incident and by a machine during a deploy, because it's one vault with one audit log for both. The CLI resolves op:// references at launch:

# .env.template — safe to commit; contains references, not values
DATABASE_URL=op://prod/postgres/url
STRIPE_SECRET_KEY=op://prod/stripe/secret_key

op run --env-file=.env.template -- node server.js

Committing a template of references is the part that quietly fixes onboarding — a new hire clones, runs, and gets the right values without anyone DM'ing them a file.

Vault (or OpenBao, the Linux Foundation fork created after HashiCorp's 2023 license change) earns its complexity on exactly one feature: dynamic secrets. It can mint a Postgres user valid for an hour and revoke it automatically, so a leaked credential expires on its own. If nobody on your team wants to own unseal keys and HA topology, use the managed offering or don't use Vault — a badly-run Vault is worse than Parameter Store.

Vault is a correct answer to a question most small teams don't have yet; the question it answers is "how do I make leaked credentials expire by themselves."

How do you get secrets into CI without storing secrets in CI?

This is the part teams skip, and it's where the highest-value credentials sit. A long-lived cloud access key pasted into repository secrets is the single most valuable thing an attacker can get from your CI — it survives rotation of everything else.

Use OIDC federation instead. GitHub Actions can exchange a short-lived workflow identity token for cloud credentials, so no static key exists to steal:

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy
          aws-region: us-east-1
      - run: ./deploy.sh

The trust policy on that IAM role should pin the repository and the branch or environment — a wildcard subject condition means any workflow in your org can assume it. GitLab CI, CircleCI, and Buildkite all have equivalent OIDC flows.

If your CI still holds a long-lived cloud key in 2026, fixing that beats every other item on this list.

When is an encrypted-file approach (SOPS) actually right?

SOPS encrypts only the values in a YAML/JSON file, leaving keys readable, so a diff still shows which setting changed without revealing anything:

sops --encrypt --age age1ql3z... secrets.dev.yaml > secrets.dev.enc.yaml
sops --decrypt secrets.dev.enc.yaml

That property — reviewable diffs — is the whole argument. It fits GitOps and Kubernetes flows where config already ships through git, and it works offline with no vendor in the boot path.

The cost is that rotation means a commit, a merge, and a deploy, and every old value stays in git history forever. Once a secret has been in a repo, rotating it is the only real remediation — deleting the commit is not, since clones and forks keep the object. Treat SOPS as configuration-that-happens-to-be-sensitive, not as a credential vault.

FAQ

How should a small startup manage secrets across dev, staging, and production?
One store, three scopes, no files on laptops. Use your cloud provider's secret store if you're single-cloud, or Doppler/1Password if you aren't, and inject values into the process at launch instead of writing .env files to disk. Validate every required variable at boot so a missing secret crashes instead of falling back.

Is it safe to commit a .env file if it's encrypted?
Encrypted with SOPS or git-crypt, yes, with two caveats: the decryption key must live outside the repo, and any value that has ever been committed in plaintext must be rotated, because git history and existing clones keep it permanently.

Do I need HashiCorp Vault for a five-person team?
Almost certainly not. Vault's payoff is dynamic, short-lived credentials and fine-grained policy; below that bar, a managed secret store plus OIDC in CI gives you most of the security benefit with none of the unseal-and-HA operational burden.

Bottom line

Single-cloud teams should use their provider's secret store with OIDC in CI and stop there — it's the lowest-cost, lowest-drama option, and it removes the long-lived keys that actually get exploited. Teams deploying across several platforms, or handing credentials to people who don't touch infrastructure, get real time back from Doppler or 1Password's CLI injection. Reach for SOPS when sensitive config genuinely needs to be reviewed in pull requests, and for Vault or OpenBao only when you specifically want credentials that expire on their own. Whatever you choose, the boot-time validation and the CI key removal matter more than the vendor.

Your Postgres Migration Runner Needs a Retry Contract, Not Just a Lock Timeout

Libme — Mon, 17 Aug 2026 15:23:11 +0000

Setting lock_timeout on a migration keeps a blocked ALTER TABLE from freezing your traffic, but on its own it just converts one outage into a flaky deploy. The complete version is five rules: exactly one migrator at a time (advisory lock), a short lock timeout scoped to the transaction, the blocker logged before every retry, jittered backoff so replicas don't wake in lockstep, and a wall-clock budget after which the deploy fails instead of retrying forever. A reader raised this on an earlier post about migration locks, and it's the part most runners get wrong.

Everything here holds for Postgres 12 and up, as of mid-2026.

Why do several replicas try to run the same migration at once?

Most deploy systems run migrations as a pre-start step in the application container. Roll out four replicas and you have four processes racing to apply the same DDL. Usually the migration table's own row lock hides this, and you never notice.

You notice when the DDL is blocked. All four sit in the lock queue behind the same idle-in-transaction session. Then the blocker commits, all four wake, and whichever loses the race hits a duplicate-object error or, worse, a partially applied migration:

ERROR:  column "shipped_at" of relation "orders" already exists

Now the deploy is red for a reason that has nothing to do with the real problem. Worse, if you added deterministic exponential backoff — sleep 1s, 2s, 4s — every replica computes the same delays, so they retry in unison and keep colliding on the same schedule. Backoff without jitter is a synchronized herd.

Takeaway: a blocked migration turns a harmless replica race into a deploy failure, so serialization has to happen before the retry logic, not after.

How do I make sure only one migrator runs at a time?

Postgres advisory locks. They're application-defined locks on an arbitrary bigint key, scoped to the database, with no table involved:

-- session-level: held until unlocked or the connection closes
SELECT pg_try_advisory_lock(8675309);   -- true = you are the migrator
-- ...run migrations...
SELECT pg_advisory_unlock(8675309);

Use pg_try_advisory_lock (returns immediately) rather than pg_advisory_lock (waits forever) so a losing replica can decide what to do rather than hanging. Two reasonable choices: exit 0 and let the winner apply the schema, or wait a bounded time if the replica cannot safely serve traffic against the old schema. Pick one deliberately — silently exiting 0 is a real footgun when the new code needs the new column.

One deployment detail decides which lock scope you can use. If your connection goes through PgBouncer in transaction pooling mode, session-level advisory locks are unsafe, because the connection returns to the pool between transactions while the lock is still attached to it. There, use pg_advisory_xact_lock(8675309) inside the migration transaction, which Postgres releases automatically at commit or rollback.

Some runners already do this for you, and it's worth checking before you build your own:

Runner	Cross-process locking (as of mid-2026)	Watch out for
Flyway	Yes, database-level lock on Postgres	Verify behavior when running through a transaction pooler
Liquibase	Yes, via a `DATABASECHANGELOGLOCK` row	A killed run can leave the lock row set; needs manual release
Rails Active Record	Yes, advisory lock by default	Can be disabled in config; check it wasn't turned off
golang-migrate	Yes, advisory lock on the Postgres driver	Lock is per-database, so shared databases share the lock
Alembic	No built-in lock	Serialization is your job — wrap the runner yourself

If you want serialization without writing a runner, Flyway is the one that takes a database lock for you and fails the second process cleanly instead of letting it race.

Takeaway: check whether your migration tool already serializes runs before adding your own advisory lock — two locking schemes are not safer than one.

Where should lock_timeout actually be set?

Inside the transaction, with SET LOCAL, so it evaporates at commit:

BEGIN;
SET LOCAL lock_timeout = '3s';
SET LOCAL application_name = 'migrator:add_shipped_at';
ALTER TABLE orders ADD COLUMN shipped_at timestamptz;
COMMIT;

The reason to prefer SET LOCAL over a plain SET is pooling: a plain SET sticks to the backend, and a pooled connection can hand that 3-second timeout to unrelated application queries later. The distinctive application_name costs nothing and makes the blocker/waiter pair obvious in pg_stat_activity at 2am.

Two exceptions worth knowing. CREATE INDEX CONCURRENTLY cannot run inside a transaction block, so SET LOCAL doesn't apply — issue a session-level SET lock_timeout on that connection instead. And don't wrap CONCURRENTLY work in a tight statement_timeout: if the timeout kills it, you're left with an INVALID index that you must drop before retrying.

SELECT indexrelid::regclass FROM pg_index WHERE NOT indisvalid;

Takeaway: SET LOCAL lock_timeout bounds lock acquisition for this transaction only, which is exactly the scope you want on a pooled connection.

How should the retry loop back off, and what should it log first?

Full jitter — sleep a uniform random amount between zero and the current exponential cap — is what breaks the synchronized herd. And before each sleep, capture who blocked you, because a retry that succeeds tells you nothing about the query that will block tomorrow's migration too.

There's a catch people get wrong: you cannot call pg_blocking_pids(pg_backend_pid()) from the connection that is currently blocked — it's busy waiting. Sampling it live requires a second observer connection. The cheap approximation is to probe right after the lock_timeout fires, from the same connection, for the oldest transactions holding locks on that table:

SELECT a.pid, a.application_name, a.state,
       now() - a.xact_start AS xact_age,
       left(a.query, 80) AS query
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.pid
WHERE l.relation = 'public.orders'::regclass
  AND a.pid <> pg_backend_pid()
ORDER BY a.xact_start
LIMIT 5;

Occasionally the blocker has already finished and you log nothing. That's an acceptable trade for not running an observer thread. Here is the whole contract in one runner, using psycopg 3:

import random, time
import psycopg

LOCK_KEY = 8675309       # same constant in every replica
BUDGET_SECONDS = 300     # hard stop for the whole migration step
BASE, CAP = 1.0, 20.0

def run_migration(dsn, ddl, table):
    conn = psycopg.connect(dsn, autocommit=True,
                           application_name="migrator")
    with conn.cursor() as cur:
        cur.execute("SELECT pg_try_advisory_lock(%s)", (LOCK_KEY,))
        if not cur.fetchone()[0]:
            print("another migrator holds the lock; nothing to do")
            return 0

    deadline, attempt = time.monotonic() + BUDGET_SECONDS, 0
    try:
        while True:
            attempt += 1
            try:
                with conn.transaction(), conn.cursor() as cur:
                    cur.execute("SET LOCAL lock_timeout = '3s'")
                    cur.execute(ddl)
                print(f"applied on attempt {attempt}")
                return 0
            except psycopg.errors.LockNotAvailable:
                log_blockers(conn, table)
                delay = random.uniform(0, min(CAP, BASE * 2 ** (attempt - 1)))
                if time.monotonic() + delay > deadline:
                    print("lock budget exhausted; failing the deploy")
                    return 1
                print(f"blocked; retry {attempt} in {delay:.1f}s")
                time.sleep(delay)
    finally:
        with conn.cursor() as cur:
            cur.execute("SELECT pg_advisory_unlock(%s)", (LOCK_KEY,))
        conn.close()

Note that only LockNotAvailable (SQLSTATE 55P03) is retried. A syntax error or a constraint violation is not a transient condition, and retrying it just burns the budget.

Takeaway: retry only the lock-timeout error, sleep a random interval rather than a computed one, and log the blocker before every sleep.

When should the deploy just fail?

When the budget runs out. A migration that has been blocked for five minutes is not waiting on a slow query; it's waiting on something structural — an idle-in-transaction connection from a pool with no idle_in_transaction_session_timeout, a BI tool holding a long read, a pg_dump that overlaps your deploy window. Retrying past that point hides the diagnosis you actually need.

Symptom in the retry log	Likely cause	Fix, not a retry
Blocker `state = idle in transaction`, age growing	App or pool leaking an open transaction	Set `idle_in_transaction_session_timeout`
Blocker is a long analytics `SELECT`	Reporting traffic on the primary	Move reads to a replica, or migrate off-peak
Blocker is another `migrator:*`	Advisory lock missing or disabled	Fix serialization first
No blocker found, still timing out	Blocker is short but constant	Raise `lock_timeout` slightly, or use a quieter window

Takeaway: the budget's job is to convert an invisible hang into a red deploy with the blocker's identity attached.

FAQ

Why does my Postgres migration fail with "canceling statement due to lock timeout"?
Because lock_timeout is doing its job: your DDL waited longer than that limit for a lock another session held. That error means your app stayed up. Look at what held the lock — usually an idle-in-transaction connection or a long-running read — rather than raising the timeout.

Can two application replicas run database migrations at the same time?
Yes, unless something stops them. Serialize the runner with pg_try_advisory_lock on a fixed key, or confirm your migration tool takes its own lock. Behind PgBouncer in transaction pooling mode, use pg_advisory_xact_lock instead, since session-level locks outlive the transaction that took them.

Should migration retries use exponential backoff?
Use exponential backoff with full jitter — a random sleep between zero and the current cap. Deterministic delays make every replica wake at the same instant and contend again. Cap total retry time with a wall-clock budget and fail the deploy when it's exhausted.

Bottom line

If your migrations already use lock_timeout and a non-blocking two-step for rewrites, the missing piece is the runner around them. Add one migrator via advisory lock (or verify your tool has one), scope the timeout with SET LOCAL, tag the session with a distinctive application_name, log blockers before each jittered retry, and stop hard at a fixed budget. Teams on Flyway, Liquibase, Rails, or golang-migrate mostly need to verify the lock is on and add the budget; teams on Alembic or a hand-rolled script need the whole contract. The point isn't to make blocked migrations succeed — it's to make them fail fast, once, with the blocker named.

Your Postgres Backups Are Untested Until You Restore One: A Drill for Small Teams

Libme — Mon, 17 Aug 2026 15:20:20 +0000

A backup job that exits 0 tells you a file was written. It does not tell you the file can become a running database with your schema, your extensions, and your roles intact. The only way to know is to restore it on a schedule and time yourself, and the first restore you ever attempt should not be during an outage.

This is the drill I run — roughly monthly, and always after any change to the schema, the extension list, or the Postgres major version. It takes about twenty minutes once it's scripted, and every single time I've introduced it somewhere, the first run failed.

Why does a backup that "succeeded" fail to restore?

Because pg_dump captures the contents of one database, not the environment it lived in. The three things it leaves behind are the three things that break your restore:

Roles are cluster-level, not database-level. pg_dump does not include CREATE ROLE. Restore into a fresh cluster and you get:

pg_restore: error: could not execute query: ERROR:  role "app_user" does not exist
Command was: ALTER TABLE public.orders OWNER TO app_user;

Extensions must exist as installed binaries on the target machine. The dump contains CREATE EXTENSION vector;, but that only works if the target already has the pgvector shared library on disk:

ERROR:  could not open extension control file "/usr/share/postgresql/16/extension/vector.control": No such file or directory

This is the one that bites people who dump from a managed provider with extensions preinstalled and restore into a stock postgres:16 container.

Client and server versions have to line up. pg_dump refuses to dump from a server newer than itself:

pg_dump: error: server version: 16.3; pg_dump version: 15.6
pg_dump: error: aborting because of server version mismatch

The failure mode that actually hurts is silent: your cron box quietly ships an older pg_dump than your upgraded server, the job starts failing, and nobody reads the log because nothing pages on a backup job. Alert on backup failure and staleness, not just on failure.

A backup job's exit code proves a file exists; only a restore proves the file is a database.

What does the drill actually look like?

Five steps, all scriptable, all runnable on a laptop.

Pull the most recent backup artifact — from object storage, not from a copy sitting on the same host as production.
Start a throwaway Postgres of the same major version as production.
Restore into it with errors treated as fatal.
Run assertions: row counts on your three or four most important tables, plus a query that exercises an extension.
Record the wall-clock time and tear it down.

Here's the core of it. Taking the dump:

#!/usr/bin/env bash
set -euo pipefail

pg_dump \
  --format=custom \
  --no-owner \
  --no-privileges \
  --file="backup_$(date -u +%Y%m%dT%H%M%SZ).dump" \
  "$DATABASE_URL"

--no-owner and --no-privileges are what make the dump portable: ownership statements get dropped, so the restore doesn't demand that app_user exists on the target. You then re-apply grants from your migration tooling, where they belong. Keep roles in version control as SQL; do not rely on them surviving in a dump.

The drill itself:

#!/usr/bin/env bash
set -euo pipefail

DUMP="$1"
PGVER="${PGVER:-16}"
CONTAINER="restore-drill-$$"

start=$(date +%s)

docker run -d --name "$CONTAINER" \
  -e POSTGRES_PASSWORD=drill \
  -p 55432:5432 \
  "pgvector/pgvector:pg${PGVER}"

until docker exec "$CONTAINER" pg_isready -U postgres -q; do sleep 1; done

export PGPASSWORD=drill
CONN="postgresql://postgres@localhost:55432/postgres"

pg_restore --dbname="$CONN" --exit-on-error --jobs=4 "$DUMP"

psql "$CONN" -v ON_ERROR_STOP=1 -f drill_assertions.sql

echo "restore completed in $(( $(date +%s) - start ))s"
docker rm -f "$CONTAINER" >/dev/null

Two flags carry most of the weight. --exit-on-error is the important one: by default pg_restore prints errors, keeps going, and exits 0, which means an unattended restore check without it will happily report success on a half-populated database. --jobs=4 parallelizes table data and index builds, and it only works with the custom or directory formats — another reason to stop using plain SQL dumps for anything large.

The assertions file is deliberately boring, and it should fail loudly:

\set ON_ERROR_STOP on

-- structural: does the extension work, not just exist?
SELECT '[1,2,3]'::vector <-> '[3,2,1]'::vector AS distance_check;

-- volumetric: catch a restore that "worked" but landed empty
DO $$
DECLARE n bigint;
BEGIN
  SELECT count(*) INTO n FROM orders;
  IF n < 1000 THEN
    RAISE EXCEPTION 'orders table has only % rows — restore is suspect', n;
  END IF;
END $$;

-- freshness: how much data would we actually have lost?
SELECT now() - max(created_at) AS data_age FROM orders;

That last query is the one worth reading out loud in a team channel. It converts an abstract retention policy into a number: if we restored right now, we would be missing this much. That number is your real RPO, and it is usually worse than whatever the backup docs implied.

Assert on row counts and data age, not on the restore's exit code — an empty database restores perfectly.

Nightly dumps or point-in-time recovery?

pg_dump gives you a consistent snapshot and nothing between snapshots. If it runs at 03:00 and you lose the primary at 17:00, you have lost fourteen hours. Point-in-time recovery closes that gap by shipping the write-ahead log continuously, so you can replay to a chosen moment — including "one second before that DELETE without a WHERE."

Approach	Typical RPO	Restore complexity	Where it fits
`pg_dump` custom format to object storage	Since last dump (hours)	Low — one command	Side projects, small internal apps, portable migrations
Managed provider snapshots + PITR	Seconds to minutes	Low, but provider-shaped	Anything on RDS, Cloud SQL, Supabase, Neon, Crunchy Bridge
pgBackRest or WAL-G to your own bucket	Seconds to minutes	Medium — real config, real ops	Self-hosted Postgres you intend to keep
Physical replica only	Near zero for hardware loss	N/A	Not a backup — replicates your `DROP TABLE` faithfully

That last row is the trap I see most often. A standby protects you from a dead machine; it does not protect you from a bad migration, because the destructive statement replicates in milliseconds. Keep both.

If you self-host and want continuous archiving without writing your own WAL shipping, pgBackRest is the option that handles full/differential/incremental backups, parallel compression, and retention expiry with one config file and one command — at the cost of a genuinely non-trivial setup pass and a config format you will have to read the docs for every time you touch it. WAL-G is the leaner alternative when you want to push straight to S3-compatible storage with minimal moving parts, though you'll find fewer worked examples when something goes wrong. On the managed side, Neon's branching turns a restore drill into creating a branch from a past timestamp and pointing a test connection string at it, which is the lowest-friction version of this whole workflow — the constraint being that it's provider-specific, so you should still keep an independent logical dump if you ever want to leave.

Whatever you pick, store backups in an account or bucket that your application's credentials cannot delete. Ransomware and a bad terraform destroy fail the same way.

A replica is not a backup, and a backup you cannot restore without the original provider is a hostage.

How often should you run the drill?

Monthly is a reasonable floor for a small team. Run it additionally after: a Postgres major version upgrade, adding or removing an extension, changing the backup tool or its flags, and any change to who owns the storage bucket. Wire it into CI on a schedule if you can — a weekly GitHub Actions job that restores yesterday's dump into a service container and runs the assertions gives you a red build instead of a discovery at 2 a.m.

Track exactly two numbers over time: minutes to a usable database (your RTO) and the data_age from the assertions (your RPO). If either number surprises someone on the team, you've found the actual gap. As of mid-2026 I've never seen a first drill where both numbers matched what people assumed.

FAQ

How do I test a Postgres backup without touching production?
Restore the dump into a disposable container on a non-production port, run assertions against it, then destroy the container. Nothing in the drill connects to production except the read that fetched the backup file from object storage.

Why does pg_restore exit 0 even though it printed errors?
Because pg_restore treats most errors as non-fatal by default so it can restore as much as possible. Pass --exit-on-error (and --single-transaction if you want all-or-nothing) whenever a script is checking the result.

Is a read replica enough of a backup for a small app?
No. A replica protects against losing a machine but faithfully replicates destructive SQL such as a bad DELETE or migration. You need point-in-time recovery or periodic dumps to recover from a mistake you made yourself.

Bottom line

If you run a side project or a small internal app, nightly pg_dump --format=custom --no-owner to object storage plus a monthly scripted restore drill is enough, and it's an afternoon of work. Once real customer data is involved, move to continuous archiving — your provider's PITR if you're managed, pgBackRest or WAL-G if you're self-hosted — and keep the logical dump as your escape hatch from the provider. Never count a physical replica as a backup. And measure the drill: the restore you have never timed is the one that takes three hours on the worst possible day.

Why Your Docker Build Takes 11 Minutes in CI When It Takes 20 Seconds Locally

Libme — Sun, 16 Aug 2026 15:04:48 +0000

If your Docker build is fast locally and slow in CI, the base image is almost never the problem. CI runners are ephemeral, so they start with an empty layer cache unless you explicitly wire one up, and a COPY . . placed above your dependency install throws away whatever cache you did manage to restore. Fix the layer ordering first, then attach a cache backend, and measure with --progress=plain so you can see which steps actually say CACHED.

This is the version of the problem I keep running into on small teams: the build was fine when it lived on one laptop, then it moved to a hosted runner and quietly became the longest step in the pipeline.

Why is the Docker cache empty on every CI run?

Locally, your daemon keeps every intermediate layer on disk between builds. A hosted runner is a fresh VM. When the job starts, docker build has nothing to compare against, so every RUN re-executes from scratch — including the three-minute npm ci or pip install you never think about.

You can confirm it in one run. Add plain progress output and read the log:

docker build --progress=plain -t myapp:ci . 2>&1 | grep -E 'CACHED|DONE'

On a warm local build you'll see a wall of CACHED lines. On a cold runner you'll see almost none. That difference — not the image size, not the base distro — is your eleven minutes.

The related trap is that people "fix" this by switching from node:22 to node:22-alpine and are confused when the build time barely moves. Image size affects push and pull time; cache hits affect build time. They're different bills.

Takeaway: a slow CI build is a cache-miss problem until you have proven otherwise with --progress=plain.

How should a Dockerfile be ordered so the cache actually holds?

BuildKit invalidates a layer when its inputs change, and every layer after it. So the rule is: things that change rarely go up top, things that change on every commit go at the bottom. In practice that means copying your manifest and lockfile alone, installing, and only then copying source.

Here's a Node example with the two common mistakes removed:

# syntax=docker/dockerfile:1
FROM node:22-slim AS deps
WORKDIR /app

# Only the lockfile inputs — this layer survives ordinary code changes.
COPY package.json package-lock.json ./
RUN npm ci --omit=dev

FROM node:22-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
# Source last: a code-only commit invalidates nothing above this line.
COPY . .
USER node
CMD ["node", "server.js"]

The Python shape is identical — COPY requirements.txt . and RUN pip install -r requirements.txt before COPY . .. If you use Poetry or uv, copy the manifest and the lockfile together, because installing from a manifest without its lock defeats the point of a reproducible layer.

Two details that bite people:

A .dockerignore that misses .git or node_modules means your COPY . . context changes on every build for reasons unrelated to your code. Check it before blaming BuildKit.
Anything that writes a timestamp or a build ID into an early layer will invalidate everything below it, forever. Push those to the last stage.

Takeaway: if COPY . . appears above your dependency install, no cache backend on earth will save that build.

How do you make the cache survive between CI runs?

Ordering only pays off if there's something to restore from. BuildKit can export its cache to an external backend and import it on the next run. On GitHub Actions the least-effort option is the built-in cache backend:

name: build
on: [push]

jobs:
  image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          tags: myapp:ci
          cache-from: type=gha
          cache-to: type=gha,mode=max

mode=max exports intermediate stage layers too, not just the final image — which is what you want when you have a multi-stage build, since the expensive npm ci lives in a stage that never ships. The tradeoff is a larger cache, and GitHub's Actions cache is capped per repository (10 GB as of mid-2026) with least-recently-used eviction, so a mode=max cache on a busy monorepo can push your other caches out.

The other option is a registry cache, which works on any CI provider and has no 10 GB ceiling:

          cache-from: type=registry,ref=ghcr.io/you/myapp:buildcache
          cache-to: type=registry,ref=ghcr.io/you/myapp:buildcache,mode=max

If you want this without maintaining runner infrastructure, Depot is the managed builder that keeps a persistent BuildKit cache volume across runs so you skip the export/import round trip entirely.

One version note: the type=gha backend was rewritten to use GitHub's newer cache service, and older Buildx releases talked to an API that GitHub has since retired. If you pinned setup-buildx-action or a Buildx version a couple of years ago and your cache silently stopped working, that's the first thing to check.

Takeaway: cache-from/cache-to is the line that turns a correct Dockerfile into a fast pipeline; without it the ordering work is invisible.

What about `RUN --mount=type=cache`?

Cache mounts are the other half, and they're the part that surprises people. They give a RUN step a persistent directory for a package manager's own cache:

RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

This is genuinely good on a long-lived builder: even when the lockfile changes and the layer must rebuild, the package manager re-downloads almost nothing.

The catch: cache mount contents are builder-local state. They are not part of the exported layer cache, so cache-to: type=gha does not carry them to the next ephemeral runner. On hosted CI they mostly help within a single build, or when you run a self-hosted/persistent builder. Treat them as a bonus on top of correct layer ordering, not a substitute for it.

Takeaway: cache mounts speed up rebuilds on a builder that sticks around; layer cache export is what helps a fresh runner.

Which fix applies to which symptom?

Symptom	Likely cause	Fix
Every step rebuilds, no `CACHED` lines	No cache backend on an ephemeral runner	`cache-from`/`cache-to` (gha or registry)
Dependency install reruns on code-only commits	`COPY . .` above the install	Copy manifest + lockfile first
Cache hits locally, misses in CI on the same commit	Build context differs	Fix `.dockerignore`, check generated files
Build is fast, deploy is slow	Image size, not cache	Multi-stage, slim base, drop build tools
Cache worked, then stopped after months	Cache evicted or backend API changed	Check size limits; update Buildx/actions

Takeaway: match the fix to the symptom you measured, because "slow build" and "slow deploy" have almost no overlap in causes.

FAQ

Why is my Docker build slow in GitHub Actions but fast locally?
Because the hosted runner is a fresh machine with no layer cache. Local builds reuse layers your daemon kept on disk; CI has nothing to reuse until you configure cache-from/cache-to with a backend like type=gha or type=registry.

Does docker build --no-cache explain the difference?
Only if it's actually in your workflow — check for it, since people add it to debug a stale build and forget to remove it. Otherwise the empty cache is structural to the runner, not a flag.

Will a smaller base image make my build faster?
Rarely. A slim or Alpine base makes the image faster to push and pull, but build time is dominated by cache misses on install steps. Fix ordering and cache export first, then optimize size for deploy speed.

Bottom line

Start by reading --progress=plain output on a CI run and counting CACHED lines — that tells you whether you have a cache problem or a size problem. If you have a cache problem, reorder the Dockerfile so lockfiles land above source, then add a cache backend: type=gha if you live on GitHub Actions and your cache fits under the repo limit, type=registry if you're on another provider or you keep getting evicted. Add cache mounts only after those two are in place, and only expect them to pay off on a persistent builder. Most of the eleven minutes goes away at step two.

Getting 429 Too Many Requests? Fix the Client Before You Ask for a Higher Quota

Libme — Sun, 16 Aug 2026 00:25:01 +0000

If your integration is throwing 429 Too Many Requests, adding a retry loop usually makes it worse: every retry is another request against the same quota, and a fleet of workers all retrying at the same interval turns a small overage into a synchronized stampede. The fix is to stop sending too fast in the first place — a token bucket in front of the client — and to treat the server's Retry-After header as the authority whenever you do get throttled. Asking the vendor for a quota increase should be the last step, not the first.

This is the thing I get wrong most often when I move fast: I wire up an API, hit the limit under real traffic, wrap the call in try/except with a sleep(1), and ship it. It works in staging with one worker and falls apart the moment two processes run concurrently.

Why does adding retries make 429s worse?

Three failure modes, in the order I usually hit them.

Retries spend quota. A naive retry loop on a rate-limited endpoint sends attempt after attempt into a bucket that is already empty. If the limiter counts rejected requests (many do — the rejection still costs the gateway work), your retries actively delay recovery.

Fixed sleeps synchronize. Every worker that hits the limit at 12:00:00 and sleeps exactly one second wakes at 12:00:01 together. You've built a metronome. This is the classic thundering herd, and it's why jitter isn't optional decoration.

Retrying non-idempotent calls duplicates work. A POST /charges that returns 429 might have been rejected before any work happened — or it might have been rejected by a downstream limiter after the charge was created. Without an idempotency key, a retry is a coin flip on double-charging someone.

The takeaway: retries are the recovery path, not the rate control; if retries are your only rate control, you don't have any.

What should the client actually do with `Retry-After`?

Read it, and parse both forms. Retry-After is allowed to be either a delay in seconds or an HTTP-date, and I have been bitten by a client that assumed seconds, got a date string, and threw a ValueError inside its own error handler.

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def retry_after_seconds(header: str | None, fallback: float) -> float:
    """Parse Retry-After (delay-seconds OR HTTP-date). Falls back on garbage."""
    if not header:
        return fallback
    header = header.strip()
    try:
        return max(0.0, float(header))
    except ValueError:
        pass
    try:
        when = parsedate_to_datetime(header)
    except (TypeError, ValueError):
        return fallback
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

Clamp the result before you use it. A server that is having a bad day can return Retry-After: 3600, and a worker that blindly sleeps for an hour looks exactly like a hung process to whatever is watching it. I cap at something like 60 seconds and let the job fail into a retry queue beyond that.

Beyond Retry-After, what you get is vendor-specific. Many APIs expose X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset; the IETF has been working on standardizing RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset, but as of mid-2026 that is still a draft, so read the docs for the specific API rather than assuming the header names. When Remaining is exposed, feeding it back into your limiter is the single cheapest improvement you can make — you stop guessing at the budget and start reading it.

The takeaway: treat Retry-After as authoritative but bounded, and treat every other rate-limit header as vendor-specific until you've read the docs.

How do I limit the client side without over-engineering it?

A token bucket is about twenty lines and covers most cases. It allows a burst up to the bucket capacity, then settles into a steady rate — which is what most APIs actually enforce.

import asyncio
import time


class TokenBucket:
    def __init__(self, rate_per_sec: float, burst: int):
        self.rate = rate_per_sec
        self.capacity = float(burst)
        self.tokens = float(burst)
        self.updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self, tokens: float = 1.0) -> None:
        while True:
            async with self._lock:
                now = time.monotonic()
                self.tokens = min(
                    self.capacity, self.tokens + (now - self.updated) * self.rate
                )
                self.updated = now
                if self.tokens >= tokens:
                    self.tokens -= tokens
                    return
                wait = (tokens - self.tokens) / self.rate
            await asyncio.sleep(wait)

Use time.monotonic(), not time.time(). A clock adjustment (NTP step, container migration) can move wall-clock time backwards, and a limiter that computes a negative elapsed interval will either stall or hand out free tokens.

One honest limitation: this implementation is not FIFO. Waiters wake up and re-race for the lock, so under heavy contention a request can wait longer than others that arrived after it. For a background worker that's fine. If you need fairness guarantees for user-facing latency, hand out ordered tickets instead of re-looping.

Wiring it together with retries, jitter, and a bounded sleep:

import random
import httpx

RETRYABLE = {429, 500, 502, 503, 504}


async def request_with_retry(
    client: httpx.AsyncClient,
    bucket: TokenBucket,
    method: str,
    url: str,
    *,
    max_attempts: int = 5,
    **kwargs,
) -> httpx.Response:
    for attempt in range(1, max_attempts + 1):
        await bucket.acquire()
        resp = await client.request(method, url, **kwargs)
        if resp.status_code not in RETRYABLE or attempt == max_attempts:
            return resp
        backoff = min(60.0, 0.5 * 2 ** (attempt - 1))
        delay = min(60.0, retry_after_seconds(resp.headers.get("Retry-After"), backoff))
        await asyncio.sleep(delay + random.uniform(0.0, 0.3 * delay + 0.1))
    raise AssertionError("unreachable")

Only retry POST/PATCH through this path if the request carries an idempotency key the server honors. Otherwise restrict retries to GET/PUT/DELETE and let write failures surface.

The takeaway: a token bucket plus exponential backoff with jitter, bounded by a hard ceiling, handles the overwhelming majority of third-party API throttling.

Which limiting strategy fits which problem?

Strategy	What it controls	Burst behavior	Needs shared state	Use it when
Fixed window counter	Requests per calendar window	Allows 2× at window boundaries	Yes (if multi-process)	Simplest server-side enforcement
Sliding window log	Requests over a rolling window	Accurate, no boundary spike	Yes, and it's memory-hungry	You must match a strict rolling quota
Token bucket	Sustained rate + burst size	Burst up to capacity, then steady	Only for multi-process	Client-side pacing of a third-party API
Concurrency semaphore	In-flight requests	Unbounded rate, bounded parallelism	Per-process usually fine	The limit is really about connections or memory

The distinction people miss: a concurrency limit and a rate limit are different constraints. Ten concurrent requests that each take 50ms is roughly 200 requests/second; ten concurrent requests that each take 5 seconds is 2 requests/second. If the API's limit is expressed in requests per second, a semaphore will not save you.

The takeaway: pick the strategy that matches how the limit is expressed, not the one that's easiest to implement.

What changes when you run more than one worker?

An in-process bucket controls one process. Run four replicas and you send four times your intended rate. Once you're multi-process, the counter has to live somewhere shared, and the usual answer is Redis with a small Lua script so the check-and-increment is atomic:

-- KEYS[1] = bucket key, ARGV[1] = limit, ARGV[2] = window in ms
local count = redis.call('INCR', KEYS[1])
if count == 1 then
  redis.call('PEXPIRE', KEYS[1], ARGV[2])
end
if count > tonumber(ARGV[1]) then
  return {0, redis.call('PTTL', KEYS[1])}
end
return {1, 0}

That's a fixed window, so it permits up to 2× the limit across a window boundary — acceptable when your client-side target is set below the real quota, not when you're pacing right at the ceiling. If you need a shared limiter from serverless functions where holding a TCP connection is awkward, Upstash Redis is the one that speaks HTTP and includes a rate-limiting helper, at the cost of per-request latency you wouldn't pay with a co-located Redis. If the traffic you need to shape is inbound rather than outbound, push the limit to the edge instead: Kong and Envoy both enforce rate limits at the proxy layer, which keeps the logic out of every service that sits behind them.

The takeaway: the moment you scale past one process, your rate limiter needs shared state or it silently stops being a limit.

FAQ

What does HTTP 429 Too Many Requests mean?
It means the server accepted your request as well-formed and authenticated, but you exceeded a rate limit, so it refused to process it. It is a client-side pacing problem, not an error in the request itself, and the response often carries a Retry-After header telling you how long to wait.

Should I automatically retry after a 429?
Yes, but only with backoff plus jitter, only up to a small number of attempts, and only for requests that are safe to repeat. Retrying a non-idempotent write without an idempotency key risks duplicating the operation.

Why am I still getting 429s after adding a rate limiter?
Almost always because the limiter is per-process and you're running multiple workers, or because the limit is enforced per-endpoint or per-resource rather than globally. Check whether your effective rate is your configured rate multiplied by your replica count.

Bottom line

If you're seeing 429s, put a token bucket in front of the client and set it a comfortable margin below the documented quota before you touch anything else. Honor Retry-After when it's present, clamp it so a bad value can't hang a worker, and add jitter so your fleet doesn't retry in lockstep. Move the counter into Redis the moment you run more than one replica, and push enforcement to a proxy like Kong or Envoy if you're limiting inbound traffic across many services. Ask for a quota increase only after your own numbers show you're pacing correctly and still hitting the ceiling.

Your Password Reset Emails Are Going to Spam: Choosing Between Resend, Postmark, and Amazon SES

Libme — Sat, 15 Aug 2026 23:08:50 +0000

If your transactional email is landing in spam, the provider is almost never the first thing to fix — misaligned SPF/DKIM/DMARC records and a shared sending domain are. Once DNS is correct, the choice comes down to what you want to operate: Amazon SES is the cheapest per message and the most work, Postmark buys you strict stream separation and deliverability support, and Resend buys you the fastest path from npm install to a sent email. Below is the failure mode, the DNS fix, and where each provider actually breaks down.

Why does "delivered" in the dashboard still mean spam folder?

This is the part that wastes an afternoon. Every provider's dashboard reports a message as Delivered when the receiving mail server returned 250 OK at the end of the SMTP transaction. That acknowledgement means Gmail accepted the message for processing. It says nothing about which folder Gmail then put it in — inbox, Promotions, or Spam. No sending API can see that, because the receiving side never tells you.

So the symptom looks like this: your logs are clean, the provider shows 100% delivery, and users keep saying the reset link never arrived. The next dead end most people try is rewriting the email copy — removing the word "free", shortening the subject line, stripping images. That occasionally helps at the margin, but it is downstream of the real problem, which is that the receiving domain cannot verify you are who your From header claims.

A 250 OK is proof of acceptance, not proof of inbox placement — treat provider "delivered" counts as a floor, not a result.

What DNS records do you actually need before blaming the provider?

Three records, and the one that trips people up is alignment rather than existence. SPF and DKIM can both pass while DMARC still fails, because DMARC requires the passing mechanism's domain to align with the domain in the visible From header. If you send as noreply@example.com but your SPF-authorized envelope sender is your provider's bounce domain and DKIM signs with a subdomain the provider owns, alignment fails and DMARC fails with it.

Check what you have published before changing anything:

# SPF — must exist exactly once on the sending domain
dig +short TXT mail.example.com | grep spf1

# DKIM — key selector varies per provider; check the one they gave you
dig +short TXT resend._domainkey.mail.example.com

# DMARC — always at _dmarc.<domain>
dig +short TXT _dmarc.example.com

A workable starting DMARC record is v=DMARC1; p=none; rua=mailto:dmarc@example.com — report-only, so you can read aggregate reports for a week and confirm alignment before moving to p=quarantine. Publishing p=reject first is how people silently kill their own mail.

Two more things that are non-negotiable as of mid-2026: since Google and Yahoo tightened their bulk sender requirements in February 2024, senders at volume need SPF, DKIM, and DMARC, plus one-click unsubscribe on marketing mail and a spam complaint rate held well below their published threshold. And transactional mail should leave from a dedicated subdomain (mail.example.com), so a bad marketing campaign cannot poison the reputation that carries your password resets.

Fix alignment and subdomain separation first; switching providers to solve a DNS problem just moves the problem.

How do Resend, Postmark, and SES actually differ?

	Amazon SES	Postmark	Resend
Pricing model	Per thousand messages, cheapest at volume; data transfer and attachment size billed separately	Per message, tiered by monthly volume; premium relative to SES	Per message with a free developer tier, tiered plans above it
Setup friction	Highest — sandbox by default, production access is a request form	Low — domain verification, then send	Lowest — API key and a verified domain
Bounce/complaint handling	You wire SNS or EventBridge to a configuration set and process events yourself	Built-in suppression plus a searchable activity view	Webhook events for delivery, bounce, and complaint
Stream separation	Configuration sets, self-managed	First-class: transactional and broadcast streams are enforced	Supported via separate domains/audiences, less opinionated
Deliverability support	AWS Support ticket, general-purpose	Deliverability-specialist support is the product	Younger track record; support scales with plan
Best when	You have volume and infra people	Reset/receipt mail must not fail	You want to ship this afternoon

The three are genuinely different products that happen to share an API shape:

// Amazon SES v2 — cheap, explicit, and you own the event plumbing
import { SESv2Client, SendEmailCommand } from "@aws-sdk/client-sesv2";

const ses = new SESv2Client({ region: "us-east-1" });

await ses.send(new SendEmailCommand({
  FromEmailAddress: "Acme <noreply@mail.example.com>",
  Destination: { ToAddresses: [user.email] },
  ConfigurationSetName: "transactional", // routes bounce/complaint events
  Content: {
    Simple: {
      Subject: { Data: "Reset your password" },
      Body: { Text: { Data: resetBody } },
    },
  },
}));

// Postmark — message streams are a required argument, not an afterthought
import { ServerClient } from "postmark";

const postmark = new ServerClient(process.env.POSTMARK_TOKEN);

await postmark.sendEmail({
  From: "Acme <noreply@mail.example.com>",
  To: user.email,
  Subject: "Reset your password",
  TextBody: resetBody,
  MessageStream: "outbound", // 'broadcast' mail must use a different stream
});

// Resend — the least ceremony between you and a sent message
import { Resend } from "resend";

const resend = new Resend(process.env.RESEND_API_KEY);

await resend.emails.send({
  from: "Acme <noreply@mail.example.com>",
  to: [user.email],
  subject: "Reset your password",
  text: resetBody,
});

The API call is the easy part in all three; what differs is who operates bounce handling, reputation, and stream hygiene — you, or the vendor.

When is SES the wrong default despite the price?

SES is unbeatable on unit cost and it is the correct answer for high-volume senders who already run AWS. The trap is that new accounts start in a sandbox: you can only send to verified addresses, under a low daily quota (200 messages per 24 hours and one message per second at the time of writing), until you file for production access and get approved. Discovering that on launch day is a genuinely bad afternoon, and the approval is not instant.

The second cost is operational. SES gives you an account-level suppression list and event streams, but the loop that turns a hard bounce into "stop emailing this address and mark it invalid in our users table" is yours to build:

// SNS -> your endpoint. Hard bounces and complaints must reach your own suppression table.
export async function handleSesEvent(req) {
  const event = JSON.parse(JSON.parse(req.body).Message);

  if (event.eventType === "Bounce" && event.bounce.bounceType === "Permanent") {
    for (const r of event.bounce.bouncedRecipients) {
      await db.suppress(r.emailAddress, "hard_bounce");
    }
  }
  if (event.eventType === "Complaint") {
    for (const r of event.complaint.complainedRecipients) {
      await db.suppress(r.emailAddress, "complaint");
    }
  }
}

If you want managed sending where bounce suppression, stream separation, and a human deliverability team come with the bill instead of the backlog, Postmark is the one that treats transactional reputation as the product rather than a feature. Its honest drawback: it is priced well above SES per message, it deliberately does not want to be your bulk marketing platform, and message content is retained only for a limited window, so your own logs still have to be the system of record.

SES saves money per message and spends it back in engineering hours — budget the bounce pipeline as part of the decision, not after it.

Where does Resend fit for a small team?

Resend's pitch is developer experience, and it delivers: domain verification is quick, the SDKs are clean, and templating with React Email removes the worst part of HTML email if you already write React. For a solo developer or an early team, it is the shortest distance from zero to a correctly authenticated sending domain.

The honest caveat is track record. Deliverability at shared-IP providers is partly a function of how aggressively the vendor polices its other customers, and that reputation compounds over years. Resend is the newest of the three, so if you are sending high-stakes mail at meaningful volume, plan to monitor Google Postmaster Tools yourself rather than assuming the platform reputation carries you. If your priority is getting authenticated transactional mail out this week with the least configuration, Resend is the one that gets you there fastest.

Pick Resend for speed of setup, but instrument your own deliverability monitoring regardless of provider.

How do you migrate without burning your domain?

Do not repoint an established sending domain at a new provider all at once. Verify the new provider on a fresh subdomain, send a low-stakes category of mail through it first (receipts, not password resets), watch complaint and bounce rates for a couple of weeks, then move the rest. If you are moving to a dedicated IP, warming is mandatory — a cold IP suddenly emitting thousands of messages looks exactly like a compromised host to every receiver, because usually it is one.

Migrate one mail category at a time on a new subdomain; a big-bang cutover risks the one email your product cannot afford to lose.

FAQ

Why do my emails go to spam even though SPF and DKIM pass?
Because DMARC requires alignment, not just passing checks: the domain that passes SPF or DKIM must match the domain in the visible From header. Publish a p=none DMARC record with a rua address and read the aggregate reports — they will name the mechanism that is failing alignment.

Is Amazon SES cheaper than Postmark or Resend?
Per message, yes, and by a wide margin at volume. The cost moves into engineering: SES starts in a sandbox that requires an approval request, and you build bounce, complaint, and suppression handling yourself from SNS or EventBridge events.

Should transactional and marketing email use the same domain?
No. Send transactional mail from a dedicated subdomain and marketing from a different one, so a campaign that draws spam complaints cannot damage the reputation delivering your password resets.

Bottom line

If you are on AWS, sending at volume, and have someone who will own the bounce pipeline, SES is the right economics. If password resets and receipts are business-critical and you would rather pay than operate, Postmark is worth the premium for enforced stream separation and deliverability support. If you are a small team optimizing for time-to-first-correct-send, start with Resend and add your own monitoring. Whichever you pick, fix DNS alignment and split transactional onto its own subdomain first — that single change fixes more spam-folder problems than any migration.

Postgres Says "Too Many Clients Already": Diagnose It Before You Add a Pooler

Libme — Sat, 15 Aug 2026 14:10:10 +0000

FATAL: sorry, too many clients already almost never means your database is out of capacity. It means something in your stack opened more connections than max_connections allows — usually a per-process pool multiplied by more processes than you remembered running, or connections parked in idle in transaction. Adding PgBouncer fixes the symptom, but if you install it without finding the multiplier first, you get the same failure a month later with an extra moving part in the path.

Here is the order I work through it: count the connections and their states, find the multiplier, shrink the app-side pool, and only then decide whether a pooler is actually warranted.

What does "sorry, too many clients already" actually mean?

Postgres allocates a backend process per connection, and max_connections is a hard ceiling set at server start. When clients exceed it, the connection attempt is rejected outright. You'll see one of two messages:

FATAL:  sorry, too many clients already
FATAL:  remaining connection slots are reserved for non-replication superuser connections

The second one is worth knowing separately: it means you've hit max_connections minus superuser_reserved_connections, so ordinary users are locked out while the reserved slots keep a superuser login available for exactly this situation. That's the escape hatch that lets you connect as a superuser and look around while the app is failing.

Both are refusals at the door, not signs of load. CPU and IO can be near idle while this happens.

Takeaway: this error is a counting problem, not a capacity problem — treat it as arithmetic until the arithmetic proves otherwise.

How do I find out where the connections are going?

Connect (as superuser if you're locked out) and group pg_stat_activity by the two things that matter: who opened it, and what it's doing.

SELECT
  usename,
  application_name,
  state,
  count(*),
  max(now() - state_change) AS longest_in_state
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY 1, 2, 3
ORDER BY count(*) DESC;

Three patterns show up over and over:

A wall of idle connections with a single application_name. That's a pool that's sized correctly per process but replicated across more processes than you think.
idle in transaction, with longest_in_state in minutes. A code path opened a transaction, did something slow or fallible outside the database (an HTTP call, usually), and never committed. Those connections hold locks and are unusable by anyone else.
A scattering of active connections all above your intended pool size. Something bypasses the pool — a migration runner, a cron job, an admin tool, a metrics exporter.

The idle in transaction case has a server-side guard worth setting regardless of how you resolve the rest:

ALTER SYSTEM SET idle_in_transaction_session_timeout = '60s';
SELECT pg_reload_conf();

That kills sessions that hold a transaction open past a minute. It converts a silent connection leak into a loud, attributable application error, which is strictly better.

Takeaway: if idle in transaction shows up at all, fix that before touching pool sizes — a pooler will happily leak those too.

Why is my app pool bigger than I think?

The multiplier is where the number actually comes from. Your config says pool size 10; your database sees 240. The count is:

pool_size x processes_per_instance x instances (x 2 if you run a separate replica/read pool)

A Gunicorn or Puma deployment with 8 workers, a pool of 10, across 3 pods is 240 connections before anything unusual happens. Add a background worker deployment with the same defaults and you're past a default max_connections of 100 several times over. Serverless is the extreme version: every warm function instance holds its own connection, and concurrency spikes create instances faster than any pool config can constrain.

Most application pools also have a burst setting beyond the nominal size — SQLAlchemy's max_overflow, HikariCP's maximumPoolSize versus minimum idle — so multiply the ceiling, not the steady state.

And the pool is usually too big to begin with. The sizing guidance that has held up best for me is the one HikariCP has documented for years: connections should be a small multiple of your core count, not a function of your request concurrency. Beyond that, the extra connections just queue inside Postgres instead of inside your app.

Takeaway: multiply your pool ceiling by every process and every replica before you conclude the database is the problem.

When is a connection pooler actually worth adding?

Shrinking pools solves it for a fixed set of long-lived processes. A pooler earns its place when the client count is genuinely unbounded — serverless, autoscaled workers, many small services sharing one database — or when you need thousands of client connections multiplexed onto a few dozen server ones.

Option	Runs where	Best fit	Main drawback
Smaller app pool	Nowhere new	Fixed process count, single service	Doesn't survive autoscaling or serverless
PgBouncer	Sidecar, VM, or container	Anything self-managed; the default choice	Single-threaded per process; you manage HA yourself
Supavisor	Managed on Supabase	Postgres on Supabase, incl. serverless clients	Tied to that platform
pgcat	Sidecar or standalone	Pooling plus load balancing across replicas	Smaller ecosystem; fewer people to ask when it misbehaves
RDS Proxy	Managed by AWS	Lambda against RDS/Aurora, IAM auth	Per-connection pricing; pinning silently kills the benefit

If you're self-hosting and want the option with the longest operational track record, PgBouncer is the one that multiplexes thousands of clients onto a small server pool with a config file you can read in one sitting. If your Postgres is already on Supabase, Supavisor is the pooler that's designed for that platform's connection strings and handles serverless client churn without extra infrastructure. If you're running Lambda against RDS or Aurora and want AWS to own the uptime, RDS Proxy is the one that integrates with IAM authentication and Secrets Manager instead of asking you to distribute database passwords.

A minimal PgBouncer config for the common case:

[databases]
appdb = host=10.0.1.20 port=5432 dbname=appdb

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = transaction
default_pool_size = 20
max_client_conn = 1000
server_idle_timeout = 600

default_pool_size is how many real Postgres connections this pool will ever hold; max_client_conn is how many clients may queue against it. Once it's running, SHOW POOLS; on the admin console tells you whether the size is right — cl_waiting and maxwait above zero mean clients are queuing for a server connection, which is your signal to raise default_pool_size or find the slow queries holding connections.

Takeaway: a pooler is for unbounded client counts; for a fixed number of processes, the cheaper fix is to stop over-provisioning the pool.

What breaks when you switch to transaction pooling?

This is the part that surprises people, because the migration looks like a hostname and port change. In pool_mode = transaction, a server connection is handed back to the pool at the end of every transaction, so any state that lives on the session rather than the transaction is gone or, worse, leaks to whoever gets that connection next. That rules out LISTEN/NOTIFY, session-level advisory locks, SET outside a transaction, WITH HOLD cursors, and temp tables.

The one that actually bites is prepared statements. Drivers that transparently prepare and cache statements — JDBC, asyncpg, psycopg3 — will send a named statement on one server connection and try to execute it on another:

ERROR:  prepared statement "S_1" does not exist
ERROR:  prepared statement "S_1" already exists

Modern PgBouncer (1.21 and later) can track named prepared statements in transaction mode, but you have to opt in with max_prepared_statements; if it's left at the disabled default, you get the errors above no matter how new the binary is. Check the default for the exact version you're running before assuming it's on.

The client-side fixes, if you'd rather not rely on the pooler:

# asyncpg: disable the statement cache
conn = await asyncpg.connect(dsn, statement_cache_size=0)

# psycopg3: never use the extended protocol's named prepares
conn = psycopg.connect(dsn, prepare_threshold=None)

# SQLAlchemy in serverless or behind a transaction-mode pooler:
# don't pool twice — let the pooler own it
from sqlalchemy import create_engine
from sqlalchemy.pool import NullPool

engine = create_engine(DSN, poolclass=NullPool)

For JDBC, the equivalent is prepareThreshold=0 in the connection URL.

Keep one session-mode port (or a separate connection string on 5432) for migrations and admin tools. Schema migrations, CREATE INDEX CONCURRENTLY, and anything using advisory locks belong on a direct connection, not the transaction pool.

Takeaway: transaction pooling is a contract change, not a hostname change — audit for session state and prepared statements before you cut over.

FAQ

Why do I get "too many clients already" when my Postgres CPU is idle?
Because the limit is max_connections, a fixed count set at server start, not a resource threshold. Postgres refuses the connection at the door regardless of how little work the existing backends are doing.

Should I just raise max_connections instead of adding a pooler?
Only modestly, and only if you have the memory for it — each connection is a backend process with its own overhead, and thousands of mostly-idle backends degrade throughput even when they're doing nothing. Raising it from 100 to a few hundred on a well-provisioned server is reasonable; treating it as the answer to serverless connection churn is not.

Can I use PgBouncer transaction mode with an ORM?
Yes, provided you disable the driver's prepared-statement cache (or enable max_prepared_statements in PgBouncer), stop the ORM from maintaining its own connection pool on top, and route migrations to a direct session-mode connection.

Bottom line

Start by counting: group pg_stat_activity by state and application, and set idle_in_transaction_session_timeout so leaks announce themselves. If your process count is fixed, shrink the per-process pool to a small multiple of your cores and the error usually disappears without new infrastructure. Add a pooler when client count is genuinely unbounded — PgBouncer for self-managed setups, Supavisor if you're on Supabase, RDS Proxy if you want AWS to own it. Whichever you choose, treat the move to transaction mode as an application change: audit session state, disable driver-side prepared statements, and keep a direct connection reserved for migrations.

Why Your Postgres Migration Locked the Whole Table (and the Pattern That Doesn't)

Libme — Sat, 15 Aug 2026 12:13:15 +0000

The ALTER TABLE itself is usually not slow — the outage comes from Postgres queueing every other query behind the lock it is waiting for. The fix is two habits: set lock_timeout on every migration session so a blocked DDL statement fails fast instead of freezing traffic, and split anything that rewrites or validates a table into a non-blocking two-step (NOT VALID then VALIDATE, or CREATE INDEX CONCURRENTLY). Everything else in this post is detail on which operations need which treatment.

This is written for Postgres 12 and up, which is where the cheap paths for most of these operations landed. Behavior described here holds as of mid-2026.

Why did a one-second ALTER TABLE take the whole app down?

The symptom is confusing the first time. Your migration adds a column. The migration log shows it eventually completed in a few hundred milliseconds. Meanwhile the API returned 504s for four minutes, and your error tracker is full of:

ERROR:  canceling statement due to statement timeout

Here is what actually happened. Most DDL takes an ACCESS EXCLUSIVE lock, which conflicts with everything, including plain SELECT. Your ALTER TABLE asked for that lock while some long-running query — an analytics SELECT, an idle-in-transaction session left open by a connection pool, a pg_dump — still held ACCESS SHARE. So the ALTER went into the lock queue and waited.

The part people miss: Postgres queues lock requests in order. Once your ALTER is waiting, every new query that needs any conflicting lock on that table lines up behind it, even though those queries could have run fine against the old table. One waiting DDL statement converts a slow query into a full table outage.

You can watch it happen:

SELECT pid, state, wait_event_type, left(query, 60) AS query
FROM pg_stat_activity
WHERE pid = ANY(pg_blocking_pids(<your_migration_pid>));

If that returns a 20-minute-old SELECT from a BI tool, you have found the real cause. The migration was never the problem; it was the thing standing in the doorway.

Takeaway: a blocked migration is more dangerous than a slow one, because everything behind it in the lock queue blocks too.

Which Postgres migrations are actually safe to run at any time?

Not every DDL statement is a hazard. The distinction that matters is whether the operation needs to rewrite the table, scan it, or merely update the catalog.

Operation	Lock behavior	Safe on a large, busy table?
`ADD COLUMN` (nullable, or with a non-volatile `DEFAULT`)	Brief `ACCESS EXCLUSIVE`, catalog-only	Yes, with `lock_timeout`
`ADD COLUMN ... DEFAULT now()` (volatile default)	Full table rewrite	No — backfill instead
`DROP COLUMN`	Brief `ACCESS EXCLUSIVE`, catalog-only	Yes, but breaks `SELECT *` clients
`CREATE INDEX`	Blocks writes for the whole build	No — use `CONCURRENTLY`
`CREATE INDEX CONCURRENTLY`	`SHARE UPDATE EXCLUSIVE`, allows reads and writes	Yes
`SET NOT NULL` directly	`ACCESS EXCLUSIVE` + full scan	No — validate a `CHECK` first
`ADD FOREIGN KEY`	`ACCESS EXCLUSIVE` + scan of both tables	No — use `NOT VALID` then `VALIDATE`
`ALTER COLUMN TYPE varchar(50) -> varchar(100)`	Catalog-only	Yes
`ALTER COLUMN TYPE int -> bigint`	Full table rewrite	No — new column, backfill, swap
`RENAME COLUMN`	Catalog-only, instant	Only with expand/contract deploys

The two columns to internalize: "catalog-only" operations are cheap but still need the lock, so they still need a timeout. "Rewrite or scan" operations are the ones that need a different shape entirely.

Takeaway: the question is never "is this ALTER fast?" but "does it rewrite, does it scan, and how long will it wait for its lock?"

The lock_timeout retry pattern

Every migration session should refuse to wait. Set lock_timeout low — a couple of seconds — so a contended DDL statement gives up instead of building a queue behind it:

SET lock_timeout = '3s';
ALTER TABLE orders ADD COLUMN fulfillment_note text;

If the lock is busy, you get this instead of an outage:

ERROR:  canceling statement due to lock timeout

That error is a success. It means the guard worked. Then you retry, because the blocking query has usually finished by the next attempt:

import time
import psycopg

DDL = "ALTER TABLE orders ADD COLUMN fulfillment_note text"

def migrate(dsn: str, attempts: int = 10) -> None:
    for attempt in range(1, attempts + 1):
        try:
            with psycopg.connect(dsn, autocommit=False) as conn:
                with conn.cursor() as cur:
                    cur.execute("SET lock_timeout = '3s'")
                    cur.execute(DDL)
                conn.commit()
            print(f"applied on attempt {attempt}")
            return
        except psycopg.errors.LockNotAvailable:
            wait = min(2 ** attempt, 60)
            print(f"attempt {attempt} blocked; retrying in {wait}s")
            time.sleep(wait)
    raise RuntimeError("could not acquire lock; investigate long-running queries")

Two details that trip people up. lock_timeout is per-session, not per-statement, so it has to be set on the same connection that runs the DDL — setting it in a different session from your migration tool does nothing. And statement_timeout is not a substitute: it caps total execution time, which will also kill a legitimate long VALIDATE CONSTRAINT you actually wanted to finish.

Most migration frameworks let you set this once. In Django you can put SET lock_timeout in a RunSQL at the top of the migration; in Rails, disable_ddl_transaction! plus a lock_timeout in the connection setup does it. If you would rather not hand-roll the guard, Squawk lints Postgres migration files in CI and fails the build when a statement takes an unsafe lock, which catches the mistake before it reaches production.

Takeaway: a migration that fails with "lock timeout" is a working safety net, not a broken deploy.

How do you add a NOT NULL column to a large table without a rewrite?

Directly running ALTER TABLE ... SET NOT NULL forces a full sequential scan under ACCESS EXCLUSIVE to prove no nulls exist. On a large table that is minutes of total downtime.

Since Postgres 12, you can prove the same thing with a CHECK constraint that validates without blocking writes, and Postgres will then accept SET NOT NULL without rescanning:

-- 1. Add the constraint unvalidated. Instant; applies to new rows only.
ALTER TABLE orders
  ADD CONSTRAINT orders_fulfillment_note_not_null
  CHECK (fulfillment_note IS NOT NULL) NOT VALID;

-- 2. Backfill in batches, committing each batch.
UPDATE orders SET fulfillment_note = ''
WHERE fulfillment_note IS NULL
  AND id IN (SELECT id FROM orders WHERE fulfillment_note IS NULL LIMIT 5000);

-- 3. Validate. Takes SHARE UPDATE EXCLUSIVE: reads and writes keep running.
ALTER TABLE orders VALIDATE CONSTRAINT orders_fulfillment_note_not_null;

-- 4. Now cheap, because the validated CHECK already proves it.
SET lock_timeout = '3s';
ALTER TABLE orders ALTER COLUMN fulfillment_note SET NOT NULL;
ALTER TABLE orders DROP CONSTRAINT orders_fulfillment_note_not_null;

The same NOT VALID / VALIDATE split works for foreign keys, and it is the single most useful trick in this whole area.

Index creation has its own version of this: CREATE INDEX CONCURRENTLY cannot run inside a transaction block, which is why ORMs that wrap every migration in a transaction fail with CREATE INDEX CONCURRENTLY cannot run inside a transaction block. Mark the migration non-transactional. Also check the result afterward — a concurrent build that fails leaves an invalid index behind that silently does nothing:

SELECT indexrelid::regclass AS index_name
FROM pg_index WHERE NOT indisvalid;

Drop and rebuild anything that shows up there.

Takeaway: any operation that needs to scan the table has a two-step form that moves the scan out from under the exclusive lock.

What about renames, drops, and type changes?

These are not lock problems — they are deploy-ordering problems. RENAME COLUMN is instant in the catalog, and that is exactly what makes it dangerous: the old application code is still running and still selecting the old name.

The pattern is expand/contract, and it takes three deploys:

Expand. Add the new column. Write to both old and new from application code. Backfill the old rows.
Migrate reads. Deploy code that reads the new column. Old code is gone at this point.
Contract. Stop writing the old column, then drop it in a later deploy.

The same three-step shape covers int → bigint on a primary key, splitting one column into two, and moving a column to another table. It is slow in calendar time and completely uneventful in production, which is the trade you want.

One more footgun worth naming: DROP COLUMN is catalog-only and fast, but any client doing SELECT * into a struct or row class with strict field mapping will break the moment the column disappears. Deploy the code change first.

If you would rather have this orchestrated for you than hand-written, pgroll implements expand/contract by exposing versioned views of the schema so old and new application versions each see the shape they expect. For teams that want migrations reviewed as declarative schema state with destructive-change linting in CI, Atlas is the one that treats the schema as code and diffs it rather than asking you to write the steps by hand.

Takeaway: renames and drops are safe at the database level and unsafe at the deploy level — sequence the code, not the lock.

FAQ

What does "canceling statement due to lock timeout" mean in Postgres?
It means your statement waited longer than lock_timeout for a lock another session was holding, so Postgres aborted it. It is a protective failure, not corruption — nothing was applied, and retrying after the blocking query finishes normally succeeds.

Does ALTER TABLE ADD COLUMN lock the table in Postgres?
Yes, it takes a brief ACCESS EXCLUSIVE lock, but since Postgres 11 adding a column with a constant default no longer rewrites the table, so the lock is held for milliseconds. The risk is not the duration of the ALTER — it is how long the ALTER waits in the lock queue while other queries pile up behind it.

How do I add an index in production without downtime?
Use CREATE INDEX CONCURRENTLY, and run it outside a transaction block. It takes roughly twice as long and can leave an invalid index behind if it fails, so check pg_index WHERE NOT indisvalid afterward and rebuild anything listed.

Bottom line

If you change one thing after reading this, set lock_timeout on your migration connection and add a retry loop — that alone converts the most common Postgres migration outage into a retried deploy step. Learn the NOT VALID / VALIDATE split next, since it covers NOT NULL, foreign keys, and check constraints with one idea. Use expand/contract for anything that changes a column's name, type, or existence, and accept that it costs three deploys. Reach for tooling — a CI linter like Squawk, or a full orchestrator like pgroll or Atlas — once you have more than a couple of people writing migrations and code review stops catching the unsafe ones reliably.

Free Tier Traps: How Hobby Plans Are Designed and When They Flip to Paid

Libme — Fri, 14 Aug 2026 22:57:07 +0000

A free tier is a marketing budget with a shutoff valve, and the valve is placed on whatever metric grows when your project starts working. The trap is rarely the price you eventually pay — it's that the flip arrives on the vendor's schedule, not yours, usually as a hard stop rather than a slow ramp. The practical defense is to know which dimension each vendor meters, and to keep the two or three pieces that would be painful to move sitting behind an interface you control.

I've had a side project paused, a database run out of connection slots on a Sunday, and a CI pipeline stop mid-release because the month's minutes were gone. None of those were surprises in hindsight. All of them were surprises at the time.

Why do vendors give away a free tier at all?

There are only a few reasons a company hands you compute for nothing, and each one produces a different shape of limit.

Acquisition funnels. The free tier exists to get a credit card on file eventually. These are usually generous on the dimension you notice (deploys, projects, requests) and tight on the dimension that costs the vendor money (bandwidth, storage, seats). They're stable for years because the free users are the pipeline.

Developer mindshare. The vendor wants you to reach for them at work. Limits here tend to be per-project rather than per-account, and the friction is deliberately placed at team features — SSO, roles, audit logs, shared environments — not at raw capacity.

Capacity dumping. Sometimes free tier is just idle infrastructure being monetized as attention. This is the least stable category, because when the underlying economics move, the tier moves with them.

Open core. The free thing is the software, not the hosting. These flip in a different way: the product stays free, but the feature you now depend on migrates into an enterprise edition.

That last distinction matters more than most people weigh it. A hosted free tier can be withdrawn overnight; a permissively-licensed open source project can only be forked away from you slowly, and you keep the running copy either way.

Takeaway: figure out which of the four reasons you're benefiting from, because it predicts how much notice you'll get.

What actually flips a hobby plan to paid?

The flip almost never comes from the metric in the marketing table. It comes from a second-order metric you weren't tracking.

Flip trigger	What it looks like in practice	Warning you get
Sustained egress/bandwidth	One image-heavy page gets linked somewhere	Usually an email, sometimes after the fact
Compute-seconds, not requests	A slow endpoint or an N+1 in a background job	Dashboard graph you have to look at
Concurrent connections	`remaining connection slots are reserved` at the worst time	None — it's a hard error
Inactivity	Project auto-paused after days of no traffic	Notification you'll miss
Rows / storage growth	A logging or events table nobody prunes	Gradual, then a write failure
Seats	You add one collaborator	Immediate paywall
Policy change	The tier itself is retired	Weeks to months of notice

The connection-limit case is the one that ruins weekends. On a small managed Postgres, the connection ceiling is low and serverless functions each want their own connection, so a modest traffic bump produces:

FATAL: remaining connection slots are reserved for non-replication superuser connections

This is not a "you've grown, please upgrade" message. It's an outage. The fix is a pooler, not a plan upgrade — and knowing that difference is worth real money.

// Pool once per process, not per request. In serverless, cap it hard
// and point at the pooler endpoint, not the direct database host.
import { Pool } from 'pg'

const pool = new Pool({
  connectionString: process.env.DATABASE_URL, // pooler URL (often port 6543)
  max: 1,                        // one connection per function instance
  idleTimeoutMillis: 10_000,     // release fast so instances don't hoard slots
  connectionTimeoutMillis: 5_000 // fail fast instead of piling up
})

export async function query(text, params) {
  const client = await pool.connect()
  try {
    return await client.query(text, params)
  } finally {
    client.release()
  }
}

If your platform's free tier gives you a pooled connection string, use it from day one even when you don't need it. Migrating to it under load is a much worse afternoon.

Takeaway: the limit that stops you is the one that isn't on the pricing page.

Which limits bite first, by service category?

As of mid-2026 the patterns below hold across most vendors in each category, though the specific numbers change constantly — always check the current pricing page rather than trusting a number you read in a blog post, including this one.

Category	Metered dimension that bites	Typically fine on free tier	Move-off signal
Static/edge hosting	Bandwidth, build minutes, commercial-use clause	Personal sites, docs, demos	Any revenue-generating traffic
Managed Postgres	Connections, storage, inactivity pausing	Prototypes, low-write apps	Background jobs + real users
Serverless functions	GB-seconds and cold-start behavior	Sporadic traffic	Anything with steady load
CI	Minutes/month, concurrency	Small repos, few branches	Matrix builds, monorepos
Error tracking	Event volume, retention window	Early apps	Any noisy dependency
Auth	MAU threshold, and which features sit above it	Single-app, single-tenant	Orgs, SSO, custom domains

Two specifics worth naming. Cloudflare's free tier is the most durable in the static/edge category because its economics come from network scale rather than from converting hobbyists, but its worker runtime is genuinely different from Node and porting is real work. Supabase's free tier is the most complete "whole backend" offer, but free projects pause after a stretch of inactivity — fine for a portfolio piece, disqualifying for anything a client might open unannounced.

Also, read the commercial-use language. Several excellent free tiers are explicitly for non-commercial or personal projects, and the moment your side project takes payments you are out of compliance regardless of usage volume.

Takeaway: pick the free tier whose metered dimension is the one your app grows slowest on.

How do you know a free tier is about to disappear?

History gives a usable prior. Heroku retired free dynos in late 2022, Railway replaced its free tier with a trial in 2023, and PlanetScale removed its hobby tier in 2024 — each after a period where the free plan was visibly the most-discussed thing about the product. The pattern is consistent: free tiers get withdrawn when the company shifts from growth-at-any-cost to gross-margin discipline, which for infrastructure vendors tends to follow a funding round or an enterprise push.

Signals I now treat as a soft deadline:

The free tier stops being mentioned in launch posts and docs quickstarts.
Support responses start routing free users to community channels only.
A "usage-based" plan appears below the old paid plan — the free tier is being repriced, not removed, and it's next.
The company announces enterprise features (SSO, compliance certifications) as its main roadmap theme.

None of these are proof. All of them are enough to spend an hour making sure you could leave.

Takeaway: when a vendor stops marketing its free tier, start planning like it's already gone.

What does a portable setup actually look like?

You don't need multi-cloud abstraction. You need three things checked before you build on any free tier:

Can I export my data unassisted? A pg_dump-compatible database or an S3-compatible bucket is portable. A proprietary document store with an export button that emails you a link is not.
Is the runtime standard? Code that runs on plain Node/Python containers moves in an afternoon. Code written against a vendor-specific runtime, KV store, or auth SDK moves in a sprint.
Is config in the repo? If redeploying elsewhere means recreating settings you only ever clicked into a dashboard, you don't have a project, you have a pet.

Concretely, keep vendor SDKs behind a thin module of your own — one file with getUser(), sendEmail(), putObject() — so the blast radius of a pricing change is a file, not a codebase. This costs an hour up front and it's the single highest-return hour in a side project.

You can also just watch your usage. Most platforms expose it via API, so a weekly check costs nothing:

#!/usr/bin/env bash
set -euo pipefail
# GitHub Actions minutes for the authenticated user (endpoint as of mid-2026;
# verify against current REST docs if it 404s).
gh api /users/"$(gh api /user --jq .login)"/settings/billing/actions \
  --jq '"used: \(.total_minutes_used)/\(.included_minutes) minutes"'

Wire that into a weekly cron or a CI job and the flip stops being an ambush.

Takeaway: portability is a property of your export path and your runtime, not of your provider's promises.

FAQ

Is it safe to build a real product on a free tier?
Build on it, yes; depend on it, no. Use the free tier for the prototype and the pre-revenue phase, but only if the data export path and the runtime are standard enough that you could move in a day. The moment the project takes payments, budget for the paid plan and check the free tier's commercial-use terms.

Why did my free tier database suddenly stop accepting connections?
Almost always the connection ceiling, not storage. Serverless functions open a connection per instance, so a small traffic spike exhausts the slots and you get remaining connection slots are reserved for non-replication superuser connections. Switch to the provider's pooled connection string and cap max at 1 per instance before you consider upgrading.

Which free tiers are the most stable long-term?
The ones where free users cost the vendor almost nothing relative to its scale, or where the free thing is open source software you can self-host. Free tiers offered as an acquisition funnel by a venture-funded startup are the least durable, because their economics depend on a conversion rate that has to improve eventually.

Bottom line

If you're building a portfolio project or a prototype, take the most generous free tier you can find and don't think about it — that's what it's for. If you're building something that might make money, choose based on the metered dimension: pick the vendor whose meter runs on whatever your app produces least. If your project already has users who'd notice an outage, pay for the database and the error tracking first, since those are the two places where a free-tier limit turns into a hard failure rather than a slowdown. And regardless of tier, spend the hour on exports and a thin vendor wrapper — it's cheap insurance against a pricing email you didn't ask for.

The Real Break-Even for AI Coding Tools Includes Review Time, Not Just Typing Saved

Libme — Fri, 14 Aug 2026 20:26:34 +0000

Most break-even math for AI coding assistants counts only one side of the ledger: the minutes you save not typing boilerplate. That math is incomplete and usually too optimistic. AI-generated code still has to be read, understood, and verified before it ships, and that review cost lands on a human — often a different, more expensive human than the one who prompted it. The tool pays off only when the time it saves you writing code is larger than the extra time your team spends confirming that code does what it claims.

I run AI assistants daily across TypeScript, Python, and Go, and the place they quietly stop paying for themselves is not the subscription line item. It's the pull request that took four minutes to generate and forty minutes to review because nobody could tell at a glance whether the generated error handling was correct or just plausible.

Why does typing-time-saved overstate the value?

The standard pitch measures productivity in keystrokes avoided. Autocomplete finishes your function, an agent scaffolds a module, and you feel fast because your hands did less. But shipping software isn't bottlenecked on typing speed — it's bottlenecked on being sure the code is right.

AI-generated code has a specific failure signature that makes review more expensive than reviewing a teammate's work: it is fluent, idiomatic, and confidently wrong in ways that don't announce themselves. A human writing unfamiliar code tends to leave signals — a hesitant comment, an obviously naive first pass, a TODO. A model emits the same polished surface whether it nailed the edge case or hallucinated an API that doesn't exist. That fluency inverts the normal review heuristic where "reads cleanly" correlates with "probably fine."

So the code arrives faster, but each line carries less built-in evidence of correctness. The reviewer has to supply that evidence themselves, which is slower per line than reviewing code whose author can explain every decision.

Takeaway: the tool moves work from typing (cheap, fast) to verification (expensive, slow) — and if you only count the first, you'll conclude it's free when it isn't.

What does the corrected break-even formula look like?

Here's the version I actually use. Let:

Tw = time the tool saves you writing a change
Tr = extra review/verification time the generated change adds versus code you'd have written yourself
Rgen = your effective hourly rate (the person prompting)
Rrev = the reviewer's effective hourly rate (often a senior, so usually higher)

The change is worth it when the value of writing time saved exceeds the added review cost:

Tw * Rgen  >  Tr * Rrev

Two things fall out of this immediately. First, review time is weighted by a higher rate than the time you saved, because in most teams the reviewer is more senior than the author. Second, Tr can easily exceed Tw on unfamiliar or subtle code, which flips the inequality even before you factor in rates.

A quick worked example, using round numbers so you can substitute your own:

def worth_it(write_saved_min, extra_review_min, gen_rate, rev_rate):
    saved = (write_saved_min / 60) * gen_rate
    cost  = (extra_review_min / 60) * rev_rate
    return saved - cost, saved > cost

# CRUD endpoint: tool saves 20 min, adds ~5 min review, similar rates
print(worth_it(20, 5, 80, 80))   # (+20.0, True)  -> clear win

# Concurrency fix: saves 15 min typing, adds 30 min senior review
print(worth_it(15, 30, 80, 140)) # (-50.0, False) -> net loss

The subscription fee barely registers next to these numbers. For any working developer, the monthly cost of the tool is dwarfed by a single hour of misdirected review time. The subscription was never the real decision — the review load is.

Takeaway: if you're arguing about whether the plan is worth the monthly fee, you're optimizing the smallest term in the equation.

When does review cost stay low enough to win?

The inequality holds comfortably in exactly the situations the marketing implies, and it collapses in the ones the marketing ignores. The dividing line is verification difficulty, not code volume.

Work type	Typing saved	Added review cost	Net verdict
Boilerplate CRUD, DTOs, config	High	Low — behavior is obvious on read	Strong win
Test scaffolding you then tighten	High	Low-medium — you own the assertions	Win
Well-specified pure functions	Medium	Low — easy to unit-test	Win
Glue code across unfamiliar APIs	Medium	High — must verify the API is real	Toss-up
Concurrency, auth, money, migrations	Medium	Very high — subtle failure modes	Usually a loss
Large multi-file refactors	High	Very high — diff is too big to trust	Depends entirely on tests

The pattern: AI coding tools win where correctness is cheap to confirm — where a human can read the diff and know, or a fast test can prove it. They lose where confirming correctness requires holding a lot of context in your head, exactly where a subtle bug is most expensive to let through.

Takeaway: match the tool to code whose correctness you can verify quickly, and the break-even math takes care of itself.

How do you keep review cost from eating the savings?

The comment that prompted this post put it well: the savings disappear if a team accepts code faster than it can verify behavior. The fix isn't to use the tool less — it's to make verification cheaper so Tr stays small.

Concretely, what has actually lowered my review cost:

Generate the test alongside the code, then read the test first. If you can verify the behavior from a focused test you actually understand, you don't have to trace the implementation line by line. The test becomes the receipt.
Keep AI-authored diffs small and single-purpose. A 40-line diff is reviewable; a 400-line generated refactor is a rubber stamp waiting to happen. Constrain the prompt to one change.
Make the model explain its risky decisions in the PR description, then review the explanation against the code. Where they disagree is your bug.
Route generated code through the same CI gates as everything else — type checks, linters, tests, and a static analyzer catch the confident-but-wrong output cheaply, before a human spends senior time on it. If you want a managed layer that runs deeper semantic checks on every pull request without standing up your own pipeline, GitHub's own Copilot-based PR review and third-party services like SonarQube Cloud are the ones that slot into an existing repo without extra infrastructure.

None of this is free either — writing the test costs time. But it's time that converts expensive, unbounded human review into cheap, repeatable machine verification, which is the only move that reliably keeps the inequality on the right side.

Takeaway: you don't reduce review cost by trusting the tool more — you reduce it by making correctness cheap to prove.

FAQ

Does an AI coding assistant actually save time overall?
It saves net time on code whose correctness is cheap to verify — boilerplate, DTOs, well-specified functions, test scaffolding. On subtle code (concurrency, auth, money, large refactors), the added review time often exceeds the typing time saved, so it can be a net loss even though it feels faster.

Should I include code review time in AI tool ROI calculations?
Yes. The honest formula compares writing time saved against extra verification time added, and weights review by the reviewer's hourly rate, which is usually higher than the author's. Leaving review cost out is the single most common way these calculations lie.

Is the monthly subscription the main cost of using Copilot?
No. For most working developers the subscription is a rounding error next to the value of a single hour of review time. The real cost is human verification of generated code, and that's where the decision should focus.

Bottom line

Count both sides of the ledger. AI coding assistants earn their keep on code you can verify quickly and lose money on code that takes a senior engineer an hour to trust — and the subscription fee is almost never the deciding term. Point the tool at boilerplate and well-tested pure functions, keep generated diffs small, and make tests carry the verification load so review time stays low. Do that, and the break-even math isn't close; skip it, and you'll ship faster right up until the review queue and the 2 a.m. bugs quietly erase the savings.

Build vs Buy: Authentication in 2026 (Auth0, Clerk, Supabase Auth, or Roll Your Own)

Libme — Fri, 14 Aug 2026 06:08:09 +0000

If you are choosing auth for a new product in 2026, the honest default is: buy a managed provider for anything user-facing, and only roll your own when you have a specific reason the managed options can't satisfy. Auth0 is the safe enterprise choice with the widest protocol coverage, Clerk is the fastest way to ship a polished sign-in flow for a React/Next.js app, and Supabase Auth is the pragmatic pick when Postgres is already your source of truth. Rolling your own is defensible far less often than developers think — the login form is the easy 10%, and the other 90% is the part that pages you at 2am.

I've shipped all four approaches at different times, and the mistake I made early was treating this as a code decision. It's an operations decision. Below is the framework I use now, the concrete trade-offs, and the failure modes each option hides.

What "roll your own" actually costs

The seductive thing about building auth is that the first commit feels trivial. Hash a password with a modern KDF, store it, compare on login:

from argon2 import PasswordHasher

ph = PasswordHasher()

def register(email: str, password: str) -> str:
    return ph.hash(password)  # store this

def verify(stored_hash: str, password: str) -> bool:
    try:
        ph.verify(stored_hash, password)
        return True
    except Exception:
        return False

That's correct and it's maybe 1% of the job. The bill comes due when you list everything a real auth system owns: session management and rotation, secure cookie flags, CSRF protection, email verification, password reset with single-use expiring tokens, rate limiting and lockout on login, OAuth social logins (each with its own redirect and token-refresh quirks), MFA (TOTP plus recovery codes), account enumeration protection, and an audit trail. Then the recurring work: rotating signing keys, responding to the next credential-stuffing wave, and keeping up with library CVEs.

The failure mode I've watched happen twice: a team ships hand-rolled auth, it works fine at 500 users, then a bot runs a credential-stuffing list against the login endpoint. There's no per-account lockout, no anomaly detection, and the "forgot password" flow leaks whether an email exists via a different response time. None of that was in the original ticket, and all of it is table stakes a managed provider gives you on day one.

Rolling your own auth means signing up to operate a security-critical service forever, not to write a login form once.

When is a managed auth provider worth paying for?

Buy when authentication is necessary but not your differentiator — which is almost always. The value isn't the code you avoid writing; it's the on-call security work you outsource to a team whose entire job is auth.

Buy specifically when you need social logins and SSO without gluing five OAuth flows together, when SOC 2 or enterprise SSO (SAML/OIDC) is on your roadmap, when you want MFA and passwordless without building the recovery-code edge cases, or when you simply don't have someone who wants to own auth incident response.

Lean toward building only when you have a genuinely unusual identity model a provider can't express, a hard data-residency or air-gapped requirement, a cost structure where per-MAU pricing becomes punishing at your scale, or auth is your product. Those are real cases — they're just rarer than the "it's just a login form" instinct suggests.

If auth isn't the thing customers pay you for, paying someone else to operate it is usually the cheaper decision once you price in your own time and risk.

Auth0 vs Clerk vs Supabase Auth vs Roll Your Own

Dimension	Auth0	Clerk	Supabase Auth	Roll your own
Best fit	Enterprise, broad protocol needs	React/Next.js apps wanting polished UX fast	Apps already on Postgres/Supabase	Unusual identity model or hard constraints
Protocol breadth	Widest (OIDC, SAML, enterprise connections)	Growing; strong OAuth/social	OAuth social + email; SAML on higher tiers	Whatever you build
Pre-built UI	Hosted pages, customizable	Drop-in components, best-in-class DX	Basic UI helpers; you build most of it	You build all of it
Data ownership	Provider-hosted identity store	Provider-hosted identity store	Users live in your Postgres	Fully yours
Lock-in risk	Higher (proprietary rules/actions)	Moderate (component-coupled)	Lower (it's your DB)	None
Operational burden	Low	Low	Low-to-medium	High and permanent
Where cost bites	Enterprise-tier feature gating	Per-MAU as you scale	Bundled with platform usage	Your engineering + incident time

A few notes from using these rather than reading their marketing. Auth0's strength is also its complexity: its Actions/Rules pipeline is powerful but proprietary, so heavy customization is exactly what makes migrating away painful later. Clerk's developer experience is the best I've used — you can have a real, styled, MFA-capable sign-in flow in an afternoon — but that convenience comes from tightly coupled components, so it's most compelling if you're committed to the React ecosystem. Supabase Auth's underrated advantage is that your users table lives in your own Postgres, so joining identity to application data is a normal SQL query and there's no separate system to reconcile; the trade-off is that it's less turnkey for complex enterprise SSO than Auth0.

If you want the managed version of enterprise-grade auth with the broadest protocol support, Auth0 is the one that handles SAML, OIDC, and enterprise connections without you assembling them yourself. If you want the fastest path to a polished, secure sign-in flow in a React or Next.js app, Clerk is the one that gives you drop-in components and MFA without building the UI. If your data already lives in Postgres and you want identity to be a normal table you can join against, Supabase Auth is the one that keeps users in your own database.

Pick the provider whose default assumptions match where your data and your stack already are — that's what determines integration pain, more than any feature checklist.

How do you avoid painful lock-in?

You can't eliminate lock-in, but you can contain it. The pattern that has saved me: never let provider-specific types leak past a thin boundary. Wrap the SDK so your application code only ever sees your own User shape.

// auth.ts — the ONLY file that imports the vendor SDK
import { verifyToken } from "@vendor/sdk";

export interface AppUser {
  id: string;
  email: string;
  roles: string[];
}

export async function getUser(token: string): Promise<AppUser | null> {
  const claims = await verifyToken(token);
  if (!claims) return null;
  return {
    id: claims.sub,
    email: claims.email,
    roles: (claims["app_roles"] as string[]) ?? [],
  };
}

Your routes call getUser, not the vendor SDK. When you migrate providers, you rewrite one file instead of grepping the whole codebase. The other durable habit: keep authorization (what a user can do) in your own database, not in the provider's roles/permissions system. Providers are good at authentication; owning authorization yourself keeps your access model portable.

Contain each auth provider behind a single adapter module so switching costs stay a one-file problem instead of a rewrite.

FAQ

Is it safe to build my own authentication in 2026?

It can be safe, but only if you're prepared to own session management, rate limiting, MFA, account-enumeration protection, and CVE patching indefinitely. For most teams the managed providers are safer because auth is their full-time job and yours is your product.

Auth0 vs Clerk: which should I choose?

Choose Clerk if you're building a React or Next.js app and want the fastest polished sign-in experience with minimal UI work. Choose Auth0 if you need broad enterprise protocol support like SAML and OIDC connections or expect complex compliance requirements.

Is Supabase Auth good enough for production?

Yes, especially if your data already lives in Postgres, since users become a table you can join against directly. Its main limitation is that advanced enterprise SSO is less turnkey than Auth0, so evaluate it against your specific SSO roadmap.

Bottom line

Buy your authentication unless you have a concrete reason not to. Choose Auth0 when enterprise protocol breadth and compliance drive the decision, Clerk when developer experience and a React/Next.js sign-in flow matter most, and Supabase Auth when Postgres is already your center of gravity and you want identity to be just another table. Roll your own only when an unusual identity model or a hard constraint genuinely rules the managed options out — and go in knowing you've signed up to operate a security service, not to write a login form. Whatever you pick, hide it behind a one-file adapter and keep authorization in your own database, so the decision stays reversible.

DEV Community: Libme

When Should You Move Off Managed Hosting to Kubernetes? The Thresholds That Actually Matter

Why is "we're getting too much traffic" almost never the real trigger?

What are the structural signals that actually justify a cluster?

What team size can actually carry a cluster?

How do you make the switch without a big-bang migration?

FAQ

Bottom line

Related reading

Where Should a Small Team Store Its Secrets? Comparing .env Files, SOPS, Doppler, 1Password, and Vault

The failure mode that ends the .env era

What are the real options, and what does each cost you?

How do you get secrets into CI without storing secrets in CI?

When is an encrypted-file approach (SOPS) actually right?

FAQ

Bottom line

Related reading

Your Postgres Migration Runner Needs a Retry Contract, Not Just a Lock Timeout

Why do several replicas try to run the same migration at once?

How do I make sure only one migrator runs at a time?

Where should lock_timeout actually be set?

How should the retry loop back off, and what should it log first?

When should the deploy just fail?

FAQ

Bottom line

Related reading

Your Postgres Backups Are Untested Until You Restore One: A Drill for Small Teams

Why does a backup that "succeeded" fail to restore?

What does the drill actually look like?

Nightly dumps or point-in-time recovery?

How often should you run the drill?

FAQ

Bottom line

Related reading

Why Your Docker Build Takes 11 Minutes in CI When It Takes 20 Seconds Locally

Why is the Docker cache empty on every CI run?

How should a Dockerfile be ordered so the cache actually holds?

How do you make the cache survive between CI runs?

What about RUN --mount=type=cache?

Which fix applies to which symptom?

FAQ

Bottom line

Related reading

Getting 429 Too Many Requests? Fix the Client Before You Ask for a Higher Quota

Why does adding retries make 429s worse?

What should the client actually do with Retry-After?

How do I limit the client side without over-engineering it?

Which limiting strategy fits which problem?

What changes when you run more than one worker?

FAQ

Bottom line

Related reading

Your Password Reset Emails Are Going to Spam: Choosing Between Resend, Postmark, and Amazon SES

Why does "delivered" in the dashboard still mean spam folder?

What DNS records do you actually need before blaming the provider?

How do Resend, Postmark, and SES actually differ?

When is SES the wrong default despite the price?

Where does Resend fit for a small team?

How do you migrate without burning your domain?

FAQ

Bottom line

Related reading

Postgres Says "Too Many Clients Already": Diagnose It Before You Add a Pooler

What does "sorry, too many clients already" actually mean?

How do I find out where the connections are going?

Why is my app pool bigger than I think?

When is a connection pooler actually worth adding?

What breaks when you switch to transaction pooling?

FAQ

Bottom line

Related reading

Why Your Postgres Migration Locked the Whole Table (and the Pattern That Doesn't)

Why did a one-second ALTER TABLE take the whole app down?

Which Postgres migrations are actually safe to run at any time?

The lock_timeout retry pattern

How do you add a NOT NULL column to a large table without a rewrite?

What about renames, drops, and type changes?

FAQ

Bottom line

Related reading

What about `RUN --mount=type=cache`?

What should the client actually do with `Retry-After`?