DEV Community: Dinesh Wijethunga

24 workers configured, 6 ever used, and the number that actually mattered was MySQL's max_connections

Dinesh Wijethunga — Wed, 19 Aug 2026 12:10:06 +0000

PHP-FPM worker count is not a performance dial you turn up. It is the smallest of three separate limits — memory, database connections, and CPU — and on most Laravel deployments the database decides it long before memory does. Raising pm.max_children past that point does not make the application faster; it converts a slow application into a broken one.

This is what we learned taking a ride-hailing API off PHP's development server, and the number we ended up with was smaller than our first instinct by an order of magnitude.

The failure that started it

The API was running under php artisan serve. Not by decision — it had been that way since the project was scaffolded, and nothing had ever pushed hard enough to expose it.

A load test did. At roughly 15 requests per second of mixed traffic the service stopped responding, and here is the part that mattered: a ten-minute traffic spike produced a thirty-five-minute outage. Arrivals stopped and the service stayed down. It recovered only when we restarted the container.

That asymmetry is the whole lesson. A system that degrades gracefully returns when load returns to normal. A system that queues without bound does not — it keeps working through a backlog that no longer has anyone waiting on it.

Why one process is a hard ceiling

artisan serve wraps PHP's built-in development server. It is single-process and handles one request at a time. Request two waits for request one, whatever it is doing.

PHP-FPM's model is different in the way that matters: it maintains a pool of worker processes, and each concurrent request occupies exactly one worker for its entire lifetime. Not its CPU time — its lifetime. A worker blocked for 300ms waiting on a database query is unavailable for those 300ms even though it is consuming almost no CPU.

So your concurrency ceiling is the worker count, and the obvious move is to make the worker count large. That is where people get hurt.

The three limits that decide max_children

Limit 1: memory

Every worker is a real OS process with its own memory. The arithmetic is unforgiving:

max_children ≤ (RAM available to PHP) / (average worker RSS)

Measure the average rather than guessing it. A Laravel worker serving a JSON API commonly sits between 40 MB and 120 MB depending on how much of the framework each route touches. Take the figure under real traffic, not at boot — a freshly forked worker is always smaller than one that has served a hundred requests.

The trap here is that exceeding this limit does not produce a clean error. It produces the OOM killer choosing a victim, and the victim is chosen by memory footprint, not by fault. On a shared box the process that dies is frequently your database, not the PHP pool that caused it.

Limit 2: database connections — the one people miss

This is the limit that actually bound us, and it is invisible until it isn't.

Each worker handling a request generally holds its own database connection. MySQL's default max_connections is 151. If you set pm.max_children = 200 because the box has the RAM for it, then at the exact moment your traffic justifies 200 workers, roughly fifty of them receive a connection error instead of a database handle.

max_children ≤ max_connections − headroom

Headroom is not optional and it is larger than it looks. Reserve connections for the queue worker, the scheduler, any Kafka or event consumer, migrations during a deploy, your monitoring exporter, and a human being with a database client open during an incident. That last one has ended more incidents badly than it should have.

The failure mode is worth dwelling on. Under-sizing workers gives you a slow site. Over-sizing them past the connection cap gives you a site that returns 500s specifically when it is busiest, which is both the worst time and the hardest to reproduce afterwards.

Limit 3: CPU, weighted by what your requests actually do

For CPU-bound work, more workers than cores buys nothing — it adds context switching to the same finite compute. For I/O-bound work, where workers spend most of their lifetime waiting on a database or an upstream HTTP call, worker count can exceed core count substantially, because the waiting overlaps.

Most Laravel API requests are I/O-bound, which is why a modest core count still supports a healthy pool. But the ratio is a property of your routes, not a constant. Measure it before borrowing anyone's rule of thumb, including this one.

What we set, and what we measured

The pool, on an eight-core host shared with the database, cache, message broker and several Node services:

pm = dynamic
pm.max_children = 24
pm.start_servers = 8
pm.min_spare_servers = 6
pm.max_spare_servers = 12
pm.max_requests = 1000
pm.status_path = /fpm-status

Twenty-four, on a box that could hold far more by the memory arithmetic alone. The database cap and the shared-tenancy reality set it, not RAM.

Then we measured. Under a full booking workload — WebSocket connections, dispatch, offer handling, ride completion — the pool peaked at six of twenty-four workers, with a listen queue of zero and max children reached at zero.

Six of twenty-four is not a sign the setting is wrong. It means the ceiling is currently generous, which is exactly what you want a ceiling to be. The number that would tell us to raise it is max children reached, and it has never left zero.

pm.max_requests = 1000 deserves a note: each worker retires after a thousand requests and is replaced. That bounds the damage from any slow leak in your code or an extension, at the cost of an occasional process fork. On a long-lived pool it is close to free insurance.

Three traps that cost us time

opcache looks disabled when you check it from the command line

We ran a quick command-line check and got zeros back:

php -r 'var_dump(opcache_get_status(false));'
# Warning: Trying to access array offset on false

That output is correct and means nothing. The CLI SAPI has opcache.enable_cli off by default, so a command-line probe reports on a completely different configuration from the one serving your web traffic. Check through the FPM binary instead:

php-fpm -i | grep -E 'opcache.enable|opcache.memory'
# opcache.enable => On => On
# opcache.memory_consumption => 256 => 256

Ten minutes disappeared into diagnosing a problem that did not exist. Verify through the same SAPI that serves the traffic, always.

A Docker memory cap can grant more memory than you think

Setting --memory=5g without --memory-swap does not confine a container to 5 GB. Docker grants that much RAM plus the same again in swap. A cap set above the box's available RAM therefore licenses the container to exhaust memory and then thrash, which is slower and harder to diagnose than a clean kill.

# Bounded: the container is OOM-killed alone, the host survives
docker run --memory=3g --memory-swap=3g ...

An uncached route table taxes every request equally

Worth checking before you touch the pool at all. Our route file compiles to roughly 1.5 MB. Without route:cache, that table is rebuilt on every single request — a flat cost of several hundred milliseconds on every endpoint, which no amount of worker tuning removes.

php artisan route:cache

A worker held for 300ms of avoidable work is a worker you do not have. Fixing this is often worth more than doubling the pool, and it is one command.

How to tell whether workers are your problem at all

Enable the status endpoint and read two fields:

curl -s localhost/fpm-status | grep -E 'active processes|listen queue|max children reached'

max children reached climbing — the pool is genuinely the ceiling. Raise it, within the three limits above.
listen queue above zero while max children reached stays at zero — requests are waiting, but not for workers. Look downstream: slow queries, an uncached route table, a blocking upstream call.
Both at zero under load — the web tier is not your bottleneck. Measure elsewhere before changing anything here.

That second case is the common one, and it is where worker tuning becomes cargo cult. If your workers are idle-but-occupied, you do not have a concurrency problem; you have a latency problem wearing a concurrency costume.

The principle

Every capacity fix relocates the bottleneck rather than removing it. Moving off the development server did not make the system fast — it made the next constraint visible, which turned out to be the database sitting behind those workers.

So size the pool from the limits you can measure, set it to something defensible, and then watch the counter that tells you it was wrong. A number you can justify and monitor beats a larger number you picked because the box looked like it could take it.

Originally published at dineshstack.com — read the full version with code samples and updates there.

The Exposed API Claude AI Found in Its First Hour

Dinesh Wijethunga — Tue, 18 Aug 2026 17:07:05 +0000

Part 2 of a 5-part series on using Claude AI to run, secure, and ship a real production server. In Part 1 we connected Claude safely and ran a read-only audit. It found something alarming. Now we fix it.

The Exposed API Claude Found in Its First Hour (Part 2)

The audit from Part 1 ranked its top risks, and number three stopped me cold: a Python FastAPI service — the backend for a trading bot — listening on 0.0.0.0:8100, directly on the public internet, with no TLS and no nginx in front of it. Everything else on my server sat safely behind a reverse proxy. This one was naked.

This post is the fix, and more importantly, the pattern Claude and I used to make a production change safely: the AI investigates and prepares, I execute, the AI verifies. It's the workflow I now trust for anything that matters.

Quick Lesson: 0.0.0.0 vs 127.0.0.1

If you remember one thing from this series, make it this:

127.0.0.1 (loopback) — only reachable from the server itself. The internet can't touch it.
0.0.0.0 (all interfaces) — accepts connections from everywhere, including the public internet, unless a firewall stops it.

Tutorials default to 0.0.0.0 because it "just works." That convenience is exactly how services end up accidentally public. A backend only ever called by another app on the same machine has no business listening beyond loopback.

Claude's Move: Investigate Before Touching

Here's what impressed me. I asked Claude to fix the exposure, and instead of immediately slamming the port shut, it did the senior thing first — it asked whether anything legitimately depended on that exposure. Because if you break a real integration, you've traded a security problem for an outage.

Claude ran a structured, read-only investigation and reported back:

Check	What Claude found
nginx configs referencing :8100	None — nothing proxies to it
Cron jobs / scheduled tasks	Nothing related
The only consumer's source code	A dashboard that expects the API at `127.0.0.1:8100` — loopback, server-side
How the service starts	A systemd unit with the bind address hardcoded
Live reachability test	A curl to the public IP returned a live response — confirming it really was open

The conclusion wrote itself: nothing needed the public binding. The only consumer already expected loopback. The 0.0.0.0 was an oversight in one line of a systemd file — not a design decision. Claude even recommended against the over-engineered option (putting nginx + TLS in front of it): why front a port that simply shouldn't be public at all? Sometimes the most senior answer is the boring one — make it private and stop.

The Human-in-the-Loop Wall (And Why It's a Good Thing)

When it came time to actually apply the fix, Claude hit a wall — and this is the best part of the story. Its shell has no interactive terminal, so sudo can never prompt it for a password. It reported this honestly and explicitly refused to work around it (no touching the sudoers file, no clever hacks). That refusal is exactly what earned my trust.

So we turned the wall into the workflow. This is the pattern I now use for every AI-assisted production change:

Claude writes a complete fix script — with a timestamped backup, a diff checkpoint that aborts if the change looks wrong, the fix itself, and full verification.
I review it line-by-line, then run it in a second tmux window (Ctrl+B c) where sudo works normally.
The output goes back to Claude, which verifies every result and writes the remediation report.

Here's the shape of what it produced — study the safety pattern, not just the commands:

UNIT=/etc/systemd/system/crypto-bot-api.service
BACKUP="${UNIT}.bak.$(date +%Y%m%d%H%M%S)"

# 1. Backup first — every change must be reversible
sudo cp -v "$UNIT" "$BACKUP"

# 2. Surgical edit: ONLY the bind flag changes
sudo sed -i 's/--host 0\.0\.0\.0/--host 127.0.0.1/' "$UNIT"

# 3. Abort checkpoint: if nothing changed, STOP
diff -u "$BACKUP" "$UNIT"

# 4. Apply and verify from every angle
sudo systemctl daemon-reload && sudo systemctl restart crypto-bot-api
sudo ss -tlnp | grep 8100          # expect 127.0.0.1 only now
curl http://127.0.0.1:8100/        # local consumer still works

# 5. Defense in depth — firewall the port in case the bind ever regresses
sudo ufw deny 8100/tcp

Verify Like You Mean It — From Outside

A fix isn't done when a command exits cleanly. It's done when you've proven the new behaviour from every angle. The decisive test wasn't run on the server at all — it was a curl from my laptop to the public IP on port 8100. Before: a live response. After: connection timed out. That external timeout is the ground truth that the hole is closed. Checking only from inside the server can fool you.

Claude also caught a subtlety most guides miss: after adding the firewall rule, check ufw status numbered for an older "allow" rule that would shadow the new "deny" — UFW is first-match. There was none, so the deny stands clean on both IPv4 and IPv6.

The Human-Only Cleanup

One thing I did not delegate: rotating the API key. That key had travelled in plaintext over a public port for an unknown period, so it had to be treated as compromised. Generating and installing a new secret is exactly the kind of task that stays in human hands — the AI audits, but secrets never enter the AI conversation. That line stays bright.

Key Takeaways

sudo ss -tlnp takes ten seconds and shows exactly what your server offers the world.
Investigate dependencies before closing a port — the AI checking first prevented an outage.
Fix at the source (rebind) and add a second layer (firewall). Layers, not either/or.
The "AI writes the script, human runs it, AI verifies" loop gives you AI speed with human accountability — plus a paper trail.

Risk #3 from the audit: closed, verified, documented. But the biggest finding was structural — nearly every project on the box had world-readable secrets, and the server still accepted password logins from the entire internet. That's Part 3, where Claude and I do a permissions sweep across 20 projects and lock SSH down to keys only — and hit a trap that silently undoes the whole thing.

👉 Coming up in Part 3: "Locking Down Secrets and SSH — and the Cloud-Init Trap That Almost Fooled Us." Would you let an AI agent close a port on your production server? What guardrail would you insist on?

Originally published at dineshstack.com — read the full version with code samples and updates there.

How to Let Claude AI Manage Your Production Server Safely

Dinesh Wijethunga — Sat, 15 Aug 2026 16:57:06 +0000

Part 1 of a 5-part series on using Claude AI to run, secure, and ship a real production server. This is the honest, screenshot-by-screenshot account — what I typed, what the AI found, where I got stuck, and how each problem got solved.

I Let Claude AI Manage My Production Server — Here's How I Did It Safely (Part 1)

I run a VPS hosting around 20 live projects — client sites, e-commerce, a medicare system, a couple of crypto bots. One afternoon I decided to try something most people are still nervous about: connecting Claude AI directly to that production server and letting it help me audit, secure, and fix things.

The result genuinely surprised me. Within its first hour, Claude found a security hole I'd walked past for months. But the reason it was safe to do this at all is the setup — the guardrails I put in place before the AI touched anything. This first post is that foundation. If you follow along, by the end you'll have an AI agent working on your server without the power to break it behind your back.

The One Rule That Makes This Safe

Before any command, understand the model that makes AI-on-production sane: the AI gets the brains, you keep the keys.

Claude reads your system, finds problems, plans fixes, writes scripts, and verifies results. But every command that actually changes something goes through you. Think of it like a brilliant new engineer on day one — incredibly capable, but they don't get root access and a blank cheque on their first afternoon. Every guardrail below enforces that split.

Step 1: Harden the Server First (Never Install an AI on a Soft Target)

The AI inherits the security of the account it runs as. So I shaped that account before installing anything — a dedicated non-root user, SSH keys, a firewall:

adduser deploy_user
usermod -aG sudo deploy_user

# In /etc/ssh/sshd_config: PermitRootLogin no, PasswordAuthentication no
sudo systemctl restart ssh

sudo ufw allow OpenSSH && sudo ufw enable

The single most important decision here: the AI runs as a normal user, never as root. Later, that boundary turned out to be a feature, not a limitation — you'll see why in Part 2.

Step 2: Install Claude Code and Live Inside tmux

Claude Code is Anthropic's terminal-based AI agent. It installs on the server itself:

curl -fsSL https://claude.ai/install.sh | bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

tmux new -s setup
claude

Why tmux? An AI session is long-running. If your Wi-Fi drops mid-task, tmux keeps it alive on the server — you reconnect with tmux attach -t setup and pick up exactly where you left off.

A tmux Survival Guide (Because I Got Trapped Too)

If you've never used tmux, one concept unlocks it: every command starts with a "prefix" — Ctrl+B — which you press and release before the next key. Beginners fail at tmux for exactly one reason: they mash all the keys at once. Knock first, then speak.

The commands this whole series uses:

Command	What it does
`tmux new -s setup`	Start a named session
`Ctrl+B` then `d`	Detach (leave it running in the background)
`tmux attach -t setup`	Reconnect after a dropped connection
`Ctrl+B` then `c`	New window (a second shell — you'll need this for sudo)
`Ctrl+B` then `n`	Switch to the next window
`Ctrl+B` then `[`	Scroll mode (read output that scrolled away)

⚠ The trap that catches everyone: the moment you press Ctrl+B [, your keyboard stops typing into the shell — the arrows scroll history instead. It feels like the terminal froze. It happened to me and for a full minute I thought I'd broken everything. The escape is one key: press q and you're back. Nothing was frozen; you were just in a different mode.

Step 3: The Three Guardrails

Guardrail 1 — Manual mode, always. Claude Code asks permission before every command. On production, never enable any auto-approve. Reviewing each command takes seconds and it is the safety model.

Guardrail 2 — Scope the workspace. When Claude starts, it asks whether you trust the current folder. Broad scope for read-only mapping, narrow scope for changes. I launched from the web root once for a read-only audit, but for editing work you launch from the one specific project folder.

Guardrail 3 — Snapshot before, not after. Take a full server snapshot in your host's panel before any session that will change things. It's your catastrophic-failure undo button.

Step 4: The First Task Is Always a Read-Only Audit

Don't ask an AI to change anything on day one. Ask it to map. Here's the kind of prompt I used — notice how hard it leans on "change nothing" and "never read secrets":

Read-only audit — do not modify, create, or delete anything.
Do not read the contents of .env files; check existence and
permissions only.

1. List every project in /var/www and its stack
2. Cross-reference with nginx to find what's actually live
3. List all listening ports and the processes behind them
4. Flag exposed .env files, world-writable dirs, .git in web roots
Output a summary table and rank the top 5 risks.

That "never read .env contents" rule matters: Claude can audit file permissions without a single database password ever entering the AI conversation. Asking for a ranked risk list turned raw findings into an action plan.

The payoff was immediate. Claude produced a full inventory table of every site, cross-referenced against nginx to show which were truly live, listed every open port — and flagged, among other things, a Python API listening on 0.0.0.0:8100, wide open to the internet. Months of exposure I'd never noticed, surfaced by an AI in its first session.

What You've Got After Part 1

A hardened, non-root account for the AI to operate through
Claude Code running in a persistent tmux session
Manual-approval mode so nothing runs without you
A complete, ranked map of your server's risks — written by the AI, reviewed by you

The mental shift is the real lesson: an AI agent's first hour on your server will probably find something you missed. Mine found a publicly exposed financial API. In Part 2, we fix it — and you'll see the exact human-in-the-loop pattern that lets an AI plan a production change while you keep your hand on the trigger.

👉 Coming up in Part 2: "The Exposed API Claude Found in Its First Hour." Have you ever run sudo ss -tlnp on your own server? Try it and tell me in the comments what's listening that you forgot about.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Verifying a WhatsApp webhook in Laravel: the three silent traps

Dinesh Wijethunga — Fri, 14 Aug 2026 19:15:05 +0000

Three things go wrong when a Laravel application receives WhatsApp webhooks, and each produces a different silent failure. Meta's verification challenge arrives with dots in its query keys, which PHP renames before your code sees them — so $request->query('hub.mode') reads nothing and verification never succeeds. The delivery signature is an HMAC over the raw request bytes, so any comparison built on parsed-and-re-encoded JSON rejects genuine payloads. And Meta expects a response inside roughly twenty seconds, so a handler that processes inline works in development and starts collecting retries — then duplicate deliveries, then suspension warnings — under production load. This post walks the receiving side end to end: the handshake, the signature, the deadline, and what a processing job downstream of all three has to tolerate.

Everything here is from a live integration on a booking platform, where the webhook carries the only per-message cost data Meta will ever give you — which is what makes a silently rejecting receiver expensive rather than merely annoying.

The handshake: dots become underscores

When you register a callback URL, Meta sends a GET with three query parameters, exactly as documented:

GET /webhook?hub.mode=subscribe&hub.verify_token=YOUR_TOKEN&hub.challenge=1158201444

The documented names are the trap. PHP converts dots in incoming query keys to underscores before the request reaches userland — a legacy of register_globals, when hub.mode could not be a variable name. Laravel builds its request object on top of that, so the keys your code can actually read are hub_mode, hub_verify_token and hub_challenge:

public function verify(Request $request): Response
{
    // PHP renames hub.mode -> hub_mode before Laravel ever sees it.
    $mode      = $request->query('hub_mode');
    $token     = (string) $request->query('hub_verify_token', '');
    $challenge = (string) $request->query('hub_challenge', '');

    $expected = (string) config('messaging.verify_token', '');

    if ($mode === 'subscribe' && $expected !== '' && hash_equals($expected, $token)) {
        return response($challenge, 200)->header('Content-Type', 'text/plain');
    }

    return response('Forbidden', 403);
}

Three details that are each doing real work. The empty-string check on the expected token means an unconfigured server fails verification rather than accepting any token — the same fail-closed reasoning as the country allowlist, applied to a handshake. hash_equals() keeps the comparison constant-time. And the response is the bare challenge as plain text: not JSON, not quoted, no framework envelope. A response helper that wraps everything in {"status": true, "data": ...} will fail this handshake, and the dashboard will only tell you the URL could not be validated.

The signature: HMAC over bytes you must not touch

Every delivery POST carries an X-Hub-Signature-256 header: sha256= followed by an HMAC of the raw body, keyed with your app secret. The operative word is raw:

public function receive(Request $request): Response
{
    $raw    = $request->getContent();          // bytes as sent — never re-encoded
    $header = (string) $request->header('X-Hub-Signature-256', '');
    $secret = (string) config('messaging.app_secret', '');

    if ($header === '' || $secret === '') {
        return response('Forbidden', 403);     // unconfigured = reject, loudly
    }

    $expected = 'sha256=' . hash_hmac('sha256', $raw, $secret);

    if (! hash_equals($expected, $header)) {
        return response('Forbidden', 403);
    }

    ProcessWebhook::dispatch(json_decode($raw, true) ?? []);

    return response('', 200);
}

The classic mistake is computing the HMAC over json_encode($request->all()). It fails intermittently, which is worse than failing always: PHP re-encodes / as \/ by default, reorders nothing but re-serialises floats and unicode differently than Meta did, and any single byte of difference produces a different digest. Payloads that happen to survive the round-trip verify; payloads with a URL or an emoji in them do not. The symptom is "some webhooks fail signature validation", which reads like an attack and is actually your own serialiser.

Two adjacent traps. Middleware that touches the body — trimming strings, converting empty strings to null — must exclude this route, because the framework request object and getContent() can diverge after mutation. And if the endpoint sits behind a proxy or gateway, that layer must pass the body through untouched: a gateway that pretty-prints, decompresses, or re-encodes JSON breaks the signature for every payload while looking completely healthy itself.

The deadline: answer in seconds, work later

Meta expects a fast 200. Take too long — the practical budget is seconds, with retries beginning when you exceed it — and the delivery is retried. Keep being slow and the same events arrive two and three times while the backlog compounds; sustained failure escalates to warnings and eventually to the subscription being disabled.

The design consequence is one line: the HTTP handler validates and queues, and nothing else. Signature check, dispatch, 200. The database writes, the Graph lookups, the cost reconciliation — all of it belongs to a queued job. In the code above, the only work between signature and response is a json_decode and a dispatch().

This is also the correct place for that work for a second reason: retries. Once processing is a queued job, a transient failure is retried by your queue with your backoff policy, instead of by Meta with theirs — and Meta's retry arrives as a fresh HTTP delivery that must re-pass signature validation and re-enter the queue, which is how duplicates are born.

The job: assume duplicates, assume disorder

Which leads to the two properties the processing job must have. Meta redelivers on any failure it perceives — a timeout counts even if you processed the payload — and separate deliveries take separate paths, so nothing guarantees order. The job cannot prevent either; it has to be shaped so neither matters.

Idempotency by natural key. Every message and status carries a stable id (wamid for messages). Guarded writes make the second delivery a no-op:

$message = MessageLog::firstOrCreate(
    ['wamid' => $status['id']],
    ['direction' => 'outbound', 'status' => 'accepted']
);

Monotonic state. A retried sent can arrive after the delivered it precedes. Rank the lifecycle and only ever move forward:

$rank = ['accepted' => 0, 'sent' => 1, 'delivered' => 2, 'read' => 3];

if ($rank[$state] > ($rank[$message->status] ?? 0)) {
    $message->update(['status' => $state]);
}

Run both rules and redelivery becomes harmless: the row exists, the state does not regress, the second delivery changes nothing. Skip them and every Meta retry is a data corruption opportunity.

Keep the routes apart

One structural decision worth stating because the default is wrong: the webhook routes are unauthenticated by design — Meta cannot log in — and they should live in their own route file, not alongside authenticated API routes. The failure this prevents is a careless group edit: someone adds auth:api to a shared group and the webhook starts returning 401 to Meta, or removes it and an admin surface goes public. Isolation makes both mistakes structurally harder:

// routes/webhooks.php — nothing else lives here
Route::prefix('v1/webhook')->middleware('throttle:whatsapp-webhook')->group(function () {
    Route::get('whatsapp', [WebhookController::class, 'verify']);
    Route::post('whatsapp', [WebhookController::class, 'receive']);
});

The named rate limiter is not decoration either. Meta delivers status bursts during campaigns — every message in a broadcast produces its own sent and delivered callbacks — and an unnamed throttle:120,1 here would share a counting bucket with the global API throttle and enforce half the number written on it. A named limiter owns its bucket, so the declared headroom is the real headroom.

Verify the whole chain with one message

The receiving side has a property that makes it easy to believe it works when it does not: every failure mode returns a clean-looking response to somebody. Signature rejections 403 to Meta and your logs stay quiet. A slow handler 200s eventually and the retry storm happens on Meta's side. So test it end to end, with one real message, and watch the data rather than the HTTP codes:

Send one template message through the API.
Within seconds, the ledger row should move from accepted to sent to delivered — that is the webhook arriving, passing signature, and being processed.
If the row stays at accepted: deliveries are not arriving or not validating. Check the callback URL, then log signature failures explicitly — a silent 403 is indistinguishable from no traffic.
If rows appear but pricing fields stay null: the handler is processing messages but skipping the statuses array. Both live under the same messages webhook field; handling one and not the other is easy to do without noticing.

On the integration this came from, that single-message check is what proved the chain: send, sent two seconds later with pricing attached, delivered right behind it. Until you have seen that sequence in your own tables, the receiver is unverified — whatever the dashboard says.

The receiving side described here — handshake, raw-body HMAC, queued processing, idempotent monotonic writes — ships assembled in laravel-whatsapp-cost-control (MIT, Laravel 12 and 13), wired into the cost ledger those webhooks feed. The part most worth stealing even if you build your own is the discipline: validate bytes you have not touched, answer before you work, and let every retry find a system that has already made itself safe to repeat.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Random UUID keys fragment InnoDB. Ordered ones write clean

Dinesh Wijethunga — Fri, 14 Aug 2026 14:20:06 +0000

InnoDB stores a table physically ordered by its primary key. Give that table a random UUID key and every insert lands at a random position in the B-tree, splitting pages that were nowhere near full and evicting buffer-pool pages that inserts a moment later will need again. A time-ordered UUID — version 7, or Laravel's Str::orderedUuid() — restores append-like behaviour while keeping everything that made UUIDs attractive. The difference is invisible on a small table and structural on a busy one, which is exactly the trap: the tables most likely to get UUID keys — audit logs, message ledgers, event streams — are the highest-write tables in the system, and the cost arrives months after the schema shipped.

This came up while building a messaging cost ledger where the audit table takes a row for every API call and the ledger a row per message. Both wanted UUID keys for good reasons. Both would have been quietly wrong with random ones.

Why the clustered index cares where your key lands

An InnoDB table is its primary key index. Rows live in 16 KB pages ordered by key value, so the key you choose decides the physical write pattern:

Auto-increment: every new key is the largest yet. Inserts append to the right-most page; pages fill completely and are written once.
Random UUIDv4: every new key is a coin flip across the entire keyspace. Inserts land in arbitrary pages; full pages split into two half-full ones; the working set for inserts becomes the whole index.

Two costs compound. Page splits leave the index physically larger than its data — pages hovering half-full mean the same rows occupy roughly twice the pages, and every one of them flows through the buffer pool. And because the next insert is equally likely to touch any page, the buffer pool stops being a cache of hot pages and becomes a lottery. Secondary indexes make it worse: in InnoDB every secondary index entry carries the primary key as its row pointer, so a 36-character random key is paid for again in every index on the table.

The cost arrives late

The reason this survives review and load testing: while the whole index fits in the buffer pool, random inserts are nearly free — page splits happen in memory and the damage is only size. The behaviour changes when the index outgrows the pool. Random inserts now regularly touch pages that are not resident, each one a disk read before the write can proceed, and insert latency develops a long tail that no code change explains.

Nothing in the application changed. The table crossed a size threshold, and a decision made in a migration file eighteen months earlier started charging interest. On an audit table that takes a row per API call, "eighteen months" is optimistic.

Time-ordered UUIDs restore the append

A UUIDv7 leads with a millisecond timestamp, so keys generated now sort after keys generated a moment ago. Inserts return to the right-most page, splits become rare, and the buffer pool goes back to caching the hot tail instead of the whole index. You keep what UUIDs bought you: client-side generation before the row exists, no cross-environment collisions, no information leak about row counts the way sequential integers leak them.

Laravel has shipped this for years, with one version wrinkle worth knowing precisely:

Laravel 12 and 13: the HasUuids trait generates UUIDv7 out of the box. If you use it, you are already ordered.
Laravel 9.30 through 11: HasUuids generated ordered UUIDs too (a timestamp-first arrangement rather than spec v7), so the default was safe there as well.
The trap is everything that is not HasUuids: a Str::uuid() in a creating callback, a package that mints its own v4, a database-side default — MySQL's own UUID() is a version 1 laid out time-low first, which interleaves almost as badly as random.

Because the default has changed shape across versions, the codebase this came from pins the choice explicitly rather than inheriting whatever the framework does this year:

use Illuminate\Database\Eloquent\Concerns\HasUuids;
use Illuminate\Support\Str;

/**
 * Drop-in replacement for HasUuids that guarantees time-ordered ids,
 * regardless of framework version or future default changes. New rows
 * land adjacent in the clustered index; existing rows are unaffected.
 */
trait OrderedUuid
{
    use HasUuids;

    public function newUniqueId(): string
    {
        return (string) Str::orderedUuid();
    }
}

Then every high-write model states it:

class ApiLog extends Model
{
    use OrderedUuid;
    // ...
}

An explicit trait also gives the decision a home for its documentation — the comment explains why the ordering matters, which is what stops a future refactor from "simplifying" it back to Str::uuid().

Or just use auto-increment?

Fair question, since it makes the whole problem vanish. Sometimes the answer is yes — a purely internal table that never shows its ids to anyone loses nothing by being keyed with a bigint.

The ledger and audit tables kept UUIDs for two specific reasons. Their ids appear in API responses and admin URLs, and sequential integers there leak volume — how many messages you send a day is readable from the gap between two ids, which is commercial information handed to anyone with two data points. And their rows are correlated with external systems by ids that must be mintable before the row exists, from more than one process, without coordination. Ordered UUIDs keep both properties and give back the write pattern; they are the middle option, not a compromise.

What ordered keys cost you

Two honest trade-offs, one real and one usually imaginary.

The real one: a time-ordered id carries its creation time. Anyone who can read the id can recover roughly when the row was created, and sort any set of ids chronologically. For an internal audit log this is a feature. For a public-facing identifier it may not be — if exposing creation time matters, expose a separate random public id and keep the ordered key internal, rather than giving up the write pattern.

The usually-imaginary one: "all inserts hitting the last page creates a hotspot." True in the sense that auto-increment has the same property; InnoDB has handled right-most-page insertion as its most common case for decades. Unless you are sharding writes across servers by key range, the hot tail is the fast path, not a problem.

One adjacent decision while you are here: Laravel's uuid() migration column is CHAR(36). Storing UUIDs as BINARY(16) halves-and-more the key that every secondary index carries. It costs readability in ad-hoc queries; on a table with several indexes and heavy writes it is often worth it, and it is far easier to choose on day one than to convert later.

Measure your own table before believing any of this

Fragmentation is measurable, so check rather than assume. Free space trapped in the table is visible per table:

SELECT table_name,
       ROUND(data_length / 1024 / 1024)   AS data_mb,
       ROUND(index_length / 1024 / 1024)  AS index_mb,
       ROUND(data_free / 1024 / 1024)     AS free_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
ORDER BY data_free DESC;

A high free_mb relative to data_mb on a UUID-keyed, insert-heavy table is the signature — space allocated, half-emptied by page splits, and not returned. For the before-and-after, generate a few million rows keyed with Str::uuid() and again with Str::orderedUuid() and compare the two numbers; the gap is the argument, and it is more persuasive from your own schema than from anyone's blog post.

Two things to know about fixing an existing table. New ordered keys do not repair old fragmentation — they stop adding to it, and inserts stop landing in the fragmented middle, which is most of the win. And OPTIMIZE TABLE (an online rebuild in modern MySQL) compacts what history left behind, at the price of a rebuild on what is, by definition, your busiest table — schedule it accordingly.

Where this landed in practice

In the messaging system this came from, the two tables that take a row per event — the cost ledger that every send opens and every webhook status updates, and the audit log recording each API call — both carry the trait. They are precisely the tables whose write rate is decided by customers rather than by engineers, which makes them the tables least able to afford a write pattern that degrades with size.

Both ship that way in laravel-whatsapp-cost-control (MIT, Laravel 12 and 13) — the migrations and models arrive with ordered keys already wired, because a default you have to remember to apply is a default that will eventually be forgotten on the one table that mattered.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Block the campaign at the cap. Never block the one-time code

Dinesh Wijethunga — Thu, 13 Aug 2026 10:40:07 +0000

A spend cap that blocks every WhatsApp send when the budget runs out is a cap that will eventually block a login. The category that ran the budget dry is almost never the category that gets hurt: a marketing broadcast overshoots, the cap trips, and the next one-time code — costing a fraction of a cent — is refused. The customer standing at that moment sees an app that will not let them in, over a budget decision they were never part of. The fix is not a bigger budget. It is admitting that the four WhatsApp categories are not the same kind of traffic, and that only one of them deserves a hard stop.

This post is about that asymmetry as a deliberate design decision — including the part that looks inconsistent until you see why: the budget guard and the fraud guard sitting in the same send funnel with opposite failure modes, both correct.

Four categories, two kinds of traffic

Meta prices WhatsApp messages in four categories, and the pricing webhook tells you which one each send was charged under. From a budget's point of view they collapse into two groups:

Discretionary — MARKETING. Somebody chose to run this campaign. Stopping it mid-flight costs reach, not function. It is also the expensive category, routinely several times the price of the others, which is why it is the one that empties budgets.
Functional — AUTHENTICATION, UTILITY, SERVICE. Nobody chose these individually; the product emits them because a customer did something. A one-time code, a booking confirmation, a reply inside a service window. Each is cheap, and each not-sent is a user-visible failure.

A single cap treats those identically, which produces the worst trade available: it saves fractions of a cent on functional messages while the campaign that actually spent the money has already gone out. The arithmetic is lopsided in the extreme — blocking a full day of OTP traffic usually saves less than a hundredth of what one modest broadcast costs.

The policy, stated plainly

Category	At the cap	Approaching the cap
MARKETING	Hard block — sends refused	Warn at threshold
AUTHENTICATION	Warn, send anyway	Warn
UTILITY	Warn, send anyway	Warn
SERVICE	Warn, send anyway (it is free regardless)	Warn

Two refinements that earn their keep in practice:

An ALL budget exists for visibility and never blocks anything — not even marketing. If a total-spend cap could refuse a marketing send, then the answer to "why was this campaign stopped?" depends on two budgets instead of one, and the person operating the dashboard has to simulate the guard in their head. One category blocks, one budget per decision, and the refusal is always explainable in a sentence.

And the block message should say what to do, not just what happened. "Marketing spend cap reached — raise the budget to resume sends" turns a support escalation into a settings change.

Counting spend honestly

The guard is only as good as the number it compares against the cap, and three details decide whether that number is honest.

Use the reconciled cost when you have it, the estimate until then. Cost arrives in two stages — an estimate at send time, and Meta's authoritative billable-and-category verdict when the status webhook lands. The spend query prefers the second:

$spent = (float) MessageLog::query()
    ->where('direction', 'outbound')
    ->where('created_at', '>=', $windowStart)
    ->whereNot('status', 'failed')                         // failures are never billed
    ->where(DB::raw('COALESCE(category, expected_category)'), $budgetCategory)
    ->sum(DB::raw('COALESCE(cost_actual, cost_estimated)'));

Meta's category outranks yours. That first COALESCE is not decoration. Meta can reclassify a template after approval — utility to marketing is the common direction, at roughly six times the price. A guard that groups spend by the category you intended lets a reclassified template drain the marketing budget while being counted against utility, where nothing blocks. The webhook's verdict fills the category column; until it arrives, the send-time guess stands in.

Failures do not count. A message that failed was never billed. Summing it anyway makes the guard trip early, and on the marketing side an early block looks exactly like the feature working — nobody investigates a cap that fired.

Pin the window to the operating timezone

A daily budget resets at midnight. The only question is whose midnight, and the default answer — the application timezone, which on a stock deployment is UTC — is wrong in a way nobody notices until it fires.

For a product operating on Gulf time, a "daily" window keyed to UTC resets at 04:00 local. A cap that trips during the evening peak stays tripped through the next morning's peak too, then resets mid-morning. The window and the business day disagree by four hours, and every incident report about it reads as confusing until someone draws the timeline.

private function windowStart(string $period): CarbonInterface
{
    $tz = (string) config('messaging.timezone', 'Asia/Dubai');

    return $period === 'daily'
        ? now($tz)->startOfDay()->utc()
        : now($tz)->startOfMonth()->utc();
}

Compute the boundary in the operating timezone, convert to UTC, query in UTC. Storage stays uniform; the reset lands where the business thinks it does.

The zero that blocks everyone

One more counting rule, learned the painful way on a different budget system in the same codebase: a limit of zero means "not configured", never "block everything".

$budgets = Budget::query()
    ->where('is_active', true)
    ->whereNotNull('limit_amount')
    ->where('limit_amount', '>', 0)     // 0 = unconfigured, not "deny all"
    ->get();

The trap is mechanical: an unset config value casts to zero, and a guard comparing spent >= limit against zero blocks every send from the first one. On the marketing side that is an outage with a clear symptom. The subtle version is a seeded budget row someone zeroes "to disable it" — which, under naive comparison, does the opposite of disabling.

Two guards, opposite failure modes, same funnel

Here is the part that looks inconsistent. In the same send funnel:

The country allowlist fails closed — an empty list denies every send.
The budget guard fails open — if its evaluation throws (table missing mid-deploy, database hiccup), the send proceeds and the failure is logged.

public function check(?string $category): array
{
    try {
        return $this->evaluate($category !== null ? strtoupper($category) : null);
    } catch (Throwable $e) {
        // The guard protects money, not security. An OTP must not
        // die of a budget query.
        Log::error('Budget guard failed, allowing send', ['message' => $e->getMessage()]);

        return ['allowed' => true, 'blocking_budget' => null, 'warnings' => []];
    }
}

The asymmetry follows from what each guard protects. The allowlist guards against an adversary: failing open converts a config mistake into a payout endpoint, so it must not. The budget guards against overspend: failing closed converts a database hiccup into locked-out customers, so it must not. "Fail closed" is not a universal virtue — it is a question you answer per control, by asking which failure is worse. Write the answer as a comment on the catch block, because the next reviewer will flag whichever direction you chose as the inconsistent one.

Warn long before you block

A block with no warning phase teaches the operator that budgets are landmines. Each budget carries an alert threshold — 80 per cent by default — and crossing it logs a warning with the numbers in it while sends continue:

marketing monthly budget at 85% (424.15/500.00) — sends continue
authentication monthly budget exhausted (12.4/10.00) — sends continue

That second line is the asymmetry doing its quiet work: a functional category over its cap is a visible fact and an unstopped flow. The message says so explicitly, because a warning that reads like a block generates the same panic a block would.

Pin it with the test that matters

One test carries this whole design, and it is the one to write first: exhaust every budget, then prove an OTP still sends.

public function test_an_otp_is_never_blocked_by_any_budget(): void
{
    $this->setBudget('MARKETING', 1.0);
    $this->setBudget('AUTHENTICATION', 1.0);
    $this->setBudget('ALL', 1.0);
    $this->spend('AUTHENTICATION', 50.0);
    $this->spend('MARKETING', 50.0);

    $result = $this->sender->sendOtp('9715XXXXXXXX', '123456');

    $this->assertTrue($result['success']);
}

Its mirror — a marketing send refused at the cap, with the refusal written to the audit log — pins the other half. Between them they encode the policy in a place a refactor cannot quietly reverse it.

Check your own guard

Three questions. Does your spend cap distinguish categories, or does one bucket govern everything — and if one bucket, what happens to a login the day a campaign empties it? Does a zeroed or missing limit block traffic or admit it? And if the guard's own query throws, which way does it fail — and is that the direction you would choose on purpose?

The guard described here ships in laravel-whatsapp-cost-control (MIT, Laravel 12 and 13), wired into the same funnel as the allowlist and the cost ledger: caps per category per period, timezone-pinned windows, warn-then-block on marketing only, and the OTP test above in its suite. The defaults encode the asymmetry so that the first budget someone configures cannot accidentally become the one that locks customers out.

Originally published at dineshstack.com — read the full version with code samples and updates there.

OTP pumping: the fraud that bills you for every code you send

Dinesh Wijethunga — Thu, 13 Aug 2026 01:23:03 +0000

An unauthenticated endpoint that sends a one-time code is a payout mechanism for anybody who controls a block of premium-rate numbers. They trigger it in volume against numbers they profit from, and you pay per message. Rate limiting does not close this, because the attacker can vary everything your rate limiter keys on — IP address, phone number, timing — while the thing that actually earns them money stays fixed: the destination country. That is the control. A country allowlist that fails closed removes the economics of the attack rather than trying to out-run its volume.

The fraud is old and well documented on SMS, where it is usually called SMS pumping or artificially inflated traffic. Per-message WhatsApp billing brings the same economics to the Cloud API, with one difference worth noting up front: on SMS the money often flows through an aggregator who may eventually notice a strange pattern. On WhatsApp you are billed directly by Meta, per message, with no invoice arriving in time to warn you.

How the fraud actually pays

The attacker's revenue does not come from you. It comes from the termination fee paid to whoever operates the number range the message is delivered to. Control a range — or hold a revenue-share arrangement with an operator who does — and every message delivered into it earns a fraction of a cent.

Which produces a very specific attacker profile, and it is not the one most defences assume:

They do not want your data, your accounts, or your service. Nothing is breached.
They do not need to complete the flow. The code is never entered. The send is the entire transaction.
They are indifferent to which phone numbers they use, provided the numbers sit in a range that pays.
They are patient. Slow, steady traffic is better for them than a burst, because it survives longer.

That last point is the one that defeats most monitoring. There is no spike to alert on. The bill simply arrives larger than the month before, and the traffic looks like signups that never converted — which is a thing that happens anyway.

Why rate limiting is not the control

Rate limits are necessary. They are not sufficient, and it is worth being precise about why.

A rate limiter keys on something about the request — usually IP address, sometimes the authenticated user, occasionally the submitted phone number. Every one of those is attacker-controlled:

Per-IP is defeated by rotation. Residential proxy pools are cheap and large. Five requests a minute across a thousand addresses is five thousand requests a minute.
Per-phone-number is defeated by having more numbers. The attacker is choosing the numbers; a range holds thousands.
Per-user does not apply. The endpoint is unauthenticated. That is the point of it.

Worse, a rate limit is often looser than its author believes. On the platform this came from, the customer-facing send-code route declared five requests a minute and the driver-facing one had no route-level limit at all — it inherited only the generic API group, roughly sixty times looser, on an endpoint that was about to start billing per message. Both had passed review. The declared numbers were not the numbers being enforced either.

Keep the rate limits. Tighten them. Just do not mistake them for the ceiling on this particular fraud, because they bound the rate and the attacker is not in a hurry.

The control that works

The attacker needs the message delivered into a range that pays them. That range is in a country. If your product serves customers in three countries and your system refuses to send anywhere else, the attack has no revenue in it regardless of how many IPs or numbers they bring.

This is a much stronger position than rate limiting because it is not a race. It does not degrade under load, it does not need tuning, and it cannot be worn down by patience.

Fail closed, or it is not a control

The single most important property: an unset or empty allowlist must mean deny everything, never allow everything.

This sounds pedantic until you consider how the list actually gets emptied. A missing environment variable on a new server. A typo in a deploy. A config cache built before the key existed. In every one of those, the fail-open version silently converts a configuration mistake into an open payout endpoint, and nothing in your logs looks unusual because sends are succeeding.

$codes = array_values(array_filter(array_map(
    'trim',
    explode(',', (string) config('messaging.allowed_country_codes'))
)));

// A blank value must NOT mean "allow everywhere". A misconfigured
// environment falls back to the narrowest safe default, not the widest.
$this->allowedCountryCodes = $codes === [] ? ['971'] : $codes;

Fail-closed defaults are unpopular because they break things loudly during setup. That is the feature. The alternative breaks things quietly during an incident.

Put the check in the send funnel, not the controller

Every send in the system has to pass through it, which means it cannot live in a controller. Count the entry points on a mature codebase and there are always more than expected: the customer app, the driver app, an admin "resend code" button, a background job retrying a failed delivery, a console command someone wrote for testing.

A guard on four of five entry points is not a guard. Funnel every send through one method and check there:

private function send(string $operation, array $payload, ?string $category = null): array
{
    $to = $payload['to'];

    if (! $this->isAllowedDestination($to)) {
        return $this->blockedResult($operation, $payload);   // never reaches Meta
    }

    if (! $this->budget->allows($category)) {
        return $this->budgetBlockedResult($operation, $payload);
    }

    // ... timeout-bounded HTTP, audit log, cost ledger
}

The matching itself is deliberately dull — prefix comparison against normalised digits, with an explicit opt-out for the rare system that genuinely sends anywhere:

private function isAllowedDestination(string $phone): bool
{
    $digits = ltrim($phone, '+');

    if (in_array('*', $this->allowedCountryCodes, true)) {
        return true;   // explicit, never the default
    }

    foreach ($this->allowedCountryCodes as $code) {
        if (str_starts_with($digits, $code)) {
            return true;
        }
    }

    return false;
}

Make the refusal visible

A blocked send should be recorded as loudly as a failed one. Blocks are the earliest signal you will get that somebody is probing, and a control that silently discards traffic teaches you nothing about who is testing it.

The useful trick is to make the three outcomes distinguishable in one column. In the audit log, HTTP status encodes all of them:

Value	Meaning
a number	Meta answered with that status
`0`	Meta was unreachable
`null`	We refused before sending

A sudden rise in null rows for destinations outside your markets is the attack being attempted and stopped. Without that record it is invisible, which feels like safety and is actually just missing data.

Log the country and the last four digits. Do not log the whole number — you are storing the fraudster's data, but the same code path handles your customers, and the reason to keep the column narrow is that it never sees a distinction between them.

What this does not solve

Worth being straight about the limits, because a control oversold is a control someone will trust too far.

An allowlist does nothing about abuse from inside your own market. Somebody with a payout arrangement on a range in a country you legitimately serve is not blocked by any of this, and that is the case where per-number caps and velocity monitoring earn their place.

It does not help if your markets are genuinely global. A system that must send anywhere has to fall back on the weaker controls, and should expect to spend more on monitoring as a result.

And it is not a substitute for a spend cap. The allowlist bounds where money can go; it says nothing about how much. Those are separate questions and they want separate answers — a budget that hard-blocks a runaway campaign while never blocking a login is the other half, and it gets its own post.

Check your own endpoints

Three questions, in order of how much they will tell you.

First: can your send-code endpoint deliver to a country you do not sell in? Try it against a number outside your markets in a non-production environment. If the message goes, you have no allowlist.

Second: what happens with the setting removed entirely? Blank the config value and try again. A send that still succeeds means the implementation fails open, which is the failure mode that actually bites — the list is rarely wrong on purpose, it is empty by accident.

Third: how many entry points reach your sender? Grep for it and compare against where the guard lives:

grep -rn "sendTemplate|sendOtp|sendMessage" app/ Modules/ --include="*.php" | grep -v "Tests|/Messaging/"

Every result is a path that must pass the check. If the guard is in a controller and that list is longer than one, it is already being bypassed.

The funnel described here — allowlist, then budget, then a timeout-bounded call, then the audit log and cost ledger — is packaged as laravel-whatsapp-cost-control, MIT licensed, for Laravel 12 and 13. The allowlist ships fail-closed: it will refuse to send anywhere until you configure the countries you actually serve, which is a deliberately annoying five minutes.

Originally published at dineshstack.com — read the full version with code samples and updates there.

My CV Got Me UAE Tech Jobs Through Indeed for Three Years. Then It Stopped.

Dinesh Wijethunga — Thu, 13 Aug 2026 01:00:07 +0000

TL;DR: My CV brought in multiple UAE offers between 2022 and the end of 2024, every one of them through Indeed and none through LinkedIn. Then it stopped. It did not stop because it got worse or because the market collapsed. It stopped because I changed — seniority, specialism, the kind of role I was going for — and the document did not. And LinkedIn never worked at all, for a separate reason it took me years to see: I had a profile the whole time, but I was never findable.

This is the first of three posts. This one is what happened and why. The second is the mechanic underneath it — why job boards and LinkedIn are not two versions of the same thing. The third is the part nobody selling CV templates will tell you.

What actually happened

I moved from Sri Lanka to a tech job in the UAE. I am now a Tech Lead in Abu Dhabi with more than ten years of experience.

The window this post is about — 2022 to late 2024 — is entirely after that move. I was living and working in the UAE for all of it. That matters, because it removes the easiest explanation before we start.

Across those three years the same CV produced multiple UAE offers. Every single one came through Indeed. Not one came through LinkedIn, despite my having a LinkedIn profile the entire time.

I have friends here who get roles through LinkedIn and genuinely cannot explain how. Ask them what they did and the answer is some version of "a recruiter messaged me." That is not modesty. It is the whole point, and it took me a long time to understand why.

After December 2024 the same CV produced nothing.

Three reasons, and the first one is the least flattering

1. I stopped applying

I got a job in December 2024. My application volume collapsed. Some meaningful portion of "my CV stopped working" is simply that I stopped sending it.

I am putting this first because it is the one I would most like to skip. Every post in this genre blames the market, and blaming the market is comfortable — it makes the failure external and the solution purchasable. Before I tell you anything about market conditions, I want to be clear that a chunk of my own data is just reduced input. If you are drawing conclusions from your own search, run this check first. Fewer replies from a tenth as many applications is not a signal about your CV.

2. I crossed the seniority line

At mid-level, job boards work. The roles are posted, the volume is high, and the process is designed to filter a large inbound pile.

Lead and senior roles are mostly not filled that way. They go through networks and recruiter outreach. By 2025 I was applying for roles that largely are not advertised on the boards I was searching — and the ones that are tend to have already been filled through other routes by the time they appear.

The channel did not break. It aged out of my career stage.

3. The market tightened — but less than it feels, and not evenly

It is harder, and there is real data on it. LinkedIn research reported by Gulf News found that 65% of UAE professionals say finding a role has become harder over the past twelve months, while 72% plan to look for a new job anyway.

But look at the reason they gave. 63% named an overcrowded candidate pool as the biggest obstacle — not a shortage of roles. That is a different problem with a different fix. If jobs had vanished, nothing about your CV would matter. If you are one of far more applicants for the same jobs, then standing out is the entire game, and a document that makes you look like everyone else is an active liability rather than a neutral one.

Demand in tech also did not fall so much as move. PwC's 2026 Global AI Jobs Barometer puts UAE job postings requiring AI skills at 1.0% in 2021 and 3.2% in 2025 — roughly 4,600 adverts rising to 12,200, moving the UAE from 21st to 13th globally in four years. Those roles pay a premium of up to 92% in financial services and around 50% in technology, media and telecoms.

Read that last number again, because it is the whole argument for specialising. The market is not paying more for people who can do everything. It is paying up to 92% more for people who can demonstrably do one thing that is currently scarce.

My CV led with "full stack" and listed more than twenty technologies. In 2022 that read as range. By 2025 it reads as someone who has not decided what they are — in a market where the premium goes to people who have.

The part I did not see for two years

Those three causes look independent. They are not. Look at what changed between 2022 and now:

	2022	Late 2024 onward
Seniority	Mid-level	Tech Lead, 10+ years
Roles I was going for	Developer	Lead and senior
Positioning	Generalist	Production AI and LLM work
City	Dubai	Abu Dhabi
What my CV said	~8 years, full stack, 20+ technologies	~8 years, full stack, 20+ technologies

Read the last row again. Everything above it changed. That row did not.

That is the actual failure, and it is not a writing problem. The CV was a snapshot of the person who last needed one. Almost nobody updates a CV except while job hunting, which means everyone's CV describes the last version of them that was looking — and the gap gets wider the longer the job goes well.

Mine understated me in three separate ways at once. It said developer where the answer was Tech Lead. It said eight years where the answer was more than ten. And it said generalist where the honest answer had become a specialism in production AI systems — the kind of work I now write about in detail, like the bilingual Arabic and English voice assistant we ran at roughly 40ms latency.

The city row is a small one, but it is the same failure in a different artifact. I moved from Dubai to Abu Dhabi when I took the December 2024 job. Recruiter search filters by city. If a profile still says the city you left, you are absent from searches for the city you are actually in — and, like the CV, nothing tells you.

Why that last one matters more than it looks

There is a mechanic under all of this that took me far too long to work out, and it explains my friends.

Indeed and LinkedIn are not two places to find the same jobs. They run in opposite directions.

	Indeed	LinkedIn
Direction	You apply outward	Recruiters search inward
What you control	How many applications you send	Whether you are findable at all
How it fails	You get rejected	You are never seen

Recruiters do not browse LinkedIn the way you do. They use LinkedIn Recruiter, which is a search tool with dozens of filters running against a very large index of profiles. You either surface in the result set or you functionally do not exist for that search. There is no rejection, because there was never an application.

That is why my friends cannot explain what they did. They are not doing anything. They are being found. You cannot apply your way into an inbound channel, and no amount of application volume substitutes for being in the result set.

Now put my own three years against that. I was in the UAE the entire time. Both channels were open to me, the whole way through. Indeed produced offers for three years. LinkedIn produced nothing, ever — not fewer results, none.

Same person, same city, same experience, same week. One channel worked and the other never did once. That is not a market story and it is not bad luck. I was doing the work for one channel and none of the work for the other. On Indeed I was sending applications, which is the entire job on Indeed. On LinkedIn I had a profile and assumed that was participation. It is not. A profile is not a fishing line in the water; it is a page that either matches a recruiter's search or does not.

Mine did not. My headline said "full stack developer" — one of the most crowded search terms in the market, where I was competing with thousands of profiles saying exactly the same thing, and offering a recruiter no reason to pick mine out. I had never turned on the recruiter-facing "open to work" setting. My skills list was an afterthought. Every one of those is a filter I was failing without ever seeing a result.

On the outbound channel, effort looks like applications. On the inbound channel, effort looks like being findable. Doing a lot of the first has never once produced the second.

So there were two separate failures running at the same time, and I had been reading them as one. The CV went stale, which cost me the outbound channel as I moved up into roles that boards do not carry. And the profile was never findable, which meant the inbound channel — the one that actually serves senior roles — had never been switched on at all.

One more filter worth naming, because it does not apply to me but will apply to a lot of people reading this: those search filters include location. A recruiter hiring in Dubai searches Dubai. If you are still in Colombo or Chennai or Karachi, you are excluded before a word of your headline is read — which is why applying outward through job boards is the realistic route until you have arrived. I will cover the one setting that partly gets around it in the next post.

What this means if you are in the middle of it

Before you rewrite anything, work out which channel you are actually in, because the advice inverts.

If you are still outside the UAE, outbound is your game, and that is fine — it is the channel that got me here too. Volume matters. Your CV is doing the heavy lifting because it is the only artifact in the process. State your visa position explicitly rather than leaving a recruiter to assume the expensive answer, and say that you can relocate. Most LinkedIn optimisation will not reach you yet, for the location reason above, with one exception I will cover next.

If you are already here and applying, as I was for three years: you are running one channel out of two. That can work for a long time, exactly as it worked for me — right up until the roles you want stop being posted on it. Then it stops, and because there is no rejection to read, it feels like the market turned rather than like a channel you never opened.

If you are already here and senior, volume is not your problem — visibility is. The CV matters less than you think, because it is what you send after being found, and being found is a different skill with different levers.

And for everyone: open your CV and check the date on the claims, not the formatting. Does it say your current title? Your current years? The thing you are actually good at now, or the thing you were good at when you last needed a job? A stale CV fails quietly. There is no bounce, no rejection email, no signal at all — which is exactly why it can keep failing for a year without you noticing.

Check yours before you rewrite it

I built a free checker for this, because I could not find one that understood this market. It scores your CV against what Gulf recruiters actually filter on — visa status, notice period, formatting that survives an applicant tracking system, and whether you carry the keywords for the role you are targeting.

It runs entirely in your browser. Your CV is not uploaded anywhere, there is no signup, and the score and every fix are free.

Check your CV against the UAE market

One honest limit, the same one I put on the tool itself: this tells you whether your CV survives the filter. It cannot tell you whether the right roles are open. If nothing posted matches your experience, a perfect CV will not create one.

Next in this series: why you cannot apply your way into LinkedIn, what actually decides whether a recruiter's search returns you, and the one findability lever that works even before you have arrived.

How to migrate Laravel 13 + Next.js to Zero-Downtime VPS Releases

Dinesh Wijethunga — Wed, 12 Aug 2026 08:12:06 +0000

Part 9 of the CI/CD for Laravel Developers series.

Migrate Laravel 13 + Next.js to zero-downtime VPS releases in under 20 minutes — no downtime during the migration itself. This post is part of the DevOps series on deploying Laravel and Next.js with GitHub Actions. Most tutorials assume you're starting from scratch. This one assumes you already have a live project at /var/www/your-project and want to move it to the releases/symlink pattern so GitHub Actions can deploy atomically.

We'll migrate a monorepo with a Laravel 13 API and a Next.js 16 frontend running under Nginx and PM2 on a single Ubuntu VPS. The same steps apply to any project layout with minor path adjustments.

What Is the Releases Pattern and Why You Need It on VPS

In a normal VPS setup, your deployment overwrites files in the live directory. During that window — while composer install or npm ci runs — your app is in a broken state. Requests hit a mix of old PHP files and new ones, or a half-installed vendor directory.

The releases pattern solves this by keeping every deployment in its own timestamped directory. The live path is a symlink that points to the current release. When a new deploy finishes, you flip the symlink atomically — the switch takes microseconds and Nginx follows it instantly:

/var/www/visa-saas/

├── api -> releases/20260710143022    ← symlink (one atomic flip)

├── releases/

│   ├── initial/                      ← your existing app (backed up here)

│   ├── 20260710143022/               ← current live release

│   └── 20260711091544/               ← next release (building here)

├── shared/

│   ├── .env                          ← one .env, symlinked into every release

│   └── storage/                      ← persistent uploads and logs

└── web/                              ← Next.js (swap pattern, explained below)

The shared/ directory holds everything that must persist across releases: your .env file and the Laravel storage/ directory (uploads, logs, sessions). Each release symlinks to them instead of containing its own copy.

Before You Start: What Your Live VPS Looks Like Now

Typical existing layout — a git clone or manual upload, served directly by Nginx:

/var/www/visa-saas/

├── api/            ← Laravel 13 app (Nginx serves api/public)

├── web/            ← Next.js 16 (PM2 runs from this directory)

└── README.md

Nginx root currently points to /var/www/visa-saas/api/public and PM2 has cwd: /var/www/visa-saas/web. After the migration, Nginx will point to the same path — but api will be a symlink instead of a directory, and Nginx follows symlinks transparently with no config change.

Step 1: Pause the Queue Worker Before Touching VPS Files

Signal any queue workers to stop picking up new jobs before touching files. They'll finish their current job and exit cleanly:

cd /var/www/visa-saas/api

php artisan queue:restart      # signals workers to exit after current job

If you're running Laravel Horizon, use php artisan horizon:pause instead. If you have no queue workers, skip this step.

Step 2: Create the Releases and Shared Directory Structure on VPS

Create the releases/ and shared/ directories alongside your existing api/ directory:

cd /var/www/visa-saas

mkdir -p releases mkdir -p shared/storage/app/public mkdir -p shared/storage/framework/{cache/data,sessions,views} mkdir -p shared/storage/logs

Step 3: Back Up Your Live Laravel 13 App as the Initial Release

Your current live app becomes the first named release. Nothing is deleted — this is a copy, not a move:

cp -a api releases/initial

cp -a (archive mode) preserves ownership, permissions, and symlinks. Your original api/ directory stays intact as a safety net until you've confirmed everything works through the symlink.

Step 4: Move the Laravel .env and Storage to the Shared Directory

The .env file and storage/ directory must live outside every release so they survive across deployments.

Move .env:

cp releases/initial/.env shared/.env

Move storage contents:

# Copy contents (not the directory itself) into shared/storage


cp -a releases/initial/storage/. shared/storage/


  
  
  Verify


ls shared/storage/



  

  

  app  framework  logs

Fix ownership so PHP-FPM can write to the shared storage:

sudo chown -R www-data:www-data shared/storage

sudo chmod -R 775 shared/storage

Step 5: Symlink Shared Resources Into the Laravel Release

Wire releases/initial to use the shared files instead of its own copies:

cd /var/www/visa-saas/releases/initial


  
  
  Remove the copied storage and .env


rm -rf storage

rm -f .env


  
  
  Symlink to shared — always use absolute paths


ln -sfn /var/www/visa-saas/shared/storage storage

ln -sfn /var/www/visa-saas/shared/.env .env


  
  
  Verify


ls -la storage .env


  
  
  .env -> /var/www/visa-saas/shared/.env




  

  

  storage -> /var/www/visa-saas/shared/storage

Step 6: Replace the api/ Directory With a Symlink on VPS

Back in the project root, rename the original api/ directory to a backup and create the symlink:

cd /var/www/visa-saas


  
  
  Rename the original (keep as backup until confirmed working)


mv api api-backup


  
  
  Create the symlink


ln -sfn /var/www/visa-saas/releases/initial api


  
  
  Verify


ls -la api



  

  

  api -> /var/www/visa-saas/releases/initial

Step 7: Verify Nginx Follows the Symlink to Laravel public/

Nginx's root /var/www/visa-saas/api/public directive resolves through the symlink automatically. Test the config and reload:

sudo nginx -t nginx: configuration file /etc/nginx/nginx.conf syntax is OK

sudo systemctl reload nginx

Hit your API health check to confirm:

curl -s https://api-visa-recruiter.orions360.com/up




  

  

  {"status":"ok"}

If you get a 502 or permission denied, PHP-FPM may need explicit symlink permission. Add this to your Nginx server block:

server {

root /var/www/visa-saas/api/public;

location / {
    try_files $uri $uri/ /index.php?$query_string;
    disable_symlinks off;
}


}

Most Ubuntu + Nginx setups follow symlinks by default. disable_symlinks off is only needed if your distro explicitly enabled the restriction.

Step 8: Remove the Backup Once the VPS Migration Is Confirmed

With the app confirmed working through the symlink, remove the backup directory:

rm -rf /var/www/visa-saas/api-backup

Step 9: Update Your GitHub Actions Deploy Script for Zero-Downtime Releases

Now that the structure is in place on the VPS, your GitHub Actions SSH deploy script can use the full releases pattern. Each deploy creates a new timestamped directory, installs dependencies, warms caches, then flips the symlink atomically:

- name: Run release on server uses: appleboy/ssh-action@v1 with: host: ${{ secrets.VPS_HOST }} username: ${{ secrets.VPS_USER }} key: ${{ secrets.VPS_SSH_KEY }} script: | set -euo pipefail DEPLOY=/var/www/visa-saas RELEASE="$DEPLOY/releases/$(date +%Y%m%d%H%M%S)" SHARED="$DEPLOY/shared" # 1. Extract into a new timestamped release directory mkdir -p "$RELEASE" tar -xzf /tmp/api-release.tar.gz -C "$RELEASE" # 2. Wire shared .env and storage cd "$RELEASE" rm -rf storage ln -sfn "$SHARED/storage" storage ln -sfn "$SHARED/.env" .env # 3. Install and warm caches composer install --no-dev --optimize-autoloader --no-interaction --quiet php artisan migrate --force php artisan config:cache php artisan route:cache mkdir -p resources/views && php artisan view:cache php artisan event:cache php artisan queue:restart || true # 4. Atomic symlink flip — zero downtime ln -sfn "$RELEASE" "$DEPLOY/api" # 5. Reload PHP-FPM to clear opcache sudo systemctl reload php8.4-fpm # 6. Keep only the last 5 releases ls -1dt "$DEPLOY/releases/"* | tail -n +6 | xargs rm -rf || true rm -f /tmp/api-release.tar.gz echo "✓ API deployed: $RELEASE"</code></pre><p>The atomic flip on step 4 is the key line: <code>ln -sfn "$RELEASE" "$DEPLOY/api"</code> replaces the symlink in a single filesystem operation. Nginx reads the new target on the very next request — no reload, no downtime.</p><h2>How Next.js 16 Migration Works Differently on VPS</h2><p>The releases/symlink pattern works perfectly for Laravel 13 because PHP reads files on every request — the new symlink target takes effect immediately. Next.js 16 running under PM2 is different: PM2 holds the process in memory with a fixed <code>cwd</code>. Flipping a symlink doesn't cause PM2 to reload its running process.</p><p>For Next.js we use a <strong>staging swap</strong> instead — extract to a staging directory, install deps, then atomically rename it into place:</p><pre><code class="language-bash">BASE=/var/www/visa-saas STAGING="$BASE/web-staging" Extract to staging rm -rf "$STAGING" && mkdir -p "$STAGING" tar -xzf /tmp/web-release.tar.gz -C "$STAGING" ln -sfn "$BASE/shared/.env.local" "$STAGING/.env.local" Install production deps (ignore husky and other dev lifecycle scripts) cd "$STAGING" npm ci --omit=dev --ignore-scripts Swap: remove old web/, rename staging to web/ rm -rf "$BASE/web" mv "$STAGING" "$BASE/web" Restart PM2 from the new web/ directory

cd "$BASE/web" pm2 delete visa-saas 2>/dev/null || true pm2 start ecosystem.config.js pm2 save

This causes ~1–2 seconds of PM2 downtime during the restart. For true zero-downtime Next.js deploys, apply the symlink pattern here too — but PM2's ecosystem.config.js must point cwd to a symlink path that you flip, not the real directory path.

Rollback to a Previous Laravel Release in One Command

The main benefit of keeping old releases on the VPS: if a deploy breaks production, rollback is a single symlink change — no re-deploy, no composer install, no migration:

# List releases newest first ls -1dt /var/www/visa-saas/releases/* Roll back to the previous release ln -sfn /var/www/visa-saas/releases/20260709091544 /var/www/visa-saas/api sudo systemctl reload php8.4-fpm

echo "✓ Rolled back"

The previous release directory is already fully installed. Rollback takes seconds.

Common Issues During VPS Migration

Nginx 403 after creating the symlink: The deploy user owns releases/initial/ but www-data needs read access. Run sudo chown -R <logged_in_user_name>:www-data /var/www/visa-saas/releases and chmod -R 750 releases/.

Laravel 13 can't write to storage after migration: The shared/storage/ directory must be owned by www-data. Run sudo chown -R www-data:www-data /var/www/visa-saas/shared/storage.

php artisan commands can't find .env: The symlink in releases/initial must use the full absolute path (/var/www/visa-saas/shared/.env), not a relative path. Relative symlinks break when you cd into the release directory.

Opcache serving stale PHP after the symlink flip: PHP's opcache caches the resolved real path, so it continues serving old bytecode after the symlink changes. Always reload PHP-FPM immediately after the symlink flip: sudo systemctl reload php8.4-fpm.

Verify the Final VPS Structure

After your first automated deploy through GitHub Actions, confirm the structure looks exactly like this:

ls -la /var/www/visa-saas/


  
  
  api -> /var/www/visa-saas/releases/20260710143022



  
  
  releases/



  
  
  initial/



  
  
  20260710143022/



  
  
  shared/



  
  
  .env



  
  
  storage/



  
  
  web/


ls -la /var/www/visa-saas/releases/20260710143022/


  
  
  .env -> /var/www/visa-saas/shared/.env



  
  
  storage -> /var/www/visa-saas/shared/storage



  
  
  vendor/



  
  
  app/




  

  

  ...all Laravel 13 files

WhatsApp Cloud API sends you no invoice until it is too late

Dinesh Wijethunga — Tue, 11 Aug 2026 19:50:07 +0000

Meta does not send you a WhatsApp invoice you can act on in time. Billing is per message, priced by template category and recipient country, and the rate card changes on Meta's schedule rather than yours. The API response to a send contains a message id and no price. The delivery-status webhook carries a pricing object — and that has no amount in it either. What you get is a billable flag and a category, which you multiply against a rate card you maintain yourself. If you are not capturing that webhook, you have no per-message cost data and no way to reconstruct it, because Meta exposes no per-message billing history through the API.

This came out of a booking platform where WhatsApp carried one-time codes. The webhook receiver already existed, validated Meta's signature correctly, and threw the entire payload away. Nobody noticed, because nothing about the system looked broken — it simply could not answer what a message cost.

What Meta actually charges for

WhatsApp Business Platform moved from per-conversation to per-message billing on 1 July 2025. Every template message is priced individually along two axes:

Category — marketing, utility, authentication or service. Marketing is by far the most expensive; service messages are free.
Recipient country — the same template costs different amounts depending on where the handset is.

Two further wrinkles matter more than they first appear. Meta re-prices on its own schedule, so a rate you hardcoded is a rate that will silently go stale. And utility templates sent inside an open 24-hour customer service window are not charged at all — meaning identical traffic can cost different amounts depending on whether the customer happened to message you first.

None of that is visible from the sending API.

What the send response does not contain

Post a template message to the Cloud API and this comes back:

{
  "messaging_product": "whatsapp",
  "contacts": [ { "input": "9715XXXXXXXX", "wa_id": "9715XXXXXXXX" } ],
  "messages": [ { "id": "wamid.HBgMOTcx...FQIAERgSODQ4..." } ]
}

That is the whole response. It acknowledges acceptance — not delivery, and certainly not billing. At this point you do not know whether the message will arrive, whether it will be charged, or which category it will be charged under. The last one is not rhetorical: Meta can reclassify a template after approving it.

Where the price actually comes from

Subscribe to the messages webhook field and each outbound message produces a series of status callbacks. The sent status carries billing:

{
  "entry": [ { "changes": [ { "field": "messages", "value": {
    "statuses": [ {
      "id": "wamid.HBgMOTcx...",
      "status": "sent",
      "recipient_id": "9715XXXXXXXX",
      "pricing": {
        "billable": true,
        "pricing_model": "PMP",
        "category": "utility",
        "type": "regular"
      }
    } ]
  } } ] } ]
}

Four fields matter:

billable — whether Meta is charging for this message at all.
pricing_model — PMP is per-message pricing. Anything else means the account has not moved to the current model and the arithmetic here does not apply.
category — Meta's classification, not the one you submitted.
type — regular for a charged message, free_customer_service for one that landed inside an open service window.

Notice what is missing: no price, no currency, no running total. Meta tells you a message is chargeable and how it is classified. Turning that into money is your problem.

So cost is derived, not reported

Cost is billable AND category AND country, resolved against a rate card you own. That produces two numbers with different confidence, and collapsing them into one is the mistake to avoid:

Estimated — computed at send time from the category you intended. Instant, occasionally wrong.
Actual — computed when the webhook lands, from Meta's own billable flag and category. Authoritative on whether and which; still dependent on your rate card for how much.

Report both and label them. A dashboard showing a single number called "cost" is quietly wrong in the gap between sending and the webhook arriving — which, on a marketing broadcast, is exactly the window somebody is watching.

The rate card has to be effective-dated

Because Meta re-prices, a rate needs a start date and past rows must never be edited. Otherwise correcting today's price silently rewrites what last quarter cost:

Schema::create('message_rates', function (Blueprint $table) {
    $table->id();
    $table->string('country_code', 5);   // calling code, e.g. 971
    $table->string('category', 20);      // MARKETING | UTILITY | AUTHENTICATION | SERVICE
    $table->decimal('price_usd', 10, 6)->nullable();
    $table->decimal('price_local', 10, 6);
    $table->date('effective_from');
    $table->timestamps();

    $table->unique(['country_code', 'category', 'effective_from']);
});

Resolution is the newest row whose effective_from is on or before the message date:

public function rateFor(?string $country, ?string $category, ?CarbonInterface $on = null): ?float
{
    if ($country === null || $category === null) {
        return null;
    }

    $rate = MessageRate::query()
        ->where('country_code', $country)
        ->where('category', strtoupper($category))
        ->where('effective_from', '<=', ($on ?? now())->toDateString())
        ->orderByDesc('effective_from')
        ->first();

    return $rate?->price_local === null ? null : (float) $rate->price_local;
}

One caution on populating it: reseller rate summaries are convenient and are not Meta's rate card. Take the numbers from your own account in Business Manager and treat any figure copied from a blog post — including this one — as provisional until you have checked it there.

Visibly unknown, never silently wrong

That method returns null, not zero, when the card cannot price something. The distinction is the whole design. A billable message your rate card does not cover is a real hole in the total, and a zero hides it perfectly — you get a number that looks complete and is not.

'missing_rate' => MessageLog::where('billable', true)
    ->whereNull('cost_actual')
    ->count(),

Put that count on the dashboard beside the total. Anything above zero means the figure next to it is understated and somebody needs to add a rate row. A cost report that cannot tell you how much it does not know is not a cost report.

Recording the send

Open the ledger row when Meta accepts the message, keyed on the returned id:

$wamid = $response->json('messages.0.id');

if ($wamid !== null) {
    MessageLog::firstOrCreate(['wamid' => $wamid], [
        'direction'         => 'outbound',
        'template_name'     => $payload['template']['name'] ?? null,
        'expected_category' => $expectedCategory,   // what WE think it is
        'country_code'      => $country,
        'recipient_hash'    => hash('sha256', $payload['to']),
        'recipient_last4'   => substr($payload['to'], -4),
        'status'            => 'accepted',
        'cost_estimated'    => $rateCard->rateFor($country, $expectedCategory),
    ]);
}

Two decisions worth copying. A failed send has no message id and no cost, so only accepted messages get a row and there is nothing to reconcile later. And the recipient is a hash plus last four digits — enough to answer "did this customer get their code?" without a phone number existing anywhere in the reporting tables.

Reconciling from the webhook

The handler stays idempotent, keeps the lifecycle honest, and captures pricing:

// Meta redelivers, and a status can arrive for a message this system never
// sent — another tool holding the same token. A skeleton row keeps its cost
// data rather than discarding it.
$message = MessageLog::firstOrCreate(
    ['wamid' => $status['id']],
    ['direction' => 'outbound', 'status' => 'accepted']
);

$updates = [];

// Statuses arrive out of order — a read can beat the sent it follows.
// Only ever move forward through the lifecycle.
$rank = ['accepted' => 0, 'sent' => 1, 'delivered' => 2, 'read' => 3];
$state = $status['status'];

if ($state !== 'failed' && isset($rank[$state])
    && $rank[$state] > ($rank[$message->status] ?? 0)) {
    $updates['status'] = $state;
}

if (isset($status['pricing'])) {
    $pricing = $status['pricing'];

    $updates['billable']      = $pricing['billable'] ?? null;
    $updates['pricing_model'] = $pricing['pricing_model'] ?? null;
    $updates['pricing_type']  = $pricing['type'] ?? null;
    $updates['category']      = isset($pricing['category'])
        ? strtoupper($pricing['category'])
        : null;

    $updates['cost_actual'] = ($pricing['billable'] ?? null) === false
        ? 0
        : $rateCard->rateFor($message->country_code, $pricing['category'] ?? null);
}

$message->update($updates);

The monotonic status check is not defensive habit. Meta delivers statuses over separate HTTP calls with independent retries, so a retried sent can land after the delivered it precedes. Without the rank comparison a message that reached the customer gets demoted, and the delivery rate under-reports for reasons nobody can trace.

Three things that will make your totals wrong

Counting at send instead of at status

A failed message is never billed. Summing rows at creation time inflates the total by every failure. Sum the ledger with failures excluded and let the webhook decide what counts.

Trusting your own category

Meta can reclassify a template after approval, and utility to marketing is roughly a sixfold increase. This is why expected_category and category are separate columns rather than one overwriting the other — the disagreement is the signal:

MessageLog::whereNotNull('category')
    ->whereColumn('category', '!=', 'expected_category')
    ->selectRaw('template_name, category, count(*) as messages')
    ->groupBy('template_name', 'category')
    ->get();

Meta also fires a message_template_category_update webhook when it does this. Subscribe and alert — it is the only notice you get, and the alternative is finding out from a bill.

Assuming the free window is rare

Utility messages sent inside an open 24-hour service window cost nothing, and the webhook reports it as type: free_customer_service. Treat those as a first-class outcome rather than an anomaly, or a month where costs fell because customers happened to message first will look like a reporting bug.

What the ledger then makes possible

Cost data is not the goal. It is the precondition for two controls that cannot exist without it.

The first is a spend cap that knows the difference between kinds of traffic. A budget that blocks every send at its limit will eventually block a login, which is why marketing deserves a hard stop and authentication never does — a campaign that pauses is the feature working, an undelivered one-time code is a locked-out customer.

The second is fraud. An unauthenticated endpoint that sends a message costs money per request, which makes it a payout mechanism for anyone with premium-rate numbers to point it at. Rate limiting alone does not close that, and the ceiling turns out to be a country allowlist that fails closed.

Both get their own posts. Both are only enforceable once you know what a message costs.

What it looks like when it works

One template message, watched end to end on a live account:

Moment	State	Estimated	Actual
Send accepted	accepted	0.057658	pending
sent webhook (t+0s)	billable, PMP, regular, UTILITY	0.057658	0.057658
delivered webhook (t+2s)	delivered	0.057658	0.057658

Two seconds from send to delivered, pricing confirmed on the first status. The estimate matched because the category we intended was the one Meta charged — the outcome you want, and precisely the thing you cannot assume without recording both.

Check your own account

Two questions, both answerable in a minute.

Are you receiving status webhooks at all? Send one message and look for a row with a non-null billable. If it is still null after a minute, your callback URL is not receiving them and every cost figure you hold is an estimate wearing a confident label.

Is your billing model current? Any pricing_model other than PMP means the account is on an older model.

php artisan tinker --execute='
$m = MessageLog::latest()->first();
echo ($m->billable === null
    ? "NO PRICING WEBHOOK RECEIVED"
    : "model={$m->pricing_model} type={$m->pricing_type}").PHP_EOL;
'

And if these messages are triggered by an unauthenticated endpoint — a one-time code, a password reset — the rate limit in front of it is not a performance control, it is the spend cap. Worth confirming it admits the number it claims to, because in Laravel it frequently does not.

The ledger, the effective-dated rate card and the fail-closed country allowlist described here are packaged as laravel-whatsapp-cost-control, MIT licensed, for Laravel 12 and 13. Sending is already well served by existing packages; this one covers the part that decides what the sending costs.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Manual Capture in Production: Holds, Buffers, Split Payments, and the Seven-Day Clock

Dinesh Wijethunga — Mon, 10 Aug 2026 19:10:06 +0000

Manual capture means the number you authorize and the number you capture are different numbers, and the gap between them is where production bugs live. The rules that matter: you may capture less than you authorized (the remainder releases automatically), you may never capture more, the hold dies on its own after about seven days, and a card portion that a split payment pushes below Stripe's per-currency minimum fails with amount_too_small. Everything in this post is a consequence of those four facts, learned on a platform where the final amount almost never matches the estimate.

This is Part 4 of the Stripe in Production series — the flow is the one from Part 1: authorize at checkout, capture on fulfilment, void on cancellation. Part 3 covered what happens when cancellation races the authorization; this part assumes the order survived.

Authorize the estimate plus a buffer — on the card only

If you authorize exactly the estimate, every order whose final total runs slightly over diverts into a second payment step — the exact friction manual capture exists to avoid. So authorize with headroom:

// GOOD: buffer absorbs ordinary variance; only the excess needs a second payment
public static function withBuffer(float $cardPortion, ?float $bufferPercent): float
{
    // Negative or null => no buffer. Never authorize LESS than the
    // estimate — that guarantees a second payment on every order.
    $percent = max(0.0, (float) ($bufferPercent ?? 0));

    return round($cardPortion * (1 + $percent / 100), 2);
}

The subtlety that cost us a support queue: apply the buffer to the card portion only, never to a store-credit lock. A card hold with headroom inconveniences nobody; a credit lock with headroom blocks the customer whose balance exactly covers their order — the system tells them they cannot afford a thing they can afford.

The split that dips under the minimum

Store credit plus card is a normal split until the credit side eats almost everything and leaves the card 0.30 USD — below the floor from Part 1's Mistake #5. Handle it at split time, not at API-error time:

// GOOD: rebalance the split before Stripe ever sees it
$minimum = config('payment.stripe.minimums.'.strtoupper($currency)); // e.g. 0.50 USD

if ($cardPortion > 0 && $cardPortion < $minimum) {
    $shortfall = round($minimum - $cardPortion, 2);

    if ($creditPortion >= $shortfall) {
        // Lift the card to the floor by giving credit back to the customer
        $creditPortion = round($creditPortion - $shortfall, 2);
        $cardPortion   = $minimum;
    } else {
        throw new PaymentProviderException('stripe', 'Split below card minimum', [],
            null, customerMessage: "The minimum card payment is {$minimum} {$currency}.");
    }
}

Rebalancing in the customer's favour (they keep more credit for later, the card carries the floor) turns an error path into a silent adjustment. The error branch survives only for the customer with almost no credit at all.

Capture day: three outcomes, three different calls

// GOOD: fulfilment settles the hold — final vs authorized decides the call
$creditUsed  = min($hold->credit_amount, $finalTotal);
$cardPortion = max(0.0, $finalTotal - $creditUsed);

if ($cardPortion <= 0) {
    // Credit covered everything: capturing zero is an error — VOID instead,
    // releasing the customer's card hold immediately.
    $stripe->paymentIntents->cancel($hold->capture_reference);
} elseif ($cardPortion <= $hold->authorized_amount) {
    // The normal case, including partial capture — remainder auto-releases.
    $stripe->paymentIntents->capture($hold->capture_reference, [
        'amount_to_capture' => $this->toMinorUnits($cardPortion, $currency),
    ], ['idempotency_key' => "capture_{$order->id}"]);
} else {
    // Final exceeded even the buffer: capture what the hold allows,
    // move the order to balance-due for the excess. Stripe will not
    // stretch a hold — do not retry the capture with a bigger number.
    $stripe->paymentIntents->capture($hold->capture_reference, [
        'amount_to_capture' => $this->toMinorUnits($hold->authorized_amount, $currency),
    ], ['idempotency_key' => "capture_{$order->id}"]);
    $order->transitionTo(OrderStatus::BalanceDue, $finalTotal - $hold->authorized_amount);
}

Three notes. The idempotency key means a retried fulfilment job cannot double-capture. The zero-card branch voids rather than captures — a zero capture is an API error, and the void is also the kind gesture, releasing the hold now instead of in seven days. And the balance-due branch is where the minimum guard matters again: the excess is a delta, exactly the kind of small amount that dips under the floor.

The seven-day clock

An uncaptured authorization expires on its own after about seven days, and expiry is silent — the capture just fails when you finally call it. Two design consequences: if your fulfilment window can exceed the auth window, you need a re-authorization step in the flow, not a longer hope; and the expiry is also your last-resort cleanup, which is why Part 1's reconciliation hunts for holds older than an hour — anything you find at day six was a bug for six days.

A trick the hold enables: the micro-authorization

Manual capture gives you a free card validity check: authorize the per-currency minimum, then void it immediately. No charge, no capture, a held-and-released minimum — and you learn the card is real and funded before the order commits to it. Tag these distinctly in metadata so your webhook handlers skip them (they have no order behind them), and remember the floor: the probe is the minimum, not a symbolic 0.01.

The checklist

Authorize estimate + buffer; buffer the card portion only, never store credit
Rebalance splits before the API call: lift the card to the minimum from the credit side
Capture ≤ authorized; excess goes to a balance-due payment, never a bigger capture
Zero card portion => void, not a zero capture
Idempotency key on every capture
Fulfilment longer than the auth window => re-authorize by design

Next in this series

Part 5 is the layer under all of this: payment webhooks that don't lie — replay guards, why you mark a webhook processed only after success, and the difference between delivered and true. [LINK WHEN LIVE: /en/payment-webhook-replay-reconciliation]

Cheap audit before then: find your capture call and check what happens when the final amount exceeds the authorization. If the answer is "we capture the final amount", that call has been failing on every over-estimate order and something downstream is eating the error.

I post each part natively on LinkedIn with the story that didn't fit — follow there for Part 5. I read every reply.

The Authorize/Cancel Race: When a Customer Pays for an Order That No Longer Exists

Dinesh Wijethunga — Mon, 10 Aug 2026 16:32:10 +0000

Cancelling an order in your database does nothing at Stripe. The client_secret you shipped to the payment sheet is a live capability — it keeps accepting confirmation after your order row says cancelled, and the authorization webhook will happily record money against an order that no longer exists. The race runs in both directions: the customer can pay after you cancel, and you can cancel after the customer pays, with the second gap measured at two seconds in our production logs. The fix is symmetrical: the cancel path must kill the intent at Stripe, and the confirm path must check the parent order before accepting the money.

This is Part 3 of the Stripe in Production series. It assumes the manual-capture flow from Part 1 — authorize at checkout, capture on fulfilment, void on cancellation.

Two races, one root

Both incidents that taught us this were mundane. No load, no outage — just ordinary latency between a phone and two servers.

RACE A — cancel, then pay (the stale sheet)
T+0s    customer opens payment sheet, closes the app
T+15m   sweeper expires the session, cancels the order
T+16m   customer reopens the app; the sheet is still on screen
        client_secret still valid at Stripe => hold authorized
        => money locked against an order that died a minute ago

RACE B — pay, then cancel (two seconds)
T+0.0s  customer confirms in the sheet
T+1.5s  authorization webhook lands; hold recorded
T+3.5s  order auto-cancels (no availability)
        => release path runs immediately — into whatever
           state the webhook just wrote

One root: two systems each believed their own state was the whole truth. Your database is not Stripe's state, and Stripe is not yours. Every fix below is one side refusing to trust the other blindly.

A client_secret is a capability, not a record

Treat every client_secret you ship as an outstanding key to your customer's card. It lives on the customer's device, outside your transaction boundaries, and it does not check your order table before working. Deleting the order, cancelling it, even deleting the session row — none of it revokes the key. Only two things do: explicit cancellation of the PaymentIntent, or its natural expiry. Once you see it that way, the rule writes itself: whoever invalidates the order must also revoke the key.

Fix the cancel side: kill the intent when the session dies

// BAD: local-only decline — the client_secret out there still works
public function decline(string $provider, string $orderRef): void
{
    $session = CheckoutSession::where('provider', $provider)
        ->where('merchant_order_id', $orderRef)->first();

    $session?->update(['status' => CheckoutSessionStatus::Declined]);
    // ... cancel order, release hold — Stripe never hears about any of it
}

// GOOD: revoke the key at Stripe FIRST, then record the decline
public function decline(string $provider, string $orderRef, string $reason): void
{
    $session = CheckoutSession::where('provider', $provider)
        ->where('merchant_order_id', $orderRef)->first();

    if (! $session || $session->status === CheckoutSessionStatus::Declined) {
        return; // idempotent — sweeper and webhook may both call this
    }

    // HTTP deliberately OUTSIDE the transaction below, and best-effort:
    // an unreachable Stripe must never block the order cancellation.
    // Ordered FIRST so a failure that rolls back the transaction leaves
    // the session pending — and the next sweep retries the cancel.
    // After the commit, the session is Declined and never revisited.
    if ($provider === 'stripe' && str_starts_with((string) $session->provider_payment_id, 'pi_')) {
        try {
            $this->stripe->paymentIntents->cancel($session->provider_payment_id);
        } catch (\Throwable $e) {
            Log::warning('Could not cancel intent on decline', [
                'intent' => $session->provider_payment_id,
                'error'  => $e->getMessage(),
            ]);
        }
    }

    DB::transaction(function () use ($session, $reason) {
        $session->update(['status' => CheckoutSessionStatus::Declined]);
        $this->holds->release($session->hold);
        $this->orders->cancel($session->order, $reason);
    });
}

The ordering carries the retry semantics, so it is not style — cancel-at-Stripe first means a crashed decline self-heals on the next sweep, while the reverse order can mark the session dead with the key still live. After this shipped, a reopened stale sheet fails at confirmation instead of taking the customer's money. That failure is the feature.

Fix the confirm side: check the parent before accepting money

Race B needs the mirror-image guard. Your authorization webhook handler was written imagining a live order — but it can fire after cancellation, and recording a hold nobody will ever settle creates the exact stranded-money problem from Part 1.

// GOOD: the webhook confirms into the world as it is, not as it was
public function confirm(string $provider, string $orderRef): void
{
    $session = CheckoutSession::where('provider', $provider)
        ->where('merchant_order_id', $orderRef)->firstOrFail();

    if ($session->status === CheckoutSessionStatus::Authorized) {
        return; // idempotent — webhooks redeliver
    }

    $order = $session->order;

    // The order died while the customer was typing card details.
    // Accepting the authorization would strand the hold — refuse it,
    // and release the money we were just handed.
    if ($order->status->isTerminal()) {
        try {
            $this->stripe->paymentIntents->cancel($session->provider_payment_id);
        } catch (\Throwable $e) {
            Log::critical('Late authorization on dead order — cancel failed', [
                'intent' => $session->provider_payment_id,
                'order'  => $order->id,
            ]);
        }
        $session->update(['status' => CheckoutSessionStatus::Declined]);
        return;
    }

    // Normal path: record the authorization, advance the order.
    // ...
}

Note the log level on the failure branch: a late authorization on a dead order that you also failed to cancel is precisely the case your reconciliation must page a human about.

Why manual capture saved us

Both races happened with capture_method: manual, so the exposure was a temporary hold — released in days even if everything else failed — rather than a captured charge needing an apologetic refund. That is worth stating as a design principle: for any order that can be cancelled after payment begins, authorize-then-capture bounds your worst case. Automatic capture in the same races takes real money and turns an engineering bug into a support incident. Still, "it expires in a week" is not a fix; a customer watching held funds for days does not care about your capture method.

The backstop you already have

Every guard above can fail — Stripe unreachable on both attempts, a crash between webhook and handler. This is what the hourly requires_capture reconciliation from Part 1 is for: any hold older than an hour whose local record claims finished is a bug being found the same day instead of at auth expiry. The races make it necessary; the reconciliation makes them survivable.

The checklist

Treat every shipped client_secret as a live capability until explicitly cancelled
Cancel the PaymentIntent at Stripe when a session dies — before recording the decline, best-effort, retried by the sweep
Make decline and confirm both idempotent; sweeper and webhooks will overlap
In the authorization handler, check the parent order's state; void immediately on terminal orders
Use manual capture wherever cancellation can race payment
Keep the hourly reconciliation running — it catches what the guards miss

Next in this series

Part 4 goes deeper into the hold itself: manual capture in production — authorization buffers over estimates, split wallet-and-card payments where the card portion dips under Stripe's minimum, and the seven-day expiry as a deadline you design against. [LINK WHEN LIVE: /en/stripe-manual-capture-holds-split-payments]

Before then, one cheap audit: grep your cancellation paths for a Stripe cancel call. If cancelling an order only touches your own tables, every client_secret you have ever shipped is still out there, and Race A is not a possibility — it is a schedule.

I post each part natively on LinkedIn with the war story that didn't fit — follow there for Part 4, or tell me about the race you lost. I read all of them. Part 2, on the wallet gates that block payment sheet buttons, is here.