Sid Ngeth's Blog

the second opinion needs to read the code

2026-03-29T00:00:00+00:00

the premise
the experiment
run 1: the reviewer can’t read the code
run 2: the reviewer gets equal tools
what changed
what the literature missed
what this means

the premise

developers get stuck in single-model ruts. you ask Claude to fix a bug, rephrase three times, get variations of the same wrong answer. so you switch to GPT. different perspective, maybe better.

this works often enough that developers keep doing it. so i asked: what if the second model explicitly reviewed the first model’s fix and pointed out what’s wrong, then the first model revised? i built this and tested it with ground truth – SWE-bench tasks where the patches either pass the test suite or they don’t.

the first result was clear: the critique made things worse. then i gave the reviewer actual tools, and the result flipped.

the experiment

30 tasks from SWE-bench Lite – real GitHub issues across 10 Python repositories. Claude Opus 4.6 fixes the bug with full agentic repo access (file browsing, code search, shell execution via the Claude Agent SDK). three patches per task:

v1 – solo. Claude fixes the bug alone. no review.

v2 – cross-model review. GPT-5.4 critiques Claude’s fix. Claude reads the critique, gets a fresh copy of the repo, and revises.

v3 – self-review. Claude critiques its own fix, then revises on a fresh copy.

same author model, same repo access, same task. one variable: who wrote the critique. all three patches run through SWE-bench’s test harness. pass or fail. no judgment calls.

the flow

each agent runs in an isolated Docker container with the target repo mounted. Claude uses the Claude Agent SDK (Claude Code) with full agentic tools — file read/write, bash, grep, code search. in run 2, GPT uses OpenAI Codex CLI with equivalent capabilities. in run 1, GPT runs bare chat completions with no tools.

for each task, the flow is:

Claude solo — gets the issue description and the repo. explores the codebase, finds the bug, edits the files. the git diff of the modified repo is patch v1.
GPT critique — gets the issue description, Claude’s full response (reasoning + code changes), and in run 2, the repo. produces its own analysis of the problem and a critique of Claude’s approach.
Claude revision — gets the issue description, its own original response, GPT’s critique, and a fresh copy of the repo (no edits from step 1). reads the critique and applies a revised fix. the diff is patch v2.
Claude self-critique — same as step 2 but Claude critiques its own step 1 response.
Claude self-revision — same as step 3 but using the self-critique. the diff is patch v3.

the fresh repo copy in steps 3 and 5 matters — Claude isn’t editing on top of its previous fix. it starts clean and applies the revision from scratch. this means if the critique says “your approach is wrong, do X instead,” Claude can take a completely different approach.

i ran this twice. the only difference: how much access GPT had in step 2.

run 1: the reviewer can’t read the code

GPT-5.4 runs as bare chat completions. no repo access, no file browsing, no shell. it sees Claude’s explanation of its fix – the reasoning and code snippets – but can’t read the actual source files or verify its claims against real code.

condition	passed	failed	error	pass rate
solo (v1)	16	10	4	53%
self-review (v3)	14	8	8	47%
cross-review (v2)	10	12	8	33%

outcome	cross-review	self-review
solo passed, review broke	6	5
solo missed, review fixed	0	3
net	-6	-2

cross-model review broke 6 working patches and fixed zero. purely destructive.

the critiques sounded authoritative. GPT identified “issues” that were specific, technical, and plausible:

“your fix only handles the forward migration, you missed the backward direction”
“booleans are categorical, not continuous – casting to float is wrong”
“your change targets code that doesn’t exist in the actual module structure”

some of these were correct observations. some were hallucinated. studies of LLM-generated code reviews found hallucinations in 43-47% of generated review comments on fine-tuned models – input inconsistencies, logic contradictions, and intent deviations. frontier models likely hallucinate less, but the failure mode is the same: the review identifies problems that don’t exist. Claude couldn’t tell the difference. it found the critiques persuasive and revised its working fixes into broken ones.

models trained with RLHF learn to accommodate feedback. Sharma et al. (2024) showed this is a structural property: RLHF optimizes for user approval over truthfulness, and both humans and preference models prefer sycophantic responses over correct ones. GPT’s critique has high perplexity from Claude’s perspective – unfamiliar reasoning patterns and vocabulary that read as novelty rather than noise. Claude interprets the foreign-sounding feedback as genuine outside perspective that deserves accommodation, even when the original fix was correct.

self-review (v3) was less destructive because Claude’s own critique sounds familiar. it partially sees through its own doubts and is less compelled to act on them.

run 2: the reviewer gets equal tools

GPT-5.4 runs OpenAI’s Codex CLI – the same class of agentic tool as Claude Code. full repo access, file browsing, shell execution, code search. when it critiques Claude’s fix, it reads the actual files and verifies its claims against real code.

condition	passed	failed	error	pass rate
solo (v1)	16	12	1	55%
self-review (v3)	16	8	3	59%
cross-review (v2)	15	8	4	56%

outcome	cross-review	self-review
solo passed, review broke	5	4
solo missed, review fixed	4	4
net	-1	0

cross-model review still broke 5 tasks – sycophancy doesn’t disappear – but it also fixed 4 that solo couldn’t solve. the net dropped from -6 to -1. self-review was perfectly neutral: broke 4, fixed 4.

the tasks GPT’s grounded critique fixed were bugs where Claude’s solo approach was fundamentally wrong and GPT – having read the actual code – identified the correct direction.

what changed

the critique quality. not the model, not the prompts, not the tasks. just whether the reviewer could read the code.

ungrounded critique (run 1): “your approach probably doesn’t handle the backward migration.” GPT reasons from Claude’s description. it might be right, it might be hallucinating a problem that doesn’t exist.

grounded critique (run 2): “i read the migration file at line 340 and the backward path already handles this case. your fix is correct but you should also add a test.” GPT verified its claim against the code. the critique is specific, falsifiable, and anchored to a real file.

the sycophancy problem persists either way – cross-review still broke 5 tasks in run 2. but grounded critique adds enough genuine signal to offset the losses. the reviewer catching real bugs compensates for the cases where the author over-accommodates.

what the literature missed

the self-preference bias literature (Panickssery et al., 2024; Bavaresco et al., 2025) measures bias in evaluation settings – score two outputs side by side. they found models favor their own. in revision, i found the opposite: models defer to the other’s critique. these are the same perplexity mechanism in different settings. familiar text gets preference in evaluation; unfamiliar critique gets deference in revision.

the Rethinking Mixture-of-Agents paper (2025) found that Self-MoA (same model, multiple runs) outperforms Mixed-MoA (different models). they attributed this to model quality. the data here suggests a different variable: tool access. GPT-5.4 and Claude Opus 4.6 trade leads depending on the benchmark – Opus leads on SWE-bench Verified (80.8% vs ~80%), GPT-5.4 leads on the harder SWE-bench Pro (57.7% vs ~45%). neither is categorically weaker. but without repo access, GPT produces weaker critiques regardless of its model capability. the same model went from net -6 (no tools) to net -1 (equal tools) on the same tasks.

industrial-scale code review research identifies false alarm reduction as a key open challenge – LLM reviewers produce redundant or hallucinated comments frequently enough to require multi-stage filtering before reaching developers. analysis of iterative AI code generation found that critical vulnerabilities increase across successive LLM-driven revision rounds, with statistically significant degradation after iteration 5 – the feedback loop amplifies flaws rather than correcting them. evaluations of Copilot’s security review found it frequently missed critical vulnerabilities while proposing insecure changes.

the production code review tools have already converged on this lesson. GitHub Copilot’s code review switched to an agentic architecture in March 2026 — the reviewer now browses the full repo, reads cross-file dependencies, and gathers directory structure before commenting. earlier versions reviewed diffs without repo context and produced enough false positives that GitHub had to blend LLM review with deterministic tools like ESLint and CodeQL. CodeRabbit spins up sandboxed environments per PR with shell access, grep, and ast-grep — the reviewer runs analysis code against the actual codebase. both moved from reasoning-about-diffs to reading-the-repo for the same reason the data here shows: unverified review produces hallucinated critique.

none of this work compared the same reviewer with and without repo access on the same tasks using ground truth, or measured whether cross-model review is worse than self-review.

what this means

ungrounded review is worse than no review. if the reviewer can’t read the actual code, it hallucinates bugs, and the author accommodates the hallucinations. this applies to any workflow where a model critiques code it can’t see – including pasting snippets into a chat window and asking “what’s wrong with this.”

grounded review is net -1. when the reviewer has the same tools as the author, it breaks about as many things as it fixes. the value is in catching bugs the author can’t see – a different model with different reasoning patterns identifies failure modes the author’s model is blind to.

self-review is the safest option. Claude reviewing its own work was net zero in both runs. the model partially sees through its own doubts, making it less prone to false positive accommodation.

the reviewer needs to verify its claims. in human code review, the reviewer reads the actual code before commenting. in AI code review, the reviewer needs the same repo access so it can check whether its objections are real before raising them. a reviewer that can’t verify its own claims produces hallucinated critiques that the author model treats as authoritative.

switching models when stuck is the same dynamic. when you switch from Claude to GPT in OpenCode or similar tools, GPT sees the full conversation history including the prior model’s failed fix. switching to GPT and reading its perspective yourself is fine – you’re the filter. the risk is switching back and letting Claude revise based on GPT’s feedback without scrutiny. if GPT has full agentic access to the repo, its feedback is more likely to be grounded. if it’s just a chat window, it’s run 1.

don’t auto-apply review suggestions. cross-model critique is valuable when a human reads it and decides whether to act. the problem is when the critique feeds back into the author model automatically. the model can’t distinguish a valid objection from a confident hallucination, so it accommodates both.

experimental details

system. Claude Opus 4.6 runs in Docker with the Claude Agent SDK – file I/O, bash, web search, code editing. GPT-5.4 runs either bare chat completions (run 1) or OpenAI Codex CLI (run 2) with equivalent agentic capabilities.

tasks. 30 from SWE-bench Lite, 3 per repository across 10 Python projects.

verification. SWE-bench’s Docker-based test harness. each patch is applied to the repo at the base commit and tested against the task’s test suite. pass/fail.

prior bias runs. before ground truth, three runs measured whether the orchestrator (Claude judging Claude vs GPT disagreements) showed self-preference bias. none did – the orchestrator selected GPT’s position 50-52% across all conditions (Sonnet orchestrator, Opus orchestrator, and Opus without critiques).

limitations. n=30. no test-feedback loop (agents get one session to explore and fix, no iterative debugging against test results). results may differ with different models, different tasks, or different critique prompts.

nanoclaw's memory architecture: sqlite + 200-message rolling window

2026-03-15T00:00:00+00:00

what is nanoclaw
the memory question
architecture overview
the 200-message rolling window
cursor-based retrieval mechanism
accessing context beyond the window
comparison to RAG-based systems
trade-offs and design decisions
source citations

what is nanoclaw

nanoclaw is a WhatsApp-integrated AI assistant built on Claude that runs in Docker containers with per-group isolation. it’s designed for multi-group conversations with separate memory contexts, scheduled tasks, and persistent message history.

the interesting question isn’t what it does – it’s how it manages conversation context across multiple groups with potentially thousands of messages.

the memory question

when building LLM-based agents, one of the first architectural decisions is: how do you handle conversation history that exceeds the context window?

popular approaches:

RAG (Retrieval-Augmented Generation): embed messages, store in vector DB, retrieve semantically relevant chunks
Summarization: periodically summarize old messages, keep summaries in context
Sliding window: keep last N messages, drop older ones
Hybrid: combine multiple strategies

nanoclaw uses none of these. let’s see what it actually does.

architecture overview

nanoclaw’s memory system has two primary components:

SQLite database (store/messages.db) - stores all messages with metadata
Markdown files (conversations/ folder) - searchable conversation exports

the database schema is straightforward:

CREATE TABLE messages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  chat_jid TEXT NOT NULL,           -- WhatsApp group ID
  sender TEXT NOT NULL,              -- Phone number
  sender_name TEXT,                  -- Display name
  content TEXT NOT NULL,             -- Message text
  timestamp TEXT NOT NULL,           -- ISO 8601
  is_from_me INTEGER DEFAULT 0,     -- Bot's own messages
  is_bot_message INTEGER DEFAULT 0  -- Messages from bot
);

source: src/db.ts lines 15-25

no embeddings. no vector columns. no fancy indexing beyond the standard B-tree on chat_jid and timestamp.

the 200-message rolling window

when nanoclaw processes a new message, it retrieves conversation history using getMessagesSince():

export function getMessagesSince(
  chatJid: string,
  sinceTimestamp: string,
  botPrefix: string,
  limit: number = 200,
): NewMessage[] {
  const sql = `
    SELECT * FROM (
      SELECT id, chat_jid, sender, sender_name, content, timestamp, is_from_me
      FROM messages
      WHERE chat_jid = ? AND timestamp > ?
        AND is_bot_message = 0 AND content NOT LIKE ?
        AND content != '' AND content IS NOT NULL
      ORDER BY timestamp DESC
      LIMIT ?
    ) ORDER BY timestamp
  `;
  return db.prepare(sql).all(chatJid, sinceTimestamp, `${botPrefix}:%`, limit);
}

source: src/db.ts lines 341-364

key observations:

Hard limit of 200 messages - enforced at the SQL level via LIMIT ?
Cursor-based pagination - uses sinceTimestamp instead of fixed time window
Chronological order - ORDER BY timestamp ensures messages are in conversation order
Filters bot’s own messages - excludes is_bot_message = 1 to avoid self-references

the 200-message limit is not configurable and not adaptive based on token count. it’s a simple message count cap.

cursor-based retrieval mechanism

the “rolling window” isn’t time-based – it’s cursor-based. here’s how it works:

// src/index.ts lines 158-163
const sinceTimestamp = lastAgentTimestamp[chatJid] || '';
const missedMessages = getMessagesSince(chatJid, sinceTimestamp, ASSISTANT_NAME);

// After processing...
// src/index.ts lines 183-184
lastAgentTimestamp[chatJid] = missedMessages[missedMessages.length - 1].timestamp;

flow:

Initial state: lastAgentTimestamp is empty string ''
- retrieves last 200 messages from entire history
After first run: cursor advances to timestamp of last processed message
Next run: retrieves all messages AFTER that timestamp (up to 200)
If < 200 new messages: gets all of them
If > 200 new messages: gets only 200 most recent, older ones are dropped

this means:

if you send 500 messages while the bot is offline, it only sees the last 200
the cursor never goes backward
there’s no “lookback” or “re-retrieval” of older context

accessing context beyond the window

so what happens if you reference something from message #201?

nanoclaw provides manual retrieval tools:

1. conversation folder exports

# conversations/
The `conversations/` folder contains searchable history of
past conversations. Use this to recall context from previous sessions.

source: groups/main/CLAUDE.md line 39

agents can use the Read tool to read exported conversation markdown files.

2. grep tool

agents can search message content using the Grep tool:

Grep({
  pattern: "budget discussion",
  path: "/workspace/project/conversations/",
  output_mode: "content"
})

3. direct database queries

agents can query the SQLite database directly via Bash tool:

sqlite3 /workspace/project/store/messages.db "
  SELECT timestamp, sender_name, content
  FROM messages
  WHERE chat_jid = '120363336345536173@g.us'
    AND content LIKE '%budget%'
  ORDER BY timestamp DESC
  LIMIT 10;
"

key point: retrieval is manual and tool-initiated. the agent must explicitly decide to search for old context. it’s not automatic like RAG.

comparison to RAG-based systems

feature	nanoclaw	typical RAG system
storage	SQLite (relational)	Vector DB (Pinecone, Chroma, Weaviate)
retrieval	manual tool calls	automatic semantic search
context selection	chronological (last 200)	semantic similarity top-k
embeddings	none	required
search	SQL WHERE / Grep	vector similarity (cosine, euclidean)
latency	sub-millisecond SQL	depends on vector DB, usually 10-100ms
cost	zero (SQLite is free)	vector DB hosting + embedding API calls
complexity	low (just SQL)	medium-high (embedding pipeline, vector indexing)

why this matters:

RAG systems automatically retrieve relevant context based on semantic similarity:

user asks “what was our budget discussion?”
system embeds the query
retrieves top 5 semantically similar messages
adds them to context

nanoclaw requires the agent to explicitly search:

agent sees “what was our budget discussion?”
agent thinks “I need to search for this”
agent calls Grep or Read tool
agent adds findings to response

this is more transparent (you see the search happening) but less automatic (agent might forget to search).

trade-offs and design decisions

why 200 messages?

likely a balance between:

context window limits: keeping token count manageable
conversation coherence: 200 messages covers most multi-turn conversations
query performance: SQLite LIMIT 200 is fast even on large tables

why no embeddings?

embeddings add complexity:

need embedding API (OpenAI, Cohere, etc.) or local model
need vector storage and indexing
need embedding refresh on message updates
adds latency and cost

for a personal assistant handling dozens of groups, simplicity > sophistication.

why cursor-based?

alternatives:

time window (last 7 days): breaks if conversation is inactive for a week
fixed offset (messages 1000-1200): doesn’t adapt to conversation growth
cursor (since last processed): always picks up where you left off

cursor-based ensures continuity even with irregular message patterns.

why manual retrieval?

automatic RAG retrieval can:

add irrelevant context (semantic similarity isn’t perfect)
increase latency (every message triggers vector search)
use more tokens (retrieved chunks added to every request)

manual retrieval gives the agent control over when to pay the cost of searching.

source citations

all analysis based on nanoclaw repository source code:

database schema: src/db.ts lines 15-25
getMessagesSince function: src/db.ts lines 341-364
cursor advancement: src/index.ts lines 158-163, 183-184
conversation folder docs: groups/main/CLAUDE.md line 39
message filtering: excludes is_bot_message and bot-prefixed content

bottom line: nanoclaw’s memory is simple by design. no embeddings, no RAG, just SQLite with a 200-message rolling window and manual tool-based search. it trades automatic semantic retrieval for simplicity, transparency, and zero external dependencies.

for a multi-group WhatsApp assistant, that’s probably the right call.

adding vim keybindings to opencode's tui

2026-03-11T00:00:00+00:00

the request
what claude code’s vim mode actually supports
opencode’s TUI architecture
the four-layer architecture
cross-referencing against neovim’s source
full vim mode reference
how to try it
files changed
more gotchas (found after shipping)
escape key routing during generation

the request

claude code shipped vim mode in late 2025. within days, opencode had two open issues asking for the same thing – #1764 and #11111, the latter with 20+ comments and a fair amount of frustration from people who’d switched from claude code specifically to get an open-source alternative but then had to give up muscle memory every time they opened the prompt.

opencode is exactly that: an open-source alternative to claude code. same agentic AI coding concept, same terminal-first workflow, but the source is public and you can modify it. the TUI is built on SolidJS and @opentui/core, a terminal rendering framework. the prompt input is a TextareaRenderable widget that handles text, cursor position, and syntax highlighting via extmarks.

the goal was exact parity with claude code’s vim feature set. not a subset where we skip the hard parts, not a superset where we add visual mode and count prefixes. whatever claude code supports, we support. whatever it doesn’t, we don’t. that constraint actually made the scope manageable.

what claude code’s vim mode actually supports

the official docs at code.claude.com/docs/en/interactive-mode list the full feature set. two modes only: NORMAL and INSERT. no visual mode, no command mode, no ex commands.

mode switching:

i – insert before cursor
I – insert at first non-blank of line
a – insert after cursor
A – insert at end of line
o – open new line below, enter insert mode
O – open new line above, enter insert mode
Esc – return to normal mode

navigation:

h/j/k/l – left/down/up/right
w/e/b – word motions
0/$ – line start/end
^ – first non-blank
gg/G – buffer start/end
f/F/t/T{char} – character search on current line
;/, – repeat last f/F/t/T forward/backward

editing:

x – delete character
dd/D – delete line / delete to end
d{motion} – delete with motion
cc/C – change line / change to end
c{motion} – change with motion
yy/Y – yank line
y{motion} – yank with motion
p/P – paste after/before
>>/<< – indent/dedent
J – join lines
. – repeat last change

text objects (used with d/c/y):

iw/aw – inner/around word
iW/aW – inner/around WORD (whitespace-delimited)
i"/a" – inner/around double quotes
i'/a' – inner/around single quotes
i(/a( – inner/around parentheses
i[/a[ – inner/around square brackets
i{/a{ – inner/around curly braces

explicitly not supported:

no visual mode (v/V)
no count prefixes (3j, 5dd)
no undo/redo (u, Ctrl+R)
no / search
no r (replace char in place)

that’s the spec. now let’s look at the codebase we’re modifying.

opencode’s TUI architecture

opencode is a monorepo. the TUI lives in packages/opencode/. the relevant source tree:

packages/opencode/src/
  config/
    tui-schema.ts          # Zod schema for tui.json
    tui.ts                 # config loading
  cli/cmd/tui/
    app.tsx                # root component, provider tree
    context/
      keybind.tsx          # keybind context (existing pattern we follow)
      vim.tsx              # NEW: vim context
    component/
      prompt/
        index.tsx          # ~1171 lines, the prompt component
        textarea-keybindings.ts  # maps keybinds to KeyBinding[]
    lib/
      vim-engine.ts        # NEW: pure TS state machine

if you’re wondering “why is a terminal app using SolidJS” – the JSX you see in opencode’s source isn’t React. it’s the same angle-bracket syntax, but SolidJS compiles it into direct reactive wiring with no virtual DOM and no diffing. the rendering stack is:

SolidJS (reactive state) → @opentui/solid (binding layer) → @opentui/core (Zig-based terminal engine)

SolidJS manages signals and effects. @opentui/solid translates JSX elements like <box> and <textarea> into native renderables backed by @opentui/core, which is written in Zig. bun is just the JavaScript runtime; there’s no browser, no electron, no DOM at any point.

the way Zig ends up rendering to your terminal is through bun’s FFI (foreign function interface). @opentui/core ships a platform-specific shared library (.dylib on macOS, .so on Linux) compiled from Zig source. the TypeScript side calls into it via bun:ffi – you can see this in the type definitions: createRenderer returns a Pointer, render takes a Pointer, everything crosses the FFI boundary as raw pointers and primitives. no serialization, no IPC, just direct function calls into native code.

on the TypeScript side, each renderable (BoxRenderable, TextRenderable, TextareaRenderable) holds a Yoga layout node – that’s Facebook’s cross-platform flexbox engine. when you write <box flexGrow={1} paddingLeft={2}>, Solid creates a renderable with those layout properties set on its Yoga node. the layout tree is computed in JavaScript, but the actual painting happens in Zig.

the render loop looks roughly like this:

something changes – a signal updates, text is typed, the terminal resizes
Yoga recomputes the layout tree (which nodes moved, which resized)
each renderable paints its content into an OptimizedBuffer – a Zig-allocated character grid where each cell holds a character, foreground color, background color, and text attributes
the Zig renderer diffs the current buffer against the previous frame
only the cells that actually changed get written to stdout as ANSI escape codes – cursor movement sequences to skip unchanged regions, SGR sequences for colors and styles, then the character data

step 4 is where the performance comes from. a full-screen terminal might be 200x50 = 10,000 cells. if you type one character in the prompt, maybe 20 cells change. the Zig renderer writes escape codes for those 20 cells and skips the other 9,980. this is the same approach that tmux and neovim use – compute the minimal diff, emit the minimal escape sequences.

when a signal like vim.mode changes, only the one <text> node showing -- INSERT -- updates. that update is a direct property mutation on the Zig-backed renderable, which marks those cells dirty for the next paint. nothing else in the tree re-evaluates. this is why SolidJS is a good fit for TUI work: its granular reactivity maps directly to granular terminal updates.

the terminal rendering framework is @opentui/core at v0.1.87. it provides TextareaRenderable, which is the core input widget. the interface you care about:

interface TextareaRenderable {
  plainText: string; // current text content
  cursorOffset: number; // get/set cursor position (byte offset)
  setText(text: string): void; // replace entire content
  insertText(text: string): void; // insert at cursor
  clear(): void;
  extmarks: Extmark[]; // syntax highlighting regions
}

key flow for keyboard input: config in tui-schema.ts defines what keybinds are valid, keybind.ts parses them, keybind.tsx provides them via SolidJS context, and textarea-keybindings.ts maps them to KeyBinding[] objects that the textarea widget understands. the prompt component (prompt/index.tsx) has an onKeyDown handler that intercepts keys before they reach the textarea.

the context pattern throughout the codebase uses a createSimpleContext helper:

// from keybind.tsx -- the pattern we follow
export const { use: useKeybind, provider: KeybindProvider } =
  createSimpleContext({
    name: "Keybind",
    init: () => {
      // ... setup logic
      return {
        /* context value */
      };
    },
  });

createSimpleContext returns a { provider, use } pair. the provider wraps children and makes the context available. use is the hook that components call to access it. we follow this exact pattern for vim.

the implementation of createSimpleContext is straightforward – it’s a thin wrapper around SolidJS’s createContext and useContext:

// simplified from context/common.ts
export function createSimpleContext<T>(opts: {
  name: string
  init: () => T
}) {
  const Context = createContext<T | undefined>(undefined)

  function provider(props: { children: JSX.Element }) {
    const value = opts.init()
    return <Context.Provider value={value}>{props.children}</Context.Provider>
  }

  function use(): T {
    const ctx = useContext(Context)
    if (!ctx) throw new Error(`${opts.name} context not found`)
    return ctx
  }

  return { provider, use }
}

the provider tree in app.tsx is a nested stack of these providers. order matters – a provider can only call use on contexts that are higher in the tree (already initialized). VimProvider reads from TuiConfigProvider, so it must be nested inside it:

// app.tsx (simplified)
export function App() {
  return (
    <TuiConfigProvider>
      <ThemeProvider>
        <KeybindProvider>
          <VimProvider>
            {" "}
            {/* reads TuiConfig, provides vim context */}
            <Prompt />
          </VimProvider>
        </KeybindProvider>
      </ThemeProvider>
    </TuiConfigProvider>
  );
}

the actual tree has more providers, but this shows the nesting relationship. Prompt can call useVim(), useKeybind(), useTheme(), and useTuiConfig() because all four are ancestors in the tree.

the four-layer architecture

the implementation splits across four layers. each layer has a single responsibility and doesn’t reach into the others’ concerns.

layer 1: config

packages/opencode/src/config/tui-schema.ts defines the shape of tui.json. we add one field:

vim: z.boolean()
  .optional()
  .describe("Enable vim editing mode in the prompt input");

opt-in via { "vim": true } in .opencode/tui.json. defaults to false (or rather, undefined, which we treat as false).

we used .optional() instead of .default(false) deliberately. the tui.ts loader has several fallback paths that construct empty {} objects and merge them. with .default(false), Zod’s output type inference makes vim non-optional (boolean instead of boolean | undefined), which causes type errors at those empty-object fallback sites. .optional() keeps vim as boolean | undefined throughout, and we handle the ?? false defaulting in the context layer where we actually use it.

layer 2: VimEngine

packages/opencode/src/cli/cmd/tui/lib/vim-engine.ts is 957 lines of pure TypeScript. zero imports. no SolidJS, no @opentui/core, no external dependencies at all.

the public interface:

export type VimMode = "normal" | "insert";

export interface VimKeyEvent {
  key: string;
  ctrl: boolean;
  shift: boolean;
  meta: boolean;
}

export interface VimResult {
  consumed: boolean;
  newText?: string;
  newCursor?: number;
  modeChange?: VimMode;
}

export class VimEngine {
  getMode(): VimMode;
  handleKey(event: VimKeyEvent, text: string, cursor: number): VimResult;
  reset(): void;
}

handleKey takes the current text and cursor position as arguments. it returns what changed. the engine never holds a reference to the textarea – it doesn’t know TextareaRenderable exists. the caller reads the current state, passes it in, and applies the result. this makes the engine testable with plain strings and integers, no UI framework required.

internal state:

private mode: VimMode = "normal"
private pending: string = ""          // multi-key buffer: "d", "c", "y", "g", ">", "<", "f", "F", "t", "T"
private register: string = ""         // unnamed yank register
private lastChange: ChangeRecord | null = null   // for dot repeat
private lastFtMotion: FtMotion | null = null     // for ; and , repeat
private insertSession: InsertSession | null = null  // tracks text on insert entry

pending is how operator + motion composition works. pressing d doesn’t immediately do anything – it sets pending = "d" and returns { consumed: true }. the next key press sees pending === "d" and knows to interpret the key as a motion. pressing w with pending === "d" triggers applyOperator("d", text, cursor, "w").

applyOperator calls getMotionRange to find the affected character range, then deletes it and stores the deleted text in register. for the c operator, it also switches to insert mode after deleting. for y, it just stores the range in register without modifying text.

text objects (iw, a", i{, etc.) go through applyTextObjectOperator instead, which calls findTextObjectRange to locate the object boundaries.

lastChange is a ChangeRecord:

interface ChangeRecord {
  type: "operator-motion" | "operator-textobj" | "insert-change" | "simple";
  operator?: string;
  motion?: string;
  textobj?: string;
  insertedText?: string; // what was typed during the insert session
  simpleOp?: () => VimResult; // for x, dd, D, J, >>, <<
}

dot repeat replays whatever lastChange describes. for dw, it re-runs applyOperator("d", text, cursor, "w") against the current text and cursor. for cw{text}Esc, it re-runs the delete, switches to insert mode, inserts insertedText, then switches back to normal.

insertSession tracks the text content when we entered insert mode. on Esc, we diff the current text against insertSession.textBefore to find what was typed, and store that as insertedText in lastChange. this is how dot repeat knows what to re-insert.

lastFtMotion stores the last f/F/t/T operation:

interface FtMotion {
  type: "f" | "F" | "t" | "T";
  char: string;
}

; replays it in the same direction. , replays it in the opposite direction (f becomes F, t becomes T, and vice versa).

the key dispatch loop:

handleKey(event: VimKeyEvent, text: string, cursor: number): VimResult {
  if (this.mode === "insert") {
    if (event.key === "escape") {
      return this.enterNormal(text, cursor)
    }
    return { consumed: false }  // let the textarea handle it
  }

  // normal mode
  const key = this.resolveKey(event)  // normalize shift+key combos

  if (this.pending) {
    return this.handlePendingKey(key, text, cursor)
  }

  return this.handleNormalKey(key, text, cursor)
}

in insert mode, the engine only intercepts Escape. everything else returns { consumed: false }, which tells the prompt component to let the key fall through to the textarea’s normal handling. this is the key insight: vim mode doesn’t replace the textarea’s input handling, it wraps it. regular typing in insert mode goes through the existing path unchanged.

layer 3: VimProvider

packages/opencode/src/cli/cmd/tui/context/vim.tsx is 35 lines:

import { createSignal } from "solid-js";
import { createSimpleContext } from "./common";
import { useTuiConfig } from "../../config/tui";
import { VimEngine, type VimMode, type VimKeyEvent } from "../lib/vim-engine";

export const { use: useVim, provider: VimProvider } = createSimpleContext({
  name: "Vim",
  init: () => {
    const config = useTuiConfig();
    const enabled = config.vim ?? false;
    const engine = new VimEngine();
    const [mode, setMode] = createSignal<VimMode>(
      enabled ? "normal" : "insert",
    );

    return {
      get enabled() {
        return enabled;
      },
      get mode() {
        return mode();
      },
      handleKey(event: VimKeyEvent, text: string, cursor: number) {
        const result = engine.handleKey(event, text, cursor);
        if (result.modeChange) {
          setMode(result.modeChange);
        }
        return result;
      },
      reset() {
        engine.reset();
        setMode(enabled ? "normal" : "insert");
      },
    };
  },
});

the provider’s job is to bridge the pure engine to SolidJS reactivity. mode is a signal so any component that reads vim.mode will re-render when the mode changes. the engine itself doesn’t know about signals – it returns modeChange in the result, and the provider calls setMode when it sees one.

VimProvider gets added to the provider tree in app.tsx, nested inside TuiConfigProvider (since it reads the config) but outside Prompt (since the prompt consumes it).

layer 4: prompt integration

packages/opencode/src/cli/cmd/tui/component/prompt/index.tsx has three touch points.

1. hook call – alongside the other context hooks at the top of the component:

const vim = useVim();

2. key interception – at the top of the onKeyDown handler, right after the disabled check:

if (vim.enabled) {
  const result = vim.handleKey(
    { key: e.name, ctrl: !!e.ctrl, shift: !!e.shift, meta: !!e.meta },
    input.plainText,
    input.cursorOffset,
  );
  if (result.consumed) {
    e.preventDefault();
    if (result.newText !== undefined) {
      input.setText(result.newText);
      syncExtmarksWithPromptParts();
      setStore("prompt", "input", result.newText);
    }
    if (result.newCursor !== undefined) {
      input.cursorOffset = result.newCursor;
    }
    return;
  }
}

when consumed is true, we apply the result and return early. the key never reaches the textarea’s default handler. when consumed is false (insert mode, regular typing), we fall through and the textarea handles it normally.

syncExtmarksWithPromptParts() is an existing function in the prompt component that re-applies syntax highlighting after text changes. we call it whenever we modify text through the vim engine, same as any other text modification path.

setStore("prompt", "input", result.newText) keeps the SolidJS store in sync with the textarea content. the store is the source of truth for the prompt value that gets sent to the agent.

3. mode indicator – in the tray section of the prompt JSX:

<Show when={vim.enabled}>
  <text fg={theme.textMuted}>
    {vim.mode === "normal" ? "-- NORMAL --" : "-- INSERT --"}
  </text>
</Show>

this only renders when vim is enabled. the vim.mode signal drives it reactively – when you press i to enter insert mode, the indicator updates immediately.

cross-referencing against neovim’s source

the spec from claude code’s docs tells you what commands exist. it doesn’t tell you the exact semantics. for that, we went to neovim’s C source.

the key files:

src/nvim/textobject.c – word motions (fwd_word, bck_word, end_word) and the cls() character classification function
src/nvim/normal.c – normal mode dispatch table, including the cw special case
src/nvim/charset.c / src/nvim/mbyte.c – character classification (utf_class)

character classification

word motions in vim depend on character classification. from textobject.c around line 271:

static int cls(void) {
  int c = gchar_cursor();
  if (c == ' ' || c == '\t' || c == NUL) return 0;
  c = utf_class(c);
  if (c != 0 && cls_bigword) return 1;  // W/B/E: all non-blank = class 1
  return c;
}

three classes: 0 = whitespace, 1 = punctuation, 2+ = keyword characters. cls_bigword is set when processing W/B/E motions (WORD, not word), which treat all non-whitespace as the same class.

the default iskeyword setting is @,48-57,_,192-255, which expands to [a-zA-Z0-9_] plus the Latin-1 supplement range (characters 192-255). our isWordChar function uses /[A-Za-z0-9_]/, which matches the ASCII portion. for a prompt input that’s mostly code and English text, this is correct.

the w motion algorithm from fwd_word in textobject.c:

get the class of the character at cursor
skip all characters of the same class
skip whitespace
you’re now at the start of the next word

in C, the inner loop looks roughly like:

// simplified from fwd_word()
int cls_start = cls();
while (!end_of_buffer()) {
  inc_cursor();
  if (cls() != cls_start) break;
}
// now skip whitespace
while (!end_of_buffer() && cls() == 0) {
  inc_cursor();
}

our TypeScript equivalent:

function findNextWordStart(text: string, cursor: number): number {
  if (cursor >= text.length - 1) return cursor;
  const startClass = charClass(text[cursor]);
  let i = cursor;
  // skip same class
  while (i < text.length - 1 && charClass(text[i + 1]) === startClass) {
    i++;
  }
  i++;
  // skip whitespace
  while (i < text.length && (text[i] === " " || text[i] === "\t")) {
    i++;
  }
  return i;
}

function charClass(ch: string): number {
  if (ch === " " || ch === "\t" || ch === "") return 0;
  if (isWordChar(ch)) return 2;
  return 1; // punctuation
}

the b motion is the reverse: skip whitespace backward, then skip same-class characters backward. in TypeScript:

function findPrevWordStart(text: string, cursor: number): number {
  if (cursor <= 0) return 0;
  let i = cursor - 1;
  // skip whitespace
  while (i > 0 && (text[i] === " " || text[i] === "\t")) {
    i--;
  }
  // skip same class backward
  const endClass = charClass(text[i]);
  while (i > 0 && charClass(text[i - 1]) === endClass) {
    i--;
  }
  return i;
}

the symmetry between w and b is clean. w skips forward past same-class then whitespace. b skips backward past whitespace then same-class. the only asymmetry is that w ends at the start of the next word while b ends at the start of the previous word – both are word-start positions, just in different directions.

three bugs found

bug 1: cw should behave like ce

from normal.c around line 5951:

/*
 * "cw" is a special case - it works like "ce" if the cursor is
 * on a non-blank.  This is not Vi compatible, but it's what Vim
 * has always done.
 */
if (cap->cmdchar == 'c' && cap->nchar == 'w'
    && !u_save_cursor()
    && !lineempty(curwin->w_cursor.lnum)) {
  if (!vim_iswhite(gchar_cursor())) {
    cap->nchar = 'e';
  }
}

the comment says it all. cw maps to ce when the cursor is on a non-blank character. this is not Vi compatible – it’s a Vim-specific behavior that’s been there long enough that everyone expects it.

our initial implementation treated cw as a normal c + w composition, which deleted from cursor to the start of the next word (including the whitespace between words). the fix: in applyOperator, when the operator is c and the motion is w, check if the cursor is on whitespace. if not, redirect the motion to e.

// in applyOperator
if (op === "c" && motion === "w") {
  const ch = text[cursor] ?? "";
  if (ch !== " " && ch !== "\t" && ch !== "") {
    motion = "e";
  }
}

bug 2: e motion starting position

neovim’s end_word function always calls inc_cursor() first – it advances at least one character before looking for the end of a word. this means pressing e when you’re already at the end of a word moves to the end of the next word.

our initial findNextWordEnd started at cursor:

// wrong
let i = cursor;
while (i < text.length - 1 && isWordChar(text[i + 1])) {
  i++;
}

if the cursor is at the last character of a word, this loop doesn’t advance at all and returns cursor. pressing e does nothing.

the fix is let i = cursor + 1:

// correct
let i = cursor + 1;
if (i >= text.length) return cursor;
// skip whitespace
while (i < text.length && (text[i] === " " || text[i] === "\t")) {
  i++;
}
// find end of word
while (i < text.length - 1 && isWordChar(text[i]) && isWordChar(text[i + 1])) {
  i++;
}
return i;

bug 3: iw on whitespace

vim’s documentation for iw (inner word): “Select [count] words (see word ). White space between words is counted too, see aw if you don’t want this.”

the actual behavior when the cursor is on whitespace: iw selects the whitespace run itself, not the adjacent word. aw selects the whitespace plus the adjacent word.

our initial implementation jumped to the adjacent word when the cursor was on whitespace, which is wrong. the fix:

function findWordRange(
  text: string,
  cursor: number,
  inner: boolean,
): [number, number] {
  const onWhitespace = text[cursor] === " " || text[cursor] === "\t";

  if (onWhitespace) {
    // find the whitespace run
    let start = cursor;
    let end = cursor;
    while (start > 0 && (text[start - 1] === " " || text[start - 1] === "\t"))
      start--;
    while (
      end < text.length - 1 &&
      (text[end + 1] === " " || text[end + 1] === "\t")
    )
      end++;

    if (!inner) {
      // aw: include the adjacent word (prefer the word after the whitespace)
      if (end + 1 < text.length && isWordChar(text[end + 1])) {
        while (end + 1 < text.length && isWordChar(text[end + 1])) end++;
      } else if (start > 0 && isWordChar(text[start - 1])) {
        while (start > 0 && isWordChar(text[start - 1])) start--;
      }
    }
    return [start, end];
  }

  // cursor is on a word character
  let start = cursor;
  let end = cursor;
  while (start > 0 && isWordChar(text[start - 1])) start--;
  while (end < text.length - 1 && isWordChar(text[end + 1])) end++;

  if (!inner) {
    // aw: include trailing whitespace (or leading if at end of line)
    if (
      end + 1 < text.length &&
      (text[end + 1] === " " || text[end + 1] === "\t")
    ) {
      while (
        end + 1 < text.length &&
        (text[end + 1] === " " || text[end + 1] === "\t")
      )
        end++;
    } else if (
      start > 0 &&
      (text[start - 1] === " " || text[start - 1] === "\t")
    ) {
      while (start > 0 && (text[start - 1] === " " || text[start - 1] === "\t"))
        start--;
    }
  }
  return [start, end];
}

the escape key name bug

this one wasn’t from neovim source – it came up during live testing. @opentui/core sends the escape key as "escape" (lowercase). we were checking for "Escape" (capital E). the engine never saw escape key events, so you couldn’t exit insert mode.

the fix is a single character change in the key dispatch:

if (event.key === "escape") {
  // not "Escape"
  return this.enterNormal(text, cursor);
}

worth noting because it’s the kind of thing that’s invisible in unit tests if you’re not careful about what key names your test framework uses.

full vim mode reference

mode switching

key	action
`i`	enter insert mode at cursor
`I`	enter insert mode at first non-blank of line
`a`	enter insert mode after cursor
`A`	enter insert mode at end of line
`o`	open new line below, enter insert mode
`O`	open new line above, enter insert mode
`Esc`	return to normal mode / clear pending command

key	action
`h`	move left
`j`	move down (next line)
`k`	move up (previous line)
`l`	move right
`w`	next word start
`e`	end of current/next word
`b`	previous word start
`W`	next WORD start (whitespace-delimited)
`E`	end of current/next WORD
`B`	previous WORD start
`0`	beginning of line
`$`	end of line
`^`	first non-blank character of line
`gg`	beginning of buffer
`G`	end of buffer
`f{char}`	jump to next occurrence of {char} on current line
`F{char}`	jump to previous occurrence of {char} on current line
`t{char}`	jump to just before next {char} on current line
`T{char}`	jump to just after previous {char} on current line
`;`	repeat last f/F/t/T in same direction
`,`	repeat last f/F/t/T in opposite direction

editing (normal mode)

key	action
`x`	delete character at cursor
`dd`	delete entire line
`D`	delete from cursor to end of line
`dw`	delete to next word start
`de`	delete to end of word
`db`	delete to previous word start
`d0`	delete to beginning of line
`d$`	delete to end of line
`d{motion}`	delete with any supported motion
`cc`	change entire line (delete + enter insert)
`C`	change from cursor to end of line
`cw`	change to end of word (behaves like `ce`)
`ce`	change to end of word
`cb`	change to previous word start
`c{motion}`	change with any supported motion
`yy` / `Y`	yank entire line into register
`yw`	yank to next word start
`ye`	yank to end of word
`yb`	yank to previous word start
`y{motion}`	yank with any supported motion
`p`	paste register after cursor (linewise: below current line)
`P`	paste register before cursor (linewise: above current line)
`>>`	indent current line by one shiftwidth
`<<`	dedent current line by one shiftwidth
`J`	join current line with next (adds space, strips leading whitespace)
`.`	repeat last change

text objects (used with d/c/y in normal mode)

text object	description
`iw`	inner word – the word under cursor, or whitespace run if on whitespace
`aw`	around word – word plus surrounding whitespace
`iW`	inner WORD – whitespace-delimited token
`aW`	around WORD – WORD plus surrounding whitespace
`i"`	inner double quotes – content between `"..."`
`a"`	around double quotes – includes the quote characters
`i'`	inner single quotes – content between `'...'`
`a'`	around single quotes – includes the quote characters
`i(`	inner parentheses – content between `(...)`
`a(`	around parentheses – includes the parentheses
`i[`	inner square brackets – content between `[...]`
`a[`	around square brackets – includes the brackets
`i{`	inner curly braces – content between `{...}`
`a{`	around curly braces – includes the braces

not supported (matching claude code)

feature	reason
visual mode (`v/V`)	not in claude code’s feature set
count prefixes (`3j`, `5dd`)	not in claude code’s feature set
undo/redo (`u`, `Ctrl+R`)	not in claude code’s feature set
search (`/`, `?`, `n`, `N`)	not in claude code’s feature set
replace char (`r`)	not in claude code’s feature set
marks (`m`, `, `'`)	not in claude code’s feature set
registers (`"a`, `"b`, etc.)	not in claude code’s feature set
ex commands (`:`)	not in claude code’s feature set
macros (`q`, `@`)	not in claude code’s feature set

how to try it

the full source is on github: sngeth/opencode (feat/vim-keybindings)

git clone https://github.com/sngeth/opencode.git
cd opencode
git checkout feat/vim-keybindings
bun install

create ~/.config/opencode/tui.json to enable vim mode globally:

{
  "vim": true
}

you can also use .opencode/tui.json in a specific project directory, but the global path is better when running the compiled binary since CWD changes per project.

running as a compiled binary (recommended)

bun run dev from packages/opencode/ works but locks your terminal to the repo directory. sessions and file paths resolve relative to process.cwd(), so you want to invoke it from the project you’re actually working on.

compile a native binary instead. bun bundles everything – dependencies, the SolidJS JSX transform, tree-sitter parsers, migrations – into a single executable:

# from packages/opencode
bun run build --single --skip-install

this produces a ~116MB binary at dist/opencode-<platform>-<arch>/bin/opencode. symlink it somewhere on your $PATH under a name that won’t shadow the official opencode binary:

ln -sf $(pwd)/dist/opencode-darwin-arm64/bin/opencode ~/.local/bin/opencode-vim

now opencode-vim works from any directory. sessions are scoped correctly and the official opencode stays untouched. rebuild after source changes with --single --skip-install (~10 seconds).

the mode indicator appears in the prompt tray. it shows -- NORMAL -- when you’re in normal mode and -- INSERT -- when you’re in insert mode. opencode starts in normal mode when vim is enabled, same as neovim’s default.

if you want to start in insert mode automatically, press i after the prompt loads. there’s no config option for the initial mode – that’s a deliberate choice to match how vim itself works.

files changed

file	change
`packages/opencode/src/config/tui-schema.ts`	added `vim: z.boolean().optional()` field
`packages/opencode/src/cli/cmd/tui/lib/vim-engine.ts`	NEW – 957-line pure TypeScript state machine
`packages/opencode/src/cli/cmd/tui/context/vim.tsx`	NEW – 35-line SolidJS context wrapping the engine
`packages/opencode/src/cli/cmd/tui/app.tsx`	added `VimProvider` to the provider tree
`packages/opencode/src/cli/cmd/tui/component/prompt/index.tsx`	key interception in `onKeyDown` + mode indicator in tray

five files. the engine is the bulk of it. the integration is thin by design – the engine does the work, the prompt just routes keys through it and applies the results.

the pure engine approach paid off during debugging. when the cw behavior was wrong, we could write a test that called engine.handleKey directly with a string and cursor position, without spinning up a terminal or a SolidJS component tree. the neovim source comparison was also easier because we were comparing TypeScript logic to C logic, not trying to trace through UI event handling at the same time.

the escape key name bug is a good reminder that integration tests matter even when unit tests pass. the engine was correct. the key name mismatch was in the boundary between @opentui/core’s event format and our engine’s expected format. that boundary only exists in the running application.

more gotchas (found after shipping)

the initial post covered three bugs found during development. three more showed up after the implementation was running in daily use.

`x` silently fails on trailing spaces

the symptom: pressing x at the end of a line with a trailing space does nothing. the character is there, the cursor is on it, but x returns without deleting.

the root cause is a cursor model mismatch. @opentui/core’s TextareaRenderable.cursorOffset is a gap-position – it sits between characters, like a text insertion point. vim’s normal mode cursor is a cell-position – it sits on a character. for the string "xxx " (four characters), cursorOffset=4 means “after the space” in the gap model. in vim’s model, that position doesn’t exist: the valid range is 0-3.

the x handler had an early return for this case:

if (safeCursor >= text.length) return { consumed: true };

when cursorOffset=4 and text.length=4, this fires and returns without deleting anything. the user sees nothing happen.

the same mismatch caused a second problem: pressing Escape to exit insert mode didn’t move the cursor left by one position. real neovim does this – if you’re at the end of a word in insert mode and press Escape, the cursor steps back one character so it’s sitting on the last character rather than past it. without that adjustment, the cursor stays at offset 4 after switching to normal mode, which puts it in the gap-position zone where operations silently fail.

two fixes in vim-engine.ts:

// before -- safeCursor could equal text.length in normal mode
const safeCursor = this.clamp(cursor, 0, text.length);

// after -- clamp to text.length - 1 so cursor is always on a character
const safeCursor = text.length > 0 ? this.clamp(cursor, 0, text.length - 1) : 0;

// before -- escape exits insert mode but leaves cursor at end-of-text gap position
return { consumed: true, modeChange: "normal" };

// after -- step cursor back one, matching neovim's behavior
return {
  consumed: true,
  newCursor: text.length > 0 ? Math.max(0, cursor - 1) : 0,
  modeChange: "normal",
};

the clamping fix is the more important one. any operation that checked safeCursor >= text.length or safeCursor >= end would silently fail when the cursor was at the end of text. x was the most visible case, but the same condition could affect other operators.

every shifted command is silently broken

pressing Shift+j to join lines moved the cursor down instead. pressing Shift+o to open a line above did nothing visible. every uppercase command – J, O, A, I, G, D, C, Y, P, W, E, B, F, T – was broken.

@opentui/core sends key events with the base character in e.name (always lowercase) and shift state as a separate boolean e.shift. so Shift+j arrives as { name: "j", shift: true }, not { name: "J" }. the vim engine was doing const key = event.key and checking against uppercase literals like if (key === "J"). the key was always lowercase. the uppercase branch never matched.

the lowercase branch matched instead: if (key === "j") → move down. Shift+j moved the cursor down when it should have joined lines. Shift+o hit if (key === "o") → open line below instead of above.

the fix is one line at the top of handleKey:

// before
const key = event.key;

// after
const key =
  event.shift && event.key.length === 1 ? event.key.toUpperCase() : event.key;

when shift is held and the key is a single character, uppercase it before dispatch. this is the kind of bug that’s invisible in unit tests where you construct key events with the character you expect ("J") rather than the character the framework actually sends ("j" + shift: true).

escape key routing during generation

opencode has a session interrupt command – when a task is running, pressing Escape should abort the generation. after adding vim mode, that stopped working. pressing Escape during generation always triggered the interrupt instead of letting vim switch from insert to normal mode.

the root cause wasn’t in the textarea’s onKeyDown handler at all. opencode’s command system registers a global useKeyboard handler in dialog-command.tsx that fires before any component-level key handler:

// dialog-command.tsx -- global handler, fires first
useKeyboard((evt) => {
  if (suspended()) return;
  if (dialog.stack.length > 0) return;
  for (const option of entries()) {
    if (!isEnabled(option)) continue;
    if (option.keybind && keybind.match(option.keybind, evt)) {
      evt.preventDefault();
      option.onSelect?.(dialog);
      return;
    }
  }
});

the session_interrupt command is registered with keybind: "session_interrupt" (mapped to "escape") and enabled: status().type !== "idle". during generation, every Escape keypress matches the interrupt command before the textarea ever sees it. vim never gets a chance to handle the key.

the fix is one line – make the interrupt command aware of vim’s mode:

// before
enabled: status().type !== "idle",

// after
enabled: status().type !== "idle" && !(vim.enabled && vim.mode === "insert"),

when vim is in insert mode, the interrupt command is disabled. Escape passes through the global handler without matching, reaches the textarea’s onKeyDown, and vim processes it normally (insert to normal). on the next Escape, vim is in normal mode, the interrupt command is enabled again, and the abort fires.

the resulting behavior:

mode	task running	`Escape` does
insert	yes	switches to normal mode
normal	yes	interrupts the generation
insert	no	switches to normal mode
normal	no	clears pending command

this gives you the two-step sequence that feels natural: Escape to get to normal mode, Escape again to interrupt. it matches the mental model of “escape gets me out of whatever I’m in” – first out of insert mode, then out of the generation.

opencode permission rules: protecting your code from ai agents

2026-03-10T00:00:00+00:00

a terraform destroy that wiped production

in february 2026, alexey grigorev let his claude code agent run terraform destroy on what he thought were duplicate resources. the agent had silently unpacked an old terraform state file that pointed at production infrastructure instead. the entire course management platform for datatalks.club went down – database, vpc, ecs cluster, load balancers, bastion host, all gone. every automated snapshot was deleted with it.

“i over-relied on the ai agent to run terraform commands. i treated plan, apply, and destroy as something that could be delegated. that removed the last safety layer.”

the agent told him it would run terraform destroy. he approved it. the permission system worked – it asked. but he didn’t realize the state file had changed underneath him, so “yes, destroy those duplicates” actually meant “yes, destroy production.”

this is the hard lesson: ask only protects you if you understand what you’re approving. he saw terraform destroy and it looked logical in context. the agent had even explained its reasoning. but the premise was wrong because the state file was wrong, and no amount of prompting catches that.

“what happened was that i didn’t notice claude unpacking my terraform archive. it replaced my current state file with an older one that had all the info about the datatalks.club course management platform.”

his fix was to stop delegating entirely:

“agents no longer execute commands. every plan is reviewed manually. every destructive action is run by me.”

that works, but it’s also the nuclear option. opencode’s permission system offers a middle ground: let agents run safe commands freely, prompt you for risky ones, and hard-block the ones where no amount of context makes them safe to delegate.

the same class of problem applies to git worktrees. an agent running git restore, git reset --hard, or git clean can silently wipe uncommitted changes with no recovery path. here’s a practical config that draws the line.

the configuration

add this to your ~/.config/opencode/opencode.json:

{
  "permission": {
    "bash": {
      "*": "ask",
      "git status*": "allow",
      "git diff*": "allow",
      "git log*": "allow",
      "git add*": "allow",
      "git commit*": "ask",
      "git stash*": "ask",
      "git checkout*": "ask",
      "git restore*": "ask",
      "git reset*": "ask",
      "git clean*": "deny",
      "git worktree*": "ask"
    }
  }
}

how it works

opencode evaluates permission rules using last-match-wins ordering. the "*": "ask" catch-all at the top means any bash command not explicitly listed requires your approval. more specific rules below it override the default for their matched patterns.

three permission levels:

allow – runs silently, no prompt
ask – prompts you with once/always/reject before executing
deny – blocked outright, the agent cannot run it at all

test results

all tests run against a stub git repo with uncommitted modifications and untracked files present.

“allow” commands – ran without prompting

these are safe, read-only (or low-risk staging) operations. the agent runs them freely.

test 1: `git status`

rule: "git status*": "allow" result: executed immediately, no prompt.

$ git status
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   main.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	untracked-work.txt

no changes added to commit (use "git add" and/or "git commit -a")

test 2: `git diff`

rule: "git diff*": "allow" result: executed immediately, no prompt.

$ git diff
diff --git a/main.py b/main.py
index 110c807..11a6bc3 100644
--- a/main.py
+++ b/main.py
@@ -4,3 +4,7 @@ def greet(name):

 if __name__ == "__main__":
     print(greet("world"))
+
+
+def farewell(name):
+    return f"Goodbye, {name}!"

test 3: `git log`

rule: "git log*": "allow" result: executed immediately, no prompt.

$ git log --oneline
bf23d2b initial commit

test 4: `git add`

rule: "git add*": "allow" result: executed immediately, no prompt.

$ git add .
(no output -- files staged successfully)

“ask” commands – prompted for approval

these commands can modify your working tree or history. opencode shows a prompt with once, always, or reject options before the command runs.

test 5: `git commit`

rule: "git commit*": "ask" result: prompted, approved, then executed.

$ git commit -m "test commit from agent"
[main 7db8a3c] test commit from agent
 2 files changed, 6 insertions(+)
 create mode 100644 untracked-work.txt

test 6: `git stash`

rule: "git stash*": "ask" result: prompted, approved, then executed.

$ git stash
Saved working directory and index state WIP on main: 7db8a3c test commit from agent

test 7: `git restore`

rule: "git restore*": "ask" result: prompted, approved, then executed.

$ git restore main.py
(no output -- changes silently discarded)

this is one of the most dangerous “quiet” commands. without the ask rule, an agent could silently discard your uncommitted modifications with zero output to indicate anything happened.

test 8: `git checkout`

rule: "git checkout*": "ask" result: prompted, approved, then executed.

$ git checkout feature-branch
Switched to branch 'feature-branch'

test 9: `git reset --hard`

rule: "git reset*": "ask" result: prompted, approved, then executed.

$ git reset --hard HEAD
HEAD is now at c44ce28 test commit from agent

this catches the most destructive variant (--hard) which resets the index and working tree, throwing away all uncommitted changes.

test 10: `git worktree`

rule: "git worktree*": "ask" result: prompted, approved, then executed.

$ git worktree add ../permission-test-wt main
Preparing worktree (checking out 'main')
HEAD is now at 7db8a3c test commit from agent

“deny” command – blocked entirely

test 11: `git clean -fd`

rule: "git clean*": "deny" result: blocked. the agent received an error and the command never executed.

$ git clean -fd
Error: The user has specified a rule which prevents you from using
this specific tool call. Here are some of the relevant rules:
[
  {"permission":"*","pattern":"*","action":"allow"},
  {"permission":"bash","pattern":"*","action":"ask"},
  {"permission":"bash","pattern":"git status*","action":"allow"},
  {"permission":"bash","pattern":"git diff*","action":"allow"},
  {"permission":"bash","pattern":"git log*","action":"allow"},
  {"permission":"bash","pattern":"git add*","action":"allow"},
  {"permission":"bash","pattern":"git commit*","action":"ask"},
  {"permission":"bash","pattern":"git stash*","action":"ask"},
  {"permission":"bash","pattern":"git checkout*","action":"ask"},
  {"permission":"bash","pattern":"git restore*","action":"ask"},
  {"permission":"bash","pattern":"git reset*","action":"ask"},
  {"permission":"bash","pattern":"git clean*","action":"deny"},
  {"permission":"bash","pattern":"git worktree*","action":"ask"}
]

the untracked file (expendable.txt) survived:

$ ls expendable.txt
expendable.txt

git clean is the only command set to deny because it permanently removes untracked files with no recovery path – not even git reflog can help you. the ask tier at least gives you a chance to think; deny takes the option off the table entirely.

why this matters for worktrees

when you use git worktree to run multiple agents in parallel on different branches, each worktree has its own working tree but shares the same .git directory. this creates a unique risk surface.

commands that destroy work inside a worktree

git restore <file> – the most common agent culprit. agents run this reflexively to “undo” a bad edit. it silently discards your modifications with zero output. you’d never know it happened.
git reset --hard – agents reach for this when things get messy and they want a “clean slate.” resets the index and working tree, all uncommitted changes gone.
git checkout <branch> – if uncommitted changes conflict with the target branch, git overwrites your working tree files. agents switch branches casually.
git clean -fd – deletes all untracked files permanently. new files that were never staged are gone with no recovery path, not even git reflog.

worktree management commands that destroy work

the worktree management commands themselves are also dangerous:

git worktree remove <path> – deletes the entire worktree directory. git refuses if there are uncommitted modifications, but agents will retry with --force when they hit the error, which bypasses the safety check and nukes everything.
git worktree remove --force <path> – skips the dirty-tree check entirely. if an agent decides to “clean up” a worktree you’re still working in, all uncommitted files in that worktree directory are gone.
git branch -D <branch> – while not a worktree command, an agent in one worktree can delete a branch that’s checked out in another worktree, leaving it in a broken state.

shared state risks across worktrees

all worktrees share the same .git directory. an agent operating in one worktree can affect others:

git gc --prune=now can clean up loose objects that other worktrees still reference
ref updates (tags, branches) in one worktree are immediately visible to all others
git stash in one worktree is accessible from all worktrees – an agent could accidentally pop another worktree’s stash

the "*": "ask" catch-all is critical. it means any command not explicitly listed – rm, curl, python, whatever – still requires your approval rather than running blind.

applying this to terraform

the datatalks.club incident would have been prevented with:

{
  "permission": {
    "bash": {
      "terraform plan*": "ask",
      "terraform apply*": "deny",
      "terraform destroy*": "deny",
      "terraform import*": "ask"
    }
  }
}

deny on terraform apply and terraform destroy forces you to run them yourself in a separate terminal where you see the full plan output and can verify which state file terraform is using. the agent can still generate plans and write config – it just can’t pull the trigger.

this wouldn’t have helped if he’d rubber-stamped an ask prompt the same way he approved the agent’s reasoning in chat. but deny removes the option entirely – the agent physically cannot execute the command, no matter how logical its explanation sounds.

additional recommendations

if you’re using multiple worktrees in different directories, add external_directory rules to control cross-worktree access:

{
  "permission": {
    "external_directory": {
      "~/Code/my-project-wt-*": "ask"
    }
  }
}

this ensures the agent in one worktree has to get your approval before touching files in another worktree that’s outside the directory where you launched opencode.

important notes

restart required: permission changes in opencode.json take effect on the next opencode launch, not mid-session.
last-match-wins: rules are evaluated in order and the last matching rule wins. put the "*" catch-all first, specific rules after.
wildcard behavior: * matches zero or more of any character (including spaces). "git status*" matches both git status and git status --porcelain.
per-agent overrides: you can set different permissions per agent in the "agent" config section. for example, a review-only agent could have "edit": "deny".
“always” is session-only: when opencode prompts you and you choose “always”, that approval only lasts for the current session. it lives in memory, not on disk – it does not update opencode.json. once you restart opencode, you’ll be prompted again. if you want a permanent allow rule, add it to opencode.json yourself.

server-side vs client-side tools in llm apis

2026-01-30T00:00:00+00:00

when building agentic applications with the anthropic api, understanding the difference between client tools and server tools is essential. they have different execution models, response structures, and handling requirements.

the two types of tools

from the anthropic documentation:

Claude supports two types of tools:

Client tools: Tools that execute on your systems

Server tools: Tools that execute on Anthropic’s servers, like the web search and web fetch tools. These tools must be specified in the API request but don’t require implementation on your part.

aspect	client tools	server tools
execution	your systems	anthropic’s servers
response handling	you must return `tool_result`	results arrive automatically
content blocks	`tool_use`	`server_tool_use` + `web_search_tool_result`
definition	`input_schema` object	versioned `type` field

defining each type

client tool - you provide a schema, claude generates input, you execute it:

{
  name: "get_weather",
  description: "Get the current weather in a given location",
  input_schema: {
    type: "object",
    properties: {
      location: {
        type: "string",
        description: "The city and state, e.g. San Francisco, CA"
      }
    },
    required: ["location"]
  }
}

server tool - you specify a versioned type, anthropic handles execution:

{
  type: "web_search_20250305",
  name: "web_search",
  max_uses: 5
}

server tools use versioned types (e.g., web_search_20250305) to ensure compatibility across model versions.

different workflows

client tool workflow

you define the tool with a schema
claude returns a tool_use block with stop_reason: "tool_use"
you execute the tool on your systems
you return results via tool_result
claude formulates the final response

// client tool response
{
  "stop_reason": "tool_use",
  "content": [
    {
      "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": {"location": "San Francisco, CA"}
    }
  ]
}

you then execute and return:

messages.push({
  role: "user",
  content: [{
    type: "tool_result",
    tool_use_id: "toolu_01A09q90qw90lq917835lq9",
    content: "72°F, sunny"
  }]
});

server tool workflow

you specify the server tool in your request
claude executes the tool automatically
results are embedded in the response as web_search_tool_result
no tool_result needed from you
claude formulates the final response

// server tool response
{
  "content": [
    {
      "type": "text",
      "text": "I'll search for that information."
    },
    {
      "type": "server_tool_use",
      "id": "srvtoolu_01WYG3ziw53XMcoyKL4XcZmE",
      "name": "web_search",
      "input": { "query": "claude shannon birth date" }
    },
    {
      "type": "web_search_tool_result",
      "tool_use_id": "srvtoolu_01WYG3ziw53XMcoyKL4XcZmE",
      "content": [
        {
          "type": "web_search_result",
          "url": "https://en.wikipedia.org/wiki/Claude_Shannon",
          "title": "Claude Shannon - Wikipedia",
          "encrypted_content": "..."
        }
      ]
    },
    {
      "type": "text",
      "text": "Claude Shannon was born on April 30, 1916...",
      "citations": [...]
    }
  ]
}

the key difference: server tool results are already in the response. anthropic executed the search; you just continue the conversation.

handling both in an agentic loop

when building an agentic loop that uses both tool types, you need to distinguish between them:

if (response.stop_reason === "tool_use") {
  messages.push({ role: "assistant", content: response.content });

  // find client-side tools that need results
  const clientToolBlocks = response.content.filter(
    (block): block is Anthropic.ToolUseBlock =>
      block.type === "tool_use"
  );

  if (clientToolBlocks.length > 0) {
    // execute client tools and return results
    const toolResults: Anthropic.ToolResultBlockParam[] = [];
    for (const block of clientToolBlocks) {
      const result = await executeMyTool(block.name, block.input);
      toolResults.push({
        type: "tool_result",
        tool_use_id: block.id,
        content: result,
      });
    }
    messages.push({ role: "user", content: toolResults });
  }
}

// for server tools, check for server_tool_use blocks
const hasServerTool = response.content.some(
  block => block.type === "server_tool_use"
);

the server_tool_use block type tells you anthropic is handling execution. the results come back as web_search_tool_result blocks in the same response.

common mistake: sending empty tool_result for server tools

a common bug is treating server tools like client tools:

// wrong: sending tool_result for server-side tools
for (const block of response.content) {
  if (block.type === "tool_use") {
    toolResults.push({
      type: "tool_result",
      tool_use_id: block.id,
      content: "",  // what would you even put here?
    });
  }
}

this happens when code is adapted from client-tool examples without accounting for server tools. sending empty tool_result messages injects noise into the conversation - the model may interpret it as the tool failing.

pricing

server tools have usage-based pricing in addition to token costs. from the web search documentation:

Web search is available on the Claude API for $10 per 1,000 searches, plus standard token costs for search-generated content.

the usage is tracked in the response:

"usage": {
  "input_tokens": 105,
  "output_tokens": 6039,
  "server_tool_use": {
    "web_search_requests": 1
  }
}

key points

client tools: you define, you execute, you return tool_result
server tools: you specify, anthropic executes, results arrive automatically
detect by block type: tool_use vs server_tool_use
never send tool_result for server tools - the results are already in the response
server tools are versioned (e.g., web_search_20250305) for compatibility

references

tool use with claude - official documentation
web search tool - server tool example with pricing

migrating to zig 0.15: the roadblocks nobody warned you about

2025-10-24T00:00:00+00:00

i built a command-line security tool to analyze shell scripts before executing them (preventing those dangerous curl | bash situations). starting with zig 0.15 meant hitting every breaking change head-on. here’s what actually broke and how to fix it.

the project: safe-curl

the tool analyzes shell scripts for malicious patterns:

recursive file deletion (rm -rf /)
code obfuscation (base64 decoding, eval)
privilege escalation (sudo)
remote code execution

zig seemed perfect for a simple CLI tool with minimal dependencies. then i hit the 0.15 changes.

source: safe-curl on github

roadblock 1: arraylist requires allocator everywhere

the change: zig 0.15 replaced std.ArrayList with std.array_list.Managed as the default. the “managed” variant now requires passing an allocator to every method call.

official reasoning: zig 0.15 release notes explain: “Having an extra field is more complicated than not having an extra field.” the unmanaged variant is now the primary implementation, with the managed version as a wrapper.

what broke

my initial attempt looked like this:

const Finding = struct {
    severity: Severity,
    message: []const u8,
    line_num: usize,
};

const AnalysisResult = struct {
    findings: std.ArrayList(Finding),

    fn init(allocator: std.mem.Allocator) AnalysisResult {
        return .{
            .findings = std.ArrayList(Finding).init(allocator),
        };
    }

    fn addFinding(self: *AnalysisResult, finding: Finding) !void {
        try self.findings.append(finding);  // Error: missing allocator
    }
};

error message:

error: expected 2 arguments, found 1

the fix

you have two options in 0.15:

option 1: store the allocator and pass it to methods

const AnalysisResult = struct {
    findings: std.ArrayList(Finding),
    allocator: std.mem.Allocator,  // Store allocator

    fn init(allocator: std.mem.Allocator) AnalysisResult {
        return .{
            .findings = std.ArrayList(Finding).init(allocator),
            .allocator = allocator,
        };
    }

    fn addFinding(self: *AnalysisResult, finding: Finding) !void {
        try self.findings.append(self.allocator, finding);  // Pass allocator
    }

    fn deinit(self: *AnalysisResult) void {
        self.findings.deinit(self.allocator);  // Pass here too
    }
};

option 2: use the unmanaged variant

const AnalysisResult = struct {
    findings: std.ArrayListUnmanaged(Finding),

    fn init() AnalysisResult {
        return .{
            .findings = .{},  // Empty initialization
        };
    }

    fn addFinding(self: *AnalysisResult, allocator: std.mem.Allocator, finding: Finding) !void {
        try self.findings.append(allocator, finding);
    }

    fn deinit(self: *AnalysisResult, allocator: std.mem.Allocator) void {
        self.findings.deinit(allocator);
    }
};

i went with option 1 for familiarity, but option 2 is more idiomatic in 0.15.

why this change?

the zig team explains that storing the allocator in the struct adds complexity. with the unmanaged variant as default, you get:

simpler method signatures
static initialization support (.{})
explicit allocator lifetime management

trade-off: you pass the allocator everywhere, but your data structures are cleaner.

roadblock 2: empty struct initialization `.{}`

the pattern: zig 0.15 introduced a shorthand for empty struct initialization.

what this enables

before, initializing an empty arraylist required:

var findings = std.ArrayList(Finding).init(allocator);

now you can use struct field inference:

const AnalysisResult = struct {
    findings: std.ArrayList(Finding),

    fn init(allocator: std.mem.Allocator) AnalysisResult {
        return .{
            .findings = .{},  // Compiler infers std.ArrayList(Finding).init(allocator)
            .allocator = allocator,
        };
    }
};

this syntax confused me initially because .{} looks like an empty struct literal, but it actually calls the appropriate init function based on the field type.

when it works: field type is clear from context when it breaks: compiler can’t infer the type

var list: std.ArrayList(Item) = .{};  // Works
var list = .{};  // Error: cannot infer type

roadblock 3: process api breaking changes

the change: std.process.Child.run() return type changed significantly.

what broke

fn fetchFromUrl(allocator: std.mem.Allocator, url: []const u8) ![]const u8 {
    const result = try std.process.Child.run(.{
        .allocator = allocator,
        .argv = &[_][]const u8{ "curl", "-fsSL", url },
    });
    defer allocator.free(result.stderr);

    // This line broke
    if (result.term.Exited != 0) {
        allocator.free(result.stdout);
        return error.HttpRequestFailed;
    }

    return result.stdout;
}

error: no field named 'Exited' in union 'std.process.Child.Term'

the fix

the term field changed from having an Exited field to being a tagged union:

fn fetchFromUrl(allocator: std.mem.Allocator, url: []const u8) ![]const u8 {
    const result = try std.process.Child.run(.{
        .allocator = allocator,
        .argv = &[_][]const u8{ "curl", "-fsSL", url },
    });
    defer allocator.free(result.stderr);

    // Check the union variant properly
    switch (result.term) {
        .Exited => |code| {
            if (code != 0) {
                allocator.free(result.stdout);
                return error.HttpRequestFailed;
            }
        },
        else => {
            allocator.free(result.stdout);
            return error.ProcessFailed;
        },
    }

    return result.stdout;
}

this is more explicit about handling different termination types (signal, unknown, etc.).

roadblock 4: http client instability

the problem: zig’s std.http.Client is still evolving rapidly between versions.

what i tried

fn fetchFromUrl(allocator: std.mem.Allocator, url: []const u8) ![]const u8 {
    var client = std.http.Client{ .allocator = allocator };
    defer client.deinit();

    const uri = try std.Uri.parse(url);
    var server_header_buffer: [16384]u8 = undefined;

    var req = try client.open(.GET, uri, .{
        .server_header_buffer = &server_header_buffer,
    });
    defer req.deinit();

    try req.send();
    try req.wait();

    // ... read response
}

errors:

API mismatches between documentation and actual implementation
buffer size requirements unclear
response reading patterns changed between minor versions

the workaround

the zig 0.15 release notes acknowledge: “HTTP client/server completely reworked to depend only on I/O streams, not networking directly.”

this instability meant falling back to shelling out:

fn fetchFromUrl(allocator: std.mem.Allocator, url: []const u8) ![]const u8 {
    // Use curl as a fallback since the Zig HTTP client API is too unstable
    const result = try std.process.Child.run(.{
        .allocator = allocator,
        .argv = &[_][]const u8{ "curl", "-fsSL", url },
    });
    defer allocator.free(result.stderr);

    switch (result.term) {
        .Exited => |code| {
            if (code != 0) {
                allocator.free(result.stdout);
                return error.HttpRequestFailed;
            }
        },
        else => {
            allocator.free(result.stdout);
            return error.ProcessFailed;
        },
    }

    return result.stdout;
}

not ideal for a “zero dependency” tool, but pragmatic given the api churn.

roadblock 5: reader/writer overhaul (“writergate”)

the change: zig 0.15 completely redesigned std.io.Reader and std.io.Writer interfaces.

from the release notes: “A complete overhaul of the standard library Reader and Writer interfaces… designed to usher in a new era of performance and drastically reduce unnecessary copies.”

what changed

before (0.14):

const stdout = std.io.getStdOut().writer();
try stdout.print("Hello {s}\n", .{"world"});

after (0.15):

const stdout = std.fs.File.stdout();
try stdout.writeAll("Hello world\n");

// For formatted output, you need a buffer
var stdout_buffer: [4096]u8 = undefined;
var stdout_writer = stdout.writer(&stdout_buffer);
try stdout_writer.print("Hello {s}\n", .{"world"});

why this matters

the old api wrapped streams in multiple layers of abstraction. the new api:

builds buffering directly into reader/writer
supports zero-copy operations (file-to-file transfers)
provides precise error sets
enables vector i/o and advanced operations

but it requires more explicit buffer management.

my approach

i created a helper function to hide the complexity:

fn printf(allocator: std.mem.Allocator, comptime fmt: []const u8, args: anytype) !void {
    const stdout = std.fs.File.stdout();
    const msg = try std.fmt.allocPrint(allocator, fmt, args);
    defer allocator.free(msg);
    try stdout.writeAll(msg);
}

this allocates for the formatted string, but keeps the call sites clean:

try printf(allocator, "{s}[{s}]{s} {s}\n", .{
    color_code,
    severity_name,
    Color.NC,
    finding.message
});

roadblock 6: undefined behavior rules tightened

the change: zig 0.15 standardizes when undefined is allowed.

from the release notes: “Only operators which can never trigger Illegal Behavior permit undefined as an operand.”

what this means

// This now errors at compile time
const x: i32 = undefined;
const y = x + 1;  // Error: undefined used in arithmetic

// Safe uses of undefined
var buffer: [256]u8 = undefined;  // OK: just reserves space
const ptr: *u8 = undefined;  // OK: pointers can be undefined

this catches bugs earlier but requires more explicit initialization.

the practical impact

in my code, i couldn’t do:

var line_num: usize = undefined;
while (condition) : (line_num += 1) {  // Error
    // ...
}

had to initialize explicitly:

var line_num: usize = 1;
while (condition) : (line_num += 1) {
    // ...
}

the verdict: worth it?

what’s better in 0.15:

5× faster debug compilation with x86 backend
clearer allocator lifetime management
more explicit, less magic
better performance fundamentals

what hurts:

breaking changes everywhere
documentation lags behind implementation
http client still unstable
community examples are all outdated

resources

zig 0.15.1 release notes - official breaking changes
std.ArrayList documentation - new allocator patterns
safe-curl source code - working 0.15 example

analyzing ocaml patterns in a real web app: what melange code actually looks like

2025-10-15T00:00:00+00:00

i built an options max pain calculator using melange (ocaml that compiles to javascript). this post analyzes the actual code—what patterns emerge, how the type system shapes the implementation, and what makes functional programming different in practice.

the project: what is max pain?

the app calculates “maximum pain” for stock options. this is the strike price where option holders (collectively) lose the most money at expiration.

here’s how it works:

the setup: every option contract has a strike price and open interest (number of contracts outstanding). at expiration, options are worth either:

calls: max(0, stock_price - strike) per share (100 shares per contract)
puts: max(0, strike - stock_price) per share

the calculation: for each possible strike price, calculate total value of all calls + all puts if the stock expires at that price. the strike where this total is minimized is “max pain”—where option holders lose the most.

why it matters: some traders believe stocks tend to gravitate toward max pain at expiration due to market maker hedging. whether that’s true is debatable, but it’s an interesting calculation that requires processing options chain data.

the algorithm:

fetch all option contracts (calls and puts) for an expiration date
extract unique strike prices
for each strike, sum up intrinsic value of all options if stock expires there
find the strike with minimum total value

live demo: options-max-pain.pages.dev

this problem shows where ocaml differs from javascript/typescript:

json parsing: requires explicit decoders instead of just calling .json()
data transformations: immutable pipelines with fold instead of mutable loops
error handling: option types and pattern matching instead of null checks
finding the minimum: explicit handling of empty lists (can’t just return undefined)

let’s analyze how these patterns appear in the actual code.

pattern 1: types as documentation

ocaml makes you define your data structures upfront:

type optionContract = {
  strike: float,
  openInterest: int,
  optionType: string, /* "call" or "put" */
};

type painByStrike = {
  strike_price: float,
  call_pain: float,
  put_pain: float,
  total_pain: float,
};

type maxPainResult = {
  strikePrice: float,
  totalPain: float,
  painBreakdown: list(painByStrike),
};

what’s different from typescript:

these aren’t optional annotations you can skip with any
the compiler tracks every field access and ensures consistency
misspell a field name anywhere? compile error
try to access a field that doesn’t exist? compile error
no runtime overhead—all this type information gets erased during compilation

this upfront ceremony pays off later. refactoring is mechanical: change a type, follow the compiler errors.

pattern 2: pipeline composition with the pipe operator

the core calculation looks like this:

let calculateMaxPain = (contracts: list(optionContract)): option(maxPainResult) => {
  /* get unique strike prices */
  let strikes =
    contracts
    |> List.map(c => c.strike)
    |> List.sort_uniq(compare);

  /* calculate pain for each strike */
  let painByStrikeList =
    strikes
    |> List.map(strike => {
         let (callPain, putPain) =
           contracts
           |> List.fold_left(
                (acc, contract) => {
                  let (accCallPain, accPutPain) = acc;
                  switch (contract.optionType) {
                  | "call" =>
                    let pain =
                      strike > contract.strike
                        ? (strike -. contract.strike)
                          *. float_of_int(contract.openInterest)
                          *. 100.0
                        : 0.0;
                    (accCallPain +. pain, accPutPain);
                  | "put" =>
                    let pain =
                      strike < contract.strike
                        ? (contract.strike -. strike)
                          *. float_of_int(contract.openInterest)
                          *. 100.0
                        : 0.0;
                    (accCallPain, accPutPain +. pain);
                  | _ => acc
                  };
                },
                (0.0, 0.0),
              );
         {
           strike_price: strike,
           call_pain: callPain,
           put_pain: putPain,
           total_pain: callPain +. putPain,
         };
       });

  /* find minimum pain strike */
  let minPainStrike =
    painByStrikeList
    |> List.fold_left(
         (minResult, current) =>
           switch (minResult) {
           | None => Some(current)
           | Some(min) =>
             current.total_pain < min.total_pain ? Some(current) : Some(min)
           },
         None,
       );

  /* return result */
  switch (minPainStrike) {
  | None => None
  | Some(minStrike) =>
    Some({
      strikePrice: minStrike.strike_price,
      totalPain: minStrike.total_pain,
      painBreakdown: painByStrikeList,
    })
  };
};

key patterns:

pipe operator (|>): threads data through transformations left-to-right. compare to javascript:

// javascript
const strikes = Array.from(
  new Set(contracts.map(c => c.strike))
).sort((a, b) => a - b);

// ocaml with pipes
let strikes =
  contracts
  |> List.map(c => c.strike)
  |> List.sort_uniq(compare);

fold_left for accumulation: like reduce() in javascript, but with explicit accumulator type. the tuple (accCallPain, accPutPain) carries both values through the fold. no mutation—each iteration returns a new tuple.

pattern matching for control flow: the switch on optionType is checked at compile time. the | _ => acc case handles unexpected values (though using a string here instead of a variant type is a missed opportunity).

return type is option(maxPainResult): this function might not have a result. what if the contracts list is empty? in javascript, you’d return null or undefined and hope callers check. in ocaml, the return type is explicitly option(maxPainResult):

type option('a) =
  | Some('a)
  | None;

callers must pattern match on the result:

switch (calculateMaxPain(contracts)) {
| Some(result) => /* use result.strikePrice */
| None => /* handle empty case */
}

the compiler won’t let you access result.strikePrice without handling the None case first. no silent failures, no forgotten null checks.

pattern 3: composable json decoders

the json parsing code is verbose but interesting:

|> Js.Promise.then_((json) => {
     open Melange_json.Of_json;

     /* compose decoders from small pieces */
     let decodeDetails = (json) => {
       let strikePrice = json |> field("strike_price", float);
       let contractType = json |> field("contract_type", string);
       (strikePrice, contractType);
     };

     let decodeResult = (json) => {
       let (strike, optionType) = json |> field("details", decodeDetails);
       let openInterest = json |> field("open_interest", int);
       {strike, openInterest, optionType};
     };

     try {
       let contracts =
         json
         |> field("results", array(decodeResult))
         |> Array.to_list;

       callback(Ok(contracts));
       Js.Promise.resolve();
     } {
       | Melange_json.Of_json_error(error) => {
         let msg = Melange_json.of_json_error_to_string(error);
         callback(Error("JSON decode error: " ++ msg));
         Js.Promise.resolve();
       }
     };
   })

trade-offs here:

in typescript, you’d write:

const response = await fetch(url);
const data = await response.json();
const contracts = data.results; // hope it's the right shape

in ocaml, you build composable decoders:

decodeDetails decodes nested details object
decodeResult uses decodeDetails to decode each array item
final decoder: field("results", array(decodeResult))

upside: if the json doesn’t match, you get a specific error: “field ‘strike_price’ not found” or “expected float, got string”. the decoder tells you exactly what failed.

downside: you write the structure twice—once in the type definition, once in the decoder. every field requires explicit decoder logic.

what you gain: confidence. once decoded, the type system guarantees contracts is list(optionContract). no runtime type checking needed anywhere else in the codebase.

pattern 4: explicit nullability in react

the react component shows how the option type forces explicit error handling:

[@react.component]
let make = () => {
  let (result, setResult) = React.useState(() => None);
  let (error, setError) = React.useState(() => None);

  let handleCalculate = _ => {
    fetchOptionsData(
      ticker,
      expirationDate,
      response => {
        switch (response) {
        | Ok(contracts) =>
          let maxPain = calculateMaxPain(contracts);
          setResult(_ => maxPain);
        | Error(msg) => setError(_ => Some(msg))
        };
      },
    );
  };

  /* render jsx */
  <div>
    {switch (error) {
     | Some(msg) => <p> {React.string("Error: " ++ msg)} </p>
     | None => React.null
     }}
    {switch (result) {
     | Some({strikePrice, totalPain, painBreakdown}) =>
       <div>
         <p> {React.string("$" ++ Js.Float.toFixed(strikePrice))} </p>
       </div>
     | None => React.null
     }}
  </div>;
};

what’s happening:

state initialization: React.useState(() => None) creates state with an option type. error is option(string), result is option(maxPainResult). there’s no null or undefined—just None.

pattern matching in jsx: can’t access result.strikePrice directly. must pattern match:

switch (result) {
| Some({strikePrice, totalPain, painBreakdown}) => /* use it */
| None => React.null
}

the compiler won’t let you forget the None case.

explicit string wrapping: React.string() converts ocaml strings to react elements. this looks verbose, but it prevents accidentally rendering objects or functions. in typescript/react:

<p>{someObject}</p> // renders "[object Object]"

in ocaml:

<p> someObject </p> /* compile error: expected React.element, got object */

what the compiled javascript looks like

melange outputs readable javascript. here’s what calculateMaxPain compiles to:

function calculateMaxPain(contracts) {
  var strikes = List.sort_uniq(Caml_obj.compare, List.map(
    function (c) { return c.strike; },
    contracts
  ));

  var painByStrikeList = List.map(function (strike) {
    var match = List.fold_left(/* ... fold logic ... */, [0.0, 0.0], contracts);
    return {
      strike_price: strike,
      call_pain: match[0],
      put_pain: match[1],
      total_pain: match[0] + match[1]
    };
  }, strikes);

  // ... rest of function
}

observations:

no type annotations - all stripped during compilation
readable structure - mirrors the source ocaml
minimal runtime - just a few helper functions for lists and comparisons
no null checks - the compiler already verified everything

the types exist only at compile time. runtime javascript is clean and fast.

what i learned

the type system catches real bugs early

multiple times during development, i refactored the pain calculation logic. each time, the compiler caught every place that needed updating. change a field name? compiler shows every access. change a function signature? compiler shows every call site.

this isn’t theoretical. it saved me from shipping broken code.

json decoding is verbose but worth it

writing decoders feels like busy work at first. but when the api changed (polygon.io updated their response structure), i got immediate compile errors showing exactly which decoders needed updating. no silent failures. no runtime surprises.

pattern matching changes how you think

instead of defensive if (data && data.results && data.results.length > 0) checks everywhere, you model the states explicitly:

None when there’s no data
Some(data) when there is

the compiler ensures you handle both cases. no forgotten null checks.

the trade-off analysis

what you gain:

soundness - if it compiles, types are guaranteed correct
refactoring confidence - compiler guides you through changes
explicit error handling - no forgotten null checks or error cases
immutability by default - no accidental mutations

what you pay:

upfront type ceremony - define types, write decoders, handle all cases
smaller ecosystem - fewer libraries than typescript/javascript
build complexity - opam + dune + webpack/bundler setup
team learning curve - functional programming concepts aren’t mainstream

bottom line

analyzing this ocaml code shows functional programming isn’t just academic theory. the type system, pattern matching, and explicit error handling eliminate entire classes of bugs. but they require upfront investment in types and decoders.

the code is longer and more explicit than equivalent typescript. but it’s also more maintainable and correct by construction.

live demo: options-max-pain.pages.dev

lateral joins: the rails optimization nobody talks about

2025-10-09T00:00:00+00:00

i benchmarked lateral joins against window functions and N+1 queries for the classic “top N per group” problem in rails. lateral joins were 57% faster than window functions and 3.5x faster than N+1 queries.

here’s why almost nobody uses them, and why you should.

the problem: top N per group

you’ve hit this before. you have posts with comments. you want the top 3 highest-scored comments for each post.

# the N+1 approach (what most rails apps do)
@posts = Post.all
@posts.each do |post|
  post.comments.order(score: :desc).limit(3)
end

this works. it’s also slow. for 100 posts, that’s 101 queries.

most rails developers either live with the N+1 or preload everything into memory and filter in ruby. both options suck at scale.

there’s a better way hiding in plain sql: lateral joins.

what are lateral joins?

lateral joins let you write correlated subqueries that reference the outer query. think of it as a “for each row” loop at the database level.

SELECT posts.*, top_comments.*
FROM posts
LEFT JOIN LATERAL (
  SELECT comments.*
  FROM comments
  WHERE comments.post_id = posts.id  -- references outer query!
  ORDER BY score DESC
  LIMIT 3
) top_comments ON true

that WHERE comments.post_id = posts.id inside the subquery is the magic. for each post, postgres runs the inner query and limits to 3 results before doing anything else.

this means it’s only processing 300 rows (100 posts × 3 comments) instead of all 5,000 comments.

why nobody uses them in rails

no native activerecord support - you have to write raw sql
window functions exist - they solve the same problem and are easier to understand
the problem is rare - most apps don’t hit scale where this matters

but when you do need them, the performance difference is massive.

the benchmark

i built a minimal activerecord benchmark comparing four approaches:

N+1 queries (the rails way)
window functions (the smart way)
lateral joins (the postgres way)
preload all + ruby filter (the memory-heavy way)

dataset: 100 posts, 50 comments each = 5,000 total comments. task: find top 3 comments per post.

sqlite3 results (no lateral support)

Window function (1 query):      363.3 i/s
N+1 queries (100 queries):       70.0 i/s - 5.19x slower
Preload all + Ruby filter:       34.2 i/s - 10.61x slower

note: i/s = iterations per second (higher is better). 363.3 i/s means the query completed 363 times in one second, or ~2.75ms per iteration.

window functions are already 5x faster than N+1 queries. nice speedup. let’s stop there, right?

wrong.

postgresql results (with lateral joins)

LATERAL join (1 query):         130.9 i/s - FASTEST
Window function (1 query):       83.2 i/s - 1.57x slower
N+1 queries (100 queries):       37.0 i/s - 3.53x slower
Preload all + Ruby filter:       11.8 i/s - 11.08x slower

lateral joins are 57% faster than window functions.

that’s not a typo. same dataset, same queries, lateral joins just win.

why lateral is faster: the execution plan

let’s look at what postgres actually does.

window function approach (12.5ms)

SELECT posts.*, comments.*
FROM posts
INNER JOIN (
  SELECT comments.*,
         ROW_NUMBER() OVER (PARTITION BY post_id ORDER BY score DESC) as row_num
  FROM comments
) comments ON comments.post_id = posts.id
WHERE comments.row_num <= 3
ORDER BY posts.id, comments.score DESC

postgres has to:

scan all 5,000 comments
compute ROW_NUMBER for every single row
filter to row_num <= 3
join with posts

it processes all 5,000 comments even though we only need 300 results.

lateral join approach (5.6ms)

SELECT posts.*, top_comments.*
FROM posts
LEFT JOIN LATERAL (
  SELECT comments.*
  FROM comments
  WHERE comments.post_id = posts.id
  ORDER BY score DESC
  LIMIT 3
) top_comments ON true
ORDER BY posts.id, top_comments.score DESC

postgres does:

scan posts
for each post, find top 3 comments using index on (post_id, score)
stop after 3 rows per post

it only processes 300 comments total. the LIMIT happens inside the correlated subquery, so postgres can use indexes efficiently and bail early.

5.6ms vs 12.5ms - lateral is 2.2x faster in raw query execution time.

when the gap widens

the performance advantage scales with data volume. here’s where lateral really shines:

hot paths

if this query runs 1,000 times per second (homepage, api endpoint), lateral saves:

4.4ms × 1,000 = 4.4 seconds of total query time per second

that’s 4.4 seconds of database cpu you’re not paying for.

implementing lateral joins in rails

activerecord doesn’t support lateral natively, so you write raw sql. i usually wrap it in a scope:

class Post < ApplicationRecord
  has_many :comments

  def self.with_top_comments(limit = 3)
    sql = <<~SQL
      SELECT posts.*, top_comments.*
      FROM posts
      LEFT JOIN LATERAL (
        SELECT comments.*
        FROM comments
        WHERE comments.post_id = posts.id
        ORDER BY score DESC
        LIMIT #{sanitize_sql(limit)}
      ) top_comments ON true
      ORDER BY posts.id, top_comments.score DESC
    SQL

    connection.exec_query(sql)
  end
end

then use it like any query:

results = Post.with_top_comments(3)

you can also build it with arel if you want more composability:

def self.with_top_comments(limit = 3)
  lateral_query = Comment
    .where('comments.post_id = posts.id')
    .order(score: :desc)
    .limit(limit)
    .to_sql

  from("posts")
    .joins("LEFT JOIN LATERAL (#{lateral_query}) top_comments ON true")
    .select('posts.*, top_comments.*')
end

not as clean as activerecord, but not terrible either.

the complete benchmark code

#!/usr/bin/env ruby
require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'activerecord', '~> 7.0'
  gem 'pg'
  gem 'benchmark-ips'
end

require 'active_record'
require 'benchmark/ips'

# setup database
ActiveRecord::Base.establish_connection(
  adapter: 'postgresql',
  database: 'benchmark_db',
  username: ENV['USER']
)

# create schema
ActiveRecord::Schema.define do
  create_table :posts, force: true do |t|
    t.string :title
  end

  create_table :comments, force: true do |t|
    t.integer :post_id
    t.integer :score
  end

  add_index :comments, [:post_id, :score]
end

# models
class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post
end

# seed data
100.times do |i|
  post = Post.create!(title: "Post #{i}")
  50.times { Comment.create!(post_id: post.id, score: rand(1..100)) }
end

# benchmark approaches
def approach_lateral(top_n)
  sql = <<~SQL
    SELECT posts.*, top_comments.*
    FROM posts
    LEFT JOIN LATERAL (
      SELECT comments.*
      FROM comments
      WHERE comments.post_id = posts.id
      ORDER BY score DESC
      LIMIT #{top_n}
    ) top_comments ON true
  SQL

  ActiveRecord::Base.connection.exec_query(sql)
end

def approach_window(top_n)
  sql = <<~SQL
    SELECT posts.*, comments.*
    FROM posts
    INNER JOIN (
      SELECT comments.*,
             ROW_NUMBER() OVER (PARTITION BY post_id ORDER BY score DESC) as row_num
      FROM comments
    ) comments ON comments.post_id = posts.id
    WHERE comments.row_num <= #{top_n}
  SQL

  ActiveRecord::Base.connection.exec_query(sql)
end

Benchmark.ips do |x|
  x.report("LATERAL join") { approach_lateral(3) }
  x.report("Window function") { approach_window(3) }
  x.compare!
end

run with:

ruby lateral_join_benchmark_postgres.rb

it creates a test database, seeds data, runs benchmarks, and cleans up. takes about 30 seconds.

you can adjust the dataset size by changing NUM_POSTS and COMMENTS_PER_POST at the top of the script.

real-world use cases

reddit-style “show top comments per post”

class PostsController < ApplicationController
  def index
    @posts = Post.with_top_comments(5)
  end
end

instead of N+1 queries or eager loading thousands of comments, one lateral query gets exactly what you need.

analytics: “show top 10 products by revenue per category”

SELECT categories.*, top_products.*
FROM categories
LEFT JOIN LATERAL (
  SELECT products.*, SUM(order_items.price) as revenue
  FROM products
  JOIN order_items ON order_items.product_id = products.id
  WHERE products.category_id = categories.id
  GROUP BY products.id
  ORDER BY revenue DESC
  LIMIT 10
) top_products ON true

this would be brutal with window functions on millions of order items.

“most active users per region”

SELECT regions.*, active_users.*
FROM regions
LEFT JOIN LATERAL (
  SELECT users.*, COUNT(activities.id) as activity_count
  FROM users
  JOIN activities ON activities.user_id = users.id
  WHERE users.region_id = regions.id
  GROUP BY users.id
  ORDER BY activity_count DESC
  LIMIT 20
) active_users ON true

lateral lets you push down the LIMIT before aggregating. huge win.

window functions vs lateral: when to use which

use window functions when:

you need all rows with ranking metadata (e.g., show every comment with its rank)
database doesn’t support lateral (mysql pre-8.0.14)
query is already fast enough

use lateral when:

you only need top N per group
dataset is large (10k+ rows per group)
query is on a hot path
you have proper indexes on join/order columns

lateral requires indexes on (group_column, order_column) to be fast. in our case: (post_id, score).

without indexes, lateral can actually be slower than window functions because it’s running N correlated subqueries.

other databases

postgresql: native support since 9.3 (2013). works great.

sqlite: added in 3.39.0 (2022), but not widely deployed. most rails apps on older sqlite.

mysql: supports lateral since 8.0.14 (2019). same syntax, slightly different optimizer behavior.

sql server: uses CROSS APPLY instead of CROSS JOIN LATERAL. same concept.

gotchas

correlated subquery performance: lateral executes the inner query for each outer row. bad indexes = bad performance.

result shape: lateral returns flattened rows. you get (post1, comment1), (post1, comment2), (post1, comment3) not nested objects. you have to group in ruby if you need nesting.

results = Post.with_top_comments(3)

posts_hash = results.group_by { |row| row['post_id'] }
posts_hash.each do |post_id, rows|
  post = rows.first
  comments = rows.map { |r| r.slice('comment_id', 'body', 'score') }
  # do something with post and comments
end

sql injection: if you’re interpolating user input into the lateral query, sanitize it:

lateral_query = Comment
  .where('comments.post_id = posts.id')
  .where('comments.author = ?', params[:author])  # sanitized
  .order(score: :desc)
  .limit(sanitize_sql(limit))
  .to_sql

why this matters

let’s be clear: N+1 queries are never fine. always eager load with includes() as your default. even on small datasets, N+1 queries waste round-trip time and teach bad habits.

but eager loading has its own problem for “top N per group” queries:

# eager loading still loads ALL comments
@posts = Post.includes(:comments).limit(100)

# then you filter in ruby
@posts.each do |post|
  top_3 = post.comments.sort_by(&:score).reverse.take(3)
end

you’ve loaded 5,000 comments into memory when you only need 300. that’s where lateral joins shine.

“can’t you eager load and then filter via SQL?” you can add SQL conditions to eager loading:

# this works for global filters
@posts = Post.includes(:comments)
  .where("comments.score > ?", 50)
  .references(:comments)

but there’s no way to do “top N per group” with includes(). activerecord’s eager loading loads entire associations - you can’t tell it to “only load the top 3 comments per post” without writing custom SQL. at that point, you might as well use lateral joins or window functions.

lateral joins aren’t a fix for lazy N+1 queries. they’re an optimization on top of good eager loading practices.

they solve the specific case where:

you’ve already eliminated N+1 queries with eager loading
but you’re loading too much data because you only need a subset per group
and filtering in ruby is inefficient at scale

the frustrating part is that rails doesn’t make this easy. no native activerecord support. you have to know postgres-specific features and drop down to raw sql.

but 57% faster query execution is worth writing some sql.

the bottom line

lateral joins are 1.57x faster than window functions for top N per group queries
they scale better with data volume
they require proper indexes to be fast
activerecord doesn’t support them natively
most rails apps don’t need them, but when you do, they’re a massive win

if you’re doing “top N per group” queries at scale, benchmark lateral joins. they might surprise you.

Debugging GitHub's Pinned Items Performance Issue in Safari

2025-10-07T00:00:00+00:00

I recently encountered a frustrating performance issue on GitHub’s profile settings page. When trying to pin repositories to my profile, each checkbox click would freeze for several seconds in Safari, but worked instantly in Firefox. This is a case study of how I used Safari’s Timelines tool to track down the root cause.

The Problem

On GitHub’s “Edit pinned items” dialog, clicking a checkbox to pin/unpin a repository had a 3-4 second delay before the checkbox would visually update - but only in Safari. Firefox was instant.

Initial Investigation

My first instinct was to determine if this was a backend or frontend issue. I opened Safari DevTools and recorded a Timeline profile while clicking a checkbox.

What I Found

The Timeline showed:

Minimal network activity - No API calls during the delay
Fast JavaScript execution - Initial event handlers completed in ~43ms
Long Layout & Rendering gap - Seconds of nothing happening
Massive JavaScript & Events block - A 3.5 second JavaScript execution block

This ruled out backend issues. The problem was entirely in the frontend.

Digging Deeper

I added some instrumentation to measure the checkbox event handling:

document.querySelectorAll('input[type="checkbox"]').forEach(checkbox => {
  checkbox.addEventListener('change', (e) => {
    console.time('checkbox-change');
  }, true);

  checkbox.addEventListener('change', (e) => {
    console.timeEnd('checkbox-change');
  }, false);
});

The result: 43ms. The JavaScript completed quickly, but the visual update took 3+ seconds.

This was confusing - if JavaScript finished in 43ms, why did it take seconds to render?

The Real Culprit

Looking more carefully at the Timeline, I noticed a second “Change Event Dispatched” that took 3.47 seconds. The 43ms I measured was just the first event handler.

Clicking into that slow event revealed the call stack pointing to bind.js:67 - part of GitHub Catalyst, GitHub’s web component framework.

The Catalyst Event System

GitHub uses Catalyst’s event binding system. Here’s the handleEvent function from bind.ts:

function handleEvent(event) {
  const el = event.currentTarget;
  for (const binding of bindings(el)) {
    if (event.type === binding.type) {
      const controller = el.closest(binding.tag);
      if (controllers.has(controller) && typeof controller[binding.method] === 'function') {
        controller[binding.method](event);
      }
      // ... Shadow DOM handling
    }
  }
}

This routes events to web component controller methods. In this case, the checkbox change event was calling ProfilePinsElement.limitPins().

The Performance Bug

Here’s the limitPins() method from GitHub’s source:

async limitPins() {
  await Promise.resolve()
  const checkboxes = this.checkboxes  // queries ALL checkboxes
  const used = checkboxes.filter(el => el.checked).length
  const limit = parseInt(this.getAttribute('max')!, 10)

  for (const el of checkboxes) {
    el.disabled = used === limit && !el.checked
  }

  const label = this.limitNotice.getAttribute('data-remaining-label') || ''
  const diff = limit - used
  this.limitNotice.textContent = `${diff} ${label}`
  this.limitNotice.classList.toggle('color-fg-danger', diff < 1)
}

The issue? Scale.

The dialog loads up to 100 pages of repositories (controlled by autoreloadCount = 100). For a user with 100+ repositories, this means:

this.checkboxes queries 100+ checkbox elements from the DOM
.filter(el => el.checked) loops through 100+ checkboxes
for (const el of checkboxes) loops through 100+ again setting .disabled
Each .disabled = ... triggers:
- Style recalculation (:disabled pseudo-class)
- MutationObserver callbacks
- Layout updates

100+ DOM property updates × Safari’s slower DOM performance = 3.5 seconds

Why Safari and Not Firefox?

Safari’s JavaScript engine (JavaScriptCore) and rendering engine (WebKit) are notoriously slower at certain DOM operations compared to Firefox’s SpiderMonkey and Gecko:

DOM property access (.checked, .disabled)
Style recalculation for pseudo-classes
MutationObserver callbacks
Layout computation

Firefox brute-forces through the inefficiency faster. Safari exposes the O(n²) behavior more clearly.

The Fix

The proper fix would be to only update checkboxes that actually need their disabled state changed:

async limitPins() {
  await Promise.resolve()
  const checkboxes = this.checkboxes
  const used = checkboxes.filter(el => el.checked).length
  const limit = parseInt(this.getAttribute('max')!, 10)

  for (const el of checkboxes) {
    const shouldDisable = used === limit && !el.checked
    // Only update if it needs to change
    if (el.disabled !== shouldDisable) {
      el.disabled = shouldDisable
    }
  }
  // ... rest
}

This would reduce the number of DOM mutations from 100+ to typically just a few. Most checkbox clicks don’t actually require any disabled state changes - only when you hit or leave the 6-item limit. By checking if the value needs to change first, you go from 100+ DOM mutations on every click to 0 mutations for most clicks, and ~94 mutations only when crossing the limit threshold.

Key Takeaways

Browser differences matter - Performance issues may only manifest in certain browsers
Safari Timelines is powerful - The visual breakdown of JavaScript, Layout, and Rendering clearly showed where time was spent
Look at ALL the events - I initially missed the second, slower event handler
Scale reveals bugs - This code probably works fine with 10 repos, but breaks down at 100+
Check your assumptions - I thought it was React, turns out it was web components (Catalyst)

Tools Used

Safari DevTools > Timelines tab - Visual breakdown of performance
Console timing - console.time() / console.timeEnd() for quick measurements
Network tab - Rule out backend issues
Call stack inspection - Find the actual slow code

If you’re experiencing performance issues in your web apps, Safari’s Timelines tool is an excellent starting point for investigation. The visual breakdown makes it easy to see whether you’re blocked on JavaScript, layout, rendering, or network.

why i refactored 1200 lines of vanilla js to vue

2025-10-03T00:00:00+00:00

i spent the last few hours refactoring my productivity app from vanilla javascript to vue 3. deleted 1200 lines of manual DOM manipulation and replaced it with 680 lines of reactive components. here’s why it was worth it and what i learned.

the app: get stuff done

quick context - this is an ai-powered goal-setting app that:

generates SMART goals using ai
breaks them into daily/monthly/yearly tasks
syncs to the cloud with clerk auth + stripe billing
handles subtasks, dark mode, pdf export, etc.

the original version was vanilla js + alpine.js (barely used) + a lot of innerHTML. it worked fine. but was kind of messy

the breaking point

here’s what finally pushed me to refactor:

// Before: updating a task checkbox
function toggleTaskComplete(category, index) {
    const goalSet = goalSets[activeGoalSetId];
    goalSet[category][index].completed = !goalSet[category][index].completed;

    // Now update localStorage
    localStorage.setItem('productivityGoalSets', JSON.stringify(goalSets));

    // Don't forget to update the DOM!
    renderTasks(goalSet[category], category);

    // And the stats section
    updateTaskStats();

    // Oh and save to the cloud
    saveToCloud();
}

four different places to update for one checkbox. miss any of them? bugs. guaranteed.

what vue fixes

1. reactivity eliminates manual dom updates

before:

function renderTasks(tasks, category) {
    let html = '<div class="task-list">';
    tasks.forEach((task, index) => {
        html += `
            <div class="task-item ${task.completed ? 'completed' : ''}">
                <input type="checkbox"
                       onchange="toggleTaskComplete('${category}', ${index})"
                       ${task.completed ? 'checked' : ''}>
                <span>${escapeHtml(task.text)}</span>
            </div>
        `;
    });
    html += '</div>';
    document.getElementById(`${category}-tasks`).innerHTML = html;
}

after:

<!-- TaskItem.vue -->
<template>
  <div class="task-item" :class="{ completed: task.completed }">
    <input
      type="checkbox"
      :checked="task.completed"
      @change="$emit('toggle-complete')">
    <span></span>
  </div>
</template>

<script setup>
defineProps({
  task: { type: Object, required: true }
});
defineEmits(['toggle-complete']);
</script>

no string concatenation. no manual xss protection. no inline event handlers. just declare what it should look like and vue handles the updates.

2. composables solve state management

the vanilla version had state everywhere:

global variables (goalSets, activeGoalSetId)
localStorage (primary source of truth)
dom state (checkbox values, input text)
cloud database (async sync)

vue’s composables pattern fixed this:

// composables/useGoals.js
const goalSets = ref({});
const activeGoalSetId = ref(null);

export function useGoals() {
  const activeGoalSet = computed(() => {
    return activeGoalSetId.value ? goalSets.value[activeGoalSetId.value] : null;
  });

  const taskStats = computed(() => {
    const allTasks = [
      ...(activeGoalSet.value?.today || []),
      ...(activeGoalSet.value?.month || []),
      ...(activeGoalSet.value?.year || [])
    ];

    return {
      total: allTasks.length,
      completed: allTasks.filter(t => t.completed).length,
      important: allTasks.filter(t => t.important).length
    };
  });

  const toggleTaskComplete = (category, index) => {
    const task = activeGoalSet.value[category][index];
    task.completed = !task.completed;
    saveGoalSets();  // handles localStorage + cloud sync
  };

  return {
    goalSets,
    activeGoalSet,
    taskStats,
    toggleTaskComplete
  };
}

single source of truth. computed properties auto-update the ui. no manual synchronization.

every component that needs goal state just calls useGoals() and gets the same reactive data:

<!-- AppInterface.vue -->
<script setup>
import { useGoals } from '@/composables/useGoals';

const { activeGoalSet, taskStats, toggleTaskComplete } = useGoals();
</script>

<template>
  <div>Total: </div>
  <div>Completed: </div>
</template>

change activeGoalSet anywhere in the app, and everything updates automatically.

3. components make code reusable

before, i had three copies of task rendering logic (one for each timeframe: today/month/year). different enough that extracting a function was awkward, similar enough that bugs appeared in all three.

after:

<!-- TaskList.vue - used for all three timeframes -->
<template>
  <div class="task-list">
    <TaskItem
      v-for="(task, index) in tasks"
      :key="index"
      :task="task"
      :category="category"
      :index="index"
      @toggle-complete="onToggleComplete(index)"
      @toggle-important="onToggleImportant(index)"
      @delete="onDelete(index)" />
  </div>
</template>

<script setup>
import { useGoals } from '@/composables/useGoals';

const props = defineProps({
  tasks: Array,
  category: String
});

const { toggleTaskComplete, toggleTaskImportant, deleteTask } = useGoals();

function onToggleComplete(index) {
  toggleTaskComplete(props.category, index);
}

// etc...
</script>

one component, three usages. fix a bug once, it’s fixed everywhere.

the migration process

day 1: infrastructure

started with the basics:

npm install vue@latest vue-router@latest @clerk/vue@latest
npm install --save-dev vite @vitejs/plugin-vue

created a minimal vite config:

// vite.config.js
import { defineConfig } from 'vite';
import vue from '@vitejs/plugin-vue';

export default defineConfig({
  plugins: [vue()],
  server: { port: 5173 }
});

backed up the old files (index.html → index-old.html) and created a new minimal entry point:

<!-- index.html -->
<!DOCTYPE html>
<html>
<head>
  <title>Get Stuff Done</title>
  <link href="./css/output.css" rel="stylesheet">
</head>
<body>
  <div id="app"></div>
  <script type="module" src="/src/main.js"></script>
</body>
</html>

day 2: composables first

extracted state management before building ui. this was key - having the composables working meant i could test each piece independently.

created useAuth() first (authentication is the foundation):

// composables/useAuth.js
import { ref, computed, watch } from 'vue';
import { useRouter } from 'vue-router';
import { useClerk, useUser } from '@clerk/vue';

const userUsage = ref({ goalGenerations: 0, isSubscribed: false });

export function useAuth() {
  const router = useRouter();
  const clerk = useClerk();
  const { user, isSignedIn } = useUser();

  const currentUser = computed(() => user.value);
  const isAuthenticated = computed(() => isSignedIn.value);

  watch(isSignedIn, async (signedIn) => {
    if (signedIn && user.value) {
      router.push('/app');
      loadUserData();
    } else {
      router.push('/');
    }
  });

  return {
    currentUser,
    isAuthenticated,
    userUsage,
    signIn,
    signOut
  };
}

then useGoals() for the core app logic. tested both in isolation before touching any ui code.

day 3: components

built components bottom-up (leaf components first):

TaskItem.vue - single task with checkbox
TaskList.vue - container for tasks
AuthSection.vue - sign in/out buttons
ProfileModal.vue / PaywallModal.vue - modals
LandingPage.vue / AppInterface.vue - top-level views

each component was small and focused. made debugging easy.

the results

code metrics

vanilla js: ~1,200 lines across 3 files
vue: ~680 lines across 15 files
43% reduction in code
100% reduction in manual dom manipulation

before/after: adding a feature

before (vanilla js):

to add a “priority” field to tasks:

update task object when creating (3 places)
update rendering logic (3 timeframes × 2 views = 6 places)
add ui controls (3 timeframes)
add event handlers (global functions)
update stats calculation
update cloud sync schema
update localStorage schema

estimated: 2-3 hours, high chance of missing something

after (vue):

add priority to task object in useGoals()
add ui in TaskItem.vue component
add handler that emits event
update computed stats in useGoals()

estimated: 30 minutes, low chance of bugs

performance

bundle size went up (added vue framework):

before: 45 kb
after: 87 kb (vue included), 32 kb gzipped

time to interactive actually got faster because vue’s virtual dom is more efficient than my string concatenation.

one weird trick: the shared state pattern

this was my favorite vue discovery. you can create shared state by defining refs outside the composable function:

// composables/useAuth.js

// State OUTSIDE the function = shared across all components
const currentUser = ref(null);
const isAuthenticated = ref(false);

export function useAuth() {
  // Every component that calls useAuth() gets the same refs
  return { currentUser, isAuthenticated };
}

now any component can get the auth state:

<!-- Header.vue -->
<script setup>
import { useAuth } from '@/composables/useAuth';
const { currentUser } = useAuth();
</script>
<template>
  <div></div>
</template>

<!-- Dashboard.vue -->
<script setup>
import { useAuth } from '@/composables/useAuth';
const { currentUser } = useAuth();  // Same user ref as Header!
</script>

it’s like a global store but type-safe and composable. no need for vuex/pinia for simple apps.

conclusion

if you’re maintaining a vanilla js app and:

you dread adding features
you’re debugging state sync issues
you’re copying code between components

…give vue a shot. the reactive primitives alone are worth it.

app: actuallydostuff.com

appendix: vue 3 primer

if you want to dive deeper into vue concepts, here’s a practical primer covering everything you need to know.

for react developers

if you’re coming from react, here’s the quick translation guide:

React	Vue 3	Key Difference
`useState(0)`	`ref(0)`	Access with `.value` in script, auto-unwrap in template
`useMemo()`	`computed()`	Same concept, different syntax
`useEffect()`	`watch()`	More explicit dependencies
JSX	Templates	HTML-like syntax, no curly braces for text
Custom Hooks	Composables	Very similar pattern
Props + Callbacks	Props + Events	Events instead of callback props
`useContext()`	Shared Composable	Define refs outside function
React Router	Vue Router	Similar API, `useRouter()` / `useRoute()`

quick example comparison:

react:

function Counter() {
  const [count, setCount] = useState(0);
  const doubled = useMemo(() => count * 2, [count]);

  useEffect(() => {
    console.log('Count changed:', count);
  }, [count]);

  return (
    <div>
      <div>{count}</div>
      <div>Doubled: {doubled}</div>
      <button onClick={() => setCount(count + 1)}>Increment</button>
    </div>
  );
}

vue:

<script setup>
import { ref, computed, watch } from 'vue';

const count = ref(0);
const doubled = computed(() => count.value * 2);

watch(count, (newVal) => {
  console.log('Count changed:', newVal);
});
</script>

<template>
  <div>
    <div>8</div>
    <div>Doubled: </div>
    <button @click="count++">Increment</button>
  </div>
</template>

key differences:

vue uses ref() instead of useState(), access with .value in script
templates use `` instead of jsx’s { }, and no .value needed
vue’s @click vs react’s onClick
can mutate state directly in vue (count++), no setter needed

reactivity system

vue 3’s reactivity is powered by javascript proxies.

ref() - reactive primitives:

import { ref } from 'vue';

const count = ref(0);           // number
const message = ref('Hello');   // string
const isActive = ref(true);     // boolean

// access/modify with .value
console.log(count.value);  // 0
count.value++;             // 1

// in templates, .value is automatic:
// <div>8</div>  ← no .value needed!

computed() - derived state:

import { ref, computed } from 'vue';

const tasks = ref([
  { text: 'Buy milk', completed: true },
  { text: 'Walk dog', completed: false }
]);

// recalculates when tasks changes
const completedCount = computed(() => {
  return tasks.value.filter(t => t.completed).length;
});

watch() - side effects:

import { ref, watch } from 'vue';

const username = ref('');

watch(username, (newValue, oldValue) => {
  console.log(`Changed from ${oldValue} to ${newValue}`);
  // save to localStorage, call api, etc.
});

template syntax

text interpolation:

<div></div>
<div>8</div>

attribute binding:

<img :src="imageUrl" :alt="imageAlt">
<div :class="{ active: isActive }">

event handling:

<button @click="handleClick">Click</button>
<button @click="count++">Increment</button>
<form @submit.prevent="onSubmit">  <!-- preventDefault() -->

conditional rendering:

<div v-if="isLoggedIn">Welcome back!</div>
<div v-else>Please log in</div>

list rendering:

<ul>
  <li v-for="task in tasks" :key="task.id">
    
  </li>
</ul>

two-way binding:

<input v-model="message">
<!-- for text inputs, equivalent to: -->
<input :value="message" @input="message = $event.target.value">

composables pattern

composables are reusable functions that encapsulate reactive state and logic.

// composables/useCounter.js
import { ref, computed } from 'vue';

export function useCounter(initialValue = 0) {
  const count = ref(initialValue);
  const doubleCount = computed(() => count.value * 2);

  function increment() {
    count.value++;
  }

  return { count, doubleCount, increment };
}

using it:

<script setup>
import { useCounter } from './composables/useCounter';

const { count, doubleCount, increment } = useCounter(10);
</script>

<template>
  <div>Count: 8</div>
  <div>Double: </div>
  <button @click="increment">+</button>
</template>

shared state pattern:

define refs outside the function to share across all components:

// composables/useAuth.js
import { ref } from 'vue';

// state outside = shared across all components
const currentUser = ref(null);
const isAuthenticated = ref(false);

export function useAuth() {
  function signIn(credentials) {
    currentUser.value = userData;
    isAuthenticated.value = true;
  }

  return { currentUser, isAuthenticated, signIn };
}

now every component that calls useAuth() gets the same currentUser and isAuthenticated.

props & events

parent passes data down:

<TaskItem :task="myTask" :index="0" />

child receives props:

<!-- TaskItem.vue -->
<script setup>
const props = defineProps({
  task: { type: Object, required: true },
  index: Number
});
</script>

child emits events to parent:

<script setup>
const emit = defineEmits(['toggle-complete', 'delete']);

function handleCheckbox() {
  emit('toggle-complete');
}
</script>

<template>
  <input @change="handleCheckbox">
</template>

parent listens:

<TaskItem @toggle-complete="onToggleComplete" />

lifecycle hooks

import { onMounted, onUnmounted } from 'vue';

onMounted(() => {
  console.log('Component mounted');
  // fetch data, add event listeners
});

onUnmounted(() => {
  console.log('Cleaning up');
  // remove event listeners, cancel timers
});

common gotchas

1. don’t mutate props:

<!-- ❌ don't do this -->
<script setup>
const props = defineProps({ task: Object });
props.task.completed = true;  // mutating prop!
</script>

<!-- ✅ do this instead -->
<script setup>
const emit = defineEmits(['update']);
emit('update', { ...props.task, completed: true });
</script>

2. v-for needs :key:

<!-- ❌ missing key -->
<div v-for="task in tasks"></div>

<!-- ✅ with key -->
<div v-for="task in tasks" :key="task.id"></div>

3. remember .value in script:

const count = ref(0);

// ❌ won't work
console.log(count);    // RefImpl object
count++;              // NaN

// ✅ correct
console.log(count.value);
count.value++;

templates don’t need .value, but scripts do.

quick reference

reactivity:

import { ref, reactive, computed, watch } from 'vue';

const count = ref(0);
const state = reactive({ name: 'Alice' });
const doubled = computed(() => count.value * 2);

watch(count, (newVal, oldVal) => {
  console.log(`Changed from ${oldVal} to ${newVal}`);
});

component communication:

<!-- parent -->
<Child :msg="message" @update="handleUpdate" />

<!-- child -->
<script setup>
defineProps({ msg: String });
const emit = defineEmits(['update']);
emit('update', newValue);
</script>

composable pattern:

// outside = shared
const state = ref({});

export function useFeature() {
  function method() { /* ... */ }
  return { state, method };
}

lifecycle:

import { onMounted, onUnmounted } from 'vue';

onMounted(() => console.log('Component mounted'));
onUnmounted(() => console.log('Cleanup'));

resources:

vue 3 docs
vue router docs
vueuse - collection of useful composables

materialized views made my dashboard 9000x faster

2025-10-03T00:00:00+00:00

i built a rails dashboard to analyze millions of records. first pass: painfully slow. adding materialized views with the scenic gem: absurdly fast.

here’s what i learned benchmarking against 100k users, 1M orders, and 5M user activities.

the problem

dashboard queries were hitting multiple tables with joins and aggregations. every page load meant scanning millions of rows, grouping, sorting… you know the drill.

daily sales query: 7.1 seconds per request. user engagement: same pain. this is fine for batch reports but unusable for a dashboard people actually look at.

materialized views in 30 seconds

regular views are just saved queries. postgres re-runs them every time.

materialized views are snapshots. postgres runs the query once, stores the results as a real table. subsequent reads? just SELECT from that cached table.

trade-off: data gets stale until you refresh. for dashboards where 5-60 minute staleness is fine, this works great.

scenic gem setup

scenic wraps postgres materialized views in rails migrations. feels native.

rails generate scenic:view daily_sales

creates two files:

migration file to create the view
sql file for your query

here’s the daily sales view:

SELECT
  DATE(orders.order_date) AS sale_date,
  COUNT(DISTINCT orders.id) AS total_orders,
  COUNT(DISTINCT orders.user_id) AS unique_customers,
  SUM(orders.total_amount) AS total_revenue,
  AVG(orders.total_amount) AS average_order_value,
  SUM(CASE WHEN orders.status = 'completed' THEN 1 ELSE 0 END) AS completed_orders,
  SUM(CASE WHEN orders.status = 'cancelled' THEN 1 ELSE 0 END) AS cancelled_orders
FROM orders
GROUP BY DATE(orders.order_date)
ORDER BY sale_date DESC

normal complex aggregation. but instead of running this every time, we materialize it:

class CreateDailySales < ActiveRecord::Migration[8.0]
  def change
    create_view :daily_sales, materialized: true
    add_index :daily_sales, :sale_date, unique: true
  end
end

now you can query it like any rails model:

class DailySale < ApplicationRecord
  def readonly?
    true
  end

  def self.refresh
    Scenic.database.refresh_materialized_view(table_name, concurrently: false, cascade: false)
  end
end

# in your controller
@daily_sales = DailySale.order(sale_date: :desc).limit(30)

postgres reads from a pre-computed table instead of scanning orders every time.

the benchmarks

i built 4 materialized views:

daily_sales - revenue metrics by day
top_products - product performance
user_engagements - customer lifetime value
category_revenues - category breakdowns

then benchmarked raw queries vs materialized views using benchmark-ips.

results

daily sales summary

raw query: 6.25 iterations/sec (160ms per query)
materialized view: 2,191 iterations/sec (456 microseconds per query)
350x faster

top products by revenue

raw query: 0.69 iterations/sec (1.44 seconds per query)
materialized view: 438 iterations/sec (2.28ms per query)
633x faster

user engagement metrics

raw query: 0.14 iterations/sec (7.12 seconds per query)
materialized view: 135 iterations/sec (7.39ms per query)
963x faster

category revenue analysis

raw query: 0.29 iterations/sec (3.41 seconds per query)
materialized view: 2,715 iterations/sec (368 microseconds per query)
9,252x faster

the user engagement query went from 7 seconds to 7 milliseconds. category revenue from 3.4 seconds to 368 microseconds.

how the queries work

let’s look at the user engagement view since it had the biggest pain:

SELECT
  users.id AS user_id,
  users.email,
  users.name,
  COUNT(DISTINCT orders.id) AS total_orders,
  SUM(orders.total_amount) AS lifetime_value,
  AVG(orders.total_amount) AS avg_order_value,
  COUNT(DISTINCT user_activities.id) AS total_activities,
  COUNT(DISTINCT CASE WHEN user_activities.activity_type = 'page_view' THEN user_activities.id END) AS page_views,
  MAX(orders.order_date) AS last_order_date,
  MAX(user_activities.occurred_at) AS last_activity_date,
  DATE_PART('day', NOW() - MAX(user_activities.occurred_at)) AS days_since_last_activity
FROM users
LEFT JOIN orders ON users.id = orders.user_id
LEFT JOIN user_activities ON users.id = user_activities.user_id
GROUP BY users.id, users.email, users.name
ORDER BY lifetime_value DESC NULLS LAST

two left joins across 100k users, 1M orders, and 5M activities. grouping, aggregating, sorting. every single time someone loads the dashboard.

materialized it? 100k rows pre-computed. SELECT with a simple ORDER BY and LIMIT.

the indexes matter too:

add_index :user_engagements, :user_id, unique: true

postgres can use the index for lookups. filtering by high-value customers? instant.

why materialized views are faster: database internals

ran EXPLAIN ANALYZE on both approaches to see what postgres is actually doing. the difference is wild.

raw query execution (7.1 seconds)

Limit  (cost=1666565.79..1666566.04 rows=100)
  Buffers: shared hit=383450 read=135233 written=1559
  ->  Sort  (top-N heapsort)
        ->  GroupAggregate  (rows=100000)
              ->  Merge Left Join  (rows=50455739)  ← 50 MILLION intermediate rows
                    ->  Gather Merge (parallel workers: 2)
                          ->  Incremental Sort
                                ->  Merge Left Join (users + orders)
                    ->  Materialize (user_activities, 5M rows)

what’s happening:

joins 100k users + 1M orders + 5M activities
creates 50 million intermediate rows
groups all 100k users
sorts by lifetime value
reads 135,233 disk blocks from storage
takes top 100

the query is scanning millions of rows, doing complex joins, aggregating, then sorting. postgres is working hard.

materialized view execution (7.4ms)

Limit  (cost=0.29..8.87 rows=100)
  Buffers: shared hit=103
  ->  Index Scan using index_user_engagements_on_user_id
        Order By: lifetime_value DESC

what’s happening:

uses index to read rows sorted by lifetime_value
reads 103 blocks (all from cache)
stops after 100 rows

no joins. no aggregation. no sorting. just reading pre-computed results.

buffer analysis: cache hits matter

postgres tracks how often data is read from RAM (cache hits) vs disk:

base tables getting hammered by raw queries:

order_items:      5.2M disk reads, 76% cache hit ❌
user_activities:  1.3M disk reads, 91% cache hit ❌
orders:           817K disk reads, 95% cache hit ⚠️

materialized views:

daily_sales:        37 disk reads, 99.88% cache hit ✅
user_engagements: 9,612 disk reads, 99.71% cache hit ✅
top_products:     1,168 disk reads, 99.87% cache hit ✅

disk reads are ~1000x slower than RAM. materialized views stay in cache because they’re small and accessed frequently.

query cost comparison

postgres estimates query cost before execution:

query	raw cost	view cost	ratio
daily sales	101,503	0.96	105,628x
user engagement	763,318	2.86	266,860x
top products	101,996	2.54	40,156x

these aren’t execution times, they’re cost units. includes disk I/O, CPU operations, memory usage. lower is better.

raw query for user engagement costs 763,318 units. materialized view: 2.86 units.

the memory problem: external sorts

raw daily sales query execution plan shows this:

Sort Method: external merge  Disk: 14208kB
  Worker 0: Disk: 12200kB
  Worker 1: Disk: 13736kB

sorting 1M rows doesn’t fit in work_mem, so postgres spills to disk. writes ~40MB of temporary files across 3 parallel workers.

disk I/O during sorting kills performance.

materialized views? no sorting needed. data is already sorted via indexes.

sequential scans vs index scans

checked how often postgres uses indexes vs scanning entire tables:

base tables:

orders:      5.5M index scans (99.99% index usage) ✅
users:       6.2M index scans (100% index usage)   ✅
products:    7.0M index scans (100% index usage)   ✅

every raw query hits these tables with index lookups. millions of operations putting load on the database.

materialized views:

category_revenues: 38,642 sequential scans (0 index scans) ✅
top_products:       5,988 sequential scans (0 index scans) ✅
daily_sales:            5 seq scans, 29,578 index scans   ✅

materialized views are small. sequential scans are actually faster than indexes for small tables (no index overhead).

real I/O impact

ran rails sql:analysis to get detailed buffer statistics:

raw user engagement query:

135,233 disk blocks read
383,450 cache blocks read
1,559 blocks written (temp data)
9.26 seconds execution

materialized view:

0 disk blocks read
103 cache blocks read
0 blocks written
0.0074 seconds execution

the raw query is doing 1300x more I/O. that’s why it’s slow.

tools for analysis

added comprehensive SQL analysis tools to the repo:

# full analysis report
rails sql:analysis

# shows: execution plans, buffer usage, cache hit ratios,
# index usage, query costs, table statistics

# analyze specific query
rails sql:analyze_query QUERY='SELECT * FROM orders WHERE status = "completed"'

# compare raw vs materialized views
rails benchmark:compare

the EXPLAIN ANALYZE output shows exactly what postgres is doing: parallel workers, sort methods, join types, buffer usage, actual row counts.

check out PERFORMANCE_ANALYSIS.md in the repo for the complete breakdown with execution plans and statistics.

refreshing the views

views get stale. you need to refresh them.

i use a background job with solid queue:

class RefreshMaterializedViewsJob < ApplicationJob
  queue_as :default

  def perform
    DailySale.refresh
    TopProduct.refresh
    UserEngagement.refresh
    CategoryRevenue.refresh
  end
end

scheduled hourly in production:

# config/recurring.yml
production:
  refresh_materialized_views:
    class: RefreshMaterializedViewsJob
    queue: default
    schedule: every hour

refreshing all 4 views takes about 27 seconds with my dataset. once an hour is negligible overhead for 350-9000x query speedups.

for larger views or high-traffic sites, use CONCURRENTLY:

def self.refresh
  Scenic.database.refresh_materialized_view(table_name, concurrently: true)
end

requires unique indexes but lets you refresh without locking the view. users can keep querying during refresh.

when this makes sense

materialized views work when:

you have complex aggregations that run often
data staleness of 5-60 minutes is acceptable
reads massively outnumber writes
the underlying query is expensive (>500ms)

don’t use them for:

real-time data requirements
simple queries already fast with indexes
write-heavy tables that change constantly

my dashboard checks all the boxes. analytics data where hour-old numbers are fine. users hitting the same queries hundreds of times per day.

the full setup

i open sourced the complete case study. includes:

production-ready schema (users, products, orders, activities)
4 materialized views with sql
seed script that generates millions of records
benchmark rake tasks
dashboard ui
automated refresh jobs

you can clone it and run benchmarks yourself:

git clone https://github.com/sngeth/scenic-materialized-views-demo
cd scenic-materialized-views-demo
bundle install
rails db:create db:migrate
rails db:seed
rails benchmark:refresh
rails benchmark:compare

customize data volume with env vars:

USERS_COUNT=50000 PRODUCTS_COUNT=5000 rails db:seed

some specifics on scenic

scenic handles view versioning like migrations. updating a view:

rails generate scenic:view daily_sales --version 2

creates daily_sales_v02.sql. modify the query, run migrations, scenic handles the swap.

you can also drop down to raw sql when needed:

ActiveRecord::Base.connection.execute("REFRESH MATERIALIZED VIEW CONCURRENTLY daily_sales")

scenic mostly stays out of your way. it’s a thin wrapper that makes postgres materialized views feel like rails.

monitoring refresh performance

track how long refreshes take:

def perform
  Rails.logger.info "Starting materialized views refresh..."
  start_time = Time.now

  DailySale.refresh
  Rails.logger.info "  ✓ DailySale refreshed"

  # ... other views

  elapsed_time = Time.now - start_time
  Rails.logger.info "Completed in #{elapsed_time.round(2)}s"
end

watch for degradation as data grows. if refreshes start taking too long, consider:

refreshing views separately with different schedules
using incremental refresh patterns
partitioning underlying tables

practical example: the dashboard controller

here’s how simple the controller gets:

class DashboardController < ApplicationController
  def index
    @daily_sales = DailySale.order(sale_date: :desc).limit(30)
    @top_products = TopProduct.order(total_revenue: :desc).limit(10)
    @category_revenues = CategoryRevenue.order(total_revenue: :desc)
    @top_users = UserEngagement.order(lifetime_value: :desc).limit(10)
  end
end

four simple queries. no joins, no aggregations, no complexity. just reading pre-computed data.

response time? 50-100ms total including rendering. used to be 10+ seconds with raw queries.

the views handle all the heavy lifting in the background refresh job.

cost analysis

refreshing 4 views takes 27 seconds every hour = 648 seconds per day.

without materialized views, if the dashboard gets hit 1000 times per day (conservative):

1000 requests × 4 queries × 3 seconds average = 12,000 seconds of query time
plus database load, connection pool pressure, etc.

the math checks out. background refresh overhead is tiny compared to saved query time.

edge cases

partial data during refresh: use CONCURRENTLY to avoid downtime, but it requires unique indexes and takes longer.

view dependencies: if views reference other views, refresh order matters. scenic handles this with cascade options.

schema changes: changing underlying tables requires updating and versioning the views. scenic makes this manageable with version files.

storage: materialized views duplicate data. monitor disk usage. my 4 views add maybe 50mb on top of 2gb of base tables. negligible.

wrapping up

350x to 9000x faster queries. 27 seconds of refresh time per hour. hour-old data that’s perfectly acceptable for analytics.

materialized views aren’t magic. they’re cached query results. but for dashboards on millions of rows, they transform unusable into instant.

the scenic gem makes them feel native to rails. write sql, run migrations, query like models.

check out the full repo if you want to try it. includes all the benchmarks, views, and a working dashboard you can load with test data.

tracking your wins with git

2025-09-22T00:00:00+00:00

another year down, and i’m trying to remember what i actually built this year. honestly? it’s all a blur.

you know the feeling. you’ve been shipping code consistently, fixing bugs, building features, but when someone asks “what did you accomplish this year?” your brain just goes blank. was that auth refactor in march or july? did i ship the analytics dashboard before or after the mobile redesign?

the problem with developer memory

we’re constantly context switching. one day you’re debugging a race condition in the payment flow, the next you’re building a new onboarding experience, then suddenly you’re optimizing database queries because the dashboard is slow. each task feels important in the moment, but they all blend together over months.

whether it’s performance reviews, job interviews, or just internal reflection, people want concrete examples of your impact. “tell us about a complex technical challenge you solved” or “describe how you improved system performance.” but when everything feels like just another tuesday, it’s hard to remember which wins were actually significant.

your git history is your accomplishment log

every commit you make is a timestamp of progress. your git history contains:

exact dates of when you shipped features
the complexity and scope of changes
how many bugs you fixed vs features you built
patterns in your work (are you always fixing the same types of issues?)
collaboration evidence (co-authored commits, code reviews)

the trick is turning that raw commit data into a coherent story of growth and impact.

the magic command

here’s what i fed into an LLM to generate my yearly summary:

# get a year's worth of commits with stats
git log --author="your-email@company.com" \
        --since="2024-01-01" \
        --until="2024-12-31" \
        --pretty=format:"%h|%ad|%s" \
        --date=short \
        --all | head -50

then prompt your favorite LLM with:

“analyze these git commits and create a technical accomplishments summary. group by major themes like features, bug fixes, performance improvements, and security. highlight the business impact and technical complexity. include specific metrics where possible.”

automating with a script

i’ve been using this technique for weekly standups too. here’s a script that automates the whole process:

#!/bin/bash
# git-standup.sh - Generate AI-powered standup reports from git commits

set -e

DAYS=${1:-7}  # Default to last 7 days
AUTHOR=${2:-$(git config user.email)}
ENV_FILE=${3:-~/.env}

# Source environment variables
source "$ENV_FILE"

# Get git commits
COMMITS=$(git log --author="$AUTHOR" \
    --since="$DAYS days ago" \
    --pretty=format:"%h|%ad|%s" \
    --date=short \
    --all \
    --no-merges | head -50)

# Prepare the prompt
PROMPT="Analyze these git commits and create a concise standup update. Focus on:
- What was accomplished (group similar work)
- Any blockers or challenges implied by the commits
- Key technical wins or improvements
- Format as: Completed, In Progress, Blockers, Notes

Commits (format: hash|date|message):
$COMMITS"

# Use OpenAI API
if [[ -n "$OPENAI_API_KEY" ]]; then
    ESCAPED_PROMPT=$(echo "$PROMPT" | jq -Rs .)

    curl -s https://api.openai.com/v1/chat/completions \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $OPENAI_API_KEY" \
        -d "{
            \"model\": \"gpt-4o-mini\",
            \"messages\": [{
                \"role\": \"user\",
                \"content\": $ESCAPED_PROMPT
            }],
            \"max_tokens\": 1000
        }" | jq -r '.choices[0].message.content'
fi

just add your OPENAI_API_KEY to ~/.env and run:

./git-standup.sh           # last 7 days
./git-standup.sh 3         # last 3 days
./git-standup.sh 14 your.email@company.com  # custom timeframe/author

what the analysis revealed

looking at my own year through this lens was… honestly pretty shocking. here’s what git actually tracked:

major feature developments:

webinar platform (2025)
- enhanced VCF platform to support webinars including:
  - post-registration functionality and user workflows
  - custom marketing capabilities for webinar events
  - early start/late end time configuration system
  - private webinar filtering for staff interfaces
- technical impact: enabled virtual employment workshops, expanding platform capabilities beyond traditional job fairs
virtual career fair (VCF) enhancements
- developed VCF featured jobs system - complete job highlighting and promotion feature
- built pre-event search functionality - allowing candidates to discover opportunities before events
- enhanced chat welcome message formatting with WYSIWYG input
- improved message template positioning and dropdown functionality
- created mobile-responsive interfaces for exhibitor lists and candidate interactions
event management & analytics
- built sold-out event handling system with automatic waitlist functionality
- created comprehensive messaging analytics with campaign details and performance tracking
- enhanced control center real-time statistics display

security & performance contributions:

security hardening

strengthened password requirements and implemented secure reset flows
prevented user enumeration attacks in authentication systems
replaced insecure staff password generation with secure reset links
added CSRF protection and input sanitization improvements

performance optimization

resolved N+1 query issues in candidate searches and exhibitor displays
optimized database queries and added proper indexing
implemented efficient search filtering with elasticsearch integration
added query optimization for large dataset operations

technical problem solving:

mobile & responsive design

fixed critical mobile responsiveness issues across VCF interfaces
resolved viewport and layout problems for exhibitor schedules
implemented x-teleport solutions for dropdown menu clipping issues
enhanced mobile chat functionality with proper input visibility

data export & reporting

built comprehensive CSV export systems for:
- staff organization users
- client job postings with missing columns
- candidate applications with enhanced filtering
- event folder candidate downloads with rep attendance data

UI/UX improvements

implemented advanced filtering systems with sidebar interfaces
created sortable lists for completed one-on-one meetings

business impact contributions:

event operations

enhanced attendee tracking with view confirmation systems
implemented booth management with presence and broadcasting fixes
created candidate folder filtering for improved exhibitor experience
built time zone handling for multi-region events

client tools

developed job deletion workflows with automatic credit refunds
enhanced job application search with advanced filtering options
created messaging template systems with positioning improvements
implemented draft job management capabilities

quality assurance & testing

fixed flaky test issues with proper mocking and stubbing
implemented integration specs for complex workflows
added test helpers for consistent testing patterns

recent high-impact work (2024-2025):

september 2025: sold-out event handling and waitlist system
august 2025: VCF enhancements and messaging analytics
july 2025: staff candidate tracking and view confirmation
june 2025: security hardening and password requirements

technical skills demonstrated:

full-stack ruby on rails development
javascript/stimulus frontend frameworks
elasticsearch implementation and optimization
database design and query optimization
real-time features with turbo streams
mobile-responsive design patterns
security best practices implementation

beyond the basic stats

the real value isn’t just counting commits. it’s seeing patterns:

what types of problems do you gravitate toward? my commits showed i spend a lot of time on integration challenges, mobile responsiveness, and real-time features.

when are you most productive? my commit timestamps revealed patterns i never noticed. heavy feature work in morning sprints, bug fixes and optimization in the afternoon.

what’s your technical growth path? the progression from simple bug fixes early in the year to building complete subsystems later shows clear skill development. commits touching multiple systems prove comfort with complex, cross-cutting changes.

staying on top of it

keep a running note of the big wins as they happen. git gives you the data, but you need to capture the context: why was this hard? what would’ve happened if you didn’t fix it? how many users did this impact?

your commits are proof you’ve been busy. turning them into a story of impact? that’s the difference between “i wrote code” and “i moved the business forward.”

how that cloudflare outage happened (and how to avoid it)

2025-09-14T00:00:00+00:00

so cloudflare had this massive outage recently. their tenant service api went down, taking the dashboard and a bunch of other apis with it. the root cause? a react useEffect dependency array bug that made their dashboard hammer the api with unnecessary requests.

here’s what went wrong…

the setup

they had a react component that needed to fetch data from their tenant service api. pretty standard stuff - throw it in a useEffect, call it a day:

useEffect(() => {
  fetchTenantData(config);
}, [config]);

looks fine, right? except config was an object that got recreated on every render.

why objects break dependency arrays

react’s dependency array uses Object.is() to check if dependencies changed (verified in react’s source - see packages/shared/objectIs.js). for primitives like strings and numbers, this works great:

Object.is('hello', 'hello') // true
Object.is(42, 42) // true

but for objects and arrays? different story:

Object.is({a: 1}, {a: 1}) // false!
Object.is([1, 2], [1, 2]) // false!

even if the contents are identical, they’re different object references. so when you do this:

function Dashboard() {
  const config = { endpoint: '/api/tenant' }; // new object every render!

  useEffect(() => {
    fetchData(config);
  }, [config]); // this runs every single render
}

that effect runs on every render. every state update. every prop change. everything.

the cascade failure

here’s where it gets interesting. the dashboard wasn’t just making one extra call - it was making dozens. why? because the api call itself was probably updating state:

component renders → creates new config object
useEffect sees “new” dependency → calls api
api response updates state → triggers re-render
go to step 1

add multiple components doing this, users refreshing the page, and a recent service update that made the tenant service less stable… boom. you’ve got an outage.

how to fix it

few options here:

option 1: useMemo

memoize the object so it keeps the same reference:

const config = useMemo(() => ({
  endpoint: '/api/tenant'
}), []); // only create once

useEffect(() => {
  fetchData(config);
}, [config]); // now this only runs once

option 2: primitive dependencies

instead of passing the whole object, use primitive values:

const endpoint = '/api/tenant';

useEffect(() => {
  fetchData({ endpoint });
}, [endpoint]); // strings compare by value

option 3: move it outside

if the config never changes, define it outside the component:

const CONFIG = { endpoint: '/api/tenant' };

function Dashboard() {
  useEffect(() => {
    fetchData(CONFIG);
  }, []); // no dependency needed
}

how eslint might have made it worse

here’s the ironic part: the exhaustive-deps rule might have actually caused this bug!

{
  "rules": {
    "react-hooks/exhaustive-deps": "error"
  }
}

imagine you start with this:

function Dashboard() {
  const config = { endpoint: '/api/tenant' };

  useEffect(() => {
    fetchData(config);
  }, []); // eslint error: missing dependency 'config'
}

the linter complains that config is used but not in the deps array. so you “fix” it:

useEffect(() => {
  fetchData(config);
}, [config]); // linter happy, performance dead

now your effect runs on every render because config is a new object each time. the linter pushed you into the bug!

the real fix is understanding why the warning exists and addressing the root cause (memoizing the object, using primitives, or moving it outside the component) rather than just making the linter happy.

deploying rails to aws ecs fargate with application load balancer health checks

2025-09-13T00:00:00+00:00

a complete guide to containerizing a rails application and deploying it to aws ecs fargate with proper alb health check configuration.

overview

this guide walks through deploying a rails 8 application to aws using:

ecs fargate for serverless container orchestration
application load balancer (alb) for traffic routing and health checks
ecr for container image storage
secrets manager for secure configuration management
cloudwatch for logging

important security note: replace all placeholder values like [APP-NAME] and [ACCOUNT-ID] with your actual values. never commit these actual values to version control.

why ecs fargate over traditional deployment?

benefits of fargate:

no server management - aws handles os patches, scaling, security
pay-per-use pricing model
built-in integration with alb and other aws services
automatic scaling and load balancing
perfect for microservices and containerized applications

vs. traditional ec2:

no ssh access needed
no ami management
scales to zero for cost savings
simpler operations and ci/cd

prerequisites

aws cli configured with appropriate permissions
docker installed locally
rails application with health check endpoint

step 1: containerizing the rails application

1.1 create dockerfile

rails 8 generates an excellent production-ready dockerfile. key components:

# multi-stage build for smaller final image
ARG RUBY_VERSION=3.2.9
FROM ruby:$RUBY_VERSION-slim as base

# production environment configuration
ENV RAILS_ENV="production" \
    BUNDLE_DEPLOYMENT="1" \
    BUNDLE_PATH="/usr/local/bundle"

# thruster configuration for http proxy (recommended)
ENV TARGET_PORT=3000
ENV HTTP_PORT=80
EXPOSE 80

# use thruster to proxy port 80 → rails on port 3000
CMD ["./bin/thrust", "./bin/rails", "server", "-b", "0.0.0.0", "-p", "3000"]

1.2 health check endpoint

create a robust health check endpoint:

# app/controllers/health_controller.rb
class HealthController < ApplicationController
  def check
    render json: {
      status: "ok",
      timestamp: Time.current.iso8601,
      rails_version: Rails.version,
      environment: Rails.env
    }, status: :ok
  end
end

# config/routes.rb
Rails.application.routes.draw do
  get "health/check"
  # other routes...
end

1.3 docker entrypoint

simplify the entrypoint for containerized deployment:

#!/bin/bash -e
# bin/docker-entrypoint

# enable jemalloc for reduced memory usage
if [ -z "${LD_PRELOAD+x}" ]; then
    LD_PRELOAD=$(find /usr/lib -name libjemalloc.so.2 -print -quit)
    export LD_PRELOAD
fi

echo "starting rails server without database setup..."
exec "${@}"

1.4 platform compatibility

add linux platforms to gemfile.lock for cross-platform builds:

bundle lock --add-platform x86_64-linux aarch64-linux

step 2: aws infrastructure setup

2.1 create ecr repository

security note: use unique repository names to avoid conflicts with existing resources.

aws ecr create-repository --repository-name [APP-NAME] --region us-east-1

2.2 build and push docker image

# build for production architecture
docker buildx build --platform linux/amd64 -t [APP-NAME]:latest .

# tag and push to ecr
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com

docker tag [APP-NAME]:latest [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com/[APP-NAME]:latest
docker push [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com/[APP-NAME]:latest

2.3 vpc and networking setup

security consideration: this creates a new vpc. if you have existing infrastructure, consider using existing vpcs and subnets instead.

# create vpc
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --region us-east-1 --query Vpc.VpcId --output text)

# create public subnets in different azs
SUBNET1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.1.0/24 --availability-zone us-east-1a --query Subnet.SubnetId --output text)
SUBNET2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.2.0/24 --availability-zone us-east-1b --query Subnet.SubnetId --output text)

# internet gateway and routing
IGW_ID=$(aws ec2 create-internet-gateway --query InternetGateway.InternetGatewayId --output text)
aws ec2 attach-internet-gateway --vpc-id $VPC_ID --internet-gateway-id $IGW_ID

# route table configuration
RT_ID=$(aws ec2 create-route-table --vpc-id $VPC_ID --query RouteTable.RouteTableId --output text)
aws ec2 create-route --route-table-id $RT_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGW_ID
aws ec2 associate-route-table --subnet-id $SUBNET1 --route-table-id $RT_ID
aws ec2 associate-route-table --subnet-id $SUBNET2 --route-table-id $RT_ID

# enable auto-assign public ips
aws ec2 modify-subnet-attribute --subnet-id $SUBNET1 --map-public-ip-on-launch
aws ec2 modify-subnet-attribute --subnet-id $SUBNET2 --map-public-ip-on-launch

step 3: application load balancer configuration

3.1 security groups

security note: the alb security group allows traffic from the entire internet (0.0.0.0/0). this is appropriate for public web applications but consider restricting if needed.

# alb security group
ALB_SG=$(aws ec2 create-security-group \
  --group-name [APP-NAME]-alb-sg \
  --description "security group for alb" \
  --vpc-id $VPC_ID \
  --query GroupId --output text)

aws ec2 authorize-security-group-ingress \
  --group-id $ALB_SG \
  --protocol tcp --port 80 --cidr 0.0.0.0/0

# ecs security group
ECS_SG=$(aws ec2 create-security-group \
  --group-name [APP-NAME]-ecs-sg \
  --description "security group for ecs tasks" \
  --vpc-id $VPC_ID \
  --query GroupId --output text)

aws ec2 authorize-security-group-ingress \
  --group-id $ECS_SG \
  --protocol tcp --port 80 --source-group $ALB_SG

3.2 create application load balancer

# create alb
ALB_ARN=$(aws elbv2 create-load-balancer \
  --name [APP-NAME]-alb \
  --subnets $SUBNET1 $SUBNET2 \
  --security-groups $ALB_SG \
  --query 'LoadBalancers[0].LoadBalancerArn' --output text)

# create target group with health check configuration
TG_ARN=$(aws elbv2 create-target-group \
  --name [APP-NAME]-targets \
  --protocol HTTP --port 80 \
  --vpc-id $VPC_ID \
  --target-type ip \
  --health-check-path /health/check \
  --health-check-protocol HTTP \
  --health-check-interval-seconds 30 \
  --health-check-timeout-seconds 5 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 3 \
  --matcher HttpCode=200 \
  --query 'TargetGroups[0].TargetGroupArn' --output text)

# create listener
aws elbv2 create-listener \
  --load-balancer-arn $ALB_ARN \
  --protocol HTTP --port 80 \
  --default-actions Type=forward,TargetGroupArn=$TG_ARN

3.3 health check configuration details

the alb health check configuration is critical for proper operation:

path: /health/check - your rails endpoint
success codes: 200 - http ok status
interval: 30 seconds - check frequency
timeout: 5 seconds - request timeout
healthy threshold: 2 - consecutive successful checks to mark healthy
unhealthy threshold: 3 - consecutive failed checks to mark unhealthy

step 4: ecs configuration

4.1 create ecs cluster

aws ecs create-cluster --cluster-name [APP-NAME]-cluster

4.2 iam role for task execution

security note: check if ecstaskexecutionrole already exists in your account before creating it to avoid conflicts.

# create execution role (skip if it already exists)
aws iam create-role \
  --role-name ecsTaskExecutionRole \
  --assume-role-policy-document '{
    "Version":"2012-10-17",
    "Statement":[{
      "Effect":"Allow",
      "Principal":{"Service":"ecs-tasks.amazonaws.com"},
      "Action":"sts:AssumeRole"
    }]
  }'

# attach required policies
aws iam attach-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

aws iam attach-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-arn arn:aws:iam::aws:policy/SecretsManagerReadWrite

4.3 secrets management

store sensitive configuration in aws secrets manager:

# store rails master key
aws secretsmanager create-secret \
  --name [APP-NAME]/rails_master_key \
  --secret-string "$(cat config/master.key)"

4.4 ecs task definition

{
  "family": "[APP-NAME]-task",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::[ACCOUNT-ID]:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "[APP-NAME]",
      "image": "[ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com/[APP-NAME]:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 80,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {
          "name": "RAILS_ENV",
          "value": "production"
        },
        {
          "name": "RAILS_LOG_TO_STDOUT",
          "value": "true"
        }
      ],
      "secrets": [
        {
          "name": "RAILS_MASTER_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:[ACCOUNT-ID]:secret:[APP-NAME]/rails_master_key"
        }
      ],
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost/health/check || exit 1"],
        "interval": 30,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 60
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/[APP-NAME]",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

4.5 register task definition and create service

# create cloudwatch log group
aws logs create-log-group --log-group-name /ecs/[APP-NAME]

# register task definition
TASK_DEF_ARN=$(aws ecs register-task-definition \
  --cli-input-json file://ecs-task-definition.json \
  --query 'taskDefinition.taskDefinitionArn' --output text)

# create ecs service
aws ecs create-service \
  --cluster [APP-NAME]-cluster \
  --service-name [APP-NAME]-service \
  --task-definition $TASK_DEF_ARN \
  --desired-count 2 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={
    subnets=[$SUBNET1,$SUBNET2],
    securityGroups=[$ECS_SG],
    assignPublicIp=ENABLED
  }" \
  --load-balancers "targetGroupArn=$TG_ARN,containerName=[APP-NAME],containerPort=80"

step 5: deployment and testing

5.1 monitor deployment

# check service status
aws ecs describe-services --cluster [APP-NAME]-cluster --services [APP-NAME]-service --region us-east-1

# check target health
aws elbv2 describe-target-health --target-group-arn $TG_ARN --region us-east-1

# view logs
aws logs get-log-events --log-group-name /ecs/[APP-NAME] --log-stream-name [LOG-STREAM] --region us-east-1

5.2 test health checks

# get alb dns name
ALB_DNS=$(aws elbv2 describe-load-balancers \
  --load-balancer-arns $ALB_ARN \
  --query 'LoadBalancers[0].DNSName' --output text)

# test health check endpoint
curl http://$ALB_DNS/health/check

expected response:

{
  "status": "ok",
  "timestamp": "2025-09-13t21:28:14z",
  "rails_version": "8.0.2.1",
  "environment": "production"
}

common issues and solutions

container permission errors

issue: permission denied - bind(2) for "0.0.0.0" port 80

solution options:

option a: use non-privileged port (recommended for security)

# run as non-root user on port 3000
USER rails:rails
EXPOSE 3000
CMD ["./bin/rails", "server", "-b", "0.0.0.0", "-p", "3000"]

# update alb target group to port 3000
# update ecs security group to allow port 3000 from alb

option b: use thruster proxy (better performance)

# run as root to bind privileged port, but thruster drops privileges
ENV TARGET_PORT=3000
ENV HTTP_PORT=80
EXPOSE 80
CMD ["./bin/thrust", "./bin/rails", "server", "-b", "0.0.0.0", "-p", "3000"]

# benefits: http/2, compression, static file serving, caching
# security: thruster runs as root but rails process runs as rails user

thruster benefits you lose with option a:

http/2 support
automatic compression (gzip/brotli)
static file serving optimizations
built-in caching
x-sendfile support for efficient file downloads

health check failures

issue: alb showing 502/503 errors

solutions:

verify health check path matches your rails route
ensure container is listening on the correct port
check security group allows alb → ecs communication
review container logs for startup errors

platform compatibility

issue: exec format error in container logs

solution: build for correct architecture:

docker buildx build --platform linux/amd64 -t [APP-NAME]:latest .

security considerations

best practices implemented

secrets management: sensitive data stored in aws secrets manager
network security: security groups restrict access between components
least privilege: iam roles with minimal required permissions
container security: multi-stage builds reduce attack surface

security group rules

with option a (port 3000):

alb sg: allow http (80) from internet
ecs sg: allow http (3000) only from alb sg
alb handles port 80 → 3000 mapping

with option b (thruster):

alb sg: allow http (80) from internet
ecs sg: allow http (80) only from alb sg
thruster handles http optimizations

security trade-offs

option a (non-privileged port):

✅ better: no root processes
✅ better: principle of least privilege
❌ worse: no http/2, compression, caching
❌ worse: higher resource usage for static files

option b (thruster):

✅ better: http/2, compression, optimizations
✅ better: rails process still runs as non-root
⚠️ acceptable: thruster proxy runs as root (industry standard)
⚠️ acceptable: container isolation provides security boundary

recommendation: use thruster (option b) unless you have strict security requirements that prohibit any root processes.

cost optimization

fargate pricing factors

cpu allocation: 256 cpu units (0.25 vcpu)
memory allocation: 512 mb ram
running time: pay per second, minimum 1 minute

cost-saving tips

right-size resources: start small, monitor, and adjust
use spot pricing: for non-critical workloads
scale to zero: during low-traffic periods
monitor usage: cloudwatch metrics for optimization

monitoring and logging

cloudwatch integration

container logs: automatically streamed to cloudwatch
metrics: cpu, memory, network utilization
alarms: set up alerts for health check failures

health check monitoring

# create cloudwatch alarm for unhealthy targets
aws cloudwatch put-metric-alarm \
  --alarm-name "[APP-NAME]-unhealthy-targets" \
  --alarm-description "alb has unhealthy targets" \
  --metric-name UnHealthyHostCount \
  --namespace AWS/ApplicationELB \
  --statistic Average \
  --period 300 \
  --evaluation-periods 2 \
  --threshold 0 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=TargetGroup,Value=$TG_ARN

deployment commands summary

here’s the complete sequence of commands to deploy your rails app:

# 1. build and push image
docker buildx build --platform linux/amd64 -t [APP-NAME]:latest .
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com
docker tag [APP-NAME]:latest [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com/[APP-NAME]:latest
docker push [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com/[APP-NAME]:latest

# 2. create infrastructure
aws ecs create-cluster --cluster-name [APP-NAME]-cluster --region us-east-1
aws logs create-log-group --log-group-name /ecs/[APP-NAME] --region us-east-1

# 3. register task definition and deploy
aws ecs register-task-definition --cli-input-json file://ecs-task-definition.json --region us-east-1
aws ecs create-service --cluster [APP-NAME]-cluster --service-name [APP-NAME]-service --task-definition [APP-NAME]-task:1 --desired-count 2 --launch-type FARGATE --network-configuration "awsvpcConfiguration={subnets=[subnet-ids],securityGroups=[ecs-sg-id],assignPublicIp=ENABLED}" --load-balancers "targetGroupArn=[tg-arn],containerName=[APP-NAME],containerPort=80" --region us-east-1

# 4. test deployment
curl http://[alb-dns]/health/check

high availability and reliability patterns

current availability with 2 containers

our basic deployment with desired-count: 2 provides:

basic redundancy: if one container fails, traffic routes to the healthy container
rolling updates: ecs can update one container at a time without downtime
automatic recovery: failed containers are automatically restarted
estimated availability: ~99.5% (basic level)

achieving higher availability (99.9%+)

for production applications requiring maximum uptime, implement these patterns:

1. multi-az deployment with increased capacity

{
  "serviceName": "[APP-NAME]-service-ha",
  "desiredCount": 4,
  "deploymentConfiguration": {
    "maximumPercent": 200,
    "minimumHealthyPercent": 50,
    "deploymentCircuitBreaker": {
      "enable": true,
      "rollback": true
    }
  },
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": ["subnet-1a", "subnet-1b", "subnet-1c"],
      "securityGroups": ["sg-ecs"],
      "assignPublicIp": "ENABLED"
    }
  }
}

benefits:

4 containers across 3 availability zones
can lose entire az and maintain service
circuit breaker automatically rolls back failed deployments
deployment flexibility allows 100% capacity increase during deployments

2. auto scaling configuration

# create auto scaling target
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/[APP-NAME]-cluster/[APP-NAME]-service \
  --scalable-dimension ecs:service:DesiredCount \
  --min-capacity 4 \
  --max-capacity 20

# cpu-based scaling policy
aws application-autoscaling put-scaling-policy \
  --service-namespace ecs \
  --resource-id service/[APP-NAME]-cluster/[APP-NAME]-service \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-name cpu-scaling \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 70.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
    },
    "ScaleOutCooldown": 300,
    "ScaleInCooldown": 300
  }'

3. enhanced health checks

extend your health controller for comprehensive monitoring:

# app/controllers/health_controller.rb
class HealthController < ApplicationController
  def check
    health_data = {
      status: "ok",
      timestamp: Time.current.iso8601,
      rails_version: Rails.version,
      environment: Rails.env,
      uptime: uptime_seconds,
      memory: memory_usage,
      checks: {
        database: database_check,
        redis: redis_check,
        storage: storage_check
      }
    }

    if health_data[:checks].values.all? { |check| check[:status] == "ok" }
      render json: health_data, status: :ok
    else
      render json: health_data, status: :service_unavailable
    end
  end

  private

  def database_check
    ActiveRecord::Base.connection.execute("SELECT 1")
    { status: "ok", response_time_ms: 0 }
  rescue => e
    { status: "error", message: e.message }
  end

  def memory_usage
    return {} unless defined?(GC)

    {
      rss_mb: `ps -o rss= -p #{Process.pid}`.strip.to_i / 1024,
      gc_count: GC.count,
      heap_slots: GC.stat[:heap_live_slots]
    }
  end

  def uptime_seconds
    Process.clock_gettime(Process::CLOCK_UPTIME).to_i
  end
end

4. monitoring and alerting setup

# create comprehensive alarms
aws cloudwatch put-metric-alarm \
  --alarm-name "[APP-NAME]-high-cpu" \
  --alarm-description "high cpu utilization" \
  --metric-name CPUUtilization \
  --namespace AWS/ECS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 2 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=ServiceName,Value=[APP-NAME]-service Name=ClusterName,Value=[APP-NAME]-cluster

aws cloudwatch put-metric-alarm \
  --alarm-name "[APP-NAME]-response-time" \
  --alarm-description "high response time" \
  --metric-name TargetResponseTime \
  --namespace AWS/ApplicationELB \
  --statistic Average \
  --period 300 \
  --evaluation-periods 3 \
  --threshold 2.0 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=LoadBalancer,Value=[ALB-FULL-NAME]

5. graceful shutdown handling

rails applications handle sigterm gracefully by default with puma. configure ecs task definition for proper shutdown timing:

{
  "containerDefinitions": [{
    "stopTimeout": 30,
    "healthCheck": {
      "command": ["CMD-SHELL", "curl -f http://localhost/health/check || exit 1"],
      "interval": 15,
      "timeout": 5,
      "retries": 3,
      "startPeriod": 45
    }
  }]
}

availability comparison

pattern	containers	azs	estimated availability	recovery time
basic	2	2	99.5%	2-3 minutes
enhanced	4	3	99.9%	30 seconds
enterprise	6+	3+	99.95%+	10 seconds

cost vs availability trade-offs

basic deployment (2 containers):

cost: ~$30/month for small workloads
availability: sufficient for internal tools, staging
recovery: manual intervention may be needed

high availability (4+ containers):

cost: ~$60-120/month depending on scale
availability: production-ready for business applications
recovery: automatic with circuit breakers

enterprise (6+ containers + auto-scaling):

cost: variable, $100-500+/month based on traffic
availability: mission-critical applications
recovery: instant failover across multiple zones

deployment pipeline for ha

# 1. build and test
docker buildx build --platform linux/amd64 -t [APP-NAME]:latest .
docker run --rm -p 3000:80 [APP-NAME]:latest &
sleep 10
curl -f http://localhost:3000/health/check || exit 1

# 2. push to ecr
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com
docker tag [APP-NAME]:latest [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com/[APP-NAME]:$(git rev-parse --short HEAD)
docker push [ACCOUNT-ID].dkr.ecr.us-east-1.amazonaws.com/[APP-NAME]:$(git rev-parse --short HEAD)

# 3. update task definition with new image
sed "s/:latest/:$(git rev-parse --short HEAD)/g" ecs-task-definition.json > ecs-task-definition-$(git rev-parse --short HEAD).json
aws ecs register-task-definition --cli-input-json file://ecs-task-definition-$(git rev-parse --short HEAD).json

# 4. update service (ecs handles rolling deployment)
aws ecs update-service \
  --cluster [APP-NAME]-cluster \
  --service [APP-NAME]-service \
  --task-definition [APP-NAME]-task:$(aws ecs list-task-definitions --family-prefix [APP-NAME]-task --status ACTIVE --sort DESC --max-items 1 --query 'taskDefinitionArns[0]' --output text | cut -d'/' -f2)

# 5. wait for deployment to complete
aws ecs wait services-stable --cluster [APP-NAME]-cluster --services [APP-NAME]-service

recommended ha configuration

for most production applications, this configuration provides excellent availability:

{
  "desiredCount": 4,
  "deploymentConfiguration": {
    "maximumPercent": 150,
    "minimumHealthyPercent": 75,
    "deploymentCircuitBreaker": {
      "enable": true,
      "rollback": true
    }
  },
  "healthCheckGracePeriodSeconds": 60
}

key benefits:

4 containers provide redundancy across az failures
75% minimum ensures 3 containers always running during deployments
circuit breaker prevents bad deployments from taking down service
reasonable costs while maintaining high availability

conclusion

this deployment approach provides:

scalable architecture that grows with your application
high availability across multiple azs with configurable redundancy levels
proper health monitoring with comprehensive alb and container health checks
security best practices with secrets management
cost-effective operations with serverless containers
reliability patterns including auto-scaling, circuit breakers, and graceful shutdowns

the combination of alb health checks, ecs service management, and proper application health endpoints creates a robust production deployment that can achieve 99.9%+ availability for business-critical applications.

for production environments, consider adding:

database integration (rds with multi-az)
ssl/tls termination at alb
cdn (cloudfront) for global performance
comprehensive monitoring and alerting
backup and disaster recovery strategies
blue/green or canary deployments

repository structure

├── dockerfile                 # container definition
├── docker-compose.yml        # local development
├── ecs-task-definition.json  # ecs configuration
├── app/
│   └── controllers/
│       └── health_controller.rb
├── config/
│   └── routes.rb
└── bin/
    └── docker-entrypoint

this guide demonstrates a complete production-ready rails deployment on aws using modern containerization and infrastructure practices.

Building a Terminal IRC Client with Bubble Tea: A Deep Dive into Go's TUI Framework

2025-08-17T00:00:00+00:00

When I decided to build a modern IRC client for the terminal, I wanted something more sophisticated than the typical ncurses-based applications. Enter Bubble Tea, Charm’s powerful framework for building terminal user interfaces in Go. In this post, I’ll walk through how Bubble Tea works and how I used it to create a feature-rich IRC client.

What is Bubble Tea?

Bubble Tea is based on The Elm Architecture, bringing functional programming concepts to terminal UIs. It follows a simple pattern:

Model: Your application state
Update: A function that modifies state based on messages
View: A function that renders the current state

This architecture makes applications predictable, testable, and easy to reason about.

The Elm Architecture in Bubble Tea

According to the Bubble Tea repository, it’s “based on the functional design paradigms of The Elm Architecture”. Here’s how it works:

The Four Pillars

Every Bubble Tea program consists of:

Model: A struct that holds your entire application state
Init(): Returns the initial model and any startup commands
Update(msg tea.Msg): Receives messages and returns an updated model
View(): Takes the model and returns a string representation

Here’s the minimal interface every Bubble Tea program must implement:

type Model interface {
    Init() Cmd
    Update(Msg) (Model, Cmd)
    View() string
}

How It Works

A key concept: You implement these methods, but you never call them. The framework calls your code:

// What you write:
func (m IRCModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.KeyMsg:
        // Handle user input
    case msgConnected:
        // Handle IRC connection
    }
    return m, nil
}

func main() {
    p := tea.NewProgram(InitialModel())
    p.Run()  // You call this once, then Bubble Tea takes over
}

Inside p.Run(), Bubble Tea’s event loop calls your methods:

// What Bubble Tea does (you never write this):
for {
    select {
    case msg := <-p.msgs:
        model, cmd = model.Update(msg)  // Framework calls YOUR Update
        handleCommand(cmd)              // Framework handles returned command
        render(model.View())            // Framework calls YOUR View
    }
}

The Message Flow

The genius of this architecture is its unidirectional data flow:

    ┌─────────────────┐
    │                 │
    │     Model       │◄─────────────┐
    │                 │              │
    └────────┬────────┘              │
             │                       │
             ▼                       │
    ┌─────────────────┐              │
    │                 │              │
    │      View       │              │
    │                 │              │
    └────────┬────────┘              │
             │                       │
             ▼                       │
        Terminal                     │
         Display                     │
             │                       │
         User Input                  │
             │                       │
             ▼                       │
    ┌─────────────────┐              │
    │                 │              │
    │     Update      │──────────────┘
    │                 │
    └─────────────────┘

Messages flow in one direction: User input → Update → Model → View → Display.

Why this matters: In traditional UI programming, different parts of your app can modify state directly, leading to chaos:

// Traditional approach - multiple places changing state
func onKeyPress() {
    sidebar.addChannel("#golang")
    chatArea.updateUserCount(42)
    statusBar.setConnected(true)
    // Who changed what? When? In what order?
}

func onNetworkEvent() {
    sidebar.removeUser("bob")
    chatArea.addMessage("bob left")
    // Now sidebar and chat area might be out of sync!
}

With Bubble Tea’s unidirectional flow, only one place can change state:

// Bubble Tea approach - all changes go through Update
func (m IRCModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case UserLeftMsg:
        // Remove from users list
        delete(m.channelUsers[msg.channel], msg.user)
        // Add to message history  
        m.addMessage(msg.channel, fmt.Sprintf("%s left", msg.user))
        // State is always consistent!
    }
    return m, nil
}

This guarantees your UI state is always consistent because there’s only one path for changes.

Why This Matters for Terminal UIs

Traditional terminal UI libraries like ncurses use imperative updates:

// ncurses - imperative, stateful
mvprintw(10, 20, "Status: ");
if (connected) {
    attron(COLOR_PAIR(GREEN));
    printw("Connected");
} else {
    attron(COLOR_PAIR(RED));
    printw("Disconnected");
}
refresh();

With Bubble Tea’s Elm Architecture:

// Bubble Tea - declarative, functional
func (m Model) View() string {
    status := "Disconnected"
    if m.connected {
        status = "Connected"
    }
    return fmt.Sprintf("Status: %s", status)
}

The framework handles all the diffing, rendering, and optimization. You just describe what you want to see.

The Core Architecture

Here’s how I structured the IRC client using Bubble Tea:

type IRCModel struct {
    // UI components
    viewport        viewport.Model
    sidebarViewport viewport.Model
    textarea        textarea.Model
    
    // Application state
    allMessages     map[string][]string
    channels        map[string]bool
    channelUsers    map[string][]string
    activeChannel   string
    sidebarFocused  bool
    connected       bool
    
    // Layout
    width           int
    height          int
    sidebarWidth    int
}

The model contains both UI components (viewports, textarea) and application state (channels, messages, users). This separation allows for clean state management while leveraging Bubble Tea’s built-in components.

The Update Loop

The heart of any Bubble Tea application is the Update function, which handles all events:

func (m IRCModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    // Route input based on focus
    if !m.sidebarFocused {
        m.textarea, tiCmd = m.textarea.Update(msg)
        m.viewport, vpCmd = m.viewport.Update(msg)
    } else {
        m.sidebarViewport, svpCmd = m.sidebarViewport.Update(msg)
    }

    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        m.handleResize(msg)
        
    case tea.KeyMsg:
        return m.handleKeypress(msg)
        
    case msgConnected:
        return m.handleConnection(msg)
        
    case msgReceived:
        return m.handleIRCMessage(msg)
    }
    
    return m, tea.Batch(tiCmd, vpCmd, svpCmd)
}

Notice how different message types are handled separately. This pattern makes it easy to add new features without breaking existing functionality.

Custom Message Types

One powerful feature of Bubble Tea is custom message types. For IRC, I created specific messages for different network events:

type msgConnected struct {
    conn net.Conn
}

type msgReceived struct {
    text string
}

type errMsg error

These messages are sent through commands, which are functions that return messages:

func connectToIRC(server, nickname string) tea.Cmd {
    return func() tea.Msg {
        conn, err := net.Dial("tcp", server)
        if err != nil {
            return errMsg(err)
        }
        
        // Send IRC registration
        writer := bufio.NewWriter(conn)
        writer.WriteString(fmt.Sprintf("NICK %s\r\n", nickname))
        writer.WriteString(fmt.Sprintf("USER %s 0 * :%s\r\n", nickname, nickname))
        writer.Flush()
        
        return msgConnected{conn: conn}
    }
}

This approach keeps the UI responsive while handling network operations in the background.

Layout with Golden Ratio

For the visual design, I implemented a golden ratio layout to create pleasing proportions:

goldenRatio := 1.618
m.sidebarWidth = int(float64(msg.Width) / (goldenRatio + 1.0))

// Ensure reasonable bounds
if m.sidebarWidth < 15 {
    m.sidebarWidth = 15
}
if m.sidebarWidth > 25 {
    m.sidebarWidth = 25
}

This creates a sidebar that’s approximately 38% of the screen width, following the golden ratio principle for visual harmony.

Independent Scrolling with Focus Management

One challenge was implementing independent scrolling for the sidebar and main chat area. I solved this with a focus system:

case tea.KeyTab:
    m.sidebarFocused = !m.sidebarFocused
    if m.sidebarFocused {
        m.textarea.Blur()
    } else {
        m.textarea.Focus()
    }

When the sidebar is focused, arrow keys scroll through channels and users. When the chat is focused, they scroll through message history. This gives users full control over both areas independently.

Real-time Updates

IRC requires real-time message handling. I set up a continuous message loop:

func waitForMessage(conn net.Conn) tea.Cmd {
    return func() tea.Msg {
        scanner := bufio.NewScanner(conn)
        if scanner.Scan() {
            return msgReceived{text: scanner.Text()}
        }
        if err := scanner.Err(); err != nil {
            return errMsg(err)
        }
        return nil
    }
}

Each time a message is received, it triggers an update, parses the IRC protocol, and updates the appropriate channel or user list.

Styling with Lipgloss

Bubble Tea integrates beautifully with Lipgloss for styling. I created adaptive styles that work in both light and dark terminals:

var (
    titleStyle = lipgloss.NewStyle().
        Foreground(lipgloss.AdaptiveColor{Light: "#FFFFFF", Dark: "#FFFDF5"}).
        Background(lipgloss.AdaptiveColor{Light: "#0969DA", Dark: "#25A065"}).
        Padding(0, 1)

    userStyle = lipgloss.NewStyle().
        Foreground(lipgloss.AdaptiveColor{Light: "#1A7F37", Dark: "#7EE787"})
)

This ensures the client looks great regardless of the terminal’s color scheme.

Under the Hood: How Bubble Tea Prevents UI Blocking

Looking at the Bubble Tea source code reveals elegant concurrency patterns that keep the UI responsive. Here’s how it actually works:

The Message Channel Architecture

Bubble Tea uses a central message channel (p.msgs) as the communication hub:

func (p *Program) Send(msg Msg) {
    select {
    case <-p.ctx.Done():
    case p.msgs <- msg:
    }
}

This channel allows background goroutines to safely send messages back to the main event loop without blocking.

Command Execution in Goroutines

When you return a tea.Cmd, Bubble Tea spawns a goroutine to execute it:

func (p *Program) handleCommands(cmds chan Cmd) chan struct{} {
    go func() {
        for {
            select {
            case cmd := <-cmds:
                go func() {
                    // Each command runs in its own goroutine
                    msg := cmd()
                    p.Send(msg)  // Send result back to main loop
                }()
            }
        }
    }()
}

Key benefits:

Non-blocking execution - Long-running operations don’t freeze the UI
Automatic panic recovery - Crashed commands don’t take down the app
Graceful cleanup - Context cancellation stops all goroutines on exit

The Event Loop

The main event loop processes messages sequentially, ensuring thread safety:

func (p *Program) eventLoop(model Model, cmds chan Cmd) (Model, error) {
    for {
        select {
        case msg := <-p.msgs:
            // Update model (always on main thread)
            model, cmd = model.Update(msg)
            
            // Send new commands for background execution
            select {
            case cmds <- cmd:
            case <-p.ctx.Done():
                return model, nil
            }
            
            // Render immediately with updated model
            p.renderer.write(model.View())
        }
    }
}

Why This Design Matters

In your IRC client, when connectToIRC() makes a network call:

Network operation runs in background goroutine (doesn’t block UI)
User can still type, scroll, resize (UI remains responsive)
When connection completes, sends msgConnected (thread-safe communication)
Main loop processes message and updates model (sequential, no race conditions)
UI re-renders with new state (immediate visual feedback)

This is why you can have dozens of ongoing network operations (IRC reads, user lookups, etc.) without any UI lag or complex synchronization code.

Source Code

You can check out the complete IRC client source code at github.com/sngeth/chat.

ai-powered goal setting with smart caching

2025-07-29T00:00:00+00:00

built a goal-setting app that transforms problems into actionable smart goals, then provides conversational ai guidance for each task. the challenge? making ai interactions fast, cost-effective, and genuinely helpful. the solution? a multi-layer caching approach with semantic similarity matching.

the problem

traditional goal-setting apps are static lists. when users get stuck on “start a 4-day upper lower split” or “learn spanish,” they’re on their own. adding ai help seems obvious, but creates new problems:

cost escalation: every “how do I do this?” question hits expensive openai apis
response inconsistency: same question gets different answers due to ai randomness
context loss: follow-up questions lack memory of the original task
user frustration: waiting 2+ seconds for common questions like “what about sets and reps?”

how it works

implemented a simple flow: users describe problems, get structured goals, then ask follow-up questions about how to complete them.

layer 1: problem to smart goals

transforms user problems into specific, measurable, achievable, relevant, and time-bound goals:

// user input: "How do I quit smoking?"
const response = await fetch('/api/generate-goals', {
  method: 'POST',
  headers: { 'Authorization': `Bearer ${token}` },
  body: JSON.stringify({ problem: userInput })
});

// generates structured output:
{
  "today": ["throw away all cigarettes and smoking accessories"],
  "month": ["complete 30-day nicotine replacement therapy program"],
  "year": ["maintain smoke-free lifestyle for 365 consecutive days"]
}

uses embedding-based caching - similar problems get similar goals without redundant api calls.

layer 2: contextual how-to guides

each goal gets an interactive ❓ button for ai-powered guidance:

async function showHowTo(goalSetId, listType, taskId) {
  const task = goalSets[goalSetId][listType].find(t => t.id === taskId);
  
  const response = await fetch('/api/how-to', {
    method: 'POST',
    body: JSON.stringify({
      taskText: task.text,
      goalContext: {
        goalSetName: goalSet.name,
        timeframe: listType // today, month, year
      }
    })
  });
  
  return response.json(); // structured guide with steps, tips, timing
}

returns structured guidance:

overview: brief explanation of the task
steps: actionable numbered instructions
proTip: expert advice or common pitfalls
timeNeeded: realistic estimates
difficulty: complexity assessment

layer 3: conversational follow-ups

users can ask clarifying questions that maintain full context:

// conversation state preserved across questions
currentConversation = {
  taskId: "start_4_day_split_123",
  goalSetId: "fitness_goals", 
  listType: "today",
  messages: [
    { type: 'assistant', content: originalGuide },
    { type: 'user', content: "What about sets and reps?" },
    { type: 'assistant_followup', content: "For upper body days..." }
  ]
};

understanding openai embeddings

before diving into caching, it’s crucial to understand what embeddings are and why they’re revolutionary for text similarity.

what are embeddings?

embeddings convert text into high-dimensional vectors that capture semantic meaning. think of them as “fingerprints” for concepts - similar ideas get similar fingerprints.

openai’s text-embedding-3-small model transforms any text into a 1536-dimensional array of floating-point numbers:

async function getEmbedding(text, apiKey) {
  const response = await fetch('https://api.openai.com/v1/embeddings', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: 'text-embedding-3-small',
      input: text.toLowerCase().trim(),
      encoding_format: 'float'
    })
  });
  
  const data = await response.json();
  return data.data[0].embedding; // array of 1536 numbers
}

// "start a 4 day upper lower split" becomes:
// [0.12, -0.05, 0.73, 0.41, -0.19, 0.84, ...]  // 1536 floating-point numbers

the magic of semantic similarity

the breakthrough insight: semantically similar text produces similar vectors, even with completely different words.

// these all get embedded into nearby points in 1536-dimensional space:
"start a 4 day upper lower split"     // [0.12, -0.05, 0.73, 0.41, ...]
"begin 4-day upper/lower routine"     // [0.11, -0.04, 0.74, 0.42, ...]  
"initiate four day upper-lower workout" // [0.13, -0.06, 0.72, 0.40, ...]

// while completely different concepts are far apart:
"bake chocolate chip cookies"         // [0.89, 0.34, -0.12, -0.67, ...]

the ai model has learned through massive training that these phrases represent the same underlying concept, despite using different words.

measuring similarity with cosine similarity

cosine similarity measures the angle between two vectors in high-dimensional space, returning a score from 0.0 to 1.0:

function cosineSimilarity(vecA, vecB) {
  // calculate dot product (how aligned the vectors are)
  const dotProduct = vecA.reduce((sum, a, i) => sum + a * vecB[i], 0);
  
  // calculate magnitudes (lengths of the vectors)
  const magnitudeA = Math.sqrt(vecA.reduce((sum, a) => sum + a * a, 0));
  const magnitudeB = Math.sqrt(vecB.reduce((sum, b) => sum + b * b, 0));
  
  // cosine similarity = dot product / (magnitude_a * magnitude_b)
  return dotProduct / (magnitudeA * magnitudeB);
}

// similarity score meanings:
// 0.95-1.0: nearly identical meaning ("start workout" vs "begin workout")
// 0.8-0.95:  very similar concepts ("sets and reps" vs "repetitions")  
// 0.6-0.8:   related but different ("workout" vs "exercise routine")
// 0.0-0.6:   unrelated concepts ("workout" vs "baking cookies")

real similarity examples from the app

here are actual similarity scores measured during development:

// high task similarity (0.92) - correctly matched
cosineSimilarity(
  embedding("start a 4 day upper lower split"),
  embedding("begin 4-day upper/lower routine")
) // returns 0.92

// high question similarity (0.87) - correctly matched
cosineSimilarity(
  embedding("what about sets and reps?"),
  embedding("how many sets and repetitions?")
) // returns 0.87

// low task similarity (0.23) - correctly rejected  
cosineSimilarity(
  embedding("start a 4 day upper lower split"),
  embedding("bake chocolate chip cookies")
) // returns 0.23

// medium question similarity (0.64) - correctly rejected
cosineSimilarity(
  embedding("what about sets and reps?"),
  embedding("what about rest periods?")
) // returns 0.64 (related but different question)

why embeddings beat traditional approaches

traditional string matching would completely fail on natural language variations:

// string matching approach (brittle and inflexible)
function isSetRepsQuestion(question) {
  const q = question.toLowerCase();
  return q.includes("sets") && q.includes("reps");
}

isSetRepsQuestion("what about sets and reps?");     // ✅ true
isSetRepsQuestion("how many repetitions per set?"); // ❌ false (missed!)
isSetRepsQuestion("what's the rep and set scheme?"); // ❌ false (missed!)

// embedding approach (semantic understanding)
function isSetRepsQuestion(question) {
  const questionEmbedding = getEmbedding(question);
  const referenceEmbedding = getEmbedding("sets and reps");
  return cosineSimilarity(questionEmbedding, referenceEmbedding) >= 0.8;
}

isSetRepsQuestion("what about sets and reps?");     // ✅ true (0.95)
isSetRepsQuestion("how many repetitions per set?"); // ✅ true (0.84)  
isSetRepsQuestion("what's the rep and set scheme?"); // ✅ true (0.81)

semantic caching approach

armed with understanding embeddings, we can now implement intelligent caching using dual similarity matching:

dual similarity matching

the breakthrough insight: we need both the task context and question intent to match for a valid cache hit.

asking “what about sets and reps?” means completely different things for:

“start a 4 day upper lower split” → workout programming advice
“bake chocolate chip cookies” → nonsensical question

async function findSimilarFollowUp(taskText, question, env) {
  // convert both text inputs to semantic vectors  
  const taskEmbedding = await getEmbedding(taskText, env.OPENAI_API_KEY);
  const questionEmbedding = await getEmbedding(question, env.OPENAI_API_KEY);
  
  // search all cached responses
  const list = await env.GOAL_CACHE.list({ prefix: 'followup_' });
  
  for (const item of list.keys) {
    const cached = await env.GOAL_CACHE.get(item.name, 'json');
    
    // calculate semantic similarity for both dimensions
    const taskSimilarity = cosineSimilarity(taskEmbedding, cached.taskEmbedding);
    const questionSimilarity = cosineSimilarity(questionEmbedding, cached.questionEmbedding);
    
    // both thresholds must be exceeded for cache hit
    if (taskSimilarity >= 0.85 && questionSimilarity >= 0.8) {
      console.log(`Cache hit: task=${taskSimilarity.toFixed(3)}, question=${questionSimilarity.toFixed(3)}`);
      return cached.answer;
    }
  }
  
  return null; // no semantic match found
}

threshold tuning through experimentation

the similarity thresholds were determined through empirical testing:

task similarity: 0.85 - tasks must be very similar
- fitness tasks vs cooking tasks score ~0.2 (correctly rejected)
- “upper lower split” vs “upper/lower routine” score ~0.92 (correctly matched)
question similarity: 0.8 - questions can vary more linguistically
- “sets and reps” vs “repetitions” score ~0.87 (correctly matched)
- “sets and reps” vs “rest periods” score ~0.64 (correctly rejected)

too low and you get wrong answers for the wrong context, too high and you miss valid linguistic variations.

cache structure evolution: from duplication to referential integrity

the initial cache structure stored complete task embeddings with every follow-up question:

// inefficient: task embeddings duplicated across follow-ups
"followup_1703512345_abc123" → {
  taskText: "start a 4 day Upper Lower Split",
  taskEmbedding: [0.1, 0.2, 0.3, ...], // 1536 dimensions - DUPLICATED
  question: "What about sets and reps?",
  questionEmbedding: [0.7, 0.8, 0.9, ...],
  answer: "For upper body days, aim for 3-4 sets of 8-12 reps...",
  hitCount: 23,
  timestamp: "2024-01-15T10:30:00Z"
}

the optimized structure uses a relational approach with task hash references:

// task embeddings stored once and referenced by hash
"task_embedding_abc123" → {
  taskText: "start a 4 day Upper Lower Split", 
  taskEmbedding: [0.1, 0.2, 0.3, ...], // 1536 dimensions - STORED ONCE
  taskHash: "abc123",
  usageCount: 15,
  timestamp: "2024-01-15T10:30:00Z"
}

// follow-up entries reference the task by hash
"followup_v2_abc123_def456_1703512345" → {
  taskHash: "abc123", // FOREIGN KEY REFERENCE
  question: "What about sets and reps?",
  questionEmbedding: [0.7, 0.8, 0.9, ...],
  answer: "For upper body days, aim for 3-4 sets of 8-12 reps...",
  hitCount: 23,
  timestamp: "2024-01-15T10:30:00Z"
}

task embedding lookup mechanism

when a cached follow-up is found, the system performs a two-step lookup to verify task similarity:

// step 1: find potential follow-up cache matches
const followupList = await env.GOAL_CACHE.list({ prefix: 'followup_v2_' });

for (const item of followupList.keys) {
  const cached = await env.GOAL_CACHE.get(item.name, 'json');
  
  if (cached && cached.taskHash && cached.questionEmbedding) {
    // step 2: lookup task embedding by hash reference
    const cachedTaskEmbedding = await getTaskEmbedding(cached.taskHash, env);
    
    if (cachedTaskEmbedding) {
      // step 3: verify both task and question similarity
      const taskSimilarity = cosineSimilarity(currentTaskEmbedding, cachedTaskEmbedding);
      const questionSimilarity = cosineSimilarity(currentQuestionEmbedding, cached.questionEmbedding);
      
      if (taskSimilarity >= 0.85 && questionSimilarity >= 0.8) {
        return cached.answer; // cache hit!
      }
    }
  }
}

the getTaskEmbedding() function resolves the hash reference:

async function getTaskEmbedding(taskHash, env) {
  const taskCacheKey = `task_embedding_${taskHash}`;
  const cachedTask = await env.GOAL_CACHE.get(taskCacheKey, 'json');
  
  if (cachedTask && cachedTask.taskEmbedding) {
    return cachedTask.taskEmbedding; // return the 1536-dimensional vector
  }
  
  return null; // task embedding not found (shouldn't happen in normal operation)
}

the reality: javascript does all the work

important distinction: cloudflare kv is a simple key-value store with no query capabilities. it only supports:

get(key) - retrieve value by exact key match
put(key, value) - store value at key
list({ prefix }) - list keys with a given prefix
delete(key) - remove key

kv does not have:

sql queries
indexing beyond key prefixes
semantic search capabilities
similarity functions
relational joins

this means all semantic matching happens in javascript:

// what actually happens during cache lookup:

// 1. javascript fetches ALL follow-up cache entries (brute force)
const followupList = await env.GOAL_CACHE.list({ prefix: 'followup_v2_', limit: 50 });

// 2. javascript loops through each entry one by one
for (const item of followupList.keys) {
  const cached = await env.GOAL_CACHE.get(item.name, 'json'); // 🔥 KV API CALL
  
  // 3. javascript makes ANOTHER kv call to get the task embedding
  const cachedTaskEmbedding = await getTaskEmbedding(cached.taskHash, env); // 🔥 ANOTHER KV API CALL
  
  // 4. javascript calculates cosine similarity in memory
  const taskSimilarity = cosineSimilarity(currentTaskEmbedding, cachedTaskEmbedding);
  const questionSimilarity = cosineSimilarity(currentQuestionEmbedding, cached.questionEmbedding);
  
  // 5. javascript evaluates thresholds
  if (taskSimilarity >= 0.85 && questionSimilarity >= 0.8) {
    return cached.answer; // found match!
  }
}

performance implications:

searching 50 follow-up entries = 50 get() calls to kv
each entry requires another get() call for task embedding = 50 more calls
total: 100 kv api calls for a single cache lookup
each call has ~5-15ms latency from edge to kv storage
semantic similarity calculations happen in cloudflare’s v8 javascript runtime

this is why we limit searches (limit: 50) and use early return on first match - the “database” is actually just a distributed hashtable with javascript doing all the intelligent work.

why O(1) lookup is impossible with embeddings

the fundamental problem: you cannot create deterministic cache keys from semantic similarity.

// two questions that mean the same thing to humans
const question1 = "what about sets and reps?";
const question2 = "how many repetitions?";

// but AI embeddings convert them to completely different number arrays
embedding(question1) → [0.123, 0.456, 0.789, ...]  // 1536 numbers
embedding(question2) → [0.187, 0.423, 0.801, ...]  // 1536 different numbers

// to create cache keys, we hash the text (not the embeddings)
hash(question1) → "abc123"  // deterministic based on exact text
hash(question2) → "xyz789"  // different text = different hash

// KV can only find exact key matches
await env.GOAL_CACHE.get("followup_task1_abc123"); // ✅ finds cached answer for "sets and reps"
await env.GOAL_CACHE.get("followup_task1_xyz789"); // ❌ cache miss for "repetitions"

// even though humans know these questions are asking the same thing!

what if we hashed the embeddings instead?

// you could hash the embedding arrays...
embedding(question1) → [0.123, 0.456, 0.789, ...]
hash([0.123, 0.456, 0.789, ...]) → "def456"

embedding(question2) → [0.187, 0.423, 0.801, ...]  
hash([0.187, 0.423, 0.801, ...]) → "ghi789"

// but you still get different hashes for similar meanings!
// hashing doesn't make semantically similar vectors produce similar hashes

the problem: hash functions are designed to produce completely different outputs for even tiny input changes. this is the opposite of what we want for semantic similarity.

// tiny difference in embeddings = completely different hash
embedding("sets and reps")      → hash → "abc123"  
embedding("sets and reps!")     → hash → "xyz789"  // just added "!" 
embedding("reps and sets")      → hash → "def456"  // just swapped order

// semantic similarity is about finding vectors that are *close* in high-dimensional space
// but hash functions are designed to make similar inputs produce *distant* outputs

to find semantic matches, you must compare embeddings:

// the only way to know if two questions are similar:
const similarity = cosineSimilarity(
  embedding("how many repetitions?"),     // user's question
  embedding("what about sets and reps?") // cached question
); // returns 0.87 - they ARE similar!

// but you can't know this without calculating similarity for every cached question

attempted workarounds and why they fail:

// ❌ canonical mapping: requires manual maintenance, misses variations
const canonicalMap = {
  "sets_and_reps": ["sets and reps", "repetitions", "how many reps"],
  // what about "rep count"? "set/rep scheme"? "lifting numbers"?
};

// ❌ embedding bucketing: complex, approximate, still requires similarity search
function bucketEmbedding(embedding) {
  return embedding.slice(0, 10).map(x => Math.round(x * 100)).join('_');
}
const bucket = bucketEmbedding(questionEmbedding); // still need O(k) search in bucket

// ❌ locality-sensitive hashing: difficult to implement correctly, approximate results

the harsh reality: if you need semantic similarity, you need either:

O(n) search through all candidates (what we built)
specialized vector database with optimized similarity algorithms

key-value stores excel at exact lookups, but semantic similarity requires mathematical comparison of high-dimensional vectors - a fundamentally different operation.

performance reality check

this approach has significant scalability issues:

cache size	kv api calls	lookup latency	bottleneck
10 entries	~20 calls	~100-300ms	acceptable
50 entries	~100 calls	~500-1500ms	slower than openai
100 entries	~200 calls	~1000-3000ms	unusable

why this can be slower than calling openai directly:

openai api: 1 call, ~2000ms response time
our cache lookup: 100+ kv calls, potentially 1500ms+ just for network overhead
plus javascript cpu time for 50+ cosine similarity calculations

architectural limitations:

o(n) search complexity - performance degrades linearly with cache size
api call explosion - each cache lookup requires dozens of network requests
no indexing - cloudflare kv provides no query optimization beyond key prefixes
cpu intensive - 1536-dimensional vector math in javascript runtime

better approaches for production scale

1. cloudflare vectorize (purpose-built for this):

// cloudflare's native vector database - perfect fit
const results = await env.VECTORIZE_INDEX.query(questionEmbedding, {
  filter: { taskHash: { $eq: "abc123" } }, // filter by task first
  topK: 3,
  returnMetadata: true
}); // single api call, sub-100ms response, same infrastructure

// would store vectors like:
await env.VECTORIZE_INDEX.upsert([{
  id: "followup_abc123_def456",
  values: questionEmbedding, // 1536 dimensions
  metadata: {
    taskHash: "abc123",
    question: "what about sets and reps?",
    answer: "For upper body days, aim for 3-4 sets...",
    taskSimilarity: 0.92 // pre-computed for filtering
  }
}]);

2. third-party vector databases:

// pinecone, weaviate, or similar if you need features vectorize lacks
const results = await vectorDB.query({
  vector: questionEmbedding,
  filter: { taskSimilarity: { $gte: 0.85 } },
  topK: 1
}); // single api call, sub-100ms response

3. pre-computed similarity indices:

// compute similarities at write-time, not read-time
"task_questions_abc123" → {
  taskHash: "abc123",
  questions: [
    { text: "sets and reps?", embedding: [...], answers: ["key1", "key2"] },
    { text: "how often?", embedding: [...], answers: ["key3"] }
  ]
} // one kv call gets all questions for a task

4. hybrid approach with similarity caching:

// cache the similarity calculations themselves
"question_matches_def456" → {
  questionHash: "def456",
  matches: [
    { followupKey: "followup_v2_abc123_ghi789", similarity: 0.87 },
    { followupKey: "followup_v2_xyz789_mno123", similarity: 0.82 }
  ],
  timestamp: "2024-01-15T10:30:00Z"
} // amortize expensive similarity calculations

the vectorize migration

after running the kv-based system in production and validating user demand, we migrated to cloudflare vectorize for true semantic caching.

migration results

performance improvements:

cache lookup: 500-1500ms → <100ms
api calls: 100+ kv operations → 1 vectorize query
cache accuracy: string matching → semantic similarity

real-world impact:

// before: brute force kv iteration
const followupList = await env.GOAL_CACHE.list({ prefix: 'followup_v2_', limit: 50 });
for (const item of followupList.keys) {
  const cached = await env.GOAL_CACHE.get(item.name, 'json'); // 50+ api calls
  // ... similarity calculations in javascript
}

// after: native vectorize query  
const matches = await env.SEMANTIC_CACHE.query(questionEmbedding, {
  filter: { 
    type: { $eq: "followup" },
    taskHash: { $eq: taskHash }
  },
  topK: 3,
  returnMetadata: "all"
}); // single api call with hardware-accelerated similarity search

unified semantic cache setup

single vectorize index handles all cache types:

// goal generation cache
{ 
  id: "goal_abc123_timestamp",
  values: problemEmbedding,
  metadata: { 
    type: "goal_generation", 
    problem: "How do I learn Spanish?", 
    goals: "{...}" 
  }
}

// how-to guide cache
{
  id: "guide_def456_timestamp", 
  values: taskEmbedding,
  metadata: { 
    type: "how_to_guide", 
    taskText: "start 4 day upper lower split", 
    guide: "{...}" 
  }
}

// follow-up question cache
{
  id: "followup_ghi789_timestamp",
  values: questionEmbedding, 
  metadata: { 
    type: "followup", 
    taskHash: "def456", 
    question: "what about sets and reps?", 
    answer: "..."
  }
}

production migration approach

phase 1: dual system with fallbacks

async function findSimilarFollowUpVectorize(taskText, question, env) {
  if (!env.SEMANTIC_CACHE) {
    // fallback to legacy kv caching if vectorize unavailable
    return await findSimilarFollowUp(taskText, question, env);
  }
  
  try {
    // vectorize query
    const matches = await env.SEMANTIC_CACHE.query(questionEmbedding, {...});
    return matches.length > 0 ? matches[0].metadata.answer : null;
  } catch (error) {
    console.error('vectorize error:', error);
    // graceful fallback to kv on errors
    return await findSimilarFollowUp(taskText, question, env);
  }
}

phase 2: monitoring and validation

cache headers distinguish systems: X-Cache-Status: HIT-VECTORIZE vs HIT
performance monitoring shows dramatic improvements
error rates remain low with reliable fallbacks

why the migration made sense:

architectural mismatch resolved - we were forcing kv (key-value store) to do similarity search (vector database job)
performance unpredictability - 100+ api calls per cache lookup could be slower than openai depending on network conditions
complexity reduction - eliminated 200+ lines of similarity calculation code
timing alignment - vectorize became available when we needed it

engineering reality: we built a working but inefficient system, then migrated when better infrastructure became available. the kv approach taught us the requirements and validated user demand before committing to specialized tooling.

why this matters: the taskHash acts as a foreign key that enables semantic deduplication while maintaining referential integrity. multiple follow-up questions about the same task concept share a single task embedding, but each has its own question embedding and cached answer.

this creates a many-to-one relationship where:

one task embedding (e.g., “start 4 day upper lower split”)
supports multiple follow-up caches (e.g., “sets and reps?”, “how often?”, “what weight?”)
without duplicating the expensive 1536-dimensional task vector

cross-user cache benefits

the magic happens when multiple users ask similar questions:

user a: “start 4 day upper lower split” + “what about sets and reps?” → cache miss → ai generates → cached

user b: “begin 4-day upper/lower routine” + “sets and repetitions?” → cache hit → instant response

the semantic matching catches variations:

“What about sets and reps?” ≈ “How many sets and repetitions?”
“start 4 day upper lower split” ≈ “begin upper/lower 4-day routine”
but rejects wrong contexts: “sets and reps” + “bake cookies” → no match

performance characteristics

two distinct response patterns:

scenario	network	openai cost	latency	hit rate
cache miss + openai	openai api	~$0.005	~6.5s	0%
vectorize cache hit	vectorize query	$0	~2.1s	40-60%

vectorize caching implementation

the system now implements unified vectorize caching with semantic similarity search:

single index, three cache types

all caching flows through one vectorize index with metadata-based filtering:

// goal generation vectors
{
  id: "goal_776b24e8b76a40ad_1753846452714",
  values: problemEmbedding, // 1536 dimensions
  metadata: {
    type: "goal_generation",
    problemHash: "776b24e8b76a40ad", 
    problem: "get jacked",
    goals: "{\"today\":[...], \"month\":[...], \"year\":[...]}"
  }
}

// how-to guide vectors  
{
  id: "guide_3933f9b7ab9b0f6f_1753847607690",
  values: taskEmbedding, // 1536 dimensions
  metadata: {
    type: "how_to_guide",
    taskHash: "3933f9b7ab9b0f6f",
    taskText: "complete a 30-minute strength training workout", 
    guide: "{\"overview\":\"...\", \"steps\":[...], \"proTip\":\"...\"}"
  }
}

// follow-up question vectors
{
  id: "followup_3933f9b7ab9b0f6f_5d67953f891f8d41_1753847633760", 
  values: questionEmbedding, // 1536 dimensions
  metadata: {
    type: "followup",
    taskHash: "3933f9b7ab9b0f6f", // links to how-to guide
    question: "how to do a lunge",
    answer: "To perform a lunge: 1. Stand with your feet hip-width apart..."
  }
}

vectorize query patterns

goal generation lookup:

const matches = await env.SEMANTIC_CACHE.query(problemEmbedding, {
  filter: { type: { $eq: "goal_generation" } },
  topK: 3,
  returnMetadata: "all"
});

how-to guide lookup:

const matches = await env.SEMANTIC_CACHE.query(taskEmbedding, {
  filter: { type: { $eq: "how_to_guide" } },
  topK: 3, 
  returnMetadata: "all"
});

follow-up question lookup:

const matches = await env.SEMANTIC_CACHE.query(questionEmbedding, {
  filter: { 
    type: { $eq: "followup" },
    taskHash: { $eq: taskHash }
  },
  topK: 3,
  returnMetadata: "all"
});

performance: native vector similarity search with hardware acceleration, ~2.1s cache hits vs ~6.5s cache misses in production

cost analysis

cost breakdown for 1000 users on similar fitness tasks:

without any caching:

initial guides: 1000 × $0.005 = $5.00
follow-up questions: 1000 × $0.005 = $5.00
task embeddings: 2000 × $0.0001 = $0.20
total: $10.20

with complete semantic caching:

initial guides: 1 × $0.005 = $0.005
follow-up questions: varies by uniqueness ≈ $0.05
task embeddings: 10 × $0.0001 = $0.0001
total: $0.055 (99.5% savings)

the first user asking about any topic creates a “knowledge seed” that benefits all future users with similar needs.

implementation details

rate limiting with graceful degradation

if (response.status === 429) {
  const retryAfter = errorData.retryAfter || 60;
  const minutes = Math.ceil(retryAfter / 60);
  throw new Error(
    `Too many questions! Please wait ${minutes} minute${minutes !== 1 ? 's' : ''} before asking another.`
  );
}

cache warming effects

the vectorize system creates organic cache warming through semantic similarity:

goal generation caching:

“get jacked” → instant responses for “build muscle”, “gain strength”, “get buff”
“learn spanish” → instant responses for “study spanish”, “spanish fluency”
each problem type builds a knowledge base that benefits similar requests

how-to guide caching:

“start 4 day upper lower split” → instant for “begin upper/lower routine”, “4-day workout plan”
“bake chocolate chip cookies” → instant for “make chocolate cookies”, “cookie baking”
semantic matching catches variations without exact string matches

follow-up question caching:

“what about sets and reps?” cached once, serves “how many repetitions?”, “sets and repetitions?”
“what temperature?” cached once, serves “baking temperature?”, “oven temp?”
context-aware caching ensures answers match the original task domain

monitoring and debugging

cache monitoring tracks vectorize performance:

// cache status headers distinguish between systems
return new Response(JSON.stringify(response), {
  headers: {
    'X-Cache-Status': cached ? 'HIT-VECTORIZE' : 'MISS-VECTORIZE',
    'X-Cache-Similarity-Score': bestMatch?.score?.toFixed(3),
    'X-Cache-Vector-Count': matches.matches?.length
  }
});

// console logging shows vectorize query results
console.log('=== GOAL GENERATION REQUEST START ===');
console.log('SEMANTIC_CACHE available:', !!env.SEMANTIC_CACHE);
console.log('Goal generation raw Vectorize response:', JSON.stringify(matches, null, 2));
console.log(`Goal generation search found ${matches.matches?.length || 0} potential matches`);

vectorize vector id structure enables easy debugging:

goal_{problemHash}_{timestamp} - goal generation vectors
guide_{taskHash}_{timestamp} - how-to guide vectors
followup_{taskHash}_{questionHash}_{timestamp} - follow-up q&a vectors

debug cli queries for troubleshooting:

# check metadata indexes
npx wrangler vectorize list-metadata-index semantic-cache

# query by type filter  
npx wrangler vectorize query semantic-cache --vector [...] --filter '{"type": "goal_generation"}'

# check vector count
npx wrangler vectorize info semantic-cache

can monitor hit rates in browser devtools and server logs, plus use cli tools to debug vectorize filtering issues.

alternatives considered

simple string matching: too brittle, misses semantic variations single embedding per question: loses task context, wrong answers llm-based similarity: too expensive for cache lookup hash-based caching: can’t handle natural language variations redis/database: unnecessary infrastructure complexity

lessons learned

migration insights

start simple, upgrade strategically - kv validation → vectorize optimization
reliable fallbacks enable confidence - dual systems during migration prevent downtime
semantic caching scales exponentially - each user benefits from all previous interactions
infrastructure timing matters - vectorize ga made the migration viable
monitoring distinguishes systems - cache headers enable performance comparison

technical learnings

metadata filtering is crucial - type and taskHash filters prevent wrong context matches
similarity thresholds matter - 0.8+ cosine similarity for reliable semantic matching
cache warming is organic - no need to pre-populate, users do it naturally
conversation state is fragile - clear on modal close to prevent bugs
graceful degradation - if caching fails, still call openai
vectorize beats kv - native similarity search vs javascript iterations

production results

performance improvements measured:

cache hit latency: 6.5s → 2.1s (~3x faster)
api call reduction: 100+ kv operations → 1 vectorize query
cost reduction: 99.5% for popular interactions
cache accuracy: improved semantic matching vs string similarity

user experience impact:

instant responses for cached questions
better cross-user knowledge sharing
more consistent ai guidance
reduced rate limiting due to cache hits

metadata indexing gotcha: vectors vs indexes timing

after deploying the vectorize migration, we discovered a critical timing issue that broke filtering for goal generation and follow-up caching.

the problem: retroactive indexing doesn’t work

what happened:

vectors were inserted into vectorize with metadata like type: "goal_generation"
metadata indexes were created later via wrangler vectorize create-metadata-index
filtered queries returned 0 results despite vectors having the correct metadata

// vectors inserted BEFORE metadata index creation
// ❌ cannot be found by filtered queries
const matches = await env.SEMANTIC_CACHE.query(embedding, {
  filter: { type: { $eq: "goal_generation" } }
}); // returns 0 results

// ✅ but can be found by unfiltered queries  
const allMatches = await env.SEMANTIC_CACHE.query(embedding, {
  topK: 10
}); // returns vectors with metadata intact

the discovery process

debugging revealed the issue:

# unfiltered query found 10 vectors including goals
npx wrangler vectorize query semantic-cache --vector [...] --top-k 10

# filtered query found only 2 newer vectors
npx wrangler vectorize query semantic-cache --vector [...] --filter '{"type": "goal_generation"}'

key insight: cloudflare vectorize metadata indexes only apply to vectors inserted after the index creation. existing vectors become invisible to filtered queries.

the solution: nuclear option

since the site had few users, we chose the clean slate approach:

# delete entire index
npx wrangler vectorize delete semantic-cache

# recreate with metadata indexes first
npx wrangler vectorize create semantic-cache --dimensions 1536 --metric cosine
npx wrangler vectorize create-metadata-index semantic-cache --propertyName=type --type=string
npx wrangler vectorize create-metadata-index semantic-cache --propertyName=taskHash --type=string

result: all new vectors are properly indexed and findable by filtered queries.

lessons for production systems

1. create metadata indexes before inserting vectors

# ✅ correct order for new vectorize setup

# step 1: create vectorize index
npx wrangler vectorize create semantic-cache --dimensions 1536 --metric cosine

# step 2: create metadata indexes BEFORE inserting any vectors
npx wrangler vectorize create-metadata-index semantic-cache --propertyName=type --type=string
npx wrangler vectorize create-metadata-index semantic-cache --propertyName=taskHash --type=string

# step 3: verify indexes are ready
npx wrangler vectorize list-metadata-index semantic-cache

# step 4: NOW safe to insert vectors with metadata
# vectors inserted after this point will be findable by filtered queries

in your application code:

// this will work correctly because metadata indexes exist
await env.SEMANTIC_CACHE.upsert([{
  id: "goal_abc123_timestamp",
  values: problemEmbedding,
  metadata: {
    type: "goal_generation",  // ✅ indexed
    problemHash: "abc123"     // ✅ indexed via taskHash
  }
}]);

// filtered queries will find the vector
const matches = await env.SEMANTIC_CACHE.query(embedding, {
  filter: { type: { $eq: "goal_generation" } }  // ✅ works
});

2. vectorize lacks reindexing capabilities unlike elasticsearch or other databases, vectorize doesn’t offer:

reindex command to rebuild metadata indexes
bulk export/import for data migration
retroactive index application

3. migration strategies for production data

small datasets: nuclear option (delete/recreate)
large datasets: build custom export/import pipeline
critical systems: implement dual-write during transition

4. monitoring metadata filtering add debug logging to detect filtering issues:

const filteredMatches = await env.SEMANTIC_CACHE.query(embedding, {
  filter: { type: { $eq: "goal_generation" } }
});

if (filteredMatches.matches.length === 0) {
  console.warn('Filtered query returned 0 results - check metadata indexes');
}

this metadata indexing gotcha cost us a few hours of debugging but taught valuable lessons about vectorize operational characteristics that aren’t well documented.

conclusion

semantic caching with vector databases transforms ai applications from expensive, slow interactions into fast, cost-effective systems that improve with every user.

key insights for ai application developers:

use the right tool for the job - vector databases excel at similarity search, traditional databases at exact lookups
semantic caching scales exponentially - each user interaction creates value for all future similar requests
ai responses don’t need to be unique - they need to be contextually appropriate and fast
metadata filtering is crucial - combine semantic similarity with structured filters for precise results
create metadata indexes before inserting vectors - retroactive indexing doesn’t work in most vector databases

when to use vector databases for caching:

✅ user queries have natural language variations (“sets and reps” vs “repetitions”)
✅ exact string matching misses too many valid cache hits
✅ content generation is expensive (time or cost)
✅ semantic similarity matters more than exact matches
❌ simple key-value lookups work fine
❌ transactional consistency is required

bottom line: if you’re building ai applications with expensive generation costs, semantic caching with vector databases can deliver 3x performance improvements and 99%+ cost reductions for popular content. the infrastructure investment pays for itself quickly through improved user experience and reduced ai api costs.

try the live system: actuallydostuff.com - click the ❓ on any goal to experience ~2s cached responses vs ~6s generated responses, powered by cloudflare vectorize.

dual-layer caching with cloudflare kv and localstorage

2025-07-13T00:00:00+00:00

built a movie recommendation app that uses ai to analyze content themes. the challenge? each openai api call costs money and takes time. the solution? a dual-layer caching strategy that maximizes performance while minimizing costs.

the problem

calling openai’s api for every movie analysis is expensive and slow:

cost: $0.002 per 1k tokens with gpt-4o-mini
latency: 500-2000ms per request
scale: same movies analyzed repeatedly by different users

simple client-side caching helps individual users, but doesn’t solve the broader cost problem when multiple users analyze the same popular movies.

dual-layer approach

implemented two complementary caching layers:

layer 1: localstorage (client-side)

function getCachedAnalysis(movieTitle, year) {
  const cache = JSON.parse(localStorage.getItem('movieAnalysisCache') || '{}');
  const key = `${movieTitle}_${year}`;
  const cached = cache[key];
  
  if (cached && cached.timestamp) {
    const ageInDays = (Date.now() - cached.timestamp) / (1000 * 60 * 60 * 24);
    if (ageInDays < 30) {
      return cached.analysis;
    }
  }
  return null;
}

layer 2: cloudflare kv (server-side)

export async function onRequest(context) {
  const { request, env } = context;
  const { title, synopsis, year } = await request.json();
  
  const cacheKey = `analysis:${title}_${year}`.replace(/[^a-zA-Z0-9_-]/g, '_');
  
  // check kv cache first
  const cached = await env.KV.get(cacheKey, { type: 'json' });
  if (cached) {
    return new Response(JSON.stringify(cached), {
      headers: { 'X-Cache': 'HIT' }
    });
  }
  
  // cache miss - call openai
  const analysis = await callOpenAI(title, synopsis, year);
  
  // store in kv for 30 days
  await env.KV.put(cacheKey, JSON.stringify(analysis), {
    expirationTtl: 2592000
  });
  
  return new Response(JSON.stringify(analysis), {
    headers: { 'X-Cache': 'MISS' }
  });
}

how they work together

the caching cascade works like this:

client checks localstorage - if hit, no network request needed
if miss, calls api - server function gets invoked
server checks kv cache - if hit, returns cached result (no openai cost)
if miss, calls openai - makes expensive api call
server stores in kv - future users benefit from cache
client stores in localstorage - future requests from same user skip network

// client-side flow
const cachedAnalysis = getCachedAnalysis(movie.title, movie.year);
if (cachedAnalysis) {
  // best case: instant local cache hit
  displayAnalysis(cachedAnalysis);
  return;
}

// cache miss - make api call
const response = await fetch('/api/analyze-content', {
  method: 'POST',
  body: JSON.stringify({ title: movie.title, synopsis: movie.synopsis, year: movie.year })
});

const analysis = await response.json();
setCachedAnalysis(movie.title, movie.year, analysis); // store locally
displayAnalysis(analysis);

performance characteristics

this creates three distinct performance scenarios:

scenario	network	openai cost	latency
localstorage hit	none	$0	~1ms
kv hit	api call	$0	~100ms
cache miss	api call	~$0.01	~1500ms

cost analysis

for a popular movie analyzed by 1000 users:

without caching: 1000 × $0.01 = $10.00
with localstorage only: 1000 × $0.01 = $10.00 (no sharing)
with dual-layer: 1 × $0.01 = $0.01 (99.9% savings)

the kv cache effectively amortizes the openai cost across all users.

cloudflare kv setup

cloudflare pages automatically injects kv bindings into function context through the dashboard:

create kv namespace in cloudflare dashboard
bind it to your pages project via dashboard ui (variable name: KV)
the binding appears as env.KV in your functions automatically

no configuration files needed - it’s all done through the cloudflare dashboard interface.

cache invalidation

both layers use 30-day expiration:

localstorage: timestamp-based expiration check
kv: cloudflare’s native ttl handling

// localstorage expiration
const ageInDays = (Date.now() - cached.timestamp) / (1000 * 60 * 60 * 24);
if (ageInDays < 30) {
  return cached.analysis;
}

// kv expiration (automatic)
await env.KV.put(cacheKey, data, {
  expirationTtl: 2592000 // 30 days in seconds
});

debugging cache behavior

added cache status headers to track hit/miss patterns:

return new Response(JSON.stringify(analysis), {
  headers: {
    'Content-Type': 'application/json',
    'X-Cache': cached ? 'HIT' : 'MISS'
  }
});

can monitor in browser devtools to verify caching is working.

alternatives considered

redis: would work but adds infrastructure complexity cdn caching: doesn’t work for dynamic post requests database caching: overkill for simple key-value needs memory caching: doesn’t persist across deployments

cloudflare kv hits the sweet spot - globally distributed, zero infrastructure, tight pages integration.

trade-offs

pros:

massive cost reduction for popular content
improved user experience (faster responses)
zero infrastructure management
global edge caching

cons:

slightly more complex than single-layer
potential for stale data (30-day window)
kv has eventual consistency (rare edge case)

implementation tips

cache key normalization - strip special characters to avoid kv key issues
graceful degradation - if kv fails, still call openai
cache warming - consider pre-caching popular movies
monitoring - track hit rates and costs

conclusion

dual-layer caching with localstorage and cloudflare kv provides the best of both worlds - instant local performance and shared cost benefits. for api-heavy applications, this pattern can dramatically reduce costs while improving user experience.

the key insight is that not all caching needs to be shared - combining individual and collective caching layers creates optimal performance characteristics for different use cases.

adding fuzzy search to a jekyll blog

2025-07-06T00:00:00+00:00

wanted to add search to my blog without any server-side complexity or external services. turns out jekyll’s liquid templating makes this surprisingly elegant.

the liquid magic

jekyll can generate json files during build time. here’s the key insight - we can loop through all posts and create a searchable index:

---
layout: null
---
[
  {% for post in site.posts %}
    {
      "title": "{{ post.title | escape }}",
      "excerpt": "{{ post.excerpt | strip_html | truncatewords: 50 | escape }}",
      "url": "{{ post.url }}",
      "date": "{{ post.date | date: '%B %d, %Y' }}",
      "categories": {{ post.categories | jsonify }}
    }{% unless forloop.last %},{% endunless %}
  {% endfor %}
]

the layout: null tells jekyll to output raw json without any html wrapper. the {% unless forloop.last %} handles the trailing comma problem that would break json parsing.

automatic reindexing

this is the beautiful part - no rake tasks or manual reindexing needed. every time you run jekyll build or jekyll serve, the search.json gets regenerated with your latest posts. jekyll’s build process handles the entire search index automatically.

fuse.js integration

for the actual search, fuse.js does the heavy lifting. it’s 6kb gzipped and handles fuzzy matching really well:

fetch('/search.json')
  .then(response => response.json())
  .then(data => {
    fuse = new Fuse(data, {
      keys: ['title', 'excerpt', 'categories'],
      threshold: 0.3,
      includeScore: true
    });
  });

the threshold: 0.3 is the sweet spot - strict enough to avoid nonsense results but loose enough to catch typos and partial matches.

why this approach works

no server required - everything happens client-side
no build complexity - uses jekyll’s existing templating
always current - updates with every build
fast - json loads once, search happens locally
lightweight - fuse.js is tiny, no dependencies

search ui

added a simple input to the sidebar that shows results as you type:

searchInput.addEventListener('input', function(e) {
  const query = e.target.value.trim();

  if (query.length < 2) {
    searchResults.innerHTML = '';
    return;
  }

  const results = fuse.search(query);
  displayResults(results);
});

only triggers after 2 characters to avoid noise. shows up to 5 results with title, excerpt, and date.

liquid templating gotchas

few things to watch out for:

json escaping - use | jsonify filter instead of | escape for proper json encoding
strip html and newlines - excerpts need | strip_html | strip_newlines to avoid json breaks
arrays - jekyll’s | jsonify filter handles arrays and escaping automatically
trailing commas - the {% unless forloop.last %} pattern prevents json errors

csp considerations

if you’re using content security policy, you’ll need to allow the fuse.js cdn and local fetch requests:

script-src 'self' https://cdn.jsdelivr.net 'unsafe-inline'
connect-src 'self' https://disqus.com

or download fuse.js locally to avoid external dependencies entirely.

parse don't validate in ruby: building safer applications

2025-07-05T00:00:00+00:00

Dynamic languages like Ruby give you flexibility, but they also put the burden of data safety on you. Without compile-time type checking, how do you ensure your application doesn’t crash when it receives unexpected data?

The answer is the “parse don’t validate” pattern - a technique popularized by Alexis King’s influential 2019 blog post that transforms unknown input into well-defined, validated objects before it reaches your business logic.

the problem with validation-only approaches

Traditional validation approaches check if data is correct, but then continue working with the original, unstructured data:

# Don't do this - validation without transformation
def create_user(params)
  if params[:email].present? && params[:name].present?
    User.create(params) # Still working with unstructured hash
    # What if params has unexpected keys?
    # What if email is nil despite the check?
    # What if someone changes the validation logic?
  end
end

This leaves you vulnerable to runtime errors when the raw data doesn’t match your assumptions.

parse don’t validate: transform input into structured objects

Instead of just checking validity, transform unknown data into known, typed structures:

class UserForm
  include ActiveModel::Model
  include ActiveModel::Attributes

  attribute :name, :string
  attribute :email, :string
  attribute :id, :integer

  validates :name, presence: true
  validates :email, presence: true, format: { with: URI::MailTo::EMAIL_REGEXP }
  validates :id, presence: true, numericality: { greater_than: 0 }

  def self.parse(data)
    form = new(data)
    raise ArgumentError, form.errors.full_messages.join(', ') unless form.valid?
    form
  end
end

# Usage
user_form = UserForm.parse(params) # Returns UserForm or raises
User.create!(user_form.attributes)

why this pattern matters

Explicit contracts - Clear what each component expects and returns
Fail fast - Catch invalid data at system boundaries, not deep in business logic
Self-documenting - Code clearly shows what data flows through the system
Centralized validation - All validation rules in one place per data type
Better error messages - Specific, actionable feedback about what’s wrong

controllers: transform filtered params into validated objects

Strong parameters handle security (preventing mass assignment), but they still return unvalidated hashes. Add a parsing layer for data integrity:

class UsersController < ApplicationController
  def create
    # Strong parameters filter, then parse for validation
    user_form = UserForm.parse(user_params)
    @user = UserCreationService.call(user_form)
    render json: UserSerializer.new(@user).to_h
  rescue ArgumentError => e
    render json: { error: e.message }, status: 422
  end

  private

  def user_params
    params.require(:user).permit(:name, :email, :id)
  end
end

services: accept structured objects

Services should work with validated, structured data rather than raw hashes:

class UserCreationService
  def self.call(user_form) # Explicit contract, not random hash
    user = User.create!(user_form.attributes)
    NotificationMailer.welcome_email(user).deliver_later
    user
  end
end

external api integration

Transform external responses into internal objects to maintain consistent data contracts:

class StripeChargeResult
  include ActiveModel::Model
  include ActiveModel::Attributes

  attribute :charge_id, :string
  attribute :amount_cents, :integer
  attribute :status, :string
  attribute :created_at, :datetime

  def self.from_stripe_response(response)
    new(
      charge_id: response[:id],
      amount_cents: response[:amount],
      status: response[:status],
      created_at: Time.at(response[:created])
    )
  end

  def successful? = status == 'succeeded'
  def amount_dollars = amount_cents / 100.0
end

# Usage
stripe_response = stripe_client.charges.create(charge_params)
charge_result = StripeChargeResult.from_stripe_response(stripe_response)

if charge_result.successful?
  record_payment(charge_result)
end

background jobs: structured arguments

Instead of working with argument hashes, parse job parameters into validated objects:

class EmailJobParams
  include ActiveModel::Model
  include ActiveModel::Attributes

  attribute :user_id, :integer
  attribute :template, :string, default: 'welcome'
  attribute :delay_minutes, :integer, default: 0

  validates :user_id, presence: true
  validates :template, inclusion: { in: %w[welcome premium reminder] }

  def self.parse(args)
    params = new(args)
    raise ArgumentError, params.errors.full_messages.join(', ') unless params.valid?
    params
  end

  def user
    @user ||= User.find(user_id)
  end
end

class WelcomeEmailJob < ApplicationJob
  def perform(raw_args)
    job_params = EmailJobParams.parse(raw_args)
    WelcomeMailer.send_email(job_params.user, job_params.template).deliver_now
  end
end

alternative: result objects

For applications that prefer explicit success/failure handling over exceptions:

require 'dry/monads'

class UserForm
  include Dry::Monads[:result]
  include ActiveModel::Model
  include ActiveModel::Attributes

  attribute :name, :string
  attribute :email, :string
  attribute :id, :integer

  validates :name, presence: true
  validates :email, format: { with: URI::MailTo::EMAIL_REGEXP }

  def self.safe_parse(data)
    form = new(data)
    return Failure(form.errors.full_messages) unless form.valid?
    Success(form)
  end
end

# Usage
case UserForm.safe_parse(params)
in Success(user_form)
  User.create!(user_form.attributes)
in Failure(errors)
  render json: { errors: errors }, status: 422
end

libraries to consider

dry-validation - Advanced validation with detailed error handling
dry-monads - Result objects and functional patterns
dry-struct - Immutable value objects with type coercion
reform - Form objects that integrate seamlessly with Rails

implementation strategy

Start with new features - Apply parsing pattern to all new controllers and services
Focus on boundaries - Prioritize user input, external APIs, and background jobs
Refactor incrementally - Convert existing code one component at a time

the outcome

By parsing unknown data into known structures at every boundary, you eliminate a whole class of runtime errors. Your code becomes more predictable, easier to debug, and self-documenting.

In dynamic languages, explicit data contracts aren’t just good practice - they’re essential for building reliable applications that handle real-world data gracefully.

form validation with jitter effects

2025-07-02T00:00:00+00:00

Ever notice how form validation can feel… boring? You click submit, some fields turn red, maybe you get an error message. Functional, sure. But what if we could make invalid fields literally shake their heads at you?

try it out

Jitter Effect Preview

Fill out this form and try the validation buttons to see different jitter animations on required empty fields.

the problem with standard validation

HTML5 gives us built-in form validation with the required attribute and input types like email. But the default browser behavior is pretty bland. And if you add novalidate to your form (which many of us do for custom validation), you’re on your own for feedback.

<form id="testForm" novalidate>
  <input type="email" required>
</form>

That novalidate attribute tells the browser “thanks but no thanks, I’ll handle validation myself.” Which opens the door for more creative feedback…

enter jitter animations

Instead of just turning fields red, what if they physically reacted to being empty? Here’s a collection of CSS animations that give form fields personality:

@keyframes jitter-shake {
  0%, 100% { transform: translateX(0); }
  10%, 30%, 50%, 70%, 90% { transform: translateX(-4px); }
  20%, 40%, 60%, 80% { transform: translateX(4px); }
}

@keyframes jitter-bounce {
  0%, 100% { transform: translateY(0); }
  25% { transform: translateY(-8px); }
  50% { transform: translateY(-4px); }
  75% { transform: translateY(-2px); }
}

@keyframes jitter-pulse {
  0%, 100% { transform: scale(1); }
  50% { transform: scale(1.05); }
}

@keyframes jitter-wobble {
  0%, 100% { transform: rotate(0deg); }
  25% { transform: rotate(-2deg); }
  50% { transform: rotate(2deg); }
  75% { transform: rotate(-1deg); }
}

Each animation has its own personality and color:

shake: the classic “nope” head shake (red #f7768e)
bounce: a gentle hop like “hey, over here!” (orange #ff9e64)
pulse: subtle breathing effect for a softer touch (purple #bb9af7)
wobble: playful rotation that feels less aggressive (pink #f7768e)

The colors are applied along with the animation class:

.jitter-shake {
    animation: jitter-shake 0.6s ease-in-out;
    border-color: #f7768e !important;
}

This means when an input gets the jitter effect, it not only animates but also changes its border color to match the animation’s personality. The !important ensures the color override takes precedence during the animation.

the validation logic

Here’s where it gets interesting. Instead of validating fields one by one, we collect all empty required fields and animate them simultaneously. The function gets called from the test buttons, each passing a different animation type:

<button type="button" onclick="validateForm('shake')">🔸 Test Shake</button>
<button type="button" onclick="validateForm('bounce')">🔹 Test Bounce</button>
<button type="button" onclick="validateForm('pulse')">🔸 Test Pulse</button>
<button type="button" onclick="validateForm('wobble')">🔹 Test Wobble</button>

And here’s the validation function itself:

function validateForm(animationType = 'shake') {
  const form = document.getElementById('testForm');
  const requiredInputs = form.querySelectorAll('input[required], textarea[required], select[required]');

  let emptyFields = [];
  let emptyInputs = [];

  // first pass: identify all empty fields
  requiredInputs.forEach(input => {
    if (isEmpty(input)) {
      emptyFields.push(input.name || input.id);
      emptyInputs.push(input);
    }
  });

  // second pass: jitter all empty fields simultaneously
  if (emptyInputs.length > 0) {
    emptyInputs.forEach(input => {
      jitterElement(input.id, animationType);
    });

    statusDiv.className = 'status error';
    statusDiv.textContent = `❌ Please fill in: ${emptyFields.join(', ')}`;
    return false;
  }

  return true;
}

The two-pass approach means all invalid fields react at once, creating a unified “these need attention” moment rather than a sequential cascade.

applying the effect

The jitter function handles the animation lifecycle cleanly:

function jitterElement(elementId, animationType = 'shake') {
  const element = document.getElementById(elementId);
  const className = `jitter-${animationType}`;

  // only remove classes if this element doesn't already have the target class
  if (!element.classList.contains(className)) {
    // remove any existing jitter classes
    element.classList.remove('jitter-shake', 'jitter-bounce', 'jitter-pulse', 'jitter-wobble');
  }

  // add the new jitter class
  element.classList.add(className);

  // remove after animation completes
  setTimeout(() => {
    element.classList.remove(className);
  }, 600);
}

subtle touches

You can also add jitter on blur for immediate feedback:

document.querySelectorAll('input[required]').forEach(input => {
  input.addEventListener('blur', function() {
    if (isEmpty(this)) {
      setTimeout(() => jitterElement(this.id, 'shake'), 100);
    }
  });
});

That 100ms delay prevents the animation from feeling too aggressive when users are just tabbing through fields.

The form submission also triggers validation:

document.getElementById('testForm').addEventListener('submit', function(e) {
  e.preventDefault();

  if (validateForm('shake')) {
    // form is valid, show success message
  }
});

So validation happens in three scenarios:

When clicking any of the test buttons (with their specific animation)
When blurring out of a required field that’s empty (always uses shake)
When submitting the form (uses shake by default)

when to use what

Different animations work better for different contexts:

use shake for critical errors or final form submission
use bounce for friendly reminders
use pulse for subtle hints in longer forms
use wobble for playful interfaces or less serious applications

The key is matching the animation personality to your form’s context. A medical form probably wants subtle pulse effects. A game signup might embrace the full wobble.

performance notes

CSS transforms are GPU-accelerated, so these animations won’t cause layout thrashing. The 600ms duration is long enough to be noticeable but short enough to not feel sluggish. And since we’re using classes rather than inline styles, the browser can optimize the animations.

Form validation doesn’t have to be boring. Sometimes a little shake is all you need to make the experience memorable without being annoying.

why your brain actually wants you to struggle

2025-07-01T00:00:00+00:00

Your brain has this weird thing where it actually wants you to struggle. Not the soul-crushing, life-ending kind… the manageable “this is just hard enough to make me sweat” kind.

After 20 years of lifting and 6 years competing in powerlifting, I’ve realized something interesting. The iron doesn’t just forge stronger bodies - it builds antifragile minds. Controlled challenge exposure does something crazy - it rewires your entire system.

When you face manageable challenges, your brain literally changes structure. Neuroimaging studies show controlled stress increases prefrontal cortex volume. Diffusion tensor imaging reveals enhanced white matter myelination in executive control areas, creating faster neural highways between thinking and emotional centers.

your stress system becomes precision tuned

The hypothalamic-pituitary-adrenal axis (yeah that’s a mouthful) develops crazy adaptability through repeated controlled challenges. People who experience manageable stress develop enhanced glucocorticoid receptor sensitivity… basically more precise cortisol regulation. Their stress response becomes like a finely-tuned instrument.

Neurotransmitter systems coordinate too. Dopamine learns to find rewarding aspects in challenging situations. Serotonin adapts for mood regulation during stress. Norepinephrine optimizes release patterns - moderate levels engage high-affinity receptors that strengthen prefrontal function without the excessive release that screws up cognitive control.

stress inoculation actually works??

Stress inoculation training operates on an immunization model. Expose people to manageable doses of stress, build psychological immunity. Meta-analyses show large effect sizes across populations from military to anxiety patients.

Fear extinction creates new safety memories that compete with original fear memories. You’re not erasing stress responses but building a more complex, adaptive system. 65 studies with nearly 5,000 ptsd patients showed medium to large effects from exposure therapy, benefits maintained long-term.

The process involves implicit habituation and explicit skill development. Prefrontal cortex learns to regulate amygdala fear responses better while hippocampus improves contextual processing.

the 4% sweet spot (powerlifting gets this right)

Optimal growth happens when challenges are roughly 4% beyond current skill level. This maintains the flow channel - enough challenge to engage full attention without triggering overwhelm.

Powerlifting naturally implements this through systematic progression. Adding 2.5-5 pounds to a 400lb squat? That’s about 1% increase. Most good programs cycle between 70-95% of max, keeping you in that optimal challenge zone. It’s stress inoculation training disguised as sport.

4% challenge increase triggers ideal dopamine and norepinephrine levels. Enough to enhance focus without impairing prefrontal function. Maintains growth mindset where challenges become opportunities rather than threats.

This is why heavy singles work so well psychologically. When you’re regularly handling 92-93% in training, that 90% competition opener feels routine instead of terrifying. Your nervous system adapts to the stress pattern.

building antifragile systems (not just resilient ones)

Taleb’s antifragility concept is fascinating - systems that actually improve from volatility and stress. Unlike resilience (bouncing back) or robustness (resisting change), antifragile systems get stronger.

Powerlifting exemplifies this perfectly. You literally damage muscle fibers to promote growth through supercompensation. Each training cycle involves embracing uncertainty (will I make this lift?), processing failure (missed attempts provide feedback), and emerging stronger both physically and psychologically.

Elite powerlifters view failures as “nothing more than a statistic” rather than identity-threatening events. The sport teaches that “failure is the most consistent part of lifting.” This normalizes setbacks as information for improvement rather than defeats.

Hormesis effect - small stress doses strengthen the system. People experiencing varied, manageable stressors develop “redundancy in coping” - multiple strategies preventing single points of failure.

Antifragility needs skin in the game though. Personal investment ensuring learning from outcomes. In powerlifting, there’s no hiding from results. The bar either goes up or it doesn’t. This creates accountability that forces genuine adaptation.

resilience isn’t a personality trait

Resilience emerges from interaction of protective factors: individual attributes (cognitive ability, self-regulation), relationships (caring connections, high expectations), community resources (effective institutions, prosocial networks).

Most common response to adversity (35-65% of people) is “minimal-impact resilience” - maintaining stable functioning despite challenges. Not extraordinary characteristics but flexible coping strategies adapted to context. “Coping flexibility” - matching strategies to situational demands.

Avoiding personalization, pervasiveness, and permanence in explanations of negative events maintains psychological resources for growth. Positive emotions during stress create upward spirals.

the dose makes the poison (and powerlifters understand dosage)

HPA axis shows rapid habituation to identical stressors, cortisol responses decreasing over time. 47 studies show moderate to large effect sizes for habituation capacity.

Elite powerlifters show 40-60% cortisol elevations during competition - massive stress responses that would impair function in untrained individuals. Yet they maintain peak performance because they’ve adapted their stress systems through systematic exposure.

Psychological challenge follows inverted-u curve though. Insufficient challenge = no growth. Excessive challenge = harm. This is why periodization works - accumulation phases build volume tolerance, intensification phases develop comfort with heavy loads, realization phases peak psychological readiness.

Progressive overload applies to psychological strength like physical training. Start at 60-70% max capacity, increase 10-15% for optimal adaptation without overwhelming systems. Good powerlifting programs naturally implement this through percentage-based training.

Active engagement is crucial. You can’t just passively experience stress - you need deliberate practice of coping strategies. In powerlifting, this means visualization, breathing techniques, self-talk patterns, arousal control. The Barnicle & Lepage study showed sport psych interventions produced 6% performance improvements in elite powerlifters.

Recovery periods essential. Understanding that training stress, work stress, and life stress all contribute to total fatigue load. Modern powerlifters track HRV, prioritize sleep, manage nutrition timing. You get better by training, but gains are made when you rest.

This represents a shift from viewing stress as harmful to understanding it as potentially beneficial when structured right. Our brains are remarkably plastic, capable of developing enhanced stress management through experience.

Twenty years of lifting has taught me something most people miss - the iron is just a tool for building psychological capacity. The real weight being moved is mental. Each rep under the bar is practice for handling life’s inevitable pressures.

Research comparing powerlifters to other athletes shows distinct psychological advantages. Superior goal achievement psychology. Greater self-reliance and personal accountability. Better tolerance for training failure and more adaptive responses to setbacks.

The key isn’t avoiding challenges but engaging with them to promote growth rather than damage. Graduated challenges with adequate support and recovery create antifragile humans. Whether you’re adding 5 pounds to your deadlift or tackling a new project at work, the principles remain the same.

We’re designed to grow stronger through struggle, more capable through challenge, more resilient through adversity… when those challenges are calibrated right. The barbell just happens to be one of the best teachers for this lesson.

breaking down 'are your lights on?' chapter by chapter

2025-06-29T00:00:00+00:00

part 1: what is the problem?

chapter 1: a problem

the famous brontosaurus tower story kicks things off. 73-story building, elevators are terrible, tenants threatening to leave. everyone immediately jumps to solutions: faster elevators, more elevators, outside shafts, whatever.

but the authors stop and ask… whose problem is this actually??

if you think it’s the tenants’ problem, you get solutions like “speed up elevators.” if you think it’s the landlord’s problem, you get totally different solutions like “raise rents so you need fewer tenants” or my personal favorite, “burn down the building for insurance.”

the key insight: the fledgling problem solver invariably rushes in with solutions before taking time to define the problem being solved.

we’re all guilty of this. someone says “the website is slow” and we immediately start optimizing database queries instead of asking… slow for who? slow compared to what? is slow actually the problem or just a symptom?

chapter 2: peter pigeonhole prepares a petition

peter the mailboy organizes a petition about the elevators. landlord ignores it. workers escalate - they show up at his house with picket signs and stink bombs. suddenly his wife gets involved and now it’s definitely his problem too.

this introduces the “walking in moccasins” principle. problems don’t get solved until the pain is shared. when one party feels pain in sync with another, that’s when resolution happens.

also learned: if it’s their problem, make it their problem. peter gets assigned to solve the elevator issue because he started the whole thing. taking initiative means it becomes your problem.

chapter 3: what’s your problem?

here’s where they define what a problem actually is: a problem is a difference between things as desired and things as perceived.

if you want the room at 72 degrees but it feels like 68, you have a problem. doesn’t matter if the thermometer says 77 - if you perceive it as cold, the problem is real.

peter learns this and realizes elevator problems could be solved by changing perceptions instead of reality. so they install mirrors by the elevators. people get distracted checking themselves out and stop noticing the wait.

then vandals cover the mirrors with graffiti. peter’s solution? give everyone crayons to make their own graffiti while waiting.

plot twist: during annual inspection, they find a dead rat jammed in the master relay. the elevators were broken the whole time, not slow by design. when they fix it, elevators work perfectly… but now everyone rushes to the subway at once and the landlord gets trampled to death.

phantom problems are real problems - even if the root cause isn’t what you think, the experience of the problem is genuine and needs addressing.

part 2: what is the problem?

chapter 4: billy brighteyes bests the bidders

billy is a programmer who gets hired to solve a bidding problem. company has sealed bids from competitors and wants to compute 4 million possible combinations to find the optimal strategy.

billy looks at the rules for 5 minutes and solves it instantly with basic logic. the executives don’t believe him because if you solve their problem too readily, they’ll never believe you’ve solved their real problem.

also: don’t take their solution method for a problem definition. just because they want 4 million calculations doesn’t mean that’s actually the problem.

chapter 5: billy bites his tongue

plot twist - billy later discovers the same problem was solved by another company using linear programming packages. cost them $1,400 and took 3 days. his 5-minute solution saved way more money but wasn’t trusted.

lesson: don’t mistake a solution method for a problem definition, especially if it’s your own solution method.

sometimes people want the comfort of a complex, expensive solution because it feels more legitimate than a simple one.

chapter 6: billy back to the bidders

billy realizes the whole bidding situation was probably rigged from the start. if everyone bought the “secret” bids and changed theirs accordingly, then the original bids were just meant to mislead competitors.

this leads to infinite recursion: if i know you know i know you know… it becomes impossible to determine what’s real.

you can never be sure you have a correct definition, even after the problem is solved.

but that’s okay. the real lesson is: don’t leap to conclusions, but don’t ignore your first impression.

part 3: what is the problem, really?

chapter 7: the endless chain

dan the engineer creates a tool to mark precise measurements on paper - aluminum bar with pins that punch holes exactly 8 inches apart. works great until someone sets it down pin-side up and the section chief sits on it, getting two precise holes punched in his rear end.

each solution is the source of the next problem.

they “solve” this by grinding the legs into semicircles so it can’t stand upright. but this probably creates new problems too.

if you can’t think of at least three things that might be wrong with your understanding of the problem, you don’t understand the problem.

chapter 8: missing the misfit

designers create solutions for people they never meet, leading to “misfits” - solutions that don’t match actual human needs.

example: disposable razor blades were great for men who cut themselves sharpening razors, but terrible for wives and maids who cut themselves disposing of the blades. eventually someone invented packages that could hold used blades, but it took years to recognize the disposal problem.

each new point of view will produce a new misfit.

solution: test your definition on a foreigner, someone blind, or a child, or make yourself foreign, blind, or childlike.

chapter 9: landing on the level

shows a circle and asks “what is it?” most people say “circle” immediately. but change the problem statement to “this shows a very unfamiliar object” and suddenly people get creative: “hole,” “hula hoop,” “helicopter landing pad.”

the lesson is about “semantic levels” - we unconsciously decide what type of problem we’re dealing with, which constrains our solutions.

as you wander along the weary path of problem definition, check back home once in a while to see if you haven’t lost your way.

chapter 10: mind your meaning

“nothing is too good for our customers” - does this mean customers deserve the best, or that giving them nothing would be too generous?

words are tricky. the book gives examples of $500k mistakes caused by misplacing “too” in a sentence.

once you have a problem statement in words, play with the words until the statement is in everyone’s head.

they provide a “golden list” of word games: change positives to negatives, change “may” to “must,” replace “and” with “or,” draw pictures of sentences, express words as equations.

part 4: whose problem is it?

chapter 11: smoke gets in your eyes

classroom with 11 non-smokers and 1 cigar smoker. instead of the teacher mandating a solution, they let the students work it out. the smoker voluntarily agrees to stop smoking if everyone brings interesting snacks to share instead.

don’t solve other people’s problems when they can solve them perfectly well themselves.

if it’s their problem, make it their problem.

chapter 12: the campus that was all spaced out

university parking problem. students start parking in the president’s space, he threatens expulsion, they escalate to slashing his tires. meanwhile some faculty take a different approach: try blaming yourself for a change, even for a moment.

instead of “there aren’t enough parking spaces,” they reframe it as “i’m too lazy to walk far” or “i need exercise anyway.” some professors start deliberately parking in the farthest spots and walking for exercise.

if a person is in a position to do something about a problem, but doesn’t have the problem, then do something so he does.

chapter 13: the lights at the end of the tunnel

new tunnel has a sign: “turn your headlights on.” but at the scenic overlook after the tunnel, tourists leave lights on and drain batteries.

engineer considers complex solutions, then just puts up a simple sign at tunnel exit: “are your lights on?”

if people really have their lights on, a little reminder may be more effective than your complicated solution.

this is where the book title comes from.

part 5: where does it come from?

chapter 14: janet jaworski joggles a jerk

janet travels to poland to visit her grandmother. gray bureaucrat claims she’s missing one notarized copy of her paperwork. instead of assuming “bureaucracy is evil,” she asks: where does this problem come from?

could be: attendant lost the copy, she never had it, bureaucrat is incompetent, bureaucrat has different goals, bureaucrat lacks authority to make exceptions.

by not attributing it to “nature” or “that’s just how things are,” she keeps the problem solvable.

chapter 15: mister matczyszyn mends the matter

janet stops treating the bureaucrat like “mr. grayface” and learns his name is jan matczyszyn. turns out he’s human, they bond over family history, and he helps her make a copy for the missing paperwork.

the source of the problem is most often within you.

brutal but true. how we approach people affects how they respond.

chapter 16: make-works and take-credits

some people create work for others, some people do actual work. the book gives an example of a ridiculous memo about comma usage in punctuation reports.

in the valley of the problem solvers, the problem creator is king.

sometimes the “problem” exists just to justify someone’s job. solution: route the memo back with “fascinating concept, let’s discuss” written on it and watch the originator tie themselves in knots.

chapter 17: examinations and other puzzles

students learn to game homework by knowing which week’s material will be covered, then get destroyed on comprehensive finals. but exam problems aren’t from storks - they come from professors with biases and limitations.

who sent this problem? what’s he trying to do to me?

understanding the source helps solve the problem. multiple choice tests can often be solved by analyzing answer patterns rather than doing the math.

part 6: do we really want to solve it?

chapter 18: tom tireless tinkers with toys

young programmer finds a toy company with shipping cost optimization problem. he discovers they could save tons of money by closing two factories and making everything at the most efficient one.

“that’s true, but we can’t accept that solution.”

“why not?”

“because the president lives near the atlantic plant and the chairman lives in kansas city. they wouldn’t move for anything.”

in spite of appearances, people seldom know what they want until you give them what they ask for.

chapter 19: patience plays politics

programmer builds tax assessment system, treasurer complains the total is off by one penny due to rounding. she offers to donate a dollar to the state to cover rounding errors for the next decade.

not too many people, in the final analysis, really want their problems solved.

some problems are more useful unsolved because they justify someone’s existence or provide an excuse for inaction.

chapter 20: a priority assignment

code breaker spends two years cracking diplomatic codes, only to discover they’re just expense accounts. “twenty-three bottles scotch, fifty-nine wine…”

do i really want a solution?

sometimes the solution is so trivial it makes all your effort feel worthless. sometimes solving the problem eliminates your job or reveals uncomfortable truths.

the book doesn’t end with a neat conclusion because that would go against its entire philosophy. instead it just reminds you that problem solving is never morally neutral, and to “thine own self be true.”

the whole point is that there are no perfect problem definitions, no final solutions, and no universal frameworks. just keep asking the right questions and stay suspicious of your own assumptions.

how many times have i built exactly what the client asked for, only to have them realize it wasn’t what they needed? how many features have i added that created more problems than they solved?

sometimes the best solution is to do nothing at all. the elevator in brontosaurus tower worked fine once they removed the dead rat. sometimes the problem really is just a dead rat in the machine, and all the sophisticated solutions in the world won’t help until you deal with that basic fact.

Effect's pipe: the backbone of composable TypeScript

2025-06-21T00:00:00+00:00

You know that feeling when you chain Promises and suddenly you’re drowning in nested try-catch blocks? Effect’s pipe function takes a radically different approach. It’s not just another functional programming utility—it’s a complete rethinking of how we compose operations in TypeScript.

The pipe that knows what can go wrong

Here’s what makes Effect’s pipe different right off the bat:

const result = pipe(
  Effect.succeed(42),
  Effect.map(n => n / 2),
  Effect.flatMap(n => n > 20 ? Effect.succeed(n) : Effect.fail("too small")),
  Effect.mapError(msg => new ValidationError(msg))
)
// Type: Effect<number, ValidationError, never>

See that type signature? Every possible error is tracked. Every dependency is explicit. This isn’t your typical pipe function that just threads values through transformations. Effect’s pipe carries three pieces of information through every step: the success value, possible errors, and required dependencies.

How Effect pulls off this magic

The implementation relies on TypeScript’s overloading system pushed to its limits. Effect provides 20+ overloads to maintain type safety through long pipelines:

pipe<A, B, C, D>(
  a: A, 
  ab: (a: A) => B, 
  bc: (b: B) => C, 
  cd: (c: C) => D
): D

But here’s where it gets interesting. Effect uses a “dual API” pattern. Every function works two ways:

import { dual, pipe } from "effect/Function"

const sum = dual<
  (that: number) => (self: number) => number,
  (self: number, that: number) => number
>(2, (self, that) => self + that)

// Both styles work
sum(2, 3)        // Data-first: 5
pipe(2, sum(3))  // Data-last: 5

This isn’t just syntactic sugar. It’s about flexibility. Sometimes you want method chaining. Sometimes you want function composition. Effect says: why choose?

Real-world patterns

Here’s a common pattern that works well:

const createInvitation = ({ email, role, userId, orgId }) =>
  pipe(
    validateInvitationData({ email, role }),
    Effect.andThen(() => checkUserPermissions({ userId, orgId })),
    Effect.andThen(() => generateInvitationId()),
    Effect.flatMap(invitationId => 
      saveInvitation({ invitationId, email, role, orgId })
    ),
    Effect.tap(() => sendInvitationEmail(email)),
    Effect.mapError(error => 
      error instanceof DatabaseError 
        ? new InternalServerError("Database operation failed")
        : error
    ),
    Effect.retry({ times: 3, schedule: Schedule.exponential(1000) }),
    Effect.timeout(10000)
  )

Notice how each concern is separate? Validation, authorization, business logic, error handling, retries, timeouts—all composed through pipe. In Promise-land, this would be a tangled mess of try-catch blocks and manual retry logic.

Performance: the elephant in the room

Let’s address it head-on. Effect has overhead. The fiber-based runtime adds ~15KB to your bundle (compressed). Initial execution is slower than raw Promises.

But here’s the thing: Effect shines when complexity grows. That weather app fetching data from three APIs? Effect’s concurrent execution model offers advantages over Promise.all:

const weatherData = pipe(
  Effect.all({
    current: fetchCurrentWeather(city),
    forecast: fetchForecast(city),
    alerts: fetchWeatherAlerts(city)
  }, { concurrency: "unbounded" }),
  Effect.retry({ times: 3, schedule: Schedule.exponential(500) }),
  Effect.timeout(5000)
)

Each operation gets its own retry logic. One fails? Others continue. The fiber runtime handles this elegantly while Promises would require custom orchestration.

Comparing to other pipes

F#’s pipe operator |> provides readable left-to-right composition:

[1; 2; 3; 4; 5]
|> List.filter (fun x -> x % 2 = 0)
|> List.map (fun x -> x * x)
|> List.sum

Beautiful. Type-safe. But it’s just syntax. No error tracking, no async handling, no dependency injection.

Elixir also uses pipe operators for data transformation:

"hello world"
|> String.split()
|> Enum.map(&String.capitalize/1)
|> Enum.join(" ")

Elixir’s strength is combining pipes with pattern matching in function definitions, letting you handle different data shapes elegantly. But like F#, when you need explicit error tracking at the type level, you’re managing it yourself.

Unix pipes? They’re the OG:

cat file.txt | grep "pattern" | sort | uniq -c

Text streams, process isolation, parallel execution. These concepts influenced many modern functional programming approaches, including Effect’s composable operations.

Advanced patterns worth stealing

The service layer pattern works well for organizing Effect code:

const userService = {
  create: (userData: UserData) =>
    pipe(
      Schema.decodeUnknown(UserSchema)(userData),
      Effect.andThen(checkEmailUniqueness),
      Effect.andThen(hashPassword),
      Effect.flatMap(saveToDatabase),
      Effect.tap(sendWelcomeEmail),
      Effect.catchTags({
        ParseError: () => Effect.fail(new ValidationError("Invalid user data")),
        DatabaseError: () => Effect.fail(new ServiceUnavailable())
      })
    )
}

Each method is a pipeline. Errors bubble up with proper types. Testing? Swap the database layer:

const testLayer = Layer.succeed(DatabaseService, {
  save: () => Effect.succeed({ id: "test-id" }),
  find: () => Effect.succeed(null)
})

const result = pipe(
  userService.create(userData),
  Effect.provide(testLayer),
  Effect.runPromise
)

The retry scheduler pattern prevents naive retry loops:

const smartRetry = pipe(
  Schedule.exponential(Duration.seconds(1)),
  Schedule.jittered,
  Schedule.whileOutput(Duration.lessThanOrEqualTo(Duration.minutes(1))),
  Schedule.tapOutput(duration =>
    Effect.log(`Retrying after ${duration.seconds} seconds`)
  )
)

Exponential backoff with jitter, maximum duration, and logging. Try implementing that with Promises.

When Effect’s pipe actually helps (and when it doesn’t)

Effect shines when you have:

Complex error scenarios that need explicit handling
Multiple async operations that might fail independently
Business logic requiring retries, timeouts, and fallbacks
Team members who keep shipping bugs because “we forgot to handle that error”

Skip Effect when you’re:

Building a simple CRUD app with predictable failures
Working with a team hostile to functional programming
Prototyping something you’ll throw away next week

The migration path that actually works

Teams succeeding with Effect don’t rewrite everything. They start at the boundaries:

// Old Promise-based code
async function fetchUserData(id: string): Promise<User | null> {
  try {
    const response = await fetch(`/api/users/${id}`)
    if (!response.ok) throw new Error('User not found')
    return await response.json()
  } catch {
    return null
  }
}

// Gradual Effect adoption
const fetchUserData = (id: string) =>
  pipe(
    Effect.tryPromise({
      try: () => fetch(`/api/users/${id}`),
      catch: () => new NetworkError()
    }),
    Effect.filterOrFail(
      response => response.ok,
      () => new UserNotFoundError()
    ),
    Effect.andThen(response => 
      Effect.tryPromise({
        try: () => response.json(),
        catch: () => new ParseError()
      })
    ),
    Effect.andThen(Schema.decodeUnknown(UserSchema))
  )

Now errors are explicit. The compiler catches missing error handling. The team gradually learns Effect patterns without a big-bang rewrite.

The philosophical shift

Effect’s pipe isn’t just about chaining functions. It’s about making the implicit explicit. Every function in a pipeline must declare what it needs, what it returns, and what can go wrong.

Traditional error handling hides failure:

try {
  const user = await getUser(id)
  const profile = await getProfile(user.id)
  return await enrichProfile(profile)
} catch (error) {
  console.error('Something went wrong:', error)
  return null
}

What failed? Who knows. Effect forces honesty:

pipe(
  getUser(id),                    // Effect<User, UserNotFoundError, never>
  Effect.andThen(u => getProfile(u.id)), // Effect<Profile, ProfileError, never>
  Effect.andThen(enrichProfile),  // Effect<RichProfile, EnrichmentError, never>
  Effect.catchTags({
    UserNotFoundError: () => Effect.succeed(guestProfile),
    ProfileError: (e) => Effect.fail(new IncompleteDataError()),
    EnrichmentError: () => Effect.succeed(basicProfile)
  })
)

Every failure mode is visible. Every recovery strategy is explicit. The types tell the whole story.

Type inference that doesn’t make you cry

Effect preserves types through insanely long pipelines:

const complexPipeline = pipe(
  Effect.succeed({ name: "Alice", age: 30 }),
  Effect.map(user => ({ ...user, id: generateId() })),
  Effect.flatMap(user => 
    user.age >= 18 
      ? Effect.succeed(user)
      : Effect.fail(new UnderageError())
  ),
  Effect.andThen(user => fetchPremiumStatus(user.id)),
  Effect.map(status => status.isPremium ? "premium" : "basic"),
  Effect.retry({ times: 3 }),
  Effect.timeout(5000)
)
// Type: Effect<"premium" | "basic", UnderageError | FetchError | TimeoutException, never>

Twenty transformations deep? Types still flow. Compare this to Promise chains where you’re adding type annotations every other line just to keep TypeScript happy.

The ecosystem bonus

When you buy into Effect’s pipe, you get an entire ecosystem designed around it:

// HTTP client with built-in Effect support
const userData = pipe(
  HttpClientRequest.get("/api/user"),
  HttpClient.execute,
  Effect.andThen(response => response.json),
  Effect.andThen(Schema.decodeUnknown(UserSchema)),
  Effect.retry({ times: 3 }),
  Effect.provide(FetchHttpClient.layer)
)

// Stream processing 
const processedStream = pipe(
  Stream.fromIterable(largeDataset),
  Stream.map(processItem),
  Stream.filter(isValid),
  Stream.groupByKey(item => item.category),
  Stream.runCollect
)

Everything speaks the same language. Everything composes the same way.

Performance tricks

Memoization for expensive operations:

const expensiveCalculation = pipe(
  Effect.sync(() => {
    console.log("This only runs once!")
    return heavyComputation()
  }),
  Effect.cached
)

// Multiple calls, single execution
Effect.all([
  expensiveCalculation,
  expensiveCalculation,
  expensiveCalculation
])

Batching for database operations:

const getUserById = (id: string) =>
  pipe(
    Effect.request(GetUserById(id)),
    Effect.flatMap(Schema.decodeUnknown(UserSchema))
  )

// Automatically batches multiple getUserById calls
const users = Effect.all([
  getUserById("1"),
  getUserById("2"),
  getUserById("3")
])

The gotchas nobody talks about

Effect.gen looks tempting for developers coming from async/await:

const program = Effect.gen(function* () {
  const user = yield* getUser(id)
  const profile = yield* getProfile(user.id)
  return yield* enrichProfile(profile)
})

But overusing generators defeats the purpose. You lose the composability that makes pipe powerful. Save generators for complex control flow, not simple sequences.

Another gotcha: nested pipes get ugly fast:

// Don't do this
pipe(
  data,
  x => pipe(
    x,
    transform1,
    y => pipe(
      y,
      transform2,
      z => pipe(z, transform3)
    )
  )
)

Flatten it out. Extract functions. Keep pipes linear.

Effect’s pipe function represents a philosophical shift: stop pretending errors don’t exist. Stop hiding dependencies. Make everything explicit, and let the compiler help you build robust systems. It’s not always easy, but for complex applications, it’s a compelling choice for many complex applications.

Creating a Retro CSS Glitch Text Effect

2025-06-19T00:00:00+00:00

Remember those corrupted VHS tapes where the image would tear and shift with weird colors? Here’s how to recreate that effect with CSS.

See It In Action

GLITCH EFFECT!

The Trick

Start with this HTML:

<h1 class="glitch" data-text="GLITCH EFFECT!">GLITCH EFFECT!</h1>

That data-text attribute? We’ll use it to create ghost copies of the text. First, give your text that classic red/blue shift from old CRT monitors:

.glitch {
  font-size: 2.5rem;
  font-weight: bold;
  color: #fff;
  text-shadow: 
    2px 2px 0 #ff00ff,    /* Magenta shadow */
    -2px -2px 0 #00ffff;  /* Cyan shadow */
  position: relative;
  animation: glitch 2s infinite;
}

Now here’s where it gets fun. Use pseudo-elements to stack two more copies of the text:

.glitch::before,
.glitch::after {
  content: attr(data-text);
  position: absolute;
  left: 0;
  top: 0;
  width: 100%;
  height: 100%;
}

.glitch::before {
  animation: glitch-1 0.5s infinite;
  color: #ff00ff;
  z-index: -1;
}

.glitch::after {
  animation: glitch-2 0.5s infinite;
  color: #00ffff;
  z-index: -2;
}

The content: attr(data-text) is doing the heavy lifting here. It grabs the text from that data attribute we set up. Now you’ve got three layers of text ready to glitch out.

For the main text, animate those shadow positions to create movement:

@keyframes glitch {
  0%, 100% { text-shadow: 2px 2px 0 #ff00ff, -2px -2px 0 #00ffff; }
  25% { text-shadow: -2px 2px 0 #ff00ff, 2px -2px 0 #00ffff; }
  50% { text-shadow: 2px -2px 0 #ff00ff, -2px 2px 0 #00ffff; }
  75% { text-shadow: -2px -2px 0 #ff00ff, 2px 2px 0 #00ffff; }
}

But the real magic happens with clip-path. This is what creates that digital tearing effect:

@keyframes glitch-1 {
  0%, 100% { 
    clip-path: inset(0 0 0 0); 
    transform: translate(0); 
  }
  20% { 
    clip-path: inset(0 0 50% 0);
    transform: translate(-5px);
  }
  40% { 
    clip-path: inset(50% 0 0 0);
    transform: translate(5px);
  }
  60% { 
    clip-path: inset(0 0 0 0); 
    transform: translate(0); 
  }
}

clip-path: inset() chops off parts of the text. inset(0 0 50% 0) hides the bottom half, inset(50% 0 0 0) hides the top. Combine that with some horizontal shifts and you get that corrupted video look.

Making It Your Own

Want a subtle glitch? Slow down the animations. Want chaos? Speed them up to 0.2s. The sweet spot is usually around 0.5s for the pseudo-elements.

Different color combos work great too. Try green-on-green for that terminal look, or go full cyberpunk with purple and yellow. The key is high contrast. You want those layers to pop.

If you really want to sell the effect, add some JavaScript to trigger random intense glitches:

setInterval(() => {
  if (Math.random() > 0.7) {
    glitchElement.style.animation = 'glitch 0.2s infinite';
    setTimeout(() => {
      glitchElement.style.animation = 'glitch 2s infinite';
    }, 200 + Math.random() * 300);
  }
}, 3000);

Quick Notes

clip-path works everywhere that matters. If you’re worried about performance, stick to one or two glitch elements per page. Three layers of animated text can get heavy.

Oh, and for accessibility, kill the animations for people who prefer reduced motion:

@media (prefers-reduced-motion: reduce) {
  .glitch,
  .glitch::before,
  .glitch::after {
    animation: none !important;
  }
}

That’s it. Pure CSS, no libraries, just some clever use of pseudo-elements and animations. Perfect for headers, loading screens, or anywhere you want that corrupted digital aesthetic.

Print Debugging

2025-05-22T00:00:00+00:00

The Paradox of Choice was popularized by psychologist Barry Schwartz in his 2004 book of the same name, though the underlying concept has roots in earlier psychological and economic research.

Schwartz argued against the conventional wisdom that more options always lead to better outcomes and greater satisfaction. Instead, he demonstrated through various studies that beyond a certain point, additional choices can lead to decision paralysis, increased anxiety, and decreased satisfaction with whatever choice is ultimately made. His famous example involved a study of jam purchases at a grocery store: when customers were presented with 24 varieties of jam, fewer people made purchases compared to when only 6 varieties were available, and those who did buy from the larger selection were less satisfied with their choice.

This principle perfectly explains why developers default to print debugging despite having superior tools available. When faced with a bug, modern IDEs offer an overwhelming array of sophisticated options: integrated debuggers with breakpoints, step-through execution, variable watches, call stack inspection, conditional breakpoints, and profiling tools. Each tool requires learning its interface, making decisions about where to set breakpoints, and understanding how to interpret the results. This cognitive overhead creates decision paralysis, so developers instinctively retreat to the simplest option that requires zero choices: scattering print statements throughout their code. It’s the debugging equivalent of ordering the same meal at a restaurant with a 50-page menu - the familiar option feels safer than navigating all the potentially better alternatives, even when you know those alternatives would ultimately serve you better.

Every developer has reached for console.log or other similiar print statements when code misbehaves. You sprinkle logs throughout your functions, refresh the browser, and squint at the console trying to piece together what went wrong. While this approach feels familiar, it’s one of the least efficient ways to debug code, especially when tracing complex program flow or call stacks.

At its core, console.log debugging is just an inefficient, stone-age version of watch variables. You’re essentially trying to manually recreate what modern debuggers do automatically - except you have to guess what to watch, modify your source code, and parse through cluttered output instead of seeing clean, real-time variable updates.

The Three Core Problems with Print Debugging

First, you’re debugging blind. When tracking down bugs in deep call stacks, you’re playing a guessing game about where to place diagnostic output. You don’t know the actual execution path, so you scatter logs based on hunches. Consider this recursive function that’s causing stack overflows:

// Initial version - just crashes somewhere
function processNestedData(data, depth = 0) {
    if (!data) return null;

    if (data.children) {
        return data.children.map(child => processNestedData(child, depth + 1));
    }

    return data.value;
}

// First debugging attempt - add some logs
function processNestedData(data, depth = 0) {
    console.log('processNestedData called with:', data, 'depth:', depth);

    if (!data) {
        console.log('data is null/undefined, returning');
        return null;
    }

    console.log('about to check children');
    if (data.children) {
        console.log('has children, recursing');
        return data.children.map(child => processNestedData(child, depth + 1));
    }

    console.log('returning value:', data.value);
    return data.value;
}

// Still crashing, need more detail
function processNestedData(data, depth = 0) {
    console.log(`[DEPTH ${depth}] Processing:`, JSON.stringify(data));

    if (!data) return null;

    console.log(`[DEPTH ${depth}] Data type:`, typeof data);
    console.log(`[DEPTH ${depth}] Has children:`, !!data.children);

    if (data.children) {
        console.log(`[DEPTH ${depth}] Children count:`, data.children.length);
        console.log(`[DEPTH ${depth}] About to map over children`);

        const result = data.children.map((child, index) => {
            console.log(`[DEPTH ${depth}] Processing child ${index}:`, child);
            return processNestedData(child, depth + 1);
        });

        console.log(`[DEPTH ${depth}] Finished processing children`);
        return result;
    }

    return data.value;
}

You’re stuck in an endless cycle of adding logs, refreshing, running, and still not knowing exactly where the circular reference is or when the infinite recursion starts. Console.log forces you to predict what variables matter, while debugger watch variables let you inspect anything on demand.

Second, the development cycle kills productivity. Each investigation requires modifying source code, rebuilding (if using a bundler), and reloading. Modern build tools might be fast, but you’re still waiting to see if your latest batch of console.log statements provides useful insight. When they don’t—which happens frequently—you’re back to modifying code and waiting for another reload cycle. Look at how this simple bug hunt spirals out of control:

// First attempt - add some logs
async function fetchUserProfile(userId) {
    console.log('Fetching profile for user:', userId);  // Round 1

    const response = await fetch(`/api/users/${userId}`);
    const userData = await response.json();

    if (validateUserData(userData)) {
        updateUserCache(userData);
        renderUserProfile(userData);
    }
}

// Still not clear what's wrong, add more logs
async function fetchUserProfile(userId) {
    console.log('Fetching profile for user:', userId);
    console.log('User ID type:', typeof userId, 'Value:', userId);  // Round 2

    const response = await fetch(`/api/users/${userId}`);
    console.log('Response status:', response.status);  // Round 2
    console.log('Response headers:', response.headers);  // Round 2

    const userData = await response.json();
    console.log('Parsed user data:', userData);  // Round 2

    if (validateUserData(userData)) {
        console.log('Validation passed');  // Round 2
        updateUserCache(userData);
        renderUserProfile(userData);
    } else {
        console.log('Validation failed');  // Round 2
    }
}

// Still need to dig deeper, modify validation function
function validateUserData(userData) {
    console.log('Validating user data:', userData);  // Round 3
    console.log('Has email:', !!userData.email);  // Round 3
    console.log('Email value:', userData.email);  // Round 3

    if (!userData.email || userData.email.length === 0) {
        console.log('Email validation failed');  // Round 3
        return false;
    }
    // ... more validation logic with more logs
}

This is the stone age approach to variable inspection. You’re manually recreating what watch variables do automatically, but with all the overhead of code modification and none of the flexibility.

Third, you lose the bigger picture. Console.log debugging only shows you the specific points you thought to instrument, potentially reinforcing incorrect assumptions about how your code behaves. It’s like trying to watch a movie by taking random screenshots - you miss the flow and context. Consider this Promise chain where the bug could be anywhere:

function handleUserAction(actionType, payload) {
    console.log('Handling action:', actionType, payload);

    switch(actionType) {
        case 'LOGIN':
            return authenticateUser(payload)
                .then(result => {
                    console.log('Auth result:', result);  // Is this even called?
                    return updateUserSession(result);
                })
                .then(session => {
                    console.log('Session updated:', session);  // What about this?
                    return redirectToProfile(session.userId);
                });
        case 'LOGOUT':
            return clearUserSession()
                .then(() => {
                    console.log('Session cleared');  // Silent failure?
                    return redirectToHome();
                });
    }
}

async function authenticateUser(credentials) {
    console.log('Authenticating with:', credentials.username);
    // Where do I put the next log? In the API call? In error handling?
    try {
        const response = await fetch('/api/auth', {
            method: 'POST',
            body: JSON.stringify(credentials)
        });
        return await response.json();
    } catch (error) {
        console.log('Auth error:', error);  // But what kind of error?
        throw error;
    }
}

You’re instrumenting based on assumptions, potentially missing the real problem entirely.

The Three Advantages of Interactive Debugging

Start high and step down systematically. With a debugger, you set one breakpoint and step through the actual execution path. Watch variables show you real-time updates without any code modification:

function processNestedData(data, depth = 0) {
    if (!data) return null;  // <- Breakpoint here, add 'data' to watch

    if (data.children) {     // <- Step through, watch 'data.children' update
        return data.children.map(child => processNestedData(child, depth + 1));
    }

    return data.value;
}

In your browser debugger watch panel, you immediately see the problem:

▼ data: Object
  ▼ children: Array(1)
    ▼ 0: Object
        value: "child"
      ▼ parent: Object [Circular *1]  <- Circular reference detected!

No code modification needed. No parsing through console spam. Just clean, structured variable inspection.

Adapt your investigation in real time. Instead of guessing what to log, you examine variables on demand and add new watches instantly:

async function fetchUserProfile(userId) {
    // Breakpoint here - add userId to watch, see it's undefined
    const response = await fetch(`/api/users/${userId}`);  // Add response to watch
    const userData = await response.json();  // Add userData to watch

    if (validateUserData(userData)) {  // Step into this, watch validation logic
        updateUserCache(userData);     // Watch cache updates in real-time
        renderUserProfile(userData);
    }
}

In the debugger watch window (updated in real-time as you step):

userId: undefined        <- Added to watch when you noticed the problem
response.status: 404     <- Added when you stepped to this line
userData: {error: "User not found"}  <- Added automatically as you stepped
validateUserData(userData): false    <- Added to watch the function result

This is like having a time machine for your variables - you can watch how they change over time without having to predict what you’ll need to see.

See the complete context. For complex async flows, you can trace the entire chain while watching variables update across the entire call stack:

function handleUserAction(actionType, payload) {
    // Breakpoint here, step through the switch
    // Watch panel shows: actionType: "LOGIN", payload: {username: "john"}
    switch(actionType) {
        case 'LOGIN':
            return authenticateUser(payload)  // Step into, watch variables flow down
                .then(result => updateUserSession(result))  // Watch result propagate
                .then(session => redirectToProfile(session.userId));
    }
}

async function authenticateUser(credentials) {
    // Watch panel now shows: credentials: {username: "john", password: "***"}
    const response = await fetch('/api/auth', {
        method: 'POST',
        body: JSON.stringify(credentials)
    });
    // Watch panel updates: response: Response {status: 200, ok: true, ...}
    return await response.json();
}

Your debugger shows the complete call stack and async flow, with variables updating in real-time across all scopes.

The transition from print debugging requires some initial learning investment, but the productivity gains are substantial.

Modern browsers make this easier than ever, with powerful debugging tools built right into DevTools. The time spent learning these tools pays dividends in every debugging session thereafter, turning what used to be a time-consuming guessing game into a systematic process of investigation and understanding.

Understanding Rails HTTP Processing

2024-11-20T00:00:00+00:00

I had an interesting interview recently where the interviewer asked me what do you write in a Rails controller. I’ve written a lot of web apps and APIs and I really couldn’t understand the aim of the question. Is it a probe in the technical depth of how web frameworks work, or appreciation of how the controller organizes seperation of concerns? Something i’ve realized is modern day to day work is simply magic we don’t think about.

To appreciate why this matters, let’s remember what web development looked like before MVC frameworks. Here’s a typical PHP script from the early 2000s:

<?php
// article.php - A single file handling database, business logic, and view
$conn = mysql_connect("localhost", "user", "password");
mysql_select_db("blog");

if ($_POST['action'] == 'create') {
    $title = mysql_real_escape_string($_POST['title']);
    $content = mysql_real_escape_string($_POST['content']);
    mysql_query("INSERT INTO posts (title, content) VALUES ('$title', '$content')");
    header("Location: article.php");
}

$result = mysql_query("SELECT * FROM posts ORDER BY created_at DESC");
?>

<html>
<body>
    <form method="post">
        <input type="hidden" name="action" value="create">
        Title: <input type="text" name="title"><br>
        Content: <textarea name="content"></textarea><br>
        <input type="submit" value="Create Post">
    </form>

    <?php while ($row = mysql_fetch_assoc($result)) { ?>
        <h2><?php echo htmlspecialchars($row['title']); ?></h2>
        <p><?php echo htmlspecialchars($row['content']); ?></p>
    <?php } ?>
</body>
</html>

The PHP example shows:

Database connection mixed with business logic
SQL queries embedded directly in the page
HTML templates mixed with PHP code
No separation of concerns
Basic security handled manually (mysql_real_escape_string)

However, understanding this “magic” becomes crucial when:

Debugging complex routing issues
Building custom middleware
Optimizing application performance
Designing APIs that deviate from Rails conventions

Let’s peek behind the curtain and see how Rails transforms raw HTTP into the clean, object-oriented code we work with daily.

The HTTP Request Journey

Step 1: Raw HTTP Arrives

Everything starts with a raw HTTP request hitting your server:

GET /posts/5 HTTP/1.1
Host: example.com
Accept: text/html
User-Agent: Mozilla/5.0
Cookie: session=abc123

This text-based protocol carries all the information needed to process the request: method, path, headers, and potentially a body. But working with raw text would be cumbersome and error-prone.

Step 2: Rack Middleware Processing

Rack, the unsung hero of Ruby web applications, transforms raw HTTP into something more manageable.

env = {
  'REQUEST_METHOD' => 'GET',
  'PATH_INFO' => '/posts/5',
  'HTTP_ACCEPT' => 'text/html',
  'HTTP_COOKIE' => 'session=abc123'
}

This standardized hash becomes the common currency of request handling. Every piece of middleware can modify this hash, adding features like:

Session handling
Request parsing
Authentication
Logging

Step 3: Rails Router Takes Control

The router (config/routes.rb) acts as traffic control, determining which code should handle the request:

Rails.application.routes.draw do
  resources :posts
  # Expands to multiple routes including:
  # GET /posts/:id => posts#show
end

When a request arrives, the router:

Matches the path and HTTP method against defined routes
Extracts parameters from dynamic segments
Identifies the target controller and action

Step 4: Controller Processing

Finally, we reach familiar territory. The controller receives a clean, normalized request:

class PostsController < ApplicationController
  def show
    @post = Post.find(params[:id])
    render json: @post
  end
end

All the complexity of HTTP has been abstracted away. We can focus on business logic rather than protocol details.

Why This Matters

For Development

Understanding the request cycle helps you debug issues more effectively
Knowledge of middleware lets you add cross-cutting concerns cleanly
Appreciation of HTTP details improves API design

For Performance

Each layer adds some processing overhead
Knowing the flow helps identify optimization opportunities
Understanding middleware ordering can prevent unnecessary processing

Modernizing the Spring Boot Kotlin Tutorial: A Developer's Guide with Neovim

2024-11-17T00:00:00+00:00

After over a 6-year hiatus from Spring Boot in my professional work, I found myself curious about how the ecosystem had evolved. I wondered if I could make it work without reaching for the traditional IntelliJ IDEA approach, because “nvim btw”. Here’s what I discovered while attempting to modernize a Spring Boot Kotlin tutorial using just Neovim as my development environment. Neovim boots up in less than a second and compilation is blazing fast.

The State of the Tutorial

The original Spring Boot Kotlin tutorial, despite its age, remains surprisingly relevant. However, there are several key modern considerations:

Gradle configuration now uses the Kotlin DSL (build.gradle.kts)
Updated dependencies including Spring Boot 3.x and Kotlin 1.9.x
Modern Java toolchain requirements (JDK 21)
Usage of Jakarta EE instead of javax packages
Enhanced Kotlin compiler options and Spring plugin configurations

Project Setup

Before diving into development, you’ll need to set up your environment. Here’s what you need:

Prerequisites

Java Development Kit (JDK) 21
```
brew install openjdk@21
```
After installation, make sure JAVA_HOME is set correctly in your shell configuration.
Gradle Build Tool
```
brew install gradle
```
While the project includes the Gradle wrapper (gradlew), having Gradle installed locally can be helpful for other Kotlin projects.

Clone and Build

git clone https://github.com/sngeth/spring-boot-kotlin-demo
cd spring-boot-kotlin-demo
./gradlew build

Neovim as Your Kotlin IDE

One of the most interesting aspects of this journey was using Neovim instead of traditional IDEs like IntelliJ IDEA. Here’s how to make Neovim a powerful Kotlin development environment:

Essential Tools

Kotlin Language Server: The backbone of Kotlin development in Neovim
- Provides code completion
- Offers type information
- Handles syntax highlighting
- Detects missing imports automatically
```
brew install kotlin-language-server
```

LSP Configuration:

require('lspconfig').kotlin_language_server.setup({
  cmd = { "kotlin-language-server" },
  filetypes = { "kotlin" },
  root_dir = require('lspconfig.util').root_pattern(
    "settings.gradle",
    "settings.gradle.kts",
    "build.gradle",
    "build.gradle.kts"
  )
})

Code Actions: This functionality comes from multiple sources:
- The built-in LSP client in Neovim (:h lsp)
- nvim-cmp for completion
- null-ls or none-ls for additional formatting
- Popular plugins like nvim-code-action-menu

Development Workflow

Instead of relying on IDE-integrated build tools, the workflow becomes:

./gradlew build -t

This continuous build approach:

Watches for file changes
Recompiles automatically
Provides quick feedback on errors
Eliminates the need for IDE-specific build processes

Advantages of This Setup

Lightweight Development Environment
- Faster startup times
- Lower resource usage
- Familiar Vim keybindings
Modern Development Features
- Code completion via LSP
- Inline error highlighting
- Jump-to-definition functionality
- Symbol search across the project
Build Process Transparency
- Clear visibility of the build process
- Direct control over Gradle commands
- No IDE abstraction layer

Common Pitfalls and Solutions

Import Management
- Challenge: Missing IDE auto-import
- Solution: Kotlin Language Server handles this effectively
- Tip: Use :LspRestart if imports aren’t updating
Compile Error Detection
- Challenge: No immediate visual feedback
- Solution: Continuous Gradle build (-t flag)
- Enhancement: Configure quickfix list integration
Code Actions Availability
- Challenge: Understanding source of actions
- Solution: Multiple plugins working together:
  - LSP for basic actions
  - Additional plugins for enhanced functionality
  - Custom keybindings for frequent actions

Recommended Neovim Plugins for Kotlin Development

Base LSP Setup:

'neovim/nvim-lspconfig'
'hrsh7th/nvim-cmp'
'hrsh7th/cmp-nvim-lsp'

Enhanced Functionality:

'nvim-code-action-menu'
'telescope.nvim'
'null-ls.nvim'

Conclusion

While the Spring Boot Kotlin tutorial may be showing its age, the core concepts remain solid. Combining it with modern tools like Neovim and the Kotlin Language Server creates a powerful, efficient development environment. The continuous build approach with Gradle provides quick feedback, while LSP integration offers many features traditionally associated with full IDEs.

This setup proves that you don’t need a heavy IDE to develop Spring Boot applications with Kotlin effectively. The combination of Neovim’s efficiency, LSP’s intelligence, and Gradle’s build tools creates a streamlined, modern development experience that can rival traditional IDEs while maintaining the flexibility and lightness of a terminal-based workflow.

The Surprising Connection Between Smalltalk, Ruby, and C#: A Tale of Extension Methods

2024-11-14T00:00:00+00:00

When developers think of C#’s extension methods, they often associate them with LINQ and .NET’s modern features. But did you know that this powerful programming concept has roots dating back to the 1970s?

The Smalltalk Origins

In the 1970s, Smalltalk introduced a revolutionary concept called “open classes.” This feature allowed developers to add methods to existing classes, even core library classes, at runtime. Here’s what it looked like:

Object subclass: #String
    instanceMethods: [
        shout [
            ^self asUppercase, '!'
        ]
    ]

This was groundbreaking at the time - the ability to modify existing types without subclassing them. This feature would later influence many modern programming languages.

Ruby’s “Monkey Patching”

Fast forward to the 1990s, and Ruby embraced this concept with its own implementation of open classes:

class String
  def shout
    self.upcase + "!"
  end
end

# Now every String instance has access to this method
puts "hello".shout  # Outputs: HELLO!

This feature became so popular (and sometimes notorious) in Ruby that it earned the nickname “monkey patching.” While powerful, it highlighted both the benefits and potential dangers of unrestricted type extension.

C#: Formalizing the Concept

In 2007, C# 3.0 introduced what we now know as “extension methods.” This implementation took the lessons learned from dynamic languages and added static typing safety:

public static class StringExtensions
{
    public static string Shout(this string str)
    {
        return str.ToUpper() + "!";
    }
}

// Usage
"hello".Shout();  // Returns: "HELLO!"

C# formalized the concept and coined the term “extension methods” that many modern languages now use. The implementation addressed several key challenges:

Safety: Extensions can’t override existing methods
Scope: Extensions must be explicitly imported
Clarity: The syntax makes it clear that it’s an extension
Performance: Compile-time binding for better efficiency

Modern Language Implementations

CLOS (Common Lisp Object System)

;; Using generic functions
(defgeneric shout (thing))

(defmethod shout ((text string))
  (concatenate 'string (string-upcase text) "!"))

;; Usage
(shout "hello") ; Returns "HELLO!"

Kotlin

// Extension function
fun String.shout() = this.uppercase() + "!"

// Extension property
val String.doubleLength: Int
    get() = this.length * 2

// Extension with generics
fun <T> List<T>.secondOrNull(): T? = if (this.size >= 2) this[1] else null

// Usage
println("hello".shout())          // HELLO!
println("test".doubleLength)      // 8
listOf(1,2,3).secondOrNull()      // returns 2

Swift

extension String {
    // Simple extension
    func shout() -> String {
        return self.uppercased() + "!"
    }

    // Computed property
    var doubleLength: Int {
        return self.count * 2
    }

    // With parameters
    func repeated(times: Int) -> String {
        return String(repeating: self, count: times)
    }
}

// Usage
"hello".shout()           // "HELLO!"
"test".doubleLength       // 8
"ha".repeated(times: 3)   // "hahaha"

Scala

object StringExtensions {
    implicit class StringOps(val s: String) {
        def shout: String = s.toUpperCase + "!"

        def reverseShout: String = s.reverse.toUpperCase + "!"

        // With parameters
        def repeatTimes(n: Int): String = s * n
    }
}

// Usage
import StringExtensions._
"hello".shout        // "HELLO!"
"hello".reverseShout // "OLLEH!"
"ha".repeatTimes(3)  // "hahaha"

TypeScript

declare global {
    interface String {
        shout(): string;
        reverseShout(): string;
    }
}

String.prototype.shout = function(): string {
    return this.toUpperCase() + "!";
};

String.prototype.reverseShout = function(): string {
    return this.split('').reverse().join('').toUpperCase() + "!";
};

// Usage
"hello".shout();        // "HELLO!"
"hello".reverseShout(); // "OLLEH!"

Rust (Using Traits)

trait StringExtension {
    fn shout(&self) -> String;
    fn reverse_shout(&self) -> String;
}

impl StringExtension for String {
    fn shout(&self) -> String {
        format!("{}!", self.to_uppercase())
    }

    fn reverse_shout(&self) -> String {
        format!("{}!", self.chars().rev()
            .collect::<String>().to_uppercase())
    }
}

// Usage
let text = String::from("hello");
println!("{}", text.shout());         // "HELLO!"
println!("{}", text.reverse_shout()); // "OLLEH!"

What Makes This Interesting?

The evolution of extension methods shows how programming languages learn from each other:

Smalltalk proved the concept was useful
Ruby showed both the power and potential pitfalls
C# formalized the approach with static typing
Modern languages refined the implementation further

Key Takeaways

Many modern programming features have roots in older languages
Different languages can learn from each other across paradigms
Good ideas evolve as we learn their strengths and weaknesses
Static and dynamic languages can inspire each other’s features

Alternatives to Data-Test Attributes in Modern Testing

2024-11-10T00:00:00+00:00

Building maintainable end-to-end tests doesn’t have to be complicated. While data-test attributes are a common approach, there’s an elegant alternative that aligns better with how users actually interact with your application. Let’s explore how we can write more resilient tests by focusing on user behavior and accessibility patterns.

Building User-Centric Tests

Let’s explore how we can build tests that mirror real user interactions. Here’s a typical user settings panel implementation:

// Initial component implementation
const UserSettingsPanel = () => {
  return (
    <div data-testid="settings-panel">
      <div data-testid="settings-header">
        <h2 data-testid="settings-title">User Settings</h2>
      </div>
      <div data-testid="settings-form-container">
        <form data-testid="settings-form">
          <div data-testid="email-group">
            <label data-testid="email-label">
              Email
              <input
                data-testid="email-input"
                type="email"
                name="email"
              />
            </label>
          </div>
          <button
            data-testid="save-button"
            type="submit"
          >
            Save Changes
          </button>
        </form>
      </div>
    </div>
  );
};

// Initial tests
describe('User Settings Panel', () => {
  it('should update user settings', () => {
    cy.get('[data-testid="settings-panel"]').should('be.visible');
    cy.get('[data-testid="email-input"]').type('new@email.com');
    cy.get('[data-testid="save-button"]').click();
    cy.get('[data-testid="success-message"]').should('be.visible');
  });
});

Later, your team decides to improve accessibility and semantic structure. A careful developer might update the component like this:

// Refactored component maintaining both test IDs and semantic structure
const UserSettingsPanel = () => {
  return (
    <main data-testid="settings-panel">
      <header data-testid="settings-header">
        <h2 data-testid="settings-title">User Settings</h2>
      </header>
      <form data-testid="settings-form">
        <fieldset>
          <legend className="sr-only">User Preferences</legend>
          <div className="form-group" data-testid="email-group">
            <label htmlFor="email" data-testid="email-label">Email</label>
            <input
              id="email"
              type="email"
              name="email"
              aria-describedby="email-help"
              data-testid="email-input"
            />
          </div>
          <button
            type="submit"
            aria-label="Save user settings"
            data-testid="save-button"
          >
            Save Changes
          </button>
        </form>
    </main>
  );
};

By focusing on semantic HTML and accessibility patterns, we can create a more maintainable approach that:

Mirrors User Behavior: Tests interact with elements the same way users do
Promotes Accessibility: Using ARIA labels and semantic HTML improves both testing and user experience
Simplifies Component Structure: Components remain clean and focused on their primary purpose

A Better Approach

Instead of maintaining parallel identification systems, we could focus on testing the way users actually interact with our application:

// Component focusing on user interaction patterns
const UserSettingsPanel = () => {
  return (
    <main>
      <form aria-label="User Settings">
        <div>
          <label htmlFor="email">Email</label>
          <input
            id="email"
            type="email"
            name="email"
          />
        </div>
        <button type="submit">Save Changes</button>
      </form>
    </main>
  );
};

// Tests focusing on user interaction patterns
describe('User Settings Panel', () => {
  it('should update user settings', () => {
    cy.findByRole('form', { name: /user settings/i }).within(() => {
      cy.findByLabelText(/email/i).type('new@email.com');
      cy.findByRole('button', { name: /save changes/i }).click();
    });
    cy.findByRole('alert').should('be.visible');
  });
});

This approach offers several benefits:

Tests actual user experience
Encourages good accessibility practices
Reduces maintenance burden
Naturally resilient to refactoring

The Cypress Perspective

While Cypress actually recommends using data-* attributes in their Best Practices guide, there’s a growing movement in the testing community to reconsider this approach. Here’s why:

// Traditional Cypress test with data-test attributes
cy.get('[data-testid="login-form"]')
  .find('[data-testid="username-input"]')
  .type('user@example.com')

cy.get('[data-testid="submit-button"]').click()

// More resilient approach using user-centric selectors
cy.get('form').within(() => {
  cy.get('input[type="email"]').type('user@example.com')
  cy.get('button[type="submit"]').click()
})

Better Alternatives for Cypress

1. Use Semantic HTML and ARIA Roles

// Instead of:
cy.get('[data-testid="navigation"]')

// Use:
cy.get('nav')
// Or even better:
cy.get('[role="navigation"]')

2. Leverage Text Content and Labels

// Instead of:
cy.get('[data-testid="login-button"]')

// Use:
cy.contains('button', 'Log in')
// Or:
cy.get('button').contains('Log in')

3. Use Form Elements Wisely

// Instead of:
cy.get('[data-testid="email-input"]')

// Use:
cy.get('input[type="email"]')
// Or:
cy.get('label').contains('Email').siblings('input')

When Data Attributes Might Make Sense in Cypress

There are legitimate cases for data attributes in Cypress tests:

Dynamic content where text might change:

// Valid use case
cy.get('[data-testid="user-notification"]').should('be.visible')

Internationalized applications:

// Text content might vary by locale
cy.get('[data-testid="welcome-message"]').should('be.visible')

Complex data grids or tables:

cy.get('[data-testid="data-grid-row-1"]')
  .should('contain', expectedData)

Conclusion

While Cypress’s documentation suggests using data-test attributes, we can write more maintainable tests by prioritizing selectors that reflect how users actually interact with our applications. The key is finding the right balance between test reliability and maintenance overhead.

When writing Cypress tests, ask yourself: “How would a user find this element?” If the answer isn’t “by looking for a data-test attribute,” consider using a more user-centric selector.

Sources:

Cypress Best Practices Guide
Kent C. Dodds: “Making your UI tests resilient to change”

Making Invalid States Unrepresentable: From Java to Haskell

2024-11-03T00:00:00+00:00

Introduction

Recently, I came across an interesting example on X that demonstrates this concept using Java. Let’s explore how we can take this further using Haskell’s type system.

The Java Approach

Here’s the original Java code:

record Book(String title, ISBN isbn, List<Author> authors) {
    Book {
        Objects.requireNonNull(title);
        if (title.isBlank())
            throw new IllegalArgumentException("Title must not be blank");

        Objects.requireNonNull(isbn);
        Objects.requireNonNull(authors);
        if (authors.isEmpty())
            throw new IllegalArgumentException("There must be at least one author");

        // plus immutable copies as in the previous article
    }
}

This code does enforce our business rules:

Title cannot be null or blank
ISBN cannot be null
Authors list cannot be null or empty

However, there are several issues with this approach:

Runtime Validation: All checks happen at runtime. We won’t know about invalid data until the program is running.
Defensive Programming: We need to manually check for all invalid cases.
Documentation Required: Nothing in the type signature tells us about these constraints.
Partial Functions: The constructor can fail at runtime.

The Haskell Approach

Let’s see how we can use Haskell’s type system to enforce these rules at compile time while adding additional type safety:

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}
{-# LANGUAGE StrictData #-}

module Main where

import Control.Monad (when)
import Data.Text (Text)
import qualified Data.Text as T
import Data.List.NonEmpty (NonEmpty(..))
import qualified Data.List.NonEmpty as NE
import Data.Char (isDigit, isSpace)

-- New ISBN newtype for type safety
newtype ISBN = ISBN { unISBN :: Text }
  deriving (Show, Eq)

-- New Author newtype for type safety
newtype Author = Author { unAuthor :: Text }
  deriving (Show, Eq)

-- Improved Book type with newtypes
data Book = Book
  { bookTitle   :: !Text      -- Strict fields
  , bookISBN    :: !ISBN      -- Using ISBN newtype
  , bookAuthors :: !(NonEmpty Author)  -- Using Author newtype
  } deriving Show

-- Smart constructor for ISBN
mkISBN :: Text -> Either String ISBN
mkISBN text = do
  when (T.null text) $
    Left "ISBN must not be blank"
  let digits = T.filter isDigit text
  if T.length digits == 13 && T.all isDigit digits
    then Right $ ISBN text
    else Left "ISBN must be 13 digits"

-- Smart constructor for Author
mkAuthor :: Text -> Either String Author
mkAuthor text =
  if T.null text || T.all isSpace text
    then Left "Author name cannot be blank"
    else Right $ Author text

-- Smart constructor for NonEmpty list of authors
mkAuthors :: [Text] -> Either String (NonEmpty Author)
mkAuthors [] = Left "There must be at least one author"
mkAuthors texts = do
  authors <- traverse mkAuthor texts
  case authors of
    [] -> Left "There must be at least one author"  -- Should never happen due to previous check
    (a:as) -> Right $ a :| as

-- Smart constructor for Book
mkBook :: Text -> Text -> [Text] -> Either String Book
mkBook titleText isbnText authorTexts = do
  -- Title validation
  when (T.null titleText || T.all isSpace titleText) $
    Left "Title must not be blank"

  -- ISBN validation using smart constructor
  isbn <- mkISBN isbnText

  -- Authors validation using smart constructor
  authors <- mkAuthors authorTexts

  -- Create the book if all validations pass
  Right $ Book
    { bookTitle = titleText
    , bookISBN = isbn
    , bookAuthors = authors
    }

Key Improvements

Stronger Type Safety
- Introduced ISBN and Author newtypes to prevent mixing up text fields
- Each domain type has its own smart constructor with validation
- Strict fields for better performance
NonEmpty List for Authors
- Guarantees at least one author at the type level
- No runtime checks needed for non-emptiness
- Makes the constraint explicit in the type signature
Smart Constructors with Validation
- Separate validation logic for each domain type
- Composable validation using Either
- Clear error messages for each validation failure
Type-Level Guarantees
- Once a Book exists, we know it’s valid
- Invalid states are impossible to represent
- Compiler enforces our domain rules

Here’s an example of using this improved implementation:

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}
{-# LANGUAGE StrictData #-}

module Main where

import Control.Monad (when)
import Data.Text (Text)
import qualified Data.Text as T
import Data.List.NonEmpty (NonEmpty(..))
import qualified Data.List.NonEmpty as NE
import Data.Char (isDigit, isSpace)

-- Domain Types
newtype ISBN = ISBN { unISBN :: Text }
  deriving (Show, Eq)

newtype Author = Author { unAuthor :: Text }
  deriving (Show, Eq)

data Book = Book
  { bookTitle   :: !Text
  , bookISBN    :: !ISBN
  , bookAuthors :: !(NonEmpty Author)
  } deriving Show

-- Smart Constructors
mkISBN :: Text -> Either String ISBN
mkISBN text = do
  when (T.null text) $
    Left "ISBN must not be blank"
  let digits = T.filter isDigit text
  if T.length digits == 13 && T.all isDigit digits
    then Right $ ISBN text
    else Left "ISBN must be 13 digits"

mkAuthor :: Text -> Either String Author
mkAuthor text =
  if T.null text || T.all isSpace text
    then Left "Author name cannot be blank"
    else Right $ Author text

mkAuthors :: [Text] -> Either String (NonEmpty Author)
mkAuthors [] = Left "There must be at least one author"
mkAuthors texts = do
  authors <- traverse mkAuthor texts
  case authors of
    [] -> Left "There must be at least one author"
    (a:as) -> Right $ a :| as

mkBook :: Text -> Text -> [Text] -> Either String Book
mkBook titleText isbnText authorTexts = do
  when (T.null titleText || T.all isSpace titleText) $
    Left "Title must not be blank"

  isbn <- mkISBN isbnText
  authors <- mkAuthors authorTexts

  Right $ Book
    { bookTitle = titleText
    , bookISBN = isbn
    , bookAuthors = authors
    }

-- Helper function to display validation result
displayResult :: Either String Book -> IO ()
displayResult result = case result of
  Left err -> putStrLn $ "Error: " ++ err
  Right book -> putStrLn $ "Successfully created book: " ++ show book

-- Main function with test cases
main :: IO ()
main = do
  putStrLn "Testing book validation..."
  putStrLn "\n1. Valid book:"
  displayResult $ mkBook "Dune" "9780441172719" ["Frank Herbert"]

  putStrLn "\n2. Valid book with multiple authors:"
  displayResult $ mkBook "Good Omens" "9780441172719" ["Terry Pratchett", "Neil Gaiman"]

  putStrLn "\n3. Invalid: Empty title:"
  displayResult $ mkBook "" "9780441172719" ["Frank Herbert"]

  putStrLn "\n4. Invalid: Blank title (only spaces):"
  displayResult $ mkBook "   " "9780441172719" ["Frank Herbert"]

  putStrLn "\n5. Invalid: Wrong ISBN format:"
  displayResult $ mkBook "Dune" "123" ["Frank Herbert"]

  putStrLn "\n6. Invalid: No authors:"
  displayResult $ mkBook "Dune" "9780441172719" []

  putStrLn "\n7. Invalid: Blank author name:"
  displayResult $ mkBook "Dune" "9780441172719" ["  "]

  putStrLn "\n8. Invalid: Mix of valid and invalid authors:"
  displayResult $ mkBook "Dune" "9780441172719" ["Frank Herbert", "", "Kevin J. Anderson"]

  putStrLn "\n9. Invalid: Non-numeric ISBN:"
  displayResult $ mkBook "Dune" "abc0441172719" ["Frank Herbert"]

  putStrLn "\n10. Valid: ISBN with hyphens (still has 13 digits):"
  displayResult $ mkBook "Dune" "978-0441172719" ["Frank Herbert"]

Benefits of This Approach

Type-Level Guarantees
- Domain rules enforced by the compiler
- No need for defensive programming
- Invalid states are unrepresentable
Self-Documenting Code
- Types clearly indicate constraints
- Newtypes make the domain model explicit
- Smart constructors document validation rules
Better Error Handling
- Validation results are explicit in return types
- No runtime exceptions
- Composable error handling
Maintainability
- Easy to add new constraints
- Validation rules are reusable
- Type system catches errors at compile time

Conclusion

While the Java solution enforces business rules through runtime checks and exceptions, our improved Haskell approach pushes these checks to the type system and makes invalid states truly unrepresentable. The addition of newtypes and strict fields further enhances type safety and performance.

Key takeaways:

Use types to encode business rules when possible
Push validation to the boundaries of your system
Make impossible states impossible, not just checked
Let the compiler help you enforce your domain rules
Use newtypes to prevent mixing up similar types
Make data structures strict when appropriate

The end result is a system that’s not only safer but also more maintainable and self-documenting.

Using Longest Increasing Subsequence to Analyze Training Block Effectiveness

2024-11-03T00:00:00+00:00

When designing strength programs, we often organize training into 4-week blocks. By combining the Longest Increasing Subsequence (LIS) algorithm with training block analysis, we can identify which combinations of volume, intensity, and exercise variations lead to the best E1RM progressions.

Understanding the Core Algorithm

First, let’s look at the elegant algorithm that powers our analysis - the LIS implementation using Patience Sorting:

def find_progression(nums, min_improvement \\ 2.5) do
  # dp[i] stores smallest number that can end subsequence of length i+1
  dp = []
  prev = %{}  # For reconstructing the sequence

  {dp, prev} = Enum.with_index(nums)
    |> Enum.reduce({dp, prev}, fn {num, i}, {dp, prev} ->
      # Find position where this number belongs
      pos = find_position(dp, num, min_improvement)

      # Track for reconstruction
      prev = if pos > 0,
        do: Map.put(prev, i, {pos - 1, Enum.at(dp, pos - 1)}),
        else: prev

      # Update dp array
      dp = if pos == length(dp),
        do: dp ++ [num],
        else: List.replace_at(dp, pos, num)

      {dp, prev}
    end)

  # Reconstruct the sequence
  {length(dp), reconstruct(nums, prev, length(dp) - 1, dp)}
end

defp find_position(dp, target, min_improvement) do
  do_binary_search(dp, target, min_improvement, 0, length(dp))
end

defp do_binary_search(dp, target, min_improvement, left, right) when left < right do
  mid = div(left + right, 2)
  mid_val = Enum.at(dp, mid)

  cond do
    mid_val == nil -> left
    target - mid_val >= min_improvement ->
      do_binary_search(dp, target, min_improvement, mid + 1, right)
    true ->
      do_binary_search(dp, target, min_improvement, left, mid)
  end
end

defp do_binary_search(_, _, _, left, _), do: left

defp reconstruct(nums, prev, pos, dp) do
  case Map.get(prev, pos) do
    nil -> [Enum.at(dp, pos)]
    {prev_pos, prev_num} ->
      reconstruct(nums, prev, prev_pos, dp) ++ [Enum.at(dp, pos)]
  end
end

This algorithm finds the longest sequence of E1RMs where each value is at least min_improvement greater than the previous. Time complexity is O(n log n).

Structuring Training Data

defmodule TrainingBlock do
  defstruct [
    :block_number,
    :start_date,
    :end_date,
    :primary_movement,     # e.g., "Comp Squat", "Paused Bench"
    :volume_per_session,   # sets * reps
    :intensity_range,      # % of E1RM
    :frequency_per_week,
    :variations_used,      # e.g., ["Paused", "Tempo", "Close Grip"]
    :sets,                 # List of actual training sets
    :starting_e1rm,
    :ending_e1rm
  ]
end

defmodule TrainingSet do
  defstruct [:date, :weight, :reps, :rpe, :e1rm]

  def calculate_e1rm(weight, reps) do
    weight * (36 / (37 - reps))  # Brzycki formula
  end
end

Block Analysis System

defmodule BlockAnalyzer do
  def analyze_progression_patterns(blocks, min_improvement \\ 2.5) do
    # Group blocks by exercise
    blocks_by_exercise = Enum.group_by(blocks, & &1.primary_movement)

    # Analyze each exercise
    Enum.map(blocks_by_exercise, fn {exercise, exercise_blocks} ->
      # Get E1RMs
      e1rms = Enum.map(exercise_blocks, & &1.ending_e1rm)

      # Find progression using LIS
      {length, progression} = find_progression(e1rms, min_improvement)

      # Map back to block characteristics
      successful_blocks =
        exercise_blocks
        |> Enum.filter(& &1.ending_e1rm in progression)
        |> Enum.sort_by(& &1.block_number)

      {exercise, analyze_characteristics(successful_blocks)}
    end)
  end

  defp analyze_characteristics(blocks) do
    %{
      avg_volume_per_session: average_volume(blocks),
      most_successful_intensity: common_intensity_range(blocks),
      optimal_frequency: most_common_frequency(blocks),
      effective_variations: most_effective_variations(blocks),
      block_sequence: extract_block_sequence(blocks)
    }
  end
end

Example Usage and Output

Here’s how we can analyze 6 months (6 blocks) of training:

blocks = [
  %TrainingBlock{
    block_number: 1,
    primary_movement: "Competition Bench",
    volume_per_session: 15,  # 5 sets of 3
    intensity_range: "80-85%",
    frequency_per_week: 2,
    variations_used: ["Paused"],
    starting_e1rm: 100,
    ending_e1rm: 102.5
  },
  %TrainingBlock{
    block_number: 2,
    primary_movement: "Competition Bench",
    volume_per_session: 24,  # 6 sets of 4
    intensity_range: "75-80%",
    frequency_per_week: 2,
    variations_used: ["Paused", "Tempo"],
    starting_e1rm: 102.5,
    ending_e1rm: 105
  },
  # ... more blocks ...
]

analysis = BlockAnalyzer.analyze_progression_patterns(blocks)

The output shows us the successful progression patterns:

%{
  "Competition Bench" => %{
    progression: [102.5, 105.0, 108.5, 112.0],
    characteristics: %{
      avg_volume_per_session: 20,
      most_successful_intensity: "75-80%",
      optimal_frequency: 2,
      effective_variations: ["Paused", "Tempo"],
      block_sequence: [
        %{volume: "moderate", intensity: "moderate"},
        %{volume: "high", intensity: "moderate"},
        %{volume: "moderate", intensity: "high"},
        %{volume: "low", intensity: "very high"}
      ]
    }
  }
}

How It Works

Finding True Progression

# Example E1RMs: [100, 102.5, 101, 105, 104, 108]
# With min_improvement = 2.5kg:

Step 1: [100]
Step 2: [100, 102.5]     # Valid as 102.5 - 100 = 2.5kg
Step 3: [100, 102.5]     # Skip 101 as it's lower than 102.5
Step 4: [100, 105]       # Replace 102.5 with 105 (better improvement)
Step 5: [100, 105]       # Skip 104 as it's lower than 105
Step 6: [100, 105, 108]  # Add 108 (valid 3kg improvement)

# Result: [100, 105, 108] - Three block progression

Block Pattern Analysis
- Each number in our progression represents a successful block
- We analyze characteristics of these blocks
- Look for patterns in volume, intensity, and variation

Using the Results

Program Design

def design_next_block(current_e1rm, analysis_results) do
 successful_pattern = find_matching_pattern(analysis_results)
 next_block_characteristics = predict_next_block(successful_pattern)

 %{
   suggested_volume: next_block_characteristics.volume,
   suggested_intensity: next_block_characteristics.intensity,
   suggested_variations: next_block_characteristics.variations,
   expected_improvement: next_block_characteristics.expected_gain
 }
end

Progress Prediction

def predict_block_outcome(current_e1rm, block_characteristics) do
 similar_blocks = find_similar_blocks(block_characteristics)
 average_improvement = calculate_avg_improvement(similar_blocks)

 %{
   expected_improvement: average_improvement,
   confidence: calculate_confidence(similar_blocks),
   recommended_modifications: suggest_modifications(similar_blocks)
 }
end

Conclusion

By using the LIS algorithm with Patience Sorting, we can:

Find genuine progression patterns in our training
Identify block characteristics that lead to consistent progress
Make data-driven decisions about program design
Predict likely outcomes of different block structures

The algorithm’s efficiency (O(n log n)) makes it practical for analyzing large training histories, while its ability to find strictly increasing sequences with minimum improvements makes it perfect for strength training analysis.

Making a D3 Sankey Chart Responsive in React

2024-10-26T00:00:00+00:00

Creating data visualizations that work well across different screen sizes can be challenging. Today, I’ll walk you through how I enhanced a D3.js Sankey chart to be more responsive in a React application. Here’s how we made our chart adapt seamlessly to any screen size. The app is deployed here and the full source code is http://github.com/sngeth/cash-flow

The Challenge

Our initial Sankey chart worked well on desktop but had several limitations on smaller screens:

Labels would overlap on narrow viewports
Node spacing was too wide for mobile screens
Font sizes were too large for smaller displays
The chart wouldn’t resize smoothly on window resize

Key Responsive Improvements

1. Dynamic SVG Dimensions

export default function SankeyChart({ income, savings, billItems }: SankeyChartProps) {
  const svgRef = useRef<SVGSVGElement>(null);

  const createChart = useCallback(() => {
    if (!svgRef.current) return;

    const svg = d3.select(svgRef.current);
    const width = svg.node()!.getBoundingClientRect().width;
    const height = svg.node()!.getBoundingClientRect().height;
  ...
}

2. Adaptive Node Padding

We adjust the spacing between nodes based on screen width:

const nodePadding = width < 600 ? 10 : 20;

const sankeyGenerator = sankey<SankeyNodeExtended, SankeyLink<SankeyNodeExtended, {}>>()
  .nodeWidth(10)
  .nodePadding(nodePadding)
  .extent([[1, 1], [width - 1, height - 6]]);

This provides:

Comfortable spacing on desktop (20px)
Compact layout on mobile (10px)
Better use of available space across all devices

3. Responsive Text Handling

We implement dynamic font sizing based on viewport width:

const fontSize = width < 600 ? "10px" : "12px";

node.append("text")
  .attr("font-size", fontSize)
  .attr("x", d => (d.x0 ?? 0) < width / 2 ? (d.x1 ?? 0) + 6 : (d.x0 ?? 0) - 6)
  .attr("y", d => ((d.y1 ?? 0) + (d.y0 ?? 0)) / 2)
  .attr("dy", "0.35em")
  .attr("text-anchor", d => (d.x0 ?? 0) < width / 2 ? "start" : "end")
  .text(d => `${d.name}: $${d.value ?? 0}`);

Key features:

Smaller font on mobile devices
Dynamic text positioning
Smart text anchor points based on node position

4. Smooth Resize Handling

We implemented efficient window resize handling:

useEffect(() => {
  createChart();
  const handleResize = () => {
    createChart();
  };
  window.addEventListener('resize', handleResize);
  return () => {
    window.removeEventListener('resize', handleResize);
  };
}, [createChart]);

This ensures:

Chart redraws on window resize
Clean cleanup of event listeners
Smooth transitions between sizes

5. Clean Redraws

Before each redraw, we clear the previous chart:

svg.selectAll('*').remove();

Locality of Behavior vs SOLID: Finding Balance in Code Organization

2024-10-26T00:00:00+00:00

Software companies often push for modular, highly-abstracted code in pursuit of flexibility and maintainability. However, this approach can inadvertently create significant cognitive overhead for developers, especially those new to a codebase. As codebases grow more complex and distributed, developers increasingly face mental fatigue from juggling numerous abstractions and navigating sprawling file structures. This raises an important question: Are our current practices truly serving us, or are they contributing to developer burnout? The resurgence of interest in locality of behavior, along with the popularity of tools like HTMX and the emergence of “anti-design patterns,” suggests a growing desire for simpler, more cognitively manageable code structures. But how do we balance these competing concerns?

Reflecting on my experience applying for a software internship in 2008, I recall being bombarded with questions about object-oriented programming (OOP), inheritance, and polymorphism. At the time, these concepts were considered essential for writing and understanding modular code. The industry’s focus on these principles stemmed from the belief that they led to more maintainable and scalable software. However, this approach raises an important question: Did the emphasis on OOP truly prepare developers for the complexities of real-world software development? While these concepts can be powerful tools, they don’t necessarily justify the cognitive overhead they introduce. Interview questions rarely addressed the critical skill of determining when such complexity is warranted or how to balance modularity with code readability and maintainability. This disconnect between interview practices and practical development needs highlights the ongoing challenge of finding the right balance in code organization and design.

Understanding Locality of Behavior

Before diving into code organization patterns, let’s understand a fundamental principle that often conflicts with traditional SOLID advice: Locality of Behavior (LoB).

Locality of Behavior was prominently discussed by Richard P. Gabriel in his patterns work and gained more attention through Alan Kay’s ideas about object-oriented programming. However, it really entered mainstream discussion through Rich Hickey (creator of Clojure) who has spoken about it extensively.

The core idea is simple but powerful: code should be organized so that related behaviors are kept close together. In other words, all the code needed to understand a particular operation should be in the same place.

This principle has strong academic roots:

Richard P. Gabriel discussed it in “Patterns of Software: Tales from the Software Community” (1996)
Rich Hickey’s “Simple Made Easy” presentation explores the cognitive overhead of scattered code
John Ousterhout’s “A Philosophy of Software Design” (2018) discusses “deep modules” that keep implementation details close to their interface

Let’s examine how this principle plays out in real code.

The Case for Keeping Things Together

First, let’s look at code with high locality of behavior:

class FileProcessor
  def process(file)
    case file.extension
    when '.csv'
      process_csv(file)    # CSV behavior is local
    when '.json'
      process_json(file)   # JSON behavior is local
    end
  end

  private

  def process_csv(file)
    CSV.read(file.path).map { |row| row.map(&:strip) }  # The full CSV behavior is visible right here
  end

  def process_json(file)
    JSON.parse(File.read(file.path))                    # The full JSON behavior is visible right here
  end
end

Compare this with code that has low locality of behavior:

class FileProcessor
  def process(file)
    processor_for(file.extension).process(file)  # Have to look elsewhere to find the processor
  end
end

class CsvProcessor
  def process(file)
    clean_values(           # Have to look elsewhere to find what clean_values does
      read_csv(file)       # Have to look elsewhere to find what read_csv does
    )
  end
end

module ValueCleaner
  def clean_values(data)   # The actual behavior is far from where it's used
    data.map { |row| row.map(&:strip) }
  end
end

The Great SOLID Debate

Before we dive deeper, let’s address the elephant in the room: SOLID principles, particularly the Open/Closed Principle (OCP), have faced criticism in recent years. Critics argue that breaking everything into separate files and abstractions can actually make code harder to understand. They have a point – let’s look at both sides.

Different Approaches to Code Organization

The Inheritance Approach

Here’s how many developers first attempt to separate concerns:

# base_processor.rb
class BaseProcessor
  def process(file)
    raise NotImplementedError
  end

  protected

  def strip_values(data)
    data.map { |row| row.map(&:strip) }
  end
end

# csv_processor.rb
class CsvProcessor < BaseProcessor
  def process(file)
    data = CSV.read(file.path)
    strip_values(data)
  end
end

# json_processor.rb
class JsonProcessor < BaseProcessor
  def process(file)
    JSON.parse(File.read(file.path))
  end
end

# file_processor.rb
class FileProcessor
  PROCESSORS = {
    '.csv' => CsvProcessor,
    '.json' => JsonProcessor
  }

  def process(file)
    processor_class = PROCESSORS[file.extension] ||
      raise("Unsupported format: #{file.extension}")

    processor_class.new.process(file)
  end
end

Mental Model Required:

Understand class inheritance
Know to look in multiple files
Grasp abstract base classes
Learn about class registration patterns

New Developer Questions:

“Why do we need a BaseProcessor? Where are the actual processing methods? How do I find which processor handles which format? Why is strip_values in the base class?”

The Composition Approach

Here’s a composition-based approach:

# processors/csv.rb
module Processors
  class Csv
    def self.process(file)
      new(file).process
    end

    def initialize(file)
      @file = file
    end

    def process
      ValueCleaner.new(
        CsvReader.new(@file)
      ).process
    end
  end
end

# processors/components/csv_reader.rb
class CsvReader
  def initialize(file)
    @file = file
  end

  def process
    CSV.read(@file.path)
  end
end

# processors/components/value_cleaner.rb
class ValueCleaner
  def initialize(source)
    @source = source
  end

  def process
    @source.process.map { |row| row.map(&:strip) }
  end
end

# file_processor.rb
class FileProcessor
  PROCESSORS = {
    '.csv' => Processors::Csv,
    '.json' => Processors::Json
  }

  def process(file)
    processor_class = PROCESSORS[file.extension] ||
      raise("Unsupported format: #{file.extension}")

    processor_class.process(file)
  end
end

Mental Model Required:

Understand object composition
Grasp dependency injection
Know about component assembly
Navigate deeper directory structures

New Developer Questions:

“Why are there so many small classes? How do these pieces fit together? Where does the processing actually happen? How do I trace the flow?”

Finding Balance: A More Approachable Solution

Here’s a middle ground that maintains separation while being more approachable:

# file_processor.rb
class FileProcessor
  def process(file)
    processor_for(file.extension).process(file)
  end

  private

  def processor_for(extension)
    case extension
    when '.csv' then CsvProcessor.new
    when '.json' then JsonProcessor.new
    else raise "Unsupported format: #{extension}"
    end
  end
end

# processors.rb
class CsvProcessor
  def process(file)
    clean_values(
      read_csv(file)
    )
  end

  private

  def read_csv(file)
    CSV.read(file.path)
  end

  def clean_values(data)
    data.map { |row| row.map(&:strip) }
  end
end

class JsonProcessor
  def process(file)
    JSON.parse(File.read(file.path))
  end
end

Mental Model Required:

Basic object-oriented programming
Simple method delegation
Two files to navigate

New Developer Experience:

“I can see how processors are selected and where their logic lives. Adding a new format means adding a new processor class with a process method. The processing steps are clear within each processor.”

Key Insights for Real-World Development

Cognitive Load Matters
- Every layer of abstraction is a concept developers must hold in their head
- More files = more context switching
- Simpler patterns = faster onboarding
The Cost of Flexibility
- Inheritance creates rigid hierarchies that are hard to change
- Deep composition can make code flow hard to follow
- Not every difference needs its own abstraction
Signs You Might Be Over-Separating
- You need a diagram to explain the code structure
- New developers frequently ask “where does X happen?”
- Changes require touching many files
- Test setup becomes complex
When Separation Makes Sense
- Processing logic is complex (>20-30 lines)
- Components have different deployment/testing needs
- Different teams own different processors
- Performance requires lazy loading

Practical Guidelines

Start Together
- Keep code in one place until patterns emerge
- Don’t separate based on speculation
- Let real requirements drive design
Separate Gradually
- Move code out when it proves necessary
- Keep related code close together
- Document why separation was needed
Optimize for Understanding
- Could a new developer understand this in their first week?
- Is the separation making the code clearer or just more “proper”?
- Are you solving real problems or theoretical ones?

Benefits of Locality of Behavior

Reduced cognitive load - developers don’t have to jump between files
Easier debugging - the full context is visible
Better performance - related code tends to be loaded together
Simpler testing - fewer dependencies to mock

The principle doesn’t mean “put everything in one file” but rather “keep related behaviors together.” The challenge is determining what “related” means in your specific context.

Conclusion

The best code isn’t the most perfectly separated – it’s the code that helps your team move quickly and confidently. Sometimes that means keeping things together, even if it doesn’t satisfy every SOLID principle.

Remember: Every layer of indirection you add is a concept that must live in a developer’s mental model of the system. Choose wisely.

What’s your experience with code organization patterns? How do you balance separation with understandability? Share your thoughts in the comments below.

References

Gabriel, Richard P. (1996). “Patterns of Software: Tales from the Software Community”
Hickey, Rich. “Simple Made Easy” presentation
Ousterhout, John. (2018). “A Philosophy of Software Design”

Database Indexes

2024-10-25T00:00:00+00:00

Why This Interview Question Needs a Rethink

Software companies frequently ask about database indexing during interviews, which might seem puzzling given that complexity analysis can be easily googled. Even more puzzling: if the goal is to assess system design knowledge, why not directly ask about specific scaling challenges or data access patterns?

The truth is, this question often reveals more about the interviewer’s habits than their assessment goals. A better line of questioning might be:

“What read/write patterns in your current system influenced your indexing strategy?”
“How did you determine when to add or remove indexes in production?”
“What monitoring helped you identify index-related performance issues?”

These questions would better reveal an engineer’s practical experience with database performance tuning. Nevertheless, let’s explore both the theoretical foundations and real-world implications that make index knowledge crucial for day-to-day engineering decisions.

The Theoretical Foundation

Complexity Analysis

Without an index (sequential scan):

Time complexity: O(n) where n is the number of rows
Every row must be examined to find matches
Optimal for scanning large portions of the table (>15-20% of rows)

With a B-tree index:

Time complexity: O(log n) for lookups, inserts, and deletes
B-tree height typically remains 2-4 levels even with millions of rows
Each level requires one disk I/O operation
Ideal for highly selective queries

Consider a table with 1,000,000 rows:

Sequential scan requires checking all 1,000,000 rows
B-tree index typically needs only 3-4 lookups

Inside a B-tree Index: A Practical Example

Understanding how B-tree indexes actually work helps explain both their performance characteristics and limitations. Let’s walk through a concrete example.

B-tree Structure

Consider a table of users with an index on the age column:

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    name TEXT,
    age INTEGER,
    email TEXT
);

CREATE INDEX idx_users_age ON users(age);

The resulting B-tree structure might look like this:

Root Node (Level 0)
[20, 40, 60]
 |   |   |   |
 v   v   v   v
Level 1 Nodes
[10,15] [25,30,35] [45,50,55] [70,80,90]
 |  |  |  |  |  |  |  |  |   |  |  |
 v  v  v  v  v  v  v  v  v   v  v  v
Leaf Nodes (Level 2)
[Pointers to actual table rows...]

How Lookups Work

Let’s trace what happens when we execute:

SELECT * FROM users WHERE age = 25;

Root Node Traversal

def find_in_node(node, target):
    # Binary search within node's keys
    for i, key in enumerate(node.keys):
        if target < key:
            return node.children[i]
        elif target == key:
            return node.children[i + 1]
    return node.children[-1]

def btree_search(root, target):
    current = root
    while not current.is_leaf:
        current = find_in_node(current, target)

Leaf Node Access

class LeafNode:
    def __init__(self):
        self.keys = []  # The indexed values
        self.row_pointers = []  # Pointers to actual table rows
        self.next_leaf = None  # For range scans

def get_row_pointers(leaf_node, target):
    matches = []
    for i, key in enumerate(leaf_node.keys):
        if key == target:
            matches.append(leaf_node.row_pointers[i])
    return matches

Insert Operations

When inserting a new record:

def insert(root, key, row_pointer):
    # Find the appropriate leaf node
    leaf = find_leaf_node(root, key)

    # If leaf has space, simply insert
    if len(leaf.keys) < MAX_KEYS:
        insert_in_leaf(leaf, key, row_pointer)
        return root

    # Otherwise, split the node
    new_leaf = split_leaf(leaf, key, row_pointer)

    # Propagate the split upward if necessary
    return propagate_split(root, leaf, new_leaf)

The Real-World Impact

This is where theoretical knowledge transforms into practical engineering decisions. Here’s what actually happens in production systems:

Write Performance Impact

Single Record Operations
- Base insert without index: ~1ms
- Each additional index adds: ~2-10ms overhead
- Impact: 4-5 indexes can make inserts 3-5x slower
Bulk Operations
- 1M row import with no indexes: ~2-3 minutes
- Same import with 3 indexes: ~5-8 minutes
- With 5+ indexes: Can extend to 15+ minutes or more

When It Really Hurts

The performance impact becomes particularly noticeable in:

High-frequency Insert Systems
- Logging systems
- Real-time data pipelines
- IoT data collection
- High-volume transaction systems
Development Pain Points
- Adding “just one more index” suddenly making writes noticeably slower
- Background index creation blocking production writes
- Unexpected storage growth (each index can add 20-30% to table size)

Making the Right Engineering Decisions

Understanding both theoretical and practical aspects helps engineers make better decisions:

Index Strategically
- Don’t index everything that could be queried
- Consider query-to-write ratio for each table
- Monitor index usage and remove unused indexes
Balance Performance Tradeoffs
- Accept slower writes for critical read performance
- Consider partial indexes for large tables
- Use covering indexes for crucial queries
Plan for Scale
- Anticipate growth in both data volume and query patterns
- Consider index maintenance windows
- Monitor index bloat and performance degradation

Conclusion

While understanding B-tree complexity is important, the real engineering value comes from:

Recognizing specific access patterns in your system
Understanding the concrete performance implications
Making informed tradeoffs based on actual requirements

The next time you’re interviewing candidates, consider skipping the theoretical complexity question. Instead, ask about their experience with real database performance challenges and how they measured, monitored, and resolved them. These answers will tell you far more about their engineering capabilities than whether they can recite trivia knowledge.

C# Evolution: A Practical Implementation Guide (6.0 to 12.0)

2024-10-24T00:00:00+00:00

C# 6.0 marked my introduction to the language. After going back to full time professional Ruby development, it seems i’ve missed quite a bit. Features like tuples and pattern matching, which I don’t recall using, are particularly fun from other languages. The full history of changes can be found here but I’d appreciate insights on any other important day-to-day concepts I might have overlooked.

C# 6.0 (2015)
C# 7.0 (2017)
C# 8.0 (2019)
C# 9.0 (2020)
C# 10.0 (2021)
C# 11.0 (2022)
C# 12.0 (2023)

C# 6.0 (2015)

Focus on developer productivity and code readability.

String Interpolation

// Old way
string message = string.Format("Hello {0}, you are {1} years old", name, age);

// New way
string message = $"Hello {name}, you are {age} years old";
string complex = $"Math: {2 + 2}, Method: {CalculateValue()}";

When to use: Any time you need to embed values or expressions within strings Why use it:

More readable than string.Format()
Compile-time checking of interpolated values
IntelliSense support for embedded expressions

Best practices:

Use for simple string formatting
Consider traditional format strings for complex formatting scenarios
Be careful with complex expressions - extract to variables if they become hard to read

Null Propagation (?.)

// Old way
var zipCode = customer != null
    ? customer.Address != null
        ? customer.Address.ZipCode
        : null
    : null;

// New way
var zipCode = customer?.Address?.ZipCode;
var length = customer?.Name?.Length ?? 0;

When to use:

Accessing properties or methods on potentially null objects
Chaining multiple null-checkable operations

Why use it:

Eliminates verbose null-checking code
Prevents null reference exceptions
Makes code more readable

Best practices:

Combine with ?? operator for default values
Don’t overuse - if you find too many null checks, consider redesigning

C# 7.0 (2017)

Introduction of tuples and pattern matching.

Tuples

// Method returning multiple values
public (string name, int age) GetPersonDetails()
{
    return ("John", 30);
}

// With deconstruction
var (name, age) = GetPersonDetails();

// Tuple usage in LINQ
var statistics = orders
    .Select(o => (o.Date, o.Total))
    .GroupBy(x => x.Date.Month)
    .Select(g => (month: g.Key, total: g.Sum(x => x.Total)));

When to use:

Returning multiple values from methods
Temporary grouping of related data
LINQ projections

Why use it:

Cleaner than out parameters
More structured than anonymous types
Better performance than small classes

Best practices:

Name tuple elements for clarity
Use for internal implementation details
Consider proper classes for public APIs

Pattern Matching

// Type patterns with when
switch (shape)
{
    case Circle c when c.Radius > 10:
        return $"Large circle: {c.Radius}";
    case Rectangle r when r.Width == r.Height:
        return "Square";
    case Rectangle r:
        return $"Rectangle: {r.Width}x{r.Height}";
    case null:
        throw new ArgumentNullException(nameof(shape));
    default:
        return "Unknown shape";
}

// Property patterns
if (order is { Status: OrderStatus.Paid, Total: > 1000 })
{
    // Process premium order
}

When to use:

Type checking and casting in one operation
Complex conditional logic
Object property validation

Why use it:

More concise than traditional type checking
Safer than manual casting
More maintainable than nested if statements

C# 8.0 (2019)

Focus on null safety and improved patterns.

Nullable Reference Types

#nullable enable

public class Customer
{
    public string Name { get; set; } = null!; // Must be initialized
    public string? MiddleName { get; set; }   // Can be null

    public string GetFullName(string? title)
    {
        return title is null
            ? Name
            : $"{title} {Name}";
    }
}

When to use:

New projects where null safety is important
Gradually in existing projects
APIs where null semantics matter

Why use it:

Catches null reference bugs at compile time
Makes null handling intentions clear
Improves code documentation

Switch Expressions

public decimal CalculateDiscount(Customer customer) =>
    customer.Type switch
    {
        CustomerType.New => 0.1m,
        CustomerType.Regular when customer.Orders.Count > 100 => 0.2m,
        CustomerType.Regular => 0.15m,
        CustomerType.VIP => 0.3m,
        _ => throw new ArgumentException($"Unknown customer type: {customer.Type}")
    };

When to use:

Converting one type to another based on conditions
Simple pattern matching scenarios
Replacing switch statements with expressions

Why use it:

More concise than switch statements
Forces exhaustive matching
Better type safety

C# 9.0 (2020)

Records

// Immutable record
public record Person(string Name, int Age);

// Record with additional members
public record Employee(string Name, int Age, string Department)
{
    public bool IsManager { get; init; }
    public decimal CalculateBonus() => IsManager ? 5000m : 1000m;
}

// Inheritance
public record Manager(string Name, int Age, string Department)
    : Employee(Name, Age, Department)
{
    public int TeamSize { get; init; }
}

When to use:

Data-centric types
Domain models
DTOs
Immutable objects

Why use it:

Built-in value equality
Immutability by default
Concise syntax for data classes

Best practices:

Use for immutable data models
Consider inheritance hierarchy
Use with pattern matching

C# 10.0 (2021)

Global Using Directives

// In a central file (e.g., GlobalUsings.cs)
global using System.Collections.Generic;
global using System.Linq;
global using System.Text.Json;
global using static System.Math;

// File scoped namespaces
namespace MyApp;

public class Program { }

When to use:

Common imports across many files
Framework-specific imports
Large projects with consistent dependencies

Why use it:

Reduces code repetition
Centralizes dependency management
Cleaner source files

C# 11.0 (2022)

Raw String Literals

var json = """
    {
        "name": "John Doe",
        "age": 30,
        "addresses": [
            {
                "type": "home",
                "street": "123 Main St"
            }
        ]
    }
    """;

var sql = """
    SELECT u.Name, u.Email
    FROM Users u
    WHERE u.Status = 'Active'
        AND u.LastLoginDate >= @date
    """;

When to use:

JSON templates
SQL queries
HTML/XML content
Any multi-line string with special characters

Why use it:

No escape sequences needed
Preserves formatting
More readable

C# 12.0 (2023)

Primary Constructors

public class CustomerService(
    ILogger logger,
    IRepository repository,
    IValidator validator)
{
    public async Task<Customer> CreateCustomer(CustomerDto dto)
    {
        logger.Log("Creating customer");

        if (!validator.Validate(dto))
            throw new ValidationException();

        var customer = new Customer(dto);
        await repository.Save(customer);
        return customer;
    }
}

When to use:

Service classes with dependencies
Classes with simple initialization
When constructor parameters are used throughout the class

Why use it:

Reduces boilerplate
Clear dependency declaration
Improved readability

Collection Expressions

// Array initialization
int[] numbers = [1, 2, 3, 4, 5];

// List creation with spread operator
var existing = new List<int> { 1, 2, 3 };
var combined = [..existing, 4, 5, 6];

// Dictionary initialization
var config = new Dictionary<string, int>
{
    ["MaxRetries"] = 3,
    ["Timeout"] = 1000
};

When to use:

Simple collection initialization
Combining collections
Creating fixed-size arrays

Why use it:

More concise syntax
Clearer intent
Reduced ceremony

Using search_after with Elasticsearch-Rails: Efficient Pagination for Large Datasets

2024-10-12T00:00:00+00:00

When working with large datasets in Elasticsearch, efficient pagination becomes crucial for maintaining performance. The search_after parameter offers a solution by allowing you to paginate through results based on the sorting values of the last document from the previous page. In this post, we’ll explore how to implement search_after pagination using the Elasticsearch-Rails gem.

Why Use `search_after`?

Traditional pagination using from and size parameters can become inefficient for deep pages, as Elasticsearch needs to calculate and skip over a large number of documents. search_after provides a more scalable approach by using the sort values of the last seen document as a search marker.

Implementation Steps

1. Set Up Your Model

First, ensure your model is set up with Elasticsearch:

class Product < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  # Define your index settings and mappings here
end

2. Define a Custom Search Method

Create a class method in your model to handle the search_after logic:

class Product < ApplicationRecord
  # ...

  def self.search_with_after(query, sort_field, search_after = nil, size = 20)
    search_definition = {
      query: query,
      sort: [
        { sort_field => { order: 'asc' } },
        { id: { order: 'asc' } }  # Tie-breaker
      ],
      size: size
    }

    search_definition[:search_after] = search_after if search_after

    __elasticsearch__.search(search_definition)
  end
end

3. Perform the Initial Search

In your controller or service object, perform the initial search:

query = { match: { name: 'example' } }
results = Product.search_with_after(query, :created_at)

@products = results.records
@last_sort_values = results.records.last&.slice(:created_at, :id)&.values

4. Implement Pagination in Your View

In your view, add a “Next Page” link that includes the last sort values:

<% @products.each do |product| %>
  <!-- Display product information -->
<% end %>

<% if @last_sort_values %>
  <%= link_to "Next Page", products_path(search_after: @last_sort_values.to_json) %>
<% end %>

5. Handle Subsequent Searches

In your controller, handle the search_after parameter for subsequent pages:

class ProductsController < ApplicationController
  def index
    query = { match: { name: params[:q] } }
    search_after = JSON.parse(params[:search_after]) if params[:search_after]

    results = Product.search_with_after(query, :created_at, search_after)

    @products = results.records
    @last_sort_values = results.records.last&.slice(:created_at, :id)&.values
  end
end

Considerations and Best Practices

Consistent Sorting: Ensure your sort fields are consistent across searches to maintain proper pagination.
Tie-breakers: Always include a tie-breaker field (like id) in your sort to handle documents with identical sort values.
Statelessness: search_after is stateless, making it suitable for scenarios where users might open multiple browser tabs or share links.
Performance: While more efficient than deep offset pagination, be mindful of very large result sets and consider implementing reasonable limits.

Neovim Telescope

2024-10-11T00:00:00+00:00

If you’re a longtime Vim user like me, you’ve probably used the CtrlP plugin at some point in your workflow. One of my favorite features in CtrlP was being able to hit <C-p> twice and instantly resume the last fuzzy search. This simple trick saved me from having to repeat the same query again and again.

Recently, while transitioning to Neovim, I started using Telescope, which offers a more powerful and extensible fuzzy finder. But one feature I was really missing from CtrlP was the ability to resume the last search without retyping it.

Resuming Your Last Search with Telescope

While this feature is mentioned in the Github Docs under the Vim Pickers section, there is no default keybinding for it, making it easy to overlook.

Telescope’s resume feature allows you to pull up your last search exactly where you left off. This is especially useful when you’ve closed the fuzzy finder, but want to bring it back with all the previous search context.

Here’s how you can map the resume command to a shortcut key:

vim.keymap.set('n', '<leader>tr', '<cmd>Telescope resume<CR>', { noremap = true, silent = true })

Now, with <leader>tr, I can resume my last Telescope search instantly, just like I used to do with CtrlP by hitting <C-p> twice.

Use Cases for `Telescope resume`

There are several moments when resume really shines:

Switching between buffers: Maybe you started a fuzzy search for a file, got distracted by something else, and need to get back to it quickly.
Live grep continuation: If you were performing a project-wide search using live_grep and closed the picker, you can immediately resume and continue refining your search.
Navigating between recently opened files: If you were exploring multiple files and closed the search prematurely, resume will restore that exact file list.

Sending Telescope Results to Quickfix List

Another powerful feature in Telescope that enhances your workflow is the ability to send your current search results to a quickfix list. This is particularly useful when you want to work with multiple results from your search.

By default, you can use <C-q> while in a Telescope picker to send all the current results to the quickfix list. This works for any Telescope picker, including file search, grep results, and more.

Here are some benefits of using this feature:

Persistent Results: The quickfix list persists even after you close the Telescope window, allowing you to refer back to your search results later.
Batch Operations: You can perform operations on multiple files in the quickfix list, such as search and replace across all results.
Navigation: Quickly jump between search results using quickfix navigation commands.

To use this feature:

Open any Telescope picker (e.g., <leader>ff for file search or <leader>fg for live grep)
Perform your search as usual
Press <C-q> to send the current results to the quickfix list
Close the Telescope window (optional)
Use :copen to open the quickfix window and navigate through your results

Using git checkout -

2024-10-10T00:00:00+00:00

As a developer, you often switch between branches or commits while working on multiple tasks, testing out code, or reviewing changes. One handy Git command that can help speed up this process is git checkout -. This command acts like a toggle, letting you jump back and forth between your current branch or commit and the last one you were on. Let’s explore a good use case for this feature and why it can be so useful in your daily workflow.

What is `git checkout -`?

In Git, git checkout - is shorthand for checking out the previously checked-out reference, whether that reference is a branch, commit, or tag. It allows you to quickly toggle between your current location and the one you just left, no matter if it’s a branch or a specific commit.

Example Scenario: Reviewing and Hotfixing

Let’s say you’re working on a feature branch, feature/new-onboarding-flow, and you’re in the middle of implementing some new functionality. Suddenly, you receive an urgent request to fix a bug in production.

Here’s how git checkout - can come in handy:

Step 1: Switch to the `main` Branch for the Hotfix

You need to fix the bug, so you quickly switch from your feature branch to the main branch:

git checkout main

Now you’re on the main branch, where you can apply the necessary hotfix.

Step 2: Make the Hotfix and Push the Changes

You make the changes, commit the fix, and push it to the remote repository:

git commit -m "Fix critical bug in production"
git push origin main

Step 3: Switch Back to Your Feature Branch

After the fix is done, you want to quickly get back to working on your feature. Instead of typing the branch name again or checking the logs for where you were, you can simply run:

git checkout -

This command takes you back to the feature/new-onboarding-flow branch where you left off, allowing you to continue right where you were without extra steps.

Step 4: Toggle Back Again

Need to check something else on the main branch again? Just run git checkout - once more to toggle back, and you’re instantly switched to main.

Handling Uncommitted Changes

While git checkout - is a powerful shortcut, you need to be mindful of uncommitted changes when using it. If you have uncommitted changes, Git will do one of two things:

If there are no conflicts: Git will apply your uncommitted changes when switching between branches or commits. This can be convenient when you want to keep working on something across branches.

If there are conflicts: Git will prevent the switch and display an error message like:

 error: Your local changes to the following files would be overwritten by checkout:
         <file-name>
 Please commit your changes or stash them before you switch branches.

In this case, you’ll need to either commit your changes, discard them, or stash them using git stash.

Using `git stash` to Save Uncommitted Changes

If you want to switch branches without losing your current work, you can use git stash to temporarily save your uncommitted changes, then switch branches, and later reapply the changes.

Here’s how to do it:

git stash           # save your uncommitted changes
git checkout -      # switch back to the previous branch
git stash pop       # reapply your changes

This workflow ensures that you can quickly move between branches or commits without losing your in-progress work.

Toggling Between Branches and Commits

Let’s say before fixing the bug, you briefly switched to an older commit to test some behavior:

git checkout <commit-hash>

After testing the old commit, running git checkout - will take you right back to the branch you were working on. This flexibility to switch between branches and specific commits makes git checkout - particularly useful during testing or debugging sessions.

Why Use `git checkout -`?

1. Speed:

When you’re working on multiple branches or need to switch between a branch and a specific commit, typing git checkout - is much faster than remembering the branch name or commit hash.

2. Convenience:

If you frequently switch between a feature branch and the main or develop branch, or between a commit and your branch, git checkout - saves you from having to repeat the full checkout commands.

3. Simplicity:

For quick, temporary switches (such as testing a fix, reviewing a commit, or updating something on a different branch), you can toggle back and forth without cluttering your workflow or history. You just need to remember to use git stash if you have uncommitted changes that might cause conflicts.

Additional Tips

You can combine git checkout - with other Git commands like git rebase or git merge if you frequently move between branches during development.
This command is context-aware, meaning it works whether you last checked out a branch, commit, or even tag. It simply returns you to where you came from.

Integrating Prometheus for Monitoring a Rails Application

2024-10-04T00:00:00+00:00

Monitoring is a crucial aspect of maintaining the reliability and performance of any application. In this post, we’ll explore how to integrate Prometheus, a free and open-source monitoring system, with a Ruby on Rails application for observability. We’ll cover tracking HTTP requests and exceptions, visualizing metrics, and setting up alerts, all using open-source tools.

What is Prometheus?

Prometheus is a powerful monitoring and alerting toolkit designed for recording real-time metrics in a time-series database. It allows you to scrape metrics from your application and provides a flexible query language, PromQL, for analyzing them.

Setting Up Prometheus with Rails

To integrate Prometheus into your Rails application, you will need to follow these steps:

1. Install the `prometheus-client` Gem

Add the prometheus-client gem to your Gemfile:

gem 'prometheus-client'

Then run:

bundle install

2. Configure Metrics in an Initializer

Create a new initializer (e.g., config/initializers/prometheus_metrics.rb) and set up your metrics:

require "prometheus/client"

prometheus = Prometheus::Client.registry

# Counter for HTTP requests
http_requests = prometheus.counter(:http_requests_total, docstring: "A counter of HTTP requests made.", labels: [:method, :path])

# Counter for exceptions
http_exceptions = prometheus.counter(:http_exceptions_total, docstring: "Total number of exceptions raised.")

# Expose metrics for Rails requests
ActiveSupport::Notifications.subscribe("process_action.action_controller") do |name, start, finish, id, payload|
  http_requests.increment(labels: { method: payload[:method], path: payload[:path] })
end

3. Handle Exceptions with Notifications

In your ApplicationController, use ActiveSupport Notifications to capture exceptions:

# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
  # Emit a notification for unhandled exceptions
  rescue_from StandardError do |exception|
    # Notify about the exception
    ActiveSupport::Notifications.instrument("exception.action_controller", exception: exception)

    # Optionally log the exception
    Rails.logger.error(exception.message)

    # Render a generic error message
    render plain: 'Internal Server Error', status: :internal_server_error
  end
end

And subscribe to this notification in your initializer:

# Subscribe to the exception notification
ActiveSupport::Notifications.subscribe("exception.action_controller") do |name, start, finish, id, payload|
  http_exceptions.increment
end

4. Expose a Metrics Endpoint

To collect and expose your application’s metrics efficiently, you can use Prometheus middleware in your Rack configuration. This middleware automatically handles metrics collection and exposure, making it easy to get started.

# config.ru
require 'rack'
require 'prometheus/middleware/collector'
require 'prometheus/middleware/exporter'

use Rack::Deflater
use Prometheus::Middleware::Collector
use Prometheus::Middleware::Exporter

run Rails.application

With this setup, the middleware will automatically expose your Prometheus metrics at the /metrics endpoint, allowing Prometheus to scrape the data without the need for additional controller actions.

Benefits of Using Standardized Metrics

Automatic Collection: The middleware collects standardized metrics such as request counts, response times, and error rates without any manual coding required. This minimizes the chances of errors in metric implementation and ensures consistency across different applications.
Simplicity: By leveraging standardized metrics, you can avoid the overhead of managing custom metrics unless absolutely necessary, allowing you to focus on your application’s core functionality.

When to Use Custom Metrics

While standardized metrics cover many common use cases, there may be instances where custom metrics are essential to capture specific aspects of your application’s performance or business logic.

Tailored Insights: Custom metrics allow you to track unique user interactions or application performance indicators that standardized metrics might not address.
Greater Flexibility: Implementing custom metrics provides you the flexibility to adapt your monitoring strategy as your application evolves. However, keep in mind that this approach may require additional maintenance and careful planning to ensure that the metrics remain relevant and accurately reflect your application’s behavior.

In summary, using the middleware to expose a /metrics endpoint provides a quick and efficient way to leverage the power of Prometheus in your Rails application, whether through standardized metrics or tailored custom ones.

5. Start Prometheus

Make sure you have Prometheus installed and running. Use a configuration file (e.g., prometheus.yml) to define your scrape configuration:

scrape_configs:
  - job_name: 'rails_app'
    static_configs:
      - targets: ['localhost:3000']  # Adjust the port if needed

6. Access the Prometheus Web Interface

Prometheus provides a built-in web interface to view metrics. You can access it at http://localhost:9090. Use the Graph tab to query metrics using PromQL.

7. Setting Up Grafana (Optional)

While Prometheus has a web interface, many users prefer to visualize metrics using Grafana. You can find installation instructions on the Grafana website.

Start Grafana on a different port (e.g., 3001 set in the grafana.ini or defaults.ini if you run rails on 3000):

grafana server # you will have to login with default user and pass admin/admin

Add Prometheus as a data source in Grafana by navigating to Configuration > Data Sources and selecting Prometheus.

8. Alerts with Prometheus

To set up alerts, define alerting rules in your Prometheus configuration. For instance, you can create an alert for exceptions:

groups:
  - name: alerting_rules
    rules:
      - alert: HighHttpExceptions
        expr: http_exceptions_total > 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High number of HTTP exceptions"
          description: "More than 10 exceptions in the last 5 minutes."

Viewing Metrics Without Grafana

You can view Prometheus metrics directly without Grafana. Use the Prometheus web interface to run queries and visualize metrics as graphs. Simply navigate to http://localhost:9090 and explore the metrics.

Why Does AI Like to 'Delve' So Much?

2024-10-03T00:00:00+00:00

If you’ve interacted with an AI language model, you’ve probably noticed that certain words or phrases tend to come up again and again. One word that often appears is “delve” — as in “let’s delve into this topic.” But why does AI “like” to use certain words so much? The answer lies in how these models generate text and, more specifically, in the processes known as greedy decoding and stochastic sampling.

The Basics of AI Text Generation

AI language models, like GPT, generate text by predicting the next word in a sequence based on previous words. The goal is to model the probability of the next word given the context. Formally, this looks something like this:

P(x_t | x_1, x_2, ..., x_t-1)

In plain terms, the model predicts the probability of the next word (x_t) based on all the words that came before it (x_1, x_2, ..., x_t-1). To actually generate text, we need a strategy for choosing the next word. This is where greedy decoding and stochastic sampling come into play.

Greedy Decoding

In greedy decoding, the model always picks the word with the highest probability at each step. This results in a very focused and deterministic sequence of text. While this can be useful for ensuring coherence, it tends to make the output repetitive or overly cautious — which explains why certain phrases like “delve” might get overused.

Here’s an Elixir example of greedy decoding:

defmodule GreedyDecoder do
  def predict_next_word(tokens, model) do
    tokens
    |> Enum.map(&model.(&1))
    |> Enum.max_by(fn {_, probability} -> probability end)
    |> elem(0)
  end
end

# Example usage
vocab = ["explore", "delve", "analyze", "investigate"]
model = fn token -> Enum.zip(vocab, [0.6, 0.3, 0.05, 0.05]) end
tokens = ["The", "researcher", "decided", "to"]

GreedyDecoder.predict_next_word(tokens, model)
# Output: "explore"

In this case, the word with the highest probability (“explore”) is always selected. If “delve” has a consistently high probability in similar contexts, you’ll see it again and again.

Stochastic Sampling

Stochastic sampling, on the other hand, introduces some randomness. Instead of always picking the word with the highest probability, the model samples from the probability distribution — meaning that words with lower probabilities still have a chance to be chosen. This method encourages more variety and creativity in the generated text.

Here’s how you could implement stochastic sampling in Elixir:

defmodule StochasticDecoder do
  def predict_next_word(tokens, model) do
    tokens
    |> Enum.map(&model.(&1))
    |> Enum.reduce([], fn {word, prob}, acc -> acc ++ List.duplicate(word, round(prob * 100)) end)
    |> Enum.shuffle()
    |> List.first()
  end
end

# Example usage
vocab = ["explore", "delve", "analyze", "investigate"]
model = fn token -> Enum.zip(vocab, [0.6, 0.3, 0.05, 0.05]) end
tokens = ["The", "researcher", "decided", "to"]

StochasticDecoder.predict_next_word(tokens, model)
# Possible outputs: "delve", "explore", "investigate", or "analyze" (depending on the random sampling)

By using stochastic sampling, there’s a chance the AI will pick words other than “explore”, even though it has the highest probability. This can lead to more diverse and creative outputs.

How to Adjust the “Temperature” for More Creative Output

If you want the AI to generate more creative and varied responses, one of the most important parameters to adjust is the temperature.

Lower temperatures (closer to 0) make the model more deterministic, meaning it will stick to the most probable word choices.
Higher temperatures (closer to 1) introduce more randomness, encouraging the model to pick less common words and be more inventive.

How to Change the Temperature

In practice, changing the temperature depends on the tool or platform you’re using:

OpenAI API: You can set the temperature parameter when making API calls. For example, in OpenAI’s API:
```
{
  "model": "gpt-4",
  "prompt": "The researcher decided to",
  "temperature": 0.8
}
```
Playground: If you’re using OpenAI’s Playground, you can adjust the temperature with a slider before generating text.
Other apps and services: Many tools and platforms that use AI allow you to adjust the creativity or style of the generated text. These usually map to the temperature setting internally.

Here’s an example showing how to adjust the temperature in Elixir:

defmodule TemperatureDecoder do
  def predict_next_word(tokens, model, temperature \\ 1.0) do
    tokens
    |> Enum.map(&model.(&1))
    |> Enum.map(fn {word, prob} -> {word, :math.pow(prob, 1.0 / temperature)} end)
    |> Enum.reduce([], fn {word, prob}, acc -> acc ++ List.duplicate(word, round(prob * 100)) end)
    |> Enum.shuffle()
    |> List.first()
  end
end

# Example usage with temperature
vocab = ["explore", "delve", "analyze", "investigate"]
model = fn token -> Enum.zip(vocab, [0.6, 0.3, 0.05, 0.05]) end
tokens = ["The", "researcher", "decided", "to"]

TemperatureDecoder.predict_next_word(tokens, model, 0.7)
# Output will vary based on the adjusted temperature

Lowering the temperature parameter leads to more predictable outcomes, while raising it adds creativity and variety to the AI’s word choices.

Conclusion

AI language models generate text by predicting the next word based on prior context. Techniques like greedy decoding result in more deterministic, focused text, while stochastic sampling can add variety and creativity. If you’re looking to guide the AI towards more varied, creative outputs, adjusting the temperature is key. Lower temperatures make responses more predictable, while higher temperatures foster creativity by allowing more diverse word choices.

So, why does AI like to “delve” so much? It’s not that the model has a preference, but rather that certain words come up more often in certain contexts due to how probabilities are calculated. By understanding and tweaking decoding strategies, you can have more control over the AI’s output — and maybe even get it to “explore” a little more often than “delve.”

Understanding NTP and Fixing Time Sync Issues

2024-09-29T00:00:00+00:00

In any time-sensitive application, such as those relying on SAML authentication, ensuring your server’s time is accurate is critical. A common culprit for time-related issues is clock drift, which happens when your server’s time gradually becomes inaccurate. This blog post will cover how to detect and fix clock drift using NTP (Network Time Protocol), ensuring your system is always in sync with global time standards.

What is Clock Drift?

Clock drift refers to the gradual deviation of a system’s clock from the actual, accurate time. It happens because every system has an internal clock that measures time, and these clocks aren’t always perfectly accurate. The drift may be small, but over time it can accumulate, causing significant discrepancies between your system’s time and the real time.

For example, a system clock might drift a few seconds per day. Over a few weeks, this could turn into minutes or even hours of difference, depending on the system. This can lead to major problems in time-sensitive operations like:

Authentication (e.g., SAML, OAuth)
Cache expiration
Database transactions with time-based queries
Time-sensitive event scheduling

What is NTP?

NTP (Network Time Protocol) is a protocol used to synchronize your system’s clock with a highly accurate time source, like an atomic clock or a GPS clock. NTP works by periodically querying remote time servers, which provide the correct time, and adjusting the system’s clock as necessary to stay in sync.

How NTP Sync Fixes Clock Drift

NTP fixes clock drift by ensuring that your system regularly checks a reliable time source and corrects its internal clock. Once NTP is installed and configured, your system will constantly adjust its time to stay as close as possible to the global time standard.

Fixing NTP Sync on a System

Here’s how you can check and fix NTP synchronization issues on a typical Linux server running an application.

Step 1: Check Current Time Sync Status

Before making changes, you can check if your system’s time is synchronized by using the following command:

timedatectl status

This command will tell you if your system clock is synchronized or if there are issues.

Here’s an example of what you might see:

               Local time: Fri 2024-09-30 14:15:02 UTC
           Universal time: Fri 2024-09-30 14:15:02 UTC
                 RTC time: Fri 2024-09-30 14:15:02
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

In this case, “System clock synchronized: no” indicates that the system is not currently synchronized, which could lead to clock drift.

Step 2: Install NTP or Chrony

On most Linux distributions, you can use NTP or Chrony to sync your system’s clock. Chrony is often preferred for modern systems because it’s faster and more efficient, but NTP is still widely used.

Installing NTP:

On Ubuntu or Debian systems, install NTP with:

sudo apt update
sudo apt install ntp

For Red Hat or CentOS systems:

sudo yum install ntp

Installing Chrony:

If you prefer Chrony instead:

sudo apt install chrony   # Ubuntu/Debian
sudo yum install chrony   # CentOS/RedHat

Step 3: Enable and Start the NTP Service

Once NTP (or Chrony) is installed, you need to enable and start the service to begin syncing the time.

For NTP:

sudo systemctl enable ntp
sudo systemctl start ntp

For Chrony:

sudo systemctl enable chronyd
sudo systemctl start chronyd

Step 4: Verify Time Synchronization

After starting the service, you can check if the synchronization is working with the following command:

timedatectl status

You should now see something like:

               Local time: Fri 2024-09-30 14:20:00 UTC
           Universal time: Fri 2024-09-30 14:20:00 UTC
                 RTC time: Fri 2024-09-30 14:20:00
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

The key field here is “System clock synchronized: yes”, which indicates that your server’s time is now properly synchronized.

Step 5: Test NTP Sync

To check how your system is performing in terms of time synchronization, you can use the ntpq command to get details about the time servers your system is syncing with:

ntpq -p

This will give you a table showing the servers being used for synchronization, along with the delay and offset:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*time.nist.gov   .NIST.           1 u   23   64  377   52.123   -0.987   1.145
+clock.isc.org   .GPS.            1 u   30   64  377   30.245    0.245   0.987

In this table, you want to see low offsets and jitter values, indicating that your system’s clock is close to being perfectly in sync with the external time source.

What Happens if NTP is Not Set Up?

Without NTP, your system will slowly experience clock drift, and over time, the system clock will become inaccurate. This can cause issues in:

Authentication protocols like SAML, where tokens are only valid for short time windows.
Scheduled tasks that rely on accurate timing.
Logging systems, where out-of-sync times can make debugging difficult.
Database records, where time-sensitive queries (updated_at, created_at) can lead to inconsistencies.

In extreme cases, significant time discrepancies can cause downtime, particularly for systems dependent on strict time validation.

How Does Clock Drift Occur?

Clock drift happens because no hardware clock is perfect. The reasons include:

Hardware limitations: System clocks are built with internal quartz crystals that vibrate to keep time. However, due to imperfections in the crystals, temperature changes, or electrical noise, the vibrations may vary slightly, leading to time drift.
System load: If your server is under heavy load, its internal clock may lose time accuracy as the CPU focuses on other tasks.
Power fluctuations: Inconsistent power delivery to a system can cause its clock to drift.

Without regular synchronization via NTP, these issues accumulate over time, making the system clock increasingly inaccurate.

Conclusion

Ensuring that your system stays in sync with global time standards using NTP is critical for time-sensitive operations. Whether you’re dealing with authentication systems, database queries, or scheduled tasks, clock drift can cause serious issues if left unchecked. With NTP or Chrony properly configured, your system can automatically correct its clock drift, keeping your applications running smoothly and securely.

Point-free programming

2024-09-25T00:00:00+00:00

Point-free style, also known as tacit programming, is a method of writing functions without explicitly mentioning their arguments. Instead of focusing on what the arguments are, point-free style emphasizes how functions are combined. This style is prominent in functional programming and can be found in many functional languages, such as Haskell and Elixir.

In this blog post, we’ll explore the origins of point-free style, how it’s implemented in Haskell and Elixir, and how these two languages compare when it comes to this technique.

The Origins of Point-Free Style

The roots of point-free style go back to combinatory logic, a branch of mathematical logic developed by Moses Schönfinkel in the 1920s and later extended by Haskell Curry. Combinatory logic aimed to eliminate the need for variables in functions by focusing solely on function composition. This allowed expressions to be more concise and abstract.

The goal of combinatory logic was to describe computations purely through function application and composition without referring to the individual arguments, hence “point-free” (where “points” refer to arguments).

While combinatory logic originated in mathematics, its ideas have strongly influenced functional programming languages. APL, developed in the 1960s, was one of the first languages to use a similar concept, and Miranda, developed in the 1980s, popularized it further. Haskell, born in the 1990s, adopted and refined this approach, making point-free style a hallmark of functional programming.

Point-Free Style in Haskell

Haskell is known for its elegant use of function composition and point-free style. In Haskell, you can define functions by composing other functions, without mentioning the arguments.

Here’s a simple example of a point-free function in Haskell:

makeGreeting = (<>) . (<> " ")

In this case:

The function (<>) is the string concatenation operator.
The expression (<>) . (<> " ") creates a new function that concatenates a space between two strings.

When you call makeGreeting "hello" "world", it returns "hello world". The magic of point-free style here is that you never explicitly mention the arguments name1 or name2 — everything is done through function composition.

Benefits in Haskell

Conciseness: Point-free style reduces the verbosity by removing explicit references to arguments.
Readability (in some cases): For those accustomed to functional programming, this style can make code more expressive and elegant by highlighting the transformations applied to the data rather than focusing on the data itself.

However, point-free style can become unreadable if overused, especially when the composition involves many functions. This is where balance is key.

Point-Free Style in Elixir

Elixir, while also a functional programming language, has a slightly different syntax and approach to anonymous functions compared to Haskell. Elixir doesn’t support point-free style as directly as Haskell, but it still allows for a concise definition of anonymous functions using the capture operator (&).

Here’s the equivalent of the Haskell point-free example in Elixir:

make_greeting = &(&1 <> " " <> &2)

In this Elixir version:

The & symbol is a shorthand for anonymous functions.
&1 and &2 refer to the first and second arguments, respectively.

This function behaves the same as its Haskell counterpart: calling make_greeting.("hello", "world") will return "hello world".

Differences in Elixir

Unlike Haskell, Elixir does not emphasize point-free style as much, mainly due to its roots in the Erlang VM (which was not designed with pure functional programming in mind). However, Elixir provides the & capture operator for defining short, anonymous functions in a concise way, which offers some point-free-like benefits.

Example: Haskell vs Elixir

Let’s compare a slightly more complex example in both languages: a function that doubles each element in a list.

Haskell:
```
doubleAll = map (* 2)
```
In Haskell, doubleAll is a point-free function that uses map to apply (* 2) to each element in a list. The function is defined entirely without mentioning the list argument.
Elixir:
```
double_all = &Enum.map(&1, &(&1 * 2))
```
In Elixir, while you can write it concisely using the capture operator (&), it’s not as purely point-free as Haskell. Here, you still need to refer to &1 to indicate the argument, and you call the Enum.map/2 function.

Pros and Cons of Point-Free Style

Advantages

Conciseness: Point-free style leads to more compact code, which can be beneficial when dealing with simple function compositions.
Abstraction: By focusing on composition rather than individual arguments, the code often becomes more abstract and modular.
Readability (for small compositions): When used correctly, point-free style can make code more expressive by focusing on the logic of function composition.

Disadvantages

Readability (for complex compositions): As compositions grow in complexity, point-free style can become difficult to follow, making code harder to understand and maintain.
Debugging: When errors occur, it can be harder to track down the source, as the explicit argument handling is missing.

Conclusion

Point-free style has its origins in combinatory logic and has found a home in functional programming, particularly in languages like Haskell. Haskell makes point-free style a natural part of its programming model, allowing for elegant function compositions without mentioning arguments.

In contrast, while Elixir supports concise anonymous functions using the capture operator (&), it doesn’t fully embrace point-free style in the same way as Haskell. However, both languages allow for powerful functional programming techniques, and knowing when to use or avoid point-free style can lead to clearer, more maintainable code.

The key takeaway is that point-free style is a useful tool, but like any tool, it should be used judiciously, especially as the complexity of your code grows.

References

Haskell Documentation: Haskell Function Composition
Elixir Documentation: Elixir Anonymous Functions

I hope this gives you a good overview of point-free style and how it compares between Haskell and Elixir.

Enhancing Your Codebase with Commitizen

2024-09-16T00:00:00+00:00

One often overlooked aspect of a code base is the commit history. Enter Commitizen, a powerful tool that standardizes commit messages and automates changelog generation. In this post, we’ll explore how Commitizen can improve your development workflow and discuss some common commit guidelines.

What is Commitizen?

Commitizen is a command-line utility that helps developers write standardized commit messages. It provides an interactive prompt that guides you through the commit process, ensuring that your commits follow a consistent format.

Installing and Configuring Commitizen

Installation

You can install Commitizen globally using either npm or yarn:

Using npm:

npm install -g commitizen

Using yarn:

yarn global add commitizen

Configuration

To customize Commitizen for your project, you can create a .czrc file in your project root or home directory. Here’s an example of a .czrc file:

{
  "path": "cz-conventional-changelog",
  "types": {
    "feat": {
      "description": "A new feature"
    },
    "fix": {
      "description": "A bug fix"
    },
    "docs": {
      "description": "Documentation only changes"
    },
    "style": {
      "description": "Changes that do not affect the meaning of the code"
    },
    "refactor": {
      "description": "A code change that neither fixes a bug nor adds a feature"
    },
    "perf": {
      "description": "A code change that improves performance"
    },
    "test": {
      "description": "Adding missing tests"
    },
    "chore": {
      "description": "Changes to the build process or auxiliary tools"
    },
    "a11y": {
      "description": "Adding accessibility features"
    },
    "sec": {
      "description": "A code change related to security"
    },
    "devops": {
      "description": "A code change related to devops"
    }
  },
  "messages": {
    "type": "Select the type of change you're committing:",
    "subject": "Write a short, imperative tense description of the change:\n",
    "body": "Provide a longer description of the change (optional). Use Markdown for formatting:\n",
    "breaking": "List any breaking changes:\n",
    "footer": "List any issues closed by this change (optional):\n"
  },
  "subjectLimit": 100,
  "upperCaseSubject": false,
  "allowCustomScopes": false,
  "allowBreakingChanges": ["feat", "fix"],
  "footerPrefix": "ISSUES CLOSED:"
}

This configuration file allows you to customize the commit types, prompts, and other aspects of Commitizen’s behavior. You can adjust these settings to match your project’s specific needs.

For more detailed information on configuration options, you can refer to the Commitizen GitHub documentation.

Benefits of Using Commitizen

Consistency: Enforces a uniform commit message format across your team.
Clarity: Makes it easier to understand the purpose of each commit at a glance.
Automation: Facilitates automatic generation of changelogs.
Better Code Reviews: Standardized commits make it easier to review code changes.

Common Commit Types

Let’s break down some commonly used commit types and their purposes:

feat: A new feature
fix: A bug fix
docs: Documentation only changes
style: Changes that do not affect the meaning of the code (e.g., formatting)
refactor: A code change that neither fixes a bug nor adds a feature
perf: A code change that improves performance
test: Adding missing tests
chore: Changes to the build process or auxiliary tools
a11y: Adding accessibility features
sec: A code change related to security
devops: A code change related to DevOps

Anatomy of a Good Commit Message

A well-structured commit message typically includes:

Type: The category of the change (e.g., feat, fix, docs)
Subject: A short, imperative description of the change
Body (optional): A longer description of the change, using Markdown for formatting
Breaking Changes (if any): List of any breaking changes introduced
Footer (optional): References to issues closed by the change

Using Commitizen: A Walkthrough

Let’s walk through the process of creating a commit using Commitizen. After you’ve made your changes and staged them with git add, instead of using git commit, you’ll use git cz. Here’s what the process looks like:

In your terminal, type:
```
git cz
```

Commitizen will launch and guide you through the commit process:

? Select the type of change you're committing: (Use arrow keys)
❯ feat:     A new feature
  fix:      A bug fix
  docs:     Documentation only changes
  style:    Changes that do not affect the meaning of the code
  refactor: A code change that neither fixes a bug nor adds a feature
  perf:     A code change that improves performance
  test:     Adding missing tests
(Move up and down to reveal more choices)

Use the arrow keys to select the appropriate type and press Enter.

Next, you’ll be prompted to enter the scope of the change (optional):
```
? What is the scope of this change (e.g. component or file name): (press enter to skip)
auth
```
Enter the scope or press Enter to skip.

Now, enter a short description of the change:

? Write a short, imperative tense description of the change:
implement JWT-based user authentication

Provide a longer description if needed:

? Provide a longer description of the change: (press enter to skip)
Implement JWT-based authentication for user login and registration.
This includes:
- Creating login and registration endpoints
- Implementing password hashing
- Setting up JWT token generation and validation

Indicate if there are any breaking changes:

? Are there any breaking changes? (y/N)
y

If you answer yes, you’ll be prompted to describe the breaking changes:

? Describe the breaking changes:
API now requires authentication for most endpoints

Finally, indicate if this change affects any open issues:

? Does this change affect any open issues?
y

If yes, you’ll be prompted to add issue references:

? Add issue references (e.g. "fix #123", "re #123".):
Closes #123, #124

Commitizen will then generate the commit message based on your inputs and create the commit:

[master 5c25de1] feat(auth): implement JWT-based user authentication

Implement JWT-based authentication for user login and registration.
This includes:
- Creating login and registration endpoints
- Implementing password hashing
- Setting up JWT token generation and validation

BREAKING CHANGE: API now requires authentication for most endpoints

Closes #123, #124

This process ensures that your commit message is structured consistently and contains all the necessary information. The resulting commit message will look like this:

feat(auth): implement JWT-based user authentication

Implement JWT-based authentication for user login and registration.
This includes:
- Creating login and registration endpoints
- Implementing password hashing
- Setting up JWT token generation and validation

BREAKING CHANGE: API now requires authentication for most endpoints

Closes #123, #124

By following this process, you create commits that are not only informative but also adhere to a consistent format, making it easier for your team to understand changes and automatically generate meaningful changelogs.

Sample Problematic Commit

While Commitizen won’t automatically reject commits, it guides users towards more descriptive messages. Here’s an example of a commit that Commitizen would try to improve:

type: updated stuff

Select the type of change you're committing:
> feat
Write a short, imperative tense description of the change:
> updated stuff
Provide a longer description of the change (optional). Use Markdown for formatting:
>
List any breaking changes:
>
List any issues closed by this change (optional):
>

In this scenario, Commitizen would create a commit message like:

feat: updated stuff

While this commit would be accepted, it still lacks specificity and doesn’t provide useful information about the changes made. Commitizen’s prompts encourage developers to provide more details, but it ultimately depends on the user to input meaningful information.

Generating and Understanding Changelogs

While Commitizen helps in creating structured commit messages, generating changelogs requires additional tools that work well with Commitizen’s commit format. Let’s explore how to generate a changelog and what the result looks like.

Generating a Changelog

Two popular options for generating changelogs are standard-version and conventional-changelog-cli.

Using standard-version

Install standard-version:
```
npm install -g standard-version
```
Generate a changelog:
```
standard-version
```
This command analyzes your commits, updates your version in package.json, generates a changelog, creates a new commit with these changes, and creates a new tag with the new version number.
To generate the changelog without bumping the version:
```
standard-version --skip.bump --skip.tag
```

Using conventional-changelog-cli

Install conventional-changelog-cli:

npm install -g conventional-changelog-cli

Generate the changelog:

conventional-changelog -p angular -i CHANGELOG.md -s

To generate the entire changelog from scratch:

conventional-changelog -p angular -i CHANGELOG.md -s -r 0

Automating Changelog Generation

Add this script to your package.json to automate changelog generation:

{
  "scripts": {
    "version": "conventional-changelog -p angular -i CHANGELOG.md -s && git add CHANGELOG.md"
  }
}

Now, running npm version will automatically update your changelog.

Example Changelog

Here’s an example of what a generated changelog might look like when following Commitizen conventions:

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.2.0] - 2024-09-20

### Added
- feat: implement user authentication system (#123)
  - Added JWT-based authentication for login and registration
  - Created new endpoints for user management
- feat: add dark mode to user interface (#145)
- feat(api): introduce rate limiting for public endpoints (#150)

### Changed
- refactor: optimize database queries for better performance (#135)
- style: update color scheme across the application (#140)

### Fixed
- fix: resolve issue with password reset functionality (#128)
- fix(ui): correct alignment of buttons in mobile view (#142)

### Security
- sec: update dependencies to address known vulnerabilities (#155)

## [1.1.0] - 2024-08-15

### Added
- feat: introduce search functionality for products (#110)
- feat(api): add pagination to list endpoints (#115)

### Changed
- refactor: restructure folder organization for better modularity (#105)
- perf: optimize image loading for faster page renders (#120)

### Fixed
- fix: resolve cart calculation errors (#112)
- fix(ui): correct responsive layout issues on tablet devices (#118)

### Documentation
- docs: update API documentation with new endpoints (#125)

## [1.0.1] - 2024-07-30

### Fixed
- fix: resolve critical bug in payment processing (#102)
- fix(ui): correct typos in error messages (#104)

### Security
- sec: implement additional checks for user input sanitization (#103)

## [1.0.0] - 2024-07-15

### Added
- Initial release of the application
- feat: core e-commerce functionality including product listing, cart, and checkout
- feat: user registration and profile management
- feat: basic reporting and analytics for administrators

### Documentation
- docs: create initial README and contribution guidelines (#95)
- docs: add inline code comments for complex functions (#97)

### DevOps
- devops: set up CI/CD pipeline for automated testing and deployment (#90)

This changelog demonstrates how structured commit messages translate into a clear, informative record of project changes. It’s organized by version, with changes categorized based on their type. Each entry corresponds directly to a commit, making it easy to trace changes back to their source.

The effectiveness of these changelog generation tools depends on the quality and consistency of your commit messages. This is where Commitizen really shines, ensuring that all your commits follow a format that these changelog generators can understand and process.

Conclusion

By adopting Commitizen and following consistent commit guidelines, you can significantly improve your codebase’s maintainability and documentation. While Commitizen provides a framework for better commits, it’s up to the development team to ensure that the content of the commits is meaningful and descriptive.

Remember, good commit messages are a form of documentation. They not only help your team understand changes but also make it easier for future developers (including yourself) to understand the project’s history and evolution. The more specific and descriptive your commit messages are, the more valuable they become in the long run, especially when it comes to generating comprehensive and useful changelogs.

Implementing Stripe Payments in Phoenix 1.7 with Stripity Stripe

2024-09-11T00:00:00+00:00

This guide outlines the process of integrating Stripe payments into a Phoenix 1.7 application using the Stripity Stripe library. It covers the key steps in implementation and highlights the differences in Phoenix 1.7’s architecture that may affect the integration process.

Advantages of Stripity Stripe

Stripity Stripe offers several benefits for handling payments in Elixir:

Elixir-native implementation: Designed specifically for use with Elixir.
Comprehensive coverage: Supports most features of the Stripe API.
Regular maintenance: Kept up-to-date with Stripe’s API changes.
Type safety: Utilizes Elixir’s type system to reduce runtime errors.

Compared to manual API integration, Stripity Stripe simplifies API calls, webhook handling, and error management.

Architectural Changes in Phoenix 1.7

Phoenix 1.7 introduces several changes to the framework’s structure:

Function Components: Replaces traditional templates with function components in a dedicated HTML module.
Embedded Templates: Templates are now embedded directly in the HTML module using embed_templates "*.html".
Updated Path Helpers: The ~p sigil replaces the previous Routes.x_path syntax.

These changes affect how views and templates are structured and how routes are referenced within the application.

Implementation Steps

1. Project Setup

Add Stripity Stripe to your mix.exs:

{:stripity_stripe, "~> 2.0"}

Run mix deps.get to install the dependency.

2. Stripe Configuration

In config/config.exs, add:

config :stripity_stripe, api_key: System.get_env("STRIPE_SECRET_KEY")

3. Create and Run Migration

Before setting up the controller, we need to create a database table to store payment information. Let’s create a migration:

mix ecto.gen.migration create_payments

This will create a new migration file in the priv/repo/migrations directory. Open the newly created file and add the following content:

defmodule YourApp.Repo.Migrations.CreatePayments do
  use Ecto.Migration

  def change do
    create table(:payments) do
      add :amount, :integer
      add :stripe_id, :string
      add :status, :string

      timestamps()
    end

    create index(:payments, [:stripe_id])
  end
end

This migration creates a payments table with fields for the amount, Stripe ID, and status of the payment. The timestamps() function adds inserted_at and updated_at fields.

Now, run the migration:

mix ecto.migrate

4. Create Payment Schema

After creating the database table, we need to define a schema for it. Create a new file lib/your_app/payments/payment.ex:

defmodule YourApp.Payments.Payment do
  use Ecto.Schema
  import Ecto.Changeset

  schema "payments" do
    field :amount, :integer
    field :stripe_id, :string
    field :status, :string

    timestamps()
  end

  def changeset(payment, attrs) do
    payment
    |> cast(attrs, [:amount, :stripe_id, :status])
    |> validate_required([:amount, :stripe_id, :status])
  end
end

This schema corresponds to the database table we just created and provides a changeset function for validating and casting payment data.

5. Payment Controller

Create lib/your_app_web/controllers/payment_controller.ex:

defmodule YourAppWeb.PaymentController do
  use YourAppWeb, :controller
  alias YourApp.Payments.Payment

  def new(conn, _params) do
    changeset = Payment.changeset(%Payment{}, %{})
    stripe_publishable_key = Application.get_env(:liftforge, :stripe_publishable_key)
    render(conn, :new, changeset: changeset, stripe_publishable_key: stripe_publishable_key)
  end

  def create(conn, %{"payment" => payment_params}) do
    amount = payment_params["amount"]
    token = payment_params["token"]
    case Stripe.Charge.create(%{
      amount: amount,
      currency: "usd",
      source: token,
      description: "Example charge"
    }) do
      {:ok, charge} ->
        {:ok, _payment} =
          %Payment{}
          |> Payment.changeset(%{amount: amount, stripe_id: charge.id})
          |> YourApp.Repo.insert()
        conn
        |> put_flash(:info, "Payment successful.")
        |> redirect(to: ~p"/payment/thank-you?amount=#{amount}")
      {:error, error} ->
        conn
        |> put_flash(:error, "Payment failed: #{error.message}")
        |> render(:new, changeset: Payment.changeset(%Payment{}, payment_params))
    end
  end

  def thank_you(conn, %{"amount" => amount}) do
    render(conn, :thank_you, amount: amount)
  end
end

6. HTML Components

Create lib/your_app_web/controllers/payment_html.ex:

defmodule YourAppWeb.PaymentHTML do
  use YourAppWeb, :html
  embed_templates "payment_html/*"

  attr :changeset, Ecto.Changeset, required: true
  attr :action, :string, required: true
  def payment_form(assigns) do
    ~H"""
    <div class="max-w-md mx-auto mt-8 p-6 bg-white rounded-lg shadow-md">
      <.form :let={f} for={@changeset} action={@action} class="space-y-6">
        <div>
          <.input field={f[:amount]} type="number" label="Amount (in cents)" class="mt-1 block w-full" />
        </div>
        <div>
          <label for="card-element" class="block text-sm font-medium text-gray-700">Credit or debit card</label>
          <div id="card-element" class="mt-1 block w-full">
            <!-- Stripe Elements will insert the card input here -->
          </div>
          <div id="card-errors" role="alert" class="mt-2 text-sm text-red-600"></div>
        </div>
        <input type="hidden" name={input_name(f, :token)} id="stripe_token" value={input_value(f, :token)} />
        <div>
          <.button type="submit" class="w-full">Pay Now</.button>
        </div>
      </.form>
    </div>
    """
  end

  attr :amount, :string, required: true
  def thank_you(assigns)
end

7. Templates

Create lib/your_app_web/controllers/payment_html/new.html.heex:

<h1 class="text-3xl font-bold text-center mt-8 mb-6">New Payment</h1>
<.payment_form changeset={@changeset} action={~p"/payment"} />

<script src="https://js.stripe.com/v3/"></script>
<script>
  var stripe = Stripe('<%= @stripe_publishable_key %>');
  var elements = stripe.elements();
  var style = {
    base: {
      fontSize: '16px',
      color: '#32325d',
      '::placeholder': {
        color: '#aab7c4'
      },
    },
    invalid: {
      color: '#fa755a',
      iconColor: '#fa755a'
    }
  };
  var card = elements.create('card', {style: style});
  card.mount('#card-element');

  card.addEventListener('change', function(event) {
    var displayError = document.getElementById('card-errors');
    if (event.error) {
      displayError.textContent = event.error.message;
    } else {
      displayError.textContent = '';
    }
  });

  var form = document.querySelector('form');
  form.addEventListener('submit', function(event) {
    event.preventDefault();

    stripe.createToken(card).then(function(result) {
      if (result.error) {
        var errorElement = document.getElementById('card-errors');
        errorElement.textContent = result.error.message;
      } else {
        var tokenInput = document.getElementById('stripe_token');
        tokenInput.value = result.token.id;
        form.submit();
      }
    });
  });
</script>

Create lib/your_app_web/controllers/payment_html/thank_you.html.heex:

<div class="max-w-md mx-auto mt-8 p-6 bg-white rounded-lg shadow-md text-center">
  <h1 class="text-3xl font-bold mb-4">Thank You!</h1>
  <p class="text-xl mb-4">Your payment of $<%= String.to_float(@amount) / 100 %> was successful.</p>
  <p class="mb-6">We appreciate your business and hope you enjoy your purchase.</p>
  <.link href={~p"/"} class="inline-block bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded">
    Return to Home
  </.link>
</div>

8. Wire up the router

In lib/your_app_web/router.ex:

scope "/", YourAppWeb do
  pipe_through :browser

  # existing routes

  get "/payment/new", PaymentController, :new
  post "/payment", PaymentController, :create
  get "/payment/thank-you", PaymentController, :thank_you

end

Stripity Stripe Integration

The create action in the PaymentController demonstrates the simplicity of using Stripity Stripe. The Stripe.Charge.create/1 function encapsulates the complexities of creating a charge, including constructing headers, encoding the request body, and parsing the response.

Conclusion

This guide demonstrates the basic implementation of Stripe payments in a Phoenix 1.7 application using Stripity Stripe. The new component-based structure in Phoenix 1.7 promotes modular code organization, although it requires some adjustment in development approach.

This implementation covers the fundamentals of payment processing. For more advanced features such as subscriptions or invoicing, refer to the Stripity Stripe documentation.

Print function in Zig

2024-09-03T00:00:00+00:00

Zig’s Print Function: Why the Empty Struct?

If you’ve used Zig, you’ve likely encountered this syntax:

std.debug.print("Hello, world!\n", .{});

The empty struct (.{}) at the end might seem puzzling, especially when printing a static string. This post explores the rationale behind this design choice and its implications for type safety and security.

The Printf Legacy and Its Drawbacks

C’s printf function has long been a standard for formatted output in many languages. However, it comes with significant drawbacks in type safety and security. Zig’s approach addresses these issues, albeit with a syntax that may initially seem counterintuitive.

Let’s examine the problems with printf-style functions and how Zig’s solution, while more verbose, provides important safeguards.

The Problem with Printf

C’s printf function and its variants are variadic, meaning they can take a variable number of arguments. While this provides flexibility, it also opens the door to several potential issues.

1. Type Safety Concerns

The printf function relies on format specifiers in the format string to determine how to interpret and print its arguments. However, the compiler can’t always verify if the format specifiers match the types of the provided arguments. This can lead to various problems:

#include <stdio.h>

int main() {
    // Example 1: Type mismatch
    int num = 42;
    printf("The number is %f\n", num);  // Should be %d for int

    // Example 2: Wrong number of arguments
    printf("Name: %s, Age: %d\n", "Alice");  // Missing the age argument

    // Example 3: Passing wrong type of pointer
    char* str = "Hello";
    printf("String: %s\n", &str);  // Should be just str, not &str

    return 0;
}

In these examples:

We’re passing an integer to a float format specifier.
We’re missing an argument for the age.
We’re passing a pointer to a string pointer instead of just the string pointer.

All of these will compile with warnings (if warnings are enabled), but they can lead to undefined behavior at runtime. The program might crash, print garbage values, or in some cases, even appear to work correctly (which can be dangerous as it might go unnoticed).

2. Security Vulnerabilities

Even more concerning are the potential security vulnerabilities, particularly format string attacks. These occur when an attacker can control the format string passed to a printf-style function. Here’s a vulnerable example:

#include <stdio.h>

void vulnerable_function(char *user_input) {
    printf(user_input);
}

int main() {
    char user_input[100];
    printf("Enter a string: ");
    fgets(user_input, sizeof(user_input), stdin);
    vulnerable_function(user_input);
    return 0;
}

In this code, vulnerable_function directly passes user_input as the format string to printf. An attacker could exploit this by entering format specifiers as input:

%x %x %x %x could print values from the stack, potentially leaking sensitive information.
%n could be used to write to memory locations, potentially overwriting important data or hijacking program flow.

Modern Solutions: The Zig Approach

Modern languages like Zig have recognized these issues and taken steps to address them. Let’s look at how Zig handles string formatting:

const std = @import("std");

pub fn main() !void {
    const number = 42;
    std.debug.print("The number is {}\n", .{number});
}

Key differences in Zig’s approach:

Type Safety: The compiler always knows the types of the arguments being passed, ensuring type safety at compile-time.
Format String Separation: The format string and the arguments are separate parameters. The .{number} syntax creates an anonymous struct of arguments.
Compile-time Checking: Zig performs compile-time checking to ensure the format string matches the provided arguments.
No Variadic Functions: By not using variadic functions, Zig eliminates an entire class of potential issues.

While this approach may seem more verbose for simple cases (e.g., std.debug.print("Hello, world!\n", .{});), it provides significant safety benefits:

Impossible to mismatch format specifiers and argument types
No risk of format string attacks
Clear separation between the format string and the data being formatted

Zig’s Design Philosophy

Zig’s approach to string formatting isn’t just a technical solution; it’s a reflection of the language’s broader design philosophy. One of Zig’s zen tenets particularly relevant to this discussion is:

“Compile errors are better than runtime crashes.”

This principle is clearly illustrated in Zig’s handling of string formatting:

Type Checking at Compile Time: By separating the format string and the arguments, and using compile-time type checking, Zig ensures that type mismatches are caught before the program ever runs.
No Format String Vulnerabilities: The design makes it impossible to introduce format string vulnerabilities, eliminating an entire class of runtime security issues at compile time.
Explicit Argument Passing: The .{} syntax for passing arguments is more explicit, reducing the chance of accidentally omitting arguments or passing the wrong number of arguments.

By prioritizing compile-time checks and explicit syntax, Zig embodies its philosophy of preferring compile errors to runtime crashes. This not only improves program safety but also enhances the developer experience by catching potential issues early in the development process.

Conclusion

The design choices in Zig’s string formatting system, guided by principles like “Compile errors are better than runtime crashes,” demonstrate how language design can significantly impact code safety, readability, and maintainability. As we’ve seen, these choices can eliminate entire categories of bugs and vulnerabilities, making a strong case for prioritizing safety and explicitness in language design.

Zig Memory Allocation

2024-08-29T00:00:00+00:00

Memory Management in Zig: A Comprehensive Guide

Zig, a modern systems programming language, offers a unique approach to memory management. Unlike languages with garbage collection or those requiring manual memory management, Zig provides a set of tools and patterns that allow developers to choose the most appropriate memory management strategy for their specific needs. In this post, we’ll explore the various ways to allocate and manage memory in Zig, including both heap and stack allocation.

Heap Allocation Methods

1. General Purpose Allocator (GPA)

The General Purpose Allocator is Zig’s default allocator for most applications. It’s designed to be efficient for a wide range of allocation patterns.

var gpa = std.heap.GeneralPurposeAllocator(.{}){};
defer {
    const leaked = gpa.deinit();
    if (leaked) std.debug.print("Memory leak detected!\n", .{});
}
const allocator = gpa.allocator();

const memory = try allocator.alloc(u8, 100);
defer allocator.free(memory);

Management: Manual deallocation required, but the GPA can detect leaks.

2. Arena Allocator

The Arena Allocator is perfect for scenarios where you need to allocate memory frequently and free it all at once.

var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
defer arena.deinit();
const allocator = arena.allocator();

const memory1 = try allocator.alloc(u8, 100);
const memory2 = try allocator.alloc(u8, 200);
// No need to free memory1 or memory2 individually

Management: All memory is freed at once when arena.deinit() is called.

3. Fixed Buffer Allocator

When you have a fixed amount of memory to work with, the Fixed Buffer Allocator is an excellent choice.

var buffer: [1024]u8 = undefined;
var fba = std.heap.FixedBufferAllocator.init(&buffer);
const allocator = fba.allocator();

const memory = try allocator.alloc(u8, 100);
// No need to free; memory is invalid when buffer goes out of scope

Management: No explicit deallocation needed; memory is reclaimed when the buffer goes out of scope.

4. Page Allocator

For large allocations, the Page Allocator directly interfaces with the operating system to allocate memory in page-sized chunks.

const allocator = std.heap.page_allocator;
const memory = try allocator.alloc(u8, 4096);
defer allocator.free(memory);

Management: Manual deallocation required.

5. C Allocator

When interfacing with C libraries or when you need C-compatible allocation, the C Allocator is available.

const allocator = std.heap.c_allocator;
const memory = try allocator.alloc(u8, 100);
defer allocator.free(memory);

Management: Manual deallocation required, follows C allocation patterns.

6. Allocator-Aware Types

Many types in Zig’s standard library are allocator-aware, managing their own memory when given an allocator.

var list = std.ArrayList(u32).init(allocator);
defer list.deinit();

try list.append(42);

Management: Memory is managed by the type itself, freed when deinit() is called.

7. Scratch Allocator

For temporary allocations within a single function call, the Scratch Allocator can be very efficient.

var scratch_buffer: [1024]u8 = undefined;
var fba = std.heap.FixedBufferAllocator.init(&scratch_buffer);
var scratch = std.heap.ScratchAllocator.init(&fba);
const allocator = scratch.allocator();

const memory = try allocator.alloc(u8, 100);
// No need to free; memory is reclaimed when scratch goes out of scope

Management: Memory is automatically reclaimed when the scratch allocator goes out of scope.

8. Memory Pools

For scenarios where you need to frequently allocate and deallocate objects of the same size, a memory pool can be very efficient.

const MyStruct = struct { value: u32 };
var pool = std.heap.MemoryPool(MyStruct).init(allocator);
defer pool.deinit();

const obj = try pool.create();
defer pool.destroy(obj);

Management: Objects are returned to the pool when destroyed, the entire pool is freed on deinit().

9. Garbage Collection

While Zig doesn’t have built-in garbage collection, you can implement or use third-party GC libraries for specific use cases.

// Example using a hypothetical GC library
const GC = @import("gc");
var gc = GC.init(allocator);
defer gc.deinit();

const memory = try gc.alloc(u8, 100);
// No need to manually free; GC will handle it

Management: Automatic, handled by the garbage collector.

Stack Allocation

While heap allocation is powerful and flexible, Zig also provides efficient stack allocation for scenarios where it’s more appropriate. Stack allocation is faster and doesn’t require manual memory management, but it’s limited in size and scope.

fn stackExample() void {
    var buffer: [1024]u8 = undefined;
    var number: i32 = 42;
    // buffer and number are allocated on the stack and automatically freed when the function returns
}

Characteristics of Stack Allocation:

Fast allocation and deallocation
Memory is automatically managed
Limited in size (typically a few MB)
LIFO (Last In, First Out) structure

Use Cases for Stack Allocation:

Local variables
Function parameters
Small, fixed-size data structures
Short-lived data

Stack vs Heap: When to Use Which

Understanding when to use stack allocation versus heap allocation is crucial for efficient memory management in Zig.

Use Stack When:

You know the size of data at compile time
Data is small and short-lived
You need fast allocation/deallocation
You’re working with function-local data
You want to avoid memory fragmentation

Use Heap When:

Data size is unknown at compile time or can grow
You need data to persist beyond function calls
You’re working with large datasets
You need to share data between different parts of your program
You’re implementing complex data structures like trees or graphs

Example: Choosing Between Stack and Heap

Here’s an example that demonstrates how you might choose between stack and heap allocation based on runtime conditions:

fn example(allocator: std.mem.Allocator, input: []const u8) !void {
    if (input.len <= 1024) {
        var buffer: [1024]u8 = undefined;
        @memcpy(buffer[0..input.len], input);
        // Use buffer (stack allocation)
    } else {
        var dynamic_buffer = try allocator.alloc(u8, input.len);
        defer allocator.free(dynamic_buffer);
        @memcpy(dynamic_buffer, input);
        // Use dynamic_buffer (heap allocation)
    }
}

This example uses stack allocation for small inputs and falls back to heap allocation for larger inputs, demonstrating how Zig allows for flexible memory management strategies.

Welcoming Dumb Ideas

2024-04-11T00:00:00+00:00

0 interest is an interest level

I just stumbled upon a viral-ish X post where most of the commenters made fun of Grant Cardone, a known “grifter” in a lot of people’s eyes: Grant Cardone has an IQ of 42 and is worth $600 million. What is holding you back?

A comical thesis he proposed was that “0 interest, is an interest level”. Which I don’t recommend you try unless you want HR to randomly Zoom you.

When you have a perception of a person, it’s hard to not have a certain preconceived bias. Instead of taking it face value I thought about it a bit more. Is there any possible nugget of wisdom or something we can learn here? While you may not need to agree 100%, you should be aware of these biases that might potentially shoot down ideas pre-emptively.

After some more thought, I believe what he was trying to say is when you want to sell something to someone who has 0 interest level, it’s not the end of the world. The “0 interest level” is a litmus or calibration point to effectively tell you where you stand in this person’s mind.

In software engineering, we are constantly exchanging ideas back and forth and trying to build consensus. Believe it or not, one does not simply read How to Win Friends and Influence People and become an expert in soft skills as repeatedly pitched by managers around the world.

My personal problem with the book is some what summarized from another Reddit user:

However, after 80 years in the field, its techniques have been vastly surpassed by later developments in human psychology and influence. I’m in a role where I have to deal with dozens of salesmen hitting me up every week. If I see a Dale Carnegie technique (‘complementing’, ‘using my name’, ‘talking in terms of my interests’) I know I’m dealing with someone very new in a sales role.

Although the principles I will highlight later make them quite evident, I find that they need more concrete ways of applications as a day to day software developer.

Something I experienced in my last company was the idea of Test Driven Development or Testing culture in general. Principal engineers would vouch for the bugs saved or the cost savings from re-work. Being able to be agile and move fast while having regression safety net. On the flip side there are cultures that do not embrace testing as it slows them down, etc. Regardless to say, testing was mandated, but how effective was it exactly? Thats not the topic of this post in particular, but the semination of ideas into your organization.

Building organizational consensus for effectuating change:

Build the why:

Listing benefits that people already know isn’t going to necessarily help in my experience. Non technical people prefer ROI/Cost savings, data helps. Listening and solving their major pain points from their point of view helps. If you are just simply trying to practice resume driven development or over optimize/refactor a piece of your code base with no convincable benefits, you are probably going to get shot down. Avoid Shiny Object Syndrome for the sake of shiny.

Principle 1: Fundamental Techniques in Handling People. By listening to their ideas and solving their major pain points, you’re showing genuine interest in their perspective and fostering a positive relationship.

On a random side note on Clean Code. I highly recommend this very informative debate between Bob Martin and Casey M: https://github.com/unclebob/cmuratori-discussion/blob/main/cleancodeqa.md It’s a very good case study of two programmers coming from two different backgrounds on why they believe in their philosophies.

Provide Education and Training for Ease of adoptability:

Offer training on the importance, methodologies, and best practices. Empowering team members with knowledge helps them embrace the change confidently. To introduce a new process or idea, it must be low barrier or small investment for users to adopt. If it’s something completely ridiculous and creates more work it’s obviously a no.

Having an “escape hatch” or a backup plan can provide reassurance and mitigate the fear of failure when implementing changes in a company. It allows for a safety net in case the change doesn’t yield the expected results or faces unforeseen challenges. This backup plan can include alternative strategies, contingency measures, or even a way to revert to previous processes if necessary.

Principle 2: Six Ways to Make People Like You. By offering training on the importance, methodologies, and best practices, you’re demonstrating genuine interest in their development and success. Empowering team members with knowledge helps them embrace the change confidently and builds trust and rapport.

Involve your team early:

I think a more effective approach is talking to your co-workers one by one, and listening to their ideas. You can then achieve better overall buy-in now that everyone is on board. If you can then sell why your new idea or approach is saving them from a pain point(or for non technical people, make $), you are going to better be able to get consensus. Include team members in decision-making from the start, seeking their input and concerns. This fosters ownership and commitment to the change.

Principle 3: How to Win People to Your Way of Thinking. By including team members in decision-making from the start and seeking their input and concerns, you’re showing respect for their opinions and making them feel valued. Additionally, your emphasis on selling the benefits of your ideas by addressing their pain points ties into the principle of appealing to the other person’s interests.

Conclusion

Sometimes dumb ideas can have benefits that you never thought. It’s more of a matter of building the proper consensus with your team. Tying back into the idea that a company culture that embraces psychological safety will have a lot more potential for innovation compared to ideas simply mandated from an ivory tower. If all else fails just become the CEO of your own company 😂

Embracing the Weight

2024-04-05T00:00:00+00:00

Confronting Impostor Syndrome in Software Engineering and Powerlifting

In the realms of both software engineering and powerlifting, there exists a common adversary that haunts the minds of many practitioners: impostor syndrome. This psychological phenomenon, characterized by feelings of inadequacy and self-doubt despite evident competence, manifests itself in strikingly similar ways in these seemingly disparate domains.

Imagine stepping into a weight room, the clinks of iron plates echoing as seasoned lifters go about their routines with apparent ease. As a novice, each lift feels like a monumental task, and the specter of failure looms large. Similarly, in the world of software engineering, entering a new project or tackling a complex problem can evoke that same sense of trepidation. Despite possessing the requisite skills and knowledge, doubts creep in, whispering that perhaps you don’t belong among the ranks of your peers.

Yet, it is precisely within these moments of doubt and discomfort that the parallels between powerlifting and software engineering reveal themselves most vividly. Consider the act of progressively overloading in powerlifting, where lifters continually push themselves to lift heavier weights. The process is not one of the weights becoming easier; rather, it is the lifter who grows stronger, adapting to the increasing demands placed upon them.

Similarly, in software engineering, the challenges we face do not diminish with experience; if anything, they become more complex and nuanced. Yet, it is through confronting these challenges head-on, stepping outside our comfort zones, that we cultivate resilience and expertise. Each bug squashed, each feature implemented is akin to adding another plate to the bar, pushing ourselves into the realm of the unknown and emerging stronger for it.

Moreover, both powerlifting and software engineering are communal endeavors, where individuals often find support and camaraderie within a community of peers. Just as a powerlifter may seek guidance from more experienced lifters to refine their technique and overcome plateaus, software engineers can benefit immensely from mentorship and collaboration. By fostering an environment where vulnerability is embraced and knowledge is shared freely, we can alleviate the burdens of impostor syndrome and empower each other to reach new heights.

Continuous Improvement: The Shared Path of Powerlifting and Software Engineering

At the heart of both powerlifting and software engineering lies a fundamental principle: the pursuit of continuous improvement. In powerlifting, this manifests as the relentless quest to lift heavier weights, to surpass previous personal bests, and to refine lifting technique for optimal performance. Similarly, in software engineering, the ethos of continuous improvement drives us to refine our coding practices, to seek out more efficient algorithms, and to constantly expand our knowledge base.

Just as a powerlifter meticulously tracks their progress, recording every lift and analyzing each session for areas of improvement, software engineers employ methodologies like Agile development to iteratively refine their work. Through practices such as regular code reviews, retrospectives, and continuous integration, we strive to identify weaknesses, address shortcomings, and evolve our skills over time.

Moreover, both powerlifting and software engineering recognize that the path to mastery is not a linear one. Setbacks and failures are inevitable, but it is how we respond to these challenges that ultimately defines our progress. Just as a missed lift serves as a learning opportunity for a powerlifter, a software bug or a failed deployment can provide valuable insights for an engineer. By embracing failure as an essential part of the learning process, we foster resilience and adaptability, propelling ourselves ever closer to our goals.

Furthermore, the concept of continuous improvement extends beyond individual practice to encompass the broader communities of powerlifting and software engineering. Lifters share training tips, programming strategies, and motivational support with one another, while software engineers collaborate on open-source projects, contribute to online forums, and attend conferences to stay abreast of the latest developments in the field. In doing so, we create a culture of learning and growth, where knowledge is freely exchanged, and innovation flourishes.

Culture Beats Talent: Cultivating Fearlessness in Collaboration

In the realms of powerlifting and software engineering alike, it’s often said that culture beats talent. While raw skill and technical prowess certainly have their place, it is the collective ethos of a team that truly sets the stage for success. In both disciplines, cultivating a fearless, collaborative culture can spell the difference between mediocrity and excellence.

Consider the powerlifting gym, where lifters of all skill levels gather to train, support, and motivate one another. Here, it’s not just about individual achievement but about lifting each other up—both literally and metaphorically. Seasoned lifters mentor novices, sharing their knowledge and expertise, while beginners infuse the gym with a sense of enthusiasm and camaraderie. Together, they create an environment where everyone feels empowered to push their limits, knowing that they have a community behind them every step of the way.

Similarly, in software engineering, the most successful teams are those that foster a culture of psychological safety, where team members feel comfortable taking risks, sharing ideas, and admitting mistakes without fear of judgment or reprisal. It’s a culture where egos are checked at the door, and collaboration takes precedence over individual glory. In such an environment, innovation flourishes, as diverse perspectives and complementary skill sets converge to solve complex problems with creativity and agility.

Moreover, a fearless, collaborative culture in both powerlifting and software engineering serves as a bulwark against the insidious influence of impostor syndrome. When individuals feel supported and valued by their peers, they are less likely to succumb to self-doubt and more inclined to embrace challenges head-on. It’s not about being the strongest lifter in the gym or the most brilliant coder in the room; it’s about contributing to something greater than oneself and lifting each other up in the process.

Ultimately, whether we’re striving to lift heavier weights or to write more elegant code, the power of collaboration cannot be overstated. By cultivating a fearless, collaborative culture—one that celebrates diversity, encourages experimentation, and values collective achievement—we can overcome the limitations of talent alone and achieve feats that would be impossible on our own.

Practical Advice: Navigating Impostor Syndrome with Powerlifting and Software Engineering

In the face of impostor syndrome, it’s essential to arm ourselves with practical strategies for overcoming self-doubt and forging ahead on our journey of growth and self-improvement. Here are three actionable tips to help navigate the challenges of impostor syndrome in both powerlifting and software engineering:

Set Realistic Goals: Break down your long-term aspirations into smaller, achievable milestones. Whether it’s increasing your deadlift by 10 pounds or mastering a new programming language, setting clear, measurable goals can provide a sense of direction and progress.
Seek Support and Mentorship: Don’t be afraid to reach out to more experienced lifters or seasoned software engineers for guidance and support. Mentorship can provide invaluable insights, encouragement, and perspective, helping to dispel feelings of isolation and inadequacy.
Celebrate Your Progress: Acknowledge and celebrate your achievements, no matter how small they may seem. Whether it’s completing a challenging workout or successfully debugging a piece of code, take pride in your accomplishments and use them as fuel to propel you forward on your journey.

By incorporating these practical tips into your practice, you can cultivate a mindset of resilience, self-confidence, and growth, both in the weight room and in the world of software engineering. Remember, overcoming impostor syndrome is not about eliminating self-doubt entirely but rather learning to navigate it constructively, using it as fuel to propel you forward on your journey of self-discovery and mastery. Embrace the challenges, seek support from your peers, and never underestimate the power of continuous improvement and collaboration. With perseverance and determination, you’ll not only conquer impostor syndrome but emerge stronger and more resilient than ever before.

Beyond Documentation: The Art of Reverse Engineering in Development

2024-03-07T00:00:00+00:00

Introduction:

In the world of software development, documentation is often hailed as the cornerstone of understanding and maintaining complex systems. Whether it’s meticulously crafted API references, detailed architectural diagrams, or comprehensive user manuals, documentation serves as the guiding light for developers navigating through the intricate labyrinth of code. However, while documentation undoubtedly offers valuable insights into the inner workings of a system, it’s essential to recognize that it’s not always the silver bullet it’s made out to be.

In the fast-paced reality of the software industry, adaptability and the ability to learn quickly are paramount. Throughout my career, which began in the realm of Ruby on Rails development, I’ve encountered diverse systems, often without the luxury of comprehensive documentation. From navigating through C# and .NET to delving into Java and Spring development, and even spearheading migrations from JQuery to React, each transition has underscored the importance of adeptness in reverse engineering principles. In environments where documentation may be scarce or outdated, the ability to dissect and comprehend existing codebases becomes invaluable for effective problem-solving and innovation. This journey through various technologies has shaped my perspective on the symbiotic relationship between documentation and reverse engineering.

The Myth of Documentation as a Silver Bullet:

Before delving into the depths of reverse engineering, it’s crucial to acknowledge the merits of documentation. After all, well-written documentation can significantly streamline onboarding processes, facilitate collaboration among team members, and provide crucial context for future enhancements or bug fixes. From high-level overviews to granular code comments, documentation comes in various forms, each serving its unique purpose.

API Documentation: API documentation provides developers with a roadmap for interacting with external services or libraries. It outlines available endpoints, parameters, response formats, and usage examples, empowering developers to leverage third-party functionality efficiently.
Architectural Documentation: Architectural diagrams, such as UML diagrams or system flowcharts, offer a bird’s-eye view of the system’s structure and components. They help developers understand how different modules interact with each other and how data flows throughout the system.
Code Comments and Inline Documentation: Within the code itself, comments and inline documentation serve as invaluable signposts for understanding implementation details, rationale behind design decisions, and potential pitfalls to watch out for. Well-documented code is not only easier to maintain but also fosters knowledge sharing and code reuse across teams.

While documentation undoubtedly offers numerous benefits, it’s essential to recognize its inherent limitations. No matter how comprehensive or meticulously maintained, documentation inevitably suffers from a few common pitfalls:

Incompleteness: Despite the best intentions, documentation is rarely exhaustive. It’s challenging to capture every edge case, exception, or subtle nuance in written form, leading to gaps in understanding for developers.
Outdated Information: In fast-paced development environments, code evolves rapidly, often outpacing the corresponding documentation. As a result, developers may find themselves relying on outdated or inaccurate information, leading to confusion and frustration.
Lack of Context: Documentation provides a snapshot of the system at a particular point in time, but it often lacks the contextual richness necessary for fully understanding the underlying rationale or trade-offs behind design decisions.

Reverse Engineering a Full-Stack CRUD Feature in Rails:

Imagine you’re a new engineer tasked with understanding and modifying a full-stack CRUD (Create, Read, Update, Delete) feature in a Ruby on Rails application. At first glance, the codebase may appear daunting, with multiple layers of abstraction and interconnected components. However, by applying the principles of reverse engineering, you can systematically unravel the complexities and gain a deeper understanding of how the system functions.

Understanding the MVC Pattern:

Ruby on Rails follows the Model-View-Controller (MVC) architectural pattern, which divides an application into three interconnected components:

Model: Represents the data and business logic of the application. In Rails, models typically correspond to database tables and encapsulate operations such as querying, creating, updating, and deleting records.
View: Handles the presentation layer of the application, rendering HTML templates and responding to user interactions. Views in Rails are often written in embedded Ruby (ERB) or utilize front-end frameworks like React or Angular for more complex interfaces.
Controller: Acts as an intermediary between the model and view components, handling incoming requests, processing user input, and orchestrating the flow of data. Controllers in Rails contain action methods corresponding to CRUD operations (e.g., create, index, show, update, destroy).

Reverse Engineering Process:

Start with the Controller: Begin your reverse engineering journey by examining the controller responsible for the CRUD action you’re investigating. Locate the corresponding action methods, such as create, update, or destroy, and analyze their logic and interactions with the model layer.
Explore the Model Layer: Dive deeper into the model layer to understand how data is structured, validated, and persisted in the database. Identify the ActiveRecord models associated with the CRUD action and inspect their attributes, associations, and callbacks.
Inspect the View Templates: Next, inspect the view templates associated with the CRUD action to understand how data is presented to users. Look for ERB files or front-end framework components that render forms, tables, or other UI elements relevant to the CRUD operations.

Deeper Debugging Principles and Skills in Rails Web Applications:

In the realm of Rails web applications, mastering debugging principles and skills beyond basic breakpoint-based debugging can significantly enhance a developer’s ability to diagnose issues, both in development and production environments. Let’s explore some advanced techniques and tools that senior developers employ to tackle complex problems effectively.

Debugging in JavaScript Environments:

In Rails development, debugging often involves navigating through the server-side codebase to identify and resolve issues. While tools like Pry, Byebug, and Rails console offer invaluable insights into application state and behavior, developers can also leverage browser-based debugging tools to diagnose frontend issues.

When it comes to debugging JavaScript code in web applications, Browser Developer Tools, such as Chrome DevTools and Firefox Developer Tools, are indispensable. These tools provide a suite of features for inspecting HTML elements, monitoring network requests, and debugging JavaScript code in real-time.

Key features of browser DevTools include:

Console: The Console tab allows developers to log messages, execute JavaScript code snippets, and debug runtime errors directly within the browser environment.
Debugger: The Debugger tab provides a powerful interface for setting breakpoints, stepping through code execution, and inspecting variable values during runtime. Developers can pinpoint the exact location of errors and trace the execution flow through complex JavaScript functions.
Network Analysis: The Network tab enables developers to monitor HTTP requests and responses, analyze network performance, and identify potential bottlenecks or errors in API interactions.
DOM Inspection: The Elements tab provides a visual representation of the Document Object Model (DOM), allowing developers to inspect and manipulate HTML elements, CSS styles, and event listeners dynamically.

By leveraging browser DevTools in conjunction with server-side debugging techniques, developers can diagnose and resolve issues more effectively, regardless of whether they originate from frontend or backend code. This integrated approach to debugging empowers developers to gain comprehensive insights into application behavior.

Command Line Interface (CLI) Tools:

Rails developers often rely on command-line tools to streamline development workflows and troubleshoot issues efficiently. Leveraging CLI tools like Pry, Byebug, or Rails console allows developers to interactively explore the application state, execute ad-hoc queries, and simulate edge cases in a controlled environment. Additionally, tools like Rails ERD or Brakeman help analyze database schemas and identify security vulnerabilities, respectively.

Creative Debugging Techniques:

Sometimes, resolving obscure bugs in Rails applications requires thinking outside the box and employing creative debugging techniques. This could involve using logging libraries like Airbrake to capture detailed application logs and trace the execution flow across multiple components. Additionally, implementing feature flags or toggles using tools like Flipper can enable developers to selectively enable or disable specific features in production to isolate and diagnose issues.

Diagnosing Issues in Production Systems:

Debugging issues in production systems requires a different approach due to the constraints of real-world environments. Senior developers often employ strategies such as log aggregation and monitoring using tools like Elasticsearch, Kibana, or New Relic to gain insights into application performance, error rates, and resource utilization. By setting up robust alerting mechanisms and implementing automated recovery procedures, developers can proactively detect and mitigate issues before they impact end-users.

Debugging Production Issues with AWS CloudWatch Logs:

When facing production issues in a Rails application deployed on AWS, leveraging AWS CloudWatch Logs can provide valuable insights into the root cause of the problem. Here’s a concise guide to debugging based on logs alone:

Log Collection Setup: Ensure Rails application logs are configured to capture relevant information. By default, Rails logs to log/production.log.
CloudWatch Logs Configuration: Set up AWS resources to send log data to CloudWatch Logs. Install the CloudWatch Logs agent or configure log streaming from application code.
Log Analysis and Insights: Use the CloudWatch Logs console to search, filter, and analyze log events in real-time. Create metric filters and alarms to detect specific patterns or anomalies.
Debugging Based on Logs Alone: Identify relevant log entries related to the reported problem. Look for error messages, exceptions, or warning signs to trace the issue’s root cause.
Sample Queries: For CloudWatch Logs, a sample query might look like this:

fields @timestamp, @message
| filter @message like /NoMethodError/
| sort @timestamp desc

Sample Rails Log Entry: A sample Rails log entry showing a “NoMethodError” with a larger stack trace might look like this:

[ERROR] [2024-03-10 15:45:21] NoMethodError (undefined method 'name' for nil:NilClass):
Traceback (most recent call last):
  app/controllers/users_controller.rb:25:in `show'
  app/controllers/application_controller.rb:42:in `authorize'
  app/models/user.rb:36:in `get_name'
  app/models/user.rb:15:in `full_name'
  app/views/users/show.html.erb:12:in `_app_views_users_show_html_erb___123456789'
...

Remote Debugging and Error Tracking with Airbrake

In addition to leveraging logs for debugging, utilizing specialized error tracking and monitoring tools like Airbrake can enhance the debugging process further. Airbrake provides real-time error tracking for Rails applications, capturing exceptions and errors as they occur in production environments. By integrating Airbrake with your Rails application, you can gain insights into the frequency, severity, and impact of errors, allowing you to prioritize and address critical issues promptly.

Using Airbrake for Remote Debugging

Airbrake offers remote debugging capabilities, enabling developers to diagnose and troubleshoot errors without direct access to the production environment. When an error occurs, Airbrake captures relevant information such as stack traces, request parameters, and environment details, providing valuable context for identifying the root cause of the issue. Developers can then use the Airbrake dashboard to view and analyze error occurrences, track their resolution progress, and collaborate with team members to implement fixes efficiently.

Benefits of Airbrake Integration

Real-time Error Notifications: Receive instant notifications when errors occur in your Rails application, allowing you to respond promptly and minimize downtime.
Detailed Error Reports: Access detailed error reports with stack traces, environment information, and user context to gain insights into the circumstances surrounding each error occurrence.
Workflow Integration: Seamlessly integrate Airbrake with your existing development workflow, enabling you to triage, prioritize, and address errors alongside feature development and bug fixes.
Historical Error Data: Leverage historical error data and trends to identify recurring issues, prioritize bug fixes, and proactively address potential stability and performance issues.

By incorporating Airbrake into your debugging toolkit, you can streamline the debugging process, improve the reliability of your Rails applications, and deliver a better experience for your users.

In conclusion, while documentation serves as a crucial reference point in understanding system architecture and functionality, it’s essential to acknowledge its limitations, especially in complex and evolving systems. The ability of senior developers to navigate through codebases via reverse engineering principles offers a complementary approach to understanding system intricacies. By combining thorough documentation with the adeptness to reverse engineer, developers can gain deeper insights into the inner workings of applications, fostering robust debugging, and problem-solving capabilities. Tools like AWS CloudWatch Logs, Airbrake, and other logging libraries further augment this process by providing real-time visibility into application behavior.

Ultimately, the synergy between documentation and reverse engineering empowers developers to navigate intricate codebases effectively, paving the way for increased productivity and efficiency.

Sprout Method

2019-08-19T00:00:00+00:00

Write tests for new code even if the old code isn’t under test. Although this isn’t a good long term solution it can help move the design forward.

For simple changes that can be captured in a method we can TDD that.

Let’s say we have some existing code:

public class TransactionGate
{
  public void postEntries(List entries) {
    // iterate through entries list and persist the transaction
  }
}

Now we want to remove duplicate entry’s before persisting them. A quick thing to do is write the new logic directly inside postEntries but this is bad for several reasons such as SRP violation. Instead we can sprout a new method called uniqueEntries and call it inside postEntries

public class TransactionGate
{
  // TDD this new method
  List uniqueEntries(List entries) {
    //remove duplicate entries
  }

  public void postEntries(List entries) {
    List entriesToAdd = uniqueEntries(entries);
    // iterate through entries list and persist the transaction
  }
}

Addendum (3/7/2024):

In response to feedback and to provide further clarity, I’d like to emphasize the importance of understanding the “sprout method” concept and its application in software development.

Understanding the “Sprout Method” Concept:

The “sprout method” pattern involves extracting a new method from existing code to encapsulate specific functionality. This approach promotes code modularity, readability, and maintainability by adhering to the Single Responsibility Principle (SRP). By isolating distinct tasks into separate methods, developers can create more cohesive and reusable code.

Benefits of Using the “Sprout Method” Pattern:

Improved Code Organization: The “sprout method” pattern allows developers to organize code more effectively by grouping related functionality into separate methods. This enhances code readability and makes it easier to understand the purpose and behavior of each method.
Enhanced Testability: Extracting logic into separate methods facilitates unit testing, as it isolates specific behaviors for testing. With focused, well-defined methods, developers can write more targeted tests with clear inputs and expected outputs.
Easier Maintenance and Refactoring: By adhering to the SRP and separating concerns, the “sprout method” pattern simplifies maintenance and refactoring efforts. Developers can modify or extend individual methods without impacting other parts of the codebase, reducing the risk of unintended side effects.

Applying the “Sprout Method” Pattern in Practice:

When faced with the task of adding new functionality or modifying existing code, consider whether the “sprout method” pattern can help improve code structure and maintainability. Look for opportunities to extract cohesive, reusable methods that encapsulate specific behaviors or operations.

In summary, understanding and applying the “sprout method” pattern can lead to cleaner, more maintainable codebases. By leveraging this pattern effectively, developers can enhance code organization, testability, and overall software quality.

Extract Interface

2019-08-16T00:00:00+00:00

Note: The sample code in this post is from Feathers’ book Working Effectively With Legacy Code which i’m trying to read through and summarize my knowledge.

Sometimes your design shows it could use improvement if you notice it’s hard to test. When it’s hard to test your code, you eventually won’t even bother.

A common thing that occurs is when you try to put a class under test it has dependencies/collaborating classes which need to be created. Below the Sale class depends on a ArtR56Display class.

public class Sale {
  public void scan() {
      ArtR56Display display = new ArtR56Display();
      display.showLine();
    }
  }
}

First, we need to use dependency injection on the display object. One easy way to do this is by passing it through the constructor. Typically instantiating objects inside another class will highlight dependencies you need to overcome (although there are exceptions e.g. basic value model objects).

public class Sale {
  private ArtR56Display display;

  public Sale(ArtR56Display display) {
    this.display = display;
  }

  public void scan() {
      display.showLine();
    }
  }
}

Now we want to test our sale class. We create a test which will instantiate a ArtR56Display but suprise, it’s connected to a physical cash register. It’s not very convenient to write unit tests that rely on this sort of external dependency just to run. Other typical dependencies such as DB connections also suffer from this issue.

So the solution here is to extract the display to an interface which has the side effect of promoting loose coupling and replaceability. It allows the Sale class to be extended but not modified. So in the future our Sale class can be hooked up to any other type of display.

public interface Display
{
  void showLine(String line);
}

public class Sale
{
  private Display display;

  public Sale(Display display) {
    this.display = display;
  }

  public void scan() {
    ...
    String itemLine = item.name() + " " + item.price().asDisplayText();
    display.showLine(itemLine);
  }
}

At this point we could create a FakeDisplay that implements showLine(). In our tests now we can inject our FakeDisplay and still test the scan method to ensure it still sends the right text to the display when the Sale class is used without depending on the physical cash register.

public class FakeDisplay implements Display {
  private String lastLine = "";

  public void showLine(String line) {
    lastLine = line;
  }

  public String getLastLine() {
    return lastLine;
  }
}

As a side note most Mocking libraries can handle most of this stuff for us by creating a proxy class for any interfaces we would like to fake and configuring it to execute dummy behavior.

Drawing Graphics in Elm

2017-06-28T00:00:00+00:00

I’m learning how to write games in Elm and there does not seem to be very many resources updated for Elm 0.18. The old Graphics module has been moved out of elm-lang/core and into evancz/elm-graphics which might cause some confusion/compilation errors when looking at older examples.

The elm-graphics package gives us modules that interface with HTML 5 Canvas API. So lets see how we can draw something basic to the browser. In my later posts we will continue to build upon this program.

Make sure you have the required packages installed in your project directory:

elm package install elm-lang/html
elm package install evancz/elm-graphics

Summary:

We learned how to render a “canvas” on the screen by using Element.toHtml which takes an Element created by Collage.collage. While the Collage mainly deals with free form graphics, the Element and Text Modules can help you plug in various other types of graphics into your collage once converted into a form. I recommend checking out the elm-graphics documentation to see what other Forms you can create in your collages. So far, drawing graphics to the screen may seem somewhat tedious as it is in most other libraries. Elm’s architecture will start to shine in its simplicity in how updates and events are handled in your app.

Refactoring Part 1

2017-04-13T00:00:00+00:00

You may hear many developers groan at working on a legacy code base out of control versus a greenfield project. However, I believe working in a large legacy codebase provides a wonderful opportunity to improve your refactoring skills.

This will be a series of connected blog posts on refactoring real world code at my current job. I’ve attempted to obfuscate the code to make it more generic and not tied to my company but the spirit should be the same. Hopefully you are able to notice these issues in your own code and improve it.

So the first thing I did was run the flog gem(it essentially scores an ABC metric: Assignments, Branches, Calls) on my app/ folder and tackled the highest rated file which not suprisingly, was a controller:

1: flog total
1: flog/method average

1: RaterController#index            app/controllers/rater_controller.rb:3-2

I will go ahead and make the commentary before you look at the wall of abomination. At first glance the code is so dense that it is not clear at all what we’re trying to do in the index action.

I hope this can convince you on the importance of having cleanly formatted code right off the bat. While it’s a very small file, it contains immense complexity. The code smells I wanted to refactor in this first attempt:

1) Long method

It’s a 24 line index action method. Cannot tell at glance what it’s doing.

2) Single Responsibility Principle violation

While it’s reasonable to have a controller to delegate rating different objects, it can get out of control to put that logic directly in the controller. We could at first just perform Extract Method but that would just be shuffling complexity within the file and does not let us take advantage of fixing the next issue.

3) Case statement that type checks and Open/Close Principle Violation

We observe an if-else block that kept getting piled on as more domain objects were needed to be rated

class RaterController < AuthenticatedUserController

  def index
    @title = I18n.t 'rater.title'
    @active_rating   = manager_selection_is_self? ? params[:employee_rating   ] : params[:manager_rating]
    @inactive_rating = manager_selection_is_self? ? params[:manager_rating] : params[:employee_rating   ]
    @fullscreen_form_action = ''

    if params[:rated_item_type] == 'movies'
      movie = Movie.find(params[:id])
      @rated_item = MoviePresenter.new movie
      @fullscreen_form_action = rate_movie_path(id: movie.id, is_self: manager_selection_is_self?)
    elsif params[:rated_item_type] == 'books'
      book = Book.find(params[:id])

      ratee = manager_selection_is_self? ? current_user.id : selected_user.id
      rater = current_user.id

      @fullscreen_form_action = rate_book_path(id: Book.id, rater:rater, ratee: ratee, manager_rating: params[:manager_rating], employee_rating:params[:employee_rating])

      @rated_item = BookPresenter.new Book, selected_user_id: @selected_user_id
    elsif params[:rated_item_type] == 'restaurants'
      @fullscreen_form_action = rate_restaurants_path(id: selected_user.id, is_self: manager_selection_is_self?, rater_id: current_user.id, old_employee_rating: params[:employee_rating], old_manager_rating: params[:manager_rating])
      @rated_item = RestaurantRating.new(selected_user)
    end

    render layout: 'fullscreen_form_dialog'
  end

end

You probably didn’t even bother to try and understand what was going on above, huh? Here is the refactored code with changes in comments

class RaterController < AuthenticatedUserController
  def index
    @title                  = I18n.t 'rater.title'
    @active_rating          = active_rating?
    @inactive_rating        = inactive_rating?
    @fullscreen_form_action = ''

    # Extracted logic to object_rater method to do all the work
    @fullscreen_form_action, @rated_item = object_rater
    render layout: 'fullscreen_form_dialog'
  end

  private

  # Use a rater "factory" to instantiate specific raters
  # Note we are using convention over configuration here:
  # We assume that we have classes with name DomainItemRate
  def object_rater
    object_rater_class.new(rating_params).call
  end

  # The if-else can be removed now since we have a factory
  # that knows how to instantiate itself based off item type.
  # This takes advantage of Ruby's dynamic dispatching instead
  # instead of needing to type check.
  def object_rater_class
    "#{params[:rated_item_type]}Rate".classify.constantize
  end

  # Instantiate a Parameter Object to pass to rater
  def rating_params
    "#{rating_type_rating}".classify.constantize.new(params)
  end

  # Remaining methods are helpers to parse params
  def active_rating?
    # return active rating check based on user
  end

  def inactive_rating?
    # return inactive rating check based on user
  end

  def id
    params['id'].to_i
  end

  def employee_rating
    params[:employee_rating]
  end

  def manager_rating
    params[:manager_rating]
  end

  def ratee
    # return the ratee
  end

  def rater
   # return the rater
  end
end

We don’t have to modify the index action at all now and have a nice convention for maintaining new Rater types. We simply just implement a Rater class that “duck types” what the index action wants returned so we can maintain Open/Close principles.

Example Rater Class:

class BookRater
  def initialize(args)
    @id = args[:id].to_i
    @is_self = args[:is_self]
  end

  def call
    fullscreen_form_action = rate_book_path(id: book.id,
                                             is_self: @is_self)
    rated_item = BookPresenter.new book

    [fullscreen_form_action, rated_item]
  end

  private

  def book
    Book.find(id)
  end
end

Summary: We broke out the rating logic for each domain type into plain old Ruby objects(POROs) . We were able to instantiate these on the fly and remove the if-else type checking by utilizing a factory pattern to dynamically dispatch to our POROs.

Functional Programming in Ruby

2017-02-04T00:00:00+00:00

Although I’ve been focusing learning functional programming through Elixir and Elm these days, it’s interesting to go back to the first programming language I love to see what it can do.

These are some examples of Ruby’s functional programming abilities taken from HackerRank’s Ruby challenges.

Ruby can support higher-order functions i.e. can take and pass functions as arguments through several mechanisms: Blocks, Procs, and Lambdas. I’m not going to go into super detail as there are plenty good posts already out there on the topic. But a basic primer goes as follows:

Passing a Block to a function as nameless method:

def calculate(a,b)
    yield(a, b)
end

puts calculate(15, 10) {|a, b| a - b} # 5

Assigning a Proc to a variable. Proc’s are like a “saved” block:

def foo(a, b, my_proc)
    my_proc.call(a, b)
end

add = proc {|x, y| x + y}

puts foo(15, 10, add) # 25

Very similar to Procs but differing how they return (see link mentioned beginning of post) out of a function call is Ruby’s lambda.

# the -> keyword is Ruby >= 1.9's lambda syntax
area = ->(a, b) { a * b }

x = 10.0; y = 20.0

area_rectangle = area.(x, y)
area_triangle = 0.5 * area.call(x, y)

puts area_rectangle     #200.0
puts area_triangle      #100.0

Let’s dig into an interesting example where we show partial function applications

def factorial(n)
    (1..n).inject(:*) || 1
end


combination = -> (n) do
    -> (r) do
        factorial(n) / (factorial(r) * factorial(n-r))
    end
end

#How many combinations in 52 card deck when taking 5 cards?
n = gets.to_i # Let's enter 52
r = gets.to_i # Let's enter 5
nCr = combination.(n) # returns a lambda object
puts nCr.(r) # call our lambda with 2nd argument to be applied, result is 2,598,960

So similar to partial function applications is the concept of currying. Currying is converting a single function of n arguments into n functions with a single argument each.

power_function = -> (x, z) {
    (x) ** z
}

base = gets.to_i
raise_to_power = power_function.curry.(base)

power = gets.to_i
puts raise_to_power.(power)

JS Map Implementation

2016-12-29T00:00:00+00:00

This is my answer to one of the questions from Quora I stumbled upon concerning how to judge a good JS dev.

“JavaScript is a functional programming language. Give me an example of how you can use higher order functions in JavaScript to iterate over and apply a function to every element in an array. Do not use a for or while loop.”

I didn’t find that this particular challenge noted much advance knowledge of JS. But some key things it does touch on are adding prototype functions, loss of functional scope of ‘this’, and basic idea of how recursion works. Note: I am using some ES6 features and wrote it via a Jasmine test.

describe("map recursive implementation", () => {
  it("applies function over array", () => {

     // Let's add our function directly to all Array's via prototype
     // So we can make calls like [1,2,3].myMap(...)
     Array.prototype.myMap = function myMap(callback) {
      // function implicitly binded to the original calling object
      // i.e. [1,5,10,15].myMap(...)
      // therefore 'this' is equal to the array
      // so we can save this to a new variable called arr
      let arr = this;
      let newArray = [];

      function map(arr) {
        if(arr.length === 0) {
          console.log("DONE processing, new array is " + newArray)
          console.log(newArray.length);
          return newArray;
        }
        else {
          //process the head first
          console.log("processing head of array: " + arr[0])
          console.log("pushing: " + callback(arr[0]))
          newArray.push(callback(arr[0]))

          //process the rest of the list
          arr.shift()
          console.log("processing rest of list: " + arr)

          //recursive call here
          //we pass the new array which removed first processed item
          return map(arr, callback);
        }
      }

      return map(this);
    }


    let doubles = [1,5,10,15].myMap(function(x) {
      return x * 2;
    });

    expect(doubles).toEqual([2,10,20,30])
  })
})

Prototypes in Javascript

2016-12-20T00:00:00+00:00

What is a prototype?

An object that exists on every function in javascript.

A function’s prototype: A function’s prototype is the object instance that will become the prototype for all objects created using this function as a constructor

An object’s prototype: An object’s prototype is the object instance from which the object is inherited.

var myFunc = function() {}
console.log(myFunc.prototype) // empty protototype object

var cat = {name: 'Fluffy'}
console.log(cat.__proto__); // note objects don't have a prototype but have a __proto__ object

function Cat(name, color) {
  this.name = name
  this.color = color
}

var fluffy = new Cat('Fluffy', 'White')

Cat.prototype.age = 3

console.log(Cat.prototype) // has a property age: 3
console.log(fluffy.__proto__) // has a property age: 3
console.log(Cat.prototype === fluffy.__proto__) // true - same prototype instance

var muffin = new Cat('Muffin', 'Brown')
console.log(muffin.__proto__) // has a property age: 3


fluffy.age = 4

//instance properties override prototype properties
console.log(fluffy.age) // 4
console.log(fluffy.__proto__.age) // 3

delete fluffy.age
console.log(fluffy.age) // 3 - looks up prototype chain

//Changing a Function's prototype
Cat.prototype = {age: 5}

var snowbell = new Cat('Snowbell', 'White')

console.log(fluffy.age) // 3 still refers to original prototype instance
console.log(muffin.age) // 3 still refers to original prototype instance
console.log(snowbell.age) // 5 gains new defined prototype
console.log(snowbell.age) // 5 gains new defined prototype

//Prototype chains
console.log(fluffy.__proto__) // Cat { age: 4 }
console.log(fluffy.__proto__.__proto__) // Object {}
console.log(fluffy.__proto__.__proto__.__proto__) // null

Prototypical inheritance

function Animal(voice) {
  this.voice = voice || 'grunt'
}

Animal.prototype.speak = function() {
  console.log('Grunt')
}

function Cat(name, color) {
  //call parent constructor for parent related initialization
  Animal.call(this, 'Meow')
  this.name = name
  this.color = color
}

Cat.prototype = Object.create(Animal.prototype) // Link the prototypes
Cat.prototype.constructor = Cat // Necessary to correct prototype chain references

var fluffy = new Cat('Fluffy', 'White')

fluffy.speak() // 'Grunt'

Inheritance using new ES6 class syntax

We have some syntatic sugar that makes it cleaner to set up inheritance

class Animal {
  constructor(voice) {
    this.voice = voice || 'grunt'
  }

  speak() {
    console.log(this.voice)
  }
}

class Cat extends Animal {
  constructor(name, color) {
    super('Meow')
    this.name = name
    this.color = color
  }
}

var fluffy = new Cat('Fluffy', 'White')
fluffy.speak() // "Meow"

An important note is that the constructor object is a class not a constructor function on the object.

Also, members of classes are not enumerable by default. E.g., the speak function will not show up on the Object.keys or loop over the properties of the Animal class.

Closures in Javascript

2016-12-16T00:00:00+00:00

What is a closure?

A closure is when a function “remembers” its lexical scope even when the function is executed outside that lexical scope.

function makeAdder(x) {
  return function(y) {
    return x + y;
  };
}

var add5 = makeAdder(5);
var add10 = makeAdder(10);

console.log(add5(2));  // 7
console.log(add10(2)); // 12

add5 and add10 have both become closures because they share the same function body definition but store different environments and close over the x and y independently. The key mechanism at work here is the inner function is be transported out and returned.

Module pattern

A useful application of closures is the module pattern which can provide a means of encapsulation.

var foo = (function() {
  var o = { bar: "bar" };

  return {
    bar: function() {
      console.log(o.bar);
    }
  };
})();

foo.bar(); // "bar"
foo.o // "undefined" - akin to a private member

this keyword in Javascript

2016-12-15T00:00:00+00:00

The following notes are a resummarization from Kyle Simpson as a note to myself

What is this? Every function, while executing, has a reference to its current execution context

What determines which object a functions this points to? What’s the default?

Default and Implicit Binding rule

function foo() {
  console.log(this.bar);
}

var bar = "bar1";
var o2 = { bar: "bar2", foo: foo };
var o3 = { bar: "bar3", foo: foo };

//the call below is a normal call and uses default binding
//in strict mode this will be undefined
//in non strict mode it wil be the global
foo();    //"bar1"

//below we illustrate implicit binding
//we make a reference to o2 as the owner or containing object
//and use it to call the function, 'this' now points to o2's 'this' context
o2.foo(); //"bar2"
o3.foo(); //"bar3"

Using the new keyword

function foo() {
  this.baz = "baz";
  console.log(this.bar + " " + baz);
}

var baz = "bar";

// below we will get a brand new object with a new 'this' context
// also the object inherits from foo's prototype
// if there is no return statement, 'this' will be implicitly returned
var baz = new foo();

//the above will log "undefined bar"
//this is because this.bar is set on our new this context

Explicit Binding with a call, apply or bind(as of ES5)

function foo(a,b) {
  console.log(a + " " + b + " " + this.bar);
}

var bar = "bar1";
var obj = { bar: "bar2" };

foo(); // "bar1" due to default binding

//first argument is the owner object or 'this' context
foo.call(obj, "a", "b"); // "a b bar2" - explicitly use obj as this
foo.apply(obj, ["a", "b"]); // "a b bar2" - same except extra arguments in array
foo = foo.bind(obj, "a", "b");
foo(); // "a b bar2"  - note we have to explicitly call foo after binding

Double Dispatch

2016-09-14T00:00:00+00:00

How can you code a computation that has many cases, the cross product of two families of classes? In a language like Ruby that does not support method overloading, we would need to rely on a low level pattern known as double dispatch.

The example below illustrates the dependency on the class of the object(Doctor or Dentist) and the class of the input object(Adult or Child). If we don’t use double dispatch we get a nasty case of if or switch statements

#Not using double dispatch
class Doctor
  def work_on(patient)
    if patient.is_a? Child
      #patient.do_child_work
    elsif patient.is_a? Adult
      #patient.do_adult_work
      #elseif potential maintenance explosions here
    end
  end
end

class Dentist
  def work_on(patient)
    if patient.is_a? Child
      #patient.do_child_work
    elsif patient.is_a? Adult
      #patient.do_adult_work
      #elseif potential maintenance explosions here
    end
  end
end

Using double dispatch below, we can cleanly decouple the type of work to be done and get rid of type checking

#Using double dispatch
class Doctor
  def work_on(patient)
    patient.dispatch_work(self)
  end

  def work_on_adult(patient)
    do_checkup(patient)
  end

  def work_on_child(patient)
    assure_presence_of(patient.guardian)
    ask_questions_to(patient.guardian)
    do_checkup(patient)
    give_cheap_toy_to(patient)
  end

  private
  def do_checkup(patient)
    puts "Checking all your adult aparts"
  end
end

class Dentist
  def work_on(patient)
    patient.dispatch_work(self)
  end

  def work_on_adult(patient)
    drill_as_hard_as_you_can(patient)
  end

  def work_on_child(patient)
    use_bubble_gum_toothpaste(patient)
    give_toothbrush_to(patient)
  end

  private
  def drill_as_hard_as_you_can(patient)
    puts "Drilling very hard! on #{patient}"
  end

  def use_bubble_gum_toothpaste(patient)
    puts "Heard you like bubble gum?"
  end

  def give_toothbrush_to(patient)
    puts "Here's a free toothbrush!"
  end
end

class Adult
  def dispatch_work(doctor)
    puts "dispatch_work called on an adult"
    doctor.work_on_adult(self)
  end
end

class Child
  def dispatch_work(doctor)
    puts "dispatch_work called on an child"
    doctor.work_on_child(self)
  end
end

p1 = Adult.new
c1 = Child.new

d1 = Dentist.new
d1.work_on p1
d1.work_on c1

d2 = Doctor.new
d2.work_on p1

Addendum (3/7/2024): Understanding Double Dispatch

In software development, particularly in object-oriented programming, handling complex interactions between different types of objects can be challenging. One such scenario arises when we need to perform computations that involve multiple combinations of object types. In languages like Ruby, which lack native support for method overloading, we often rely on patterns like double dispatch to address these challenges.

Understanding Double Dispatch:

Double dispatch is a design pattern that allows us to dynamically dispatch method calls based on the types of two objects involved in a computation. It involves a two-step process:

First Dispatch: The method call is dispatched based on the type of the first object.
Second Dispatch: Within the method called in the first dispatch, another method call is dispatched based on the type of the second object.

This approach effectively decouples the logic for handling different combinations of object types, leading to cleaner and more maintainable code.

Possible Alternative:

Alternatively, we could refactor the code to use duck typing and interface extraction. This approach involves defining a Patient interface with a do_work method, which both Adult and Child classes implement. The Doctor and Dentist classes then directly call the do_work method on patient objects, eliminating the need for double dispatch.

# Define an interface for patients
module Patient
  def do_work(doctor)
    raise NotImplementedError, "This method must be implemented by subclasses"
  end
end

# Define classes for Adult and Child patients implementing the Patient interface
class Adult
  include Patient

  def do_work(doctor)
    doctor.work_on_adult(self)
  end
end

class Child
  include Patient

  def do_work(doctor)
    doctor.work_on_child(self)
  end
end

# Define classes for Doctor and Dentist
class Doctor
  def work_on_adult(adult)
    puts "Checking all your adult parts"
  end

  def work_on_child(child)
    assure_presence_of(child.guardian)
    ask_questions_to(child.guardian)
    puts "Checking all your child parts"
    give_cheap_toy_to(child)
  end

  private

  def assure_presence_of(guardian)
    # Logic to assure the presence of a guardian
  end

  def ask_questions_to(guardian)
    # Logic to ask questions to the guardian
  end

  def give_cheap_toy_to(child)
    puts "Here's a free toy for you!"
  end
end

class Dentist
  def work_on_adult(adult)
    puts "Drilling very hard!"
  end

  def work_on_child(child)
    puts "Using bubble gum toothpaste"
    puts "Here's a free toothbrush for you!"
  end
end

# Usage
p1 = Adult.new
c1 = Child.new

d1 = Dentist.new
d1.work_on p1
d1.work_on c1

d2 = Doctor.new
d2.work_on p1

This refactoring promotes a more flexible and maintainable design, aligning with the Interface Segregation Principle (ISP) of the SOLID principles.

In summary, double dispatch is a powerful design pattern for handling complex interactions between objects in object-oriented systems. By leveraging double dispatch effectively, developers can write cleaner, more maintainable code that is better suited to handle the intricacies of real-world applications.

Javascript click vs touch events

2016-03-05T00:00:00+00:00

If your application needs to respond to both mobile touches and desktop clicks you could write the following

function bind_button(selector, subselector) {
  $(selector).on('click', subselector, function(event) {
    handleEvent();
    return false;
  });

  if('touchstart' in window) {
    $(selector).on('touchstart', subselector, function(event) {
      handleEvent();
      return false;
    });
  }
}

Composed Method

2016-02-28T00:00:00+00:00

Definition: Breaking down a method into smaller intention revealing methods

Example:

def controlActivity
  controlInitialize
  controlLoop
  controlTerminate
end

The above code is very readable and communicates at the same abstraction level. We are guiding the structure of the system subtly by the naming and chunking.

In fact we can design our system top down and leave the implementation details for later.

A pleasant side effect of intention revealing methods within this composed method is that we hide the implementation details of methods making it more extensible and easier to modify. E.g.:

# original implementation
def controlInitialize(args)
  activity_name = args[:activity_name]
end

# adding a description
def controlInitialize(args)
  activity_name = args[:activity_name]
  description = args[:description]
end

We can also actually use Composed Method bottom-up to DRY up our code. I.e controlInitialize can now be called in multiple places and changed in one place.

If the above method was an overridable method in a superclass, we now only have 1 method we need to override vs large chunks of random code in the superclass.

Blogging on GitHub

2014-10-29T00:00:00+00:00

This is my new blog using Jekyll. It’s hosted using GitHub Pages and using the theme and layout from Poole for now.

The setup took less than 15 minutes, so I would consider this a pretty lightweight, fast, and free way to start blogging.

Sid Ngeth's Blog

the second opinion needs to read the code

the premise

the experiment

the flow

run 1: the reviewer can’t read the code

run 2: the reviewer gets equal tools

what changed

what the literature missed

what this means

experimental details

nanoclaw's memory architecture: sqlite + 200-message rolling window

what is nanoclaw

the memory question

architecture overview

the 200-message rolling window

cursor-based retrieval mechanism

accessing context beyond the window

1. conversation folder exports

2. grep tool

3. direct database queries

comparison to RAG-based systems

trade-offs and design decisions

why 200 messages?

why no embeddings?

why cursor-based?

why manual retrieval?

source citations

adding vim keybindings to opencode's tui

the request

what claude code’s vim mode actually supports

opencode’s TUI architecture

the four-layer architecture

layer 1: config

layer 2: VimEngine

layer 3: VimProvider

layer 4: prompt integration

cross-referencing against neovim’s source

character classification

three bugs found

full vim mode reference

mode switching

navigation (normal mode)

editing (normal mode)

text objects (used with d/c/y in normal mode)

not supported (matching claude code)

how to try it

running as a compiled binary (recommended)

files changed

more gotchas (found after shipping)

x silently fails on trailing spaces

every shifted command is silently broken

escape key routing during generation

opencode permission rules: protecting your code from ai agents

a terraform destroy that wiped production

the configuration

how it works

test results

“allow” commands – ran without prompting

test 1: git status

test 2: git diff

test 3: git log

test 4: git add

“ask” commands – prompted for approval

test 5: git commit

test 6: git stash

test 7: git restore

test 8: git checkout

test 9: git reset --hard

test 10: git worktree

“deny” command – blocked entirely

test 11: git clean -fd

why this matters for worktrees

commands that destroy work inside a worktree

worktree management commands that destroy work

shared state risks across worktrees

applying this to terraform

additional recommendations

important notes

server-side vs client-side tools in llm apis

`x` silently fails on trailing spaces

test 1: `git status`

test 2: `git diff`

test 3: `git log`

test 4: `git add`

test 5: `git commit`

test 6: `git stash`

test 7: `git restore`

test 8: `git checkout`

test 9: `git reset --hard`

test 10: `git worktree`

test 11: `git clean -fd`

roadblock 2: empty struct initialization `.{}`