I really love visual exploration. But there's a problem: given no constraints, it's so easy to get wrapped up into a one idea and go too far down that path.

Here's what happened: I found a cool trick to "warp" real DOM elements. I took this approach to the extreme and tried to build my whole site around it. Here was one early version:

I still really like this. It feels like my personal messy desk that that world can see.

But that's also a problem: it's too messy. The hardest part about design is executing it in a way that doesn't detract from the original need. It still needs to fulfill the original requirement.

In this case, the requirement is I should be able to quickly write content for you to read. I spent almost a week working on this before realizing this approach just fundamentally detracted from the experience too much. Maybe a better designer could have figured out how to execute this better.

For now, I ripped it out and implemented what you see in ONE HOUR. It's time to focus on content again.

I left my in-progress work up so you can see it on this page.

(I also wrote this up as a thread which has a few more details & videos)

This is my favorite part of the design by far. I made the demo videos into little polaroid pictures. These are real videos that are playing! They animate in like a paper would, and there's a nice paper-like interaction on hover, and you can drag them around:

(I wrote some notes when implementing this: https://jlongster.com/slicing-up-a-video)

I learned a ton throughout this process. The biggest thing was I used AI to create design tools on the fly. For example, I create a full animation keyframe editor that let me create keyframes of the warped DOM nodes, and then a runtime player plays these keyframes to make the animation:

I installed the perfect-freehand library and created a live animation drawing tool. You can press record, draw stuff on the screen, and stop recording and get an animation back. It automatically saves the animation to the real site:

The site has 3 tools available: paper, text, and animation tools. Paper lets you create and edit paper animations, even opening up the animation keyframe editor itself! Text just lets me position those text and arrows. And animation is the global animation coordinator that lets me change the delay and other properties, and retrigger animations to see them live:

This was one of the most fun things I've done recently, but alas, I just couldn't get it to work with the content in a satisfying way. I needed to timebox this to a week, so it's time to move on.

You can view this experiment live on this page.

^ef0fc3

I’m James Long, and I love building products. You might know me as the creator of Prettier, absurd-sql, Actual, or maybe something else.

I left Stripe in October and am currently looking for my next role (email me at longster@gmail.com). Please get in touch if I can help you!

About me

"Design engineer" is probably the best description of me. I love being creative and working closely with designers to build products, and bending web browsers to create great experiences. I'm more of an engineer than designer, but always looking to get better at design.

I obsess over ideas. There’s nothing better than creating something from nothing. Sometimes I can’t go to sleep until I’ve figured out a problem. I sweat the details in all of my work. I love working on foundational problems that require deep technical expertise. I’ve implemented a custom CRDT sync engine, built a SQL engine for the web, dissected the chromaticity diagram, and many other things.

• Location: Richmond, VA
• Online: Twitter/X

My work

Read more about my work over here.

Projects:

• Prettier
• absurd-sql
• Actual

Podcasts/talks:

• Discussion about local-first on localfirst.fm
• CRDTs for Mortals
• My History with Papers

Posts:

• A case study of complex table design
• Verlet integration demo

^865093

Resume

At Stripe, I helped build a design system from scratch used by most products today. I wrote the layering system, collections, and other systems from the ground up, and helped build a wide set of components to power Stripe. I also worked on fundamental pieces of the Dashboard. My last year there was spent building out Stripe's AI Assistant product.

I created a product Actual entirely by myself, with the goal of getting it bootstrapped. I acquired a decent amount of users, but eventually open-sourced it and left the project.

I've also worked for Mozilla on the Firefox developer tools and other projects.

Other work

Prettier

I created the Prettier JavaScript formatter alongside vjeux. I wrote the initial version, and Christopher got involved soon after and helped evolve it into the mainstream version you know today. I helped for about the first 9 months and moved to an advisory role after to focus on my consulting business.

absurd-sql

In 2021 I created the absurd-sql project. It was more of a proof-of-concept, but it worked well enough that it powered the web version of Actual in production. I released it to inspire others, but did not intend for it to become a big maintained project itself (due to my lack of time). To this day Actual uses it on the web.

absurd-sql was one of the early SQLite builds for the web. The unique feature was to hook into the filesystem calls and translate page reads and writes to IndexedDB for persistence. It even mapped transactional semantics correctly, which means you can read and write from multiple tabs at the same time. This is something that is still difficult today in most modern libraries.

IndexedDB is very slow, and one of the most surprising results from this was how much it blew away IDB in performance. You essentially get batching for free and it works so well. Read the post for more details.

Blog posts

UI design: My post A case study of complex table design is good example of how I think through complex UI problems.

React: I was an early React adopter and wrote many blog posts about it. One of them got very popular and supposedly was very influential in getting React adopted.

^110e6b

I've been working on a design and found a cool effect that would make a great bookmarklet. A bookmarklet is a JS snippet that you can drag to your bookmarks bar and run on any page.

If you've ever wanted to warp DOM elements on a page, you're dream has now come true. Here's what it looks like:

Just drag this to your bookmarks bar and click it on any page:

Drag this to your bookmarks bar

Video demo, including how to install it:

<p class="scroller">
  <video id="scrollVid"
    muted playsinline preload="auto" controls
    src="https://static.jlongster.com/20251119/output-web2.mp4">
  </video>
</p>

^855968

const video = document.getElementById("scrollVid");
const section = document.querySelector(".scroller");

// Optional: if you want the video visible only while the section is on screen:
const io = new IntersectionObserver(
  entries => {
    const onScreen = entries[0].isIntersecting;
    if (onScreen) {
      video.play();
    } else {
      video.pause();
    }
  },
  { threshold: 0 }
);
io.observe(section);

^b50209

Hope you like it!

The idea was to meld a live evaluation environment together with AI. By using QuickJS as an evaluator, we get more than just a sandbox: we are able to inspect bytecode of functions, we could pause execution as needed, etc.

I thought we might discover use cases that would be difficult to do by sending JS to eval off to a remote sandbox (or a separate node instance). It's also a UX experiment: you can either write code or talk to AI.

In the end, it didn't prove to be super useful. The cases where I want to provide deep introspection at runtime to AI are where I'm already working with a very complex codebase, not writing simple scripts. For the UX, I generally want to write code in a dedicated editor and send it off to be evaluated. I still think there's merit in the idea of sharing an executing environment with AI with the debugger hooks enabled; I'm researching companies working on this. Let me know if you know of any.

Still, it was fun to give AI the ability to get the bytecode for a function and describe what the function does at a very low-level. I asked to write a function that sums the numbers in an array, and then asked it to get the bytecode and describe it:

Some notable things that happened during this:

• QuickJS doesn't come with a WASM build by default. I told Claude Code to compile it to WebAssembly, and it got Emscripten all setup and successfully built it to WASM
• Sonnet 4.5 was able to very easily write a bunch of C code to provide new APIs in QuickJS
• Changing how error handling works, another example of how nice it was to get AI to work with the C code
  • I intended to eventually dig into QuickJS and change some fundamental features but decided to pause this experiment
• Adding tools is really fun. Here I added a tool to get the bytecode of a function
  • It took a while to get this working, skip to this part where I started testing it

Admittedly, I'm kind of stuck on Anthropic's models. I need to branch out and start testing other models now that I'm getting more familiar with how to test their behaviors.

I've thrown up the source for this on github: https://github.com/jlongster/code-agent.

Now that I don't have a job I finally have time to figure out a better workflow to manage life. I have a good workflow for writing notes and managing tasks in Bear, but my biggest weakness is staying on top of communication. What happens frequently is I'll have a backlog so big that I end up ignoring it, which is really bad as I miss out on opportunities or important emails.

We now have AI as an incredible tool. I was always too busy to sit down and learn how to leverage AI to transform my life; it's so early and this space is super chaotic. However, I finally have time to figure out new workflows that use AI.

My first attempt was to run my own instance of n8n on fly and setup some workflows for processing email. I got it working (and accidentally spammed my wife with multiple emails every hour, which she is still annoyed by), but I realized right now I don't need automations. I just need new tools to help manage my life.

Next, I turned to Claude Code. I use it all the time while writing code, and I knew there was a way to extend it. So I thought of a specific use case: after announcing I left my job, I received a lot of emails from interested companies. Since I'm still dealing with some burnout, it's been difficult to stay on top of. How can AI help us here?

I thought of a few questions:

• Which email threads should I follow-up on? Are there any important ones that I forgot to reply to?
• How can I more clearly see all these companies to help decide what my next step should be? I want to be able to clearly see a list of companies and info about them

The first step is to give Claude Code access to my gmail.

Writing a Gmail Skill to access email

I've used tool calling before only when interacting with an LLM via an API. How do I give Claude Code, a product, the ability to read my email?

It looked like MCP was the way to do it. Claude Code has an /mcp command to add an MCP server which would provide tools that it could call. However, I've never liked MCP. The amount of work that this would require (setting up and running a server, writing glue code to wire it all up, etc) felt insane. I just want to hit gmail directly from my local computer! Well, I don't want to, I just need a way to tell Claude to do so.

I could write a doc that describes how to do it, including where my OAuth credentials are stored for the Gmail API. Every time I open Claude, I could tell it to read this doc. That would probably work, but annoying to always have to load this into context.

As timing would have it, Claude just came out with Skills which is basically the doc idea, but more automated. A Skill is just a markdown doc with some accompanying scripts which provide new functionality. For Claude Code, all you have to do is put them in ~/.claude/skills and every single Claude Code session is automatically aware of them. So much simpler than MCP.

I created the folder ~/.claude/skills/gmail and added this SKILL.md:

---
name: gmail
description: Allows access to read and search the user's email in gmail
---

# gmail

## Instructions

Script available in the `scripts` folder to help query the user's email:

* `search_gmail.js`: Run this script when you need to retrieve a set of emails from the user's gmail account from a search filter. Simply pass the filter as the only argument.

## Examples

User: which emails have the filter "companies"?
Claude Code: <run `node search_gmail.js label:companies`>

^e7cd3f

Then I got Claude Code to write the search_gmail.js script for me which takes a filter expression and hits the Gmail API to fetch the emails.

I also had to setup a project and credentials in google cloud console, of course. Claude Code wrote a setup_auth.js script which ran me through the OAuth flow to grab my authorization code (using the "out of band" redirect flow). Once I had my credentials and authorization code, it was ready to cook:

Notice how it says The "gmail" skill is running, and it figured out how to invoke the script with the correct args to get my latest email.

What can we do?

Now that Claude Code knows how to fetch my email, let's ask it some interesting questions.

This is where AI really shines. It's amazing what it can do when you give it a few fundamental pieces. This is currently my favorite use of AI: connecting a lot of disparate data from various APIs together. Normally you have to write tons of boilerplate glue code to do this, but now AI can just figure it out.

I can see why the CRM space (and others) are going to be entirely reinvented because of this.

Watch this video (click to watch it in fullscreen). Here I'm asking it to get the latest 5 emails from someone, and then I'm asking how those differed from previous emails. It was smart enough to fetch the emails in the 6-10 range, compared them, and gave me results:

<section class="scroller">
  <video id="scrollVid"
    muted playsinline preload="auto" controls
    src="https://static.jlongster.com/20251114/output.mp4">
  </video>
</section>

^f4c38a

const video = document.getElementById("scrollVid");
const section = document.querySelector(".scroller");

// Optional: if you want the video visible only while the section is on screen:
const io = new IntersectionObserver(
  entries => {
    const onScreen = entries[0].isIntersecting;
    if (onScreen) {
      video.play();
    } else {
      video.pause();
    }
  },
  { threshold: 0 }
);
io.observe(section);

^51f353

The comparison is simplistic. You can argue that each individual step here isn't that mindblowing. But the fact that all I had to do was give this tool the ability to fetch emails, and then I can build up more complex questions and it all works is simply incredible.

My original asks

Let's revisit the questions I had at the beginning:

1. Which email threads should I follow-up on? Are there any important ones that I forgot to reply to?
2. How can I more clearly see all these companies to help decide what my next step should be? I want to be able to clearly see a list of companies and info about them

Let's start with the first one. The first attempt didn't quite work; it searched for emails starting with Re: and used that as a response indicator. While clever, it missed email threads that I had responded to but someone replied back; I wasn't the "last" person to email in the conversation.

To really get this to work, you need to reconstruct the "thread" from a list of emails. I had Claude Code add a new script to the Skill called find_unanswered.js which did this: it found email threads where I was not the latest person.

With this new functionality, Claude Code worked beautifully. I'm able to ask the question: "get all the emails with the "companies" label. which emails did I not reply to in the three weeks?" and it worked great. It figured out to use the after: filter and the date command to generate a date of 3 weeks ago:

We can now use this functionality to ask more and more questions and cross-reference other emails.

What about my second question? I want to compile info from all these emails and put them in a spreadsheet. I gave it a more complicated prompt to do this:

Go through all the emails with the "companies" label. These emails are discussions with companies that I might want to work for. Go through all of the emails and extract out all of the companies from them. Then, do some research online about each company. When finished, write a CSV file that I can import into a spreadsheet which contains a list of all of the companies, who I was talking to from that company, a summary of the email discussion, and additional information about the company that you found online

I'll spare you the full transcript (especially because it contains sensitive info) but it spent about 5 minutes, got all the emails, extracted out a list of companies, performed a bunch of web searches. Oh, it also needed to get full access to the email content (the list API doesn't give you that) so it wrote a script to do that and used it to fetch the full contents.

It compiled it all into a CSV file that I dumped into a spreadsheet. For obvious reasons I can't share a screenshot of the real spreadsheet, but I got Claude to generate fake data to show you what it looks like:

The more interesting columns that contain the results of the web research:

It's incredible how easy this kind of stuff is now. I know I'm late to the party, but if you're not thinking about how to rethink your workflows with AI you really should be.

What I learned

A few thoughts after experimenting with this for a day. Overall, I'm extremely happy with it.

Context is powerful

It's great that I can give AI a complex prompt (like the one above) and it one-shots it to completion. However, I find it even cooler that you can ask AI to pull in some data, and then continue to work with that data with follow-up questions. This works because the results of a tool call (or script, or API call, whatever gets the data) are kept in the context of a conversation.

This is not very token-efficient so it probably isn't good for large sets of data. But it just works so well that it's so convenient to just include it in the context. For example, I can get it to pull in 10 emails and then continually refine my questions about those emails and it all just works.

This reminds me of REPL-driven development, something much more popular in the ~late 2000s. In a way a REPL is a conversation with code. You type some code, press enter, it evaluates it, and shows you the result. Rinse and repeat. You can build up "context" by declaring variables. You can always inspect data from previous steps, and keep refining until you're satisfied.

The problem with REPLs is it becomes unwieldy for anything complex. You don't want to type a large function in a REPL, and it's easy to forget variables you declared earlier. The steps you got to a certain point aren't clear. Still, I always loved this style of development. Notebooks (like Jupyter) are essentially a modern-day REPL version of it.

This makes me want to build something in this space: imaging a REPL or notebook that allowed you to switch between natural language and code seamlessly, and they both share the same access to a serializable heap. Hmmm.

Skills are great

It's just so easy and incredible to drop a bunch of markdown files and scripts locally into my Claude Code instance that gives it superpowers. I can't wait to keep augmenting it.

One weird issue with skills

I ran into one strange behavior. I accidentally ran claude inside my gmail skill. Well, it was intentional to get it to write the skill but then I forget that it was running in that directory.

Getting it to use the skill inside the directory skill was weird. It seemed to work at first. In fact, I thought the Claude was smart enough to add a new script to the skill because it required new functionality. But it turns out that I was just running Claude in that directory, and it did it's normal thing of writing a script so it can run it. It wasn't intentionally trying to augment a skill.

Things started to fall apart though. I would change scripts inside the skill, or change the SKILL.md file to change how the skill worked, and Claude Code just seemed to flat out ignore it. For example, I asked it "get the latest 5 emails" and I realized it was fetching all 100 emails first (the max in my script) and then filtering which was slow. I added a --limit arg to the script, but no matter what I did, it always passed the arg as --max-results. I wrote out a huge section in SKILL.md to yell at it to never do that and use --limit, but it never obeyed.

I grepped and discovered that the gmail API calls the arg --max-results, so it's probably inferring it from the gmail API. But why could I not instruct it to do the right thing?

This happened with several other behaviors; it simply seemed to be ignoring my instructions.

Turns out you shouldn't expect skills to work if you are running Claude inside the skill itself. Once I ran Claude Code in a different directory everything worked great. For some reason, it seemed to ignore the SKILL.md when running inside the skill itself. In fact, now that I think about it, I never saw the message "The gmail skill is running" so I think the issue was that it wasn't running the skill it all. Because it saw local scripts that seemed to achieve the same thing, it chose to run those scripts directly instead of the skill. Sadly, it did a bad job inspecting and learning the right API to run those scripts.

Definitely not a big issue, and I'm still extremely happy with how this turned out.

^ddb372

<div class="demo-full-screen">
<div class="player">
  <video
    autoplay
    loop
    muted
    playsinline
    preload="auto"
    style="display: block; object-fit:contain; width: 100%; height: 100%"
  >
    <source src="https://static.jlongster.com/20250826/ball.mp4" type="video/mp4" />
  </video>

  <div class="video-box-fade"><div class="fade"></div></div>
</div>
</div>

^c098af

^0686c4

^d6f59f

^67fd6e

<div class="demo-full-screen">
    <div class="relative" style="width: 100%; height: 100%; max-width: 800px; max-height: 700px;">
      <canvas style="width: 100%; height: 100%; object-fit: contain"></canvas>
      <div class="demo-touch-area absolute" style="top: 25%; left: 25%; right: 25%; bottom: 25%; user-select: no-repeat"></div>
    </div>
  </div>

^043ea2

(function () {
  'use strict';

  var X = 0;
  var Y = 1;
  var Z = 2;
  var W = 3;

  function vec(x, y, z, w) {
    var arr;
    var len = w != undefined ? 4 : z != undefined ? 3 : y != undefined ? 2 : 1;

    if (typeof ArrayBuffer !== 'undefined') {
      var buffer = new ArrayBuffer(4 * len);
      arr = new Float32Array(buffer);
    } else {
      arr = [];
    }

    arr[X] = x;
    len >= 2 && (arr[Y] = y);
    len >= 3 && (arr[Z] = z);
    len == 4 && (arr[W] = w);
    return arr;
  }

  function vec_equals(v1, v2) {
    function fleq(x, y) {
      if (isNaN(x) && isNaN(y)) return true;

      return Math.abs(x - y) < 0.0000000001;
    }

    return (
      fleq(v1[X], v2[X]) &&
      fleq(v1[Y], v2[Y]) &&
      fleq(v1[Z], v2[Z]) &&
      fleq(v1[W], v2[W])
    );
  }

  function vec_copy(v1) {
    if (typeof Float32Array !== 'undefined' && v1 instanceof Float32Array) {
      var buffer = new ArrayBuffer(4 * v1.length);
      var arr = new Float32Array(buffer);
      arr[X] = v1[X];
      arr[Y] = v1[Y];
      arr[Z] = v1[Z];
      arr[W] = v1[W];
      return arr;
    }
    return Array.prototype.slice.call(v1);
  }

  function vec_pure_operation(op) {
    return function(v1, v2) {
      v1 = vec_copy(v1);
      op(v1, v2);
      return v1;
    };
  }

  function _vec_subtract(v1, v2) {
    v1[X] = v1[X] - v2[X];
    v1.length >= 2 && (v1[Y] = v1[Y] - v2[Y]);
    v1.length >= 3 && (v1[Z] = v1[Z] - v2[Z]);
    v1.length == 4 && (v1[W] = v1[W] - v2[W]);
  }
  var vec_subtract = vec_pure_operation(_vec_subtract);

  function _vec_multiply(v1, v2) {
    v1[X] = v1[X] * v2[X];
    v1.length >= 2 && (v1[Y] = v1[Y] * v2[Y]);
    v1.length >= 3 && (v1[Z] = v1[Z] * v2[Z]);
    v1.length == 4 && (v1[W] = v1[W] * v2[W]);
  }

  function _vec_add(v1, v2) {
    (v1[X] = v1[X] + v2[X]), v1.length >= 2 && (v1[Y] = v1[Y] + v2[Y]);
    v1.length >= 3 && (v1[Z] = v1[Z] + v2[Z]);
    v1.length == 4 && (v1[W] = v1[W] + v2[W]);
  }
  var vec_add = vec_pure_operation(_vec_add);

  function vec_dot(v1, v2) {
    return (
      v1[X] * v2[X] +
      (v1.length >= 2 ? v1[Y] * v2[Y] : 0) +
      (v1.length >= 3 ? v1[Z] * v2[Z] : 0) +
      (v1.length == 4 ? v1[W] * v2[W] : 0)
    );
  }

  function _vec_cross(v1, v2) {
    if (v1.length < 3) return;

    var x = v1[Y] * v2[Z] - v1[Z] * v2[Y];
    var y = v1[Z] * v2[X] - v1[X] * v2[Z];
    var z = v1[X] * v2[Y] - v1[Y] * v2[X];
    v1[X] = x;
    v1[Y] = y;
    v1[Z] = z;
  }
  var vec_cross = vec_pure_operation(_vec_cross);

  function vec_length(v1) {
    return Math.sqrt(
      v1[X] * v1[X] +
        v1[Y] * v1[Y] +
        (v1.length >= 3 ? v1[Z] * v1[Z] : 0) +
        (v1.length == 4 ? v1[W] * v1[W] : 0)
    );
  }

  function _vec_3drotateX(v1, angle) {
    var y = v1[Y] * Math.cos(angle) - v1[Z] * Math.sin(angle);
    var z = v1[Y] * Math.sin(angle) + v1[Z] * Math.cos(angle);
    v1[Y] = y;
    v1[Z] = z;
  }
  var vec_3drotateX = vec_pure_operation(_vec_3drotateX);

  function _vec_3drotateY(v1, angle) {
    var x = v1[Z] * Math.sin(angle) + v1[X] * Math.cos(angle);
    var z = v1[Z] * Math.cos(angle) - v1[X] * Math.sin(angle);
    v1[X] = x;
    v1[Z] = z;
  }
  var vec_3drotateY = vec_pure_operation(_vec_3drotateY);

  function _vec_3drotateZ(v1, angle) {
    var x = v1[X] * Math.cos(angle) - v1[Y] * Math.sin(angle);
    var y = v1[X] * Math.sin(angle) + v1[Y] * Math.cos(angle);
    v1[X] = x;
    v1[Y] = y;
  }
  var vec_3drotateZ = vec_pure_operation(_vec_3drotateZ);

  function _vec_unit(v1) {
    var len = vec_length(v1);
    v1[X] = v1[X] / len;
    v1[Y] = v1[Y] / len;
    v1[Z] = v1[Z] / len;
  }
  var vec_unit = vec_pure_operation(_vec_unit);

  // tests

  function assert(msg, exp) {
    if (!exp) throw '[FAILED] ' + msg;
  }

  function assert_equal(msg, v1, v2) {
    if (typeof v1 == 'object') {
      assert(msg + ' ' + v1 + ' ' + v2, vec_equals(v1, v2));
    } else {
      assert(msg + ' ' + v1 + ' ' + v2, v1 == v2);
    }
  }

  var x = vec(4, 5, 6);
  var y;
  var z = vec(2, 3, 4);

  assert_equal('vec_subtract', vec_subtract(x, z), vec(2, 2, 2));
  assert_equal('vec_add', vec_add(x, z), vec(6, 8, 10));

  x = vec(0, 1, 0);
  y = vec(1, 0, 0);
  assert_equal('vec_dot', vec_dot(x, y), 0);

  x = vec(-1, 0, 0);
  assert_equal('vec_dot', vec_dot(x, y), -1);

  x = vec(1, 1, 0);
  y = vec(1.5, 1, 0);
  assert_equal('vec_dot', vec_dot(x, y), 2.5);

  x = vec(0, 1, 0);
  y = vec(1, 0, 0);
  assert_equal('vec_cross', vec_cross(x, y), vec(0, 0, -1));

  assert_equal('vec_3drotateX', vec_3drotateX(x, Math.PI / 2.0), vec(0, 0, 1));
  assert_equal('vec_3drotateY', vec_3drotateY(y, Math.PI / 2.0), vec(0, 0, -1));
  assert_equal('vec_3drotateZ', vec_3drotateZ(x, Math.PI / 2.0), vec(-1, 0, 0));

  var LEFT = 2;
  var RIGHT = 3;

  function make_heap() {
    return [null, null, null, null];
  }

  function heap_add(heap, line) {
    var z = (line[0][Z] + line[1][Z]) / 2.0;
    _heap_insert(heap, line, z);
  }

  function _heap_insert(heap, line, z) {
    if (!heap[0]) {
      heap[0] = line;
      heap[1] = z;
    } else if (z > heap[1]) {
      if (!heap[LEFT]) {
        heap[LEFT] = [line, z, null, null];
      } else {
        _heap_insert(heap[LEFT], line, z);
      }
    } else {
      if (!heap[RIGHT]) {
        heap[RIGHT] = [line, z, null, null];
      } else {
        _heap_insert(heap[RIGHT], line, z);
      }
    }
  }

  function heap_depth_first(heap, func) {
    if (heap[LEFT]) {
      heap_depth_first(heap[LEFT], func);
    }

    if (heap[0]) {
      func(heap[0]);
    }

    if (heap[RIGHT]) {
      heap_depth_first(heap[RIGHT], func);
    }
  }

  let canvas = document.querySelector('canvas');
  let size = [800, 700];
  canvas.width = size[0] * 2;
  canvas.height = size[1] * 2;
  // canvas.style.width = size[0] + 'px';
  // canvas.style.height = size[1] + 'px';
  let ctx = canvas.getContext('2d');
  ctx.scale(2, 2);

  let camera = vec(0, 0, -1);
  let frustum = make_frustum(60.0, size[0] / size[1], 1.0, 1000.0);
  let currentColor = 'red';

  let codeString = `(define (shift* f) (let* ((parent-denv (vector-ref *meta-continuation* 2)) (curr-denv (current-dynamic-env)) (diff-denv (dynamic-env-sub curr-denv parent-denv))) (let ((v (call/cc (lambda (k) (abort-env* (lambda () (f (lambda (v) (reset (k v)))))))))) (current-dynamic-env-set! (dynamic-env-add diff-denv (vector-ref *meta-continuation* 2))) v))) (define (reset* thunk) (let ((mc *meta-continuation*) (denv (current-dynamic-env))) (continuation-capture (lambda (k-pure) (current-dynamic-wind-set! ##initial-dynwind) (abort-pure* ((call/cc (lambda (k) (set! *meta-continuation* (make-vector-values (lambda (v) (set! *meta-continuation* mc) (current-dynamic-env-set! denv) (##continuation-return-no-winding k-pure v)) k denv)) thunk))))))))`;

  function getCodeIndex(i) {
    // Walk forwards until it's not pointing to a space
    while (codeString[i] === ' ') {
      i++;
    }
    return i;
  }

  let meshes = [
    // lines
    {
      yaw: 0,
      pitch: Math.PI / 2,
      translate: vec(0, 0, 100),
      scale: vec(7, 7, 7),
      color: '#00ff00',
      data: [
        [[1.490947, 2.589018, 4.14739], [1.490947, 2.570464, -0.952927]],
        [[0.82669, 0.939331, 3.573287], [0.82669, 0.913141, -2.007562]],
        [[1.987827, -0.359111, 5.698228], [1.987827, -0.405375, -4.160209]]
      ]
    },

    // computer
    {
      yaw: 0,
      pitch: Math.PI / 4,
      translate: vec(0, 0, 100),
      scale: vec(7, 7, 7),
      color: '#FF9B9B',
      data: [
        [[4.146833, -2.017568, 3.361326], [-0.369471, -2.052729, 3.361326]],
        [[-0.369471, -2.052729, 3.361326], [-0.369471, -2.052729, -2.481019]],
        [[-0.369471, -2.052729, -2.481019], [4.146832, -2.017568, -2.481019]],
        [[4.146832, -2.017568, -2.481019], [4.146833, -2.017568, 3.361326]],
        [[-0.378355, -2.046666, 3.320174], [-2.182173, 2.093923, 3.320174]],
        [[-2.182173, 2.093923, 3.320174], [-2.182173, 2.093923, -2.522171]],
        [[-2.182173, 2.093923, -2.522171], [-0.378356, -2.046666, -2.522171]],
        [[-0.378356, -2.046666, -2.522171], [-0.378355, -2.046666, 3.320174]],
        [[0.080366, -0.902821, 2.749107], [-1.056806, 1.70751, 2.749107]],
        [[-1.056806, 1.70751, 2.749107], [-1.056806, 1.70751, -2.03474]],
        [[-1.056806, 1.70751, -2.03474], [0.080366, -0.902821, -2.03474]],
        [[0.080366, -0.902821, -2.03474], [0.080366, -0.902821, 2.749107]]
      ]
    },
    // curve
    {
      yaw: 0,
      pitch: Math.PI / 4,
      translate: vec(0, 0, 100),
      scale: vec(0, 0, 0),
      color: '#f0a0f0',
      data: [
        [[0, -3.733048, 2.712219], [0, -1.425897, 4.388462]],
        [[0, -1.425897, 4.388462], [0, 1.425898, 4.388462]],
        [[0, 1.425898, 4.388462], [0, 3.733049, 2.712218]],
        [[-1.917828, -3.733048, 1.917828], [-3.103111, -1.425897, 3.103111]],
        [[-3.103111, -1.425897, 3.103111], [-3.103111, 1.425898, 3.103111]],
        [[-3.103111, 1.425898, 3.103111], [-1.917828, 3.733049, 1.917828]],
        [[0, 3.733049, 2.712218], [-1.917828, 3.733049, 1.917828]],
        [[-3.103111, 1.425898, 3.103111], [0, 1.425898, 4.388462]],
        [[0, -1.425897, 4.388462], [-3.103111, -1.425897, 3.103111]],
        [[-1.917828, -3.733048, 1.917828], [0, -3.733048, 2.712219]],
        [[-2.712219, -3.733048, 0], [-4.388463, -1.425897, 0]],
        [[-4.388463, -1.425897, 0], [-4.388462, 1.425898, 0]],
        [[-4.388462, 1.425898, 0], [-2.712219, 3.733049, 0]],
        [[-2.712219, 3.733049, 0], [0, 4.614302, 0]],
        [[-2.712219, 3.733049, 0], [-1.917828, 3.733049, 1.917828]],
        [[-3.103111, 1.425898, 3.103111], [-4.388462, 1.425898, 0]],
        [[-4.388463, -1.425897, 0], [-3.103111, -1.425897, 3.103111]],
        [[-1.917828, -3.733048, 1.917828], [-2.712219, -3.733048, 0]],
        [[-1.917828, -3.733048, -1.917829], [-3.103111, -1.425897, -3.103112]],
        [[-3.103111, -1.425897, -3.103112], [-3.103111, 1.425898, -3.103111]],
        [[-3.103111, 1.425898, -3.103111], [-1.917828, 3.733049, -1.917828]],
        [[-1.917828, 3.733049, -1.917828], [-2.712219, 3.733049, 0]],
        [[-4.388462, 1.425898, 0], [-3.103111, 1.425898, -3.103111]],
        [[-3.103111, -1.425897, -3.103112], [-4.388463, -1.425897, 0]],
        [[-2.712219, -3.733048, 0], [-1.917828, -3.733048, -1.917829]],
        [[0, -3.733048, -2.712219], [0, -1.425897, -4.388462]],
        [[0, -1.425897, -4.388462], [0, 1.425898, -4.388462]],
        [[0, 1.425898, -4.388462], [0, 3.733049, -2.712219]],
        [[0, 3.733049, -2.712219], [-1.917828, 3.733049, -1.917828]],
        [[-3.103111, 1.425898, -3.103111], [0, 1.425898, -4.388462]],
        [[0, -1.425897, -4.388462], [-3.103111, -1.425897, -3.103112]],
        [[-1.917828, -3.733048, -1.917829], [0, -3.733048, -2.712219]],
        [[0, -4.614302, -0.000001], [1.917828, -3.733048, -1.917829]],
        [[1.917828, -3.733048, -1.917829], [3.103111, -1.425897, -3.103111]],
        [[3.103111, -1.425897, -3.103111], [3.103111, 1.425898, -3.103111]],
        [[3.103111, 1.425898, -3.103111], [1.917828, 3.733049, -1.917828]],
        [[1.917828, 3.733049, -1.917828], [0, 3.733049, -2.712219]],
        [[0, 1.425898, -4.388462], [3.103111, 1.425898, -3.103111]],
        [[3.103111, -1.425897, -3.103111], [0, -1.425897, -4.388462]],
        [[0, -3.733048, -2.712219], [1.917828, -3.733048, -1.917829]],
        [[2.712219, -3.733048, 0], [4.388462, -1.425897, 0]],
        [[4.388462, -1.425897, 0], [4.388462, 1.425898, 0]],
        [[4.388462, 1.425898, 0], [2.712218, 3.733049, 0]],
        [[2.712218, 3.733049, 0], [1.917828, 3.733049, -1.917828]],
        [[3.103111, 1.425898, -3.103111], [4.388462, 1.425898, 0]],
        [[4.388462, -1.425897, 0], [3.103111, -1.425897, -3.103111]],
        [[1.917828, -3.733048, -1.917829], [2.712219, -3.733048, 0]],
        [[1.917828, -3.733048, 1.917828], [3.103111, -1.425897, 3.103111]],
        [[3.103111, -1.425897, 3.103111], [3.103111, 1.425898, 3.103111]],
        [[3.103111, 1.425898, 3.103111], [1.917828, 3.733049, 1.917827]],
        [[1.917828, 3.733049, 1.917827], [2.712218, 3.733049, 0]],
        [[4.388462, 1.425898, 0], [3.103111, 1.425898, 3.103111]],
        [[3.103111, -1.425897, 3.103111], [4.388462, -1.425897, 0]],
        [[2.712219, -3.733048, 0], [1.917828, -3.733048, 1.917828]],
        [[0, -4.614302, -0.000001], [0, -3.733048, 2.712219]],
        [[0, 3.733049, 2.712218], [0, 4.614302, 0]],
        [[0, -4.614302, -0.000001], [-1.917828, -3.733048, 1.917828]],
        [[-1.917828, 3.733049, 1.917828], [0, 4.614302, 0]],
        [[0, -4.614302, -0.000001], [-2.712219, -3.733048, 0]],
        [[0, -4.614302, -0.000001], [-1.917828, -3.733048, -1.917829]],
        [[-1.917828, 3.733049, -1.917828], [0, 4.614302, 0]],
        [[0, -4.614302, -0.000001], [0, -3.733048, -2.712219]],
        [[0, 3.733049, -2.712219], [0, 4.614302, 0]],
        [[1.917828, 3.733049, -1.917828], [0, 4.614302, 0]],
        [[0, -4.614302, -0.000001], [2.712219, -3.733048, 0]],
        [[2.712218, 3.733049, 0], [0, 4.614302, 0]],
        [[0, -4.614302, -0.000001], [1.917828, -3.733048, 1.917828]],
        [[1.917828, 3.733049, 1.917827], [0, 4.614302, 0]],
        [[0, 3.733049, 2.712218], [1.917828, 3.733049, 1.917827]],
        [[3.103111, 1.425898, 3.103111], [0, 1.425898, 4.388462]],
        [[0, -1.425897, 4.388462], [3.103111, -1.425897, 3.103111]],
        [[1.917828, -3.733048, 1.917828], [0, -3.733048, 2.712219]]
      ]
    }
  ];

  let explodeStarted = null;
  let explodeEnded = null;
  let touchArea = document.querySelector('.demo-touch-area');

  function startExplode(e) {
    explodeStarted = Date.now();
    explodeEnded = null;

    let curves = meshes[2];
    if (curves.opacity === 0) {
      curves.scale = [0, 0, 0];
    }

    meshes.forEach(mesh => {
      mesh.originalScale = mesh.scale;
      mesh.originalOpacity = mesh.opacity;
    });
  }

  function stopExplode() {
    explodeStarted = null;
    explodeEnded = Date.now();
    meshes.forEach(mesh => {
      mesh.originalScale = mesh.scale;
      mesh.originalOpacity = mesh.opacity;
    });
  }

  touchArea.addEventListener('mouseenter', startExplode);
  touchArea.addEventListener('mouseleave', stopExplode);
  touchArea.addEventListener('touchstart', startExplode);

  // Newer iOS phones have sucky tendency to bring up a bottom tab bar
  // but still think that 100vh means you want to go underneath it. This
  // is stupid. window.innerHeight is correct so set it to that, and we
  // have to do it a little in the future because it's racy.
  if (window.innerWidth < 500) {
    setTimeout(() => {
      document.querySelector('.demo-full-screen').style.height =
        window.innerHeight + 'px';
    }, 100);
  }

  let resumeTimeout;
  let animationFrame;
  window.addEventListener('resize', () => {
    clearTimeout(resumeTimeout);
    cancelAnimationFrame(animationFrame);

    resumeTimeout = setTimeout(() => {
      let width = window.innerWidth;

      frame(0, 0, ctx);
    }, 250);
  });

  frame(0, 0, ctx);

  function make_frustum(fovy, aspect, znear, zfar) {
    var range = znear * Math.tan((fovy * Math.PI) / 360.0);

    return {
      xmax: range,
      xmin: -range,
      ymax: range / aspect,
      ymin: -range / aspect,
      znear: znear,
      zfar: zfar
    };
  }

  function project2d(points, frustum) {
    function proj(point, frustum) {
      var x = point[X] / point[Z];
      var y = point[Y] / point[Z];

      x = (frustum.xmax - x) / (frustum.xmax - frustum.xmin);
      y = (frustum.ymax - y) / (frustum.ymax - frustum.ymin);

      return vec(x * size[0], y * size[1]);
    }

    return [proj(points[0], frustum), proj(points[1], frustum)];
  }

  function _transform_points(mesh, points) {
    var p = [vec_copy(points[0]), vec_copy(points[1])];

    if (mesh.scale) {
      _line_apply(p, function(v) {
        _vec_multiply(v, mesh.scale);
      });
    }

    if (mesh.yaw) {
      _line_apply(p, function(v) {
        _vec_3drotateX(v, mesh.yaw);
      });
    }

    if (mesh.pitch) {
      _line_apply(p, function(v) {
        _vec_3drotateY(v, mesh.pitch);
      });
    }

    if (mesh.roll) {
      _line_apply(p, function(v) {
        _vec_3drotateZ(v, mesh.roll);
      });
    }

    if (mesh.translate) {
      _line_apply(p, function(v) {
        _vec_add(v, mesh.translate);
      });
    }

    return p;
  }

  function _line_apply(tri, transform) {
    transform(tri[0]);
    transform(tri[1]);
  }

  function frame(time, lastTime, ctx) {
    update(time - lastTime);
    render(ctx);

    animationFrame = requestAnimationFrame(newTime => {
      frame(newTime, time, ctx);
    });
  }

  function x$1(progress) {
    let damping = 10.0;
    let mass = 1.0;
    let stiffness = 100.0;
    let velocity = 0.0;

    let beta = damping / (2 * mass);
    let omega0 = Math.sqrt(stiffness / mass);
    let omega1 = Math.sqrt(omega0 * omega0 - beta * beta);
    let omega2 = Math.sqrt(beta * beta - omega0 * omega0);

    let x0 = -1;

    let oscillation;

    if (beta < omega0) {
      // Underdamped
      oscillation = t => {
        let envelope = Math.exp(-beta * t);

        let part2 = x0 * Math.cos(omega1 * t);
        let part3 = ((beta * x0 + velocity) / omega1) * Math.sin(omega1 * t);
        return -x0 + envelope * (part2 + part3);
      };
    } else if (beta == omega0) {
      // Critically damped
      oscillation = t => {
        let envelope = Math.exp(-beta * t);
        return -x0 + envelope * (x0 + (beta * x0 + velocity) * t);
      };
    } else {
      // Overdamped
      oscillation = t => {
        let envelope = Math.exp(-beta * t);
        let part2 = x0 * Math.cosh(omega2 * t);
        let part3 = ((beta * x0 + velocity) / omega2) * Math.sinh(omega2 * t);
        return -x0 + envelope * (part2 + part3);
      };
    }

    return oscillation(progress);
  }

  function spring(v1, v2, progress) {
    return (v2 - v1) * x$1(progress) + v1;
  }

  function lerp(v1, v2, progress, func) {
    return (v2 - v1) * (func ? func(progress) : progress) + v1;
  }

  function update(dt) {
    let computer = meshes[1];
    let lines = meshes[0];
    let curve = meshes[2];

    computer.pitch += 0.0001 * dt;
    lines.pitch += 0.00025 * dt;
    curve.pitch += 0.0005 * dt;

    lines.data[0].codeIndex = 0;
    lines.data[1].codeIndex = getCodeIndex(25);

    if (explodeStarted) {
      meshes.forEach(mesh => {
        if (mesh === meshes[2]) {
          let d = Math.min((Date.now() - explodeStarted) / 500, 1);

          if (mesh.originalOpacity > 0) {
            mesh.opacity = lerp(mesh.originalOpacity, 1, d);
          } else {
            mesh.opacity = 1;
          }

          mesh.scale = [
            spring(mesh.originalScale[0], 8.8, d),
            spring(mesh.originalScale[1], 8.8, d),
            spring(mesh.originalScale[2], 8.8, d)
          ];
        } else if (mesh.id !== 'box') {
          let d = Math.min((Date.now() - explodeStarted) / 1500, 1);
          mesh.scale = [
            spring(mesh.originalScale[0], 4.5, d),
            spring(mesh.originalScale[1], 4.5, d),
            spring(mesh.originalScale[2], 4.5, d)
          ];
        }
      });
    } else if (explodeEnded) {

      meshes.forEach(mesh => {
        if (mesh === meshes[2]) {
          let dOpacity = Math.min((Date.now() - explodeEnded) / 500, 1);
          mesh.opacity = lerp(mesh.originalOpacity, 0, dOpacity);
        } else if (mesh.id !== 'box') {
          let d = Math.min((Date.now() - explodeEnded) / 1500, 1);
          mesh.scale = [
            spring(mesh.originalScale[0], 7, d),
            spring(mesh.originalScale[1], 7, d),
            spring(mesh.originalScale[2], 7, d)
          ];
        }
      });
    }
  }

  function render(ctx) {
    let heap = make_heap();

    ctx.clearRect(0, 0, size[0], size[1]);

    for (var i = 0; i < meshes.length; i++) {
      if (!meshes[i].hidden) {
        renderMesh(meshes[i], heap);
      }
    }

    heap_depth_first(heap, function(line) {
      render3d(ctx, line, camera, frustum);
    });
  }

  function renderMesh(mesh, heap) {
    let { data } = mesh;
    for (var i = 0; i < data.length; i++) {
      if (data[i].codeIndex == null) {
        data[i].codeIndex = getCodeIndex(
          (Math.random() * (codeString.length - 15)) | 0
        );
      }

      var line = _transform_points(mesh, data[i]);

      _line_apply(line, function(v) {
        _vec_subtract(v, camera);
      });

      line.color = mesh.color || currentColor;
      line.opacity = mesh.opacity != null ? mesh.opacity : 1;
      line.codeIndex = data[i].codeIndex;
      line.meshId = mesh.id;

      heap_add(heap, line);
    }
  }

  function render3d(ctx, points, camera, frustum) {
    var p_camera = [
      vec_subtract(points[0], camera),
      vec_subtract(points[1], camera)
    ];

    // var tri_ca = vec_subtract(p_camera[2], p_camera[0]);
    // var tri_cb = vec_subtract(p_camera[2], p_camera[1]);

    // var normal_camera = vec_cross(tri_ca, tri_cb);
    // var angle = vec_dot(p_camera[0], normal_camera);

    // // don't render back faces of triangles
    // if (angle >= 0) {
    //   return;
    // }

    // lighting
    // var p_ba = vec_subtract(points[1], points[0]);
    // var p_ca = vec_subtract(points[2], points[0]);
    // var normal = vec_unit(vec_cross(p_ba, p_ca));

    var color = points.color;

    // var angle = vec_dot(normal, light);
    // var ambient = 0.3;
    // var shade = Math.min(1.0, Math.max(0.0, angle));
    // shade = Math.min(1.0, shade + ambient);

    var points2d = project2d(p_camera, frustum);

    if (points.meshId === 'box') {
      render2d(ctx, points2d, color);
    } else {
      render2dWords(ctx, points2d, color, points.opacity, points.codeIndex);
    }
  }

  function render2d(ctx, points, color) {
    ctx.beginPath();
    ctx.moveTo(points[0][X], points[0][Y]);
    ctx.lineTo(points[1][X], points[1][Y]);
    ctx.lineWidth = 1;
    ctx.strokeStyle = color;
    ctx.stroke();
  }

  function render2dWords(ctx, points, color, opacity, codeIndex) {
    // ctx.lineWidth = 20;

    let line = vec_subtract(points[1], points[0]);
    let length = vec_length(line);
    let v = vec_unit(line);
    // Invert the Y, positive should be going up
    v[Y] = -v[Y];
    let axis = vec(1, 0);

    let angle = Math.acos(vec_dot(v, axis));
    if (v[Y] > 0) {
      angle = -angle;
    }

    if (opacity === 1) {
      ctx.save();
      ctx.globalCompositeOperation = 'destination-out';
      ctx.beginPath();
      ctx.translate(points[0][X], points[0][Y]);
      ctx.rotate(angle);
      ctx.translate(5, -7);
      ctx.moveTo(0, 0);
      ctx.lineTo(length - 15, 0);
      ctx.lineWidth = 18;
      ctx.strokeStyle = 'black';
      ctx.stroke();
      ctx.restore();
    }

    if (opacity > 0) {
      ctx.save();
      ctx.translate(points[0][X], points[0][Y]);
      ctx.rotate(angle);
      ctx.fillStyle = color;
      ctx.globalAlpha = opacity;
      ctx.font = '16px monaco';
      ctx.fillText(codeString.slice(codeIndex, codeIndex + length / 10), 0, 0);
      ctx.restore();
    }

    // Debug lines
    // ctx.beginPath();
    // ctx.moveTo(points[0][X], points[0][Y]);
    // ctx.lineTo(points[1][X], points[1][Y]);
    // ctx.lineWidth = 1;
    // ctx.strokeStyle = color;
    // ctx.stroke();
  }

}());

^eb9fe9

^96c620

<div id="demo-mount"></div>

^f31aec

    <div class="sizer">
      <input name="size" type="range" min="100" max="350" value="100" />
      <div class="output">32 x 33</div>
    </div>

    <a class="info">?</a>

    <div class="info closed">
      <p>
        Drag the cloth around with the left mouse button, and tear it with the right button.
      </p>
      <p>
        This was written
        with <a href="https://github.com/jlongster/LLJS">my fork</a>
        of <a href="http://lljs.org/">LLJS</a> that compiles
        to <a href="http://asmjs.org/">asm.js</a>.
        Get <a href="https://github.com/jlongster/lljs-cloth">the
        code</a>, and see more demos at <a href="https://jlongster.com">my site</a>. <span>- James Long</span>
      </p>
    </div>
    
    <script src="https://archive.jlongster.com/lljs-cloth/stats.min.js"></script>
    <script src="https://archive.jlongster.com/lljs-cloth/gl-matrix.js"></script>
    <script src="https://archive.jlongster.com/lljs-cloth/gl-renderer.js"></script>
    <script src="https://archive.jlongster.com/lljs-cloth/canvas-renderer.js"></script>
    <script src="https://archive.jlongster.com/lljs-cloth/verlet-run.js"></script>
    <script src="https://archive.jlongster.com/lljs-cloth/verlet.js"></script>

^be0572

In JavaScript we have two APIs to work with weak references: WeakMap and WeakRef. (Before I wrote this article I thought WeakRef was only a proposal, but it turns out most browsers have already implemented it)

One of the more common use cases uses WeakMap. This data structure keeps a weak reference to the keys in the map, and a strong reference between the keys and values. I guess I should explain what a "weak reference" is: usually if you have a reference to an object in a variable, it stops the garbage collector from deleting it (which makes sense). It'd be weird if suddenly your variable pointed to nothing right?

A "weak reference" doesn't stop the object from being garbage collected. Usually languages' semantics don't allow a variable to change in the middle of execution however:

// this doesn't exist, but what if we could create a weak reference like this?
let weak trans = new Transaction();

// do a bunch of things...

// error! trans is... nothing? it got garbage collected
trans.transfer();

^3d7637

Wouldn't it be weird if trans changed in the middle of execution? If that was possible all bets would be off for everything in the entire program because these weak references could be passed anywhere.

Instead, APIs for weak references force you to call a function to get the value. The WeakRef class has a deref method to get the object.

const ref = new WeakRef(obj)

// get the object
ref.deref()

// do a bunch of things...

ref.deref()

^ae238d

In the above example, if nothing has a reference to obj, it'll eventually get garbage collected. That means deref might return obj the first time, but the second time it might return undefined.

Let's get back to WeakMap which is the more common usage of weak references. (Direct weak references have a lot of weird behaviors and they should be considered very low-level.) A WeakMap has the same API as Map, but it holds a weak reference to the keys:

const map = new WeakMap();

map.set(obj, new Thing());

// later on
const thing = map.get(obj)

^dba43e

This map will not retain obj. Once obj gets garbage collected, it will no longer be in the map. Note that there is a strong reference to the value thing, meaning that as long as obj exists, so will thing. However, because obj is weakly referenced, once it gets garbage collected it no longer references thing in the map. That means the strong reference to the values will never result in memory leaks. It only means that thing will be alive as long as obj is.

This brings me to my favorite use of weak references: subverting control of abstractions.

Let's say you have a Transaction class:

class Transaction {
  // implementation
}

^e66bd4

Now let's say we have a function that builds a textual representation of a transaction. To do this, it fetches a bunch of extra information from a remote server.

async function describe(transaction) {
  const customer = await fetchCustomer(...)
  const conversions = await fetchCurrencyConversions(...)
  // etc

  // The rest of the implementation sets `description`

  return description;
}

^718ba0

This brings up a lot of interesting questions about the shape of your abstractions. You probably don't want to re-fetch all of these details every single time you call describe. Even if you are sure describe is only called once, what will happen is at some point in the future you will say "oh shoot, I want to call that again".

So we need to figure out how to avoid all the refetching work. One solution is to have a caching layer that owns all of that work. You'd delegate it to a network abstraction that ensures everything gets cached.

That forces the describe function to buy-in to your entire networking layer though. Maybe that's not possible for any number of reasons: this is an experimental feature and the backend APIs don't work well with the frontend caching yet, or this describe is used in other products that don't have the same networking layer. Regardless, maybe you are doing a lot of CPU-intensive work and you simply cache the result.

Using WeakMap you can cache the result based off of transaction:

const CACHE = new WeakMap();

async function describe(transaction) {
  let cached = CACHE.get(transaction);
  if(cached) {
    return cached;
  }

  // Not cached, do all the work...

  CACHE.set(transaction, description);
  return description;
}

^2494ea

What's cool about this is you never have to worry about memory leaks. The CACHE map will never accidentally keep growing with references to object we don't need anymore. When transaction gets GCed it simply won't be in the map anymore.

You might recognize this as simple memoization. This is only one simple use case though. You can use this trick in any cache where you want to "cache" something, regardless if it's based on the arguments or not:

function describe(transaction) {
  let currency = getCurrency(transaction);

  let converter = CACHE.get(currency);
  if(!converter) {
    // get it...
  }

  // keep doing stuff...
}

^7731a8

Maybe "converter" here is some kind of currency converter, and to get it requires an expensive API call that you want to cache.

Again, you could lift this caching up into a network layer. But this is useful when you are stuck in a place where for some reason you just can't use that. Another strategy would be to have the transaction object itself handle the caching. You could add something on the Transaction class:

class Transaction {
  _cachedCurrencyConverter = null;

  // rest of the implementation
}

^d014f1

Now in describe we could set it on the Transaction instance:

function describe(transaction) {
  let currency = getCurrency(transaction);

  let converter = transaction._cachedCurrencyConverter;
  if(!converter) {
    converter = /* get it */

    transaction._cachedCurrencyConverter = converter;
  }
}

^62b9a8

This could work. However, we are assuming the Transaction class is something we control. Similar to the networking layer problem, we might not be able to change it. We could get spicy in set it on transaction ourselves without it being defined in the class. Now you are monkeypatching instances and these could clash across abstractions (maybe another dev sets something with the same name).

I also believe stuffing classes with semi-related data also leads to bloated abstractions. There are better ways to coordinate.

If you are in a weird situation where you really want to write a describe function with the caching semantics, but you don't have control over Transaction and you can't rely on the networking layer, what are you to do?

This WeakMap technique subverts control of the program: it gives you the power to add these semantics yourself. You need to track state somewhere. If you can't control Transaction or the networking layer, you're kinda stuck. You could use a normal Map but then you need to figure out an eviction strategy and the semantics just wouldn't be as good.

Tracking more state

This made me wonder: does this open up other interesting patterns? What if we used it to track a bunch of other state?

Let's say I'm on a team working on currency conversions. We are writing a lot of code for this, and we want to structure it in a class to provide all the functionality:

class CurrencyConversion {
  // implementation
}

^650363

We want this code to live as if it was an extension of the Transaction class: the decisions of which systems to use all apply here. Anywhere you can use Transaction, you can also use this class to manage currency conversions. In fact, if we controlled Transaction, we would just add all of this code in there. But let's assume we don't own it and the team is pushing back on our APIs.

We kind of want to "extend" Transaction with our APIs and treat it like this:

+------------------------------------+
|+-------------+                     |
|| Transaction | Currency conversion |
|+-------------+                     |
+------------------------------------+

^cda5ed

We won't go so far as monkeypatching Transaction, so the APIs will stay separate. But how can we make it as seamless as possible? What if we had a getConverter function that created our APIs?

const CONVERTERS = new WeakMap();

function getConverter(transaction) {
  const converter =
    CONVERTERS.get(transaction) || new CurrencyConversion(transaction, {
      /* options */
    });
  CONVERTERS.set(transaction, converter);
  return converter;
}

^c6adcf

Now we can get access to our APIs whenever we need to:

function describe(transaction) {
  if (!getConverter(transaction).isReady()) {
    // do something..
  }

  let amount = getConverter(transaction).convert();

  // generate a description
}

^027bfe

This is a silly example. In real code, you have shared systems that make it easy to write new abstractions that can coordinate easily. However, after working for a large company for several years I've seen how many times you want more control yourself. It's too easy to discover the the existing systems don't work for you.

What is so different between the above technique and just simply doing this?

function describe(transaction) {
  const converter = new CurrencyConversion(transaction);

  // Use the converter throughout this code
}

^e066fb

Hopefully by now the difference is obvious: here converter only exists for the lifetime to the describe function call. When we store it in a weak map, it exists for the entire lifetime of the transaction class. That allows us to write code the assumes the same lifetime (caching things, using instance equality, etc)

Admittedly, this is more of an edge case. I encourage you to be intentional about how you shape your abstractions however. How you structure code and coordinate abstractions can make or break your codebase, so while there's never a single best solution, the more tools we have the better.

Tonight I thought I'd give Cursor a chance to create a quick synth from scratch to play with harmonics. I also wanted to visualize the frequencies. Here's the Cursor came up with really quickly while the kids brush their teeth:

^d66d99

^7b915a

^f3bf01

^aeae51

A collection of notes as I work through The Little Learner.

^0c0c15

Chapter 1

1.5: graph a line with $w$ and $b$ parameters

const f = (w, b) => x => w * x + b;

render(graph(f(100, 10)));
render(graph(f(10, 0)));

^2b4784

1.14: reverse the order to make $x$ an argument and $w$ and $p$ parameters

// A version where w and b become parameters _after_ `x`
const f = x => (w, b) => w * x + b;

return graph(x => f(x)(2, 20));

^14f8ec

x is the argument of the line, while w and b which come after are parameters

1.19 plot the xs, ys dataset

const xs = [2, 1, 4, 3];
const ys = [1.8, 1.2, 4.2, 3.3];

return graph({
  marks: Plot.dot(zip(xs, ys), { x: n => n[0], y: n => n[1], fill: "black" }),
  domainX: [0, 5],
  domainY: [0, 5]
});

^dc0f98

Rule of parameters: every parameter is a number

Given x and y, or arguments to a function, we can walk backwards and figure out the parameters and then use that to predict other y values for a given x

θ is the parameter set (lowercase theta)

Given θ, there parameters if it referred to as θ$_1$, θ$_2$, etc

const f = θ => (θ_1, θ_2) => θ_1 * x + θ_2

^7052b4

Chapter 2

"Scalars" are real numbers

A "tensor" is a vector of scalars: [2.0, 1.0, 4.3, 4.2]

The book uses tensor$^1$ with a superscript

Tensors can be nested, and the superscript indicates the level of "nested"

"elements" are the individual values in the tensor

I think tensor$^1$ (with the 1 superscript) specifically means a vector of scalars, and higher tensors have tensors as elements

All tensors$^m$ must have the same number of elements

A scalar is atensor$^0$

9 is tensor$^0$

[9, 9, 9] is tensor$^1$

[[9, 9, 9] [9, 9, 9]] is tensor$^2$

2.25: define a function that finds the rank of a tensor

window.scalarp = v => typeof v === "number";
window.rank = t => (scalarp(t) ? 0 : 1 + rank(t[0]));

return log(output => {
  output(rank(4), 0);
  output(rank([4, 1]), 1);
  output(rank([[4, 1], [3, 6]]), 2);
  output(rank([[[[[3]]]]]), 5);
});

^ccfa8a

2.37 define a function that finds the shape of a tensor t

window.shape = (t, acc = []) => {
  if (!scalarp(t)) {
    acc.push(t.length);
    shape(t[0], acc);
  }
  return acc;
};

return log(output => {
  output(shape(5), []);
  output(shape([[4, 1], [3, 6], [2, 9]]), [3, 2]);
  output(shape([[[[[3]]]]]), [1, 1, 1, 1, 1]);
});

^80fdfc

Lists have members while non-scalar tensors have elements

2.42: how are rank and shape related?

// `rank` could also be defined as the length of `shape`
window.rank2 = t => shape(t).length;

return log(output => {
  output(rank2(4), 0);
  output(rank2([4, 1]), 1);
  output(rank2([[4, 1], [3, 6]]), 2);
  output(rank2([[[[[3]]]]]), 5);
});

^20a217

Interlude I

I.8: addition of tensors

const tadd = (t1, t2) =>
  t1.map((v, idx) => (scalarp(v) ? v + t2[idx] : tadd(v, t2[idx])));

return log(output => {
  output(tadd([1, 2, 3], [4, 5, 6]), [5, 7, 9]);
  output(tadd([[1, 2], [1, 2]], [[3, 4], [3, 4]]), [[4, 6], [4, 6]]);
  output(tadd([[[[[5]]]]], [[[[[1]]]]]), [[[[[6]]]]]);
});

^65d238

Currently, tensors must be of the same shape to add them with +

Making + work on tensors of arbitrary rank is called an extension of +

Functions built using extensions are called extended functions

Note: extension applies both to making an operator work on a single argument that has an arbitrary rank, and also making it work with two arguments that may have different ranks. A non-extended operator is one that only works with one specific rank

I.14 extended addition

Let's implement tadd_ as an extended addition operator. Note: click view source to see the implementation of these operators.

^eb1c7f

We can do this with all mathematical operations: *, sqrt, etc

It looks like the order shouldn't matter. With our last definition, we are assuming t2 has a higher rank, but we can make it work so that the higher rank can be in either position

^2b3daa

Not all math operations descend; for example sum$^1$. sum$^1$ has a superscript to make it clear it always expects a tensor. Let's implement tsum:

^e122ac

sum is the extended version of sum$^1$ which descends until it fins a tensor$^1$

^779995

Rule of sum: a tensor $t$ with a rank $r$, the rank of $sum(t)$ is $r - 1$

So far, we've implemented tadd_ andtsum_. These operators will be used for the rest of the page. We will need more operators, so let's go ahead and implement more:

tsub_:

^611105

tmul_:

^35802f

tsqr_:

^3ceeb6

Chapter 3

Fitting: a well-fitted θ is one the finds the best fit for a given data set

Let's take this line equation again:

window.line = x => (w, b) => tadd_(tmul_(w, x), b)

^10517d

We start with w (or θ$^0$) and b (or θ$^1$) both being 0.0

return log(output => {
  output(line([2, 1, 4, 3])(0, 0));
});

^d0d5b0

The loss is how far away our parameters are. The best fit would be loss as close to 0 as possible

How do you calculate loss?

First step: line-ys - predicted-line-ys, where predicted-line-ys is line(line-xs)(Θ_1, Θ_2}). So it becomes:

line-ys - line(line-xs)(Θ_1, Θ_2})

That produces a tensor$^1$ though; how do we get a scalar?

We sum it together, but also need to square it to get rid of negative values

sum(sqr(ys - pred-ys))

We can create an l2loss function using the operators we've defined above (tsum_ etc) to implement this:

window.l2loss = target => (xs, ys) => {
  return (...params) => {
    const pred_ys = target(xs)(params[0], params[1]);
    return tsum_(tsqr_(tsub_(ys, pred_ys)));
  };
};

return log(output => {
  const loss1 = l2loss(line)([2, 1, 4, 3], [1.8, 1.2, 4.2, 3.3])(0, 0);
  const loss2 = l2loss(line)([2, 1, 4, 3], [1.8, 1.2, 4.2, 3.3])(0.0099, 0);
  output(loss1);
  output(loss2);
  output("difference", loss2 - loss1);
});

^c5cf79

θ is the parameter set, it has nothing to do with the line function. It's just a list or vector with the right number of parameters needed

well, it does kind of have to do with line

parameters are the inputs need for a kind of "transformation" function. for example w*x + b. you have an input value, the transformation function, and the parameters for the transformation. at least that's how I'm thinking about it

An expectant function is the (xs, ys) => ... piece: it's a function that expects a data set as arguments

An objective function is the function which takes a parameter set and returns a scalar representing the loss

In the above code we output two losses: the first one with θ$^0$ set to 0 and the second with θ$^0$ to 0.0099. We increase it by a small number to figure out a rate of change (which is relative to this arbitrary small amount)

The rant of change here is $-0.62 / 0.0099 = -62.63$

This helps "seed" our rate of learning because otherwise we'd have no idea how fast changing the parameters changes the loss

Given the rate of change derived from this arbitrary small value, we use another small value and multiple it by this rate of change

This is called the learning rate and is represented by ⍺. Let's use 0.01

It's basically a "step size"

$⍺ * -62.63 = -0.6263$

^f89bc4

The rate of change depends on θ$0$. Now that we are using a new value for it, we need to get the new rate of change by using the 0.0099 constant again. Rate of change here is -25.12

It's not great to derive the rate of change this way though

Chapter 4

Graph of loss where X is θ$^0$

return graph(x => l2loss(line)([2, 1, 4, 3], [1.8, 1.2, 4.2, 3.3])(x, 0), {
  domainX: [-1, 5]
});

^d58ab8

We also refer to the x-axis as weight

Rate of change is important. It's called gradient and we can define a function to find it for any function

A gradient function takes a function and a parameter set, and returns a list of gradients for each parameter in the parameter set

This is also known as ∇ (del)

My own break: automatic differentiation

It turns out the book never defines ∇ which makes it difficult to continue to build running examples. It casually defines it as a function that gets the gradient (or rate of change) at a specific value of a function. I was annoyed that it never gave a definition.

After doing some research, it turns out that ∇ essentially the derivative of a one-dimensional function. Defining this isn't simple, so the book assumes this already provided. So far the book has done a great job building things up from scratch, so this was a confusing turning point and it should have done a better job explaining exactly what ∇ is and why it's not providing a definition. At least point the reader to where they can find more information about it.

To define ∇ we need to implement automatic differentiation. This is a way to deriving an expression automatically. We build up a graph of operators and use rules of differentiation to get a new graph of operators representing the differentiated expression.

I'm using this observable as inspiration which in turn is inspired by micrograd from Andrej Karpathy.

This uses a neat technique called backwards differentiation which is possible because of a few constraints: we don't need a full derivative, but just a derivative of a single variable, and we never need the derivative of a derivative.

The basic idea is we want to know for a given expression that uses a variable x, how much does changing x affect the result? We can figure this out with a few steps:

Construct a graph of operations, compute the final value along the way eagerly (the forward pass)

Given the final value, perform a backwards pass to compute the amount that each piece of the expression contributed to the final result. Each operation implements its own backwards calculate according to various mathematical rules

The value for the single variable x in the expression given by the backwards pass is the gradient, or rate of change, for that variable.

First we define a Value class that represents the a single value in an equation, and then we define a couple operations like vadd, vmul, vpow, and more to operate on these values. (The v prefix means we're operating on values)

^6c9ea4

Using it looks like this. This is how we represent the equation $(x + -4)^2 + 10$:

const variable = new Value(x);
vadd(vpow(vadd(variable, new Value(-4)), 2), new Value(10));

^3c8182

Notice how there isn't really any difference between the x value and the other constants in here. They all operate as a Value. This simplifies our work and makes it more efficient, but it's important to remember they actually are different. x here is an actual variable and needs to be the single input into the equation, and if that's the case reading the x.grad value makes sense.

The other numbers like -4 and 10 are constants and it wouldn't make any sense to read the grad value of them. It took me a while to understand this: if you just did new Value(4) to represent y=4, which would simply render a horizontal line, and then ran the backwards pass and got the gradient, the value would be 1! The issue is we are assuming the value is the input variable and if it were, 1 would be the correct slope because we'd actually be working with the y=x equation.

Let's visualize this. This graph shows the equation $(x - 4)^2 + 10$ and we apply automatic differentiation at each whole number to visualize the gradient at that point.

const gradients = range(-4, 12).map(x => {
  const target = new Value(x);
  const top = vadd(vpow(vadd(target, new Value(-4)), 4), new Value(10));
  top.backward();

  return v => (v - target.data) * target.grad + top.data;
});

return graph(...gradients, x => Math.pow(x - 4, 4) + 10, {
  domainX: [-5, 15],
  colors: [...gradients.map(_ => "#a0a0a0"), "#e60049"]
});

^0d3a41

Feels really good to understand this and be able to work with a real system written from scratch. I'll be able to continue to run examples from the book!

However, first we need to extend our tensor functions to support our now automatic differentiation system. Remember when we defined tmul, tadd, etc? That feels very similar to our new operation above right? We can combine both together into new tensor operations that support both properties: extended math functions that work with tensors that also support automatic differentiation.

^af3032

That works!

Chapter 4 (continued)

Revisions

4.23: define a revise function which helps revise parameters given a function

window.revise = function revise(func, revs, params) {
  while (revs > 0) {
    params = func(params);
    revs--;
  }
  return params;
};

return log(output => {
  output(revise(params => params.map(p => p - 3), 5, [1, 2, 3]), [
    -14,
    -13,
    -12
  ]);
});

^4d6977

Now let's use this revise function to help adjust parameters to reduce loss

const learning_rate = 0.01;
const obj = l2loss(line)(tdual_([2, 1, 4, 3]), tdual_([1.8, 1.2, 4.2, 3.3]));

window.lineParams = revise(
  params => {
    const gs = gradient_of(obj, params);
    return [
      params[0] - gs[0] * learning_rate,
      params[1] - gs[1] * learning_rate
    ];
  },
  1000,
  [0, 0]
);

return log(output => output(lineParams));

^7b1e95

We found our line! Let's plot it against the xs, ys dataset and see how it fits:

^f055bf

Hell yeah!

This is the optimization of gradient descent. We used a learning rate of 0.01 with our loss function and revised 1000 times to find the parameter set that fits the dataset bet

So far we've hardcoded the amount of parameters in our revise usage, let's generalize that. We can also generalize the other constants like learning rate and objective function into a general gradient_descent function:

window.gradient_descent = function gradient_descent(obj, initialp, α, revs) {
  return revise(
    params => {
      const gs = gradient_of(obj, params);
      return gs.map((g, idx) => params[idx] - g * α);
    },
    revs,
    initialp
  );
};

^be6caa

return log(output => {
  const obj = l2loss(line)(tdual_([2, 1, 4, 3]), tdual_([1.8, 1.2, 4.2, 3.3]));
  output(gradient_descent(obj, [0, 0], 0.01, 1000));
});

^b1f226

Θ is the symbol used for the revised parameters passed to the revision function. (The capitalized version of θ)

Interlude II

TODO: Keep going

I use this site not just for writing content, but also quick sketches of interactive code. I can easily make a code block interactive by tagging it to run inline in the page, making it easy to develop quick ideas. For this to work, publishing needs to be instant and the editing experience is vital. In logseq, it took 3-4 seconds to publish content (some of this is due to how I wired up the process) but worse is the editing experience in large posts with lots of code was very slow and frustrating.

My new setup uses obsidian to write content which I've found extremely polished and fast. A big difference is logseq is an "outliner" and obsidian treats content as a simple file of text. I'll miss certain features of outliners but overall the UX improvement is very much worth it.

New site allows code with syntax highlighting (just as before):

function calc(x) {
  return x + 6 * 10000;
}

^659b71

I can make code blocks easily run by adding the run tag to a codeblock. It looks like this:

```js run
function calc(x) {
  return x + 5 * 10;
}

return calc(10);
```

^4536cf

If I want to show something from the code I can just return the value and it appears in the DOM:

^ef4d56

Alternatively I could use a render function within the code block to do the same thing: render(calc(10)). It's a little more verbose, but the benefit with this is I can render multiple things and at different places in the call stack.

I also took the time to support math which is rendered with katex. For example $g(\\sqrt{a^2 + b^2})$ renders as this: $g(\sqrt{a^2 + b^2})$.

This new workflow will be instrumental in my AI studies, which was the primary motivation for this rehaul. Watch this space: I have a feed if you want to keep up to date.

^30b920

I've always wanted to understand color theory, so I started reading about the XYZ color space which looked like it was the mother of all color spaces. I had no idea what that meant, but it was created in 1931 so studying 93-year old research seemed like a good place to start.

When reading about the XYZ color space, this cursed image keeps popping up:

I say "cursed" because I have no idea what that means. What the heck is that shape??

I couldn't find any reasonably clear answer to my question. It's obviously not a formula like x = func(y). Why is it that shape, and where did the colors come from? Obviously the edges are wavelengths which have a specific color, but how did the image above compute every pixel?

I became obsessed with this question. Below is the path I took to try to answer it.

I'll spoil the answer but it might not make sense until you read this article: the shape comes from how our eyes perceive red, green, and blue relative to each other. Skip to the last section if you want to see some direct examples.

The fill colors inside the shape are another story, but a simple explanation is there is some math to calculate the mixture of colors and we can draw the above by sampling millions of points in the space and rendering them onto the 2d image.

Let's dig in more.

^f0af28

^82e29b

Color matching functions

The first place to start is color matching functions. These functions determine the strength of specific wavelengths (color) to contribute so that our eyes perceive a target wavelength (color). We have 3 color matching functions for red, green, and blue (at wavelengths 700, 546, and 435 respectively), and these functions specify how to mix RGB to so that we visually see a spectral color.

More simply put: imagine that you have red, green, and blue light sources. What is the intensity of each one so that the resulting light matches a specific color on the spectrum?

Note that these are spectral colors: monochromatic light with a single wavelength. Think of colors on the rainbow. Many colors are not spectral, and are a mix of many spectral colors.

The CIE 1931 color space defines these RGB color matching functions. The red, green, and blue lines represent the intensity of each RGB light source:

^4f5638

Note: this plot uses the table from the original study. This raw data must not be used anymore because I couldn't find it anywhere. I had to extract it myself from an appendix in the original report.

Given a wavelength on the X axis, you can see how to "mix" the RGB wavelengths to produce the target color.

How did they come up with these? They scientifically studied how our eyes mix RGB colors by sitting people down in a room with multiple light sources. One light source was the target color, and the other side had red, green, and blue light sources. People had to adjust the strength of the RGB sources until it matched the target color. They literally had people manually adjust lights and recorded the values! There's a great article that explains the experiments in more detail.

There's a big problem with the above functions. Can you see it? What do you think a negative red light source means?

It's nonsense! That means with this model, given pure RGB lights, there are certain spectral colors that are impossible to recreate. However, this data is still incredibly useful and we can transform it into something meaningful.

Introducing the XYZ color matching functions. The XYZ color space is simply the RGB color space, but multiplied with a matrix to transform it a bit. The important part is this is a linear transform: it's literally the same thing, just reshaped a little.

I found a raw table for the XYZ color matching functions here and this is what it looks like. The CIE 1931 XYZ color matching functions:

^2dad83

Wikipedia defines the RGB matrix transform as this:

matrix = [
  2.364613,  -0.89654, -0.468073,
 -0.515166,  1.426408, 0.088758,
  0.005203, -0.014408, 1.009204
]

[R, G, B] = matrix * [X, Y, Z]

^a90681

We can take the XYZ table and transform it with the above matrix, and doing so produces this graph. Look familiar? This is exactly what the RGB graph above looks like (plotted directly from the data table)!

^6de61f

Wikipedia also documents an analytical approximation of this data, which means we can use mathematical functions to generate the data instead of using tables. Press "view source" to see the algorithm:

^bcc10d

How is this useful?

Ok, so we have these color matching functions. When displaying these colors with RGB lights though, we can't even show all of the spectral colors. Transforming it into XYZ space, where everything is positive, fixes the numbers but what's the point if we still can't physically show them?

The XYZ space describes all colors, even colors that are impossible to display. It's become a standard space to encode colors in a device-independent way, and it's up to a specific device to interpret them into a space that it can physically produce. This is nice because we have a standard way to encode color information without restricting the possibilities of the future -- as devices become better at displaying more and more colors, they can automatically start displaying them without requiring any infrastructure changes.

Chromaticity

Now let's get back to that cursed shape. That's actually a chromaticity diagram, which is "objective specification of the quality of a color regardless of its luminance".

We can derive the chromaticity for a color by taking the XYZ values for it dividing each by the total:

const x = X / (X + Y + Z)
const y = Y / (X + Y + Z)
const z = Z / (X + Y + Z) = 1 - x - y

^0ec113

We don't actually need z because we can derive it given x and y. Hence we have the "xy chromaticity diagram". Remember how I said it's a 3d curve projected onto a 2d space? We've done that by just dropping z.

If we want to go back to XYZ from xy, we need the Y value. This is called the xyY color space and is another way to encode colors.

Alright, let's try this out. Let's take the RGB table we rendered above, and plot the chromaticity. We do this by using the above functions, and plotting the x and y points (the colors are a basic estimation):

^58d0f7

Hey! Look at that! That looks familiar. Why is it so slanted though? If you look at the x axis, it actually goes into negative! That's because the RGB data is representing impossible colors.

Let's use an RGB to XYZ matrix to transform it into XYZ space (the opposite of what we did before, where we transformed XYZ into RGB) space. If we render the same data but transformed, it looks like this:

^d7d4da

Now that's looking really familiar!

Just to double-check, let's render the chromaticity of the XYZ table data. Note that we have more granular data here, so there are more points, but it matches:

^c7a085

Ok, so what about colors? How do we fill the middle part with all the colors? Note: this is where I really start to get out my league, but here's my best attempt.

What if we iterate over every single pixel in the canvas and try to plot a color for it? The question is given x and y, how do we get a color?

Here are some steps:

We scale each x and y point in the canvas to a value between 0 and 1

Remember above I said we need the Y value to transform back into XYZ space? Turns out that the XYZ space intentionally made Y map to the luminance value of a color, so that means we can... make it up?

What if we just try to use a luminance value of 1?

That lets us generate XYZ values, which we then translate into sRGB space (don't worry about the s there, it's just RGB space with some gamma correction)

One immediate problem you hit this produces many invalid colors. We also want to experiment with different values of Y. The demo below has controls to customize its behavior: change Y from 0 to 1, and hide colors with elements below 0 or 255.

^31f373

^fba881

^4e7753

That's neat! We're getting somewhere, and are obviously constrained by the RGB space. By default, it clips colors with negative values and that produces this triangle. Feels like the dots are starting to connect: the above image is clearly showing connections between XYZ/RGB and limitations of representable colors.

Even more interesting is if you turn on "clip colors max". You only see a small slice of color, and you need to move the Y slider morph the shape to "fill" the triangle. Almost like we're moving through 3d space.

For each point, there must be a different Y value that is the most optimal representation of that color. For example, blues are rich when Y is low, but greens are only rich when Y is higher.

Taking a break: spectrums

I'm still confused how to fill that space within the chromaticity diagram, so let's take a break.

Let's create a spectrum. Take the original color matching function. Since that is telling us the XYZ values needed to create a spectral color, shouldn't we be able to iterate over the wavelengths of visible colors (400-720), get the XYZ values for each one, and convert them to RGB and render a spectrum?

^3222ac

^0ffafa

This looks pretty bad, but why? I found a nice article about rendering spectra which seems like another deep hole. My problems aren't even close to that kind of accuracy; the above isn't remotely close.

Turn out I need to convert XYZ to sRGB because that's what the rgb() color function is assuming when rendering to canvas. The main difference is gamma correction which is another topic.

^5cabf2

We've learned that sRGB can only render a subset of all colors, and turns out there are other color spaces we can use to tell browsers to render more colors. The p3 wide gamut color space is larger than sRGB, and many browsers and displays support it now, so let's test it.

You specify this color space by using the color function in CSS, for example: color(display-p3 r, g, b). I ran into the same problems where the colors were all wrong, which was surprising because everything I read implied it was linear. Turns out the p3 color space in browsers has the same gamma correction as sRGB, so I needed to include that to get it to work:

^808408

If you are seeing this on a wide gamut compatible browser and display, you will see more intense colors. I love that this is a thing, and the idea that so many users are using apps that could be more richly displayed if they supported p3.

I started having an existential crisis around this point. What are my eyes actually seeing? How do displays... actually work? Looking at the wide gamut spectrum above, what happens if I take a screenshot of it in macOS and send it to a user using a display that doesn't support p3?

To test this I started a zoom chat with a friend and shared my screen and showed them the wide gamut spectrum and asked if they could see a difference (the top and bottom should look different). Turns out they could! I have no idea if macOS, zoom, or something else is translating it into sRGB (thus "downgrading" the colors) or actually transmitting p3. (Also, PNG supports p3, but what do monitors that don't support it do?)

The sheer complexity of abstractions between my eyes and pixels is overwhelming. There are so many layers which handle reading and writing the individual pixels on my screen, and making it all work across zoom chats, screenshots, and everything is making my mind melt.

Let's move on.

A little question: why does printing use the CMY color system with the primaries of cyan, magenta, and yellow, while digital displays build pixels with the primaries of reg, green, and blue? If cyan, magenta, and yellow allow a wider range of colors via mixing why is RGB better digitally? Answer: because RGB is an additive color system and CMY is a subtractive color system. Materials absorb light, while digital displays emit light.

Back to the grind

We're not giving up on figuring out the colors of the chromaticity diagram yet.

I found this incredible article about how to populate chromaticity diagrams. I still have no idea if this is how the original ones were generated. After all, the colors shown are just an approximation (your screen can't actually display the true colors near the edges), so maybe there's some other kind of formula.

So that I can get back to my daily life and be present with my family, I'm accepting that this is how those images are generated. Let's try do it ourselves.

There's no way to go from an x, y point in the canvas to a color. There's no formula tells us if it's a valid point in space or how to approximate a color for it.

We need to do the opposite: start with an value in the XYZ color space, compute an approximate color, and plot it at the right point by converting it into xy space. But how do we even find valid XYZ values? Not all points are valid inside that space (between 0 and 1 on all three axes). To do that we have to take another step back.

I got this technique from the incredible article linked above. What we're trying to is render all colors in existence. Obviously we can't actually do that, so we need an approximation. Here's the approach we'll take:

First, we need to generate an arbitrary color. The only way to do this is to generate a spectral line shape. Basically it's a line across all wavelengths (the X axis) that defines how much each wavelength contributes to the color.

To get the xy, coordinate on the canvas, we need to get the XYZ values for the color. To do that, we multiply the XYZ color matching functions with the spectral line, and then take the integral of each line to get the final XYZ values.

We do the same for the RGB color. We multiply the RGB color matching functions with the spectral line and take the integral of each one for the final RGB color. (We'll talk about the colors more later)

I don't know if that made any sense, but here's a demo which might help. The graph in the bottom left is the spectral line we are generating. This represents a specific color, which is shown in the top left. Finally, on the right we plot the color on the chromaticity diagram by summing up the area of the spectral line multiplied by the XYZ color matching functions.

We generated the spectral line graph with two simple sine curves with a specific width and offset. You can change the offset of each curve with the sliders below. You can see that moving those curves, which generates a different spectral line (and thus color) which plots different points on the diagram.

By adjusting the sliders, you are basically painting the chromaticity diagram!

^60d1c4

^ec88e9

^73d5bc

You can see how all of this works but pressing "view source" to see the code.

Obviously this is a very poor representation of the chromaticity diagram. It's difficult to cover the whole area; adjusting the offset of the curves only allows you to walk through a subset of the entire space. We would need to change how we are generating spectral lines to fully walk through the space.

Here's a demo which attempts to automate this. It's using the same code as above, except it's changing both offset and width of the curves and walking through the space better:

^554867

^bba249

I created an isolated codepen if you want to play with this yourself. If you let this run for a while, you'll end up with a shape like this:

We're still not walking through the full space, but it's not bad! It at least... vaguely resembles the original diagram?

What's going on with the colors?

Our coloring isn't quite right. It's missing the white spot in the middle and it's too dark in certain places. Let me explain a little more how we generated these colors.

After all, didn't we generate RGB colors? If so, why weren't they clipped and showing a triangle like before? Or at least we should see more "maxing out" of colors near the edges.

My first attempts at the above did show this. Here's a picture where I only took the integral to find the XYZ values, and then took those values and used XYZ_to_sRGB to transform them into RGB colors:

We do get more of the bright white spot in the middle, but the colors are far too saturated. It's clear that many of these colors are actually invalid (they are not in between 0 and 255).

Another technique I learned from the incredible article is to avoid using the XYZ points to find the color, and instead do the same integration over the RGB color mapping functions. So we take our spectral line, multiple by each of the RGB functions, and then take the sum of each result to find the individual RGB values.

Even though this still produces invalid colors, intuitively I can see how it more directly maps onto the RGB space and provides a better interpolation.

That's about as far as I got. I wish I had a better answer for how to generate the colors here, and maybe you know? If so, give me a shout! I'm satisfied with how far I got, and I bet the final answer uses slightly different color matching functions or something, but it doesn't feel far off.

If you have ideas to improve this, please do so in this demo! I'd love to see any improvements.

I want to drive home that my above implementation is still generating invalid colors. For example, if I add clipping and avoid rendering any colors with elements outside of the 0-255 range, I get the familiar sRGB triangle:

It turns out that even though colors outside the triangle aren't rendering accurately, we're still able to represent a change of color because only 1 or 2 of the RGB channels have maxed out. If green maxes out, changes in the red and blue channels will still show up.

More shape explorations

But really, why that specific shape? I know it derives from how we perceive red, green, and blue relative to each other. Let's look at the XYZ color matching functions again:

^140cd2

The shape is derived from these shapes. To render chromaticity, you walk through each wavelength above and calculate the percentage of each XYZ value of the total. So there's a direct relationship.

Let's drive this home by generating our own random color matching functions. We generate them with some simple sine waves (view source to see the code):

^7d34e3

Now let's render the chromaticity according to our nonsensical color matching functions:

^6bf276

The shape is very different! So that's it: the shape is due to the XYZ color matching functions, which were derived from experiments that studied how our eyes perceive red, green, and blue light. That's why the chromaticity diagram represents something meaningful: it's how our eyes perceive color.

Resources:

• The CIE RGB color space
• A Beginner’s Guide to (CIE) Colorimetry
• CIE Chromaticity and Perception