CodeTV Blog RSS Feed

Web Dev Challenge Hackathon S2.E11: Twilio Hotline Challenge

Mon, 03 Nov 2025 00:00:00 GMT

Press 1 for an awesome AI-Powered Hotline

Ever thought: gee, I wish there was a hotline for that? Well now’s your chance to build it!

Think: an office-hours hotline, community resources line, pizza-ordering line, event status line, bug triage, wake‑up calls, volunteer coordination, campus shuttle tracker — if it rings it fits.

The Tool: ConversationRelay

Your hotline must use ConversationRelay as part of the build.

Twilio’s ConversationRelay lets you place a relay layer between callers and your logic so you can route messages, swap in bots or humans, and keep a clean separation of concerns.

Use it with Twilio Programmable Voice and/or Messaging (SMS/WhatsApp), webhooks, and your runtime of choice (Twilio Functions/Studio/Serverless or your own stack).

Ready to get started? Check out the ConversationRelay docs, dive into one of the tutorials on the Twilio blog, or check out the starter repo on Github.

Make new connections and get expert guidance

The Twilio team has set up a dedicated channel on the Twilio Discord for brainstorming, sharing ideas, and keeping each other accountable.

Multiple Ways to Win!

The Twilio team will choose three built hotline submissions to win some great Twilio swag from our merch store and Twilio credits.

The first five qualifying hotlines submitted will also receive an item of their choice (up to $150) from Ugmonk, Drop, or Cocoon.

Finally, whether you’ve built a hotline or just have a great idea, share it in the challenge thread on Twilio's X account or LinkedIn post. The top five ideas by total likes, comments, and reshares at the deadline will each receive a gift card valued at $25. As a bonus, we will be building one standout idea live on Twitch with Alex and Jason.

Watch the episode for inspiration

See how Jack, Sherm, Brian, Pedro, Avi, and Dibyanshu tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Team up with a friend or work on your onw
Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use ConversationRelay to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
11:59 pm Pacific on Friday, Nov 21, 2025 by replying to Twilio's post on X or on LinkedIn

Bonus: you can also win for a great idea!

If you don’t have time to build a hotline, you can also win by submitting our favorite hotline idea. Reply to the post with your idea, and if we decide to build it, you’ll win a prize!

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon S2.E9: Breakfast Apps

Tue, 23 Sep 2025 00:00:00 GMT

Build an app that’s about breakfast

It’s the most important ~~meal~~ app of the day! Build an app that is somehow related to breakfast.

Manage your favorite recipes, track the best breakfast burritos, build a digital shrine to pancakes — we don’t care what you build as long as it has something to do with breakfast.

The Tool: Hashbrown

Your app must use Hashbrown as part of the build.

Hashbrown allows developers to create generative UIs using your own components. It’s fully free and open source, released under the MIT license.

With Hashbrown, developers can provide a set of custom components with a clear schema, and use that to give your LLM guardrails for outputting generative — but quality controlled! — user interfaces that adapt to meet the needs of your users.

You’ve also got access to safe function calling, structured outputs, and a JavaScript Runtime, which open up some pretty interesting opportunities within apps.

Make new connections and get expert guidance

We've set up a dedicated channel in the CodeTV Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Shaundai, Scott, James, Brian, Jane, and David tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Team up with a friend or work on your onw
Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Hashbrown to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, Sep 8, 2025

Submit your web app

Happy building! Let's have some fun.

CSS overrides without important using layers in Astro components

Sat, 30 Aug 2025 00:00:00 GMT

Recently I was working on a shared Astro component and found an edge case that required me to override the CSS of the Astro component from within the page that imported it.

I thought the fix was neat, so I figured I'd write it up in case it helps anyone else.

Minimal reproduction of CSS override challenges is Astro

To see the problem in action, let's imagine a simple Astro site that has a layout, a heading component, and a page that imports that heading component.

---
const { title } = Astro.props;
---

<html lang="en">
	<head>
		<meta charset="utf-8" />
		<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
		<meta name="viewport" content="width=device-width" />
		<meta name="generator" content={Astro.generator} />
		<title>{title}</title>
	</head>
	<body>
		<slot />
	</body>
</html>

<style>
	html,
	body {
		font-family: system-ui, sans-serif;
		margin: 20px;
		text-align: center;
	}
</style>

---
const { title } = Astro.props;
---

<h1 class="heading">{title}</h1>

<style>
	.heading {
		color: red;
	}
</style>

---
import Layout from '../components/layout.astro';
import Heading from '../components/heading.astro';

const title = 'Style overrides in Astro with CSS layers';
---

<Layout title={title}>
	<Heading title={title} />
</Layout>

These three files result in a simple Astro site.

Why standard overrides don't work in Astro components

One of Astro's strengths is that, by default, all CSS is scoped to the component. This solves one of the biggest sources of frustration that many devs feel with CSS: the cascade and inheritance, which can become really hard to keep track of in large code bases.

The way Astro does this is by applying a unique generated data attribute to the built HTML, which is then appended to every CSS selector.

The problem comes in when we do want to use the cascade. For example, if an edge case arises where the headline needs to be blue, we may try something like adding a global override to the .heading class in the page using the Heading component:

---
import Layout from '../components/layout.astro';
import Heading from '../components/heading.astro';

const title = 'Style overrides in Astro with CSS layers';
---

<Layout title={title}>
	<Heading title={title} />
</Layout>

<style is:global>
	.heading {
		color: blue;
	}
</style>

This won't work. If we inspect the element, we can see that the scoped component style has higher specificity, so we can't override it this way.

If we want to override the CSS in an Astro component, we're going to need to try something else.

CSS overrides in Astro the old way: `!important`

Previously, I would have reluctantly slapped an !important at the end of the override. It's heavy-handed and feels kinda gross, but it works. Them's the breaks.

---
import Layout from '../components/layout.astro';
import Heading from '../components/heading.astro';

const title = 'Style overrides in Astro with CSS layers';
---

<Layout title={title}>
	<Heading title={title} />
</Layout>

<style is:global>
	.heading {
		/* not ideal */
		color: blue !important;
	}
</style>

But it's not 2021 anymore! We have a better option: enter CSS layers.

CSS overrides in Astro the new way: CSS layers

Since March 2022, @layer is part of Baseline.

This means we can assign a given rule or rules to a named layer. For example, we can define a layer called component for our heading component styles:

---
const { title } = Astro.props;
---

<h1 class="heading">{title}</h1>

<style>
	@layer component {
		.heading {
			color: red;
		}
	}
</style>

Next, we can define an additional layer called overrides in the page that imports the component:

---
import Layout from '../components/layout.astro';
import Heading from '../components/heading.astro';

const title = 'Style overrides in Astro with CSS layers';
---

<Layout title={title}>
	<Heading title={title} />
</Layout>

<style is:global>
	@layer overrides {
		.heading {
			color: blue !important;
			color: blue;
		}
	}
</style>

Because these layers are arbitrarily named, there's no magic involved — we can either rely on the layers to be applied in the order they were defined (which will get very confusing with components), or we can manually define the layer order.

To manually define the layer order, add a @layer declaration to the top of the layout component that tells the browser to render the component styles first, then apply the overrides:

---
const { title } = Astro.props;
---

<html lang="en">
	<head>
		<meta charset="utf-8" />
		<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
		<meta name="viewport" content="width=device-width" />
		<meta name="generator" content={Astro.generator} />
		<title>{title}</title>
	</head>
	<body>
		<slot />
	</body>
</html>

<style>
	@layer component, overrides;

	html,
	body {
		font-family: system-ui, sans-serif;
		margin: 20px;
		text-align: center;
	}
</style>

Because of how cascade layers work, we don't need the !important hack to override the component styles because our layer order tells the browser that the overrides take precedence.

Modern CSS overrides, no hacks

CSS layers are great because they give all the control of things like !important, but without the compounding frustration of ever-increasing specificity wars. As developers, we can choose what takes precedence by placing our styles into layers — and we choose the order of importance for the layers.

I was really happy to see this feature land in all modern browsers, and even happier that Astro is built in such a way that we can use the platform to accomplish edge case goals like these.

Resources

Web Dev Challenge Hackathon S2.E7: Touch Grass

Tue, 26 Aug 2025 00:00:00 GMT

Build an app that connects us to the physical world

So much of the software out there today seems to be reducing our connections — we talk to fewer people, have fewer reasons to go outside, etc. — so let’s build apps that help connect us to the physical world.

Face-to-face. IRL. Meatspace. Between no-contact deliveries and every AI app trying to replace another human relationship — from pair programming to therapy to romantic partners — it feels like so much of the technology being produced right now is designed to minimize, or even eliminate, our need to interact with other humans.

Let’s do our small part to change that. Your challenge is to build an app that encourages people to connect with the physical world and/or each other.

What helps you disconnect from the digital firehose and spend time making offline connections? Go in whatever direction inspires you, whether that’s a hobby, different ways to spend time together with other people, connecting to nature, or something completely different.

Your app should help ground your users and reconnect them to the world beyond their screens.

The Tool: Apollo Connectors

Your app must use Apollo Connectors as part of the build.

Apollo has been helping developers build flexible APIs via GraphQL since the early days of the technology, and Connectors make it possible to bring any REST API into GraphQL quickly and painlessly. This is exciting because it means you can extend your own data with third-party API data (or vice versa) to build out anything you can imagine without a lot of data plumbing hassle.

For this challenge, you’ll have access to a large collection of pre-built Connectors for everything from Pokémon to Stripe data to LLM providers like Anthropic and OpenAI. You can also quickly hook up your own REST APIs to a custom Connector.

Make new connections and get expert guidance

We've set up a dedicated channel in the CodeTV Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

Win a ticket to GraphQL Summit!

The Apollo team will choose 3 hackathon submissions to win a free ticket to the GraphQL Summit.

In addition, the first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Israa, Hosna, Nicole, Ryan, Mark, and Joe tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Team up with a friend or work on your onw
Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Apollo Connectors to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, Sep 8, 2025

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon S2.E5: Give in to your worst developer urges

Tue, 01 Jul 2025 00:00:00 GMT

Give in to your worst developer urges and automate something unnecessary

As developers, we all know the urge to spend hours automating a task that would have taken us minutes is overpowering. In this challenge, we want you to give in to that urge and automate something in your life that you find mildly inconvenient.

Why do a thing when you could spend an inordinate amount of time building an app that does the thing for you? No task is too small. No inconvenience too minor. We want to see you create apps that automate your life in extremely specific, personal ways.

Do you have to send that one friend a reminder text to make sure they show up on time for your plans? Build an app for that! Sick of doing tech support for your family? Create an AI agent that reads the manual and responds on your behalf. You can keep it simple and practical, or go big and channel your inner Rube Goldberg — anything goes!

The Sponsor: Intuit

Intuit wants to empower the builders in our community. There’s no required tool this time around — you are the secret ingredient here.

Combine your skills in web development, problem solving, creative thinking, and leveraging new technologies like AI agents or MCP to build your personal problem-solving web app.

Make new connections and get expert guidance

We've set up a dedicated channel in the CodeTV Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Lyn and Ebrima, Bryn and Jacob, and Ben and Seve tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Team up with a friend or work on your onw
Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Intuit to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, July 14, 2025

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon S2.E3: Create a devious web-based video player

Tue, 13 May 2025 00:00:00 GMT

Build the most unhinged, Byzantine, devious, unusual video player UX you can imagine.

Programmer social media passes around intentionally bad UI designs every once in a while, and it’s both very fun and also deeply painful to see how creatively a developer can make something like a simple button hurt so much to use.

Your challenge is to embody this kind of mischievous energy and build out a truly heinous video player UX built on top of the Mux video player base. The video must be actually playable (though we’ll let you get creative with what “playable” means).

Apps must use Mux as part of the build.

Introduction to Mux

In the kickoff call for this hackathon, Dave Kiss from Mux gives an overview of how Mux and Media Chrome works.

Big prizes for this hackathon!

We always have goodies for the hackathon, but Mux is stepping it up this time:

Winning submission gets a trip to Vercel Ship — fly to New York City and attend the conference. Mux covers your ticket, flight, and hotel.
Runner up gets an Elgato streaming bundle — Prompter, 2x Key Light MK.2, Stream Deck, and Wave:3 mic.
First 50 entrants get a Media Chrome Thrift-tee — Retro vibes guaranteed to help you learn to write COBOL
First 100 entrants get $100 Mux credit — Get Mux bucks, on us.

Get more details on the Mux site

Make new connections and get expert guidance

We've set up a dedicated channel in the CodeTV Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

Find an accountabilibuddy, get expert guidance from the industry professionals, run your idea past Jason and other community members, and listen in to what the rest of the community is cooking up in this open call to kick off the hackathon.

RSVP to add it to your calendar:

Watch the episode for inspiration

See how Chan and Chance, Eve and Roxy, and Braydon and Devyn tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Team up with a friend or work on your onw
Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Mux to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, June 15, 2025

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon S2.E2: Build a game played on at least 2 devices

Tue, 29 Apr 2025 00:00:00 GMT

The prompt: Build a game that’s played on at least 2 devices.

Single player, multiplayer, cooperative, competitive, or something totally different — your challenge is to come up with something fun that is played across at least two devices. Temporal’s workflow tools will allow you to manage sending information between devices dependably.

Your game could be something like Jackbox, where a tv is the “game board” and each player’s phone is how they interact with it on their turn. You could make a game that uses phone APIs like the camera or gyroscope. Or you can implement a simple word game like the New York Times connection puzzles, but with the twist that it's designed to be solved collaboratively, and a session can persist beyond the players closing the web app.

Get creative and have fun with it!

Apps must use Temporal as part of the build.

Introduction to Temporal + devs discuss their ideas

In the kickoff call for this hackathon, Alex Garnett from Temporal gives an overview of how Temporal works, including a live coded demo, and answers questions from developers.

Make new connections and get expert guidance

We've set up a dedicated channel in the CodeTV Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Sarah and Nikki, Shashi and Nick, and Adam and Lane tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Team up with a friend or work on your onw
Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Temporal to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, May 12, 2025

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon S2.E1: Build a Custom API + App

Tue, 15 Apr 2025 00:00:00 GMT

The prompt: Build a custom API — and an app that consumes it — that makes something in your life a little more convenient.

Not every app needs to be the Next Big Thing. Sometimes you have a specific problem, challenge, or even just a pet peeve that can be solved with software. And these days, building an app that’s just for personal use is faster than ever.

For this challenge, your team needs to create a custom API that uses at least one third-party source and extends it in some way. You’ll also need to build a web app that interacts with your API. The end result should be a system that adds convenience to your life.

This could be something larger and practical, like automating something tedious you have to do over and over again (e.g. turn important incoming emails into calendar invites and/or Discord notifications), or it could be something fun and silly (e.g. use data from your wearable fitness device to calculate how many mini Snickers bars you’ve earned today).

Apps must use Postman as part of the build.

Make new connections and get expert guidance

We've set up a dedicated channel in the CodeTV Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Michael and Will, Brittany and Dave, and Abbey and Alex tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Postman to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Wednesday, April 28, 2025

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon 10: Workshop Woes

Sat, 14 Dec 2024 00:00:00 GMT

The prompt: The holidays are upon us and Santa’s elves are behind schedule! Build an app to get the holiday back on track.

Santa’s workshop is in chaos. The drop on the 25th is getting closer by every hour, every day You’ve been brought in as a consultant to help get things back in order. Your job: decide what the cause of the chaos is — then plan and build an app to solve the problem.

The team at Sanity will provide us with a starter dataset and schema giving us some information about Santa’s workshop and content for all the kids in the world (the good, bad, young and old ones) — things like the staff, the inventory, services, etc. — and you can extend or modify that schema as you see fit (or, if you prefer, ignore it entirely and do something else).

Apps must use Sanity as part of the build.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Aaron, Charlie, Kent, and Dom tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Sanity to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Wednesday, January 1, 2025

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon 9: Blvck Spades

Tue, 03 Dec 2024 00:00:00 GMT

The prompt: Come up with ways to connect people’s behavior to e-commerce incentives

Creating brand loyalty is everything to an e-commerce store. Our friends at Blvck Spades have a great product, and they need help coming up with ways to incentivize users to interact with their brand.

Your challenge is to build software that connects a user’s activity to incentives. Beyond an e-commerce store, many businesses might have an app, partnerships, or physical locations or events — our job is to connect a user’s activity in these other places to incentives in the e-commerce store using the Mailchimp users as the source of truth.

For example, something like “play 10 rounds of our game and get 10% off your order in the store”, or “show up to a local meetup and earn an exclusive, limited edition t-shirt”.

Apps must use Mailchimp as part of the build.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Bree, Robbie, Ximena, and Anthony tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Mailchimp to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, Dec 16, 2024

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon 7: Spooky Apps

Wed, 30 Oct 2024 00:00:00 GMT

The prompt: build an app that’s haunted. Make it spooky, make it creepy — let’s have a little Halloween fun.

Your app should get into the Halloween spirit in its functionality, user experience, or design. The specifics are up to you: it can be fun or funny, like a Scooby-Doo episode; it can be scary like a horror movie; it can be something totally unexpected.

Let’s build some apps to put a few “bytes” into Fright Night!

Apps must use Convex as part of the build.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Joel, Doug, Danny, and Jason tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Convex to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, Nov 11, 2024

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon 6: Capture Memories

Wed, 09 Oct 2024 00:00:00 GMT

The prompt: build an app that helps people capture memories.

There are so many things we experience that might seem small in the moment, but that we’d see as a treasured memory looking back on it years from now. A special event, a dinner with loved ones, a small adventure — or a big one.

Your challenge is to create an app to help people capture a special kind of memory. You get to choose what kind of memories your app will capture, but it should allow users to upload images and/or videos.

Apps must use Tigris Data as part of the build, and use of Fly.io is encouraged.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Lawrence, Amy, Josh, and Jason tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Tigris Data and, optionally, Fly.io to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, Oct 21, 2024

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon 5: The Local Food Scene

Tue, 17 Sep 2024 00:00:00 GMT

The prompt: make something that gets people excited about their local food scene.

The way restaurants run changed in a big way during the pandemic. The big food delivery apps control which restaurants get seen, and social media algorithm changes make it harder than ever for these businesses to get noticed. Your challenge is to come up with some way of getting more people to notice — and get involved in — their local food scene.

This can be anything web-based, whether you want to create some way for people to discover new restaurants and chefs, create an event or a social game, or even tackle the notoriously awful restaurant website design aesthetic.

Apps must use Convex, Clerk, and/or Raycast as part of the build.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

Find an accountabilibuddy, get expert guidance from the Clerk, Convex, and Raycast teams, run your idea past Jason and other community members, and listen in to what the rest of the community is cooking up in this open call to kick off the hackathon.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Maxi, Natalie, John, and Jason tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Convex, Clerk, and/or Raycast to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, September 30, 2024

Submit your web app

Happy building! Let's have some fun.

Web Dev Challenge Hackathon 4: Monsters!

Tue, 27 Aug 2024 00:00:00 GMT

The prompt: monsters have arrived — build an app to help out in this new world!

What kind of monster is up to you! Maybe kaiju have emerged from the ocean. Maybe Pokémon are actually real. Maybe gremlins were discovered. Maybe cryptids finally got caught on camera. Make ‘em cute, make ‘em scary, make ‘em fun — your call!

And “help” is also up for interpretation. Maybe this is an exciting scientific opportunity and your app supports that exploration. Maybe you’re helping people survive monster attacks. Or maybe you’re building an app to remote operate a monster-hunting robot to save humanity (if you can build that in 4 hours, you’re my hero).

Apps must use Convex, Clerk, and/or Raycast as part of the build.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

We'll have devs from the Algolia team available to provide technical guidance as you're building.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Annie, Shruti, Zrybea, and Jason tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Convex, Clerk, and/or Raycast to build your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, September 9, 2024

Submit your web app

Happy building! Let's have some fun.

Join the Web Dev Challenge Hackathon (E-comm Edition)

Tue, 06 Aug 2024 00:00:00 GMT

The prompt: build an e-commerce site... with a twist

E-commerce makes the world go round, and we want you to shake up the standard e-comm approaches with something a little different. The twist can be anything: make us laugh, add some M. Night Shyamalan flair — have fun with it!

Apps must use Algolia as part of the build. Algolia provides solutions for quickly adding search, recommendations, and more to web apps with pre-built components, fast responses, and painless integration.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

We'll have devs from the Algolia team available to provide technical guidance as you're building.

Find an accountabilibuddy, get expert guidance from the Algolia team, run your idea past Jason and other community members, and listen in to what the rest of the community is cooking up in this open call to kick off the hackathon.

RSVP to add it to your calendar:

The first 5 devs to submit a qualifying app get their choice of gear

The first 5 qualifying apps submitted will receive an item of their choice, up to $150 value, from one of the following sites:

Watch the episode for inspiration

See how Sidney, Alex, Shaundai, and Jason tackled the challenge in the latest episode of Web Dev Challenge to jumpstart your own creativity.

Watch on YouTube

The rules and how to submit

If you want to play along, here’s how:

Build a web app that meets the challenge
Spend 30 minutes¹ planning your app
Spend 4 hours¹ building your app
Use Algolia as part of your web app
Publish the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, August 19, 2024

Submit your web app

Happy building! Let's have some fun.

"Businesses hate fun" and other lies, with Salma Alam-Naylor

Mon, 08 Jul 2024 00:00:00 GMT

Watch on YouTube

What to expect in this episode of Web Lunch

I had the pleasure of joining Salma Alam-Naylor for a meal and talking to her about being a person vs. being a persona, aligning creativity and self-expression with business goals, and inventing villains as an excuse for not doing our best work.

Coming to terms with being extremely online

How much of yourself is too much to put online? Should we truly be ourselves, or a persona that we inhabit online? This is... complicated. We dig into it.

How to reconcile our creativity with business needs

Is it possible to be creative at a job that expects us to deliver business value? (Spoiler: yes.) Salma and I discuss how we find creative opportunities within the constraints of a full-time job, the power of learning the business side of things, and our strategies for connecting our ideas to monetary return to the company.

How to get buy-in for creative ideas

It's possible to pitch and work on extremely fun, creatively challenging work as part of your day job — if we can convince stakeholders that it's going to get them great results. We talk about how to make the business case for creative ideas.

What success actually means (maybe)

How do you know you're "successful"? Is it financial gain? Fame? Title? Salma talks about different measures of success that are more fulfilling than any of these for her.

We talk about what kind of impact we care about — and which kinds we actively reject these days.

What is Web Lunch?

In Web Lunch, host Jason Lengstorf joins talented, successful, and — most importantly — happy web professionals for a meal and a conversation.

These experts have:

Built careers that are resilient in the face of an ever-changing software industry
Achieved proper compensation and recognition for their efforts
Found joy and growth in their work — instead (or sometimes on the other side) of burnout

In each conversation, Jason aims to learn the strategies, mental frameworks, and techniques that got them there. Watch to learn how you can build your own resilient, lucrative, joyful career.

Inside Figma’s HQ: talking creativity with Jake Albaugh

Mon, 01 Jul 2024 00:00:00 GMT

Watch on YouTube

Where are the communities for creative coders? For designers? Not just to show their work, but to actually connect, collaborate, and grow together? I got the chance to visit Figma's developer advocate, Jake Albaugh, in their San Francisco office to talk about why creativity matters — and how to find people to share and grow with.

Jake Albaugh wants you to build stuff simply for the joy of creation. The act of creating has intrinsic value; it doesn't need to be monetized to matter. This mindset is crucial for fostering innovation and maintaining a sense of curiosity in an increasingly commercialized digital landscape.

The web started out as an open playground, filled with limitless possibilities and mostly free of consequences. Creative experimentation was the norm, because no one had agreed on how to use the internet yet. But as the internet has become more of a vehicle for commerce, the spirit of free experimentation moved to the fringes.

"It feels like there are fewer and fewer places to do that goofy stuff," Jake said. There's pressure to make everything commercially viable — to always be productive — that makes the idea of playful experimentation start to feel like a guilty pleasure.

But play is a core part of how we learn. Making space to goof around with something new in a low-stakes environment is a proven path to innovation. It's counterintuitive, but actively choosing not to be productive is a great way to discover new ideas and connections that might just end up as the catalyst for your next great career move.

Jake tries to lead by example here, constantly using (and misusing) tools to make things that generate absolutely no shareholder value, and along the way he's been able to create a body of work that landed him a job as a developer advocate at Figma.

The importance of community is something Jake believes in strongly. Surrounding yourself with others who celebrate exploration and playful uses of technology is the key to staying inspired.

Everything is so serious — finding a way to stay playful, to stay curious, gives us a chance to find joy in a world that can otherwise try to force us to only value direct return on investment.

The fun is the point. The community is the point. The growth is a bonus.

If you want to keep goofing around on the web, do it in public! Be weird! Share it with Jake and me! Bring back fun on the web.

Join the Web Dev Challenge mini-hackathon (retro gaming edition)

Mon, 24 Jun 2024 00:00:00 GMT

Watch the episode for inspiration

Watch on YouTube

See how Dev, Ben, Lindsay, and Jason tackled the challenge to get your creative gears turning!

What is AWS Amplify?

AWS Amplify has everything you need to create and deploy web and mobile apps in just a few hours! In this challenge, you'll have the opportunity to get started with AWS and level up your skill set.

Amplify has a variety of features that you can use in this challenge.

Data - Build secure, real-time APIs backed by AWS databases.
Auth - Enable secure authentication flows and control access to data, files, and more.
Storage - Storage and manage app content and data.
Fuctions - Add functions and configure environment variables.
Hosting - Deploy your website globally.

To get started, use the quickstart guide. Amplify is part of the AWS free tier!

You have the chance to win a dream desk setup!

In every episode of Web Dev Challenge, developers from around the community challenge themselves to build an app that meets a prompt and uses a given tool.

Now you can be one of those devs! Build your own take on the app prompt and submit it to the showcase to learn something new, show off your work, have some fun with the web dev community, and learn (or practice) practical skills.

And, as a bonus, one dev will win a dream desk setup: Ugmonk's complete Gather collection.

The prompt: “build a retro games-themed web app using AWS Amplify”

For this episode, the devs are building a web app for any kind of retro gaming theme they can think of. A Dr. Mario tournament app? Pong leaderboard? Investment advice for Monopoly players? It's all fair game!

Amazon Web Services is sponsoring this episode and giveaway, and the apps must be built using AWS Amplify, which can be used for the frontend deployment and/or for the backend of your web app.

Make new connections and get expert guidance

We've set up a dedicated channel in the Learn With Jason Discord called #builder-chat for brainstorming, sharing ideas, and keeping each other accountable.

We've also got our friends from the AWS Amplify team available to provide technical guidance as you're building.

The rules and how to submit

If you want to play along, here’s how:

Build a web app that has a retro games theme
Spend 30 minutes planning your app¹
Spend 4 hours building your app¹
Use AWS Amplify as part of your web app
Release the source code as a public GitHub repo
Publish the web app to a public URL
Submit your web app by 11:59 pm Pacific on Monday, July 29, 2024

Submit your web app

Happy building! Let's have some fun.

Not another f%*#in' chatbot — Web Dev Challenge S1E1

Sun, 23 Jun 2024 00:00:00 GMT

Watch on YouTube

The Challenge

Build an AI-powered app that's not another f%*#in' chatbot.

So much of what we see in the AI space are variations on the chatbot interface. Carter at DataStax challenged the devs to build something different.

The Tool

All apps must use Astra DB as part of the final build. Astra DB provides a vector store and tools for generating vector embeddings from any data you choose.

The Rules

In the Web Dev Challenge, the rules are:

4 web developers each build a web app
The web app is built to complete a specific challenge
The web app must include a given tool or technology, which changes each episode
Devs are given 30 minutes to plan
Devs are given 4 hours to build
At the end of 4 hours, the devs demo the web app they've built

Web Dev Challenge is not a competition. Instead, it's a chance to build something with low stakes, learn something new, and have fun with friends.

The Web Devs

Lizzie Siegle — https://www.lizziesiegle.xyz

Chance Strickland — https://chance.dev

Jack Herrington — https://youtube.com/@jherr

And, of course, your friend Jason.

The Advisor

Carter Rabasa, Head of Developer Relations at DataStax, was in studio to support the devs, teach us about vector databases, and answer questions.

The Apps

Jack's DnD Encounter Generator: source code

Jason's related videos PoC: live app · source code

Lizzie's Star Wars fanfic generator: source code

Chance's burger generator: source code

The Giveaway

We opened up this challenge to the whole community to see what you would build, and you could win a dream desk setup by Ugmonk by building your own app.

Huge congrats to Linda Thompson, who won the dream desk setup!

Understand your work archetype: explorers, villagers, and town planners

Fri, 31 May 2024 00:00:00 GMT

In 2018, I saw Simon Wardley speak at Craft Conf. In his talk, he mentioned in passing a model for types of work and the people best suited for each.

His casual aside was a "holy crap" moment for me because this framing unlocked a new perspective on my working history. I suddenly understood why I'd succeeded in certain jobs I'd had — and why I'd burned out quickly in others.

Let me explain.

What we do every day matters

Imagine two people with the job title "software engineer".

What do they actually do every day?

The first might handle Jira tickets for their team's section (billing) of an enterprise product's dashboard. The tickets are mostly minor tweaks and bug fixes. Over the course of an average quarter, there aren't many large changes to the codebase. Instead, there are long-running initiatives to modernize outdated code, fix performance issues, and handle edge cases as they're reported by customers.

The second might work at a startup and every day is a new adventure: sometimes the CEO codes up a new proof of concept over the weekend and now their task is to turn it into a production-ready feature; other times they're handling priority tickets from important customers; other other times they get handed a partially completed set of requirements and a tight deadline.

These people have the same job title, but these are two completely different roles.

Our working archetype should match our job duties

In the two hypothetical software engineering roles described above, I feel very different about one vs. the other. I could see myself thriving in one of the roles, and in the other I'm relatively confident I'd be back on the job market pretty quickly.

I would wager that you, dear reader, probably feel the same way. But whether you and I feel that way about the same roles is a coin toss. That's the critical point here: each of us gets our energy from certain types of work, and feels drained and/or stressed by other types of work. It's up to each of us to figure out what our individual strengths are — and then work toward landing roles that let us spend most of our time leveraging them.

Simon Wardley describes "explorers, villagers, and town planners" as the three core archetypes that make a company function.

People are complex, so I don't think we all fit tidily into one of these three boxes, but I do think there's a spectrum, and each of us can observe our own behavior to get a pretty good idea of where we fall on it.

Explorers thrive in the unknown

The explorer archetype is at their best when they have a problem to solve and very little structure around how to solve it. Explorers are comfortable with ambiguity and excited to try something new — in fact, they're often bored by things that aren't new.

Explorers thrive when:

Working with uncertainty — they thrive on chaos and get energy from creating new ideas
Trusting their instincts — they have a good gut feel for what will work and have enough strength in their convictions to act on them
Experimenting and "failing fast" — they see failure as progress and roll the lessons of unsuccessful experiments into new hypotheses

Explorers will struggle with:

Highly structured environments
Prescriptive direction (dictating how to do things)

Villagers turn promise into product

The villager archetype shines in roles where there's strong evidence that an idea will work, if only someone would sit down and do the work properly. Villagers find satisfaction in turning potential into reality, combining customer feedback and iterative testing to smooth out the rough edges and build something commercially viable.

Villagers thrive when:

Turning good ideas into actual products — they turn a cool demo into actual revenue
Getting customer feedback — they take pride in user feedback and act quickly to respond to problems
Getting the angles right — they're iterating toward success and energized by steady progress

Villagers struggle with:

Losing progress due to upstream changes
Isolation and lack of feedback

Town planners build resilience and scale

The town planner archetype is driven by the pursuit of a flawless system. They're process-driven, meticulously organized, and patient — willing to put in the hours to hunt down the most elusive edge cases and mysterious imperfections.

Town planners thrive when:

Standardizing and operationalizing — they work toward consistency and repeatability in a detail-oriented, very thorough manner
Responding to analytics and data — they care about clearly defined goals and verifiable measures of success
Steadily working toward perfection — they aim to perfect systems through small, incremental improvements

Town planners struggle with:

Lack of clarity and/or consistency
Rapidly changing priorities

It takes all three to run a successful company

In case it's not clear: none of these archetypes are better or worse than the others. All three have unique strengths and weaknesses, and any successful company will need all three archetypes in roles that match their strengths.

Knowing your archetype has nothing to do with your intrinsic value. Instead, having a clear idea of your archetype and the types of work that energize you will help you optimize for roles that take advantage of your strengths and don't force you into work that drains you.

While many jobs have the same job title, what actually matters is the day-to-day duties. It's important that your duties match up with your archetype if you want to remain happy in your role for a long time.

Which archetype best matches you?

As you read the descriptions of the archetypes above, did one feel the most "you"?

I believe these archetypes are more of a spectrum than a clear delineation, and I know people who feel like they're a blend of two of the archetypes. (In fact, I'd wager most of us will see parts of ourselves in two of the archetypes.)

I don't think it's important to force ourselves into one of these archetypes. The point isn't to shrink ourselves down to fit into a box; it's to understand what gives us energy so we can optimize our career efforts toward landing roles that make us feel happy and fulfilled by lining them up with our preferred working styles.

4 Web Devs 1 App Not Another Chatbot Dream Desk Giveaway

Mon, 29 Apr 2024 00:00:00 GMT

A new season of 4 Web Devs 1 App is coming, and we want everyone — especially you! — to join in on the fun. And to make things even more exciting, we've added a new feature: a community challenge and giveaway!

We’re giving away a dream desk setup!

In every episode of 4 Web Devs 1 App, developers from around the community challenge themselves to build an app that meets a prompt and uses a given tool.

And, as a bonus, one dev will win a dream desk setup: Ugmonk's complete Gather collection.

The prompt: “build an app using AI that isn’t another chatbot”

For this episode, the devs are building a web app that uses AI — but there's a catch: it can't just be another chatbot.

The team over at DataStax are sponsoring this episode, and the apps must be built using Astra DB.

How to enter

If you want to play along, here’s how to enter:

Build a web app that includes AI, but is not a chatbot
Use Astra DB as part of the web app
Release the source code as a public GitHub repo
Publish the web app to a hosting provider of your choice
Submit your web app using the form below by 11:59pm on Monday, May 13

Submissions are now closed

This giveaway has ended, but there will be more soon. Sign up for newsletter updates to hear about them first!

Play along with friends!

If you're looking for others who are getting involved, we've set up a dedicated channel in the Learn With Jason Discord (called #builder-chat) for people to brainstorm, share ideas, and keep each other accountable.

We've also got Carter and friends from the DataStax team available to answer your Astra DB questions as you're building.

Happy building! Let's have some fun.

Fix Drizzle "SqliteError: no such table" error — how to create tables

Sat, 17 Feb 2024 00:00:00 GMT

I was trying to build something with Drizzle but I couldn't figure out how to actually create tables in my SQLite database. Here are the docs I wish I'd had that would have saved me an hour of Googling things.

Create a new Node project

If you don't already have one, create a new Node project.

# create a folder 
mkdir my-drizzle-project
cd my-drizzle-project/

# initialize
npm init -y
git init

# install the required deps
npm i drizzle-orm better-sqlite3
npm i -D @types/better-sqlite3 @types/node drizzle-kit ts-node typescript

Update `package.json`

In package.json, make the highlighted changes. More on the command prefixed with db: later in this tutorial.

{
	"engines": {
		"node": ">=20.6.0"
	},
	"type": "module",
	"name": "drizzle-sqlite-basic-example",
	"version": "1.0.0",
	"description": "",
	"main": "index.js",
	"scripts": {
		"db:studio": "drizzle-kit studio",
		"build": "npx tsc",
		"dev": "node --env-file=.env --watch --loader ts-node/esm index.ts",
		"test": "echo \"Error: no test specified\" && exit 1"
	},
	"author": "Jason Lengstorf <jason@learnwithjason.dev>",
	"license": "ISC",
	"dependencies": {
		"better-sqlite3": "^9.4.1",
		"drizzle-orm": "^0.29.3"
	},
	"devDependencies": {
		"@types/better-sqlite3": "^7.6.9",
		"@types/node": "^20.11.19",
		"drizzle-kit": "^0.20.14",
		"ts-node": "^10.9.2",
		"typescript": "^5.3.3"
	}
}

Add `tsconfig.json`

You can use any TypeScript setup you prefer, but here's the one I tested with.

{
	"compilerOptions": {
		"strict": true,
		"target": "ESNext",
		"module": "NodeNext",
		"esModuleInterop": true,
		"forceConsistentCasingInFileNames": true,
		"skipLibCheck": true
	}
}

Create a Drizzle config file

import type { Config } from 'drizzle-kit';

export default {
	schema: './db/schema.ts',
	out: './db/migrations',
	driver: 'better-sqlite',
	dbCredentials: {
		url: './db/demo.db',
	},
} satisfies Config;

Create a SQLite database schema

Your schema can define any tables you need for your app. Here's what mine looks like:

import { integer, text, sqliteTable } from 'drizzle-orm/sqlite-core';

export const users = sqliteTable('users', {
	id: integer('id', { mode: 'number' }).primaryKey({ autoIncrement: true }),
	name: text('name'),
});

export const ideas = sqliteTable('ideas', {
	id: integer('id', { mode: 'number' }).primaryKey({ autoIncrement: true }),
	text: text('text'),
	status: text('status', { enum: ['approved', 'rejected', 'pending'] }),
	creator: integer('creator_id').references(() => users.id),
});

export type User = typeof users.$inferSelect;
export type NewUser = typeof users.$inferInsert;

export type Idea = typeof ideas.$inferSelect;
export type NewIdea = typeof ideas.$inferInsert;

Create the SQLite database tables from your Drizzle schema

Right now, the schema is defined but the tables haven't been created in the SQLite database yet.

This is where I got stuck. I tried to run my app, but I kept getting a SqliteError: no such table: users error. I couldn't find any reference to how to actually create the database tables in the Drizzle docs.

After searching around, I figured out two ways to get the SQLite tables created using Drizzle:

Generate a migration (the SQL query to create the tables as defined in your schema), which can be applied manually, using Drizzle's migrate helper, or with third-party tools — this is a production-friendly approach
Push the schema changes directly, which is exactly what I was looking for to set up my local database and is what I'll cover in this tutorial.

Both generating migrations and pushing schemas can be done using Drizzle's CLI helper, Drizzle Kit.

Push the SQLite database changes using Drizzle Kit

For prototyping, local dev, or initializing a new database, the push command will use the defined schema to update the database — which, in our case, will create the missing tables.

npx drizzle-kit push:sqlite

View the SQLite database in Drizzle Studio

To verify that the tables were created, you can use Drizzle Studio, which gives you a browser-based interface for viewing and working with your SQLite data.

npm run db:studio

This runs drizzle-kit studio under the hood and will start up Drizzle Studio at https://local.drizzle.studio/.

Test CRUD in SQLite using Drizzle ORM

Now the tables exist, you can actually insert and select data from the database.

To try this out, create index.ts at the root of the project and add the following:

import { drizzle } from 'drizzle-orm/better-sqlite3';
import Database from 'better-sqlite3';
import {
	users,
	ideas,
	type NewIdea,
	type NewUser,
	type User,
} from './db/schema.js';
import { eq } from 'drizzle-orm';

const sqlite = new Database('./db/demo.db');
const db = drizzle(sqlite);

// inserts a new user and returns the newly created entry
async function addUser(user: NewUser): Promise<User> {
	return (await db.insert(users).values(user).returning()).at(0)!;
}

async function addIdea(idea: NewIdea): void {
	await db.insert(ideas).values(idea);
}

// joins the ideas and users tables to select ideas with creator name
async function getIdeas() {
	return await db
		.select({
			id: ideas.id,
			text: ideas.text,
			status: ideas.status,
			creator: users.name, // <= use the creator's name instead of ID
		})
		.from(ideas)
		.leftJoin(users, eq(ideas.creator, users.id));
}

const user = await addUser({ name: 'Jason Lengstorf' });

await addIdea({
	text: 'Learn how ORMs work',
	status: 'pending',
	creator: user.id, // <= use the new user's ID as the creator
});

const allIdeas = await getIdeas();

console.log(allIdeas);

After saving this, start the dev server by running npm run dev and you'll see output similar to the following:

❯ npm run dev

> drizzle-sqlite-basic-example@1.0.0 dev
> node --env-file=.env --watch --loader ts-node/esm index.ts

(node:82803) ExperimentalWarning: Watch mode is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:82808) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("ts-node/esm", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
[
  {
    id: 1,
    text: 'Learn how ORMs work',
    status: 'pending',
    creator: 'Jason Lengstorf'
  },
]
Completed running 'index.ts'

This creates both a user and an idea, then does a partial select to create a combined view of the idea and user.

If you're like me and this is your first time using Drizzle and SQLite: congratulations! You just created and interacted with a database in a Node and TypeScript project, and you're ready to build your data-powered app.

Happy building!

How to set up a Node server with TypeScript in 2024

Wed, 14 Feb 2024 00:00:00 GMT

I've had to look up how to create a new Node server enough times that I'm writing it down to make it easier for me to find in the future. This is a quick and dirty post without much context or explanation. Feel free to ask me for details if something I said isn't clear.

This is the fastest way I know of to get a Node server running with TypeScript as of 2024. If you know a better way, please do let me know!

Set up the project

# create a new project
mkdir my-node-app
cd my-node-app/

# initialize the project
git init
npm init -y

# install dev dependencies
npm i -D typescript ts-node @types/node

# initialize TypeScript
npx tsc --init

Next, open package.json and add the following:

{
	"engines": {
		"node": ">=20.6.0"
	},
	"name": "my-node-app",
	"version": "1.0.0",
	"description": "",
	"main": "index.js",
	"scripts": {
		"build": "tsc",
		"dev": "node --env-file=.env --watch -r ts-node/register src/index.ts",
		"test": "echo \"Error: no test specified\" && exit 1"
	},
	"author": "Jason Lengstorf <jason@learnwithjason.dev>",
	"license": "ISC",
	"devDependencies": {
		"@types/node": "^20.11.17",
		"ts-node": "^10.9.2",
		"typescript": "^5.3.3"
	}
}

Add environment variables, if needed

Create .env and put any secrets inside:

TEST_VALUE=hello

Create the main Node app file

Create src/index.ts and put something inside:

function test(): void {
	console.log(process.env.TEST_VALUE);
}

test();

Start the Node server and test live reload

Start the server with npm run dev and you'll see the following output:

❯ npm run dev

> my-node-app@1.0.0 dev
> node --env-file=.env --watch -r ts-node/register src/index.ts

(node:29702) ExperimentalWarning: Watch mode is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
hello
Completed running 'src/index.ts'

Make a change to src/index.ts or .env and the server will automatically restart and show your changes in the console.

And that's it — now you've got a Node app with TypeScript running with live reloading and environment variables, using as few dependencies as possible, modernized for building apps in 2024.

Happy building!

I miss RSS

Tue, 16 Jan 2024 00:00:00 GMT

There was this incredible moment in the mid-2000s where it felt like everyone had their own blog, and those blogs all had RSS feeds. I'd start my day by scrolling through the latest posts (RIP Google Reader).

It was sort of like social media today: ideas and insights from my industry peers were available to me in a handy little scrollable list, all in one place.

RSS was missing features... and I think that was good, actually?

RSS feeds didn't have some of the things social media would introduce later:

there was no "like" or "retweet" option
discussion was broken up into individual comment sections on the blogs (if the blog had comments at all)
discovery was harder because there was no Algorithm™ to speak of — you curated your own list (because the feed reader had no opinions or business goals)

But in retrospect, what RSS was missing might have been what made it great:

no likes or retweets meant that the point was to share, not to farm engagement
localized discussion meant the issue of context collapse was much more contained
having a small list meant I actually read things instead of skimming to try and finish reading the whole internet

It was far from perfect, but I can't help but feel like we lost something when we gave up on RSS.

We lost control of what we see

The biggest loss of the last 15-ish years, internet-wise, is that at some point we lost our ability to choose what we see online. Every social site has some kind of recommendation algorithm, and while I understand why this starts, it ultimately turns into "show people the most inflammatory stuff because engagement makes investors and advertisers open their wallets".

Cassidy Williams put this really well in a recent post:

"In the earlier internet days, you went to a fun website or read the latest thing because you decided to go do it. Now, all of this content is pushed in your face, designed to be as addicting as possible, so you keep coming back. You can curate it to a point, but companies design these systems this way on purpose."

That's fine, I guess, but the side effect is that now I see all sorts of junk on social media that I didn't choose to see. Suggested posts and promoted posts often make up more of what I see than the people I actually, y'know subscribe to.

And for every post that gets shoved into my feed that I enjoy, there are a hundred or so that either make me feel sad, angry, or annoyed — when did we decide this was the right default?

I'm not sure what happened, but it feels... bad.

You could run out of stuff to read

This may come as a surprise to folks who haven't been on the internet as long as I have, but there used to be a point where you could run out of things to read online.

It could be frustrating sometimes, sure, but there was a sense of satisfaction that came with knowing that you'd read everything you intended to read. That last entry would get marked as read and you could just, like, do other stuff and not worry that you were being left out of the conversation.

Can RSS make a comeback?

I don't know. I hope so.

This blog has an RSS feed.

I know some of my friends never stopped publishing RSS (examples: Chris Coyier and Cassidy Williams). Sites like YouTube still publish RSS feeds. Even some of our social media provides an RSS feed, with options to convert other feeds like Twitter to RSS if you want!

And there are still RSS readers out there like Feedly and Feeder.

It feels like there's a moment happening where this idea of building our own little spaces on the internet and curating our own little feeds of what our friends and colleagues are up to is maybe — just maybe — something that would work.

So I don't now. Maybe add an RSS feed to your site and let people know about it.

Add user management to a Next.js site with React server components, server actions, and AuthKit

Thu, 30 Nov 2023 00:00:00 GMT

Watch on YouTube

There are lots of options to get auth set up quickly, but they often aren't built to handle the more complex, enterprise features that you hope to grow into over time. In the other direction, user management that's powerful enough for a more mature company is usually so cumbersome and expensive to set up that it's borderline foolish to start with it.

AuthKit caught my attention because its stated intention is to fit the needs of early-stage companies as well as more mature, complex use cases. They have a hosted UI for getting full-featured user management set up in a few lines of code, plus they have a full SDK for building custom user management flows and adding enterprise features like SSO. I love this, because it lines up with my views on progressive disclosure of complexity in software.

In this tutorial, you'll use AuthKit to add user management to a Next.js project using the app directory, React server components, and server actions.

Enable authentication in your WorkOS dashboard

To enable authentication, head to dashboard.workos.com, then click the Authentication tab and make sure that at least one of the methods is enabled.

While you're in here, also make sure to visit the Redirects tab and make sure http://localhost:3000/callback is added as a sign-in callback.

Get the project cloned and running

We'll be starting from a modified version of the official AuthKit example app, which is built using the app directory in Next.js and Radix (the same UI library AuthKit uses to create the hosted UI). We're not going to focus on the UI — instead, we'll only be looking at the specific code that makes user login and session management work in the app.

To get started, clone the app:

# clone the repo using the start branch
gh repo clone jlengstorf/next-authkit-example -- -b start

# move into the directory
cd workos-authkit/

# install dependencies
npm i

Next, copy .env.local.example to .env.local and add the required environment variables:

# Available in your WorkOS dashboard
WORKOS_CLIENT_ID="..."
WORKOS_API_KEY="..."
WORKOS_REDIRECT_URI="http://localhost:3000/callback"

# Your JWT secret key
JWT_SECRET_KEY="..."

To get these values:

Head to dashboard.workos.com and head to the API Keys tab
Copy the Client ID from the top as WORKOS_CLIENT_ID
Click "+ Create Key" to generate a new API key to use as WORKOS_API_KEY
Leave the WORKOS_REDIRECT_URI set to http://localhost:3000/callback — this must match one of the callbacks in your WorkOS dashboard Redirects tab
Set JWT_SECRET_KEY to any value — I used https://www.uuidgenerator.net/

Finally, start up local development to see the app.

npm run dev

Open http://localhost:3000 to see the running site.

Get the URL for signing in or signing up via AuthKit

Our first step is to get the URL that sends a user to AuthKit's hosted sign in/sign up flow. To do that, make the following changes in src/auth.ts:

import { cookies } from "next/headers";
import { redirect } from "next/navigation";
import WorkOS, { User } from "@workos-inc/node";
import { jwtVerify } from "jose";

// TODO Initialize the WorkOS client
// Initialize the WorkOS client
export const workos = new WorkOS(process.env.WORKOS_API_KEY);

export function getClientId() {
  const clientId = process.env.WORKOS_CLIENT_ID;

  if (!clientId) {
    throw new Error("WORKOS_CLIENT_ID is not set");
  }

  return clientId;
}

// TODO get the authorization URL for logging in and signing up with AuthKit
export async function getAuthorizationUrl() {
  const redirectUri = process.env.WORKOS_REDIRECT_URI;

  if (!redirectUri) {
    throw new Error("WORKOS_REDIRECT_URI is not set");
  }

  const authorizationUrl = workos.userManagement.getAuthorizationUrl({
    provider: "authkit",
    clientId: getClientId(),
    // The endpoint that WorkOS will redirect to after a user authenticates
    redirectUri,
  });

  return authorizationUrl;
}

export function getJwtSecretKey() {
  const secret = process.env.JWT_SECRET_KEY;

  if (!secret) {
    throw new Error("JWT_SECRET_KEY is not set");
  }

  return new Uint8Array(Buffer.from(secret, "base64"));
}

// TODO verify that the JWT is valid

// TODO determine whether a user is authenticated and return user details if so

// TODO log the user out by deleting their token

After initializing a new instance of the WorkOS Node SDK, this file also exports a new function called getAuthorizationUrl().

This function uses the user management API — WorkOS's collection of features that make up AuthKit's core functionality. The getAuthorizationUrl() method takes in the client ID and redirect URI (which were set via .env.local earlier) and the provider, which is set to 'authkit' to use the hosted UI.

Put this authorization URL to work by making the following changes in src/app/components/sign-in-button.tsx:

import { Button, Flex } from "@radix-ui/themes";
import { getAuthorizationUrl } from '../../auth';

export async function SignInButton({ large }: { large?: boolean }) {
  // TODO determine login status from AuthKit
  const isAuthenticated = false;

  // TODO get AuthKit authorization URL
  const authorizationUrl = "#login";
  const authorizationUrl = await getAuthorizationUrl();

  if (isAuthenticated) {
    return (
      <Flex gap="3">
        <form
          action={async () => {
            "use server";
            // TODO log the user out
          }}
        >
          <Button type="submit" size={large ? "3" : "2"}>
            Sign Out
          </Button>
        </form>
      </Flex>
    );
  }

  return (
    <Button asChild size={large ? "3" : "2"}>
      <a href={authorizationUrl}>Sign In {large && "with AuthKit"}</a>
    </Button>
  );
}

Save these changes, then click the "Sign In with AuthKit" button on the home page. You'll be redirected to the hosted sign in/sign up flow

The user is authorizing the app, but there's no handler in place to turn the authorization code into a user token. In the next step, we'll implement the callback handler to exchange the code for a user token.

Create a callback handler for user logins

Exchange the code for user details by replacing the contents of src/app/callback/route.tsx with the following code:

import { SignJWT } from "jose";
import { NextRequest, NextResponse } from "next/server";
import { getJwtSecretKey, workos, getClientId } from "../../auth";

export async function GET(request: NextRequest) {
  const code = request.nextUrl.searchParams.get("code");

  if (code) {
    try {
      // Use the code returned to us by AuthKit and authenticate the user with WorkOS
      const { user } = await workos.userManagement.authenticateWithCode({
        clientId: getClientId(),
        code,
      });

      // Create a JWT token with the user's information
      const token = await new SignJWT({
        // Here you might lookup and retrieve user details from your database
        user,
      })
        .setProtectedHeader({ alg: "HS256", typ: "JWT" })
        .setIssuedAt()
        .setExpirationTime("1h")
        .sign(getJwtSecretKey());

      const url = request.nextUrl.clone();

      // Cleanup params
      url.searchParams.delete("code");

      // Redirect to the requested path and store the session
      url.pathname = "/";
      const response = NextResponse.redirect(url);

      response.cookies.set({
        name: "token",
        value: token,
        path: "/",
        httpOnly: true,
      });

      return response;
    } catch (error) {
      return NextResponse.json(error);
    }
  }

  return NextResponse.json({
    error: "No authorization code was received from AuthKit",
  });
}

To exchange the authorization code for user information, this code uses the authenticateWithCode() method.

Next, the user details are encoded into a JSON web token (JWT) using the jose package.

Once a token is created, the current URL is cloned, updated to point to the home page, and the code is deleted. This updated URL is where the user will be redirected after a successful authorization.

Finally, the token is added as an HTTP-only cookie before returning the response.

After saving, click the "Sign In with AuthKit" button again. You'll end up back on the home page, which still needs changes to show a logged in user, but if you check the "application" tab in your devtools and look at cookies, you'll see the token cookie is now present.

With the user token set, the next step is to read it and update the UI based on whether a user is logged in.

Add logic to verify and load a logged in user + log out

Now that a token is set when a user is logged in, add the logic to check and verify that token by making the following changes in src/auth.ts:

import { cookies } from "next/headers";
import { redirect } from "next/navigation";
import WorkOS, { User } from "@workos-inc/node";
import { jwtVerify } from "jose";

// Initialize the WorkOS client
export const workos = new WorkOS(process.env.WORKOS_API_KEY);

export function getClientId() { /* unchanged */ }

export async function getAuthorizationUrl() { /* unchanged */ }

export function getJwtSecretKey() { /* unchanged */ }

// TODO verify that the JWT is valid
export async function verifyJwtToken(token: string) {
  try {
    const { payload } = await jwtVerify(token, getJwtSecretKey());
    return payload;
  } catch (error) {
    return null;
  }
}

// TODO determine whether a user is authenticated and return user details if so
export async function getUser(): Promise<{
  isAuthenticated: boolean;
  user?: User | null;
}> {
  const token = cookies().get("token")?.value;
  const verifiedToken = token && (await verifyJwtToken(token));

  if (verifiedToken) {
    return {
      isAuthenticated: true,
      user: verifiedToken.user as User | null,
    };
  }

  return { isAuthenticated: false };
}

// TODO log the user out by deleting their token
export async function clearCookie() {
  cookies().delete("token");
  redirect("/");
}

The verifyJwtToken() function uses the jose package to make sure the token was created using the secret key set in .env.local.

The getUser() function loads the token cookie, verifies it using verifyJwtToken(), then returns an object with the current authentication status (isAuthenticated) and, when a user is logged in, the user's details such as name and email.

The clearCookie() function deletes the cookie and redirects the user to the home page — this will be used to handle the logout action.

Only show the account details page to logged in users

For protected routes, such as the account details page, use Next.js middleware to check for a valid token and, if none is found, redirect the user to the authorization URL. Replace the contents of src/middleware.ts with the following code:

import { NextRequest, NextResponse } from "next/server";
import { getAuthorizationUrl, verifyJwtToken } from "./auth";

export async function middleware(request: NextRequest) {
  const { cookies } = request;
  const { value: token } = cookies.get("token") ?? { value: null };

  const hasVerifiedToken = token && (await verifyJwtToken(token));

  // Redirect unauthenticated users to the AuthKit flow
  if (!hasVerifiedToken) {
    const authorizationUrl = await getAuthorizationUrl();
    const response = NextResponse.redirect(authorizationUrl);

    response.cookies.delete("token");

    return response;
  }

  return NextResponse.next();
}

// Match against the account page
export const config = { matcher: ["/account/:path*"] };

After saving, delete the token cookie in the Application tab of your devtools and try to visit the account tab — you'll be redirected to the AuthKit authorization flow.

Update the UI to use authentication information

The last step is to make the UI react to the presence of a logged in user.

Add user details to the account page

First, make the following changes to src/app/account/page.tsx:

import { Text, Heading, TextFieldInput, Flex, Box } from "@radix-ui/themes";
import { getUser } from "../../auth";

export default async function AccountPage() {
  const user: any = {};
  const { user } = await getUser();

  const userFields = user && [
    ["First name", user.firstName],
    ["Last name", user.lastName],
    ["Email", user.email],
    ["Id", user.id],
  ];

  return (
    <>
      <Flex direction="column" gap="2" mb="7">
        <Heading size="8" align="center">
          Account details
        </Heading>
        <Text size="5" align="center" color="gray">
          Below are your account details
        </Text>
      </Flex>

      {userFields && (
        <Flex
          direction="column"
          gap="3"
          style={{ width: 400 }}
          justify="center"
        >
          {userFields.map(([label, value]) => (
            <Flex asChild align="center" gap="6" key={value}>
              <label>
                <Text weight="bold" size="3" style={{ width: 100 }}>
                  {label}
                </Text>

                <Box grow="1">
                  <TextFieldInput value={value || ""} readOnly />
                </Box>
              </label>
            </Flex>
          ))}
        </Flex>
      )}
    </>
  );
}

The account page is only available to logged in accounts, so all it needs is the user details returned by getUser().

Turn the "sign in" button to a "sign out" button when logged in

Next, make the following changes to src/app/components/sign-in-button.tsx:

import { getAuthorizationUrl } from "../../auth";
import { clearCookie, getAuthorizationUrl, getUser } from "../../auth";
import { Button, Flex } from "@radix-ui/themes";

export async function SignInButton({ large }: { large?: boolean }) {
  // TODO determine login status from AuthKit
  const isAuthenticated = false;
  const { isAuthenticated } = await getUser();
  const authorizationUrl = await getAuthorizationUrl();

  if (isAuthenticated) {
    return (
      <Flex gap="3">
        <form
          action={async () => {
            "use server";
            // TODO log the user out
            await clearCookie();
          }}
        >
          <Button type="submit" size={large ? "3" : "2"}>
            Sign Out
          </Button>
        </form>
      </Flex>
    );
  }

  return (
    <Button asChild size={large ? "3" : "2"}>
      <a href={authorizationUrl}>Sign In {large && "with AuthKit"}</a>
    </Button>
  );
}

This change uses the actual authenticated status to show a "sign out" button when a user is logged in. It also uses a server action to delete the token cookie when clicked, allowing the user to log out and be redirected to the home page.

Update the home page for logged in users

Finally, make the following changes to src/app/page.tsx:

import { Button, Flex, Heading, Text } from "@radix-ui/themes";
import NextLink from "next/link";
import { SignInButton } from "./components/sign-in-button";
import { getUser } from "../auth";

export default async function HomePage() {
  // TODO determine login status from AuthKit
  const isAuthenticated = false;

  // TODO load user details from AuthKit when logged in
  const user: any = {};
  const { isAuthenticated, user } = await getUser();

  return (
    <Flex direction="column" align="center" gap="2">
      {isAuthenticated ? (
        <>
          <Heading size="8">
            Welcome back{user?.firstName && `, ${user?.firstName}`}
          </Heading>
          <Text size="5" color="gray">
            You are now authenticated into the application
          </Text>
          <Flex align="center" gap="3" mt="4">
            <Button asChild size="3" variant="soft">
              <NextLink href="/account">View account</NextLink>
            </Button>
            <SignInButton large />
          </Flex>
        </>
      ) : (
        <>
          <Heading size="8">AuthKit authentication example</Heading>
          <Text size="5" color="gray" mb="4">
            Sign in to view your account details
          </Text>
          <SignInButton large />
        </>
      )}
    </Flex>
  );
}

This swaps out the hard-coded values for real login status and user details. Save to see the user details displayed on the home page when a user is logged in.

Going further with user management

As your app grows, you may want to add new features or customize how things look. This is where AuthKit is uniquely valuable, as far as I can tell.

Bring your own UI

If you want to customize the UI instead of using a hosted workflow, the user management APIs are available in the WorkOS Node SDK to do anything AuthKit can do within your own UI.

I won't go into how, but they have a ton of examples up on GitHub to get you started.

Add enterprise features

You might have spotted it when you enabled authentication in your WorkOS dashboard, but adding enterprise user management features like single sign-on is a checkbox away. For more details, check out my tutorial on SSO with WorkOS in a Node app.

Resources and further reading

Add audit logging and log streams to a Node Express app

Tue, 14 Nov 2023 00:00:00 GMT

In this tutorial, we're going to learn how to add audit logging and log streams to an existing Node app built with Express using WorkOS.

Watch on YouTube

Get the project cloned and running

This demo adds audit logging to an existing Node app built using Express. We'll focus exclusively on how to configure and send audit logging events via Node middleware using the WorkOS audit logging functionality.

To get started, clone the app:

# clone the repo
gh repo clone learnwithjason/node-express-audit-log-event-stream-workos

# move into the directory
cd node-express-audit-log-event-stream-workos/

# install dependencies
npm i

Next, get the required environment variables:

WORKOS_API_KEY=""
WORKOS_CLIENT_ID=""
WORKOS_REDIRECT_URI="http://localhost:3000/auth/callback"
WORKOS_ORG_ID=""
WORKOS_DIRECTORY_ID=""
WORKOS_WEBHOOK_SECRET=""

SESSION_SECRET=""

These environment variables are all part of getting the SSO and SCIM integrations running. Once these are in place, there are no additional credentials required to add audit logging.

With the environment variables saved, start the app locally:

npm run dev

This will open the app at http://localhost:3000. Open it in your browser and log in with your SSO credentials to see the dashboard.

What is audit logging?

On bigger teams, knowing who did what inside the app is a big deal. It creates accountability and makes sure the company is confident that every action can be attributed to one of their team members.

That's what audit logging is designed to solve: provide a uniform API for keeping track of who did what inside the app to ensure there's a paper trail.

Practically speaking, an audit log is a JSON object with data about who did what to which parts of the app that gets sent as an event. These events are stored somewhere (in our case, in WorkOS), and can be further sent (via log streams) to destinations like Amazon S3, Datadog, or Splunk.

In WorkOS, an audit logging event looks something like this:

{
  "actor": {
    "id": "00ua...",
    "type": "user"
  },
  "action": "user.login",
  "context": {
    "location": "0.0.0.0",
    "user_agent": "Mozilla/5.0 ..."
  },
  "targets": [
    {
      "id": "00ua...",
      "type": "user"
    },
    {
      "id": "directory_group_01...",
      "type": "team"
    }
  ],
  "occurred_at": "2023-11-07T00:21:32.898Z"
}

There are 5 parts to it:

actor — who is taking the action? Providing a type (e.g. user or system) along with an identifier provides auditors with the ability to identify who did the thing.
action — what was done? This is a unique identifier chosen by whoever sets up audit logging for the team.
context — location and browser information about the actor
targets — an array of what was acted upon. For example, an event might include the post that was affected, along with the user's team. These targets can be used to group events during audits (e.g. “show me everything that’s happened to this post”).
occurred_at — a timestamp for when the action happened.

The good news is that after a bit of initial setup, audit logging is as straightforward as console logging in an Express app. Let's build it!

Create a new file for audit logging middleware

Usually, audit logging is being added to an already existing app. Because of that, it's likely that we already have most of the information we need configured: the user's information, the actions they can take, and so on.

In Express, this information can be added to session storage, and since much of what we'll need is going to be the same for every event (e.g. the user's ID), we can use Express middleware to provide a helper function for logging that will have most of the work already done for the dev — it makes the right thing the easy thing, which (hopefully) means it will actually get used without someone needing to hound the dev team to implement it.

Set up this middleware by creating a new file at src/middleware/audit-logging.js and add the following code inside:

const { WorkOS } = require('@workos-inc/node');

const workos = new WorkOS(process.env.WORKOS_API_KEY);

exports.auditLoggingMiddleware = (req, res, next) => {
	// get the IP and user agent of the request, if available
	const ip = req.headers['x-forwarded-for'] ?? '';
	const userAgent = req.headers['user-agent'] ?? '';

	req.log = async ({ action, targets = [] }) => {
		if (!req.session.user || !req.session.user.idpId) {
			console.error('unable to send audit log events without a valid user');
			return;
		}

		const userId = req.session.user.idpId;
		const groups = req.session.user.groups ?? ['missing'];

		// all events get attached to the user and team(s) they’re part of
		const defaultTargets = [
			{ type: 'user', id: userId },
			...groups.map((groupId) => ({ type: 'team', id: groupId })),
		];

		await workos.auditLogs.createEvent(process.env.WORKOS_ORG_ID, {
			action,
			actor: {
				type: 'user',
				id: userId,
			},
			occurredAt: new Date(),
			targets: defaultTargets.concat(targets),
			context: {
				location: ip,
				userAgent,
			},
		});
	};

	next();
};

After getting a new instance of the WorkOS Node SDK, this file exports a new middleware function called auditLoggingMiddleware.

Inside, it tries to grab the IP address and user agent from the current request, then attaches a new function called log to the request before continuing the request with next(). This middleware will run before all routes, which means every route will now have access to req.log() for sending audit logging events.

The req.log function accepts an options argument with the name of the action and an optional array of targets. Everything else required will already be present in the session thanks to the auth middleware. (We'll look at specifically what's needed in the next section.)

Inside req.log, the function will:

Check for a logged in user and bail if none is found
Grab the user's ID and group(s) from the session
Create default targets for the user and each group the user belongs to
Call the WorkOS SDK's createEvent method to send an audit logging event

createEvent requires two arguments:

The organization ID to attach the event to
The event to log

The event itself is mostly populated from existing data. The only things that the developer needs to provide are the action type and an optional array of additional targets to attach the event to.

This setup makes it far less cumbersome to send an audit logging event. In the minimum use case, all the developer needs to do is something like this in a route:

req.log({ action: 'some.action' });

This will create an event that's properly associated with the user and their group(s), along with additional information to help auditors.

To add additional targets, the developer only needs the ID to send along:

req.log({
	action: 'some.action',
	targets: [
		{
			type: 'document',
			id: documentId,
		},
	],
});

Now that we've got this set up, let's add it to the app.

Use the custom Express middleware for audit logging

In src/index.js, add the following code:

	const { join } = require('node:path');
	const express = require('express');
	const session = require('express-session');
	const pgSimple = require('connect-pg-simple');

	const app = express();
	const sessionStore = pgSimple(session);
+	const { auditLoggingMiddleware } = require('./middleware/audit-logging');

	app.set('view engine', 'ejs');
	app.set('views', join(__dirname, 'views'));

	app.use(
		session({
			secret: process.env.SESSION_SECRET,
			resave: false,
			saveUninitialized: false,
			store: new sessionStore({
				createTableIfMissing: true,
				conObject: {
					database: 'postgres',
				},
			}),
		}),
	);
	app.use(express.static(join(__dirname, 'static')));
	app.use(express.json());
	app.use(express.urlencoded({ extended: true }));

+	app.use(auditLoggingMiddleware);

	app.use('/', require('./routes/public'));
	app.use('/auth', require('./routes/auth'));
	app.use('/dashboard', require('./routes/dashboard'));
	app.use('/api', require('./routes/api'));

	const port = 3000;
	app.listen(port, () => {
		console.log(`app listening at http://localhost:${port}`);
	});

Once this is added, every route in the app will have access to the req.log method!

Make sure the required data is available in the session

Because we want to attach audit logging events to both the user and the user's teams, we need to make sure the appropriate data is added to the user session.

Inside src/routes/auth.js, find the req.get('/callback', ...) route and make the following changes:

	router.get('/callback', async (req, res) => {
		const { code } = req.query;

		const { profile } = await workos.sso.getProfileAndToken({
			code,
			clientID: process.env.WORKOS_CLIENT_ID,
		});

		// store the user in the app database
		const user = await db.getUserByEmail(profile.email);
+		const dirUser = await workos.directorySync
+			.listUsers({
+				directory: process.env.WORKOS_DIRECTORY_ID,
+			})
+			.then(({ data }) => data.find((u) => u.idpId === profile.idpId));

		req.session.user = profile;
		req.session.user.id = user.id;
+		req.session.user.idpId = profile.idpId;
		req.session.user.roles = user.roles;
+		req.session.user.groups = dirUser.groups.map((g) => g.id);

		res.redirect('/dashboard');
	});

Users in this app log in with SSO and are synced with a user directory using SCIM, so we use WorkOS's directory sync to look up all the users and match the current user to get access to all their current groups.

Next, we attach the identity provider ID and an array of group IDs to the user's session object.

Define audit logging events in WorkOS

Next, head over to your WorkOS dashboard, navigate to Configuration > Audit Logs, and click "+ Create an event" to add all the events required for this app. This demo uses 5 event types:

user.login, with target types of user and team
user.logout, with target types of user and team
user.invite, with target types of user and team
post.create, with target types of post, user and team
post.delete, with target types of post, user and team

Once these are created, we can add the audit log calls inside our app.

Add audit logging to an Express app

In any route that should be logged, add a new call to the req.log method.

In src/routes/api.js, update the /delete-post/:post_id route:

	router.get('/invite/:id', async (req, res) => {
		// TODO implement invite system
		console.log(`Invited user: ${req.params.id}`);
+		req.log({
+			action: 'user.invite',
+			targets: [{ type: 'user', id: String(req.params.id) }],
+		});
+
		res.redirect('/dashboard/team');
	});

	 router.get('/delete-post/:post_id', async (req, res) => {
+		req.log({
+			action: 'post.delete',
+			targets: [{ type: 'post', id: String(req.params.post_id) }],
+		});
+
		await db.query(
			db.sql`
				DELETE FROM posts
				WHERE id = $1
			`,
			[req.params.post_id],
		);

		res.redirect('/dashboard');
	});

In src/routes/auth.js, update the /callback and /logout routes:

	router.get('/callback', async (req, res) => {
		const { code } = req.query;

		const { profile } = await workos.sso.getProfileAndToken({
			code,
			clientID: process.env.WORKOS_CLIENT_ID,
		});

		// store the user in the app database
		const user = await db.getUserByEmail(profile.email);
		const dirUser = await workos.directorySync
			.listUsers({
				directory: process.env.WORKOS_DIRECTORY_ID,
			})
			.then(({ data }) => data.find((u) => u.idpId === profile.idpId));

		req.session.user = profile;
		req.session.user.id = user.id;
		req.session.user.idpId = profile.idpId;
		req.session.user.roles = user.roles;
		req.session.user.groups = dirUser.groups.map((g) => g.id);

+		await req.log({ action: 'user.login' });
+
		res.redirect('/dashboard');
	});

-	router.get('/logout', (req, res) => {
+	router.get('/logout', async (req, res) => {
+		await req.log({ action: 'user.logout' });
+
		req.session.user = null;
		req.session.save();

		res.redirect('/');
	});

And finally, in src/routes/dashboard.js, update the /new route for POST requests:

	router.post('/new', async (req, res) => {
		const { title, content } = req.body;
		const user_id = req.session.user.id;

		try {
			const result = await db.query(
				db.sql`
					INSERT INTO posts (user_id, title, content)
						VALUES ($1, $2, $3)
+						RETURNING id;
				`,
				[user_id, title, content],
			);

			console.log(result);
+			await req.log({
+				action: 'post.create',
+				targets: [{ type: 'post', id: String(result.rows.at(0).id) }],
+			});
		} catch (err) {
			console.error(err);
		}

		res.redirect('/dashboard');
	});

Perform actions and check the log in WorkOS

After saving, try logging in, logging out, creating and deleting posts, or inviting a team member. After the actions have been performed, head to your WorkOS dashboard and view the log for your organization. You'll see audit logging events showing up in real time!

Add log streams to integrate with your Security Incident and Event Management (SIEM) provider

No team wants to open a bunch of different dashboards to monitor the dozens of tools used by the company. Setting up log streams allows them to bring all the logs together into a central SIEM provider, and it all happens without any code changes required.

Inside WorkOS, go to your organization and click "Configure manually" under Log Streams, then choose your SIEM provider (currently supported: Amazon S3, Datadog, and Splunk), then add the configuration details on the next screen to get logs streaming to your provider.

And that's it! Log streams are handled and your teams don't need to add another dashboard to their list.

Audit logging and log streaming are important to larger customers

Remember: as companies grow, features like audit logging and log streams start to become critical factors in their decision to adopt a new product.

These kinds of features maybe don't feel exciting to implement, but you know what is exciting? Closing a six-figure deal with a huge new customer.

Thanks again to WorkOS for sponsoring this tutorial. Happy building, friends!

Resources & further reading

Animated CSS gradient borders (no JavaScript, no hacks)

Sat, 04 Nov 2023 00:00:00 GMT

Watch on YouTube

So much cool stuff has been happening in CSS lately — if you haven’t looked at it in a few years, it might be worth giving things another shot. A lot has changed!

Here’s what you’ll build by the end of this article.

CodePen: Gorgeous animated gradient borders using only CSS

To follow along, create your own CodePen or open an HTML document where you can modify the HTML and CSS.

Set up the HTML markup

Because of the way these borders are set up, we don’t need any extra container elements. We can use straightforward semantic markup.

<main>
  <article>
    <h1>Hey look, this is only CSS!</h1>
    <p>
      I didn’t know you could do gradient borders like this. Hover over this
      element to see the gradient animate!
    </p>
  </article>
</main>

The article will be the element that we style with the border.

Add base styles

This step is optional. To make our demo look a little nicer, let’s add some base styles:

a basic reset
background colors
centering and spacing
miscellaneous touches to make it feel nice

* {
  box-sizing: border-box;
}

html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica,
    Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';
  font-size: 18px;
  line-height: 1.45;
}

body {
  margin: 0;
}

main {
  background: radial-gradient(
      circle,
      oklch(0.15 0.2 330 / 0),
      oklch(0.15 0.2 330 / 1)
    ), linear-gradient(344deg in oklch, oklch(0.3 0.37 310), oklch(0.35 0.37 330), oklch(0.3
          0.37 310));
  display: grid;
  height: 100svh;
  place-items: center;
}

article {
  border-radius: 1rem;
  box-shadow: 0.125rem 0.25rem 0.25rem 0.5rem oklch(0.1 0.37 315 / 0.25);
  color: white;
  padding: 1rem;
  width: min(400px, 90vw);

  & h1 {
    line-height: 1.1;
    margin: 0;
  }

  & p {
    margin: 0.75rem 0 0;
  }
}

Things won’t quite look right yet, but you’ll be looking much better than the default styles.

Create a gradient border using `background-origin` and `conic-gradient()`

To add the gradient border, we need to add two backgrounds to the article:

one to be the solid, interior background (the dark color that makes the text visible)
one to be the gradient border (a conical gradient)

To add these, add the following styles:

article {
  background: linear-gradient(
        to bottom,
        oklch(0.1 0.2 240 / 0.95),
        oklch(0.1 0.2 240 / 0.95)
      ) padding-box, conic-gradient(
        from 0deg in oklch longer hue,
        oklch(1 0.37 0) 0 0
      ) border-box;

  border: 6px solid transparent;
}

After saving, the border will appear!

Let’s break down how this works.

Use multiple backgrounds

We add two backgrounds to our element:

The first background is the solid background inside the element that the text is placed on top of.

The second background is the gradient that will be used for the border.

Use `background-origin` to place the gradient in the border

A setting I’d never given much thought to before was background-origin. This tells the browser where the background should be contained (e.g. should it go to the border? the end of the padding? just the content area?).

By setting our first background to padding-box and the second to border-box, the gradient background extends further than the inner background, allowing it to become a “border”.

This means we control the border mostly in the usual way: we set a border property and choose a width. The only difference is that we set the border’s color to transparent, which allows the gradient background to shine through.

Use OKLCH for better-looking gradients

I had so much to say about why OKLCH is my new favorite way to do color in CSS that I had to split it out into its own post.

The gist is this: gradients in CSS used to look bad, but HSL and LCH make them look good. This is partly due to how gradients are calculated in different color spaces, and partly due to LCH’s ability to represent about 50% more colors.

In our gradient code, we set a super bright pink using oklch(1 0.37 0).

Use a `conic-gradient()` to have the gradient “wrap” the whole border

A conical gradient starts from a center point and goes around it in a circle (or, you know, a cone). By placing the gradient in the middle of our container, the parts that are visible at the border appear to follow the border itself, effectively “wrapping” the component in a gradient.

Pretty nice, right?

But the syntax looks a little wild at first, so let’s break down what’s happening.

    conic-gradient(
        from 0deg in oklch longer hue,
        oklch(1 0.37 0) 0 0
      )

There’s a surprising amount of information packed into these four lines of code, so let’s break it down piece by piece, starting with the first line:

from 0deg — this is the starting angle of the gradient
in oklch — this tells CSS to use the OKLCH color space to calculate the gradient (since we also use oklch() to set the color, this isn’t stricly necessary, but it doesn’t hurt to be explicit)
longer hue — I learned this from Temani Afif, who pointed me to Adam Argyle’s illustrated explanation of how hues are calculated in gradients
- tl;dr: hues are a circle, and we can tell CSS to take the long way around the circle to calculate the gradient, resulting in additional colors

Next, let’s take a look at the second line:

oklch(1 0.37 0) — this is our starting color, which is a vivid pink
0 0 — this is shorthand for “use the same color” (another tip from Temani)

Because we’re using the longer hue setting, creating a conic-gradient() with the same start and end color gives us a full spin around the color wheel, resulting in a very colorful gradient with a single line of code!

Animate the gradient border

To animate the border, we need exactly one thing to change: the angle of the gradient. By animating the starting angle from 0deg to 360deg, we get an infinitely spinning animated gradient border.

This is harder than it looks — why doesn’t this work?

This is deceptively hard, though. My original thought was that I could do something with a CSS custom property:

/* this doesn’t work? why?! */
:root {
  --bg-angle: 0deg;
}

@keyframes spin {
  to {
    --bg-angle: 360deg;
  }
}

This approach results in an animation that “pops” from one state to the next, so it looks like nothing is happening. If you change the keyframes value to 180deg, though, you can see what’s actually happening more clearly.

There was no way I knew of to solve this without JavaScript. At least, until very recently.

`@property` makes angle interpolation work in CSS

However, @property makes interpolation work by telling CSS what kind of unit is stored in the custom property — this means animations work smoothly! It’s in all modern browsers now with the exception of Firefox, which will release support on December 19, 2023.

The main difference is that we can give the custom property a type. Here’s how it works:

@property --bg-angle {
  inherits: false;
  initial-value: 0deg;
  syntax: '<angle>';
}

The syntax is the secret sauce that makes this work. By specifying "<angle>", CSS knows how to interpolate changed values in animation, which means the animations can be smooth now.

Update the conic-gradient() to use the custom property and add the animation — but let’s start out the animation as paused. We only want to set the animation-play-state to running when the element is hovered.

+	@property --bg-angle {
+		inherits: false;
+		initial-value: 0deg;
+		syntax: "<angle>";
+	}
+
+	@keyframes spin {
+		to {
+			--bg-angle: 180deg;
+		}
+	}
+
	article {
+		animation: spin 1.5s linear infinite paused;
		background: linear-gradient(
					to bottom,
					oklch(0.1 0.2 240 / 0.95),
					oklch(0.1 0.2 240 / 0.95)
				)
				padding-box,
			conic-gradient(
-					from 0deg in oklch longer hue,
+					from var(--bg-angle) in oklch longer hue,
					oklch(1 0.37 0) 0 0
				)
				border-box;

		border: 6px solid transparent;
+
+		&:hover {
+			animation-play-state: running;
+		}
	}

Once you’ve made these changes, you should see the gradient borders and they’ll animate when you hover!

CodePen: Gorgeous animated gradient borders using only CSS

It blows my mind that this kind of stuff is not only possible in CSS these days, but that we can do it without complex hacks, pseudo-elements, hidden elements, or other messy implementations that always felt fragile to me.

Resources & Further Reading

OKLCH for better color in the browser

Sat, 04 Nov 2023 00:00:00 GMT

Something that has always bugged me about CSS is the dullness of most gradients.

You’d have this beautiful idea for your design, set up the gradient in your CSS, and when it rendered in the browser it was just... blech.

The middle gets all muddy and the whole thing just feels kinda flat and lifeless.

But watch what happens when you do the exact same gradient, but in the OKLCH color space, which I learned was available to use from Chris Coyier’s post crowning OKLCH as the “winner” for color in CSS:

Now that looks great!

So... what exactly is happening here, and why does OKLCH make such a big difference?

I am not the expert here

Before I start explaining anything: I’m not an expert in this stuff. I read a bunch of material written by actual experts, and I’m collecting what I learned here in hopes that some of the knowledge gets caught in my brain wrinkles along the way.

If you want to read the experts, check out the links and resources at the end of this post.

What the heck is a color space?

There’s an entire field of study behind color spaces, so let me gloss over most of it and say: color spaces are the math computers use to turn stuff like rgb(100 200 0 / 0.5) into color on the screen.

The default browser color space, sRGB, is called a rectangular color space. In addition to srgb, browsers also support rectangular color spaces srgb-linear, lab, oklab, xyz, xyz-d50, and xyz-d65.

OKLCH, however, is a polar color space. The browser can support several polar color spaces: hsl, hwb, lch, and oklch.

What does color space change how gradients look?

From what I understand, the reason gradients look so much better in OKLCH is that rectangular color spaces draw a straight line through the color space between the gradient colors, where polar color spaces draw an arc through the hues to get there.

That’s why the same gradient looks muddy in srgb and looks great in oklch. To (very roughly) visualize what’s happening with a gradient, take a look at the lines drawn between the two colors from our gradients above:

So in a rectangular color space, the interpolation takes the middle colors from the muddy middle of the color wheel, which Adam Argyle lovingly calls “the dead zone” in this article that has more information about color than anyone could ask for.

In a polar color space the arc keeps the gradient along the hues at roughly the same distance from the “center” (again: lots of math happening here that I’m skimming past).

OKLCH is good for more than just gradients, though

On top of making gradients look actually good, OKLCH moves us into a color space that literally has more colors in it. sRGB is limited because when it was defined, monitors couldn’t display all the colors it defined. These days, though, we’re getting HDR monitors that support nearly double the colors, and OKLCH lets us start using them.

The short version is that if we switch to OKLCH, we get a larger palette of brighter, more vivid colors. You can see the extra colors visualized in Evil Martian’s OKLCH color picker (note the difference between the selected color and the fallback).

The long version is far better explained in Lea Verou’s post on LCH in CSS.

See the OKLCH difference for yourself

I threw together a quick CodePen showing how a gradient looks in all the different color spaces. Edit this to change out the colors and see how each color space handles it.

CodePen: Visualizing all CSS color-interpolation-methods in gradients

I’ll let you draw your own conclusions since this all comes down to preference, but for me, OKLCH will be my new default choice for colors in the browser.

Resources & Further Reading

The killer tool for devs who don't want to touch design is... Wix?

Mon, 23 Oct 2023 00:00:00 GMT

Watch on YouTube

A lot of developers I've spoken to really enjoy working with the data layer of web apps, but wish they never had to touch UI code. If you're one of those developers, Wix might actually be an excellent fit for your dev stack.

Wait, you mean THAT Wix?

If you thought Wix was only for no-code sites, you're not alone. Up until the team at Wix hit me up and asked if I'd be interested in trying out their developer tools, I thought the same thing.

Wix Studio, which Wix describes as "a new end-to-end web creation platform for freelancers and agencies", gives us access to a new tools and a library of developer APIs that let us handle the backend of our client sites while letting our clients continue managing their site's look and feel in a visual, no-code interface.

What does Wix Studio offer developers?

When I started looking through the developer-focused tools Wix Studio offers, two things caught my eye:

There's a VS Code-based Wix IDE that lets you work on you Wix site code in a familiar interface (if you're a VS Code user).
There's a set of APIs called Velo that provides JavaScript APIs for modifying Wix element data, handling animation, auth, and all sorts of other advanced controls I wasn't expecting to see.

I'm not going to get into all the details during this build, so if you're interested go hit up their docs to see all the things you're able to do.

What we'll cover in this tutorial is a first look at how a developer can manage the data layer of a client's Wix site using an IDE and APIs that are familiar and surprisingly nice to work with.

The setup: create a Wix site with a Pro Gallery element

To get started, we need a Wix site we can add custom code to. For my demo, I created a new site with the "start from a blank canvas" option, then added a Pro Gallery element to the page.

That's really all we need. You can play with it a bit, but really the point here is that we don't need to worry about what it looks like. The client will do whatever they want to on the presentation side; we only need to worry about getting the right image data into the gallery.

Enable the Wix Studio developer tools

To get started with coding, click the curly bois ({}) in the left-hand nav and click the "Start Coding" button.

This opens up the Wix Studio code tools, which are fine, but what I really want is that VS Code-based IDE — so click the "Code in Wix IDE" button at the top of the Page Code section.

This opens the familiar VS Code UI right in the browser, which is pretty slick.

Use code to set Wix gallery element images

Now that we've got a site and we're in the Wix IDE, let's use the Wix Editor Elements Velo API to tell the page which images should be displayed in the gallery.

In the explorer, go into the src/pages/ directory and look for the file called Home with a random-looking suffix and a .js extension. In my site, the file is src/pages/Home.c1dmp.js.

Inside, replace the contents with the following code:

// docs: https://www.wix.com/velo/reference/$w/gallery
$w.onReady(function () {
  // @ts-expect-error known issue where IFrame type is inferred incorrectly
  $w('#gallery1').items = [
    {
      type: 'image',
      title: 'Yellow Tracksuit',
      description: 'Photo by Dom Hill',
      alt: 'woman in yellow tracksuit standing on basketball court side',
      src: 'https://images.unsplash.com/photo-1515886657613-9f3515b0c78f?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8Mnx8ZmFzaGlvbnxlbnwwfHwwfHx8MA%3D%3D&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/nimElTcTNyY',
    },
    {
      type: 'image',
      title: 'Striped V-Neck Tee',
      description: 'Photo by Ayo Ogunseinde',
      alt: 'black haired man making face',
      src: 'https://images.unsplash.com/photo-1463453091185-61582044d556?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8N3x8YmxhY2slMjBtYW58ZW58MHwwfDB8fHww&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/sibVwORYqs0',
    },
    {
      type: 'image',
      title: 'Coral Blouse',
      description: 'Photo by Eye for Ebony',
      alt: 'woman standing in front of red brick wall',
      src: 'https://images.unsplash.com/photo-1502764613149-7f1d229e230f?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MTF8fGJsYWNrJTIwd29tYW58ZW58MHwwfDB8fHww&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/OExQjtxbIpE',
    },
    {
      type: 'image',
      title: 'Pinstripe Parachute Pants',
      description: 'Photo by Ahmed Carter',
      alt: 'posing woman in white sleeveless top',
      src: 'https://images.unsplash.com/photo-1509631179647-0177331693ae?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MTR8fGZhc2hpb258ZW58MHx8MHx8fDA%3D&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/posing-woman-in-white-sleeveless-top-tiWcNvpQF4E',
    },
    {
      type: 'image',
      title: 'White Dress',
      description: 'Photo by Vladimir Yelizarov',
      alt: 'woman in white dress sitting on white textile',
      src: 'https://images.unsplash.com/photo-1617551307578-7f5160d6615e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MTh8fGJsYWNrJTIwd29tYW58ZW58MHwwfDB8fHww&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/PfbItsR8Gqo',
    },
    {
      type: 'image',
      title: 'Wool Overcoat',
      description: 'Photo by Alexandru Zdrobău',
      alt: 'woman holding brown leather bag in bokeh photography',
      src: 'https://images.unsplash.com/photo-1485968579580-b6d095142e6e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MzF8fGZhc2hpb258ZW58MHx8MHx8fDA%3D&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/juESZxMhtXk',
    },
    {
      type: 'image',
      title: 'Gray Jumper',
      description: 'Photo by Judeus Samson',
      alt: 'woman standing in the middle of the road',
      src: 'https://images.unsplash.com/photo-1545291730-faff8ca1d4b0?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MjR8fGZhc2hpb258ZW58MHx8MHx8fDA%3D&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/0UECcInuCR4',
    },
    {
      type: 'image',
      title: 'Leather Jacket',
      description: 'Photo by Dami Adebayo',
      alt: 'man in brown leather coat',
      src: 'https://images.unsplash.com/photo-1487222477894-8943e31ef7b2?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8Mjh8fGZhc2hpb258ZW58MHx8MHx8fDA%3D&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/k6aQzmIbR1s',
    },
    {
      type: 'image',
      title: 'Wool Sweater',
      description: 'Photo by Maia Habegger',
      alt: 'woman wearing brown sweater standing on forest',
      src: 'https://images.unsplash.com/photo-1520508358701-63027028d924?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8M3x8d29tYW4lMjBzd2VhdGVyfGVufDB8MHwwfHx8MA%3D%3D&auto=format&fit=crop&w=1200&q=60',
      link: 'https://unsplash.com/photos/mAg7Dz1IQQU',
    },
  ];
});

If you've ever used jQuery or the browser's built-in querySelector API, this might look familiar.

The $w() function accepts a CSS selector as an argument, and in the Wix IDE the available elements will autocomplete once you type the quotes. Every Wix element automatically gets an ID that matches its type, so if you create your first gallery element, the ID will be #gallery1.

After selecting the gallery, we can set the items to be displayed by providing an array to the .items property.

Each item in the array is an object containing either image or video information. In this example we're only providing images.

The nice thing about this API is that there's barely any code required to insert our images — nearly all of this code is the image data itself.

See the custom-coded images in the site

To test that the images are loading as expected, go back to the Wix Studio and click the eye icon at the very top right of the UI, next to the "Publish" button.

Assuming you used the same images from the code sample above, your gallery will look similar to the one in the screenshot above. (The specific layout will depend on options you selected for the gallery element.)

With that, you've successfully custom coded images into your client's Wix site. They can change all the settings, move the layout around, and whatever else they want to do, and you as the developer only need to worry about setting this one call to $w() up.

Going beyond hard-coded images

You're not limited to hard-coded image data, either — you can grab images from any API and display them in your Wix gallery elements.

For example, here's how you would display a list of corgi images from the Dog API using the web standard Fetch API:

// docs: https://www.wix.com/velo/reference/$w/gallery
$w.onReady(async function () {
  const res = await fetch('https://dog.ceo/api/breed/corgi/images');

  if (!res.ok) {
    console.error('something went wrong loading the images');
    return;
  }

  const data = await res.json();
  const images = data.message.map((image, i) => {
    return {
      type: 'image',
      title: `Corgi ${i}`,
      description: 'Images from the Dog API',
      alt: 'a corgi',
      src: image,
      link: 'https://dog.ceo/dog-api/documentation/breed',
    };
  });

  // @ts-expect-error known issue where IFrame type is inferred incorrectly
  $w('#gallery1').items = images;
});

After saving, preview the site to see the gallery images update.

No-code for clients doesn't mean no-code for devs

We only looked at a single workflow in this tutorial, but the fact that we can work in a VS Code-based IDE, with autocomplete and a familiar, selector-based API, is a welcome surprise to devs like me who thought choosing Wix mean it was impossible to make code changes to client sites.

And, for those devs who never want to touch the design parts of web dev, Wix might actually be a welcome addition to the way they work with clients.

Resources & Further Reading

SCIM Provisioning in Node Express Using WorkOS and Okta

Tue, 10 Oct 2023 00:00:00 GMT

Watch on YouTube

Prerequisites

This tutorial extends an app that uses Single Sign-On (SSO) for user auth. If you're not familiar with SSO or how to implement it, you can see the implementation in the source code or learn how to implement SSO in a Node app in this tutorial.

If you want to build along with this tutorial, you'll need:

Node v20.6.0 or later
PostgreSQL available in your development environment (I used v14.9)
A WorkOS account (you can sign up without a credit card and build in dev mode for free)
An Okta account to use as your identity provider (a dev account is free while you build)
ngrok or a similar tool for exposing your dev environment via URL (for webhook testing)

Add SCIM support to a Node Express app

WorkOS calls this Directory Sync. Quote:

System for Cross-domain Identity Management (or SCIM) is an open standard for managing automated user and group provisioning.

Set up your development environment

Clone the repo

# clone the repo on the start branch (using the GitHub CLI: https://cli.github.com)
gh repo clone learnwithjason/node-express-scim-workos-example -- -b start

# move into it and install dependencies
cd node-express-scim-workos-example/
npm i

Get environment variables

For WorkOS SSO to work, you'll need to have your identity provider linked to your WorkOS account for single sign-on. If you're not sure how to do this, I have a Node SSO with WorkOS tutorial that walks though how to set up SSO using Okta as your identity provider.

This app already has the SSO flow set up, so all you'll need are your WorkOS credentials and redirect URI.

Rename .env.EXAMPLE to .env, then update the following values:

+	WORKOS_API_KEY="sk_test_..."
+	WORKOS_CLIENT_ID="client_..."
+	WORKOS_REDIRECT_URI="http://localhost:3000/auth/callback"
+	WORKOS_ORG_ID="org_..."
	WORKOS_DIRECTORY_ID=""
	WORKOS_WEBHOOK_SECRET=""

+	SESSION_SECRET="secret sauce"

WORKOS_API_KEY is found on your WorkOS Dashboard under the API keys section
WORKOS_CLIENT_ID is found on the Configuration section of your dashboard
WORKOS_REDIRECT_URI is also found in the Configuration section
- Note that you can make this any path you prefer, but if you change it from the value shown above you'll need to update the route in src/routes/auth.js to match
WORKOS_ORG_ID can be found in the Organizations tab at the top of the org you set up for SSO
SESSION_SECRET is any string value — this is used by the session middleware to protect user sessions from tampering

We'll add both WORKOS_DIRECTORY_ID and WORKOS_WEBHOOK_SECRET as part of this tutorial, so leave them blank for now.

Start the app locally

With the environment variables updated, start the app:

npm run dev

The app will start, and you'll be able to access it at http://localhost:3000. Open it in your browser and log in via SSO.

Add Directory Sync using SCIM

Add a new directory in WorkOS

In your WorkOS dashboard, find the organization you want to set up Directory Sync for, then open the Actions dropdown and choose "Add Directory".

Choose Okta as your provider, then choose a display name (I chose "Okta").

On the next screen, the "Directory Details" panel will show you the custom endpoint and bearer token that you'll need to configure Okta.

Grab the "Directory ID" from the top of this page — this is what you'll need in the .env. as the value of WORKOS_DIRECTORY_ID.

	WORKOS_API_KEY="sk_test_..."
	WORKOS_CLIENT_ID="client_..."
	WORKOS_REDIRECT_URI="http://localhost:3000/auth/callback"
	WORKOS_ORG_ID="org_..."
+	WORKOS_DIRECTORY_ID="directory_..."
	WORKOS_WEBHOOK_SECRET=""

	SESSION_SECRET="secret sauce"

Enable SCIM provisioning in Okta

Next, open your Okta dashboard and choose the application you've configured for SSO. (Or, if you prefer, create a new application.)

On the General tab of your app's home page, click the "Edit" option in App Settings, then check the box to "Enable SCIM provisioning".

Configure SCIM provisioning in Okta

The app will now have a Provisioning tab. Click that, then click the Edit option for the SCIM connection. And set the following options:

For the "SCIM connector base URL", use the endpoint from the WorkOS directory details from earlier
For "Unique identifier field for users", use "email"
Check the boxes for "Push New Users", "Push Profile Updates", and "Push Groups"
Change the "Authentication Mode" to "HTTP Header"
Add the Bearer Token from the WorkOS directory details to the "Authorization" field

On the next page, click the Edit option for "Provisioning to App" and check the boxes for:

Create Users
Update User Attributes
Deactivate Users

Assign the app to people in your company

Next, navigate to your Okta app's Assignments tab. Open the "Assign" dropdown, then choose "Assign to Groups" (you can also manually assign to each person if you prefer). Since my Okta account is a test instance, I assigned the app to everyone, but you can do whatever makes the most sense for your testing.

For this app specifically, create a new group called "Authors" and assign at least one person to that group for testing. Assign this group to the app as well.

Push groups to WorkOS

With the groups assigned, it's time to try things out. Go to the "Push Groups" tab in your Okta app's dashboard, then open the "Push Groups" dropdown and choose "Find groups by name". Find "Authors" in the dropdown, ensure the "Push group memberships immediately" box is checked, and click save.

Once the push status changes to "Active", check your WorkOS directory and you'll see that the group and its users have been synced to WorkOS. That means SCIM is set up properly and Directory Sync is working!

Any changes in your Okta directory will now update WorkOS as well. In the next section we'll set up a webhook so our app is also updated to stay in sync with user permissions.

Set up a webhook to sync directory changes to a Node app

Now that Okta events are syncing to WorkOS, we need to ensure that our app updates whenever users or groups change.

In our app, we rely on WorkOS for user login, so if a user's account is deactivated, we need our app to make sure they're logged out immediately (i.e. their next request will bounce them out to the login screen).

We're also going to rely on groups for permissions in the app (i.e. authors can create and delete their own posts, but not delete the posts of others, but admins can delete anyone's posts). This means that we need to immediately sync group changes so that a user's membership in a permission group is up-to-date on every request.

To do that, we need to set up a webhook endpoint in our app, then tell WorkOS to send all directory sync events to that endpoint.

Set up a webhook endpoint in the Node app

In the example app, a stubbed out endpoint already exists at /api/directory-sync (defined in src/routes/api.js). This is good enough for testing, so our only task at this point is to make sure it's accessible to WorkOS for testing.

To do that, we'll use ngrok in this tutorial. This avoids the need to deploy the app to the web for testing webhooks by allowing us to expose our own localhost as a public URL for as long as we keep the ngrok command running.

With your app still running, open a new terminal and run:

ngrok http 3000

This will give you a "Forwarding" URL that looks similar to this:

https://56cd-2603-3004-6e3-8100-7429-8493-1862-fbb8.ngrok-free.app

If you visit the URL, you'll get a notice from ngrok, and if you click "Visit Site" you'll see your local app.

Copy this URL and keep both your dev command and ngrok running — this is the URL we'll use to test the webhook.

Register a webhook with WorkOS

In the WorkOS dashboard, click the Webhooks section, then create a new webhook. In the "Endpoint URL" field, add your ngrok URL with the path /api/directory-sync appended. It should look similar to this:

https://56cd-2603-3004-6e3-8100-7429-8493-1862-fbb8.ngrok-free.app/api/directory-sync

Check the boxes for "Directory Events", "Directory Group Events", and "Directory User Events" (this will select all the boxes below each of these event categories), then save.

Store the webhook secret as an environment variable

After creating the webhook, copy the Signing Secret value from the top of the new webhook's dashboard and store it in .env as WORKOS_WEBHOOK_SECRET:

	WORKOS_API_KEY="sk_test_..."
	WORKOS_CLIENT_ID="client_..."
	WORKOS_REDIRECT_URI="http://localhost:3000/auth/callback"
	WORKOS_ORG_ID="org_..."
	WORKOS_DIRECTORY_ID="directory_..."
+	WORKOS_WEBHOOK_SECRET="abc..."

	SESSION_SECRET="secret sauce"

Validate that webhooks are sending

To make sure your webhook events are sending, make a change in your Okta directory: add a user to a group, create a new group, or otherwise modify your users and groups in a way that will trigger a sync with WorkOS.

Once you've made a change, the console of your dev process will show the incoming webhook. The data will look something like this:

{
  id: 'event_01HBF492M21X1QYD1TKCQK7GSE',
  data: {
    id: 'directory_group_01HBF491X5ECAHXQ0J13KXGEH6',
    name: 'Authors',
    idp_id: 'Authors',
    object: 'directory_group',
    created_at: '2023-09-29T00:09:07.748Z',
    updated_at: '2023-09-29T00:09:08.479Z',
    directory_id: 'directory_01HBF1A4XRKNJDVSX1NJW5TGJ5',
    raw_attributes: {},
    organization_id: 'org_01HBF10W8ZVRY2VG96HZ6KHX72',
    previous_attributes: {}
  },
  event: 'dsync.group.updated',
  created_at: '2023-09-29T00:09:08.482Z'
}

Update the app in response to webhook events

Now that the app is receiving webhook events, we need to write the code to update our app in response to those events.

Validate that incoming webhook requests are valid

To ensure that only requests sent from WorkOS's Directory Sync updates are able to modify our app, we need to start by validating every request that's made to our endpoint.

To do this, make the following changes to the /directory-sync route in src/routes/api.js:

	const Router = require('express-promise-router');
+	const { WorkOS } = require('@workos-inc/node');
	const db = require('../db');

	const router = new Router();
+	const workos = new WorkOS(process.env.WORKOS_API_KEY);

	router.post('/directory-sync', async (req, res) => {
-		console.log(req.body);
-
-		// TODO implement directory sync
+		const payload = req.body;
+		const sigHeader = req.headers['workos-signature'];
+
+		// validate the event and get the data
+		const webhook = workos.webhooks.constructEvent({
+			payload,
+			sigHeader,
+			secret: process.env.WORKOS_WEBHOOK_SECRET,
+		});
+
		res.send('ok');
	});

WorkOS requests include a workos-signature header, which is the result of hashing the request payload with the webhook secret. If the signature matches, it's a valid webhook and the event data is returned from the constructEvent method.

Handle changes in group membership

The first group of events our app needs to handle are group membership updates. These are how our app knows which permissions each user has. In this example, we store the group names as part of their user data, but this could also map to roles, permissions, or any other approach you prefer.

Set up a switch on the event value of the webhook payload, then handle the dsync.group.user_added and dsync.group.user_removed events by adding the following code to the /directory-sync route in src/routes/api.js:

	router.post('/directory-sync', async (req, res) => {
		const payload = req.body;
		const sigHeader = req.headers['workos-signature'];

		// validate the event and get the data
		const webhook = workos.webhooks.constructEvent({
			payload,
			sigHeader,
			secret: process.env.WORKOS_WEBHOOK_SECRET,
		});
+
+		switch (webhook.event) {
+			case 'dsync.group.user_added':
+			case 'dsync.group.user_removed':
+				const { user, group } = webhook.data;
+				const roles = await db.getUserRolesByEmail(user.username);
+
+				if (webhook.event === 'dsync.group.user_added') {
+					roles.add(group.name.toUpperCase());
+				} else {
+					roles.delete(group.name.toUpperCase());
+				}
+
+				await db.createOrUpdateUser({
+					email: user.username,
+					firstName: user.firstName,
+					lastName: user.lastName,
+					roles: [...roles.values()],
+					active: user.state === 'active',
+				});
+				break;
+
+			default:
+				console.log(`TODO: handle ${webhook.event} events`);
+		}

		res.send('ok');
	});

When a user is added to or removed from a group, this code loads the affected user's profile, adds or removes the group according to the event type, then updates the user in the app's database with a new set of roles.

Add new users

Next, your app needs to create new users when they're added to the app. Update the /directory-sync endpoint in src/routes/api.js with the following:

	router.post('/directory-sync', async (req, res) => {
		const payload = req.body;
		const sigHeader = req.headers['workos-signature'];

		// validate the event and get the data
		const webhook = workos.webhooks.constructEvent({
			payload,
			sigHeader,
			secret: process.env.WORKOS_WEBHOOK_SECRET,
		});

		switch (webhook.event) {
			case 'dsync.group.user_added':
			case 'dsync.group.user_removed':
				const { user, group } = webhook.data;
				const roles = await db.getUserRolesByEmail(user.username);

				if (webhook.event === 'dsync.group.user_added') {
					roles.add(group.name.toUpperCase());
				} else {
					roles.delete(group.name.toUpperCase());
				}

				await db.createOrUpdateUser({
					email: user.username,
					firstName: user.firstName,
					lastName: user.lastName,
					roles: [...roles.values()],
					active: user.state === 'active',
				});
				break;
+
+			case 'dsync.user.created':
+				await db.createOrUpdateUser({
+					email: webhook.data.username,
+					firstName: webhook.data.firstName,
+					lastName: webhook.data.lastName,
+					roles: [],
+					active: webhook.data.state === 'active',
+				});
+				break;

			default:
				console.log(`TODO: handle ${webhook.event} events`);
		}

		res.send('ok');
	});

To do this, we directly insert the necessary details into our user database.

Deactivate deleted users

Finally, if a user's account is disabled — whether that means it's set to inactive or suspended, or deleted altogether — we need to make sure that user is immediately logged out. The SSO auth flow will already make sure they're unable to log back in again.

To do this, make one last set of changes in the /directory-sync endpoint in src/routes/api.js:

	router.post('/directory-sync', async (req, res) => {
		const payload = req.body;
		const sigHeader = req.headers['workos-signature'];

		// validate the event and get the data
		const webhook = workos.webhooks.constructEvent({
			payload,
			sigHeader,
			secret: process.env.WORKOS_WEBHOOK_SECRET,
		});

		switch (webhook.event) {
			case 'dsync.group.user_added':
			case 'dsync.group.user_removed':
				const { user, group } = webhook.data;
				const roles = await db.getUserRolesByEmail(user.username);

				if (webhook.event === 'dsync.group.user_added') {
					roles.add(group.name.toUpperCase());
				} else {
					roles.delete(group.name.toUpperCase());
				}

				await db.createOrUpdateUser({
					email: user.username,
					firstName: user.firstName,
					lastName: user.lastName,
					roles: [...roles.values()],
					active: user.state === 'active',
				});
				break;

			case 'dsync.user.created':
				await db.createOrUpdateUser({
					email: webhook.data.username,
					firstName: webhook.data.firstName,
					lastName: webhook.data.lastName,
					roles: [],
					active: webhook.data.state === 'active',
				});
				break;
+
+			case 'dsync.user.updated':
+				if (
+					webhook.data.state === 'inactive' ||
+					webhook.data.state === 'suspended'
+				) {
+					await db.deactivateUserByEmail(webhook.data.username);
+				}
+				break;
+
+			case 'dsync.user.deleted':
+				await db.deactivateUserByEmail(webhook.data.username);
+				break;

			default:
				console.log(`TODO: handle ${webhook.event} events`);
		}

		res.send('ok');
	});

This code listens for both updated and deleted users, and deactivates the user's account if any of the conditions are met. Once deactivated, any active sessions are destroyed for that user, meaning they'll be logged out on their next request.

Test the sync

To see this in action, try any of the following actions:

Create a group called "Admins" and add a user to it. Note that users in the Admins group will have a "delete" option for all posts, not just those they created.
Change a user's groups and watch the permissions update on each page refresh.
Remove a user from the app while logged in as that user. Note that a removed user is immediately moved back to the home page (logged out) on their next request.

The tutorial app is stripped down for the sake of clarity. In a production app, you can do much more with Directory Sync to keep your app synced with the customer's team as it grows and changes.

Bonus tip: use Directory Sync to expand your adoption

Many SaaS apps today charge based on user seats, so another interesting way to use SCIM and Directory Sync is to build an "invite your teammates" flow into the app.

This is a win-win, because for your customer, their team has a fast path to find and invite the people they need to collaborate with, and for your company, there's an organic growth loop where each employee of your customer that starts using your app is another opportunity to add more users as they bring their team along.

To see the simplest implementation, add the following code to src/routes/dashboard.js at the top of the file and in the /team route:

	const Router = require('express-promise-router');
+	const { WorkOS } = require('@workos-inc/node');
	const db = require('../db');

	const router = new Router();
+	const workos = new WorkOS(process.env.WORKOS_API_KEY);

	/* unchanged code omitted for brevity */

	router.get('/team', async (req, res) => {
-		const teammates = [];
+		const users = await workos.directorySync.listUsers({
+			directory: process.env.WORKOS_DIRECTORY_ID,
+		});
+
+		const teammates = users.list.data
+			.filter((user) => {
+				return !user.emails.some((e) => e.value === req.session.user.email);
+			})
+			.map(({ id, emails, firstName, lastName, groups }) => {
+				console.log(groups);
+				return {
+					firstName,
+					lastName,
+					email: emails.at(0).value,
+					groups: groups.map((g) => g.name).join(', '),
+					inviteLink: `/api/invite/${id}`,
+				};
+			});

		res.render('dashboard/team', { teammates, user: req.session.user });
	});

This code pulls a list of all the users that could create an account in your app and shows them to the logged-in user with an invite link.

Save, then visit http://localhost:3000/dashboard/team to see the teammates listed.

To take this further, you could get only the employees that are on the current user's team or who are in their department, depending on the metadata supplied by the customer.

SCIM means your biggest customers feel comfortable signing bigger contracts

From admin overhead to security risks, third-party SaaS products cause a lot of stress for large companies' IT teams. By adding support for SCIM, your SaaS product leverages the customer's already-vetted directory and user management software, so the customer doesn't have to manually provision user accounts or worry that a terminated employee will still have access to things for a period after being removed from the central system.

As the size of the companies you're pursuing for deals increases, these types of concerns become more and more of a blocker to finalizing a signed contract. Adding SCIM support alongside your SSO gives even the largest potential customer confidence that your app can handle their scale and pass their security and compliance audits.

Resources and further reading

Add notifications to a Node app (app inbox, SMS, and email)

Tue, 19 Sep 2023 00:00:00 GMT

Watch on YouTube

Configure Courier for your app

First, log in or create an account at courier.com

On the dashboard, at the bottom of the left-hand sidebar, there's a toggle that lets you choose either test or production data. For this tutorial, we'll work with test data.

Our first step is to add the notification channels we want our app to support. Courier has built-in integrations with dozens of tools, plus ways to add your own with webhooks and SDKs.

This tutorial will set up in-app notifications, email, and SMS.

Add support for in-app notifications

A notification inbox in an app is a great way to unobtrusively let users know what's happening. Courier provides everything you need to add one, including React components (which we'll look at later in this tutorial).

To set it up, go to your channels and choose Courier under the Inbox section. On the next screen, scroll all the way down and click "install provider".

Head back to your channels and it'll show up under your configured providers as well as in your routing settings.

Add support for SMS and email

To send notifications to users who aren't actively viewing your app dashboard, you'll need additional channels.

First, add email by choosing Gmail from the Email providers and authorizing with any of your Gmail accounts. (If you don't have one, you can skip this or use a different provider.)

Next, add SMS using Twilio. You'll need your account SID and auth token, which are available at on your Twilio console home page, and any active Twilio phone number on your account.

Once this is done, both email and SMS will be available under configured providers.

Add channels to your default routing

Courier allows you to choose whether a given channel should always be notified, or if only the best available option out of a set of channels should be used.

This is a great way to allow you to reach your users without being too overbearing about it. No one likes to get the same notification duplicated across every account and device they own.

How this works in practice is that Courier will always deliver a notification to the in-app inbox, and will choose one of either SMS or email, using SMS first if both are available. These are only defaults, though — you can also allow your users to choose which platforms they want to get notified on, and if they want an alert on every device and account they have, you can make that happen!

Set up the app for local development

For this tutorial, the app we're working with is a React-powered app dashboard. The dashboard itself doesn't do anything — it's only there so we have a plausible app to work with.

Under the hood, this demo uses a couple things to make our lives easier with setup:

Notifications themselves are managed by Courier. We'll spend most of this tutorial on how to configure, display, and send notifications in a React app.
User auth is handled by Clerk. Having a logged-in user is important because it means we have a unique ID for each person viewing the dashboard, which allows Courier to route notifications to people properly.
This app uses a webhook to sync users between Clerk and Courier. The webhook will be built on Netlify Functions both so we don't have to think about how to deploy a server and also because we get a way to expose our local dev environment as a live URL for testing.
To send notifications, we'll use a Netlify Function to keep secret credentials secure.

To grab the tutorial repo at the starting point, clone the start branch:

# clone the start branch of the repo using the GitHub CLI (https://cli.github.com)
gh repo clone learnwithjason/courier-notifications -- -b start

# move into the folder
cd courier-notifications/

# install dependencies
npm i

Get Clerk credentials

To use the auth, you'll need a free Clerk account. This will take less than five minutes to set up.

Log in or create an account at clerk.com
Create a new app and choose any auth provider that you want to use to log in with — this tutorial uses GitHub
Go to API Keys in the left-hand nav
Copy your publishable key and your secret key

In your code, rename .env.EXAMPLE to .env and add your publishable key as VITE_CLERK_PUBLISHABLE_KEY and your secret key as CLERK_SECRET_KEY.

Start the dev server

With the credentials saved, you're able to start the app and log in.

# start the dev server
npm run dev

Open http://localhost:5173 in your browser and you'll be redirected to the login page

Use your GitHub account to log in and you'll see the dashboard.

Now that we're able to log in, we want to add a notifications inbox up in the right side of the header next to our user info.

Display notifications in the app

To add a notifications inbox to our app, we'll take advantage of Courier's ready-made React components.

Get your Courier credentials

First, we need our client key from Courier. To find this, head to the settings for your Courier inbox channel, scroll down to the "Test Configuration" section, and copy the "Client Key (Public)". Add it to your .env file as the value of VITE_COURIER_CLIENT_KEY.

Install Courier npm packages

The demo repo already has dependencies installed. For posterity, you can install everything you need to run Courier notifications in a React app by installing these npm packages:

npm i @trycourier/courier @trycourier/react-inbox @trycourier/react-provider

Create a Notifications component

In your code, create a new file at src/components/notifications.tsx and add the following code inside:

import { CourierProvider } from '@trycourier/react-provider';
import { Inbox } from '@trycourier/react-inbox';
import { useUser } from '@clerk/clerk-react';

export function Notifications() {
  const { user } = useUser();

  console.log(user?.id);

  return (
    <CourierProvider
      userId={user?.id}
      clientKey={import.meta.env.VITE_COURIER_CLIENT_KEY}
    >
      <div className="notifications-wrapper">
        <Inbox />
      </div>
    </CourierProvider>
  );
}

This component places the Inbox component inside a CourierProvider that receives the client key from Courier and the user ID from Clerk.

We also log the user ID so it's easy to find for testing.

To put this into our app, add the Notifications component to src/components/dashboard.tsx:

	import {
		RedirectToSignIn,
		SignedIn,
		SignedOut,
		UserButton,
	} from '@clerk/clerk-react';
+	import { Notifications } from './notifications';

	import styles from './dashboard.module.css';

	export const Dashboard = () => {
		return (
			<>
				<header className={styles.header}>
					<a className={styles.homeLink} href="/" rel="home">
						Toofshine
					</a>

					<nav className={styles.nav}>
						<SignedIn>
+							<Notifications />
							<UserButton
	// ...unchanged below this point...

Save and look at your app dashboard. You'll see the inbox at the top right and the user ID logged in the console.

Send a notification manually

To send a notification manually, copy the user ID and head to the Courier users page. (As always, make sure you're still in test mode.) Create a new user — all you'll need is the ID, which should match the ID from Clerk.

Save this, then click the "Home" option, then the "send a message" button on the home page.

You'll see a form. On the right-hand side of the screen, choose "Push" from the channel dropdown, then Courier from the provider dropdown.

In the "To" field, add the same user ID, then click "send now".

Back in the app, you'll see a new notification in the inbox.

Automatically sync users to Courier

To make it possible to send notifications without manually creating each user, you'll need a way to sync users from your user management tool into Courier. In Clerk, this can be done using a webhook that will be called whenever a user is created, updated, or deleted.

Get a Courier API key

To set this up, go to your Courier API keys page and copy the key with "Published" scope. Set it as the value of COURIER_API_KEY in .env.

Configure Clerk with a webhook and get the signing secret

Next, head to your Clerk dashboard, choose your app, and click the "Webhooks" option in the left-hand nav.

Inside, set any URL to start (we'll change this in a later step) and check the "user" box so the webhook is called for all user events.

On the next screen, look for the "Signing Secret" toward the bottom of the right-hand column. Copy that value and set it as CLERK_WEBHOOK_SECRET.

Set up a live URL for testing local dev with Netlify Dev

To make debugging easier, we'll be using Netlify Dev, which will set up a live tunnel and allow us to create a public URL that can access our local code while we're developing. This is extremely helpful for debugging things like webhooks because it removes the need to deploy every change to see if it works.

In your terminal, install the Netlify CLI if you don't have it, make sure you're logged in, and then create a new project.

# if you don't already have it, install the Netlify CLI globally
npm i -g netlify-cli

# make sure you're logged in
ntl login

# create a new project
ntl init

You can keep the defaults during initialization.

Once the project is hooked up to Netlify, run Netlify Dev with the --live flag to get a public URL:

ntl dev --live dev

This will create a live URL that looks like https://dev--<your_site_name>.netlify.live.

Back in the Clerk webhook configuration, edit the webhook URL to be your live dev URL from Netlify and add the path /api/user-sync to the end. We'll build this endpoint in the next step.

Build the user sync webhook

Open netlify/functions/user-sync.ts in your code and replace the contents with the following:

import type { Handler } from '@netlify/functions';
import type { WebhookEvent } from '@clerk/clerk-sdk-node';
import { Webhook } from 'svix';
import { CourierClient } from '@trycourier/courier';

const courier = CourierClient({
  authorizationToken: process.env.COURIER_API_KEY,
});

type ProfileData = {
  email?: string;
  phone_number?: string;
};

function validateWebhook(req) {
  try {
    const wh = new Webhook(process.env.CLERK_WEBHOOK_SECRET!);
    wh.verify(req.body ?? '', req.headers as any);
    return true;
  } catch (err) {
    console.log(err);
    return false;
  }
}

function getProfileDetails({ email_addresses, phone_numbers }): ProfileData {
  const profile: ProfileData = {};

  if (email_addresses.length > 0 && email_addresses[0].email_address) {
    profile.email = email_addresses[0].email_address;
  }

  if (phone_numbers.length > 0 && phone_numbers[0].phone_number) {
    profile.phone_number = phone_numbers[0].phone_number;
  }

  return profile;
}

export const handler: Handler = async (req) => {
  if (!validateWebhook(req)) {
    return {
      statusCode: 400,
      body: 'Bad Request',
    };
  }

  const event = JSON.parse(req.body ?? '') as WebhookEvent;

  switch (event.type) {
    case 'user.created':
    case 'user.updated':
      await courier.replaceProfile({
        recipientId: event.data.id,
        profile: getProfileDetails(event.data),
      });
      break;

    case 'user.deleted':
      if (!event.data.id) {
        break;
      }

      await courier.deleteProfile({
        recipientId: event.data.id,
      });
      break;

    default:
      return {
        statusCode: 400,
        body: 'Bad Request',
      };
  }

  return {
    statusCode: 200,
    body: 'OK',
  };
};

At the top, we create a new instance of the Courier Node SDK with our API key. This provides methods for us to handle all the Courier-related tasks we need to accomplish.

Next we define a couple helper functions:

validateWebhook uses the signing secret from Clerk to ensure that the webhook is legitimate to prevent someone from impersonating Clerk and messing with user data
getProfileDetails pulls the primary email address and phone number out of the Clerk user data and drops everything else since Courier won't need it

In the actual exported handler, the incoming request is validated, and then we set up a switch on the event type.

When a user is created or updated, we want to sync their user ID, phone number, and email to Courier to make sure they're getting notifications properly
When a user is deleted, we want to remove them from Courier entirely

Finally, we return a 200 status to let Clerk know the webhook was processed successfully.

To test this, Clerk has a "Testing" tab in the webhook config. Choose "user.created" from the dropdown and then click "Send Example". You'll see a successful call in the dev console, and the Courier users page will update with the example user.

Now, whenever a user is created, updated, or deleted, the necessary information required by Courier to send notifications will be automatically synced to Courier.

Create a custom audience in Courier

Sending notifications to individual users can be done using their IDs, but what if you want to send a notification to every active user — for instance, when a new feature launches?

In the Courier users page, click the "+ Audience" button, name the audience "Active Members", and save. The ID will automatically generate as active-members.

Inside, click "+Add Condition" and choose email, then scroll down to "exists" and choose "True".

Next, click the "+" at the end of email and choose phone_number, then set "exists" to "True".

Use the "Calculate Audience" button to see the users that will be targeted. This should include your user ID from Clerk.

Now you're able to send a notification to everyone in this audience, which greatly simplifies the process of sending notifications.

Send a notification using the Courier Node SDK

Now that users are properly synced and our notification channels are configured, let's take a look at how the Courier Node SDK allows you to send notifications.

Open netlify/functions/send-notification.ts and replace the contents with the following:

import type { Handler } from '@netlify/functions';
import { CourierClient } from '@trycourier/courier';

const courier = CourierClient({
  authorizationToken: process.env.COURIER_API_KEY,
});

export const handler: Handler = async (req) => {
  const { title, body } = JSON.parse(req.body ?? '');

  if (req.httpMethod !== 'POST' || !title || !body) {
    return {
      statusCode: 400,
      body: 'Bad Request',
    };
  }

  const res = await courier.send({
    message: {
      to: {
        audience_id: 'active-members',
      },
      content: {
        title,
        body,
      },
    },
  });

  return {
    statusCode: 200,
    body: JSON.stringify(res),
  };
};

This is a Netlify Function that sets up the Courier Node SDK, pulls the title and body out of the POST request, and then uses the send method of the SDK to deliver the notification to everyone in the active-members audience.

It then returns a 200 to let you know the function ran successfully.

With your Netlify Dev process still running, use a tool like Postman or a cURL command to send a POST request to the /api/send-notification endpoint of your live URL.

The notification will show up in your in-app inbox.

If you look at the Courier logs, you'll see that the notification was sent to the inbox and also to the best of either SMS or email. If you registered using GitHub, it's likely that you only have an email on your account, so you'll get an email from Clerk with your notification.

If you update Clerk to use phone numbers for registration and create a new account, sending notifications will now hit the inbox and SMS for that user.

Notifications don't have to be hard or annoying

Courier has made it possible to get notifications running within an app in a few minutes and without much additional code at all, all while making it less complicated to send notifications in a way that won't overwhelm or annoy your app's users.

Thanks again to Courier for making this tutorial possible.

Go build something cool (with notifications)!

Resources and further reading

Add SSO to a SaaS app using WorkOS & Okta in under 10 minutes

Thu, 07 Sep 2023 00:00:00 GMT

Watch on YouTube

The scenario: enterprise-sized deals require enterprise-level auth support

Your team has been growing your SaaS offering quickly, and your first big customer is ready to sign a six-figure annual contract, but they've got Okta as an identity provider (IdP) and they'll only finalize the deal if your app will let their team use single sign-on (SSO).

For any developers who have implemented a SAML 2.0 workflow or other federated identity management manually in the past, the thought of adding SSO to a SaaS app might cause you to break into a cold sweat. I can remember spending days struggling to get it set up for a client project years ago — it was not a great experience.

But things have improved significantly in this space, and tools like WorkOS make adding SSO to a Node app a task that you can fit in before lunch. In this tutorial, we'll add SSO powered by WorkOS, using Okta as an IdP.

Create a WorkOS organization and connect it to Okta as an identity provider

First, register or log in at https://workos.com.

Add a redirect URI

As part of the SSO flow, the user will be redirected to a URI that can exchange a temporary code for an auth token. For security, these redirect URIs need to be explicitly added to WorkOS.

In your WorkOS dashboard, click configuration in the left-hand nav, then add http://localhost:3000/auth/sso/redirect as a redirect URI for local dev. You'll need to come back to this page later to add more URIs for your other environments, such as staging and production.

Create an organization in WorkOS

Once logged in, click "Organizations" in the left-hand sidebar, then click the button to create a new one. Give your organization a name and add the domain(s) associated with users in your IdP.

On the organization home page, look for "Single Sign-On" under the Features section. Click "Configure".

Choose Okta from the dropdown and give the connection a name (I chose "Okta").

On the next screen you'll see several URLs and configuration options. Keep this page open, then continue with the next section.

Create an application in Okta

Sign up or register for Okta. We'll be using Okta as our Identity Provider (IdP), so our users' data will be stored here, and we'll use an app integration to allow WorkOS to manage the SSO flow.

In your account, navigate to Applications using the top-left menu.

Create a new app integration and choose SAML 2.0 as the sign-in method. Give the app a name, then click next.

Under SAML settings, add the ACS URL from WorkOS as the Single sign-on URL in Okta.

Next, add the SP Entity ID from WorkOS as the Audience URI in Okta.

Under attribute statements, map the four bits of user data we want for this app:

id => user.id
email => user.email
firstName => user.firstName
lastName => user.lastName

Click next.

On the next screen, choose "I'm an Okta customer adding an internal app" and click Finish.

The next screen will be your Okta app settings, and under "SAML 2.0" you'll see the "Metadata URL" — copy this, then head to your WorkOS organization. Under the URLs you copied before is a section called "Identity Provider Configuration". Click "Edit Configuration".

Paste in the metadata URL and save. Your connection will now show active!

Assign users to your app integration

In order to allow your users to log in via WorkOS SSO, you'll need to assign the app to them in Okta. This is one of the major reasons companies choose SSO: it allows them to manage user access to all apps in a single location.

To do this, go to the Directory in the menu, then choose Groups. Select the Applications tab, then click "Assign applications" and choose the WorkOS SSO app you just created.

Set up your dev environment

Our starting point is a Node.js app built with Fastify, which is very similar to Express. There's a home page that's public, and a dashboard that requires the user to be logged in.

Clone the start branch of the repo to get started:

# clone the start branch
gh repo clone learnwithjason/workos-six-figure-saas-contracts -- -b start

# move into the directory
cd workos-six-figure-saas-contracts/

# install dependencies
npm i

Add environment variables

Make a copy of .env.EXAMPLE and rename it to .env. Inside, add the following values.

SESSION_SECRET — a random value (I used a UUID generator to get one) used by the Fastify session plugin
REDIRECT_URI — this is pre-filled for development. If you change this, make sure it matches the value set as a redirect URI in your WorkOS organization
WORKOS_API_KEY — click "API Keys" in the left-hand nav of your WorkOS account and copy the secret key
WORKOS_CLIENT_ID — the client ID is displayed at the top of the API Keys screen in your WorkOS account
WORKOS_ORG_ID — head to your organization in the WorkOS dashboard and the organization ID will be displayed at the top of the org home page

Add a route to handle SSO login

In src/app.ts, make the following changes:

	import "dotenv/config";

	import type { FastifyPluginAsync } from "fastify";
	import type { StaticRequest } from "./types.js";
	import * as path from "path";
	import { fileURLToPath } from "url";
	import ejs from "ejs";
	import fastifyView from "@fastify/view";
	import fastifyStatic from "@fastify/static";
+	import { WorkOS } from "@workos-inc/node";

	const __filename = fileURLToPath(import.meta.url);
	const __dirname = path.dirname(__filename);
	const publicDirRoot = path.join(__dirname, "..", "public");

+	const workos = new WorkOS(process.env.WORKOS_API_KEY);

	const app: FastifyPluginAsync = async (fastify, _opts) => {
		fastify.register(fastifyStatic, { root: publicDirRoot });
		fastify.register(fastifyView, { engine: { ejs } });
+
+		// WorkOS SSO flow
+		fastify.get("/auth/sso", async (_req, res) => {
+			const authorizationUrl = workos.sso.getAuthorizationURL({
+				clientID: process.env.WORKOS_CLIENT_ID!,
+				organization: process.env.WORKOS_ORG_ID,
+				redirectURI: process.env.REDIRECT_URI!,
+			});
+
+			res.redirect(authorizationUrl);
+		});

		// App views
		fastify.get("/", async (req, reply) => {
			return reply.view("src/templates/index.ejs", {
				user: undefined,
			});
		});

		fastify.get("/dashboard", async (req, reply) => {
			// TODO only show the dashboard if the user is logged in

			return reply.view("src/templates/dashboard.ejs", {
				user: { first_name: "Guest" },
			});
		});

		// all other requests fall through to static files
		fastify.get<StaticRequest>("/:filename", async function (req, reply) {
			const { filename } = req.params;

			return reply.sendFile(filename);
		});
	};

	export default app;

Using the @workos-inc/node package and our API key, this code registers a new route at /auth/sso, which calls the getAuthorizationUrl method. Using our client ID, organization ID, and redirect URI, this method generates a URL that will ask the user to log into their IdP (in this example, Okta).

Once the user has authenticated with their IdP, they'll be redirected to the redirectURI specified here.

Handle calls to the redirect URI

When a user has successfully authenticated with their IdP, they'll be sent to the redirect URI with a code that can be exchanged for an auth token.

To handle this exchange, add the redirect URI route in src/app.ts:

	import "dotenv/config";

	import type { FastifyPluginAsync } from "fastify";
-	import type { StaticRequest } from "./types.js";
+	import type { StaticRequest, RedirectRequest } from "./types.js";
	import * as path from "path";
	import { fileURLToPath } from "url";
	import ejs from "ejs";
	import fastifyView from "@fastify/view";
	import fastifyStatic from "@fastify/static";
	import { WorkOS } from "@workos-inc/node";

	const __filename = fileURLToPath(import.meta.url);
	const __dirname = path.dirname(__filename);
	const publicDirRoot = path.join(__dirname, "..", "public");

	const workos = new WorkOS(process.env.WORKOS_API_KEY);

	const app: FastifyPluginAsync = async (fastify, _opts) => {
		fastify.register(fastifyStatic, { root: publicDirRoot });
		fastify.register(fastifyView, { engine: { ejs } });

		// WorkOS SSO flow
		fastify.get("/auth/sso", async (_req, res) => {
			const authorizationUrl = workos.sso.getAuthorizationURL({
				clientID: process.env.WORKOS_CLIENT_ID!,
				organization: process.env.WORKOS_ORG_ID,
				redirectURI: process.env.REDIRECT_URI!,
			});

			res.redirect(authorizationUrl);
		});
+
+		fastify.get<RedirectRequest>("/auth/sso/redirect", async (req, res) => {
+			const { code } = req.query;
+			const { profile } = await workos.sso.getProfileAndToken({
+				code,
+				clientID: process.env.WORKOS_CLIENT_ID!,
+			});
+
+			// TODO store the user profile in the session
+			console.log({ profile });
+
+			return res.redirect("/dashboard");
+		});

		// App views
		fastify.get("/", async (req, reply) => {
			return reply.view("src/templates/index.ejs", {
				user: undefined,
			});
		});

		fastify.get("/dashboard", async (req, reply) => {
			// TODO only show the dashboard if the user is logged in

			return reply.view("src/templates/dashboard.ejs", {
				user: { first_name: "Guest" },
			});
		});

		// all other requests fall through to static files
		fastify.get<StaticRequest>("/:filename", async function (req, reply) {
			const { filename } = req.params;

			return reply.sendFile(filename);
		});
	};

	export default app;

Save, then navigate to http://localhost:3000/auth/sso. You'll be redirected to the Okta login. After signing in, you'll end up on the app dashboard and your profile data will be logged in the console.

Log in with your Okta credentials and you'll land on http://localhost:3000/dashboard. This means the login flow is working, but our app isn't checking for a logged in user yet or using that to restrict access to private pages.

Store user profile data in a session

To let the app display user data and prevent the user from needing to log in on every page navigation, let's add the user's profile data to a session cookie. In this app, we'll use the @fastify/session package, but the approach of storing the user data in a cookie will work in just about every framework.

To add support for sessions, bring in the required Fastify plugins in src/app.ts and register them.

	import "dotenv/config";

	import type { FastifyPluginAsync } from "fastify";
	import type { StaticRequest, RedirectRequest } from "./types.js";
	import * as path from "path";
	import { fileURLToPath } from "url";
	import ejs from "ejs";
	import fastifyView from "@fastify/view";
	import fastifyStatic from "@fastify/static";
+	import fastifyCookie from "@fastify/cookie";
+	import fastifySession from "@fastify/session";
	import { WorkOS } from "@workos-inc/node";

	const __filename = fileURLToPath(import.meta.url);
	const __dirname = path.dirname(__filename);
	const publicDirRoot = path.join(__dirname, "..", "public");

	const workos = new WorkOS(process.env.WORKOS_API_KEY);

	const app: FastifyPluginAsync = async (fastify, _opts) => {
		fastify.register(fastifyStatic, { root: publicDirRoot });
		fastify.register(fastifyView, { engine: { ejs } });
+		fastify.register(fastifyCookie);
+		fastify.register(fastifySession, {
+			secret: process.env.SESSION_SECRET!,
+			cookie:
+				process.env.NODE_ENV === "production"
+					? {
+							path: "/",
+							maxAge: 86400,
+							httpOnly: true,
+							secure: true,
+							sameSite: true,
+					  }
+					: { secure: false },
+		});

		// WorkOS SSO flow
		fastify.get("/auth/sso", async (_req, res) => {
			const authorizationUrl = workos.sso.getAuthorizationURL({
				clientID: process.env.WORKOS_CLIENT_ID!,
				organization: process.env.WORKOS_ORG_ID,
				redirectURI: process.env.REDIRECT_URI!,
			});

			res.redirect(authorizationUrl);
		});

		fastify.get<RedirectRequest>("/auth/sso/redirect", async (req, res) => {
			const { code } = req.query;
			const { profile } = await workos.sso.getProfileAndToken({
				code,
				clientID: process.env.WORKOS_CLIENT_ID!,
			});

-			// TODO store the user profile in the session
-			console.log({ profile });
+			req.session.user = {
+				id: profile.id,
+				first_name: profile.first_name,
+				last_name: profile.last_name,
+				email: profile.email,
+			};

			return res.redirect("/dashboard");
		});

		// App views
		fastify.get("/", async (req, reply) => {
			return reply.view("src/templates/index.ejs", {
+				user: req.session.user,
			});
		});

		fastify.get("/dashboard", async (req, reply) => {
			// TODO only show the dashboard if the user is logged in

			return reply.view("src/templates/dashboard.ejs", {
				user: { first_name: "Guest" },
			});
		});

		// all other requests fall through to static files
		fastify.get<StaticRequest>("/:filename", async function (req, reply) {
			const { filename } = req.params;

			return reply.sendFile(filename);
		});
	};

	export default app;

The required plugins allow our app to store data in a session, and we use that to store the user profile in the redirect handler after a user successfully logs in with their IdP.

On the home route (/), we pass the user object from the session into the template, which will allow the app UI to be updated with logged in user data.

Only show the dashboard to logged in users

To use the SSO auth data to protect the authenticated area, make the following changes to the /dashboard route in src/app.ts:

	fastify.get("/dashboard", async (req, reply) => {
-		// TODO only show the dashboard if the user is logged in
+		if (!req.session.user?.id) {
+			return reply.redirect("/auth/sso");
+		}

		return reply.view("src/templates/dashboard.ejs", {
-			user: { first_name: "Guest" },
+			user: req.session.user,
		});
	});

Handle logout

Finally, to handle logout, all we need to do is destroy the session. Add a new route definition for /auth/sso/logout in src/app.ts to handle logging out:

fastify.get('/auth/logout', async (req, reply) => {
  req.session.destroy();
  reply.redirect('/');
});

This will remove the user's data from the session and send them back to the public home page. They'll need to log in again to return to the dashboard.

Go land those big contracts!

Believe it or not, that's it. Single sign on for enterprise customers used to be a nightmare, but the developer experience these days is actually pretty pleasant thanks to WorkOS.

So don't let adding SAML 2.0 or other SSO solutions to your app be a blocker for your product to land larger customers. Happy building!

Resources and further reading

Handle webhooks, user auth, database, & file storage with Convex

Tue, 22 Aug 2023 00:00:00 GMT

Watch on YouTube

I had a wild idea recently: what if I wanted to send a text message to my app and have it show up on the web? Could I send images, too? How hard would that be to build?

My gut instinct was that it's possible, but it would be really complicated. I'd need to:

Have a way to turn incoming SMS messages into code
Set up a webhook to handle those incoming SMS messages
Validate them to make sure they're real
Set up user auth on the app
Set up a database to store the messages
Set up file storage for incoming images

That feels like a lot, right?

But modern dev tooling is, like, good good. Twilio makes handling incoming SMS extremely approachable. Clerk makes user auth so fast to set up that it feels like cheating. And Convex handles literally everything else on the requirement list, from storing data to exposing webhooks to storing images.

So let's build it. Today we'll build a React + TypeScript app that you can send text and images to via SMS, and we'll power it all with Convex, Twilio, and Clerk.

Set up your dev environment

For this app, we'll focus on the database specifically. Clone the start branch of the demo app's repo to get an app that's working except for data.

# clone the start branch of the repo using the GitHub CLI
gh repo clone learnwithjason/convex-twilio-text-log -- -b start

# move into the repo
cd convex-twilio-text-log/

# install dependencies
npm i

Set up Clerk

This app uses Clerk to allow users to create accounts and log in, so before we can start developing we'll need a Clerk account and a publishable key.

Sign up or sign in at https://clerk.com
Click the "add application" button
Give your new application a name (e.g. "Snack Tracker")
Under "how will your users sign in?", choose only "phone number" — our whole app is built around texting, so this is important!
Click create application
On the next screen, copy your publishable key

Back in your code, rename .env.local.EXAMPLE to .env.local and paste the publishable key as the value of VITE_CLERK_PUBLISHABLE_KEY.

Start the dev server

Once the Clerk publishable key is saved, start the dev server.

npm run dev

Open http://localhost:5173 in your browser to see the app.

Get a Twilio phone number to accept incoming SMS messages

For this app to work, we need a way to relay incoming SMS messages to our code. Twilio makes this possible. If you've never used Twilio before, you can get a free trial and a trial number for testing. If you already use Twilio, setting up a new phone number costs $1.15/month (in the US at the time of writing).

To set up your Twilio number:

Go to the Twilio console
Find "phone numbers" either in the left-hand sidebar or by searching
Create a new number
- If you're in a free trial, click "get a trial number"
- If you're not, click "buy a number"
On the next screen, choose your country, make sure "SMS" and "MMS" are checked under Capabilities, and choose any number.
Buy the number and copy it

Open .env.local in your code and add the phone number, including country code, as the value of VITE_TWILIO_PHONE_NUMBER. It should be formatted like this:

VITE_TWILIO_PHONE_NUMBER="+1 555-555-5555"

Save and you'll see your Twilio number displayed in the app.

At this point, you're ready to add a database!

Set up Convex

Now that the app is up and running, let's add a database to store messages sent by users and the webhook that will handle new incoming messages.

Install Convex in the app by adding the convex package:

npm i convex

Next, start the Convex dev process in a second terminal window (the app's dev process should still be running) to initialize Convex for your app:

npx convex dev

Choose "create a new project"
If necessary, you'll be prompted to create an account or log in
Choose which team the project belongs to
Give the project a name (e.g. snack-tracker)

This creates a new folder called convex in the app, which is where all of the schema, data access, and HTTP actions for the app will be created and managed.

Create a database table to store messages

Before we do anything else, let's define a schema for our messages. Create a new file at convex/schema.ts and add the following code:

import { defineSchema, defineTable } from 'convex/server';
import { v } from 'convex/values';

export const MessageFields = {
  text: v.string(),
  sender: v.string(),
  image: v.union(
    v.object({
      id: v.string(),
      url: v.union(v.string(), v.null()),
    }),
    v.null()
  ),
};

export default defineSchema({
  messages: defineTable(MessageFields).index('by_sender', ['sender']),
});

Exporting MessageFields separately means we can import that to use as a TypeScript type anywhere we need it in our app.

Using defineSchema, we pass in an object that describes all the tables in our app. We pass MessageFields to the defineTable function to use that schema for our messages, and to speed up searching by sender (which will be how we query for messages), we add an index to the messages table on the sender field.

After saving, Convex will automatically update and create the messages table, which you can view in the Convex dashboard.

Connect Convex to Clerk auth

This app requires a user to be logged in to view posts, and they're only able to see their own posts. How this translates to code is that we need to get the currently logged in user from Clerk and use that as part of our query to Convex.

Fortunately, Clerk and Convex have a first-class integration, so we're able to do this with a few clicks and a few lines of code.

Configure Clerk to integrate with Convex

Head to the Clerk dashboard and choose your app.

Click "JWT Templates" from the left-hand nav
Click "New template"
Choose Convex
Copy the "Issuer" URL that appears on the next screen
Click "apply changes"

Add auth config to Convex

With the issuer URL copied, create a new file at convex/auth.config.js and add the following:

export default {
  providers: [
    {
      domain: 'YOUR_CLERK_ISSUER_URL',
      applicationID: 'convex',
    },
  ],
};

Save and Convex will auto-detect the new config and update.

This tells Convex to use Clerk's configuration for auth, and will give us access to the currently logged in user within our Convex calls.

Add a Convex provider to the app

To use Convex in the app UI, we need to add a provider. Since we're using Clerk for auth, we'll use a special provider from Convex called ConvexProviderWithClerk.

The provider accepts a client, which we need to configure with our Convex URL. To get this, go to the Convex dashboard, choose your project, and navigate to settings. Click the toggle to show your development credentials and copy the Deployment URL.

Store the deployment URL as VITE_CONVEX_URL in .env.local.

Next, open src/main.tsx and make the following changes:

	import React from 'react';
	import ReactDOM from 'react-dom/client';
+	import { ConvexProviderWithClerk } from 'convex/react-clerk';
+	import { ConvexReactClient } from 'convex/react';
-	import { ClerkProvider } from '@clerk/clerk-react';
+	import { ClerkProvider, useAuth } from '@clerk/clerk-react';
	import { App } from './components/app';

	import './styles/global.css';

+	const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string);

	ReactDOM.createRoot(document.getElementById('root') as HTMLElement).render(
		<React.StrictMode>
			<ClerkProvider publishableKey={import.meta.env.VITE_CLERK_PUBLISHABLE_KEY}>
+				<ConvexProviderWithClerk client={convex} useAuth={useAuth}>
					<App />
+				</ConvexProviderWithClerk>
			</ClerkProvider>
		</React.StrictMode>,
	);

This change allows Clerk to provide auth data to the Convex provider via the useAuth hook — and that's all the setup that's required to integrate Convex and Clerk.

Add a query to load messages by user

Now that we have access to Convex and the current user in our app, we need a way to query for the messages they're authorized to see.

To do that, we'll define our first Convex query. Create a new file at convex/messages.ts and add the following code:

import { query } from './_generated/server';

export const get = query({
  handler: async (ctx) => {
    const identity = await ctx.auth.getUserIdentity();
    if (!identity || !identity.phoneNumberVerified) {
      return [];
    }

    const messages = await ctx.db
      .query('messages')
      .withIndex('by_sender', (q) => q.eq('sender', identity.phoneNumber!))
      .collect();

    return messages;
  },
});

The query helper from Convex is loaded from the _generated directory, which Convex uses to provide us with autocompletion and other quality-of-life enhancements as we write our apps.

Inside we define a handler that receives a context object (ctx) from Convex. This object contains multiple helpful utilities, including auth, which has a method for loading the current user's details, and db, which has methods for querying our database tables.

After loading the current user (and returning an empty result set if no user is found), this code queries the messages table using that by_sender index we defined earlier, and uses the q helper to filter down results to only those where the sender of the stored message matches the phone number of the current user.

Run the query in the React UI

To use this query in the app, make the following changes in src/components/messages.tsx:

+	import { useQuery } from 'convex/react';
+	import { api } from '../../convex/_generated/api';
+
	import styles from './messages.module.css';

	const Empty = () => {
		const phone = import.meta.env.VITE_TWILIO_PHONE_NUMBER;

		return (
			<div className={styles.empty}>
				<h1>Ready to start logging?</h1>
				<p>
					Text a photo and any details or description you want to include about it
					to:
					<a href={`sms:${phone}`}>{phone}</a>
					Send your first text to see it logged here!
				</p>
			</div>
		);
	};

	export const Messages = () => {
-		// TODO: load all messages from the currently logged in user
-		const messages = [];
+		// load all messages from the currently logged in user
+		const messages = useQuery(api.messages.get) || [];

		return (
			<section className={styles.wrapper}>
				{messages.length > 0 ? (
					<ul className={styles.messages}>
-						{/* TODO: display messages */}
+						{messages.map(({ _id, _creationTime, text, image }) => {
+							return (
+								<li key={_id} className={styles.message}>
+									{image && image.url ? (
+										<img className={styles.image} src={image.url} alt={text} />
+									) : null}
+									<p className={styles.text}>{text}</p>
+									<p className={styles.meta}>
+										Posted {new Date(_creationTime).toLocaleString()}
+									</p>
+								</li>
+							);
+						})}
					</ul>
				) : (
					<Empty />
				)}
			</section>
		);
	};

Convex generates an api object that will autocomplete with all available tables and their related queries, mutations, and actions. The useQuery hook runs the get query we just defined and returns an array of messages.

To display the messages, we loop over them and destructure out the fields we need. We also use two system-generated fields: _id, which is an auto-generated unique identifier for each entry, and _creationTime, which is the timestamp at which the entry was created.

Right now our table is empty. You can create an entry or two manually through the Convex dashboard to test this if you'd like. This has the bonus effect of showing the automatic real-time nature of working with Convex: as soon as you save an entry, it will show up in the app UI in real time.

Create new messages from incoming SMS messages

To allow our users to create messages, we need a way to process SMS messages sent to our Twilio number. To do this, we'll use Convex HTTP actions, which are similar to queries and mutations, but are exposed as HTTP endpoints so they can interact with third-party systems.

This makes Convex HTTP actions an ideal solution for building webhooks.

Create a Convex HTTP action

A Convex HTTP action receives a standard Request object in addition to the Convex context, and it needs to return a standard Response object.

Export a new method called save from convex/messages.ts with the following code:

import type { WithoutSystemFields } from 'convex/server';
import type { Doc } from './_generated/dataModel';
import { httpAction, query } from './_generated/server';

type Message = WithoutSystemFields<Doc<'messages'>>;

export const get = query({
  /* unchanged */
});

/*
 * An HTTP action exposes an endpoint, which we’ll add as the webhook URL for
 * incoming Twilio SMS messages. This will be called every time someone texts
 * the phone number provided by our app.
 */
export const save = httpAction(async (ctx, req) => {
  const body = await req.text();

  // Twilio params: https://www.twilio.com/docs/messaging/guides/webhook-request
  const message = new URLSearchParams(body);

  // TODO: validate the webhook for security

  const text = message.get('Body') ?? '';
  const sender = message.get('From');
  const imageUrl = message.get('MediaUrl0');

  if (!sender) {
    return new Response(null, {
      status: 400,
    });
  }

  const msg: Message = {
    text,
    sender,
    image: null,
  };

  if (imageUrl) {
    // TODO: store any images sent by the user
  }

  // TODO: save the message
  console.log(JSON.stringify(msg, null, 2));

  return new Response(null, {
    status: 200,
  });
});

At the top, we import the httpAction helper, as well as two types: WithoutSystemFields and Doc.

The two types allow us to create a Message type that matches our message table schema but leaves out fields that are generated by Convex, such as _id. This lets us add type checking without having to worry about missing system fields before saving.

In the save function, we get the body of the request as text because Twilio sends the body as query parameters (e.g. key1=val1&key2=val2).

Our function needs the sender's phone number, the text from the message, and the URL of the first image, if any were sent.

Organize those details in to an object that matches the Message type and it's ready for saving! We'll add the mutation to actually save entries in a moment, but for now this is good enough to test that it's working once we integrate with Twilio.

Expose the HTTP action in a URL endpoint

To make our HTTP action callable, we need to give it a public URL. To do this, create a new file at convex/http.ts and add the following code:

import { httpRouter } from 'convex/server';
import { save } from './messages';

const http = httpRouter();

http.route({
  path: '/messages',
  method: 'POST',
  handler: save,
});

export default http;

We define a new route at /messages, then add our save function as the handler for requests sent to that endpoint via POST requests. Once we save, our HTTP action is now usable by a third-party service.

HTTP actions are exposed at https://<your deployment name>.convex.site. Grab the value you stored in VITE_CONVEX_URL and replace .cloud with .site, then append /messages for the full URL to your HTTP action (e.g. https://energized-rooster-480.convex.site/messages).

Register the Convex HTTP action as a webhook for incoming Twilio messages

In the Twilio console, navigate to your active numbers and choose the one you purchased earlier.

Under the "Configure" tab, scroll down to "Messaging Configuration"
In the section for "A message comes in", make sure "Webhook" is selected
Add your HTTP action endpoint as the webhook URL
Make sure "HTTP POST" is selected
Click "Save configuration"

Once this is saved, send a text message to your Twilio number, then look at the logs in Convex. You'll see your number and the contents of your text message logged there, which means the webhook is configured properly.

Validate Twilio webhook requests

To make sure some mischief-maker out there doesn't spam or otherwise abuse the app, let's make sure every request received by our HTTP action is a valid Twilio request before taking any action.

Validation will be handled in a Convex internal function. To do that, we'll use Twilio's Node SDK, which we can install by running the following in our terminal:

npm i twilio

Create a new file at convex/validate.ts with the following code inside:

'use node';

import { v } from 'convex/values';
import twilio from 'twilio';
import { internalAction } from './_generated/server';

export const twilioWebhook = internalAction({
  args: {
    signature: v.string(),
    url: v.string(),
    params: v.any(), // Twilio sends a lot of fields that might vary
  },
  handler: async (_, args) => {
    return twilio.validateRequest(
      process.env.TWILIO_AUTH_TOKEN!,
      args.signature,
      args.url,
      args.params
    );
  },
});

To run this code, we'll need our Twilio auth token:

Navigate to https://console.twilio.com
Copy the "Auth token" field
Open the Convex dashboard
Choose your project
Click "Settings" in the left-hand nav
Add TWILIO_AUTH_TOKEN as the key of a new environment variable
Add your copied auth token as the value

To actually call the validation action, make the following changes to convex/messages.ts:

	import type { WithoutSystemFields } from 'convex/server';
	import type { Doc } from './_generated/dataModel';
	import { httpAction, query } from './_generated/server';
+	import { internal } from './_generated/api';

	type Message = WithoutSystemFields<Doc<'messages'>>;

	export const get = query({
		handler: async (ctx) => {
			const identity = await ctx.auth.getUserIdentity();
			if (!identity || !identity.phoneNumberVerified) {
				return [];
			}

			const messages = await ctx.db
				.query('messages')
				.withIndex('by_sender', (q) => q.eq('sender', identity.phoneNumber!))
				.collect();

			return messages;
		},
	});

	/*
	 * An HTTP action exposes an endpoint, which we’ll add as the webhook URL for
	 * incoming Twilio SMS messages. This will be called every time someone texts
	 * the phone number provided by our app.
	 */
	export const save = httpAction(async (ctx, req) => {
		const body = await req.text();

		// Twilio params: https://www.twilio.com/docs/messaging/guides/webhook-request
		const message = new URLSearchParams(body);

-		// TODO: validate the webhook for security
+		const isValidWebhook = await ctx.runAction(internal.validate.twilioWebhook, {
+			url: req.url,
+			signature: req.headers.get('x-twilio-signature') ?? '',
+			params: Object.fromEntries(message.entries()),
+		});
+
+		if (!isValidWebhook) {
+			return new Response(null, {
+				status: 422,
+			});
+		}

		const text = message.get('Body') ?? '';
		const sender = message.get('From');
		const imageUrl = message.get('MediaUrl0');

		/* unchanged below this point */

This code grabs the URL, request signature, and parameters sent by Twilio, then passes them to our internal validation action. If the request is valid, the code continues to run as usual, but if the signatures don't match our HTTP action will now return a 422 HTTP response code ("unprocessable content").

Send another text to your Twilio number to validate that it still works as expected with valid requests. If you want to test invalid requests, you can use something like Postman to send a POST request to the HTTP action — you'll now receive a 422 response.

Save new messages in Convex

Now that we're receiving messages from Twilio and we're confident that the requests are valid, let's save them in the database.

To do that, we'll use another internal function, but this time it'll be a mutation. Make the following changes to convex/messages.ts:

	import type { WithoutSystemFields } from 'convex/server';
	import type { Doc } from './_generated/dataModel';
-	import { httpAction, query } from './_generated/server';
+	import { httpAction, query, internalMutation } from './_generated/server';
	import { internal } from './_generated/api';
+	import { MessageFields } from './schema';

	type Message = WithoutSystemFields<Doc<'messages'>>;

	export const get = query({ /* unchanged */ });

	/*
	* An HTTP action exposes an endpoint, which we’ll add as the webhook URL for
	* incoming Twilio SMS messages. This will be called every time someone texts
	* the phone number provided by our app.
	*/
	export const save = httpAction(async (ctx, req) => {
		const body = await req.text();

		// Twilio params: https://www.twilio.com/docs/messaging/guides/webhook-request
		const message = new URLSearchParams(body);

		// TODO: validate the webhook for security
		const isValidWebhook = await ctx.runAction(internal.validate.twilioWebhook, {
			url: req.url,
			signature: req.headers.get('x-twilio-signature') ?? '',
			params: Object.fromEntries(message.entries()),
		});

		if (!isValidWebhook) {
			return new Response(null, {
				status: 422,
			});
		}

		const text = message.get('Body') ?? '';
		const sender = message.get('From');
		const imageUrl = message.get('MediaUrl0');

		if (!sender) {
			return new Response(null, {
				status: 400,
			});
		}

		const msg: Message = {
			text,
			sender,
			image: null,
		};

		if (imageUrl) {
			// TODO: store any images sent by the user
		}

-		// TODO: save the message
-		console.log(JSON.stringify(msg, null, 2));
+		ctx.runMutation(internal.messages.saveMessage, msg);

		return new Response(null, {
			status: 200,
		});
	});
+
+	export const saveMessage = internalMutation({
+		args: MessageFields,
+		handler: async (ctx, args) => {
+			await ctx.db.insert('messages', args);
+		},
+	});

Save, then send a text message to your Twilio number. After a few seconds it will appear in your app UI.

This is already pretty dang cool, but we want to make it better: let's add support for sending and saving images as well.

Save incoming images in messages to Convex

Twilio automatically sends along images in webhook requests. In our app, we want to save those images to the same place as the rest of our data, so we'll be using Convex file storage to download and deliver them.

And before you wave this off as too complicated: this will take about 20 lines of code to implement!

Make the following changes in convex/messages.ts:

	import type { WithoutSystemFields } from 'convex/server';
	import type { Doc } from './_generated/dataModel';
	import {
+		type ActionCtx,
		httpAction,
		query,
		internalMutation,
	} from './_generated/server';
	import { internal } from './_generated/api';
	import { MessageFields } from './schema';

	type Message = WithoutSystemFields<Doc<'messages'>>;

	export const get = query({ /* unchanged */ });

	/*
	* An HTTP action exposes an endpoint, which we’ll add as the webhook URL for
	* incoming Twilio SMS messages. This will be called every time someone texts
	* the phone number provided by our app.
	*/
	export const save = httpAction(async (ctx, req) => {
		const body = await req.text();

		// Twilio params: https://www.twilio.com/docs/messaging/guides/webhook-request
		const message = new URLSearchParams(body);

		// TODO: validate the webhook for security
		const isValidWebhook = await ctx.runAction(internal.validate.twilioWebhook, {
			url: req.url,
			signature: req.headers.get('x-twilio-signature') ?? '',
			params: Object.fromEntries(message.entries()),
		});

		if (!isValidWebhook) {
			return new Response(null, {
				status: 422,
			});
		}

		const text = message.get('Body') ?? '';
		const sender = message.get('From');
		const imageUrl = message.get('MediaUrl0');

		if (!sender) {
			return new Response(null, {
				status: 400,
			});
		}

		const msg: Message = {
			text,
			sender,
			image: null,
		};

		if (imageUrl) {
+			try {
+				msg.image = await storeImage(ctx, imageUrl);
+			} catch (err) {
+				console.error(`failed to store image (url: ${imageUrl})`);
+			}
		}

		ctx.runMutation(internal.messages.saveMessage, msg);

		return new Response(null, {
			status: 200,
		});
	});

	export const saveMessage = internalMutation({
		args: MessageFields,
		handler: async (ctx, args) => {
			await ctx.db.insert('messages', args);
		},
	});

+	export const storeImage = async (ctx: ActionCtx, imageUrl: string) => {
+		const res = await fetch(imageUrl);
+
+		if (!res.ok) {
+			console.error(res);
+			return null;
+		}
+
+		const blob = await res.blob();
+		const id = await ctx.storage.store(blob);
+		const url = await ctx.storage.getUrl(id);
+
+		return { id, url };
+	};

The storeImage function loads the provided image from its Twilio URL, then sends it to Convex's storage as a blob. The returned ID of the stored file is then used to generate its public URL, and both the ID and URL are returned.

The save action runs storeImage if there's an image in the message and wraps it in a try ... catch block just in case the file is incompatible or otherwise unusable.

And... that's it. Save this and send an image to your app's phone number to see it show up in the dashboard.

Stop worrying that databases are too hard and go build cool stuff

I've let a lot of good ideas die because I didn't want to deal with setting up or managing a database. These days, though, tools like Convex make it so dang easy that I can't make excuses — it's fun to put together a database like this. It's fun to hook up different third-party APIs.

I really love the web today, because these tools are here to let me just go build my ideas instead of having to spend all my time creating the boilerplate that makes my ideas function.

I'm excited for what this unlocks for the web. I hope you're excited, too. I hope you show me what you build.

Resources and further reading

Use AI to moderate abusive and vulgar comments (full tutorial)

Mon, 24 Jul 2023 00:00:00 GMT

Watch on YouTube

Internal tools don't get prioritized. That's why so many developers have horror stories of teams running superadmin commands against the production database that they copy-pasted from a doc somewhere.

In this tutorial, you'll learn how to build an internal dashboard to moderate a custom comments database. You won't have to write much code, but everything you create will be stored as human-readable code that can be checked into source control and edited manually without breaking the low-code workflow.

As an added bonus, we'll also look at how we can integrate AI (using OpenAI/Chat-GPT integrated into Airplane) to automatically flag the most abusive and vulgar comments, decreasing the psychic damage taken by human moderators.

Create a new Airplane project

To start, register for or sign into your Airplane account.

Next, install the Airplane CLI so you can develop locally:

brew install airplanedev/tap/airplane

Clone the starter repo, then move into it and start Airplane

# clone the start branch of the repo
gh repo clone learnwithjason/airplane-content-moderation -- -b start

# move into the cloned app directory
cd airplane-content-moderation/

Inside, this app contains the boilerplate for a project — such as a tsconfig.json and a package.json — as well as a file called airplane.yaml. The only thing inside this file is a note about the Node version (set to Node 18). This is a pretty cool feature of Airplane: there's very little boilerplate required.

There's also an example app folder, which you can ignore for now. We'll come back to that after we've got the Airplane tasks and views built.

Inside the project folder, use the Airplane CLI to start the project.

# start Airplane in dev mode (will prompt for login on first run)
airplane dev

The first time you run this command, you'll be asked to log in.

Next, the CLI will start the dev server and give you the option to press enter to open the Airplane Studio, which is a UI for local development that updates your local files in real time. Press enter to open the studio.

Set up the database and initial queries

Our first step will be to make sure we have data to work with and that we're able to read it out of our database in Airplane.

Create a task to initialize the comments table

To begin, let's create a table to store comments and add a few entries so we can verify things are working as expected.

To get the work done that our tutorial requires, we'll be using Airplane tasks. These can take a few forms, but we'll start with a SQL task.

Create a new task in the Studio called "comments_db_reset" and choose the SQL option.

Once you create the task, two new files will be created in your working directory:

comments_db_reset.sql
comments_db_reset.task.yaml

It's possible to edit these files directly — we'll do that for a future task — but in many cases it's much more convenient to use the Studio UI.

In the Studio, update the details in the "Define" section:

Name: Reset comments database
Description: Deletes the current comments table, including all comment entries, then creates a new comments table with a few seed entries.

The local files will update as you type.

Next, scroll down to the "Build" section and choose "[Demo DB]" from the "Database Resource" dropdown.

Under "Query", add the following:

DROP TABLE comments;

CREATE TABLE IF NOT EXISTS
  comments (
    id      SERIAL PRIMARY KEY,
    comment TEXT NOT NULL,
    flagged BOOLEAN
  );

INSERT INTO
  comments (comment, flagged)
VALUES
  ('this looks so delicious omg', false),
  ('I think you suck', true),
  ('I do not want to eat this', false),
  ('eat poop', true)
;

This is a set of SQL instructions that removes the comments table, creates a new one with the necessary fields, and then inserts example entries into it that we can use to build out the rest of our dashboard.

Our changes save as we type, so once everything is entered, we can click the "Execute Task" button in the top panel.

That's the whole process for setting up Airplane and modifying a database. I love this flow because it feels magical, but nothing that happens is "magic" or hidden from me — everything I entered in the UI is stored in my code base now.

The UI is a convenience, not a requirement — I can choose not to use it if I prefer.

Create a task to list all comments

Next, let's add another task to list all of our comments. Create a new task in the Studio, choose SQL, and name it "comments_list_all". Add the following details in the "Define" section:

Name: List all comments
Description: Lists all comments in the database, regardless of flagged status

Choose the demo DB from the database dropdown and add the following query:

SELECT
  id,
  comment,
  flagged
FROM
  comments
ORDER BY
  flagged
;

Click the "Execute Task" button and you'll see the seed comments listed on the screen.

Create a task to list flagged comments

Next, repeat this process to create a task called "comments_list_flagged" to list only flagged comments using the following details:

Name: List flagged comments
Description: List all comments that have been flagged as abusive or otherwise problematic.

Choose the demo DB and add the following query:

SELECT
  id,
  comment,
  flagged
FROM
  comments
WHERE
  flagged = true
;

Create a task to list approved comments

We'll also need to be able to select all the unflagged (approved) comments. For these, we can copy-paste the comments_list_flagged files and make small adjustments right in the code.

Rename both files to comments_list_approved, keeping their respective extensions. In comments_list_approved.task.yaml, make the following edits:

	slug: comments_list_approved
	name: List approved comments
	description: List all comments that have been approved.
	sql:
	  resource: demo_db
	  entrypoint: comments_list_approved.sql

Next, replace the contents of comments_list_approved.sql with the following:

SELECT
  id,
  comment,
  flagged
FROM
  comments
WHERE
  flagged = false
;

Build a comment moderation dashboard view

So far, we've only looked at tasks in Airplane. Next, let's dig into how Airplane views work.

Create an Airplane view

In the Studio, create a new view by clicking the + at the top of the explorer. Give it the following details:

Name: Comment Moderation Dashboard
Description: Allows admins to see all approved comments, and optionally see flagged comments. They're also able to change the approved/flagged state of a comment and delete comments permanently.

This will create a new file in your local directory called CommentModerationDashboard.airplane.tsx — you can rename this if you want, but we'll leave it as-is for this project.

Update the view with a table to display approved comments

Inside, you'll see an example component. Replace the file contents with the following:

import { Heading, Stack, Table } from '@airplane/views';
import airplane from 'airplane';

const CommentModerationDashboard = () => {
  return (
    <Stack>
      <Heading>Comment Moderation Dashboard</Heading>

      <Table
        title="Approved Comments"
        task="comments_list_approved"
        defaultPageSize={20}
        hiddenColumns={['flagged']}
      />
    </Stack>
  );
};

export default airplane.view(
  {
    slug: 'comment_moderation_dashboard',
    name: 'Comment Moderation Dashboard',
    description:
      "Allows admins to see all approved comments, and optionally see flagged comments. They're also able to change the approved/flagged state of a comment and delete comments permanently.",
  },
  CommentModerationDashboard
);

Airplane provides a suite of React UI components to make building dashboards as straightforward as snapping together components.

A Stack is a container for content, and inside we've added a Heading to let the viewer know what this dashboard is for, followed by a Table to display approved comments.

The Table accepts a few props. The title is displayed at the top, the defaultPageSize tells the table how many rows to show before paginating, and hiddenColumns lets us leave out columns from the table that we don't need.

The really interesting prop here is the task prop. Many Airplane components can be task-backed, which means we can perform tasks (such as loading data or performing a query) using their slugs. This is a great productivity boost, because we don't have to mess with calling APIs to load data, then looping through them to build table views — we just say, "Give me a table with the result of the task called comments_list_approved" and Airplane does the rest. Neat!

Once we've saved this component, we'll see a new icon pop up in the explorer for a view called "Comment Moderation Dashboard". Click on it and you'll see the layout you just built, including the approved comment entries displayed in the table.

Add another table to display flagged comments

We also need a way to see flagged comments, so add another Table that uses the comments_list_flagged task:

	import { Heading, Stack, Table } from '@airplane/views';
	import airplane from 'airplane';

	const CommentModerationDashboard = () => {
		return (
			<Stack>
				<Heading>Comment Moderation Dashboard</Heading>

				<Table
					title="Approved Comments"
					task="comments_list_approved"
					defaultPageSize={20}
					hiddenColumns={['flagged']}
				/>
+
+				<Table
+					title="Flagged Comments"
+					task="comments_list_flagged"
+					defaultPageSize={20}
+					hiddenColumns={['flagged']}
+				/>
			</Stack>
		);
	};

	export default airplane.view(
		{
			slug: 'comment_moderation_dashboard',
			name: 'Comment Moderation Dashboard',
			description:
				"Allows admins to see all approved comments, and optionally see flagged comments. They're also able to change the approved/flagged state of a comment and delete comments permanently.",
		},
		CommentModerationDashboard,
	);

Save and the flagged comments appear, but this isn't ideal — we don't want to subject our admins to potentially abusive comments every time they load the dashboard. Instead, let's hide the flagged comments by default and only show them if the admin clicks a checkbox to confirm that they want to review flagged comments.

Inside the view, let's add a Checkbox from the Airplane component library, as well as the useComponentState hook that will let us check whether it's checked or not:

	import {
+		type CheckboxState,
		Heading,
		Stack,
		Table,
+		Checkbox,
+		useComponentState,
	} from '@airplane/views';
	import airplane from 'airplane';

	const CommentModerationDashboard = () => {
+		const { id, checked } = useComponentState<CheckboxState>();

		return (
			<Stack>
				<Heading>Comment Moderation Dashboard</Heading>

				<Table
					title="Approved Comments"
					task="comments_list_approved"
					defaultPageSize={20}
					hiddenColumns={['flagged']}
				/>

+				<Checkbox
+					id={id}
+					label="Show flagged comments (view at your own risk!)"
+				/>
+
+				{checked ? (
					<Table
						title="Flagged Comments"
						task="comments_list_flagged"
						defaultPageSize={20}
						hiddenColumns={['flagged']}
					/>
+				) : null}
			</Stack>
		);
	};

	export default airplane.view(
		{
			slug: 'comment_moderation_dashboard',
			name: 'Comment Moderation Dashboard',
			description:
				"Allows admins to see all approved comments, and optionally see flagged comments. They're also able to change the approved/flagged state of a comment and delete comments permanently.",
		},
		CommentModerationDashboard,
	);

Now the dashboard hides the comments that could ruin someone's day by default, and they only have to be viewed if it becomes necessary to review them.

Add a task to flag comments as abusive

If a comment is approved by mistake, we need the ability to manually flag it. To do that, create a new task called "comment_flag" with the following details:

Name: Flag comment as abusive

Under parameters, click the "Add parameter" button and add the following values:

Name: id
Description: The ID of the comment to flag.
Type: Integer
Required: true

All the other values can remain unchanged.

Click Update to save.

Next, choose the demo DB as the database resource and add the following query:

UPDATE
  comments
SET
  flagged = true
WHERE
  id = :id
;

Finally, set the query argument to be "id" and the value to be {{params.id}}, which connects the parameter of the task to the query argument of this query.

To test, grab an ID from one of the approved comments on the dashboard, enter it into the ID field of the "Flag comment as abusive" task, and click the "Execute task" button.

The previously approved comment will now be flagged, which you can verify by visiting the dashboard again.

Add a task to unflag comments

Now, it's not against our site rules to dislike something, so that comment should be approved. Let's add another task to allow us to do that.

Create a task called "comment_approve" with the following details:

Name: Approve comment
Description: Approve a comment for public display.

Add a parameter called id as an integer with the description, "The ID of the comment to approve".

Next, set the demo DB as the database resource and add this query:

UPDATE
  comments
SET
  flagged = false
WHERE
  id = :id
;

For query arguments, add a new one called id with the value of {{params.id}}.

Use the same comment ID that you just flagged and execute the task. It will now be back on the approved list in the dashboard.

Create a SQL task to delete comments

After an admin has reviewed a flagged comment to confirm that, yep, this comment is terrible, we want to let them delete it permanently — no reason for anyone else to have to see that trash.

To do that, create a new task called "comment_delete" with the following details:

Name: Delete comment permanently
Description: Removes a comment permanently. There is no undo for this action!

Add a parameter called id as an integer with the description, "The ID of the comment to delete".

Next, set the demo DB as the database resource and add this query:

DELETE FROM
  comments
WHERE
  id = :id
;

For query arguments, add a new one called id with the value of {{params.id}}.

Test this by adding a comment ID in the field and clicking "execute task".

Call tasks from table rows in Airplane

At this point, what we've built is already pretty useful. We can:

View comments (both flagged and approved)
Toggle the flagged status of comments
Delete comments

This could be considered good enough. But we can make this much more user friendly with only a few more lines of code thanks to a built-in Airplane feature called row actions.

Add a row action to flag or approve a comment

In Airplane Table components, we can add a rowActions prop that adds a button in each row and performs the specified action for the current row when clicked.

There are a few ways to do this, up to and including fully custom solutions. For our needs, the task-backed row actions are perfect: they will automatically pass through the current comment's ID — we only need to provide the task to be performed and label for it!

In CommentModerationDashboard.airplane.ts, make the following changes:

	import {
		type CheckboxState,
		Heading,
		Stack,
		Table,
		Checkbox,
		useComponentState,
	} from '@airplane/views';
	import airplane from 'airplane';

	const CommentModerationDashboard = () => {
		const { id, checked } = useComponentState<CheckboxState>();

		return (
			<Stack>
				<Heading>Comment Moderation Dashboard</Heading>

				<Table
					title="Approved Comments"
					task="comments_list_approved"
					defaultPageSize={20}
					hiddenColumns={['flagged']}
+					rowActions={{
+						slug: 'comment_flag',
+						label: 'flag',
+					}}
				/>

				<Checkbox
					id={id}
					label="Show flagged comments (view at your own risk!)"
				/>

				{checked ? (
					<Table
						title="Flagged Comments"
						task="comments_list_flagged"
						defaultPageSize={20}
						hiddenColumns={['flagged']}
+						rowActions={[
+							{
+								slug: 'comment_approve',
+								label: 'approve',
+							},
+							{
+								slug: 'comment_delete',
+								label: 'delete',
+							},
+						]}
					/>
				) : null}
			</Stack>
		);
	};

	export default airplane.view(
		{
			slug: 'comment_moderation_dashboard',
			name: 'Comment Moderation Dashboard',
			description:
				"Allows admins to see all approved comments, and optionally see flagged comments. They're also able to change the approved/flagged state of a comment and delete comments permanently.",
		},
		CommentModerationDashboard,
	);

Save, refresh the studio, and your dashboard will now show the "flag" option on approved comments, and the "approve" and "delete" options on flagged comments.

Test out the buttons to see row actions in, uh, action!

Refresh other tables when row actions are performed

You may have noticed that right now, the table that contains the updated row updates, but other tables require a page refresh to show the changes.

Let's fix that.

Airplane provides a hook called useTaskQuery that lets us, among other things, force a refetch of the given task, causing all components using it to update.

Make the following changes in CommentModerationDashboard.airplane.ts to refetch all tables whenever a comment is modified:

	import {
		type CheckboxState,
		Heading,
		Stack,
		Table,
		Checkbox,
		useComponentState,
+		useTaskQuery,
	} from '@airplane/views';
	import airplane from 'airplane';

	const CommentModerationDashboard = () => {
		const { id, checked } = useComponentState<CheckboxState>();
+		const flagged = useTaskQuery('comments_list_flagged');
+		const approved = useTaskQuery('comments_list_approved');

		return (
			<Stack>
				<Heading>Comment Moderation Dashboard</Heading>

				<Table
					title="Approved Comments"
					task="comments_list_approved"
					defaultPageSize={20}
					hiddenColumns={['flagged']}
					rowActions={{
						slug: 'comment_flag',
						label: 'flag',
+						onSuccess: () => flagged.refetch(),
					}}
				/>

				<Checkbox
					id={id}
					label="Show flagged comments (view at your own risk!)"
				/>

				{checked ? (
					<Table
						title="Flagged Comments"
						task="comments_list_flagged"
						defaultPageSize={20}
						hiddenColumns={['flagged']}
						rowActions={[
							{
								slug: 'comment_approve',
								label: 'approve',
+								onSuccess: () => approved.refetch(),
							},
							{
								slug: 'comment_delete',
								label: 'delete',
							},
						]}
					/>
				) : null}
			</Stack>
		);
	};

	export default airplane.view(
		{
			slug: 'comment_moderation_dashboard',
			name: 'Comment Moderation Dashboard',
			description:
				"Allows admins to see all approved comments, and optionally see flagged comments. They're also able to change the approved/flagged state of a comment and delete comments permanently.",
		},
		CommentModerationDashboard,
	);

Save, then try again. Now both tables update whenever a comment is modified.

Create a TypeScript task to add a new comment

To this point, we've been using only SQL tasks, but Airplane also supports tasks written in JavaScript, Python, and more.

Create a new task and choose JavaScript as the type. Give it the name "comment_add" and leave TypeScript selected. Use the following details:

Name: Add a comment
Description: Save a new comment in the database. Includes a step to check for abusive comments and flag them to limit problematic content from becoming visible.
Parameters: comment, type "Long text"

Add the demo DB as a resource, then open comment_add.airplane.ts in your editor. The second argument to airplane.task is an async function, which contains everything you want the task to do when called.

Replace the boilerplate function with the following:

import airplane from 'airplane';

export default airplane.task(
  {
    slug: 'comment_add',
    name: 'Add a comment',
    description:
      'Save a new comment in the database. Includes a step to check for abusive comments and flag them to limit problematic content from becoming visible.',
    parameters: {
      comment: {
        name: 'comment',
        type: 'shorttext',
      },
    },
    resources: ['demo_db'],
  },
  async (params) => {
    const { comment } = params;

    // TODO add check for abusive content
    const output = { flagged: false };

    const res = await airplane.sql.query(
      'demo_db',
      'INSERT INTO comments (comment, flagged) VALUES (:comment, :flagged);',
      { args: { comment, flagged: output.flagged } }
    );

    console.log(res);

    let message = `Your comment was saved.`;

    return { message, flagged: output.flagged };
  }
);

Here's what this code does:

Get the comment out of the params so we can work with it
For now, temporarily hard-code the output, which we'll build for real in the next section
Use the built-in airplane.sql.query method to run an INSERT in our demo database to save the new comment along with its flagged status
Return a message and whether the comment was flagged

Save, then write a new comment in the task's input in the Studio and execute the task. Check the dashboard to see your new comment saved.

Use AI to automatically flag abusive comments

Moderation is not optional when we're opening up spaces for public comments. Flagging abusive content is a must if you want to have a space free of harassment and other unacceptable behavior.

The challenges with moderation are enormous and complicated. We won't cover all of them in this post, but we will look at two:

The people who moderate content prevent us from seeing the most terrible things that are said in the comments section — but for them to do their jobs, they have to read that awful content. A moderator is forced to confront the absolute worst the web has to offer every day, and that takes its toll.
Good moderation means not showing comments until they've been checked. This adds a significant delay between posting a comment and seeing the comment live, which can prevent conversations from happening because it takes too long to see responses.

To address these two challenges, one possible solution is using a large language model (LLM) as a kind of "first line defense" for comment moderation.

For this use case, we're attempting to catch abusive and hateful comments. We're also saving all comments for a final human review before deletion, which will allow us to adjust the instructions we're providing the LLM if it's incorrectly flagging comments.

Use Airplane's built-in AI functions to moderate user input

Airplane provides several built-in operations, including AI support for both OpenAI and Anthropic.

For our app, we'll use OpenAI, which currently provides $5 in credit to new accounts, which is more than enough to build and test this feature.

Next, head to the the Airplane Studio and create a config var (the icon that looks like (x) in the left-hand sidebar). Name the config OPENAI_API_KEY and paste in your OpenAI key.

Go back to your "Add a comment" task in the explorer and scroll down to the "build" section of the config. Under "Environment variables", click "add variable". Name it OPENAI_API_KEY, then choose "From config var" from the dropdown. In the new dropdown that appears, choose the OPENAI_API_KEY config.

This will update the task file with a reference to the config var, which makes it safe to commit the task file (but, again, do not commit airplane.dev.yaml).

With the API key available, modify comment_add.airplane.ts to add the AI moderation step:

	import airplane from 'airplane';

	export default airplane.task(
		{
			slug: 'comment_add',
			name: 'Add a comment',
			description:
				'Save a new comment in the database. Includes a step to check for abusive comments and flag them to limit problematic content from becoming visible.',
			parameters: {
				comment: {
					name: 'comment',
					type: 'shorttext',
				},
			},
			resources: ['demo_db'],
			envVars: {
				OPENAI_API_KEY: {
					config: 'OPENAI_API_KEY',
				},
			},
		},
		async (params) => {
			const { comment } = params;

-			// TODO add check for abusive content
-			const output = { flagged: false };
+			const getSentiment = airplane.ai.func(
+				'Identify abusive and vulgar comments. Negative opinions are allowed but personal attacks are not.',
+				[
+					{
+						input: 'This is the shit!',
+						output: { flagged: false, sentiment: 'positive' },
+					},
+					{
+						input: 'You are stupid!',
+						output: { flagged: true, sentiment: 'negative' },
+					},
+					{
+						input: 'Burgers are gross',
+						output: { flagged: false, sentiment: 'negative' },
+					},
+				],
+			);
+
+			const { output, confidence } = await getSentiment(comment);
+
+			if (typeof output === 'string') {
+				return { message: 'unparseable input' };
+			}

			const res = await airplane.sql.query(
				'demo_db',
				'INSERT INTO comments (comment, flagged) VALUES (:comment, :flagged);',
				{ args: { comment, flagged: output.flagged } },
			);

			console.log(res);

-			let message = `Your comment was saved.`;
+			let message = `Your comment was saved. The sentiment was read as ${output.sentiment}.`;
+			if (output.flagged === true && confidence >= 0.75) {
+				message = 'Wow, you kiss your mother with that mouth?';
+			}

			return { message, flagged: output.flagged };
		},
	);

Save, then add an abusive comment to make sure it'll get caught (I recommend trying, "you're a doofus").

Less than 25 lines of code, and we've got a pretty okay auto-moderation flow in place — that's pretty impressive!

And again: remember that this is not a replacement for human moderation. It's a tool that can help the moderators work more effectively.

Deploy your Airplane dashboard

So far we've only been working locally. To make this dashboard available to your team, you'll need to deploy it.

Add config vars for production

For your tasks to work properly, you'll need to add a production config var with your OpenAI API key. Name it OPENAI_API_KEY — save it and you're all set.

Deploy Airplane tasks and views to production

To deploy, open the terminal and run the following command:

airplane deploy

Once the deployment completes, head to https://app.airplane.dev and check the library to see your deployed tasks and views.

Try the demo app to see the comment moderation flow in action

To provide a way to test the moderation workflow, the demo app for this project executes Airplane tasks directly via API. You could also set up a webhook to run the moderation flow in the background after a comment is added to your production database.

We won't go through exactly how to build this demo app, but the overview is:

It's an Astro site
The site runs in hybrid mode so it can process form submissions
Approved comments are loaded by executing the comments_list_approved task and getting its outputs
New comments are created by executing the comment_add task and returning its outputs

To see the source code, check out the GitHub repo.

For a user on the site, submitting a comment will take a few seconds and then immediate feedback will be sent: either the comment is approved and visible immediately, or it's flagged and the comment form will chide the user for posting abusive or vulgar comments.

Congratulations! You've built a complete comment moderation dashboard, including a first line of defense against the most vulgar and abusive comments powered by OpenAI.

Links and additional resources

Source code: https://github.com/learnwithjason/airplane-content-moderation
Get started with Airplane
More information on the ethical concerns around AI
A reminder that biases in AI can quite literally kill people if we don’t actively check the results

Clean as you go

Mon, 10 Jul 2023 00:00:00 GMT

Watch on YouTube

I learned to cook in a restaurant. I was a prep cook.

Every morning, I clocked in to stacks of produce I needed to have ready before the lunch rush. Every morning, I grabbed a cutting board and a knife, opened the first box of romaine lettuce, and got to chopping. And every morning, I'd make an enormous mess.

As I chopped, I pushed the scraps aside into a pile to deal with them later. Cleaning, after all, happened after I finished prep.

Big messes require big efforts to clean

But the scraps piled up, and my workspace got tighter and tighter. Onion skins stuck to the tomatoes; remnants of diced parsley speckled the cucumbers. My scrap pile got so out of control that I'd occasionally end up dumping handfuls of veggie waste onto the floor by mistake.

Working around my mess made prep stressful, but that paled in comparison to the cleanup. My prep technique left the kitchen looking like the aftermath of a food fight. It took me so long to clean up that I had to rush to avoid delaying the lunch shift from getting started. (There should have been an hour or so of downtime.)

The more experienced staff was horrified by this.

And, fortunately for me, they decided to help me instead of just firing me immediately.

Clean as you go to avoid big messes

One of the cooks very patiently introduced me to the concept of cleaning as you go. They had a bench scraper and a bleach bucket standing by, and positioned the trash can right next to the table. As I chopped, they encouraged me to move the scraps directly to the trash instead of making a pile. When I finished an ingredient, I grabbed the bench scraper to clear away all the left behind bits, then gave a quick wipe with a rag to reset my station to clean before starting the next one.

At first, I resisted this. Cleaning at each step had to be slower, right? After all, context switching is a terrible idea, and this was clearly going to make me even slower for sure.

But my options were to do it their way or get fired, so I stuck to it. And a few things happened:

The quality of my work went up. No more bits of the previous veggie showing up in the next one. And because my workspace was less cramped, I could see what I was doing and make fine adjustments more easily.
The prep itself went faster. Starting with a clean station made it faster to do the prep because I wasn't working around previous mess.
Cleanup went faster. By cleaning as I went, cleaning was as simple as a few swipes with a bench scraper and a quick wipe-down with a rag. Nothing piled up, so my last veggie was as easy to clean up after as the first one.

I wasn't rushing to get everything done and cleaned anymore. Instead of the cooks worrying about me, they started giving me additional training and opportunities.

It felt counterintuitive, but cleaning as I went had made me both better and faster at my job.

Isn't stopping to clean slower?

My knee-jerk reaction to cleaning as I went was to condemn it as multitasking, which makes work slower. But... was it?

Cleaning as you go seems like multitasking, but it's actually an efficient way of sequentially tackling work.

Small messes clean up fast

By not cleaning as we go, multiple messes compound on each other. A small mess is fast to clean, but a dozen or more small messes become a much bigger mess that's harder to clean up.

In food prep, the scrap pile might spill off the table. Wet scraps might dry to the table, making them far more difficult to clean up. A large pile has a tendency to spread out and get additional dishes and utensils dirty.

By contrast, a small mess is almost always a couple quick wipes away from being clean.

"Clean as you go" as a guiding principle for everything

This lesson didn't just save my high school job — it made a huge impression on my outlook on life. I try to clean as I go wherever I can these days, whether I'm cooking dinner at home or writing code at work.

In a codebase, we always make small messes as we work. We're solving problems, and until we hit a solution that works our code is full of small choices based on educated guesses that are directionally correct, but that probably wander a bit since they lack the full context of the working solution.

If we ship the code as-is, everything will be fine. The code works, and that's ultimately the point.

But we left a small mess. And if we do that every time we build a new feature, the small messes start to build into a big mess — this is how we end up with tech debt that's so overwhelming that we start to argue that the only reasonable solution is to start over from scratch.

If, instead, we choose to clean as we go — to do an immediate small refactor as part of submitting the PR, for example — we'll clean up those small messes before they can pile up into big ones. And because the messes are being continually addressed, we have cleaner, easier to manage code, and that makes us faster.

Building new habits is hard, but worth it

Cleaning as you go is a tricky habit to build. It requires a concentrated effort to change the way you work, and if you're in a team it's even trickier because you're trying to shift the team culture.

However, if you put in the work to build the habit, it pays dividends in every area of your life where you choose to adopt it.

It’s not worth switching tech stacks

Tue, 04 Jul 2023 00:00:00 GMT

Watch on YouTube

The JavaScript landscape is noisy. And it can be pushy. We hear a lot of opinions about what we should and shouldn't be using to build websites.

How do you choose when just about every JavaScript think piece contradicts the other advice you've seen and also insists that you need to learn the hot new thing?

My hot take is that a lot of developers tend to switch to new stacks as a way of kicking the can down the road on building features that users actually want and care about — so the best tech stack is the one you can build something useful in today.

We're just building websites here.

The fastest website you can build is HTML and CSS on a global CDN. Anything else we add to that site is going to slow it down — so when we're going to add extra steps, we need to make sure the trade-offs are actually worth it.

Ultimately, the technology we choose to build websites doesn't matter. There are a huge number of thriving businesses that make eye-watering profits, and very few are using bleeding edge web frameworks to ship.

In fact, most of them ship jQuery.

jQuery still powers 77% of websites.

Inside the tech hype bubbles, jQuery is DEAD-dead. We talk about it as if it's been fully abandoned for years and no one ever uses it anymore.

In reality, 3 of every 4 websites on the internet still ship jQuery.

Could you build a great website with jQuery today? You can! And people do!

Social media hype bubbles are small.

In online tech circles like Twitter, YouTube, Hacker News, Reddit, and their ilk, we can end up with a skewed perception of what the world of web dev actually looks like. A relatively tiny number of tech influencers — like me! — spend a lot of time discussing the bleeding edge of tech. We make it sound like everyone is doing the new thing, and everyone has left behind the old things.

But that's not reality. That's just me and a bunch of nerds like me who are extremely into web tech talking about our hobbies. Sure, we're using this bleeding edge stuff in production, but we're the exception, not the rule.

The FoMO factory of social media remains a menace to actually shipping things.

If you build something cool, the tools are irrelevant.

If you build a web app that people enjoy using and are willing to pay for, it doesn't really matter what you used to build it. What matters is that you can get the idea out of your head and into a working app. It doesn't matter if you built it with Java, .Net, Node, Ruby on Rails, or Dreamweaver.

Don't let someone who speaks authoritatively on Twitter make you feel bad about what you build with. What matters is that you built something.

When does it make sense to switch?

When there's a valid reason to rebuild an entire app, it'll be fairly obvious and compelling. Rebuilding an intranet that was pinned to IE6 has a host of compelling security and feature reasons to make the switch.

Shaving 50 milliseconds off a page load by spending months switching from Node to Rust, though? It just doesn't make enough of an impact to be worth the cost.

The best tech stack is whatever you're shipping with.

Technology is a tool, and it only matters as a means of delivering ideas to people who will use, enjoy, and (hopefully) pay for the things we build with it.

Outside of that, additional attention spent on tools is a hobby unless your job is building tools. Hobbies are wonderful and they move the web forward and we need folks on the bleeding edge to experiment, but you bear no obligation to change the way you build for the web just because it seems like your current stack is no longer trendy.

We're in this field to solve real problems for real people. The rest is details.

How to make $200K+ as a dev

Mon, 26 Jun 2023 00:00:00 GMT

Watch on YouTube

As you advance in your career as a developer, the key to reaching the next level of influence, leadership, and salary shifts away from improving technical skills. What makes a senior engineer qualified for staff, principal, and beyond? Interpersonal, leadership, and problem-solving skills.

Senior-plus salary bands expect you to build teams, not code.

Hitting senior software developer means you should be capable of handling just about any technical task thrown your way. Advancing to a senior-plus level requires making an impact that's bigger than what you can code on your own.

As a senior-plus dev, your impact isn't that you can code faster. Your impact is that you can:

Break concepts down in a way that helps less experienced engineers move more quickly
Convey context clearly and cross-functionally
Communicate with stakeholders in language that they understand

A senior-plus dev has to be capable of more than coding up something impressive — you need to be able to understand what to build (and why), then communicate with teammates and leadership well enough that they agree and support your insight.

Learn empathy for other departments — and how to communicate with them.

Learn the motivations and pain points of sales, marketing, product, and other departments in your organization. Know their metrics, objectives, and challenges. By understanding where they're coming from, you can reframe proposals in the context of their goals.

Sales needs new features and value adds that they can bring up in sales meetings so they can hit their sales goals
Marketing is on the hook to launch things and drive conversation, interest, and new signups
Product is under pressure to increase activation, retention, and performance metrics

Instead of trying to convince these teams that they should care about the underlying tech involved, start thinking about how different engineering choices will impact all of these metrics, both now and in the future — it completely changes the conversations and allows you to wield much greater influence.

The ability to communicate empathetically and persuasively to teammates and leaders in every department is a strong indicator of a senior-plus developer.

Become the stabilizer and foundation for your team.

A vital part of a senior-plus developer's role is to provide clarity to their team regarding the tasks and goals at hand. When the team is confident in the direction and deliverables, they can focus on being productive and efficient.

Support your manager in understanding project goals and specifications.
Collaborate with other departments to ensure they know what they are asking for and that their requirements are complete.
Consider user experience and user flow early in the project to prevent costly surprises midway through.

At every step, you're not only thinking of the technical implementation, but also how the tech fits into the company's goal and long-term success.

What you really build as a senior-plus dev is a high-performing team.

A team that gels well together and has confidence in their work is a sign of strong leadership and effective communication. Senior-plus developers and leaders should focus on clarity and alignment across all teams, speaking the same language, and working towards the same outcomes.

Reaching the next salary band requires growing your influence and leadership skills. Excellence as a coder is a requirement, but the higher levels require stronger communication and team-mindedness. By demonstrating that you understand the needs of stakeholders, that you can communicate effectively, and that you're focused not just on your own output, but on your entire team's success, you signal to leadership at your company that you're ready for your next promotion and raise.

Resources and further reading

Salary info on levels.fyi
Career ladder info by Sarah Drasner

What you need to know about getting hired in devrel

Tue, 20 Jun 2023 00:00:00 GMT

I have yet to see a job description for devrel that matches what someone actually does in their day-to-day work.

Watch on YouTube

Part of the problem is that a lot of devrel teams are built by someone who's had success in their own devrel career and is building a team around their own history and preferences. But most of the problem is that companies have an extremely poor understanding of what devrel actually means, so they tend to make it mean anything.

An incomplete list of all the different jobs I've seen described as devrel.

I've seen "developer relations" used to describe a wide range of entirely different roles and skillsets:

Community building — creating spaces and systems for others to succeed
Content creation — tutorials, videos, demos, etc.
Documentation — a different kind of writing entirely
Sales support — talk to prospective customers (usually their engineering teams)
Product support — relaying community feedback for planning and prioritization
Engineering support — acting as "User Zero" for new APIs and features to give early feedback
Marketing support — helping shape the messaging and positioning of the company
R&D support — prototyping and validation of wild ideas
"Being famous" — sometimes a company just wants to be associated with a well-known name (I have feelings about this, but that's for another time)
...and this isn't even a complete list — it's just what I can remember about the work I did in my own devrel roles

None of these skills are necessarily required to get into devrel — but depending on how a given company understands devrel, they might be.

Companies don't think about devrel strategy — so you have to.

Every company has a complex set of strengths, weaknesses, and needs. Devrel will take a different shape in each company.

If the company's leadership isn't comfortable going out in public and talking about the company's vision and product's value, devrel will probably need to tackle that. They'll be looking for someone to get up on stages and get a crowd hyped on your company's promised future.

If the company doesn't have a Documentation team, devrel might get tapped for that.

If the company is weak on Marketing, devrel can fill in for a while. Ditto for Product, Sales, Support, and a host of other roles.

Devrel is organizational duct tape.

Devrel has a tendency to become organizational duct tape. It's a role that attracts generalists who — while they can't do everything — are typically capable of handling some of the tasks of many roles. This partial coverage can help companies handle shifting needs for a short time while they figure out more permanent solutions.

The adaptability of devrel is one of its greatest strengths. But it's also one of the biggest challenges when joining a new company. If the company hasn't really thought through how devrel fits into the organization, it's incredibly hard to predict what kind of tasks you're signing up for.

Ask the right questions so you don't end up with unpleasant surprises.

When you're interviewing, ask the company what the daily tasks for devrel are. Make sure they expect you to do the things you want to do. And if they have no idea, you can make the choice on whether you want to do the work to shape the devrel team at this company — or whether you'd prefer to look elsewhere for a team with a clearer idea of what success looks like in the role.

I've watched extremely talented people take devrel jobs, only to leave the job within their first year — often with a lot of frustration on both sides of the relationship.

If you're deeply technical and your goal in devrel is to be heavily engaged with the Engineering team, building demos, and workshopping APIs to ensure the developer experience is top-notch, that needs to be what the company is looking for in their devrel team.

Otherwise, you might think devrel means digging into the tech, and the company might think it means writing two blog posts per week. Without setting clear expectations, there could be mismatched goals — and you'll end up leaving the job unhappy.

Know what drives you — and what doesn't.

If you're looking to get into devrel, it's really important to understand all the different facets of what devrel can be and to make sure you know which ones you enjoy.

For example, if you love writing and creating tutorials, make sure that's what the devrel team does. There might be a different team with technical writers — and you may want to apply for that role instead.

If you want to go to conferences and give talks on stage, make sure that's what the company expects out of devrel. Especially today, with budgets tightening and conferences struggling to get attendees back after COVID put in-person events on hold, not every devrel team is going to have a travel budget to send you to conferences.

Ultimately, you need to ask yourself: How do I create the role that gives me energy that makes me want to come to work that I'm excited about doing?

Devrel is mostly a vibe — get the job details in writing.

Devrel is too new as a role to have any kind of standardized job description. Every company will define, measure, and prioritize devrel tasks differently.

I know folks who work full time in devrel at two different companies. In one, their job is exclusively conferences: they go to events, work the booth, and generate leads. The other spends exclusively works on writing tutorials, blog posts, and demo apps. Both teams have "Developer Relations" as a title, but the Venn diagram of their actual work is two separate circles.

They're completely different jobs, despite having the same job title.

The Island of Misfit Toys

I often joke that the devrel is the Island of Misfit Toys. Our job in devrel is to understand the gaps and weaknesses in a company, and then fill those gaps. That might mean that we're working on the product one day, then working in engineering the next. Other days, you're working on a marketing campaign or writing docs.

All of these tasks fit under the devrel umbrella.

But remember: depending on the company and the company's maturity (and who the company has already hired), you might never do any of those things — whether or not you want to.

If you want to land a devrel job you'll enjoy and thrive in, you need to write your own ideal job description — then use it to filter the companies until you find one who defines the role the same way.

Getting your first devrel job is hard, but you can give yourself an advantage.

Breaking into devrel is tough, especially if you don't already have a platform. However, having a clear understanding of the broad spectrum of what devrel can be, what your strengths and goals are, and how all that fits into the company's strategy puts you head and shoulders above most candidates.

Most candidates aren't thinking about this.

Most companies aren't thinking about this.

Devrel is not about being an niche internet microcelebrity. You're there to do work, and you're there to provide value to the company.

Remember: if you want to be in devrel, do devrel.

A career is a pie-eating contest where the prize for winning is more pie.

If you want to do a thing, do that thing! If you show somebody that you're good at it, they will reward you by offering you more of that work.

If you show that you understand the space and put out work you want to get more of, you'll eventually land that job. Lots of companies will make accommodations to allow you to write or speak about your work at the company. It's extra press for them, after all.

And remember: it will take time. Keep applying constant, gentle pressure and you'll get the job. 💜

How I balance tech debt vs. feature development

Tue, 13 Jun 2023 00:00:00 GMT

Watch on YouTube

We should treat maintainability as an upstream factor that improves velocity as a downstream effect.

If we build code that’s easy to understand and maintain, velocity stays high.

I’ve worked on dozens of codebases throughout my career. In some cases, changes took weeks (or even months) and were full of frustration, chasing down other teams to address issues our changes would cause in their code, and other complex shenanigans. In others, changes shipped in hours, with minimum hassle and very little stress or cross-team challenges.

The specific architecture didn’t matter. What always made the difference was clear, strongly enforced API boundaries.

The benefit came from:

Extremely clear contracts establishing how the tool could be used (the API)
Architectural enforcement that prevented working around those API boundaries (e.g. no way to reach into the internals to create hard-to-track dependencies)

A clear API contract means changes only have to account for the internal code and the API contract itself. As long as the API surface is unchanged, the team only had to think about the internal code instead of reasoning through the entire company codebase for every update.

This meant the code was separated into independent, reasonably sized, fully understandable chunks. Teams could deliver new features quickly because they needed far less context to feel confident and test their work.

Build for maintainability to boost shipping velocity.

Maintainable code should be easy to delete. Optimizing for deletion makes code easier to refactor, or even fully replace — and it can be hot-swapped without other systems ever noticing or needing to care.

Reducing the external surface area of the codebase starts as building for maintainability, but it results in much higher velocity.

For example, your team might have an API written in Node. Later, you may decide to rebuild in Go or Rust to solve a performance bottleneck. As long as the rebuilt API has the same endpoints, arguments, and return values, none of the services consuming that API don’t even need to be made aware of the change — they’ll just see a performance boost once the new version rolls out.

Nothing can reach its internals and touch things if the API stays accurate. You don't have to worry that changing a line of code will break the whole codebase. There's no way for some other piece of code to reach beyond the API boundary.

Maintainability should be considered a feature of velocity.

By treating maintainability and velocity as separate concerns, it opens the door for the team to say, “We’re going to build fast and not worry about code quality. We’ll come back later and clean up tech debt.”

But later never comes — it is extremely challenging to halt feature work for long enough to tackle a large-scale refactor solely for tech debt cleanup. So instead, tech debt piles up, and every additional mess adds drag to the team’s ability to ship.

The team loses momentum. And momentum is everything. Lost momentum can drain morale, which further decreases velocity as well as care for maintainability (”the app’s already a mess, so why worry about clean code?”) — and that’s a vicious cycle that grinds teams to a halt.

Treat maintainability as a facet of velocity instead of a separate component of the codebase.

In restaurants, cooks are trained to clean as they go. Wipe down surfaces in between steps; wash dishes while waiting for a pot to boil; put equipment away instead of to the side.

All of this reduces clutter and means the kitchen never becomes a giant mess, because the restaurant can’t afford to keep customers waiting while the kitchen shuts down for cleaning mid-service.

When I first started cooking, I struggled with this and pushed back. I’d ask, “Doesn’t this make us way slower?” And the chef would patiently explain that cleaning as part of cooking is what makes us fast. Clean stations reduce mistakes, waste, stress, and much more. We just have to build the habit.

Coding on a team is much the same. It might feel like fixing tech debt and building features are entirely separate tasks, but if we build the habit of cleaning as we go, we can use every pull request as an opportunity to handle a small amount of tech debt and avoid creating more.

It’s not easy, but it can be done.

Change the way you talk about the codebase.

If maintainability and velocity are treated as two sides of the same coin (as opposed to a separate concern), it tends to change the discussion about how we build.

We stop asking, “When will we come back to clean this up?”

Instead, we start asking, “Are we willing to take the velocity hit required to rush this out the door now?”

What used to be a vague “later” becomes a pretty clear “no”.

Companies that ignore this run into compounding slowdowns.

When a company chooses to ignore maintainability under the false assumption that it will make them faster, the tech debt starts to accumulate. At first, it’s not that noticeable — the features are shipping, the codebase is still small, the team still has all the context because it’s relatively new.

But as time continues, as original team members leave and new team members join, as new tech debt stacks on top of old tech debt, things start to slow down. It’s more and more difficult to build things without breaking other things, and velocity grinds to a halt.

Momentum is everything in a company. If you have momentum, it allows for faster learning, quicker adjustment, tighter feedback loops, and overall healthier companies and products.

Maintainability enables velocity, and velocity creates momentum. If you want to build a culture of shipping fast, architect for maintainability and build the habit of cleaning as you go.

Add Feature Flags to a React App (for FREE)

Tue, 06 Jun 2023 00:00:00 GMT

In this tutorial, we'll look at how we can use feature flags to safely and quickly ship new ideas to a small number of users, allowing you to gather real data on how people use it. We'll do this by adding a feature in a React app using DevCycle for feature flags. And we'll build it all in minutes using just a few lines of code.

Watch on YouTube

Demo: https://feature-flag-devcycle.netlify.app/
Repo: https://github.com/learnwithjason/feature-flag-devcycle

Why you should consider feature flags

There's a constant tension between "shipping quickly" and "making sure we only ship things people want" — it undermines our confidence, slows us down, and leads to pretty frustrating meetings.

Feature flags are a programming pattern where we put functionality inside a conditional check, and only show it if the feature flag is set to the correct value.

Used well, feature flags allow us to deploy to production with lower risk.

By lowering the risk, you can ship faster.

By shipping faster, you get real data and feedback.

By getting real data and feedback, you validate that what you're building is the right thing.

By validating your idea, you're far less likely to waste time building and shipping things that no one wants.

Feature flags get you out of hypotheticals and into reality faster.

To get a feel for how to add feature flags into an existing app, let's ship a new feature behind a feature flag in a React app.

Source code: https://github.com/learnwithjason/feature-flag-devcycle Demo: https://feature-flag-devcycle.netlify.app/dashboard/

Set up your dev environment

To start, clone the start branch of the repo, fork it, and install dependencies:

# clone the repo (this uses the GitHub CLI)
gh repo clone learnwithjason/feature-flag-devcycle -- -b start

# move into the project
cd feature-flag-devcycle/

# fork the repo
gh repo fork

# install dependencies
npm i

# start the dev server
npm run dev

Open http://localhost:5173 in your browser to see the app we're going to build.

Create a DevCycle account and project

We'll be using DevCycle to power the feature flags in our app. The free tier will be more than enough to handle our needs.

Head to the DevCycle home page and click the "create account" button, then sign up. I used my GitHub account, but you can use whatever you prefer.

Next, create a new project. You can name this whatever you like — it's only used internally and won't be visible to anyone outside your team.

On the next screen, you'll see your SDK keys for different environments. Copy the client key from the Development environment and put it into the .env file in the app as the value of VITE_DEVCYCLE_CLIENT_KEY.

Vite will detect that the .env file has changed and restart automatically, which is pretty dang cool.

Create a feature flag in DevCycle

In the DevCycle dashboard, go to the Features tab and click "Create New Feature". This opens up a modal that asks what type of feature you want to add.

We want to do a limited release of a new feature, so select "Release".

On the next screen, name the feature "Waff-fulfillment". The key and variable fields will autocomplete. Adding a description is optional, but will be helpful for remembering what the flag is for.

On the next screen, note that the variable key is a boolean value where true maps to "Variation On" and false maps to "Variation Off". Each user on our site can have their own value for the feature flag, which is how we control who sees the new feature or not.

To decide who will see the new feature, scroll down to the "Users & Targeting" section and find the settings for development.

By default, the feature flag will turn the new feature on for all users. We have options to change which users we target, which we'll look at a bit later. We also have options on how to serve the variations of our feature, which is what we want to look at now.

Open the "Serve" dropdown and choose "Random Distribution". This updates the UI to show percentages for both the "on" and "off" variations. By default they're set to 50/50, but we can choose any combination we want.

For this app, 50/50 makes sense, but if you're working on a more established app it might make sense to only show 5% (or even 1%) of users the new feature at first to gather data and feedback before rolling out more widely.

Click save to update the settings.

Add the DevCycle provider and identify the current user

With the SDK key in our environment and a feature flag set up in DevCycle, we're ready to write some code!

Because we're working in React, accessing the feature flag data is made possible by wrapping our app in a provider. DevCycle will let us pass in the current user as an argument to the provider, which means the feature flags will be tied to a user account. This is useful because it means the user will have the same experience across all devices.

To set this up, open src/app.tsx and make the following changes:

	import { BrowserRouter, Route, Routes, useNavigate } from 'react-router-dom';
	import { ClerkProvider, SignIn, SignUp, useUser } from '@clerk/clerk-react';
+	import {
+		useIsDVCInitialized,
+		withDVCProvider,
+	} from '@devcycle/devcycle-react-sdk';
	import { Layout } from './components/_layout';
	import { HomePage } from './components/home';
	import { DashboardLayout } from './components/dashboard/_dashboard-layout';
	import { DashboardHome } from './components/dashboard/dashboard-home';
	import { DashboardWaffles } from './components/dashboard/dashboard-waffles';
	import { DashboardProgress } from './components/dashboard/dashboard-progress';

	import './styles/main.css';

	const MainApp = () => {
-		const { isLoaded } = useUser();
+		const { isLoaded, user } = useUser();

+		// this little maneuver saves us from having yet another split out component
+		const MainAppWithFeatureFlags = withDVCProvider({
+			sdkKey: import.meta.env.VITE_DEVCYCLE_CLIENT_KEY,
+			user: {
+				user_id: user?.id,
+				name: user?.firstName ?? '',
+				email: user?.emailAddresses[0].emailAddress,
+			},
+		})(() => {
+			const dvcReady = useIsDVCInitialized();
+
-			if (!isLoaded) {
+			if (!dvcReady || !isLoaded) {
				return (
					<div className="loading">
						<p>loading...</p>
					</div>
				);
			}

			return (
				<Routes>
					<Route element={<Layout />}>
						<Route path="/" element={<HomePage />} />
						<Route
							path="/login/*"
							element={<SignIn routing="path" path="/login" />}
						/>
						<Route
							path="/register/*"
							element={<SignUp routing="path" path="/register" />}
						/>
						<Route path="/dashboard" element={<DashboardLayout />}>
							<Route index element={<DashboardHome />} />
							<Route path="waffles" element={<DashboardWaffles />} />
							<Route path="progress" element={<DashboardProgress />} />
						</Route>
					</Route>
				</Routes>
			);
+		});
+
+		return <MainAppWithFeatureFlags />;
	};

	/*
	* Clerk needs access to the React Router context, so we need to split out the
	* component to allow for that.
	*/
	const ClerkProviderWithRoutes = () => {
		const navigate = useNavigate();

		return (
			<ClerkProvider
				publishableKey={import.meta.env.VITE_CLERK_PUBLISHABLE_KEY}
				navigate={(to) => navigate(to)}
			>
				<MainApp />
			</ClerkProvider>
		);
	};

	export const App = () => {
		return (
			<BrowserRouter>
				<ClerkProviderWithRoutes />
			</BrowserRouter>
		);
	};

This code has a few key features:

The original output of MainApp gets wrapped with DevCycle's provider using withDVCProvider, which gets stored in a component and returned from MainApp. This might look a bit strange, but it simplifies getting access to the user value from Clerk's useUser() hook.
The SDK key and user details get passed as arguments to withDVCProvider, which connects the app to your DevCycle account and ties the feature flag to the current user.
An additional readiness check is added to make sure DevCycle is loaded before rendering the app using the useIsDVCInitialized() hook.

Once this is saved, the app is now ready for feature flagging!

Modify app navigation based on feature flags

The app dashboard right now shows the experimental "WAF-FULFILLMENT" feature in the left-hand navigation. Our first order of business is making sure only users in our test cohort can see this nav item.

To do that, modify src/components/dashboard/_dashboard-layout.tsx with the following code:

	import { RedirectToSignIn, SignedIn, SignedOut } from '@clerk/clerk-react';
	import { NavLink, Outlet } from 'react-router-dom';
+	import { useVariableValue } from '@devcycle/devcycle-react-sdk';
	import styles from './_dashboard-layout.module.css';

	export const DashboardLayout = () => {
+		const showWaffFulfillment = useVariableValue('waff-fulfillment', false);
+
		return (
			<>
				<SignedIn>
					<div className={styles.dashboard}>
						<nav className={styles.nav}>
							<NavLink
								to="/dashboard"
								className={({ isActive }) => (isActive ? styles.active : '')}
								end
							>
								Dashboard
							</NavLink>
							<NavLink
								to="/dashboard/waffles"
								className={({ isActive }) => (isActive ? styles.active : '')}
							>
								Your Waffles
							</NavLink>
+							{showWaffFulfillment ? (
								<NavLink
									to="/dashboard/progress"
									className={({ isActive }) => (isActive ? styles.active : '')}
							>
									Waff-fulfillment
								</NavLink>
+							) : null}
						</nav>
						<section className={styles.content}>
							<Outlet />
						</section>
					</div>
				</SignedIn>
				<SignedOut>
					<RedirectToSignIn />
				</SignedOut>
			</>
		);
	};

Save the page and — if you're one of the 50% of users to whom the feature flag is set to true — the nav item will disappear.

Add additional targeting to allow easier development

During development, it's helpful to be able to toggle the feature flag on or off for your user to make sure things are working as expected. To do this, head back to your DevCycle dashboard and go back to the "Users & Targeting" section for development.

Click the "Add Targeting Rule" option below the original definition, then use the up arrow button at the right to move the new targeting rule to the top of the list. These rules are evaluated in order, so more specific rules go first.

Give the new rule a name of "Developer Targeting". For the definition, select "User Email". In the second dropdown select "is". In the final input, add your email address.

With this in place, you can update the "Serve" option to be on or off, and when you save the flag will be updated in the app. This works without a reload, which is really powerful because it means you have full control.

Turn the variation on and you'll see the nav item appear. Turn it off and it'll disappear.

Show or hide an announcement banner based on a feature flag

Next, let's update the dashboard so it only shows the announcement banner at the top if the feature flag is enabled.

To do this, make the following changes in src/components/dashboard/dashboard-home.tsx:

+	import { useVariableValue } from '@devcycle/devcycle-react-sdk';
	import { Link } from 'react-router-dom';
	import waffles from '../../data/waffles.json';
	import styles from './_dashboard-layout.module.css';

	export const DashboardHome = () => {
+		const showWaffFulfillment = useVariableValue('waff-fulfillment', false);
+
		return (
			<>
+				{showWaffFulfillment ? (
					<section className="box">
						<div className="boxTopper">
							<h2>NEW! Your Journey Toward Waff-fulfillment</h2>
							<div className={styles.boxControls}>
								<Link to="/dashboard/progress" className={styles.button}>
									check it out &rarr;
								</Link>
							</div>
						</div>
					</section>
+				) : null}
				<section className="box">
					{/* unchanged below this line */}

Save and the banner will disappear when the variation is turned off.

Disable a feature route using a feature flag

As it stands, there are no links presented to a user in the "off" variation. However, if they knew the URL they could still get to the feature manually.

To disable the feature entirely, modify src/app.tsx to only render the route if the feature flag is true for the given user.

	import { BrowserRouter, Route, Routes, useNavigate } from 'react-router-dom';
	import { ClerkProvider, SignIn, SignUp, useUser } from '@clerk/clerk-react';
	import {
		useIsDVCInitialized,
+		useVariableValue,
		withDVCProvider,
	} from '@devcycle/devcycle-react-sdk';
	import { Layout } from './components/_layout';
	import { HomePage } from './components/home';
	import { DashboardLayout } from './components/dashboard/_dashboard-layout';
	import { DashboardHome } from './components/dashboard/dashboard-home';
	import { DashboardWaffles } from './components/dashboard/dashboard-waffles';
	import { DashboardProgress } from './components/dashboard/dashboard-progress';

	import './styles/main.css';

	const MainApp = () => {
		const { isLoaded, user } = useUser();

		// this little maneuver saves us from having yet another split out component
		const MainAppWithFeatureFlags = withDVCProvider({
			sdkKey: import.meta.env.VITE_DEVCYCLE_CLIENT_KEY,
			user: {
				user_id: user?.id,
				name: user?.firstName ?? '',
				email: user?.emailAddresses[0].emailAddress,
			},
		})(() => {
			const dvcReady = useIsDVCInitialized();
+			const showWaffFulfillment = useVariableValue('waff-fulfillment', false);

			if (!dvcReady || !isLoaded) {
				return (
					<div className="loading">
						<p>loading...</p>
					</div>
				);
			}

			return (
				<Routes>
					<Route element={<Layout />}>
						<Route path="/" element={<HomePage />} />
						<Route
							path="/login/*"
							element={<SignIn routing="path" path="/login" />}
						/>
						<Route
							path="/register/*"
							element={<SignUp routing="path" path="/register" />}
						/>
						<Route path="/dashboard" element={<DashboardLayout />}>
							<Route index element={<DashboardHome />} />
							<Route path="waffles" element={<DashboardWaffles />} />
+							{showWaffFulfillment ? (
								<Route path="progress" element={<DashboardProgress />} />
+							) : null}
						</Route>
					</Route>
				</Routes>
			);
		});

		return <MainAppWithFeatureFlags />;
	};

	/* unchanged below this line */

With that change, it's no longer possible to access the feature in any way unless the feature flag is on for the current user.

Feature flags let you stop guessing and start learning

Feature flags are one of the best ways to take the risk out of shipping so you can gather real data instead of guessing what your users want. A good feature flagging workflow gives your team both safety and speed so you can ship new features and experiments faster than ever.

Thanks again to DevCycle for sponsoring this video. Learn more about what you can do with DevCycle on their website.

Resources & Next Steps

How I prep for frontend interviews

Wed, 17 May 2023 00:00:00 GMT

Watch on YouTube

How should devs prepare for frontend interviews?

Today’s interview landscape is wild. One company might ask you to code a page based on a design file. Another might ask you to implement a binary sorting algorithm on a whiteboard from scratch. The style and quality of technical interviewing varies so much between companies that it’s impossible to give a definitive “how to prep for a frontend interview” answer.

Use working in public to ease (or even bypass) technical interviews.

In my career, I’ve focused on preempting the technical interview through building lots of things and posting them in public places. For many companies — at least for the companies that chose to hire me — my public work was a sufficient technical interview because they could see tons of examples of my knowledge and skill level.

When I was the hiring manager for my team, I would use someone’s public work as proof they could do the job, and the technical interview became a conversation where they’d walk me through a piece of code they were proud of and talk about what they liked and why.

To algorithm or not to algorithm?

A lot of companies aren’t willing to accept public work, however, and they use a templated technical interview. For me, this spells doom.

I'll probably never work at Google, because I can’t pass their tech interview. I interviewed at Facebook about ten years ago and I failed the interview so badly that they thought I was the wrong person on the call.

This may have changed, but I interviewed for a front-end position at Facebook and my interview was "write a binary sorting algorithm from scratch" and "write the most efficient query to join two tables". I felt like I was in the wrong interview.
— Jason Lengstorf ⚡️ (@jlengstorf) December 5, 2017

I've never passed an algorithms interview. I don't have a background in computer science. I can't write a bubble sort from scratch. I work with languages that have implemented those algorithms for me already, and I’ve never been able to bring myself to care about learning things solely to pass an interview.

In practice, that’s meant that if I’m in an interview loop, I call out early that I won’t be able to pass an algorithms interview and that we should end now if that’s a requirement. I’ve lost opportunities that way, but choosing to be upfront about it has led to less frustration on both sides.

If you care about working at big companies, you might need to practice your algorithms.

Especially at big companies, the desire to standardize often leads to one-size-fits-all tech interviews, and that often means computer science trivia and whiteboarding. If your goals include adding those companies to your resume, then it’s probably worth diving into something like MIT’s free computer science materials or LeetCode.

Is it practical knowledge that you’ll use every day? Probably not.

But if it helps you land a job that’s meaningful to you, sometimes it’s worth jumping through silly hoops to get there.

Be open and curious to give yourself the best chances.

If your hiring manager is good (by my highly subjective definition of “good”), they’ll be looking at far more than just your ability to memorize an Intro to Algorithms course.

They’ll be looking for someone who’s thoughtful, good at communication, aware of what the business does, and interested in things beyond code.

Building lots of varied projects and sharing them in a place where a potential employer can find them is a great way to broaden your skillset. Thinking about the projects beyond the code — how would the business work? who would the customer be? what would the marketing positioning be? — also demonstrates breadth in a way that can really set you apart.

Learning doesn’t have downsides. Gaining new experience in virtually any field will help you throughout your career. Broad experience means you have additional pools of knowledge to draw from, and that adds unique value.

Remember that employment is a relationship and it needs to be a good fit.

Interviewing is a grab bag of experiences. You get a glimpse into a company’s culture as you interview. Maybe they’ll show you empathy and the tech interview will be highly relevant to the work you’d be doing. Maybe they’ll play gotcha with algorithm trivia. Maybe they feel like they suffered in their coding interview, so they need to make you suffer in turn.

In any case, don’t ignore that information. Remember that while the company is interviewing you, you’re also interviewing the company. If you see something that makes you worry, don’t shrug it off.

Getting a job means you have to do the job you got hired for with the people who hired you. If the interview feels bad, it can often be a sign that the rest of your employment experience won’t be much better.

It can be hard, but if the vibes are off it might be worth declining that job.

Keep learning, keep building, keep sharing, keep participating in the community, and keep applying. You’ll find a job that’s a good fit.

JS vs. No-JS for Websites & How Astro Bridges the Gap

Wed, 10 May 2023 00:00:00 GMT

Watch on YouTube

Why am I so excited about Astro?

In a field that tends toward absolutes, I’ve really loved that Astro has identified a specific set of use cases, embraced that there’s no one-size-fits-all approach to building for the web, and given us smart defaults to start with, plus full control of how far we can choose to go with our apps.

By default, Astro ships HTML and CSS. No JavaScript at all. This is ideal for a substantial portion of the sites on the internet — most of these sites show us text and images without much interactivity or state.

However, unlike previous tools that are — implicitly or otherwise — against JavaScript, Astro fully embraces it. You can add any JavaScript you want, using any framework you want (or no framework), and opt-in to shipping JS on the client side. Astro’s got you covered.

And no matter which approach you choose, you get to keep the modern, component-based workflow that makes the developer experience so pleasant in all our favorite frameworks.

There is a big push right now to recenter on simplicity.

More and more people seem to be questioning why we’re spending so much time on complex solutions to problems that most of don’t have, and I’m extremely here for it.

Web standards have made massive leaps forward since the last wave of JS frameworks rose to popularity. Many things that made something like React necessary in the first place have been implemented natively, and if you keep an eye on the spec discussions, it’s pretty clear that these working groups have no intention of slowing down.

The web is ready to evolve to its next, simpler iteration.

We moved to tools like React because they simplified the mental model of building web apps.

But over time — as any successful tool will do — JavaScript frameworks have expanded.

The cost of success is complexity: more use cases, more edge cases, backward compatibility, a Cambrian explosion of ecosystem tools, and devs using these frameworks to do things they were never intended for.

All of this has led to commentary like Andrew Clark saying, “If you use React, you should be using a meta-framework.” This makes me wonder if we’ve reached the apex of React’s dominance (and maybe the all-in JS framework approach?), and the coming years will see a new approach move in as the “default modern approach” to building for the web.

If you use React, you should be using a React framework. If your existing app doesn't use a framework, you should incrementally migrate to one. If you're creating a new React project, you should use a framework from the beginning.
— Andrew Clark (@acdlite)
January 23, 2023

We've started applying some very high-complexity app approaches to very low-complexity websites.

When I’m lucky enough to get into a deeper conversation about how we build for the web with people who have been doing it for a long time, one of the more common pain points we talk about is how easy it is to go overboard with our projects “just in case” (or worse, “because it’s fun”).

The assignment might be to build a marketing home page, newsletter capture, and blog for our company. Somehow, we end up shipping a server-side rendered behemoth that clocks in at 1.2 MB of JavaScript, and the only interactivity is a fun little easter egg we hid in the hero 🤡

React was built to power Facebook, arguably the most complex app on the internet at the time.

It excels at building highly complex applications. And if we find ourselves building one of those, we should absolutely choose the right tool for the job. But if we’re *not***** building something monstrously complex, well… we should still choose the right tool for the job, which might just be shipping only HTML and CSS.

There are no “sides” in building for the web.

This all brings me back to why I think what Astro is doing is so smart.

They’re telling us to rely on the platform as a default, then giving us a clear, flexible API to add any frameworks and abstractions we need on top of it as part of the Astro authoring experience.

They’re giving us a bridge where other frameworks appear to draw lines in the sand.

On one side, we have a raft of excellent tools like Eleventy, Hugo, and others that excel at producing plain HTML and CSS. However, they actively discourage JavaScript through a complete lack of API support.

The guidance is, effectively: “We can’t stop you from using JS, but we won’t help you either.”

On the other side, JS meta frameworks are pushing an all-JavaScript approach where every bit of the DOM has to be rehydrated into a JS-powered layout, even if there’s no interactivity on the page.

Both sides have strong arguments for why their approach is correct (and, in many cases, it starts to sound like there’s an argument for which side is more “morally correct” which worries me quite a bit). Depending on the project, where I land in the argument will change.

What if the “JS vs. no-JS” argument didn’t have to exist?

Astro takes a completely different approach. “There’s nothing to argue about,” they seem to be saying. “If you don’t need JS, don’t ship it. (We’ll help.) And if you need JS, use whatever you need. (We’ll help with that, too.)”

Astro embraces the experience of building with our favorite frameworks without the trade-off of shipping a pile of unnecessary JavaScript at the end.

In many ways, we get to have our cake and eat it, too.

Rather than accepting the false premise that everyone needs to “choose a side” in the JS wars, Astro just… builds websites. And I love that.

Animations that feel alive using GSAP randomization

Tue, 14 Mar 2023 00:00:00 GMT

Animations are more fun — and more engaging — when they feel "alive". Let’s build a hero section to learn how to take animations from stiff and robotic to something much more natural-feeling with just a few lines of code using GreenSock.

Rather watch along than read? No problem!

A video version of this tutorial is available on YouTube. Don't forget to subscribe!

Watch on YouTube

Set up the project with HTML and CSS

For our animation to work, we need four components:

A cutout image to be in the foreground
A container with a background image that will hold our animations
A button to trigger new animations
A template that contains the markup we want to animate on each button click

To get the starter code that contains the styles and markup used in this tutorial, clone the start branch of the repo:

# clone the start branch of the tutorial repo
git clone git@github.com:learnwithjason/gsap-randomized-animations.git -b start

# move into the cloned repo
cd gsap-randomized-animations/

# install dependencies
npm i

Start the dev server

To start the site in local dev mode, run the following command:

npm run dev

The server will start up the site running at http://localhost:5173. Open it in your browser and you should see a hero banner extolling the virtues of donuts:

Animate an image whenever the button is clicked

First, let's get a basic animation in place. We'll be using GreenSock — also known as GSAP — because it's extremely powerful, free, and compatible with any framework (or no framework at all).

To do that, we will:

Get the template, container, and button using document.querySelector so we can work with them
Create a variable called endY that will tell the heart where it should be at the end of the animation
Add an event listener to the button for click events and prevent the default behavior
Create a copy of the heart image
Add the cloned heart inside the container
Use GSAP to animate the heart using .to(), from its starting position at the bottom of the container to the endY value, which is out of view at the top
At the end of the animation, remove the heart

Removing the heart at the end is important with an animation like this. Since someone can click the button an unlimited number of times, we need to make sure we're not continuously increasing the number of DOM nodes on the page.

import { gsap } from 'gsap';

import './style.css';

const heartTemplate: HTMLTemplateElement = document.querySelector('#heart')!;
const container = document.querySelector('.container')!;
const button = document.querySelector('.button')!;

// move hearts to a position 20% beyond the top of the container so they disappear
const endY = container.clientHeight * -1.2;

button?.addEventListener('click', (event) => {
  event.preventDefault();

  // create a new node from the img element in the template
  const heart = heartTemplate.content.firstElementChild!.cloneNode(true);

  // add the new node to the DOM inside the container
  container.appendChild(heart);

  // animate the heart from its starting position to the endY we defined above
  gsap.to(heart, {
    duration: 2,
    y: endY,
    onComplete: () => {
      container.removeChild(heart);
    },
  });
});

Save these changes, then click the button in your browser. The heart will animate in exactly the same way every time you click.

This is good, but we can make it better. Let's add some randomness to make the animation feel less robotic.

Start the animation from a random position in the container

To make the animation feel a little more alive, we'll have each heart generate from a random position along the bottom of the container. To do that, we'll need to:

Figure out the width of the container
Set the width of the heart
Get a random value between 0 and the width of the container, minus the width of the heart

GSAP provides a helpful set of utility classes, including a random helper. We'll use that to randomize the start position.

Add the following changes to src/main.ts:

	import { gsap } from "gsap";

	import "./style.css";

	const heartTemplate: HTMLTemplateElement = document.querySelector("#heart")!;
	const container = document.querySelector(".container")!;
	const button = document.querySelector(".button")!;

	// move hearts to a position 20% beyond the top of the container so they disappear
	const endY = container.clientHeight * -1.2;
+	const w = container.clientWidth;

	button?.addEventListener("click", (event) => {
		event.preventDefault();

		// create a new node from the img element in the template
		const heart = heartTemplate.content.firstElementChild!.cloneNode(true);

+		// vary the size of the hearts a bit
+		const width = gsap.utils.random(40, 70);

+		// choose a random starting point along the width of the container
+		const initialX = gsap.utils.random(0, w - width);

+		// set initial values for the heart
+		gsap.set(heart, {
+			width,
+			x: initialX,
+		});

		// add the new node to the DOM inside the container
		container.appendChild(heart);

		// animate the heart from its starting position to the endY we defined above
		gsap.to(heart, {
			duration: 2,
			y: endY,
			onComplete: () => {
				container.removeChild(heart);
			},
		});
	});

Save and click the button in your browser a few times. The hearts now show up at different sizes and from different positions.

By randomizing the starting position and size, the animation feels more fun and "alive". But we can do better!

Add a "floating" effect to the animation path

To polish off the animation, we'll bring in GSAP's MotionPathPlugin, which will allow us to make the hearts "float" as they animate upward. Combined with some randomization, we can make the hearts look like they're floating bubbles, drifting upward organically as we click.

To accomplish this, we need to:

Import and register the MotionPathPlugin
Create a floatDirection variables to hold -1 or 1, which we'll use to determine if the heart should float left or right
A function called getNextX() to determine the distance each heart should drift (using the floatDirection to determine to which side it drifts)
Replace the y value in gsap.to() with motionPath config.
1. The hearts should turn to match their path direction. autoRotate: 90 sets the top of the image (default is right side) as the part that should face the path
2. Setting curviness: 1.25 "softens" the float. Try setting it to 0 to see the hearts follow a jagged path
Finally, add an easing function to give the float a bit more dynamism.

Make the following changes to src/main.ts to update the animation:

	import { gsap } from "gsap";
+	import { MotionPathPlugin } from "gsap/all";

	import "./style.css";

+	gsap.registerPlugin(MotionPathPlugin);

	const heartTemplate: HTMLTemplateElement = document.querySelector("#heart")!;
	const container = document.querySelector(".container")!;
	const button = document.querySelector(".button")!;

	// move hearts to a position 20% beyond the top of the container so they disappear
	const endY = container.clientHeight * -1.2;
	const w = container.clientWidth;

	button?.addEventListener("click", (event) => {
		event.preventDefault();

		// create a new node from the img element in the template
		const heart = heartTemplate.content.firstElementChild!.cloneNode(true);

		// vary the size of the hearts a bit
		const width = gsap.utils.random(40, 70);

		// choose a random starting point along the width of the container
		const initialX = gsap.utils.random(0, w - width);

+		// randomize the initial direction of the float
+		const floatDirection = gsap.utils.random([-1, 1]);

+		// get a distance between the starting point & 200px in the given direction
+		const getNextX = (dir: number): number => {
+			return gsap.utils.random(initialX, initialX + 200 * dir);
+		};

		// set initial values for the heart
		gsap.set(heart, {
			width,
			x: initialX,
		});

		// add the new node to the DOM inside the container
		container.appendChild(heart);

		// animate the heart from its starting position to the endY we defined above
		gsap.to(heart, {
			duration: 2,
-			y: endY,
+			motionPath: {
+				autoRotate: 90,
+				curviness: 1.25,
+				path: [
+					{
+						x: getNextX(floatDirection),
+						y: endY / gsap.utils.random(2, 4), // switch up the turning point
+					},
+					{
+						x: getNextX(floatDirection * -1), // reverse float direction
+						y: endY,
+					},
+				],
+			},
+			ease: "power1.in",
			onComplete: () => {
				container.removeChild(heart);
			},
		});
	});

Save and click the button in your browser. You'll see floaty animated hearts!

Give your animations a little more life with randomized values

Adding animation makes apps more engaging, more interactive, and more likely to be shared. And now that you know how to use randomization to make your animations feel a little more alive, you're well on your way to building engaging, interactive, and shareable apps of your own!

Share what you build in the comments! Don't forget to like and subscribe. If you want to learn more about animation, check out the recommended video. See you next time.

Resources

Type-safe, data-driven apps, even if databases freak you out

Tue, 28 Feb 2023 00:00:00 GMT

You can build a database-powered app with end-to-end type safety and real-time updates without needing to learn how to manage databases. This tutorial will show you how.

Finished app: https://convex-dashboard-react.netlify.app/
Source code: https://github.com/learnwithjason/dashboard-convex-react

Prefer to watch instead of reading? I've got you!

Watch on YouTube

If you learn better by watching and listening, I've got a video version of this tutorial up on YouTube. (Don't forget to subscribe if it's useful!)

For frontend devs, databases can be intimidating

If you're a web developer like me, you probably have exciting ideas for new apps all the time.

Most of my app ideas hit the same challenge: building them would require setting up a database.

And I don't know you, but when I start thinking about all the work that goes into setting up and managing a database, I always tend to land on the same solution: I convince myself that my app idea wasn't that good, and I give up.

At least that's how I used to feel. These days, there's a new wave of innovation happening in the database space that's flattening the learning curve for web developers who want to add data to their apps, even if they'd describe themselves as a frontend developer or — as Brad Frost once said — a "front-of-the-frontend" developer.

Build a data-powered app for reacting to dogs

This tutorial will step through adding a Convex database to a web app and using it to store information about dogs and how people have reacted to those dogs. It will have a home page where people can react, and a stats view for showing aggregate data about reactions for each dog.

The app is built using React started from the Vite React TypeScript starter. (npm create vite --template react-ts)

Clone the repo and get local development running

For this app, we'll start with the frontend already built out and styled. Clone the start branch of the tutorial repo, install dependencies, and start the dev server:

# clone the start branch
git clone git@github.com:learnwithjason/dashboard-convex-react.git -b start

# move into the project
cd dashboard-convex-react/

# install dependencies
npm install

# start the local dev server
npm run dev

This is a Vite-powered project, so the local dev server starts at http://localhost:5173. Open it in your browser and you'll see the app:

The home page shows a list of pups, each with a few buttons to let us react. If we toggle to the stats page, we can see the aggregate of how many times each reaction has been hit for each pup.

Right now all the data is hard-coded. We're going to use Convex to store both our pup data and reaction data.

Install and start Convex in the web app

Keep the local dev server running in your terminal. Open up a second terminal and install the convex package.

npm i convex

Next, start Convex in dev mode:

npx convex dev

This will take you through the process of setting up a Convex account, then ask you for the name of your project and generate .env files for local and production development.

The dev command will stay running while we work to keep our local app connected to the Convex datastore.

Define a type-safe database schema

Because we want type safety and autocomplete, our first step is to define a schema. The convex package provides the helpers we need to define the overall schema, data tables, and the field types within the table.

Create a new file at convex/schema.ts:

import { defineSchema, defineTable, s } from 'convex/schema';

export default defineSchema({
  pups: defineTable({
    name: s.string(),
    photo: s.string(),
  }),
  reactions: defineTable({
    pup: s.id('pups'),
    type: s.union(
      s.literal('heart'),
      s.literal('cute'),
      s.literal('star_eyes')
    ),
  }),
});

(see this change on GitHub)

The Convex CLI will automatically update once this file is saved, configuring everything on the backend for us so we don't need to deal with dashboards or additional config.

Wrap the React app with Convex provider

To make sure our app has access to Convex data, wrap it in the Convex provider component by making the following changes in src/main.tsx:

	import React from 'react';
	import ReactDOM from 'react-dom/client';
+	import { ConvexProvider, ConvexReactClient } from 'convex/react';
	import { Toggle } from './components/toggle';

	import './styles/global.css';

+	const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL);

	ReactDOM.createRoot(document.getElementById('root') as HTMLElement).render(
		<React.StrictMode>
+			<ConvexProvider client={convex}>
				<Toggle />
+			</ConvexProvider>
			<footer>
				a <a href="https://www.codetv.dev/">CodeTV</a> creation
				·{' '}
				<a href="https://github.com/learnwithjason/dashboard-convex-react">
					source code
				</a>
			</footer>
		</React.StrictMode>,
	);

(see this change on GitHub)

Creating a client lets the provider know which Convex project we're working with, and the provider makes sure we can run queries and mutations from any component that's rendered inside the provider.

The environment variable (import.meta.env.VITE_CONVEX_URL) was automatically added to .env.local when we started Convex.

Create and display pup entries

Next, we can define how we create and read data to the tables we just defined. Let's start with the pups.

Create a new file at convex/pups.ts:

import { mutation, query } from './_generated/server';

export const add = mutation(async ({ db }, name, photo) => {
  await db.insert('pups', {
    name,
    photo,
  });
});

export const get = query(async ({ db }) => {
  return await db.query('pups').collect();
});

(see this change on GitHub)

Make sure to type out the db.insert() part of this code in your editor. It's extremely cool that the function autocompletes with the names of our two schema tables. Plus, once we select a table, the second argument to db.insert() knows what the fields are.

This is a killer feature of Convex: the database is automatically type safe and we haven't defined any types ourselves, let alone had to mess with configuration files. It Just Works™.

Next up, let's modify the home page to add pups to the database and read them out on screen.

Open src/components/list.tsx and make the following changes:

+	import { useQuery, useMutation } from '../../convex/_generated/react';
	import { seedPups, reactionTypes } from '../util/helpers';

	export function List() {
-		const pups = seedPups;
+		const pups = useQuery('pups:get');
		const addPup = (..._args: any) => {};
		const addReaction = (..._args: any) => {};

		if (!pups) {
			return null;
		}

		if (Array.isArray(pups) && pups.length === 0) {
			seedPups.forEach((pup) => {
				addPup(pup.name, pup.photo);
			});
		}

		return (
			<div className="pups">
				{pups?.map((pup) => {
					return (
						<div className="pup" key={pup._id.toString()}>
							<h2>{pup.name}</h2>
							<img src={pup.photo} alt={pup.name} />

							<div className="reactions">
								{reactionTypes.map((reaction) => {
									return (
										<button
											onClick={() => addReaction(pup._id, reaction.name)}
											key={reaction.label + pup._id}
										>
											<span role="img" aria-label={reaction.name}>
												{reaction.label}
											</span>
										</button>
									);
								})}
							</div>
						</div>
					);
				})}
			</div>
		);
	}

(see this change on GitHub)

Did you see useQuery() autocomplete with our pup query? Again: this Just Works™. Hover over pups and you'll see what data each pup includes.

Right now, the home page will be empty since there aren't any pups in the database yet. Let's fix that.

Update the addPup() function to use a Convex mutation by making the following change:

	import { useQuery, useMutation } from '../../convex/_generated/react';
	import { seedPups, reactionTypes } from '../util/helpers';

	export function List() {
		const pups = useQuery('pups:get');
-		const addPup = (..._args: any) => {};
+		const addPup = useMutation('pups:add');
		const addReaction = (..._args: any) => {};

		if (!pups) {
			return null;
		}

		if (Array.isArray(pups) && pups.length === 0) {
			seedPups.forEach((pup) => {
				addPup(pup.name, pup.photo);
			});
		}

		return (
			<div className="pups">
				{/* ...unchanged... */}
			</div>
		);
	}

(see this change on GitHub)

The code is already written to check if the pups array is empty and add the seed data if so, which means once we save this file, our pup data will be sent to Convex and we'll see it loaded on the page.

Create and display reactions

Next, we want users in our app to be able to react to each pup and see how other people have reacted to them as well.

Let's start by defining a mutation to store new reactions and a query to read them out grouped by reaction type and the pup that was reacted to. In a new file at convex/reactions.ts, add the following:

import { mutation, query } from './_generated/server';
import { reactionTypes } from '../src/util/helpers';

interface DataPoint {
  name: string;
  count: number;
}

interface DataSeries {
  label: string;
  data: DataPoint[];
}

export const add = mutation(async ({ db }, pup, type) => {
  await db.insert('reactions', {
    pup,
    type,
  });
});

export const getByPup = query(async ({ db }) => {
  const reactionsRaw = await db.query('reactions').collect();

  const reactions = await Promise.all(
    reactionsRaw.map(async (reaction) => {
      const pupEntry = await db.get(reaction.pup);

      return { ...reaction, name: pupEntry?.name };
    })
  );

  return reactionTypes.reduce<DataSeries[]>((dataseries, { name, label }) => {
    return [
      ...dataseries,
      {
        label,
        data: reactions
          .filter((r) => r.type === name)
          .reduce<DataPoint[]>((datapoints, r) => {
            const index = datapoints.findIndex((d) => d.name === r.name);

            if (index >= 0) {
              datapoints[index].count += 1;
            } else if (r.name) {
              datapoints.push({
                name: r.name,
                count: 1,
              });
            }

            return datapoints;
          }, []),
      },
    ];
  }, []);
});

(see this change on GitHub)

The mutation is pretty straightforward, but that query might look a little intense. For the bar chart to work, we need to deliver reaction data in the following format:

[
	{
		"label": "💜",
		"data": [
			{
				"name": "Floof",
				"count": 23
			},
			...
		]
	},
	...
]

The code inside the query uses the reaction data to load the right pup data, then uses nested reducers (😅) to shape the data into the format we need.

Next, let's hook up our buttons to save a new reaction in the database. Make one last change to src/components/list.tsx:

	import { useQuery, useMutation } from '../../convex/_generated/react';
	import { seedPups, reactionTypes } from '../util/helpers';

	export function List() {
		const pups = useQuery('pups:get');
		const addPup = useMutation('pups:add');
-		const addReaction = (..._args: any) => {};
+		const addReaction = useMutation('reactions:add');

		if (!pups) {
			return null;
		}

		if (Array.isArray(pups) && pups.length === 0) {
			seedPups.forEach((pup) => {
				addPup(pup.name, pup.photo);
			});
		}

		return (
			<div className="pups">
				{/* ...unchanged... */}
			</div>
		);
	}

(see this change in GitHub)

Save, then click a few of the reactions on your favorite pups to add some reaction entries to the database.

To display these reaction stats, open up src/components/bar-chart.tsx and make the following change:

	import React from 'react';
	import { useMemo } from 'react';
	import { Chart } from 'react-charts';
-	import { getPlaceholderReactionData } from '../util/helpers';
+	import { useQuery } from '../../convex/_generated/react';

	interface DataPoint {
		name: string;
		count: number;
	}

	export function BarChart() {
		const primaryAxis = React.useMemo(
			() => ({
				getValue: (datum: DataPoint) => datum.name,
				showGrid: false,
				innerBandPadding: 0.3,
				innerSeriesBandPadding: 0.05,
			}),
			[],
		);

		const secondaryAxes = useMemo(() => {
			return [
				{
					getValue: (d: DataPoint) => d.count,
					hardMin: 0,
					showGrid: false,
				},
			];
		}, []);

-		const reactions = getPlaceholderReactionData();
+		const reactions = useQuery('reactions:getByPup');

		if (!reactions) {
			return null;
		}

		return (
			<div className="chart-container">
				<Chart
					options={{
						data: reactions,
						primaryAxis,
						secondaryAxes,
						interactionMode: 'primary',
					}}
				/>
			</div>
		);
	}

(see this change in GitHub)

Save and navigate to the stats and you'll see that the reaction data now corresponds to how many reactions you've added.

Deploy a Convex app to production

To deploy the app, follow the deployment instructions in the Convex docs.

Commit all your changes (including the .env file — the Convex URL is not sensitive) and get them up on GitHub
Go to app.netlify.com and either sign in or create an account using your GitHub login
Get your deploy key from the Convex Dashboard
Add it as an environment variable called CONVEX_DEPLOY_KEY in Netlify
Set your build command to npx convex deploy && npm run build

The site will build and deploy to a public URL. Open it up and add some reactions! Your database-powered app is now deployed and running!

But wait, there's more!

If getting a fully type-safe data workflow isn't cool enough for you, there's one more built-in bonus when you choose Convex as your data layer: real-time updates Just Work™.

Open your live site in two browser windows. On one, load the home page with all the pups. In the other, load the stats. Add some reactions and watch the chart update in real time! For even more fun, open the site on your phone and watch the browser update as you tap away on your favorite pups.

This reactivity is automatic, so you don't have to configure anything for it to work. You'll even see realtime updates if you edit your data directly in the Convex dashboard!

Databases are for everyone now — even front-of-the-frontend developers

For web devs who would describe themselves as frontend developers or even "front of the frontend devs", working with data might feel like a lot — but hopefully this tutorial made it feel a little more approachable.

Get out there and build all those fun app ideas!

Resources

Finished app: https://convex-dashboard-react.netlify.app/
Source code: https://github.com/learnwithjason/dashboard-convex-react
Watch this app get built on YouTube
Convex documentation
Unsplash — images of pups came from here
React Charts — used for the stats view

Typesafe Markdown With Astro Content Collections

Tue, 24 Jan 2023 00:00:00 GMT

If you want to learn how to get typesafety and autocomplete into your Markdown blog, we'll go through the whole process of creating a brand new blog powered by Astro content collections in this article.

If you prefer video, I've got you covered

If you prefer learning by watching something get built, I recorded building the demo app in this post as a video tutorial. Learn about Astro content collections and typesafe Markdown on YouTube.

Watch on YouTube

Skip to the end

If you'd rather jump straight to looking at the source code, you can find it here:

Repo: https://github.com/learnwithjason/astro-content-collections
Demo: https://astro-content-collections.netlify.app

Step 1: Set up a new Astro site

To get started, create a new Astro site:

npm create astro

This command kicks off a guided experience (complete with a cute robot mascot) that walks through setting up a new Astro site.

You will be asked to:

Choose a name for the directory where the project will live
Select "an empty project" so we can focus on content collections.
Install npm dependencies.
Initialize a new git repository
- This is only necessary if you want to deploy this site. If you're just playing with content collections to learn, you can skip this step.

Once the site is created, move into the project directory:

cd astro-content-collections/

Start the project to make sure everything is running as expected:

npm run dev

Open up localhost:3000 in your browser and you should see this bare-bones page:

Open the project in your code editor of choice and you're ready to code!

Step 2: Add a layout

To give your blog a cohesive feel, create a layout for the site at src/layouts/default.astro that will be shared across all pages:

---
const { title } = Astro.props;
---

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
    <meta name="viewport" content="width=device-width" />
    <meta name="generator" content="{Astro.generator}" />

    <title>{title}</title>
  </head>
  <body>
    <header>
      <a href="/" rel="home">
        <span role="img" aria-label="Cheese">🧀</span>
      </a>
      <nav>
        <a href="/">Blog</a>
      </nav>
    </header>
    <main>
      <slot />
    </main>
  </body>
</html>

<style is:global>
  * {
    box-sizing: border-box;
  }

  html {
    color: #484844;
    font-family: system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto,
      Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
    font-size: 18px;
    line-height: 1.45;
  }

  body {
    margin: 0;
  }

  h1,
  h2 {
    color: #383833;
    line-height: 1.1;
  }

  img {
    max-width: 100%;
  }
</style>

<style>
  header {
    align-items: center;
    background-color: bisque;
    display: flex;
    justify-content: space-between;
    padding: 0.5rem 5%;
  }

  a[rel='home'] {
    font-size: 2.5rem;
    text-decoration: none;
  }

  nav a {
    color: #383833;
    text-decoration: none;
  }

  main {
    margin: 3rem auto;
    width: min(54ch, 90%);
  }
</style>

Edit the home page to use the new layout

Update the home page to use the layout by editing src/pages/index.astro:

---
import Layout from '../layouts/default.astro';
---

<Layout>
  <h1>Please Read My Excellent Blog</h1>
  <p>
    I write the truth many are too afraid to speak. But I’m not afraid to speak.
    Speaking is just making mouth sounds, and to be honest it’s not very scary.
  </p>
</Layout>

<style>
  article {
    margin-top: 2rem;
  }

  h2 {
    font-size: 1.25rem;
  }

  h2 a {
    color: inherit;
    text-decoration: none;
  }

  h2 a:focus,
  h2 a:hover {
    text-decoration: underline;
  }
</style>

Run npm run dev to see the updated site using the layout:

Step 3: Define a content collection schema

Create a new directory at src/content/ — this is both where our content collections are defined and where the content of our blog posts will live.

Create a config file at src/content/config.ts:

import { z, defineCollection } from 'astro:content';

export const collections = {};

Decide what fields you want in your blog metadata

For our blog, we want the following metadata to be supplied:

Draft status — we don't want to show work-in-progress posts on the live site
Publish date
Title
Category — each post needs to have exactly one category, and we only want to allow two categories: "food" and "wisdom"
Tags — posts can optionally add tags for grouping posts together
Sharing details — for social sharing cards, search results, and other external services, each post can specify custom details:
- Image
- Title
- Description

In a standard Markdown blog, this is a lot of metadata and we'd almost certainly forget things, add extra stuff, or otherwise make a mess of our frontmatter. That's what Astro's content collections are here to help us solve: we can now be strict about what frontmatter is allowed and required for each post and provide helpful error messages if things aren't set up properly!

To do that, we'll use the defineCollection helper and Zod, a schema validation library that's included as z for convenience.

In src/content/config.ts, define your blog schema:

import { z, defineCollection } from 'astro:content';

export const collections = {
  blog: defineCollection({
    schema: z.object({
      draft: z.boolean().default(false),
      date: z.date().transform((str) => new Date(str)),
      title: z.string(),
      category: z.enum(['food', 'wisdom']),
      tags: z.array(z.string()).optional(),
      share: z
        .object({
          image: z.string().url().optional(),
          title: z.string(),
          description: z.string(),
        })
        .strict(),
    }),
  }),
};

Zod provides a clear API for defining how our frontmatter should look.

For the most part, Zod works by defining the property of the schema and using one of Zod's types as the value. For example, title: z.string() lets the schema know that a title must be set and it must be a string.

For more specialized use cases, we can add defaults and transforms, as well as marking things as optional. This is done using schema methods.

For example, the draft field can default to false so that the field can be omitted on publishable posts. We specify that by chaining .default(false) onto the property definition.

The date field will be used as a JavaScript Date object, so we can use the .transform() helper to convert the string representation of the date in frontmatter into a Date object for use in our code.

The share object uses .strict() to ensure that no additional fields are added to it.

Step 4: Show a list of blog posts on the blog home page

With the definitions in place, we can start using the collection to display blog posts on our site! Update src/pages/index.astro to pull in our blog collection and display it on the home page:

---
import { getCollection } from 'astro:content';
import Layout from '../layouts/default.astro';

const posts = await getCollection('blog', (post) => {
  return import.meta.env.MODE !== 'production' || post.data.draft === false;
});
---

<Layout>
  <h1>Please Read My Excellent Blog</h1>
  <p>
    I write the truth many are too afraid to speak. But I’m not afraid to speak.
    Speaking is just making mouth sounds, and to be honest it’s not very scary.
  </p>

  {
    posts.map((post) => {
      return (
        <article>
          <h2>
            <a href={`/blog/${post.slug}`}>{post.data.share.title}</a>
          </h2>
          <p>{post.data.share.description}</p>
          <p>
            <a href={`/blog/${post.slug}`}>full post &rarr;</a>
          </p>
        </article>
      );
    })
  }
</Layout>

<style>
  /* styles are unchanged */
</style>

The getCollection helper allows us to pull in everything in the blog collection, which we then filter to show posts where draft is true in development mode, but not in production.

In the page body, we map over the loaded posts and add markup to display a preview of each post.

If we save this, nothing changes in the browser. This is a good thing: we haven't added any blog posts yet! Empty collections don't throw errors. We could add logic to check for an empty posts array and show an empty state, but in our case we're going to publish a couple blogs right away, so we won't need it.

Step 5: Create your first blog posts

Create your first blog post by adding the following to a new file at src/content/blog/cheese.md:

---
draft: true

date: 2023-01-14
title: Eat Cheese Every Day

category: food
tags:
  - gouda
  - cheddar
  - brie

share:
  image: https://res.cloudinary.com/jlengstorf/image/upload/v1674096555/blog/eat-cheese-every-day.jpg
  title: One Mind-Blowing Life Hack That Will Change The Way You Eat
  description: >
    Forget about keto, Atkins, Whole 30, and every other diet — this revolutionary breakthrough in how we eat will forever change your relationship with food.
---

The new food pyramid:

![a food pyramid where every tier is cheese](https://res.cloudinary.com/jlengstorf/image/upload/v1674096555/blog/eat-cheese-every-day.jpg)

Eat cheese every day and your whole life will change.

The home page will now show the blog preview:

Create a second post by adding the following to a new file at src/content/blog/good-advice.md:

---
draft: false

date: 2023-01-17
title: The Yellow Ones Don’t Stop

tags:
  - survival

share:
  image: https://res.cloudinary.com/jlengstorf/image/upload/v1674108800/blog/yellow-ones-dont-stop.jpg
  title: The Harrowing Truth of Living In New York, Exposed!
  description: >
    The Big Apple may be called a concrete jungle, but if you don’t have the cheat codes for this game of Frogger then it’ll be game over for you, friendo.
---

Learn to spot New York’s apex predator.

![a yellow cab with the text overlay “the color of death”](https://res.cloudinary.com/jlengstorf/image/upload/v1674108800/blog/yellow-ones-dont-stop.jpg)

They’re everywhere. They’re looking for you. And they’re _very_ angry.

After saving, the homepage is now showing an error!

The error message says:

blog → good-advice.md frontmatter does not match collection schema.
"category" is required.

What a great error message! We forgot to add a category field to the frontmatter, and since that one is required the schema validation fails.

This is ideal, because without the typesafety we might not notice this until our build failed, or — worse — until a user let us know that our site is broken.

Edit src/content/blog/good-advice.md to include category: wisdom in the frontmatter, then save and check out the home page again. Everything works as expected!

Right now, though, if we click to read one of these posts we'll get a 404. In the next section we'll create individual blog pages from the collection.

Step 6: Create individual blog pages from an Astro content collection

Since we don't want to create an individual page every time we write a blog post, we'll use a dynamic route for our blog posts.

Create a new file at src/pages/blog/[slug].astro and add the following:

---
import { CollectionEntry, getCollection } from 'astro:content';
import Layout from '../../layouts/default.astro';

export async function getStaticPaths() {
  const posts = await getCollection('blog');

  return posts.map((post) => {
    return {
      params: {
        slug: post.slug,
      },
      props: {
        post,
      },
    };
  });
}

interface Props {
  post: CollectionEntry<'blog'>;
}

const { post } = Astro.props;
const { Content } = await post.render();
---

<Layout>
  <header>
    <h1>{post.data.title}</h1>
    <p>{post.data.share.description}</p>
  </header>

  <Content />

  <footer>
    <ul class="tags">
      {post.data.tags?.map((tag) => <li>{tag}</li>)}
    </ul>
    <p>
      <a href="/">&larr; back to all posts</a>
    </p>
  </footer>
</Layout>

<style>
  header {
    margin-bottom: 2rem;
  }

  header p {
    font-size: 1.125rem;
  }

  footer {
    border-top: 1px solid #d8d8d3;
    margin-top: 2rem;
  }

  .tags {
    color: #787873;
    display: flex;
    gap: 1rem;
    list-style: none;
    margin-bottom: 2rem;
    padding: 0;
  }

  .tags li::before {
    content: '#';
  }
</style>

Using getCollection again, we load all the blog posts, then use getStaticPaths let Astro know the slug and post data for each of our blog posts.

Then, we grab the post out of the props and define our markup in the Astro page body.

To get autocomplete, we define an interface for Props that uses the CollectionEntry<'blog'> type for our post. This is incredibly handy because it means we can press control + space to pull up a list of all the available properties on our post object.

Loading the Markdown content of the post is done by calling await post.render(), which returns an object including a Content component that we can place wherever we want the post body to be displayed.

Save this, click on one of the blog posts, and you've got a fully functional, fully typesafe Markdown blog running using Astro content collections!

Resources and next steps

Congrats on building your first content collection-powered Astro site!

To see this in action and review the source code, check the following links:

Repo: https://github.com/learnwithjason/astro-content-collections
Demo: https://astro-content-collections.netlify.app

Additional resources:

Hardware Accelerated Video Encoding on Apple Silicon (M1 Max, etc.) With FFmpeg

Thu, 19 Jan 2023 00:00:00 GMT

I make a lot of video, and I work with an editor to polish things up before they go out into the world. My cameras' recordings are huge, though (upwards of 50GB each), so we were running into time and storage issues trying to sync these huge files back and forth.

To fix it, I use FFmpeg to transcode the videos at a smaller size. When I first started doing this, it took forever. My recordings are hours long, and running FFmpeg with a software encoder was taking nearly as long as the video itself to encode — when I need to send over 3 camera angles of a 3-hour video, that's a full day of monitoring FFmpeg!

Use your Apple Silicon (M1 / M2 Max, etc.) to run FFmpeg on multiple cores with hardware acceleration

Fortunately, I have a computer with the M1 Max chip in it, so I can use hardware acceleration and the chip's multiple cores to drastically speed up FFmpeg on my MacBook Pro.

Here's the command I use (pulled together from helpful answers like this one) broken up by option with a comment explaining what each one does:

# command for easy copy-paste (replace filenames!)
ffmpeg -i original-file.mp4 -an -c:v h264_videotoolbox -q:v 50 output-file.mp4

# same command, broken up for legibility with comments:
ffmpeg \
  -i original-file.mp4 \   # the input file name
  -an \                    # this drops the audio entirely
  -c:v h264_videotoolbox \ # use Apple Silicon hardware acceleration
  -q:v 50 \                # quality (0 is worst, 100 is best)
  output-file.mp4          # output file name

Switching to the Video Toolbox hardware encoder sped things up to roughly 8.5x speed, meaning a 10-minute video will finish encoding in about 1.2 minutes. Encoding a full 3-hour video took just 21 minutes 34 seconds and dropped the file size from 54.83GB to 2.23GB.

For comparison, using software encoding was closer to 1.5x speed.

Now I can get all camera angles encoded in a couple hours and send less than 20GB of video to my editor. A huge win for a small change to my FFmpeg command setup!

I’m Going Full-Time on Learn With Jason

Mon, 05 Dec 2022 00:00:00 GMT

I’m now full-time on Learn With Jason!

Since the show started in 2018, I’ve made over 300 episodes with close to 250 experts from around the web development community. Episodes of LWJ are now watched more than half a million times every year, with tens of thousands of developers subscribing to the show on YouTube, Twitch, and the Learn With Jason newsletter.

But until now, the show has always been what I’ve done in addition to my day job. Most weeks I only had enough time to show up and record the episodes — there was no time left over for making the show better or exploring new ideas.

That’s why I’m incredibly excited to announce that Learn With Jason is now my full-time job.

Focusing fully on LWJ will allow me to:

Improve the quality of the show (e.g. 1080p streaming)
Develop and release new features (e.g. a weekly newsletter!)
Experiment with new ideas
Create additional content (e.g. more blogs, more videos)

I have a big ol’ backlog of things that I’ve been putting off, saying, “I’d love to do that — if only I had time.” I have time now. Let’s go!

Subscribe to my newsletter for more like this!

How will Learn With Jason pay the bills?

The show has had sponsors (check them out on the home page!) for quite a while now, and that pays the majority of the bills. If your company is interested in sponsoring the show, get in touch at info@learnwithjason.dev!

In addition to sponsorship, I plan on offering a few services starting in 2023:

I’ll run workshops for companies to help teams create and/or improve their devrel programs
I’ll partner with companies to create ambitious media and events (like this project for Netlify)

Since 2020, many of the “tried and true” ideas for connecting with developer communities don’t work as well anymore. I’ve been proving out new and different strategies, and I’m available to help your company adapt to the modern developer ecosystem.

Build a CSS Theme Switcher With No Flash of the Wrong Theme

Wed, 12 Oct 2022 00:00:00 GMT

Not too long ago, the introduction of React Context led to a wave of color theme switchers. For a brief, glorious moment, color theme switchers dethroned TODO apps as the default tutorial.

However, these color theme switchers all shared a flaw: they were all (a little bit) broken. When you selected a non-default theme, they would either:

Briefly flash the wrong color theme (which Chris Coyier dubbed FART, as only Chris Coyier can)
Delay the rendering of the page until client-side JavaScript is able to load and execute to update the theme

Both of these workarounds aren't ideal, and there haven't been many (any?) solutions offered that don't require running a server. In fact, Chris summed up his article on this problem by saying:

I’m not convinced there is a good way to avoid FART without a server-side language or force-delayed page renders.

But — good news, everybody! — I have a solution that will allow any site, whether it's built with a server-side language, a JS framework, or plain ol' HTML and CSS, to add a color theme switcher that works without client-side JS and doesn't delay the page render.

To do this, we're going to use edge computing, which might sound like a lot, but only requires about 60 lines of code and a free Netlify account to add.

Jump to the end: demo · source code

1. Start with a basic HTML and CSS site

For this tutorial, we'll be working from this starter repo.

The solution we're building requires the use of Netlify Edge Functions, which means we'll need the Netlify CLI for local testing and to deploy the site to Netlify.

Clone and deploy the repo

For convenience, click here to create and deploy the repo all at once. This will take you through Netlify's deployment flow, connect your GitHub, create a copy of the repo, and deploy it to your Netlify account.

Set up local development

Once you've got that set up, set up the repo locally (make sure to replace <username> and <repo> with your own username and repo name):

# clone your copy of the repo
git clone git@github.com:<username>/<repo>.git

# move into the cloned repo
cd <repo>

# OPTIONAL: install the Netlify CLI globally
# (v12.1.0 at time of writing)
npm i -g netlify-cli

# start the Netlify CLI
ntl dev

# or, if you don't have the CLI installed globally:
# npx ntl dev

This will start the site at localhost:8888:

There is a theme selection dropdown in the sidebar. Right now, choosing a new theme doesn't change anything — it just adds a ?theme=<selected_theme> to the URL.

Get familiar with the code

Inside the repo, you'll see a src folder that has two HTML files and a CSS file inside.

Open src/index.html and take a look at the top of the file:

<!DOCTYPE html>
<!--     ℹ️ this is the important thing -->
<!--            ⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄⌄    -->
<html lang="en" data-theme="default">
  <!--          ^^^^^^^^^^^^^^^^^^^^    -->
  <head></head>
</html>

The data-theme="default" is the critical piece of this markup.

Below that, there's a form allowing people to choose their theme:

<aside>
  <h2>Choose a Theme</h2>
  <form>
    <select title="Choose a theme" name="theme">
      <option value="default">System Default</option>
      <option value="light">Light</option>
      <option value="dark">Dark</option>
      <option value="pink">Pink</option>
    </select>
    <button type="submit">Switch Theme</button>
  </form>
</aside>

The rest of the markup sets up the example page, but doesn't have any impact on the theme switcher functionality and can be ignored.

Next, open up src/styles.css and look at the top of the file.

html[data-theme='default'],
html[data-theme='light'] {
  --color-bg: #fcfaff;
  --color-header-bg: #2a1f3f;
  --color-header-text: #fcfaff;
  --color-border: #54495f;
  --color-link: #1600ed;
  --color-text: #464064;
  --color-text-heading: #1d1036;
  --color-text-muted: #575280;
}

html[data-theme='dark'] {
  --color-bg: #242526;
  --color-header-bg: #18111f;
  --color-header-text: #e9e9f1;
  --color-border: #54495f;
  --color-link: #617ff5;
  --color-text: #cdcdcf;
  --color-text-heading: #dedeee;
  --color-text-muted: #bebebf;
}

/* ...full file omitted for brevity... */

By targeting the html element using the data-theme attribute, we're able to update the color theme by changing CSS variable values.

To see this in action, open the inspector in your browser and edit the attribute to data-theme="pink". This will cause the site's theme to change.

For our color switcher to work, we need to handle form submissions from the theme chooser, store that value somewhere, then use the value to update the data-theme attribute.

2. Create an edge function

What is an edge function?

An edge function is a small piece of business logic that's executed at the network's edge (which is another way of saying "as close to the user as possible").

This logic is executed between receiving the request from the user and returning the response from the network. An edge function can modify the response before it's returned — if you've ever worked with middleware before, it's similar to that.

The outcome of using edge functions is that you can modify responses based on details about the specific request — the user's cookies, geolocation, or other data sent in the request — without requiring the whole site to be powered by a server or sending client-side JavaScript. This allows us to create personalized experiences in a way that's performant and pleasant to use without a steep learning curve or changing our entire codebase to support edge functions.

Create a Netlify Edge Function in your project

To set up a Netlify Edge Function, create a new folder called netlify, and another folder called edge-functions inside of it. Edge functions will be automatically detected by Netlify when they're stored in this folder structure.

Create a new file at netlify/edge-functions/color-theme.ts. Inside, let’s create the "hello world" of edge functions:

export default async () => {
  return new Response('hello world');
};

Next, we need to configure which route or routes should trigger the edge function. For a color theme, we want that to affect every page on the site, so we'll target all routes.

Open netlify.toml at the root of your project and add the following:

  [build]
    publish = "src"

+ [[edge_functions]]
+   path = "/*"
+   function = "color-theme"

Stop the server (ctrl + C in your terminal), then restart it using ntl dev — localhost:8888 is replaced by the response from our edge function.

Return the original response through the edge function

If an edge function doesn't return anything, by default the page will remain unchanged. This can be helpful for logging or setting cookies without modifying the response itself.

In this example, however, we do want to modify the response, so we need to send the original page response back.

To do this, modify netlify/edge-functions/color-theme.ts with the following:

+ import type { Context } from 'https://edge.netlify.com/';
+
- export default async () => {
+ export default async (request: Request, context: Context) => {
+   const res = await context.next();
+
+   console.log(request.url);
+
-   return new Response('hello world');
+   return res;
  };

Reload the page in your browser and the site will render the same as it did before the edge function was added. However, if you look in the terminal, you'll see that URLs were logged:

◈ Reloading edge functions...
◈ Reloaded edge function color-theme
[color-theme] http://localhost:8888/
[color-theme] http://localhost:8888/styles.css

The HTML file ran through the edge function, which is exactly what we wanted!

However, the CSS file also triggered the edge function, and that's not what we wanted.

Only transform HTML files

To make sure edge functions only execute on the requests we intend to modify, we can check the Content-Type header and bail if it's not text/html.

Update netlify/edge-functions/color-theme.ts with the following:

  import type { Context } from 'https://edge.netlify.com/';

  export default async (request: Request, context: Context) => {
    const res = await context.next();
+   const type = res.headers.get('content-type') as string;
+   if (!type.startsWith('text/html')) {
+     return;
+   }

    console.log(request.url);

    return res;
  };

Reload the page and you'll see that only the URL for the HTML itself shows up in the terminal logs now:

◈ Reloading edge functions...
◈ Reloaded edge function color-theme
[color-theme] http://localhost:8888/

3. Update the HTML with the correct theme

To update the HTML with our correct theme, we need to do three things:

Check a cookie to track which theme is currently selected
Set the cookie to a new theme value when the user makes a theme selection
Transform the data-theme attribute in the HTML to insert the current theme
Update the selected attribute of the theme switcher to make sure the current theme is selected in the dropdown

Use a cookie to track the currently selected theme

In Netlify Edge Functions, cookie management is simplified thanks to the built in context object.

In netlify/edge-functions/color-theme.ts, make the following changes:

  import type { Context } from 'https://edge.netlify.com/';

  export default async (request: Request, context: Context) => {
    const res = await context.next();
    const type = res.headers.get('content-type') as string;
    if (!type.startsWith('text/html')) {
      return;
    }

-   console.log(request.url);
+   const theme = context.cookies.get('lwj-color-theme') ?? 'default';
+   console.log({ theme });

    return res;
  };

Reload the page and see the theme logged in the terminal:

◈ Reloaded edge function color-theme
[color-theme] { theme: "default" }

We haven't set the cookie yet, so it falls back to "default".

Set a cookie with the selected theme

When someone chooses a new theme, it will submit the form to the same page and add the theme as a query string. This is the default behavior of a form that doesn't have an action or method set.

In our app, the edge function will check for a query string and — if one is set — create a cookie using the selected theme.

Add the following to netlify/edge-functions/color-theme.ts:

  import type { Context } from 'https://edge.netlify.com/';

  export default async (request: Request, context: Context) => {
    const res = await context.next();
    const type = res.headers.get('content-type') as string;
    if (!type.startsWith('text/html')) {
      return;
    }

+   const url = new URL(request.url);
+
+   console.log(url.search);
+   if (url.searchParams.has('theme')) {
+     const availableThemes = ['default', 'light', 'dark', 'pink'];
+     const requestedTheme = url.searchParams.get('theme') ?? 'default';
+
+     console.log({ requestedTheme });
+     if (availableThemes.includes(requestedTheme)) {
+       context.cookies.set({
+         name: 'lwj-color-theme',
+         value: requestedTheme,
+         path: '/',
+         secure: true,
+         httpOnly: true,
+         sameSite: 'Strict',
+         expires: new Date(Date.now() + 30 * 24 * 60 * 60 * 1000),
+       });
+     }
+
+     return new Response('Redirecting...', {
+       status: 301,
+       headers: {
+         Location: url.pathname,
+         'Cache-Control': 'no-cache',
+       },
+     });
+   }

    const theme = context.cookies.get('lwj-color-theme') ?? 'default';
    console.log({ theme });

    return res;
  };

This code starts by turning the requested URL into a web standard URL interface, then checks to see if the query parameters include theme. If set, it checks the requested theme against the available themes, then sets a cookie with the requested theme.

Finally, to remove the query string from the URL, the code redirects to the current URL without the query string.

Transform the `data-theme` attribute

Now that we have the current theme name set in a cookie, we can use it to transform the request and set the displayed color theme.

To do this, we'll use a powerful library called HTMLRewriter that allows us to update elements in the response using CSS-like selectors and a straightforward API.

Update netlify/edge-functions/color-theme.ts to import HTMLRewriter and the Element type, then set up the transform for the html element that updates the data-theme attribute:

  import type { Context } from 'https://edge.netlify.com/';
+ import {
+   HTMLRewriter,
+   Element,
+ } from 'https://ghuc.cc/worker-tools/html-rewriter/index.ts';

  export default async (request: Request, context: Context) => {
    const res = await context.next();
    const type = res.headers.get('content-type') as string;
    if (!type.startsWith('text/html')) {
      return;
    }

    const url = new URL(request.url);

    console.log(url.search);
    if (url.searchParams.has('theme')) {
      const availableThemes = ['default', 'light', 'dark', 'pink'];
      const requestedTheme = url.searchParams.get('theme') ?? 'default';

      console.log({ requestedTheme });
      if (availableThemes.includes(requestedTheme)) {
        context.cookies.set({
          name: 'lwj-color-theme',
          value: requestedTheme,
          path: '/',
          secure: true,
          httpOnly: true,
          sameSite: 'Strict',
          expires: new Date(Date.now() + 30 * 24 * 60 * 60 * 1000),
        });
      }

      return new Response('Redirecting...', {
        status: 301,
        headers: {
          Location: url.pathname,
          'Cache-Control': 'no-cache',
        },
      });
    }

    const theme = context.cookies.get('lwj-color-theme') ?? 'default';
    console.log({ theme });

-   return res;
+   return new HTMLRewriter()
+     .on('html', {
+       element(element: Element) {
+         element.setAttribute('data-theme', theme);
+       },
+     })
+     .transform(res);
  };

Save this and reload the browser. Choose a theme other than the default and see that it actually updates now.

However, the dropdown will always show "system default", which isn't ideal.

Update the currently selected theme in the dropdown

To update the dropdown, we need to add another transformation in our HTMLRewriter chain.

Modify netlify/edge-functions/color-theme.ts to transform the option element with a value that matches the currently selected theme and set that to selected="selected":

  import type { Context } from 'https://edge.netlify.com/';
  import {
    HTMLRewriter,
    Element,
  } from 'https://ghuc.cc/worker-tools/html-rewriter/index.ts';

  export default async (request: Request, context: Context) => {
    const res = await context.next();
    const type = res.headers.get('content-type') as string;
    if (!type.startsWith('text/html')) {
      return;
    }

    const url = new URL(request.url);

    console.log(url.search);
    if (url.searchParams.has('theme')) {
      const availableThemes = ['default', 'light', 'dark', 'pink'];
      const requestedTheme = url.searchParams.get('theme') ?? 'default';

      console.log({ requestedTheme });
      if (availableThemes.includes(requestedTheme)) {
        context.cookies.set({
          name: 'lwj-color-theme',
          value: requestedTheme,
          path: '/',
          secure: true,
          httpOnly: true,
          sameSite: 'Strict',
          expires: new Date(Date.now() + 30 * 24 * 60 * 60 * 1000),
        });
      }

      return new Response('Redirecting...', {
        status: 301,
        headers: {
          Location: url.pathname,
          'Cache-Control': 'no-cache',
        },
      });
    }

    const theme = context.cookies.get('lwj-color-theme') ?? 'default';
    console.log({ theme });

    return new HTMLRewriter()
      .on('html', {
        element(element: Element) {
          element.setAttribute('data-theme', theme);
        },
      })
+     .on(`option[value='${theme}']`, {
+       element(element: Element) {
+         element.setAttribute('selected', 'selected');
+       },
+     })
      .transform(res);
  };

Save and reload — it all works!

Fast, no flash, no client-side JavaScript color themes are possible without managing a server

Before edge functions entered the equation, there wasn't a good solution for managing color themes without either some jank on the client side or some extra management on the server side. But today, we're now able to get the benefits of server side HTML rendering without having to set up a server.

Edge computing gives us the capability to make small transforms on the fly, whether we're updating static HTML or transforming SSR-generated pages in a framework like Next.js. It also means we can get some of the benefits of servers without having to pay for servers — edge functions are free!

View the full source code for this demo, or check out the live demo to try it for yourself.

Replace Text & Images Using Edge Functions and HTMLRewriter

Tue, 03 May 2022 00:00:00 GMT

Edge Functions (also referred to as cloud workers or edge compute) are a powerful way to modify requests to web content without modifying the source code or using client-side JavaScript.

Developers can use edge functions to manage a range of tasks, including:

Auth and access control
Proxying and rewriting content
Personalization and customization
Localization and translation
Content transformation
Custom headers
Cookies
and more

In this tutorial, we'll take a look at how to transform an HTML response using HTMLRewriter in a Netlify Edge Function.

Jump to the end: source code | demo

1. Set up an HTML page

For this example we'll work with a simple HTML page with a little bit of custom CSS to make it look nice.

Expand to see the full HTML and CSS used in this example

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>A Treatise on Pizza</title>
    <link rel="stylesheet" href="/styles/main.css" />
  </head>
  <body>
    <main>
      <h1>A Treatise on Pizza</h1>

      <p>
        It is empirically provable that pizza is the superior foodstuff. In this
        essay I will lay out the data-driven case for pizza’s scientific
        superiority.
      </p>

      <h2>Exhibit A: Melty Cheese</h2>

      <img
        class="right cheesy"
        src="https://images.unsplash.com/photo-1593504049359-74330189a345?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=600&ar=0.85:1&fit=crop&q=80&crop=entropy"
        alt="melty cheese pizza"
      />

      <p>
        There’s nothing better in life than melty cheese. It has been proven: in
        a study conducted at my friend’s house, rats were offered a choice
        between a document containing the true meaning of life and a small
        amount of melty cheese. The rats chose the melty cheese
        <em>every time</em> — it doesn’t get more conclusive than that.
      </p>

      <p>
        In another study, I asked my friends to write a list of the things they
        were most willing to suffer for. 87% of respondents were willing to
        suffer intestinal distress if it meant they could eat more cheese.
      </p>

      <h2>Exhibit B: Salty Meats</h2>

      <img
        class="left salty"
        src="https://images.unsplash.com/photo-1601924582970-9238bcb495d9?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=988&q=80&ar=0.85:1&fit=crop"
        alt="a pepperoni pizza"
      />

      <p>
        The presence of a savory, salty ingredient adds a distinct advantage for
        pizza over lesser foodstuffs. If you’ve ever wondered why just about
        every restaurant table includes a salt shaker, but they rarely contain a
        shaker of, like, rosemary, it’s because the human palate has evolved to
        recognize the best flavors, and the best flavors are salty.
      </p>

      <p>
        In a study where half of participants were given a salt shaker filled
        with salt, and the other half of participants were given a salt shaker
        filled with granulated sugar, 100% of respondents were very upset with
        me for putting sugar in a salt shaker.
      </p>
      <p>This is evidence that 100% of people prefer salt.</p>

      <h2>Conclusion: Pizza Is the Best Food</h2>

      <p>
        Based on the ironclad scientific evidence above, we can conclude with
        absolute certainty that pizza, which is basically melty cheese and salty
        meats on bread (which is the second-best foodstuff), is the best
        foodstuff.
      </p>
    </main>
  </body>
</html>

* {
  box-sizing: border-box;
}

html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto,
    Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;
  font-size: 18px;
}

main {
  margin: 4rem auto 6rem;
  width: min(90vw, 54ch);
}

h2 {
  clear: both;
}

.left,
.right {
  aspect-rato: 0.85 / 1;
  margin: 0.75rem 0 1.5rem;
  max-width: 40%;
}

.left {
  float: left;
  margin-right: 1.5rem;
}

.right {
  float: right;
  margin-left: 1.5rem;
}

Create a folder for your project and place the HTML and CSS in a public folder. The folder structure should look like this:

.
└── public
    ├── index.html
    └── styles
        └── main.css

2. Set up local dev and deployment

To make it possible to develop and test Edge Functions locally — and deploy them to production once they're running — create a netlify.toml at the folder root and place the following inside:

[build]
  publish = "public"

Next, install the Netlify CLI if you don't already have it:

npm i -g netlify-cli

Start the site locally by running the following command from your project root:

ntl dev

This will open http://localhost:8888 in your browser and display the HTML.

3. Create an Edge Function

Create your first Edge Function by adding a new file at netlify/edge-functions/pizza-to-tacos.js. Once it’s created, the file tree should look like this:

❯ tree .
.
├── netlify
│   └── edge-functions
│       └── pizza-to-tacos.js
├── netlify.toml
└── public
    ├── index.html
    └── styles
        └── main.css

Inside the Edge Function, let's start by making sure it's running as expected. Add the following code:

export default async () => {
  console.log('Edge Functions are working!');
};

To turn this on, we need to modify our netlify.toml with the following:

  [build]
    publish = "public"
+
+ [[edge_functions]]
+   path = "/"
+   function = "pizza-to-tacos"

The path tells Netlify what URL path should trigger an Edge Function, and the function identifies which Edge Function should be triggered.

Restart your local dev server (ctrl + C, then run ntl dev again), then load the homepage and check the console output to see the log. This means our Edge Function is running.

4. Transform streaming text with an Edge Function

In order to keep responses fast, Edge Functions will stream responses, which means that the response is sent to the browser in chunks instead of waiting until the full document is processed.

This is great for performance, but it can be confusing for transformations because we can't just run a find-and-replace — it's possible that the text we're searching for will be broken across chunks! For example, the sentence "A great big ol' honkin' donut." may be sent in two chunks: "A great big " and "ol' honkin' donut." This means that if we were trying to replace "big ol'" in our text, a straightforward find-and-replace would fail.

Set up HTMLRewriter

To solve for this, we can use the HTMLRewriter tool, which was originally created by the Cloudflare team and later ported for use with Deno and other open runtimes by Florian Klampfer.

Let's implement the HTMLRewriter in our Edge Function by replacing the contents of pizza-to-tacos.js with the following:

import { HTMLRewriter } from 'https://ghuc.cc/worker-tools/html-rewriter/index.ts';

export default async (request, context) => {
  const response = await context.next();

  return new HTMLRewriter().transform(response);
};

It's worth noting that the import is done using Deno-style, URL-based packages. This means we don't need a package.json or other installation step — it will Just Work™.

The Edge Function takes two arguments:

request — a standards-based Request object with information from the incoming request
context — a collection of Netlify-specific helpers and data

To get the data that will be returned to the browser, we use the context.next() helper. Next, we return a new instance of the HTMLRewriter that calls the transform() method on the response.

Write a transform for all text nodes

Transformation works by using an element selector (very similar to CSS selectors), then running a function against the matched elements.

To transform text, we'll use the text method, put each chunk of streamed text into a buffer, and wait until we reach the end of the text node before running our transform to ensure all text is properly transformed.

Add the following code to transform every instance of the word "pizza" to the word "TACOS":

  import { HTMLRewriter } from 'https://ghuc.cc/worker-tools/html-rewriter/index.ts';

+ let buffer = '';
  export default async (request, context) => {
    const response = await context.next();

    return (
      new HTMLRewriter()
+       .on('*', {
+         text(text) {
+           buffer += text.text;
+
+           if (text.lastInTextNode) {
+             text.replace(buffer.replace(/pizza/gi, 'TACOS'));
+             buffer = '';
+           } else {
+             text.remove();
+           }
+         },
+       })
        .transform(response)
    );
  };

Calling text.remove() prevents the text from streaming to the browser before it's replaced. We can do this safely because we're storing all the text into the buffer variable.

Save the changes and reload your local page to see the transformation in action:

5. Transform element attributes in streaming HTML responses

Our transformation isn't complete yet — we're still showing photos of pizza despite updating the text to be about tacos.

To fix this, we'll add an additional transform to select images. On each matched image, we'll get the classes applied to each. If the classes contain either cheesy or salty as class names, we'll transform the src attribute for the element to use a different image and add matching alt text.

Make the following changes to pizza-to-tacos.js:

  import { HTMLRewriter } from 'https://ghuc.cc/worker-tools/html-rewriter/index.ts';

  let buffer = '';
  export default async (request, context) => {
    const url = new URL(request.url);
    if (url.searchParams.get('transform') === 'false') {
      return context.next();
    }

    const response = await context.next();

    return (
      new HTMLRewriter()
        .on('*', {
          text(text) {
            buffer += text.text;

            if (text.lastInTextNode) {
              text.replace(buffer.replace(/pizza/gi, 'TACOS'));
              buffer = '';
            } else {
              text.remove();
            }
          },
        })
+       .on('img', {
+         element(element) {
+           const classes = element.getAttribute('class');
+
+           let newSrc = false;
+           let newAlt = false;
+           if (classes.includes('cheesy')) {
+             newSrc =
+               'https://images.unsplash.com/photo-1464219222984-216ebffaaf85?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=600&q=80&ar=0.85:1&crop=entropy';
+             newAlt = 'hands pulling chips from a cheesy pile of nachos';
+           } else if (classes.includes('salty')) {
+             newSrc =
+               'https://images.unsplash.com/photo-1513456852971-30c0b8199d4d?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=600&q=80&ar=0.85:1&crop=entropy';
+             newAlt = 'a close-up of nachos on a plate';
+           }
+
+           if (newSrc) {
+             element.setAttribute('src', newSrc);
+             element.setAttribute('alt', newAlt);
+           }
+         },
+       })
        .transform(response)
    );
  };

Save the changes, then reload the browser to see the images transformed:

6. Deploy to production

Now that we have a working Edge Function transforming our HTML, we can deploy it to production on Netlify.

To start, make sure you're logged into your Netlify account in the CLI:

ntl login

Next, create a new GitHub repository and push your local code to it. We'll use the GitHub CLI for this example.

git init

# don't track local development files
echo .netlify >> .gitignore

# add all the files we've created
git add -A
git commit -m 'feat: site ready to deploy'

# create a new repo on GitHub using the GitHub CLI
gh repo create

Once the repo is created and you've pushed up your code, initialize a new site with the Netlify CLI:

ntl init

Create a new site, then link your new GitHub repository to it. It will start building immediately and will rebuild any time you push to GitHub.

To view the site, click the link from the CLI output, visit your Netlify dashboard, or run this CLI command:

ntl open:site

Your site is deployed to production and the HTML is being transformed. 😎

To see this site running in production, check out the demo. You can also review the source code on GitHub.

Resources

Acknowledgments

Huge thanks to the Cloudflare team and Florian Klampfer for making HTMLRewriter available. Thanks to @harris on the Cloudflare team for writing up an example of using HTMLRewriter on streaming responses. Also shout-out to Phil and Salma for creating a library of Edge Functions examples that made writing this so much easier.

Use Promise.all to Stop Async/Await from Blocking Execution in JS

Fri, 05 Jun 2020 00:00:00 GMT

Promises are extremely powerful for handling asynchronous operations in JavaScript. Async functions make them easier to read and reason about. However, they also introduce some sneaky traps that can lead to slowdowns if we're not careful.

tl;dr

To make sure your async code isn’t slowing down your apps, check your code to ensure that:

If you’re calling async functions with await, don’t let unrelated async calls block each other.
Don’t use await inside loops. Create an array of Promises and await Promise.all instead.

This article will walk through a few examples and how they can be refactored to avoid blocking execution when using await.

Promises are a wonderful, powerful tool.

When sending off requests to load third-party data or do other asynchronous work, using a Promise has become a common pattern for telling our code to wait until the async work is done before continuing.

We can see this in action by writing a function that simulates a slow network connection: when the function is called, it will wait 1 second before resolving (that's Promise-jargon for "the asynchronous work is complete").

// -------------------------------------------------------------------
// this function simulates a request that needs to run async
// -------------------------------------------------------------------
function getFakeData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve({ action: 'boop' });
    }, 1000);
  });
}

// -------------------------------------------------------------------
// this is the code that actually loads data
// -------------------------------------------------------------------
getFakeData().then((response) => {
  console.log(response.action);
  //=> boop
});

This is a code-along article!

All the code samples in this article can be run in your console. To see these in action, open up developer tools (command + option + I), copy paste the code sample into the console, and press enter.

If you don't want to use your console, you can also use CodePen or create a JavaScript file in your favorite editor to run in your browser.

Promises power many of our data fetching workflows.

One of the most common ways we work with Promises is when loading data with the Fetch API or a library like Axios. For example, if we want to load a random dog image from Dog CEO, our code might use the Fetch API (and Promises) like this:

function getData() {
  // the Fetch API returns a Promise
  fetch('https://dog.ceo/api/breeds/image/random')
    .then((response) => {
      // `.then()` is called after the request is complete
      // this is part of the Fetch API for handling JSON-encoded responses
      return response.json();
    })
    .then((response) => {
      // We can do whatever we want with the data now!
      console.log(response);
    });
}

getData();

Fetch sends off a request to the Dog CEO REST API and waits for a response. Once the response comes back, it resolves a Promise with the data that came back. It provides helpers for handling different response types (JSON in this case), which are called in a .then, and after that point we can do whatever we want with the data by chaining additional .then calls.

Promises don’t return values for use outside themselves.

When I started learning Promises, I struggled to wrap my head around the idea that there’s no straightforward way to get the data out of a Promise; you have to work inside the .then once you’ve started using them:

// -------------------------------------------------------------------
// this function simulates a request that needs to run async
// -------------------------------------------------------------------
function favoritePupper(favorite) {
  return new Promise((resolve, reject) => {
    if (favorite === 'corgi') {
      resolve(favorite);
    } else {
      reject('your preference is incorrect');
    }
  });
}

// -------------------------------------------------------------------
// this is the code that actually loads data
// -------------------------------------------------------------------

// 🚫 this doesn’t work
function getFavoritePupperBroken() {
  // the return value is a Promise, not the value that resolved the Promise
  const pupper = favoritePupper('corgi');

  // that means we can’t return the value from this function 😱
  console.log(`My favorite pupper is a ${pupper}.`);
  //=> My favorite pupper is a [object Promise].
}

// ✅ this works
function getFavoritePupper() {
  const pupper = favoritePupper('corgi');

  // we can only get to the value using the `.then` chain
  pupper.then((doggo) => {
    console.log(`My favorite pupper is a ${doggo}.`);
    //=> My favorite pupper is a corgi.
  });
}

getFavoritePupper();

The reasoning makes sense: Promises are asynchronous, but they shouldn't block synchronous code that doesn't depend on them, so the asynchronous code has to be isolated.

However, trying to "think in Promises" has caused me a lot of headaches. It always takes me a minute to get my brain into the right mode to write Promise-based code.

Nested Promises are hard to keep track of.

If our content has multiple async steps, the nesting in Promises can become challenging to keep track of:

// -------------------------------------------------------------------
// this function simulates a request that needs to run async
// -------------------------------------------------------------------
function doOtherAsyncThing() {
  return new Promise((resolve) => {
    setTimeout(() => resolve('it’s done!'), 500);
  });
}

// -------------------------------------------------------------------
// this is the code that actually loads data
// -------------------------------------------------------------------
function getData() {
  fetch('https://dog.ceo/api/breeds/image/random')
    .then((response) => response.json())
    .then((response) => {
      doOtherAsyncThing().then((otherResponse) => {
        // do stuff with `response` and `otherResponse`
        console.log({ response, otherResponse });
      });
    });
}

getData();

Because next steps have to happen inside a .then(), each async action further nests our code, which adds cognitive overhead and can make code hard to refactor and maintain as time goes on.

Async functions make Promises easier to use...

To make Promises easier to work with, async functions introduce the async and await keywords that allow us to get the benefits of Promises — waiting for an async all to complete before continuing — without the mental overhead of chaining .then calls and nesting Promises.

Let‘s refactor the code we’ve written so far in this article using async functions to make it a little easier to read:

async function getData() {
  // using await means the resolved value of the Promise is returned!
  const response = await fetch('https://dog.ceo/api/breeds/image/random').then(
    (response) => response.json()
  ); // .then still works when it makes sense!

  const otherResponse = await doOtherAsyncThing();

  // do stuff with `response` and `otherResponse`
  console.log({ response, otherResponse });
}

getData();

async function getFavoritePupper() {
  const pupper = await favoritePupper('corgi');

  console.log(`My favorite pupper is a ${pupper}.`);
  //=> My favorite pupper is a corgi.
}

getFavoritePupper();

The await tells our code to wait for the Promise to resolve, then hands back the resolved value of the Promise as the return value. This removes a lot of boilerplate associated with Promises, and that’s a Good Thing™.

...but async functions also introduce new challenges.

Unfortunately, the convenience of async functions comes with some traps that aren't immediately obvious.

Let’s take a look at two examples:

Two unrelated async operations that load data, then do separate things.
An async chain: one operation that loads an array, then a series of calls that depend on the result of the original call.

Two unrelated async operations

With Promises, we might set this up like so:

// -------------------------------------------------------------------
// these functions simulate requests that need to run async
// -------------------------------------------------------------------
function asyncThing1() {
  return new Promise((resolve) => {
    setTimeout(() => resolve('Thing 1 is done!'), 2000);
  });
}

function asyncThing2() {
  return new Promise((resolve) => {
    setTimeout(() => resolve('Thing 2 is done!'), 2000);
  });
}

// -------------------------------------------------------------------
// this is the code that actually loads data
// -------------------------------------------------------------------
function doThings() {
  asyncThing1().then((thing1) => {
    // do something with the first response
    console.log(thing1);
  });

  asyncThing2().then((thing2) => {
    // do something with the second response
    console.log(thing2);
  });
}

doThings();

If you run this in your console, you’ll see that Thing 1 and Thing 2 complete at nearly the same time.

If we refactor to use async functions, our code might look like this:

// 🚫 don’t do this in your real code; it’s slow!
async function doThings() {
  const thing1 = await asyncThing1();
  console.log(thing1);

  const thing2 = await asyncThing2();
  console.log(thing2);
}

doThings();

This looks nice and clean, but if we run it in our console we’ll notice an issue: the code now takes twice as long to run! 😱

Promises run in parallel because they don’t care what happens outside of them — that‘s why we can only access their content inside a .then. This means that both asyncThing1() and asyncThing2() run at the same time in our first example.

In async functions, await blocks any code that follows from executing until the Promise has resolves, which means that our refactored code doesn't even start asyncThing2() until asyncThing1() has completed — that’s not good.

There’s good news, though: we can fix this without giving up on the benefits of async functions!

// ✅ do this — async code is run in parallel!
async function doThings() {
  const p1 = asyncThing1();
  const p2 = asyncThing2();

  const [thing1, thing2] = await Promise.all([p1, p2]);

  console.log(thing1);
  console.log(thing2);
}

doThings();

Because async functions are Promises under the hood, we can run both asyncThing1() and asyncThing2() in parallel by calling them without await. Then we can use await and Promise.all, which returns an array of results once all Promises have completed.

This allows the Promises to run in parallel again, but still gives us a pleasant-to-use syntax that avoids chaining and lets us treat the values in our Promises as standard return values.

Async operations with dependencies

In some cases, we'll have async operations that depend on the results of previous async operations. For example, if we want to load a list of blog posts from one API endpoint, then load comment data for each blog post from another.

With Promises, that setup might look like this:

// -------------------------------------------------------------------
// these functions simulate network requests to load data from an API
// -------------------------------------------------------------------
function getBlogPosts() {
  const posts = [
    { id: 1, title: 'Post One', body: 'A blog post!' },
    { id: 2, title: 'Post Two', body: 'Another blog post!' },
    { id: 3, title: 'Post Three', body: 'A third blog post!' },
  ];

  return new Promise((resolve) => {
    setTimeout(() => resolve(posts), 200);
  });
}

function getBlogComments(postId) {
  const comments = [
    { postId: 1, comment: 'Great post!' },
    { postId: 2, comment: 'I like it.' },
    { postId: 1, comment: 'You make interesting points.' },
    { postId: 3, comment: 'Needs more corgis.' },
    { postId: 2, comment: 'Nice work!' },
  ];

  // get comments for the given post
  const postComments = comments.filter((comment) => comment.postId === postId);

  return new Promise((resolve) => {
    setTimeout(() => resolve(postComments), 300);
  });
}

// -------------------------------------------------------------------
// this is the code that actually loads data
// -------------------------------------------------------------------
function loadContent() {
  getBlogPosts().then((posts) => {
    for (const post of posts) {
      getBlogComments(post.id).then((comments) => {
        console.log({ ...post, comments });
      });
    }
  });
}

loadContent();

This code is perfectly fine. However, the nesting is pretty deep here, and it's a little challenging to track where things are happening — we've got callbacks in loops in callbacks.

To clean this up, our first instinct might be to refactor this code to use async functions:

async function loadContent() {
  const posts = await getBlogPosts();

  for (post of posts) {
    // using await here means the Promise has to resolve before the loop continue
    const comments = await getBlogComments(post.id);

    console.log({ ...post, comments });
  }
}

loadContent();

So much cleaner! 😍

Unfortunately, each request for comments now has to wait for the one before it to complete, meaning our code is now significantly slower — and it'll only get worse as we add more posts!

To clean this up, we can use .map instead of a for loop and create an array of Promises, then use Promise.all to wait for them to complete.

async function loadContent() {
  const posts = await getBlogPosts();

  // instead of awaiting this call, create an array of Promises
  const promises = posts.map((post) => {
    return getBlogComments(post.id).then((comments) => {
      return { ...post, comments };
    });
  });

  // use await on Promise.all so the Promises execute in parallel
  const postsWithComments = await Promise.all(promises);

  console.log(postsWithComments);
}

loadContent();

This code is still much more readable than the nested Promises and for loops, and it allows the requests to load comments to happen in parallel.

Keep async functions fast!

This is just one optimization to keep in mind when writing asynchronous code. Tuck it into your toolkit and keep your async functions fast!

Have other ideas for speeding up async functions? Hit me up on Twitter!

WTF is the Jamstack? A goofy name for a great web architecture.

Fri, 27 Mar 2020 00:00:00 GMT

At a high level, the Jamstack is an architectural approach to web apps that focuses on:

generating cacheable, static assets at build time whenever possible,
deploying those assets to CDNs, and
using client-side JavaScript to call third-party APIs and serverless functions for dynamic interactions and data.

For individual developers, the Jamstack lowers the barrier to entry, cuts down on the number of tools to learn, and provides modern tooling to build web apps.

For teams, the Jamstack helps enforce a more maintainable architecture and decreases iteration time, enabling faster delivery along with increased security and scalability.

For users, Jamstack apps typically load faster and are more reliable, which makes the web a less frustrating place.

The Jamstack is not a traditional tech stack.

The Jamstack is not a “stack” in the same way the term is commonly used, such as the LAMP stack (Linux, Apache, MySQL, PHP) or MERN stack (Mongo, Express, React, Node).

It is true that the origin of Jamstack comes from the acronym JAM, standing for JavaScript, APIs, and Markup (or maybe Markdown? who knows?). However, as people started adopting the principals of Jamstack, it immediately outgrew this stack and came to represent a more generalized approach.

Instead of being a descriptive acronym describing a particular tech stack, Jamstack joins “radar” and “laser” as an often-inaccurately-used acronym that represents a general idea instead of a precise origin.

Jamstack is more of an architecture.

The Jamstack is similar to terms like “microservices“ and “monolith“ because it doesn’t describe the specifics of implementation. Instead, these terms communicate the high-level details about how the code is organized.

If we hear someone describe a codebase as a monolith or microservices, we get a broad idea of the architecture and can make some high-level assumptions about how the code works. However, knowing the high-level architecture doesn’t tell us anything about the specifics of how the code is implemented.

Jamstack is like that: if we hear an app described as a Jamstack app, we can make broad architectural assumptions, but the implementation details can vary widely between teams.

What architectural decisions make an app a Jamstack app?

If the Jamstack allows us to make architectural assumptions, what are those assumptions? In the broadest strokes, the software architecture decisions that make up the Jamstack are:

Assets are generated at build time, not request time.
Apps are deployed to CDNs instead of always-on servers.
Deployments ship atomic, static assets instead of dynamic, derived assets.
Dynamic interactions use APIs and serverless functions instead of monolithic servers.

Assets are generated at build time, not request time

Any performance-minded developer will tell you to cache responses to both speed up our apps and reduce load on our servers. In just about any production stack out there, we can expect to find caching in place.

Often, this is done by waiting for a site visitor to hit a URL, doing work on the server at request time to generate the assets required to render the page, then caching that response so the next visitor is able to get the assets without waiting for the server to do the work.

The Jamstack takes this one step further: if you generate the cache at build time, no one ever has to wait for assets to be generated.

This gives us much more certainty about our deployment process because, as Phil Hawksworth puts it, “by pre-rendering our sites we can be certain that our pages are correct before we deploy them”.

Apps are deployed to CDNs instead of always-on servers

By thinking of our apps as pre-generated caches, we eliminate the need for always-on servers. This creates several huge benefits:

Our sites load faster because we can deploy to Content Delivery Networks (CDNs), putting our app assets closer to the people trying to load them.
We cut down on costs. Using CDNs means we don’t have to pay for massively scalable servers to handle traffic spikes. We’re shipping more reliable sites and reducing operational overhead.
Our apps are more secure because there’s no active connection to a server or database. Sites are made of pre-generated, static assets, which means we don’t need a server or database to display them.
Our apps are more stable. There are very few moving parts between your site visitors and the content they’re requesting, so there are fewer points of failure in the request chain.

Deployments ship atomic, static assets instead of dynamic, derived assets

Because we build the entire site once, we get a very cool benefit: atomic deploys. Once the site is built, we have all of the markup, styles, scripts, data — everything in one place that won't be modified again after the build. This means that if we make a mistake, we can roll back to a previous deploy by straight-up replacing the bad deploy’s files with a previous good deploy.

This also means we can set up previews for new ideas quickly and easily: we can create a branch in our repo, run a build, and then host that site at a private, temporary URL to send the idea around for feedback.

At Netlify, where I work, instant rollbacks, branch deploys, and deploy previews for pull requests are all powered by this concept. I rarely hear anyone talk about it, but this is one of the most powerful benefits of the Jamstack.

Dynamic data uses APIs and serverless functions instead of monolithic servers

Not everything can be cached, however: what if our site’s users log in and need to see their own data? It doesn’t make sense to try and build that ahead of time — it would be a huge security risk, for one thing — so we still need a way to handle user input and load personalized data on-demand.

Handling dynamic data and interactions is a core part of the Jamstack approach, which may seem counterintuitive after we’ve just spent most of this article talking about generating static assets.

This is an important distinction: static assets do not mean static apps.

A JavaScript file is a static asset. Using JavaScript, we can make calls out to other APIs, third-party services, and all sorts of other data sources — this means we have all the same flexibility to generate dynamic content that we have with a traditional server request.

The benefit here is that we get to focus just on the data and business logic instead of building out all the boilerplate for a service and dealing with the operational overhead of keeping it deployed and maintained.

Isn’t this the same as what we used to do?

In a sense, the Jamstack is a return to a very old method of building websites: we create a static asset — an HTML file — and put that online somewhere. If you’ve ever dragged a file into an FTP program to make website changes, this may feel familiar.

The difficulty of manually editing every file on a site and uploading the changes via FTP was high, and it left a lot of room for human error.

As web development evolved, more tooling was created to generate assets automatically. We saw the rise of PHP, followed by the even more dramatic rise of WordPress. We saw Node enter the scene, with frameworks like Express for building our own template-driven servers. We saw Grunt, Gulp, Webpack, and other tools enter the mix to automate complicated build processes.

And somewhere along the way, it became complicated to get a website up on the web.

Jamstack returns to the old way of deploying static files as a folder — we’re swapping out traditional hosting for a CDN — and introduces new innovations that make the process of creating sites boring again.

The Jamstack combines tried-and-true development and deployment strategies with more ergonomic tooling, better abstractions around common infrastructure, and improvements to the JavaScript ecosystem have helped smooth out the developer experience around all this powerful tooling, decreasing the friction to build and deploy web apps.

The Jamstack is not just marketing fluff.

A common criticism of the Jamstack is that it’s just a marketing term for stuff that already exists.

Depending on what you’re predisposed to believe, this statement can fall anywhere on the spectrum of true and false.

Put cynically, the Jamstack is “just putting static files on S3”.

Put charitably, the Jamstack is “revolutionizing web development by removing barriers and giving frontend developers superpowers”.

Somewhere in the middle lies a more objective assessment: the Jamstack is a label for a broad assortment of website-building techniques. These techniques are based on several mature solutions that have existed long before the Jamstack was around to describe it, and also introduces new innovations that streamline the process and make building in this way more approachable.

The Jamstack is dope.

To recap, let’s run through the Big Ideas™ of the Jamstack that we covered in this article:

The Jamstack is not just marketing fluff.
The Jamstack is not actually a tech stack.
The Jamstack is an architectural approach to building web apps that focuses on static assets shipped to CDNs using APIs and/or serverless functions to add dynamic functoinality.
For a large number of web apps, the Jamstack will be faster, more reliable, and more efficient than other architectural approaches.

I’m obviously biased, given that I work at Netlify. However, I love the Jamstack and have been recommending it since long before I took a job at Netlify — I was pushing for this approach when I worked at IBM back in 2016 and didn’t have a word to describe the approach yet. 💜

Are you exploring the Jamstack for an upcoming project? What questions do you have about it? I really want to hear from you — hit me up on Twitter or reply to any of my newsletter messages! (I’m serious, too: hearing what you find confusing, inspiring, frustrating, etc. about the Jamstack helps me know what I should create content about, so please don’t hold back!)

Access Query String Parameters in Serverless Functions

Fri, 10 Jan 2020 00:00:00 GMT

Serverless functions really start to show their potential when we can accept user input and respond to it. The most straightforward way to do this is with query parameters in the browser using GET requests. In this article, we'll look at how to retrieve query parameters in a serverless function and use them to affect the output of our function.

Create your dev environment

If you've been following along with the full series on serverless functions, you can use the same codebase you originally set up. If you're starting fresh, you can clone the starter repo:

# clone the starter branch of the repo
git clone --single-branch --branch starter https://github.com/jlengstorf/serverless-functions.git

# move into the project
cd serverless-functions/

# install dependencies (can also use `npm install`)
yarn

# install the Netlify CLI (can also use npm i -g netlify-cli)
yarn global add netlify-cli

Write the serverless function

To start, let’s create a serverless function that will allow us to boop our friends. Create a file called boop-a-friend.js in the functions folder of your project and write the following code inside:

exports.handler = async () => {
  // TODO get this value from the query string
  const boopee = 'Jason';

  return {
    statusCode: 200,
    body: `You booped ${boopee} on the nose. Boop!`,
  };
};

Run netlify dev and visit http://localhost:9999/.netlify/functions/boop-a-friend and we see the return text, "You booped Jason on the nose. Boop!"

We want to be able to choose who we boop, though, and we want to do that from the browser — meaning we'll be using the GET method to call our serverless function. In practice, it will look like this:

http://localhost:9999/.netlify/functions/boop-a-friend?boopee=Marisa

We're using a query string (the ?boopee=Marisa part) to set our boopee value to "Marisa", because that's who we want to boop.

Next, we need to use the query string value in our serverless function.

Retrieve values from query strings in serverless functions

Because we're using Netlify Functions, we receive the event as the first argument to our serverless function, and it contains a property called queryStringParameters, which contains any query string values as an object.

What this looks like in practice is that if we call our serverless function using the query string above:

http://localhost:9999/.netlify/functions/boop-a-friend?boopee=Marisa

We can access query parameters in our serverless function by making the following changes:

- exports.handler = async () => {
+ exports.handler = async event => {
+   console.log(event.queryStringParameters);

    // TODO get this value from the query string
    const boopee = 'Jason';

    return {
      statusCode: 200,
      body: `You booped ${boopee} on the nose. Boop!`,
    };
  };

When we run netlify dev and call our function, the logs will show us the following output:

Request from ::1: GET /.netlify/functions/boop-a-friend?boopee=Marisa
[Object: null prototype] { boopee: 'Marisa' }
Response with status 200 in 0 ms.

Our boopee is there!

Now we can use it by making a few more adjustments to our code:

  exports.handler = async event => {
-   console.log(event.queryStringParameters);
-
-   // TODO get this value from the query string
-   const boopee = 'Jason';
+   const querystring = event.queryStringParameters;
+   const boopee = querystring.boopee || 'a friend';

    return {
      statusCode: 200,
      body: `You booped ${boopee} on the nose. Boop!`,
    };
  };

Now we can call our function by visiting http://localhost:9999/.netlify/functions/boop-a-friend?boopee=Marisa and we'll see that we're booping Marisa, just like we wanted to!

If we don't add a query string, we fall back to the default text:

We did it! We can now send, parse, and use query string parameters in our serverless functions!

What to do next

Deploy Your First Serverless Function Using JavaScript

Wed, 08 Jan 2020 00:00:00 GMT

Serverless functions are a powerful solution for adding additional functionality to Jamstack sites. In this post, we'll step through creating and deploying your first serverless function using Netlify Functions.

Write your first serverless function

Our first step is to write the serverless function itself. In an empty folder, create a folder called functions, and create a new file called my-first-function.js inside with the following code:

exports.handler = async () => ({
  statusCode: 200,
  body: 'boop',
});

That's all there is to it — you've just written your first serverless function! 🎉 The rest of this article is about getting this function online; we're done coding now.

What are the requirements of a serverless function?

There are three required components in a serverless function:

The file needs to export a function named handler — this is what exports.handler is doing on line 1 above
The function needs to return an object with a statusCode matching a valid HTTP response code
The response object also needs to include a body value, which is plain text by default

Configure your project for deployment to Netlify

With Netlify Functions, we only need two lines of configuration, which we need to save in netlify.toml at the root of the folder:

[build]
  functions = "functions"

This tells Netlify that our functions live in the functions folder.

Create the repo and push to GitHub

At this point, we‘re ready to get this function on the internet!

Create a new repo on GitHub, then add and push our code to it:

# add your new repo as an origin
# IMPORTANT: make sure to use your own username/repo name!
git remote add origin git@github.com:yourusername/yourreponame.git

# add all the files
git add -A

# commit the files
git commit -m 'my first serverless function'

# push the changes to GitHub
git push -u origin master

Create a new Netlify site

You can create your site through the Netlify dashboard or through the CLI. The CLI is really convenient and powerful, so let's use that for this site.

# install the Netlify CLI globally
npm install --global netlify-cli

# log into your Netlify account
netlify login

# initialize a new site
netlify init

This command will set up a new Netlify site in your account connected to the GitHub repo we just created.

It will ask several questions:

What would you like to do? — choose "Create & configure a new site"
Team — choose which Netlify team you want to add this site to
Site name (optional) — choose a name for your site, or press enter to get a randomly-generated name
Your build command — press enter to leave this blank; we don't need it for running functions
Directory to deploy — hit backspace to remove the suggested value, then press enter to leave it blank

Once the site has been created, we can grab the URL from the terminal output. In the above screenshot, the generated site name was:

https://confident-nightingale-4e5a0b.netlify.com/

By default, Netlify functions live at the URL endpoint /.netlify/functions/<function-name> — this is to minimize the chances that the route will conflict with other routes on your site. (We can customize our function URLs with redirects, if we want to.)

Our function file is called my-first-function.js, so it will be accessible on the web at https://confident-nightingale-4e5a0b.netlify.com/.netlify/functions/my-first-function. Go ahead and click that link — it works!

That's all there is to it! You've successfully deployed your first serverless function to Netlify.

What to do next

See the full collection of serverless function examples
Read the Netlify CLI docs on setting up continuous deployment
Learn how to use redirects in Netlify

What Are Serverless Functions and How Do I Use Them?

Wed, 08 Jan 2020 00:00:00 GMT

Serverless functions are an approach to writing back-end code that doesn't require writing a back-end.

In the simplest terms: we write a function using our preferred language, like JavaScript; we send that function to a serverless provider; and then we can call that function just like any API using HTTP methods.

This is huge for front-end developers. We're now able to add powerful "back-end" logic to our apps just by writing JavaScript — no devops, no server code, no fuss.

This also means that our Jamstack apps gain all the benefits that come along with server-hosted apps, but with significantly lower setup and maintenance costs. We can deploy in seconds using Netlify Functions and use our serverless functions immediately.

Put another way: serverless functions turn front-end developers into full-stack developers without requiring us to learn or manage the full stack.

Serverless function examples and tutorials

To explore what serverless functions can do, I created a bunch of small examples — I've collected them in this post, and I'll add additional posts as they’re published.

What to do next

Check out the source code for these examples
Read the Netlify Functions docs

Automatically Generate Social Images for Blog Posts

Mon, 06 Jan 2020 00:00:00 GMT

Creating sharing images for social media is critical if you want your content to stand out, but a lot of content — for example, code-related blog posts — don't always have great visuals to share. If our Twitter cards don't have images, they get lost in the noise — but at the same time, a random image from Unsplash doesn't really communicate what the post contains, either.

The need to create a social media image creates extra chores before publishing: in addition to writing the post, you now need to go find (or create) an image for sharing. It may not be a ton of work, but it's still one more hurdle between you and a published post.

Fortunately, tools exist that allow us to automatically generate social media images. In this post, we'll use Cloudinary, which combines asset hosting with APIs to modify media. This requires a Cloudinary account — the free tier should be more than enough for most personal sites.

Prior art and alternatives.

Before we start, there are a few other options for generating social sharing images out there. These require a little more setup, but they're built completely from open source tools.

Chris Biscardi created gatsby-plugin-printer, which can be used to generate social media images as part of the build process if you're using Gatsby. (The docs are limited, so you'll probably want to look at the source of Chris's website for how this works.)
Shawn Wang wrote a post detailing options for creating social media images, including a DIY solution at the bottom.

For me, this was more than I wanted to take on and maintain, so I opted for a (free) hosted service — Cloudinary — that lets me forget about it entirely after setting it up the first time.

Make a plan: what is the desired outcome?

Before we get started, we need to know where we're headed — so let's make a plan and identify our outcomes. By the end of this article, we want to have a utility function that we can call to generate a social media-friendly image URL that will be unique to our post and help it stand out on people's timelines.

It should look something like this:

Write a function to dynamically generate cards

Now that we know how to build the Cloudinary URLs to create custom social sharing cards, we can write a utility function that builds them for us.

To start, create a new file and add a simple function declaration:

export default function generateSocialImage() {
  // TODO write our function
}

Our image URL has four major components:

The image transforms
The title text overlay configuration
The tagline text overlay configuration
The sharing image template's public ID

Each of those configuration options should have good defaults, but allow us to customize if we want.

Set up the base dimensions and configuration for the social card template

Our first step is to create a configuration object that will be passed to the function. We'll set defaults where possible to give a good outcome without needing to tweak the settings.

Then we'll create our first transformation block, which are the Cloudinary transforms that set the dimensions, cropping, quality, and format of the final image:

export default function generateSocialImage({
  imageWidth = 1280,
  imageHeight = 669,
}: Config): string {
  // configure social media image dimensions, quality, and format
  const imageConfig = [
    `w_${imageWidth}`,
    `h_${imageHeight}`,
    'c_fill',
    'q_auto',
    'f_auto',
  ].join(',');

  // TODO finish the function
}

This places each transform option for the sharing image template into an array, allowing the width and height to optionally be set through the config, and then joined using commas to create a URL-ready transformation.

We can repeat that process for the title and tagline transformation blocks, adding the transformation configurations and the config arguments to make them customizable.

  export default function generateSocialImage({
+   title,
+   tagline,
+   titleFont = 'arial',
+   titleExtraConfig = '',
+   taglineExtraConfig = '',
+   taglineFont = 'arial',
    imageWidth = 1280,
    imageHeight = 669,
+   textAreaWidth = 760,
+   textLeftOffset = 480,
+   titleBottomOffset = 254,
+   taglineTopOffset = 445,
+   textColor = '000000',
+   titleFontSize = 64,
+   taglineFontSize = 48,
  }) {
    // configure social media image dimensions, quality, and format
    const imageConfig = [
      `w_${imageWidth}`,
      `h_${imageHeight}`,
      'c_fill',
      'q_auto',
      'f_auto',
    ].join(',');

+   // configure the title text
+   const titleConfig = [
+     `w_${textAreaWidth}`,
+     'c_fit',
+     `co_rgb:${textColor}`,
+     'g_south_west',
+     `x_${textLeftOffset}`,
+     `y_${titleBottomOffset}`,
+     `l_text:${titleFont}_${titleFontSize}${titleExtraConfig}:${encodeURIComponent(
+       title,
+     )}`,
+   ].join(',');
+
+   // configure the tagline text
+   const taglineConfig = [
+     `w_${textAreaWidth}`,
+     'c_fit',
+     `co_rgb:${textColor}`,
+     'g_north_west',
+     `x_${textLeftOffset}`,
+     `y_${taglineTopOffset}`,
+     `l_text:${taglineFont}_${taglineFontSize}${taglineExtraConfig}:${encodeURIComponent(
+       tagline,
+     )}`,
+   ].join(',');
+
    // TODO finish the function
  }

Only the title and tagline config options are required — everything else has defaults set to match the social sharing card template designed in a companion post.

Next, we need to add these three configuration blocks into a valid Cloudinary URL and add the version (if one is provided) and the image public ID:

  export default function generateSocialImage({
    title,
    tagline,
+   cloudName,
+   imagePublicID,
+   cloudinaryUrlBase = 'https://res.cloudinary.com',
+   version = null,
    titleFont = 'arial',
    titleExtraConfig = '',
    taglineExtraConfig = '',
    taglineFont = 'arial',
    imageWidth = 1280,
    imageHeight = 669,
    textAreaWidth = 760,
    textLeftOffset = 480,
    titleBottomOffset = 254,
    taglineTopOffset = 445,
    textColor = '000000',
    titleFontSize = 64,
    taglineFontSize = 48,
  }: Config): string {
    // configure social media image dimensions, quality, and format
    const imageConfig = [
      `w_${imageWidth}`,
      `h_${imageHeight}`,
      'c_fill',
      'q_auto',
      'f_auto',
    ].join(',');

    // configure the title text
    const titleConfig = [
      `w_${textAreaWidth}`,
      'c_fit',
      `co_rgb:${textColor}`,
      'g_south_west',
      `x_${textLeftOffset}`,
      `y_${titleBottomOffset}`,
      `l_text:${titleFont}_${titleFontSize}${titleExtraConfig}:${encodeURIComponent(
        title,
      )}`,
    ].join(',');

    // configure the tagline text
    const taglineConfig = [
      `w_${textAreaWidth}`,
      'c_fit',
      `co_rgb:${textColor}`,
      'g_north_west',
      `x_${textLeftOffset}`,
      `y_${taglineTopOffset}`,
      `l_text:${taglineFont}_${taglineFontSize}${taglineExtraConfig}:${encodeURIComponent(
        tagline,
      )}`,
    ].join(',');

+   // combine all the pieces required to generate a Cloudinary URL
+   const urlParts = [
+     cloudinaryUrlBase,
+     cloudName,
+     'image',
+     'upload',
+     imageConfig,
+     titleConfig,
+     taglineConfig,
+     version,
+     imagePublicID,
+   ];
+
+   // remove any falsy sections of the URL (e.g. an undefined version)
+   const validParts = urlParts.filter(Boolean);
+
+   // join all the parts into a valid URL to the generated image
+   return validParts.join('/');
  }

Because the version might be null, we run a quick filter to remove any falsy values (like null) from the array, then join all parts using a forward slash (/) to combine everything into a valid URL.

Now we can use our template to create custom title cards!

Create custom sharing cards using the helper function

I've uploaded my template to Cloudinary, and the URL I got back is:

https://res.cloudinary.com/jlengstorf/image/upload/v1578253116/lwj/blog-post-card.jpg

The defaults in our function are based on the template we designed earlier, so with no changes we should end up pretty close to the desired outcome. Let's start by plugging in our cloud name, image public ID, and the title and tagline we want to display:

const socialImage = getShareImage({
  title: 'This Is a Post With an Automatically Generated Social Sharing Card',
  tagline: 'Writing blog posts is fun when the robots do some of the work!',
  cloudName: 'jlengstorf',
  imagePublicID: 'lwj/blog-post-card',
});

console.log(socialImage);

If we visit the URL this produces, we'll see the social card generated with the defaults:

Not bad!

However, I want to use a custom font, tweak the color a bit, and — because I can’t help myself — make a minor adjustment to the line-height of the title. To do that, let's add a few more thing to the config:

  const socialImage = getShareImage({
    title: 'This Is a Post With an Automatically Generated Social Sharing Card',
    tagline: 'Writing blog posts is fun when the robots do some of the work!',
    cloudName: 'jlengstorf',
    imagePublicID: 'lwj/blog-post-card',
+   titleFont: 'lwj-title.otf',
+   titleExtraConfig: '_line_spacing_-10',
+   taglineFont: 'lwj-tagline.otf',
+   textColor: '232129',
  });

  console.log(socialImage);

Now the URL shows us a more customized-looking sharing card!

That's more like it! We now get an automatically generated social card that has my branding on it — this will save me a lot of time as I write more content, and it looks the same as it would if I opened Figma and filled out the template manually.

What to do next

At this point, you have all the tools you need to add automatically generated social media sharing cards to your own blog!

Learn how to design a social media sharing card
Read the Cloudinary docs on text overlays
Use the Figma template to create your own design
Install the npm package for automatically generating social sharing images to add this function to your site
See the source code for this site to see how to use the package in production

After you’ve implemented this, send me a note on Twitter so I can see what you come up with!

Design a Flexible Social Card Template for Sharing Content

Mon, 06 Jan 2020 00:00:00 GMT

When sharing content on Twitter, Facebook, and other social media platforms, adding an eye-catching image is critical to making sure your content doesn't get lost in the timeline. In this post, we'll look at a strategy for designing a flexible, reusable template for sharing posts that helps brand your content without requiring you to create a bespoke image for each post.

Define a template for social media sharing images

In order to create a reusable template for our social sharing images, we need to create a template that will work for any post we create.

For our template, we want to include:

Our logo (or photo) — showing our brand helps associate us with our content.
Post title — this should be limited by the length that Google will show in search results. We can use an SEO snippet generator to check the length of our post titles.
Tagline — this can be any additional text we want to show on the sharing image. In this example, we'll use the post tags.

Our template will put the branding on the left, with the title and tagline on the right, like this:

This means that we need two text overlays — and it needs to look good with any length of title and tagline.

Position the heading on the image

Let's start by getting the heading in the right place on our image.

Because the title needs to be near the tagline no matter what the length of either, we want the title to always be positioned at the bottom of its available area. We'll do this using the gravity transform, setting it to south (for "bottom") and west (for left).

Cloudinary does alignment using directions (i.e. north, south, west, east) instead of calling it top, bottom, left, right — this can be a little confusing the first time you use it if you're not ready for it.

This means that the title will still align well with the tagline, even when the title is short:

Next, we'll set up the tagline.

Position the tagline on the image

The tagline should be near the title no matter its length, so we'll use north (for "top") and west (for "left") as the gravity settings.

This means that even short titles and taglines still look right in the image:

With this setup, we can use any combination of title and tagline that fits within their respective areas and it'll appear properly aligned on the card.

The image template can be anything as long as it leaves the text area clear

Now that we have our text overlays positioned, our sharing image can look any way we want — the only requirement is that the text areas stay clear and have enough contrast to be legible.

The simplest possible image might just be a white background with our logo:

For my site, adding the logo looks like this:

This already looks pretty good! However, I wanted to add a little something extra to make it look more branded, so I put in top and bottom borders and a subtle background pattern, which ended up here:

A longer title and tagline still looks right in this template:

This looks pretty good, if I do say so myself. 😎

What to do next

Use the Figma template to design your own social sharing card
Use Cloudinary to automatically generate your social sharing cards

Upload a Custom Font to Cloudinary Using the Media Library UI

Sun, 05 Jan 2020 00:00:00 GMT

If you want to use custom fonts with Cloudinary, you need to upload them as authenticated assets.

While you can use Cloudinary Node SDK to upload custom fonts (or one of the other SDKs), it's sometime more convenient to use the Media Library UI to drag and drop fonts. However, it's not immediately clear how to upload fonts as authenticated assets.

In this post, we'll look at how to configure the Media Library to upload font files as authenticated assets so we can use them with text overlays.

Create an upload preset

In your Cloudinary console, create a new upload preset by visiting https://cloudinary.com/console/lui/upload_presets/new — you can navigate to this page by clicking the gear in the top-right, then the "upload" tab, and scrolling down near the bottom and clicking the link that says, "Add upload preset".

You'll see the following screen:

On this page, we need to:

Give the preset a name — I recommend using something obvious, like "custom_font_upload_preset"
Choose "Authenticated" from the "Delivery type" dropdown
Click "Save" at the top-right of the page

Use the upload preset for raw files in the media library

Now that we have a preset, we need to tell Cloudinary to use it for "raw" files, which is any file that's not an image or video.

Head to https://cloudinary.com/console/settings/upload — or click the gear, then the "Upload" tab — and scroll all the way to the bottom to find the "Media library’s upload presets".

Under "Raw", choose the upload preset you just created.

Click "Save" at the bottom of the page.

Upload a custom font using the media library

Now that the preset is enabled, we can test it out by uploading a custom font. We're going to use the delightful Snowballs font from Stereo Type.

NOTE: Make sure you have the appropriate licenses for any fonts you upload!

Drag the TTF file onto the media library at https://cloudinary.com/console/media_library — once it's finished, we'll see it show up in our list of assets.

Heads up! Depending on your settings, you may see the font uploaded with a random string instead of the font name — if this happens, you can click on the font and edit the name in the right-hand sidebar that appears.

We know that our upload preset is working because the font has a lock in the bottom-right corner, which means that it's an authenticated upload — this means that there's no public access to the font, which is necessary to avoid violating the terms of most font licenses.

Use the custom font in a text overlay

Once our custom font is uploaded, we can use it by adding its public ID as the font name in a text overlay transformation.

Using an image from our own media library, we can add the text overlay like so:

https://res.cloudinary.com/jlengstorf/image/upload/w_800/g_west,x_30,w_350,c_fit,co_white,bo_4px_solid_black,l_text:snowballs.ttf_180_stroke:Let%20it%20snow!/corgi.jpg

The URL above will display this image with a text overlay using our custom font!

What to do next

For next steps:

Learn how to use text overlays to automatically generate social sharing cards for blog posts
Get more detail on how text overlays work
Use the Node SDK to upload custom fonts programmatically
Read the Cloudinary docs on text overlays

Upload a Custom Font to Cloudinary Using the Node SDK

Sun, 05 Jan 2020 00:00:00 GMT

If you want to use custom fonts with Cloudinary, you need to upload them as authenticated assets. This script will allow you to upload an OTF or TTF to use when adding text overlays to images.

Create a project and install dependencies

Make sure you've created a folder, initialized the Node project, and installed the cloudinary Node SDK:

mkdir upload-fonts

cd upload-fonts/

npm init -y

npm install --save cloudinary

Upload the custom font to Cloudinary

Create a file in this folder called upload.js and add the following inside:

const cloudinary = require('cloudinary').v2;

// this is shown at the top right of https://cloudinary.com/console
const CLOUDINARY_CLOUD_NAME = '<YOUR CLOUD NAME>';

// find these at https://cloudinary.com/console/settings/security
const CLOUDINARY_API_KEY = '<YOUR CLOUDINARY API KEY>';
const CLOUDINARY_API_SECRET = '<YOUR CLOUDINARY API SECRET>';

// path to the custom font (TTF or OTF only), relative to this file
const PATH_TO_FILE = 'my-font.ttf';

// used in Cloudinary URLs — no underscores allowed!
const PUBLIC_ID = 'my-font.ttf';

const uploadToCloudinary = async () => {
  cloudinary.config({
    cloud_name: CLOUDINARY_CLOUD_NAME,
    api_key: CLOUDINARY_API_KEY,
    api_secret: CLOUDINARY_API_SECRET,
  });

  const result = await cloudinary.uploader.upload(PATH_TO_FILE, {
    resource_type: 'raw',
    type: 'authenticated',
    public_id: PUBLIC_ID,
  });

  console.log(result);
};

uploadToCloudinary();

Update the information at the top with your own Cloudinary and font info, then run the script:

node upload.js

You will see the output of the script showing the upload details (or an error, if one occurred). It will look something like this (some details masked):

{ public_id: 'Tahu!.ttf',
  version: 1578268616,
  signature: '342a1f68ea8815b2894dadf8ecc65935ca7daade',
  resource_type: 'raw',
  created_at: '2020-01-05T23:56:56Z',
  tags: [],
  bytes: 39972,
  type: 'authenticated',
  etag: '98c36c6da792d6726ffdba9e6d6989ab',
  placeholder: false,
  url:
    '<http://res.cloudinary.com/jlengstorf/raw/authenticated/[redacted]/v1578268616/Tahu%21.ttf>',
  secure_url:
    '<https://res.cloudinary.com/jlengstorf/raw/authenticated/[redacted]/v1578268616/Tahu%21.ttf>',
  access_mode: 'public',
  original_filename: 'Tahu!' }

NOTE: this example uses the free Tahu! font.

Use the custom font in a Cloudinary image

Now that we've uploaded the font, we can use it for image overlays.

Let's start with this image of a llama:

The URL to display this image is:

https://res.cloudinary.com/jlengstorf/image/upload/w_800,q_auto,f_auto/v1578269094/llama-kimmy-williams.jpg

We can break this URL into four pieces:

https://res.cloudinary.com/jlengstorf/image/upload — this is the URL base, which is going to be the same for all images in my Cloudinary account
/w_800,q_auto,f_auto — these are image transforms tells Cloudinary that I want the image at 800 pixels wide and that the format and quality should be set automatically to best suit this browser
/v1578269094 — this is teh version, an optional piece that allows Cloudinary to cache the image aggressively to cut down on bandwidth and processing
/llama-kimmy-williams.jpg — this is the public ID of the image we want to display and an optional extension telling Cloudinary what format should be used (this will be overridden by f_auto, though)

To insert our custom text overlay, we'll add a new piece in between the image transforms and the version, which will configure text:

/g_south,y_20,co_white,l_text:Tahu!.ttf_70:Have%20you%20been%20to%20Machu%20Picchu%3F

This sets a few things for our text:

g_south — put the text at the bottom-center of the image
y_20 — move the text 20px away from the bottom of the image
co_white — set the text color to white
l_text:Tahu!.ttf_70 — l_text is the overlay config, then we use the custom font's public ID, then _70 to set the font size
:Have%20you%20been%20to%20Machu%20Picchu%3F — we can set any text we want here — the text needs to be URI-encoded

Once we've added this to the URL, it looks like this:

https://res.cloudinary.com/jlengstorf/image/upload/w_800,q_auto,f_auto/g_south,y_20,co_white,l_text:Tahu!.ttf_70:Have%20you%20been%20to%20Machu%20Picchu%3F/v1578269094/llama-kimmy-williams.jpg

And the image now includes our text overlay using the custom font!

What to do next

For next steps:

Learn how to use text overlays to automatically generate social sharing cards for blog posts
Get more detail on how text overlays work
Use the media library to upload custom fonts via UI
Read the Cloudinary docs on text overlays

Add Text Overlays to Images Using Cloudinary

Sat, 04 Jan 2020 00:00:00 GMT

In many cases, it's helpful to add text overlays to images. Sharing content on social media, for example, can be more effective with a sharing image including information about the content.

In this post, we'll look at how we can use Cloudinary to add text overlays to images using URL-based APIs.

How do text overlays work with Cloudinary?

Cloudinary has a whole set of APIs to add overlays to images. We can combine those with other transforms to place text anywhere on an image. All of these transformations are added in the URL of our image.

As a simplified example, let's add some text to a picture of a corgi:

The URL for this image is:

https://res.cloudinary.com/jlengstorf/image/upload/w_800/corgi.jpg

This will display the corgi image at 800px wide because we use the w_800 transform in the URL.

Add a text overlay to an image using Cloudinary

To add text, we're going to add a second transform — transforms in Cloudinary are separated by forward slashes (/) — that tells Cloudinary to add a text overlay using the font Arial at 64pt size:

https://res.cloudinary.com/jlengstorf/image/upload/w_800/l_text:arial_64:Ready%20to%20party/corgi.jpg

The change here is this string:

/l_text:arial_64:Ready%20to%20Party

l_text: tells Cloudinary that we're going to do a text overlay
arial_64: configures the text overlay to use Arial at 64pt
Ready%20to%20party is the URL-encoded text we want to display

If we open this URL in a browser, we’ll see the text overlaid on the image:

Position the text overlay on an image using Cloudinary

Our image looks okay, but it would be better if the text wasn't on top of our doggo. Fortunately, Cloudinary allows us to position text using additional transforms.

First, let's set the "gravity" of the text so it's anchored to the bottom-left corner. This is done using directions for whatever reason, so bottom and left are considered south and west.

Add g_south_west to the text overlay transformation section of the URL, separating it with a comma:

https://res.cloudinary.com/jlengstorf/image/upload/w_800/g_south_west,l_text:arial_64:Ready%20to%20party/corgi.jpg

Now the image looks like this:

This is better, but now the text is crammed against the edge of the photo — not great.

Add X and Y offsets to text overlays in Cloudinary

We can adjust this by changing the X and Y offsets, though. The offsets start from wherever the gravity is set, so in this case our X offset will be from the left edge and the Y offset will be from the bottom.

These are set as x_40,y_40 in the text overlay transformation section:

https://res.cloudinary.com/jlengstorf/image/upload/w_800/g_south_west,x_40,y_40,l_text:arial_64:Ready%20to%20party/corgi.jpg

Now the image is offset from the edge of the image by 40px at the left and bottom:

Almost there!

Set text overlays to wrap at a specified width in Cloudinary

Now let's get the text to wrap so it doesn't overlay the corgi at all. We can do this by setting a width and telling the text to "fit" using the crop settings.

These are set as w_250,c_fit in the transformation section:

https://res.cloudinary.com/jlengstorf/image/upload/w_800/g_south_west,x_40,y_40,w_250,c_fit,l_text:arial_64:Ready%20to%20party/corgi.jpg

Now it looks pretty good!

What to do next

To take your text overlays to the next level, try:

Data abstraction in Gatsby themes (and React apps in general)

Mon, 25 Mar 2019 00:00:00 GMT

It’s an inevitability that, over time, any actively maintained application will grow in complexity.

In general, tools are designed for a specific level of complexity. We call this tool “entry level”, while that one is “enterprise-ready”.

Level-based tooling works on the assumption that applications won’t graduate from one level to the next. (Or, if they do move to the next level of complexity, they’ll be able to support a complete rewrite in a tool better suited for their new needs.)

This seems to be a generally accepted practice: “Just get something quick-and-dirty built to prove this out, then we’ll go back and rebuild it to be ‘enterprise-ready’ later.”

But to me this feels... wasteful.

Why shouldn’t our tools grow up with us? Why should we be forced to throw away previous work because our app needs new features?

We should build tools that adapt to increasing demands

If we’re deliberate about the architecture of our tools, we can use progressive disclosure of complexity to create apps that start as entry-level, beginner-friendly starter kits, but allow developers to opt out of specific abstractions on a per-case basis — all the way up to taking full control.

This is something we’ve been thinking hard about at Gatsby, and it’s the philosophical underpinning of what we’re trying to accomplish with themes.

The goal is to start with nothing but a theme and a data source (like a folder full of Markdown files), but allow developers to progressively peel back the abstractions until they’re poking at the underlying Webpack and Babel configuration.

Layers of abstraction in a typical web app

If we leave the really hardcore customization out for now, we’re left with three main levels of abstraction:

The content layer — the information displayed on the site. This could be a folder full of Markdown, a headless CMS, or some kind of dashboard or database. No code is required at this layer.
Presentation — this is a theme that defines markup and styles for presentational components. This layer is only concerned with the UI; it doesn’t care where data comes from.
Data orchestration — this theme executes queries and provides the data to components as props. This layer is what actually connects to the source of content, whether that’s the filesystem, a database, or a third-party API.

How to set up abstraction layers in Gatsby themes

As theme authors, we can choose the level of abstraction each theme addresses. Later in this post we’ll look at real site using a Gatsby theme, but — in reality — our theme is actually two themes:

a parent theme to handle the data layer
a child theme to handle the presentation layer

There’s a reason for this, and it’s among the more exciting parts of Gatsby themes in my mind: by abstracting data management, we can define a schema for data — and then stop worrying about where the data comes from.

In practical terms, this means — assuming the abstractions are done properly — a Gatsby theme for managing Post data can be created as a schema contract, and dozens (or hundreds, or thousands) of presentation-layer themes can be built against that schema contract: they know that a Post has a title and content, so they confidently grab that data and style it.

But then — and here’s where this gets really exciting — any data source can be adapted into the Post format. In this example we’re mapping MarkdownRemark nodes to the Post type, but there’s nothing stopping us from also mapping data from a headless CMS, JSON, or literally any other data type to the Post type.

This is huge, because it means that we have the potential to create a shared pool of themes that work for any data source. To the best of my knowledge, this has never been possible before; if you like a Woocommerce WordPress theme but want to use Shopify, you’d need a developer to port the WordPress theme into a Shopify theme.

With data abstraction in Gatsby themes, the base theme would query against a Product type, and product data from WordPress and product data from Shopify would be mapped to the Product schema.

This means that the same theme that worked for WordPress will work for Shopify with zero code changes.

To put these abstractions in context, let’s build a blog using a Gatsby theme, look at the layers of abstraction, then opt out of them to customize our app.

Build a site using Gatsby themes

To start, let’s create a site using a Gatsby theme that requires almost no code or configuration.

Step 1: create a folder for the site

To start, create a new directory and move into it. This will be our blog site.

# Create the site folder and move into it.
mkdir data-abstraction-example-site
cd data-abstraction-example-site/

Step 2: add a `package.json`

Next, create a package.json:

# Create a package.json with the default settings.
npm init -y

Step 3: install dependencies

To use a Gatsby theme, we need to install Gatsby, React, React DOM, and the theme itself:

npm install gatsby react react-dom @jlengstorf/gatsby-theme-style

Step 4: tell Gatsby to use the theme

As our final code step, let’s tell Gatsby to use the theme by creating a gatsby-config.js:

module.exports = {
  __experimentalThemes: ['@jlengstorf/gatsby-theme-style'],
};

Step 5: add a post

Next, we can add content. Create a folder called content/posts/, then add a new file called foo.md with the following content:

---
title: Super Sweet Post
date: 2019-02-28
author: Jason Lengstorf
---

This is blog content!

[Gatsby themes](https://www.gatsbyjs.org/blog/2019-03-11-gatsby-themes-roadmap/) are cool.

Step 6: start the site

Once this is set up, we can start the app and see our post:

gatsby develop

Once the server starts, we can open http://localhost:8000/posts/ to see the content we created.

What makes Gatsby themes different?

With almost no code, we’ve created a complete blog from scratch. On its own, this is exciting, but not groundbreaking. Pretty much every website builder out there has themes in one form or another.

So what’s so special about Gatsby themes?

What sets Gatsby themes apart from other theming systems is the ability to selectively opt out of parts of the abstraction. We can keep the convenience in all the places where the defaults suit our needs and make customizations where they don’t.

In short: Gatsby themes don’t force us to choose between convenience and control.

To understand the power of Gatsby themes, we need to start evolving our site and outgrowing the default settings.

Selectively opting out of abstractions

As our site grows, we may decide that we want more control over the layout. Perhaps we want to add a link to our Twitter to the end of each post.

To do this, we take advantage of component shadowing. This is a technique that allows us to selectively replace parts of the theme without needing to eject the entire theme.

Gatsby handles component shadowing by looking in a site’s src directory for a folder named after the theme being used, then looking for paths that match the theme’s structure.

For example, if the theme is called gatsby-theme-foo and it has a file located at src/components/bar.js:

.
└── gatsby-theme-foo
   └── src
   └── components
   └── bar.js

Then we can shadow this component by creating a new file in our site here:

.
└── gatsby-theme-foo
└── src
└── gatsby-theme-foo
└── components
└── bar.js

To shadow our <Post> component, we need to create a new file in our site at src/@jlengstorf/gatsby-theme-data/components/post.js and add the following:

import React from 'react';

const Post = ({ title, content, author, date }) => (
  <React.Fragment>
    <h1 className="post-heading">{title}</h1>
    <p className="post-byline">
      Posted on{' '}
      <time dateTime={new Date(date).toISOString()}>
        {new Date(date).toLocaleDateString('en-US', {
          year: 'numeric',
          month: 'long',
          day: 'numeric',
        })}
      </time>{' '}
      by {author.name}
    </p>
    <div
      className="post-content"
      dangerouslySetInnerHTML={{ __html: content }}
    />
    // highlight-start
    <p>
      For more content like this, you should{' '}
      <a href="https://twitter.com/jlengstorf">follow me on Twitter</a>.
    </p>
    // highlight-end
  </React.Fragment>
);

export default Post;

After saving the new component, stop the site (control + C), then start it again:

gatsby develop

The post now shows a link to Twitter at the bottom.

We can go much deeper

It’s also possible to modify the underlying data, add new components, or even compose themes together (e.g. a blog theme and an ecommerce theme). We won’t get into that in this post, but the potential of Gatsby themes is extremely high.

Automatic WordPress Deployment + Free SSL: Trellis How-To

Mon, 17 Oct 2016 00:00:00 GMT

Watch on YouTube

Elevator Pitch:

Set up a new WordPress site using the Roots stack.

Prerequisites

A private server (we'll walk through how to set this up for $5/month) — Trellis cannot run on a shared host, so this isn't optional.
Homebrew (v1.0.7 used in this tutorial)

# Install Homebrew.
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Git (v2.10.1 used in this tutorial)

# Install Git with Homebrew
brew install git

Ansible (v2.1.2.0 used in this tutorial)

# Install Ansible with Homebrew
brew install ansible

Composer (v1.2.1 used in this tutorial)

# Install Composer with Homebrew
brew install composer

Virtualbox (v5.0.18r106667 used in this tutorial)
Vagrant (v1.8.5 used in this tutorial)

Part I: Install Trellis

To get things rolling, we need to grab a copy of Trellis from GitHub.

Create a new directory for the site.

# Move into the directory where where you keep dev projects.
cd ~/dev/code.lengstorf.com/projects/

# Create a new directory for this project
mkdir learn-trellis

# Move into the new directory.
cd learn-trellis/

Get a copy of Trellis to manage environments and deployment.

Since we want to track our copy of Trellis as its own project, we don't actually want the Trellis repo information. By adding --depth=1 to the clone command, we're able to get only the most recently committed version (avoiding a lot of wasted bandwidth downloading the commit history). Then, we delete the .git directory so we can have our own Git project.

# Clone Trellis, but without all the Git history.
git clone --depth=1 git@github.com:roots/trellis.git

# Delete the `.git` file so we can have our own Git repo.
rm -rf trellis/.git

Install Ansible dependencies.

One of the most powerful parts of Ansible is the ability to use community-supplied scripts — called "roles" in the Ansible world — rather than having to write them from scratch or copy-paste them from various forums and tutorials.

This collection of community roles is called Galaxy, and it's home to thousands of roles. Trellis uses several of these to configure a WordPress server, so we need to get those installed.

# Move into the Trellis directory
cd trellis/

# Install the Ansible dependencies for Trellis.
ansible-galaxy install -r requirements.yml

Part II: Install Bedrock

We're not going to do talk much about Bedrock, but the short version is this: Bedrock makes WordPress development on a team much less frustrating by:

Using Composer to manage all dependencies — including WordPress plugins and the core itself — to make it easier to work with version control and develop across teams
Easier environment-specific configuration
Better security through directory structure improvements and the use of better password encryption

Get a copy of Bedrock to make WordPress's file structure sane.

Just like we did with Trellis, we want to get a copy of Bedrock without the Git repository. So we do a shallow clone and then remove the .git folder.

# Move back into the project root.
cd ..

# Clone Bedrock to the `site` directory.
git clone --depth=1 git@github.com:roots/bedrock.git site

# Remove the `.git` file so we can have our own Git repo.
rm -rf site/.git

Part III: Configure a Development Site

With all the proper dependencies installed, we can start configuring our development site, which will run on our computer.

Add site details to `wordpress_sites.yml`.

Open trellis/group_vars/development/wordpress_sites.yml in your editor:

# Documentation: https://roots.io/trellis/docs/local-development-setup/
# `wordpress_sites` options: https://roots.io/trellis/docs/wordpress-sites
# Define accompanying passwords/secrets in group_vars/development/vault.yml

wordpress_sites:
  example.com:
    site_hosts:
      - canonical: example.dev
        redirects:
          - www.example.dev
    local_path: ../site # path targeting local Bedrock site directory (relative to Ansible root)
    admin_email: admin@example.dev
    multisite:
      enabled: false
    ssl:
      enabled: false
      provider: self-signed
    cache:
      enabled: false

If you're not familiar with YAML, it's a common way of describing data. It's indentation-based, so the default file creates a wordpress_sites object, and that contains an example.com object, which holds config properties (e.g. site_hosts).

Each site is identified by a key — in the example, the key is example.com — which allows us to link together our development, staging, and production environments without a bunch of duplicated configuration.

As a general rule, the production domain name is a good key to use.

With that in mind, let's set up our site by making the following changes to wordpress_sites.yml:

  wordpress_sites:
+   roots.code.lengstorf.com:
      site_hosts:
+       - canonical: roots.dev
-         redirects:
-           - www.example.dev
      local_path: ../site # path targeting local Bedrock site directory (relative to Ansible root)
+     admin_email: jason@lengstorf.com
      multisite:
        enabled: false
      ssl:
        enabled: false
        provider: self-signed
      cache:
        enabled: false

I'll be deploying the site to a production domain of roots.code.lengstorf.com, so that's my site key. For local development, we'll use roots.dev as the URL, and we don't need the redirects here, so we can remove them.

Finally, we updated the admin_email.

Save the changes and we're ready to move on.

Add credentials to `vault.yml`.

Next, we'll open trellis/group_vars/development/vault.yml:

# Documentation: https://roots.io/trellis/docs/vault/
vault_mysql_root_password: devpw

# Variables to accompany `group_vars/development/wordpress_sites.yml`
# Note: the site name (`example.com`) must match up with the site name in the above file.
vault_wordpress_sites:
  example.com:
    admin_password: admin
    env:
      db_password: example_dbpassword

Here we can see that the site key is example.com, so we'll need to update that.

We also need to update admin_password, which is the password we'll use to log into WordPress's dashboard.

And finally, we'll add strong passwords for the MySQL root user and the site's DB access.

Make the following changes in vault.yml:

  # Documentation: https://roots.io/trellis/docs/vault/
+ vault_mysql_root_password: "xy&G6o2kKH$#AFz247N."

  # Variables to accompany `group_vars/development/wordpress_sites.yml`
  # Note: the site name (`example.com`) must match up with the site name in the   above file.
  vault_wordpress_sites:
+   roots.code.lengstorf.com:
+     admin_password: "DM93zj,o29KjT/bh$8G$"
      env:
+       db_password: "qP42q2*?hjt.P+x7Bzc6"

Save the changes — now we're ready to fire up a local development box.

Part IV: Start a Local Instance of the WordPress Site Using Vagrant

Here's where the power of Trellis starts to become apparent.

Assuming the required software is installed, only four steps are required to get to this point:

Clone Trellis.
Clone Bedrock.
Update wordpress_sites.yaml
Update vault.yaml

When you compare that to the "normal" WordPress setup — clone WordPress, create a database, configure your local hosts file to give you a development URL, and so on — Trellis is far simpler. And we're not even to the really good stuff yet.

Start the development site.

To start the development site, it's one simple command:

vagrant up

The first time you run this, it'll take a 5–10 minutes. This is because Vagrant needs to download and configure all the pieces required to get the box up and running properly. After the first time, a lot of dependencies will be cached, which makes things much quicker for subsequent calls to vagrant up.

Check the development site on your local machine.

Once Vagrant is done, we can open the dev site by visiting http://roots.dev/ in our browser.

Log into the WordPress dashboard.

To log into the WordPress dashboard, head to http://roots.dev/wp/wp-admin/ in your browser and use the admin_password we set in vault.yml earlier.

Part V: Configure a Production Server

Wait, what? We're already deploying to production?

Yep. That's how easy Trellis is.

Create a Digital Ocean droplet to host the site.

It's hard to beat $5/month for hosting a website, so Digital Ocean will be our choice of production host.

To get started, create or log into your DigitalOcean account, then create a new droplet.

Choose Ubuntu 16.04.1 x64 for the distribution, $5/mo for the size, and choose any datacenter region.

Update `hosts/production`.

To tell Trellis where the production server lives, we need to add its IP address to trellis/hosts/production.

# Add each host to the [production] group and to a "type" group such as [web] or [db].
# List each machine only once per [group], even if it will host multiple sites.

[production]
your_server_hostname

[web]
your_server_hostname

Make the following edits:

  # Add each host to the [production] group and to a "type" group such as [web] or [db].
  # List each machine only once per [group], even if it will host multiple sites.

  [production]
+ 162.243.171.188

  [web]
+ 162.243.171.188

Enable free SSL and caching for production.

Setting up the production site configuration is nearly identical to the development site, except we need to add a few more settings for things like security. We can get these in place by editing trellis/group_vars/production/wordpress_sites.yml:

# Documentation: https://roots.io/trellis/docs/remote-server-setup/
# `wordpress_sites` options: https://roots.io/trellis/docs/wordpress-sites
# Define accompanying passwords/secrets in group_vars/production/vault.yml

wordpress_sites:
  example.com:
    site_hosts:
      - canonical: example.com
        redirects:
          - www.example.com
    local_path: ../site # path targeting local Bedrock site directory (relative to Ansible root)
    repo: git@github.com:example/example.com.git # replace with your Git repo URL
    repo_subtree_path: site # relative path to your Bedrock/WP directory in your repo
    branch: master
    multisite:
      enabled: false
    ssl:
      enabled: false
      provider: letsencrypt
    cache:
      enabled: false

Inside, make the following edits:

  # Documentation: https://roots.io/trellis/docs/remote-server-setup/
  # `wordpress_sites` options: https://roots.io/trellis/docs/wordpress-sites
  # Define accompanying passwords/secrets in group_vars/production/vault.yml

  wordpress_sites:
+   roots.code.lengstorf.com:
      site_hosts:
+       - canonical: roots.code.lengstorf.com
-         redirects:
-           - www.example.com
      local_path: ../site # path targeting local Bedrock site directory (relative to Ansible root)
+     repo: git@github.com:jlengstorf/roots.code.lengstorf.com.git # replace with your Git repo URL
      repo_subtree_path: site # relative path to your Bedrock/WP directory in your repo
      branch: master
      multisite:
        enabled: false
      ssl:
+       enabled: true
        provider: letsencrypt
      cache:
+       enabled: true

In addition to setting the site's URL, we also tell Trellis where the source code can be found with the repo setting, and enable SSL and caching for a faster, more secure site.

Add security settings to `vault.yml`.

Next, we need to add passwords and encryption keys for the site. We do this by editing trellis/group_vars/production/vault.yml:

# Documentation: https://roots.io/trellis/docs/vault/
vault_mysql_root_password: productionpw

# Documentation: https://roots.io/trellis/docs/security/
vault_users:
  - name: '{{ admin_user }}'
    password: example_password
    salt: 'generateme'

# Variables to accompany `group_vars/production/wordpress_sites.yml`
# Note: the site name (`example.com`) must match up with the site name in the above file.
vault_wordpress_sites:
  example.com:
    env:
      db_password: example_dbpassword
      # Generate your keys here: https://roots.io/salts.html
      auth_key: 'generateme'
      secure_auth_key: 'generateme'
      logged_in_key: 'generateme'
      nonce_key: 'generateme'
      auth_salt: 'generateme'
      secure_auth_salt: 'generateme'
      logged_in_salt: 'generateme'
      nonce_salt: 'generateme'

Make the following changes:

  # Documentation: https://roots.io/trellis/docs/vault/
+ vault_mysql_root_password: "z3Q6m3x8y?j@k&3+xuBq"

  # Documentation: https://roots.io/trellis/docs/security/
  vault_users:
    - name: "{{ admin_user }}"
+     password: "vH.827WQ2,t9?vyZyuB@"
+     salt: "jRMB764/EpB+,j(hvL98"

  # Variables to accompany `group_vars/production/wordpress_sites.yml`
  # Note: the site name (`example.com`) must match up with the site name in the above file.
  vault_wordpress_sites:
+   roots.code.lengstorf.com:
      env:
+       db_password: "#RLLE3)h9z9RDMT6d/4%"
        # Generate your keys here: https://roots.io/salts.html
+       auth_key: "&/qSrw23*@HeP2Kk#{^Ntx[!N>7#IdA=pCtI5gkpfgnn8{gDQ]2PQye]OkI.-p9f"
+       secure_auth_key: "LwX}3v}-P72LyH<o+kK&&M]^F3/#*&[um5$OiV@v:b!052Kaq%b]OQy=$@7F>=fF"
+       logged_in_key: "k+;dLHoBtR)5Y4VfyxMmm(fKp+Z<Uy]1PZvS_f#o`xGy7e=GNN1BEkd11s035t1:"
+       nonce_key: "!7#7ow=)d17Y[RlzSVA)_?GH<.e7-|SvD*&|;_5Y7)J@w]Dl,Q9_o!hUP8]G]n.k"
+       auth_salt: "G)0!0Z?MGKS?K,s$03=4e5Xu+[l:hw|X5Llr^H.e#[^Yd*m[)uOBYLh9/Zdwp{ir"
+       secure_auth_salt: "<tXa0,1PN;4}]VkkY|-[B$`AGi]KT{z5H:F/0EtFCBJ,KE/j%(5[.$pZ<1WP8y<I"
+       logged_in_salt: "lT*P7xsVeh=f}u9?b#F>4h8dY?]?>t{5cXby=jziz:1!o,gGO#z*lIw|[#y%,/SN"
+       nonce_salt: "BZqsYn4aC}?z@`HSi22n]z$qw>?2Y^$>M:PZ1eMHj*ucI)rnYi1jKld3):n/|1(5"

Note that you can automatically generate YAML-formatted encryption keys using the Roots salt generator.

Encrypt sensitive data with Ansible Vault.

Since it's always a bad idea to commit plain text credentials in a repo, we're going to use Ansible's built-in encryption to keep passwords and other sensitive data safe.

To do this, we create a password that will only be stored on our computer, which Ansible can use to decrypt the files. Trellis is already configured to ignore the password, so someone would need physical access to our computer to get the credentials.

Create a password for encrypting and decrypting files.

First, create a new file at trellis/.vault_pass:

# Make sure we’re in the `trellis/` directory
pwd
# Output => /Users/jlengstorf/dev/code.lengstorf.com/projects/learn-trellis/trellis

# Create a new file called `.vault_pass`
touch .vault_pass

Open trellis/.vault_pass for editing, then add a strong password:

Z+6Cm>TaofG=[379sED6

Save and close this file.

Update the Ansible config to use the vault password.

Open trellis/ansible.cfg for editing and add the highlighted line:

  [defaults]
  callback_plugins = ~/.ansible/plugins/callback_plugins/:/usr/share/ansible_plugins/callback_plugins:lib/trellis/plugins/callback
  stdout_callback = output
  filter_plugins = ~/.ansible/plugins/filter_plugins/:/usr/share/ansible_plugins/filter_plugins:lib/trellis/plugins/filter
  force_color = True
  force_handlers = True
  inventory = hosts
  nocows = 1
  roles_path = vendor/roles
  vars_plugins = ~/.ansible/plugins/vars_plugins/:/usr/share/ansible_plugins/vars_plugins:lib/trellis/plugins/vars
+ vault_password_file = .vault_pass

  [ssh_connection]
  ssh_args = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s

This tells Ansible where to look for the vault password, rather than prompting for it.

Use `ansible-vault` to encrypt the files.

To actually encrypt the files, run the following command:

ansible-vault encrypt group_vars/production/vault.yml

The output is a garbled mass of shit. This is a good thing — it means it's encrypted. Here's a snippet of what the encrypted file looks like:

$ANSIBLE_VAULT;1.1;AES256
62343434643862366430393366333661306366363937623561323637363033353366636134336230
3765633530336234393636306130346434333239636532650a336235356564303133303562626462
35363437383263653830313766646463646164303338626666366130396161383930373963613066
3366313862333134620a356563656432306335636331633063653163626638306532666335306239
...

Part VI: Provision the Production Server

Now that the proper configuration is in place, we need to let Trellis provision — or configure — the production server. Before we can do that, we need to update a few more configuration options, then make sure our source code is in a public repo and our domain name points to the production server.

Set up DNS so Let's Encrypt can run properly.

Since Let's Encrypt needs to verify the domain name in order to create an SSL certificate, we need to make sure the domain name points to the server before we try to provision it. Otherwise, we'll get an error during the SSL step.

In your DNS manager of choice (typically the site you bought the domain name through, e.g. Namecheap or GoDaddy), update the A record for your domain to point to your DigitalOcean droplet's IP address.

Use your GitHub keys for deployment.

To make sure cloning the repo on the production server goes smoothly, Ansible needs to know about your GitHub account's public keys.

Open trellis/group_vars/all/users.yml for editing, uncomment the GitHub URLs, and edit them to reflect your GitHub username:

# Documentation: https://roots.io/trellis/docs/ssh-keys/
admin_user: admin

  # Also define 'vault_users' (`group_vars/staging/vault.yml`, `group_vars/production/vault.yml`)
  users:
    - name: "{{ web_user }}"
      groups:
        - "{{ web_group }}"
      keys:
        - "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
+       # IMPORTANT: Add YOUR GitHub username here.
+       - https://github.com/jlengstorf.keys
    - name: "{{ admin_user }}"
      groups:
        - sudo
      keys:
        - "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
+       # IMPORTANT: Add YOUR GitHub username here.
+       - https://github.com/jlengstorf.keys

  web_user: web
  web_group: www-data
  web_sudoers:
    - "/usr/sbin/service php7.0-fpm *"

Disable root login for better security.

This step is optional, but unless you have a really good reason to keep the root user enabled, it's a good idea to disable it. The root user can wreak havoc on a server, so we can sleep better knowing that it's disabled and can't hurt anyone anymore.

Open trellis/group_vars/all/security.yml for editing and change the highlighted line:

  ferm_input_list:
    - type: dport_accept
      dport: [http, https]
      filename: nginx_accept
    - type: dport_accept
      dport: [ssh]
      saddr: "{{ ip_whitelist }}"
    - type: dport_limit
      dport: [ssh]
      seconds: 300
      hits: 20

  # Documentation: https://roots.io/trellis/docs/security/
  # If sshd_permit_root_login: false, admin_user must be in 'users' (`group_vars/all/users.yml`) with sudo group
  # and in 'vault_users' (`group_vars/staging/vault.yml`, `group_vars/production/vault.yml`)
+ sshd_permit_root_login: false
  sshd_password_authentication: false

Commit the site's code to GitHub.

For this tutorial, I created a repo called roots.code.lengstorf.com. You'll want to create your own for this step.

To get our site's code up to that repo, run the following commands:

# Move to the project root
cd ..
pwd
# Output => /Users/jlengstorf/dev/code.lengstorf.com/projects/learn-trellis

# Initialize a Git repository.
git init

# Add all the files to the repo.
git add -A

# Commit the files.
git commit -m 'Initial commit.'

# Add the GitHub repo as a remote repository.
# IMPORTANT: Replace this with YOUR GitHub repository!
git remote add origin git@github.com:jlengstorf/roots.code.lengstorf.com.git

# Push the `master` branch to GitHub and set it up for tracking.
git push --set-upstream origin master

Run the server provisioning script.

With the repo ready, the DNS configured, and some security measures in place, we're ready to provision the server.

Provisioning means getting the server ready: Ansible will run through a series of commands that download, install, and configure our server based on the information we've provided in the configuration files.

To make it happen, all we need to do is run the following:

# Move into the Trellis directory.
cd trellis/

# Provision the server using the `production` configuration options.
ansible-playbook server.yml -e env=production

This takes about 5–10 minutes. The output in the command line walks through everything that's being installed and configured, so if you're curious you can follow along and see what the Roots maintainers consider best practices for server configuration.

Or, we can walk away and let the robots do our bidding.

When it's all done, we'll see a "play recap" from Ansible that looks something like this:

PLAY RECAP **\*\***\*\*\*\***\*\***\*\***\*\***\*\*\*\***\*\***\***\*\***\*\*\*\***\*\***\*\***\*\***\*\*\*\***\*\***
162.243.171.188 : ok=130 changed=93 unreachable=0 failed=0
localhost : ok=0 changed=0 unreachable=0 failed=0

With 0 failures, we're ready to deploy!

Part VII: Deploy the Site to Production

Our last step — deploying our WordPress site to the DigitalOcean droplet — is really fucking easy — even compared to the rest of the steps in this walkthrough. We've already configured everything. We've already committed all our site's files to GitHub. We've already provisioned a server.

Now we just need Ansible to copy the site files and fire it up.

Enter the following to make it happen:

# Make sure we're in the `trellis/` directory.
pwd
# Output => /Users/jlengstorf/dev/code.lengstorf.com/projects/learn-trellis/trellis

# Deploy the `roots.code.lengstorf.com` site using `production` options.
# NOTE: Replace this site key with YOUR site key.
./bin/deploy.sh production roots.code.lengstorf.com

This takes a minute or two, and ends with a "play recap", just like provisioning. It should look something like this:

PLAY RECAP **\*\***\*\*\*\***\*\***\*\***\*\***\*\*\*\***\*\***\***\*\***\*\*\*\***\*\***\*\***\*\***\*\*\*\***\*\***
162.243.171.188 : ok=27 changed=12 unreachable=0 failed=0
localhost : ok=0 changed=0 unreachable=0 failed=0

Check the Live Site

And that's it. We've successfully deployed an SSL-secured (for free), painlessly-source-controlled WordPress site on a live domain name, all for about 20 minutes and $5/month.

BONUS: Synchronize Databases Using WP Sync DB

Trellis only manages files, so you'll notice that the new site doesn't match up with the development site. This isn't a big deal when you first start out, but as the production site starts to grow and the content builds up, it's a pain in the ass to pull a copy of the database so you can develop using real data.

Fortunately, the free plugin WP Sync DB makes synchronizing databases really painless. So let's install that and try it out.

Install WP Sync DB on the site.

Since we're using Bedrock, installing the plugin is as simple as telling Composer we want to use it.

# Make sure we’re in the site/ directory.
cd ../site
pwd
# Output => /Users/dev/code.lengstorf.com/projects/learn-trellis/site

# Use Composer to install WP Sync DB
composer require wp-sync-db/wp-sync-db:dev-master@dev

# Use Composer to install the media attachments plugin for WP Sync DB
composer require wp-sync-db/wp-sync-db-media-files:dev-master

Commit and redeploy to production.

After WP Sync DB is installed, we need to get it installed on the production server. This is a good example of how easy it is to deploy changes using Trellis: if you run git status, you'll see that composer.json and composer.lock have been updated; simply commit and push those files, then redeploy to production.

# Commit the changes and push them.
git commit -Am 'feat(wp-sync-db): installed WP Sync DB'
git push

# Deploy the changes to production.
cd ../trellis
./bin/deploy.sh production roots.code.lengstorf.com

Activate WP Sync DB on the production site and get the API key.

Once the deploy is complete, go to the production site's WordPress dashboard (e.g. https://roots.code.lengstorf.com/wp/wp-admin/), log in, then click "Plugins" in the left-hand menu.

Activate both the "WP Sync DB" and "WP Sync DB Media Files" plugins, then navigate to "Tools", then "Migrate DB" in the left-hand menu.

Click the "Settings" tab, then do the following:

Check the "Accept push requests..." option.
Copy the link in the "Connection Info" box.

Now the plugin is able to receive a database update from an external site (our development site, in this example).

Activate WP Sync DB on the development site and configure the sync.

On your development site, head to the "Plugins" page on your WordPress dashboard to activate the WP Sync DB and WP Sync DB Media Files plugins, then navigate to Tools > Migrate DB.

On the "Migrate" tab, update the settings as shown:

Choose the "Push" option.
Paste the production site's connection info into the text area that appears (what we copied in the previous step).
Under Advanced options:
- Uncheck the "Replace GUIDs" option.
- Check the "exclude spam comments" option.
[OPTIONAL] Check the "Backup the remote database before replacing it" option. This is probably a good idea if the production site has real data on it.
Check the "Media Files" option.
[OPTIONAL] Check the "Save Migration Profile" option and create a name so you can quickly push changes to production.

Once all these settings have been updated, click the "Migrate DB & Save" button.

Once this is done, you can reload the production site and see that the database from the development site is now live on production.

Bundle Stylesheets and Add LiveReload With Rollup

Thu, 25 Aug 2016 00:00:00 GMT

Watch on YouTube

In the first part of this series, we walked through the process of setting up Rollup as a front-end build tool for JavaScript.

This article covers parts two and three.

First, we’ll continue working on that project in Part II to add support for stylesheet processing through Rollup, using PostCSS to run some transforms and allow us to use syntactic sugar like simpler variable syntax and nested rules.

After that, we’ll wrap up with Part III, where we’ll add file watching and live reloading to the project so we don’t have to manually regenerate the bundle whenever files are changed.

Prerequisites

We’ll be continuing with the project we started last week, so if you haven’t gone through that part yet, it’s probably worth a look.

Series Navigation

Part II: How to Use Rollup.js for Your Next Project: PostCSS

Another part of Rollup that’s nice, depending on how your project is set up, is that you can easily process CSS and inject it into the head of the document.

On the plus side, this keeps all your build steps in one place, which keeps the complexity down in our development process — that’s a big help, especially if we’re working on a team.

But on the down side, we’re making our stylesheets rely on JavaScript, and creating a brief flicker of unstyled HTML before the styles are injected. So this approach may not make sense for some projects, and should be weighed against approaches like using PostCSS separately.

Since this article is about Rollup, though: fuck it. Let’s use Rollup!

Step 0: Load the stylesheet in `main.js`.

This is a little funky if you’ve never used a build tool before, but stick with me. To use our styles in the document, we’re not going to use a <link> tag like we normally would; instead, we’re going to use an import statement in main.min.js.

Right at the top of src/scripts/main.js, load the stylesheet:

+ // Import styles (automatically injected into <head>).
+ import '../styles/main.css';

  // Import a couple modules for testing.
  import { sayHelloTo } from './modules/mod1';
  import addArray from './modules/mod2';

  // Import a logger for easier debugging.
  import debug from 'debug';
  const log = debug('app:log');

  // The logger should only be disabled if we’re not in production.
  if (ENV !== 'production') {

    // Enable the logger.
    debug.enable('*');
    log('Logging is enabled!');
  } else {
    debug.disable();
  }

  // Run some functions from our imported modules.
  const result1 = sayHelloTo('Jason');
  const result2 = addArray([1, 2, 3, 4]);

  // Print the results on the page.
  const printTarget = document.getElementsByClassName('debug__output')[0];

  printTarget.innerText = `sayHelloTo('Jason') => ${result1}\n\n`;
  printTarget.innerText += `addArray([1, 2, 3, 4]) => ${result2}`;

Step 1: Install PostCSS as a Rollup plugin.

The first thing we need is Rollup’s PostCSS plugin, so install that with the following:

npm install --save-dev rollup-plugin-postcss

Step 2: Update `rollup.config.js`.

Next, let’s add the plugin to our rollup.config.js:

  // Rollup plugins
  import babel from 'rollup-plugin-babel';
  import eslint from 'rollup-plugin-eslint';
  import resolve from 'rollup-plugin-node-resolve';
  import commonjs from 'rollup-plugin-commonjs';
  import replace from 'rollup-plugin-replace';
  import uglify from 'rollup-plugin-uglify';
+ import postcss from 'rollup-plugin-postcss';

  export default {
    entry: 'src/scripts/main.js',
    dest: 'build/js/main.min.js',
    format: 'iife',
    sourceMap: 'inline',
    plugins: [
+     postcss({
+       extensions: [ '.css' ],
+     }),
      resolve({
        jsnext: true,
        main: true,
        browser: true,
      }),
      commonjs(),
      eslint({
        exclude: [
          'src/styles/**',
        ]
      }),
      babel({
        exclude: 'node_modules/**',
      }),
      replace({
        ENV: JSON.stringify(process.env.NODE_ENV || 'development'),
      }),
      (process.env.NODE_ENV === 'production' && uglify()),
    ],
  };

Take a look at the generated bundle.

Now that we’re able to process the stylesheet, we can regenerate the bundle and see how this all works.

Run ./node_modules/.bin/rollup -c, then look at the generated bundle at build/js/main.min.js, right near the top. You’ll see a new function called __$styleInject():

function __$styleInject(css) {
  css = css || '';
  var head = document.head || document.getElementsByTagName('head')[0];
  var style = document.createElement('style');
  style.type = 'text/css';
  if (style.styleSheet) {
    style.styleSheet.cssText = css;
  } else {
    style.appendChild(document.createTextNode(css));
  }
  head.appendChild(style);
}
__$styleInject('/* Styles omitted for brevity... */');

In a nutshell, this function creates a <style> element, sets the stylesheet as its content, and appends that to the document’s <head>.

Just below the function declaration, we can see that it’s called with the styles output by PostCSS. Pretty snazzy, right?

Except right now, those styles aren’t actually being processed; PostCSS is just passing our stylesheet straight across. So let’s add the PostCSS plugins we need to make our stylesheet work in our target browsers.

Step 3: Install the necessary PostCSS plugins.

I love PostCSS. I started out in the LESS camp, found myself more or less forced into the Sass camp when everyone abandoned LESS, and then was extremely happy to learn that PostCSS existed.

I like it because it gives me access to the parts of LESS and Sass that I liked — nesting, simple variables — and doesn’t open me up to the parts of LESS and Sass that I think were tempting and dangerous,[^dangerous] like logical operators.

[^dangerous]: I say "dangerous" because the logical features of LESS/Sass always felt a little flimsy to me, and in discussions with people they were always a sticking point. That was a red flag: using them introduced a kind of brittleness in a stylesheet, and while one person may be perfectly clear on what’s going on, the rest of the team may feel like mixins are voodoo pixie magic — and that’s never good for maintainability.

One of the things that I like most about it is the use of plugins, rather than an overarching language construct called "PostCSS". We can choose only the features we’ll actually use — and more importantly, we can leave out the features we don’t want used.

So in our project, we’ll only be using four plugins — two for syntactic sugar, one to support new CSS features in older browsers, and one to compress and minify the resulting stylesheet:

postcss-simple-vars — This allows the use of Sass-style variables (e.g. $myColor: #fff;, used as color: $myColor;) instead of the more verbose CSS syntax (e.g. :root { --myColor: #fff; }, used as color: var(--myColor);). This is purely preferential; I like the shorter syntax better.
postcss-nested — This allows rules to be nested. I actually don’t use this to nest rules; I use it as a shortcut for creating BEM-friendly selectors and grouping my blocks, elements, and modifiers into single CSS blocks.
postcss-cssnext — This is a bundle of plugins that enables the most current CSS syntax (according to the latest CSS specs), transpiling it to work, even in older browsers that don’t support the new features.
cssnano — This compresses and minifies the CSS output. This is to CSS what UglifyJS is to JavaScript.

To install these plugins, use this command:

npm install --save-dev postcss-simple-vars postcss-nested postcss-cssnext cssnano

Step 4: Update `rollup.config.js`.

Next, let’s include our PostCSS plugins in rollup.config.js by adding a plugins property to the postcss configuration object:

  // Rollup plugins
  import babel from 'rollup-plugin-babel';
  import eslint from 'rollup-plugin-eslint';
  import resolve from 'rollup-plugin-node-resolve';
  import commonjs from 'rollup-plugin-commonjs';
  import replace from 'rollup-plugin-replace';
  import uglify from 'rollup-plugin-uglify';
  import postcss from 'rollup-plugin-postcss';

+ // PostCSS plugins
+ import simplevars from 'postcss-simple-vars';
+ import nested from 'postcss-nested';
+ import cssnext from 'postcss-cssnext';
+ import cssnano from 'cssnano';

  export default {
    entry: 'src/scripts/main.js',
    dest: 'build/js/main.min.js',
    format: 'iife',
    sourceMap: 'inline',
    plugins: [
      postcss({
+       plugins: [
+         simplevars(),
+         nested(),
+         cssnext({ warnForDuplicates: false, }),
+         cssnano(),
+       ],
        extensions: [ '.css' ],
      }),
      resolve({
        jsnext: true,
        main: true,
        browser: true,
      }),
      commonjs(),
      eslint({
        exclude: [
          'src/styles/**',
        ]
      }),
      babel({
        exclude: 'node_modules/**',
      }),
      replace({
        ENV: JSON.stringify(process.env.NODE_ENV || 'development'),
      }),
      (process.env.NODE_ENV === 'production' && uglify()),
    ],
  };

Check the output in the `<head>`.

With the plugins installed, we can rebuild our bundle (./node_modules/.bin/rollup -c) and open build/index.html in our browser. We’ll see that the page is now styled, and if we inspect the document we can see the stylesheet was injected in the head, compressed and minified and with all the vendor prefixes and other goodies we expected from PostCSS:

Great! So now we have a pretty solid build process: our JavaScript is bundled, unused code is removed, and the output is compressed and minified, and our stylesheets are processed by PostCSS and injected into the head.

However, it’s still kind of a pain in the ass to have to manually rebuild the bundle every time we make a change. So in the next part, we’ll have Rollup watch our files for changes and reload the browser whenever a file is changed.

Part III: How to Use Rollup.js for Your Next Project: LiveReload

At this point, our project is successfully bundling JavaScript and stylesheets, but it’s still a manual process. And since every manual step in a process is a higher risk for failure than an automated step — and because it’s a pain in the ass to have to run ./node_modules/.bin/rollup -c every time we change a file — we want to make rebuilding the bundle automatic.

Step 0: Add a watch plugin to Rollup.

A common development tool when working with Node.js is a watcher. This may be familiar if you’ve worked with webpack, Grunt, Gulp, and other build tools in the past.

A watcher is a process that runs while you work on a project, and when you change files in any of the folders it’s monitoring, it triggers an action. In the case of build tools, the most common action is to trigger a rebuild.

In our project, we want to watch the src/ directory for any file changes, and when changes are detected we want to trigger a new bundle creation from Rollup.

To do that, we’ll use the rollup-watch plugin, which is a little different from the other Rollup plugins we’ve used so far — but more on that in a bit. Let’s start by installing the plugin:

npm install --save-dev rollup-watch

Step 1: Run Rollup with the `--watch` flag.

The difference between rollup-watch and other Rollup plugins is that we don’t have to make any changes to rollup.config.js in order to use it.

Instead, we add a flag to our terminal command:

# Run Rollup with the watch plugin enabled
./node_modules/.bin/rollup -c --watch

After running the command, we see a different output in the console:

$ ./node_modules/.bin/rollup -c --watch
checking rollup-watch version...
bundling...
bundled in 949ms. Watching for changes...

The process stays active, and it’s now watching for changes.

So if we make a small change to src/main.js — say adding a comment — as soon as we save it a new bundle is generated.

This is a big timesaver in development, but we can take it a step further. Right now we still have to refresh the browser to see the updated bundle — so let’s add a tool to refresh the browser automatically when our bundle is updated.

Step 2: Install LiveReload to refresh the browser automatically.

A common tool for speeding up development is LiveReload. This is a process that runs in the background and tells the browser to refresh whenever a file it’s watching is changed.

To start, we need to install the plugin:

npm install --save-dev livereload

Step 3: Inject the LiveReload script into the project.

Before LiveReload can work, we need to include a script in our page that talks to the LiveReload server.

Since this is only required in development, we’re going to take advantage of our environment variables to only inject the script if we’re not in production mode.

In src/main.js, add the following:

  // Import styles (automatically injected into <head>).
  import '../styles/main.css';

  // Import a couple modules for testing.
  import { sayHelloTo } from './modules/mod1';
  import addArray from './modules/mod2';

  // Import a logger for easier debugging.
  import debug from 'debug';
  const log = debug('app:log');

  // The logger should only be disabled if we’re not in production.
  if (ENV !== 'production') {

    // Enable the logger.
    debug.enable('*');
    log('Logging is enabled!');

+   // Enable LiveReload
+   document.write(
+     '<script src="http://' + (location.host || 'localhost').split(':')[0] +
+     ':35729/livereload.js?snipver=1"></' + 'script>'
+   );
  } else {
    debug.disable();
  }

  // Run some functions from our imported modules.
  const result1 = sayHelloTo('Jason');
  const result2 = addArray([1, 2, 3, 4]);

  // Print the results on the page.
  const printTarget = document.getElementsByClassName('debug__output')[0];

  printTarget.innerText = `sayHelloTo('Jason') => ${result1}\n\n`;
  printTarget.innerText += `addArray([1, 2, 3, 4]) => ${result2}`;

Save the file once you’ve made the changes, and now we’re ready to try it out.

Step 4: Run LiveReload.

With LiveReload installed and the script injected into our document, we can fire it up to watch our build/ directory:

./node_modules/.bin/livereload 'build/'

This results in output similar to the following:

$ ./node_modules/.bin/livereload 'build/'
Starting LiveReload v0.5.0 for /Users/jlengstorf/dev/code.lengstorf.com/projects/learn-rollup/build on port 35729.

And if we open our browser to view build/index.html — make sure to refresh the page after starting LiveReload to ensure the socket connection is active — we can see that making a change to build/index.html will automatically reload the browser and show us our changes:

This is great, but we’re still a little stuck: right now, we can only get Rollup’s watch function or LiveReload running unless we open multiple terminal sessions. That’s not ideal. In the next two steps, we’ll create a workaround for that.

Step 5: Use `package.json` scripts to simplify startup.

So far in this tutorial, we’ve had to type the full path to the rollup script, which — I’m sure you’ve noticed by now — is a pain in the ass.

Because of this, and because the tool we’ll be using to run watch and LiveReload at the same time, we’re going to add both the rollup command and the livereload command as scripts in package.json.

Open package.json — it’s in the root of the learn-rollup/ project directory. Inside, you should see the following:

{
  "name": "learn-rollup",
  "version": "0.0.0",
  "description": "This is an example project to accompany a tutorial on using Rollup.",
  "main": "build/js/main.min.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+ssh://git@github.com/jlengstorf/learn-rollup.git"
  },
  "author": "Jason Lengstorf <jason@lengstorf.com> (@jlengstorf)",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/jlengstorf/learn-rollup/issues"
  },
  "homepage": "https://github.com/jlengstorf/learn-rollup#readme",
  "devDependencies": {
    "babel-preset-es2015-rollup": "^1.2.0",
    "cssnano": "^3.7.4",
    "livereload": "^0.5.0",
    "npm-run-all": "^3.0.0",
    "postcss-cssnext": "^2.7.0",
    "postcss-nested": "^1.0.0",
    "postcss-simple-vars": "^3.0.0",
    "rollup": "^0.34.9",
    "rollup-plugin-babel": "^2.6.1",
    "rollup-plugin-commonjs": "^3.3.1",
    "rollup-plugin-eslint": "^2.0.2",
    "rollup-plugin-node-resolve": "^2.0.0",
    "rollup-plugin-postcss": "^0.1.1",
    "rollup-plugin-replace": "^1.1.1",
    "rollup-plugin-uglify": "^1.0.1",
    "rollup-watch": "^2.5.0"
  },
  "dependencies": {
    "debug": "^2.2.0"
  }
}

See that scripts property? We’re going to add two new ones:

A script to run our Rollup bundle generation command
A script to turn on LiveReload

Add the following to package.json:

  {
    "name": "learn-rollup",
    "version": "0.0.0",
    "description": "This is an example project to accompany a tutorial on using Rollup.",
    "main": "build/js/main.min.js",
    "scripts": {
+     "dev": "rollup -c --watch",
+     "reload": "livereload 'build/'",
      "test": "echo \"Error: no test specified\" && exit 1"
    },
    "repository": {
      "type": "git",
      "url": "git+ssh://git@github.com/jlengstorf/learn-rollup.git"
    },
    "author": "Jason Lengstorf <jason@lengstorf.com> (@jlengstorf)",
    "license": "ISC",
    "bugs": {
      "url": "https://github.com/jlengstorf/learn-rollup/issues"
    },
    "homepage": "https://github.com/jlengstorf/learn-rollup#readme",
    "devDependencies": {
      "babel-preset-es2015-rollup": "^1.2.0",
      "cssnano": "^3.7.4",
      "livereload": "^0.5.0",
      "npm-run-all": "^3.0.0",
      "postcss-cssnext": "^2.7.0",
      "postcss-nested": "^1.0.0",
      "postcss-simple-vars": "^3.0.0",
      "rollup": "^0.34.9",
      "rollup-plugin-babel": "^2.6.1",
      "rollup-plugin-commonjs": "^3.3.1",
      "rollup-plugin-eslint": "^2.0.2",
      "rollup-plugin-node-resolve": "^2.0.0",
      "rollup-plugin-postcss": "^0.1.1",
      "rollup-plugin-replace": "^1.1.1",
      "rollup-plugin-uglify": "^1.0.1",
      "rollup-watch": "^2.5.0"
    },
    "dependencies": {
      "debug": "^2.2.0"
    }
  }

These scripts allow us to use a shortcut for executing the script of our choice.

To run Rollup, we can now use npm run dev.

To run LiveReload, we can use npm run reload.

All that’s left now is to get them both running together.

Step 6: Install a tool to run the watcher and LiveReload in parallel.

In order to get both Rollup and LiveReload working at the same time, we’re going to use a utility called npm-run-all.

This is a powerful tool, so we won’t talk about everything it can do. What we’re using it for is its ability to run scripts in parallel — meaning both run at the same time inside the same terminal session.

Start by installing npm-run-all:

npm install --save-dev npm-run-all

Next, we need to add one more script to package.json that calls npm-run-all. In the scripts block, add the following (note that I’ve left out most of the file for brevity):

    "scripts": {
      "dev": "rollup -c --watch",
      "reload": "livereload 'build/' -d",
+     "watch": "npm-run-all --parallel reload dev",
      "test": "echo \"Error: no test specified\" && exit 1"
    }

Save your changes, then go to the terminal. We’re ready for the last step!

Step 7: Run the final `watch` script.

That’s it. We did it.

In your terminal, run the following:

npm run watch

Then reload your browser, make a change in the CSS or JS, and watch the browser refresh with the updated bundle — all like magic!

We’re now Rollup masters. Our code bundles will be smaller and faster, and our development process will be painless and quick.

How to Bundle JavaScript With Rollup — Step-by-Step Tutorial

Fri, 19 Aug 2016 00:00:00 GMT

Watch on YouTube

This week, we're going to build our first project using Rollup, which is a build tool for bundling JavaScript (and stylesheets, but we'll get to that next week).

By the end of this tutorial, we'll have Rollup configured to:

combine our scripts,
remove unused code,
transpile it to work with older browsers,
support the use of Node modules in the browser,
work with environment variables, and
compress and minify our code for the smallest possible file size.

Prerequisites

This will make more sense if you know at least a little bit of JavaScript.
Initial familiarity with ES2015 modules doesn't hurt, either.
You'll need npm installed on your machine. (Don't have it? Install Node.js here.)

Series Navigation

What Is Rollup?

In their own words:

Rollup is a next-generation JavaScript module bundler. Author your app or library using ES2015 modules, then efficiently bundle them up into a single file for use in browsers and Node.js.

It's similar to Browserify and webpack.

You could also call Rollup a build tool, which would put it in the company of things like Grunt and Gulp. However, it's important to note that while you can use Grunt and Gulp to handle tasks like creating JavaScript bundles, those tools would use something like Rollup, Browserify, or webpack under the hood.

Why should you care about Rollup?

What makes Rollup exciting, though, is its ability to keep files small. This gets pretty nerdy, so the tl;dr version is this: compared to the other tools for creating JavaScript bundles, Rollup will almost always create a smaller, faster bundle.

This happens because Rollup is based on ES2015 modules, which are more efficient than CommonJS modules, which are what webpack and Browserify use. It's also much easier for Rollup to remove unused code from modules using something called tree-shaking, which basically just means only the code we actually need is included in the final bundle.

Tree-shaking becomes really important when we're including third-party tools or frameworks that have dozens of functions and methods available. If we're only using one or two — think lodash or jQuery — there's a lot of wasted overhead in loading the rest of the library.

Browserify and webpack will end up including a lot of unused code right now. But Rollup doesn't — it'll only bring in what we're actually using.

And that's huge.

Part I: How to Use Rollup to Process and Bundle JavaScript Files

To show how effective Rollup is, let's walk through the process of building an extremely simple project that uses Rollup to bundle JavaScript.

Step 0: Create a project with JavaScript and CSS to be compiled.

In order to get started, we need to have some code to work with. For this tutorial, we'll be working with a small app, available on GitHub.

The folder structure looks like this:

learn-rollup/
├── build/
│ └── index.html
├── src/
│ ├── scripts/
│ │ ├── modules/
│ │ │ ├── mod1.js
│ │ │ └── mod2.js
│ │ └── main.js
│ └── styles/
│ └── main.css
└── package.json

You can install the app we'll be working with during this tutorial by running the following command into your terminal.

# Move to the folder where you keep your dev projects.
cd /path/to/your/projects

# Clone the starter branch of the app from GitHub.
git clone -b step-0 --single-branch https://github.com/jlengstorf/learn-rollup.git

# The files are downloaded to /path/to/your/projects/learn-rollup/

Step 1: Install Rollup and create a configuration file.

To get started, install Rollup with the following command:

npm install --save-dev rollup

Next, create a new file called rollup.config.js in the learn-rollup folder. Inside, add the following.

export default {
  entry: 'src/scripts/main.js',
  dest: 'build/js/main.min.js',
  format: 'iife',
  sourceMap: 'inline',
};

Let's talk about what each of these configuration options actually does:

entry — this is the file we want Rollup to process. In most apps, this would be the main JavaScript file, which initializes everything and actually starts the show.
dest — this is the location where the processed scripts should be saved.
format — Rollup supports several output formats. Since we're running in the browser, we want to use an immediately-invoked function expression (IIFE).

(This is a fairly complex concept to understand, so don't stress if it doesn't make total sense. In a nutshell, we want our code to be inside its own scope, which prevents conflicts with other scripts. An IIFE is a closure that contains our code in its own scope.)
sourceMap — it's extremely helpful for debugging to provide a sourcemap. This option adds a sourcemap inside the generated file, which keeps things simple.

Test the Rollup configuration.

Once we've created the config file, we can test that everything is working by running the following command in our terminal:

./node_modules/.bin/rollup -c

This will create a new folder called build in your project, with a js subfolder that contains our generated main.min.js file.

We can see that the bundle was created properly by opening build/index.html in our browser:

Look at the Bundled Output

What makes Rollup powerful is the fact that it uses "tree-shaking", which leaves out unused code in the modules we reference. For example, in src/scripts/modules/mod1.js, there's a function called sayGoodbyeTo() that isn't used in our app — and since it's never used, Rollup doesn't include it in our bundle:

(function () {
  'use strict';

  /**
   * Says hello.
   * @param  {String} name a name
   * @return {String}      a greeting for `name`
   */
  function sayHelloTo(name) {
    const toSay = `Hello, ${name}!`;

    return toSay;
  }

  /**
   * Adds all the values in an array.
   * @param  {Array} arr an array of numbers
   * @return {Number}    the sum of all the array values
   */
  const addArray = (arr) => {
    const result = arr.reduce((a, b) => a + b, 0);

    return result;
  };

  // Import a couple modules for testing.
  // Run some functions from our imported modules.
  const result1 = sayHelloTo('Jason');
  const result2 = addArray([1, 2, 3, 4]);

  // Print the results on the page.
  const printTarget = document.getElementsByClassName('debug__output')[0];

  printTarget.innerText = `sayHelloTo('Jason') => ${result1}\n\n`;
  printTarget.innerText += `addArray([1, 2, 3, 4]) => ${result2}`;
})();
//# sourceMappingURL=data:application/json;charset=utf-8;base64,...

In other build tools that's not always the case, and bundles can get really large if we include everything inside a bigger library like lodash just to reference one or two functions.

For example, using webpack, the sayGoodbyeTo() function is included, and the resulting bundle is more than double the size of what Rollup generates.

Step 2: Set up Babel so we can use new JavaScript features now.

At this point, we've got a code bundle that will work in modern browsers, but it'll break if the browser is even a couple versions old in some cases — that's not ideal.

Fortunately, Babel has us covered. This project transpiles new features of JavaScript (ES6/ES2015 and so on) into ES5, which will run on virtually any browser that's still used today.

If you've never used Babel, your life as a developer is about to change forever. Having access to the new features of JavaScript makes the language simpler, cleaner, and more pleasant in general.

So let's make that part of our Rollup process so we don't have to think about it.

Install the necessary modules.

First, we need to install the Babel Rollup plugin and the appropriate Babel preset.

# Install Rollup’s Babel plugin.
npm install --save-dev rollup-plugin-babel

# Install the Babel preset for transpiling ES2015.
npm install --save-dev babel-preset-es2015

# Install Babel’s external helpers for module support.
npm install --save-dev babel-plugin-external-helpers

Create a `.babelrc`.

Next, create a new file called .babelrc in your project's root directory (learn-rollup/). Inside, add the following JSON:

{
  "presets": [
    [
      "es2015",
      {
        "modules": false
      }
    ]
  ],
  "plugins": ["external-helpers"]
}

This tells Babel which preset it should use during transpiling.

Update `rollup.config.js`.

To make this actually do stuff, we need to update rollup.config.js.

Inside, we import the Babel plugin, then add it to a new configuration property called plugins, which will hold an array of plugins.

// Rollup plugins
import babel from 'rollup-plugin-babel';

export default {
  entry: 'src/scripts/main.js',
  dest: 'build/js/main.min.js',
  format: 'iife',
  sourceMap: 'inline',
  plugins: [
    babel({
      exclude: 'node_modules/**',
    }),
  ],
};

In order to avoid transpiling third-party scripts, we set an exclude config property to ignore the node_modules directory.

Check the bundle output.

With everything installed and configured, we can rebuild the bundle:

./node_modules/.bin/rollup -c

When we look at the output, it looks mostly the same. But there are a few key differences: for example, look at the addArray() function:

var addArray = function addArray(arr) {
  var result = arr.reduce(function (a, b) {
    return a + b;
  }, 0);

  return result;
};

See how Babel converted the fat-arrow function (arr.reduce((a, b) => a + b, 0))to a regular function?

That's transpiling in action: the result is the same, but the code is now supported back to IE9.

Step 3: Add ESLint to check for common JavaScript errors.

It's always a good idea to use a linter for your code, because it enforces consistent coding practices and helps catch tricky bugs like missing brackets or parentheses.

For this project, we'll be using ESLint.

Install the Modules.

In order to use ESLint, we'll want to install the ESLint Rollup plugin:

npm install --save-dev rollup-plugin-eslint

Generate a `.eslintrc.json`.

To make sure we only get errors we want, we need to configure ESLint first. Fortunately, we can automatically generate most of this configuration by running the following command:

$ ./node_modules/.bin/eslint --init
? How would you like to configure ESLint? Answer questions about your style
? Are you using ECMAScript 6 features? Yes
? Are you using ES6 modules? Yes
? Where will your code run? Browser
? Do you use CommonJS? No
? Do you use JSX? No
? What style of indentation do you use? Spaces
? What quotes do you use for strings? Single
? What line endings do you use? Unix
? Do you require semicolons? Yes
? What format do you want your config file to be in? JSON
Successfully created .eslintrc.json file in /Users/jlengstorf/dev/code.lengstorf.com/projects/learn-rollup

If you answer the questions as shown above, you'll get the following output in .eslintrc.json:

{
  "env": {
    "browser": true,
    "es6": true
  },
  "extends": "eslint:recommended",
  "parserOptions": {
    "sourceType": "module"
  },
  "rules": {
    "indent": ["error", 4],
    "linebreak-style": ["error", "unix"],
    "quotes": ["error", "single"],
    "semi": ["error", "always"]
  }
}

Tweak `.eslintrc.json`.

However, we need to make a couple adjustments to avoid errors for our project:

We're using 2 spaces instead of 4.
We will use a global variable called ENV later, so we need to whitelist that.

Make the following adjustments — the globals property and the adjustment to the indent property — to your .eslintrc.json:

{
  "env": {
    "browser": true,
    "es6": true
  },
  "globals": {
    "ENV": true
  },
  "extends": "eslint:recommended",
  "parserOptions": {
    "sourceType": "module"
  },
  "rules": {
    "indent": ["error", 2],
    "linebreak-style": ["error", "unix"],
    "quotes": ["error", "single"],
    "semi": ["error", "always"]
  }
}

Update `rollup.config.js`.

Next, import the ESLint plugin and add it to the Rollup configuration:

// Rollup plugins
import babel from 'rollup-plugin-babel';
import eslint from 'rollup-plugin-eslint';

export default {
  entry: 'src/scripts/main.js',
  dest: 'build/js/main.min.js',
  format: 'iife',
  sourceMap: 'inline',
  plugins: [
    eslint({
      exclude: ['src/styles/**'],
    }),
    babel({
      exclude: 'node_modules/**',
    }),
  ],
};

Check the console output.

At first, when we run ./node_modules/.bin/rollup -c, nothing seems to be happening. That's because as it stands, the app's code passes the linter without issues.

But if we introduce an issue — say removing a semicolon — we'll see how ESLint helps:

$ ./node_modules/.bin/rollup -c

/Users/jlengstorf/dev/code.lengstorf.com/projects/learn-rollup/src/scripts/main.js
12:64 error Missing semicolon semi

✖ 1 problem (1 error, 0 warnings)

Something that has the potential to introduce a mystery bug is now pointed out instantly, including the file, line, and column where the issue is happening.

While this won't eliminate all of our problems with debugging, it goes a long way toward squashing bugs that are due to obvious typos and oversights.

(As someone who has previously spent — ahem — numerous hours chasing bugs that ended up being something as silly as a misspelled variable name, it's hard to overstate the efficiency boost that working with a linter provides.)

Step 4: Add plugins to handle non-ES modules.

This is important if any of your dependencies use Node-style modules. Without it, you'll get errors about require.

Add a Node module as a dependency.

It would be easy to bang through this sample project without referencing a third-party module, but that's not going to cut it in real projects. So, in the interest of making our Rollup setup actually useful, let's make sure we can also reference third-party modules in our code.

For simplicity, we'll add a simple logger to our code using the debug package. Start by installing it:

npm install --save debug

Then, inside src/scripts/main.js, let's add some simple logging:

// Import a couple modules for testing.
import { sayHelloTo } from './modules/mod1';
import addArray from './modules/mod2';

// Import a logger for easier debugging.
import debug from 'debug';
const log = debug('app:log');

// Enable the logger.
debug.enable('*');
log('Logging is enabled!');

// Run some functions from our imported modules.
const result1 = sayHelloTo('Jason');
const result2 = addArray([1, 2, 3, 4]);

// Print the results on the page.
const printTarget = document.getElementsByClassName('debug__output')[0];

printTarget.innerText = `sayHelloTo('Jason') => ${result1}\n\n`;
printTarget.innerText += `addArray([1, 2, 3, 4]) => ${result2}`;

So far so good, but when we run rollup we get a warning:

$ ./node_modules/.bin/rollup -c
Treating 'debug' as external dependency
No name was provided for external module 'debug' in options.globals – guessing 'debug'

And if we check our index.html again, we can see that a ReferenceError was thrown for debug:

Well, shit. That didn't work at all.

This happens because Node modules use CommonJS, which isn't compatible with Rollup out of the box. To solve this, we need to add a couple plugins for handling Node dependencies and CommonJS modules.

Install the modules.

To work around this problem, we're going to add two plugins to Rollup:

rollup-plugin-node-resolve, which allows us to load third-party modules in node_modules.
rollup-plugin-commonjs, which coverts CommonJS modules to ES6, which stops them from breaking Rollup.

Install both plugins with the following command:

npm install --save-dev rollup-plugin-node-resolve rollup-plugin-commonjs

Update `rollup.config.js`.

Next, import and add the plugins to the Rollup config:

// Rollup plugins
import babel from 'rollup-plugin-babel';
import eslint from 'rollup-plugin-eslint';
import resolve from 'rollup-plugin-node-resolve';
import commonjs from 'rollup-plugin-commonjs';

export default {
  entry: 'src/scripts/main.js',
  dest: 'build/js/main.min.js',
  format: 'iife',
  sourceMap: 'inline',
  plugins: [
    resolve({
      jsnext: true,
      main: true,
      browser: true,
    }),
    commonjs(),
    eslint({
      exclude: ['src/styles/**'],
    }),
    babel({
      exclude: 'node_modules/**',
    }),
  ],
};

Check the console output.

Rebuild the bundle with ./node_modules/.bin/rollup -c, then check the browser again to see the output:

Step 5: Add a plugin to replace environment variables.

Environment variables add a lot of power to our development flow, and give us the ability to do things such as turning logging off and on, injecting dev-only scripts, and more.

So let's make sure Rollup will enable us to use them.

Add an `ENV`-based conditional in `main.js`.

Let's make use of an environment variable and only enable our logging script if we're not in production mode. In src/scripts/main.js, let's change the way our log() is initialized:

// Import a logger for easier debugging.
import debug from 'debug';
const log = debug('app:log');

// The logger should only be disabled if we’re not in production.
if (ENV !== 'production') {
  // Enable the logger.
  debug.enable('*');
  log('Logging is enabled!');
} else {
  debug.disable();
}

However, after we rebuild our bundle (./node_modules/.bin/rollup -c) and check the browser, we can see that this gives us a ReferenceError for ENV.

That shouldn't be surprising, though, because we haven't defined it anywhere. But if we try something like ENV=production ./node_modules/.bin/rollup -c, it still doesn't work. This is because setting an environment variable that way only makes it available to Rollup, not to the bundle created by Rollup.

We'll need to use a plugin to pass our environment variables into the bundle.

Install the modules.

Start by installing rollup-plugin-replace, which is essentially just a find-and-replace utility. It can do a lot of things, but for our purposes we'll have it simply find an occurrence of an environment variable and replace it with the actual value (e.g. all occurrences of ENV would be replaced with "production" in the bundle).

npm install --save-dev rollup-plugin-replace

Update `rollup.config.js`.

Inside rollup.config.js, let's import the plugin and add it to our list of plugins.

The configuration is pretty straightforward: we can just add a list of key: value pairs, where the key is the string to replace, and the value is what it should be replaced with.

// Rollup plugins
import babel from 'rollup-plugin-babel';
import eslint from 'rollup-plugin-eslint';
import resolve from 'rollup-plugin-node-resolve';
import commonjs from 'rollup-plugin-commonjs';
import replace from 'rollup-plugin-replace';

export default {
  entry: 'src/scripts/main.js',
  dest: 'build/js/main.min.js',
  format: 'iife',
  sourceMap: 'inline',
  plugins: [
    resolve({
      jsnext: true,
      main: true,
      browser: true,
    }),
    commonjs(),
    eslint({
      exclude: ['src/styles/**'],
    }),
    babel({
      exclude: 'node_modules/**',
    }),
    replace({
      exclude: 'node_modules/**',
      ENV: JSON.stringify(process.env.NODE_ENV || 'development'),
    }),
  ],
};

In our configuration, we're going to find every occurence of ENV and replace it with either the value of process.env.NODE_ENV — the conventional way of setting the environment in Node apps — or "development". We use JSON.stringify() to make sure the value is wrapped in double quotes, since ENV is not.

To make sure we don't cause issues with third-party code, we also set the exclude property to ignore our node_modules directory and all the packages it contains. (Thanks to @wesleycoder for the heads-up on this.)

Check the results.

To start, rebuild the bundle and check the browser. The console log should show up, just like before. That's good — that means our default value was applied.

To see where the power comes in, let's run the command in production mode:

NODE_ENV=production ./node_modules/.bin/rollup -c

When we reload the browser, there's nothing logged to the console:

Step 6: Add UglifyJS to compress and minify our generated script.

The last JavaScript step we'll go through in this tutorial is adding UglifyJS to minify and compress the bundle. This can hugely reduce the size of a bundle by removing comments, shortening variable names, and otherwise mangling the hell out of the code — which makes it more or less unreadable for humans, but much more efficient to deliver over a network.

Install the plugin.

We'll be using UglifyJS to compress the bundle, by way of rollup-plugin-uglify.

Install it with the following:

npm install --save-dev rollup-plugin-uglify

Update `rollup.config.js`.

Next, let's add Uglify to our Rollup config. However, for legibility during development, let's make uglification a production-only feature:

// Rollup plugins
import babel from 'rollup-plugin-babel';
import eslint from 'rollup-plugin-eslint';
import resolve from 'rollup-plugin-node-resolve';
import commonjs from 'rollup-plugin-commonjs';
import replace from 'rollup-plugin-replace';
import uglify from 'rollup-plugin-uglify';

export default {
  entry: 'src/scripts/main.js',
  dest: 'build/js/main.min.js',
  format: 'iife',
  sourceMap: 'inline',
  plugins: [
    resolve({
      jsnext: true,
      main: true,
      browser: true,
    }),
    commonjs(),
    eslint({
      exclude: ['src/styles/**'],
    }),
    babel({
      exclude: 'node_modules/**',
    }),
    replace({
      ENV: JSON.stringify(process.env.NODE_ENV || 'development'),
    }),
    process.env.NODE_ENV === 'production' && uglify(),
  ],
};

We're using something called short-circuit evaluation, which is a common (though debatably evil) shortcut for conditionally setting a value. (For example, it's pretty common to see this used to assign default values, like var value = maybeThisExists || 'default'.)

In our case, we're only loading uglify() if NODE_ENV is set to "production".

Check the minified bundle.

With the configuration saved, let's run Rollup with NODE_ENV in production:

NODE_ENV=production ./node_modules/.bin/rollup -c

The output isn't pretty, but it's much smaller. Here's a screenshot of what build/js/main.min.js looks like now:

Before, our bundle was ~42KB. After running it through UglifyJS, it's down to ~29KB — we just saved over 30% on file size with no additional effort.

Coming Up Next

In the next installment of this series, we'll look at handling stylesheets through Rollup using PostCSS, as well as adding live reloading so we can see our changes near-instantaneously as we make them.

Deploy a Node.js App to DigitalOcean with SSL

Tue, 09 Aug 2016 00:00:00 GMT

Watch on YouTube

In this post, we'll walk through the process, from start to finish, of creating a new server, deploying a Node.js app, securing it (for free!) with an SSL certificate, and pointing a domain name to it.

Watch the video above to see the whole process live — with clever commentary, of course — or jump to just the bits you need in the write-up below.

Prerequisites

A domain name that you can modify DNS records for.
A sense of adventure.

Set Up and Configure Your Server

Before we can do anything, we need a server that can be accessed publicly. There are lots of options out there, so don't feel locked into DigitalOcean — however, for this tutorial it'll probably be easiest to follow if you're using exactly the same setup.

Create a new droplet on DigitalOcean.

To start create an account on DigitalOcean, or log into your existing account.

Once you're logged in, click the "Create Droplet" button at the top of your screen.

Choose the $5/month option with Ubuntu 16.04.1 x64. Select a region closest to your users.

Finally, add your SSH key and

How to find your SSH key

First, open Terminal[^windows] and check for existing SSH keys:

[^windows]: If you're on Windows, check out Git Bash for a way to run these commands on your computer.

ls -la ~/.ssh

If you already have SSH keys set up, you should see a file called id_rsa.pub. (If there's a file ending in .pub, it's very likely an SSH key.)

To copy your SSH key, use one of the following commands:

# This copies the key so you can paste with command + V
pbcopy < ~/.ssh/id_rsa.pub

# This prints it in the command line for manual copying
cat ~/.ssh/id_rsa.pub

Add your SSH key to the droplet

Back on the DigitalOcean droplet creation screen, click the "New SSH Key" button and paste your SSH key into the field that opens.

Click "Add SSH Key" to save it, then make sure it's selected, name your droplet, and hit the big "Create" button to get your server online.

Your new droplet will display its IP address once it's set up. You can click on it to copy the IP to your clipboard.

Connect to the server using SSH

DigitalOcean droplets are created with a root user, and since we added our SSH keys, we can now log in without a password. Like magic!

# Make sure to replace the IP below with your server's IP address
ssh root@192.168.1.1

You will most likely be asked if you want to continue connecting the first time you log in. Type yes to continue, and you'll see something similar to the following:

$ ssh root@138.68.11.65
The authenticity of host '138.68.11.65 (138.68.11.65)' can't be established.
ECDSA key fingerprint is SHA256:f1qsLkumkNyRNfDVgjJk2R7kRlonuce1IMoEVTL2sfE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '138.68.11.65' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-31-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@nodejs-ssl-deploy:~#

Configure the server with basic security.

Once we're logged into the server, we need to get a few things configured to keep it secure.[^security]

[^security]: Keep in mind that this is not a security tutorial, and these are bare minimum security measures. For a production app, consult a security specialist.

Create an SSH user

First, we're going to add a new user with sudo privileges. To do this, run the following command while logged into your droplet:

# You can choose any username you want here.
adduser jason

This command prompts us for a password, and then for some additional, optional details.

Afterward, we can see that our user has been created by running id <your_username>, which should output something like the following:

root@nodejs-ssl-deploy:~# id jason
uid=1000(jason) gid=1000(jason) groups=1000(jason)

In order to run some of the commands on the server, such as restarting services, we need to add our new user to the sudo group. Do this by running the following command:

# Don't forget: use your own username here
usermod -aG sudo jason

Now if we run id jason we can see the sudo group has been applied.

root@nodejs-ssl-deploy:~# id jason
uid=1000(jason) gid=1000(jason) groups=1000(jason),27(sudo)

Add your SSH key for the new user

Next, we need to add our SSH key to the new user. This allows us to log in without a password, which is important because we're planning to disable password logins for this server.

# Become the new user
su - jason

# Create a new directory for SSH stuff
mkdir ~/.ssh

# Set the permissions to only allow this user into it
chmod 700 ~/.ssh

# Create a file for SSH keys
nano ~/.ssh/authorized_keys

The nano editor allows us to copy-paste your SSH key — the same one we copied to DigitalOcean when we created the droplet — into the new file, then press control + X to exit. Type Y to save the file, and press enter to confirm the file name.

We can make sure the SSH key is saved by running cat ~/.ssh/authorized_keys; if the SSH key is printed in the terminal, it's been saved.

# Set the permissions to only allow this user to access it
chmod 600 ~/.ssh/authorized_keys

# Stop acting as the new user and become root again
exit

Disable password login

Since every server has a default root account that's a target for automated server attacks — and because that account has unlimited power inside the server — it's a good idea to make sure no one can use it.

After the previous step, you should be logged into your server as root. Let's make sure the new account works and has sudo access:

# Log out of the server as root
exit

# Log into your server as the new user
ssh jason@138.68.11.65

Inside, we need to update the SSH configuration to disable password logins, and to disable logging in as root altogether.

To do this, use the following command to open the SSH configuration file for editing:

sudo nano /etc/ssh/sshd_config

Inside, you need to update two settings:

Find PermitRootLogin yes and change it to PermitRootLogin no
Find #PasswordAuthentication yes and change it to PasswordAuthentication no

Save the file by pressing control + X, then Y, then enter.

Finally, restart the SSH service with this command:

# Reloads the configuration we just changed
sudo systemctl reload sshd

Test your login by opening a new tab in Terminal (command + T on Mac) and logging into your server again.

If we log in as our new user, everything works as expected. However, if we try to log in as root, we get an error:

$ ssh root@138.68.11.65
Permission denied (publickey).

Set up a basic firewall

Next, we're going to configure a simple firewall. We're going to configure it to deny all traffic except through standard web traffic ports (80 for HTTP, and 443 for HTTPS), and to allow SSH logins.

This, in theory at least, should eliminate a lot of security risks on our server. (But again — this is not a security article; these are just basic precautions.)

We're going to run three commands to configure the firewall — called ufw — and then we'll enable it. Enter the following while logged into the server:

# Enable OpenSSH connections
sudo ufw allow OpenSSH

# Enable HTTP traffic
sudo ufw allow http

# Enable HTTPS traffic
sudo ufw allow https

# Turn the firewall on
sudo ufw enable

To check the status of the firewall, run sudo ufw status, which will give you the following:

jason@nodejs-ssl-deploy:~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
80                         ALLOW       Anywhere
443                        ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
80 (v6)                    ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

Get Your App Up and Running

Now that the server is set up, we can get our app installed.

Install Git.

In order to get a copy of our app to this server, we're going to use Git. Fortunately, Ubuntu makes it really easy to install common tools, so all we need to do is run this command:

sudo apt-get install git

We can validate that Git was installed properly by running git --version:

jason@nodejs-ssl-deploy:~$ git --version
git version 2.7.4

Set up Node.js.

Node.js is a little more complex than Git, because there are several different versions of Node that are used in production environments. Therefore, we need to update apt-get with the right version for our app before we install it.

Tell `apt-get` which Node.js version to download

The folks at NodeSource have made it really easy to install our desired Node version. For this tutorial, we'll be using the latest 6.x release.

Run the following commands to download and execute the setup script:

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -

This takes a few seconds to complete.

Install Node.js v6.x

With the NodeSource script complete, we can simply use apt-get to install Node.js:

sudo apt-get install nodejs

Once it's complete, we can verify that node is available by running node --version:

jason@nodejs-ssl-deploy:~$ node --version
v6.3.1

Clone the app

Now we can actually clone a copy of our app to the server — things are really getting exciting now.

It doesn't matter where you install the app, so let's create an apps dir in our user's home folder and clone the app into a folder named after our domain — this makes it really easy to remember which app is which.

# Make sure you’re in your home folder
cd ~

# Create the new directory and move into it
mkdir apps
cd apps/

# Clone your app into a new directory named for your domain
git clone https://github.com/jlengstorf/tutorial-deploy-nodejs-ssl-digitalocean-app.git app.example.com

Test the app

To make sure your app is installed and working, move into the new folder and start it:

# Move into the app directory
cd app.example.com

# Start the app
node app

The example app listens at http://localhost:5000, so we can test if it's working by opening a new Terminal session, logging into our server, and making a curl request to the app.

jason@nodejs-ssl-deploy:~$ curl http://localhost:5000/
<h1>I’m a Node app!</h1><p>And I’m <em>sooooo</em> secure.</p>

Awesome — we have a running app. Now we just need to make it accessible to the outside world.

We can exit from the test session (the one we just ran the curl command in), and we can stop the app in our other session using control + C.

Start Your App Using a Process Manager

Simply starting the app manually is technically enough to get the app deployed, but if the server restarts, that means we have to manually start the app again.

And in production apps, we want to eliminate as many — if not all — manual steps to get the app deployed. So we're going to use a process manager called PM2 to run our app. This also gives us benefits like easy-to-access logs, and a simple way to start, stop, and restart the app.

PM2 also allows us to start the app automatically when the server restarts, which means one less thing we need to worry about.

Install PM2

Unlike the other tools we've installed, pm2 is a Node package. We install it using the npm command, which is the default package manager for Node.js.

sudo npm install -g pm2

Start your app using PM2

With PM2 installed, we can now start the app like this:

# Make sure you're in the app directory
cd ~/apps/app.example.com

# Start the app with PM2
pm2 start app

Once the app is started, we see the status displayed:

And, conveniently, the app is running without locking up our session. In the same session we're able to run curl http://localhost:5000 to make sure it's running properly:

jason@nodejs-ssl-deploy:~/apps/app.example.com$ curl http://localhost:5000/

<h1>I’m a Node app!</h1><p>And I’m <em>sooooo</em> secure.</p>

Start your app automatically when the server restarts

We're almost done with getting the app running — just one more step.

The last thing to do is to make sure that when the server restarts, PM2 starts our app again.

This is a two-step process, which we kick off by running pm2 startup systemd:

jason@nodejs-ssl-deploy:~/app.example.com$ pm2 startup
[PM2] You have to run this command as root. Execute the following command:
sudo su -c "env PATH=$PATH:/usr/bin pm2 startup systemd -u jason --hp /home/jason"

PM2 prints out a command that we need to run using sudo. Copy-paste that to finish the process.

sudo su -c "env PATH=$PATH:/usr/bin pm2 startup systemd -u jason --hp /home/jason"

This will step through the process of updating the server to run PM2 on startup, and then you're all set.

Now we've got a running app — in the next section, we'll make the app securely accessible to the rest of the world!

Get a Free SSL Certificate With Let's Encrypt

SSL was a big hurdle for a long time for two reasons:

It was expensive.
It was hard.

Fortunately, some very smart, very kind-hearted people created Let's Encrypt, which is:

Free
Easy

So now there's really no excuse not to set up SSL for our domains.

Install Let's Encrypt

To start, we need to install some tools that Let's Encrypt depends on, then clone the letsencrypt repository to our server.

# Install tools that Let’s Encrypt requires
sudo apt-get install bc

# Clone the Let’s Encrypt repository to your server
sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Configure your domain to point to the server

Log into your DNS provider. I use CloudFlare; you'll want to log into whichever service you either bought your domain name through (Namecheap and GoDaddy, for example), or the service you use to manage your DNS (such as CloudFlare or DNSimple).

Add an A record for your domain that points to your droplet's IP address.

To check that the domain is pointing to your droplet, run the following (make sure to replace app.example.com with the domain you just configured):

dig +short app.example.com
# output should be your droplet’s IP address, e.g. 138.68.11.65

Generate the SSL certificate.

Now that the domain is pointed to our server, we can generate the SSL certificate:

# Move into the Let’s Encrypt directory
cd /opt/letsencrypt

# Create the SSL certificate
./certbot-auto certonly --standalone

The tool will run for a while to initialize itself, and then we'll be asked for an admin email address, to agree to the terms, and to specify our domain name or names. Once that's done, the certificate will be stored on the server for use with our app.

For now, that's all we need. We'll come back to these in a minute when we configure the domain.

Setup auto-renewal for the SSL certificate

For security, Let’s Encrypt certificates expire every 90 days, which seems pretty short. (By contrast, most paid SSL certificates are valid for at least a year.)

It turns out, though, that Let’s Encrypt has an one-step command to renew certificates:

/opt/letsencrypt/certbot-auto renew

This command checks if the certificate is near its expiration date and, when necessary, it generates an updated certificate that's good for another 90 days.

Now, it would be a huge pain in the ass if we had to manually log into the server and renew the certificate four times a year — and most likely we'd end up forgetting at least once — so we're going to use a built-in tool called cron to handle the renewal automatically.

To set this up, run the following command in the terminal to edit the server's cron jobs:

sudo crontab -e

We get an option for which editor to use here. Since nano is easier than the others, we'll stick with that.

When the editor opens, head to the bottom of the file and add the following two lines:

00 1 * * 1 /opt/letsencrypt/certbot-auto renew >> /var/log/letsencrypt-renewal.log
30 1 * * 1 /bin/systemctl reload nginx

The first line tells cron to run the renewal command, with the output logged so we can check on it when necessary, every Monday at 1 in the morning.

The second restarts NGINX — which we haven't set up yet, so don't worry — at 1:30 to make sure the new cert is being used.

Save and exit by pressing control + X, then Y, then enter.

That's it for the SSL cert. The last thing left is to make your app accessible by visiting our domain name in a browser.

Point Your Domain to the App

In order to make our app accessible, we need to send visitors to our domain to our app. To do this, we'll be using NGINX as a reverse proxy because it's faster and less painful than handling it through Node.js.

Install NGINX

Installing NGINX is no different from most of the other tools we've downloaded so far. Use apt-get to download and install it:

sudo apt-get install nginx

Make sure all traffic is secure

Next, we need to make sure that all traffic is served over SSL. To do this, we'll add a redirect for any non-SSL traffic to the SSL version. That way, if someone visits http://app.example.com, they'll be automatically redirected to https://app.example.com.

To do this, we need to edit NGINX's configuration files. Run the following command to open the file for editing:

sudo nano /etc/nginx/sites-enabled/default

Inside, delete everything and add the following:

# HTTP — redirect all traffic to HTTPS
server {
    listen 80;
    listen [::]:80 default_server ipv6only=on;
    return 301 https://$host$request_uri;
}

Save and exit by pressing control + X, then Y, then enter.

Create a secure Diffie-Hellman Group

It only takes a couple extra minutes to create a really secure SSL setup, so we might as well do it. One of the ways to do that is to use a strong Diffie-Hellman group, which helps ensure that our secure app stays secure.

Run the following command on your server:

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

This takes a minute or two — encryption should be hard for computers — and when it's done we can move on for now. We'll use this file in the next section.

Create a configuration file for SSL

Since I'm not a security expert, we're going to defer to an actual security expert for NGINX's SSL settings.

We need to create a new file on our server to hold these settings — if we add another domain to this server, we can reuse them this way — which we'll do with the following command:

sudo nano /etc/nginx/snippets/ssl-params.conf

Inside, we can copy-paste the following settings.

# See https://cipherli.st/ for details on this configuration
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
ssl_stapling on; # Requires nginx >= 1.3.7
ssl_stapling_verify on; # Requires nginx => 1.3.7
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;

# Add our strong Diffie-Hellman group
ssl_dhparam /etc/ssl/certs/dhparam.pem;

Save and exit by pressing control + X, then Y, then enter.

Configure your domain to use SSL

This is the last configuration step, I promise.

Now that we've got a certificate, a strong Diffie-Hellman group, and a secure SSL configuration, all that's left to do is actually set up the reverse proxy.

Open the site configuration again:

sudo nano /etc/nginx/sites-enabled/default

Inside, add the following below the block we added earlier:

# HTTPS — proxy all requests to the Node app
server {
    # Enable HTTP/2
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name app.example.com;

    # Use the Let’s Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    # Include the SSL configuration from cipherli.st
    include snippets/ssl-params.conf;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://localhost:5000/;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
    }
}

Save and exit by pressing control + X, then Y, then enter.

This configuration listens for connections to our domain on port 443 (the HTTPS port), uses the certificate we generated to secure the connection, and then proxies our app's output out to the browser.

Before we start the server, we should test the NGINX configuration with sudo nginx -t. If we didn't make any typos and everything looks good, we'll get the following:

jason@nodejs-ssl-deploy:~$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Enable NGINX

The very last step in this process is to start NGINX.

sudo systemctl start nginx

Test Your App

And now: the big moment. We can now visit our domain in a browser, and we'll see our app.

The configuration steps get pretty mind-numbing toward the end, but there's a huge payoff: we can now bask in the glory of a server that took about 30 minutes to set up, costs $5/month, and — as a bonus — gets an A+ for SSL security.

Not bad for 30 minutes' worth of setup, right?

Additional Resources

How to Convert HTML Form Field Values to a JSON Object

Sun, 07 Aug 2016 00:00:00 GMT

Getting form values as a JSON object can be a little confusing, but there’s good news! Browsers have implemented a built-in API for getting form values that makes this straightforward and approachable!

Use the FormData API to access form values in JavaScript

Before I learned about the FormData API, I thought accessing form values in JavaScript was a pain. But after Suz Hinton made me aware of it, that all changed.

In a nutshell, the FormData API lets us access any field value in a submitted form using a straightforward API.

For a quick example, let's assume we have this form:

<form>
  <label for="email">Email</label>
  <input type="email" name="email" id="email" />

  <button type="submit">Submit</button>
</form>

To handle submissions in JavaScript, we can use the FormData API like this:

function handleSubmit(event) {
  event.preventDefault();

  const data = new FormData(event.target);

  const value = data.get('email');

  console.log({ value });
}

const form = document.querySelector('form');
form.addEventListener('submit', handleSubmit);

If we run this code and submit the form, the value we enter into the email input will be logged. I don’t know about you, but the first time I tried this I wept happy tears — this is so much simpler than what I used to do! (My previous, more complicated approach is still at the bottom of this article if you want to compare.)

How to get all values from a form as a JSON object using the FormData API

If we want to get all of the values from a form, there’s an extra step. Let’s expand our form with a name field:

<form>
  <label for="name">Name</label>
  <input type="text" name="name" id="name" />

  <label for="email">Email</label>
  <input type="email" name="email" id="email" />

  <button type="submit">Submit</button>
</form>

To access all entries,

function handleSubmit(event) {
  event.preventDefault();

  const data = new FormData(event.target);

  // Do a bit of work to convert the entries to a plain JS object
  const value = Object.fromEntries(data.entries());

  console.log({ value });
}

const form = document.querySelector('form');
form.addEventListener('submit', handleSubmit);

The FormData API doesn’t directly convert form values to JSON, but we can get there by using the entries method and passing its return value to Object.fromEntries, which returns a plain JavaScript object.

This is compatible with JSON.stringify, so we can use it for sending JSON-encoded data to APIs or any other thing we might want to do with a JavaScript object.

Get multi-select values like checkboxes as JSON with the FormData API

The fromEntries approach in the previous section works great for most form inputs, but if the input allows multiple values — such as a checkbox — we'd only see one value in the resulting object.

Fortunately, the workaround for this only requires one more line of JavaScript for each multi-value input.

Let's add a field with multiple potential values to our form:

<form>
  <label for="name">Name</label>
  <input type="text" name="name" id="name" />

  <label for="email">Email</label>
  <input type="email" name="email" id="email" />

  <p>
    <input type="checkbox" name="topics" id="javascript" value="javascript" />
    <label for="javascript">JavaScript</label>

    <input type="checkbox" name="topics" id="html" value="html" />
    <label for="html">HTML</label>

    <input type="checkbox" name="topics" id="css" value="css" />
    <label for="css">CSS</label>
  </p>

  <button type="submit">Submit</button>
</form>

Getting all the topics requires using the .getAll() method:

function handleSubmit(event) {
  event.preventDefault();

  const data = new FormData(event.target);

  const value = Object.fromEntries(data.entries());

  value.topics = data.getAll("topics");

  console.log({ value });
}

const form = document.querySelector("form");
form.addEventListener("submit", handleSubmit);

Now the object contains an array in topics that contains all checked values!

A full example of multiple input types with the FormData API

For a full example of using the FormData API with lots of different input types, check out this CodePen (the form values get printed below the form on submission).

CodePen: [UPDATED] How to Get Form Field Data as JSON Using Plain JavaScript

Original Article Text

Using AJAX is really common, but it’s still tricky to get the values out of a form without using a library.

And that’s because it seems pretty intimidating to set up all the loops and checks required to deal with parsing a form and all its child elements. You get into heavy discussions of whether you should use for, for...in, for...of, or forEach, and after trying to keep up with the various performance, semantic, and stylistic reasons for making those choices, your brain starts to liquefy and drip out your ears — at which point it’s easy to just say, "Forget it; let’s just use jQuery."

But for simple sites that don’t need much beyond grabbing form data as an object to use with JSON, jQuery (or any big library or framework) includes a lot of overhead for only one or two functions that you’ll be using.

(Even if it’s not something we’d ever use in production, writing our own utility scripts is a fantastic way to increase our understanding of how things work. If we rely too much on a tool’s "magic" to make our apps work, it becomes really hard to debug them when we find a problem that falls outside of the tool’s scope.)

So in this walkthrough, we’ll be writing our own script — in plain JavaScript — to pull the values of a form’s fields into an object, which we could then use for AJAX, updating information on other parts of the page, and anything else you might want to do with a form’s data.

What We’ll Be Building

At the end of this walkthrough, we’ll have built the form shown in this pen:

CodePen: Finished code on Codepen.

If you fill the form and hit the “Send It!” button, the form data will be output as JSON in the “Form Data” section below.

Before We Get Started: Goals and Plans

To save ourselves a lot of headache and heartache, we’re going to start our project with an clear plan. This’ll keep our goals clear, and helps define the structure and purpose of the code before we ever write a line.

Start with a goal: what should we end up with?

Before we write any JavaScript, let’s start by deciding how we want the output to look.

If I’ve filled out the form above completely, we’d want the resulting object to look like this:

{
  "salutation": "Mr.",
  "name": "Jason Lengstorf",
  "email": "jason@lengstorf.com",
  "subject": "I have a general question.",
  "message": "Is this thing on?",
  "snacks": ["pizza"],
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"
}

Each field’s name attribute is used as the object’s key, and the field’s value is set as the object’s value.

This is ideal, because it means that we can do something like this:

// Find our form in the DOM using its class name.
const form = document.getElementByClassName('.contact-form')[0];

// Get the form data with our (yet to be defined) function.
const data = getFormDataAsJSON(form);

// Do something with the email address.
doSomething(data.email);

This is straightforward, easy to read as a human, and also easy to send to APIs that accept application/json data in requests (which is most of them these days).

So let’s shoot for that.

Make a plan: how can we convert form fields to JSON?

When we’re finished, our JavaScript should accomplish the following goals:

Capture the form’s submit event and prevent the default submission.
Convert the form’s child elements to JSON.
Check to make sure only form field elements are added to the object.
Add a safeguard to only store checkable fields if the checked attribute is set.
Handle inputs that allow multiple values, like checkboxes.

Ready to flex that big-ass brain of yours? Create a fork of the markup-and-styles-only pen, and let’s jump in and start writing some JavaScript.

Getting Started: Create a Form for Testing

To avoid the hassle of setting up front-end tooling (we’re using Babel to transpile the newer features of JavaScript, such as fat-arrow functions), we’re going to work through this project on Codepen.

To start, create a fork of this pen, which contains form markup with common inputs, and some styles to make it display nicely.

CodePen: Starting code on Codepen.

Step 1: Add a Listener to the `submit` Event for the Form

Before we do anything else, we need to listen for the submit event on our form, and prevent it from doing its usual thing.

To do this, let’s create a function called handleSubmit(), then use getElementsByClassName() to find our form, and attach the function to the form’s submit event.

Create a `handleSubmit()` function.

At the moment, this function isn’t going to do much. To start, we’ll prevent the default submit action, create a variable called data to store the output (which we’ll be building in a moment), then find our output container and print out the data variable as JSON.

In order to prevent the default action, this function needs to accept one argument: the event that’s created when the user clicks the submit button on the form. We can stop the form from submitting the usual way (which triggers the browser to go somewhere else) using event.preventDefault().

/**
 * A handler function to prevent default submission and run our custom script.
 * @param  {Event} event  the submit event triggered by the user
 * @return {void}
 */
const handleFormSubmit = (event) => {
  // Stop the form from submitting since we’re handling that with AJAX.
  event.preventDefault();

  // TODO: Call our function to get the form data.
  const data = {};

  // Demo only: print the form data onscreen as a formatted JSON object.
  const dataContainer = document.getElementsByClassName('results__display')[0];

  // Use `JSON.stringify()` to make the output valid, human-readable JSON.
  dataContainer.textContent = JSON.stringify(data, null, '  ');

  // ...this is where we’d actually do something with the form data...
};

Attach an event listener to the form.

With the event handler created, we need to add a listener to the form so we can actually handle the event.

To do this, we use getElementsByClassName() to target the form, then store the first item in the resulting collection as form.

Next, using addEventListener(), we hook handleSubmit() to the submit event, which will allow it to run whenever the user clicks to submit the form.

const handleFormSubmit = (event) => {
  /* omitted for brevity */
};

/*
 * This is where things actually get started. We find the form element using
 * its class name, then attach the `handleFormSubmit()` function to the
 * `submit` event.
 */
const form = document.getElementsByClassName('contact-form')[0];
form.addEventListener('submit', handleFormSubmit);

At this point we can test that things are working properly by clicking the "Send It!" button on the form. We should see {} in the "Form Data" output box.

Step 2: Extract the Values of Form Fields As JSON

Next up, we need to actually grab values from the form fields.

To do this, we’ll use something that — at first — might look scary as shit: reduce() combined with call().

We’ll dive into the dirty details of what reduce() is actually doing in the next section, but for now let’s focus on how we’re actually using it.

/**
 * Retrieves input data from a form and returns it as a JSON object.
 * @param  {HTMLFormControlsCollection} elements  the form elements
 * @return {Object}                               form data as an object literal
 */
const formToJSON = (elements) =>
  [].reduce.call(
    elements,
    (data, element) => {
      data[element.name] = element.value;
      return data;
    },
    {}
  );

const handleFormSubmit = (event) => {
  // Stop the form from submitting since we’re handling that with AJAX.
  event.preventDefault();

  // Call our function to get the form data.
  const data = formToJSON(form.elements);

  // Demo only: print the form data onscreen as a formatted JSON object.
  const dataContainer = document.getElementsByClassName('results__display')[0];

  // Use `JSON.stringify()` to make the output valid, human-readable JSON.
  dataContainer.textContent = JSON.stringify(data, null, '  ');

  // ...this is where we’d actually do something with the form data...
};

const form = document.getElementsByClassName('contact-form')[0];
form.addEventListener('submit', handleFormSubmit);

I know. I know. It looks hairy. But let’s dig in and see what this is doing.

First, let’s break this into its component parts:

We have a function called formToJSON(), which accepts one argument: form
Inside that function, we return the value of [].reduce.call(), which accepts three arguments: a form, a function, and an empty object literal ({})
The function argument accepts the arguments data and child, and adds a new property with the key of child.name and the value child.value, finally returning the data object

After we’ve added that code to our pen, we need to call the function from handleSubmit(). Find const data = {}; inside the function and replace it with const data = formToJSON(form.elements);.

Now we can run it by clicking the "Send It!" button will now output this:

{
  "salutation": "Ms.",
  "name": "",
  "email": "",
  "subject": "I have a problem.",
  "message": "",
  "snacks": "cake",
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7",
  "": ""
}

There are some issues here — for example, neither "Ms." nor "Cake" was actually selected on the form, and there’s an empty entry at the bottom (which is our button) — but this isn’t too bad for a first step.

So how did that just happen? Let’s go step by step to figure it out.

Step 2.1 — Understand how `reduce()` works.

The simplest explanation for reduce() is this:

The reduce() method uses a function to convert an array into a single value.

This method is part of the Array prototype, so it can be applied to any array value.

It takes two arguments:

A reducer function, which is required.
An initial value, which is optional (defaults to 0).

The reducer function is applied to each element of the array. This function accepts four arguments:

The value returned by the reducer function when it ran on the previous element (or the initial value, if this is the first element).
The current array element.
The current array index.
The whole array, in case the reducer needs a reference to it.

For our reducer, we only need the first two arguments.

A really simple example of reducing an array.

Let’s say we have an array of numbers, which represent sales for the day:

const sales = [100.12, 19.49, 10, 42.18, 99.62];

We need to determine total sales for the day, so we set up this simple function to add up sales:

function getTotalSales(previousTotal, currentSaleAmount) {
  return previousTotal + currentSaleAmount;
}

Then we use reduce() to apply the function to the array of sales:

const sales = [100.12, 19.49, 10, 42.18, 99.62];

function getTotalSales(previousTotal, currentSaleAmount) {
  return previousTotal + currentSaleAmount;
}

sales.reduce(getTotalSales);
// result: 271.41

Now, if we want to condense this code a little, we can actually write the whole thing like this:

const sales = [100.12, 19.49, 10, 42.18, 99.62];
sales.reduce((prev, curr) => prev + curr);
// result: 271.41

When this is called, reduce() starts with 0 as the value of prev, and takes the first element of the array, 100.12, as the value of curr. It adds those together and returns them.

Now reduce() moves to the second element in the array, 19.49, and this time the value of prev is the value returned last time: 100.12.

This process is repeated until all of the elements have been added together, and we end up with our total sales for the day: 271.41.

Step 2.2 — Deconstruct the function.

As it stands, formToJSON() is actually made of three parts:

A reducer function to combine our form elements into a single object.
An initial value of {} to hold our form data.
A call to reduce() using call(), which allows us to force reduce() to work with elements, even though it’s technically not an array.

Step 2.3 — Write the reducer function.

First up, we need to have our reducer function. In the simple example of reducing an array, we used single values, which won’t work in this case. Instead, we want to add each field to an object with a format like this:

{
    field_name: "field_value",
}

So our reducer function works like this:

// This is the function that is called on each element of the array.
const reducerFunction = (data, element) => {
  // Add the current field to the object.
  data[element.name] = element.value;

  // For the demo only: show each step in the reducer’s progress.
  console.log(JSON.stringify(data));

  return data;
};

The data object is the previous value of the reducer, and element is the current form element in the array. We then add a new property to the object using the element’s name property — this is the input’s name attribute in the HTML — and store its value there.

When we return data, we make the updated object available to the next call of the funciton, which allows us to add each field to the object, one by one.

Step 2.4 — Call the reducer.

To make it a little more obvious what’s happening in the formToJSON() function, here’s what it looks like if we break it up into more verbose code:

const formToJSON_deconstructed = (elements) => {
  // This is the function that is called on each element of the array.
  const reducerFunction = (data, element) => {
    // Add the current field to the object.
    data[element.name] = element.value;

    // For the demo only: show each step in the reducer’s progress.
    console.log(JSON.stringify(data));

    return data;
  };

  // This is used as the initial value of `data` in `reducerFunction()`.
  const reducerInitialValue = {};

  // To help visualize what happens, log the inital value.
  console.log('Initial `data` value:', JSON.stringify(reducerInitialValue));

  // Now we reduce by `call`-ing `Array.prototype.reduce()` on `elements`.
  const formData = [].reduce.call(
    elements,
    reducerFunction,
    reducerInitialValue
  );

  // The result is then returned for use elsewhere.
  return formData;
};

In the above example, we do exactly the same thing as in formToJSON(), but we’ve broken it down into its component parts.

We can see the output if we update handleSubmit() and change the call to formToJSON(form.elements) to formToJSON_deconstructed(form.elements). Check the console to see this output:

Initial `data` value: {}
{"salutation":"Mr."}
{"salutation":"Mrs."}
{"salutation":"Ms."}
{"salutation":"Ms.","name":""}
{"salutation":"Ms.","name":"","email":""}
{"salutation":"Ms.","name":"","email":"","subject":"I have a problem."}
{"salutation":"Ms.","name":"","email":"","subject":"I have a problem.","message":""}
{"salutation":"Ms.","name":"","email":"","subject":"I have a problem.","message":"","snacks":"pizza"}
{"salutation":"Ms.","name":"","email":"","subject":"I have a problem.","message":"","snacks":"cake"}
{"salutation":"Ms.","name":"","email":"","subject":"I have a problem.","message":"","snacks":"cake","secret":"1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"}
{"salutation":"Ms.","name":"","email":"","subject":"I have a problem.","message":"","snacks":"cake","secret":"1b3a9374-1a8e-434e-90ab-21aa7b9b80e7","":""}

We can see here that the reducer is called for every form element, and the object grows with each subsequent call until we’ve got an entry for every name value in the form.

Change handleSubmit() back to using formToJSON(form.elements), and let’s move on to cleaning up this output to only include fields it should include.

Step 3: Add a Check to Make Sure Only the Fields We Want Are Collected

The first problem we can see in the output is that fields with both empty name and empty value attributes have been added to the array. This isn’t what we want in this case, so we need to add a quick check to verify that fields have both a name and a value before we add them.

Step 3.1 — Create a function to check for valid elements.

First, let’s add a new function to our pen called isValidElement(). This function will accept one argument — the element — and return either true or false.

To return true, the element must have:

A non-empty name property.
A non-empty value property.

Implement this check like so:

/**
 * Checks that an element has a non-empty `name` and `value` property.
 * @param  {Element} element  the element to check
 * @return {Bool}             true if the element is an input, false if not
 */
const isValidElement = (element) => {
  return element.name && element.value;
};

Pretty simple, right?

This gives us a flag that lets us avoid unused elements (like the button) and unfilled fields (such as an empty Email field) from being added to the form data object.

Step 3.2 — Add the check to `formToJSON()`.

Next, we need to add an if check for whether or not our element is valid in formToJSON(). Since we don’t want to add anything if the element is not valid, we can simply do the following:

const formToJSON = (elements) =>
  [].reduce.call(
    elements,
    (data, element) => {
      // Make sure the element has the required properties.
      if (isValidElement(element)) {
        data[element.name] = element.value;
      }

      return data;
    },
    {}
  );

Now when we submit our form, the output is much cleaner:

{
  "salutation": "Ms.",
  "subject": "I have a problem.",
  "snacks": "cake",
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"
}

However, we’re still not there yet. In the next step, we’ll deal with checkable elements like radio inputs and checkboxes.

Step 4: Only Store Checkable Fields If a Field Is In `checked` State

Now we need another check to identify whether or not an element should be added to the array. For instance, right now the salutation field is being stored with the value Ms., even though that value is not selected in the form.

Obviously, this is bad news. So let’s fix it.

Step 4.1 — Create a function to check for checkable elements.

First, let’s add a new function to check whether or not an element’s value should be considered valid for inclusion in the object.

Our criteria for determining a "valid" element are:

The element is not a checkbox or radio input.
If the element is a checkbox or radio input, it has the checked attribute.

Add the following to create this check:

/**
 * Checks if an element’s value can be saved (e.g. not an unselected checkbox).
 * @param  {Element} element  the element to check
 * @return {Boolean}          true if the value should be added, false if not
 */
const isValidValue = (element) => {
  return !['checkbox', 'radio'].includes(element.type) || element.checked;
};

Step 4.2 — Add the check to `formToJSON()`.

Now we can add this check to formToJSON(), which is as simple as adding a second condition to our existing if check:

const formToJSON = (elements) =>
  [].reduce.call(
    elements,
    (data, element) => {
      // Make sure the element has the required properties and should be added.
      if (isValidElement(element) && isValidValue(element)) {
        data[element.name] = element.value;
      }

      return data;
    },
    {}
  );

Now we can run our code and see that the output is much cleaner:

{
  "subject": "I have a problem.",
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"
}

This is much better — now we only get elements that actually have a value set.

Step 5: If a Field Allows Multiple Values, Store Them In an Array

But we’re not quite done yet, because the form is still messing up the snacks field — which is clearly the most important field.

Try selecting both "Pizza" and "Cake" to see the problem:

{
  "subject": "I have a problem.",
  "snacks": "cake",
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"
}

Nope. This is a disaster. We need both pizza AND cake. So let’s make sure that can happen.

Step 5.1 — Create checks for elements that accept multiple values.

The check for whether or not multiple values are allowed has two parts, because there are two elements that allow multiple values.

First, we need to add a check for any checkboxes. This is simple enough: we just check if the type is checkbox:

/**
 * Checks if an input is a checkbox, because checkboxes allow multiple values.
 * @param  {Element} element  the element to check
 * @return {Boolean}          true if the element is a checkbox, false if not
 */
const isCheckbox = (element) => element.type === 'checkbox';

Second, we need to add a check for a select element with the multiple attribute.

This is a bit trickier, but still pretty straightforward. A select has a property called options, so we’ll check for that first. Next, we check for the multiple property. If both exist, our check will return true:

/**
 * Checks if an input is a `select` with the `multiple` attribute.
 * @param  {Element} element  the element to check
 * @return {Boolean}          true if the element is a multiselect, false if not
 */
const isMultiSelect = (element) => element.options && element.multiple;

Step 5.2 — Handle checkboxes in `formToJSON()`.

Inside formToJSON(), we need to add another if block for our isCheckbox() function.

If the current element is a checkbox, we need to store its value(s) in an array. Let’s take a look at the code first, and then we’ll talk about how it works.

const formToJSON = (elements) =>
  [].reduce.call(
    elements,
    (data, element) => {
      // Make sure the element has the required properties and should be added.
      if (isValidElement(element) && isValidValue(element)) {
        /*
         * Some fields allow for more than one value, so we need to check if this
         * is one of those fields and, if so, store the values as an array.
         */
        if (isCheckbox(element)) {
          data[element.name] = (data[element.name] || []).concat(element.value);
        } else {
          data[element.name] = element.value;
        }
      }

      return data;
    },
    {}
  );

Since we need to get the element’s values into an array, we use a bit of shorthand in (data[element.name] || []), which means, "use the existing array, or a new, empty one".

Then we use concat() to add the current value to the array.

Now if we check the options for both pizza and cake, we see the following:

{
  "subject": "I have a problem.",
  "snacks": ["pizza", "cake"],
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"
}

Much better. Crisis averted, everyone!

Step 5.3 — Write a function to retrieve values from multi-selects.

Our very last step before we can call this sucker done is to add a check for select fields that support multiple selected options. I’m not a big fan of this input type, because I think it’s a confusing input for people to use — one that’s easily replaced with checkboxes — but for the sake of covering bases we’ll plug it in.

The selected options from a multi-select are stored in their own array-like object, called an HTMLOptionsCollection, so we need to run reduce() on this as well.

Let’s keep things clean by moving this out into its own function.

This function will accept each options, check if the selected property is true, then add its value to an array called values, which will ultimately be returned containing the values of all selected options.

Add the following to the pen:

/**
 * Retrieves the selected options from a multi-select as an array.
 * @param  {HTMLOptionsCollection} options  the options for the select
 * @return {Array}                          an array of selected option values
 */
const getSelectValues = (options) =>
  [].reduce.call(
    options,
    (values, option) => {
      return option.selected ? values.concat(option.value) : values;
    },
    []
  );

Step 5.4 — Handle multi-select values in `formToJSON()`.

To put a bow on all this, we need to add an else if block in our formToJSON() function.

After the isCheckbox() check, we’ll add a isMultiSelect() check. If that returns true, we’ll add the select’s values to the object as an array using getSelectValues().

Make the following updates to the pen:

const formToJSON = (elements) =>
  [].reduce.call(
    elements,
    (data, element) => {
      // Make sure the element has the required properties and should be added.
      if (isValidElement(element) && isValidValue(element)) {
        /*
         * Some fields allow for more than one value, so we need to check if this
         * is one of those fields and, if so, store the values as an array.
         */
        if (isCheckbox(element)) {
          data[element.name] = (data[element.name] || []).concat(element.value);
        } else if (isMultiSelect(element)) {
          data[element.name] = getSelectValues(element);
        } else {
          data[element.name] = element.value;
        }
      }

      return data;
    },
    {}
  );

Run a quick test of multi-select values.

Since our current form doesn't have a select with the multiple attribute, so let's quickly add that to the subject field in our pen.

Look for the select in the HTML pane and add the multiple attribute like so:

      select#subject.contact-form__input.contact-form__input--select(
        name="subject"
        multiple
      )

Now we can actually test. Click on both options and submit the form. The out put will now be:

{
  "subject": ["I have a problem.", "I have a general question."],
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"
}

After we’ve tested, we can remove the multiple attribute from the select input.

The Final Result: Form Field Values Are Collected in an Object for Use as JSON

At this point, we’ve built a small script that will extract the values from a form as an object literal, which can easily be converted to JSON using JSON.stringify().

We can test by filling out our form with dummy data and submitting it.

Use your own fork of the pen, or enter dummy data in the form below:

CodePen: Work in progress on Codepen.

After submitting, we’ll see the info we entered, and it’ll look something like the following:

{
  "salutation": "Ms.",
  "name": "Pac-Man",
  "email": "mspacman@example.com",
  "subject": "I have a problem.",
  "message": "These ghosts keep chasing me!",
  "snacks": ["pizza", "cake"],
  "secret": "1b3a9374-1a8e-434e-90ab-21aa7b9b80e7"
}

CodeTV Blog RSS Feed

Web Dev Challenge Hackathon S2.E11: Twilio Hotline Challenge

Press 1 for an awesome AI-Powered Hotline

The Tool: ConversationRelay

Make new connections and get expert guidance

Multiple Ways to Win!

Watch the episode for inspiration

The rules and how to submit

Bonus: you can also win for a great idea!

Web Dev Challenge Hackathon S2.E9: Breakfast Apps

Build an app that’s about breakfast

The Tool: Hashbrown

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

The rules and how to submit

CSS overrides without important using layers in Astro components

Minimal reproduction of CSS override challenges is Astro

Why standard overrides don't work in Astro components

CSS overrides in Astro the old way: !important

CSS overrides in Astro the new way: CSS layers

Modern CSS overrides, no hacks

Resources

Web Dev Challenge Hackathon S2.E7: Touch Grass

Build an app that connects us to the physical world

The Tool: Apollo Connectors

Make new connections and get expert guidance

Win a ticket to GraphQL Summit!

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon S2.E5: Give in to your worst developer urges

Give in to your worst developer urges and automate something unnecessary

The Sponsor: Intuit

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon S2.E3: Create a devious web-based video player

Build the most unhinged, Byzantine, devious, unusual video player UX you can imagine.

Introduction to Mux

Big prizes for this hackathon!

Make new connections and get expert guidance

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon S2.E2: Build a game played on at least 2 devices

The prompt: Build a game that’s played on at least 2 devices.

Introduction to Temporal + devs discuss their ideas

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon S2.E1: Build a Custom API + App

The prompt: Build a custom API — and an app that consumes it — that makes something in your life a little more convenient.

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon 10: Workshop Woes

The prompt: The holidays are upon us and Santa’s elves are behind schedule! Build an app to get the holiday back on track.

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon 9: Blvck Spades

The prompt: Come up with ways to connect people’s behavior to e-commerce incentives

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon 7: Spooky Apps

The prompt: build an app that’s haunted. Make it spooky, make it creepy — let’s have a little Halloween fun.

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

The rules and how to submit

Web Dev Challenge Hackathon 6: Capture Memories

The prompt: build an app that helps people capture memories.

Make new connections and get expert guidance

The first 5 devs to submit a qualifying app get their choice of gear

Watch the episode for inspiration

CSS overrides in Astro the old way: `!important`