tk.gg

Lime's billing model is encouraging cyclists to run red lights

Wed, 24 Dec 2025 19:00:00 GMT

import { LimeBikeSimulator } from './ssr-safe.js'; import './style.css'; import route from './blackfriars-to-oxford.json'

You're a cyclist at an intersection in London waiting for a red light. You're on a dockless hire bike. Every second you wait, you're paying for being conscientious.

Every junction you stop at, another cyclist chooses to run the light.

This isn't a bug. It's encoded into the design of the pricing. And it creates a dangerous incentive.

Try it yourself. The simulator below shows a route derived from Lime's own data of the most popular locations: Blackfriars Bridge to Oxford Street. Watch how much time is possibly spent waiting at lights, and how much that could cost.

Lime charges by the minute. That's a reasonable business model for a service where time correlates with value, but actually we're focused on distance when thinking about a journey. Time spent waiting at a red light isn't valuable to you. It's time spent obeying traffic law. And unlike the same situation in a taxi, the legal consequences for skipping a red light are inconsequential most of the time, and self directed. It is far more now a social contract than it is perceived as a legal obligation.

Here's the problem: with this model, running a red light saves money. The longer the light, the more you save. You can feel it every time you approach a junction and the light turns amber. Do you brake and pay to wait? Or do you push harder?

Putting a monetary incentive for skipping that light is dangerous for pedestrians, for other cyclists, and for the riders themselves. And the danger multiplies with the weight of a 30kg electric bike underneath you.

Lime's billing model has turned riding safely into a premium feature you pay an extra 10-30% on your ride for.

A trivial fix?

Lime bikes have GPS. They track your speed in real-time. They know when you're moving and when you're not. The data exists. You can even download it from your account.

Pausing billing when a rider's speed drops below a threshold for more than a few seconds would be straightforward. No new hardware. No new data. Just a different calculation in the billing logic. It could even work as a refund if the calculation requires journey and bike telemetry data.

If Lime wanted to make this only work at lights, this demo shows how it's possible with open data to do that. If Lime wanted to make sure it wasn't abused, they could set a proportionate limit of 'free' stoppage time per trip, or change their model entirely to focus on a combination of distance and time.

The choice to charge for stop time isn't a technical limitation. It's a product decision that prioritises revenue over rider safety.

In a quote to The Times, Hal Stevenson, director of policy at Lime, said that the “pricing model had a very low impact on whether people did or didn’t stop at red lights. We don’t subscribe to the idea that people are making these decisions about their safety, for the basis of 10 or 20p.”

But this came from research that Lime commissioned themselves in July. They launched a 'Respect the Red' campaign to 'encourage riders to stop at red lights'. But surely a crucial part of that campaign would be looking at the behavioural effect of how they charge for time.

Hackney found a workaround

Hackney Council has partially addressed this. From October 2025, e-bike rides in the borough were capped at £1.75 for 30 minutes — the same price as a bus fare. The meter still runs, but the financial pressure disappears. You pay the same whether you sprint through junctions or wait patiently at every light, provided your journey is less than 30 minutes total.

It's a pragmatic fix that removes the incentive without requiring Lime to change their billing logic (they just sell 30-minute bundles for a cheaper price). But it only works within Hackney's boundaries, and it required council intervention to negotiate. The rest of London is still paying by the minute.[^1]

Try building your own commute and see how much Lime's tax on safety might cost you.

<details class='mb-8 rounded bg-slate-400/10 p-4'> <summary>About this simulation</summary> This is a simplified model. Traffic lights are placed using OpenStreetMap data, which includes signals on main roads even where separate cycleways exist. Light timings are randomised and cycle predictably rather than responding to real-world factors like traffic density or time of day. The actual stop-time cost on your commute will vary. </details>

[^1]: All of the above also applies to Forest and Voi, who also charge per minute in most of London.

Your newsroom should be using AI for this today

Wed, 08 Oct 2025 12:00:00 GMT

import { Image } from 'astro:assets' import AIComponent from './AIComponent'

import nyt_hp_img from './nyt_hp.png' import bbc_hp_img from './bbc_hp.png' import guardian_hp_img from './guardian_hp.png' import spark_alt_text_img from './spark_alt_text.png'

The easiest way to show that your newsroom is actually thinking about AI isn't the 50th summarisation tool, chatbot article, or headline generator. It's in automating the jobs people aren't good at and don't really want to do in the first place.

Alt text is a perfect example.

Most images on your site need it, hardly anyone writes it consistently, and most editors would rather not spend their day doing it. But with AI and LLMs, you can fix that in minutes.

Try the demo below. It'll generate alt text for any image you upload. Then we'll dive into why it works, what makes good alt text for journalism, and what we learned building this at the Financial Times.

Try it out below

What and why

Alt text is the short line of text a screen reader uses to describe an image to someone who can't see it. It's one of those things that every website should consider, but few newsrooms really give consistent attention to. Most alt text is either empty, badly duplicated, or overly descriptive, and that inconsistency means people relying on screen readers often have a worse experience on your site.

Take a look at this example from the BBC and the New York Times. Each of their homepages marked up with this accessibility tool shows how alt text appears for images.

<div

</div>

In this example, The New York Times has videos for both of its leading stories, which have no text representation at all. However, the small headlines image for the daily podcast has a detailed description of the Supreme Court. Columnist headshots have their names associated with them. And images of iPhone homescreens have an alt text describing just that. Towards the bottom of the page, two images are marked as 'decorative', which we'll get on to later.

By contrast, the BBC's page is scattered with images, all of which have text. We have a variety of lengths, from "Dame Jilly Cooper laughing" or "Lecornu and Macron" to a detailed description of a composite image around the US-hosted World Cup. Charlotte Church's image describes her outfit, the event and the location, whereas the image to the left of her simply reads "The Strictly contestants".

Imagine you couldn't see those pages and text was all you had to go on. Would the information the images are adding be sufficient? Would it actually be too much? Think about the hierarchy: would hearing the alt text first before the headline be a help or a hindrance to getting to where you need to go?

Alt text is really hard to get right. There are a lot of guides online, but they generally end up focused on the same few principles, and they typically focus on 'functional' images, or images that are there to convey information. But in reality, most images on a news site are there to add context, to break up text, or to add some visual interest. They aren't always strictly necessary to understand the content of the article, nor are they necessarily 'adding' anything to the story that is not in the text already.

A homepage is a great example of this contrast: the images are accompanied by headlines and text that convey the information that you need to take the next action: clicking through to the story, where you will more than likely see the image again. In this context the image is typically decorative. And by definition decorative images don't need alt text.

Some images however are crucial to comprehension. In articles, the image often holds a lot of weight. Photos of historic moments, unbelievable damage after a natural disaster, or the emotion captured in a person's face are all adding something important to the story. The term photojournalism exists for a reason, and in many cases striking photography is as much a part of the experience as the writing itself.

How we built it at the FT

When we built this at the Financial Times, we made it part of the CMS's native image component. When an editor uploads an image, they can click a button to generate alt text using the AI model. The generated text is then populated into the alt text field, where the editor can review and edit it as needed before saving.

We went further by encouraging the editor to check a box confirming they had reviewed the text before saving. If they didn't, the text would stay highlighted in orange, drawing attention from other editors until it was approved.

Our prompt took some work to get right. It took a series of tests with different models and prompt iterations to reach a point where the text was consistently good enough to publish with minimal edits. We also limited the tool to images that came with photo agency captions and metadata, as we found that passing this information through aided in identification significantly.

<details class='mb-8 rounded bg-slate-400/10 p-4'> <summary>Example prompt used above</summary> You are an AI alt text generator. For each image, write a concise, objective description that conveys the essential visual information to someone who cannot see it. Focus on key elements, context, and purpose. Avoid subjective judgments, assumptions, or detail not critical to comprehension. Be clear, accurate, and specific. Only name people in the photo if you are certain of their name and they are a famous individual. Otherwise gently describe people without making judgements about protected characteristics. If the image is a chart, then discuss the axes, important values, and the general trend. Do not discuss visual information that would not make sense to someone who cannot see the chart, such as the color of the trend line or the color of the bars. Always return your response in <alt> tags only, without any additional commentary or punctuation. You MUST keep your response to two sentences, or around 150 characters, unless in exceptional circumstances such as a chart where it is necessary to go longer. For charts, you may respond with up to four sentences, and your character count is unlimited. </details>

The output can be a little generic, and that's fine. The goal isn't to replace editors but to help them start from something consistent. Adding a human in the loop ensures the final alt text fits your newsroom's tone and context.

A few things we learned

Decide whether the image is decorative. If it doesn't add essential meaning, use an empty alt="".
Describe what's essential, not everything. Focus on who or what's in the image and why it matters to the story. If the person is notable, name them. Describe what they are wearing or doing only if it's relevant.
Keep it short. Usually under 150 characters, though for complex images and charts you can afford to flex this a bit.
Don't duplicate captions. Screen readers will read both. You can use aria-hidden on captions in this case. [^1]
Punctuate like a sentence. Skip "Image of..." intros unless it is a "Screenshot of" or "Cartoon of" where the medium is important.

It's a simple workflow, but it adds up fast: accessibility audit scores improved, editors saved time, and, most importantly, we served readers who use assistive tech much better.

And it's the kind of thing any media organisation can do this quarter. You don't need a data science team or a complex pipeline, just a text-generation API call, some sensible defaults, and someone to review the results. Let me know if you give it a go.

Aside

I'm certain that in the future this will be something that is provided by local AI models anyway. Google's Gemini Nano is getting good enough at this kind of thing, and small enough, that integration with Chrome could enable this for images on any website you visit. However! This model may not have all the information your organisation has available to it when adding this image, so if you want to provide the best experience possible, keeping control over what you are serving is still valuable.

</div>

[^1]: Full disclosure: the FT still has some challenges in delivering this to the frontend, but I have faith in my excellent former-colleagues who are working on it.

What mistakes are newsrooms making with push notifications?

Sun, 10 Aug 2025 22:00:00 GMT

On one of those summer Fridays at the end of June, you might have had time to glance through the Reuters Institute Digital News Report. This is an annual treat for those looking to map our changing business environment and the challenges facing the news industry.

I appreciated seeing push notifications in this year’s report. Not least of all because a tool I’ve been working on for a couple of years now to track them, Project Push, was cited.

The report’s main author, Nic Newman, has written a special report: Walking the notification tightrope: how to engage audiences while avoiding overload. It looks into an audience of obsessives — the people who have left their news notifications on — and questions the value of them to this audience. It also discusses how news orgs can use this channel to best effect.

To dive deeper into this topic, I chatted with Angelica Hill, Technical Product Manager on The New York Times’ Messaging team. No one else is deeper in the detail than she is on this, and we had a great discussion about some of the report’s findings. Here’s some of how she thinks about notifications after seven years working in this area, including what makes push work and what doesn’t.

The challenges of working in notifications

One of the challenges of Project Push has been that I can capture the notification content, but I can’t see how well a notification performed. Angelica has the benefit of being able to see behind the scenes, but even at the NYT, she still doesn’t have all the data she wants. Publishers are typically working off of what the platforms provide, and while you can send 1,000 notifications to 1,000 devices, you’ll only know if someone saw it if they tap through.

Tracking attribution is therefore paramount. Knowing that a visit came from a notification — and knowing which one — can help your teams understand which kinds of notifications work for each audience. Understanding recirculation rates is important too: A challenge for Angelica’s team is how to use notifications in the most effective way to drive engagement and impact for the user and the company.

Do you encourage readers to engage more with the app as a whole, not just the story or nudge that was pushed to them, and if so, how? Or do you prioritize engagement with the push and its content only? Not to mention the challenges of measuring the halo effect of push notifications, i.e., how much a user’s engagement with push impacts their engagement and behavior across other areas of your product portfolio.

One advantage the NYT has, of course, is its breadth of product: Games, Cooking, Wirecutter, and The Athletic are all on hand to maintain engagement once a visitor taps through.

“In any strategy, I’m trying to push people into the app, getting them to engage with that piece of content deeply, then trying to get them to engage across our other products.” - Angelica Hill

Testing and optimization

As with newsletters, testing and optimizing your push notifications can eke out some wins. But the nature of the product, and the controls you have, come with downsides. You could A/B test a regular notification or a personalized one, but A/B testing a single breaking news alert won’t necessarily teach you much.

Your test might also be successful on a metric you so easily can’t measure: satisfaction. If a reworded notification tells the recipient enough about the story that they don’t need to visit, you’ve still done your job in delivering that information to them. In a subscription environment, that might be enough.

Pushes are also subject to external factors, and user behavior can be seasonal. Being conscious of the news agenda and what else might be affecting user behavior is important to keep in mind when talking to editors about data.

“You might see great engagement on the same push notification from month to month, but then the next month is going to go down. And your data won’t tell you why, but if you look out into the world, you’ll realize why.” - Angelica Hill

Offering control

Spurring users to turn off notifications for your app entirely, rather than more selectively unsubscribing, can destroy your strategy. Angelica suggests making sure that your user preferences are easily reachable, and ideally having a variety of channels for the user to opt out of individually.

The New York Times has a lot of these, and Angelica has noticed how they can actually be an encouragement for annoyed users to opt into other channels. A user looking to turn off their Politics notifications, for instance, might serendipitously opt into Cooking or Wellness updates.

“We want to keep our push-eligible user pool as big as possible while giving as much granularity within the app as possible, to give them that flexibility, that curation, but also to stop them from opting out.” - Angelica Hill

The email overlap

Prior to working on Project Push, I tried doing a similar thing for email newsletters. The two methods of reaching audiences have some interesting parallels. On the lock screen, or via an inbox, they skirt the on-platform/off-platform divide in levels of customization and ownership.

One query that Angelica and I are both interested in is how readers prefer to engage. If you can guide newsletter readers to an app, you can gather more data. But do you lose readers who like the features of an inbox? Angelica has discovered readers who repeatedly return throughout the day from a single newsletter, something that’s impossible to do from a push notification.

There’s so much opportunity

Push is such an interesting platform, and new stuff is always arriving. In the news publishing space, where opportunities for innovation can feel scarce, push offers ways for publishers to try something new, with options such as Apple’s Live Activities or Android’s Live Updates notifications, audio autoplay, and video notifications.

These short formats also tend to be a productive area of collaboration with editors, who may be more open to experimenting with things your brand might otherwise be a bit snooty over, like emojis. Building relationships with editors across teams is a good bonding experience, dealing with the challenges of the technology and format.

“I think there’s just so much [value] in having like the persistent tabs on a mobile device. Outside of push notifications, just in general, the mobile home-page space and having that direct access to put your message right in front of a user is really exciting and interesting.” - Angelica Hill

Some practical takeaways

Push is a fertile ground for experimentation and data collection. Try to prioritize attribution and watch recirculation.

Test notifications thoughtfully, but encourage editorial teams to be creative and try new formats and content.
Offer granular notification settings to give your users effective opt-outs and opportunities for discovery.
Consider a strategic overlap with other messaging products like newsletters to understand engagement better.

Push notifications are far from just the latest news alerts. Done well, they can provide a multitude of trigger points to encourage good user habits and engagement with your newsroom. I hope this manages to inspire some thoughts and experimentation.

Wondering what other newsrooms do with push? You can sign up for a beta of Project Push today (just hit the Login button), and find an archive of 180,000 notifications. If you have any thoughts or comments, as always, I’m on Slack or Bluesky.

[!NOTE] You can sign up to the Product Notes newsletter here.

How tiny teams can have an outsized impact with AI

Fri, 06 Jun 2025 00:00:00 GMT

Like many of you, I’ve been paying closer attention to how large language models (LLMs) are impacting our work. They’re a transformational technology, and we’re only starting to understand what that scale of change might look like in our work. In news product, most experiments so far have focused on summaries, search, and conversational experiences. A good all-encompassing example is Time’s Person of the Year feature, which combines summaries, chat, and audio into one experience.

These experiments are a good start. They give us a chance to collect usage data and learn what readers actually want from AI; and on our own platforms, rather than through third-party remixes that increasingly strip out attribution and clicks. But I think the biggest shift, at least for now, is happening internally. AI gives product and editorial teams a low-risk way to extend their capabilities and move faster. As larger publishers grapple with lawsuits, brand risks, and internal inertia, smaller newsrooms have a real chance to move first. There’s more upside, and less to lose. When I sat down to prepare this newsletter, I wondered how Product Notes had changed over time. With almost 40 issues published, that’s a lot of content to analyse. So I opened Google’s AI Studio, where their latest Gemini models are freely available, and asked it to write a Python script that could scrape the NPA website and pull out the newsletter content. From there, I fed it back into the model to surface themes and trends.

According to the analysis, AI is the fourth most frequent topic—behind newsletters, but ahead of technical tooling. Seems I’m in good company.

Doing this job reminded me of the XKCD comic “Is it Worth the Time?”, which charts how much time you should spend automating a task based on how often you do it. That comic just turned twelve, and while the principles have held up well, I wonder if the numbers are increasingly shifting to the low-end. If it used to take five hours to build something and now you can do it in five minutes with the help of AI, the maths changes. Technological advances have made automating things like scraping the newsletter archive feasible within 30 minutes.

Of course, it’s not just about scripting or scraping or writing SQL. The more interesting question is how AI can help product teams build, test, and iterate faster. Recently, I’ve been experimenting with tools like Vercel’s V0, Lovable, Stackblitz Bolt, and Val Town’s Townie. These are prompt-driven interfaces for building front ends. So far these tools are mostly marginal gains: good for green field ideas, basic internal tooling and early prototyping, but you’re not likely to be able to use them to ship something much more useful. It’s still early days, but the pace of improvement is fast and the competition is intense.

At the FT, we’ve been exploring how AI could assist in the editing process—flagging errors or inconsistencies before a story is published. To prototype some ideas, I used v0 and Lovable to generate sample interfaces and see how they might plug into our own text editor infrastructure. It’s been a good way to test possibilities and get clearer on implementation paths.

Better results though have come from ideas I’ve got from colleagues that I’ve been able to quickly prototype in code using other methods. More modern, development-tuned LLMs with large context windows like OpenAI’s GPT-4.1 have made ‘vibe coding’ something really enjoyable, rapid, and productive. In just a few hours I managed to put together an app that transcribed live German radio into English and made it searchable via a web interface.

One of the advantages of using AI for code is that it’s easy to verify: does the output run? Does it do what you asked? While there are risks—especially if you’re building without understanding the underlying logic—for prototyping and lightweight tooling, it’s remarkably effective. A half hour of back-and-forth with an LLM can take you further than you might expect.

Product managers are often expected to be generalists, working across domains without always having deep technical skills. But that balance is shifting. These tools extend what a generalist can accomplish, letting you bring more ideas to life without waiting for engineering time or even full specs. That, I think, is where the most immediate opportunity lies.

[!NOTE] You can sign up to the Product Notes newsletter here.

Tracking newsletter read time with AMP

Tue, 19 Nov 2024 21:00:00 GMT

import { Image } from 'astro:assets' import { BlueskyPost, YouTube } from 'astro-embed'

A year or so ago, I noticed the Washington Post had started to send their breaking news email notifications with AMP.

AMP, or Accelerated Mobile Pages, is a Google web framework designed to create lightweight experiences that — Google being Google — you could still chuck a lot of ads into.

Nowadays, as a web framework, it is mostly dead, but it lives on through email, where a few companies use it to send what Google calls 'Dynamic email'. These emails have the superpower of being able to load in new data, or dynamically update their content. This is how Google Docs sends you comment notifications that update with replies, and it's how the Washington Post gives you a time interval since the event happened in their breaking news emails.

<figure> <Image src='https://assets.tk.gg/chrome_hEe5fFgbQ1.png' width={679} height={263} alt="A screenshot of a Washington Post Breaking News alert with the time '2 hours ago' displayed" /> <figcaption> We're talking about the red bar that says '2 hours ago'. Believe it or not, that's impossible in a regular email. </figcaption> </figure>

This led me to wonder: could AMP be used to enable better read-time tracking in email newsletters, even though AMP support is limited primarily to Gmail and Yahoo Mail?

So what if you could leverage AMP to do something that everyone wants to do in newsletters but can't: better analytics.

Read time

To fully understand the magnitude of this opportunity, we first need to examine the current status quo for read time tracking in email.

Read time on the web is usually calculated with some javascript that runs on the page to detect when various actions occur, like scrolling, and summing up all the time spent by the user between entering the page and their last action. This doesn't work in email because famously no javascript is supported. No dynamic scripting at all is supported, and everything has to be a combination of HTML and (usually inline) CSS.

Read receipts (which have become less reliable since Apple's Mail Privacy Policy and Google's image proxy) instead use a small unique image in each email, and a remote server records a request for that image as an indication that the email has been opened.

The best read time tracking you can get without javascript works similarly. A small unique image, but this time the remote server deliberately throttles data out to a very slow rate. It's pretty error prone, and doesn't work with things like image proxying. Any numbers you get back will be wildly inaccurate.

Enter AMP

AMP allows you to include some very specific kinds of interactivity in email. They have a list of components that are supported including cool things like dynamic lists, and autocomplete fields. This is how you can respond to a comment in your Google Doc from your Gmail app.

One of those elements is a carousel for images. Common to the web, carousels progressively slide or fade through images in a finite list. One of the magic things about the carousel in AMP is that the images are loaded progressively.

With this in mind, an experiment occurred to me. Could you create a carousel of X number of images, loading every Y seconds, and get up to X * Y duration tracking for your email? If you bucketed read times into something like ten seconds you would suddenly get a decent chunk of very valuable information as to whether your Substack essays are actually landing with readers as they spend a minute reading or if people are nope-ing out immediately.

So I did end up prototyping this and it worked! Here's a short video going through the feature:

Yes the first and last slide load at the same time as you load, but then, every five seconds on the dot a new request, the next slide, with a unique identifier.

You can test this too by running the code, putting yourself an AMP email together, and sending yourself a test AMP email.

How to use this

There's a few things you'd need to do to enable this: firstly you'd need to get your sender approved to send AMP emails. It turns out that Google doesn't allow just anyone to do this, and you need to send examples of the kinds of emails you're going to send to them first to get your sending domains allowlisted.

Secondly you'll need to actually build the tracking software yourself and deploy it at scale. My prototype is not going to be anywhere near enough, and if you're going to do that you may as well at the same time take over the responsibility of open rate tracking from your <span title="email send provider">ESP</span>.

Each user and each send to that user is going to need a unique hash to identify that send and that user, and you'll need to append this to the request for each of the images you include in your carousel. The rest can stay functionally the same as the prototype, but that's quite a big ask.

Anyway, I don't suspect I'll actually convince anyone to do this further than the test that I've put together. It's quite a lot of work. But I would love to hear about anyone with plans to give it a go. Please do reach out.

You can view the prototype proof-of-concept code on my GitHub here.

Another real name commenting policy?

Tue, 20 Dec 2022 00:00:00 GMT

Well, this is weird. The Times have today decided to implement a real-name policy for commenting on their website.

They claim this will increase the quality of debate, something we know there is limited evidence for in the first place. Let's go through their FAQ...

This is such a weird decision because there is no real anonymity on The Times website in the first place. It's a subscription community and the customer team will always have access to your billing details.

And they admit to this in the FAQ.

"Why doesn't The Times block accounts instead?" "Why don't you moderate comments instead?"

Both of those things already happen. There is no "animosity protected by anonymity" because you can ban the user by their billing details.

Anonymity has a mixed effect in academic research on the quality of comments and amount of abuse. It's a well debated topic that I won't get into because of this point: pseudonymity != anonymity, so The Times already isn't an anonymous community.

The only thing this policy changes is whether people's real names are tied publicly rather than privately to their comments on news stories.

It is exposing people to others in the community and online.

They address this too, but skirt around it entirely. Certain people will not be able to abide by this policy without a degree of pseudonymity, and there is no allowance for that.

This will bar those people from the community. Teachers, doctors, social workers, academics.

So what are the alternatives to achieve the same effect?

Well, moderation for one, and a strong set of community guidelines, both things The Times already claims to have. Why are these not working? That's not addressed.

The cost of moderating a community that is pay to play vs one that is effectively open access is much less already.

Surely it's net positive at the cost of The Times? If new accounts are needed with new payment details that's a £30 charge to post.

There are so many other ways you could experiment with this: like putting a delay on posting for new accounts, or putting them in a new moderation queue.

You could lock pseudonyms from being changed, or reduce the number of articles you can comment on, or comments you can post.

Interestingly this also has pretty major implications for the "real" identifies of marginalised groups that The Times has editorially taken a line against, like the trans community, who might want to use a name other than their legal name. That's also not allowed.

It's going to be an interesting (if ideologically flawed) example of a pseudonymous community changing to an identifiable community, and the impact of that on the civility, volume and diversity of that community should be measured.

I really hope there are plans to record this.

Lastly, and the reason this feels like a very lazy brush stroke more than anything else, is that this still allows hateful people to take that hate and now apply it in a more dangerous way. Against people posting under their real names, anonymously, without participation.

A link to that FAQ to see for yourself. I'd love to be pointed to the latest in this by some experts in the field.

LFT notifications in your browser

Sun, 23 Jan 2022 00:00:00 GMT

import { Tweet } from 'astro-embed'

On the 9th of January 2022, The Sunday Times published a story that asserted that the UK Government were in the process of phasing out a policy that had arguably kept normal life running during many months of the Covid-19 pandemic.

Where other countries had made Covid testing something citizens had to seek out privately, the 'Test and trace' programme in the UK — aside from doing very little tracing — had made free lateral flow tests available, delivered to your door, in packs of 7.[^1]

Given this policy was supposedly coming to an end whilst the Omicron wave was still out in force, and public life was still heavily disrupted for those looking to avoid catching the virus, there was a rush on Covid tests. The website crashed repeatedly, and frequently was returning a page stating that home delivery tests were out of stock.

Enter the Twitter bot, @LFT_Alert, the inspiration for this tool, which alerted people in their Twitter feed whenever tests became available. A fantastic service, and anonymously set up on New Years Eve 2021, this bot tweeted to its 9000 followers whenever tests went in and out of stock.

But there were two things I could see about this approach that could be improved.

Firstly, whoever was running LFT_Alert needed to keep a computer running themselves in order to repeatedly verify if tests were available. This effectively became a single point of failure to the system (and a point of maintenace and annoyance for the owner) as disruption to the machine would end the service.

And secondly, Twitter is a mess. For me, following over 3000 people, my Twitter feed is awash with hundreds of different faces. To pick LFT Alert's valuable messages amongst the noise of retweets and chatter would be difficult. Twitter offers notifications for any one account, but this had the secondary effect of notifying me whenever tests went in or out of stock. It only matters to get tests when you are running out. To be alerted every time would be a hassle.

So, far too late to the party, as I would soon find out, I picked up my keyboard and spent a few hours on the night of the 9th of January building an alternative to solve this.

lft.tk.gg was born.

The app

Built on two principles: keep the work to the person searching for the test, and don't abuse the LFT portal.

When a user loads the page, they immediately start a timer that executes every minute[^2] which triggers a check to a shared serverless API endpoint, that is cached for 30 seconds. This endpoint goes to the NHS service and uses a private API to check if LFT orders are open or closed. The API returns the state of the ordering, and the last time the check ran.


{
"status": "OPEN",
"timeChecked": 1642919745205
}

The page then responds, updating the last time the check ran. If the ordering is open, it then uses the web notifications API to send a push notification to the user, triggering a link through to the order page.

I thought this was the simplest solution to the problem. A user needing tests could simply open the page in the morning, work through the day, and get the notification as soon as tests were available.

It's a little frustrating that the government service didn't offer something similar to this. The only major downside I had was learning that the Notification API was (of course) not available for mobile browsers (and full on crashes Safari when used natively). I thought about working out how to send notifications to a mobile browser. This would have required some extra work to set up a service worker, and then using the Push API instead. I planned to do this if the service got a lot of traffic (and may still), but the short term supply issue appears to have been resolved, and the status is now very rarely CLOSED.

Still, I enjoyed building the app and learning a few new things about notifications, and keeping browser tabs awake (it turns out you can play audio to keep something permanently awake, but Chrome checks to see if the audio is audible!).

I hope it really won't be useful in the future! Stay safe.

[^1]: Just a aside on how infuriating it is that then they also enforced mandatory private lateral flow testing, at a cost of around £20 per flight, for international travellers, despite everyone keeping a half dozen identical tests at home, provisioned free at a massive discount by a public provider...

[^2]: Chrome throttles background functions running more frequently than this)

The Unintended Consequences of Apple's War on Email Tracking

Fri, 06 Aug 2021 00:00:00 GMT

At WWDC yesterday Apple announced Mail Privacy Protection, an initiative seeking to bring blocking of Superhuman style tracking pixels to Apple's Mail app.

In the video demo Katie Skinner, Manager of User Privacy Software, introduced marketing emails as using hidden pixels to collect your activity, such as your IP address, or when you open or take an action in these emails. Their solution is Mail Privacy Protection.

It hides your IP address so senders can't link it to your online activity, or determine your location. And it prevents senders from seeing if and when you've opened their email. So now you can catch up on email with greater peace of mind.

That's it. That's all we know so far. And though focused in this context on marketing emails, it has significant implications for publishers and the newsletter boom.

Litmus, an email marketing consultancy, estimated that the top email client globally is Mail on iPhone, with 38.9% of opens coming from the app in the first three months of this year. On desktop, if this feature launches on Apple Mail there, that's another 11.5% of sends. The current market share of both is expected around 57%.

This is probably a fuzzier statistic than I'd like it to be, but we can I think safely say that a lot of emails are going through this ecosystem.

As a publisher you probably have two statistics that you are using to determine performance via your <span style={{borderBottom: "1px dashed black", cursor: "help"}} title="Email Send Platform">ESP</span>: open rate (the number of people who opened your email for any amount of time, judged by a small invisible image that is loaded uniquely per subscriber), and click through rate (the number of people who clicked on one or more of the links in the email, judged by each link being tracked via a forwarding service). Apple have just attempted to wipe out one of those.

We don't know how Apple accomplish this, but I think it's safe to presume that it's similar to how Hey.com announced their privacy push in their email client. Hey keep a denylist of tracker addresses, and scan your email for elements that look like they could hide a tracker (anything that's a 1x1 gif) to remove them.

Hey has a second line of defence also: if a tracker slips through via some more structural image, Hey proxies the request to the image via their servers, meaning your IP and approximate location are never leaked to the sender. Apple may or may not do this.

I'm broadly not against privacy online, and I think Hey.com's second step is especially good at protecting users from invasive tracking. In fact it's exactly what Google Mail started doing back in 2013. But, much like other generally-sensible-in-theory provision like the GDPR, moves like this from Apple are going to hit hardest the solo and freelance publishers with their Substack newsletters.

A major publisher will be hurt, sure. They'll lose a lot of data on which they sold their newsletter sponsorships. They'll be less able to confidently purge subscribers who haven't opened their newsletter in months (what if they're iPhone users?). They'll see their open rates drop 30%+ overnight.

But there will be ways around it. And suddenly, with over half your email data at stake, this is what both marketers and publishers will start working on in the coming months. Until emails are wholly plaintext there's always something you can make uniquely identifiable.

A smaller publisher, a local newspaper, a solo freelancer, a small blog; all these will lose data on a significant part of their audience. A likely valuable part of their audience. And it may stifle or slow their growth or opportunities.

Where previously you could unsubscribe readers who hadn't opened your newsletter to save money, now you don't know if they're loyal or not. You'll have to find other ways to entice them to let you know they are reading. A larger publisher can afford to keep 20,000 recipients on a list that never open an email. A smaller outfit cannot.

Apple's fight for privacy is really a fight against the web. In signing up for a newsletter, a publisher or marketer already has a more valuable piece of PII: your email address. By focusing on IP addresses, and blocking trackers rather than proxying them on a fuzzy delay[^1] (which would provide the same useful publisher data without any PII leak of location or time), Apple are not really fighting for their users so much as they are fighting against email.

Apple every year see that the walls of their garden get higher. It's no doubt that whenever something like this happens, Apple have an in-ecosystem solution for you: launch an app on their store, post to their own publishing system Apple News, just don't use the open web.

[^1]: This really should be what Apple do instead: they know what the trackers are, so load them later, from a server rather than from the phone. Give the publisher the data that the email has been opened, and by whom (via the email address), but not where or when.

A retrospective on five years of digital editions

Fri, 07 May 2021 00:00:00 GMT

import { Tweet } from 'astro-embed'

The 30th of March 2021 marked five years since the launch of The Times and Sunday Times 'digital editions'. A project that took two years from the initial concepts, to a final launch.

Now, 1500 editions later, not much has changed of the original. Though it was initially received with skepticism in bundling publish times and generally bringing a newspaper's then-tablet workflow to the web, it has at least been influential on several other publishers since.

In 2019, the i launched "three daily editions" on its apps, alongside a livelier breaking news operation.
The Guardian recently relaunched their paper tablet app as 'Guardian Daily[^1]', supporting readers wanting to read "coverage at a slower pace".
MailPlus, the slightly-weird paid newspaper-made-digital version of the Daily Mail (not to be confused with dailymail.co.uk 🤦‍♂️), have a homepage that looks remarkably similar to The Times, and has stories updated at set times of the day, with the update time called out specifically in the first paragraph.

But even looking past legacy publishers with the kind of content scale that would permit editions, the product themes (timeliness, bundles, and finishability) are behind the rise of many new faces.

Daily deliveries like Morning Brew briefing, the bulleted 'why it matters' summaries of Axios, the timely written prose of the Quartz homepage, and the growing array of newsletters on Substack that cover news at a slower pace, in a more digestible manner, all exhibit parts of these three components.

Five years on though, the industry is still scratching the surface of what an edition product might look like. Instead of working on the job an edition does for a reader, we are once again focused on what job a digital offering does for the journalists and editors that produce it.

We are not our readers.

The story of editions outside of the digital-first startups can be told as an allegory of the fundamental disconnect between newsrooms and the concept of news product that seems to constantly evade them.

In technology roles you are encouraged to take the time to look back, to retrospectively analyse your successes and failures to learn from them, something that there is rarely the opportunity to do in editorial given the relentless pace. Five years after I helped build it, and as many of that team depart The Times and Sunday Times I think editions as a concept deserves a retro.

So I thought I would write about why I think editions done well are still a compelling product, and then how we continue to get them wrong.

What do I mean by editions?

There are three concepts that I think go into building editions, and they align reasonably well with the concept of RFV as a business metric. Recency, frequency, and value (as in recency of visit, regularity of visit, and length of visit) are common metrics that news providers might use to determine their performance amongst readers. Generally if you can bring people in, give them reasons to come back, and serve them well during the time they are with you, you have a happy reader.

One of the driving forces behind this behaviour, and something that editions explicitly encourage is creating habit. A bundle of high quality, curated, and timely content, produced regularly, are excellent foundations on which to build a habitual, valuable audience.

Loops

This works best if you can provide a platform that marries this bundling with other features that help build further habit. As Nir Eyal describes in his book Hooked: How to Build Habit-Forming Products, you want to build loops that the user is prompted to come back to, however you may create or deliver that prompt, and then provide value and opportunities to trigger further prompts in the future.

These loops are a part of why that slim proportion of readers that engage in comments are so much more likely to return: they typically read more content, and spend more time reading it, than those that do not engage. Leaving a comment is an 'investment' as Eyal describes it in the last part of the hook. A comment may also, depending on your capabilities, provide the first part of the hook, the trigger ("Matt liked your comment", "Matt replied to your comment" as an email notification or otherwise). Creating touchpoints for readers to engage in ways that permit you to bring them back are gold.

So how does an edition relate to a comment?

It's in the product's ability to create that reader investment, and initial trigger for them to return. With a live news site, you could allow readers to follow topics (like My FT), or authors (like... er... My FT). Both of these up front investments likely result in notifications (a trigger), but one that could potentially come at any time, with an unknown volume.[^2]

An edition model, done well, enables the ultimate end-goal in planning: an open newsroom with your readers, telling them what you are producing and when they can expect it. They can sign up to a known and predictable quantity and timing of editions, with the potential for them to engage further in highlighting coverage they want more of in coming editions.

Digital technologies allow the same kinds of notification for individual stories or storylines, and they provide regular touchpoints (the edition publish times) to re-engage readers to subscribe to the next touch point, or the one after that.

An enforced limit

The other aspect of editions I think a lot of is 'finishability', described as creating a valuable visit for the reader through enabling them to see—though perhaps not meet—the finish line.

It would be very difficult for you to 'complete' the digital product of The Guardian today. You'd need to find every story The Guardian has written, quickly check it out and determine whether you wanted to read it or not, then potentially read it. This is the power of the alternative in Guardian Daily. It's a front-to-back bundle of editorially curated items, and once you've finished browsing through them you're done. Your trust is in the publisher to curate what is important for you to see, which is a service you pay for just as much as the journalism itself.

This idea of 'finishability' is fundamental to the success of sites like Axios, which trades in both breaking news and detailed analysis, but lets no story lose the reader's attention through a succinct summary delivered in bullet points.[^3] Brevity is crucial.

The Guardian, years ago now, used audience data to cut their output by a third. As described by Chris Moran in this exceedingly worthwhile interview with Emily Bell, the problem presented by the internet was a combination of the feeling of unlimited space, whilst still being limited by opportunities for promotion. A homepage, or a social feed, or a newsletter can only display so many possible items for consumption. Why not limit yourself to a much more succinct service through design once again?

Editions, at their best, should attempt to reintroduce the limitation of print space. They should make editors acutely aware of the attention commitment they are asking of the reader, and of the reader's potential need at the time. Journalists have a very odd fascination for the idea that more stories, and more choices, are always better than less. We need to remember that we are not our audience, and their needs at any time are very different from ours.

Attention in an article is an important metric, but when applied to the reader's entire visit it is their quality of attention we should strive for: how quickly can you get a reader into a value-providing action such as reading an article or watching a video. Time spent hunting is time they could afford to save.

Of course there is a lot more that is possible: in learning more about habits and interests of readers you could deliver them partially or entirely custom bundles, or more effectively A:B test structure and content than it may be possible to do in a more real-time traditional digital news product.

Timeliness

Editions rebundle articles. This is a step back from the course of the internet where the article has become the atomic unit of consumption. When it was impossible in a printed world to buy an individual article, in the digital domain you likely land on an article page directly when searching for news, and may never see the overall package.

This is fine for flyby consumers, but in publishers with membership models the homepage, and other elements of the digital offering, have increased value. Readers may visit directly and use it to skim your reporting, or to get a feel for what your editorial focus is.

Homepages today are (almost all) live news wires, with latest news interspersed with high-performing features and older articles. Repeat visitors over the course of the day (themselves rare, admittedly) may see articles they have already read sitting in the page, in a different position or with a different headline[^5].

Editions offer the opportunity in rebundling articles to create periodic updates to your homepage or homepage-like features (such as a specific 'edition' page). Your predictable updates provide opportunities for nudges and notifications at known times for your audience.

And, should it be useful to your content strategy, editions can help to move newsrooms away from the always-breaking news agenda that is endemic of the transition to the internet. A rush to publish first means a multitude of updates, work passing between hands, and readers not knowing the difference between your outlets and the dozens of others carrying the same stories at the same frenzied pace. National broadcasters and cable networks will always cover breaking news, so bring your expertise and angle when it's ready in the next edition.

The state of editions

Moving on from the thematic elements of editions to the practicalities of introducing them to readers, it's no surprise to see that in practice little of the potential is realised for readers.

The Times, MailPlus and the i all intersperse new stories with old, they all have long scrolling homepages based on print sections rather than reader need. None meaningfully personalise or engage further than comments or social pushes, and none offer to notify me about editions, leaving me to learn for myself when the next update is by deciphering unloved page furniture.

The Times has even recently started to completely break with the concept, starting rolling live news coverage on their homepage, something which has become increasingly common as time has gone on. Journalists are working rigorously to serve themselves, and the social and search giants, rather than their loyal paying customers, or carving any kind of niche provision.

The editions model is the clearest indication of the product/journalism divide I think there has been. In a world of print, publishers competed on content, geographical monopolies aside. In a global digital world, attention is more sought after than ever, and content is most commonly combined with some valued delivery mechanism to provide the best experience. This could be a movie streamed at adaptive resolutions straight to your smart TV app, or a song from a playlist constructed from your listening habits to introduce you to new artists.

In our world of journalism it could equally be an edition. Some form of the packaging of journalism that isn't atomic, isn't gamed for search, and doesn't purely rely on the levers of social networks. A product in its own right that delivers a unique service for readers[^4].

That is where we could be.

[^1]: The GitHub code repository for this app is even called 'editions', although it was originally called Mallard, which is super cute 🦆.

[^2]: Speaking of volume of stories, Feedbin, Feedly and other RSS readers do this really nicely by showing you the number of stories you are likely to receive by subscribing to any one feed. eg. one article per month, or twelve articles per day. Don't tell me that information wouldn't help you make a better decision as to what to spend your time with.

[^3]: Axios also have this lovable, almost-pointless Go deeper clickthrough for stories on their list pages that calculates the minutes to read. More often that not, that time reads (<3 minutes).

[^4]: I feel it's worth nothing that this might not, and probably should not be the only product that you offer to readers as part of your service (unless you are really committed to this specific branch of readers), but increasingly could be something that an editor produces from your coverage, or is almost entirely automated. Publishers that jump into editions one day, and out of them the other, do nobody a good service. How on earth are your readers supposed to know what's in your head?

How to create a voicemail inbox for your journalism

Sat, 13 Feb 2021 00:00:00 GMT

The New York Times recently produced an awesome article series called Primal Scream, where a published phone number allowed readers to call in and talk about their experience raising children during the pandemic.

Listening to your readers, however you can, is crucially important in evolving your delivery of digital journalism. Entire consultancies such as Hearken are built on involving communities in business, especially journalism. City Bureau in Chicago bring people together in workshops to discuss the problems they are facing, ultimately producing media that is more honest, more involving, and crucially more desired.

But how do you do that when you're a national newsroom at scale? Or even at the scale of a single city? How can we use technology to make it easier than ever for readers to get in touch with journalists and becoming involved in our storytelling.

Whilst years ago running a phone in line like this was something that only the largest newsrooms with the deepest pockets could accomplish, advances in digital technologies have made creating the Primal Scream phone line possible for even the smallest reporting outfits.

Recording hundreds of voicemails to a dedicated phone number could cost you as little as $20. It requires very little technical knowledge, and just a few short steps to get everything up and running.

We used the same technology when I worked for The Times for our NHS memories project. Hearing the voices of readers telling their own stories adds impact that a typical written interview wouldn't have. Primal Scream I think shows that impact in a whole new way.

So how do you do it?

Basically we have a company called Twilio to thank for this. Twilio bring all that complexity in call-centres and telecoms networks and make it accessible and programmable over simple control panels. Fortunately what we'll be asking it to do is very easy to follow, and you don't need to know any code to get started.

There are three main parts to this project. You're going to need to set up two things: a phone number, and a handler function for calls to that number. And you will need to record one piece of audio to upload that will act as your voicemail greeting.

Let's get going:

Sign up to Twilio
To start, let's get a phone number. From your control panel find 'Programmable Voice' from the overflow menu (pin it for easy use later), and then go to 'Numbers'
You'll need to upgrade your account by adding a small amount of cash at some point around about here. If you aren't prompted, or can't find out how, there's a guide on how to do that here. Unless you enable the auto-recharge feature, should you run out of credit, Twilio will simply cease to operate service to you.
Find 'Get a number' and search for a number you can use. Your number will need to support voice calls, so look for the 📞 phone icon.
Once you've purchased your number, we need to set up your voicemail. Let's start with the recording. Use any audio recording program, such as Audacity or any online tool to get a file that you are happy with.
Now go to Functions->Runtime in the menu (pin this also) and click Functions Classic. The new Services feature doesn't support what we need to do yet. Click the red plus and give this any name you want voicemail-recorder for example.
To upload the file we're going to need to open the menu again and go to Runtime->Assets. Select Assets (Classic) in the sidebar, and upload your file. Make sure it's public. Once it's uploaded select the URL in the PATH field and save it somewhere, you'll need this in a moment.
Now back to the Functions Classic menu, and open your function. Let's give it a PATH like /voicemail (this must start with a slash). Underneath in the configuration field make sure Access Control is disabled, select the Event to be Incoming Voice Calls, and then in the Code box enter the following. Make sure to put your saved asset URL in!

exports.handler = function (context, event, callback) {
  const twiml = new Twilio.twiml.VoiceResponse()
  twiml.play('YOUR FILE') // Put your asset URL in this between the quote marks
  twiml.record({
    timeout: 10, // The number of seconds of silence to wait before hanging up
    transcribe: false, // Change this to true if you want Twilio to try to transcribe your recordings ($)
    maxLength: 300, // This is the maximum length that the audio recording can be.
    playBeep: true // This tells the function to play a bleep after your message or not.
  })
  twiml.hangup()
  callback(null, twiml)
}

Hit save! Hopefully there are no issues. Now we need to make it so when someone calls your number, this code is run. So go back to the programmable voice menu for your phone number.
From here, scroll down to the Voice & Fax options. Select Accept Incoming and check it's set only to Voice Calls. Then where it reads A call comes in change the first option in the drop down menu to Function, and now in the Service menu that appears to the right, select default, and from the Function path menu, your path.
Everything's set! Call your number and check that it's all working. You can find recordings in the Calls log->Incoming menu of your phone number.

If you have questions, or can't get it working, give me a tweet and I'll see if I can help you out.

Be aware!

It is possible for a hostile actor to attack you by repeatedly calling your number and leaving long messages. Though they will not be able to block legitimate contributors, they can cost you money, and they can cost you time in sifting through the results.

It is also possible for your fun experiment to turn into $1 a month forever unless you turn off your phone number at the end of the project. Make sure to remove any reference to the phone number from your reporting at this point, as otherwise it is possible for another actor to potentially take over the number.

What we learned from rebuilding the editions CMS for The Times

Mon, 03 Aug 2020 00:00:00 GMT

In the last post I took you through a short summary of what it took to get Edition Builder live at The Times and Sunday Times. If you haven’t read it, you should, it’s pretty good background.

(In the last post I also mentioned I’d be writing this in a few weeks. That was pre-coronavirus, so these posts have taken a seat at the very back of a very long series of seats. Apologies for that!)

As one of the larger engineering projects we have undertaken, Edition Builder broke new ground in a bunch of areas. We tried things that we hadn’t attempted before, and pushed systems to their limits (and well past them in one incident).

There were a lot of challenges we encountered as a team that we had to work out how to navigate. Though its not likely that you’ll learn a lot about the specific issues we encountered, I hope there are patterns you can recognise that are best avoided, and things you can think about should you work on anything similar.

Searching for articles

Search in Edition Builder was, for most of its life, unworkable. I don’t know why I persisted so long convincing myself that it would work, but it was only ever going to get worse from the moment we realised this problem. Not raising this earlier in the lifecycle of the project was a big mistake from me, I think because I thought it was unsolvable, but as with most things online, someone else has usually solved the problem you are facing, and often they’ve built a SaaS solution that will cost substantially less than your engineering time building and maintaining it.

On to how search was terrible. Our content API was never built for search. This is because of the link between our CMS and our content api was only one way: via publishing. Our readers used a managed SOLR cluster for site search. Editorial users only ever needed to work with content inside the CMS, which had reasonable search and categorisation functionality. In the world of Edition Builder existing, suddenly there was a need to find articles before they had gone through the publishing process.

<figure> <img src='https://assets.tk.gg/1_EvqKLCBT8axTCVp7OYqgTw.png' height={916} width={1119} /> <figcaption> All 776 of those articles were stored in full in your browser’s memory, because performance be damned, we needed somewhere to put them! </figcaption> </figure>

“No problem”, we thought, “we’ll set up another version of that API” (because building versioning in to the API was not something we wanted to be responsible for). (and it has ended up coming anyway as part of a later rewrite).

Except of course there was no reasonable way to perform a scan or search as the endpoints just didn’t exist. So in the end we settled for fetching a range of data between two dates, and storing that in the application state to then sift through locally in the browser.

Constructing that index took as long as three seconds per 24hr period searched, so we limited it to three days at a time. Worse: to get new articles, you had to refresh your local index, causing another delay of however long you had selected.

The moment this hit user testing it became clear to us it was a big problem. The search was just too slow. But was it even within our power to fix? We sat on the problem, hoping a solution would rear its head, as every fix we could think of involved work well outside the scope of the project (and adding months to its delivery).

Step forward a couple of months and a couple of members of the team had been tasked with something new. We’d just signed up for Algolia, a SaaS search provider, and needed to get it working for our readers as soon as possible, decommissioning the SOLR cluster as we went.

Algolia is a breeze to work with, and we quickly saw significant improvements on what SOLR had been returning. We feature flagged one engine over the other, and, with journalists being some of the world’s best critics on search engines to surface their own writing, I jerry-rigged a glitch.com app that we sent round to find out which performed better.

A few tweaks to the ranking algorithm later, we were satisfied. And then it dawned on us that we could use exactly the same thing in Edition Builder.

Tearing out the old system and replacing it with a prototype took just a few days. Again we played around with ranking and faceting to get what we needed, and then we threw it in front of users to test. Immediately we saw improvements. A local in-browser index was no-longer necessary. Every query went fresh to the remote index, resulting in more recent results. We could return results almost instantly, just as you typed into the box.

Even better, when users talked about issues they were having with search, they were relatively easy to diagnose. We could replicate the same searches, see how the ranking affected the results, and adjust accordingly, even on a per-query basis.

How we structure edition updates

A second issue we looked to solve focused on how we structured editions. On reflection, I think we could have gone further, and we shouldn’t have shut conversations down so early without seeing them out. Again here there was too much of an assumption we had the best solution at the time, and not enough thought to restructure things early on.

In the previous system, editions over the course of the day were overwritten when the latest update was published. This meant throwing a lot of useful data away, a lot of which we were technically paying for in terms of journalistic decision making. Wouldn’t it potentially be useful to be able to learn about how stories move around the edition, and save different updates separately.

We initially explored a git-style branching system where root editions would have branches with changes on them that then potentially were merged back over time. This had the advantages of immediately making sense to developers, and the disadvantage of it not making any sense to any of our users 😭. Can you imagine playing with git-editions on deadline?!

<figure> <img src='https://assets.tk.gg/0_fFwub9PiTsadeVGj.jfif' height={353} width={700} /> <figcaption> How a branching model might have enabled multiple versions to work together with each other </figcaption> </figure>

The ‘git’ style feature would have brought us some level of historical and history data, which we could have used for scrubbing back and forth, but the complexity add at the time, along with the UI challenges of having this work seamlessly for users was problematic.

We set out to solve the problem actually in front of us, and built the simplest solution we could see that did that.

<figure> <img src='https://assets.tk.gg/0_3blbvJMGpwdg_m0V.jfif' height={920} width={1920} /> <figcaption> Our old system never stopped being edition A. In the new system we just create a new edition each time that is a copy of the previous one, whilst saving its parent. </figcaption> </figure>

Simply enabling users to copy past published editions into future timeslots solved the data problem, and enabled users to work on future editions whilst still being able to make tweaks and changes to past ones.

In hindsight though, it didn’t solve enough of the problem. For example, in a short timeframe you might want to urgently prepare two options for a future version. A direct copy gives you very little flexibility as other stories that might change need to then be made in two places. We thought at the time about creating overlays you could flatten into an edition update, but again it was difficult to work out if the value would ever be delivered.

Looking back I wish we’d thought harder about how sections interacted with editions and made that less fixed. In reality an edition is just a bundle of sections, and making them updatable independently and then bundleable for an update might have both provided an easier structure to manage, and an easier product to break up should we plan to in the future.

The saving grace at least, given the changes made, is that it will be significantly easier to integrate our bespoke analytics service (INCA) into this system, so as to show journalists when they place stories what the story is expected to do based on its position in terms of our key metrics. This will help journalists understand the power of their decision making better (as stories towards the top of the edition have generally higher numbers than those lower, stories in larger slots likewise etc).

Publishing and the initial breakdown

Finally we have the bug that held us up for four months. The lesson: never trust orphaned systems to do anything.

This is the working version (we also added lazy loading… which would have been a really good idea at the start)

Edition Builder has a feature that shows the entire edition to you at once in a single screen. In order to do this we have to fetch all the articles that exist in this view from the API.

In addition, Edition Builder, for the first time, had to handle published and unpublished content (as mentioned in the search feature). This meant that two versions of the same article could exist at the same time, given editorially you might want to feature either (the unadjusted published version, or the perhaps more-up-to-date unpublished version).

The consequence being this view needed to switch from two sourcing content APIs, the in-production API, and our preview API, depending on the article.

And when we published articles, we switched any reference for this lookup from the preview version to the production version. Simple! When it came to running our first live editions everything went really smoothly. We worked our way back from the 5pm edition to the 9am, in order to have the least impact in the most supportable working hours, and finally had the midnight edition in our sights.

<figure> <img src='https://assets.tk.gg/1_uwg3a5UmyrFOBBWMD1stYw.png' height={226} width={564} /> <figcaption> I don’t always @channel, but when I do it’s the best @channel in the world </figcaption> </figure>

Where the 9am, 12pm, and 5pm editions are relatively small updates on the main bulk, the midnight edition publishes several hundred articles all at once. For architecture that was not necessarily built for this in mind, bursts like this can be fatal… which we learned to our peril.

<figure> <img src='https://assets.tk.gg/1_Ljto-7_OnPus-WRFdeJJjA.png' height={428} width={546} /> <figcaption>The as-promised ‘crippling load’ 🔥</figcaption> </figure>

When Edition Builder publishes the midnight edition, all of these articles get posted into the content API in the same queue. This usually causes a little bit of a spike.

But Edition Builders incredibly smart and well-meaning feature that showed you the entire edition after publish, then immediately tried to load these hundreds of articles from the same system that was trying desperately to ingest them.

And what’s more, in order to check if there were any changes, it reloaded them every thirty seconds.

Cue heavy breathing from our operations team as our apps started to intermittently return 500 errors. And by intermittently, I mean all the time for about fifteen minutes. 😬

...

Fixing this was no mean feat. The content API at this stage was practically orphaned, with a replacement on the way, but with no clear end in sight.

What’s more, this problem only occurred in production, and we couldn’t work out how to replicate it on our pre-production instances, which all handled the load fine (perhaps because it was the only load they were seeing 👀).

We, sensibly, were not allowed to continue our production rollout whilst this error still occurred, but we couldn’t replicate the error to show it was solved, so our catch-22 just kept enumerating.

After far too long, our solution came in the form of new talent. Stefano Berdusco joined me on the long road to salvation and together we wrote load tests (he wrote them), we built a perfect replica of production (he built it), and we flipped the sourcing to the new API, so as not to cripple the old one (maybe my idea, but given my track record probably also his).

😍 Our tests, with load applied, showed the new API (built with this burst in mind) was able to easily handle the preview state. And we were back on track.

So that’s that. Thanks for reading, I hope we all learned something. (Also in going through our Slack logs for this post, perhaps my favourite short-lived bug below)

Building Editions at The Times

Thu, 23 Jan 2020 00:00:00 GMT

Internal tools never get enough attention.

Building them can be a thankless, unseen task that’s rarely prioritised, as it is not immediately obvious how to tie investment in them back to the bottom line. It can be too easy to jump on the customer-facing speedboat and jet away from them, leaving a wake of stopgaps that can often leave you with tightly coupled intricate workarounds that are enormously difficult to pick apart.

This was the case at The Times for the last four years. While we spent a long time discussing and designing how editions might work on our readers’ screens, we didn’t place any focus on how to make building these experiences manageable for our editors. We quickly shoehorned the process into our generic digital CMS, making it difficult to maintain or build upon, and creating a process for our editors that took much longer than it should have done.

Edition Builder was our solution to this problem, and has been two years in the making. At its heart, it is a simple drag and drop interface that allows you to place and publish stories in our edition. Beneath the surface is a huge amount of user research and UX refinement and, crucially, the foundations for future newsroom tools.

Late last year The Times went live, and this week The Sunday Times joined them, and now all editions from The Times and The Sunday Times are produced on Edition Builder. I’d love to tell you more about the story of how we got here and what it means for the future.

<figure> <img src='https://assets.tk.gg/newspaper-bundle.png' /> <figcaption> Editions replicate the print bundle in a digital form: timely, broad, and finite. </figcaption> </figure>

From prototype to production

I’m well known as someone who is not easily impressed, but from day one editions excited me: the idea of publishing fewer stories, of higher quality, in a bundled format was novel. However, the way we produced them irked me. Our editors were fighting with the system to make the most basic changes. The more time went on, the more we wanted to do with editions and the more painful it came to be.

<figure> <img src='https://assets.tk.gg/eb-prototype.png' /> <figcaption> {' '} The first Edition Builder prototype showing the menu to create article containers. </figcaption> </figure>

In a break from our day-to-day jobs supporting editorial, Chris Hutchinson (now a Principal Engineer at The Times) and I took some time out with the implicit instruction to build something cool that we thought needed to exist. We only had a week, and we went slightly over schedule, but we knew from the start we wanted to solve our edition curation problem.

The value we were able to provide from our experience understanding the editorial pain-points, being able to theorise a solution, and eventually prototyping it wouldn’t have happened outside of our cross-functional editorial development team. These groups of product and engineering-focused, journalism-obsessed individuals that have sprung up at news orgs all around the world are, in my opinion, some of the most valuable people in the business.

Our team’s dream was to empower the newsroom to do much more with editions, faster and more intuitively than before, but the current monolithic CMS made this more difficult than it should have been.

Edition Builder’s prototype came out of that week, and our demo roadshow made the rounds through the technology and editorial departments. There was ambition to build more. Finally, after a presentation that included way too many Jurassic Park gifs for the seniority of the audience, we got there, and the investment case was signed off.

<figure> <img src='https://assets.tk.gg/methode-is-gross.png' /> <figcaption> {' '} Our first challenge was working out what features from the old system were useful, and which were not. This is our old edition plan, built into the same system that made our printed product. </figcaption> </figure>

We formed a team of six engineers, a user-experience designer, and myself, acting as the product manager. As a new team, with a clear purpose, we dived in and (of course) began making mistakes.

It quickly became apparent that some of our assumptions were misplaced, or just wrong. We designed an introduction screen that we thought made it clear which edition was live at any one time: nobody from editorial understood it. We thought that previewing articles was something people did habitually but didn’t really need: it turns out they did. And we thought, most wrongly, that getting articles out of the system, and publishing our eventual output would be the easy bit: publishing functionality alone took six months to build.

Inevitably there are more unknowns than you realise entering a project like this. Our objective was very clear: build a slimmer, faster, more efficient, and more extensible system for editions; and sliced up, the journey ahead looked manageable. But between the cracks were services that hadn’t seen new deployments in months, orphaned systems held together with the code equivalent of a low-budget PVA glue, and teams across the business skittish and burnt from too many lookalike projects gone wrong.

This is mostly why it took just over two years to bring Edition Builder into production. When a single tool is responsible for a disproportionate amount of your output, replacing it becomes a project of either changing what you do with it, or covering off every feature it has.

<figure> <img src='https://assets.tk.gg/open-articles-prototype.jpeg'

<figcaption>Making it so that users could open up articles in the original editing system was important. This was the initial paper prototype I threw together.</figcaption> </figure>

Making it so that users could open up articles in the original editing system was important. This was the initial paper prototype I threw together.

There was no logical physical analogy that didn’t sound absurdly risky: like shifting your entire printing press all at once to another that has identical output, but also looks and works completely differently.

Getting editorial users on side was a crucial part of this project from the start. Over time we probably ended up building a few small features that weren’t needed (and we are now in the process of gradually removing), but the goodwill gained from these relatively small features greatly outweighed the development cost.

That established trust was important to move forward with some bigger work items without consulting users every step of the way, whilst working more closely with them on items we knew they cared about. It meant we slowly built up a relationship where we felt aligned in the combined goal of transforming this workflow to make a better product, rather than being forced to replicate every step for a feeling of safety and familiarity.

Ultimately this all paid off. Even though it took a little longer than expected, and the team working on it changed in its entirety when we hit a particularly sticky deployment issue towards the end. Some of our heaviest editorial users have given us rave reviews: it is “far faster", the time savings “impressive” and the rollout “exemplary”.

In our prototype, we worked out a time saving of approximately 5x on doing some basic actions. Edition Builder in production is in some cases 50–100x faster than the previous workflow to perform the same task. We really should hold everything used this regularly to the same standards.

With that I’d like to thank the people on the team that brought us to where we are now: Himesh Ladva, Tom Chambers, Elliot Davies, Cassie Best, Leah Haskoylu, Chris Hutchinson, James Fuller, Geoff Ford, Chris Jordan, Stefano Berdusco, Jack Stevens, Tom Hoad and Adele Kufour.👏

I’d also like to thank everyone in the newsroom that worked with us to help develop the tool and get it into production, including Wes Rock, Gary Mitchell, Abhi Ahluwalia, George Lindsay-Watson, David Rankin, Sadie Gray, Hannah Rock, Hannah Scott, and Michael Brear.

<figure> <img src='https://assets.tk.gg/eb-early.png' /> <figcaption> An early build of the live system with a simple article search loaded into the browser’s memory. </figcaption> </figure>

Read part two here, and learn the lessons we learned putting this system together.

And, enjoy last weekend’s The Sunday Times; the first Sunday edition made with Edition Builder. 🎉