Mapzen

Mapzen project updates December 2019

Fri, 27 Dec 2019 00:00:00 +0000

The Mapzen diaspora was active in 2019 with thousands of pull requests. There continues to be a future for the projects incubated at Mapzen and those tools and data continue to be used and updated after our January 2019 transition to the Linux Foundation.

project	2019 closed PRs	stars
Pelias	720	1,799
Valhalla	236	999
Tangram JS/ES	102	1,506/540
Tilezen	143	394
Who’s On First	30	n/a
Who’s On First data	1,554	232

We’ve also been making a few updates to this website like renaming “products” to “projects” to reflect our new status with the Linux Foundation. Also, come by and say hi on Twitter!

Project updates since January 2018:

Map display

Tangram JS

Tangram JS has added several new capabilities over the past two years, along with a new build system to output native ES6 JavaScript modules (since v0.16), syntax refinements to avoid boilerplate code, and performance and memory usage improvements for labels and other areas.

New data source parameters for controlling level-of-detail support cases where tiles may be either too large (in file size) or too detailed (in feature density or coordinate precision) than desirable for the current application, such as generalizing raster data or optimizing for low network bandwidth. The zoom_offset parameter (v0.17) down-samples tile data by one or more zoom levels, while the zooms array (v0.18) provides fine-grained control over which zoom levels will load new tiles.

For example, we can use zoom_offset: 1 to opt for softer and more performant terrain tiles:

Geo-referenced (un-tiled) raster image layers were added in v0.17, useful for ad hoc overlays such as historical images or local aerial imagery; the scene author can apply alpha processing or composite multiple images into a single layer. Together these features extend Tangram’s unique flexibility for real-time raster processing, with the ability to mask, filter, and analyze multiple raster and vector layers simultaneously with custom GLSL shader code.

Agricultural drone imagery filtered by raster-derived values in a Tangram shader:

Or use a slide tool to compare contemporary topographical data (shown here from Nextzen terrain tiles) with contour lines from a USGS historical map:

Significant label improvements arrived in v0.15, with labels now properly colliding across multiple data sources and tile boundaries – enabling more seamless data overlays interleaved with Mapzen basemaps. Label collision is also now re-evaluated in between zoom levels, for increased label density.

Filtering and styling syntax also gained new capabilities with native support for complex JSON objects (v0.19), whether included in GeoJSON properties, or embedded as stringified JSON in MVT as output by Mapbox’s Tippecanoe tiling library. These objects can now be automatically parsed from MVT data sources, accessed in layers with common JS-style “dot notation” syntax, and filtered with new array-to-array set operations (includes_all and includes_any). Tangram’s hierarchical layer structure is powerful, but managing parallel filter sub-trees can be complex; we’ve added new priority and exclusive keywords (v0.18) for more control over the layer matching logic, useful for disambiguating competing sub-layers or short-circuiting filter paths.

Finally, several syntax extensions focus on removing repetitive boilerplate code in scene files. Most notably, the blend_order parameter can now be set within draw groups (v0.18), greatly simplifying complex stacks of label and sprite symbol layers. v0.19 brought built-in rendering styles for all combinations of blend and base, removing another common pain point with blend modes. And the new all_layers data source parameter composites an entire vector source into one scene layer, useful for creating wireframe views or loading MVT tiles with variable or unknown layer names.

Tangram ES

Tangram ES has steadily stabilized and improved its own capabilities with a host of changes over the past two years, all bundled up into our latest release: Tangram ES 0.11.

This release includes an essential change for Android users. Following the guidelines for Google Play, the Tangram Android SDK now includes 64-bit binaries for both the ARM and x86 architectures. If you have an app on Google Play that uses Tangram, you should upgrade to this release!

zoom_offset for data sources, as described in Tangram JS above, is now supported in Tangram ES as well.

Android apps can now render a Tangram map into a TextureView. This lets you embed maps into more complex View hierarchies or apply bitmap effects that aren’t possible with a SurfaceView.

A new JavaScript engine abstraction adds the ability to compile Tangram ES with JSCore, the native JS engine on iOS and macOS, reducing the size of your compiled binary.

Both the iOS and Android Tangram SDKs have a new, much faster interface for adding custom data layers. The scene below used to take 4.5 seconds to load on a phone - now it takes 22 milliseconds!

These are just a few of the changes in Tangram ES since becoming part of the Linux Foundation. We’ve also made loads of bug fixes, performance enhancements, and developer workflow improvements. Head on over to our repository to see all the activity!

Search

The Pelias geocoder has seen many improvements in the last two years. Active development has been continued by the growing Pelias community and by Geocode Earth, who offers a hosted geocoding API and consulting for custom geocoding installation.

Notable developments since joining the Linux Foundation include:

A major focus on customization including support for importing custom data via CSV, customization of “boost” levels to fine tune results, and display of additional data through the “addendum” feature
Improved autocomplete through the new Pelias Parser
Massively improved query performance through multiple optimizations to Elasticsearch queries, use of the libpostal service, and more
An easy to use Docker framework for quickly getting started with Pelias
Improved deduplication logic to avoid presenting identical looking (and confusing!) results
New query parameters such as boundary.gid for filtering by parent administrative area
Support for importing OpenStreetMap relation records
A new Transit importer supporting GTFS and other common formats
Support for Elasticsearch 5, Elasticsearch 6, and soon Elasticsearch 7

Routing

The main themes in the Valhalla project have been centered around maturity and efficiency. The code is easier to build on platforms like Windows, Android and iOS. We’ve added some new features like 2 new costing modes and are working towards a multimodal bike-share routing. V3.0 of the Valhalla tile spec was released about a year ago and allowed us to make some gains in tile sizes where we know the use-case the tiles would be targeted at. The bulk of the rest of the contributions have been largely around maturity of the product adding new features for properly selecting origin and destination edges in the graph or allowing/disallowing routes based on time of day or access restrictions (ie accuracy of the routing). We’ve also generally improved the performance and usability in scenarios where you are disk limited (server-less or mobile).

Windows build support
Motorcycle costing
Experimental nodejs bindings
Supported for predicted traffic influenced routing
Reduced boost dependencies to header only for easier crossplatform development
Time dependent isochrones (isochrones differ depending on predicted traffic)
Valhalla V3.0
- Smaller tiles especially when omitting elevation sources
- Improved performance (~100ms cross country US routing)
- Support for filtering out parts of the graph based on mode of travel (remove bike-only edges or remove pedestrian-only edges for example)
Support for polyline5 and geojson geometry output
Taxi costing
Preferred side of street (and tolerance) for arrival and departure locations
Max search radius for locations
Via and break through location types allowing for u-turns with no separate leg and continuing in the same direction with a separate leg respectively
Internal API now tracks turn lane information at the nodes along the route
Improve reachability checks to search in both directions
Caching improvements for on-the-fly tile fetching (serverless mode)
Allow time-independent routes to ignore timed restrictions
Allow u-turns where no other option is possible
Bike-share multimodal coming very soon!

Vector Tiles

Long term support for free hosted access to vector and terrain Tilezen tiles continue to be supported by generous AWS Research Grants thru April 2020, and global builds sponsored by HERE on Nextzen. Please email if you’re interested in sponsoring a build or access to tiles!

v1.5 added an inexpensive global tile build, international road shields (img), and many more kinds of points of interest.
v1.6 significantly reduced file sizes between 23% (p50) and 30% (p90) globally by additional geometry simplification, dropping features, dropping properties, and more aggressive merging to multi-lines and multi-polygons in low- and mid-zooms.
v1.7 added more green areas, prototyped point-of-view for disputed boundaries, and added collision_rank for more predictable label collisions, and population_rank for sizing place labels, and modeling support for showing traffic flow lines and traffic incidents.
v1.8 support ~30 points of view for disputed boundaries, country, region, and capitals for high-zooms from OpenStreetMap and low-zooms from Natural Earth, added wikidata_id concordances. The version hosted on Nextzen since July 2019.
v1.9 (under development) focused on adding more landuse green areas at low- and mid-zooms, contours, and outdoor feature enhancements

(above) Chart shows sizes in bytes (logarithmic scale), based on top 100,000 tiles from openstreetmap.org logs at 512 pixel zoom.

We will continue to host indefinitely the raw metatiles assets in an S3 requester pays bucket. You can now run your own tile server via tapalcatl-py: it’s cheap for low usage and easy to configure as an on-demand Lambda + API Gateway with Zappa. This is the same software and configuration that powers the existing endpoint, but billed to your own AWS account. Setup instructions are in the project README and most users should expect to see AWS bills less than $5 / month.

If you’re running a larger commercial service you’d also want to add CloudFront edge cache and look at running tapalcatl-py via Docker in EKS.

Terrain Tiles

Tilezen terrain tiles are available via Nextzen with an API key, and are still accessible in 256 sized tiles via the existing Amazon public dataset S3 endpoints.

The 512 and buffered 260 and 516 sized terrain tiles on Nextzen are made possible by running Zaloa as a Lambda function.

We did additional work extending Marble Cutter to tile indexed landcover sources into tiles, and prototyped a vector export.

Who’s On First

On the data side, Who’s On First gained comprehensive worldwide locality coverage by adding over 4 million new locality points from GeoNames. Projects like Pelias can now import Who’s On First alone without having to worry about deduplicating results between the two datasets.
Added 11.7 million more name translations to 972k places, thanks Wikidata.
Posting monthly changelogs (1, 2) at whosonfirst.org/blog
Legal audit of WOF datasources terms and conditions, we’re enterprise ready.
Expanded and validated Disputed area coverage, including guidance on label hierarchy display
Split the mono repo into per country repos for the administrative data at the request of Github.
Alternate geometries are more explicit, easier to index, and include in distributions.
All features now include mz:min_zoom and point geometries include label bbox
Easier to work with the data in databases, documenting fields, removing nested : strings in property names, simplify default geometries, resolve geometry self intersections
Label shortcodes and abbreviations for country and regions
Capital cities declared with new data properties
Per country rebuilds, including: Norway, Poland, France, Switzerland, Denmark, Germany, India, Ireland, United Kingdom, Sweden, Finland, Belgium, Australia, Austria, Saudi Arabia, United Arab Emirates, Canada, United States (townships for localadmin, reset geometry of oversized mostly rural neighbourhoods), and Netherlands
Other small fixes from customer feedback

Pre-GeoNames import locality coverage visualized:

Post-GeoNames import locality coverage visualized:

Transitland

Transitland started in 2014 as a data platform aggregating transit data in the San Francisco and New York metro regions. Today Transitland makes available stop locations, route lines, schedules, and other data from 2,626 transit operators in 59 countries. There’s an API and a range of different user interfaces.

Transitland is managed by Interline Technologies, a mobility and transportation technology firm founded by Mapzen alums. Interline provides custom Transitland integrations and consulting to transit agencies and private mobility providers (think e-scooters and ride-hail cars). Here’s one example of how we’re using Transitland open-source components to support the 30+ transit agencies in the San Francisco Bay Area.

Thanks to generous support from Amazon Web Services, Transitland’s APIs have been free to other startups, advocacy groups, and transit enthusiasts. Learn more about Transitland’s general governance and growth.

Thanks to a grant from the United States’ Transportation Research Board, Interline is currently expanding Transitland to support real-time transit data.

And finally, we’re ending 2019 with major technical changes: Transitland Version 2.0. The new version is faster (to handle an entire country’s worth of transit data in one import!) and distributed (to allow Interline’s client, other organizations, and enthusiasts to run their own deployments of Transitland tooling.

Metro Extracts

Daily OSM extracts are available from Interline, making it easy to get started with Valhalla, Pelias, or Tilezen with a small local area instead of the entire, very large OpenStreetMap planet file. Interline OSM Extracts are powered by the open-source PlanetUtils package, which wraps up a variety of OSM processing toolchains.

header image: Calendar, June; Japan; 1985-44-1-7, Cooper Hewitt, Smithsonian Design Museum

Mapzen is now a Linux Foundation project!

Tue, 29 Jan 2019 00:00:00 +0000

We have good news! Mapzen is now a Linux Foundation project!

Mapzen the company may have shut down, but the open source tools that the services were based upon live on and continue to grow. As we noted back in 2015, the very premise of Mapzen was longevity:

One of the core tenets of Mapzen is that all our software and data must be open. This has been our guiding principle since the beginning, and as sure as all things must end, we will follow this through to our end (hopefully a hundred or more years from now… By building everything in the open we can build for real permanence, not just the short-lived permanence of a hot brand in this turbulent space.

We’re humbled to see so many people continuing to use our tools, but we also understand that some organizations need more reassurance and clarity. The Mapzen Diaspora decided to work with the Linux Foundation because it allows us to give the projects and intellectual property a clear legal framework. Note that no one is going to work for the Linux Foundation, nor does this mean that the Mapzen APIs are coming back — these had considerable infrastructure costs — but it does give us a place for fundraising to support the AWS, S3 and Github bills for the Tilezen vector tiles and Who’s On First data.

The work accomplished in past year shows that there’s a strong future for what we were lucky enough to be able to create. The tools and data continue to be used and updated, with thousands of PRs added in the past year.

project	2018 closed PRs	stars
Pelias	1131	1,509
Valhalla	361	760
Tangram JS/ES	258	1,324/480
Tilezen	242	363
Who’s On First	49	n/a
Who’s On First data	149	217

We’ve also been making a few updates to this website — we are now directly pointing to each project’s documentation sources, and the blog posts are all up and continue to be interesting and useful. Also, come by and say hi on Twitter!

header image: Stick Navigation Chart (Marshall Islands), Before 1950, Cooper Hewitt, Smithsonian Design Museum

'/v1/search'ing for the Road Ahead

julian@mapzen.com — Wed, 31 Jan 2018 00:00:00 +0000

Image Credit: NASA/Mark Vande Hei

Here on the geocoding team at Mapzen, we’ve grown to love what we do. As soon as the Mapzen shutdown was announced, we started looking into ways where we could continue to work on geocoding and Pelias, the open source geocoder we’ve been building at Mapzen.

Just last week we announced that Geocode Earth is live and ready. It’s hosted geocoding, based on Pelias. It’s owned and operated by a new, independent company that we’ve founded.

We’ve done this for two reasons. The first, obviously, is to provide an easy migration path for anyone using Mapzen Search. Thanks to the open source commitment at Mapzen not just for the geocoder itself, but for the infrastructure around it, we’ve been able to do that in a very short time.

If you were using @mapzen and worried about the shutdown next week, https://t.co/cKYQ88lu99 is a perfect drop-in replacement. Literally just changed API keys and endpoints, and it worked. https://t.co/S2AyzCLXSI
— Andy Baio (@waxpancake) January 26, 2018

The second reason we launched Geocode Earth is to provide a way for us to continue to spend time growing the Pelias open source project.

In the days after Mapzen’s shutdown announcement we heard from people all over the world who were using and loving Pelias. Some we knew, but others were reaching out for the first time.

A great example is the NYC Planning Labs. They launched their own Pelias-based geocoder with features specifically to handle the nuances of New York City address data in under a week.

We started standing up @mapzen Pelias geocoder on Monday, @juliansimioni showed up to our sprint demo on Friday with words of encouragement and technical guidance! This is why we are #opensource at @nycplanninglabs. Thanks!
— Chris Whong (@chris_whong) January 13, 2018

The Planning Labs is already contributing back to the project in big ways. Many others have been doing so for years. We’ll continue to be right there with them.

In other words, we’ve listened to Velvet Underground’s first album, and we’re starting a band. We look forward to performing for, and with, all of you. See you at conferences, events, and of course, around the internet!

Long-term support for Mapzen maps, vector & terrain tiles

Tue, 30 Jan 2018 00:00:00 +0000

As Mapzen hosted services enter final shutdown we’re proud to offer long-term support for the maps, basemap designs, and the vector and terrain tile data that has powered Mapzen’s cartography. To enjoy the maps into the future, you’ll need to make a few updates to your projects.

This post details new long-term support endpoints for various map display related services and libraries, and provides information on accessing the new requester pays S3 bucket which features archival December 2017 vector tiles for the entire world.

See it all together at tile.nextzen.org/preview.html.

Importing a Basemap

All basemaps have been updated to use the new Nextzen sources, but you’ll need to import a new major version from a new web location.

For example, this Tangram scene file example imports Refill, adds Blue as a color theme, shows more labels, using the new Nextzen endpoints:

import:
    - https://www.nextzen.org/carto/refill-style/11/refill-style.zip
    - https://www.nextzen.org/carto/refill-style/11/themes/color-blue.zip
    - https://www.nextzen.org/carto/refill-style/11/themes/label-10.zip

The updated basemap scenes files also include many new icons and a few bug fixes, and are fully documented below.

Rendering maps with Tangram JS

The basemaps are written in the Tangram scene file syntax, and the Tangram JS library is available for import to your Javascript projects via a new Nextzen endpoint:

<script type="text/javascript" src="https://www.nextzen.org/tangram/0.14/tangram.min.js"></script>

Tangram JS, Tangram ES, and Tangram Play remain open source projects.

NOTE: The Mapzen.js wrapper around Tangram is not currently ported over to Nextzen. Stay in touch if that’s important to you.

Vector Tiles

If you were using Tilezen vector tiles via the Mapzen Vector Tile service you’ll need to get an API key (more on that later, learn more) and update your endpoints to the new Nextzen locations:

https://tile.nextzen.org/tilezen/vector/v1/{tilesize}/all/{z}/{x}/{y}.mvt?api_key=your-nextzen-api-key
https://tile.nextzen.org/tilezen/vector/v1/512/all/{z}/{x}/{y}.mvt?api_key=your-nextzen-api-key
https://tile.nextzen.org/tilezen/vector/v1/256/all/{z}/{x}/{y}.mvt?api_key=your-nextzen-api-key
https://tile.nextzen.org/tilezen/vector/v1/all/{z}/{x}/{y}.mvt?api_key=your-nextzen-api-key

We’re also making an entire global vector tile build, based on December 2017 data, available as a requester pays S3 endpoint:

vector-tiles-prod/20171221/{hashed-z-x-y-coordinate}.zip

To access the Tilezen vector tiles you’ll need to run the same middleware called Tapalcatl-py that powers the tile.nextzen.org service. Tapalcatl reads the hashed-x-y-z-coordinate metatiles (which group many tiles together in one ZIP archive) and extracts and returns the requested tile in the correct format (MVT, GeoJSON, and TopoJSON) and tile size (256 and 512 pixel).

Tapalcatl is similar to TileLive and other MBTiles based servers that vastly reduce financial costs associated with S3 gets and puts. This minimizes what would otherwise be billions of individual files, while introducing just a few milliseconds of latency at the edge. Amazon S3 GET pricing in us-east is currently $0.004 per 10,000 requests, and running Lambda and/or CloudFront on top of that adds only a little bit more.

Instructions for running Tapalcatl as an AWS Lambda function with Zappa are included in the project’s readme and enable a “serverless” (i.e.: low maintenance and generally cheap) way to serve vector tiles. Or join our mailing list if you’d like information about hosted instances or future data updates.

Terrain Tiles

If you were using Tilezen terrain tiles via the Mapzen Terrain Tiles service you’ll need to get an API key and update your endpoints to the new Nextzen locations:

https://tile.nextzen.org/tilezen/terrain/v1/{tilesize}/terrarium/{z}/{x}/{y}.png?api_key=your-nextzen-api-key
https://tile.nextzen.org/tilezen/terrain/v1/{tilesize}/normal/{z}/{x}/{y}.png?api_key=your-nextzen-api-key
https://tile.nextzen.org/tilezen/terrain/v1/geotiff/{z}/{x}/{y}.tif?api_key=your-nextzen-api-key
https://tile.nextzen.org/tilezen/terrain/v1/skadi/{N|S}{y}/{N|S}{y}{E|W}{x}.hgt.gz?api_key=your-nextzen-api-key

You can still access the Tilezen terrain tiles in 256 sized tiles via the existing Amazon public dataset S3 endpoints:

elevation-tiles-prod/terrarium/{z}/{x}/{y}.png
elevation-tiles-prod/normal/{z}/{x}/{y}.png
elevation-tiles-prod/geotiff/{z}/{x}/{y}.tif
elevation-tiles-prod/skadi/{N|S}{y}/{N|S}{y}{E|W}{x}.hgt.gz

The 512 and buffered 260 and 516 sized terrain tiles are made possible by running Zaloa as a Lambda function.

All together now…

The following versions of the Tangram basemap styles introduce support for Nextzen sources:

Bubble Wrap: 9.0.0
Cinnabar: 9.0.0
Refill: 11.0.0
Tron: 6.0.0
Walkabout 7.0.0
SDK Deault 1.1.0

New Nextzen basemap endpoints:

https://www.nextzen.org/carto/bubble-wrap-style/9/bubble-wrap-style.zip
https://www.nextzen.org/carto/cinnabar-style/9/cinnabar-style.zip
https://www.nextzen.org/carto/refill-style/11/refill-style.zip
https://www.nextzen.org/carto/tron-style/6/tron-style.zip
https://www.nextzen.org/carto/walkabout-style/7/walkabout-style.zip
https://www.nextzen.org/carto/sdk-default-style/1.1/sdk-default-style.zip

Icon Library: sprites on their own

The full Icon Library is available for use in your favorite mapping or graphics software as individual sprite images, independent of the basemaps. Currently only png format at 2x resolution is supported.

https://www.nextzen.org/carto/bubble-wrap-style/9/icons/{resolution}/{sprite}.{format}
https://www.nextzen.org/carto/cinnabar-style/9/icons/{resolution}/{sprite}.{format}
https://www.nextzen.org/carto/refill-style/11/icons/{resolution}/{sprite}.{format}
https://www.nextzen.org/carto/tron-style/6/icons/{resolution}/{sprite}.{format}
https://www.nextzen.org/carto/walkabout-style/7/icons/{resolution}/{sprite}.{format}
https://www.nextzen.org/carto/sdk-default-style/1.1/icons/{resolution}/{sprite}.{format}

For example:

Load the adit sprite in the Bubble Wrap style at 2x in png format:

https://www.nextzen.org/carto/bubble-wrap-style/9/icons/2x/adit.png

Thank you

It’s been a pleasure serving you the last few years! If you’d like to continue or otherwise sponsor the Tilezen open source project let us know :)

See you on the other side!

PS: Show me the icons

This release of the basemaps and stand-alone icon libraries triples the number of map icons, with 100% interoperability with the Maki icon set, the default openstreetmap.org cartography, the HOT icon set, and adds icons for all Tilezen POI kinds. (When there is a name discrepancy between Tilezen and Maki, for instance, we re-export the same artwork per name variant.) We also added nearly 200 shields, each with 5 character widths to support localized road shields in more than 40 countries.

The artwork is available in 3 different styles: Bubble Wrap full color, Walkabout silhouettes, and Refill’s greyscale. We were lucky to contract with the extraordinary Denis Sazhin of iconka to extend Geraldine’s original set of icons. The raw Illustrator files is also available under an open license, please use the artwork far and wide :)

674 map and ux icons
958 road shields
- 290 United States road shields for all 50 states, US Routes, US Interstates, county routes, US Forest Service highways and roads, and more.
- 668 international road shields, including in: Argentina, Australia, Austria, Bangladesh, Belgium, Brazil, Canada (including all provinces), Chile, China, Denmark, Finland, France, Germany, Greece, India, Indonesia, Iraq, Ireland, Italy, Japan, Luxembourg, Malaysia, Mexico, Netherlands, New Zealand, Norway, Peru, Poland, Portugal, Romania, Russia, Saudi Arabia, Singapore, South Africa, South Korea, Spain, Sweden, Switzerland, Thailand, Turkey, Ukraine, United Kingdom, and Vietnam.

Post Mortem III

randy@mapzen.com — Mon, 29 Jan 2018 00:00:00 +0000

A Letter Just to You, with Flowers

Early springtime lilacs. Shelter Island, NY

Hi there -

It’s been a while since the first two posts in our Mapzen Post Mortem series, but we’ve been busy. You’ve heard that Mapzen is shutting down, right? If you still depend on the services, please check our migration guide soon because February 1 is right around the corner… We’ll keep adding relevant substitutes to the guide over the next week so do take a look if you need to move.

Most at Mapzen are gone, but a few are here for the next week. One week! Not much time left. On the subject of time, we all have so little of it to begin with, so if you’re fortunate enough to have the choice, make sure you’re working on things that matter to you and the people you care about. If you’re reading this post, you probably have more choice than you realize: more choice and less time. Mapzen was lucky to be able to work on things that mattered to us for the last four-and-a-half years, and we wish the same for you.

On that subject, I love this Tweet and send it warmly to the Mapzen team:

listen. if you have the chance to create something with your friends, do it. do not wait. the chance may not come again. you will not find these people, as they are, again. as much as it may feel like it, opportunity doesnt last forever. it barely even lasts to begin with. go!
— jomny sun (@jonnysun) December 17, 2017

It was an honor and pleasure to build all this with you!

Post Blame, Less Mortem

As Mapzen shuts down, I’ve been thinking about legacy, continuity, and like everyone here, I’ve been thinking about what’s next. We’re in an interesting position where much of what we built will live on, and I wanted to talk more about that.

I also promised a reply to Kellan:

I’ve always been inspired by Mapzen’s commitment to build sustained long term improvements to the ecosystem – I hope they write more about those choices and how their work will live on because of them. https://t.co/4KbRYS2tM6
— kellan (@kellan) January 2, 2018

Kellan is the former CTO of Etsy, which popularized the blameless post-mortem, a way to understand and learn from failure in a constructive way. I find it amusing that our post mortem series includes both poems about death and also quotes from him, and I hope he doesn’t mind being dragged into this. So here goes…

Mapzen had a mandate to work only on open software and data, a very rare opportunity. Usually, companies that pursue open software or data draw a strategic line about what will be open and what will be proprietary. This makes a lot of sense; it was the case at my previous companies and will certainly be the case at my next company. But some kind of open strategy remains critical in mapping, a field dominated by incumbents who have invested billions of dollars and who use their advantages to restrict products across many industries. No one can compete in mapping at this point without embracing some collaborative efforts. At Mapzen, we went to the extreme and drew the line at near-zero proprietary elements, basically just this website and some operational code that didn’t make sense to publish.

Would I undertake this strategy again? For a large company trying to find flexibility in a restrictive area like mapping, I think the Mapzen strategy makes a lot of sense. If two or more large companies put significant, long-term resources behind what we were doing, the difficult problem of mapping would quickly be solved and companies could move the competition to a more interesting area: their products. For companies focused specifically on mapping and location services, the Mapzen strategy makes less sense. They need to find the right strategic line and hope that their proprietary elements remain stable.

In the end, I wouldn’t bet on any proprietary elements in mapping remaining stable.

How to Survive Death

Forsythia in bloom with later-spring lilacs behind. Shelter Island, NY

Early on at Mapzen, we were planning some new project. I asked a question that kind of scared me: “How would this survive the death of the company?” We weren’t close to death, and it scared me because who wants to hear that from the CEO? But it captured what we were doing in this special point in time and inspired our work. We had the chance to build everything to survive, and we did it. And we planned it all along the way to survive past our own existence.

Because we built everything like this, others joined in. They trusted that we were working on permanent infrastructure not tied to a specific company. Our gazetteer, routing engine, geocoder, tile server, and rendering engines were all set up as independent Github orgs with liberal licenses to ensure their durability. Others contribute to them, and their reach is wide. You’re using Mapzen’s legacy when you drive certain cars, play popular AR games, or interact with location data all over the Internet. Neither the shutdown of our hosted services nor the dissolution of the team can change this.

We were able to do this because we were pursuing the strategy outlined above: trying to provide flexibility in a restrictive area. Having others join as equals wasn’t a threat, it was a success to be celebrated. Someone once asked me, “What happens if the big mapping providers decide to open their data as well?” I replied, “We can all just go home!” Of course there would still be work to do, but Mapzen would have considered this a big success for our mission.

On January 2nd we announced our shutdown. December 2017 was one of the hardest months of my life, letting go one of the best teams in the industry and many good friends. As I came into the office on January 2nd and made the announcement at noon eastern time, I braced myself for a wave of internet snark and cynicism. Would people make fun of Mapzen for our tone and our all-open positioning? No. Instead, we all marveled at the overwhelming support and sadness from the community that loved our work and appreciated what we did. We spent most of the day at the top of Hacker News:

not quite the way I wanted @Mapzen to be number one on Hacker News pic.twitter.com/nq3ArEsBLO
— Burrito Justice (@burritojustice) January 2, 2018

No snark was detected. Our inboxes filled up. Twitter went wild. Corporate development teams from every company you might imagine - and many you might not - reached out to acquire us. Recruiters reached out to many on the team. The idea of hiring entire teams, whether they focused on search, navigation, or map rendering, became a hot topic. People learned that the tools and services we worked on could continue intact. January was a great month and gives me a lot of hope.

We knew that building technology to last would be valuable for the community, but over the last month we realized it’s also valuable to individuals on the team. It turns out that working entirely in public on big problems is good for your career. The Mapzen team has a high profile and we’re humbled by the hiring demands, and want to remind companies that a lot of our high profile is due to non-engineering work. If you’re at a company that needs to improve its own profile, consider looking at our marketing, community, support, and business teams.

Case Study: Mobility

Let’s look specifically at one of our products and how it will survive our death.

Years ago I arrived at MapQuest through a startup acquisition, and I began thinking about open strategies for larger companies. The problem was that data providers were too restrictive and expensive. Data providers who were formerly partners suddenly seemed more like competitors, and it was becoming harder to build new products without control over the entire map stack including data. I had worked with OpenStreetMap data previously, and wondered if that might be a solution: share the data with others and compete on products instead. MapQuest became the earliest corporate backer of OpenStreetMap, and did some work on Mapnik and Nominatim which would be used well into the future. I started to think about the elements that would later become Mapzen.

When I left MapQuest, I always regretted not giving more access to the excellent navigation engine the MapQuest team built. So when we had the chance at Mapzen, we hired that team in its entirety and built the Valhalla routing engine. I’m very proud of this work: it’s the first global, open, multi-modal (meaning transit!) navigation engine. It’s built for low-memory usage to make it possible to use in embedded systems, like a car. Valhalla makes accessible to all developers what was previously reserved for Google and Apple. Their work is freely available under the liberal MIT license.

I’m very happy that Mapbox hired this team and will continue the work around Valhalla in conjunction with their other navigation services. I’m confident that they will move this work forward, and pursue a sensible strategic line around open and proprietary. I look forward to see what they do here!

But the key to Mapzen’s legacy is that you are free to run Valhalla yourself. As we helped a well-known customer move off the paid services, we showed them how to do just this. They said, “I thought this was going to be a disaster, but actually it’s kind of awesome.” I agree!

On the subject of awesome, Drew Dara-Abrams and Ian Rees, who worked on mobility and transit at Mapzen, have announced a company called Interline Technologies. If you need help running Valhalla yourself, or want to work more closely with transit data, they’re available.

Brian Eno once said about the Velvet Underground that their first album sold few copies, but that everyone who bought one formed a band. I’d like to see something similar happen with Mapzen’s work - I want to see a lot of new companies pop up here with what we built. I’d like to think back on Mapzen as the Velvet Underground of mapping.

Other Case Studies

If you want to read more on the same topic, check out Aaron’s post about Who’s On First. Also we’ll announce some exciting developments for our search and vector tiles services before we close later this week, so watch this space. Spoiler alert: all of our work will continue to be supported and accessible to all.

What’s Next for Me

Only three or four more gigantic problems to solve and this should all work pretty well
— Randy Meech (@randyme) March 1, 2016

What’s next! Follow me above to see what’s next very soon (it will be awesome).

I’ve been working in open mapping for about ten years now. While Mapzen made a lot of progress, a few of the problems we wanted to solve still exist. To my mind, high quality points of interest (restaurants, bars, parks, etc) are still too inaccessible, and this area has not improved as fast as other areas of mapping. I want to focus more directly on this as Mapzen had a very wide focus. I’m also interested in exploring some of the decentralizing technologies that are emerging now; my sense is that they’re a good way to express the work we’ve already been doing with open software and data.

Let’s keep in touch! Thanks for working with us and thanks for caring!

Much love,
Randy

Lilacs in full bloom. Shelter Island, NY

Post Mortem II

randy@mapzen.com — Tue, 09 Jan 2018 00:00:00 +0000

This is a republication of Our Magna Carto from July 1, 2015, and the second installment of our post mortem series. Read the first installment here.

One of my favorite pages on Wikipedia lists the oldest companies in the world. While mapping is indeed an ancient profession, you won’t find any mapping companies there: the oldest companies make alcoholic beverages, knives and swords, or provide restaurant and hotel services. The oldest I’ve done business with is probably Stella Artois (1366).

The oldest company in the world was until recently Kongō Gumi, a Japanese construction company founded in 578. Yet even good old Kongō Gumi ran into difficulties in 2006 and was absorbed into a larger construction company, proving that nothing in this world is permanent. Even Shitennō-ji Buddhist temple, which it originally built in the 6th century, has been completely rebuilt a number of times over the centuries:

Shitennō-ji: younger than it looks. image credit 663highland, CC BY 2.5

I started in tech in 2000, and moved into the mapping industry in 2008. Not a long time ago by any means, although feeling like I’ve seen things before happens more frequently to me at this phase of my career. And what I’ve seen over the years is change and impermanence. Giants have tripped, hot startups have been acquired and fallen quite literally off the map. What has remained and been carried forward?

Seeking permanence

One of the core tenets of Mapzen is that all our software and data must be open. This has been our guiding principle since the beginning, and as sure as all things must end, we will follow this through to our end (hopefully a hundred or more years from now, like Kongō Gumi). By building everything in the open we can build for real permanence, not just the short-lived permanence of a hot brand in this turbulent space.

Our motto is “Start where you are.” This means many different things, but I often think about it when we’re looking at an area where open data is clearly inferior to proprietary, such as geocoding. A conventional company would abandon its open philosophy and strike a deal with a proprietary geocoding service. But to quote Google, we are not a conventional company: we will start where we are and insist on open data. This is part of our work with Code for America and OpenAddresses: we would rather spend $500,000 improving open address data than strike a quick deal to provide a better geocoder. We would rather do things that benefit all in the open.

Why? Because striking a quick deal robs from our collective future. It props up old, struggling business models, and prevents us and our developers from using and improving data as a community. It’s not that we don’t have the money to work with traditional map data providers. We won’t work with them because it would divert focus from the real work of improving open data, which will help everyone. We will never take harmful shortcuts, even if they make short-term business sense.

At the US State of the Map conference, I spoke on businesses and OpenStreetMap. At the end of the talk I invited the whole community - individuals, businesses, governments - to work together to get all of this data into the open. Doing this will allow us to build something that will outlast many things, and will certainly outlast the older proprietary models that we see collapsing around us now. Then we can build much more interesting things on top.

As companies and individuals, let’s all work on things that will be around for the next hundred years.

Mapzen Post Mortem I

Mon, 08 Jan 2018 00:00:00 +0000

The Seventh Seal, Danse Macabre

O dark dark dark. They all go into the dark,
The vacant interstellar spaces, the vacant into the vacant,
The captains, merchant bankers, eminent men of letters,
The generous patrons of art, the statesmen and the rulers,
Distinguished civil servants, chairmen of many committees,
Industrial lords and petty contractors, all go into the dark,
And dark the Sun and Moon, and the Almanach de Gotha
And the Stock Exchange Gazette, the Directory of Directors,
And cold the sense and lost the motive of action.
And we all go with them, into the silent funeral,
Nobody’s funeral, for there is no one to bury.

I said to my soul, be still, and let the dark come upon you
Which shall be the darkness of God. As, in a theatre,
The lights are extinguished, for the scene to be changed
With a hollow rumble of wings, with a movement of darkness on darkness,
And we know that the hills and the trees, the distant panorama
And the bold imposing facade are all being rolled away -
Or as, when an underground train, in the tube, stops too long between stations
And the conversation rises and slowly fades into silence
And you see behind every face the mental emptiness deepen
Leaving only the growing terror of nothing to think about;
Or when, under ether, the mind is conscious but conscious of nothing—

I said to my soul, be still, and wait without hope
For hope would be hope for the wrong thing; wait without love
For love would be love of the wrong thing; there is yet faith
But the faith and the love and the hope are all in the waiting.
Wait without thought, for you are not ready for thought:
So the darkness shall be the light, and the stillness the dancing.

Whisper of running streams, and winter lightning.
The wild thyme unseen and the wild strawberry,
The laughter in the garden, echoed ecstasy
Not lost, but requiring, pointing to the agony
Of death and birth.

                                          You say I am repeating
Something I have said before. I shall say it again,
Shall I say it again? In order to arrive there,
To arrive where you are, to get from where you are not,
    You must go by a way wherein there is no ecstasy.
In order to arrive at what you do not know
    You must go by a way which is the way of ignorance.
In order to possess what you do not possess
    You must go by the way of dispossession.
In order to arrive at what you are not
    You must go through the way in which you are not.
And what you do not know is the only thing you know
And what you own is what you do not own
And where you are is where you are not.

-T.S. Eliot from “The Four Quartets” - East Coker III

Mapzen Shutdown :(

Tue, 02 Jan 2018 00:00:00 +0000

Mapzen is now a Linux Foundation project!

There continues to be a future for the projects incubated at Mapzen and those tools and data continue to be used and updated in 2020 and beyond after our January 2019 transition to the Linux Foundation.

Now back to 2018… Unfortunately, we have some sad news. Mapzen will cease operations at the end of January 2018. Our hosted APIs and all related support and services will turn off on February 1, 2018. You will not be charged for API usage in December/January. We know this is an inconvenience and have provided a migration guide to similar services for our developer community. Our goal is to help as much as possible to ensure continuity in the services that you have built with us.

Fortunately, the core products of Mapzen are built entirely on open software and data. As a result, there are options to run Mapzen services yourself or to switch to other service providers.

Thanks for being with us over the past four years. We’re grateful that the open work we’ve done can continue outside of Mapzen and while we know this is a sad day, we’re optimistic about what’s next.

Preview image: Air Afrique Map of the World by M. Bourie from the David Rumsey Map Collection

Mapzen Alternatives

Tue, 02 Jan 2018 00:00:00 +0000

Mapzen will cease operations at the end of January 2018, and its hosted APIs and all related support and services will turn off on February 1, 2018.

Your two options are to switch to another hosted API that offers similar functionality, or to run your own servers with the open-source projects that powered Mapzen services. We list several options below to make this as easy as possible.

Alternate services and hosted APIs

Mapzen Vector Tiles and Cartography

For basemaps and vector tiles, you may want to consider:

Nextzen (A long-term support version of own own Tilezen)
Mapbox
OpenMapTiles
Thunderforest
MAPCAT

Long-term support for Vector Tiles is available via Nextzen through 2018 supported by Amazon Web Services, thanks to an award from the Earth on AWS Cloud Credits for Research Program. This includes hosted versions of Tangram JS and the map styles. The vector tile data is an archived December 2017 global build available in a requestor pays S3 bucket with associated serverless software.

Terrain Tiles will continue to be available as an Amazon Public Dataset.

Mapzen Vector Tiles and Cartography are based on the Tangram and Tilezen open-source projects. Read below for information on how to run these services locally.

Tangram scene files saved in a Tangram Play account can be saved to a GitHub Gist or downloaded locally, while Mapzen services are still running.

Mapzen Turn-by-Turn routing

For routing and navigation, you may want to consider these services:

Mapzen’s routing and navigation projects were backed by Valhalla, an open-source project. Read below for how to run Valhalla locally.

Mapzen Search

For search and geocoding, you may want to consider these services:

Geocode Earth (A new hosted version of our very own Pelias)
Nominatim
OpenCage Data
OSMNames
Mapbox Search API

Mapzen Search was backed by Pelias, an open source project. Read below for how to run this service locally.

Mobile

For Android and iOS, you may want to consider these third party SDKs:

Mapbox

In the event you decide to host a version of Valhalla and Pelias locally, you can continue to use our mobile SDKs we’ve built to interact with that service.

Mapzen iOS and Android SDKs, along with On The Road SDKs for Valhalla will be available at https://github.com/nextzen.

Our LOST project for Android will be available at https://github.com/lostzen/.

Pelias and tangram-es SDKs will continue to be available from their respective repositories.

Transitland

Transitland will continue as an open-data project at https://transit.land

Transitland’s server and data operations through 2018 will be supported by Amazon Web Services, thanks to an award from the Earth on AWS Cloud Credits for Research Program.

Continue to follow @transitland on Twitter for updates.

Who’s On First / Mapzen Places

Data (individual records) and source code for Who’s On First will continue to be hosted at https://github.com/whosonfirst, but whosonfirst.mapzen.com and the Mapzen Places API will go offline.

Who’s On First will continue (in a scaled back version, for now) at https://whosonfirst.org.

Data (individual records) and source code are at https://github.com/whosonfirst.
All of the data and code and blog posts have also been cloned to the Internet Archive.
The Spelunker will go offline, in the short term.
Batch processing and cascading edits will stop in the short term.
A limited version of the API that returns individual records will be at https://whosonfirst.org, with additional functionality growing over time (spatial queries, point-in-polygon, editorial tool, cascading updates).

Follow along at https://www.whosonfirst.org for updates.

Metro Extracts

Metro Extracts will no longer be updated, and the custom extract service will no longer be available. We are working on making a copy of the existing extracts available.

The Humanitarian Open Streetmap Team has created the HOT Export Tool that creates customized extracts of up-to-date OSM data in various file formats, including PBFs. Please consider donating to HOT, whether or not you use this service.

For non planet-sized PBF extracts, you may want to consider Geofabrik or the Overpass API.

Also, OSMaxx can be used for downloading OSM data.

Documentation and blog posts

Blog posts will be archived on https://medium.com/postzen.

Documentation will be available in each open-source GitHub repo:

Documentation section	Source location
Overview	https://github.com/mapzen/mapzen-docs-generator/tree/master/docs
mapzen.js	https://github.com/mapzen/mapzen.js/tree/master/docs
Tangram	https://github.com/tangrams/tangram-docs
Vector tiles	https://github.com/tilezen/vector-datasource/tree/master/docs
Search	https://github.com/pelias/pelias-doc
Mobility	https://github.com/valhalla/valhalla-docs
Metro Extracts	https://github.com/mapzen/metro-extracts/tree/master/docs
Terrain tiles	https://github.com/tilezen/joerd
Elevation	https://github.com/valhalla/valhalla-docs
Android SDK	https://github.com/mapzen/android/tree/master/docs
iOS SDK	https://github.com/mapzen/ios/blob/master/docs
Address parsing/libpostal	https://github.com/whosonfirst/go-whosonfirst-libpostal/blob/master/docs
Who’s On First	https://github.com/whosonfirst/whosonfirst-www-api/tree/master/docs
Cartography	https://github.com/tangrams/cartography-docs/

Run open-source versions of Mapzen services

Mapzen services are backed by open-source software projects, and use open data. Even with limited technical knowledge, you can run these yourself for metro-sized areas using a local web server or Docker containers.

Tangram and Tilezen (Cartography and Vector Tiles)

Documentation for Tangram is available at https://github.com/tangrams/tangram-docs.

tangram.js, tangram.es, Mapzen.js, Tangram Play are all open-source projects that can be run locally.

Basemap styles

Refill, Walkabout, Bubble-wrap, Tron and Cinnabar basemap styles will continue to be available in their repositories in https://github.com/tangrams and served via tangrams.github.io and via Nextzen.org.

Map data

While Tangram can import non-tiled GeoJSON and TopoJSON, vector tiles are what makes it interesting.

The Docker version of Tilezen can dynamically serve tiles from a Metro Extract or reasonably-sized PBF.

Tilezen in Docker https://github.com/tilezen/tileserver/blob/master/DOCKER.md

To run a map for a city or a region with no network dependencies:

Save mapzen.js / tangram.js, index.html and scene.yaml file in a directory
Extract a pyramid of tiles for area of interest using tilepack and save them in that same directory. (Note you don’t need to go beyond z15 tiles as they contain all the data in higher (>= 16) tiles, and Tangram overzooms.)

To download tiles from our hosted services before shutdown:

https://github.com/tilezen/tilepacks MAPZEN_API_KEY=mapzen-xxxxxx tilepack -122.51489 37.70808 -122.35698 37.83239 10 15 sf_mvt_10_15 --tile-size 512

Size and download speeds makes this viable for a city and perhaps a region. The example above generates a 30MB pyramid of MVT tiles, or 138MB of topojson, for San Francisco. For GeoJSON, SF is 67MB at z14, and 130MB at z15.

Serve tiles via a web server (localhost or hosted).
Create a scene file for Tangram. Note you will need to override the mapzen source in the scene file if you are importing a basemap.

sources:
    mapzen:
        type: MVT
        url:  localhost:8000/sf_mvt_10_15/all/{z}/{x}/{y}.mvt
        tile_size: 512
        max_zoom: 15

Valhalla (routing)

Documentation for Valhalla is available at https://github.com/valhalla/valhalla-docs.

Instructions on running Valhalla in Docker using data from a Metro Extract or PBF are at https://github.com/valhalla/docker.

Pelias (search)

Documentation for Pelias is available at http://pelias.io and https://github.com/pelias/pelias-doc.

Instructions on how to run Pelias in Docker with data from a Metro Extract or PBF and OpenAddresses data are at https://github.com/pelias/dockerfiles/blob/master/how_to_guide.pdf.

General instructions on running and installing Pelias and Elasticsearch: http://pelias.io/install.html

Mobile

Android and iOS SDK source code and LOST will available at https://github.com/lostzen

Preview image: World Emigration Map by Charles Minard from the David Rumsey Map Collection

WOF in a Box (part 2)

dan@mapzen.com — Fri, 29 Dec 2017 00:00:00 +0000

This is the second of the two part guide to running Who’s On First tools on your own hardware. You may want to read part 1 first.

Before we jump into installing the Who’s On First REST API and editor, let’s briefly take stock of what we have running from the first part of this guide.

We have a git repo of WOF administrative data
Incoming web requests get proxied by nginx
We can explore the data with our own installation of the WOF Spelunker

Some of the stuff we set up in part one, like Elasticsearch and nginx, are dependencies for this second part of the stack too.

These next parts we’re setting up run on a LAMP stack, each built with Flamework, a set of libraries and conventions originally created for the Flickr back-end. It still works just fine, although you can find some crufty bits if you poke around.

The two remaining web apps exist separately for now, but could very well be combined into a single codebase at some point in the future. The diagram shown above is rather simplified, and reflects a slight adjustment from the plan in the first part of this guide. Instead of API requests getting routed through /api, we will use the top-level / path.

The Who’s On First documentation we used to have at the top-level / path is still available online, if not on your NUC.

Getting the time right

Before we start, there is one minor change from how we set up the server in part one of the guide. During the Ubuntu server setup I mentioned that I just accept the defaults except for a handful important changes. One important change that I left out was to set the server’s timezone to UTC (Coordinated Universal Time).

We can easily revisit that step in the setup menu.

sudo dpkg-reconfigure tzdata

Then choose None of the above, then UTC. Then select Ok.

Let’s err on the side of caution and install some software to ensure the server has the most up-to-date time set.

sudo apt-get update
sudo apt-get install -y ntpdate

Now we can update the time from the National Institute of Standards and Technology.

sudo ntpdate time.nist.gov

With that settled, let’s continue.

Setting up the API

Okay, let’s begin setting up the API. I’m going to assume you’re already SSH’d into the NUC machine.

cd /usr/local/mapzen
git clone https://github.com/whosonfirst/whosonfirst-www-api.git

Much of the API’s setup process is automated, but if you look inside the Makefile you can inspect each step in detail.

Since we are proxying requests with nginx, we will set things up without SSL support (for now).

cd /usr/local/mapzen/whosonfirst-www-api
make setup-nossl

You may be prompted for your password for sudo. You will also be asked to confirm that you want to install unattended upgrades (Yes) and which updates to install (the default is fine). Finally, you will choose a MySQL root password (and confirm it), then enter it once more to set up the database schema.

When all of this finishes your API setup will almost be in working order. Some things you should know about how the stack is set up:

The API’s Apache configuration is symlinked from whosonfirst-www-api/config/whosonfirst-www-api-apache.conf
The database is called wof_api and is accessed by user wof_api
All passwords and tokens are stored in www/include/secrets.php
You will learn a lot by reading www/.htaccess
There’s also a great number of configuration tweaks contained in www/include/config.php (the convention is to set overrides in a separate file, we’ll get to that down below)

Configuring Apache

We still need to adjust a couple configurations in Apache, to make the API run on port 8888.

cd /usr/local/mapzen/whosonfirst-www-api
nano config/whosonfirst-www-api-apache.conf

Edit the first line to use port 8888:

<VirtualHost *:8888>

Next, edit Apache’s ports.conf.

sudo nano /etc/apache2/ports.conf

We’re instructing Apache to listen on 8888 and also 9999 (for Boundary Issues). We can get rid of the existing configurations for port 80 and 443. Those ports are handled by nginx.

Listen 0.0.0.0:8888    # API
Listen 0.0.0.0:9999    # Boundary Issues

Let’s restart Apache and check to make sure we can reach port 8888.

sudo service apache2 restart
curl -I http://localhost:8888/

If you see HTTP/1.1 200 OK then you are in good shape.

GitHub OAuth application

We use GitHub’s OAuth mechanism to authenticate users. We will need to set some things up at GitHub in order for that to work.

Go to GitHub’s Developer settings and navigate to OAuth Apps
Click the New OAuth App button
Enter an Application Name (WOF in a Box is a good choice)
For the Homepage URL, enter your server’s IP address, for example http://192.168.0.47/
The Authorization callback URL is similar to the Homepage URL, but with an /auth path suffix, e.g., http://192.168.0.47/auth
Click the Register Application button

Edit /usr/local/mapzen/whosonfirst-www-api/www/include/secrets.php and copy/paste the GitHub OAuth tokens using the following configurations. You’ll want to add this just before # the end.

$GLOBALS['cfg']['github_oauth_key'] = '(your client ID)';
$GLOBALS['cfg']['github_oauth_secret'] = '(your client secret)';

Secret hashes

While we’re looking at this file, check to see if the first three configurations are set. If they have values like You must set cfg.crypto_password_secret that means we need to set them still. These are used for things like cookie sessions, passwords, and CSRF-deterrent crumbs.

Generate a new secret hash for each of the three configurations, using the following script:

cd /usr/local/mapzen/whosonfirst-www-api
php bin/generate_secret.php
php bin/generate_secret.php
php bin/generate_secret.php

Those hashes get copied into secrets.php like this (your hashes will be different):

$GLOBALS['cfg']['crypto_cookie_secret'] = 'lfMi9sHoC2gnFKmMBgG8wUWqvqWO80eq';
$GLOBALS['cfg']['crypto_password_secret'] = 'afEp7fVgVe7TeO07jhud1y9opqgmhsmy';
$GLOBALS['cfg']['crypto_crumb_secret'] = 'u73GlqrDxqBOriFK921H8a0OsT8YZExz';

Configuring nginx

Before we can test everything, we will need to make one last adjustment to the nginx config. Instead of routing traffic from /api to port 8888, we will send everything from the top level / path.

sudo nano /etc/nginx/sites-enabled/whosonfirst

First, delete this block:

location / {
    try_files $uri $uri/ =404;
}

Then modify the API block to use / instead of /api. Notice there are two places that need editing, the location / part and the proxy_set_header X-Proxy-Path / part.

location / {
    proxy_pass http://localhost:8888;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Scheme $scheme;
    proxy_set_header X-Proxy-Path /;
}

Let’s restart nginx to make those configurations take effect.

sudo service nginx restart

Logging in with GitHub

There is one last configuration we need to make for all of this to run on plain vanilla HTTP. It’s not so much that we wouldn’t want to run on HTTPS, just that we would need a public hostname, or a self-signed certificate.

If file config_local.php exists, it gets loaded in to override the default values defined in config.php.

cd /usr/local/mapzen/whosonfirst-www-api
nano www/include/config_local.php

Add the following to disable HTTPS and, while we’re in here, use the right Elasticsearch index:

<?php

    $GLOBALS['cfg']['server_force_https'] = 0;
    $GLOBALS['cfg']['api_require_ssl'] = 0;
    $GLOBALS['cfg']['elasticsearch_spelunker_index'] = 'spelunker';

We don’t need a trailing ?> since it is optional, and can lead to unexpected side-effects.

Save your config_local.php configuration, and then load up your WOF in a Box’s IP address (something like http://192.168.0.47/) in a browser.

In the top-right corner, choose Sign In from the drop-down menu labeled You.

GitHub will ask for permission to authenticate you with the OAuth application you just set up. Click the green button to login.

You should get redirected back to the WOF in a Box homepage. If all goes well you should see your name appear in the top-right drop-down menu. You have been OAuth’d by GitHub.

Create an access token

Go to the to-right drop-down menu and choose Developer. From here you can explore some of the API methods available, and configure OAuth access.

Click on Your Access Tokens, then Create a new access token for yourself.

Give your token a name (“WOF in a Box” is a safe choice), permissions (“Read” is good), and expiration (“until I revoke it”).

Then you have to click one more I agree! checkbox and you should be set.

You should see a green “hooray” message, with a hexidecimal token for you to copy/paste.

For the sake of having a realistic looking example token, we’ll go with 87ffbaee33443e4b9b385606d6b9af28, but yours will be different.

Accessing the API

If you navigate to the API methods section of the Developer pages, you can try out some of the methods using the web-based API explorer.

Let’s try calling some API methods using curl.

curl -s "http://192.168.0.47/api/rest/?method=api.test.echo&access_token=87ffbaee33443e4b9b385606d6b9af28" | jq

You should see the following result:

{
  "method": "api.test.echo",
  "access_token": "87ffbaee33443e4b9b385606d6b9af28",
  "stat": "ok"
}

We can use a shell variable to store the IP address and access token, which should make the examples easier to copy/paste. Note, again, that your own IP and token will be different.

wof_api="http://192.168.0.47"
wof_token="87ffbaee33443e4b9b385606d6b9af28"

Now we should be able to test out the REST API. Let’s get a list of all the API method names.

curl -s "$wof_api/api/rest/?method=api.spec.methods&access_token=$wof_token" | jq "[ .methods[].name ]"

Let’s load up some info about a specific WOF record.

curl -s "$wof_api/api/rest/?method=whosonfirst.places.getInfo&id=102030609&access_token=$wof_token" | jq

You should see a record for Mumbai.

One last example, all the microhoods in San Francisco:

curl -s "$wof_api/api/rest/?method=whosonfirst.places.getDescendants&id=85922583&placetype=microhood&access_token=$wof_token" | jq '[ .places[]["wof:name"] ]'

You should take some time to explore the API documentation, and try out the standalone API Explorer desktop app. It has a settings pane that lets you change the API endpoint, so you can test out requests against your own WOF in a Box rig.

Important note: not all of the API methods will work for you at this point. We need some additional software for spatial queries and an additional Elasticsearch index for brands to work. Those can be subjects of some future WOF guide.

Setting up Boundary Issues

We are in the home stretch. The final part of the stack is our bespoke web-based WOF editor, called Boundary Issues. It has a similar setup process as the API.

cd /usr/local/mapzen
git clone https://github.com/whosonfirst/whosonfirst-www-boundaryissues.git

Like the API, much of the setup has been automated with a Makefile to run a sequence of shell scripts. And as with the API, we will set things up without HTTPS, since we’re proxying requests through nginx.

cd /usr/local/mapzen/whosonfirst-www-boundaryissues
make setup-nossl

You will be prompted for your sudo password, and then for the MySQL root password.

There is a second set of installation scripts that install everything required for offline tasks. The offline task functionality was designed to be an optional add-on to Boundary Issues, so it gets installed with a separate Makefile target.

make setup-offline

Most of the offline task dependencies are pulled in from modular single-purpose Flamework repos that take care of setting up each of:

You may notice the process of setting up gearman has an exceptionally large set of dependencies. This is the unfortunate consequence of apt-get install gearman not working in Ubuntu 16.04. Instead we install a few low-level dependencies and then build it from source. The process is automated by a shell script, so you shouldn’t have to type anything extra, but you will notice a whole lot more scrolling by.

Also, for some reason logstash doesn’t get validated during its installation, so you will need to manually confirm that you’re okay with that.

Once all that finishes, things will be set up similarly to the API:

The API’s Apache configuration is symlinked from whosonfirst-www-boundaryissues/config/whosonfirst-www-boundaryissues-apache.conf
The database is called boundaryissues and is accessed by user boundaryissues
All passwords and tokens are stored in www/include/secrets.php
Reading the bottom part of www/.htaccess will show you all the URL path routing logic
There’s also an awful lot of configuration tweaks contained in www/include/config.php (we’ll set some overrides in a config_local.php down below)

Configuring Apache (again)

Like with the API, we need to adjust a couple configurations to make Boundary Issues run on port 9999.

cd /usr/local/mapzen/whosonfirst-www-boundaryissues
nano config/whosonfirst-www-boundaryissues-apache.conf

Edit the first line to use port 9999:

<VirtualHost *:9999>

Also, add this additional config after the DocumentRoot part, to get the /boundaryissues path to work properly:

Alias /boundaryissues /usr/local/mapzen/whosonfirst-www-boundaryissues/www

We already took care of /etc/apache2/ports.conf earlier, so we can just restart Apache and check to make sure we can reach port 9999.

sudo service apache2 restart
curl -I http://localhost:9999/

If you see HTTP/1.1 200 OK then take a moment to celebrate quietly before we move on.

Indexing Elasticsearch (again)

One of the quirks about the Who’s On First ecosystem is that Boundary Issues and the Spelunker run on separate Elasticsearch indexes. This was intended to reduce the likelihood of accidentally screwing up live data in the Spelunker. We may combine them at some point, but for now it means we have some more schemas and indexing to take care of.

Let’s start with the schemas, yes plural. Boundary Issues uses three of them! I’m assuming we already did all the other Elasticsearch setup from part one of the guide.

cd /usr/local/mapzen/es-whosonfirst-schema
cat schema/2.4/mappings.boundaryissues.json | curl -X PUT http://localhost:9200/boundaryissues_20171229 -d @-
curl -X POST http://localhost:9200/_aliases -d '{ "actions": [ { "add": { "alias": "boundaryissues", "index": "boundaryissues_20171229" } } ] }'
cat schema/2.4/mappings.offline_tasks.json | curl -X PUT http://localhost:9200/offline_tasks_20171229 -d @-
curl -X POST http://localhost:9200/_aliases -d '{ "actions": [ { "add": { "alias": "offline_tasks", "index": "offline_tasks_20171229" } } ] }'
cat schema/2.4/mappings.audit_trail.json | curl -X PUT http://localhost:9200/audit_trail_20171229 -d @-
curl -X POST http://localhost:9200/_aliases -d '{ "actions": [ { "add": { "alias": "audit_trail", "index": "audit_trail_20171229" } } ] }'

That’s a whole lot of schemas and aliases. Basically what we did was set up three new indexes, aliased to make future upgrades easier:

boundaryissues aliased to boundaryissues_20171229
offline_tasks aliased to offline_tasks_20171229
audit_trail aliased to audit_trail_20171229

Now we can start the process of indexing the WOF records into the boundaryissues index. As with the Spelunker’s Elasticsearch index, this will take some time to finish, so you may want to use screen for it.

cd /usr/local/data/whosonfirst-data
wof-es-index -s . --index=boundaryissues -b

Let that run in the background while we finish up the rest of the Boundary Issues setup.

Secret hashes and GitHub OAuth (again again)

Okay, so a lot of what’s below may seem familiar after setting up the API. And I know you are eager to start editing WOF records. We are getting there!

Unfortunately, because the API and Boundary Issues are separate apps, we need to set up a second GitHub OAuth application to handle redirects properly.

Go to GitHub’s Developer settings and click the New OAuth App button
Enter another Application Name (Boundary Issues in a Box is a good choice)
For the Homepage URL, enter your server’s IP address, for example http://192.168.0.47/boundaryissues
The Authorization callback URL is similar to the Homepage URL, but with an /auth path suffix, e.g., http://192.168.0.47/boundaryissues/auth
Click the Register Application button

Now let’s open up secrets.php and insert those GitHub OAuth credentials.

$GLOBALS['cfg']['github_oauth_key'] = '(your client ID)';
$GLOBALS['cfg']['github_oauth_secret'] = '(your client secret)';

Next we’ll generate three new secret hashes, for the crypto hash configs:

cd /usr/local/mapzen/whosonfirst-www-boundaryissues
php bin/generate_secret.php
php bin/generate_secret.php
php bin/generate_secret.php

Those hashes get copied into secrets.php like this (your hashes will be different):

$GLOBALS['cfg']['crypto_cookie_secret'] = '9NBEy7d484AN3muSObRPsicoQIpjrhxQ';
$GLOBALS['cfg']['crypto_password_secret'] = 'HRoQz6jcuCB6lHhSmOhlWmfH3X7Nw9xa';
$GLOBALS['cfg']['crypto_crumb_secret'] = 'AxM2yCFsTwY9lVgyDUpJphrv4Qxf3gwI';

The pending folder

To accommodate the process of saving WOF records, we need to set up a www-writeable directory for incoming edits. It works kind of similarly to a Git index, with subfolders to keep the present edited state and a snapshot record of each edit.

The convention is to put the pending folder into the same structure as the other data repos.

sudo mkdir /usr/local/data/whosonfirst-pending
sudo chown www-data:www-data /usr/local/data/whosonfirst-pending

… and then symlink that into the Boundary Issues folder.

sudo ln -s /usr/local/data/whosonfirst-pending /usr/local/mapzen/whosonfirst-www-boundaryissues/pending

Let’s test things out

Now that we’ve set a whole bunch of Boundary Issues stuff… let’s maybe try it?

Go to http://192.168.0.47/boundaryissues/ in a browser (again, with your IP subbed in), and see if it loads up. You should see something similar, but not quite the same as, the API’s homepage.

Click on the same You menu and then Sign in with GitHub. You should see a similar OAuth confirmation page on GitHub, and then redirect back to the Boundary Issues homepage.

Once you get redirected back, you will have to accept some terms before you can continue.

Let’s search for a record, for example Brooklyn.

You want to click on the one that’s a Borough placetype.

Okay now we are getting somewhere, although…

… by default, new users don’t have permissions to edit records.

Add a user role

One last setup task: we just need to add the role admin to your user account and you should be all set.

Here’s how that works (again, swap in the email address you used to sign up for your GitHub account):

cd /usr/local/mapzen/whosonfirst-www-boundaryissues
php bin/add_user_role.php your.email@example.com admin

You should be prompted with a confirmation (press Y or just press enter).

There is actually another configuration we should set, inside a new config_local.php file.

cd /usr/local/mapzen/whosonfirst-www-boundaryissues/www/include
nano config_local.php

This will make Boundary Issues update the Spelunker’s Elasticsearch index, upon saving.

<?php

    $GLOBALS['cfg']['enable_feature_index_spelunker'] = 1;

And one more thing (I know, this is complicated), we need to make sure we have the offline tasks running, since they take care of saving stuff after you hit the Save button.

sudo /etc/init.d/gearmand start
sudo supervisorctl start all

Okay, now we should really be ready.

Edit a WOF record

Go back to the Edit Brooklyn page in Boundary Issues and reload it. You should be able to edit things now, yay!

Scroll down to the name section, and expand it open. Add the colonial name “Breuckelen” to the list of variant names (technically this should be under the Dutch language, not English, but one thing at a time).

Notice how the interface added a blue indicator next to the field? That helps you keep track of which things you’ve edited.

When you hit the save button, if all goes well, it should replace the list of changed properties with Saved. If it gets stuck on Saving… you may want to try again and keep an eye on the JavaScript console looking for error messages.

Finally, let’s try searching for updated name using the Spelunker.

Hooray, you made an edit and it showed up in the Spelunker. There is more to all of this that I won’t go into in this blog post. Right now all the changes you make end up in /usr/local/data/whosonfirst-pending, without modifying the actual WOF Git repos. If we wanted to actually upstream the changes to GitHub, we would need to set up a new user on the server (we tend to go with botsonfirst) with credentials to push up to GitHub. And that user would execute the script bin/save_pending.php on a cron job.

There is also the whole subject of pipeline tasks vs. offline tasks, but maybe you can ask me about that over a beer some day.

Bundler config

Okay one final set of config changes before we call it a day. The Spelunker has a very handy tool called the Bundler and we should get it working with our new WOF in a Box stack.

SSH into the NUC server and edit the flask config.

cd /usr/local/mapzen/whosonfirst-www-spelunker/config/
nano whosonfirst-www-spelunker-flask.cfg

We’ll make two edits here, the first enables the bundler feature:

enable_feature_bundler=1

The second will swap where the Spelunker looks for its WOF records from (once again, with your own IP swapped in):

data_root=https://192.168.0.47/data

There are a couple more edits to another config file: mapzen.whosonfirst.config.js

cd /usr/local/mapzen/whosonfirst-www-spelunker/www/static/javascript/
nano mapzen.whosonfirst.config.js

We are going to scroll down to the bottom and edit this part:

if (mapzen.whosonfirst.api) {
        mapzen.whosonfirst.api.set_endpoint('https://' + api_host + api_path);
        mapzen.whosonfirst.api.set_key(api_key);
}

Change it so that it looks like this (remember the access token we generated for the API?):

if (mapzen.whosonfirst.api) {
        mapzen.whosonfirst.api.set_endpoint('http://192.168.0.47/api/rest/');
        mapzen.whosonfirst.api.set_token('87ffbaee33443e4b9b385606d6b9af28');
}

Note that we changed the set_key function call to set_token. This is an important difference.

Okay, now we can try it all out!

First, restart the Spelunker for the config changes to take effect.

sudo /etc/init.d/whosonfirst-www-spelunker.sh restart

Next, load up the Spelunker page for Brooklyn (or Breuckelen). The URL should be something like http://192.168.0.47/spelunker/id/421205765/.

Now, scroll down until you get to the “Other” links, below the Hierarchy.

You should see a new link for Download descendants of Brooklyn. Click on that.

If everything is working, you should now be able to bundle and download all the descendants records of Brooklyn, using the NUC’s API and data. Let’s try getting all the Brooklyn neighbourhoods.

That’s all, folks

That was a whole lotta WOF in a Box, thanks for reading this far!