How Python is Used at CloudMade

This is a repost from my personal blog.

I’m starting a series on explaining how and why CloudMade uses Python. The following one explains why we ditched OSM’s stack in favour of in-house solution.

Intro

It’s been almost 2 years as CloudMade has ditched mod_tile and renderd as main rendering solution in favour of in-house solution. As the principle designer of the said alternative, I must say that this decision led to higher development pace. This article will try to cover the general architecture approach, reasons of decisions made and short comparison to other rendering alternatives.

Before The Switch

As some of you might know, CloudMade has its roots in OpenStreetMap and it was quite natural to adopt OSM’s software stack to have something to start with. But as CloudMade grew, the needs and requirements changed rapidly and the task of supporting and developing mod_tile became more of a burden, the decision to switch to more high-level language as the main was made. The language of choice was Python, due to its generous set of already existing spatial libraries (e.g. Shapely, GeoAlchemy, Mapnik bindings, etc), ease of deployment and its simpler support for cross-platform development. And, well, I knew it better than Scala, Ruby or Perl at that moment. Here goes a list of our tasks with mod_tile and renderd that we found easier to implement with Python:

Variable priorities
mod_tile has the notion of “dirty” and “general” requests, with dirty requests having lower priority and thus having the property of being rendered when there’s little-to-none on-demand rendering required. While this seems enough for most applications, it does has its warts, as it makes the priority system overall less general. What this means in practice, is that every time we need to add some special priority (i.e. in case we need to health-check system by forcing rendering) we get into adding quite a lot of code, rather than changing the “priority” property of the request. It might seem silly, but off the top of my head I can remember that we have at least 6 different priorities now
Replicating cache
When it comes to scaling rendering and serving of tiles, the simplest solution that comes to mind is adding more servers. It’s as simple, as pushing several links in web interface or even using automated process and Amazon Web Services API. But when you add new server with rendering stack installed you lose all the cache that has been on other servers and furthermore all the instances don’t share cache, which makes the cacheto use system less effective. There’re several solutions to this issue, each of them making use networking or database libraries, programming against which is tedious task in C (and C++).
Being tied to Apache
mod_tile is an Apache module, which makes it less interesting if you look at it from “commodity server” perspective. Having to program against a monster that is Apache, using its APR library is one giant leap into full-blown programmer depression. The autogenerated documentation make the matters even worse. And two last things about Apache are its comparatively slow serving of static files and complicated configuration scheme. One might say that Apache might be winning in other parts of comparison, but the things that have been mentioned were essential to our rendering services.

These were the main reasons to switch, as mod_tile and renderd didn’t seem like the right thing for CloudMade. Of course, there were a lot of others, more and less subjective reasons, but having even before mentioned ones, it was enough to seriously consider a switch.

The Switch

With all the warts of the existing system and requirements for the future in mind, we decided to move on with the new approach. There were several things to consider in our system:

Decoupling
This was our main goal — thoroughly decoupled system, where every part does one thing and does it good. This makes scaling much easier, but also incurs additional penalty on the amount of code, because of the need to write communication utilities. This also makes the system as a whole seem much more stable, as every other part of the system can work as a replacement in case of failure. Of course, the price is having network overhead and supervising system parts.
Handling styles
One of the main CloudMade web-services is the style editor, which gives ability to edit map styles using WYSIWYG technique. Handling thousands of Mapnik styles wasn’t something any existing system was prepared for, so unique way of doing exactly this had to be devised. Of course, this meant that style state in every part of the system had to be consistent at any given moment of time, making this even harder to accomplish.
Cache expiry
To minimize load on the system, as much cache as possible has to be available. But for rapidly changing OpenStreetMap data, having all tiles cached for month wouldn’t work and at the same time rendering all images on the fly would be an enormously heavy goal to accomplish. Whatever cache update approach is taken, unless there’s a hardware possibility to render maps on the fly, someone will be unhappy about cache expiry scheme.
Health monitoring and high availability
In order to meet requirement of having usable web services, one of the most important things to consider is having as high service uptime as possible. Without having health monitoring which knows about state of every part of the system the said objective is almost unreachable. Of course, the ideal can not achieved, but having a setup that covers at least 80% of the nodes would satisfy our needs.

The system that’s currently in use at CloudMade has been developed with exactly these goals in mind, with minor additions and subtractions along the way. To summarize, the goal was the system where every part has a maximum level of independency from every other while succumbing to the general goal of having fast and easily-deployed rendering stack.

To Be Continued

I’ll continue the talk about moving from mod_tile to our in-house system in follow-ups, where I’ll try to get into technical details, explain our shortcomings and issues that arised while developing.

Stay tuned.

January 17th, 2011 - Posted by Andrii Mishkovskyi in for developers, openstreetmap, technologies

Your Future is Customized – State of the Map Presentation from CloudMade

In July this year, 250 of the leading lights in community mapping and geo application development gathered in Amsterdam for the annual OpenStreetMap Foundation conference The State of the Map. Video and audio recordings form the three day conference are being processed one by one by a dedicated team of volunteers who recently published the video of Nick Black’s talk: “Your Future is Customized”.

The talk asks why most current users of geodata experience maps through one of two ways: in car sat navs, or online mapping portals and looks towards a future of app stores, specialized map data and geo applications that match the exact needs of consumers.

Enable your applications with CloudMade – Nick Black (CloudMade) from State of the Map 2009 on Vimeo.

Follow along with the slides from the presentation:

August 28th, 2009 - Posted by in api, for developers, geodata, openstreetmap, products

Building the Map

Whether you saw it at State of the Map or in the posting below, many of you have seen and commented on Mapzen, CloudMade’s upcoming web-based mapping tool.  We’ve gotten a lot of good feedback from people on what they like and what they’d like to see.  I am pretty excited about it.  We hope that it will introduce more people into the OSM community, accelerate the growth of the map worldwide and also improve the quality of the map by making it easier to add the attributes that sometimes aren’t seen but really make the map work.

CloudMade is committed to helping the OpenStreetMap community build the best map in the world.  That goes right to our roots and our founders’ passion for OSM.  With Mapzen we are building the next generation of mapping tool, that will greatly improve mapping for existing mappers and make it easier for new mappers to get started with mapping. This will help ensure the best of community mapping, rapid response and map quality.  We think Mapzen is a great step in that direction and offer it to the community in that spirit.

As many of you know, CloudMade’s Community Ambassadors have been working very hard to build the map through outreach to local groups, holding mapping parties and working on behalf of OSM at trade shows and conferences. This has helped spread the word on OSM in selected regions in the US, but in the process we have learned that we can get a wider reach by working with larger national and local community groups. Based on this, we have made the decision to dissolve the ambassador team and build a team that will focus on partnering with national and local groups, support finding and importing public data and communicating local success stories.

Going forward, CloudMade will be actively working with national and local groups to promote OSM and community mapping.  We are working with the Geo-Georgia group on a very cool project to map the city of Atlanta in a weekend (September 11-13, if you want to help). The goal is to make Atlanta a showcase city for the OSM map and there’s a great team of people working on that. You’ll hear more about that as the time gets closer.

Our goal here is simple:  Build the Community.  Build the Map. We hope that our efforts can make a difference, but the community will build the map.  If you know of ways you can help us in that, or ways we can help you, let us know. Send a message to community@cloudmade.com.

July 23rd, 2009 - Posted by Marc Prioleau in openstreetmap

CloudMade releases Style Editor 1.1

Great news for Style Editor users: we’ve launched an updated version of Style Editor (v1.1) that makes discovering, sharing and creating new map styles even easier!

Style Editor is CloudMade‘s online cartography tool for creating customized map styles.  Application developers can use Style Editor to quickly create their own map styles that give applications a unique look & feel.  Style Editor gives you complete control over which objects are displayed on the map, and how they will appear on your map at each zoom level.  Developers use CloudMade’s Style Editor to design customized maps built for a particular use.   For example, if you were building a cycling application that helps users get from point A to point B by bicycle, you’d want to highlight the cycle paths and roads better suited for cyclists while de-emphasising main roads and motorways.

Style Editor now has a style gallery for discovering new map styles

Style Editor now has a style gallery for discovering new map styles

The most important improvements in Style Editor 1.1 is the addition of the style gallery.  Style gallery will allow you to quickly discover public map styles created by CloudMade and our community.  Style Editor also allows for private styles available only to the author, which you can view by pressing the My Styles tab on the Style Editor toolbar.  By default the map styles are sorted by popularity.  You may also chose to look for CloudMade styles which were created by professional cartographers including some by Stamen Design by clicking on the CloudMade tab.  Or you can browse the featured styles (Featured tab) which showcase a mix of CloudMade and community styles that our cartographers deem fantanstic!

The Style Editor tool bar allows for quick discovery of public maps styles for editing or direct use

The Style Editor tool bar allows for quick discovery of public maps styles for editing or direct use

Another cool new feature in style gallery is the ability to sort map styles based on color.  To the top of the Style Editor toolbar is a Color Filter control.  This control will filter map styles in the gallery based on where you drag the Color Filter bar.  The default setting on the far left shows all colors.  By dragging the Color Bar towards the right, you can select only the map styles with a particular shade of yellow, for example.  In this way you can find map styles that suit the look and feel of your website or application.  Don’t forget to reset the Color Bar back to the far left when your done so that you can see all the styles again.

Each map style in the style gallery shows the style name, author and style ID.  The style ID comes in handy when coding with CloudMade’s API libraries.  You can quickly change the Style ID and see a completely different map in your app!  The search bar in the far right of the tool bar allows for style searches based on style name, author or style ID.

The last new feature we wanted to talk about is the Style RSS feed.  You can now subscribe to all new style updates via RSS.  This is a super easy way to get all the latest updates to Style Gallery via RSS.

Style Editor is open to the public and available to use as a free service.   So go ahead and start creating your own new map styles and publish them to Style Gallery so the world can see and use them!

Style Editor Team @CloudMade

Creating new map styles is simple with CloudMade's Style Editor

Creating new map styles is simple with CloudMade’s Style Editor

July 16th, 2009 - Posted by Cragg Nilson in cartography, for developers, openstreetmap, products, style editor

Mapping with the kids

I took my children to a mapping party last weekend in Milton Keynes (UK).  My daughters are 4, 6 and 8 years old.  We had three GPSs, three scooters, 2 cameras and a pad of paper.

My girls had mapped with me before near our home, but always incidental mapping (i.e. we were going on a bike ride or walk and did a bit of mapping).   This was the first time we would have an assigned area to look at.

So, for those of you who want to map and have kids you don’t want to neglect, here is what I learned:

  • Bring some toys (in my case each girl had a back pack and a nintendo and/or ipod).  The times you spend talking to other adults and/or working on the computer are pretty dull to watch.
  • Bring a GPS for each person…  The kids love mapping and will happily run down that trail you want mapped and back (when your legs are too tired).  Each kid wants to hold a GPS and they make you alot more effective with limitless energy.
    [Note to tools developers, the ability to overlay multiple traces would be very useful... ]
  • Map something they can enjoy…  So, we got to do Campbell park in Milton Keynes…  They could play, run, have fun…  Sitting in a car while daddy did the parking lots and bus stops on the way over was not much fun for them…
  • Two 2 hour sessions is alot of mapping for a kid…  mine managed two ninety minute sessions with a longish lunch in the middle.  Plan for more down time
  • The kids will notice things that should be mapped that you might not…  Mapping stuff kids find important, means making a map that is meaningful to families…
  • Finally: Mappers are very nice to kids :) Everyone seemed a bit suprised, but pleased to see them…

Overall, if you map and have kids…  Take them to a party.  They will love it, you will see new things to map through their eyes and they can make you more effective while on foot (in particular).

Mapping with kids in Milton Keynes

Mapping with kids in Milton Keynes

May 19th, 2009 - Posted by Jim Brown in openstreetmap

A Week of Changesets

One of the new features in OpenStreetMap’s API 0.6 is changesets, which track the bounding box of the edits associated with them. You can check out how useful this feature is at the changeset browser page, or the user edits page. But it isn’t easy to get an overview of editing activity directly from the browser page, so here’s an animation showing all the changesets as they’re created (red) and closed (green) and fade away. You can download the full version here.


Kudos to our German contributors for their relentlessly energetic pace!

May 8th, 2009 - Posted by Matt Amos in openstreetmap

New OSM db server “smaug” – First Pics!

Let’s cut to the chase – you all want to see pictures of the new OpenStreetMap core database server, and here it is:

Smaug - the new OSM DB server

It’s got blue LEDs on the front. It must be good.

For the techies amongst us, you’ll probably want to know some juicy specs – 2 Quad Core Intel Xeon E5420 processors, 32 Gigabytes of DDR2 RAM, 10 450Gb 15k SAS disks for the database (and another 2 73Gb disks for the operating system). All in all, a worthy machine to sit at the heart of OpenStreetMap, storing all the geodata and powering all kinds of things. You can see more pictures of the machine as it was being unboxed.

Thanks go of course to everyone who contributed to the recent OpenStreetMap foundation donation drive, and we can’t wait to see it up and running with the next version of the OSM API in a few weeks time. Once Matt has finished deafening everyone in the CloudMade London Office (as he “tries it out”) it’ll be moved to its permanent home in UCL to join the rest of the OSM servers.

March 27th, 2009 - Posted by Andy Allan in news, openstreetmap

A Summary of the Future of Mapping

Over the last few days we’ve launched the CloudMade Developer Programme in front of packed audiences in San Francisco and London.  There was a real buzz at both events – a feeling that the tools and services we’ve been working on are going to make a real difference to developers.

There’s a lot of anticipation at CloudMade about the applications that can be built using our mapping APIs.  If you’re looking for inspiration, you need a quick recap, or you couldn’t make it to the events, read on to find out more.

Cragg Nilson presents the agenda.

(more…)

February 16th, 2009 - Posted by Nick Black in api, customers, for developers, openstreetmap, products, talks

(Nearly) Live Tiles

Waiting to see your edits is boring. Worse than that, it delays the sense of satisfaction that comes with seeing your contributions and reduces the addictiveness of mapping.

At CloudMade, we’ve been running prototype (nearly) live updating tile servers internally for a while, but we thought it would be nice to share these with the community. This sandbox tile server is updated from the minute diffs and runs approximately 10 minutes behind the main database, due to the time necessary to dump the diffs, download and import them. You can see the results here by switching back and forth between the minutely layer and the weekly layer. (Thanks to Shaun for the continuous stream of edits!)

If you’re looking at an area and you think that it should be updated, try hitting the permalink and refreshing a few times, as your browser may be showing you cached tiles.

Please feel free to use these tiles, but remember that this is an experimental sandbox – please don’t download tiles in bulk. The server will be slow under high load so, please, if you see a warning message then come back a few minutes later.

Enjoy!

January 23rd, 2009 - Posted by Matt Amos in openstreetmap

An Animated Year of Edits on OSM

We were inspired by ITO’s “a year of edits on OSM“, but thought that it needed a little extra something – animation!

 

(more…)

December 15th, 2008 - Posted by Matt Amos in openstreetmap

Next Page »