CloudMade » openstreetmap

How Python is Used at CloudMade

mishok13 — Mon, 17 Jan 2011 12:09:02 +0000

This is a repost from my personal blog.

I’m starting a series on explaining how and why CloudMade uses Python. The following one explains why we ditched OSM’s stack in favour of in-house solution.

Intro

It’s been almost 2 years as CloudMade has ditched mod_tile and renderd as main rendering solution in favour of in-house solution. As the principle designer of the said alternative, I must say that this decision led to higher development pace. This article will try to cover the general architecture approach, reasons of decisions made and short comparison to other rendering alternatives.

Before The Switch

As some of you might know, CloudMade has its roots in OpenStreetMap and it was quite natural to adopt OSM’s software stack to have something to start with. But as CloudMade grew, the needs and requirements changed rapidly and the task of supporting and developing mod_tile became more of a burden, the decision to switch to more high-level language as the main was made. The language of choice was Python, due to its generous set of already existing spatial libraries (e.g. Shapely, GeoAlchemy, Mapnik bindings, etc), ease of deployment and its simpler support for cross-platform development. And, well, I knew it better than Scala, Ruby or Perl at that moment. Here goes a list of our tasks with mod_tile and renderd that we found easier to implement with Python:

Variable priorities: mod_tile has the notion of “dirty” and “general” requests, with dirty requests having lower priority and thus having the property of being rendered when there’s little-to-none on-demand rendering required. While this seems enough for most applications, it does has its warts, as it makes the priority system overall less general. What this means in practice, is that every time we need to add some special priority (i.e. in case we need to health-check system by forcing rendering) we get into adding quite a lot of code, rather than changing the “priority” property of the request. It might seem silly, but off the top of my head I can remember that we have at least 6 different priorities now
Replicating cache: When it comes to scaling rendering and serving of tiles, the simplest solution that comes to mind is adding more servers. It’s as simple, as pushing several links in web interface or even using automated process and Amazon Web Services API. But when you add new server with rendering stack installed you lose all the cache that has been on other servers and furthermore all the instances don’t share cache, which makes the cacheto use system less effective. There’re several solutions to this issue, each of them making use networking or database libraries, programming against which is tedious task in C (and C++).
Being tied to Apache: mod_tile is an Apache module, which makes it less interesting if you look at it from “commodity server” perspective. Having to program against a monster that is Apache, using its APR library is one giant leap into full-blown programmer depression. The autogenerated documentation make the matters even worse. And two last things about Apache are its comparatively slow serving of static files and complicated configuration scheme. One might say that Apache might be winning in other parts of comparison, but the things that have been mentioned were essential to our rendering services.

These were the main reasons to switch, as mod_tile and renderd didn’t seem like the right thing for CloudMade. Of course, there were a lot of others, more and less subjective reasons, but having even before mentioned ones, it was enough to seriously consider a switch.

The Switch

With all the warts of the existing system and requirements for the future in mind, we decided to move on with the new approach. There were several things to consider in our system:

Decoupling: This was our main goal — thoroughly decoupled system, where every part does one thing and does it good. This makes scaling much easier, but also incurs additional penalty on the amount of code, because of the need to write communication utilities. This also makes the system as a whole seem much more stable, as every other part of the system can work as a replacement in case of failure. Of course, the price is having network overhead and supervising system parts.
Handling styles: One of the main CloudMade web-services is the style editor, which gives ability to edit map styles using WYSIWYG technique. Handling thousands of Mapnik styles wasn’t something any existing system was prepared for, so unique way of doing exactly this had to be devised. Of course, this meant that style state in every part of the system had to be consistent at any given moment of time, making this even harder to accomplish.
Cache expiry: To minimize load on the system, as much cache as possible has to be available. But for rapidly changing OpenStreetMap data, having all tiles cached for month wouldn’t work and at the same time rendering all images on the fly would be an enormously heavy goal to accomplish. Whatever cache update approach is taken, unless there’s a hardware possibility to render maps on the fly, someone will be unhappy about cache expiry scheme.
Health monitoring and high availability: In order to meet requirement of having usable web services, one of the most important things to consider is having as high service uptime as possible. Without having health monitoring which knows about state of every part of the system the said objective is almost unreachable. Of course, the ideal can not achieved, but having a setup that covers at least 80% of the nodes would satisfy our needs.

The system that’s currently in use at CloudMade has been developed with exactly these goals in mind, with minor additions and subtractions along the way. To summarize, the goal was the system where every part has a maximum level of independency from every other while succumbing to the general goal of having fast and easily-deployed rendering stack.

To Be Continued

I’ll continue the talk about moving from mod_tile to our in-house system in follow-ups, where I’ll try to get into technical details, explain our shortcomings and issues that arised while developing.

Stay tuned.

How Python is Used at CloudMade

New Mapzen Updates Make Editing Smoother and Faster

pauljarratt — Wed, 10 Mar 2010 21:25:36 +0000

We’ve taken on board a lot of feedback and made lots of fixes and updates to Mapzen which we think will make a significant difference to your editing experience. They’re aimed at helping you map more by providing even greater flexibility within Mapzen.

Here’s what we’ve added:

1. Easier to work in areas with lots of land use

It’s now a lot easier to work more accurately with land use and other areas, with the ability to turn their filling on or off to provide a clearer view of what you are putting a perimeter around when viewing the satellite imagery or background images.

2. Easier to align roads to satellite imagery

A new Hide/ Show Elements button has been added to the bottom of the editing window allowing you to quickly turn on / off elements to better see the background objects. This is really useful when trying to align roads to satellite imagery.

3. More control when adding lines and areas

A much requested feature that you’ve all been waiting for. Now you can press ESC to cancel any editing made to shapes of lines and press ‘Enter’ to end editing with line types. This is really useful when adding new roads.

4. Mac Users: no more annoying zoom in / zoom out

If you have a magic mouse from Apple or use a MacBook touchpad you might have experienced sudden jumps in zoom level. If you have suffered this problem, you can now disable the ‘scroll’ function to prevent the map from accidentally zooming in or out.

We hope you’ll find these updates useful – please continue to email us feedback, it’s incredibly useful to help us evolve and improve Mapzen. Here’s how you can get in touch

Leave a post in the Mapzen forums
Report a bug on the Mapzen issue tracker
Send us a mail to mapzen@cloudmade.com
Get in touch with Nick, Yuliya or Anton through Mapzen

The next Mapzen blog post will look at some new features coming up in the April release of Mapzen Beta. Until then, if you don’t have a Mapzen account already, you can sign up for one and start using Mapzen here – happy mapping!

Mapzen Updates and Sneak Preview of the Next Release

nick — Sat, 20 Feb 2010 18:10:28 +0000

There’s been a lot of work going on behind the scenes to make Mapzen, CloudMade’s family of easy to use OpenStreetMap editing tools, even easier, even more useful and even more fun to use. So what’s new?

Mapzen POI Collector 1.1 – Now in the App Store

Mapzen POI Collector is CloudMade’s smash hit iPhone mapping app that lets you easily and quickly add a huge range of places of interest to OpenStreetMap. The latest version includes numerous updates and improvements – all based on feedback we received from Mapzen users.

Mapzen POI Collector 1.1 lets you map faster and smarter:

Choose from over 45 new POI types based on user requests, including ‘Grit Bins’ for winter mapping, ‘Traffic Cams’ and many more
All POI icons are full color for a more vibrant mapping experience
Quickly find the POI type you are looking for with a slick new search feature
Add even more detail to the map with new attributes including disability access, seating capacity and religious denomination

Other bug fixes and enhancements:

GPS location functionality is faster and more reliable
Faster map loading with an enhanced map engine

You can get the latest version for free from the App Store. Existing users can give feedback and help make Mapzen POI Collector even better.

Share your mapping adventures with the world

With the latest release of Mapzen, mappers can share their mapping adventures with friends and family or publish them for the whole world to see. An exciting cross between mapping and blogging, mappers can publish their mapping activity feed – which includes details of recently mapped areas as well as posts made by the user. Posts can range from short, Twitter style updates to longer, blog style entries that let mappers share their experiences. Mappers’ activity feeds are tagged with the location of their posts which means they can be used by other applications – like this demo shows.

Stay up to date with your friends’ mapping activities

Mapzen now makes it easier than ever before to keep up to date with updates from your mapping network. Your Dashboard now includes a mapping activity feed from all of your mapping friends – so you know who’s been mapping what, where.

Map faster, worry less with the new Undo/Redo history panel

Mapzen’s undo and redo feature lets you map without having to worry about messing up anyone else’s work. Mistakes can be undone with the click of a button and now you can view a full history of your edits – making it easy to jump back to the exact point before it all went wrong.

Go to Mapzen to try out all the new features for yourself!

Coming Soon – What’s Next for Mapzen?

The next release of Mapzen Beta is just a few weeks away. We’ve been listening to your feedback to make it a better, faster, more fun tool to use. Mapzen Beta Belz includes these top requested features:

Natural features like lakes, forests or landuse zones and other areas can be tricky to map, with lots of overlapping layers getting in the way of each other. The next release of Mapzen gives mappers control over the appearance of natural features and other landuse areas, making it easier to map in places where lots of different types of landuse areas are required
Faster loading of Mapzen will let you quickly start mapping sessions and swap between the “View” and “Edit” modes
Ending line or shape features can be tricky in Mapzen – especially when you want to end a line that you’ve just added a point to. Keyboard controls in the next version of Mapzen will make it easy to end line and shape drawing.
Some Mac users have experienced problems with the scroll to zoom functions in Mapzen, especially when using an Apple Magic Mouse or MacBook or MacBook Pro touchpad. These problems will be fixed in the next release of Mapzen

If you can’t wait to try out Mapzen Beta Belz you can join the Mapzen Community Panel to have the chance to try out the latest features before they hit the web. Community Panel members get access to unreleased features and give feedback that makes Mapzen better for everyone.

Waiting for a feature that we haven’t talked about?

If you didn’t see the feature you want in the list for Mapzen Beta Belz, don’t worry. The features listed below are planned for the April release of Mapzen. We’d love to hear what you think about them. To provide feedback, comment on this post or send a mail to mapzen@cloudmade.com or check out the Mapzen forums.

Merging lines is a top requested feature that will appear in the April release of Mapzen
The presets panel is starting to get a bit crowded with all of the line, point and shape types that can be added. A handy search feature will make finding the right object easy
Hint strings will help new Mapzen users get started quickly and help existing users get more out of Mapzen
Mapzen can be slow sometimes – but don’t worry – we’ve got a few tricks up our sleeves that will make mapping a lot faster

Tell us what you think

The Mapzen team want to know what you like and what you don’t like about Mapzen. There are loads of ways to get in touch with us:

Leave a post in the Mapzen forums
Report a bug on the Mapzen issue tracker
Send us a mail to mapzen@cloudmade.com
Get in touch with Nick, Yuliya or Anton through Mapzen

Making Mapzen Even Easier and More Fun to Use

nick — Mon, 04 Jan 2010 11:41:18 +0000

Its just over a month since we launched Mapzen – the easy to use OpenStreetMap editor. Since then, the intuitive user interface and focus on ease of use has caught the attention of users and the media (see New Scientist, Mac World, ZDNet and our own Press Zone). Over the holidays we released a set of updates to Mapzen that address many of the comments early users have made. Here’s a summary of the updates that were made in the “Mapzen Beta Uhniv” release, on the 24th December 2009:

Bug fixes

One of the biggest problems reported by some Mapzen users was with saving data – the annoying “Oops – a big error has occurred” problem was our highest priority for this release. We tracked down all of the known causes of this problem and they are now be fixed for good. We also added an automatic reporting system – so Mapzen will tell us if it incurs any problems when saving or editing data. Other bug fixes included more reliable loading of satellite imagery and more reliable rendering of lines – stopping the problems that occasionally made lines disappear.

Giving you more space to work

No-one likes working in a small, constrained screen space, so we increased the size of the work area in Mapzen:

More objects to edit

There are thousands of different types of map features in OpenStreetMap – each often with several variations and combinations. WIth Mapzen we boil all of this choice down to the features and attributes you need for mapping. With each release of Mapzen we’ll add some new map features, modify others and even remove features or attributes that user do not find useful. In this release of Mapzen we added loads of new features, including:

New icons for a whole load of exciting features including funicular railways, gondolas and drag lifts – just in time for the skiing season
New icons for boundaries including city walls, fences, hedge lines and gates
New landuse types including scrub
New POIs including drinking water and recycling

New features make map editing in ski resorts easy

Another hugely requested feature was clearer one-way arrows for roads, also updated in this release:

Coming up next

The next release of Mapzen will focus on improving usability – particularly the way in which users interact with roads. For example, we know you find it annoying to have to click once to select a node and again to drag it when you are re-shaping a road, so we’ll be updating this behaviour so that to make it easier and quicker for you to drag and re-align roads – something that all the TIGER editors out there spend a lot of time doing.

We’ve also heard that users in areas with a landuse shapes can find it hard to edit roads. To solve this problem we’ll be adding controls that let you turn off landuse – making it far easier to edit roads in these areas. We’ll post some screenshots as soon as we have working internal versions.

What else?

There are lots of exciting features in the queue for Mapzen from speed enhancements to improved hints to merging of ways to easier selection of map features. We’ll be releasing a poll in the next few weeks that will let you choose which features you want to see implemented first – so stay tuned.

In the meantime you can leave feedback about Mapzen or report bugs at the Mapzen Issue Tracker or you can get in touch via email.

Mapzen POI Collector is now in the App Store

nick — Thu, 26 Nov 2009 18:10:57 +0000

CloudMade’s Mapzen POI Collector – an iPhone application that makes it really easy to add places of interest to OpenStreetMap is now available for free download from the App Store. Click here to find out more.

Union Square, San Francisco in Mapzen POI Collector

Mapzen POI Collector’s fast, easy to use POI menu

Stay tuned for more Mapzen news coming soon.

Calling all OpenStreetMappers! Get Ready for Mapzen – editing and adding POIs just got easier

emzwmz — Wed, 07 Oct 2009 18:23:01 +0000

We’ve just released the Alpha version of Mapzen, our OpenStreetMap community editing tool which lets anyone easily contribute to the map, and are actively recruiting people to help test and provide feedback on it. To be a part of this program please sign up here.

We’re only able to handle a certain number of Alpha testers so if we’re unable to accept you to the program don’t worry, Mapzen Beta will be available for everyone later on this year.

CloudMade has developed Mapzen to help reduce the time it takes to create and edit OpenStreetMap so you can spend more time doing what you like best, mapping. It’s the first community mapping tool that let’s anyone easily contribute to OpenStreetMap.

Here’s a quick overview of some features:

• Intuitive user interface and workflow dramatically reduce editing time
• Graphical menus mean no more searching for obscure tags
• Multi level undo and save modes mean no more fear of ruining the map
• Built with social networks in mind so sharing mapping experiences with Facebook and Twitter friends is simple
And much more….

We’ve also been busy working on Mapzen POI Collector for iPhone which will be in the Apple App Store in a few weeks time, so make sure you sign up here to be informed when it’s available. Adding places of interest (POIs) to OpenStreetMap on location is now a simple 5-click process and there are hundreds of categories to choose from, everything from trees, volcanoes and gas stations to ATMs, vets, picnic spots and aerial way stations, so most interests should be covered. Let us know if there’s something you’re crying out for and we’ll do our best to get it into the next release.

More updates coming soon…

Your Future is Customized – State of the Map Presentation from CloudMade

Fri, 28 Aug 2009 14:04:56 +0000

In July this year, 250 of the leading lights in community mapping and geo application development gathered in Amsterdam for the annual OpenStreetMap Foundation conference The State of the Map. Video and audio recordings form the three day conference are being processed one by one by a dedicated team of volunteers who recently published the video of Nick Black’s talk: “Your Future is Customized”.

The talk asks why most current users of geodata experience maps through one of two ways: in car sat navs, or online mapping portals and looks towards a future of app stores, specialized map data and geo applications that match the exact needs of consumers.

Enable your applications with CloudMade – Nick Black (CloudMade) from State of the Map 2009 on Vimeo.

Follow along with the slides from the presentation:

Your Future Is Customized

View more presentations from nickb.

Find out more about Mapzen at the OpenStreetMap meet-up, London, 30th July

nick — Tue, 28 Jul 2009 16:22:43 +0000

If you’ve been dying to hear more about Mapzen, CloudMade’s under-development OpenStreetMap editor, you should call into the next OpenStreeMap meetup, in London this Thursday

The “Map Mayfair” evening starts at 6.30pm with a mapping session around Mayfair. The fun will continue in a nearby pub – “The Iron Duke”, where mappers will be meeting from 8.00pm onwards. OpenStreetMap meetups are a great way to get a quick introduction to mapping. For more details, take a look at the page below or get in touch with Harry, the event organiser.

You can find more details and sign up for the meet-up here.

Mapzen: An easy to use editor for OpenStreetMap

nick — Wed, 22 Jul 2009 14:27:46 +0000

At the recent State of the Map conference we unveiled our plans for an OpenStreetMap editor: Mapzen. Mapzen is a web based map editing tool that will make it far easier for new-commers to OpenStreetMap to get started editing and creating maps. Lets take a look at some of Mapzen’s features:

In Application Tutorials

No one likes searching wikis and web pages to learn how to use a tool. With Mapzen, you learn as you map. In application tutorials teach you to map, step by step.

Graphical Menus

Many new mappers are confused by the large number of different road, track and point of interest types. Mapzen has an intuitive graphical menu, letting you pick out different features to add to the map.

Context Sensitive Menus

After choosing which object you want to add to the map, the next hurdle is knowing which attributes to add. Mapzen’s context sensitive menu prompts you to add only the attributes that are needed for a particular object.

Specialized Junction Editing

Modeling complex junctions, with one-ways and turn restrictions can be difficult, with so many tags and relations to keep track of. Mapzen gives mappers a graphical interface that makes modeling junctions quick and fun.

Find Out More

Mapzen is currently under active development. We’re going to be carrying out user testing over the summer – if you’d like to take part, please mail mapzen@cloudmade.com.

Mapzen is open source (GPLv2) – if you are a developer, you can get hold of the source from here (right click in the window).

If you have any feature requests, or would like any more information, please send us a mail.

CloudMade releases Style Editor 1.1

Cragg — Thu, 16 Jul 2009 14:05:22 +0000

Great news for Style Editor users: we’ve launched an updated version of Style Editor (v1.1) that makes discovering, sharing and creating new map styles even easier!

Style Editor is CloudMade’s online cartography tool for creating customized map styles. Application developers can use Style Editor to quickly create their own map styles that give applications a unique look & feel. Style Editor gives you complete control over which objects are displayed on the map, and how they will appear on your map at each zoom level. Developers use CloudMade’s Style Editor to design customized maps built for a particular use. For example, if you were building a cycling application that helps users get from point A to point B by bicycle, you’d want to highlight the cycle paths and roads better suited for cyclists while de-emphasising main roads and motorways.

Style Editor now has a style gallery for discovering new map styles

The most important improvements in Style Editor 1.1 is the addition of the style gallery. Style gallery will allow you to quickly discover public map styles created by CloudMade and our community. Style Editor also allows for private styles available only to the author, which you can view by pressing the My Styles tab on the Style Editor toolbar. By default the map styles are sorted by popularity. You may also chose to look for CloudMade styles which were created by professional cartographers including some by Stamen Design by clicking on the CloudMade tab. Or you can browse the featured styles (Featured tab) which showcase a mix of CloudMade and community styles that our cartographers deem fantanstic!

The Style Editor tool bar allows for quick discovery of public maps styles for editing or direct use

Another cool new feature in style gallery is the ability to sort map styles based on color. To the top of the Style Editor toolbar is a Color Filter control. This control will filter map styles in the gallery based on where you drag the Color Filter bar. The default setting on the far left shows all colors. By dragging the Color Bar towards the right, you can select only the map styles with a particular shade of yellow, for example. In this way you can find map styles that suit the look and feel of your website or application. Don’t forget to reset the Color Bar back to the far left when your done so that you can see all the styles again.

Each map style in the style gallery shows the style name, author and style ID. The style ID comes in handy when coding with CloudMade’s API libraries. You can quickly change the Style ID and see a completely different map in your app! The search bar in the far right of the tool bar allows for style searches based on style name, author or style ID.

The last new feature we wanted to talk about is the Style RSS feed. You can now subscribe to all new style updates via RSS. This is a super easy way to get all the latest updates to Style Gallery via RSS. You can subscribe here.

Style Editor is open to the public and available to use as a free service here. So go ahead and start creating your own new map styles and publish them to Style Gallery so the world can see and use them!

Style Editor Team @CloudMade

Creating new map styles is simple with CloudMade's Style Editor