Hiking Maps Follow-Up: Large Area Maps

The approach for creating feature-rich hiking maps described in my earlier post was limited in area, since it uses the Overpass API to download OSM data. The Overpass API has a restriction with regard to the amount of data downloadable in one go. In this post I describe a way to download a much larger area if needed, using the PBF files available from Geofrabrik and processing them with Osmosis.

Motivation

Me living in the Cologne-Bonn area, I love hiking in the Eifel, the Rhine and Mosel valleys, the Hunsrück and in the Westerwald. These are relatively close-by, attractive hiking regions with beautiful landscapes and quiet forests. For planning tours, I use my “Superatlas” – for details refer to my post “Creating the “Perfect” Hiking Map for Germany and other Countries”. Unfortunately, it is not possible to download the whole region mentioned from the Overpass API, but I wanted to have the whole area available in MOBAC, for spontaneous planning without first having to wait for the Maperitive jobs to finish. Doing one sub-area after another is no option, since in the overlapping parts you’d have “white spots” from the margins Maperitive creates.

Here’s how I was able to create a large area tile store for my favorite hiking region.

General Approach

My method is:

• Download the PBF files that cover the required area using curl
• Use Osmosis with a bounding box to reduce the data to the necessary amount
• and to merge the individual files into one, which then I
• load in Maperitive using the load-source command as replacement for download-osm-overpass
• and do the rest the same way as before.

I also tried to reduce the amount of data by filtering the tags included, but in the end the reduction was not worth the effort, considering that I may even miss something due to an error made.

All these steps I put into a batch file that I include into MOBAC as external tool the same way as the other Maperitive-tasks, as outlined in the older post mentioned.

In Detail

N.b.: To fully understand everything here, you may need to read my first hiking-map post beforehand.

Getting the PBF Files

In the download-section of Geofabrik there are PBF files available for the whole world, updated daily. For Germany, you may go down to federal state or even district level. For the mentioned area above I use Hessen, Rheinland-Pfalz and Nordrhein-Westfalen (Hesse, Rhineland-Palatinate and North Rhine-Westphalia). Download may be done e.g. using curl, wget or PowerShell Invoke-Webrequest cmdlet. The latter has the advantage that nothing needs to be installed since it is part of the OS, but I found Invoke-Webrequest surprisingly slow. So I went for curl. Commands are straightforward (Here for Hesse):

(Replace <Path-to-curl> and <PBF-Path> with values for your environment)

Reduce Data with Osmosis

The most recent version of Osmosis can be downloaded here. Osmosis can consume all current OSM data formats including PBF, process them in many ways and output the processed data again in OSM data formats. Cool stuff, thanks to the authors! It requires Java.

What I do is:

• Read in the first Geofabrik-PBF
• clip it via a bounding box
• write it to a temporary PBF.
• Now with each next Geofabrik PBF, do the following:
  • Load the temporary PBF,
  • load the next Geofabrik PBF and clip it
  • merge them
  • and write a new temporary PBF.

The first task is:

Again, adjust the paths to your needs. %1-%4 are later inserted by MOBAC and are the longitudes and latitudes of the bounding box bounds.

The remaining PBFs now go like this:

For the last run omit the move command.

You may do all this in just one Osmosis-run as well, putting all –rb … –bounding-box … parts into one command line (each –rb needs its own –bounding-box!). I found it a bit more easy to maintain the other way.

Optional: Tag-Filtering

You may even reduce data further by doing tag filtering. If you use my Suparatlas maperative rules, the Osmosis command would look like this:

<Path-to-Osmosis>\bin\osmosis.bat –rb <PBF-Path>\rheinland-pfalz-latest.osm.pbf –tf accept-relations boundary=national_park waterway=waterfall –tf accept-nodes accomodation=shelter amenity=bbq,bench,biergarten,cafe,drinking_water,firepit,fountain,grave_yard,parking,place_of_worship,pub,ranger_station,restaurant,shelter,table,toilets,water_point boundary=national_park building=cathedral,chapel,church,mosque,synagoge,temple,wayside_chapel,wayside_shrine embankment=yes geological=outcrop historic=aqueduct,archaeological_site,battlefield,boundary_stone,castle,castle,city_gate,fort,memorial,milestone,mine,monastery,monument,pillory,ruins,rune_stone,stone,tomb,tree_shrine,wayside_cross,wayside_shrine,gallows information=board,guidepost,map,office landuse=cemetery,forest,quarry,surface_mining,vineyard leisure=bird_hide,firepit,nature_reserve,picnic_table,swimming_area,swimming_pool,water_park,wildlife_hide man_made=adit,beacon,communications_tower,cross,dyke,embankment,lighthouse,mineshaft,observatory,survey_point,telescope,tower,water_tap,water_well,watermill,windmill natural=arete,bare_rock,cave_entrance,cliff,geyser,glacier,heath,moor,mud,peak,ridge,rock,scree,scrub,spring,stone,tree,wetland,wood sport=climbing tourism=alpine_hut,artwork,attraction,information,picnic_site,viewpoint,wilderness_hut waterway=waterfall –rb <PBF-Path>\rheinland-pfalz-latest.osm.pbf –tf reject-relations –tf reject-ways highway=motorway,motorway_link,trunk,trunk_link –used-node –merge –wb <PBF-Path>\myregion.pbf

Please note: You’ll need to read in the source PBF twice, since the first filter will lose all ways, the second will restore them, and only them. For details refer to the Osmosis Detailed Usage pages.

I did not use this in the end, since the data reduction was something of about 10% only, and I am afraid that my filters may filter something I’d like to keep in the end. Also, when changing my rendering rules, I would need to think of updating this also.

Changing the Maperative Command File

The only thing to change in the Maperative script file if to replace the line

download-osm-overpass

by

load-source <PBF-Path>\myregion.pbf

And that’s it – you’re done!

MOBAC Batch for External Command

Here’s my batch file – make sure to replace the path-placeholders:

• <Path-to-curl>
• <Path-to-Osmosis>
• <Path-to-Maperitive>
• <Path-to-MOBAC>
• <PBF-Path>

Also, put in the download links for your regions and add/modify the Osmosis commands to pick up those files then!

Due to whatever reason the Osmosis-commands need their own shell (hence cmd /c …) – otherwise, the MOBAC batch would just stop after the first Osmosis-part. I guess it’s because Osmosis is wrapped in its own batch file.

Performance

On my five-year old 16 GB Core i5-4590 @ 3.3 GHz machine, processing an area from Aachen in the top left to Worms in the lower right, which is an area of about 16,000 km², took that much (RAM did not be too important by the way):

• Downloading the files: ~1 minute (Depends on internet bandwidth of course)
• Processing the Osmosis tasks: ~5 minutes
• Loading and rendering the result PBF in maperative: ~5 minutes
• Creating tiles for zoom levels 9-17: 9 hours
• Resulting tile store: ~16 GB in ~800,000 files

So this is certainly not running on a daily basis here 🙂 I guess my workflow will be: Use the large-area Superatlas with a seperate, “static” tile store for planning activities, and then for the actual hiking tour create a much smaller regional map, with up-to-date data, the “old” way using Overpass, to put on my smartphone. When in a hurry, I may use the “static” tiles for the smartphone atlas – better than nothing. Once every few months I’ll then update my static atlas.

Limitations

While the area limitation of Overpass is now overcome, I’d still not reommend to create a smartphone atlas from such a large area. The tile download from Basemap DE may still take ages, may upset the service provider, and the resulting file may be that large that the smartphone may struggle. Still, having the local tiles ready at hand is helpful already!

Credits

Thanks to the authors of curl and of Osmosis! Great work!