Mapnik implemention of Mapbox Vector Tile specification

Hi, While I tried to make I faced this error

In file included from Release/obj/gen/vector_tile.pb.cc:5:0: Release/obj/gen/vector_tile.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is #error This file was generated by a newer version of protoc which is

I tried both protobuf 3.3.0 and protobuf 3.4.1 but there was not any change, where am I doing wrong?

Hi! I'm trying mapnik (branch v3.0.x) and mapnik-vector-tile (v1.2.2). I build x64 debug configuration for Windows. My test code:

mapnik::datasource_cache::instance().register_datasource(mapnik_dir + "/mapnik/input/shape.input");
mapnik::Map m(256, 256);
load_map(m, "style.xml");	
m.zoom_all();
mapnik::vector_tile_impl::processor ren(m);
mapnik::vector_tile_impl::merc_tile tile(0, 0, 0, 1024);
ren.update_tile(tile);

My simplified mapnik style:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map[]>
<Map srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m [email protected] +wktext +no_defs +over" background-color="#b5d0d0">

<Parameters>
  <Parameter name="scale">1</Parameter>
  <Parameter name="metatile">2</Parameter>
  <Parameter name="name"><![CDATA[OpenStreetMap Carto]]></Parameter>
  <Parameter name="description"><![CDATA[A general-purpose OpenStreetMap mapnik style, in CartoCSS]]></Parameter>
  <Parameter name="bounds">-180,-85.05112877980659,180,85.05112877980659</Parameter>
  <Parameter name="center">0,0,4</Parameter>
  <Parameter name="format">png</Parameter>
  <Parameter name="minzoom">0</Parameter>
  <Parameter name="maxzoom">22</Parameter>
</Parameters>
<Style name="world" filter-mode="first">
  <Rule>
    <MinScaleDenominator>750000</MinScaleDenominator>
    <PolygonSymbolizer fill="#f2efe9" />
  </Rule>
</Style>
<Layer name="world"
  minimum-scale-denominator="750000"
  srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m [email protected] +wktext +no_defs +over">
    <StyleName>world</StyleName>
    <Datasource>
       <Parameter name="file"><![CDATA[Path to\simplified_land_polygons.shp]]></Parameter>
       <Parameter name="type"><![CDATA[shape]]></Parameter>
    </Datasource>
  </Layer>

</Map>

I get an error like this? call stack:

>	test.exe!std::vector<ClipperLib::OutRec * __ptr64,std::allocator<ClipperLib::OutRec * __ptr64> >::operator[](unsigned __int64 _Pos) Line 1237	C++
 	test.exe!ClipperLib::Clipper::AppendPolygon(ClipperLib::TEdge * e1, ClipperLib::TEdge * e2) Line 2735	C++
 	test.exe!ClipperLib::Clipper::AddLocalMaxPoly(ClipperLib::TEdge * e1, ClipperLib::TEdge * e2, const mapnik::geometry::point<__int64> & Pt) Line 2267	C++
 	test.exe!ClipperLib::Clipper::ProcessHorizontal(ClipperLib::TEdge * horzEdge) Line 3103	C++
 	test.exe!ClipperLib::Clipper::ProcessHorizontals() Line 2881	C++
 	test.exe!ClipperLib::Clipper::ProcessEdgesAtTopOfScanbeam(const __int64 topY) Line 3434	C++
 	test.exe!ClipperLib::Clipper::ExecuteInternal() Line 1952	C++
 	test.exe!ClipperLib::Clipper::Execute(ClipperLib::ClipType clipType, ClipperLib::PolyTree & polytree, ClipperLib::PolyFillType subjFillType, ClipperLib::PolyFillType clipFillType) Line 1907	C++
 	test.exe!mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor>::operator()(mapnik::geometry::polygon<__int64,mapnik::geometry::rings_container> & geom) Line 292	C++
 	test.exe!mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> >::operator()(const mapnik::geometry::polygon<double,mapnik::geometry::rings_container> & geom) Line 238	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 313	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::detail::dispatcher<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void,mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::apply_const(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 319	C++
 	test.exe!mapbox::util::variant<mapnik::geometry::geometry_empty,mapnik::geometry::point<double>,mapnik::geometry::line_string<double>,mapnik::geometry::polygon<double,mapnik::geometry::rings_container>,mapnik::geometry::multi_point<double>,mapnik::geometry::multi_line_string<double>,mapnik::geometry::multi_polygon<double>,mapnik::geometry::geometry_collection<double> >::visit<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double>,void>(const mapnik::geometry::geometry<double> & v, mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f) Line 865	C++
 	test.exe!mapnik::util::apply_visitor<mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & __ptr64,mapnik::geometry::geometry<double> >(mapnik::vector_tile_impl::transform_visitor<mapnik::vector_tile_impl::vector_tile_strategy,mapnik::vector_tile_impl::geometry_clipper<mapnik::vector_tile_impl::geometry_to_feature_pbf_visitor> > & f, const mapnik::geometry::geometry<double> & v) Line 43	C++
 	test.exe!mapnik::vector_tile_impl::detail::create_geom_layer(mapnik::vector_tile_impl::tile_layer & layer, double simplify_distance, double area_threshold, mapnik::vector_tile_impl::polygon_fill_type fill_type, bool strictly_simple, bool multi_polygon_union, bool process_all_rings) Line 161	C++
 	test.exe!mapnik::vector_tile_impl::processor::update_tile(mapnik::vector_tile_impl::tile & t, double scale_denom, int offset_x, int offset_y) Line 311	C++
 	test.exe!main(int argc, char * * argv) Line 43	C++

This is a bug or am I doing something wrong? (ClipperLib is here) Thanx!

Hi, I am using mapbox GL JS (including newest 1.0.0. and some older versions) with my own vector tiles, generated with mapnik-vector-tile from OSM data. Works fine with vast majority of tiles, but I have noticed that some tiles in Japan are not rendered and console saying only Unimplemented type: 4. I tried to isolate the issue, and it seems to me, that is happens when feature property (name) contains japanese characters. I have no idea, what to do next about this - if this is bug in encoder (mapnik-vector-tile) or decoder (mapbox GL JS). But I can decode the tile without issues with protobuf-net and I can see japanese names in values just fine in debugger.

See attached sample tile, layer road osm_13_7179_3254_broken.zip

Hi,

I'm trying to buid Mapnik vector tile on Windows. I'm able to build 'mapnik' using guidelines posted at https://github.com/mapbox/windows-builds.

Is there is a how to steps to follow in order to build 'Mapnik with vector tile support' ?

Thanks Noury

Looks like we've still got some memory corruption lurking in the v2_spec branch. I'm seeing this happen sometimes when running make test:

$ make test
make[1]: Nothing to be done for `all'.
./build/Release/tests

tests is a Catch v1.1 b3 host application.
Run with -? for options

-------------------------------------------------------------------------------
geometries visual tests
-------------------------------------------------------------------------------
../test/geometry_visual_test.cpp:265
...............................................................................

../test/geometry_visual_test.cpp:69: FAILED:
  CHECK( buffer2 == buffer )
with expansion:
  "I
?????(? x"      ??ZF?>??>?? ?   ??  ?"?!?
  ==
  "I
??(??er"5   ??ZF?>??>?? ?   ??  ?"?!?
                                                  x"

../test/utils/geometry_equal.hpp:140: FAILED:
  REQUIRE( p.template get<0>().y == Approx(p.template get<1>().y) )
with expansion:
  40.6390923571 == Approx( 40.6137897117 )

===============================================================================
test cases:    181 |    180 passed | 1 failed
assertions: 544800 | 544798 passed | 2 failed

make: *** [test] Error 2

/cc @flippmoke

It appears that with high tolerance settings the start and end vertex may be discarded when encoding lines. We need tests to ensure this cannot occur and fix any bugs around tolerance handling to prevent it.

Should also take a look at #29 at the same time.

Tasks include:

• [x] - ensure that any internal move_to commands are preserved for multilines that happen to be encoded this way (b515cf754fcd1609).
• [x] - ensure that move_to commands are never dropped (8950587). For example if two move_to commands are found in sequence we currently drop the second if the tolerance is such that it is not a valid movement. But this could create gaps - the last move_to in a sequence is the critical one to keep.
• [x] - ensure that the last vertex in lines is never dropped no matter how high the tolerance (2664718)
• [x] - fix tilelive-bridge tests after 2664718: https://gist.github.com/springmeyer/f04660a83823a03121cd
• [x] - should we try to drop all but the last if multiple move_to are found in sequence?
• [x] - ensure that the last vertex in polygons is never dropped no matter how high the tolerance
• [x] - ensure that the line_to vertex before an internal move_to in lines is never dropped no matter how high the tolerance
• [x] - ensure that the line_to vertex before an internal move_to in polygons is never dropped no matter how high the tolerance

We are seeing that processes that need to reproject data will crash on windows. This has likely be happening for some time (causing crashing for Mapbox Studio windows users). This may be a mapnik core problem in how Mapnik uses proj4. But so far it has only seemed to happen in Mapbox Studio (and therefore seems related to vector tiles). So, let's track this issue here for now.

This ticket is to track another regression in v0.8.x (after the move to using boost::geometry::transform) and therefore this is another followup to #111/#116 and #120.

What I'm seeing is that the conversion of the buffered bbox to integer coordinates may throw. When fixing #116 I noticed this might throw but could not think of a case where the buffered bbox would be outside the range of proj4 enough to trigger a throw. That was naive as I've now found that is highly probable when dealing with source data in a different projection than mercator.

Next steps:

• How did this work before (in mapnik-vt v0.7.1)?
• Should we be clipping the buffered box / buffered_query_ext to the layer extent / layer_ext2 (and if so, how?)
• testcase (can use the NZ_Coastline_NZMG.shp from https://github.com/mapbox/mapbox-studio/issues/1390)
• what can we learn from https://github.com/mapbox/node-wmshp/pull/9

Lately I've been investigating one issue around MVTs using the Postgis plugin to retrieve the geometries from the database and found that, when using the simplify_geometries parameter, the geometries are simplified considering a 256 pixel size, not a vector_layer_extent size.

Consider the attached xml (postgis_simplification.txt), or this extract:

       <Parameter name="simplify_geometries"><![CDATA[true]]></Parameter>
       <Parameter name="twkb_encoding"><![CDATA[true]]></Parameter>
       <Parameter name="vector_layer_extent"><![CDATA[4096]]></Parameter>

Following the twkb tolerance calculation when requesting a tile of zoom 12, (12,12,12) we should have:

• Extent: 40075017 / 2^12 = 9783.939697
• Resolution: $extent / 4096 = 2.388657153
• Tolerance: POW(10, ROUND(LOG10($resolution)) - 1) = 0.1

Instead we see this query:

SELECT ST_AsTWKB(ST_Simplify(ST_RemoveRepeatedPoints("the_geom",10),10,true),-1) AS geom,"cartodb_id" FROM (SELECT 2 AS cartodb_id, 'SRID=3857;LINESTRING(0 20037508, 0 0, 0 10037508, 0 -10037508, 0 -20037508)'::geometry as the_geom) as cdbq WHERE "the_geom" && ST_SetSRID('BOX3D(-19959236.82582522 19871181.3692407,-19871181.3692407 19959236.82582522)'::box3d, 3857)

Which comes from using 256 as the extent:

• Extent: 40075017 / 2^12 = 9783.939697
• Resolution: $extent / 256 = 38.21851444
• Tolerance: POW(10, ROUND(LOG10($resolution)) - 1) = 10

This means that for a 4096 tile we are having a tolerance 100 times bigger than expected which leads to oversimplification and partially breaking the ability of over-zooming. It is even more painful if you try to do some kind of one-tile-optimization (request a big part of the map with less or no simplification), for example requesting the tile (0,0,0) with an extent of 256*2^18 and simplifying in the client. As the requested extent isn't being considered, the geometries will just disappear due to the simplification.

I've been investigating and this hasn't always been like this. It used to be just layer_extent_ / qw (which is what I expected and want in this case), but it was modified in 041aed410dd147b4beeebd1723f452268a018a83 to use image_size instead, and then modified to its current status in 75c4868d0900963c19e0adcd00ec8d5c87572a57 to use VT_LEGACY_IMAGE_SIZE: https://github.com/mapbox/mapnik-vector-tile/blob/61aebcbaf4552dc410a8bdbbe09f9e5d7cbbb076/src/vector_tile_layer.hpp#L331-L340

@flippmoke Could you please shed some light on this? Is there any special requirement to maintain the 256 there? I've changed it locally and it passes all the tests. Should I open a PR with the change?

Do geometry rings have a defined winding order in vector tiles? E.g. counter-clockwise for outer rings and clockwise for rings? Or should I expect any order of any particular ring?

Winding order isn't mentioned anywhere in spec, but some polygon algorithms depend on a certain winding order. Also, some algorithms (especially triangulation) need to know if a polygon ring is a hole or an outer ring, which could also be derived from winding order if geometry is flattened.

It seems like the vector::processor sometimes fails to get some lines (from a shapefile) on the buffered part of the tile, especially when the line seems to be north/south of the tile.

I have rendered all geometries in some tiles and the resulting image shows that some smaller roads and (which are more problematic for my case) label-lines are removed in the buffered part (compared to the surrounding tiles actual data/image). This results in cut off labels (the line is the center of the text) for some tiles.

I have tried to take a larger buffer but it doesn´t fix it. Is this a known/unknown problem or have I misunderstood something?

Ran into this recetly, whilst trying to install @mapbox/spritezero

npm ERR! In file included from ../src/node_mapnik.cpp:7:
npm ERR! In file included from ../src/mapnik_vector_tile.hpp:7:
npm ERR! In file included from ../../mapnik-vector-tile/src/vector_tile_merc_tile.hpp:5:
npm ERR! ../../mapnik-vector-tile/src/vector_tile_tile.hpp:11:10: fatal error: 'mapnik/geometry/box2d.hpp' file not found
npm ERR! #include <mapnik/geometry/box2d.hpp>
npm ERR!          ^~~~~~~~~~~~~~~~~~~~~~~~~~~

Looks like it's related to https://github.com/mapbox/mapnik-vector-tile/issues/261 even though it's been some time since that issue has been closed.

Any suggestion how work resolve/work around the issue?

This might be expected.

First run:

-------------------------------------------------------------------------------                                               
geometries visual tests
-------------------------------------------------------------------------------
../test/geometry_visual_test.cpp:250
...............................................................................                                               

../test/utils/geometry_equal.hpp:112: FAILED:
  REQUIRE( false )                                                                                                            
with messages:                                                                                                                
  mapnik::geometry::multi_polygon<double, std::vector>
  mapbox::geometry::polygon<double, std::vector>

===============================================================================
test cases:  202 |  201 passed | 1 failed
assertions: 1801 | 1800 passed | 1 failed

Subsequent runs:

-------------------------------------------------------------------------------                                               
geometries visual tests
-------------------------------------------------------------------------------
../test/geometry_visual_test.cpp:250
...............................................................................                                               

../test/utils/geometry_equal.hpp:112: FAILED:
  REQUIRE( false )                                                                                                            
with messages:                                                                                                                
  mapnik::geometry::multi_polygon<double, std::vector>
  mapbox::geometry::polygon<double, std::vector>

===============================================================================
test cases:  202 |  201 passed | 1 failed
assertions: 1903 | 1902 passed | 1 failed

when running make test, one test fails:

-------------------------------------------------------------------------------                                               
geometries visual tests
-------------------------------------------------------------------------------
../test/geometry_visual_test.cpp:250
...............................................................................                                               

../test/utils/geometry_equal.hpp:112: FAILED:
  REQUIRE( false )                                                                                                            
with messages:                                                                                                                
  mapnik::geometry::multi_polygon<double, std::vector>
  mapbox::geometry::polygon<double, std::vector>

===============================================================================
test cases:  202 |  201 passed | 1 failed
assertions: 1903 | 1902 passed | 1 failed

code of the failing test:

template <typename T1, typename T2>
void operator() (T1 const&, T2 const&)
{
    // comparing two different types!
    INFO(type_name<T1>());
    INFO(type_name<T2>());
    REQUIRE(false);
}

https://github.com/Microsoft/WSL/issues/2465:

There is solution of problem with fakeroot: sudo update-alternatives --set fakeroot /usr/bin/fakeroot-tcp

As per my discussion with @springmeyer in https://github.com/mapbox/mapnik-vector-tile/issues/284 there appears to be an issue with the support of GeometryCollections when creating tiles from geojson.

In some cases the features appear to be dropped across tile boundaries, though I have been unable to determine what the exact offending circumstances are. I isolated a cleaner test case from one of the culprit data sets, which you can get here.

I am using https://github.com/mapbox/mapbox-tile-copy v5.1.1 to generate the tiles but I tested with the latest release as well, and the issue persists.

Here is what the feature is supposed to look like (I flattened the geojson in this example using https://github.com/Turfjs/turf/tree/master/packages/turf-flatten):

And here is what it looks like without flattening the geojson:

I've created a working example here.

Please let me know if there is anything else I can provide to help isolate the problem.

Cheers.