Efficiency Comparison of Generating Tiles by Multi-process

During multi-process tile generation, various factors such as storage formats (original, compact, or MongoDB), tile data sources, and storage locations (local or shared) can affect efficiency. To help users configure optimal solutions for their environments, we conducted repeated tests and organized four test scenarios. Through comparative analysis of tile generation efficiency in these scenarios, we derived reasonable conclusions, providing references and recommendations for deploying multi-process tile generation.

Scenario 1. Multi-process Tile Generation: Compare efficiency when generating original, compact, and MongoDB tiles.

Scenario 2. Local vs. Remote Path Tile Generation: Compare how local/remote tile data and storage affect generation speed.

Scenario 3. Vector/Raster Map Tile Generation: Compare multi-process efficiency for vector and raster maps.

Test Data and Environment

All three scenarios used identical vector map data, tile ranges, map scales, and tile parameters:

  • Test Data: Vector map with maximum object count 8,324,287; file size 12.4G; 326 layers.
  • Tile Range: [Left:11427641.47674698, Bottom:2817774.6107045934, Right:12993071.816027392, Top:4383204.9499850012], divided into 100 sub-processes, generating 1,638,400 tiles.
  • Map Scale: 1:18,055.9909335062 (single scale)
  • Tile Parameters: JPG format; 256*256 tile size; 75% compression ratio.

Multi-process Tile Generation

Figure: Multi-process Generating MongoDB Tiles
Figure: Multi-process Generating Original Tiles
Figure: Multi-process Generating Compact Tiles
  • MongoDB Tiles: MongoDB storage format; local map; 64-bit 4-core CPU. Efficiency comparison with increasing processes:
  • Original Tiles: Original storage format; local map; 32-bit 2-core CPU. Efficiency comparison with increasing processes:
  • Compact Tiles: Compact storage format; local map; 64-bit 4-core CPU. Efficiency comparison with increasing processes:
  • Analysis:

    ① Compared to single-process caching, multi-process significantly improves efficiency for all storage formats.

    ② Optimal process count per machine: CPU cores × 2 (e.g., 4 processes for 2-core CPU).

    ③ MongoDB shows highest efficiency improvement (4-5× average, up to 6×). Efficiency order: MongoDB > Compact > Original.

Local vs. Remote Path Tile Generation

Four scenarios formed by combining local/shared map data and storage paths. Four sub-tasks per machine:

Figure: Tile Generation Efficiency of Local vs. Remote Paths
  • Local Map-Local Tiles: Both data and storage local.
  • Local Map-Remote Tiles: Local data, remote storage.
  • Remote Map-Local Tiles: Remote data, local storage.
  • Remote Map-Remote Tiles: Both data and storage remote.
  • Analysis:

    ① Original tiles achieve highest efficiency with local data and storage. Recommended for large-scale tasks.

    ② Compact tiles show slightly lower efficiency with shared directories, comparable to MongoDB.

    ③ MongoDB tile efficiency remains stable regardless of data source/storage location.

Vector/Raster Map Tile Generation

  • Both vector and raster maps (822 90m-resolution images, 44.7G total) generated 1,638,400 tiles at same scale. Three scenarios tested with 4 processes per node:
    Figure: Vector/Raster Map Generation Efficiency
  • Analysis:

    Raster map generation takes significantly longer than vector maps.

Conclusions

  • Conclusion 1: Multi-process improves speed proportionally to CPU cores (4× for 4-core, 16× for 16-core).
  • Conclusion 2: Recommend launching CPU×2 tasks per node.
  • Conclusion 3: MongoDB tiles support more parallel nodes.
  • Conclusion 4: Compact tiles work best with local data and storage.
  • Conclusion 5: Compact tiles show highest efficiency; recommend high-performance physical machines for moderate workloads.
  • Conclusion 6: Multi-process recommended for compact/original tiles.
  • Conclusion 7: Deploy MongoDB on separate machines to avoid memory contention.

Select tile generation methods based on data characteristics, tile types, and hardware capabilities. Adjust process counts according to resources to maximize efficiency.

Related Topics

Generate Map Tiles

Single Process Map Cache

Generate Tile by Multi-process