Generate Map Tiles by Multiple Tasks efficiency comparison

In The process of Generate Map Tiles by Multiple Tasks, various factors such as different Storage Formats (original, compact or MongoDB), the storage location of Tile Data and Tile (local or shared) will have a certain impact on the efficiency of graph cutting. In order to enable users to configure the optimal cutting scheme suitable for their current environment, through repeated testing and verification, four types of test scenarios are sorted out, and through the comparison of the cutting efficiency of the following scenarios, a reasonable Conclusion is obtained through analysis and summary. Provide reference and suggestions for the deployment of Generate Map Tiles by Multiple Tasks.

Scenario 1. Generate Map Tiles by Multiple Tasks. : Compare the cutting efficiency of multi-task cutting primitive, compact or MongoDB Tile.

Scenario 2. Local and remote path cuts : Compare the impact of Tile Data and Tile Storage on the graph cutting rate locally and remotely.

Scene 3. Vector/Raster Map Cut : Compare the cutting efficiency of multi-task cutting vector map and grid map.

Test Data and Environment

The same vector map data, cutting range, cutting scale and Tile parameters are used for the above three types of scenes, and the specific information is as follows:

  • Test Data: Select a vector map, the maximum Object Count of Vector Dataset is the 8324287, the file size is 12.4G, and the number of layers is 326.
  • Cutting range: left: 11427641.47674698, bottom: 2817774. 6107045934, right: 12993071. 816027392, top: 4383204. 9499850012, the number of unpacked tasks is 100, In total, 1638400 Tile are generated.
  • Cutting scale: 1: 18,055. 9909335062 (only 1 scale);
  • Set Tile parameters: The key parameters of Map Tiles for all test Scene Settings are: Tile format is JPG; The size of tiles (PX) is 256 * 256; Tile compression ratio is 75%.

Generate Map Tiles by Multiple Tasks

  • MongoDB Tile: TileStorage Format is MongoDB; Map local map; Test machine: 64-bit, 4-core CPU. Increase the number of processes to get the following efficiency comparison chart:
    • Figure: Multitasking to MongoDB Tile
  • Original Tile: TileStorage Format is original; Map local map; Test machine: 32-bit, 2-core CPU. Increase the number of processes to get the following efficiency comparison chart:
    • Figure: Multitask Cut Original Tile
  • Compact Tile: TileS torage Format is Compact; local map; test machine: 64-bit, 4-core CPU. Increase the number of processes to get the following efficiency comparison chart:
    • Figure: Multitasking to Compact Tile
  • Result analysis:

    ① For Tile with primitive, compact and MongoDB Storage Format, Generate Map Tiles by Multiple Tasks can significantly improve the efficiency of graph cutting compared with single-task graph cutting on.

    ② The most suitable number of processes for each graph cutting machine is the number of CPU cores * 2, that is, the dual-core CPU has 4 processes, the 4-core CPU has 8 processes, and so on.

    ③ The efficiency of MongoDB TileStorage Format improvement is the highest, the average improvement of ordinary 4-core PC is 4-5 times, and the highest improvement can be nearly 6 times in the test. MongoDB & > Compact & > Original.

Local and remote path cutting graph

Whether the cutting map and Tile Storage location are from the Local Directory or the shared directory, four cutting scenarios are formed.

  • Local Map-Local Tiles: The Tile Data is stored locally, and the cut Tile is also stored locally.
  • Local Map-Remote Tile: The Tile Data is stored locally, and the cut Tile is also stored in the remote machine.
  • Remote Map-Local Tiles: The Tile Data is placed remotely, and the cut Tile is also stored locally.
  • Remote Map-Remote Tile: Tile Data is placed in the remote, and the cut Tile is also stored in the remote.

    • Each machine starts four cutting sub-tasks, and the following efficiency comparison chart is obtained:

    Figure: Local and Remote Path Cut Graph Efficiency
  • Result analysis:

    ① Original Tile, when the map comes from the local and the tile is stored locally, the efficiency is the highest. This scenario is recommended for large-scale graph cutting applications.

    ② Compact Tile is slightly less efficient when the data comes from a shared directory, but it is also as efficient as MongoDB.

    ③ MongoDB Tile is not affected by data source and storage location.

Vector/Raster Map Cutting

    The vector and raster maps are from the local area. The raster map is Mosaic Dataset, containing 822 IMG images with a resolution of 90 meters. The number of rows and columns of a single image is 6000 * 6000. The file size is 44.7 G; The number of tiles obtained by cutting the map is the same as the number of tiles obtained by cutting the vector map, and 1638400 tiles are generated in total; and the scales of the cut maps are the same.

    There are three scenarios: Local MongDB, Remote MongDB and Local Compact. A graph cutting node is used, and each node starts four graph cutting processes. The following efficiency comparison chart is obtained:

    Figure: Vector/Raster Map Cutting Efficiency
  • Result analysis: The cutting time of

    raster map is obviously longer than that of vector map.

Conclusion

  • Conclusion 1: Multi-process can effectively improve the speed of graph cutting, and the efficiency is a multiple of the number of local CPU cores. For example, a 4-core CPU machine can be increased to about 4 times, and a 16-core CPU machine can be increased to about 16 times.
  • Conclusion 2: For each graph cutting node, it is recommended to start CPU * 2 graph cutting tasks.
  • Conclusion 3: Cutting MongoDB Tile can support more cutting graph nodes to work in parallel.
  • Conclusion 4: Compact Tile, it is recommended that the map data should be deployed in the local machine, and the Tile should also be stored in the local machine, which is the most efficient. The original Tile is not suitable for Data Migration, and no conclusion will be made here.
  • Conclusion 5: The efficiency of Compact Tile is the highest for the same number of processes of graph cutting tasks.When the amount of Tile tasks is not particularly large, the recommended deployment scheme is to use a high-performance physical machine and use the Generate Tile by Mulit-process.
  • Conclusion 6: Compact and original Tile, multiple processes are recommended.
  • Conclusion 7: MongoDB Tile, it is recommended that MongoDB services be deployed on machines that are not graph cutting nodes. MongoDB is memory-hungry and deployed on the graph cutting node, which will affect the graph cutting speed.

Through the above conclusion, we can choose the way of cutting graph reasonably according to our own Tile Data situation, Tile Type:, and computer performance. In the process of cutting, increase/decrease the number of processes according to the machine resources, so as to make rational and effective use of computer resources and maximize the efficiency of cutting.

Related topics

Generate Map Tiles

Single-task cutting graph

Generate Map Tiles by Multiple Tasks