{"id":108,"date":"2019-01-13T21:18:11","date_gmt":"2019-01-13T21:18:11","guid":{"rendered":"https:\/\/rossedwards.co.uk\/blog\/?p=108"},"modified":"2019-05-05T22:32:40","modified_gmt":"2019-05-05T22:32:40","slug":"real-world-data-qgis-to-rv4","status":"publish","type":"post","link":"https:\/\/rossedwards.co.uk\/blog\/2019\/01\/13\/real-world-data-qgis-to-rv4\/","title":{"rendered":"Real world data QGIS to RV4"},"content":{"rendered":"\n<p>I recently finished off a tutorial covering steps required to get real world data into Arma 3 (RV4 Engine) using only open source tools: <br><a href=\"https:\/\/github.com\/rossoe\/Arma3_QGIS\">https:\/\/github.com\/rossoe\/Arma3_QGIS<\/a><br><br>Whilst working out how to manage heightmap and satellite imagery within QGIS in preparation for loading into RV4 Engine, I discovered some interesting things along the way.<br><br>The first major issue reared it&#8217;s head following a <a href=\"http:\/\/trac.osgeo.org\/gdal\/wiki\/Release\/2.4.0-News\">GDAL update<\/a> affecting the gdalwarp command, which broke a step I had been relying on &#8211; &#8216;<strong>Clip to raster by mask layer<\/strong>&#8216; I asked the <a href=\"https:\/\/github.com\/OSGeo\/gdal\/issues\/1176\">question<\/a> on gdal&#8217;s github page, and it was confirmed that I had to switch to use full extents, instead of the easier shapefile mask layer. See below original pipeline which mainly relied upon QGIS GUI config.<br><\/p>\n\n\n\n<div class=\"wp-block-envira-envira-gallery\"><div id=\"envira-gallery-wrap-257\" class=\"envira-gallery-wrap envira-gallery-theme-base envira-lightbox-theme-base_dark\" itemscope itemtype=\"https:\/\/schema.org\/ImageGallery\"><div data-row-height=\"\" data-gallery-theme=\"\" id=\"envira-gallery-257\" class=\"envira-gallery-public  envira-gallery-1-columns envira-clear enviratope envira-gallery-css-animations\" data-envira-columns=\"1\"><div id=\"envira-gallery-item-258\" class=\"envira-gallery-item enviratope-item envira-gallery-item-1 envira-lazy-load\" style=\"padding-left: 1px; padding-bottom: 1px; padding-right: 1px;\"  itemscope itemtype=\"https:\/\/schema.org\/ImageObject\"><div class=\"envira-gallery-item-inner\"><div class=\"envira-gallery-position-overlay  envira-gallery-top-left\"><\/div><div class=\"envira-gallery-position-overlay  envira-gallery-top-right\"><\/div><div class=\"envira-gallery-position-overlay  envira-gallery-bottom-left\"><\/div><div class=\"envira-gallery-position-overlay  envira-gallery-bottom-right\"><\/div><a href=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/QGIS_satellite_image_export_process-1.jpg\" class=\"envira-gallery-257 envira-gallery-link\" rel=\"enviragallery257\" title=\"Original QGIS satellite image export process\" data-envira-caption=\"Original QGIS satellite image export process\" data-envira-retina=\"\" data-thumbnail=\"\"  itemprop=\"contentUrl\"><div class=\"envira-lazy\" data-test-width=\"640\" data-test-height=\"480\" style=\"padding-bottom:75%;\"><img id=\"envira-gallery-image-258\" class=\"envira-gallery-image envira-gallery-image-1\" data-envira-index=\"1\" src=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/QGIS_satellite_image_export_process-1-1024x659-640x480.jpg\" data-envira-src=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/QGIS_satellite_image_export_process-1-1024x659-640x480.jpg\" data-envira-gallery-id=\"257\" data-envira-item-id=\"258\" data-envira-caption=\"Original QGIS satellite image export process\" alt=\"\" title=\"Original QGIS satellite image export process\"  itemprop=\"thumbnailUrl\" data-envira-srcset=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/QGIS_satellite_image_export_process-1-1024x659-640x480.jpg 400w,https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/QGIS_satellite_image_export_process-1-1024x659-640x480.jpg 2x\" srcset=\"data:image\/gif;base64,R0lGODlhAQABAIAAAP\/\/\/\/\/\/\/yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\" \/><\/div><\/a><\/div><\/div><\/div><\/div><noscript><img src=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/QGIS_satellite_image_export_process-1.jpg\" alt=\"\" \/><\/noscript><\/div>\n\n\n\n<p>This lead me to invest more time learning the construct of the GDAL commands within QGIS&#8217;s python console. There is of course a lot more flexibility within command line than using the GUI.<\/p>\n\n\n\n<!--more-->\n\n\n\n<p><strong>Summary of the key discoveries:<\/strong><br><br><strong>1.<\/strong>  Added <strong>-wo SOURCE_EXTRA=1000&nbsp;<\/strong><br>This removed corruption of data during CRS reprojection step.<br><br>NASA&#8217;s Robert Simmon had some very useful tips re constructing GDAL commands, and noted the benefit of using <strong>SOURCE_EXTRA<\/strong>, he also mentioned compression which feeds into my second point<br><a rel=\"noreferrer noopener\" href=\"https:\/\/medium.com\/planet-stories\/a-gentle-introduction-to-gdal-part-2-map-projections-gdalwarp-e05173bd710a\" target=\"_blank\">https:\/\/medium.com\/planet-stories\/a-gentle-introduction-to-gdal-part-2-map-projections-gdalwarp-e05173bd710a<\/a><\/p>\n\n\n\n<p>&#8220;<em>-r lanczos defines the resampling method, with a few more options than gdal_translate. Lanczos is slow but high quality. -wo SOURCE_EXTRA=1000 is an example of a warp option\u2014advanced parameters that determine how the reprojection is calculated. SOURCE_EXTRA adds a buffer of pixels around the map as it is reprojected, which helps prevent gaps in the output. Not all reprojections require it, but it doesn\u2019t hurt to add the option to be on the safe side.<\/em>&#8221; <\/p>\n\n\n\n<p><strong>2.<\/strong>  Added compression using <strong>-r cubic<\/strong><br>An alternative <strong>-r lanczos<\/strong> was noted in article above, but it resulted in degradation of raster.<br><\/p>\n\n\n\n<p><strong>3.<\/strong>  Used <strong>-te<\/strong> instead of <strong>-crop_to_cutline<\/strong> <br>Although this required acquisition of precise extents rather than just selecting a shapefile to cut by, it avoided the incorrect resolution output due to the GDAL update for <strong>gdalwarp<\/strong> command.<\/p>\n\n\n\n<p>I initially managed to reduce 3 tasks &#8211; <strong>reproject CRS<\/strong>, <strong>set cell size<\/strong> &amp; <strong>cut to extents<\/strong> into 2 commands (see below), but I could not escape slight degradation with the raster output. I had hoped that by adding <strong>-r cubic<\/strong> to both of the following commands, it would resolve the issue, but same result. <\/p>\n\n\n\n<p><strong>Two gdalwarp commands to reproject CRS &amp; then set cell size + cut to extents:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import os\nos.system(r'''gdalwarp -t_srs EPSG:32631 -wo SOURCE_EXTRA=1000 -tr 5.0 5.0 -r cubic -of GTiff \"E:\/Heightmaps Opentopo\/output_srtm.asc\" D:\/QGIS\/crm4.tif''')\n\nimport os\nos.system(r'''gdalwarp -tr 5.0 5.0 -r cubic -te 174425.817977 5600556.828797 215386.817977 5641517.828797 D:\/Arma\/QGIS\/crm3.tif D:\/QGIS\/cut12.tif''')<\/code><\/pre>\n\n\n\n<p>It was only when I compressed the above into a single command that I got a perfect output raster, I suspect this was due to too much processing on the raster.<\/p>\n\n\n\n<p><strong>Single gdalwarp command to reproject CRS, Set cell size &amp; Cut to extents:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import os\nos.system(r'''gdalwarp -t_srs EPSG:32631 -wo SOURCE_EXTRA=1000 -tr 5.0 5.0 -r cubic -ot Float32 -of GTiff -te 173421.201222 5600629.600063 214381.201222 5641589.600063 \"E:\/Heightmaps Opentopo\/output_srtm.asc\" E:\/QGIS\/converted9.tif''')<\/code><\/pre>\n\n\n\n<div class=\"wp-block-envira-envira-gallery\"><div id=\"envira-gallery-wrap-285\" class=\"envira-gallery-wrap envira-gallery-theme-base envira-lightbox-theme-base_dark\" itemscope itemtype=\"https:\/\/schema.org\/ImageGallery\"><div data-row-height=\"\" data-gallery-theme=\"\" id=\"envira-gallery-285\" class=\"envira-gallery-public  envira-gallery-1-columns envira-clear enviratope envira-gallery-css-animations\" data-envira-columns=\"1\"><div id=\"envira-gallery-item-299\" class=\"envira-gallery-item enviratope-item envira-gallery-item-1 envira-lazy-load\" style=\"padding-left: 1px; padding-bottom: 1px; padding-right: 1px;\"  itemscope itemtype=\"https:\/\/schema.org\/ImageObject\"><div class=\"envira-gallery-item-inner\"><div class=\"envira-gallery-position-overlay  envira-gallery-top-left\"><\/div><div class=\"envira-gallery-position-overlay  envira-gallery-top-right\"><\/div><div class=\"envira-gallery-position-overlay  envira-gallery-bottom-left\"><\/div><div class=\"envira-gallery-position-overlay  envira-gallery-bottom-right\"><\/div><a href=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/IOW2-1.png\" class=\"envira-gallery-285 envira-gallery-link\" rel=\"enviragallery285\" title=\"Heightmap loaded into QGIS with hillshade colour ramp\" data-envira-caption=\"Heightmap loaded into QGIS with hillshade colour ramp\" data-envira-retina=\"\" data-thumbnail=\"\"  itemprop=\"contentUrl\"><div class=\"envira-lazy\" data-test-width=\"640\" data-test-height=\"480\" style=\"padding-bottom:75%;\"><img id=\"envira-gallery-image-299\" class=\"envira-gallery-image envira-gallery-image-1\" data-envira-index=\"1\" src=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/IOW2-1-1024x605-640x480.png\" data-envira-src=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/IOW2-1-1024x605-640x480.png\" data-envira-gallery-id=\"285\" data-envira-item-id=\"299\" data-envira-caption=\"Heightmap loaded into QGIS with hillshade colour ramp\" alt=\"\" title=\"Heightmap loaded into QGIS with hillshade colour ramp\"  itemprop=\"thumbnailUrl\" data-envira-srcset=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/IOW2-1-1024x605-640x480.png 400w,https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/IOW2-1-1024x605-640x480.png 2x\" srcset=\"data:image\/gif;base64,R0lGODlhAQABAIAAAP\/\/\/\/\/\/\/yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\" \/><\/div><\/a><\/div><\/div><\/div><\/div><noscript><img src=\"https:\/\/rossedwards.co.uk\/blog\/wp-content\/uploads\/2019\/03\/IOW2-1.png\" alt=\"\" \/><\/noscript><\/div>\n\n\n\n<p><br><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I recently finished off a tutorial covering steps required to get real world data into Arma 3 (RV4 Engine) using only open source tools: https:\/\/github.com\/rossoe\/Arma3_QGIS Whilst working out how to manage heightmap and satellite imagery within QGIS in preparation for loading into RV4 Engine, I discovered some interesting things along the way. The first major &hellip; <span class=\"read-more\"><a href=\"https:\/\/rossedwards.co.uk\/blog\/2019\/01\/13\/real-world-data-qgis-to-rv4\/\" class=\"more-link\">Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":224,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3,4],"tags":[],"_links":{"self":[{"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/posts\/108"}],"collection":[{"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=108"}],"version-history":[{"count":50,"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/posts\/108\/revisions"}],"predecessor-version":[{"id":1240,"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/posts\/108\/revisions\/1240"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/media\/224"}],"wp:attachment":[{"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=108"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=108"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rossedwards.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=108"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}