[QGIS] in Basic Field Archaeology and LIDAR Processing

124 Pages·2017·17.36 MB·English

by  Stephen Eastmead

LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

Use of QGIS Geographical Information System
in Basic Field Archaeology and LIDAR Processing

LIDAR data and other data sources are processed to describe the basic use of QGIS,
and how to process the UK Open Data LIDAR files into 2D images and 3D models.

Stephen Eastmead

First published November 2017
Current Revision 4 February 2018

Chapter 1: LIDAR and Mapping in QGIS Page 1
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

Introduction
Fair Use
It has taken hundreds of hours of work, experimentation and recording to assemble the
information in these pages. I would be grateful if you could recognise this by making a small
donation to Altogether Archaeology, which is a small community archaeology group that
relies heavily on membership subscriptions to fund archaeological fieldwork.

You can simply make the donation either by bank transfer or by post, using this webpage:
https://altogetherarchaeology.org/DonationsMenu.php or by using PayPal sending the donation to
this email address: [email protected]
Suggested donation: £10-£15 in the UK or $20-$25 using PayPal from other countries.

Copyright © Stephen Eastmead 2017
Please do not pass the document on to others, but tell them where it can be downloaded
from by sending them this download link: https://eastmead.com/QGIS-LIDAR.htm
I would be grateful if you would acknowledge my efforts if you use any of the information to
provide maps used within a presentation or publication. Thanks.
Acknowledgements
My thanks to Sue Goldsborough for spending considerable time in testing these procedures
and making valuable suggestions to the text, and to my wife Pam, for trying to locate the
inevitable typos that escaped our scrutiny. All the rest of the typos are 100% my own work.
General Information
Originally this was seven separate documents, but I have since amalgamated them into this
seven-chapter document. Each chapter is a natural follow-on from the previous chapter,
except for the concluding chapter, which is a miscellany of useful QGIS and LIDAR related
snippets.
The Contents page indicates the scope of each chapter. The data downloads described in
Chapter 1 are used throughout the rest of the chapters, so make sure you have downloaded
them first.
This document contains both internal (document) and external (Internet) links. Each internal
link has a [BackLink] which returns you either to the original link page, or to the next section
in the described process.
If every minute step was described, this would be a long manuscript. So, I have assumed
that readers have a basic knowledge of how to use Microsoft Windows software. The
described procedures have all been developed on a Windows 10 Home 64-bit OS computer.
I haven’t been able to test them on earlier versions of Windows.
MAC OS X
Sue Goldsborough has tested the instructions on her MacBook Pro and has found that they
can easily be followed. There are specific MAC OS X instructions in addition to the Windows
10 procedures. As of September 2017, it was necessary to use the Firefox browser instead
of Safari or Chrome.

Edition / Revision update history
See this link

Chapter 1: LIDAR and Mapping in QGIS Page 2
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

Contents

Introduction ....................................................................................................................... 2
Fair Use ......................................................................................................................... 2
Copyright © Stephen Eastmead 2017 ............................................................................ 2
Acknowledgements ........................................................................................................ 2
General Information ....................................................................................................... 2
MAC OS X ..................................................................................................................... 2
Edition / Revision update history .................................................................................... 2
Contents ............................................................................................................................ 3
Chapter 1: LIDAR and Mapping in QGIS ............................................................................... 8
Chapter 1: Introduction ...................................................................................................... 8
Step 1 The Site .............................................................................................................. 8
Step 2 Create some folders ......................................................................................... 10
Step 3 LIDAR data - Background ................................................................................. 10
Step 4 LIDAR data - Downloads .................................................................................. 11
Step 5 Raster and Vector Maps – Background ............................................................ 14
Step 6 Open Data Maps – Background ........................................................................ 14
Step 7 Ordnance Survey Open Data Maps - Background ............................................ 15
Step 8 Vector and Raster Map – Downloads ................................................................ 15
Step 9 QGIS - Download .............................................................................................. 16
Step 10 QGIS Main Window ........................................................................................ 17
Chapter 1: QGIS Main Window General Notes: ........................................................... 18
Chapter 1: Appendix 1 OS Open Data Download Examples ............................................ 19
Chapter 2: LiDAR Processing in QGIS - 2D ........................................................................ 20
Editor’s Note: New Chapter 2 functionality ....................................................................... 20
Digital Elevation Model creation (DEM) ........................................................................... 20
ASC Files ..................................................................................................................... 20
GDAL: Geospatial Data Abstraction Library ..................................................................... 21
LiDAR General Processing Procedures ........................................................................... 21
Hillshade Processing Methodology .............................................................................. 21
Step1 – Single and multiple merged asc files into a DEM layer ....................................... 23
Step 1 Option 1: Single asc file DEM: Using the ‘Add Raster Layer’ icon ..................... 23
Step 1 Option 2: Merging multiple raster (asc) files in a DEM ....................................... 23
Step 1 Option 3: Virtual DEM alternative method ......................................................... 24
Step 2 – Filling nodata -9999 ‘holes’ in LiDAR data ......................................................... 25
Method ......................................................................................................................... 26

Chapter 1: LIDAR and Mapping in QGIS Page 3
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

Step 3 – Applying Hillshade to the DEM file ..................................................................... 27
Step 3 Option 1 ............................................................................................................ 27
Step 3 Option 2 ............................................................................................................ 28
Chapter 2 Appendix 1 – Previous Version of Chapter 2 ................................................... 30
Introduction .................................................................................................................. 30
DEM Creation Procedure ............................................................................................. 30
Single asc file DEM: ..................................................................................................... 30
Creating a multiple asc file DEM .................................................................................. 31
Creating Hillshade Images ............................................................................................... 32
Hillshade Procedure ..................................................................................................... 32
Chapter 3: LIDAR Processing – Colour Ramps ................................................................... 34
Installing the Plugin ......................................................................................................... 34
Creating a LIDAR image with colour banded elevations .................................................. 36
Preparation: ................................................................................................................. 36
Colour banding elevation: organising the Layers Panel: ............................................... 37
Colour banding elevation: DEM layer processing ......................................................... 37
Colour banding elevation: Hillshade Layer processing ................................................. 38
Colour Banding Elevation with OS CONTOURS-terr50 Data ....................................... 39
Chapter 4: LIDAR Processing – 3D Models ....................................................................... 40
Chapter 4: Introduction .................................................................................................... 40
Install Plugin .................................................................................................................... 40
Clipping Raster Files ....................................................................................................... 40
Clipping Method for LIDAR during 3D model generation ................................................. 41
3D models – Options ....................................................................................................... 43
Software for the Sketchfab option .................................................................................... 43
Saving the Python Code to your Windows PC ................................................................. 44
Generating a HTML model from your DEM and Hillshade .tif files ................................... 44
Simple ‘White Space’ Removal .................................................................................... 45
Alternative method of eliminating ‘white space’ ............................................................ 47
Generating a 3D model from your DEM and Hillshade tif files and importing it into
Sketchfab ........................................................................................................................ 48
Generating the Python command line .............................................................................. 49
Concise Procedure Windows 10 computers .................................................................... 49
Completing the upload to Sketchfab ................................................................................ 50
Chapter 4: Appendix 1 - Python Code ............................................................................. 53
Chapter 4: Appendix 2 - Command text editing in detail Windows 10 .............................. 56
Chapter 4: Appendix 3 - Notepad Python command text ................................................. 58
Chapter 4: Appendix 4 - Sketchfab Post Processing........................................................ 59

Chapter 1: LIDAR and Mapping in QGIS Page 4
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

Chapter 4: Appendix 5 - ‘White Space’ Removal using MeshLab .................................... 60
Chapter 4: Appendix 6 - Saving the Python Code to your MAC OS X Computer ............. 61
Chapter 4: Appendix 7 - Concise Procedure MAC OS X Computers ............................... 62
Screen Images ............................................................................................................. 63
Chapter 5: Using GPS data, Contours, Raster, Vector and On-line Maps ........................... 65
Downloading the GPS Data ............................................................................................. 65
GPS Tracks Line: Style ................................................................................................ 67
Labels .......................................................................................................................... 69
Raster Maps .................................................................................................................... 72
Vector Maps .................................................................................................................... 75
Raster Tile File Sizes ................................................................................................... 75
Vector Files Sizes ........................................................................................................ 75
Online Maps .................................................................................................................... 78
Contour Lines .................................................................................................................. 79
Chapter 6: QGIS Print Composer ........................................................................................ 80
Chapter 6: Introduction .................................................................................................... 80
Procedure ........................................................................................................................ 80
Step 1: Configuring the data to map ............................................................................. 80
Step 2: Creating a new Print Composer Map ............................................................... 80
Step 3 Viewing the map in Print Composer .................................................................. 81
Step 4 Adding: Border, Grid and Coordinates .............................................................. 82
Step 5: Adding a Scalebar ........................................................................................... 83
Step 6: Adding: Title, copyright and feature text ........................................................... 84
Completed Map............................................................................................................ 85
Locating an Agisoft trench plan image on a map using the trench’s GPS coordinates . 85
Map Output ..................................................................................................................... 88
Landscape Map Features ................................................................................................ 88
Chapter 7: QGIS, LIDAR & Other – Miscellaneous Snippets ............................................... 89
Chapter 7: Introduction .................................................................................................... 89
1 QGIS Decorations ........................................................................................................ 89
2 Processing LIDAR Point Cloud Data using LAStools .................................................... 89
LAStools Installation .................................................................................................... 90
Point Cloud (.laz) QGIS Procedure .............................................................................. 90
Blast2dem - Creating New Classification Combinations ............................................... 92
3 Planlauf TERRAIN ........................................................................................................ 92
4 Relief Visualization Toolbox version 1.3 ....................................................................... 93
5 QGIS Software Bugs .................................................................................................... 95

Chapter 1: LIDAR and Mapping in QGIS Page 5
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

6 Plugins installed ........................................................................................................... 95
7 Google Earth ................................................................................................................ 96
8 Google Earth and the British National Grid ................................................................... 97
GPS Utility Preliminary Procedure ............................................................................... 98
Google Earth Error Assessment ................................................................................... 98
GPSU Data Correction Procedure ............................................................................... 99
9 Add a Geology Layer in QGIS using WMS/WMTS ...................................................... 101
Geology Example ....................................................................................................... 101
10 QGIS LIDAR 1M DSM & 1M DTM England Zoomable using WMS/WMTS ............... 103
11 Ordnance Survey Open Data Raster and Vector Maps Issue ................................... 106
12 Ordnance Survey OS Net ......................................................................................... 106
13 IrfanView Windows Image Editor .............................................................................. 107
14 Transferring Large Files for Free using the Internet .................................................. 107
15 Editing Navigation Controls in Sketchfab .................................................................. 108
Sketchfab Navigation Editing Procedure .................................................................... 108
16 QGIS SAGA LIDAR Tools ........................................................................................ 111
17 LIDAR Resolution Comparison: 200cm - 100cm - 50cm - 25cm ............................... 112
18 GPS Software: GPSU and GPS TrackMaker ............................................................ 114
19 QGIS Website links .................................................................................................. 114
Gentle Introduction to GIS .......................................................................................... 114
Training and User Guides both as webpage and pdf download .................................. 114
QGIS Current versions ............................................................................................... 114
QGIS Previous versions ............................................................................................. 114
QGIS Development .................................................................................................... 114
QGIS Issue Tracking .................................................................................................. 114
QGIS Bug Reporting .................................................................................................. 114
Chapter 8 planlauf/TERRAIN ............................................................................................ 115
Introduction ................................................................................................................... 115
Procedures .................................................................................................................... 115
Methodology .................................................................................................................. 115
Files ........................................................................................................................... 115
Menu Options ............................................................................................................ 116
Basic Processing Steps ................................................................................................. 117
Stage 1 Merging LiDAR asc files ................................................................................ 117
Stage 2 Creating the plfter file .................................................................................... 118
Step 3 Creating the Local Relief and Slope Models ....................................................... 119
Step 4 Color Ramps ................................................................................................... 119

Chapter 1: LIDAR and Mapping in QGIS Page 6
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

Step 5 Export Options ................................................................................................ 122
Summary ................................................................................................................... 123
Acknowledgments ............................................................................................................. 123
Feedback ...................................................................................................................... 124

[BackLinkToChapter7]

Chapter 1: LIDAR and Mapping in QGIS Page 7
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

Chapter 1: LIDAR and Mapping in QGIS
Chapter 1: Introduction
If you intend following this series of basic use chapters to record archaeology in QGIS, or to
manipulate LIDAR data, then follow Chapter 1 so that you can obtain all the necessary data
files. You will be downloading: LIDAR tiles, OS Open Data: raster and vector maps, and a
GPS (gpx) file type. The data downloads are of How Hill, near Downholme in Swaledale,
North Yorkshire.
By the end of the document you will be able to obtain and process your own maps and
LIDAR data, GPS waypoint or track data as well as dipping your toe into 3D modelling.
I have chosen the How Hill site as it demonstrates interesting multi-period LIDAR features
and produces nice LIDAR colour coded elevation bands. Whilst this series of chapters are
LIDAR based, all that you learn will be applicable using QGIS for non-LIDAR archaeological
data. Note American spelling is used when referring to software features e.g. color.
Step 1 The Site
The site chosen to illustrate the use of QGIS in this document can be seen on the 1:25,000
OS map below.
Note: the blue grid lines denote 1 km by 1 km squares.
a) Decide which 1km squares you require data for. Obviously How Hill is divided across
the left-hand pair and I know there is archaeology on the opposite hillside across
both right-hand squares.
b) Having established that four squares of LIDAR and Map data will be used, you then
need to identify them. Memory-Map will indicate the coordinates as you move the
cursor over each square see above. If you do not have access to a digital map or a
paper map, then go to: https://www.gridreferencefinder.com/bing.php If it is not set to
Aerial View, click on ‘Aerial View’ and select Ordnance Survey. Zoom into the site,

Chapter 1: LIDAR and Mapping in QGIS Page 8
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

and then right-click on the map to display British National Grid Coordinates. See red
pins on the map (above), where I have right-clicked on all four 1km squares.
NOTE: The difference between the Grid Reference and the X(Easting) and
Y(Northing) which has the 2-letter grid prefix replaced by 4’s in this case. This is
explained in Paragraph e) below.

c) Starting with the bottom left grid record each of the four grids’ 2-digit coordinates like
this:
a. Bottom left SE 10 97 (Point A)
b. Top left SE 10 98 (Point B)
c. Top right SE 11 98 (Point C)
d. Bottom right SE 11 97 (Point D)
and note of the 10km by 10km grid co-ordinate which is the 1-digit co-ordinate i.e. SE
19. If you are unlucky, the project may cross over more than one 10km grid.
d) You will see when using QGIS that the British National Grid (BNG) or EPSG:27700
doesn’t display the leading grid letters like SE for How Hill, but a numeric equivalent.
SE uses a 4 before each easting and northing, so the reference shown on the map of
SE 10766 97945 becomes 410766 497945. See how 4 and 4 for grid SE is derived
on the map below.

Chapter 1: LIDAR and Mapping in QGIS Page 9
LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h

e) Note: Remember that the OS British National Grid’s datum is known as OSGB36,
and in QGIS the BNG is called EPSG:27700.
f) The yellow highlighted data in paragraphs: c) and d) above is used to obtain the
LIDAR and Map data.
Step 2 Create some folders
You could potentially create a lot of new files. I recommend creating either
C:\LIDAR\projectname\ folder system, or C:\QGIS\projectname\ system, and then creating
suitable sub-folders as suggested in this document. Remember not to use spaces in-
between folder names and file names, hence using projectname rather than project name,
and keep names short as possible. Following these recommendations will eliminate potential
problems as you proceed through this document. Secondly, create a master map folder
containing sub-folders for your various map downloads e.g. C:\MAPS\.

Step 3 LIDAR data - Background
Historic England has a free pdf document on LIDAR. The first two images below are taken
from that publication. They provide useful background information. The pdf can be found
here: Historic England LIDAR publication: https://content.historicengland.org.uk/images-
books/publications/light-fantastic/light-fantastic.pdf/

Chapter 1: LIDAR and Mapping in QGIS Page 10