DEM Generation module

Name	Caption
Output Folder	Output DEM folder
Output DEM Base Name	Output DEM base name
Output File Type	Output DEM file format
Overwrite Results	Overwrite existing results
Output Map Units	Output DEM projection units
Tile Size	Output DEM tile size
Resolution	DEM pixel resolution
Elevation Units	Output elevation units
Background Elevation	Output background elevation
Elevation Reference	Output vertical reference for elevation values
Points	Input point data
Contours	Input contour data
Break Lines	Input break-line data
3-D Structures	Input 3-D structure data
Elevation Options	Options for points, contours, and break lines
Elevation Range	Elevation range for points, contours, and break lines
Ridges	Input ridge data
Valleys	Input valley data
Cliffs	Input cliff data
Unit Conversion	Elevation offset and scale

Parameter descriptions

Output Folder

The full path and name of the folder that contains the index files (index.pix and index.txt) generated by the DEM Index File Creator module. The specified folder will contain the output raster DEM files.

This parameter is mandatory.

Output DEM Base Name

The base name of the output digital surface model (DSM) and digital terrain model (DTM) files. This name is prepended to all of the output DSM and DTM files.

If no value is specified for this parameter, the default digital elevation model (DEM) base name is dem.

Output File Type

The file format for the output DEM.

Overwrite Results

Select this check box to overwrite the existing output files, if any exist. If this check box is left clear, and an output file exists in the relevant folder, the status of the job displays a message informing you of the existence and name of the output file. The message is also written to the event log of the job.

Output Map Units

The output projection of the ingested vectors. All input will be reprojected automatically to the output projection.

If no value is specified for this parameter, the output projection uses the projection of the first input vector file found, by default.

Valid projections include:

UTM: Universal Transverse Mercator

The value specified can be the UTM grid-zone number and row, and Earth model, as follows:
```
UTM [mm] [r] [Ennn]
```
where:
- [mm] is the two-digit zone number between 1 and 60
- [r] is the zone row: a single letter between N and X for north of the equator and between C and M for south of the equator. Zone rows A, B, Y, and Z are not supported; rows I and O do not exist.
- [Ennn] specifies the Earth model, where the model number is between 0 and 19. If the Earth model is not specified, it is assumed to be E000 (Clarke 1866).
Long/Lat: longitude and latitude

The Earth model can be specified for LONG/LAT (and other units except PIXEL), as follows:
```
LONG/LAT [Ennn]
```
If the Earth model is not specified, it is assumed to be E000 (Clarke 1866).
SPCS: State Plane Coordinate System

The SPCS zone number and Earth model can be specified as follows:
```
SPCS [mmmm] [Ennn]
```
where:
- [mmmm] is the four-digit zone number
- [Ennn] specifies the Earth model, where the model number is between 0 and 19. If the Earth model is not specified, it is assumed to be E000 (Clarke 1866).
ProjectionName: the label of a user-defined projection, if it exists in the file userproj.txt.

For more information about the available projections, see Projections and Earth models.

This parameter is optional.

Tile Size

The dimensions of each output DEM tile, in pixels and lines. If a single value is specified, it is used for both the horizontal and vertical dimensions. If no value is specified, a single tile is created.

If the output format is JP2, the tile-size values must be multiples of 512, because JP2 creates internal tiles of 512 x 512.

Resolution

The resolution of the output DEM pixels. The resolution must be specified in the units of the Output Map Units parameter.

Note: To eliminate deformations in bridges, the DEM resolution should be two-to-three times the resolution of the output orthorectified files. For example, if you want an ortho resolution of 0.2 meters, set the resolution from 0.4 meters through 0.6 meters.

This parameter is mandatory.

Elevation Units

The units of the elevation values stored as pixel values in the output DEM.

Available units are:

Meters
Feet
US Feet

Note: Feet is defined as 0.3048 meters (corresponding to International Feet); US Feet is defined as 1200/3937 meters (corresponding to U.S. Survey Feet).

Background Elevation

The background (No Data) elevation value for the output DEM.

If no value is specified for this parameter, the default value of -32,768 is used.

This parameter is optional.

Elevation Reference

The vertical reference for the elevation values in the output DEM.

Supported options are:

Mean Sea Level: elevations are referenced to Mean Sea Level. The EGM2008 geoid model is used to convert these values to ellipsoidal heights.
Ellipsoid: elevations are referenced to the ellipsoid. The coordinate system of the DEM raster file is examined to determine which ellipsoid and datum to use; for instance, if the DEM file is referenced to NAD83, the heights are assumed to be NAD83 ellipsoidal heights. If the DEM and math model are based on different coordinate systems, the heights are automatically converted to the math model coordinate system before use.

Points

The input file names, folders, or both containing the elevation points.

Elevation points are individual x and y points with an associated elevation. The points from all input files are collected and output to a vector segment named 'POINTS'.

The supplied string can be used to specify a single file, a recursive folder search or, by using the asterisk (*) wildcard, a more sophisticated match. For examples, see Module details.

Supported input file types include:

.shp (ESRI shapefile)
.txt (ASCII text)
.xyz (ASCII text)
.las (LIDAR)
.laz (LIDAR compressed)
.pix (PCIDSK)
.dxf (AutoCAD)
.dgn (older versions of MicroStation)

Contours

The input file names, folders, or both containing the contours.

Contours are 3-D lines that have a single elevation for the entire line (break lines, in contrast, have a different elevation at each vertex). The contours from all input files are collected and output to a vector segment named 'CONTOURS'.

Break Lines

The input file names, folders, or both containing the break lines.

Break lines are 3-D lines in which every vertex can have a different elevation. The break lines from all input files are collected and output to a vector segment named BREAKLIN.

3-D Structures

The input file names, folders, or both containing 3-D structures.

These are typically man-made structures, such as bridges or buildings, and represented by 3-D polygons. The polygons from all input files are collected and output to a vector segment named 'STRUCT3D'.

Elevation Options

Defines the following elevation options:

The names of attribute fields containing elevation data. These names are used only with point and contour data.

If the point or contour data has the elevation value specified as a numeric attribute, the name of the field is specified here. Multiple names can be specified, separated by spaces or commas. If multiple names are provided, the first name matching a field will be used.
The value RETURN1 specifies the first return, RETURNALL for all returns, and RETURNLAST for the last return. RETURN1 and RETURNLAST cannot be used at the same time. By default, the first return is selected.

With LIDAR-point files, RETURN1, RETURNALL, or RETURNLAST can also include an optional suffix to indicate which classifications to process. For example, RETURN1:"Ground,Road Surface" only processes "Ground" and "Road Surface" points. Class-name matching is case-insensitive. Use a comma to separate class names. Surround the class names with double quotes if supplying more than one class or if there are spaces in the class name. Partial class names will process those points that have the partial class name as a substring of the point's class. For example, RETURNALL:"ro" processes only points that have "ro" anywhere in the class name.
If the option NONAME is specified, all vector segments will be named NONAME rather than POINTS, CONTOURS, and so on. This is sometimes useful in complex situations.
With POINT data, each point is considered a separate shape. By specifying the option PACKPOINTS, 10,000 points are packed into each shape. This reduces the size on disk by half and allows the module to run faster. The points, however, are in a nonstandard representation.

If no value is specified for this parameter, PACKPOINTS is used by default.

Elevation Range

The range of elevation values [maximum, minimum] accepted for points, contours, and break lines.

If a single value is specified, it is considered a maximum value.

Note: Points, contours, or break lines containing values outside this range are not ingested. If any vertex of a break line is outside this range, the entire break line is ignored. The elevation-range check is done after any elevation conversion is applied.

Ridges

The input file names, folders, or both containing ridges.

Ridges are 2-D lines that have no associated elevation, but represent lines where elevation falls off on either side. The ridges from all input files are collected and output to a vector segment named RIDGES.

Valleys

The input file names, folders, or both containing valleys.

Valleys are 2-D lines that have no associated elevation, but represent lines where elevation rises on either side. The valleys from all input files are collected and output to a vector segment named VALLEYS.

Cliffs

The input file names, folders, or both containing cliffs.

Cliffs are 2-D lines that have no associated elevation, but represent discontinuities where elevations on either side are independently interpolated from surrounding information. The cliffs from all input files are collected and output to a vector segment named CLIFFS.

Unit Conversion

The conversion to apply to the input vector data to transform it to the units specified for the Elevation Units parameter; for example, converting from feet to meters.

If no value is specified for this parameter, the elevation values provided with the input vector data are assumed to be those specified for the Elevation Units parameter, and no conversion is applied.

This parameter specifies two values:

offset, scale

The conversion formula is:

elevation value = scale * (vector elevation value + offset)

Details

General job details

The DEM Generation module provides a series of predefined configuration files that define the type of output to generate. These are defined in the settings.py file located in the PROHOME\exe\PGS\config folder of your CATALYST Enterprise installation.

Preprocessing requirements

Before running this module, the following requirements must be met to ensure the job processes successfully and produces accurate results:

For every file matching the given inputs, only the first vector layer is used. For best results, consider:
- Each input file should contain only one type of information (for example, contours and break lines should be in separate files)
- Each input file should have only a single layer of information (for example, only contours or only points)
- File names should uniquely represent the data they contain

Module details

The DEM Generation module reads multiple vector files containing points, break lines, contours, valleys, ridges, and cliffs, and generates a raster digital elevation model (DEM) by interpolating elevation values of points, contours, and 3-D structure lines. The process is separated into two main sections: Height-data preparation and DEM generation.

Height-data preparation

Often, you do not have raster DEMs; rather, you have hundreds, perhaps thousands, of small files containing points and vector representations of contours, break lines, and features, such as ridges, valleys, and cliffs. The DEM Generation module merges all the information in these point and vector files into a single PCIDSK (.pix) file, which can then be used to create a high-quality raster DEM.

The parameters Points, Contours, Break Lines, 3-D Structures, Ridges, Valleys, and Cliffs specify the location of the input files. The input files can be a single specific file or a group of files indicated by a wildcard, and can include a recursive folder search.

For example:

points="abc.shp" a single specific file abc.shp
points="abc*.pix" wildcard search for all .pix files starting with abc
points="abc*.*, -abcd.shp" all files starting with abc, excluding file abcd.shp
points="data" all files in the data folder (and each of its subfolders)
cliffs="d:\\lines\\cliff*" all files in the lines folder starting with cliff (no subfolders)
cliffs="d:\\lines\\**\\cliff*" all files in the lines folder (and each of its subfolders).

It is recommended to specify the full path (for example, "D:\\lines" in Windows); otherwise, the module uses the current working folder.

Up to 32 exclusions can be specified using -xxx separated by commas; for example: "-abc, -ignore, -x". Use of the asterisk wildcard (*) is not supported. If any portion of a file name or a folder name is matched, it is excluded.

Point, vector data or both, representing elevation comes in a wide variety of file formats, organization on disk, and internal representations. While the DEM Generation module has considerable interpretation capabilities, it does not directly solve every circumstance. In some cases, it may be necessary to reorganize files into folders or to reformat data using other third-party utilities before using this module.

For best results, consider:

Each file should contain only one type of information; for example, contours and break lines should be in separate files)
Each file should have only a single layer (with the exception of .dxf files)
File names should uniquely represent the data they contain
ESRI Shapefile (.shp) is the preferred file format

Supported input file formats are:

PCIDSK files (.pix): these files are scanned for vector segments and the first vector segment found is used as input. An exception to this is that segments named POINTS are assumed to contain points, segments named CONTOURS are assumed to contain contours, and so on. Vector segments named EXTENTS and OUTLINE are ignored. This special handling of names allows output from the DEM Generation module to be used as input on a second run of the module.
ESRI shape files (.shp): in most cases, points and contours are already in 3-D but, if necessary, the Elevation Options parameter can specify the field name with the z-elevation value. It is assumed that the associated .dbf, .prj, and .shx files are also present.
MicroStation files (.dgn): older-format MicroStation files
APSPRS LiDAR files (.las): these files contain elevation points and often contain multiple returns (first return, up to fifth return). By default, only the last return is read (RETURNLAST) because it most closely represents bare earth. The Elevation Options parameter can be used to request only the first return (RETURN1) or all returns (RETURNALL).
raw ASCII text files (.txt): ASCII text files can be used as input, but must be accompanied by a generic ASCII vector file (.gav) of the same name. The .gav file describes the layout of the data in the text file. The .gav file is a PCI Geomatics convention. The file must be created before running the DEM Generation module. If a .gav file is not found, a default .gav file named xyz.gav is used.
AutoCAD files (.dxf): The first layer in a .dxf file contains header information and is skipped. The DEM Generation module requires all actual input data to be in the second layer.

Height-data preparation results

The height-data preparation process generates DEM index files named index.pix and index.txt, and a PCIDSK file named demprep.pix in the output folder.

Demprep.pix file

The demprep.pix file contains the following vector segments, depending on the input files:

EXTENTS: polygon extents for each ingested file; used for informational and quality-assurance (QA) purposes
POINTS: points with elevation
CONTOURS: lines with a single elevation
BREAKLIN: 3-D lines with a elevation value at each vertex
STRUCT3D: 3-D structures (such as bridges or buildings), with an elevation value at each vertex
VALLEYS: 2-D lines with no elevation
RIDGES: 2-D lines with no elevation
CLIFFS: 2-D lines with no elevation
OUTLINE: a single polygon outlining the extents of all ingested files
SUMMARY: holds a copy of the generated processing report for informational and QA purposes

The vertices in all created vector segments are 3-D and have no attribute fields. Any attribute fields existing in the original input file are not copied. The VALLEYS, RIDGES, and CLIFFS segments are also 3-D by default, and may contain elevations, but this module ignores any elevation information in those segments.

To limit memory consumption, vector segments greater than 1 GB in size are divided into multiple segments, with the same name.

The DEM Generation module uses the POINTS, CONTOURS, BREAKLIN, STRUCT3D, VALLEYS, RIDGES, and CLIFFS segments to interpolate a raster DEM. The module can use the OUTLINE segment to constrain the interpolated DEM so that it does not bleed into the the background.

The EXTENTS segment contains a polygon outline and information, such as the file name, for every file ingested. Although the DEM Generation module does not use this segment, it can be useful for QA purposes. For example, if the raster DEM generated by this module is loaded in CATALYST Professional Focus, this segment can be loaded as a vector overlay. If the DEM contains anomalies, the input vector file associated with the data in that area can be identified and loaded using Attribute Manager in Focus.

The SUMMARY segment contains a copy of the text report; this is useful if the original report was not generated or was lost. The report can be saved to a separate text file or viewed in Focus.

DEM index file

The DEM index files (index.pix and index.txt) list all the DEM tiles created by the DEM Generation module. The results of this module can be used as input for other CATALYST Enterprise processes, such as orthorectification, GCP collection, and tie-point collection. The index.pix file contains footprints of all DEM tiles; these can be viewed in Focus.

DEM generation

The DEM-generation process generates a raster DEM by interpolating elevation values of points, contours, and 3-D structure lines in vector segments. In addition, 2-D break lines, such as valleys, ridges, and cliffs with no elevation value are also accepted as constraints for interpolation.

Typically, multiple DEM-generation child jobs are created; each DEM-generation child job generates a single DEM tile.

DEM-generation algorithm

The DEM-generation algorithm consists of the following three steps, plus an optional step for processing 2-D features:

Scan conversion of the 3-D vector data
Initial interpolation of source elevations
Optional: scan and initialization of the 2-D vector data
Iterative smoothing of the interpolated values

In the first step, the input 3-D vector lines and points are reprojected to the raster coordinates. They are then burned into the raster buffer with the elevations interpolated linearly between vector nodes. At this stage, the 2-D layers are ignored. If multiple elevation values are scanned into a single pixel, the minimum elevation value is assigned to the pixel, the maximum elevation is recorded for use in interpolation, and the pixel is marked as a cliff.

In the second step, the elevation value in each DEM pixel is interpolated from the source elevation data. An interpolation is made between the source elevations and elevations that are at an equal distance from the source locations. During the interpolation, the source elevation closer in value to the current elevation is used. As a result, the multi-valued pixels retain the discontinuity between their 'low' and 'high' sides.

In the optional step, any 2-D vector layers present are scanned and converted into a flag buffer. The 2-D features are also initialized for use during the iterative smoothing process.

In the final step, a finite difference method is used to iteratively smooth the DEM grid. The module also uses the over-relaxation technique to accelerate convergence. When the DEM grid is being smoothed, the source elevation values remain unchanged, while the interpolated values are updated based on the neighborhood values.

Module results

On successful completion, this module produces a series of output files. The following is an example of files generated by the DEM Generation module, where the prefix ottawa_ represents the value specified for the Output DEM Base Name parameter:

ottawa_demprep.pix
ottawa_demprep.report.txt
index.pix
index.txt
ottawa_0_0.jp2
ottawa_0_0.jp2.pox
ottawa_0_0.jp2.report.txt
ottawa_0_1.jp2
ottawa_0_1.jp2.pox
ottawa_0_1.jp2.report.txt
ottawa_1_0.jp2
ottawa_1_0.jp2.pox
ottawa_1_0.jp2.report.txt
ottawa_1_1.jp2
ottawa_1_1.jp2.pox
ottawa_1_1.jp2.report.txt

The DEM files are the DEM index files (index.pix and index.txt) and the DEM tiles (in the example, the JP2 files). The POX files are created only when the output DEM format is TIF or JP2. The remaining files are reports on the processing.

The results of the DEM Generation module can be used by other CATALYST Enterprise modules requiring DEMs as input, such as orthorectification.

Modifying the output DEM coverage area

In some cases, background values outside the area of interest may not be assigned correctly; in such cases, you must manually modify the polygon that defines the output DEM coverage area.

To modify the output DEM coverage area

In CATALYST Professional Focus, open the *_demprep.pix file generated by the DEM Generation module.
This file is located in the folder specified for the Output Folder parameter of the DEM Generation module.
Modify the polygon in the vector layer named OUTLINE so it correctly outlines the data.
Rerun the DEM Generator child job from the initial DEM Generation module run.
For more information, see Resubmitting (copying) a job.

On successful completion of the job, the data outside the outline polygon is filled with NoData values.

DEM Generation module

Description

Parameters

Parameter descriptions

Details