Automatic Accuracy Assessment module

Name	Caption
Scene Folder	Location of one or more georeferenced images on which to perform an accuracy assessment (input)
Output Folder	Output folder
Overwrite Results	Overwrite existing results
Scale	Amount by which to scale the check-point-displacement lines
Assessment Mode	Mode of accuracy assessment to use
Measurement Mode	Mode of measurement to use
Control Images or Chip Database	Folder of reference images or chip database
Control Channel for Matching	Image channel from one or more control images used for feature matching to automatically collect check points
Master Matching Channel	Image channel from input scenes used for feature matching to automatically collect check points
Sampling Method	Check-point sampling method
Check Point Samples	Image channel from one or more control images used for feature-matching to automatically collect check points
Matching Algorithm	Matching algorithm
Search Radius	Search radius (pixels)
Absolute Minimum Intersection	Minimum intersection between the ortho image and control source for check-point collection that must exist for absolute assessment
Absolute Minimum Intersection Unit	Unit of measure of Absolute Minimum Intersection
Relative Minimum Intersection	Minimum intersection between two ortho images for check-point collection that must exist for relative assessment
Relative Minimum Intersection Unit	Unit of measure of Relative Minimum Intersection
Blunder Threshold	Absolute distance threshold over which check points are considered blunders and eliminated
Blunder Threshold Unit	Units of the blunder threshold
Local Mask Layer	Layer number of the local exclusion mask (typically, a cloud or UDM layer)
Global Mask File	Input PCIDSK file that contains a global exclusion mask (typically, a water mask)
Global Mask Layer	Layer number of the global exclusion mask

Parameter descriptions

Scene Folder

The path and name of the folder containing one or more orthorectified images to be assessed for positional accuracy.

If necessary, you can use a wildcard to filter the images. For example, you can enter *.JPG to assess only .JPG images in the folder.

The folder must contain at least one orthorectified image.

This parameter is mandatory.

Output Folder

The path and name of the folder to which to write the output files.

Overwrite Results

Select this check box to overwrite the existing output files, if any exist. If this check box is left clear, and an output file exists in the relevant folder, the status of the job displays a message informing you of the existence and name of the output file. The message is also written to the event log of the job.

Scale

Displacement lines provide a visual illustration of the offset of the source (ortho) image from the control image. These lines are typically very small and can be difficult to see. The Automatic Accuracy Assessment module generates a second displacement line, which is scaled according to the value specified herein. This can make the displacement lines much easier to see in the image overview.

For example, a scale value of 100 creates displacement lines with a length 100 times greater than the actual error being measured.

Note: This parameter creates two outputs: unscaled displacement lines and scaled displacement lines.

If no value is specified for this parameter, a default value of 100 is applied.

Assessment Mode

The mode of accuracy assessment to use: Absolute, Relative, or All.

Absolute: compares the position of pixels (features) in each input image with each valid control image (with overlap)
Relative: compares the position of pixels (features) in each input image with other input images with valid overlap
All: performs the absolute and relative accuracy assessments and creates the outputs of both

Measurement Mode

The mode of measurement to use: Locations or Pixel Values.

Locations: uses collected ground control points (GCP) to measure the positional accuracy of the image. This method is useful in assessing the geocoded accuracy of orthorectified imagery.
Pixel Values: compares pixel values at random locations and measures the differences. This mode is useful in comparing finite values across imagery, and was designed to verify the accuracy of DEMs. In this mode, the following parameters are ignored:
- Scale
- Sampling Method
- Matching Algorithm
- Search Radius
- Blunder Threshold
- Blunder Threshold Unit
- Local Mask Layer
- Global Mask File
- Global Mask Layer

Control Images or Chip Database

For use only when Assessment Mode is Absolute or All, the path and file name of a single control image or a path to a folder containing one or more control images. If more than one control image is provided, each must be in the same folder.

Alternatively, you can specify as input the file name of a chip database (.cdb). When you specify a chip database, the following conditions apply:

Measurement Mode must be Locations.
You need not enter or select a value for:
- Control Channel for Matching
- Sampling Method
- Check Point Samples
- Absolute Minimum Intersection
That is, the preceding will be ignored regardless of any value specified.

Control images are those orthorectified previously with known and acceptable positional accuracy. They are used to compare the positional accuracy of the input ortho images. Image-matching techniques are used to automatically match the pixels (features) of input ortho images with one or more of the control images.

A chip database is a compilation of individual image samples, called chips, usually measuring 256 pixels by 256 pixels or smaller.

Control Channel for Matching

The channel in the control image to use for matching when collecting check points. If no value is specified for this parameter, the first channel in the control image is used.

Master Matching Channel

The channel from the input image to use for automatic feature-matching when collecting check points. If no value is specified for this parameter, the third channel is used for images with three or more channels, and the first channel is used for images with one or two channels.

Sampling Method

The method to use to create check-point samples from the source imagery.

Available options are:

Grid: sample points are generated automatically, using sample points distributed evenly across the overlap region
Susan: sample points are generated automatically using the Susan corner-detection algorithm; this is the default value

The Susan and Grid options determine how to find the initial candidate positions in one image (the source image) for collecting sample points. The function builds a patch around each candidate position that it searches for in overlapping images.

The Susan option finds the candidates by running a corner-detection algorithm on the image, which looks for corner-like features to use as candidates.
The Grid option creates candidates in a grid-like pattern. This option tends to be faster, as it does not preprocess the image. However, it also finds fewer matches because the grid point might fall on a featureless flat patch somewhere in the image that cannot be matched to anything in the overlapping reference images.

For check-point collection, the Susan option is often preferred because it facilitates performing quality assurance on the collected check points: you can visualize a recognizable feature (perhaps a building corner or specific tree, for example) and compare it with the other image to see if it matches correctly.

Check Point Samples

The maximum number of check points to collect on each individual control image that intersects the ortho image being assessed.

If using the GRID option, available values are:

num_samples: sample points are generated automatically and distributed evenly across the overlap region. The overlap region is divided into a grid, and the candidates are placed in the center of the grid cells. num_samples defines the number of samples to use. The size of the grid cells is chosen automatically, based on the number of samples requested. num_samples is the default value.
num_samples, margin: sample points are generated automatically and distributed evenly across the overlap region. The distribution is similar to the previous GRID option, except the distribution of the points starts at the edges of the overlap rather than in the centre of the grid cells, which improves the chances of finding matches at the edge of the image. The margin value controls the minimum distance between the edge of the image and the placement of the candidates. Candidates that are too close to the image edge may not be matched properly, so adjusting the margin value may improve the result.

If using the SUSAN option, available values are:

num_samples: sample points are generated automatically using the SUSAN corner-detection algorithm. num_samples defines the number of samples to use.

The default number of samples to look for on each control image that overlaps the source is 25.

This parameter is optional.

Matching Algorithm

The algorithm to use for automated point matching.

Available options are:

Fast Fourier Transform Phase: when two images have a relative shift between them, the result is a phase difference in the frequency domain. Fast Fourier Transform Phase (FFTP) determines the shift between images using this phase difference.
Normalized Cross-Correlation: this method finds the relative shift between two images by finding the one that produces the maximum cross-correlation coefficient of the gray values in the images.

When the two images being matched have similar gray values and appearances, Normalized Cross-Correlation (NCC) generally produces acceptable results. When there is a rotation or image-size error in the initial math models, NCC may produce better matching results than FFTP. Typically, NCC also generates faster results, because the template size it uses is smaller than that used by FFTP.

For more consistently accurate results, FFTP is recommended. This method uses a larger template size than NCC and, because it works in the frequency domain, it looks at the patterns of details in the image rather than the gray values in a small neighborhood, which NCC uses. This makes FFTP more robust than NCC when there is a large difference in brightness between images or when a major land-use change has occurred between the images. FFTP also allows for a better match between images of the same area from different sensors or spectral bands.

Search Radius

The distance in the x and y directions from a starting location on the reference image over which the search for the best match with a fixed point on the input image is conducted.

The search radius is an estimation of error with the raw image's positional information and the digital elevation model (DEM) accuracy. If you know that your image is accurate to within 80 meters, and your DEM is accurate to within 200 meters, set the search radius to 280 meters. A larger search radius requires more processing time, because more locations are evaluated to determine the best match for a ground control point (GCP).

If this parameter is not specified, the function uses a default search radius based on the resolution of input data.

Absolute Minimum Intersection

The minimum amount of overlap that must exist between the source image (ortho) and control image for check-point collection to be performed on a given control image.

This parameter is only for the absolute assessment.

If no value is specified for this parameter, the module will default to accept all control images that intersect the source (ortho) image, regardless of the amount of overlap.

Absolute Minimum Intersection Unit

The unit of measure of the Absolute Minimum Intersection parameter.

Relative Minimum Intersection

The minimum amount of overlap that must exist between two source (ortho) images for check point collection to be performed between the two source (ortho) images.

This parameter is only for the relative assessment.

If no value is specified for this parameter, the module will default to collect check points on all source (ortho) images that intersect each other, regardless of the amount of overlap.

Relative Minimum Intersection Unit

The unit of measure of the Relative Minimum Intersection parameter.

Blunder Threshold

The absolute distance threshold over which check points are considered blunders and eliminated.

This parameter expects one number greater than 0.

Blunder Threshold Unit

The unit of the blunder threshold.

Available values are:

Pixels: the blunder threshold is assumed to be a pixel value. For example, if the value of the Blunder Threshold parameter is 5, the absolute distance is measured by 5 pixels. This is the default value.
Ground: the blunder threshold is assumed to be a ground-unit value. For example, if the input orthos are in a meter projection, the value will be assumed to be in meters.

Local Mask Layer

The PCIDSK layer that contains vector-exclusion polygons or bitmap masks for each input image.

Typically, the exclusion mask corresponds to a cloud mask layer generated from the Cloud Detection and Haze Removal module.

The Automatic Accuracy Assessment module will remove any check points that intersect the vector polygons or bitmap in the local mask layer. This ensures that the check points included in the accuracy-assessment report are valid.

Global Mask File

The path and file name of a global-mask file that contains vector-exclusion polygons for the entire project (dataset). Typically, these are water-body polygons.

The Automatic Accuracy Assessment module will remove all check points that intersect any of the polygons in the global-mask-vector layer.

Global Mask Layer

The layer (segment) number that contains the vector-global-exclusion mask.

If you specify a global-mask file, you must also specify a global-mask layer.

Details

General job details

This section describes the preprocessing requirements, module details, and job results, as applicable.

Preprocessing requirements

Before running this module, the following requirements must be met to ensure the job processes successfully and produces accurate results:

Input imagery: The input imagery is the data for which the positional accuracy will be assessed. The imagery must be in a supported format, such as .PIX or .TIF. Ensure that all of the input imagery you want to examine is in the same folder. Ideally, the imagery should be output from either the Orthorectification module or the Mosaic Generation module.
Control imagery: Ensure that the control imagery provided is accurately georeferenced and provided in a GDB-compatible format. A single control image covering the area of interest (AOI) or a set of control images may be provided. If a set of control images is provided, ensure that all images are in the same folder. For best results, ensure that the control imagery is well distributed throughout the AOI, and that the resolution is within 1:5 m of the resolution of the input imagery. Control imagery is required only when the accuracy assessment mode is Absolute or All.
Global masks: This is an optional input that allows for the removal of potential blunder check points captured on erroneous features that do not vary from day to day, such as water bodies. Check points that intersect polygons in the global mask will be removed. The global mask must be a vector-polygon layer provided in a supported GDB format, such as .PIX, shapefile, or another supported format.
Local masks: This is an optional input that allows for the removal of potential blunder check points captured on erroneous features that vary from day to day, such as clouds and or if pixels have no data. Check points that intersect polygons in the local masks will be removed. The local masks are polygon or bitmap layers provided as a segment directly in one or more of the input files.

Module details

The Automatic Accuracy Assessment module can assess the absolute accuracy, relative accuracy, or both. The absolute accuracy is a measure of how well the input (ortho) images are aligned to a third-party control source (imagery). The relative accuracy is a measure of how well the overlap areas of the input (ortho) images are aligned relative to each other.

The module assesses the absolute accuracy by automatically collecting check points on each input image from a single control image, or a set. The module assesses the relative accuracy by automatically collecting check points in the overlapping regions of the input images to assess the relative accuracy. A variety of outputs is provided to help with validating, analyzing, and summarizing the results.

Job results

The Automatic Accuracy Assessment module creates various files, according to the applicable assessment and measurement modes, which you can use to quickly and easily validate, analyze, and summarize the check points collected, as described in the following table. For example, the first four rows in the body of the table describe the files produced when Measurement Mode is Locations, and Assessment Mode is Relative.

Measurement Mode	Assessment Mode	Creates	Description
Locations	Relative	REL_QC.csv	An ASCII file of comma-separated values containing statistical information on the positional accuracy reported by each check point from the Relative assessment. You can use this file directly with the QC_Summary.xlsm file to calculate and organize the summary statistics for each image, and the entire project.
		REL_<Input_Image_Name>.pix	PCIDSK (.pix) link file for each input (ortho) image. Each contains the following layers: Input image channels: Similar in behavior to regular channels, but linked to the input ortho image. The number of channels depends on those in the input image. These layers can be used to verify check points with high residuals. Merged check-point segment: Appendage of all check points found in all control images that intersect with the input (ortho) image. Vector-displacement lines: Vector lines representing the radial-residual displacement of the check points collected in the overlap areas of the adjacent input ortho images.
		ORTHO_index.pix	PCIDSK (.pix) index file containing the vector footprints of one or more of the input images
		REL_summary.pix	Generated PCIDSK (.pix) file containing check points, displacement lines, and footprints for all overlapping ortho images: Check points collected on the input imagery. Check points collected on the output imagery. Scaled displacement between the input and reference check points Polygon footprints of all input scenes (equivalent to ORTHO_index.pix).
	Absolute	ABS_QC.csv	An ASCII file of comma-separated values containing statistical information about the positional accuracy reported by each check point. This file can be used directly with the QC_Summary.xlsm file to calculate and organize the summary statistics for each image, and the entire project.
		ABS_<Input_Image_Name>.pix	A PCIDSK (.pix) link file for each input (ortho) image. Each contains the following layers: Input image channels: Similar in behavior to regular channels, but linked to the input ortho image. The number of channels depends on those in the input image. These layers can be used to verify check points with high residuals. Merged CP segment: Appendage of all check points found in all control images that intersect with the input (ortho) image. Vector-displacement lines: Vector lines representing the radial-residual displacement of the check points collected in the overlap areas of the adjacent input ortho files.
		index.pix	PCIDSK (.pix) index file containing the footprints of one or more of the control images. That is, when this file exists in the Control Images folder, it is used as is. However, when the file does not exist in the folder, it is created only in the system's temporary folder. Tip: Rather than create an index file each time you run the Automatic Accuracy Assessment module, create the file with the Index PIX File Creator module.
		ORTHO_index.pix	PCIDSK (.pix) index file containing the footprints of one or more of the input ortho images.
		ABS_summary.pix	Generated PCIDSK (.pix) file containing check points, displacement lines, and footprints for all overlapping ortho images: Check points collected on the input imagery. Check points collected on the output imagery. Displacement between the input and reference check points. Polygon footprints of all input scenes (equivalent to ORTHO_index.pix). Polygon footprint of all reference imagery (equivalent to index.pix).
	All	Files merged from the output of Absolute and Relative modes.
Pixel Values	Relative	REL_<Input_Image_Name>.pix	A PCIDSK (.pix) link file for each input (ortho) image. Each contains the following layers: Input image channels: Similar in behavior to regular channels, but linked to the input ortho image. The number of channels depends on those in the input image. These layers can be used to verify check points with a high degree of change in pixel values. Matches: One vector segment for each reference image matched against.
		ORTHO_index.pix	PCIDSK (.pix) index file containing the footprints of one or more of the input ortho images.
		REL_summary.pix	Generated PCIDSK (.pix) file containing check points for all overlapping ortho images: Merged check points collected on all imagery. Unmerged check points collected on all imagery.
	Absolute	ABS_<Input_Image_Name>.pix	A PCIDSK (.pix) link file for each input (ortho) image. Each contains the following layers: Input image channels: Similar in behavior to regular channels, but linked to the input ortho image. The number of channels depends on those in the input image. These layers can be used to verify check points with a high degree of change in pixel values. Matches: One vector segment for each reference image matched against.
		ORTHO_index.pix	PCIDSK (.pix) index file containing the footprints of one or more of the input ortho images.
		ABS_summary.pix	Generated PCIDSK (.pix) file containing check points for all overlapping ortho images: Merged check points collected on all imagery. Unmerged check points collected on all imagery.
	All	Files merged from the output of Absolute and Relative modes.

Working with the results (Measurement Mode is Locations)

This section describes how to get the most out of the results of an accuracy assessment.

Generating summary statistics

On completion of an accuracy assessment, you should next create and review the summary statistics. These statistics provide insight into the overall positional accuracy of the project, the positional accuracy of each image, and which images require further inspection.

Note: To generate summary statistics, you must have the file qc_summary.xlsm. If you do not have this file, contact PCI Customer Support at [email protected].

To generate the summary statistics

From your output folder, open the file qc_summary.xlsm.
Note: For Absolute calculations, refer to the Absolute Accuracy table (shaded blue). For Relative calculations, refer to the Relative Accuracy table (shaded red).
Click Import.
The data on the collected check points is imported into the Absolute_CPs or Relavtive_CPs worksheet, as applicable.

Note: The file ABS_QC.csv, REL_QC.csv, or both is required to calculate the absolute and relative statistics, respectively, as created by the module. The .csv files must exist in the same folder as the file qc_summary.xlsm you opened in step 1.
Click Calculate.
The summary statistics are calculated for the entire project in the Project Summary Statistics table, and for each image separately in the Image Summary Statistics table.

Working with the results (Measurement Mode is Pixel Values)

When you run Automatic Accuracy Assessment module with Measurement Mode set to Pixel Values, All of the information is stored in attribute tables for each image. Each output vector segment (matches, merged, and so on) contains the structure shown in the following table.

Note: In the following table, the units associated with Measured value, Reference value, and Difference are dynamic. That is, the unit depends on what the pixel value represents. For example, with two DEMs measured in meters, the units would represent the elevation, in meters.

ShapeID	X	Y	Measured value	Reference value	Difference	Input image	Reference image
0	15.427	47.1045	36	56	20	<path_name>\a.pix	<path_name>\b.pix

Automatic Accuracy Assessment module

Description

Parameters

Parameter descriptions

Details