Data Ingest module

Description

The Data Ingest module reads the source folder for raw image scenes and creates a data-ingest job for each scene found. The module imports raw satellite sensor data into PCIDSK format, allowing it to be processed through other CATALYST Enterprise modules.

Parameters

Name	Caption
Send Email	Email notification settings
Input Scenes	Input scene folder
Output Folder	Output folder
Ingest Method	Link or import data
Build Pyramids	Builds pyramids
Create GCP/TP Exclusion Mask	Create an exclusion mask
Overwrite Results	Overwrite existing results
Ingest Stereo Data	Ingest stereo data
Tri-Stereo Look Directions	View or views to ingest
Ingest Multispectral	Ingest multispectral data
Ingest Panchromatic	Ingest panchromatic data
Ingest Pansharpened	Ingest pansharpened data
Ingest Other	Ingest other data
Math Model Filter	Input math model filter

Parameter descriptions

Send Email

If necessary, you can set up CATALYST Enterprise to send an email notification on job start and job completion.

With this check box selected, an email message is sent to each address specified in the Email Addresses box after the job starts and on completion.

You can specify one or more addresses, and each must be separated by a comma or a semi-colon. The email address of the user currently logged in displays by default.

Input Scenes

The path and name of the folder containing RAW satellite scenes to ingest.

The specified path must point to a folder containing raw imagery scenes as acquired from the satellite vendor. The folder must not contain a mixture of sensor types; rather, keep multiple sensor types separate, as the module is capable of traversing nested folders.

This parameter is mandatory.

Output Folder

The path and name of the folder to which to write the output files.

Ingest Method

Select whether to link or import the data specified in the scene folder.

You can select from the following:

Automatic: Links or imports the data based on the sensor.
Import: Creates from the file in its raw vendor format a PCIDISK file that contains the input scene.
This method may take longer to ingest the data; however, it improves the overall performance of the workflow.

Important: The Import method requires twice the amount of disk space, because it duplicates the pixel data.
Link: Creates a PCIDISK file that points to the input scene.
This method ingests data very quickly; however, subsequent steps of the workflow may take longer to complete.

Build Pyramids

This check box controls whether to build image pyramids (raster overviews) for the ingested scenes.

Building pyramids improves performance in some subsequent processing steps. They can use the pyramids instead of reading all the pixels. However, building pyramids can cause the ingest job to take longer to complete. If immediately after ingestion you intend only to create orthorectified images, you need not build pyramids.

When to turn on pyramids:

View or edit the data in: CATALYST Professional Focus
Perform some tasks in: CATALYST Professional OrthoEngine
Run any of the aerial triangulation jobs:

For more information on pyramids, see the Pyramid module.

Create GCP/TP Exclusion Mask

Selected by default, this check box controls whether to create a bitmap exclusion mask for each multispectral (MS) scene.

An exclusion mask prevents ground control points (GCP), tie points (TP), or both from being created in areas identified by the mask when you run a module that performs GCP collection.

For more information about using an exclusion mask, see the GCP/TP Exclusion-Mask Generation module.

Overwrite Results

Select this check box to overwrite the existing output files, if any exist. If this check box is left clear, and an output file exists in the relevant folder, the status of the job displays a message informing you of the existence and name of the output file. The message is also written to the event log of the job.

Ingest Stereo Data

When selected, this check box ingests stereo data, if any exists in the scene folder.

You can use this parameter in conjunction with the Tri-Stereo Look Directions parameter to select only the specific view or views you want to ingest.

Tri-Stereo Look Directions

Available only when the Ingest Stereo Data check box is selected, and when the data is tri-stereo, such as ZY-3 scenes, select from the list the specific view or views you want to ingest.

Available options are:

All (Nadir, Forward, Backward)
Forward, Backward
Nadir, Forward
Nadir, Backward
Nadir
Forward
Backward

Ingest Multispectral

Selected by default, this check box controls whether to ingest multispectral data, if any exists in the scene folder.

Ingest Panchromatic

Selected by default, this check box controls whether to ingest panchromatic data, if any exists in the scene folder.

Ingest Pansharpened

Selected by default, this check box controls whether to ingest pansharpened data, if any exists in the scene folder.

Ingest Other

To ingest other types of data, if any exists in the scene folder, select this check box.

Math Model Filter

The math model to use for collection of ground control points (GCPs) and, subsequently, orthorectification.

Available options are:

Automatic: available in data-ingest-related modules only, such as the Data Ingest and GCP Collection module. With this option selected, any and all supported math models is added to the output file for each scene. Typically, only the selected math model is added to the output files.
Rational Function Model (RPC): uses the Rational Polynomial Coefficients (RPCs) that are distributed with the raw imagery. With most modern sensors, this is the most robust model to use, as the distributed RPCs are very well calibrated (accurate). Therefore, fewer GCPs are required, generally, and positional improvements are not constrained to areas within the outer bounds of the GCPs collected. Other abbreviations for the Rational Function Model include RF, RFMODEL, RFI, RF0, RF1, and RF2.
Toutin's Satellite Orbital Model (Rigorous): CATALYST Enterprise uses the Toutin rigorous math-model solution, which calculates the position and orientation of the sensor at the time of image capture from an orbital or ephemeris segment. When this information has been identified, it can then be used to accurately account for known distortions in the image. After the sensor orientation and position have been calculated, this information can be used to drive other processes, such as orthorectification. Other abbreviations for Toutin's Satellite Orbital Model include SAT, RIG, SRIT, and TOUTIN.

The module first attempts to use the selected math model. If required information is missing from the user-defined math model, the module automatically tries to use another math-model option, and displays a warning message.

Details

General job details

This module provides a series of predefined configuration files that define the type of output to create. These are defined in the settings.py file in the PROHOME\exe\PGS\config folder of your CATALYST Enterprise installation. In the configuration file, you can specify whether the output file is a linked file (in PCIDSK format) or a standalone file.

Module details

With the Data Ingest module, you can automatically import raw satellite sensor data into the PCIDSK format. This is a mandatory step in the CATALYST Enterprise workflows, and allows a multitude of actions to take place, including:

Ingesting the nominal math model provided with the raw sensor data (orbital model, RPC model, and so on)
Setting default metadata to the ingested PCIDSK file

This module supports a multitude of sensors and data products. With each data product, the module can import the multispectral, panchromatic, and pansharpened (raw) images.

For each scene in the Input Scenes folder, the Data Ingest module creates:

a PCIDSK file containing the multispectral image bands
a PCIDSK file containing the panchromatic image band
a PCIDSK file containing the pansharpened multispectral bands (when the Input Scenes folder contains raw pansharpened data)

Restriction: When running this module on a Linux operating system, do not use circular symbolic links, because this will cause the job to loop continuously. For example, when you ingest the contents of folder /abc, make sure the following does not exist:

/abc -> /def
/def -> /xyz
/xyz -> /abc

Job results

For each raw scene found in the specified Input Scenes folder, the module creates a PCIDSK file with an automatically generated file name in the specified output folder. Each file name contains a scene ID, a state (includes a _RAW tag in the file name) and a type (such as _PAN, _MS, or _PSH).

The following is an example of images imported to PCIDSK using the Data Ingest module.

IKONOS Geo Ortho Kit – NITF Format
po_199802_blu_0070000.ntf
po_199802_grn_0070000.ntf
po_199802_nir_0070000.ntf
po_199802_red_0070000.ntf
po_199802_pan_0070000.ntf

	|
	| 
	V

po_199802_ 0070000_RAW_MS.pix
po_199802_ 0070000_RAW_PAN.pix

The output folder and its files follow a naming convention based on the input-scene ID, the current state of the image product, and the image type.

A typical IKONOS scene ID is:

po_199802_pan_0070000.ntf

The output file name resembles the following schema:

<sceneid>_RAW_<type>.pix

where:

sceneid: the image scene ID (for example po_199802_0070000)
type: the image type (for example, PAN, MS, PSH)

For example:

po_199802_ 0070000_RAW_PAN.pix

When the Create GCP/TP Exclusion Mask check box is selected, a file-level metadata tag, NominalCloudCoverPercentage is created during processing. The tag provides the percentage of cloud cover identified in an image. When an image has a cloud-cover percentage greater than 30 precent, a warning message is displayed.

The module also automatically creates OrthoEngine project files for each type of scene ingested successfully. For example, when multispectral and panchromatic data is ingested, two project files, data-ingest_RAW_MS.prj and data-ingest_RAW_PAN.prj are created in the specified output folder.