Catalyst.earth Logo
Catalyst.earth Logo

CATALYST Insights API via AWS Data Exchange

Generic Featured Image

Table of Contents

WORKFLOW OVERVIEW

Programmatically download Insights Ground Displacement sample data (Kings County) via AWS Data Exchange with Python.

REQUIREMENTS

Subscriptions

    • Amazon Web Services (AWS)
    • California InSAR Displacement – Kings County Sample

    Libraries and Scripts

    • Python 3.x

    SUBSCRIBING TO CATALYST INSIGHTS FREE SAMPLE DATA ON ADX

    1. Register for an AWS Account if you do not already have one.
      • Please note, a credit card is required to sign up. As you will note upon retrieving the dataset, however, is that it costs $0.
    2. Sign into your AWS account.
    3. In the main search bar, type insar.
    4. Under Marketplace, find the result titled California InSAR Displacement – King County Sample.
    5. Click Continue to subscribe button in the top right of your AWS console.
    6. An AWS Data Exchange window opens. Scroll to the bottom and click Subscribe.

      Note: Your subscription may take a few minutes to set up. You will be notified when your subscription is ready.

      Image of subscription success

      REVIEWING AND ACCESSING YOUR ENTITLED DATA IN BROWSER (OPTIONAL)

      1. Click on View Subscription.
      2.  Scroll down to the Entitled data sets section.
      3. Click on California InSAR Deformation – King County Sample.
      4. In the Products tree-list, select the latest revision.
      5. Click on the checkbox of the Asset you wish to download:
        • e.g. Kings_County_Sample_20201105_To_20210523.csv.
      6. Select the Exportaction you wish:
        • e.g. Download selected assets, or Export selected assets to S3.
      Image of download asset

      ACCESSING YOUR DATA PROGRAMMATICALLY WITH PYTHON

      1.Download requirements.txt and download-entitled-assets.py from GitHub.

      • To download the files, you must go to the parent folder by clicking Code in the top-left corner. Alternatively, you can simply copy the text of the files into a text editor and save them with the file names listed above.

      2. Open the download-entitled-assets.py file in a text editor.

      3. On line 11, change region_name to ‘us-east-2’.

        dx = boto3.client('dataexchange', region_name='us-east-2')

        4. Open a command console (Miniconda3 was used for this demo).

        5. Type pip install -r requirements.txt

        6. Set AWS Access Key and Secret (learn how to get your AWS Access Key ID and Secret).

        • set AWS_ACCESS_KEY_ID=
        • set AWS_SECRET_ACCESS_KEY=

        7. Run the python script you downloaded from GitHub, download-entiled-assets.py.

          #install requirements.txt local file
pip install -r requirements.txt

#Set AWS acces key ID and Secret (export command for unix)
set AWS_ACCESS_KEY_ID=<your-access-key-here>
set AWS_SECRET_ACCESS_KEY=<your-secret-here>

#Run the download-entitled-assets.py script you downloaded from github
download-entitled-assets.py

          8. Your download will now be accessible in your root folder.

            • The data is downloaded as a .CSV file in a new folder named California-InSAR-Deformation-Kings-County-Sample.
            • The data columns are structured as follows: Geopoint, Capture Time, Movement, and Velocity.
            Image of data structure
              envelopephone-handsetcross