WORKFLOW OVERVIEW
Programmatically download Insights Ground Displacement sample data (Kings County) via AWS Data Exchange with Python.
REQUIREMENTS
Subscriptions
- Amazon Web Services (AWS)
- California InSAR Displacement – Kings County Sample
Libraries and Scripts
SUBSCRIBING TO CATALYST INSIGHTS FREE SAMPLE DATA ON ADX
- Register for an AWS Account if you do not already have one.
- Please note, a credit card is required to sign up. As you will note upon retrieving the dataset, however, is that it costs $0.
- Sign into your AWS account.
- In the main search bar, type insar.
- Under Marketplace, find the result titled California InSAR Displacement – King County Sample.
- Click Continue to subscribe button in the top right of your AWS console.
- An AWS Data Exchange window opens. Scroll to the bottom and click Subscribe.
Note: Your subscription may take a few minutes to set up. You will be notified when your subscription is ready.
REVIEWING AND ACCESSING YOUR ENTITLED DATA IN BROWSER (OPTIONAL)
- Click on View Subscription.
- Scroll down to the Entitled data sets section.
- Click on California InSAR Deformation – King County Sample.
- In the Products tree-list, select the latest revision.
- Click on the checkbox of the Asset you wish to download:
- e.g. Kings_County_Sample_20201105_To_20210523.csv.
- Select the Exportaction you wish:
- e.g. Download selected assets, or Export selected assets to S3.
ACCESSING YOUR DATA PROGRAMMATICALLY WITH PYTHON
1.Download requirements.txt and download-entitled-assets.py from GitHub.
- To download the files, you must go to the parent folder by clicking Code in the top-left corner. Alternatively, you can simply copy the text of the files into a text editor and save them with the file names listed above.
2. Open the download-entitled-assets.py file in a text editor.
3. On line 11, change region_name to ‘us-east-2’.
dx = boto3.client('dataexchange', region_name='us-east-2')
4. Open a command console (Miniconda3 was used for this demo).
5. Type pip install -r requirements.txt
6. Set AWS Access Key and Secret (learn how to get your AWS Access Key ID and Secret).
- set AWS_ACCESS_KEY_ID=
- set AWS_SECRET_ACCESS_KEY=
7. Run the python script you downloaded from GitHub, download-entiled-assets.py.
#install requirements.txt local file
pip install -r requirements.txt
#Set AWS acces key ID and Secret (export command for unix)
set AWS_ACCESS_KEY_ID=<your-access-key-here>
set AWS_SECRET_ACCESS_KEY=<your-secret-here>
#Run the download-entitled-assets.py script you downloaded from github
download-entitled-assets.py
8. Your download will now be accessible in your root folder.
- The data is downloaded as a .CSV file in a new folder named California-InSAR-Deformation-Kings-County-Sample.
- The data columns are structured as follows: Geopoint, Capture Time, Movement, and Velocity.