Technical Support for Environmental Data Collection Software
Views: 450
Attachments: 0
Related Articles: 1
Feedback: 0
Helpful: 0
Not Helpful: 0
File Attachments
No attachments were found.

Decode USGS Web Download

  • Author: Michael Maloney
  • Create Time: 01/06/2015 13:05:20
  • Last Update Time: 01/07/2015 01:27:28
This article will explain how to set up data-source, configuration, site, platform, and routing spec to decode data downloaded from USGS's website. The URL we will be using as a test is:

http://waterservices.usgs.gov/nwis/iv/?format=rdb&sites=08041749&period=P1D&parameterCd=00060

Note the USGS Site Number 08041749 in the URL. We will use that as the DECODES Transport Medium ID.

Step 1: Create the Data Source Record

On the "Sources" tab of the DECODES Database Editor create a new data source as shown below.
The important items are:
  • Select "abstractweb" for "Type" in the upper left.
  • Set property "OneMessageFile" to true. This tells DECODES that the entire file should be treated as a single message, rather than a collection of messages with some delimiter.
  • Set property "header" to "other". This tells DECODES to not make any assumptions about a formatted header at the top of the file.
  • Set property "abstractUrl" to the web site's URL. See in the figure below that the value is being edited. Note that ${MEDIUMID} will be replaced by the USGS number when the routing spec runs.



Step 2: Create a Site Record

On the "Sites" tab, create a record as shown below. You can add other descriptive information and other names that you need. The minimal info in a site record is a single name. We have entered the USGS Site Number as a name. If you are using CWMS, your site should also have a CWMS name.



Step 3: Create the Configuration

On the "Configs" tab, create a new configuration, or if this is a site that also uses GOES or some other means to collect data, simply add the needed sensors to the existing configuration. The web download we are processing contains a single sensor, which we have called "Discharge".

At the bottom of the config screen, create a new DECODES script called "web". Create the script as shown and test the decoding until it is working. Of note on this screen:
  • Again, set Header Type to 'other'.
  • The script has two lines: CheckLine checks for 'USGS' at the start of a data line. ParseLine decodes the line once that is found.
  • Note that the two lines with the same label "ParseLine" are really continuations of the same line. It's okay to break up long lines like this.
  • The columns in the file are tab-delimited. '\t' means a tab character.
  • The ParseLine script skips to the Date Field, then the Time Field, then the Data Field.



Step 4: Create a Platform Record


On the "Platforms" tab, create a new platform. Or if this is an existing GOES platform, open it and create the new Transport Medium for the web download. Assign the platform to your Site and Config records.



The Transport Medium record is shown below. Note the following:
  • Medium Type: other
  • Medium Identifier is the USGS Site Number. This will replace ${MEDIUMID} in the URL.
  • Select the "web" decoding script.
  • Set Timezone "CST" to match the date/time values in the downloaded file.




Step 5: Create a Network List

On the Network List Tab, create a new list for all of your USGS web downloads. Again, note that we select Transport Medium Type "other". You can select whatever Site Name Type Preference you like. Click Select Platforms and add the platform record that we created above.



Step 6: The Routing Spec

On the Routing Spec Tab create a new Routing Spec. This ties everything together. Select your Web Data Source, whatever desitination you want, and add the network list.



You run the routing spec either under the scheduler or with the "rs" command as shown below:



After running, you can find the log file in the routstat directory under the installation.


Post Feedback
This article is:
Name: *
Email: *
Comment:
Verification Code: *
*Required Field
Related Articles
Feedback
No Feedback was found.