Technical Support for Environmental Data Collection Software
Views: 196
Attachments: 0
Related Articles: 0
Feedback: 0
Helpful: 0
Not Helpful: 0
File Attachments
No attachments were found.

Decode Web Report

  • Author: Michael Maloney
  • Create Time: 10/06/2016 15:34:27
  • Last Update Time: 10/07/2016 08:28:51
DECODES can download reports from the web and decode the data therein. This article shows how to download hourly and daily reports for a group of stations, and then decode the data.

The following two URLs show the reports that we will decode:

http://www.meds-sdmm.dfo-mpo.gc.ca/ALPHAPRO/TWL/Hourly_Heights.txt

http://www.meds-sdmm.dfo-mpo.gc.ca/ALPHAPRO/TWL/Daily_Means.txt

In DECODES, we will use a WebDataSource to download these files in our routing specs.

The Hourly Report data looks like this:

Hourly Heights for 10050 THUNDER BAY                    IGLD(1985)  183.200
20161002
00  0.482 01  0.487 02  0.500 03  0.505 04  0.476 05  0.500 06  0.505 07  0.484
08  0.485 09  0.484 10  0.483 11  0.478 12  0.484 13  0.486 14  0.499 15  0.495
16  0.504 17  0.511 18  0.494 19  0.503 20  0.494 21  0.491 22  0.485 23  0.482
20161003
00  0.493 01  0.483 02  0.484 03  0.498 04  0.506 05  0.495 06  0.505 07  0.507
08  0.502 09  0.503 10  0.493 11  0.490 12  0.495 13  0.498 14  0.504 15  0.512
16  0.509 17  0.502 18  0.516 19  0.513 20  0.506 21  0.511 22  0.506 23  0.503
20161004
00  0.514 01  0.507 02  0.513 03  0.515 04  0.499 05  0.531 06  0.516 07  0.528
08  0.519 09  0.510 10  0.516 11  0.488 12  0.513 13  0.513 14  0.517 15  0.532
16  0.531 17  0.537 18  0.523 19  0.522 20  0.531 21  0.531 22  0.514 23  0.541
20161005
00  0.524 01  0.506 02  0.536 03  0.514 04  0.537 05  0.543 06  0.545 07  0.553
08  0.575 10  0.537 11  0.541 12  0.559 13  0.551 14  0.560 15  0.535 16  0.538
17  0.546 18  0.542 19  0.524 20  0.526 21  0.503 22  0.478 23  0.517
20161006
00  0.525 01  0.501 02  0.537 03  0.525 04  0.523 05  0.515 06  0.498
Hourly Heights for 10220 ROSSPORT                       IGLD(1985)  183.200
20161002
00  0.429 01  0.425 02  0.424 03  0.419 04  0.429 05  0.423 06  0.435 07  0.440
08  0.429 09  0.440 10  0.430 11  0.435 12  0.425 13  0.441 14  0.441 15  0.440
16  0.434 17  0.433 18  0.446 19  0.442 20  0.441 21  0.453 22  0.434 23  0.434
20161003
00  0.424 01  0.430 02  0.430 03  0.419 04  0.434 05  0.423 06  0.430 07  0.435

We will have the Web Data Source pick out each platform's data and process each block as a separate 'message'. Notice that each block begins with the string "Hourly Heights for" (highlighted above). We will use this as the start delimiter in each message. (More on this below.)

Transport Medium Types for the Reports
 

Thus, at the start of each "message", we see a 5-digit unique identifier for the platform. We will use this as the transport medium ID.

To start, we will set up two new Transport Medium Types, one for each of the report types.  Start the Reference List Editor with the 'rledit' command. On the enumerations tab, set up two new types as shown below:




Set the Executable Java Class exactly as shown: "decodes.datasource.IdStartPMParser". This code tells DECODES that the ID (the 5 digit identifier) is at the start of the message. After creating the two new transport medium types, hit File - Save to DB. Then exit the reference list editor.

Web Data Source Records

Next, start the DECODES database editor. On the Sources tab, create a new data source records for the Hourly report. We called it "ALPHAPRO-Hourly".




Note the property settings:
  • header: set to "alphapro-daily" or "alphapro-hourly". This tells DECODES how to parse the header of each message to extract the station ID.
  • mediumType: Set to the same as header. This tells DECODES what Transport Medium record to match in the platform record.
  • OneMessageFile: Set to false. The file (web report) contains many messages that we will delimit as described above.
  • url: Set to the URL for the web report.
  • before: This defines the start-delimiter for each message. Set to "Hourly Heights for\040". The "\040" is the ASCII code for a space character.

Now do the same for the daily report. The only difference is the URL and the "before" property:



DECODES Configuration

A single DECODES configuration record will suffice for both reports. We will have separate scripts for each report. If this were a GOES DCP, you could also have a script to handle the GOES messages.

Create a configuration. We called it "ALPHAPRO". Add the two sensor records for hourly and daily water level.



Now hit Commit to save the sensors. Then hit 'Add' in the lower right and create a script called 'hourly'. Then ...

  1. Set Data Order to Ascending and Header Type to "alphapro-hourly".
  2. Define the units in the Sensor Unit Conversion (We assume meters since this is a Canadian report.)
  3. On a web browser, bring up the hourly report and Copy/Paste a single "message" into the Sample Message area. Recall that a "message" starts with the 5-digit ID after the text "Hourly Heights for".


Write the script as shown above.

   date      F(YR,A,4,F(MN,A,2),F(DY,A,2), >data

The 'date' statement parses the date field (e.g. "20161002"). Since this doesn't correspond to one of DECODE's canned date formats, we have to parse YR, MN, and DY separately. If successful, we jump to the "data" statement. If an error occurs in parsing a date field we jump to the "error" statement, which simply skips the line and tries again on the next line.

   data       3(/, 8(w, F(HR,A,2), w, F(S,A,8d' ', 1))), /, >date

The data line parses the 3 lines of data. Within each line, there are 8 values. For each value we have an HR field followed by a sensor value. After the 3rd line, we jump back to the "date" line to get the next day.

After you write the script, you can hit the 'Decode' button and see the successfully decoded data at the bottom of the screen.

Site Records

 
Create a DECODES Site record for each site in the report that you want to process. We created two with local names "THUNDER_BAY" and "ROSSPORT". Only the name is necessary. All other Site fields may be left blank.

Platform Records

 
Create a Platform record for each site.
  1. Select the Site that you created above
  2. Select the ALPHAPRO Configuration that you created above.
  3. Click Add in the lower right to create a transport medium. Select medium type "alphapro-hourly", Decoding script "hourly", and enter the unique medium ID 10050.



Routing Spec Records

 
After you have created both Platform records, you are ready to create a Routing Spec record that will tie everything together. Below we describe the Routing Spec for the Hourly report. The Daily routing spec is left as an exercise for the student.

  1. Select the ALPHAPRO-Hourly data source. Recall that this will call up the web report and parse it into separate "messages" by the start delimiter.
  2. For now, select Destination "pipe" and Output Format "human-readable". This will print the decoded data to the screen. After you are satisfied that everything is working, you can change these to the output format and destination you require (e.g. CWMS).
  3. Add your platforms in the list at the bottom. NOTE: you could also create a network list of these platforms and then add the network list here.


Finally, test the routing spec with the 'rs' command:

    rs ALPHAPRO-Hourly

 You should see the formatted data print to the screen. If you do not, look at the log file in your 'routstat' directory for a clue as to what went wrong. If needed, you can also run the 'rs' command with a -d3 argument for more debugging information.

Scheduling

 
You probably want these reports to be fetched and decoded on a schedule. The snapshot below shows a schedule entry that will run the Hourly report every hour at 10 minutes after the hour (We added a delay to ensure that the hourly data is present on the web site).

 

Post Feedback
This article is:
Name: *
Email: *
Comment:
Verification Code: *
*Required Field
Related Articles
No Related Articles were found.
Feedback
No Feedback was found.