IOC

WMO

SOOP Data Management


  1. Data distribution and dissemination (how to get the data?)

    1. Global Temperature and Salinity Profile Project (GTSPP)
      • The data management activities of the program continue to be undertaken in collaboration with the Global Temperature and Salinity Profile Programme (GTSPP) and the AOML GOOS Center (UOT/DAC). GTSPP continues to support the real-time data exchange and quality control mainly via ISDM , Canada. This is the data provided for operational applications, such as ENSO prediction. The Science Centres (AOML, SIO, CSIRO/BMRC JAFOOS), which are operated by the GTSPP, are responsible for the scientific quality control and delivery of products from the high-resolution, delayed mode data set. This data set is managed by the global archive for both programmes (NODC), as is used for scientific research and Climatology development. NODC agreed to make developments to permit access to the data via the Distributed Oceanographic Data system (DODS) and will therefore establish a DODS server in the next 6 to 12 months. Data distributed via DODS (in NetCDF) will be the best current copy of GTSPP data and will be updated on a monthly basis.

        WOCE produced an Upper Ocean Thermal data CD-Rom (current version is version 2) which contains all SOOP observations collected until 1999 plus other types of upper ocean thermal data (e.g. TAO). A version 3 of this CD-Rom will soon be published. All data on the CD will be corrected for XBTs for which old fall rate equation coefficients were used. 1999 and 2000 SOOP data will be included in the CD-Rom although not all of these data sets could be quality controlled by science centres in time (this will be documented in the CD). Data on the CD-Rom will be provided in NetCDF format.

        Real-time data are sent from MEDS to US/NODC and other clients three times each week. Real-time data provided by MEDS to clients are either for the whole world or a a specific area. Both MEDS and US/NODC have clients that receive regular dispatches of data. Once a year, all data collected two years previously are divided into three oceans and forwarded for scientific QC in the US (AOML and Scripps) and Australia, (CSIRO/BoM/Joint Australian Facility for Ocean Observing Systems (JAFOOS)). The results are returned to US/NODC and updated into the archives. These centres also contribute to the WOCE DAC activities.

        GTSPP has developed a number of users of the data. MEDS has about 6 users that receive data three times a week. The US/NODC has more than a dozen users receiving data either weekly or monthly.

        The data volumes in the continuously managed database are about 1 million stations from 1990 to 1999. A significant fraction of data is still received as real-time reports. Increasing the timeliness of flow of delayed mode data to the archives is still an important function that needs attention, including by SOOPIP.

        SOOP is a significant contributor of data in both real-time and delayed mode. In support of SOOPIP, GTSPP makes available monitoring reports such as data sampling information on a monthly basis. The data quality statistics are posted here each month. Also shown are maps of the sampling both in the past month and over the past twelve months.

        Getting the data:

    2. Real time GTS distribution

      GTSPP provides real-time data exchange of quality controlled data via MEDS , Canada. SOOP data are also distributed in real-time onto the Global Telecommunication System (GTS) of WMO for insertion into coupled ocean-atmosphere numerical models although the level of QC of GTS data is limited (mainly simple automatic quality control checks). Temperature and salinity profile data that are transmitted in near real-time are sent as BATHY and TESAC reports respectively. These BATHY and TESAC messages are sent by radio or satellite to Shore Stations which places the data reports onto the Global Telecommunications System (GTS). GTS bulletin headers (see below) permit to identify these sources. Only lower resolution data (limited vertical resolution, 0.1C temperature resolution for BATHY, and 0.01C for TESAC) are distributed on GTS for the time being. It is planned to distribute higher resolution data in real-time via GTSPP (better QC, NetCDF format).

      1. What is GTS?
      2. GTS codes
      3. GTS routing, GTS bulletin headers
      4. Monitoring




  2. Quality control

    Extensive programme monitoring and data quality activities continue to be implemented in conjunction with the Global Temperature Salinity Profile Programme (GTSPP). Feedback mechanisms have been instigated to ensure data flow and quality. Once a year, all data collected two years previously are divided into three oceans and forwarded for scientific QC in the US (AOML and SIO), and Australia (JAFOOS). These mechanisms have proven to be effective in increasing the amount of real-time data flow and data quality. Delayed mode data submissions are also being tracked to increase the amount of high-resolution data being made available to the global archives in a timely manner. GTSPP is introducing a data state indicator scheme where it will be easier for the users of the data to identify the type of data processing and quality control which was applied to the data. 3 levels are proposed (0 raw, 1 geo-physical units independent of instrument knowledge, 2 complete geo-spatial and temporal references, data may be averaged, sub-sampled but no assumptions of scale or variability or thermodynamic relationships were made, 3 assumptions were made, data reduced to regular space, time intervals). 3 classes were proposed as well as sub-classes for each level (e.g. class A no scrutiny, value judgement, B data scrutinized automatically, C data scrutinized scientifically). The use of unique data tags associated to every observation will also permit to filter duplicates out more easily.

    Deferred time guidelines: SOOP is making available a
    Quality Control Cookbook for XBT data. The Quality Control (QC) procedures described in the cookbook have been developed by oceanographers participating in the CSIRO XBT Program, specifically to assure the research quality of expendable bathythermograph (XBT) data of 0.6 to 2.0 m resolution, at the delayed mode stage. This manual provides an overview of real oceanographic features and instrument errors that have been monitored over a period of years and from which the CSIRO procedures and QC codes (flags) have been developed. An example of each flag is provided to illustrate the subtle differences in structure that occur when either an error or feature is recorded in the XBT profile data. We believe that checking XBT data for the occurrence of these errors and features, combined with comparisons to climatological data and to neighbouring profile data, forms the basis of scientific quality control. The ability to judge whether a feature is realistic for a particular region cannot easily be passed on to a QC operator from a manual such as this. Local knowledge of water mass structure, statistics of data anomalies, the depth and gradient of the thermocline, as well as reference material from atlas data must be incorporated into the validation process. These ingredients all help to establish if the specific event (e.g., an inversion) identified in the profile is probable or not for the region in which it was observed. The potential types of malfunctions of the XBT instrument must also be kept in mind when performing QC. If uncertain, the QC operator should proceed with caution whilst ensuring that no good data is thrown away. The type of flag to assign under these circumstances will depend on the type of error or feature, and the appropriate action to take will become clearer as each flag is discussed in detail. This "cookbook" is intended to make the identification of real and erroneous features easier. It must be kept in mind, however, that the ability to make valid scientific QC decisions requires training under expert supervision. These procedures can also be applied to low to medium resolution, inflection point XBT data. However, the lower resolution makes it more difficult to identify the structures described here.

    Real time guidelines: The GTSPP real-time Quality Control Manual (
    WMO-IOC manual and guides No. 22) provides guidance to ensure that the quality of the temperature and salinity profile data, coming from different sources and types of instruments (e.g. water samplers, CTDs, XBTs, thermistor chains from data buoys, thermosalinographs), is consistent in the whole GTSPP data-set. The purpose of this Manual is to set standards for quality control of real-time data and to describe exactly the screening process that is employed. By reading this document, users may assess the applicability of the procedures to their requirements and thereby judge whether they need do further work before using the data. The Manual therefore describes procedures that make extensive use of flags to indicate data quality. GTSPP participants agreed to permit the selection of data from their archives based on quality flags as well as other criteria. These flags are always included with any data transfers that take place. Because the flags are always included, and because of the policy regarding changes to data, as described in the manual, a user can expect the participants to disseminate data at any stage of processing. Furthermore, GTSPP participants have also agreed to retain copies of the data as originally received and to make these available to the user if requested. Note that the implementation of the tests in the Manual requires interactive software to be written.

    The "Guide to operational procedures for the collection and exchange of JCOMM oceanographic data" (WMO-IOC manual and guides No. 3, UNESCO 1999) is a general guide to the operational procedures for the collection, encoding, quality control and exchange of oceanic surface and sub-surface temperature, salinity and current (BATHY, TESAC and TRACKOB) data. The guide defines notions of operational data, non-operational data, timeliness, oceanographic product, operational product, observational reports, and GTS bulletins. It provides general information on instrumentation being used as well as relevant observing platforms such as ships and buoys. Space/time sampling frequencies are also detailed. Finally, the guide provides practical information regarding data encoding and routing for real time distribution as well as specific recommendations before transmission and upon reception of the data to ensure a good level of quality for distributed data and products. Among recommended operational error checks are (i) bulletin format check, (ii) GTS report format check, and (iii) geo-physical values check. Information is provided regarding the type of quality control provided with the data acquisition systems onboard the ships.


  3. Data Formats

    1. GTS Formats (formal description in WMO manual on codes)
      1. BATHY (see also previous version of BATHY code)
      2. TESAC (see also previous version of TESAC code)
      3. TRACKOB (Thermosalinograph data mainly)
      4. WMO Code tables for BATHY, TESAC, TRACKOB
      5. BUFR
      6. BUFR Tables
    2. NetCDF
    3. NetCDF (as coded for WOCE UOT CD-Rom)
    4. MEDS ASCII Ocean Profile data format
    5. GTSPP Code tables



See also SOOP monitoring activities.