NOAA S-111 Surface Water Currents Data on AWS

Overview 🔗︎

Forecast guidance of surface water currents from NOAA’s operational oceanographic forecast modeling systems are now available in S-111/HDF5 format on Amazon S3. These files are part of a preliminary service for delivering surface water currents for marine navigation systems. The goal of this service is to provide manufacturers with samples of S-111 surface water currents files to test and implement into their navigational systems.

This service is made possible through a public-private partnership enabled by the NOAA Open Data Dissemination Program.

Accessing the Data on AWS 🔗︎

The Amazon Resource Name (ARN) for the S3 Bucket is: arn:aws:s3:::noaa-s111-pds

The Amazon S3 Bucket URI is: s3://noaa-s111-pds

AWS S3 provides direct HTTP access to the files, so datasets can be downloaded using any HTTP client such as a web browser. To make discovering and accessing the data easier, a simple bucket explorer web application is also provided which allows you to navigate through the object key structure and download files using a web browser.

Access to the datasets is being provided on Amazon Web Services (AWS) via Simple Storage Service (S3). Files can be downloaded via the native AWS S3 API using tools such as the AWS CLI or AWS SDK libraries such as the Python boto3 package.

Common AWS CLI Access Examples (No AWS account required):

  • To list all files in the noaa-s111-pds bucket: 
    	aws s3 ls --no-sign-request s3://noaa-s111-pds/
  • To copy all files in the noaa-s111-pds bucket to your local machine: 
    	aws s3 sync --no-sign-request s3://noaa-s111-pds/ /path/to/local/directory/

Additionally, an AWS Simple Notification Service (SNS) Topic has also been created to provide automated notifications when new files are added to the S3 Bucket. Users who operate their own infrastructure on AWS can integrate directly with this SNS Topic to trigger automated workflows using AWS Lambda or other services.

The Amazon Resource Name (ARN) for the SNS Topic is: arn:aws:sns:us-east-1:123901341784:NewS111Object.

Format 🔗︎

S-111 is a data encoding standard which is part of the International Hydrographic Organization's (IHO) S-100 Universal Hydrographic Data Model. Individual S-111 files are encoded in S-111 Standard. Because S-111 is built upon HDF5, files can be read and written using existing software such as the official HDF5 libraries/utilities and h5py.

For the purpose of supporting marine navigation, surface currents can be defined as the horizontal motion of the water at a depth which directly affects surface vessels. The S-111 files in this data collection contain predictions of surface currents calculated at a depth of 4.5 meters below water surface, or half the distance to the seafloor, whichever is shallower.

The S-111 standard supports encoding the data using any of the following Data Coding Formats (DCFs):

  • 1: Time-series at fixed station
  • 2: Regularly-gridded arrays
  • 3: Ungeorectified gridded arrays (i.e. irregular grid)
  • 4: Time-series for moving platform

Presently, this collection provides S-111 files encoded in both DCF-2 - Regularly-gridded arrays ("regular grids"), and DCF-3 - Ungeorectified gridded arrays (i.e. irregular grid) in order to allow users to evaluate both formats.

In most cases, the underlying data sources (modeling systems) generate predictions at irregular or unstructured gridpoints placed at varying spatial resolutions. In order to promote interoperability, these values have been interpolated to regular grids at a defined resolution, then encoded into S-111 DCF-2 (regular grid) format. These DCF-2 files are being generated for the entire model domain (stored in the S3 bucket under the "regional/" object-key structure), but output is also being subset into tiles and encoded into individual DCF-2 files (stored in the S3 bucket under the "tiles/" object-key structure). This tiling scheme is designed to keep file sizes small and to allow users to obtain data for only their area of interest, with each tile covering a small geographic area corresponding with the OCS Nautical Product Tiling Scheme, thus matching the coverage areas of other OCS products that will be disseminated in the future.

In addition to the DCF-2 tiled and regional files, DCF-3 (irregular grid) regional files are also being produced to provide surface current model forecast guidance at the original horizontal resolution of the underlying modeling system for users who would prefer that format.

Multiple data encoding options are being provided for evaluation because OCS recognizes that some users may prefer one format over the other depending on their specific use-case. In the future, it is likely that the resolution of the DCF-2 (regular grid) S-111 files will increase to more closely match the underlying model resolution. Additional S-111 files encoded in different Data Coding Formats may also be added in the future.

Data Sources 🔗︎

The National Oceanic and Atmospheric Administration (NOAA) operates a set of 3-D Hydrodynamic Modeling Systems, commonly referred to as Operational Forecast Systems (OFS), which produce predictions for various ocean conditions such as water currents, salinity, water temperature, and water level. The majority of these OFS are run four times per day and generate forecast guidance for the next 36 to 48 hours (and out to 120 hours for two of the Great Lakes OFS). The S-111 files in this data collection are generated by extracting, postprocessing, and interpolating the forecast guidance output from these modeling systems to produce predictions of surface water currents, which are then encoded in the S-111 HDF5 format.

S-111 Surface Water Currents datasets are provided from the following forecast modeling systems:

Data Organization 🔗︎

S-111 datasets are organized in the S3 bucket using an object-key structure similar to a directory hierarchy. This key structure includes the version (edition) of the S-111 standard with which the files are compliant, the data source (i.e. a particular OFS), the date/time of the model forecast run cycle (i.e. model initialization time), the S-111 data coding format (e.g. 'dcf2' == Regular Grid), and the geographic coverage (e.g. small individual tiles vs. entire region) of the files.

Note: This key structure is subject to change in the future pending feedback from users. Please see the Contact section below to provide feedback.

Example Object Keys:

  • ed1.0.1/model_forecast_guidance/rtofs_east/2020/07/01/00/dcf2/tiles/111US00_RTOFS_EAST_20200701T00Z_US2GOMBC.h5
    • ed1.0.1 - Files compliant with edition 1.0.1 of the S-111 Standard
    • model_forecast_guidance - Files contain surface current forecast guidance derived from a numerical modeling system
    • rtofs_east - Files contain output from the eastern (Atlantic) domain of NCEP's Global Real-Time Ocean Forecast System (RTOFS)
    • 2020/07/01/00 - Files contain forecast guidance derived from a model run initialized on July 1, 2020 at 00 UTC
    • dcf2 - Files are encoded using S-111 Data Coding Format 2, which represents Regular Grids
    • tiles - The full dataset has been split into individual files each covering a distinct geographic area according to the predefined OCS Nautical Product Tiling Scheme
    • 111US00_RTOFS_EAST_20200701T00Z_US2GOMBC.h5
      • 111US00 - This is an S-111 file produced by the U.S. Government
      • RTOFS_EAST - The file contains output from RTOFS for its Eastern (Atlantic) domain
      • 20200701T00Z - The file contains forecast guidance derived from a model run initialized on July 1, 2020 at 00 UTC
      • US2GOMBC - The file contains output for a geographic area corresponding with the Office of Coast Survey's (OCS) Nautical Product Tiling Scheme for Cell Identifier US2GOMBC
      • .h5 - The file extension denotes that this is an HDF5 encoded file.
  • ed1.0.1/model_forecast_guidance/gomofs/2020/08/11/00/dcf3/regional/111US00_GOMOFS_20200811T12Z.h5
    • ed1.0.1 - Files compliant with edition 1.0.1 of the S-111 Standard
    • model_forecast_guidance - Files contain surface current forecast guidance derived from a numerical modeling system
    • gomofs - Files contain output from the NOS Gulf of Maine Operational Forecast System (GoMOFS)
    • 2020/08/11/12 - Files contain forecast guidance derived from a model run initialized on August 11, 2020 at 1200 UTC
    • dcf3 - Files are encoded using S-111 Data Coding Format 3, which represents Ungeorectified Gridded Arrays (i.e. irregular grids)
    • regional - Each file contains data for the entire coverage area of the underlying model domain
    • 111US00_GOMOFS_20200811T12Z.h5
      • 111US00 - This is an S-111 file produced by the U.S. Government
      • GOMOFS - The file contains output from the Gulf of Maine Operational Forecast System
      • 20200811T12Z - The file contains forecast guidance derived from a model run initialized on August 11, 2020 at 1200 UTC
      • .h5 - The file extension denotes that this is an HDF5 encoded file.

S-111 Exchange Set Catalogs (XML Metadata) 🔗︎

Example Exchange Set:

  • US_111_20220805150000_12_ExchangeSet.zip - S-111 Exchange Set filename
    • US - Producing Agency
    • 111 - Product Code
    • 20220805150000 - Date and time S-111 Exchange Set was produced
    • 12 - Region Identifier
    • Example Exchange Set contents:
    • 111US00_SFBOFS_20220805T15Z_US4CA2BI.h5 - S-111 HDF5 file
    • 111US00_SFBOFS_20220805T15Z_US4CA2CJ.h5 - S-111 HDF5 file
    • 111US00_SFBOFS_20220805T15Z_US4CA2CM.h5 - S-111 HDF5 file
    • 111US00_SFBOFS_20220805T15Z_US4CA1AL.h5 - S-111 HDF5 file
    • CATALOG.xml - S-111 Exchange Set Catalog file (XML Metadata)

Per the S-100 specification, S-111 files are packaged into "Exchange Sets," which are collections of individual S-111 datasets whose basic metadata are documented in an accompanying Catalog XML file. The Catalog XML file is provided in order to allow users to easily determine what S-111 files are available along with some basic information about each file (e.g. geographic coverage area) without requiring the user to first download these datasets.

Presently, exchange set catalog XML files are only being generated for the S-111 DCF-2 (regular grid) "tile" files, where this metadata will have the biggest benefit for users because of its ability to provide basic file metadata about each tile before a user system decides which file(s) to download. The DCF-2 and DCF-3 "regional" files are being provided to allow users to have additional S-111 files to evaluate, but are not yet being packaged into full Exchange Sets.

In this S3 bucket, an Exchange Set is comprised of all S-111 HDF5 DCF-2 (regular grid) tiles from a distinct model run cycle. The corresponding Exchange Set Catalog can be found in the same directory (S3 key prefix) as the S-111 files. For example, the S-111 Exchange Set for the 8/05/2022 15:00 UTC run cycle for SFBOFS (data coding format 2 [regular grid]) can be found at the path ed1.0.1/model_forecast_guidance/sfbofs/2022/08/05/15/dcf2/tiles/US_111_20220805150000_12_ExchangeSet.zip.

In addition to the dynamic path shown above, which will change based on date/time of the target OFS forecast cycle, a second Exchange Set Catalog is also provided for each OFS at the static path ed1.0.1/model_forecast_guidance/<ofs>/catalog.xml. This file is overwritten each time a new OFS forecast run becomes available, and thus can be downloaded at any time to determine which S-111 files are the latest available for a particular OFS.

Contact Information 🔗︎

If you have any questions or would like to provide feedback about these datasets, please submit your inquiry using the NOAA Office of Coast Survey (OCS) Assist Tool