ERDDAP Easier access to scientific data

Brought to you by NOAA NMFS SWFSC ERD

ERDDAP > Computer Programs

Accessing ERDDAP as a Web Service

• A web application – a web site that humans with browsers can use
  (in this case, to get data, graphs, and information about datasets).
   
• A web service – a web site that computer programs can use
  (in this case, to get data, graphs, and information about datasets).

Build Things on Top of ERDDAP
There are many features in ERDDAP that can be used by computer programs or scripts
that you write. You can use them to build other web applications or web services on
top of ERDDAP, making ERDDAP do most of the work!
So if you have an idea for a better interface to the data that ERDDAP serves or a web
page that needs an easy way to access data, we encourage you to build your own web
web application, web service, or web page and use ERDDAP as the foundation.
Your system can get data, graphs, and other information from ERD's ERDDAP or from
other ERDDAP installations, or you can set up your own ERDDAP server, which can be
publicly accessible or just privately accessible.

URL Requests
Requests for user-interface information from ERDDAP (for example, search results)
use the web's universal standard for requests: URLs sent via HTTP GET.
This is the same mechanism that your browser uses when you fill out a form
on a web page and click on Submit.
To use HTTP GET, you generate a specially formed URL (which may include a query)
and send it with HTTP GET. You can form these URLs by hand and enter them in
the address textfield of your browser (for example,
http://edac-dap2.northerngulfinstitute.org/erddap/search/index.json?searchFor=temperature)
Or, you can write a computer program or web page script to create a URL, send it,
and get the response. URLs via HTTP GET were chosen because

• They are simple to use.
• They work.
• They are universally supported (in browsers, computer languages, operating system
  tools, etc).
• They are a foundation of Representational State Transfer (REST) and
  Resource Oriented Architecture (ROA).
• They facilitate using the World Wide Web as a big distributed application,
  for example via mashups and AJAX applications.
• They are stateless, as is ERDDAP, which makes the system simpler.
• A URL completely define a given request, so you can bookmark it in your browser,
  write it in your notes, email it to a friend, etc.

Percent Encoding
In URLs, some characters are not allowed (for example, spaces) and other characters
have special meanings (for example, '&' separates key=value pairs in a query).
When you fill out a form on a web page and click on Submit, your browser automatically
percent encodes the special characters in the URL (for example, by replacing ' ' in a query
value with "%20", for example,
http://edac-dap2.northerngulfinstitute.org/erddap/search/index.html?searchFor=temperature%20wind%20speed
But if your computer program or script generates the URLs, it may need to do the percent
encoding itself. Programming languages have tools to do this (for example, see Java's
java.net.URLEncoder).

Requesting Compressed Files
ERDDAP doesn't offer results stored in compressed (e.g., .zip or .gzip) files.
Instead, ERDDAP looks for accept-encoding in the HTTP GET request header sent
by the client. If a supported compression type ("gzip", "x-gzip", or "deflate") is found
in the accept-encoding list, ERDDAP includes "content-encoding" in the HTTP response
header and compresses the data as it transmits it.
It is up to the client program to look for "content-encoding" and decompress the data.
Browsers and OPeNDAP clients do this by default. They request compressed data and
decompress the returned data automatically.
Other clients (e.g., Java programs) have to do this explicitly.

Response File Types
Although humans using browsers want to receive user-interface results (for example,
search results) as HTML documents, computer programs often prefer to get results in
simple, easily parsed, less verbose documents. ERDDAP can return user-interface
results as a table of data in these common, computer-program friendly, file types:

• .csv - Download a comma-separated ASCII text table (line 1: names; line 2: units; ISO 8601 times). (more info)
• .htmlTable - View a .html web page with the data in a table. Times are ISO 8601 strings. (more info)
• .json - Download a JSON table (missing value = 'null'; times are ISO 8601 strings). (more info)
• .mat - Download a MATLAB binary file. (more info)
• .tsv - Download a tab-separated ASCII text table (line 1: names; line 2: units; ISO 8601 times). (more info)
• .xhtml - View an XHTML (XML) file with the data in a table. Times are ISO 8601 strings. (more info)

• Each column has a column name and one type of information.
• The first row of the table has the column names.
• Subsequent rows have the information you requested.

The content in these plain file types is also slightly different from the .html
response -- it is intentionally bare-boned, so that it is easier for a computer
program to work with.

A Consistent Data Structure for the Responses
All of the user-interface services described on this page can return a table of
data in any of the common file formats listed above. Hopefully, you can write
just one procedure to parse a table of data in one of the formats. Then you can
re-use that procedure to parse the response from any of these services. This
should make it easier to deal with ERDDAP.

.csv and .tsv Details

• If a datum in a .csv file has internal double quotes or commas, ERDDAP follows the
  .csv specification strictly: it puts double quotes around the datum and doubles
  the internal double quotes.
• If a datum in a .csv or .tsv file has internal newline characters, ERDDAP converts
  the newline characters to character #166 (¦). This is non-standard.

jsonp
Requests for .json files may now include an optional jsonp request by
adding "&.jsonp=functionName" to the end of the query. Basically, this tells
ERDDAP to add "functionName(" to the beginning of the response and ")" to the
end of the response. If originally there was no query, leave off the "&" in your query.

griddap and tabledap Offer Different File Types
The file types listed above are file types ERDDAP can use to respond to
user-interface types of requests (for example, search requests). ERDDAP supports
a different set of file types for scientific data (for example, satellite and buoy
data) requests (see the griddap and tabledap documentation.

Access URLs for ERDDAP's Services
ERDDAP has these URL access points for computer programs:

• To get the list of the main resource access URLs,
  use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
   
• To get the current list of all datasets,
  use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
   
• To get metadata for a specific data set
  (the list of variables and their attributes),
  use http://edac-dap2.northerngulfinstitute.org/erddap/info/datasetID/index.fileType ,
  for example, .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
   
• To get the results of full text searches for datasets
  (using "searchFor=temperature%20wind%20speed" as the example),
  use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
  (Your program or script may need to percent-encode the value in the query.)
   
• To get the results of advanced searches for datasets
  (using "searchFor=temperature%20wind%20speed" as the example),
  use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
  But experiment with Advanced Search in a browser to figure out all of the
  optional parameters.
  (Your program or script may need to percent-encode the value in the query.)
   
• To get the list of categoryAttributes
  (e.g., institution, long_name, standard_name),
  use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
   
• To get the list of categories for a specific categoryAttribute
  (using "standard_name" as the example),
  use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
   
• To get the list of datasets in a specific category
  (using "standard_name=time" as the example),
  use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
   
• To get the current list of datasets available via a specific protocol,
  • For griddap: use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
  • For tabledap: use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
  • For WMS: use .csv, .htmlTable, .json, .mat, .tsv, or .xhtml.
     
• Griddap and tabledap have many web services that you can use.
  • The Data Access Forms are just simple web pages to generate URLs which
    request data (e.g., satellite and buoy data). The data can be in any of
    several common file formats. Your program can generate these URLs directly.
    For more information, see the griddap documentation and
    tabledap documentation.
     
  • The Make A Graph pages are just simple web pages to generate URLs which
    request graphs of a subset of the data. The graphs can be in any of several
    common file formats. Your program can generate these URLs directly. For
    more information, see the griddap documentation and tabledap documentation.
     
  • To get a dataset's structure, including variable names and data types,
    use a standard OPeNDAP .dds resquest. For example,
    http://edac-dap2.northerngulfinstitute.org/erddap/griddap/erdBAssta5day.dds or
    http://edac-dap2.northerngulfinstitute.org/erddap/tabledap/pmelTao.dds .
     
  • To get a dataset's metadata, use a standard OPeNDAP .das resquest.
    For example,
    http://edac-dap2.northerngulfinstitute.org/erddap/griddap/erdBAssta5day.das or
    http://edac-dap2.northerngulfinstitute.org/erddap/tabledap/pmelTao.das .
     
• ERDDAP's other protocols also have web services that you can use.
  See ERDDAP's WMS documentation.
   
• ERDDAP offers RSS subscriptions, so that your computer program find out if a
  dataset has changed.
   
• ERDDAP offers email/URL subscriptions, which notify your computer program
  whenever a dataset changes.
   

Using ERDDAP as a Data Source within Your Java Program

Or, since ERDDAP is an all-open source program, you can also set up your own copy of
ERDDAP on your own server (publicly accessible or not) to serve your own data. Your Java
programs can get data from that copy of ERDDAP. See Set Up Your Own ERDDAP.

ERDDAP Version