The BioCatalogue REST API

The BioCatalogue provides a set of public RESTful ¹⁾ endpoints that allow you to query the registry programmatically and integrate the data and functionality into your own scripts, workflows, apps, tools and mashups. XML is the primary output format, with support for JSON ²⁾ as the secondary format. We are also looking into having RDF, FOAF, SKOS and others output formats in the future and where applicable.

The BioCatalogue API is read only, with write support for submission of REST and SOAP services, as well as annotations.

The API follows a consistent URI scheme and Linked Data principles. Content Negotiation ³⁾ is the recommended way to access the outputs (though for purposes of this documentation and testing in a browser you may see URIs with '.xml' in them). More detailed information is available on this.

One key notion in the data model presented by the REST API is that a Service in the BioCatalogue is really a container for one or more ServiceDeployments and one or more variants, where a variant is either a SoapService or RestService.

Jump to the Quick Examples

Jump to the Table of Endpoints

Go to the Endpoints Detailed Docs

The last change to the API (for the main BioCatalogue site) was:

v1.1.0b - 25th June 2010

• Added viewings and favourites counts to the /services/{id} endpoint (in the <summary> section).
• New endpoint - /soap_services/wsdl_locations. Allows you to get a list of all the WSDL locations of all the SOAP based services in the catalogue.

See the full changelog for more details

Search

• For things with the keyword “ebi”
  • http://www.biocatalogue.org/search.xml?q=ebi
  • http://www.biocatalogue.org/search.json?q=ebi
• For Services and SoapOperations with the keywords “blast sequence”
  • http://www.biocatalogue.org/search.xml?q=blast+sequence&scope=services,soap_operations
  • http://www.biocatalogue.org/search.json?q=blast+sequence&scope=services,soap_operations
• For Users with the keyword “franck”
  • http://www.biocatalogue.org/search.xml?q=franck&scope=users
  • http://www.biocatalogue.org/search.json?q=franck&scope=users

Lookup

For a particular WSDL http://ws.adaptivedisclosure.org/axis/services/IndexWS?wsdl…

• Get the relevant SoapService
  • http://www.biocatalogue.org/lookup.xml?wsdl_location=http%3A%2F%2Fws.adaptivedisclosure.org%2Faxis%2Fservices%2FIndexWS%3Fwsdl%0A
  • http://www.biocatalogue.org/lookup.json?wsdl_location=http%3A%2F%2Fws.adaptivedisclosure.org%2Faxis%2Fservices%2FIndexWS%3Fwsdl%0A
• Get the SoapOperation “streamContent”
  • http://www.biocatalogue.org/lookup.xml?wsdl_location=http%3A%2F%2Fws.adaptivedisclosure.org%2Faxis%2Fservices%2FIndexWS%3Fwsdl%0A&operation_name=streamContent
  • http://www.biocatalogue.org/lookup.json?wsdl_location=http%3A%2F%2Fws.adaptivedisclosure.org%2Faxis%2Fservices%2FIndexWS%3Fwsdl%0A&operation_name=streamContent
• Get the SoapInput “generateContent” for the SoapOperation “streamContent”
  • http://www.biocatalogue.org/lookup.xml?wsdl_location=http%3A%2F%2Fws.adaptivedisclosure.org%2Faxis%2Fservices%2FIndexWS%3Fwsdl%0A&operation_name=streamContent&input_name=generateContent
  • http://www.biocatalogue.org/lookup.json?wsdl_location=http%3A%2F%2Fws.adaptivedisclosure.org%2Faxis%2Fservices%2FIndexWS%3Fwsdl%0A&operation_name=streamContent&input_name=generateContent

Services

• Latest
  • http://www.biocatalogue.org/services.xml
  • http://www.biocatalogue.org/services.json
• Filtered by service type SOAP
  • http://www.biocatalogue.org/services.xml?t=%5bSOAP%5d
  • http://www.biocatalogue.org/services.json?t=%5bSOAP%5d
• Filtered by service type SOAP, and provided by the EBI or NCBI, and with the keyword “fetch” in them somewhere
  • http://www.biocatalogue.org/services.xml?t=%5bSOAP%5d&p=%5b1%5d%2c%5b6%5d&q=fetch
  • http://www.biocatalogue.org/services.json?t=%5bSOAP%5d&p=%5b1%5d%2c%5b6%5d&q=fetch
• All the possible filters that can be applied to the Services index
  • http://www.biocatalogue.org/services/filters.xml
  • http://www.biocatalogue.org/services/filters.json

SOAP operations

• Latest
  • http://www.biocatalogue.org/soap_operations.xml
  • http://www.biocatalogue.org/soap_operations.json
• Filtered by the tags “blast” or “sequence” anywhere within the operation and it's inputs and outputs
  • http://www.biocatalogue.org/soap_operations.xml?tag=%5bblast%5d%2c%5bsequence%5d
  • http://www.biocatalogue.org/soap_operations.json?tag=%5bblast%5d%2c%5bsequence%5d
• Filtered by the tag “protein_database” anywhere within the operation and it's inputs
  • http://www.biocatalogue.org/soap_operations.xml?tag_ins=%5bprotein_database%5d
  • http://www.biocatalogue.org/soap_operations.json?tag_ins=%5bprotein_database%5d
• All the possible filters that can be applied to the SoapOperations index
  • http://www.biocatalogue.org/soap_operations/filters.xml
  • http://www.biocatalogue.org/soap_operations/filters.json

REST methods

• Latest
  • http://www.biocatalogue.org/rest_methods.xml
  • http://www.biocatalogue.org/rest_methods.json
• Filtered by the tags “blast” or “sequence” anywhere within the REST method and it's REST input and output parameters and representations
  • http://www.biocatalogue.org/rest_methods.xml?tag=%5bblast%5d%2c%5bsequence%5d
  • http://www.biocatalogue.org/rest_methods.json?tag=%5bblast%5d%2c%5bsequence%5d
• Filtered by the tag “database” anywhere within the operation and it's REST input parameters and representations
  • http://www.biocatalogue.org/rest_methods.xml?tag_ins=%5bdatabase%5d
  • http://www.biocatalogue.org/rest_methods.json?tag_ins=%5bdatabase%5d
• All the possible filters that can be applied to the RestMethods index
  • http://www.biocatalogue.org/rest_methods/filters.xml
  • http://www.biocatalogue.org/rest_methods/filters.json

Details of a Service

• Default view
  • http://www.biocatalogue.org/services/4.xml
  • http://www.biocatalogue.org/services/4.json
• Summary only
  • http://www.biocatalogue.org/services/4/summary.xml
  • http://www.biocatalogue.org/services/4/summary.json
• Monitoring info only
  • http://www.biocatalogue.org/services/4/monitoring.xml
  • http://www.biocatalogue.org/services/4/monitoring.json
• Annotations on the Service (not including the annotations on the other parts of the service, like the ServiceDeployments, SoapServices, etc)
  • http://www.biocatalogue.org/services/4/annotations.xml
  • http://www.biocatalogue.org/services/4/annotations.json

Categories and Tags

• Top level service categories
  • http://www.biocatalogue.org/categories.xml
  • http://www.biocatalogue.org/categories.json
• The Services that have a particular category (and all other categories narrower/below that category)
  • http://www.biocatalogue.org/categories/5/services.xml
  • http://www.biocatalogue.org/categories/5/services.json
• Get the top 10 tags
  • http://www.biocatalogue.org/tags.xml?limit=10
  • http://www.biocatalogue.org/tags.json?limit=10
• Get more information about a regular keyword tag
  • http://www.biocatalogue.org/tags/soaplab.xml
  • http://www.biocatalogue.org/tags/soaplab.json
• Get more information about a controlled vocabulary term tag (currently only supported for the myGrid ontology)
  • http://www.biocatalogue.org/tags/retrieving.xml?namespace=mygrid-domain-ontology
  • http://www.biocatalogue.org/tags/retrieving.json?namespace=mygrid-domain-ontology

• Balance - between simplicity and utility. A number of different use cases for the API have been considered and an attempt has been made to balance simplicity and utility of the API.

• Consistency - in the URIs and outputs of the API. Consistent URIs and patterns are used throughout.

• Discoverability - of related resources, next pages of results, and so on. Additional links to resources are added to outputs wherever possible and appropriate.

• Manageability - of URIs, outputs and data schemas. We aim to provide a stable and manageable experience for API consumers. Breaking changes will be avoided as much as possible, but in the event where they are absolutely necessary they will be trialled and announced in the appropriate places. The XML outputs are fully schema driven so that changes to client code can be done in a manageable way.

The data model exposed by the API is shown below.

Key:

• The orange boxes represent logical groups.
• The blue boxes represent actual entities exposed in the API.
• The purple boxes represent polymorphic references to entities and not actual entities.
  • For example, an Annotation has a Source, but no actual Source entity exists. This is just a reference to another entity type that can be the source of the annotation (see more information about Annotations, below).

The table below details the various entities that make up this data model. It also specifies whether the API exposes them as resources or not, and whether a corresponding index/collection resource is available.

A resource is defined as: something that has a unique dereferenceable URI that can be accessed to retrieve a representation of that thing, via the API.

Annotations are the extensible mechanism by which additional metadata is added about services (or their constituents) in the BioCatalogue, to incrementally document services at various levels of granularities.

Things like…

• Tags
• Categories
• Additional descriptions
• Documentation URLs
• Example data

… and so on, are stored and exposed using the generic Annotations mechanism.

The common principle we attempt to follow: metadata from service description documents and from service monitoring are part of the core model, but all other metadata is stored and exposed using Annotations.

Each resource has it's own set of annotations. This allows different levels granularity. E.g.: 'tag' annotations on a Service apply to the whole service whereas 'tag' annotations on a SoapOperation are specific to that SOAP operation. See the breakdown of what main annotation fields are on what resource types.

At a conceptual level, an Annotation consists of:

• a link to the thing that the Annotation is about - the annotatable.
• a link to the source of the Annotation - the source.
• a link to the annotationAttribute which specifies what kind of Annotation this is (i.e.: what field).
• the value of the annotation (some textual content with an optional reference to an entity that is the actual value).
• the latest version number of the Annotation.
• timestamps for when the Annotation was created and last modified.

The following resource types can be the source of Annotations:

• User
• ServiceProvider
• Registry
• Agent

Below are the main annotation fields (i.e.: annotation attributes) for all the possible annotatable resource types within the BioCatalogue:

• Service:
  • tag
  • category
  • display_name
  • alternative_name
  • rating.xx

• ServiceDeployment:
  • cost
  • usage_condition
  • contact

• SoapService, RestService:
  • documentation_url
  • description
  • license
  • publication
  • citation

• SoapOperation:
  • alternative_name
  • tag
  • description

• SoapInput:
  • alternative_name
  • tag
  • description
  • format
  • example_data

• SoapOutput:
  • alternative_name
  • tag
  • description
  • format
  • example_data

• ServiceProvider:
  • display_name
  • alternative_name
  • description
  • website
  • contact

The BioCatalogue maintains a consistent URI scheme and currently the API (XML, JSON, ATOM outputs) follows Linked Data ⁶⁾ principles ⁷⁾. We use the extensions way of defining the URIs for the different representations (see below for more info).

Content Negotiation ⁸⁾ is the recommended way to access the outputs. I.e.: by using the HTTP Accept header and not calling the ”.xml” etc URLs directly.

This means that, for example:

The following is a non-information resource URI for the services index:

• http://www.biocatalogue.org/services

This URI is considered to be the unique dereferenceable URI (aka identifier) for that particular resource.

Which then has the following representations:

• http://www.biocatalogue.org/services.html
• http://www.biocatalogue.org/services.xml
• http://www.biocatalogue.org/services.atom
• http://www.biocatalogue.org/services.json

All data outputs from the API only ever contain identifiers (that are the non-information resource URIs). Content negotiation can then be used to access specific representations of these identifiers (via an HTTP 303 redirect).

The figure below illustrates another example of how this works, showing the relevant 303 redirect:

The table below enumerates all the main endpoints that can be used to access resources via the API.

Note:

• Template parameters are shown in curly brackets, e.g.: {id} needs to be replaced with a value, in order to specify a specific resource.
• Not all endpoints have an HTML output representation, but all will have at least XML.
• Detailed docs for each endpoint are available and each entry in the table has a corresponding link to these docs.

The XML output for all index/collection resources have the following structure:

<resource>
  <parameters>...</parameters>
  <statistics>...</statistics>
  <results>...</results>
  <related>...</related>
</resource>

This allows for additional information to be present, to aid consistency and manageability of the outputs, as well discoverability of further associated resources.

The JSON output for all index/collection resources have the following structure:

{
  'resource' : {
    'per_page' : ... ,
    'current_page' : ... ,
    'pages' : ... ,
    'total' : ... ,
    'search_query' : ... ,
    'results' : [ 
      {
        'resource_property_x' : ...,
        'resource_property_y' : ...,
        'resource_property_z' : ...,
        ...
      },
      ...
    ]
  }
}

This allows for key information to be present, to aid consistency and manageability of the outputs.

These index/collection resources will contain the following properties:

• per_page - the number of result items to show per page..
• current_page - the current page being viewed.
• pages - the total number of pages of results.
• total - the total number of items of that resource type found for that particular resource call.
• search_query - the search query applied where applicable.
• results - the list of result items found for that particular resource call.

The majority of the XML outputs have a <related> section with links to other associated resources.

In the case of index/collection resources (like /services) there will be a <related> section for associated links/resources to the index/collection, and usually there will also be individual <related> sections for the resources mentioned in the <results>.

The <statistics> section within index/collection resources will usually contain at least the following:

• <pages> - the total number of pages of results.
• <results> - the total number of result items found for that particular resource call.
• <total> - the total number of items of that resource type available in the system. Note that this is only shown for index/collection resources that are about a single resource type (e.g.: /services but NOT /search).

This allows you to get information like “1585 out of 1628 services are SOAP based services” from a filtered /services index.

Whenever possible, sub-collections on a resource take you to a filtered index/collection resource.

For example:

http://www.biocatalogue.org/services/1928/annotations.xml (the Annotations on the Service with ID 1928)

… will redirect you to http://www.biocatalogue.org/annotations.xml?as=%5b1928%5d

This means that client code only needs to be able to parse the index resource and be aware of filtering, in order to handle the majority of sub-collections.

In places, you will see the presence of urlKey and urlValue attributes. These specify what URL query parameters can be used (or have been used, in the case of <parameters>) on the resource that the output is about.

Note that for filtering, urlValue should be used in a different way (more information about this below).

All index/collection resources are paged by default, meaning that the whole result set is split into pages of a certain size and only one page is returned at a time. Subsequent pages can then be accessed by new requests.

In order to assist with pagination, the XML outputs provide:

• Current <page> and <pageSize> values, within the <parameters> section (together with what urlKey to use for the query parameters).
• <pages> value, within the <statistics> section.
• <previous> and <next> links (where applicable), within the <related> section.

In JSON, the following elements are available:

• per_page
• current_page
• pages

The relevant URL query parameters are:

So for example, the following URL gets the 5th page of the Services index and asks for 20 items per page:

http://www.biocatalogue.org/services.xml?page=5&per_page=20 http://www.biocatalogue.org/services.json?page=5&per_page=20

Most of the index/collection resources allow for sorting of the results.

The include URL query parameter can be used to include certain sub-sections of data, that are not normally shown, within the outputs.

This works in two different ways depending on the type of resource:

1. For index/collection resources, include acts on the resources returned within the <results> section.
   • For example, http://www.biocatalogue.org/services.xml?include=summary would give you back the first page of the Services index with the <summary> sections included for each <service> within <results>.
2. For single resources, include just includes the specified sub-section(s).
   • For example, http://www.biocatalogue.org/services/4.xml?include=summary would give the Service resource with it's <summary> sub-section also included.

To include multiple sub-sections, separate out the names with commas. E.g.:

http://www.biocatalogue.org/services/4.xml?include=summary,monitoring

To include all possible sub-sections, use include=all. E.g.:

http://www.biocatalogue.org/services/4.xml?include=all

The structure of Filters (i.e.: the <filters> top level resource and <filters> in <parameters> have the following structure:

<filters>
  <group>
    <type>
      <filter>...
      <filter>...
      ...
    </type>
    ...
  </group>
  ...
</filters>

When combining filters, the following logic applies:

• all <filter> within a <type> are OR'ed
• all <type> within a <group> are OR'ed
• all <group> are AND'ed

NB: All filtered indexes allow for allow for query based filtering via the q query parameter:

• GET /services?q={query}
• GET /soap_operations?q={query}
• GET /rest_methods?q={query}
• GET /users?q={query}
• GET /service_providers?q={query}

Errors in API calls will result in a response with an appropriate HTTP/1.1 Status Code.

The following table shows the relevant status codes:

Wherever possible, the body of an error response will contain an Errors structure like below, to provide more information on the error(s) raised:

<errors>
<error>Error message</error>
<error>...</error>
</errors>

You can use a command line tool like curl to quickly test out API endpoints:

curl -i -H "Accept: application/xml" http://www.biocatalogue.org/services.xml

Below is an example of how to access JSON endpoints in a Ruby script/application using OAuth…

Firstly, register your application on the BioCatalogue website. Please note your consumer key and consumer secret as these will be required by your application/script.

Then, install the JSON and OAuth gems:

gem install json oauth

After this, you can then use code like:

require 'rubygems'
require 'oauth'
require "json/ext"
 
key = "..." # get your consumer key from the BioCatalogue
secret = "..." # get your consumer key from the BioCatalogue
 
site = "http://www.biocatalogue.org"
 
# ===== the tedious one time setup ===== #
 
# the first time you use OAuth, you will need to get an access token; you only need to do this once
consumer = OAuth::Consumer.new(key, secret, { :site => site })
request_token = consumer.get_request_token
 
# the user needs to ask users to allow you application/script to access their BioCatalogue content
# after authorisation the user will receive a token which your application/script needs for the next step! make sure you have a way to capture this token :-)
request_token.authorize_url # point your browser to this URL
...
 
# :oauth_verifier is the key that was provided to the user in the last step
access_token = request_token.get_access_token(:oauth_verifier => "...")
 
# store your access token (access_token)
...
 
# in future, all you need to do is recall your access token and you are ready to go :-)
 
# =======================================#
 
response = access_token.get(site + '/users/738.json')
json_doc = JSON.parse(response.body)
 
# Do some stuff with the JSON document (json_doc)
name = json_doc["user"]["name"]
 
# Do some other stuff with the JSON document (json_doc)
...

Below is an example of how to access JSON endpoints in a Ruby script/application…

First, install the JSON gem:

gem install json

Then you can use code like:

# Require the necessary components
require 'rubygems'
require 'open-uri'
require 'json/ext'
 
# Set up the endpoint URL you want to access
endpoint_url = "http://www.biocatalogue.org/services"
 
# Set up an informative HTTP USER AGENT
http_user_agent = "My test script; Ruby/#{RUBY_VERSION}" "MyTestScript/1.0 (X11; U; Linux x86_64; en-US) Ruby/#{RUBY_VERSION}"
 
# Fetch and parse the JSON document
json = open(endpoint_url, "Accept" => "application/json", "User-Agent" => http_user_agent).read
document = JSON(json)
 
puts "The #{document['services']['per_page']} latest results provided by '#{endpoint_url}' are:"
 
document['services']['results'].each { |service|
  # Do some stuff with the object
  ...
}
 
# Do some other stuff with the JSON document
...

Below is an example of how to access XML endpoints in a Ruby script/app…

First, install the libxml gem:

gem install libxml-ruby

Then you can use code like:

# Require the necessary components
require 'rubygems'
require 'open-uri'
require 'libxml'
 
include LibXML
 
# Set up the endpoint URL you want to access
endpoint_url = "http://www.biocatalogue.org/services"
 
# Set up an informative HTTP USER AGENT
http_user_agent = "My test script; Ruby/#{RUBY_VERSION}" "MyTestScript/1.0 (X11; U; Linux x86_64; en-US) Ruby/#{RUBY_VERSION}"
 
# Fetch and parse the XML document
xml = open(endpoint_url, "Accept" => "application/xml", "User-Agent" => http_user_agent).read
document = XML::Document.string(xml)
 
# Validate the output against the schema
schema = XML::Schema.new(SCHEMA_FILE_PATH)
is_valid = document.validate_schema(schema) do |message,flag|
  puts "\n#{(flag ? 'ERROR' : 'WARNING')}: #{message}"
end
puts "\nValid output? #{is_valid}\n"
 
# Do some other stuff with the XML document
...

Download this document to get detailed usage information on calling and using the BioCatalogue API in Java. This includes instructions on using XmlBeans to automatically generate the API binding classes from the XSD, which removes the need to manually parse the XML files yourself.

This document is authored by Sergejs Aleksejevs (who is building the BioCatalogue plugin for Taverna).

The following projects currently make use of the BioCatalogue API:

• An alpha version BioCatalogue plugin for Taverna has been released. See the Taverna plugins page for more info.
• The BioCatalogue official iOS application. See the github page for more information.
• The ONIX Portal by the NCRI harvests all the information about web services from the BioCatalogue and makes it searchable and browsable.
• We are currently integrating BioCatalogue into myExperiment by harvesting the service data and making them searchable and viewable in myExperiment. Also, linking workflows with services.
• An alpha version application for mass curation. See the Mass Curator github page for more information.
• The BioCatalogue iPhone app developed by Emory University.

The API coverage page details current and planned API coverage.

The BioCatalogue API is being actively development and maintained by:

• Jiten Bhagat (Jits)
• Mannie Tagarira

A number of people have helped drive the requirements and design of the BioCatalogue API. In particular:

• Sergejs Aleksejevs
• Stian Soiland-Reyes
• The NCRI ONIX team
• The myExperiment Team