Using APIs

Some of the data presented on the JSNA web site is accessed in real-time from government sources using Application Programming Interfaces, or APIs for short. The benefits to us in working this way include:

  • Not having to download, prepare and store the data ourselves, locally. Saving time and money.
  • The assurance that the our web pages will always be displaying the very latest published data.
  • The ability to present data from a disparate set of sources in a consistent style.

We are not expert programmers, but we’re happy to share what we’ve learnt so far.

ONS nomis

The API for the Office for National Statistics nomis web site is by far the most accessible and well supported API we’ve used so far. Nomis is a service provided by the ONS to give free access to the most detailed and up-to-date UK labour market statistics from official sources. It also hosts the data tables from the 2011 Census, and large amounts of additional census material from 2001, 1991, and even some from 1981. There is also an increasing amount of demographic data on nomis, including population estimates and projections, mortality and marriage data.

The nomis API can handle discovery and data download requests, and supports a variety of data formats – SDMX, RSS feeds, CSV, JSON and Excel. In our experience, pretty much all the data on nomis can be accessed through the API, and the support team are good at answering questions.

The easiest way to get started is to use the nomis site in the usual way to find some data, and then before downloading, go to the Format/Layout options and select Nomis API from the formats list. When you then click on Download Data you’ll be presented with details of the data that the API request will return, and you have the opportunity to refine the amount of data you’ll be served using the Column Selection and Tabulation tool. (This can take a bit of getting used to, but it’s very useful). For those people wanting to extract nomis data and see it visualized in a web page there is a massive helping hand waiting on the API Links tab, by clicking on the <EMBED /> on my site link. This produces the html and JavaScript code necessary to display the data using the Google Visualization tools. Just copy/paste this into a file and load into your web browser, et voila, most of the time anyway.

Here are some examples we’ve worked up so far.

LG Inform Plus

If you’re a subscriber to LG Inform Plus (formerly known as esd-toolkit) then you’ll have access to an API that can serve discovery information and data from the LG Inform database. We had a few problems getting started, but the support and development team were great and helped us get the proxy scripting working and provided us with worked examples to get us started. API requests have to be passed through a proxy script written in PHP. This handles the secure keys necessary to protect what is a subscription service. The LG Inform Plus web site has API documentation and tools to help build the query strings.

Here are some examples we’ve worked up so far.

DWP Stat-Xplore

The Stat-Xplore API provides access to a wide range of benefits and other DWP data. You’ll need to register for an API key, but that’s straightforward and free. The Stat-Xplore user guide provides some information about the API end points and there are a couple of examples of the API in use which you can unpick to see how the AJAX queries are put together. To make sense of things it is probably best to spend a bit of time querying the /schema endpoint to see how things are named.

Stat-Xplore uses the SuperSTAR software from Space-Time Research, an Australian company. The Space-Time Research web site (Support … Documentation) also has documentation about the API.

The most useful support is built into Stat-Xplore itself. If you create a table of data using the standard selection tools, you have the option to Download Table (top-right of screen) in .json format. If you proceed in this manner then a file will be generated showing you how the call to the /table endpoint should be made. At the moment, the API doesn’t have the ability to work with relative dates, e.g. latest, or latest-1. Not specifying a date period is likely to return data for all periods held in the database for that table, which can be quite a lot of data to then filter through to get the date(s) you want. The data scientists at DWP intend to implement relative dates in a future release of the API (personal communication November 2017).

Here are some examples we’ve worked up so far.

fingertips

From the fingertips User Guide

Fingertips is a web platform that provides easy access to in-depth analysis of a wide range of health and health related data in thematic Profiles. All profiles can be accessed via:
http://fingertips.phe.org.uk/
It was initially designed and built by the Eastern Region Public Health Observatory, and is now owned by Public Health England. The platform grows quickly both in functionality and content and is served by one common database called PHOLIO.

An Application Programming Interface (API) is available that allows developers to obtain Fingertips data directly in order to use it in other web applications. A library for R has been developed for people wanting to pull public health data into R for analysis. More details of this work can be found on GitHub.

For our work though, we’ve been interested in how we can write client-side JavaScript to pull the fingertips data into our web pages. There is a useful API discovery tool, but other documentation is rather thin on the ground. However, by opening up a copy of a fingertips profile (e.g. the Public Health Outcomes Framework) and examining the address URL as you navigate around, it is possible to work out most of the parameters that the discovery tool requires, and so start to built up API URLs and see the JSON they produce.

We’d usually look to use JSONP to help us around the problem of cross-domain policies in web browsers. (You are not allowed to make AJAX requests to a web page perceived to be on a different server by the browser). Unfortunately, the fingertips API does not support JSONP so we’ve used Ben Alman’s Simple PHP Proxy to work around the cross-domain problem.

Here are the examples we’ve worked up so far.

Google Maps JavaScript API

We’ve decided to explore this API with a view to using it to add further choropleth mapping capability to the site. The data to be mapped might be stored in the web page as a simple array, or the data may itself be accessed using an API, from nomis, for example. In both cases, the data values are merged with the GeoJSON file containing the spatial boundary data and then the boundary polygons are shaded using various algorithms for defining data intervals or algorithms for continuous colour ramps.

GeoJSON data for the standard administrative and statistical boundaries can be found on the ONS Open Geography Portal. This resource includes generalised and super-generalised versions of the popular boundary files.

Google’s documentation for the API contains many worked examples in additional to the usual reference material.

Visit Tom MacWright’s web site for discussion and the JavaScript to classify data using the Jenks natural breaks algorithm.

We’re still at an early stage with this work, but here is an example we are using for testing

Other APIs we’ve yet to explore

  • OpenDataCommunities – Department for Communities and Local Government. Housing, deprivation, planning etc.
  • HM Land Registry – House sales, etc.
  • maps.police.uk – crime, ASB and outcomes data
Print Friendly, PDF & Email