Democratising data: how improving Big Data accessibility is helping to protect crucial ecosystems

Collating Big Data. Shubham Dhage | Unsplash

Following the boom of new innovative monitoring technologies, a lack of environmental data is often no longer a limiting factor. However, the scientific community’s capacity to handle and process the reams of data being generated has been identified as a bottleneck. A key challenge has been to facilitate an accurate and reliable stream of verified data to a variety of end users and this has been playing out in the satellite Earth Observation (EO) sector in recent years.

Whether by design or default, satellite EO technology advanced rapidly, allowing the collection of unparalleled levels of data, with far great volume and storage space required, and this was thankfully met by advances in information technology and artificial intelligence. From this, new data technologies, infrastructures and architectures emerged that help streamline data distribution, management and quality control as well as lowering technical barriers for access by potential data end users.

One such example of this is the Open Data Cube (ODC), which seeks to provide an open and freely available data architecture solution that has value to global users and overall, increases the beneficial impact of EO satellite data.

ODCs are gridded and stacked arrays of different data sets, that can be interrogated easily and efficiently by scientists and other users. They often include  a set of libraries to build and manage the data cube, allowing multiple users to easily interact with large archives of data. Fundamentally, the ODC acts like a universal translator that helps different data ‘languages’ work together.

The Environmental Intelligence Group at Plymouth Marine Laboratory (PML) have been collaborating with other scientists around the globe to help optimise this potential. In a recent study, the ODC was used to build analytical EO processing capacity that enables Papua New Guinea (PNG) to routinely map ‘blue carbon’ habitats across the country and monitor the status of associated ecosystems and carbon stocks.

This study focused on mapping the extent of blue carbon ecosystems (BCE) in PNG using EO data pulled from a range of sources for the year 2020, reporting on biomass and carbon storage services. Funnelling the assorted data through the ODC allowed the team to estimate that the total BCE area in PNG (14,353km²) comprised 30% mangrove, 65% lowland peat swamp forest, 3% saltmarsh, and 2% seagrass.

The data also showed that lowland peat swamp forests contribute the greatest biomass at around 70.87 – 205.04 teragrams (Tg = 1 million tonnes; 137.94 ± 67.10 Tg) followed by mangroves at 44.63 – 98.95 Tg (71.79 ± 27.16 Tg), with a total biomass of around 117.10 – 308.88 Tg (212.99 ± 95.89 Tg).

Across PNG, a total of approximately 347.71 – 1073.21 Tg of carbon (710.46 ± 362.75 Tg C) were estimated for belowground carbon of BCEs (reporting to 1m depth), almost seven times more than that of aboveground carbon at around 56.17 – 148.11 Tg C (102.14 ± 45.97 Tg C).

The study highlights the need for a consistent and standardised framework for mapping BCEs, and the application of the ODC in this way can be used as a demonstration for other nations where similar opportunities and challenges may exist for mapping BCE using EO.

Dr Dan Clewley, Lead Research Software Engineer at Plymouth Marine Laboratory (PML) and co-author on the study, commented:

“Being able to bring together multiple satellite data sources into a single application that users can then use to map ecosystems is extremely valuable and really shows the usefulness and versatility of Earth observation data. With this new Open Data Cube capability, maps of blue carbon habitats in Papua New Guinea can now be routinely generated and with a few tweaks, this new digital infrastructure technique could be applied anywhere in the world.”

Expanding this capability will be a game changer for scientists, environmental managers and policy makers, globally, and the PNG study has reinforced the ODC’s adaptability. However, this capability did not appear overnight and its roots can be found in an Australian initiative from over a decade ago.

Early days of the Open Data Cube

In response to the United States Geological Survey (USGS) Landsat archive being made freely available to all users in 2008, an Australian collaboration developed the Australian Geoscience Open Data Cube (AG-ODC). The combination of high-performance computing and data infrastructure unlocked the value in the Landsat data, which won it the Content Platform of the Year at the Geospatial World Leadership Awards 2016. The AG-ODC showed much promise and it was not long before a community formed around it to help maximise its potential for other uses around the world. Now simply called the Open Data Cube (ODC), it was used in a collaborative project with the Digital Earth Australia team to produce land cover maps of the country for the last 30 years, as well as exploring the implementation of a global framework for standardising and classifying land cover in support of sustainable development goals.

Dr Clewley said:

“As part of this Digital Earth Australia land-cover project, we developed core software to allow the ODC to  produce multi-year maps and monitoring going forward”.

Running in parallel, and again with expertise provided by PML scientists, a dedicated ODC of Landsat, Sentinel-1 radar, and Sentinel-2 optical data was also established for Wales. This provides the Welsh Government with a fully operational tool to map and monitor national land cover for Wales, in support of protecting livelihoods, communities, biodiversity and the overall environment.

Monitoring water quality

Beyond mapping land cover in Australia, the ODC has proven valuable for environmental monitoring closer to home. One of the PML projects that is benefiting from the ODC approach is investigating the use of remote monitoring devices to observe changes in water quality.

The ‘VISualisation and Assessment of water quality using an Open Data Cube FOR the weStern English channel (Vis4Sea)’ project was funded to measure the impacts of flooding on the water quality of Plymouth Sound and the wider Western English Channel. The data available for this area includes samples collected by boats and data from marine buoys, satellites and drones.

To make the processing of all of this data more efficient, the team developed a Vis4Sea ODC instance that not only integrates data from the variety of sources but also applied advanced machine- and deep-learning algorithms to satellite data. This enables the team to generate actionable water quality information, such as maps of total suspended matter and Escherichia coli (E. coli). It also helps track flooding events in the local area as well as assess their impacts on water quality, mud flats and sea grass beds.

The project found that both heavy rainfall and south-west surface currents resulted in high cell counts of E. coli on local tourist beaches, Plymouth Sound and shellfish beds in Lyme Bay. Heavy rainfall also significantly increased the total and inorganic suspended matter, impacting light penetration in certain zones that may reduce the growth of submerged vegetation.

Dr Gavin Tilstone, Bio-optical Oceanographer and PML and lead of the Vis4Sea project, said:

“Using the ODC for observing the rivers, estuaries and coastal waters around Plymouth and the south-west UK, has transformed the way in which we can monitor water quality in the region by combining multi-source, multi-sensor data with deep learning and particle tracking models.”

Next generation tools

Seeing the potential of the ODC and taking the approach a step further, the PML and PML Applications teams have re-imagined the ODC offering into a fully comprehensive suite of EO, modelling and in situ data management tools.

The Central Open Repository of Data, Information, Analysis & Learning (CORDIAL) is a collaborative space that aims to transform the way EO data is analysed and interpreted. By using an ODC at its core, CORDIAL operates in a scalable cloud environment, eliminating the need for massive downloads or costly hardware investments, whilst also transforming petabytes of data into clear, actionable insights.

The CORDIAL service has already been used to: assess marine biodiversity; support environmental compliance and security; track environmental changes; detect oil spills; monitor habitat extent and condition; analyse water quality; explore climate change indicators; support natural disasters mapping; quantify land use change; predict crop yield; track air quality; monitor forest cover, and generally facilitate oceanographic and environmental research.

As the team spends more time exploring what the ODC approach has to offer, the more they are finding potential application opportunities. Across the world, successful ODC implementations currently exist in Australia, Colombia, Switzerland and Taiwan and are under development or consideration in another 39 countries so this is definitely a big data platform to watch.

Coming soon: latest innovation using the Open Data Cube

An international study team, including Dr Dan Clewley from PML, will soon have research published on their development of a ‘National Digital Infrastructure’ for Wales that uses the ODC. The study demonstrated that this approach brings major advantages, including: the sharing of consistent datasets across governments and partner organisations; minimising duplication of effort; improving transparency, traceability and reproducibility; fostering collaboration between diverse stakeholders and communities; promoting inclusivity in environmental management decision-making, and supporting sustainable outcomes. Keep a look out on PML’s website for more information soon.