Destination Earth was well represented at the European Geosciences Union’s 2023 General Assembly. A dedicated session on 26 April gathered scientists from the three participating entities (ECMWF, ESA, EUMETSAT) as well as contractors developing different components of the ambitious initiative of the European Union to create digital twins of the Earth system. In addition to a long list of presentations from scientists, ECMWF presented three posters to introduce key aspects of the software infrastructure behind the Weather-Induced Extremes Digital Twin, the Climate Adaptation Digital Twin and the Digital Twin Engine – which ECMWF is responsible to deliver in DestinE.
One of the main challenges for the digital twins’ infrastructure will be to handle the massive amounts of data that the high-resolution simulations will produce every day. Anticipating these challenges, ECMWF scientists are working on various software solutions to ensure that users can access the data they need efficiently.
ECMWF scientists for DestinE, Mathilde Leuridan, Antonino Bonanni and Domokos Sármány, took their posters carefully packed all the way from Bonn and Reading to Vienna, to share with their fellow scientists the latest about the software solutions they are working on for Destination Earth’s digital twins.
“EGU is a great place to meet all actors in the Destination Earth initiative and to collaborate with colleagues,“ said Mathilde Leuridan. “It’s great to see so much interest in ECMWF’s work on the Destination Earth initiative, with the sessions on digital twins being very popular amongst attendees!” she added.
“Our poster session on ‘First steps towards Destination Earth’ felt very engaging and interactive with many people asking questions and also providing very valuable feedback!” said Antonino Bonanni.
Here follows a brief description of the three posters presented by ECMWF’s Destination Earth team at the EGU’s General Assembly.
Mathilde Leuridan, James Hawkes, Tiago Quintino, Simon Smart and Emanuele Danovaro – Polytope: a revolutionary algorithm for accessing big data
ECMWF real-time forecast activities already produce 120 terabytes of data daily and with the Destination Earth digital twins, this is expected to reach more than 1 petabyte per day.
ECMWF’s weather data can be represented as a multi-dimensional datacube (6D or 7D). The existing data access mechanisms on such datacubes are however quite limited. They can indeed only perform a limited range of user requests, in effect “bounding boxes” around data of interest, and in fact need to “read” the complete global fields before extracting data. This process can prove to be very inefficient when working with petabytes of data.
Instead, with the prototype Polytope software developed by ECMWF, Destination Earth users will be able to extract complex shapes of data, such as spatio-temporal paths, from datacubes. This significantly reduces the volumes of data delivered to the user. At the same time, Polytope removes post-processing complexity as the user will only receive the specific data requested rather than bounding boxes of data around the requested shapes.
The first examples show a promising 30-70% data reduction in country data extraction and over 99.9% reduction for more complex requests when using Polytope rather than the current bounding box extraction technique.
To make Polytope more user-friendly, three different API levels have been implemented: the high-level, mid-level and low-level API. This allows the software to accommodate different users and their specific types of requests.
Domokos Sármány Philipp Geier, Mirco Valentini, Simon Smart, James Hawkes, and Tiago Quintino – MultIO: message-driven data for distributed Earth-system models
In recent years, the need for increased model resolution for the sake of ever more accurate weather forecasts and climate projections has brought new challenges in terms of performance and scalability. Instead of floating-point operations, access to sufficient data in memory and, especially, throughput to storage, have become significant bottlenecks – a phenomenon often referred to as the input/output (I/O) performance gap.
MultIO, an open-source software library, offers a two-fold message-driven solution to this problem. The first is an I/O-server functionality to create global horizontal fields from distributed meteorological and Earth-system models, efficiently aggregating fields from massively-parallel simulations. The second is on-the-fly (in memory) post-processing to compute derived meteorological products as these fields pass through MultIO.
MultIO is based on “message-driven data routing”: routing decisions are based on the message metadata and the configuration of the post-processing pipelines, through which each message passes. This makes it simple for users to define on-the-fly processing pipelines for particular fields. The result is a simple framework for users to move calculation closer to the data, and thus reduce the burden on the storage significantly.
When acting as a generic I/O server, MultIO permits asynchronous I/O, ensuring that model computations are neither blocked nor significantly slowed down during data output.
MultIO is a core component of ECMWF’s software infrastructure, partly co-developed via ECMWF’s participation in Destination Earth and is a key component of the Digital Twin Engine (DTE). It is also partially co-developed in the ACROSS project, funded by the EuroHPC JU.
The software is available at github.com/ecmwf/multio
Antonino Bonanni, James Hawkes, Tiago Quintino – Plume: a plugin mechanism for Numerical Weather Prediction models
Plume is a Plugin Mechanism for Numerical Weather Prediction (NWP) models, allowing them to load and run plugins implementing specific model functionalities.
Plume offers a controlled and flexible execution environment where plugins can access and process selected model data directly in memory, therefore reducing the need for computationally expensive and resource-consuming I/O operations.
Plugins are loaded at runtime and data are accessed through a well-defined interface provided by Atlas, a software library created by ECMWF that supports the development of Earth system model components and data processing.
At the beginning of the model run, Plume negotiates with prescribed plugins and decides which ones are activated according to both data offered by the model and data required by each plugin (see illustration above). During the simulation, data are then updated by the model and plugins can access and process them on the fly.
The software is available at github.com/ecmwf/plume
Destination Earth is a European Union-funded initiative launched in 2022, with the aim to build a digital replica of the Earth system by 2030. The initiative is being jointly implemented under the leadership of the EU Commission’s DG CNECT by three entrusted entities: the European Centre for Medium-Range Weather Forecasts (ECMWF), responsible for the creation of the first two ‘digital twins’ and the ‘Digital Twin Engine’, the European Space Agency (ESA) responsible for building the ‘Core Service Platform’, and the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT), responsible for the creation of the ‘Data Lake’.
We acknowledge the EuroHPC Joint Undertaking for awarding this project strategic access to the EuroHPC supercomputers LUMI, hosted by CSC (Finland), and the LUMI consortium, Marenostrum5, hosted by BSC (Spain) Leonardo, hosted by Cineca (Italy) and MeluXina, hosted by LuxProvide (Luxembourg) through a EuroHPC Special Access call.
More information about Destination Earth is on the Destination Earth website and the EU Commission website.
For more information about ECMWF’s role visit ecmwf.int/DestinE
For any questions related to the role of ECMWF in Destination Earth, please use the following email links: