Artificial Intelligence in DestinE

The explainer

Destination Earth (DestinE) – the ambitious initiative of the European Commission (EC) to create digital twins of our planet – is harnessing recent breakthroughs in the application of artificial intelligence (AI) and machine learning (ML) in weather and climate to augment its capabilities.  Implemented under the leadership of the EC’s DG CNECT, by ECMWF, ESA, EUMETSAT and many partners throughout Europe, and through a strategic partnership with the European HPC Joint Undertaking (EuroHPC JU), DestinE will also pave the way for new advancements in AI for Earth system sciences.

In the second phase of DestinE, a wide range of AI-focused activities are being undertaken, engaging all areas of the DestinE system, from technological infrastructure to weather and climate modelling.  The activities led by ECMWF and its partners focus on developing ML-based Earth-system model components to advance towards a fully integrated Earth-system ML model. They also involve expanding the Digital Twin Engine software solutions to support training and running this new generation of ML models by harnessing the world-class supercomputers of the EuroHPC JU. Practically, these AI activities will help quantify uncertainty around the digital twins’ high-resolution simulations and enhance the system’s interactive features.

This explainer revisits recent advances in AI for weather forecasting and describes the different AI-focused activities carried out in DestinE by ECMWF and its partners.

AI activities - DestinE

AI in Weather Forecasting

A groundbreaking evolution

In recent years, advances in AI have transformed weather forecasting with a series of machine learning or data-driven-based models demonstrating impressive improvements in accuracy and speed. These models can generate 10-day forecasts in minutes, compared to the average one hour to generate a forecast with a physics-based model, while consuming up to 100 times less energy. The rapid advancement of AI in weather forecasting is often described as revolutionary: starting only in 2018, progress has accelerated in the last couple of years to achieve a level of performance that is comparable and, for certain metrics, better than traditional numerical weather prediction models.

Progress of AI-weather models over time, chart courtesy of Stephan Rasp

The Artificial Intelligence Forecasting System

Also playing a key role in this revolution, ECMWF and its Member States rapidly developed their own AI-based weather prediction model, the Artificial Intelligence/Integrated Forecasting System (AIFS). Since 2024, the AIFS is running four times daily, providing open charts and data.

This animation shows a 10-day forecast of meridional wind at 850 hPa using ECMWF’s Artificial Intelligence Forecasting System (AIFS).

Anemoi: A Collaborative Software Framework for AI Models 

Shortly after the launch of AIFS, ECMWF and its Member States have introduced Anemoi, an open-source software application designed to provide the essential building blocks for training state-of-the-art data-driven models and deploying them in an operational setting. Anemoi encompasses a comprehensive ecosystem of Python packages, addressing the entire life cycle of data-driven modelling – from data preparation and model development to operational execution – ensuring a seamless and efficient workflow for advanced modelling applications. 

Illustration of the promising results for a 7-day forecast of 10 m wind speed (shading) and sea-level pressure (contours), obtained with the regionally high-resolution AI-based model Bris, developed by MET Norway and partners, making use of the Anemoi framework. The model has learned to forecast at high resolution (here about 2.5 km) inside the Nordic region, and at low resolution (here about 30 km) outside of this domain. The model successfully creates a higher-resolution structure over the Nordics.

AI Fundamentals

Physics-based models

Traditional, physics-based, weather forecasting models use a three-dimensional grid to model physical processes, by solving the equations of the atmospheric flow and by making assumptions about the impacts of the processes that take place at scales smaller than the model grid box.  Starting from the best possible reconstruction of the earth system state at a given time – or “initial conditions” – the models solve these equations over time requiring billions of calculations at grid points, utilising between 10 and 1,000 supercomputer nodes, depending on the model resolution.

Ai models

AI models use the same “initial conditions” as a starting point, but the way they predict the future from this state is different. As statistical models, they have analysed patterns in historical data from the past few decades to build a fully learnt model of the Earth and then use this to make predictions.

Training the ML models

AI models are trained on nearly 40 years of historical weather data (e.g., ECMWF’s ERA5 reanalysis produced in the framework of the Copernicus Climate Change Service) to identify patterns and relationships in key variables like temperature, humidity, and wind speed.

Training phase

The training phase is relatively computationally intensive, requiring a few weeks and the combined power of more than 50 Graphics Processing Units (GPUs). However, this is still far smaller than the computer power being used to train the state-of-the-art large language models (e.g. ChatGPT). 

Inference phase

Once trained, the AI model enters the “inference phase”, where it applies its learning to make predictions on new data. In this phase, the model showcases its true efficiency: it can generate 10-day weather forecasts in a couple of minutes using just a single GPU. This underlines the transformative potential of AI in weather forecasting.

Benefits

Machine learning models such as ECMWF’s AIFS have shown great potential in advancing weather forecasting. For some phenomena, they have been shown to enable more accurate predictions, for example, identifying the most likely path of tropical cyclones. With significantly lower computational requirements, the AI models can deliver powerful insights quickly, opening new possibilities for maximising the potential of all forecasting techniques.

AI in DestinE

Facilitating their training is essential for advancing the next generation of data-driven models. Not only is this necessary for achieving the developments and integrations of a new generation of ML models within DestinE, but it supports the broader landscape of AI advancements in weather and climate applications across Europe. Although computationally demanding, these training activities are a key catalyst for progress.

EuroHPC in DestinE: a Premier Resource for AI Training

The EuroHPC supercomputers at the heart of DestinE are equipped with the latest accelerator technology, making them the ideal environment for AI training. These accelerators, such as GPUs, can perform massive numbers of calculations simultaneously, which is a strong asset for the demanding AI training tasks.

By providing cutting-edge computational performance, the EuroHPC supercomputers can help accelerate the training of ML models, enabling developers to iterate on models more quickly and unlock breakthroughs in AI capabilities. However, as the application of ML techniques for Earth-system modelling is still relatively new, their full exploitation requires the deployment of novel approaches for data management and of specific software.

Expanding software solutions for the new generation of AI models 

Due to its complex decentralised environment and the unprecedented volume of data generated by the digital twins, DestinE has brought innovative data handling techniques into its architecture, for example through the Digital Twin Engine (DTE), one of DestinE’s main components. Among its various functions, the DTE ensures the seamless management of the vast data volumes produced by the digital twins. This advanced framework not only facilitates efficient data management but also provides a solid foundation for effectively leveraging both DestinE digital twin data and the EuroHPC supercomputers for training a new generation of ML models in DestinE.

The activities undertaken by ECMWF and its partners in DestinE’s second phase are further expanding the software solutions of the DTE, supporting the development of AI-driven weather and climate models, and their integration into supercomputing environments like those of EuroHPC.

For example, some of the main developments will enhance the Digital Twin Engine by embedding ML pipelines into its framework. The goal is to further develop its data handling and processing capabilities, ensuring they enable the development, training, and execution of ML/AI models on EuroHPC supercomputers by exploiting the DestinE digital twins data.

Key efforts include optimising GPU utilisation and parallelisation for more efficient data management, designing new pipelines and workflows, and integrating ML processes into the DestinE DTE production environment. This integration will streamline the configuration, execution, and monitoring of ML training and inference of the ML Earth-system components developed in DestinE.

Scaling efficiency of training AIFS on EuroHPC systems. For both EuroHPC computers (Leonardo and MareNostrum5), efficiency is close to 1.

The activities carried-out in DestinE will also lead to further performance improvements to maximise EuroHPC utilisation and adaptations to allow the development of AI Earth-system models. These include handling of domain-specific variables and easy coupling of ML Earth-system components.

The importance of quality data: unlocking new horizons with DestinE’s Digital Twins

The quality of data is essential for developing accurate and reliable AI models, especially for weather and climate predictions. Currently, AI models are trained on high-quality atmospheric and land data, such as that provided by the ERA5 reanalysis. However, the resolution of this data is still fairly coarse (about 30km) and does not cover all Earth-system components. Training AI models with a broader range of high-resolution data, that also covers other Earth system components like ocean, sea ice and hydrology – such as that provided by DestinE’s digital twins – would allow for the development of new, more integrated and precise models. The ongoing infrastructure adaptations, as previously outlined, are essential for enabling more efficient AI training and leveraging the digital twins’ data to advance progress toward a fully integrated Earth-system model.

Towards an ML-based Earth System model within DestinE

DestinE’s AI activities led by ECMWF and its partners aim at expanding ECMWF’s AIFS model, which currently only represents the atmosphere, to all components of the Earth System, including land, ocean, sea ice, waves and hydrology to make progress towards an ML-based Earth System model.

Components of Earth system model

At ECMWF in the context of DestinE, work is ongoing to develop ML components for land, ocean, sea ice, waves and hydrology, with exciting first results having already been obtained with early prototypes.

Activities to develop a prototype ML-based Climate Emulator have also recently begun, aimed at expanding the ML modelling capabilities towards longer climate timescales by exploiting the data produced by the Climate Adaptation Digital Twin.

The development of a data-driven Earth-system model will allow it to complement the cutting-edge physics-based Earth system models underpinning DestinE’s digital twins. For example, machine learning modelling can support uncertainty quantification for the high-resolution physically-based simulations of the digital twins, and help increase the interactivity of the DestinE system.

Early results from AI Earth system components for DestinE

Sea Ice

Ocean

Ocean Surface Waves

Hydrology

Applying DestinE model developments:

Ensembles forecasting

Uncertainty is an inherent factor in weather and climate predictions due to the chaotic nature of the atmosphere. To address this, an “ensemble approach” is often used, which involves running multiple simulations with slight physical variations in the initial conditions and the model integrations. This method generates a range of possible outcomes, helping indicate the likelihood that the outcome of a single forecast – known as a deterministic forecast – will occur. This is particularly valuable for extremes weather events and efficient decision-making.

While essential, this method is also computationally demanding, particularly as the models’ resolution increases.

DestinE’s Digital Twins run at very high resolution: using 4.4 to 10 km resolution simulations for the Climate Adaptation Digital Twin, 4.4 km resolution for the global component and down to a 500 m regional model for the on-demand component of the Weather-Induced Extremes Digital Twin. These simulations harness the power of EuroHPC supercomputers to provide multidecadal climate predictions and detailed information on extreme weather events for a few days ahead.

Running tens of these high-resolution simulations in a timely manner would be very expensive and would require computational capabilities that currently exceed what is feasible today.

The power of machine learning to help quantify uncertainty

This is where machine learning shows great potential. Due to their ability to run similar forecasts using less computational resources, data-driven weather models are considered a valuable complement to physically-based models. They can be used to perform ensemble forecasts, which help estimate the uncertainty of a single deterministic forecast generated by a physics-based model. As an example, with the AIFS, an ensemble of 100 forecasts can be produced at relatively low cost.

Applying this approach in DestinE in the future, is a powerful way to enhance the system’s modelling capabilities, which are already supported by the cutting-edge Earth system models underpinning the digital twins.

DestinE is an interactive system designed to facilitate access to the unprecedented amount of data and simulations produced by the digital twins. ECMWF also undertakes new AI-focused activities to expand these interactive features.

Towards an AI “forecast in a box”

For example, ECMWF is working towards developing a ‘forecast in a box’ concept.  This AI-driven solution aims to leverage the speed and low-compute-cost capabilities of AI models, allowing specific users to run their own forecasts directly on their own computers and facilities (local hardware or cloud).

How will this work? Users will be provided with initial conditions for the forecasts based on ECMWF’s state-of-the-art analysis and machine learning models packaged together and will be able to run these simulations within few minutes to get results tailored to their needs.

DestinE weather and climate Chatbots

The data and information generated by weather and climate models are not always accessible to the general public without extensive training in earth system science. At the same time, the need for localised information on the impacts of climate change and extreme events is increasing, and it is also not practically achievable for scientists to accommodate the vast array of weather and climate-related questions to all specific cases.

Addressing users’ specific scenarios is inherently part of DestinE, as it aims to provide bespoke simulations through its digital twins and to involve interactive features.  Harnessing machine learning tools, specifically large language models (LLM) in combination with the digital twins’ data, opens new horizons to reinforce and achieve this interactivity.

One of the AI developments currently being carried out by ECMWF and its partners in DestinE focuses on developing prototype weather and climate chatbots. These chatbots will be able to provide answers regarding for example adaptation to a changing climate guided by digital twin data and other data sources.

This set of activities in DestinE builds on these existing efforts, enhancing and expanding them to provide tailored climate solutions for users. For example, promising results have been shown with the ClimSight prototype developed by the Alfred Wegener Institute, which demonstrates how large-language-model systems can provide localised climate information, overcoming challenges in obtaining user-specific data.

To ensure decision makers can have trust and confidence in new built tools and algorithms in DestinE, the use of emerging AI technologies must align with principles of ethics and responsibility. Dedicated tasks and activities are also underway in DestinE to support the establishment of guidelines for the ethical and responsible use of AI. Through a contract procured by ECMWF, DestinE will deliver multiple white papers and guidelines, providing insights to inform policies and standards on ethical AI use and shaping the future of AI strategy within DestinE and beyond.

Driving Innovation

The AI advances in DestinE contribute to the wider AI-driven innovation, particularly supporting Europe’s efforts to develop “AI factories”- dynamic ecosystems designed to create cutting-edge generative AI models.

Overall, the combination of advanced AI, the digital twin technology pioneered in DestinE, and the world-class supercomputing facilities of EuroHPC will contribute to enhancing disaster resilience, supporting climate adaptation, and developing next-generation AI solutions in Europe.

Watch DestinE videos on Artificial Intelligence

Destination Earth is a European Union funded initiative launched in 2022, with the aim to build a digital replica of the Earth system by 2030. The initiative is being jointly implemented by three entrusted entities: the European Centre for Medium-Range Weather Forecasts (ECMWF) responsible for the creation of the first two ‘digital twins’ and the ‘Digital Twin Engine’, the European Space Agency (ESA) responsible for building the ‘Core Service Platform’, and the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT), responsible for the creation of the ‘Data Lake’.

We acknowledge the EuroHPC Joint Undertaking for awarding this project strategic access to the EuroHPC supercomputers LUMI, hosted by CSC (Finland) and the LUMI consortium, Marenostrum5, hosted by BSC (Spain) Leonardo, hosted by Cineca (Italy) and MeluXina, hosted by LuxProvide (Luxembourg) through a EuroHPC Special Access call. 

More information about Destination Earth is on the Destination Earth website and the EU Commission website.

For more information about ECMWF’s role visit ecmwf.int/DestinE

For any questions related to the role of ECMWF in Destination Earth, please use the following email links:

General enquiries