Most state-of-the-art approaches for weather and climate modeling are based
on physics-informed numerical models of the atmosphere. These approaches aim to
model the non-linear dynamics and complex interactions between multiple
variables, which are challenging to approximate. Additionally, many such
numerical models are computationally intensive, especially when modeling the
atmospheric phenomenon at a fine-grained spatial and temporal resolution.
Recent data-driven approaches based on machine learning instead aim to directly
solve a downstream forecasting or projection task by learning a data-driven
functional mapping using deep neural networks. However, these networks are
trained using curated and homogeneous climate datasets for specific
spatiotemporal tasks, and thus lack the generality of numerical models. We
develop and demonstrate ClimaX, a flexible and generalizable deep learning
model for weather and climate science that can be trained using heterogeneous
datasets spanning different variables, spatio-temporal coverage, and physical
groundings. ClimaX extends the Transformer architecture with novel encoding and
aggregation blocks that allow effective use of available compute while
maintaining general utility. ClimaX is pre-trained with a self-supervised
learning objective on climate datasets derived from CMIP6. The pre-trained
ClimaX can then be fine-tuned to address a breadth of climate and weather
tasks, including those that involve atmospheric variables and spatio-temporal
scales unseen during pretraining. Compared to existing data-driven baselines,
we show that this generality in ClimaX results in superior performance on
benchmarks for weather forecasting and climate projections, even when
pretrained at lower resolutions and compute budgets. The source code is
available at https://github.com/microsoft/ClimaX.
Description
[2301.10343] ClimaX: A foundation model for weather and climate
%0 Generic
%1 nguyen2023climax
%A Nguyen, Tung
%A Brandstetter, Johannes
%A Kapoor, Ashish
%A Gupta, Jayesh K.
%A Grover, Aditya
%D 2023
%K climate deeplearning idea:big_data_geo_2 transformer
%T ClimaX: A foundation model for weather and climate
%U http://arxiv.org/abs/2301.10343
%X Most state-of-the-art approaches for weather and climate modeling are based
on physics-informed numerical models of the atmosphere. These approaches aim to
model the non-linear dynamics and complex interactions between multiple
variables, which are challenging to approximate. Additionally, many such
numerical models are computationally intensive, especially when modeling the
atmospheric phenomenon at a fine-grained spatial and temporal resolution.
Recent data-driven approaches based on machine learning instead aim to directly
solve a downstream forecasting or projection task by learning a data-driven
functional mapping using deep neural networks. However, these networks are
trained using curated and homogeneous climate datasets for specific
spatiotemporal tasks, and thus lack the generality of numerical models. We
develop and demonstrate ClimaX, a flexible and generalizable deep learning
model for weather and climate science that can be trained using heterogeneous
datasets spanning different variables, spatio-temporal coverage, and physical
groundings. ClimaX extends the Transformer architecture with novel encoding and
aggregation blocks that allow effective use of available compute while
maintaining general utility. ClimaX is pre-trained with a self-supervised
learning objective on climate datasets derived from CMIP6. The pre-trained
ClimaX can then be fine-tuned to address a breadth of climate and weather
tasks, including those that involve atmospheric variables and spatio-temporal
scales unseen during pretraining. Compared to existing data-driven baselines,
we show that this generality in ClimaX results in superior performance on
benchmarks for weather forecasting and climate projections, even when
pretrained at lower resolutions and compute budgets. The source code is
available at https://github.com/microsoft/ClimaX.
@misc{nguyen2023climax,
abstract = {Most state-of-the-art approaches for weather and climate modeling are based
on physics-informed numerical models of the atmosphere. These approaches aim to
model the non-linear dynamics and complex interactions between multiple
variables, which are challenging to approximate. Additionally, many such
numerical models are computationally intensive, especially when modeling the
atmospheric phenomenon at a fine-grained spatial and temporal resolution.
Recent data-driven approaches based on machine learning instead aim to directly
solve a downstream forecasting or projection task by learning a data-driven
functional mapping using deep neural networks. However, these networks are
trained using curated and homogeneous climate datasets for specific
spatiotemporal tasks, and thus lack the generality of numerical models. We
develop and demonstrate ClimaX, a flexible and generalizable deep learning
model for weather and climate science that can be trained using heterogeneous
datasets spanning different variables, spatio-temporal coverage, and physical
groundings. ClimaX extends the Transformer architecture with novel encoding and
aggregation blocks that allow effective use of available compute while
maintaining general utility. ClimaX is pre-trained with a self-supervised
learning objective on climate datasets derived from CMIP6. The pre-trained
ClimaX can then be fine-tuned to address a breadth of climate and weather
tasks, including those that involve atmospheric variables and spatio-temporal
scales unseen during pretraining. Compared to existing data-driven baselines,
we show that this generality in ClimaX results in superior performance on
benchmarks for weather forecasting and climate projections, even when
pretrained at lower resolutions and compute budgets. The source code is
available at https://github.com/microsoft/ClimaX.},
added-at = {2023-11-15T11:53:39.000+0100},
author = {Nguyen, Tung and Brandstetter, Johannes and Kapoor, Ashish and Gupta, Jayesh K. and Grover, Aditya},
biburl = {https://www.bibsonomy.org/bibtex/2733d856a9c4a0ea41394caa8737e0b85/annakrause},
description = {[2301.10343] ClimaX: A foundation model for weather and climate},
interhash = {986c206f0d65a07c80471b0020a62517},
intrahash = {733d856a9c4a0ea41394caa8737e0b85},
keywords = {climate deeplearning idea:big_data_geo_2 transformer},
note = {cite arxiv:2301.10343Comment: International Conference on Machine Learning 2023},
timestamp = {2023-11-15T11:53:39.000+0100},
title = {ClimaX: A foundation model for weather and climate},
url = {http://arxiv.org/abs/2301.10343},
year = 2023
}