Accuracy Improved by 400%! The Indian Monsoon Prediction Model, Based on 36 Meteorological Stations, Enables Detailed Forecasts at the City level.
Mumbai, India, enters monsoon season from June to September each year. Extreme rainfall events have become frequent in recent years, with average rainfall increasing by nearly 401 TP3 tons compared to pre-2019 levels. This coastal city of 18 million people is often thrown into chaos by heavy downpours: delayed weather warnings often lead to work and school suspensions and even severe flooding. The devastating monsoon disasters create an urgent need for more accurate localized weather forecasts.
However, in tropical monsoon climates, the resolution of standard global weather models, approximately 25 square kilometers, makes it difficult to capture the subtle differences in local weather systems. The complexity of the terrain also exacerbates the spatial uncertainty of floods. Therefore, monsoon forecasts have previously been limited to macro-trends.
To fill the gap in urban flood risk prediction, the Indian Institute of Technology Bombay and the University of Maryland research team collaborated to develop a hyperlocal prediction model based on convolutional neural networks (CNN) and transfer learning (CNN-TL), which was able to predict most heavy rainfall events several days in advance.According to the latest report from Science, Mumbai is now considering incorporating it into its official early warning system, marking a new stage of hyperlocal flood forecasting in South Asian cities.
The related research results are pre-published on SSRN under the title "Hyperlocal Extreme Rainfall Forecasts in Mumbai: Convolutional Neural Network Transfer Learning-Based Downscaling Approach."
Research highlights:
* Achieve hyperlocal forecasts at the city scale, reducing the resolution to the urban area level, addressing the shortcomings of flood risk prediction in traditional numerical models (25km grid);
* Using a combination of convolutional neural networks and transfer learning, the model is trained based on full rainfall data and fine-tuned using extreme rainfall samples, effectively improving the model's ability to capture extreme events.
* The spatial synchronization pattern of extreme rainfall in Mumbai was revealed by event synchronization method and Louvain algorithm, which is more efficient than traditional clustering methods.
Paper address:
https://go.hyper.ai/j05Vt
Follow the official account and reply "India Monsoon Forecast" to get the full PDF
More AI frontier papers:
"Model data + forecast data", screening 36 observation stations based on data gaps
The datasets used in this study are divided into two categories: model data and measured data:
- Mode data:The Global Forecast System (GFS) from the U.S. National Centers for Environmental Prediction (NCEP) covers the period from June 2015 to September 2023, with a spatial range of 18°–20°N and 72°–74°E, and a resolution of 0.25°×0.25°;
- Forecast data:The data comes from Automatic Weather Stations (AWS) deployed by the Municipal Corporation of Mumbai (MCGM), with a recording period of every 15 minutes, covering the period from 2006 to 2023 (excluding 2014). Due to missing measurements and discontinuous data at some stations, only 36 observation stations with relatively complete data during the 2015-2023 monsoon season (June to September) were selected for analysis.
To ensure the quality of input data and improve the predictive performance of the model, the research team processed the data by eliminating missing data, screening variables, and performing spatial matching.For the raw observation data from automatic weather stations, the research team eliminated sites with significant data gaps, retaining only those with complete records for at least five consecutive years during the monsoon season (June–September). Furthermore, the researchers used correlation screening within the model data to select meteorological factors with a strong correlation with rainfall (such as precipitable water, precipitation, relative humidity, temperature, and air pressure) as predictor variables.
After preprocessing, the research team aligned the GFS grid data with the site observation data to construct "input-output" pairs as training samples for convolutional neural networks and transfer learning models.
Rainfall prediction based on convolutional neural network and transfer learning
The model uses GFS data as input and combines it with measured rainfall. It first uses a convolutional neural network to downgrade the coarse-resolution forecast to the site scale, then uses transfer learning to optimize extreme rainfall predictions. Finally, it combines site comparisons to identify regional rainstorm synchrony and improve overall forecast accuracy.
Louvain clustering based on event synchronization method
This study introduced the Event Synchronization method to identify regional rainstorms by comparing rainfall time series at different stations, revealing the spatial synchronization pattern of extreme rainfall in cities.Compared with traditional linear correlation analysis, this method is more suitable for capturing extreme rainfall conditions with strong suddenness and obvious nonlinear characteristics.
After inputting the preprocessed data, the research team used correlation analysis to select the GFS variables most closely related to rainfall. They considered the extreme rainfall processes at each station during the observation period as "events" and evaluated their synchronization in the time series by calculating the time difference and frequency of events between stations.
After outputting the event synchronization matrix between stations, the researchers further used the Louvain algorithm to cluster the relationships and divide stations with similar rainfall synchronization characteristics into the same group.
Downscaling Convolutional Neural Network Models
After completing the clustering,Based on the convolutional neural network model, the research team gradually downscaled the low-resolution GFS data to the site scale.
The researchers first screened the GFS gridded meteorological variables, retaining the factors most closely related to rainfall, such as precipitation rate, humidity, temperature and air pressure.And input into the convolutional neural network model in four-dimensional form.
The convolutional neural network used in this study is divided into convolutional layers and flattening layers. The convolutional layers extract spatial and temporal features from the input variables, while the flattening layers subsequently align these features with rainfall data measured at Mumbai's automatic weather station, establishing an input-output relationship. The local features captured and extracted by the convolutional neural network are then output to directly predict daily rainfall at each station, achieving a nonlinear down-conversion from coarse-resolution GFS data to fine-grained station-level rainfall.
It is worth mentioning thatThe researchers also introduced measures such as early stopping to avoid overfitting.
Transfer Learning Fine-tuning Optimization
In this study,The team introduced transfer learning methods to enable the model to recognize the characteristics of both general rainfall conditions and extreme rainfall events:Among them, the basic convolutional layer remains unchanged, while the part close to the output layer allows parameter updates to avoid the problems of high computational cost and insufficient samples, while preventing the model from overfitting on extreme samples.
The research team first trained the convolutional neural network basic model based on the complete rainfall data of the Mumbai observation station from 2015 to 2023 to ensure the model's stable prediction of general medium and low-intensity rainfall. In addition, during the fine-tuning stage, the researchers used high-intensity rainfall samples exceeding the 95% or 99% quantiles.Make the model "focus on learning" extreme events and enhance its sensitivity to extreme samples.
Verification of model performance in time and space
The research team used a variety of statistical indicators and real-world cases to verify the model's system performance. The results showed that the forecast model significantly outperformed the traditional Global Forecast System (GFS) and unoptimized convolutional neural network models in many aspects.The evaluation uses common quantitative indicators, including correlation coefficient (CC), root mean square error (RMSE), false alarm rate (FA), etc.
Overall rainfall forecast performance
In terms of overall rainfall prediction, the transfer learning model performs better than GFS.Downscaling the convolutional neural network significantly improved spatial accuracy, with generally higher correlation coefficients and significantly lower root mean square error (RMSE), effectively bridging the gap between the model and observations. The introduction of transfer learning made the model more robust in predicting moderate to high-intensity rainfall, reducing underestimations. Overall, transfer learning outperformed other models for forecasts one to three days in advance, with particularly strong performance for forecasts two days in advance.
In extreme rainfall prediction, transfer learning enhances the model's ability to identify extreme samples.In the test of 95% and 99% quantiles, the time prediction accuracy of the transfer learning model is 60%–400% higher than that of GFS. The 1st and 2nd day forecasts can capture most rainstorm processes earlier and effectively reduce the false alarm rate.
Model regional performance and practical application
The study validated the model's ability to synchronize rainfall forecasts across different regions of Mumbai using event synchronization and spatial clustering methods. The results showed that the transfer learning model not only outperformed the pre-optimized model in single-point forecasts but also reproduced the spatial clustering characteristics between sites.
Building a large-scale model of local sovereignty, India accelerates its entry into the field of artificial intelligence
In fact, in addition to the academic community's attention to the application of AI achievements such as monsoon prediction models,The Indian government is also leading the development of AI.Ashwini Vaishnaw, Minister of Electronics and Information Technology, recently announced the launch of the national "IndiaAI Mission," with the goal of completing the development of multiple indigenous foundational models within the next six to ten months. These large-scale language models will be designed with India's multilingual and multicultural environment in mind and optimized for local social needs. To this end, India has purchased approximately 18,600 GPUs from around the world to establish the IndiaAI Compute Facility, ensuring a continuous supply of basic computing power.
at the same time,In order to form independent and controllable core technologies, the Indian government also selected local AI company Sarvam as a key partner to undertake the task of building India's first "sovereign big model".This model will be able to handle multiple languages in India and emphasize reasoning and adaptability to large-scale application scenarios. In the future, it will be able to serve multiple fields such as healthcare and government affairs.
In addition, the Indian government has also launched the AI Kosha data platform, which aims to centrally manage, open and share public and non-sensitive data that can be used to train AI. The India AI Compute Facility will also provide subsidized GPU usage rights to startups and research teams.We hope to quickly build a R&D environment comparable to those in Europe, America, and East Asia.
As the IndiaAI Mission progresses, it is not difficult to foresee that India will gradually transform from a follower to an active participant in the global AI competition landscape.
Reference Links:
1.https://timesofindia.indiatimes.com/technology/artificial-intelligence/government-selects-indian-ai-company-sarvam-to-develop-nations-first-sovereign-large-language-model/articleshow/120645896.cms
2.https://indiaai.gov.in/article/union-minister-of-electronics-it-railways-and-i-b-announces-the-availability-of-18-000-affordable-ai-compute-units
3.https://www.electronicsforyou.biz/industry-buzz/indiaai-mission-expands-to-18693-gpus-for-rd/