Based on 2,500 Square Kilometers of Real-world Data, the Beijing Normal University Team Proposed the StarFusion Model to Achieve High Spatial Resolution Image Prediction
Driven by the wave of science and technology, traditional agriculture is undergoing an intelligent transformation.As the "eyes and ears" of modern agriculture, agricultural remote sensing collects remote sensing data from platforms such as satellites and drones, and is widely used to monitor crop growth conditions, land use and cover changes, pest and disease conditions, water resource management, and the evolution of the agricultural environment.Agricultural remote sensing can provide decision makers with real-time and in-depth insights, ensuring the accuracy of information and the foresight of decisions.
However, the application of remote sensing data in agriculture usually requires high spatial resolution (HSR) and frequent observations. Due to factors such as the revisit period of optical satellite images and cloud contamination, the ability of satellites to monitor crop growth is often severely affected.
In order to obtain composite images with high spatial and temporal resolution, many spatiotemporal fusion methods have been developed, but the existing methods mainly focus on fusing low and medium spatial resolution satellite data in terms of model development and verification. When it comes to fusing medium and high spatial resolution images, their applicability still faces various challenges.
To solve these problems,Chen Jin's team from the State Key Laboratory of Earth Surface Processes and Resource Ecology at Beijing Normal University proposed a dual-stream spatiotemporal decoupling fusion architecture model called StarFusion.This model combines the traditional STF method with deep learning, which can overcome the problem that most existing deep learning algorithms require HSR time series images for training and fully realize the prediction of high spatial resolution images.
Research highlights:
* StarFusion fuses Gaofen-1 and Sentinel-2 satellite data to generate high temporal and spatial resolution images
* StarFusion maintains high fusion accuracy and good spatial details by combining a deep learning-based super-resolution method with a partial least squares regression model and using edge and color weighted loss functions
* StarFusion model shows better overall performance and temporal transferability than traditional spatiotemporal fusion methods and deep learning-based methods in 3 different experimental locations
Paper address:
https://doi.org/10.34133/remotesensing.0159
Dataset download address:
https://go.hyper.ai/LGmkW
The open source project "awesome-ai4s" brings together more than 100 AI4S paper interpretations and provides massive data sets and tools:
https://github.com/hyperai/awesome-ai4s
Dataset: Based on two counties in Shandong, integrated use of Gaofen-1 and Sentinel-2 satellite data
The study designed comprehensive evaluation experiments using datasets from Gaofen-1 and Sentinel-2 satellites, each covering the same geographical area.
* Study location
The study was conducted in Feixian and Yuncheng counties in Shandong Province, covering an area of approximately 2,567 square kilometers with a variety of land cover types. To ensure landscape diversity, the study identified three sub-regions as experimental sites, all of which were a mixture of farmland, water bodies, roads, and buildings, and the heterogeneity of the landscape within the region also showed obvious differences.
* Data preprocessing
As shown in the table below, the study mainly used the Gaofen-1 satellite to produce relatively clear images with a resolution of 2 meters, and the Sentinel-2 satellite to produce relatively coarse images with a resolution of 10 meters. The study then aligned and georectified the Sentinel-2 images with the 2-meter spatial resolution that matched the Gaofen-1 data.
In order to coordinate the spectral characteristics of Gaofen-1 and Sentinel-2, the study used a linear regression model, taking each band of Sentinel data as the dependent variable and the corresponding band of Gaofen-1 data as the independent variable, to construct a univariate regression model, and then applied the slope and intercept of the constructed model to each pixel of the Gaofen-1 image in each band. This fitting process ensures that the converted Gaofen-1 image can closely reflect the spectral response of Sentinel-2, thus effectively reducing the spectral mismatch between them.
StarFusion: Model integration of SRGAN and PLSR, supplemented with ECW weighting function
As shown in the figure below, StarFusion fuses Gaofen-1 data acquired on the base date (T1) with Sentinel-2 data acquired on the base date (T1) and the forecast date (T2).Based on the collected data, StarFusion integrates the two fusion models of SRGAN (Grad-SRGAN-STF) and PLSR into a two-stream spatiotemporal decoupled fusion architecture, which can bring out the advantages of the two models.
In the partial least squares regression model (PLSR) part, the study established a multivariate regression relationship model with higher degrees of freedom based on the scale invariant assumption, thereby more accurately predicting temporal changes. Considering that the spatial resolution of Sentinel-2 images is lower than that of Gaofen-1 images, the study also designed a gradient map SRGAN model (Grad-SRGAN) to handle the downscaling task from coarse images to fine images.
Furthermore, this study also proposed a SRGAN-STF model with gradient graph (Grad-SRGAN-STF).As shown in the figure below, Grad-SRGAN-STF is based on temporal transfer performance and mainly consists of three parts:
1) Generator: It consists of two parallel surface roughness SR (surface roughness) parameters, one for collecting the rough reflection image (Ref-SR) and the other for collecting the rough gradient image (Grad-SR).
2) Discriminator: Two independent discriminators DRef and DGrad are designed to determine whether the generated reflection image and gradient map are real or forged, respectively.
3) Loss function: It consists of two parts: content loss and adversarial loss. The loss function of the generator Ref-SR and Grad-SR is the weighted sum of content loss and adversarial loss. Among them, content loss can be used to evaluate the difference between the predicted image and the reference image, and adversarial loss can measure the probability that the generated image is recognized as a real image by the discriminator.
More importantly,The study also designed a new weight function called edge and color weight (ECW). ECW combines the powerful spatial detail recovery capability of the SRGAN method and the accurate temporal variation estimation of the PLSR method, and can provide more accurate predictions.
Research results: StarFusion performs best, Grad-SRGAN contributes the most
Experiment I: StarFusion has the best spatial detail fidelity compared to other algorithms
To evaluate the effectiveness of StarFusion, this study compared StarFusion with other representative algorithms (STARFM, FSDAF, Fit-FC, FIRST and SRGAN) based on Gaofen-1 and Sentinel-2 images taken on 2 different dates in 3 sub-areas, focusing on factors such as the accuracy of predicted image generation and spatial details.
The results are shown in the figure below.The image predicted by the StarFusion method (Figure G below) shows significant similarity to the reference image (Figure A below), demonstrating superior spatial detail fidelity.In contrast, the images generated by STARFM, FSDAF, Fit-FC, FIRST, and SRGAN all show some patches of blurred areas, and the field boundaries are not clear enough.
In addition, the following APA diagram also shows thatThe StarFusion method achieves strong spatial transfer capability without significant loss of accuracy or spatial details.It’s worth noting that StarFusion is visually on the edge of the “good” range, but doesn’t fall into that range, suggesting that its performance still has room for improvement in the future.
Experiment II: StarFusion predicts the best image quality over a long time span
Since Sub-Region-3 has complex landscape features, the study plans to use images of different base dates (T1, T2, T3) and their gradient maps to predict high-resolution images of T4. At the same time, the study also trained the GradSRGAN (Grad-SRGAN-Multi) and StarFusion (StarFusion-Multi) models and compared their performance.
The study found that when the prediction date is close to the benchmark date, all fusion methods work well, but when the time span increases, the image quality of STARFM, FSDAF, Fit-FC, FIRST and SRGAN decreases, and blurring and color distortion occur. StarFusion maintains good spatial details and color fidelity when inputting image pairs with different base dates, outperforming the other five methods.
also,StarFusion-Multi and Grad-SRGAN-Multi perform better when trained with multiple pairs of images than with single pair training, and StarFusion-Multi consistently outperforms Grad-SRGAN-Multi in terms of RMSE and Roberts margin.
Experiment III: Grad-SRGAN contributes most to StarFusion
Finally, this study also conducted an ablation study in sub-region-1 to demonstrate the contribution of each part in the proposed StarFusion method, including Grad-SRGAN, PLSR, and ECW.The results show that the introduction of Grad-SRGAN, PLSR and ECW functions all help to improve the fusion accuracy. In terms of their contribution, the introduction of Grad-SRGAN has the greatest contribution, followed by ECW weighting function and PLSR.
AI helps agricultural remote sensing enter the era of intelligent driving
With the continuous development of AI technology, digital agriculture is gradually becoming a new trend in agricultural development. The agricultural satellite remote sensing industry is in a critical period of gradually shifting from the traditional data-dependent model to the intelligent-driven model with AI as the core. More and more companies are deploying and competing in this field.
For example,Artificial intelligence companies represented by SenseTime have used their independently developed large-scale base models toThe launch of the "SenseTime Land" AI remote sensing large model for remote sensing has realized the automation, intelligence and normalization of arable land plots, crop identification and non-grain element identification. In addition,Technology giants such as Huawei used their Ascend AI to help Wuhan University build LuoJiaNet, a special framework for intelligent interpretation of remote sensing images, and LuoJiaSET, a remote sensing sample library.Accelerate the automatic interpretation of remote sensing images and enable the widespread application of intelligent remote sensing technology in industries such as agriculture and natural resources.
In the academic field, relying on the National Key Laboratory of Remote Sensing Science jointly established by the Institute of Remote Sensing Applications of the Chinese Academy of Sciences and Beijing Normal University,A large number of industry elites, including the author of this article, Professor Chen Jin, have emerged in my country and are gradually making important contributions to the development and progress of agricultural remote sensing technology.
As early as 1988, the laboratory established the Global Agricultural Remote Sensing Rapid Reporting System (Cropwatch). Based on remote sensing and ground observation data, the system can independently evaluate crop growth, yield and other related information at the national and global scales. With the help of AI technology, the laboratory has launched a number of achievements in the past period of time, including the new ultra-high-resolution optical forest three-dimensional remote sensing method of the AGAR algorithm and the wildfire satellite monitoring and early warning system with remote sensing AI algorithms.
The application of AI in agricultural remote sensing not only improves the efficiency and accuracy of agricultural production, but also provides strong technical support for the sustainable development of agriculture. It is foreseeable that the application of AI in agricultural remote sensing will become more extensive, providing more and more solutions for plant pest and disease monitoring, crop growth status and yield assessment, efficient management and optimization of agricultural production, etc.
Finally, I recommend an academic sharing activity!
The third live broadcast of Meet AI4S invited Zhou Ziyi, a postdoctoral fellow at the Institute of Natural Sciences of Shanghai Jiao Tong University and Shanghai National Center for Applied Mathematics. Click here to make an appointment to watch the live broadcast!
https://hdxu.cn/6Bjomhdxu.cn/6Bjom
