A Physics-Informed Machine Learning Approach utilizing Multiband Satellite Data for Solar Irradiance Estimation

Abstract

Solar irradiance is fundamental data crucial for analyses related to weather and climate. High-precision estimation models are necessary to create areal data for solar irradiance. In this study, we developed a novel estimation model by utilizing machine learning and multiband data from meteorological satellite observations. Particularly under clear-sky and thin clouds, satellite observations can be influenced by surface reflections, which may lead to overfitting to ground observations. To make the model applicable at any location, we constructed the model incorporating prior information such as radiative transfer models and clear-sky probability, based on physical and meteorological knowledge. As a result, the estimation accuracy significantly improved at validation sites.

1 Introduction

Solar irradiance data is essential for monitoring solar power, assessing potential output, climate simulations, and climate monitoring. These applications often require high-precision areal data reflecting regional characteristics. In principle, it is possible to estimate solar irradiance through radiative transfer calculations (Mueller et al., 2009; Xie et al., 2016; Zhang et al., 2018); however there are some challenges such as cloud modeling and computational costs. As a result, statistical models based on meteorological satellite data are widely used (Engerer et al., 2017; Saito et al., 2018; Hashimoto & Yoshimoto, 2023).

However, these standard models are simplistic, utilizing limited band information. For example, Saito et al. (2018) uses visible and infrared band data for weather classification with rule-based algorithms and piecewise linear regression for estimation. Actually, the relationship between solar irradiance and satellite observations is highly complex due to the diversity of clouds, suggesting that combining multiband observation data with machine learning models can lead to more precise estimations of solar irradiance.

In previous studies where machine learning was applied to solar irradiance estimation models, primarily limited elements such as the visible band and simpler architectures like MLPs (Multilayer Perceptrons) were used, indicating that there is potential for improvement in accuracy (Cornejo-Bueno et al., 2019; Jiang et al., 2019; Yeom et al., 2019; Palacios et al., 2022).

This study aims to develop a high-precision model applicable at any location, leveraging multiband satellite images and machine learning. To avoid overfitting to ground observations and ensure applicability at any site, we incorporated constraints based on physical and meteorological knowledge to enhance the model’s generalizability. we have quantitatively demonstrated the model’s superiority by comparing it with the Saito’s standard model (2018) under consistent conditions.

2 Data description

In this study, we used satellite image data from Japan’s Himawari-8, the imager on board of which has 16 observation bands ranging from $0.47\mu$m to $13.3\mu$m in wavelength. Six of these bands are in the shortwave spectrum (visible and near-infrared) and ten in the longwave spectrum (infrared), with the visible bands strongly correlating with solar irradiance due to their high spectral intensity. Near-infrared bands reflect the size and phase state (water or ice) of cloud particles, and infrared bands vary in vertical sensitivity according to their wavelengths, providing vertical structural information of clouds (Shimizu et al., 2017).

For model training, we used one-minute interval ground observations from 47 sites across Japan, provided by the Japan Meteorological Agency. For validation, we used solar irradiance data independently observed in this study (Ishii et al., 2013). The observation sites are Naganuma, Tsukuba, Gifu, Tosu, and Okinoerabu, referred to as ST1, ST2, ST3, ST4, and ST5, respectively. Each site has distinct solar irradiance characteristics and surface conditions, allowing for a comprehensive evaluation of the model’s accuracy and generality across diverse environmental conditions. The model was trained over four years, from 2016 to 2020, excluding 2018, with a specific test period set for 2018.

3 Methodology

In this study, we utilized a deep learning framework to model the relationship between satellite imagery and solar irradiance. Specifically, we input data from 16 satellite observation bands and trained the model using ground-measured solar irradiance as the truth values. However, under clear skies and thin clouds, the contribution of reflections and emissions from the surface to the satellite observations is significant. Due to the limited number of ground observation sites compared to the diversity of surface conditions (such as urban areas, forests, seas, and snow covers), there is a risk of the model overfitting to the characteristics of the observation sites. To address this, we incorporated a radiative transfer model and clear-sky probability to improve the model’s generalizability to unknown locations. The developed model consists of the following three sub-models:

• Clear-Sky Solar Irradiance Estimation Model.

• All-Sky Solar Irradiance Estimation Model.

• Clear-Sky Probability Estimation Model.

3.1 Clear-Sky Solar Irradiance Estimation Model

This sub-model is based on the model by Kondo (1994) and is represented by the following equations:

$$I_{\text{clr}}=a_{0}I_{0N}\cos Z(1+a_{1}\times 10^{-a_{2}m})C_{w}$$ (1)

$$C_{w}=[1+a_{3}\log_{10}w+a_{4}\log^{2}_{10}w+a_{5}m\log_{10}w]$$ (2)

Where $I_{\text{clr}}$ is the solar irradiance under clear-sky conditions, $I_{0N}$ is the extraterrestrial solar irradiance, $Z$ is the solar zenith angle, $m$ is the air mass, and $w$ is the precipitable water vapor. $C_{w}$ represents the term for seasonal variation of solar irradiance due to changes in water vapor amount. The precipitable water vapor $w$ is estimated based on the date and latitude of the target site. The parameters $a_{0}$ to $a_{5}$ are optimized using ground-measured solar irradiance data. Since this model is determined only by the location and date/time and does not rely on satellite observations, it can be applied to any region.

3.2 All-Sky Solar Irradiance Estimation Model

Figure 1: Architecture of the all-sky solar irradiance estimation model. Satellite images from multiple bands are cropped around the target site and used as input. Features of the spatial distribution of clouds are extracted using a CNN, and the transmittance and reflectance of clouds are predicted using an MLP. Finally, downward and upward shortwave radiation are calculated using radiative transfer equations. Additionally, upward shortwave radiation is directly calculated from satellite observation data.

This sub-model calculates the all-sky solar irradiance from 16-band satellite images. First, we formulated a radiative transfer model based on Liou (2002). When the flux corresponding to clear-sky irradiance, $I_{\text{clr}}$, is incident on a multilayer atmospheric system consisting of clouds, atmosphere, and surface, the radiative transfer equations, considering multiple reflections between clouds and the surface, are expressed as follows:

$$I_{\text{all}}=\frac{T_{c}}{1-R_{s}R_{c}}I_{\text{clr}},I_{\text{up}}=\left(R_{c}+\frac{T_{c}^{2}R_{s}}{1-R_{s}R_{c}}\right)I_{\text{clr}}$$ (3)

Here, $I_{\text{all}}$ represents the downward shortwave radiation at the surface, corresponding to the all-sky solar irradiance, and $I_{\text{up}}$ is the upward shortwave radiation at the top of the atmosphere. $T_{c}$ and $R_{c}$ are the transmittance and reflectance of clouds, respectively, and $R_{s}$ represents the surface albedo. $I_{\text{clr}}$ is estimated using equation 1,2, and $R_{s}$ is estimated from the most recent visible band images under clear-sky conditions.

The transmittance $T_{c}$ and reflectance $R_{c}$ are calculated using a neural network with multiband satellite images as inputs. The architecture, as shown in Figure 1, consists of a CNN (Convolutional Neural Network) and an MLP. Here, the input data comprises satellite images of a rectangular area approximately 20 km around the target site. It is expected that the CNN can extract features of the cloud’s spatial structure, thereby improving the representation of scattered irradiance.

The calculated $T_{c}$ and $R_{c}$ are converted into the upward and downward shortwave radiations through equation 3. Here, $I_{\text{all}}$ is optimized using the ground-measured irradiance $I_{s}^{(\text{o})}$, while $I_{\text{up}}$ corresponds to the shortwave radiation observed by the satellite, integrated over the wavelength region, denoted as $I_{\text{up}}^{\prime}$, which can approximately be expressed as:

$$I_{\text{up}}^{\prime}=I_{0N}\left(\sum\nolimits_{j=1}^{6}\theta_{j}B_{j}+\theta_{0}\right)$$

Here, $B_{j}$ represents the shortwave radiation observed by the satellite, and $\theta_{j}$ are the learning parameters. Assuming that absorption by the atmosphere, ice clouds, and water clouds is low (Hale & Querry, 1973; Warren, 1984), $R_{c}+T_{c}\approx 1$, the overall loss function can be expressed using the hyperparameters $\alpha$, $\beta$, $\gamma$ as follows:

$$\mathcal{L}=\alpha\|I_{\text{all}}-I_{s}^{(\text{o})}\|_{2}+\beta\|R_{c}+T_{c}-1\|_{2}+\gamma\|I_{\text{up}}-I_{\text{up}}^{\prime}\|_{2}$$

Through optimization with this loss function, the energy balance is maintained so that $I_{\text{all}}$ fits the ground observations while $I_{\text{up}}$ remains consistent with satellite observations, $I_{\text{up}}^{\prime}$, thus preventing overfitting to observation sites. In practice, optimizing without energy-related constraints resulted in outliers under complex conditions such as thin clouds over snow cover (Figure 2).

Figure 2: Differences in estimated solar irradiance with and without radiative energy constraints. Black dotted, red, and blue lines indicate ground observations, estimation without constraints, and estimation with constraints, respectively. Time series for ST1, March 7, 2018.

3.3 Clear-Sky Probability Estimation Model

Figure 3: Processing flow and example results of the clear-sky probability estimation model. Satellite images from multiple bands are used as inputs to U-net, which outputs the contribution maps for different radiation sources such as seas, forests, snow covers, upper clouds and middle clouds. Input data include images from various bands (B03, B04, B05, B06, B10, B13) as well as surface albedo from Band03 and Band06 (SfcB03, SfcB06), and surface brightness temperature from Band13 (SfcB13). The results shown here are for the Kanto region, spanning latitudes 35°N to 38°N and longitudes 137°E to 143°E, on January 15, 2018, at 14:00.

Especially under clear sky conditions, the influence of reflections and emissions from the surface is significant, and there are limitations in enhancing the generalizability of the all-sky solar irradiance estimation model to unknown locations. Therefore, we introduce the clear-sky probability, $P_{\text{clr}}$, defined as a random variable that takes the value 1 in clear-sky and 0 in cloudy-sky conditions. The final estimated solar irradiance, $I_{s}$, is calculated as the expected value based on the clear-sky probability:

$$I_{s}=P_{\text{clr}}\times I_{\text{clr}}+(1-P_{\text{clr}})\times I_{\text{all}}$$

This approach reduces the influence of the surface and ensures accuracy for unknown locations by using a larger proportion of the clear-sky solar irradiance, $I_{\text{clr}}$, which is independent of satellite observations, under clear-sky conditions.

This sub-model is designed based on the model by Akimoto et al. (2020). First, we perform cluster analysis on states defined by combinations of multiband satellite observation values, $B_{i}$ (where $i$ denotes the band type), such as $(B_{i_{1}},B_{i_{2}},...)$. The centroid of each cluster, $b_{ji}$ (where $j$ is the cluster number), can be considered as representative states corresponding to different radiation sources. By estimating coefficients $g_{j}$ that reproduce the original satellite observation values $B_{i}$ through a linear combination of $b_{ji}$, $g_{j}$ can be considered as the contribution of each radiation source to the satellite observations. In practice, a U-net (Ronneberger et al., 2015) was optimized to output the contribution map $g_{j}(\bm{x})$ from the input satellite images $B_{i}(\bm{x})$. The loss function was defined as follows:

$$\mathcal{L}=\left\|B_{i}(\bm{x})-\sum\nolimits_{j}g_{j}(\bm{x})b_{ji}(\bm{x})\right\|_{1}s.t.\sum\nolimits_{j}g_{j}(\bm{x})=1$$

Here, $\bm{x}$ represents the image coordinates.

The process flow and an example of the results are shown in Figure 3. Here, other information such as outputs from numerical weather predictions and temporal changes in satellite images was referenced to identify the radiation sources for each cluster $j$. By aggregating the contribution of radiation sources corresponding to the surface, such as seas, forests, and snow covers, the clear sky probability can be calculated:

$$P_{\text{clr}}(\bm{x})=\sum\nolimits_{j\in\text{Surface}}g_{j}(\bm{x})$$

In an event shown in Figure 4, ST1 site is covered with snow under clear-sky. Although clouds in the vicinity cannot be distinguished from snow covers in visible images, the clear-sky probability calculated by this sub-model shows a value close to 1, confirming the model’s validity.

Figure 4: Visible satellite image (left) and distribution of estimated clear-sky probability (right) for the Hokkaido region, spanning latitudes 41°N to 46°N and longitudes 138°E to 146°E, on December 3, 2018, at 12:00. The yellow arrow points to ST1, the ground observation site.

4 Results and Conclusions

In this paper, we demonstrated the effectiveness of utilizing machine learning and multiband data by comparing the developed model against the standard model across cases of thin clouds and snow covers.

For the thin clouds case, we refer to an event for ST2 on August 15, 2018. The presence of thin clouds in the area around ST2 reduced solar irradiance, but the standard model overestimated the solar irradiance around 10:00 (see Figure 5, left). This overestimation is likely due to surface reflections transmitted through thin clouds being observed in the visible band, leading to an underestimation of cloud thickness. In contrast, the developed model accurately captured the reduction in solar irradiance caused by the thin clouds. For this event, when only the visible band was used in the developed model, an overestimation similar to that of the standard model occurred. This experimental result indicate that by extracting information about the refractive index of ice particles and the vertical structure of upper-level clouds from the near-infrared and infrared band data, the cloud thickness could be accurately estimated.

For the snow case, we refer to an event for ST1 on December 3, 2018, at 12:00, shown in Figure 4. The solar irradiance time series for the day appears in Figure 5 (right). The standard model cannot distinguish snow cover from clouds, leading to an underestimation of solar irradiance. However, the developed model accurately estimates solar irradiance. This accuracy is achieved by correctly identifying snow cover based on the clear-sky probability. Snow, composed of ice particles and accumulated on the ground, can be distinguished by appropriately combining near-infrared and infrared bands, although the relationship is inherently complex. The standard model, which employs a rule-based classification method utilizing several bands, often misidentifies snow as clouds due to its simplistic approach. In contrast, the developed model uses six bands and leverages machine learning to accurately represent this complexity, thus achieving precise identification of snow and clouds.

Figure 5: Comparison of solar irradiance time series. Black dotted, red, and blue lines indicate ground observations, the standard model, and the developed model, respectively. Left: ST2, August 15, 2018, at 10:00. Right: ST1, December 3, 2018.

To quantitatively compare the standard and developed models, we calculated the Root Mean Square Error (RMSE) for the year 2018. The calculations used one-minute average values for ground observations and instantaneous values estimated from satellite observation data. Table 1 shows the RMSE for each model by validation site. The developed model shows an improvement trend at all sites compared to the standard model, with more than a 10% improvement effect, particularly when excluding ST5, where localized clouds frequently occur that are not captured by satellites.

These results confirm that the developed model outperforms the standard model. The main factors for improvement include enhanced representation of thin clouds and convective clouds (not shown in this paper) through the use of multiband data and machine learning, and the introduction of radiative transfer models and clear-sky probability, which have improved the generalization performance for various surface conditions including snow covers. Additionally, the use of CNN has contributed by taking into account the spatial structure of clouds and improving the representation of scattered irradiance.

Table 1: Comparison of RMSE$[\text{W/m}^{2}]$ between ths standard model and the developed model by validation site.

Model	ST1	ST2	ST3	ST4	ST5
Standard	106.9	98.2	101.5	111.3	124.8
Developed	86.6	86.1	86.0	94.2	118.6

This research provides a versatile framework for constructing solar irradiance estimation models applicable to any region, based on limited ground observation data, not only in Japan but worldwide. This contributes to the creation of foundational data for meteorological and climatic analyses.

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning and Earth Science. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.