Minimalist and High-Quality Panoramic Imaging with PSF-aware Transformers

Recent research interest in panoramic images is booming for immersive visual experiences [11, 39]. Semantic segmentation [1, 40], depth estimation [2, 3], and visual Simultaneous Localization and Mapping (SLAM) [41, 42] are widely explored on panoramic images for a holistic understanding of the surrounding scene. To this intent, high-quality panoramic images are urgently required for robust performance. A considerate amount of work is conducted to improve the image quality of panoramic images, such as super-resolution [5, 6, 7] and distortion correction [8, 9, 10].

However, the above image processing of panoramic images is based on the aberration-free images captured by the conventional panoramic lens, where multiple sets of lenses with complex surface types [11, 12, 13] are applied for high-quality imaging. This work focuses on capturing panoramic images with a Minimalist Optical System (MOS) composed of much fewer lenses for volume-limited applications, where we process the aberration images via computational methods.

The aberration-induced image blur is inevitable for MOS due to insufficient lens groups for aberration correction. Recently, computational imaging methods for MOS appear as a preferred solution to this issue, where optical designs with few necessary optical components are equipped with image recovery pipelines for both minimalist and aberration-free imaging [16, 17, 43]. Some research works [18, 44, 45] even design end-to-end deep learning frameworks for joint optimization of MOS and post-processing networks to achieve the best match between MOS and recovery models to further improve the final imaging performance.

However, computational imaging for minimalist panoramic systems is scarcely explored. In a preliminary study, Jiang et al. [28] propose an Annular Computational Imaging (ACI) framework to break the optical limit of minimalist Panoramic Annular Lens (PAL), where the image processing is conducted on unfolded PAL images. To further develop a general framework for minimalist and high-quality panoramic imaging, this work fills the gap in high-quality PAL image datasets and designs the Panoramic Computational Imaging Engine (PCIE) directly on annular PAL images, in terms of both optical design and aberration-images recovery.

The aberration-induced blur is always spatially-variant, i.e.  Linear Shift Variant (LSV), due to the uneven thicknesses of the lenses. Several efforts have been made for the LSV system, spanning path-wise restoration [46, 47], experimental PSFs calibration and non-blind deconvolution [43, 48], and low-rank decomposition [49, 50], based on the degradation model of aberration-images [51, 52, 53].

Recent works tend to adopt the data-driven learning-based image restoration networks [54, 38]. These methods typically use a U-shaped network with an encoder-decoder structure [16, 55, 56] to achieve more efficient and robust recovery results, which can also be easily inserted into an end-to-end framework for joint optimization. To break the bottleneck of data-driven methods under scarce data, the PSF information is explored to design physical-informed networks, where model-based methods are characterized by Convolutional Neural Networks (CNNs) for learning ill-posed terms [14, 57]. Explorations have also been made in [27, 35, 36, 58] to apply transformers for solving the inverse problem, leveraging its strong long-range modeling capabilities.

Differently, we make a pioneering effort and investigate the potential of transformer-based SR models in aberration correction rather than conventional Deblur models. Then, the PSFs are transformed into an additional modality of the aberration image, based on which we design PSF-aware mechanisms for achieving better results. The proposed PSF-aware Aberration-image Recovery Transformer (PART) is a successful attempt to engage PSF information in the representation learning stage of SR models for recovering aberration images.

The overview of PCIE is shown in Fig. 3. It provides a powerful framework for minimalist and high-quality panoramic imaging, where optical design (detailed in Sec. III) and learning-based model (presented in Sec. IV) are intertwined to achieve impressive imaging results.

To boost scene understanding with larger FoV, panoramic optical systems are emerging, including fisheye optical systems, refractive panoramic systems, panoramic annular optical systems, etc. [11]. In most modern designs of panoramic lenses [12, 21, 22], a panoramic head is applied for collecting incident rays of 360° FoV, while a set of relay lenses is designed to bend the rays and correct the aberrations. Based on the structure, we propose the Minimalist Panoramic Imaging Prototype (MPIP), including an essential panoramic head and a simple relay lens group, as shown in Fig. 1(a).

Specifically, we adopt a more compact and efficient solution, i.e. Panoramic Annular Lens (PAL) [23, 24], in MPIP samples, where a catadioptric PAL head is equipped for 360° annular imaging. For minimalist design and convenient manufacture, spherical lenses are applied in the relay lens group and the number is reduced to fewer than $3$. To illustrate the imaging results of different relay lens groups, we define MPIP-P1 and MPIP-P2 in Fig. 4(a), whose relay lens group is composed of two lenses and a single lens, respectively.

The lack of enough lenses for aberration correction makes the imaging point spread from an ideal point, inducing spatially-variant PSFs with large kernel sizes, as shown in Fig. 4(b). The average geometric spot radius of MPIP-P1 is $13.78{\mu}m$, whereas that of MPIP-P2 is $46.26{\mu}m$. As a result, the captured images of MPIP suffer from spatially-variant aberration-induced blur, especially for MPIP-P2 with fewer lenses, as shown in Fig. 4(c).

In addition to the uncorrected optical aberrations, the limited image plane size due to the small aperture of MPIP presents the issue of image resolution. To fit the small image plane of the MPIP, an image sensor with a smaller pixel size can be applied to maintain high resolution, but it makes the system more sensitive to aberration-induced blur. As shown in Fig 5(a), the diffused optical spot of fixed physical size affects more pixels for the sensor with smaller pixel sizes and higher resolution. The opposite solution with large pixel sizes is less sensitive to the diffused spot, but the reduced image resolution also brings degradation to the images, which is especially harmful to panoramic images with large FoV [6].

To address this dilemma, we propose two pipelines for solving the contradictory problems, as shown in Fig. 5(b), where a learning-based model is applied to process different image recovery tasks. For image sensors with smaller pixel sizes, we define the Aberration Correction (AC) task, where the goal is to recover a clear image ${x}_{hq}\in\mathbb{R}^{H{\times}W{\times}3}$ from a high-resolution input aberration-image ${x}_{ab}\in\mathbb{R}^{H{\times}W{\times}3}$. Whereas for image sensor with larger pixel size, the Super-Resolution and Aberration Correction (SR$\&$AC) task is raised to recover a high-resolution aberration-free image ${x}_{hq}\in\mathbb{R}^{H{\times}W{\times}3}$ from a low-resolution input aberration-image ${x}_{{lq}}{\in}\mathbb{R}^{\frac{{H}}{{s}}\times\frac{{W}}{{s}}\times 3}$, where $s$ is the scale factor of SR.

The lack of high-quality image datasets for PAL comes as a bottleneck to the above tasks. A piece of previous work for CI of PAL, ACI [28], unfolds the annular PAL images into perspective ones to utilize the publicly available datasets, i.e. DIV2K [59]. However, the asymmetrical interpolation during unfolding induces extra image degradation, which further complicates the image degradation factors for MPIP. In addition, the annular image is more appealing for the simulation of aberrations in the original image plane and necessary for some vision tasks like PAL-based SLAM [41, 42]. In the case of benchmarks for processing panoramic images, e.g., the ODI-SR dataset [60] and the SUN360 panorama dataset [61], which are taken via fisheye cameras, the imaging process is also quite different from that of PAL [11]. These concerns raise an urgent request for high-quality panoramic annular image datasets.

To this intent, we propose PALHQ, a dataset of high-quality PAL images, to facilitate network training and evaluation of PAL-based low-level vision tasks. A well-designed PAL of $11$ lenses and a Sony $\alpha$6600 camera are applied to capture high-resolution PAL images with negligible primary aberrations. PALHQ contains $550$ clear PAL images with a resolution of $3152\times 3152$, covering rich and varied scenes of indoor, natural, urban, campus, and scenic spots. We divide PALHQ into $500$ images for the training set and $50$ images for the validation set (refer to the appendix for sample images of PALHQ). In PCIE, we benchmark both AC and SR$\&$AC on PALHQ, where the corresponding aberration images are generated by the imaging simulation model depicted below. Furthermore, PALHQ can be also transmitted to unfolded panoramas via equirectangular projection (ERP), which can support various panoramic image processing applications.

To quantitatively benchmark the raised two tasks and enable supervised training of learning-based models, paired aberration images and clear images are required. Following previous super-resolution works [62, 63] and CI works [28, 56], we construct an imaging simulation model to generate synthetic aberration-images in batches.

The wave-based simulation pipeline with random perturbation in [28] is adopted to generate multiple aberration distributions directly on clear annular PAL images. Specifically, the clear raw image $R$ is modulated by an optical system and then processed by ISP $\Gamma(\cdot)$ to produce the final imaging result $A$:

where ${r_{\lambda}}$ is the wave response of the sensor. The noise $N$ and the sampling process $(\cdot)\downarrow$ of the image sensor are also included in the model. We divide the image into patches for patch-wise convolution with PSFs $K_{\theta}(x,y,\lambda)$ under different FoV $\theta$. Different from [28], the division of FoV is centrosymmetric for annular images as is shown in Fig. 4(b). Through scalar diffraction integral [64], $K_{\theta}(x,y,\lambda)$ is calculated based on the wavefront $\Phi_{\theta}(x^{\prime},y^{\prime},\lambda)$ on exit pupil plane, which is described by Zernike polynomials [65] mathematically:

where $C(\theta,\lambda)$ denotes Zernike coefficients under FoV $\theta$ and wavelength $\lambda$ and $Z$ refers to polynomials of the coordinate $(x^{\prime},y^{\prime})$ on exit pupil. The combination of different $m$ and $n$ represents different orders. Finally, we apply the random disturbance strategy in [28] to fine-tune the ideal $C(\theta,\lambda)$ from the $Zemax$ software, generating synthetic aberration images with diverse aberration distributions.

For non-blind optimization-based recovery methods in aberration correction, e.g. Wiener filter [51], PSFs $K$ of the system are exploited to predict the clear image $x_{hq}$ from the aberration-image $x_{ab}$ by deconvolution. However, this method often fails when the PSFs deviate from the design stage during manufacture and require time-consuming strategies [43, 48] for processing complex spatially-variant blur. Data-driven learning-based models [27, 36], which can be plugged directly into existing end-to-end frameworks of lens design [18, 19, 20], have demonstrated more powerful abilities in image recovery, but may hit a bottleneck when the training data is scarce.

This motivates us to break the bottleneck by utilizing PSFs of the optical system in a learning-based model. The PSFs of $n$ sampled FoVs of the applied MIIP can be calculated based on the wavefront as depicted in Eq. (2):

The $\mathcal{S}(\cdot)$ denotes scalar diffraction integral (refer to [64] for more details). Previous methods tend to use the kernel size of $K_{i}$ to guide the network [28] or refine the ill-posed term in deconvolution through a learning-based model [57]. Although these attempts can improve the recovery benefiting from the applications of $K_{i}$, the spatial intensity distribution of PSFs is not fully exploited to guide the deep feature extraction of the general image recovery paradigm.

To this intent, we propose a PSF representation method to produce a PSF map, containing both intensity and size distributions of PSF kernels, which is aligned with the image feature map. We first map the spatial PSFs $K_{i}\in\mathbb{R}^{k_{i}{\times}{k_{i}}{\times}3}$ into the image feature shape ${H{\times}{W}{\times}{C^{\prime}}}$, where $k_{i}$ is the kernel size of PSF under the $i_{th}$ FoV and $C^{\prime}$ denotes the channels of mapped PSFs, serving as an additional modality aligned with the aberration-image. As previously shown in [66], the spatial-to-channel arrangement helps transform spatially-variant kernels into a feature map. Similarly, to produce a PSF feature map, the spatial PSFs $K_{i}$ are arranged into the channel dimension. Concretely, for a pixel at $(x,y)$ of the image, we first calculate the vector $\overrightarrow{p}$ from the image center $(0,0)$ to $(x,y)$, and define the vertical unit vector$\overrightarrow{a}(1,0)$. The PSF of the corresponding FoV is located by $\left|\overrightarrow{p}\right|/\max\left(\left|{\overrightarrow{p}}\right|\right)$, and rotated by the angle $\arccos\left(\frac{\vec{p}\cdot\vec{a}}{\left|\vec{p}\right|\cdot\left|\vec{a}\right|}\right)$, producing the PSF $K_{x,y}$ of the pixel. For memory-friendly computation, we pad all $K_{x,y}$ into unified size of $\max\limits_{i}k_{i}$ and compress them into ${k^{\prime}\times{k^{\prime}}\times 1}$ via adaptive average pooling:

where the choice of compressed size $k^{\prime}$ is ablated in Sec. V-F. The $\hat{K_{x,y}}$ is then reshaped into ${1\times 1\times(k^{\prime 2})}$ and inserted into each pixel to produce the PSF feature map $x_{int}\in\mathbb{R}^{H\times{W}\times(k^{\prime 2})}$. In addition, considering the lost PSF size information during compressing, we also generate the size distribution map $x_{s}\in\mathbb{R}^{H\times{W}\times 3}$ of RGB channels, where the value of each pixel represents the kernel size. Finally, the PSF map $x_{psf}$ is produced via Eq. (5):

and the visualized PSF map is shown in the appendix. PSF map is an aligned modality of the aberration image characterizing the image degradation over FoVs, based on which we design a PSF-aware transformer, as described in the next subsection.

CNN layers have shown impressive abilities of local feature extraction, but are restricted to the fixed spatially-invariant kernels. However, the mathematical imaging model in Eq. (1) reveals that the aberration-induced blur is only generated by convolution with spatially-variant PSF kernels, whose inverse solution cannot be modeled by the fixed convolution kernels [55, 56].

To extract adaptive image features with the guidance of spatially-variant PSF kernels, we propose the PSF-aware Feature Modulator (PFM), as shown in the lower left of Fig. 6. PFM builds on the idea of filter adaptive convolution [66, 67], where a kernel map of ${H{\times}{W}{\times}(Ck^{2})}$ is predicted from feature map of ${H{\times}{W}{\times}{C}}$. Differently, in PFM, the kernel map $x_{kernel}$ is predicated on the features of PSF map $x_{psf}$, which has been compressed into a similar form as $x_{kernel}$. We first apply $E_{psf}$ as a $3\times{3}$ convolution layer to extract features of PSF map $x_{psf}$, as depicted in Eq. (6):

where $x^{\prime}_{psf}\in\mathbb{R}^{H\times{W}\times{C}}$ is the extracted PSF feature map. Then, a lightweight kernel predictor composed of several convolution layers is proposed to output the kernel map $x_{kernel}\in\mathbb{R}^{H{\times}{W}{\times}(Ck^{2})}$ based on $x^{\prime}_{psf}$, as in Eq. (7):

To reduce the memory cost and inference latency of kernel prediction, the predictor $P$ computes the kernel map on the downsampled features (by $4{\times}4$ average pooling). Benefiting from that the PSF map shares a similar form with the kernel map, we further simplify the $P$ where only one Max Pooling layer and one residual block of $1{\times}1$ convolution layers are applied, to predict the kernel map $x^{\prime}_{kernel}\in\mathbb{R}^{\frac{H}{8}{\times}\frac{W}{8}{\times}(Ck^{2})}$ in a smaller resolution. The final kernel map $x_{kernel}$ is then obtained by ${\times}8$ upsampling via bilinear interpolation. Finally, we reshape the $x_{kernel}$ into a list of per-pixel kernels of $k{\times}{k}{\times}{C}$ and apply them to the corresponding pixels of image feature $x^{\prime}_{img}$. PFM attempts to model the inverse process of the aberration-induced blur, i.e. deconvolution, which promotes the dynamic feature extraction of the aberration-image.

We put forward the PSF-aware Mix-Attention Block (PMAB) as the basic unit of our PSF-aware transformer, to process aberration images assisted with the PSF map, as shown at the middle bottom of Fig. 6. The Window-based Multi-head Self-Attention (W-MSA) of the Swin-T block [27] is first adopted to be the baseline attention mechanism for modeling spatially-variant convolution and long-range dependency, which is also important for stable training of the network.

To address the drawback of fixed window size in vanilla W-MSA, we further propose the PSF-aware Varied-Size Attention (P-VSA), shown on the lower right of Fig. 6. The vanilla varied-size attention [68] in high-level tasks predicts the sizes and locations of the windows from input features for computing self-attention on dynamic windows. Meanwhile, the kernel sizes of PSFs in different FoV regions reveal the severity of aberration-induced blur, which is relevant to the calculation of window-based self-attention. To better adaptively modulate the windows according to the PSF kernels, we make use of the PSF map features $x^{\prime}_{psf}$ to generate PSF-aware varied-size windows. Concretely, the scale $S$ and offset $O$ of the varied-size windows are predicated on $x^{\prime}_{psf}$ by the Window Transform block, which is composed of a $1\times{1}$ convolution layer. Then, we sample the projected key and value tokens $K,V$ of image features $x^{\prime}_{img}$ based on the transformed window to obtain $K_{vs},V_{vs}$. The cross-attention is computed between query $Q$ of the default window and $K_{vs},V_{vs}$. The operation of P-VSA can be expressed as:

where $\rm{WinPar}$ denotes the window partition operation of Swin-T and $d$ is the dimension of tokens.

Additionally, some works [30, 32] apply channel-attention-based convolution blocks in parallel with the self-attention to enhance the representation ability of the network. We insert the proposed PFM to PMAB in the same parallel way, where the filter adaptive convolution mechanism can better model the spatially-variant blur compared to channel-attention-based convolution.

Finally, PMAB is the mixing of W-MSA and P-VSA with a parallel $1{\times}1$ PFM. For the self-attention module, the image feature map $x^{\prime}_{img}$ is equally split along the channel dimension and processed by parallel W-MSA and P-VSA, then concatenated along the channel dimension again. The modulated feature map by parallel PFM is multiplied by a constant $\alpha$, to be added to the result of self-attention and the original feature map as common practice for stable training [32]. The whole process of PMAB is computed as:

where ${\rm{FFN}}$ is a common Feed Forward Network composed of a LayerNorm and a Multi-Layer Perceptron (MLP) layer.

Most previous networks [16, 55] for aberration corrections often utilize the architecture of image deblurring methods, i.e. U-Net. However, for MPIP images with a high resolution (e.g., $3K$), the U-Net methods incur unacceptable computational costs due to the large image sizes at shallow layers. Differently, we look into the tasks from the perspective of image super-resolution, which processes image features with low resolution and reconstructs the high-quality image via an upsampling module. The aberration-induced blur brings aliasing between pixels and losses of image details, which can also be interpreted as “low resolution”. Thereby, as shown in Fig. 6, the PSF-aware Aberration-image Recovery Transformer (PART) is set up based on the structure of SwinIR [27] and our proposed PSF-aware mechanisms.

A Task-Processing module is first applied to transform the input image and PSF map to a small spatial size, where pixel-unshuffle [63] is leveraged for AC and no operation is entailed for SR$\&$AC. The PSF map is also concatenated with the aberration image as the input of the network. More precisely, PART contains three parts. (1) A feature extraction layer converts the input to image feature maps via a $3{\times}3$ convolution. (2) The representation learning stage applies stacks of transformer-based blocks ending with a convolution layer to enrich the learned degradation information of aberration-induced blur progressively. We design the PSF-aware Residual Transformer Block (PRTB) with several PMAB layers and a convolution layer. The PFM is inserted into each PRTB to modulate the learned features and model the inverse process of the aberration-induced blur. We also implement PFM at the beginning and end of the representation learning stage, for adaptive feature extraction and feature fusion based on PSF information. (3) The image reconstruction module further fuses the extracted deep features and recovers a high-quality image with higher resolution. With PART, we can recover a high-quality aberration-free image $x_{hq}$ from either a high-resolution aberration image $x_{ab}$ or a low-resolution one $x_{lq}$, providing a general solution to AC and SR$\&$AC:

For real datasets without reference clear image, we employ no-reference metrics, i.e. NIQE and BRISQUE, to evaluate the image quality of MPIP images in terms of natural images. The qualitative visual results are also provided for an intuitive evaluation. However, the NIQE [72] and BRISQUE [73] are built on the statistics of perspective natural images, which are challenging for assessing the MPIP images with the annular distribution of image content. Considering the specific tasks of correcting optical aberrations, we define the Optical-based Image Quality Evaluator (OIQE) for credible evaluation, based on the Modulation Transfer Function (MTF) of the imaging system calculated by a set of testing checkerboard images.

To be specific, we follow Spatial Frequency Response (SFR) [56] testing to calculate MTFs on image patches of “knife-edge” of different FoVs from different testing images. $MTF50$ and $MTFarea$ are used to characterize the MTF curves, where the former is the frequency when the MTF drops $50\%$ and the latter is the area under the MTF curve. We further define $OIQE50$ and $OIQEarea$ as the ratio of the average $MTF50$ and $MTFarea$ of the testing imaging pipeline to those of a well-designed panoramic imaging system. Accordingly, $OIQE$ is defined as:

which measures the gap between the results of PICE and conventional panoramic lenses in terms of MTF. OIQE is only applied in the AC pipeline due to its specific design for evaluating the ability of the model to remove aberration-induced blur.

In addition, with the testing checkerboard images of OIQE, we generate the ground-truth images through edge extraction and re-coloring following [55], so that the PSNR and SSIM can be applied as metrics in this setting.

Finally, we conduct a user study as a subjective evaluation method. The results of the User Study (U.S.) will be presented as the percentage of times that each method’s results were chosen as the best.

The implementation details of the ground-truth generation pipeline and user study are depicted in the Appendix. Based on the above evaluation pipelines and metrics, a comprehensive evaluation of competitive recovery models on real-world datasets will be presented in Section V-C.

All the models are retrained on PALHQ-SynMPIP-P1 and PALHQ-SynMPIP-P2 with their original optimizers, learning rates, and schedulers, where the number of training iterations and the batch size are set the same as PART for a fair comparison. Additionally, we apply task-processing for all the SR models the same as PART.

PART is trained on L1Loss, while other loss functions are explored in Sec. V-D. We train the models with the Adam optimizer with an initial learning rate of $2e{-}4$ and a batch size of $8$ on a single A800 GPU. For data augmentation, random crop, flip, and rotation are applied, where the ground-truth crop size is $256{\times}256$ for AC and $196{\times}196$ for SR$\&$AC to keep an image size of $64$ in the representation learning stage. The number of training iterations is set to $200k$ and the learning rate is halved at $100k$, $160k$, $180k$, and $190k$.

Overall, PART brings better results for PCIE, yielding state-of-the-art performance on two benchmarks, in terms of both fidelity-based metrics (PSNR and SSIM) and perceptual-based metrics (LPIPS and FID). As all the methods produce aberration-free visual results with some lost textures and artifacts, the recovered image of PART shows more visually pleasant details, as shown in Fig. 7(a), in all FoVs.

Further, applied with only one more spherical lens, the PCIE results of MPIP-P1 outperform those of MPIP-P2 by a large margin. For example, the PSNR drops by $3.050dB{\sim}6.101dB$ when MPIP-P2 is equipped. As shown in Fig. 7(b), PCIE with MPIP-P2 delivers moderate clear aberration-free images. Yet, suffering from severe aberrations, its detailed textures are heavily corrupted, especially for large FoVs. In this sense, MPIP-P1 is a superior choice for PCIE to achieve minimalist and high-quality panoramic imaging.

Comparing SR$\&$AC (Table II) with AC (Table II), we observe that the loss of spatial resolution in aberration-images causes significant deterioration to the imaging quality of PCIE, e.g., to an amount of ${-}4.258dB{\sim}{-}3.674dB$ in PSNR. The visual quality comparison between the two pipelines is provided in Fig. 8, where the imaging results of AC reveal richer and more realistic details. In this case, AC is a more competitive pipeline for reconstructing high-resolution aberration-free images, where the real sampled pixels of the sensor offer more convincing imaging features than super-resolved ones despite more aberration-induced blur.

As shown in Table III, PCIE with representative models makes significant contributions to the removal of the aberration-induced blur of real-world MPIP images. To be specific, OIQE improves from $55.22\%$ to $58.28\%{\sim}77.87\%$, and PSNR/SSM improves from $16.215dB/0.7995$ to $18.193dB/0.8690{\sim}19.841dB/0.8943$. The results on NIQE and BRISQUE reveal a large variance, which is attributed to that these metrics are designed for perspective natural images rather than annular MPIP images. For a comprehensive and intuitive evaluation, we rank each method on each metric and provide the average rank (A.R.). In the real-world case, PART outperforms other models, achieving the best OIQE ($77.87\%$), and the best A.R. (2.2). The subjective evaluation of the User Study (U.S.) also illustrates that PART delivers more visual-pleasant panoramic images, which has far superior selection rates. The visual results of PCIE on real-world scenes are provided in Fig. 9. PCIE enables most methods to deliver high-quality panoramic images with few aberrations and high resolution, where PART sets the state of the art in terms of higher contrast, sharper edges, and fewer artifacts. Additionally, consistent with experiments on synthetic data, the recovered images of the SR$\&$AC pipeline reveal perceptually unpleasant artifacts.

To generate richer details for recovered images, we investigate GAN-based training strategies on classical models RRDB, SwinIR, and our PART. Following [76], the GAN-based loss functions in ESRGAN [29] and Local Discriminative Learning (LDL) [76] are adopted, where the former is a classical GAN-based framework and the latter is an improved strategy to remove artifacts. We take models trained with L1Loss, i.e. PSNR-oriented models, as pre-training generators, then apply GAN and LDL loss functions to enable these networks to generate more textures respectively. As shown in Table IV, on synthetic data, both GAN and LDL lead to a decrease in recovery accuracy (PSNR and SSIM), while bringing great gains under the perceptual quality metrics. LDL is a more competitive strategy that outperforms GAN with higher fidelity and fewer visual artifacts, especially with PART.

Regarding real-world data, we present the OIQE and qualitative results in Fig. 10. GAN-based training further contributes to the removal of the aberration-induced blur, achieving better OIQE with higher image contrast. Aside from this, GAN-based models deliver more realistic imaging results with richer textures, which also bring some perceptually unpleasant artifacts and fake details despite being trained with LDL. We point out that the GAN-based strategies offer the potential for learning a more realistic high-quality MPIP image. Still, the local statistics in LDL of perspective images may need to be adapted to annular images for better suppression of artifacts.

We have further explored other potential generative models, e.g., the diffusion model [77, 78]. Please refer to the Appendix for more results.

The collected PALHQ demonstrates an impressive ability to train the model for recovering both synthetic and real-world MPIP images in previous experiments. In this section, we explore whether PALHQ is necessary for PCIE. As an alternative to PALHQ, we simulate the aberrations of MPIP-P1 directly on the publicly available perspective image dataset, i.e. Flickr2K [79], creating PanoFlickr2K for training.

We compare representative models trained on PanoFlickr2K and PALHQ on both synthetic and real-world benchmarks in Table V and Fig. 11. It becomes clear that PALHQ contributes significantly to high-quality panoramic imaging, where the numerical results in all metrics are improved by a large margin and the visual results are more perceptually pleasant with sharper edges, fewer artifacts, and fewer noises.

We conduct ablation studies to investigate how PSF-aware mechanisms contribute to high-quality MPIP image reconstruction. In all cases, the experiments are implemented with the AC pipeline on PALHQ-SynMPIP-P1, evaluated by PSNR and SSIM, and set up on the baseline model SwinIR.

The extensive experiments illustrate the critical points in the proposed PCIE for achieving minimalist and high-quality panoramic imaging. We summarize the following primary findings of our experiments:

• •

  The proposed PCIE presents impressive high-quality imaging results, where the MPIP-P1 and AC pipeline are superior choices for delivering aberration-free panoramic images with much more realistic details.

• •

  In PCIE, we find that window-attention-based models reveal better results. Furthermore, PSF-aware mechanisms are effective for improving the performance of SR models, where the proposed PSF-aware transformer, i.e. PART, sets state of the art.

• •

  The PSF representation plays a significant role in PSF-aware mechanisms, facilitating effective learning of the inverse process of the aberration-induced blur.

• •

  Regarding the training strategies, GAN-based methods contribute to more realistic recovered images, but with some visually unpleasant artifacts and fake details. The generative model appears to be more competitive in PCIE if a good balance is struck when generating rich details and suppressing artifacts.

• •

  Comparing with the adaptation of perspective images, the collected high-quality panoramic annular images dataset, i.e. PALHQ, brings considerable improvements. PALHQ serves as the cornerstone of our PCIE for training a robust model to process MPIP images.

We hope that the PCIE can bring inspiration from optical design, network architecture, sensor choice, data preparation, and training strategies, for minimalist and high-quality panoramic imaging in mobile and wearable applications.

In this paper, we design PCIE to present a general solution to minimalist and high-quality panoramic imaging. Based on the idea of PAL, the MPIP is proposed for 360° panoramic imaging with less than three lenses. Then, learning-based models, which are trained on synthetic aberration images from simulation, are applied to solve the aberration-induced blur and low resolution of MPIP images. A new dataset PALHQ is collected to fill the gap of high-quality PAL images for low-level vision. We explore utilizing PSF information of the optical system to improve the performance of models and design a PSF-aware transformer PART with PSF-aware mechanisms. The plug-and-play mechanism PFM can enhance modern SR models for removing aberration-induced blur, while PART with PMAB delivers state-of-the-art performance on both synthetic and real-world benchmarks. Extensive experiments are conducted to investigate how to improve PCIE, providing heuristic findings for constructing a computational-imaging-based minimalist panoramic system with impressive imaging quality, in terms of optical design, network architecture, sensor selection, training strategies, and data preparation.

There are still some limitations in PCIE, which call for further investigation into extremely high-quality imaging. First, the PSF-aware mechanisms are designed in a straightforward way, which improves the performance, yet, with extra parameters and computational overhead. More efficient and effective PSF-aware architectures or training strategies are expected to further enhance the performance. Meanwhile, the improvements on CNN-based models are less pronounced compared to those on transformer models. We are interested in the design of learnable PSF representation, PSF-aware dynamic, deformable, and dilated convolution, or PSF-aware varied-shape window attention for better exploration of PSF information. Then, we investigate state-of-the-art GAN-based training strategies, while there is open research space for further suppressing artifacts. Aside from this, the results of PCIE on real-world data are not as good as on synthetic data, where artifacts and fake details exist in some recovered images. The considerable synthetic-to-real gap needs future research on domain adaptation. The image number of PALHQ is also limited due to the difficulties of capturing high-quality PAL images under various scenes. We intend to design a hybrid training approach to take advantage of the large data size of the publicly available perspective datasets while improving the training with PALHQ. Finally, an end-to-end framework for joint optimization of MPIP design and recovery model will be focused on presenting a more general engine of minimalist and high-quality panoramic imaging.