RTGS: Enabling Real-Time Gaussian Splatting on Mobile Devices Using Efficiency-Guided Pruning and Foveated Rendering

PBNR is a class of neural rendering techniques, exemplified by the 3D Gaussian Splatting (3DGS) algorithm (Kerbl2023GaussianSplatting, ) and its descendants (fan2023lightgaussian, ; lee2024compact, ; fang2024mini, ; niemeyer2024radsplat, ; girish2023eagles, ). Compared to previous neural rendering techniques, a.k.a., the NeRF-family algorithms (mildenhall2021nerf, ; barron2022mip, ; muller2022instant, ; chen2022tensorf, ; sun2022direct, ), PBNR is fundamentally more efficient (e.g., usually over 1,000 times faster), because it parameterizes the scene with discrete points (rather than voxels) to avoid redundant computations and renders via a lightweight rasterization-based process called splatting (rusinkiewicz2000qsplat, ; zwicker2001surface, ; ren2002object, ) rather than the heavy MLP inference.

The goal of our pruning is to judiciously reduce tile-ellipse intersections without affecting the visual quality. The key to our pruning is a metric that we call Computational Efficiency (CE), which intuitively describes how much contribution a point makes to pixel values per unit cost of compute. Intuitively, we would like to prioritize pruning points with low CEs, as they consume a lot of computation without making much contribution to pixel values. For every point $i$ in a dense model, its CE is defined as:

$\text{Val}_{i}$, contribution of a point $i$ to pixel values, is defined as the number of pixels that are “dominated” by that point. A pixel is dominated by a point if and only if that point, among all the points, has the highest numerical contribution to the pixel value during rasterization (Sec. 2.1). The numerical contribution of a point $i$ is quantified by $T_{i}\alpha_{i}$ in Eqn. 1a.

$\text{Comp}_{i}$, the compute cost of a point $i$, which is ignored in all existing pruning methods, is quantified by the number of tiles that intersect and use (the ellipse of) that point, which directly affects the rendering speed as established above.

In actual rendering, a point will be used in different frames based on the camera pose. Thus, a point’s CE is frame-specific; in extreme cases, a point could be outside the camera’s viewing frustum and thus makes no contribution to the image. We empirically find that the final CE of a point is adequately measured by the maximum CE across all poses (as opposed to the average, which is susceptible to dataset bias) in the training set.

With this metric, during pruning we sort all the points by their CEs and remove a certain portion of points with the lowest CEs. How many points to remove must be done in conjunction with controlling the quality of the pruned model, which we will discuss in Sec. 3.4.

Orthogonal to pruning points, another way to reduce tile-ellipse intersections is to reduce the ellipse size/scale, which we call “scale decay.” In particular, we want to focus on scaling ellipses that are both large and are used by a lot of tiles in rendering. To guide scale decay, we propose a metric called Weighted Scale (WS) that weighs the point sizes with how often they are used in rendering:

The WS metric is a general metric characterizing point/ellipse scales in PBNR. We empirically find that it is particularly effective when integrated into the training process as an additional term to the loss function $\mathcal{L}$, which ordinarily is concerned only with the rendering quality ($\mathcal{L}_{\text{quality}}$):

Pruning and scale decay are conceptually orthogonal, but, importantly, scaling an ellipse’s size also changes its CE. Thus, scale decay must be done in conjunction with pruning. Fig. 6 illustrates the general procedure.

Given a dense model, we first compute the CE for all the points, and repetitively prune a small percentage ($R=10\%$ in our implementation) of the points with the lowest CEs until the quality loss ($\mathcal{L}_{\text{quality}}$ in Eqn. 6) is above a prescribed threshold. We then train the pruned model again to regain the quality, but using the composite loss $\mathcal{L}$ in Eqn. 6 in order to apply scale decay. The re-training continues until $\mathcal{L}_{\text{quality}}$ is once again below the threshold, at which point we apply intersection-aware pruning again. We iteratively apply pruning and scale decay in such a way until a certain number of iterations is reached.

Note that $\mathcal{L}_{\text{quality}}$ is usually PSNR or SSIM but can be any other quality metric of interest. In the next section we will show how we can use a human vision-inspired quality metric to account for the eccentricity dependence of visual quality.

Our iterative procedure has two advantages. First, it combines pruning and scale decay. Second, it does not require quality-specific hyper-parameter tuning to achieve a specific visual quality: monitoring and controlling for $\mathcal{L}_{quality}$ automatically yield a model at a given quality.

We accelerate rendering by relaxing the rendering quality at the visual periphery, leveraging the low peripheral visual acuity in HVS. We illustrate the idea in Fig. 7, panel .

To address both the performance and storage overhead, we propose an efficient data representation that allows models at different quality levels to share computation and parameters. The key idea is that points used to train and render a lower quality level are strictly a subset of the points used by a higher quality level. Panel in Fig. 7 illustrates how the original points in Panel are organized after subsetting. Level 1 ($L_{1}$) model is trained with the most points and thus would offer the highest quality, and Level 4 ($L_{4}$) model has the fewest points and lowest quality.

Subsetting mitigates both the performance and storage overhead, because the total number of points across all $N$ models, $P_{total}$, is the same as that of the highest-quality model, $P_{1}$, rather than the sum of all $N$ models. That is, $P_{total}=\text{max}_{i=1}^{N}{P_{i}}=P_{1}<\sum_{i=1}^{N}{P_{i}}$. As a result, there is no storage overhead. The compute overhead is small too, since the Projection and Filtering stages are executed only once, rather than once for each of the $N$ models.

Under subsetting, each point is simultaneously used in models $[L_{1},\cdots,L_{m}]$, where $m$ is the highest level beyond which the point is not used and is called the quality bound of the point. For instance in Fig. 7, $m=3$ for Points 4. During the Projection stage, each point is projected to a tile, which has a specific eccentricity and thus a corresponding quality level $t$. If $t>m$, the point does not participate in the rest of rendering. This is the Filtering stage in Panel .

The discussion so far has focused on performance, but equally important to FR is the visual quality: how much weaker can higher-level models be while maintaining subjectively good visual quality across quality levels/eccentricies?

To answer this question, we turn to the HVSQ metric discussed in Sec. 2.2. The HVSQ metric quantifies the subjective visual quality between a reference image (e.g., rendered from a dense PBNR model) and an altered image (e.g., rendered from a pruned PBNR model), accounting for the eccentricity-dependent visual acuity of HVS. Conveniently, while the vanilla HVSQ metric in Eqn. 2 is applied to an entire image, it can be easily adapted to a selected region — by simply iterating over the spatial poolings (pixels) in the selected region rather than over the entire image.

That way, each quality region has a unique HVSQ measure, and our goal is to ensure the HVSQs across all quality levels are similar to the HVSQ of the baseline model. To that end, we first train the highest-quality, $L_{1}$ model, which itself can be pruned and scale-decayed from a dense model. We then prune a $L_{1}$ model to obtain a $L_{2}$ model, which is pruned down to obtain a $L_{3}$ model; this continues until the desired level is achieved. The way to obtain a $L_{i+1}$ model follows the exact procedure as laid out in Sec. 3.4 (i.e., iteratively apply pruning and re-training to a $L_{i}$ model while controlling for the quality loss $\mathcal{L}_{quality}$) — with two key differences.

First, instead of using the usual PSNR/SSIM metrics, we use HVSQ as $\mathcal{L}_{quality}$ in Eqn. 6. In particular, when obtaining a $L_{i}$ model we use the HVSQ corresponding to level $i$. We control for $\mathcal{L}_{quality}$ so that the HVSQ at all quality levels is the same as that of $L_{1}$ such that the human visual quality is consistent across the entire visual field. Second, during iterative re-training we do not apply scale decay, because an ellipse scale is not part of the multi-versioned parameters.

Fig. 10 shows the average number of participants who prefer the two methods for each video. A tie would be 4-vs-4, since each video is watched eight times by each user. We find that users either have no preference or prefer our method over Mini-Splatting-D. The results are statistically significant through a binomial test with $p$ ¡ 0.01; the null hypothesis is “users prefer Mini-Splatting-D more than 50% of the time”.

It might initially look surprising that we have equal or better subjective quality than Mini-Splatting-D, a dense model from which we prune and build our FR model. Further inspection and interviewing participants show two reasons. First, our HVS-aware fine-tuning (Sec. 5) better aligns the statistics of human-sensitive features with the ground truth. Second, some points in the dense model are trained with inconsistent information across camera poses, leading to incorrect luminance changes over time; pruning those points helps alleviate this inconsistency.

We now show the performance results on the mobile Volta GPU on Nvidia Jetson AGX Xavier (xaviersoc, ), a representative mobile device for use-cases such as VR. Fig. 10 shows the results. We execute each model five times for each camera pose in each scene in all the datasets, and report the average FPS.

To put the performance results in context, we also compare the PSNR metric of our variants with the baselines. PSNR is a good objective quality measure of the region under the user’s gaze and is universally reported in prior work. Using an objective metric also allows us to scale up the study to more traces. The SSIM results are similar and omitted.

Our three variants Pareto-dominate the baselines. Our slowest variant RTGS-H is 1.9$\times$ faster than the fastest baseline while being 0.1 dB higher than the best-quality baseline. The fastest variant RTGS-L is 7.9$\times$ faster than 3DGS, and can be up to 19.8$\times$ on the largest bicycle trace.

RTGS also out-performs the two FR baselines. Tbl. 1 compares the FPS (on the mobile Volta GPU), storage requirement, and the HVSQ metric across different quality regions/layers. The results are averaged across all the datasets.

SMFR has been seen as a variant of our FR with strict subsetting (no multi-versioning). Thus, it is the fastest, but has excessively low visual quality, because it simply sub-samples pre-trained points to render low-quality regions. Its HVSQ in $L_{4}$ is over 10$\times$ worse than the other two methods. Users confirm that subjectively this gives the worst quality.

Our method selectively multi-versioned four trainable parameters out of about 60 (Sec. 4.2 and in Fig. 7), so the additional storage requirement is small (about 6%). Note that RTGS-H already reduces the dense model size to 16% as shown in Sec. 5.

MMFR can be seen as a variant of our FR that multi-versions all parameters. It is thus the slowest of the three — its FPS is way below a 90 FPS real-time requirement, and has the largest storage requirement. This is due to the compute and storage overhead discussed in Sec. 4.1. Its HVSQ metrics in higher levels (lower-quality regions) are better than that of ours. Given that our method is already subjectively no-worse than even a dense model (Sec. 6.1), this suggests that MMFR unnecessarily optimizes for details that are imperceptible to users, which we confirm with users.