Standardized Data-Parallel Rendering Using ANARI

Sci-vis applications can be classified as either post-hoc or in-situ/in-transfer, where in the latter case the visualization and analysis pipeline is executed as the data is generated. Post-hoc is the alternative processing paradigm to in-situ/in-transfer, where data is saved to permanent storage and then loaded by the sci-vis application to perform visualization and analysis.

At scale, the dominating operation of a traditional sci-vis renderer that uses rasterization, simple shading, etc., is sort-last image compositing; the literature has focused on optimizing this operation through efficient algorithms such as parallel direct send [8, 14] or radix-k [20]. These developments culminated in the IceT image compositing library [24] which has become the de-facto standard for real-time distributed renderers. The distributed rendering back-ends of visualization packages like VisIt [6] or ParaView [2], as well as in-situ visualization frameworks like libsim [39], Catalyst2 [21], or Ascent [22, 18] all internally build on IceT.

The implication of using sort-last is that compositing happens at the end, once each rank has reduced its rendering operation to a single fragment (color, opacity, and depth) per pixel. In reality, that restricts the application in what kind of content it can render in this way; in particular, compositing with a single fragment only works correctly if data is partitioned in a way where each rank’s data is convex. To remove some of these restrictions researchers have looked at various forms of “deep” frame buffers that can store more than one fragment per pixel, which in turn allows for better handling transparency when data is partitioned in non-convex ways. A very early example of this was proposed by Ma [23], more recent ones by Binyahib et al. [4] and Sahistan et al. [30]. The same concept is also used by OSPRay [37], whose data-parallel rendering mode relies on using a distributed frame buffer [35] where ranks can also produce more than one fragment per pixel. One an interesting contradiction becomes obvious when looking at OSPRay: while in non-data parallel OSPRay supports advanced rendering techniques realized with Monte Carlo ray tracing, when rendering in parallel within ParaView or VisIt, OSPRay has to switch to simple ray casting without secondary rays, resulting in similar quality as the typical rasterization renderers used by other sci-vis applications.

For ray tracing, data parallel rendering means that any ray on any rank may at any time need intersection with geometry stored by any other rank. This can be done by using either one of two alternative techniques: fetching the remote data to the rank that traces the ray (typically involving some caching of said data, e.g. [19, 7]), or sending the ray to the rank that has the data (see, e.g., [31, 28, 1, 11, 38, 36], or some combination thereof (e.g., [29, 27, 26]).

Though some of these approaches have been around for decades, this kind of data parallel rendering has only recently seen interest from the sci-vis community, probably because only recently hardware has become capable enough to actually do this. In particular, SpRay [28], Galaxy [1], BriX [38], and RQS [36] are all been shown to achieve interactive performance for non-trivial data.

Data parallel CPU and GPU ray tracers aimed at sci-vis achieve illumination effects, including shadows or ambient occlusion, but also global illumination with diffuse reflection, by queuing and exchanging rays across ranks [1, 38, 36]. Shading effects are computed by light rays “bouncing”, i.e., they change direction upon interaction with surfaces, volumes, or lights. Each time a ray bounces it consequently has to visit scene content in completely different spots, resulting in highly incoherent patterns.

To avoid latency, when exchanging rays across ranks, data parallel ray tracers do that in batches rather than exchanging small groups or even individual rays. A core concept for that is the ray wave-front: instead of computing the recursive ray tracing function per each ray individually, compute kernels are executed per bounce, on all the rays the rank is currently responsible for. After a bounce finished, the ray wave-fronts are synchronized, can optionally be sorted and compacted; this is also the point where a data parallel wave-front ray tracer will synchronize its wave-fronts across ranks using MPI broad-casts or uni-casts. The ranks exchange the rays they are responsible for until a maximum number of bounces was reached, until all wave-fronts contain zero rays; etc. The main difference of state-of-the-art methods these days is how the renderers compute, cull, and assign ray wave-fronts to ranks in-between bounces.

Traditionally, the reasoning when distributing rays was to reduce the overall bandwidth. Optimizing ray tracers account for that by building complex culling accleration structures [1, 38] to minimize the overall number of rays exchanged. Those data structures can be extremely unbalanced, and especially if the data itself is instanced requires to add additional constraints to avoid overflowing memory [40].

An alternative approach, also adopted in this work, by the data parallel ANARI back-end described in Section 4.2, is to distribute the scene data in a very simple way—e.g., round-robin, bin-packing until all GPU memory is used, etc.—and then just generate full wave-fronts on all ranks. The ranks can initially discard some of the rays in their wave-fronts by testing them for visibility against their geometry, but still generally exchange more rays than with an optimized partitioning; wave-fronts are exchanged in a ring buffer or similar pattern. The reason that this works and is not overly costly is that wave-fronts are sent in lock-step, i.e., all the communication happens at the same time and is then limited by the pair of ranks with the highest ray count to exchange. When adding more ranks, the overall ray count stays the same, resulting in more ranks exchanging fewer rays individually. This concept was evaluated by Wald et al. [36] on multi-GPU systems with NV-Link interconnect, but also extends to massively parallel architectures like GPU clusters. One compelling reason for preferring this method over approaches that optimize for bandwidth is its negligible pre-processing time; whereas building an optimized acceleration structure can often take hours and is then impractical for visualization in a prodcution context.

ANARI is a cross-vendor 3D rendering API maintained by the Khronos Group. ANARI connects applications from diverse domains to any 3D rendering engine implementing the ANARI API while still giving implementations a vast degree of freedom of how exactly rendering is done. ANARI has already been adopted by major scientific visualization packages, namely VTK [32], VTK-m [25], ParaView [2], VisIt [6], VMD [17], and OVITO [34]. ANARI is not limited to scientific visualization, and has also been integrated, for example, in Blender [12], and OpenUSD’s Hydra subsystem [9].

A comprehensive introduction to ANARI is out of the scope of this work; instead we briefly summarize what we consider the main components of the API that are relevant for data parallel rendering. For a complete overview we refer the reader to [33].

ANARI’s core design centers around opaque handles to objects representing the various bespoke actors commonly found in rendering an image—surfaces, materials, volumes, cameras, lights, frames etc. These objects are parameterized through generic parameters represented by name/value pairs and are transitioned between states using parameter commit semantics—specifically that committing the object’s parameters indicates that those changes should be visible in the next rendered frame. Alas, frames are rendered asynchronously, where the application triggers a render operation to start, and then is free to synchronize with it to access resulting output buffers.

The foundational object for most API calls is the device, which represents the instance of the rendering engine handling ANARI API calls. After creating a device, applications make all ANARI API calls through this special handle, which provides implementations a point to reconcile any common implementation-wide state and gives applications a clear set of rules for how implementations can be used concurrently. Thus for the rest of this paper, the phrases “ANARI implementation” and “ANARI device” will be used synonymously.

Once the ANARI device was initialized, a valid ANARIFrame object is all that is required for rendering. A frame object has color, depth, and other optional auxiliary memory buffers allocated that it can retire rendered pixels to. A valid frame has a ANARIRenderer object, a ANARICamera, and ANARIWorld assigned to it. The ANARI world defines an immediate mode scene graph. It is a special group node serving as a collection of surfaces, volumes, and light objects. In addition to that it may also contain ANARIInstance objects that can themselves contain group nodes; ANARIWorld is a special group node that can contain instance nodes; i.e., the the scene graph has depth two at most (two-level hierarchy). Buffer objects for input and output are realized as ANARIArray s. Arrrays deviate from the otherwise strict immediate mode model in that they can be mapped and their content altered by the user.

A concept that occurs in distributed rendering is how objects relate to one another between nodes. For some objects, they are only defined and interacted with on the node in which they are created, while for others there must be a global definition to them on every rank. The following describes the different application of object definitions with respect to their global or local definitions.

Most ANARI API calls can be done independently on each rank, but a few will behave as a rank synchronizing operation—in other words, some ANARI API calls will necessarily require all ranks to participate in and will implicitly barrier at that call.

The first and most obvious synchronizing API call is anariRenderFrame, as it is the central place that all object parameter transactions must be completed and where the vast majority of the implementation’s work is done. While the mechanics of rendering a frame is intentionally left as implementation defined, applications must follow this semantic to express that every node is ready for its local ANARIWorld is ready to render.

Similarly, anariGetProperty is a synchronizing API for objects that have a global identity (ANARIFrame and ANARIWorld). This is to ensure that implementations can guarantee global consistency to those objects when properties are queried by the application. However, this constraint is only required when applications pass ANARI_WAIT to anariGetProperty, as ANARI_NO_WAIT would indicate it is permissible for the device to return a previously held value (or none at all) in order to prevent blocking.

Finally, anariRelease is a synchronizing API when called on the ANARIDevice which has no remaining application references. This permits implementations to rely on the device being destroyed in lock-step and guarantee no additional ANARI API calls will be made using the device.

We observe that this new data-parallel paradigm in practice actually matches how existing data parallel applications work. An application using IceT for compositing would also use MPI for synchronization. The individual ranks would operate largely independent of each other and render different content using calls to whatever library the app uses for per-rank rendering. Similarly, that application would also have to ensure consistency between different ranks’ global rendering information like camera, background color, etc., so would likely already have some mechanism to synchronize such information before it calls rendering. For our paradigm, this is exactly the same, except that we apply it to ANARI, and formalize the process.

In Fig. 1 we illustrate exactly that workflow: Upon startup all ranks would (collaboratively) create their DP-ANARI device, then do whatever the app wants to do for data loading, iso-surface extraction, etc. At some point each rank would create its local ANARI world, and populate it with different ANARI objects—these calls are not collaborative, so different ranks can do as many of those—and whichever ones they want—without any other ranks even being aware. Once all ranks are ready for rendering they all call anariRenderFrame(), at which point they will implicitly synchronize until the frame is done. Rank 0 can then map the generated frame buffer, save or display it, and wait for UI events or user input, at which point it will instruct its worker ranks to do whatever scene updates are required, etc.

The classical ANARI API—i.e., ANARI without the paradigms introduced in this paper—has no concept of a “data parallel world”. However, that does not mean that applications using ANARI could not do data-parallel rendering of their own: such applications could use ANARI to render data locally on each rank, and then rely on depth- and/or alpha compositing to somehow composite the resulting images. This approach has some obvious limitations in terms of what effects can or cannot be rendered, but these limitations are not specific to ANARI: they are not different from those when using any other rendering back-end for the per-rank rendering.

Where these limitations are acceptable, one could use exactly that same approach also within an ANARI device to implement the data-parallel ANARI paradigm we have introduced above. In such a device, almost all functions would behave in exactly the same way as in any other local device; namely, they would set up the scene to be rendered, and perform rendering of a local frame buffer. The only thing this device would need to modify is anariRenderFrame(), in which it would first perform its local rendering, and then composite the results. We decided to implement this approach, for which we need two main ingredients: a means of performing local-node rendering, and a means of compositing.

Barney is a new—and still under development—project for data-parallel path tracing on multi-node and multi-GPU hardware. For rendering, Barney relies on ray forwarding similar to what is done by the recently published Brix [38] and RQS [36] papers, where rays are sent to the node(s) that may have geometry that may intersect a given ray—and where each ray will always find its respectively closest intersection no matter which rank the ray was spawned on, or which rank holds that respective geometry.

Unlike ANARI, Barney was built with parallel rendering—and in particular, data parallel rendering—in mind from its very inception. In addition to the relatively simple mode we described above—where each rank has exactly one part of the data to be rendered—Barney also offers various additional modes such as, for example, data-replicated rendering, islands-parallel rendering, non-MPI multi-GPU rendering, additional multi-GPU data-parallel rendering within a given rank, etc. Despite this bigger set of functionalities, Barney follows the same general paradigm described above: it has the concept of a data-parallel world, different ranks can independently specify different pieces of this world, and render operations need to be synchronous.

We are ultimately most interested in how hard it is to integrate our paradigm into actual end-user applications like VisIt or ParaView, or widely used frameworks such as VTK. However, we could not evaluate that until some implementation(s) existed, but neither could we have developed the aforementioned back-ends without applications to exercise it. To break this chicken-and-egg problem we decided to defer integration into actual end-user tools to the end, and in the meanwhile, relied on developing both back-ends and several different front-ends in parallel.

While mini-apps and HayStack were invaluable from a developer’s point of view, the eventual goal for any data-parallel ANARI effort must necessarily lie with actual “real” end-user vis apps such as ParaView, VisIt, or various in-situ frameworks.

Our first observation is that our paradigm is purely additive in that it does not take anything away from existing state of the art. A single-process app already using ANARI (e.g,, TSD or Blender (cf. Fig. 4) will use ANARI in the exact same way, which will have the exact same performance implications as it did before. Similarly, an existing data-parallel app that however already uses ANARI for local, per-rank rendering (e.g., mainline ParaView without our modifications) would also see no negative effects whatsoever when using the DP-ANARI paradigm and whatever means it uses to render images in parallel.

Let us now consider this same app to follow the (simple) steps outlined above (Section 5.2.1) to actually enable that device’s data-parallel capabilities, and follow our paradigm. Assuming it simply used our compositing device (passing through to whatever it used before) it still would not see any difference whatsoever. At this point the app could already get rid of its own compositing layer, and still not “lose” anything it could have done before. Though the app would still not have seen any benefit, it also would not have lost anything by adopting our paradigm.

Our motivation for a DP-ANARI paradigm is to enable global effects usually realized with ray tracing in data-parallel sci-vis renderers. Ordinary data parallel renderers used in sci-vis cannot produce these effects without artifacts, as can be seen in Fig. 5. To illustrate the effect of missing global effects for some more realistic data, in Fig. 6 we show two large data sets (one volumetric, one surface based) rendered with Barney, and in exactly the same data-parallel configuration—but once with only local shading, and once with path tracing turned on. Clearly the images with global effects are not only “nicer”, but also convey more information (which is what visualization is about)—but with classical ANARI (i.e., without exposing the concept of a data parallel world) this could not be reproduced. An application could of course still decide to not use ANARI for such use cases at all, and instead integrate into libraries like Barney in the first place—but this would lose all the benefits of why applications are integrating the first place.

Whereas Fig. 6 explicitly illustrated data-parallel rendering with and without advanced ray tracing effects, in Standardized Data-Parallel Rendering Using ANARI and Fig. 7 we provide several more examples of what an application could expect when adopting a data-parallel path tracing paradigm.

The most obvious limitation of our approach is that it is not as easily tangible as any new API extension would be. At its heart our paradigm only specifies a convention, and even that would eventually need some sort of formalization in the ANARI spec. This is, however, not all that different from how ANARI works in general: for example, ANARI does the API call for creating a material with a given name, but it does not make any guarantees how (or even whether) a given device will implement a given material. In practice, this still works, due to what we call a normative pressure: once enough applications start to expect a given material to operate in a given way, device developers come under significant pressure to implement it the way that those applications expect. We fully expect adoption of our paradigm to work exactly the same way.

Another limitation is that our paradigm specifies how a given scene is to be created, but does not make any guarantees about how a given device will then render it. This is yet another example of ANARI being intentionally vague, and relying on said normative pressure; however, we fully expect compositing-based devices to remain in use for a considerable while, and applications will have to decide how to deal with that.

One key issue for data parallel rendering—which we have completely skirted so far—is how the scene is partitioned across the different ranks. This is important because many data-parallel renderers will only work for certain types of data partitioning. For example, IceT’s alpha blending mode [38] requires a spatial partitioning of the scene as well as an a-priori known compositing order; and similar limitations would apply to other renderers. In Sci-vis, this problem gets even more interesting because it is not the renderer that does the scene partitioning, but the application—such as pvserver for ParaView, or libsim for VisIt.

Clearly, if there is such a strong dependence on how the scene is partitioned, any API or paradigm for data parallel rendering must have a means of communicating what the back-end can consume, and/or what the front-end has generated. One way of solving this would be to specify a certain partitioning requirement in the API, but this would unduly restrict what kind of renderers could or could not be implemented.

Instead, we suggest to handle that by having the apps pass such meta information by setting a set of parameters on the underlying device. For example, apps that do want to use devices that use IceT could set some int compositingOrder and box3 boundingBox parameter on the device. Of course, this only works if the app can actually provide such data, but if it couldn’t it wouldn’t be able to use IceT, anyway. Here, however, we would expect the aforementioned normative pressure to eventually assert itself, too: if some devices have more constraints than others then clearly these will see some pressure to relieve these constraints.

The key remaining issue is to get the developers of actual tools like ParaView and VisIt to adopt this paradigm. Though we believe this paper to have made a strong argument that they should, this will likely not happen over night.

Ultimately this will also require more work on the device side. OSPRay, for example, already more or less follows the same paradigm, and already has a (single-rank) ANARI interface—but would yet have to merge these two. For our own devices, much is left to be done, too: for Barney, there are still rather large gaps between what Barney supports and what ANARI would expect. Adding these missing features—and changing existing ones to be more ANARI-like—will require significant effort. However, applications will not adopt it until it supports enough of the ANARI features that said application requires. This is another example of the aforementioned normative pressure, but it will still not happen over night.