Gemini: Integrating Full-fledged Sensing upon Millimeter Wave Communications

Compared with conventional (communication) channel, the mmWave MIMO channels for ISAC can be far more complicated, as they include at least four components: Tx interference, monostatic sensing (reflection), communication, and multi-static sensing (reflection). In a typical indoor mmWave MIMO setting, the AP has $N$ Tx/Rx chains, each of them equipped with an $M$-elements phased array. For ISAC-oriented temporal scheduling, the AP selects $N_{\mathrm{U}}$ UEs and $N_{\mathrm{S}}$ subjects (where $N=N_{\mathrm{U}}+N_{\mathrm{S}}$) from all UEs and subjects to serve at each time slot. Since the MU-MIMO via hybrid beamforming is adopted by IEEE 802.11ay (802_11ay, ), we consider the AP performing the MU-MIMO to the UEs. We let $\bm{s}(t)=[s_{1}(t),\cdots,s_{N}(t)]$, $\bm{y}^{\mathrm{u}}(t)=[y^{\mathrm{u}}_{1}(t),\cdots,y^{\mathrm{u}}_{N_{\mathrm{U}}}(t)]$, and $\bm{y}^{\mathrm{s}}(t)=[y^{\mathrm{s}}_{1}(t),\cdots,y^{\mathrm{s}}_{N_{\mathrm{S}}}(t)]$ respectively represent the Tx signals from the AP, the Rx signals at the UEs, and the monostatic sensing signals received by the AP. Note that, $\bm{y}^{\mathrm{u}}(t)$ and $\bm{y}^{\mathrm{s}}(t)$, even when happening to the same device, are temporally separated by the MAC protocol (see Section 2.1), while $\bm{y}^{\mathrm{u}}(t)$ is the superposition of the multi-static sensing signals $\bm{y}^{\mathrm{us}}(t)$ and the pure communication signals $\bm{y}^{\mathrm{uc}}(t)$, i.e., $\bm{y}^{\mathrm{u}}(t)=\bm{y}^{\mathrm{us}}(t)+\bm{y}^{\mathrm{uc}}(t)$. In total, we can characterize the simultaneous ISAC (both AP and UE) Rx signals by:

where $\bm{W}_{\mathrm{DB}}^{\mathrm{Rx}}$, $\bm{W}_{\mathrm{DB}}^{\mathrm{Tx}}$, $\bm{W}_{\mathrm{AB}}^{\mathrm{Rx}}$, and $\bm{W}_{\mathrm{AB}}^{\mathrm{Tx}}$ are the Rx digital beamformers, the Tx digital beamformers, the Rx analog beamformers, and the Tx analog beamformers, respectively, $[\cdot]^{\prime}$ denotes matrix transpose, and $\bm{n}(t)$ is the additive Gaussian noise with variance $\sigma^{2}$. It is worth noting that $\bm{W}_{*}^{\mathrm{Rx}}$ is of block-diagonal form, as it involves the beamformer of one device (e.g., UE) for $\bm{y}^{\mathrm{u}}(t)$ and that of another (e.g., AP) for $\bm{y}^{\mathrm{s}}(t)$.

The $\bm{H}(t)$ in Eqn. (1) represents the mmWave MIMO channel; it involves the following major components:

where the 4 terms on the right-hand side denote the Tx interference channels, monostatic sensing channels, communication channels, and multi-static sensing channels, respectively. According to Section 2.2, $\bm{H}_{\mathrm{ti}}(t)$ may strongly affect $\bm{H}_{\mathrm{s}}(t)$, but the impact from $\bm{H}_{\mathrm{c}}(t)$ to $\bm{H}_{\mathrm{ms}}(t)$ can be readily handled (mmtrack-INFOCOM20, ). Moreover, Eqn. (1) confirms what we have observed in Section 2.3: the hybrid beamformers can be scheduled to largely reshape the channels so as to affect performance in both communication and sensing. Consequently, though the two groups of channels, i.e., $\{\bm{H}_{\mathrm{ti}}(t),\bm{H}_{\mathrm{s}}(t)\}$ vs $\{\bm{H}_{\mathrm{c}}(t),\bm{H}_{\mathrm{ms}}(t)\}$, are seemingly independent of each other as they are physically or temporally separated, they become correlated as they may share the same Tx beamformers $\bm{W}_{*}^{\mathrm{Tx}}$. In a nutshell, the goal of Gemini is to optimize the hybrid Tx/Rx beamformers on all these channels in $\bm{H}(t)$ for balancing the performance between sensing and communication. We take a divide-and-conquer method to approach this optimization, by eliminating $\bm{H}_{\mathrm{ti}}(t)$ in Section 3.2 and jointly scheduling $\{\bm{H}_{\mathrm{s}}(t),\bm{H}_{\mathrm{c}}(t),\bm{H}_{\mathrm{ms}}(t)\}$ in Section 3.3. However, before executing this plan, a scheme to probe the channel states needs to be in place.

Recall that a probing scheme is employed for mmWave communications to acquire $\bm{H}_{\mathrm{c}}$ in Section 2.1 under the sector sweep phase, which also yields $\bm{H}_{\mathrm{ms}}(t)$ and the bearings of subjects via static background removal (adib2015multi, ) and signal detection (richards2014fundamentals, ). Since $\{\bm{H}_{\mathrm{ti}}(t),\bm{H}_{\mathrm{s}}(t)\}$ cannot be directly acquired by this scheme, we piggyback an additional probing scheme on the existing one by emulating its behavior. Consequently, we mainly focus on this added scheme aiming to estimate the Tx interference channels $\bm{H}_{\mathrm{ti}}(t)$ for the follow-up cancellation that obtains monostatic sensing channel $\bm{H}_{\mathrm{s}}(t)$ in Section 3.2. A byproduct of this two-round probing and the later resulting $\bm{H}_{\mathrm{s}}(t)$ is the locations of all subjects, as subject ranges can be inferred from $\bm{H}_{\mathrm{s}}(t)$’s time-of-flight.

Since the phased array equipped at Tx chain has multiple beam patterns and each of them causes different Tx interference, a probing scheme needs to collect all kinds of corresponding Tx interference channels. Therefore, our added scheme involves an additional round of SSW with a substantially lower Tx power to create a “wireless shortcut” for probing $\bm{H}_{\mathrm{ti}}(t)$, in order to avoid the interference from surrounding environments. As shown in Figure 6, upon transmitting SSW frames with Tx sector ID sequentially from a device (the pink sectors), the phased arrays of its Rx chains are set to quasi-omni-directional patterns to receive SSW frames (the blue sectors). In this way, the channel estimation field with TRN of SSW is utilized to extract the Tx interference channels $\bm{H}_{\mathrm{ti}}(t)$ via a minimum mean square-error estimation (speth1999optimum, ).

Whereas the UE channel in Figure 7(a) is discovered by the 1st round SSW, the two sensing channels in Figure 7(b) are probed during the same SSW round. Apparently, the beamforming for UE may allow for sensing ‘Subject1’ at the same time, yet sensing ‘Subject2’ has to be scheduled in a different time slot. Moreover, the 2nd round SSW obtains Tx interference states shown in Figure 7(c): the impact appears to be more intensive at 0^∘ and is otherwise quasi-symmetric on two sides. We also measure the correlation between any two Tx interference channels at different time slots, and the results in Figure 7(d) demonstrate correlation coefficients mostly above 0.8. Therefore, the information provided by one round SSW can help train a cancellation process that remains valid for a long period.

We leverage the hybrid beamforming to reduce the Tx interference at the first stage. Since advanced phased array allows for controlling both amplitude and phase (SiversBF01_2021, ) via antenna weights vectors (AWV), we leverage this ability to offer efficient solutions for future developments. To achieve beam nulling for the $i$-th sector containing at least a subject, we leverage the hybrid beamforming to minimize Tx interference $\bm{H}^{i}_{\mathrm{ti}}(t)$:

where $\|\cdot\|_{F}$ is the Frobenius norm. Since the hybrid beamformers involve a large amount of parameters to be optimized, searching for an optimal solution directly using conventional optimization methods can be highly inefficient. Moreover, since these parameters will also be needed for the joint scheduling in Section 3.3, the degree of freedom in fine-tuning them is limited.

Fortunately, we find the problem (4) can be approached as training a neural network, given the similarity between the data pipeline (see Figure 5) and a linear autoencoder (kunin2019loss, ). We reformulate Eqn. (4) into an inverse neural network shown in Figure 8, where the Tx interference channels and the Tx/Rx hybrid beamformers are respectively modeled as the bottleneck layer (weights frozen as the channel $\bm{H}^{i}_{\mathrm{ti}}(t)$ is physically determined) and decoder/encoder whose weights are beamformer parameters, while the Tx signals $\bm{s}$ become the TRN. The objective of a hypothetical training process should result in the output $\bm{x}$ being the $\bm{H}_{\mathrm{s}}(t)$ after canceling the overwhelming $\bm{H}^{i}_{\mathrm{ti}}(t)$. However, the network cannot be trained without any ground truth dataset for $\bm{H}_{\mathrm{s}}(t)$.

By far, we have reformed the problem to reconstructing $\bm{H}_{\mathrm{s}}(t)$ from the observation $\bm{H}^{\prime}(t)=\bm{H}^{i}_{\mathrm{ti}}(t)+g\left(\bm{H}_{\mathrm{s}}(t)\right)$ where the $g(\cdot)$ is a deterministic mapping introduced by the beamforming operations. Since obtaining the ground truth for $\bm{H}_{\mathrm{s}}(t)$ is infeasible, we try to bypass this issue by using a mmWave radar at the 60 ​GHz with 2 ​GHz bandwidth (TImmWave, ) to collect a dataset for $\bm{H}^{\prime}_{\mathrm{s}}(t)$: since the radar is designed for sensing but with system parameters similar to Gemini, $\bm{H}_{\mathrm{s}}(t)$ and $\bm{H}^{\prime}_{\mathrm{s}}(t)$ should share high-level yet intrinsic features. It should be stressed that the mmWave radar is only deployed during the training stage and is surely not required by Gemini in runtime. Though we cannot directly use $\bm{H}^{\prime}_{\mathrm{s}}(t)$ as ground truth for supervised learning, exploiting it to perform adversarial learning is certainly feasible. To this end, we append an digital cancellation neural network $\bm{C}_{\mathrm{D}}(t)$ to the above model (see Figure 8), and we train the whole model as a cGAN (conditional Generative Adversarial Network) (mirza2014conditional, ). Our model is trained to learn how to cancel Tx interference regardless of all environment inferences, so only one training is needed for each type of mmWave device. We denote by $\mathcal{G}$ the whole neural network constructed so far; it generates samples $\bm{x}_{g}=\mathcal{G}(\bm{z}|\bm{s})$ with $\bm{z}$ being the background Gaussian noise introduced by the signal processing/propagation pipeline (LAE). We further employ a discriminator network $\mathcal{D}(\bm{x})$ aiming to recognize if $\bm{x}$ comes from $\{\bm{H}^{\prime}_{\mathrm{s}}(t)\}$. According to (mirza2014conditional, ), $\mathcal{D}$ and $\mathcal{G}$ play a min-max game modeled by:

where $\mathcal{N}$ denotes the Gaussian noise distribution. Essentially, we use the cGAN as a powerful non-linear filter to obtain the monostatic sensing channels by “generating” it out of the channel states contaminated by $\bm{H}^{i}_{\mathrm{ti}}(t)$.

To verify the effectiveness of our design, we let a subject stand in the front of our platform with $15^{\circ}$ bearing and extract its respiration via the proposed pipeline. We first show the heatmaps of our two-stage cancellation in Figures 9(a) and 9(b); it clearly demonstrates the gradually “diminishing” of $\bm{H}^{i}_{\mathrm{ti}}(t)$ through the pipeline. Further plotting the respiration sensing results in Figure 9(c) confirms the largely suppressed Tx interference ²²2While Gemini enables Tx interference cancellation, packet detection delays may introduce ranging errors. This is mitigated through an RF loopback calibration technique (guan-TMTT2021, ) in the RF front-end to correct the corresponding phase offsets; it processes Rx signals before ranging..

UE selection has a default implementation in 802.11ay as MU-MIMO. In particular, MU-MIMO starts with AP using the SSW (see Section 2.1) to obtain channel states from UEs and grouping UEs with correlated channels (of similar bearing) (shen2015sieve, ) as a set to be covered by the same analog beam pattern. Then the bearing scheduling arranges UEs from the orthogonal (non-correlated) sets to communicate via distinct Tx chains simultaneously. For UEs covered by the same beam pattern or unable to be scheduled spatially, a temporal scheduling operating in a round-robin fashion is adopted to serve them. In summary, the communication-oriented UE selection operates in a single dimension manner for bearing scheduling, leading to inefficiency in handling the conflict among pure communication, pure sensing, and simultaneous communication and sensing (see Section 2.3) under ISAC context.

Unlike communication, sensing allows subjects with the same bearing but different ranges to be differentiated, given a sufficient range resolution. With the 2 ​GHz bandwidth of Gemini, the resulting centimeter-level resolution may enable many subjects to be simultaneously covered by one beam pattern, adding one more dimension for the ISAC scheduling. In addition, locations of UEs and subjects are known (see Sections 2.1 and 3.1.2). Consequently, we upgrade the correlated set definition to a two-dimensional Beam-Compatible Set (BC-Set) in the beamforming phase; it aims to group both UEs and subjects under a single beam pattern that may serve one UE and all the covered subjects simultaneously. We plot five BC-Sets in Figure 10; they are meant for illustrative purpose only, as realistic beam patterns (shown on the right part) can be far more irregular.

To determine the BC-Sets and their corresponding beam patterns, we start from the default communication scheduling that outputs the correlated communication sets $\mathcal{U}=\{U_{1},\cdots\}$ (802_11ay, ). As illustrated in Algorithm 1, our algorithm essentially executes in a greedy manner: it first adds as many subjects as possible to sets in $\mathcal{U}$ for forming the initial BC-Sets, then it constructs new BS-Sets to cover the remaining subjects. It first gathers all positions covered by beam patterns of individual sets in $\mathcal{U}$ (line 1). Then it performs a range-wise depth-first search (DFS) to gather subjects that can be covered by these sets (line 1), since initially beam patterns are often very narrow. If possible, it adjusts the original narrow beam pattern to trade its range for beam width, making it possible to conduct a bearing-wise breadth search for covering more subjects (line 1). Upon upgrading all sets in $\mathcal{U}$ to BC-Sets, the algorithm again proceeds in the greedy DFS manner in the next stage, aiming to construct new BC-Sets to cover the remaining subjects (line 1 till the end).

What Algorithm 1 outputs are only (logical) BC-Sets; they need to be “translated” into (physical) beam patterns. Essentially, the beam patterns should be shaped by both the distribution of UEs/subjects covered by the BC-Sets and the capability of the phased array (gu2021packaging, ). The procedure of shaping a beam pattern follows two basic principles: i) adopting wider beams to trade range for width coverage and ii) adjusting the direction of the main beam to balance the coverage among multiple entities. Since the resulting patterns have not taken into account the beam nulling requirements (see Section 3.2), the corresponding beamformers are used as the initial setting for the problem (4) (hence the LAE) to derive proper cancellation schemes for individual BC-Sets. Finally, we analyze the complexity of Algorithm 1 to demonstrate its efficiency for real-time execution. Let the number of UEs be $K$ and that for range (resp. bearing) bins be $K_{\mathrm{R}}$ (resp. $K_{\mathrm{D}}$), the complexity is $O(KK_{\mathrm{D}}K_{\mathrm{R}})$, often up to the scale of several hundreds only.

We may walk through the algorithm using the BC-Sets in Figure 10 as examples. During the 1st stage, the three UE sets (red, green, and blue) are sequentially upgraded to form BC-Sets by adding in more subjects. Then the remaining three subjects are covered by two new BC-Sets (yellow and cyan). While the red and yellow BC-Sets can be covered by the default (narrow) beam sectors, others should be reshaped: in particular, both principles apply to the blue one. We also perform a trace-driven emulation to evaluate the advantages of BC-Sets. We set 6 UEs and 14 subjects in our experiment, and randomize their positions in each experiment. We run three algorithms each for 100 trials, and show the average time spans for serving all UEs and subjects in Figure 11. Apparently, our scheduling requires only half of RR’s time span and goes close to the optimal solution.

We begin by quantifying the interference cancellation ability at different stages. We set the Tx power to 40​ dBm and observe the signal strength at different stages indicated in Figure 8. The results shown in Figure 14 confirm that LAE reduces Tx interference by about 39​ dB, and deep denoising further suppresses it by 27​ dB. Overall, the total cancellation is approximately 66​ dB, and the remaining Tx-interference power goes very close to the noise floor. Apparently, the monostatic reflection channels $\bm{H}_{\mathrm{s}}(t)$ has been distilled from $\bm{H}(t)$ in Eqn. (3).

The ability to ranging is a fundamental and crucial sensing feature, but it can only be accomplished in the monostatic mode (isacot, ). We study the ranging performance with/without Tx interference cancellation, given the subject staying statically in the range of $[1,5]$ ​m with a step size of 0.5 ​m from Gemini, and Gemini performing ranging via CIR. We use the TI radar as a baseline by co-locating it with Gemini. The range errors shown in Figure 15(a) confirm that Gemini can obtain comparable performance to the radar, but much higher median range errors up to 30 ​cm are introduced without the Tx interference cancellation. The range errors of both TI radar and Gemini slightly grow as the distance increases, potentially because the adopted beam pattern covers more background clutters at further distances and multipath reflections from them affect the ranging accuracy. Moreover, reflected (sensing) signals attenuate in distance and hence also result in degraded performance.

To study another fundamental sensing function, we leverage the phased arrays equipped with both Gemini and the TI cascaded radar (TImmWave, ) to estimate object bearing (or AoA). We let the object stay within a bearing of $[-45^{\circ},45^{\circ}]$ and vary with a step size of $15^{\circ}$. As shown in Figure 15(b), Gemini achieves better performance than that of radar because Gemini has a much more powerful phased array than that of radar. Again, Tx interference significantly degrades the bearing estimation performance if not properly handled.

The motion sensing is also crucial to the capability of Gemini, so we conducted tests where the object moves at a varying speed ranging from 0.5​ m/s to 3.0​ m/s within a 3 $\times$ 20​ $\text{m}^{2}$ corridor. Since the LiDAR cannot monitor motion, we have to change the role of the TI radar from baseline to measuring the ground truth. The evaluation results shown in Figure 15(c) clearly demonstrate that Gemini achieves a much lower speed estimation error with the enhancement offered by the Tx interference cancellation.

This experiment takes the subject’s respiration as the sensing target, aiming to accurately estimate the breath rate. The results plotted in Figure 17 showcase the obtained throughput against corresponding sensing accuracy. We may observe that, while Gemini achieves a sensing accuracy almost the same as that for both SO and TI radar (which is much better than that for CO and RR), its throughput is only marginally lower than that of CO (but still much higher than that for SO or RR). In fact, for continuous yet spatially coarse-grained sensing applications, Gemini can always obtain nearly perfect sensing performance and barely sacrifice throughput.

Different from continuous sensing application in Section 5.2.1, point cloud is a one-shot sensing application (e.g., 1​ s duration in our experiment). In such applications, a much narrower beam is required to generate dense point cloud, yet Gemini still needs to maintain its beam schedules. Our solution is to have asymmetric Tx and Rx beam patterns for the AP: while the Tx one is designed to match that determined by the corresponding BC-Set, the Rx chain tunes its beam to the finest $1.5^{\circ}$ beamwidth and quickly switches to distinct azimuth and elevation angles for “scanning” the subject under the BC-Set coverage. According to the reflection power at different angle pairs, we set a threshold to filter out reflection signals with very lower power, and map the residue signals from polar coordinates to Cartesian coordinates so as to generate point cloud (gao2019experiments, ).

We test Gemini and the baselines in sequence and report the results in Figure 18: while throughput is statistics from all trials, the one-shot point cloud results are arbitrarily chosen examples (a human figure) for demonstration purpose only. Results in Figure 18(a) suggest that the throughput gap between Gemini and CO gets slightly wider, but Gemini still outperforms other two baselines: to generate 3D point cloud, Gemini has to temporarily sacrifice a bit throughput to capture fine-grained reflection signals. Though SO in Figure 18(c) performs marginally better than Gemini in Figure 18(b), the main beam of SO has to solely target the subject, hence resulting in the lowest throughput. RR sits in the middle of CO and SO for both throughput and point cloud performance shown in Figure 18(a) and Figure 18(e), due to its time-division nature. Finally, the TI cascaded radar with a number of antennas (12 Tx $\times$ 16 Rx) also performs poorly with its lower bearing resolution than Sivers’ phased array antennas (SiversBF01_2021, ) illustrated in Figure 18(f).

We captured the human movements in a room approximately 21$\times$18 feet, slightly larger than a typical US family room, accommodating scenarios with larger movements involving multiple UEs and several subjects. Human subjects follow the predefined paths marked on the floor, specifically ’Rectangular’ and ’Straight Line’ trajectories as depicted in Figure 19. Throughput and tracking performance are depicted in Figure 19. As for the sensing performance shown in Figures 19(c) to 19(g), human movement exhibits a similar performance ranking as static point cloud generation, as it is also about producing point cloud but in both temporal and spatial dimensions. For the case involving multiple human subjects and UEs as shown in Figure 19(h), we placed two frontends as UEs to receive the data from the AP, with each subject following one of the two predefined paths. Despite the slight decreases in throughput and tracking accuracy observed with two UEs tracking two targets (MT), Gemini still outperforms baselines with only one UE and one target. It is attributed to our beam scheduling algorithm, which effectively balances communication and sensing tasks.

In fact, the idea of ISAC is not meant to benefit any specific applications; it is basically striking a balance between sensing and communication, at the benefit of having them co-existing in one hardware. Meanwhile, Gemini, with its innovative scheduling driven by BC-Set, is superior to existing mmWave-ISAC solutions, as it is demonstrated by our intensive experiments to have obtained performance very close to either SO (for sensing) or CO (for communications), given that both sensing and communication functions co-exist in one hardware and operate simultaneously.