Enhancement of Over-the-Air Federated Learning by Using AI-based Fluid Antenna System

Abstract

This letter investigates an over-the-air federated learning (OTA-FL) system that employs fluid antennas (FAs) at the access point (AP) to enhance learning performance by leveraging the additional degrees of freedom provided by antenna mobility. First, we analyze the convergence of the OTA-FL system and derive the optimality gap to illustrate the influence of FAs on learning performance. Then, we formulate a nonconvex optimization problem to minimize the optimality gap by jointly optimizing the positions of the FAs and the beamforming vector. To address the dynamic environment, we cast this optimization problem as a Markov decision process (MDP) and propose the recurrent deep deterministic policy gradient (RDPG) algorithm. Finally, extensive simulations show that the FA-assisted OTA-FL system outperforms systems with fixed-position antennas and that the RDPG algorithm surpasses the existing methods.

I Introduction

Federated learning (FL) has gained significant prominence in the field of machine learning in recent years due to its decentralized framework and robust privacy protection measures. FL leverages the computational capabilities of edge devices to collectively train a unified global model, all while ensuring that sensitive locally stored data remains confidential. This approach is particularly suitable for applications in various Internet of Mobile Things (IoMT) scenarios, including the Internet of Drones (IoD)[1] and mobile crowd sensing[2], among others. To mitigate communication latency and costs associated with FL implementation, over-the-air computation (AirComp) for model aggregation has emerged as a promising approach. It leverages the superposition property of wireless multiple access channels (MAC) [3]. However, the performance of model aggregation in over-the-air FL (OTA-FL) is severely affected by adverse wireless propagation channel conditions, especially in massive mobile IoT scenarios.

To address this challenge, previous research has extensively explored the integration of intelligent reflecting surfaces (IRSs) into OTA-FL to achieve reliable model aggregation. This is accomplished by reconfiguring wireless channels through adjusting the passive reflecting coefficients of the IRS [3]. Additionally, other studies have investigated beamforming techniques at the receiver server to enhance OTA-FL performance by leveraging the degrees of freedom (DoF) in the spatial domain [4]. However, the fixed positions of receiver antennas can impose limitations on the potential gains from beamforming. To overcome these limitations, we propose the use of fluid antennas (FAs) in OTA-FL to dynamically manipulate wireless channel conditions through adaptive movement of antennas. Unlike fixed-position antennas (FPA), FAs offer the capability to reshape channel environments, thereby introducing new DoFs to enhance OTA-FL performance [5]. Previous studies have demonstrated the superior performance of FAs over traditional FPAs in various communication systems, including AirComp systems [6], multi-user uplink communications [7, 5, 8], and mobile edge computing [9]. Moreover, FAs have been investigated to maximize network sum-rate in multiple access communication systems through deep reinforcement learning (DRL) [10]. However, the integration of FAs into OTA-FL systems has not yet been addressed in current academic literature to the best of our knowledge.

In this paper, we propose the integration of FA systems with OTA-FL to enhance convergence performance. Our objective is to minimize the optimality gap through joint optimization of the beamforming vector and the antenna position vector at the access point (AP) under practical dynamic conditions [11]. First, we derive the optimality gap between the actual loss and the optimal loss for OTA-FL to quantify the impact of the beamforming vector and antenna positioning. Subsequently, based on the convergence analysis, we formulate a non-convex optimization problem to enhance learning performance. We then reformulate it as a Markov decision process (MDP) and leverage deep reinforcement learning (DRL) techniques suitable for dynamic environments. We introduce the recurrent deep deterministic policy gradient (RDPG) algorithm, with actor and critic networks specifically designed to capture the temporal correlation of state features, enabling real-time decisions under dynamic conditions. Extensive simulations are undertaken to evaluate the performance of the integrated FAs in OTA-FL systems compared to FPA, and the performance of the proposed RDPG algorithm in comparison to standard DRL algorithms such as soft actor-critic (SAC) and deep deterministic policy gradient (DDPG). Simulation results indicate that the proposed RDPG algorithm outperforms both SAC and DDPG methods in terms of performance and stability in both scenarios. Furthermore, the FA-assisted OTA-FL system consistently outperforms the FPA scenarios.

Notations: Italicized letters represent scalars, while boldface letters denote vectors. $(\cdot)^{T}$ is the transpose, $(\cdot)^{H}$ the conjugate transpose, and $\mathbb{E}[\cdot]$ the expectation operation. $|\cdot|$ signifies the magnitude of a scalar or the cardinality of a set. The Euclidean norm of a vector is represented by $\left\|.\right\|$.

II System Model

We consider an OTA-FL system comprising $K$ single-antenna user equipment (UE) devices, denoted as $\text{UE}_{k}$, $\forall k\in\mathcal{K}\triangleq\{1,2,\ldots,K\}$. These devices could be include mobile IoT devices, UAVs, drones, or similar entities. The UEs are randomly distributed and move dynamically within a designated area of interest, where they collect local data samples. These samples are collaboratively utilized to train a global model at an AP equipped with $N$ FAs.

II-A OTA-FL Model

We consider an OTA-FL framework with full participation that executes sequential actions at each iteration $t$ over $T$ training rounds as follows:

• Global model broadcast: The AP broadcasts the current global model $\boldsymbol{w}_{t}\in\mathbb{R}^{d}$ to all UEs, where $d$ represents the dimensionality of the model parameter space.

• Local model update: Each UE updates its local model using the gradient descent algorithm as $\boldsymbol{w}_{k,t}=\boldsymbol{w}_{t}-\gamma\nabla{F}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{k})$, where $\gamma$ is the learning rate, $\nabla{F}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{k})$ represents the gradient of the local loss function, and $\mathbf{\mathcal{D}}_{k}$ is the local dataset for $\text{UE}_{k}$ with a local dataset size denoted by $|\mathbf{\mathcal{D}}_{k}|=D$.

• Model aggregation: Each UE transmits its local model to the AP, which then performs aggregation by averaging to update the global model as:

  $$\boldsymbol{w}_{t+1} =\frac{1}{K}\sum_{k\in\mathcal{K}}\boldsymbol{w}_{k,t}.$$ (1)

The procedure continues iteratively until reaching the maximum specified number of outer iterations.

II-B Communication Model

We consider the uploading phase within the OTA-FL system, where each UE synchronously transmits its updated model parameters to the AP. The AP is equipped with an array of FAs, facilitating the adjustment of each FA along a one-dimensional line segment that spans a length of $X$. Each FA’s position is constrained within the interval $[0,X]$ with a minimum distance $X_{0}$ between adjacent FAs to prevent antenna coupling. The collective locations of all $N$ FAs are represented as the vector $\mathbf{x}=[x_{1},\ldots,x_{N}]^{T}$, with their movement along one dimension restricted by $x_{1}<x_{2}<\ldots<x_{N}$. For the purposes of brevity and clarity, time indices are omitted in this subsection. Assuming line-of-sight (LoS) propagation conditions, the channel between the $\text{UE}_{k}$ and the AP, represented by $\mathbf{h}_{k}[\mathbf{x}]\in\mathbb{C}^{N\times 1}$, is given by

$$\mathbf{h}_{k}[\mathbf{x}]=\sqrt{\frac{l_{0}}{{d^{\alpha}_{k}}}}\left[e^{j\frac{2\pi}{\lambda}x_{1}\cos(\phi_{k})},\ldots,e^{j\frac{2\pi}{\lambda}x_{N}\cos(\phi_{k})}\right]^{T},$$ (2)

where $l_{0}$, $\lambda$, and $\alpha$ denote the path loss at the reference distance, the wavelength, and the path loss exponent, respectively. $d_{k}$ and $\phi_{k}$ represent the distance between the FAs and $\text{UE}_{k}$ and the angle of arrival (AoA) of the LoS path, respectively, which are determined by the location of the UEs in each training round. In this system, we assume each UE moves within an designated area and then transmits model parameters from a stationary position [2]. Moreover, given that the signal path length significantly exceeds the FA movement area, the far field condition between the AP and UEs is assumed. Consequently, $\phi_{k}$ and $d_{k}$ are treated as constants during transmission, regardless of FA positional changes [9, 7].

The AP receives the local model parameters from all UEs in the $t$-th training round as:

$$\mathbf{y}=\sum_{k\in\mathcal{K}}{p}_{k}\mathbf{{h}}_{k}[\mathbf{x}]\boldsymbol{w}_{k}+\mathbf{z},$$ (3)

where, $p_{k}$ denotes the transmission power factor for the $k$-th UE, and $\mathbf{z}\in\mathbb{C}^{N\times d}$ represents an additive white Gaussian noise (AWGN) matrix with elements following a complex normal distribution $\mathcal{CN}(0,\sigma^{2})$.

The aggregated model parameter vector, $\boldsymbol{\hat{w}}$, in the $t$-th training round is estimated by conducting post-processing on the received signal at the AP as follows:

$$\boldsymbol{\hat{w}} =\frac{1}{K}(\frac{1}{\sqrt{\eta}}\boldsymbol{m}^{H}\mathbf{y})$$ (4)

$$=\frac{1}{K}(\sum_{k\in\mathcal{K}}\frac{1}{\sqrt{\eta}}\boldsymbol{m}^{H}{p}_{k}\mathbf{h}_{k}[\mathbf{x}]\boldsymbol{w}_{k}+\frac{\boldsymbol{m}^{H}\mathbf{z}}{\sqrt{\eta}}),$$

where, $\boldsymbol{m}\in\mathbb{C}^{N\times 1}$ denotes the beamforming vector at the AP, and $\eta$ represents the scaling factor for signal amplitude alignment.

According to [4, 3], in order to minimize the mean square error (MSE) in the OTA-FL system, the transmit power factor in each client is designed as follows:

$${p}_{k}=\frac{\sqrt{\eta}\left(\boldsymbol{m}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]\right)^{H}}{\left|\boldsymbol{m}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]\right|^{2}}.$$ (5)

We consider that the transmission power allocated to each $\text{UE}_{k}$ does not exceed the maximum transmission power limit $p_{\text{max}}$, expressed as:

$$\frac{1}{d}p_{k}^{2}\mathbb{E}\left[\|\boldsymbol{w}_{k}\|^{2}\right]\leq p_{\max},\quad\forall k\in\mathcal{K}.$$ (6)

Moreover, under the assumption of full participation in FL to adhere to the maximum power budget for each client, the upper bound of $\eta$ must satisfy the following condition:

$$\eta\leq\frac{dp_{\text{max}}\left|\boldsymbol{m}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]\right|^{2}}{\mathbb{E}\left[\|\boldsymbol{w}_{k}\|^{2}\right]},\quad\forall k\in\mathcal{K}.$$ (7)

III Convergence Analysis

In order to facilitate our analysis of convergence, we incorporate the following commonly accepted assumptions as discussed in [3, 1, 12]:

Assumption 1: The global loss function $F(\boldsymbol{w})$ is $\ell$-smooth. Namely, for any given model parameters $\boldsymbol{w},\boldsymbol{v}\in\mathbb{R}^{d}$, there exists a nonnegative constant $\ell$, such that

$$F(\boldsymbol{w})-F(\boldsymbol{v})\leq(\boldsymbol{w}-\boldsymbol{v})^{T}\nabla F(\boldsymbol{v})+\frac{\ell}{2}\|\boldsymbol{w}-\boldsymbol{v}\|^{2}.$$ (8)

Assumption 2: The loss function satisfies the Polyak-Lojasiewicz inequality, where $F(\boldsymbol{\boldsymbol{w}}^{*})$ denotes the optimal global loss value and $\mu>0$, as follows:

$$\|\nabla F(\boldsymbol{w})\|^{2}\geq 2\mu[F(\boldsymbol{w})-F(\boldsymbol{w}^{*})].$$ (9)

Assumption 3: The upper limit of the model parameter for $\text{UE}_{k}$ is denoted as $\Gamma\geq 0$. This ensures:

$$\mathbb{E}\left[\left\|\boldsymbol{w}_{k}\right\|^{2}\right]\leq\Gamma,\quad\forall k\in\mathcal{K}.$$ (10)

Theorm 1: Under the conditions outlined in Assumptions III, III, and III, and setting the learning rate to $1/\ell$, the optimality gap after $T$ rounds of training is bounded as follows:

$$\mathbb{E}[F(\boldsymbol{w}_{T+1})]-F(\boldsymbol{w}^{*}) \leq\psi^{T}\left(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*})\right)$$ (11)

$$+\sum_{t=1}^{T}\psi^{T-t}\Theta_{t}=\Phi_{T}(\boldsymbol{m},\boldsymbol{x}),$$

where, $\Theta_{t}$ and $\psi$ can be represented as follows:

$$\Theta_{t}=\frac{l\sigma^{2}\Gamma}{2K^{2}p_{\max}}\max_{k\in\mathcal{K}}\frac{\|\boldsymbol{m}_{t}^{H}\|^{2}}{|\boldsymbol{m}_{t}^{H}\boldsymbol{h}_{k}[\boldsymbol{x}]|^{2}},$$ (12)

$$\psi=1-\frac{\mu}{l}.$$ (13)

Proof:

See Appendix. ∎

IV Problem Formulation

In this paper, our objective is to enhance learning performance in OTA-FL through the design of FA systems within dynamic environments. According to Theorem III, the optimality gap is influenced by the configuration of the beamforming vector and the FA locations in the training iterations. Thus, we formulate an optimization problem to jointly optimize $\boldsymbol{m}=[m_{1},\ldots,m_{N}]^{T}$ and $\boldsymbol{x}=[x_{1},\ldots,x_{N}]^{T}$ for all $t\in[1,\ldots,T]$, aiming to minimize the total optimality gap as follows:

$$\mathcal{P}_{1}: \min_{\boldsymbol{m},\boldsymbol{x}}\hskip 5.69046pt\Phi_{T}(\boldsymbol{m},\boldsymbol{x})$$ (14)

$$\text{s.t.}\quad C_{1}:0\leq x_{n}\leq X,\ \ \ \ \ \ \forall n\in[1,\ldots,N],$$

$$\ \ \ \ \ \ C_{2}:x_{1}<x_{2}<\ldots<x_{N},$$

$$\ \ \ \ \ \ C_{3}:x_{n}-x_{n-1}>X_{0},\ \forall n\in[2,\ldots,N],$$

where, $C_{1}$ constrains the permissible range for FA locations, $C_{2}$ determines the order of FA placements, and $C_{3}$ enforces a minimum separation distance between adjacent FAs.

Due to the non-convex nature of the objective function and the stochasticity inherent in the dynamic environment, particularly in massive mobile IoT scenarios, traditional optimization methods face challenges in solving the original intractable problem $\mathcal{P}_{1}$. In order to tackle this issue, we propose transforming problem $\mathcal{P}_{1}$ into an online optimization problem, subsequently reformulating it as an MDP. Based on Theorem III, the optimality gap at the $t$-th training round, denoted as $\Phi_{t}(\boldsymbol{m},\boldsymbol{x})$, is bounded as follows:

$$\Phi_{t}(\boldsymbol{m},\boldsymbol{x}) \leq\text{$\Phi_{t-1}(\boldsymbol{m},\boldsymbol{x})$}$$ (15)

$$+(\psi^{t}-\psi^{t-1})(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*}))+\Theta_{t}(\boldsymbol{m},\boldsymbol{x}),$$

where (15) indicates that when $\psi$ and the initial optimality gap $(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*}))$ are known, the optimality gap is determined by $\Theta_{t}$ and the previous optimality gap $\Phi_{t-1}(\boldsymbol{m},\boldsymbol{x})$. Thus, the problem $\mathcal{P}_{1}$ of minimizing the optimality gap after $T$ communication rounds can be transformed into minimizing $\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$ in each round. This reformulation is expressed as follows:

$$\mathcal{P}_{2}:\min_{\boldsymbol{m},\boldsymbol{x}}\hskip 5.69046pt\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$$ (16)

$$\ \ \ \ \text{s.t. }C_{1},C_{2},C_{3}.$$

V Proposed DRL Algorithm

To address Problem $\mathcal{P}_{2}$, we deploy a DRL agent on the AP to learn an optimal decision policy that simultaneously optimizes the beamforming vector $\boldsymbol{m}$ and the FA locations $\boldsymbol{x}$ in each training round in order to minimize $\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$. Details of the MDP are provided below:

• State Space: The state space at time slot $t$ consists of the distances $d_{k}$ between the FAs and the $\text{UE}_{k}$, and the AoA of the LoS paths $\phi_{k}$, $\forall k\in\mathcal{K}$. The state space can be expressed as: $s_{t}=[[d_{1},\ldots,d_{K}],[\phi_{1},\ldots,\phi_{K}]]$.

• Action space: The action space at each time slot $t$ comprises the beamforming vector and the locations of the FAs. Consequently, the action space at time slot $t$ can be expressed as:

  $$a_{t}=[[m_{1},\ldots,m_{N}],[x_{1},\ldots,x_{N}]].$$ (17)

• Reward function: Based on (12), in order to minimize $\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$, the reward function can be formulated as:

  $$r(s_{t},a_{t})=\begin{cases}r_{1},\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \|\boldsymbol{m}\|=0,\\ r_{2}\max_{k\in\mathcal{K}}\left(\frac{\|\boldsymbol{m}\|^{2}}{|\boldsymbol{m}^{H}\mathbf{h}_{k}[\boldsymbol{x}]|^{2}}\right),\ \ \ \ \ \text{otherwise},\end{cases}$$ (18)

  where, the constants $r_{1}$ and $r_{2}$ are negative values that require tuning during the simulation process to achieve better convergence. Notably, the reward function is formulated as a negative value. Therefore, by maximizing this (negative) reward, the agent effectively minimizes $\Theta_{t}(\boldsymbol{m},\boldsymbol{x})$.

Since the action space is continuous, we cannot use model-free value-based DRL algorithms such as deep Q-network (DQN), as they can only handle discrete action spaces. Instead, we utilize policy gradient-based reinforcement learning methods. The DDPG algorithm is a suitable off-policy actor-critic approach capable of managing continuous action spaces. However, the fully-connected deep neural networks (DNNs) employed in conventional DDPG are inadequate for capturing the temporal patterns of environmental dynamics, such as user mobility [13, 11]. Therefore, we adjust the RDPG approach by incorporating long short-term memory (LSTM) into the DDPG architecture to exploit temporal state patterns and adapt continuously to environmental dynamics.

The proposed RDPG algorithm utilizes four neural networks: an Actor Network (policy network) denoted by $\pi_{\phi}$ with parameter $\phi$, which determines actions $a_{t}=\pi_{\phi}(s_{t})+\xi$ based on states $s_{t}$, where $\xi$ is a random process added to actions for exploration; a Critic Network (Q-network) with parameters $\theta$ that computes Q-values $Q_{\theta}(s_{t},a_{t};\theta)$ for state-action pairs; a Target Actor Network, which is an older version of the actor network; and a Target Critic Network, which is an older version of the critic network.

In our optimization problem, the objective is to minimize the optimality gap by maximizing the expected reward $r(s_{t},a_{t})$ in each training round. The goal of the RDPG, given the state $s_{t}$ and action $a_{t}$, is to identify a policy that maximizes the expected cumulative reward, defined as:

$$\pi^{*}=\arg\max_{\pi}\mathbb{E}_{s_{t},a_{t}}\left[\sum_{t=0}^{\infty}r(s_{t},a_{t})\right],$$ (19)

to achieve this, the Actor Network is optimized based on the gradient of the objective function $J(\phi)$ as follows:

$$\nabla_{\phi}J(\phi)=\mathbb{E}\left[\nabla_{a_{t}}Q_{\theta_{1}}(s_{t},a_{t})\bigg{|}_{a_{t}=\pi_{\phi}(s_{t})}\nabla_{\phi}\pi_{\phi}(s_{t})\right],$$ (20)

while the critic network is trained to minimize the loss function relative to the target value $Y_{t}$, defined as:

$$Y_{t}=r_{t}+\gamma Q_{\theta_{i}^{\prime}}(s_{t+1},\pi_{\phi^{\prime}}(s_{t+1})+{\xi}).$$ (21)

The proposed RDPG method is described in Algorithm 1.

Initialize: experience replay memory $M$, mini-batch size $H$, the actor network $\pi_{\phi}$, the critic network $Q_{\theta}$ with random values, and create the target networks by setting $\theta^{\prime}\leftarrow\theta$ and $\phi^{\prime}\leftarrow\phi$.

Set: Set $E$ and $T$ as the maximum number of episodes and episode length, respectively.

for each episode $e:E{}$ do

      Initialize the environment state $s_{0}$, and the exploration noise $\xi$;

       for $t=1:T$ do

            Receive $s_{t}$ from the environment;

             Obtain $a_{t}=\pi_{\phi}(s_{t})+\xi$ from the actor network and re-shape it;

             Obtain $r_{t}$ based on equation (18);

             Observe the new state, $s_{t+1}$;

             Store transition $(s_{t},a_{t},r_{t},s_{t+1})$ into $M$;

            

       end for

      Randomly sample a $H$ mini-batch of transitions from $M$;

       Compute the target function $Y_{t}$ according to (21);

       Update the actor and critic networks using the Adam optimizer.

       Soft update the target actor and target critics with $\tau\in[0,1]$, as the soft update coefficient:

$$\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime},\quad\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}$$

end for

Algorithm 1 The RDPG Algorithm

VI Simulation Results

In this section, we provide numerical results illustrating how combining FA arrays with the proposed RDPG algorithm can improve OTA-FL learning performance. We consider the distance between users and the AP to be uniformly distributed in the range $[20,100]$ m, and the AoAs to be uniformly distributed over $[-\pi/2,\pi/2]$ rad. The parameters for the FA arrays are set with $X_{0}=0.5\lambda$ and $X=8\lambda$. The RDPG algorithm is configured with a learning rate of 0.0005, a replay buffer size of $10^{4}$, a batch size of 64, a soft update parameter of 0.001, and a discount factor of 0.9. For performance evaluation, we consider two comparisons. Initially, we evaluate the performance of FA relative to FPA with a predetermined location vector $\boldsymbol{x}=\left[\frac{X}{N+1},\ldots,\frac{NX}{N+1}\right]^{T}$. Subsequently, we compare the proposed RDPG algorithm with SAC [14] and DDPG [13], which are conventional DRL algorithms. Learning performance is evaluated by computing the average rewards over 100 episodes, which is determined at episode $e$ by employing the formula ${R_{\text{avg}}(e)}=\frac{1}{100}\sum_{i=e-100}^{e}R_{i}$, where $R_{i}$ signifies the mean reward of episode $i$.

Figure 1: Comparing DRL Algorithms in FA and FPA systems across (a) training episodes; (b) antenna numbers; and (c) client numbers.

Fig. 1 (a) demonstrates the convergence characteristics of different DRL algorithms, depicting the average rewards with solid curves and showing the standard deviations as shaded regions. The RDPG algorithm exhibits higher average rewards and lower variance compared to standard DRLs, demonstrating superior performance and improved stability in dynamic environments.

To assess the performance of the proposed algorithms across varying numbers of antennas, we kept the number of clients fixed and varied the antenna count in both the FA and FPA scenarios. As depicted in Fig. 1 (b), the average reward performance of all DRL methods improves with increasing $N$, although this improvement diminishes as $N$ continues to increase. Furthermore, due to the increased DoF provided by antenna adjustments in FA systems, FAs consistently outperform FPAs at all values $N$. Moreover, RDPG demonstrating superior performance over other DRL algorithms.

Fig. 1(c) provides a detailed comparison of the performance of FAs and the RDPG algorithm across varying numbers of users. It is evident from the results that as the number of users increases, there is a noticeable decrease in performance for both FA and FPA scenarios. This decline is attributed to the increased challenge of optimizing the beamforming vector and the antenna position vector of the AP in the presence of more dynamic users. Despite these challenges, FAs consistently outperform FPAs in all tested scenarios, highlighting the efficacy of FAs in enhancing OTA-FL system performance. Moreover, the RDPG algorithm consistently exhibits superior performance compared to other optimization methods in mitigating the adverse effects of dynamic user dynamics on system performance.

VII Conclusion

In this study, we investigated the integration of FAs into AP to improve the performance of OTA-FL systems. Our convergence analysis highlighted the significant impact of FA positions and the beamforming vector on the optimality gap. We addressed this issue with a non-convex optimization problem and proposed the RDPG algorithm for real-time optimization. Through simulations, we demonstrated that the OTA-FL system enhanced by FAs outperformed conventional FPA systems. Moreover, RDPG demonstrates superior performance and stability compared to existing methods, validating its effectiveness in dynamic environments.

Proof of Theorem III

In the $t$-th communication round, the global model update is formulated by integrating (5) into (4), as follows:

$$\boldsymbol{w}_{t+1} =\frac{1}{M}\sum_{m\in\mathcal{M}}\boldsymbol{w}_{m,t}+\frac{1}{M\sqrt{\eta_{t}}}\boldsymbol{m}_{t}^{H}\mathbf{z}_{t}$$ (22)

$$=\frac{1}{M}\sum_{m\in\mathcal{M}}(\boldsymbol{w}_{t}-\gamma\nabla{F}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{m}))+\frac{1}{M\sqrt{\eta_{t}}}\boldsymbol{m}_{t}^{H}\mathbf{z}_{t}$$

$$=\boldsymbol{w}_{t}-\gamma(\nabla{F}(\boldsymbol{w}_{t})-\mathbf{e}_{t}),$$

where, $\nabla{F}(\boldsymbol{w}_{t})=\frac{1}{M}\sum_{m\in\mathcal{M}}{\nabla{F}_{m}(\boldsymbol{w}_{t},\mathbf{\mathcal{D}}_{m})}$ is the global gradient and $\mathbf{e}_{t}=\frac{\boldsymbol{m}_{t}^{H}\mathbf{z}_{t}}{\gamma M\sqrt{\eta_{t}}}$ is the model aggregation error caused by wireless communication in the $t$-th communication round. By applying (22) in Assumption III, and setting $\eta=\frac{1}{l}$, after taking expectation, we have:

$$\mathbb{E}[{F}(\boldsymbol{w}_{t+1})] \leq\mathbb{E}[{F}(\boldsymbol{w}_{t})]-\frac{1}{2l}\|\nabla{F}(\boldsymbol{w}_{t})\|_{2}^{2}+\frac{1}{2l}\mathbb{E}[\|\mathbf{e}_{t}\|_{2}^{2}]$$ (23)

$$\leq\mathbb{E}[{F}(\boldsymbol{w}_{t})]-\frac{1}{2l}\|\nabla{F}(\boldsymbol{w}_{t})\|_{2}^{2}+\frac{l}{2}\frac{d\sigma^{2}\|\boldsymbol{m}_{t}^{H}\|_{2}^{2}}{\eta M^{2}}.$$

By applying the upper bound of $\eta_{t}$ based on (LABEL:eta) and employing Assumption III and III, then subtracting $F(\boldsymbol{w}^{*})$ from both sides of (23), we have:

$$\mathbb{E}[F(w_{t+1})]-F(w^{*})\leq(1-\frac{\mu}{l})(\mathbb{E}[F(w_{t})]-F(w^{*}))\\ +\frac{l\sigma^{2}\Gamma}{2M^{2}p_{\max}}\max_{k\in\mathcal{K}}\frac{\|m_{t}^{H}\|_{2}^{2}}{|m_{t}^{H}h_{m}[x]|^{2}}.$$ (24)

By recursively operating on (24) and using the definitions of $\Theta_{t}$ and $\psi$ in (12) and (13), the cumulative optimality gap is calculated as follows:

$$\mathbb{E}[F(\boldsymbol{w}_{T+1}) ]-F(\boldsymbol{w}^{*})\leq\psi(\mathbb{E}[F(\boldsymbol{w}_{T})]-F(\boldsymbol{w}^{*})+\Theta_{T}$$ (25)

$$\leq\psi(\psi(\mathbb{E}[F(\boldsymbol{w}_{T-1})]-F(\boldsymbol{w}^{*}))+\Theta_{T-1})+\Theta_{T})$$

$$\leq\ldots\leq\psi^{T}(\mathbb{E}[F(\boldsymbol{w}_{1})]-F(\boldsymbol{w}^{*}))+\sum_{t=1}^{T}\psi^{T-t}\Theta_{t}.$$

Proof of Theorem III is complete.

Appendix A References Section