Showing 1–50 of 86 results for author: Lin, S

SensEmo: Enabling Affective Learning through Real-time Emotion Recognition with Smartwatches

Authors: Kushan Choksi, Hongkai Chen, Karan Joshi, Sukrutha Jade, Shahriar Nirjon, Shan Lin

Abstract: Recent research has demonstrated the capability of physiological signals to infer both user emotional and attention responses. This presents an opportunity for leveraging widely available physiological sensors in smartwatches, to detect real-time emotional cues in users, such as stress and excitement. In this paper, we introduce SensEmo, a smartwatch-based system designed for affective learning. S… ▽ More Recent research has demonstrated the capability of physiological signals to infer both user emotional and attention responses. This presents an opportunity for leveraging widely available physiological sensors in smartwatches, to detect real-time emotional cues in users, such as stress and excitement. In this paper, we introduce SensEmo, a smartwatch-based system designed for affective learning. SensEmo utilizes multiple physiological sensor data, including heart rate and galvanic skin response, to recognize a student's motivation and concentration levels during class. This recognition is facilitated by a personalized emotion recognition model that predicts emotional states based on degrees of valence and arousal. With real-time emotion and attention feedback from students, we design a Markov decision process-based algorithm to enhance student learning effectiveness and experience by by offering suggestions to the teacher regarding teaching content and pacing. We evaluate SensEmo with 22 participants in real-world classroom environments. Evaluation results show that SensEmo recognizes student emotion with an average of 88.9% accuracy. More importantly, SensEmo assists students to achieve better online learning outcomes, e.g., an average of 40.0% higher grades in quizzes, over the traditional learning without student emotional feedback. △ Less

Submitted 13 July, 2024; originally announced July 2024.

Comments: 7 pages, 7 figures, 2 tables. IEEE MASS 2024

ACM Class: C.3.3; J.3.2; J.4.2

Stacked Intelligent Metasurfaces for Wireless Sensing and Communication: Applications and Challenges

Authors: Hao Liu, Jiancheng An, Xing Jia, Shining Lin, Xianghao Yao, Lu Gan, Bruno Clerckx, Chau Yuen, Mehdi Bennis, Mérouane Debbah

Abstract: The rapid advancement of wireless communication technologies has precipitated an unprecedented demand for high data rates, extremely low latency, and ubiquitous connectivity. In order to achieve these goals, stacked intelligent metasurfaces (SIM) has been developed as a novel solution to perform advanced signal processing tasks directly in the electromagnetic wave domain, thus achieving ultra-fast… ▽ More The rapid advancement of wireless communication technologies has precipitated an unprecedented demand for high data rates, extremely low latency, and ubiquitous connectivity. In order to achieve these goals, stacked intelligent metasurfaces (SIM) has been developed as a novel solution to perform advanced signal processing tasks directly in the electromagnetic wave domain, thus achieving ultra-fast computing speed and reducing hardware complexity. This article provides an overview of the SIM technology by discussing its hardware architectures, advantages, and potential applications for wireless sensing and communication. Specifically, we explore the utilization of SIMs in enabling wave-domain beamforming, channel modeling and estimation in SIM-assisted communication systems. Furthermore, we elaborate on the potential of utilizing a SIM to build a hybrid optical-electronic neural network (HOENN) and demonstrate its efficacy by examining two case studies: disaster monitoring and direction-of-arrival estimation. Finally, we identify key implementation challenges, including practical hardware imperfections, efficient SIM configuration for realizing wave-domain signal processing, and performance analysis to motivate future research on this important and far-reaching topic. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Comments: 8 pages, 5 figures, 1 table

PretVM: Predictable, Efficient Virtual Machine for Real-Time Concurrency

Authors: Shaokai Lin, Erling Jellum, Mirco Theile, Tassilo Tanneberger, Binqi Sun, Chadlia Jerad, Ruomu Xu, Guangyu Feng, Christian Menard, Marten Lohstroh, Jeronimo Castrillon, Sanjit Seshia, Edward Lee

Abstract: This paper introduces the Precision-Timed Virtual Machine (PretVM), an intermediate platform facilitating the execution of quasi-static schedules compiled from a subset of programs written in the Lingua Franca (LF) coordination language. The subset consists of those programs that in principle should have statically verifiable and predictable timing behavior. The PretVM provides a schedule with wel… ▽ More This paper introduces the Precision-Timed Virtual Machine (PretVM), an intermediate platform facilitating the execution of quasi-static schedules compiled from a subset of programs written in the Lingua Franca (LF) coordination language. The subset consists of those programs that in principle should have statically verifiable and predictable timing behavior. The PretVM provides a schedule with well-defined worst-case timing bounds. The PretVM provides a clean separation between application logic and coordination logic, yielding more analyzable program executions. Experiments compare the PretVM against the default (more dynamic) LF scheduler and show that it delivers time-accurate deterministic execution. △ Less

Submitted 25 June, 2024; v1 submitted 10 June, 2024; originally announced June 2024.

The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report

Authors: Bin Ren, Yawei Li, Nancy Mehta, Radu Timofte, Hongyuan Yu, Cheng Wan, Yuxin Hong, Bingnan Han, Zhuoyuan Wu, Yajun Zou, Yuqing Liu, Jizhe Li, Keji He, Chao Fan, Heng Zhang, Xiaolin Zhang, Xuanwu Yin, Kunlong Zuo, Bohao Liao, Peizhe Xia, Long Peng, Zhibo Du, Xin Di, Wangkai Li, Yang Wang , et al. (109 additional authors not shown)

Abstract: This paper provides a comprehensive review of the NTIRE 2024 challenge, focusing on efficient single-image super-resolution (ESR) solutions and their outcomes. The task of this challenge is to super-resolve an input image with a magnification factor of x4 based on pairs of low and corresponding high-resolution images. The primary objective is to develop networks that optimize various aspects such… ▽ More This paper provides a comprehensive review of the NTIRE 2024 challenge, focusing on efficient single-image super-resolution (ESR) solutions and their outcomes. The task of this challenge is to super-resolve an input image with a magnification factor of x4 based on pairs of low and corresponding high-resolution images. The primary objective is to develop networks that optimize various aspects such as runtime, parameters, and FLOPs, while still maintaining a peak signal-to-noise ratio (PSNR) of approximately 26.90 dB on the DIV2K_LSDIR_valid dataset and 26.99 dB on the DIV2K_LSDIR_test dataset. In addition, this challenge has 4 tracks including the main track (overall performance), sub-track 1 (runtime), sub-track 2 (FLOPs), and sub-track 3 (parameters). In the main track, all three metrics (ie runtime, FLOPs, and parameter count) were considered. The ranking of the main track is calculated based on a weighted sum-up of the scores of all other sub-tracks. In sub-track 1, the practical runtime performance of the submissions was evaluated, and the corresponding score was used to determine the ranking. In sub-track 2, the number of FLOPs was considered. The score calculated based on the corresponding FLOPs was used to determine the ranking. In sub-track 3, the number of parameters was considered. The score calculated based on the corresponding parameters was used to determine the ranking. RLFN is set as the baseline for efficiency measurement. The challenge had 262 registered participants, and 34 teams made valid submissions. They gauge the state-of-the-art in efficient single-image super-resolution. To facilitate the reproducibility of the challenge and enable other researchers to build upon these findings, the code and the pre-trained model of validated solutions are made publicly available at https://github.com/Amazingren/NTIRE2024_ESR/. △ Less

Submitted 25 June, 2024; v1 submitted 16 April, 2024; originally announced April 2024.

Comments: The report paper of NTIRE2024 Efficient Super-resolution, accepted by CVPRW2024

A Novel State-Centric Necessary Condition for Time-Optimal Control of Controllable Linear Systems Based on Augmented Switching Laws

Authors: Yunan Wang, Chuxiong Hu, Yujie Lin, Zeyang Li, Shize Lin, Suqin He

Abstract: Most existing necessary conditions for optimal control based on adjoining methods require both state information and costate information, yet the lack of costates for a given feasible trajectory in practice impedes the determination of optimality. This paper establishes a novel theoretical framework for time-optimal control of controllable linear systems, proposing the augmented switching law that… ▽ More Most existing necessary conditions for optimal control based on adjoining methods require both state information and costate information, yet the lack of costates for a given feasible trajectory in practice impedes the determination of optimality. This paper establishes a novel theoretical framework for time-optimal control of controllable linear systems, proposing the augmented switching law that represents the input control and the feasibility in a compact form. Given a feasible trajectory, the disturbed trajectory under the constraints of augmented switching law is guaranteed to be feasible, resulting in a novel state-centric necessary condition without dependence on costate information. A first order necessary condition is proposed that the Jacobian matrix of the augmented switching law is not full row rank, which also results in an approach to optimizing a given feasible trajectory further. The proposed necessary condition is applied to the chain-of-integrators systems with full box constraints, contributing to some conclusions challenging to reason by traditional costate-based necessary conditions. △ Less

Submitted 13 April, 2024; originally announced April 2024.

A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challenge

Authors: Ezequiel de la Rosa, Mauricio Reyes, Sook-Lei Liew, Alexandre Hutton, Roland Wiest, Johannes Kaesmacher, Uta Hanning, Arsany Hakim, Richard Zubal, Waldo Valenzuela, David Robben, Diana M. Sima, Vincenzo Anania, Arne Brys, James A. Meakin, Anne Mickan, Gabriel Broocks, Christian Heitkamp, Shengbo Gao, Kongming Liang, Ziji Zhang, Md Mahfuzur Rahman Siddiquee, Andriy Myronenko, Pooya Ashtari, Sabine Van Huffel , et al. (33 additional authors not shown)

Abstract: Diffusion-weighted MRI (DWI) is essential for stroke diagnosis, treatment decisions, and prognosis. However, image and disease variability hinder the development of generalizable AI algorithms with clinical value. We address this gap by presenting a novel ensemble algorithm derived from the 2022 Ischemic Stroke Lesion Segmentation (ISLES) challenge. ISLES'22 provided 400 patient scans with ischemi… ▽ More Diffusion-weighted MRI (DWI) is essential for stroke diagnosis, treatment decisions, and prognosis. However, image and disease variability hinder the development of generalizable AI algorithms with clinical value. We address this gap by presenting a novel ensemble algorithm derived from the 2022 Ischemic Stroke Lesion Segmentation (ISLES) challenge. ISLES'22 provided 400 patient scans with ischemic stroke from various medical centers, facilitating the development of a wide range of cutting-edge segmentation algorithms by the research community. Through collaboration with leading teams, we combined top-performing algorithms into an ensemble model that overcomes the limitations of individual solutions. Our ensemble model achieved superior ischemic lesion detection and segmentation accuracy on our internal test set compared to individual algorithms. This accuracy generalized well across diverse image and disease variables. Furthermore, the model excelled in extracting clinical biomarkers. Notably, in a Turing-like test, neuroradiologists consistently preferred the algorithm's segmentations over manual expert efforts, highlighting increased comprehensiveness and precision. Validation using a real-world external dataset (N=1686) confirmed the model's generalizability. The algorithm's outputs also demonstrated strong correlations with clinical scores (admission NIHSS and 90-day mRS) on par with or exceeding expert-derived results, underlining its clinical relevance. This study offers two key findings. First, we present an ensemble algorithm (https://github.com/Tabrisrei/ISLES22_Ensemble) that detects and segments ischemic stroke lesions on DWI across diverse scenarios on par with expert (neuro)radiologists. Second, we show the potential for biomedical challenge outputs to extend beyond the challenge's initial objectives, demonstrating their real-world clinical applicability. △ Less

Submitted 3 April, 2024; v1 submitted 28 March, 2024; originally announced March 2024.

Chattering Phenomena in Time-Optimal Control for High-Order Chain-of-Integrators Systems with Full State Constraints

Authors: Yunan Wang, Chuxiong Hu, Zeyang Li, Yujie Lin, Shize Lin, Suqin He

Abstract: Time-optimal control for high-order chain-of-integrators systems with full state constraints remains an open and challenging problem in the optimal control theory domain. The behaviors of optimal control in high-order problems lack precision characterization, even where the existence of the chattering phenomenon remains unknown and overlooked. This paper establishes a theoretical framework for cha… ▽ More Time-optimal control for high-order chain-of-integrators systems with full state constraints remains an open and challenging problem in the optimal control theory domain. The behaviors of optimal control in high-order problems lack precision characterization, even where the existence of the chattering phenomenon remains unknown and overlooked. This paper establishes a theoretical framework for chattering phenomena in the considered problem, providing novel findings on the uniqueness of state constraints inducing chattering, the upper bound on switching times in an unconstrained arc during chattering, and the convergence of states and costates to the chattering limit point. For the first time, this paper proves the existence of the chattering phenomenon in the considered problem. The chattering optimal control for 4th order problems with velocity constraints is precisely solved, providing an approach to plan strictly time-optimal snap-limited trajectories. Other cases of order $n\leq4$ are proved not to allow chattering. The conclusions correct the longstanding misconception in the industry regarding the time-optimality of S-shaped trajectories with minimal switching times. △ Less

Submitted 29 March, 2024; v1 submitted 26 March, 2024; originally announced March 2024.

HemoSet: The First Blood Segmentation Dataset for Automation of Hemostasis Management

Authors: Albert J. Miao, Shan Lin, Jingpei Lu, Florian Richter, Benjamin Ostrander, Emily K. Funk, Ryan K. Orosco, Michael C. Yip

Abstract: Hemorrhaging occurs in surgeries of all types, forcing surgeons to quickly adapt to the visual interference that results from blood rapidly filling the surgical field. Introducing automation into the crucial surgical task of hemostasis management would offload mental and physical tasks from the surgeon and surgical assistants while simultaneously increasing the efficiency and safety of the operati… ▽ More Hemorrhaging occurs in surgeries of all types, forcing surgeons to quickly adapt to the visual interference that results from blood rapidly filling the surgical field. Introducing automation into the crucial surgical task of hemostasis management would offload mental and physical tasks from the surgeon and surgical assistants while simultaneously increasing the efficiency and safety of the operation. The first step in automation of hemostasis management is detection of blood in the surgical field. To propel the development of blood detection algorithms in surgeries, we present HemoSet, the first blood segmentation dataset based on bleeding during a live animal robotic surgery. Our dataset features vessel hemorrhage scenarios where turbulent flow leads to abnormal pooling geometries in surgical fields. These pools are formed in conditions endemic to surgical procedures -- uneven heterogeneous tissue, under glossy lighting conditions and rapid tool movement. We benchmark several state-of-the-art segmentation models and provide insight into the difficulties specific to blood detection. We intend for HemoSet to spur development of autonomous blood suction tools by providing a platform for training and refining blood segmentation models, addressing the precision needed for such robotics. △ Less

Submitted 2 June, 2024; v1 submitted 24 March, 2024; originally announced March 2024.

Stacked Intelligent Metasurface Enabled LEO Satellite Communications Relying on Statistical CSI

Authors: Shining Lin, Jiancheng An, Lu Gan, Mérouane Debbah, Chau Yuen

Abstract: Low earth orbit (LEO) satellite communication systems have gained increasing attention as a crucial supplement to terrestrial wireless networks due to their extensive coverage area. This letter presents a novel system design for LEO satellite systems by leveraging stacked intelligent metasurface (SIM) technology. Specifically, the lightweight and energy-efficient SIM is mounted on a satellite to a… ▽ More Low earth orbit (LEO) satellite communication systems have gained increasing attention as a crucial supplement to terrestrial wireless networks due to their extensive coverage area. This letter presents a novel system design for LEO satellite systems by leveraging stacked intelligent metasurface (SIM) technology. Specifically, the lightweight and energy-efficient SIM is mounted on a satellite to achieve multiuser beamforming directly in the electromagnetic wave domain, which substantially reduces the processing delay and computational load of the satellite compared to the traditional digital beamforming scheme. To overcome the challenges of obtaining instantaneous channel state information (CSI) at the transmitter and maximize the system's performance, a joint power allocation and SIM phase shift optimization problem for maximizing the ergodic sum rate is formulated based on statistical CSI, and an alternating optimization (AO) algorithm is customized to solve it efficiently. Additionally, a user grouping method based on channel correlation and an antenna selection algorithm are proposed to further improve the system performance. Simulation results demonstrate the effectiveness of the proposed SIM-based LEO satellite system design and statistical CSI-based AO algorithm. △ Less

Submitted 9 March, 2024; originally announced March 2024.

Comments: 14 pages, 4 figures, accepted by IEEE WCL

Closing the AI generalization gap by adjusting for dermatology condition distribution differences across clinical settings

Authors: Rajeev V. Rikhye, Aaron Loh, Grace Eunhae Hong, Preeti Singh, Margaret Ann Smith, Vijaytha Muralidharan, Doris Wong, Rory Sayres, Michelle Phung, Nicolas Betancourt, Bradley Fong, Rachna Sahasrabudhe, Khoban Nasim, Alec Eschholz, Basil Mustafa, Jan Freyberg, Terry Spitz, Yossi Matias, Greg S. Corrado, Katherine Chou, Dale R. Webster, Peggy Bui, Yuan Liu, Yun Liu, Justin Ko , et al. (1 additional authors not shown)

Abstract: Recently, there has been great progress in the ability of artificial intelligence (AI) algorithms to classify dermatological conditions from clinical photographs. However, little is known about the robustness of these algorithms in real-world settings where several factors can lead to a loss of generalizability. Understanding and overcoming these limitations will permit the development of generali… ▽ More Recently, there has been great progress in the ability of artificial intelligence (AI) algorithms to classify dermatological conditions from clinical photographs. However, little is known about the robustness of these algorithms in real-world settings where several factors can lead to a loss of generalizability. Understanding and overcoming these limitations will permit the development of generalizable AI that can aid in the diagnosis of skin conditions across a variety of clinical settings. In this retrospective study, we demonstrate that differences in skin condition distribution, rather than in demographics or image capture mode are the main source of errors when an AI algorithm is evaluated on data from a previously unseen source. We demonstrate a series of steps to close this generalization gap, requiring progressively more information about the new source, ranging from the condition distribution to training data enriched for data less frequently seen during training. Our results also suggest comparable performance from end-to-end fine tuning versus fine tuning solely the classification layer on top of a frozen embedding model. Our approach can inform the adaptation of AI algorithms to new settings, based on the information and resources available. △ Less

Submitted 23 February, 2024; originally announced February 2024.

Dynamic Fault Characteristics Evaluation in Power Grid

Authors: Hao Pei, Si Lin, Chuanfu Li, Che Wang, Haoming Chen, Sizhe Li

Abstract: To enhance the intelligence degree in operation and maintenance, a novel method for fault detection in power grids is proposed. The proposed GNN-based approach first identifies fault nodes through a specialized feature extraction method coupled with a knowledge graph. By incorporating temporal data, the method leverages the status of nodes from preceding and subsequent time periods to help current… ▽ More To enhance the intelligence degree in operation and maintenance, a novel method for fault detection in power grids is proposed. The proposed GNN-based approach first identifies fault nodes through a specialized feature extraction method coupled with a knowledge graph. By incorporating temporal data, the method leverages the status of nodes from preceding and subsequent time periods to help current fault detection. To validate the effectiveness of the node features, a correlation analysis of the output features from each node was conducted. The results from experiments show that this method can accurately locate fault nodes in simulation scenarios with a remarkable accuracy. Additionally, the graph neural network based feature modeling allows for a qualitative examination of how faults spread across nodes, which provides valuable insights for analyzing fault nodes. △ Less

Submitted 27 January, 2024; v1 submitted 28 November, 2023; originally announced November 2023.

Time-Optimal Control for High-Order Chain-of-Integrators Systems with Full State Constraints and Arbitrary Terminal States (Extended Version)

Authors: Yunan Wang, Chuxiong Hu, Zeyang Li, Shize Lin, Suqin He, Yu Zhu

Abstract: Time-optimal control for high-order chain-of-integrators systems with full state constraints and arbitrarily given terminal states remains a challenging problem in the optimal control theory domain, yet to be resolved. To enhance further comprehension of the problem, this paper establishes a novel notation system and theoretical framework, providing the switching manifold for high-order problems i… ▽ More Time-optimal control for high-order chain-of-integrators systems with full state constraints and arbitrarily given terminal states remains a challenging problem in the optimal control theory domain, yet to be resolved. To enhance further comprehension of the problem, this paper establishes a novel notation system and theoretical framework, providing the switching manifold for high-order problems in the form of switching laws. Through deriving properties of switching laws regarding signs and dimension, this paper proposes a definite condition for time-optimal control. Guided by the developed theory, a trajectory planning method named the manifold-intercept method (MIM) is developed. The proposed MIM can plan time-optimal jerk-limited trajectories with full state constraints, and can also plan near-optimal non-chattering higher-order trajectories with negligible extra motion time compared to optimal profiles. Numerical results indicate that the proposed MIM outperforms all baselines in computational time, computational accuracy, and trajectory quality by a large gap. △ Less

Submitted 28 March, 2024; v1 submitted 12 November, 2023; originally announced November 2023.

AI-Enabled Unmanned Vehicle-Assisted Reconfigurable Intelligent Surfaces: Deployment, Prototyping, Experiments, and Opportunities

Authors: Li-Hsiang Shen, Kai-Ten Feng, Ta-Sung Lee, Yuan-Chun Lin, Shih-Cheng Lin, Chia-Chan Chang, Sheng-Fuh Chang

Abstract: The requirement of wireless data demands is increasingly high as the sixth-generation (6G) technology evolves. Reconfigurable intelligent surface (RIS) is promisingly deemed to be one of 6G techniques for extending service coverage, reducing power consumption, and enhancing spectral efficiency. In this article, we have provided some fundamentals of RIS deployment in theory and hardware perspective… ▽ More The requirement of wireless data demands is increasingly high as the sixth-generation (6G) technology evolves. Reconfigurable intelligent surface (RIS) is promisingly deemed to be one of 6G techniques for extending service coverage, reducing power consumption, and enhancing spectral efficiency. In this article, we have provided some fundamentals of RIS deployment in theory and hardware perspectives as well as utilization of artificial intelligence (AI) and machine learning. We conducted an intelligent deployment of RIS (i-Dris) prototype, including dual-band auto-guided vehicle (AGV) assisted RISs associated with an mmWave base station (BS) and a receiver. The RISs are deployed on the AGV with configured incident/reflection angles. While, both the mmWave BS and receiver are associated with an edge server monitoring downlink packets for obtaining system throughput. We have designed a federated multi-agent reinforcement learning scheme associated with several AGV-RIS agents and sub-agents per AGV-RIS consisting of the deployment of position, height, orientation and elevation angles. The experimental results presented the stationary measurement in different aspects and scenarios. The i-Dris can reach up to 980 Mbps transmission throughput under a bandwidth of 100 MHz with comparably low complexity as well as rapid deployment, which outperforms the other existing works. At last, we highlight some opportunities and future issues in leveraging RIS-empowered wireless communication networks. △ Less

Submitted 6 November, 2023; originally announced November 2023.

FPM-INR: Fourier ptychographic microscopy image stack reconstruction using implicit neural representations

Authors: Haowen Zhou, Brandon Y. Feng, Haiyun Guo, Siyu Lin, Mingshu Liang, Christopher A. Metzler, Changhuei Yang

Abstract: Image stacks provide invaluable 3D information in various biological and pathological imaging applications. Fourier ptychographic microscopy (FPM) enables reconstructing high-resolution, wide field-of-view image stacks without z-stack scanning, thus significantly accelerating image acquisition. However, existing FPM methods take tens of minutes to reconstruct and gigabytes of memory to store a hig… ▽ More Image stacks provide invaluable 3D information in various biological and pathological imaging applications. Fourier ptychographic microscopy (FPM) enables reconstructing high-resolution, wide field-of-view image stacks without z-stack scanning, thus significantly accelerating image acquisition. However, existing FPM methods take tens of minutes to reconstruct and gigabytes of memory to store a high-resolution volumetric scene, impeding fast gigapixel-scale remote digital pathology. While deep learning approaches have been explored to address this challenge, existing methods poorly generalize to novel datasets and can produce unreliable hallucinations. This work presents FPM-INR, a compact and efficient framework that integrates physics-based optical models with implicit neural representations (INR) to represent and reconstruct FPM image stacks. FPM-INR is agnostic to system design or sample types and does not require external training data. In our demonstrated experiments, FPM-INR substantially outperforms traditional FPM algorithms with up to a 25-fold increase in speed and an 80-fold reduction in memory usage for continuous image stack representations. △ Less

Submitted 31 October, 2023; v1 submitted 27 October, 2023; originally announced October 2023.

Comments: Project Page: https://hwzhou2020.github.io/FPM-INR-Web/

Real-to-Sim Deformable Object Manipulation: Optimizing Physics Models with Residual Mappings for Robotic Surgery

Authors: Xiao Liang, Fei Liu, Yutong Zhang, Yuelei Li, Shan Lin, Michael Yip

Abstract: Accurate deformable object manipulation (DOM) is essential for achieving autonomy in robotic surgery, where soft tissues are being displaced, stretched, and dissected. Many DOM methods can be powered by simulation, which ensures realistic deformation by adhering to the governing physical constraints and allowing for model prediction and control. However, real soft objects in robotic surgery, such… ▽ More Accurate deformable object manipulation (DOM) is essential for achieving autonomy in robotic surgery, where soft tissues are being displaced, stretched, and dissected. Many DOM methods can be powered by simulation, which ensures realistic deformation by adhering to the governing physical constraints and allowing for model prediction and control. However, real soft objects in robotic surgery, such as membranes and soft tissues, have complex, anisotropic physical parameters that a simulation with simple initialization from cameras may not fully capture. To use the simulation techniques in real surgical tasks, the "real-to-sim" gap needs to be properly compensated. In this work, we propose an online, adaptive parameter tuning approach for simulation optimization that (1) bridges the real-to-sim gap between a physics simulation and observations obtained 3D perceptions through estimating a residual mapping and (2) optimizes its stiffness parameters online. Our method ensures a small residual gap between the simulation and observation and improves the simulation's predictive capabilities. The effectiveness of the proposed mechanism is evaluated in the manipulation of both a thin-shell and volumetric tissue, representative of most tissue scenarios. This work contributes to the advancement of simulation-based deformable tissue manipulation and holds potential for improving surgical autonomy. △ Less

Submitted 29 May, 2024; v1 submitted 20 September, 2023; originally announced September 2023.

Exploring Sentence Type Effects on the Lombard Effect and Intelligibility Enhancement: A Comparative Study of Natural and Grid Sentences

Authors: Hongyang Chen, Yuhong Yang, Zhongyuan Wang, Weiping Tu, Haojun Ai, Song Lin

Abstract: This study explores how sentence types affect the Lombard effect and intelligibility enhancement, focusing on comparisons between natural and grid sentences. Using the Lombard Chinese-TIMIT (LCT) corpus and the Enhanced MAndarin Lombard Grid (EMALG) corpus, we analyze changes in phonetic and acoustic features across different noise levels. Our results show that grid sentences produce more pronounc… ▽ More This study explores how sentence types affect the Lombard effect and intelligibility enhancement, focusing on comparisons between natural and grid sentences. Using the Lombard Chinese-TIMIT (LCT) corpus and the Enhanced MAndarin Lombard Grid (EMALG) corpus, we analyze changes in phonetic and acoustic features across different noise levels. Our results show that grid sentences produce more pronounced Lombard effects than natural sentences. Then, we develop and test a normal-to-Lombard conversion model, trained separately on LCT and EMALG corpora. Through subjective and objective evaluations, natural sentences are superior in maintaining speech quality in intelligibility enhancement. In contrast, grid sentences could provide superior intelligibility due to the more pronounced Lombard effect. This study provides a valuable perspective on enhancing speech communication in noisy environments. △ Less

Submitted 8 July, 2024; v1 submitted 19 September, 2023; originally announced September 2023.

Mandarin Lombard Flavor Classification

Authors: Qingmu Liu, Yuhong Yang, Baifeng Li, Hongyang Chen, Weiping Tu, Song Lin

Abstract: The Lombard effect refers to individuals' unconscious modulation of vocal effort in response to variations in the ambient noise levels, intending to enhance speech intelligibility. The impact of different decibel levels and types of background noise on Lombard effects remains unclear. Building upon the characteristic of Lombard speech that individuals adjust their speech to improve intelligibility… ▽ More The Lombard effect refers to individuals' unconscious modulation of vocal effort in response to variations in the ambient noise levels, intending to enhance speech intelligibility. The impact of different decibel levels and types of background noise on Lombard effects remains unclear. Building upon the characteristic of Lombard speech that individuals adjust their speech to improve intelligibility dynamically based on the self-feedback speech, we propose a flavor classification approach for the Lombard effect. We first collected Mandarin Lombard speech under different noise conditions, then simulated self-feedback speech, and ultimately conducted the statistical test on the word correct rate. We found that both SSN and babble noise types result in four distinct categories of Mandarin Lombard speech in the range of 30 to 80 dBA with different transition points. △ Less

Submitted 14 September, 2023; originally announced September 2023.

EMALG: An Enhanced Mandarin Lombard Grid Corpus with Meaningful Sentences

Authors: Baifeng Li, Qingmu Liu, Yuhong Yang, Hongyang Chen, Weiping Tu, Song Lin

Abstract: This study investigates the Lombard effect, where individuals adapt their speech in noisy environments. We introduce an enhanced Mandarin Lombard grid (EMALG) corpus with meaningful sentences , enhancing the Mandarin Lombard grid (MALG) corpus. EMALG features 34 speakers and improves recording setups, addressing challenges faced by MALG with nonsense sentences. Our findings reveal that in Mandarin… ▽ More This study investigates the Lombard effect, where individuals adapt their speech in noisy environments. We introduce an enhanced Mandarin Lombard grid (EMALG) corpus with meaningful sentences , enhancing the Mandarin Lombard grid (MALG) corpus. EMALG features 34 speakers and improves recording setups, addressing challenges faced by MALG with nonsense sentences. Our findings reveal that in Mandarin, meaningful sentences are more effective in enhancing the Lombard effect. Additionally, we uncover that female exhibit a more pronounced Lombard effect than male when uttering meaningful sentences. Moreover, our results reaffirm the consistency in the Lombard effect comparison between English and Mandarin found in previous research. △ Less

Submitted 9 January, 2024; v1 submitted 13 September, 2023; originally announced September 2023.

ORRN: An ODE-based Recursive Registration Network for Deformable Respiratory Motion Estimation with Lung 4DCT Images

Authors: Xiao Liang, Shan Lin, Fei Liu, Dimitri Schreiber, Michael Yip

Abstract: Deformable Image Registration (DIR) plays a significant role in quantifying deformation in medical data. Recent Deep Learning methods have shown promising accuracy and speedup for registering a pair of medical images. However, in 4D (3D + time) medical data, organ motion, such as respiratory motion and heart beating, can not be effectively modeled by pair-wise methods as they were optimized for im… ▽ More Deformable Image Registration (DIR) plays a significant role in quantifying deformation in medical data. Recent Deep Learning methods have shown promising accuracy and speedup for registering a pair of medical images. However, in 4D (3D + time) medical data, organ motion, such as respiratory motion and heart beating, can not be effectively modeled by pair-wise methods as they were optimized for image pairs but did not consider the organ motion patterns necessary when considering 4D data. This paper presents ORRN, an Ordinary Differential Equations (ODE)-based recursive image registration network. Our network learns to estimate time-varying voxel velocities for an ODE that models deformation in 4D image data. It adopts a recursive registration strategy to progressively estimate a deformation field through ODE integration of voxel velocities. We evaluate the proposed method on two publicly available lung 4DCT datasets, DIRLab and CREATIS, for two tasks: 1) registering all images to the extreme inhale image for 3D+t deformation tracking and 2) registering extreme exhale to inhale phase images. Our method outperforms other learning-based methods in both tasks, producing the smallest Target Registration Error of 1.24mm and 1.26mm, respectively. Additionally, it produces less than 0.001\% unrealistic image folding, and the computation speed is less than 1 second for each CT volume. ORRN demonstrates promising registration accuracy, deformation plausibility, and computation efficiency on group-wise and pair-wise registration tasks. It has significant implications in enabling fast and accurate respiratory motion estimation for treatment planning in radiation therapy or robot motion planning in thoracic needle insertion. △ Less

Submitted 25 May, 2023; v1 submitted 23 May, 2023; originally announced May 2023.

Comments: Accepted by IEEE Transactions on Biomedical Engineering

An Ensemble Learning Approach for Exercise Detection in Type 1 Diabetes Patients

Authors: Ke Ma, Hongkai Chen, Shan Lin

Abstract: Type 1 diabetes is a serious disease in which individuals are unable to regulate their blood glucose levels, leading to various medical complications. Artificial pancreas (AP) systems have been developed as a solution for type 1 diabetic patients to mimic the behavior of the pancreas and regulate blood glucose levels. However, current AP systems lack detection capabilities for exercise-induced glu… ▽ More Type 1 diabetes is a serious disease in which individuals are unable to regulate their blood glucose levels, leading to various medical complications. Artificial pancreas (AP) systems have been developed as a solution for type 1 diabetic patients to mimic the behavior of the pancreas and regulate blood glucose levels. However, current AP systems lack detection capabilities for exercise-induced glucose intake, which can last up to 4 to 8 hours. This incapability can lead to hypoglycemia, which if left untreated, could have serious consequences, including death. Existing exercise detection methods are either limited to single sensor data or use inaccurate models for exercise detection, making them less effective in practice. In this work, we propose an ensemble learning framework that combines a data-driven physiological model and a Siamese network to leverage multiple physiological signal streams for exercise detection with high accuracy. To evaluate the effectiveness of our proposed approach, we utilized a public dataset with 12 diabetic patients collected from an 8-week clinical trial. Our approach achieves a true positive rate for exercise detection of 86.4% and a true negative rate of 99.1%, outperforming state-of-the-art solutions. △ Less

Submitted 11 May, 2023; originally announced May 2023.

Comments: 10 pages, 7 figures, 2 tables

MSC Class: 68T07 (Primary) 34A05 (Secondary) ACM Class: J.3

WiRiS: Transformer for RIS-Assisted Device-Free Sensing for Joint People Counting and Localization using Wi-Fi CSI

Authors: Wei-Yu Chung, Li-Hsiang Shen, Kai-Ten Feng, Yuan-Chun Lin, Shih-Cheng Lin, Sheng-Fuh Chang

Abstract: Channel State Information (CSI) is widely adopted as a feature for indoor localization. Taking advantage of the abundant information from the CSI, people can be accurately sensed even without equipped devices. However, the positioning error increases severely in non-line-of-sight (NLoS) regions. Reconfigurable intelligent surface (RIS) has been introduced to improve signal coverage in NLoS areas,… ▽ More Channel State Information (CSI) is widely adopted as a feature for indoor localization. Taking advantage of the abundant information from the CSI, people can be accurately sensed even without equipped devices. However, the positioning error increases severely in non-line-of-sight (NLoS) regions. Reconfigurable intelligent surface (RIS) has been introduced to improve signal coverage in NLoS areas, which can re-direct and enhance reflective signals with massive meta-material elements. In this paper, we have proposed a Transformer-based RIS-assisted device-free sensing for joint people counting and localization (WiRiS) system to precisely predict the number of people and their corresponding locations through configuring RIS. A series of predefined RIS beams is employed to create inputs of fingerprinting CSI features as sequence-to-sequence learning database for Transformer. We have evaluated the performance of proposed WiRiS system in both ray-tracing simulators and experiments. Both simulation and real-world experiments demonstrate that people counting accuracy exceeds 90\%, and the localization error can achieve the centimeter-level, which outperforms the existing benchmarks without employment of RIS. △ Less

Submitted 9 November, 2023; v1 submitted 25 March, 2023; originally announced April 2023.

Multi-Channel Attentive Feature Fusion for Radio Frequency Fingerprinting

Authors: Yuan Zeng, Yi Gong, Jiawei Liu, Shangao Lin, Zidong Han, Ruoxiao Cao, Kaibin Huang, Khaled Ben Letaief

Abstract: Radio frequency fingerprinting (RFF) is a promising device authentication technique for securing the Internet of things. It exploits the intrinsic and unique hardware impairments of the transmitters for RF device identification. In real-world communication systems, hardware impairments across transmitters are subtle, which are difficult to model explicitly. Recently, due to the superior performanc… ▽ More Radio frequency fingerprinting (RFF) is a promising device authentication technique for securing the Internet of things. It exploits the intrinsic and unique hardware impairments of the transmitters for RF device identification. In real-world communication systems, hardware impairments across transmitters are subtle, which are difficult to model explicitly. Recently, due to the superior performance of deep learning (DL)-based classification models on real-world datasets, DL networks have been explored for RFF. Most existing DL-based RFF models use a single representation of radio signals as the input. Multi-channel input model can leverage information from different representations of radio signals and improve the identification accuracy of the RF fingerprint. In this work, we propose a novel multi-channel attentive feature fusion (McAFF) method for RFF. It utilizes multi-channel neural features extracted from multiple representations of radio signals, including IQ samples, carrier frequency offset, fast Fourier transform coefficients and short-time Fourier transform coefficients, for better RF fingerprint identification. The features extracted from different channels are fused adaptively using a shared attention module, where the weights of neural features from multiple channels are learned during training the McAFF model. In addition, we design a signal identification module using a convolution-based ResNeXt block to map the fused features to device identities. To evaluate the identification performance of the proposed method, we construct a WiFi dataset, named WFDI, using commercial WiFi end-devices as the transmitters and a Universal Software Radio Peripheral (USRP) as the receiver. ... △ Less

Submitted 23 June, 2023; v1 submitted 19 March, 2023; originally announced March 2023.

Ultrafast CMOS image sensors and data-enabled super-resolution for multimodal radiographic imaging and tomography

Authors: Xin Yue, Shanny Lin, Wenting Li, Bradley T. Wolfe, Steven Clayton, Mark Makela, C. L. Morris, Simon Spannagel, Erik Ramberg, Juan Estrada, Hao Zhu, Jifeng Liu, Eric R. Fossum, Zhehui Wang

Abstract: We summarize recent progress in ultrafast Complementary Metal Oxide Semiconductor (CMOS) image sensor development and the application of neural networks for post-processing of CMOS and charge-coupled device (CCD) image data to achieve sub-pixel resolution (thus $super$-$resolution$). The combination of novel CMOS pixel designs and data-enabled image post-processing provides a promising path toward… ▽ More We summarize recent progress in ultrafast Complementary Metal Oxide Semiconductor (CMOS) image sensor development and the application of neural networks for post-processing of CMOS and charge-coupled device (CCD) image data to achieve sub-pixel resolution (thus $super$-$resolution$). The combination of novel CMOS pixel designs and data-enabled image post-processing provides a promising path towards ultrafast high-resolution multi-modal radiographic imaging and tomography applications. △ Less

Submitted 27 January, 2023; originally announced January 2023.

Comments: 12 pages, 10 figures

Report number: Los Alamos National Laboratory report number LA-UR-23-20744

Journal ref: Proceedings of Science ; Vol.420, p.041, 8 May 2023

CarFi: Rider Localization Using Wi-Fi CSI

Authors: Sirajum Munir, Hongkai Chen, Shiwei Fang, Mahathir Monjur, Shan Lin, Shahriar Nirjon

Abstract: With the rise of hailing services, people are increasingly relying on shared mobility (e.g., Uber, Lyft) drivers to pick up for transportation. However, such drivers and riders have difficulties finding each other in urban areas as GPS signals get blocked by skyscrapers, in crowded environments (e.g., in stadiums, airports, and bars), at night, and in bad weather. It wastes their time, creates a b… ▽ More With the rise of hailing services, people are increasingly relying on shared mobility (e.g., Uber, Lyft) drivers to pick up for transportation. However, such drivers and riders have difficulties finding each other in urban areas as GPS signals get blocked by skyscrapers, in crowded environments (e.g., in stadiums, airports, and bars), at night, and in bad weather. It wastes their time, creates a bad user experience, and causes more CO2 emissions due to idle driving. In this work, we explore the potential of Wi-Fi to help drivers to determine the street side of the riders. Our proposed system is called CarFi that uses Wi-Fi CSI from two antennas placed inside a moving vehicle and a data-driven technique to determine the street side of the rider. By collecting real-world data in realistic and challenging settings by blocking the signal with other people and other parked cars, we see that CarFi is 95.44% accurate in rider-side determination in both line of sight (LoS) and non-line of sight (nLoS) conditions, and can be run on an embedded GPU in real-time. △ Less

Submitted 21 December, 2022; originally announced January 2023.

ACM Class: C.3

DCS-RISR: Dynamic Channel Splitting for Efficient Real-world Image Super-Resolution

Authors: Junbo Qiao, Shaohui Lin, Yunlun Zhang, Wei Li, Jie Hu, Gaoqi He, Changbo Wang, Lizhuang Ma

Abstract: Real-world image super-resolution (RISR) has received increased focus for improving the quality of SR images under unknown complex degradation. Existing methods rely on the heavy SR models to enhance low-resolution (LR) images of different degradation levels, which significantly restricts their practical deployments on resource-limited devices. In this paper, we propose a novel Dynamic Channel Spl… ▽ More Real-world image super-resolution (RISR) has received increased focus for improving the quality of SR images under unknown complex degradation. Existing methods rely on the heavy SR models to enhance low-resolution (LR) images of different degradation levels, which significantly restricts their practical deployments on resource-limited devices. In this paper, we propose a novel Dynamic Channel Splitting scheme for efficient Real-world Image Super-Resolution, termed DCS-RISR. Specifically, we first introduce the light degradation prediction network to regress the degradation vector to simulate the real-world degradations, upon which the channel splitting vector is generated as the input for an efficient SR model. Then, a learnable octave convolution block is proposed to adaptively decide the channel splitting scale for low- and high-frequency features at each block, reducing computation overhead and memory cost by offering the large scale to low-frequency features and the small scale to the high ones. To further improve the RISR performance, Non-local regularization is employed to supplement the knowledge of patches from LR and HR subspace with free-computation inference. Extensive experiments demonstrate the effectiveness of DCS-RISR on different benchmark datasets. Our DCS-RISR not only achieves the best trade-off between computation/parameter and PSNR/SSIM metric, and also effectively handles real-world images with different degradation levels. △ Less

Submitted 1 January, 2023; v1 submitted 14 December, 2022; originally announced December 2022.

An STL-based Approach to Resilient Control for Cyber-Physical Systems

Authors: Hongkai Chen, Scott A. Smolka, Nicola Paoletti, Shan Lin

Abstract: We present ResilienC, a framework for resilient control of Cyber-Physical Systems subject to STL-based requirements. ResilienC utilizes a recently developed formalism for specifying CPS resiliency in terms of sets of $(\mathit{rec},\mathit{dur})$ real-valued pairs, where $\mathit{rec}$ represents the system's capability to rapidly recover from a property violation (recoverability), and… ▽ More We present ResilienC, a framework for resilient control of Cyber-Physical Systems subject to STL-based requirements. ResilienC utilizes a recently developed formalism for specifying CPS resiliency in terms of sets of $(\mathit{rec},\mathit{dur})$ real-valued pairs, where $\mathit{rec}$ represents the system's capability to rapidly recover from a property violation (recoverability), and $\mathit{dur}$ is reflective of its ability to avoid violations post-recovery (durability). We define the resilient STL control problem as one of multi-objective optimization, where the recoverability and durability of the desired STL specification are maximized. When neither objective is prioritized over the other, the solution to the problem is a set of Pareto-optimal system trajectories. We present a precise solution method to the resilient STL control problem using a mixed-integer linear programming encoding and an a posteriori $ε$-constraint approach for efficiently retrieving the complete set of optimally resilient solutions. In ResilienC, at each time-step, the optimal control action selected from the set of Pareto-optimal solutions by a Decision Maker strategy realizes a form of Model Predictive Control. We demonstrate the practical utility of the ResilienC framework on two significant case studies: autonomous vehicle lane keeping and deadline-driven, multi-region package delivery. △ Less

Submitted 4 November, 2022; originally announced November 2022.

Comments: 11 pages, 6 figures

Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking

Authors: Shan Lin, Albert J. Miao, Jingpei Lu, Shunkai Yu, Zih-Yun Chiu, Florian Richter, Michael C. Yip

Abstract: Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensiv… ▽ More Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical perception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting downstream tasks like surgical navigation. The proposed framework is demonstrated on challenging endoscopic data with deforming tissue, showing its advantages over our baseline and several other state-of the-art approaches. Our code and dataset are available at https://github.com/ucsdarclab/Python-SuPer. △ Less

Submitted 20 February, 2023; v1 submitted 29 October, 2022; originally announced October 2022.

Comments: IEEE International Conference on Robotics and Automation (ICRA) 2023

WakeUpNet: A Mobile-Transformer based Framework for End-to-End Streaming Voice Trigger

Authors: Zixing Zhang, Thorin Farnsworth, Senling Lin, Salah Karout

Abstract: End-to-end models have gradually become the main technical stream for voice trigger, aiming to achieve an utmost prediction accuracy but with a small footprint. In present paper, we propose an end-to-end voice trigger framework, namely WakeupNet, which is basically structured on a Transformer encoder. The purpose of this framework is to explore the context-capturing capability of Transformer, as s… ▽ More End-to-end models have gradually become the main technical stream for voice trigger, aiming to achieve an utmost prediction accuracy but with a small footprint. In present paper, we propose an end-to-end voice trigger framework, namely WakeupNet, which is basically structured on a Transformer encoder. The purpose of this framework is to explore the context-capturing capability of Transformer, as sequential information is vital for wakeup-word detection. However, the conventional Transformer encoder is too large to fit our task. To address this issue, we introduce different model compression approaches to shrink the vanilla one into a tiny one, called mobile-Transformer. To evaluate the performance of mobile-Transformer, we conduct extensive experiments on a large public-available dataset HiMia. The obtained results indicate that introduced mobile-Transformer significantly outperforms other frequently used models for voice trigger in both clean and noisy scenarios. △ Less

Submitted 6 October, 2022; originally announced October 2022.

Mind Reader: Reconstructing complex images from brain activities

Authors: Sikun Lin, Thomas Sprague, Ambuj K Singh

Abstract: Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our… ▽ More Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our work aims to reconstruct image stimuli that are rich in semantics, closer to everyday scenes, and can reveal more perspectives. However, data scarcity of fMRI datasets is the main obstacle to applying state-of-the-art deep learning models to this problem. We find that incorporating an additional text modality is beneficial for the reconstruction problem compared to directly translating brain signals to images. Therefore, the modalities involved in our method are: (i) voxel-level fMRI signals, (ii) observed images that trigger the brain signals, and (iii) textual description of the images. To further address data scarcity, we leverage an aligned vision-language latent space pre-trained on massive datasets. Instead of training models from scratch to find a latent space shared by the three modalities, we encode fMRI signals into this pre-aligned latent space. Then, conditioned on embeddings in this space, we reconstruct images with a generative model. The reconstructed images from our pipeline balance both naturalness and fidelity: they are photo-realistic and capture the ground truth image contents well. △ Less

Submitted 30 September, 2022; originally announced October 2022.

Hierarchical Reinforcement Learning Based Video Semantic Coding for Segmentation

Authors: Guangqi Xie, Xin Li, Shiqi Lin, Li Zhang, Kai Zhang, Yue Li, Zhibo Chen

Abstract: The rapid development of intelligent tasks, e.g., segmentation, detection, classification, etc, has brought an urgent need for semantic compression, which aims to reduce the compression cost while maintaining the original semantic information. However, it is impractical to directly integrate the semantic metric into the traditional codecs since they cannot be optimized in an end-to-end manner. To… ▽ More The rapid development of intelligent tasks, e.g., segmentation, detection, classification, etc, has brought an urgent need for semantic compression, which aims to reduce the compression cost while maintaining the original semantic information. However, it is impractical to directly integrate the semantic metric into the traditional codecs since they cannot be optimized in an end-to-end manner. To solve this problem, some pioneering works have applied reinforcement learning to implement image-wise semantic compression. Nevertheless, video semantic compression has not been explored since its complex reference architectures and compression modes. In this paper, we take a step forward to video semantic compression and propose the Hierarchical Reinforcement Learning based task-driven Video Semantic Coding, named as HRLVSC. Specifically, to simplify the complex mode decision of video semantic coding, we divided the action space into frame-level and CTU-level spaces in a hierarchical manner, and then explore the best mode selection for them progressively with the cooperation of frame-level and CTU-level agents. Moreover, since the modes of video semantic coding will exponentially increase with the number of frames in a Group of Pictures (GOP), we carefully investigate the effects of different mode selections for video semantic coding and design a simple but effective mode simplification strategy for it. We have validated our HRLVSC on the video segmentation task with HEVC reference software HM16.19. Extensive experimental results demonstrated that our HRLVSC can achieve over 39% BD-rate saving for video semantic coding under the Low Delay P configuration. △ Less

Submitted 24 August, 2022; originally announced August 2022.

Comments: Accepted by VCIP2022

FAIVConf: Face enhancement for AI-based Video Conference with Low Bit-rate

Authors: Zhengang Li, Sheng Lin, Shan Liu, Songnan Li, Xue Lin, Wei Wang, Wei Jiang

Abstract: Recently, high-quality video conferencing with fewer transmission bits has become a very hot and challenging problem. We propose FAIVConf, a specially designed video compression framework for video conferencing, based on the effective neural human face generation techniques. FAIVConf brings together several designs to improve the system robustness in real video conference scenarios: face-swapping… ▽ More Recently, high-quality video conferencing with fewer transmission bits has become a very hot and challenging problem. We propose FAIVConf, a specially designed video compression framework for video conferencing, based on the effective neural human face generation techniques. FAIVConf brings together several designs to improve the system robustness in real video conference scenarios: face-swapping to avoid artifacts in background animation; facial blurring to decrease transmission bit-rate and maintain the quality of extracted facial landmarks; and dynamic source update for face view interpolation to accommodate a large range of head poses. Our method achieves a significant bit-rate reduction in the video conference and gives much better visual quality under the same bit-rate compared with H.264 and H.265 coding schemes. △ Less

Submitted 8 July, 2022; originally announced July 2022.

Comments: ICME 2022

6G-AUTOR: Autonomic CSI-Free Transceiver via Realtime On-Device Signal Analytics

Authors: Shih-Chun Lin, Chia-Hung Lin, K V S Rohit, Liang C Chu

Abstract: Next-generation wireless systems aim at fulfilling diverse application requirements but fundamentally rely on point-to-point transmission qualities. Aligning with recent AI-enabled wireless implementations, this paper introduces autonomic radios, 6G-AUTOR, that leverage novel algorithm-hardware separation platforms, softwarization of transmission (TX) and reception (RX) operations, and automatic r… ▽ More Next-generation wireless systems aim at fulfilling diverse application requirements but fundamentally rely on point-to-point transmission qualities. Aligning with recent AI-enabled wireless implementations, this paper introduces autonomic radios, 6G-AUTOR, that leverage novel algorithm-hardware separation platforms, softwarization of transmission (TX) and reception (RX) operations, and automatic reconfiguration of RF frontends, to support link performance and resilience. As a comprehensive transceiver solution, our design encompasses several ML-driven models, each enhancing a specific aspect of either TX or RX, leading to robust transceiver operation under tight constraints of future wireless systems. A data-driven radio management module was developed via deep Q-networks to support fast-reconfiguration of TX resource blocks (RB) and proactive multi-agent access. Also, a ResNet-inspired fast-beamforming solution was employed to enable robust communication to multiple receivers over the same RB, which has potential applications in realisation of cell-free infrastructures. As a receiver the system was equipped with a capability of ultra-broadband spectrum recognition. Apart from this, a fundamental tool - automatic modulation classification (AMC) which involves a complex correntropy extraction, followed by a convolutional neural network (CNN)-based classification, and a deep learning-based LDPC decoder were added to improve the reception quality and radio performance. Simulations of individual algorithms demonstrate that under appropriate training, each of the corresponding radio functions have either outperformed or have performed on-par with the benchmark solutions. △ Less

Submitted 7 June, 2022; originally announced June 2022.

Comments: Preprint submitted to Ad Hoc Networks

Zero-Touch Network on Industrial IoT: An End-to-End Machine Learning Approach

Authors: Shih-Chun Lin, Chia-Hung Lin, Wei-Chi Chen

Abstract: Industry 4.0-enabled smart factory is expected to realize the next revolution for manufacturers. Although artificial intelligence (AI) technologies have improved productivity, current use cases belong to small-scale and single-task operations. To unbound the potential of smart factory, this paper develops zero-touch network systems for intelligent manufacturing and facilitates distributed AI appli… ▽ More Industry 4.0-enabled smart factory is expected to realize the next revolution for manufacturers. Although artificial intelligence (AI) technologies have improved productivity, current use cases belong to small-scale and single-task operations. To unbound the potential of smart factory, this paper develops zero-touch network systems for intelligent manufacturing and facilitates distributed AI applications in both training and inferring stages in a large-scale manner. The open radio access network (O-RAN) architecture is first introduced for the zero-touch platform to enable globally controlling communications and computation infrastructure capability in the field. The designed serverless framework allows intelligent and efficient learning assignments and resource allocations. Hence, requested learning tasks can be assigned to appropriate robots, and the underlying infrastructure can be used to support the learning tasks without expert knowledge. Moreover, due to the proposed network system's flexibility, powerful AI-enabled networking algorithms can be utilized to ensure service-level agreements and superior performances for factory workloads. Finally, three open research directions of backward compatibility, end-to-end enhancements, and cybersecurity are discussed for zero-touch smart factory. △ Less

Submitted 26 April, 2022; originally announced April 2022.

Comments: Submitted for publication in the IEEE Network

Efficient Content Delivery in User-Centric and Cache-Enabled Vehicular Edge Networks with Deadline-Constrained Heterogeneous Demands

Authors: Md Ferdous Pervej, Richeng Jin, Shih-Chun Lin, Huaiyu Dai

Abstract: Modern connected vehicles (CVs) frequently require diverse types of content for mission-critical decision-making and onboard users' entertainment. These contents are required to be fully delivered to the requester CVs within stringent deadlines that the existing radio access technology (RAT) solutions may fail to ensure. Motivated by the above consideration, this paper exploits content caching in… ▽ More Modern connected vehicles (CVs) frequently require diverse types of content for mission-critical decision-making and onboard users' entertainment. These contents are required to be fully delivered to the requester CVs within stringent deadlines that the existing radio access technology (RAT) solutions may fail to ensure. Motivated by the above consideration, this paper exploits content caching in vehicular edge networks (VENs) with a software-defined user-centric virtual cell (VC) based RAT solution for delivering the requested contents from a proximity edge server. Moreover, to capture the heterogeneous demands of the CVs, we introduce a preference-popularity tradeoff in their content request model. To that end, we formulate a joint optimization problem for content placement, CV scheduling, VC configuration, VC-CV association and radio resource allocation to minimize long-term content delivery delay. However, the joint problem is highly complex and cannot be solved efficiently in polynomial time. As such, we decompose the original problem into a cache placement problem and a content delivery delay minimization problem given the cache placement policy. We use deep reinforcement learning (DRL) as a learning solution for the first sub-problem. Furthermore, we transform the delay minimization problem into a priority-based weighted sum rate (WSR) maximization problem, which is solved leveraging maximum bipartite matching (MWBM) and a simple linear search algorithm. Our extensive simulation results demonstrate the effectiveness of the proposed method compared to existing baselines in terms of cache hit ratio (CHR), deadline violation and content delivery delay. △ Less

Submitted 29 March, 2023; v1 submitted 15 February, 2022; originally announced February 2022.

Comments: Under review for possible publication in IEEE Transactions on Vehicular Technology

Intelligent Reflecting Surface-Aided Spectrum Sensing for Cognitive Radio

Authors: Shaoe Lin, Beixiong Zheng, Fangjiong Chen, Rui Zhang

Abstract: Spectrum sensing is a key enabling technique for cognitive radio (CR), which provides essential information on the spectrum availability. However, due to severe wireless channel fading and path loss, the primary user (PU) signals received at the CR or secondary user (SU) can be practically too weak for reliable detection. To tackle this issue, we consider in this letter a new intelligent reflectin… ▽ More Spectrum sensing is a key enabling technique for cognitive radio (CR), which provides essential information on the spectrum availability. However, due to severe wireless channel fading and path loss, the primary user (PU) signals received at the CR or secondary user (SU) can be practically too weak for reliable detection. To tackle this issue, we consider in this letter a new intelligent reflecting surface (IRS)-aided spectrum sensing scheme for CR, by exploiting the large aperture and passive beamforming gains of IRS to boost the PU signal strength received at the SU to facilitate its spectrum sensing. Specifically, by dynamically changing the IRS reflection over time according to a given codebook, its reflected signal power varies substantially at the SU, which is utilized for opportunistic signal detection. Furthermore, we propose a weighted energy detection method by combining the received signal power values over different IRS reflections, which significantly improves the detection performance. Simulation results validate the performance gain of the proposed IRS-aided spectrum sensing scheme, as compared to different benchmark schemes. △ Less

Submitted 5 February, 2022; originally announced February 2022.

Comments: Accepted by IEEE Wireless Communications Letters (5 pages, 4 figures)

Journal ref: IEEE Wireless Communications Letters, 2022

Intelligent Reflecting Surface-Aided LEO Satellite Communication: Cooperative Passive Beamforming and Distributed Channel Estimation

Authors: Beixiong Zheng, Shaoe Lin, Rui Zhang

Abstract: We consider in this paper a new intelligent reflecting surface (IRS)-aided LEO satellite communication system, by utilizing the controllable phase shifts of massive passive reflecting elements to achieve flexible beamforming, which copes with the time-varying channel between the high-mobility satellite (SAT) and ground node (GN) cost-effectively. In particular, we propose a new architecture for IR… ▽ More We consider in this paper a new intelligent reflecting surface (IRS)-aided LEO satellite communication system, by utilizing the controllable phase shifts of massive passive reflecting elements to achieve flexible beamforming, which copes with the time-varying channel between the high-mobility satellite (SAT) and ground node (GN) cost-effectively. In particular, we propose a new architecture for IRS-aided LEO satellite communication where IRSs are deployed at both sides of the SAT and GN, and study their cooperative passive beamforming (CPB) design over line-of-sight (LoS)-dominant single-reflection and double-reflection channels. Specifically, we jointly optimize the active transmit/receive beamforming at the SAT/GN as well as the CPB at two-sided IRSs to maximize the overall channel gain from the SAT to each GN. Interestingly, we show that under LoS channel conditions, the high-dimensional SAT-GN channel can be decomposed into the outer product of two low-dimensional vectors. By exploiting the decomposed SAT-GN channel, we decouple the original beamforming optimization problem into two simpler subproblems corresponding to the SAT and GN sides, respectively, which are both solved in closed-form. Furthermore, we propose an efficient transmission protocol to conduct channel estimation and beam tracking, which only requires independent processing of the SAT and GN in a distributed manner, thus substantially reducing the implementation complexity. Simulation results validate the performance advantages of the proposed IRS-aided LEO satellite communication system with two-sided cooperative IRSs, as compared to various baseline schemes such as the conventional reflect-array and one-sided IRS. △ Less

Submitted 8 January, 2022; originally announced January 2022.

Comments: major revision, JSAC

Lane Departure Prediction Based on Closed-Loop Vehicle Dynamics

Authors: Daofei Li, Siyuan Lin, Guanming Liu

Abstract: An automated driving system should have the ability to supervise its own performance and to request human driver to take over when necessary. In the lane keeping scenario, the prediction of vehicle future trajectory is the key to realize safe and trustworthy driving automation. Previous studies on vehicle trajectory prediction mainly fall into two categories, i.e. physics-based and manoeuvre-based… ▽ More An automated driving system should have the ability to supervise its own performance and to request human driver to take over when necessary. In the lane keeping scenario, the prediction of vehicle future trajectory is the key to realize safe and trustworthy driving automation. Previous studies on vehicle trajectory prediction mainly fall into two categories, i.e. physics-based and manoeuvre-based methods. Using a physics-based methodology, this paper proposes a lane departure prediction algorithm based on closed-loop vehicle dynamics model. We use extended Kalman filter to estimate the current vehicle states based on sensing module outputs. Then a Kalman Predictor with actual lane keeping control law is used to predict steering actions and vehicle states in the future. A lane departure assessment module evaluates the probabilistic distribution of vehicle corner positions and decides whether to initiate a human takeover request. The prediction algorithm is capable to describe the stochastic characteristics of future vehicle pose, which is preliminarily proved in simulated tests. Finally, the on-road tests at speeds of 15 to 50 km/h further show that the pro-posed method can accurately predict vehicle future trajectory. It may work as a promising solution to lane departure risk assessment for automated lane keeping functions. △ Less

Submitted 12 February, 2022; v1 submitted 20 December, 2021; originally announced December 2021.

Comments: 12 pages, 23 figures

Privacy-Preserving Serverless Edge Learning with Decentralized Small Data

Authors: Shih-Chun Lin, Chia-Hung Lin

Abstract: In the last decade, data-driven algorithms outperformed traditional optimization-based algorithms in many research areas, such as computer vision, natural language processing, etc. However, extensive data usages bring a new challenge or even threat to deep learning algorithms, i.e., privacy-preserving. Distributed training strategies have recently become a promising approach to ensure data privacy… ▽ More In the last decade, data-driven algorithms outperformed traditional optimization-based algorithms in many research areas, such as computer vision, natural language processing, etc. However, extensive data usages bring a new challenge or even threat to deep learning algorithms, i.e., privacy-preserving. Distributed training strategies have recently become a promising approach to ensure data privacy when training deep models. This paper extends conventional serverless platforms with serverless edge learning architectures and provides an efficient distributed training framework from the networking perspective. This framework dynamically orchestrates available resources among heterogeneous physical units to efficiently fulfill deep learning objectives. The design jointly considers learning task requests and underlying infrastructure heterogeneity, including last-mile transmissions, computation abilities of mobile devices, edge and cloud computing centers, and devices battery status. Furthermore, to significantly reduce distributed training overheads, small-scale data training is proposed by integrating with a general, simple data classifier. This low-load enhancement can seamlessly work with various distributed deep models to improve communications and computation efficiencies during the training phase. Finally, open challenges and future research directions encourage the research community to develop efficient distributed deep learning techniques. △ Less

Submitted 30 November, 2021; v1 submitted 29 November, 2021; originally announced November 2021.

Comments: Submitted for publication in the IEEE Network

Learning of Time-Frequency Attention Mechanism for Automatic Modulation Recognition

Authors: Shangao Lin, Yuan Zeng, Yi Gong

Abstract: Recent learning-based image classification and speech recognition approaches make extensive use of attention mechanisms to achieve state-of-the-art recognition power, which demonstrates the effectiveness of attention mechanisms. Motivated by the fact that the frequency and time information of modulated radio signals are crucial for modulation mode recognition, this paper proposes a time-frequency… ▽ More Recent learning-based image classification and speech recognition approaches make extensive use of attention mechanisms to achieve state-of-the-art recognition power, which demonstrates the effectiveness of attention mechanisms. Motivated by the fact that the frequency and time information of modulated radio signals are crucial for modulation mode recognition, this paper proposes a time-frequency attention mechanism for a convolutional neural network (CNN)-based modulation recognition framework. The proposed time-frequency attention module is designed to learn which channel, frequency and time information is more meaningful in CNN for modulation recognition. We analyze the effectiveness of the proposed time-frequency attention mechanism and compare the proposed method with two existing learning-based methods. Experiments on an open-source modulation recognition dataset show that the recognition performance of the proposed framework is better than those of the framework without time-frequency attention and existing learning-based methods. △ Less

Submitted 11 January, 2022; v1 submitted 5 November, 2021; originally announced November 2021.

COVID-19 Detection in Chest X-ray Images Using Swin-Transformer and Transformer in Transformer

Authors: Juntao Jiang, Shuyi Lin

Abstract: The Coronavirus Disease 2019 (COVID-19) has spread globally and caused serious damage. Chest X-ray images are widely used for COVID-19 diagnosis and the Artificial Intelligence method can increase efficiency and accuracy. In the Challenge of Chest XR COVID-19 detection in Ethics and Explainability for Responsible Data Science (EE-RDS) conference 2021, we proposed a method that combined Swin Transf… ▽ More The Coronavirus Disease 2019 (COVID-19) has spread globally and caused serious damage. Chest X-ray images are widely used for COVID-19 diagnosis and the Artificial Intelligence method can increase efficiency and accuracy. In the Challenge of Chest XR COVID-19 detection in Ethics and Explainability for Responsible Data Science (EE-RDS) conference 2021, we proposed a method that combined Swin Transformer and Transformer in Transformer to classify chest X-ray images as three classes: COVID-19, Pneumonia, and Normal (healthy) and achieved 0.9475 accuracies on the test set. △ Less

Submitted 31 December, 2022; v1 submitted 15 October, 2021; originally announced October 2021.

Risk-Aware Learning for Scalable Voltage Optimization in Distribution Grids

Authors: Shanny Lin, Shaohui Liu, Hao Zhu

Abstract: Real-time coordination of distributed energy resources (DERs) is crucial for regulating the voltage profile in distribution grids. By capitalizing on a scalable neural network (NN) architecture, one can attain decentralized DER decisions to address the lack of real-time communications. This paper develops an advanced learning-enabled DER coordination scheme by accounting for the potential risks as… ▽ More Real-time coordination of distributed energy resources (DERs) is crucial for regulating the voltage profile in distribution grids. By capitalizing on a scalable neural network (NN) architecture, one can attain decentralized DER decisions to address the lack of real-time communications. This paper develops an advanced learning-enabled DER coordination scheme by accounting for the potential risks associated with reactive power prediction and voltage deviation. Such risks are quantified by the conditional value-at-risk (CVaR) using the worst-case samples only, and we propose a mini-batch selection algorithm to address the training speed issue in minimizing the CVaR-regularized loss. Numerical tests using real-world data on the IEEE 123-bus test case have demonstrated the computation and safety improvements of the proposed risk-aware learning algorithm for decentralized DER decision making, especially in terms of reducing feeder voltage violations. △ Less

Submitted 18 April, 2022; v1 submitted 4 October, 2021; originally announced October 2021.

Decoupling Speaker-Independent Emotions for Voice Conversion Via Source-Filter Networks

Authors: Zhaojie Luo, Shoufeng Lin, Rui Liu, Jun Baba, Yuichiro Yoshikawa, Ishiguro Hiroshi

Abstract: Emotional voice conversion (VC) aims to convert a neutral voice to an emotional (e.g. happy) one while retaining the linguistic information and speaker identity. We note that the decoupling of emotional features from other speech information (such as speaker, content, etc.) is the key to achieving remarkable performance. Some recent attempts about speech representation decoupling on the neutral sp… ▽ More Emotional voice conversion (VC) aims to convert a neutral voice to an emotional (e.g. happy) one while retaining the linguistic information and speaker identity. We note that the decoupling of emotional features from other speech information (such as speaker, content, etc.) is the key to achieving remarkable performance. Some recent attempts about speech representation decoupling on the neutral speech can not work well on the emotional speech, due to the more complex acoustic properties involved in the latter. To address this problem, here we propose a novel Source-Filter-based Emotional VC model (SFEVC) to achieve proper filtering of speaker-independent emotion features from both the timbre and pitch features. Our SFEVC model consists of multi-channel encoders, emotion separate encoders, and one decoder. Note that all encoder modules adopt a designed information bottlenecks auto-encoder. Additionally, to further improve the conversion quality for various emotions, a novel two-stage training strategy based on the 2D Valence-Arousal (VA) space was proposed. Experimental results show that the proposed SFEVC along with a two-stage training strategy outperforms all baselines and achieves the state-of-the-art performance in speaker-independent emotional VC with nonparallel data. △ Less

Submitted 3 October, 2021; originally announced October 2021.

Nana-HDR: A Non-attentive Non-autoregressive Hybrid Model for TTS

Authors: Shilun Lin, Wenchao Su, Li Meng, Fenglong Xie, Xinhui Li, Li Lu

Abstract: This paper presents Nana-HDR, a new non-attentive non-autoregressive model with hybrid Transformer-based Dense-fuse encoder and RNN-based decoder for TTS. It mainly consists of three parts: Firstly, a novel Dense-fuse encoder with dense connections between basic Transformer blocks for coarse feature fusion and a multi-head attention layer for fine feature fusion. Secondly, a single-layer non-autor… ▽ More This paper presents Nana-HDR, a new non-attentive non-autoregressive model with hybrid Transformer-based Dense-fuse encoder and RNN-based decoder for TTS. It mainly consists of three parts: Firstly, a novel Dense-fuse encoder with dense connections between basic Transformer blocks for coarse feature fusion and a multi-head attention layer for fine feature fusion. Secondly, a single-layer non-autoregressive RNN-based decoder. Thirdly, a duration predictor instead of an attention model that connects the above hybrid encoder and decoder. Experiments indicate that Nana-HDR gives full play to the advantages of each component, such as strong text encoding ability of Transformer-based encoder, stateful decoding without being bothered by exposure bias and local information preference, and stable alignment provided by duration predictor. Due to these advantages, Nana-HDR achieves competitive performance in naturalness and robustness on two Mandarin corpora. △ Less

Submitted 28 September, 2021; originally announced September 2021.

Data-driven Modeling for Distribution Grids Under Partial Observability

Authors: Shanny Lin, Hao Zhu

Abstract: Accurately modeling power distribution grids is crucial for designing effective monitoring and decision making algorithms. This paper addresses the partial observability issue of data-driven distribution modeling in order to improve the accuracy of line parameter estimation. Inspired by the sparse changes in residential loads, we advocate to regularize the group sparsity of the unobservable inject… ▽ More Accurately modeling power distribution grids is crucial for designing effective monitoring and decision making algorithms. This paper addresses the partial observability issue of data-driven distribution modeling in order to improve the accuracy of line parameter estimation. Inspired by the sparse changes in residential loads, we advocate to regularize the group sparsity of the unobservable injections in a bi-linear estimation problem. The alternating minimization scheme of guaranteed convergence is proposed to take advantage of convex subproblems with efficient solutions. Numerical results using real-world load data on the single-phase equivalent of the IEEE 123-bus test case have demonstrated the accuracy improvements of the proposed solution over existing work for both parameter estimation and voltage modeling. △ Less

Submitted 7 October, 2021; v1 submitted 18 August, 2021; originally announced August 2021.

NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results

Authors: Ren Yang, Radu Timofte, Jing Liu, Yi Xu, Xinjian Zhang, Minyi Zhao, Shuigeng Zhou, Kelvin C. K. Chan, Shangchen Zhou, Xiangyu Xu, Chen Change Loy, Xin Li, Fanglong Liu, He Zheng, Lielin Jiang, Qi Zhang, Dongliang He, Fu Li, Qingqing Dang, Yibin Huang, Matteo Maggioni, Zhongqian Fu, Shuai Xiao, Cheng li, Thomas Tanay , et al. (47 additional authors not shown)

Abstract: This paper reviews the first NTIRE challenge on quality enhancement of compressed video, with a focus on the proposed methods and results. In this challenge, the new Large-scale Diverse Video (LDV) dataset is employed. The challenge has three tracks. Tracks 1 and 2 aim at enhancing the videos compressed by HEVC at a fixed QP, while Track 3 is designed for enhancing the videos compressed by x265 at… ▽ More This paper reviews the first NTIRE challenge on quality enhancement of compressed video, with a focus on the proposed methods and results. In this challenge, the new Large-scale Diverse Video (LDV) dataset is employed. The challenge has three tracks. Tracks 1 and 2 aim at enhancing the videos compressed by HEVC at a fixed QP, while Track 3 is designed for enhancing the videos compressed by x265 at a fixed bit-rate. Besides, the quality enhancement of Tracks 1 and 3 targets at improving the fidelity (PSNR), and Track 2 targets at enhancing the perceptual quality. The three tracks totally attract 482 registrations. In the test phase, 12 teams, 8 teams and 11 teams submitted the final results of Tracks 1, 2 and 3, respectively. The proposed methods and solutions gauge the state-of-the-art of video quality enhancement. The homepage of the challenge: https://github.com/RenYang-home/NTIRE21_VEnh △ Less

Submitted 31 August, 2022; v1 submitted 21 April, 2021; originally announced April 2021.

Comments: Corrected the MOS values in Table 2, and corrected some minor typos

SD-VEC: Software-Defined Vehicular Edge Computing with Ultra-Low Latency

Authors: Shih-Chun Lin, Kwang-Cheng Chen, Ali Karimoddini

Abstract: New paradigm shifts and 6G technological revolution in vehicular services have emerged toward unmanned driving, automated transportation, and self-driving vehicles. As the technology for autonomous vehicles becomes mature, real challenges come from reliable, safe, real-time connected transportation operations to achieve ubiquitous and prompt information exchanges with massive connected and autonom… ▽ More New paradigm shifts and 6G technological revolution in vehicular services have emerged toward unmanned driving, automated transportation, and self-driving vehicles. As the technology for autonomous vehicles becomes mature, real challenges come from reliable, safe, real-time connected transportation operations to achieve ubiquitous and prompt information exchanges with massive connected and autonomous vehicles. This article aims at introducing novel wireless distributed architectures that embed the edge computing capability inside software-defined vehicular networking infrastructure. Such edge networks consist of open-loop grant-free communications and computing-based control frameworks, which enable dynamic eco-routing with ultra-low latency and mobile data-driven orchestration. Thus, this work advances the frontiers of machine learning potentials and next-generation mobile system realization in vehicular networking applications. △ Less

Submitted 25 March, 2021; originally announced March 2021.

Enhancing the Spatio-temporal Observability of Grid-Edge Resources in Distribution Grids

Authors: Shanny Lin, Hao Zhu

Abstract: Enhancing the spatio-temporal observability of distributed energy resources (DERs) is crucial for achieving secure and efficient operations in distribution grids. This paper puts forth a joint recovery framework for residential loads by leveraging the complimentary strengths of heterogeneous measurements in real time. The proposed framework integrates low-resolution smart meter data collected at e… ▽ More Enhancing the spatio-temporal observability of distributed energy resources (DERs) is crucial for achieving secure and efficient operations in distribution grids. This paper puts forth a joint recovery framework for residential loads by leveraging the complimentary strengths of heterogeneous measurements in real time. The proposed framework integrates low-resolution smart meter data collected at every load node with fast-sampled feeder-level measurements from limited number of distribution phasor measurement units. To address the lack of data, we exploit two key characteristics for the loads and DERs, namely the sparse changes due to infrequent activities of appliances and electric vehicles (EVs) and the locational dependence of solar photovoltaic (PV) generation. Accordingly, meaningful regularization terms are introduced to cast a convex load recovery problem, which will be further simplified to reduce the computational complexity. The load recovery solutions can be utilized to identify the EV charging events at each load node and to infer the total behind-the-meter PV output. Numerical tests using real-world data have demonstrated the effectiveness of the proposed approaches in enhancing the visibility of grid-edge DERs. △ Less

Submitted 20 August, 2021; v1 submitted 15 February, 2021; originally announced February 2021.

Triple M: A Practical Text-to-speech Synthesis System With Multi-guidance Attention And Multi-band Multi-time LPCNet

Authors: Shilun Lin, Fenglong Xie, Li Meng, Xinhui Li, Li Lu

Abstract: In this work, a robust and efficient text-to-speech (TTS) synthesis system named Triple M is proposed for large-scale online application. The key components of Triple M are: 1) A sequence-to-sequence model adopts a novel multi-guidance attention to transfer complementary advantages from guiding attention mechanisms to the basic attention mechanism without in-domain performance loss and online serv… ▽ More In this work, a robust and efficient text-to-speech (TTS) synthesis system named Triple M is proposed for large-scale online application. The key components of Triple M are: 1) A sequence-to-sequence model adopts a novel multi-guidance attention to transfer complementary advantages from guiding attention mechanisms to the basic attention mechanism without in-domain performance loss and online service modification. Compared with single attention mechanism, multi-guidance attention not only brings better naturalness to long sentence synthesis, but also reduces the word error rate by 26.8%. 2) A new efficient multi-band multi-time vocoder framework, which reduces the computational complexity from 2.8 to 1.0 GFLOP and speeds up LPCNet by 2.75x on a single CPU. △ Less

Submitted 7 April, 2021; v1 submitted 30 January, 2021; originally announced February 2021.

Continual Learning of Generative Models with Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence

Authors: Mehmet Dedeoglu, Sen Lin, Zhaofeng Zhang, Junshan Zhang

Abstract: Learning generative models is challenging for a network edge node with limited data and computing power. Since tasks in similar environments share model similarity, it is plausible to leverage pre-trained generative models from the cloud or other edge nodes. Appealing to optimal transport theory tailored towards Wasserstein-1 generative adversarial networks (WGAN), this study aims to develop a fra… ▽ More Learning generative models is challenging for a network edge node with limited data and computing power. Since tasks in similar environments share model similarity, it is plausible to leverage pre-trained generative models from the cloud or other edge nodes. Appealing to optimal transport theory tailored towards Wasserstein-1 generative adversarial networks (WGAN), this study aims to develop a framework which systematically optimizes continual learning of generative models using local data at the edge node while exploiting adaptive coalescence of pre-trained generative models. Specifically, by treating the knowledge transfer from other nodes as Wasserstein balls centered around their pre-trained models, continual learning of generative models is cast as a constrained optimization problem, which is further reduced to a Wasserstein-1 barycenter problem. A two-stage approach is devised accordingly: 1) The barycenters among the pre-trained models are computed offline, where displacement interpolation is used as the theoretic foundation for finding adaptive barycenters via a "recursive" WGAN configuration; 2) the barycenter computed offline is used as meta-model initialization for continual learning and then fast adaptation is carried out to find the generative model using the local samples at the target edge node. Finally, a weight ternarization method, based on joint optimization of weights and threshold for quantization, is developed to compress the generative model further. △ Less

Submitted 22 January, 2021; originally announced January 2021.

Vehicular Network Slicing for Reliable Access and Deadline-Constrained Data Offloading: A Multi-Agent On-Device Learning Approach

Authors: Md Ferdous Pervej, Shih-Chun Lin

Abstract: Efficient data offloading plays a pivotal role in computational-intensive platforms as data rate over wireless channels is fundamentally limited. On top of that, high mobility adds an extra burden in vehicular edge networks (VENs), bolstering the desire for efficient user-centric solutions. Therefore, unlike the legacy inflexible network-centric approach, this paper exploits a software-defined fle… ▽ More Efficient data offloading plays a pivotal role in computational-intensive platforms as data rate over wireless channels is fundamentally limited. On top of that, high mobility adds an extra burden in vehicular edge networks (VENs), bolstering the desire for efficient user-centric solutions. Therefore, unlike the legacy inflexible network-centric approach, this paper exploits a software-defined flexible, open, and programmable networking platform for an efficient user-centric, fast, reliable, and deadline-constrained offloading solution in VENs. In the proposed model, each active vehicle user (VU) is served from multiple low-powered access points (APs) by creating a noble virtual cell (VC). A joint node association, power allocation, and distributed resource allocation problem is formulated. As centralized learning is not practical in many real-world problems, following the distributed nature of autonomous VUs, each VU is considered an edge learning agent. To that end, considering practical location-aware node associations, a joint radio and power resource allocation non-cooperative stochastic game is formulated. Leveraging reinforcement learning's (RL) efficacy, a multi-agent RL (MARL) solution is proposed where the edge learners aim to learn the Nash equilibrium (NE) strategies to solve the game efficiently. Besides, real-world map data, with a practical microscopic mobility model, are used for the simulation. Results suggest that the proposed user-centric approach can deliver remarkable performances in VENs. Moreover, the proposed MARL solution delivers near-optimal performances with approximately 3% collision probabilities in case of distributed random access in the uplink. △ Less

Submitted 14 October, 2021; v1 submitted 31 December, 2020; originally announced December 2020.

Comments: This paper was rejected from JSAC; we need to address the reviewers' comments