Showing 1–32 of 32 results for author: Ho, A

Combinatorial Reasoning: Selecting Reasons in Generative AI Pipelines via Combinatorial Optimization

Authors: Mert Esencan, Tarun Advaith Kumar, Ata Akbari Asanjan, P. Aaron Lott, Masoud Mohseni, Can Unlu, Davide Venturelli, Alan Ho

Abstract: Recent Large Language Models (LLMs) have demonstrated impressive capabilities at tasks that require human intelligence and are a significant step towards human-like artificial intelligence (AI). Yet the performance of LLMs at reasoning tasks have been subpar and the reasoning capability of LLMs is a matter of significant debate. While it has been shown that the choice of the prompting technique to… ▽ More Recent Large Language Models (LLMs) have demonstrated impressive capabilities at tasks that require human intelligence and are a significant step towards human-like artificial intelligence (AI). Yet the performance of LLMs at reasoning tasks have been subpar and the reasoning capability of LLMs is a matter of significant debate. While it has been shown that the choice of the prompting technique to the LLM can alter its performance on a multitude of tasks, including reasoning, the best performing techniques require human-made prompts with the knowledge of the tasks at hand. We introduce a framework for what we call Combinatorial Reasoning (CR), a fully-automated prompting method, where reasons are sampled from an LLM pipeline and mapped into a Quadratic Unconstrained Binary Optimization (QUBO) problem. The framework investigates whether QUBO solutions can be profitably used to select a useful subset of the reasons to construct a Chain-of-Thought style prompt. We explore the acceleration of CR with specialized solvers. We also investigate the performance of simpler zero-shot strategies such as linear majority rule or random selection of reasons. Our preliminary study indicates that coupling a combinatorial solver to generative AI pipelines is an interesting avenue for AI reasoning and elucidates design principles for future CR methods. △ Less

Submitted 19 June, 2024; originally announced July 2024.

Comments: 13 pages, 3 figures

Context-aware LLM-based Safe Control Against Latent Risks

Authors: Quan Khanh Luu, Xiyu Deng, Anh Van Ho, Yorie Nakahira

Abstract: It is challenging for autonomous control systems to perform complex tasks in the presence of latent risks. Motivated by this challenge, this paper proposes an integrated framework that involves Large Language Models (LLMs), stochastic gradient descent (SGD), and optimization-based control. In the first phrase, the proposed framework breaks down complex tasks into a sequence of smaller subtasks, wh… ▽ More It is challenging for autonomous control systems to perform complex tasks in the presence of latent risks. Motivated by this challenge, this paper proposes an integrated framework that involves Large Language Models (LLMs), stochastic gradient descent (SGD), and optimization-based control. In the first phrase, the proposed framework breaks down complex tasks into a sequence of smaller subtasks, whose specifications account for contextual information and latent risks. In the second phase, these subtasks and their parameters are refined through a dual process involving LLMs and SGD. LLMs are used to generate rough guesses and failure explanations, and SGD is used to fine-tune parameters. The proposed framework is tested using simulated case studies of robots and vehicles. The experiments demonstrate that the proposed framework can mediate actions based on the context and latent risks and learn complex behaviors efficiently. △ Less

Submitted 18 March, 2024; originally announced March 2024.

Algorithmic progress in language models

Authors: Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, Jaime Sevilla

Abstract: We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months,… ▽ More We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms. △ Less

Submitted 9 March, 2024; originally announced March 2024.

ROSE: Rotation-based Squeezing Robotic Gripper toward Universal Handling of Objects

Authors: Son Tien Bui, Shinya Kawano, Van Anh Ho

Abstract: Robotics hand/grippers nowadays are not limited to manufacturing lines; instead, they are widely utilized in cluttered environments, such as restaurants, farms, and warehouses. In such scenarios, they need to deal with high uncertainty of the grasped objects' shapes, postures, surfaces, and material properties, which requires complex integration of sensing and decision-making process. On the other… ▽ More Robotics hand/grippers nowadays are not limited to manufacturing lines; instead, they are widely utilized in cluttered environments, such as restaurants, farms, and warehouses. In such scenarios, they need to deal with high uncertainty of the grasped objects' shapes, postures, surfaces, and material properties, which requires complex integration of sensing and decision-making process. On the other hand, integrating soft materials into the gripper's design may tolerate the above uncertainties and reduce complexity in control. In this paper, we introduce ROSE, a novel soft gripper that can embrace the object and squeeze it by buckling a funnel-liked thin-walled soft membrane around the object by simple rotation of the base. Thanks to this design, ROSE hand can adapt to a wide range of objects that can fit in the funnel and handle with gentle gripping force. Regardless of this, ROSE can generate a high lift force (up to 33kgf) while significantly reducing the normal pressure on the gripped objects. In our experiment, a 198g ROSE can be integrated into a robot arm with a single actuation and successfully lift various types of objects, even after 400,000 trials. The embracing mechanism helps reduce the dependence of friction between the object and the membrane, as ROSE could pick up a chicken egg submerged inside an olive oil tank. We also report a feasible design for equipping the ROSE hand with tactile sensing while appealing to the scalability of the design to fit a wide range of objects. Video: https://youtu.be/E1wAI09LaoY △ Less

Submitted 10 February, 2024; originally announced February 2024.

Comments: 9 pages, 9 figures, RSS2023 conference

Journal ref: Robotics: Science and System 2023

EuroPED-NN: Uncertainty aware surrogate model

Authors: A. Panera Alvarez, A. Ho, A. Jarvinen, S. Saarelma, S. Wiesen, JET Contributors

Abstract: This work successfully generates uncertainty aware surrogate models, via the Bayesian neural network with noise contrastive prior (BNN-NCP) technique, of the EuroPED plasma pedestal model using data from the JET-ILW pedestal database and subsequent model evaluations. All this conform EuroPED-NN. The BNN-NCP technique is proven to be a good fit for uncertainty aware surrogate models, matching the o… ▽ More This work successfully generates uncertainty aware surrogate models, via the Bayesian neural network with noise contrastive prior (BNN-NCP) technique, of the EuroPED plasma pedestal model using data from the JET-ILW pedestal database and subsequent model evaluations. All this conform EuroPED-NN. The BNN-NCP technique is proven to be a good fit for uncertainty aware surrogate models, matching the output results as a regular neural network, providing prediction's confidence as uncertainties, and highlighting the out of distribution (OOD) regions using surrogate model uncertainties. This provides critical insights into model robustness and reliability. EuroPED-NN has been physically validated, first, analyzing electron density $n_e\!\left(ψ_{\text{pol}}=0.94\right)$ with respect to increasing plasma current, $I_p$, and second, validating the $Δ-β_{p,ped}$ relation associated with the EuroPED model. Affirming the robustness of the underlying physics learned by the surrogate model. △ Less

Submitted 1 February, 2024; originally announced February 2024.

Evaluating Language-Model Agents on Realistic Autonomous Tasks

Authors: Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R. Lin, Hjalmar Wijk, Joel Burget, Aaron Ho, Elizabeth Barnes, Paul Christiano

Abstract: In this report, we explore the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. We refer to this cluster of capabilities as "autonomous replication and adaptation" or ARA. We believe that systems capable of ARA could have wide-reaching and hard-to-anticipate consequences, and that measuring and forecasting… ▽ More In this report, we explore the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. We refer to this cluster of capabilities as "autonomous replication and adaptation" or ARA. We believe that systems capable of ARA could have wide-reaching and hard-to-anticipate consequences, and that measuring and forecasting ARA may be useful for informing measures around security, monitoring, and alignment. Additionally, once a system is capable of ARA, placing bounds on a system's capabilities may become significantly more difficult. We construct four simple example agents that combine language models with tools that allow them to take actions in the world. We then evaluate these agents on 12 tasks relevant to ARA. We find that these language model agents can only complete the easiest tasks from this list, although they make some progress on the more challenging tasks. Unfortunately, these evaluations are not adequate to rule out the possibility that near-future agents will be capable of ARA. In particular, we do not think that these evaluations provide good assurance that the ``next generation'' of language models (e.g. 100x effective compute scaleup on existing models) will not yield agents capable of ARA, unless intermediate evaluations are performed during pretraining. Relatedly, we expect that fine-tuning of the existing models could produce substantially more competent agents, even if the fine-tuning is not directly targeted at ARA. △ Less

Submitted 4 January, 2024; v1 submitted 18 December, 2023; originally announced December 2023.

Comments: 14 pages

Limits to the Energy Efficiency of CMOS Microprocessors

Authors: Anson Ho, Ege Erdil, Tamay Besiroglu

Abstract: CMOS microprocessors have achieved massive energy efficiency gains but may reach limits soon. This paper presents an approach to estimating the limits on the maximum floating point operations per Joule (FLOP/J) for CMOS microprocessors. We analyze the three primary sources of energy dissipation: transistor switching, interconnect capacitances and leakage power. Using first-principles calculations… ▽ More CMOS microprocessors have achieved massive energy efficiency gains but may reach limits soon. This paper presents an approach to estimating the limits on the maximum floating point operations per Joule (FLOP/J) for CMOS microprocessors. We analyze the three primary sources of energy dissipation: transistor switching, interconnect capacitances and leakage power. Using first-principles calculations of minimum energy costs based on Landauer's principle, prior estimates of relevant parameters, and empirical data on hardware, we derive the energy cost per FLOP for each component. Combining these yields a geometric mean estimate of 4.7e15 FP4/J for the maximum CMOS energy efficiency, roughly two hundred-fold more efficient than current microprocessors. △ Less

Submitted 13 December, 2023; originally announced December 2023.

Building Footprint Extraction in Dense Areas using Super Resolution and Frame Field Learning

Authors: Vuong Nguyen, Anh Ho, Duc-Anh Vu, Nguyen Thi Ngoc Anh, Tran Ngoc Thang

Abstract: Despite notable results on standard aerial datasets, current state-of-the-arts fail to produce accurate building footprints in dense areas due to challenging properties posed by these areas and limited data availability. In this paper, we propose a framework to address such issues in polygonal building extraction. First, super resolution is employed to enhance the spatial resolution of aerial imag… ▽ More Despite notable results on standard aerial datasets, current state-of-the-arts fail to produce accurate building footprints in dense areas due to challenging properties posed by these areas and limited data availability. In this paper, we propose a framework to address such issues in polygonal building extraction. First, super resolution is employed to enhance the spatial resolution of aerial image, allowing for finer details to be captured. This enhanced imagery serves as input to a multitask learning module, which consists of a segmentation head and a frame field learning head to effectively handle the irregular building structures. Our model is supervised by adaptive loss weighting, enabling extraction of sharp edges and fine-grained polygons which is difficult due to overlapping buildings and low data quality. Extensive experiments on a slum area in India that mimics a dense area demonstrate that our proposed approach significantly outperforms the current state-of-the-art methods by a large margin. △ Less

Submitted 4 September, 2023; originally announced September 2023.

Comments: Accepted at The 12th International Conference on Awareness Science and Technology

Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translation

Authors: Shravan Nayak, Surangika Ranathunga, Sarubi Thillainathan, Rikki Hung, Anthony Rinaldi, Yining Wang, Jonah Mackey, Andrew Ho, En-Shiun Annie Lee

Abstract: NMT systems trained on Pre-trained Multilingual Sequence-Sequence (PMSS) models flounder when sufficient amounts of parallel data is not available for fine-tuning. This specifically holds for languages missing/under-represented in these models. The problem gets aggravated when the data comes from different domains. In this paper, we show that intermediate-task fine-tuning (ITFT) of PMSS models is… ▽ More NMT systems trained on Pre-trained Multilingual Sequence-Sequence (PMSS) models flounder when sufficient amounts of parallel data is not available for fine-tuning. This specifically holds for languages missing/under-represented in these models. The problem gets aggravated when the data comes from different domains. In this paper, we show that intermediate-task fine-tuning (ITFT) of PMSS models is extremely beneficial for domain-specific NMT, especially when target domain data is limited/unavailable and the considered languages are missing or under-represented in the PMSS model. We quantify the domain-specific results variations using a domain-divergence test, and show that ITFT can mitigate the impact of domain divergence to some extent. △ Less

Submitted 23 September, 2023; v1 submitted 2 June, 2023; originally announced June 2023.

Comments: Accepted for poster presentation at the Practical Machine Learning for Developing Countries (PML4DC) workshop, ICLR 2023

Will we run out of data? Limits of LLM scaling based on human-generated data

Authors: Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Marius Hobbhahn

Abstract: We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock o… ▽ More We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress. △ Less

Submitted 4 June, 2024; v1 submitted 25 October, 2022; originally announced November 2022.

Soft Robotic Link with Controllable Transparency for Vision-based Tactile and Proximity Sensing

Authors: Quan Khanh Luu, Dinh Quang Nguyen, Nhan Huu Nguyen, Van Anh Ho

Abstract: Robots have been brought to work close to humans in many scenarios. For coexistence and collaboration, robots should be safe and pleasant for humans to interact with. To this end, the robots could be both physically soft with multimodal sensing/perception, so that the robots could have better awareness of the surrounding environment, as well as to respond properly to humans' action/intention. This… ▽ More Robots have been brought to work close to humans in many scenarios. For coexistence and collaboration, robots should be safe and pleasant for humans to interact with. To this end, the robots could be both physically soft with multimodal sensing/perception, so that the robots could have better awareness of the surrounding environment, as well as to respond properly to humans' action/intention. This paper introduces a novel soft robotic link, named ProTac, that possesses multiple sensing modes: tactile and proximity sensing, based on computer vision and a functional material. These modalities come from a layered structure of a soft transparent silicon skin, a polymer dispersed liquid crystal (PDLC) film, and reflective markers. Here, the PDLC film can switch actively between the opaque and the transparent state, from which the tactile sensing and proximity sensing can be obtained by using cameras solely built inside the ProTac link. In this paper, inference algorithms for tactile proximity perception are introduced. Evaluation results of two sensing modalities demonstrated that, with a simple activation strategy, ProTac link could effectively perceive useful information from both approaching and in-contact obstacles. The proposed sensing device is expected to bring in ultimate solutions for design of robots with softness, whole-body and multimodal sensing, and safety control strategies. △ Less

Submitted 6 November, 2022; originally announced November 2022.

Comments: Submitted to RoboSoft 2023 for review. Final content subjected to change

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Authors: Tilman Räuker, Anson Ho, Stephen Casper, Dylan Hadfield-Menell

Abstract: The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to… ▽ More The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to identify problems, fix bugs, and improve basic understanding. In particular, "inner" interpretability techniques, which focus on explaining the internal components of DNNs, are well-suited for developing a mechanistic understanding, guiding manual modifications, and reverse engineering solutions. Much recent work has focused on DNN interpretability, and rapid progress has thus far made a thorough systematization of methods difficult. In this survey, we review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain (weights, neurons, subnetworks, or latent representations) and whether they are implemented during (intrinsic) or after (post hoc) training. To our knowledge, we are also the first to survey a number of connections between interpretability research and work in adversarial robustness, continual learning, modularity, network compression, and studying the human visual system. We discuss key challenges and argue that the status quo in interpretability research is largely unproductive. Finally, we highlight the importance of future work that emphasizes diagnostics, debugging, adversaries, and benchmarking in order to make interpretability tools more useful to engineers in practical applications. △ Less

Submitted 18 August, 2023; v1 submitted 26 July, 2022; originally announced July 2022.

Machine Learning Model Sizes and the Parameter Gap

Authors: Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho, Marius Hobbhahn

Abstract: We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven orders of magnitude. The trend then accelerated, with model size increasing by another five orders of magnitude in just 4 years from 2018 to 2022. Vision models grew at a more constant pace, totaling 7 orders of magnitude… ▽ More We study trends in model size of notable machine learning systems over time using a curated dataset. From 1950 to 2018, model size in language models increased steadily by seven orders of magnitude. The trend then accelerated, with model size increasing by another five orders of magnitude in just 4 years from 2018 to 2022. Vision models grew at a more constant pace, totaling 7 orders of magnitude of growth between 1950 and 2022. We also identify that, since 2020, there have been many language models below 20B parameters, many models above 70B parameters, but a scarcity of models in the 20-70B parameter range. We refer to that scarcity as the parameter gap. We provide some stylized facts about the parameter gap and propose a few hypotheses to explain it. The explanations we favor are: (a) increasing model size beyond 20B parameters requires adopting different parallelism techniques, which makes mid-sized models less cost-effective, (b) GPT-3 was one order of magnitude larger than previous language models, and researchers afterwards primarily experimented with bigger models to outperform it. While these dynamics likely exist, and we believe they play some role in generating the gap, we don't have high confidence that there are no other, more important dynamics at play. △ Less

Submitted 5 July, 2022; originally announced July 2022.

Causal Analysis of Generic Time Series Data Applied for Market Prediction

Authors: Anton Kolonin, Ali Raheman, Mukul Vishwas, Ikram Ansari, Juan Pinzon, Alice Ho

Abstract: We explore the applicability of the causal analysis based on temporally shifted (lagged) Pearson correlation applied to diverse time series of different natures in context of the problem of financial market prediction. Theoretical discussion is followed by description of the practical approach for specific environment of time series data with diverse nature and sparsity, as applied for environment… ▽ More We explore the applicability of the causal analysis based on temporally shifted (lagged) Pearson correlation applied to diverse time series of different natures in context of the problem of financial market prediction. Theoretical discussion is followed by description of the practical approach for specific environment of time series data with diverse nature and sparsity, as applied for environments of financial markets. The data involves various financial metrics computable from raw market data such as real-time trades and snapshots of the limit order book as well as metrics determined upon social media news streams such as sentiment and different cognitive distortions. The approach is backed up with presentation of algorithmic framework for data acquisition and analysis, concluded with experimental results, and summary pointing out at the possibility to discriminate causal connections between different sorts of real field market data with further discussion on present issues and possible directions of the following work. △ Less

Submitted 22 April, 2022; originally announced April 2022.

Comments: 10 pages, 4 figures, submitted to Artificial General Intelligence 2022 conference

Tombo Propeller: Bio-Inspired Deformable Structure toward Collision-Accommodated Control for Drones

Authors: Son Tien Bui, Quan Khanh Luu, Dinh Quang Nguyen, Nhat Dinh Minh Le, Giuseppe Loianno, Van Anh Ho

Abstract: There is a growing need for vertical take-off and landing vehicles, including drones, which are safe to use and can adapt to collisions. The risks of damage by collision, to humans, obstacles in the environment, and drones themselves, are significant. This has prompted a search into nature for a highly resilient structure that can inform a design of propellers to reduce those risks and enhance saf… ▽ More There is a growing need for vertical take-off and landing vehicles, including drones, which are safe to use and can adapt to collisions. The risks of damage by collision, to humans, obstacles in the environment, and drones themselves, are significant. This has prompted a search into nature for a highly resilient structure that can inform a design of propellers to reduce those risks and enhance safety. Inspired by the flexibility and resilience of dragonfly wings, we propose a novel design for a biomimetic drone propeller called Tombo propeller. Here, we report on the design and fabrication process of this biomimetic propeller that can accommodate collisions and recover quickly, while maintaining sufficient thrust force to hover and fly. We describe the development of an aerodynamic model and experiments conducted to investigate performance characteristics for various configurations of the propeller morphology, and related properties, such as generated thrust force, thrust force deviation, collision force, recovery time, lift-to-drag ratio, and noise. Finally, we design and showcase a control strategy for a drone equipped with Tombo propellers that collides in mid-air with an obstacle and recovers from collision continuing flying. The results show that the maximum collision force generated by the proposed Tombo propeller is less than two-thirds that of a traditional rigid propeller, which suggests the concrete possibility to employ deformable propellers for drones flying in a cluttered environment. This research can contribute to morphological design of flying vehicles for agile and resilient performance. △ Less

Submitted 14 February, 2022; originally announced February 2022.

Compute Trends Across Three Eras of Machine Learning

Authors: Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos

Abstract: Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compu… ▽ More Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems. △ Less

Submitted 9 March, 2022; v1 submitted 11 February, 2022; originally announced February 2022.

Journal ref: 2022 International Joint Conference on Neural Networks (IJCNN), Padua, Italy, 2022, pp. 1-8

Nonequilibrium Monte Carlo for unfreezing variables in hard combinatorial optimization

Authors: Masoud Mohseni, Daniel Eppens, Johan Strumpfer, Raffaele Marino, Vasil Denchev, Alan K. Ho, Sergei V. Isakov, Sergio Boixo, Federico Ricci-Tersenghi, Hartmut Neven

Abstract: Optimizing highly complex cost/energy functions over discrete variables is at the heart of many open problems across different scientific disciplines and industries. A major obstacle is the emergence of many-body effects among certain subsets of variables in hard instances leading to critical slowing down or collective freezing for known stochastic local search strategies. An exponential computati… ▽ More Optimizing highly complex cost/energy functions over discrete variables is at the heart of many open problems across different scientific disciplines and industries. A major obstacle is the emergence of many-body effects among certain subsets of variables in hard instances leading to critical slowing down or collective freezing for known stochastic local search strategies. An exponential computational effort is generally required to unfreeze such variables and explore other unseen regions of the configuration space. Here, we introduce a quantum-inspired family of nonlocal Nonequilibrium Monte Carlo (NMC) algorithms by developing an adaptive gradient-free strategy that can efficiently learn key instance-wise geometrical features of the cost function. That information is employed on-the-fly to construct spatially inhomogeneous thermal fluctuations for collectively unfreezing variables at various length scales, circumventing costly exploration versus exploitation trade-offs. We apply our algorithm to two of the most challenging combinatorial optimization problems: random k-satisfiability (k-SAT) near the computational phase transitions and Quadratic Assignment Problems (QAP). We observe significant speedup and robustness over both specialized deterministic solvers and generic stochastic solvers. In particular, for 90% of random 4-SAT instances we find solutions that are inaccessible for the best specialized deterministic algorithm known as Survey Propagation (SP) with an order of magnitude improvement in the quality of solutions for the hardest 10% instances. We also demonstrate two orders of magnitude improvement in time-to-solution over the state-of-the-art generic stochastic solver known as Adaptive Parallel Tempering (APT). △ Less

Submitted 26 November, 2021; originally announced November 2021.

Comments: 28 pages, 18 figures

Two step clustering for data reduction combining DBSCAN and k-means clustering

Authors: Bart J. J. Kremers, Aaron Ho, Jonathan Citrin, Karel L. van de Plassche

Abstract: A novel combination of two widely-used clustering algorithms is proposed here for the detection and reduction of high data density regions. The Density Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is used for the detection of high data density regions and the k-means algorithm for reduction. The proposed algorithm iterates while successively decrementing the DBSCAN search… ▽ More A novel combination of two widely-used clustering algorithms is proposed here for the detection and reduction of high data density regions. The Density Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is used for the detection of high data density regions and the k-means algorithm for reduction. The proposed algorithm iterates while successively decrementing the DBSCAN search radius, allowing for an adaptive reduction factor based on the effective data density. The algorithm is demonstrated for a physics simulation application, where a surrogate model for fusion reactor plasma turbulence is generated with neural networks. A training dataset for the surrogate model is created with a quasilinear gyrokinetics code for turbulent transport calculations in fusion plasmas. The training set consists of model inputs derived from a repository of experimental measurements, meaning there is a potential risk of over-representing specific regions of this input parameter space. By applying the proposed reduction algorithm to this dataset, this study demonstrates that the training dataset can be reduced by a factor ~20 using the proposed algorithm, without a noticeable loss in the surrogate model accuracy. This reduction provides a novel way of analyzing existing high-dimensional datasets for biases and consequently reducing them, which lowers the cost of re-populating that parameter space with higher quality data. △ Less

Submitted 22 November, 2021; originally announced November 2021.

ACM Class: I.0

HRnV-Calc: A software package for heart rate n-variability and heart rate variability analysis

Authors: Chenglin Niu, Dagang Guo, Marcus Eng Hock Ong, Zhi Xiong Koh, Andrew Fu Wah Ho, Zhiping Lin, Chengyu Liu, Gari D. Clifford, Nan Liu

Abstract: Objective: Heart rate variability (HRV) has been proven to be an important indicator of physiological status for numerous applications. Despite the progress and active developments made in HRV metric research over the last few decades, the representation of the heartbeat sequence upon which HRV is based has received relatively little attention. The recently introduced heart rate n-variability (HRn… ▽ More Objective: Heart rate variability (HRV) has been proven to be an important indicator of physiological status for numerous applications. Despite the progress and active developments made in HRV metric research over the last few decades, the representation of the heartbeat sequence upon which HRV is based has received relatively little attention. The recently introduced heart rate n-variability (HRnV) offers an alternative to R-to-R peak interval representations which complements conventional HRV analysis by considering HRV behavior on varying scales. Although HRnV has been shown to improve triage in pilot studies, there is currently no open and standard software to support future research of HRnV and its broader clinical applications. We aimed to develop an open, reliable, and easy to use software package implementing HRnV for further research and improvements of HRnV. This package has been designed to facilitate collaborative investigations between clinicians and researchers to study HRnV in various contexts and applications. Approach: We developed an open-source software, HRnV-Calc, based on the PhysioNet Cardiovascular Signal Toolbox (PCST), which features comprehensive graphical user interfaces (GUIs) for HRnV and HRV analysis. Main results: While preserving the core functionalities and performance of PCST, HRnV-Calc enables step-by-step manual inspection and configuration of HRV and HRnV analysis, so that results can be debugged, easily interpreted, and integrated to downstream applications. Significance: The open-source HRnV-Calc software, an accessible and standardized HRV and HRnV analysis platform, enhances the scope of HRV assessment and is designed to assist in future improvements and applications of HRnV and related research. △ Less

Submitted 18 November, 2021; originally announced November 2021.

Learning Temporally Causal Latent Processes from General Temporal Data

Authors: Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, Kun Zhang

Abstract: Our goal is to recover time-delayed latent causal variables and identify their relations from measured temporal data. Estimating causally-related latent variables from observations is particularly challenging as the latent variables are not uniquely recoverable in the most general case. In this work, we consider both a nonparametric, nonstationary setting and a parametric setting for the latent pr… ▽ More Our goal is to recover time-delayed latent causal variables and identify their relations from measured temporal data. Estimating causally-related latent variables from observations is particularly challenging as the latent variables are not uniquely recoverable in the most general case. In this work, we consider both a nonparametric, nonstationary setting and a parametric setting for the latent processes and propose two provable conditions under which temporally causal latent processes can be identified from their nonlinear mixtures. We propose LEAP, a theoretically-grounded framework that extends Variational AutoEncoders (VAEs) by enforcing our conditions through proper constraints in causal process prior. Experimental results on various datasets demonstrate that temporally causal latent processes are reliably identified from observed variables under different dependency structures and that our approach considerably outperforms baselines that do not properly leverage history or nonstationarity information. This demonstrates that using temporal information to learn latent processes from their invertible nonlinear mixtures in an unsupervised manner, for which we believe our work is one of the first, seems promising even without sparsity or minimality assumptions. △ Less

Submitted 8 February, 2022; v1 submitted 11 October, 2021; originally announced October 2021.

Comments: ICLR 2022: https://openreview.net/forum?id=RDlLMjLJXdq

Multi-directional Bicycle Robot for Steel Structure Inspection

Authors: Son Thanh Nguyen, Hai Nguyen, Son Tien Bui, Van Anh Ho, Hung Manh La

Abstract: This paper presents a novel design of a multi-directional bicycle robot, which targets inspecting general ferromagnetic structures including complex-shaped structures. The locomotion concept is based on arranging two magnetic wheels in a bicycle-like configuration with two independent steering actuators. This configuration allows the robot to possess multi-directional mobility. An additional free… ▽ More This paper presents a novel design of a multi-directional bicycle robot, which targets inspecting general ferromagnetic structures including complex-shaped structures. The locomotion concept is based on arranging two magnetic wheels in a bicycle-like configuration with two independent steering actuators. This configuration allows the robot to possess multi-directional mobility. An additional free joint helps the robot naturally adapt to non-flat and complex surfaces of steel structures. The robot has the biggest advantage to be mechanically simple with high mobility. Besides, the robot is equipped with sensing tools for structure health monitoring. We demonstrate the deployment of our robot to perform steel rust detection on steel bridges. The final inspection results are visualized as 3D models of the bridges together with marked locations of detected rusty areas. △ Less

Submitted 27 March, 2021; v1 submitted 21 March, 2021; originally announced March 2021.

Comments: Under review at IROS 2021

BPActuators: Lightweight and Low-Cost Soft Actuators by Balloons and Plastics

Authors: Qiukai Qi, Shogo Yoshida, Genki Kakihana, Takuma Torii, Van Anh Ho, Haoran Xie

Abstract: To increase the awareness and impact, soft robotics needs to go beyond the lab environment and should be readily accessible to those even with no robotic expertise. However, most prevailing manufacturing methodologies require either professional equipment or materials that are not usually available to common people, thereby constraining the accessibility of soft robotics. In this communication, we… ▽ More To increase the awareness and impact, soft robotics needs to go beyond the lab environment and should be readily accessible to those even with no robotic expertise. However, most prevailing manufacturing methodologies require either professional equipment or materials that are not usually available to common people, thereby constraining the accessibility of soft robotics. In this communication, we propose a lightweight and low-cost soft bending actuator, called BPActuator, that can be easily fabricated with plastics and balloons. We fabricated a range of actuators with various morphology for characterization in terms of deformation and load-bearing capacity, and demonstrated that they can bend up to 35 degrees and exert force at the tip around 0.070$\pm$0.015N, which is over 5 times higher than their average gravity. We further implemented a gripper with three fingers using the proposed actuators, and found that the gripper can realize human-like grasp of a range of daily objects. The gripper can lift objects at least 8 times heavier than its own weight. Furthermore, the BPActuator is cost effective and each costs about 0.22 USD. Given these advantages, the BPActuators are expected to significantly improve the accessibility of soft robotics to a wider group without robotic expertise. △ Less

Submitted 4 March, 2021; v1 submitted 1 March, 2021; originally announced March 2021.

Comments: Accepted to the 4th IEEE International Conference on Soft Robotics (RoboSoft), IEEE copyright

Generative latent neural models for automatic word alignment

Authors: Anh Khoa Ngo Ho, François Yvon

Abstract: Word alignments identify translational correspondences between words in a parallel sentence pair and are used, for instance, to learn bilingual dictionaries, to train statistical machine translation systems or to perform quality estimation. Variational autoencoders have been recently used in various of natural language processing to learn in an unsupervised way latent representations that are usef… ▽ More Word alignments identify translational correspondences between words in a parallel sentence pair and are used, for instance, to learn bilingual dictionaries, to train statistical machine translation systems or to perform quality estimation. Variational autoencoders have been recently used in various of natural language processing to learn in an unsupervised way latent representations that are useful for language generation tasks. In this paper, we study these models for the task of word alignment and propose and assess several evolutions of a vanilla variational autoencoders. We demonstrate that these techniques can yield competitive results as compared to Giza++ and to a strong neural network alignment system for two language pairs. △ Less

Submitted 28 September, 2020; originally announced September 2020.

Journal ref: The Association for Machine Translation in the Americas, Oct 2020, Florida, United States

Neural Baselines for Word Alignment

Authors: Anh Khoa Ngo Ho, François Yvon

Abstract: Word alignments identify translational correspondences between words in a parallel sentence pair and is used, for instance, to learn bilingual dictionaries, to train statistical machine translation systems , or to perform quality estimation. In most areas of natural language processing, neural network models nowadays constitute the preferred approach, a situation that might also apply to word alig… ▽ More Word alignments identify translational correspondences between words in a parallel sentence pair and is used, for instance, to learn bilingual dictionaries, to train statistical machine translation systems , or to perform quality estimation. In most areas of natural language processing, neural network models nowadays constitute the preferred approach, a situation that might also apply to word alignment models. In this work, we study and comprehensively evaluate neural models for unsupervised word alignment for four language pairs, contrasting several variants of neural models. We show that in most settings, neural versions of the IBM-1 and hidden Markov models vastly outperform their discrete counterparts. We also analyze typical alignment errors of the baselines that our models overcome to illustrate the benefits-and the limitations-of these new models for morphologically rich languages. △ Less

Submitted 28 September, 2020; originally announced September 2020.

Comments: The 16th International Workshop on Spoken Language Translation, Nov 2019, Hong Kong, Hong Kong SAR China

TensorFlow Quantum: A Software Framework for Quantum Machine Learning

Authors: Michael Broughton, Guillaume Verdon, Trevor McCourt, Antonio J. Martinez, Jae Hyeon Yoo, Sergei V. Isakov, Philip Massey, Ramin Halavati, Murphy Yuezhen Niu, Alexander Zlokapa, Evan Peters, Owen Lockwood, Andrea Skolik, Sofiene Jerbi, Vedran Dunjko, Martin Leib, Michael Streif, David Von Dollen, Hongxiang Chen, Shuxiang Cao, Roeland Wiersema, Hsin-Yuan Huang, Jarrod R. McClean, Ryan Babbush, Sergio Boixo , et al. (4 additional authors not shown)

Abstract: We introduce TensorFlow Quantum (TFQ), an open source library for the rapid prototyping of hybrid quantum-classical models for classical or quantum data. This framework offers high-level abstractions for the design and training of both discriminative and generative quantum models under TensorFlow and supports high-performance quantum circuit simulators. We provide an overview of the software archi… ▽ More We introduce TensorFlow Quantum (TFQ), an open source library for the rapid prototyping of hybrid quantum-classical models for classical or quantum data. This framework offers high-level abstractions for the design and training of both discriminative and generative quantum models under TensorFlow and supports high-performance quantum circuit simulators. We provide an overview of the software architecture and building blocks through several examples and review the theory of hybrid quantum-classical neural networks. We illustrate TFQ functionalities via several basic applications including supervised learning for quantum classification, quantum control, simulating noisy quantum circuits, and quantum approximate optimization. Moreover, we demonstrate how one can apply TFQ to tackle advanced quantum learning tasks including meta-learning, layerwise learning, Hamiltonian learning, sampling thermal states, variational quantum eigensolvers, classification of quantum phase transitions, generative adversarial networks, and reinforcement learning. We hope this framework provides the necessary tools for the quantum computing and machine learning research communities to explore models of both natural and artificial quantum systems, and ultimately discover new quantum algorithms which could potentially yield a quantum advantage. △ Less

Submitted 26 August, 2021; v1 submitted 5 March, 2020; originally announced March 2020.

Comments: 56 pages, 34 figures, many updates throughout the manuscript, several new sections are added

Emotion Recognition for Vietnamese Social Media Text

Authors: Vong Anh Ho, Duong Huynh-Cong Nguyen, Danh Hoang Nguyen, Linh Thi-Van Pham, Duc-Vu Nguyen, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

Abstract: Emotion recognition or emotion prediction is a higher approach or a special case of sentiment analysis. In this task, the result is not produced in terms of either polarity: positive or negative or in the form of rating (from 1 to 5) but of a more detailed level of analysis in which the results are depicted in more expressions like sadness, enjoyment, anger, disgust, fear, and surprise. Emotion re… ▽ More Emotion recognition or emotion prediction is a higher approach or a special case of sentiment analysis. In this task, the result is not produced in terms of either polarity: positive or negative or in the form of rating (from 1 to 5) but of a more detailed level of analysis in which the results are depicted in more expressions like sadness, enjoyment, anger, disgust, fear, and surprise. Emotion recognition plays a critical role in measuring the brand value of a product by recognizing specific emotions of customers' comments. In this study, we have achieved two targets. First and foremost, we built a standard Vietnamese Social Media Emotion Corpus (UIT-VSMEC) with exactly 6,927 emotion-annotated sentences, contributing to emotion recognition research in Vietnamese which is a low-resource language in natural language processing (NLP). Secondly, we assessed and measured machine learning and deep neural network models on our UIT-VSMEC corpus. As a result, the CNN model achieved the highest performance with the weighted F1-score of 59.74%. Our corpus is available at our research website. △ Less

Submitted 26 January, 2020; v1 submitted 21 November, 2019; originally announced November 2019.

Comments: PACLING 2019

Journal ref: In Proceeding of PACLING 2019

Establishing the Quantum Supremacy Frontier with a 281 Pflop/s Simulation

Authors: Benjamin Villalonga, Dmitry Lyakh, Sergio Boixo, Hartmut Neven, Travis S. Humble, Rupak Biswas, Eleanor G. Rieffel, Alan Ho, Salvatore Mandrà

Abstract: Noisy Intermediate-Scale Quantum (NISQ) computers are entering an era in which they can perform computational tasks beyond the capabilities of the most powerful classical computers, thereby achieving "Quantum Supremacy", a major milestone in quantum computing. NISQ Supremacy requires comparison with a state-of-the-art classical simulator. We report HPC simulations of hard random quantum circuits (… ▽ More Noisy Intermediate-Scale Quantum (NISQ) computers are entering an era in which they can perform computational tasks beyond the capabilities of the most powerful classical computers, thereby achieving "Quantum Supremacy", a major milestone in quantum computing. NISQ Supremacy requires comparison with a state-of-the-art classical simulator. We report HPC simulations of hard random quantum circuits (RQC), which have been recently used as a benchmark for the first experimental demonstration of Quantum Supremacy, sustaining an average performance of 281 Pflop/s (true single precision) on Summit, currently the fastest supercomputer in the World. These simulations were carried out using qFlex, a tensor-network-based classical high-performance simulator of RQCs. Our results show an advantage of many orders of magnitude in energy consumption of NISQ devices over classical supercomputers. In addition, we propose a standard benchmark for NISQ computers based on qFlex. △ Less

Submitted 6 May, 2020; v1 submitted 1 May, 2019; originally announced May 2019.

Comments: The paper has been published in Quantum Science and Technology

Journal ref: Quantum Science and Technology 5, 3 (2020)

Lost in the Digital Wild: Hiding Information in Digital Activities

Authors: Shujun Li, Anthony T. S. Ho, Zichi Wang, Xinpeng Zhang

Abstract: This paper presents a new general framework of information hiding, in which the hidden information is embedded into a collection of activities conducted by selected human and computer entities (e.g., a number of online accounts of one or more online social networks) in a selected digital world. Different from other traditional schemes, where the hidden information is embedded into one or more sele… ▽ More This paper presents a new general framework of information hiding, in which the hidden information is embedded into a collection of activities conducted by selected human and computer entities (e.g., a number of online accounts of one or more online social networks) in a selected digital world. Different from other traditional schemes, where the hidden information is embedded into one or more selected or generated cover objects, in the new framework the hidden information is embedded in the fact that some particular digital activities with some particular attributes took place in some particular ways in the receiver-observable digital world. In the new framework the concept of "cover" almost disappears, or one can say that now the whole digital world selected becomes the cover. The new framework can find applications in both security (e.g., steganography) and non-security domains (e.g., gaming). For security applications we expect that the new framework calls for completely new steganalysis techniques, which are likely more complicated, less effective and less efficient than existing ones due to the need to monitor and analyze the whole digital world constantly and in real time. A proof-of-concept system was developed as a mobile app based on Twitter activities to demonstrate the information hiding framework works. We are developing a more hybrid system involving several online social networks. △ Less

Submitted 8 September, 2018; originally announced September 2018.

Comments: 11 pages, 4 figures, accepted to MPS 2018 (2nd International Workshop on Multimedia Privacy and Security)

Distributed Caching for Complex Querying of Raw Arrays

Authors: Weijie Zhao, Florin Rusu, Bin Dong, Kesheng Wu, Anna Y. Q. Ho, Peter Nugent

Abstract: As applications continue to generate multi-dimensional data at exponentially increasing rates, fast analytics to extract meaningful results is becoming extremely important. The database community has developed array databases that alleviate this problem through a series of techniques. In-situ mechanisms provide direct access to raw data in the original format---without loading and partitioning. Pa… ▽ More As applications continue to generate multi-dimensional data at exponentially increasing rates, fast analytics to extract meaningful results is becoming extremely important. The database community has developed array databases that alleviate this problem through a series of techniques. In-situ mechanisms provide direct access to raw data in the original format---without loading and partitioning. Parallel processing scales to the largest datasets. In-memory caching reduces latency when the same data are accessed across a workload of queries. However, we are not aware of any work on distributed caching of multi-dimensional raw arrays. In this paper, we introduce a distributed framework for cost-based caching of multi-dimensional arrays in native format. Given a set of files that contain portions of an array and an online query workload, the framework computes an effective caching plan in two stages. First, the plan identifies the cells to be cached locally from each of the input files by continuously refining an evolving R-tree index. In the second stage, an optimal assignment of cells to nodes that collocates dependent cells in order to minimize the overall data transfer is determined. We design cache eviction and placement heuristic algorithms that consider the historical query workload. A thorough experimental evaluation over two real datasets in three file formats confirms the superiority -- by as much as two orders of magnitude -- of the proposed framework over existing techniques in terms of cache overhead and workload execution time. △ Less

Submitted 16 March, 2018; originally announced March 2018.

"Flow Size Difference" Can Make a Difference: Detecting Malicious TCP Network Flows Based on Benford's Law

Authors: Aamo Iorliam, Santosh Tirunagari, Anthony T. S. Ho, Shujun Li, Adrian Waller, Norman Poh

Abstract: Statistical characteristics of network traffic have attracted a significant amount of research for automated network intrusion detection, some of which looked at applications of natural statistical laws such as Zipf's law, Benford's law and the Pareto distribution. In this paper, we present the application of Benford's law to a new network flow metric "flow size difference", which have not been st… ▽ More Statistical characteristics of network traffic have attracted a significant amount of research for automated network intrusion detection, some of which looked at applications of natural statistical laws such as Zipf's law, Benford's law and the Pareto distribution. In this paper, we present the application of Benford's law to a new network flow metric "flow size difference", which have not been studied before by other researchers, to build an unsupervised flow-based intrusion detection system (IDS). The method was inspired by our observation on a large number of TCP flow datasets where normal flows tend to follow Benford's law closely but malicious flows tend to deviate significantly from it. The proposed IDS is unsupervised, so it can be easily deployed without any training. It has two simple operational parameters with a clear semantic meaning, allowing the IDS operator to set and adapt their values intuitively to adjust the overall performance of the IDS. We tested the proposed IDS on two (one closed and one public) datasets, and proved its efficiency in terms of AUC (area under the ROC curve). Our work showed the "flow size difference" has a great potential to improve the performance of any flow-based network IDSs. △ Less

Submitted 20 January, 2017; v1 submitted 14 September, 2016; originally announced September 2016.

Comments: 13 pages, 3 figures

ACM Class: C.2; K.6.5

Detecting and Preventing "Multiple-Account" Cheating in Massive Open Online Courses

Authors: Curtis G. Northcutt, Andrew D. Ho, Isaac L. Chuang

Abstract: We describe a cheating strategy enabled by the features of massive open online courses (MOOCs) and detectable by virtue of the sophisticated data systems that MOOCs provide. The strategy, Copying Answers using Multiple Existences Online (CAMEO), involves a user who gathers solutions to assessment questions using a "harvester" account and then submits correct answers using a separate "master" accou… ▽ More We describe a cheating strategy enabled by the features of massive open online courses (MOOCs) and detectable by virtue of the sophisticated data systems that MOOCs provide. The strategy, Copying Answers using Multiple Existences Online (CAMEO), involves a user who gathers solutions to assessment questions using a "harvester" account and then submits correct answers using a separate "master" account. We use "clickstream" learner data to detect CAMEO use among 1.9 million course participants in 115 MOOCs from two universities. Using conservative thresholds, we estimate CAMEO prevalence at 1,237 certificates, accounting for 1.3% of the certificates in the 69 MOOCs with CAMEO users. Among earners of 20 or more certificates, 25% have used the CAMEO strategy. CAMEO users are more likely to be young, male, and international than other MOOC certificate earners. We identify preventive strategies that can decrease CAMEO rates and show evidence of their effectiveness in science courses. △ Less

Submitted 8 September, 2015; v1 submitted 24 August, 2015; originally announced August 2015.

OR-Benchmark: An Open and Reconfigurable Digital Watermarking Benchmarking Framework

Authors: Hui Wang, Anthony TS Ho, Shujun Li

Abstract: Benchmarking digital watermarking algorithms is not an easy task because different applications of digital watermarking often have very different sets of requirements and trade-offs between conflicting requirements. While there have been some general-purpose digital watermarking benchmarking systems available, they normally do not support complicated benchmarking tasks and cannot be easily reconfi… ▽ More Benchmarking digital watermarking algorithms is not an easy task because different applications of digital watermarking often have very different sets of requirements and trade-offs between conflicting requirements. While there have been some general-purpose digital watermarking benchmarking systems available, they normally do not support complicated benchmarking tasks and cannot be easily reconfigured to work with different watermarking algorithms and testing conditions. In this paper, we propose OR-Benchmark, an open and highly reconfigurable general-purpose digital watermarking benchmarking framework, which has the following two key features: 1) all the interfaces are public and general enough to support all watermarking applications and benchmarking tasks we can think of; 2) end users can easily extend the functionalities and freely configure what watermarking algorithms are tested, what system components are used, how the benchmarking process runs, and what results should be produced. We implemented a prototype of this framework as a MATLAB software package and used it to benchmark a number of digital watermarking algorithms involving two types of watermarks for content authentication and self-restoration purposes. The benchmarking results demonstrated the advantages of the proposed benchmarking framework, and also gave us some useful insights about existing image authentication and self-restoration watermarking algorithms which are an important but less studied topic in digital watermarking. △ Less

Submitted 5 June, 2015; v1 submitted 31 May, 2015; originally announced June 2015.