Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all available GPUs in the Cloud

A talk by Rob Emanuele given at FedGeoDay 2016 about using GeoMesa, GeoWave, and GeoTrellis to work with geospatial data on Apache Spark and Accumulo.

The document discusses the CERN OpenStack cloud, which provides compute resources for the Large Hadron Collider experiment at CERN. It details the scale of the cloud, including over 6,700 hypervisors, 190,000 cores, and 20,000 VMs. It also describes the various use cases served, wide range of hardware, and operations of the cloud, including a retirement campaign and network migration to Neutron.

ISC Cloud‘13, Heidelberg (Germany) Sep. 23-24th, 2013 A. Gómez, L.M. Carril, R. Valin, J.C. Mouriño, C. Cotelo

This document summarizes an international collaboration between the National Computational Infrastructure (NCI) in Australia and A*Star in Singapore to accelerate DNA analysis. The collaboration utilizes trans-Pacific extended InfiniBand networks and supercomputers to: 1) Transfer large genetic sequence datasets from NCI in Canberra to A*Star in Singapore for analysis on the A*Star Aurora system and return results. 2) Utilize NCI's InfiniCloud HPC system for visualization of genetic data results produced by Aurora. 3) Demonstrate long distance high-speed data transfers between Australia and Singapore leveraging extended InfiniBand networks.

The document discusses OpenStack at CERN. It provides details on: - OpenStack has been in production at CERN for 3 years, managing over 190,000 cores and 7,000 hypervisors. - Major cultural and technology changes were required and have been successfully addressed to transition to OpenStack. - Contributing back to the upstream OpenStack community has led to sustainable tools and effective technology transfer.

This document summarizes Tim Bell's presentation on OpenStack at CERN. It discusses how CERN adopted OpenStack in 2011 to manage its growing computing infrastructure needs for processing massive data sets from the Large Hadron Collider. OpenStack has since been scaled up to manage over 300,000 CPU cores and 500,000 physics jobs per day across CERN's private cloud. The document also briefly outlines CERN's use of other open source technologies like Ceph and Kubernetes.

In this slidecast, Jason Stowe from Cycle Computing describes the company's recent record-breaking Petascale CycleCloud HPC production run. "For this big workload, a 156,314-core CycleCloud behemoth spanning 8 AWS regions, totaling 1.21 petaFLOPS (RPeak, not RMax) of aggregate compute power, to simulate 205,000 materials, crunched 264 compute years in only 18 hours. Thanks to Cycle's software and Amazon's Spot Instances, a supercomputing environment worth $68M if you had bought it, ran 2.3 Million hours of material science, approximately 264 compute-years, of simulation in only 18 hours, cost only $33,000, or $0.16 per molecule." Learn more: http://blog.cyclecomputing.com/2013/11/back-to-the-future-121-petaflopsrpeak-156000-core-cyclecloud-hpc-runs-264-years-of-materials-science.html Watch the video presentation: http://wp.me/p3RLHQ-aO9

Tim Bell from CERN gave a presentation on "Understanding the Universe through Clouds" at OpenStack UK Days on September 26th, 2017. Some key points: - CERN operates one of the world's largest private OpenStack clouds to support the Large Hadron Collider, with over 8000 hypervisors and 33,000 VMs. - The Worldwide LHC Computing Grid distributes and analyzes LHC data across 600 PB of storage and 750k CPU cores at 170 sites in 42 countries. - CERN has been an early adopter of OpenStack technologies like Nova, Glance, Horizon, and Neutron since 2011 and contributes code back to the community. - New services like Mag

This document discusses OpenStack cloud computing at CERN. It notes that CERN has 4 OpenStack clouds with over 120,000 cores total, and is migrating to the Kilo release of OpenStack. It then describes OpenStack components like Keystone for authentication, Glance for images, Nova for compute, and Cinder for block storage. The document outlines how OpenStack supports federated identity through options like Active Directory, OpenID Connect, and SAML. It provides examples of how federation could allow access to external clouds and shares experiences in deploying federated OpenStack.

CERN operates the largest particle physics laboratory in the world. It manages over 8,000 servers to support its research. In 2012, CERN recognized limits with its existing infrastructure management tools and formed a team to define a new "Agile Infrastructure Project." The project goals were to improve resource provisioning time, enable cloud interfaces, improve monitoring and accounting, and boost efficiency. The team adopted open source tools like OpenStack, Puppet, and Ceph to create a new cloud service spanning two data centers. This allowed on-demand provisioning in minutes versus months and helped CERN better support its expanding computing needs for research.

MACPAC is a federal legislative branch agency tasked with reviewing state and federal Medicaid and Children's Health Insurance Program (CHIP) access and payment policies and making recommendations to Congress. By March 15 and again by June 15 each year, the agency produces a comprehensive report for Congress that compiles results from Medicaid and CHIP data sources for the 50 states and territories. The CIO of MACPAC wanted a secure, cost-effective, high performance platform that met their needs to crunch this large amount of health data. In this session, learn how MACPAC and 8KMiles helped set up the agency’s Big Data/HPC analytics platform on AWS using SAS analytics software.

Wireless sensor networks use many types of wireless sensors to configure network. However batteries in wireless sensor nodes are energy limited and consume considerable energy for data transmission. Therefore, data merging is used as a means to increase energy efficiency in data transmission. In this paper, we propose Differential Data Processing(DDP), which reduces the size of data transmitted from the sensor node to increase the energy efficiency of the wireless sensor network. Experimental results show that processing the differential temperature data reduces the average data size of the sensor node by 30%.

This document introduces BioPig, a Hadoop-based analytic toolkit for large-scale genomic sequence analysis. BioPig aims to provide a flexible, high-level, and scalable platform to enable domain experts to build custom analysis pipelines. It leverages Hadoop's data parallelism to speed up bioinformatics tasks like k-mer counting and assembly. The document demonstrates how BioPig can analyze over 1 terabase of metagenomic data using just 7 lines of code, much more simply than alternative MPI-based solutions. While challenges remain around optimization and integration, BioPig shows promise for scalable genomic analytics on very large datasets.

This document describes XeMPUPiL, a performance-aware power capping orchestrator for the Xen hypervisor. It aims to maximize performance under a power cap using a hybrid approach. The key challenges addressed are instrumentation-free workload monitoring and balancing hardware and software power management techniques. Experimental results show XeMPUPiL outperforms a pure hardware approach for I/O, memory, and mixed workloads by better balancing efficiency and timeliness. Future work includes integrating the orchestrator logic into the scheduler and exploring new resource assignment policies.

CERN operates the largest machine on Earth, the Large Hadron Collider (LHC), which produces over 1 billion collisions per second and records over 0.5 petabytes of data per day. CERN relies heavily on OpenStack, with over 190,000 CPU cores and 5,000 VMs under OpenStack management, accounting for over 90% of CERN's computing resources. CERN plans to add over 100,000 more CPU cores in the next 6 months and explores using public clouds and containers to help process the massive amount of data generated by the LHC.