Scylla Summit 2017: The Upcoming HPC Evolution

PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
The Upcoming HPC Evolution
Los Alamos National Lab, Scientist/HPC Engineer
Joseph ‘Joshi’ Fullop
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
2
Joshi a high-performance computing (HPC)
scientist/engineer in scheduling, monitoring, and
analytics at Los Alamos National Lab.
Formerly with the National Center for
Supercomputing Applications (NCSA) for 19 years.
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
▪ Department of Energy (DoE) weapons laboratory with priorities set by National Nuclear
Security Administration (NNSA)
▪ Established in 1943 as site Y of the Manhattan Project
▪ In 1992, the U.S. stopped full-scale testing of nuclear weapons, which means using other
methods to assure the safety of the U.S. stockpile and requiring a tremendous amount of
science, technology, computational horsepower, and analytical tools.
▪ We perform work for the Department of Defense (DoD), Intelligence Community (IC), and
Department of Homeland Security (DHS), among others. As a result, our strategy reflects
U.S. priorities spanning nuclear security, intelligence, defense, emergency response,
nonproliferation, counterterrorism, energy security, emerging threats, and environmental
management.
3LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
SECTION 1
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
What is HPC?
▪ High Performance Computing
o Delivering extreme scale, parallel, floating-point operations for scientific, engineering
and business endeavors using supercomputers.
o 10s of thousands servers, 300k-1M+ cores, ~20PetaFlops, ~80PB storage
o Many different combinations and use of technologies to serve the machine’s purpose,
which is driven by the applications that is serves. (e.g. GPUs, FPGAs)
o Application architectures mostly based on Message Passing Interface (MPI)
▪ 20 PetaFlops: If you did one 14-digit calculation per second without
stopping to eat or sleep, it would take 634 million years to do what a
supercomputer can do in 1 second.
5LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Two Examples of HPC Supercomputers
6
▪ Trinity @ LANL
~20k nodes
http://www.lanl.gov/projects/trinity/
▪ Blue Waters @ NCSA
~27k nodes
http://www.ncsa.illinois.edu/enabling/bluewaters
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
HPV vs Cloud Computing
▪ Some vast over generalizations
o Network
• HPC: Extremely low latency AND High Bandwidth. Multi-dimensional architectures.
• Cloud: Tends to be more commodity based.
o Parallel File System
• HPC: Focal point of contention. Major factor of scalability.
• Cloud: May not even be required.
o Workload
• HPC: GB in / TB out. Not always embarrassingly parallel.
• Cloud: GB or TB in / KB out. Tends to be more parallel or asynchronous by nature.
o Infrastructure
• HPC: Generally persistent
• Cloud: Can be highly dynamic.
7LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
New HPC Application Architecture
8
▪ Traditional HPC applications
oSimulation based
• Create a model.
• Feed it a data set.
• Turn the crank.
• Analyze what comes out.
oMeasured by number of
calculations and model size.
▪ Data Analytics applications
oQuery based
• Pose a question.
• Traverse dataset.
• Do statistics.
• Make a plausible
determination.
oMeasured by the amount of
data assessed.
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Supercomputer Job Cycle
9
Job
Launch
Compute
Store
Results
Visualize
Job Launch
• Massive number & bandwidth of reads
• Node communication & synchronization
Compute
• Requires very low latency network
• Intolerant to interruptions (jitter)
• Potential check-pointing (burst writes)
Store Results
• Massive file system write(s)
Visualize
• Significantly different I/O patterns
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Opportunity in HPC
1. Data engines become part of the infrastructure
- Applications will need to be re-written
- Separate budget line item – requires proven need
2. Data engines provide a file system interface
Immediate adoption
Prove in place
Potential game changer – handle burst usage
Will need a Consistency sync function
10LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Benefits of Containers in HPC
• Custom Environment for Users
• Not all applications benefit from changes
• No Conflicting or evolving default packages
• Consistent, Reproducible Workflow
• Fewer things changing under the application
• More defendable performance comparisons
• Fewer required application rebuilds
• Some applications take over a day to rebuild
• Unknown when/if a rebuild may be required
11LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Hurdles for Containers in HPC
• Security
• Enforcement of required patches
• Restriction of certain software
• Localized node security
• Perceived performance penalty
• Environment management at scale
12LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Dilemma for Many HPC Sites
• Provide cycles for legacy/mission codes
• These pay the bills
• Codes do not change much
• Adverse to change in environment
• Facilitate new applications & investigations
• Minimize ‘Time to Science’ by making the systems easy to learn and use
• Embrace new architectures
13LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Virtual Cluster Architecture
14
Scalable
Database
Virtual HPC Cluster Visualization
Cluster
V I R T U A L M A C H I N E R E S O U R C E S
B A R E M E T A L M A C H I N E S
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Custom Container Cluster Architecture
15
Job 123 Job 456
Scalable Database Compute Nodes Visualization Compute Nodes
B A R E M E T A L M A C H I N E S
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Relief for Parallel File System
• Burst Buffers / Data Warp
• Regional SSD file system for staging data into/out of parallel file system
• Provide large data sets to applications
• Local OS buffering
• Add local storage to each node
• Use out of band service network to pre-stage container directly onto node
• Job launch now from local storage
• Normal launch from network storage is a scalability limiter
• Greatly reduced launch latency
16LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
17LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
SECTION 2
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
As clusters continue to scale…
• Management of these resources also needs to scale
• “If you don’t monitor it, you can’t manage it.”
• Monitoring efforts need to scale at:
• Collection
• Aggregation
• Storage
• Query time
• Visualization
19LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Types of Monitoring Data
• Events
• Something that occurred at a time and location
• Syslog … and application logs … and external events
• Metrics
• A measurable value that can be assessed over time
• Context/Job
• What was running on what nodes at what time
20LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Audiences for Monitoring Data
• Admins
• Track the current health of the system
• Users
• See how their jobs used the allocated resources
• Code Developers
• Locate bottlenecks and inefficiencies in their codes
• System Designers
• Determine where to improve performance
• Researchers
• To find the location of the Holy Grail?
21LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Feeds and Speeds
• Logs
• Average in the 10s of GB/day, can peak at 200GB+/day.
• Asynchronous: Can have huge spikes.
• Metrics
• 20k nodes * 200 metrics/node/second = 4M metrics/sec (4TB/day)
• Can easily step up to 10Hz
• Fairly regular flow rate.
• Multiply that by 12-15 systems (*not all quite that size)
22LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
What to do with the data?
23
Visualization Understanding Automation
LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
24LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Workload Throughput
25LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
HELO & Baler
• Machine Learning log classification systems
• Builds a central template library and uses that to pattern match incoming logs messages.
• Tags each log message with the template integer ID (much easier to use later).
• If a log message does not match a pattern, it is analyzed against the template library to see if
there is a very similar one that could be modified to include the new one.
- If a close candidate is found, the template is modified and the message is tagged.
- If there is no close match, a new template with ID is created and used to tag the message.
• Makes possible various types of analyses
• Event macro-event detection (fingerprints).
• Correlation and Cascades
• Event Prediction
26LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Log Comparator
27LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Log Comparator
28LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
29LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Event Correlation & Prediction
30LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Bringing it all together
• Management tools and methods need to scale.
• Provisioning & Job Launch
• Monitoring Data Storage & Access
• Container Management
• Scalable visualizations are needed.
• We need to leverage each others successes.
31LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
Parting Thought
High Performance Computing and Cloud Computing are like teen siblings that
share a lot of clothes but play different sports. Yet, at the end of the day, it is
good to come home for dinner together.
32LA-UR-17-29616

AND ON TWO LINES
First and last name
Position, company
THANK YOU
fullop@lanl.gov
Please stay in touch
Any questions?
http://hpc.lanl.gov
LA-UR-17-29616

Scylla Summit 2017: The Upcoming HPC Evolution

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Scylla Summit 2017: The Upcoming HPC Evolution

Similar to Scylla Summit 2017: The Upcoming HPC Evolution (20)

More from ScyllaDB

More from ScyllaDB (20)

Recently uploaded

Recently uploaded (20)

Scylla Summit 2017: The Upcoming HPC Evolution