Xtreme Deployment
- 2. Who are we?
Brian Clapper
CTO of Circonus
Whipping boy of the modern SaaS stack.
Theo Schlossnagle
CEO of Circonus
Repeat denier of reality.
Tuesday, October 2, 12
- 3. How the world used to look (less xtreme)
Architectures driven by user interaction:
(e.g. web page load ➠ web page response)
Tuesday, October 2, 12
- 4. How the world used to look (less xtreme)
isolated
user A’s page load doesn’t effect user B.
latent
user A’s affect shows up “later” in user B’s view.
Tuesday, October 2, 12
- 5. How the world used to look (less xtreme)
Problem reproducibility is achievable
(even if sometimes painful)
Tuesday, October 2, 12
- 6. How the world looks now (more xtreme)
Architectures driven by
high volume data and users
Tuesday, October 2, 12
- 7. How the world looks now (more xtreme)
every piece of data entering the architecture
can effect a user’s subsequent experience.
Tuesday, October 2, 12
- 8. How the world looks now (more xtreme)
sometimes
the data must effect a user’s
subsequent experience.
Tuesday, October 2, 12
- 9. How the world looks now (more xtreme)
Social systems cause indirect impact
via user action eroding isolation.
Tuesday, October 2, 12
- 10. How the world looks now (more xtreme)
A more responsive web is eroding
users’ acceptance of latent state.
Tuesday, October 2, 12
- 11. How the world looks now (more xtreme)
Highly distributed systems
(due to complexity and scalability) means
no single world state
Tuesday, October 2, 12
- 12. How the world looks now (more xtreme)
Highly distributed systems
(due to complexity and scalability) means
difficult repeatability of state and failures
Tuesday, October 2, 12
- 13. How the world looks now (more xtreme)
Highly distributed systems
(due to complexity and scalability) means
treacherous debugability
Tuesday, October 2, 12
- 14. Despair all ye’ who debug here
I studied distributed systems long enough
to understand “the suck” you do not wish to
have in your lives.
Tuesday, October 2, 12
- 15. Despair all ye’ who debug here
Expedient debugging requires a quick
realization of a precipitating state of failure.
Tuesday, October 2, 12
- 16. Despair all ye’ who debug here
In distributed systems,
post-mortem techniques are less useful
because recreating possible global state is a
mind-bending (or brain-damaging) exercise.
Tuesday, October 2, 12
- 17. Despair all ye’ who debug here
Applying older techniques
to small controlled distributed systems
is still feasible,
but complex, asynchronous architectures
can be nearly impossible.
Tuesday, October 2, 12
- 18. Some context about us
Circonus ingests telemetry data
across an entire business
Tuesday, October 2, 12
- 19. Some context about us
alerting • correlation • visualization • analytics
Tuesday, October 2, 12
- 20. How we see the world today
High velocity non-user-generated data inflow.
Tuesday, October 2, 12
- 21. How we see the world today
Users expect sub-second reflection of
all input data and drawn-conclusions.
Tuesday, October 2, 12
- 22. How we see the world today
Users expect extremely high availability
(100%, you can only disappoint)
Tuesday, October 2, 12
- 23. How we see the world today
I was surprised how
impatient and demanding users are...
until I used the tool...
and felt exactly the same way.
Tuesday, October 2, 12
- 26. Circonus architecture
Voltron
• distributed data collection (via brokers)
• distributed data ingestion (via aggregators)
Tuesday, October 2, 12
- 27. Circonus architecture
Voltron
• distributed data collection (via brokers)
• distributed data ingestion (via aggregators)
• distributed data distribution! (via message queueing)
Tuesday, October 2, 12
- 28. Circonus architecture
Voltron
• distributed data collection (via brokers)
• distributed data ingestion (via aggregators)
• distributed data distribution! (via message queueing)
• distributed data storage (via snowth)
Tuesday, October 2, 12
- 29. Circonus architecture
Voltron
• distributed data collection (via brokers)
• distributed data ingestion (via aggregators)
• distributed data distribution! (via message queueing)
• distributed data storage (via snowth)
• distributed streaming fault detection (via muppet love)
Tuesday, October 2, 12
- 30. Circonus architecture
Voltron
• distributed data collection (via brokers)
• distributed data ingestion (via aggregators)
• distributed data distribution! (via message queueing)
• distributed data storage (via snowth)
• distributed streaming fault detection (via muppet love)
• ... distributed nightmare.
Tuesday, October 2, 12
- 33. Hope (keys to the city)
• Leveraging
• stream mirroring and
• idempotent behavior.
Tuesday, October 2, 12
- 34. Hope
Allows for the recklessness that we all
appreciate in a no-consequences
development environment...
Tuesday, October 2, 12
- 35. Hope
with
production data,
production volume, and
production “personality.”
Tuesday, October 2, 12
- 41. EXPLORATION
Safety & Happiness
Tuesday, October 2, 12
- 42. Hope actualized
Ex #2: Dual streaming analysis tools
Tuesday, October 2, 12
- 49. Hope actualized
Ex #3: Dual messages queueing systems
Tuesday, October 2, 12
- 50. The Future
This is the foundation of
real responsive design
going forward.
Tuesday, October 2, 12
- 51. Thank You
Questions?
Tuesday, October 2, 12