Graphing Enterprise IT – Representing IT Infrastructure and Business Processes as a Graph - Alan Robertson @ GraphConnect SF 2013

Modeling IT Infrastucture
using
The Assimilation Project
#AssimProj

@OSSAlanR

http://assimproj.org/
http://bit.ly/AssimGC2013
Alan Robertson <alanr@unix.sh>
Assimilation Systems Limited
http://assimilationsystems.com

G
R
A
P
H
C
O
N
N
E
C
T

Upcoming Events
GraphConnect San Francisco (today!)
Open Source Monitoring Conference - Nürnberg
NSA / Homeland Security Assimilation Technical Talk
Large Installation System Administration Conference - DC
Colorado Springs Open Source User’s Group
linux.conf.au – Awesome Australian Linux Conf - Perth
Details on http://assimilationsystems.com/

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013
© 2013 Assimilation Systems Limited

2/25

Talk Outline
●

Project Overview

●

Infrastructure Schema

●

Python Object-Graph-Mapping API

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

3/25

Assimilation Project History
●

Inspired by 2 million core computer (cyclops64)

●

Concerns for extreme scale

●

Topology aware monitoring

●

Topology discovery w/out security issues

=►Discovery of everything!

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

4/25

Project Scope
Zero-network-footprint continuous Discovery
integrated with extreme-scale Monitoring
●

Continuous extensible discovery
–

●

Extensible exception monitoring
–

●

systems, switches, services, dependencies
– zero network footprint

G
R
A
P
H
C
O
N
N
E
C
T

more than 100K systems

All data goes into central graph database

GraphConnect
4 October
2013

5/25

Discovery
Discovering
●

systems you've forgotten

●

what you're not monitoring

●

whatever you'd like

●

without setting off security alarms

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

6/25

Why Discovery?
●

Documentation: incomplete, incorrect

●

Dependencies: unknown

●

Planning: Needs accurate data

●

●

Best Practices: Verification needs
data
ITIL CMDB (Configuration Mgmt
DataBase)

Our Discovery: continuous, low-profile

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

7/25

Why Neo4j (graph db)?
●
●

●

●
●

●

Dependency & Discovery information: graph
Speed of graph traversals depends on size
of subgraph, not total graph size
Root cause queries  graph traversals –
notoriously slow in relational databases
Visualization of relationships
Schema-less design: good for constantly
changing heterogeneous environment
Graph Model === Object Model

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

8/25

Assimilation Communication
Neighbor-Rings

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

9/25

Ring Representation Schema

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

10/25

ssh -> sshd dependency graph

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

11/25

Switch Discovery Data
from LLDP (or CDP)

CRM transforms LLDP (CDP) Data to JSON

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

12/25

OGM – Object Graph Mapping
●

Managing the Graph Nodes “disappears”

●

The Object Model is the Graph Model

●

Significant Improvement in Thinking
Clarity
–

The “mud” is gone

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

13/25

OGM rules
●
●

●

Don't use Constructors
Relationships replace hash tables and
object references and so on
Constructor parameter names match
attribute names

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

14/25

OGM sample
@RegisterGraphClass
class Person(GraphNode):
'A Person - or at least an electronic representation of one'
def __init__(self, firstname, lastname, dateofbirth=None):
GraphNode.__init__(self, domain='global')
self.firstname = firstname
self.lastname = lastname
if dateofbirth is not None:
self.dateofbirth = dateofbirth
else:
self.dateofbirth='unknown'
@staticmethod
def __meta_keyattrs__():
'Return our key attributes in order of significance'
return ['lastname', 'firstname']

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

15/25

OGM sample
@RegisterGraphClass
class TestSystem(GraphNode):
'Some kind of semi-intelligent system'
def __init__(self, designation, roles=[]):
GraphNode.__init__(self, domain)
self.designation = designation.lower()
if roles == []:
roles = ['']
self.roles = roles
def addroles(self, role):
if self.roles[0] == '''':
del self.roles[0]
if isinstance(role, list):
for arole in role:
self.addroles(arole)
elif role not in self.roles:
self.roles.append(role)


G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

16/25

OGM sample
@staticmethod
def __meta_keyattrs__():
'Return our key attributes in order of significance'
return ['designation']

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

17/25

OGM sample
# (seven)-[:formerly]->(Annika)
Annika = store.load_or_create(Person,
firstname='Annika',
lastname='Hansen')
seven = store.load_or_create(Drone,
designation='SevenOfNine', roles='Borg')
store.relate(seven, 'formerly', Annika)
store.commit()

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

18/25

OGM sample
@RegisterGraphClass
class TestDrone(TestSystem):
def __init__(self, designation, roles=[]):
TestSystem.__init__(self, designation=designation)
if isinstance(roles, str):
roles = [roles]
roles.extend(['host', 'Drone'])
System.__init__(self, designation, roles=roles)

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

19/25

Current State
●

First release was April 2013

●

Great unit test infrastructure

●

Nanoprobe code – works well
●

Service monitoring works

●

Lacks digital signatures, encryption, compression

●

Reliable UDP comm code working

●

Several discovery methods written

●

CMA and database code restructuring near-complete

●

G
R
A
P
H
C
O
N
N
E
C
T

UI development underway

●

Licensed under the GPL, commercial license available

GraphConnect
4 October
2013

20/25

Future Plans
●

Production grade by end of year

●

Purchased support

●

“Real digital signatures, compression, encryption

●

Other security enhancements

●

Much more discovery

●

GUI

●

Alerting

●

Reporting

●

Add Statistical Monitoring

●

Best Practice Audits

●

Dynamic (aka cloud) specialization

●

G
R
A
P
H
C
O
N
N
E
C
T

Hundreds more ideas
–

See: https://trello.com/b/OpaED3AT

GraphConnect
4 October
2013

21/25

Get Involved!
Powerful Ideas and Infrastucture
Fun, ground-breaking project
Looking for early adopters, testers!!
Needs for every kind of skill
●
●
●
●
●
●
●

Awesome User Interfaces (UI/UX)
Evangelism, community building
Test Code (simulate 106 servers!)
Python, C, script coding
Documentation
Feedback: Testing, Ideas, Plans
Many others!

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

22/25

Resistance Is Futile!
#AssimProj

@OSSAlanR

#AssimMon

Project Web Site
http://assimproj.org

G
R
A
P
H
C
O
N
N
E
C
T

Blog
techthoughts.typepad.com
lists.community.tummy.com/cgi-bin/mailman/admin/assimilation

GraphConnect
4 October
2013

23/25

The Elder GeekGirl

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

24/25

Younger GeekGirl's Computer

Running Linux
Of Course!

G
R
A
P
H
C
O
N
N
E
C
T

GraphConnect
4 October
2013

25/25

Graphing Enterprise IT – Representing IT Infrastructure and Business Processes as a Graph - Alan Robertson @ GraphConnect SF 2013

More Related Content

What's hot

What's hot (12)

Viewers also liked

Viewers also liked (6)

Similar to Graphing Enterprise IT – Representing IT Infrastructure and Business Processes as a Graph - Alan Robertson @ GraphConnect SF 2013

Similar to Graphing Enterprise IT – Representing IT Infrastructure and Business Processes as a Graph - Alan Robertson @ GraphConnect SF 2013 (20)

More from Neo4j

More from Neo4j (20)

Recently uploaded

Recently uploaded (20)

Graphing Enterprise IT – Representing IT Infrastructure and Business Processes as a Graph - Alan Robertson @ GraphConnect SF 2013