A Little Graph Theory for the Busy Developer - Jim Webber @ GraphConnect Chicago 2013

Editor's Notes

1. Vanity slide - Work for Neo Technology, the commercial backer of the Neo4j open source graph database
2. We’ve all been there – trudging through 3 intermediate tables just to get the data you want.
3. We have key-value stores, typically very highly available and scalable for simple key-value data
4. Column stores naturally-indexed value storesContrary to common belief – Google’s big table isn’t the world’s most famous column storeBritish museum London is: it’s got columns and it’s where we stored all the stuff we nicked from the British Empire!
5. Aggregates:No more impedance mismatch, just decompose to documents/ k-v / columnsThe data model is less expressive than RDBMS, but that’s OK because:RDBMS constraints don’t really match application concernsThe expressiveness is in your domain logic, right?
6. People talk about Codd’s relational model being mature because it was proposed in 1969 – 42 years old.Euler’s graph theory was proposed in 1736 – 275 years old.
7. Graphs are the most natural way to model most domains. You already know this because you draw graphs on a whiteboard, but you’ve never had the opportunity to take that down into the database before.Nodes are a bit like documents, but they’re flat at present in Neo4j.You pour data into your nodes and then connect them – easy peasy.This enables high fidelity domain modeling because this is how your domains work.And you don’t have to do this stuff in your application code – it’s right there in the databaseLet’s prove it by exploring a fun domain…
8. Start off sketching the domain. That’s your model done – we see this when we revisit databases months after they’re been designed and put into productionNo decomposition, ER design, normalisation/denormalisation as you need with RDBMS.
9. Predictive analytics
10. Euler reduced the problem into an abstract formFrom geography to diagrammatic through to a graph.It’s the first documented case of representing the real world as a graph.
11. First we need to talk about some local properties
12. A triadic closure is a local property of (social) graphs whereby if two nodes are connected via a path involving a third node, there is an increased likelihood that the two nodes will become directly connected in future. This is a familiar enough situation for us in a social setting whereby if we happen to be friends with two people, ultimately there's an increased chance that those people will become direct friends too, since by being our friend in the first place, it's an indication of social similarity and suitability. It’s called triadic closure, because we try to close the triangle.
13. We see this all the time – it’s likely that if we have two friends, that they will also become at least acquaintances and potentially friends themselves!In general, if a node A has relationships to B & C then the relationship between B&C is likely to form – especially if the existing relationships are both strong.This is an incredibly strong assertion and will not be typically upheld by all subgraphs in a graph. Nonetheless it is sufficiently commonplace (particularly in social networks) to be trusted as a predictive aid.
14. Sentiment plays a role in how closures form too – there is a notion of balance.
15. From a triadic closure perspective this is OK, but intuitively it seems odd.Cartman’s friends shouldn’t be friends with his enemies. Nor should Cartman’s enemies be friends with his friends.
16. This makes sense – Cartman’s friend Craig is also an enemy of Cartman’s enemy TweekTwo negative sentiments and one positive sentiment is a balanced structure – and it makes sense too since we gang up with our friends on our poor beleaguered enemy
17. Another balanced – and more pleasant – arrangement is for three positive sentiments, in this case mutual friends.
18. A starting point for a network of friends and enemiesRed links indicate enemy of relationshipBlack links indicate friend of relationshipThe Three Emperor’s league
19. Italy forms the with Austria and Germany – a balanced +++ triadic closureIf Italy had made only a single alliance (or enemy) it would have been unstable and another relationship would be likely to form anyway!Triple Alliance
20. Russia becomes hostile to Austria and Germany – a balance --+ d triadic closure becomes agnostic towards France.German-Russian Lapse
21. The French and Russians ally, forming a balanced --+ triadic closure with the UKFrench-Russian Alliance
22. The UK and France enter into the famous Entente CordialeThis produces an unbalanced ++- triadic closure with Russia, and the graph doesn’t like it.
23. The British and Russians form an alliance, thereby changing their previously unbalanced triadic closure into a balanced one.Other local pressures on the graph make other closures form.Italy becomes hostile to Russia, forming a balanced --+ closure with the France, and another balanced --+ closure with the UK.Germany and the UK become hostile forming a balanced --+ closure with Austria and another balanced --+ closure with ItalyBritish-Russian Alliance
24. That WWI can be predicted without domain knowledge by iterating a graph and applying local structural constraints is nothing short of astonishing to me.Note how the network slides into a balanced labeling — and into World War I.
25. In this case the string triadic closure property still holds – though it is a weak link that characterises the relationship between Stan and Cartman.Given a starting graph, we can apply this simple local principal to see how it would evolve.
26. In this case the string triadic closure property still holds – though it is a weak link that characterises the relationship between Stan and Cartman.Given a starting graph, we can apply this simple local principal to see how it would evolve.
27. A local bridge acts as a link – perhaps the only realistic link - between two otherwise distant (or separate) subgraphs.Local bridges are semantically rich – they provide conduits for information flow between otherwise independent groups.In this case DATING is a local bridge – it must also be a weak relationship according to our definition of a local bridgeIntuitively this makes sense – your girl/boyfriend is rather less important at age 8 than your regular friends, IIRC.
28. How do we identify local bridges? Any weak link which would cause a component of the graph to become disconnected.Being able to identify local bridges is important – in this case it’s the only know conduit to allow the girls and boys to communicate.In real life local bridges are apparent in your organisation as experts (or managers); appear as nexus in fraud cases;
29. Zachary in the Journal of Anthropological Research 1977Intuitively we can see “clumps” in this graph.But how do we separate them out? It’s called minimum cut.
30. What’s interesting is that it’s mechanical – no domain knowledge is necessary. There’s only one failure with the method Zachary chose to partition the graph: node 9 should have gone to the instructor’s club but instead went with the original president of the club (node 34).Why? Because the student was three weeks away from completing a four-year quest to obtain a black belt, which he could only do with the instructor (node 1)Other minimum cut approaches might deliver slightly different results, but on the whole it’s amazing you get such insight from an algorithm!
31. But is there enough information in the graph itself to predict the schism?
32. But is there enough information in the graph itself to predict the schism?
33. We can use graph matching to look for behavioural patterns in the graph too!
34. The insight here is that we have a typical young father who buys beer, nappies and a game console simply by reducing subgraphWe have a pattern to search for
35. Now we look for young fathers – implied by beer and nappies purchases – who haven’t bought a game console.
36. START n=node(*), r=rel(*) DELETE r,nCREATE daddy1 = { name: 'Mickey Smith', dob: 19781006 }CREATE beer = { category: 'beer' } CREATE alcohol = { category: 'alcoholicdrinks' } CREATE peeweePilsner = { sku: '2555f258', product: 'PeeweePilsner' } CREATE badgersNadgers = { sku: '5e175641', product: 'BadgersNadgers Ale' }CREATE peeweePilsner-[:MEMBER_OF]->beerCREATE badgersNadgers-[:MEMBER_OF]->beerCREATE beer-[:MEMBER_OF]->alcoholCREATE daddy1-[:BOUGHT]->peeweePilsnerCREATE daddy1-[:BOUGHT]->badgersNadgersCREATE baby = { category: 'baby' }CREATE nappies = { category: 'nappies' }CREATE nappies-[:MEMBER_OF]->babyCREATE babyDryNights = { sku: '49d102bc', product: 'Baby DryNights'} CREATE babyDryNights-[:MEMBER_OF]-nappiesCREATE daddy1-[:BOUGHT]->babyDryNightsCREATE consumerElectronics = { category: 'consumerelectronics' }CREATE console = { category: 'console' }CREATE xbox = { sku: '49d102bc', product: 'XBox 360' }CREATE xbox-[:MEMBER_OF]->(console)-[:MEMBER_OF]->consumerElectronicsCREATE daddy1-[:BOUGHT]->xboxCREATE mummy11 = { name: 'Rose Tyler', dob: 19800317 }CREATE wine = { sku:'3a3f22bc', product: 'Shiraz' }CREATE mummy1-[:BOUGHT]->wineCREATE mummy1-[:BOUGHT]->babyDryNights CREATE daddy2 = { name: 'Rory Williams', dob: 19880121 }CREATE daddy2-[:BOUGHT]->peeweePilsnerCREATE daddy2-[:BOUGHT]->babyDryNightsSTART beer=node(2), nappies=node(7), xbox=node(11)MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer), (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies), (daddy)-[b?:BOUGHT]->(xbox)WHERE b isnullRETURN distinctdaddy
37. Note that Max de Marzireimplemented a functionally better Graph Search with Neo4j and some Ruby gems for language processing in a weekend!
38. Image: real-timeBeing able to look for all young fathers who might be tempted to buy a new game console is helpful, but not dramatically different from what we have nowIt’s much faster to process in a graph, but still a latent business activity (e.g. mailshot)But you can take the same idea and run the query in real time foran individual customer at they go through the checkout!And you can do more too – you can add in more/less dimensions to the search.Does the shop have stock?Does the young father live in the right target area (r-tree, postcode)?We can vary the number of dimensions we include to tailor the search for performance/accuracy very easily in a graph – query latency is proportional to the amount of graph searched, not data set size.