Big Data graph Clustering with Laurence O'Toole - Digital Marketing Show, November 2015

Editor's Notes

1. Preferential Attachment - The Rich get Richer Power Laws When I come back to clustering graph data later – mention Kevin Bacon and 6 degrees of separation Mention Google trying to apply structure to the Web with Entity Search – its Knowledge Graph Mention Google using Rank Brain – Artificial Intelligence – so you need to use Artificial Intelligence for Natural Search
2. Visitors to a website, pages in a site, links to a site/page City sizes Incomes Earthquake magnitudes Frequency of words in most languages The second answer is easy, too. Powerlaws come about when you have three conditions:  Variety  Inequality  Network effects (word of mouth, for example) to amplify the differences between them.    In others words, powerlaw distributions occur where things are different, some are better than others, and network effects can work to promote the good and suppress the bad. This results in what Vilfredo Pareto called the predictable imbalance of markets, culture and society: success breeds success, rich get richer and so on. Needless to say, these forces describe a good fraction of the world around us. http://longtail.typepad.com/the_long_tail/2005/05/powerlaw_101.html
3. How do you get Direct Answers and Knowledge graph results?
4. What it really looks like for a real Market!
5. Every ‘Graph’ has a ‘Point Of Origin’ (a ‘POO’) – Yes, we’d prefer a different acronym but that is what it is called! From the POO you can traverse the breadth and depth of the Graph to understand the relationships between Domains, Hosts, Pages, Keywords, Competing Pages, Competing Keywords and Competing Hosts and Domains. For each Host we analyse all its ranking URLs. From there we look at all the keywords those pages rank for. For each keyword we look for all competing pages (up to 30 per keyword) and find all the keywords that page also ranks for. We then whittle down the list of discovered Hosts to the top 100 by frequency of Common Keywords. We then look at Search Volume x CTR% (for ranking position) to estimate the Total Number of Unique Keywords in the Market, Total Market Demand in terms of Search Volume, Total Market Visibility in terms of estimated organic traffic and Total Market Value in the local currency based on the suggested CPC bid for each keyword in the market. We then calculate the Market Share for each of these measures for each of the 100 hosts.
6. In the process of being reworded Our Hypothesis….. Google’s algorithm effectively builds its own index of the web based on the structure of the web and the relationship between entities across the web, as well as based on consumer’s searching and click-through behaviour. By undertaking graph analysis of the SERPs using the network of ranking URLs for millions of keywords discovered, we are effectively observing the structure created by Google’s ranking algorithms, by highlighting Google’s view of the market (e.g. How it organises the web into related centres of authority) that we interpret and display as clusters – e.g. Communities or hubs of related keywords that correspond to related and relevant content and sites. How a cluster is formed ….. Each cluster should be dissimilar to other clusters in the graph (measured by distance from each other in the graph, weighted by ranking) Each keyword and page in a cluster is put into a cluster based on distance from other items in the cluster so that the items that are closest together are grouped together. This process stops when the optimal quality score (modularity) for each cluster is reached. * Connections are weighted by rank of a keyword to a page using the formula: 1/rank, e.g. 1/7.
7. In the process of being reworded Our Hypothesis….. Google’s algorithm effectively builds its own index of the web based on the structure of the web and the relationship between entities across the web, as well as based on consumer’s searching and click-through behaviour. By undertaking graph analysis of the SERPs using the network of ranking URLs for millions of keywords discovered, we are effectively observing the structure created by Google’s ranking algorithms, by highlighting Google’s view of the market (e.g. How it organises the web into related centres of authority) that we interpret and display as clusters – e.g. Communities or hubs of related keywords that correspond to related and relevant content and sites. How a cluster is formed ….. Each cluster should be dissimilar to other clusters in the graph (measured by distance from each other in the graph, weighted by ranking) Each keyword and page in a cluster is put into a cluster based on distance from other items in the cluster so that the items that are closest together are grouped together. This process stops when the optimal quality score (modularity) for each cluster is reached. * Connections are weighted by rank of a keyword to a page using the formula: 1/rank, e.g. 1/7.
8. Catch before any filtering taken place Up and coming competitors Promotions / renegation element Virgin Active – poor data
9. This slide shows the top 100 domains in the market plotted with their average relative strength (x axis) versus the sum of their share of market visibility (y axis – log scale base 10).
10. Please note this document contains details of future product plans and is therefore confidential – please do not disclose, circulate or publish without Analytics SEO’s written permission. Please remind your team of this obligation.
11. Clustered by host (Great British Chefs) market common keyword topics Relative strength: host domain versus competing URLs’ domain strength (averaged for the cluster) Growth Potential from competing, higher ranking URLs’ common keywords calculated traffic (CTRxSV) (summed for the cluster) Bubbles are sized by the number of keywords per cluster
12. Clustering of host (Great British Chefs) URLs, by topic Relative strength: host domain versus competing URLs’ domain strength (averaged for the cluster) Growth Potential from competing URLs’ opportunity keywords calculated traffic (CTRxSV) (averaged for the cluster) Bubbles are sized by the number of keywords for that cluster of URLs
13. Relevant opportunity keywords clustered by connected topic for this host (Great British Chefs)’s market Relative strength: host domain versus competing URLs (also within that cluster)’s domain strength Growth Potential from average competing URLs’ opportunity keywords calculated traffic (CTRxSV) – log scaled for clearer visibility Bubbles are sized by the number of keywords for the cluster in question
14. Values in red to be created, and updated