Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
•
43 likes•6,827 views
Shirshanka Das and Yael Garten describe how LinkedIn redesigned its data analytics ecosystem in the face of a significant product rewrite, covering the infrastructure changes that enable LinkedIn to roll out future product innovations with minimal downstream impact. Shirshanka and Yael explore the motivations and the building blocks for this reimagined data analytics ecosystem, the technical details of LinkedIn’s new client-side tracking infrastructure, its unified reporting platform, and its data virtualization layer on top of Hadoop and share lessons learned from data producers and consumers that are participating in this governance model. Along the way, they offer some anecdotal evidence during the rollout that validated some of their decisions and are also shaping the future roadmap of these efforts.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (7)
Similar to Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Similar to Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem (20)
Recently uploaded
Recently uploaded (20)
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
- 1. Architecting for change: LinkedIn's new data ecosystem Sept 28, 2016 Shirshanka Das, Principal Staff Engineer, LinkedIn Yael Garten, Director of Data Science, LinkedIn @shirshanka, @yaelgarten
- 2. Design for change. Expect it. Embrace it.
- 3. Product Change Technology Culture & Process Learnings
- 4. The Product Change: Launch a completely rewritten LinkedIn mobile app
- 5. What does this impact? Data driven product
- 6. Tracking data records user activity InvitationClickEvent()
- 7. Tracking data records user activity InvitationClickEvent() Scale fact: ~ 1000 tracking event types, ~ Double-digit TB per day, hundreds of metrics & data products
- 8. user engagement tracking data metric scripts production code Tracking Data Lifecycle TransportProduce Consume Member facing data products Business facing decision making
- 9. Tracking Data Lifecycle & Teams TransportProduce Consume Product or App teams: PMs, Developers, TestEng Infra teams: Hadoop, Kafka, DWH, ... Data teams: Analytics, Relevance Engineers,... user engagement tracking data metric scripts production code Member facing data products Business facing decision making
- 10. How do we calculate a metric: ProfileViews PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } Metric: ProfileViews = sum(PageViewEvent) where pageKey = profile_page PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "new_profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } or new_profile_page
- 11. PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } CASE WHEN trackingInfo["profileIds"] ... WHEN trackingInfo["profileid"] ... WHEN trackingInfo["profileId"] ... WHEN trackingInfo["url$profileIds"] ... WHEN trackingInfo["11"] LIKE '%profileIds=%' THEN SUBSTRING(trackingInfo["11"],9,60) WHEN trackingInfo["12"] LIKE '%priceIds=%' THEN SUBSTRING(trackingInfo["12"],9,60) ELSE NULL END AS profile_id Evolution as we mature and grow... Metric: ProfileViews = sum(PageViewEvent where pagekey = profile_page and memberID != trackinginfo[vieweeID] )
- 13. PageViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 1454745292951, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "trackingInfo" : { ["vieweeID" : "23456"], ... } } We wanted to move to better data models LI_ProfileViewEvent Record 1: { "header" : { "memberId" : 12345, "time" : 4745292951145, "appName" : { "string" : "LinkedIn" "pageKey" : "profile_page" }, }, "entityView" : { "viewType" : "profile-view", "viewerId" : “12345”, "vieweeId" : “23456”, }, }
- 14. Two options: 1. Keep the old tracking: a. Cost: producers (try to) replicate it (write bad old code from scratch), b. Save: consumers avoid migrating. 2. Evolve. a. Cost: time on data modeling, and on consumer migration, b. Save: pays down data modeling tech debt How much work would it be?
- 15. How much work would it be? Two options: 1. Keep the old tracking: a. Cost: producers (try to) replicate it (write bad old code from scratch), b. Save: consumers avoid migrating. 2. Evolve. a. Cost: time on data modeling, and on consumer migration, b. Save: pays down data modeling tech debt 2000 daysEstimated cost to update consumers to new tracking with clean, committee-approved data models Estimated cost for producers to attempt to replicate old tracking 5000 days #AnalyticsHappiness
- 16. The Task and Opportunity Must do: So we will do the data modeling, and rewrite all the metrics to account for the changes happening upstream… but… Extra credit points: How do we make sure that the cost is not this high the next time? How do we handle evolution in a principled way?
- 17. Product Change Technology Culture & Process Learnings
- 18. Metrics ecosystem at LinkedIn: 3 yrs ago Operational Challenges Diminished Trust due to multiple sources of truth
- 19. Data Stages Ingest Process Serve VisualizeCreate
- 20. Ingest Process Serve VisualizeCreate Tracking Kafka Espresso …
- 21. Tracking Architecture SDKs in different frameworks (server, client) Tracking front-end Monitoring Tools Components KafkaClient-side Tracking Tracking Frontend Services Tools Create
- 22. Data Stages Ingest Process Serve VisualizeCreate
- 23. Unified Ingestion with Hundreds of TB / day Thousands of datasets 80+% of data ingest Ingest
- 24. In production @ LinkedIn, Intel, Swisscom, NerdWallet, PayPal Ingest
- 25. Ingest Process Serve VisualizeCreate Hadoop
- 26. Processing engines @ LinkedInProcess
- 27. Ingest Process Serve VisualizeCreate Pinot
- 28. Pinot Kafka Hadoop Samza Jobs Pinot minutes hour + Distributed Multi-dimensional OLAP Columnar + indexes No joins Latency: low ms to sub-second Serve
- 29. Site-facing Apps Reporting dashboards Monitoring In production @ LinkedIn, Uber Serve
- 30. Ingest Process VisualizeCreate Hadoop Pinot Raptor Serve
- 31. Ingest Process VisualizeCreate Unified Metrics Platform (UMP) Hadoop Pinot Raptor Serve
- 32. Unified Metrics Platform Metrics Logic Raw Data Pinot UMP Harness Incremental Aggregate Backfill Auto-join Raptor dashboards HDFS Aggregated Data Experiment Analysis Relevance ... HDFS Ad-hoc
- 33. Ingest Process Serve VisualizeCreate RaptorKafka Espresso … Hadoop Pinot Tracking Unified Metrics Platform (UMP)
- 35. How do we handle old and new? PageViewEvent ProfileViewEvent Producers Consumers old new Relevance Analytics
- 36. The Big Challenge load “/data/tracking/PageViewEvent” using AvroStorage() (Pig scripts) My Raw Data Our scripts were doing ….
- 37. My Raw Data My Data API We need “microservices" for Data
- 38. The Database community solved this decades ago... Views!
- 39. We had been working on something that could help... A Data Access Layer for Linkedin Abstract away underlying physical details to allow users to focus solely on the logical concerns Logical Tables + Views Logical FileSystem
- 41. Views ecosystem 41 Producers Consumers LinkedInProfileView JSAProfileView Job Seeker App (JSA) LinkedIn App UnifiedProfileView
- 42. Data Catalog + Discovery (DALI) DaliFileSystem Client Data Source (HDFS) Data Sink (HDFS) Processing Engine (MapReduce, Spark) DALI Datasets (Tables + Views) Query Layers (Hive, Pig, Spark) View Defs + UDFs (Artifactory, Git) Dataflow APIs (MR, Spark, Scalding) DALI CLI Dali: Implementation Details in Context
- 43. From load ‘/data/tracking/PageViewEvent’ using AvroStorage(); To load ‘tracking.UnifiedProfileView’ using DaliStorage(); One small step for a script
- 44. A Few Hard Problems Versioning Views and UDFs Mapping to Hive metastore entities Development lifecycle Git as source of truth Gradle for build LinkedIn tooling integration for deployment
- 45. Early experiences with Dali views How we executed Lots of work to get the infra ready Closed beta model Tons of training and education (hand holding) for all Governance body Feedback from analysts is overwhelmingly positive: + Much simpler to share and standardize data cleansing code with peers + Provides effective insulation to scripts from upstream changes - Harder to debug where problems are due to additional layer
- 46. State of the world today ~100 producer views ~200 consumer views ~30% of UMP metrics use Dali data sources ~80 unique tracking event data sources ProfileViews MessagesSent Searches InvitationsSent ArticlesRead JobApplications ...
- 47. What’s next for Dali? Real-time Views on streaming data Selective materialization Hive is an implementation detail, not a long term bet Open source Data Quality Framework
- 48. Product Change Technology Culture & Process Learnings
- 49. Infrastructure enables, but culture really preserves get_tracking_codes = foreach get_domain_rolled_up generate ..entry_domain_rollup, ( (tracking_code matches 'eml-ced.*' or tracking_code matches 'eml-b2_content _ecosystem_digest.*' or (referer is not null and (referer matches '.*touch.linkedin.com.*trk=eml-ced.*' or referer matches '.*touch.linkedin.com.*trk=eml-b2_content_ecosystem_digest.*')) ? 'Email - CED' : (tracking_code matches 'eml-.*' or (referer is not null and referer matches '.*touch.linkedin .com.*trk=eml-.*') or entry_domain_rollup == 'Email' ? 'Email - Other' : (tracking_code == 'hp-feed-article-title-hpm' and entry_domain_rollup == 'Linkedin' ? 'Homepage Pulse Module' : ((tracking_code matches 'hp-feed-.*' and entry_domain_rollup == 'Linkedin') or (std_user_interface matches '(phone app|tablet app|phone browser|tablet browser)' and tracking_code == 'v-feed') or (tracking_code == 'Organic Traffic' and entry_domain_rollup == 'Linkedin' and (referer == 'https://www.linkedin.com/nhome' or referer == 'http://www.linkedin.com/nhome')) ? 'Feed' : (tracking_code matches 'hb_ntf_MEGAPHONE_.*' and entry_domain_rollup == 'Linkedin' ? 'Desktop Notifications' : (tracking_code == 'm_sim2_native_reader_swipe_right' ? 'Push Notification' : (tracking_code == 'pulse_dexter_stream_scroll' and entry_domain_rollup == 'Linkedin' ? 'Pulse -
- 50. For a great data ecosystem that can handle change: 1. Standardize core data entities 2. Create clear maintainable contracts between data producers & consumers 3. Ensure dialogue between data producers & consumers
- 51. 1. Standardize core data entities • Event types and names: Page, Action, Impression • Framework level client side tracking: views, clicks, flows • For all else (custom) - guide when to create a new Event or Dali view Navigation Page View Control Interaction
- 52. 2. Create clear maintainable contracts 1 1. Tracking specification with monitoring: clear, visual, consistent contract Need tooling to support culture shift Tracking specification Tool 2 2. Dali dataset specification with data quality rules
- 53. 3. Ensure dialogue between Producers & Consumers • Awareness: Train about end-to-end data pipeline, data modeling • Instill communication & collaborative ownership process between all: a step-by-step playbook for who & how to develop and own tracking PM → Analyst → Engineer → All3 → TestEng → Analyst user engagement tracking data metric scripts production code Member facing data products Business facing decision making
- 54. Product Change Technology Culture & Process Learnings
- 55. Our Learnings Culture and Process ● Spend time to identify what needs culture & process, and what needs tools & tech ● Big changes can mean big opportunities ● Very hard to massively change things like data culture or data tech debt; never a good time to invest in “invisible” behind-the-scenes change → Make it non-invisible -- try to clarify or size out the cost of NOT doing it → needed strong leaders, and a village Tech ● Must build tooling to support that culture change otherwise culture will revert ● Work hard to make any new layer as frictionless as possible ● Virtual views on Hadoop data can work at scale! (Dali views)
- 56. For a great data ecosystem that can handle change: 1. Standardize core data entities 2. Create clear maintainable contracts between data producers & consumers 3. Ensure dialogue between data producers & consumers Design for change. Expect it. Embrace it.
- 57. Did we succeed? We just handled another huge change! #AnalyticsHappiness