Floods of Twitter Data - StampedeCon 2016
•
2 likes•442 views
The Twitter data firehose delivers hundreds of millions of Tweets every day. This data flood comes with many ‘big data’ challenges in terms of both data volumes and velocities. This presentation will focus on tools that help you find your data ‘signal’ of interest, and will include several demos that focus on using Twitter for flood early-warning systems. These demos will highlight the real-time, public broadcast nature of Twitter, examples of real-time firehose filtering, as well as recent Internet of Things (IoT) Twitter integrations.
Report
Share
Floods of Twitter Data - StampedeCon 2016
- 1. Floods of Twitter Data Jim Moffitt Developer Advocate @Twitter @snowman StampedeCon 2016
- 2. Gnip develops the enterprise-grade data APIs and services to help unlock the power of Twitter data
- 3. agenda (6 characters) • Twitter 101 - Public, mobile, real- time and big data. • Finding your signal of interest. • Twitter, early-warning systems and IoT.
- 4. Twitter is public, mobile, and real-time (40 characters)
- 5. Firehose volume: • Hundreds of millions of Tweets every day. • Billions of Tweets per month. Twitter is big data
- 6. Twitter is big data Firehose velocity: • Many thousands of Tweets per second. • 20-25K per second now common during events. • Current record: 140K tweets per second.
- 8. Twitter is really fast #StampedeCon OR #realtime OR @snowman OR (point_radius:[-90.202222 38.624419 1mi] (@StampedeCon OR #demo OR "big data" OR #BigData))
- 13. Publisher Data Platform Sample Analytics Providers Brands Acting on Insights from Twitter Data Providing Analysis from Twitter Data Filtering, Enriching & Distributing Content Creating Social Media Content TWITTER DATA ECOSYSTEM
- 14. SYSTEM REDUNDANCY CONFIGURATION APPLICATION FILTERING AT SCALE DATA FIDELITY Enterprise Data Services
- 16. FIREHOSE FILTERING STACK 1 CUSTOMER STREAMING STACK 2 STACK 3 CUSTOMER 1 CUSTOMER 2 CUSTOMER 3 CUSTOMER 4 CUSTOMER 5 CUSTOMER 6 CUSTOMER 7 CUSTOMER 8
- 17. • Agnostic APIs. • Most used languages: Java, Python, Ruby, C#, Node.js • Understanding Twitter metadata and filtering syntax. • Stream consumer is lightweight and writes Tweets to a queue. • JSON parser is flexible and tolerant. • Bending and not breaking. • Storage. Client-side details
- 18. A Twitter Data Story 2013 Colorado Flood
- 19. Public Broadcasts Tweeting Alerts and Notifications
- 20. BoulderCreek StreamLevel(Feet) 2.25 4.5 6.75 9 Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17 HourlyRainfall (Inches) 0.3 0.6 0.9 1.2 Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17 TweetsPerHour 1500 3000 4500 6000 Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17 Major Flooding Moderate Flooding Stream Bank Full Event Tweets Baseline Tweets
- 21. Audiences tune in during events…
- 22. TotalRain(Inches) 4 8 12 16 Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17
- 23. NewFollowers 1500 3000 4500 6000 Sep-10 Sep-11 Sep-12 Sep-13 Sep-14 Sep-15 Sep-16 Sep-17
- 26. IoT Examples