Big Data Analytics Powerpoint Presentation SlideSlideTeam
If it’s that time to make analysis for the predicament of the management system or simply to present deafening data in front of your qualified team then you have reached the right match. SlideTeam presents you classy and eternally approaching PowerPoint slides for big data analytics. Data analysis agendas and big data plans are shown through captivating icons and subheadings for a precise and interesting approach. This unique PPT slide is useful for studying business and marketing related topics, approaching the correct conclusions and keeping a track on business growth. Make an outstanding presentation for your viewers with this unique PPT slide and deliver your message in an effective manner using Big data analytics Powerpoint Presentation slide and make your pathways more defining. Most of the elements of the slide are highly customizable. The text boxes help you in adding more information about the point mentioned and its associated icon. Every detail in our Big Data Analytics Powerpoint Presentation Slide is doubly cross checked. You can be certain of it's authenticity. https://bit.ly/3fvnRVK
This document provides an overview of big data, including its definition, characteristics, sources, tools used, applications, benefits, and impact on IT. Big data is a term used to describe the large volumes of data, both structured and unstructured, that are so large they are difficult to process using traditional database and software techniques. It is characterized by high volume, velocity, variety, and veracity. Common sources of big data include mobile devices, sensors, social media, and software/application logs. Tools like Hadoop, MongoDB, and MapReduce are used to store, process, and analyze big data. Key applications areas include homeland security, healthcare, manufacturing, and financial trading. Benefits include better decision making, cost reductions
This document contains information about a group project on big data. It lists the group members and their student IDs. It then provides a table of contents and summaries various topics related to big data, including what big data is, data sources, characteristics of big data like volume, variety and velocity, storing and processing big data using Hadoop, where big data is used, risks and benefits of big data, and the future of big data.
The document discusses big data analytics. It begins by defining big data as large datasets that are difficult to capture, store, manage and analyze using traditional database management tools. It notes that big data is characterized by the three V's - volume, variety and velocity. The document then covers topics such as unstructured data, trends in data storage, and examples of big data in industries like digital marketing, finance and healthcare.
This document provides a syllabus for a course on big data. The course introduces students to big data concepts like characteristics of data, structured and unstructured data sources, and big data platforms and tools. Students will learn data analysis using R software, big data technologies like Hadoop and MapReduce, mining techniques for frequent patterns and clustering, and analytical frameworks and visualization tools. The goal is for students to be able to identify domains suitable for big data analytics, perform data analysis in R, use Hadoop and MapReduce, apply big data to problems, and suggest ways to use big data to increase business outcomes.
A Seminar Presentation on Big Data for Students.
Big data refers to a process that is used when traditional data mining and handling techniques cannot uncover the insights and meaning of the underlying data. Data that is unstructured or time sensitive or simply very large cannot be processed by relational database engines. This type of data requires a different processing approach called big data, which uses massive parallelism on readily-available hardware.
This document defines big data and discusses techniques for integrating large and complex datasets. It describes big data as collections that are too large for traditional database tools to handle. It outlines the "3Vs" of big data: volume, velocity, and variety. It also discusses challenges like heterogeneous structures, dynamic and continuous changes to data sources. The document summarizes techniques for big data integration including schema mapping, record linkage, data fusion, MapReduce, and adaptive blocking that help address these challenges at scale.
This document provides an overview of big data, including its definition, characteristics, sources, tools, applications, risks and benefits. It defines big data as large volumes of diverse data that can be analyzed to reveal patterns and trends. The three key characteristics are volume, velocity and variety. Examples of big data sources include social media, sensors and user data. Tools used for big data include Hadoop, MongoDB and analytics programs. Big data has many applications and benefits but also risks regarding privacy and regulation. The future of big data is strong with the market expected to grow significantly in coming years.
Big data Analytics is a process to extract meaningful insight from big such as hidden patterns, unknown correlations, market trends and customer preferences
Big data is high-volume, high-velocity, and high-variety data that is difficult to process using traditional data management tools. It is characterized by 3Vs: volume of data is growing exponentially, velocity as data streams in real-time, and variety as data comes from many different sources and formats. The document discusses big data analytics techniques to gain insights from large and complex datasets and provides examples of big data sources and applications.
7 Big Data Challenges and How to Overcome ThemQubole
Implementing a big data project is difficult. Hadoop is complex, and data governance is crucial. Learn common big data challenges and how to overcome them.
It is an introduction to Data Analytics, its applications in different domains, the stages of Analytics project and the different phases of Data Analytics life cycle.
I deeply acknowledge the sources from which I could consolidate the material.
Big data is data that is too large or complex for traditional data processing applications to analyze in a timely manner. It is characterized by high volume, velocity, and variety. Big data comes from a variety of sources, including business transactions, social media, sensors, and call center notes. It can be structured, unstructured, or semi-structured. Tools used for big data include NoSQL databases, MapReduce, HDFS, and analytics platforms. Big data analytics extracts useful insights from large, diverse data sets. It has applications in various domains like healthcare, retail, and transportation.
The document is a chapter from a textbook on data mining written by Akannsha A. Totewar, a professor at YCCE in Nagpur, India. It provides an introduction to data mining, including definitions of data mining, the motivation and evolution of the field, common data mining tasks, and major issues in data mining such as methodology, performance, and privacy.
The document discusses big data, providing definitions and facts about the volume of data being created. It describes the characteristics of big data using the 5 V's model (volume, velocity, variety, veracity, value). Different types of data are mentioned, from unstructured to structured. Hadoop is introduced as an open source software framework for distributed processing and analyzing large datasets using MapReduce and HDFS. Hardware and software requirements for working with big data and Hadoop are listed.
The document provides an introduction to data analytics, including defining key terms like data, information, and analytics. It outlines the learning outcomes which are the basic definition of data analytics concepts, different variable types, types of analytics, and the analytics life cycle. The analytics life cycle is described in detail and involves problem identification, hypothesis formulation, data collection, data exploration, model building, and model validation/evaluation. Different variable types like numerical, categorical, and ordinal variables are also defined.
This presentation briefly explains the following topics:
Why is Data Analytics important?
What is Data Analytics?
Top Data Analytics Tools
How to Become a Data Analyst?
This document provides an overview of big data. It defines big data as large volumes of diverse data that are growing rapidly and require new techniques to capture, store, distribute, manage, and analyze. The key characteristics of big data are volume, velocity, and variety. Common sources of big data include sensors, mobile devices, social media, and business transactions. Tools like Hadoop and MapReduce are used to store and process big data across distributed systems. Applications of big data include smarter healthcare, traffic control, and personalized marketing. The future of big data is promising with the market expected to grow substantially in the coming years.
Big Data Analytics and its Application in E-CommerceUyoyo Edosio
Abstract-This era unlike any, is faced with explosive
growth in the size of data generated/captured. Data
growth has undergone a renaissance, influenced
primarily by ever cheaper computing power and
the ubiquity of the internet. This has led to a
paradigm shift in the E-commerce sector; as data is
no longer seen as the byproduct of their business
activities, but as their biggest asset providing: key
insights to the needs of their customers, predicting
trends in customer’s behavior, democratizing of
advertisement to suits consumers varied taste, as
well as providing a performance metric to assess the
effectiveness in meeting customers’ needs.
This paper presents an overview of the unique
features that differentiate big data from traditional
datasets. In addition, the application of big data
analytics in the E-commerce and the various
technologies that make analytics of consumer data
possible is discussed.
Further this paper will present some case studies of
how leading Ecommerce vendors like Amazon.com,
Walmart Inc, and Adidas apply Big Data analytics in
their business strategies/activities to improve their
competitive advantage. Lastly we identify some
challenges these E-commerce vendors face while
implementing big data analytic
This document provides an overview of big data analytics. It discusses challenges of big data like increased storage needs and handling varied data formats. The document introduces Hadoop and Spark as approaches for processing large, unstructured data at scale. Descriptive and predictive analytics are defined, and a sample use case of sentiment analysis on Twitter data is presented, demonstrating data collection, modeling, and scoring workflows. Finally, the author's skills in areas like Java, Python, SQL, Hadoop, and predictive analytics tools are outlined.
The document discusses cloud computing, big data, and big data analytics. It defines cloud computing as an internet-based technology that provides on-demand access to computing resources and data storage. Big data is described as large and complex datasets that are difficult to process using traditional databases due to their size, variety, and speed of growth. Hadoop is presented as an open-source framework for distributed storage and processing of big data using MapReduce. The document outlines the importance of analyzing big data using descriptive, diagnostic, predictive, and prescriptive analytics to gain insights.
Just finished a basic course on data science (highly recommend it if you wish to explore what data science is all about). Here are my takeaways from the course.
This video will give you an idea about Data science for beginners.
Also explain Data Science Process , Data Science Job Roles , Stages in Data Science Project
This document discusses cloud computing, big data, Hadoop, and data analytics. It begins with an introduction to cloud computing, explaining its benefits like scalability, reliability, and low costs. It then covers big data concepts like the 3 Vs (volume, variety, velocity), Hadoop for processing large datasets, and MapReduce as a programming model. The document also discusses data analytics, describing different types like descriptive, diagnostic, predictive, and prescriptive analytics. It emphasizes that insights from analyzing big data are more valuable than raw data. Finally, it concludes that cloud computing can enhance business efficiency by enabling flexible access to computing resources for tasks like big data analytics.
Data science Nagarajan and madhav.pptxNagarajanG35
This document summarizes a presentation on data science. It includes details about the presenters, date, time and login details for a seminar on data science. It then provides definitions and explanations of key concepts in data science including machine learning, deep learning, statistics and visualization. It describes common data science jobs and roles and lists popular tools used in data science. Finally, it discusses applications of data science and some challenges in the field.
Developed by Google’s Artificial Intelligence division, the Sycamore quantum processor boasts 53 qubits1.
In 2019, it achieved a feat that would take a state-of-the-art supercomputer 10,000 years to accomplish: completing a specific task in just 200 seconds1
This document discusses big data workflows. It begins by defining big data and workflows, noting that workflows are task-oriented processes for decision making. Big data workflows require many servers to run one application, unlike traditional IT workflows which run on one server. The document then covers the 5Vs and 1C characteristics of big data: volume, velocity, variety, variability, veracity, and complexity. It lists software tools for big data platforms, business analytics, databases, data mining, and programming. Challenges of big data are also discussed: dealing with size and variety of data, scalability, analysis, and management issues. Major application areas are listed in private sector domains like retail, banking, manufacturing, and government.
Purpose of this presentation is to highlight how end to end machine learning looks like in real world enterprise. This is to provide insight to aspiring data scientist who have been through courses or education in ML that mostly focus on ML algorithms and not end to end pipeline.
Architecture and components mentioned in Slide 11 will be discussed in detailed in series of post on LinkedIn over the course of next few month
To get updates on this follow me on LinkedIn or search/follow hashtag #end2endDS. Post will be active in August 2019 and will be posted till September 2019
A New Paradigm on Analytic-Driven Information and Automation V2.pdfArmyTrilidiaDevegaSK
The document proposes an end-to-end methodology for developing analytic-driven information and automation systems based on big data, data science, and artificial intelligence. The methodology involves 6 steps: 1) collecting data from multiple sources, 2) preprocessing the data, 3) extracting features from the data, 4) clustering and interpreting the data, 5) designing applications, and 6) implementing and evaluating the systems. It then provides an example of applying this methodology to develop an early warning system for monitoring higher education institutions in Indonesia. The system would collect data from various sources, analyze it using machine learning techniques, predict and prescribe interventions for student groups.
1. Introduction and how to get into Data
2. Data Engineering and skills needed
3. Comparison of Data Analytics for statistic and real time streaming data
4. Bayesian Reasoning for Data
Big Data Analytics: From SQL to Machine Learning and Graph AnalysisYuanyuan Tian
This document discusses big data analytics and different types of analytics that can be performed on big data, including SQL, machine learning, and graph analytics. It provides an overview of various big data analytics systems and techniques for different data types and complexity levels. Integrated analytics that combine multiple types of analytics are also discussed. The key challenges of big data analytics and how different systems address them are covered.
Introduction to Data Analysis Course Notes.pdfGraceOkeke3
"Embark on a journey into data analysis with our Introduction to Data Analysis slides. Uncover the fundamentals and prerequisites for effective analysis, explore types of data, and discover essential tools and methodologies. Equip yourself with the skills to unlock valuable insights.
A presentation covers how data science is connected to build effective machine learning solutions. How to build end to end solutions in Azure ML. How to build, model, and evaluate algorithms in Azure ML.
This document provides an overview of the key concepts in the syllabus for a course on data science and big data. It covers 5 units: 1) an introduction to data science and big data, 2) descriptive analytics using statistics, 3) predictive modeling and machine learning, 4) data analytical frameworks, and 5) data science using Python. Key topics include data types, analytics classifications, statistical analysis techniques, predictive models, Hadoop, NoSQL databases, and Python packages for data science. The goal is to equip students with the skills to work with large and diverse datasets using various data science tools and techniques.
Advanced Analytics and Machine Learning with Data VirtualizationDenodo
Watch: https://bit.ly/2DYsUhD
Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way.
Attend this webinar and learn:
- How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice
- How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo
- How you can use the Denodo Platform with large data volumes in an efficient way
- How Prologis accelerated their use of Machine Learning with data virtualization
The document discusses big data and big data analytics in banking. It defines big data as large, complex datasets that are difficult to process and store using traditional databases. Sources of big data include social media, sensors, transportation services, online shopping, and mobile apps. Characteristics of big data include volume, velocity, and variety. Hadoop is presented as an open source framework for analyzing big data using HDFS for storage and MapReduce for processing. The benefits of big data analytics in banking include fraud detection, risk management, customer segmentation, churn analysis, and sentiment analysis to improve customer experience.
Data mining involves extracting useful patterns from large amounts of data. It uses software tools to analyze raw data and discover hidden patterns that can help businesses gain insights into customer behavior and develop effective marketing strategies. The key processes involved in data mining are data cleaning, integration, transformation, mining, pattern evaluation, and presentation. Data mining is commonly used for applications like classification, association, prediction, clustering, and regression. It has proven useful in domains such as healthcare for improving treatment and education for enhancing learning outcomes. Popular tools for data mining include Oracle Data Mining, IBM SPSS Modeler, RapidMiner, Apache Mahout, Weka, and KNIME.
The document discusses big data analysis and provides an introduction to key concepts. It is divided into three parts: Part 1 introduces big data and Hadoop, the open-source software framework for storing and processing large datasets. Part 2 provides a very quick introduction to understanding data and analyzing data, intended for those new to the topic. Part 3 discusses concepts and references to use cases for big data analysis in the airline industry, intended for more advanced readers. The document aims to familiarize business and management users with big data analysis terms and thinking processes for formulating analytical questions to address business problems.
Cómo hemos implementado semántica de "Exactly Once" en nuestra base de datos ...javier ramirez
Los sistemas distribuidos son difíciles. Los sistemas distribuidos de alto rendimiento, más. Latencias de red, mensajes sin confirmación de recibo, reinicios de servidores, fallos de hardware, bugs en el software, releases problemáticas, timeouts... hay un montón de motivos por los que es muy difícil saber si un mensaje que has enviado se ha recibido y procesado correctamente en destino. Así que para asegurar mandas el mensaje otra vez.. y otra... y cruzas los dedos para que el sistema del otro lado tenga tolerancia a los duplicados.
QuestDB es una base de datos open source diseñada para alto rendimiento. Nos queríamos asegurar de poder ofrecer garantías de "exactly once", deduplicando mensajes en tiempo de ingestión. En esta charla, te cuento cómo diseñamos e implementamos la palabra clave DEDUP en QuestDB, permitiendo deduplicar y además permitiendo Upserts en datos en tiempo real, añadiendo solo un 8% de tiempo de proceso, incluso en flujos con millones de inserciones por segundo.
Además, explicaré nuestra arquitectura de log de escrituras (WAL) paralelo y multithread. Por supuesto, todo esto te lo cuento con demos, para que veas cómo funciona en la práctica.
How We Added Replication to QuestDB - JonTheBeachjavier ramirez
Building a database that can beat industry benchmarks is hard work, and we had to use every trick in the book to keep as close to the hardware as possible. In doing so, we initially decided QuestDB would scale only vertically, on a single instance.
A few years later, data replication —for horizontally scaling reads and for high availability— became one of the most demanded features, especially for enterprise and cloud environments. So, we rolled up our sleeves and made it happen.
Today, QuestDB supports an unbounded number of geographically distributed read-replicas without slowing down reads on the primary node, which can ingest data at over 4 million rows per second.
In this talk, I will tell you about the technical decisions we made, and their trade offs. You'll learn how we had to revamp the whole ingestion layer, and how we actually made the primary faster than before when we added multi-threaded Write Ahead Logs to deal with data replication. I'll also discuss how we are leveraging object storage as a central part of the process. And of course, I'll show you a live demo of high-performance multi-region replication in action.
LLM powered contract compliance application which uses Advanced RAG method Self-RAG and Knowledge Graph together for the first time.
It provides highest accuracy for contract compliance recorded so far for Oil and Gas Industry.
Amazon Aurora 클러스터를 초당 수백만 건의 쓰기 트랜잭션으로 확장하고 페타바이트 규모의 데이터를 관리할 수 있으며, 사용자 지정 애플리케이션 로직을 생성하거나 여러 데이터베이스를 관리할 필요 없이 Aurora에서 관계형 데이터베이스 워크로드를 단일 Aurora 라이터 인스턴스의 한도 이상으로 확장할 수 있는 Amazon Aurora Limitless Database를 소개합니다.
2. ABOUT ME
Currently work in Telkomsel as senior data analyst
8 years professional experience with 4 years in big data
and predictive analytics field in telecommunication
industry
Bachelor from Computer Science, Gadjah Mada
University & get master degree from Magister of
Information Technology, Universitas Indonesia
Lecturer in Muhammadiyah Jakarta University
https://id.linkedin.com/pub/ghulam-imaduddin/32/a21/507
ghulam@ideweb.co.id
3. WHAT’S IN THIS SLIDE
[BIG] DATA ANALYTICS
Intro & Data Trends
Challenges
Tech Approach
Big Data Tools
Type of Analytics
Tools
Analytics Lifecycle
Use Cases (Sentiment Analysis)
What’s Trending
Where to Start
Methodology
4. THE WORLD OF DATA
Source: http://www.cision.com/us/2012/10/big-data-and-big-analytics/
5. DATA VS BIG DATA
Big data is just data with:
More volume
Faster data generation (velocity)
Multiple data format (variety)
World's data volume to grow 40% per year
& 50 times by 2020 [1]
Data coming from various human & machine
activity
[1] http://e27.co/worlds-data-volume-to-grow-40-per-year-50-times-by-2020-aureus-20150115-2/
6. CHALLENGES
More data = more storage space
More storage = more money to spend (RDBMS server needs very costly
storage)
Data coming faster
Speed up data processing or we’ll have backlog
Needs to handle various data structure
How do we put JSON data format in standard RDBMS?
Hey, we also have XML format from other sources
Other system give us compressed data in gzip format
Agile business requirement.
On initial discussion, they only need 10 information, now they ask for 25? Can
we do that? We only put that 10 in our database
Our standard ETL process can’t handle this
7. STORAGE COST
In Terms of storage cost, Hadoop has lower comparing to standard
RDBMS.
Hadoop provides highly scalable storage and process with fraction of
the EDW Cost
8. STORAGE & COMPUTE
TOGETHER
The Hadoop WayThe Old Way
• Hard to scale
• Network is a bottleneck
• Only handles relational data
• Difficult to add new fields & data types
Expensive, Special purpose, “Reliable” Servers
Expensive Licensed Software
Network
Data Storage
(SAN, NAS)
Compute
(RDBMS, EDW)
• Scales out forever
• No bottlenecks
• Easy to ingest any data
• Agile data access
Commodity “Unreliable” Servers
Hybrid Open Source Software
Compute
(CPU)
Memory Storage
(Disk)
z
z
Source: Cloudera Presentation Deck by Amr Awadallah
9. MAP REDUCE APPROACH
Process data in parallel way using distributed algorithm on a cluster
Map procedure performs filtering and sorting data locally
Reduce procedure performs a summary operation (count, sum,
average, etc.)
10. HADOOP vs UNSTRUCTURED
DATA
Hadoop has HDFS (Hadoop Distributed File System)
It is just file system, so what you need is just drop the file there
Schema on read concept
Source Data
Database Table
Load the data
Metadata
Applying schema
User
Application (BI Tools)
RDBMS
APPROACH
HADOOP
APPROACH
11. HIVE
The Apache Hive ™ data warehouse software facilitates querying and
managing large datasets residing in distributed storage.
With Hive you can write the schema for the data in HDFS
Hive provide many library that enable you to read various data type
like XML, JSON, or even compressed format
You can create your own data parser with Java language
Hive support SQL language to read from your data
Hive will convert your SQL into Java MapReduce code, and run it in
cluster
12. Apache spark is fast and general engine for large-scale data processing
Run programs up to 100x faster than Hadoop MapReduce in memory,
or 10x faster on disk
You can write spark application in Java, Scala, Python, or R
Spark support library to run SQL, streaming, and complex analysis like
graph computation and machine learning
https://spark.apache.org/
14. ANALYTICS IS IN YOUR BLOOD
Do you realize that you do analytics everyday?
I need to go to campus faster!
Hmm.. Looking at the sky today, I think it’ll be rain
Based on my mid term and assignment score, I need to get at least 80
in my final exam to pass this course
I stalked her social media. I think she is single because most of her
post only about food :p
15. DESCRIPTIVE & PREDICTIVE
Descriptive statistics is the term given to the analysis of data that helps
describe, show or summarize data in a meaningful way such that, for
example, patterns might emerge from the data.
In Information System Design course, most of the student get C grade (11
people). There is 4 people get A, 7 get B, 7 get D, and 7 get E
Fulan only post his activity on Facebook at weekend
Predictive analytics is the branch of data mining concerned with the
prediction of future probabilities and trends.
The central element of predictive analytics is the predictor, a variable
that can be measured for an individual or other entity to predict future
behavior.
Fulan should be has a job. Because he always left home at 7 in the morning
and get back at 6 afternoon
16. PREDICTIVE ANALYTICS
There is 2 types of predictive analytics:
◦ Supervised
Supervised analytics is when we know the truth about something in the past
Example:
we have historical weather data. The temperature, humidity, cloud density and
weather type (rain, cloudy, or sunny). Then we can predict today weather
based on temp, humidity, and cloud density today
Machine learning to be used: Regression, decision tree, SVM, ANN, etc.
◦ Unsupervised
Unsupervised is when we don’t know the truth about something in the past.
The result is segment that we need to interpret
Example:
We want to do segmentation over the student based on the historical exam
score, attendance, and late history
17. APPLYING THE CONTEXTSource
Raw&unstructured
Location
Socmed data,
Complaint,
Survey
URL access CDRDevice info
IMEI
&
TAC
Point Of Interest, sentiment library, socmed buzzer, website category
ContextDerived
Information
Commute pattern
Hangout location
Idols
Political view
Pain point
Community leader
Family member
Communication spending
18. ANALYTICS LIFECYCLE
- Defining target variable
- Splitting data for training and
validating the model
- Defining analysis time frame
for training and validation
- Correlation analysis and
variable selection
- Selecting right data mining
algorithm
- Do validation by measuring
accuracy, sensitivity, and
model lift
- Data mining and modeling is
an iterative process
Data
Mining
& Modeling
- Define variables to
support hypothesis
- Cleaning &
transforming the data
- Create longitudinal
data/trend data
- Ingesting additional
data if needed
- Build analytical data
mart
- Gathering problem
information
- Defining the goal to
solve the problem
- Defining expected
output
- Defining hypothesis
- Defining analysis
methodology
- Measuring the
business value
Data
Understanding
Business
Understanding
19. ANALYTICS LIFECYCLE
- Create monitoring
process for model
evaluation
- Evaluate the model
based on real-world
result
- Monitor and evaluate
the business impact
Model
Monitoring
- Define the model scoring
period
- Integrate model result
with execution system
(campaign system, CRM,
etc)
- Create operational
process that timely,
consistent, and efficient
Model
Operationalization
- Describe the importance
of each variable
- Visualize overall model
by creating decision tree
for example
- Define business action
based on the model
result
Model
Interpretation
Analytics and modeling is an iterative process. Data model will become
obsolete and need to evolve to accommodate changes in behavior
20. BUILDING THE
METHODOLOGY
Analysis Domain
• What is the analysis domain? Is it for male only? Is it for housewife or worker? Your
“customer” segment has different behavior
Type of Analysis
• Do we need only descriptive analysis? Or we need to go with predictive analysis?
Supervised or Unsupervised?
• Do we need to build unsupervised clustering/segmentation for this analysis?
Define Analysis Time Window
• What time window of data we need for behavior observation?
• What is the prediction time window?
• Is there any seasonal event on that time window?
21. ANALYTICS TOOLS
Microsoft Excel. Very powerful tools to do statistical data manipulation, pivoting, even doing
simple prediction
SQL is just the language. Your data lying in database? SQL will help to filter, aggregate and
extract your data
RapidMiner provide built-in RDBMS connector, parser for common data format (csv, xml),
data manipulation, and many machine learning algorithm. We can also create our own library.
Latest version of RapidMiner can connect to Hadoop and do more complex analysis like text
mining. Free version is available (community edition)
KNIME. Known as a powerful tools to do predictive analytics. Overall function is similar to
RapidMiner. Latest version of KNIME can connect to Hadoop and do more complex analysis
such as text mining. Free version is available
Tableau is one of the famous tools to build visualization on top of the data. Tableau also
powerful to create interactive dashboard. Free version is available with some limitation
QlikView. Similar to Tableau, QlikView designed to enable data analyst to develop a
dashboard or just simple visualization on top of the data. Free version is available
23. BACKGROUND
Objective
Measuring customer sentiment over big tree telecommunication provider in
Indonesia (Telkomsel, XL, Indosat)
Metric
Measuring NPS (Net Promotor Score) for each operator using twitter data.
NPS calculated as percentage of positive tweets minus percentage of
negative tweets.
Putra, B. P. (2015). Analisis Sentimen Layanan Telekomunikasi pada Pengguna Media Sosial Twitter. Jakarta: Universitas Indonesia
24. WORKFLOW
- Generate word vector
using machine
learning algorithm
based on training
dataset
- Using SVM and C4.5
- The result is 2
different model
- Select the best model
by comparing the
accuracy
Data
Modeling
- Deduplication
- Convert to lower case
- Tokenization
- Filter stop word
Data
Preparation
- Label some sample
for training dataset
- This part done with
crowdsourcing
Data
Labeling
- Create twitter crawler
with python and
twitter API
- Run the crawler with
selected keyword,
parse, and store to
RDBMS
- Collection for tweet
generated in April
2015
Data
Collection
25. WORKFLOW
- Aggregate scoring
result by telco
provider to get count
of positive tweets
and negative tweets
- Calculate the NPS for
each telco provider
- Visualize the result as
a bar chart
NPS
Calculation
- Using best model,
score the rest dataset
- Scoring result is a
label
(positive/negative/
neutral) for each
tweet
Data
Scoring
26. DATA COLLECTION
We run the crawler 3 times, one time for each operator. We only
search tweets containing some keywords
Parse the json result using json parser library embedded in python 2.7,
form it as CSV (comma separated value)
Load the csv into database (we use MySQL in this experiment)
• Telepon
• SMS
• Internet
• Jaringan
• Telkomsel
• Indosat
• XL
27. DATA LABELING
The objective is to build the ground truth
Using crowdsourcing approach. We build online questionnaire and ask
people to define each tweets if it is negative, positive, or neutral
We label 100 tweets by ourselves as a validated tweets for
questionnaire validation
We put 20 tweets for each questionnaire. 5 tweets for Indosat, 5 for
XL, 5 for Telkomsel, and the rest 5 is random validated tweets
If 4 out of 5 validated tweets answered correctly, then we flag a
questionnaire as a valid questionnaire
This approach used to eliminate the answer submitted by people who
do it randomly
28. DATA PREPARATION
Deduplication process is to remove duplicated tweets
Tokenization is a process to split a sentence into words. This should be
done because the model will generate the word vector instead of
sentence.
29. DATA PREPARATION
Filtering stop words. We eliminate non useful word (word that doesn���t
reflect to positive or negative means)
30. TOOLS USED
Data preparation modeling done with RapidMiner software
RapidMiner has text analysis function and procedure. We can found
procedure to do tokenize, convert case, deduplication, and filter stop
word
RapidMiner also has SVM and C4.5 algorithm to do modeling
31. MODEL ACCURACY
Model accuracy measurement done by confusion matrix
In this experiment, we found that SVM performs better than C4.5
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
(𝑇𝑃 + 𝑇𝑁)
(𝑇𝑃 + 𝐹𝑃 + 𝑇𝑁 + 𝐹𝑁)
32. NPS Result
After we do aggregation for scored dataset, we found that Indosat has
higher NPS than the others.
Telco % Promoters % Detractors NPS
Indosat 37% 14% 23%
Telkomsel 30% 27% 3%
XL 19% 37% -18%
34. BACKGROUND
This is the demonstration how to use Apache Spark to extract some
information from twitter data
Twitter data collected with some crawler made with python language,
and store as it is (JSON formatted data)
35. DATA EXPLORATION
Load JSON data to memory
val tweets = sqlContext.jsonFile("/user/flume/tweets/2015/09/01/*/*")
Looking the data schema, and select useful field only
tweets.printSchema
36. DATA EXPLORATION
Finding top 10 users based on tweet count
tweets.
select("user.screen_name").
rdd.map(x => (x(0).toString,1)).
reduceByKey(_+_).
map(_.swap).
sortByKey(false).
map(_.swap).
take(10).
foreach(println)
37. DATA EXPLORATION
Finding top words
tweets.select("text").rdd.
flatMap(x => x(0).toString.toLowerCase.
split(“[^A-Za-z0-9]+")).
map(x => (x,1)).
filter(x => x._1.length >= 3).
reduceByKey(_+_).
map(_.swap).
sortByKey(false).
map(_.swap).
take(20).foreach(println)
38. DATA EXPLORATION
Finding top words with stop word exclusion
val stop_words = sc.textFile("/user/ghulam/stopwords.txt")
val bc_stop = sc.broadcast(stop_words.collect)
tweets.select("text").rdd.
flatMap(x => x(0).toString.toLowerCase.split("[^A-Za-z0-9]+")).
map(x => (x,1)).
filter(x => x._1.length > 3 & !bc_stop.value.contains(x._1)).
reduceByKey(_+_).
map(_.swap).sortByKey(false).map(_.swap).
take(20).foreach(println)
39. DATA EXPLORATION
Words Chain (Market Basket Analysis)
import org.apache.spark.mllib.fpm.FPGrowth
val stop_words = sc.broadcast(sc.textFile("/user/hadoop-
user/ghulam/stopwords.txt").collect)
val tweets = sqlContext.jsonFile("/user/flume/tweets/2015/09/01/*/*")
val trx = tweets.select("text").rdd.
filter(!_(0).toString.toLowerCase.contains("ini 20 finalis aplikasi")).
filter(!_(0).toString.toLowerCase.contains("telkomsel jaring 20 devel")).
filter(!_(0).toString.toLowerCase.contains("[jual")).
filter(!_(0).toString.toLowerCase.contains("lelang acc")).
filter(!_(0).toString.toLowerCase.matches(".*theme.*line.*")).
filter(!_(0).toString.toLowerCase.matches(".*fol.*back.*")).
filter(!_(0).toString.toLowerCase.matches(".*favorite.*digital.*")).
filter(!_(0).toString.toLowerCase.startsWith("rt @")).
map(x => x(0).toString.toLowerCase.split("[^A-Za-z0-9]+").filter(x =>
x.length > 3 & !stop_words.value.contains(x)).distinct)
val fpg = new FPGrowth().setMinSupport(0.01).setNumPartitions(10)
val model = fpg.run(trx)
model.freqItemsets.filter(x => x.items.length >= 3).take(20).foreach {
itemset =>
println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}
47. SKILLS NEEDED
Business Acumen
In terms of data science, being able to discern which problems are
important to solve for the business is critical, in addition to identifying
new ways the business should be leveraging its data.
Python, Scala, and SQL
SQL skills is a must! Python and Scala also become a common language to
do data processing, along with Java, Perl, or C/C++
Hadoop Platform
It is heavily preferred in many cases. Having experience with Hive or Pig is
also a strong selling point. Familiarity with cloud tools such as Amazon S3
can also be beneficial.
SAS or R or other predictive analytics tools
In-depth knowledge of at least one of these analytical tools, for data
science R is generally preferred. Along with this, statistical knowledge also
important
48. SKILLS NEEDED
Intellectual curiosity
Curiosity to dig deeper into data and solving a problem by finding a
root cause of it
Communication & Presentation
Companies searching for a strong data scientist are looking for
someone who can clearly and fluently translate their technical findings
to a non-technical team. A data scientist must enable the business to
make decisions by arming them with quantified insights
Summarized from http://www.kdnuggets.com/2014/11/9-must-have-skills-data-scientist.html
49. [BIG] DATA SOURCES
Social media platform. Most of social media provided some API to
fetch the data from there. Twitter and Facebook is the most common
example
KDNuggets (http://www.kdnuggets.com/datasets/index.html)
Kaggle (https://www.kaggle.com/)
Portal Data Indonesia (http://data.go.id/)
Your WhatsApp group conversation