Hadoop tools with Examples

Intro to the Hadoop
T Chain
ool
HDFS, Sqoop, Pig, Hive, Map Reduce, & Oozie
By Example
Joe McT
ee

Tuesday, March 4, 2014

About Me

Principal Engineer at T
endril Networks
Experience with embedded programming, server-side
Java, big data, very basic web dev and sql

JEKLsoft is my sandbox for experiment and play
Contact me at
mcjoe@jeklsoft.com / jmctee@tendrilinc.com
@jmctee on twitter
https://github.com/jmctee for teh codez


About T
endril

We develop products that
bring consumers, utilities,
and consumer product

manufacturers together in
a partnership to save

energy while maintaining
quality of life.

http://tendrilinc.com


What is Hadoop
A distributed file system
Defaults to HDFS but others supported
An environment to run Map-Reduce jobs
T
ypically batch mode
Other
NOSQL Database (HBase)
Real-time query engine (Impala)
New uses are cropping up all the time


What is Map-Reduce
The “assembly language” of Hadoop
An algorithm for efficiently processing large amounts
of data in parallel

Rigorous mathematical underpinnings that we will
ignore

See http://en.wikipedia.org/wiki/MapReduce
Optimized for data locality
Don’t move data around, move code around, it is
smaller

Map (Wikipedia)
"Map" step: The master node takes the input, divides it
into smaller sub-problems, and distributes them to

worker nodes. A worker node may do this again in turn,
leading to a multi-level tree structure. The worker node
processes the smaller problem, and passes the answer
back to its master node.


Reduce (Wikipedia)
"Reduce" step: The master node then
collects the answers to all the sub-

problems and combines them in some way
to form the output – the answer to the

problem it was originally trying to solve.


Shuffle-Sort
After the map phase, data is “shuffled”
to combine similar data from different
mappers on a single reducer

Sometimes, to optimize, a pre-reduce is
run on the data prior to shuffle

This can decrease the amount of data
transferred to other nodes

Pig & Hive
It can be tricky to set up a problem as a
pure map-reduce job

Pig & Hive are tools that abstract the
problem definition

Underneath, they transform work into
a set of map-reduce jobs


Playing card example
For an excellent explanation, checkout:
http://www.youtube.com/watch?v=bcjSe0xCHbE

One nit, not crazy about his use of
the term magic

Makes something simple seem
complex


The Cloudera CDH4 Distro
Cloudera Manager Standard 4.8.2
CDH 4.6.0
Oozie
3.3.2

Sqoop 2
1.4.3

Hive

0.10.0

Pig

0.11.0

Map-Reduce 1
2.0.0

HDFS
2.0.0

Not covered, but included in CDH4: Hue, Map-Reduce 2,
Mahout, Impala, Solr Integration, more...


The Cluster

Using Digital Ocean Cloud Hosting for test cluster
https://www.digitalocean.com/?refcode=12768ce5992e
Shameless plug: Use this link to sign up and I get credit

Hadoop is resource hungry, particularly for RAM
Was unable to get install to work with < 4 GB
$40/month per instance and I have 4 (OUCH!)
So I only spin them up when I am testing
Currently, requires reinstall of Hadoop on spin
up. Have it down to ~60 minutes.

Reinstall required because Hadoop stores IP
addressed instead of host names (grrr!).

The Cluster (cont)
hadoop-nn

Name Node
Job T
racker
Hive MetaStore
Sqoop MetaStore
Oozie Server

hadoop-dn1

hadoop-dn2

hadoop-dn3

Data Node
T
ask T
racker
Hive Gateway

T DB
est
(postgreSQL
All Remote Nodes:
CentOS 6
Dual Core
4GB Ram
60GB SSD Disk
Hosted in New York
$0.06/hr


My Mac

The Code

T
endril Home Energy Report

A real world product produced with
Hadoop


Home Energy Report (pg 1)

Compared to similar homes

Current month’s usage
comparison


Customized savings tips and
tricks

Home Energy Report (pg 2)
Historical usage comparison

Customizable utility
marketing messages
- Service notices
- Program announcements


Not Just about the Petabytes
1.7 million total accounts

Eligibility, e.g., voluntary opt out, control group, and VIP
500K “control” group accounts
13 months of billing data for most accounts
9 prioritized demographics sources
Accounts clustered into “like” neighborhoods
Dynamic clustering use Mahout (Canopy & K-Means)
Business Rules, e.g., no rentals
Customized savings tips
Proprietary physics-based building model
Customizable utility marketing messages
Filters, e.g., not enough bills, small neighborhood, address hygiene
End of run analytics

Resulting in...
1.2 Million reports per month (and growing)!

M&V (Energy savings of treatment vs control groups)
is ongoing

Initial value is redacted kWh per household


The “Contrived” Problem
An energy monitoring company has the following data:
List of accounts, some customers have solar, some don’t
Hourly residential solar production readings
Weather data for all customer locations
Sunrise/Sunset data for all customer locations
A list of “Do not call” accounts not to market to
These include solar and non-solar customers


The “Contrived” Problem (cont)
Need to create daily sunrise-sunset graphs (png’s?)
for each solar customer

Should integrate weather data in the graphs
T
emperature and cloud cover
Will be added to newsletters sent to the customers
Need to create a report for marketing containing all

accounts without solar who can be contacted as part
of the current marketing campaign
Other contrivey things...


How Do We Get Data Into
the Cluster?


The HDFS CLI

For some data types, this is the easiest way to import
data

T
ypically used for static files that don’t get
updated a lot


The HDFS CLI (cont)
Command format:

hadoop fs -<command> args
File specification: From network
Use hdfs://<namenode>:8020 prefix
e.g. hdfs://hadoop-nn:8020/tmp
From name node, prefix not needed
Commands are mostly familiar linux file commands
ls, cat, mkdir, rm, cp, mv, etc.
A few hdfs specific
put, getmerge, copyfromlocal, copytolocal, etc.

The HDFS CLI (cont)
Some hdfs specifics

Files are stored in blocked chunks
Default is 128 MB
So not optimized for small files
Blocks for a given file not necessarily stored on the same
data node
But will be replicated on more than one node
Name Node keeps track of what’s where
hdfs won’t overwrite a file
Pattern to use is delete, then write
Or write to new file, then delete old file and move
Directories sometimes act as files

A note on examples
Will use the scriptRunner.sh tool from project
(mostly)

Example slides will list the scripts to run
Scripts are run on name-node (hadoop-nn)
For convenience, could run elsewhere
Assumes cluster is configured and ssh keys
have been configured to avoid passwords


HDFS Example
We have some bulk files on our filesystem
that we need to import

prepareForT
alk.script (1)
bulkCopy.script (2)
Review /solar in HDFS browser
We’ll come back to some other HDFS
commands later

Let’s Sqoop some data!
As cool as NOSQL is, there are still a
“couple” of relational DB’s out there

Sqoop is the tool to import (and export)
from SQL DB’s

T avoid typing passwords...
o
.pgpass in my home dir
/home/root/pg_password in HDFS

Sqoop Example 1
Populate the DB with 3 accounts
populateWith3.script (3)
Now sqoop data
simpleSqoop.script (4)
Note that data is now in /solar/accounts
Also note that sqoop has helpfully created a

file, accounts.java, for your use, if you choose.


A Brief Digression
Whats up with this file?
The data was processed by 3 mappers
Still one file, but written by three processes
Let’s use getmerge to view this file


Example - getMerge
getMergeAccounts.script (5)
Note that all “part” files have been concatenated
(order is not guaranteed, don’t count on it).

Note the crc file. If you intend to modify this file and

re-upload, make sure and delete it, or the upload won’t
work.


Sqoop Example 2
Populate the DB with 3 more accounts
populateWith3More.script (6)
And sqoop data
simpleSqoop.script (7)
Oops...
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /solar/accounts already exists


Let’s Fix This
Could delete file and sqoop all data each time
Should we sqoop 1.7 million accounts when 10K
changed?

Let’s add some new options to our sqoop
--incremental append
--check-column last_updated
--last-value '2000-01-01'
Requires relational table to have an appropriate
column


Sqoop Example 3
removeAccounts.script (8)
And sqoop data
advancedSqoop.script (10)
Now edit script (yuk!), then
advancedSqoop.script (12)

Enter Sqoop Saved Job
Job details are saved in sqoop metastore
By default, an HSQL DB
After job runs, last value is auto-updated
Had to make some changes to support Oozie
Start a server (on name node):
sqoop metastore --meta-connect jdbc:hsqldb:hsql://hadoop-nn:
16000/sqoop &
Add argument to sqoop commands
--meta-connect jdbc:hsqldb:hsql://hadoop-nn:16000/
Not needed for command line use


Sqoop Example 4
createSavedJob.script (13)
runSqoopJob.script (16)
runSqoopJob.script (18)


More on saved jobs
sqoop job --list
sqoop job --show sqoopAccounts
sqoop job --delete sqoopAccounts


Now let’s process the data
Pig
A DSL for creating data processing pipelines
DSL is converted into series of map-reduce jobs
DSL is called pig latin
REPL-like shell called Grunt
Easy to add User Defined Functions (UDFs)
For our system, written in Java
Stored in the piggybank
Who says devs don’t have a sense of humor?


Step 1 - Date Normalization

Dates in the raw data are stored in different formats
Later processing will be easier if we convert to a
common format

Will use a UDF for the actual conversion, pig will make
sure processing is parallelized
This UDF is an evalFunc
Will use a filterFunc in next example
Many others, these seem to be the most common
normalize.script (19)
While it is running, we’ll look at the code


Step 2 - Aggregate
Want to merge account info, solar, and weather
Remove night time readings (a filterFunc UDF)
Remove dupes (this was a reported defect)
Calculate day of reading (yup, an evalFunc UDF)
Group data by account and day of reading
Sort by time of individual readings
mergeAndFilter.script (20)
While it is running, we’ll look at the code


Contrived Map-Reduce
Group all solar readings
by account/date (day)/system size
This example is trivial, we already solved similar
problem in Pig, but...

Long term, we want this data to be turned into a

graph (probably a png) and stored in HDFS for
other processes to use (stamped onto pdfs or
emails).

M-R is a good tool for this (in conjunction with
other libraries)


Map Reduce Example 1 a & b
Run scripts/runBasicAggregator.sh
Note: run from ~, not scripts directory


Map Reduce Example 2
Reformat the data, don’t include the key
Mapper is same as previous example
Added a combiner, same code as reducer in
previous example
New reducer
Run scripts/runAdvancedAggregator.sh


Map Reduce Example 3
Save each reading in separate file (long term, this will
be the png, not a csv)

Demonstrates custom output format class
Mapper and Combiner are still the same, new Reducer
Run scripts/runMultiOutput.sh
Results stored in hierarchical structure
/solar/daily_multi/account/date
See example /solar/daily_multi/3/20130701


Hive
A SQL-like tool for bulk data query
Often referred to as data warehousing
Most things you can do in Pig can be done in Hive and
vice versa

Hive is designed for SQL devs (I’m not one)


Hive Example 1

Simple example
Find which accounts are solar and which are not
getSolarAndNonSolarAccounts.script (21)


Hive Example 2

Need a list of all non-solar customers who are not on do not call list
Would like to do the following (but can’t):
SELECT accounts.id
FROM accounts
LEFT JOIN do_not_call ON accounts.id <> do_not_call.id
WHERE accounts.capacity == 0;
Hive’s syntax is “almost” SQL
Biggest incompatibility is that only equi-joins are supported (will likely
be fixed in a future release)
There is a hacky work around using an outer join (which can be
expensive!)
getAccountsT
oContact.script (22)


Hive Example 2 (cont)
Why is the outer join expense?
Creates more intermediate rows that we need, which then have to be
filtered
Account ID

4
5
6
7
8
9

DNC ID

NULL
5
NULL
7
NULL
NULL

Oozie example
Will take a while to run, so will start, then
discuss

Workflow bundle must be deployed on HDFS
TODO: incorporate into gradle script
deployLibs.script (only when jar
dependencies change)

deployWorkflow.script - when app changes


Oozie example (cont.)
oozie.script (26)
Review progress in Oozie Console


Oozie - Putting it all together
Oozie is Hadoop’s workflow scheduler
Oozie Workflow jobs are Directed Acyclical Graphs
(DAGs) of actions.

Oozie Coordinator jobs are recurrent Oozie Workflow
jobs triggered by time (frequency) and data
availability.

Oozie is integrated in Hadoop stack supporting

several types of jobs out of the box (map-reduce, Pig,
Hive, Sqoop, java, etc.)


But...
It puts the hell in XML-hell!
Anti-DRY
The web-based tool is kind of lame
LOTS of clicking to find what you are after
Somewhat non-intuitive interface


workflow.xml
Describes your processing pipeline
Each job (pig, MR, hive, java, etc.) is enclosed
in an action block

Prepare - setup steps
Configure - define properties
OK - next step after success
Error - next step after a failure

Fork/Join
Easier than scripting the same
Allows independent jobs to run in parallel
Makes optimal use of your cluster
resources


Sqoop workaround
Evidently, when using a password file, there is a known
issue in Sqoop where it closes filesystem prematurely
Workaround
Set fs.hdfs.impl.disable.cache to true
http://stackoverflow.com/questions/19735672


Running in parallel


A successful run...


References

This project, https://github.com/jmctee/hadoopT
ools
Hadoop: The Definitive Guide, 3rd Edition - T White
om
http://shop.oreilly.com/product/0636920021773.do
Programming Pig - Alan Gates
Programming Hive - Edward Capriolo, Dean Wampler, & Jason Rutherglen
Apache Sqoop Cookbook - By Kathleen Ting & Jarek Jarcec Cecho
Oozie
http://oozie.apache.org/docs/3.3.2/
Cloudera CDH4 Documentation
http://www.cloudera.com/content/support/en/documentation/cdh4-documentation/cdh4documentation-v4-latest.html
CDH Users Forum
https://groups.google.com/a/cloudera.org/forum/#!forum/cdh-user


Thank You! (and go solar!)


Hadoop tools with Examples

Related slideshows

More Related Content

Hadoop tools with Examples