Cloud Foundry for Data Science

1© Copyright 2015 Pivotal. All rights reserved.
@ianhuston
1© Copyright 2014 Pivotal. All rights reserved. 1© Copyright 2015 Pivotal. All rights reserved. 1© Copyright 2015 Pivotal. All rights reserved. 1© Copyright 2014 Pivotal. All rights reserved.
Cloud Foundry
for Data Science
Ian Huston, Data Scientist

@ianhuston
Who am I?
Ÿ  Ian Huston
Ÿ  @ianhuston
Ÿ  github.com/ihuston
Ÿ  www.ianhuston.net
Ÿ  Talk resources:
http://tinyurl.com/cf4ds-talk
Ÿ  Data Scientist
Ÿ  Previously a theoretical
physicist using Python for
numerical simulations & HPC
Ÿ  Use Python/R/SQL/… for
predictive analytics and
machine learning

@ianhuston
What is Cloud Foundry?
http://cloudfoundry.org
Open Source Cloud Platform
Simple App Deployment,
Scaling & Availability
No Cloud Provider Lock In

@ianhuston
How can CF help data scientists?
Ÿ  Jamie is a data scientist who has just finished some
analysis. They want to put up a simple internal web app with
Javascript visualisations connected to internal data stores.
Ÿ  Sam is a data engineer who wants to set up a REST API to
expose a production machine learning model as a service.
Ÿ  Alex is a data scientist who has an existing RShiny or
Python app that they want to make available with multiple
instances.

@ianhuston
Cloud Foundry is a Platform
You bring the apps, the rest
is taken care of!
Source: Albert Barron (IBM),
https://www.linkedin.com/pulse/20140730172610-9679881-pizza-as-a-service

@ianhuston
Cloud Foundry Foundation: Industry Standard
Gold
Silver

@ianhuston
CF for data scientists & developers
Easily deploy your web app
cf
push
myapp

Scale up and out quickly
cf
scale
myapp
–i
5
–m
1G

Create and bind services
cf
bind-‐service
myapp
redis

@ianhuston
Simple Python Flask App Demo
Ÿ  Simple one page “Hello World” web app
Ÿ  Video: https://www.youtube.com/watch?v=QOfD6tnoAB8
Ÿ  Demonstrates:
–  Installation of requirements
–  Scaling properties
Ÿ  Need to Provide:
–  App files
–  Dependencies listed in requirements.txt file
–  Optional manifest.yml file with configuration for deployment

@ianhuston
Official Language Support on Cloud Foundry
Go
Ruby

@ianhuston
Buildpacks – How to Customise CF
Use Heroku-like buildpacks to add additional runtimes:
Ÿ  Community Buildpacks for Clojure, Haskell, .NET, Erlang,…
Ÿ  R Buildpack
–  https://github.com/alexkago/cf-buildpack-r
–  Works with RShiny apps
Ÿ  PyData Buildpack
–  https://github.com/ihuston/python-conda-buildpack
–  Uses conda for hard to install dependencies

@ianhuston
Comparison with Docker
runtime layer
OS image
application layer
Docker
system brings fixed
host OS Kernel
* Devs may bring a custom buildpack
runtime layer*
OS image
application layer
CF Buildpack
system brings fixed
host OS Kernel
App container
System Provides
Dev Provides

@ianhuston
Services now available
These are available on Pivotal CF (Pivotal’s packaged Cloud Foundry offering).
See http://run.pivotal.io for the services available on Pivotal Web Services.

@ianhuston
Data Services
Ÿ  Easy to bind and scale services
–  Databases, NoSQL, message queues etc.
$
cf
create-‐service
rediscloud
PLAN_NAME
INSTANCE_NAME

$
cf
bind-‐service
APP_NAME
INSTANCE_NAME

Ÿ  Details provided in VCAP_APP_SERVICES env variable:
{
"rediscloud":
[

{
"name":
"rediscloud-‐42",
"label":
"rediscloud",
"plan":
"20mb",

"credentials":
{
"port":
"6379",

"hostname":
"pub-‐redis-‐6379.us-‐east-‐1-‐2.3.ec2.redislabs.com",

"password":
"your_redis_password”

}

}
]

}

@ianhuston
What can you build with CF?
Some ideas:
Ÿ  Interactive data exploration apps
Ÿ  APIs for a machine learning model
Ÿ  Internet of Things data collection and modelling tools
Ÿ  Microservices architecture for data analysis
Ÿ  Interface to a natural language system like IBM’s Watson

@ianhuston
Examples
Ÿ  Simple HTML + JS
app
Ÿ  Uses data in S3
Ÿ  Scale out when
heavy load
expected with auto
load balancing
http://ds-demo-transport.cfapps.io

@ianhuston
Examples
Ÿ  RShiny app
Ÿ  Interactive controls
with very little
additional coding.
Ÿ  Runs RandomForest
model in background
https://ak-insurance-demo.cfapps.io:4443/

@ianhuston
Send me your examples!
Show off your data science related Cloud Foundry
apps:
Twitter: @dsoncf
http://dsoncf.com
http://DataScienceOnCloudFoundry.com

@ianhuston
Try Cloud Foundry today!
Pivotal Web Services: http://run.pivotal.io
IBM Bluemix: http://bluemix.net
Anynines: http://anynines.com
HP Helion: http://horizon.hpcloud.com

@ianhuston
More info and resources
Ÿ  Material for this talk: http://tinyurl.com/cf4ds-talk
Ÿ  Cloud Foundry: http://cloudfoundry.org
Ÿ  Python Flask Mega Meta Tutorial for DS:
http://tinyurl.com/flaskDS
Ÿ  Twelve Factor Apps: http://12factor.net/
Ÿ  Meetups: http://cloud-foundry.meetup.com/

Cloud Foundry for Data Science

More Related Content

Cloud Foundry for Data Science