CloudStack Metering – Working with the Usage Data

CloudStack Metering
Working with the Usage Data
Tariq Iqbal
Senior Consultant
tariq.iqbal@shapeblue.com
Twitter: @TariqIqbal_ @ShapeBlue

@ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
 Involved with CloudStack before donation to Apache
 Built and deployed CloudStack/CloudPlatform based clouds for
Enterprises and Service Providers globally
 Specialise in integrating CloudStack with Business Support and
Operational Support systems, which include:
 CloudPortals (including Payment Gateways, Fraud Control solutions)
 Ticketing Systems
 CRM/ERP Systems
 Monitoring Solutions
About Me

“ShapeBlue are expert builders of public & private
clouds. They are the leading global independent
CloudStack / CloudPlatform integrator &
consultancy”
About ShapeBlue

 CloudStack metering - What and how?
 CloudStack configuration to enable Usage capture
 Usage Data generation process
 Methods of accessing the Usage Data
 Real World Use Cases
Overview

 Organisations need to be able to meter their cloud's
consumption and monetize the cloud offerings by charging for
usage
 Majority of organisations already have existing billing or business
support systems and just need access to CloudStack Usage Data
 Organisations tend to have different billing criteria and charging
models requiring a comprehensive integrated solution
Metering Requirements

CloudStack Usage?
CloudStack Metering

 CloudStack does not offer any integrated usage monitoring or
billing/chargeback capability
 Root Admin can view
System Capacity
 Domain Admin and Users can view
VM status
Metering in CloudStack today

 CloudStack includes a Usage Server that creates summary usage records for
the various resources consumed in CloudStack
 The Usage Server is an optional, separately-installed component called
‘cloudstack-usage’
 The Usage Server runs at least once per day. It can be configured to run
multiple times per day and it’s behaviour is controlled by the following global
configuration settings:
•
Cloud Usage Server

 As CloudStack resources are created, consumed and
destroyed, appropriate Event records are created in the ‘cloud’
database table: usage_events
 There are 110
different Events:
(51 shown)
CloudStack Events

 The Usage Server schedules a job at the time specified by
usage.stats.job.exec.time parameter. This job queries the ‘cloud’
database table: usage_events and generates summary usage records in
the ‘cloud_usage’ database table: cloud_usage as follows:
1. CloudStack events have been logged into the cloud.usage_event table
2. cloudstack_usage job gets a list of the latest usage events
3. Inserts the latest usage events into the cloud_usage.usage_events table
4. Helper tables populated from cloud_usage.usage_event table after
parsing
5. cloud_usage.cloud_usage table populated with aggregation range wise
data. Helper table data is used here
Usage Record Generation

 The Usage Server collects 14 different types of resource usage:
Usage Types

 The Usage Records show the amount of resources consumed by guest instances.
 The Usage Record format for Allocated & Running VMs is shown below:
 The other 7 Usage Record formats are:
 Network Usage
 IP Address
 Disk Volume
 Template, ISO & Snapshot
 Load Balancer & Port Fwd
 Network Offering
 VPN User
Usage Records

Usage Record Example

Accessing Usage Data
CloudStack Metering

 The usage records can be accessed through the CloudStack API
 This is best done using the authentication port 8080 and requires
both the API Key and Secret Key and for the API request to be
signed
 CloudStack provides the following Usage API calls with Root
Admin access and supports XML and JSON response formats
 listUsageTypes – list of Usage Types accessible
 generateUsageRecords - asynchronous usage record generation
 listUsageRecords - provides Usage records for a date range
CloudStack API

 The base command is:
http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y
yyy-MM-dd&enddate=yyyy-MM-dd&signature
 Append the various options:
ListUsageRecords API

 <listusagerecordsresponse>
<count>1</count>
<usagerecord>
<account>user5</account>
<accountid>10004</accountid>
<domainid>1</domainid>
<zoneid>1</zoneid>
<description>i-3-4-WC running time (Service Offering: 1) (Template: 3)</description>
<usage>2.95288 Hrs</usage>
<usagetype>1</usagetype>
<rawusage>2.95288</rawusage>
<virtualmachineid>4</virtualmachineid>
<name>i-3-4-WC</name>
<offeringid>1</offeringid>
<templateid>3</templateid>
<usageid>245554</usageid>
<type>XenServer</type>
<startdate>2009-09-15T00:00:00-0700</startdate>
<enddate>2009-09-18T16:14:26-0700</enddate>
</usagerecord>
</listusagerecordsresponse>
ListUsageRecords Response

 CloudMonkey is a CLI tool for CloudStack and can be used as an
interactive shell or from within a shell script to call the API
 It can output the Usage Data in both JSON and Tabular form to a
file or pipe it to another application for further processing
 Filtering can be used to limit the result set
 It supports argument passing and shell automation
 The host, port, apikey and secretkey can be configured on setup
CloudMonkey CLI

CloudMonkey list usagerecords
The command to retrieve Usage Records is:
cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f-
080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2

 The usage records can be also accessed by directly querying the
cloud_usage.cloud_usage MySQL table in CloudStack
 Usage Data can be exported in a CSV format from a SELECT
query
 The SQL query can become complex if the ID fields in the data
need to be de-referenced
 Typically multiple passes of the query results may be required to
extract the required information
SQL

 Once you are able to access the Usage Data, you will need to
design the mapping between the fields in the CloudStack Usage
Data and the fields of the target system.
 Considerations:
 Do you actually need all the Usage Data for all Resources?
 Any UUID values of the CloudStack resources should also be
imported into the target system for reference purposes
 Decide which system will be the master for the user account data
and if synchronisation is required
Mediation

Real World Use Cases
CloudStack Metering

 MS Excel can be used to analyse CloudStack Usage Records
 The usage records are imported from the cloud_usage.
cloud_usage table into MS Excel via an MySQL ODBC connection
 Either PivotTables or reports can be used to present the data.
 This is convenient and
great for integration
testing between CloudStack
and external systems.
MS Excel

 Citrix CloudPortal Business Manager tightly integrates with the
cloud_usage database on CloudStack/CloudPlatform.
 The Usage Data is cross referenced against the utility pricing and
subscription pricing in CPBM.
 Scheduled Billing and Invoicing jobs generate the invoices in XML
or PDF based on the
tenants billing cycle.
Citrix CloudPortal

 splunk> is an operational intelligence tool that was initially used
for collecting and indexing infrastructure logs
 We then connected splunk> to the CloudStack Usage database
and automated the collection and indexing of Usage Data
 Setup feature-rich dashboards
and the capability to drill down into
the Usage Data for BI reporting
splunk>

 CloudStack Developers Guide, API Documentation & Wiki:
 http://cloudstack.apache.org/
 CloudMonkey
 https://cwiki.apache.org/CLOUDSTACK/cloudstack-cloudmonkey-
cli.html
 How to Use MS Excel to Analyze CloudStack Usage Records:
 http://support.citrix.com/article/CTX132030
 CloudStack Mailing Lists (Users, Development, Marketing):
 http://cloudstack.apache.org/mailing-lists.html
Further Information

CloudStack Metering – Working with the Usage Data

Related slideshows

More Related Content

CloudStack Metering – Working with the Usage Data

Editor's Notes