fiware-lab-dev-6.pdf

Yahya Al-Hazmi | Technische Universität Berlin
yahya.al-hazmi@tu-berlin.de
XIFI Webinar | GoToWebinar | February 23, 2015, 11-12 AM CET
FIWARE Lab Solution for Managing
Resources & Services in a Cloud
Federation
http://lab.fiware.org

Agenda
 Introduction
 FIWARE Lab
 Federation Architecture Overview
 Cloud Federation Management
 Conclusion
 Discussion
1

Introduction (1)
 We need an open ecosystem or platform for open innovation
 Cloud capacity and services are of great value to support open innovation
and facilitate start-up incubation through infrastructure resource
 How to create a large distributed cloud to boost innovation across
European regions?
• This requires
› either large investments by a single player or
› an agreement among many players (federation)
• Of course, when different players team up together within a federation, they do not
want to lose all the control on their own infrastructure.
 There are several challenges and motivations behind cloud federation in
the literature, as well as different federation models and approaches
2

Introduction (2)
 There is no „best“ cloud federation approach
• Different approaches exist and each has its pros/cons
• Depend on different requirements
 We focus on Cloud Federation conducted within XIFI project to build the
federation of cloud infrastructures for the Future Internet Lab (FIWARE Lab)
 FIWARE Lab:
• Is an open innovation platform to develop Future Internet (FI) applications
• Offers a rich catalogue of services available either in SaaS or PaaS modality, the
so called Generic Enabler implementations (GEis)
› FIWARE GEs: set of general-purpose platform functions available through APIs
• Provides also a wide offer of FI facilities (e.g. sensor networks, 4G networks,
etc.) that provide advanced experimental capacities to developers allowing them
to link their applications with actual infrastructures and test them in real world
settings
3

FIWARE Lab
 FIWARE Lab is the Community Cloud for European FI-PPP developers
enabled by advanced FI infrastructures in Europe
 XIFI, as part of the overall vision of FI-PPP and following the principle “eat
your own dog food”, is based on FI-PPP technologies delivered by FIWARE
4

FIWARE Lab: the “meeting point”
where innovation takes place
5
Entrepreneurs, Developers
• Develop once for a large market
• Easily meet potential customers
• Marketing, promotion
• Ability to test with real data and
end users
• Simple yet powerful APIs that
accelerate product development
App Customers and Data providers
• Connect to entrepreneurs
• Put their data at work
• Bring new innovative services to end
users
• Be more efficient
• Social Reputation
FIWARE Technology Providers
• “Competitive” approach
• Connect to entrepreneurs: jointly
exploit the opportunities
 4,2 M€ promotion campaign
• Campus Party events
• Startup Weekend events
• Chambers of Commerce
• 870 K€ in prizes
 100 M€ of funding devoted
to entrepreneurs in phase 3
of the FIWARE program

Distributed Cloud for FIWARE Lab
6
• 17 nodes in Europe
providing up to
3000+ cores,
16TB+ Ram,
750TB+ HD
• A node is a data-
center (or more
linked)
• Connectivity node-
to-node offered
through GEANT /
NRENs
• Service topology
depending on GEs
release and UC
requirements
• 17 nodes in Europe
providing up to
3000+ cores,
16TB+ Ram,
750TB+ HD
• A node is a data-
center (or more
linked)
• Connectivity node-
to-node offered
through GEANT /
NRENs
• Service topology
depending on GEs
release and UC
requirements

The Federation “topology”
Identity Federation
Identity Federation
Spain
(M)
Spain
(M)
Italy
(M)
Italy
(M)
France
(S)
France
(S)
Ireland
(S)
Ireland
(S)
German
y (S)
Load Balancer
Synchronize
Spain
(M)
Italy
(M)
France
(S)
Ireland
(S)
Germany
(S)
Note:
• Not all the links are
shown
• For simplicity only five
nodes are depicted
M (master):
management and
operation layers
S (slave):
operation layer
7

General Architecture
8
Centralized Federation
Management
• Find available resources
• Compare resources
• Deploy GEs
• Federated Monitoring
Distributed Resources
• Access GEs
• Access VMs
• Monitoring data
• IdM & AC
• Network configuration

FIWARE Lab cloud federation management
from App developers’ viewpoint
 Focusing on innovative components that, working in an orchestrated
manner, perform core operations dealing with the management of
resources and services from an application developer viewpoint, in
particular:
i. the ability to deploy multi-tier applications across the federation
a. Due to regulatory or security reasons each tier may need to be deployed in different
infrastructure
b. Select among a set of configuration tools helping developers to minimize chronophage
operations
ii. the provisioning of real-time information about the services (GEis) available
in the federation through standardized RESTfull API
iii. a centralized mechanism to handle large-scale monitoring data gathered
through the federated underlying resources, by establishing a well-defined
and standardized API for storing, aggregating and publishing such data
9

Simplified architecture: cloud federation
management from App developers’ viewpoint
10
IdM: Identity Management
IMM: Infrastructure Monitoring
Middleware
DCRM: Data Center Resource
Management ( OpenStack-
Based)

Platform-as-a-Services (Pegasus)
 Enables a user to deploy easily any kind of application (single VM, single GEi or
multiple GEis)
 Offers the opportunity to deploy multi-tier applications, where each tier can be
accommodated by a different infrastructure of the federation
11
 No restriction of the
platform technology
• The developer is free to
select his own platform
• Products that will be
supported: J2EE, Apache,
DB: MySQL, PostgreSQL,
PHP, …
 Automate how you build,
deploy, and manage your
infrastructure using Chef
or Puppet (Sagitta)

Platform-as-a-Services (Pegasus)
 Deployment design: concrete number of machines, VM structure: products,
application components, load balancers, final network structure
 Deployment execution: orchestration of different steps: configuration of the
cloud services, configuration of IaaS/NaaS, creation of the VMs with the
software inside, configuration of the monitoring system, etc.
 Infrastructure control layer: creation and configuration of VMs+software,
interaction with IaaS/NaaS providers, deployment and execution of VMs
 Monitoring: configuration of the monitoring probes in order to recover the
data from the VMs, network and or process
 Adaptation and scalability of the applications (multi-tenancy)
12

Deployment and Configuration Adapter (DCA)
 DCA caters for the persistency of all
pertinent information related to the
whole lifecycle of services (GEis)
 DCA exposes a RESTful API that
can be used by interested users to
collect all needed information
regarding GE instances available in
the cloud federation either as SaaS
or PaaS offerings
 DCA is a flexible component that
accommodates different accessing
policies, following infrastructure
owners’ requirements
13

Deployment and Configuration Adapter (DCA)
 Challenges:
i. Respecting potential stringent access policies applied by the infrastructure
owners (Slave Nodes) not willing to provide administrative privileges to external
parties
ii. Unambiguous identification of the deployment of a specific GEi in several
infrastructures
 Solutions:
i. Software component (Python script), which is able to collect respective
information through the OpenStack Nova and Glance components, is used by
the infrastructure administrator to edit and customize the parameters of the
Python script before installing it on the controller of the infrastructure (DCRM)
ii. The information is made available to the DCA through a particular configuration
done by the Pegasus during GEi deployment through the use of the
metadata service offered by the OpenStack
› inserting a specific value (called NID) in the Glance metadata that uniquely and
unambiguously allows for GEi identification across the cloud federation
14

Federation Monitoring
 Attached to the local monitoring
system(s), a cross-domain
adaptation mechanism, denoted
as Infrastructure Monitoring
Middleware (IMM), unifies the
format of and the accessibility
to the collected data
 The Federation Monitoring
fulfilling the next operational
layer is in charge of storing and
publishing the unified data-set
by defining a Fed. Monitoring
API
• This layer is able to elaborate
the data by leveraging on Big
Data analysis techniques and
providing aggregation features
15

 Fed. Monitoring components:
• Context Broker: the IMM notify
the monitoring data as NGSI
context format into the CB that
handles the datasets and updates
the metrics into Hadoop
• Apache Hadoop: provides
scalable, reliable and distributed
data processing and storage
› perform aggregating operations to
the data coming from the
subscription with the CB
› all Hadoops are federated in order
to maintain the service in high
availability
› only the Hadoop deployed in the
Master Node is allowed to perform
operations with the relational
database
16

• Relational database: required to
store the elaborated data
• Fed. Monitoring API Server: API
for users to access the processed
monitoring data stored in the
relational database, and real-time
data from the federation
 Only master node hosts specific
federation-aware functionalities
17

Managing resources in multi-tier application:
a real case scenario
 An application developer willing to offer a weather service
• by allowing subscribed users to check, via a website, weather conditions or be
automatically informed of sudden weather changes
• Weather information are collected through a network including several sensors
 S/he selects:
• the Orion Context Broker GEi: to publish the data collected by the sensor
• the Complex Event Processing GEi: to process the information from sensors and to
create events through customized threshold
• the WireCloud GEi: to properly display this information in a webpage,
 Due to regulatory reason, Pegasus deploys them in two clouds (two-
infrastructure application)
 DCA allows a user to query its API providing all VMs created by him
 Monitoring API is used to get monitoring infos using a unique ID of the VM
18

Conclusion
 Addressed FIWARE Lab solutions
• for seamless deployment of services across the federation and ability of services to
span across different members of the federation
• For monitoring of the resources and data which can be aggregated with a common
structure
• be offered as an open ecosystem for innovation at the developers’ disposal
 These solutions are implemented and deployed in FIWARE Lab that includes
a running federation of 17 infrastructures distributed across Europe
 Developers can get an account by registering through the Cloud Portal and
enjoy the offerings
 How to use/interact/start working is presented in the upcoming Webinar “XIFI
for developers” on Wednesday, February 25, 2015.
19

Acknowledgment
The research work has received funding
from the EU FP7 grant agreement no.
604590 XIFI Project. The authors would
like to thank the project partners for their
contribution, especially Panos Trakadas,
Pablo Rodríguez, Silvio Cretti and Attilio
Broglio.
20

http://xifi.org
http://fiware.org
http://lab.fiware.org
Join us!
21

fiware-lab-dev-6.pdf

Related slideshows

More Related Content

fiware-lab-dev-6.pdf