How to create a secured cloudera cluster

How-to create a Secured
Cloudera Cluster
Cloudera Manager + Kerberos + CentOS

Introduction
This presentation, it’s for everyone that is curious with Big Data and does have the know
how to start learning...
With this, you will be able to create quickly a Kerberos secured Cloudera Cluster.

Architecture
This architecture enables a Cluster with several layers of security.

Pre-Assumptions
1. VM Centos 6.9
2. VM Requirements 26Gb with 4 vcores
3. root user or user with sudo access

Prepare Cloudera Installation
Set VM Hostname
sudo nano /etc/hosts;
10.111.22.3 cm515.localdomain
sudo hostname cm515.localdomain;
sudo nano /etc/sysconfig/network;
HOSTNAME=cm515.localdomain
Clear iptables Rules
sudo iptables -F;
sudo nano /etc/rc.local;
iptables -F
Note: just need to change the highlighted values for your own values.
Make iptables Rules Executable on Startup
sudo chmod +x /etc/rc.d/rc.local;
sudo sed -i
's/IPTABLES_SAVE_ON_RESTART="no"/IPTABLES_SAVE_ON_RESTART="yes"/'
/etc/sysconfig/iptables-config;
Make Cloudera Manager Rules Executable on Startup
echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local;
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local;
echo never > /sys/kernel/mm/transparent_hugepage/enabled;
echo never > /sys/kernel/mm/transparent_hugepage/defrag;

Prepare Cloudera Installation
Disable Selinux
sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/'
/etc/selinux/config;
sudo setenforce 0;
Reboot
sudo reboot
Check Selinux
sestatus
Install & Start NTP service
sudo yum install ntp;
sudo service ntpd status;
sudo service ntpd start;
Configure NTP service
sudo nano /etc/ntp.conf
server 10.111.22.5 prefer iburst burst minpoll 4 maxpoll 4
server 10.111.22.6 iburst burst minpoll 4 maxpoll 4
sudo service ntpd restart;
Change Host Check Parameters
sudo sysctl vm.swappiness=10;
sudo echo "vm.swappiness = 10" >> /etc/sysctl.conf
Disable IPV6
sudo nano /etc/sysctl.conf
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.all.disable_ipv6=1

Cloudera Manager Installation
There’s many ways to do the Cloudera Cluster installation, such as a installation binary wizard file.
But i've found that the most reliable way is the following:
Create Cloudera Repo
sudo nano /etc/yum.repos.d/cloudera-manager.repo
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.15.1/
gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1
sudo yum repolist;

Install Java & Cloudera Server + Agent
sudo yum install oracle-j2sdk1.8 -y;
sudo yum install cloudera-manager-agent -y;
sudo yum install cloudera-manager-server -y;
Install JDBC Connector for MariaDB
sudo mkdir -p /usr/share/java/;
cd /usr/share/java/;
sudo chmod 777 /usr/share/java;
sudo wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz;
sudo unzip mysql-connector-java-5.1.46.zip;
cd mysql-connector-java-5.1.46;
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar;
sudo chmod 777 /usr/share/java/mysql-connector-java.jar;
Note 1: just need to change the highlighted values for your own values.
Note 2: Cloudera agent installation must be done in each cluster node and the server only on the server node.
Note 3: It was chosen the MariaDB for Cloudera DB due to is reallibily and easy configuration.
MariaDB Repo configuration
sudo nano /etc/yum.repos.d/MariaDB.repo;
[mariadb]
name = MariaDB
baseurl =
https://yum.mariadb.org/10.0/centos/6/x86_64/
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaD
gpgcheck=1

MariaDB Server Installation
sudo yum clean all;
sudo yum install mariadb-server -y;
Enable MariaDB Start on boot
cd /etc/init.d;
sudo chkconfig --add mysql;
sudo chkconfig mysql on;
sudo chkconfig --list mysql;
sudo service mysql start;
Set MariaDB root password
sudo mysql_secure_installation
Remove anonymous users? [Y/n] - Y
Disallow root login remotely? [Y/n] - N
Remove test database and access to it? [Y/n] - Y
Reload privilege tables now? [Y/n] - Y
Create Cloudera Services Databases & Users
mysql -u root -p
- - User: ClouderaManagerServer
create user 'scm'@'%' IDENTIFIED BY 'password';
grant all privileges ON *.* TO 'scm'@'%' WITH GRANT OPTION;
create database scm default character set utf8;
- - User: Sentry
create database sentry default character set utf8;
grant all on sentry.* to 'sentry'@'%' identified by 'password';
- - User: Hive
create database metastore default character set utf8;
grant all on metastore.* to 'hive'@'%' identified by 'password';
- - User: Hue
create database hue default character set utf8;
grant all on hue.* to 'hue'@'%' identified by 'password';

Create Cloudera Services Databases & Users
- - User: Oozie
create database oozie default character set utf8;
grant all on oozie.* to 'oozie'@'%' identified by 'password';
- - User: Activity Monitor
create database amon default character set utf8;
grant all on amon.* to 'amon'@'%' identified by 'password';
- - User: Reports Manager
create database rman default character set utf8;
grant all on rman.* to 'rman'@'%' identified by 'password';
exit;
Create Cloudera DB
/usr/share/cmf/schema/scm_prepare_database.sh mysql --host cm515.localdomain --port 3306 scm scm;
Start Cloudera Server & Agent
sudo service cloudera-scm-server start;
sudo service cloudera-scm-agent start;

Configure Cloudera Manager
If you are running you VM in a Windows PC, you should add an entry on your host file that’s locate in
c:WindowsSystem32driversetchosts. After you can login in your browser an open the url http://cm515.localdomain:7180
The login details will be: Username: admin Password: admin
Configure CDH Hosts
Select Repository
Note 2: always chose Parcels.

JDK Installation (Optional)
Single User Mode
Note: On Single User Mode: don't select it, because If single user mode is not enabled, the Cloudera Manager Agent runs as root so that it can make
sure the required directories are created and that processes and files are owned by the appropriate user (for example, the hdfs and mapred users).

Login Credentials
Note: Choose root and select an appropriate password.
Installation Agents and Parcels (Wizard - Automatic)

Choose Services
Note: Choose only services that you need, more services requires more resources
Assign Roles
Note: Don’t assign the role Telemetry Publisher!

Configure Cluster Databases
Note: this is a mere example of cloudera services (there are more…)
Kudu Service Additional Configs (Opticional)
Note: you can chose a different directory, for example /u01/kudu/...

After Cluster Started
You will have some issues, therefore i’ll recommend a full restart to the Cluster.
Then you should have only two issues and both on HDFS service, such as:
Start & Add Services (rpc, nfs)
sudo service rpcbind start;
sudo service nfs start; sudo service nfs stop;
sudo chkconfig rpcbind on;
sudo chkconfig nfs on; Restart NFS Gateway Role
(CM - HDFS)

After Cluster Started
Fix HDFS Under-Replicated Blocks
sudo -u hdfs hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; sudo -u hdfs hadoop fs -setrep 3 $hdfsfile; done
Note 2: this is issue will appear on a VM (one node cluster) for test purpose you won’t need the replication by 3.

Secure the Cluster
There can be many ways to secure your Cluster, i’ve found that the best way in cost and benefits it’s with Kerberos, but keep
in mind that with future configurations such as Zeppelin, Jupyter and so on there i’ll be some “hurdles”.
Set Kerberos Realm
sudo yum install -y krb5-server openldap-clients krb5-workstation
Install Kerberos Server and Open Ldap Client
sudo sed -i.orig 's/EXAMPLE.COM/DOMAIN.COM/g' /etc/krb5.conf
Set the hostname for the kerberos server
sudo sed -i.m1 's/kerberos.example.com/cm515.localdomain/g' /etc/krb5.conf
Change Domain Name
sudo sed -i.m2 's/example.com/domain.com/g' /etc/krb5.conf
Create the kerberos database
sudo kdb5_util create -s
Note 1: enter a new password for kerberos database

Secure the Cluster
Update ACL File to enable */admin with admin privileges
sudo sed -i 's/EXAMPLE.COM/DOMAIN.COM/' /var/kerberos/krb5kdc/kadm5.acl;
Update the kdc.conf file to allow renewable
sudo sed -i.m3 '/supported_enctypes/a default_principal_flags = +renewable, +forwardable' /var/kerberos/krb5kdc/kdc.conf
Fix the indenting
sudo sed -i.m4 's/^default_principal_flags/ default_principal_flags/' /var/kerberos/krb5kdc/kdc.conf
Update kdc.conf file
sudo sed -i.orig 's/EXAMPLE.COM/DOMAIN.COM/g' /var/kerberos/krb5kdc/kdc.conf
Add a line to the file with ticket life
sudo sed -i.m1 '/dict_file/a max_life = 1d' /var/kerberos/krb5kdc/kdc.conf
Add a max renewable life
sudo sed -i.m1 '/dict_file/a max_life = 1d' /var/kerberos/krb5kdc/kdc.conf

Secure the Cluster
Indent the two new lines in the file
sudo sed -i.m3 's/^max_/ max_/' /var/kerberos/krb5kdc/kdc.conf
Start up the kdc server and the admin server
sudo service krb5kdc start;
sudo service kadmin start;
Create krbtgt principal
sudo kadmin.local <<eoj
modprinc -maxrenewlife 1week krbtgt/DOMAIN.COM@DOMAIN.COM
eoj
Add the admin user that CM will use to provision kerberos in the cluster
addprinc -pw password cloudera-scm/admin@DOMAIN.COM
modprinc -maxrenewlife 1week cloudera-scm/admin@DOMAIN.COM
eoj

Secure the Cluster
Add user to be super user for HDFS
addprinc -pw password hdfs@DOMAIN.COM
modprinc -maxrenewlife 1week hdfs@DOMAIN.COM
eoj
Test authentication as the CM admin user
kinit cloudera-scm/admin@DOMAIN.COM
Check characteristics of the ticket
klist -e
Note1: this info will be used later on Cloudera Manager
Note2: just need to change the highlighted values for your own values.
Make the kerberos services start on boot
sudo chkconfig kadmin on;
sudo chkconfig krb5kdc on;

Cloudera Manager - Kerberos Configuration
For start you should open your browser and login to Cloudera manager with the url: http://cm515.localdomain:7180/cmf/login
Enable Kerberos
Note 1:
check all boxes.

Setup KDC
Kerberos Encryption Types: aes256-cts-hmac-sha1-96
KDC Security Realm: cm515.localdomain
Kerberos Server Host: DOMAIN.COM
KDC Admin Server Host: cm515.localdomain
Domain Realm: domain.com
Maximum Renewable Life: 7
Configure krb5.conf
Note 1: check the manage krb5.conf box and forwardable tickets .

Setup KDC
Username: cloudera-scm/admin @ DOMAIN.COM
Password: password
Configure Kerberos Principals
Configure Ports

A successful output of a enabled Kerberos on Cloudera Manager should me something like this:

Additional Issues
In some cases it can appear some additional issues such as: Error on the Role Instance “Telemetry Publisher” and access
denied to the root user on mysql.
Error on Role Instance - Telemetry Publisher
Usually this role isn’t necessary, therefore you need to remove it (Delete)
To do so...you have to stop all Cloudera Management Services and then on Actions: Delete the role.

Additional Issues
Access denied for user 'root'@'localhost'
sudo nano /etc/my.cnf;
[mysqld]
skip-grant-tables
sudo service mysql restart;
mysql -u root -p
Note 1: just press enter - no password
FLUSH privileges;
ALTER USER 'root'@'localhost' IDENTIFIED BY 'welcome1';
exit;
mysql -u root -p
Note 2: retry again and enter the new chosen password
Remove Grant Flag
sudo nano /etc/my.cnf;
[mysqld]
#skip-grant-tables
sudo service mysql restart;

Thanks
Big Data Engineer
Tiago Simões

How to create a secured cloudera cluster

Related slideshows

More Related Content

How to create a secured cloudera cluster