Securing Big Data in Hadoop

Instructor - Miguel Guirao

Pre-Requisites - In order to get out the most of this workshop, the student needs to be confortable working in the command line, moving around the filesyetem, editing files with vi or nano, visualizing and understanding processes and the top or htop command outputs. If you have been using the UNIX or UNIX-like command line for a time, you should be good and al set.

Abstract - Big Data have been for quiet a good time the driving force for innovation in many markets around. Data is the current asset that companies from around the world look for to crunch and extract information and knowledge, get new insights in order to create new services and products to deliver to their customers and finally improve their profits.

Because of that, it is imperative to protect such an important asset. In this workshop we will look at Haddoop from the point of view of security. We will learn what the Hadoop ecosystem has to offer us to protect our data, starting with Kerberos, perimeter security with Apache Knox, Configuring authorization with Apache Ranger and enabling encryption of the HDFS (Hadoop File System).

1) Kerberos is used in Hadoop to provide an authentication system for users and other system interacting with the Hadoop cluster and it's services. Strongly authenticating and establishing a user's identity is the basis for secure access in Hadoop. Users need to be able to reliably “identify” themselves and then have that identity propagated throughout the Hadoop cluster. Hadoop uses Kerberos as the basis for strong authentication and identity propagation for both user and services. More info: https://web.mit.edu/kerberos/

2) The Apache Knox Gateway (Knox) is a system to extend the reach of Apache Hadoop® services to users outside of a Hadoop cluster without reducing Hadoop Security. Knox also simplifies Hadoop security for users who access the cluster data and execute jobs. The Knox Gateway is designed as a reverse proxy. The Apache Knox Gateway is an Application Gateway for interacting with the REST APIs and UIs of Apache Hadoop deployments. The Knox Gateway provides a single access point for all REST and HTTP interactions with Apache Hadoop clusters. More info: https://knox.apache.org/

3) Apache Ranger is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The vision with Ranger is to provide comprehensive security across the Apache Hadoop ecosystem

Required Materials - Since this is NOT a class of how to setup a Hadoop cluster, but instead on how to secure a Hadoop cluster, it is a must that students taking this workshop come with the Hortonworks Data Platform (HDP) Docker image (https://hortonworks.com/products/sandbox/) already installed! The Docker image image is very big and it will take you a considerable time to download it during the workshop. Warning!! DO NOT download Hortonworks Data Flow (HDF), it is NOT THE SAME!
Please read the Intall Guide for the the type of the HDP and OS you will be using!
The workshop is prepared using Ubuntu Linux 18.04 and Docker!