Hadoop Administration training course

Introduction to Data Storage and Processing

• Installing the Hadoop Distributed File System (HDFS)
•     Defining key design assumptions and architecture
•     Configuring and setting up the file system
•     Issuing commands from the console
•     Reading and writing files
• Setting the stage for MapReduce
•     Reviewing the MapReduce approach
•     Introducing the computing daemons
•     Dissecting a MapReduce job

Defining Hadoop Cluster Requirements

• Planning the architecture
•     Selecting appropriate hardware
•     Designing a scalable cluster
• Building the cluster
•     Installing Hadoop daemons
•     Optimising the network architecture

Configuring a Cluster

• Preparing HDFS
•     Setting basic configuration parameters
•     Configuring block allocation, redundancy and replication
• Deploying MapReduce
•     Installing and setting up the MapReduce environment
•     Delivering redundant load balancing via Rack Awareness

Maximising HDFS Robustness

• Creating a fault–tolerant file system
•     Isolating single points of failure
•     Maintaining High Availability
•     Triggering manual failover
•     Automating failover with Zookeeper
• Leveraging NameNode Federation
•     Extending HDFS resources
•     Managing the namespace volumes
• Introducing YARN
•     Critiquing the YARN architecture
•     Identifying the new daemons

Managing Resources and Cluster Health

• Allocating resources
•     Setting quotas to constrain HDFS utilisation
•     Prioritising access to MapReduce using schedulers
• Maintaining HDFS
•     Starting and stopping Hadoop daemons
•     Monitoring HDFS status
•     Adding and removing data nodes
• Administering MapReduce
•     Managing MapReduce jobs
•     Tracking progress with monitoring tools
•     Commissioning and decommissioning compute nodes

Maintaining a Cluster

• Employing the standard built–in tools
•     Managing and debugging processes using JVM metrics
•     Performing Hadoop status checks
• Tuning with supplementary tools
•     Assessing performance with Ganglia
•     Benchmarking to ensure continued performance

Extending Hadoop

• Simplifying information access
•     Enabling SQL–like querying with Hive
•     Installing Pig to create MapReduce jobs
• Integrating additional elements of the ecosystem
•     Imposing a tabular view on HDFS with HBase
•     Configuring Oozie to schedule workflows

Implementing Data Ingress and Egress

• Facilitating generic input/output
•     Moving bulk data into and out of Hadoop
•     Transmitting HDFS data over HTTP with WebHDFS
• Acquiring application–specific data
•     Collecting multi–sourced log files with Flume
•     Importing and exporting relational information with Sqoop
• Planning for Backup, Recovery and Security
•     Coping with inevitable hardware failures
•     Securing your Hadoop cluster