Announcement: Mindasys is now Skillet! Check out our new website.

Key points about this course

Duration : 3 Days
Course Fee : RM4,199.00

HRD Corp Claimable Course

Managing Big Data in Clusters and Cloud Storage

Live Virtual Class

Public Class

In-House Training

Private Class

Course Overview

In this course, you'll learn how to manage big datasets, how to load them into clusters and cloud storage, and how to apply structure to the data so that you can run queries on it using distributed SQL engines like Apache Hive and Apache Impala. You’ll learn how to choose the right data types, storage systems, and file formats based on which tools you’ll use and what performance you need.

What you will learn

  • Use different tools to browse existing databases and tables in big data systems

  • Use different tools to explore files in distributed big data filesystems and cloud storage

  • Create and manage big data databases and tables using Apache Hive and Apache Impala

  • Describe and choose among different data types and file formats for big data systems

Course Prerequisites

To use the hands-on environment for this course, you need to download and install a virtual machine and the software on which to run it. Before continuing, be sure that you have access to a computer that meets the following hardware and software requirements:

• Windows, macOS, or Linux operating system (iPads and Android tablets will not work)

• 64-bit operating system (32-bit operating systems will not work)

• 8 GB RAM or more

• 25GB free disk space or more

• Intel VT-x or AMD-V virtualization support enabled (on Mac computers with Intel processors, this is always enabled; on Windows and Linux computers, you might need to enable it in the BIOS)

• For Windows XP computers only: You must have an unzip utility such as 7-Zip or WinZip installed (Windows XP’s built-in unzip utility will not work)

Course Objectives

By the end of the course, you will be able to

• use different tools to browse existing databases and tables in big data systems;

• use different tools to explore files in distributed big data filesystems and cloud storage;

• create and manage big data databases and tables using Apache Hive and Apache Impala; and

• describe and choose among different data types and file formats for big data systems.

Course Content

Module 1: Orientation to Data in Cluster and Cloud Storage

  • Browsing Tables with Hue
  • Browsing Tables with SQL Utility Statements
  • Browsing HDFS with the Hue File Browser
  • Browsing HDFS from the Command Line
  • Understanding S3 and Other Cloud Storage Platforms
  • Browsing S3 Buckets from the Command Line

Module 2: Defining Databases, Tables and Columns

  • Introduction to the CREATE TABLE Statement
  • Using Different Schemas on the Same Data
  • Specifying TBLPROPERTIES
  • Examining, Modifying, and Removing Tables
  • Hive and Impala Interoperability
  • Impala Metadata Refresh

Module 3: Data Types and Files Types

  • Overview of Data Types
  • Choosing the Right Data Types
  • Overview of File Types
  • Choosing the Right File Types

Module 4: Managing Datasets in Cluster and Cloud Storage

  • Refresh Impala's Metadata Cache after Loading Data
  • Loading Files into HDFS with Hue's Table Browser
  • Loading Files into HDFS with Hue's File Browser
  • Loading Files into HDFS from the Command Line
  • Loading Files into S3 from the Command Line
  • Using Hive and Impala to Load Data into Tables
  • Conclusion

Module 5: Optimizing Hive and Impala (Honors)

  • What to Do When Queries Are Too Complex
  • What to Do When Queries Take Too Long
  • When to Use Table Partitioning
  • When to Use Complex Columns
  • File Systems versus Storage Engines

  • Managing Big Data in Clusters and Cloud Storage

  • Ask For