Memunic Chronicles in Atomic Arena

Hive is a data warehousing infrastructure based on the Hadoop. Hadoop provides massive scale out and fault tolerance capabilities for data storage and processing (using the map-reduce programming paradigm) on commodity hardware. Hive is designed to enable easy data summarization, ad-hoc querying and analysis of large volumes of data. It provides a simple query language called Hive QL, which is based on SQL and which enables users familiar with SQL to do ad-hoc querying, summarization and data analysis easily. At the same time, Hive QL also allows traditional map/reduce programmers to be able to plug in their custom mappers and reducers to do more sophisticated analysis that may not be supported by the built-in capabilities of the language. Hadoop is a batch processing system and Hadoop jobs tend to have high latency and incur substantial overheads in job submission and scheduling. As a result - latency for Hive queries is generally very high (minutes) even when data sets invol...

Apache Pig provides an engine for executing data flows in parallel on Hadoop. Pig's infrastructure layer consists of a compiler that produces sequences of MapReduce programs, for which large-scale parallel implementations already exist (e.g., the Hadoop subproject).  Pig's language layer currently consists of a textual language called Pig Latin, for expressing these data flows. Pig Latin includes operators for many of the traditional data operations (join, sort, filter, group etc.), as well as the ability for users to develop their own functions for reading, processing, and writing data.  Key Properties of Pig Latin because of which it is so popular esp on Hadoop ecosystem: Ease of programming  Optimization opportunities Extensibility This post will talk about installing Pig on top of Your hadoop cluster(You can also run it in local mode) and running a pig script. The pre requisite for this post is that You already have Your hadoop cluster set up and Yo...