Speed-up Extension to Hadoop System
Journal Title: INTERNATIONAL JOURNAL OF ENGINEERING TRENDS AND TECHNOLOGY - Year 2014, Vol 12, Issue 2
Abstract
For storage and analysis of online or streaming data which is too big in size most organization are moving toward using Apaches Hadoop- HDFS. Applications like log processors, search engines etc. using Hadoop Map Reduce for computing and HDFS for storage. Hadoop is most popular for analysis, storage and processing very large data but there need to be lots of changes in hadoop system. Here problem of data storage and data processing try to solve which helps hadoop system to improve processing speed and reduce time to execute the task. Hadoop application requires streaming access to data files. During placement of data files default placement of Hadoop does not consider any data characteristics. If the related set of files is stored in the same set of nodes, the efficiency and access latency can be increased. Hadoop uses Map Reduce framework for implementing large-scale distributed computing on unpredicted data sets. There are potential duplicate computations being performed in this process. No mechanism is to identify such duplicate computations which increase processing time. Solution for above problem is to co-locate related files by considering content and using locality sensitive hashing algorithm which is a clustering based algorithm will try to co -locate related file streams to the same set of nodes without affecting the default scalability and fault tolerance properties of Hadoop and for avoiding duplicate computation processing mechanism is developed which store executed task with result and before execution of any task stored executed tasks are compared if task find then direct result will be provided . By storing related files in same cluster which improve data locality mechanism and avoiding repeated execution of task improves processing time, both helps to speed up execution of Hadoop.
Authors and Affiliations
Sayali Ashok Shivarkar
Keywords
Related Articles
- EP ID EP142234
- DOI -
- Views 104
- Downloads 0
An Efficient and elastic approach for partial shape matching using DTW
We present the Partial shape matching for scale invariant and deformation tolerant 2D images. Scale invariance means a feature of objects that do not change if scale or length of objects changes. Deformation to...
A Review Paper on RF MEMS Switch for Wireless Communication
This paper deals with the RF (Radio Frequency)-MEMS (Micro-Electro-Mechanical-System) switch importance in the wireless communication system. Also explains the dominance of RF-Switch over existing devices like PIN...
Investigation of Natural Variants for Antimicrobial Finishes in Innerwear A Review Paper for Promotion of Natural Hygiene in Innerwear
This paper gives a review for scope of natural variants in the field of antimicrobial finishes. The main reason of increased interest in this field include: an increased awareness towards personal hygiene, demand f...
Design of Convolutional Codes for varying Constraint Lengths
Convolutional codes play a vital role in wireless communication with the increase in usage of low-latency applications operating at high data rates. As the current technologies demand high data rates, the de...
Performance, Combustion and Emission Characteristics of Corn oil blended with Diesel
Petroleum based fuels is a finite resource that is rapidly depleting. Consequently, petroleum reserves are not sufficient enough to last many years. Biodiesel is one of the alternative fuel made from vegetable oil,...