A Simple Measuring Model for Evaluating the Performance of Small Block Size Accesses in Lustre File System
Journal Title: Engineering, Technology & Applied Science Research - Year 2017, Vol 7, Issue 6
Abstract
Storage performance is one of the vital characteristics of a big data environment. Data throughput can be increased to some extent using storage virtualization and parallel data paths. Technology has enhanced the various SANs and storage topologies to be adaptable for diverse applications that improve end to end performance. In big data environments the mostly used file systems are HDFS (Hadoop Distributed File System) and Lustre. There are environments in which both HDFS and Lustre are connected, and the applications directly work on Lustre. In Lustre architecture with out-of-band storage virtualization system, the separation of data path from metadata path is acceptable (and even desirable) for large files since one MDT (Metadata Target) open RPC is typically a small fraction of the total number of read or write RPCs. This hurts small file performance significantly when there is only a single read or write RPC for the file data. Since applications require data for processing and considering in-situ architecture which brings data or metadata close to applications for processing, how the in-situ processing can be exploited in Lustre is the domain of this dissertation work. The earlier research exploited Lustre supporting in-situ processing when Hadoop/MapReduce is integrated with Lustre, but still, the scope of performance improvement existed in Lustre. The aim of the research is to check whether it is feasible and beneficial to move the small files to the MDT so that additional RPCs and I/O overhead can be eliminated, and read/write performance of Lustre file system can be improved.
Authors and Affiliations
N. Jayakumar, A. M. Kulkarni
Keywords
Related Articles
- EP ID EP155536
- DOI -
- Views 246
- Downloads 0
Modeling a Distributed Power Flow Controller with a PEM Fuel Cell for Power Quality Improvement
Electrical power demand is increasing at a relatively fast rate over the last years. Because of this increasing demand the power system is becoming very complex. Both electric utilities and end users of electric power ar...
Phi Shape UWB Antenna with Band Notch Characteristics
In this paper a novel band notch antenna in UWB frequency range is designed using split rings. Split rings are overlapped with designed monopole to give phi shape. The slit gap gives band-notch operation from 5.1GHz to 6...
Assessing Power System Stability Following Load Changes and Considering Uncertainty
An increase in load capacity during the operation of a power system usually causes voltage drop and leads to system instability, so it is necessary to monitor the effect of load changes. This article presents a method of...
Aspect Βased Classification Model for Social Reviews
Aspect based opinion mining investigates deeply, the emotions related to one’s aspects. Aspects and opinion word identification is the core task of aspect based opinion mining. In previous studies aspect based opinion mi...
Analysis of PM Magnetization Field Effects on the Unbalanced Magnetic Forces due to Rotor Eccentricity in BLDC Motors
This paper investigates both static and dynamic eccentricities in single phase brushless DC (BLDC) motors and analyzes the effect of the PM magnetization field on unbalanced magnetic forces acting on the rotor. Three com...