A Simple Measuring Model for Evaluating the Performance of Small Block Size Accesses in Lustre File System
Journal Title: Engineering, Technology & Applied Science Research - Year 2017, Vol 7, Issue 6
Abstract
Storage performance is one of the vital characteristics of a big data environment. Data throughput can be increased to some extent using storage virtualization and parallel data paths. Technology has enhanced the various SANs and storage topologies to be adaptable for diverse applications that improve end to end performance. In big data environments the mostly used file systems are HDFS (Hadoop Distributed File System) and Lustre. There are environments in which both HDFS and Lustre are connected, and the applications directly work on Lustre. In Lustre architecture with out-of-band storage virtualization system, the separation of data path from metadata path is acceptable (and even desirable) for large files since one MDT (Metadata Target) open RPC is typically a small fraction of the total number of read or write RPCs. This hurts small file performance significantly when there is only a single read or write RPC for the file data. Since applications require data for processing and considering in-situ architecture which brings data or metadata close to applications for processing, how the in-situ processing can be exploited in Lustre is the domain of this dissertation work. The earlier research exploited Lustre supporting in-situ processing when Hadoop/MapReduce is integrated with Lustre, but still, the scope of performance improvement existed in Lustre. The aim of the research is to check whether it is feasible and beneficial to move the small files to the MDT so that additional RPCs and I/O overhead can be eliminated, and read/write performance of Lustre file system can be improved.
Authors and Affiliations
N. Jayakumar, A. M. Kulkarni
Keywords
Related Articles
- EP ID EP155536
- DOI -
- Views 214
- Downloads 0
Preventive Maintenance Interval Prediction: a Spare Parts Inventory Cost and Lost Earning Based Model
In this paper, some preventive maintenance parameters in manufacturing firms were identified and used to develop cost based functions in terms of machine preventive maintenance. The proposed cost based model considers sy...
Electrical Trees in a Composite Insulating System Consisted of Epoxy Resin and Mica: The Case of Multiple Mica Sheets For Machine Insulation
Epoxy resin and mica sheets consist the essential insulation of rotating machine stator bars. Such an insulation, although very resistant to partial discharges, is subjected to considerable electrical stresses and conseq...
Biodiesel Production Potential from Native Tehran Oil Crops Using GIS
One of the main factors in the alternative fuel economy is the primary raw materials. Importing raw materials and preparation conditions for cultivation of non-native species require high expenditure. Therefore, using na...
Mitigating the Adverse Impact of Un-Deterministic Distributed Generation on a Distribution System Considering Voltage Profile
Electric power systems are enforced to operate near to their stability limit due to the fast increase in power demand. Therefore, voltage stability has become a primary concern. The main cause of voltage variations is th...
Simulating the Various Subsystems of a Coal Mine
A set of simulation models of various subsystems of a coal mine was developed with the help of a new visual interactive simulation system of technological processes. This paper contains a brief description of this simula...