A Simple Measuring Model for Evaluating the Performance of Small Block Size Accesses in Lustre File System
Journal Title: Engineering, Technology & Applied Science Research - Year 2017, Vol 7, Issue 6
Abstract
Storage performance is one of the vital characteristics of a big data environment. Data throughput can be increased to some extent using storage virtualization and parallel data paths. Technology has enhanced the various SANs and storage topologies to be adaptable for diverse applications that improve end to end performance. In big data environments the mostly used file systems are HDFS (Hadoop Distributed File System) and Lustre. There are environments in which both HDFS and Lustre are connected, and the applications directly work on Lustre. In Lustre architecture with out-of-band storage virtualization system, the separation of data path from metadata path is acceptable (and even desirable) for large files since one MDT (Metadata Target) open RPC is typically a small fraction of the total number of read or write RPCs. This hurts small file performance significantly when there is only a single read or write RPC for the file data. Since applications require data for processing and considering in-situ architecture which brings data or metadata close to applications for processing, how the in-situ processing can be exploited in Lustre is the domain of this dissertation work. The earlier research exploited Lustre supporting in-situ processing when Hadoop/MapReduce is integrated with Lustre, but still, the scope of performance improvement existed in Lustre. The aim of the research is to check whether it is feasible and beneficial to move the small files to the MDT so that additional RPCs and I/O overhead can be eliminated, and read/write performance of Lustre file system can be improved.
Authors and Affiliations
N. Jayakumar, A. M. Kulkarni
Keywords
Related Articles
- EP ID EP155536
- DOI -
- Views 220
- Downloads 0
Design of the Administrative Building of Kuhsar City under a Sustainable Architecture Concept Approach
This study investigates a sustainable approach in the design of the administrative building of the municipality and Islamic council of Kuhsar city, Iran. The initial design is inspired by a flower bud and the various sta...
Using Length of Bilge Keel to Length of Waterline Ratio to Reduce Ship Rolling Motion
Fishing vessels must have good stability and manoeuvrability. Hull with round bottom shape has a relatively poor rolling duration compared to other forms. Rolling duration reduction will improve the quality of stability...
Effect of Shear Flow on Crystallization of Sydiotactic Polypropylene/Clay Composites
The high sensitivity of crystallization to shear flow is a subject of great research interest the last several years. A set of syndiotactic polypropylene/clay composite samples were used to examine the effect of shear fl...
Modeling of Fatigue Crack Propagation in Aluminum Alloys Using an Energy Based Approach
Materials fatigue is a particularly serious and unsafe kind of material destruction. Investigations of the fatigue crack growth rate and fatigue life constitute very important and complex problems in mechanics. The under...
A Novel Hybrid Algorithm for Software Cost Estimation Based on Cuckoo Optimization and K-Nearest Neighbors Algorithms
The inherent uncertainty to factors such as technology and creativity in evolving software development is a major challenge for the management of software projects. To address these challenges the project manager, in add...