DATA MIGRATION FOR LARGE SCIENTIFIC DATASETS IN CLOUDS
Journal Title: Azerbaijan Journal of High Performance Computing - Year 2018, Vol 1, Issue 1
Abstract
Transferring large data files between various storages including cloud storages is an important task both for academic and commercial users. This should be done in an efficient and secure way. The paper describes Data Avenue that fulfills all these conditions. Data Avenue can efficiently transfer large files even in the range of TerraBytes among storages having very different access protocols (Amazon S3, OpenStack Swift, SFTP, SRM, iRODS, etc.). It can be used in personal, organizational and public deployment with all the security mechanisms required for these usage configurations. Data Avenue can be used by a GUI as well as by a REST API. The paper describes in detail all these features and usage modes of Data Avenue and also provides performance measurement results proving the efficiency of the tool that can be accessed and used via several public web pages.
Authors and Affiliations
Akos Hajnal, Eniko Nagy, Peter Kacsuk, Istvan Marton
Keywords
Related Articles
- EP ID EP525573
- DOI 10.32010/26166127.2018.1.1.66.86
- Views 73
- Downloads 0
CHALLENGES OF RESOURCE DISCOVERY TO SUPPORT DISTRIBUTED EXASCALE COMPUTING ENVIRONMENT
The resource discovery management unit (RD) in distributed Exascale systems needs to be able to manage the occurrence of dynamic and interactive events in the requesting process when running activities related to RD. The...
A SURVEY OF RESOURCE MANAGEMENT CHALLENGES IN MULTI-CLOUD ENVIRONMENT: TAXONOMY AND EMPIRICAL ANALYSIS
HOME ABOUT SUBMISSION SEARCH CURRENT ARCHIVES A SURVEY OF RESOURCE MANAGEMENT CHALLENGES IN MULTI-CLOUD ENVIRONMENT: TAXONOMY AND EMPIRICAL ANALYSIS Hits: 159 PDF Volume 1 (1), July 2018, Pages 51-65 Bandar Aldawsari1,...
CONTROLLING OPTIMIZATION SOFTWARE PACKAGES WITH THE APPLICATION OF PARALLEL COMPUTING
The paper is devoted to the analysis of techniques and algorithms of controlling computational process of solution to complex optimization problems with the use of multiprocessor and/or multicore computer systems. We hav...
CLOUD-BASED FLOWBSTER PORTAL TO DESIGN AND DEPLOY SCIENTIFIC WORKFLOWS
A workflow system called Flowbster has been designed to create efficient data pipelines in clouds. The entire Flowbster workflow is dynamically built by using virtual machines on a target cloud. The paper describes a rec...
CONVERGENCE OF HPC AND AI: TWO DIRECTIONS OF CONNECTION PDF
This paper examines the role of HPC systems in the solution of the most AI problems, on the other hand, assesses the impact of the application of AI methods on the resolution of different tasks in distributed systems. Th...