Object storage: how chunky would you like your data?

Brown, Matt ORCID: https://orcid.org/0000-0003-1127-0279; Chevuturi, Amulya ORCID: https://orcid.org/0000-0003-2815-7221; Fry, Matt ORCID: https://orcid.org/0000-0003-1142-4039. Object storage: how chunky would you like your data? [Poster] In: NERC Digital Gathering 2023, British Antarctic Survey, Cambridge, 10-11 July 2023. Wallingford, UK, UK Centre for Ecology & Hydrology. (Unpublished)

Before downloading, please read NORA policies.

Preview

Text
N535988PO.pdf
Download (377kB) | Preview

Official URL: https://digitalenvironment.org/events/dg23-digital...

Abstract/Summary

In this study we examine object storage, a cutting-edge cloud-native technology specifically designed for efficiently managing large datasets. While object storage offers significant cost-effectiveness compared to disk storage, it requires data to be appropriately adapted to fully realise its benefits. Data retrieval from object storage is over HTTP in complete "objects," which are either entire files or file chunks. As this is relatively new technology, there is a clear lack of established tools and best-practice for converting various file types for optimal use with object storage, particularly for large gridded and N-dimensional datasets used in environmental and climate science. The performance and speed of object storage are contingent upon the data's structure, chunking, and the specific analysis requirements of the user. Consequently, a better understanding of these interactions is essential before widespread adoption. To address this need, our study conducted a series of experiments using gridded data with different chunking strategies, aiming to identify the most efficient approach for utilizing and accessing data stored in an object store. Our findings highlight the need for comprehensive understanding of object storage before its widespread adoption, and serve as a valuable resource for guiding future users in utilizing object storage effectively.

Item Type:	Publication - Conference Item (Poster)
UKCEH and CEH Sections/Science Areas:	Hydro-climate Risks (Science Area 2017-) Water Resources (Science Area 2017-)
NORA Subject Terms:	Computer Science
Date made live:	21 Nov 2023 14:39 +0 (UTC)
URI:	https://nora.nerc.ac.uk/id/eprint/535988

Item Type:

Publication - Conference Item (Poster)

UKCEH and CEH Sections/Science Areas:

Hydro-climate Risks (Science Area 2017-)
Water Resources (Science Area 2017-)

NORA Subject Terms:

Computer Science

Date made live:

21 Nov 2023 14:39 +0 (UTC)

URI:

https://nora.nerc.ac.uk/id/eprint/535988