Next-generation sensor data acquisition onboard NERC deep water research vessels: a deliverable report for the AMPLIFY-EDS project

This report describes the development of the Data Acquisition and Metadata Platform (DAMP), a real time sensor logging system created by BAS and NOC for use on NERC’s deep-water research vessels. DAMP was designed to address limitations in the long standing Research Vessel Data Acquisition System (RVDAS) by adopting a modular architecture built on Kubernetes and Apache Kafka. Its containerised components (per sensor readers, parsers, file writers, and database sinks) operate independently and communicate through Kafka topics, enabling flexible scaling, dynamic reconfiguration, and improved resilience. The work aimed to explore architectural approaches that could inform a future NERC Environmental Data Service sensor commons. A prototype DAMP installation was deployed on the RRS Sir David Attenborough in October 2024, running alongside RVDAS for the 2024/25 Antarctic season. It successfully ingested high volume sensor streams, wrote raw and parsed data to disk and database, and mirrored all messages to a shore side replica with sub second latency. Following this successful trial, DAMP was integrated into BAS IT infrastructure and became the primary logging system ahead of the 2025/26 season. The project highlights lessons around Kubernetes networking, the importance of early IT engagement, and the growing need for “software patchworking” skills to integrate open source components into operational marine systems. Future work includes multi node Kubernetes deployments, replacing bespoke translators with Apache NiFi, adding real time stream processing, and preparing the codebase for potential open sourcing.