Assessing and designing heterogeneous data management solutions for health-related research applications

Data management may be a complex challenge in fields such as bioinformatics and health sciences, which continuously generate extensive heterogeneous datasets. In the context of collaborative global health initiatives, secure storage and sharing of data are crucial to support impactful research. However, the absence of a unified data management platform complicates efficient data exchange and governance within these initiatives. In this paper, we introduce the design process of a data management prototype platform based on a data lakehouse architecture, data federation, and the FAIR principles. The platform is designed using open-source tools, guided by system requirements identified in previously published studies and complemented by insights from the existing literature. The current prototype platform comprises a user-friendly website, an open API, Python and R packages, allowing users to interact with the platform in multiple ways. Through a user study that included participants with varying technical backgrounds, we showed that our proposed data management prototype is both usable and useful. Our prototype design showcases the adaptability, scalability, and reproducibility of a lakehouse system that can be used by any organisation. It is designed as a flexible and complementary approach that allows organisations to customise data management systems to their specific requirements and resources, including cloud-based or self-hosted storage choices.

Grantor dc:publisher

Stellenbosch : Stellenbosch University

Year dc:date.issued

2026

Author dc:creator

• De Castro Silva, Danilo

Advisors dc:contributor.advisor

• Dunaiski, Marcel
• Moir, Monika
• Xavier, Joicymara

Language dc:language.iso

en

Repository record dc:identifier.uri

https://scholar.sun.ac.za/handle/10019.1/135725

OAI identifier oai:identifier

oai:scholar.sun.ac.za:10019.1/135725