Automated Graph-Based Data Integration and Information Fusion: An Open-Source Ecosystem of Tools With a Focus on Maintainability and Usability

Information is a vital part of research. While there are likely exceptions, almost all research projects process or produce data, or both. Storage of information in digital form provides many advantages, such as large capacities or easy and immediate access across locations. However, new challenges arise as well necessitating specialized software and paradigms for efficient data storage, integration, and analysis.<br /> The ever-growing number of available online databases is a vast source of information for research projects. Longevity and maintenance of these databases is, however, a non-trivial task and finding relevant databases for a project has become difficult. Meta-registries provide curated information on available databases and allow searching for specific keywords, topics, and more.<br /> Data integration tools and workflows are available providing researchers with the means to combine and interconnect heterogeneous source databases. Each tool uses specific database systems and integration methodologies but also introduces certain constraints.<br /> This thesis evaluates existing meta-registries for life science databases and identifies the requirements to capture reliable metadata on database longevity. As a result, two novel meta-registries, miRNAverse and NAR DB Status, are presented. Both focus on providing up-to-date manual metadata annotations regarding availability and longevity of databases. Analyses of both meta-registries show a database availability between 62% and 66%.<br /> Moreover, this thesis compares existing data integration tools and identifies functional and non-functional requirements for such tools. A novel concept is formulated covering the entire data integration and mapping process from automatic database version retrieval to an analysis-ready data warehouse. To demonstrate the feasibility of the concept, the BioDWH2 ecosystem of tools is implemented. A significant number of data source modules are already available and extensible using a modular approach. Data source versions are automatically retrieved and downloaded if necessary. Furthermore, the entire mapping process is developed to be transparent and analyzable. The novel MappingSpy tool allows the visual inspection of mappings and calculates scores to hint at potential issues. BioDWH2 is open-source and freely available at https://github.com/BioDWH2/.

Level thesis:degree_level

thesis.doctoral

Grantor dc:publisher

Universität Bielefeld

Year

2025

Author dc:creator

• Friedrichs, Marcel

Repository record source_url

https://pub.uni-bielefeld.de/record/3000735

OAI identifier oai:identifier

oai:pub.uni-bielefeld.de:3000735