Automated Care Pathway Modeling Using Agentic and Knowledge-Aware LLMs

Clinical pathways (CPWs) translate evidence-based guidance into stepwise care but are often disseminated as free text, obscuring control-flow semantics needed for clarity and computability. Formalizing CPWs as process models - e.g., in the Business Process Model and Notation (BPMN) - improves comprehensibility and enables downstream automation. This thesis designs, implements, and evaluates LLM4CPW, a pipeline for automatic guideline-to-BPMN modeling using large language models (LLMs). We compare two contemporary frameworks - MAO (agentic, multi-role orchestration) and ProMoAI (single-agent with self-refinement loop) - under controlled execution with standardized evaluation. Automated metrics combine node-level and structural similarity (after Dijkman et al.) with graph-edit distance; a clinician provides fidelity ratings with qualitative annotations. We then investigate knowledge-aware modeling via a categorization of recurrent errors and curated clinical statements, testing prompt-only injections versus a dedicated Knowledge Advisor agent placed at different stages of the workflow. Across four Ontario stroke Quality-Based Procedures (QBPs; n = 15 runs per framework), MAO attains higher node similarity (≈ 0.782 vs. ≈ 0.696) and structural similarity (≈ 0.630 vs. ≈ 0.585) than ProMoAI with large effects and p < 0.001, and exhibits markedly lower run-to-run variability; expert ratings also favor MAO. Knowledge-aware variants of MAO yield measurable gains: introducing a Knowledge Advisor after semantic review phase improves node similarity by > 4 points and reduces Graph Edit Distance by ∼ 13 (to∼ 97), with statistically comparable outcomes when placed before review; expert deltas likewise favor the advisor-based designs. Refining statement wording improves medians and interpretability without shifting means. Contributions. This thesis contributes (i) an auditable LLM4CPW pipeline and evaluation protocol; (ii) empirical evidence that agentic orchestration improves fidelity and stability; and (iii) a principled, deployable strategy for knowledge-enhanced modeling via a specialized advisory phase. Collectively, the findings demonstrate the feasibility of reliable, automatically extracted BPMN models from concise clinical guidelines and chart a path toward broader, clinically grounded automation.

Grantor dc:publisher

Université d'Ottawa / University of Ottawa

Year dc:date

2025

Author dc:creator

• Houshidari, Alireza

Contributors dc:contributor

• Van Woensel, William
• Amyot, Daniel

• LLMs
• Process Mining
• Knowledge-aware process mining
• Process extraction
• BPMN
• Clinical Pathways
• Clinical Guidelines

Statement dc:rights

• Attribution-NonCommercial 4.0 International

Licence

• http://creativecommons.org/licenses/by-nc/4.0/

Language dc:language

en

Identifier

https://doi.org/10.20381/ruor-31573

OAI identifier oai:identifier

oai:ruor.uottawa.ca:10393/51118