Data-Driven sequential decision making with learning under ambiguity

Markov decision processes are often used to model sequential decision-making problems in uncertain dynamic environments, such as equipment maintenance and replacement problems, and inventory control problems. The objective of these problems is to find a policy or strategy, which is a prescription of which action to choose, to maximize a predetermined reward function. Determining an optimal policy for a Markov decision process model, first and foremost, requires the full knowledge of the reward and transition parameters, which are, however, fundamentally unknown in many applications. In conventional approach, these parameters are estimated from historical data, and treated as known quantities in the decision-making process. This approach ignores parameter uncertainty introduced by data inadequacy (e.g., limited availability of data, data contained with noise), making the robustness of decisions questionable. Two critical research questions arise in sequential decision making: 1) How can we incorporate the parameter uncertainty into the decision-making process to obtain more data-driven decisions? 2) What are the asymptotic behaviors of the decision-making process as more observations that reflect unknown parameters become available? This dissertation provides answers to these questions by developing novel data-driven sequential decision-making models in the presence of parameter uncertainty, and investigating the performance of the proposed decision-making models in maintenance and remanufacturing planning problems based on real-world cases. We first develop a novel distributionally robust Markov decision process model that incorporates a decision maker’s prior probabilistic information to model uncertain transition probabilities in a Bayesian framework. We first construct a set of prior distributions for unknown transition probabilities. With the evidence of historical data, the prior distributions in the ambiguity set are reevaluated using a likelihood-ratio test, and only admitted prior distributions are updated using the standard Bayesian updating rule. The objective of the proposed model is to maximize the expected total discounted reward over the worst posterior distribution. We investigate the asymptotic properties of the proposed multiple-priors model with a continuous support of prior distributions. We further develop an efficient approximation method to solve the resulting optimization model, and provide theoretical analysis on the performance of such an approximation. The utility of the proposed decision framework is demonstrated using a remanufacturing planning example. We then consider two sequential maintenance planning problems for both single- and multi-component systems. We first consider the problem of optimally maintaining a periodically inspected system with multi-level preventive maintenance whose effects are complex. We formulate the problem as an infinite-horizon Markov decision process with the objective of minimizing the expected total discounted inspection and maintenance cost. Sufficient conditions are established to ensure the existence of an optimal monotone control-limit type policy with respect to the system’s deterioration level and age. We further consider an integrated budget allocation and preventive maintenance optimization problem for multi-facility deteriorating transportation infrastructure systems, where preventive maintenance has multiple types and each maintenance type introduces complex maintenance effects. We formulate the problem as a sum of multiple Markov decision process models with budget constraints. A priority-based twostage method is developed to find optimal maintenance decisions for large-scale problems. Real-world deterioration data are used to demonstrate the effectiveness and efficiency of the proposed models and solution algorithms.

Name thesis:degree_name

Doctor of Philosophy

Level thesis:degree_level

Doctoral

Discipline thesis:degree_discipline

Industrial Engineering

Grantor

Texas Tech University

Year dc:date.issued

2022

Author dc:creator

• Shi, Yue

Chair dc:contributor.committeechair

• Xiang, Yisha

Committee members dc:contributor.committeemember

• Du, Dongping
• Matis, Timothy
• Coit, David W.

• Sequential Decision Making
• Data-Driven Method
• Bayesian Learning Method
• Maintenance Optimization

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/2346/90236

OAI identifier oai:identifier

oai:ttu-ir.tdl.org:2346/90236