A primer on POMDPs

We wrote a primer on POMDPs! This was on my bucket list for some time now, but it seemed to complex or difficult at the time. Few years after my first attempt, here it is. I shared some of my experience modelling problems as POMDPs over the last 10 years to speed up the process and avoid some pitfalls. We also put together a repository of POMDP problems to help solving your first POMDP!

Thanks to my co-authors for making this paper a reality and the reviewers for providing precious insights on how we could clarify our ideas.

Chadès, I., Pascal, L. V., Nicol, S., Fletcher, C. S., & Ferrer-Mestres, J. (2021). A primer on partially observable Markov decision processes (POMDPs). Methods in Ecology and Evolution, 12, 2058– 2072. https://doi.org/10.1111/2041-210X.13692

Abstract:

1. Partially observable Markov decision processes (POMDPs) are a convenient mathematical model to solve sequential decision-making problems under imperfect observations. Most notably for ecologists, POMDPs have helped solve the trade-offs between investing in management or surveillance and, more recently, to optimise adaptive management problems.
2. Despite an increasing number of applications in ecology and natural resources, POMDPs are still poorly understood. The complexity of the mathematics, the inaccessibility of POMDP solvers developed by the Artificial Intelligence (AI) community, and the lack of introductory material are likely reasons for this.
3. We propose to bridge this gap by providing a primer on POMDPs, a typology of case studies drawn from the literature, and a repository of POMDP problems.
4. We explain the steps required to define a POMDP when the state of the system is imperfectly detected (state uncertainty) and when the dynamics of the system are unknown (model uncertainty). We provide input files and solutions to a selected number of problems, reflect on lessons learned applying these models over the last 10 years and discuss future research required on interpretable AI.
5. Partially observable Markov decision processes are powerful decision models that allow users to make decisions under imperfect observations over time. This primer will provide a much-needed entry point to ecologists.