Decentralized partially observable Markov decision process

The decentralized partially observable Markov decision process (Dec-POMDP)^{[1] [2]} is a model for coordination and decision-making among multiple agents. It is a probabilistic model that can consider uncertainty in outcomes, sensors and communication (i.e., costly, delayed, noisy or nonexistent communication).

It is a generalization of a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) to consider multiple decentralized agents.^[3]

A Dec-POMDP is a 7-tuple ${\displaystyle (S,\{A_{i}\},T,R,\{\Omega _{i}\},O,\gamma )}${\displaystyle (S,\{A_{i}\},T,R,\{\Omega _{i}\},O,\gamma )}, where

• ${\displaystyle S}${\displaystyle S} is a set of states,
• ${\displaystyle A_{i}}${\displaystyle A_{i}} is a set of actions for agent ${\displaystyle i}${\displaystyle i}, with ${\displaystyle A=\times _{i}A_{i}}${\displaystyle A=\times _{i}A_{i}} is the set of joint actions,
• ${\displaystyle T}${\displaystyle T} is a set of conditional transition probabilities between states, ${\displaystyle T(s,a,s')=P(s'\mid s,a)}${\displaystyle T(s,a,s')=P(s'\mid s,a)},
• ${\displaystyle R:S\times A\to \mathbb {R} }${\displaystyle R:S\times A\to \mathbb {R} } is the reward function.
• ${\displaystyle \Omega _{i}}${\displaystyle \Omega _{i}} is a set of observations for agent ${\displaystyle i}${\displaystyle i}, with ${\displaystyle \Omega =\times _{i}\Omega _{i}}${\displaystyle \Omega =\times _{i}\Omega _{i}} is the set of joint observations,
• ${\displaystyle O}${\displaystyle O} is a set of conditional observation probabilities ${\displaystyle O(s',a,o)=P(o\mid s',a)}${\displaystyle O(s',a,o)=P(o\mid s',a)}, and
• ${\displaystyle \gamma \in [0,1]}${\displaystyle \gamma \in [0,1]} is the discount factor.

At each time step, each agent takes an action ${\displaystyle a_{i}\in A_{i}}${\displaystyle a_{i}\in A_{i}}, the state updates based on the transition function ${\displaystyle T(s,a,s')}${\displaystyle T(s,a,s')} (using the current state and the joint action), each agent observes an observation based on the observation function ${\displaystyle O(s',a,o)}${\displaystyle O(s',a,o)} (using the next state and the joint action) and a reward is generated for the whole team based on the reward function ${\displaystyle R(s,a)}${\displaystyle R(s,a)}. The goal is to maximize expected cumulative reward over a finite or infinite number of steps. These time steps repeat until some given horizon (called finite horizon) or forever (called infinite horizon). The discount factor ${\displaystyle \gamma }${\displaystyle \gamma } maintains a finite sum in the infinite-horizon case (${\displaystyle \gamma \in [0,1)}${\displaystyle \gamma \in [0,1)}).

• maspan.org
• The Dec-POMDP page

• Markov processes

• Articles with short description
• Short description matches Wikidata