• Definitions
  • Markov Decision Process
  • Rewards
  • Policies
    • Value functions
• Policy evaluation
  • (Q)-function
• Value Function Policies
  • Advantage function
• Policy Iteration
• Value iteration
• Bonus: Linear Program (LP) Formulation
  • Linear Systems with Quadratic Reward
• Footnotes

Definitions

Markov Decision Process

A Markov Decision Process (MDP) models choosing an action $a_t$ in an action space $\mathcal{A}$ while in a state $s_t$ (from a state space $\mathcal{S}$) and thence receiving a reward $r_t$. Choosing such an action evolves the process to a state $s_{t+1}$ according to a probability distribution $P(S_{t+1}\mid S_t,A_t)$ over states and actions¹. The Markov assumption assumes that $S_{t+1}$ only depends on $S_t$ and $A_t$ (rather than the entire history/trajectory of the process). Stationary MDPs are those for which $P(S_{t+1}\mid S_t,A_t)$ does not vary in time (i.e. only depends on states and actions and not at which $t$ those states and actions were occupied/chosen). Therefore, the state transition model $T(s^{\prime }\mid s,a)$ represents that probability of transitioning from state $s$ to state $s^{\prime }$ after choosing action $a$ and the reward function $R(s,a)$ represents the expected reward when executing action $a$ from state $s$.

Rewards

$R(s,a)$ is a deterministic function of $s,a$ because it is an expectation but it may be “generated” stochastically; for example, if the reward depends on the next state (as given by a modified $R(s,a,s^{\prime })$) then

In a finite horizon problem with $n$ decisions the utility associated (also called the reward) with a sequence of rewards $r_{1:n}:=r_1,r_2,\dots ,r_n$ is

In an infinite horizon the discounted return is given by

where $0\le \gamma <1$ guarantees that the sum converges. Note that the “first” return $r_1$ is “undiscounted” (${\gamma }^{1-1}=1$).

Policies

Given a history of states and actions $h_t:=(s_{1:t},a_{1:t})$ a policy ${\pi }_t(h_t)$ sets which actions are taken at each time step. Since for MDPs future states and rewards depend only on the current state, we restrict our attention to policies that only depend on the current state, i.e. ${\pi }_t(h_t)\equiv {\pi }_t(s_t)$. A deterministic policy is one where the action taken is deterministically a function of the history. Alternatively, a stochastic policy is one where the action taken is actually drawn from a probability distribution ${\pi }_t(a_t\mid s_t)$. For infinite horizon problems with stationary transitions and rewards stationary policies, naturally, do not depend on time i.e.

Note that for finite horizon problems (even with stationary transitions and rewards) it maybe advantageous to consider non-stationary policies.

Value functions

The expected utility of executing policy $\pi$ from state $s$ is denoted ${\mathcal{U}}^{\pi }(s)$. For MDPs ${\mathcal{U}}^{\pi }$ is called the value function (because it captures the value of being in a state, given some policy for actions to take). An optimal policy ${\pi }^{\ast }$ is a policy that maximizes expected utility

The value function ${\mathcal{U}}^{\ast }(s):={\mathcal{U}}^{{\pi }^{\ast }}(s)$ corresponding to the optimal policy is called the optimal value function. An optimal policy can be found using dynamic programming².

Policy evaluation

Policy evaluation involves estimating the value function ${\mathcal{U}}^{\pi }$ forward in time. This can be done inductively (iteratively); for a single step from the state $s$ the value function is just the reward received when performing action $\pi (s)$

Then ${\mathcal{U}}_{k+1}^{\pi }(s)$ at some time step $k+1$ is estimated using the lookahead equation; the lookahead equation estimates ${\mathcal{U}}_{k+1}^{\pi }(s)$ by combining

1. the reward received by taking action $\pi (s)$
2. the average value of the $k$th (current) estimate of the policy, ${\mathcal{U}}_k^{\pi }(s^{\prime })$, over all possible states $s^{\prime }$ that can be transitioned to when starting in state $s$ and taking action $\pi (s)$

That is to say

Note that since $0\le \gamma <1$ it’s the case that ${\mathcal{U}}_k^{\pi }\to {\mathcal{U}}_{k+1}^{\pi }$ is a contraction mapping³ and therefore policy evaluation converges

and therefore for an infinite horizon problem, at convergence, we have

${\mathcal{U}}^{\pi }(s)$ can also be solved for by solving a system of equations

where ${\mathbf{U}}^{\pi },{\mathbf{R}}^{\pi }$ are $|\mathcal{S}|$-length vectors and ${\mathbf{T}}^{\pi }$ is a $|\mathcal{S}|\times |\mathcal{S}|$ matrix with state $i$ $\to$ state $j$ transition probabilities $T_{ij}^{\pi }$. Solving for ${\mathbf{U}}^{\pi }$ we get

which has runtime $O(|\mathcal{S}{|}^3)$.

$Q$-function

An alternative way to represent an arbitrary value function $\mathcal{U}(s^{\prime })$ is in terms of the action as well as the state, namely by the action value function (also called the $Q$-function)

Storing $Q$ costs $O(|\mathcal{S}|\times |\mathcal{A}|)$ as opposed to just $O(|\mathcal{S}|)$ for storing $\mathcal{U}$ but it saves having to store and query $R,T$.

Value Function Policies

The previous section computed a value function given a policy. Similarly, given a value function $\mathcal{U}(s)$, we can compute the policy that maximizes that value function

called the greedy policy. If $\mathcal{U}(s)$ is the optimal value function then $\pi (s)$ is the optimal policy.

Both $\mathcal{U}(s)$ and $\pi (s)$ can be obtained from the $Q$-function

Advantage function

Policies can also be represented using the advantage function, which quantifies the “advantage” of taking some action relative to the greedy action (action dictated by the greedy policy $\mathcal{U}(s)$)

Note that greedy actions have “zero advantage” and non-greedy actions have “negative advantage”.

Policy Iteration

Policy iteration is an algorithm for computing an optimal policy. It starts with any policy (e.g. $\pi (s)=0$) and alternates between policy evaluation and policy improvement (using the lookahead function).

def lookahead$(\mathcal{S},T,R,\gamma ,\mathcal{U},s,a)$:
    return $R(s,a)+\gamma \sum _{s^{\prime }\in \mathcal{S}}T(s^{\prime }\mid s,a)\mathcal{U}(s^{\prime })$

def greedy_action$(\mathcal{S},T,R,\mathcal{A},\gamma ,\mathcal{U},s)$:
    return ${\mathrm{argmax}}_{a^{\prime }\in \mathcal{A}}$ lookahead$(\mathcal{S},T,R,\gamma ,\mathcal{U},s,a^{\prime })$

def policy_iteration$(\mathcal{S},\pi ,\mathcal{A},T,R,\gamma ,$k_max=100$)$:
    # vectorize by evaluating args
    for k = 1:k_max
        # perform policy evaluation
        $\mathbf{T}$ = $[T(s^{\prime }\mid s,\pi (s))$ for $s\in \mathcal{S}$ for $s^{\prime }\in \mathcal{S}]$
        $\mathbf{R}$ = $[R(s,\pi (s))$ for $s\in \mathcal{S}]$
        $\mathbf{U}$ = ${(\mathbf{I}-\gamma \mathbf{T})}^{-1}\mathbf{R}$

        # check convergence
        # lambdas 
        $\mathcal{U}$ = $s$ -> $\mathcal{U}(s)$
        ${\pi }^{\prime }$ = $s$ -> greedy_action$(\mathcal{S},T,R,\mathcal{A},\gamma ,\mathcal{U},s)$
        if all$(\pi (s)$ == ${\pi }^{\prime }(s)$ for $s\in \mathcal{S})$:
            break
        $\pi$ = ${\pi }^{\prime }$
    return $\pi$

Value iteration

Value iteration is an alternative to policy iteration that updates the value function directly (instead of updating it through the policy). It starts with any bounded $|\mathcal{U}(s)|<\mathrm{\infty }$, e.g. $\mathcal{U}(s)=0$, and improves it using the Bellman equation

This is called backup.

def backup$(R,\mathcal{A},\mathcal{S},\mathcal{U})$:
    return $\underset{a\in \mathcal{A}}{max}(R(s,a)+\gamma \sum _{s^{\prime }\in \mathcal{S}}T(s^{\prime }\mid s,a)\mathcal{U}(s^{\prime }))$

As already mentioned this is guaranteed to converge and the optimal policy is guaranteed to satisfy

That is to say ${\mathcal{U}}^{\ast }(s)$ is a fixed-point of the Bellman equation.

def value_iteration$(R,T,\mathcal{A},\mathcal{S},\gamma ,$k_max=100$)$:
    ${\mathcal{U}}_1$ = 0
    for k = 1:k_max
        ${\mathcal{U}}_{k+1}$ = $[$backup$(R,\mathcal{A},\mathcal{S},{\mathcal{U}}_k)$ for $s\in \mathcal{S}]$
    $\pi$ = $s$ -> greedy_action$(\mathcal{S},T,R,\mathcal{A},\gamma ,{\mathcal{U}}_{\mathtt{k}\mathtt{\_}\mathtt{m}\mathtt{a}\mathtt{x}\mathtt{+}\mathtt{1}},s)$
    return $\pi$

Note that we could also terminate value iteration conditional on $\Vert {\mathcal{U}}_{k+1}-{\mathcal{U}}_k{\Vert }_{\mathrm{\infty }}$⁴, called the Bellman residual. A Bellman residual of $\delta$ guarantees that the current iteration is within $\epsilon =\delta \gamma /(1-\gamma )$ of the optimal value function ${\mathcal{U}}^{\ast }$. Similarly the policy loss $\Vert {\mathcal{U}}^{\pi }-{\mathcal{U}}^{\ast }{\Vert }_{\mathrm{\infty }}$ bounds the deviation of the total reward obtained under the policy extracted from value iteration $\pi$; policy loss is bounded by $2\epsilon$.

Bonus: Linear Program (LP) Formulation

Finding the optimal policy can be reformulated as a linear program. Begin by replacing the equality in equation $\text{(17)}$ with a set of inequality constraints and a minimization object

Intuitively the minimization objective induces/encourages equality. The $\underset{a}{max}$ is a nonlinear constraint; it can be replaced by a set of linear constraints

This is therefore a linear program because both the objective and constraints are linear in the variables $\mathcal{U}(s)$.

Linear Systems with Quadratic Reward

The Bellman equation⁵ for continuous, vector-valued states and actions is

Note that here we’re assuming finite horizon and no discounting. In general this is difficult to solve. On the contrary, if a problem has linear dynamics and a quadratic reward then the optimal policy can be computed in closed form; a problem has linear dynamics if

where

are matrices that represent the mean of the next state ${\mathbf{s}}^{\prime }$ given $\mathbf{s}$ and $\mathbf{a}$ respectively, and $\mathbf{w}$ is zero mean, finite variance noise that is independent of both $\mathbf{s},\mathbf{a}$ (e.g. multivariate Gaussian). A reward function is quadratic if it can be written

where ${\mathbf{R}}_{\mathbf{s}},{\mathbf{R}}_{\mathbf{a}}$ are matrices that determine the contributions of state and action to reward. Further more ${\mathbf{R}}_{\mathbf{s}},{\mathbf{R}}_{\mathbf{a}}$ should be negative semi-definite⁶ and negative definite, respectively; such a reward penalizes states and actions different from $\mathbf{0}$. In control theory such a system is called a linear quadratic regulator (LQR).

Substituting eqns. $\text{(21)}$ and $\text{(23)}$ into the Bellman equation

The optimal one-step lookahead function is

for which the optimal action is $\mathbf{a}=0$ (since ${\mathbf{R}}_{\mathbf{a}}$ is negative semi-definite). We now prove that ${\mathcal{U}}_h(\mathbf{s})={\mathbf{s}}^{⊺}{\mathbf{V}}_h\mathbf{s}+q_h$ for a symmetric ${\mathbf{V}}_h$, with ${\mathbf{V}}_1={\mathbf{R}}_{\mathbf{s}}$ and $q_1=0$. Making this substitution into the right-hand side of eqn. $\text{(24)}$

Using $\int p(\mathbf{w})\text{d}\mathbf{w}=1$ and $\int \mathbf{w}p(\mathbf{w})\text{d}\mathbf{w}=0$ (zero mean) to simplify terms with a $q_h$ factor

Using

differentiating wrt $\mathbf{a}$ and setting equal to $\mathbf{0}$

Using the fact that ${\mathbf{R}}_{\mathbf{a}}+{\mathbf{T}}_{\mathbf{a}}^{⊺}{\mathbf{V}}_h{\mathbf{T}}_{\mathbf{a}}$ is negative definite and therefore invertible we obtain the optimal action

and substituting into ${\mathcal{U}}_{h+1}(\mathbf{s})$ produces ${\mathcal{U}}_{h+1}(\mathbf{s})={\mathbf{s}}^{⊺}{\mathbf{V}}_{h+1}\mathbf{s}+q_{h+1}$ with

and

If $\mathbf{w}\sim \mathcal{N}(\mathbf{0},\mathbf{\Sigma })$

Equation $\text{(31)}$ is called the discrete-time Ricatti equation. Note that the optimal action is independent of $\mathbf{w}$ but the variance of $\mathbf{w}$ does affect the expected utility through $q_{h+1}$.

Footnotes