# Exercise 3: Dynamic Programming

## 1) Policy Evaluation

After your master thesis you decide to study on and do you beer-bachelor.
Therefore, you have to drink three beers in three different pubs.
There are six pubs available in town, you start at home and will (hopefully) end up at home. The problem is depicted in the following picture:

In our first example we follow the 50/50 policy.
So after drinking in a pub - e.g. Auld Triangle, there is a $50 \, \%$ probability to go "up" to the Globetrotter and  $50\, \%$ probability to go "down" to the Black Sheep.
Evaluate the state values using policy evaluation ($v_\mathcal{X} = \mathcal{R}_\mathcal{X} + \gamma \mathcal{P}_{xx'} v_\mathcal{X}$):

\begin{align*}
\begin{bmatrix}
v^{50/50}_{1}\\
.\\
.\\
.\\
v^{50/50}_{n}\\
\end{bmatrix}
=
\begin{bmatrix}
\mathcal{R}^{50/50}_{1}\\
.\\
.\\
.\\
\mathcal{R}^{50/50}_{n}\\
\end{bmatrix}
+
\gamma
\begin{bmatrix}
{p}^{50/50}_{11}&...&{p}^{50/50}_{1n}\\
.& &.\\
.& &.\\
.& &.\\
{p}^{50/50}_{n1}&...&{p}^{50/50}_{nn}\\
\end{bmatrix}
\begin{bmatrix}
v^{50/50}_{1}\\
.\\
.\\
.\\
v^{50/50}_{n}\\
\end{bmatrix}
\end{align*}

The rewards are given as negative numbers next to the arrows and represent the distances between two bars as a penalty.
In this exercise we will set $\gamma = 0.9$.
In the shown problem we have $n = 8$ states (pubs, including start-home and end-home), ordered as given by the state space:

\begin{align*}
\mathcal{X} =
\left\lbrace \begin{matrix}
\text{Start: Home}\\
\text{Auld Triangle}\\
\text{Lötlampe}\\
\text{Globetrotter}\\
\text{Black Sheep}\\
\text{Limericks}\\
\text{Fat Louis}\\
\text{End: Home}\\
\end{matrix}
\right\rbrace
\end{align*}

Use a little python script to calculate the state values!

## 1) Solution

The state transition matrix is given by:

$\mathcal{P}_{xx'} =
\begin{bmatrix}
0 & 0.5 & 0.5 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 0.5 & 0.5 & 0 & 0 & 0\\
0 & 0 & 0 & 0.5 & 0.5 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & 0 & 0.5 & 0.5 & 0\\
0 & 0 & 0 & 0 & 0 & 0.5 & 0.5 & 0\\
0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\
0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\
0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\\
\end{bmatrix}
$

By matrix inversion we can calculate the state values for the given policy like already learned in exercise 2 by:
\begin{align}
v_\mathcal{X} &= \mathcal{R}_\mathcal{X} + \gamma \mathcal{P}_{xx'} v_\mathcal{X}\\
v_\mathcal{X} - \gamma \mathcal{P}_{xx'} v_\mathcal{X} &= \mathcal{R}_\mathcal{X}\\
\left( I_6 - \gamma \mathcal{P}_{xx'} \right) v_\mathcal{X} &= \mathcal{R}_\mathcal{X}\\
v_\mathcal{X} &= \left( I_6 - \gamma \mathcal{P}_{xx'} \right)^{-1} \mathcal{R}_\mathcal{X}\\
\end{align}

First, we have to calculate the expected reward using the 50/50 policy.
Concerning the home state, it is 50 % likely to receive a negative reward of -3 on the way to the Auld Triangle and 50 % likely to receive a negative reward of -1 on the way to Lötlampe.
So the expected reward is given by: $\mathcal{R}_1 = 0.5 \cdot (-3) + 0.5 \cdot (-1) = -2$

Following this concept, the expected reward vector is given by:

$\mathcal{R} =
\begin{bmatrix}
-2\\
-3\\
-4\\
-4.5\\
-5.5\\
-6\\
-7\\
0
\end{bmatrix}
$

Making use of this knowledge, we can calculate the state values as follows:

```python
import numpy as np

# define given parameters
gamma = 0.9 # discount factor

### BEGIN SOLUTION

P_xx = np.array([[0, 0.5, 0.5,   0,   0,   0,   0,   0],
                 [0,   0,   0, 0.5, 0.5,   0,   0,   0],
                 [0,   0,   0, 0.5, 0.5,   0,   0,   0],
                 [0,   0,   0,   0,   0, 0.5, 0.5,   0],
                 [0,   0,   0,   0,   0, 0.5, 0.5,   0],
                 [0,   0,   0,   0,   0,   0,   0,   1],
                 [0,   0,   0,   0,   0,   0,   0,   1],
                 [0,   0,   0,   0,   0,   0,   0,   1]]) # state trasition probability

r_X = np.array([-2, -3, -4, -4.5, -5.5, -6, -7, 0]) # rewards
r_X = r_X.reshape(-1, 1) # make column vector

v_X = np.matmul(np.linalg.inv(np.eye(8)-gamma*P_xx) , r_X)

### END SOLUTION

print(v_X)

```

    [[-13.9385]
     [-12.765 ]
     [-13.765 ]
     [-10.35  ]
     [-11.35  ]
     [ -6.    ]
     [ -7.    ]
     [  0.    ]]

## 2) Exhaustive Policy Search

From now on use $\gamma = 1$.

As you have pre knowledge from your master degree, you try to minimize the distance of the way you have to take during your tour in order to have more time in the pubs. Therefore, you perform the following exhaustive search algorithm:

1. Write down all possible path-permutations and calculate the distances.
2. Which is the best path concerning most beer per distance?
3. Derive the formula to calculate the number of necessary path comparisons.

## 2) Solution

Auld Triangle, Globetrotter, Limericks = -15

Auld Triangle, Globetrotter, Fat Louis = -17

Auld Triangle, Black Sheep, Limericks = -18

Auld Triangle, Black Sheep, Fat Louis = -20

Lötlampe, Black Sheep, Fat Louis = -19

Lötlampe, Black Sheep, Limericks = -17

Lötlampe, Globetrotter, Limericks = -14

Lötlampe, Globetrotter, Fat Louis = -16

Choice of two, three times. Order of chosen actions $\{up, down\}$ is important and number of choices stays the same. The number of different paths is therefore given by: $N^k = 2^3 = 8$. So the number of necessary path comparisons is $N^k -1 = 2^3 -1= 7$.

With number of decisions $k$ and possible options per decision $N$.

## 3) Dynamic Programming - The Idea

Trying out all combinations might not be best for your liver, so you want to solve the problem above using dynamic programming.

Making use of value iteration, derive the values resulting from the optimal policy: $v_{i+1}^*(x_k) = \text{max}_u (r_{k+1} + v_{i}^*(x_{k+1}))$.

Hint: There is only one policy improvement step needed.

How many value comparisons have to be made?

## 3) Solution

In this situation, we are looking for the state with maximum value denoted by the asterisk $^*$.
At first we initialize all state values with zero and update from start to end using the equation above.

For the first iteration $i=0$, all state values are zero, so for $i=1$, the best immediate reward is taken as new state value due to $v_{i=1}^*(x_k) = \text{max}_u (r_{k+1} + \underbrace{v_{i=0}^*(x_{k+1}))}_{=0}$.

For $i=2$ we have to consider that the state value $v_{i=1}^*(x_{k+1})$ is not zero anymore.

For example for the "Start" state we have the options:
1. To A: $v_{i=2}(Start) = -3 -2 = -5$
2. To Lö:$v_{i=2}(Start) = -1 -3 = -4$

As we are looking for the optimal policy (remember $^*$) we take the better value (to Lö).

For the state A we have the options:
1. To G: $v_{i=2}(A) = -2 -4 = -6$
2. To B: $v_{i=2}(A) = -4 -5 = -9$

...

i \ x   | Start  |   A    |  Lö    |   G    |   B    |  Li    |   F    |  End
-------- | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------
    0    |   0    |   0    |   0    |   0    |   0    |   0    |   0    |   0
    1    |   -1   |   -2   |   -3   |   -4   |   -5   |   -6   |   -7   |   0
    2    |   -4   |   -6   |   -7  |   -10   |   -11   |   -6   |   -7   |   0
    3    |   ...   |

This would take a lot of time and it would be better to write a little programm.
But if we change the order of state updates in a clever way, like explained in the lecture, it needs only one iteration to find the result!

Since we know the model, we are able to begin the evaluation in the end state and can derive the final state values directly:

If in Limericks or Fat Louis, there is no choice of way, so the value of these locations is directly given.

\begin{align}
v(\text{Limericks})&=-6\\
v(\text{Fat Louis})&=-7
\end{align}

If in Globetrotter or Black Sheep, the choice where to go is determined by the distance and the future value:

\begin{align}
v(\text{Globetrotter})&=-4+v(\text{Limericks})=-10 \hspace{0.5cm} \Leftarrow \text{optimal choice}\\
v(\text{Globetrotter})&=-5+v(\text{Fat Louis})=-12
\end{align}

\begin{align}
v(\text{Black Sheep})&=-5+v(\text{Limericks})=-11 \hspace{0.5cm} \Leftarrow \text{optimal choice}\\
v(\text{Black Sheep})&=-6+v(\text{Fat Louis})=-13
\end{align}

And go on like this:

\begin{align}
v(\text{Auld Triangle})&=-2+v(\text{Globetrotter})=-12 \hspace{0.5cm} \Leftarrow \text{optimal choice}\\
v(\text{Auld Triangle})&=-4+v(\text{Black Sheep})=-15
\end{align}

\begin{align}
v(\text{Lötlampe})&=-3+v(\text{Globetrotter})=-13 \hspace{0.5cm} \Leftarrow \text{optimal choice}\\
v(\text{Lötlampe})&=-5+v(\text{Black Sheep})=-16
\end{align}

Finally, we decide where to start:

\begin{align}
v(\text{Home})&=-3+v(\text{Auld Triangle})=-15 \\
v(\text{Home})&=-1+v(\text{Lötlampe})=-14 \hspace{0.5cm} \Leftarrow \text{optimal choice}
\end{align}

i \ x      | Start  |   A    |  Lö    |   G    |   B    |  Li    |   F    |  End
-------- | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------
    0    |   0    |   0    |   0    |   0    |   0    |   0    |   0    |   0
    1    |   -14  |  -12   |   -13  |   -10  |   -11  |   -6   |   -7   |   0

So we have to calculate the value of 5 states (4 pubs and 1 starting node) for both of the possible choices to be made. The 2 states in direct reach of the terminal node do not allow choices. The number of comparisons is therefore given by $N k -1 = 2 \cdot 3 -1 = 5$ which is much better for large problems than an exhaustive search, if you start clever.

## 4) Value Iteration in Stochastic Environments

All of the pubs have different special offers on some days of the week.
Due to general confusion you have no clue, which day of the week we currently have.
You only know, for example, that Globetrotter has one happy-hour-day per week, but Black Sheep has four days per week.
So, the chance to get a positive reward in the Black Sheep is higher than in the Globetrotter.

To find the best path we can use the Bellman optimality equation we know from the lecture:

$v_\pi(x_k) = \text{max}_{u_k\in \mathcal{U}} \mathbb{E}\left[R_{k+1} + \gamma v_\pi(X_{k+1}|X_k = x_k, U_k = u_k)\right]$

## Comparison to lecture:
In the tree example from lecture we have deterministic rewards and a stochastic state transition.
So $v_\pi(x_k) = \text{max}_{u_k\in \mathcal{U}} r_x^u + \gamma \Sigma_{x_{k+1}\in \mathcal{X}}p_{xx'}^u v_\pi(x_{k+1})$

In our problem we have deterministic state transitions because we reach the bar we plan to visit for sure.
But the reward in our case has a stochastic offset.
If happy-hour-day (randomly, we do not know the weekday), we get an additional positive reward.
The probability to get that is dependent on the number of happy-hour-days per week.
For example:

As can be seen, in Globetrotter we have in 1 of 7 cases (days) an additional happy-hour-reward and in Black Sheep in 4 of 7 cases.

The states are defined in the following order:

$\mathcal{X} = \left\lbrace \begin{matrix}
\text{Start: Home}\\
\text{Auld Triangle}\\
\text{Lötlampe}\\
\text{Globetrotter}\\
\text{Black Sheep}\\
\text{Limericks}\\
\text{Fat Louis}\\
\text{End: Home}\\
\end{matrix}
\right\rbrace$

The probability to get the positive reward (happy hour) is defined by:

$p_{xr_+} = \left[ \begin{matrix}
\frac{3}{7} & \frac{1}{7} & \frac{1}{7} & \frac{1}{7} & \frac{1}{7} & 0 & 0\\
\frac{6}{7} & \frac{4}{7} & \frac{4}{7} & \frac{5}{7} & \frac{5}{7} & 0 & 0\\
\end{matrix}
\right]$

$r_{+} = \left[ \begin{matrix}
12 & 16 & 16 & 16 & 16 & 0 & 0\\
6 & 10 & 10 & 8 & 8 & 0 & 0\\
\end{matrix}
\right]$

The probability to get the no extra reward is $p_{xr_-} = 1 - p_{xr_+}$.
In that case you take the "no extra reward" $r_-$ is zero in all cases.

$r_{-} = \left[ \begin{matrix}
0 & 0 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & 0 & 0 & 0\\
\end{matrix}
\right]$

The actions to choose are $\textit{up}$ and $\textit{down}$ (for states Li and F, $\textit{up}$ and $\textit{down}$ mean the same).

So if you are at home and you go $\textit{up}$ you get a fixed reward of $-3$ on the way.
With a probability of $p_{x=0,r_+} = \frac{3}{7}$ there is happy hour in the Auld Triangle and you get an additional reward of $+12$.

1. Examine the pseudocode for value iteration as presented in the lecture. Does this pseudocode already contain the concept of a stochastic reward and where do we find it?
2. Make use of value iteration to (use $\gamma = 1$):
    1. Find the state value for each state
    2. Find the optimal policy
3. Check your solution by dynamic programming by hand like in 3)

### Pseudocode:
***
- **input:** Full model of the MDP i.e. $\left\langle\mathcal{X}, \mathcal{U}, \mathcal{P}, \mathcal{R}, \gamma \right\rangle$
- **parameter:** $\delta>0$ as accuracy termination threshold
- **init:** $v_0(x_k)\, \forall \, x_k\in\mathcal{X}$ arbitrary except $v_0(x_k)=0$ if $x_k$ is terminal
- **repeat**
    - $\Delta \leftarrow 0 $
    - **for** $\forall \, x_k\in\mathcal{X}$
        - $\tilde{v}\leftarrow \hat{v}(x_k)$
		- $\hat{v}(x_k)\leftarrow  \max_{u_k\in\mathcal{U}}\left(\mathcal{R}^u_x + \gamma\sum_{x_{k+1}\in\mathcal{X}}p_{xx'}^u \hat{v}(x_{k+1})\right)$
		- $\Delta \leftarrow \max\left(\Delta, |\tilde{v}-\hat{v}(x_k) |\right)$
    - **end**
- **until** $\Delta < \delta$
- **output:** Deterministic policy $\pi\approx\pi^*$, such that

$\pi(x_k)\leftarrow  \text{arg} \, \text{max}_{u_k\in\mathcal{U}}\left(\mathcal{R}^u_x + \gamma\sum_{x_{k+1}\in\mathcal{X}}p_{xx'}^u \hat{v}(x_{k+1})\right)$
***
Value iteration (note: compared to policy iteration, value iteration doesn't require an initial policy but only a state-value guess)

## 4) Solution

```python
import numpy as np
np.set_printoptions(linewidth=100)

r_ways = np.array([[-3, -2, -3, -4, -5, -6, -7],
                   [-1, -4, -5, -5, -6, -6, -7]]) # fixed rewards for the upwards or downwards path

p_xr = np.array([[3/7, 1/7, 1/7, 1/7, 1/7, 0, 0],
                 [6/7, 4/7, 4/7, 5/7, 5/7, 0, 0]]) #probability of success for the upwards or downwards path

r_happy = np.array([[12, 16, 16, 16, 16, 0, 0],
                    [ 6, 10, 10,  8,  8, 0, 0]])

expected_rewards = r_ways + p_xr*r_happy + (1-p_xr)*0
values = np.zeros([8])

delta = 0.1 # lower tolerance boundary

### BEGIN SOLUTION

error = 100

iteration_idx=0
while error > delta:
    iteration_idx += 1
    error = 0
    for state_idx in range(len(values)-1):
        v_tilde = values[state_idx]
        up_value = expected_rewards[0, state_idx] + values[state_idx + (state_idx % 2 + 1)]
        if state_idx == 5 or state_idx == 6:
            down_value = up_value
        else:
            down_value = expected_rewards[1, state_idx] + values[state_idx + (state_idx % 2 + 2)]
        values[state_idx] = np.max([up_value, down_value])

        error = np.max([error, np.sum(np.abs(v_tilde-values[state_idx]))])

# Alternative, faster solution
values2 = np.zeros(8)
iteration_idx2 = 0
error = 100
transition_indices = np.arange(1, 8)
transition_indices += 1 - transition_indices % 2  # results in [1 3 3 5 5 7 7]
transition_indices = np.column_stack([transition_indices, transition_indices + 1]).clip(max=7).T
while error > delta:
    iteration_idx2 += 1
    updated_values2 = np.max(expected_rewards + values2[transition_indices], axis=0)
    error = np.abs(values2[:-1] - updated_values2).max()  # last state in values2 is never updated
    values2[:-1] = updated_values2  # terminal state will never be updated
assert np.allclose(values, values2)
assert iteration_idx == iteration_idx2

### END SOLUTION
print(values)
print(iteration_idx)
```

    [-2.42857143 -5.57142857 -6.57142857 -6.28571429 -7.28571429 -6.         -7.          0.        ]
    5

```python
policy = []
for state_idx in range(len(values)-1):
    up_value = expected_rewards[0, state_idx] + values[state_idx + (state_idx % 2 + 1)]
    if state_idx == 5 or state_idx == 6:
        down_value = up_value
    else:
        down_value = expected_rewards[1, state_idx] + values[state_idx + (state_idx % 2 + 2)]
    if up_value > down_value:
        policy.append("Up")
    elif up_value < down_value:
        policy.append("Down")
    else:
        policy.append("Graduated! Go home!")
print(policy)

# Alternative, faster solution
actions = ['Up', 'Down']
states = ['Start: Home', 'Auld Triangle', 'Lötlampe', 'Globetrotter', 'Black Sheep', 'Limericks', 'Fat Louis']
greedy_policy = np.argmax(expected_rewards + values2[transition_indices], axis=0)
for i, (s, p) in enumerate(zip(states, greedy_policy.tolist())):
    # From last two states there is only one direction
    direction = f'Go {actions[p]}' if i < len(states) - 2 else 'Graduated! Go home!'
    print(f'When in {s} - {direction}')
```

    ['Down', 'Down', 'Down', 'Down', 'Down', 'Graduated! Go home!', 'Graduated! Go home!']
    When in Start: Home - Go Down
    When in Auld Triangle - Go Down
    When in Lötlampe - Go Down
    When in Globetrotter - Go Down
    When in Black Sheep - Go Down
    When in Limericks - Graduated! Go home!
    When in Fat Louis - Graduated! Go home!


# Linear Regression Tutorial

```python
import numpy as np
import scipy.linalg
import matplotlib.pyplot as plt
%matplotlib inline
```

We consider a linear regression problem of the form
$$
y = \boldsymbol x^T\boldsymbol\theta + \epsilon\,,\quad \epsilon \sim \mathcal N(0, \sigma^2)
$$
where $\boldsymbol x\in\mathbb{R}^D$ are inputs and $y\in\mathbb{R}$ are noisy observations. The parameter vector $\boldsymbol\theta\in\mathbb{R}^D$ parametrizes the function.

We assume we have a training set $(\boldsymbol x_n, y_n)$, $n=1,\ldots, N$. We summarize the sets of training inputs in $\mathcal X = \{\boldsymbol x_1, \ldots, \boldsymbol x_N\}$ and corresponding training targets $\mathcal Y = \{y_1, \ldots, y_N\}$, respectively.

In this tutorial, we are interested in finding good parameters $\boldsymbol\theta$.

```python
# Define training set
X = np.array([-3, -1, 0, 1, 3]).reshape(-1,1) # 5x1 vector, N=5, D=1
y = np.array([-1.2, -0.7, 0.14, 0.67, 1.67]).reshape(-1,1) # 5x1 vector

# Plot the training set
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

## 1. Maximum Likelihood
We will start with maximum likelihood estimation of the parameters $\boldsymbol\theta$. In maximum likelihood estimation, we find the parameters $\boldsymbol\theta^{\mathrm{ML}}$ that maximize the likelihood
$$
p(\mathcal Y | \mathcal X, \boldsymbol\theta) = \prod_{n=1}^N p(y_n | \boldsymbol x_n, \boldsymbol\theta)\,.
$$
From the lecture we know that the maximum likelihood estimator is given by
$$
\boldsymbol\theta^{\text{ML}} = (\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\in\mathbb{R}^D\,,
$$
where
$$
\boldsymbol X = [\boldsymbol x_1, \ldots, \boldsymbol x_N]^T\in\mathbb{R}^{N\times D}\,,\quad \boldsymbol y = [y_1, \ldots, y_N]^T \in\mathbb{R}^N\,.
$$

Let us compute the maximum likelihood estimate for a given training set

```python
## EDIT THIS FUNCTION
def max_lik_estimate(X, y):

    # X: N x D matrix of training inputs
    # y: N x 1 vector of training targets/observations
    # returns: maximum likelihood parameters (D x 1)

    N, D = X.shape
    theta_ml = np.zeros((D,1)) ## <-- EDIT THIS LINE
    return theta_ml
```

```python
# get maximum likelihood estimate
theta_ml = max_lik_estimate(X,y)

#@test
assert np.any(theta_ml[0]), "max_lik_estimate is not implemented"
```

Now, make a prediction using the maximum likelihood estimate that we just found

```python
## EDIT THIS FUNCTION
def predict_with_estimate(Xtest, theta):

    # Xtest: K x D matrix of test inputs
    # theta: D x 1 vector of parameters
    # returns: prediction of f(Xtest); K x 1 vector

    prediction = Xtest ## <-- EDIT THIS LINE

    return prediction
```

Now, let's see whether we got something useful:

```python
# define a test set
Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest, theta_ml)

#@test
assert not np.array_equal(ml_prediction, Xtest), "predict_with_estimate not implemented"

# plot
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Questions
1. Does the solution above look reasonable?
2. Play around with different values of $\theta$. How do the corresponding functions change?
3. Modify the training targets $\mathcal Y$ and re-run your computation. What changes?

Let us now look at a different training set, where we add 2.0 to every $y$-value, and compute the maximum likelihood estimate

```python
ynew = y + 2.0

plt.figure()
plt.plot(X, ynew, '+', markersize=10)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

```python
# get maximum likelihood estimate
theta_ml = max_lik_estimate(X, ynew)
print(theta_ml)

# define a test set
Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest, theta_ml)

# plot
plt.figure()
plt.plot(X, ynew, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Question:
1. This maximum likelihood estimate doesn't look too good: The orange line is too far away from the observations although we just shifted them by 2. Why is this the case?
2. How can we fix this problem?

Let us now define a linear regression model that is slightly more flexible:
$$
y = \theta_0 + \boldsymbol x^T \boldsymbol\theta_1 + \epsilon\,,\quad \epsilon\sim\mathcal N(0,\sigma^2)
$$
Here, we added an offset (bias) parameter $\theta_0$ to our original model.

#### Question:
1. What is the effect of this bias parameter, i.e., what additional flexibility does it offer?

If we now define the inputs to be the augmented vector $\boldsymbol x_{\text{aug}} = \begin{bmatrix}1\\\boldsymbol x\end{bmatrix}$, we can write the new linear regression model as
$$
y = \boldsymbol x_{\text{aug}}^T\boldsymbol\theta_{\text{aug}} + \epsilon\,,\quad \boldsymbol\theta_{\text{aug}} = \begin{bmatrix}
\theta_0\\
\boldsymbol\theta_1
\end{bmatrix}\,.
$$

```python
N, D = X.shape
X_aug = np.hstack([np.ones((N,1)), X]) # augmented training inputs of size N x (D+1)
theta_aug = np.zeros((D+1, 1)) # new theta vector of size (D+1) x 1
```

Let us now compute the maximum likelihood estimator for this setting.
_Hint:_ If possible, re-use code that you have already written

```python
## EDIT THIS FUNCTION
def max_lik_estimate_aug(X_aug, y):

    theta_aug_ml = np.zeros((D+1,1)) ## <-- EDIT THIS LINE

    return theta_aug_ml
```

```python
theta_aug_ml = max_lik_estimate_aug(X_aug, y)

#@test
assert np.any(theta_aug_ml), "Max_lik_estimate_aug not implemented"
```

Now, we can make predictions again:

```python
# define a test set (we also need to augment the test inputs with ones)
Xtest_aug = np.hstack([np.ones((Xtest.shape[0],1)), Xtest]) # 100 x (D + 1) vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest_aug, theta_aug_ml)

# plot
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

It seems this has solved our problem!
#### Question:
1. Play around with the first parameter of $\boldsymbol\theta_{\text{aug}}$ and see how the fit of the function changes.
2. Play around with the second parameter of $\boldsymbol\theta_{\text{aug}}$ and see how the fit of the function changes.

### Nonlinear Features
So far, we have looked at linear regression with linear features. This allowed us to fit straight lines. However, linear regression also allows us to fit functions that are nonlinear in the inputs $\boldsymbol x$, as long as the parameters $\boldsymbol\theta$ appear linearly. This means, we can learn functions of the form
$$
f(\boldsymbol x, \boldsymbol\theta) = \sum_{k = 1}^K \theta_k \phi_k(\boldsymbol x)\,,
$$
where the features $\phi_k(\boldsymbol x)$ are (possibly nonlinear) transformations of the inputs $\boldsymbol x$.

Let us have a look at an example where the observations clearly do not lie on a straight line:

```python
y = np.array([10.05, 1.5, -1.234, 0.02, 8.03]).reshape(-1,1)
plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Polynomial Regression
One class of functions that is covered by linear regression is the family of polynomials because we can write a polynomial of degree $K$ as
$$
\sum_{k=0}^K \theta_k x^k = \boldsymbol \phi(x)^T\boldsymbol\theta\,,\quad
\boldsymbol\phi(x)=
\begin{bmatrix}
x^0\\
x^1\\
\vdots\\
x^K
\end{bmatrix}\in\mathbb{R}^{K+1}\,.
$$
Here, $\boldsymbol\phi(x)$ is a nonlinear feature transformation of the inputs $x\in\mathbb{R}$.

Similar to the earlier case we can define a matrix that collects all the feature transformations of the training inputs:
$$
\boldsymbol\Phi = \begin{bmatrix}
\boldsymbol\phi(x_1) & \boldsymbol\phi(x_2) & \cdots & \boldsymbol\phi(x_n)
\end{bmatrix}^T \in\mathbb{R}^{N\times K+1}
$$

Let us start by computing the feature matrix $\boldsymbol \Phi$

```python
## EDIT THIS FUNCTION
def poly_features(X, K):

    #X: inputs of size N x 1
    #K: degree of the polynomial
    # computes the feature matrix Phi (N x (K+1))

    X = X.flatten()
    N = X.shape[0]

    #initialize Phi
    Phi = np.zeros((N, K+1))

    # Compute the feature matrix in stages
    Phi = np.zeros((N, K+1)) ## <-- EDIT THIS LINE
    return Phi
```

With this feature matrix we get the maximum likelihood estimator as
$$
\boldsymbol \theta^\text{ML} = (\boldsymbol\Phi^T\boldsymbol\Phi)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$
For reasons of numerical stability, we often add a small diagonal "jitter" $\kappa>0$ to $\boldsymbol\Phi^T\boldsymbol\Phi$ so that we can invert the matrix without significant problems so that the maximum likelihood estimate becomes
$$
\boldsymbol \theta^\text{ML} = (\boldsymbol\Phi^T\boldsymbol\Phi + \kappa\boldsymbol I)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$

```python
## EDIT THIS FUNCTION
def nonlinear_features_maximum_likelihood(Phi, y):
    # Phi: features matrix for training inputs. Size of N x D
    # y: training targets. Size of N by 1
    # returns: maximum likelihood estimator theta_ml. Size of D x 1

    jitter = 1e-08 # good for numerical stability

    K = Phi.shape[1]

    # maximum likelihood estimate
    theta_ml = np.zeros((K,1)) ## <-- EDIT THIS LINE

    return theta_ml
```

Now we have all the ingredients together: The computation of the feature matrix and the computation of the maximum likelihood estimator for polynomial regression. Let's see how this works.

To make predictions at test inputs $\boldsymbol X_{\text{test}}\in\mathbb{R}$, we need to compute the features (nonlinear transformations) $\boldsymbol\Phi_{\text{test}}= \boldsymbol\phi(\boldsymbol X_{\text{test}})$ of $\boldsymbol X_{\text{test}}$ to give us the predicted mean
$$
\mathbb{E}[\boldsymbol y_{\text{test}}] = \boldsymbol \Phi_{\text{test}}\boldsymbol\theta^{\text{ML}}
$$

```python
K = 5 # Define the degree of the polynomial we wish to fit
Phi = poly_features(X, K) # N x (K+1) feature matrix

#@Test
assert np.any(Phi), "Poly_features not implemented"

theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator

#@Test
assert np.any(theta_ml), "nonlinear_features_maximum_likelihood"

# test inputs
Xtest = np.linspace(-4,4,100).reshape(-1,1)

# feature matrix for test inputs
Phi_test = poly_features(Xtest, K)

#@Test
assert np.any(theta_ml), "nonlinear_features_maximum_likelihood"

y_pred = Phi_test @ theta_ml # predicted y-values

plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

Experiment with different polynomial degrees in the code above.
#### Questions:
1. What do you observe?
2. What is a good fit?

## Evaluating the Quality of the Model

Let us have a look at a more interesting data set

```python
def f(x):
    return np.cos(x) + 0.2*np.random.normal(size=(x.shape))

X = np.linspace(-4,4,20).reshape(-1,1)
y = f(X)

plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

Now, let us use the work from above and fit polynomials to this dataset.

```python
## EDIT THIS CELL
K = 2 # Define the degree of the polynomial we wish to fit

Phi = poly_features(X, K) # N x (K+1) feature matrix

#@Test
assert np.any(Phi), "Poly_features not implemented"

theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator

#@Test
assert np.any(theta_ml), "nonlinear_features_maximum_likelihood"

# test inputs
Xtest = np.linspace(-5,5,100).reshape(-1,1)
ytest = f(Xtest) # ground-truth y-values

# feature matrix for test inputs
Phi_test = poly_features(Xtest, K)

#@Test
assert np.any(Phi), "Poly_features not implemented"

y_pred = Xtest*0 # <-- EDIT THIS LINE

#@Test
assert np.any(y_pred), "Prediction not implemented"

# plot
plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred)
plt.plot(Xtest, ytest)
plt.legend(["data", "prediction", "ground truth observations"])
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Questions:
1. Try out different degrees of polynomials.
2. Based on visual inspection, what looks like the best fit?

Let us now look at a more systematic way to assess the quality of the polynomial that we are trying to fit. For this, we compute the root-mean-squared-error (RMSE) between the $y$-values predicted by our polynomial and the ground-truth $y$-values. The RMSE is then defined as
$$
\text{RMSE} = \sqrt{\frac{1}{N}\sum_{n=1}^N(y_n - y_n^\text{pred})^2}
$$
Write a function that computes the RMSE.

```python
## EDIT THIS FUNCTION
def RMSE(y, ypred):
    rmse = -1 ## <-- EDIT THIS LINE
    return rmse
```

Now compute the RMSE for different degrees of the polynomial we want to fit.

```python
## EDIT THIS CELL
K_max = 20
rmse_train = np.zeros((K_max+1,))

for k in range(K_max+1):

    rmse_train[k] = -1 # <-- EDIT THIS LINE

    assert rmse_train[k] != -1, "RMSE not implemented"

plt.figure()
plt.plot(rmse_train)
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE");
```

#### Question:
1. What do you observe?
2. What is the best polynomial fit according to this plot?
3. Write some code that plots the function that uses the best polynomial degree (use the test set for this plot). What do you observe now?

```python
# WRITE THE PLOTTING CODE HERE
plt.figure()
plt.plot(X, y, '+')
ypred_test = Xtest*0 ## <--- EDIT THIS LINE (hint: you may require a few lines to do the computation)

#@Test
assert any(ypred_test), "Prediction not implemented"

plt.plot(Xtest, ypred_test)
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.legend(["data", "maximum likelihood fit"]);
```

The RMSE on the training data is somewhat misleading, because we are interested in the generalization performance of the model. Therefore, we are going to compute the RMSE on the test set and use this to choose a good polynomial degree.

```python
## EDIT THIS CELL
K_max = 20
rmse_train = np.zeros((K_max+1,))
rmse_test = np.zeros((K_max+1,))

for k in range(K_max+1):

    # feature matrix
    Phi = 0 ## <--- EDIT THIS LINE

    # maximum likelihood estimate
    theta_ml = 0 ## <--- EDIT THIS LINE

    # predict y-values of training set
    ypred_train = 0 ## <--- EDIT THIS LINE

    # RMSE on training set
    rmse_train[k] = 0 ## <--- EDIT THIS LINE

    # feature matrix for test inputs
    Phi_test = 0 ## <--- EDIT THIS LINE

    # prediction (test set)
    ypred_test = 0 ## <--- EDIT THIS LINE

    # RMSE on test set
    rmse_test[k] = -1 ## <--- EDIT THIS LINE

plt.figure()
plt.semilogy(rmse_train) # this plots the RMSE on a logarithmic scale
plt.semilogy(rmse_test) # this plots the RMSE on a logarithmic scale
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE")
plt.legend(["training set", "test set"]);
```

#### Questions:
1. What do you observe now?
2. Why does the RMSE for the test set not always go down?
3. Which polynomial degree would you choose now?
4. Plot the fit for the "best" polynomial degree.

```python
# WRITE THE PLOTTING CODE HERE
plt.figure()
plt.plot(X, y, '+')
ypred_test = Xtest*0 ## <--- EDIT THIS LINE (hint: you may require a few lines to do the computation)

#@Test
assert any(ypred_test), "Prediction not implemented"

plt.plot(Xtest, ypred_test)
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.legend(["data", "maximum likelihood fit"]);
```

#### Question
If you did not have a designated test set, what could you do to estimate the generalization error (purely using the training set)?

## 2. Maximum A Posteriori Estimation

We are still considering the model
$$
y = \boldsymbol\phi(\boldsymbol x)^T\boldsymbol\theta + \epsilon\,,\quad \epsilon\sim\mathcal N(0,\sigma^2)\,.
$$
We assume that the noise variance $\sigma^2$ is known.

Instead of maximizing the likelihood, we can look at the maximum of the posterior distribution on the parameters $\boldsymbol\theta$, which is given as
$$
p(\boldsymbol\theta|\mathcal X, \mathcal Y) = \frac{\overbrace{p(\mathcal Y|\mathcal X, \boldsymbol\theta)}^{\text{likelihood}}\overbrace{p(\boldsymbol\theta)}^{\text{prior}}}{\underbrace{p(\mathcal Y|\mathcal X)}_{\text{evidence}}}
$$
The purpose of the parameter prior $p(\boldsymbol\theta)$ is to discourage the parameters to attain extreme values, a sign that the model overfits. The prior allows us to specify a "reasonable" range of parameter values. Typically, we choose a Gaussian prior $\mathcal N(\boldsymbol 0, \alpha^2\boldsymbol I)$, centered at $\boldsymbol 0$ with variance $\alpha^2$ along each parameter dimension.

The MAP estimate of the parameters is
$$
\boldsymbol\theta^{\text{MAP}} = (\boldsymbol\Phi^T\boldsymbol\Phi + \frac{\sigma^2}{\alpha^2}\boldsymbol I)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$
where $\sigma^2$ is the variance of the noise.

```python
## EDIT THIS FUNCTION
def map_estimate_poly(Phi, y, K, sigma, alpha):
    # Phi: training inputs, Size of N x (K+1)
    # y: training targets, Size of D x 1
    # K: degree of the polynomial
    # sigma: standard deviation of the noise
    # alpha: standard deviation of the prior on the parameters
    # returns: MAP estimate theta_map, Size of (K+1) x 1

    theta_map = np.zeros((K+1,1)) ## <-- EDIT THIS LINE

    return theta_map
```

```python
# define the function we wish to estimate later
def g(x, sigma):
    p = np.hstack([x**0, x**1, np.sin(x)])
    w = np.array([-1.0, 0.1, 1.0]).reshape(-1,1)
    return p @ w + sigma*np.random.normal(size=x.shape)
```

```python
# Generate some data
sigma = 1.0 # noise standard deviation
alpha = 1.0 # standard deviation of the parameter prior
N = 20

np.random.seed(42)

X = (np.random.rand(N)*10.0 - 5.0).reshape(-1,1)
y = g(X, sigma) # training targets

plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

```python
# get the MAP estimate
K = 8 # polynomial degree

# feature matrix
Phi = poly_features(X, K)

theta_map = map_estimate_poly(Phi, y, K, sigma, alpha)

#@Test
assert any(theta_map), "map_estimate_poly not implemented"

# maximum likelihood estimate
theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

Xtest = np.linspace(-5,5,100).reshape(-1,1)
ytest = g(Xtest, sigma)

Phi_test = poly_features(Xtest, K)
y_pred_map = Phi_test @ theta_map

y_pred_mle = Phi_test @ theta_ml

plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred_map)
plt.plot(Xtest, g(Xtest, 0))
plt.plot(Xtest, y_pred_mle)

plt.legend(["data", "map prediction", "ground truth function", "maximum likelihood"]);
```

```python
print(np.hstack([theta_ml, theta_map]))
```

Now, let us compute the RMSE for different polynomial degrees and see whether the MAP estimate addresses the overfitting issue we encountered with the maximum likelihood estimate.

```python
## EDIT THIS CELL

K_max = 12 # this is the maximum degree of polynomial we will consider
assert(K_max < N) # this is the latest point when we'll run into numerical problems

rmse_mle = np.zeros((K_max+1,))
rmse_map = np.zeros((K_max+1,))

for k in range(K_max+1):

    rmse_mle[k] = -1 ## Compute the maximum likelihood estimator, compute the test-set predicitons, compute the RMSE
    rmse_map[k] = -1 ## Compute the MAP estimator, compute the test-set predicitons, compute the RMSE

    #@Test
    assert rmse_mle[k] != -1, "mle prediction not implemented"
    assert rmse_map[k] != -1, "map prediction not implemented"

plt.figure()
plt.semilogy(rmse_mle) # this plots the RMSE on a logarithmic scale
plt.semilogy(rmse_map) # this plots the RMSE on a logarithmic scale
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE")
plt.legend(["Maximum likelihood", "MAP"])
```

#### Questions:
1. What do you observe?
2. What is the influence of the prior variance on the parameters ($\alpha^2$)? Change the parameter and describe what happens.

## 3. Bayesian Linear Regression

```python
# Test inputs
Ntest = 200
Xtest = np.linspace(-5, 5, Ntest).reshape(-1,1) # test inputs

prior_var = 2.0 # variance of the parameter prior (alpha^2). We assume this is known.
noise_var = 1.0 # noise variance (sigma^2). We assume this is known.

pol_deg = 3 # degree of the polynomial we consider at the moment
```

Assume a parameter prior $p(\boldsymbol\theta) = \mathcal N (\boldsymbol 0, \alpha^2\boldsymbol I)$. For every test input $\boldsymbol x_*$ we obtain the
prior mean
$$
E[f(\boldsymbol x_*)] = 0
$$
and the prior (marginal) variance (ignoring the noise contribution)
$$
V[f(\boldsymbol x_*)] = \alpha^2\boldsymbol\phi(\boldsymbol x_*) \boldsymbol\phi(\boldsymbol x_*)^\top
$$
where $\boldsymbol\phi(\cdot)$ is the feature map.

```python
## EDIT THIS CELL

# compute the feature matrix for the test inputs
Phi_test = np.zeros((Ntest, pol_deg+1))  # N x (pol_deg+1) feature matrix <--- EDIT THIS LINE
raise NotImplementedError

# compute the (marginal) prior at the test input locations
# prior mean
prior_mean = np.ones((Ntest,1))  # prior mean at test inputs (size: (Ntest,1)) <-- EDIT THIS LINE
raise NotImplementedError

# prior variance
full_covariance = np.zeros((Ntest, Ntest)) # N x N covariance matrix of all function values <-- EDIT THIS LINE
prior_marginal_var = 0 # marginal of size (N, )
raise NotImplementedError

# Let us visualize the prior over functions
plt.figure()
plt.plot(Xtest, prior_mean, color="k")

conf_bound1 = np.sqrt(prior_marginal_var).flatten()
conf_bound2 = 2.0*np.sqrt(prior_marginal_var).flatten()
conf_bound3 = 2.0*np.sqrt(prior_marginal_var).flatten() + 2.0*np.sqrt(noise_var)
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound1,
             prior_mean.flatten() - conf_bound1, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound2,
                 prior_mean.flatten() - conf_bound2, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound3,
                 prior_mean.flatten() - conf_bound3, alpha = 0.1, color="k")

plt.xlabel('$x$')
plt.ylabel('$y$')
plt.title("Prior over functions");
```

Now, we will use this prior distribution and sample functions from it.

```python
## EDIT THIS CELL

# samples from the prior
num_samples = 10

# We first need to generate random weights theta_i, which we sample from the parameter prior
random_weights = np.random.normal(size=(pol_deg+1,num_samples), scale=np.sqrt(prior_var))

# Now, we compute the induced random functions, evaluated at the test input locations
# Every function sample is given as f_i = Phi * theta_i,
# where theta_i is a sample from the parameter prior

sample_function = np.zeros((Ntest,)) # <-- EDIT THIS LINE
raise NotImplementedError

plt.figure()
plt.plot(Xtest, sample_function, color="r")
plt.title("Plausible functions under the prior")
print("Every sampled function is a polynomial of degree "+str(pol_deg));
```

Now we are given some training inputs $\boldsymbol x_1, \dotsc, \boldsymbol x_N$, which we collect in a matrix $\boldsymbol X = [\boldsymbol x_1, \dotsc, \boldsymbol x_N]^\top\in\mathbb{R}^{N\times D}$

```python
N = 10
X = np.random.uniform(high=5, low=-5, size=(N,1)) # training inputs, size Nx1
y = g(X, np.sqrt(noise_var)) # training targets, size Nx1
```

Now, let us compute the posterior

```python
## EDIT THIS FUNCTION

def polyfit(X, y, K, prior_var, noise_var):
    # X: training inputs, size NxD
    # y: training targets, size Nx1
    # K: degree of polynomial we consider
    # prior_var: prior variance of the parameter distribution
    # sigma: noise variance

    jitter = 1e-08

    Phi = poly_features(X, K) # N x (K+1) feature matrix

    # Compute maximum likelihood estimate
    theta_ml = np.zeros((K+1,1)) # <-- EDIT THIS LINE

    # MAP estimate
    theta_map = np.zeros((K+1,1)) # <-- EDIT THIS LINE

    # Parameter posterior
    SN = np.zeros(K+1) # covariance matrix of the parameter posterior # <-- EDIT THIS LINE
    mN = np.zeros((K+1,1)) # mean vector of the parameter posterior   # <-- EDIT THIS LINE

    raise NotImplementedError

    return (theta_ml, theta_map, mN, SN)
```

```python
theta_ml, theta_map, theta_mean, theta_var = polyfit(X, y, pol_deg, alpha, sigma)
```

Now, let's make predictions (ignoring the measurement noise). We obtain three predictors:
\begin{align}
&\text{Maximum likelihood: }E[f(\boldsymbol X_{\text{test}})] = \boldsymbol \phi(X_{\text{test}})\boldsymbol \theta_{ml}\\
&\text{Maximum a posteriori: } E[f(\boldsymbol X_{\text{test}})] = \boldsymbol \phi(X_{\text{test}})\boldsymbol \theta_{map}\\
&\text{Bayesian: } p(f(\boldsymbol X_{\text{test}})) = \mathcal N(f(\boldsymbol X_{\text{test}}) \,|\, \boldsymbol \phi(X_{\text{test}}) \boldsymbol\theta_{\text{mean}},\, \boldsymbol\phi(X_{\text{test}}) \boldsymbol\theta_{\text{var}}  \boldsymbol\phi(X_{\text{test}})^\top)
\end{align}
We already computed all quantities. Write some code that implements all three predictors.

```python
## EDIT THIS CELL

# predictions (ignoring the measurement/observations noise)

# maximum likelihood predictions (just the mean)
m_mle_test = np.zeros((Ntest,1)) # <-- EDIT THIS LINE

# MAP predictions (just the mean)
m_map_test = np.zeros((Ntest,1)) # <-- EDIT THIS LINE

# predictive distribution (Bayesian linear regression)
# mean prediction
mean_blr = np.zeros((Ntest,1)) # <-- EDIT THIS LINE
# variance prediction
cov_blr =  np.ones((Ntest,Ntest)) # <-- EDIT THIS LINE

raise NotImplementedError
```

```python
# plot the posterior
plt.figure()
plt.plot(X, y, "+")
plt.plot(Xtest, m_mle_test)
plt.plot(Xtest, m_map_test)
var_blr = np.diag(cov_blr)
conf_bound1 = np.sqrt(var_blr).flatten()
conf_bound2 = 2.0*np.sqrt(var_blr).flatten()
conf_bound3 = 2.0*np.sqrt(var_blr).flatten() + 2.0*sigma

plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound1,
                 mean_blr.flatten() - conf_bound1, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound2,
                 mean_blr.flatten() - conf_bound2, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound3,
                 mean_blr.flatten() - conf_bound3, alpha = 0.1, color="k")
plt.legend(["Training data", "MLE", "MAP", "BLR"])
plt.xlabel('$x$');
plt.ylabel('$y$');
```

```python

```


# Compute $\boldsymbol{J}$ in Kinetic Energy

We can define the rotational kinetic energy of a single body as

$$
\frac{1}{2} \boldsymbol{\omega}^{T} \boldsymbol{I} \boldsymbol{\omega}=\frac{1}{2} \operatorname{tr}\left([\boldsymbol{\omega}] \boldsymbol{J}[\boldsymbol{\omega}]^{T}\right),
$$

where $\boldsymbol{J}$ is a diagonal PD matrix that ensures the equality above holds. The following notebook derives the definition of $\boldsymbol{J}$.

```python
import numpy
import sympy.vector
from IPython.display import Latex, Math, display
from sympy import *

init_printing()
```

```python
def hat(x):
    return numpy.array([[0, -x[2], x[1]], [x[2], 0, -x[0]], [-x[1], x[0], 0]],
                       dtype=x.dtype)
```

```python
omega = numpy.array(symbols("omega_x omega_y omega_z"))
I = numpy.diag(symbols("I_x I_y I_z"))
J = numpy.diag(symbols("J_0 J_1 J_2"))
```

```python
lhs = omega.T @ I @ omega
display(lhs.collect(omega))
lhs_coeffs = [lhs.coeff(omega[i], 2) for i in range(omega.size)]
display(lhs_coeffs)
```

```python
rhs = numpy.trace(hat(omega) @ J @ hat(omega).T)
display(rhs.collect(omega))
rhs_coeffs = [rhs.coeff(omega[i], 2) for i in range(omega.size)]
display(rhs_coeffs)
```

```python
eqs = [
    Eq(lhs_coeff, rhs_coeff)
    for lhs_coeff, rhs_coeff in zip(lhs_coeffs, rhs_coeffs)
]
display(eqs)
J_of_I = solve(eqs, numpy.diag(J).tolist())
J_solved = numpy.diag([J_of_I[numpy.diag(J)[i]] for i in range(3)])
display(Latex("$\\boldsymbol{J}=$"), Matrix(J_solved))
```

```python
x = numpy.array(symbols("x y z"), dtype=Symbol)
(x.T @ J_solved @ x).collect(numpy.diag(I))
```


# Deadbeat controller for the double integrator

Consider a mass of 1kg moving in one direction on a friction-free horizontal surface. We can apply a force to the mass (input signal $u$), and the mass is also subject to disturbance forces $v$. We are interested in controlling the position $z$ of the mass. In continuous time the dynamics are described by
$$ \ddot{z} = u + v. $$
Introducing the state variables $x_1=z$ and $x_2=\dot{z}$, the system can also be represented on state-space form with state vector $x = \begin{bmatrix} z & \dot{z}\end{bmatrix}^T$ as
\begin{align}
\dot{x} &= \underbrace{\begin{bmatrix} 0 & 1\\0 & 0\end{bmatrix}}_{A}x + \underbrace{\begin{bmatrix}0\\1\end{bmatrix}}_{B}u + \underbrace{\begin{bmatrix}0\\1\end{bmatrix}}_{B}v\\
y &= \underbrace{\begin{bmatrix}1 & 0 \end{bmatrix}}_C x
\end{align}

## Discrete-time state-space model
The discrete-time state-space model using a sampling period $h$ is
\begin{align}
x(k+1) &= \Phi(h)x(k) + \Gamma(h)u + \Gamma(h)v\\
y(k) &= Cx(k)
\end{align}
where
$$ \Phi(h) = \mathrm{e}^{Ah} = \begin{bmatrix} 1 & h\\0 & 1 \end{bmatrix}$$
and
$$ \Gamma(h) = \int_0^h \mathrm{e}^{As}B ds = \begin{bmatrix} \frac{h^2}{2}\\h \end{bmatrix}.$$
### Verification by symbolic computation

```python
import numpy as np
import sympy as sy
sy.init_printing(use_latex='mathjax', order='lex')

h = sy.symbols('h', real=True, positive=True)
A = sy.Matrix([[0,1], [0,0]])
B = sy.Matrix([[0],[1]])
Phi = sy.simplify(sy.exp(A*h))
Phi
```

$$\left[\begin{matrix}1 & h\\0 & 1\end{matrix}\right]$$

```python
s = sy.symbols('s')
Gamma = sy.integrate(sy.exp(A*s)*B, (s, 0, h))
Gamma
```

$$\left[\begin{matrix}\frac{h^{2}}{2}\\h\end{matrix}\right]$$

## Reachability
The controllability matrix for this second order system becomes
$$ W_c = \begin{bmatrix} \Gamma & \Phi\Gamma \end{bmatrix} = \begin{bmatrix} \frac{h^2}{2} & \frac{3h^2}{2}\\h & h\end{bmatrix}, $$
with determinant
$$\det W_c = \frac{h^3}{2}(1 - 3) = -h^3,$$
which is different from zero since $h>0$.
It is hence possible to reach any point in the state-space from any other point in just two steps (two sampling periods).
### Verification by symbolic computation

```python
Wc = sy.BlockMatrix([[Gamma, Phi*Gamma]]).as_explicit()
Wc
```

$$\left[\begin{matrix}\frac{h^{2}}{2} & \frac{3 h^{2}}{2}\\h & h\end{matrix}\right]$$

```python
sy.det(Wc)
```

$$- h^{3}$$

## Designing an input sequence
We now know that the system is reachable. This means that we can take the system from the origin in the state-space (position zero and velocity zero) to any other point in state-space. And it can be done in only two steps with the input sequence
$$ u(0), \, u(1).$$
Let's say we want to reach the point
$$ x_d = \begin{bmatrix} a\\b \end{bmatrix},$$
which in words is that we want the mass to be at $z=a$ and having the velocity $\dot{z}=b$. The general solution for an n-th order discrete-time state space system is
\begin{align}
x(n) &= \Phi^n x(0) + \Phi^{n-1}\Gamma u(0) + \Phi^{n-2}\Gamma u(1) + \cdots + \Gamma u(n-1)\\
&= \Phi^n x(0) + W_cU,
\end{align}
where
$$ U = \begin{bmatrix} u(n-1)\\u(n-2)\\\vdots\\u(0)\end{bmatrix}. $$
In the case here we have $x(0)=0$ and this leads to the equation
$$ W_cU = x_d, \qquad \text{with solution}$$
\begin{align}
U &= \begin{bmatrix}u(1)\\u(0)\end{bmatrix} = W_c^{-1}x_d = \begin{bmatrix} \frac{h^2}{2} & \frac{3h^2}{2}\\h & h\end{bmatrix}^{-1} \begin{bmatrix} a\\b \end{bmatrix}\\
&= \frac{1}{-h^3} \begin{bmatrix} h & -\frac{3h^2}{2}\\-h & \frac{h^2}{2} \end{bmatrix} \begin{bmatrix} a\\b \end{bmatrix}\\
&= \begin{bmatrix} -\frac{1}{h^2} & \frac{3}{2h}\\\frac{1}{h^2} & -\frac{1}{2h} \end{bmatrix} \begin{bmatrix} a\\b \end{bmatrix}\\
&= \begin{bmatrix} -\frac{a}{h^2} + \frac{3b}{2h}\\ \frac{a}{h^2} - \frac{b}{2h} \end{bmatrix}.
\end{align}
Thus the input sequence becomes $u(0) = \frac{a}{h^2} - \frac{b}{2h}$, $u(1) = \frac{-a}{h^2} + \frac{3b}{2h}$.
### Verification with symbolic computation

```python
# Verify
a,b = sy.symbols('a,b')
U = Wc.inv()*sy.Matrix([[a],[b]])
U
```

$$\left[\begin{matrix}- \frac{a}{h^{2}} + \frac{3 b}{2 h}\\\frac{a}{h^{2}} - \frac{b}{2 h}\end{matrix}\right]$$

```python
# Simulate
u0 = U[1,0]
u1 = U[0,0]
x0 = sy.Matrix([[0],[0]])
x1 = Phi*x0 + Gamma*u0
x2 = Phi*x1 + Gamma*u1
sy.simplify(x2)
```

$$\left[\begin{matrix}a\\b\end{matrix}\right]$$

## State feedback
Introducing the state-feedback control law
$$ u = -l_1x_1 - l_2 x_2 + l_0y_{ref} = -Lx + l_0y_{ref}$$
gives the closed-loop state-space system
\begin{align}
x(k+1) &= \Phi x(k) +\Gamma\big(-Lx(k) + l_0y_{ref}(k)\big) + \Gamma v(k) = \left( \Phi - \Gamma L \right) x(k) + l_0\Gamma y_{ref}(k) + \Gamma v(k)\\
y(k) &= C x(k)
\end{align}
with characteristic polynomial given by
\begin{align}
\det \left( zI - (\Phi-\Gamma L) \right) &= \det \left( \begin{bmatrix} z & 0\\0 & z \end{bmatrix} - \begin{bmatrix} 1 & h\\0 & 1 \end{bmatrix} + \begin{bmatrix} l_1\frac{h^2}{2} & l_2\frac{h^2}{2}\\ l_1h & l_2h \end{bmatrix} \right)\\
&= \det \begin{bmatrix} z-1+l_1\frac{h^2}{2} & -h+l_2\frac{h^2}{2}\\l_1h & z-1+l_2h
\end{bmatrix}\\
&= (z-1+l_1\frac{h^2}{2})(z-1+l_2h) - l_1h(-h + l_2\frac{h^2}{2})\\
&= z^2 + (-1+l_2h-1+l_1\frac{h^2}{2}) z + (1-l_2h - l_1\frac{h^2}{2} + l_1l_2\frac{h^3}{2} +l_1h^2 -l_1l_2\frac{h^3}{2})\\
&= z^2 + (l_1\frac{h^2}{2}+l_2h-2) z + (1 +l_1\frac{h^2}{2} -l_2h)
\end{align}
### Verification by symbolic computation

```python
l1, l2 = sy.symbols('l1, l2', real=True)
z = sy.symbols('z')
L = sy.Matrix([[l1, l2]])
ch_poly = sy.Poly((z*sy.eye(2) - (Phi - Gamma*L)).det(), z)
ch_poly.as_expr()
```

$$\frac{h^{2} l_{1}}{2} - h l_{2} + z^{2} + z \left(\frac{h^{2} l_{1}}{2} + h l_{2} - 2\right) + 1$$

### Desired closed-loop characteristic polynomial
Here we are interested in designing a deadbeat controller, so the desired closed-loop poles are
$$ p_1 = 0, \qquad p_2=0,$$
and the desired characteristic polynomial is
$$ A_c(z) = (z-p_1)(z-p_2) = z^2. $$
In the same spirit as when designing an RST controller using the polynomial approach, we set the calculated characteristic polynomial - obtained when introducing the linear state feedback- equal to the desired characteristic polynomial.
\begin{align}
z^1: \qquad l_1\frac{h^2}{2} + l_2h -2 &= 0\\
z^0: \qquad l_1\frac{h^2}{2} - l_2h+1 &= 0
\end{align}
which can be written as the system of equations
$$ \underbrace{\begin{bmatrix} \frac{h^2}{2} & h\\\frac{h^2}{2} & -h \end{bmatrix}}_{M} \underbrace{\begin{bmatrix} l_1\\l_2\end{bmatrix}}_{L^T} = \underbrace{\begin{bmatrix}2\\-1\end{bmatrix}}_{b} $$
with solution given by

$$L^T = M^{-1}b = \frac{1}{-h^3} \begin{bmatrix} -h & -h\\-\frac{h^2}{2} & \frac{h^2}{2} \end{bmatrix} \begin{bmatrix} 2\\-1 \end{bmatrix}$$

$$ = -\frac{1}{h^3} \begin{bmatrix} -2h+h\\-h^2-\frac{h^2}{2}\end{bmatrix} = \begin{bmatrix} \frac{1}{h^2}\\\frac{3}{2h} \end{bmatrix} $$
### Verification by symbolic calculation

```python
des_ch_poly = sy.Poly(z*z, z)
dioph_eqn = ch_poly - des_ch_poly
sol = sy.solve(dioph_eqn.coeffs(), (l1,l2))
sol
```

$$\left \{ l_{1} : \frac{1}{h^{2}}, \quad l_{2} : \frac{3}{2 h}\right \}$$

In the system of equations $ML^T=b$ above, note that the matrix $M$ can be written
$$ M = \begin{bmatrix} \frac{h^2}{2} & h\\\frac{h^2}{2} & -h \end{bmatrix} = \begin{bmatrix}1 & 0\\-2 & 1\end{bmatrix}\underbrace{\begin{bmatrix} \frac{h^2}{2} & h \\ \frac{3h^2}{2} & h\end{bmatrix}}_{W_c^T}, $$
so $M$ will be invertible if and only if $\det W_c^T = \det W_c \neq 0$.

## The resulting closed-loop system
So, we have found the control law
$$ u(k) = -Lx(k) + l_0y_{ref}(k) = -\begin{bmatrix} \frac{1}{h^2} & \frac{3}{2h} \end{bmatrix}x(k) + l_0 y_{ref}(k)$$
which gives a closed-loop system with poles in the origin, i.e. deadbeat control. The closed-loop system becomes
\begin{align*}
 x(k+1) &= \big( \Phi - \Gamma L \big) x(k) + \Gamma l_0 y_{ref}(k) + \Gamma v(k)\\
 &= \left( \begin{bmatrix} 1 & h\\0 & 1\end{bmatrix} - \begin{bmatrix} \frac{h^2}{2}\\h\end{bmatrix}\begin{bmatrix} \frac{1}{h^2} & \frac{3}{2h} \end{bmatrix}  \right) x(k) + \Gamma l_0 y_{ref}(k) + \Gamma v(k)\\
 &= \left( \begin{bmatrix} 1 & h\\0 & 1\end{bmatrix} - \begin{bmatrix} \frac{1}{2} & \frac{3h}{4}\\ \frac{1}{h} & \frac{3}{2}\end{bmatrix}\right) x(k) + \Gamma l_0 y_{ref}(k) + \Gamma v(k)\\
 &= \underbrace{\begin{bmatrix} \frac{1}{2} & \frac{h}{4} \\-\frac{1}{h} & -\frac{1}{2}\end{bmatrix}}_{\Phi_c}x(k) + \begin{bmatrix}\frac{h^2}{2}\\h\end{bmatrix} l_0 y_{ref}(k) + \begin{bmatrix}\frac{h^2}{2}\\h\end{bmatrix}  v(k)\\
 y(k) &= \begin{bmatrix} 1 & 0 \end{bmatrix} x(k)
 \end{align*}
 ### Verification using symbolic computations

```python
L = sy.Matrix([[sol[l1], sol[l2]]])
Phic = Phi - Gamma*L
Phic
```

$$\left[\begin{matrix}\frac{1}{2} & \frac{h}{4}\\- \frac{1}{h} & - \frac{1}{2}\end{matrix}\right]$$

## Determining the reference signal gain $l_0$
Consider the steady-state solution for a unit step in the reference signal. We set $y_{ref}=1$ and $v = 0$. This gives
$$ x(k+1) = \Phi_c x(k) + \Gamma l_0. $$
In steady-state there is no change in the state, so $x(k+1)=x(k)=x_{ss}$, which leads to
$$ x_{ss} = \Phi_c x_{ss} + \Gamma l_0$$
$$ (I - \Phi_c)x_{ss} = \Gamma l_0$$
\begin{align}
x_{ss} &= (I - \Phi_c)^{-1}\Gamma l_0\\
&= \begin{bmatrix} \frac{1}{2} &-\frac{h}{4}\\ \frac{1}{h} & \frac{3}{2} \end{bmatrix}^{-1} \begin{bmatrix} \frac{h^2}{2}\\h \end{bmatrix} l_0\\
&= \begin{bmatrix}\frac{3}{2} & \frac{h}{4}\\-\frac{1}{h} & \frac{1}{2} \end{bmatrix} \begin{bmatrix} \frac{h^2}{2}\\h\end{bmatrix} l_0\\
&= \begin{bmatrix}\frac{3h^2}{4} + \frac{h^2}{4}\\-\frac{h}{2} + \frac{h}{2} \end{bmatrix}l_0= \begin{bmatrix}h^2\\ 0 \end{bmatrix}l_0\\
\end{align}
which means that the steady-state velocity $\dot{z}(\infty) = x_2(\infty) = 0$. This makes sense.

We can now determine $l_0$. Since  $y(k)=x_1(k)$ then $y_{ss} = h^2 l_0$ for a unit step in the reference signal. We would like the steady-state value $y_{ss}$ to be the same as the reference signal (which is equal to one, of course) so this gives
$$ h^2l_0 = 1 \quad \Rightarrow \quad l_0 = \frac{1}{h^2}. $$

##  Simulate step responses (symbolically)
### Step response from the reference

```python
l0 = 1/(h*h)
C = sy.Matrix([[1,0]])
x = sy.Matrix([[0],[0]]) # Initial state
yref = sy.Matrix([[1]])
xs = [x] # List to hold state trajectory
us = [[0]] # and control signal
ys = [[0]] # and system output
for k in range(6): # No need to simulate too long. It is deadbeat control after all
    us.append(-L*x + l0*yref)
    x = Phic*x + Gamma*l0*yref
    xs.append(x)
    ys.append(C*x)
xs
```

$$\left [ \left[\begin{matrix}0\\0\end{matrix}\right], \quad \left[\begin{matrix}\frac{1}{2}\\\frac{1}{h}\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right], \quad \left[\begin{matrix}1\\0\end{matrix}\right]\right ]$$

```python
us
```

$$\left [ \left [ 0\right ], \quad \left[\begin{matrix}\frac{1}{h^{2}}\end{matrix}\right], \quad \left[\begin{matrix}- \frac{1}{h^{2}}\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right], \quad \left[\begin{matrix}0\end{matrix}\right]\right ]$$

### Step response from the disturbance

```python
x = sy.Matrix([[0],[0]]) # Initial state
yref = sy.Matrix([[0]])
v = sy.Matrix([[1]])
xs = [x] # List to hold state trajectory
us = [[0]] # and control signal
ys = [[0]] # and system output
for k in range(6): # No need to simulate too long. It is deadbeat control after all
    us.append(-L*x + l0*yref)
    x = Phic*x + Gamma*l0*yref + Gamma*v
    xs.append(x)
    ys.append(C*x)
xs
```

$$\left [ \left[\begin{matrix}0\\0\end{matrix}\right], \quad \left[\begin{matrix}\frac{h^{2}}{2}\\h\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right], \quad \left[\begin{matrix}h^{2}\\0\end{matrix}\right]\right ]$$

## Simulate step-responses (numerically)

```python
import control as ctrl
import matplotlib.pyplot as plt
# Convert to from sympy matrices to numpy
hval = .1
Phi_np = np.array(Phi.subs({h:hval})).astype(np.float64)
Gamma_np = np.array(Gamma.subs({h:hval})).astype(np.float64)
L_np = np.array(L.subs({h:hval})).astype(np.float64)
l0_np = np.array(l0.subs({h:hval})).astype(np.float64)
Phic_np = Phi_np - Gamma_np*L_np
C_np = np.array(C).astype(np.float64)
D_np = np.array([[0]])
sys_c = ctrl.ss(Phic_np, Gamma_np*l0_np, C_np, D_np, hval) # From ref signal
sys_cv = ctrl.ss(Phic_np, Gamma_np, C_np, D_np, hval) # From disturbance signal

```

```python
tvec = np.asarray(np.arange(8))*hval
T, yout = ctrl.step_response(sys_c, tvec)
T, yout_v = ctrl.step_response(sys_cv, tvec)
```

```python
plt.figure(figsize=(14,3))
plt.step(tvec, yout.flatten())
```

```python
plt.figure(figsize=(14,3))
plt.step(tvec, yout_v.flatten())
```

# Exercises

## Design a less agressive controller
Consider to let the closed-loop poles be less fast. Choose something reasonable, for instance a double pole in $z=0.5$, or a pair of complex-conjugated poles in $z=0.6 \pm i0.3$. Redo the design, following the example above. Find the state feedback and simulate step-responses.

## Design a deadbeat controller for the DC-motor
From the textbook (Åström & Wittenmark) Appendix:

1. Use symbolic calculations to find the discrete-time state-space model for arbitrary sampling period $h$.
2. Design a deadbeat controller for arbitrary sampling period.
3. Assume a disturbance is acting on the input to the system, as an unknown torque on the motor shaft. This means that the disturbance enters into the system in the same way as the disturbance on the mass on frictionless surface analyzed above. Simulate step-responses for the closed-loop system.

```python

```

state-space/notebooks/Deadbeat controller for the double integrator.ipynb


## Structural Model
$X_1 \sim Ber(.3)$

$X_2 \sim \mathcal{N}(X_1,.1)$

$X_3 = X_2 ^2$

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
np.random.seed(40997)

def f1():
    return np.random.binomial(1, 0.3)

def f2(x1):
    return np.random.normal(loc=x1, scale=.1)

def f3(x2):
    return x2 ** 2

x1 = f1()
x2 = f2(x1)
x3 = f3(x2)

print("x1:{}, x2:{:.3f}, x3:{:.3f}".format(x1,x2,x3))
```

    x1:0, x2:0.012, x3:0.000

```python
#
# Plot Entropy = - p log p
#
# 0 to sqrt(2) so the entire area under the Prob curve is 1
x = np.linspace(1e-10,1,100)
y1 = np.sqrt(2) * x
y2 = -x * np.log(x)
plt.plot(x, y1, 'b', label='y = x*sqrt(2)')
plt.plot(x, y2, 'r', label='Entropy: -p log p')
plt.title('Entropy: -p log p')
plt.xticks([np.min(x), np.exp(-1), np.average(x), np.max(x)]) #max at 1/e
plt.legend()
```

## Causal Graphical Models
pip install causalgraphicalmodels

```python
from causalgraphicalmodels import CausalGraphicalModel
sprinkler = CausalGraphicalModel(
    nodes=['season','rain',  'sprinkler', 'wet', 'slippery'],
    edges=[
        ('season', 'rain'),
        ('season', 'sprinkler'),
        ('rain', 'wet'),
        ('sprinkler', 'wet'),
        ('wet', 'slippery')
    ]
)
sprinkler.draw()

```

```python
print(sprinkler.get_distribution())
```

    P(season)P(rain|season)P(sprinkler|season)P(wet|rain,sprinkler)P(slippery|wet)

```python
sprinkler_do = sprinkler.do('rain')
print(sprinkler_do.get_distribution())
sprinkler_do.draw()
```

    P(season)P(sprinkler|season)P(wet|do(rain),sprinkler)P(slippery|wet)

### Causal Inference with Causal Graphical Models
Can we make inferences from observational data? Consider, the following confounder setup where $Z$

confounds the relationship between $X$ and $Y$. If we intervene on $X$, then we get the DAG

farther below and then that decouples $Z$ and $X$.

```python
confounder = CausalGraphicalModel(
    nodes=['X','Y','Z'],
    edges=[
        ('X', 'Y'),
        ('Z', 'Y'),
        ('Z', 'X'),
    ]
)
confounder.draw()
```

```python
confounder_do = confounder.do('X')
confounder_do.draw()
```

```latex
%%latex

\begin{align}
P(Y|do(X)) &= \sum_Z P(Y|do(X),Z)P(Z|do(X))  ~~~~~~ \text{axioms of prob} \\
    &= \sum_Z P(Y|do(X),Z)P(Z) ~~~~~~ \text{$Z$ is independent of $do(X)$} \\
    &= \sum_Z P(Y|X, Z)P(Z) ~~~~~~ \text{stability assumption, that $Y$ behaves the same given its parents} \\
    \\
P(Y|do(X)) &= \sum_Z P(Y|X, Z)P(Z) ~~~~~~ \textbf{Adjustment Formula / g-formula}\\
\end{align}
Compare the above to the conditional on the observed data

$P(Y|X) = \sum_Z P(Y|X, Z)P(Z|X) $

```

\begin{align}
P(Y|do(X)) &= \sum_Z P(Y|do(X),Z)P(Z|do(X))  ~~~~~~ \text{axioms of prob} \\
    &= \sum_Z P(Y|do(X),Z)P(Z) ~~~~~~ \text{$Z$ is independent of $do(X)$} \\
    &= \sum_Z P(Y|X, Z)P(Z) ~~~~~~ \text{stability assumption, that $Y$ behaves the same given its parents} \\
    \\
P(Y|do(X)) &= \sum_Z P(Y|X, Z)P(Z) ~~~~~~ \textbf{Adjustment Formula / g-formula}\\
\end{align}
Compare the above to the conditional on the observed data

$P(Y|X) = \sum_Z P(Y|X, Z)P(Z|X) $

## Adjustment Formula
When is the adjustment formula true? It is true only when the following apply:

$Z$ blocks all backdoor paths between $X$ and $Y$

$Z$ does not contain any descendants of $X$

```python
from causalgraphicalmodels.examples import big_csm
example_cgm = big_csm.cgm
example_cgm.draw()
```

```python
# get all backdoor adjustment sets
example_cgm.get_all_backdoor_adjustment_sets('x','y')
```

    frozenset({frozenset({'d', 'e'}),
               frozenset({'a', 'e'}),
               frozenset({'a', 'd', 'e'}),
               frozenset({'b'}),
               frozenset({'b', 'd'}),
               frozenset({'a', 'b'}),
               frozenset({'d'}),
               frozenset({'a', 'd'}),
               frozenset({'a', 'b', 'd'}),
               frozenset({'e'}),
               frozenset({'b', 'e'}),
               frozenset({'a', 'b', 'e'}),
               frozenset({'b', 'd', 'e'}),
               frozenset({'a', 'b', 'd', 'e'})})

```python
big_csm.sample(5)
```

      b
      d
      e
      a
      x
      c
      y
      f
      h

      0
      -0.779979
      0.592671
      2.108259
      -0.211792
      0
      -0.234952
      -3.149868
      -1.834548
      -4.576666

      1
      -1.058772
      1.024925
      0.622312
      -0.221537
      0
      0.057270
      0.867680
      -0.026985
      0.124077

      2
      -0.110376
      -0.042870
      -0.455473
      -0.115910
      0
      -0.820117
      -2.491658
      -1.045292
      -4.123815

      3
      -0.630953
      -0.198309
      -0.737376
      -1.339891
      0
      -0.168683
      1.148839
      1.531666
      -0.113968

      4
      -0.532262
      0.367744
      2.193800
      -1.814282
      1
      1.729301
      3.186595
      3.141783
      0.772651

```python
# Estimate ATE from this observational data
from causalinference import CausalModel
def estimate_ate(dataset, adjustment_set=None, method='matching'):
    if adjustment_set is None:
        y0 = dataset.loc[lambda df: df.x==0].y.mean()
        y1 = dataset.loc[lambda df: df.x==1].y.mean()

        y0_var = dataset.loc[lambda df: df.x==0].y.var()
        y1_var = dataset.loc[lambda df: df.x==1].y.var()

        n_y0 = dataset.loc[lambda df: df.x==0].shape[0]
        n_y1 = dataset.loc[lambda df: df.x==1].shape[0]
        return {
            'ate': y1 - y0,
            'ate_se': 2 * np.sqrt(y1_var/n_y1 + y0_var/n_y0)
        }

    cm = CausalModel(
        Y=dataset.y.values, #outcome
        D=dataset.x.values, #treatment
        X=dataset[adjustment_set].values #confounders
    )
    cm.est_via_ols()
    cm.est_via_matching()

    cm.est_propensity_s()
    cm.est_via_weighting()

    cm.stratify_s()
    cm.est_via_blocking()

    return {
        'ate': cm.estimates[method]['ate'],
        'ate_se': cm.estimates[method]['ate_se']
    }

```

```python
estimate_ate(big_csm.sample(10000)) # not real ATE because it is observational
```

    {'ate': 4.484538064771581, 'ate_se': 0.17055645606035957}

```python
nsamples = 10000
ds = big_csm.sample(nsamples)
#
# Generate samples from interventional distribution
# Values of X are assigned randomly as in A/B test
#
ds_intervention = (
    big_csm.do('x')
    .sample(
        10000,
        set_values={'x': np.random.binomial(n=1, size=10000, p=0.5)}
    )
)
true_ate = estimate_ate(ds_intervention)
print(true_ate)

#
# Generate results for a few different adjustment sets
#
results = {
    'no_adjustment': estimate_ate(ds),
    'adjustment_b': estimate_ate(ds, ['b']),
    'adjustment_bde': estimate_ate(ds, ['b','d','e']),
    'adjustment_bh': estimate_ate(ds, ['b', 'h']),
    'adjustment_bc': estimate_ate(ds, ['b','c']),
    'adjustment_all': estimate_ate(ds, ['a','b','c','d','e','f','h']),
}
print(results)
```

    {'ate': 2.9813784652161837, 'ate_se': 0.17391798378482043}
    {'no_adjustment': {'ate': 4.459599599419482, 'ate_se': 0.17303193947682885}, 'adjustment_b': {'ate': 2.8848192161447574, 'ate_se': 0.14373454279146847}, 'adjustment_bde': {'ate': 2.9963612366576262, 'ate_se': 0.10917635552681271}, 'adjustment_bh': {'ate': 1.5821370929380225, 'ate_se': 0.056654666704890423}, 'adjustment_bc': {'ate': -0.045882294748157797, 'ate_se': 0.10655645525591194}, 'adjustment_all': {'ate': 0.17598153754726498, 'ate_se': 0.038851708994256287}}

```python
x_label = list(results.keys())
x = np.arange(len(x_label))
y, yerr = [], []
for l in x_label:
    if results[l] is not None:
        y += results[l]['ate'],
        yerr += results[l]['ate_se'],

print(x,y,yerr)
plt.figure(figsize=(10,8))
plt.errorbar(x=x,y=y,yerr=yerr, linestyle='none', capsize=5, marker='o')
plt.xticks(x, x_label, rotation=45,fontsize=10)
plt.title('Estimated ATE', color='g', fontsize=18)
xmin, xmax = plt.xlim()
plt.hlines(true_ate['ate'], xmin, xmax, linestyle='dashed')
```

```python

```


[Decode Ways](https://leetcode.com/problems/decode-ways/)。**2019哔哩哔哩笔试题**。设$A-Z$26个字母对应$[1,26]$个数字，给一串数字，求该串数字能表示成多少种字符串。

思路：设置$dp$数组，其中$dp[i]$表示长度为$i$的数字部分能表示多少种字符串。易得$dp[0]=1$，即空数字表示空字串。然后推状态转移方程，新增一个数字$x$，若$x\in{[1,9]}$，则$dp[i]$至少要等于$dp[i-1]$，因为$x$可以独立表示成一个字母；若$x$跟之前一个数字的范围在$[10,26]$内，说明到达$dp[i]$就有两条路径：$dp[i]=dp[i-1]+dp[i-2]$。注意这里还忽略了一个特殊情况，当$x=0$时，其自身是无法表示字母的，只能跟之前的数字合成$10$或$20$来表示字母。综上，状态转移方程为：
$$
\begin{align}
dp[i]=
\begin{cases}
dp[i-1] & ,s[i-1]\in{[1,9]} \\
dp[i-2] & ,s[i-2:i]=10\ or \ s[i-2:i]=20 \\
dp[i-1]+dp[i-2] & ,s[i-2:i]\in{[11,19]\cup[21,26]} \\
\end{cases}
\end{align}
$$

```python
def numDecodings(self, s: str) -> int:
    n = len(s)
    dp = [0]*(n+1)
    dp[0] = 1

    if s[0] == '0':
        return 0

    dp[1] = 1

    for i in range(2, n+1):
        if int(s[i-2:i]) in (10, 20):
            dp[i] = dp[i-2]
        elif int(s[i-2:i]) in range(11, 27):
            dp[i] = dp[i-1]+dp[i-2]
        elif int(s[i-1]) in range(1, 10):
            dp[i] = dp[i-1]

    return dp[-1]
```

    3

[House Robber II](https://leetcode.com/problems/house-robber-ii/)。同上题，不过现在房子围成了一个圈，此时给出的数组首位相等。

思路：现在首尾相连，那么首尾就成了相邻的情况。直接分成两种情况来分析，有首无尾和无首有尾。，对这两种情况，解法与上题一致。

```python
def rob(nums) -> int:
    if not nums:
        return 0
    if len(nums) < 3:
        return max(nums)

    n = len(nums)
    nums_1, nums_2 = nums[1:], nums[:-1]
    dp_1, dp_2 = [0]*(n-1), [0]*(n-1)
    dp_1[0], dp_2[0] = nums_1[0], nums_2[0]
    dp_1[1], dp_2[1] = max(nums_1[0], nums_1[1]), max(nums_2[0], nums_2[1])

    for i in range(2, n-1):
        dp_1[i] = max(dp_1[i-1], dp_1[i-2]+nums_1[i])
        dp_2[i] = max(dp_2[i-1], dp_2[i-2]+nums_2[i])

    return max(dp_1[-1], dp_2[-1])
```

    3

[House Robber III](https://leetcode.com/problems/house-robber-iii/)。房屋呈二叉树结构，小偷不能连续偷有边相连的房屋，求小偷能偷得的最多财富。

思路：树形DP。每一层有两种选择，偷或不偷，所以需要设置两个dp数组，设$dp_{do}[cur\_node]$表示偷当前节点所能获得的最大财富，$dp_{dont}[cur\_node]$表示不偷当前节点所能获得的最大财富，则态转移方程为：
$$
\begin{align}
dp_{do}[cur\_node]&=cur\_node.val+dp_{dont}[left]+dp_{dont}[right] \\
dp_{dont}[cur\_node]&=max(dp_{do}[left],dp_{dont}[left])+max(dp_{do}[right],dp_{dont}[right]) \\
\end{align}
$$
$dp$数组由底往上更新，可使用递归来实现，最终返回$max(dp_{do}[root],dp_{dont}[root])$。

```python
def rob(root: TreeNode) -> int:
    def dp_rec(node):
        '''
        return (dp_do,dp_dont)
        '''
        if not node:
            return (0, 0)
        if not node.left and not node.right:
            return (node.val, 0)

        do_left, dont_left = dp_rec(node.left)
        do_right, do_right = dp_rec(node.right)

        return (node.val+dont_left+dont_right,
                max(do_left, dont_left)+max(do_right, dont_right))

    return dp_rec(root)
```

[All Possible Full Binary Trees](https://leetcode.com/problems/all-possible-full-binary-trees/)。给$N$个节点，求所有可能的满二叉树结构。

思路：满二叉树的节点数肯定是奇数，如果$N$是偶数直接返回空。设立dp数组，其中$dp[i]$存储$i$个节点下所有可能的满二叉树。新增一个节点时，可把新增的节点作为根节点，然后遍历所有可能的奇数作为左分支的节点数，剩余的节点作为右分支的节点数，分别在$dp$数组中取出所有可能的结构拼接到根节点上。

```python
def allPossibleFBT(N: int):
    if N % 2 == 0:
        return list()

    dp = [list() for _ in range(N+1)]
    dp[1].append(TreeNode(0))

    for node_cnt in range(3, N+1, 2):    # 总结点数
        for left_cnt in range(1, node_cnt-1, 2):    # 左分支节点数
            right_cnt = node_cnt-left_cnt-1    # 右分支节点数

            for left in dp[left_cnt]:
                for right in dp[right_cnt]:
                    root = TreeNode(0)
                    root.left = left
                    root.right = right
                    dp[node_cnt].append(root)

    return dp[-1]
```

[Triangle](https://leetcode.com/problems/triangle/)。给一个三角形数组，求一个自顶向下路径的最小和。
```
   [2],
  [3,4],
 [6,5,7],
[4,1,8,3]
```

思路：很明显的dp问题。若自顶向下计算，边界位置与非边界位置需要分开考虑；自底向上计算，很简单。开一个形状与原数据相同的异构嵌套数组，显然初始状态为$dp[-1]=nums[-1]$，状态转移方程为：

$$
dp[row][col]=min(dp[row+1][col],dp[row+1][col+1])+nums[row][col]
$$

```python
def minimumTotal(triangle) -> int:
    n = len(triangle)
    dp = [[0]*row for row in range(1, n+1)]
    dp[-1] = triangle[-1]

    for row in range(n-2, -1, -1):
        for col in range(row+1):
            dp[row][col] = min(dp[row+1][col],
                               dp[row+1][col+1])+triangle[row][col]

    return dp[0][0]
```

    11

排名问题(2019疯狂游戏笔试题)。有$n$个人，对其进行排名次，允许并列同一名，总共有多少种排序方案。

思路：设立二维dp数组，其中$dp[i][j]$表示将$i$个人排成$j$种名词的排序方案，那么初始dp的第一行与第一列全是$1$，对角线上(没有并列名次)是一个全排列，难点在于状态转移方程。增加一个人且构成$j$种名次的话，那么之前的状态只可能有两种：一是之前的$i-1$个人构成$j$种名次，然后第$i$个人与之前的人并列一个名次，没有增加名次的种类；二是之前$i-1$个人构成$j-1$种名次，第$i$个人独占一位，增加一种名次。得：
$$
\begin{align}
dp[i][j]=j*dp[i-1][j]+j*dp[i-1][j-1]
\end{align}
$$

```python
def ranking(n: int) -> int:
    if n < 0:
        return -1

    dp = [[0] * n for _ in range(n)]
    for row in range(n):
        dp[row][0] = 1
    for col in range(n):
        dp[0][col] = 1
    for idx in range(1, n):
        dp[idx][idx] = dp[idx-1][idx-1]*(idx + 1)

    for row in range(1, n):
        for col in range(1, n):
            if row == col:
                dp[row][col] = dp[row-1][col-1]*(row+1)
            else:
                dp[row][col] = (col + 1) * dp[row - 1][col] + \
                    (col + 1) * dp[row - 1][col - 1]

    res = 0
    for col in range(n):
        res += dp[-1][col]

    return res
```

[Maximal Square](https://www.cnblogs.com/grandyang/p/4550604.html)。**百度2019提前批手撕代码题**。给一个含$0$和$1$的矩阵，求其中$1$构成的最大方阵的面积。

思路：设立二维dp数组，其中$dp[i][j]$表示以$(i,j)$为右下角且包含$(i,j)$位置时的最大全$1$方阵的边长。初始状态，第一行与第一列等于原矩阵的值，然后看状态转移方程。首先只有当$(i,j)$本身为$1$时才有可能构成全$1$方阵，然后看该位置的左边、上边跟左上，包含当前位置时所能构成的最大方阵由其最小值决定，所以有：
$$
\begin{align}
dp[i][j]=
\begin{cases}
0 & ,A[i][j]=0 \\
\min(dp[i-1][j],dp[i][j-1],dp[i-1][j-1])+1 & ,A[i][j]=1 \\
\end{cases}
\end{align}
$$

```python
def maximalSquare(matrix) -> int:
    res = 0
    if not matrix or not matrix[0]:
        return 0

    rows, cols = len(matrix), len(matrix[0])
    dp = [[0]*cols for _ in range(rows)]
    res = 0

    for row in range(rows):
        dp[row][0] = int(matrix[row][0])
        res = max(res, dp[row][0])
    for col in range(cols):
        dp[0][col] = int(matrix[0][col])
        res = max(res, dp[0][col])

    for row in range(1, rows):
        for col in range(1, cols):
            if matrix[row][col] == '1':
                dp[row][col] = min(dp[row-1][col],
                                   dp[row][col-1],
                                   dp[row-1][col-1])+1
                res = max(res, dp[row][col])

    return res*res
```

[Arithmetic Slices](https://leetcode.com/problems/arithmetic-slices/)。给一数组，求其中等差数列的个数。要求等差数列的长度至少为$3$。

思路：设立dp数组，其中$dp[i]$表示以$i$结尾且包含$nums[i]$的数字序列能构成多少个等差数列。当新增一个数时，若新数与之前两个数等差，则以$i+1$结尾且包含$nums[i+1]$的等差数列就有$dp[i]+1$个。这里的状态转移方程是个难点，需要举个简单例子在纸上列一下。那么总共的等差数列就是对dp数组求和。

```python
def numberOfArithmeticSlices(A) -> int:
    n = len(A)
    if n < 3:
        return 0
    dp = [0]*n

    for idx in range(2, n):
        if A[idx]-A[idx-1] == A[idx-1]-A[idx-2]:
            dp[idx] = dp[idx-1]+1

    return sum(dp)
```

    3

[Unique Substrings in Wraparound String](https://leetcode.com/problems/unique-substrings-in-wraparound-string/)。给一无限循环的顺序字母序列$abc...xyzabc...$，再给一字串$p$，求字串$p$中有多少子串是$s$的子串。

思路：该题类似于求等差数列，但是数字变成了字符，需要考虑重复的情况，题目中的$s$其实可以忽略。设立大小为$26$的$dp$数组，其中$dp[i]$表示以字符$i$结尾且包含字符$i$的等差字串有多少种。在等差数列那一题中知道，每增加一位，那么种类也是只增加一个的。

```python
def findSubstringInWraproundString(p: str) -> int:
    if not p:
        return 0

    n = len(p)
    dp = [0]*26
    dp[ord(p[0])-97] = 1

    cur_len = 1
    for idx in range(1, n):
        if ord(p[idx])-ord(p[idx-1]) == 1 or ord(p[idx-1])-ord(p[idx]) == 25:
            cur_len += 1
        else:
            cur_len = 1
        dp[ord(p[idx])-97] = max(dp[ord(p[idx])-97], cur_len)

    return sum(dp)
```

    6

[Partition Equal Subset Sum](https://leetcode.com/problems/partition-equal-subset-sum/)。给一非空非负数组，判断该数组能否分成和相等的两部分。

思路：01背包的变种。将每个数字看作一个物品，每个物品重量为自身的值，该题需要求能否装满一个负重量为$sum(nums)/2$的背包。

```python
def canPartition(nums) -> bool:
    arr_sum = sum(nums)
    if arr_sum & 1:
        return False

    n = len(nums)
    target_sum = arr_sum//2

    dp = [[False]*(target_sum+1) for _ in range(n+1)]
    for row in range(n+1):
        dp[row][0] = True

    for i in range(1, n+1):
        for j in range(1, target_sum+1):
            dp[i][j] = dp[i-1][j]    # 不拿的情况

            if j >= nums[i-1]:    # 拿取第$i$个数字
                dp[i][j] = dp[i][j] or dp[i-1][j -nums[i-1]]    # 不拿或拿，只要有一种情况满足即满足

    return dp[-1][-1]
```

    True

[Out of Boundary Paths](https://leetcode.com/problems/out-of-boundary-paths/)。给一个$m$行$n$列的矩阵，有一个球在$[i,j]$处，在最多走$N$步的条件下，把这个球移动到矩阵外面有多少条可走的路径。

思路：带时间状态的DP问题，时间维度为$N$。设立三维$dp$数组，其中$dp[t][x][y]$表示走$t$步的情况下从$[x,y]$坐标走出去的解法数。那么状态转移方程为：
$$
dp[t][x][y]=dp[t-1][x-1][y]+dp[t-1][x+1][y]+dp[t-1][x][y-1]+dp[t-1][x][y+1]
$$
分别代表走$t-1$步时上下左右四个格子的解法。

```python
def findPaths(m: int, n: int, N: int, i: int, j: int) -> int:
    dp = [[[0]*n for _ in range(m)] for _ in range(N+1)]

    for z in range(1, N+1):
        for x in range(m):
            for y in range(n):
                up = 1 if x == 0 else dp[z-1][x-1][y]
                down = 1 if x == m-1 else dp[z-1][x+1][y]
                left = 1 if y == 0 else dp[z-1][x][y-1]
                right = 1 if y == n-1 else dp[z-1][x][y+1]
                dp[z][x][y] = up+down+left+right

    return dp[-1][i][j] % 1000000007
```

    6

[Delete Operation for Two Strings](https://leetcode.com/problems/delete-operation-for-two-strings/)。给两字符串，问最少需要做多少次删除操作才能使得两字符串相等。

思路：该题是求两字符串最长公共子序列的反向题目。设一个二维数组$dp$，其中$dp[i][j]$表示字串$1$的前$i$位与字串$2$的前$j$位的最长公共子序列长度，易得$dp[0][0]=0$。不难得出状态转移方程为：

$$
dp[i][j]=
\begin{cases}
dp[i-1][j-1]+1 & str_{1}[i1]=str_{2}[j1] \\
max(dp[i][j-1],dp[i-1][j]) & str_{1}[i1]{\ne}str_{2}[j1] \\
\end{cases}
$$

原理是这样：若两字符串的某一位置相等，则LCS长度可以直接加一；若不相等，则需要取错位比较下的最大长度。

```python
def minDistance(word1: str, word2: str) -> int:
    len_1, len_2 = len(word1), len(word2)
    dp = [[0]*(len_2+1) for _ in range(len_1+1)]

    for i in range(1, len_1+1):
        for j in range(1, len_2+1):
            if word1[i-1] == word2[j-1]:    # 注意dp索引与字串索引大1
                dp[i][j] = dp[i-1][j-1]+1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    return len_1+len_2-2*dp[-1][-1]
```

    2

[Longest Palindromic Subsequence](https://leetcode.com/problems/longest-palindromic-subsequence/)。给一字符串，求其最长回文子序列。

思路：找回文的方法就是从中心往两边扩展。因为子序列不要求连续，所以使用dp的方法来解。首先$i$表示当前字串的末尾位置，$i$是从前往后扫描的；令$j$表示当前字串的开始位置，$j$是从$i-1$往前扫描的。设一个二维数组$dp$，其中$dp[i][j]$表示$[j,i]$区间内的最长回文子序列的长度，那么易得有$0\le{j\le{i}}$，所以只使用dp数组的下三角。扩展即可得到状态转移方程：

$$
dp[i][j]=
\begin{cases}
dp[i-1][j+1]+2 & s[i]=s[j] \\
max(dp[i-1][j],dp[i][j+1]) & s[i]\ne{s[j]} \\
\end{cases}
$$

注意最后返回的答案是$dp[-1][0]$。

```python
def longestPalindromeSubseq(s: str) -> int:
    n = len(s)
    dp = [[0]*n for _ in range(n)]
    for i in range(n):
        dp[i][i] = 1

    for i in range(n):
        for j in range(i-1, -1, -1):    # j由后往前移
            if s[i] == s[j]:
                dp[i][j] = dp[i-1][j+1]+2
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j+1])

    return dp[-1][0]
```

    2

[Unique Paths II](https://leetcode.com/problems/unique-paths-ii/)。给一$01$矩阵，$0$代表可通行而$1$表示障碍物。从左上角出发到达右下角，问有几条路径。

思路：设立二维$dp$数组，其中$dp[i][j]$表示到达$(i,j)$坐标有几条路径。初始状态就是首行首列，首行首列障碍物后面的坐标都是不可达的，需要特殊处理。而递推式就很简单了：
$$
dp[i][j]=dp[i-1][j]+dp[i][j-1]
$$

```python
def uniquePathsWithObstacles(obstacleGrid) -> int:
    if obstacleGrid[0][0] == 1:
        return 0

    rows, cols = len(obstacleGrid), len(obstacleGrid[0])
    dp = [[0]*cols for _ in range(rows)]

    dp[0][0] = 1
    for row in range(1, rows):
        dp[row][0] = dp[row-1][0] if obstacleGrid[row][0] == 0 else 0
    for col in range(1, cols):
        dp[0][col] = dp[0][col-1] if obstacleGrid[0][col] == 0 else 0

    for row in range(1, rows):
        for col in range(1, cols):
            dp[row][col] = dp[row-1][col]+dp[row][col - 1] \
                if obstacleGrid[row][col] == 0 else 0

    return dp[-1][-1]
```

    2

[拼凑硬币](https://www.nowcoder.com/questionTerminal/2479839aa61e44f39aa3268160650e17)。现有$2^{i}$面值的硬币各2个，指定一个值，求出使用这些硬币拼凑的方案数。

思路：设立$dp$数组，其中$dp[i]$表示拼凑$i$有多少种方案。易得初始状态为$dp[0]=dp[1]=1$，状态转移方程是难点。因为每种面值硬币都有两个，且考虑到重复值如$(1,2)$跟$(2,1)$。
$$
\begin{align}
dp[i]=
\begin{cases}
& dp[i/2] , i为奇数\\
& dp[(i-1)/2]+dp[(i-1)/2-1] , i为偶数\\
\end{cases}
\end{align}
$$
因为测试用例有大数，而DP需要计算所有的方案，为省内存使用带```mem```数组的递归。

```python
def coin(n):
    mem = {0: 1, 1: 1}

    def rec(i):
        if i in mem.keys():
            return mem[i]
        elif i & 1 == 1:    # 奇数
            mem[i] = rec(i >> 1)
        else:    # 偶数
            mem[i] = rec(i >> 1)+rec((i >> 1)-1)
        return mem[i]

    return rec(n)
```

[骰子游戏](https://www.nowcoder.com/practice/0e83797c34e54cca91179fe9ad681bc4?tpId=90&&tqId=30849&rp=1&ru=/activity/oj&qru=/ta/2018test/question-ranking)。掷一个骰子$n$次，求累计点数和大于等于$x$的概率。

思路：该题关键是求$n$个骰子点数的分布，设二维$dp$矩阵，其中$dp[i][j]$表示$i$个骰子得到$j$点的期望频率。易得状态转移方程为：
$$
\begin{align}
dp[i][j]&=dp[i-1][j-1]+dp[i-1][j-2]+dp[i-1][j-3] \\
&+dp[i-1][j-4]+dp[i-1][j-5]+dp[i-1][j-6] \\
\end{align}
$$

```python
def func(n, x):
    dp = [[0] * (6 * n + 1) for _ in range(n + 1)]
    for i in range(1, 6 + 1):
        dp[1][i] = 1

    for i in range(2, n + 1):
        for j in range(i, 6 * i + 1):
            if j > 1:
                dp[i][j] += dp[i - 1][j - 1]
            if j > 2:
                dp[i][j] += dp[i - 1][j - 2]
            if j > 3:
                dp[i][j] += dp[i - 1][j - 3]
            if j > 4:
                dp[i][j] += dp[i - 1][j - 4]
            if j > 5:
                dp[i][j] += dp[i - 1][j - 5]
            if j > 6:
                dp[i][j] += dp[i - 1][j - 6]

    return sum(dp[-1][x:])/sum(dp[-1])
```

[歌单方案](https://www.nowcoder.com/questionTerminal/f3ab6fe72af34b71a2fd1d83304cbbb3?orderByHotValue=1&page=1&onlyReference=false)。现有$X$首$A$分钟的歌和$Y$首$B$分钟的歌，要组成一个总长$K$分钟的歌单有几种方案？输出模$1000000007$的结果。

思路：$01$背包方案数问题。现有$X+Y$件物品，一部分重量为$A$而另一部分重量为$B$，拿满一个负重量为$K$的背包有多少种拿法。

```python
def func(K, A, X, B, Y):
    dp = [[0] * (K + 1) for _ in range(X + Y + 1)]
    w = [0] * (X + Y + 1)
    for i in range(1, X + 1):
        w[i] = A
    for i in range(X + 1, X + Y + 1):
        w[i] = B
    dp[0][0] = 1

    for i in range(1, X + Y + 1):
        for j in range(0, K + 1):
            dp[i][j] = dp[i - 1][j]
            if j >= w[i]:
                dp[i][j] += dp[i - 1][j - w[i]]
                dp[i][j] %= 1000000007

    return dp[-1][-1]
```

2019贝壳笔试题。在你面前依次排着n棵树，而你有两种工具，锯子和斧头，它们砍第$i$棵树的时间分别为$a[i]$和$b[i]$，一开始你拿的是斧头，而砍第$i$棵树前交换工具需要花费$c[i]$的时间，问题来了，依次砍完这些树的时间最短为多少呢？

思路：双状态DP问题，设立两个dp数组$dp_{a}$和$dp_{b}$，其中$dp_{a}[i]$表示用锯子砍第$i$颗树的最小时间点，而$dp_{b}[i]$表示用斧头砍第$i$颗树的最小时间点。因为最开始拿的是斧子，所以初始状态为$dp_{a}[0]=a[0]+c[0]$，$dp_{b}[0]=b[0]$，状态转移方程为：
$$
\begin{align}
dp_{a}[i]&=\min(dp_{a}[i - 1], dp_{b}[i - 1] + c[i]) + a[i] \\
dp_{b}[i]&=\min(dp_{b}[i - 1], dp_{a}[i - 1] + c[i]) + b[i]
\end{align}
$$

```python
import sys

n = int(sys.stdin.readline().strip())
a, b, c = [0] * n, [0] * n, [0] * n

for i in range(n):
    a[i], b[i], c[i] = list(map(int, sys.stdin.readline().strip().split()))

def func(a, b, c):
    dp_a, dp_b = [0] * n, [0] * n

    dp_a[0], dp_b[0] = a[0]+c[0], b[0]

    for i in range(1, n):
        dp_a[i] = min(dp_a[i - 1], dp_b[i - 1] + c[i]) + a[i]
        dp_b[i] = min(dp_b[i - 1], dp_a[i - 1] + c[i]) + b[i]

    return min(dp_a[-1], dp_b[-1])

print(func(a, b, c))
```

[Trapping Rain Water](https://leetcode.com/problems/trapping-rain-water/)。给一非负整数数组，每个位置上的数字代表该位置的陆地高度，求该地势在雨天的最大蓄水量。

思路：木桶原理，一个坑的最大蓄水量取决于其左右挡板中最低那一块的高度。维护两个数组$left$和$right$，其中$left[i]$记录$i$位置左边的最高挡板高度，$right[i]$记录$i$位置右边的最高挡板高度。当前位置的最大蓄水量为：

$$
min(left[i],right[i])-height[i]
$$

```python
def trap(height) -> int:
    n = len(height)
    res = 0
    if n < 3:
        return res

    left, right = [0]*n, [0]*n

    left[0] = height[0]
    for i in range(1, n):
        left[i] = max(left[i-1], height[i])

    right[-1] = height[-1]
    for i in range(n-2, 0, -1):
        right[i] = max(right[i+1], height[i])

    for i in range(1, n-1):
        res += min(left[i], right[i])-height[i]

    return res
```

    6

[Increasing Triplet Subsequence](https://leetcode.com/problems/increasing-triplet-subsequence/)。判断数组中是否存在一个递增的三元组(不要求连续)。

思路：换句话说，去除首尾元素，要在中间元素中找这样一个位置，其左边有小于它的值，其右边有大于它的值。设立两个$dp$数组，```left_min```保存左边的最小值，```right_max```保存右边的最大值。

```python
def increasingTriplet(nums) -> bool:
    n = len(nums)
    left_min, right_max = nums[:], nums[:]

    for i in range(1, n):
        left_min[i] = min(left_min[i-1], nums[i])
    for i in range(n-2, -1, -1):
        right_max[i] = max(right_max[i+1], nums[i])

    for i in range(1, n-1):
        if left_min[i] < nums[i] < right_max[i]:
            return True

    return False
```

    False

[Unique Binary Search Trees](https://leetcode.com/problems/unique-binary-search-trees/)。给$n$个不同的数字，每个数字形成一个节点，求这些节点能构成多少种不同的BST。

思路：设立$dp$数组，其中$dp[i]$表示$i$个节点所能表示的二叉树的数量。那么对于一个具有$k$个节点，左子树有$i$个节点的二叉树，就有$dp[i]*dp[k-i-1]$种。

```python
def numTrees(n: int) -> int:
    dp = [0]*(n+1)
    dp[0] = 1
    dp[1] = 1

    for k in range(2, n+1):
        for i in range(k):
            dp[k] += dp[i]*dp[k-i-1]
    return dp[-1]
```

    5

[Valid Permutations for DI Sequence](https://leetcode.com/problems/valid-permutations-for-di-sequence/)。给一长度为$n$且只含'D'和'I'的字串，其中'D'表示递减而'I'表示递增。求有多少个长度为$n+1$的自然数排列满足字串限定的大小关系。

思路：**2019爱奇艺笔试题**。设立二维$dp$数组，其中$dp[i][j]$表示长度为$i$且以数字$j$结尾的答案数，易得$j<i$。状态转移方程的推导是难点。假设最后一位要求升序$I$，那么长度为$i$且以数字$j$结尾，前一位必须小于$j$，即：

$$
dp[i][j]=dp[i-1][0]+...+dp[i-1][j-1]
$$

假设最后一位要求降序$D$，同理，前一状态的最后一位都必须大于等于$j$(难点，建议看[此视频](https://www.youtube.com/watch?v=_uqusS_6rvw)了解)，即：

$$
dp[i][j]=dp[i-1][j]+...+dp[i-1][i-1]
$$

```python
def numPermsDISequence(S: str) -> int:
    n = len(S) + 1
    mod = 1e9 + 7
    dp = [[0] * (n + 1) for _ in range(n + 1)]
    dp[1][0] = 1
    for i in range(2, n + 1):
        for j in range(i):
            if S[i - 2] == 'I':
                dp[i][j] = int(sum(dp[i - 1][:j]) % mod)
            else:
                dp[i][j] = int(sum(dp[i - 1][j:]) % mod)

    return int(sum(dp[-1]) % mod)
```


## Hierarchical Regression

Hierarchical models enable sharing of features among groups. The parameters of the model are assumed to be sampled from a common distribution that models similarity between groups. The figure below shows three different scenarios that illustrate the benefit of hierarchical modeling. In the figure on the left, we have a single set of parameters $\theta$ that model the entire sequence of observations referred to as a pooled model. Here any variation in data is not modelled explicitly since we are assuming a common set of parameters that give rise to the data. On the other hand, we have an unpooled scenario where we model a different set of parameters for each observation. In the unpooled case, we are assuming that there is no sharing of parameters between groups of observations and that each parameter is independent. The hierarchical model combines the best of both worlds: it assumes that there's a common distribution from which individual parameters are sampled and therefore captures similarities between groups.

In Bayesian Hierarchical Regression, we can assign priors on model parameters and use MCMC sampling to infer posterior distribution. We use the radon dataset to regress radon gas levels in houses of different counties based on the floor number (in particular if there's a basement or not). Thus our regression model will look like the following:
\begin{equation}
    \alpha_c \sim N(\mu_a, \sigma_{a}^{2})\\
    \beta_c \sim N(\mu_{\beta}, \sigma_{\beta}^{2})\\
    \mathrm{radon}_c = \alpha_c + \beta_c \times \mathrm{floor}_{i,c} + \epsilon_c
\end{equation}

Notice that subscript $c$ indicates a county, thus we are learning an intercept and a slope for each county sampled from a shared Gaussian distribution. Thus, we are assuming a hierarchical model in which our parameters ($\alpha_c$ and $\beta_c$) are sampled from a common distribution. Having specified the model, we can implement it using PyMC3.

```python
%matplotlib inline

import numpy as np
import pandas as pd

import seaborn as sns
import matplotlib.pyplot as plt

import pymc3 as pm
```

```python
#load data
data = pd.read_csv('./radon.txt')
data.head()
```

      Unnamed: 0
      idnum
      state
      state2
      stfips
      zip
      region
      typebldg
      floor
      room
      ...
      pcterr
      adjwt
      dupflag
      zipflag
      cntyfips
      county
      fips
      Uppm
      county_code
      log_radon

      0
      0
      5081.0
      MN
      MN
      27.0
      55735
      5.0
      1.0
      1.0
      3.0
      ...
      9.7
      1146.499190
      1.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      1
      1
      5082.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      ...
      14.5
      471.366223
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      2
      2
      5083.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      ...
      9.6
      433.316718
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      1.098612

      3
      3
      5084.0
      MN
      MN
      27.0
      56469
      5.0
      1.0
      0.0
      4.0
      ...
      24.3
      461.623670
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.095310

      4
      4
      5085.0
      MN
      MN
      27.0
      55011
      3.0
      1.0
      0.0
      4.0
      ...
      13.8
      433.316718
      0.0
      0.0
      3.0
      ANOKA
      27003.0
      0.428565
      1
      1.163151

5 rows × 30 columns

```python
county_names = data.county.unique()
county_idx = data['county_code'].values
```

```python
with pm.Model() as hierarchical_model:

    # Hyperpriors
    mu_a = pm.Normal('mu_alpha', mu=0., sd=100**2)
    sigma_a = pm.Uniform('sigma_alpha', lower=0, upper=100)
    mu_b = pm.Normal('mu_beta', mu=0., sd=100**2)
    sigma_b = pm.Uniform('sigma_beta', lower=0, upper=100)

    # Intercept for each county, distributed around group mean mu_a
    a = pm.Normal('alpha', mu=mu_a, sd=sigma_a, shape=len(data.county.unique()))
    # Intercept for each county, distributed around group mean mu_a
    b = pm.Normal('beta', mu=mu_b, sd=sigma_b, shape=len(data.county.unique()))

    # Model error
    eps = pm.Uniform('eps', lower=0, upper=100)

    # Expected value
    radon_est = a[county_idx] + b[county_idx] * data.floor.values

    # Data likelihood
    y_like = pm.Normal('y_like', mu=radon_est, sd=eps, observed=data.log_radon)
```

Having specified the graphical model, we can run inference using the state of the art No-U-Turn Sampler (NUTS) initialized with ADVI.

```python
with hierarchical_model:
    # Use ADVI for initialization
    mu, sds, elbo = pm.variational.advi(n=100000)
    step = pm.NUTS(scaling=hierarchical_model.dict_to_array(sds)**2, is_cov=True)
    hierarchical_trace = pm.sample(5000, step, start=mu)
```

    /usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:3: DeprecationWarning: Old ADVI interface and sample_vp is deprecated and will be removed in future, use pm.fit and pm.sample_approx instead
      This is separate from the ipykernel package so we can avoid doing imports until
    Average ELBO = -1,118.2:  49%|████▉     | 48907/100000 [00:15<00:16, 3116.50it/s]Median ELBO converged.
    Finished [100%]: Average ELBO = -1,103.6

    WARNING (theano.tensor.blas): We did not found a dynamic library into the library_dir of the library we use for blas. If you use ATLAS, make sure to compile it with dynamics library.
    100%|█████████▉| 5494/5500 [00:37<00:00, 143.16it/s]/usr/local/lib/python2.7/dist-packages/pymc3/step_methods/hmc/nuts.py:440: UserWarning: The acceptance probability in chain 0 does not match the target. It is 0.667219685913, but should be close to 0.8. Try to increase the number of tuning steps.
      % (self._chain_id, mean_accept, target_accept))
    100%|██████████| 5500/5500 [00:37<00:00, 144.83it/s]

Let's examine the trace plots of all our latent variables in the hierarchical regression model:

```python
pm.traceplot(hierarchical_trace[500:])
plt.show()
```

From the traceplots, we can see convergence in our posterior distributions for $\alpha_c$ and $\beta_c$ indicating different intercepts and slopes for different counties. In addition, we also recover the posterior distribution of the shared parameters. $\mu_{\alpha}$ tells us that the group mean of log radon levels is close to $1.5$, while $\mu_{\beta}$ tells us that the slope is negative with a mean of $-0.65$ and thus having no basement decreases radon levels.

```python

```