```python
import numpy as np
import math
from scipy import optimize
import numba as nb
import cap_equi as func
import interactive_plots as int_plot

import scipy.stats as stats

# For plots:
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('seaborn-whitegrid')
mpl.style.use('seaborn')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
```

# PS2, part 1: Welfare effects of capital injection

The exercise consists of the core setup:
* Two periods, $t=0,1$,
* Two types of agents: Consumers and entrepreneurs.
* **Consumers:**
    * Linear utility $u_c = c_0^c+c_1^c$.
    * Large endowment of consumption goods in each period: $e$.
    * Labor endowment of $e_l=1$. Inelastically supplied in a competitive labor market at the price $w_1$.
* **Entrepreneurs:**
    * Linear utility $u_e = c_0^e+c_1^e$.
    * Small endowment $n_0$ in period 0.
    * Borrow $b_1$, invest in capital $k_1$.
    * Period 1 they hire labor $l_1$ at wage $w_1$.
    * Entrepreneurs utilize capital and labor in $t=1$ with Cobb-Douglas production function
    $$ \begin{align}
    y_1 = k_1^{\alpha}l_1^{1-\alpha} \notag
    \end{align} $$
    * **Entrepreneurs are borrowing constrained:**
    $$\begin{align}
    b_1\leq \lambda \left(y_1-w_1l_1\right), && \lambda\in (0,1) \notag
    \end{align} $$
* Assume that endowment of consumption goods $(e)$ is *sufficiently* large, for the interest rate to be 1 in equilibrium. (We will show what *sufficiently* means here).

## Q1: Three parts. (i) State the entrepreneur's problem. (ii) Argue that entrepreneurs will choose to maximize profits in period 1, a linear function of $k_1$, and (3) restate entrepreneur's problem as a simple linear problem.

### (i) Entrepreneur problem:

Interpret bullets from **entrepreneurs** above as a problem:

$$
\begin{align}
    \max_{c_0,c_1,k_1,l_1,y_1,b_1} \mbox{ } c_0+c_1,& \notag \\
    \text{s.t. }\mbox{ }c_0+k_1 &= n_0+b_1 \notag \\
                        c_1+b_1 &= y_1-w_1l_1 \tag{Problem 1}\\
                            y_1 &= k_1^{\alpha}l_1^{1-\alpha} \notag \\
                            b_1\leq &\lambda \left(y_1-w_1l_1\right) \notag
\end{align}$$

States that:
1. Maximize consumption (risk-neutrality).
2. Budget $t=0$: Consumption $(c_0)$ and investment $(k_1)$ equals income from endowment $(n_0)$ and borrowing $(b_1)$.
3. Budget $t=1$: Consumption $(c_1)$ and repayment of debt $(b_1)$ equals income from production $(y_1)$ minues wages $(w_1l_1)$.
4. Production is Cobb-Douglas.
5. Entrepreneur is credit-constrained, with $\lambda$ denoting the degree of future profits that can be used as collateral.

Substitute for $c_0$ and $c_1$ into the utility function, using the budgets for $t=0,1$:

$$\begin{align}
    c_0+c_1 &= n_0+b_1-k_1+y_1-w_1l_1-b_1 \\
            &= n_0+y_1-k_1-w_1l_1
\end{align}$$

Substitue for $y_1$ using the Cobb-Douglas function (remember you should substitute for $y_1$ in both the object function **and** the constraints). The problem is then reduced to:

$$\begin{align}
    \max_{k_1,l_1,b_1} n_0+k_1^{\alpha}l_1^{1-\alpha}-k_1-w_1l_1, && s.t. b_1\leq \lambda \left(k_1^{\alpha}l_1^{1-\alpha}-w_1l_1\right). \tag{Problem 2}
\end{align}$$

The Lagrangian function corresponding to this problem is then given by:

$$\begin{align}
    \mathcal{L}_2 = n_0+k_1^{\alpha}l_1^{1-\alpha}-k_1-w_1l_1+\mu\left[\lambda\left(k_1^{\alpha}l_1^{1-\alpha}-w_1l_1\right)-b_1\right],
\end{align}$$

where $\mu\geq0$ is the shadow-variable related to the credit-constraint. As in problem set, exercise 3.5 in (JT), this problem is characterized by a set of first order conditions with Karush-Kuhn-Tucker conditions for $\mu$ and the credit constraint. This is because of the inequality constraint (instead of standard Lagrangian approach when the constraint is with "=").

### (ii) Optimal choice of labor:

Note that profits in $t=1$ is given by:

$$\begin{align}
    \pi_1 = k_1^{\alpha}l_1^{1-\alpha}-w_1l_1.
\end{align}$$

Optimal choice of labor $(l^*)$ is then characterized by:

$$\begin{align}
    \dfrac{\partial \mathcal{L}}{\partial l_1} = (1-\alpha)k_1^{\alpha}l_1^{-\alpha} -w_1 + \mu\lambda\left[(1-\alpha)k_1^{\alpha}l_1^{-\alpha}-w_1\right] = 0
\end{align}$$

We can rewrite this condition as:

$$\begin{align}
    \left[(1-\alpha)k_1^{\alpha}l_1^{-\alpha}-w_1\right](1+\mu\lambda)=0 && \Rightarrow && w_1 = (1-\alpha)k_1^{\alpha}l_1^{-\alpha}
\end{align}$$

Note that this corresponds to maximizing $\pi_1$ wrt. labor. Note further that we can use this to write up two other terms:

$$\begin{align}
w_1l_1^* = (1-\alpha)k_1^{\alpha}(l_1^*)^{1-\alpha}, \tag{Labor Share}
\end{align}$$

and by isolating for $l_1$:

$$\begin{align}
    l_1^* = \dfrac{(1-\alpha)^{1/\alpha}k_1}{w_1^{1/\alpha}}. \tag{$l^*$}
\end{align}$$

To show that the profit function is linear in $k_1$, when labor is chosen optimally, start by using (Labor Share):

$$\begin{align}
    \pi_1(l=l^*) &= k_1^{\alpha}(l_1^*)^{1-\alpha}-w_1l_1^* \notag \\
                 &= \alpha k_1^{\alpha}(l_1^*)^{1-\alpha}
\end{align}$$

Now substitute for $(l^*)$:

$$\begin{align}
    \pi_1(l=l^*) &= \alpha k_1^{\alpha}\dfrac{(1-\alpha)^{(1-\alpha)/\alpha}k_1^{1-\alpha}}{w_1^{(1-\alpha)/\alpha}} \notag \\
                 &= \underbrace{\dfrac{\alpha (1-\alpha)^{(1-\alpha)/\alpha}}{w_1^{(1-\alpha)/\alpha}}}_{\equiv R(w_1)}k_1 \tag{Profit}
\end{align}$$

### (iii) Restate as linear problem:

Using (Profit) in (Problem 2):

$$\begin{align}
    \max_{k_1,b_1}\mbox{ }n_0-k_1+R(w_1)k_1, && \text{s.t. } b_1\leq \lambda R(w_1)k_1 \tag{Problem 3}.
\end{align} $$

## Q2: Derive entrepreneur's demand for capital, and argue how the solution depends on $\lambda$ and $R(w_1)$.

Problem $3$ is linear in $k_1$. Thus capital demand depends on:

* If $R(w_1)<1$: Investment in capital is not profitable, $k_1^*=0$. (it is implicitly assumed that $k\geq0$)
* If $\lambda R(w_1)>1$: Investment in capital is so profitable that borrowing constraint never binds, $k_1^*=\infty$.
* If $\lambda R(w_1)<1<R(w_1)$: Capital is postponed as much as possible, but borrowing constraint will bind. Solution is then to set $c_0=0$ in the budget at $t=0$, and use that $b_1$ is determined by the borrowing constraint:

$$\begin{align}
    k_1^* &= n_0 +b_1 \\
          &= n_0+\lambda R(w_1)k_1^* \\
\Rightarrow k_1^* &= \dfrac{n_0}{1-\lambda R(w_1)}
\end{align}$$

* Finally, if $R(w_1)=1$ the entrepreneur is indifferent between consumption and investment. In this case capital demand is indeterminate:
$$\begin{align}
    k_1^*\in\left[0,\mbox{ }\dfrac{n_0}{1-\lambda}\right].
\end{align}$$

**Illustrating demand for capital, and equilibrium capital**:

Before we proceed to the next questions, let us briefly illustrate how capital demand looks like, and how it behaves in equilibrium. *(NB: You don't need to change anything in the cell with code below (you can though), just use the sliders in the graph.)*

The first part here plots the capital demand and the interest rate function $(R=\alpha k^{\alpha-1})$.

```python
# Baseline parameters:
n_0base, lambda_base, alpha_base, Inf = 0.1, 0.55, 0.9, 10**3

# Exogenous grid of R (interest rate):
R_min, R_max, N_R= 0.1, 2, 100
R_grid = np.linspace(R_min, R_max, N_R)

# Upper and lower limit of sliders to operate below:
n_0_lower, n_0_upper, n_0_step = 0, 0.5, 0.05
lambda_lower, lambda_upper, lambda_step = 0, 0.9, 0.05

# Plot:
int_plot.interactive_cap(n_0base, lambda_base, alpha_base, Inf, N_R, R_grid, n_0_lower, n_0_upper, n_0_step, lambda_lower, lambda_upper, lambda_step)
```

    interactive(children=(FloatSlider(value=0.55, description='$\\lambda$', max=0.9, step=0.05), FloatSlider(value…

The second part here plots the equilibrium value across $(\lambda,n)$.

```python
# Grids:
n_0_lower, n_0_upper, N_n = 0, 0.5, 51
lambda_lower, lambda_upper, N_lambda = 0, 0.9, 91
n_grid = np.linspace(n_0_lower, n_0_upper,N_n, dtype=np.float)
lambda_grid, lambda_step = np.linspace(lambda_lower, lambda_upper, N_lambda, retstep=True, dtype=np.float)
# baseline:
n_0base, lambda_base, alpha = 0.1, 0.5, 0.9
# Plot:
int_plot.interactive_equicap(N_n,N_lambda,alpha,n_grid,lambda_grid,math.ceil(N_lambda/2),lambda_lower, lambda_upper,lambda_step)
```

    interactive(children=(FloatSlider(value=0.46, description='$\\lambda$', max=0.9, step=0.01), Output()), _dom_c…

## Q3: Show the cutoff $n^*$, where $n_0 \geq n^*$$ implies that the first-best level of capital can be financed, and the borrowing constraint is not binding.

Let us define the first-best level of capital. Recall that in equilibrium $l_1=1$ (as this is the endowment supplied by consumers). Define first-best as **the level of capital that maximizes net output**:

$$\begin{align}
    \max_{k_1} k_1^{\alpha}-k_1 && \Rightarrow && k_1^{fb} = \alpha^{1/(1-\alpha)}.
\end{align}$$

In equilibrium the interest rate $R(w_1)$ is defined by the marginal product of capital:

$$\begin{align}
    R(w_1) = \alpha k_1^{\alpha-1}, && \Rightarrow && k_1 = \left(\dfrac{\alpha}{R(w_1)}\right)^{1/(1-\alpha)}
\end{align}$$

Thus for $k_1=k_1^{fb}$ we need $R(w_1)=1$. When $R(w_1)=1$ the capital demand was indeterminate, i.e. it covered a range of possible solutions. The optimal level $k_1^{fb}$ is one of the feasible capital demand levels as long as:

$$\begin{align}
k_1^{fb}\equiv \alpha^{1/(1-\alpha)}\in\left[0,\mbox{ }\dfrac{n_0}{1-\lambda}\right].
\end{align}$$

As long as $n_0$ is sufficiently large the first best is thus an equilibrium. The threshold $n^*$ where the first best is an equilibrium for $n_0\geq n^*$ is then given by the condition above with equality:

$$\begin{align}
    \alpha^{1/(1-\alpha)} = \dfrac{n^*}{1-\lambda} && \Rightarrow && n^* = (1-\lambda)\alpha^{1/(1-\alpha)}
\end{align}$$

**Addtional remark:** Give another explanation for why we need R=1 for optimality:

Optimality requires that the $\color{blue}{\text{marginal rate of transformation}}$ equals $\color{blue}{\text{the marginal rate of substitution}}$

## Q4: Show that if n<n* there is an equilibrium where the entrepreneurs are constrained and $k_1$ becomes an increasing function of $n_0$.

* **Capital demand:** Recall that when $n_0<n^*$ the entrepreneurs do not have sufficient equity; thus they are credit constrained.
* In the credit constrained case recall that capital was given by the budget constraint for $t=0$ with $c_0=0$ imposed, i.e.:

$$\begin{align}
    k_1 = n_0+\lambda R(w_1) k_1.
\end{align}$$

Substituting for $R(w_1)$ this yields:

$$\begin{align}
    k_1 - \lambda\alpha k_1^{\alpha}=n_0. \tag{$k_1^*$, credit-constrained}
\end{align}$$

As long as $n_0<n^*$ this equations determines $k_1$. Differentiation wrt. $n_0$ then yields

$$\begin{align}
    \dfrac{\partial k_1}{\partial n_0}-\lambda \alpha^2k_1^{\alpha-1}\dfrac{\partial k_1}{\partial n_0}=1 && \Rightarrow && \dfrac{\partial k_1}{\partial n_0} = \left(1-\lambda \alpha k_1^{\alpha}\dfrac{\alpha}{k_1}\right)^{-1}.
\end{align}$$

Using the expression above for the credit constrained $k_1$ we can rearrange slightly to show that

$$\begin{align}
\dfrac{\partial k_1}{\partial n_0} = \dfrac{1}{1-\alpha(k_1-n_0)/k_1}>1
\end{align}$$

another neat way of writing this is:

$$\begin{align}
    \dfrac{\partial k_1}{\partial n_0} = \dfrac{1}{1-\alpha \lambda R(w_1)}>1.
\end{align}$$

Capital thus increases more than 1-to-1 with the entrepreneur's equity here. The reason for this is the following:

In the credit constrained case the entrepreneur prefers to allocate **all** of his consumption to $t=1$: Thus when $n_0$ increases marginally so does $k_1$, initially 1-to-1. When $k_1$ increases the entrepreneur increases his future profits $(y_1-w_1l_1)$; this relaxes the credit-constraint, allowing him to borrow more against future profits. This increases investments further.

## Q5: Introduce a lump-sum tax on consumers in period 0 that transfers wealth to entrepreneurs. Derive expected utility as a function of the lump sum tax rate.

Let us start with the consumers' utility. This is straightforward:

$$\begin{align}
    U_C(\tau) = 2e-\tau + \overbrace{(1-\alpha)k_1^{\alpha}}^{\equiv w_1(\tau)}
\end{align}$$

**The entrepreneur part:**

Setup is as defined in (Problem 3) earlier, simply with an added $\tau$:

$$\begin{align}
    U_E(\tau) = n_0+\tau +k_1\left[R(w_1)-1\right]
\end{align}$$

Substitute for the equilibrium interest rate we then have

$$\begin{align}
    U_E(\tau) = n_0+\tau + \alpha k_1^{\alpha}-k_1
\end{align}$$

Now we have two cases:

### The credit-constrained case

If $n_0+\tau<n^*$: The entrepreneur is **credit-constrained**, and capital is defined as earlier by the equality

$$\begin{align}
k_1-\lambda \alpha k_1^{\alpha} = n_0+\tau.
\end{align}$$

Substituting for this we get

$$\begin{align}
    U_E(\tau) = \alpha k_1^{\alpha}(1-\lambda)
\end{align}$$

Increasing the tax $\tau$ corresponds to increasing the available endowment for entrepreneurs at $t=0$. As we confirmed above, capital is increasing in $n_0$ and thus in $\tau$, when the credit constraint is binding. Thus $\partial U_E/\partial\tau>0$ when the entrepreneur is constrained. Technically we can confirm this:

$$\begin{align}
    \dfrac{\partial U_E}{\partial \tau} &= \alpha^2k_1^{\alpha-1}(1-\lambda)\dfrac{\partial k_1}{\partial n_0} \\
                                        &= \alpha R(w_1)(1-\lambda)\dfrac{\partial k_1}{\partial n_0}>0
\end{align}$$

Using the derivative $\partial k_1/\partial n_0$, we can (if we prefer) simplify this further as

$$\begin{align}
    \dfrac{\partial U_E}{\partial \tau} &= \dfrac{\alpha R(w_1)(1-\lambda)}{1-\alpha \lambda R(w_1)}>0, && n_0+\tau<n^*
\end{align}$$

### The non-constrained case:

If $n_0+\tau\geq n^*$ the capital equilibrium is unaffected. In this case the tax simply transfers wealth from consumers to entrepreneurs, who then immediately consume it. In other words we sum up:

$$\begin{align}
    \dfrac{\partial U_e}{\partial \tau} = \left\lbrace \begin{array}{ll}  1 & n_0+\tau\geq n^* \\
    \dfrac{\alpha R(w_1)(1-\lambda)}{1-\alpha \lambda R(w_1)} & n_0+\tau<n^* \end{array}\right.
\end{align}$$

## Q6: Show that the relationship between consumer utility and the lump-sum tax is non-monotone. Show that for sufficiently small $n_0$ the tax can be pareto-improving.

Entrepreneurs always benefit from the tax. For consumers the tax includes three potential effects:

$$\begin{align}
    \dfrac{\partial U_c}{\partial \tau} = \left\lbrace \begin{array}{ll} \color{red}{-1}, & n_0+\tau\geq n^* \\ \color{red}{-1} +\color{blue}{(1-\alpha)R(w_1)}\color{blueviolet}{\dfrac{\partial k_1}{\partial \tau}}, & n_0+\tau<n^*\end{array}\right.
\end{align}$$

When $n_0+\tau\geq n^*$ the effect is clearly negative. However, when entrepreneurs are credit-constrained:

1. Negative wealth transfer effect: $\color{red}{-1}$.
2. Entrepreneurs increase capital investment, which increases labor wages: $\color{blue}{(1-\alpha)R(w_1)}$.
3. **(Credit multiplier)** The transfer loosens credit constraitn incuding further investment: $\color{blueviolet}{\partial k_1/\partial \tau}>1$.

If effects 2-3. dominate effect 1., the tax has a pareto-improving effect. Effects 2-3. are largest for small values of $k_1$ (thus $n_0$).

## Q7: Discuss how this justifies government's capital injection during the financial crisis. What features might the model be missing?

The role for capital injections in this model:

* A capital injection relaxes the banks' borrowing constraint, increases lending to the real sector and thus investments. This increases employment (not in our model though) and wages, which can offset the effects of the recession.

* *Caveats (a few of potentially many):*
    1. Moral hazard (reputation effect): If financial institutions expect bailouts $\Rightarrow$ increase risky investments.
    2. Endogenous uncertainty: A feature of the financial crisis was precautionary behavior in light of increased uncertainty. The capital injection reduces uncertainty of asset prices.

## Q8: How does key findings in Q6 depend on the type of borrowing constraint?

The short answer: The borrowing constraint is quite standard. There are microfoundations that lead to this type of constraint.

In this model future profits can (partly) be collateralized. There are several potential explanations why future profits cannot entirely be collateralized.

* **Informational/enforcement frictions** are at the heart of any borrowing constraint (to the best of my knowledge at least).
* **Moral Hazard**: Behave/shirk.
* **Collateralized assets** as Kiyotaki and Moore (1997) leads to similar results.

# PS2, part 2: Costly state verification and optimality of debt

The core of the model setup:

* **Utility** is linear: $U_E=c_0+c_1$.
* Entrepreneur invests at time $0$, cost $I$. Return at time $t=1$ is random, distributed according to $g(s)$:

$$\begin{align}
    Return = Rs, && s\sim g(s), && \text{Support on }S\equiv \left[s^{min}, s^{max}\right]
\end{align}$$
Also assumed that
$$\begin{align}
    \mathbb{E}[s]=1, && R>I.
\end{align}$$

* **Asymmetric information:** State $s$ is unknown at time $t=0$. At time $t=1$ the entrepreneur (E) observes the true value of $s$ costlessly. The financier (F) can choose to audit and observe it, but at the cost $\kappa>0$.
* **Assumption:** The financier chooses a *deterministic* audit rule. I.e. no *random/stochastic* audit rule. By the *revelation principle:* Only focus on contracts in which E has no incentive to lie.
* **Contracts:** Specified by the triplet

$$\begin{align}
    Contract = \left( R^E(s), \mbox{ }R^F(s), \mbox{ }S_A\right), && S_A\subset S
\end{align}$$
where $R^E(s)$ and $R^F(s)$ denotes the payoffs for E,F (net of audit cost). $S_A$ is the set of states $S$, where $F$ chooses to audit.

* **Resource constraint:**

$$\begin{align}
    R^E(s)+R^F(s)+\kappa \mathbf{1}_{s\in S_A} = Rs, && R^E(s)\geq 0.
\end{align}$$

where $\mathbf{1}_{s\in S_A}$ indicates auditing, and the inequality captures *limited liability* of the entrepreneur.

* **Participation constraint:** (for F)

$$\begin{align}
    \int_S R^F(s)g(s)\mbox{ }ds\geq I.
\end{align}$$

States that expected revenue covers investment cost.

### Q1: Write down the IC constraint for E. What does it imply for F's net payoff?

**IC constraint** with a continuous set of possible outcomes: *In any given state $s$ it must not be profitable to deviate to *any* state $s'$.*

Another useful definition is **the revelation principle**. The principle roughly states that every potential equilibrium outcome, can be achieved by an incentive-compatible strategy.

The IC constraint then states that

$$\begin{align}
    Rs-R^F(s)-\kappa \mathbf{1}_{s\in S_A}\geq Rs-R^F(s'), && \forall s'\in\left(S\backslash S_A\right)
\end{align}$$

The left-hand side (LHS) is the profit from E telling the true $s$. The RHS is the outcome if E is in state $s$, but signal to F that he is in state $s'$. Note that we only consider deviations to $s'\in (S\backslash S_A)$. There is no incentive to lie about a state $s'$, where E knows he will be audited.

**What does it imply for F's net payoff?**

* Consider two states, $s,s'$, both in the no-audit region. The IC constraint for state $s$ states $R^F(s)\geq R^F(s')$. The IC constraint for state $s$ states $R^F(s')\geq R^F(s)$.

* For them both to hold simultaneously, there must be exactly one payoff level for all $s\notin S_A$. Denote this $R^F(\bar{s})$. If there was a state with higher payoffs in the region $S\backslash S_A$, where E is not audited, he would always report this.

* Now consider the case where $s\in S_A$ (recall that alternative $s'\notin S_A$): The $(IC)$ then states that

$$\begin{align}
    Rs-R^F(s)-\kappa \geq Rs-R^F(\bar{s}) && \Rightarrow && R^F(s)\leq R^F(\bar{s}), && \forall s\in S_A.
\end{align}$$

* This states that in the audit region, the financier (F) has to accept lower returns $R^F(s)$ than in the non-audit region. If this was not the case, E would always report a state in the no-audit region, and thus get higher profits.

### Q2: Define a standard debt contract:

$$\begin{align}
    R^F(s) = \left\lbrace \begin{array}{ll} R\bar{s}, & s\geq \bar{s} \\ Rs-\kappa, & s\leq \bar{s} \end{array}\right., && S_A=\left[s^{min},\mbox{ }\bar{s}\right].
\end{align}$$

### Check that any level $\bar{s}$ satisfies both the resource constraint and the (IC).

* **(IC) constraint:**
    * When $s\geq \bar{s}:$ The entrepreneur receives $R(s-\bar{s})$ by telling the truth. The same if he over-reports. If he under-reports in the audit-region he receives 0.
    * When $s<\bar{s}:$ If E over-reports (in the no-audit region) he receives $R(s-\bar{s})<0$. By true reporting he receives 0.

* **Resource constraint:**
    * If $s\geq \bar{s}$ the resource constraint reads:
    $$\begin{align}
        \overbrace{R(s-\bar{s})}^{R^E}+\overbrace{R\bar{s}}^{R^F} + \overbrace{0}^{audit}= Rs
    \end{align}$$
    * If $s<\bar{s}$ the resource constraint reads:

$$\begin{align}
    \overbrace{0}^{R^E} +\overbrace{Rs-\kappa}^{R^F} + \overbrace{\kappa}^{audit} = Rs
\end{align}$$

### Q3: Prove that the standard contract is optimal.

Consider a non-standard contract:

$$\begin{align}
    (\tilde{R}^E, \tilde{R}^F, \tilde{S}_A)
\end{align}$$

* The (IC) constraint still needs to hold. Thus for $s\notin \tilde{S}_A$ the non-standard contract still has to offer the same return. Denote this constant $\tilde{R}^F$.
* Note furthermore that $\tilde{S}_A$ has to be a closed interval on the form $\tilde{S}_A = [s^{min},\tilde{s}]$: Assume there is a level $s$ that is not audited, but $s'>s$ is audited. If E is in state s', he can report $s$ without being audited and profit from it.
* This implies that $\tilde{R}^F=R\tilde{s}$.

Consider now the various regions of $s$:

* For $s\in\left(\tilde{S}_A \cap S_A\right):$ F receives $Rs-\kappa$ in both cases.
* For $s\in\left(\tilde{S}_A \cap (S\backslash S_A)\right):$ F receives higher payoff in standard debt contract, where he does not pay the audit cost.
* For $s$ in non-audit region: Let the standard contract offer the level $\bar{R}^F= \tilde{R}^F$. Thus F is indifferent in these cases.

For entrepreneurs: The standard debt contract minimizes audit-costs (given IC and participation constraints). Given that they receive the residual income F's payment, E prefers a standard contract as well.

### Q4: Assume that $s$ is uniform. Characterizes the riskiness of the optimal debt contract.

Assume that $s$ is uniform on $[1-\Delta, 1+\Delta]$. In other words the pdf (f) and cdf (F) are given by:

$$\begin{align}
f(s) = \dfrac{1}{2\Delta}, && F(s) = \dfrac{s-(1-\Delta)}{2\Delta}, && s\in [1-\Delta, 1+\Delta]
\end{align}$$

The entrepreneur then solves:

$$\begin{align}
    \max_{\bar{s}} \dfrac{1}{2\Delta} \int_{\bar{s}}^{1+\Delta}(Rs-R\bar{s})\mbox{ }ds
\end{align}$$

subject to F's participation constraint

$$\begin{align}
    \dfrac{1}{2\Delta} \int_{\bar{s}}^{1+\Delta} R\bar{s}\mbox{ }ds+\dfrac{1}{2\Delta}\int_{1-\Delta}^{\bar{s}}(Rs-\kappa)\mbox{ }ds\geq I
\end{align}$$

Integrating out the expression we write this as:

$$\begin{align}
       \left[1+\Delta-\bar{s}\right]R\bar{s}+\left[\dfrac{R}{2}\bar{s}^2-\bar{s}\kappa\right]-\left[\dfrac{R}{2}(1-\Delta)^2-(1-\Delta)\kappa\right]\geq 2\Delta I,
\end{align}$$

or rewritten as

$$\begin{align}
    -\dfrac{R}{2}\bar{s}^2+\left[R(1+\Delta)-\kappa\right]\bar{s}+(1-\Delta)\kappa-\dfrac{R}{2}(1-\Delta)^2\geq 2\Delta I. \tag{PC}
\end{align}$$

We note here that:
* For $s<\bar{s}:$ the entrepreneur receives 0. Thus the object function integrates from $\bar{s}$ to $1+\Delta$.
* For $\bar{s}\leq s \leq 1+\Delta$, the financier receives $R\bar{s}$. In the non-audit region he receives $Rs-\kappa$.

It is straightforward to see that for the entrepreneur, the lowest possible $\bar{s}$ is preferred, where the participation constraint still holds.

To do this proceed as follows:

* If $I$ is sufficiently small, the obvious solution is to choose $\bar{s}=1-\Delta$. This is feasible (participation constraint holds) if:

$$\begin{align}
        R(1-\Delta)\geq I
\end{align}$$

* There are two options now: Either there is **no** solution within $[1-\Delta, 1+\Delta]$. Otherwise the solution is given in the case where $(PC)$ holds with equality.

Solving for $(PC)$ with equality, we have:

$$\begin{align}
 \bar{s}^* = 1+\Delta-\dfrac{\kappa}{R}-\dfrac{1}{R}\sqrt{(R(1+\Delta)-\kappa)^2+2R*\left[(1-\Delta)\kappa-\dfrac{R}{2}(1-\Delta)^2-2\Delta I\right]}, \tag{$s^*$}
\end{align}$$

Note that there might not be any solution at all, if the term in the $\sqrt{\cdot}$ is negative.

```python
# Parameter values:
R = 2
I = 1.5
Delta = 0.5
kappa = 0.1
# Grid of sbar:
sbar = np.linspace(1-Delta, 1+Delta, 101)
# PC constraint:
int_plot.interactive_pc(R,I,Delta,kappa,sbar)
s_star = 1+Delta-kappa/R-(1/R)*math.sqrt((R*(1+Delta)-kappa)**2+2*R*((1-Delta)*kappa-(R/2)*(1-Delta)**2-2*Delta*I))
print(s_star)
```

    interactive(children=(FloatSlider(value=1.5, description='I', max=2.0), FloatSlider(value=0.1, description='$\…

    0.8155711229775239

In the figure above (with values $R=2,I=1.5,\Delta=0.5,\kappa=0.1$) note that the PC constraint holds for values of $\bar{s}$ slightly larger 0.8.

### Q5: Find the threshold cost $I$, where the project is profitable.

Using the result from Q4 the threshold $I^{max}$, where the project is financed as long as $I\leq I^{\max}$ is defined as follows:

* If $R(1-\Delta)\geq I$ then $s^*=1-\Delta$. In this case the largest cost $I$ that is financed, is given by $R(1-\Delta)$.
* If $R(1-\Delta)<I$ then there exists a solution, recall that there exists a solution as long as the term $\sqrt{\cdot}$ in $s^*$ is non-negative.
* Note furthermore, that $I$ enters negatively in that expression; thus the higher $I$, the closer we get to a scenario, where there is no $\bar{s}^*$.
* In other words, the largest $I$ that allows a feasible solution is defined by the condition:

$$\begin{align}
(R(1+\Delta)-\kappa)^2+2R*\left[(1-\Delta)\kappa-\dfrac{R}{2}(1-\Delta)^2-2\Delta I\right] = 0 && \Rightarrow && I = R-\kappa+\dfrac{\kappa^2}{4\Delta R}.
\end{align}$$

Combining the two potential scenarios the highest possible $I$ that can be financed is thus given by

$$\begin{align}
     I^{max} = \max \Bigg\lbrace \underbrace{R-\kappa+\dfrac{\kappa^2}{4\Delta R}}_{\text{Scenario 1}},\mbox{ }\mbox{ }\underbrace{R(1-\Delta)}_{\text{Scenario 2}}\Bigg\rbrace.
\end{align}$$

A few comments on this solution:

* **When $\kappa<2R\Delta$:** The max-operator returns *scenario 1*. In this case $I^{max}$ is decreasing in both $\kappa$ and $\Delta$. Why?

    * When $\kappa$ increases the cost of auditing goes up. When $\kappa<2\Delta$ the $(PC)$ constraint is binding implying that $s^*>1-\Delta$; in other words, the optimal debt contract involves some costly auditing. In this scenario a larger audit-cost makes investment projects less profitable over all $\Rightarrow I^{max}$ decreases.

    * An increase in $\Delta$ increases the uncertainty of the outcome. You may verify (either from just adjusting the graph above, or solving for the derivative) that $s^*$ in this case is increasing in $\Delta$. In other words, the increase in $\Delta$ increases the region of $s$, where the financier has to audit, in order to ensure zero expected profits. This lowers the profitability of the project for F, which lowers $I^{max}$.

* **When $\kappa\geq 2R\Delta$:** The max operator returns *scenario 2*.

    * In this case $I^{max}$ is simply given by $R(1-\Delta)$: When there is no auditing at all, the project is still profitable for F. Thus any change in auditing-costs $(\kappa)$ does not affect $I^{max}$.

    * An increase in $\Delta$ clearly lowers $s^*$ proportionally. As the financier in this case always receive the simple payoff $R^F=Rs^* = R(1-\Delta)$, an increase in $\Delta$ lowers the profits from the project, thus $I$ decreases.


$\newcommand{\xv}{\mathbf{x}}
 \newcommand{\wv}{\mathbf{w}}
 \newcommand{\yv}{\mathbf{y}}
 \newcommand{\zv}{\mathbf{z}}
 \newcommand{\Chi}{\mathcal{X}}
 \newcommand{\R}{\rm I\!R}
 \newcommand{\sign}{\text{sign}}
 \newcommand{\Tm}{\mathbf{T}}
 \newcommand{\Xm}{\mathbf{X}}
 \newcommand{\Zm}{\mathbf{Z}}
 \newcommand{\I}{\mathbf{I}}
 \newcommand{\muv}{\boldsymbol\mu}
 \newcommand{\Sigmav}{\boldsymbol\Sigma}
$
### ITCS6155

# Reinforcement Learning

Along with supervised and unsupervised learning, reinforcement learning is one of the interesting fields of machine learining.
As we shortly discussed in the first week of class, reinforcement learning is different from other machine learning paradigms as summarized below:

- only reward signal as feedback,
- the feedback can be delayed,
- sequential data,
- interaction based on the actions taken.

The reinforcement learning resembles huuman learning or animal training that treats reward good behavior.
When the series of actions end up with good results, we can **reinforce** those actions by giving some rewards.

## Applictions

There are many possible applications:

- walking robot,
- playing boardgames,
- controlling joysticks for video games,
- smart thermostat,
- stock trading,
- music personalization,
- recommendation system,
- marketing,
- product delivery,
- and so on...

## Terminology

- Reward: $R_t$ is a scalar feedback signal at time $t$. It indicates how well agent is doing at the time.

- State: $S_t$ represents what happens currently and next.

- Action: $A_t$ is how an agent affects to the environment that can change the state.

- Observation: $O_t$ is what an agent recognizes the world for the state $S_t$.

- Policy: A function that determines an agent's behavior or a function that selects its action, $\pi(S_t)$.

Q: Think about the rewards for the example applications above, and answer the possible rewards for them.

 Answer HERE

## Modeling

How can we simplify the listed problems to solve? We have the sets of decision making problems or control problems.
With the assumption of **Markov property**, we can model the problems as Makrov Decision Process (MDP).

Here, the definition of Markov property is as follows,

**Definition**: A state $S_t$ is Markov if and only if

$$P [ S_{t+1} | S_t ] = P[ S_{t+1} | S_1, ..., S_t ].$$

That is, we can say the state transition model has Markov property when the future is dependent only on the current state not the past.

### Markov Decision Processes
So, the MDP can be defined as follows and the example state transition diagram has the transition probabilities and rewards along with discounting factor. The discounting factor ensures the mathematical simplicity, and it allows the modeling the uncertainty of the future as well.

 $ ( S, A, P, R, \gamma ) $
* $S$ : a finite set of states
* $A$ : a finite set of actions
* $P$ : a state transition probability
* $P^a_{ss^\prime} = P [ S_{t+1} = s^\prime | S_t = s, A_t = a ]$
* $R$ : a reward function
* $\gamma$ : a discount factor

 MDP with 3 states and 2 actions (wikipedia)

### Policy

The policy $\pi(S_t)$ is a function that maps from a state to an action. Thus, it can be easily represented by a probability:

$$ \pi(a | s) = P[A_t = a | S_t = s]. $$

Or, it can be a machine learning model to generate such probability for a certain behavior of an agent.

## Goal

Now with the MDP model, what we want to do is find a policy $\pi$ that maximizes the long-term rewards. The long-term rewards can be modeled as the sum of rewards. Here, considering the different importance on the actions (the current reward is not equivalent to future rewards), we can define the **Return**, $G_t$,

$$ G_t = R_{t+1} + \gamma R_{t+2} + \cdots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}. $$

From this, we can model the long-term value of a certain state $s$ with the return:

$$ V(s) = E [ G_t | S_t = s ]. $$

We, we set our objective to maximize the return. Developing the optimal policy can be made by defining the evaluation function of state and action pair, the Q funciton:

$$
\begin{align}
    Q^\pi(s, a) &= E_\pi [ G_t | S_t = s, A_t = a ] \\
                &= E_\pi [ R_{t+1} + \gamma Q^\pi (S_{t+1}, A_{t+1}) | S_t = s, A_t = a ]
\end{align}
$$

## How to reach the goal? / How to find the optimal policy?

There are several ways to reach the goal such as value iteration, policy iteration, linear programming, and temporal difference learning.

First, let us examine the bootstraping representation of the value function to solve the problem.
This is called *Bellman equation*.

$$
\begin{aligned}
  V(s) &= E [ G_t | S_t = s ] \\
       &= E [ R_{t+1} + \gamma R_{t+2} + \cdots | S_t = s ] \\
       &= E [ R_{t+1} + \gamma ( R_{t+2} + \gamma R_{t+3} + \cdots ) | S_t = s ] \\
       &= E [ R_{t+1} + \gamma G_{t+1} | S_t = s ] \\
       &= E [ R_{t+1} + \gamma V(s_{t+1}) | S_t= s ]\\
      \\
  V(s) &= R(s) + \gamma \sum_{s^\prime \in S} P_{ss^\prime} V(s^\prime)
\end{aligned}
$$

With matrix representation, we can rewrite this as follows:

$$
 \begin{align}
  \begin{pmatrix}
  V_1 \\
  V_2 \\
  \vdots \\
  V_n
 \end{pmatrix} &=
  \begin{pmatrix}
  R_1 \\
  R_2 \\
  \vdots \\
  R_n
 \end{pmatrix} +
 \gamma
 \begin{pmatrix}
  P_{11} & P_{12} & \cdots & P_{1n} \\
  P_{21} & P_{22} & \cdots & P_{2n} \\
  \vdots  & \vdots  & \ddots & \vdots  \\
  P_{n1} & P_{n2} & \cdots & P_{nn}
 \end{pmatrix}
  \begin{pmatrix}
  V_1 \\
  V_2 \\
  \vdots \\
  V_n
 \end{pmatrix}
 \\
 \\
 V &= R + \gamma PV
 \end{align}
$$

This can be directly solved with

$$ V = (I - \gamma P)^{-1} R. $$

Considering the computational complexity with inverse matrix, however, we can consider iterative solutions such as dynamic programming, Monte-Carlo evaluation, or temporal difference learning.

# Value Iteration

* For state transition from $i$ to $j$ with action $k$
$$ V^{n+1} (s_i) = \max_k \Big[ R_i + \gamma \sum_{j=1}^N P^k_{ij} V^n (s_j) \Big] $$

* Value Iteration:

$$
\begin{align}
n=0 \quad \forall i, \quad &V^1 (s_i) = \max_k \Big[ R_i + \gamma \sum_{j=1}^N P^k_{ij} V^0 (s_j) \Big] \\
n=1 \quad \forall i, \quad &V^2 (s_i) = \max_k \Big[ R_i + \gamma \sum_{j=1}^N P^k_{ij} V^1 (s_j) \Big] \\
n=2 \quad \forall i, \quad &V^3 (s_i) = \max_k \Big[ R_i + \gamma \sum_{j=1}^N P^k_{ij} V^2 (s_j) \Big] \\
\vdots
\end{align}
$$

* Convergence can be tested as follows:

$$ \max_i \Big| V^{n+1}(s_i) - V^n(s_i) \Big| \lt \epsilon. $$

* Computing values for all $i$'s with Dynamic Programming.

Here follows an example from Wikipedia.

Let $\gamma = 1$. The values for each state can be iteratively updated as the following table.

| t | $V^\pi_{S_0}$ | $V^\pi_{S_1}$ | $V^\pi_{S_2}$ |
|:--|:-------------|:-------------|:-------------|
| 0 |      0       |       0.0      |      0       |
| 1 |      0       |       3.5      |    0.75      |
| 2 |      0       |       3.85     |    0.75      |
| 3 |      0.75       |      4.035      |    1.155      |
| 4 |      1.155       |      4.6595      |    1.5975      |
| 5 |      1.5975     |      5.09395   |    2.08335      |

In Python, we can write the codes.

```python
import collections
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
gamma = 1.0
n_states = 3
n_actions = 2

# value vector
V = np.zeros(n_states).reshape((-1, 1))
# Reward dictionary
R = {(1,2,0): -1, (0,1,0): 5}
# transition probability: key (action, from, to)
P = {(0,0,0): 0.5, (0,0,2): 0.5, (1,0,2): 1.0,
     (0,1,0): 0.7, (0,1,1): 0.1, (0,1,2): 0.2,
     (1,1,1): 0.95, (1,1,2): 0.05,
     (0,2,0): 0.4, (0,2,2): 0.6,
     (1,2,0): 0.3, (1,2,1): 0.3, (1,2,2): 0.4 }

def Rf(s, sn, a):
    if (a, s, sn) in R:
        return R[(a,s,sn)]
    return 0

def Pf(s, sn, a):
    if (a, s, sn) in P:
        return P[(a,s,sn)]
    return 0.

""" with no matrix"""
def Rs(s, a):
    r = 0
    for j in range(3):
        if (a, s, j) in P:
            r += P[(a, s, j)] * Rf(a,s, j)
    return r

def Pfv(s, a):
    ret = np.zeros(3).reshape((-1, 1))
    for j in range(3):
        if (a, s, j) in P:
            ret[j] = P[(a, s, j)]
    return ret

```

```python
Rmat = np.zeros((n_states, n_actions))
Pmat = np.zeros((n_actions, n_states, n_states))

for i in range(n_states):
    for k in range(n_actions):
        for j in range(n_states):
            if (k, i, j) in P:
                Rmat[i, k] += P[(k, i, j)] * Rf(i, j, k)

for k, i, j in P.keys():
    Pmat[k,i,j] = P[(k, i, j)]

for t in range(5):
    print("t=", t, "\tV = ", V.T)
    # Bellman
    newV = Rmat + gamma * np.sum(Pmat @ V, axis=2).T
    V[:] = np.max(newV, axis=1, keepdims=True)

print("t=", t+1, "\tV = ", V.T)
```

    t= 0 	V =  [[ 0.  0.  0.]]
    t= 1 	V =  [[ 0.   3.5  0. ]]
    t= 2 	V =  [[ 0.    3.85  0.75]]
    t= 3 	V =  [[ 0.75   4.035  1.155]]
    t= 4 	V =  [[ 1.155   4.6595  1.5975]]
    t= 5 	V =  [[ 1.5975   5.09395  2.08335]]

```python
gamma=1
for t in range(50):
    print("t=", t, "\tV = ", V.T)
    for i in range(3):
        # Bellman
        V[i] = max(Rs(i, 0) + gamma * np.sum(Pfv(i, 0) * V),
                   Rs(i, 1) + gamma * np.sum(Pfv(i, 1) * V))
print("t=", t+1, "\tV = ", V.T)
```

    t= 0 	V =  [[  0.   17.5  -1.5]]
    t= 1 	V =  [[ -0.75  16.55   3.84]]
    t= 2 	V =  [[  3.84     15.9145    7.16235]]
    t= 3 	V =  [[  7.16235     15.4768925    9.35671275]]
    t= 4 	V =  [[  9.35671275  15.17088351  10.80096398]]
    t= 5 	V =  [[ 10.80096398  14.95238754  11.74639105]]
    t= 6 	V =  [[ 11.74639105  15.56699069  12.59257094]]
    t= 7 	V =  [[ 12.59257094  16.39001292  13.43180353]]
    t= 8 	V =  [[ 13.43180353  17.22762447  14.27054981]]
    t= 9 	V =  [[ 14.27054981  18.06625728  15.10926205]]
    t= 10 	V =  [[ 15.10926205  18.90496158  15.94797191]]
    t= 11 	V =  [[ 15.94797191  19.74367088  16.7866816 ]]
    t= 12 	V =  [[ 16.7866816   20.58238053  17.62539128]]
    t= 13 	V =  [[ 17.62539128  21.4210902   18.46410096]]
    t= 14 	V =  [[ 18.46410096  22.25979988  19.30281063]]
    t= 15 	V =  [[ 19.30281063  23.09850956  20.14152031]]
    t= 16 	V =  [[ 20.14152031  23.93721924  20.98022999]]
    t= 17 	V =  [[ 20.98022999  24.77592891  21.81893967]]
    t= 18 	V =  [[ 21.81893967  25.61463859  22.65764934]]
    t= 19 	V =  [[ 22.65764934  26.45334827  23.49635902]]
    t= 20 	V =  [[ 23.49635902  27.29205795  24.3350687 ]]
    t= 21 	V =  [[ 24.3350687   28.13076762  25.17377838]]
    t= 22 	V =  [[ 25.17377838  28.9694773   26.01248805]]
    t= 23 	V =  [[ 26.01248805  29.80818698  26.85119773]]
    t= 24 	V =  [[ 26.85119773  30.64689666  27.68990741]]
    t= 25 	V =  [[ 27.68990741  31.48560633  28.52861709]]
    t= 26 	V =  [[ 28.52861709  32.32431601  29.36732676]]
    t= 27 	V =  [[ 29.36732676  33.16302569  30.20603644]]
    t= 28 	V =  [[ 30.20603644  34.00173537  31.04474612]]
    t= 29 	V =  [[ 31.04474612  34.84044504  31.8834558 ]]
    t= 30 	V =  [[ 31.8834558   35.67915472  32.72216547]]
    t= 31 	V =  [[ 32.72216547  36.5178644   33.56087515]]
    t= 32 	V =  [[ 33.56087515  37.35657408  34.39958483]]
    t= 33 	V =  [[ 34.39958483  38.19528375  35.23829451]]
    t= 34 	V =  [[ 35.23829451  39.03399343  36.07700418]]
    t= 35 	V =  [[ 36.07700418  39.87270311  36.91571386]]
    t= 36 	V =  [[ 36.91571386  40.71141278  37.75442354]]
    t= 37 	V =  [[ 37.75442354  41.55012246  38.59313321]]
    t= 38 	V =  [[ 38.59313321  42.38883214  39.43184289]]
    t= 39 	V =  [[ 39.43184289  43.22754182  40.27055257]]
    t= 40 	V =  [[ 40.27055257  44.06625149  41.10926225]]
    t= 41 	V =  [[ 41.10926225  44.90496117  41.94797192]]
    t= 42 	V =  [[ 41.94797192  45.74367085  42.7866816 ]]
    t= 43 	V =  [[ 42.7866816   46.58238053  43.62539128]]
    t= 44 	V =  [[ 43.62539128  47.4210902   44.46410096]]
    t= 45 	V =  [[ 44.46410096  48.25979988  45.30281063]]
    t= 46 	V =  [[ 45.30281063  49.09850956  46.14152031]]
    t= 47 	V =  [[ 46.14152031  49.93721924  46.98022999]]
    t= 48 	V =  [[ 46.98022999  50.77592891  47.81893967]]
    t= 49 	V =  [[ 47.81893967  51.61463859  48.65764934]]
    t= 50 	V =  [[ 48.65764934  52.45334827  49.49635902]]

# Policy Iteration

* Start with an initial policy $\pi^0$.
* Iteratively,
    * Evaluate policy ($V^n(x) = V^{\pi^n}(x)$):
    $$ V^n(s) = R(s, a = \pi^n(s)) + \gamma \sum_{s^\prime} P(s^\prime | s, a = \pi^n(s)) V^{n}(s^\prime) $$
    * Improve policy:
    $$ \pi_{n+1}(s) = \arg \max_a \Big[ R(s, a) + \gamma \sum_{s^\prime} P(s^\prime | s, a) V^{n}(s^\prime) \Big] $$

* Stop condition:
    * Policy does not change anymore (for about 10 iterations)
    * The changes in evaluation of values are minor

```python
n_iter = 10
gamma = 1.
V = np.zeros(3)

pi = np.random.randint(2, size=n_states)

for n in range(n_iter):
    print("n = ", n, "\t", pi, end="")

    # evaluate
    for s in range(n_states):
        V[s] = Rs(s, pi[s]) + gamma * np.sum([Pf(s, sn, pi[s]) * V[sn] for sn in range(n_states)])
    print("\t", V)

    # improve
    for s in range(n_states):
        pi[s] = np.argmax([Rs(s, a) + gamma * np.sum([Pf(s, sn, a) * V[sn] for sn in range(n_states)])  for a in range(n_actions)] )
```

    n =  0 	 [0 0 1]	 [ 0.    3.5   0.75]
    n =  1 	 [1 0 1]	 [ 0.75    4.525   1.5825]
    n =  2 	 [1 0 1]	 [ 1.5825    5.37675   2.420775]
    n =  3 	 [1 0 1]	 [ 2.420775    6.2163725   3.25945425]
    n =  4 	 [1 0 1]	 [ 3.25945425  7.05514607  4.0981618 ]
    n =  5 	 [1 0 1]	 [ 4.0981618   7.89386023  4.93687133]
    n =  6 	 [1 0 1]	 [ 4.93687133  8.73257022  5.77558099]
    n =  7 	 [1 0 1]	 [ 5.77558099  9.57127992  6.61429067]
    n =  8 	 [1 0 1]	 [  6.61429067  10.40998959   7.45300035]
    n =  9 	 [1 0 1]	 [  7.45300035  11.24869927   8.29171002]

# Linear Programming

                                                                                [Manne '60]

\begin{equation*}
\begin{aligned}
& \underset{x}{\text{minimize}}
& & \sum_x V(x) \\
& \text{subject to}
& & V(x) \geq R(x, u) + \gamma \sum_{x^\prime} P(x^\prime | x, u) V^{n}(x^\prime) &\forall x, a.
\end{aligned}
\end{equation*}

# Temporal Difference Learning

When there are large number of states, the memory requirement grows exponentially.
*Temporal difference (TD) learning* considers that the agent knows only the partial information of the MDP.
With only current and next state transition and without any model transition probability, TD lets the agent explore the environment to examine the random policy.
With an estiate of the value function $V(s)$, $\hat{V}(s)$,

   $$
        \begin{align}
        V(s_t) &= R_{t+1} + \gamma V(s_{t+1}) \\
        V(s_t) &\sim R_{t+1} + \gamma \hat{V}(s_{t+1}) \\
        \Rightarrow \quad \delta_t &= R_{t+1} + \gamma \hat{V}(s_{t+1}) - V(s_t).
        \end{align}
    $$

Here, $\delta$ represents the *temporal diffrence error*.
We can use this error as a gradient to update the value estimation.

$$
    \begin{align}
        V(s_t) &\leftarrow R_{t+1} + \alpha \delta_t \\
        V(s_t) &\leftarrow R_{t+1} + \alpha (R_{t+1} + \gamma \hat{V}(s_{t+1}) - V(s_t))
    \end{align}
$$

## Example Model

```python
# model (problem to solve)
# a: 0 - loop  1 - to next state
# r:       +1                 +0      +100 when reaches 4
#
n_states = 5
n_actions = 2

def transition(s, a):
    s1 = s + a
    if s == s1:
        r = 1
    elif s1 == n_states-1:
        r = 100
    else:
        r = 0
    return s1, r

def pause_print():
    sys.stdout.write('\r')
    sys.stdout.write("\033[F")
    sys.stdout.flush()
    time.sleep(0.5)
```

```python
# Example TD(0) Update
alpha = 1.0
gamma = 0.9
V = np.zeros(n_states)  # 5 states
pi = np.random.randint(n_actions, size=n_states)

np.set_printoptions(precision=2)
for k in range(2):
    if k == 1:
        pi = np.ones(n_states, dtype=np.int)
    print("Policy:", pi)
    # Evaluation of the random policy
    for e in range(20):
        s = 0

        print("\r\tTraj: ", s, end=" ")
        for step in range(10):
            a = pi[s]
            s1, r1 = transition(s, a)
            V[s] += alpha * (r1 + gamma * V[s1] - V[s])
            s = s1

            print(a, s, end=" ")
            if s == n_states-1:
                break

        print("\t", V, end="\n")
        #pause_print()
    print()
```

    Policy: [0 0 0 1 0]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [ 6.51  0.    0.    0.    0.  ]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [ 8.78  0.    0.    0.    0.  ]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [ 9.58  0.    0.    0.    0.  ]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [ 9.85  0.    0.    0.    0.  ]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [ 9.95  0.    0.    0.    0.  ]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [ 9.98  0.    0.    0.    0.  ]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [ 9.99  0.    0.    0.    0.  ]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]
    	Traj:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 	 [10.  0.  0.  0.  0.]

    Policy: [1 1 1 1 1]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [   0.    0.    0.  100.    0.]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [   0.    0.   90.  100.    0.]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [   0.   81.   90.  100.    0.]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]
    	Traj:  0 1 1 1 2 1 3 1 4 	 [  72.9   81.    90.   100.     0. ]

# Control Problems and Q function

For control problems, we have defined Q function above to evaluate the state and action altogether.
Updating the Q values with TD learning is similar to previous update with two different considerations.
First, we update the Q with assumption that we follow a certain behavior policy. Thus, we call this as *on-policy control*, or **SARSA**.

$$
    Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha ( R_{t+1} + \gamma Q(s_{t+1}, a_{t+1}) - Q(s_t, a_t))
$$

Next, without making assumption of behavior policy, we can explore other possible policies to update the Q. We call this as *off-policy control*, or **Q-learning**.

$$
    Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha ( R_{t+1} + \gamma \max_a Q(s_{t+1}, a) - Q(s_t, a_t))
$$

Here describes the psedocode for each algorithm.

**[Algorithm: TD Learning]**

**[Algorithm: SARSA]**

**[Algorithm: Q-learning]**

# Choosing an Action

Picking an action can be simple by selecting one with maximum Q value, so *Greedy!*.

$$
a^* = \arg \max_a Q(S_t, a)
$$

However, this can cause limited experience to develop good Q estimation, and eventually a good policy.
Without new data, greedy action selection will repeat the same actions, or repeatedly *exploit* your current knowledge. Thus, you need to *explore* other non-greedy actions to increase the experience to improve the Q estimation.

This is called "exploration-exploitation dilemma."

One of the way for this dilemma is $\epsilon$-greedy action selection. With a parameter $\epsilon \in [0, 1]$, we can control the exploration and exploitation level. When $\epsilon = 0$, the actions are selected in greedy manner, but when $\epsilon = 1$, the actions are selected randomly.

```python
# action selection
def greedy(Q, s):
    return np.argmax(Q[s])  # greedy action selection

def e_greedy(Q, s, e):
    if np.random.rand() < e:
        return np.random.randint(n_actions)
    else:
        return greedy(Q,s)
```

```python
# SARSA example
alpha = 1.  # learning rate
gamma = 0.9 # discount factor
epsilon = 0.1

# tabular approximation
Q = np.random.rand(n_states, n_actions)  # 5 states and 2 actions

for e in range(20):
    s = 0
    a = e_greedy(Q, s, epsilon)  # greedy action selection

    for step in range(100):
        s1, r1 = transition(s, a)
        a1 = e_greedy(Q, s, epsilon)

        # TODO: update Q table here!

        #print("s: ", s, "a: ", a, "s1:", s1, "a1:", a1, "Q: ", Q[s,a])
        s, a = s1, a1
        if s == n_states-1:
            break

print("Final Q: ", Q)
print("Policy:", np.argmax(Q, axis=1))
```

    Final Q:  [[   2.42   73.  ]
     [  74.     81.12]
     [  82.12   90.13]
     [  91.13  100.14]
     [   0.33    0.16]]
    Policy: [1 1 1 1 0]

```python
# Q-learning example
alpha = 1.  # learning rate
gamma = 0.9 # discount factor
epsilon = 0.1

# tabular approximation
Q = np.random.rand(n_states, n_actions)  # 5 states and 2 actions

for e in range(20):
    s = 0

    for step in range(100):
        a = e_greedy(Q, s, epsilon)  # greedy action selection
        s1, r1 = transition(s, a)

        # TODO: update Q table here!

        print("s: ", s, "a: ", "s1:", s1, "a1:", a1, "Q: ", Q[s,a])
        s = s1
        if s == n_states-1:
            break

print("Final Q: ", Q)
print("Policy:", np.argmax(Q, axis=1))
```

    s:  0 a:  s1: 0 a1: 1 Q:  1.24006438385
    s:  0 a:  s1: 0 a1: 1 Q:  2.11605794546
    s:  0 a:  s1: 0 a1: 1 Q:  2.90445215091
    s:  0 a:  s1: 0 a1: 1 Q:  3.61400693582
    s:  0 a:  s1: 0 a1: 1 Q:  4.25260624224
    s:  0 a:  s1: 0 a1: 1 Q:  4.82734561802
    s:  0 a:  s1: 1 a1: 1 Q:  0.215898229533
    s:  1 a:  s1: 2 a1: 1 Q:  0.317105678191
    s:  2 a:  s1: 2 a1: 1 Q:  1.31710567819
    s:  2 a:  s1: 2 a1: 1 Q:  2.18539511037
    s:  2 a:  s1: 2 a1: 1 Q:  2.96685559933
    s:  2 a:  s1: 2 a1: 1 Q:  3.6701700394
    s:  2 a:  s1: 2 a1: 1 Q:  4.30315303546
    s:  2 a:  s1: 2 a1: 1 Q:  4.87283773191
    s:  2 a:  s1: 2 a1: 1 Q:  5.38555395872
    s:  2 a:  s1: 2 a1: 1 Q:  5.84699856285
    s:  2 a:  s1: 2 a1: 1 Q:  6.26229870657
    s:  2 a:  s1: 2 a1: 1 Q:  6.63606883591
    s:  2 a:  s1: 2 a1: 1 Q:  6.97246195232
    s:  2 a:  s1: 2 a1: 1 Q:  7.27521575709
    s:  2 a:  s1: 2 a1: 1 Q:  7.54769418138
    s:  2 a:  s1: 2 a1: 1 Q:  7.79292476324
    s:  2 a:  s1: 2 a1: 1 Q:  8.01363228692
    s:  2 a:  s1: 2 a1: 1 Q:  8.21226905822
    s:  2 a:  s1: 2 a1: 1 Q:  8.3910421524
    s:  2 a:  s1: 2 a1: 1 Q:  8.55193793716
    s:  2 a:  s1: 2 a1: 1 Q:  8.69674414345
    s:  2 a:  s1: 2 a1: 1 Q:  8.8270697291
    s:  2 a:  s1: 2 a1: 1 Q:  8.94436275619
    s:  2 a:  s1: 2 a1: 1 Q:  9.04992648057
    s:  2 a:  s1: 2 a1: 1 Q:  9.14493383251
    s:  2 a:  s1: 2 a1: 1 Q:  9.23044044926
    s:  2 a:  s1: 2 a1: 1 Q:  9.30739640434
    s:  2 a:  s1: 2 a1: 1 Q:  9.3766567639
    s:  2 a:  s1: 2 a1: 1 Q:  9.43899108751
    s:  2 a:  s1: 2 a1: 1 Q:  9.49509197876
    s:  2 a:  s1: 2 a1: 1 Q:  9.54558278089
    s:  2 a:  s1: 2 a1: 1 Q:  9.5910245028
    s:  2 a:  s1: 2 a1: 1 Q:  9.63192205252
    s:  2 a:  s1: 2 a1: 1 Q:  9.66872984727
    s:  2 a:  s1: 2 a1: 1 Q:  9.70185686254
    s:  2 a:  s1: 2 a1: 1 Q:  9.73167117629
    s:  2 a:  s1: 2 a1: 1 Q:  9.75850405866
    s:  2 a:  s1: 2 a1: 1 Q:  9.78265365279
    s:  2 a:  s1: 3 a1: 1 Q:  0.477446642928
    s:  3 a:  s1: 3 a1: 1 Q:  1.47744664293
    s:  3 a:  s1: 3 a1: 1 Q:  2.32970197863
    s:  3 a:  s1: 3 a1: 1 Q:  3.09673178077
    s:  3 a:  s1: 3 a1: 1 Q:  3.78705860269
    s:  3 a:  s1: 3 a1: 1 Q:  4.40835274242
    s:  3 a:  s1: 3 a1: 1 Q:  4.96751746818
    s:  3 a:  s1: 3 a1: 1 Q:  5.47076572136
    s:  3 a:  s1: 3 a1: 1 Q:  5.92368914923
    s:  3 a:  s1: 3 a1: 1 Q:  6.3313202343
    s:  3 a:  s1: 3 a1: 1 Q:  6.69818821087
    s:  3 a:  s1: 3 a1: 1 Q:  7.02836938979
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 0 a1: 1 Q:  5.34461105622
    s:  0 a:  s1: 0 a1: 1 Q:  5.81014995059
    s:  0 a:  s1: 0 a1: 1 Q:  6.22913495553
    s:  0 a:  s1: 0 a1: 1 Q:  6.60622145998
    s:  0 a:  s1: 1 a1: 1 Q:  0.285395110372
    s:  1 a:  s1: 2 a1: 1 Q:  8.80438828751
    s:  2 a:  s1: 2 a1: 1 Q:  9.80438828751
    s:  2 a:  s1: 2 a1: 1 Q:  9.82394945876
    s:  2 a:  s1: 2 a1: 1 Q:  9.84155451288
    s:  2 a:  s1: 2 a1: 1 Q:  9.8573990616
    s:  2 a:  s1: 2 a1: 1 Q:  9.87165915544
    s:  2 a:  s1: 2 a1: 1 Q:  9.88449323989
    s:  2 a:  s1: 2 a1: 1 Q:  9.8960439159
    s:  2 a:  s1: 2 a1: 1 Q:  9.90643952431
    s:  2 a:  s1: 2 a1: 1 Q:  9.91579557188
    s:  2 a:  s1: 2 a1: 1 Q:  9.92421601469
    s:  2 a:  s1: 2 a1: 1 Q:  9.93179441322
    s:  2 a:  s1: 2 a1: 1 Q:  9.9386149719
    s:  2 a:  s1: 2 a1: 1 Q:  9.94475347471
    s:  2 a:  s1: 2 a1: 1 Q:  9.95027812724
    s:  2 a:  s1: 2 a1: 1 Q:  9.95525031452
    s:  2 a:  s1: 2 a1: 1 Q:  9.95972528306
    s:  2 a:  s1: 2 a1: 1 Q:  9.96375275476
    s:  2 a:  s1: 2 a1: 1 Q:  9.96737747928
    s:  2 a:  s1: 2 a1: 1 Q:  9.97063973135
    s:  2 a:  s1: 2 a1: 1 Q:  9.97357575822
    s:  2 a:  s1: 2 a1: 1 Q:  9.9762181824
    s:  2 a:  s1: 2 a1: 1 Q:  9.97859636416
    s:  2 a:  s1: 2 a1: 1 Q:  9.98073672774
    s:  2 a:  s1: 2 a1: 1 Q:  9.98266305497
    s:  2 a:  s1: 2 a1: 1 Q:  9.98439674947
    s:  2 a:  s1: 2 a1: 1 Q:  9.98595707452
    s:  2 a:  s1: 2 a1: 1 Q:  9.98736136707
    s:  2 a:  s1: 2 a1: 1 Q:  9.98862523036
    s:  2 a:  s1: 2 a1: 1 Q:  9.98976270733
    s:  2 a:  s1: 2 a1: 1 Q:  9.99078643659
    s:  2 a:  s1: 2 a1: 1 Q:  9.99170779294
    s:  2 a:  s1: 2 a1: 1 Q:  9.99253701364
    s:  2 a:  s1: 2 a1: 1 Q:  9.99328331228
    s:  2 a:  s1: 2 a1: 1 Q:  9.99395498105
    s:  2 a:  s1: 2 a1: 1 Q:  9.99455948294
    s:  2 a:  s1: 2 a1: 1 Q:  9.99510353465
    s:  2 a:  s1: 2 a1: 1 Q:  9.99559318119
    s:  2 a:  s1: 2 a1: 1 Q:  9.99603386307
    s:  2 a:  s1: 2 a1: 1 Q:  9.99643047676
    s:  2 a:  s1: 2 a1: 1 Q:  9.99678742908
    s:  2 a:  s1: 2 a1: 1 Q:  9.99710868618
    s:  2 a:  s1: 2 a1: 1 Q:  9.99739781756
    s:  2 a:  s1: 2 a1: 1 Q:  9.9976580358
    s:  2 a:  s1: 2 a1: 1 Q:  9.99789223222
    s:  2 a:  s1: 2 a1: 1 Q:  9.998103009
    s:  2 a:  s1: 2 a1: 1 Q:  9.9982927081
    s:  2 a:  s1: 2 a1: 1 Q:  9.99846343729
    s:  2 a:  s1: 2 a1: 1 Q:  9.99861709356
    s:  2 a:  s1: 2 a1: 1 Q:  9.9987553842
    s:  2 a:  s1: 2 a1: 1 Q:  9.99887984578
    s:  2 a:  s1: 2 a1: 1 Q:  9.99899186121
    s:  2 a:  s1: 2 a1: 1 Q:  9.99909267509
    s:  2 a:  s1: 2 a1: 1 Q:  9.99918340758
    s:  2 a:  s1: 2 a1: 1 Q:  9.99926506682
    s:  2 a:  s1: 2 a1: 1 Q:  9.99933856014
    s:  2 a:  s1: 2 a1: 1 Q:  9.99940470412
    s:  2 a:  s1: 2 a1: 1 Q:  9.99946423371
    s:  2 a:  s1: 2 a1: 1 Q:  9.99951781034
    s:  2 a:  s1: 2 a1: 1 Q:  9.99956602931
    s:  2 a:  s1: 2 a1: 1 Q:  9.99960942638
    s:  2 a:  s1: 2 a1: 1 Q:  9.99964848374
    s:  2 a:  s1: 2 a1: 1 Q:  9.99968363536
    s:  2 a:  s1: 2 a1: 1 Q:  9.99971527183
    s:  2 a:  s1: 2 a1: 1 Q:  9.99974374464
    s:  2 a:  s1: 2 a1: 1 Q:  9.99976937018
    s:  2 a:  s1: 2 a1: 1 Q:  9.99979243316
    s:  2 a:  s1: 2 a1: 1 Q:  9.99981318985
    s:  2 a:  s1: 2 a1: 1 Q:  9.99983187086
    s:  2 a:  s1: 2 a1: 1 Q:  9.99984868378
    s:  2 a:  s1: 2 a1: 1 Q:  9.9998638154
    s:  2 a:  s1: 2 a1: 1 Q:  9.99987743386
    s:  2 a:  s1: 2 a1: 1 Q:  9.99988969047
    s:  2 a:  s1: 2 a1: 1 Q:  9.99990072143
    s:  2 a:  s1: 2 a1: 1 Q:  9.99991064928
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 0 a1: 1 Q:  6.94559931398
    s:  0 a:  s1: 0 a1: 1 Q:  7.25103938258
    s:  0 a:  s1: 0 a1: 1 Q:  7.52593544433
    s:  0 a:  s1: 0 a1: 1 Q:  7.77334189989
    s:  0 a:  s1: 0 a1: 1 Q:  7.9960077099
    s:  0 a:  s1: 0 a1: 1 Q:  8.19640693891
    s:  0 a:  s1: 0 a1: 1 Q:  8.37676624502
    s:  0 a:  s1: 0 a1: 1 Q:  8.53908962052
    s:  0 a:  s1: 1 a1: 1 Q:  7.92394945876
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 0 a1: 1 Q:  8.68518065847
    s:  0 a:  s1: 0 a1: 1 Q:  8.81666259262
    s:  0 a:  s1: 0 a1: 1 Q:  8.93499633336
    s:  0 a:  s1: 0 a1: 1 Q:  9.04149670002
    s:  0 a:  s1: 0 a1: 1 Q:  9.13734703002
    s:  0 a:  s1: 0 a1: 1 Q:  9.22361232702
    s:  0 a:  s1: 0 a1: 1 Q:  9.30125109432
    s:  0 a:  s1: 0 a1: 1 Q:  9.37112598489
    s:  0 a:  s1: 0 a1: 1 Q:  9.4340133864
    s:  0 a:  s1: 0 a1: 1 Q:  9.49061204776
    s:  0 a:  s1: 0 a1: 1 Q:  9.54155084298
    s:  0 a:  s1: 0 a1: 1 Q:  9.58739575868
    s:  0 a:  s1: 0 a1: 1 Q:  9.62865618281
    s:  0 a:  s1: 0 a1: 1 Q:  9.66579056453
    s:  0 a:  s1: 0 a1: 1 Q:  9.69921150808
    s:  0 a:  s1: 0 a1: 1 Q:  9.72929035727
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 1 a1: 1 Q:  74.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 3 a1: 1 Q:  91.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 1 a1: 1 Q:  74.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    s:  0 a:  s1: 1 a1: 1 Q:  73.0558878812
    s:  1 a:  s1: 2 a1: 1 Q:  81.1732087569
    s:  2 a:  s1: 3 a1: 1 Q:  90.1924541743
    s:  3 a:  s1: 4 a1: 1 Q:  100.213837971
    Final Q:  [[  9.73e+00   7.31e+01]
     [  7.41e+01   8.12e+01]
     [  1.00e+01   9.02e+01]
     [  9.12e+01   1.00e+02]
     [  2.38e-01   5.53e-02]]
    Policy: [1 1 1 1 0]

# Maze Example

Now, let us solve some maze problems. To deal with various maze configuration, we define our class that can handle a file input of maze. Here the file is expected to have 3 letters: 'O' for open space, 'H' for obstacles (walls), and 'G' for a goal position. The example grid.txt file looks as follows.

Note: this bash command runs only on Linux or Mac. The shell is to show the content of file only. You can skip running the cell and creat a file with the same content.

```bash
%%bash
cat grid.txt
```

    OOOHOOOOO
    OOOHOOHOO
    OOOOOOHOO
    OOOOHHHOO
    OOHOOOOOH
    OOHOOOGOO
    OOOOOOOOO

Now, let us define the environment class. The environment constructor gets only one input, the filename. Along with some utility functions, the main simulation functions are init() and next() for initiating and running one step simulation. Please take a look at the example use and try them in a separate cell.

```python
# maze example

class GridWorld:
    """ Grid World environment
            there are four actions (left, right, up, and down) to move an agent
            In a grid, if it reaches a goal, it get 30 points of reward.
            If it falls in a hole or moves out of the grid world, it gets -5.
            Each step costs -1 point.

        to test GridWorld, run the following sample codes:

            env = GridWorld('grid.txt')

            env.print_map()
            print [2,3], env.check_state([2,3])
            print [0,0], env.check_state([0,0])
            print [3,4], env.check_state([3,4])
            print [10,3], env.check_state([10,3])

            env.init([0,0])
            print env.next(1)  # right
            print env.next(3)  # down
            print env.next(0)  # left
            print env.next(2)  # up
            print env.next(2)  # up

        Parameters
        ==========
        _map        ndarray
                    string array read from a file input
        _size       1d array
                    the size of _map in ndarray
        goal_pos    tuple
                    the index for the goal location
        _actions    list
                    list of actions for 4 actions
        _s          1d array
                    current state
    """
    def __init__(self, fn):
        # read a map from a file
        self._map = self.read_map(fn)
        self._size = np.asarray(self._map.shape)
        self.goal_pos = np.where(self._map == 'G')

        # definition of actions (left, right, up, and down repectively)
        self._actions = [[0, -1], [0, 1], [-1, 0], [1, 0]]
        self._s = None

    def get_cur_state(self):
        return self._s

    def get_size(self):
        return self._size

    def read_map(self, fn):
        grid = []
        with open(fn) as f:
            for line in f:
               grid.append(list(line.strip()))
        return np.asarray(grid)

    def print_map(self):
        print( self._map )

    def check_state(self, s):
        if isinstance(s, collections.Iterable) and len(s) == 2:
            if s[0] < 0 or s[1] < 0 or\
               s[0] >= self._size[0] or s[1] >= self._size[1]:
               return 'N'
            return self._map[tuple(s)].upper()
        else:
            return 'F'  # wrong input

    def init(self, state=None):
        if state is None:
            s = [0, 0]
        else:
            s = state

        if self.check_state(s) == 'O':
            self._s = np.asarray(state)
        else:
            raise ValueError("Invalid state for init")

    def next(self, a):
        s1 = self._s + self._actions[a]
        # state transition
        curr = self.check_state(s1)

        if curr == 'H' or curr == 'N':
            return -5
        elif curr == 'F':
            warnings.warn("invalid state " + str(s1))
            return -5
        elif curr == 'G':
            self._s = s1
            return 30
        else:
            self._s = s1
            return -1

    def is_goal(self):
        return self.check_state(self._s) == 'G'

    def get_actions(self):
        return self._actions
```

```python
env = GridWorld("grid.txt")
env.print_map()
```

    [['O' 'O' 'O' 'H' 'O' 'O' 'O' 'O' 'O']
     ['O' 'O' 'O' 'H' 'O' 'O' 'H' 'O' 'O']
     ['O' 'O' 'O' 'O' 'O' 'O' 'H' 'O' 'O']
     ['O' 'O' 'O' 'O' 'H' 'H' 'H' 'O' 'O']
     ['O' 'O' 'H' 'O' 'O' 'O' 'O' 'O' 'H']
     ['O' 'O' 'H' 'O' 'O' 'O' 'G' 'O' 'O']
     ['O' 'O' 'O' 'O' 'O' 'O' 'O' 'O' 'O']]

## Agent

Let us define an RL agent class. Two main methods are train() and test() as other ML classes. Unlike supervised learining, train function actively collect data while interacting with an environment and update its Q estimation with SARSA (or Q-learning). The next cell defines a utility to adjust coordianates to start from left-top.

```python
def coord_convert(s, sz):
    return [s[1], sz[0]-s[0]-1]
```

```python
class RLAgent:
    """
        Reinforcement Learning Agent Model for training/testing
        with Tabular function approximation

    """

    def __init__(self, evn):
        self.env = env
        self.size = env.get_size()
        self.n_a = len(env.get_actions())
        # self.Q table including the surrounding border
        self.Q = np.zeros((self.size[0], self.size[1], self.n_a))

    def epsilon_greed(self, epsilon, s):
        if np.random.uniform() < epsilon:
            a = np.random.randint(self.n_a)
        else:
            i_max = np.where(self.Q[s[0], s[1], :] == np.max(self.Q[s[0], s[1], :]))[0]
            a = int(np.random.choice(i_max))
        return a

    def train(self, start, **params):

        # parameters
        gamma = params.pop('gamma', 0.99)
        alpha = params.pop('alpha', 0.1)
        epsilon= params.pop('epsilon', 0.1)
        maxiter= params.pop('maxiter', 1000)
        maxstep= params.pop('maxstep', 1000)

        # init self.Q matrix
        self.Q[...] = 0
        self.Q[self.env._map == 'H'] = -np.inf

        # online train
        # rewards and step trace
        rtrace = []
        steps = []
        for j in range(maxiter):

            env.init(start)
            s = env.get_cur_state()
            # selection an action
            a = self.epsilon_greed(epsilon, s)

            rewards = []
            trace = np.array(coord_convert(s, self.size))
            # run simulation for max number of steps
            for step in range(maxstep):
                # move
                r = env.next(a)
                s1 = env.get_cur_state()
                a1 = self.epsilon_greed(epsilon, s1)

                rewards.append(r)
                trace = np.vstack((trace, coord_convert(s1, self.size)))

                # ToDo: update self.Q table (SARSA)

                if env.is_goal(): # reached the goal
                    # ToDo: update Q table

                    break

                s = s1
                a = a1

            rtrace.append(np.sum(rewards))
            steps.append(step+1)
        return rtrace, steps, trace # last trace of trajectory

    def test(self, start, maxstep=1000):

        #### ToDo: Here is one episode of simulation for testing.
        ####       Add one line to make this code test the trained agent

        env.init(start)
        s = env.get_cur_state()
        # selection an action
        a = self.epsilon_greed(epsilon, s)

        trace = np.array(coord_convert(s, self.size))
        # run simulation for max number of steps
        for step in range(maxstep):
            # move
            r = env.next(a)
            s1 = env.get_cur_state()
            a1 = self.epsilon_greed(epsilon, s1)

            trace = np.vstack((trace, coord_convert(s1, self.size)))

            if env.is_goal(): # reached the goal
                break

            s = s1
            a = a1

        return trace
```

```python
### Plotting tools

def plot_trace(agent, start, trace, title="test trajectory"):
    plt.plot(trace[:, 0], trace[:, 1], "ko-")
    plt.text(env.goal_pos[1], agent.size[0]-env.goal_pos[0]-1, 'G')
    plt.text(start[1], agent.size[0]-start[0]-1, 'S')
    plt.xlim([0, agent.size[1]])
    plt.ylim([0, agent.size[0]])

def plot_train(agent, rtrace, steps, trace, start):

    fig = plt.figure(figsize=(10,8))
    ax = fig.add_subplot(221)
    plt.plot(rtrace, "b-")
    plt.ylabel("sum of rewards")

    ax1 = fig.add_subplot(222)
    plt.plot(steps)
    plt.ylabel("# steps")

    # contour plot for agent.Q
    ax2 = fig.add_subplot(223)
    xs = range(agent.size[1])
    ys = range(agent.size[0])
    maxQ = np.max(agent.Q, axis=2)
    h_b = (maxQ==-np.inf)
    maxQ[h_b] = 0
    maxQ[h_b] = np.min(maxQ) - 100
    cs = plt.contourf(xs, ys[::-1], maxQ)
    plt.colorbar(cs)
    plt.text(env.goal_pos[1], agent.size[0]-env.goal_pos[0]-1, 'G')
    plt.text(start[1], agent.size[0]-start[0]-1, 'S')
    plt.ylabel("max agent.Q")

    # plot traces
    ax3 = fig.add_subplot(224)
    plot_trace(agent, start, trace, "trace of the last episode")

    plt.plot()
```

```python
agent = RLAgent(env)
start = [0,0]
rtrace, steps, trace = agent.train(start,
                                   amma=0.99,
                                   alpha=0.1,
                                   epsilon=0.1,
                                   maxiter=100,
                                   maxstep=1000)
```

```python
plot_train(agent, rtrace, steps, trace, start)
```

```python
test_start = [0,2]
test_trace = agent.test(test_start)
plot_trace(agent, test_start, test_trace)
```

```python
test_start = [1,8]
test_trace = agent.test(test_start)
plot_trace(agent, test_start, test_trace)
```

Q: What do you think about the test results above? Does the agent run fine?


Lambda School Data Science, Unit 2: Predictive Modeling

# Regression & Classification, Module 2

## Objectives
- Go from simple regression (1 feature) to multiple regression (2+ features)
- Use regression metrics: MAE (Mean Absolute Error), RMSE (Root Mean Squared Error), and $R^2$ Score
- Understand how ordinary least squares regression minimizes the sum of squared errors
- Get and plot coefficients
- Explain why overfitting is a problem. Do train/test split

## Setup

```python
# If you're in Colab...
import os, sys
in_colab = 'google.colab' in sys.modules

if in_colab:
    # Install required python packages:
    # pandas-profiling, version >= 2.0
    # plotly, version >= 4.0
    !pip install --upgrade pandas-profiling plotly

    # Pull files from Github repo
    os.chdir('/content')
    !git init .
    !git remote add origin https://github.com/LambdaSchool/DS-Unit-2-Regression-Classification.git
    !git pull origin master

    # Change into directory for module
    os.chdir('module2')
```

```python
# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
```

```python
# Use these functions later

%matplotlib inline
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import numpy as np
import plotly.express as px
import plotly.graph_objs as go
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

def regression_3d(df, x, y, z, **kwargs):
    """
    Visualize linear regression in 3D: 2 features + 1 target

    df : Pandas DataFrame
    x : string, feature 1 column in df
    y : string, feature 2 column in df
    z : string, target column in df
    """

    # Plot data
    fig = px.scatter_3d(df, x, y, z, **kwargs)

    # Fit Linear Regression
    features = [x, y]
    target = z
    model = LinearRegression()
    model.fit(df[features], df[target])

    # Define grid of four points in the feature space
    xmin, xmax = df[x].min(), df[x].max()
    ymin, ymax = df[y].min(), df[y].max()
    coords = [[xmin, ymin],
              [xmin, ymax],
              [xmax, ymin],
              [xmax, ymax]]

    # Make predictions for the grid
    Z = model.predict(coords).reshape((2,2), order='F')

    # Plot predictions as a 3D surface (plane)
    fig.add_trace(go.Surface(x=[xmin,xmax], y=[ymin,ymax], z=Z))

    return fig

def regression_residuals(df, feature, target, m, b):
    """
    Visualize linear regression, with residual errors,
    in 2D: 1 feature + 1 target.

    Use the m & b parameters to "fit the model" manually.

    df : Pandas DataFrame
    feature : string, feature column in df
    target : string, target column in df
    m : numeric, slope for linear equation
    b : numeric, intercept for linear requation
    """

    # Plot data
    df.plot.scatter(feature, target)

    # Make predictions
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    # Plot predictions
    plt.plot(x, y_pred)

    # Plot residual errors
    for x, y1, y2 in zip(x, y, y_pred):
        plt.plot((x, x), (y1, y2), color='grey')

    # Print regression metrics
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

def regression_squared_errors(df, feature, target, m, b):
    """
    Visualize linear regression, with squared errors,
    in 2D: 1 feature + 1 target.

    Use the m & b parameters to "fit the model" manually.

    df : Pandas DataFrame
    feature : string, feature column in df
    target : string, target column in df
    m : numeric, slope for linear equation
    b : numeric, intercept for linear requation
    """

    # Plot data
    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    df.plot.scatter(feature, target, ax=ax)

    # Make predictions
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    # Plot predictions
    ax.plot(x, y_pred)

    # Plot squared errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    # Print regression metrics
    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

# Credit: Jake VanderPlas, Python Data Science Handbook, Chapter 5.3
# https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#Validation-curves-in-Scikit-Learn
def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree),
                         LinearRegression(**kwargs))
```

# Predict Elections! 🗳

```python
import pandas as pd
import plotly.express as px

df = pd.read_csv('../data/bread_peace_voting.csv')
px.scatter(
    df,
    x='Average Recent Growth in Personal Incomes',
    y='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2016',
    trendline='ols',
)
```

<meta charset="utf-8" />

#### Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:

> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

#### Data sources
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33

```python
# Look at the data
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Incumbent Party Vote Share

      0
      1952
      Stevenson
      Eisenhower
      2.40
      190
      44.60

      1
      1956
      Eisenhower
      Stevenson
      2.89
      0
      57.76

      2
      1960
      Nixon
      Kennedy
      0.85
      0
      49.91

      3
      1964
      Johnson
      Goldwater
      4.21
      1
      61.34

      4
      1968
      Humphrey
      Nixon
      3.02
      146
      49.60

      5
      1972
      Nixon
      McGovern
      3.62
      0
      61.79

      6
      1976
      Ford
      Carter
      1.08
      2
      48.95

      7
      1980
      Carter
      Reagan
      -0.39
      0
      44.70

      8
      1984
      Reagan
      Mondale
      3.86
      0
      59.17

      9
      1988
      Bush, Sr.
      Dukakis
      2.27
      0
      53.94

      10
      1992
      Bush, Sr.
      Clinton
      0.38
      0
      46.55

      11
      1996
      Clinton
      Dole
      1.04
      0
      54.74

      12
      2000
      Gore
      Bush, Jr.
      2.36
      0
      50.27

      13
      2004
      Bush, Jr.
      Kerry
      1.72
      4
      51.24

      14
      2008
      McCain
      Obama
      0.10
      14
      46.32

      15
      2012
      Obama
      Romney
      0.95
      5
      52.00

      16
      2016
      Clinton
      Trump
      0.10
      5
      48.20

```python
df['Incumbent Party Vote Share'].mean()
```

    51.82823529411765

```python
# Add another feature to the scatterplot
px.scatter_3d(
    df,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2016'
)
```

<meta charset="utf-8" />

```python
# Fit a multiple regression model and plot the hyperplane of best fit
regression_3d(
    df,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2016'
)
```

<meta charset="utf-8" />

```python
# How's the error?

model = LinearRegression()

# Assign X matrix & y vector
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']
target = 'Incumbent Party Vote Share'

X = df[features]
y = df[target]

model.fit(X, y)
y_pred = model.predict(X)

df['Predicted'] = y_pred
df['Error'] = y_pred - y

df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Incumbent Party Vote Share
      Predicted
      Error

      0
      1952
      Stevenson
      Eisenhower
      2.40
      190
      44.60
      44.847507
      0.247507

      1
      1956
      Eisenhower
      Stevenson
      2.89
      0
      57.76
      56.729476
      -1.030524

      2
      1960
      Nixon
      Kennedy
      0.85
      0
      49.91
      49.780799
      -0.129201

      3
      1964
      Johnson
      Goldwater
      4.21
      1
      61.34
      61.171926
      -0.168074

      4
      1968
      Humphrey
      Nixon
      3.02
      146
      49.60
      49.324458
      -0.275542

      5
      1972
      Nixon
      McGovern
      3.62
      0
      61.79
      59.216012
      -2.573988

      6
      1976
      Ford
      Carter
      1.08
      2
      48.95
      50.456724
      1.506724

      7
      1980
      Carter
      Reagan
      -0.39
      0
      44.70
      45.557093
      0.857093

      8
      1984
      Reagan
      Mondale
      3.86
      0
      59.17
      60.033503
      0.863503

      9
      1988
      Bush, Sr.
      Dukakis
      2.27
      0
      53.94
      54.617623
      0.677623

      10
      1992
      Bush, Sr.
      Clinton
      0.38
      0
      46.55
      48.179878
      1.629878

      11
      1996
      Clinton
      Dole
      1.04
      0
      54.74
      50.427980
      -4.312020

      12
      2000
      Gore
      Bush, Jr.
      2.36
      0
      50.27
      54.924182
      4.654182

      13
      2004
      Bush, Jr.
      Kerry
      1.72
      4
      51.24
      52.529196
      1.289196

      14
      2008
      McCain
      Obama
      0.10
      14
      46.32
      46.473607
      0.153607

      15
      2012
      Obama
      Romney
      0.95
      5
      52.00
      49.852659
      -2.147341

      16
      2016
      Clinton
      Trump
      0.10
      5
      48.20
      46.957377
      -1.242623

```python
df['Error'].mean()
```

    -3.761696836377001e-15

```python
df['Error'].abs().mean()
```

    1.3975663494016117

```python
from sklearn.metrics import mean_absolute_error
mean_absolute_error(y, y_pred)
```

    1.3975663494016117

```python
# How does this compare to guessing?

# What was the average incumbent party vote share?
y.mean()
```

    51.82823529411765

```python
# What if we guessed this number for every elecetion?
guesses = [y.mean()] * len(y)

# How far off would this be on average?
mean_absolute_error(y, guesses)
```

    4.463806228373702

```python
feature = 'Average Recent Growth in Personal Incomes'
regression_residuals(df, feature, target, m=0, b=y.mean())
```

```python
regression_residuals(df, feature, target, m=3, b=46)
```

```python
from ipywidgets import interact, fixed

interact(
    regression_residuals,
    df=fixed(df),
    feature=fixed(feature),
    target=fixed(target),
    m=(-5,5,0.5),
    b=(40,60,0.5)
);
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=5.0, min=-5.0, step=0.5), FloatSlider(value=…

```python
# But Ordinary Least Squares Regression *doesn't* directly minimize MAE or R^2...
```

```python
regression_squared_errors(df, feature, target, m=3, b=46)
```

## Linear Algebra

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

\begin{align}
\hat{\beta} = (X^{T}X)^{-1}X^{T}y
\end{align}

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

### Lets calculate our $\beta$ coefficients with numpy!

```python
# This is NOT an objective you'll be tested on. It's just a demo.

# X is a matrix. Add constant for the intercept.
from statsmodels.api import add_constant
X = add_constant(df[feature].values)
print('X')
print(X)

# y is a column vector
y = df[target].values[:, np.newaxis]
print('y')
print(y)

# Least squares solution in code
X_transpose = X.T
X_transpose_X = X_transpose @ X
X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
X_transpose_y = X_transpose @ y
beta_hat = X_transpose_X_inverse @ X_transpose_y

print('Beta Hat')
print(beta_hat)
```

    X
    [[ 1.    2.4 ]
     [ 1.    2.89]
     [ 1.    0.85]
     [ 1.    4.21]
     [ 1.    3.02]
     [ 1.    3.62]
     [ 1.    1.08]
     [ 1.   -0.39]
     [ 1.    3.86]
     [ 1.    2.27]
     [ 1.    0.38]
     [ 1.    1.04]
     [ 1.    2.36]
     [ 1.    1.72]
     [ 1.    0.1 ]
     [ 1.    0.95]
     [ 1.    0.1 ]]
    y
    [[44.6 ]
     [57.76]
     [49.91]
     [61.34]
     [49.6 ]
     [61.79]
     [48.95]
     [44.7 ]
     [59.17]
     [53.94]
     [46.55]
     [54.74]
     [50.27]
     [51.24]
     [46.32]
     [52.  ]
     [48.2 ]]
    Beta Hat
    [[46.49920976]
     [ 2.97417709]]

```python
# Scikit-learn gives the exact same results
from sklearn.linear_model import LinearRegression
model = LinearRegression()
features = ['Average Recent Growth in Personal Incomes']
target = 'Incumbent Party Vote Share'
X = df[features]
y = df[target]
model.fit(X, y)
model.intercept_, model.coef_
```

    (46.499209757741625, array([2.97417709]))

```python
# Fit a multiple regression model and plot the hyperplane of best fit
regression_3d(
    df,
    x='Average Recent Growth in Personal Incomes',
    y='US Military Fatalities per Million',
    z='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2016'
)
```

<meta charset="utf-8" />

```python
# What's the equation for the hyperplane?
# Can you relate the intercept and coefficients to what you see in the plot?
from sklearn.linear_model import LinearRegression
model = LinearRegression()
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']
target = 'Incumbent Party Vote Share'
X = df[features]
y = df[target]
model.fit(X, y)
model.intercept_, model.coef_
```

    (46.88551694148949, array([ 3.40621407, -0.05375223]))

```python
# This is easier to read
print('Intercept', model.intercept_)
coefficients = pd.Series(model.coef_, features)
print(coefficients.to_string())
```

    Intercept 46.88551694148949
    Average Recent Growth in Personal Incomes    3.406214
    US Military Fatalities per Million          -0.053752

```python
# What if ...

# Income growth = 0%, fatalities = 0
model.predict([[0, 0]])
```

    array([46.88551694])

```python
# Income growth = 1% (fatalities = 0)
model.predict([[1, 0]])
```

    array([50.29173101])

```python
# The difference between these predictions = income coefficient
model.predict([[1, 0]]) - model.predict([[0, 0]])
```

    array([3.40621407])

```python
# What if... income growth = 2% (fatalities = 0)
model.predict([[2, 0]])
```

    array([53.69794508])

```python
# The difference between these predictions = income coefficient
model.predict([[2, 0]]) - model.predict([[1, 0]])
```

    array([3.40621407])

```python
# What if... (income growth=2%) fatalities = 100
model.predict([[2, 100]])
```

    array([48.32272205])

```python
# The difference between these predictions = 100 * fatalities coefficient
model.predict([[2, 100]]) - model.predict([[2, 0]])
```

    array([-5.37522303])

```python
# What if income growth = 3% (fatalities = 100)
model.predict([[3, 100]])
```

    array([51.72893612])

```python
# The difference between these predictions = income coefficient
model.predict([[3, 100]]) - model.predict([[2, 100]])
```

    array([3.40621407])

```python
# What if (income growth = 3%) fatalities = 200
model.predict([[3, 200]])
```

    array([46.35371309])

```python
# The difference between these predictions = 100 * fatalities coefficient
model.predict([[3, 200]]) - model.predict([[3, 100]])
```

    array([-5.37522303])

```python
# Plot coefficients
%matplotlib inline
coefficients.sort_values().plot.barh(color='grey');
```

```python
# Note that the features are on different scales,
# so the coefficients are on different scales too
df[features].describe()
```

      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million

      count
      17.000000
      17.000000

      mean
      1.791765
      21.588235

      std
      1.419812
      55.767440

      min
      -0.390000
      0.000000

      25%
      0.850000
      0.000000

      50%
      1.720000
      0.000000

      75%
      2.890000
      5.000000

      max
      4.210000
      190.000000

## Overfitting Demo

```python
# Read New York City property sales data, from first 4 months of 2019.
# Dataset has 23040 rows, 21 columns.
df = pd.read_csv('../data/NYC_Citywide_Rolling_Calendar_Sales.csv')
assert df.shape == (23040, 21)

# Change column names. Replace spaces with underscores
df.columns = [col.replace(' ', '_') for col in df]

# Remove symbols from SALE_PRICE string, convert to integer
df['SALE_PRICE'] = (
    df['SALE_PRICE']
    .str.replace('$','')
    .str.replace('-','')
    .str.replace(',','')
    .astype(int)
)

# Keep subset of rows:
# Tribeca neighborhood, Condos - Elevator Apartments,
# 1 unit, sale price more than $1, less than $35 million
mask = (
    (df['NEIGHBORHOOD'].str.contains('TRIBECA')) &
    (df['BUILDING_CLASS_CATEGORY'] == '13 CONDOS - ELEVATOR APARTMENTS') &
    (df['TOTAL_UNITS'] == 1) &
    (df['SALE_PRICE'] > 0) &
    (df['SALE_PRICE'] < 35000000)
)
df = df[mask]

# Data now has 90 rows, 21 columns
assert df.shape == (90, 21)

# Convert SALE_DATE to datetime
df['SALE_DATE'] = pd.to_datetime(df['SALE_DATE'], infer_datetime_format=True)

from ipywidgets import interact
import pandas as pd
from sklearn.linear_model import LinearRegression

# Read New York City property sales data, from first 4 months of 2019.
# Dataset has 23040 rows, 21 columns.
df = pd.read_csv('../data/NYC_Citywide_Rolling_Calendar_Sales.csv')
assert df.shape == (23040, 21)

# Change column names. Replace spaces with underscores
df.columns = [col.replace(' ', '_') for col in df]

# Remove symbols from SALE_PRICE string, convert to integer
df['SALE_PRICE'] = (
    df['SALE_PRICE']
    .str.replace('$','')
    .str.replace('-','')
    .str.replace(',','')
    .astype(int)
)

# Keep subset of rows:
# Tribeca neighborhood, Condos - Elevator Apartments,
# 1 unit, sale price more than $1, less than $35 million
mask = (
    (df['NEIGHBORHOOD'].str.contains('TRIBECA')) &
    (df['BUILDING_CLASS_CATEGORY'] == '13 CONDOS - ELEVATOR APARTMENTS') &
    (df['TOTAL_UNITS'] == 1) &
    (df['SALE_PRICE'] > 0) &
    (df['SALE_PRICE'] < 35000000)
)
df = df[mask]

# Data now has 90 rows, 21 columns
assert df.shape == (90, 21)

# Convert SALE_DATE to datetime
df['SALE_DATE'] = pd.to_datetime(df['SALE_DATE'], infer_datetime_format=True)

# Arrange X features matrix & y target vector
features = ['GROSS_SQUARE_FEET']
target = 'SALE_PRICE'
X = df[features]
y = df[target]
```

```python
from IPython.display import display, HTML
from sklearn.model_selection import train_test_split

# Train/Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=11)

# Repeatedly fit increasingly complex models, and keep track of the scores
polynomial_degrees = range(1, 10, 2)
train_r2s = []
test_r2s = []

for degree in polynomial_degrees:
    model = PolynomialRegression(degree)
    display(HTML(f'Polynomial degree={degree}'))

    model.fit(X_train, y_train)
    train_r2 = model.score(X_train, y_train)
    test_r2 = model.score(X_test, y_test)
    display(HTML(f'<b style="color: blue">Train R2 {train_r2:.2f}</b>'))
    display(HTML(f'<b style="color: red">Test R2 {test_r2:.2f}</b>'))

    plt.scatter(X_train, y_train, color='blue', alpha=0.5)
    plt.scatter(X_test, y_test, color='red', alpha=0.5)
    plt.xlabel(features)
    plt.ylabel(target)

    x_domain = np.linspace(X.min(), X.max())
    curve = model.predict(x_domain)
    plt.plot(x_domain, curve, color='blue')
    plt.show()
    display(HTML('<hr/>'))

    train_r2s.append(train_r2)
    test_r2s.append(test_r2)

display(HTML('Validation Curve'))
plt.plot(polynomial_degrees, train_r2s, color='blue', label='Train')
plt.plot(polynomial_degrees, test_r2s, color='red', label='Test')
plt.xlabel('Model Complexity (Polynomial Degree)')
plt.ylabel('R^2 Score')
plt.legend()
plt.show()
```

#### Jake VanderPlas, [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off)