# Bayesian decision theory

```python
import numpy as np
np.set_printoptions(precision=3)

import pandas as pd
```

# From beliefs to actions

* Probabilistic inference let us compute our distribution over the possible **states of nature** $p(H=h|X=x)$; this is called our **belief state**. But ultimately we must convert beliefs into actions.
* The optimal way to make **decisions under uncertainty** is to use **decision theory**.
* If the **decision maker** or **agent** picks action $a \in \cal{A}$ when the true state of nature is $h \in \cal{H}$, then we assume they incur a **loss** of $\ell(h,a)$.
* Since the state is usually hidden, we must compute the  the **expected loss** or **risk** for each action:
$$
R(a|x) = \sum_{h \in \cal{H}} p(h|x) \ell(h,a)
$$
* The **optimal policy** specifies which action to take for each possible observation so as to minimize this risk:
$$
a^* = \pi^*(x)  = \arg \min_{a \in \cal{A}}
 \sum_{h \in \cal{H}} p(h|x) \ell(h,a)
$$
* Equivalently, we can **maximize expected utility**
$$
a^* = \pi^*(x) = \arg \max_{a \in \cal{A}} \sum_{h \in \cal{H}} p(h|x) U(h,a)
$$

# Example:  Optimal policy for treating COVID patients

* Suppose we are a doctor who can either do nothing or give an expensive and painful drug to a patient.
* Suppose the patient either has or does not have COVID, and is either young or old.
* Suppose the loss function $\ell(h,a)$ is as shown below, where the units are [Quality Adjusted Life Years](https://en.wikipedia.org/wiki/Quality-adjusted_life_year#:~:text=The%20quality%2Dadjusted%20life%20year,one%20year%20in%20perfect%20health.). This matrix encodes the assumption that a young person is likely to live much longer than an old person, and that the drug is so bad that it is like losing 8 (quality-adjusted) years of life. (We will vary these assumptions later.)

* We assume the age is a visible variable, but the COVID status is hidden.
* The doctor can use Bayes rule to infer $p(h|x)$, where $x$ is a diagnostic test.
* Given the belief state, we can compute the optimal policy, which is shown below.
* We see that we should only give the drug to young people who test positive.

## Sensitivity to assumptions

* If we reduce the cost of the drug from 8 to 5, we get the new optimal policy shown below.
* Now we see that we should give the drug to anyone who tests positive.

* If we keep the cost of the drug at 8, but increase the sensititivity of the test from 0.875 to 0.975, we get the new optimal policy shown below.
* Now we see that we should give the drug to anyone who tests positive, since the chance that they actually have COVID is higher.

## Code to reproduce the above.

```python
def make_loss_fun(loss_drug=15):
  loss_covid_young = 60
  loss_covid_old = 10
  loss_mat = np.zeros((4, 2))
  loss_mat[0, :] = [0, loss_drug]
  loss_mat[1, :] = [loss_covid_young, loss_drug]
  loss_mat[2, :] = [0, loss_drug]
  loss_mat[3, :] = [loss_covid_old, loss_drug]
  return loss_mat
```

```python
loss_mat = make_loss_fun()
records = []
records.append(('No COVID, young', loss_mat[0,0], loss_mat[0,1]))
records.append(('COVID, young', loss_mat[1,0], loss_mat[1,1]))
records.append(('No COVID, old', loss_mat[2,0], loss_mat[2,1]))
records.append(('COVID, old', loss_mat[3,0], loss_mat[3,1]))

df = pd.DataFrame.from_records(records, columns=['State', 'Nothing', 'Drugs'])
display(df)
```

      State
      Nothing
      Drugs

      0
      No COVID, young
      0.0
      15.0

      1
      COVID, young
      60.0
      15.0

      2
      No COVID, old
      0.0
      15.0

      3
      COVID, old
      10.0
      15.0

```python
def normalize(x):
  return x / np.sum(x)

def posterior_covid(observed, prevalence=None, sensitivity=None):
  # observed = 0 for negative test, 1 for positive test
  # hidden state = 0 if no-covid, 1 if have-covid
  if sensitivity is None:
    sensitivity = 0.875
  specificity = 0.975
  TPR = sensitivity;
  FNR = 1-TPR
  TNR = specificity
  FPR = 1-TNR
  # likelihood(hidden, obs)
  likelihood_fn = np.array([[TNR, FPR], [FNR, TPR]])
  # prior(hidden)
  if prevalence is None:
    prevalence = 0.1
  prior = np.array([1-prevalence, prevalence])
  likelihood = likelihood_fn[:, observed].T
  posterior = normalize(prior * likelihood)
  return posterior
```

```python
def compute_belief(obs, prevalence=None, sensitivity=None):
  test_result, age = obs
  # returns bel[0:3] for states (not-covid+not-old, covid+not-old, not-covid+old, covid+old)
  post = posterior_covid(test_result, prevalence, sensitivity)
  bel = np.zeros(4)
  if age==0:
    bel[0:2] = post
  else:
    bel[2:4] = post
  return bel

def policy(obs):
  bel = compute_belief(obs)
  loss_mat = make_loss_fun(loss_drug)
  expected_loss = np.dot(bel, loss_mat)
  return np.argmin(expected_loss)

```

```python
def make_table(cost_drug, prevalence=None, sensitivity=None):
  records = []
  for test_result in [0,1]:
    for age in [0,1]:
      obs = (test_result, age)
      bel = compute_belief(obs, prevalence, sensitivity)
      loss_mat = make_loss_fun(cost_drug)
      expected_loss = np.dot(bel, loss_mat)
      act = np.argmin(expected_loss)
      if age==0:
        post_covid = bel[1]
      else:
        post_covid = bel[3]
      records.append((test_result, age, post_covid, expected_loss[0], expected_loss[1], act))

  df = pd.DataFrame.from_records(records, columns=['test', 'age', 'pr(covid)', 'cost-noop', 'cost-drugs', 'action'])
  return df

```

```python
df = make_table(cost_drug=8)
display(df)
print(df.to_latex(index=False, float_format="%.2f" ))

```

      test
      age
      pr(covid)
      cost-noop
      cost-drugs
      action

      0
      0
      0
      0.014045
      0.842697
      8.0
      0

      1
      0
      1
      0.014045
      0.140449
      8.0
      0

      2
      1
      0
      0.795455
      47.727273
      8.0
      1

      3
      1
      1
      0.795455
      7.954545
      8.0
      0

    \begin{tabular}{rrrrrr}
    \toprule
     test &  age &  pr(covid) &  cost-noop &  cost-drugs &  action \\
    \midrule
        0 &    0 &       0.01 &       0.84 &        8.00 &       0 \\
        0 &    1 &       0.01 &       0.14 &        8.00 &       0 \\
        1 &    0 &       0.80 &      47.73 &        8.00 &       1 \\
        1 &    1 &       0.80 &       7.95 &        8.00 &       0 \\
    \bottomrule
    \end{tabular}

```python
df = make_table(cost_drug=5)
display(df)
print(df.to_latex(index=False, float_format="%.2f" ))
```

      test
      age
      pr(covid)
      cost-noop
      cost-drugs
      action

      0
      0
      0
      0.014045
      0.842697
      5.0
      0

      1
      0
      1
      0.014045
      0.140449
      5.0
      0

      2
      1
      0
      0.795455
      47.727273
      5.0
      1

      3
      1
      1
      0.795455
      7.954545
      5.0
      1

    \begin{tabular}{rrrrrr}
    \toprule
     test &  age &  pr(covid) &  cost-noop &  cost-drugs &  action \\
    \midrule
        0 &    0 &       0.01 &       0.84 &        5.00 &       0 \\
        0 &    1 &       0.01 &       0.14 &        5.00 &       0 \\
        1 &    0 &       0.80 &      47.73 &        5.00 &       1 \\
        1 &    1 &       0.80 &       7.95 &        5.00 &       1 \\
    \bottomrule
    \end{tabular}

```python
df = make_table(cost_drug=8, sensitivity=0.975)
display(df)
print(df.to_latex(index=False, float_format="%.2f" ))
```

      test
      age
      pr(covid)
      cost-noop
      cost-drugs
      action

      0
      0
      0
      0.002841
      0.170455
      8.0
      0

      1
      0
      1
      0.002841
      0.028409
      8.0
      0

      2
      1
      0
      0.812500
      48.750000
      8.0
      1

      3
      1
      1
      0.812500
      8.125000
      8.0
      1

    \begin{tabular}{rrrrrr}
    \toprule
     test &  age &  pr(covid) &  cost-noop &  cost-drugs &  action \\
    \midrule
        0 &    0 &       0.00 &       0.17 &        8.00 &       0 \\
        0 &    1 &       0.00 &       0.03 &        8.00 &       0 \\
        1 &    0 &       0.81 &      48.75 &        8.00 &       1 \\
        1 &    1 &       0.81 &       8.12 &        8.00 &       1 \\
    \bottomrule
    \end{tabular}

# Decision-theoretic classification

* In classification problems, the unknown state of nature is the "true" class label $y^* \in \cal{Y}$, where $\cal{Y} = \{1,\ldots,C\}$.
* We typically assume that the set of possible actions is to pick one of the labels, $\hat{y} \in \cal{Y}$.
* We often assume a **zero-one** loss matrix $\ell(h,a)$, as follows:

$$
\begin{array}{c|cc}
 & \hat{y}=0 & \hat{y}=1  \\ \hline
 y^*=0 & 0 & 1  \\
 y^*=1 & 1 & 0
 \end{array}
$$
* We can write this more concisely as follows:
$$
\ell(y^*,\hat{y}) = I(y^* \neq \hat{y})
$$
* In this case,
the posterior expected loss is
$$
R(\hat{y}|x)
 = p(\hat{y} \neq y^* |x) = 1-p(y^*=\hat{y}|x)
$$
Hence the action that minimizes the expected loss is to choose the
most probable label:
$$
\pi(x)  = \arg \max_{y \in \cal{Y}} p(y|x)
$$
* This  corresponds to
the **mode** of the posterior distribution,
also known as the **maximum a posteriori**
or  **MAP estimate**.

## Classification with a "reject" option

* In some applications (e.g., medicine, finance), we have an asymmetric loss, where the cost of a false positives is much higher than a cost of a false negative.
* In such settings, it is useful to allow for addition **rejection action**, in which we **refuse to classify** if we are uncertain. Such inputs can then be handled by some other system (e.g., a human).
* We define the loss function as follows:
$$
\ell(h, a) =
\left\{ \begin{array}{cc}
  0 & \mbox{if $h=a$ and $a \in \{1,\ldots,C\}$}\\
  \lambda_r & \mbox{if $a=0$}\\
  \lambda_e & \mbox{otherwise}
\end{array}
\right.
$$
where $\lambda_r$ is the cost of a rejection, and $\lambda_e$ is a cost of a misclassification error.
* It is easy to show that the optimal policy is as follows:
$$
\begin{align}
a^* &=
\begin{cases}
y^* & \text{ if } p(y^*|x) >  1-\frac{\lambda_r}{\lambda_e} \\
\text{reject} & \text{otherwise}
\end{cases}
\end{align}
$$
where $y^* = \arg \max_{y} p(y|x)$ is the MAP estimate.

# Bandit problems

## Multi-armed bandits

The classic multi-armed bandit is illustrated below.
The agent (the octopus) must choose which "arm" to pull.
Each slot machine (aka one-armed bandit) has a different, but unknown, payout rate (reward), denoted $R(a)$.

* In a MAB problem, the goal is to choose a sequence of actions (arm pulls) that maximizes the sum of expected rewards:
$$
a_{1:T} = \arg \max_{a_{1:T}} \sum_{t=}^T E_{a_t \sim \pi}[R(a_t | \theta)]
$$
where $T$ is the **finite horizon**, $\pi()$ is the (stochastic) **policy**, $R(a|\theta)$ is the reward function for action $a$ using model parameters $\theta$.
* In the **infinite horizon** case, we use a **discount factor** $\gamma < 1$ to ensure the sum converges:
$$
J = \sum_{t=1}^\infty \gamma^{t-1} E_{ a_t \sim \pi}[R(a_t | \theta)]
$$
* For binary rewards, we can use a **Bernoulli bandit**
$$
p(R|a) = \text{Ber}(R|\mu_a)
$$
where $\mu_a=E[R|a]=p(R=1|a)$.
* For real-valued rewards, we can use a **Gaussian bandit**
$$
p(R|a) = \text{Gauss}(R|\mu_a, \sigma_a^2)
$$

## Contextual bandits

* In **contextual bandits**, the agent is in a random state $s_t$ at each step. States are drawn iid from some unknown distribution $p(s_t)$. The policy should now depend on the inputs, $a_t \sim \pi(s_t)$.
* The goal is to maximimize
$$
J = \sum_{t=1}^\infty \gamma^{t-1} E_{ s_t \sim p(s), a_t \sim \pi(a|s_t)}[R(s_t, a_t | \theta)]
$$
* There are numerous applications, e.g.
** **Online advertising**, where the action specifies which ad to show, and the state specifies user features (e.g., search history) and/or web page features (e.g., content).
** **Recommender systems**, where the action specifies which (set of) item(s)  to show, and the state specifies user features.
** **Clinical trials**, where the action specifies which drug to give, and the state specifies patient features.
* For binary rewards, we can use logistic regression model
$$
p(R|s,a;\theta) = \text{Ber}(R|\sigma(\theta^T \phi(s,a)))
$$
where $\phi(s,a)$ is a feature vector, possibly computed using a deep neural network, and $\sigma(x)=1/(1+e^{-x})$ is the sigmoid or logistic function.
* For real-valued rewards, we can replace the sigmoid output with a linear output, and use a Gaussian observation model.
* A key problem is that the rewards for each (state,action) pair (represented by model parameters $\theta$) are unknown.
* The agent can infer the parameters in an online (recursive) fashion given the data stream $D_{1:t} = (s_{1:t}, a_{1:t}, r_{1:t})$ using **sequential Bayesian updating**:
$$
\begin{align}
p(\theta|D_{1:t}) &=
p(\theta|D_t, D_{1:t-1})
= \frac{p(D_t|\theta,D_{1:t-1}) p(\theta|D_{1:t-1})}
{p(D_t|D_{1:t-1})}
\propto p(D_t|\theta) p(\theta|D_{1:t-1})
\end{align}
$$
* The likelihood is $p(D_t|\theta)=p(r_t|s_t, a_t,\theta)$ and the prior is the belief state from the previous time step, $p(\theta|D_{1:t-1})$.

## Exploration-exploitation tradeoff

* The agent must try each (state, action) pair (possibly multiple times) in order to collect data, but these experiments incur an opportunity cost. This is called the **exploration-exploitation tradeoff**.
* A common heuristic solution is the **upper confidence bound** method, which selects the action according to
$$
a_t = \arg \max_a E[R(s_t,a;\theta)] + c \;\text{std}[R(s_t,a;\theta)]
$$
* The  constant $c \geq 0$ reflects the amount of optimism about unknown rewards, and hence the amount of exploration.
* Below we illustrate the distribution over possible rewards for  a 3-armed bandit (without contextual features).

## Reinforcement learning

* In **reinforcement learning**, the setup is similar to a contextual bandit, except now the current state depends on the previously visited states, as well as the actions that the agent has taken.
* If we assume a Markov model for the enviroment, we get a **Markov decision process** or **MDP**.
$$
J = \sum_{t=1}^\infty \gamma^{t-1} E_{ s_t \sim p(s|s_{t-1}, a_{t-1}; \theta), a_t \sim \pi(a|s_{1:t})}[R(s_t, a_t | \theta)]
$$
* Solving for the optimal policy when the parameters $\theta$ of the environment are unknown is challenging. This is beyond the scope of this tutorial.

```python

```


# TD 1 : Inference Learning / Perceptron : Berlemont Kevin

## Simple Perceptron Algorithm

In this section we are going to study the behavior of a perceptron when trying to learn one classification about datas. Let us first take $M$ points, denote by $x_i$ in $\mathbb{R}^N$. Suppose that each of them are labelled by a binary label, denote by $\sigma_i$. How can the perceptron learn this classification ?

A perceptron who receives an input $x$ will compute a function of the form :
\begin{equation}
f(x) = \text{sign} (w \cdot x  - b)
\end{equation}
for vector $x \in \mathbb{R}^N$, where $w=(w_1,\dots, w_N) \in \mathbb{R}^N$ and $b \in \mathbb{R}$ are adjustable parameters, or weights.

In order to have simplified notation, up to now we will include the biaises $b$ in the vector $w$ by writting :
$$ (w_1, \dots, w_N) \rightarrow (b,w_1,\dots,w_N)$$
$$ (x_1, \dots, x_N) \rightarrow (1,x_1,\dots x_N)$$

Let us now specify the problem. Indeed, we have taken inputs which are distributed along a Gaussain in $\mathbb{R}^N$, but we will not take the classification totally randomly. Indeed, if the classification is totally random, the perceptron might not be able to learn the classification (problem of capacity). Hence we are going to take a TEACHER perceptron, with random weights, and the classification will be done according to this one :
$$ \sigma_i = \text{sign} (w_{\text{teacher}} \cdot x )$$

Why are we taking this assumption ? It is because then, the ensemble $\{ x_i, \sigma_i >0\}$ and $\{x_i, ~ \sigma_i <0 \}$ are absolutely linearly separable.

Two sets $A$ and $B$ of points in a $n$-dimensional space are called absolutely linearly separable if $n+1$ real number exists $w_1, \dots, w_{n+1}$ exist such that $(x_1,\dots, x_n) \in A$ satisfies $\sum_i w_i x_i > w_{n+1}$ and for $B$ we have $\sum_i w_i x_i < w_{n+1}$

A perceptron allow us to fin a separation between sets wich are linearly separable. This is useful because if we take the following algorithm (simple perceptron algorithm) :

**We have a theorem :**
>Assume that there exists some parameter vector $w^{\star}$ such that $\left \lVert w^{\star}\right\rVert=1$ and some $\gamma >0$ such that for all $t=1, \dots , n$
>$$
y_t (x_t \cdot w^{\star}) \geq \gamma
$$
>Assume in additional that $\left\lVert x_t \right\rVert \leq R$. Then the perceptron algorithm makes at most $\frac{R^2}{\gamma^2}$ errors.

--------------------------------------------
*This theorem can be demonstrate as follows :*

Let us first denote $w^k$ the parameter vector when the algorithm makes the $k$th error. We clearly have, by initialization :
$$ w^1 = 0$$
If the $k+1$'th error is made on example $t$, we have :
\begin{align}
w^{k+1} \cdot w^{\star} & = (w^k + y_t x_t) \cdot w^{\star} \\
& = w^k \cdot w^{\star} + y_t x_t \cdot w^{\star} \\
& \geq w^k \cdot w^{\star} + \gamma
\end{align}
This last inequality comes from :
\begin{equation}
y_t x_t \cdot w^{\star} \geq \gamma
\end{equation}
Basic induction on $k$ leads to :
\begin{equation}
w^{k+1} \cdot w^{\star} \geq k \gamma
\end{equation}
Using Cauchy-Schwarz we find $\left \lVert w^{k+1}\right \rVert \times \left \lVert w^{\star} \right \rVert \geq w^{k+1} \cdot w^{\star}$ and $\left \lVert w^{\star} \right \rVert=1$ :
$$
\left \lVert w^{k+1} \right \rVert \geq k \gamma
$$
Now we need to find an upper bound on $\left \lVert w^{k+1} \right \rVert$.
\begin{align}
\left \lVert w^{k+1} \right \rVert^2 &= \left \lVert w^k + y_t x_t \right \rVert^2 \\
& \leq \left \lVert w^k \right \rVert^2 + R^2
\end{align}
The last inequality comes from the fact that we had an error at example $t$. Using again an induction reasoning :
\begin{equation}
\left \lVert w^{k+1}\right \rVert^2 \leq k R^2
\end{equation}
Combining the two bounds we find :
\begin{equation} k \leq \frac{R^2}{\gamma^2}\end{equation}

--------------------------------------------------
We just show that we should be able to solve our problem numerically, and that's what the following simulations will do, trying to understand the limits of this approach and if indeed convergence is obtained, and the time to get it.

```python
from IPython.display import HTML

HTML('''
The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click <a href="javascript:code_toggle()">here</a>.''')
```

The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click here.

```python
from pylab import rand,plot,show,norm
import numpy as np
import gzip

def generateData(N,M):
 """
  generates a N-D  dataset with M samples.
  M = nombres exemples
  N = dimensions
 """
 x = np.random.randn(M, N)
 x = np.concatenate((np.ones((M,1)),x), axis=1)

 return x/np.linalg.norm(x)

def generateTeacher(M):
 """
  generates the weights of the teacher.

 """
 w_teacher  = np.random.uniform(low=-0.5, high=0.5, size=N+1)# +1 à expliquer car insertion du biais
 return w_teacher/np.linalg.norm(w_teacher)

def generateLabel(x,w_teacher):
 """
  generates the N-D linearly separable dataset.
  The classification is done using the weight of the teacher
  return a couple :
      - First argument is the sample
      - Second his the binary classification associated
 """
 label=np.zeros(M)
 for i in range(M):
        label[i] = np.sign(np.sum(x[i]*w_teacher))
 x= zip(x,label)

 return x

class Perceptron:
 def __init__(self,N=50,M=100):
  """ perceptron initialization """

  self.N=N # Dimension
  w = np.random.uniform(low=-1, high=1, size=N+1) # weights
  self.w = w
  self.learningRate = 0.1
  self.error = list() # list of error through the training
  self.M = M # number example
  self.weight_list = list() # evolution of the weights through the training
  self.weight_list.append(self.w)

 def response(self,x):
  """ perceptron output """
  y = np.sign(np.sum(self.w * x)) # dot product between w and x
  return y

 def updateWeights(self,x,iterError):
  """
   updates the weights status, w at time t+1 is
       w(t+1) = w(t) + learningRate*(d-r)*x
   where d is desired output and r the perceptron response
   iterError is (d-r)
  """
  for i in range (self.N):
        self.w[i] += self.learningRate*iterError*x[i]

 def train(self,data):
  """
   trains all the vector in data.
   Data must be a couple. The second element should be the label.
   Return the number of iterations needed to perform the training
  """
  learned = False # Do we continue or not the training ?
  iteration = 0
  while not learned:
   globalError = 0.0
   for (x,label) in data: # for each sample
    r = self.response(x)
    if label != r: # if we have a wrong response
     iterError = label- r # desired response - actual response
     self.updateWeights(x,iterError)
     globalError += abs(iterError)
   self.error.append(globalError/self.M)

   iteration += 1
   #Normalization (don't forget to explain in theory why we do this)
   norm_temp = np.linalg.norm(self.w)
   self.w = self.w/norm_temp
   self.weight_list.append(self.w)
   #print (iteration)
   if globalError == 0.0 or iteration >= 100: # stop criteria
    #print ('iterations',iteration)
    learned = True # stop learning

  return iteration

```

```python
#N=50
#M=100
#w_teacher = generateTeacher(N)
#x_trainset = generateData(N,M)

#x_data = generateLabel(x_trainset, w_teacher)

#-----------------------
# DATA LIST !!!!!!! Ne pas relancer, passer directement à la récup des fichiers sauf si nombre N ou M change
#---------------------------
N_list=np.array([10,50,100,500,1000,2000])
M_list=np.array([20,100,200,1000,5000,10000,20000])

iterations_matrix=np.zeros((len(N_list),len(M_list))) # abscisses : dimensions, ordonnées : nbr exemples
weight_matrix = [] # sera composé de N matrices temporaires qui auront chacune M vetcteurs poids
norme_matrix = np.zeros((len(N_list),len(M_list))) # norme error mais suelement la dernière
teacher_matrix = list() # on va metter un couple N,teacher
#error_matrix = np.zeros((N,M))

for i,N in enumerate(N_list):
    w_teacher = generateTeacher(N)
    teacher_matrix.append((N,w_teacher))
    #print N
    for j,M in enumerate(M_list):
        x_trainset=generateData(N,M)
        #print M
        x_data = generateLabel(x_trainset,w_teacher)
        perceptron = Perceptron(N,M)
        iterations = perceptron.train(x_data)
        iterations_matrix[i,j] = iterations
        #error_matrix[i,j] = perceptron.error
        norme_matrix[i,j]=np.linalg.norm(perceptron.weight_list[-1]-w_teacher)

```

```python
try:
   import cPickle as pickle
except:
   import pickle

with open('donnees_perceptron_nouveaucode','wb') as fichier:
    my_pickler= pickle.Pickler(fichier)
    my_pickler.dump(teacher_matrix)
    my_pickler.dump(iterations_matrix)
    my_pickler.dump(N_list)
    my_pickler.dump(M_list)
    my_pickler.dump(norme_matrix)
    readme=['normematrix','Mlist','Nlist','iterations','teacher']
    my_pickler.dump(readme)

```

```python
with open('donnees_perceptron','rb') as fichier:
    my_pickler= pickle.Unpickler(fichier)
    readme=my_pickler.load()
    norme_matrix=my_pickler.load()
    M_list = my_pickler.load()
    N_list = my_pickler.load()
    iterations_matrix = my_pickler.load()
    teacher_matrix = my_pickler.load()
```

```python
import matplotlib.pyplot as plt
%matplotlib inline

from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter

fig = plt.figure(figsize=(10, 5))

ax = fig.add_subplot(111)
ax.set_title('Norme of the difference')
plt.imshow(norme_matrix,origin='lower')
ax.set_aspect('equal')
ax.set_ylabel('Dimension')
ax.set_xlabel('Number examples')
ax.set_xticklabels(['',20,100,200,1000,5000,10000,20000])
ax.set_yticklabels(['',10,50,100,500,1000,2000])

plt.colorbar()
plt.show()
```

This figure shows us how our perceptron is able to generalize after his training. Indeed, he will generalize perfectly if and only if $w_{\text{perceptron}} = w_{\text{teacher}}$. Given two parameters ; dimension of the input space and the number of example we can observe two different behaviors.

As the number of examples increases the difference between the two weight vectors decreases, which means that our perceptron learns more precisely the teacher's vector. And he will be able to generalize more easily.

However as the dimension of the input space increases, it's the opposite effect. To generalize high-dimensional inputs we need more examples.

*The question we can ask is : Does the loss of generalization come from the fact that our algorithm goes not over 100 iterations ?*

```python
fig = plt.figure(figsize=(10, 5))

ax = fig.add_subplot(111)
ax.set_title('Iterations')
plt.imshow(iterations_matrix,origin='lower')
ax.set_aspect('equal')
ax.set_ylabel('Dimension')
ax.set_xlabel('Number examples')
ax.set_xticklabels(['',20,100,200,1000,5000,10000,20000])
ax.set_yticklabels(['',10,50,100,500,1000,2000])

plt.colorbar()
plt.show()
```

This figure allows us to answer the previous question. Indeed the variations of the number of iterationso ur algorithm makes is the inverse of the one of the diiference between the wieght vectors.

For high-dimensional space our algorithm converges in a few iterations to a good classifier. However he will be bad at generalization. In contrary for high number of example the algorithm converges slowly but the obtained perceptron will be good at generalization.

This result is in accordance to the theorem of *convergence of perceptron algorithm*, which statest hat the number of errors our algorithms is making, depends strongly on the *distance* betwwen the two classifications. If we increase the number of example, the distance is smaller, it's harder to classify but it will be easier to generalize because we need to be more precise to classify the examples.

```python
N=500
w_teacher = generateTeacher(N)

M_list=[10,100,500,1000,10000]

error_list = list()

for i,M in enumerate(M_list):
    x_trainset = generateData(N,M)

    x_data = generateLabel(x_trainset, w_teacher)
    perceptron = Perceptron(N,M)
    iterations = perceptron.train(x_data)
    error_list.append(perceptron.error)

import matplotlib.pyplot as plt
%matplotlib inline

```

```python
plt.figure(1)

for i,error in enumerate(error_list) :

    plt.plot(error,label='Number of example is ' +str(M_list[i]))
plt.xlabel('Epochs')
plt.ylabel('Global error on the training ')
plt.axis([0,60,0,0.8])
plt.title('Global error variations in dimension ' + str(N))
plt.legend(loc=1)
plt.show()

N=2000
w_teacher = generateTeacher(N)

M_list=[10,100,500,1000,10000]

error_list = list()

for i,M in enumerate(M_list):
    x_trainset = generateData(N,M)

    x_data = generateLabel(x_trainset, w_teacher)
    perceptron = Perceptron(N,M)
    iterations = perceptron.train(x_data)
    error_list.append(perceptron.error)

plt.figure(2)

for i,error in enumerate(error_list) :

    plt.plot(error,label='Number of example is ' +str(M_list[i]))
plt.xlabel('Epochs')
plt.ylabel('Global error on the training ')
plt.axis([0,60,0,0.8])
plt.title('Global error variations in dimension ' + str(N))
plt.legend(loc=1)
plt.show()
```

Those figure show the evolution of the global error on the training set during the training. We have tried for several dimensions of the input space, and several number of examples and we observe the same behavior as for the final distribution of the norm.

We could just add that even if for large sample in a small input space the error decreases faster than for less examples, it will achive total training slower anyways. We just see in another form the fact that it's hard to be extremly precise during the training with a great number of examples.

## MNIST Data

We now want to classify the digits in the MNIST database. For this purpose we will still use the perceptron algorithm. However in contrary to the previous part we do not have only binary label.

Hence we need to take a model of $10$ perceptrons, taking an input of dimensions $784$. Each perceptron gives as output a number and the largest number between all the perceptrons will correspond to the digit our network has choose.

We just need to compare with the correct label and apply the same perceptron algorithm rule as before.

```python
# import pychecker.checker
import numpy as np
import argparse
import six
import pickle
from six.moves.urllib import request
import os
import bz2

trainurl = 'https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass/mnist.bz2'
testurl = 'https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass/mnist.t.bz2'

def read_file(filename):
    """Read data file and make features
    Args:
        filename: filename to read (str with relative path)
    Returns:
        ans: A dict mapping keys, consisting of:
            {'target': #TARGET, 'features': #np.2darray}
        max_dim: max dimension of input featureset
    Features 2darray has the shape of [[dimensions array],[valus array]].
    Be careful this is not direct expression of matrix.
    """
    print "reading and making pickles..."
    ans = []
    max_dim = 1
    for line in open(filename, 'r'):
        sind = line.find(" ")
        label = int(line[0:sind])    # separated by space
        one = {}
        one['target'] = label

        features = np.zeros([2, 1])
        length = line[sind + 1:].find(" ")
        while 1:
            tempstr = line[sind + 1:sind + 1 + length]
            if tempstr.find(':') > 0:
#                print tempstr
                fdim = int(tempstr[:tempstr.find(":")])
                # get max dimension of input feature to make matrix
                max_dim = np.max([max_dim, fdim])
                fval = float(tempstr[tempstr.find(":") + 1:])
                f = np.array([[fdim], [fval]])
                features = np.append(features, f, axis=1)
                sind, length = sind + length + 1, line[sind + length + 2:].find(" ")
            if length < 0:
                break

        features = features.T[1:].T
        one['features'] = features
        ans.append(one)
    return ans, max_dim

def make_featuremat(dicts, dim):
    """Make feature matrix for input and output
    Args:
        dicts, dim: Returns of function "read_file"
    Returns:
        data: feature matrix
    """
    data = {}
    data['x'] = np.zeros([len(dicts), dim])
    data['y'] = np.zeros([len(dicts)])
    for i in range(0, len(dicts)):
        data['y'][i] = dicts[i]['target']
        temp = dicts[i]['features']

        for j in range(0, len(temp[0])):
            data['x'][i][temp[0][j] - 1] = temp[1][j]

    return data

def save_pkl(data, filename):
    savename = filename + '.pkl'
    print 'Save datasets as pkl: %s' % (savename)
    with open(savename, 'wb') as output:
        six.moves.cPickle.dump(data, output, -1)
    print('Done')

def download_dataset():
    print 'Downloading data/mnist.bz2'
    request.urlretrieve(trainurl, filename="data/mnist.bz2")
    bz2.decompress("data/mnist.bz2")
    print('Done')
    print 'Downloading data/mnist.t.bz2'
    request.urlretrieve(testurl + '.t', filename="data/mnist.t.bz2")
    bz2.decompress("data/mnist.t.bz2")
    print('Done')

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Read MNIST dataset')
    fname = "data/mnist"

    if not os.path.exists(fname):
        download_dataset()

    ans, max_dim = read_file(fname)
    print len(ans), max_dim
    data = make_featuremat(ans, max_dim)
    save_pkl(data, fname)

    fname = "data/mnist.t"
    ans, max_dim = read_file(fname)
    print len(ans), max_dim
    data = make_featuremat(ans, max_dim)
    save_pkl(data, fname)

"""
open pickle
with open(filename, 'rb') as D_pickle:
    D = six.moves.cPickle.load(D_pickle)
"""
```

    reading and making pickles...
    60000 779

    /usr/local/lib/python2.7/dist-packages/ipykernel/__main__.py:70: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future

    Save datasets as pkl: data/mnist.pkl
    Done
    reading and making pickles...
    10000 777
    Save datasets as pkl: data/mnist.t.pkl
    Done

    "\nopen pickle\nwith open(filename, 'rb') as D_pickle:\n    D = six.moves.cPickle.load(D_pickle)\n"

```python
import pickle
import six
import numpy as np
import argparse

def read_pkl(fname):
    with open(fname, 'rb') as d_pickle:
        data = six.moves.cPickle.load(d_pickle)
    return data

class Perceptron:
    """Train and test the perceptron.
    This class has one perceptron model, which has weight and bias.
    Before setting the class, input "Train data" and "Test data",
    in order to set feature matrix due to feature dimension.
    """
    def __init__(self, traindata, testdata):

        self.num_ofdim = np.max([len(traindata['x'][0]), len(testdata['x'][0])])
        self.num_ofclass = int(np.max(traindata['y']) + 1) # +1 à expliquer en theorie
        self.w = np.random.rand(self.num_ofclass, self.num_ofdim + 1)
        self.accuracy_train = list()
        self.accuracy_test = list()

    def reshape_mat(self, data):
        """Reshaping input feature matrix, because either train/test data
        does not always have all the features that the other has.
        """
        if len(data['x'][0]) < self.num_ofdim:
            x = data['x']
            x_plus = np.zeros([len(x), self.num_ofdim - len(x[0])])
            data['x'] = np.append(x, x_plus, axis=1)
        return data

    def predict(self, w, x):
        """Prediction function.
        N-class discrimination: from 0 to (n-1)
        """
        pred_val = np.dot(w, np.append(x, 1))
        pred_class = np.argmax(pred_val)
        return pred_class # expliquer théorie

    def w_update(self, x, y, pred_class, alpha=0.001):
        """Weight update function.
        alpha is learning rate.
        """
        tempw = self.w[pred_class]
        self.w[y] += alpha * np.append(x, 1)
        self.w[pred_class] -= alpha * np.append(x, 1)

    def train(self, data, iteration):
        """Train function.
        """
        num = len(data['y'])
        data = self.reshape_mat(data)

        for j in range(0, iteration):
            acc = 0
            order = np.random.permutation(num) # some statistical inference stuff (stochastic descent)
            for i in range(0, num):
                x, y = data['x'][order[i]], data['y'][order[i]]
                yhat = self.predict(self.w, x)
                if y == yhat:
                    acc += 1
                else:
                    self.w_update(x, y, yhat)
            self.accuracy_train.append(acc / float(num))
            #print 'Iteration %d: acc = %f' % (j+1, acc / float(num)*100)

    def test(self, data):
        """Test function.
        """
        num = len(data['y'])
        self.reshape_mat(data)

        acc = 0
        order = np.random.permutation(num)
        for i in range(0, num):
            x, y = data['x'][order[i]], data['y'][order[i]]
            yhat = self.predict(self.w, x)
            if y == yhat:
                acc += 1
        self.accuracy_test.append(acc / float(num))
        #print 'Test: acc = %f' % (acc / float(num) * 100)

```

```python
fname = "mnist"

traindata = read_pkl("data/" + fname + ".pkl")
testdata = read_pkl("data/" + fname + ".t.pkl")

Perc = Perceptron(traindata, testdata)

iterations = 40
for i in range(iterations):
        Perc.test(testdata)
        Perc.train(traindata, 1)

```

    /usr/local/lib/python2.7/dist-packages/ipykernel/__main__.py:52: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future

```python
import matplotlib.pyplot as plt
%matplotlib inline

plt.figure(2)

plt.plot(Perc.accuracy_train,label='Train accuracy')
plt.plot(Perc.accuracy_test,label='Test accuracy')
plt.axis([0,40,0.8,0.95])
plt.title('Accuracy on the MNIST data for simple perceptron algorithm ')
plt.xlabel('Epochs')
plt.ylabel('Accuracy in percent')
plt.legend()
plt.show()
```

On this figure we see how the accuracy increases with the number of epochs. The interesting thing is that our system is not able to reach perfect accuracy, even on training set. This is in accordance with the results of the previous part because we have $60000$ inputs in only a 784 dimensional space.

```python
fname = "mnist"

traindata = read_pkl("data/" + fname + ".pkl")
testdata = read_pkl("data/" + fname + ".t.pkl")

Perc = Perceptron(traindata, testdata)

iterations = 100
for i in range(iterations):
        Perc.test(testdata)
        Perc.train(traindata, 1)
plt.figure(2)

plt.plot(Perc.accuracy_train,label='Train accuracy')
plt.plot(Perc.accuracy_test,label='Test accuracy')
plt.axis([0,100,0.8,0.95])
plt.title('Accuracy on the MNIST data for simple perceptron algorithm ')
plt.xlabel('Epochs')
plt.ylabel('Accuracy in percent')
plt.legend()
plt.show()
```

If we keep the computation durong more time we see the same behavior. The training accuracy seems to have reach a plateau. However our algorithm is still modifying the weight vector. Indeed we see some oscillations of the test accuracy, around the mean value of train accuracy. This stands for the fact that globally the train accuracy is constant but the network is always changing his weights, and thus modifying the vectors for which he misclassified.

This figure allows us to understand the difficulty to learn some specif features with a good accuracy, and the difficulty of generalization. Indeed the results on the test datas are very noisy, and in reality we want good performance on the test data, the training data are just here to be able to train.

Because the behavior of the global error is really different of previously, it decreases a lot slower, we can deduce that our datas are probably not linearly separable or we should be able to classify perfectly the training data at least.

To improve the results on accuracy we should work with non-linear classifier (like SVM for example) or preprocessing the datas in order to find an input space in which the datas arem more likely to be linear eparable sets.


# Big Assignment 4

### A number of organizations and governments are considering policies to supply students with access to free computers, in an effort to boost learning outcomes. Suppose that you want to learn about the effect of having a free computer on test scores. For a sample of high school students, you collect data on test scores (`test` ), parent’s income (`inc`), and whether they have a computer at home (`comp`).

### Part A:
#### Suppose that a local organization held a lottery, and offered computers at random to a number of students, indicated by freecomp. Suppose that among the population who were not offered a free computer, 60% have a computer, whereas after the lottery, 90% of people offered a computer had a computer. Compare the specifications

\begin{align}
test=\beta_0+\beta_1comp+u
\end{align}

#### or

\begin{align}
test=\beta_0+\beta_1freecomp+u
\end{align}

#### Both tell you something different about the effect of computers on test scores. Describe the limitations of each approach.

#### ANSWER:

 The first specification
\begin{align}
test= \beta_0+\beta_1 comp+u
\end{align}
 describes the effect of having a computer on test scores, while the second specification
\begin{align}
test= \beta_0+\beta_1 freecomp+u
\end{align}

 describes the effect of being offered a computer on test scores. In other words, the first specification is explicitly about ownership of a computer, irrespective of whether or not one had one before the random distribution, while the second specification is interested in the individual having been offered a computer, irrespective of whether or not one had one before the random distribution, and with no conclusion yet as to whether the individual accepted the computer.

 Regarding limitations: In the first specification, one might be worried about unobservables that are correlated with having computer that are also correlated with test scores. For example, students who are more driven academically and try harder on tests may also be those who will pressure their parents to purchase them computers to work on. Meanwhile, in the second specification, we don’t learn what the effect of computers are on test scores. Instead, we learn about the effect of the program’s offer, which may not exactly be our interest.

### Part B:
#### You are concerned that household income may be correlated with test scores for a variety of reasons. Suggest an alternate specification that holds income constant, and discuss how you anticipate it changing inference using each estimated $\beta_1$ above.

#### ANSWER:

Our specifications are now
\begin{align}
test= \beta_0+\beta_1 comp+\beta_2 income+u
\end{align}
and
\begin{align}
test= \beta_0+\beta_1 freecomp+\beta_2 income+u
\end{align}

For specification 1, the exclusion of income from the specification previously was likely leading to omitted variable bias for beta 1, such that our estimate for beta 1 now suffers from less bias.

For specification 2, because freecomp is random, we know that E[u|T]=0 will be true in expectation but not always in samples, such that including more control variables can reduce bias. However, we usually don’t expect that adding a control will change the estimate of interest here. Including control variables will also increase the precision of estimates.

### Part C:
#### Propose a “Treatment on the treated” estimator for the free computer lottery.

#### ANSWER:

A treatment on the treated estimator for the effect of owning a computer can be written as follows:
\begin{align}
\frac{\overline{(test)_P}-\overline{(test)_{NP}}}{\overline{(comp)_P}-\overline{(comp)_{NP}} }
\end{align}
where P is assigned to group receiving and offer for a computer,NP is assigned to the group not receiving an offer for a computer, and comp is the proportion owning a computer.

### Part D:
#### You think computers might have two different effects on test scores. On the one hand, they are a productive tool which can help learning. On the other hand, they open up recreational opportunities (like social networking) which may not be very useful for learning. As a result, you wonder what the effects of a free computer would be if you could hold recreational screentime constant. That is, you would like to estimate
\begin{align}
test=\beta_0+\beta_1freecomp+\beta_2screentime+u
\end{align}

#### However, in practice you cannot observe screentime. You have two options:

#### - either you can just use whether they have a smart phone as a proxy variable for screentime (variable `Smartphone`),

#### -or you can ask the students to report how much screentime they spend per day (`repscreentime`).
#### Describe algebraically how each approach will affect estimation of $\beta_1$ and $\beta_2$ in the equation for part D, and the analogue to MLR4 which is necessary to grant unbiased estimates. Discuss different potential forms for any measurement error, and the likely consequences of that measurement error.

#### ANSWER:

We start out with our original model:
\begin{align}
test=\beta_0+\beta_1freecomp+\beta_2screentime+u
\end{align}
Case 1: Using `smartphone` represents the case described in Wooldridge 9.2 – Using Proxy Variables for Unobserved Explanatory Variables. `smartphone` is a proxy variable for `screentime`, which means we have

\begin{align}
screentime=\delta_0+\delta_2smartphone+\nu_2
\end{align}

where $\nu_2$ is an error due to the fact that `screentime` and `smartphone` are not related. If we plug in smartphone into the original model, then whether the proxy delivers consistent estimators depends on assumptions about $u$ and $\nu_2$.

(1) $u$ is uncorrelated with freecomp, screentime and smartphone, which are fairly standard assumptions implying that E$[u|freecomp, screentime, smartphone] = 0$.

(2)$\nu_2$ is uncorrelated with freecomp and smartphone. Assuming that $\nu_2$ is uncorrelated with freecomp requires smartphone to be a good proxy for screentime, which we can see by writing these assumptions in terms of conditional expectations
\begin{align}
E[screentime│freecomp,smartphone]=E[screentime│smartphone]\\
= \delta_0+\delta_2 smartphone
\end{align}

which is to say that once we control for smartphones, the expected value of screentime does not depend on freecomp. This may or may not be a reasonable assumption. But if we do assume this, then by plugging in we see that

\begin{align}
y=(\beta_0+\beta_2\delta_0 )+\beta_1 freecomp+ \beta_2 \delta_2 smartphone+u+\beta_2 \nu_2
\end{align}

the composite error can be written as $e = u + \beta_2 \nu_2$. Since both $u$ and $v_2$ have zero mean and each is uncorrelated with `freecomp` and `smartphone`, $e$ also has zero mean and is uncorrelated with `freecomp` and `smartphone`. We can write the above as
\begin{align}
y=\alpha_0+\beta_1 freecomp+ \alpha_2 smartphone+e
\end{align}

Thus, when we run the regression of our y on freecomp and smartphone, we will not recover unbiased estimators of $\beta_0$ and $\beta_2$, but we will instead get unbiased estimators of $\alpha_0$, $\beta_1$, and $\alpha_2$.

If instead screentime was related to all observed variables by
\begin{align}
screentime= \delta_0+\delta_1 freecomp+\delta_2 smartphone+\nu_2
\end{align}

where $\nu_2$ has zero mean and is uncorrelated with `freecomp` and `smartphone`.  Substituting the above equation into the equation with y on the LHS, we have
\begin{align}
y=(\beta_0+\beta_2 \delta_0 )+(\beta_1+\beta_2 \delta_1)freecomp+ \beta_2 \delta_2 smartphone+u+\beta_2 \nu_2
\end{align}
such that the probability limit of estimated $\beta_1 = \beta_1 + \beta_2 \delta_1$, so we can still get an upward bias for `freecomp` if `smartphone` is not a good proxy.

Case 2: Using repscreentime represents the case described in Wooldridge 9.4b.We can have measurement error of
\begin{align}
e_2=repscreentime-screentime
\end{align}
and we assume $E[e_2] = 0$. An important assumption is that u is uncorrelated with freecomp, repscreentime, and screentime. Usually $e_2$ is assumed to be uncorrelated with freecomp. However, the key issue is whether $e_2$ is uncorrelated with repscreentime. If it is uncorrelated, then the OLS regression of y on freecomp and repscreentime produces consistent estimators, which can be seen by writing
\begin{align}
y=\beta_0+\beta_1 freecomp+\beta_2 repscreentime+u-\beta_2 e_2
\end{align}

where $u$ and $e_2$ are both uncorrelated with all the explanatory variables. Under the CEV assumption that measurement error is uncorrelated with the unobserved explanatory variable, OLS will be biased and inconsistent, because $e_1$ is correlated with repscreentime, which means that all OLS estimators will be biased. There will also be attenuation bias on $\beta_2$.

### Part E:

#### Suppose instead that the organization offered free computers to the students whose parent’s income was in the bottom 20% of the school. Write down a regression equation which will implement a regression-discontinuity estimator to learn the impact of receiving a free computer. Discuss why results from that estimator may be different from the results from the randomized lottery in part A.

#### ANSWER:

To learn the impact of receiving a free computer, we can implement the following equation:
\begin{align}
test= \beta_0+\beta_1 bottom20_i+\beta_2 income_i+\beta_3 bottom20_i*income_i+u_i
\end{align}

where `bottom20` is an indicator variable for being a student whose parent’s income was in the bottom 20% of the school, and income is the income percentile that the individual’s parents fall into within the school.

Results from this estimator may be different from the randomized lottery in part A for a number of reasons. For one, results may be different depending on the linearity between income and test scores, which would motivate us to limit our data to observations around the threshold.We are estimating the treatment effect on a different local population: in the RCT TOT estimator, we get a LATE on compliers, while in the RD we get a LATE on the people around the threshold. We might also be concerned about other policies that may have been enacted at or close to the threshold.Finally, we would be worried about possible gaming around the threshold, which would motivate us to see if there are more individuals reporting just under or at 20% compared to right above 20%. We will also want to know how well the cutoff was respected, while in the RCT presumably the protocols regarding offers being made or not made were well respected.

```R

```


# Neural Estimation via DV bound

$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}$

```python
from gaussian import *
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import torch
import torch.optim as optim
from torch import Tensor, nn
from torch.nn import functional as F
from torch.utils.tensorboard import SummaryWriter
import tensorboard as tb

%load_ext tensorboard
%matplotlib inline
```

    The tensorboard extension is already loaded. To reload it, use:
      %reload_ext tensorboard

**How to formulate the problem of mutual information estimation?**

The problem of estimating the mutual information is:

---
Given $n$ samples

$$(\R{X}_1,\R{Y}_1),\dots, (\R{X}_n,\R{Y}_n) \stackrel{iid}{\sim} P_{\R{X},\R{Y}}\in \mc{P}(\mc{X},\mc{Y})$$

i.i.d. drawn from an *unknown* probability measure $P_{\R{X},\R{Y}}$ from the space $\mc{X}\times \mc{Y}$, estimate the *mutual information (MI)*

$$
\begin{align}
I(\R{X}\wedge\R{Y}) &:= E\left[\log \frac{d P_{\R{X},\R{Y}}(\R{X},\R{Y})}{d (P_{\R{X}} \times P_{\R{Y}})(\R{X},\R{Y})} \right].
\end{align}
$$ (MI)

---

Run the following code, which uses `numpy` to
- generate i.i.d. samples from a multivariate gaussian distribution, and
- store the samples as numpy arrays assigned to `XY`.

```python
import numpy as np

# Seeded random number generator for reproducibility
XY_rng = np.random.default_rng(SEED)

# Sampling from an unknown probability measure
rho = 1 - 0.19 * XY_rng.random()
mean, cov, n = [0, 0], [[1, rho], [rho, 1]], 1000
XY = XY_rng.multivariate_normal(mean, cov, n)
plt.scatter(XY[:, 0], XY[:, 1], s=2)
```

You may find more details below:
- https://numpy.org/doc/stable/reference/random/generated/numpy.random.multivariate_normal.html
- https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.scatter.html

---

**Tips**

To learn about a python object in JupyterLab:

- **Docstring**:
  - Move the cursor to the object and
    - click `Help->Show Contextual Help` or
    - click `Shift-Tab` if you have limited screen space.
- **Directory/Source**:
  - Right click on a notebook and choose `New Console for Notebook`.
  - Run `dir(obj)` for a previously defined object `obj` To see the available methods/properties of `obj`.
  - Run `obj??` to print the source code defining `obj`, if available.

---

---

**Exercise** What is unknown about the above sampling distribution?

The density is

$$
\frac{d P_{\R{X},\R{Y}}}{dxdy} = \mc{N}_{\M{0},\left[\begin{smallmatrix}1 & \rho \\ \rho & 1\end{smallmatrix}\right]}(x,y)
$$

but $\rho$ is unknown (uniformly random over $[0.8,0.99)$).

---

To show the data samples using `pandas`:

```python
import pandas as pd

XY_df = pd.DataFrame(XY, columns=["X", "Y"])
XY_df
```

      X
      Y

      0
      -0.029492
      0.285584

      1
      0.030093
      -0.233446

      2
      -0.671256
      -0.029712

      3
      -0.744867
      -1.091092

      4
      1.379858
      1.073218

      ...
      ...
      ...

      995
      -1.740407
      -0.884540

      996
      -0.370652
      0.244189

      997
      -0.590843
      -0.864294

      998
      2.180489
      1.733067

      999
      -0.461016
      -0.254751

1000 rows × 2 columns

To plot the data using `seaborn`:

```python
def plot_samples_with_kde(df, **kwargs):
    p = sns.PairGrid(df, **kwargs)
    p.map_lower(sns.scatterplot, s=2)  # scatter plot of samples
    p.map_upper(sns.kdeplot)  # kernel density estimate for pXY
    p.map_diag(sns.kdeplot)  # kde for pX and pY
    return p

plot_samples_with_kde(XY_df)
plt.show()
```

**Exercise** Complete the following code by replacing the blanks `___` so that `XY_ref` stores the i.i.d. samples of $(\R{X}',\R{Y}')$ where $\R{X}'$ and $\R{Y}'$ are zero-mean independent gaussian random variables with unit variance.

```Python
...
cov_ref, n_ = ___, n
XY_ref = XY_ref_rng_ref.___(mean, ___, n_)
...
```

```python
XY_ref_rng = np.random.default_rng(SEED)
### BEGIN SOLUTION
cov_ref, n_ = [[1, 0], [0, 1]], n
XY_ref = XY_ref_rng.multivariate_normal(mean, cov_ref, n_)
### END SOLUTION
XY_ref_df = pd.DataFrame(XY_ref, columns=["X", "Y"])
plot_samples_with_kde(XY_ref_df)
plt.show()
```

**Can we generalize the problem further?**

Estimating MI may be viewed as a special case of the following problem:

---

Estimate the KL *divergence*

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) &:= E\left[\log \frac{d P_{\R{Z}}(\R{Z})}{d P_{\R{Z}'}(\R{Z})} \right]
\end{align}
$$ (D)

using
- a sequence $\R{Z}^n:=(\R{Z}_1,\dots, \R{Z}_n)\sim P_{\R{Z}}^n$ of i.i.d. samples from $P_{\R{Z}}$ if $P_{\R{Z}}$ is unknown, and
- another sequence ${\R{Z}'}^{n'}\sim P_{\R{Z}'}^{n'}$ of i.i.d. samples from $P_{\R{Z}'}$  if $P_{\R{Z}'}$, the *reference measure* of $P_{\R{Z}}$, is also unknown.

---

---
**Exercise**

Although $\R{X}^n$ and $\R{Y}^n$ for MI estimation should have the same length, $\R{Z}^n$ and ${\R{Z}'}^{n'}$ can have different lengths, i.e., $n \not\equiv n'$. Why?

**Solution** The dependency between $\R{Z}$ and $\R{Z}'$ does not affect the divergence.

---

Regarding the mutual information as a divergence from joint to product distributions, the problem can be further generalized to estimtate other divergences such as the $f$-divergence:

For a strictly convex function $f$ with $f(1)=0$,

$$
\begin{align}
D_f(P_{\R{Z}}\|P_{\R{Z}'}) &:= E\left[ f\left(\frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')}\right) \right].
\end{align}
$$ (f-D)

$f$-divergence in {eq}`f-D` reduces to KL divergence when $f=u \log u$:

$$
\begin{align}
E\left[ \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')} \log \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')}  \right] &= \int_{\mc{Z}} \color{gray}{d P_{\R{Z}'}(z)} \cdot \frac{d P_{\R{Z}}(z)}{\color{gray}{d P_{\R{Z}'}(z)}} \log \frac{d P_{\R{Z}}(z)}{d P_{\R{Z}'}(z)}.
\end{align}
$$

---

**Exercise**

Show that $D_f(P_{\R{Z}}\|P_{\R{Z}'})\geq 0$ with equality iff $P_{\R{Z}}=P_{\R{Z}'}$ using Jensen's inequality and the properties of $f$.

**Solution**

It is a valid divergence because, by Jensen's inequality,

$$
D_f(P_{\R{Z}}\|P_{\R{Z}'}) \geq  f\bigg( \underbrace{E\left[ \frac{d P_{\R{Z}}(\R{Z}')}{d P_{\R{Z}'}(\R{Z}')} \right]}_{=1}\bigg) = 0
$$

with equality iff $P_{\R{Z}}=P_{\R{Z}'}$.

---

Regarding the divergence as an expectation, it is approximated by the sample average:

$$
\begin{align}
D_f(P_{\R{Z}}\|P_{\R{Z}'}) &\approx
\frac1n \sum_{i\in [n]} f\left(\frac{d P_{\R{Z}}(\R{Z}'_i)}{d P_{\R{Z}'}(\R{Z}'_i)}\right).
\end{align}
$$ (avg-f-D)

However, this is not a valid estimate because it involves the unknown measures $P_{\R{Z}}$ and $P_{\R{Z}'}$.

One may further estimate the *density ratio*

$$
\begin{align}
z \mapsto \frac{d P_{\R{Z}}(z)}{d P_{\R{Z}'}(z)}
\end{align}
$$ (dP-ratio)

or estimate the density defined respective to some reference measure $\mu$:

$$
\begin{align}
p_{\R{Z}}&:=\frac{dP_{\R{Z}}}{d\mu} \in \mc{P}_{\mu}(\mc{Z}).
\end{align}
$$ (density)

## Neural estimation of KL divergence

Estimating MI well neither require nor imply the divergence/density to be estimated well. However,
- MI estimation is often not the end goal, but an objective to train a neural network to return the divergence/density.
- The features/representations learned by the neural network may be applicable to different downstream inference tasks.

To explain the idea of neural estimation, consider the following characterization of divergence:

---
**Proposition**

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) & =  \sup_{Q\in \mc{P}(\mc{Z})} E \left[ \log \frac{dQ(\R{Z})}{dP_{\R{Z}'}(\R{Z})} \right]
\end{align}
$$ (D1)

where the unique optimal solution is $Q=P_{\R{Z}}$.

---

{eq}`D1` is {eq}`D` but with $P_{\R{Z}}$ replaced by a parameter $Q$.

- The proposition essentially gives a tight lower bound on KL divergence.
- The unknown distribution is recovered as the optimal solution.

---

**Proof**

To prove {eq}`D1`,

$$
\begin{align*}
D(P_{\R{Z}}\|P_{\R{Z}'})  &= D(P_{\R{Z}}\|P_{\R{Z}'}) - \inf_{Q\in \mc{P}(\mc{Z})} \underbrace{D(P_{\R{Z}}\|Q)}_{\geq 0 \text{ with equality iff } Q=P_{\R{Z}}\kern-3em} \\
&= \sup_{Q\in \mc{P}(\mc{Z})}  \underbrace{D(P_{\R{Z}}\|P_{\R{Z}'})}_{=E \left[\frac{dP_{\R{Z}}(\R{Z})}{dP_{\R{Z}'}(\R{Z})}\right]} -  \underbrace{D(P_{\R{Z}}\|Q)}_{=E \left[\frac{dP_{\R{Z}}(\R{Z})}{dQ(\R{Z})}\right]}\\
&= \sup_{Q\in \mc{P}(\mc{Z})} E \left[\frac{dQ(\R{Z})}{dP_{\R{Z}'}(\R{Z})}\right]
\end{align*}
$$

---

The idea of neural estimation is to

- estimate the expectation in {eq}`D1` by the sample average

$$
\frac1n \sum_{i\in [n]} \log \underbrace{\frac{dQ(\R{Z}_i)}{dP_{\R{Z}'}(\R{Z}_i)}}_{\text{(*)}},
$$

- use a neural network to compute the density ratio (*), and train the network to maximizes the expectation, e.g., by gradient ascent on the above sample average.

Since $Q$ is arbitrary, the sample average above is a valid estimate.

**But how to compute the density ratio?**

We will first consider estimating the KL divergence $D(P_{\R{Z}}\|P_{\R{Z}'})$ when both $P_{\R{Z}}$ and $P_{\R{Z}'}$ are unknown.

## Donsker-Varadhan formula

If $P_{\R{Z}'}$ is unknown, we can apply a change of variable

$$
r(z) = \frac{dQ(z)}{dP_{\R{Z}'}(z)},
$$ (Q->r)

which absorbs the unknown reference into the parameter.

---
**Proposition**

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) & =  \sup_{\substack{r:\mc{Z}\to \mathbb{R}_+\\ E[r(\R{Z}')]=1}} E \left[ \log r(\R{Z}) \right]
\end{align}
$$ (D2)

where the optimal $r$ satisfies
$
r(\R{Z}) = \frac{dP_{\R{Z}}(\R{Z})}{dP_{\R{Z}'}(\R{Z})}.
$

---

---

**Exercise** Show using {eq}`Q->r` that the optimal solution satisfies the constraint stated in the supremum {eq}`D2`.

The constraint on $r$ is obtained from the constraint on $Q\in \mc{P}(\mc{Z})$, i.e., with $dQ(z)=r(z)dP_{\R{Z}'}(z)$,

$$
\begin{align*}
dQ(z) \geq 0 &\iff r(z)\geq 0\\
\int_{\mc{Z}}dQ(z)=1 &\iff E[r(\R{Z}')]=1.
\end{align*}
$$

---

The next step is to train a neural network that computes $r$. What about?

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) \approx \sup_{\substack{r:\mc{Z}\to \mathbb{R}_+\\ \frac1{n'}\sum_{i\in [n']} r(\R{Z}'_i)]=1}} \frac1n \sum_{i\in [n]} \log r(\R{Z}_i)
\end{align}
$$ (avg-D1)

**How to impose the constraint on $r$ when training a neural network?**

We can apply by a change of variable:

$$
\begin{align}
r(z)&=\frac{e^{t(z)}}{E[e^{t(\R{Z}')}]}.
\end{align}
$$ (r->t)

**Exercise** Show that $r$ defined in {eq}`r->t` satisfies the constraint in {eq}`D1` for all real-valued function $t:\mc{Z}\to \mathbb{R}$.

**Proof**

$$
\begin{align}
E\left[ \frac{e^{t(\R{Z}')}}{E[e^{t(\R{Z}')}]} \right] =  \frac{E\left[ e^{t(\R{Z}')} \right]}{E[e^{t(\R{Z}')}]} = 1.
\end{align}
$$

Substituting {eq}`r->t` into {eq}`D1` gives the well-known *Donsker-Varadhan (DV)* formula:

---
**Corollary** [{cite}`donsker1983asymptotic`]

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) =  \sup_{t: \mc{Z} \to \mathbb{R}} E[t(\R{Z})] - \log E[e^{t(\R{Z}')}]
\end{align}
$$ (DV)

where the optimal $t$ satisfies
$$
\begin{align}
t(\R{Z}) = \log \frac{dP_{\R{Z}}(\R{Z})}{dP_{\R{Z}'}(\R{Z})} + c
\end{align}
$$ (DV:sol)

almost surely for some constant $c$.

---

The divergence can be estimated as follows instead of {eq}`avg-D1`:

$$
\begin{align}
D(P_{\R{Z}}\|P_{\R{Z}'}) \approx \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t(\R{Z}_i) - \frac1{n'}\sum_{i\in [n']} e^{t(\R{Z}'_i)}
\end{align}
$$ (avg-DV)

In summary, the neural estimation of KL divergence is a sample average of {eq}`D` but with the unknown density ratio replaced by {eq}`r->t` trained as a neural network to maximize the estimate.

$$
D(P_{\R{Z}}\| P_{\R{Z}'}) = \underbrace{}_{\sup_{t}}\underbrace{E}_{\op{avg}} \bigg[ \log \underbrace{\frac{P_{\R{Z}}(\R{Z})}{P_{\R{Z}'}(\R{Z})}}_{\frac{e^{t(\R{Z})}}{\underbrace{E}_{\op{avg}}[e^{t(\R{Z}')}]}} \bigg].
$$

## Training a neural network

We will train a neural network with `torch` and use GPU if available:

```python
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

if DEVICE == "cuda":  # print current GPU name if available
    print("Using GPU:", torch.cuda.get_device_name(torch.cuda.current_device()))
```

    Using GPU: Tesla T4

- https://pytorch.org/docs/stable/notes/randomness.html
- https://pytorch.org/docs/stable/cuda.html?highlight=cuda#module-torch.cuda

When GPU is available, you can use GPU DASHBOARDS on the left to monitor GPU utilizations.

**How to train a neural network by gradient descent?**

We will first consider a simple implementation.

### A simple implementation of gradient descent

For simplicity, suppose we want to apply gradient descend algorithm to solve

$$ \inf_{w\in \mathbb{R}} \overbrace{e^{w\cdot z}}^{L(w):=}$$

for a given $z\in \mathbb{R}$. This can be viewed as training a simple neural network with one parameter, namely, $w$, to minimize the loss function $L(w)$.

Consider $z=-1$ in particular. Then,

$$
L(w) = e^{-w} \geq 0
$$

which is achievable with equality as $w\to \infty$. The following implements the loss function:

**How to implement the loss function?**

We will define the loss function using tensors:

```python
z = Tensor([-1]).to(DEVICE)  # default tensor type on a designated device

def L(w):
    return (w * z).exp()

L(float("inf"))
```

    tensor([0.], device='cuda:0')

The function `L` is vectorized because Tensor follows the [broadcasting rules of `numpy`](https://numpy.org/doc/stable/user/basics.broadcasting.html):

```python
ww = np.linspace(0, 10, 100)
ax = sns.lineplot(
    x=ww,
    y=L(Tensor(ww).to(DEVICE)).cpu().numpy(),  # convert to numpy array for plotting
)
ax.set(xlabel=r"$w$", title=r"$L(w)=e^{-w}$")
ax.axhline(L(float("inf")), ls="--", c="r")
```

**What is gradient descent?**

A gradient descent algorithm updates the parameter $w$ iteratively starting with some initial weight $w^{(0)}$ as follows:

$$w^{(i+1)} = w^{(i)} - s^{(i)} \nabla L(w^{(i)}) \qquad \text{for }i\geq 0,$$

where $s$ is a small step size (or *learning rate*). For simplicity, fix $s^{(i)}=0.001$ independent of $i$.

**How to compute the gradient?**

With $w^{(0)}=0$,

$$\nabla L(w^{(0)}) = \left.-e^{-w}\right|_{w=0}=-1,$$

which can be computed using `backward` ([backpropagation][bp]):

[bp]: https://en.wikipedia.org/wiki/Backpropagation

```python
w = Tensor([0]).to(DEVICE).requires_grad_()  # requires gradient calculation for w
L(w).backward()  # calculate the gradient by backpropagation
w.grad
```

    tensor([-1.], device='cuda:0')

Under the hood, a computational graph is generated for backpropagation to the parameters that requires gradient calculations.

**How to implement the gradient descent?**

The following implements the gradient algorithm with a learning rate of `0.001`:

```python
for i in range(1000):
    w.grad = None  # zero the gradient to avoid accumulation
    L(w).backward()
    w.grad
    with torch.no_grad():  # updates the weights in place without gradient calculation
        w -= w.grad * 1e-3

print("w:", w.item(), "\nL(w):", L(w).item())
```

    w: 0.6933202147483826
    L(w): 0.4999134838581085

[`with torch.no_grad():`][no_grad] sets up a context where computations are not recorded in the computational graph for backpropagation. The weight update, in particular, should not be differentiated in the subsequent calculate of the gradient.

[no_grad]: https://pytorch.org/docs/stable/generated/torch.no_grad.html

---

**Exercise** Repeatedly run the above cell until you get `L(w)` below `0.001`. How large is the value of `w`? What is the limitations of the simple gradient descent algorithm?

**Solution** The value of `w` needs to be smaller than `6.9`. The convergence can be slow, especially when the learning rate is small. Also, `w` can be far away from its optimal value even if `L(w)` is close to its minimum.

---

### A practical implementation

For a neural network to fit a sophisticated function, it needs to have many degrees of freedom, i.e., number of parameters.

**How to define a neural network?**

The following code [defines a simple neural networks][define] with 3 fully-connected (fc) hidden layers:

where

- $\M{W}_l$ and $\M{b}_l$ are the weight and bias respectively for the linear transformation $\R{W}_l a_l + b_l$ of the $l$-th layer; and
- $\sigma$ for the first 2 hidden layers is an activation function chosen to be the [*exponential linear unit (ELU)*](https://pytorch.org/docs/stable/generated/torch.nn.ELU.html).

[define]: https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#define-the-network

```python
class Net(nn.Module):
    def __init__(self, input_size=2, hidden_size=100, sigma=0.02):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)  # fully-connected (fc) layer
        self.fc2 = nn.Linear(hidden_size, hidden_size)  # layer 2
        self.fc3 = nn.Linear(hidden_size, 1)  # layer 3
        nn.init.normal_(self.fc1.weight, std=sigma)  #
        nn.init.constant_(self.fc1.bias, 0)
        nn.init.normal_(self.fc2.weight, std=sigma)
        nn.init.constant_(self.fc2.bias, 0)
        nn.init.normal_(self.fc3.weight, std=sigma)
        nn.init.constant_(self.fc3.bias, 0)

    def forward(self, z):
        a1 = F.elu(self.fc1(z))
        a2 = F.elu(self.fc2(a1))
        t = self.fc3(a2)
        return t

torch.manual_seed(SEED)  # seed RNG for PyTorch
net = Net()
net.to(DEVICE)
print(net)
```

    Net(
      (fc1): Linear(in_features=2, out_features=100, bias=True)
      (fc2): Linear(in_features=100, out_features=100, bias=True)
      (fc3): Linear(in_features=100, out_features=1, bias=True)
    )

The neural network is a vectorized function, i.e., we can pass in multiple values of $z$'s (along the first dimension) to obtain multiple $t(z)$'s. E.g., the following plots the density estimate of $t(\R{Z}_i)$'s and $t(\R{Z}'_i)$'s.

```python
Z = Tensor(XY).to(DEVICE)
Z_ref = Tensor(XY_ref).to(DEVICE)

tZ = (
    net(torch.cat((Z, Z_ref), dim=0))  # compute t(Z_i)'s and t(Z'_i)
    # output needs to be converted back to an array on CPU for plotting
    .cpu()  # copy back to CPU
    .detach()  # detach from current graph (no gradient calculation)
    .numpy()  # convert output back to numpy
)

tZ_df = pd.DataFrame(data=tZ, columns=["t"])
sns.kdeplot(data=tZ_df, x="t")
plt.show()
```

---

**Exercise** Why are the values concentrated around $0$?

**Solution** The values are all very close to $0$ as we have set a small variance `sigma=0.02` to initialize the neural network parameters:

- The linear transformation $\M{W}_l (\cdot) + \M{b}_l$ is close to $0$ for when the weight and bias are close to $0$.
- The ELU activation function $\sigma$ is also close to $0$ if its input is close to $0$.

---

We can use the neural network to compute the approximate divergence lower bound in {eq}`avg-DV` as follows:

$$
\begin{align}
\R{L}(\theta) &:= \underbrace{\frac1{n} \sum_{i\in [n]} t(\R{Z}_i)}_{\text{(a)}} - \underbrace{\log \frac1{n'} \sum_{i\in [n']} e^{t(\R{Z}'_i)}}_{ \underbrace{\log \sum_{i\in [n']} e^{t(\R{Z}'_i)}}_{\text{(b)}} - \underbrace{\log n'}_{\text{(c)}}}
\end{align}
$$

where $\theta$ is a tuple of parameters (weights and biases) of the neural network that computes $t$:

$$
\theta := (\M{W}_l,\M{b}_l|l\in [3]).
$$

```python
def DV(Z, Z_ref, net):
    avg_tZ = net(Z).mean()  # (a)
    avg_etZ_ref = net(Z_ref).logsumexp(dim=0) - np.log(Z_ref.shape[0])  # (b) - (c)
    return avg_tZ - avg_etZ_ref

DV_estimate = DV(Z, Z_ref, net)
```

---

**Exercise** Why is it preferrable to use `logsumexp(dim=0)` instead of `.exp().mean().log()`?

**Solution** `logsumexp(dim=0)` is numerically more stable than `.exp().mean().log()` especially when the output of the exponential function is too large to be represented by the default floating point type. This can lead to an overall value of `NaN`, even if the output after taking the `mean()` and `log()` is representable.

---

To calculate the gradient of the divergence estimate with respect to $\theta$:

```python
net.zero_grad()  # zero the gradient
DV(Z, Z_ref, net).backward()  # calculate the gradient
a_param = next(net.parameters())
```

`a_param` is a (module) parameter in $\theta$ retrieved from the parameter iterator `parameters()`.

---

**Exercise** Check the value of `a_param.grad` is non-zero. Is `a_param` a weight or a bias?

**Solution** It should be the weight matrix $\M{W}_1$ because the shape is `torch.Size([100, 2])`.

---

We will use the [*Adam's* gradient descend algorithm][adam] implemented as an optimizer [`optim.Adam`][optimAdam]:

[adam]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent#cite_note-Adam2014-28
[optimAdam]: https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam

```python
optimizer = optim.Adam(net.parameters())  # Allow Adam's optimizer to update the neural network parameters
optimizer.step() # perform one step of the gradient descent
```

To alleviate the problem of overfitting, the gradient is often calculated on randomly chosen batches of the samples:

```python
n_iters_per_epoch = 10  # ideally a divisor of both n and n'
batch_size = int((Z.shape[0] + 0.5) / n_iters_per_epoch)
batch_size_ref = int((Z_ref.shape[0] + 0.5) / n_iters_per_epoch)
```

We will use `tensorboard` to show the training logs. Rerun the following to create a new log, for instance, after a change of parameters.

```python
n_iter = n_epochs = 0  # keep counts for logging
writer = SummaryWriter()  # create a new folder under runs/ for logging
torch.manual_seed(SEED) # for reproducibility
```

The following code carries out Adam's gradient descent on batch loss:

$$
\begin{align}
\R{L}(\theta) := - \bigg[\frac1{\abs{\R{B}}} \sum_{i\in \R{B}} t(\R{Z}_i) - \log \frac1{\abs{\R{B}'}} \sum_{i\in \R{B}'} e^{t(\R{Z}'_i)} - \log \abs{\R{B}'} \bigg],
\end{align}
$$

which is the negative lower bound of the VD formula in {eq}`DV` but on the minibatches

$$\R{Z}_{\R{B}}:=(\R{Z}_i\mid i\in \R{B})\quad \text{and}\quad \R{Z}'_{\R{B}'}$$

where $\R{B}$ and $\R{B}'$ are uniformly randomly chosen indices from $[n]$ and $[n']$ respectively.

```python
if input('Train? [Y/n]').lower() != 'n':
    for i in range(100):  # loop through entire data multiple times
        n_epochs += 1
        # Random indices for selecting samples for all batches in one epoch
        idx = torch.randperm(Z.shape[0])
        idx_ref = torch.randperm(Z_ref.shape[0])

        for j in range(n_iters_per_epoch):  # loop through multiple batches
            n_iter += 1
            optimizer.zero_grad()
            # obtain a random batch of samples
            batch_Z = Z[idx[i : Z.shape[0] : batch_size]]
            batch_Z_ref = Z_ref[idx_ref[i : Z_ref.shape[0] : batch_size_ref]]
            # define loss as negative divergence lower bound from VD formula
            loss = -DV(batch_Z, batch_Z_ref, net)
            loss.backward()  # calculate gradient
            optimizer.step() # descend

        writer.add_scalar("Loss/train", loss.item(), global_step=n_epochs)

    # Estimate the divergence using all data
    with torch.no_grad():
        writer.add_scalar("Div estimate", DV(Z, Z_ref, net).item(), global_step=n_epochs)
```

Run the following to show the losses and divergence estimate in `tensorboard`. You can rerun the above cell to train the neural network more.

```python
if input('Run tensorboard? [Y/n]').lower() != 'n':
    %tensorboard --logdir=runs
```

    The tensorboard extension is already loaded. To reload it, use:
      %reload_ext tensorboard

    Reusing TensorBoard on port 6006 (pid 74055), started 0:07:10 ago. (Use '!kill 74055' to kill it.)

**Exercise** The ground truth of is given by

$$D(P_{\R{Z}}\|P_{\R{Z}'}) = \frac12 \log(1-\rho^2) $$

where $\rho$ is the randomly generated correlation in the previous notebook. See if you can get an estimate close to this value by training the neural network repeatedly.

```python
ground_truth = -0.5*np.log(1-rho**2)
ground_truth
```

    0.7405246745135301

## Clean-up

It is important to release the resources if it is no longer used. You can release the memory or GPU memory by `Kernel->Shut Down Kernel`.

To clear the logs:

```python
if input('Delete logs? [y/N]').lower() == 'y':
    !rm -rf ./runs
```

    Delete logs? [y/N]

To kill a tensorboard instance without shutting down the notebook kernel:

```python
tb.notebook.list() # list all the running TensorBoard notebooks.
while (pid := input('pid to kill? (press enter to exit)')):
    !kill {pid}
```

    No known TensorBoard instances running.

    pid to kill? (press enter to exit)


```
# default_exp assignment_part_1_solution
```

# Assignment Part 1 Solution

--------------------

## Laplace Transforms

1. Calculate the Laplace Transform $X(s)$ of the signal $x(t)$:

$$
x(t) = (4t-3cos(5t))e^{-2t}
$$

$$
x(t)=4te^{-2t} - 3cos(5t)e^{-2t} \rightarrow X(s) = 4 \frac{1}{(s+2)^2} - 3 \frac{(s+2)}{(s+2)^2 + 25}
$$

Let's verify it with Sympy

```
import matplotlib.pyplot as plt # import matplotlib - it will be useful later.
import sympy # import sympy
t, s = sympy.symbols('t, s')
```

```
f1 = 4*t*sympy.exp(-2*t)
f2 = - 3*sympy.cos(5*t)*sympy.exp(-2*t)
print('{} {}'.format(f1, f2))
```

    4*t*exp(-2*t) -3*exp(-2*t)*cos(5*t)

```
F1 = sympy.laplace_transform(f1, t, s, noconds=True)
F2 = sympy.laplace_transform(f2, t, s, noconds=True)
print('{} {}'.format(F1, F2))
```

    4/(s + 2)**2 -(3*s + 6)/((s + 2)**2 + 25)

2. Calculate the Inverse Laplace Transform $g(t)$ of the transfer function $G(s)$:

$$
G(s)=2 + \frac{3}{2s^3+3s^2+s}
$$

$$
G(s)=2 + \frac{3}{2s^3+3s^2+s} = 2 + \frac{3}{s(2s^2+3s+1)} = 2 + \frac{3}{2s(s+1)(s+\frac{1}{2})} = 2 + \frac{3}{s} + \frac{3}{s+1} + \frac{-6}{s+\frac{1}{2}}
$$

$$
g(t) = 2\delta(t) + 3 + 3e^{-t} -6e^{-0.5t}
$$

```
F1 = 2
F2 = 3/(2*s**3+3*s**2+s)
```

```
f1 = sympy.inverse_laplace_transform(F1, s, t)
f2 = sympy.inverse_laplace_transform(F2, s, t)
print('{} + {}'.format(f1, f2))
```

    2*DiracDelta(t) + 3*Heaviside(t) + 3*exp(-t)*Heaviside(t) - 6*exp(-t/2)*Heaviside(t)

```
f3 = sympy.inverse_laplace_transform(F1+F2, s, t)
print('{}'.format(f3))
```

    2*DiracDelta(t) + 3*Heaviside(t) + 3*exp(-t)*Heaviside(t) - 6*exp(-t/2)*Heaviside(t)

## Block Diagrams

3. Calculate the equivalent transfer function $G_t(s)=\frac{Y(s)}{R_1(s)}$ of the following block diagram:

$$
\frac{G_1G_4(G_2+G_3)}{1+G_1G_4H_1 + H_2G_1G_4(G_2+G_3)}
$$

## System Response

4. Write the transfer function to a step input of a second order system characterised by:
- Static gain $G(0)=5$
- Damping ratio $\xi=0.5$
- Settling time $t_s=3 s$
- No zeros

We would like a system with form:
$$
\frac{K}{s^2+2\xi\omega_ns+\omega_n^2}
$$

we also know that:

$$
t_s \approx -\frac{1}{\xi\omega_n}ln(0.05)\;\; s
$$

from which:

$$
\omega_n = -\frac{1}{3\cdot0.5}ln(0.05) = 2 \;\; rad/s
$$

which means:

$$
\frac{K}{s^2+2s+4}
$$

since we want $G(0)=5 \rightarrow K=20$

$$
\frac{20}{s^2+2s+4}
$$

5. Plot the qualitative behaviour of the step response of the system:

$$
G(s)=\frac{20(3+0.1s)(s^2+10s+160)}{(2s+10)(0.1s+5)(s^2+2s+400)}
$$

Let's re-write the transfer function slightly:

$$
G(s)=\frac{20\cdot0.1(30+s)(s^2+10s+160)}{2(s+\frac{10}{2})0.1(s+\frac{5}{0.1})(s^2+2s+400)}
$$

The poles are:

$$
s = -5, -50, -1.+19.97498436j, -1.-19.97498436j
$$

and the zeros are:

$$
s = -30, -5.+11.61895004j, -5.-11.61895004j
$$

The system is asymptotically stable, we can use the dominant poles approximation:

$$
G(s)=\frac{20\cdot0.1\cdot30\cdot160}{2\cdot5\cdot0.1\cdot50(s^2+2s+400)} = \frac{192}{(s^2+2s+400)}
$$

The response to a step input can be obtained as:

$$
Y(s) = G(s)\frac{1}{s} = \frac{192}{(s^2+2s+400)}\frac{1}{s}
$$

And, using partial fraction decomposition:

$$
Y(s) = \frac{K}{s} + \frac{A}{s-p_1} + \frac{A^*}{s-p_1^*}
$$

Where

$$
K = \frac{192}{(s^2+2s+400)} \bigg |_{s=0} = \frac{192}{400} = 0.48
$$

And for the pole $p_1=−1.+19.97498436𝑗$

We know that when we have complex conjugate poles - see notebook `03_Transfer_function`:

$$
f(t) = |A|e^{\sigma t} \big [   e^{j(\Phi_A+\omega t)} + e^{-j(\Phi_A+\omega t)}  \big ] = 2|A|e^{\sigma t}\cos(\Phi_A+\omega t)
$$

where:

$$
A = (s-p_1)\frac{192}{s(s-p_1)(s-p_1^*)} \bigg |_{s=-1+20j} = \frac{192}{s(s-p_1^*)} \bigg |_{s=-1+20j} = \frac{192}{(-1+20j)(
40j)} = -0.24+0.012j
$$

```
abs(-0.24+0.012)
```

    0.22799999999999998

```
abs(192/(40j*(-1+20j)))
```

    0.2397005613306827

```
import numpy as np
np.sqrt(399)
```

    19.974984355438178

and the response is

$$
y(t) = K + Ae^{p_1}t + A^*e^{p_1^*}t = K + 2|A|e^{\sigma t}cos(\omega t + \angle{A})
$$

where $p_1=\sigma+\omega t$ and $A=|A|e^{j\angle{A}}$

$$
y(t) = 0.48 + 2*0.23*e^{-1t}cos(20t -\frac{\pi}{2})
$$

We can check that the inverse laplace of the system transfer function using Sympy:

```
def evaluate(f, times):
    res = []
    for time in times:
        res.append(f.evalf(subs={t:time}).n(chop=1e-5))
    return res
def L(f):
    return sympy.laplace_transform(f, t, s, noconds=True)
def invL(F):
    return sympy.inverse_laplace_transform(F, s, t)
```

```
t, s = sympy.symbols('t, s')
F = 192/(s**2+2*s+400)
invL(F)
```

$\displaystyle \frac{64 \sqrt{399} e^{- t} \sin{\left(\sqrt{399} t \right)} \theta\left(t\right)}{133}$

Where:

```
import numpy as np
import cmath
np.sqrt(399)
```

    19.974984355438178

And now we can plot it to verify the step response:

```
time = np.linspace(0, 7, 1000)
y = 0.48 + 2*0.23*np.exp(-time)
plt.plot(time, y)
plt.xlabel('time');
```

Finally, let's compare the result we got with what Sympy calculates

```
fig, ax = plt.subplots(1,1,figsize=(8,5))

time = np.linspace(0,7,1000)

ax.plot(time, evaluate(invL(F*1/s), time), linewidth=3)
ax.plot(time, y, 'g')

ax.set_title(f'y(end)={y[-1]}');
```

Let's look at the response more closely:

```
plt.plot(time[:100], y[:100])
```

We can also use the Python Control Library to verify our dominant pole approximation

```
import control
import matplotlib.pyplot as plt
```

```
s = control.TransferFunction.s
G = (20*(3+0.1*s)*(s**2+10*s+160))/((2*s+10)*(0.1*s+5)*(s**2 + 2*s + 400))
G_approx = 192/(s**2 + 2*s + 400)
```

```
T, yout = control.step_response(G, T=np.linspace(0, 7, 1000));
Tap, youtap = control.step_response(G_approx, T=np.linspace(0, 7, 1000));

plt.plot(T, yout, 'b', label='original')
plt.plot(Tap, youtap, 'r', label='approx d.p.')
plt.legend()
plt.grid()
```

6. For the system $G(s)$ defined above, calculate:

- Its steady state value $y_\infty$ to a step input (when $t \rightarrow \infty$)
- Its settling time $t_s$
- If the system oscillates, calculate the period $T_\omega$ of the oscillation

- $y(t\rightarrow\infty) = 0.48$
- $t_s \approx -\frac{1}{\xi\omega_n}ln(0.05) \approx 3$

given that:
 - $\omega_n=\sqrt{399}\approx20$
 - $2\xi\omega_n=2 \rightarrow 20 \xi =1 \rightarrow \xi = 0.05$

And finally:
- $T_{P} = \frac{2\pi}{w_n\sqrt{1-\xi^2}} = 0.314s$

```
xi = 0.05
wn = 20
t_s = -1/(xi*wn)*np.log(0.05)
print('t_s = {}'.format(t_s))

Tp = 2*3.14/(wn*np.sqrt(1-xi**2))

print('T_p = {}'.format(Tp))
```

    t_s = 2.995732273553991
    T_ = 0.3143932374740646

## Routh Criterion

7. Determine the values of $K$ for which the following feedback system is asymptotically stable:

The characteristic equation of the feedback system is:

$$
1+K\frac{s^2+2s+64}{s^2(s+0.5)} = 0 \rightarrow s^2(s+0.5)+K(s^2+2s+64) = 0 \rightarrow s^3 + (0.5+K)s^2 + 2Ks+64K=0
$$

The Routh table is:

| | | |
|-|-|-|
|$3$| $1$ | $2K$ |
|$2$| $0.5+K$ | $64K$ |
|$1$| $(0.5+K)2K-64K$ | $0$ |
|$0$| $64K$ | |

from which it is possible to find the following constraints on $K$:

$$
0.5 + K > 0, \hspace{0.5cm} 2K^2 +1K-64K > 0, \hspace{0.5cm}K>0
$$

$$
K>0.5, \hspace{0.5cm} K>31.5, \hspace{0.5cm} K>0
$$

And the system is asymptotically stable for $K>31.5$

## Bode Plots

8. Plot the Bode amplitude and phase plots for the system discussed in question 7.

$$
G_d(s) = \frac{s^2+2s+64}{s^2(s+0.5)}= \frac{64(\frac{s^2}{64}+\frac{2s}{64}+1)}{0.5s^2(\frac{s}{0.5}+1)}
$$

$K_{dB}=20\log(64/0.5) \approx 42 dB$

## DC Motor Transfer Functions

The figures below represents a DC motor attached to an inertial load.

The input voltage can be applied to either the field or the armature terminals.
The voltages applied to the field and armature sides of the motor are represented by $V_f$ and $V_a$.

The resistances and inductances of the field and armature sides of the motor are represented by $R_f$ , $L_f$, $R_a$, and $L_a$.
The torque generated by the motor can be assumed to be related linarly to $i_f$ and $i_a$, the currents in the field and armature sides of the motor, as follows:

$$
T_m = K i_f i_a  \;\;\;\;(1)
$$

From the equation above, clearly to retain the linearity, one current must be kept constant, while the other becomes the input current.

This makes it possible to have field-current or armature-current controlled motors.
We will focus on the field-current motor for our analysis.

This means that in a field-current controlled motor, the armature current is kept constant, while the field current $i_a$ is controlled through the field voltage $V_f$.

$$
T_m = K i_f i_a = K_m i_f  \;\;\;\;(2)
$$

where $K_m$ is defined as the motor constant. The motor torque increases linearly with the field current.

**1. Write the Transfer Function from the input current to the resulting torque of the previous equation**

$$
 \frac{T_m(s)}{I_f(s)} = K_m
$$

For the field side of the motor the voltage/current relationship is
$$
V_f =V_R +V_L
 = R_f i_f +L_f (di_f dt)   \;\;\;\;(3)
$$

 **2. Write the Transfer Function from the input voltage to the resulting current**

 Vf = RfI+LsI

 I = Vf/Lf/(Rf/Lf+s)

$$
\frac{I_f(s)}{V_f(s)} = \frac{\frac{1}{L_f}}{s+\frac{R_f}{L_f}}  \;\;\;\;(4)
$$

**3. Write the Transfer Function from the input voltage to the resulting motor torque**

$$
\frac{T_m(s)}{V_f(s)} = \frac{\frac{K_m}{L_f}}{s+\frac{R_f}{L_f}}   \;\;\;\;(5)
$$

**4. Discuss how the motor torque behaves with respect to different input signals (e.g., step input in field voltage, etc.)**

This is a first-order system (type 0), and a step input in field voltage $V_f$ results in an exponential rise in the motor torque.

------------------

We can now add load to the motor and verify how its behaviour changes.

The motor torque $T_m(s)$ is equal to the torque delivered to the load, and this relation can be expressed as:

$$
T_m(s) = T_L(s) + T_d(s) \;\;\;\;(6)
$$

where $T_L(s)$, and $T_d(s)$ is the disturbance torque (e.g., external forces acting on the load).

We can calculate the rotational motion of the inertial load summing moments (see Figures above):

$$
\sum{M} = T_L - f\omega = J \dot{\omega} \;\;\;\;(7)
$$

where $f$ is due to friction, and $J$ is the load inertia.

**5. Write the Laplace transform of the load torque $T_L(s)$ using using equation (7) above**

$$
T_L(s) = Js\Omega(s)+f\Omega(s)
$$

**6. Given that the relationship between position and angular velocity $\omega$, refine the equation you have just calculated to explicit the rotor position $\theta$ (_hint:_ $\dot\theta = \omega$)**

$$
T_L(s) = Js^2\theta(s) + fs\theta(s)
$$

where $\Omega(s) = \mathcal{L}(\omega(t))$

**7. Put together equations (2,4,6) and calculate the transfer function of the motor-load combination**

$$
\frac{\theta(s)}{V_f(s)} = \frac{K_m}{s(Js+f)(L_fs+R_f)} = \frac{\frac{K_m}{JL_f}}{s(s+\frac{f}{J})(s+\frac{R_f}{L_f})}. \;\;\;\;(8)
$$

**8. Discuss the resuting system (order, dominant pole approximations, what happens when parameters change, select specific values for the parameters and draw its step response.)**

$$
\frac{\theta(s)}{V_f(s)} = \frac{\frac{K_m}{JL_f}}{s(s+\frac{f}{J})(s+\frac{R_f}{L_f})} = \frac{\frac{K_m}{fR_f}}{s(\tau_f s+1)(\tau_L s+1)}
$$

where $\tau_f=\frac{L_f}{R_f}$ and $\tau_L=\frac{J}{f}$.

When $\tau_L > \tau_f$, the field time constant $\tau_f$ can be neglected.

Let's choose the following values:

- (J)     moment of inertia of the rotor     0.01 kg.m^2
- (f)     motor viscous friction constant    0.1 N.m.s
- (Ke)    electromotive force constant       0.01 V/rad/sec
- (Kt)    motor torque constant              0.01 N.m/Amp
- (R)     electric resistance                1 Ohm
- (L)     electric inductance                0.5 H

_Note that in SI units, $K_e = K_t = K$_

$$
\frac{\theta(s)}{V_f(s)} = \frac{\frac{0.01}{0.1\cdot1}}{s(0.5/1s+1)(0.01/0.1s+1)} = \frac{0.1}{s(0.5s+1)(0.1s+1)}
$$

```
s = control.TransferFunction.s
G_motor = 0.1/(s*(0.5*s+1)*(0.1*s+1))
```

```
T, yout = control.step_response(G_motor, T=np.linspace(0, 7, 1000));

plt.plot(T, yout, 'b', label='dc-motor-step-response')
plt.legend()
plt.xlabel('time')
plt.grid()
```

**9. Draw a block diagram of the field controlled DC motor from the field voltage to the position output using all the relevant equations calculated so far. Make sure you include the disturbance $T_d(s)$ as per equation (6).**

```

```


# 14 Linear Algebra: String Problem
## Motivating problem: Two masses on three strings
Two masses $M_1$ and $M_2$ are hung from a horizontal rod with length $L$ in such a way that a rope of length $L_1$ connects the left end of the rod to $M_1$, a rope of length $L_2$ connects $M_1$ and $M_2$, and a rope of length $L_3$ connects $M_2$ to the right end of the rod. The system is at rest (in equilibrium under gravity).

Find the angles that the ropes make with the rod and the tension forces in the ropes.

## Theoretical background
Treat $\sin\theta_i$ and $\cos\theta_i$ together with $T_i$, $1\leq i \leq 3$, as unknowns that have to simultaneously fulfill the nine equations
\begin{align}
-T_1 \cos\theta_1 + T_2\cos\theta_2 &= 0\\
 T_1 \sin\theta_1 - T_2\sin\theta_2 - W_1 &= 0\\
                   -T_2\cos\theta_2 + T_3\cos\theta_3 &= 0\\
                    T_2\sin\theta_2 + T_3\sin\theta_3 - W_2 &= 0\\
 L_1\cos\theta_1 + L_2\cos\theta_2 + L_3\cos\theta_3 - L &= 0\\
-L_1\sin\theta_1 -L_2\sin\theta_2 + L_3\sin\theta_3  &= 0\\
\sin^2\theta_1 + \cos^2\theta_1 - 1 &= 0\\
\sin^2\theta_2 + \cos^2\theta_2 - 1 &= 0\\
\sin^2\theta_3 + \cos^2\theta_3 - 1 &= 0
\end{align}

Consider the nine equations a vector function $\mathbf{f}$ that takes a 9-vector $\mathbf{x}$ of the unknowns as argument:
\begin{align}
\mathbf{f}(\mathbf{x}) &= 0\\
\mathbf{x} &= \left(\begin{array}{c}
x_0 \\ x_1 \\ x_2 \\
x_3 \\ x_4 \\ x_5 \\
x_6 \\ x_7 \\ x_8
\end{array}\right)
 =
\left(\begin{array}{c}
\sin\theta_1 \\ \sin\theta_2 \\ \sin\theta_3 \\
\cos\theta_1 \\ \cos\theta_2 \\ \cos\theta_3 \\
T_1 \\ T_2 \\ T_3
\end{array}\right) \\
\mathbf{L} &= \left(\begin{array}{c}
L \\ L_1 \\ L_2 \\ L_3
\end{array}\right), \quad
\mathbf{W} = \left(\begin{array}{c}
W_1 \\ W_2
\end{array}\right)
\end{align}

Solve with generalized Newton-Raphson:
$$
\mathsf{J}(\mathbf{x}) \Delta\mathbf{x} = -\mathbf{f}(\mathbf{x})
$$
and
$$
\mathbf{x} \leftarrow \mathbf{x} + \Delta\mathbf{x}.
$$

## Problem setup
Set the problem parameters and the objective function $\mathbf{f}(\mathbf{x})$

```python
import numpy as np

# problem parameters
W = np.array([10, 20])
L = np.array([8, 3, 4, 4])

def f_2masses(x, L, W):
    return np.array([
            -x[6]*x[3] + x[7]*x[4],
             x[6]*x[0] - x[7]*x[1] - W[0],
            -x[7]*x[4] + x[8]*x[5],
             x[7]*x[1] + x[8]*x[2] - W[1],
             L[1]*x[3] + L[2]*x[4] + L[3]*x[5] - L[0],
            -L[1]*x[0] - L[2]*x[1] + L[3]*x[2],
            x[0]**2 + x[3]**2 - 1,
            x[1]**2 + x[4]**2 - 1,
            x[2]**2 + x[5]**2 - 1,
        ])

def fLW(x):
    return f_2masses(x, L, W)
```

### Initial values
Guess some initial values (they don't have to fullfil the equations!):

```python
# initial parameters
#theta0 = np.deg2rad([45, 45, 90])
#T0 = np.array([1, 1, 2])
#x0 = np.concatenate([np.sin(theta0), np.cos(theta0), T0])

x0 = np.array([1.5, 0.5, 0.5, 0.5, 0.5, 0.5, 1, 1, 1])
```

```python
x0
```

    array([ 0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  1. ,  1. ,  1. ])

```python
f_2masses(x0, L, W)
```

    array([  0. , -10. ,   0. , -19. ,  -2.5,  -1.5,  -0.5,  -0.5,  -0.5])

### Visualization
Plot the positions of the 2 masses and the 3 strings for any solution vector $\mathbf{x}$:

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def plot_2masses(x, L, W, **kwargs):
    """Plot 2 mass/3 string problem for parameter vector x and parameters L and W"""

    kwargs.setdefault('linestyle', '-')
    kwargs.setdefault('marker', 'o')
    kwargs.setdefault('linewidth', 1)

    r0 = np.array([0, 0])
    r1 = r0 + np.array([L[0], 0])
    rod = np.transpose([r0, r1])

    L1 = r0 + np.array([L[1]*x[3], -L[1]*x[0]])
    L2 = L1 + np.array([L[2]*x[4], -L[2]*x[1]])
    L3 = L2 + np.array([L[3]*x[5], L[3]*x[2]])
    strings = np.transpose([r0, L1, L2, L3])

    ax = plt.subplot(111)
    ax.plot(rod[0], rod[1], color="black", marker="d", linewidth=4)
    ax.plot(strings[0], strings[1], **kwargs)
    ax.set_aspect(1)
    return ax
```

What does the initial guess look like?

```python
plot_2masses(x0, L, W)
```

## Jacobian
Write a function `Jacobian(f, x, h=1e-5)` that computes the Jacobian matrix numerically (use the central difference algorithm).

\begin{align}
\mathbf{J} &= \frac{\partial \mathbf{f}}{\partial\mathbf{x}} \\
J_{ij} &= \frac{\partial f_i(x_1, \dots, x_j, \dots)}{\partial x_j} \\
       &\approx \frac{f_i(x_1, \dots, x_j + \frac{h}{2}, \dots) - f_i(x_1, \dots, x_j - \frac{h}{2}, \dots)}{h}
\end{align}

```python
def Jacobian(f, x, h=1e-5):
    """df_i/dx_j with central difference (fi(xj+h/2)-fi(xj-h/2))/h"""
    J = np.zeros((len(f(x)), len(x)), dtype=np.float64)
    hvec = np.zeros_like(x, dtype=np.float64)
    for j in range(len(x)):
        hvec *= 0
        hvec[j] = 0.5*h
        J[:, j] = (f(x + hvec) - f(x - hvec))/h
    return J
```

Test Jacobian on
$$
\mathbf{f}(\mathbf{x}) = \left( \begin{array}{c}
  x_0^2 - x_1 \\ x_0
  \end{array}\right)
$$
with analytical result
$$
\mathsf{J} = \frac{\partial f_i}{\partial x_j} =
\left( \begin{array}{cc}
  2 x_0 & -1\\
  1 & 0
\end{array}\right)
$$

```python
def ftest(x):
    return np.array([
            x[0]**2 - x[1],
            x[0]
        ])
x0test = np.array([1, 0])
J = Jacobian(ftest, x0test)
print(J)
```

    [[ 2. -1.]
     [ 1.  0.]]

Test that it also works for our starting vector:

```python
Jacobian(fLW, x0)
```

    array([[ 0.        ,  0.        ,  0.        , -1.        ,  1.        ,
             0.        , -0.70710678,  0.70710678,  0.        ],
           [ 1.        , -1.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.70710678, -0.70710678,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        , -1.        ,
             2.        ,  0.        , -0.70710678,  0.        ],
           [ 0.        ,  1.        ,  2.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.70710678,  1.        ],
           [ 0.        ,  0.        ,  0.        ,  3.        ,  4.        ,
             4.        ,  0.        ,  0.        ,  0.        ],
           [-3.        , -4.        ,  4.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ],
           [ 1.41421356,  0.        ,  0.        ,  1.41421356,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  1.41421356,  0.        ,  0.        ,  1.41421356,
             0.        ,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  2.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ]])

## n-D Newton-Raphson Root Finding
Write a function `newton_raphson(f, x, Nmax=100, tol=1e-8, h=1e-5)` to find a root for a vector function `f(x)=0`. (See also [13 Root-finding by trial-and-error](https://asu-compmethodsphysics-phy494.github.io/ASU-PHY494/2019/03/19/13_Root_finding/) and the _1D Newton-Raphson algorithm_ in [13-Root-finding.ipynb](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/13_root_finding/13-Root-finding.ipynb).) As a convergence criterion we demand that the length of the vector `f(x)` (the norm --- see `np.linalg.norm`) be less than the tolerance.

```python
def newton_raphson(f, x, Nmax=100, tol=1e-8, h=1e-5):
    """n-D Newton-Raphson: solves f(x) = 0.

    Iterate until |f(x)| < tol or nmax steps.
    """
    x = x.copy()
    for istep in range(Nmax):
        fx = f(x)
        if np.linalg.norm(fx) < tol:
            break
        J = Jacobian(f, x, h=h)
        Delta_x = np.linalg.solve(J, -fx)
        x += Delta_x
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
            "best guess is {2} with error {3}".format(Nmax, tol, x, fx))
    return x
```

### Solve 2 masses/3 strings string problem

#### Solution

```python
x = newton_raphson(fLW, x0)
print(x0)
print(x)
```

    [ 1.5  0.5  0.5  0.5  0.5  0.5  1.   1.   1. ]
    [  0.76100269   0.26495381   0.83570583   0.64874872   0.9642611
       0.54917735  17.16020978  11.54527968  20.27152804]

Plot the starting configuration and the solution:

```python
plot_2masses(x0, L, W)
plot_2masses(x, L, W)
```

Pretty-print the solution (angles in degrees):

```python
def pretty_print(x):
    theta = np.rad2deg(np.arcsin(x[0:3]))
    tensions = x[6:]
    print("theta1 = {0[0]:.1f} \t theta2 = {0[1]:.1f} \t theta3 = {0[2]:.1f}".format(theta))
    print("T1     = {0[0]:.1f} \t T2     = {0[1]:.1f} \t T3     = {0[2]:.1f}".format(tensions))
```

```python
print("Starting values")
pretty_print(x0)
print()
print("Solution")
pretty_print(x)
```

    Starting values
    theta1 = 45.0 	 theta2 = 45.0 	 theta3 = 90.0
    T1     = 1.0 	 T2     = 1.0 	 T3     = 2.0

    Solution
    theta1 = 49.6 	 theta2 = 15.4 	 theta3 = 56.7
    T1     = 17.2 	 T2     = 11.5 	 T3     = 20.3

#### Show intermediate steps
Create a new function `newton_raphson_intermediates()` based on `newton_raphson()` that returns *all* trial `x` values including the last one.

```python
def newton_raphson_intermediates(f, x, Nmax=100, tol=1e-8, h=1e-5):
    """n-D Newton-Raphson: solves f(x) = 0.

    Iterate until |f(x)| < tol or nmax steps.

    Returns all intermediates.
    """
    intermediates = []
    x = x.copy()
    for istep in range(Nmax):
        fx = f(x)
        if np.linalg.norm(fx) < tol:
            break
        J = Jacobian(f, x, h=h)
        Delta_x = np.linalg.solve(J, -fx)
        intermediates.append(x.copy())
        x += Delta_x
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
            "best guess is {2} with error {3}".format(Nmax, tol, x, fx))
    return np.array(intermediates)
```

Visualize the intermediate configurations:

```python
x_series = newton_raphson_intermediates(fLW, x0)
```

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis_r(i) for i in np.linspace(0, 1, len(x_series))])
for iteration, x in enumerate(x_series):
    plot_2masses(x, L, W, label=str(iteration))
plt.legend(loc="best");
```

It's convenient to turn the above plotting code into a function that we can reuse:

```python
def plot_series(x_series, L, W):
    """Plot all N masses/strings solution vectors in x_series (N, 9) array"""
    ax = plt.subplot(111)
    ax.set_prop_cycle("color", [plt.cm.viridis_r(i) for i in np.linspace(0, 1, len(x_series))])
    for iteration, x in enumerate(x_series):
        plot_2masses(x, L, W, label=str(iteration))
    plt.legend(loc="best")
    return ax
```

## Additional work
Try different masses, e.g. M1 = M2 = 10, or M1= 0 , M2 = 10.

Use nicer starting parameters that already fulfill the angle equations (7) - (9) (but it works with pretty much any guess):

```python
# initial parameters
theta0 = np.deg2rad([45, 45, 90])
T0 = np.array([1, 1, 2])
x0 = np.concatenate([np.sin(theta0), np.cos(theta0), T0])
```

### M1 = M2 = 10

```python
W_2 = np.array([10, 10])
def fLW_2(x):
    return f_2masses(x, L, W_2)
```

```python
x_series_2 = newton_raphson_intermediates(fLW_2, x0)
pretty_print(x_series_2[-1])
```

    theta1 = 57.9 	 theta2 = 8.8 	 theta3 = 52.1
    T1     = 13.1 	 T2     = 7.0 	 T3     = 11.3

```python
plot_series(x_series_2, L, W_2)
```

### M1 = 0, M2 = 10

```python
W_3 = np.array([0, 10])
def fLW_3(x):
    return f_2masses(x, L, W_3)
```

```python
x_series_3 = newton_raphson_intermediates(fLW_3, x0)
pretty_print(x_series_3[-1])
```

    theta1 = 30.0 	 theta2 = 30.0 	 theta3 = 61.0
    T1     = 4.8 	 T2     = 4.8 	 T3     = 8.7

```python
plot_series(x_series_3, L, W_3)
```

```python

```


```python
%matplotlib widget
# %matplotlib inline

from sympy import *
from IPython.display import Latex
import matplotlib.pyplot as plt

plt.style.use('dark_background')
init_printing(use_latex=True)
_ = var('c k t x y z')
```

```python
lhs = sin(x)**2 + cos(x)**2
Eq(lhs, simplify(lhs))
```

```python
p = (x + 1)**2 - 1
p, solve(p, x)
```

```python
lhs = x**2 - 7*x + 12
Eq(lhs, factor(lhs))
```

```python
eqs = [
    Eq(x + y, 4),
    Eq(x - y, 2) ]
(eqs), solve(eqs)
```

```python
A = Matrix([[1, 1], [1, -1]])
b = Matrix([4, 2])
X = MatrixSymbol('x', 2, 1)
X = A.LUsolve(b)
Eq(MatMul(A, X), b)
```

```python
expr = Integral(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
Eq(expr, expr.doit())
```

```python
f = Function('f')
eq = Eq(f(x).diff(x), k * f(x))
eq, dsolve(eq)
```

```python
plot(sin(x)**2, (x, -3, 1))
```

```python
from sympy.plotting import plot3d
plot3d(sin(x*10)*cos(y*5) - x*y)
```