In order to successfully complete this assignment you need to participate both individually and in groups during class on **Monday March 23**.

# In-Class Assignment: Least Squares in Python

Todays in-class assignment includes multiple Least Squares Fit models.  The goal is to see the types of models that can be solved using least squares fit.  Even though this is a Linear Algebra Method the models do not need to be linear.

**As soon as you get to class, download and start working on this notebook**: Try and help your neighbors. the instructor will go over solutions but make sure you try to understand and solve them on your own.

### Agenda for today's class (80 minutes)

1. [(20 minutes) Pre-class Review](#Pre-class_Review)
2. [(20 minutes) LSF Example: Tracking the Planets](#Tracking_the_Planets)
1. [(20 minutes) LSF Example: Predator-Pray Model (revisited)](#Predator-Pray_Model)
2. [(20 minutes) LSF Example: Estimating the best Ellipses](#Estimating_the_best_Ellipses)

---

# 1. Pre-class Review

* [0322--LA-pre-class-assignment](0322--LA-pre-class-assignment.ipynb)

----

# 2. LSF Example: Tracking the Planets

The following table lists the average distance from the sun to each of the first seven planets, using Earth's distance as a unit of measure (AUs).

| Mercury | Venus | Earth | Mars | Jupiter | Satern | Uranus |
|---------|-------|-------|------|---------|--------|--------|
| 0.39    | 0.72  | 1.00  | 1.52 | 5.20    | 9.54   | 19.2   |

The following is a plot of the data:

```python
# Here are some libraries you may need to use

%matplotlib inline
import matplotlib.pylab as plt
import numpy as np
import sympy as sym
import math
sym.init_printing()
```

```python
distances = [0.39, 0.72, 1.00, 1.52, 5.20, 9.54, 19.2]
planets = ['Mercury', 'Venus', 'Earth', 'Mars', 'Jupiter', 'Satern','Uranus']
ind = [1.0,2.0,3.0,4.0,5.0,6.0,7.0]

plt.scatter(ind, distances);
plt.xticks(ind,planets)
plt.ylabel('Distance (AU)')
```

**Note** that the above plot does not look like a line, and so finding the line of best fit is not fruitful.  It does, however look like an exponential curve (maybe a polynomial?). The following step transforms the distances using the numpy **```log```** function and generates a plot that looks much more linear.

```python
log_distances = np.log(distances)

plt.scatter(ind,log_distances)
plt.xticks(ind,planets)
plt.ylabel('Distance (log(AU))')
```

For this question we are going to find the coefficients ($c$) for the best fit line of the form $c_1 + c_2i= \log{d}$, where $i$ is the index of the planet and $d$ is the distance.

The following code constructs this problem in the form $Ax=b$ and define the $A$ matrix and the $b$ matrix as numpy matrices

```python
A = np.matrix(np.vstack((np.ones(len(ind)),ind))).T
b = np.matrix(log_distances).T
sym.Matrix(A)
```

```python
sym.Matrix(b)
```

✅ **DO THIS:** Solve for the best fit of $Ax=b$ and define a new variable $c$ which consists of the of the two coefficients used to define the line $(\log{d} = c_1 + c_2i)$

```python
##Put your answer here:
```

✅ **DO THIS:** Modify the following code (as needed) to plot your best estimates of $c_1$ and $c_2$ against the provided data.

```python
## Modify the following code

est_log_distances = (c[0] + c[1]*np.matrix(ind)).tolist()[0]
plt.plot(ind,est_log_distances)

plt.scatter(ind,log_distances)
plt.xticks(ind,planets)
plt.ylabel('Distance (log(AU))')
```

We can determine the quality of this line fit by calculating the root mean squared error between the estimate and the actual data:

```python
rmse = np.sqrt(((np.array(log_distances)  - np.array(est_log_distances)) ** 2).mean())
rmse
```

Finally, we can also make the plot on the original axis using the inverse of the log (i.e. the exp function):

```python
est_distances = np.exp(est_log_distances)
plt.scatter(ind,distances)
plt.plot(ind,est_distances)
plt.xticks(ind,planets)
plt.ylabel('Distance (AU)')
```

The asteroid belt between Mars and Jupiter is what is left of a planet that broke apart.  Let's the above calculation again but renumber so that  the index of Jupyter is 6, Saturn is 7 and Uranus is 8 as follows:

```python
distances = [0.39, 0.72, 1.00, 1.52, 5.20, 9.54, 19.2]
planets = ['Mercury', 'Venus', 'Earth', 'Mars', 'Jupiter', 'Satern','Uranus']
ind = [1,2,3,4,6,7,8]

log_distances = np.log(distances)
```

✅ **DO THIS:** Repeat the calculations from above with the updated model.  Plot the results and compare the RMSE.

```python
## Copy and Paste code from above
```

```python
## Copy and Paste code from above
est_log_distances = (c[0] + c[1]*np.matrix(ind)).tolist()[0]

est_distances = np.exp(est_log_distances)
plt.scatter(ind,distances)
plt.plot(ind,est_distances)
plt.xticks(ind,planets)
plt.ylabel('Distance (AU)')

rmse = np.sqrt(((np.array(log_distances)  - np.array(est_log_distances)) ** 2).mean())
rmse
## Copy and Paste code from above
```

This model of planet location was used to help discover Neptune and prompted people to look for the "missing planet" in position 5 which resulted in the discovery of the asteroid belt.  Based on the above model, what is the estimated  distance of the asteroid belt and Neptune (index 9) from the sun in AUs? (**Hint:** you can check your answer by searching for the answer on-line).

```python
#Put your prediction calcluation here
```

----

# 3. LSF Example: Predator-Pray Model (revisited)

Let's revisit the predator-pray model from [0318-Pred-Pray_in-class-assignment](0318-Pred-Pray_in-class-assignment.ipynb) and see if we can solve it using LSF?

$$dx = ax + bxy$$
$$dy = cy + dxy$$

```python
# (* The first column is time 't', the second column is coordinate 'x', and the last column is coordinate 'y'. *)
%matplotlib inline
import matplotlib.pylab as plt
import numpy as np

data=[[11,45.79,41.4],
      [12,53.03,38.9],[13,64.05,36.78],
      [14,75.4,36.04],[15,90.36,33.78],
      [16,107.14,35.4],[17,127.79,34.68],
      [18,150.77,36.61], [19,179.65,37.71],
      [20,211.82,41.98],[21,249.91,45.72],
      [22,291.31,53.1],[23,334.95,65.44],
      [24,380.67,83.],[25,420.28,108.74],
      [26,445.56,150.01],[27,447.63,205.61],
      [28,414.04,281.6],[29,347.04,364.56],
      [30,265.33,440.3],[31,187.57,489.68],
      [32,128.,512.95],[33,85.25,510.01],
      [34,57.17,491.06],[35,39.96,462.22],
      [36,29.22,430.15],[37,22.3,396.95],
      [38,16.52,364.87],[39,14.41,333.16],
      [40,11.58,304.97],[41,10.41,277.73],
      [42,10.17,253.16],[43,7.86,229.66],
      [44,9.23,209.53],[45,8.22,190.07],
      [46,8.76,173.58],[47,7.9,156.4],
      [48,8.38,143.05],[49,9.53,130.75],
      [50,9.33,117.49],[51,9.72,108.16],
      [52,10.55,98.08],[53,13.05,88.91],
      [54,13.58,82.28],[55,16.31,75.42],
      [56,17.75,69.58],[57,20.11,62.58],
      [58,23.98,59.22],[59,28.51,54.91],
      [60,31.61,49.79],[61,37.13,45.94],
      [62,45.06,43.41],[63,53.4,41.3],
      [64,62.39,40.28],[65,72.89,37.71],
      [66,86.92,36.58],[67,103.32,36.98],
      [68,121.7,36.65],[69,144.86,37.87],
      [70,171.92,39.63],[71,202.51,42.97],
      [72,237.69,46.95],[73,276.77,54.93],
      [74,319.76,64.61],[75,362.05,81.28],
      [76,400.11,105.5],[77,427.79,143.03],
      [78,434.56,192.45],[79,410.31,260.84],
      [80,354.18,339.39],[81,278.49,413.79],
      [82,203.72,466.94],[83,141.06,494.72],
      [84,95.08,499.37],[85,66.76,484.58],
      [86,45.41,460.63],[87,33.13,429.79],
      [88,25.89,398.77],[89,20.51,366.49],
      [90,17.11,336.56],[91,12.69,306.39],
      [92,11.76,279.53],[93,11.22,254.95],
      [94,10.29,233.5],[95,8.82,212.74],
      [96,9.51,193.61],[97,8.69,175.01],
      [98,9.53,160.59],[99,8.68,146.12],[100,10.82,131.85]]
data = np.array(data)
t = data[:,0]
x = data[:,1]
y = data[:,2]
plt.scatter(t,x)
plt.scatter(t,y)
plt.legend(('prey', 'preditor'))
plt.xlabel('Time')
plt.title('Population');
```

✅ **DO THIS** Use Numerical Differentiation to calculate $dx$ and $dy$ from $x$ and $y$. Discuss with people at your table how to write efficient code.  See if you can plot $x,dx$ nad $y,dy$ on a couple of plots. Use the plots to try and check to make sure your results make senes.

```python
# Put your answer here
```

✅ **DO THIS**  Formulate two linear systems ($Ax=b$) and solve them using LSF as we did in the pre-class. Use one to solve the first ODE and the second to solve the second ODE.  Remember, we are trying to estimate values for $a,b,c,d$

```python
#Put your answer here.
```

Assuming everything worked the following should plot the result.

```python
from scipy.integrate import odeint
# The above ODE model sutiable for ODEINT
def deriv(position,t,a,b,c,d):
    x = position[0]
    y = position[1]
    dx = a*x - b*x*y
    dy = -c*y + d*x*y
    return (dx,dy)

# Integrate equations over the time grid, t.
ret = odeint(deriv, (data[0,1],data[0,2]), t, args=(a,b,c,d))

#Plot the model on the data
plt.plot(t,ret)
plt.scatter(t, data[:,1])
plt.scatter(t, data[:,2]);
plt.legend(('x est', 'y est', 'x', 'y'))
plt.xlabel('Time');
```

✅ **QUESTION** Compare these results with those from the prior class.  Are they different? If so which is better and why? Discuss ideas with your group and be prepared to share your ideas with the class.

Put your answers to the above questions here.

----

# 4. Example: Estimating the best Ellipses

```python
%matplotlib inline
import matplotlib.pylab as plt
import numpy as np
import sympy as sym
sym.init_printing(use_unicode=True)
```

Now consider the following sets of points. Think of these as observations of planet moving in an elliptical orbit.

```python
x=[0, 1.0, 1.1, -1.1, -1.2, 1.3]
y =[2*1.5, 2*1.0, 2*-0.99, 2*-1.02, 2*1.2, 2*0]

plt.scatter(x,y)
plt.axis('equal')
```

In this problem we want to try to fit an ellipse to the above data.  First lets look at a general equation for an ellipse:

$$ \frac{(u+x)^2}{a^2} + \frac{(v+y)^2}{b^2} = 1 \qquad \qquad \text{    (1)}$$

Where $u$ and $v$ are the $x$ and $y$ coordinates for the center of the ellipse and $a$ and $b$ are the lengths of the axes sizes of the ellipse.  A quick search on how to plot an ellipse in python comes up with the following example:

```python
# Code from: https://stackoverflow.com/questions/10952060/plot-ellipse-with-matplotlib-pyplot-python

u=1.     #x-position of the center
v=0.5    #y-position of the center
a=2.     #radius on the x-axis
b=1.5    #radius on the y-axis

t = np.linspace(0, 2*np.pi, 100)
plt.plot( u+a*np.cos(t) , v+b*np.sin(t) )
plt.grid(color='lightgray',linestyle='--')
plt.show()
```

Notice this example uses equations of the form:

$$t = [0, \dots, 2\pi]$$
$$x = u+a\cos(t)$$
$$y = v+b\sin(t)$$

Turns out that this form of the equation is easier to plot and the variables $u,v,a,b$ are the same as our original equation.

Now lets expand the original equation (equation 1 from above) and we get the following:

$$x^2−2ux-u^2+y^2−2vy+v^2=r^2 \qquad \qquad \text{    (2)}$$

✅ **QUESTION:** Why can't we convert equation 2 into the form $Ax=b$ and solve using Least Squares Fit? Discuss with your group and be prepared to share your thought with the class.

Put your answer to the above question here.

If we look at our data more closely we can simplify equations 1 and 2 by assuming the the centroid (u,v) is at the origin.  This assumption results in the following equation:

$$ \frac{x^2}{a^2} + \frac{y^2}{b^2} = 1 $$

Notice we can rewrite this into a standard linear set of equations by defining $c_o = \frac{1}{a^2}$ and $c_1 = \frac{1}{b^2}$ and rewriting the equation as follows:

$$ c_0x^2 + c_1y^2 = 1 $$

✅ **DO THIS** Given that we know the $x$ and $y$ values of our point observations, equation 4 is now linear and can be solved using Least Squares Fit.  Using the observation points from above construct  ```A``` and ```b``` as numpy matrixes for the overdefined system $Ax=b$

Put your answer to the above question here.

```python
sym.Matrix(A)
```

```python
sym.Matrix(b)
```

✅ **DO THIS:** Solve the above over defined system of linear equations for $c_0$ and $c_1$ using LSF.

```python
# Put your answer to the above question here
```

Assuming we have $c$ in the correct format, we can now calculate $a$ and $b$ from the solution for $c_o$ and $c_1$ calculated in the previous step and plot using our plotting code:

```python
c = 1/np.sqrt(np.abs(c))
b=c[1,0]
a=c[0,0]
print(a,b)
```

```python
u=0     #x-position of the center
v=0     #y-position of the center

t = np.linspace(0, 2*np.pi, 100)
plt.plot(u+a*np.cos(t) , v+b*np.sin(t) )
plt.scatter(x,y)
plt.grid(color='lightgray',linestyle='--')
plt.axis('equal');
```

-----
### Congratulations, we're done!

### Course Resources:

- [Syllabus](https://docs.google.com/document/d/e/2PACX-1vTW4OzeUNhsuG_zvh06MT4r1tguxLFXGFCiMVN49XJJRYfekb7E6LyfGLP5tyLcHqcUNJjH2Vk-Isd8/pub)
- [Preliminary Schedule](https://docs.google.com/spreadsheets/d/e/2PACX-1vRsQcyH1nlbSD4x7zvHWAbAcLrGWRo_RqeFyt2loQPgt3MxirrI5ADVFW9IoeLGSBSu_Uo6e8BE4IQc/pubhtml?gid=2142090757&single=true)
- [D2L Page](https://d2l.msu.edu/d2l/home/912152)
- [Git Repository](https://gitlab.msu.edu/colbrydi/cmse802-s20)

© Copyright 2020,  Michigan State University Board of Trustees


```python
import sympy as sp
import numpy as np
from scipy.optimize import fsolve, fmin, minimize_scalar, curve_fit
import matplotlib.pyplot as mpl
```

```python
import pathlib
if not pathlib.Path("mpl_utils.py").exists():
  !curl -O https://raw.githubusercontent.com/joaochenriques/MCTE_2022/main/libs/mpl_utils.py &> /dev/null

import mpl_utils as mut
mut.config_plots()

%config InlineBackend.figure_formats = ['svg']
```

# **Warning**

Before running this notebook locus of optimal values of $C_P$ as function of $C_T$ and $\mathrm{Fr}_1$ in the notebook:

https://github.com/joaochenriques/MCTE_2022/blob/main/ChannelFlows/DiskActuator/SensitivityAnalysis_V02.ipynb

```python
def compute_C_T_and_C_P( Fr4b, Fr1, B ):

    # These Eqs. are described in the course Lecture Notes

    ζ4 = (1/2.)*Fr1**2 - 1/2.*Fr4b**2 + 1.0

    Fr4t = (Fr1 - Fr4b*ζ4 + np.sqrt(B**2*Fr4b**2 - 2*B*Fr1**2 + 2*B*Fr1*Fr4b \
            + B*ζ4**2 - B + Fr1**2 - 2*Fr1*Fr4b*ζ4 + Fr4b**2*ζ4**2))/B

    ζ4b =  (Fr1 - Fr4t*ζ4)/(Fr4b - Fr4t)
    ζ4t = -(Fr1 - Fr4b*ζ4)/(Fr4b - Fr4t)

    Fr2t = Fr4t*ζ4t/B

    C_T = (Fr4b**2 - Fr4t**2)/Fr1**2
    C_P = C_T*Fr2t/Fr1

    return C_T, C_P

def find_minus_C_P( Fr4b, Fr1, B ):
    # function created to discard the C_T when calling "compute_C_T_and_C_P"
    C_T, C_P = compute_C_T_and_C_P( Fr4b, Fr1, B )
    return -C_P # Minus C_P to allow minimization
```

```python
# Blockage factor
B = 0.1

# define Fr1 interval and number of points
Fr1_min = 1E-3
Fr1_max = 0.4
Fr1_num = 30

Fr1_opt_vec = np.linspace( Fr1_min, Fr1_max, Fr1_num )

C_P_opt_vec = np.zeros( Fr1_num )
C_T_opt_vec = np.zeros( Fr1_num )

for i, Fr1 in enumerate( Fr1_opt_vec ):

  res = minimize_scalar( find_minus_C_P, args=(Fr1, B), bounds=[0,1], method='bounded',
                         options={ 'xatol': 1e-08, 'maxiter': 500, 'disp': 1 } )
  Fr4b = res.x # optimal value

  C_T, C_P = compute_C_T_and_C_P( Fr4b, Fr1, B )

  C_T_opt_vec[i] = C_T
  C_P_opt_vec[i] = C_P

fig, (ax1, ax2) = mpl.subplots(1,2, figsize=(12, 4.5) )
fig.subplots_adjust( wspace = 0.19 )

ax1.plot( Fr1_opt_vec, C_P_opt_vec, 'o-' )
ax1.set_title( "B = %.2f" % B )
ax1.set_xlabel("$\mathrm{Fr}_1$")
ax1.set_ylabel("$C_P^\mathrm{opt}$")
ax1.text(-0.17, 1.05, 'a)', transform=ax1.transAxes, size=16, weight='semibold')
ax1.grid()

ax2.plot( Fr1_opt_vec, C_T_opt_vec, 'ro-' )
ax2.set_title( "B = %.2f" % B )
ax2.set_xlabel("$\mathrm{Fr}_1$")
ax2.set_ylabel("$C_T^\mathrm{opt}$")
ax2.text(-0.17, 1.05, 'b)', transform=ax2.transAxes, size=16, weight='semibold');
ax2.grid()

mpl.savefig('CP_CT_optimal_B%4.2f.pdf' % B, bbox_inches='tight', pad_inches=0.02);
```

## **Polynomial fitting**

Fit a polynomial of the type

$$ a x^6 + b x^4 + c x^2 + d$$

to the optimal $C_T$ and $C_P$. This polynomial has only even monomial to avoid double curvature.

```python
def fitting_func( x, a, b, c, d ):
    x2 = x*x
    return ( ( ( a * x2 + b ) * x2 + c ) * x2 ) + d

C_P_popt, C_P_pcov = curve_fit( fitting_func, Fr1_opt_vec, C_P_opt_vec )
C_T_popt, C_P_pcov = curve_fit( fitting_func, Fr1_opt_vec, C_T_opt_vec )
```

## **Optimal $C_T$ and $C_P$**

```python
sFr_1, sC_P, sC_T = sp.symbols( "\mathrm{Fr}_1, C_\mathrm{P}, C_\mathrm{T}" )
eqCP = sp.Eq( sC_P, fitting_func( sFr_1, *C_P_popt ) )
sp.expand( eqCP )
```

$\displaystyle C_\mathrm{P} = 0.950469005167641 \mathrm{Fr}_1^{6} + 0.235239872607705 \mathrm{Fr}_1^{4} + 0.189205872691749 \mathrm{Fr}_1^{2} + 0.731588254027737$

```python
eqCT = sp.Eq( sC_T, fitting_func( sFr_1, *C_T_popt ) )
sp.expand( eqCT )
```

$\displaystyle C_\mathrm{T} = 3.47964521886281 \mathrm{Fr}_1^{6} + 0.595218942569153 \mathrm{Fr}_1^{4} + 0.474817875529668 \mathrm{Fr}_1^{2} + 1.20709928871966$

## **Plot results of the optimization and interpolation of the maxima**

```python
fit_C_P = fitting_func( Fr1_opt_vec, *C_P_popt )
fit_C_T = fitting_func( Fr1_opt_vec, *C_T_popt )
```

```python
mpl.plot( Fr1_opt_vec, fit_C_P, label="$(ax^6+bx^4+cx^2+d)$ fitting" )
mpl.plot( Fr1_opt_vec, C_P_opt_vec, 'ro', label="Optimal values" )

mpl.title( "B = %.2f" % B )
mpl.xlabel("$\mathrm{Fr}_1$")
mpl.ylabel("$C_P^\mathrm{opt}$")
mpl.legend(loc='upper left')
mpl.grid()
```

```python
mpl.plot( Fr1_opt_vec, fit_C_T, label="$(ax^6+bx^4+cx^2+d)$ fitting" )
mpl.plot( Fr1_opt_vec, C_T_opt_vec, 'ro', label="Optimal values" )

mpl.title( "B = %.2f" % B )
mpl.xlabel("$\mathrm{Fr}_1$")
mpl.ylabel("$C_T^\mathrm{opt}$")
mpl.legend(loc='upper left')
mpl.grid()
```

```python

```


# BAYESIAN INFERENCE USING MARKOV CHAIN MONTE CARLO
##      -- ESTIMATING MARKETING ROI IN A REVOLUTIONAL WAY

**MCMC**, short for **Markov Chain Monte Carlo** is a particularly useful tool for solving sampling problems
* from a complex probability density function;
* in high dimensional spaces.

You might wonder, what's it to do with Bayesian inference? Isn't MCMC just a sampling tool?

In real life cases, Bayesian inferences have very **complicated posterior distribution** together with **high-dimensions**. In this sense, the two traditional ways - either mathematical deduction or numerical integral- to integrate posterior are not favorable. So how should we get the posterior? [Ref 7]

The answer is - maybe we just forget about the integral. What we can do here, we **generate enough samples from the posterior distribution**. With these samples, we will be able to approximate any important values we want to know about the posterior distribution. That's where MCMC comes into place - because it's working for very complex posterior distribution!

This tutorial assumes no prior knowledge for Markov Chain Monte Carlo and will introduces it in a lot details. It will also go into some practical perspectives and talks about an application case in marketing RoI in general.

Enjoy reading :)

### PART I: INTRODUCTION TO MARKOV CHAIN
The firststep to understand MCMC is to know three important concepts of Markov Chain:
* **Markov Propoerty:** "Probability of moving to next state depends only on the present state and not on the previous states."
  $$\Pr(X_{n+1}=x\mid X_{1}=x_{1},X_{2}=x_{2},\ldots ,X_{n}=x_{n})=\Pr(X_{n+1}=x\mid X_{n}=x_{n})$$
* **Stationary Distribution:** With transition matrix P, when transitions occurs infinite times, the state of X will reach a stationary distribution, **regardless of its initial state**.
 $$x_{n + 1} = x_n \times P $$

 $${\displaystyle \lim _{n\rightarrow \infty }x_1 \times P^n= {C}} $$

* **Detailed Balance Condition**:  Each transition matrix correspond to one stationary distribution that, for all i, j, it satisfies:
 $$X(i)Pij= X(j)Pji $$

Now let's follow a small story about Gary's and John's family below to get an understanding.

#### 1.1 Gary's and John's Story

##### Markov Property, Transition Matrix
There are three classes of people in this society, upper class, middle class and lower class. From generations to generations, people may squander the fortune and drop to lower classes; people may also try very hard to accumulate back fortune and reputations and go back to upper classes. The probabilities for all the **transition matrix** is shown as below:

\begin{align}
                          & \quad Child   &\quad Child   &\quad Child\\
                              & \quad  Upper Class &\quad Middle Class &\quad Lower Class \\
Parent: Upper Class	& \quad    0.65 & \quad  0.32 & \quad 0.03\\
Parent: Middle Class & \quad    0.1  & \quad  0.8  & \quad  0.1\\
Parent: Lower Class  & \quad    0.01 & \quad 0.08  & \quad 0.91 \\
\end{align}

Note that
* Proportions of this generation's social classes is dependent of its parent generation only, nothing to do with grandpa's generation or even earlier;

```python
import numpy as np
transition_p = np.array([[0.65, 0.32, 0.03], [0.10, 0.8, 0.10], [0.01, 0.08, 0.91]])
print transition_p
```

    [[ 0.65  0.32  0.03]
     [ 0.1   0.8   0.1 ]
     [ 0.01  0.08  0.91]]

##### Stationary Distribution
Now let's do a little experiment. Suppose from there are two families, Gary's and John's, with proportions denoted in the code:

**In Gary's house,**
* 55% of family members belong to upper class;
* 40% belong to middle class;
* 5% belong to lower class;

**In John's house,**
* 5% of family members belong to upper class;
* 80% of family members belong to middle class;
* 15% of family members belong to lower class;

Guess what will happen after 200 generations?

```python
# set initial status for Gary's and John's
gary = np.array([[0.55, 0.4, 0.05]])
john = np.array([[0.05, 0.8, 0.15]])

#iterate 200 generations - update status
for i in range(200):
    gary = np.dot(gary, transition_p);
    john = np.dot(john, transition_p);

#print results
print "Gray's - 200th generation", gary
print "John's - 200th generation", john
#test if are identical
print "Identical?", np.array_equal(gary, john)
```

    Gray's - 200th generation [[ 0.12626263  0.39393939  0.47979798]]
    John's - 200th generation [[ 0.12626263  0.39393939  0.47979798]]
    Identical? True

It's easy to see that, regardless of initial difference, Gary's and John's family have the same probability components from upper class, middle class and lower class at 200th generation. This is not a coincidence, but it demonstrates the idea of **stationary distribution**, that when the iteration of transitions approaches infinity (in our case 200 is a large enough figure), the proportion of the 200th generation becomes the same regardless of differences in initial status.What's more, the proportions will have **no furthur change** if we does one more iteration of transition again. Let's evolve another 100 generations and look at the plot:

```python
import matplotlib.pyplot as plt

gary_generations = gary.copy()
for i in range(100):
    gary = np.dot(gary, transition_p);
    gary_generations = np.vstack( (gary_generations, gary)  )
plt.plot(gary_generations[range(100), 0 ], label = "Upper class")
plt.plot(gary_generations[range(100), 1 ], label = "Middle class")
plt.plot(gary_generations[range(100), 2 ], label = "Lower class")
plt.xlabel('Generations') ; plt.ylabel('Proportion of Classes')
plt.title("I told you that's stationary!")
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.);
plt.show()
```

##### Detailed Balance Condition

Why the proportion doesn't change any more? Because now it satisfies detailed balance condition, that every probability "goes out" equals the probabilities that "comes back". Let's look at cases for upper class.

```python
p_upper_goingout = gary[0,0]* (transition_p[0,1] + transition_p[0,2])
print "Proportions going out from upper class: ", p_upper_goingout

p_upper_comingback = gary[0,1]*transition_p[1,0] + gary[0,2]*transition_p[2,0]
print "Proportions coming back to upper class: ",p_upper_comingback
```

    Proportions going out from upper class:  0.0441919191919
    Proportions coming back to upper class:  0.0441919191919

#### 1.2 Markov Trick for Bayesian Inference

In the introduction section, we mention generating samples from the posterior distribution using MCMC. How?

##### Motivation
If we have our posterior as stationary distribution of a Markov Chain, then we can generate as many samples as we want from the posterior distribution. (If you want to know more about sampling from Markov Chain, please visit  here.) This is key idea of MCMC algorithm!

###### Posterior Distribution as Stationary Distribution
Suppose we have:
$$ h(θ, X)$$
where
$$ Parameters_ {unknown}:θ = {θ1,...,θk}  $$
$$ variables_ {known}: X = {X1,...,Xn}  $$

Writing it in a Bayesian fashion,

$$ p(\mathbf{θ}|\mathbf{Y, X}) \propto p(\mathbf{Y}|\mathbf{θ, X}) \times p(θ) $$

Here, p(θ|Y,X) is our function of interest. Let's make it our stationary distribution.

##### Evolving Regardless of Initial State
Note that, once we have the correct transition matrix for the posterior distribution, regardless of what value of θ we start with, we will always guarantee that - samples we generate belong to the posterior distribution once we iterate enough times and reach stationary distribution. Then how many time should we iterate? We don't know! But most likely, iterating 10000 is enough; and we then could simply abondom the first thousands of samples that are generated before we reach stationary distribution.

However, there is one question remains, what should the transition matrix be?

### PART II: MARKOV CHAIN MONTE CARLO

MCMC is to perform monte carlo for a Markov Chain whose stationary distribution is our posterior integral in our Bayesian inference. In the previous section we have a glance at how Markov Chain relates to posterior distribution of interest; now we will look at how to construct transition matrix.

#### 2.1 Intuition from Reject Sampling

Look at the distribution below. How can we sample from such a complex distribution (green line)?

We can have a "wrapper" probability density function outside our target probability function, and generate samples from the wrapper function. Then we will use some tricks to determine whether we want this sample or not.

```python
import scipy.stats as stat

# Plot between -10 and 10 with .001 steps.
x_axis = np.arange(-15, 15, 0.001)
target_func = stat.norm.pdf(x_axis,3,1) + stat.norm.pdf(x_axis, -5, 2)
plt.plot(x_axis, target_func, color = 'green', label = 'target')

# wrapper function
wrapper_func = stat.norm.pdf(x_axis,0,4)
c =  np.max(target_func/wrapper_func)

#PLOT SCALED PROPOSAL/ENVELOP DISTRIBUTION
plt.plot(x_axis,c*wrapper_func, color = 'pink', label = 'wrapper');
plt.legend()
plt.show()
```

Steps for Reject Sampling are as follows:

* Sample from wrapper function, obtain X_i
* Calculate acceptance probability
$$α=\frac {p(x_i)} {Cq(x_i)}$$
* Compare acceptance probability with a random number between Uniform [0,1], say, u
* Accept X_i as a sample if α>= u, else reject X_i

Let's look at the below code to strengthen our understanding:

```python
#probability of getting X at the target function;
def target_p(x):
    p = stat.norm.pdf(x,3,1) + stat.norm.pdf(x, -5, 2);
    return p;

#perform reject_sampling to get samples at size sampling_nums
def reject_sampling( sampling_nums):
    samples = []
    sample_num = 0;
    while(sample_num < sampling_nums):
        #sampling from wrapper function
        X_i = np.random.normal(0,4,1)[0];
        #calculate acceptance probability
            # q(X_i) is the probability of X_i given wrapper function; where loc = mean, scale = sigma (which isstdev, sqrt (4))
        a = target_p(X_i)/ (c*(stat.norm.pdf(X_i, loc=0, scale=2)));
        #compare with random value from Uniform
        u = np.random.uniform(0,1,1);
        # if acceptance rate bigger than a random probability value, we accept X_i; vice versa
        if a>= u:
            samples.append(X_i);
            sample_num += 1;
        else:
            pass;
    return samples;

samples = reject_sampling(10000);

#plot the shistogram
plt.hist(samples, bins = 500, color = 'gray');
plt.show();
```

#### 2.2 A Transition Matrix with Acceptance Probability

Now let's assume we don't know the two means for the target function (we do know standard deviation), what can we do?

$$ θ= mean1, mean2 $$
$$ p(\mathbf{θ}|\mathbf{samples}) \propto p(\mathbf{samples}|\mathbf{θ}) \times p(θ) $$

In continuous cases, transition matrix is just how we update our target distribution, which is θ in our case.
Remember, we need to satisfy the detailed balance condition, where
$$p(θ_{proposed}|samples) P(θ_{original}|θ_{proposed})= p(θ_{original}|samples)P(θ_{proposed}|θ_{original}) $$
But most likely, this equation doesn't hold.

What we can do? We multiply the right half with acceptance probability a, where
\begin{align}
a & = \frac {equation_{left}} {equation_{right}}\\
  & = \frac {P(θ_{proposed}|samples) P(θ_{original}|θ_{proposed})}{P(θ_{original}|samples)P(θ_{proposed}|θ_{original})}
\end{align}

Substitude posterior P(θ|samples) with its likelihood*prior, we then have:
\begin{align}
a  & = \frac {P(samples|θ_{proposed}) P(θ_{proposed})P(θ_{original}|θ_{proposed})}
{P(samples|θ_{original})P(θ_{original})P(θ_{proposed}|θ_{original})} \\
  & = \frac {P(samples|θ_{proposed}) P(θ_{original},θ_{proposed})}{P(samples|θ_{original})P(θ_{proposed},θ_{original})} \\
  & = \frac {P(samples|θ_{proposed}) }{P(samples|θ_{original})} \\
\end{align}

So the idea for the transition matrix for the posterior distribution of interest is pretty simple: we propose a new θ as a random walk N(0,1) from the old θ; and then we decide to accept the new θ at probability a if a < 1; will just accept new θ if a >= 1.

```python
##Learning purpose only. In last section we will have reproducible code

# log probability of getting X at the target function with mean1 and mean2;
# note: using log is only to avoid producing 0 denominator and numerator because sample no. is large
def target_p(mean1, mean2):
    p = np.sum(  np.log(stat.norm.pdf(samples,mean1,1) + stat.norm.pdf(samples,mean2, 2)));
    return p;

#perform mcmc to generate samples posterior distributions
# iterations means total iterations, burning means the number of iterations we will abandon in the end
def mcmc(iterations, burning):
    mean1 = 3;  #initial mean1
    mean2 = -3;  #initial mean2
    mean1_list = []; #to store mean1 samples
    mean2_list = []; #to store mean2 samples
    sample_num = 0; #valid samples so far
    p_samples_theta = target_p(mean1, mean2);#P(samples|θoriginal)

    while(sample_num < iterations):
        #random work: serve as transition matrix
        random_work = np.random.normal(0,3,2);  #we can make the step bigger so I use N(0,3)
        #proposed thetas
        mean1_proposed = mean1 + random_work[0] ;
        mean2_proposed = mean2 + random_work[1] ;
        #P(samples|θ_proposed)
        p_samples_theta_proposed = target_p(mean1_proposed, mean2_proposed);
        #calculate log acceptance probability
        a = p_samples_theta_proposed - p_samples_theta;
        #compare with random value from Uniform
        log_u = np.log(np.random.uniform(0,1,1));
        # if acceptance rate bigger than a random probability value, we accept X_i; vice versa
        if a>= log_u:
            mean1_list.append(mean1_proposed);
            mean2_list.append(mean2_proposed);
            sample_num += 1;
        else:
            pass;
    mean1_approximate = np.mean(mean1_list[burning:]);
    mean2_approximate = np.mean(mean2_list[burning:]);
    return [mean1_approximate, mean2_approximate];

mean1_approximate, mean2_approximate = mcmc(10000,3000);
print "true mean1: 3", "   estimation: ", mean1_approximate;
print "true mean2: -5",  "  estimation: ", mean2_approximate
```

    true mean1: 3    estimation:  3.68754057967
    true mean2: -5   estimation:  -4.80500118953

### PART III: CALCULATING RETURN ON INVESTMENT

Companies care about one thing: how much sales could be boosted if we spend this amount of money in advertisment?

This question is answered by RoI figure:
\begin{equation*}
\frac{Profits - Investment}{Investment}
\end{equation*}

Let's look at how we solve RoI for a juice company.

#### 3.1 Backgrounds
JuicyJuice spent 10,000 dollars in their Pittsburgh market while they spent $0 in their Austin market on April 2016.

Sales follow certain degree of seasonal changes as in summer people prefer to drink juices while in winter they don't. Pittsburgh has a larger market and sales are also higher than that in Austin. Mainly, the sales in Pittsburgh are higher than Austin by around \$2000 or \$3000, but it also happens that the differences go up to \$4000 or go down to $1000 due to competitors' strategy or sports competitions.

```python
import numpy as np
from datetime import datetime as dt

np.random.seed(1349)

investment = 10000/1000

# sales of austin (in $1000)
austin = np.array([ 9.4793,  10.1584,   11.1288,   11.3445,  10.7590,  9.0659,  8.6782,  8.3428,   8.3733,    8.4991,  9.5095,   9.5457,    9.4601,    9.6153,   10.3252, 9.3139,  9.7216,   9.7592,  9.3066,   9.2432,  9.1111,   9.3399,    9.1782,  9.7632,  9.7600, 9.9181,  9.8263,   9.5090,   8.8818,   8.9620])

#sales of pittsburgh (in $1000):
pittsburgh = np.hstack(  (austin[:10] +   np.random.normal(2,1, 10), austin[10:20] + np.random.normal(3, 1, 10), austin[20:24]  +   np.random.normal(2,1, 4),
                          austin[24:28] +  np.random.normal(2, 1, 4) + np.random.normal(5,1, 4), austin[28:] +   np.random.normal(3, 1, 2) + np.random.normal(5,1, 2)  ) )

#prepare x-axis labels denoting months
sd = dt.strptime('2014-5-10', "%Y-%m-%d") ; ed = dt.strptime('2016-10-10', "%Y-%m-%d")
month = [dt.strptime('%2.2d-%2.2d' % (y, m), '%Y-%m').strftime('%b-%y') for y in xrange(sd.year, ed.year+1)
       for m in xrange(sd.month if y==sd.year else 1, ed.month+1 if y == ed.year else 13)]

plt.xticks(range(len(month)), month)
locs, labels = plt.xticks()  #makes the months (string) corresponds to int axis
plt.setp(labels, rotation=90);  #rotate x axis strings to increase readability

#plot Austin and Pittsburgh sales correspondingly
plt.plot(range(len(month)), austin, label = "Austin", color = 'navy')
plt.plot(range(len(month)), pittsburgh, label = "Pittsburgh", color = 'orange')

#plot advertisement time point as a vertical line
plt.axvline(x=23,color='k',ls='dashed');

#labels and legend
plt.ylabel('Sales $000') ;
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.);
plt.show()
```

#### 3.2 Traditionally Difference-in-Difference Approach

Traditionally, we use diff-in-diff approach to measure RoI. We compare the sales of control market Austin and test market Pittsburgh and calculate:

\begin{align}
              & \quad Pittsburgh - Austin\\
Before-Ad-Period	& \quad        a\\
Post- Ad-Period    & \quad        b\\
Diff-in-Diff      & \quad      b-a  \\
\end{align}

```python
#calculates sales differences of Austin and Pittsburgh before April 2016
diff_before_ad = np.sum(pittsburgh[:-7] - austin[:-7])

#calculates sales differences of Austin and Pittsburgh after April 2016
diff_after_ad  = np.sum(pittsburgh[-7:] - austin[-7:])

#estimate a "pure" amount of sales attribution that advertisement brings to JuicyJuice Pittsburgh market
attained_profits = diff_after_ad - diff_before_ad*7/len(austin)
print "RoI by Diff-in-Diff Approach",  round((attained_profits - investment)/ investment, 2)
```

    RoI by Diff-in-Diff Approach 2.18

However, this approach assumes that a nature of static regression, and requires the sales differences of Austin and Pittsburgh before and after the ad period are **i.i.d**, i.e. independent and identically distributed from one distribution. However, after posting ads most likely the distribution of sales differences change, that means, the assumption for Diff-in-Diff doesn't hold any more. Therefore, the MCMC approach gives a much better solution.

#### 3.3 Bayesian Inference using MCMC

Let's note four facts from our JuicyJuice story that,
* (1) Pittsburgh's market has certain degree of linear correlation, and is always larger than Austin's.
* (2) Sometimes its competitor will go on some promotions, which results a fall of JuicyJuice's Pittsburgh sales.
* (3) In the case 2, sales difference for the cities are mainly around \$2000, otherwise its's around \$3000.
* (4) Fluctuations for sales for Austin and Pittsburgh are alike, at most, it will \$1000.

(Note, though that's how I make the data so I know it :p
 but they are also easy to see if we calculate the mean, median and standard deviations of the two series).

Let's decompose our marketing understanding to some formulars that describe the two markets:

\begin{align}
(1) \quad  & Sales_{Pittsburgh, t}  = b0_t + b1\times Sales_{Austin, t}\\
(2)\quad  & p_t  ∝  Binomial(1, p)\\
(3)\quad  & b0_t  ∝  Gaussian (mean_1, std dev^2) \quad if \quad p_t = 1\\
(4)\quad  & b0_t  ∝  Gaussian (mean_2, std dev^2) \quad if \quad p_t = 0\\
\end{align}

###### Bayesian Expressions
Writing it in a Bayesian fashion,

$$ p(\mathbf{θ}|\mathbf{Sales_{Pittsburgh}, Sales_{Austin}}) \propto p(\mathbf{Sales_{Pittsburgh}}|\mathbf{θ, Sales_{Austin}}) \times prior(θ) $$

where θ denotes a collection of the parameters,
$$ θ = [b1, p, mean_1, mean_2, std dev^2] $$

Let's now build model using PyMC

```python
from pymc import deterministic,Normal, Uniform, Bernoulli, graph, MCMC, MAP

class mcmc_model:
    data = pittsburgh[:24]
    #2nd equation - binormial distribution with 0<=p<=1;
    # As we have no preference over what value should we choose about p, so we use a uniform distribution
    p = Uniform("p", lower=0, upper=1)
    bern = Bernoulli("bern", p=p, size=len(data))

    #Gaussian distribution in 3rd and 4th equation - Gaussian distribution;
    # both mean are bounded by the min and max difference of the Pittsburgh sales and Austin sales
    # standard deviation is again bounded by the difference
    diff = pittsburgh - austin
    mean1 = Uniform('mean1', lower=min(diff), upper=max(diff))
    mean2 = Uniform('mean2', lower=min(diff), upper=max(diff))
    std_dev = Uniform('std_dev', lower=0, upper= np.sqrt(max(diff) - min(diff)))

    #in order to formulate normal distribution in PyMC, we need its precision, which is 1/variance rather than stdev
    @deterministic(plot=False)
    def precision(std_dev=std_dev):
        return 1.0 / (std_dev * std_dev)

    b0_1  = Normal('b0_1', mu = mean1, tau=precision)
    b0_2 = Normal('b0_2', mu = mean2, tau = precision)

    # "model" the observed x values as a normal random variable
    # because x is observed, it doesn't actually matter how we choose to model x -- PyMC isn't going to change x's values
    X_austin = Normal('X_austin', mu = 0, tau = 1, value=austin[:24], observed=True)

    #As we have no preference for b1 value so we again choose a uniform value.
    #As we could observe from graph that sales of Pittsburgh always large than that of Austin then we choose lower bound 1
    b1 = Uniform('b1', lower=1, upper=max(data)/min(X_austin))

    ## 1st equation
    # if p = 1, then we will go for the random variable from the first gaussian distribution with mean_1
    # if p = 0, then we will go for the random variable from the second gaussian distribution with mean_2
    # the slope of the regression component is b1 and the observed variable is X_austin
    @deterministic(plot = False)
    def pred(bern = bern, b0_1 = b0_1, b0_2=b0_2, b1 = b1, x=X_austin):
        return  bern *b0_1 + (1 - bern) * b0_2 + b1*x

    # "model" the observed y values as a normal random variable
    # its mean is pred  and value is the data (pittburgh[:24])
    y = Normal('y', mu = pred, tau=precision, value=data, observed=True)

#formulate mcmc using the model
model = MCMC(mcmc_model)
#sample 10000 times and burn the first 8000
model.sample(iter=10000, burn=3000)
# create a MAP
model_map = MAP(model)
model_map.fit()

#put the coefficients together
coef = {}
for variable in model_map.variables:
    coef[str(variable)]= variable.value
#pymc.Matplot.plot(model)
```

     [-----------------100%-----------------] 10000 of 10000 complete in 11.2 secWarning: Stochastic bern's value is neither numerical nor array with floating-point dtype. Recommend fitting method fmin (default).

    D:\Anaconda3\envs\py27\lib\site-packages\scipy\optimize\optimize.py:1881: RuntimeWarning: invalid value encountered in absolute
      tmp2 = numpy.abs(tmp2)

```python
# prediction using the data
def prediction(data):
    binom = np.random.binomial(n = 1, p = coef['p'], size = len(data)) # equation 4
    stdev = np.sqrt (1/ coef['precision']) #equation 2&3
    mean1_rand = np.random.normal(coef['mean1'], stdev, size = len(data)) #equation 2
    mean2_rand = np.random.normal(coef['mean2'], stdev, size = len(data)) #euqation 3
    b1 = coef['b1']; # equation 1
    result = b1*data + binom*mean1_rand + (1-binom)*mean2_rand #equation 1
    return result

#putting the predictions for time preriod before the advertisement and predictions after the advertisement together
pittsburgh_pred = np.hstack(   (coef['pred'], prediction(austin[24:])  ))

#calculate ROI
RoI_mcmc = ( np.sum(pittsburgh[24:] - pittsburgh_pred[24:]) - investment)/investment

#plotting
plt.xticks(range(len(month)), month); locs, labels = plt.xticks();  #makes the months (string) corresponds to int axis
plt.setp(labels, rotation=90); #rotate x axis strings to increase readability
plt.plot(range(len(month)), austin, label = "Austin", color = 'navy'), plt.plot(range(len(month)), pittsburgh, label = "Pittsburgh", color = 'orange') #plot sales
plt.plot(range(len(month)), pittsburgh_pred, label= "pittsburgh_predict_mcmc", color = "magenta", linestyle = "dashdot" )
plt.axvline(x=23,color='k',ls='dashed'); #plot advertisement time point as a vertical line
plt.ylabel('Sales $000') ; plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.); #labels and legend
plt.title('RoI by MCMC Approach: ' +  str( round(RoI_mcmc,2)))
plt.show()
```

What's going on here? We are creating a **counterfactual** prediction for sales in pittsburgh using MCMC - what will happen if the advertisement doesn't display in pittsburgh?

This way is much more convincing and theoretically strong than the traditional diff-in-diff approach; worth to mention, it's advocated by Google [Ref 6].

### CONCLUSION

This tutorial aims to guide reader through Markov Chain, Reject Sampling to understand MCMC in detail. A business application case for MCMC is also given, where implementation using PyMC is also a reproducible case for future usage.

Thank you for your time. Please contact yiyoulin@cmu.edu for any questions. :)

### REFERENCES
1. State space estimation in Python via Metropolis-Hastings http://www.chadfulton.com/posts/state_space_mh.html
* Markov Chain Monte Carlo (MCMC) https://people.duke.edu/~ccc14/sta-663/MCMC.html
* MCMC and Gibbs Sampling http://cos.name/2013/01/lda-math-mcmc-and-gibbs-sampling/
* Monte Carlo: Reject Sampling http://blog.csdn.net/baimafujinji/article/details/51407703
* PyMC Tutorial: https://pymc-devs.github.io/pymc/tutorial.html
* Google Causal Impact: https://github.com/google/CausalImpact
* Appendix: Markov Chain Monte Carlo: https://pymc-devs.github.io/pymc/theory.html


# Neural Network Optimization and Tuning

You've learned how to build computational graphs in PyTorch and compute gradients. The final piece to training a network is applying the gradients to update the network parameters. In this tutorial you will learn how to implement a number of optimization techniques in PyTorch along with other tuning methods.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torchvision import transforms, datasets, models

import numpy as np
import matplotlib.pyplot as plt

from collections import namedtuple
from IPython.display import Image
%matplotlib inline

np.random.seed(2020)
```

We use the PyTorch dataset API to load a dataset with exactly the same properties as the MNIST handwritten digits dataset. However, instead of handwritten digits, this dataset contains images of 10 different **common clothing items**, hence the name **Fashion-MNIST** . Performance on MNIST saturates quickly with simple network architectures and optimization methods. This dataset is more difficult than MNIST and is useful to demonstrate the relative improvements of different optimization methods.

Some of the characteristics are mentioned below.

- 28x28 images
- 10 classes
- Single color channel (B&W)
- Centered objects
- 50000 training set members
- 10000 test set members

```python
# Fashion Class that enables the Dataset download and basic transformations
class Fashion(datasets.MNIST):
    def __init__(self, root, train=True, transform=None, target_transform=None, download=False):
        self.urls = [
            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz',
            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz',
            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz',
            'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz',
        ]
        super(Fashion, self).__init__(
            root, train=train, transform=transform, target_transform=target_transform, download=download
        )

def decode_label(l):
    return ["Top",
     "Trouser",
     "Pullover",
     "Dress",
     "Coat",
     "Sandal",
     "Shirt",
     "Sneaker",
     "Bag",
     "Ankle boot"
    ][l]
```

```python
train_data = Fashion('data', train=True, download=True, transform=transforms.Compose([
                                    transforms.ToTensor(),
                                    transforms.Normalize((0.1307,), (0.3081,))
                            ]))

test_data = Fashion('data', train=False, download=True, transform=transforms.Compose([
                                   transforms.ToTensor(),
                                   transforms.Normalize((0.1307,), (0.3081,))
                            ]))
```

    0it [00:00, ?it/s]Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to data/Fashion/raw/train-images-idx3-ubyte.gz
     99%|█████████▉| 26132480/26421880 [00:06<00:00, 5399928.22it/s]Extracting data/Fashion/raw/train-images-idx3-ubyte.gz to data/Fashion/raw

    0it [00:00, ?it/s]Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to data/Fashion/raw/train-labels-idx1-ubyte.gz

      0%|          | 0/29515 [00:00<?, ?it/s]
     56%|█████▌    | 16384/29515 [00:00<00:00, 132285.13it/s]

    0it [00:00, ?it/s]Extracting data/Fashion/raw/train-labels-idx1-ubyte.gz to data/Fashion/raw
    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to data/Fashion/raw/t10k-images-idx3-ubyte.gz

      0%|          | 0/4422102 [00:00<?, ?it/s]

      0%|          | 16384/4422102 [00:00<00:32, 137149.57it/s]

      1%|          | 40960/4422102 [00:00<00:44, 97976.17it/s]

      2%|▏         | 98304/4422102 [00:00<00:33, 128786.41it/s]

      5%|▍         | 212992/4422102 [00:01<00:24, 173985.88it/s]

     10%|▉         | 434176/4422102 [00:01<00:16, 238654.49it/s]

     15%|█▍        | 647168/4422102 [00:01<00:11, 320444.58it/s]

     27%|██▋       | 1187840/4422102 [00:01<00:07, 442308.79it/s]

     41%|████      | 1810432/4422102 [00:01<00:04, 613186.16it/s]

     55%|█████▍    | 2416640/4422102 [00:01<00:02, 838131.02it/s]

     64%|██████▎   | 2818048/4422102 [00:01<00:01, 1077138.04it/s]

     79%|███████▊  | 3481600/4422102 [00:01<00:00, 1431633.75it/s]

     96%|█████████▌| 4251648/4422102 [00:01<00:00, 1891269.47it/s]

    0it [00:00, ?it/s]Extracting data/Fashion/raw/t10k-images-idx3-ubyte.gz to data/Fashion/raw
    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to data/Fashion/raw/t10k-labels-idx1-ubyte.gz

      0%|          | 0/5148 [00:00<?, ?it/s]Extracting data/Fashion/raw/t10k-labels-idx1-ubyte.gz to data/Fashion/raw
    Processing...
    Done!

    26427392it [00:18, 5399928.22it/s]

    4423680it [00:21, 1891269.47it/s]

## Random examples from the Fashion-MNIST dataset

```python
idxs = np.random.randint(100, size=8)
f, a = plt.subplots(2, 4, figsize=(10, 5))
for i in range(8):
    X = train_data.train_data[idxs[i]]
    Y = train_data.train_labels[idxs[i]]
    r, c = i // 4, i % 4
    a[r][c].set_title(decode_label(Y))
    a[r][c].axis('off')
    a[r][c].imshow(X.numpy())
plt.draw()
```

## Build a model

As we are more focussed on evaluating how the different optimization methods perform, we'll be constructing a very simple feedforward network.

```python
class FashionModel(nn.Module):
    def __init__(self):
        super(FashionModel, self).__init__()
        self.fc1 = nn.Linear(784, 64)
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 10)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.log_softmax(self.fc3(x))
        return x
print(FashionModel())
```

    FashionModel(
      (fc1): Linear(in_features=784, out_features=64, bias=True)
      (fc2): Linear(in_features=64, out_features=32, bias=True)
      (fc3): Linear(in_features=32, out_features=10, bias=True)
    )

## A Simple Optimizer in PyTorch

The most simple optimization method is to update the network parameters by adding the negative of the gradient scaled by a fixed learning rate $\eta$.

$$ \textbf{W'} \leftarrow \textbf{W} - \eta \nabla L(\textbf{W}) $$

PyTorch provides us a very simple and expressive API for creating custom optimizers. An optimizer in PyTorch subclasses `torch.optim.Optimizer` and is required to specify two methods.

`__init__`: Must call the superclass `__init__` and provide a list of network parameters, `params`, to optimize and a dictionary of default values provided to each parameter group, `defaults`.

`step`: Performs an update on the network parameters.

The meat of your optimizer logic lies in the `step` method. In this method you should update your model parameters with the help of some useful internal datastructures. Let's define these to make the following code more clear.

`self.param_groups`: When you initialize an optimizer object, you are required to provide the list of parameter objects to be optimized. In the case of `FashionModel`, there are 6 parameters -- each `Linear` layer has a weight matrix parameter and a bias vector. All of these 6 parameters are considered within a single `param_group`. This `group` will be a dictionary with an entry `params` that contains an iterable of all 6 parameters, as well as entries for all `defaults`. These `defaults` are generally useful for storing small values like hyperparameters that are standard across all parameter groups. There are more advanced cases where it can come in handy to have different values for certain entities depending on the `param_group`.

`self.state`: This maintains state for a given parameter. Essentially it maps a parameter to a dictionary of data that you want to keep track of. This is useful in cases where you want to keep state on a per-parameter basis.

**IMPORTANT**: Unlike most other use cases in PyTorch, operations on parameters and state data should be done inplace. This ensures that the updated parameters are not updated copies of the original. In the following sample implementations, you may see some unfamiliar operations. Functions like `torch.add_` and `torch.mul_` are just inplace analogues of standard PyTorch functions. See http://pytorch.org/docs/master/torch.html for further details.

## Let's write our *Trainer*

```python
train_size = train_data.train_data.shape[0]

val_size, train_size = int(0.20 * train_size), int(0.80 * train_size) # 80 / 20 train-val split

test_size = test_data.test_data.shape[0]

batch_size = 100

# Add dataset to dataloader that handles batching
train_loader = torch.utils.data.DataLoader(train_data,
                                           batch_size=batch_size,
                                           sampler=torch.utils.data.sampler.SubsetRandomSampler(np.arange(val_size, val_size+train_size)))
val_loader = torch.utils.data.DataLoader(train_data,
                                           batch_size=batch_size,
                                           sampler=torch.utils.data.sampler.SubsetRandomSampler(np.arange(0, val_size)))

test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, shuffle=False)

# Setup metric class
Metric = namedtuple('Metric', ['loss', 'train_error', 'val_error'])
```

# How to adjust Learning Rate

`torch.optim.lr_scheduler` provides several methods to adjust the learning rate based on the number of epochs. Learning rate scheduling should be applied after optimizer’s update. See https://pytorch.org/docs/stable/optim.html for details.

```python
def inference(model, loader, n_members):
    correct = 0
    for data, label in loader:
        X = Variable(data.view(-1, 784))
        Y = Variable(label)
        out = model(X)
        pred = out.data.max(1, keepdim=True)[1]
        predicted = pred.eq(Y.data.view_as(pred))
        correct += predicted.sum()
    return correct.numpy() / n_members

class Trainer():
    """
    A simple training cradle
    """

    def __init__(self, model, optimizer, load_path=None):
        self.model = model
        if load_path is not None:
            self.model = torch.load(load_path)
        self.optimizer = optimizer

    def save_model(self, path):
        torch.save(self.model.state_dict(), path)

    def run(self, epochs):
        print("Start Training...")
        self.metrics = []

        for e in range(n_epochs):
            scheduler.step()
            epoch_loss = 0
            correct = 0

            for batch_idx, (data, label) in enumerate(train_loader):
                self.optimizer.zero_grad()

                X = Variable(data.view(-1, 784))
                Y = Variable(label)

                out = self.model(X)

                pred = out.data.max(1, keepdim=True)[1]
                predicted = pred.eq(Y.data.view_as(pred))

                correct += predicted.sum()
                loss = F.nll_loss(out, Y)
                loss.backward()
                self.optimizer.step()
                epoch_loss += loss.item()

            total_loss = epoch_loss/train_size
            train_error = 1.0 - correct/train_size
            val_error = 1.0 - inference(self.model, val_loader, val_size)

            print("epoch: {0}, loss: {1:.8f}".format(e+1, total_loss))
            self.metrics.append(Metric(loss=total_loss,
                                  train_error=train_error,
                                  val_error=val_error))

```

```python
### LET'S TRAIN ###

# A function to apply "normal" distribution on the parameters
def init_randn(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0,1)

# We first initialize a Fashion Object and initialize the parameters "normally".
normalmodel = FashionModel()
normalmodel.apply(init_randn)

n_epochs = 8

print("SGD OPTIMIZER")
SGDOptimizer = torch.optim.SGD(normalmodel.parameters(), lr=0.01)
scheduler = torch.optim.lr_scheduler.StepLR(SGDOptimizer, step_size=4, gamma=0.1)

sgd_trainer = Trainer(normalmodel, SGDOptimizer)
sgd_trainer.run(n_epochs)

sgd_trainer.save_model('./sgd_model.pt')
print('')

print("ADAM OPTIMIZER")
normalmodel = FashionModel()
normalmodel.apply(init_randn)

AdamOptimizer = torch.optim.Adam(normalmodel.parameters(), lr=0.01)
scheduler = torch.optim.lr_scheduler.StepLR(AdamOptimizer, step_size=4, gamma=0.1)

adam_trainer = Trainer(normalmodel, AdamOptimizer)
adam_trainer.run(n_epochs)
adam_trainer.save_model('./adam_model.pt')
print('')

print("RMSPROP OPTIMIZER")
normalmodel = FashionModel()
normalmodel.apply(init_randn)
RMSPropOptimizer = torch.optim.RMSprop(normalmodel.parameters(), lr=0.01)
scheduler = torch.optim.lr_scheduler.StepLR(RMSPropOptimizer, step_size=4, gamma=0.1)
rms_trainer = Trainer(normalmodel, RMSPropOptimizer)
rms_trainer.run(n_epochs)
rms_trainer.save_model('./rmsprop_model.pt')
print('')

### TEST ###
model = FashionModel()
model.load_state_dict(torch.load('./sgd_model.pt'))
test_acc = inference(model, test_loader, test_size)
print("Test accuracy of model optimizer with SGD: {0:.2f}".format(test_acc * 100))

model = FashionModel()
model.load_state_dict(torch.load('./adam_model.pt'))
test_acc = inference(model, test_loader, test_size)
print("Test accuracy of model optimizer with Adam: {0:.2f}".format(test_acc * 100))

model = FashionModel()
model.load_state_dict(torch.load('./rmsprop_model.pt'))
test_acc = inference(model, test_loader, test_size)
print("Test accuracy of model optimizer with RMSProp: {0:.2f}".format(test_acc * 100))
```

    SGD OPTIMIZER
    Start Training...
    epoch: 1, loss: 0.11102368
    epoch: 2, loss: 0.02646282
    epoch: 3, loss: 0.02322329
    epoch: 4, loss: 0.02115949
    epoch: 5, loss: 0.02084071
    epoch: 6, loss: 0.02058258
    epoch: 7, loss: 0.02039195
    epoch: 8, loss: 0.02022394

    ADAM OPTIMIZER
    Start Training...
    epoch: 1, loss: 0.28471655
    epoch: 2, loss: 0.02147725
    epoch: 3, loss: 0.01294323
    epoch: 4, loss: 0.00916920
    epoch: 5, loss: 0.00843018
    epoch: 6, loss: 0.00801231
    epoch: 7, loss: 0.00771024
    epoch: 8, loss: 0.00727162

    RMSPROP OPTIMIZER
    Start Training...
    epoch: 1, loss: 0.19673135
    epoch: 2, loss: 0.01578333
    epoch: 3, loss: 0.01320449
    epoch: 4, loss: 0.01104938
    epoch: 5, loss: 0.01017495
    epoch: 6, loss: 0.00946062
    epoch: 7, loss: 0.00891735
    epoch: 8, loss: 0.00849683

    Test accuracy of model optimizer with SGD: 29.80
    Test accuracy of model optimizer with Adam: 74.45
    Test accuracy of model optimizer with RMSProp: 65.29

```python
### VISUALIZATION ###
def training_plot(metrics):
    plt.figure(1)
    plt.plot([m.loss for m in metrics], 'b')
    plt.title('Training Loss')
    plt.show()

training_plot(sgd_trainer.metrics)
training_plot(adam_trainer.metrics)
training_plot(rms_trainer.metrics)
```

## Parameter Initialization

While training a network, the initial value of the weights plays a significant role. In the extreme case, an oracle could just set the weights directly to values that minimize the objective function, and in practical cases a good initialization can bring us to a more favorable starting position in the parameter space.

This raises the question of how to choose these weights.

- What happens if all the weights are set to zero? The gradients become zero, and the network finds itself without a direction.
- What if all of them are set to the same non-zero value? Although the gradients are no longer zero, each neuron has the same weight and follows the same gradient. Such neurons will continue to have the same value, since they're identical.

So any initialization scheme must break this symmetry somehow, and randomly initializing the weights is a first step in that direction.

Let's begin with creating a weight initialization function that samples from **N(0,1)**. A clean way of initializing the weights is to access the network parameters by traversing all modules inside the network, and then applying the desired initialization. This method also allows us to encapsulate all the initializations into a single function.

```python
def init_randn(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0,1)
```

Now let's use this scheme to initialize the network.

Note that *apply(fn)* applies the function *fn* recursively to every submodule (as returned by .children()) as well as self. Also, since it is applied to itself as well, you must take care to select the appropriate type of module *m* and apply the initialization to it.

```python
normalmodel = FashionModel()
normalmodel.apply(init_randn)
```

    FashionModel(
      (fc1): Linear(in_features=784, out_features=64, bias=True)
      (fc2): Linear(in_features=64, out_features=32, bias=True)
      (fc3): Linear(in_features=32, out_features=10, bias=True)
    )

## Custom initializations

We could also choose a different way to initialize the weights, where you explicitly copy some values into the weights.

```python
def init_custom(m):
    if type(m) == nn.Linear:
        rw = torch.randn(m.weight.data.size())
        m.weight.data.copy_(rw)
```

Now let's use this initialization scheme to implement Xavier initialization.

Xavier initialization is a way of initializing the weights such that the variance of the inputs is the same as the variance of the outputs. At each layer, the fan_in and fan_out (i.e. input connections and output connections) might be different. To calculate the variance, you will multiply each weight with the inputs. Evidently, if the number of inputs is less, they will need to be multiplied with higher weights so that they can sum up to the product of a larger number of outputs with smaller weights. This is the intuition behind Xavier initialization.

```python
def init_xavier(m):
    if type(m) == nn.Linear:
        fan_in = m.weight.size()[1]
        fan_out = m.weight.size()[0]
        std = np.sqrt(2.0 / (fan_in + fan_out))
        m.weight.data.normal_(0,std)
```

```python
xaviermodel = FashionModel()
xaviermodel.apply(init_xavier)
```

    FashionModel(
      (fc1): Linear(in_features=784, out_features=64, bias=True)
      (fc2): Linear(in_features=64, out_features=32, bias=True)
      (fc3): Linear(in_features=32, out_features=10, bias=True)
    )

```python
### LET'S TRAIN ###
n_epochs = 3

print("NORMAL INIT WEIGHTS")
AdamOptimizer = torch.optim.Adam(normalmodel.parameters(), lr=0.001)
normal_trainer = Trainer(normalmodel, AdamOptimizer)
normal_trainer.run(n_epochs)
normal_trainer.save_model('./normal_model.pt')
print('')

print("XAVIER INIT WEIGHTS")
AdamOptimizer = torch.optim.Adam(xaviermodel.parameters(), lr=0.001)
xavier_trainer = Trainer(xaviermodel, AdamOptimizer)
xavier_trainer.run(n_epochs)
xavier_trainer.save_model('./xavier_model.pt')
print('')
```

    NORMAL INIT WEIGHTS
    Start Training...
    epoch: 1, loss: 1.04583949
    epoch: 2, loss: 0.06138324
    epoch: 3, loss: 0.03620169

    XAVIER INIT WEIGHTS
    Start Training...
    epoch: 1, loss: 0.00553334
    epoch: 2, loss: 0.00382321
    epoch: 3, loss: 0.00341060

```python
### VISUALIZATION ###
def training_plot(metrics):
    plt.figure(1)
    plt.plot([m.loss for m in metrics], 'b')
    plt.title('Training Loss')
    plt.show()

training_plot(normal_trainer.metrics)
training_plot(xavier_trainer.metrics)
```

## Using pretrained weights

In the previous section we saw that initializations can start the training from a good spot. In addition to these schemes, you might also need to have specific methods to initialize the weights in different layers. For example, you might want to use a pretrained model like Alexnet to give your network a head start for visual recognition tasks. Let's load the pretrained Alexnet model and see how it works.

```python
alexnet_model = models.alexnet(pretrained=True)
```

    Downloading: "https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth" to /Users/kalpak/.cache/torch/checkpoints/alexnet-owt-4df8aa71.pth
    100%|██████████| 233M/233M [01:23<00:00, 2.93MB/s]

## Adding Momentum

We can make use of the `self.state` data-structure to maintain a copy of an accumulated gradient that we also decay at each step. Once again we use inplace operations to avoid unneccesary buffer allocation. Recall a standard update with momentum given decay rate $\mu$.

$$ \begin{align}
\textbf{V'} &= \mu \textbf{V} - \eta \nabla L(\textbf{W})\\
\textbf{W'} &= \textbf{W} + \textbf{V'}\\
\end{align}
$$

## Batch Normalization

Batch normalization is a relatively simple but significant improvement in training neural networks. In machine learning, *covariate shift* is a phenomenon in which the covariate distribution is non-stationary over the course of training. This is a common phenomenon in online learning. When training a neural network on a fixed dataset, there is no covariate shift (excluding sample noise from minibatch approximation), but the distribution of individual node and layer activity shifts as the network parameters are updated. As an abstraction, we can consider each node's activity to be a covariate of the following nodes in the network. Thus we can think of the non-stationarity of node (and layer) activations as a sort of *internal covariate shift*.

Why is internal covariate shift a problem? Each subsequent layer has to account for a shifting distribution of its inputs. For saturating non-linearities the problem becomes even more dire, as the shift in activity will more likely place the unit output in the saturated region of the non-linearity.

```python
class BatchNorm(nn.Module):

    def __init__(self, num_features):
        super(BatchNorm, self).__init__()
        self.num_features = num_features
        self.affine = affine
        self.weight = Parameter(torch.Tensor(num_features))
        self.bias = Parameter(torch.Tensor(num_features))
        self.register_buffer('running_mean', torch.zeros(num_features))
        self.register_buffer('running_var', torch.ones(num_features))
        self.reset_parameters()

    def reset_parameters(self):
        self.running_mean.zero_()
        self.running_var.fill_(1)
        self.weight.data.uniform_()
        self.bias.data.zero_()

    def forward(self, x):
        pass
```

## Overfitting
Deep neural networks contain multiple non-linear hidden layers and this makes them very
expressive models that can learn very complicated relationships between their inputs and
outputs. With limited training data, however, many of these complicated relationships
will be the result of sampling noise, so they will exist in the training set but not in real
test data even if it is drawn from the same distribution. This leads to overfitting and many
methods have been developed for reducing it.

## Dropout
Dropout is a regularization technique for reducing overfitting in neural networks by preventing complex co-adaptations on training data. It is a very efficient way of performing model averaging with neural networks. The term "dropout" refers to dropping out units in a neural network.

## Regularization (weight_decay)

Weight decay specifies regularization in the neural network.
During training, a regularization term is added to the network's loss to compute the backpropagation gradient. The weight decay value determines how dominant this regularization term will be in the gradient computation.

As a rule of thumb, the more training examples you have, the weaker this term should be. The more parameters you have the higher this term should be.

```python
class FashionModel_Tricks(nn.Module):

    def __init__(self):
        super(FashionModel_Tricks, self).__init__()
        self.fc1 = nn.Linear(784, 64)
        self.bnorm1 = nn.BatchNorm1d(64)
        self.dp1 = nn.Dropout(p=0.2)
        self.fc2 = nn.Linear(64, 32)
        self.bnorm2 = nn.BatchNorm1d(32)
        self.dp2 = nn.Dropout(p=0.1)
        self.fc3 = nn.Linear(32, 10)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.dp1(self.bnorm1(x))
        x = F.relu(self.fc2(x))
        x = self.dp2(self.bnorm2(x))
        x = F.log_softmax(self.fc3(x))
        return x
print(FashionModel_Tricks())
```

    FashionModel_Tricks(
      (fc1): Linear(in_features=784, out_features=64, bias=True)
      (bnorm1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (dp1): Dropout(p=0.2)
      (fc2): Linear(in_features=64, out_features=32, bias=True)
      (bnorm2): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (dp2): Dropout(p=0.1)
      (fc3): Linear(in_features=32, out_features=10, bias=True)
    )

```python
### TRAIN MODELS WITH BATCHNORM AND DROPOUT ###
n_epochs = 10

model = FashionModel_Tricks()
optimizer = torch.optim.SGD(model.parameters(), lr = 0.001, momentum = 0.9, weight_decay = 0.001)
btrainer = Trainer(model, optimizer)
btrainer.run(n_epochs)
btrainer.save_model('./dropout-batchnorm_optimized_model.pt')

training_plot(btrainer.metrics)

print('')
```

## Gradient Clipping

During experimentation, once the gradient value grows extremely large, it causes an overflow (i.e. NaN) which is easily detectable at runtime or in a less extreme situation, the Model starts overshooting past our Minima; this issue is called the Gradient Explosion Problem.

Gradient clipping will ‘clip’ the gradients or cap them to a Threshold value to prevent the gradients from getting too large.

```python
#Gradient Clipping
# `clip_grad_norm` helps prevent the exploding gradient problem. To be used before optimizer.step()during training
torch.nn.utils.clip_grad_norm(model.parameters(), 0.25)
```

    /Users/advaitgadhikar/anaconda/lib/python2.7/site-packages/ipykernel/__main__.py:3: UserWarning: torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.
      app.launch_new_instance()

    tensor(2.6596)

## Annealing Learning Rate
In training deep networks, it is usually helpful to anneal the learning rate over time. Good intuition to have in mind is that with a high learning rate, the system contains too much kinetic energy and the parameter vector bounces around chaotically, unable to settle down into deeper, but narrower parts of the loss function. Knowing when to decay the learning rate can be tricky: Decay it slowly and you’ll be wasting computation bouncing around chaotically with little improvement for a long time. But decay it too aggressively and the system will cool too quickly, unable to reach the best position it can. One way of doing it is using step decay. Step decay schedule drops the learning rate by a factor every few epochs. The mathematical form of step decay is:

$$\eta = \eta_0 * drop^{\floor ( \frac{epoch}{epochs\_drop})}$$

```python
def step_decay(epoch):
    initial_lrate = 0.1
    drop = 0.5
    epochs_drop = 10.0
    lrate = initial_lrate * math.pow(drop,
           math.floor((1+epoch)/epochs_drop))
    return lrate
```


$$
\newcommand{\EE}{\mathbb{E}}
\newcommand{\wb}{\mathbf{w}}
\newcommand{\zb}{\mathbf{z}}
\newcommand{\Poisson}{\text{Poisson}}
\newcommand{\Dir}{\text{Dir}}
\newcommand{\Multinomial}{\text{Multinomial}}
$$

```python
%matplotlib inline
```

```python
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
sns.set_style("ticks")
sns.set_context("talk", font_scale=2, rc={"lines.linewidth": 2})
```

<sup>* Picture are reproduced from [original paper](http://jmlr.csail.mit.edu/papers/v3/blei03a.html)</sup>

+ A **word** is the basic unit of discrete data, defined to be an item from a vocabulary indexed by $\{1,...,V\}$.

+ A **document** is a sequence of N words denoted by $\wb = (w_1,w_2,...,w_N)$, where $w_n$ is the nth word in the sequence.

+ A **corpus** is a collection of M documents denoted by $D = \{w_1,w_2,...,w_M\}$.

> We wish to find a probabilistic model of a corpus that not only assigns high probability to
members of the corpus, but also assigns high probability to other “similar” documents.

## Latent Dirichlet Allocation

1. Choose $N \sim \Poisson(\zeta)$.
2. Choose $\theta \sim \Dir(\alpha)$.
3. For each of the $N$ words $w_n$:

  (a) Choose a topic $z_n \sim \Multinomial(\theta)$.

  (b) Choose a word $w_n$ from $p(w_n \vert z_n,\beta)$, a multinomial probability conditioned on the topic $z_n$.

```python
W = 200
D = 200
D_test = 160
K = 4
N = 50
alpha = np.array(np.linspace(0.2, 1, 4))
beta = np.random.uniform(size=(N, K)).T
beta = beta / beta.sum(axis=1)[:, np.newaxis]
data = []
topics = []
data_test = []
topics_test = []
for d in range(D):
    d__ = []
    theta = np.random.dirichlet(alpha, 1)[0]

    for w in range(W):
        z = np.random.multinomial(n=1, pvals=theta, size=1)
        z_ = z.astype(bool)[0]
        rand_ = np.random.uniform()
        d__.append((rand_ > np.cumsum(beta[z_, :])).sum())
    if d > D_test:
        data_test.append(d__)
        topics_test.append(theta)
    else:
        data.append(d__)
        topics.append(theta)
topics = np.array(topics)
data = np.array(data)
```

```python
plt.figure(figsize=(6 * 3, 6))
plt.subplot(1, 2, 1)
vals, counts = np.unique(data, return_counts=True)
plt.bar(vals, counts / counts.sum())
plt.xlabel("Word")
_ = plt.ylabel("Frequency of words \n in corpus")
plt.subplot(1, 2, 2)
plt.bar(np.arange(topics.shape[1]), topics.sum(axis=0))
plt.xlabel("Topics")
plt.xticks(np.arange(topics.shape[1]))
_ = plt.ylabel("Count")
```

### Likelihood function

Given parameters $\alpha$ and $\beta$, the joint distribution of a topic mixture $\mathbf{\theta}$, a set of $N$ topics $\mathbf{z}$, and a set of $N$ words $\mathbf{w}$ is given by:

\begin{align}
p(\mathbf{\theta},\mathbf{z},\mathbf{w}\vert \alpha,\beta)=p(\mathbf{\theta}\vert \alpha)\prod_{n=1}^{N}p(z_n\vert \mathbf{\theta})p(w_n\vert z_n,\beta)
\end{align}

We only observe $\wb$ while $\theta$ and $\zb$ are not observed. Integrating over $\mathbf{\theta}$ and summing over $z$, we get marginal probabilities of a document:

\begin{align}
p(\mathbf{w}\vert \alpha,\beta)=\int p(\mathbf{\theta}\vert \alpha)\big(\prod_{n=1}^{N}\sum_{z_{dn}}p(z_n\vert \mathbf{\theta})p(w_n\vert z_n,\beta)\big)d\theta
\end{align}

Finally the likelihood of observing a corpus is:

\begin{align}
p(D\vert \alpha,\beta)=\prod_{d=1}^{M}\int p(\mathbf{\theta}_d\vert \alpha)\big(\prod_{n=1}^{N}\sum_{z_{dn}}p(z_n\vert \mathbf{\theta}_d)p(w_n\vert z_n,\beta)\big)d\theta_d
\end{align}

----

## Inference

We take a Bayesian perspective, and are interested in posterior distribtion
after observing data.

for a given document we have:

\begin{align}
p(\mathbf{\theta},\mathbf{z}\vert \mathbf{w},\alpha,\beta) = \frac{p(\mathbf{\theta},\mathbf{z},\mathbf{w}|\alpha,\beta)}{p(\mathbf{w}\vert \alpha,\beta)}
\end{align}

Normalization constant in this distribution is intractable, therefore, we try other methods such as **Variational Inference**, *Sampling* or *Approxiamtion*.

### Variational Inference

> The basic idea of convexity-based variational inference is to make use of Jensen’s inequality to obtain
an adjustable lower bound on the log likelihood (Jordan et al., 1999). Essentially, one considers
a family of lower bounds, indexed by a set of variational parameters. The variational parameters
are chosen by an optimization procedure that attempts to find the tightest possible lower bound.

Take a prticular family of variational distributions:
\begin{align}
q(\mathbf{\theta},\mathbf{z}|\gamma,\phi) = q(\theta|\gamma)\prod_{n=1}^{N} q(z_n|\mathbf{\phi}_n)
\end{align}

Set up an optimization problem that determines the values of the variational parameters $\gamma$ and $\phi$:

\begin{align}
(\gamma^{*},\phi^{*}) = \text{argmin}_{(\gamma,\phi)}
\text{D}\big[q(\mathbf{\theta},\mathbf{z}|\gamma,\phi) ||p(\mathbf{\theta},\mathbf{z}|\mathbf{w},\alpha,\beta)\big].
\end{align}

Which results in following pair of update rules:

\begin{align}
\phi_{ni} \propto \beta_{iv} \exp{(\Psi(\gamma_i)-\Psi(\sum_{j=1}^{k}\gamma_i))}.
\end{align}

\begin{align}
\gamma_{i} = \alpha_i+\sum_{n=1}^{N} \phi_{ni}
\end{align}

#### Derivation
\begin{align}
p(\mathbf{\theta},\mathbf{z},\mathbf{w}|\alpha,\beta)=p(\mathbf{\theta}|\alpha)\prod_{n=1}^{N}p(z_n|\mathbf{\theta})p(w_n|z_n,\beta)
\end{align}

\begin{align}
\log{p(\mathbf{w}|\alpha,\beta}) =~& \log{\int \sum_{\mathbf{z}} p(\theta,\mathbf{z},\mathbf{w}|\alpha,\beta)}d\theta\nonumber\\
=~&\log{\int \sum_{\mathbf{z}}\frac{ p(\theta,\mathbf{z},\mathbf{w}|\alpha,\beta)q(\theta,\mathbf{z})}{q(\theta,\mathbf{z})}}d\theta\nonumber\\
\geq~&\int \sum_{\mathbf{z}} q(\theta,\mathbf{z})\log{p(\theta,\mathbf{z},\mathbf{w}|\alpha,\beta)}-q(\theta,\mathbf{z})\log{q(\theta,\mathbf{z}})d\theta\tag*{Jensen's inequality}\\
=~& \EE_q[{\log{p(\theta,\zb,\wb|\alpha,\beta})}]-\EE_q[\log{q(\theta,\zb)}] = L (\gamma,\phi;\alpha,\beta)
\end{align}

\begin{align}
L(\gamma,\phi;\alpha,\beta) = \EE_q[\log{p(\theta|\alpha)}]+ \EE_q[\log{p(\zb|\theta)}]+\EE_q[\log{p(\wb|\zb,\beta)}]- \EE_q[\log{q(\theta)}] - \EE_q[\log{q(\zb)}]
\end{align}

\begin{align}
L(\gamma,\phi;\alpha,\beta) &~= \log{\Gamma(\sum_{j=1}^k \alpha_j)} - \sum_{i=1}^k \log{\Gamma(\alpha_i)} + \sum_{i=1}^k (\alpha_i - 1)\big(\Psi(\gamma_i)-\Psi(\sum_{j=1}^k \gamma_j)\big)\\
&~+ \sum_{n=1}^N \sum_{i=1}^k \phi_{ni}\big(\Psi(\gamma_i) - \Psi(\sum_{j=1}^k \gamma_j)\big)\\
&~+ \sum_{n=1}^N \sum_{i=1}^k \phi_{ni}w_n^j \log{\beta_{ij}}\\
&~- \log{\Gamma(\sum_{j=1}^k \gamma_j)} + \sum_{i=1}^k \log{\Gamma(\gamma_i)} - \sum_{i=1}^k (\gamma_i - 1)(\Psi(\gamma_i)-\Psi(\sum_{j=1}^k \gamma_j))\\
&~- \sum_{n=1}^N \sum_{i=1}^k \phi_{ni}\log{\phi_{ni}}
\end{align}

Maximizing the above equation wrt to each parameter:

\begin{align}
L_{[\phi_{ni}]} =~&  \phi_{ni}(\Psi(\gamma_i)-\Psi(\sum_{j=1}^{k}\gamma_i) + \phi_{ni}\log{\beta_{iv}} - \phi_{ni}\log(\phi_{ni}) + \lambda_n(\sum_{j=1}^{k} \phi_{nj} -1)\\
\frac{\partial L_{[\phi_{ni}]}}{\partial \phi_{ni}} =~&  \Psi(\gamma_i)-\Psi(\sum_{j=1}^{k}\gamma_i) + \log{\beta_{iv}} - \log(\phi_{ni}) -1 + \lambda\\
\phi_{ni} \propto~&  \beta_{iv} \exp{(\Psi(\gamma_i)-\Psi(\sum_{j=1}^{k}\gamma_i))}.
\end{align}

---

\begin{align}
L_{[\gamma_{i}]} =~& \sum_{i=1}^{k}(\alpha_i - 1)(\Psi(\gamma_i)-\Psi(\sum_{j=1}^{k}\gamma_j)) + \sum_{n=1}^{N} \phi_{ni}(\Psi(\gamma_i) - \Psi(\sum_{j=1}^{k}\gamma_j))\\
%
&-\log{\Gamma(\sum_{j=1}^{k}\gamma_j)}+\log{\Gamma(\gamma_i)-\sum_{i=1}^{k}(\gamma_i-1)(\Psi(\gamma_i)-\Psi(\sum_{j=1}^{k}\gamma_j))}\\
%%
=~& \sum_{i=1}^{k}(\Psi(\gamma_i)-\Psi(\sum_{j=1}^{k}\gamma_j)(\alpha_i + \sum_{n=1}^{N}\phi_{ni}-\gamma_{i})-\log{\Gamma(\sum_{j=1}^{k}\gamma_j)}+\log{\Gamma(\gamma_i)}\\
%
\frac{\partial L_{[\gamma_{i}]}}{\partial\gamma_{i}} =~& \Psi'(\gamma_i)(\alpha_i + \sum_{n=1}^{N}\phi_{ni}-\gamma_{i})-\Psi'(\sum_{j=1}^{k}\gamma_j)\sum_{j=1}^{k}(\alpha_j + \sum_{n=1}^{N}\phi_{nj}-\gamma_{j})\\
\gamma_{i} =~& \alpha_i+\sum_{n=1}^{N} \phi_{ni}
\end{align}

## Experiments

```python
alpha0 = np.ones(K)
phi = np.zeros((data.shape[0], data.shape[1], K))
phi[:, :, :] = 1 / K
gamma = np.zeros((data.shape[0], K))
gamma[:, :] = alpha + phi.shape[0] / K
beta0 = np.random.uniform(size=(K, N))
beta0 = beta / beta.sum(axis=1)[:, np.newaxis]
```

```python
def L(phi, gamma, beta0, alpha0):
    temp = 0
    for m in range(phi.shape[0]):
        ss = sp.special.digamma(gamma[m, :]) -\
             sp.special.digamma(gamma[m, :].sum())

        temp = sp.special.loggamma(
            alpha0.sum()) - sp.special.loggamma(alpha0).sum() + ((alpha0 - 1) * ss).sum()

        temp += (phi[m, :, :] * ss).sum()

        temp += (phi[m, :, :] * beta0[:, data[m, :]].T).sum()

        temp += -sp.special.loggamma(gamma[m, :].sum()) + sp.special.loggamma(
            gamma[m, :]).sum() - ((gamma[m, :] - 1) * ss).sum()

        temp += (-phi[m, :, :] * np.log(phi[m, :, :])).sum()

    return np.real(temp)

def update_vi(phi, gamma, beta0, alpha0):
    m = 0
    history = []
    iter_ = 0
    beta = beta0.copy()
    while(True):
        phi_temp = phi.copy()
        gamma_temp = gamma.copy()
        for m in range(phi.shape[0]):
            for n in range(phi.shape[1]):
                for i in range(gamma.shape[1]):
                    phi_temp[m, n, i] = beta[i, data[m, n]] * np.exp(sp.special.digamma(gamma[m, i]) -\
                                                                     sp.special.digamma(gamma[m, :].sum()))

            phi_temp[m, :, :] = phi_temp[m, :, :] /\
                                phi_temp[m, :, :].sum(axis=1)[:, np.newaxis]
        for m in range(gamma.shape[0]):
            for i in range(gamma.shape[1]):
                gamma_temp[m, i] = alpha0[i] + phi_temp[m, :, i].sum()

        diff_gamma = np.mean((gamma_temp - gamma)**2)
        diff_phi = np.mean((phi_temp - phi)**2)

        if ((diff_phi < 1e-3) & (diff_gamma < 1e-3)):
            break

        phi = phi_temp
        gamma = gamma_temp

        L_ = L(phi, gamma, beta, alpha0)

        print("diff gamma %f" % diff_gamma, "diff phi %f" %
              diff_phi, "L %f" % L_)
        history.append((beta, phi, gamma, L_))
        iter_ += 1
    return phi, gamma, beta, iter_, history
```

```python
new_phi, new_gamma, new_beta, iter_, history = update_vi(
    phi, gamma, beta0, alpha0)

diff_gamma = np.mean(np.abs(new_gamma - gamma))
diff_phi = np.mean(np.abs(new_phi - phi))
diff_beta = np.mean(np.abs(new_beta - beta))
print("iter", iter_)
print("diff gamma %f" % diff_gamma, "diff phi %f" % diff_phi)
print("diff beta %f" % diff_beta)
```

    diff gamma 133.635209 diff phi 0.014895 L -23.776968
    diff gamma 25.256380 diff phi 0.000735 L -23.141265
    diff gamma 22.212411 diff phi 0.000636 L -22.511695
    diff gamma 18.884462 diff phi 0.000534 L -21.896922
    diff gamma 15.560183 diff phi 0.000436 L -21.305556
    diff gamma 12.495897 diff phi 0.000349 L -20.745106
    diff gamma 9.842222 diff phi 0.000274 L -20.221272
    diff gamma 7.646850 diff phi 0.000213 L -19.737676
    diff gamma 5.888301 diff phi 0.000164 L -19.295954
    diff gamma 4.510381 diff phi 0.000126 L -18.896069
    diff gamma 3.446265 diff phi 0.000097 L -18.536718
    diff gamma 2.631902 diff phi 0.000074 L -18.215718
    diff gamma 2.011879 diff phi 0.000057 L -17.930349
    diff gamma 1.540939 diff phi 0.000044 L -17.677624
    diff gamma 1.183386 diff phi 0.000034 L -17.454480
    diff gamma 0.911664 diff phi 0.000026 L -17.257917
    diff gamma 0.704776 diff phi 0.000020 L -17.085083
    diff gamma 0.546850 diff phi 0.000016 L -16.933323
    diff gamma 0.425937 diff phi 0.000012 L -16.800206
    diff gamma 0.333056 diff phi 0.000010 L -16.683532
    diff gamma 0.261459 diff phi 0.000008 L -16.581325
    diff gamma 0.206069 diff phi 0.000006 L -16.491823
    diff gamma 0.163057 diff phi 0.000005 L -16.413463
    diff gamma 0.129534 diff phi 0.000004 L -16.344865
    diff gamma 0.103307 diff phi 0.000003 L -16.284812
    diff gamma 0.082711 diff phi 0.000002 L -16.232234
    diff gamma 0.066476 diff phi 0.000002 L -16.186193
    diff gamma 0.053631 diff phi 0.000002 L -16.145866
    diff gamma 0.043431 diff phi 0.000001 L -16.110535
    diff gamma 0.035300 diff phi 0.000001 L -16.079568
    diff gamma 0.028795 diff phi 0.000001 L -16.052417
    diff gamma 0.023572 diff phi 0.000001 L -16.028601
    diff gamma 0.019363 diff phi 0.000001 L -16.007698
    diff gamma 0.015959 diff phi 0.000000 L -15.989344
    diff gamma 0.013197 diff phi 0.000000 L -15.973218
    diff gamma 0.010948 diff phi 0.000000 L -15.959041
    diff gamma 0.009111 diff phi 0.000000 L -15.946569
    diff gamma 0.007605 diff phi 0.000000 L -15.935589
    diff gamma 0.006366 diff phi 0.000000 L -15.925916
    diff gamma 0.005344 diff phi 0.000000 L -15.917387
    diff gamma 0.004498 diff phi 0.000000 L -15.909861
    diff gamma 0.003796 diff phi 0.000000 L -15.903213
    diff gamma 0.003211 diff phi 0.000000 L -15.897338
    diff gamma 0.002723 diff phi 0.000000 L -15.892138
    diff gamma 0.002315 diff phi 0.000000 L -15.887534
    diff gamma 0.001972 diff phi 0.000000 L -15.883451
    diff gamma 0.001683 diff phi 0.000000 L -15.879829
    diff gamma 0.001440 diff phi 0.000000 L -15.876610
    diff gamma 0.001234 diff phi 0.000000 L -15.873747
    diff gamma 0.001059 diff phi 0.000000 L -15.871198
    iter 50
    diff gamma 40.601717 diff phi 0.232107
    diff beta 0.000000

```python
vals = []
val_name = []
val_exp = []
val_unit = []
for ind, h in enumerate(history):
    t = np.abs(h[2] / h[2].sum(axis=1)[:, np.newaxis] - topics)
    vals.extend(t.flatten().tolist())
    val_name.extend(['$\\bar{\\gamma}-\\theta$'] * t.shape[0] * t.shape[1])
    val_exp.extend([ind] * t.shape[0] * t.shape[1])
    val_unit.extend(np.arange(t.shape[0] * t.shape[1]))

    l = np.exp(h[3])
    vals.extend([l])
    val_name.extend(['$L$'])
    val_exp.extend([ind])
    val_unit.extend([0])
```

```python
import pandas as pd

df = pd.DataFrame(data={"vals": vals, "param": val_name,
                        "exp": val_exp, "unit": val_unit})
```

```python
g = sns.factorplot(data=df, x="exp", y="vals",
                   row="param", sharey=False, aspect=3).set_titles("{row_name}")
g.axes[0, 0].set_ylabel(g.axes[0, 0].get_title())
g.axes[1, 0].set_ylabel(g.axes[1, 0].get_title())
g.axes[0, 0].set_title("")
g.axes[1, 0].set_title("")
g.axes[1, 0].grid("on")
g.axes[0, 0].grid("on")
g.axes[0, 0].set_xticks(np.arange(0, len(history), 5))
g.axes[0, 0].set_xticklabels(np.arange(0, len(history), 5))
plt.xlabel("iteration")
```


```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

# MEC6514 - Masse ajoutée

Ce document est un support au cours sur la masse ajoutée. Vous y retrouverez la démarche pour obtenir la masse ajoutée d'un cylindre circulaire, à l'aide de Python, ainsi que les formules pour d'autres formes plus complexes, fournies par Naudascher et Rockwell (1994).

## Masse ajoutée d'un cylindre circulaire

Pour pouvoir déterminer la masse ajoutée d'un cylindre circulaire, on applique la théorie des écoulements potentiels, qui nous permet de simplifier les équations de Navier-Stokes. Ceci amène les hypothèses suivantes :
- Fluide parfait ($R_e \rightarrow \infty$ et viscosité négligeable)
- Ecoulement irrotationnel (vorticité nulle)
- Fluide incompressible (hypothèse que l'on ajoute et non obligatoire)

Avant de décrire le cylindre, on importe les librairies de Python nécessaires :

```python
# librairie et outils pour le calcul symbolique
from sympy import symbols,dsolve,diff,Function,lambdify
import sympy as sp

# librairie pour le calcul numérique
import numpy as np

# librairie pour les figures et tracés
import matplotlib.pyplot as plt
```

On considère un cylindre circulaire de rayon $a$ se déplaçant à la vitesse $U$ et l'accélération $\dot{U}$ dans un fluide au repos remplissant un domaine infini (de masse volumique $\rho$).

```python
t = symbols('t') # permet de définir t comme une variable inconnue

# les paramètres suivants peuvent être changés à votre guise
a = 0.02         # rayon du cylindre (en m)
U = t+2          # Vitesse dépendant du temps, pour une accélération non nulle
rho = 1000       # masse volumique du fluide (en kg/m^3)
```

On définit les coordonnées comme cylindriques (ou polaires). Pour obtenir la masse ajoutée du cylindre, il faut déterminer dans un premier temps le champs de vitesse du fluide environnant. Pour cela, on définit les conditions limites :
- Condition de non-pénétration sur la paroi du cylindre :
    à $r=a$, la vitesse radiale $u_r=U\cos{\theta}$
- Loin du cylindre, pas d'effet de celui-ci sur l'écoulement :
    à $r=\infty$, la vitesse radiale $u_r=0$ et la vitesse azimutale $u_{\theta}=0$

On résout l'équation de Laplace en coordonnées cylindriques (et 2D), dont la solution est le potentiel de déplacement fluide $\phi(r,\theta)$ :
\begin{equation*}
\frac{1}{r} \frac{\partial}{\partial r}\left(r \frac{\partial\phi}{\partial r}\right) + \frac{1}{r^2} \frac{\partial^2\phi}{\partial\theta^2}= 0
\end{equation*}
Pour résoudre cette équation, on applique la méthode de séparation des variables : $\phi(r,\theta) = R(r)T(\theta)$, ce qui nous donne :
\begin{equation*}
\frac{r}{R(r)}\frac{\partial}{\partial r}\left(r \frac{\partial R}{\partial r}\right)=-\frac{1}{T(\theta)} \frac{\partial^2 T}{\partial\theta^2}
\end{equation*}
Les termes $R(r)$ et $T(\theta)$ étant indépendants, chaque côté de l'expression est égal à une constante. En posant cette constante égale à $1$, on obtient deux équations découplées, qu'il nous est possible de résoudre avec Python :
\begin{equation*}
\frac{\partial^2 T}{\partial\theta^2}+T(\theta)=0 \textrm{ et }r^2 \frac{\partial^2 R}{\partial r^2} +r \frac{\partial R}{\partial r} - R(r) = 0
\end{equation*}

```python
theta = symbols('theta')
T = Function('T')(theta) # on introduit T la fonction symbolique d'argument theta

# dsolve résout des EDO et diff est l'opérateur différentiel symbolique
T_theta = dsolve(diff(diff(T,theta),theta)+T,T).args[1]
print('T(theta) = ',T_theta)
```

    T(theta) =  C1*sin(theta) + C2*cos(theta)

On procède de la même manière avec $R(r)$ :

```python
r = symbols('r')
R = Function('R')(r)

R_r = dsolve(r**2*diff(diff(R,r),r)+r*diff(R,r)-R,R).args[1]
print(' R(r) = ',R_r)
```

     R(r) =  C1/r + C2*r

On applique ensuite les conditions limites pour déterminer les constantes inconnues :
- $r = \infty$, $u_r=0$ $\rightarrow$ $C_2^r=0$
- $r = \infty$, $u_{\theta}=0$ $\rightarrow$ $C_2^{\theta}=C_1^{\theta}\tan{\theta}$
- $r=a$, $u_r=U\cos{\theta}$ $\rightarrow$ $2 C_1^r C_1^{\theta} = -a^2 U$

Soit :
\begin{equation*}
\phi=-\frac{a^2}{r} U\cos{\theta}
\end{equation*}
Et :
\begin{equation*}
u_r = \frac{\partial\phi}{\partial r}=\frac{a^2}{r^2} U\cos{\theta} \textrm{, } u_{\theta} = \frac{1}{r}\frac{\partial\phi}{\partial\theta}=\frac{a^2}{r^2} U\sin{\theta}
\end{equation*}

On peut maintenant tracer les lignes de courant autour du cylindre. Ce dernier étant en accélération, il est nécessaire de choisir un instant auquel on effectue le tracé. On a arbitrairement choisi l'instant initial $t=0$, mais vous pouvez changer celui-ci.

```python
U_0 = lambdify(t,U)(0) # changement d'instant à effectuer ici
phi = -a**2/r*U_0*sp.cos(theta)

# vitesse radiale
ur = lambdify((r,theta),diff(phi,r)) # passage d'une expression symbolique à une fonction numérique

# vitesse tangentielle (ou azimutale)
utheta = lambdify((r,theta),1/r*diff(phi,theta))

''' Calcul du champs de vitesse en coordonnées cartésiennes pour tracer '''
Ux = np.zeros((int(round(5*a*1000,0))+1,int(round(5*a*1000,0))+1))
Vy = np.zeros((int(round(5*a*1000,0))+1,int(round(5*a*1000,0))+1))

# x1 -> y et x2 -> x
x1,x2 = np.arange(-2.5*a,2.5*a+0.001,0.001), np.arange(-2.5*a,2.5*a+0.001,0.001)
i = 0
while i < int(round(5*a*1000,0))+1:
    j = 0
    while j < int(round(5*a*1000,0))+1:
        # calcul des coordonnées cylindriques
        r = np.sqrt((x1[i])**2+(x2[j])**2)
        theta = np.arcsin(x1[i]/np.sqrt((x1[i])**2+(x2[j])**2))
        if r>a:
            if x1[i] > 0 and x2[j] > 0:
                # quadrant supérieur droit
                Ux[i,j] = ur(r,theta)*np.cos(theta)-utheta(r,theta)*np.sin(theta)
                Vy[i,j] = ur(r,theta)*np.sin(theta)+utheta(r,theta)*np.cos(theta)
            elif x1[i] > 0 and x2[j] < 0:
                # quadrant supérieur gauche
                Ux[i,j] = -ur(r,np.pi-theta)*np.cos(theta)-utheta(r,np.pi-theta)*np.sin(theta)
                Vy[i,j] = ur(r,np.pi-theta)*np.sin(theta)-utheta(r,np.pi-theta)*np.cos(theta)
            elif x1[i] < 0 and x2[j] < 0:
                # quadrant inférieur gauche
                Ux[i,j] = -ur(r,np.pi-theta)*np.cos(theta)-utheta(r,np.pi-theta)*np.sin(theta)
                Vy[i,j] = ur(r,np.pi-theta)*np.sin(theta)-utheta(r,np.pi-theta)*np.cos(theta)
            elif x1[i] < 0 and x2[j] > 0:
                # quadrant inférieur droit
                Ux[i,j] = ur(r,theta)*np.cos(theta)-utheta(r,theta)*np.sin(theta)
                Vy[i,j] = ur(r,theta)*np.sin(theta)+utheta(r,theta)*np.cos(theta)
        j += 1
    i += 1

''' Tracé des lignes de courant '''
X1,X2 = np.meshgrid(x1,x2)
s_p_init = [(a*np.cos(i),a*np.sin(i)) for i in np.linspace(0,2*np.pi,36+1)]
s_p = s_p_init[:8]+s_p_init[11:25]+s_p_init[29:-1] # sélection des points de départ des lignes de courant
cylinder = plt.Circle((0, 0),a,ec='k',fc='w',lw=3,zorder=10) # définition du cylindre

fig = plt.figure(figsize=(10,10))
ax = fig.gca()
ax.streamplot(X1,X2,Ux,Vy,color='k',density=10,start_points=s_p)
ax.add_patch(cylinder)
ax.set_xlim(-2.5*a,2.5*a)
ax.set_ylim(-2.5*a,2.5*a)
ax.set_xlabel('$x (m)$')
ax.set_ylabel('$y (m)$')
ax.set_title('Vélocité - Lignes de courant')
plt.show()
```

Le potentiel de déplacement fluide $\phi$ étant déterminé, on évalue la distribution de la pression $p$ avec l'équation d'Euler :
\begin{equation*}
\rho\frac{\partial\phi}{\partial t} + \rho\frac{1}{2}\left(\overrightarrow{\nabla}\phi\right)^2+p=0
\end{equation*}

```python
# on rédéfinit symboliquement ces variables car elles ont été utilisées précédemment
a,r,theta,rho,U,U_point = symbols('a r theta rho U U_point')
phi = -a**2/r*U*sp.cos(theta)
dphi_dt = -a**2/r*U_point*sp.cos(theta)

grad_phi = np.array([[diff(phi,r)],[1/r*diff(phi,theta)]]) # définition du gradient de phi
p = sp.simplify(-(rho*dphi_dt+rho*1/2*np.dot(grad_phi.T,grad_phi)))
p = p[0,0]
print('p = ',p)
```

    p =  -U**2*a**4*rho/(2*r**4) + U_point*a**2*rho*cos(theta)/r

On calcule la force exercée par le fluide sur le cylindre en intégrant la pression sur la surface du cylindre :
\begin{equation*}
F_x = \int\limits^{2\pi}_0 pa\cos{\theta}d\theta \textrm{, en }r=a
\end{equation*}

```python
Fx = sp.integrate(p*a*sp.cos(theta),(theta,0,2*sp.pi))
Fx = sp.nsimplify(Fx,tolerance=1e-10).subs(r,a)
print('Fx = ',Fx)
```

    Fx =  pi*U_point*a**2*rho

Ce qui nous permet de déterminer la masse ajoutée du cylindre par unité de longueur $m_f$ avec $F_x=m_f\dot{U}$.

```python
mf = Fx/U_point
print('Expression de mf : ',mf)

mf = mf.subs([(sp.pi,np.pi),(a,0.02),(rho,1000)]) # on remplace les variables par leur valeur, vous pouvez les modifier
print('mf = ',mf)
```

    Expression de mf :  pi*a**2*rho
    mf =  1.25663706143592

$A_{xx}$ (respectivement $A_{yy}$) est la contribution du mouvement du cylindre dans la direction $x$ (respectivement $y$) à la masse ajoutée dans la direction $x$ (respectivement $y$). Les deux sont égales pour le cylindre circulaire et c'est leur valeur que l'on vient de calculer.
Pour le cas du cylindre circulaire, la contribution du mouvement dans la direction $y$ à la
masse ajoutée dans la direction $x$ $A_{xy}$ est nul.

Il nous est possible de calculer ces termes pour d'autres formes, dont les expressions ont été évaluées par Naudascher et Rockwell (1994).

Les formes suivantes sont centrées en $(0,0)$, c'est pour cela que chaque paramètre demandé est la moitié de la longueur habituellement utilisée. De plus, pour certaines formes, les ratios entre paramètres sont limités, car nous utilisons les résultats (parfois expérimentaux) fournis par Naudascher et Rockwell (1994). Vous êtes libres de modifier les paramètres pour calculer la masse ajoutée associée.

## Masse ajoutée d'une ellipse

```python
a = 0.02   # demi grand-axe (en m)
b = 0.01   # demi petit-axe (en m)
rho = 1000 # masse volumique du fluide (en kg/m3)

print('Axx = ',rho*np.pi*b**2)
print('Ayy = ',rho*np.pi*a**2)
print('Axy = ',rho*np.pi/8*(a**2-b**2)**2)
```

    Axx =  0.3141592653589793
    Ayy =  1.2566370614359172
    Axy =  3.534291735288518e-05

## Masse ajoutée d'un rectangle

```python
a = 0.02   # demie largeur (en m)
a_b = 0.2  # ratio de la demie largeur sur la demie hauteur (en m) - à choisir parmi 0.1,0.2,0.5,1.0,2.0,5.0
rho = 1000 # masse volumique du fluide (en kg/m3)

b = a/a_b
if a_b == 0.1:
    alpha = 2.23
    alpha2 = 0.147
elif a_b == 0.2:
    alpha = 1.98
    alpha2 = 0.15
elif a_b == 0.5:
    alpha = 1.7
    alpha2 = 0.15
elif a_b == 1:
    alpha = 1.51
    alpha2 = 0.234
elif a_b == 2:
    alpha = 1.36
    alpha1 = 0.15
    alpha2 = 0
elif a_b == 5:
    alpha = 1.21
    alpha1 = 0.15
    alpha2 = 0

print('Ayy = ',alpha*rho*np.pi*a**2)
if alpha2 != 0:
    print('Axy = ',alpha2*rho*np.pi*a**4)
else:
    print('Axy = ',alpha1*rho*np.pi*b**4)
```

    Ayy =  2.4881413816431164
    Axy =  7.539822368615503e-05

## Masse ajoutée d'une plaque

L'épaisseur de la plaque est considérée négligeable devant sa demie largeur.

```python
a = 0.02   # demie largeur de la plaque (en m)
rho = 1000 # masse volumique du fluide (en kg/m3)

print('Axx = 0')
print('Ayy = ',rho*np.pi*a**2)
print('Axy = ',rho*np.pi/8*a**4)
```

    Axx = 0
    Ayy =  1.2566370614359172
    Axy =  6.283185307179586e-05

## Masse ajoutée d'un losange

```python
a = 0.02   # demie largeur (en m)
a_b = 0.2  # ratio de la demie largeur sur la demie hauteur (en m) - à choisir parmi 0.2,0.5,1.0,2.0
rho = 1000 # masse volumique du fluide (en kg/m3)

b = a/a_b
if a_b == 0.2:
    alpha = 0.61
elif a_b == 0.5:
    alpha = 0.67
elif a_b == 1:
    alpha = 0.76
elif a_b == 2.0:
    alpha = 0.85

print('Ayy = ',alpha*rho*np.pi*a**2)
```

    Ayy =  0.7665486074759096

## Masse ajoutée d'une poutre en I

```python
a = 0.02   # demie largeur de la poutre (en m)
rho = 1000 # masse volumique du fluide (en kg/m3)

# tiré des expériences de Naudascher et Rockwell
d = a/2.6
b = d/3.6

print('Ayy = ',2.11*rho*np.pi*a**2)
```

    Ayy =  2.6515041996297852

## Masse ajoutée d'une poutre en croix

```python
a = 0.02   # taille d'une branche (en m)
rho = 1000 # masse volumique du fluide (en kg/m3)

print('Axx = ', rho*np.pi*a**2)
print('Axy = ',rho*2/np.pi*a**2)
```

    Axx =  1.2566370614359172
    Axy =  0.25464790894703254