# Problem 2: The AS-AD Model

Consider the following **AS-AD model**. The **goods market equilibrium** is given by

$$ y_{t} = -\alpha r_{t} + v_{t} $$

where $y_{t}$ is the **output gap**, $r_{t}$ is the **ex ante real interest** and $v_{t}$ is a **demand disturbance**.

The central bank's **Taylor rule** is

$$ i_{t} = \pi_{t+1}^{e} + h \pi_{t} + b y_{t}$$

where $i_{t}$ is the **nominal interest rate**, $\pi_{t}$ is the **inflation gap**, and $\pi_{t+1}^{e}$ is the **expected inflation gap**.

The **exante real interest rate** is given by

$$ r_{t} = i_{t} - \pi_{t+1}^{e} $$

Together, the above implies that the **AD-curve** is

$$ \pi_{t} = \frac{1}{h\alpha}\left[v_{t} - (1+b\alpha)y_{t}\right]$$

Further, assume that the **short-run supply curve (SRAS)** is given by

$$ \pi_{t} = \pi_{t}^{e} + \gamma y_{t} + s_{t}$$

where $s_t$ is a **supply disturbance**.

**Inflation expectations are adaptive** and given by

$$ \pi_{t}^{e} = \phi\pi_{t-1}^{e} + (1-\phi)\pi_{t-1}$$

Together, this implies that the **SRAS-curve** can also be written as

$$ \pi_{t} = \pi_{t-1} + \gamma y_{t} - \phi\gamma y_{t-1} + s_{t} - \phi s_{t-1} $$

The **parameters** of the model are:

```python
par = {}

par['alpha'] = 5.76
par['h'] = 0.5
par['b'] = 0.5
par['phi'] = 0
par['gamma'] = 0.075
```

```python
import numpy as np
from scipy import linalg
from scipy import optimize
import sympy as sm
import matplotlib.pyplot as plt
import ipywidgets as widgets
```

## Question 1
Use the ``sympy`` module to solve for the equilibrium values of output, $y_t$, and inflation, $\pi_t$, (where AD = SRAS) given the parameters ($\alpha$, $h$, $b$, $\alpha$, $\gamma$) and $y_{t-1}$ , $\pi_{t-1}$, $v_t$, $s_t$, and $s_{t-1}$.

```python
# Defining the AD and the SRAS functions

# i. Defining symbols
sm.init_printing(use_unicode=True)
pit = sm.symbols('pi_t')
pit1 = sm.symbols('pi_t-1')
alpha = sm.symbols('alpha')
gamma = sm.symbols('gamma')
phi = sm.symbols('phi')
h = sm.symbols('h')
b = sm.symbols('b')
vt = sm.symbols('v_t')
yt = sm.symbols('y_t')
yt1 = sm.symbols('y_t-1')
st = sm.symbols('s_t')
st1 = sm.symbols('s_t-1')

# ii. AD-curve
AD = sm.Eq(1/(h*alpha)*(vt-(1+b*alpha)*yt), pit)

# iii. SRAS-curve
SRAS = sm.Eq(pit1 + gamma*yt - phi*gamma*yt1 + st - phi*st1, pit)

```

```python
# Solving for y

# i. Setting AD equal to SRAS
AD_SRAS = sm.Eq(1/(h*alpha)*(vt-(1+b*alpha)*yt), pit1 + gamma*yt - phi*gamma*yt1 + st - phi*st1)

# ii. Solving for y
yt_star = sm.solve(AD_SRAS, yt)

# iii. Printing y*
print(f'The equilibrium value of y is:')
yt_star
```

```python
# Solving for pi

# i. Inserting y* in the AD function
pit_star = 1/(h*alpha)*(vt-(1+b*alpha)*yt_star[0])

# ii. Printing pi*
print(f'The equilibrium value of pi is:')
pit_star
```

## Question 2
Find and illustrate the equilibrium when $y_{t-1} = \pi_{t-1} = v_t = s_t = s_{t-1} = 0$. Illustrate how the equilibrium changes when instead $v_t = 0.1$.

```python
# Defining variable values
yt1_int = 0
pit1_int = 0
vt_int = 0
st_int = 0
st1_int = 0

vt_one = 0.1

## Solving numerically for y

# a. Defining a lambda function of yt_eq
_sol_y_func = sm.lambdify((h,b,alpha,gamma,phi,vt,st,pit1,yt1,st1),yt_star[0])

# b. Defining a function which solve for y, given values and parameters
def sol_y_func(vt,st,pit1,yt1,st1,h=par['h'],alpha=par['alpha'],b=par['b'],gamma=par['gamma'],phi=par['phi']):
    return _sol_y_func(h,b,alpha,gamma,phi,vt,st,pit1,yt1,st1)

# c. Solving for y and printing the solution
y_star = sol_y_func(vt=vt_int,st=st_int,pit1=pit1_int,yt1=yt1_int,st1=st1_int)
print(f'The equilibrium value of y, given variable values and parameters is: {y_star}')

##  Solving for pi

# a. Defining a lambda function of yt_eq
_sol_pi_func = sm.lambdify((h,b,alpha,gamma,phi,yt,vt,st,pit1,yt1,st1),pit_star)

# b. Defining a function which solve for pi, given values and parameters
def sol_pi_func(vt,st,yt,pit1,yt1,st1,h=par['h'],alpha=par['alpha'],b=par['b'],gamma=par['gamma'],phi=par['phi']):
    return _sol_pi_func(h,b,alpha,gamma,phi,yt,vt,st,pit1,yt1,st1)

# c. Solving for pi, using the value of y_star, and printing the solution
pi_star = sol_pi_func(vt=vt_int,st=st_int,yt=y_star,pit1=pit1_int,yt1=yt1_int,st1=st1_int)
print(f'The equilibrium value of pi, given variable values and parameters is: {pi_star}')

```

    The equilibrium value of y, given variable values and parameters is: 0.0
    The equilibrium value of pi, given variable values and parameters is: 0.0

```python
# Illustration

# Define AD and SRAS curves as functions
def AD_curve(h,b,alpha,vt,yt):
    return 1/(h*alpha)*(vt-(1+b*alpha)*yt)

def SRAS_curve(gamma,phi,pit1,yt,yt1,st,st1):
    return pit1+gamma*yt-phi*gamma*yt1+st-phi*st1
```

```python
# Constructing graph

def harry(vt_val):
    # a. Finding equilibrium given vt
    y_opt = sol_y_func(vt=vt_val,st=st_int,pit1=pit1_int,yt1=yt1_int,st1=st1_int)
    pi_opt = sol_pi_func(vt=vt_val,st=st_int,yt=y_star,pit1=pit1_int,yt1=yt1_int,st1=st1_int)

    # b. Constructing curves
    y = np.linspace(-0.1,0.1,100)
    AD = AD_curve(h=par['h'],b=par['b'],alpha=par['alpha'],vt=vt_val,yt=y)
    SRAS = SRAS_curve(gamma=par['gamma'],phi=par['phi'],pit1=pit1_int,yt1=yt1_int,st=st_int,st1=st1_int,yt=y)

    # c. Generating graph
    plt.plot(y,AD,label='AD')
    plt.plot(y,SRAS,label='SRAS')
    plt.grid(True)

    plt.title('AD and SRAS')
    plt.xlabel('$y_t$')
    plt.ylabel('$\pi_t$')

    # c. Add lines which show optimal values of y and pi
    plt.axvline(y_opt, linestyle='dashed', color="pink", label='$y^*$')
    plt.axhline(pi_opt, linestyle='dashed', color="blue", label='$\pi^*$')

    plt.legend()

# Interactive graph
def ron():
    widgets.interact(harry,
                     vt_val=widgets.IntSlider(
                         value=0,
                         description='Value of $v_t$',
                         min=0,
                         max=0.1,
                         step=0.01,
                         disabled=False,
                         continuous_update=False));

# ron()

def hermione():
    widgets.interact(harry,
                    vt_val=widgets.Dropdown(
                    description='Value of $v_t$',
                    options=[0,0.1],
                    value=0),);

hermione();

```

    interactive(children=(Dropdown(description='Value of $v_t$', options=(0, 0.1), value=0), Output()), _dom_class…

# NOTE: CHECK RON FUNKTIONEN - SLIDEREN VIRKER IKKE

Bare hvis vi vil bruge slideren. Alternativt kunne vi lave en Checkbox

**Persistent disturbances:** Now, additionaly, assume that both the demand and the supply disturbances are AR(1) processes

$$ v_{t} = \delta v_{t-1} + x_{t} $$
$$ s_{t} = \omega s_{t-1} + c_{t} $$

where $x_{t}$ is a **demand shock**, and $c_t$ is a **supply shock**. The **autoregressive parameters** are:

```python
par['delta'] = 0.80
par['omega'] = 0.15
```

## Question 3:
Starting from $y_{-1} = \pi_{-1} = s_{-1} = 0$, how does the economy evolve for $x_0 = 0.1$, $x_t = 0, \forall t > 0$ and $c_t = 0, \forall t \geq 0$?

```python
# Constructing AR(1) processes

# Defining intial values
y1 = 0
pi_init = 0
s_init = 0

v_init = 0
x0 = 0.1
xt = 0
ct = 0

# Number of variable values
num = 1000

# Creating list, contraining x number values of vt
vt = []
vt_num = par['delta'] * v_init + x0
vt.append(vt_num)
def v_func(number=num):
    """
    Function which creates list of vt values
    Args:
        number = Number of values in list
    """
    for i in range(0,number):
        vt_i = par['delta']*vt[i] + xt
        vt.append(vt_i)

v_func();

# Creating list, contraining x number values of st
st = []
st_num = par['omega']* s_init + ct
st.append(st_num)
def s_func(number=num):
    """
    Function which creates list of st values
    Args:
        number = Number of values in list
    """
    for i in range(0,number):
        st_i = par['omega']*st[i] + ct
        st.append(st_i)

s_func();
```

```python

```

```python
# Constructing list of y* and pi*

y_stars = [0]
pi_stars = [0]
AD = []
SRAS = []

def y_solve(number=num):
    for i in range(1,num):
        # y
        y_star_i = sol_y_func(vt=vt[i],st=st[i],pit1=pi_stars[i-1],yt1=y_stars[i-1],st1=st[i-1])
        y_stars.append(y_star_i)
        # Pi
        pi_star_i = sol_pi_func(vt=vt[i],st=st[i],yt=y_stars[i],pit1=pi_stars[i-1],yt1=y_stars[i-1],st1=st[i-1])
        pi_stars.append(pi_star_i)

y_solve();

```

```python
# Illustrating the evolution of the economy

x = np.linspace(0,1,1000)
y_value = y_stars
pi_value = pi_stars

# Constructing graph
f, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(10,5))
ax1.plot(x,pi_value, label='$\pi$')
ax1.set_title('Values of $\pi$')
# ax1.ylabel('$\pi_t$')
ax1.grid(True)
ax1.legend()
ax2.plot(x,y_value, label="$y$")
ax2.set_title('Values of y')
# ax2.ylabel('$y_t$')
ax2.grid(True)
ax2.legend()

plt.show();

```

```python
# Illustrating the evolution of the economy
x = y_stars
y = pi_stars

plt.plot(x,y)
plt.grid(True)

plt.title("Evolution of the Economy with AR(1) processes")
plt.xlabel('$y_t$')
plt.ylabel('$\pi_t$')

plt.show();
```

```python
# Illustrating moves in AD and SRAS curves

# a. Shocks and Convergence in AD and SRAS curves
def AD_shock(yt,period, h=par['h'],b=par['b'],alpha=par['alpha'],vt=vt):
    """Function which generates AD curves following a shock
    Args:
        h: Parameter
        b: Parameter
        alpha: Parameter
        vt: Vector of shocks
        yt: Vector of y values
        period: Number of periods after initial shock
    """
    if period == 0:
        return 1/(h*alpha)*(0-(1+b*alpha)*yt)
    else:
        return 1/(h*alpha)*(vt[period-1]-(1+b*alpha)*yt)

def SRAS_shock(yt,period,gamma=par['gamma'],phi=par['phi'],st=st):
    """Function which generates SRAS curves following a shock
    Args:
        gamma: Parameter
        phi: Parameter
        pi: Vector of pi values
        yt: Vector of y values
        st: Vector of shocks
        period: Number of periods after initial shock
        """
    if period == 0:
        return pi_init + gamma*yt - phi*y1 + st[period] - phi*s_init
    else:
        y_values = [0]
        pi_values = [0]
        for i in range(1,num):
            y_val_i = sol_y_func(vt=vt[i],st=st[i],pit1=pi_stars[i-1],yt1=y_stars[i-1],st1=st[i-1])
            y_values.append(y_val_i)
            pi_val_i = sol_pi_func(vt=vt[i],st=st[i],yt=y_stars[i],pit1=pi_stars[i-1],yt1=y_stars[i-1],st1=st[i-1])
            pi_values.append(pi_val_i)
        return pi_values[period-1] + gamma*yt - phi*gamma*y_values[period-1] + st[period] - phi*st[period-1]

# b. Constructing graph
axis_font = {'fontname':'Arial', 'size':'14'}

def sherlock(vt,st,ymax):
    """Constructs figure of convergence in the economy following a shock
    Args:
        vt: Vector of shocks
        st: Vector of shocks
        ymax: Maxmium value of y. Minimum value of y will become -ymax."""
    # i. Constructing curves
    y = np.linspace(-ymax,ymax,100)
    AD_zero = AD_shock(y,0,vt=vt)
    AD_i = AD_shock(y,1,vt=vt)
    AD_v = AD_shock(y,5,vt=vt)
    AD_x = AD_shock(y,10,vt=vt)
    AD_c = AD_shock(y,100,vt=vt)
    AD_m = AD_shock(y,1000,vt=vt)

    SRAS_zero = SRAS_shock(y,0,st=st)
    SRAS_i = SRAS_shock(y,1,st=st)
    SRAS_v = SRAS_shock(y,5,st=st)
    SRAS_x = SRAS_shock(y,10,st=st)
    SRAS_c = SRAS_shock(y,100,st=st)

    # ii. Graphs
    # (a) AD curves
    plt.figure(figsize=(20,10))
    plt.plot(y,AD_zero,label=('AD'),linewidth=5,color='blue')
    plt.plot(y,AD_i,label='$AD_1$',color='darkblue')
    plt.plot(y,AD_v,label='$AD_5$',color='navy')
    plt.plot(y,AD_x,label='$AD_{10}$',color='indigo')
    plt.plot(y,AD_c,label='$AD_{100}$',color='aqua')

    # (b) SRAS curves
    plt.plot(y,SRAS_zero,label='SRAS',linewidth=5,color='orange')
    plt.plot(y,SRAS_i,label='$SRAS_1$',color='orangered')
    plt.plot(y,SRAS_v,label='$SRAS_5$',color='red')
    plt.plot(y,SRAS_x,label='$SRAS_{10}$',color='orchid')
    plt.plot(y,SRAS_c,label='$SRAS_{100}$',color='maroon')

    # (c) Graph properties
    plt.grid(True)
    plt.title('AD and SRAS')
    plt.xlabel('$y_t$')
    plt.ylabel('$\pi_t$')

    # d. Add lines which show optimal values of y and pi
    plt.axvline(y_star, linestyle='dashed', label='$y^*$')
    plt.axhline(pi_star, linestyle='dashed', label='$\pi^*$')

    plt.legend()

sherlock(vt,st,0.05)

```

**Stochastic shocks:** Now, additionally, assume that $x_t$ and $c_t$ are stochastic and normally distributed

$$ x_{t}\sim\mathcal{N}(0,\sigma_{x}^{2}) $$
$$ c_{t}\sim\mathcal{N}(0,\sigma_{c}^{2}) $$

The **standard deviations of the shocks** are:

```python
par['sigma_x'] = 3.492
par['sigma_c'] = 0.2
```

## Question 4
Simulate the AS-AD model for 1,000 periods. Calculate the following five statistics:

1. Variance of $y_t$, $var(y_t)$
2. Variance of $\pi_t$, $var(\pi_t)$
3. Correlation between $y_t$ and $\pi_t$, $corr(y_t,\pi_t)$
4. Auto-correlation between $y_t$ and $y_{t-1}$, $corr(y_t,y_{t-1})$
5. Auto-correlation between $\pi_t$ and $\pi_{t-1}$, $corr(\pi_t,\pi_{t-1})$

```python
seed = 11
np.random.seed(seed)

# Constructing x and c
x = np.random.normal(0,par['sigma_x'],1000)
c = np.random.normal(0,par['sigma_c'],1000)

#print(x)
```

```python
# New vt and st vecotr

# a. Number of variable values
num = 1000

# b. Creating list, contraining x number values of vt
vt_new = [0]
def v_func(number=num):
    """
    Function which creates list of vt values
    Args:
        number = Number of values in list
    """
    for i in range(1,number):
        vt_i = par['delta']*vt[i-1] + x[i]
        vt_new.append(vt_i)

v_func();

# c. Creating list, contraining x number values of st
st_new = [0]
def s_func(number=num):
    """
    Function which creates list of st values
    Args:
        number = Number of values in list
    """
    for i in range(1,number):
        st_i = par['omega']*st[i-1] + c[i]
        st_new.append(st_i)

s_func();
```

```python
# Illustrating AD and SRAS curves with new shocks

sherlock(vt_new,st_new, 1)
```

# Tilføj Equilibrium Path

## Question 6

```python

# Generating function to make the demand chok vector
def v_func(num,pars):
    """
    Function which creates list of vt values
    Args:
        number = Number of values in list
    """
    vt_sim = np.zeros(num)
    x = np.random.normal(0,pars[0],num)

    for i in range(1,num):
        vt_i = par['delta']*vt_sim[i-1] + x[i]
        vt_sim[i] += vt_i

    return vt_sim

# Generating function to make the supply chok vector
def s_func(num,pars):
    """
    Function which creates list of st values
    Args:
        number = Number of values in list
    """

    st_sim = np.zeros(num)
    c = np.random.normal(0,pars[1],num)

    for i in range(1,num):
        st_i = par['omega']*st_sim[i-1] + c[i]
        st_sim[i] += st_i

    return st_sim
```

```python
# Defining function that return the vectors
def sim(num, pars):
    y_sim = np.zeros(num)
    pi_sim = np.zeros(num)

    vt_sim = v_func(num,pars)
    st_sim = s_func(num,pars)

    for i in range(1,num):
        y_val_i = sol_y_func(vt=vt_sim[i],st=st_sim[i],pit1=pi_sim[i-1],yt1=y_sim[i-1],st1=st_sim[i-1],phi=pars[2])
        y_sim[i] += y_val_i
        pi_val_i = sol_pi_func(vt=vt_sim[i],st=st_sim[i],yt=y_sim[i],pit1=pi_sim[i-1],yt1=y_sim[i-1],st1=st_sim[i-1],phi=pars[2])
        pi_sim[i] += pi_val_i

    return y_sim, pi_sim

def var_corr_acorr(num, pars):
    y_sim, pi_sim = sim(num, pars)

    var_y = np.var(y_sim)
    var_pi = np.var(pi_sim)
    corr_ypi = np.corrcoef(y_sim,pi_sim)[0,1]
    acorr_y = np.corrcoef(y_sim[:-1],y_sim[1:])[0,1]
    acorr_pi = np.corrcoef(pi_sim[:-1],pi_sim[1:])[0,1]

    return var_y, var_pi, corr_ypi, acorr_y, acorr_pi

print(var_corr_acorr(num, [par['sigma_x'], par['sigma_c'],par['phi']]))
print(var_corr_acorr(num, [3.12396213, 0.21927556, 0.95083244]))
```

    (1.8036333683263819, 1.200728375153797, -0.08262159163119677, 0.757523287304832, 0.9797171641466312)
    (1.572196814865202, 0.06404872139836039, 0.33157562066036683, 0.7866832229024381, 0.4057389389015989)

```python
def us_cycle(num):

    def _obj(num,pars):
        seed = 11
        np.random.seed(seed)

        var_y, var_pi, corr_ypi, acorr_y, acorr_pi = var_corr_acorr(num, pars)

        test = np.zeros(5)

        test[0] += (var_y-1.64)**2
        test[1] += (var_pi-0.21)**2
        test[2] += (corr_ypi-0.31)**2
        test[3] += (acorr_y-0.84)**2
        test[4] += (acorr_pi-0.48)**2

        return  np.sum(test)

    obj = lambda x: _obj(num,x)

    print(obj([par['sigma_x'], par['sigma_c'],par['phi']]))

    x0 = [1,1,par['phi']]
    bounds = ((0.0001,10000000), (0.0001,10000000), (0,1))

    result = optimize.minimize(obj,x0,
                           method='SLSQP',
                           bounds=bounds
                           ),

    return result

seed = 11
np.random.seed(seed)

result = us_cycle(num)

print(result)

```

    1.4189899590594868
    (     fun: 0.06797191833489287
         jac: array([-0.00048065,  0.001064  , -0.14804597])
     message: 'Optimization terminated successfully.'
        nfev: 212
         nit: 37
        njev: 36
      status: 0
     success: True
           x: array([3.11069136, 0.18672897, 1.        ]),)

```python
num = 1000
seed = 11
np.random.seed(seed)

moment_fun = lambda y_sim, pi_sim: np.array([np.var(y_sim),np.var(pi_sim),
                                                 np.corrcoef(y_sim,pi_sim)[0,1],
                                                 np.corrcoef(y_sim[:-1],y_sim[1:])[0,1],
                                                 np.corrcoef(pi_sim[:-1],pi_sim[1:])[0,1]])

moment_data = np.array([1.64,0.21,0.31,0.84,0.48])

def obj(theta,moment_fun,moment_data):

    seed = 11
    np.random.seed(seed)

    x = np.random.normal(0,theta[0],1000)
    c = np.random.normal(0,theta[1],1000)

    vt_friends = [0]
    st_friends = [0]

    for i in range(1,num):
        vt_i = par['delta']*vt_friends[i-1] + x[i]
        vt_friends.append(vt_i)
    for i in range(1,num):
        st_i = par['omega']*st_friends[i-1] + c[i]
        st_friends.append(st_i)

    y_friends = [0]
    pi_friends = [0]

    for i in range(1,num):
        y_s = sol_y_func(vt=vt_friends[i],st=st_friends[i],pit1=pi_friends[i-1],yt1=y_friends[i-1],st1=st_friends[i-1],phi=theta[2])
        y_friends.append(y_s)
        pi_s = sol_pi_func(vt=vt_friends[i],st=st_friends[i],yt=y_friends[i],pit1=pi_friends[i-1],yt1=y_friends[i-1],st1=st_friends[i-1],phi=theta[2])
        pi_friends.append(pi_s)

    dist = moment_fun(y_friends,pi_friends)-moment_data
    return dist@dist

theta0 = [par['sigma_x']-0.368, par['sigma_c']+0.017,par['phi']+0.953]
theta1 = [par['sigma_x'],par['sigma_c'],par['phi']]
print(obj(theta0,moment_fun,moment_data))

obj_fun = lambda theta: obj(theta,moment_fun,moment_data)
bounds = ((0.0001,10000000), (0.0001,10000000), (0,1))

np.random.seed(seed)

print(obj_fun(theta0))

result = optimize.minimize(obj_fun,theta1,
                           method='SLSQP',
                           bounds=bounds,
                            #options={"maxiter":50000},
                           ),

print(result)

# a. Number of variable values
num = 1000

```

    0.022655219849427314
    0.022655219849427314
    (     fun: 0.06797194596486839
         jac: array([ 0.00044292, -0.00143753, -0.14896686])
     message: 'Optimization terminated successfully.'
        nfev: 42
         nit: 8
        njev: 8
      status: 0
     success: True
           x: array([3.11106962, 0.18654192, 1.        ]),)

```python
y_var_val = 1.64
pi_var_val = 0.21
y_pi_corr_val = 0.31
y_corr_val = 0.84
pi_corr_val = 0.48

def chandler(gunter):
    seed = 11
    np.random.seed(seed)

    A,B,C = gunter
    # Sigma_x and Sigma_c
    x = np.random.normal(0,A,1000)
    c = np.random.normal(0,B,1000)

    vt_friends = [0]
    st_friends = [0]
    pi_friends = [0]
    y_friends = [0]

    # Generate vt and st as functions of x and c
    for i in range(1,num):
        vt_i = par['delta']*vt_friends[i-1] + x[i]
        vt_friends.append(vt_i)
    for i in range(1,num):
        st_i = par['omega']*st_friends[i-1] + c[i]
        st_friends.append(st_i)

    # Constructing y and pi based on vt and st
    for i in range(1,num):
        y_ross = sol_y_func(vt=vt_friends[i],st=st_friends[i],pit1=pi_friends[i-1],yt1=y_friends[i-1],st1=st_friends[i-1],phi=C)
        y_friends.append(y_ross)
        pi_ross = sol_pi_func(vt=vt_friends[i],st=st_friends[i],yt=y_friends[i],pit1=pi_friends[i-1],yt1=y_friends[i-1],st1=st_friends[i-1],phi=C)
        pi_friends.append(pi_ross)
    # Calculates variance and correlations
    y_joey = np.var(y_friends) - y_var_val
    pi_joey = np.var(pi_friends) - pi_var_val
    y_pi_joey = np.corrcoef(y_friends,pi_friends)[0,1] - y_pi_corr_val
    yy_joey = np.corrcoef(y_friends[:-1],y_friends[1:])[0,1] - y_corr_val
    pipi_joey = np.corrcoef(pi_friends[:-1],pi_friends[1:])[0,1] - pi_corr_val
    # Return sum of squared differences
    return (y_joey**2 + pi_joey**2 + y_pi_joey**2 + yy_joey**2 + pipi_joey**2)

x0 = [0.5,1,1]
bounds = ((0.00001,10000000), (0.00001,10000000), (0,1))

result = optimize.minimize(chandler,theta0,method='L-BFGS-B',bounds=bounds)
result

```

    (     fun: 0.02251745906880153
          jac: array([-1.98425725e-04,  4.57731076e-05, -8.45133979e-03])
      message: 'Optimization terminated successfully.'
         nfev: 19
          nit: 3
         njev: 3
       status: 0
      success: True
            x: array([3.12396213, 0.21927556, 0.95083244]),)

```python

```


# Velocity and Acceleration
 This notebook will build on the concept of vector position, and introduce vector velocity and Acceleration.

```python
## Imports
import numpy as np
```

```python
# Some advanced stuff just so we can plot the trajectory of the objects we are playing with
import matplotlib.pyplot as plt
%matplotlib inline
def show_vels(velocities):
    x_vals = list()
    y_vals = list()
    for i in velocities:
        x_vals.append(i[0])
        y_vals.append(i[1])
    plt.plot(x_vals, y_vals)
```

So you are likely familiar with speed. This is a measure of the distance covered over time, which you can infer from the units, miles per hour, metres per second. The step we're making when speed becomes velocity is to give it a direction. 40 miles per hour is a speed, 40 miles per hour _north_ is a velocity.

The terms used for these two kinds of value, are:
1. **_Scalar_**: Objects which only have a magnitude (size)
1. **_Vector_**: Objects which have both a magnitude and a direction

## Distance and Displacement

Calculating distance and displacement is a linked task.

if you start with two points on a chart, say the cricket and the worm from the previous task, you can find the vector displacement easily. Just subtract the positions of your origin and destination. What you are left with is a vector describing the steps in each dimension to take to get from one place to the next.

### Example 1.1
Review of the cricket and worm movement from the first session

Using these two locations as reference pieces. There are two ways to describe the position change.

1. The **Scalar**: Distance, just the number representing the straight line movement
1. The **Vector**: Displacement, the a vector representing the movement with the specifics of direction

```python
cricket = np.array([1,8,0])
worm = np.array([5,2,0])

print(cricket-worm)
print(worm-cricket)

worm2cricket = cricket-worm
```

    [-4  6  0]
    [ 4 -6  0]

### Distance travelled
If you only want the number of steps taken on a grid, then you can just add the numbers up.

#### Example 1.2

```python
path = cricket-worm
# show the steps to travel
print(path)
# get the absolute distance travelled in each direction
print(abs(path))
# Add it up
print(sum(abs(cricket-worm)))
```

    [-4  6  0]
    [4 6 0]
    10

However if you want the distance in a straight line then you need to take advantage of some tricks from pythagoras.

Pythagoras' Theorem is used to calculate the side length of right-angled triangles.

Graphically shown below:

The root of the sum of the squares of the number of steps in each direction is equivalent to the number of steps in a straight line.

That number of steps is the distance.

Or, taken another way, the side lengths *a* and *b* are the steps in the _x_ & _y_ direction, and the hypotenuse (longest side) *c* is the actual distance travelled

We can apply this in python, for when we dont want to do the calculation manually.

```python
## Application

# using the diagram above to show the results
test = np.array([3, 4, 0])
print(test)
# exploiting the numpy arrays to square all the values at the same time
print(test * test)
# Add them up to replicate the pythagoras method above and get the square of the distance travelled
print(sum(test*test))
# getting the distance travelled by square rooting the value
print(np.sqrt(sum(test*test)))

#def magnitude(an_array):
    #Some code to get the magnitude

```

    [3 4 0]
    [ 9 16  0]
    25
    5.0

Now if we were to swap the sign of one of the directions, so now instead of moving up 4 position in _y_ we instead move -4 position back. We would travel the same distance, but end up in a different place.

You can test and confirm this by using the process outlined above and see you end up with the same number.

```python
test2 = np.array([3, -4, 0])

# magnitude(test2)
```

## Change over time

Its worth then thinking about how fast that movement happens.

1. The distance is 2 metres, and it happens in one second, then they have moved at 2 metres per second.

2. When moving half the distance, 1 metre per second:  1 metre in one second

3. When taking twice as long, also 1 metre per second: 2 metres in two seconds

To help demonstrate that the last two are identical, if you repeat the step in 2., what has happened.

## Perpendicular Non-interference

This is a fancy name for the idea that the different components of a vector dont interact with one another. As a basic example, if you have a vector
\begin{equation}
\begin{pmatrix} 1 & 0  & 1 \end{pmatrix}
\end{equation}

and a vector
\begin{equation}
\begin{pmatrix} 0 & 2 & 0 \end{pmatrix}
\end{equation}

and you were to add/subtract them. The components would never interact.

\begin{equation}
 \begin{pmatrix} 1 & 0  & 1 \end{pmatrix} + \begin{pmatrix}0  & 2 & 0 \end{pmatrix}  =  \begin{pmatrix} 1 & 2& 1\end{pmatrix}
\end{equation}

You'd end up with something like (1, 2, 1), where the *x* and *z* component of the first vector are unchanged, and the *y* component of the second is unchanged.

## Applications
This concept of non-interference simplifies a lot of things when dealing with a velocity.

From an initial position of (1, 1, 1)m an object travels with a velocity (1, 0, 0) m/s

After 1 second where is it?

After 3 seconds where is it?

We can answer this simple case manually, but when we start waiting for longer (or working with more complex velocities), we need to be able to instruct python to do it for us.

```python
position = np.array([1, 1, 1])
velocity = np.array([1, 0 , 0])
for i in range(1, 10):
    # Add in regular process
    print("iteration " + str(i))
    position = position
    print(position)

```

    iteration 1
    [1 1 1]
    iteration 2
    [1 1 1]
    iteration 3
    [1 1 1]
    iteration 4
    [1 1 1]
    iteration 5
    [1 1 1]
    iteration 6
    [1 1 1]
    iteration 7
    [1 1 1]
    iteration 8
    [1 1 1]
    iteration 9
    [1 1 1]

This kind of process is straight line motion. This is how things work in deep space. If you fire a pellet into deep space, it will just keep going, at the same speed, forever.

```python

```

```python

```

```python
position = np.array([1, 1, 1])
velocity = np.array([1, 0 , 0])
positions = list()
for i in range(1, 10):
    positions.append(position)
    position = position + velocity
    print(position)

show_vels(positions)
```

## Extension: Acceleration
Where the displacement has a thing called velocity which can cause it to change over time. Velocity has the same, this is called acceleration. This is the rate at which the velocity changes.

The biggest example in our lives is Gravity. This induces a constant acceleration downwards for all objects, and it's this constant downward acceleration which causes the bow shape in projectiles launched.

This also exhibits this property of the non-interference of perpendicular components. Gravity only applies in one direction, _Down_. As such only the vertical _y_ component of the velocity changes. you will see that the horizontal component doesnt change across the whole process.

```python
position = np.array([1, 1, 1])
velocity = np.array([1, 1 , 0])
acceleration = np.array([0,-0.15,0])
positions = list()
for i in range(1, 10):
    positions.append(position)
    position = position + velocity
    velocity = velocity + acceleration
show_vels(positions)
```

```python

```

```python

```


# Scikit-Learn: un API de python para el aprendizaje automatico

Scikit-Learn, libreria de python, combina una interfaz intuitiva con una implementacion altamente optimizada de varios algoritmos de clasificacion y regresion. No solo ofrece una amplia variedad de algoritmos de aprendizaje, sino tambien diferentes funciones sencillas para el preprocesamiento de datos y para la evaluacion de los modelos.

Los cinco pasos principales que se dan en el entrenamiento de un algoritmo de aprendizaje automatica se pueden resumir en:

1. Seleccionar caracteristicas y recopilar muestras de entrenamiento.
2. Elegir una medicion del rendimiento.
3. Eligir un algoritmo optimizador y de clasificacion.
4. Evaluar el rendimiento del modelo.
5. Afinar el algoritmo.

Siempre sera necesario aprender a manejar varios algoritmos de clasificacion, debido a que no todos funcionan de la misma manera ni son aptos para todos los conjuntos de datos; inclusive sus restricciones matematicas suelen ser diferentes, y por, lo tanto pueden producir resultados erroneos de no utilizarse con los datos adecuados. Mas aun, algunos de llos dependen de los llamadas _hiperparametros_ , siendo estos diferentes entre los distintos algoritmos, lo cual implica una dificultad mayor, la cual puede ser subsanada con la practica.

Para introducirnos adecuadamente en el tema, iniciaremos con la implementacion del modelo **Perceptron** de clasificacion lineal visto la semana pasada, pero esta vez implementado a traves de _scikit learn_. Primero debemos recordar y agregar lo siguiente:

1. El modelo es de clasificacion lineal, lo que significa que tratara de separar las clases por medio de lineas rectas.
2. El modelo, al ser de clasificacion, puede manejar distintas clases, y no solo dos, como se vio en la clase de la semana pasada, en la cual se manejo unicamente clasificacion binaria. Para lograr la clasificacion de mas de dos clases, utilizara un metodo llamada **OvR: One versus All**.
3. El escalado de variables, que favorecia al metodo del Descenso del gradiente usado en el modelo Adaline, tambien puede ser usado con el modelo del Perceptron a traves de la libreria scikit learn.
4. El modelo de Perceptron requiere para su convergencia que las clases sean linealmente separables, de lo contrario nunca terminar de clasificar. Para solventar esto, se introduce un numero predeterminado de iteraciones y asi asegurar su finalizacion, mas no su convergencia. Es de aclarar que el numero de iteraciones elegido no garantiza que los errores sean minimos, y en clases que no son linealmente separables, puede llegar a ser demandante encontrar este minimo. Esto representa un defecto de la funcion de activacion del Perceptron.

Scikit Learn ademas proporciona acceso a conjuntos de datos precargados, los cuales, por su popularidad, se ganaron su lugar dentro de la libreria. Otra librerias que permiten algo similar son Pandas y Numpy.

```python
# Cargando el conjunto de datos Iris
# =========================================================
from sklearn import datasets
import numpy as np

iris = datasets.load_iris()
X = iris.data[:, [2, 3]]
y = iris.target

print(f'Etiquetas de clase: {np.unique(y)}')
```

    Etiquetas de clase: [0 1 2]

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

```python
# Division del conjunto de datos en entrenamiento y test
# =========================================================
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, stratify = y)

print(f'Tamaño del conjunto X_train: {X_train.shape}')
print(f'Tamaño del conjunto X_test: {X_test.shape}')
print(f'Tamaño del conjunto y_train: {y_train.shape}')
print(f'Tamaño del conjunto y_test: {y_test.shape}')

print(f'\nConteo de etiquetas en y: {np.bincount(y)}')
print(f'Conteo de etiquetas en y_train: {np.bincount(y_train)}')
print(f'Conteo de etiquetas en y_test: {np.bincount(y_test)}')
```

    Tamaño del conjunto X_train: (105, 2)
    Tamaño del conjunto X_test: (45, 2)
    Tamaño del conjunto y_train: (105,)
    Tamaño del conjunto y_test: (45,)

    Conteo de etiquetas en y: [50 50 50]
    Conteo de etiquetas en y_train: [35 35 35]
    Conteo de etiquetas en y_test: [15 15 15]

```python
# Escalado de variables
# =========================================================
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
sc.fit(X_train) # Calculo de mu y sigma para cada caracteristica
X_train_std = sc.transform(X_train)
X_test_std = sc.transform(X_test) # X_test se escala con los mismos parametros de X_train, para que sean comparables
```

```python
# Creacion del modelo de perceptron
# =========================================================
from sklearn.linear_model import Perceptron
modelo = Perceptron(max_iter = 40, eta0 = 0.1, random_state = 1)
modelo.fit(X_train_std, y_train)
```

    Perceptron(eta0=0.1, max_iter=40, random_state=1)

```python
# Prediccion de resultados
# =========================================================
y_pred = modelo.predict(X_test_std)
print(f'Valores mal clasificados: {(y_test != y_pred).sum()}')
print(f'Cantidad de valores en y_test: {y_test.shape[0]}')
proporcion = (y_test != y_pred).sum()/y_test.shape[0]
print(f'Proporcion de valores mal clasificados: {np.round(proporcion, 3)}')
print(f'Precision de la clasificacion: {1 - np.round(proporcion, 3)}')
```

    Valores mal clasificados: 4
    Cantidad de valores en y_test: 45
    Proporcion de valores mal clasificados: 0.089
    Precision de la clasificacion: 0.911

```python
# Calculo de la precision usando accuracy_score
# =========================================================
from sklearn.metrics import accuracy_score
print(f'Precision: {np.round(accuracy_score(y_test, y_pred), 3)}')
```

    Precision: 0.911

```python
# Calculo de la precision usando score
# =========================================================
print(f'Precision: {np.round(modelo.score(X_test_std, y_test), 3)}')
```

    Precision: 0.911

```python
# Graficando las regiones
# =========================================================
from matplotlib.colors import ListedColormap
import matplotlib.pyplot as plt

def plot_decision_regions(X, y, classifier, test_idx=None, resolution=0.02):

    # setup marker generator and color map
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot the decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                           np.arange(x2_min, x2_max, resolution))
    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)
    plt.contourf(xx1, xx2, Z, alpha=0.3, cmap=cmap)
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    for idx, cl in enumerate(np.unique(y)):
        plt.scatter(x=X[y == cl, 0],
                    y=X[y == cl, 1],
                    alpha=0.8,
                    c=colors[idx],
                    marker=markers[idx],
                    label=cl,
                    edgecolor='black')

    # highlight test samples
    if test_idx:
        # plot all samples
        X_test, y_test = X[test_idx, :], y[test_idx]

        plt.scatter(X_test[:, 0],
                    X_test[:, 1],
                    c='',
                    edgecolor='black',
                    alpha=1.0,
                    linewidth=1,
                    marker='o',
                    s=100,
                    label='test set')
```

```python
%matplotlib notebook
X_combined_std = np.vstack((X_train_std, X_test_std))
y_combined = np.hstack((y_train, y_test))

plot_decision_regions(X=X_combined_std, y=y_combined,
                      classifier=modelo, test_idx=range(105, 150))
plt.xlabel('petal length [estandarizado]')
plt.ylabel('petal width [estandarizado]')
plt.legend(loc='upper left')

plt.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

    <ipython-input-11-82b7b871b197>:39: MatplotlibDeprecationWarning: Using a string of single character colors as a color sequence is deprecated. Use an explicit list instead.
      plt.scatter(X_test[:, 0],

EJERCICIO

1. Volver a usar el modelo de Perceptron con todas las caracteristicas del dataset. Calcular la precision.
2. Tomar el dataset `Social_Network_Ads.csv` y clasificar los datos de compra y no compra usando las variables 'Age' y 'EstimatedSalary'. Calcular el score y graficar para visualizar las regiones.

# Regresion Logistica

La regresion logistica es uno de los algoritmos de clasificacion mas populares, que no se usa para regresion, como su nombre pareciera indicar. Su metodo se basa en el establecimiento de probabilidades a partir de un valor de entra da de red $z$, el cual se representa de forma lineal, como en el caso del perceptron:

$$z=w_0+w_1x_1+w_2x_x+\dots+w_mx_m$$

Por lo tanto, la regresion logistica tambien es un algoritmo de clasificacion lineal, que tratara de separar las clases por un limite o frentera lineal; su ventaja sobre el modelo de Perceptron es similar a la ventaja mostrada por Adaline, la cual es la posibilidad de hallar un minimo de la funcion de costo sin depender de la convergencia del algoritmo; recordemos que Perceptron es un modelo que depende la separacion lineal de clases para su convergencia, lo cual es una limitante del modelo.

La definicion matematica de la regresion logistica es:

$$\phi(z)=\frac{1}{1+e^{-z}}$$

Para entender mejor como funciona, vamos a graficar su forma en python:

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')
%matplotlib notebook

def sigmoid(z):
    return 1./(1.+np.exp(-z))

x = np.linspace(-7, 7, 100)
y = sigmoid(x)

fig, ax = plt.subplots(figsize = (8, 5))
ax.plot(x, y, color = 'red')
plt.axhline(y = 0.5, color = 'gray', linewidth = 0.5)
plt.axhline(y = 1, color = 'gray', linewidth = 0.5)
plt.axhline(y = 0, color = 'gray', linewidth = 0.5)
plt.axvline(x = 0, color = 'k')

ax.set_xlabel('z')
ax.set_ylabel('sigmoid(z)')
ax.set_title('Comportamiento de la funcion sigmoide');
```

    <IPython.core.display.Javascript object>

Como se puede ver, a medida que x se acrca a infinito, la funcion se acerca a 1; cuando la funcion se acerca a menos infinito, la funcion se acerca a -1. La funcion se puede entender como que toma valores numero numericos en el intervalo $(-\infty, \infty)$ y los transforma al intervalo $[0, 1]$.

###### Comparacion entre Adaline y Regresion logistica

La salida de la funcion sigmoide se lee como: La probabilidad de que la muestra pertenezca a la clase 1, dados los valores $\vec{x}$ y $\vec{w}$ de entrada. En terminos mas formales, se escribe como $P(y=1|\vec{x},\vec{w})$. Por ejemplo, si $\phi(z)=0.8$, significa que la probabilidad de que la flor de entrada sea `Iris-versicolor` (etiqueta 1) es del 80%, y por lo tanto la probabilidad de que se `Iris-Setosa` es del 20%.

Lo interesante de la regresion logistica es que no solo brinda la etiqueta de clase, sino que ademas brinda la probabilidad de esta etiqueta sea la correcta; por esta razon es bastante popular en calculos meteorologicos y en medicina.

La probabilidad predicha se puede, simplemete, convertir despues en un resultado binario mediante la funcion umbral:

\begin{equation}
     \hat{y} = \left\{
	       \begin{array}{ll}
		 1      & \mathrm{si\ } \phi(z) \geq 0.5 \\
		 0 & \text{en otros casos}
	       \end{array}
	     \right.
   \end{equation}

Tambien, a partir del grafico, se puede deducir que:

\begin{equation}
     \hat{y} = \left\{
	       \begin{array}{ll}
		 1      & \mathrm{si\ } z \geq 0.0 \\
		 0 & \text{en otros casos}
	       \end{array}
	     \right.
   \end{equation}

## Aprender los pesos para la funcion logistica

Para aprender los pesos, haremos uso de la tecnica de maximizacion de la funcion de coste apropiada para la regresion logistica:

$$J(\textbf{w})=\sum_{i=1}^n \big [-y^{(i)}\log(\phi(z^{(i)}))-(1-y^{(i)})\log(1-\phi(z^{(i)})) \big]$$

Tener en cuenta que la siguiente propiedad se cumple para $J$:

\begin{equation}
     J(\phi(z), y; \textbf{w}) = \left\{
	       \begin{array}{ll}
		 -\log(\phi(z))   & \mathrm{si\ } y = 1 \\
		 -\log(1-\phi(z)) & \mathrm{si\ } y = 0
	       \end{array}
	     \right.
   \end{equation}

```python
def cost_1(z):
    return - np.log(sigmoid(z))

def cost_0(z):
    return - np.log(1 - sigmoid(z))

z = np.arange(-10, 10, 0.1)
phi_z = sigmoid(z)

fig, ax = plt.subplots(figsize = (5, 3))
c1 = [cost_1(x) for x in z]
plt.plot(phi_z, c1, label='J(w) if y=1')

c0 = [cost_0(x) for x in z]

plt.plot(phi_z, c0, linestyle='--', label='J(w) if y=0')

plt.ylim(0.0, 5.1)
plt.xlim([0, 1])
plt.xlabel('$\phi$(z)')
plt.ylabel('J(w)')
plt.legend(loc='best')
plt.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

## Implementando el algoritmo de regresion logistica

```python
class LogisticRegressionGD:
    """Logistic Regression Classifier using gradient descent.

Parametros
    ------------
    eta : float
      Rango de aprendizaje (entre 0.0 1 1.0)
    n_iter : int
      Iteracion sobre el datset de entrenamiento completo.
    random_state : int
      Semilla de numeros aleatorios

    Atributos
    -----------
    w_ : 1d-array
      Pesos de las caracteristicas.
    cost_ : list
      Suma de cuadrados de la funcion de costo en cada epoca.
    """

    def __init__(self, eta=0.05, n_iter=100, random_state=1):
        self.eta = eta
        self.n_iter = n_iter
        self.random_state = random_state

    def fit(self, X, y):
        """ Fit de datos de entrenamiento.

        Parametros
        ----------
        X : {array-like}, shape = [n_samples, n_features]
        y : array-like, shape = [n_samples]

        Returno
        -------
        self : object
        """

        rgen = np.random.RandomState(self.random_state)
        self.w_ = rgen.normal(loc=0.0, scale=0.01, size=1 + X.shape[1])
        self.cost_ = []

        for i in range(self.n_iter):
            net_input = self.net_input(X)
            output = self.activation(net_input)
            errors = (y - output)
            self.w_[1:] += self.eta * X.T.dot(errors)
            self.w_[0] += self.eta * errors.sum()

            cost = -y.dot(np.log(output)) - ((1 - y).dot(np.log(1 - output)))
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        """Calcular la entrada de red"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, z):
        """Calcular la activaciond e la funcion sigmoide"""
        return 1. / (1. + np.exp(-np.clip(z, -250, 250)))

    def predict(self, X):
        """Retornar la etiqueta de clase despues de cada paso"""
        return np.where(self.net_input(X) >= 0.0, 1, 0)

```

Es importante tener en cuenta que la regresion logistica solo sirve para clasificaciones binarias.

```python
from sklearn import datasets
import numpy as np

iris = datasets.load_iris()
X = iris.data[:, [2, 3]]
y = iris.target

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=1, stratify=y)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
sc.fit(X_train)
X_train = sc.transform(X_train)
X_test = sc.transform(X_test)
```

```python
X_train_01_subset = X_train[(y_train == 0) | (y_train == 1)]
y_train_01_subset = y_train[(y_train == 0) | (y_train == 1)]

lrgd = LogisticRegressionGD(eta=0.05, n_iter=1000, random_state=1)
lrgd.fit(X_train_01_subset, y_train_01_subset)

fig, ax = plt.subplots(figsize = (6, 4))

plot_decision_regions(X=X_train_01_subset, y=y_train_01_subset, classifier=lrgd)

plt.xlabel('petal length [normalizado]')
plt.ylabel('petal width [normalizado]')
plt.legend(loc='upper left')

plt.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

```python
y = lrgd.cost_
x = np.arange(1, len(y)+1)
fig, ax = plt.subplots()
ax.plot(x,y);
```

    <IPython.core.display.Javascript object>

EJERCICIOS

1. Realice la clasificacion de todo el dataset usando la regresion logistica. Compare los resultados con el modelo del perceptron.
2. De nuevo realice el ejericicio 2 pero con esta regresion. Verifique sus resultados y grafique.


# SVD and ICA exploration
This notebook is an attempt to understand the application of independent component analysis (ICA) on top of the output of the singular value decomposition (SVD).

The code in this notebook was written by Esten Leonardsen, and extended by me.

## 1. Synthethic data creation
To illustrate the concept, we create synthetic data which consists of three underlying sources that we are trying to find with the ICA.

```python
import math
import numpy as np
import plotly.graph_objs as go

from plotly.offline import init_notebook_mode, iplot
from plotly.subplots import make_subplots

init_notebook_mode(connected=True)

n_samples = 100
n_sources = 3 # MAX 6

sources = [
    np.arange(0, n_samples, dtype=float),
    np.sin(np.arange(0, 10, 10/n_samples)),
    np.asarray([1. if (i // (n_samples//8)) % 2 == 0 else 2. for i in range(n_samples)]),
    np.concatenate([np.arange(0, int(math.floor(n_samples // 2)), dtype=float),
                    np.arange(int(math.ceil(n_samples // 2)), 0, -1, dtype=float)]),
    np.arange(n_samples, 0, -1, dtype=float) ** 2,
    np.arange(0, 5, 5/n_samples).astype(int).astype(float)

]
sources = sources[:n_sources]
sources = np.asarray(sources)
sources = sources.T

def normalize(x):
    x -= np.amin(x)
    x /= np.amax(x)

    return x

for i in range(n_sources):
    sources[:,i] = normalize(sources[:, i])

traces = [
    go.Scatter(
        x=np.arange(n_samples),
        y=sources[:,i],
        mode='lines',
        name=f'Source {i}'
    )
for i in range(n_sources)]

layout = go.Layout(
    title = {
        'x': 0.5,
        'text': 'Original sources'
    }
)

iplot(go.Figure(traces, layout))
```

## 2. Place the signal in the the columns of our data matrix.

Create signals that are a linear combination of the three sources given above.

```python
n_idps = 50
noise = 5e-2

idps = []
formulas = []

for i in range(n_idps):
    factors = np.random.uniform(low=0, high=1, size=n_sources)
    idp = np.sum([factors[i] * sources[:,i] for i in range(n_sources)], axis=0)
    idp += np.random.normal(0, noise, size=idp.shape)
    idps.append(idp)
    formulas.append(' + '.join([f'{np.round(factors[i], 1)} * x[{i}]' for i in range(n_sources)]))

idps = np.stack(idps)
idps = idps.T
print(np.shape(idps))
traces = [
    go.Scatter(
        x=np.arange(n_samples),
        y=idps[:, i],
        mode='lines',
        name=formulas[i],
        showlegend=False
    )
for i in range(n_idps)]

layout = go.Layout(
    title = {
        'x': 0.5,
        'text': 'Derived IDPs'
    }
)

iplot(go.Figure(traces, layout))
```

    (100, 50)

Our data matrix, _idps_, is now a n_samples x n_idps matrix. The linear combination of the signal is to be found in each column.

### 2.1 Apply ICA to find the underlying signal in the columns
Now, we will apply ICA onto our data matrix. The FastICA algorithm implemented in python expects the samples to be in the columns.

```python
from sklearn.decomposition import FastICA

n_components = n_sources

ica = FastICA(n_components)
components = ica.fit_transform(idps)
componentsT = ica.fit_transform(idps.T)

figure = make_subplots(2, 1, subplot_titles=['ICA (columns)','ICA (rows)'])
for i in range(n_components):
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=components[:, i],
            mode='lines',
            name=f'ICA (column) component {i}'
        ),
        row=1,
        col=1
    )
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=componentsT[:, i],
            mode='lines',
            name=f'ICA (row) component {i}'
        ),
        row=2,
        col=1
    )

iplot(figure)
```

### 2.2 Apply SVD, then ICA
Now, let us do the singular value decomposition (SVD), and then apply ICA on the output of the left and right singular vectors.

```python
from scipy import linalg

U, s, Vh = linalg.svd(idps)
V = Vh.T

sigma = np.zeros((n_samples,n_idps))
for i in range(min(n_samples,n_idps)):
    sigma[i,i] = s[i]

US = np.dot(U,sigma)
VS = np.dot(V,sigma.T)

ica = FastICA(n_components)
components_us = ica.fit_transform(US)
components_vs = ica.fit_transform(VS)

figure = make_subplots(2, 1, subplot_titles=['US+ICA','VS+ICA'])
for i in range(n_components):
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=components_us[:, i],
            mode='lines',
            name=f'US+ICA component {i}'
        ),
        row=1,
        col=1
    )
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=components_vs[:, i],
            mode='lines',
            name=f'VS+ICA component {i}'
        ),
        row=2,
        col=1
    )

iplot(figure)
```

We see that if the signal is in the columns in our data matrix, *idps* , we will retrieve it in the U matrix in the SVD, but not in the V.

## 3. Place the signal in the rows of our data matrix
Now, let us transpose our data matrix *idps*, so that the synthetic signal generated is in each row, and run the same analysis as above

```python
A = idps.T
traces = [
    go.Scatter(
        x=np.arange(n_samples),
        y=A[:, i],
        mode='lines',
        name=formulas[i],
        showlegend=False
    )
for i in range(n_idps)]

layout = go.Layout(
    title = {
        'x': 0.5,
        'text': 'Derived IDPs'
    }
)

iplot(go.Figure(traces, layout))
```

The plotted columns now look random. As the signal is along the rows. Let us try to apply ICA the same way as we did above. Now, we should expect that the ICA should not find the sources as they are not present in the columns.

### 3.1 Apply ICA on the columns

```python
from sklearn.decomposition import FastICA

n_components = n_sources

ica = FastICA(n_components)
components = ica.fit_transform(A)
componentsT = ica.fit_transform(A.T)

figure = make_subplots(2, 1, subplot_titles=['ICA (columns)','ICA (rows)'])
for i in range(n_components):
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=components[:, i],
            mode='lines',
            name=f'ICA (column) component {i}'
        ),
        row=1,
        col=1
    )
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=componentsT[:, i],
            mode='lines',
            name=f'ICA (row) component {i}'
        ),
        row=2,
        col=1
    )

iplot(figure)
```

As expected, we do not find any signal in the columns of our data matrix, as it does not exist.

### 3.2 Apply SVD, then ICA

```python
from scipy import linalg

U, s, Vh = linalg.svd(A)
V = Vh.T

sigma = np.zeros((n_samples,n_idps)).T
for i in range(n_idps):
    sigma[i,i] = s[i]

US = np.dot(U,sigma)
VS = np.dot(V,sigma.T)

ica = FastICA(n_components)
components_us = ica.fit_transform(US)
components_vs = ica.fit_transform(VS)

from plotly.subplots import make_subplots

figure = make_subplots(2, 1, subplot_titles=['US+ICA','VS+ICA'])
for i in range(n_components):
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=components_us[:, i],
            mode='lines',
            name=f'US+ICA component {i}'
        ),
        row=1,
        col=1
    )
    figure.add_trace(
        go.Scatter(
            x=np.arange(n_samples),
            y=components_vs[:, i],
            mode='lines',
            name=f'VS+ICA component {i}'
        ),
        row=2,
        col=1
    )

iplot(figure)
```

## 4. Interpretation
When doing the singular value decomposition,
\begin{equation*}
\mathbf{A} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^{T},
\end{equation*}
- where $\mathbf{A}$ is our $(mxn)$ data matrix,
- the vectors of $\mathbf{U}$ and $\mathbf{V}$ contain the left and right singular vectors respectively, of $\mathbf{A}$, in their columns
- The diagonal entries of $\mathbf{\Sigma}$ are the singular values of $\mathbf{A}$, the rest of the entries are 0.
- $\mathbf{V}$ can be said to be the principal axes of $\mathbf{A}$,
- $\mathbf{A}$ projected onto $\mathbf{V}$ gives us: $\mathbf{AV} = \mathbf{U \Sigma V^TV} = \mathbf{U \Sigma} $, which is the principal components of $\mathbf{A}$.

If we look at $\mathbf{A}^T$, we get:
\begin{align}
\mathbf{A}^T & = (\mathbf{U} \mathbf{\Sigma} \mathbf{V}^{T})^T \\
\mathbf{A}^T & = \mathbf{V}^{TT} \mathbf{\Sigma}^T \mathbf{U}^T \\
\mathbf{A}^T & = \mathbf{V} \mathbf{\Sigma}^T \mathbf{U}^T \\
\end{align}

Here,
- $\mathbf{U}$ can be said to be the principal axes of $\mathbf{A}^T$,
- $\mathbf{A}^T$ projected onto $\mathbf{U}$ gives us: $\mathbf{A^TU} = \mathbf{V \Sigma^T U^TU} = \mathbf{V \Sigma^T} $, which is the principal components of $\mathbf{A}^T$.

So, by doing ICA on
- $\mathbf{U \Sigma}$ (the principal components of $\mathbf{A}$), we look for an underlying signal in the columns of $\mathbf{A}$.
- $\mathbf{V \Sigma}^T$ (the principal components of $\mathbf{A}^T$), we look for an underlying signal in the rows of $\mathbf{A}$.

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as st
import statsmodels.api as sm
import seaborn as sb
```

* Please email me (ianzhang@connect.hku.hk) if you find any mistakes in the script.

# Markov Chain (MC)

A **Markov chain** is sequence of random variables $\{X_1,X_2,\dots\}$ satisfying the Markov property
$$
Pr(X_{n+1}\in A|X_1=x_1,\dots,X_n=x_n)=Pr(X_{n+1}\in A|X_n=x_n)
$$
for all $A\subset\Omega$.

The Markov chain $\{X_1,X_2,\dots\}$ is **time-homogeneous** if we have
$$
Pr(X_{n+1}\in A|X_n=x_n)=Pr(X_2\in A|X_1=x_1).
$$

We will see that both the classic Metropolis–Hastings (M–H) Algorithm and the classic Gibbs Sampler generate time-homogeneous chains. So we will only consider time-homogeneous chains in this tutorial, and refer to them simply by Markov chains.

## Stationary distribution

Here for simplicity, we mainly consider the finite sample space $\Omega=\{\omega_i, i=1,\dots,m\}$, in which case a distribution $\pi=(\pi_1,\dots,\pi_m)^T$ is just a vector whose sum is 1, and the conditional probability of the time-homogeneous chain $Pr(X_2=\omega_j|X_1=\omega_i)$ can be represented by a **transition matrix** $P=[p_{ij}]$, $p_{ij}=Pr(X_2=\omega_j|X_1=\omega_i)$ like the following one:

```python
P = np.array([[0.75, 0.25, 0], [0.5, 0, 0.5], [0, 0.5, 0.5]])
print(P)
```

    [[0.75 0.25 0.  ]
     [0.5  0.   0.5 ]
     [0.   0.5  0.5 ]]

We say $\pi$ is a **stationary distribution** of the transition matrix $P$ if $\pi^T=\pi^TP$, which means $\pi$ is a left eigenvector of $P$, with eigenvalue 1.

```python
w, v = np.linalg.eig(P.T)
print(w)
print(v)

pi = v[:, 1] / v[:, 1].sum()
print(np.round(pi, 4), np.round(pi.dot(P), 4))
```

    [-0.3903882  1.         0.6403882]
    [[-0.35708982 -0.81649658 -0.77669562]
     [ 0.81444203 -0.40824829  0.17027001]
     [-0.45735221 -0.40824829  0.60642562]]
    [0.5  0.25 0.25] [0.5  0.25 0.25]

If the Markov chain is **irreducible** and **aperiodic**, then there is an unique stationary distribution $\pi$, and there is $P^n\to\mathbf{1}\pi^T$. This is stated by the [Perron–Frobenius theorem](https://en.wikipedia.org/wiki/Perron%E2%80%93Frobenius_theorem). Actually, there is a [weak form of the Perron-Frobenius Theorem](https://sv.20file.org/up1/1179_0.pdf) guarantees the existence of the stationary distribution almost without any conditions. The uniqueness of $\pi$ holds when $P$ is irreducible whether it is periodic or not (check it [here](https://services.math.duke.edu/~rtd/EOSP/EOSP2E.pdf)). The aperiodic only plays a role in $P^n\to\mathbf{1}\pi^T$.

```python
print(np.linalg.matrix_power(P, 10000))
```

    [[0.5  0.25 0.25]
     [0.5  0.25 0.25]
     [0.5  0.25 0.25]]

* Please check the definitions on the lecture note to construct transition matrices that are reducible or periodic respectively, check their theoretical properties.

```python
# reducible
P1 = np.array([[0.5, 0.5, 0], [0.5, 0.5, 0], [0, 0, 1]])
print(P1)

print('eigenvalues and eigenvectors:')
w1, v1 = np.linalg.eig(P1.T)
print(w1)
print(v1)

pi1 = v1[:, 0] / v1[:, 0].sum()
print(np.round(pi1, 4), np.round(pi1.dot(P1), 4))
pi1 = v1[:, 2] / v1[:, 2].sum()
print(np.round(pi1, 4), np.round(pi1.dot(P1), 4))

print('power of matrix:')
print(np.linalg.matrix_power(P1, 10000))
```

    [[0.5 0.5 0. ]
     [0.5 0.5 0. ]
     [0.  0.  1. ]]
    eigenvalues and eigenvectors:
    [1.00000000e+00 1.11022302e-16 1.00000000e+00]
    [[ 0.70710678 -0.70710678  0.        ]
     [ 0.70710678  0.70710678  0.        ]
     [ 0.          0.          1.        ]]
    [0.5 0.5 0. ] [0.5 0.5 0. ]
    [0. 0. 1.] [0. 0. 1.]
    power of matrix:
    [[0.5 0.5 0. ]
     [0.5 0.5 0. ]
     [0.  0.  1. ]]

```python
# periodic
P2 = np.array([[0, 0, 1], [0, 0, 1], [0.5, 0.5, 0]])
print(P2)

print('eigenvalues and eigenvectors:')
w2, v2 = np.linalg.eig(P2.T)
print(w2)
print(v2)

pi2 = v2[:, 2] / v2[:, 2].sum()
print(np.round(pi2, 4), np.round(pi2.dot(P2), 4))

print('power of matrix:')
print(np.linalg.matrix_power(P2, 10000))
print(np.linalg.matrix_power(P2, 10001))
```

    [[0.  0.  1. ]
     [0.  0.  1. ]
     [0.5 0.5 0. ]]
    eigenvalues and eigenvectors:
    [-1.00000000e+00 -8.66277536e-17  1.00000000e+00]
    [[ 4.08248290e-01 -7.07106781e-01 -4.08248290e-01]
     [ 4.08248290e-01  7.07106781e-01 -4.08248290e-01]
     [-8.16496581e-01 -1.83954421e-17 -8.16496581e-01]]
    [0.25 0.25 0.5 ] [0.25 0.25 0.5 ]
    power of matrix:
    [[0.5 0.5 0. ]
     [0.5 0.5 0. ]
     [0.  0.  1. ]]
    [[0.  0.  1. ]
     [0.  0.  1. ]
     [0.5 0.5 0. ]]

The stationary distribution on infinite and continuous state space is much more complicated, please google it if you want to know more about it. Consider for example, the chain on the space $\Omega=\{1,2,\dots\}$ in which the transitions are from $\omega$ to $\omega+1$ with probability 2/3 and from $\omega$ to $\max(\omega-1,1)$ with probability 1/3. The chain is irreducible and aperiodic, but it will wander off to positive infinity and fail to have a stationary distribution.

## Ergodic theorem

In a general sample space, if a Markov chain $\{X_1,X_2,\dots\}$ is irreducible, aperiodic, and **positive recurrent**, then it is **ergodic** and we have the **law of large numbers** for it:
$$
\lim_{n\to\infty}\frac{1}{n}\sum_{i=1}^nf(X_i)=E_\pi[f], a.s.,
$$
for any initial point $X_1=\omega_i$ and any real-valued function $f$ with finite $E_\pi[f]$, where $\pi$ is the corresponding stationary distribution. This theorem justifies the use of the Markov chains in Monte Carlo.

In a classic estimation problems about the distribution $\pi$, such as to estimate $E_\pi[f]$, the first question is how to construct a conditional distribution to build a time-homogeneous chain having $\pi$ as its stationary distribution. According to the definition of stationary distribution in the finite sample space, we need to construct a transition matrix $P=[p_{ij}]$ such that
$$
\sum_{i=1}^m\pi_ip_{ij}=\pi_j=\sum_{i=1}^m\pi_jp_{ji},
$$
for any $j$.

A sufficient condition is to focus on the matrix with the property that $\pi_ip_{ij}=\pi_jp_{ji}$ for any $i,j$, which is called the **detailed balance** condition. A chain with detailed balance is said to be reversible. To see why it is called reversible, try to prove:

* (skip) If $X_1\sim\pi$, a Markov chain with detailed balance would have $Pr(X_i=x_i,\dots,X_j=x_j)=Pr(X_i=x_j,\dots,X_j=x_i)$ for $i<j$.

Define $\pi(x_i)=Pr(X=x_i)$ for $X\sim\pi$, define $P(x_j|x_i)=Pr(X_2=x_j|X_1=x_i)$, easy to see that
\begin{align}
P(X_i=x_i,\dots,X_j=x_j)&=\pi(x_i)\prod_{k=i}^{j-1}P(x_{k+1}|x_k)\\
&=\pi(x_i)\prod_{k=i}^{j-1}\frac{\pi(x_{k+1})P(x_k|x_{k+1})}{\pi(x_k)}\\
&=\pi(x_j)\prod_{k=i}^{j-1}P(x_k|x_{k+1})\\
&=P(X_i=x_j,\dots,X_j=x_i).
\end{align}

Altought the detailed balance condition seems a very restricted condition compare to the original definition of stationary distribution, it still leaves enough degree of freedom for us to construct various transition matrix.

**Think further:** Compute and compare the degree of freedom for the general transition matrix, the transition matrix satifying the definition of a stationary distribution and the transition matrix with detailed balance.

# MCMC

## Metropolis-Hastings

Consider to draw samples from a target distribution $\pi$, given any conditional distribution $q(y|x)$ which is called the proposal distribution like in the Importance Sampling, the original transition procedure is that

*Given $X_n=x$, we draw $y$ from $q(y|x)$, and accept $X_{n+1}=y$.*

We can adjust this transition procedure by a simple acceptance-rejection step, so that the generated Markov chain would have $\pi$ as its stationary distribution:

*Given $X_n=x$, we draw $y$ from $q(y|x)$, and accept $X_{n+1}=y$ with probability $\alpha(x\to y)$, otherwise reject $y$ and put $X_{n+1}=x$.*

This adjustment is to achieve the detailed balance with repect to $\pi$, so the acceptance probability $\alpha(x\to y)$ must be related to $x$ and $y$ and have
$$
\pi(x)q(y|x)\alpha(x\to y)=\pi(y)q(x|y)\alpha(y\to x).
$$
To obtain the above equality, a simple idea is to put $\alpha(x\to y)=\lambda\pi(y)q(x|y)$ and $\alpha(y\to x)=\lambda\pi(x)q(y|x)$ where $\lambda$ is a small positive number to ensure that the two acceptance probabilities are less or equal to 1. We know that the larger $\alpha$ the quicker the Markov chain explore the sample space and so we set $\lambda=1/\max(\pi(x)q(y|x),\pi(y)q(x|y))$, which is equivalent to have
$$
\alpha(x\to y)=\frac{\pi(y)q(x|y)}{\max(\pi(x)q(y|x),\pi(y)q(x|y))}=\min(\frac{\pi(y)q(x|y)}{\pi(x)q(y|x)},1),
$$
which is the acceptance probability in **Metropolis-Hastings algorithm**. So, the MH algorithm in some sense is an algorithm with the highest acceptance probability, according to the [Peskun’s Theorem](https://www2.stat.duke.edu/~scs/Courses/Stat376/Papers/ConvergeRates/Peskun1973.pdf), a high acceptance probability can generally result in a low variance in later estimation.

It is extremely important that, different from the Rejection Sampling, when the new sample $y$ is rejected, giving $X_{n+1}=X_n=x$, that the repeated value be counted again in the average in the equation of Ergodic theorem. Those repetitions may seem inefficient but they apply a necessary reweighting to the generated points, so, just like the Importance Sampling, the resulting samples would essentially be weighted samples $\{(x_1^*,w_1),\dots,(x_n^*,w_n)\}$, where the $x_i^*$'s are distinct and $w_i$'s are the corresponding repetition times. We will later check the difference between the weights of MCMC and the weights of IS.

**Think further:** Another important comment is that chains from MH algorithm are not necessary irreducible and it depents on the target $\pi(x)$ and the proposal $q(y|x)$. Please try to give some simple examples.

### Random walk Metropolis

The **Random Walk** is the most original, natural and popular MH algorithm, we simply use a symmetric [kernel](https://en.wikipedia.org/wiki/Kernel_(statistics)) $K$ as the proposal like the uniform kernel or the Gaussian kernel, i.e., $q(y|x)=K(y-x)$. The symmetry can lead to a simplified acceptance probability
$$
\alpha(x\to y)=\min(\frac{\pi(y)}{\pi(x)},1).
$$

There are two fundamental faults. The first one is that, just like Sampling Importance Resampling, it is biased but asymptotically unbiased.

* Try to draw 5 samples from $N(0,1)$ based on the Random Walk with the Gaussian kernel. Set the initial point to be 0 and set the scale parameter or step size to be 0.5. Derive the expectation of their empirical distribution by 1000 simulations. Receive the samples after discard the first 5 values (**burn-in**). What if we instead draw 100 samples or set the step size to be 1 or 3?

```python
def RW(size, burn, x0, scale):
    pi = st.norm.pdf
    x = np.zeros(size + burn)
    x[0] = x0
    dx = st.norm(scale=scale).rvs(size=size + burn - 1)
    for i, u in enumerate(st.uniform.rvs(size=size + burn - 1)):
        new = x[i] + dx[i]
        ratio = pi(new) / pi(x[i])
        x[i + 1] = new if u <= ratio else x[i]

    return x[burn:]

def experiment(rep, size, scale):
    X = np.zeros([rep, size])
    for r in range(rep):
        X[r] = RW(size, 5, 0, scale)

    X = X.flatten()
    grids = np.linspace(-4, 4, 1000)
    plt.plot(grids, st.norm.pdf(grids))
    plt.hist(X, bins=50, density=True)
    plt.show()
```

```python
np.random.seed(1997)
experiment(rep=1000, size=5, scale=0.5)
experiment(rep=1000, size=100, scale=0.5)
experiment(rep=1000, size=5, scale=1)
experiment(rep=1000, size=5, scale=3)
```

The second fault is that, unlike most sampling schemes we learnt so far, MCMC generates statistically dependent samples whose estimation variance is much higher and much difficult to estimate. Actually, there is a [CLT for the Markov chain](https://stats.stackexchange.com/questions/243921/central-limit-theorem-for-markov-chains#:~:text=The%20Central%20Limit%20Theorem%20(CLT,(0%2C%E2%88%9An).), where the **autocovariance** in a Markov chain plays a vital role. Suppose that we want to estimate $\mu=E_\pi[f]$ by $\hat{\mu}=\frac{1}{n}\sum_{i=1}^nf(X_i)$, given that $X_1\sim\pi$, the autocovariance at lag $l$ is $\gamma_l=cov(f(X_1),f(X_{1+l}))$.

Under some conditions, there would be
$$
\sqrt{n}(\hat{\mu}-\mu)\to N(0,\sigma^2), \sigma^2=\sum_{l=-\infty}^\infty\gamma_l.
$$
Notice that $\gamma_0=var(f(X_1))$ and $\gamma_l=\gamma_{|l|}$.

In most cases, we can estimate $\gamma_l$ by the **sample autocovariance**
$$
\hat{\gamma}_l=\frac{1}{n}\sum_{i=1}^{n-l}(x_i-\hat{\mu})(x_{i+l}-\hat{\mu}),
$$
which can be used to estimate the CLT variance, while it normally does not work well. In practice, to construct the confidence interval, a very simple alternative called [batching](https://www.stat.umn.edu/geyer/5102/notes/mcmc.pdf) is widely used. The main usage of the autocovariance is to quantify the dependence between the elements of a chain by computing the **sample autocorrelation** (ACF)
$$
\hat{\rho}_l=\frac{\hat{\gamma}_l}{\hat{\gamma}_0}.
$$

* Draw 10000 samples from $N(0,1)$ based on the Random Walk with the Gaussian kernel. Set the initial point to be 0 and the step size to be 0.5. Suppose $f(x)=x$. Derive the first 200 autocorrelation and plot them. You don't need to do burn-in. What if we instead set the step size to be one of \{1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 6, 8\}? Please estimate the $\sigma^2$ in CLT and calculate the acceptance rate for each choose. Plot them against the step size. What's the best step size?

```python
def experiment(size, nlags):
    x = RW(size, 0, 0, 0.5)
    sm.graphics.tsa.plot_acf(x, lags=nlags)
    plt.show()

    scales = np.linspace(0.5, 5, 10)
    scales = np.append(scales, [6, 8])
    acc = []
    sigma2 = []
    for scale in scales:
        x = RW(size, 0, 0, scale)
        acc.append(len(set(x)) / size)
        acf = sm.tsa.acf(x, nlags=nlags, fft=False)
        sigma2.append(np.var(x) * (1 + 2 * acf[1:].sum()))

    acc = np.array(acc)
    sigma2 = np.array(sigma2)

    fig, ax_left = plt.subplots()
    ax_right = ax_left.twinx()
    ax_left.plot(scales, acc, color='red')
    ax_left.tick_params(axis='y', labelcolor='red')
    ax_left.set_ylabel('acceptance rate', color='red')
    ax_right.plot(scales, sigma2, color='green')
    ax_right.tick_params(axis='y', labelcolor='green')
    ax_right.set_ylabel('estimation variance', color='green')
    plt.show()

    mask = sigma2 == sigma2.min()
    print('best:', scales[mask], acc[mask], sigma2[mask])
```

```python
np.random.seed(1997)
experiment(size=10000, nlags=200)
```

According to a similar [Gaussian experiment](http://www.stat.columbia.edu/~gelman/research/published/A6n41.pdf), when the dimension $d$ is large, it is optimal to tune the step size so that about 23.4\% of new samples are accepted. And in multidimensional Gaussian experiments like this, the best step size is about $2.38/\sqrt{d}$. Furthermore, the efficiency of $n$ IID samples is equivalent to $3dn$ MCMC samples in the simple normal case.

**Think further:** These results bring lots of insights of MCMC, think about them after class.

**Think further:** A tradition technique called **thinning** can effectively reduce the dependence in a chain by keeping every $k$th simulation draw from a sequence and discarding the rest. Note that $(k-1)/k$ of the autocorrelations will be discarded. Think about why.

### Independent Metropolis

Another simple variant of MH algorithm is the **Independent Metropolis** algorithm whose proposal doesn't depend on the previous values in the chain, i.e., $q(y|x)=q(y)$. It does not means that the Independent Metropolis produce IID samples because the acceptance probability still depends on the late value
$$
\alpha(x\to y)=\min(\frac{\pi(y)/q(y)}{\pi(x)/q(x)},1)=\min(\frac{w(y)}{w(x)},1),
$$
where $w(x)$ is the **importance ratio function** in Importance Sampling.

A trivial and importance case is to set $q(y)=\pi(y)$, in which case we always have $\alpha(x\to y)=1$ and so the optimal proposal of MCMC is actually just the target distribution. Although the optimal proposal appear in the Independent Metropolis, it does not means that the Independent Metropolis is better than the Random Walk especially when the dimension is big, because the difficult to build a proposal similar to the target grows exponentially as dimension increases.

Remember that just like the Importance Sampling, MH algorithm can essentially generate weighted samples if we treat the repetition number of a sample in the chain as its weight, i.e., $\{(x_1^*,w_1),\dots,(x_n^*,w_n)\}$. Here we compare the weights of MCMC and IS in a novel way:

*If we pretend the resulting MCMC weights $w_i$'s are IS weights, according to the definition of IS weights, $\pi(x_i^*)/w_i$ would be proportion to a IS proposal and $\pi(x_i^*)/(w_i/\overline{w_i})$ would be close to the IS proposal evaluated at $x_i^*$, and I term it as the **reconstructed proposal**.*

* (skip) Draw 10000 samples from $N(0,1)$ based on the Independent Metroplis with the Gaussian proposal. Set the initial point to be 0 and the scale parameter to be 2. You don't need to do burn-in. Summarise the resulting samples to be a weighted sample set, compare the reconstructed proposal with the target and MCMC proposal by a plot.

```python
def IM(size, burn, x0, scale):
    x = np.zeros(size + burn)
    x[0] = x0
    w = lambda x: st.norm.pdf(x) / st.norm(scale=scale).pdf(x)
    y = st.norm(scale=scale).rvs(size=size + burn - 1)
    for i, u in enumerate(st.uniform.rvs(size=size + burn - 1)):
        ratio = w(y[i]) / w(x[i])
        x[i + 1] = y[i] if u <= ratio else x[i]

    return x[burn:]

def experiment(size, scale):
    x = IM(size, 0, 0, scale)
    samples, weights = np.unique(x, return_counts=True)
    target = st.norm.pdf(samples)
    recon_proposal = target / (weights / weights.mean())
    proposal_MCMC = st.norm(scale=scale).pdf(samples)

    plt.plot(samples, recon_proposal, label='reconstructed proposal')
    plt.plot(samples, target, label='target density')
    plt.plot(samples, proposal_MCMC, label='MCMC proposal')
    plt.legend()
    plt.show()
```

```python
np.random.seed(1997)
experiment(size=10000, scale=2)
```

This experiment shows that if we pretend MCMC samples are drawn from a IS proposal independently, the IS proposal will imitate the target distribution automatically and the IS proposal itself maybe random. This way to compare the IS and MCMC was proposed by myself. Please contact me if you have any new idea about it.

**Think further:** Try to construct the reconstructed proposal for the Random Walk. You will see that its behaviour is similar to the Independent Metropolis.

## Gibbs sampler

Different from the standard MH algorithm, **Gibbs sampler** solves a problem that what if we can easily draw samples from all the one dimensional **full conditional distributions** $\pi_k(x_k|\mathbf{x}_{-k}),k=1,\dots,K$ by some simple techiques (like those in our first computation tutorial such as the Inversion Method, the Transformation and the Rejection Sampling), how can we construct a better Markov chain leveraging these new information?

The answer is that we simply construct the conditional distribution of the Markov chain based on the full conditional distributions
$$
q(\mathbf{y}|\mathbf{x})=\prod_{k=1}^K\pi_k(y_k|[\mathbf{y}_{1:k-1},\mathbf{x}_{k+1:K}]),
$$
where $[\mathbf{y}_{1:k-1},\mathbf{x}_{k+1:K}]$ means we concatenate the first $k-1$ elements of $\mathbf{y}$ with the last $K-k$ elements of $\mathbf{x}$. Just like most MH proposals, this $q(\mathbf{y}|\mathbf{x})$ doesn't achieve detailed balance without an acceptance-rejection step. But still it has $\pi$ as its stationary distribution without the acceptance-rejection step, which differentiate it from the family of MH algorithms. (Recall that the detailed balance is only a sufficient condition. )

The above special property is achieved by a fact that each substep has the detailed balance. Firstly, the $k$'th substep is
$$
q_k(\mathbf{y}|\mathbf{x})=\delta(\mathbf{y}_{-k}=\mathbf{x}_{-k})\pi_k(y_k|[\mathbf{y}_{1:k-1},\mathbf{x}_{k+1:K}])=\delta(\mathbf{y}_{-k}=\mathbf{x}_{-k})\pi_k(y_k|\mathbf{y}_{-k}).
$$
And then, easy to see that
$$
\pi(\mathbf{x})q_k(\mathbf{y}|\mathbf{x})=\pi(\mathbf{x})\delta(\mathbf{y}_{-k}=\mathbf{x}_{-k})\pi_k(y_k|\mathbf{y}_{-k})=\pi(\mathbf{y})\delta(\mathbf{x}_{-k}=\mathbf{y}_{-k})\pi_k(x_k|\mathbf{x}_{-k})=\pi(\mathbf{y})q_k(\mathbf{x}|\mathbf{y}),
$$
which is equivalent to say that each substep is a MH step with proposal $q_k(\mathbf{y}|\mathbf{x})$ and acceptance rate 1. This is also an example that MH algorithm may produce reducible chains.

**Think further:** As each substep of the Gibbs has $\pi$ as its stationary distribution, $\pi$ must be a stationary distribution for the whole Gibbs step. Think about why.

Although each substep of the Gibbs is reducible, combining them together, we can have an irreducible chain in most cases. Just like the MH algorithm, the irreducibility is not guaranteed and it still depends on the target $\pi$.

* Try to draw samples from $\pi(\mathbf{x})=0.5N(\mathbf{x}|\mathbf{0},I_2)+0.5N(\mathbf{x}|a\mathbf{1},I_2)$ based on the Gibbs. Set $a=5$ and start at $(-10,-10)$. Have 1000 burn-in and set thinning factor to be 10, draw 2000 samples. Make trajectory plot and histogram for $x_1$? What if $a=100$, is Gibbs still irreducible?

```python
def qk(x0, a):
    w1 = 1
    w2 = np.exp(-(a**2 - 2*a*x0) / 2)
    p = np.array([w1, w2]) / (w1 + w2)
    loc = np.random.choice([0, a], p=p)
    return st.norm(loc=loc).rvs()

def Gibbs(x0, burn, size, thin, a):
    for b in range(burn):
        x0[0] = qk(x0[1], a)
        x0[1] = qk(x0[0], a)

    x = np.zeros([size, 2])
    for i in range(size):
        for j in range(thin):
            x0[0] = qk(x0[1], a)
            x0[1] = qk(x0[0], a)

        x[i] = x0

    return x

def experiment(size, a):
    x = Gibbs([-10, -10], 1000, size, 10, a)
    fig, ax = plt.subplots(1, 2, figsize=[13, 4])
    ax[0].plot(x[:, 0])
    ax[1].hist(x[:, 0], bins=20)
    plt.show()
```

```python
np.random.seed(1997)
experiment(size=2000, a=5)
experiment(size=2000, a=100)
```

The performance of the Gibbs is restricted by the structure of the target distribution in the coordinate system. When there are modes can't be connected by lines parallel to the axises, Gibbs can't generate irreducible chains. When there is strong structure not parallel to the axises, Gibbs chains may move very slow. ([draw](https://jspaint.app/))

```python

```

```python

```

STAT6011 Computational Statistics/Tut 7/Tut 7 (with answers).ipynb