 ILI285 - Computación Científica I  / INF285 - Computación Científica

 Roots of 1D equations

 [[S]cientific [C]omputing [T]eam](#acknowledgements)

 Version: 1.3

## Table of Contents
* [Introduction](#intro)
* [Bisection Method](#bisection)
* [Cobweb Plot](#cobweb)
* [Fixed Point Iteration](#fpi)
* [Newton Method](#nm)
* [Wilkinson Polynomial](#wilkinson)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
%matplotlib inline
from ipywidgets import interact
sym.init_printing()
from scipy import optimize
```

## Introduction

Hello again! In this document we're going to learn how to find a 1D equation's solution using numerical methods. First, let's start with the definition of a root:

Definition: The function $f(x)$ has a root in $x = r$ if $f(r) = 0$.

An example: Let's say we want to solve the equation $x + \log(x) = 3$. We can rearrange the equation: $x + \log(x) - 3 = 0$. That way, to find its solution we can find the root of $f(x) = x + \log(x) - 3$. Now let's study some numerical methods to solve these kinds of problems.

Defining a function $f(x)$

```python
f = lambda x: x+np.log(x)-3
```

Finding $r$ using sympy

```python
y = sym.Symbol('y')
fsym = lambda y: y+sym.log(y)-3
r_all=sym.solve(sym.Eq(fsym(y), 0), y)
r=r_all[0].evalf()
print(r)
print(r_all)
```

    2.20794003156932
    [LambertW(exp(3))]

```python
def find_root_manually(r=2.0):
    x = np.linspace(1,3,1000)
    plt.figure(figsize=(8,8))
    plt.plot(x,f(x),'b-')
    plt.grid()
    plt.ylabel('$f(x)$',fontsize=16)
    plt.xlabel('$x$',fontsize=16)
    plt.title('What is r such that $f(r)='+str(f(r))+'$? $r='+str(r)+'$',fontsize=16)
    plt.plot(r,f(r),'k.',markersize=20)
    plt.show()
interact(find_root_manually,r=(1e-5,3,1e-3))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    <function __main__.find_root_manually>

## Bisection Method

The bisection method finds the root of a function $f$, where $f$ is a **continuous** function.
If we want to know if this has a root, we have to check if there is an interval $[a,b]$ for which
$f(a)\cdot f(b) < 0$. When these 2 conditions are satisfied, it means that there is a value $r$, between $a$ and $b$, for which $f(r) = 0$. To summarize how this method works, start with the aforementioned interval (checking that there's a root in it), and split it into two smaller intervals $[a,c]$ and $[c,b]$. Then, check which of the two intervals contains a root. Keep splitting each "eligible" interval until the algorithm converges or the tolerance is surpassed.

```python
def bisect(f, a, b, tol=1e-8):
    fa = f(a)
    fb = f(b)
    i = 0
    # Just checking if the sign is not negative => not root  necessarily
    if np.sign(f(a)*f(b)) >= 0:
        print('f(a)f(b)<0 not satisfied!')
        return None

    #Printing the evolution of the computation of the root
    print(' i |     a     |     c     |     b     |     fa    |     fc     |     fb     |   b-a')
    print('----------------------------------------------------------------------------------------')

    while(b-a)/2 > tol:
        c = (a+b)/2.
        fc = f(c)
        print('%2d | %.7f | %.7f | %.7f | %.7f | %.7f | %.7f | %.7f' %
              (i+1, a, c, b, fa, fc, fb, b-a))
        # Did we find the root?
        if fc == 0:
            print('f(c)==0')
            break
        elif np.sign(fa*fc) < 0:
            b = c
            fb = fc
        else:
            a = c
            fa = fc
        i += 1

    xc = (a+b)/2.
    return xc
```

```python
## Finding a root of cos(x). What about if you change the interval?
#f = lambda x: np.cos(x)
## Another function
#f = lambda x: x**3-2*x**2+(4/3)*x-(8/27)
## Computing the cubic root of 7.
#f = lambda x: x**3-7
#bisect(f,0,2)
f = lambda x: x*np.exp(x)-2
bisect(f,0,1)
```

It's very important to define a concept called **convergence rate**.
This rate shows how fast the convergence of a method is at a specified point.

The convergence rate for the bisection is always 0.5 because this method uses the half of the interval for each iteration.

## Cobweb Plot

```python
def cobweb(x,g=None):
    min_x = np.amin(x)
    max_x = np.amax(x)

    plt.figure(figsize=(10,10))
    ax = plt.axes()
    plt.plot(np.array([min_x,max_x]),np.array([min_x,max_x]),'b-')
    for i in np.arange(x.size-1):
        delta_x = x[i+1]-x[i]
        head_length =  np.abs(delta_x)*0.04
        arrow_length = delta_x-np.sign(delta_x)*head_length
        ax.arrow(x[i], x[i], 0, arrow_length, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')
        ax.arrow(x[i], x[i+1], arrow_length, 0, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')

    if g!=None:
        y = np.linspace(min_x,max_x,1000)
        plt.plot(y,g(y),'r')

    plt.title('Cobweb diagram')
    plt.grid(True)
    plt.show()
```

## Fixed Point Iteration

To learn about the Fixed-Point Iteration we will first learn about the concept of Fixed Point.

A Fixed Point of a function $g$ is a real number $r$, where $g(r) = r$

The Fixed-Point Iteration is based in the Fixed Point concept and works like this to find the root of a function:

\begin{equation} x_{0} = initial\_guess \\ x_{i+1} = g(x_{i})\end{equation}

To find an equation's solution using this method you'll have to move around some things to rearrange the equation in the form $x = g(x)$. That way, you'll be iterating over the funcion $g(x)$, but you will **not** find $g$'s root, but $f(x) = g(x) - x$ (or $f(x) = x - g(x)$)'s root. In our following example, we'll find the solution to $f(x) = x - \cos(x)$ by iterating over the funcion $g(x) = \cos(x)$.

```python
def fpi(g, x0, k, flag_cobweb=False):
    x = np.empty(k+1)
    x[0] = x0
    error_i = np.inf

    print(' i |     x(i)     |    x(i+1)    ||x(i+1)-x(i)| | e_i/e_{i-1}')
    print('--------------------------------------------------------------')

    for i in range(k):
        x[i+1] = g(x[i])
        error_iminus1 = error_i
        error_i = abs(x[i+1]-x[i])
        print('%2d | %.10f | %.10f | %.10f | %.10f' %
              (i,x[i],x[i+1],error_i,error_i/error_iminus1))
    if flag_cobweb:
        cobweb(x,g)
    return x[-1]
```

```python
g = lambda x: np.cos(x)
fpi(g, 0.5, 15, True)
```

Let's quickly explain the Cobweb Diagram we have here. The blue line is the function $x$ and the red is the function $g(x)$. The point in which they meet is $g$'s fixed point. In this particular example, we start at $y = x = 1.5$ (the top right corner) and then we "jump" **vertically** to $y = \cos(1.5) \approx 0.07$. After this, we jump **horizontally** to $x = \cos(1.5) \approx 0.07$. Then, we jump again **vertically** to $y = \cos\left(\cos(1.5)\right) \approx 0.997$ and so on. See the pattern here? We're just iterating over $x = \cos(x)$, getting closer to the center of the diagram where the fixed point resides, in $x \approx 0.739$.

It's very important to mention that the algorithm will converge only if the rate of convergence $S < 1$, where $S = \left| g'(r) \right|$. If you want to use this method, you'll have to construct $g(x)$ starting from $f(x)$ accordingly. In this example, $g(x) = \cos(x) \Rightarrow g'(x) = -\sin(x)$ and $|-\sin(0.739)| \approx 0.67$.

### Another example. Source: https://divisbyzero.com/2008/12/18/sharkovskys-theorem/amp/?__twitter_impression=true

```python
g = lambda x: -(3/2)*x**2+(11/2)*x-2
fpi(g, 1, 15, True)
```

## Newton's Method

For this method, we want to iteratively find some function $f(x)$'s root, that is, the number $r$ for which $f(r) = 0$. The algorithm is as follows:

\begin{equation} x_0 = initial\_guess \end{equation}
\begin{equation} x_{i+1} = x_i - \cfrac{f(x_i)}{f'(x_i)} \end{equation}

which means that you won't be able to find $f$'s root if $f'(r) = 0$. In this case, you would have to use the modified version of this method, but for now let's focus on the unmodified version first. Newton's (unmodified) method is said to have quadratic convergence.

```python
def newton_method(f, fp, x0, rel_error=1e-8, m=1):
    #Initialization of hybrid error and absolute
    hybrid_error = 100
    error_i = np.inf

    print('i |     x(i)     |     x(i+1)   |      |x(i+1)-x(i)|     |  e_i/e_{i-1} | e_i/e_{i-1}^2')
    print('----------------------------------------------------------------------------------------')

    #Iteration counter
    i = 1
    while (hybrid_error > rel_error and hybrid_error < 1e12 and i < 1e4):
        #Newton's iteration
        x1 = x0-m*f(x0)/fp(x0)

        #Checking if root was found
        if f(x1) == 0.0:
            hybrid_error = 0.0
            break

        #Computation of hybrid error
        hybrid_error = abs(x1-x0)/np.max([abs(x1),1e-12])

        #Computation of absolute error
        error_iminus1 = error_i
        error_i = abs(x1-x0)

        #Increasing counter
        i += 1

        #Showing some info
        print("%d | %.10f | %.10f | %.20f | %.10f | %.10f" %
              (i, x0, x1, error_i, error_i/error_iminus1, error_i/(error_iminus1**2)))
        #Updating solution
        x0 = x1

    #Checking if solution was obtained
    if hybrid_error < rel_error:
        return x1
    elif i>=1e4:
        print('Newton''s Method diverged. Too many iterations!!')
        return None
    else:
        print('Newton''s Method diverged!')
        return None
```

```python
f = lambda x: np.sin(x)
fp = lambda x: np.cos(x) # derivative of f
newton_method(f, fp, 3.1,rel_error=1e-14)
```

# Propose Classwork
Build a FPI such that given $x$ computes $\displaystyle \frac{1}{x}$. Write down your solution below or go and see the [solution](#sol1)

```python
print('Please try to think and solve before you see the solution!!!')
```

    Please try to think and solve before you see the solution!!!

# In class

### From the textbook

```python
g1 = lambda x: 1-x**3
g2 = lambda x: (1-x)**(1/3)
g3 = lambda x: (1+2*x**3)/(1+3*x**2)
fpi(g3, 0.5, 10, True)
```

```python
g1p = lambda x: -3*x**2
g2p = lambda x: -(1/3)*(1-x)**(-2/3)
g3p = lambda x: ((1+3*x**2)*(6*x**2)-(1+2*x**3)*6*x)/((1+3*x**2)**2)
r=0.6823278038280194
print(g3p(r))
```

    0.0

### Adding another implementation of FPI including and extra column for analyzing quadratic convergence

```python
def fpi2(g, x0, k, flag_cobweb=False):
    x = np.empty(k+1)
    x[0] = x0
    error_i = np.inf

    print(' i |     x(i)     |    x(i+1)    ||x(i+1)-x(i)| | e_i/e_{i-1} | e_i/e_{i-1}^2')
    print('-----------------------------------------------------------------------------')

    for i in range(k):
        x[i+1] = g(x[i])
        error_iminus1 = error_i
        error_i = abs(x[i+1]-x[i])
        print('%2d | %.10f | %.10f | %.10f | %.10f | %.10f' %
              (i,x[i],x[i+1],error_i,error_i/error_iminus1,error_i/(error_iminus1**2)))
    if flag_cobweb:
        cobweb(x,g)
    return x[-1]
```

Which function shows quadratic convergence? Why?

```python
g1 = lambda x: (4./5.)*x+1./x
g2 = lambda x: x/2.+5./(2*x)
g3 = lambda x: (x+5.)/(x+1)
fpi2(g1, 3.0, 30, True)
```

### Building a FPI to compute the cubic root of 7

```python
# What is 'a'? Can we find another 'a'?
a = -3*(1.7**2)
print(a)
```

    -8.669999999999998

```python
f = lambda x: x**3-7
g =  lambda x: f(x)/a+x
r=fpi(g, 1.7, 14, True)
print(f(r))
```

### Playing with some roots

```python
f = lambda x: 8*x**4-12*x**3+6*x**2-x
fp = lambda x: 32*x**3-36*x**2+12*x-1

x = np.linspace(-1,1,1000)
plt.figure(figsize=(5,5))
# What are we seeing with the semiloigy plot? Is this function differentiable?
plt.semilogy(x,np.abs(f(x)),'b-')
plt.semilogy(x,np.abs(fp(x)),'r-')
plt.grid()
plt.ylabel('$f(x)$',fontsize=16)
plt.xlabel('$x$',fontsize=16)
plt.show()
```

```python
r=newton_method(f, fp, 0.3, rel_error=1e-8, m=1)
print([r,f(r)])
# Is this showing quadratic convergence? If not, can you fix it?
```

    i |     x(i)     |     x(i+1)   |      |x(i+1)-x(i)|     |  e_i/e_{i-1} | e_i/e_{i-1}^2
    ----------------------------------------------------------------------------------------
    2 | 0.3000000000 | 0.3857142857 | 0.08571428571428552079 | 0.0000000000 | 0.0000000000
    3 | 0.3857142857 | 0.4279843444 | 0.04227005870841460400 | 0.4931506849 | 5.7534246575
    4 | 0.4279843444 | 0.4534159993 | 0.02543165489222543041 | 0.6016470208 | 14.2334086872
    5 | 0.4534159993 | 0.4694946399 | 0.01607864055554952820 | 0.6322294252 | 24.8599404138
    6 | 0.4694946399 | 0.4798882000 | 0.01039356016793291371 | 0.6464203321 | 40.2036683316
    7 | 0.4798882000 | 0.4866871127 | 0.00679891264490944947 | 0.6541466576 | 62.9376890188
    8 | 0.4866871127 | 0.4911655766 | 0.00447846388652911598 | 0.6587029604 | 96.8835745915
    9 | 0.4911655766 | 0.4941281466 | 0.00296257004712746630 | 0.6615147788 | 147.7101960697
    10 | 0.4941281466 | 0.4960932149 | 0.00196506826728892747 | 0.6632984996 | 223.8929338621
    11 | 0.4960932149 | 0.4973989041 | 0.00130568918550205693 | 0.6644497839 | 338.1306364449
    12 | 0.4973989041 | 0.4982674500 | 0.00086854596470975487 | 0.6652011630 | 509.4636383483
    13 | 0.4982674500 | 0.4988456368 | 0.00057818680119792187 | 0.6656951096 | 766.4477606024
    14 | 0.4988456368 | 0.4992307216 | 0.00038508477299548094 | 0.6660213831 | 1151.9138480345
    15 | 0.4992307216 | 0.4994872795 | 0.00025655790749512519 | 0.6662374767 | 1730.1060012498
    16 | 0.4994872795 | 0.4996582449 | 0.00017096535582211692 | 0.6663811593 | 2597.3908416937
    17 | 0.4996582449 | 0.4997721891 | 0.00011394425784067019 | 0.6664757155 | 3898.3085915658
    18 | 0.4997721891 | 0.4998481375 | 0.00007594838735824894 | 0.6665398397 | 5849.7010060220
    19 | 0.4998481375 | 0.4998987634 | 0.00005062583760417905 | 0.6665821272 | 8776.7778935922
    20 | 0.4998987634 | 0.4999325121 | 0.00003374875558825874 | 0.6666310561 | 13167.8029965013
    21 | 0.4999325121 | 0.4999550059 | 0.00002249375282564747 | 0.6665061403 | 19749.0582583161
    22 | 0.4999550059 | 0.4999700110 | 0.00001500514206059789 | 0.6670804190 | 29656.2527473931
    23 | 0.4999700110 | 0.4999800060 | 0.00000999499746007215 | 0.6661048206 | 44391.7703605403
    24 | 0.4999800060 | 0.4999866717 | 0.00000666569396912120 | 0.6669030178 | 66723.6805685757
    25 | 0.4999866717 | 0.4999909946 | 0.00000432287884150062 | 0.6485264492 | 97293.1629087970
    26 | 0.4999909946 | 0.4999941319 | 0.00000313735547630145 | 0.7257560508 | 167887.2060628857
    27 | 0.4999941319 | 0.4999956097 | 0.00000147777639253333 | 0.4710261249 | 150134.7642908100
    28 | 0.4999956097 | 0.4999970497 | 0.00000144001474572386 | 0.9744469820 | 659400.8314715016
    29 | 0.4999970497 | 0.4999954553 | 0.00000159440131292099 | 1.1072117960 | 768889.2070839963
    30 | 0.4999954553 | 0.4999967992 | 0.00000134384518901687 | 0.8428525354 | 528632.6149851314
    31 | 0.4999967992 | 0.4999977022 | 0.00000090303564465044 | 0.6719789244 | 500041.9169360077
    32 | 0.4999977022 | 0.4999950737 | 0.00000262848010768035 | 2.9107157876 | 3223256.8059490011
    33 | 0.4999950737 | 0.4999964081 | 0.00000133432963278501 | 0.5076430401 | 193131.7793302043
    34 | 0.4999964081 | 0.4999982008 | 0.00000179273154921056 | 1.3435447322 | 1006906.1641414912
    35 | 0.4999982008 | 0.4999939138 | 0.00000428698813359496 | 2.3913162768 | 1333895.3497373802
    36 | 0.4999939138 | 0.4999960369 | 0.00000212305952357328 | 0.4952333567 | 115520.1137200339
    37 | 0.4999960369 | 0.4999978040 | 0.00000176715078742395 | 0.8323604533 | 392057.0497707436
    38 | 0.4999978040 | 0.4999987633 | 0.00000095926573961957 | 0.5428318548 | 307179.1375397388
    39 | 0.4999987633 | 0.4999957387 | 0.00000302458381729043 | 3.1530197445 | 3286909.5750255324
    40 | 0.4999957387 | 0.4999965029 | 0.00000076425488204634 | 0.2526810061 | 83542.4049666695
    41 | 0.4999965029 | 0.4999968812 | 0.00000037826747445457 | 0.4949493727 | 647623.4360170967
    42 | 0.4999968812 | 0.4999992591 | 0.00000237792675222837 | 6.2863632557 | 16618831.0658740196
    43 | 0.4999992591 | 0.5000161146 | 0.00001685544768070812 | 7.0882955772 | 2980872.1276002550
    44 | 0.5000161146 | 0.5000106637 | 0.00000545084237957294 | 0.3233875767 | 19185.9381501386
    45 | 0.5000106637 | 0.5000070840 | 0.00000357972935949302 | 0.6567295677 | 120482.2157677659
    46 | 0.5000070840 | 0.5000046874 | 0.00000239665618528839 | 0.6695076484 | 187027.4484852393
    47 | 0.5000046874 | 0.5000025819 | 0.00000210541208811588 | 0.8784789829 | 366543.5986752341
    48 | 0.5000025819 | 0.4999998063 | 0.00000277564977962941 | 1.3183403835 | 626167.3859103953
    49 | 0.4999998063 | 0.5003696288 | 0.00036982248520706085 | 133.2381656797 | 48002513.3781396449
    50 | 0.5003696288 | 0.5002464495 | 0.00012317931164718132 | 0.3330768587 | 900.6398259087
    51 | 0.5002464495 | 0.5001643129 | 0.00008213655673550146 | 0.6668048038 | 5413.2856799283
    52 | 0.5001643129 | 0.5001095469 | 0.00005476602556908627 | 0.6667679745 | 8117.7979821256
    53 | 0.5001095469 | 0.5000730351 | 0.00003651177553432028 | 0.6666866028 | 12173.3610546054
    54 | 0.5000730351 | 0.5000486897 | 0.00002434541746509922 | 0.6667826231 | 18262.1253924256
    55 | 0.5000486897 | 0.5000324687 | 0.00001622103474174796 | 0.6662869826 | 27368.0656125717
    56 | 0.5000324687 | 0.5000216575 | 0.00001081118876600229 | 0.6664919309 | 41088.1267130889
    57 | 0.5000216575 | 0.5000144386 | 0.00000721887681853772 | 0.6677227616 | 61762.1961847271
    58 | 0.5000144386 | 0.5000096458 | 0.00000479276611831114 | 0.6639213050 | 91970.1667908346
    59 | 0.5000096458 | 0.5000063645 | 0.00000328135142357855 | 0.6846466826 | 142850.0088969091
    60 | 0.5000063645 | 0.5000040805 | 0.00000228400611013146 | 0.6960565375 | 212124.9593909390
    61 | 0.5000040805 | 0.5000024135 | 0.00000166697042569552 | 0.7298449940 | 319545.9901534105
    62 | 0.5000024135 | 0.5000071783 | 0.00000476480902644738 | 2.8583644635 | 1714706.1635887653
    63 | 0.5000071783 | 0.5000052033 | 0.00000197501001863998 | 0.4144993026 | 86991.7976422348
    64 | 0.5000052033 | 0.5000034947 | 0.00000170858392567474 | 0.8651013967 | 438023.8016830212
    65 | 0.5000034947 | 0.5000012221 | 0.00000227260331253643 | 1.3301092667 | 778486.3516206152
    66 | 0.5000012221 | 0.4999888330 | 0.00001238911739925852 | 5.4515089945 | 2398794.7938135080
    67 | 0.4999888330 | 0.4999925798 | 0.00000374679111453391 | 0.3024259916 | 24410.6163381144
    68 | 0.4999925798 | 0.4999951003 | 0.00000252055427996112 | 0.6727234593 | 179546.5609816919
    69 | 0.4999951003 | 0.4999962565 | 0.00000115617629375953 | 0.4586992246 | 181983.4741264939
    [0.4999965866182223, 0.0]

## Wilkinson Polynomial
https://en.wikipedia.org/wiki/Wilkinson%27s_polynomial

```python
x = sym.symbols('x', reals=True)
W=1
for i in np.arange(1,21):
        W*=(x-i)
W # Printing W nicely
```

```python
# Expanding the Wilkinson polynomial
We=sym.expand(W)
We
```

```python
# Just computiong the derivative
Wep=sym.diff(We,x)
Wep
```

```python
# Lamdifying the polynomial to be used with sympy
P=sym.lambdify(x,We)
Pp=sym.lambdify(x,Wep)
```

```python
# Using scipy function to compute a root
root = optimize.newton(P,16)
print(root)
```

    16.00709959078704

Why is the root far far away from $16$?

# Acknowledgements
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. March 2016._ v1.1.

# Solutions

Problem: Build a FPI such that given $x$ computes $\displaystyle \frac{1}{x}$

```python
# We are finding the 1/a
# Solution code:
a = 2.1
g = lambda x: 2*x-a*x**2
gp = lambda x: 2-2*a*x
r=fpi2(g, 0.7, 6, flag_cobweb=True)
print([r,1/a])
# Are we seeing quadratic convergence?
```

### What is this plot telling us?

```python
xx=np.linspace(0.2,0.8,1000)
plt.figure(figsize=(10,10))
plt.plot(xx,g(xx),'-',label=r'$g(x)$')
plt.plot(xx,gp(xx),'r-',label=r'$gp(x)$')
plt.plot(xx,xx,'g-',label=r'$x$')
plt.plot(xx,0*xx+1,'k--')
plt.plot(xx,0*xx-1,'k--')
plt.legend(loc='best')
plt.grid()
plt.show()
```

```python

```


$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\L}{\mathcal{L}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Parents}{\mathrm{Parents}}
\newcommand{\NonDesc}{\mathrm{NonDesc}}
\newcommand{\I}{\mathcal{I}}
\newcommand{\dsep}{\text{d-sep}}
\newcommand{\Cat}{\mathrm{Categorical}}
\newcommand{\Bin}{\mathrm{Binomial}}
$$

```python
%matplotlib inline
from matplotlib import pyplot as plt;
import matplotlib as mpl;
import numpy as np;

```

## Coin Flipping and EM

Let's work our way through the example presented in lecture.

### Model and EM theory review

First, let's review the model:

Suppose we have two coins, each with unknown bias $\theta_A$ and $\theta_B$, and collect data in the following way:

1. Repeat for $n=1, \dots, N$:
    1. Choose a random coin $z_n$.
    2. Flip this same coin $M$ times.
    3. Record only the total number $x_n$ of heads.

The corresponding probabilistic model is a **mixture of binomials**:
$$
\begin{align}
\theta &= (\theta_A, \theta_B)  &
    &&\text{fixed coin biases} \\
Z_n &\sim \mathrm{Uniform}\{A, B \}    & \forall\, n=1,\dots,N
    &&\text{coin indicators} \\
X_n | Z_n, \theta &\sim \Bin[\theta_{Z_n}, M] & \forall\, n=1,\dots,N
    &&\text{head count}
\end{align}
$$

and the corresponding graphical model is:

The **complete data log-likelihood** for a single trial $(x_n,z_n)$ is
$$
\log p(x_n, z_n | \theta) = \log p(z_n) + \log p(x_n | z_n, \theta)
$$
Here, $P(z_n)=\frac{1}{2}$.  The remaining term is
$$
\begin{align}
\log p(x_n | z_n, \theta)
&= \log \binom{M}{x_n} \theta_{z_n}^{x_n} (1-\theta_{z_n})^{M-x_n} \\
&= \log \binom{M}{x_n} + x_n \log\theta_{z_n} + (M-x_n)\log(1-\theta_{z_n})
\end{align}
$$

### Likelihood Plot

There aren't many latent variables, so we can plot $\log P(\X|\theta_A,\theta_B)$ directly!

```python
def coin_likelihood(roll, bias):
    # P(X | Z, theta)
    numHeads = roll.count("H");
    flips = len(roll);
    return pow(bias, numHeads) * pow(1-bias, flips-numHeads);

def coin_marginal_likelihood(rolls, biasA, biasB):
    # P(X | theta)
    trials = [];
    for roll in rolls:
        h = roll.count("H");
        t = roll.count("T");
        likelihoodA = coin_likelihood(roll, biasA);
        likelihoodB = coin_likelihood(roll, biasB);
        trials.append(np.log(0.5 * (likelihoodA + likelihoodB)));
    return sum(trials);

def plot_coin_likelihood(rolls, thetas=None):
    # grid
    xvals = np.linspace(0.01,0.99,100);
    yvals = np.linspace(0.01,0.99,100);
    X,Y = np.meshgrid(xvals, yvals);

    # compute likelihood
    Z = [];
    for i,r in enumerate(X):
        z = []
        for j,c in enumerate(r):
            z.append(coin_marginal_likelihood(rolls,c,Y[i][j]));
        Z.append(z);

    # plot
    plt.figure(figsize=(10,8));
    C = plt.contour(X,Y,Z,150);
    cbar = plt.colorbar(C);
    plt.title(r"Likelihood $\log p(\mathcal{X}|\theta_A,\theta_B)$", fontsize=20);
    plt.xlabel(r"$\theta_A$", fontsize=20);
    plt.ylabel(r"$\theta_B$", fontsize=20);

    # plot thetas
    if thetas is not None:
        thetas = np.array(thetas);
        plt.plot(thetas[:,0], thetas[:,1], '-k', lw=2.0);
        plt.plot(thetas[:,0], thetas[:,1], 'ok', ms=5.0);
```

```python
plot_coin_likelihood([ "HTTTHHTHTH", "HHHHTHHHHH",
         "HTHHHHHTHH", "HTHTTTHHTT", "THHHTHHHTH"]);
```

For reference (and to make it easier to avoid looking at the solution from lecture :)), here is how to frame EM in terms of the coin flip example:

### E-Step

The **E-Step** involves writing down an expression for
$$
\begin{align}
E_q[\log p(\X, Z | \theta )]
&= E_q[\log p(\X | Z, \theta) p(Z)] \\
&= E_q[\log p(\X | Z, \theta)] + \log p(Z) \\
\end{align}
$$

The $\log p(Z)$ term is constant wrt $\theta$, so we ignore it.
> Recall $q \equiv q_{\theta_t} = p(Z | \X,\theta_t)$

\begin{align}
E_q[\log p(\X | Z, \theta)]
&= \sum_{n=1}^N E_q \bigg[
        x_n \log \theta_{z_n} + (M-x_n) \log (1-\theta_{z_n})
    \bigg] + \text{const.}
\\
&= \sum_{n=1}^N q_\vartheta(z_n=A)
    \bigg[ x_n \log \theta_A + (M-x_n) {\color{red}{ \log (1-\theta_A)}} \bigg] \\
&+ \sum_{n=1}^N q_\vartheta(z_n=B)
    \bigg[ x_n \log \theta_B + (M-x_n) {\color{red}{\log (1-\theta_B)}} \bigg]
 + \text{const.}
\end{align}

### M-Step

Let $a_n = q(z_n = A)$ and $b_n = q(z_n = B)$.  Taking derivatives with respect to $\theta_A$ and $\theta_B$, we obtain the following update rules:
$$
\theta_A = \frac{\sum_{n=1}^N a_n x_n}{\sum_{n=1}^N a_n M}
\qquad
\theta_B = \frac{\sum_{n=1}^N b_n x_n}{\sum_{n=1}^N b_n M}
$$

> **Interpretation:** For each coin, examples are *weighted* according to the probability that they belong to that coin.  Observing $M$ flips is equivalent to observing $a_n M$ *effective* flips.

## Problem: implement EM for Coin Flips

Using the definitions above and an outline below, fill in the missing pieces.

```python
def e_step(n_flips, theta_A, theta_B):
    """Produce the expected value for heads_A, tails_A, heads_B, tails_B
    over n_flipsgiven the coin biases"""

    # Replace dummy values with your implementation
    heads_A, tails_A, heads_B, tails_B = n_flips, 0, n_flips, 0

    return heads_A, tails_A, heads_B, tails_B

def m_step(heads_A, tails_A, heads_B, tails_B):
    """Produce the values for theta that maximize the expected number of heads/tails"""

    # Replace dummy values with your implementation
    theta_A, theta_B = 0.5, 0.5

    return theta_A, theta_B

def coin_em(n_flips, theta_A=None, theta_B=None, maxiter=10):
    # Initial Guess
    theta_A = theta_A or random.random();
    theta_B = theta_B or random.random();
    thetas = [(theta_A, theta_B)];
    # Iterate
    for c in range(maxiter):
        print("#%d:\t%0.2f %0.2f" % (c, theta_A, theta_B));
        heads_A, tails_A, heads_B, tails_B = e_step(n_flips, theta_A, theta_B)
        theta_A, theta_B = m_step(heads_A, tails_A, heads_B, tails_B)

    thetas.append((theta_A,theta_B));
    return thetas, (theta_A,theta_B);
```

```python
rolls = [ "HTTTHHTHTH", "HHHHTHHHHH", "HTHHHHHTHH",
          "HTHTTTHHTT", "THHHTHHHTH" ];
thetas, _ = coin_em(rolls, 0.1, 0.3, maxiter=6);
```

    #0:	0.10 0.30
    #1:	0.50 0.50
    #2:	0.50 0.50
    #3:	0.50 0.50
    #4:	0.50 0.50
    #5:	0.50 0.50

### Plotting convergence

Once you have a working implementation, re-run the code below you should be able to produce a convergence plot of the estimated thetas, which should move towards 0.5, 0.8.

```python
plot_coin_likelihood(rolls, thetas)
```

## Working with different paramaters

Let's explore some different values for the paramaters of our model.

Here's a method to generate a sample for different values of $Z$ (for the purposes of this example the probability of choosing $A$), $\theta_A$ and $\theta_B$:

```python
import random

def generate_sample(num_flips, prob_choose_a=0.5, a_bias=0.5, b_bias=0.5):
    which_coin = random.random()
    if which_coin < prob_choose_a:
        return "".join(['H' if random.random() < a_bias else 'T' for i in range(num_flips)])
    else:
        return "".join(['H' if random.random() < b_bias else 'T' for i in range(num_flips)])

[generate_sample(10),
 generate_sample(10, prob_choose_a=0.2, a_bias=0.9),
 generate_sample(10, prob_choose_a=0.9, a_bias=0.2, b_bias=0.9)]
```

    ['HHHHHTHHHT', 'TTTHHHTHTH', 'HTHHTTTTTT']

## Unequal chance of selecting a coin

Use `generate_sample` to produce a new dataset where:

- coin A is 90% biased towards heads
- coin B is 30% biased towards heads
- the probability of chooising B is 70%

```python
flips = [] # your code goes here
```

Use your EM implementation and plot its progress over on the liklihood plot and see how well it estimates the true latent parameters underlying the coin flips.

```python
# your code goes here
```


# OrthoIV and DRIV: Use Cases and Examples

OrthoIV, DMLIV, and DRIV are a suite of algorithms that use Double Machine Learning approach and Doubly Robust machine learning approach to estimate the heterogeneous treatment effect with an endogeneous treatment and an instrument.

The EconML SDK implements the following  classes:

* **OrthoIV classes**: solve the moment equation $E[(Y-E[Y|X]-\theta(X) \cdot (T-E[T|X]))(Z-E[Z|X])] = 0$
* **DMLIV classes**: minimize the square loss   $E[(Y- E[Y|X] - \theta(X) \cdot (E[T|X, Z] - E[T|X]))^2]$
* **DRIV classes**: minimize the square loss
$$E[(\theta_{pre}(X) + \frac{(Y- E[Y|X]-\theta_{pre}(X) \cdot (T- E[T|X]))(Z-E[Z|X])}{E[T\cdot Z|X]-E[T|X] \cdot E[Z|X] }-\theta(X))^2]$$
* **Intent to Treat DRIV classes**: a special case of DRIV where the instrument is the assignment in a randomized controlled trail with non-compliance

In ths notebook, we show the performance of OrthoIV and DRIV on estimating average treatment effect and heterogeneous treatment effect.

**Notebook contents:**

1. [Example Usage with Average Treatment Effects](#Example-Usage-with-Average-Treatment-Effects)
2. [Example Usage with Heterogeneous Treatment Effects](#Example-Usage-with-Heterogeneous-Treatment-Effects)

```python
# Helper imports
import numpy as np
from scipy import special
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.ensemble import RandomForestRegressor
import matplotlib.pyplot as plt
```

    <IPython.core.display.Javascript object>

# Example Usage with Average Treatment Effects

## DGP
We construct the DGP as below. The instrument corresponds to a fully randomized recommendation of treatment. Then each sample complies with the recommendation to some degree. This probability depends on both the observed feature $X$ and an unobserved confounder that has a direct effect on the outcome.

\begin{align}
W \sim \; & \text{Normal}(0,\, I_{n_w})   \tag{Observed confounders}\\
Z \sim \; & \text{Bernoulli}(p=0.5)   \tag{Instrument}\\
\nu \sim \; & \text{U}[0, 5] \tag{Unobserved confounder}\\
C \sim \; & \text{Bernoulli}(p=0.8 \cdot \text{Sigmoid}(0.4 \cdot X[0] + \nu))   \tag{Compliers when recommended}\\
C0 \sim \; & \text{Bernoulli}(p=0.006)   \tag{Non-Compliers when not recommended}\\
T = \; & C \cdot Z + C0 \cdot (1-Z)  \tag{Treatment}\\
y \sim \; & \theta \cdot T + 2 \cdot \nu + 5 \cdot (X[3]>0) + 0.1 \cdot \text{U}[0, 1]  \tag{Outcome}
\end{align}

Moreover, the constant treatment effect is presdefined here, that's what we want to learn from the model.
\begin{align}
\theta = \; & 10 \tag{ATE}\\
\end{align}

```python
# test performance
def dgp(n, p, true_fn):
    X = np.random.normal(0, 1, size=(n, p))
    Z = np.random.binomial(1, 0.5, size=(n,))
    nu = np.random.uniform(0, 5, size=(n,))
    coef_Z = 0.8
    C = np.random.binomial(
        1, coef_Z * special.expit(0.4 * X[:, 0] + nu)
    )  # Compliers when recomended
    C0 = np.random.binomial(
        1, 0.006 * np.ones(X.shape[0])
    )  # Non-compliers when not recommended
    T = C * Z + C0 * (1 - Z)
    y = (
        true_fn(X) * T
        + 2 * nu
        + 5 * (X[:, 3] > 0)
        + 0.1 * np.random.uniform(0, 1, size=(n,))
    )
    return y, T, Z, X
```

    <IPython.core.display.Javascript object>

```python
func = lambda X: 10
n = 5000
p = 10
y, T, Z, X = dgp(n, p, func)
```

    <IPython.core.display.Javascript object>

## Train Estimators
We train multiple estimators from OrthoIV and DRIV classes, and see whether they could all recover the true estimate
### OrthoIV Estimator

```python
model = lambda: LinearRegression()
model_clf = lambda: LogisticRegression()
```

    <IPython.core.display.Javascript object>

```python
from econml.iv.dml import OrthoIV

est1 = OrthoIV(projection=False, discrete_treatment=True, discrete_instrument=True)
est1.fit(y, T, Z=Z, X=None, W=X)
print("True Treatment Effect: ", func(X))
est1.summary(alpha=0.05)
```

    True Treatment Effect:  10
    Coefficient Results:  X is None, please call intercept_inference to learn the constant!

CATE Intercept Results

         	        point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	      10.04	      0.138	 72.517	   0.0	    9.768	   10.311

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

### Projected OrthoIV Estimator

```python
est2 = OrthoIV(projection=True, discrete_treatment=True, discrete_instrument=True)
est2.fit(y, T, Z=Z, X=None, W=X)
print("True Treatment Effect: ", func(X))
est2.summary(alpha=0.05)
```

    True Treatment Effect:  10
    Coefficient Results:  X is None, please call intercept_inference to learn the constant!

CATE Intercept Results

         	        point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	     10.026	      0.138	 72.874	   0.0	    9.756	   10.295

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

### DMLIV Estimator

```python
from econml.iv.dml import DMLIV, NonParamDMLIV

est3 = DMLIV(discrete_treatment=True, discrete_instrument=True)
est3.fit(y, T, Z=Z, X=None, W=X)
print("True Treatment Effect: ", func(X))
est3.summary()
```

    True Treatment Effect:  10
    Coefficient Results:  X is None, please call intercept_inference to learn the constant!

CATE Intercept Results

         	        point_estimate

  cate_intercept	     10.232

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

### Linear DRIV Estimator

```python
from econml.iv.dr import LinearDRIV, SparseLinearDRIV, ForestDRIV

est4 = LinearDRIV(discrete_instrument=True, discrete_treatment=True)
est4.fit(y, T, Z=Z, X=None, W=X)
print("True Treatment Effect: ", func(X))
est4.summary()
```

    True Treatment Effect:  10
    Coefficient Results:  X is None, please call intercept_inference to learn the constant!

CATE Intercept Results

         	        point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	     10.034	      0.139	 72.211	   0.0	    9.762	   10.306

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

### Linear Intent-to-Treat DRIV Estimator

```python
from econml.iv.dr import LinearIntentToTreatDRIV

est5 = LinearIntentToTreatDRIV(model_t_xwz=model_clf())
est5.fit(y, T, Z=Z, X=None, W=X)
print("True Treatment Effect: ", func(X))
est5.summary()
```

    True Treatment Effect:  10
    Coefficient Results:  X is None, please call intercept_inference to learn the constant!

CATE Intercept Results

         	        point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	      10.03	      0.14	  71.638	   0.0	    9.755	   10.304

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

# Example Usage with Heterogeneous Treatment Effects

## DGP
In this section, we keep the same data generation process, but only change the treatment effect $\theta(X)$ to a function of $X$, and learn the heterogeneous treatment effect. We'd like to see the effect interval learnt from each model could recover the true estimate and the coefficient estimates in the final linear model could recover the true coefficient.

The true effect function is as below:
\begin{align}
\theta = \; & 10 \cdot X[0] \tag{CATE}\\
\end{align}

```python
func = lambda X: 10 * X[:, 0]
```

    <IPython.core.display.Javascript object>

```python
n = 5000
p = 10
y, T, Z, X = dgp(n, p, func)
# Generate test data
X_test = np.linspace(-2, 2, 100).reshape(-1, 1)
```

    <IPython.core.display.Javascript object>

```python
# save output for visualization
res_pred = []
res_lb = []
res_ub = []
name_list = [
    "OrthoIV",
    "ProjectedOrthoIV",
    "NonParamDMLIV",
    "LinearDRIV",
    "ForestDRIV",
    "LinearIntentToTreatDRIV",
]
```

    <IPython.core.display.Javascript object>

## Train Estimators
### OrthoIV Estimator

```python
est1 = OrthoIV(projection=False, discrete_treatment=True, discrete_instrument=True)
est1.fit(y, T, Z=Z, X=X[:, :1], W=X[:, 1:])
print("True Treatment Effect: 10")
est1.summary()
```

    True Treatment Effect: 10

Coefficient Results

   	  point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  X0	     10.043	      0.142	 70.962	   0.0	    9.765	    10.32

CATE Intercept Results

         	        point_estimate	 stderr	 zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	      0.245	      0.137	 1.793	  0.073	  -0.023	    0.513

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

```python
te_pred1 = est1.effect(X_test)
te_pred1_lb, te_pred1_ub = est1.effect_interval(X_test, alpha=0.05)
res_pred.append(te_pred1)
res_lb.append(te_pred1_lb)
res_ub.append(te_pred1_ub)
```

    <IPython.core.display.Javascript object>

### Projected OrthoIV Estimator

```python
est2 = OrthoIV(projection=True, discrete_treatment=True, discrete_instrument=True)
est2.fit(y, T, Z=Z, X=X[:, :1], W=X[:, 1:])
print("True Treatment Effect: 10")
est2.summary()
```

    True Treatment Effect: 10

Coefficient Results

   	  point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  X0	     10.052	      0.142	 70.695	   0.0	    9.773	    10.33

CATE Intercept Results

         	        point_estimate	 stderr	 zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	      0.22	       0.137	 1.604	  0.109	  -0.049	    0.488

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

```python
te_pred2 = est2.effect(X_test)
te_pred2_lb, te_pred2_ub = est2.effect_interval(X_test, alpha=0.05)
res_pred.append(te_pred2)
res_lb.append(te_pred2_lb)
res_ub.append(te_pred2_ub)
```

    <IPython.core.display.Javascript object>

### Non-Parametric DMLIV Estimator

```python
est3 = NonParamDMLIV(
    model_final=RandomForestRegressor(),
    discrete_treatment=True,
    discrete_instrument=True,
)
est3.fit(y, T, Z=Z, X=X[:, :1], W=X[:, 1:])
```

    <econml.iv.dml._dml.NonParamDMLIV at 0x12b3f873608>

    <IPython.core.display.Javascript object>

```python
te_pred3 = est3.effect(X_test)
res_pred.append(te_pred2)
res_lb.append(np.array([]))
res_ub.append(np.array([]))
```

    <IPython.core.display.Javascript object>

### Linear DRIV Estimator

```python
est4 = LinearDRIV(discrete_instrument=True, discrete_treatment=True)
est4.fit(y, T, Z=Z, X=X[:, :1], W=X[:, 1:])
print("True Treatment Effect: 10")
est4.summary()
```

    True Treatment Effect: 10

Coefficient Results

   	  point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  X0	     10.058	      0.141	 71.339	   0.0	    9.781	   10.334

CATE Intercept Results

         	        point_estimate	 stderr	 zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	      0.245	      0.137	 1.791	  0.073	  -0.023	    0.513

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

```python
te_pred4 = est4.effect(X_test)
te_pred4_lb, te_pred4_ub = est4.effect_interval(X_test, alpha=0.05)
res_pred.append(te_pred4)
res_lb.append(te_pred4_lb)
res_ub.append(te_pred4_ub)
```

    <IPython.core.display.Javascript object>

### Forest DRIV Estimator

```python
est5 = ForestDRIV(discrete_instrument=True, discrete_treatment=True)
est5.fit(y, T, Z=Z, X=X[:, :1], W=X[:, 1:])
```

    <econml.iv.dr._dr.ForestDRIV at 0x12b3f8bcb48>

    <IPython.core.display.Javascript object>

```python
te_pred5 = est5.effect(X_test)
te_pred5_lb, te_pred5_ub = est5.effect_interval(X_test, alpha=0.05)
res_pred.append(te_pred5)
res_lb.append(te_pred5_lb)
res_ub.append(te_pred5_ub)
```

    <IPython.core.display.Javascript object>

### Linear Intent-to-Treat DRIV Estimator

```python
est6 = LinearIntentToTreatDRIV()
est6.fit(y, T, Z=Z, X=X[:, :1], W=X[:, 1:])
print("True Treatment Effect: 10")
est6.summary()
```

    True Treatment Effect: 10

Coefficient Results

   	  point_estimate	 stderr	  zstat	 pvalue	 ci_lower	 ci_upper

  X0	      9.894	      0.162	 61.058	   0.0	    9.577	   10.212

CATE Intercept Results

         	        point_estimate	 stderr	 zstat	 pvalue	 ci_lower	 ci_upper

  cate_intercept	      0.267	      0.155	 1.728	  0.084	  -0.036	    0.57

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

    <IPython.core.display.Javascript object>

```python
te_pred6 = est6.effect(X_test)
te_pred6_lb, te_pred6_ub = est6.effect_interval(X_test, alpha=0.05)
res_pred.append(te_pred6)
res_lb.append(te_pred6_lb)
res_ub.append(te_pred6_ub)
```

    <IPython.core.display.Javascript object>

##  Performance Visualization

```python
plt.figure(figsize=(18, 12))
for i in range(6):
    plt.subplot(3, 2, i + 1)
    plt.plot(X_test[:, 0], res_pred[i], label=name_list[i], alpha=0.6)
    if res_lb[i].size != 0:
        plt.fill_between(X_test[:, 0], res_lb[i], res_ub[i], alpha=0.4)
    plt.plot(X_test[:, 0], func(X_test), "b--", label="True effect")
    plt.xlabel("X[0]")
    plt.ylabel("Treatment Effect")
    plt.legend()
plt.show()
```

notebooks/OrthoIV and DRIV Examples.ipynb


## Markov switching autoregression models

This notebook provides an example of the use of Markov switching models in Statsmodels to replicate a number of results presented in Kim and Nelson (1999). It applies the Hamilton (1989) filter the Kim (1994) smoother.

This is tested against the Markov-switching models from E-views 8, which can be found at http://www.eviews.com/EViews8/ev8ecswitch_n.html#MarkovAR or the Markov-switching models of Stata 14 which can be found at http://www.stata.com/manuals14/tsmswitch.pdf.

```python
%matplotlib inline

import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import requests
from io import BytesIO

# NBER recessions
from pandas_datareader.data import DataReader
from datetime import datetime
usrec = DataReader('USREC', 'fred', start=datetime(1947, 1, 1), end=datetime(2013, 4, 1))
```

### Hamilton (1989) switching model of GNP

This replicates Hamilton's (1989) seminal paper introducing Markov-switching models. The model is an autoregressive model of order 4 in which the mean of the process switches between two regimes. It can be written:

$$
y_t = \mu_{S_t} + \phi_1 (y_{t-1} - \mu_{S_{t-1}}) + \phi_2 (y_{t-2} - \mu_{S_{t-2}}) + \phi_3 (y_{t-3} - \mu_{S_{t-3}}) + \phi_4 (y_{t-4} - \mu_{S_{t-4}}) + \varepsilon_t
$$

Each period, the regime transitions according to the following matrix of transition probabilities:

$$ P(S_t = s_t | S_{t-1} = s_{t-1}) =
\begin{bmatrix}
p_{00} & p_{10} \\
p_{01} & p_{11}
\end{bmatrix}
$$

where $p_{ij}$ is the probability of transitioning *from* regime $i$, *to* regime $j$.

The model class is `MarkovAutoregression` in the time-series part of `Statsmodels`. In order to create the model, we must specify the number of regimes with `k_regimes=2`, and the order of the autoregression with `order=4`. The default model also includes switching autoregressive coefficients, so here we also need to specify `switching_ar=False` to avoid that.

After creation, the model is `fit` via maximum likelihood estimation. Under the hood, good starting parameters are found using a number of steps of the expectation maximization (EM) algorithm, and a quasi-Newton (BFGS) algorithm is applied to quickly find the maximum.

```python
# Get the RGNP data to replicate Hamilton
from statsmodels.tsa.regime_switching.tests.test_markov_autoregression import rgnp
dta_hamilton = pd.Series(rgnp, index=pd.date_range('1951-04-01', '1984-10-01', freq='QS'))

# Plot the data
dta_hamilton.plot(title='Growth rate of Real GNP', figsize=(12,3))

# Fit the model
mod_hamilton = sm.tsa.MarkovAutoregression(dta_hamilton, k_regimes=2, order=4, switching_ar=False)
res_hamilton = mod_hamilton.fit()
```

```python
res_hamilton.summary()
```

We plot the filtered and smoothed probabilities of a recession. Filtered refers to an estimate of the probability at time $t$ based on data up to and including time $t$ (but excluding time $t+1, ..., T$). Smoothed refers to an estimate of the probability at time $t$ using all the data in the sample.

For reference, the shaded periods represent the NBER recessions.

```python
fig, axes = plt.subplots(2, figsize=(7,7))
ax = axes[0]
ax.plot(res_hamilton.filtered_marginal_probabilities[0])
ax.fill_between(usrec.index, 0, 1, where=usrec['USREC'].values, color='k', alpha=0.1)
ax.set_xlim(dta_hamilton.index[4], dta_hamilton.index[-1])
ax.set(title='Filtered probability of recession')

ax = axes[1]
ax.plot(res_hamilton.smoothed_marginal_probabilities[0])
ax.fill_between(usrec.index, 0, 1, where=usrec['USREC'].values, color='k', alpha=0.1)
ax.set_xlim(dta_hamilton.index[4], dta_hamilton.index[-1])
ax.set(title='Smoothed probability of recession')

fig.tight_layout()
```

From the estimated transition matrix we can calculate the expected duration of a recession versus an expansion.

```python
print(res_hamilton.expected_durations)
```

In this case, it is expected that a recession will last about one year (4 quarters) and an expansion about two and a half years.

### Kim, Nelson, and Startz (1998) Three-state Variance Switching

This model demonstrates estimation with regime heteroskedasticity (switching of variances) and no mean effect. The dataset can be reached at http://econ.korea.ac.kr/~cjkim/MARKOV/data/ew_excs.prn.

The model in question is:

$$
\begin{align}
y_t & = \varepsilon_t \\
\varepsilon_t & \sim N(0, \sigma_{S_t}^2)
\end{align}
$$

Since there is no autoregressive component, this model can be fit using the `MarkovRegression` class. Since there is no mean effect, we specify `trend='nc'`. There are hypotheized to be three regimes for the switching variances, so we specify `k_regimes=3` and `switching_variance=True` (by default, the variance is assumed to be the same across regimes).

```python
# Get the dataset
ew_excs = requests.get('http://econ.korea.ac.kr/~cjkim/MARKOV/data/ew_excs.prn').content
raw = pd.read_table(BytesIO(ew_excs), header=None, skipfooter=1, engine='python')
raw.index = pd.date_range('1926-01-01', '1995-12-01', freq='MS')

dta_kns = raw.ix[:'1986'] - raw.ix[:'1986'].mean()

# Plot the dataset
dta_kns[0].plot(title='Excess returns', figsize=(12, 3))

# Fit the model
mod_kns = sm.tsa.MarkovRegression(dta_kns, k_regimes=3, trend='nc', switching_variance=True)
res_kns = mod_kns.fit()
```

```python
res_kns.summary()
```

Below we plot the probabilities of being in each of the regimes; only in a few periods is a high-variance regime probable.

```python
fig, axes = plt.subplots(3, figsize=(10,7))

ax = axes[0]
ax.plot(res_kns.smoothed_marginal_probabilities[0])
ax.set(title='Smoothed probability of a low-variance regime for stock returns')

ax = axes[1]
ax.plot(res_kns.smoothed_marginal_probabilities[1])
ax.set(title='Smoothed probability of a medium-variance regime for stock returns')

ax = axes[2]
ax.plot(res_kns.smoothed_marginal_probabilities[2])
ax.set(title='Smoothed probability of a high-variance regime for stock returns')

fig.tight_layout()
```

### Filardo (1994) Time-Varying Transition Probabilities

This model demonstrates estimation with time-varying transition probabilities. The dataset can be reached at http://econ.korea.ac.kr/~cjkim/MARKOV/data/filardo.prn.

In the above models we have assumed that the transition probabilities are constant across time. Here we allow the probabilities to change with the state of the economy. Otherwise, the model is the same Markov autoregression of Hamilton (1989).

Each period, the regime now transitions according to the following matrix of time-varying transition probabilities:

$$ P(S_t = s_t | S_{t-1} = s_{t-1}) =
\begin{bmatrix}
p_{00,t} & p_{10,t} \\
p_{01,t} & p_{11,t}
\end{bmatrix}
$$

where $p_{ij,t}$ is the probability of transitioning *from* regime $i$, *to* regime $j$ in period $t$, and is defined to be:

$$
p_{ij,t} = \frac{\exp\{ x_{t-1}' \beta_{ij} \}}{1 + \exp\{ x_{t-1}' \beta_{ij} \}}
$$

Instead of estimating the transition probabilities as part of maximum likelihood, the regression coefficients $\beta_{ij}$ are estimated. These coefficients relate the transition probabilities to a vector of pre-determined or exogenous regressors $x_{t-1}$.

```python
# Get the dataset
filardo = requests.get('http://econ.korea.ac.kr/~cjkim/MARKOV/data/filardo.prn').content
dta_filardo = pd.read_table(BytesIO(filardo), sep=' +', header=None, skipfooter=1, engine='python')
dta_filardo.columns = ['month', 'ip', 'leading']
dta_filardo.index = pd.date_range('1948-01-01', '1991-04-01', freq='MS')

dta_filardo['dlip'] = np.log(dta_filardo['ip']).diff()*100
# Deflated pre-1960 observations by ratio of std. devs.
# See hmt_tvp.opt or Filardo (1994) p. 302
std_ratio = dta_filardo['dlip']['1960-01-01':].std() / dta_filardo['dlip'][:'1959-12-01'].std()
dta_filardo['dlip'][:'1959-12-01'] = dta_filardo['dlip'][:'1959-12-01'] * std_ratio

dta_filardo['dlleading'] = np.log(dta_filardo['leading']).diff()*100
dta_filardo['dmdlleading'] = dta_filardo['dlleading'] - dta_filardo['dlleading'].mean()

# Plot the data
dta_filardo['dlip'].plot(title='Standardized growth rate of industrial production', figsize=(13,3))
plt.figure()
dta_filardo['dmdlleading'].plot(title='Leading indicator', figsize=(13,3));
```

The time-varying transition probabilities are specified by the `exog_tvtp` parameter.

Here we demonstrate another feature of model fitting - the use of a random search for MLE starting parameters. Because Markov switching models are often characterized by many local maxima of the likelihood function, performing an initial optimization step can be helpful to find the best parameters.

Below, we specify that 20 random perturbations from the starting parameter vector are examined and the best one used as the actual starting parameters. Because of the random nature of the search, we seed the random number generator beforehand to allow replication of the result.

```python
mod_filardo = sm.tsa.MarkovAutoregression(
    dta_filardo.ix[2:, 'dlip'], k_regimes=2, order=4, switching_ar=False,
    exog_tvtp=sm.add_constant(dta_filardo.ix[1:-1, 'dmdlleading']))

np.random.seed(12345)
res_filardo = mod_filardo.fit(search_reps=20)
```

```python
res_filardo.summary()
```

Below we plot the smoothed probability of the economy operating in a low-production state, and again include the NBER recessions for comparison.

```python
fig, ax = plt.subplots(figsize=(12,3))

ax.plot(res_filardo.smoothed_marginal_probabilities[0])
ax.fill_between(usrec.index, 0, 1, where=usrec['USREC'].values, color='gray', alpha=0.2)
ax.set_xlim(dta_filardo.index[6], dta_filardo.index[-1])
ax.set(title='Smoothed probability of a low-production state');
```

Using the time-varying transition probabilities, we can see how the expected duration of a low-production state changes over time:

```python
res_filardo.expected_durations[0].plot(
    title='Expected duration of a low-production state', figsize=(12,3));
```

During recessions, the expected duration of a low-production state is much higher than in an expansion.


# Lending Club interest rates

Most personal US loans are made on the basis of the FICO score. These loans include auto and housing loans. In recent years, new companies have emerged that challenge this traditional loan model. One such corporation is the Lending Club.  The Lending Club is an organization that connects borrowers with investors in an online community. It provides a loaning method that is an alternative to traditional banks.  The Lending Club claims to offer a more efficient process that can yield higher returns for investors and lower rates on loans for borrowers, compared to the banking system.  We are interested in understanding the influence of borrower characteristics on the interest rate that the Lending Club offers to the borrower. In particular, how much of the variation in interest rate can be explained by FICO score and how much can be explained by non-FICO measures like debt-to-credit ratio?

## Load loans data

```R
load("./loans-data/loansData.rda")
x <- loansData
x$rate <- as.numeric(gsub("%","",x$Interest.Rate))
x$ratio <- as.numeric(gsub("%","",x$Debt.To.Income.Ratio))
x$fico.num <- as.numeric(substr(x$FICO.Range,1,3))
```

## List variables and view information about data frame

```R
names(x)
```

```R
str(x)
```

## Summarize loans data

```R
summary(x) # also shows NAs
```

## Histogram and boxplot of interest rates

```R
hist(x$ratio)
```

```R
boxplot(x$rate)
```

## Fit curve to interest rate histogram

```R
par(mfrow=c(1,1))
hist(x$rate,breaks=40,freq=FALSE,xlab="Interest rate (percent)",xlim=c(5,25),ylim=c(0,0.2),main="")
d <- density(x$rate)
lines(d$x,d$y,col="red",lwd=2)
```

## Plot loan amount funded vs loan amount requested

```R
plot(x$Amount.Requested, x$Amount.Funded.By)
```

---

# Latex demo

\begin{equation}
\huge i\hbar \frac{\partial}{\partial t} |\Psi(x, t)\rangle = \hat{H} |\Psi(x, t)\rangle
\end{equation}

\begin{equation}
\huge f(x) = \sqrt{\frac{1}{2\pi\sigma^2}} e^{-x^2/2\sigma^2}
\end{equation}

---

# Network demo

```R
library(network)
```

```R
x <- matrix(rbinom(36,1,0.3), nrow = 6, ncol = 6)
print(x)
```

```R
netx <- network(x)
netx
```

```R
as.sociomatrix(netx)
```

```R

```


# Project 2: Model Project
**Name:** Jana Ucnova (pvw581) // Spring Semester 2020

# Ramsey model

Firstly, modules needed are imported.

```python
import numpy as np
import mpmath as mp
import sympy
import matplotlib.pyplot as plt
from scipy import optimize
import sympy as sm
import ipywidgets as widgets
```

## Model description
In this project, we will focus on the neoclassic **Ramsey model** of economic growth.

In the Ramsey model, there is **a finite number of homogeneous agents** with **an infinite time horizon**. Therefore, it is considered as a representative agent model. A **closed economy** is considered and time is continuous. At any point in time, there are **three markets** - one for output goods, one for labor, and the rental market for capital goods.

There are **two types of agents**. On the one hand, there are **firms**. They rent capital and hire labor. They want to maximize their profit. On the other hand, **households** own capital, they supply labor and consume. They decide how much it is optimal to consume and save, so they can maximize their utility.

We have **four variables**:
- Output: $Y_t$
- Capital: $K_t$
- Labor supply: $L_t$
- Consumption: $C_t$

However, we will mostly consider these variables per worker, i.e.
- Output per worker: $y_t = \frac{Y_t}{L_t}$
- Capital per worker: $k_t = \frac{K_t}{L_t}$
- Labor per worker: $l_t = 1$
- Consumption per worker: $c_t = \frac{C_t}{L_t}$

Moreover, prices are taken as given by both firms and households. We have
- Wage rate: $w_t$
- Rental rate of capital: $r_t$

Capital depreciates at rate $\delta>0$ by period. Therefore, the return on household savings, $R_t$, is given by

$$ R_t = 1 + r_t - \delta$$

Firms are owned by households. They rent capital for $r_t$ and hire labor for $w_t$ to produce goods. Then profits are transferred to the households. Firms have access to production technology that is given by **production function**:

$$
Y_t = F(K_t,L_t) = f(k_t)L_t
$$

The reresentative firm maximizes profit:

$$
\underset{K_t,L_t}{\mathrm{max}} F(K_t,L_t) - r_rK_t - w_tL_t
$$

with first order conditions

$$
F_K(K_t,L_t)= f'(k_t) = r_t \\
$$

$$
F_L(K_t,L_t) = f(k_t) - f'(k_t)k_t = w_t
$$

that define demand functions for capital and labor.

### 2. Households
Households work for a real wage. They use it for either consumption or they save it in the form of
accumulating assets, on which they later get interest income. Households try to maximize their current happiness and happiness of all their future descendants, i.e. **overall utility**. They maximize the discounted sum of utility from consumption:

$$
\underset{\{c_t\}_{t=0}^{\infty}}{\mathrm{max}} \sum_{t=0}^{\infty} \beta^tu(c_t)
$$

where $u'(c_t)>0$, $u''(c_t)<0$ and $\beta \in (0,1)$ is a preference factor. Moreover constrains are given by

$$
k_{t+1} = R_tk_t + w_t - c_t \\
$$

$$
\underset{t \to \infty}{\mathrm{lim}}\frac{k_t}{\mathcal{R}_t } \geq 0 \\
$$

$$
\mathcal{R}_t = \prod_{j=0}^{t}R_j
$$
with $k_0$, and paths for $\{R_t\}_{t=0}^{\infty}$ and $\{w_t\}_{t=0}^{\infty}$ given.

Optimal behavior of household is implied by Euler equation:

$$
\frac{u'(c_t)}{u'(c_{t+1})} = \beta R_{t+1}
$$

We consider CRRA utility function
$$
u(c_t) = \frac{c_t^{1- \gamma}}{1- \gamma}
$$
where $\gamma>0$, and Cobb-Douglas production function defined as
$$
F(K_t,L_t) = AK_t^{\alpha}L_t^{1-\alpha}
$$
where $\alpha \in (0,1)$ and $A>0$.
This implies that
$$
f(k_t) = Ak_t^{\alpha}
$$
and therefore
$$
r_t = A \alpha k_t^{\alpha-1} \\
$$
$$
w_t = A(1-\alpha)k_t^{\alpha}
$$

## Steady state
### Analytical solution

To find an analytical solution of **steady-state**, we use **sympy**. Firstly, we find an analytical expression for the Euler equation with respect to $c_{t+1}$.

Then, we need to define all symbols.

```python
gamma = sm.symbols('gamma')
delta = sm.symbols('delta')
c1 = sm.symbols('c_t')
c2 = sm.symbols('c_{t+1}')
k1 = sm.symbols('k_t')
R = sm.symbols('R_{t+1}')
beta = sm.symbols('beta')
A = sm.symbols('A')
alpha = sm.symbols('alpha')
kss = sm.symbols('k_{ss}')
css = sm.symbols('c_{ss}')
u = sm.symbols('u')
r = A*alpha*k1**(alpha-1)
w = A*(1-alpha)*k1**alpha
R = 1 + r - delta
```

Utility function is given by

```python
utilityfun1 = c1**(1-gamma)/(1-gamma)
utilityfun2 = c2**(1-gamma)/(1-gamma)
display(utilityfun1,utilityfun2)
```

$\displaystyle \frac{c_{t}^{1 - \gamma}}{1 - \gamma}$

$\displaystyle \frac{c_{t+1}^{1 - \gamma}}{1 - \gamma}$

Euler equation is then given by

```python
eulereq = sm.Eq(sm.diff(utilityfun1,c1)/sm.diff(utilityfun2,c2),beta*R)
eulereq
```

$\displaystyle \frac{c_{t}^{1 - \gamma} c_{t+1} c_{t+1}^{\gamma - 1}}{c_{t}} = \beta \left(A \alpha k_{t}^{\alpha - 1} - \delta + 1\right)$

We solve Euler equation with respect to $c_{t+1}$ and obain:

```python
eulereq_sol = sm.solve(eulereq,c2)[0]
sm.Eq(c2,eulereq_sol)
```

$\displaystyle c_{t+1} = \left(\frac{\beta c_{t}^{\gamma} \left(A \alpha k_{t}^{\alpha} - \delta k_{t} + k_{t}\right)}{k_{t}}\right)^{\frac{1}{\gamma}}$

In steady state, it holds that the level of capital and consumption is constant, i.e. $k_{t+1}=k_t$ and $c_{t+1}=c_t$.
From the Euler equation, steady-state of capital can be expressed as

```python
steady_state_c = sm.Eq(eulereq_sol,c1)
steady_state_c
steady_state_c_sol = sm.solve(steady_state_c,k1)
sm.Eq(kss,steady_state_c_sol[0])
```

$\displaystyle k_{ss} = \left(\frac{\beta \left(\delta - 1\right) + 1}{A \alpha \beta}\right)^{\frac{1}{\alpha - 1}}$

From budget constraint $k_{t+1} = R_tk_t + w_t - c_t $, steady-state of consumption is expressed as

```python
steady_state_k = sm.Eq(k1,(R*k1+w-c1))
steady_state_k
steady_state_k_sol = sm.solve(steady_state_k,c1)
sm.Eq(css,steady_state_k_sol[0])
```

$\displaystyle c_{ss} = A k_{t}^{\alpha} - \delta k_{t}$

These two equations define steady-state level of capital and consumption in Ramsey model.

### Numerical solution

Firstly, we choose parameters:

```python
beta = 0.96
delta = 0.10
alpha = 1/3
gamma = 0.3
A = 2
```

We want to find steady-state levels of consumption, $c_{ss}$, and capital, $k_{ss}$.

To do that, we need to define functions from the setup.

```python
# Utility function
def u(c):
    """
    Inputs:
        1) c: current period's level of consumption per worker

    Returns:
        1) Value of current period's utility

    """
    if gamma != 1:
        return (c**(1 - gamma) - 1) / (1 - gamma)
    else:
        return np.log(c) # When gamma = 1, we obtain log function

# Lifetime utility function for an agent who lives for only two periods
# Beta is preference factor, 0 < beta < 1
def U(c1, c2):
    """
    Inputs:
        1) C1: consumption per worker in period 1
        2) C2: consumption per worker in period 2

    Returns:
        1) Value of utility for lifetime consumption

    """
    return u(c1) + beta * u(c2)

# Intensive production function f(k)
# Where A > 0 and 0 < alpha < 1
def f(k):
    """
    Inputs:
        1) k_t: current period's level of capital per worker

    Returns:
        1) y_t: current period's output per worker

    """

    return A*k**alpha

# Rental rate of capital r_t
def r(k):
    """
    Inputs:
        1) k_t: current period's level of capital per worker

    Returns:
        1) r_t: current period's level of rental rate of capital

    """

    return A*alpha*k**(alpha-1)

# Wage rate w_t
def w(k):
    """
    Inputs:
        1) k_t: current period's level of capital per worker

    Returns:
        1) w_t: current period's level of wage rate

    """

    return A*(1-alpha)*k**alpha

# Return on household savings R_t
def R(k):
    """
    Inputs:
        1) k_t: current period's level of capital per worker

    Returns:
        1) R_t: current period's level of return on household savings

    """

    return 1+r(k)-delta

```

We define another two functions. The first on is a **budget constraint**, i.e. $k_{t+1} = R_tk_t + w_t - c_t $, that gives us the level of next period's capital per worker. The second one is the Euler equation with respect to $c_{t+1}$, i.e. $c_{t+1}=c_t(\beta R_{t+1})^{1/\gamma}$.

```python
# Next period's capital stock per worker
def capital(k,c):
    """
    Inputs:
        1) k_t: current period's level of capital per worker
        2) c_t: current period's level of consumption per worker

    Returns:
        1) k_t+1: next period's capital stock per worker

    """
    return (R(k))*k + w(k) - c

# Euler equation that gives next period's consumption per worker
def euler(k,c):
    """
    Inputs:
        1) k_t: next period's level of capital per effective worker
        2) c_t: current period's level of consumption per effective worker

    Returns:
        1) c_t+1: next period's consumption per effective worker

    """
    return (beta*R(k))**(1/gamma)*c
```

We define functions that give us the difference in capital and consumption between any two periods. In steady-state, it hold that $k_{t+1}=k_t$ and $c_{t+1}=c_t$.

```python
# The change in capital stock per worker between any two periods is
# k_t+1 - k_t = capital(k, c) - k
# In steady state, capital stock per worker is constant, i.e. k_t=k_t+1
# Thus, steady state values of k and c must solve
# capital(k,c) - k = 0

def F1(k,c):

    return capital(k,c) - k

# The change in consumption between any two periods is
# c_t+1 - c_t = euler(k, c) - c
# In steady state, also consumption is constant, i.e. c_t+1=c_t
# Thus, steady state values of k and c must solve
# euler(k,c) - c = 0

def F2(k,c):

    return euler(k,c) - c

# Get steady state
def ramseySteadyState(X):
    out = [F1(X[0], X[1])]
    out.append(F2(X[0], X[1]))
    return out
```

Now, we want to find steady-state values of capital and consumption. Equations *capital* and *euler* form a two-equation dynamical system with two unknowns. It must, therefore, hold that

F1(k_ss, c_ss) = 0

F2(k_ss, c_ss) = 0

That is a root-finding problem. We fill use the function *fsolve()* to find these roots.

```python
# Our guess
k0, c0 = 1, 1

# Use solver to define steady state
k_SS, c_SS = optimize.fsolve(func=ramseySteadyState, x0=(k0, c0))
r_ss = r(k_SS)
w_ss = w(k_SS)

# Print results
print('Steady state for given parameters is defined by:')
print(f"k_ss = {k_SS:.4f}")
print(f"c_ss = {c_SS:.4f}")
print(f"r_ss = {r_ss:.4f}")
print(f"w_ss = {w_ss:.4f}")
```

    Steady state for given parameters is defined by:
    k_ss = 10.2085
    c_ss = 3.3178
    r_ss = 0.1417
    w_ss = 2.8924

Now, we would like to visualize a steady-state by the phase diagram. We define functions for a steady-state.

```python
# The steady-state level of capital stock per effective worker
def k_ss():

    return ((beta*delta-beta+1)/(A*alpha*beta))**(1/(alpha-1))

# The steady-state level of consumption per effective worker
def c_ss():

    return A*k_ss()**alpha-delta*k_ss()

# Define a function that takes k as an input and return the value of c consistent with F1(k, c) = 0
def locusK(k):
    """
    Inputs:
        1) k: capital stock per effective worker

    Returns:
        1) c: consumption per effective worker

    """
    return f(k)  - delta * k
```

And we construct the diagram.

```python
# Create a grid of points for plotting
gridmax, gridsize = 200, 10000
grid = np.linspace(0, gridmax, gridsize)

# Create a new figure
plt.figure(figsize=(16,8))

# Add the c and k
plt.plot(grid, locusK(grid), '-', color='royalblue', label=r'$k_{t+1}=k_t$, the capital isocurve')
plt.axvline(k_ss(), color='black', label=r'$c_{t+1}=c_t$, the consumption isocurve')
plt.plot(k_ss(), c_ss(), marker='.', markersize=20, color='skyblue')

# Create annotation for steady state
ss = '$k_{ss}$ = '+ str(round(k_ss(),2)) + ', ' + '$c_{ss}$ = '+ str(round(c_ss(),2))
plt.annotate(ss,
            xy=(k_ss(), c_ss()),
            xytext=(-60, 30), textcoords='offset points',
            bbox=dict(boxstyle="round", fc="0.8"),
            arrowprops=dict(arrowstyle="->",
                            shrinkA=20, shrinkB=10,
                            connectionstyle="angle,angleA=0,angleB=90,rad=10"))

# Add arrows to indicate out of steady-state dynamics
x_len = 0.025 * gridmax
y_len = 1 / x_len

plt.arrow(x=0.5 * k_ss(), y=0.5 * c_ss(), dx=0, dy=y_len, color='slategray')
plt.arrow(x=0.5 * k_ss(), y=0.5 * c_ss(), dx=x_len, dy=0, color='slategray')

plt.arrow(x=0.5 * k_ss() + x_len, y=1.5 * c_ss(), dx=0, dy=y_len, color='slategray')
plt.arrow(x=0.5 * k_ss() + x_len, y=1.5 * c_ss(), dx=-x_len, dy=0, color='slategray')

plt.arrow(x=1.5 * k_ss(), y=0.5 * c_ss() + y_len, dx=0, dy=-y_len, color='slategray')
plt.arrow(x=1.5 * k_ss(), y=0.5 * c_ss() + y_len, dx=x_len, dy=0, color='slategray')

plt.arrow(x=1.5 * k_ss() + x_len, y=1.5 * c_ss() + y_len, dx=0, dy=-y_len, color='slategray')
plt.arrow(x=1.5 * k_ss() + x_len, y=1.5 * c_ss() + y_len, dx=-x_len, dy=0, color='slategray')

# Label axis
plt.xlabel('$k_t$')
plt.ylabel('$c_t$', rotation='horizontal')

# Add a title to the plot
plt.title('Phase Diagram for the Ramsey Model', size=20)

# Add the legend
plt.legend(frameon=False)
plt.grid(True)

# Change range of axis
plt.ylim((0, 6))
plt.xlim((-1, 100))

plt.savefig('Ramsey-Phase-Diagram.png')
```

## Pernament change in preference parameter $\beta$

Preference parameter $\beta$ controls consumption preferences. The change in this parameter will affect the Euler equation. We assume that the economy is now in the steady-state that can be represented by the phase diagram above and we want to see what happens when the parameter $\beta$ changes.

To see how the economy responds to the change in parameter $\beta$, we construct an interactive plot with the slider.

We define functions of steady-state once again, however this time with input *beta*.

```python
def k_star(beta):

    return ((beta*delta-beta+1)/(A*alpha*beta))**(1/(alpha-1))

def c_star(beta):

    return A*k_star(beta)**alpha-delta*k_star(beta)

def locusK_beta(k):
    """
    Inputs:
        1) k: capital stock per effective worker

    Returns:
        1) c: consumption per effective worker

    """
    return f(k)  - delta * k
```

We define function *beta_shock*:

```python
def beta_shock(betas):

    # Create a grid of points for plotting
    gridmax, gridsize = 200, 10000
    grid = np.linspace(0, gridmax, gridsize)

    # Create a new figure
    plt.figure(figsize=(16,8))

    # Add the c and k
    plt.plot(grid, locusK_beta(grid), '-', color='royalblue', label=r'$k_{t+1}=k_t$, the capital isocurve')
    plt.axvline(k_star(betas), color='black', label=r'$c_{t+1}=c_t$, the consumption isocurve')
    plt.plot(k_star(betas), c_star(betas), marker='.', markersize=20, color='skyblue')

    # Create annotation for steady state
    ss_beta = '$k_{ss}$ = '+ str(round(k_star(betas),2)) + ', ' + '$c_{ss}$ = '+ str(round(c_star(betas),2))
    plt.annotate(ss_beta,
            xy=(k_star(betas), c_star(betas)),  # theta, radius
            xytext=(-60, 30), textcoords='offset points',
            bbox=dict(boxstyle="round", fc="0.8"),
            arrowprops=dict(arrowstyle="->",
                            shrinkA=20, shrinkB=10,
                            connectionstyle="angle,angleA=0,angleB=90,rad=10"))

    # Label axis
    plt.xlabel('$k_t$')
    plt.ylabel('$c_t$', rotation='horizontal')

    # Add a title to the plot
    plt.title('Phase Diagram for the Ramsey Model, pernament shock in preference parameter', size=20)

    # Add the legend
    plt.legend(frameon=False)
    plt.grid(True)

    # Change range of axis
    plt.ylim((0, 6))
    plt.xlim((-1, 100))
```

And we can construct an interactive phase diagram with *beta slider*. This plot shows the response of the economy to an unforeseen shock to the preference parameter $\beta$.

```python
widgets.interact(beta_shock,
    betas=widgets.FloatSlider(description='beta', min=0.01, max=0.99, step=0.01, value=0.96),
);
```

    interactive(children=(FloatSlider(value=0.96, description='beta', max=0.99, min=0.01, step=0.01), Output()), _…

It can be seen that a decrease in $\beta$ results in a decrease in both steady-state capital, $k_{ss}$, and consumption, $c_{ss}$.

## Government
Now, we introduce government in the economy. Government expenditure, $G_t>0$, does not affect households' preferences for consumption and firms' current and future productivity. That means that government expenditure does not affect the consumption
isocurve, it only shifts the capital isocurve downwards, i.e. it is now defined as

$$
c_t = Ak_t^{\alpha}-\delta k_t - G_t
$$

We define functions of steady-state once again, however, this time we include government expenditure, $G$.

```python
def k_govern():

    return ((beta*delta-beta+1)/(A*alpha*beta))**(1/(alpha-1))

def c_govern(G):

    return A*k_star(beta)**alpha-delta*k_star(beta)-G

def locusK_G(k,G):
    """
    Inputs:
        1) k: capital stock per effective worker

    Returns:
        1) c: consumption per effective worker

    """
    return f(k)  - delta * k - G
```

We define function *government_exp*:

```python
def government_exp(G):

    # Create a grid of points for plotting
    gridmax, gridsize = 200, 10000
    grid = np.linspace(0, gridmax, gridsize)

    # Create a new figure
    plt.figure(figsize=(16,8))

    # Add the c and k
    plt.plot(grid, locusK_G(grid,G), '-', color='royalblue', label=r'$k_{t+1}=k_t$, the capital isocurve')
    plt.axvline(k_govern(), color='black', label=r'$c_{t+1}=c_t$, the consumption isocurve')
    plt.plot(k_govern(), c_govern(G), marker='.', markersize=20, color='skyblue')

    # Create annotation for steady state
    ss_g = '$k_{ss}$ = '+ str(round(k_govern(),2)) + ', ' + '$c_{ss}$ = '+ str(round(c_govern(G),2))
    plt.annotate(ss_g,
            xy=(k_govern(), c_govern(G)),  # theta, radius
            xytext=(-60, 30), textcoords='offset points',
            bbox=dict(boxstyle="round", fc="0.8"),
            arrowprops=dict(arrowstyle="->",
                            shrinkA=20, shrinkB=10,
                            connectionstyle="angle,angleA=0,angleB=90,rad=10"))

    # Label axis
    plt.xlabel('$k_t$')
    plt.ylabel('$c_t$', rotation='horizontal')

    # Add a title to the plot
    plt.title('Phase Diagram for the Ramsey Model, government expenditures', size=20)

    # Add the legend
    plt.legend(frameon=False)
    plt.grid(True)

    # Change range of axis
    plt.ylim((0, 6))
    plt.xlim((-1, 100))
```

And we can construct an interactive phase diagram with *G slider*. This plot shows the response of the economy to a permanent change in government expenditures.

```python
widgets.interact(government_exp,
    G=widgets.FloatSlider(description='G', min=0, max=3, step=0.2, value=0),
);
```

    interactive(children=(FloatSlider(value=0.0, description='G', max=3.0, step=0.2), Output()), _dom_classes=('wi…

It can be seen that a permanent increase in government expenditures shifts the blue curve that illustrates the level of consumption downwards.

## Conclusion

In this project, we focused on the Ramsey model. We found both analytical and numerical solutions for a steady-state that are given by

```python
print('Analytical solution for steady state:')
display(sm.Eq(kss,steady_state_c_sol[0]),sm.Eq(css,steady_state_k_sol[0]))
print('Numerical solution for steady state for given parameters:')
print(f"k_ss = {k_SS:.4f}")
print(f"c_ss = {c_SS:.4f}")
```

    Analytical solution for steady state:

$\displaystyle k_{ss} = \left(\frac{\beta \left(\delta - 1\right) + 1}{A \alpha \beta}\right)^{\frac{1}{\alpha - 1}}$

$\displaystyle c_{ss} = A k_{t}^{\alpha} - \delta k_{t}$

    Numerical solution for steady state for given parameters:
    k_ss = 10.2085
    c_ss = 3.3178

Then we visualized this solution by the phase diagram. For our further analysis, we decided to focus on a permanent change in preference parameter $\beta$ and the introduction of government. Using an interactive phase diagram, one can see that decrease in preference parameter $\beta$ results in the shift of the consumption isocurve to the left which implies a decrease in steady-state capital and consumption. After that, we introduced government expenditure, $G_t>0$. Government expenditure does not affect households' preferences for consumption and firms' current and future productivity. Using an interactive phase diagram once again, it can be seen that an increase in government expenditure, $G_t$, results in the shift of the capital isocurve downwards. This implies that a steady-state level of capital remains the same as before, but a steady-state level of consumption decreases.


```python
### First imports and default parameters

%matplotlib inline

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib
import matplotlib.pyplot as plt

# Overwritting matplotlib default linestyle of negative contours
matplotlib.rcParams['contour.negative_linestyle'] = 'solid'

SEED = 42
```

# 1. First steps with unsupervised anomaly detection algorithms

The goal of this section is to get familiar with different unsupervised anomaly detection approaches and algorithms. In order to visualise the output of the different algorithms we consider a toy data set consisting in a two-dimensional Gaussian mixture.

### Generating the data set

```python
from utils import GaussianMixture

n_samples = 500
n_features = 2
weight_1 = 0.5
weight_2 = 0.5
mean_1 = np.zeros(n_features)
mean_2 = -1 * np.ones(n_features)
cov_1 = np.array([[2., 2.,], [2., 4.]])
cov_2 = 2 * np.identity(n_features)
weights = np.array([weight_1, weight_2])
means = np.array([mean_1, mean_2])
covars = np.array([cov_1, cov_2])

gm = GaussianMixture(weights, means, covars, random_state=SEED)
X = gm.sample(n_samples)
```

### Plot the samples and levels set of the density

```python
X_range = np.zeros((n_features, 2))
X_range[:, 0] = np.min(X, axis=0)
X_range[:, 1] = np.max(X, axis=0)

h = 0.1  # step size of the mesh
x_min, x_max = X_range[0, 0] - 0.1, X_range[0, 1] + 0.1
y_min, y_max = X_range[1, 0] - 0.1, X_range[1, 1] + 0.1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

grid = np.c_[xx.ravel(), yy.ravel()]
Z = gm.density(grid)
Z = Z.reshape(xx.shape)

plt.figure()
plt.contour(xx, yy, Z, 10, cmap=plt.cm.Blues_r)
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.show()
```

The goal is to estimate a Minimum volume set with mass at least 0.95. We know that under regularity assumption of the underlying distribution a Minimum volume set is a density level set of the density $f$.
$$
\{x, f(x) \geq F_f^{-1}(1-\alpha) \}
$$
where $F_f^{-1}(1-\alpha)$ is the quantile of order $1-\alpha$ of the distribution of the random variable $f(X)$.

1. Draw a sample $\mathcal{S}$ from the previous Gaussian mixture of size $n = 1,000,000$.

2. Using the sample $\mathcal{S}$ and the true density $f$ given by `gm.density` compute the empirical quantile of order $1-\alpha$ for $\alpha = 0.95$ and $\alpha = 0.99$. You can use for instance the `scipy.stats.mstats.mquantiles` function.

3. Plot the corresponding Minimum Volume set in a figure like the previous one, using `plt.contour` and replacing the number of levels (10) by the threshold you computed: `levels=threshold`.

4. Emphasize the outliers (for instance use a different color) in the previous plot where an outlier is a sample outside the Minimum Volume set.

## 1.1. Density estimation

We are going to use the plug-in approach to estimate the Minimum Volume set with mass at least 0.95. The plug-in approach means that we replace the density $f$ by an estimate $\hat f$.

1. Equation of the corresponding Minimum Volume set estimate.

2. Using a Gaussian kernel, compute a kernel density estimator of $f$. To stick with `sklearn` you can use `sklearn.neighbors.kde.KernelDensity` with the default bandwidth.

3. Compute the empirical quantile $\widehat F_{\hat f}^{-1}(1-\alpha)$ from the original sample for $\alpha = 0.95$ (still using the function `mquantiles`).

4. Add the corresponding Minimum Volume set estimate to the previous plot to visualize the difference between the true MV set and the estimated one.

### Bandwidth selection with cross validation

We used the default bandwidth.

1. Use cross validation to learn the bandwidth from the data. You may use `sklearn.model_selection.GridSearchCV` to perform a 3-fold cross validation.

2. Plot the Minimum Volume set estimate obtained with the learnt bandwidth.

## 1.2. One-Class SVM

The One-Class SVM separates the data from the origin by solving the following optimization problem.

\begin{equation}
\min_{\boldsymbol{w},\boldsymbol{\xi},\rho} \quad \frac{1}{2}\Vert \boldsymbol{w} \Vert^2 - \rho + \frac{1}{\nu n}\sum_{i=1}^n\xi_i\\
\text{s.t.} \quad \langle \boldsymbol{w}, \Phi(x_i) \rangle \geqslant \rho - \xi_i, \quad \xi_i \geqslant 0 \quad \quad 1 \leqslant i \leqslant n
\end{equation}

where the parameter $\nu \in (0,1)$ has to be set by the user and controls the proportion of outliers.

It is in general solved via the dual given by
\begin{equation}
\min_{\boldsymbol{\alpha}} \quad \frac{1}{2}\sum_{i,j} k(x_i, x_j)\\
\text{s.t.} \quad 0 \leqslant \alpha_i \leqslant \frac{1}{\nu n},\quad \sum_{i}\alpha_i = 1 \quad \quad 1 \leqslant i \leqslant n
\end{equation}

1. Show that when using a Gaussian kernel, all data in the feature space lies on the same hypersphere. Show that data are always linearly separable from the origin. Hint: what's the formula of $k(x, x')$ in terms of the feature map $\Phi$.

Under mild assumptions the following inequality holds true for all sample size $n$:

$$ \frac{\text{Outliers}}{n} \leq \nu \leq \frac{\text{Support Vectors}}{n} $$.

Futhermore the two fractions on the left-hand side and right-hand converge almost surely towards $\nu$.

1. If we want to estimate a Minimum Volume set with mass at least $\alpha$, how can we set $\nu$?

2. Use `sklearn.svm.OneClassSVM` with the default parameters (except the parameter $\nu$ set according to the previous question) to learn a Minimum Volume set with mass at least $0.95$.

3. Plot the Minimum Volume set estimate.

1. What do you notice? The kernel used by the One-Class SVM is given by $$ k(x, x') = \exp(-\gamma\Vert x - x' \Vert^2). $$

2. What is the default gamma used when we trained the One-Class SVM? Do we need to increase or decrease its value?

### Support vectors - Outliers

1. Check that we indeed have $$ \frac{\text{Outliers}}{n} \leq \nu \leq \frac{\text{Support Vectors}}{n} $$.

2. Use `gamma=10`. Is the inequality satisfied? Can you guess why?

Only the support vectors are involved in the decision function of the One-Class SVM.

1. Plot the level sets of the One-Class SVM decision function as we did for the true density.
2. Emphasize the Support vectors.

1. Concerning the support vectors, what is the main difference between the OneClass SVM and a Kernel Density estimator when using a Gaussian Kernel?

2. When is it particularly advantageous to use the OneClass SVM?

### What if we change the value of $\nu$?

1. Set $\nu = 0.4$. How can we estimate a Minimum Volume set with mass at least $0.95$. Hint: Use the `decision_function`.

### Equivalence with SVDD

Support Vector Data Description consists in finding the ball of minimum volume containing an arbitrary proportion of data. This is obtained by minimizing the following optimization problem.

\begin{equation}
\min_{R, \boldsymbol{\xi},\boldsymbol{c}}  \quad R^2 + \frac{1}{\nu n}\sum_{i=1}^{n}\xi_i\\
    \text{s.t.} \quad \Vert \boldsymbol{x}_i - \boldsymbol{c}\Vert^2 \leqslant R^2 + \xi_i, \quad \xi_i \geqslant 0 \quad \quad 1\leqslant i \leqslant n\\
\end{equation}

It is in general solved via the dual given by
\begin{equation}
\min_{\boldsymbol{\alpha}} \quad \sum_{i,j} k(x_i, x_j) + \sum_{i}\alpha_i k(x_i,x_i)\\
\text{s.t.} \quad 0 \leqslant \alpha_i \leqslant \frac{1}{\nu n},\quad \sum_{i}\alpha_i = 1 \quad \quad 1 \leqslant i \leqslant n
\end{equation}

1. When are SVDD and the One-Class SVM equivalent?

## 1.3. Isolation Forest

1. Run Isolation with `n_estimators=1` , i.e., only one tree will be built.

2. Plot the contour of the set with the $95\%$ most normal observations. Hint: use the `decision_function` and the `contamination` parameter. Compare it with the true Minimum Volume set with mass at least $0.95$.

3. Increase `n_estimators` to 50, 100 and then 500.

4. What does the `max_sample` parameter change?

### Application on the digits data set

```python
from sklearn import datasets
digits = datasets.load_digits()
```

The digits data set consits in images (8 x 8) of digits.

```python
images = digits.images
labels = digits.target
images.shape
```

    (1797, 8, 8)

```python
### Showing some images

i = 0

plt.figure()
plt.title('{0}'.format(labels[i]))
plt.axis('off')
plt.imshow(images[i], cmap=plt.cm.gray_r, interpolation='nearest')
plt.show()
```

To use the images as a training set we need to flatten the images.

```python
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))
data.shape
```

    (1797, 64)

We focus on the digit '5'.

```python
X = data
y = digits.target
X_5 = X[y == 5]
```

1. Use IsolationForest to find the top 5% most abnormal images.
2. Plot them using the code we used to plot some images of the data set

## 1.4. Local Outlier Factor

k-NN based anomaly detection algorithm. Available on the dev version of Scikit-Learn.

## 1.5. Unsupervised as supervised using SVM

Let $P$ be a distribution with finite support $C$. Let $\mu$ be the uniform distribution over the support $C$: $\mu(A) = \lambda(A)/\lambda(C)$. We now consider the joint distribution $Q$ of $(X, Y) \in \mathbb{R}^d \times \{-1, +1\}$ given by the class conditionals distributions:

$$
X \vert Y= +1 \sim P \quad \text{and} \quad X \vert Y=-1 \sim \mu
$$
and a priori class probabilities $p = \mathbb{P}(Y=1)$.

We are as before interested in finding a given density level set $\{ f \geq \tau \}$ where $f$ is the density of $P$ with respect to the Lebesgue measure $\lambda$.

The marginal distribution of $X$, $P_X$, is given by
$$
P_X = p P + (1-p) Q.
$$

Assuming that $P$ has a density $f$. It can be shown that

$$\eta(x) = P(Y=1\vert X=x) = \frac{pf(x)}{pf(x) + (1-p)/\lambda(C)}$$

The optimal classification set is given by $\{ \eta \geq 1/2 \}$

1. Show that if $p=1/(1 + \lambda(C)\tau)$ then $\lambda(\{\eta \geq 1/2 \} \Delta \{f \geq \tau \} ) = 0$

Thus solving a binary classification problem assigning weights $p$ and $1-p$ to the samples can lead to a density level set.

# 2. Anomaly scoring and Mass Volume curve

Usually unsupervised anomaly detection algorithms returns a scoring function $s$ such that the smaller $s(x)$ the more abnormal $x$ is. To assess the performance of such a scoring function we can use the Mass Volume curve. The MV curve of a scoring function $s$ is given for all $\alpha \in (0,1)$ by
$$
MV_s(\alpha) = \lambda\{x, s(x) \geq F_s^{-1}(1-\alpha) \}
$$
where $F_s^{-1}(1-\alpha)$ is the quantile of order $1-\alpha$ of the distribution of the random variable $s(X)$.

Under regularity assumptions, the MV curve is also given by the parametric curve

$$
(\mathbb{P}(s(X) \geq t), \lambda(x, s(x) \geq t))
$$

Assume that the random variable $X$ with distribution $P$ has a density $f$ with respect to the Lebesgue measure. Furthermore assume that for all $\alpha \in (0,1)$ the Minimum Volume set optimization problem
$$
\min_{\Omega \in \mathcal{M}} \lambda(\Omega) \quad \text{ s.t } \quad P(\Omega) \geq \alpha .
$$
has a unique solution given by
$$
\Omega^*_{\alpha} = \{x, f(x) \geq F_f^{-1}(1-\alpha) \}.
$$

1. Show that for all $\alpha \in (0,1)$, $MV_s(\alpha) - MV_f(\alpha) \leq \lambda(\Omega^*_{\alpha} \Delta \{x, s(x) \geq F_s^{-1}(1-\alpha) \})$.

2. Show that for all $\alpha \in (0,1)$, $MV_f(\alpha) \leq MV_s(\alpha)$. Hint: use the following property: for all $\alpha \in (0,1)$, $\mathbb{P}(s(X) < F_s^{-1}(1-\alpha)) \leq 1-\alpha$.

3. In what sense a scoring function minimizing the area under the Mass Volume curve is optimal?

## 2.1. Using Mass Volume curve for performance evaluation

The lower $MV_s$ the better the scoring function $s$. The goal is to compare the performance of scoring functions given by the One-Class SVM for different values of $\gamma$ using the Mass Volume curve. To easily plot the Mass Volume we are going to use the ROC curve function given by `sklearn`. The idea is to use a uniform sample as the positive class and to consider the Gaussian mixture data set as the negative class. To plot the MV curve we want to plot the volume against the mass $\alpha$. This will be given by $1-TPR$ against $1-FPR$.

1. Using `sklearn.model_selection.ShuffleSplit` split the data set into a training set and a test set.

2. Generate `n_sim=100000` uniform random variables in the hypercube enclosing the data using `numpy.random.uniform`.

3. Build a training set consisting in the concatenation of the training set of the Gaussian mixture data set (obtained in 1.) and the uniform sample.

4. Similarly, build the test set by concatenating the test set of the Gaussian mixture data set (obtained in 1.) and the uniform sample.

1. Compute the Mass Volume curve, estimated on the training set, of the scoring function obtained with the One-Class SVM for $\gamma=0.05$ and $\gamma=10$. Compute the area under the Mass Volume curve for each value of $\gamma$.

2. Compute the Mass Volume curve estimated on the test set for each value of $\gamma$. Compute the area under the Mass Volume curve for each value of $\gamma$.

# 3. Performance evaluation on real data sets

## 3.1. Shuttle data set

```python
from sklearn.datasets import fetch_mldata

dataset = fetch_mldata('shuttle')
X = dataset.data
y = dataset.target

### Usual setup when using this data set for anomaly detection
# Instances with label 4 are removed
# Normal data: label 1
ind = (y != 4)
X = X[ind, :]
y = y[ind]
y = (y == 1).astype(int)
```

```python
n_samples, n_features = X.shape
n_samples, n_features
```

    (85849, 9)

1. What's the proportion of anomalies in the data set (label 0)?
2. Split the data set into a training a test set using `StratifiedShuffleSplit` to preserve the proportion of each class in the training and test sets.
3. Compare the performance of the OneClass SVM and Isolation Forest using the ROC curve (computed with the true labels) on the train set and the set set:
    - train the algorithm on the training set (without the labels), compute the ROC curve on the training set using the labels.
    - train the algorithm on the training set (without the labels), compute the ROC curve on the test set using the labels
4. Compare the performance of the OneClass SVM and Isolation Forest using the Mass Volume curve on the training and test sets

## 3.2. Http data set - Network intrusion detection

1. Same questions with the http data set

```python
from sklearn.datasets import fetch_kddcup99

dataset = fetch_kddcup99(subset='http', shuffle=True,
                         percent10=True, random_state=SEED)
X = dataset['data']
X = X.astype('float')
y = dataset['target']
y = (y == 'normal.').astype(int)
```

```python
n_samples, n_features = X.shape
```

### Novelty detection

Now we only train the model on normal data. For instance let's train the model on half the normal data.

1. Split the data set into a set with the normal data and a set with the anomalies.

2. Randomly split the normal data set into a two data sets: a training set and a test set.

3. Build a test set from the normal test set and the anomalies.

4. Train the OneClass SVM and Isolation Forest on the normal training set and compute the ROC curve on the test set.

#### ROC - AUC on test set

#### MV curve on test set

1. Plot the MV curve estimated on the test set for the OneClass SVM and Isolation Forest.

# 4. Optional

1. Show that a density level set $\{f \geq \tau \}$ is a Minimum volume set with mass $\alpha = \mathbb{P}(f(X) \geq \tau)$.

2. Show that if the density has not flat parts, the minimum volume set optimization problem has a solution.

# 5. Overfitting for Minimum volume set estimation

Let $\lambda$ denotes the Lebesgue measure of $\mathbb{R}^d$ and let $\mathcal{M}$ be the class of all measurable sets. A Minimum Volume set with mass at least $\alpha$ is the solution of the following optimization problem:
$$
\min_{\Omega \in \mathcal{M}} \lambda(\Omega) \quad \text{ s.t } \quad P(\Omega) \geq \alpha .
$$
We do not know the distribution $P$ but only have a sample $X_1, \dots, X_n$ drawn from this distribution.

1. What is the solution for all $\alpha \in (0,1)$ of the empirical optimization problem if we solve the following empirical version of the Minimum Volume set optimization problem:
$$
\min_{\Omega \in \mathcal{M}} \lambda(\Omega) \quad \text{ s.t } \quad \widehat P(\Omega) \geq \alpha
$$
where for all $\Omega$, $\widehat P(\Omega) = \frac{1}{n}\sum_{i=1}^n \mathbb{I}\{X_i \in \Omega \}$.