 ILI285 - Computación Científica I  / INF285 - Computación Científica

 Roots of 1D equations

  [S]cientific [C]omputing [T]eam

 Version: 1.32

## Table of Contents
* [Introduction](#intro)
* [Bisection Method](#bisection)
* [Cobweb Plot](#cobweb)
* [Fixed Point Iteration](#fpi)
* [Newton Method](#nm)
* [Wilkinson Polynomial](#wilkinson)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
%matplotlib inline
from ipywidgets import interact
from ipywidgets import widgets
sym.init_printing()
from scipy import optimize
```

## Introduction

Hello again! In this document we're going to learn how to find a 1D equation's solution using numerical methods. First, let's start with the definition of a root:

Definition: The function $f(x)$ has a root in $x = r$ if $f(r) = 0$.

An example: Let's say we want to solve the equation $x + \log(x) = 3$. We can rearrange the equation: $x + \log(x) - 3 = 0$. That way, to find its solution we can find the root of $f(x) = x + \log(x) - 3$. Now let's study some numerical methods to solve these kinds of problems.

Defining a function $f(x)$

```python
f = lambda x: x+np.log(x)-3
```

Finding $r$ using sympy

```python
y = sym.Symbol('y')
fsym = lambda y: y+sym.log(y)-3
r_all=sym.solve(sym.Eq(fsym(y), 0), y)
r=r_all[0].evalf()
print(r)
print(r_all)
```

    2.20794003156932
    [LambertW(exp(3))]

```python
def find_root_manually(r=2.0):
    x = np.linspace(1,3,1000)
    plt.figure(figsize=(8,8))
    plt.plot(x,f(x),'b-')
    plt.grid()
    plt.ylabel('$f(x)$',fontsize=16)
    plt.xlabel('$x$',fontsize=16)
    plt.title('What is r such that $f(r)='+str(f(r))+'$? $r='+str(r)+'$',fontsize=16)
    plt.plot(r,f(r),'k.',markersize=20)
    plt.show()
interact(find_root_manually,r=(1e-5,3,1e-3))
```

    interactive(children=(FloatSlider(value=2.0, description='r', max=3.0, min=1e-05, step=0.001), Output()), _dom…

    <function __main__.find_root_manually(r=2.0)>

## Bisection Method

The bisection method finds the root of a function $f$, where $f$ is a **continuous** function.
If we want to know if this has a root, we have to check if there is an interval $[a,b]$ for which
$f(a)\cdot f(b) < 0$. When these 2 conditions are satisfied, it means that there is a value $r$, between $a$ and $b$, for which $f(r) = 0$. To summarize how this method works, start with the aforementioned interval (checking that there's a root in it), and split it into two smaller intervals $[a,c]$ and $[c,b]$. Then, check which of the two intervals contains a root. Keep splitting each "eligible" interval until the algorithm converges or the tolerance is surpassed.

```python
def bisect(f, a, b, tol=1e-8):
    fa = f(a)
    fb = f(b)
    i = 0
    # Just checking if the sign is not negative => not root  necessarily
    if np.sign(f(a)*f(b)) >= 0:
        print('f(a)f(b)<0 not satisfied!')
        return None

    #Printing the evolution of the computation of the root
    print(' i |     a     |     c     |     b     |     fa    |     fc     |     fb     |   b-a')
    print('----------------------------------------------------------------------------------------')

    while(b-a)/2 > tol:
        c = (a+b)/2.
        fc = f(c)
        print('%2d | %.7f | %.7f | %.7f | %.7f | %.7f | %.7f | %.7f' %
              (i+1, a, c, b, fa, fc, fb, b-a))
        # Did we find the root?
        if fc == 0:
            print('f(c)==0')
            break
        elif np.sign(fa*fc) < 0:
            b = c
            fb = fc
        else:
            a = c
            fa = fc
        i += 1

    xc = (a+b)/2.
    return xc
```

```python
## Finding a root of cos(x). What about if you change the interval?
#f = lambda x: np.cos(x)
## Another function
#f = lambda x: x**3-2*x**2+(4/3)*x-(8/27)
## Computing the cubic root of 7.
#f = lambda x: x**3-7
#bisect(f,0,2)
f = lambda x: x*np.exp(x)-3
#f2 = lambda x: np.cos(x)-x
bisect(f,0,3,tol=1e-13)
```

It's very important to define a concept called **convergence rate**.
This rate shows how fast the convergence of a method is at a specified point.

The convergence rate for the bisection is always 0.5 because this method uses the half of the interval for each iteration.

## Cobweb Plot

```python
def cobweb(x,g=None):
    min_x = np.amin(x)
    max_x = np.amax(x)

    plt.figure(figsize=(10,10))
    ax = plt.axes()
    plt.plot(np.array([min_x,max_x]),np.array([min_x,max_x]),'b-')
    for i in np.arange(x.size-1):
        delta_x = x[i+1]-x[i]
        head_length =  np.abs(delta_x)*0.04
        arrow_length = delta_x-np.sign(delta_x)*head_length
        ax.arrow(x[i], x[i], 0, arrow_length, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')
        ax.arrow(x[i], x[i+1], arrow_length, 0, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')

    if g!=None:
        y = np.linspace(min_x,max_x,1000)
        plt.plot(y,g(y),'r')

    plt.title('Cobweb diagram')
    plt.grid(True)
    plt.show()
```

## Fixed Point Iteration

To learn about the Fixed-Point Iteration we will first learn about the concept of Fixed Point.

A Fixed Point of a function $g$ is a real number $r$, where $g(r) = r$

The Fixed-Point Iteration is based in the Fixed Point concept and works like this to find the root of a function:

\begin{equation} x_{0} = initial\_guess \\ x_{i+1} = g(x_{i})\end{equation}

To find an equation's solution using this method you'll have to move around some things to rearrange the equation in the form $x = g(x)$. That way, you'll be iterating over the funcion $g(x)$, but you will **not** find $g$'s root, but $f(x) = g(x) - x$ (or $f(x) = x - g(x)$)'s root. In our following example, we'll find the solution to $f(x) = x - \cos(x)$ by iterating over the funcion $g(x) = \cos(x)$.

```python
def fpi(g, x0, k, flag_cobweb=False):
    x = np.empty(k+1)
    x[0] = x0
    error_i = np.nan

    print(' i |     x(i)     |    x(i+1)    ||x(i+1)-x(i)| | e_i/e_{i-1}')
    print('--------------------------------------------------------------')

    for i in range(k):
        x[i+1] = g(x[i])
        error_iminus1 = error_i
        error_i = abs(x[i+1]-x[i])
        print('%2d | %.10f | %.10f | %.10f | %.10f' %
              (i,x[i],x[i+1],error_i,error_i/error_iminus1))
    if flag_cobweb:
        cobweb(x,g)
    return x[-1]
```

```python
g = lambda x: np.cos(x)
fpi2(g, 2, 20, True)
```

Let's quickly explain the Cobweb Diagram we have here. The blue line is the function $x$ and the red is the function $g(x)$. The point in which they meet is $g$'s fixed point. In this particular example, we start at $y = x = 1.5$ (the top right corner) and then we "jump" **vertically** to $y = \cos(1.5) \approx 0.07$. After this, we jump **horizontally** to $x = \cos(1.5) \approx 0.07$. Then, we jump again **vertically** to $y = \cos\left(\cos(1.5)\right) \approx 0.997$ and so on. See the pattern here? We're just iterating over $x = \cos(x)$, getting closer to the center of the diagram where the fixed point resides, in $x \approx 0.739$.

It's very important to mention that the algorithm will converge only if the rate of convergence $S < 1$, where $S = \left| g'(r) \right|$. If you want to use this method, you'll have to construct $g(x)$ starting from $f(x)$ accordingly. In this example, $g(x) = \cos(x) \Rightarrow g'(x) = -\sin(x)$ and $|-\sin(0.739)| \approx 0.67$.

### Another example. Source: https://divisbyzero.com/2008/12/18/sharkovskys-theorem/amp/?__twitter_impression=true

```python
g = lambda x: -(3/2)*x**2+(11/2)*x-2
gp = lambda x: -3*x+11/2

a=-1/2.7
g2 = lambda x: x+a*(x-g(x))

#x=np.linspace(2,3,100)
#plt.plot(x,gp(x),'-')
#plt.plot(x,gp(x)*0+1,'r-')
#plt.plot(x,gp(x)*0-1,'g-')
#plt.grid(True)
#plt.show()
fpi(g2, 2.45, 12, True)
```

## Newton's Method

For this method, we want to iteratively find some function $f(x)$'s root, that is, the number $r$ for which $f(r) = 0$. The algorithm is as follows:

\begin{equation} x_0 = initial\_guess \end{equation}
\begin{equation} x_{i+1} = x_i - \cfrac{f(x_i)}{f'(x_i)} \end{equation}

which means that you won't be able to find $f$'s root if $f'(r) = 0$. In this case, you would have to use the modified version of this method, but for now let's focus on the unmodified version first. Newton's (unmodified) method is said to have quadratic convergence.

```python
def newton_method(f, fp, x0, rel_error=1e-8, m=1):
    #Initialization of hybrid error and absolute
    hybrid_error = 100
    error_i = np.inf

    print('i |     x(i)     |     x(i+1)   |      |x(i+1)-x(i)|     |  e_i/e_{i-1} | e_i/e_{i-1}^2')
    print('----------------------------------------------------------------------------------------')

    #Iteration counter
    i = 1
    while (hybrid_error > rel_error and hybrid_error < 1e12 and i < 1e4):
        #Newton's iteration
        x1 = x0-m*f(x0)/fp(x0)

        #Checking if root was found
        if f(x1) == 0.0:
            hybrid_error = 0.0
            break

        #Computation of hybrid error
        hybrid_error = abs(x1-x0)/np.max([abs(x1),1e-12])

        #Computation of absolute error
        error_iminus1 = error_i
        error_i = abs(x1-x0)

        #Increasing counter
        i += 1

        #Showing some info
        print("%d | %.10f | %.10f | %.20f | %.10f | %.10f" %
              (i, x0, x1, error_i, error_i/error_iminus1, error_i/(error_iminus1**2)))
        #Updating solution
        x0 = x1

    #Checking if solution was obtained
    if hybrid_error < rel_error:
        return x1
    elif i>=1e4:
        print('Newton''s Method diverged. Too many iterations!!')
        return None
    else:
        print('Newton''s Method diverged!')
        return None
```

```python
f = lambda x: np.sin(x)
fp = lambda x: np.cos(x) # the derivative of f
newton_method(f, fp, 3.1,rel_error=1e-14)
```

```python
f = lambda x: x**2
fp = lambda x: 2*x # the derivative of f
newton_method(f, fp, 3.1,rel_error=1e-2, m=2)
```

## Wilkinson Polynomial
https://en.wikipedia.org/wiki/Wilkinson%27s_polynomial

**Final question: Why is the root far far away from $16$?**

```python
x = sym.symbols('x', reals=True)
W=1
for i in np.arange(1,21):
        W*=(x-i)
W # Printing W nicely
```

```python
# Expanding the Wilkinson polynomial
We=sym.expand(W)
We
```

```python
# Just computiong the derivative
Wep=sym.diff(We,x)
Wep
```

```python
# Lamdifying the polynomial to be used with sympy
P=sym.lambdify(x,We)
Pp=sym.lambdify(x,Wep)
```

```python
# Using scipy function to compute a root
root = optimize.newton(P,16)
print(root)
```

# Acknowledgements
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. March 2016._ v1.1.
* _Update April 2020 - v1.32 - C.Torres_ : Re-ordering the notebook.

# Propose Classwork
Build a FPI such that given $x$ computes $\displaystyle \frac{1}{x}$. Write down your solution below or go and see the [solution](#sol1)

```python
print('Please try to think and solve before you see the solution!!!')
```

# In class

### From the textbook

```python
g1 = lambda x: 1-x**3
g2 = lambda x: (1-x)**(1/3)
g3 = lambda x: (1+2*x**3)/(1+3*x**2)
fpi(g3, 0.5, 10, True)
```

```python
g1p = lambda x: -3*x**2
g2p = lambda x: -(1/3)*(1-x)**(-2/3)
g3p = lambda x: ((1+3*x**2)*(6*x**2)-(1+2*x**3)*6*x)/((1+3*x**2)**2)
r=0.6823278038280194
print(g3p(r))
```

    0.0

### Adding another implementation of FPI including and extra column for analyzing quadratic convergence

```python
def fpi2(g, x0, k, flag_cobweb=False):
    x = np.empty(k+1)
    x[0] = x0
    error_i = np.inf

    print(' i |     x(i)     |    x(i+1)    ||x(i+1)-x(i)| | e_i/e_{i-1} | e_i/e_{i-1}^2')
    print('-----------------------------------------------------------------------------')

    for i in range(k):
        x[i+1] = g(x[i])
        error_iminus1 = error_i
        error_i = abs(x[i+1]-x[i])
        print('%2d | %.10f | %.10f | %.10f | %.10f | %.10f' %
              (i,x[i],x[i+1],error_i,error_i/error_iminus1,error_i/(error_iminus1**2)))
    if flag_cobweb:
        cobweb(x,g)
    return x[-1]
```

Which function shows quadratic convergence? Why?

```python
g1 = lambda x: (4./5.)*x+1./x
g2 = lambda x: x/2.+5./(2*x)
g3 = lambda x: (x+5.)/(x+1)
fpi2(g1, 3.0, 30, True)
```

### Building a FPI to compute the cubic root of 7

```python
# What is 'a'? Can we find another 'a'?
a = -3*(1.7**2)
print(a)
```

    -8.669999999999998

```python
f = lambda x: x**3-7
g =  lambda x: f(x)/a+x
r=fpi(g, 1.7, 14, True)
print(f(r))
```

### Playing with some roots

```python
f = lambda x: 8*x**4-12*x**3+6*x**2-x
fp = lambda x: 32*x**3-36*x**2+12*x-1

x = np.linspace(-1,1,1000)
plt.figure(figsize=(10,10))

plt.title('What are we seeing with the semiloigy plot? Is this function differentiable?')
plt.semilogy(x,np.abs(f(x)),'b-')
plt.semilogy(x,np.abs(fp(x)),'r-')
plt.grid()
plt.ylabel('$f(x)$',fontsize=16)
plt.xlabel('$x$',fontsize=16)
plt.show()
```

```python
r=newton_method(f, fp, 0.3, rel_error=1e-8, m=1)
print([r,f(r)])
# Is this showing quadratic convergence? If not, can you fix it?
```

    i |     x(i)     |     x(i+1)   |      |x(i+1)-x(i)|     |  e_i/e_{i-1} | e_i/e_{i-1}^2
    ----------------------------------------------------------------------------------------
    2 | 0.3000000000 | 0.3857142857 | 0.08571428571428552079 | 0.0000000000 | 0.0000000000
    3 | 0.3857142857 | 0.4279843444 | 0.04227005870841460400 | 0.4931506849 | 5.7534246575
    4 | 0.4279843444 | 0.4534159993 | 0.02543165489222543041 | 0.6016470208 | 14.2334086872
    5 | 0.4534159993 | 0.4694946399 | 0.01607864055554952820 | 0.6322294252 | 24.8599404138
    6 | 0.4694946399 | 0.4798882000 | 0.01039356016793291371 | 0.6464203321 | 40.2036683316
    7 | 0.4798882000 | 0.4866871127 | 0.00679891264490944947 | 0.6541466576 | 62.9376890188
    8 | 0.4866871127 | 0.4911655766 | 0.00447846388652911598 | 0.6587029604 | 96.8835745915
    9 | 0.4911655766 | 0.4941281466 | 0.00296257004712746630 | 0.6615147788 | 147.7101960697
    10 | 0.4941281466 | 0.4960932149 | 0.00196506826728892747 | 0.6632984996 | 223.8929338621
    11 | 0.4960932149 | 0.4973989041 | 0.00130568918550205693 | 0.6644497839 | 338.1306364449
    12 | 0.4973989041 | 0.4982674500 | 0.00086854596470975487 | 0.6652011630 | 509.4636383483
    13 | 0.4982674500 | 0.4988456368 | 0.00057818680119792187 | 0.6656951096 | 766.4477606024
    14 | 0.4988456368 | 0.4992307216 | 0.00038508477299548094 | 0.6660213831 | 1151.9138480345
    15 | 0.4992307216 | 0.4994872795 | 0.00025655790749512519 | 0.6662374767 | 1730.1060012498
    16 | 0.4994872795 | 0.4996582449 | 0.00017096535582211692 | 0.6663811593 | 2597.3908416937
    17 | 0.4996582449 | 0.4997721891 | 0.00011394425784067019 | 0.6664757155 | 3898.3085915658
    18 | 0.4997721891 | 0.4998481375 | 0.00007594838735824894 | 0.6665398397 | 5849.7010060220
    19 | 0.4998481375 | 0.4998987634 | 0.00005062583760417905 | 0.6665821272 | 8776.7778935922
    20 | 0.4998987634 | 0.4999325121 | 0.00003374875558825874 | 0.6666310561 | 13167.8029965013
    21 | 0.4999325121 | 0.4999550059 | 0.00002249375282564747 | 0.6665061403 | 19749.0582583161
    22 | 0.4999550059 | 0.4999700110 | 0.00001500514206059789 | 0.6670804190 | 29656.2527473931
    23 | 0.4999700110 | 0.4999800060 | 0.00000999499746007215 | 0.6661048206 | 44391.7703605403
    24 | 0.4999800060 | 0.4999866717 | 0.00000666569396912120 | 0.6669030178 | 66723.6805685757
    25 | 0.4999866717 | 0.4999909946 | 0.00000432287884150062 | 0.6485264492 | 97293.1629087970
    26 | 0.4999909946 | 0.4999941319 | 0.00000313735547630145 | 0.7257560508 | 167887.2060628857
    27 | 0.4999941319 | 0.4999956097 | 0.00000147777639253333 | 0.4710261249 | 150134.7642908100
    28 | 0.4999956097 | 0.4999970497 | 0.00000144001474572386 | 0.9744469820 | 659400.8314715016
    29 | 0.4999970497 | 0.4999954553 | 0.00000159440131292099 | 1.1072117960 | 768889.2070839963
    30 | 0.4999954553 | 0.4999967992 | 0.00000134384518901687 | 0.8428525354 | 528632.6149851314
    31 | 0.4999967992 | 0.4999977022 | 0.00000090303564465044 | 0.6719789244 | 500041.9169360077
    32 | 0.4999977022 | 0.4999950737 | 0.00000262848010768035 | 2.9107157876 | 3223256.8059490011
    33 | 0.4999950737 | 0.4999964081 | 0.00000133432963278501 | 0.5076430401 | 193131.7793302043
    34 | 0.4999964081 | 0.4999982008 | 0.00000179273154921056 | 1.3435447322 | 1006906.1641414912
    35 | 0.4999982008 | 0.4999939138 | 0.00000428698813359496 | 2.3913162768 | 1333895.3497373802
    36 | 0.4999939138 | 0.4999960369 | 0.00000212305952357328 | 0.4952333567 | 115520.1137200339
    37 | 0.4999960369 | 0.4999978040 | 0.00000176715078742395 | 0.8323604533 | 392057.0497707436
    38 | 0.4999978040 | 0.4999987633 | 0.00000095926573961957 | 0.5428318548 | 307179.1375397388
    39 | 0.4999987633 | 0.4999957387 | 0.00000302458381729043 | 3.1530197445 | 3286909.5750255324
    40 | 0.4999957387 | 0.4999965029 | 0.00000076425488204634 | 0.2526810061 | 83542.4049666695
    41 | 0.4999965029 | 0.4999968812 | 0.00000037826747445457 | 0.4949493727 | 647623.4360170967
    42 | 0.4999968812 | 0.4999992591 | 0.00000237792675222837 | 6.2863632557 | 16618831.0658740196
    43 | 0.4999992591 | 0.5000161146 | 0.00001685544768070812 | 7.0882955772 | 2980872.1276002550
    44 | 0.5000161146 | 0.5000106637 | 0.00000545084237957294 | 0.3233875767 | 19185.9381501386
    45 | 0.5000106637 | 0.5000070840 | 0.00000357972935949302 | 0.6567295677 | 120482.2157677659
    46 | 0.5000070840 | 0.5000046874 | 0.00000239665618528839 | 0.6695076484 | 187027.4484852393
    47 | 0.5000046874 | 0.5000025819 | 0.00000210541208811588 | 0.8784789829 | 366543.5986752341
    48 | 0.5000025819 | 0.4999998063 | 0.00000277564977962941 | 1.3183403835 | 626167.3859103953
    49 | 0.4999998063 | 0.5003696288 | 0.00036982248520706085 | 133.2381656797 | 48002513.3781396449
    50 | 0.5003696288 | 0.5002464495 | 0.00012317931164718132 | 0.3330768587 | 900.6398259087
    51 | 0.5002464495 | 0.5001643129 | 0.00008213655673550146 | 0.6668048038 | 5413.2856799283
    52 | 0.5001643129 | 0.5001095469 | 0.00005476602556908627 | 0.6667679745 | 8117.7979821256
    53 | 0.5001095469 | 0.5000730351 | 0.00003651177553432028 | 0.6666866028 | 12173.3610546054
    54 | 0.5000730351 | 0.5000486897 | 0.00002434541746509922 | 0.6667826231 | 18262.1253924256
    55 | 0.5000486897 | 0.5000324687 | 0.00001622103474174796 | 0.6662869826 | 27368.0656125717
    56 | 0.5000324687 | 0.5000216575 | 0.00001081118876600229 | 0.6664919309 | 41088.1267130889
    57 | 0.5000216575 | 0.5000144386 | 0.00000721887681853772 | 0.6677227616 | 61762.1961847271
    58 | 0.5000144386 | 0.5000096458 | 0.00000479276611831114 | 0.6639213050 | 91970.1667908346
    59 | 0.5000096458 | 0.5000063645 | 0.00000328135142357855 | 0.6846466826 | 142850.0088969091
    60 | 0.5000063645 | 0.5000040805 | 0.00000228400611013146 | 0.6960565375 | 212124.9593909390
    61 | 0.5000040805 | 0.5000024135 | 0.00000166697042569552 | 0.7298449940 | 319545.9901534105
    62 | 0.5000024135 | 0.5000071783 | 0.00000476480902644738 | 2.8583644635 | 1714706.1635887653
    63 | 0.5000071783 | 0.5000052033 | 0.00000197501001863998 | 0.4144993026 | 86991.7976422348
    64 | 0.5000052033 | 0.5000034947 | 0.00000170858392567474 | 0.8651013967 | 438023.8016830212
    65 | 0.5000034947 | 0.5000012221 | 0.00000227260331253643 | 1.3301092667 | 778486.3516206152
    66 | 0.5000012221 | 0.4999888330 | 0.00001238911739925852 | 5.4515089945 | 2398794.7938135080
    67 | 0.4999888330 | 0.4999925798 | 0.00000374679111453391 | 0.3024259916 | 24410.6163381144
    68 | 0.4999925798 | 0.4999951003 | 0.00000252055427996112 | 0.6727234593 | 179546.5609816919
    69 | 0.4999951003 | 0.4999962565 | 0.00000115617629375953 | 0.4586992246 | 181983.4741264939
    [0.4999965866182223, 0.0]

# Solutions

Problem: Build a FPI such that given $x$ computes $\displaystyle \frac{1}{x}$

```python
# We are finding the 1/a
# Solution code:
a = 2.1
g = lambda x: 2*x-a*x**2
gp = lambda x: 2-2*a*x
r=fpi2(g, 0.7, 7, flag_cobweb=True)
print([r,1/a])
# Are we seeing quadratic convergence?
```

### What is this plot telling us?

```python
xx=np.linspace(0.2,0.8,1000)
plt.figure(figsize=(10,10))
plt.plot(xx,g(xx),'-',label=r'$g(x)$')
plt.plot(xx,gp(xx),'r-',label=r'$gp(x)$')
plt.plot(xx,xx,'g-',label=r'$x$')
plt.plot(xx,0*xx+1,'k--')
plt.plot(xx,0*xx-1,'k--')
plt.legend(loc='best')
plt.grid()
plt.show()
```

```python

```


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

# Useful Inequalities

In practice, few quantities can be analytically
calculated. Some knowledge
of bounding inequalities helps find the ballpark for
potential solutions. This
sections discusses three key inequalities that are
important for
probability, statistics, and machine learning.

## Markov's
Inequality

Let $X$ be a non-negative random variable
and suppose that
$\mathbb{E}(X) < \infty$. Then,
for any $t>0$,

$$
\mathbb{P}(X>t)\leq \frac{\mathbb{E}(X)}{t}
$$

 This is a foundational inequality that is
used as a stepping stone to other
inequalities. It is easy
to prove. Because $X>0$, we have the following,

$$
\begin{align*}
\mathbb{E}(X)&=\int_0^\infty x f_x(x)dx =\underbrace{\int_0^t
x f_x(x)dx}_{\text{omit this}}+\int_t^\infty x f_x(x)dx \\\
&\ge\int_t^\infty x f_x(x)dx \ge t\int_t^\infty f_x(x)dx = t \mathbb{P}(X>t)
\end{align*}
$$

 The step that establishes the inequality is the part where the
$\int_0^t x
f_x(x)dx$ is omitted.  For a particular $f_x(x)$ that may be
concentrated around
the $[0,t]$ interval, this could be a lot to throw out.
For that reason, the
Markov Inequality is considered a *loose* inequality,
meaning that there is a
substantial gap between both sides of the inequality.
For example, as shown in
[Figure](#fig:ProbabilityInequalities_001), the
$\chi^2$ distribution has a lot
of its mass on the left, which would be omitted
in the  Markov Inequality.
[Figure](#fig:ProbabilityInequalities_002) shows
the two curves established by
the Markov Inequality. The gray shaded region is
the gap between the two terms
and indicates that looseness of the bound
(fatter shaded region) for this case.

The $\chi_1^2$ density has much
of its weight on the left, which is excluded in the establishment of the Markov
Inequality.

The shaded area
shows the region between the curves on either side of the Markov Inequality.

## Chebyshev's Inequality

Chebyshev's Inequality drops out
directly from the Markov Inequality.  Let
$\mu=\mathbb{E}(X)$ and
$\sigma^2=\mathbb{V}(X)$. Then, we have

$$
\mathbb{P}(\vert X-\mu\vert \ge t) \le \frac{\sigma^2}{t^2}
$$

 Note that if we normalize so that $Z=(X-\mu)/\sigma$, we
have $\mathbb{P}(\vert
Z\vert \ge k) \le 1/k^2$. In particular,
$\mathbb{P}(\vert Z\vert \ge 2) \le
1/4$. We can illustrate this
inequality using Sympy statistics module,

```python
import sympy
import sympy.stats as ss
t=sympy.symbols('t',real=True)
x=ss.ChiSquared('x',1)
```

To get the left side of the Chebyshev inequality, we
have to write this out as
the following conditional probability,

```python
r = ss.P((x-1) > t,x>1)+ss.P(-(x-1) > t,x<1)
```

We could take the above expression, which is a function of $t$ and
attempt to
compute the integral, but that would take a very long time (the
expression is
very long and complicated, which is why we did not print it out
above).  In this
situation, it's better to use the built-in cumulative density
function as in the
following (after some rearrangement of the terms),

```python
w=(1-ss.cdf(x)(t+1))+ss.cdf(x)(1-t)
```

To plot this, we can evaluated at a variety of `t` values by using
the `.subs`
substitution method, but it is more convenient to use the
`lambdify` method to
convert the expression to a function.

```python
fw=sympy.lambdify(t,w)
```

Then, we can evaluate this function using something like

```python
[fw(i) for i in [0,1,2,3,4,5]]
```

    [1.0,
     0.157299207050285,
     (0.08326451666355039+0j),
     (0.04550026389635842+0j),
     (0.0253473186774682+0j),
     (0.014305878435429631+0j)]

to produce the following [Figure](#fig:ProbabilityInequalities_003).

The
shaded area shows the region between the curves on either side of the Chebyshev
Inequality.

**Programming Tip.**

Note that we cannot use
vectorized inputs for the `lambdify` function because
it contains embedded
functions that are only available in Sympy. Otherwise, we
could have used
`lambdify(t,fw,numpy)` to specify the corresponding functions
in Numpy to use
for the expression.

## Hoeffding's Inequality

Hoeffding's Inequality is similar, but less loose,
than Markov's Inequality.
Let $X_1,\ldots,X_n$ be iid observations such that
$\mathbb{E}(X_i)=\mu$ and
$a\le X_i \le b$. Then, for any $\epsilon>0$, we have

$$
\mathbb{P}(\vert \overline{X}_n -\mu\vert \ge \epsilon) \le 2 \exp(-2
n\epsilon^2/(b-a)^2)
$$

 where $\overline{X}_n = \tfrac{1}{n}\sum_i^n X_i$. Note that we
further assume
that the individual random variables are bounded.

**Corollary.** If
$X_1,\ldots,X_n$ are independent with $\mathbb{P}(a\le X_i\le b)=1$
and all with
$\mathbb{E}(X_i)=\mu$. Then, we have

$$
\vert\overline{X}_n-\mu\vert \le \sqrt{\frac{c}{2 n}\log \frac{2}{\delta}}
$$

 where $c=(b-a)^2$. We will see this inequality again in the machine
learning
chapter. [Figure](#fig:ProbabilityInequalities_004) shows the Markov
and
Hoeffding bounds for the case of ten identically and uniformly distributed
random variables, $X_i \sim \mathcal{U}[0,1]$.  The solid line shows
$\mathbb{P}(\vert \overline{X}_n - 1/2 \vert > \epsilon)$.  Note that the
Hoeffding Inequality is tighter than the Markov Inequality and that both of
them
merge when $\epsilon$ gets big enough.

This shows the Markov and Hoeffding bounds for the case of ten identically
and uniformly distributed random variables.

### Proof of Hoeffding's Inequality

We will need the following lemma to prove
Hoeffding's inequality.

**Lemma** Let $X$ be a random variable with
$\mathbb{E}(X)=0$ and
$a\le X\le b$. Then, for any $s>0$, we have the following,

$$
\begin{equation}
\mathbb{E}(e^{s X}) \le e^{s^2(b-a)^2/8}
\label{_auto1} \tag{1}
\end{equation}
$$

Because $X$ is contained in the closed interval $[a,b]$, we can write it as a
convex
combination of the endpoints of the interval.

$$
X = \alpha_1 a + \alpha_2 b
$$

 where $\alpha_1+\alpha_2=1$. Solving for the $\alpha_i$ terms, we have

$$
\begin{align*}
\alpha_1 = & \frac{x-a}{b-a} \\
\alpha_2 = &
\frac{b-x}{b-a}
\end{align*}
$$

 From Jensen's inequality,  for a convex functions $f$, we know that

$$
f\left(\sum \alpha_i x_i\right) \le \sum \alpha_i f(x_i)
$$

 Given the convexity of $e^X$, we therefore have,

$$
e^{s X} \le \alpha_1 e^{s a} + \alpha_2 e^ {s b}
$$

 With $\mathbb{E}(X)=0$, we can write the expectation of both sides

$$
\mathbb{E}(e^{s X}) \le \mathbb{E}(\alpha_1) e^{s a}
+\mathbb{E}(\alpha_2) e^{s b}
$$

 with $\mathbb{E}(\alpha_1)=\frac{b}{b-a}$ and
$\mathbb{E}(\alpha_2)=\frac{-a}{b-a}$. Thus, we have

$$
\mathbb{E}(e^{s X}) \le \frac{b}{b-a} e^{s a} -\frac{a}{b-a} e^{s b}
$$

 Using $p:=\frac{-a}{b-a}$, we can rewrite the following,

$$
\frac{b}{b-a} e^{s a} -\frac{a}{b-a} e^{s b} = (1-p)e^{s a} + p e^{s b} =:
e^{\phi(u)}
$$

 where

$$
\phi(u)=-p u + \log(1-p+p e^{u})
$$

 and $u=s(b-a)$. Note that $\phi(0)=\phi'(0)=0$. Also, $\phi''(0) = p(1-p)\le
1/4$. Thus,
the Taylor expansion of $\phi(u)\approx \frac{u^2}{2}\phi''(t) \le
\frac{u^2}{8}$ for
$t\in [0,u] \blacksquare$.

To prove Hoeffding's inequality,
we start with Markov's inequality,

$$
\mathbb{P}(X\ge\epsilon)\le \frac{\mathbb{E}(X)}{\epsilon}
$$

 Then, given $s>0$, we have the following,

$$
\mathbb{P}(X\ge\epsilon)=\mathbb{P}(e^{s X} \ge e^{s\epsilon}) \le
\frac{\mathbb{E}(e^{s X})}{e^{s \epsilon}}
$$

 We can write the one-sided Hoeffding inequality as the following,

$$
\begin{align*}
\mathbb{P}(\overline{X}_n -\mu\ge\epsilon) & \le
e^{-s\epsilon}\mathbb{E}(\exp(\frac{s}{n}\sum_{i=1}^n (X_i-\mathbb{E}(X_i)))) \\
& = e^{-s\epsilon}\prod_{i=1}^n\mathbb{E}(e^{ \frac{s}{n} (X_i-\mathbb{E}(X_i))
}) \\
    & \le e^{-s\epsilon}\prod_{i=1}^n e^{\frac{s^2}{n^2}(b-a)^2/8 } \\
& = e^{-s\epsilon} e^{\frac{s^2}{n}(b-a)^2/8}
\end{align*}
$$

 Now, we want to pick $s>0$ to minimize this upper bound. Then, with
$s=\frac{4
n\epsilon}{(b-a)^2}$

$$
\mathbb{P}(\overline{X}_n-\mu\ge\epsilon)\le e^{-\frac{2
n\epsilon^2}{(b-a)^2}}
$$

 The other side of the inequality follows similarly to obtain Hoeffding's
inequality $\blacksquare$.

## Jensen's Inequality

If $f$ is a convex function with random variable
$v$, then

$$
\mathbb{E}(f(v))\ge f(\mathbb{E}(v))
$$

 The proof of this is straightforward. Define $L(v) = a v +b $ with
$a,b\in
\mathbb{R}$. Choose $a$ and $b$ so that
$L(\mathbb{E}(v))=f(\mathbb{E}(v))$
which makes $L$ tangent to $f$ at
$\mathbb{E}(v)$. By the convexity of $f$, we
have $f(v)\ge L(v)$. We can take
the expectation of both sides of this,

$$
\begin{align*}
\mathbb{E}(f(v)) \ge & \mathbb{E}(L(v)) \\
                  =
& \mathbb{E}(a v+b)  \\
                  =  & a\mathbb{E}(v)+b  \\
=  & L(\mathbb{E}(v))  \\
                  =  & f(\mathbb{E}(v))
\end{align*}
$$

 equality holds when $f$ is linear. For a concave function $f$, the
sense of the
inequality is reversed.

```python

```


# A New-Keynesian Model

Consider the new-Keynesian business cycle model from Walsh (2017), chapter 8 expressed in log-linear terms:

\begin{align}
y_t & = E_ty_{t+1} - \sigma^{-1} (i_t - E_t\pi_{t+1}) + g_t\\
\pi_t & = \beta  E_t\pi_{t+1} + \kappa  y_t + u_t\\
i_t & = \phi_x  y_t + \phi_{\pi}  \pi_t + v_t\\
r_t & = i_t - E_t\pi_{t+1}\\
g_{t+1} & = \rho_g g_{t} + \epsilon_{t+1}^g\\
u_{t+1} & = \rho_u u_{t} + \epsilon_{t+1}^u\\
v_{t+1} & = \rho_v v_{t} + \epsilon_{t+1}^v
\end{align}

where $y_t$ is the output gap (log-deviation of output from the natural rate), $\pi_t$ is the quarterly rate of inflation between $t-1$ and $t$, $i_t$ is the nominal interest rate on funds moving between period $t$ and $t+1$, $r_t$ is the real interest rate, $g_t$ is the exogenous component of demand, $u_t$ is an exogenous component of inflation, and $v_t$ is the exogenous component of monetary policy.

Since the model is already log-linear, there is no need to approximate the equilibrium conditions. We'll still use the `.log_linear` method to find the matrices $A$ and $B$, but we'll have to set the `islinear` option to `True` to avoid generating an error.

## Import requisite modules

```python
# Import numpy, pandas, linearsolve, matplotlib.pyplot
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

## Inintialize model and solve

```python
# Input model parameters
beta = 0.99
sigma= 1
eta  = 1
omega= 0.8
kappa= (sigma+eta)*(1-omega)*(1-beta*omega)/omega

rhor = 0.9
phipi= 1.5
phiy = 0

rhog = 0.5
rhou = 0.5
rhov = 0.9

Sigma = 0.001*np.eye(3)

# Store parameters
parameters = pd.Series({
    'beta':beta,
    'sigma':sigma,
    'eta':eta,
    'omega':omega,
    'kappa':kappa,
    'rhor':rhor,
    'phipi':phipi,
    'phiy':phiy,
    'rhog':rhog,
    'rhou':rhou,
    'rhov':rhov
})

# Define function that computes equilibrium conditions
def equations(variables_forward,variables_current,parameters):

    # Parameters
    p = parameters

    # Variables
    fwd = variables_forward
    cur = variables_current

    # Exogenous demand
    g_proc =  p.rhog*cur.g - fwd.g

    # Exogenous inflation
    u_proc =  p.rhou*cur.u - fwd.u

    # Exogenous monetary policy
    v_proc =  p.rhov*cur.v - fwd.v

    # Euler equation
    euler_eqn = fwd.y -1/p.sigma*(cur.i-fwd.pi) + fwd.g - cur.y

    # NK Phillips curve evolution
    phillips_curve = p.beta*fwd.pi + p.kappa*cur.y + fwd.u - cur.pi

    # interest rate rule
    interest_rule = p.phiy*cur.y+p.phipi*cur.pi + fwd.v - cur.i

    # Fisher equation
    fisher_eqn = cur.i - fwd.pi - cur.r

    # Stack equilibrium conditions into a numpy array
    return np.array([
            g_proc,
            u_proc,
            v_proc,
            euler_eqn,
            phillips_curve,
            interest_rule,
            fisher_eqn
        ])

# Initialize the nk model
nk = ls.model(equations=equations,
              n_states=3,
              n_exo_states = 3,
              var_names=['g','u','v','i','r','y','pi'],
              parameters=parameters)

# Set the steady state of the nk model
nk.set_ss([0,0,0,0,0,0,0])

# Find the log-linear approximation around the non-stochastic steady state
nk.linear_approximation()

# Solve the nk model
nk.solve_klein(nk.a,nk.b)
```

## Compute impulse responses and plot

Compute impulse responses of the endogenous variables to a one percent shock to each exogenous variable.

```python
# Compute impulse responses
nk.impulse(T=11,t0=1,shocks=None)

# Create the figure and axes
fig = plt.figure(figsize=(12,12))
ax1 = fig.add_subplot(3,1,1)
ax2 = fig.add_subplot(3,1,2)
ax3 = fig.add_subplot(3,1,3)

# Plot commands
nk.irs['e_g'][['g','y','i','pi','r']].plot(lw='5',alpha=0.5,grid=True,title='Demand shock',ax=ax1).legend(loc='upper right',ncol=5)
nk.irs['e_u'][['u','y','i','pi','r']].plot(lw='5',alpha=0.5,grid=True,title='Inflation shock',ax=ax2).legend(loc='upper right',ncol=5)
nk.irs['e_v'][['v','y','i','pi','r']].plot(lw='5',alpha=0.5,grid=True,title='Interest rate shock',ax=ax3).legend(loc='upper right',ncol=5)
```

## Construct a stochastic simulation and plot

Contruct a 151 period stochastic simulation by first siumlating the model for 251 periods and then dropping the first 100 values. The seed for the numpy random number generator is set to 0.

```python
# Compute stochastic simulation
nk.stoch_sim(T=151,drop_first=100,cov_mat=Sigma,seed=0)

# Create the figure and axes
fig = plt.figure(figsize=(12,8))
ax1 = fig.add_subplot(2,1,1)
ax2 = fig.add_subplot(2,1,2)

# Plot commands
nk.simulated[['y','i','pi','r']].plot(lw='5',alpha=0.5,grid=True,title='Output, inflation, and interest rates',ax=ax1).legend(ncol=4)
nk.simulated[['g','u','v']].plot(lw='5',alpha=0.5,grid=True,title='Exogenous demand, inflation, and policy',ax=ax2).legend(ncol=4,loc='lower right')
```

```python
# Plot simulated exogenous shocks
nk.simulated[['e_g','g']].plot(lw='5',alpha=0.5,grid=True).legend(ncol=2)
nk.simulated[['e_u','u']].plot(lw='5',alpha=0.5,grid=True).legend(ncol=2)
nk.simulated[['e_v','v']].plot(lw='5',alpha=0.5,grid=True).legend(ncol=2)
```


# Wave propagation in Frequency Domain

In this section we discuss how to solve the PDE for 2D sonic wave equation in frequency domain using `esys.escript`. It is assumed that you have worked through the [introduction section on `esys.escript`](escriptBasics.ipynb).
First we will provide the basic theory.

## Sonic Wave Equation in Frequency domain

The sonic wave equation in frequency domain is given as
\begin{equation}\label{eqWAVEF1}
\mathbf{V} = - \nabla u = -\begin{bmatrix}
\frac{\partial u}{\partial x_0 }\\
 \frac{\partial u}{\partial x_1 }
\end{bmatrix}
\end{equation}
\begin{equation}\label{eqWAVEF2}
\frac{\partial V_0}{\partial x_0 }+
 \frac{\partial V_1}{\partial x_1 } - k^2 u = \delta_{\mathbf{x}_s} \mbox{ with } k= \frac{\omega}{c}
\end{equation}
where $u$ is the power amplitude for angular frequency $\omega$
and source at location $\mathbf{x}_s$. $c$ is the wave propagation speed.

A PML layer of thickness $L_{pml}$ is introduced to remove reflections from the boundary.
For this we set
\begin{equation}\label{eqPMLGamma}
\gamma_i = 1 -\mathbf{j} \frac{S_{pml}}{k} Q_i
\end{equation}
where $S_{pml}$ is absorption strength parameter and $Q_i$ are the absorbing functions.
Horizontal absorption is applied near the left ($x_0=0$) and right boundary ($x_0=L_0=3km$):
\begin{equation}\label{EQQ0}
Q_0(\mathbf{x}) = \begin{cases}
= \left( \frac{L_{pml}-x_0}{L_{pml}} \right)^{M_{pml}} & \mbox{ if } 0 \le x_0 \le L_{pml}\\
= \left( \frac{x_0-L_0+L_{pml}}{L_{pml}} \right)^{M_{pml}} & \mbox{ if } L_0-L_{pml} \le x_0 \le L_0 \\
=0  & \mbox{ otherwise  }
\end{cases}
\end{equation}
while vertical absorption is applied near the bottom ($x_0=1$) only:
\begin{equation}\label{EQQ1}
Q_1(\mathbf{x}) = \begin{cases}
= \left( \frac{L_{pml}-x_1}{L_{pml}} \right)^{M_{pml}} & \mbox{ if } 0 \le x_1 \le L_{pml}\\
=0  & \mbox{ otherwise  }
\end{cases}
\end{equation}
With the variable transformation $x_i \rightarrow \gamma_i x_i$ equations
\eqref{eqWAVEF1} and \eqref{eqWAVEF2} are transfered to
\begin{equation}\label{eqWAVEF1PML}
\mathbf{V} = - \nabla u = -\begin{bmatrix}
\frac{1}{\gamma_0}\frac{\partial u}{\partial x_0 }\\
\frac{1}{\gamma_1} \frac{\partial u}{\partial x_1 }
\end{bmatrix}
\end{equation}
\begin{equation}\label{eqWAVEF2PML}
\frac{1}{\gamma_0}\frac{\partial V_0}{\partial x_0 }+
\frac{1}{\gamma_1} \frac{\partial V_1}{\partial x_1 } - k^2 u = \delta_{\mathbf{x}_s}
\end{equation}

## The PDE in `esys.escript`

The conservation equation in `esys.escript` is
\begin{equation}\label{eqEE1}
\frac{\partial F_0}{\partial x_0 }+
 \frac{\partial F_1}{\partial x_1 } D u = Y + y_{dirac}
\end{equation}
To match the \eqref{eqWAVEF2PML} we multiply this equations with $\gamma_0 \gamma_1$
and use the fact that $\gamma_0$ is not depending on $x_1$
and $\gamma_1$ is not depending on $x_0$ to obtain:
\begin{equation}\label{eqWAVEF2PML2}
\frac{\partial \gamma_1 V_0}{\partial x_0 }+
\frac{\partial \gamma_0 V_1}{\partial x_1 } - k^2 \gamma_0 \gamma_1 \; u = \delta_{\mathbf{x}_s}
\end{equation}
We also assume that the source location is not located in the absorption zone and hence
$\delta_{\mathbf{x}_s} \gamma_0 \gamma_1 =\delta_{\mathbf{x}_s}$. Comparison with \eqref{eqEE1}
shows $Y=0$,  $y_{dirac}=\delta_{\mathbf{x}_s}$, $D=- k^2 \gamma_0 \gamma_1$ and
\begin{equation}\label{EQFLUX2}
\mathbf{F} =
\begin{bmatrix}
 F_0 \\
 F_1
\end{bmatrix}
=
\begin{bmatrix}
  \gamma_1 V_0\\
\gamma_0 V_1
\end{bmatrix}
=
\begin{bmatrix}
 \frac{  \gamma_1}{\gamma_0}\frac{\partial u}{\partial x_0 }\\
 \frac{  \gamma_0}{\gamma_1}\frac{\partial u}{\partial x_1 }
\end{bmatrix}
\end{equation}
Recall the flux $\mathbf{F}$ is related with $\mathbf{\nabla} u$ in the form
\begin{equation} \label{EQFLUX}
\mathbf{F} = - \mathbf{A} \mathbf{\nabla} u +\mathbf{X}
\end{equation}
This gives $\mathbf{F}=0$ and
\begin{equation}\label{eqAF}
\begin{bmatrix}
A_{00} & A_{01} \\
A_{10}  & A_{11}
\end{bmatrix}=
\begin{bmatrix}
 \frac{  \gamma_1}{\gamma_0}\ & 0  \\
0  &  \frac{  \gamma_0}{\gamma_1}
\end{bmatrix}
\end{equation}

## Domain set up

We consider a single reflector set up. The reflector is located at a depth of $500m$
where the top layer has propagation speed of $c_{top}=1500m/s$
and the bottom layer has propagation speed of $c_{top}=3000m/s$.
Use a frequency of $f=5 Hz$. The domain has depth of $1km$ and width $3km$
with $300 \times 100$ grid.
The source is located at the surface at an offset of $1500m$ from the boundary.

```python
Width=3000       # m
Ne0=300          # number of cells along the horizontal axis
Depth=1000      # m
Ne1=100          # number of cells along the vertical axis
c_top=1800      # m/s
c_bottom=3000      # m/s
#%matplotlib notebook
```

```python
import numpy as np
Omega=2*np.pi*5. # Hz
d0=Depth-500.    # m top layer is 500m thick

```

As usual we start the model set up with defining the domain. In this application
we also want to use point source. These need to be defined when the domain is
defined. Here we use a single source named "source66". If several sources are used later
a list of source locations and name can be defined. As we will see later the names allow to
switch on sources individually.

```python
sources=  [ (Width/2, Depth) ]
source_names = [ 'source66' ]
```

We set up the domain including the location and names of potential source locations.

```python
from esys.escript import *
from esys.finley import Rectangle
domain=Rectangle(Ne0, Ne1, l0=Width,l1=Depth, diracPoints=sources,  diracTags=source_names)
```

To define the propagation speed `c` a mask `m` is introduced distinguishing between the top layer (value one)
and the bottom layer (value zero).
The propagation speed is defined at integration points in the elements.

```python
X=Function(domain).getX()
m=wherePositive(X[1]-d0)
c=c_top*m+c_bottom*(1-m)
```

Now we can set

```python
k=Omega/c
```

## PML

To minimize reflection from the boundary we introduce the PML conditions.
The key step is to introduce the absorbing functions $Q_i$.
We start with the vertical  absorbing functions $Q_1$ which is defined by  \eqref{EQQ1}.
$Q_1$ is defined at the numerical integration points.

```python
Lpml=200
Mpml=3
Spml=1
```

`Q1` is build as a product of the monomial part $\left( \frac{L_{pml}-x_1}{L_{pml}} \right)^{M_{pml}}$
and a mask that makes sure that the monomial part is applied in the layer of thickness $L_{pml}$ near the bottom
boundary only:

```python
X=Function(domain).getX()
Q1=wherePositive(Lpml-X[1])*((Lpml-X[1])/Lpml)**Mpml
gamma1=1-1j*Spml/k*Q1
```

Similarly in the horizontal direction:

```python
X=Function(domain).getX()
Q0_l=wherePositive(Lpml-X[0])*((Lpml-X[0])/Lpml)**Mpml
Q0_r=wherePositive(X[0]-(Width-Lpml))*((X[0]-(Width-Lpml))/Lpml)**Mpml
gamma0=1-1j*Spml/k*(Q0_l+Q0_r)
```

Let's make sure the Q's are set up correctly and take a look at the distribution. To reduce rendering time
we interpolate to element centers first.

```python
Q=interpolate(Q1+Q0_l+Q0_r, ReducedFunction(domain))
Q_np=convertToNumpy(Q)
x_np=convertToNumpy(Q.getFunctionSpace().getX())

import matplotlib.pyplot as plt
plt.figure(figsize=(9,3))
plt.tricontourf(x_np[0], x_np[1], Q_np[0], 15)
plt.xlabel('x0 [m]')
plt.ylabel('x1 [m]')
plt.title("Absorbing Function Q")
plt.colorbar()
```

    <matplotlib.colorbar.Colorbar at 0x7f28bf3c17b8>

## Definition of the PDE

Now we are ready to set-up the PDE. As the PML coefficient has complex values we need to set up the PDE
as a complex PDE. Also we use the direct solver which is the better choice for this PDE.

```python
from esys.escript.linearPDEs import LinearSinglePDE, SolverOptions

pde=LinearSinglePDE(domain, isComplex=True)
pde.getSolverOptions().setSolverMethod(SolverOptions.DIRECT)
```

We setting the coefficient `A` following \eqref{eqAF}. We use the
`Tensor` function to create $2 \times 2$ matrix for each integration point. It is initialized with zero (first argument) and we then modify the main diagonal entries according to the PML absorption.

```python
myA=Tensor(0.,Function(domain))
myA[0,0]=gamma1/gamma0
myA[1,1]=gamma0/gamma1
```

And now we set `A` and `D`. Don't forget the factor `gamma1*gamma0` in `D` according to PML.

```python
pde.setValue(A=myA, D=-gamma1*gamma0*k**2)
```

The missing bit is the point source. This is set using a `Data` object with
the `DiracDeltaFunctions(domain)` attribute. `Data` object with this attribute using the Dirac point
`diracPoints` and corresponding tags `diracTags` introduced into the `domain` when it was defined.
The source term - let's call it `src` - is set up in two steps:
First a `Scalar` `Data` object is created which is linked with the Dirac point is initialized with zero.

```python
src=Scalar(0., DiracDeltaFunctions(domain) )
```

Then the specific Dirac point labeled "source66" get assigned the value one. Notice that
other Dirac point are not effected. Here we assign the value `1`:

```python
src.setTaggedValue('source66' , 1.)
```

This `src` is now assigned to the `y_dirac` coefficient of the PDE:

```python
pde.setValue(y_dirac=src)
```

We are ready to get the solution:

```python
u=pde.getSolution()
print("solution u =", str(u))
```

    solution u = Summary: Lsup=1.38888 data points=30401

Let's see how it looks like:

```python
u_np=convertToNumpy(abs(u))
x_np=convertToNumpy(u.getFunctionSpace().getX())

plt.figure(figsize=(9,3))
plt.tricontourf(x_np[0], x_np[1], u_np[0], 15)
plt.xlabel('y [m]')
plt.ylabel('z [m]')
plt.title("Magnitude of Solution")
plt.colorbar()
```

## Setting  up geophones:

To create a synthetic CDP gather or even run FWI we pick value from $u$ at specific location
simulating a line of geophones. Here we assume that `NGeophone` geophones are located at
surface stating at offset `Offset` from the left side of the domain. The geophones are assumed to be equally spaced
with distance `GeophoneSpacing`:

```python
GeophoneSpacing=50
NGeophone=51
Offset=250
Surface=Depth
```

Let's check is the last geophone is still located in the domain:

```python
assert Offset+GeophoneSpacing * (NGeophone-1) < Width
```

Set the actual locations of the geophones in the domain:

```python
geophone_locations = [ (Offset+GeophoneSpacing * j, Surface) for j in range(NGeophone) ]
```

We use the `Locator` to pick values from the solution `u`:

```python
from esys.escript.pdetools import Locator
geophones=Locator(u.getFunctionSpace(), geophone_locations)
u_at_geophone=geophones.getValue(abs(u))
```

We get the true offsets of the geophones - just in case ...

```python
geophone_offsets=[ x[0] for x in geophones.getX() ]
```

And now we can plot the amplitude of `u` along the line of geophones:

```python
plt.figure(figsize=(10,5))
plt.clf()
plt.plot(geophone_offsets, u_at_geophone)
plt.scatter(geophone_offsets, u_at_geophone)
plt.xlabel('x0 [m]')
plt.ylabel('aplitude [m]')
plt.title("Amplitude vs offset")
plt.savefig("wave.png")
```


# Modelling part 1 (Week 6)

Code at: https://github.com/markbriers/data-science-jupyter

## Recap - the story so far

Our goal is to follow a data science process (CRISP-DM) to deliver a successful business outcome.

Our learning objectives are as follows:

* Describe the six stages of a data processing pipeline (using CRISP-DM)

* Demonstrate an understanding of the python programming language through the production of elementary data analysis programme

* Analyse at least three different data sources by applying at least one python data processing library to extract and explore pertinent features

* Be able to design a set of data requirements for a specified business problem

* Describe and apply (using the python programming language) the main approaches to supervised learning for a given classification problem

* Understand the use cases of Big Data technology (in particular Spark)

* Produce a report including appropriate data visualisations covering the analysis of a business problem using a data science based approach

## Learning outcome

* By the end of the lecture, you will be able to differentiate a multivariable function and functional compositions from first principles. This will allow you to select the parameters of a simple linear model.

* By the end of the lecture, you will be able to determine the parameters of a linear regression model using least squares estimation. This will allow you to predict the outcome of an event with unseen data.

***

## Functions and derivatives

A function is a relation between sets that associates to every element of a first set exactly one element of the second set
* Examples:
\begin{eqnarray}
f_1(x) & = & x^3 \\
f_2(x) & = & x+2 \\
f_3(x) & = & 2x \\
f_4(x) & = & \frac{1}{\sqrt{2\pi}}\exp{\left(-\frac{x^2}{2}\right)} \\
f_5(x,y) & = & x^{2}2y^{3}
\end{eqnarray}

We often wish to characterise how "steep" a function is at a particular point. The functions listed above (and all other _differentiable_ functions) have a _slope_ at a particlar point in the function. We can compute the extent of this slope as follows:

\begin{equation}
\mbox{slope} = \frac{\Delta_y}{\Delta_x} = \frac{\mbox{change in y}}{\mbox{change in x}}
\end{equation}

For example, lets calculate the average slope between two points:

### Introducting the derivative

* The _derivative_ of a function allows us to compute the slope at a particular point in the input space
* Calculating a derivate is the same as calculating a slope, but we use a _smarter_ mathematical technique that allows us to compute the slope at any point in the function
* We use $h$ to represent the infinitesimal distance (i.e. a very small number)

* Example:
\begin{equation}
f(\theta_{1}) = \theta_1^2
\end{equation}

We can use differentiation to compute derivatives of functions:

1. Increment $\theta_{1}$ by a very small value, $h$, such that $f(\theta_1 +h) = (\theta_1+h)^2$
2. Apply the slope formula:
\begin{equation}
\frac{f(\theta_1 +h)-f(\theta_1)}{h}
\end{equation}
3. Simplify the equation:
\begin{equation}
\frac{\theta_1^2+2\theta_1 h+h^2-\theta_1^2}{h} = \frac{2\theta_1 h+h^2}{h} = 2\theta_1+h
\end{equation}
4. Set $h$ to 0 (the limit as $h\rightarrow 0$)
\begin{equation}
2\theta_1+0 = 2\theta_1
\end{equation}

This means that the slope at an any point of $f(\theta_1)$, its derivative, is $2\theta_1$

What do we do if a function has more than one variable?

In functions with 2 or more variables (e.g. functions of vectors), the partial derivative is the derivative of one variable with respect to the others. Let's look at an example:

1. Given a multivariable function, $f(x,z) = 2z^3 x^2$ calculate the derivative $\frac{df}{dx}(x,z)$
2. Swap $2z^3$ with a constant $b$:
\begin{equation}
f(x,z) = bx^2
\end{equation}
3. Calculate the derivative with $b$ constant:
\begin{eqnarray}
\lim_{h\rightarrow 0} \frac{b(x+h)^2-bx^2}{h}\\
\lim_{h\rightarrow 0} \frac{b((x+h)(x+h))-bx^2}{h} \\
\lim_{h\rightarrow 0} \frac{bx^2+2bxh+bh^2-bx^2}{h} \\
\lim_{h\rightarrow 0} 2bx+bh
\end{eqnarray}
4. Swap $b=2z^3$ back into this equation: $4z^3x$
5. Repeat steps to calculate derivative w.r.t. $z$, $\frac{df}{dz}(x,z) = 6x^2 z^2$

We can store the partial derivates in a gradient:
\begin{equation}
\Delta f(x,z) = \left[\begin{matrix}
   \frac{df}{dx} \\
   \frac{df}{dz}\\
   \end{matrix}\right] = \left[\begin{matrix}
   4z^3x \\
   6x^2 z^2\\
   \end{matrix}\right]
\end{equation}

### Introducing a gradient

* A gradient is a vector that stores the partial derivatives of multivariable functions
* It helps us calculate the slope at a specific point on a curve for functions with multiple independent variables
* In order to calculate this more complex slope, we need to isolate each variable to determine how it impacts the output on its own
* To do this we iterate through each of the variables and calculate the derivative of the function after holding all other variables constant
* Each iteration produces a partial derivative which we store in the gradient

* A gradient always points in the direction of the greatest increase of a function
* A gradient is always zero at a local minimum or maximum

Let's compute the gradient of the function $f(\theta_{1},\theta_{2}) = \theta_{1}^2 + \theta_{2}^2$ as follows. (The function looks like this: https://www.google.com/search?q=x%5E2+%2B+y%5E2&oq=x%5E2+&aqs=chrome.0.69i59j69i57.1620j0j1&sourceid=chrome&ie=UTF-8)

In this module, we will only superficially cover differentiation (in the interests of time). However, if you are interested in learning more (which may be useful for your wider careers in science) then this resource: https://www.khanacademy.org/math/differential-calculus/dc-diff-intro provides an excellent in-depth overview.

## Introduction a linear modelling technique

We will now move onto modeling, exploring simple and complex statistical models (https://en.wikipedia.org/wiki/Statistical_model). Note that we will only be able to cover a small portion of the set of available models. Research into modeling and inference is constantly evolving - in this course, we will focus on linear models, logistic regression, and (for your wider knowledge) Deep Neural Networks.

Modeling is used to represent the data generating process. That is, a model can be considered an abstraction of the data, allowing one to derive population based quantities from sample based quantities. The modeling process combines statistical assumptions (contextually related to the real-world or otherwise), a mathematical expression, and data. From the data, we can fit the mathematical model, and in doing so, encode the statistical assumptions.

### Linear model

We will first consider a linear model. This can be thought of as fitting a Normal (or Gaussian) distribution to data, or equivalently fitting a straight line (or plane) to the datapoints. A straight line (in 2 dimensions) is mathematically described as follows:
\begin{equation}
y = ax + b
\end{equation}
where $a$ is the gradient and $b$ is the $y$-axis intercept (see http://www.mathcentre.ac.uk/resources/uploaded/mc-ty-strtlines-2009-1.pdf for instance).

We will now construct a linear model and estimate the paramters using Tensorflow. This code has been adapted from https://sanjayasubedi.com.np/deeplearning/tensorflow-2-linear-regression-from-scratch/

```
import tensorflow as tf
```

First of all, we define a linear model $y=wx+b$ as a python 'class':

```
class LinearModel:
    def __init__(self):
        self.w = tf.Variable(16.0) # Default value of W = 16.0
        self.b = tf.Variable(10.0) # Default value of b = 10.0

    def __call__(self, x):
        return self.w * x + self.b # Linear model y = wx + b
```

We will now simulate some data, setting true values of w and b (which we will subsequently try to estimate):

```
TRUE_W = 3.0 # slope
TRUE_b = 0.5 # intercept

NUM_EXAMPLES = 1000

X = tf.random.normal(shape=(NUM_EXAMPLES,))
noise = tf.random.normal(shape=(NUM_EXAMPLES,))
y = X * TRUE_W + TRUE_b + noise
```

Let's plot what the initial model looks like, with the default values specified:

```
import matplotlib.pyplot as plt
%matplotlib inline
```

```
model = LinearModel()
plt.scatter(X, y, label="true")
plt.scatter(X, model(X), label="predicted")
plt.legend()
```

Now we will define the energy (or *loss*) function, using least squares expression (defined above):

```
def loss(y, y_pred):
    return tf.reduce_mean(tf.square(y - y_pred))
```

Finally, we define the training function which uses tensorflow's ability to automatically differentiate the loss function:

```
def train(model, X, y, lr=0.01):
    with tf.GradientTape() as t:
        current_loss = loss(y, model(X))
    dW, db = t.gradient(current_loss, [model.w, model.b])
    model.w.assign_sub(lr * dW)
    model.b.assign_sub(lr * db)
```

We are now in a position to bring together the model and our training procedure:

```
model = LinearModel()
wStore, bStore = [], []
trainingIterations = 40

for epoch in range(trainingIterations):
    wStore.append(model.w.numpy())
    bStore.append(model.b.numpy())
    current_loss = loss(y, model(X))
    train(model, X, y, lr=0.1)
    print(f"Epoch {epoch}: Loss: {current_loss.numpy()}, W: {model.w.numpy()}, b: {model.b.numpy()}")

plt.plot(range(trainingIterations), wStore, 'r', range(trainingIterations), bStore, 'b')
plt.plot([TRUE_W] * trainingIterations, 'r--', [TRUE_b] * trainingIterations, 'b--')
plt.legend(['W', 'b', 'true W', 'true b'])
plt.show()
```

Lets plot the predicted data (based on the estimated model) by evaluating each of the training datapoints:

```
plt.scatter(X, y, label="true")
plt.scatter(X, model(X), label="predicted")
plt.legend()
```

```
print("True value of W = ", TRUE_W)
print("Estimated value of W = ", model.w)
print("True value of b = ", TRUE_b)
print("Estimated value of b = ", model.b)
```

    True value of W =  3.0
    Estimated value of W =  <tf.Variable 'Variable:0' shape=() dtype=float32, numpy=3.0002277>
    True value of b =  0.5
    Estimated value of b =  <tf.Variable 'Variable:0' shape=() dtype=float32, numpy=0.46493825>

***
Mandatory exercises:

- [ ] Carefully read the material above, and ask questions over email.
- [ ] Replicate all of the code in this notebook.