 ILI285 - Computación Científica I  / INF285 - Computación Científica

 Polynomial Interpolation: Vandermonde, Lagrange, Newton, Chebyshev

  [S]cientific [C]omputing [T]eam

 Version: 1.27

## Table of Contents
* [Introduction](#intro)
* [Vandermonde Matrix](#vander)
* [Lagrange Interpolation](#lagrange)
* [Runge Phenomenon](#runge)
* [Newton's Divided Difference](#DDN)
* [Interpolation Error](#Error)
* [Chebyshev Interpolation](#cheby)
* [Python Modules and Functions](#py)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp
from functools import reduce
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
%matplotlib inline
from ipywidgets import interact, fixed, IntSlider
```

## Introduction

Hello! In this notebook we will learn how to interpolate 1D data with polynomials. A polynomial interpolation consists in finding a polynomial that fits a discrete set of known data points, allowing us to construct new data points within the range of the data. Formally, a polynomial $P(x)$ interpolate the data $(x_1,y_1),...,(x_n,y_n)$ if $P(x_i)=y_i$ for all $i$ in $1,...,n$.

```python
def Y(D, xi):
    # Function that evaluates the xi's points in the polynomial
    if D['M']=='Vandermonde':
        P = lambda i: i**np.arange(len(D['P']))
    elif D['M']=='Lagrange':
        P = lambda i: [np.prod(i - np.delete(D['x'],j)) for j in range(len(D['x']))]
    elif D['M']=='Newton':
        P = lambda i: np.append([1],[np.prod(i-D['x'][:j]) for j in range(1,len(D['P']))])
    return [np.dot(D['P'], P(i)) for i in xi]

def Interpolation_Plot(D,ylim=None):
    # Function that shows the data points and the function that interpolates them.
    xi = np.linspace(min(D['x']),max(D['x']),1000)
    yi = Y(D,xi)
    plt.figure(figsize=(8,8))
    plt.plot(D['x'],D['y'],'ro',label='Interpolation points')
    plt.plot(xi,yi,'b-',label='$P(x)$')
    plt.xlim(min(xi)-0.5, max(xi)+0.5)
    if ylim:
        plt.ylim(ylim[0], ylim[1])
    else:
        plt.ylim(min(yi)-0.5, max(yi)+0.5)
    plt.grid(True)
    plt.legend(loc='best')
    plt.xlabel('$x$')
    #plt.ylabel('$P(x)$')
    plt.show()
```

## Vandermonde Matrix

First, we are going to learn the Vandermonde Matrix method. This is a $m \times m$ matrix (with $m$ being the length of the set of known data points) with the terms of a geometric progression in each row. It allows us to construct a system of linear equations with the objective of find the coefficients of the polynomial function that interpolates our data.

Example:

Given the set of known data points: $(x_1,y_1),(x_2,y_2),(x_3,y_3)$

Our system of linear equations will be:

$$ \begin{bmatrix}
    1 & x_1 & x_1^2  \\[0.3em]
    1 & x_2 & x_2^2  \\[0.3em]
    1 & x_3 & x_3^2  \end{bmatrix}
    \begin{bmatrix}
    a_1  \\[0.3em]
    a_2  \\[0.3em]
    a_3  \end{bmatrix} =
    \begin{bmatrix}
    y_1  \\[0.3em]
    y_2  \\[0.3em]
    y_3  \end{bmatrix}$$

And solving it we will find the coefficients $a_1,a_2,a_3$ that we need to construct the polynomial $P(x)=a_1+a_2x+a_3x^2$ that interpolates our data.

```python
def Vandermonde(x, y, show=False):
    # We construct the matrix and solve the system of linear equations
    A = np.array([xi**np.arange(len(x)) for xi in x])
    b = y
    xsol = np.linalg.solve(A,b)

    # The function shows the data if the flag is true
    if show:
        print('Data Points: '); print([(x[i],y[i]) for i in range(len(x))])
        print('A = '); print(np.array_str(A, precision=2, suppress_small=True))
        print("cond(A) = "+str(np.linalg.cond(A)))
        print('b = '); print(np.array_str(b, precision=2, suppress_small=True))
        print('x = '); print(np.array_str(xsol, precision=2, suppress_small=True))
        xS = sp.Symbol('x')
        F = np.dot(xS**np.arange(len(x)),xsol)
        print('Interpolation Function: ')
        print('F(x) = ')
        print(F)

    # Finally, we return a data structure with our interpolating polynomial
    D = {'M':'Vandermonde',
         'P':xsol,
         'x':x,
         'y':y}
    return D
```

```python
def show_time_V(epsilon=0):
    x = np.array([1.0,2.0,3.0+epsilon,5.0,6.5])
    y = np.array([2.0,5.0,4.0,6.0,2.0])
    D = Vandermonde(x,y,True)
    Interpolation_Plot(D,[-4,10])
interact(show_time_V,epsilon=(-1,2,0.1))
```

    interactive(children=(FloatSlider(value=0.0, description='epsilon', max=2.0, min=-1.0), Output()), _dom_classe…

    <function __main__.show_time_V(epsilon=0)>

## Lagrange Interpolation

With this method, we can interpolate data thanks to the Lagrange basis polynomials. Given a set of $n$ data points $(x_1,y_1),...,(x_n,y_n)$, the Lagrange interpolation polynomial is the following:

$$ P(x) = \sum^n_{i=1} y_i\,L_i(x),$$

where $L_i(x)$ are the Lagrange basis polynomials:

$$ L_i(x) = \prod^n_{j=1,j \neq i} \frac{x-x_j}{x_i-x_j} = \frac{x-x_1}{x_i-x_1} \cdot ... \cdot \frac{x-x_{i-1}}{x_i-x_{i-1}} \cdot \frac{x-x_{i+1}}{x_i-x_{i+1}} \cdot ... \cdot \frac{x-x_n}{x_i-x_n}$$

or simply $L_i(x)=\dfrac{l_i(x)}{l_i(x_i)}$, where $l_i(x)=\displaystyle{\prod^n_{j=1,j \neq i} (x-x_j)}$.

The most important property of these basis polynomials is:

$$ L_{j \neq i}(x_i) = 0 $$

$$ L_i(x_i) = 1 $$

So, we assure that $L(x_i) = y_i$, which indeed interpolates the data.

```python
def Lagrange(x, y, show=False):
    # We calculate the li's
    p = np.array([y[i]/np.prod(x[i] - np.delete(x,i)) for i in range(len(x))])

    # The function shows the data if the flag is true
    if show:
        print('Data Points: '); print([(x[i],y[i]) for i in range(len(x))])
        xS = sp.Symbol('x')
        L = np.dot(np.array([np.prod(xS - np.delete(x,i))/np.prod(x[i] - np.delete(x,i)) for i in range(len(x))]),y)
        print('Interpolation Function: ');
        print(L)

    # Finally, we return a data structure with our interpolating polynomial
    D = {'M':'Lagrange',
         'P':p,
         'x':x,
         'y':y}
    return D
```

```python
def show_time_L(epsilon=0):
    x = np.array([1.0,2.0,3.0+epsilon,4.0,5.0,7.0,6.0])
    y = np.array([2.0,5.0,4.0,6.0,7.0,3.0,8.0])
    D = Lagrange(x,y,True)
    Interpolation_Plot(D,[0,10])
interact(show_time_L,epsilon=(-1,1,0.1))
```

    interactive(children=(FloatSlider(value=0.0, description='epsilon', max=1.0, min=-1.0), Output()), _dom_classe…

    <function __main__.show_time_L(epsilon=0)>

```python
def show_time_Li(i=0, N=7):
    x = np.arange(N+1)
    y = np.zeros(N+1)
    y[i]=1
    D = Lagrange(x,y,True)
    Interpolation_Plot(D,[-1,2])

i_widget = IntSlider(min=0, max=7, step=1, value=0)
N_widget = IntSlider(min=5, max=20, step=1, value=7)
def update_i_range(*args):
    i_widget.max = N_widget.value
N_widget.observe(update_i_range, 'value')

interact(show_time_Li,i=i_widget,N=N_widget)
```

    interactive(children=(IntSlider(value=0, description='i', max=7), IntSlider(value=7, description='N', max=20, …

    <function __main__.show_time_Li(i=0, N=7)>

Here you get some questions about Lagrange Interpolation:
- Explain what happens with the interpolator polynomial when you add a new point to the set of points to interpolate. **Answer: We need to recalculate the polynomial**
- Why it is not a good idea to use Lagrange interpolation for a set of points which is constantly changing? **A: Because we need to compute the whole interpolation again**
- What is the operation count of obtaining the interpolator polynomial using Lagrange? What happens with the error?

## Newton's Divided Difference

In this interpolation method we will use divided differences to calculate the coefficients of our interpolation polynomial. Given a set of $n$ data points $(x_1,y_1),...,(x_n,y_n)$, the Newton polynomial is:

$$ P(x) = \sum^n_{i=1} (f[x_1 ... x_i] \cdot \prod^{i-1}_{j=1} (x-x_j)) ,$$

where $ \prod^{0}_{j=1} (x-x_j) = 0 $, and:

$$ f[x_i] = y_i $$

$$ f[x_j...x_i] = \frac{f[x_{j+1}...x_i]-f[x_j...x_{i-1}]}{x_i-x_j}$$

```python
def Divided_Differences(x, y):
    dd = np.array([y])
    for i in range(len(x)-1):
        ddi = []
        for a in range(len(x)-i-1):
            ddi.append((dd[i][a+1]-dd[i][a])/(x[a+i+1]-x[a]))
        ddi = np.append(ddi,np.full((len(x)-len(ddi),),0.0))
        dd = np.append(dd,[ddi],axis=0)
    return np.array(dd)

def Newton(x, y, show=False):
    # We calculate the divided differences and store them in a data structure
    dd = Divided_Differences(x,y)

    # The function shows the data if the flag is true
    if show:
        print('Data Points: '); print([(x[i],y[i]) for i in range(len(x))])
        xS = sp.Symbol('x')
        N = np.dot(dd[:,0],np.append([1],[np.prod(xS-x[:i]) for i in range(1,len(dd))]))
        print('Interpolation Function: ');
        print(N)

    # Finally, we return a data structure with our interpolating polynomial
    D = {'M':'Newton',
         'P':dd[:,0],
         'x':x,
         'y':y}
    return D
```

```python
def show_time_N(epsilon=0):
    x = np.array([0.0,2.0,3.0+epsilon,4.0,5.0,6.0])
    y = np.array([1.0,3.0,0.0,6.0,8.0,4.0])
    D = Newton(x,y,True)
    Interpolation_Plot(D)
interact(show_time_N,epsilon=(-1,1,0.1))
```

    interactive(children=(FloatSlider(value=0.0, description='epsilon', max=1.0, min=-1.0), Output()), _dom_classe…

    <function __main__.show_time_N(epsilon=0)>

Questions about Newton's DD:
- What is the main problem using this method (and Lagrange)? How can you fix it? **A: A problem with polynomial interpolation with equispaced date is the Runge phenomenon and can be handle with Chebyshev points**
- What to do when a new point is added? **A: Pro, is not necessary re-calculate the whole polynomial only a small piece**

## Polynomial Interpolation Error

The interpolation error is given by:

$$ f(x)-P(x) = \frac{(x-x_1) \cdot (x-x_2) \cdot ... \cdot (x-x_n)}{n!} \cdot f^{(n)}(c) ,$$

where $c$ is within the interval from the minimun value of $x$ and the maximum one.

```python
def Error(f, n, xmin, xmax, method=Lagrange, points=np.linspace, plot_flag=True):
    # This function plots f(x), the interpolating polynomial, and the associated error
    # points can be np.linspace to equidistant points or Chebyshev to get Chebyshev points
    x = points(xmin,xmax,n)
    y = f(x)

    xe = np.linspace(xmin,xmax,100)
    ye = f(xe)

    D = method(x,y)
    yi = Y(D, xe)

    if plot_flag:
        plt.figure(figsize=(5,10))
        f, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5), sharey = False)

        ax1.plot(xe, ye,'r-', label='f(x)')
        ax1.plot(x, y,'ro', label='Interpolation points')
        ax1.plot(xe, yi,'b-', label='Interpolation')
        ax1.set_xlim(xmin-0.5,xmax+0.5)
        ax1.set_ylim(min(yi)-0.5,max(yi)+0.5)
        ax1.set_title('Interpolation')
        ax1.grid(True)
        ax1.set_xlabel('$x$')
        ax1.legend(loc='best')

        ax2.semilogy(xe, abs(ye-yi),'b-', label='Absolute Error')
        ax2.set_xlim(xmin-0.5,xmax+0.5)
        ax2.set_title('Absolute Error')
        ax2.set_xlabel('$x$')
        ax2.grid(True)
        #ax2.legend(loc='best')
        plt.show()

    return max(abs(ye-yi))
```

```python
def test_error_Newton(n=5):
    #me = Error(lambda x: np.sin(x)**3, n, 1, 7, Newton)
    me = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, Newton)
    print("Max Error:", me)
```

```python
interact(test_error_Newton,n=(5,25))
```

    interactive(children=(IntSlider(value=5, description='n', max=25, min=5), Output()), _dom_classes=('widget-int…

    <function __main__.test_error_Newton(n=5)>

##  **Runge's Phenomenon**: It is a problem of oscillation of polynomials at the edges of the interval.

We are interpolating a data that is 0 almost everywhere and 1 at the middle point, notice that when $n$ increases the oscilations increase and the red dots seems to be at 0 everywhere but it is just an artifact, there must be a 1 at the middle. The oscillations you see at the end of the interval is the Runge phenomenon.

```python
def Runge(n=9):
    x = np.linspace(0,1,n)
    y = np.zeros(n)
    y[int((n-1.0)/2.)]=1
    D = Newton(x,y,False)
    Interpolation_Plot(D)
interact(Runge,n=(5,25,2))
```

    interactive(children=(IntSlider(value=9, description='n', max=25, min=5, step=2), Output()), _dom_classes=('wi…

    <function __main__.Runge(n=9)>

## Chebyshev Interpolation

With the objective of reducing the error of the polynomial interpolation, we need to find the values of $x_1,x_2,...,x_n$ that minimize $(x-x_1) \cdot (x-x_2) \cdot ... \cdot (x-x_n)$.

To choose these values of $-1 \leq x_1,x_2,...,x_n \leq 1$ (to use another interval we just need to do a change of variables) that minimize the error, we will use the roots of the Chebyshev polynomials, also called **Chebyshev nodes** (of the first kind), which are defined by:

$$ x_i = \cos\left(\frac{(2i-1)\pi}{2n}\right), i = 1,...,n $$

```python
def Chebyshev(xmin,xmax,n=5):
    # This function calculates the n Chebyshev points and plots or returns them depending on ax
    ns = np.arange(1,n+1)
    x = np.cos((2*ns-1)*np.pi/(2*n))
    y = np.sin((2*ns-1)*np.pi/(2*n))
    plt.figure(figsize=(10,5))
    plt.ylim(-0.1,1.1)
    plt.xlim(-1.1,1.1)
    plt.plot(np.cos(np.linspace(0,np.pi)),np.sin(np.linspace(0,np.pi)),'k-')
    plt.plot([-2,2],[0,0],'k-')
    plt.plot([0,0],[-1,2],'k-')
    for i in range(len(y)):
        plt.plot([x[i],x[i]],[0,y[i]],'r-')
        plt.plot([0,x[i]],[0,y[i]],'r-')
    plt.plot(x,[0]*len(x),'bo',label='Chebyshev points')
    plt.plot(x,y,'ro')
    plt.xlabel('$x$')
    plt.title('n = '+str(n))
    plt.grid(True)
    plt.legend(loc='best')
    plt.show()
def Chebyshev_points(xmin,xmax,n):
    ns = np.arange(1,n+1)
    x = np.cos((2*ns-1)*np.pi/(2*n))
    #y = np.sin((2*ns-1)*np.pi/(2*n))
    return (xmin+xmax)/2 + (xmax-xmin)*x/2
def Chebyshev_points_histogram(n=50,nbins=20):
    xCheb=Chebyshev_points(-1,1,n)
    plt.figure()
    plt.hist(xCheb,bins=nbins,density=True)
    plt.grid(True)
    plt.show()
```

```python
interact(Chebyshev,xmin=fixed(-1),xmax=fixed(1),n=(2,50))
```

    interactive(children=(IntSlider(value=5, description='n', max=50, min=2), Output()), _dom_classes=('widget-int…

    <function __main__.Chebyshev(xmin, xmax, n=5)>

```python
interact(Chebyshev_points_histogram,n=(20,10000),nbins=(20,200))
```

    interactive(children=(IntSlider(value=50, description='n', max=10000, min=20), IntSlider(value=20, description…

    <function __main__.Chebyshev_points_histogram(n=50, nbins=20)>

By using these points, we reduce the numerator of the interpolation error formula:

$$ (x-x_1) \cdot (x-x_2) \cdot ... \cdot (x-x_n) = \dfrac{1}{2^{n-1}} \cdot T_n(x), $$

where $T(x) = \cos (n \cdot \arccos (x))$ is the n-th Chebyshev polynomial.

$$ T_0(x) = 1 $$
$$ T_1(x) = x $$
$$ T_2(x) = 2x^2 -1 $$
$$...$$
$$ T_{n+1}(x) = 2 \cdot x \cdot T_n(x) - T_{n-1}(x) $$

```python
def T(n,x):
    # Recursive function that returns the n-th Chebyshev polynomial evaluated at x
    if n == 0:
        return x**0
    elif n == 1:
        return x
    else:
        return 2*x*T(n-1,x)-T(n-2,x)

def Chebyshev_Polynomials(n=2, Flag_All_Tn=False):
    # This function plots the first n Chebyshev polynomials
    x = np.linspace(-1,1,1000)
    plt.figure(figsize=(10,5))
    plt.xlim(-1, 1)
    plt.ylim(-1.1, 1.1)
    if Flag_All_Tn:
        for i in np.arange(n+1):
            y = T(i,x)
            plt.plot(x,y,label='$T_{'+str(i)+'}(x)$')
    else:
        y = T(n,x)
        plt.plot(x,y,label='$T_{'+str(n)+'}(x)$')
    #    plt.title('$T_${:}$(x)$'.format(n))
    plt.legend(loc='right')
    plt.grid(True)
    plt.xlabel('$x$')
    plt.show()
```

```python
interact(Chebyshev_Polynomials,n=(0,12),Flag_All_Tn=True)
```

    interactive(children=(IntSlider(value=2, description='n', max=12), Checkbox(value=True, description='Flag_All_…

    <function __main__.Chebyshev_Polynomials(n=2, Flag_All_Tn=False)>

```python
n=9
xmin=1
xmax=9

mee = Error(lambda x: np.sin(x)**3, n, xmin, xmax, method=Lagrange)
mec = Error(lambda x: np.sin(x)**3, n, xmin, xmax, method=Lagrange, points=Chebyshev_points)

print("Max error (equidistants points):", mee)
print("Max error (Chebyshev nodes):", mec)
```

```python
def test_error_chebyshev(n=5):
    mee = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, Lagrange)
    mec = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, method=Lagrange, points=Chebyshev_points)
    print("Max error (equidistants points):", mee)
    print("Max error (Chebyshev nodes):", mec)
```

```python
interact(test_error_chebyshev,n=(5,100,2))
```

    interactive(children=(IntSlider(value=5, description='n', min=5, step=2), Output()), _dom_classes=('widget-int…

    <function __main__.test_error_chebyshev(n=5)>

Questions about Chebyshev:
- How can you calculate the Chebyshev points in the interval [a,b] instead of [-1,1]? **A: Using a change of variable**

## Convergence analysis

```python
n=50
shift=2

my_functions={0:lambda x: (x)**10,
              1:lambda x: np.abs((x)**3),
              2:lambda x: np.exp(-((x)**-2)),
              3:lambda x: 1/(1+x**2),
              4:lambda x: np.sin(x)**3}
labels = {0: "x^{10}",
          1: "|x^3|",
          2: "\exp(-x^{-2})",
          3: "1/(1+x^2)",
          4: "\sin^3(x)"}

n_points=np.arange(shift,n)

for k in np.arange(5):
    max_error=np.zeros(n-shift)
    max_error_es=np.zeros(n-shift)
    for i in n_points:
        max_error[i-shift] = Error(my_functions[k], i, -1, 1, Newton, Chebyshev_points, plot_flag=False)
        max_error_es[i-shift] = Error(my_functions[k], i, -1, 1, Newton, points=np.linspace, plot_flag=False)
    axis=plt.figure()
    plt.semilogy(n_points,max_error,'kd',label='Chebyshev points')
    plt.semilogy(n_points,max_error_es,'k.',label='Equalspaced poins')
    plt.ylim(10**-16,10**4)
    plt.grid(True)
    plt.title('Interpolation Error of $f(x)='+str(labels[k])+"$")
    plt.xlabel('Number of points used in the interpolation')
    plt.ylabel('Max error on domain')
    plt.legend(loc='best')
    plt.show()
```

## Python Modules and Functions

Interpolation:

http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.polyfit.html

Vandermonde Matrix:

http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.vander.html

Lagrange:

http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpolate.lagrange.html

Chebyshev Points:

http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.polynomial.chebyshev.chebroots.html#numpy.polynomial.chebyshev.chebroots

# Acknowledgements
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. April 2016._
* _Material modified by Cristopher Arenas. May 2017._
* _Material modified by Claudio Torres. May 2017._
* _Bug fixed by Cristobal Carmona. April 2018._
* _Update June 2020 - v1.25 - C.Torres_ : Fixing formatting issues.
* _Update June 2020 - v1.26 - C.Torres_ : Adding "ylim" argumento to Interpolation_Plot(D,ylim=None) and addint "show_time_Li".
* _Update June 2020 - v1.27 - C.Torres_ : Adding comment that the Chebyshev nodes used are of the first kind and "Chebyshev_points_histogram".

```python

```


## 例題

3次方程式
\begin{equation}
x^3+6x^2+21x+32=0
\end{equation}
の解は
\begin{equation}
x=-9^{1/3}+3^{1/3}-2,\ -9^{1/3}e^{i2\pi/3}+3^{1/3}e^{i4\pi/3}-2,\ -9^{1/3}e^{i4\pi/3}+3^{1/3}e^{i2\pi/3}-2
\end{equation}
または
\begin{equation}
x\sim-2.64,\ -1.68-3.05i,\ -1.68+3.05i
\end{equation}

```python
def non_linear_func(x: float) -> float:
    return x**3 + 6*x**2 + 21*x + 32

def derivative_func(x: float) -> float:
    return 3*x**2 + 12*x + 21

x_real = -2.637834253
```

## 解を含む区分を狭めていく方法

- 初期値として解を含む区分を与える必要がある
- ヤコビ行列を必要としない
- 確実に収束する

### 2分法

1. 解が含まれる領域 $[a,b]$ を事前に指定する
2. 中点を利用して領域を $[a,(a+b)/2]$ と $[(a+b)/2,b]$ の2つに分割する
3. 解を含む領域を中間値の定理で判定する
4. 解を含む領域を選択して新たに計算する領域として繰り返す
5. 領域幅が一定以下になったら終了

中間値の定理から領域 $[a,b]$ に対して $f(a)f(b)<0$ なら、この領域 $[a,b]$ に $f(x)=0$ の解が含まれる事がわかる。

```python
from typing import Callable

def bisection(func: Callable[[float],float],
              left: float, right: float) -> float:
    '''
    二分法のアルゴリズム
    '''
    # 二分法で解けない場合
    if func(left) * func(right) >= 0:
        raise Exception('Cannot be solved by bisection method!')

    # 代数式の数値精度
    delta: float = 1e-10
    # 解の数値精度
    epsilon: float = 5e-6

    middle: float = 0.
    count: int = 0
    while True:
        # 中点計算
        middle = (left + right)/2.
        count += 1

        # 中間値の定理から解を含む区間を選択
        if func(left) * func(middle) < 0:
            right = middle
        else:
            left = middle

        # 終了判定
        if (abs(func(middle)) < delta):
            print(f"finished: f(x)={abs(func(middle)):.6e} (count={count})")
            break
        if (abs(left - right) < epsilon):
            print(f"finished: |left - right|={abs(left - right):.6e} (count={count})")
            break

    middle = (left+right)/2.
    return middle
```

```python
x = bisection(non_linear_func, -3.0, 0)
print(f"Δx = {x-x_real:.6e}, f(x)={non_linear_func(x):.6e}")
```

    finished: |left - right|=2.861023e-06 (count=20)
    Δx = 1.134561e-06, f(x)=1.159317e-05

### はさみうち法

はさみうち法では2分法のアレンジとして、交点を中点ではなく直線近似の解を用いる。
\begin{equation}
y(x)\equiv\frac{f(a)-f(b)}{a-b}(x-a)+f(a)
\end{equation}
とした場合の
\begin{equation}
y(c)=0 ⇔ c=-f(a)\frac{a-b}{f(a)-f(b)}+a=\frac{bf(a)-af(b)}{f(a)-f(b)}
\end{equation}

```python
def squeeze(func: Callable[[float],float],
            left: float, right: float) -> float:
    '''
    はさみうち法のアルゴリズム
    '''
    # 二分法で解けない場合
    if func(left) * func(right) >= 0:
        raise Exception('Cannot be solved by squeeze method!')

    # 代数式の数値精度
    delta: float = 1e-10
    # 解の数値精度
    epsilon: float = 5e-6

    middle: float = 0.
    count: int = 0
    while True:
        # 中点計算
        f_left, f_right = func(left), func(right)
        middle = (right*f_left - left*f_right)/(f_left - f_right)
        count += 1

        # 中間値の定理から解を含む区間を選択
        if func(left) * func(middle) < 0:
            right = middle
        else:
            left = middle

        # 終了判定
        if (abs(func(middle)) < delta):
            print(f"finished: f(x)={abs(func(middle)):.6e} (count={count})")
            break
        if (abs(left - right) < epsilon):
            print(f"finished: |left - right|={abs(left - right):.6e} (count={count})")
            break

    f_left, f_right = func(left), func(right)
    middle = (right*f_left - left*f_right)/(f_left - f_right)
    return middle
```

```python
x = squeeze(non_linear_func, -3.0, 0)
print(f"Δx = {x-x_real:.6e}, f(x)={non_linear_func(x):.6e}")
```

    finished: f(x)=6.559020e-11 (count=13)
    Δx = 2.544929e-10, f(x)=-1.033840e-11

## ヤコビ行列を使う方法

### ニュートン・ラフソン法

非線形関数の$x=a$周りのテーラー展開を用いて
\begin{equation}
f(x)=f(a)+f'(a)(x-a)+\mathcal{O}(\epsilon)=0
⇔x=a-\frac{f(a)}{f'(a)}+\mathcal{O}(\epsilon)
\end{equation}
と解を1次近似する。

```python
def newton1d(func: Callable[[float], float],
             derivative: Callable[[float], float], seed: float) -> float:
    '''
    1D のニュートン・ラフソン法
    '''
    # 代数式の数値精度
    delta: float = 1e-10

    result: float = 0.
    count: int = 0
    while True:
        # 解の更新
        result = result - func(result) / derivative(result)
        count += 1

        # 終了判定
        if (abs(func(result)) < delta):
            print(f"finished: f(x)={abs(func(result)):.4e} (count={count})")
            break

    return result
```

```python
x = newton1d(non_linear_func, derivative_func, 0)
print(f"Δx = {x-x_real:.6e}, f(x)={non_linear_func(x):.6e}")
```

    finished: f(x)=3.5527e-15 (count=5)
    Δx = 2.555036e-10, f(x)=3.552714e-15

## ヤコビ行列を使用しない方法

### 割線法

割線法はニュートン法に対し、はさみうち法のような直線近似の解を用いる。
逐次近似の直前2つの解$(x_0, f(x_0))$と$(x_1, f(x_1))$に対し
\begin{equation}
y(x)\equiv\frac{f(x_0)-f(x_1)}{x_0-x_1}(x-x_0)+f(x_0)
\end{equation}
とした場合の
\begin{equation}
y(x_2)=0 ⇔ x_2=-f(x_0)\frac{x_0-x_1}{f(x_0)-f(x_1)}+x_0=\frac{x_1f(x_0)-x_0f(x_1)}{f(x_0)-f(x_1)}
\end{equation}
を使用する。

```python
def secant(func: Callable[[float],float],
            x_0: float, x_1: float) -> float:
    '''
    割線法のアルゴリズム
    '''
    # 代数式の数値精度
    delta: float = 1e-10

    result: float = 0
    count: int = 0
    while True:
        # 解の更新
        result = (x_1*func(x_0)-x_0*func(x_1))/(func(x_0)-func(x_1))
        count += 1

        # 終了判定
        if (abs(func(result)) < delta):
            print(f"finished: f(x)={abs(func(result)):.4e} (count={count})")
            break
        else:
            x_0 = x_1
            x_1 = result

    return result
```

```python
x = secant(non_linear_func, -3.0, 0)
print(f"Δx = {x-x_real:.6e}, f(x)={non_linear_func(x):.6e}")
```

    finished: f(x)=8.8107e-13 (count=5)
    Δx = 2.554175e-10, f(x)=-8.810730e-13

### マラー法

```python

```


# Projeto III - Computação Centífica II
### Solução do Problema Planar de Bratu
> Autor: Gil Miranda

> Contato: gil.neto@ufrj.br

> Repo: [@mirandagil](https://github.com/mirandagil)

## Dependências e bibliotecas externas

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import optimize
```

---
## Da solução Analítica

Vamos aqui programar as funções que retornam a solução analítica do problema

```python
def analytical_bratu(x, l):
    b_1 = 0
    b_2 = 16
    if l >= 1.5:
        b_2 = 8
    elif l > 1:
        b_2 = 10
    z = optimize.fixed_point(solve_z, [b_1, b_2], args=(l,0))
    if z[0] != z[1]:
        tu_1 = np.cosh((x-1/2)*z[0]/2)/np.cosh(z[0]/4)
        bra_1 = -2*np.log(tu_1)
        tu_2 = np.cosh((x-1/2)*z[1]/2)/np.cosh(z[1]/4)
        bra_2 = -2*np.log(tu_2)
        bratu = [bra_1, bra_2]
    else:
        tu = np.cosh((x-1/2)*z[0]/2)/np.cosh(z[0]/4)
        bra = -2*np.log(tu)
        bratu = bra
    return bratu

def solve_z(z, l, l1 = 0):
    return np.sqrt(2*l)*np.cosh(z/4)

def lambda_c(l):
    a = optimize.fixed_point(alpha, 5)
    return 8*(a**2 -1)

def alpha(x):
    return 1/np.tanh(x)

```

### Plotando a solução
Vamos fazer um plot da solução para $\lambda = 2.5$

```python
ls = np.linspace(0, lambda_c(0))
xs = np.linspace(0,1)
sol = [analytical_bratu(xs, l) for l in ls]
phi = analytical_bratu(xs, 2.5)
ys_0 = []
ys_1 = []

## plotting time
plt.plot(xs, phi[0], label = "1st root")
plt.plot(xs, phi[1], label = "2nd root")
plt.grid(alpha = 0.5)
plt.legend()
plt.title('Bratu Problem solution $\lambda = 2.5$')
plt.show()
```

### Visualizando a correlação de bifurcação das soluções para $\lambda < \lambda_c$

Como visto no artigo, temos raíz dupla $z_1, z_2$ para $\lambda < \lambda_c$ e raíz única para $\lambda = \lambda_c$, então plotando o maior valor de cada solução em função de cada $\lambda$ temos o seguinte gráfico

```python
for y in sol:
    if type(y) is list:
            ys_0.append(max(y[0]))
            ys_1.append(max(y[1]))
    else:
        ys_0.append(max(y))
        ys_1.append(max(y))
```

```python
plt.plot(ls, ys_0, label = 'Solução para $z_1$')
plt.plot(ls[1:], ys_1[1:], label = 'Solução para $z_2$')
plt.title('Plot da solução por $\lambda$')
plt.xlabel('$\lambda$')
plt.ylabel('max(solução)')
plt.grid(alpha = 0.3)
plt.legend()
plt.show()
```

Agora temos como comparar a solução analítica com a númerica e verificar o erro do metódo.

---
## Da solução Númerica #1

```python
### Preparando o grid de x
N = 5
h = 1/N
xs = np.arange(0,1+h,h)

eps_array = [10,1,0.1,0.01]

eps = eps_array[2]

def bvp_solve(n):
    h = 1/(n+1)
    u0 = ui = 0
    u1 = 1

    phiH = 2*np.log(np.cosh(h))
    A = 1/phiH
    B = -2/phiH
    C = 1/phiH

    M_A = A*np.eye(n,k = 1)
    M_B = B*np.eye(n)
    M_C = C*np.eye(n, k = -1)

    M = M_A + M_B + M_C
   # M = np.array([
   #     [B, A, 0, 0],
   #     [C, B, A, 0],
   #     [0, C, B, A],
   #     [0, 0, C, B]
   # ])

    b = np.zeros(n)
    u = np.linalg.solve(M,b)
    hs = [0]
    while hs[-1] <= 1:
        hs.append(hs[-1] + h)
    u = np.insert(u,0,0)
    u = np.append(u,1)
    return u, hs
```

```python
bvp_solve(5)
```

    (array([0.        , 0.16666667, 0.33333333, 0.5       , 0.66666667,
            0.83333333, 1.        ]),
     [0,
      0.16666666666666666,
      0.3333333333333333,
      0.5,
      0.6666666666666666,
      0.8333333333333333,
      0.9999999999999999,
      1.1666666666666665])

```python
plt.plot(bvp_solve(5)[1][:-1], bvp_solve(5)[0])
```

```python
48.295 * 9810
```

    473773.95

$$
\begin{align}
33 \cdot 746 &= (-10+\frac{pcman}{9810}+2.18) \cdot 0.062 \cdot 9810\\
24618 &= (-10+\frac{pcman}{9810}+2.18) \cdot 0.062 \cdot 9810\\
40.475 &= -10+\frac{pcman}{9810}+2.18\\
40.475 +7.82 &= \frac{pcman}{9810}\\
pcman &= 473773.95
\end{align}
$$

```python

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, C.D. Cooper. Partly based on content  by David Ketcheson, also under CC-BY.

# Phugoid model: bonus!

_The phugoid model of glider flight_ has been such a fun problem to showcase the power of numerical solution of differential equations, we thought you'd enjoy a bonus notebook. The previous lessons were:

* [Phugoid motion](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_01_Phugoid_Theory.ipynb) —Lays the groundwork for our fun problem, with some context, a little history and a description of the physics of phugoids: curves representing the trajectory of a glider exchanging potential and kinetic energy, with no drag.
* [Phugoid oscillation](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_02_Phugoid_Oscillation.ipynb) —Develops the simple harmonic motion of an aircraft experiencing a small perturbation from the horizontal trajectory: our opportunity to introduce Euler's method, and study its convergence via an exact solution.
* [Full phugoid motion](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb) —The full model takes into account the force of drag and results in a system of two nonlinear equations. We obtain the trajectories using Euler's method in vectorized form, introduce grid-convergence analysis and finish with the paper-airplane challenge!

That is a fantastic foundation for numerical methods. It's a good time to complement it with some theory: the first screencast of the course uses Taylor series to show that _Euler's method is a first-order method_, and we also show you graphical interpretations. Many problems require a more accurate method, though: second order or higher. Among the most popular higher-order methods that we can mention are the _Runge-Kutta methods_, developed around 1900: more than 100 years after Euler published his book containing the method now named after him!

### Euler's method is a first-order method

In this screencast, we use a Taylor series expansion to analyze Euler's method and show that it incurs a truncation error of first order. We also use a graphical interpretation to motivate the _modified_ Euler method, which achieves second order.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('6i6qhqDCViA')
```

## Second-order methods

The notebook on phugoid oscillation (lesson 2) included a study of the accuracy obtained with Euler's method, using the exact solution for the simple harmonic motion. We made a _convergence plot_ and saw that as $\Delta t$ gets smaller, the error also gets smaller.

We could have drawn a line with a slope equal to 1 on that log-log plot, and you would have seen that it was parallel to the convergence line. A slope equal to 1 on a log-log convergence plot is an indication that we have a first-order method: the error scales as ${\mathcal O}(\Delta t)$.

In lesson 3, using the full phugoid model (which is nonlinear and does not have an exact solution), we did a _grid-convergence study_ with three different grids, and obtained the _observed_ order of convergence—it was very close to 1, indicating a slope of 1 on a log-log plot.

Another way to look at an ${\mathcal O}(\Delta t)$ method is to say that the error scales _linearly_ with the step size, or that they are proportional:

$$ e \propto \Delta t.$$

where $e$ stands for the error. To get more accuracy, we could use a _second-order_ method, in which the error is ${\mathcal O}(\Delta t^2)$. In general, we say that a method is of order $p$ when the error is proportional to $(\Delta t)^p$.

In the screencast titled "Euler's method is a first-order method," we used a graphical interpretation to get an idea for improving it: by estimating an intermediate point, like the **midpoint**, we can get a better approximation of the area under the curve of $u^\prime$. The scheme has two steps and is written as:

\begin{align}
u_{n+1/2}   & = u_n + \frac{\Delta t}{2} f(u_n) \\
u_{n+1} & = u_n + \Delta t \,\, f(u_{n+1/2}).
\end{align}

This method is known as the *explicit midpoint method* or the *modified Euler method*, and it is a second-order method. Notice that we had to apply the right-hand side, $~f(u)$, twice. This idea can be extended: we could imagine estimating additional points between $u_{n}$ and $u_{n+1}$ and evaulating $~f(u)$ at the intermediate points to get higher accuracy—that's the idea behind Runge-Kutta methods.

### Runge-Kutta methods

In the modified Euler method, we improve the accuracy over Euler's method by evaluating the right-hand side of the differential equation at an intermediate point: the midpoint. The same idea can be applied again, and the function $f(u)$ can be evaluated at more intermediate points, improving the accuracy even more. This is the basis of the famous *Runge-Kutta (RK) methods*, going back to Carl Runge and Martin Kutta. The modified Euler method corresponds to _second-order_ Runge-Kutta.

Here's a bit of historical coincidence that will blow your mind: Carl Runge's daughter Iris—an accomplished applied mathematician in her own right—worked assiduously over the summer of 1909 to translate Lanchester's _"Aerodonetics."_ She also reproduced his graphical method to draw the phugoid curves (Tobies, 2012).

### Phugoid model with 2nd-order RK

Let's compute the motion of a glider under the full phugoid model using the second-order Runge-Kutta method. We'll build on the _paper airplane challenge_ of lesson 3 now, and look for the horizontal distance that the plane travels until the moment it touches the ground.

As usual, let's start by importing the libraries and modules that we need, and setting up the model parameters. We also set some default plotting formats using the [`rcParams`](http://matplotlib.org/api/matplotlib_configuration_api.html#matplotlib.rcParams) module.

```python
from math import sin, cos, log
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

In the paper-airplane challenge of lesson 3, we suggested an $L/D=5.0$ as a realistic value for paper airplanes, according to experiments, and a trim velocity of 4.9 m/s. Let's start with those values, but you could experiment changing these a bit. _What do you think will happen if you make $L/D$ higher?_

```python
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 4.9    # trim velocity in m s^{-1}
C_D = 1/5.0  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = 6.5     # start at the trim velocity (or add a delta)
theta0 = -0.1 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 2.0     # initial altitude
```

Among the initial parameters that we suggest for your first experiment, we are starting with a velocity a little higher than the trim velocity, launch the paper airplane with a negative initial angle, and take the initial height to be 2 meters—all sound like reasonable choices.

Now, we can define a few functions to carry out the computation:
* The right-hand side of the phugoid model from [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb),
* One step of the Euler's method that we learned in [Lesson 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_02_Phugoid_Oscillation.ipynb), and
* Differences with respect to a fine grid, as in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb).

```python
def f(u):
    """Returns the right-hand side of the phugoid system of equations.

    Parameters
    ----------
    u : array of float
        array containing the solution at time n.

    Returns
    -------
    dudt : array of float
        array containing the RHS given u.
    """

    v = u[0]
    theta = u[1]
    x = u[2]
    y = u[3]
    return numpy.array([-g*sin(theta) - C_D/C_L*g/v_t**2*v**2,
                      -g*cos(theta)/v + g/v_t**2*v,
                      v*cos(theta),
                      v*sin(theta)])

def euler_step(u, f, dt):
    """Returns the solution at the next time-step using Euler's method.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        approximate solution at the next time step.
    """

    return u + dt * f(u)

def get_diffgrid(u_current, u_fine, dt):
    """Returns the difference between one grid and the fine one using L-1 norm.

    Parameters
    ----------
    u_current : array of float
        solution on the current grid.
    u_finest : array of float
        solution on the fine grid.
    dt : float
        time-increment on the current grid.

    Returns
    -------
    diffgrid : float
        difference computed in the L-1 norm.
    """

    N_current = len(u_current[:,0])
    N_fine = len(u_fine[:,0])

    grid_size_ratio = numpy.ceil(N_fine/N_current)

    diffgrid = dt * numpy.sum( numpy.abs(\
            u_current[:,2]- u_fine[::grid_size_ratio,2]))

    return diffgrid

```

Next, we also need to define the function `rk2_step()` that computes the next time step using the *modified Euler* method of equations $(1)$ and $(2)$, above, otherwise known as 2nd-order Runge-Kutta or RK2. This function will be called over and over again within the time loop.

```python
def rk2_step(u, f, dt):
    """Returns the solution at the next time-step using 2nd-order Runge-Kutta.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        solution at the next time step.
    """
    u_star = u + 0.5*dt*f(u)
    return u + dt*f(u_star)
```

Like in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), we first need to set up the time discretization, then initialize arrays to save the solution and we are set to go! The only difference this time is that we are using _both_ Euler's method and 2nd-order Runge-Kutta to get a solution, to compare the two.

```python
# set time-increment and discretize the time
T  = 15.0                           # final time
dt = 0.01                           # set time-increment
N  = int(T/dt) + 1                  # number of time-steps

# set initial conditions
u_euler = numpy.empty((N, 4))
u_rk2   = numpy.empty((N, 4))

# initialize the array containing the solution for each time-step
u_euler[0] = numpy.array([v0, theta0, x0, y0])
u_rk2[0]   = numpy.array([v0, theta0, x0, y0])

# use a for loop to call the function rk2_step()
for n in range(N-1):

    u_euler[n+1] = euler_step(u_euler[n], f, dt)
    u_rk2[n+1]   = rk2_step(u_rk2[n], f, dt)
```

Now we can get the position of the glider in time, according to both Euler's method and the 2nd-order Runge-Kutta method, by extracting the appropriate portions of the solution arrays:

```python
x_euler = u_euler[:,2]
y_euler = u_euler[:,3]
x_rk2 = u_rk2[:,2]
y_rk2 = u_rk2[:,3]
```

##### How far will it fly before touching the ground?

As the $y$-axis measures the vertical coordinate with respect to the ground, negative values of $y$ don't have any physical meaning: the glider would have hit the ground by then! To find out if there are any negative $y$ values we can use the handy function [`numpy.where`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html). This function returns the **indices** of the elements in an array that match a given condition. For example, `numpy.where(y_euler<0)[0]` gives an array of the indices `i` where `y_euler[i]<0` (the `[0]` is necessary as `numpy.where` returns an array, which in this case contains a single line). If no elements of the array match the condition, the array of indices comes out empty.

From the physical problem, we know that once there is one negative value, the glider has hit the ground and all the remaining time-steps are unphysical. Therefore, we are interested in finding the _first_ index where the condition applies, given by `numpy.where(y_euler<0)[0][0]`—do read the  documentation of the function if you need to!

```python
# get the index of element of y where altitude becomes negative
idx_negative_euler = numpy.where(y_euler<0.0)[0]
if len(idx_negative_euler)==0:
    idx_ground_euler = N-1
    print ('Euler integration has not touched ground yet!')
else:
    idx_ground_euler = idx_negative_euler[0]

idx_negative_rk2 = numpy.where(y_rk2<0.0)[0]
if len(idx_negative_rk2)==0:
    idx_ground_rk2 = N-1
    print ('Runge-Kutta integration has not touched ground yet!')
else:
    idx_ground_rk2 = idx_negative_rk2[0]
```

##### Do Euler and RK2 produce the same solution?

An easy way to compare the numerical results obtained with the Euler and 2nd-order Runge-Kutta methods is using [`numpy.allclose`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.allclose.html). This function compares each element of two arrays and returns `True` if each comparison is within some relative tolerance. Here, we use the default tolerance: $10^{-5}$.

```python
# check to see if the paths match
print('Are the x-values close? {}'.format(numpy.allclose(x_euler, x_rk2)))
print('Are the y-values close? {}'.format(numpy.allclose(y_euler, y_rk2)))
```

    Are the x-values close? False
    Are the y-values close? False

Hmmm, they do differ. Maybe $10^{-5}$ is too tight a tolerance, considering we're using a somewhat coarse grid with first- and second-order methods. Perhaps we can assess this visually, by plotting the glider's path? Study the code below, where we are plotting the path twice, taking a closer look in the second plot by "zooming in" to the beginning of the flight.

```python
# plot the glider path
pyplot.figure(figsize=(10,6))
pyplot.subplot(121)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_euler[:idx_ground_euler], y_euler[:idx_ground_euler], 'k-', label='Euler')
pyplot.plot(x_rk2[:idx_ground_rk2], y_rk2[:idx_ground_rk2], 'r--', label='RK2')
pyplot.title('distance traveled: {:.3f}'.format(x_rk2[idx_ground_rk2-1]))
pyplot.legend();

# Let's take a closer look!
pyplot.subplot(122)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_euler, y_euler, 'k-', label='Euler')
pyplot.plot(x_rk2, y_rk2, 'r--', label='RK2')
pyplot.xlim(0,5)
pyplot.ylim(1.8,2.5);

```

From far away, the Euler and RK2 methods seem to be producing similar answers. However, if we take a closer look, small differences become evident. Keep in mind that we are solving the same equation and both methods will converge to the same solution as we refine the grid. However, they converge to that solution at different rates: RK2 gets more accurate faster, as you make $\Delta t$ smaller.

### Grid-convergence

Just like in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), we want to do a grid-convergence study with RK2, to see if we indeed observe the expected rate of convergence. It is always an important step in a numerical solution to investigate whether the method is behaving the way we expect it to: this needs to be confirmed experimentally for every new problem we solve and for every new method we apply!

In the code below, a `for`-loop computes the solution on different time grids, with the coarsest and finest grid differing by 100x. We can use the difference between solutions to investigate convergence, as before.

```python
# use a for-loop to compute the solution on different grids
dt_values = numpy.array([0.1, 0.05, 0.01, 0.005, 0.001])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt)+1    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):
        u[n+1] = rk2_step(u[n], f, dt)

    # store the value of u related to one grid
    u_values[i] = u
```

Once those runs are done, we compute the difference between each numerical solution and the fine-grid solution.

```python
# compute diffgrid
diffgrid = numpy.empty_like(dt_values)
for i, dt in enumerate(dt_values):

    diffgrid[i] = get_diffgrid(u_values[i], u_values[-1], dt)
```

And now we plot!

```python
# plot using the matplotlib function loglog()
pyplot.figure(figsize=(6,6))
pyplot.grid(True)
pyplot.xlabel(r'$\Delta t$', fontsize=18)
pyplot.ylabel(r'$L_1$-norm of the grid differences', fontsize=18)
pyplot.xlim(1e-4,1)
pyplot.ylim(1e-4,1)
pyplot.axis('equal')
pyplot.loglog(dt_values[:-1], diffgrid[:-1], color='k', ls='--', lw=2, marker='o');
```

This is looking good! The difference relative to our fine-grid solution is decreasing with the mesh size at a faster rate than in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), but *how much faster?* When we computed the observed order of convergence with Euler's method, we got a value close to 1—it's a first-order method. Can you guess what we'll get now with RK2?

To compute the observed order of convergence, we use three grid resolutions that are refined at a constant rate, in this case $r=2$.

```python
# check convergence rate
r = 2
h = 0.001

dt_values = numpy.array([h, r*h, r**2*h])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt)+1    # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    for n in range(N-1):
        ### call rk2_step() ###
        u[n+1] = rk2_step(u[n], f, dt)

    # store the value of u related to one grid
    u_values[i] = u

# calculate the order of convergence
alpha = (log(get_diffgrid(u_values[2], u_values[1], dt_values[2]))
         - log(get_diffgrid(u_values[1], u_values[0], dt_values[1]))) / log(r)

print('The order of convergence is alpha = {:.3f}'.format(alpha))
```

    The order of convergence is alpha = 1.983

Probably you're not too surprised to see that the observed order of convergence is close to $2$. Because we used a second-order method! This means that the numerical solution is converging with the grid resolution twice as fast compared with Euler's method in [Lesson 3](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), or in other words, the error scales as ${\mathcal O}(\Delta t^2)$. That is a lot faster! However, we are paying a price here: second-order Runge-Kutta requires more computations per iteration.

##### Challenge task

How much longer does it take to get the solution with RK2, compared to Euler's method? Run the same solution (same time grid, same parameters), but find a way to *time* the calculation with Python, and compare the runtimes.

## Multi-step methods

The screencast *"Euler's method is a first-order method"* motivated graphically an idea to get increased accuracy: using intermediate points between $u_{n}$ and $u_{n+1}$ and evaluating the right-hand side of the differential equation at those intermediate points. The idea is to somehow get a better approximation using more data from the function $f(u)$.

Another way to bring more information about $f(u)$ into the numerical solution is to look at time data $t\lt t_{n}$. For example, we can involve in the calculation of the solution $u_{n+1}$ the known solution at $u_{n-1}$, in addition to $u_{n}$. Schemes that use this idea are called _multi-step methods_.

A classical multi-step method achieves second order by applying a _centered difference_ approximation of the derivative $u'$:

$$ u'(t) \approx \frac{u_{n+1} - u_{n-1}}{2\Delta t}.$$

Isolate the future value of the solution $u_{n+1}$ and apply the differential equation $u'=f(u)$, to get the following formula for this method:

$$ u_{n+1} = u_{n-1} + 2\Delta t \, f(u_n),$$

This scheme is known as the **leapfrog method**. Notice that it is using the right-hand side of the differential equation, $f(u)$, evaluated at the _midpoint_ between $u_{n-1}$ and $u_{n+1}$, where the time interval between these two solutions is $2\Delta t$. Why is it called "leapfrog"? If you imagine for a moment all of the _even_ indices $n$ of the numerical solution, you notice that these solution values are computed using the slope estimated from _odd_ values $n$, and vice-versa.

Let's define a function that computes the numerical solution using the leapfrog method:

```python
def leapfrog_step(unm1, u, f, dt):
    """Returns the solution time-step n+1) using Euler's method.

    Parameters
    ----------
    unm1 : array of float
        solution at time-step n-1.
    u : array of float
        solution at time-step n.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        solution at time-step n+1.
    """
    return unm1 + 2.0*dt*f(u)
```

But wait ... what will we do at the _initial_ time step, when we don't have information for $u_{n-1}$? This is an issue with all multi-step methods: we say that they are _not self-starting_. In the first time step, we need to use another method to get the first "kick"—either Euler's method or 2nd-order Runge Kutta could do: let's use RK2, since it's also second order.

For this calculation, we are going to re-enter the model parameters in the code cell below, so that later on we can experiment here using the leapfrog method and different starting values. At the end of this notebook, we'll give you some other model parameters to try that will create a very interesting situation!

```python
# model parameters:
g = 9.8      # gravity in m s^{-2}
v_t = 4.9    # trim velocity in m s^{-1}
C_D = 1/5.0  # drag coefficient --- or D/L if C_L=1
C_L = 1.0    # for convenience, use C_L = 1

### set initial conditions ###
v0 = 6.5     # start at the trim velocity (or add a delta)
theta0 = -0.1 # initial angle of trajectory
x0 = 0.0     # horizotal position is arbitrary
y0 = 2.0     # initial altitude

# set time-increment and discretize the time
T  = 15.0                           # final time
dt = 0.01                             # set time-increment
N  = int(T/dt) + 1                   # number of time-steps

# set initial conditions
u_leapfrog = numpy.empty((N, 4))

# initialize the array containing the solution for each time-step
u_leapfrog[0] = numpy.array([v0, theta0, x0, y0])

# first step using RK2
u_leapfrog[1] = rk2_step(u_leapfrog[0], f, dt)
```

Now we have all the required information to loop in time using the leapfrog method. The code cell below calls the leapfrog function for each time step.

```python
# use a for loop to call the function leapfrog_step()
for n in range(1,N-1):

    u_leapfrog[n+1] = leapfrog_step(u_leapfrog[n-1], u_leapfrog[n], f, dt)
```

Like before, we extract from the solution array the information about the glider's position in time and find where it reaches the ground.

```python
# get the glider position in time
x_leapfrog = u_leapfrog[:,2]
y_leapfrog = u_leapfrog[:,3]

# get the index of element of y where altitude becomes negative
idx_negative_leapfrog = numpy.where(y_leapfrog<0.0)[0]

if len(idx_negative_leapfrog)==0:
    idx_ground_leapfrog = N-1
    print ('The glider has not reached the ground yet!')
else:
    idx_ground_leapfrog = idx_negative_leapfrog[0]
```

Plotting the glider's trajectory with both the leapfrog and RK2 methods, we find that the solutions are very close to each other now: we don't see the differences that were apparent when we compared Euler's method and RK2.

```python
# plot the glider path
pyplot.figure(figsize=(11,8))
pyplot.subplot(121)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_leapfrog[:idx_ground_leapfrog], y_leapfrog[:idx_ground_leapfrog], color='k', ls='-', lw=2)
pyplot.title('distance traveled: {:.3f}'.format(x_leapfrog[idx_ground_leapfrog-1]), fontsize=18);

# Let's take a closer look!
pyplot.subplot(122)
pyplot.grid(True)
pyplot.xlabel('$x$')
pyplot.ylabel('$y$')
pyplot.plot(x_leapfrog[:idx_ground_leapfrog], y_leapfrog[:idx_ground_leapfrog], color='k', ls=':', lw=2)
pyplot.plot(x_rk2, y_rk2, 'r--', label='RK2')
pyplot.xlim(0,5)
pyplot.ylim(1.8,2.5);
```

What about the observed order of convergence? We'll repeat the process we have used before, with a grid-refinement ratio $r=2$ ... here we go:

```python
# check convergence rate
r = 2
h = 0.001

dt_values = numpy.array([h, r*h, r**2*h])

u_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):

    N = int(T/dt) + 1   # number of time-steps

    ### discretize the time t ###
    t = numpy.linspace(0.0, T, N)

    # initialize the array containing the solution for each time-step
    u = numpy.empty((N, 4))
    u[0] = numpy.array([v0, theta0, x0, y0])

    # time loop
    u[1] = rk2_step(u[0], f, dt)
    for n in range(1, N-1):
        u[n+1] = leapfrog_step(u[n-1], u[n], f, dt)

    # store the value of u related to one grid
    u_values[i] = u

# calculate the order of convergence
alpha = (log(get_diffgrid(u_values[2], u_values[1], dt_values[2]))
         - log(get_diffgrid(u_values[1], u_values[0], dt_values[1]))) / log(r)

print('The order of convergence is alpha = {:.3f}'.format(alpha))
```

    The order of convergence is alpha = 2.186

We now have numerical evidence that our calculation with the leapfrog method indeed exhibits second-order convergence, i.e., the method is ${\mathcal O}(\Delta t^2)$. _The leapfrog method is a second-order method_. Good job!

### But chew on this ...

Go back to the cell that re-enters the model parameters, just above the leapfrog-method time loop, and change the following: the initial height `y0` to 25, and the final time `T` to 36. Now re-run the leapfrog calculation and the two code cells below that, which extract the glider's position and plot it.

_What is going on?_

## Reference

Tobies, R. "Iris Runge: A life at the crossroads of mathematics, science and industry," Springer Basel, 1st ed. (2012). [Read on Google books, page 73](http://books.google.com/books?id=EDm0eQqFUQ4C&lpg=PA73&dq=%22I%20have%20been%20making%20good%20progress%20with%20Lanchester.%20The%20second%20chapter%20is%20already%20on%20your%20desk%22&pg=PA73#v=onepage&q=%22I%20have%20been%20making%20good%20progress%20with%20Lanchester.%20The%20second%20chapter%20is%20already%20on%20your%20desk%22&f=false).

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

</style>


# 词频、互信息、信息熵发现中文新词

**新词发现**任务是中文自然语言处理的重要步骤。**新词**有“新”就有“旧”，属于一个相对个概念，在相对的领域（金融、医疗），在相对的时间（过去、现在）都存在新词。[文本挖掘](https://zh.wikipedia.org/wiki/文本挖掘)会先将文本[分词](https://zh.wikipedia.org/wiki/中文自动分词)，而通用分词器精度不过，通常需要添加**自定义字典**补足精度，所以发现新词并加入字典，成为文本挖掘的一个重要工作。

[**单词**](https://zh.wikipedia.org/wiki/單詞)的定义，来自维基百科的定义如下：
>在语言学中，**单词**（又称为词、词语、单字；英语对应用语为“word”）是能独立运用并含有语义内容或语用内容（即具有表面含义或实际含义）的最小单位。单词的集合称为词汇、术语，例如：所有中文单词统称为“中文词汇”，医学上专用的词统称为“医学术语”等。词典是为词语提供音韵、词义解释、例句、用法等等的工具书，有的词典只修录特殊领域的词汇。

单从语义角度，“苹果“的法语是"pomme"，而“土豆”的法语是"pomme de terre"，若按上面的定义，“土豆”是要被拆的面目全非，但"pomme de terre"是却是表达“土豆”这个语义的最小单位；在机构名中，这类问题出现的更频繁，"Paris 3"是"巴黎第三大学"的简称，如果"Paris"和"3"分别表示地名和数字，那这两个就无法表达“巴黎第三大学”的语义。而中文也有类似的例子，“北京大学”的”北京“和”大学“都可以作为一个最小单位来使用，分别表示”地方名“和“大学”，如果这样分词，那么就可以理解为“北京的大学”了，所以“北京大学”是一个表达语义的最小单位。前几年有部电影《夏洛特烦恼》，我们是要理解为“夏洛特 烦恼“还是”夏洛 特 烦恼“，这就是很经典的分词问题。

但是从语用角度，这些问题似乎能被解决，我们知道"pomme de terre"在日常生活中一般作为“土豆”而不是“土里的苹果”，在巴黎学习都知道“Paris 3”，就像我们提到“北京大学”特指那所著名的高等学府一样。看过电影《夏洛特烦恼》的观众很容易的就能区分这个标题应该看为“夏洛 特 烦恼”。

发现新词的方法，《[互联网时代的社会语言学：基于SNS的文本数据挖掘](http://www.matrix67.com/blog/archives/5044]) 》一文，里面提到的给每一个文本串计算**文本片段**的**凝固程度**和文本串对外的使用**自由度**，通过设定阈值来将文本串分类为词和非词两类。原文给了十分通俗易懂的例子来解释凝固度和自动度。这里放上计算方法。这个方法还有许多地方需要优化，在之后的实践中慢慢调整了。

## 文本片段

**文本片段**，最常用的方法就是[n元语法(ngram)](https://zh.wikipedia.org/wiki/N元语法)，将分本分成多个n长度的文本片段。数据结构，这里采用Trie树的方案，这个方案是简单容易实现，而且用Python的字典做Hash索引实现起来也很优美，唯独的一个问题是所有的数据都存在内存中，这会使得内存占用量非常大，如果要把这个工程化使用，还需要采用其他方案，比如硬盘检索。

```python
class TrieNode(object):
    def __init__(self,
                 frequence=0,
                 children_frequence=0,
                 parent=None):

        self.parent = parent
        self.frequence = frequence
        self.children = {}
        self.children_frequence = children_frequence

    def insert(self, char):
        self.children_frequence += 1
        self.children[char] = self.children.get(char, TrieNode(parent=self))
        self.children[char].frequence += 1
        return  self.children[char]

    def fetch(self, char):
        return self.children[char]

class TrieTree(object):
    def __init__(self, size=6):
        self._root = TrieNode()
        self.size = size

    def get_root(self):
        return self._root

    def insert(self, chunk):
        node = self._root
        for char in chunk:
            node = node.insert(char)
        if len(chunk) < self.size:
            # add symbol "EOS" at end of line trunck
            node.insert("EOS")

    def fetch(self, chunk):
        node = self._root
        for char in chunk:
            node = node.fetch(char)
        return node
```

Trie树的结构上，我添加了几个参数，parent，frequence，children_frequence，他们分别是：
- parent，当前节点的父节点，如果是“树根”的时候，这个父节点为空；
- frequence，当前节点出现的频次，在Trie树上，也可以表示某个文本片段的频次，比如"中国"，“国”这个节点的frequence是100的时候，“中国”俩字也出现了100次。这个可以作为最后的词频过滤用。
- children_frequence，当前接点下有子节点的"frequence"的总和。比如在刚才的例子上加上“中间”出现了99次，那么“中”这个节点的children_frequence的值是199次。
这样的构造让第二部分的计算更加方面。

这个任务中需要构建两棵Trie树，表示正向和反向两个字符片段集。

## 自由度

**自由度**，使用信息论中的[信息熵](https://zh.wikipedia.org/wiki/熵_(信息论))构建文本片段左右熵，公式[1]。熵越大，表示该片段和左右邻字符相互关系的不稳定性越高，那么越有可能作为独立的片段使用。公式[1]第一个等号后面的I(x)表示x的自信息。
\begin{align}
 H(X) = \sum_{i} {\mathrm{P}(x_i)\,\mathrm{I}(x_i)} = -\sum_{i} {\mathrm{P}(x_i) \log \mathrm{P}(x_i)} [1]
\end{align}

```python
def calc_entropy(chunks, ngram):
    """计算信息熵
    Args:
        chunks，是所有数据的文本片段
        ngram，是Trie树
    Return:
        word2entropy，返回一个包含每个chunk和对应信息熵的字典。
    """
    def entropy(sample, total):
        """Entropy"""
        s = float(sample)
        t = float(total)
        result = - s/t * math.log(s/t)
        return result

    def parse(chunk, ngram):
        node = ngram.fetch(chunk)
        total = node.children_frequence
        return sum([entropy(sub_node.frequence,
                           total) for sub_node in node.children.values()])

    word2entropy = {}
    for chunk in chunks:
        word2entropy[chunk] = parse(chunk, ngram)
    return word2entropy
```

## 凝固度

**凝固度**，用信息论中的**互信息**表示，公式[2]。在概率论中，如果x跟y不相关，则p(x,y)=p(x)p(y)。二者相关性越大，则p(x,y)就相比于p(x)p(y)越大。用后面的式子可能更好理解，在y出现的情况下x出现的条件概率p(x|y)除以x本身出现的概率p(x)，自然就表示x跟y的相关程度。
\begin{align}
I(x;y) = \log\frac{p(x,y)}{p(x)p(y)} = \log\frac{p(x|y)}{p(x)} = \log\frac{p(y|x)}{p(y)} [2]
\end{align}

这里比较容易产生一个概念的混淆，维基百科将式[2]定义为[点互信息](https://en.wikipedia.org/wiki/Pointwise_mutual_information)，[互信息](https://zh.wikipedia.org/wiki/互信息)的定义如下：
\begin{align}
I(X;Y) = \sum_{y \in Y} \sum_{x \in X}
                 p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
                              \right) }\ [3]
\end{align}
在傅祖芸编著的《信息论——基础理论与应用（第4版）》的绪论中，把式[2]定义为互信息，而式[3]定义为平均互信息，就像信息熵指的是**平均自信息**。

```python
def calc_mutualinfo(chunks, ngram):
    """计算互信息
    Args:
        chunks，是所有数据的文本片段
        ngram，是Trie树
    Return:
        word2mutualinfo，返回一个包含每个chunk和对应互信息的字典。
    """
    def parse(chunk, root):
        sub_node_y_x = ngram.fetch(chunk)
        node = sub_node_y_x.parent
        sub_node_y = root.children[chunk[-1]]

        # 这里采用互信息log(p(y|x)/p(y))的计算方法
        prob_y_x = float(sub_node_y_x.frequence) / node.children_frequence
        prob_y = float(sub_node_y.frequence) / root.children_frequence
        mutualinfo = math.log(prob_y_x / prob_y)
        return mutualinfo, sub_node_y_x.frequence

    word2mutualinfo = {}
    root = ngram.get_root()
    for chunk in chunks:
        word2mutualinfo[chunk] = parse(chunk, root)
    return word2mutualinfo
```

## 过滤

最终计算得出互信息、信息熵，甚至也统计了词频，最后一步就是根据阈值对词进行过滤。

```python
def _fetch_final(fw_entropy，
                 bw_entropy,
                 fw_mi,
                 bw_mi
                 entropy_threshold=0.8,
                 mutualinfo_threshold=7,
                 freq_threshold=10):
        final = {}
        for k, v in fw_entropy.items():
            last_node = self.fw_ngram
            if k[::-1] in bw_mi and k in fw_mi:
                mi_min = min(fw_mi[k][0], bw_mi[k[::-1]][0])
                word_prob = min(fw_mi[k][1], bw_mi[k[::-1]][1])
                if mi_min < mutualinfo_threshold:
                    continue
            else:
                continue
            if word_prob < freq_threshold:
                 continue
            if k[::-1] in bw_entropy:
                en_min = min(v, bw_entropy[k[::-1]])
                if en_min < entropy_threshold:
                    continue
            else:
                continue
            final[k] = (word_prob, mi_min, en_min)
        return final
```

## 结果

最终，通过这个方法对这次十九大的开幕发言做的一个词汇发现，ngram的n=10，结果按词频排序输出，可以发现这次十九大谈了许多内容，不一一说了。这个结果还存在不少问题，比如“二〇”，这在阈值的设置上还不够准确，可以尝试使用机器学习的方法来获取阈值。

经济|70 改革|69 我们|64 必须|61 领导|60 完善|57 历史|44 不断|43 群众|43 教育|43 战略|42 思想|40 世界|39 问题|37 提高|37 组织|36 监督|35 加快|35 依法|34 精神|33 团结|33 复兴|32 保障|31 奋斗|30 根本|29 环境|29 军队|29 开放|27 服务|27 理论|26 干部|26 创造|26 基础|25 意识|25 维护|25 协商|24 解决|24 贯彻|23 斗争|23 目标|21 统筹|20 始终|19 方式|19 水平|19 科学|19 利益|19 市场|19 基层|19 积极|18 马克思|18 反对|18 道路|18 自然|18 增长|17 科技|17 稳定|17 原则|17 两岸|17 取得|16 质量|16 农村|16 矛盾|16 协调|15 巩固|15 收入|15 绿色|15 自觉|15 方针|15 纪律|15 长期|15 保证|15 同胞|15 命运|14 美好生活|14 五年|14 传统|14 繁荣|14 没有|14 使命|13 广泛|13 日益|13 价值|13 健康|13 资源|13 参与|13 突出|13 腐败|13 充分|13 梦想|13 任何|13 二〇|13 代表|12 阶段|12 深刻|12 布局|12 区域|12 贸易|12 核心|12 城乡|12 生态文明|12 工程|12 任务|12 地区|12 责任|12 认识|12 胜利|11 贡献|11 覆盖|11 生态环境|11 具有|11 面临|11 各种|11 培育|11 企业|11 继续|10 团结带领|10 提升|10 明显|10 弘扬|10 脱贫|10 贫困|10 标准|10 注重|10 基本实现|10 培养|10 青年|10

## 代码下载地址

git clone https://github.com/Ushiao/new-word-discovery.git


```python
import numpy as np
import sympy as sym
import pydae.build as db
```

```python
params_dict = {'R_stor':0.05,'C_u':100.0, 'soc_ref':0.5,'K_p':1.0,'K_i':1.0,
              'V_u_max':305, 'V_u_min':295}  # parameters with default values

u_ini_dict = {'p_u_ref':1.0}  # input for the initialization problem
u_run_dict = {'p_u_ref':1.0}  # input for the running problem, its value is updated

x_list = ['e_u','xi_esoc']  # dynamic states
y_ini_list = ['p_u_soc','p_u','i_u','v_u','soc']          # algebraic states for the initialization problem
y_run_list = ['p_u_soc','p_u','i_u','v_u','soc']        # algebraic for the running problem

sys_vars = {'params':params_dict,
            'u_list':u_run_dict,
            'x_list':x_list,
            'y_list':y_run_list}

exec(db.sym_gen_str())  # exec to generate the required symbolic varables and constants
```

```python
de_u = 1.0/C_u*(-i_u)
dxi_esoc = soc_ref - soc

g_p_u_soc = -p_u_soc +  K_p*(soc_ref - soc) + K_i*xi_esoc
g_p_u = -p_u + p_u_ref + p_u_soc
g_i_u = i_u*v_u - p_u
g_v_u = e_u - i_u*R_stor - v_u
#!soc = (e_u - V_u_min)/(V_u_max - V_u_min)
E_max = 0.5*C_u*(V_u_max*V_u_max - V_u_min*V_u_min)
eq_soc = -soc + 0.5*C_u*(e_u*e_u - V_u_min*V_u_min)/E_max
```

```python
sys = {'name':'bess_1',
       'params_dict':params_dict,
       'f_list':[de_u,dxi_esoc],
       'g_list':[g_p_u_soc,g_p_u,g_i_u,g_v_u,eq_soc],
       'x_list':[ e_u, xi_esoc],
       'y_ini_list':y_ini_list,
       'y_run_list':y_run_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_ini_dict,
       'h_dict':{'p_u_ref':p_u_ref}}

sys = db.system(sys)
db.sys2num(sys)
```

    jacobians respect u = 0

```python

```


JuPOT - Tutorial
================

```julia
push!(LOAD_PATH, "$(homedir())/desktop/financial-opt-tools/src")

using JuPOT

# Generate synthetic data sets for Demonstration
############
# Assets
############
n = 10 # No. Of Assets
returns = rand(n)
covarariance = let
    S = randn(n, n)
    S'S + eye(n)
end
names = [randstring(3) for i in 1:n] # List of asset names

# Assets data structure containing, names, expected returns, covarariance
assets = AssetsCollection(names, returns, covarariance)
```

    WGd    0.8750696218393272
    NUU    0.0909824764542102
    XkG    0.0421296129858959
    BVj    0.3566575879209919
    pRr    0.539297484964939
    Jzy    0.5215886520600004
    kiR    0.6376321186088576
    5ML    0.2587373064380125
    4Wu    0.6969136663434587
    bTp    0.4212944263872922

## Simple MVO

\begin{align}
&\text{minimize} && w^\top\Sigma w \\
&\text{subject to} && \mu^\top w\geq r_{\min} \\
&       && \mathbf{1}^\top w = 1 \\
&       && w \succeq 0 \\
&       && \sum_{i=1}^{\infty}{w_i} \succeq 0 \\
\end{align}

```julia
######################################################
################### SIMPLE MVO #######################
######################################################

# Example 1: Basic
target_return = 0.2
# refer to model definition for keyword arguments, etc.
mvo = SimpleMVO(assets, target_return; short_sale=false)
```

     Sense: Min

     Variables:
    w[1:10] >= 0

     Objective Function:
      dot(w,10x10 Array{Float64,2}:
     10.3866    -0.89143   -1.33051   …   0.198312   4.50745    2.37707
     -0.89143    7.09333   -3.45644       2.09936    0.348747  -1.71617
     -1.33051   -3.45644   10.2676       -5.09831   -5.36072    1.8266
     -0.844028   3.28496    0.410415      2.64611    1.41345    2.39396
     -3.33323    1.31546    0.239667      0.212938  -6.14246   -1.16482
     -1.50885    2.61073    1.95697   …  -5.33638   -3.99037   -2.14628
      5.32728   -1.83709    2.18061      -5.61788   -0.872873   3.54495
      0.198312   2.09936   -5.09831       9.98955    5.32627    0.232497
      4.50745    0.348747  -5.36072       5.32627   15.8035     2.1874
      2.37707   -1.71617    1.8266        0.232497   2.1874     7.2731   * w)

     Constraints:
    dot([0.8750696218393272,0.0909824764542102,0.0421296129858959,0.3566575879209919,0.539297484964939,0.5215886520600004,0.6376321186088576,0.2587373064380125,0.6969136663434587,0.4212944263872922],w) ≥ 0.2
    dot(ones(10),w) == 1

     Assets:
     WGd    0.8750696218393272
    NUU    0.0909824764542102
    XkG    0.0421296129858959
    BVj    0.3566575879209919
    pRr    0.539297484964939
    Jzy    0.5215886520600004
    kiR    0.6376321186088576
    5ML    0.2587373064380125
    4Wu    0.6969136663434587
    bTp    0.4212944263872922

```julia
optimize(mvo)
```

    10-element Array{Float64,1}:
     0.0170244
     0.0753511
     0.183185
     1.35262e-10
     0.185192
     0.0834505
     0.145646
     0.18291
     0.127241
     2.27348e-9

Constraints
-----------

**constraint1**: $\sum$ asset group constraints

```julia
##################################
# Example 2: Adding a Constraint #
##################################
function genTechIndicator()
    [0,0,1,1,0,1,0,1,1,0]
end

# Adding a simple weight constraint
constraints = Dict((:constraint1 => :(dot(w,tech) <= tech_thresh)),
                   (:constraint2 => :(dot(w,fin) <= Fin_thresh)))
parameters = Dict(:tech=>genTechIndicator(),
                  :tech_thresh => 0.3,
                  :fin=> [1,1,0,0,1,0,1,0,0,0],
                  :Fin_thresh => 0.05)
# refer to model definition for keyword arguments, etc.
mvo = SimpleMVO(assets, target_return, constraints; short_sale=false)
w = optimize(mvo, parameters)

# add checkers for constraints
```

    10-element Array{Float64,1}:
     1.08235e-10
     0.0404122
     4.03129e-10
     9.94083e-11
     0.00958781
     0.194771
     7.34928e-11
     0.105229
     2.20301e-10
     0.65

```julia
#######################################################
# Example 3: Changing a Constraint's parameter values #
#######################################################

# Changing values of an entered constraint
parameters[:tech_thresh] = 0.04

# refer to model definition for keyword arguments, etc.
mvo = SimpleMVO(assets, target_return, constraints; short_sale=false)
optimize(mvo, parameters)
```

    10-element Array{Float64,1}:
     4.9691e-11
     0.0499999
     1.53707e-10
     1.3058e-10
     6.88396e-8
     0.04
     3.35711e-11
     5.70681e-11
     8.17501e-11
     0.91

```julia
####################################
# Example 4: Deleting a Constraint #
####################################

# Removing a previously defined constraint
delete!(constraints, :constraint2)
delete!(parameters, :Fin_thresh)
# refer to model definition for keyword arguments, etc.
mvo = SimpleMVO(assets, target_return, constraints; short_sale=false)
optimize(mvo, parameters)
```

    10-element Array{Float64,1}:
     0.152713
     0.21905
     0.00936567
     1.96236e-11
     0.31893
     3.21006e-11
     0.0845705
     8.39941e-11
     0.0306343
     0.184737

```julia
##########################################
# Example 5: Adding multiple Constraints #
##########################################

# Adding a multiple weight constraints
# constraints = Dict(:constraint1 => :(w[1:5] .<= assets_1_5_max), :constraint2 => :(w[6:n] .<= assets_6_n_max))
# Adding a simple weight constraint
constraints_1 = [symbol("x$i") => :(min_thresh <= w[$i]) for i=1:n]
constraints_2 = [symbol("y$i") => :( w[$i] <= max_thresh) for i=1:n]
parameters_1 = Dict(:min_thresh => 0, :max_thresh => 0.5, :n => n)

# Different constraint sets can be merged to form new ones
constraints = merge(constraints,constraints_1,constraints_2)
parameters = merge(parameters,parameters_1)
# refer to model definition for keyword arguments, etc.
mvo = SimpleMVO(assets, target_return, constraints; short_sale=false)
optimize(mvo, parameters)
```

    10-element Array{Float64,1}:
     0.152713
     0.21905
     0.00936567
     6.82782e-12
     0.31893
     1.58286e-11
     0.0845705
     1.82652e-10
     0.0306343
     0.184737

```julia

#####################################
# Example 6: Using Different Assets #
#####################################

############
# Asset #2 #
############
n = 10 # No. Of Assets
returns_new = rand(n)
covarariance_new = let
    S = randn(n, n)
    S'S + eye(n)
end
names_new = [randstring(3) for i in 1:n]
# Assets data structure containing, names, expected returns, covarariance
assets_new = AssetsCollection(names_new, returns_new, covarariance_new)

# Using the same previously defined constraints we can run the model on a different set of assets effortlessly
mvo = SimpleMVO(assets_new, target_return; short_sale=false)
optimize(mvo, parameters)
```

    10-element Array{Float64,1}:
     0.059491
     2.11897e-8
     0.207639
     4.07243e-12
     2.90081e-12
     0.275566
     2.24249e-11
     0.258529
     0.00451188
     0.194264

```julia
# Forgot what constraints and parameters were defined for initial constraints? No Problem!
constraints # Prints the constraints
```

    Dict{Symbol,Expr} with 21 entries:
      :y6          => :(w[6] <= max_thresh)
      :x4          => :(min_thresh <= w[4])
      :x10         => :(min_thresh <= w[10])
      :y9          => :(w[9] <= max_thresh)
      :y10         => :(w[10] <= max_thresh)
      :y8          => :(w[8] <= max_thresh)
      :y4          => :(w[4] <= max_thresh)
      :y7          => :(w[7] <= max_thresh)
      :constraint1 => :(dot(w,tech) <= tech_thresh)
      :y1          => :(w[1] <= max_thresh)
      :x2          => :(min_thresh <= w[2])
      :x3          => :(min_thresh <= w[3])
      :x9          => :(min_thresh <= w[9])
      :y2          => :(w[2] <= max_thresh)
      :x1          => :(min_thresh <= w[1])
      :x7          => :(min_thresh <= w[7])
      :x5          => :(min_thresh <= w[5])
      :y3          => :(w[3] <= max_thresh)
      :y5          => :(w[5] <= max_thresh)
      :x6          => :(min_thresh <= w[6])
      :x8          => :(min_thresh <= w[8])

```julia
parameters # prints the parameters
```

    Dict{Symbol,Any} with 6 entries:
      :tech        => [0,0,1,1,0,1,0,1,1,0]
      :tech_thresh => 0.04
      :max_thresh  => 0.5
      :fin         => [1,1,0,0,1,0,1,0,0,0]
      :n           => 10
      :min_thresh  => 0

```julia
# Using old constraints on new assets
mvo = SimpleMVO(assets_new, target_return, constraints; short_sale=false)
optimize(mvo, parameters)
```

    10-element Array{Float64,1}:
     0.29987
     0.160755
     0.04
     4.19273e-13
     1.04397e-12
     3.50401e-12
     0.246086
     2.27611e-12
     7.99382e-13
     0.253289

```julia
##############################
#  Example 7 Using Robust MVO#
##############################

# refer to model definition for keyword arguments, etc
# If no uncertainty matrix is entered the model defaults
# to the ellipse whose axes are proportional to the
# individual variances of each asset

rmvo = RobustMVO(assets, target_return; short_sale=true)
optimize(rmvo, parameters)
```

    10-element Array{Float64,1}:
     0.0819074
     0.117726
     0.120242
     0.0572045
     0.159447
     0.0917164
     0.086467
     0.116344
     0.0741187
     0.0948265

```julia
#################################################################
#  Example 8 Creating Custom Functions (e.g efficient frontier) #
#################################################################

n = 20
risk = Array(Float32,n)
index = Array(Float32,n)
for i in 1:n
   target_ret = i/21
   mvo = SimpleMVO(assets, target_ret; short_sale=true)
   optimize(mvo, parameters)
   risk[i] = mvo.objVal
   index[i] = target_ret
end
```

```julia
risk
```

    20-element Array{Float32,1}:
     0.461836
     0.461836
     0.461836
     0.461836
     0.461836
     0.461836
     0.461836
     0.461985
     0.483955
     0.543207
     0.639742
     0.773559
     0.944659
     1.15304
     1.3987
     1.68165
     2.00188
     2.35939
     2.75418
     3.18626

```julia
index
```

    20-element Array{Float32,1}:
     0.047619
     0.0952381
     0.142857
     0.190476
     0.238095
     0.285714
     0.333333
     0.380952
     0.428571
     0.47619
     0.52381
     0.571429
     0.619048
     0.666667
     0.714286
     0.761905
     0.809524
     0.857143
     0.904762
     0.952381

```julia

```


```python
from tectosaur2.nb_config import setup

setup()

```

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
from tectosaur2 import panelize_symbolic_surface, gauss_rule, integrate_term
from tectosaur2.laplace2d import hypersingular, adjoint_double_layer, double_layer, single_layer
from tectosaur2.hmatrix import build_tree

t = sp.var("t")
surf = panelize_symbolic_surface(t, 0 * t, t, gauss_rule(6), n_panels=1000)
kernel = hypersingular

```

```python
%%time
mat = kernel.direct(surf.pts, surf)[:, :, :, 0].reshape((-1, surf.n_pts))
```

    CPU times: user 403 ms, sys: 461 ms, total: 864 ms
    Wall time: 864 ms

```python
# plt.plot(np.log10(np.linalg.svd(mat)[1]))
plt.plot(np.log10(np.linalg.svd(mat[-200:, :200])[1]))
plt.show()

```

```python
U, S, V = np.linalg.svd(mat[-200:, :200])
np.argmax(S < 1e-10)

```

    2

```python
%%time
tree = build_tree(surf.pts, np.zeros(surf.n_pts), min_pts_per_box=30)
print("center =", tree.root.center)
print("radius =", tree.root.radius)
```

    center = [0.00000000e+00 9.05571914e-17]
    radius = 0.9999324695142032
    CPU times: user 32.5 ms, sys: 19.1 ms, total: 51.6 ms
    Wall time: 14.9 ms

```python
%matplotlib inline
from tectosaur2.hmatrix.tree import plot_tree
plot_tree(tree)
```

```python
from tectosaur2.hmatrix import traverse

direct, approx = traverse(tree.root, tree.root)

```

```python
from tectosaur2.hmatrix import TempSurface, build_temp_surface, approx_block_toy_aca

tree_surf = TempSurface(
    surf.pts[tree.ordered_idxs],
    surf.normals[tree.ordered_idxs],
    surf.quad_wts[tree.ordered_idxs],
    surf.jacobians[tree.ordered_idxs],
)

def direct_block(obs_node, src_node):
    temp_src = build_temp_surface(tree_surf, src_node.idx_start, src_node.idx_end)
    obs_pts = tree_surf.pts[obs_node.idx_start : obs_node.idx_end]
    M = kernel.direct(obs_pts, temp_src)
    return M.reshape(
        (
            obs_pts.shape[0] * kernel.obs_dim,
            (src_node.idx_end - src_node.idx_start) * kernel.src_dim,
        )
    )

```

```python
%%time
direct_blocks = [direct_block(d[0], d[1]) for d in direct]
```

    CPU times: user 19.3 ms, sys: 1.73 ms, total: 21 ms
    Wall time: 20.4 ms

```python
%%time
approx_blocks = [approx_block_toy_aca(kernel, tree_surf.pts, tree_surf, d[0], d[1], 1e-10) for d in approx]
```

    CPU times: user 1.56 s, sys: 360 ms, total: 1.92 s
    Wall time: 807 ms

```python
direct_block(*approx[0]).shape

```

    (46, 23)

```python
v = np.random.rand(mat.shape[1])

```

```python
%%time
ytrue = mat.dot(v).reshape((-1, kernel.obs_dim))

```

    CPU times: user 45.2 ms, sys: 55.3 ms, total: 100 ms
    Wall time: 13 ms

```python
%%time
ytree = np.zeros(mat.shape[0])
treev = v[tree.ordered_idxs]

for i, (obs_node, src_node) in enumerate(direct):
    ytree[kernel.obs_dim*obs_node.idx_start:kernel.obs_dim*obs_node.idx_end] += direct_blocks[i].dot(treev[src_node.idx_start:src_node.idx_end])

for i, (obs_node, src_node) in enumerate(approx):
    v_subset = treev[src_node.idx_start:src_node.idx_end]
    U, V = approx_blocks[i]
    ytree[kernel.obs_dim*obs_node.idx_start:kernel.obs_dim*obs_node.idx_end] += U.dot(V.dot(v_subset))

yh = np.zeros((mat.shape[0] // kernel.obs_dim, kernel.obs_dim))
yh[tree.ordered_idxs] = ytree.reshape((-1, kernel.obs_dim))
yh[:10] - ytrue[:10]
```

    CPU times: user 17.6 ms, sys: 8.57 ms, total: 26.2 ms
    Wall time: 8.05 ms

    array([[-7.38964445e-12, -3.65048702e-17],
           [-3.58113539e-12, -1.10292626e-21],
           [ 4.54747351e-13,  1.55365435e-23],
           [ 1.13686838e-13,  1.04824964e-27],
           [-1.02318154e-12,  0.00000000e+00],
           [-4.54747351e-13,  0.00000000e+00],
           [-4.54747351e-13,  0.00000000e+00],
           [-3.63797881e-12,  0.00000000e+00],
           [-5.22959454e-12,  0.00000000e+00],
           [-4.71800377e-12,  0.00000000e+00]])

```python
approx_entries = sum(
    [U.shape[0] * U.shape[1] + V.shape[0] * V.shape[1] for U, V in approx_blocks]
)
direct_entries = sum([M.shape[0] * M.shape[1] for M in direct_blocks])
(approx_entries + direct_entries) / mat.size

```

    0.038679430555555554

```python
for i, (obs_node, src_node) in enumerate(approx[:3]):
    U, V = approx_blocks[i]
    print(
        np.sqrt(
            np.sum(
                (
                    mat[
                        kernel.obs_dim
                        * obs_node.idx_start : kernel.obs_dim
                        * obs_node.idx_end,
                        kernel.src_dim
                        * src_node.idx_start : kernel.src_dim
                        * src_node.idx_end,
                    ]
                    - U.dot(V)
                )
                ** 2
            )
        )
    )

```

    2.5661647778957052e-11
    8.960104180030024e-12
    2.7796420218408977e-11

```python

```

```python
%%time
tol = 1e-10
blocks = approx[:]
approx_obs_starts = np.array([b[0].idx_start for b in blocks])
approx_obs_ends = np.array([b[0].idx_end for b in blocks])
approx_src_starts = np.array([b[1].idx_start for b in blocks])
approx_src_ends = np.array([b[1].idx_end for b in blocks])

approx_blocks_aca = [uv for uv in aca(
    kernel,
    approx_obs_starts,
    approx_obs_ends,
    approx_src_starts,
    approx_src_ends,
    tree_surf.pts,
    tree_surf,
    np.full(len(approx), tol),
    np.full(len(approx), 200, dtype=np.int32),
    verbose=False
)]
```

    CPU times: user 270 ms, sys: 189 ms, total: 460 ms
    Wall time: 167 ms

```python
approx_blocks[0][0].shape, approx_blocks_aca[0][0].shape

```

    ((46, 7), (46, 7))

```python
for i, (obs_node, src_node) in enumerate(approx):
    U, V = approx_blocks_aca[i]
    # print(U.shape, V.shape)
    print(
        np.sqrt(
            np.sum(
                (
                    mat[
                        kernel.obs_dim
                        * obs_node.idx_start : kernel.obs_dim
                        * obs_node.idx_end,
                        kernel.src_dim
                        * src_node.idx_start : kernel.src_dim
                        * src_node.idx_end,
                    ]
                    - U.dot(V)
                )
                ** 2
            )
        )
    )

```

    3.688209901823206e-12
    2.7059580386830375e-11
    3.0836022724928006e-12
    2.366293824533172e-11
    3.1391367884622104e-11
    2.464832539736078e-11
    3.345453801651139e-12
    4.087167293511586e-12
    2.8112518206477073e-11
    2.028225490941604e-11
    3.3454524082356766e-12
    4.626256903904842e-12
    1.4366090459153643e-12
    2.3785622043636096e-11
    1.5668519108727916e-12
    2.9686204890466536e-11
    4.040208259552856e-11
    3.000571480505668e-12
    3.154045256655711e-11
    5.1013703796021566e-12
    5.5952406501077546e-11
    4.536572251125396e-12
    4.344752748437839e-11
    4.12099246548554e-12
    3.9635883533775507e-11
    5.102682007820651e-12
    4.8920085305037645e-11
    4.379105621379295e-12
    3.687748382969594e-12
    2.707797519451707e-11
    3.0836022724928006e-12
    2.364585909501039e-11
    3.13709596943891e-11
    2.4674399977006422e-11
    3.3454528512255348e-12
    4.087167455420933e-12
    2.8089314938367208e-11
    2.029712867660879e-11
    3.3454516182939294e-12
    4.618743153257659e-12
    1.4366090459153643e-12
    2.3785623791706535e-11
    1.5668515040580162e-12
    2.968620611571375e-11
    4.66724473577656e-12
    4.379105968962448e-12
    3.0005716061083707e-12
    3.154042538744386e-11
    5.102681804837178e-12
    5.5952446233396514e-11
    4.326557709033584e-11
    4.272517725999873e-11
    4.680532929441529e-12
    4.5365727565051595e-12
    4.120992249648728e-12
    3.9635883533775507e-11
    5.102682007820651e-12
    4.8920085329978045e-11
    4.058799024178058e-11
    4.369631158230885e-11
    3.687748960806218e-12
    2.7077974631157195e-11
    3.0836030083698937e-12
    2.3712612896478415e-11
    3.135965758274666e-11
    2.4648325940506298e-11
    3.3454529900879837e-12
    4.087168098980339e-12
    2.808931532797382e-11
    2.0282254861878948e-11
    3.3454530710262565e-12
    4.618743385123591e-12
    1.4366095421055137e-12
    2.3707119857209517e-11
    1.566851543452899e-12
    2.968620656042394e-11
    4.048615083272881e-11
    3.000571128726197e-12
    3.154044799474189e-11
    5.102681804837178e-12
    5.595248826560423e-11
    4.5365722176635364e-12
    4.53648493238086e-12
    4.120992285250623e-12
    3.9635884755159593e-11
    5.102682007820651e-12
    4.892006170393132e-11
    4.0687746941905615e-11
    3.688916385293867e-12
    2.707100194334829e-11
    3.0836030083698937e-12
    2.3712612896478415e-11
    3.135965758274666e-11
    2.4663874076955135e-11
    3.3454524151272236e-12
    4.087168098980339e-12
    2.808931532797382e-11
    2.0282254861878948e-11
    3.3454528620270983e-12
    4.625510084894809e-12
    1.4366095421055137e-12
    2.3785623729909962e-11
    1.566851543452899e-12
    2.958458738052048e-11
    4.658133579488804e-12
    4.3696452485927266e-11
    4.047136981327584e-11
    3.000571128726197e-12
    3.154045205854804e-11
    5.102681804837178e-12
    5.595240635291434e-11
    4.536571640720205e-12
    4.667244679820615e-12
    4.293028128304844e-11
    4.676084758025128e-12
    4.272542082013044e-11
    4.536572224332457e-12
    4.120992285250623e-12
    3.9635884755159593e-11
    5.102682007820651e-12
    4.892008043836236e-11
    4.0402176161354135e-11
    4.6805328984750975e-12
    4.32586194863294e-11
    3.688914955591416e-12
    2.7059581414057966e-11
    3.083234779860407e-12
    2.3662937935715318e-11
    3.135965731550956e-11
    2.4648325560891518e-11
    3.3444400724033915e-12
    4.087166907860128e-12
    2.808931438183142e-11
    2.0271585170102044e-11
    3.3454527159876673e-12
    4.618742836586559e-12
    1.4366120049546033e-12
    2.37343008846144e-11
    1.5668510761957497e-12
    2.964449444425456e-11
    4.060491011520864e-11
    3.0005714925539577e-12
    3.15404282966368e-11
    5.102681804837178e-12
    5.595248835164416e-11
    4.536571512952352e-12
    4.53657254365487e-12
    4.120991821136286e-12
    3.9635883533775507e-11
    5.102682007820651e-12
    4.892006747608223e-11
    4.04713698105683e-11
    3.687748382969594e-12
    2.7059581414057966e-11
    3.0836022724928006e-12
    2.3667175712274622e-11
    3.136114063100273e-11
    2.4648325560891518e-11
    3.345453463710097e-12
    4.087166907860128e-12
    2.8108085261676428e-11
    2.0282254792364027e-11
    3.3454527159876673e-12
    4.6187515579899055e-12
    1.4366120049546033e-12
    2.3785622042780097e-11
    1.5668510761957497e-12
    2.9584584921272245e-11
    4.679222397267358e-12
    4.040206894182926e-11
    3.0005710536711437e-12
    3.154042674114336e-11
    5.102683244015579e-12
    5.5952489215995334e-11
    4.5365717533859375e-12
    4.2725175454420165e-11
    4.678442374024735e-12
    4.32655599719925e-11
    4.1209904259410585e-12
    3.963588458982364e-11
    5.1026834496246985e-12
    4.892006770332984e-11
    4.3791052740842424e-12
    4.3696569982423594e-11
    3.687749140374533e-12
    2.7059581307036152e-11
    3.0836031799177726e-12
    2.3667174903978437e-11
    3.125502604288539e-11
    2.4648326640545893e-11
    3.3454520020538838e-12
    4.087167034428082e-12
    2.64424125975944e-12
    2.0271585762612644e-11
    3.3454518405209002e-12
    4.100855831911664e-11
    1.4366104952937528e-12
    2.372461501540543e-11
    1.566853155717451e-12
    2.958457824833164e-11
    4.0402069147706355e-11
    3.00057118329558e-12
    3.191876828842831e-11
    5.102682652628575e-12
    5.5952487517002287e-11
    4.536572084520695e-12
    4.340801162385396e-11
    4.120990791745212e-12
    3.963588465829096e-11
    5.1026824446115375e-12
    4.892006765092677e-11
    4.380793066900709e-12
    3.687749218602451e-12
    2.7059581026472728e-11
    3.083603615432742e-12
    2.364585931839595e-11
    3.134212258994243e-11
    2.4674401878274856e-11
    3.3454530854341473e-12
    4.087166588149772e-12
    2.810808616187811e-11
    2.0271585966743277e-11
    3.3454530806996793e-12
    4.618741123197043e-12
    1.4366080328739116e-12
    2.3724614226496932e-11
    1.5668505878004413e-12
    2.96862076296905e-11
    4.307507360324907e-11
    4.3258280205002876e-11
    4.369656974862779e-11
    4.0402068996663725e-11
    2.999993692346344e-12
    3.15404880532316e-11
    5.102681315126626e-12
    5.595248889781908e-11
    4.5359925277225905e-12
    4.666766119816482e-12
    4.658482679772333e-12
    4.64185929001485e-12
    4.642753421919598e-12
    4.293029152063198e-11
    4.2725386785616955e-11
    4.340801146283726e-11
    4.121443879732847e-12
    3.963588596739429e-11
    5.102682006462565e-12
    4.892006717325455e-11
    4.37910546791484e-12
    4.667244257389828e-12
    4.657802455080003e-12
    4.307507945402485e-11
    3.687748629691656e-12
    2.70710031282743e-11
    3.0836029136754745e-12
    2.3645859278582386e-11
    3.135965747148177e-11
    2.4663875323276704e-11
    3.345455560450038e-12
    4.0871663289694884e-12
    2.81080847491965e-11
    2.035230631735644e-11
    3.3454538621542343e-12
    4.618742091739927e-12
    1.4366098493224444e-12
    2.3763629203784608e-11
    1.5668521283513777e-12
    2.9584587911502203e-11
    4.379104306660965e-12
    2.9986066221914916e-12
    3.154042926122535e-11
    5.101370826026671e-12
    5.5952488865213537e-11
    4.326559083818681e-11
    4.536572436820889e-12
    4.1209902785581836e-12
    3.963588607960092e-11
    5.102682697921569e-12
    4.892005932475144e-11
    4.058799039290878e-11
    3.68774811481449e-12
    2.707100418020183e-11
    3.083602089120272e-12
    2.364585893824083e-11
    3.136113973135944e-11
    2.464832636208813e-11
    3.345452477297256e-12
    4.0871665491234736e-12
    2.8108085341217463e-11
    2.028225655368916e-11
    3.3444404394582702e-12
    4.6187510618080974e-12
    1.436610298425714e-12
    2.3785621675166347e-11
    1.5668525215768277e-12
    2.95845871902737e-11
    4.3696523203949766e-11
    4.379104740657026e-12
    3.0005709973565377e-12
    3.1540487879046425e-11
    5.1055001015416725e-12
    5.595240717104089e-11
    4.340801135056991e-11
    4.6792227943713145e-12
    4.2725177237456915e-11
    4.542260542185505e-12
    4.1209909496105905e-12
    3.963588418371743e-11
    5.102679997539257e-12
    4.892006935296727e-11
    4.0687746723936635e-11
    4.66724444479405e-12
    3.687747017680175e-12
    2.7059581718118117e-11
    3.083601584449822e-12
    2.364585890946777e-11
    3.1342122289891425e-11
    2.4648325324143318e-11
    3.3454527446091626e-12
    4.087166703814929e-12
    2.810808505450761e-11
    2.0282254277383337e-11
    3.345452186935944e-12
    4.618741750749425e-12
    1.4366107424033329e-12
    2.3785621909662405e-11
    1.5668516462832302e-12
    2.95845781526113e-11
    4.06011537905836e-11
    3.0005714043569138e-12
    3.154042818710303e-11
    5.102681578957659e-12
    5.595240652600971e-11
    4.536572617697011e-12
    4.536571853908049e-12
    4.120990951089601e-12
    3.9635884193199626e-11
    5.102681514240571e-12
    4.892005748949621e-11
    4.0402035585267296e-11
    3.687747017680175e-12
    2.7059581718118117e-11
    3.083601584449822e-12
    2.364584587383255e-11
    3.136114016475573e-11
    2.4648325324143318e-11
    3.3454527446091626e-12
    4.087166703814929e-12
    2.810808505450761e-11
    2.0282254277383337e-11
    3.345452186935944e-12
    4.618751443971372e-12
    1.4366107424033329e-12
    2.3785621909662405e-11
    1.5668516462832302e-12
    2.958458580391053e-11
    4.650421372253486e-12
    4.678229618792978e-12
    4.060115365424667e-11
    3.0005717314454994e-12
    3.154042829133742e-11
    5.102683169585194e-12
    5.595248804957563e-11
    4.5365718618053914e-12
    4.272518232339383e-11
    4.2930280783689376e-11
    4.6760858058293e-12
    4.680532748123715e-12
    4.535992143257609e-12
    4.126743414455358e-12
    3.9635884451784654e-11
    5.102682751722933e-12
    4.892006030141193e-11
    4.0402035459342295e-11
    4.3696676345976404e-11
    4.325829363494646e-11
    3.687749115903853e-12
    2.7028182155013944e-11
    3.0836017025824188e-12
    2.3645858722084954e-11
    3.1342122254161e-11
    2.4648326747569448e-11
    3.3454529900879837e-12
    4.087167308823661e-12
    2.810808474785147e-11
    2.0282255095069722e-11
    3.3454530710262565e-12
    4.618751755080801e-12
    1.4366102386263369e-12
    2.3763628929448232e-11
    1.5668510587811593e-12
    2.9644496364271677e-11
    4.047826674364986e-11
    2.9999942159878337e-12
    3.1540428883856136e-11
    5.102680879576466e-12
    5.595248822978836e-11
    4.542261133664809e-12
    4.326555994413536e-11
    4.1262708184631935e-12
    3.957142775449047e-11
    5.102680314062178e-12
    4.892006012725221e-11
    4.379105838830407e-12
    3.687749115903853e-12
    2.708819990082567e-11
    3.0836017025824188e-12
    2.3645858722084954e-11
    3.1360405155361596e-11
    2.4648326747569448e-11
    3.3454524151272236e-12
    4.090750418601991e-12
    2.8096613397987497e-11
    2.0271584939002712e-11
    3.3454530710262565e-12
    4.618751755080801e-12
    1.4366102386263369e-12
    2.378562219005769e-11
    1.5668510587811593e-12
    2.958458707361575e-11
    4.679223203911826e-12
    4.3791062374892274e-12
    2.9986062838867472e-12
    3.1540428883856136e-11
    5.102680879576466e-12
    5.59524068711048e-11
    4.326556065181696e-11
    4.2725172938448046e-11
    4.667244941783181e-12
    4.542260631814667e-12
    4.120991237052916e-12
    3.9635884468554216e-11
    5.102680314062178e-12
    4.8920066534084436e-11
    4.0687747493932416e-11
    4.369656591183605e-11
    3.687749115903853e-12
    2.708819990082567e-11
    3.0836017025824188e-12
    2.3645858722084954e-11
    3.1363628853345676e-11
    2.4648326747569448e-11
    3.3454524151272236e-12
    4.087167308823661e-12
    2.810808474785147e-11
    2.0282255095069722e-11
    3.3454530710262565e-12
    4.618742558741932e-12
    1.4366102386263369e-12
    2.378562219005769e-11
    1.5668510587811593e-12
    2.958458707361575e-11
    4.3791062374892274e-12
    2.9986062838867472e-12
    3.154044785865491e-11
    5.102680879576466e-12
    5.595248822978836e-11
    4.34080119272895e-11
    4.53657234725059e-12
    4.120991237052916e-12
    3.9635884468554216e-11
    5.102680314062178e-12
    4.892007006153661e-11
    4.040206008759133e-11
    3.687749115903853e-12
    2.708819990082567e-11
    3.0836017025824188e-12
    2.3645858722084954e-11
    3.137095949712751e-11
    2.4648326747569448e-11
    3.3444408200712693e-12
    4.087167308823661e-12
    2.810808474785147e-11
    2.0282255095069722e-11
    3.3454530710262565e-12
    4.618742558741932e-12
    1.4366102386263369e-12
    2.3724614253390994e-11
    1.5668510587811593e-12
    2.9686206493317e-11