# 3 Nonlinear Equations and Complementarity Problems

```python

import numpy as np
from numpy import append, array, diagonal, tril, triu
from numpy.linalg import inv
from scipy.linalg import lu
#from scipy.linalg import solve
from pprint import pprint
from numpy import array, zeros, diag, diagflat, dot

from sympy import *
import sympy as sym
init_printing()
```

One of the most basic numerical operations encountered in computational economics
is to find the solution of a system of nonlinear equations. Nonlinear equations generally
arise in one of two forms. In the nonlinear *rootfinding problem*, a function f
mapping $R^n$ to $R^n$ is given and one must compute an n-vector $x$, called a *root* of $f$,
that satisfies

$$f(x) = 0$$

In the nonlinear fixed-point problem, a function $g$ from $R^n$ to $R^n$ is given and one
must compute an n-vector x called a fixed-point of $g$, that satisfies

$$x = g(x)$$

The two forms are equivalent. The rootfinding problem may be recast as a fixed-point
problem by letting $g(x) = x - f(x)$; conversely, the fixed-point problem may be recast
as a rootfinding problem by letting $f(x) = x - g(x)$.

In the related complementarity problem, two n-vectors $a$ and $b$, with $a < b$, and
a function f from $R^n$ to $R^n$ are given, and one must compute an n-vector $x \in [a; b]$,
that satisfies

$$x_i > a_i \rightarrow f_i(x) \forall i = 1,...,n$$

$$x_i < b_i \rightarrow f_i(x) \forall i = 1,...,n$$

The rootfinding problem is a special case of complementarity problem in which $a_i =
-\inf$ and $b_i = +\inf$ for all i. However, the complementarity problem is not simply to
find a root that lies within specified bounds. An element $f_i(x)$ may be nonzero at a
solution of the complementarity problem, provided that $x_i$ equals one of the bounds
$a_i$ or $b_i$.

https://github.com/QuantEcon/QuantEcon.lectures.code

```python

```

```python

```

## 3.1 Bisection Method

The bisection method is perhaps the simplest and most robust method for computing
the root of a continuous real-valued function defined on a bounded interval of the real line.

The bisection method is an iterative procedure. Each iteration begins with an
interval known to contain or to bracket a root of f, meaning the function has diffierent
signs at the interval endpoints. The interval is bisected into two subintervals of equal
length.

The bisection method's greatest strength is its robustness. In contrast to other
rootfinding methods, the bisection method is guaranteed to compute a root to a
prescribed tolerance in a known number of iterations, provided valid data are input.

https://lectures.quantecon.org/py/scipy.html#roots-and-fixed-points

https://en.wikipedia.org/wiki/Bisection_method

```python

```

```python
#%pylab inline
%matplotlib notebook
# pylab Populating the interactive namespace from numpy and matplotlib
# numpy for numerical computation
# matplotlib for ploting
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D     # 3d
```

```python
x = np.linspace(0,3,100)
plt.plot(x,x**3-2)
plt.plot(x, x-x)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x7f3d90f10668>]

```python

```

```python
def bisect(f, a, b, tol=10e-5):
    """
    Implements the bisection root finding algorithm, assuming that f is a
    real-valued function on [a, b] satisfying f(a) < 0 < f(b).
    """
    lower, upper = a, b

    while upper - lower > tol:
        middle = 0.5 * (upper + lower)
        # === if root is between lower and middle === #
        if f(middle) > 0:
            lower, upper = lower, middle
        # === if root is between middle and upper  === #
        else:
            lower, upper = middle, upper

    return 0.5 * (upper + lower)
```

In fact SciPy provides it’s own bisection function,

```python
from scipy.optimize import bisect
f =lambda x: x**3-2
#f = lambda x: np.sin(4 * (x - 0.25)) + x + x**20 - 1
bisect(f, 0, 2)
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## 3.2 Function Iteration
Function iteration is a relatively simple technique that may be used to compute a
fixed-point, $x = g(x)$, of a function from $R^n$ to $R^n$. The technique is also applicable
to a rootfinding problem $f(x) = 0$, by recasting it as the equivalent fixed-point
problem $x = x - f(x)$.

https://en.wikipedia.org/wiki/Fixed-point_iteration

```python
def myfixpoint(f, x0, maxit = 1000,tol=10e-5):
    #
    x = x0
    for it in range(maxit):
        xold = x
        x = f(x)
        diff = np.linalg.norm(x - xold)

        if diff < tol:

            return x

```

```python
def g(x):
    return x**0.5
```

```python
g = lambda x: x**0.5
```

```python
myfixpoint(g, x0=0.4)
```

```python

```

SciPy has a function for finding (scalar) fixed points too

```python

```

```python
from scipy.optimize import fixed_point

fixed_point(lambda x: x**0.5, 0.4)  # 0.4 is an initial guess
```

    array(1.0)

```python

```

```python

```

## 3.3 Newton's Method

In practice, most nonlinear rootfinding problems are solved using *Newton's method*
or one of its variants. Newton's method is based on the principle of *successive linearization*. Successive linearization calls for a hard nonlinear problem to be replaced
with a sequence of simpler linear problems whose solutions converge to the solution
of the nonlinear problem. Newton's method is typically formulated as a rootfinding
technique, but may be used to solve a fixed-point problem $x = g(x)$ by recasting it
as the rootfinding problem $f(x) = x - g(x) = 0$.

$$x:f(x)=0\,.$$

The method starts with a function f defined over the real numbers x, the function's derivative f ′, and an initial guess x0 for a root of the function f. If the function satisfies the assumptions made in the derivation of the formula and the initial guess is close, then a better approximation x1 is

$$ x_{1}=x_{0}-{\frac {f(x_{0})}{f'(x_{0})}}\,.$$

Geometrically, $(x1, 0)$ is the intersection of the x-axis and the tangent of the graph of f at $(x0, f (x0))$.

The process is repeated as

$$ x_{n+1}=x_{n}-{\frac {f(x_{n})}{f'(x_{n})}}\,$$
until a sufficiently accurate value is reached.

#### Derivation

Suppose $f : [a, b] → ℝ$ is a differentiable function defined on the interval $[a, b]$ with values in the real numbers $ℝ$. The formula for converging on the root can be easily derived. Suppose we have some current approximation xn. Then we can derive the formula for a better approximation, $x_{n + 1}$ by referring to the diagram on the right. The equation of the tangent line to the curve $y = f (x)$ at the point $x = x_n$ is

$$ y=f'(x_{n})\,(x-x_{n})+f(x_{n}),$$

where $f′$ denotes the derivative of the function $f$.

The x-intercept of this line (the value of $x$ such that $y = 0$) is then used as the next approximation to the root, $x_{n+1}$. In other words, setting $y$ to zero and $x$ to $x_{n+1}$ gives

$$ 0=f'(x_{n})\,(x_{n+1}-x_{n})+f(x_{n}).$$
Solving for $x_{n+1}$ gives

$$ {\displaystyle x_{n+1}=x_{n}-{\frac {f(x_{n})}{f'(x_{n})}}.}$$

```python

```

https://en.wikipedia.org/wiki/Newton%27s_method

```python

```

```python

```

In SciPy this algorithm is implemented by scipy.optimize.newton

Unlike bisection, the Newton-Raphson method uses local slope information

This is a double-edged sword:

- When the function is well-behaved, the Newton-Raphson method is faster than bisection
- When the function is less well-behaved, the Newton-Raphson might fail

```python
from scipy.optimize import newton

f = lambda x: x**3 -2

newton(f, 0.2)   # Start the search at initial condition x = 0.2
```

```python
newton?
```

## Newton

Signature: newton(func, x0, fprime=None, args=(), tol=1.48e-08, maxiter=50, fprime2=None)
Docstring:
Find a zero using the Newton-Raphson or secant method.

Find a zero of the function `func` given a nearby starting point `x0`.
The Newton-Raphson method is used if the derivative `fprime` of `func`
is provided, otherwise the secant method is used.  If the second order
derivate `fprime2` of `func` is provided, parabolic Halley's method
is used.

### Parameters

func : function

    The function whose zero is wanted. It must be a function of a
    **single variable
    ** of the form f(x,a,b,c...), where a,b,c... are extra
    arguments that can be passed in the `args` parameter.

x0 : float

    An initial estimate of the zero that should be somewhere near the
    actual zero.

fprime : function, optional

    The derivative of the function when available and convenient. If it
    is None (default), then the secant method is used.

Since `newton` only take one variable function. We use `fsolve` in scipy.optimize

### Example: Cournot Duopoly

To illustrate the use of this function, consider a simple Cournot duopoly model, in which the inverse demand for a good is

```python
q = np.array([1, 1])
c = np.array([0.6, 0.8])
eta = 1.6
e = -1/eta

```

```python
fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+(e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
q = q - np.linalg.inv(fjac).dot(fval)
q
```

    array([ 0.68765683,  0.55048639])

```python
fval
```

    array([-0.1542114, -0.3542114])

```python
fjac
```

    array([[-0.67598669,  0.12664449],
           [ 0.12664449, -0.87598669]])

### Newton method

```python
maxit = 1000
tol = 1/10000
# start from [0.2 0.2] will break
#q = np.array([1, 1])
#c = np.array([0.6, 0.8])
#eta = 1.6
#e = -1/eta

for it in np.arange(maxit):

    fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
    fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+(e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
    q = q - np.linalg.inv(fjac).dot(fval)
    if np.linalg.norm(fval)<tol:
        break

q
```

    array([ 0.83961268,  0.68883816])

```python
fval
```

    array([  5.34511334e-05,   6.51508959e-05])

```python
def cournot(q):
    c = np.array([0.6, 0.8])
    eta = 1.6
    e = -1 / eta
    fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
    fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+\
            (e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
    return fval, fjac
```

```python
def mynewton(f, x0, maxit=1000, tol=1/1000 ):
    x = x0
    for it in np.arange(maxit):
        fval, fjac = f(x)
        x = x - np.linalg.inv(fjac).dot(fval)
        if np.linalg.norm(fval)<tol:
            break
    return x
```

```python
mynewton(cournot, x0 = np.array([1, 1])) # converge
```

    array([ 0.84001971,  0.689215  ])

#### without backstepping, it does not converge with x0= np.array([0.2, 0.2]). It is overstepping too far.

```python
mynewton(cournot, x0 = np.array([0.2, 0.2])) # break ? without
```

    array([ nan,  nan])

More robust implementation can be found in
https://github.com/randall-romero/CompEcon-python/blob/master/compecon/nonlinear.py

### fsolve function

Since scipy.optimize.newton only take one variable function. We use fsolve in scipy.optimize

```python
from scipy.optimize import fsolve
```

```python

```

```python

def cournot(q):
    c = np.array([0.6, 0.8])
    eta = 1.6
    e = -1/eta
    fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
    fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+(e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
    return fval

def cournotjac(q):
    c = np.array([0.6, 0.8])
    eta = 1.6
    e = -1/eta
    fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
    fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+(e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
    return fjac

```

```python

```

```python
q = fsolve(func = cournot, x0= np.array([0.2, 0.2]))  # without jacobian
```

```python
q
```

    array([ 0.8395676 ,  0.68879643])

```python
q = fsolve(func = cournot,x0= np.array([0.2, 0.2]), fprime= cournotjac)  # with jacobian
```

```python
q
```

    array([ 0.8395676 ,  0.68879643])

## 3.4 Quasi-Newton Methods

Quasi-Newton methods offer an alternative to Newton's method for solving rootfinding
problems. Quasi-Newton methods are based on the same successive linearization
principle as Newton's method, except that they replace the Jacobian $f'$ with an
estimate that is easier to compute. Quasi-Newton methods are easier to implement
and less likely to fail due to programming errors than Newton's method because the
analyst need not explicitly code the derivative expressions. Quasi-Newton methods,
however, often converge more slowly than Newton's method and additionally require
the analyst to supply an initial estimate of the function's Jacobian.

The **secant method** is the most widely used univariate quasi-Newton method. The
secant method is identical to the univariate Newton method, except that it replaces
the derivative of f with a finite-difference approximation constructed from the function
values at the two previous iterates:

https://en.wikipedia.org/wiki/Secant_method

Newton's method to find zeroes of a function ${\displaystyle g}$ of multiple variables is given by: $ x_{n+1}=x_{n}-[J_{g}(x_{n})]^{-1}g(x_{n})\,\!$ where ${\displaystyle [J_{g}(x_{n})]^{-1}}$ is the left inverse of the Jacobian matrix $ J_{g}(x_{n})$ of ${\displaystyle g}$ evaluated for ${\displaystyle x_{n}}$.

Strictly speaking, any method that replaces the exact Jacobian ${\displaystyle J_{g}(x_{n})}$ with an approximation is a **quasi-Newton method.**

$$f'(x_{n-1}) ~= {\frac {f(x_{n-1})-f(x_{n-2})} {x_{n-1}-x_{n-2}}   }$$

$${\displaystyle x_{n}=x_{n-1}-f(x_{n-1}){\frac {x_{n-1}-x_{n-2}}{f(x_{n-1})-f(x_{n-2})}}={\frac {x_{n-2}f(x_{n-1})-x_{n-1}f(x_{n-2})}{f(x_{n-1})-f(x_{n-2})}}}  $$

#### *Broyden's method*

is the most popular multivariate generalization of the univariate
secant method. Broyden's method generates a sequence of vectors $x^{(k)}$ and matrices
$A^{(k)}$ that approximate the root of $f$ and the Jacobian $f'$ at the root, respectively.

Broyden's method begins with the analyst supplying a guess $x^{(0)}$ for the root of the
function and a guess $A^{(0)}$ for the Jacobian of the function at the root. Often, $A^{(0)}$
is set equal to the numerical Jacobian of f at $x^{(0)}$.

Alternatively, some analysts use
a rescaled identity matrix for $A^{(0)}$, though this typically will require more iterations
to obtain a solution than if a numerical Jacobian is computed at the outset. Given
$x^{(k)}$ and $A^{(k)}$, one updates the root approximation by solving the linear rootfinding
problem obtained by replacing f with its first-order Taylor approximation about $x^{(k)}$:

https://en.wikipedia.org/wiki/Broyden%27s_method

To solve a system of $k$ nonlinear equations

$${\displaystyle \mathbf {f} (\mathbf {x} )=\mathbf {0} ,}$$

where $f$ is a vector-valued function of vector $x$:

$${\displaystyle \mathbf {x} =(x_{1},x_{2},x_{3},\dotsc ,x_{k})}$$

$${\displaystyle \mathbf {f} (\mathbf {x} )=(f_{1}(x_{1},x_{2},\dotsc ,x_{k}),f_{2}(x_{1},x_{2},\dotsc ,x_{k}),\dotsc ,f_{k}(x_{1},x_{2},\dotsc ,x_{k}))} $$

For such problems, Broyden gives a generalization of the one-dimensional Newton's method, replacing the derivative with the Jacobian J. The Jacobian matrix is determined iteratively based on the secant equation in the finite difference approximation:

$${\displaystyle \mathbf {J} _{n}(\mathbf {x} _{n}-\mathbf {x} _{n-1})\simeq \mathbf {f} (\mathbf {x} _{n})-\mathbf {f} (\mathbf {x} _{n-1}),} $$

where n is the iteration index. For clarity, let us define:

$${\displaystyle \mathbf {f} _{n}=\mathbf {f} (\mathbf {x} _{n}),} $$
$${\displaystyle \Delta \mathbf {x} _{n}=\mathbf {x} _{n}-\mathbf {x} _{n-1},} $$
$${\displaystyle \Delta \mathbf {f} _{n}=\mathbf {f} _{n}-\mathbf {f} _{n-1},}$$

so the above may be rewritten as:

$${\displaystyle \mathbf {J} _{n}\Delta \mathbf {x} _{n}\simeq \Delta \mathbf {f} _{n}.} $$

The above equation is underdetermined when $k$ is greater than one.

Broyden suggests using the current estimate of the Jacobian matrix $\mathbf {J} _{n-1}$ and improving upon it by taking the solution to the secant equation that is a minimal modification to $\mathbf {J} _{n-1}$:

$${\displaystyle \mathbf {J} _{n}=\mathbf {J} _{n-1}+{\frac {\Delta \mathbf {f} _{n}-\mathbf {J} _{n-1}\Delta \mathbf {x} _{n}}{\|\Delta \mathbf {x} _{n}\|^{2}}}\Delta \mathbf {x} _{n}^{\mathrm {T} }} $$

This minimizes the following Frobenius norm:

$${\displaystyle \|\mathbf {J} _{n}-\mathbf {J} _{n-1}\|_{\mathrm {f} }.} $$
We may then proceed in the Newton direction:

$${\displaystyle \mathbf {x} _{n+1}=\mathbf {x} _{n}-\mathbf {J} _{n}^{-1}\mathbf {f} (\mathbf {x} _{n}).}$$

Broyden also suggested using the Sherman-Morrison formula to update directly the inverse of the Jacobian matrix:

$${\displaystyle \mathbf {J} _{n}^{-1}=\mathbf {J} _{n-1}^{-1}+{\frac {\Delta \mathbf {x} _{n}-\mathbf {J} _{n-1}^{-1}\Delta \mathbf {f} _{n}}{\Delta \mathbf {x} _{n}^{\mathrm {T} }\mathbf {J} _{n-1}^{-1}\Delta \mathbf {f} _{n}}}\Delta \mathbf {x} _{n}^{\mathrm {T} }\mathbf {J} _{n-1}^{-1}} $$

This first method is commonly known as the **"good Broyden's method".**

In theory, Broyden's method converges if $f$ is continuously differentiable, if $x_0$
is "sufficiently" close to a root of $f$ at which $f_0$ is invertible, and if $J_0^{-1}$ are
"sufficiently" close to the Jacobian or inverse Jacobian of $f$ at that root.

Like Newton's method, the robustness of Broyden's method depends on the regularity of
f and its derivatives. Broyden's method may also have diÆculty computing a precise
root estimate if f0 is ill-conditioned near the root.

```python
#https://github.com/randall-romero/CompEcon-python/blob/master/compecon/tools.py
# The script also computes an initial guess for the inverse Jacobian by inverting the finite difierence derivative computed
# using the toolbox function fdjac, which is discussed in Chapter 5 (page 107).
def jacobian(func, x, *args, **kwargs):

    # if type(func(x, *args, **kwargs)) is tuple:
    #     F = lambda x: func(x, *args, **kwargs)[0]
    # else:
    #     F = lambda x: func(x, *args, **kwargs)
    F = lambda z: func(z, *args, **kwargs)

    x = x.flatten()
    dx = x.size
    f = F(x)
    df = f.size
    x = x.astype(float)

    ''' Compute Jacobian'''
    tol = np.spacing(1) ** (1/3)

    h = tol * np.maximum(abs(x), 1)
    x_minus_h = x - h
    x_plus_h = x + h
    deltaX = x_plus_h - x_minus_h
    fx = np.zeros((dx, df))

    for k in range(dx):
        xx = x.copy()
        xx[k] = x_plus_h[k]
        fplus = F(xx)

        xx[k] = x_minus_h[k]
        fminus = F(xx)

        fx[k] = np.squeeze((fplus - fminus) / deltaX[k])  # fixme doing this to deal with broadcasting

    return fx.T

```

```python
def cournot(q):
    c = np.array([0.6, 0.8])
    eta = 1.6
    e = -1 / eta
    fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
    fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+\
            (e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
    return fval
```

```python
f = cournot
x = np.array([0.2, 0.2])
```

```python
fval = f(x)
fval
```

    array([ 1.09894808,  1.05894808])

```python
# using function to calculate Jacobian
fjac = jacobian(cournot,np.array([0.2, 0.2]) )
fjac
```

    array([[-3.88977465, -0.5194381 ],
           [-0.5194381 , -4.08977465]])

```python
# using numpy to calculate inversed Jacobian
fjacinv = np.linalg.pinv(np.atleast_2d(fjac))
fjacinv
```

    array([[-0.26151986,  0.03321537],
           [ 0.03321537, -0.2487309 ]])

```python
maxit = 100
tol = 1/100

for it in range(maxit):
    fnorm = np.linalg.norm(fval)
    if fnorm<tol:
        break
    d = -(fjacinv.dot(fval))
    x = x+d
    fold = fval
    fval = f(x)
    u = fjacinv.dot((fval-fold))
    fjacinv = fjacinv + np.outer((d-u), np.dot(d.T, fjacinv))/np.dot(d.T,u) ## ? np.outer!!!!! Key
x
```

    array([ 0.83480764,  0.68819305])

In linear algebra, an outer product is the tensor product of two coordinate vectors, a special case of the Kronecker product of matrices. The outer product of two coordinate vectors ${\displaystyle \mathbf {u} } $  and ${\displaystyle \mathbf {v} } $ , denoted ${\displaystyle \mathbf {u} \otimes \mathbf {v} } $, is a matrix ${\displaystyle \mathbf {w} } $  such that the coordinates satisfy ${\displaystyle w_{ij}=u_{i}v_{j}} $. The outer product for general tensors is also called the tensor product.

The outer product contrasts with the dot product, which takes as input a pair of coordinate vectors and produces a scalar.

(source: https://en.wikipedia.org/wiki/Outer_product)

```python

```

```python

```

Without analytic Jacobian, we use Broyden method to find the root.

```python

```

```python
def mybroyden(f, x0, maxit = 1000, tol = 1/1000):

    x=x0
    A = f(x)
    _is_there_jacobian = (type(A) is tuple) and (len(A) == 2)

    if _is_there_jacobian:
        print('Jacobian was  provided by user!')
        fval,fjac = f(x)
    else:
        print('Jacobian was not provided by user!')
        fval = f(x)
        try:
            fjac = jacobian(f,x)
        except NameError:
            print("jacobian function Not in scope!\n Using identity matrix as jacobian matrix")
            fjac = np.identity(x.size)
        else:
            print("jacobian function In scope!")
            #fjac = jacobian(f,x)
    # using numpy to calculate inversed Jacobian, only for initialize. later, we update it without calculation of new inversed Jacobian.
    fjacinv = np.linalg.pinv(np.atleast_2d(fjac))
    # or we can use identity matrix as an initiative matrix
    #fjacinv = - np.identity(x.size)
    for it in range(maxit):
        fnorm = np.linalg.norm(fval)
        if fnorm<tol:
            break
        d = -(fjacinv.dot(fval))
        # update x
        x = x+d
        fold = fval
        fval = f(x)[0] if _is_there_jacobian else f(x)# two outputs
        # update jacobian
        u = fjacinv.dot((fval-fold))
        fjacinv = fjacinv + np.outer((d-u), np.dot(d.T, fjacinv))/np.dot(d.T,u) ## ? np.outer !!! key
    return x

```

```python
mybroyden(cournot, np.array([0.2, 0.2]))
```

    Jacobian was not provided by user!
    jacobian function In scope!

    array([ 0.83945538,  0.6889152 ])

```python

```

### Scipy.optimize.broyden1

https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.optimize.broyden1.html

scipy.optimize.broyden1

    scipy.optimize.broyden1(F, xin, iter=None, alpha=None, reduction_method='restart', max_rank=None, verbose=False, maxiter=None, f_tol=None, f_rtol=None, x_tol=None, x_rtol=None, tol_norm=None, line_search='armijo', callback=None, **kw)

Find a root of a function, using Broyden’s first Jacobian approximation.

This method is also known as “Broyden’s good method”.

This algorithm implements the inverse Jacobian Quasi-Newton update

which corresponds to Broyden’s first Jacobian update

```python
from scipy.optimize import broyden1
broyden1(cournot,np.array([0.2, 0.2]) )
```

    array([ 0.8395676 ,  0.68879643])

## 3.5 Problems With Newton Methods

Several difficulties commonly arise in the application of Newton and quasi-Newton
methods to solving multivariate non-linear equations. The most common cause of
failure of Newton-type methods is coding errors committed by the analyst. The next
most common cause of failure is the specification of a starting point that is not sufficiently
close to a root. And yet another common cause of failure is an ill-conditioned
Jacobian at the root. These problems can often be mitigated by appropriate action,
though they cannot always be eliminated altogether.

The *first cause of failure*, coding error, may seem obvious and not specific to
rootfinding problems. It must be emphasized, however, that with Newton's method,
the likelihood of committing an error in coding the analytic Jacobian of the function
is often high. A careful analyst can avoid Jacobian coding errors in two ways. First,
the analyst could use Broyden's method instead of Newton's method to solve the
rootfinding problem. Broyden's method is derivative-free and does not require the
explicit coding of the function's analytic Jacobian. Second, the analyst can perform
a simple, but highly effiective check of his code by comparing the values computed
by his analytic derivatives to those computed using finite diffierence methods. Such a
check will almost always detect an error in either the code that returns the function's
value or the code that returns its Jacobian.

The *second problem, a poor starting value*, can be partially addressed by **"backstepping"**.
If taking a full Newton (or quasi-Newton) step $x+d$ does not offer an improvement
over the current iterate $x$, then one "backsteps" toward the current iterate
$x$ by repeatedly cutting $d$ in half until $x+d$ does offer an improvement. Whether a step
$d$ offers an improvement is measured by the Euclidean norm $||f(x)|| = \frac{1}{2} f(x)^{T}f(x)$.
Clearly, $||f(x)||$ is precisely zero at a root of f, and is positive elsewhere. Thus,
one may view an iterate as yielding an improvement over the previous iterate if it
reduces the function norm, that is, if $||f(x)|| > ||f(x+ d)||$. Backstepping prevents
Newton and quasi-Newton methods from taking a large step in the wrong direction,
substantially improving their robustness.

necessarily prevent Newton type methods
from getting stuck at a local minimum of $||f(x)||$. If $||f(x)||$ must decrease with
each step, it may be difficult to find a step length that moves away from the current
value of x. Most good root-finding algorithms employ so mechanism for getting
unstuck. We use a very simple one in which the backsteps continue until either
$||f(x)|| > ||f(x+ d)||$ or $ ||f(x+ d/2)|| > ||f(x+ d)||$.

```python
def cournot(q):
    c = np.array([0.6, 0.8])
    eta = 1.6
    e = -1 / eta
    fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
    fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+\
            (e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
    return fval, fjac
```

### mynewton_backstep

```python

```

```python
x0= np.array([0.2, 0.2])
f = cournot
def mynewton_backstep(f, x0, maxit=1000, tol=1/1000, maxsteps = 1000 ):
    x = x0
    for it in np.arange(maxit):
        fval, fjac = f(x)
        fnorm = np.linalg.norm(fval)
        if np.linalg.norm(fval)<tol:
            break
        d =   - np.linalg.inv(fjac).dot(fval)
        fnormold = np.inf # initial fnormold
        for backstep in np.arange(maxsteps):
            fvalnew = f(x+d)[0] # first output/outcome
            fnormnew = np.linalg.norm(fvalnew)
            if fnormnew < fnorm:
                break
            if fnormold < fnormnew:
                d=2*d
                break
            fnormold = fnormnew;
            d = d/2;
        x = x + d

    return x
mynewton_backstep(f,x0)  # even start from [0.2,0.2], still converge with backstepping.
```

    array([ 0.83901993,  0.68828942])

#### even start from [0.2,0.2], still converge with backstepping.

### Note: Backtracking
Newton’s method may not converge for a variety of reasons. One potential problem occurs when the step from $x_k$ to $x_{k+1}$ is so large that the root is stepped over completely. Backtracking is a strategy that combats the problem of overstepping by moving only a fraction of the full step from $x_k$ to $x_{k+1}$.

https://github.com/OpenSourceMacro/BootCamp2017/blob/master/Computation/Wk4_DifIntOpt/ACME_Newtons.pdf

```python

```

```python
scipy.optimize.newton?
```

    Object `scipy.optimize.newton` not found.

```python

```

### mybroyden_backstep

```python
def cournot(q):
    c = np.array([0.6, 0.8])
    eta = 1.6
    e = -1 / eta
    fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
    fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+\
            (e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
    return fval # for Broyden, no need for analytic jacobian

x0= np.array([0.2, 0.2])
f = cournot
```

```python
def mybroyden_backstep(f, x0, maxit = 1000, tol = 1/1000, maxsteps = 100):
    x=x0
    fval = f(x)
    # using function to calculate Jacobian
    fjac = jacobian(f,x)
    # using numpy to calculate inversed Jacobian
    fjacinv = np.linalg.pinv(np.atleast_2d(fjac))
    for it in range(maxit):
        fnorm = np.linalg.norm(fval)
        if fnorm<tol:
            break
        d = -(fjacinv.dot(fval))
        fnormold = np.inf # initial fnormold
        for backstep in np.arange(maxsteps):
            fvalnew = f(x+d) # first output/outcome
            fnormnew = np.linalg.norm(fvalnew)
            if fnormnew < fnorm:
                break
            if fnormold < fnormnew:
                d=2*d
                break
            fnormold = fnormnew;
            d = d/2;
        # update x
        x = x+d
        fold = fval
        fval = f(x)
        # update jacobian
        u = fjacinv.dot((fval-fold))
        fjacinv = fjacinv + ((d-u).dot(d.T)*(fjacinv))/(d.T.dot(u))
    return x

mybroyden_backstep(f,x0)
```

    array([ 0.8395465 ,  0.68872313])

```python

```

```python

```

The **third problem, an ill-conditioned Jacobian at the root**, occurs less often, but
should not be ignored. An ill-conditioned Jacobian can render inaccurately computed
Newton step dx, creating severe difficulties for the convergence of Newton and
Newton-type methods. In some cases, ill-conditioning is a structural feature of the
underlying model and cannot be eliminated. However, in many cases, ill-conditioning
is inadvertently and unnecessarily introduced by the analyst. A common source of
avoidable ill-conditioning arises when the natural units of measurements for model
variables yield values that vary vastly in order of magnitude. When this occurs, the
analyst should consider rescaling the variables so that their values have comparable
orders of magnitude, preferably close to unity. Rescaling will generally lead to faster
execution time and more accurate results.

#### ill-conditioned
https://github.com/OpenSourceMacro/BootCamp2017/blob/master/Computation/Wk4_DifIntOpt/ACME_CondStab.pdf

```python

```

## 3.6 Choosing a Solution Method

Numerical analysts have special terms that they use to classify the rates at which
iterative routines converge. Specifically, a sequence of iterates x(k) is said to converge
to $x^{k}$ at a rate of order p if there is constant $C > 0$ such that

$$ ||x^{(k+1)} - x^*|| <=  C ||x^{(k)} - x^*||^p $$

for sufficiently large k. In particular, the rate of convergence is said to be linear if
$C < 1$ and $p = 1$, superlinear if $1 < p < 2$, and quadratic if $p = 2$.

The asymptotic rates of convergence of the nonlinear equation solution methods
discussed earlier are well known. The bisection method converges at a linear rate
with $C = 1/2$.

```python

```

## 3.7 Complementarity Problems

Many economic models naturally take the form of a complementary problem rather
than a rootfinding or fixed point problem. In the complementarity problem, two n-
vectors a and b, with a < b, and a function f from $R^n$ to $R^n$  are given, and one must
find an n-vector $x \in [a; b]$, that satisfies

$$x_i > a_i \rightarrow f_i(x) \forall i = 1,...,n$$

$$x_i < b_i \rightarrow f_i(x) \forall i = 1,...,n$$

The complementarity conditions require that $f_i(x) = 0$ whenever $a_i < x_i < b_i$. The
complementarity problem thus includes the rootfinding problem as a special case in
which $a_i = - \inf$ and $bi = + \inf$ for all $i$. The complementarity problem, however, is not
to find a root that lies within specified bounds. An element $f_i(x)$ may be nonzero at
a solution of a complementarity problem, though only if $x_i$ equals one of its bounds.
For the sake of brevity, we denote the complementarity problem $CP(f; a; b)$.

Complementarity problems arise naturally in economic equilibrium models. In
this context, $x$ is an n-vector that represents the levels of certain economic activities.
For each $i = 1; 2; : : : ; n$, $a_i$ denotes a lower bound on activity $i$, $b_i$ denotes an upper
bound on activity $i$, and $f_i(x)$ denotes the marginal arbitrage profit associated with
activity $i$. Disequilibrium arbitrage profit opportunities exist if either $x_i < b_i$ and
$f_i(x) > 0$, in which case an incentive exists to increase $x_i$, or $x_i > a_i$ and $f_i(x) < 0$, in
which case an incentive exists to decrease $x_i$. An arbitrage-free economic equilibrium
obtains if and only if $x$ solves the complementarity problem $CP(f; a; b)$.

Complementarity problems also arise naturally in economic optimization models.

## 3.8 Complementarity Methods

Although the complementarity problem appears quite diffierent from the ordinary
rootfinding problem, it actually can be reformulated as one. In particular, x solves the
complementarity problem $CP(f; a; b)$ if and only if it solves the rootfinding problem

$$ \hat{f}(x) =min(max(f(x), a - x), b - x) = 0$$

(source, http://www.karenkopecky.net/Teaching/eco613614/Notes_ComplementarityMethods.pdf)

The equivalence, however, can easily be demonstrated graphically for the univariate complementarity problem.

Figure 3.7 illustrates minmax rootfinding formulation of the same four univariate complementarity problems examined in Figure 3.6. In all four plots, the curves y = a — x and y = b — x are drawn with narrow dashed lines, the curve y = /(x) is drawn with a narrow solid line, and the curve y = f(x) is drawn with a thick solid line; clearly, in all four figures, / lies between the lines y = x — a and y = x — b and coincides with f inside the lines. In Figure 3.7a, $f(a) < 0$ and the unique solution to the complementarity problem is $x^* = a$, which coincides with the unique root of f; in Figure 3.7b,$ f (b) > 0$ and the unique solution to the complementarity problem is $x^* = b$, which coincides with the unique root of f ; in Figure 3.7c, $f (a) > 0 > f (b)$ and the unique solution to the complementarity problem lies between a and b and coincides with the unique root of f (and f). In Figure 3.7d, f is upwardly sloped and possesses multiple roots, all of which, again, coincide with roots of f .

 f'<0, f(a)>0>f(b)

f'<0, f(b)>0

b) f'<0, f(b)>0

d) f'>0

The reformulation of the complementarity problem as a rootfinding problem suggests that it may be solved using standard rootfinding algorithms, such as Newton's method. To implement Newton's method for the minmax rootfinding formulation requires computation of the Jacobian J of f . The ith row of J may be derived directly from the Jacobian J of f:

$$\hat J_i (x) = \begin{cases}
 & J_i (x),  \text{ for } a_i - x_i < f_i(x)<b_i -x_i \\
 & - I_i, \,\;\; \:\:  \text{otherwise}
\end{cases}$$

Here, Ii. is the ith row of the identity matrix.

(source https://www.ajjacobson.us/bellman-equation/complementarity-methods.html

http://www.codecogs.com/latex/eqneditor.php)

```python
# def cournot(q):
#     c = np.array([0.6, 0.8])
#     eta = 1.6
#     e = -1 / eta
#     fval = np.sum(q)**e + e * np.sum(q)**(e-1)*(q) - np.diag(c).dot(q)
#     fjac = e*np.sum(q)**(e-1)*(np.ones([2,2]))+e*np.sum(q)**(e-1)*(np.eye(2))+\
#             (e-1)*e*np.sum(q)**(e-2)*(q).dot(np.array([1, 1]))-np.diag(c)
#     return fval, fjac
# x0= np.array([0.2, 0.2])
# f = cournot
```

```python
# user-supplied
x0, maxit, tol , maxsteps =np.array([2]), 10000,  1/1000, 100 # x0 = 0 converge to 0, x0 = 2 converge to 2.005
```

```python
f =lambda x: (1.01 - (1 - x) ** 2,  2 * (1 - x))
```

```python
#%pylab inline
#%pylab notebook
# pylab Populating the interactive namespace from numpy and matplotlib
# numpy for numerical computation
# matplotlib for ploting
from matplotlib import pyplot as plt
```

```python

```

```python

```

```python

a = 0
b = np.inf
```

```python
x = np.linspace(-0.5,2.5,100)
plt.plot(x,np.fmin(np.fmax(f(x)[0], a-x),b-x))
plt.hlines(y = 0, xmin=-0.5, xmax=2.5, colors='r', linestyles='--')
```

    <matplotlib.collections.LineCollection at 0x7f3d91469a58>

```python
x = x0 # x0 = 0 converge to 0, x0 = 2 converge to 2.005
for it in np.arange(maxit):
    fval,fjac = f(x)
    fhatval = np.fmin(np.fmax(fval, a-x),b-x)
    fhatjac = -np.identity(x.size)
    i = (fval> (a-x)) & (fval<(b-x))
    if np.any(i):
        fhatjac[i] = fjac[i]
    x = x - np.linalg.inv(fhatjac).dot(fhatval)
    if np.linalg.norm(fhatval)<tol:
        break
x
```

    array([ 2.00498756])

```python
def myncpsolve(f, a,b,x0,maxit = maxit, tol = tol):
    x = x0
    for it in np.arange(maxit):
        fval,fjac = f(x)
        fhatval = np.fmin(np.fmax(fval, a-x),b-x)
        fhatjac = -np.identity(x.size)
        i = (fval> (a-x)) & (fval<(b-x))
        if np.any(i):
            fhatjac[i] = fjac[i]
        x = x - np.linalg.inv(fhatjac).dot(fhatval)
        if np.linalg.norm(fhatval)<tol:
            break
    return x
```

```python
x0=np.array([1.40])
myncpsolve(f, a,b,x0)
```

    array([ 2.00498756])

```python
x0=np.array([2.5])
myncpsolve(f, a,b,x0)
```

    array([ 2.00498756])

```python
x0=np.array([15])
myncpsolve(f, a,b,x0)
# overshooting
```

    array([-0.005])

Using Newton's method to find a root of f will often work well. However, in many cases, the **nondifferentiable kinks** in f create difficulties for Newton's method, undermining its ability to converge rapidly and possibly even causing it to cycle. One way to deal with the kinks is to replace f with a function that has the same roots, but is smoother and therefore less prone to numerical difficulties. One function that has proven very effective for solving the complementarity problem in practical applications is Fischer's function

$$\tilde{f}(x)  =  \phi^- (\phi^+(f(x),a - x),b - x)$$

, where $$\phi^{\pm}(u,v) = u_i + v_i \pm \sqrt{u_i^2 + v_i^2}$$

In Figures 3.8a and 3.8b, the functions $\hat{f}$ and $\tilde{f}$ , respectively, are drawn as thick solid lines for a representative complementarity problem. Clearly, $\hat{f}$ and $\tilde{f}$ can differ substantially. What is important for solving the complementarity problem, however, is that $\hat{f}$ and $\tilde{f}$ possess the same signs and roots and that  $\tilde{f}$ is smoother than $\hat{f}$.

a) Minimax Formulation

b) Semismooth Formulation

(source https://www.ajjacobson.us/bellman-equation/complementarity-methods.html)

```python

```

```python

```

Figure 3.9a displays $\hat f$ (dashed) and $\tilde f$ (solid) for the complementarity problem and Figure 3.9b magnifies the plot near the origin, making it clear why the problem is hard. Newton's method starting at any value slightly less than 1 will tend to move toward 0. In order to avoid convergence to this false root, Newton's method must take a sufficiently large step to exit the **region of attraction**. This will not happen with $\hat f$ because 0 poses an upper bound on the positive Newton step. With $\tilde f$ , however, the function is smooth at its local maximum near the origin, meaning that the Newton step can be very large.

(source https://www.ajjacobson.us/bellman-equation/complementarity-methods.html)

```python
#https://github.com/randall-romero/CompEcon-python/blob/master/compecon/nonlinear.py

def fischer(u, v, plus=True):
    """ Computes Fischer's function
    phi^±(u, v) = u + v ± sqrt(u^2 + v^2)
    In turn, it is assumed that u and v are functions of x. If the Jacobian of Fischer's
    function wrt x is required, then partial derivatives du and dv are required.
    Fischer's function is useful to transform a complementarity problem into
    a nonlinear root-finding problem.
    Args:
        u:    first term
        v:    second term
        plus: if True (default), compute  u + v + sqrt(u^2+v^2), else u + v - sqrt(u^2+v^2)
    Returns:
        phi:                  if either du or dv is None

    References:
        Miranda and Fackler 2002 Applied Computational Economics and Finance, pp. 49-50
    """
    s = 1 if plus else -1
    sq = np.sqrt(u * u + v * v)
    ftildeval = u + v + s * sq
    return ftildeval

```

```python
f = lambda x: (1.01 - (1 - x) ** 2, 2 * (1 - x))
a = 0
b = np.inf
x0=np.array([2.5])
x = x0

```

```python

for it in np.arange(maxit):
    fval,fjac = f(x)

    ftildeval = fval
    da = a-x
    db = b-x
    hasLowerBound = np.isfinite(a)
    hasUpperBound = np.isfinite(b)
    if np.any(hasLowerBound):  # apply the Fischer + transform
        ftildeval= fischer(ftildeval, da)
    if np.any(hasUpperBound):  # apply the Fischer - transform
        ftildeval = fischer(ftildeval, db, plus=False)

    ftildejac = -np.identity(x.size)
    i = (fval> (a-x)) & (fval<(b-x))
    if np.any(i):
        ftildejac[i] = fjac[i]
    x = x - np.linalg.inv(ftildejac).dot(ftildeval)
    if np.linalg.norm(ftildeval)<tol:
        break
```

```python
x
```

    array([ 2.00498756])

#### Performance does not improve than CP ？？

```python
def myssmooth(f, a,b,x0,maxit = maxit, tol = tol):
    """f gives two outputs, value; jacobian"""
    x = x0
    for it in np.arange(maxit):
        fval,fjac = f(x)

        ftildeval = fval
        da = a-x
        db = b-x
        hasLowerBound = np.isfinite(a)
        hasUpperBound = np.isfinite(b) # only apply Fisher if there is a finite bound  ??
        if np.any(hasLowerBound):  # apply the Fischer + transform
            ftildeval= fischer(ftildeval, da)
        if np.any(hasUpperBound):  # apply the Fischer - transform
            ftildeval = fischer(ftildeval, db, plus=False)

        ftildejac = -np.identity(x.size)
        i = (fval> (a-x)) & (fval<(b-x))
        if np.any(i):
            ftildejac[i] = fjac[i]
        x = x - np.linalg.inv(ftildejac).dot(ftildeval)
        if np.linalg.norm(ftildeval)<tol:
            break
    return x

# source
#https://github.com/randall-romero/CompEcon-python/blob/master/compecon/nonlinear.py

```

```python
f = lambda x: (1.01 - (1 - x) ** 2, 2 * (1 - x))
a = 0
b = 10
tol = 1/10000
maxit = 10000
```

```python
x0=np.array([1.1]) ##??? it does not improve from fhat
myssmooth(f, a,b,x0)
```

    array([ 0.00448019])

```python
x0=np.array([2.5])
myssmooth(f, a,b,x0)
```

    array([ 2.00498756])

```python
x0=np.array([15])
myssmooth(f, a,b,x0)
```

    array([ 0.00448019])

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

### Reference

scipy.optimize does not have ncpsolve, but dolo python package provides a ncpsolve.

import scipy.optimize as opt

https://github.com/EconForge/econforge/wiki/CompEcon

https://github.com/EconForge/dolo/blob/master/trash/dolo/numeric/solver.py

https://github.com/EconForge/dolo/blob/master/dolo/numeric/optimize/ncpsolve.py

http://nbviewer.jupyter.org/github/sbustamante/ComputationalMethods/blob/master/material/one-variable-equations.ipynb

 - Bisection Method

 - Fixed-point Iteration

 - Newton-Raphson Method

 - Secant Method

https://github.com/dingliumath/economics/blob/master/3.2%20-%20Finding%20Roots%2C%20Newton's%20Method.ipynb

https://github.com/dingliumath/economics/blob/master/3.3%20-%20Ramsey%20via%20Line%20Search%2C%20Shooting%20Method.ipynb

https://github.com/dingliumath/economics/blob/master/3.2%20-%20Finding%20Roots%2C%20Built-in%20Methods.ipynb

https://github.com/EconForge/econforge/wiki/CompEcon

```python

```

```python

```


# Aufgabe

Eine Wasserleitung CU 35$\times$1.2 soll einen Volumenstrom von $\dot V = 4.0\,\frac{\mathrm{m}^3}{\mathrm{h}}$ transportieren.

Zulässig ist eine Geschwindigkeit von $v = 1.0\,\frac{\mathrm{m}}{\mathrm{s}}$, bei höheren Geschwindigkeiten können Strömungsgeräusche auftreten.

1. Berechnen Sie die Geschwindigkeit, die sich in dem Rohr einstellt.
2. Welcher Durchmesser ist mindestens zu wählen, um die zugelassene Geschwindigkeit nicht zu überschreiten.

Rechnen Sie schrittweise und kommentieren Sie Ihre Lösung ausführlich.

Die Zusammenhänge zwischen Volumenstrom, Durchmesser und Geschwindigkeit sind durch die folgenden Formeln gegeben:

\begin{align}
  \dot V &= A\,v,
  &
  A &= \dfrac{\pi\,d^2}{4}
  \\
  v &= \dfrac{\dot V}{A}
  \\
  A &= \dfrac{\dot V}{v},
  &
  d &= \sqrt{\dfrac{4\,A}{\pi}}
\end{align}

Das Modul `math` sollten Sie importieren. Das ist in der folgenden Zelle voreingestellt.

Sie finden die Zahl $\pi$ als `math.pi`.

Die Wurzel einer Zahl `x` können Sie als `math.sqrt(x)` berechnen.

```python
import math
```

1. Berechnen Sie die Geschwindigkeit, die sich in dem Rohr einstellt.

```python
# Ihre Lösung beginnt hier
d = (35-2*1.2)*1e-3 # lichter Durchmesser in m

A = math.pi*d**2/4
A # lichter Querschnitt in m**2
```

    0.0008346897521322724

```python
dV_h = 4 # Volumenstrom in m**3/h

v = dV_h/(3600*A)
v # Geschwindigkeit in m/s
```

    1.3311665900685872

Diese Geschwindigkeit ist zu hoch! Es muss ein größerer Querschnitt gewählt werden.

2. Welcher Durchmesser ist mindestens zu wählen, um die zugelassene Geschwindigkeit nicht zu überschreiten.

```python
v = 1 # Geschwindigkeit in m/s

A = dV_h/(3600*v) # Querschnitt in m**2

d = math.sqrt(4*A/math.pi)*1000
d # Durchmesser in mm
```

    37.61263890318375

Es wird mindestens ein Durchmesser von $37.6\,mm$ benötigt. Laut Tabellenbuch muss ein Querschnitt CU 42$\times$1.5 gewählt werden. Die Geschwindigkeit ist dann:

```python
d = (42-2*1.5)*1e-3

A = math.pi*d**2/4
A # lichter Querschnitt in m**2
```

    0.001194590606527519

```python
dV_h = 4 # Volumenstrom in m**3/h

v = dV_h/(3600*A)
v # Geschwindigkeit in m/s
```

    0.9301187411316844

Die Geschwindigkeit ist nun mit $v = 0.93\,\frac{m}{s}$ zulässig.


# Pulse stream recovery under additive noise

In the previous notebooks, we considered the pulse stream recovery problem under no noise. Here we investigate the effect of **_additive noise_** on our recovery process. Moreover, we consider:

\begin{align}
y_{meas}[n] = y_{BL}[n] + w[n],   \nonumber
\end{align}

where $w[n]$ can be modelled as additive white Gaussian noise with variance $\sigma^2$.

```python
import numpy as np
import plot_settings
import sys
sys.path.append('..')

from frius import create_pulse_param, sample_ideal_project, estimate_fourier_coeff, compute_ann_filt, estimate_time_param, estimate_amplitudes, evaluate_recovered_param
from frius import add_noise  # 'us_utils.py'
```

# 1. Oversampling

Our first "mechanism" to deal with noise is to **_oversample_**. This is not too different from the classical bandlimited situation in which a particular choice for the bandwidth may be too "restrictive" as an approximation of our original signal. Therefore, in order to reduce this _approximation error_, i.e. represent our original signal more faithfully as a bandlimited signal, we may need to:
* Increase the bandwidth of our anti-aliasing filter / generation function.
* Increase the sampling rate (i.e. take more samples per unit of time).

These two steps are precisely what will be done for the pulse stream scenario _prior_ to applying a denoising algorithm.

# 2. Exploiting redundancy in extra measurements

Assuming that we are correct in our choice of the number of pulses $K$, oversampling, i.e. taking $N\geq 2K+1$ samples, would yield more measurements than degrees of freedom in the noiseless case: $\rho = 2K$. Under noise, more than $2K$ degrees of freedom are needed to represent our signal. Therefore, it is natural that we need more samples.

For the pulse stream recovery problem, we will define an oversampling factor $\beta > 1$ so that an _integer_ number of samples $N$ is given by:

\begin{align}
N = 2\beta K +1.  \nonumber
\end{align}

As a reminder, we need an odd number of measurements ($2K+1$ for critical sampling) in order to use real-valued sampling kernels.

We inevitably have to bring in some more math now to see how we can exploit the redundancy in our extra measurements. In the last notebook, in particular in the Appendix, we saw in more detail what the annihilation filter entailed, namely the discrete convolution between the (equalized) Fourier coefficients and this filter is equal to zero for all indices. We can write this annihilation property (or constraint within the context of denoising) as a matrix-vector product:

\begin{align}
\begin{bmatrix}
\hat{x}[\mathrel{{-M}{+}{K}}]&\hat{x}[\mathrel{{-M}{+}{K}{-}{1}}] & \cdots & \hat{x}[-M]\\
\vdots&\ddots &  \ddots & \vdots\\
\hat{x}[1] & \hat{x}[0] &   \cdots & \hat{x}[\mathrel{{-K}{+}{1}}] \\
\hat{x}[2] & \hat{x}[1] &  \cdots & \hat{x}[\mathrel{{-K}{+}{2}}] \\
\vdots&\vdots& \ddots & \vdots\\
\hat{x}[M] & \hat{x}[\mathrel{{M}{-}{1}}]  & \cdots & \hat{x}[\mathrel{{M}{-}{K}}]
\end{bmatrix}
\underbrace{\begin{bmatrix} a[0] \\ a[1] \\ a[2] \\ \vdots\\ a[K]\end{bmatrix}}_{\mathbf{a}}
&=
\mathbf{0}. \nonumber
\end{align}

One thing to notice about the matrix of Fourier coefficients ($\hat{x}[n]$) above is its [**_Toeplitz_**](https://en.wikipedia.org/wiki/Toeplitz_matrix) structure, i.e. descending diagonals from left to right are constant. We can write the above annihilation constraint more concisely as:

\begin{align}
\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) \hspace{0.05cm}\mathbf{a} = \mathbf{0},   \nonumber
\end{align}

where the operator $ \mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) $ forms the Toeplitz matrix from $ \mathbf{\hat{x}} = \{\hat{x}[m]\}_{m=-M}^{M} $ with $ (K+1 )$ columns.

If there are truly $K$ pulses, another (less intuitive) observation we can make about the matrix $ \mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) $ is that it has a rank of at most $K$. In the [Appendix](#app) of this notebook, we prove this rank property. With our extra measurements, we can then exploit this rank property for recovery under noise!

## Total least-squares (TLS)

The simplest "denoising" scheme, although it is more akin to parameter extraction, is to perform the [Singular Value Decomposition](https://en.wikipedia.org/wiki/Singular-value_decomposition) (SVD) of the Toeplitz matrix:

\begin{align}
\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})  = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^H, \nonumber
\end{align}

where $ \mathbf{\Sigma} \in \mathbb{C}^{(2M+1-K)\times(K+1)} $ is a rectangular diagonal matrix of singular values in _decreasing order_ along the diagonal, $ \mathbf{U}\in \mathbb{C}^{(2M+1-K)\times(2M+1-K)} $ is a [unitary](https://en.wikipedia.org/wiki/Unitary_matrix) matrix, and $ \mathbf{V}\in\mathbb{C}^{(K+1)\times(K+1)} $ is another unitary matrix. Since $ \text{rank}\big(\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) \big)=K $ for noiseless Fourier coefficients, the $ (K+1)^{th} $ singular value along the diagonal of $ \mathbf{\Sigma} $ should ideally be equal to zero!

We can write this as:

\begin{align}
\sigma_{K+1}\big(\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) \big) = 0, \nonumber
\end{align}

where $ \sigma_K(\mathbf{A}) $ denotes the $ K^{th} $ largest singular value of $ \mathbf{A} $. Therefore, the null space of $ \mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})  $ is spanned by the $ (\mathrel{{K}{+}{1}})^{th} $ column of $ \mathbf{V} $, i.e. the last column.

Mathematically, this implies:

\begin{equation}
\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})\hspace{0.05cm}\mathbf{V}_{(-1)} = \mathbf{0}, \nonumber
\end{equation}

where $ \mathbf{V}_{(-1)}$ extracts the last column of $ \mathbf{V} $. This means that $  \mathbf{V}_{(-1)} $ meets the annihilation constraint we saw earlier: $\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) \hspace{0.05cm}\mathbf{a} = \mathbf{0}$.

However, _noisy_ Fourier coefficients $ \mathbf{\hat{x}}_{noisy} $ estimated from the measured samples will typically result in a Toeplitz matrix $ \mathbf{T}(\mathbf{\hat{x}}_{noisy}, \mathrel{{K}{+}{1}} ) $ with a rank larger than $ K $. Therefore, the $ (\mathrel{{K}{+}{1}})^{th} $ singular value of $ \mathbf{T}(\mathbf{\hat{x}}_{noisy}, \mathrel{{K}{+}{1}} )$ may not be equal to zero. Nonetheless, if this singular value is significantly smaller than the $ K^{th} $ singular value, $ \mathbf{V}_{(-1)} $ may still be a sufficient candidate for the annihilating filter.

This method for obtaining the annihilating filter with the SVD is often called the **_total least-squares approach_** and abbreviated as TLS. It was first suggested within the context of FRI in [1].

Let's see how this approach works with synthetic data! We will consider additive white Gaussian noise at various _signal-to-noise ratio_ (SNR) values. The variance of the noise will be set according to the signal energy and the desired SNR in dB. Please refer the function `'add_noise'` in `'frius/us_utils.py'` for the implementation details.

# 3. Simulations

We first consider critical sampling and recovery as seen in the previous notebooks for the noiseless case _but_ with noisy samples.

```python
# signal parameters
K = 6
period = 1
ck, tk = create_pulse_param(K=K, period=period)

# critical sampling
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period)

# add noise
snr_db = 50
y_noisy = add_noise(y_samp, snr_db=snr_db)

# recovery
freqs = fs_ind_base/period
fs_coeff_hat = estimate_fourier_coeff(y_noisy, t_samp)
ann_filt = compute_ann_filt(fs_coeff_hat, K)
tk_hat = estimate_time_param(ann_filt, period)
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

evaluate_recovered_param(ck, tk, ck_hat, tk_hat, viz=True, figsize=(10,5), t_max=period)
plt.legend(loc="lower left");
```

With critical sampling, we can see that we completely miss out on one of the Diracs around $0.45$ seconds and are a little bit off for the Dirac around $0.4$ seconds.

With the same signal parameters and SNR, we will oversample the FRI by a factor of $\beta = 2$ and apply the TLS approach described above.

```python
from scipy.linalg import toeplitz, svd

def total_least_squares(fs_coeff, K):
    col1 = fs_coeff[K:]
    row1 = np.flipud(fs_coeff[:K+1])
    A_top = toeplitz(col1, r=row1)
    U, s, Vh = svd(A_top)
    print("K/(K+1) singular value = %f\n" % (s[K-1]/s[K]))
    return np.conj(Vh[-1, :])

oversample_freq = 2
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, oversample_freq=oversample_freq)

# add noise
y_noisy = add_noise(y_samp, snr_db=snr_db)

# recovery
freqs = fs_ind_base/period
fs_coeff_hat = estimate_fourier_coeff(y_noisy, t_samp)
ann_filt = total_least_squares(fs_coeff_hat, K)
# ann_filt = compute_ann_filt(fs_coeff_hat, K, print_ratio=True)
tk_hat = estimate_time_param(ann_filt, period)
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

evaluate_recovered_param(ck, tk, ck_hat, tk_hat, viz=True, figsize=(10,5), t_max=period)
plt.legend(loc="lower left");
```

And we are able to obtain the pulse parameters quite well!

It is also possible to use the same function `'compute_ann_filt'` from `'frius/fri_utils.py'`, as it will detect if we are in a scenario where we have more Fourier coefficients than $(2K+1)$ and apply TLS if it is the case.

# More noise!

An SNR of $50$ dB corresponds to very little noise. Below we plot the clean and noisy samples from our critical sampling scenario.

```python
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period)

# add noise
snr_db = 50
y_noisy = add_noise(y_samp, snr_db=snr_db)

# visualize
plt.figure(figsize=(10,5))
plt.plot(t_samp, y_samp, label="Clean")
plt.plot(t_samp, y_noisy, label="Noisy")
plt.xlabel("Time [seconds]")
plt.xlim([0, period])
plt.grid()
plt.legend();
```

The two signals are practically identical! This reflects the very sensitive behavior of the recovery algorithm even to the _slightest_ perturbation. Let's try a smaller SNR, i.e. one that will result in a noticeable perturbation of the clean samples, and with an oversampling factor of $\beta= 7$. Generally, with more noise, we will need to oversample by a larger factor.

```python
oversample_freq = 7
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, oversample_freq=oversample_freq)

# add noise
snr_db = 10
y_noisy = add_noise(y_samp, snr_db=snr_db)

# visualize
plt.figure(figsize=(10,5))
plt.plot(t_samp, y_samp, label="Clean")
plt.plot(t_samp, y_noisy, label="Noisy")
plt.xlabel("Time [seconds]")
plt.xlim([0, period])
plt.grid()
plt.legend();
```

And now let's try recovering the pulse parameters.

```python
# recovery
freqs = fs_ind_base/period
fs_coeff_hat = estimate_fourier_coeff(y_noisy, t_samp)
ann_filt = compute_ann_filt(fs_coeff_hat, K, print_ratio=True)
tk_hat = estimate_time_param(ann_filt, period)
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

evaluate_recovered_param(ck, tk, ck_hat, tk_hat, viz=True, figsize=(10,5), t_max=period)
plt.legend(loc="lower left");
```

Not so great, but we can also notice that the ratio between the $K^{th}$ and $ (\mathrel{{K}{+}{1}})^{th} $ singular value is considerably smaller ($1.34$ compared to $22.7$).

If we increase the oversampling factor to $\beta=10$, we are able to perform a better recovery. However, we still miss out on one Dirac, and the singular value ratio is still not desirable to represent a low rank matrix that we are expecting.

```python
oversample_freq = 10
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, oversample_freq=oversample_freq)

# add noise
y_noisy = add_noise(y_samp, snr_db=snr_db)

# recovery
freqs = fs_ind_base/period
fs_coeff_hat = estimate_fourier_coeff(y_noisy, t_samp)
ann_filt = compute_ann_filt(fs_coeff_hat, K, print_ratio=True)
tk_hat = estimate_time_param(ann_filt, period)
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

evaluate_recovered_param(ck, tk, ck_hat, tk_hat, viz=True, figsize=(10,5), t_max=period)
plt.legend(loc="lower left");
```

This diffulty for lower SNRs motivates another denoising approach, which we will consider in the following notebook, called **_Cadzow's iterative denoising_**. It is also suggested by [1] and again exploits the low rank property of the Toeplitz matrix of Fourier coefficient in order to perform a "model matching" step prior to TLS.

As a small teaser, below we employ Cadzow's iterative denoising to (quite) sucessfully recovery the pulse parameters for an oversampling factor of $\beta=7$.

```python
from frius import cadzow_denoising

oversample_freq = 7
snr_db = 10
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, oversample_freq=oversample_freq)

# add noise
y_noisy = add_noise(y_samp, snr_db=snr_db)

# recovery
freqs = fs_ind_base/period
fs_coeff_hat = estimate_fourier_coeff(y_noisy, t_samp)
fs_coeff_hat = cadzow_denoising(fs_coeff_hat, K, n_iter=2)
ann_filt = compute_ann_filt(fs_coeff_hat, K, print_ratio=True)
tk_hat = estimate_time_param(ann_filt, period)
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

evaluate_recovered_param(ck, tk, ck_hat, tk_hat, viz=True, figsize=(10,5), t_max=period)
plt.legend(loc="lower left");
```

That's certainly a more representation singular value ratio for a Toeplitz matrix with rank $K$!

# Appendix: Rank of Toeplitz matrix from pulse stream Fourier coefficients

Assuming that we are correct in our choice of $ K $ pulses and that we have obtained noiseless Fourier coefficients $ \{\hat{x}[m]\}_{m=-M}^{M} $  for  $ M \geq K $, the rectangular Toeplitz matrix

\begin{align}
\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) = \begin{bmatrix}
\hat{x}[\mathrel{{-M}{+}{K}}]&\hat{x}[\mathrel{{-M}{+}{K}{-}{1}}] & \cdots & \hat{x}[-M]\\
\vdots&\ddots &  \ddots & \vdots\\
\hat{x}[1] & \hat{x}[0] &   \cdots & \hat{x}[\mathrel{{-K}{+}{1}}] \\
\hat{x}[2] & \hat{x}[1] &  \cdots & \hat{x}[\mathrel{{-K}{+}{2}}] \\
\vdots&\vdots& \ddots & \vdots\\
\hat{x}[M] & \hat{x}[\mathrel{{M}{-}{1}}]  & \cdots & \hat{x}[\mathrel{{M}{-}{K}}]
\end{bmatrix},  \nonumber
\end{align}

of size $ (\mathrel{{2M}{+}{1}{-}{K}})\times(\mathrel{{K}{+}{1}}) $ has a rank of $ K $. This rank property can be verified by reminding ourselves of the expression for each $ \{\hat{x}[m]\}_{m=-M}^{M} $ for the pulse stream model (through the Fourier Series representation):

\begin{align}
\hat{x}[m] &= \sum_{k=0}^{K-1} c_k \hspace{0.05cm} \exp(-j2\pi m t_k/T) \nonumber\\
&= \sum_{k=0}^{K-1}  c_k \hspace{0.05cm} u_k^m, \nonumber
\end{align}

where $u_k = \exp(-j2\pi m t_k/T)$. We can therefore write $ \mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})$ as:

\begin{align}
\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})&=
\underbrace{
    \begin{bmatrix}
	u_0^{-M+K}&u_1^{-M+K} & \cdots & u_{K-1}^{-M+K} \\
	u_0^{-M+K-1}&u_1^{-M+K-1} & \cdots & u_{K-1}^{-M+K-1} \\
	\vdots&\vdots &  \ddots & \vdots\\[0.2cm]
	u_0^{M}&u_1^{M} & \cdots & u_{K-1}^{M}
	\end{bmatrix}}_{\mathbf{C}}
\cdot
\underbrace{
	% for same height
	\vphantom{ \begin{bmatrix}
		u_0^{-M+K}&u_1^{-M+K} & \cdots & u_{K-1}^{-M+K} \\
		u_0^{-M+K-1}&u_1^{-M+K-1} & \cdots & u_{K-1}^{-M+K-1} \\
		\vdots&\vdots &  \ddots & \vdots\\[0.2cm]
		u_0^{M}&u_1^{M} & \cdots & u_{K-1}^{M}
		\end{bmatrix}}
	\begin{bmatrix}
	c_0&0 & \cdots & 0 \\
	0&c_1& \cdots & 0\\
	\vdots&\vdots &  \ddots & \vdots\\[0.1cm]
	0&0& \cdots & c_{K-1}
	\end{bmatrix}}_{\mathbf{D}}
\cdot
\underbrace{\begin{bmatrix}
	1&u_0^{-1} & \cdots & u_{0}^{-K} \\
	1&u_1^{-1} & \cdots & u_{1}^{-K} \\
	\vdots&\vdots &  \ddots & \vdots\\[0.2cm]
	1 & u_{K-1}^{-1}  &\cdots & u_{K-1}^{-K}.
	\end{bmatrix}}_{\mathbf{E}}, \nonumber
\end{align}

where $ \mathbf{C}\in \mathbb{C}^{ (2M+1-K)\times K} $, $ \mathbf{D} \in \mathbb{C}^{K\times K} $, and $ \mathbf{E}\in \mathbb{C}^{K\times (K+1)}$. The diagonal matrix $ \mathbf{D} $ certainly has a rank of $ K $ so using the following rank properties (where $ \mathbf{A} \in \mathbb{C}^{m\times n}$):

\begin{align}
\text{rank}(\mathbf{A}) &\leq \text{min}(m,n), \nonumber\\
\text{rank}(\mathbf{A}\mathbf{B}) &\leq \min\Big(\text{rank}(\mathbf{A}), \text{rank}(\mathbf{B})\Big), \nonumber
\end{align}

we know that $ \text{rank}\big(\mathbf{T}(\mathbf{\hat{x}})\big) \leq K$ since both dimensions of $ \mathbf{C} $ and $ \mathbf{E} $ are $ \geq K $. Moreover, as $ \mathbf{C} $ and $ \mathbf{E} $ have a Vandermonde structure _and_ the $ u_k $'s are distinct (as we are assuming $ K $ different pulses), both $ \mathbf{C} $ and $ \mathbf{E} $ also have rank $ K $, i.e. their smallest dimension. Therefore, $ \text{rank}\big(\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})\big) = K$.

Even if we chose to create a Toeplitz with more columns, i.e. $ \mathbf{T}(\mathbf{\hat{x}}, L) $ with $ L > (K+1) $, the rank would still be $ K $. This can be shown by adding more rows to $ \mathbf{C} $ and columns to $ \mathbf{E} $ to the decomposition above. Nevertheless, the rank would remain $ K $ as $ \mathbf{D} $ would remain unchanged, having a rank of $K$.

# References

    [1] T. Blu, P. L. Dragotti, M. Vetterli, P. Marziliano and L. Coulot, "Sparse Sampling of Signal Innovations," in IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 31-40, March 2008.


# Information Quantities for Decision Tree Induction

**CS5483 Data Warehousing and Data Mining**
___

```python
%reset -f
%matplotlib inline
import dit
from dit.shannon import entropy, conditional_entropy, mutual_information
from IPython.display import Math
import matplotlib.pyplot as plt
```

In this notebook, we will use the [`dit` package](https://dit.readthedocs.io/en/latest/) to compute some basic information quantities used in the decision tree algorithms. A summary of Shannon's information measures and their relationships are given first.

## Information Measures

The followings are the mathemtical definitions of *entropy* and *mutual information*:

\begin{align}
H(Y) &= E\left[\log \tfrac1{P_{Y}(Y)}\right] && \text{entropy of $Y$}\\
H(Y|X) &= E\left[\log \tfrac1{P_{Y|X}(Y|X)}\right] && \text{conditional entropy of $Y$ given $X$}\\
I(X;Y) &= E\left[\log \tfrac{P_{Y|X}(Y|X)}{P_Y(Y)}\right] && \text{mutual information of $X$ and $Y$}\\
\end{align}

These information quantities can be related using a *Venn Diagram*:

\begin{align}
H(X,Y)&=H(X)+H(Y|X) && \text{chain rule of entropy}\\
&=H(Y)+H(X|Y)\\
I(X;Y)&=H(Y)-H(Y|X) && \text{mutual information in terms of entropies}\\
&=H(X)+H(Y)-H(X,Y)\\
&=H(X)-H(X|Y)
\end{align}

## Entropy

Consider the following distribution:
\begin{align}
p_k=\begin{cases}
\frac12 & k=0\\
\frac14 & k=1,2\\
0 & \text{otherwise.}
\end{cases}
\end{align}

```python
p = dit.Distribution(['0', '1', '2', '3'], [1/2, 1/4, 1/4, 0])

plt.stem(p.outcomes,p.pmf,use_line_collection=True)
plt.xlabel('k')
plt.ylabel(r'$p_k$')
plt.ylim((0,1))
plt.show()
```

```python
Math(f'h(p_1,p_2,\dots)={entropy(p)}')
```

## Information Gain

Consider the dataset $D$:

|X1|X2|X3|X4|Y|
|:---:|:---:|:---:|:---:|:-:|
|0    |0    |0    |00   |0  |
|0    |0    |0    |00   |0  |
|0    |0    |1    |01   |1  |
|1    |0    |1    |11   |1  |
|0    |1    |0    |00   |2  |
|1    |1    |0    |10   |2  |
|1    |1    |1    |11   |3  |
|1    |1    |1    |11   |3  |

**How to determine which attribute is more informative?**

First, create a uniform distribution over the instances in $D$.

```python
d = dit.uniform([('0','0','0','00','0'),
                 ('0','0','0','00','0'),
                 ('0','0','1','01','1'),
                 ('1','0','1','11','1'),
                 ('0','1','0','00','2'),
                 ('1','1','0','10','2'),
                 ('1','1','1','11','3'),
                 ('1','1','1','11','3')])
d.set_rv_names(('X1','X2','X3','X4','Y'))
d
```

We can then calculate $\text{Info}(D)$ and $\text{Info}_{X_i}(D)$ for $i=\{1,2,3,4\}$ as the entropy $H(Y)$ and conditional entropies $H(Y|X_i)$'s respectively.

```python
InfoD = entropy(d,['Y'])
InfoX1D = conditional_entropy(d,['Y'],['X1'])
InfoX2D = conditional_entropy(d,['Y'],['X2'])
InfoX3D = conditional_entropy(d,['Y'],['X3'])
InfoX4D = conditional_entropy(d,['Y'],['X4'])

Math(r'''
\begin{{aligned}}
\text{{Info}}(D)&={}\\
\text{{Info}}_{{X_1}}(D)&={:.3g}\\
\text{{Info}}_{{X_2}}(D)&={:.3g}\\
\text{{Info}}_{{X_3}}(D)&={:.3g}\\
\text{{Info}}_{{X_4}}(D)&={:.3g}\\
\end{{aligned}}
'''.format(InfoD,InfoX1D,InfoX2D,InfoX3D,InfoX4D))
```

The information gain $\text{Gain}_{X_i}(D)$ can be calculated as the mutual information $I(X_i;Y):=H(Y)-H(Y|X_i)$.

```python
GainX1D = mutual_information(d,['X1'],['Y'])
GainX2D = mutual_information(d,['X2'],['Y'])
GainX3D = mutual_information(d,['X3'],['Y'])
GainX4D = mutual_information(d,['X4'],['Y'])

Math(r'''
\begin{{aligned}}
\text{{Gain}}_{{X_1}}(D)&={:.3g}\\
\text{{Gain}}_{{X_2}}(D)&={:.3g}\\
\text{{Gain}}_{{X_3}}(D)&={:.3g}\\
\text{{Gain}}_{{X_4}}(D)&={:.3g}\\
\end{{aligned}}
'''.format(GainX1D,GainX2D,GainX3D,GainX4D))
```

**Exercise** Which attribute gives the highest information gain? Should we choose it as the splitting attribute?

YOUR ANSWER HERE

## Information Gain Ratio

To normalize information gain properly, we first calculate $\text{SplitInfo}_{X_i}(D)$ as $H(X_i)$:

```python
SplitInfoX1D = entropy(d,['X1'])
SplitInfoX2D = entropy(d,['X2'])
SplitInfoX3D = entropy(d,['X3'])
SplitInfoX4D = entropy(d,['X4'])

Math(r'''
\begin{{aligned}}
\text{{SplitInfo}}_{{X_1}}(D)&={:.3g}\\
\text{{SplitInfo}}_{{X_2}}(D)&={:.3g}\\
\text{{SplitInfo}}_{{X_3}}(D)&={:.3g}\\
\text{{SplitInfo}}_{{X_4}}(D)&={:.3g}\\
\end{{aligned}}
'''.format(SplitInfoX1D,SplitInfoX2D,SplitInfoX3D,SplitInfoX4D))
```

Finally, to calculate the information gain ratios:

```python
Math(r'''
\begin{{aligned}}
\frac{{\text{{Gain}}_{{X_1}}(D)}}{{\text{{SplitInfo}}_{{X_1}}(D)}}&={:.3g}\\
\frac{{\text{{Gain}}_{{X_2}}(D)}}{{\text{{SplitInfo}}_{{X_2}}(D)}}&={:.3g}\\
\frac{{\text{{Gain}}_{{X_3}}(D)}}{{\text{{SplitInfo}}_{{X_3}}(D)}}&={:.3g}\\
\frac{{\text{{Gain}}_{{X_4}}(D)}}{{\text{{SplitInfo}}_{{X_4}}(D)}}&={:.3g}\\
\end{{aligned}}
'''.format(GainX1D/SplitInfoX1D,GainX2D/SplitInfoX2D,GainX3D/SplitInfoX3D,GainX4D/SplitInfoX4D))
```

**Exercise** Is $X_4$ a good splitting attribute? Why?

YOUR ANSWER HERE

Tutorial3/Information Quantities for Decision Tree Induction.ipynb


The parameter space of SSMs provides a compact representation that is amenable to learning algorithms (e.g. classification or clustering), evaluation, and exploration. Some of these aspects will be explored in this part of the tutorial.

We employ shape data of 116 distal femora that where, among others, used in

[Rigid motion invariant statistical shape modeling based on discrete fundamental forms](https://doi.org/10.1016/j.media.2021.102178), F. Ambellan, S. Zachow, Christoph von Tycowicz, Medical Image Analysis (2021). [PDF](https://arxiv.org/pdf/2111.06850.pdf)

The respective segmentation masks are publicly available at [pubdata.zib.de](https://pubdata.zib.de).

To speed up things a little bit, this part of the tutorial starts right after the model creation step, i.e. we have already constructed a shape model. We will work with the shape weights stored in `SSM.coeffs` (uniquely determining all input shapes).

The data set splits into two subgoups of the same cardinality, namely, healthy and diseased femora. In the following we want to visualize the shape weights in different ways and later use them to perform a classification experiment on osteoarthritis.

# Visualization of Principal Weights

## Load data

At first we load the principal weights that are stored together with the labels (healthy/diseased) in a numpy array for PDM and FCM, repsectively. At second we define unique colors to represent FCM and PDM in all subsequent plots.

```python
import numpy as np
from tutorial2_pop_med_image_shape_ana.utils.sammon import sammon
from tutorial2_pop_med_image_shape_ana.utils.utils import runSVMClassification, plotClassificationResults
from sklearn.preprocessing import normalize
import matplotlib.pyplot as plt

dataFCM = np.load('tutorial2_pop_med_image_shape_ana/data/femurOAclassificationDataFCM.npy')
dataPDM = np.load('tutorial2_pop_med_image_shape_ana/data/femurOAclassificationDataPDM.npy')

# first row -> label {0, 1}, second to last row -> shape weights (column is sample, row is feature)
labels = dataFCM[0, :]
fcmFeatures = dataFCM[1:, :]
pdmFeatures = dataPDM[1:, :]

# dark green (FCM) and dark violet (PDM)
colors = ['#008c04', '#ae00d8']
```

## Task 4 on Visualization of Two Principal Weights

At first will will focus on different pairs of two weights, indexed by (`pw1`, `pw2`), that we can easily visualize in 2d scatter plots. We heryby assign different markers to different disease states.

Choose different values for `pw1` and `pw2` (values between 0 and 114). What is your impression?
- Are there some weights to appear more expressive than others? (w.r.t. disease state)
- If so, what do you think why?
- What can you say about the difference between FCM and PDM?

```python
[pW1, pW2] = [0, 1] # eg [0, 1], [10, 15], [0, 114] ...

# split data into healthy and diseased index lists
healthy = np.where(labels == 0)[0]
diseased = np.where(labels != 0)[0]

fig, ax = plt.subplots(1, 2, figsize=(12, 5))
data_list = [fcmFeatures, pdmFeatures]
title_list = ['FCM-weights of Two Principal Directions', ' PDM-weights of Two Principal Directions']
legendLocation_list = ['upper right', 'upper left']

for k in range(2):
    ax[k].scatter(data_list[k][pW1, healthy], data_list[k][pW2, healthy], s=40, linewidths=2, c=colors[k], label='healthy')
    ax[k].scatter(data_list[k][pW1, diseased], data_list[k][pW2, diseased], s=40, linewidths=2, c='white', edgecolors=colors[k], label='diseased')
    ax[k].set_xticks([])
    ax[k].set_yticks([])
    ax[k].set_title(title_list[k])
    ax[k].legend(loc=legendLocation_list[k])

plt.show()
```

## Task 5 on Sammon Projection to Two Dimensions

The Sammon projection tries to find a low dimensional (2d in our case) representation of some given high dimensional data, s.t. the following error is minimal

\begin{equation}
Err = \dfrac{1}{\sum_{i<j}d_{R^2}(\text{pr}(\alpha_i), \text{pr}(\alpha_i))} \sum_{i<j}\dfrac{(d_{R^2}(\text{pr}(\alpha_i), \text{pr}(\alpha_i)) - d_{R^d}(\alpha_i, \alpha_j))^2}{d_{R^2}(\text{pr}(\alpha_i), \text{pr}(\alpha_i))}.
\end{equation}

In other words: The distances between two weight vectors $\alpha_i, \alpha_j$ before projection should be close to those after projection.

(cf. [A nonlinear mapping for data structure analysis](https://doi.org/10.1109/T-C.1969.222678), JW. Sammon, IEEE Transactions on computers (1969). [PDF](http://syllabus.cs.manchester.ac.uk/pgt/2021/COMP61021/reference/Sammon.pdf))

Sammon projection can also be applied to a subset of the weight vectors, defined by the index range (`nR1`, `nR2`).

Choose different values for `nR1` and `nR2` (values between 0 and 114, `nR1` < `nR2 + 1`). What is your impression?
- Are there some weights (subsets of weights) to appear more expressive than others? (w.r.t. disease state)
- If so, what do you think why?
- What can you say about the difference between FCM and PDM?

```python
[nR1, nR2] = [0, 114] # e.g. [0, 114] (full range),  [105, 114], etc.

fcmSammon, _ = sammon(fcmFeatures[nR1:nR2, :].transpose(), 2, display=0)
pdmSammon, _ = sammon(pdmFeatures[nR1:nR2, :].transpose(), 2, display=0)

fig, ax = plt.subplots(1, 2, figsize=(12, 5))
data_list = [fcmSammon, pdmSammon]
title_list = ['Sammon Projection of FCM-weights', 'Sammon Projection of PDM-weights']

for k in range(2):
    ax[k].scatter(data_list[k][healthy, 0], data_list[k][healthy, 1], s=40, linewidths=2, c=colors[k], label='healthy')
    ax[k].scatter(data_list[k][diseased, 0], data_list[k][diseased, 1], s=40, linewidths=2, c='white', edgecolors=colors[k], label='diseased')
    ax[k].set_xticks([])
    ax[k].set_yticks([])
    ax[k].set_title(title_list[k])
    ax[k].legend(loc=legendLocation_list[k])

plt.show()
```

# Task 6 on Osteoarthritis Classification Experiment

We emply a linear Support Vector Machine (SVM) trained on the princpal weights of PDM and FCM in order to classify distal femur bones as healthy of diseased w.r.t knee osteoarthritis.

To get a more complete picture we train the SVM on diffrent partitions of the data, e.g. `nPartitions=9` indicats SVM classifers trained on 10% to 90% (randomly selected elements) of all input data, using the repsective complement as test set. In order to acknowledge the randomess in the experiment design appropriately we repeat the experiment for every partition 'nRandomSamplings' times.

Furthermore, classification can be carried out also on a a subset of the weight vectors, defined by the index range (`nR1`, `nR2`).

The averaged results are plotted together with bars quantifying the standard deviation.

Choose different values for `nR1` and `nR2` (values between 0 and 114, `nR1 < nR2`). What is your impression?
- Are there some weights (subsets of weights) to appear more expressive than others? (w.r.t. disease state)
- If so, what do you think why?
- What can you say about the difference between FCM and PDM?
- Choose different values for 'nRandomSamplings' (e.g. 10, 100, 1000). Interpret what you see.

```python
nPartitions = 9
nRandomSamplings = 100

[nR1, nR2] = [0, 114] # e.g. [0, 114] (full range), [0, 3], [105, 114], etc.

# normalize feature vectors
fcmFeaturesNorm = normalize(fcmFeatures[nR1:nR2, :], axis=0, norm="l2")
pdmFeaturesNorm = normalize(pdmFeatures[nR1:nR2, :], axis=0, norm="l2")

fcmavgAccuracyPerPartition, fcmstdDevPerPartition = runSVMClassification(nPartitions, nRandomSamplings, fcmFeaturesNorm, labels)
pdmavgAccuracyPerPartition, pdmstdDevPerPartition = runSVMClassification(nPartitions, nRandomSamplings, pdmFeaturesNorm, labels)

data_list_avg = [fcmavgAccuracyPerPartition, pdmavgAccuracyPerPartition]
data_list_std = [fcmstdDevPerPartition, fcmstdDevPerPartition]

plotClassificationResults(data_list_avg, data_list_std, plt, colors)

```

```python

```


# Linear Models

## Regression
When applying regression, we have a dataset $\textbf{X}$ such that

$$\textbf{X} = \left\{\textbf{x}^{(1)}, \ldots, \textbf{x}^{(n)}\right\}, \textrm{where}\; \textbf{x}^{(i)}\in\mathbb{R}^d$$

This dataset has corresponding labels

$$\textbf{y}=\left\{y^{(1)},\ldots,y^{(n)}\right\}, \textrm{where}\; y^{(i)}\in\mathbb{R}$$

The goal is to find a hypothesis $h$ in the set of all hypothesis $H$, such that

$$ h: \mathbb{R}^d \to \mathbb{R}, \textrm{where}\; h \in H $$

This hypothesis $h$ can be written as:

$$ h(\textbf{x}) = \sum_{j=1}^d \theta_j x_j + \theta_0 x_0 \overset{*}{=} \theta^T\textbf{x}, \textrm{where}\; x_0 = 1 $$

Mind that at $\overset{*}=$ the intercept $x_0$ is put into the vector $\textbf{x}$.

Now we would like to know how far our predictions are of from the true values. To do so we need to introduce some metrics.

## Some metrics
As a first exercise we will implement the **sum of squared errors** using simple matrix operations, do this in `./code/utils.py`

These exercises are based on the [scikit-learn User Guide](https://scikit-learn.org/stable/modules/linear_model.html) and more specifically [this example](https://scikit-learn.org/stable/auto_examplesA/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py).

This is the mentioned example from scikit-learn:

```python
import numpy as np
import sys
sys.path.append('code/')

import utils

y1 = np.asarray([1., 1.2, 3.1, 3.7])[:,np.newaxis]
y2 = np.asarray([0.8, 1.5, 2.6, 3.1])[:,np.newaxis]

sse = utils.sum_squared_errors(y1, y2)
print(sse)
```

    0.7400000000000001

The **sum of squared errors** (SSE) can be easily extended to the **mean squared error** (MSE) and the **root mean squared error** (RMSE). Do this in `./code/utils.py`.

```python
mse = utils.mean_squared_error(y1, y2)
rmse = utils.root_mean_squared_error(y1, y2)
print(mse)
print(rmse)
```

    0.18500000000000003
    0.4301162633521314

## Fitting Linear Regression
To fit a linear regression mode, one of the solutions could be to minimise the sum of squared errors:

$$ SSE(\theta)=\sum_{i=1}^n\left(y^{(i)}-\theta^T\textbf{x}^{(i)}\right)^2 $$

A general solution can be written as:

$$ \theta = \left(\textbf{X}^T \textbf{X}\right)^{-1} \textbf{X}^T \textbf{y} $$

Try to find this general solution yourself using the derivative of $SSE(\theta)$ with respect to $\theta$ and using the product rule, the first steps are given:

\begin{align}
\frac{\partial SSE(\theta)}{\partial \theta}
    &= \frac{\partial}{\partial \theta} \sum_{i=1}^n \left(y^{(i)} - \theta^T\textbf{x}^{(i)}\right)^2 \\
    &\overset{\square}= \frac{\partial}{\partial \theta} \left(\textbf{y} - \textbf{X}\theta\right)^2 \\
    &\overset{*}= 2 \textbf{X}^T (\textbf{y} - \textbf{X}\theta)
\end{align}

By using the definition of inner product at $\square$ and the product rule at $*$.

Now set the derivative to zero a

Now we can use the general solution to fit a linear regression model to some data. For this exercise, the diabetes dataset will be used that comes with *scikit-learn*.

```python
from sklearn import datasets, linear_model, model_selection

# load the dataset
diabetes_X, diabetes_y = datasets.load_diabetes(return_X_y=True)

# And inspect the shapes
print(diabetes_X.shape)
print(diabetes_y.shape)

# Split in train and test data (why?)
X_train, X_test, y_train, y_test = model_selection.train_test_split(diabetes_X, diabetes_y)
```

    (442, 10)
    (442,)

This dataset has 442 samples and the samples have 10 parameters. Thus, we need to fit a linear model having 11 parameters, the 10 parameters in the dataset and the intercept. The next exercise will be to fit a least squared erros linear model to the diabetes data.

```python
import sys
sys.path.append('code/')
import linear_models
import utils

# Fit a linear model using least squared error to the training data
theta = linear_models.lse_fit(X_train, y_train)

# Predict y
y_predicted = linear_models.lse_predict(X_test, theta)

# Get performance by evaluating y_predicted with respect to y_test
mse = utils.mean_squared_error(y_test, y_predicted)
print(mse)

```

    3149.4411216258404

## Ideas for additional topics
### polynomial linear fit


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Sequences are typically written in terms of an independent variable $n$,
so we will tell SymPy to use $n$ as a symbol, then define our sequence
in terms of $n$.

We define a term of an example sequence as $a_n=\frac{1}{n+1}$, then
build a sequence from that term.  The code `(n,0,oo)` means that $n$
starts counting at $n=0$ and goes on forever (with `oo` being the SymPy
notation for $\infty$).

```python
var( 'n' )                       # use n as a symbol
a_n = 1 / ( n + 1 )              # formula for a term
seq = sequence( a_n, (n,0,oo) )  # build the sequence
seq
```

$\displaystyle \left[1, \frac{1}{2}, \frac{1}{3}, \frac{1}{4}, \ldots\right]$

You can ask for specific terms in the sequence, or many terms in a row, as follows.

```python
seq[20]
```

$\displaystyle \frac{1}{21}$

```python
seq[:10]
```

$\displaystyle \left[ 1, \  \frac{1}{2}, \  \frac{1}{3}, \  \frac{1}{4}, \  \frac{1}{5}, \  \frac{1}{6}, \  \frac{1}{7}, \  \frac{1}{8}, \  \frac{1}{9}, \  \frac{1}{10}\right]$

You can compute the limit of a sequence,

$$ \lim_{n\to\infty} a_n. $$

```python
limit( a_n, n, oo )
```

$\displaystyle 0$

database/tasks/How to define a mathematical sequence/Python, using SymPy.ipynb


---
# 3D Wing Motion Kinematics

---

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import trapz
try:
    plt.style.use('jupyter')
except OSerror:
    print('...')
```

#### Marin Lauber 2020

The ellipsoidal wing is defined as

\begin{equation}
    \frac{x^2}{a^2} + \frac{y^2}{b^2} + \frac{z^2}{c^2} = 1,
\end{equation}

with $a=0.5$, $b=0.05$ and $c=1$.

```python
# wing
def chord(r, a=0.5):
    return np.sqrt(abs((1-r**2)*a**2))
def c(r, Rtip=1.25):
    return np.where(r<(Rtip-1.0), 0, chord(r-.75))
```

Its motion is deine by three Euler angles

\begin{split}
    \varphi(t) &= A_\varphi\cos(2\pi f_r t), \\
    \theta(t) &= A_\theta\sin(2\pi f_\theta t), \\
    \alpha(t) &= \frac{\pi}{2} - A_\alpha\sin(2\pi f_r t + \xi),
\end{split}

where $\varphi(t)$ corresponds to the stroke variation, $\alpha(t)$ to the geometrical angle of attack and $\theta(t)$ to the deviation from the horizontal stroke plane. $\xi$ is the phase difference between $\alpha$ and $\theta$. Figure-0 motion are obtained by setting $f_\theta = f_r$ and figure-of-8 by setting $f_\theta = 2f_r$

```python
# elevation angle
def theta(t, A, f):
    return A*np.sin(2*np.pi*f*t)
def thetadot(t, A, f):
    return 2*np.pi*f*A*np.cos(2*np.pi*f*t)

# stroke angle
def phi(t, A, f):
    return A*np.cos(2*np.pi*f*t)
def phidot(t, A, f):
    return -2*np.pi*f*A*np.sin(2*np.pi*f*t)

# angle of attack
def alpha(t, A, f, xi=0.):
    return np.pi/2. - A*np.sin(2*np.pi*f*t + xi)
def alphadot(t, A, f, xi=0):
    return 2*np.pi*f*A*np.cos(2*np.pi*f*t + xi)
```

The aerodynamic performance of the flapping wing is evaluated using the drag coefficient $C_D$ and lift coefficient $C_L$

\begin{split}
    C_D &= \frac{F_x\cos(\varphi) - F_z\sin(\varphi)}{\frac{1}{2}\rho U_{ref}^2S}, \\
    C_L &= \frac{F_y}{\frac{1}{2}\rho U_{ref}^2S}. \\
\end{split}

The wing area is $S=\pi/4.$ and the reference velocity is defined as (average velocity magnitude over one period)

\begin{equation}
    U_{ref} = f_r\int_{0}^{1/f_r}\sqrt{\dot{\varphi}^2 + \dot{\theta}^2}dt R_g \qquad\left( = f_r\int_{0}^{1/f_r}|\dot{\varphi}| dt R_g\right).
\end{equation}

When $\theta(t)\equiv0$.

\begin{equation}
U_{ref} = 2\pi f_r A_\varphi f_r\int_{0}^{1/f_r}|\sin(2\pi f_r t)| dt R_g =0.5 \pi A_\varphi \int_{0}^{2}|\sin(\pi t)| dt R_g \sim 0.5 \pi A_\varphi 1.2732 R_g
\end{equation}
The radius of gyration is

\begin{equation}
    R_g = \sqrt{\frac{1}{S}\int_{0}^{R}cr^2 dr}.
\end{equation}

Both those integrals are better carried numerically.

```python
# kinematic params
fr = 0.5
t, dt = np.linspace(0.0, 1./fr, 512, retstep=True)
r, dr = np.linspace(0, 2.5, 512, retstep=True)
A_phi = 0.35*np.pi
S = np.pi/2.
cbar = np.pi/4.
nu = 0.0273
```

```python
# integrals
Rg = np.sqrt(1./S*trapz(c(r)*r**2, r, dr))
Uref = fr * trapz(abs(phidot(t, A_phi, fr)), t, dt) * Rg
```

```python
## printing stuff
print('Radius of gyration : %.3f' % Rg)
print('Uref : %.3f' % Uref)
```

    Radius of gyration : 1.210
    Uref : 2.660

The Reynolds number is defined as

\begin{equation}
    Re = \frac{\bar{c}U_{ref}}{\nu} = \frac{4A_\phi f_r R_g \bar{c}}{\nu}
\end{equation}

where $\nu=0.0273$ and the average cord length is $\bar{c} = \pi/4.$

```python
# Reynolds number
def Re(Uref, cbar, nu, A_phi, fr, Rg):
    return Uref*cbar/nu, 4*A_phi*fr*Rg*cbar/nu
```

```python
print('Reynolds number : %.1f (%.1f)' % Re(Uref, cbar, nu, A_phi, fr, Rg))
```

    Reynolds number : 76.5 (76.5)

```python
t = np.arange(0,256)
T = 128
plt.plot(t/T, phi(t/T, A_phi, fr));
```

```python
t = np.arange(0,256)
T = 128
plt.plot(t/T, phidot(t/T, A_phi, fr));
```

```python
def ellipse(X, radius):
    abc = radius/max(radius)
    return np.sqrt(sum(X**2/abc**2)) - max(radius)
```

```python
ellipse(np.array([0,0,.25]),np.array([1,1,.25]))
```

    0.0

```python
D = 40
x = 0.5*D*np.linspace(-2,2,64)
radius = 0.5*D*np.array([1,.5,.05])
print(radius/max(radius))
dist = np.zeros((len(x),len(x),len(x)))
for i in range(len(x)):
    for j in range(len(x)):
        for k in range(len(x)):
            X = np.array([x[i],x[j],x[k]])
            dist[k,j,i] = ellipse(X, radius)
```

    [1.   0.5  0.05]

```python
plt.contourf(x/(0.5*D),x/(0.5*D),dist[32,:,:], levels=51)
plt.colorbar()
plt.contour(x/(0.5*D),x/(0.5*D),dist[32,:,:],levels=0, colors="red")
plt.axis("equal");
```

```python
plt.contourf(x/(0.5*D),x/(0.5*D),dist[:,32,:], levels=51)
plt.colorbar()
plt.contour(x/(0.5*D),x/(0.5*D),dist[:,32,:],levels=0, colors="red")
plt.axis("equal");
```

```python
plt.contourf(x/(0.5*D),x/(0.5*D),dist[:,:,32], levels=51)
plt.colorbar()
plt.contour(x/(0.5*D),x/(0.5*D),dist[:,:,32],levels=0, colors="red")
plt.axis("equal");
```


```python
# https://colab.research.google.com/github/kassbohm/tm-snippets/blob/master/ipynb/TM_A/TM_0/trafo_tensor/trafo_tensor_2.ipynb

from sympy.physics.units import *
from sympy import *

# Input:

# 1.
# (i1, i2, i3)-components of symmetric tensor:
T11, T12, T13 = -1, 4, 0
T22, T23 = 5, 0
T33 = 0

# 2.
# (i1, i2, i3)-components of unit vector:
d1, d2, d3 = 0, 1, 0

pprint("\nInput 1: (i1, i2, i3)-tensor-components:")
T = Matrix([
    [T11, T12, T13],
    [T12, T22, T23],
    [T13, T23, T33]
    ])
pprint(T)

pprint("\nInput 2: (i1, i2, i3)-unit-vector-components:")
d = Matrix([d1, d2, d3])
tmp = d.norm()
assert(tmp==1)

pprint("\nOutput: Tensor-component in unit-vector-direction:")
tmp = d.transpose()*T*d
pprint(tmp)

# Input 1: (i1, i2, i3)-tensor-components:
# ⎡-1  4  0⎤
# ⎢        ⎥
# ⎢4   5  0⎥
# ⎢        ⎥
# ⎣0   0  0⎦
#
# Input 2: (i1, i2, i3)-unit-vector-components:
#
# Output: Tensor-component in unit-vector-direction:
# [5]

```


## Betting Formula on Betcha
\begin{align}
\\
B (w_i,t_i) = \beta_{1}e^{-\beta_{2} t_i} \frac{w_i}{\sum^{n}_{k=1}w_k} \sum^{m}_{k=1}l_j
\\
\\
\end{align}
where $w_i$ = money bet by ith person in the winner side
      $t_i$ = time when ith person place the bet ( $t_i \in [0,1] \ \forall i={1,2,...}$ )
      $l_j$ = money bet by jth person in the loser side
the coefficient $\beta_2$ is determined by the duration between the creation and the expiry date of bet

```python
import numpy as np
import matplotlib.pyplot as plt

t = np.linspace(0,1)

plt.style.use("seaborn-whitegrid")
palette = plt.get_cmap("Spectral")
ax1 =plt.subplot(111)
ax1.set_xticks([])

for b2 in [0.1,0.5,1,2,5,10]:
    ax1.plot(t,np.exp(-b2*t),color=palette(0.5/b+0.6),label = b2)
ax1.legend(title = "different "r"$\beta_2$"" values")
plt.yticks(np.arange(0, 1, 0.1))
plt.xticks(np.arange(0, 1, 0.1))
plt.ylabel(r"$e^{-\beta_{2} t}$")
plt.xlabel("t - time")
```

\begin{align}
\\
\sum^{n}_{i=1}B(w_i,t_i) = \sum^{m}_{k=1}l_j
\\
\end{align}
The sum of the money paid to the winners is equal to the Loser Pot,
We can then use the bootstrapping method to calculate $\beta_{1}$. ( $\beta_{1}>1$ )
Hence deducing the money obtained by individuals in the winner side.

For example: Noel, Hailey, Yoon, Matthew, Natalie and some other participants are betting on the winner of the chainhack.
Noel, Hailey and Yoon have betted on Batcha to win, and turns out Betcha did win.
So they are now rewarded by sharing the Loser Pot of £50.
The judges have set $\beta_1 = 1$.
Lets say,
**Noel** has bet on Betcha at t = 0 for £5,
**Hailey** for £5 at t = 0.1 and
**Yoon** for bet £10 at t = 0.1
then we get

```python
50/(12.5+12.5*np.exp(-0.1)+25*np.exp(-0.7))
```

    1.3802584273885596

$\beta_1$ = 1.3802584273885596

```python
print(b1*12.5,b1*12.5*np.exp(-0.1),b1*25*np.exp(-0.7))
```

    17.253230342356996 15.611368395757978 17.135401261885026

**Noel** wins £17.253230342356996 (and she will get her £5 back)
**Hailey** wins £15.611368395757978 and
**Yoon** wins £17.135401261885026

```python

```


```python
import numpy as np

# the applied stats module
import applied_stats.mle as mle
import applied_stats.continuous_distributions as st
```

# About this module

This module is intended to explore object-oriented programming implementations of common statistical distributions. The actual source code is available on PyPI here: https://pypi.org/project/applied-stats/

Each distribution is created as its own class with the pdf, plotting, and probability calculations as separate methods that can then be called on the class object (as demonstrated below).

Right now, there are continuous_distribution and MLE modules which are demonstrated below.

## Normal distribution

Where X is a normal random variable:

$X \sim N(\mu,\sigma^{2})\$

A random variable is normal if it has the following pdf:

$
\begin{equation}
f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{{-\frac{1}{2}}(\frac{x-\mu}{\sigma})^2} ,  \quad  -\infty < x < \infty
\end{equation}
$

Probabilities are calculated by the module, by default, as follows:

$
\int_{-\infty}^{0} \frac{1}{\sqrt{2\pi\sigma^2}}e^{{-\frac{1}{2}}(\frac{x-\mu}{\sigma})^2} dx
$

Where the upper limit is the optional critical value argument. As shown in the second normal distribution example, this can be changed to calculate different probabilities but will always calculate the area the the left of the c.v.

```python
#a standard normal r.v. (E(x)=0, Var(x)=1)
a = st.Norm_rv(0,1)
a.plot_pdf()
a.probability_calc()
```

```python
# mean = 0, variance = 7, and a critical value of -2.5
b = st.Norm_rv(.2,5,-2.5)

#recently added the capability to plot two-tailed distributions. As of now this
# will only plot and shade symmetric areas right and left of the crit values
b.plot_pdf(two_tail=True,cv_probability=True)
b.probability_calc()
```

## Chi-squared distribution

Where X is a chi-squared random variable with r degrees of freedom:

$ X \sim \chi^2 (r) $

A random variable is said to be chi-squared if it has the following pdf:

$
\begin{align*}
f(x; r) \begin{cases}
      \dfrac{1}{\Gamma({\frac{r}{2}}) 2^{\frac{r}{2}}} x^{\frac{r}{2}-1} e^{-\frac{x}{2}}  & 0 \leq x < \infty \\
      0 & otherwise\\
\end{cases}\\
\end{align*}
$

With mean = $r$ and variance = $2r$

By default, the probability is calculated as:

$\int_{0}^{\infty} \dfrac{1}{\Gamma({\frac{r}{2}}) 2^{\frac{r}{2}}} x^{\frac{r}{2}-1} e^{-\frac{x}{2}} dx $

Though since this will return 1, it is more practical to specify a critical value

```python
q = st.ChiSq_rv(4)
q.plot_pdf()
q.probability_calc()
```

```python
z = st.ChiSq_rv(3)
z.plot_pdf(left_cv=1, right_cv=7, cv_probability=True, two_tail=True)
z.probability_calc()
```

## t-distribution

Where X has a t distribution with v degrees of freedom:

$X \sim t(v)$

X is said to have a t-distribution if it has the following pdf:

$f(x; v) = \frac {\Gamma{(\frac{v+1}{2})}} {\sqrt{\pi v} \Gamma{\frac{v}{2}} (1 + \frac{x^2}{v})^{\frac{v+1}{2}}}, \quad -\infty < x < \infty $

```python
t = st.t_rv(5,1)
t.plot_pdf(cv_probability=True)
t.probability_calc()
```

```python
f = st.F_rv(5,5,1.5)
f.plot_pdf(cv_probability=True)
f.probability_calc()
```

# MLE Examples

```python
sample_data = np.random.rand(100) * 10
```

```python
mu, variance = mle.normal(sample_data)
variance = int(round(variance,0)) # limitation to fix: variance needs to be integer currently
print(mu, variance)
```

    5.448299692053602 8

```python
#a standard normal r.v. with mean and variance as estimated above
x = st.Norm_rv(mu, variance)
x.plot_pdf()
x.probability_calc()
```

```python

```