# Best Approximation in Hilbert Spaces

```python
%matplotlib inline
import sympy as sym
import pylab as pl
import numpy as np
import numpy.polynomial.polynomial as n_poly
import numpy.polynomial.legendre as leg
```

## Mindflow

We want the best approximation (in Hilbert Spaces) of the function $f$, on the space $V = \mathrm{span}\{v_i\}$. Remember that $p\in V$ is best approximation of $f$ if and only if:

$$
(p-f,q)=0, \quad \forall q\in V.
$$

Focus one second on the fact that both $p$ and $q$ belong to $V$. We know that any $q$ can be expressed as a linear combination of the basis functions $v_i$:

$$
(p-f,v_i)=0, \quad \forall v_i\in V.
$$

Moreover $p$ is uniquely defined by the coefficients $p^j$ such that $p = p^j\,v_j$. Collecting this information together we get:

$$
(v_j,v_i) p^j = (f,v_i),\quad \forall v_i\in V.
$$

Now that we know our goal (finding these $p^j$ coefficients) we do what the rangers do: we explore!

We understand that we will need to invert the matrix:

$$
M_{ij} = (v_j,v_i) = \int v_i\cdot v_j
$$

What happens if we choose basis functions such that $(v_j,v_i) =  \delta_{ij}$?

How to construct numerical techniques to evaluate integrals in an efficient way?

Evaluate the $L^2$ projection.

## Orthogonal Polynomials

Grham Schmidt

$$
p_0(x) = 1, \qquad p_k(x) = x^k - \sum_{j=0}^{k-1} \frac{(x^k,p_j(x))}{(p_j(x),p_j(x))} p_j(x)
$$

or, alternatively

$$
p_0(x) = 1, \qquad p_k(x) = x\,p_{k-1}(x) - \sum_{j=0}^{k-1} \frac{(x p_{k-1}(x),p_j(x))}{(p_j(x),p_j(x))} p_j(x)
$$

```python
def scalar_prod(p0,p1,a=0,b=1):
    assert len(p0.free_symbols) <= 1, "I can only do this for single variable functions..."
    t = p0.free_symbols.pop() if len(p0.free_symbols) == 1 else sym.symbols('t')
    return sym.integrate(p0*p1,(t,a,b))
```

```python
t = sym.symbols('t')
#k = 3

Pk = [1+0*t] # Force it to be a sympy expression

for k in range(1,5):
    s = 0
    for j in range(0,k):
        s+= scalar_prod(t**k,Pk[j])/scalar_prod(Pk[j],Pk[j])*Pk[j]
    pk = t**k-s
    # pk = pk/sym.sqrt(scalar_prod(pk,pk))
    pk = pk/pk.subs(t,1.)
    Pk.append(pk)

M = []
for i in range(len(Pk)):
    row = []
    for j in range(len(Pk)):
        row.append(scalar_prod(Pk[i],Pk[j]))
    M.append(row)

M = sym.Matrix(M)

print(M)
print Pk

x = np.linspace(0,1,2**5 + 1)

print x

for p in Pk:
    if p != 1 :
        fs = sym.lambdify(t, p, 'numpy')
        #print x.shape
        #print fs(x)
        _ = pl.plot(x,fs(x))
```

## Theorem

Le $q$ be nonzero polynomial of degree $n+1$ and $\omega(x)$ a positive weight function, s. t.:

$$
\int_a^b x^k q(x)\, \omega(x) = 0, \quad k = 0,\ldots, n
$$

If $x_i$ are zeros of $q(x)$, then:

$$
\int_a^b f(x)\, \omega(x)\approx \sum_{i=0}^nw_i\, f(x_i)
$$

with:

$$
w_i = \int_a^b l_i(x)\, \omega(x)
$$

is exact for all polynomials of degree at most $2n+1$. Here $l_i(x)$ are the usual Lagrange interpolation polynomials.

**Proof:** assume $f(x)$ is a polynomial of degree at most $2n+1$ and show:
$$
\int_a^b f(x)\, \omega(x) = \sum_{i=0}^nw_i\, f(x_i).
$$
Usign the polynomial division we have:
$$
\underbrace{f(x)}_{2n+1} = \underbrace{q(x)}_{n+1}\, \underbrace{p(x)}_{n} + \underbrace{r(x)}_{n}.
$$
By taking $x_i$ as zeros of $q(x)$ we have:
$$
f(x_i) = r(x_i)
$$
Now:
$$
\int_a^b f(x)\, \omega(x) = \int_a^b [q(x)\, p(x) + r(x)]\, \omega(x)
$$
$$
 = \underbrace{\int_a^b q(x)\, p(x) \, \omega(x)}_{=0} + \int_a^b r(x)\, \omega(x)
$$
Since $r(x)$ is a polynomial of order $n$ this is exact:

$$
\int_a^b f(x)\, \omega(x) = \int_a^b r(x)\, \omega(x) = \sum_{i=0}^nw_i\, r(x_i)
$$

But since we chose $x_i$ such that $f(x_i) = r(x_i)$, we have:

$$
\int_a^b f(x)\, \omega(x) = \int_a^b r(x)\, \omega(x) = \sum_{i=0}^nw_i\, f(x_i)
$$

This completes the proof.

## Legendre Polynomial

Two term recursion, to obtain the same orthogonal polynomials above (defined between [-1,1]), normalized to be one in $x=1$:

$$
(n+1) p^{n+1}(x) = (2n+1)\, x\, p^n(x) - n\, p^{n-1}(x)
$$

```python
Pn = [1.,t]

#Pn = [1.,x, ((2*n+1)*x*Pn[n] - n*Pn[n-1])/(n+1.) for n in range(1,2)]

for n in range(1,5):
    pn1 = ((2*n+1)*t*Pn[n] - n*Pn[n-1])/(n+1.)
    Pn.append(sym.simplify(pn1))

print(Pn)

#print(sym.poly(p))
#print(sym.real_roots(sym.poly(p)))
print(sym.integrate(Pn[4]*Pn[3],(t,-1,1)))

x = np.linspace(-1,1,100)
for p in Pn:
    if p != 1. :
        fs = sym.lambdify(t, p, 'numpy')
        #print x.shape
        #print fs(x)
        _ = pl.plot(x,fs(x))

```

In our proof we selected to evaluate $ r $ and $f$ in $x_i$ at the zeros of the legendre polynomials, this is why we need to evaluate the zeros of the polynomials.

```python
print(sym.real_roots(sym.poly(Pn[3])))

#q = [-1.]+sym.real_roots(sym.poly(Pn[2]))+[1.]
q = sym.real_roots(sym.poly(Pn[3]))
print(q)

for p in Pn:
    if p != 1. :
        #print(sym.poly(p))
        #print(sym.real_roots(sym.poly(p)))
        print(sym.nroots(sym.poly(p)))

```

    [-sqrt(15)/5, 0, sqrt(15)/5]
    [-sqrt(15)/5, 0, sqrt(15)/5]
    [0]
    [-0.577350269189626, 0.577350269189626]
    [-0.774596669241483, 0, 0.774596669241483]
    [-0.861136311594053, -0.339981043584856, 0.339981043584856, 0.861136311594053]
    [-0.906179845938664, -0.538469310105683, 0, 0.538469310105683, 0.906179845938664]

$$
w_i = \int_{-1}^{1} l_i(x)
$$

```python
Lg = [1. for i in range(len(q))]
print(Lg)

#for i in range(n+1):
for i in range(len(q)):
    for j in range(len(q)):
        if j != i:
            Lg[i] *= (t-q[j])/(q[i]-q[j])

print(Lg)

x = np.linspace(-1,1,100)

for poly in Lg:
    fs = sym.lambdify(t, poly, 'numpy')
    _ = pl.plot(x,fs(x))
```

```python
for poly in Lg:
    print(sym.integrate(poly,(t,-1,1)))
```

    0.555555555555555
    0.888888888888889
    0.555555555555555

### Hint
Proiezione usando polinomi LEGENDRE (f,v_i)

# Now let's get Numerical

From now on I work on the $[0,1]$ interval, becouse i like it this way :)

In the previus section we explored what sympbolically was happening, now we implement things on the computer. We saw how important are the legendre plynomials. Here a little documentation on that. I pont it out not because you need to read it all, but because I would like you get some aquitance with this criptic documentation pages [doc](https://docs.scipy.org/doc/numpy/reference/generated/numpy.polynomial.legendre.legroots.html#numpy.polynomial.legendre.legroots).

The problem we aim at solving is finding the coefficents $p_j$ such that:
$$
(v_j,v_i) p^j = (f,v_i),\quad \forall v_i\in V.
$$
Remind in this section the einstein notation holds.

We can expand the compact scalar product notation:
$$
p^j \int_0^1 v_i\, v_j = \int_0^1 f\, v_i,\quad \forall v_i\in V.
$$

We consider $V = \mathrm{span}\{l_i\}$. Our problem becomes:
$$
p^j \int_0^1 l_i\, l_j = \int_0^1 f\, l_i,\quad \mathrm{for}\ i = 0,\ldots,\mathtt{deg}
$$
Let's focus on mass matrix:
$$
\int_0^1 l_i(x)\, l_j(x) = \sum_k l_i(x_k)\, w_k\, l_j(x_k) =
$$

$$
=
\left(
\begin{array}{c c c c}
l_0(x_0) & l_0(x_1) & \ldots & l_0(x_q) \\
l_1(x_0) & l_1(x_1) & \ldots & l_1(x_q) \\
 & \ldots & \ldots  & \\
l_n(x_0) & l_n(x_1) & \ldots & l_n(x_q) \\
\end{array}
\right)
\left(
\begin{array}{c c c c}
w_0 &   0 & \ldots & 0 \\
  0 & w_1 & \ldots & 0 \\
   & \ldots & \ldots  & \\
  0 &   0 & \ldots & w_q \\
\end{array}
\right)
\left(
\begin{array}{c c c c}
l_0(x_0) & l_1(x_0) & \ldots & l_n(x_0) \\
l_0(x_1) & l_1(x_1) & \ldots & l_n(x_1) \\
 & \ldots & \ldots  & \\
l_0(x_q) & l_1(x_q) & \ldots & l_n(x_q) \\
\end{array}
\right)
= B\, W\, B^T
$$

A piece of curiosity, how the the two functions to find theros in two different ways

```python
print sym.nroots(sym.poly(Pn[-1]))
coeffs = np.zeros(6)
coeffs[-1] = 1.
print(leg.legroots(coeffs))
```

    [-0.906179845938664, -0.538469310105683, 0, 0.538469310105683, 0.906179845938664]
    [ -9.06179846e-01  -5.38469310e-01  -5.96500148e-17   5.38469310e-01
       9.06179846e-01]

```python
print gauss_points(3)
print(np.sqrt(3./5.)*.5)+.5
```

    [ 0.11270167  0.5         0.88729833]
    0.887298334621

```python
def define_lagrange_basis_set(q):
    n = q.shape[0]
    L = [n_poly.Polynomial.fromroots([xj for xj in q if xj != q[i]]) for i in range(n)]
    L = [L[i]/L[i](q[i]) for i in range(n)]
    return L
```

differenza fra le roots "simboliche" e non

```python
deg = 4
Nq = deg+1
p,w = leg.leggauss(Nq)
w = .5 * w
p = .5*(p+1)
#print p
#print w
W = np.diag(w)
#print W
```

```python
int_p = np.linspace(0,1,deg+1)
L = define_lagrange_basis_set(int_p)

print(len(L))

x = np.linspace(0,1,1025)
for f in L:
    _ = pl.plot(x, f(x))
_ = pl.plot(int_p, 0*int_p, 'ro')
```

```python
B = np.zeros((0,Nq))
for l in L:
    B = np.vstack([B,l(p)])
```

Recall:
$$
B\, W\, B^T p = B W f
$$

$$
B\, W\, B^T =
\left(
\begin{array}{c c c c}
l_0(x_0) & l_0(x_1) & \ldots & l_0(x_q) \\
l_1(x_0) & l_1(x_1) & \ldots & l_1(x_q) \\
 &  & \ddots  & \\
l_n(x_0) & l_n(x_1) & \ldots & l_n(x_q) \\
\end{array}
\right)
\left(
\begin{array}{c c c c}
w_0 &   0 & \ldots & 0 \\
  0 & w_1 & \ldots & 0 \\
   &  & \ddots  & \\
  0 &   0 & \ldots & w_q \\
\end{array}
\right)
\left(
\begin{array}{c c c c}
l_0(x_0) & l_1(x_0) & \ldots & l_n(x_0) \\
l_0(x_1) & l_1(x_1) & \ldots & l_n(x_1) \\
 & & \ddots  & \\
l_0(x_q) & l_1(x_q) & \ldots & l_n(x_q) \\
\end{array}
\right)
$$

```python
print(B.shape)
_ = pl.plot(B.T)
M = B.dot(W.dot(B.T))
print np.linalg.matrix_rank(M)
print np.linalg.cond(M)
```

```python
def step_function():
    def sf(x):
        index = where((x>.3) & (x<.7))
        step = zeros(x.shape)
        step[index] = 1
        return step
    return lambda x : sf(x)
```

$$
B\, W\, f =
\left(
\begin{array}{c c c c}
l_0(x_0) & l_0(x_1) & \ldots & l_0(x_q) \\
l_1(x_0) & l_1(x_1) & \ldots & l_1(x_q) \\
 & \ldots & \ldots  & \\
l_n(x_0) & l_n(x_1) & \ldots & l_n(x_q) \\
\end{array}
\right)
\left(
\begin{array}{c c c c}
w_0 &   0 & \ldots & 0 \\
  0 & w_1 & \ldots & 0 \\
   & \ldots & \ldots  & \\
  0 &   0 & \ldots & w_q \\
\end{array}
\right)
\left(
\begin{array}{c}
f(x_0) \\
f(x_1) \\
\vdots\\
f(x_q) \\
\end{array}
\right)
$$

```python
g = lambda x: np.sin(2*np.pi*x)
#g = step_function()
p = p.reshape((p.shape[0],1))

G = g(p)
print G.shape
print B.shape
print W.shape
G = B.dot(W.dot(G))
```

    (5, 1)
    (5, 5)
    (5, 5)

```python
u = np.linalg.solve(M, G)
print u
```

    [[ -1.92161045e-01]
     [  9.79052672e-01]
     [ -1.35712301e-15]
     [ -9.79052672e-01]
     [  1.92161045e-01]]

```python
def get_interpolating_function(LL,ui):
    def func(LL,ui,x):
        acc = 0
        for L,u in zip(LL,ui):
            #print(L,u)
            acc+=u*L(x)
        return acc
    return lambda x : func(LL,ui,x)
```

```python
I = get_interpolating_function(L,u)
sampling = np.linspace(0,1,101)
_= pl.plot(sampling, I(sampling))
#plot(xp, G,'ro')
```

## Diference in between projection and interpolation runge example

Proiezione usando polinomi LEGENDRE (f,v_i) con quadratura con 18 punti

Interpolazione usando polinomi LAGRANGE (sui punti di quadratura che sono i punti di gauss della funzione sopra)

```python

```


# Polynomial interpolation and data-fitting

Given a set of * **equally spaced** point* $t_1, t_2, \dots, t_m$, and *data* $y_1, y_2, \dots, y_m$, determine a polynomial of degree $m-1$: $p(t) = x_1 + x_2 t + x_3 t^2 + \cdots + x_m t^{m-1}$, such that

$$p(t_i) = y_i, i = 1, 2, \dots , m$$

This problem can be written in terms of solving the following system of linear equations

$$\left\{
\begin{array}{ccccccccccc}
x_1 & + & x_2 t_1 & + & x_3 t_1^2 & + & \cdots  & + & x_m t_1 ^{m-1} & = & y_1 \\
x_1 & + & x_2 t_2 & + & x_3 t_2^2 & + & \cdots  & + & x_m t_2 ^{m-1} & = & y_2 \\
x_1 & + & x_2 t_3 & + & x_3 t_3^2 & + & \cdots  & + & x_m t_3 ^{m-1} & = & y_3 \\
&&&&\vdots& &&&&=& \vdots \\
x_1 & + & x_2 t_m & + & x_3 t_m^2 & + & \cdots  & + & x_m t_m ^{m-1} & = & y_m \\
\end{array}
\right .$$

or in matrix form

$$\left[
\begin{array}{ccccc}
1 & t_1 & t_1^2 & \cdots & t_1^{m-1} \\
1 & t_2 & t_2^2 & \cdots & t_2^{m-1} \\
1 & t_3 & t_3^2 & \cdots & t_3^{m-1} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
1 & t_m & t_m^2 & \cdots & t_m^{m-1}
\end{array}
\right]\left[
\begin{array}{c}
x_1 \\
x_2 \\
x_3 \\
\vdots \\
x_m
\end{array}
\right]
=\left[
\begin{array}{c}
y_1 \\
y_2 \\
y_3 \\
\vdots \\
y_m
\end{array}
\right]
$$

Here $(t_i,y_i)$ are all given for $i = 1,2,\dots,m$. Only the polynomial coefficients $x_1, x_2, \dots, x_m$ are unknown. The matrix above is obviously a Vandermonde matrix, which is invertible as long as $t_i \neq t_j$, for $i \neq j$. It also implies the uniqueness of the solution of the system.

However, when there're many points to fit, the interpolation polynomial $p(t)$ will have a high degree, and become a ill conditioned problems.

And to fix this, we can seek a polynomial of degree much less than $m-1$, like $n-1$ with $n\ll m$

|$$m=20,n=20$$|$$m=20,n=10$$|
|:-:|:-:|
|||

But it is impossible for a polynomial of degree $n-1$ to satisfy $m$ interpolation conditinos, since to satisfy such equations we need to solve:

$$\left[
\begin{array}{ccccc}
1 & t_1 & t_1^2 & \cdots & t_1^{n-1} \\
1 & t_2 & t_2^2 & \cdots & t_2^{n-1} \\
1 & t_3 & t_3^2 & \cdots & t_3^{n-1} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
1 & t_m & t_m^2 & \cdots & t_m^{n-1}
\end{array}
\right]\left[
\begin{array}{c}
x_1 \\
x_2 \\
x_3 \\
\vdots \\
x_n
\end{array}
\right]
=\left[
\begin{array}{c}
y_1 \\
y_2 \\
y_3 \\
\vdots \\
y_m
\end{array}
\right]
$$

which may be without a solution. What to do now?$\newcommand{\ffrac}{\displaystyle\frac}$

We don't require the $p(t)$ of degree $n-1$ to interpolate exactly given $m$ points, rather, we need a $p(t)$ such that the difference $\displaystyle \sum_{i=1}^{m} \left|p(t_i) - y_i \right|^2$ is minimized. This is the **linear least squares problem**. In other words, we need to find the polynomial with coefficients $x_1, x_2, \dots, x_n$, that minimize the residual of the system of linear equations. After that $x_1, x_2, \dots, x_n$ are called the **linear squares solution**.

# Linear least squares problem

Given matrix $A_{m \times n}$, $m \geq n$, full rank, and a vector $\vec{b} \in \mathbb{R}^{m}$, so generally there's no solution for equation $A\vec{x} = \vec{b}$. So instead we consider the following linear least squares problem:

$$\min_{\vec{x} \in \mathbb{R}^n}\|\vec{b} - A\vec{x}\|_2^2$$

the solution $\vec{x}^*$ would minimize $\|\vec{b} - A\vec{x}\|_2$. Here's a picture to get the initial idea:

So the residual $\vec{r} = \vec{b} - A \vec{x}^*$ must be perpendicular to the $\DeclareMathOperator*{\range}{range}\DeclareMathOperator*{\rank}{rank} \range(A)$

$Theorem$ 1

Given an $m \times n$ matrix $A$, with $m > n$ and $\rank(A) = n$, and another vector $\vec{b}\in \mathbb{R}^{m}$, then a vector $\vec{x}^* \in \mathbb{R}^n$ minimizes $\|\vec{b} - A\vec{x}\|_2^2$, thereby solving the least squares problem, $iff$ one of these three equivalent conditions are satisfied
1. $\vec{b} - A \vec{x}^* \perp \range(A)$, that is $A^{\mathrm{T}}\left( \vec{b} - A \vec{x}^* \right) = \vec{0} $
2. $A^{\mathrm{T}}A\vec{x}^* = A^{\mathrm{T}}\vec{b}$
3. $A \vec{x}^* $ is the orthogonal projection of $\vec{b}$ onto $\range(A)$.

$Proof$

- First equivalence

>First we prove: *condition 1* $\Longrightarrow \vec{x}^*$ minimizes $\|\vec{b} - A\vec{x}\|_2^2$

>We choose one vector in $\range(A)$, which is $A(\vec{x}^* - \vec{x})$, here $\vec{x}$ is any vector in $\mathbb{R}^n$. So that follow the *condition 1* we have $\vec{b} - A \vec{x}^* \perp A(\vec{x}^* - \vec{x})$. So that we have

>$$\begin{align}
&\|\vec{b} - A\vec{x}\|_2^2 = \left\|\vec{b} - A\vec{x}^* + A\left( \vec{x}^* - \vec{x} \right)\right\|_2^2 \\ \stackrel{\;\;\vec{b} - A \vec{x}^* \perp \,A(\vec{x}^* - \vec{x})}{=} \;\;&\|\vec{b} - A\vec{x}^*\|_2^2 + \left\| A\left( \vec{x}^* - \vec{x} \right) \right\|_2^2 \geq \|\vec{b} - A\vec{x}^*\|_2^2
\end{align}$$

>Therefore, $\vec{x}^*$ minimizes $\|\vec{b} - A\vec{x}\|_2^2$

>Then we prove: *condition 1* $\Longleftarrow \vec{x}^*$ minimizes $\|\vec{b} - A\vec{x}\|_2^2$

>Assume not, then we can find a different vector $\vec{y} \in \mathbb{R}^n$ such that $\vec{b} - A \vec{y}$ is orthogonal to $\range(A)$, by solving the equation:

>$$A\vec{y} = P \vec{b}$$

>where $P\vec{b}$ represents the orthogonal projection of $\vec{b}$ to $\range(A)$. So that

>$$\begin{array}{rl}
& \vec{b} - A \vec{y} = \vec{b} - P \vec{b} \perp \range(A) \\[0.5em]
\Rightarrow & \vec{b} - A \vec{y} \neq \vec{b} - A \vec{x}^*  \\[0.5em]
\Rightarrow & A\left( \vec{x}^* - \vec{y} \right) \neq \vec{0}  \\[0.6em]
\Rightarrow & \begin{align*}
\left\|\vec{b} - A\vec{x}^*\right\|_2^2 &= \left\|\vec{b} - A\vec{y} - A\left( \vec{x}^* - \vec{y} \right) \right\| _2^2 \\
&= \left\|\vec{b} - A\vec{y}\right\|_2^2 + \left\|A\left( \vec{x}^* - \vec{y} \right)\right\|_2^2 \\
&> \left\|\vec{b} - A\vec{y}\right\|_2^2
\end{align*}
\end{array}$$

>This contradiction shows that $\vec{b} - A \vec{x}^*$ must be orthogonal $\range(A)$.

- First and Second Equivalence

>It's trivial that they are equivalent:

>$$A^{\mathrm{T}}A\vec{x}^* = A^{\mathrm{T}}\vec{b} \Longleftrightarrow A^{\mathrm{T}}\left( \vec{b} - A \vec{x}^* \right) = \vec{0}
$$

- Second and Third Equivalence

> Still we can prove that they are equivalent:

> $$\begin{array}{rrl}
& A^{\mathrm{T}}A\vec{x}^* \!\!\!\!\!&= A^{\mathrm{T}}\vec{b} \\[0.4em]
\Leftrightarrow \!\!\!& \vec{x}^* \!\!\!\!\!&= \left( A^{\mathrm{T}}A \right) A^{\mathrm{T}}\vec{b} \\[0.4em]
\Leftrightarrow \!\!\!& A\vec{x}^* \!\!\!\!\!&= A\left( A^{\mathrm{T}}A \right) A^{\mathrm{T}}\vec{b}
\end{array}$$

> Since $A$ is full rank, so that the inverse exists. And we've already known that $A\left( A^{\mathrm{T}}A \right) A^{\mathrm{T}}\vec{b}$ is just the orthogonal projection of $\vec{b}$ onto $\range(A)$.

# Conditioning of linear least squares problems
The condition number of solving a linear system $A\vec{x} = \vec{b}$ is $\DeclareMathOperator*{\Cond}{Cond} \Cond(A) = \big\|A\big\|_2\left\|A^{-1}\right\|_2$, when $A$ is a square matrix.

So what's the condition number of solving the least squares problem $\displaystyle \min_{\vec{x}\in\mathbb{R}^n} \left\| \vec{b} - A \vec{x} \right\|_2$ for a given $m \times n$ matrix $A$, with $m > n$ and $\rank(A) = n$. From the **theorem 1**, we can see that the solution would be

$$\vec{x}^* = \left( A^{\mathrm{T}}A \right)^{-1} A^{\mathrm{T}} \vec{b}$$

where $A$ is column full rank matrix so that $A^{\mathrm{T}}A$ is invertible. To complete the explanation, now we define the **pseudoinverse** of $A$ as $A^+:=\left( A^{\mathrm{T}}A \right)^{-1}A^{\mathrm{T}}$. So that now we can shorten the expression like

$$\vec{x}^* = \left( A^{\mathrm{T}}A \right)^{-1} A^{\mathrm{T}} \vec{b} = A^+ \vec{b}$$

And in this situation the condition number will be

$$\Cond(A) = \big\|A\big\|_2\left\|A^{+}\right\|_2$$

There's another point: the conditioning of projecting $\vec{b}$ to $\range(A)$.

Define $\theta  := \small\langle \vec{b},\range(A) \small\rangle$ as the angle between $\vec{b}$ and its projection on $\range(A)$, ranging from $-\ffrac{\pi} {2}$ to $\ffrac{\pi} {2}$.
- If $\theta \approx 0$, then a small relative change in $\vec{b}$ will just lead to a similarly small relative change in $P\vec{b}$, well-conditioned!
- If $\theta \approx \ffrac{\pi} {2}$, then a small relative change in $\vec{b}$ will just lead to a relatively large change in $P\vec{b}$, ill-conditioned!

Based on the above obeservation, we have the following theorem

$Thorem$ 2

The condition number of solving the linear least squares problem is bounded by

$$\frac{\Cond(A)} {\cos \theta}$$

$Proof$

$$\begin{cases}
\;\vec{x} = A^{+} \vec{b} \\
\left( \vec{x} + \Delta \vec{x} \right) = A^{+} \left( \vec{b} + \Delta \vec{b} \right) \\
\end{cases} \Longrightarrow \Delta \vec{x} = A^{+} \Delta \vec{b}$$

So that the condition number of solution $\vec{x}$ with respect to the perturbations in $\vec{b}$ is

$$\begin{align}
\kappa_{\vec{b} \to \vec{x}} &= \sup_{\Delta \vec{b} \neq \vec{0}} \frac{\left\| \Delta \vec{x} \right\| / \left\| \vec{x} \right\|} {\left\| \Delta \vec{b} \right\|\big/\left\| \vec{b} \right\|} \\
&= \sup_{\Delta \vec{b} \neq \vec{0}} \frac{\left\| A^{+} \cdot \Delta \vec{b} \right\| \big/ \left\| \vec{x} \right\|} {\left\| \Delta \vec{b} \right\|\big/\left\| \vec{b} \right\|} \\
&= \frac{\left\| A^{+} \right\| \cdot \left\| \vec{b} \right\|} {\left\| \vec{x} \right\|} = \left\| A^{+} \right\| \cdot \frac{\left\| \vec{b} \right\|} {\left\| P\vec{b} \right\|} \cdot \frac{ \left\| P\vec{b} \right\|} {\left\| \vec{x} \right\|} \\
&= \left\| A^{+} \right\| \cdot \frac{1} {\cos \theta} \cdot \frac{ \left\| A\vec{x} \right\|} {\left\| \vec{x} \right\|} \\
&\leq \frac{\left\| A^{+} \right\| \cdot \left\| A \right\|} {\cos \theta} = \frac{\Cond(A)} {\cos \theta}
\end{align}$$

# Algorithms for solving linear least squares problems
## Solve normal equation
From $Theorem$ 1, we know that the solution to the least squares problem satisfies the normal equation

$$A^{\mathrm{T}}A \vec{x} = A^{\mathrm{T}}\vec{b}$$

And since $A$ is full rank, so that $A^{\mathrm{T}}A$ is symmetric positive definite, and it has a Cholesky Factorization

$$A^{\mathrm{T}}A = LL^{\mathrm{T}}$$

1. Form $A^{\mathrm{T}}A$ and $A^{\mathrm{T}}\vec{b}$
2. Factorize $A^{\mathrm{T}}A = LL^{\mathrm{T}}$, using the build-in function `chol`
3. Solve $LL^{\mathrm{T}} \vec{x} = A^{\mathrm{T}} \vec{b}$, by backward/forward substitution

## By QR factorization
Denote the reduced $QR$ factorization of $A$ by $A = \hat{Q} \hat{R}$, where $\hat{Q}$ is a $m \times n$ matrix with orthonormal columns, and $\hat{R}$ is a $n \times n$ upper triangular matrix. Then, from $A^{\mathrm{T}}A \vec{x} = A^{\mathrm{T}}\vec{b}$, so that

$$\hat{R}^{\mathrm{T}} \hat{Q}^{\mathrm{T}} \hat{Q} \hat{R} \vec{x} = \hat{R}^{\mathrm{T}} \hat{Q}^{\mathrm{T}} \vec{b}$$

which implies

$$\hat{R} \vec{x} = \hat{Q}^{\mathrm{T}} \vec{b}$$

1. Compute the $QR$ factorization $A = \hat{Q} \hat{R}$, by Housholder/Modified Gram-Schmidt
2. Computer $\hat{Q}^{\mathrm{T}} \vec{b}$
3. Solve $\hat{R} \vec{x} = \hat{Q}^{\mathrm{T}} \vec{b}$, by backward substitution
***

Another way to derive the equation: Since $A\vec{x}$ is the orthogonal projection of $\vec{b}$ to the $\range(A)$:

$$A\vec{x} = \hat{Q}\hat{Q}^{\mathrm{T}} \vec{b} \Longrightarrow \hat{R} \vec{x} = \hat{Q}^{\mathrm{T}} \vec{b}$$

## By singular value decomposition
The reduced singular value decomposition of a full rank $m \times n$ matrix $A$ can be written as

$$A = \hat{U} \hat{\Sigma} V^{\mathrm{T}}$$

where $\hat{U}$ is an $m \times n$ matrix with orthonormal columns, $\hat{\Sigma}$ is an $n \times n$ diagonal matrix with singular values of $A$ on its diagonal, and $V$ is an $n \times n$ orthogonal matrix. So that

$$
A^{\mathrm{T}}A\vec{x} = V\Sigma^{\mathrm{T}}\hat{U}^{\mathrm{T}}\hat{U} \hat{\Sigma} V^{\mathrm{T}}\vec{x} = V \Sigma\hat{U}^{\mathrm{T}} \vec{b} = A^{\mathrm{T}}\vec{b}
\\[0.8em]
\vec{x} = {V^{\mathrm{T}}}^{-1} \hat{\Sigma}^{-1} \hat{U}^{-1} \vec{b} = V \hat{\Sigma}^{-1} \hat{U}^{\mathrm{T}} \vec{b}
$$

1. Compute reduced SVD $A = \hat{U} \hat{\Sigma} V^{\mathrm{T}}$
2. $\vec{x} = V \hat{\Sigma}^{-1} \hat{U}^{\mathrm{T}} \vec{b}$

# Stability of solving least squares problems
Here using $QR$ factorization is more stable than directly solve the linear equation. Let's see from an example.

>**e.g.***
>
>Fix $e^{\sin(4t)}$ from $0$ to $1$; we'll discretize the interval by $100$ points and use a polynomial with degree $14$. The function now looks like
>
>$$\left[
\begin{array}{ccccc}
1 & x_1 & x_1^2 & \cdots & x_1^{14} \\
1 & x_2 & x_2^2 & \cdots & x_2^{14} \\
1 & x_3 & x_3^2 & \cdots & x_3^{14} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
1 & x_{100} & x_{100}^2 & \cdots & x_{100}^{14}
\end{array}
\right]\left[
\begin{array}{c}
a_1 \\
a_2 \\
a_3 \\
\vdots \\
a_{15}
\end{array}
\right]
=c \cdot \left[
\begin{array}{c}
e^{\sin(4x_1)} \\
e^{\sin(4x_2)} \\
e^{\sin(4x_3)} \\
\vdots \\
e^{\sin(4x_{100})}
\end{array}
\right]
$$

>Here, $c$ is used to scale $a_{15}$ to $1$, as a checking solution, $c = 1/2006.787678808116$. In the last case, where the normal equation is solved, $a_{15} = 0.24664$, completely different from $1$.

$Conclusion$

For [method 1](#Solve-normal-equation), $\Cond(A^{\mathrm{T}}A) = \big\|A^{\mathrm{T}}A\big\|_2\left\|\left(A^{\mathrm{T}}A\right)^{-1}\right\|_2$; then

$$\begin{align}
\big\|A^{\mathrm{T}}A\big\|_2 &= \sup_{\vec{x} \neq \vec{0}}\frac{\big\|A^{\mathrm{T}}A\vec{x}\big\|_2} {\left\|\vec{x}\right\|_2} = \lambda_{\text{max}}^2 \\
\big\|\left(A^{\mathrm{T}}A\right)^{-1}\big\|_2 &= \frac{1} {\lambda_{\text{min}}^2}\\
\Longrightarrow \Cond(A^{\mathrm{T}}A) &= \frac{\lambda_{\text{max}}^2} {\lambda_{\text{min}}^2}
\end{align}$$

So that actually $\Cond(A^{\mathrm{T}}A) = \left( \Cond(A) \right)^2$

For [method 2](#By-QR-factorization), $\Cond(A) = \Cond(\hat{R})$.

$$\begin{align}
\Cond(A) &= \Cond(\hat{Q}\hat{R}) \\
&= \left\| \hat{Q}\hat{R} \right\| \cdot \left\| \left( \hat{Q}\hat{R} \right)^{+} \right\| \\
&= \sup_{\vec{x} \neq \vec{0}} \frac{\left\| \hat{Q}\hat{R}\vec{x} \right\|} {\left\| \vec{x} \right\|} \cdot \sup_{\vec{x} \neq \vec{0}} \frac{\left\| \left( \hat{Q}\hat{R} \right)^{+}\vec{x} \right\|} {\left\| \vec{x} \right\|} \\
&= \left\| \hat{R} \right\| \cdot \left\| \left( \hat{R} \right)^{+} \right\| \\
&= \Cond(\hat{R})
\end{align}$$

$Theorem$ 3

Let the full rank least squares problem $\min_{\vec{x} \in \mathbb{R}^n}\|\vec{b} - A\vec{x}\|_2^2$ be solved by $QR$ factorization with Householder triangularization. This algorithm is stable in the sense that the computed solution $\tilde{\vec{x}}$ is the minimizer of a nearby least squares problem, $i.e.$, $\tilde{\vec{x}}$ minimizes $\left\|\vec{b} - \left(A + \delta A \right)\vec{x}\right\|_2^2$ , where $\left\| \delta A \right\|_2 = \| A \|_2 O(\epsilon_{\text{machine}} )$.

$Theorem$ 4

The solution of the full rank least squares problem $\min_{\vec{x} \in \mathbb{R}^n}\|\vec{b} - A\vec{x}\|_2^2$ via solving the normal equation $A^{\mathrm{T}}A \vec{x} = A^{\mathrm{T}}\vec{b}$ is unstable.

Computational/Intro to Numerical Computing/Note_Chap07_Solving Linear Least Squares Problems.ipynb


# Implementemos las reglas de cuadratura

```python
def cuad_pmedio(a, b, f):
    return (b-a)*f((a+b)/2)
```

```python

```

```python

```

```python

```

```python
def cuad_pmedio(a, b, f):
    """Implementación de la regla del punto medio

    Parameters
    ----------
    f: La función a integrar
    a: Límite inferior del intervalo
    b: Límite superior del intervalo

    Returns
    -------
    aprox: Aproximación de la integral por la regla del punto medio

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    """
    if a > b:
        raise ValueError("Oops!  Debe ser a<b")

    try:
        x0 = (a+b)/2
        h = f(x0)
        aprox = h*(b-a)
    except:
        print('Error: no fue posible calcular la función')
    return aprox
```

```python
def f(x):
    return x**2
```

```python
f = 9
```

```python

```

```python
cuad_pmedio(0, 1, f)
```

```python

```

```python
def cuad_trapecio(a, b, f):
    """Implementación de la regla del trapecio

    Parameters
    ----------
    f: La función a integrar
    a: Límite inferior del intervalo
    b: Límite superior del intervalo

    Returns
    -------
    aprox: Aproximación de la integral por la regla del trapecio

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    """
    if a > b:
        raise ValueError("Oops!  Debe ser a<b")
        return None
    try:
        h = f(a) + f(b)
        aprox = (b-a)/2*h
    except:
        print('Error: no fue posible calcular la función')
    return aprox
```

```python
def cuad_simpson(a, b, f):
    """Implementación de la regla de Simpson

    Parameters
    ----------
    f: La función a integrar
    a: Límite inferior del intervalo
    b: Límite superior del intervalo

    Returns
    -------
    aprox: Aproximación de la integral por la regla de Simpson

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    """
    if a > b:
        raise ValueError("Oops!  Debe ser a<b")
        return None
    try:
        x0 = (a+b)/2
        h = f(a) + f(b) + 4*f(x0)
        aprox = (b-a)/6*h
    except:
        print('Error: no fue posible calcular la función')
    return aprox
```

_____
TESTING

```python
def f(x):
    return x**2

I = cuad_pmedio(1, 2, f)
print(f'La regla del punto medio da como resultado: {I}')

I = cuad_trapecio(1, 2, f)
print(f'La regla del trapecio da como resultado: {I}')

I = cuad_simpson(1, 2, f)
print(f'La regla de simpson da como resultado: {I}')
```

    La regla del punto medio da como resultado: 2.25
    La regla del trapecio da como resultado: 2.5
    La regla de simpson da como resultado: 2.333333333333333

```python
7/3
```

    2.3333333333333335

Ahora vamos a preparar las funciones para que acepten la función o los valores

```python
def cuad_pmedio(a, b, y0):
    """Implementación de la regla del punto medio

    Parameters
    ----------
    y0: f evaluada en x0=(a+b)/2
    a: Límite inferior del intervalo
    b: Límite superior del intervalo

    Returns
    -------
    aprox: Aproximación de la integral por la regla del punto medio

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    """
    try:
        x0 = (a+b)/2
        aprox = x0*y0
    except:
        print('Error: no fue posible calcular la función')
    return aprox
```

```python
cuad_pmedio(0, 1, 0.5)
```

    0.25

```python
x = None
```

```python
x is not None
```

    False

¿Qué pasa si tenemos la posibilidad de tener la función o los datos?

```python
def cuad_pmedio(a, b, G=None, y0=None):
    """Implementación de la regla del punto medio

    Parameters
    ----------
    a: float
       Límite inferior del intervalo
    b: float
       Límite superior del intervalo
    G: function (1 parameter)
       La función a integrar
    y0: float
       El valor de y en el punto medio.

    Returns
    -------
    aprox: Aproximación de la integral por la regla del punto medio

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    """
    if a > b:
        raise ValueError("Oops!  Debe ser a<b")

    x0 = (a+b)/2
    if (G is None) and (y0 is not None):
            aprox = x0*y0
    elif (G is not None) and (y0 is None):
        try:
            h = G(x0)
        except:
            print(('Error: no fue posible calcular la función'
                   ' Si desea ingresar un dato use y0='))
        aprox = h*(b-a)

    else:
        raise ValueError("Debe ingresar la función o los datos!")

    return aprox
```

```python
cuad_pmedio(0, 1, y0=0.5)
```

    0.25

```python
cuad_pmedio(0, 1, G=f)
```

    0.25

```python

```

```python
def cuad_trapecio(x0, x1, f=None, y0=None, y1=None):
    """Implementación de la regla del trapecio

    Parameters
    ----------
    x0: float
       Límite inferior del intervalo
    x1: float
       Límite superior del intervalo
    f: function (1 parameter)
       La función a integrar
    y0: float
       El valor de y en el punto medio.
    y1: float
       El valor de y en el punto medio.

    Returns
    -------
    aprox: Aproximación de la integral por la regla del punto medio

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    Uso:
        cuad_trapecio(x0, x1, f=f)
        cuad_trapecio(x0, x1, y0=f(x0), y1=f(x1))
    """
    if x0 > x1:
        raise ValueError("Oops!  Debe ser a<b")

    if (f is None) and (y0 is not None) and (y1 is not None):
            aprox = (x1-x0)*(y0+y1)/2
    elif (f is not None) and (y0 is None):
        try:
            y0 = f(x0)
            y1 = f(x1)
        except:
            print(('Error: no fue posible calcular la función'
                   ' Si desea ingresar un dato use y0='))
        aprox = (x1-x0)*(y0+y1)/2

    else:
        raise ValueError("Debe ingresar la función o los datos!")

    return aprox
```

```python
cuad_trapecio(0, 1, f)
```

    0.5

```python
cuad_trapecio(0, 1, f=f, y0=f(0), y1=f(1))
```

#### Ahora veamos si podemos hacer que la funcion "se de cuenta" de si le pasamos la funcion o los datos

```python
def contar_argumentos(func):
    def inner(*args, **kwargs):
        nargs_in = len(args) + len(kwargs)
        return func(*args, **kwargs, nargs_in=nargs_in)
    return inner
```

```python

```

```python
@contar_argumentos
def cuad_trapecio(x0, x1, f=None, y0=None, y1=None, nargs_in=None):
    """Implementación de la regla del trapecio

    Parameters
    ----------
    x0: float
       Límite inferior del intervalo
    x1: float
       Límite superior del intervalo
    f: function (1 parameter)
       La función a integrar
    y0: float
       El valor de y en el punto medio.
    y1: float
       El valor de y en el punto medio.

    Returns
    -------
    aprox: Aproximación de la integral por la regla del punto medio

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    Uso:
        cuad_trapecio(x0, x1, f=f)
        cuad_trapecio(x0, x1, y0=f(x0), y1=f(x1))
    """
    if nargs_in==4:
        y1=y0
        y0=f
        f = None
    elif nargs_in==3:
        if type(f) is float:
            raise ValueError("Verificar los argumentos")
    else:
        raise ValueError("Verificar el número de argumentos")

    if x0 > x1:
        raise ValueError("Oops!  Debe ser a<b")

    if (f is None) and (y0 is not None) and (y1 is not None):
            aprox = (x1-x0)*(y0+y1)/2
    elif (f is not None) and (y0 is None):
        try:
            y0 = f(x0)
            y1 = f(x1)
        except:
            print(('Error: no fue posible calcular la función'
                   ' Si desea ingresar un dato use y0='))
        aprox = (x1-x0)*(y0+y1)/2

    else:
        raise ValueError("Debe ingresar la función o los datos!")

    return aprox
```

```python
cuad_trapecio(0, 1, f)
```

    0.5

```python
cuad_trapecio(0, 1, f(0), f(1))
```

    0.5

```python
@contar_argumentos
def cuad_simpson(x0, x2, f=None, y0=None, y1=None, y2=None, nargs_in=None):
    """Implementación de la regla de simpson

    Parameters
    ----------
    x0: float
       Límite inferior del intervalo
    x2: float
       Límite superior del intervalo
    f: function (1 parameter)
       La función a integrar
    y0: float
       El valor de y en el punto medio.
    y2: float
       El valor de y en el punto medio.

    Returns
    -------
    aprox: Aproximación de la integral por la regla de Simpson

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    Uso:
        cuad_simpson(x0, x2, f=f)
        cuad_simpson(x0, x2, y0=f(x0), y2=f(x2))
        cuad_simpson(x0, x2, f)
        cuad_simpson(x0, x2, y0, y2)
    """

    if nargs_in==5:
        y2=y1
        y1=y0
        y0=f
        f = None
    elif nargs_in==3:
        if type(f) is float:
            raise ValueError("Verificar los argumentos")
    else:
        raise ValueError("Verificar el número de argumentos")

    if x0 > x2:
        raise ValueError("Oops!  Debe ser a<b")

    x1 = (x0+x2)/2

    if (f is None) and (y0 is not None) and (y1 is not None):
            aprox = (x2-x0)/6 * (y0 + 4*y1 + y2)
    elif (f is not None) and (y0 is None):
        try:
            y0 = f(x0)
            y1 = f(x1)
            y2 = f(x2)
        except:
            print(('Error: no fue posible calcular la función'
                   ' Si desea ingresar un dato use y0='))
        aprox = (x2-x0)/6 * (y0 + 4*y1 + y2)

    else:
        raise ValueError("Debe ingresar la función o los datos!")

    return aprox
```

```python
cuad_simpson(0, 1, f)
```

    0.3333333333333333

```python
cuad_simpson(0, 1, f(0), f(0.5), f(1))
```

    0.3333333333333333

```python

```

______
# Reglas compuestas

```python
import numpy as np
x = np.linspace(0, 10, 11)
```

```python
x
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.])

```python
np.diff(x)
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

```python
def cuad_simpson_compuesta(x, f=None, y=None):
    """Implementación de la regla de simpson

    Parameters
    ----------
    x: list or array
       Lista de valores de x
    f: function (1 parameter)
       La función a integrar
    y: list or array
       La lista de valores de y

    Returns
    -------
    aprox: Aproximación de la integral por la regla de Simpson

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    Uso:
        cuad_simpson(x, y=y)
        cuad_simpson(x, f=f)
    """
    import numpy as np

    # Primero verificar si la particion es regular
    x = np.array(x)
    x.sort()
    H = (max(x) - min(x))/(len(x)-1)
    equiesp = np.std(np.diff(x)) < H*1.e-6

    # Calcular los valores de y (si se pasó una función)
    if (y is None) and (f is not None):
        y = f(x)

    n = len(x)

    if equiesp:
        impares = y[1:-1:2].sum()
        pares = y[2:-1:2].sum()
        H = y[0] + 2*pares + 4*impares + y[-1]
        H = H / (3*n)
        aprox = (x[-1]-x[0])*H
    else:
        aprox = 0
        for i in range(0, len(x)-2, 2):
            aprox += cuad_simpson(x[i], x[i+2], y[i], y[i+1], y[i+2])

    return aprox
```

```python
def f(x):
    return x**2
```

```python
x = np.linspace(0, 1, 999)
xr = np.random.uniform(0, 1, 1000)
y = f(x)
yr = f(xr)
```

```python
cuad_simpson_compuesta(x, y=y)
```

    0.3329996663329996

```python
cuad_simpson_compuesta(xr, y=yr)
```

    0.34178714534148646

```python
cuad_simpson_compuesta(x, f=f)
```

    0.3329996663329996

```python
from scipy import integrate
```

```python
integrate.quad(f, 0, 1)
```

    (0.33333333333333337, 3.700743415417189e-15)

##### Otra opción sería dar el intervalo y la función, e ir achicando la norma de la partición hasta que el error sea menor que algún valor dado.

```python
def cuad_simpson_compuesta_II(f, I, eps):
    """Implementación de la regla de Simpson

    Parameters
    ----------
    I: list
       Intervalo de integración, ingresado como lista de dos elementos
    f: function (1 parameter)
       La función a integrar

    Returns
    -------
    aprox: Aproximación de la integral por la regla de Simpson

    Notes
    -----
    Este código es parte del curso "Computación", Famaf
    Uso:
        cuad_simpson_compuesta_II(f, I)
        cuad_simpson_compuesta_II(f, I)
    """
    import numpy as np

    delta = 2*eps
    n = 2
    aprox_old = (I[1]-I[0])*f((I[1]+I[0])/2)

    while delta > eps:
        x = np.linspace(I[0], I[1], n)
        aprox = cuad_simpson_compuesta(x, f=f)
        delta = abs(aprox - aprox_old)
        aprox_old = aprox
        n += 10
        if n>5000:
            break

    return aprox
```

```python
I = cuad_simpson_compuesta_II(f, [0, 1], 1.e-6)
```

```python
I
```

    0.33307618132090666

```python

```

```python

```

```python

```

```python

```

_______
# Programación simbólica

```python
import sympy
```

```python
x,y = sympy.symbols('x y')
```

```python
f = sympy.sin(x)*sympy.exp(x)
```

```python
sympy.diff(f, x)
```

$\displaystyle e^{x} \sin{\left(x \right)} + e^{x} \cos{\left(x \right)}$

```python
sympy.integrate(f)
```

$\displaystyle \frac{e^{x} \sin{\left(x \right)}}{2} - \frac{e^{x} \cos{\left(x \right)}}{2}$

```python
sympy.limit(sympy.sin(x)/x, x, 0)
```

$\displaystyle 1$

```python
sympy.limit(sympy.atan(x)/(1+x**2), x, sympy.oo)
```

$\displaystyle 0$

```python
sympy.integrate(sympy.sin(x**2), (x, -sympy.oo, sympy.oo))
```

$\displaystyle \frac{\sqrt{2} \sqrt{\pi}}{2}$

```python

```


# Causality and Function Approximations

```python
import lightgbm as lgb
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy.stats
import sklearn.linear_model
import sympy as sp

from IPython.display import display_markdown, Markdown

from chmp.ds import reload

reload('chmp.ds')
from chmp.ds import (
    cached,
    define,
    loop_over,
    loop_nest,
    pd_has_ordered_assign,
    pgm,
    qlineplot,
    to_markdown,
)
from chmp.app.causality import prob

assert pd_has_ordered_assign()
```

```python
base_graph = (
    pgm()
    .node('Z', r'$Z$', 1.5, 1.75)
    .node('X', r'$X$', 1, 1)
    .node('Y', r'$Y$', 2, 1)
    .edges('Z', 'XY')
    .edges('X', 'Y')
)
```

```python
subplot_kwargs = dict(
    gridspec_kw=dict(
        top=0.925, bottom=0.075, left=0.01, right=0.99, wspace=0.1,
    ),
)
```

```python
with mpl.rc_context({'text.usetex': True}):
    _, ax1 = plt.subplots(1, 1, figsize=(3, 3.5), **subplot_kwargs)
    base_graph.render(ax=ax1)
    plt.savefig('./blog/schematic.png')
```

```python
with mpl.rc_context({'text.usetex': True}):
    _, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(3 * (3 + 0.2), 3.35), **subplot_kwargs)
    base_graph.render(ax=ax1, title=r'$\mathcal{G}$')
    base_graph.remove(outgoing='X').render(ax=ax2, title=r'$\mathcal{G}_{\underline{X}}$')
    base_graph.remove(incoming='X').render(ax=ax3, title=r'$\mathcal{G}_{\overline{X}}$')
    plt.savefig('./blog/schematic_graph_operations.png')
```

## Tabular example

```python
np.random.seed(42_21_13_7)

df = pd.DataFrame().assign(
    z=lambda df: np.random.binomial(n=1, p=0.3, size=100_000),
    x=lambda df: np.random.binomial(n=1, p=0.8 - 0.4 * df['z']),
    y=lambda df: np.random.binomial(n=1, p=0.5 + 0.4 * df['z'] - 0.2 * df['x'])
)
counts = df.groupby(['x', 'y', 'z']).size().pipe(lambda s: s.reorder_levels(sorted(s.index.names)))
reweighted_counts = (0.5 * counts / counts.pipe(prob, 'x', 'z', transform=True)).astype(int)
```

```python
renames = {
    'x': '`X`',
    'y': '`Y`',
    'z': '`Z`',
}
```

```python
print(
    pd.concat([
        counts.to_frame("`N`"),
        reweighted_counts.to_frame("`N'`")
    ], axis=1)
    .reset_index()
    .sort_values(['y', 'z', 'x'])
    .rename(columns=renames)
    .pipe(to_markdown)
)
```

    |   `X` |   `Y` |   `Z` |   `N` |   `N'` |
    |------:|------:|------:|------:|-------:|
    |     0 |     0 |     0 |  6958 |  17407 |
    |     1 |     0 |     0 | 39315 |  24567 |
    |     0 |     0 |     1 |  1842 |   1536 |
    |     1 |     0 |     1 |  3512 |   4385 |
    |     0 |     1 |     0 |  7051 |  17640 |
    |     1 |     1 |     0 | 16773 |  10481 |
    |     0 |     1 |     1 | 16087 |  13415 |
    |     1 |     1 |     1 |  8462 |  10566 |

```python
from chmp.app.causality import prob
```

```python
def _calculuate(c, label):
    return (
        c.pipe(prob, 'y', 'xz')
        .pipe(lambda s: s.xs(1, level='x') - s.xs(0, level='x'))
        .xs(1, level='y')
        .round(3)
        .to_frame(f'`{label}(Y=1/X=1,Z) - {label}(Y=1/X=0,Z)`')
    )

print(
    pd.concat([
        _calculuate(counts, "P"),
        _calculuate(reweighted_counts, "P'"),
    ], axis=1)
    .reset_index()
    .rename(columns=renames)
    .pipe(to_markdown)
)
```

    |   `Z` |   `P(Y=1/X=1,Z) - P(Y=1/X=0,Z)` |   `P'(Y=1/X=1,Z) - P'(Y=1/X=0,Z)` |
    |------:|--------------------------------:|----------------------------------:|
    |     0 |                          -0.204 |                            -0.204 |
    |     1 |                          -0.191 |                            -0.191 |

```python
print(
    counts
    .pipe(
        lambda df: (
            df.pipe(prob, 'y', 'xz') * df.pipe(prob, 'z', transform=True)
        )
    )
    .sum(level=[*'xy'])
    .pipe(lambda s: s.xs(1, level='x') - s.xs(0, level='x'))
    .pipe(lambda s: s[1])
)
```

    -0.20017232801025664

```python
counts.pipe(prob, 'y', 'x').pipe(lambda s: s.xs(1, level='x') - s.xs(0, level='x')).pipe(lambda s: s[1])
```

    -0.35370126233694327

## Continous example

```python
def get_ols_coeffs(df, features, regularization=0, sample_weight=None):
    est = sklearn.linear_model.Ridge(regularization)
    est.fit(
        df[features], df['y'],
        sample_weight=(df[sample_weight] if sample_weight is not None else None),
    )
    coef = {k: round(v, 3) for k, v in zip(features, est.coef_)}
    return coef['x']
```

```python
def sample_continuous(size=100_000):
    pz = lambda df: scipy.stats.norm(np.zeros(size), 1.0)
    px = lambda df: scipy.stats.norm(+0.4 * df['z'], 1.0)
    py = lambda df: scipy.stats.norm(+1.0 * df['z'] - 0.1 * df['x'], 1.0)

    # the target distribution of the reweighted data set
    p_do_x = lambda df, target_var: scipy.stats.norm(0, target_var ** 0.5)

    return pd.DataFrame().assign(
        z=lambda df: pz(df).rvs(),
        x=lambda df: px(df).rvs(),
        y=lambda df: py(df).rvs(),
        w_high=lambda df: p_do_x(df, 1.0).pdf(df['x']) / px(df).pdf(df['x']),
        w_low=lambda df: p_do_x(df, 0.01).pdf(df['x']) / px(df).pdf(df['x']),
    )
```

```python
@cached('.cache/blog_correlation_results.pickle')
def correlation_results():
    np.random.seed(42_42_42_42)
    correlation_results = []

    for _ in loop_over(range(500)):
        df = sample_continuous()
        correlation_results += [{
            'Method': 'Unweighted',
            'Correlation': (
                df.eval('x * z').mean() / np.sqrt(df.eval('x * x').mean() * df.eval('z * z').mean())
            ),
        }]
        correlation_results += [{
            'Method': 'Reweighted ($\sigma_t^2 = 1$)',
            'Correlation': (
                    df.eval('w_high * x * z').mean()
                / np.sqrt(df.eval('w_high * x * x').mean() * df.eval('w_high * z * z').mean())
            ),
        }]
        correlation_results += [{
            'Method': 'Reweighted ($\sigma_t^2 = 10^{-2}$)',
            'Correlation': (
                df.eval('w_low * x * z').mean()
                / np.sqrt(df.eval('w_low * x * x').mean() * df.eval('w_low * z * z').mean())
            ),
        }]

    return pd.DataFrame(correlation_results)
```

    load cache .cache/blog_correlation_results.pickle

```python
specs = [
    dict(
        label='$P(Y|X, Z)$',
        features=['x', 'z'],
    ),
    dict(
        label=r'$P(Y|X, Z)$ (reweighted, $\sigma_t = 1$)',
        features=['x', 'z'],
        sample_weight='w_high',
    ),
    dict(
        label=r'$P(Y|X, Z)$ (reweighted, $\sigma_t = 10^{-2}$)',
        features=['x', 'z'],
        sample_weight='w_low',
    ),
    dict(
        label='$P(Y|X)$',
        features=['x'],
    ),
    dict(
        label='$P(Y|X)$ (reweighted, $\sigma_t = 1$)',
        features=['x'],
        sample_weight='w_high',
    ),
    dict(
        label='$P(Y|X)$ (reweighted, $\sigma_t = 10^{-2}$)',
        features=['x'],
        sample_weight='w_low',
    ),
]

@cached('.cache/blog_post_experiment.pickle')
def results():
    np.random.seed(42_42_42_42)
    results = []

    for regularization in [0, 1e2, 1e3, 5e3, 1e4, 5e4, 1e5, 1e6, 1e7]:
        for _ in loop_over(range(500)):
            df = sample_continuous()

            for spec in specs:
                spec = dict(spec)
                label = spec.pop('label')
                result = {
                    'Method': label,
                    'Estimated effect': get_ols_coeffs(
                        df,
                        **spec,
                        regularization=regularization,
                    ),
                    'Regularization': regularization,
                }

                results += [result]
        print()

    return pd.DataFrame(results)
```

    load cache .cache/blog_post_experiment.pickle

```python
subset_p_yxz = {
    '$P(Y|X, Z)$',
    '$P(Y|X, Z)$ (reweighted, $\sigma_t = 1$)',
    #'$P(Y|X, Z)$ (reweighted, $\sigma_t = 10^{-2}$)',
}

subset_p_yx = {
    '$P(Y|X)$',
    '$P(Y|X)$ (reweighted, $\sigma_t = 1$)',
    #'$P(Y|X)$ (reweighted, $\sigma_t = 10^{-2}$)',
}
```

```python
plt.subplots(figsize=(6, 3.5), gridspec_kw=dict(top=0.975, bottom=0.15, left=0.135, right=.975))
qlineplot(
    x='Regularization',
    hue='Method',
    y='Estimated effect',
    data=results[results['Method'].isin(subset_p_yxz)],
)
plt.axhline(y=-.1, color='k', ls='--', alpha=0.5, zorder=-1)
plt.axhline(y=0, color='k', ls='--', alpha=0.5, zorder=-1)
plt.ylim(-0.15, +0.1)
plt.xscale('log')
plt.legend(loc='upper left')
plt.title('Linear model')
plt.savefig('blog/linear_model_p_yxz_regularization.png', dpi=100)
```

```python
plt.subplots(figsize=(6, 3.5), gridspec_kw=dict(top=0.975, bottom=0.15, left=0.135, right=.975))
qlineplot(
    x='Regularization',
    hue='Method',
    y='Estimated effect',
    data=results[results['Method'].isin(subset_p_yx)],
)
plt.xscale('log')
plt.legend(loc='upper right')
plt.ylim(-0.15, 0.4)
plt.axhline(y=-.1, color='k', ls='--', alpha=0.5, zorder=-1)
plt.axhline(y=0, color='k', ls='--', alpha=0.5, zorder=-1)
plt.title('Linear model')
plt.savefig('blog/linear_model_p_yx_regularization.png', dpi=100)
```

# Simulated Drug Trial

Data set setup take from https://florianwilhelm.info/2017/04/causal_inference_propensity_score/

```python
def simulate(size=100_000):
    def exp_recovery_time(df):
        return df.eval('exp(2 + 0.5 * sex + 0.03 * age + 2 * severity - 1 * medication)')

    def rvs_recovery_time(df):
        return scipy.stats.poisson.rvs(exp_recovery_time(df))

    def get_p_medication(df):
        u = df.eval('(1 / 3 * sex + 2 / 3 * severity - 0.8) / 0.15')
        return scipy.stats.norm(0, 1).cdf(u)

    return pd.DataFrame().assign(
        sex=lambda df: np.random.randint(0, 2, size=size),
        age=lambda df: scipy.stats.gamma.rvs(8, scale=4, size=size),
        severity=lambda df: scipy.stats.beta.rvs(3, 1.5, size=size),
        medication_p=lambda df: get_p_medication(df),
        medication=lambda df: np.random.binomial(n=1, p=df['medication_p']),
        recovery=lambda df: rvs_recovery_time(df),
        log_recovery=lambda df: df.eval('log(recovery)'),
        weight=lambda df: df.eval('medication / medication_p + (1 - medication) / (1 - medication_p)'),
        no_weight=lambda df: df.eval('1'),
    )
```

```python
df = simulate()
```

```python
features = ['sex', 'age', 'severity', 'medication']
target = 'log_recovery'
```

```python
import itertools as it
```

```python
@cached('.cache/blog_post_lgb_experiment.pickle')
def results_lgb():
    results = []
    for rep in loop_over(range(50)):
        df = simulate()
        for learning_rate, weight in loop_nest(
            list(
                it.product(
                    [0.75 ** e for e in range(1, 15)],
                    ['weight', 'no_weight'],
                )
            ),
            label=rep,
        ):
            est = lgb.LGBMRegressor(
                n_estimators=100,
                learning_rate=learning_rate,
            )
            est.fit(df[features], df[target], sample_weight=df[weight])

            results.append(
                dict(
                    learning_rate=learning_rate,
                    weight=weight,
                    mad=abs(est.predict(df[features]) - df[target]).mean(),
                    ace=np.mean(
                        est.predict(df.assign(medication=1)[features]) -
                        est.predict(df.assign(medication=0)[features])
                    ),
                )
            )

    return pd.DataFrame(results)
```

    load cache .cache/blog_post_lgb_experiment.pickle

```python
results_lgb['Inverse learning rate'] = 1 / results_lgb['learning_rate']
results_lgb['Weight'] = results_lgb['weight'].map(dict(weight='Reweighted', no_weight='Non-reweighted'))
results_lgb['Estimated effect'] = results_lgb['ace']
results_lgb['MAD'] = results_lgb['mad']
```

```python
plt.subplots(figsize=(6, 3.5), gridspec_kw=dict(top=0.975, bottom=0.15, left=0.135, right=.975))
qlineplot(
    x='Inverse learning rate',
    hue='Weight',
    y='Estimated effect',
    ci=0.9,
    data=results_lgb,
)
plt.xscale('log')
learning_rates = np.sort(results_lgb['learning_rate'].unique())
learning_rates = learning_rates[::2]
plt.xticks(
    1.0 / learning_rates,
    ['{:.2f}'.format(lr) for lr in learning_rates]
)
plt.xlabel('Learning rate')
plt.title('Gradient boosting')
plt.savefig('blog/lgb_effect_estimate.png', dpi=100)
```

```python
plt.subplots(figsize=(6, 3.5), gridspec_kw=dict(top=0.975, bottom=0.15, left=0.135, right=.975))
qlineplot(
    x='Inverse learning rate',
    hue='Weight',
    y='MAD',
    ci=0.9,
    data=results_lgb,
)
plt.xscale('log')
learning_rates = np.sort(results_lgb['learning_rate'].unique())
learning_rates = learning_rates[::2]
plt.xticks(
    1.0 / learning_rates,
    ['{:.2f}'.format(lr) for lr in learning_rates]
)
plt.xlabel('Learning rate')
plt.title('Gradient boosting')
plt.savefig('blog/lgb_mad.png', dpi=100)
```

# Linear model analytics

$$
\begin{align}
    z &\sim p(z) = \mathcal{N}(\mu_z, \sigma_z)
\\
    x &\sim p(x|z) = \mathcal{N}(\mu_x + \alpha_{xz} z, \sigma_x)
\\
    y &\sim p(y|x, z) = \mathcal{N}(\mu_y + \alpha_{yx} x + \alpha_{yz} z, \sigma_y)
\end{align}
$$

$$
\begin{align}
    \mathbb{E} \left[ z \right] &= \mu_z
&
    \mathbb{E} \left[ x \right] &= \mu_x + \alpha_{xz} \mu_z
&
    \mathbb{E} \left[ y \right] &=
        \mu_y
        + \alpha_{yx} \mu_x +
        (\alpha_{yx} \alpha_{xz} + \alpha_{yz}) \mu_z
\\
    \mathrm{cov} \left[ z, z \right]
    &= \sigma_z^2
&
    \mathrm{cov} \left[ x, z \right]
    &= \alpha_{xz} \sigma_z^2
&
    \mathrm{cov} \left[ y, z \right]
    &= (\alpha_{yx} \alpha_{xz} + \alpha_{yz}) \sigma_z^2
\\
    &
&
    \mathrm{cov} \left[ x, x \right]
    &= \sigma_{x}^2 + \alpha_{xz}^2 \sigma_z^2
&
    \mathrm{cov} \left[ y, x \right]
    &= (\alpha_{yx} \alpha_{xz} + \alpha_{yz}) \alpha_{xz} \sigma_z^2 + \alpha_{yx} \sigma_x^2
\\
    &
&
    &
&
    \mathrm{cov} \left[ y, y \right]
    &= (\alpha_{yx} \alpha_{xz} + \alpha_{yz})^2 \sigma_z^2
    + \alpha_{yx}^2 \sigma_{x}^2
    + \sigma_y^2
\end{align}
$$

For simplicity, assume $\mu_y = 0$ and fit the linear model

$$
\begin{align}
    \hat{y} &= \beta_{yx} x + \beta_{yz} z
\end{align}
$$

The loss function is given by

$$
\begin{align}
    \mathcal{L}
    &= \frac{1}{2} \langle (y - \hat{y})^2 \rangle
    + \frac{1}{2} \lambda \beta_{yx}^2
    + \frac{1}{2} \lambda \beta_{yz}^2
\end{align}
$$

Setting the derivatives with respect to the parameters equal to zero yields

$$
\begin{align}
    \langle yx \rangle
    &= \beta_{yx} (\lambda + \langle xx \rangle) + \beta_{yz} \langle zx \rangle
\\
    \langle yz \rangle
    &= \beta_{yx} \langle xz \rangle + \beta_{yz} (\lambda + \langle zz \rangle)
\end{align}
$$

The coefficients are given by:

$$
\begin{align}
    \beta_{yx} &=
    \frac{
        \langle yx \rangle (\lambda + \langle zz \rangle) - \langle yz \rangle \langle zx \rangle
    }{
        (\lambda + \langle xx \rangle) (\lambda + \langle zz \rangle)
        - \langle xz \rangle \langle zx \rangle
    }
\\
    \beta_{yz} &=
    \frac{
        \langle yx \rangle  \langle xz \rangle - \langle yz \rangle  (\lambda + \langle xx \rangle)
    }{
        \langle zx \rangle \langle xz \rangle -
        (\lambda + \langle zz \rangle) (\lambda + \langle xx \rangle)
    }
\end{align}
$$

```python
@define
def formulas(render=True):
    mu_x, mu_y, mu_z = sp.symbols('mu_x, mu_y, mu_z')
    sigma_x, sigma_y, sigma_z = sp.symbols('sigma_x, sigma_y, sigma_z')
    alpha_xz, alpha_yx, alpha_yz = sp.symbols('alpha_xz, alpha_yx, alpha_yz')
    beta_yx, beta_yz, lam = sp.symbols('beta_yx, beta_yz, lambda')

    mu_y = 0
    mu_z = 0
    mu_x = 0

    #lam = 0

    mean_z = mu_z
    mean_x = mu_x + alpha_xz * mean_z
    mean_y = mu_y + alpha_yx * mean_x + alpha_yz * mean_z

    cov_zz = sigma_z ** 2
    cov_zx = alpha_xz * sigma_z ** 2
    cov_zy = (alpha_yx * alpha_xz + alpha_yz) * sigma_z ** 2

    cov_xx = sigma_x ** 2 + alpha_xz ** 2 * sigma_z ** 2
    cov_xy = (
        (alpha_yx * alpha_xz + alpha_yz) * alpha_xz * sigma_z ** 2 +
        alpha_yx * sigma_x ** 2
    )

    # cov_yy = (
    #     ((alpha_yx * alpha_xz + alpha_yz) * sigma_z) ** 2
    #     + (alpha_yx * sigma_x) ** 2
    #     + sigma_y ** 2
    # )

    mean_yx = mean_y * mean_x + cov_xy
    mean_yz = mean_y * mean_z + cov_zy
    mean_xz = mean_x * mean_z + cov_zx
    mean_zz = mean_z * mean_z + cov_zz
    mean_xx = mean_x * mean_x + cov_xx
    #  mean_yy = mean_y * mean_y + cov_yy

    beta_yx = sp.simplify(
        (mean_yx * (lam + mean_zz) - mean_yz * mean_xz)
        / ((lam + mean_xx) * (lam + mean_zz) - mean_xz * mean_xz)
    )

    beta_yz = sp.simplify(
        (mean_yx * mean_xz - mean_yz * (lam + mean_xx))
        / (mean_xz * mean_xz - (lam + mean_zz) * (lam + mean_xx))
    )

    ratio = sp.simplify(beta_yx / beta_yz)

    if render:
        source = [
            r'$$'
            r'\begin{align}',
            r'\beta_{yx} &= ' + sp.latex(beta_yx),
            r'\\\\',
            r'\frac{\beta_{yx}}{\beta_{yz}} &= ' + sp.latex(ratio),
            r'\end{align}',
            '$$',
        ]

        display_markdown(Markdown('\n'.join(source)))

    else:
        print('beta_yx')
        print(sp.latex(beta_yx))
        print()
        print('beta_yx / beta_yz')
        print(sp.latex(ratio))
```

$$\begin{align}
\beta_{yx} &= \frac{\alpha_{xz} \sigma_{z}^{4} \left(\alpha_{xz} \alpha_{yx} + \alpha_{yz}\right) - \left(\lambda + \sigma_{z}^{2}\right) \left(\alpha_{xz} \sigma_{z}^{2} \left(\alpha_{xz} \alpha_{yx} + \alpha_{yz}\right) + \alpha_{yx} \sigma_{x}^{2}\right)}{\alpha_{xz}^{2} \sigma_{z}^{4} - \left(\lambda + \sigma_{z}^{2}\right) \left(\alpha_{xz}^{2} \sigma_{z}^{2} + \lambda + \sigma_{x}^{2}\right)}
\\\\
\frac{\beta_{yx}}{\beta_{yz}} &= \frac{\alpha_{xz}^{2} \alpha_{yx} \lambda \sigma_{z}^{2} + \alpha_{xz} \alpha_{yz} \lambda \sigma_{z}^{2} + \alpha_{yx} \lambda \sigma_{x}^{2} + \alpha_{yx} \sigma_{x}^{2} \sigma_{z}^{2}}{\sigma_{z}^{2} \left(\alpha_{xz} \alpha_{yx} \lambda + \alpha_{yz} \lambda + \alpha_{yz} \sigma_{x}^{2}\right)}
\end{align}
$$

For $\mu_x = 0, \mu_y = 0, \mu_z = 0$:

$$
\begin{align}
    \beta_{yx} &=
    \frac{
        \alpha_{yx}
        + \frac{\lambda}{\sigma_x^2} \alpha_{xz} \left( \alpha_{yz} + \alpha_{xz} \alpha_{yx} \right)
        + \frac{\lambda}{\sigma_z^2} \alpha_{yx}
    }{
        \frac{\lambda}{\sigma_x^2} \alpha_{xz}^{2}
        + \left(1 + \frac{\lambda}{\sigma_z^2}\right)\left(1 + \frac{\lambda}{\sigma_x^2}\right)
    }
\\
    \beta_{yz} &=
    \frac{
        \alpha_{yz}
        + \frac{\lambda}{\sigma_x^2} ( \alpha_{xz} \alpha_{yx} + \alpha_{yz})
    }{
        \frac{\lambda}{\sigma_x^2} \alpha_{xz}^{2}
        + \left(1 + \frac{\lambda}{\sigma_z^2}\right)\left(1 + \frac{\lambda}{\sigma_x^2}\right)
    }
\end{align}
$$

Multiple regimes:

- $\lambda \ll \sigma_x^2, \sigma_z^2$: regularization is not important
- $\lambda \gg \sigma_x^2, \sigma_z^2$ and $\sigma_z^2 \gg \sigma_x^2$: regularization is important, variation in $z$ is much higher than $x$.
- $\lambda \gg \sigma_x^2, \sigma_z^2$ and $\sigma_x^2 \gg \sigma_z^2$: regularization is important, variation in $x$ is much higher than $z$.

Regime I:

$$
\begin{align}
    \beta_{yx} &\approx \alpha_{yx}
\\
    \beta_{yz} &\approx \alpha_{yz}
\end{align}
$$

Regime II:

$$
\begin{align}
    \beta_{yx} &\approx \frac{1}{1 + \alpha_{xz}^{-2}}
        \left( \alpha_{yz} + \alpha_{xz} \alpha_{yx} \right)
\\
    \beta_{yz} &\approx \left( 1 - \frac{1}{1 + \alpha_{xz}^{-2}} \right)
        \left( \alpha_{xz} \alpha_{yx} + \alpha_{yz} \right)
\end{align}
$$

Regime III:

$$
\begin{align}
    \beta_{yx} &\approx \alpha_{yx}
\\
    \beta_{yz} &\approx 0
\end{align}
$$

Reweighted (setting $\alpha_{xz} = 0$ + fixing the variance):

$$
\begin{align}
    \beta_{yx} &= \frac{\alpha_{yx}}{\left(1 + \frac{\lambda}{\sigma_x^2}\right)}
\\
    \beta_{yz} &= \frac{\alpha_{yz}}{\left(1 + \frac{\lambda}{\sigma_z^2}\right)}
\end{align}
$$

```python

```


# Bouncing ball with the Euler integrator
As an introduction to solving ODEs we are solving the problem of the bouncing ball with the simplest of all integrators: the forward Euler scheme.

We have to solve the *second order ODE* (Newton's equations of motion with constant acceleration)
$$
\frac{d^2 y}{dt^2} = -g.
$$

The Euler scheme for any *first order ODE*

$$
\frac{dy}{dt} = f(y, t)
$$

is
$$
y(t + h) = y(t) + h f(y(t), t).
$$

In order to solve the original 2nd order equation of motion we make use of the fact that one $n$-th order ODE can be written as $n$ coupled first order ODEs, namely

\begin{align}
\frac{dy}{dt} &= v\\
\frac{dv}{dt} &= -g.
\end{align}

Solve each of the first order ODEs with Euler:

\begin{align}
y(t + h) &= y(t) + h v(t)\\
v(t + h) &= v(t) - h g.
\end{align}

```python
import numpy as np
```

## Free fall

Start with free fall

```python
g = -9.81
y = 0.0
v = 0.0

t = 0
dt = 0.01

data = []

while t < 10:
    y = y + v*dt
    v = v + g*dt
    data.append([t, y, v])
    t += dt

data = np.array(data)
```

```python
data.shape
```

    (1001, 3)

```python
data = data.transpose()
data.shape  # t, y, v
```

    (3, 1001)

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.plot(data[0], data[1])
plt.xlabel("time (s)")
plt.ylabel("position (m)");
```

## Bouncing
Add a floor at $y = -5$.

What happens at the floor? – The velocity changes (elastic collision).

```python
g = -9.81
y = 0.0
v = 0.0

t = 0
dt = 0.01

y_floor = -5

data = []

while t < 10:
    y += v*dt
    if y > y_floor:
        v += g*dt
    else:
        v = -v   # bounce off floor
    data.append([t, y, v])
    t += dt

data = np.array(data).transpose()
```

```python
plt.plot(data[0], data[1])
plt.xlabel("time (s)")
plt.ylabel("position (m)");
```

```python

```


# Keypoints

- Be careful of overflow like big number minus big number

- To prevent underflow, try staying in the log space

- Always using numerical libraries to prevent overflow

# Scalars

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

## Integers

### Binary representation of integers

```python
format(16, '032b')
```

    '00000000000000000000000000010000'

### Bit shifting

```python
format(16 >> 2, '032b')
```

    '00000000000000000000000000000100'

```python
16 >> 2
```

    4

```python
format(16 << 2, '032b')
```

    '00000000000000000000000001000000'

```python
16 << 2
```

    64

### Overflow

In general, the computer representation of integers has a limited range, and may overflow. The range depends on whether the integer is signed or unsigned.

For example, with 8 bits, we can represent at most $2^8 = 256$ integers.

- 0 to 255 unsigned
- -128 ti 127 signed

Signed integers

```python
np.arange(130, dtype=np.int8)[-5:]
```

    array([ 125,  126,  127, -128, -127], dtype=int8)

Unsigned integers

```python
np.arange(130, dtype=np.uint8)[-5:]
```

    array([125, 126, 127, 128, 129], dtype=uint8)

```python
np.arange(260, dtype=np.uint8)[-5:]
```

    array([255,   0,   1,   2,   3], dtype=uint8)

### Integer division

In Python 2 or other languages such as C/C++, be very careful when dividing as the division operator `/` performs integer division when both numerator and denominator are integers. This is rarely what you want. In Python 3 the `/` always performs floating point division, and you use `//` for integer division, removing a common source of bugs in numerical calculations.

```python
%%python2

import numpy as np

x = np.arange(10)
print(x/10)
```

    Couldn't find program: 'python2'

Python 3 does the "right" thing.

```python
x = np.arange(10)
x/10
```

    array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])

## Real numbers

Real numbers are represented as **floating point** numbers. A floating point number is stored in 3 pieces (sign bit, exponent, mantissa) so that every float is represetned as get +/- mantissa ^ exponent. Because of this, the interval between consecutive numbers is smallest (high precison) for numebrs close to 0 and largest for numbers close to the lower and upper bounds.

Because exponents have to be singed to represent both small and large numbers, but it is more convenint to use unsigned numbers here, the exponnent has an offset (also knwnn as the exponentn bias). For example, if the expoennt is an unsigned 8-bit number, it can rerpesent the range (0, 255). By using an offset of 128, it will now represent the range (-127, 128).

**Note**: Intervals between consecutive floating point numbers are not constant. In particular, the precision for small numbers is much larger than for large numbers. In fact, approximately half of all floating point numbers lie between -1 and 1 when using the `double` type in C/C++ (also the default for `numpy`).

Because of this, if you are adding many numbers, it is more accurate to first add the small numbers before the large numbers.

#### IEEE 754 32-bit floating point representation

See [Wikipedia](https://en.wikipedia.org/wiki/Single-precision_floating-point_format) for how this binary number is evaluated to 0.15625.

```python
from ctypes import c_int, c_float
```

```python
s = c_int.from_buffer(c_float(0.15625)).value
```

```python
s = format(s, '032b')
s
```

    '00111110001000000000000000000000'

```python
rep = {
    'sign': s[:1],
    'exponent' : s[1:9:],
    'fraction' : s[9:]
}
rep
```

    {'sign': '0', 'exponent': '01111100', 'fraction': '01000000000000000000000'}

### Most base 10 real numbers are approximations

This is simply because numbers are stored in finite-precision binary format.

```python
'%.20f' % (0.1 * 0.1 * 100)
```

    '1.00000000000000022204'

### Never check for equality of floating point numbers

```python
i = 0
loops = 0
while i != 1:
    i += 0.1 * 0.1
    loops += 1
    if loops == 1000000:
        break
i
```

    10000.000000171856

```python
i = 0
loops = 0
while np.abs(1 - i) > 1e-6:
    i += 0.1 * 0.1
    loops += 1
    if loops == 1000000:
        break
i
```

    1.0000000000000007

### Associative law does not necessarily hold

```python
6.022e23 - 6.022e23 + 1
```

    1.0

```python
1 + 6.022e23 - 6.022e23
```

    0.0

### Distributive law does not hold

```python
a = np.exp(1)
b = np.pi
c = np.sin(1)
```

```python
a*(b+c)
```

    10.82708950985241

```python
a*b + a*c
```

    10.827089509852408

### Catastrophic cancellation

Consider calculating sample variance

$$
s^2= \frac{1}{n(n-1)}\sum_{i=1}^n x_i^2 - (\sum_{i=1}^n x_i)^2
$$

Be  careful whenever you calculate the difference of potentially big numbers.

```python
def var(x):
    """Returns variance of sample data using sum of squares formula."""

    n = len(x)
    return (1.0/(n*(n-1))*(n*np.sum(x**2) - (np.sum(x))**2))
```

### Underflow

```python
np.warnings.filterwarnings('ignore')
```

```python
np.random.seed(4)
xs = np.random.random(1000)
ys = np.random.random(1000)
np.prod(xs)/np.prod(ys)
```

    nan

#### Prevent underflow by staying in log space

```python
x = np.sum(np.log(xs))
y = np.sum(np.log(ys))
np.exp(x - y)
```

    696432868222.2549

### Overflow

```python
np.exp(1000)
```

    inf

### Numerically stable algorithms

#### What is the sample variance for numbers from a normal distribution with variance 1?

```python
np.random.seed(15)
x_ = np.random.normal(0, 1, int(1e6))
x = 1e12 + x_
var(x)
```

    -1328166901.4739892

#### Use functions from numerical libraries where available

```python
np.var(x)
```

    1.001345504504934

There is also a variance function in the standard library, but it is slower for large arrays.

```python
import statistics

statistics.variance(x)
```

    1.0013464175061981

Note that `numpy` uses does not use the asymptotically unbiased estimator by default. If you want the unbiased variance, set `ddof` to 1.

```python
np.var([1,2,3,4], ddof=1)
```

    1.6666666666666667

```python
statistics.variance([1,2,3,4])
```

    1.6666666666666667

### Useful numerically stable functions

Let's calculate

$$
\log(e^{1000} + e^{1000})
$$

Using basic algebra, we get the solution $\log(2) + 1000$.

\begin{align}
\log(e^{1000} + e^{1000}) &= \log(e^{0}e^{1000} + e^{0}e^{1000}) \\
&= \log(e^{100}(e^{0} + e^{0})) \\
&= \log(e^{1000}) + \log(e^{0} + e^{0}) \\
&= 1000 + \log(2)
\end{align}

**logaddexp**

```python
x = np.array([1000, 1000])
np.log(np.sum(np.exp(x)))
```

    inf

```python
np.logaddexp(*x)
```

    1000.6931471805599

**logsumexp**

This function generalizes `logaddexp` to an arbitrary number of addends and is useful in a variety of statistical contexts.

Suppose we need to calculate a probability distribution $\pi$ parameterized by a vector $x$

$$
\pi_i = \frac{e^{x_i}}{\sum_{j=1}^n e^{x_j}}
$$

Taking logs, we get

$$
\log(\pi_i) = x_i - \log{\sum_{j=1}^n e^{x_j}}
$$

```python
x = 1e6*np.random.random(100)
```

```python
np.log(np.sum(np.exp(x)))
```

    inf

```python
from scipy.special import logsumexp
```

```python
logsumexp(x)
```

    984755.5793729819

**logp1 and expm1**

```python
np.exp(np.log(1 + 1e-6)) - 1
```

    9.999999999177334e-07

```python
np.expm1(np.log1p(1e-6))
```

    1e-06

**sinc**

```python
x = 0
```

```python
np.sin(x)/x
```

    nan

```python
np.sinc(x)
```

    1.0

```python
x = np.linspace(0.01, 2*np.pi, 100)
```

```python
plt.plot(x, np.sinc(x), label='Library function')
plt.plot(x, np.sin(np.pi*x)/(np.pi*x), label='DIY function')
plt.legend()
pass
```

```python

```


```
import ExactSol
from dolfin import *
import sympy as sy
import mshr
n = 2**6
domain = mshr.Rectangle(Point(-1., -1.), Point(1., 1.)) - mshr.Rectangle(Point(0., 0.), Point(1., 1.) )
mesh = mshr.generate_mesh(domain, n)
order = 2
Velocity = VectorFunctionSpace(mesh, "CG", order)
Pressure = FunctionSpace(mesh, "CG", order-1)
Magnetic = FunctionSpace(mesh, "N1curl", order-1)
Lagrange = FunctionSpace(mesh, "CG", order-1)
```

    DEBUG:FFC:Reusing form from cache.
    DEBUG:FFC:Reusing form from cache.
    DEBUG:FFC:Reusing form from cache.
    DEBUG:FFC:Reusing form from cache.

```
u0, p0,b0, r0, Laplacian, Advection, gradPres,CurlCurl, gradR, NS_Couple, M_Couple = ExactSol.MHD2D(1,1, mesh)
# plot(interpolate(u0, Velocity), prefix='Velocity').write_png()
# plot(interpolate(b0, Magnetic), prefix='Magnetic').write_png()
plot(interpolate(p0, Pressure),elevate=0.0, prefix='Pressure').write_png()
plot(interpolate(r0, Lagrange), elevate=0.0, prefix='Lagrange').write_png()
```

       >>>>>>>>>>>>>>>>>>>>>>>>>>
         MHD 2D Exact Solution:
       >>>>>>>>>>>>>>>>>>>>>>>>>>

     ----------------------
       NS Exact Solution:
     ----------------------
      u = ( exp(x)*sin(y) , exp(x)*cos(y) )

      p = ( sin(x)*cos(y) )

     ---------------------------
       Maxwell Exact Solution:
     ---------------------------
      b = ( exp(x + y)*cos(x) , exp(x + y)*sin(x) - exp(x + y)*cos(x) )

      p = ( x*sin(2*pi*x)*sin(2*pi*y) )

```

```

```
plot(interpolate(u0, Velocity))
```

    <dolfin.cpp.io.VTKPlotter; proxy of <Swig Object of type 'std::shared_ptr< dolfin::VTKPlotter > *' at 0x10ee79cf0> >

```

x = symbols('x')
y = symbols('y')
```

```
u = sy.sin(x)*sy.exp(y)
v = sy.cos(y)*sy.exp(x)
p = sy.sin(x)*sy.cos(y)
uu = x*y*sy.cos(x)
b = sy.diff(uu,y)
d = -sy.diff(uu,x)
r = x*sy.sin(2*sy.pi*y)*sy.sin(2*sy.pi*x)
u0 = Expression((sy.ccode(u),sy.ccode(v)))
p0 = Expression(sy.ccode(p))
b0 = Expression((sy.ccode(b),sy.ccode(d)))
r0 = Expression(sy.ccode(r).replace('M_PI','pi'))
```

```
plot(interpolate(u0, Velocity))

```

    <dolfin.cpp.io.VTKPlotter; proxy of <Swig Object of type 'std::shared_ptr< dolfin::VTKPlotter > *' at 0x108b375d0> >