# Finite Difference

```python
import sympy
from sympy import init_printing, pprint, latex
from sympy import binomial, Function, factorial, Symbol, symbols, Sum, S
# S is for SingletonRegistry
from sympy import Rational as Rat
from sympy import Matrix, solve_linear_system
```

EY : 20160627 I have no idea what `sympy.SingletonRegistry` means despite the [documentation for SingletonRegistry](http://docs.sympy.org/0.7.1/_modules/sympy/core/singleton.html), which is sparse at best.

```python
init_printing() # LaTeX printing
```

```python
n = Symbol('n',integer=True,positive=True)
i = Symbol('i',integer=True,positive=True)
j = Symbol('j',integer=True,positive=True)
x,h,x_0 = symbols('x h x_0',real=True)
f = Function('f')
```

```python
factorial(5)
```

    120

```python
C_n = Sum((-1)**i*binomial(n,i),(i,1,n))

```

```python
(C_n.subs(n,3)).evalf()
```

    -1.00000000000000

cf. From *stackexchange* [How to do a symbolic taylor expansion of an unknown function $f(x)$ using sympy](http://stackoverflow.com/questions/16869587/how-to-do-a-symbolic-taylor-expansion-of-an-unknown-function-fx-using-sympy)

```python
f(x).series(x, x0=x_0, n=3)
```

$$f{\left (x_{0} \right )} + \left(x - x_{0}\right) \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=x_{0} }} + \frac{1}{2} \left(x - x_{0}\right)^{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=x_{0} }} + \mathcal{O}\left(\left(x - x_{0}\right)^{3}; x\rightarrowx_{0}\right)$$

```python
pprint( f(x).series(x, x0=h, n=3) )
```

                                                  ⎛  2        ⎞│
                                                2 ⎜ d         ⎟│
                                        (-h + x) ⋅⎜────(f(ξ₁))⎟│
                                                  ⎜   2       ⎟│
                    ⎛ d        ⎞│                 ⎝dξ₁        ⎠│ξ₁=h    ⎛        3
    f(h) + (-h + x)⋅⎜───(f(ξ₁))⎟│     + ──────────────────────────── + O⎝(-h + x)
                    ⎝dξ₁       ⎠│ξ₁=h                2

           ⎞
    ; x → h⎠

```python
latex(f(x).series(x, x0=h, n=3) )
```

    'f{\\left (h \\right )} + \\left(- h + x\\right) \\left. \\frac{d}{d \\xi_{1}} f{\\left (\\xi_{1} \\right )} \\right|_{\\substack{ \\xi_{1}=h }} + \\frac{1}{2} \\left(- h + x\\right)^{2} \\left. \\frac{d^{2}}{d \\xi_{1}^{2}}  f{\\left (\\xi_{1} \\right )} \\right|_{\\substack{ \\xi_{1}=h }} + \\mathcal{O}\\left(\\left(- h + x\\right)^{3}; x\\rightarrowh\\right)'

```python
FW = f(x+h).series(x+h, x0=x_0, n=3)
FW = FW.subs(x-x_0,0)
```

```python
FW
```

```python
FW.args[0]
```

```python
(f(x+h).series(x+h,x0=x_0,n=3)+f(x-h).series(x-h,x0=x_0,n=3)).subs(x-x_0,0)
```

```python
( f(x+h).series(x+h,x0=x_0,n=3) + f(x-h).series(x-h,x0=x_0,n=3) + \
f(x+2*h).series(x+2*h,x0=x_0,n=3) + f(x-2*h).series(x-2*h,x0=x_0,n=3) ).subs(x-x_0,0)
```

```python
( f(x+h).series(x+h,x0=x_0,n=3) - f(x-h).series(x-h,x0=x_0,n=3) + \
f(x+2*h).series(x+2*h,x0=x_0,n=3) - f(x-2*h).series(x-2*h,x0=x_0,n=3) ).subs(x-x_0,0)
```

finite-difference for the 1st order derivative ($n=1$), to the $p$th-order, $p=8$

```python
p = 8
( Rat(4,5)*(f(x+h).series(x+h,x0=x_0,n=p) - f(x-h).series(x-h,x0=x_0,n=p)) + \
Rat(-1,5)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)-f(x-2*h).series(x-2*h,x0=x_0,n=p)) + \
Rat(4,105)*(f(x+3*h).series(x+3*h,x0=x_0,n=p)-f(x-3*h).series(x-3*h,x0=x_0,n=p)) + \
Rat(-1,280)*(f(x+4*h).series(x+4*h,x0=x_0,n=p)-f(x-4*h).series(x-4*h,x0=x_0,n=p)) ).subs(x-x_0,0)
```

```python
Rat(4,5)
```

### *centered difference approximation*

$$-i_{\text{min}} = (d+p-1)/2 $$
$$i_{\text{max}} = \lceil (d+p-1)/2 \rceil $$

cf. [Derivative Approximation by Finite Differences
David Eberly](http://math.nyu.edu/~atm262/fall06/compmethods/a1/DerivativesApproximationByFiniteDifferences.pdf)

```python
d = 2
p = 8
sympy.floor(Rat(d+p-1,2)), Rat((d+p-1),2)
```

First order derivatives $d=1$; let's aim for $-i_{\text{min}} = i_{\text{max}} = 2$ so we only need 2 neighbor values.

```python
d = 1
p = 5
sympy.floor(Rat(d+p-1,2)), Rat((d+p-1),2)
```

```python
d = 2
p = 4
sympy.floor(Rat(d+p-1,2)), Rat((d+p-1),2)
```

```python
C_1,C_2,C_3,C_4 = symbols('C_1 C_2 C_3 C_4', integer=True)
```

From `sympy` documentation, [Matrices (linear algebra), Basic Manipulation](http://docs.sympy.org/0.7.2/modules/matrices/matrices.html), I learned to construct matrices with the `lambda` function

```python
Matrix(3,4, lambda i,j: (j+1)**(2*(i+1)+1))
```

```python
Matrix(1,4,lambda i,j: 2*(j+1))
```

```python
C_nu_d1p6 = sympy.zeros(4,5) # d = 1, p = 6
C_nu_d1p6[:3,:4] = Matrix(3,4,lambda i,j: (j+1)**(2*(i+1)+1))
C_nu_d1p6[3,:4] = Matrix(1,4,lambda i,j: 2*(j+1))
C_nu_d1p6[3,4] = Rat(1)
```

```python
C_nu_d1p6
```

```python
solve_linear_system( C_nu_d1p6, C_1,C_2,C_3,C_4 )
```

Also, another way is this:

```python
C_nuM_d1p6 = sympy.zeros(4,4) # d = 1, p = 6
C_nuM_d1p6[:3,:4] = Matrix(3,4,lambda i,j: (j+1)**(2*(i+1)+1))
C_nuM_d1p6[3,:4] = Matrix(1,4,lambda i,j: 2*(j+1))
C_nuM_d1p6
```

```python
C_nuM_d1p6.inv() # and then read off the last column
```

```python
Matrix([[1,8],[2,4]]).inv() # d = 1, p=2
```

```python
C_nu_d1p4 = sympy.zeros(3,4) # d = 1, p = 4
C_nu_d1p4[:2,:3] = Matrix(2,3,lambda i,j: (j+1)**(2*(i+1)+1))
C_nu_d1p4[2,:3] = Matrix(1,3,lambda i,j: 2*(j+1))
C_nu_d1p4[2,3] = Rat(1)
```

```python
C_nu_d1p4
```

```python
solve_linear_system( C_nu_d1p4, C_1,C_2,C_3 )
```

Indeed,

```python
p = 6
( Rat(3,4)*(f(x+h).series(x+h,x0=x_0,n=p) - f(x-h).series(x-h,x0=x_0,n=p)) + \
Rat(-3,20)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)-f(x-2*h).series(x-2*h,x0=x_0,n=p)) + \
Rat(1,60)*(f(x+3*h).series(x+3*h,x0=x_0,n=p)-f(x-3*h).series(x-3*h,x0=x_0,n=p))).subs(x-x_0,0)
```

$$
\boxed{   h \frac{d}{dx}f(x) =  \frac{3}{4} ( f(x+h) - f(x-h) ) + \frac{-3}{20}( f(x+2h) - f(x-2h)) + \frac{1}{60} (f(x+3h)-f(x-3h)) +O(h^6) = \sum_{\nu = 1}^3 C_{\nu} (f(x+ \nu h) - f(x - \nu h) ) + O(h^6) }
  $$

with

 * $C_1 = \frac{3}{4}$
 * $C_2 = \frac{-3}{20}$
 * $C_3 = \frac{1}{60}$

Let's generalize this procedure:

```python
def make_C_nu_dp(NU):
    C_nu_dp = sympy.zeros(NU,NU+1)
    C_nu_dp[:NU-1,:NU] = Matrix(NU-1,NU,lambda i,j: (j+1)**(2*(i+1)+1))
    #C_nu_dp[NU-1,:NU]  = Matrix(1,NU,lambda i,j:2*(j+1))
    C_nu_dp[NU-1,:NU]  = Matrix(1,NU,lambda i,j:(j+1))
    C_nu_dp[NU-1,NU]   = Rat(1,2)

    # sanity check
    pprint( C_nu_dp)
    Cnus = symbols(' '.join( ['C_'+str(nu)for nu in range(1,NU+1)]), real=True)
    return solve_linear_system( C_nu_dp, *Cnus )
```

```python
make_C_nu_dp(2)
```

    ⎡1  8   0 ⎤
    ⎢         ⎥
    ⎣1  2  1/2⎦

    {C_1: 2/3, C_2: -1/12}

```python
# sanity check
p=2
(Rat(2,3)*(f(x+h).series(x+h,x0=x_0,n=p) - f(x-h).series(x-h,x0=x_0,n=p)) + \
Rat(-1,12)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)-f(x-2*h).series(x-2*h,x0=x_0,n=p))).simplify()

```

```python
make_C_nu_dp(3)
```

```python
make_C_nu_dp(4)
```

Let's attempt to further generalize this procedure for $d=1$.

Consider the following easy examples for $p=1$.

Now
$$
f(x \pm h) = f(x) + (\frac{ \partial f}{ \partial x}(x) )( \pm h) + \mathcal{O}(h^2)
$$
$$
f(x+h) - f(x-h) = 2h \frac{ \partial f}{ \partial x} + \mathcal{O}(h^2) \text{ or } \frac{ \partial f}{ \partial x} = \frac{ f(x+h) - f(x-h) }{2h} + \mathcal{O}(h)
$$

Likewise, for so-called forward and backward difference operators:
$$
\frac{f(x+h) - f(x)}{h} = \frac{ \partial f}{ \partial x}
$$
$$
\frac{f(x) - f(x-h)}{h} = \frac{ \partial f}{ \partial x}
$$
Conjecture that
$$
\boxed{
\frac{ \partial f}{ \partial x} = \sum^N_{\nu = 1} C_{\nu} (f(x+ \nu h) - f(x-\nu h) ) \left( \frac{1}{h} \right)
}
$$

and so doing the Taylor expansion,
$$
f(x \pm \nu h) =\sum_{j=0}^{p} \frac{ f^{(j)}(x) }{j!} ( \pm \nu h)^j + \mathcal{O}(h^{p+1}) = \sum_{j'=0}^{p'} \frac{ f^{(2j')}(x) }{ (2j')! } (\nu h)^{2j'} + \sum_{j'=0}^{p'} \frac{ f^{(2j'+1) }(x) }{ (2j'+1)!} (\nu h)^{2j'+1} (\pm 1) + \mathcal{O}(h^{p+1} )
$$
where $p := 2p'+1$.

Clearly, for $f(x+\nu h) - f(x-\nu h)$, all the "even" terms in the Taylor series expansion cancel each other out, leaving
$$
f(x+\nu h) - f(x-\nu h) = \sum_{j'=0}^{p'} \frac{ 2f^{(2j'+1)}(x) }{(2j'+1)!} (\nu h)^{2j'+1} + O(h^{p+1}) = 2\nu h \frac{ \partial f}{\partial x} + \sum_{j'=1}^{p'} \frac{2(\nu h)^{2j' +1} }{ (2j'+1)! } f^{(2j'+1) }(x) + \mathcal{O}(h^{p+1} )
$$

and so with our previous assumption (caveat),

$$
\sum_{\nu =1}^{N} C_{\nu} ( f(x+ \nu h) - f(x-\nu h)) \frac{1}{h} = \sum_{\nu =1}^N C_{\nu} \left( 2\nu \frac{ \partial f}{ \partial x} + \sum_{j'=1}^{p'} \frac{ 2\nu ^{2j'+1} }{ (2j'+1)! } f^{(2j+1) }(x) h^{2j'} \right) + \mathcal{O}(h^{p} )
$$

Then we obtain a system of linear equations that we'd want to solve:

$$
\sum_{\nu =1}^N 2C_{\nu} \nu = 1
$$

$$
\sum_{j'=1}^{p'} \left( \sum_{\nu =1}^N C_{\nu} \frac{2\nu^{2j'+1} }{ (2j'+1)! } \right) f^{(2j+1) }(x) h^{2j'} = 0 \text{ or }  \left( \sum_{\nu =1}^N C_{\nu} \frac{ 2 \nu^{2j'+1} }{ (2j'+1)! } \right) = 0 \qquad \, \forall \, j' = 1 \dots p'
$$

Thus, we confirm the `make_C_nu_dp(NU)` Python function for $d=1$.

## Backward difference operator, in the case of $d=1$, first-order derivative

For the backward difference operator, then consider the following:

$$
\frac{ \partial f}{ \partial x} = \sum_{ \nu =0}^N \frac{1}{h} C_{\nu} f(x-\nu h) = \sum_{\nu = 0}^N \frac{C_{\nu} }{ h} \left[ f(x) + \left( \frac{ \partial f}{ \partial x} \right) (-\nu h) + \sum_{j=2}^p \frac{ f^{(j)}(x) }{j!} (-\nu h)^j \right] + \mathcal{O}(h^p)
$$

which leads to a system of linear equations:

$$
\frac{1}{h} \sum_{\nu =0}^N C_{\nu} = 0
$$

$$
- \sum_{\nu =0}^N C_{\nu } \nu = 1
$$

$$
\sum_{j=2}^p \frac{(-h)^jf^{(j)}(x)}{ hj!} \sum_{ \nu =0}^N C_{\nu} \nu^j = 0
$$

and I didn't make it explicit before, and neither here, but you can divide both sides of the equation by constant values that *don't* depend on index $j$ (or $\nu$) and obtain the system of linear equations to solve.

```python
NU = 3
C_nueqns = sympy.zeros(NU+1,NU+2)
C_nueqns[0,:NU+1] = Matrix(1,NU+1,lambda i,j: 1)
C_nueqns[1,:NU+1] = Matrix(1,NU+1,lambda i,j:j)
C_nueqns[1,NU+1] = -1
if NU > 1:
    C_nueqns[2:NU+1,:NU+1] = Matrix(NU-1,NU+1,lambda i,j: j**(i+2) )
pprint(C_nueqns)
Cnus = symbols(' '.join( ['C_'+str(nu)for nu in range(NU+1)]), real=True)
pprint(Cnus)
solve_linear_system( C_nueqns,*Cnus)
```

```python
def make_backwardC_nu(NU):
    C_nueqns = sympy.zeros(NU+1,NU+2)
    C_nueqns[0,:NU+1]  = Matrix(1,NU+1,lambda i,j: 1)

    C_nueqns[1,:NU+1]  = Matrix(1,NU+1,lambda i,j: j)
    C_nueqns[1,NU+1]   = -1

    if NU > 1:
        C_nueqns[2:NU+1,:NU+1] = Matrix(NU-1,NU+1,lambda i,j: (j)**(i+2) )

    # sanity check
    pprint( C_nueqns)

    Cnus = symbols(' '.join( ['C_'+str(nu)for nu in range(NU+1)]), real=True)
    return solve_linear_system( C_nueqns, *Cnus )
```

```python
make_backwardC_nu(1)
```

```python
make_backwardC_nu(2)
```

```python
# sanity check
f(x).series(x,x0=x_0,n=4).subs(x-x_0,0)
f(x-h).series(x-h,x0=x_0,n=4).subs(x-x_0,0)
f(x-2*h).series(x-2*h,x0=x_0,n=4).subs(x-x_0,0)

p=4
( Rat(3,2)*f(x).series(x,x0=x_0,n=p).subs(x-x_0,0) + \
  Rat(-2)*( f(x-h).series(x-h,x0=x_0,n=p) ).subs(x-x_0,0) + \
Rat(1,2)*( f(x-2*h).series(x-2*h,x0=x_0,n=p)).subs(x-x_0,0) )

```

```python
make_backwardC_nu(3)
```

```python
# sanity check
p=5
( Rat(11,6)*f(x).series(x,x0=x_0,n=p).subs(x-x_0,0) + \
  Rat(-3)*( f(x-h).series(x-h,x0=x_0,n=p) ).subs(x-x_0,0) + \
Rat(3,2)*( f(x-2*h).series(x-2*h,x0=x_0,n=p)).subs(x-x_0,0) + \
Rat(-1,3)*( f(x-3*h).series(x-3*h,x0=x_0,n=p)).subs(x-x_0,0)
)

```

```python
make_backwardC_nu(4)
```

```python
# sanity check
p=6
( Rat(25,12)*f(x).series(x,x0=x_0,n=p).subs(x-x_0,0) + \
  Rat(-4)*( f(x-h).series(x-h,x0=x_0,n=p) ).subs(x-x_0,0) + \
Rat(3)*( f(x-2*h).series(x-2*h,x0=x_0,n=p)).subs(x-x_0,0) + \
Rat(-4,3)*( f(x-3*h).series(x-3*h,x0=x_0,n=p)).subs(x-x_0,0) + \
Rat(1,4)*( f(x-4*h).series(x-4*h,x0=x_0,n=p)).subs(x-x_0,0)
)
```

#### For $d=2$

```python
C_nu_d2p4 = sympy.zeros(2,3) # d = 2, p = 4
C_nu_d2p4[:1,:2] = Matrix(1,2,lambda i,j: (j+1)**(2*(i+2)))
C_nu_d2p4[1,:2] = Matrix(1,2,lambda i,j: 2*(j+1)**2)
C_nu_d2p4[1,2] = Rat(1)
```

```python
C_nu_d2p4
```

```python
solve_linear_system( C_nu_d2p4, C_1,C_2)
```

```python
C_nu_d2p6 = sympy.zeros(3,4) # d = 2, p = 6
C_nu_d2p6[:2,:3] = Matrix(2,3,lambda i,j: (j+1)**(2*(i+2)))
C_nu_d2p6[2,:3] = Matrix(1,3,lambda i,j: 2*(j+1)**2)
C_nu_d2p6[2,3] = Rat(1)
```

```python
C_nu_d2p6
```

```python
solve_linear_system( C_nu_d2p6, C_1,C_2,C_3)
```

Indeed,

```python
p = 4
( Rat(2,3)*(f(x+h).series(x+h,x0=x_0,n=p) + f(x-h).series(x-h,x0=x_0,n=p) -2*f(x_0)) + \
Rat(-1,24)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)+f(x-2*h).series(x-2*h,x0=x_0,n=p) -2*f(x_0))).subs(x-x_0,0)
```

```python
p = 6
( Rat(3,4)*(f(x+h).series(x+h,x0=x_0,n=p) + f(x-h).series(x-h,x0=x_0,n=p)-2*f(x_0)) + \
Rat(-3,40)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)+f(x-2*h).series(x-2*h,x0=x_0,n=p)-2*f(x_0)) + \
Rat(1,180)*(f(x+3*h).series(x+3*h,x0=x_0,n=p)+f(x-3*h).series(x-3*h,x0=x_0,n=p)-2*f(x_0))).subs(x-x_0,0)
```

## Central difference operator, in the case of $d=2$, 2nd-order derivative

Let's attempt to further generalize this procedure for $d=2$.

Consider the following easy examples for $p=1$.

Now
$$
f(x \pm h) = f(x) + (\frac{ \partial f}{ \partial x}(x) )( \pm h) + ( \frac{1}{2} \frac{ \partial^2 f}{ \partial x^2}(x) )h^2 + \mathcal{O}(h^2)
$$
$$
f(x+h) + f(x-h) = 2f(x) + h^2 \frac{ \partial^2 f}{ \partial x^2} + \mathcal{O}(h^3) \text{ or } \frac{ \partial^2 f}{ \partial x^2} = \frac{ f(x+h) + f(x-h) - 2f(x) }{h^2} + \mathcal{O}(h)
$$

And so

$$
f(x \pm \nu h) = f(x) + (\frac{ \partial f}{ \partial x}(x) )( \pm \nu h) + (\frac{1}{2} \frac{ \partial^2 f}{ \partial x^2}(x) ) \nu^2 h^2 + \mathcal{O}(h^2)
$$
$$
f(x+\nu h) + f(x-\nu h) = 2f(x) + \nu^2 h^2 \frac{ \partial^2 f}{ \partial x^2} + \mathcal{O}(h^3) \text{ or } \frac{ \partial^2 f}{ \partial x^2} = \frac{ f(x+\nu h) + f(x-\nu h) - 2f(x) }{\nu^2 h^2} + \mathcal{O}(h)
$$

Conjecture that
$$
\boxed{
\frac{ \partial^2 f}{ \partial x^2} = \sum^N_{\nu = 1} C_{\nu} (f(x+ \nu h) + f(x-\nu h) -2f(x) ) \left( \frac{1}{h^2} \right)
}
$$
(yes, the $\frac{1}{\nu^2}$ factor was swallowed into $C_{\nu}$.)

and so doing the Taylor expansion,
$$
f(x \pm \nu h) =\sum_{j=0}^{p} \frac{ f^{(j)}(x) }{j!} ( \pm \nu h)^j + \mathcal{O}(h^{p+1}) = \sum_{j'=0}^{p'} \frac{ f^{(2j')}(x) }{ (2j')! } (\nu h)^{2j'} + \sum_{j'=0}^{p'} \frac{ f^{(2j'+1) }(x) }{ (2j'+1)!} (\nu h)^{2j'+1} (\pm 1) + \mathcal{O}(h^{p+1} )
$$
where $p := 2p'+1$.

Clearly, for $f(x+\nu h) + f(x-\nu h)$, all the "odd" terms in the Taylor series expansion cancel each other out, leaving
$$
f(x+\nu h) + f(x-\nu h) -2f(x)= \sum_{j'=1}^{p'} \frac{ 2f^{(2j')}(x) }{(2j')!} (\nu h)^{2j'} + O(h^{p+1}) =  \frac{ 2f''(x)}{2} (\nu h)^2 + \sum_{j'=2}^{p'} \frac{2(\nu h)^{2j' } }{ (2j')! } f^{(2j') }(x) + \mathcal{O}(h^{p+1} )
$$

and so with our previous assumption (caveat),

$$
\sum_{\nu =1}^{N} C_{\nu} ( f(x+ \nu h) + f(x-\nu h) -2f(x)) \frac{1}{h^2} = \sum_{\nu =1}^N C_{\nu} \left(  \nu^2 f''(x)h^2 + \sum_{j'=2}^{p'} \frac{ 2\nu ^{2j'} }{ (2j')! } f^{(2j') }(x) h^{2j'} \right) \frac{1}{h^2} + \mathcal{O}(h^{p} ) = \sum_{\nu =1}^N \left[ C_{\nu} \nu^2  f''(x) + C_{\nu} \sum_{j'=2}^{p'} \frac{ 2\nu^{2j'} }{ (2j')!} f^{(2j') }(x)h^{2(j'-1) } \right] + \mathcal{O}(h^p)
$$

Then we obtain a system of linear equations that we'd want to solve:

$$
\sum_{\nu =1}^N \nu^2 C_{\nu}  = 1
$$

$$
\sum_{j'=2}^{p'} \left( \sum_{\nu =1}^N C_{\nu} \frac{ 2\nu^{2j'} }{ (2j')! } \right)= 0
$$

```python
def make_C_nu_d2p(NU):
    C_nu_dp = sympy.zeros(NU,NU+1)
    C_nu_dp[:NU-1,:NU] = Matrix(NU-1,NU,lambda i,j: (j+1)**(2*(i+2)))

    C_nu_dp[NU-1,:NU]  = Matrix(1,NU,lambda i,j:(j+1)**2)
    C_nu_dp[NU-1,NU]   = Rat(1)

    # sanity check
    pprint( C_nu_dp)
    Cnus = symbols(' '.join( ['C_'+str(nu)for nu in range(1,NU+1)]), real=True)
    return solve_linear_system( C_nu_dp, *Cnus )
```

```python
make_C_nu_d2p(2)
```

```python
make_C_nu_d2p(3)
```

```python
make_C_nu_d2p(4)
```

Indeed

```python
p = 6
( Rat(4,3)*(f(x+h).series(x+h,x0=x_0,n=p) + f(x-h).series(x-h,x0=x_0,n=p) -2*f(x_0)) + \
Rat(-1,12)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)+f(x-2*h).series(x-2*h,x0=x_0,n=p) -2*f(x_0))).subs(x-x_0,0)
```

```python
p = 8
( Rat(3,2)*(f(x+h).series(x+h,x0=x_0,n=p) + f(x-h).series(x-h,x0=x_0,n=p) -2*f(x_0)) + \
Rat(-3,20)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)+f(x-2*h).series(x-2*h,x0=x_0,n=p) -2*f(x_0)) + \
Rat(1,90)*(f(x+3*h).series(x+3*h,x0=x_0,n=p)+f(x-3*h).series(x-3*h,x0=x_0,n=p) -2*f(x_0)) ).subs(x-x_0,0)
```

```python
p = 10
( Rat(8,5)*(f(x+h).series(x+h,x0=x_0,n=p) + f(x-h).series(x-h,x0=x_0,n=p) -2*f(x_0)) + \
Rat(-1,5)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)+f(x-2*h).series(x-2*h,x0=x_0,n=p) -2*f(x_0)) + \
Rat(8,315)*(f(x+3*h).series(x+3*h,x0=x_0,n=p)+f(x-3*h).series(x-3*h,x0=x_0,n=p) -2*f(x_0)) + \
 Rat(-1,560)*(f(x+4*h).series(x+4*h,x0=x_0,n=p)+f(x-4*h).series(x-4*h,x0=x_0,n=p) -2*f(x_0))
).subs(x-x_0,0)
```

# finite difference sanity check on Python numpy

```python
import numpy as np
```

```python
import matplotlib
import pylab
import numpy as np
```

```python
%matplotlib inline
%config InlineBackend.figure_formats = {'svg','png'}
```

```python
l_x = 2.* np.pi
l_y = 2.* np.pi
l_z = 2.* np.pi
L_x = 64
L_y = 64
L_z = 64
f_grid3d = np.zeros((L_x,L_y,L_z))
df_grid3d_result = np.zeros((L_x,L_y,L_z))
h_x = l_x/L_x; h_y = l_y/L_y; h_z = l_z/L_z
```

```python
for i_z in range(L_z):
    for i_y in range(L_y):
        for i_x in range(L_x):
            f_grid3d[i_x][i_y][i_z] = np.sin( i_x*h_x)
```

```python
f_grid3d[:,0,0].shape
```

    (64,)

```python
pylab.plot( f_grid3d[:,5,5] )
```

```python
RAD = 3
```

```python
for i_z in range(L_z-RAD):
    for i_y in range(L_y-RAD):
        for i_x in range(L_x-RAD):
#            if (i_z + RAD >= L_z):
            df_grid3d_result[i_x][i_y][i_z] = \
                (3./(4.*h_x))*(f_grid3d[i_x+1][i_y][i_z]-f_grid3d[i_x-1][i_y][i_z]) + \
                (-3./(20.*h_x))*(f_grid3d[i_x+2][i_y][i_z]-f_grid3d[i_x-2][i_y][i_z]) + \
                (1./(60.*h_x))*(f_grid3d[i_x+3][i_y][i_z]-f_grid3d[i_x-3][i_y][i_z])
```

```python
pylab.plot( df_grid3d_result[:,5,5] )
```

```python
pylab.plot( df_grid3d_result[:,0,0] )
```

```python
f_grid3d[-2][0][0]
```

    -0.19509032201612872

```python
p = 6
( Rat(3,4)*(f(x+h).series(x+h,x0=x_0,n=p) - f(x-h).series(x-h,x0=x_0,n=p)) + \
Rat(-3,20)*(f(x+2*h).series(x+2*h,x0=x_0,n=p)-f(x-2*h).series(x-2*h,x0=x_0,n=p)) + \
Rat(1,60)*(f(x+3*h).series(x+3*h,x0=x_0,n=p)-f(x-3*h).series(x-3*h,x0=x_0,n=p))).subs(x-x_0,0)
```

# Instantons

cf. Valery Rubakov. **Classical Theory of Gauge Fields**.  Princeton, 2002.  Ch. 7.  7.1 Kink

```python
t = Symbol('t',real=True)
phi1pl1 = Symbol(r'\varphi')# phi1pl1 denote phi 1+1 spacetime dimension; pl for "plus"
phi1pl1 = Function(r'\varphi')(t,x)
```

```python
phi1pl1
```

```python
phi1pl1.series(x=x,x0=x_0,n=3)
```

$$\varphi{\left (t,x_{0} \right )} + \left(x - x_{0}\right) \left. \frac{\partial}{\partial \xi_{2}} \varphi{\left (t,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=x_{0} }} + \frac{1}{2} \left(x - x_{0}\right)^{2} \left. \frac{\partial^{2}}{\partial \xi_{2}^{2}}  \varphi{\left (t,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=x_{0} }} + \mathcal{O}\left(\left(x - x_{0}\right)^{3}; x\rightarrowx_{0}\right)$$

```python
((phi1pl1.diff(t)*phi1pl1.diff(t)) + (phi1pl1.diff(x)*phi1pl1.diff(x))).series(x=x,x0=x_0,n=3)
```

$$\left. \frac{\partial}{\partial x} \varphi{\left (t,x \right )} \right|_{\substack{ x=x + x_{0} }}^{2} + \frac{\partial}{\partial t} \varphi{\left (t,x_{0} \right )}^{2} + 2 \frac{\partial}{\partial t}\left(\frac{1}{2} \left(x - x_{0}\right)^{2} \left. \frac{\partial^{2}}{\partial \xi_{2}^{2}}  \varphi{\left (t,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=x_{0} }}\right) \frac{\partial}{\partial t} \varphi{\left (t,x_{0} \right )} + 2 \frac{\partial}{\partial t}\left(\left(x - x_{0}\right) \left. \frac{\partial}{\partial \xi_{2}} \varphi{\left (t,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=x_{0} }}\right) \frac{\partial}{\partial t} \varphi{\left (t,x_{0} \right )} + \frac{\partial}{\partial t}\left(\left(x - x_{0}\right) \left. \frac{\partial}{\partial \xi_{2}} \varphi{\left (t,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=x_{0} }}\right)^{2} + \mathcal{O}\left(\left(x - x_{0}\right)^{3}; x\rightarrowx_{0}\right)$$

```python
lambdavariablesymbol = Symbol(r'\lambda')
v = Symbol('v')
mu = Symbol(r'\mu')
m = Symbol('m')
lambdavariablesymbol/Rat(4)*(phi1pl1*phi1pl1-v**2)**2
```

```python
(lambdavariablesymbol/Rat(4)*(phi1pl1*phi1pl1-v**2)**2).series(x=x,x0=x_0,n=2).subs(v, \
                                                        mu/sympy.sqrt( lambdavariablesymbol))
```

$$\frac{\mu^{4}}{4 \lambda} + \left(x - x_{0}\right) \left(\lambda \varphi^{3}{\left (t,x_{0} \right )} \left. \frac{\partial}{\partial \xi_{2}} \varphi{\left (t,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=x_{0} }} - \mu^{2} \varphi{\left (t,x_{0} \right )} \left. \frac{\partial}{\partial \xi_{2}} \varphi{\left (t,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=x_{0} }}\right) - \frac{\mu^{2}}{2} \varphi^{2}{\left (t,x_{0} \right )} + \frac{\lambda}{4} \varphi^{4}{\left (t,x_{0} \right )} + \mathcal{O}\left(\left(x - x_{0}\right)^{2}; x\rightarrowx_{0}\right)$$

# Dealing with Boundary conditions (with Finite Difference)

I found that I needed to represent boundary conditions that are "discretized" i.e. "numerical boundary condition" - for example on pp. 148 of Cebeci, Shao, Kafyeke, Laurendeau, (E5.2.3), I need to figure out if this is a derivative or second-order derivative:

$u_I^n = 2u^n_{I-1} - u^n_{I-2}$

```python
(f(x).series(x,x0=x_0,n=3)- 2 * f(x-h).series(x-h,x0=x_0,n=3) + \
 f(x-2*h).series(x-2*h,x0=x_0,n=3)).subs(x-x_0,0)
```

So indeed this is essentially

$ \frac{ \partial^2 u}{ \partial x^2} = 0$

Also consider pp. 338 of Cebeci, Shao, Kafyeke, Laurendeau (2005), namely (11.5.7a), (11.5.7b), which purports the second-order accuracy of a boundary condition, involving a first-order partial derivative:

$ \frac{ \partial u^x }{ \partial y } = 0 $ and

$\frac{\partial p }{ \partial y} = 0 $

cf. on the "symmetry plane" of $y=H$ (these boundary conditions, $\lbrace (x,y) \in \mathbb{R}^2 | y=H \rbrace$ is a *submanifold* of $\mathbb{R}^2$ as it's a level set or from the locally imbedded theorem (from differential topology))

```python
p=3
( Rat(3,2)*f(x).series(x,x0=x_0,n=p).subs(x-x_0,0) + \
  Rat(-2)*( f(x-h).series(x-h,x0=x_0,n=p) ).subs(x-x_0,0) + \
Rat(1,2)*( f(x-2*h).series(x-2*h,x0=x_0,n=p)).subs(x-x_0,0) )

```

```python
( Rat(1)*f(x).series(x,x0=x_0,n=p).subs(x-x_0,0) + \
  Rat(-4,3)*( f(x-h).series(x-h,x0=x_0,n=p) ).subs(x-x_0,0) + \
Rat(1,3)*( f(x-2*h).series(x-2*h,x0=x_0,n=p)).subs(x-x_0,0) )
```

Then "set this" first order partial derivative to zero and then you obtain the "scheme" to "discretize" the boundary condition.

$u_{i,J} = \frac{ 4 u_{i,J-1} - u_{i,J-2} }{ 3 } $

$p_{i,J} = \frac{ 4p_{i,J-1} -p_{i,J-2} }{ 3 } $

```python

```


# Canonical Clustering

## $k$ Nearest-Neighbor
### Distance Measure
The $k$ nearest neighbor is based on the simple idea that when a sample is close to other samples, changes are that they share the same label. To know how close one sample is from another sample, there needs to be a notion of close. For this application we will use euclidian distance also known as the $L^2$ norm

$$ \left\| x \right\|_2 \overset{\textrm{def}}= \left( \sum_{i=1}^n x_i^2 \right)^{1/2}$$

### Classification
In classification we would like to assign certain samples to a class. To do so we would like to implement an algorithm that assigns the class based on the classes of the $k$ nearest neighbors. Try to do so in `.code/knn.py`

```python
import sys
sys.path.append('code/')
import knn

import numpy as np

from sklearn import datasets, model_selection
from sklearn.metrics import adjusted_rand_score

# import some data to play with
X, y = datasets.load_iris(return_X_y=True)

# inspect the shape of the data
print(X.shape)
print(y.shape)

# split data into train and test
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y)

# and predict
k=10
y_predicted = knn.predict(X_train, X_test, y_train, k)

# evaluate predictions using y_predicted and y_test
ARI = adjusted_rand_score(np.squeeze(y_predicted), np.squeeze(y_test))
print("Adjusted Rand Index is: {0}".format(ARI))
```

    (150, 4)
    (150,)
    Adjusted Rand Index is: 0.8640044217217079

## Naive Bayes
#### TODO: probably this exercise is to extensive, rewrite it with using libraries.
### Dataset
Naive Bayes classifier is often used to filter emails for spam, such as in [this article](https://www.aclweb.org/anthology/E03-1059.pdf). For this exercise we will use a dataset containing samples from email messages. This dataset is available on [openml](https://www.openml.org/d/42673) and can be read in using sklearn:

```python
from sklearn import datasets
import numpy as np
spam = datasets.fetch_openml('auml_eml_1_a')

# lets inspect the dataset
np.set_printoptions(precision=3, suppress=True)
print("Dataset Dictionary Keys: \n{0}\n".format(spam.keys()))
print("Dataset Feature Names: \n{0}\n".format(spam['feature_names']))

# print a few examples of spam and non-spam
print("Example spam data entries: \n{0}\n".format(spam['data'][spam['target']==0][:8,:].T))
print("Example spam targets: \n{0}\n".format(spam['target'][spam['target']==0][:8].T))
print("Example non-spam data entries: \n{0}\n".format(spam['data'][spam['target']==1][:8,:].T))
print("Example non-spam targets: \n{0}\n".format(spam['target'][spam['target']==1][:8].T))
```

    Dataset Dictionary Keys:
    dict_keys(['data', 'target', 'frame', 'categories', 'feature_names', 'target_names', 'DESCR', 'details', 'url'])

    Dataset Feature Names:
    ['F1_text_html_found', 'F1_number_of_links', 'F1_contains_javascript']

    Example spam data entries:
    [[0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 5. 4. 3. 3. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0.]]

    Example spam targets:
    [0. 0. 0. 0. 0. 0. 0. 0.]

    Example non-spam data entries:
    [[0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0.]]

    Example non-spam targets:
    [1. 1. 1. 1. 1. 1. 1. 1.]

This dataset contains 3 features. The first is a binary feature named 'F1_text_html_found', the second is a nominal feature named 'F1_number_of_links', and the last is again a binary feature named 'F1_contains_javascript'. Using these three features, we are going to try to predict the change of a sample being spam. To do so we use:

\begin{align}
P(Y=y_k | X=x_i) = \frac{P(Y=y_k) P(X=x_i | Y=y_k)}{P(X=x_i)}
\end{align}

This is an alternative way of writing:

\begin{align}
P(Y=y_k | X=x_i) = \frac{P(Y=y_k) P(X_1=x_{i,1} \wedge \ldots \wedge X_d=x_{i,d}) | Y=y_k)}{P(X_1=x_{i,1} \wedge \ldots \wedge X_d=x_{i,d})}
\end{align}

The nominator is often omitted, because it scales all classes the same so when comparing classes it does not matter to omit it. Furthermore, it simplifies the calculation. Thus we ultimately we will use:

\begin{align}
P(Y=y_k | X=x_i) = P(Y=y_k) P(X_1=x_{i,1} \wedge \ldots \wedge X_d=x_{i,d}) | Y=y_k)
\end{align}

To use this formula, we first need to define what the probality distribution is. For a binary this is relatively simple. For the counted feature 'F1_number_of_links', we will use a probability distribution based on a histogram with bins with a width of $10$. Implement these probabilities in `code/utils.py`.

```python
import sys
sys.path.append('code/')
import naive_bayes

import pprint
from sklearn.model_selection import train_test_split

X = spam.data
y = spam.target

X_train, X_test, y_train, y_test = train_test_split(X,y, stratify=y)

feature_types = ['binary', 'histogram', 'binary']

model = naive_bayes.fit(X_train, y_train, feature_types)

# inspect model
#pprint.pprint(model)

y_predict = naive_bayes.predict(X_test, model)
```

    {'0': 0.8903432228039558,
     '0_featureset': {'0': {'0': 0.9895459000326691, '1': 0.010454099967330937},
                      '1': {'1': 0.5050637046716759,
                            '10': 0.005227049983665469,
                            '100': 0.0,
                            '101': 0.0,
                            '102': 0.0,
                            '103': 0.0,
                            '104': 0.0,
                            '105': 0.0,
                            '106': 0.0,
                            '107': 0.0003266906239790918,
                            '108': 0.0003266906239790918,
                            '109': 0.0,
                            '11': 0.00914733747141457,
                            '110': 0.0,
                            '111': 0.0,
                            '112': 0.0,
                            '113': 0.0,
                            '114': 0.0,
                            '115': 0.0,
                            '116': 0.0,
                            '117': 0.0,
                            '118': 0.0,
                            '119': 0.0,
                            '12': 0.003266906239790918,
                            '120': 0.0,
                            '121': 0.0,
                            '122': 0.0,
                            '123': 0.0,
                            '124': 0.0,
                            '125': 0.0003266906239790918,
                            '126': 0.0,
                            '127': 0.0,
                            '128': 0.0,
                            '129': 0.0,
                            '13': 0.005880431231623653,
                            '130': 0.0,
                            '131': 0.0,
                            '132': 0.0003266906239790918,
                            '133': 0.0,
                            '134': 0.0003266906239790918,
                            '135': 0.0,
                            '136': 0.0003266906239790918,
                            '137': 0.0,
                            '138': 0.0003266906239790918,
                            '139': 0.0003266906239790918,
                            '14': 0.0022868343678536427,
                            '140': 0.0,
                            '141': 0.0003266906239790918,
                            '142': 0.0,
                            '143': 0.0,
                            '144': 0.0003266906239790918,
                            '145': 0.0,
                            '146': 0.0,
                            '147': 0.0,
                            '148': 0.0,
                            '149': 0.0,
                            '15': 0.004900359359686377,
                            '150': 0.0,
                            '151': 0.0,
                            '152': 0.0,
                            '153': 0.0,
                            '154': 0.0,
                            '155': 0.0,
                            '156': 0.0,
                            '157': 0.0,
                            '158': 0.0,
                            '159': 0.0,
                            '16': 0.0029402156158118264,
                            '160': 0.0,
                            '161': 0.0,
                            '162': 0.0,
                            '163': 0.0,
                            '164': 0.0,
                            '165': 0.0,
                            '166': 0.0,
                            '167': 0.0,
                            '168': 0.0,
                            '169': 0.0,
                            '17': 0.0019601437438745506,
                            '170': 0.0,
                            '171': 0.0,
                            '172': 0.0,
                            '173': 0.0,
                            '174': 0.0,
                            '175': 0.0,
                            '176': 0.0,
                            '177': 0.0,
                            '178': 0.0,
                            '179': 0.0,
                            '18': 0.0035935968637700096,
                            '180': 0.0,
                            '181': 0.0,
                            '182': 0.0,
                            '183': 0.0,
                            '184': 0.0,
                            '185': 0.0,
                            '186': 0.0,
                            '187': 0.0,
                            '188': 0.0,
                            '189': 0.0,
                            '19': 0.0013067624959163672,
                            '190': 0.0,
                            '191': 0.0,
                            '192': 0.0,
                            '193': 0.0,
                            '194': 0.0,
                            '195': 0.0,
                            '196': 0.0,
                            '197': 0.0,
                            '198': 0.0,
                            '199': 0.0,
                            '2': 0.12381574648807579,
                            '20': 0.0013067624959163672,
                            '200': 0.0,
                            '201': 0.0,
                            '202': 0.0,
                            '203': 0.0,
                            '204': 0.0,
                            '205': 0.0,
                            '206': 0.0,
                            '207': 0.0,
                            '208': 0.0,
                            '209': 0.0,
                            '21': 0.0029402156158118264,
                            '210': 0.0,
                            '211': 0.0,
                            '212': 0.0,
                            '22': 0.003266906239790918,
                            '23': 0.001633453119895459,
                            '24': 0.0013067624959163672,
                            '25': 0.0013067624959163672,
                            '26': 0.001633453119895459,
                            '27': 0.0009800718719372753,
                            '28': 0.0009800718719372753,
                            '29': 0.0,
                            '3': 0.06599150604377654,
                            '30': 0.0,
                            '31': 0.0006533812479581836,
                            '32': 0.0003266906239790918,
                            '33': 0.0006533812479581836,
                            '34': 0.0006533812479581836,
                            '35': 0.0003266906239790918,
                            '36': 0.0009800718719372753,
                            '37': 0.0009800718719372753,
                            '38': 0.0006533812479581836,
                            '39': 0.0006533812479581836,
                            '4': 0.04083632799738648,
                            '40': 0.0006533812479581836,
                            '41': 0.0003266906239790918,
                            '42': 0.0003266906239790918,
                            '43': 0.0006533812479581836,
                            '44': 0.0003266906239790918,
                            '45': 0.0003266906239790918,
                            '46': 0.0,
                            '47': 0.0009800718719372753,
                            '48': 0.0003266906239790918,
                            '49': 0.0013067624959163672,
                            '5': 0.045083306109114665,
                            '50': 0.0003266906239790918,
                            '51': 0.0,
                            '52': 0.0003266906239790918,
                            '53': 0.0,
                            '54': 0.0006533812479581836,
                            '55': 0.0003266906239790918,
                            '56': 0.0,
                            '57': 0.0,
                            '58': 0.0,
                            '59': 0.0,
                            '6': 0.10192747468147664,
                            '60': 0.0,
                            '61': 0.0006533812479581836,
                            '62': 0.0,
                            '63': 0.0003266906239790918,
                            '64': 0.0003266906239790918,
                            '65': 0.0003266906239790918,
                            '66': 0.0,
                            '67': 0.0,
                            '68': 0.0,
                            '69': 0.0003266906239790918,
                            '7': 0.013067624959163672,
                            '70': 0.0003266906239790918,
                            '71': 0.0,
                            '72': 0.0003266906239790918,
                            '73': 0.0,
                            '74': 0.0,
                            '75': 0.0003266906239790918,
                            '76': 0.0,
                            '77': 0.0,
                            '78': 0.0,
                            '79': 0.0003266906239790918,
                            '8': 0.01110748121528912,
                            '80': 0.0,
                            '81': 0.0,
                            '82': 0.0,
                            '83': 0.0003266906239790918,
                            '84': 0.0,
                            '85': 0.0,
                            '86': 0.0,
                            '87': 0.0,
                            '88': 0.0,
                            '89': 0.0003266906239790918,
                            '9': 0.016007840574975497,
                            '90': 0.0,
                            '91': 0.0006533812479581836,
                            '92': 0.0,
                            '93': 0.0,
                            '94': 0.0,
                            '95': 0.0,
                            '96': 0.0,
                            '97': 0.0003266906239790918,
                            '98': 0.0,
                            '99': 0.0},
                      '2': {'0': 0.999673309376021, '1': 0.0003266906239790918}},
     '1': 0.1096567771960442,
     '1_featureset': {'0': {'0': 0.4907161803713528, '1': 0.5092838196286472},
                      '1': {'1': 0.41909814323607425,
                            '10': 0.002652519893899204,
                            '100': 0.0,
                            '101': 0.0,
                            '102': 0.0,
                            '103': 0.0,
                            '104': 0.0,
                            '105': 0.0,
                            '106': 0.0,
                            '107': 0.0,
                            '108': 0.0,
                            '109': 0.0,
                            '11': 0.0,
                            '110': 0.0,
                            '111': 0.0,
                            '112': 0.0,
                            '113': 0.0,
                            '114': 0.0,
                            '115': 0.0,
                            '116': 0.0,
                            '117': 0.0,
                            '118': 0.0,
                            '119': 0.0,
                            '12': 0.0,
                            '120': 0.0,
                            '121': 0.0,
                            '122': 0.0,
                            '123': 0.0,
                            '124': 0.0,
                            '125': 0.0,
                            '126': 0.0,
                            '127': 0.0,
                            '128': 0.0,
                            '129': 0.0,
                            '13': 0.0,
                            '130': 0.0,
                            '131': 0.0,
                            '132': 0.0,
                            '133': 0.0,
                            '134': 0.0,
                            '135': 0.0,
                            '136': 0.0,
                            '137': 0.0,
                            '138': 0.0,
                            '139': 0.0,
                            '14': 0.0,
                            '140': 0.0,
                            '141': 0.0,
                            '142': 0.0,
                            '143': 0.0,
                            '144': 0.0,
                            '145': 0.0,
                            '146': 0.0,
                            '147': 0.0,
                            '148': 0.0,
                            '149': 0.0,
                            '15': 0.0,
                            '150': 0.0,
                            '151': 0.0,
                            '152': 0.0,
                            '153': 0.0,
                            '154': 0.0,
                            '155': 0.0,
                            '156': 0.0,
                            '157': 0.0,
                            '158': 0.0,
                            '159': 0.0,
                            '16': 0.002652519893899204,
                            '160': 0.0,
                            '161': 0.0,
                            '162': 0.0,
                            '163': 0.0,
                            '164': 0.0,
                            '165': 0.0,
                            '166': 0.0,
                            '167': 0.0,
                            '168': 0.0,
                            '169': 0.0,
                            '17': 0.0,
                            '170': 0.0,
                            '171': 0.0,
                            '172': 0.0,
                            '173': 0.0,
                            '174': 0.0,
                            '175': 0.0,
                            '176': 0.0,
                            '177': 0.0,
                            '178': 0.0,
                            '179': 0.0,
                            '18': 0.0,
                            '180': 0.0,
                            '181': 0.0,
                            '182': 0.0,
                            '183': 0.0,
                            '184': 0.0,
                            '185': 0.0,
                            '186': 0.0,
                            '187': 0.0,
                            '188': 0.0,
                            '189': 0.0,
                            '19': 0.0,
                            '190': 0.0,
                            '191': 0.0,
                            '192': 0.0,
                            '193': 0.0,
                            '194': 0.0,
                            '195': 0.0,
                            '196': 0.0,
                            '197': 0.0,
                            '198': 0.0,
                            '199': 0.0,
                            '2': 0.4005305039787798,
                            '20': 0.0,
                            '200': 0.0,
                            '201': 0.0,
                            '202': 0.0,
                            '203': 0.0,
                            '204': 0.0,
                            '205': 0.0,
                            '206': 0.0,
                            '207': 0.0,
                            '208': 0.0,
                            '209': 0.0,
                            '21': 0.0,
                            '210': 0.0,
                            '211': 0.0,
                            '212': 0.0,
                            '22': 0.0,
                            '23': 0.0,
                            '24': 0.0,
                            '25': 0.0,
                            '26': 0.0,
                            '27': 0.0,
                            '28': 0.0,
                            '29': 0.0,
                            '3': 0.0636604774535809,
                            '30': 0.002652519893899204,
                            '31': 0.0,
                            '32': 0.002652519893899204,
                            '33': 0.0,
                            '34': 0.0,
                            '35': 0.0,
                            '36': 0.0,
                            '37': 0.0,
                            '38': 0.0,
                            '39': 0.0,
                            '4': 0.021220159151193633,
                            '40': 0.0,
                            '41': 0.0,
                            '42': 0.0,
                            '43': 0.0,
                            '44': 0.0,
                            '45': 0.0,
                            '46': 0.0,
                            '47': 0.0,
                            '48': 0.0,
                            '49': 0.0,
                            '5': 0.04509283819628647,
                            '50': 0.0,
                            '51': 0.0,
                            '52': 0.0,
                            '53': 0.002652519893899204,
                            '54': 0.0,
                            '55': 0.0,
                            '56': 0.0,
                            '57': 0.0,
                            '58': 0.0,
                            '59': 0.0,
                            '6': 0.023872679045092837,
                            '60': 0.002652519893899204,
                            '61': 0.0,
                            '62': 0.0,
                            '63': 0.0,
                            '64': 0.0,
                            '65': 0.0,
                            '66': 0.0,
                            '67': 0.0,
                            '68': 0.0,
                            '69': 0.002652519893899204,
                            '7': 0.002652519893899204,
                            '70': 0.0,
                            '71': 0.0,
                            '72': 0.0,
                            '73': 0.0,
                            '74': 0.0,
                            '75': 0.0,
                            '76': 0.0,
                            '77': 0.0,
                            '78': 0.0,
                            '79': 0.0,
                            '8': 0.002652519893899204,
                            '80': 0.0,
                            '81': 0.0,
                            '82': 0.0,
                            '83': 0.0,
                            '84': 0.0,
                            '85': 0.0,
                            '86': 0.0,
                            '87': 0.0,
                            '88': 0.0,
                            '89': 0.0,
                            '9': 0.002652519893899204,
                            '90': 0.0,
                            '91': 0.0,
                            '92': 0.0,
                            '93': 0.0,
                            '94': 0.0,
                            '95': 0.0,
                            '96': 0.0,
                            '97': 0.0,
                            '98': 0.0,
                            '99': 0.0},
                      '2': {'0': 0.9946949602122016, '1': 0.005305039787798408}}}
    Class number: 0
    Key: 0
    {'0': 0.9895459000326691, '1': 0.010454099967330937}
    Key: 1
    {'1': 0.5050637046716759,
     '10': 0.005227049983665469,
     '100': 0.0,
     '101': 0.0,
     '102': 0.0,
     '103': 0.0,
     '104': 0.0,
     '105': 0.0,
     '106': 0.0,
     '107': 0.0003266906239790918,
     '108': 0.0003266906239790918,
     '109': 0.0,
     '11': 0.00914733747141457,
     '110': 0.0,
     '111': 0.0,
     '112': 0.0,
     '113': 0.0,
     '114': 0.0,
     '115': 0.0,
     '116': 0.0,
     '117': 0.0,
     '118': 0.0,
     '119': 0.0,
     '12': 0.003266906239790918,
     '120': 0.0,
     '121': 0.0,
     '122': 0.0,
     '123': 0.0,
     '124': 0.0,
     '125': 0.0003266906239790918,
     '126': 0.0,
     '127': 0.0,
     '128': 0.0,
     '129': 0.0,
     '13': 0.005880431231623653,
     '130': 0.0,
     '131': 0.0,
     '132': 0.0003266906239790918,
     '133': 0.0,
     '134': 0.0003266906239790918,
     '135': 0.0,
     '136': 0.0003266906239790918,
     '137': 0.0,
     '138': 0.0003266906239790918,
     '139': 0.0003266906239790918,
     '14': 0.0022868343678536427,
     '140': 0.0,
     '141': 0.0003266906239790918,
     '142': 0.0,
     '143': 0.0,
     '144': 0.0003266906239790918,
     '145': 0.0,
     '146': 0.0,
     '147': 0.0,
     '148': 0.0,
     '149': 0.0,
     '15': 0.004900359359686377,
     '150': 0.0,
     '151': 0.0,
     '152': 0.0,
     '153': 0.0,
     '154': 0.0,
     '155': 0.0,
     '156': 0.0,
     '157': 0.0,
     '158': 0.0,
     '159': 0.0,
     '16': 0.0029402156158118264,
     '160': 0.0,
     '161': 0.0,
     '162': 0.0,
     '163': 0.0,
     '164': 0.0,
     '165': 0.0,
     '166': 0.0,
     '167': 0.0,
     '168': 0.0,
     '169': 0.0,
     '17': 0.0019601437438745506,
     '170': 0.0,
     '171': 0.0,
     '172': 0.0,
     '173': 0.0,
     '174': 0.0,
     '175': 0.0,
     '176': 0.0,
     '177': 0.0,
     '178': 0.0,
     '179': 0.0,
     '18': 0.0035935968637700096,
     '180': 0.0,
     '181': 0.0,
     '182': 0.0,
     '183': 0.0,
     '184': 0.0,
     '185': 0.0,
     '186': 0.0,
     '187': 0.0,
     '188': 0.0,
     '189': 0.0,
     '19': 0.0013067624959163672,
     '190': 0.0,
     '191': 0.0,
     '192': 0.0,
     '193': 0.0,
     '194': 0.0,
     '195': 0.0,
     '196': 0.0,
     '197': 0.0,
     '198': 0.0,
     '199': 0.0,
     '2': 0.12381574648807579,
     '20': 0.0013067624959163672,
     '200': 0.0,
     '201': 0.0,
     '202': 0.0,
     '203': 0.0,
     '204': 0.0,
     '205': 0.0,
     '206': 0.0,
     '207': 0.0,
     '208': 0.0,
     '209': 0.0,
     '21': 0.0029402156158118264,
     '210': 0.0,
     '211': 0.0,
     '212': 0.0,
     '22': 0.003266906239790918,
     '23': 0.001633453119895459,
     '24': 0.0013067624959163672,
     '25': 0.0013067624959163672,
     '26': 0.001633453119895459,
     '27': 0.0009800718719372753,
     '28': 0.0009800718719372753,
     '29': 0.0,
     '3': 0.06599150604377654,
     '30': 0.0,
     '31': 0.0006533812479581836,
     '32': 0.0003266906239790918,
     '33': 0.0006533812479581836,
     '34': 0.0006533812479581836,
     '35': 0.0003266906239790918,
     '36': 0.0009800718719372753,
     '37': 0.0009800718719372753,
     '38': 0.0006533812479581836,
     '39': 0.0006533812479581836,
     '4': 0.04083632799738648,
     '40': 0.0006533812479581836,
     '41': 0.0003266906239790918,
     '42': 0.0003266906239790918,
     '43': 0.0006533812479581836,
     '44': 0.0003266906239790918,
     '45': 0.0003266906239790918,
     '46': 0.0,
     '47': 0.0009800718719372753,
     '48': 0.0003266906239790918,
     '49': 0.0013067624959163672,
     '5': 0.045083306109114665,
     '50': 0.0003266906239790918,
     '51': 0.0,
     '52': 0.0003266906239790918,
     '53': 0.0,
     '54': 0.0006533812479581836,
     '55': 0.0003266906239790918,
     '56': 0.0,
     '57': 0.0,
     '58': 0.0,
     '59': 0.0,
     '6': 0.10192747468147664,
     '60': 0.0,
     '61': 0.0006533812479581836,
     '62': 0.0,
     '63': 0.0003266906239790918,
     '64': 0.0003266906239790918,
     '65': 0.0003266906239790918,
     '66': 0.0,
     '67': 0.0,
     '68': 0.0,
     '69': 0.0003266906239790918,
     '7': 0.013067624959163672,
     '70': 0.0003266906239790918,
     '71': 0.0,
     '72': 0.0003266906239790918,
     '73': 0.0,
     '74': 0.0,
     '75': 0.0003266906239790918,
     '76': 0.0,
     '77': 0.0,
     '78': 0.0,
     '79': 0.0003266906239790918,
     '8': 0.01110748121528912,
     '80': 0.0,
     '81': 0.0,
     '82': 0.0,
     '83': 0.0003266906239790918,
     '84': 0.0,
     '85': 0.0,
     '86': 0.0,
     '87': 0.0,
     '88': 0.0,
     '89': 0.0003266906239790918,
     '9': 0.016007840574975497,
     '90': 0.0,
     '91': 0.0006533812479581836,
     '92': 0.0,
     '93': 0.0,
     '94': 0.0,
     '95': 0.0,
     '96': 0.0,
     '97': 0.0003266906239790918,
     '98': 0.0,
     '99': 0.0}
    Key: 2
    {'0': 0.999673309376021, '1': 0.0003266906239790918}
    Class number: 1
    Key: 0
    {'0': 0.4907161803713528, '1': 0.5092838196286472}
    Key: 1
    {'1': 0.41909814323607425,
     '10': 0.002652519893899204,
     '100': 0.0,
     '101': 0.0,
     '102': 0.0,
     '103': 0.0,
     '104': 0.0,
     '105': 0.0,
     '106': 0.0,
     '107': 0.0,
     '108': 0.0,
     '109': 0.0,
     '11': 0.0,
     '110': 0.0,
     '111': 0.0,
     '112': 0.0,
     '113': 0.0,
     '114': 0.0,
     '115': 0.0,
     '116': 0.0,
     '117': 0.0,
     '118': 0.0,
     '119': 0.0,
     '12': 0.0,
     '120': 0.0,
     '121': 0.0,
     '122': 0.0,
     '123': 0.0,
     '124': 0.0,
     '125': 0.0,
     '126': 0.0,
     '127': 0.0,
     '128': 0.0,
     '129': 0.0,
     '13': 0.0,
     '130': 0.0,
     '131': 0.0,
     '132': 0.0,
     '133': 0.0,
     '134': 0.0,
     '135': 0.0,
     '136': 0.0,
     '137': 0.0,
     '138': 0.0,
     '139': 0.0,
     '14': 0.0,
     '140': 0.0,
     '141': 0.0,
     '142': 0.0,
     '143': 0.0,
     '144': 0.0,
     '145': 0.0,
     '146': 0.0,
     '147': 0.0,
     '148': 0.0,
     '149': 0.0,
     '15': 0.0,
     '150': 0.0,
     '151': 0.0,
     '152': 0.0,
     '153': 0.0,
     '154': 0.0,
     '155': 0.0,
     '156': 0.0,
     '157': 0.0,
     '158': 0.0,
     '159': 0.0,
     '16': 0.002652519893899204,
     '160': 0.0,
     '161': 0.0,
     '162': 0.0,
     '163': 0.0,
     '164': 0.0,
     '165': 0.0,
     '166': 0.0,
     '167': 0.0,
     '168': 0.0,
     '169': 0.0,
     '17': 0.0,
     '170': 0.0,
     '171': 0.0,
     '172': 0.0,
     '173': 0.0,
     '174': 0.0,
     '175': 0.0,
     '176': 0.0,
     '177': 0.0,
     '178': 0.0,
     '179': 0.0,
     '18': 0.0,
     '180': 0.0,
     '181': 0.0,
     '182': 0.0,
     '183': 0.0,
     '184': 0.0,
     '185': 0.0,
     '186': 0.0,
     '187': 0.0,
     '188': 0.0,
     '189': 0.0,
     '19': 0.0,
     '190': 0.0,
     '191': 0.0,
     '192': 0.0,
     '193': 0.0,
     '194': 0.0,
     '195': 0.0,
     '196': 0.0,
     '197': 0.0,
     '198': 0.0,
     '199': 0.0,
     '2': 0.4005305039787798,
     '20': 0.0,
     '200': 0.0,
     '201': 0.0,
     '202': 0.0,
     '203': 0.0,
     '204': 0.0,
     '205': 0.0,
     '206': 0.0,
     '207': 0.0,
     '208': 0.0,
     '209': 0.0,
     '21': 0.0,
     '210': 0.0,
     '211': 0.0,
     '212': 0.0,
     '22': 0.0,
     '23': 0.0,
     '24': 0.0,
     '25': 0.0,
     '26': 0.0,
     '27': 0.0,
     '28': 0.0,
     '29': 0.0,
     '3': 0.0636604774535809,
     '30': 0.002652519893899204,
     '31': 0.0,
     '32': 0.002652519893899204,
     '33': 0.0,
     '34': 0.0,
     '35': 0.0,
     '36': 0.0,
     '37': 0.0,
     '38': 0.0,
     '39': 0.0,
     '4': 0.021220159151193633,
     '40': 0.0,
     '41': 0.0,
     '42': 0.0,
     '43': 0.0,
     '44': 0.0,
     '45': 0.0,
     '46': 0.0,
     '47': 0.0,
     '48': 0.0,
     '49': 0.0,
     '5': 0.04509283819628647,
     '50': 0.0,
     '51': 0.0,
     '52': 0.0,
     '53': 0.002652519893899204,
     '54': 0.0,
     '55': 0.0,
     '56': 0.0,
     '57': 0.0,
     '58': 0.0,
     '59': 0.0,
     '6': 0.023872679045092837,
     '60': 0.002652519893899204,
     '61': 0.0,
     '62': 0.0,
     '63': 0.0,
     '64': 0.0,
     '65': 0.0,
     '66': 0.0,
     '67': 0.0,
     '68': 0.0,
     '69': 0.002652519893899204,
     '7': 0.002652519893899204,
     '70': 0.0,
     '71': 0.0,
     '72': 0.0,
     '73': 0.0,
     '74': 0.0,
     '75': 0.0,
     '76': 0.0,
     '77': 0.0,
     '78': 0.0,
     '79': 0.0,
     '8': 0.002652519893899204,
     '80': 0.0,
     '81': 0.0,
     '82': 0.0,
     '83': 0.0,
     '84': 0.0,
     '85': 0.0,
     '86': 0.0,
     '87': 0.0,
     '88': 0.0,
     '89': 0.0,
     '9': 0.002652519893899204,
     '90': 0.0,
     '91': 0.0,
     '92': 0.0,
     '93': 0.0,
     '94': 0.0,
     '95': 0.0,
     '96': 0.0,
     '97': 0.0,
     '98': 0.0,
     '99': 0.0}
    Key: 2
    {'0': 0.9946949602122016, '1': 0.005305039787798408}
    array([[5.289, 4.508],
           [4.368, 3.588],
           [4.329, 3.548],
           ...,
           [5.289, 4.508],
           [5.289, 4.508],
           [5.289, 4.508]])

```python
print(y_predict)
```

    [0 0 0 ... 0 0 0]

```python

```

    1147

```python

```


Prove that for integers $a,\;b,\;\dots$

(1) $(a, b) = 1, \; c | a, \; d | a \implies (c, d) = 1$

Suppose $(c, d) = e > 1$. Then $e | c$ and $c | a$ implies $e | a$; similarly $e | b$ so $(a, b) > 1$, a
contradiction, and therefore $(c, d) = 1$. $\;\;\;\boxdot$

(2) $(a, b) = (a, c) = 1 \implies (a, bc) = 1$

(3) $(a, b) = 1 \implies (a^n, b^k) = 1 \; \; \forall \; \; n \ge 1, k \ge 1$

(4) $(a, b) = 1 \implies (a + b, a - b) = 1 \; or \; 2$

(5) $(a, b) = 1 \implies (a + b, a^2 - ab + b^2) = 1 \; or \; 3$

(6) $(a, b) = 1, \; d|(a + b) \implies (a, d) = (b, d) = 1$

(7) A rational number $a/b$ with $(a, b) = 1$ is a *reduced fraction*. If the sum of two
reduced fractions is an integer, say $(a/b) + (c/d) = n$, prove that $|b| = |d|$.

(8) An integer is called *squarefree* if it is not divisible by the square of any prime. Prove that
for every $n \ge 1$ there exist uniquely determined $a > 0$ and $b > 0$ such that $n=a^2b$, where $b$ is *squarefree*.

...

(11) Prove that $n^4 + 4$ is composite if $n > 1$.

***Solution***

I first tried cases for the ones-digit. For example $n$ even gives $n^4 + 4$ also even and $n$ ending in
1, 3, 7 or 9 gives $n^4 + 4$ ending in 5.
However (particularly because the last case does not resolve in this manner) the right thing to try is
factoring $n^4 + 4$ in some obvious way:
Constants 1 and 4 or 2 and 2.

$n^4 + 4 = (n^2 + a \cdot n + 2) (n^2 + b \cdot n + 2)$

This gives $n^4 + b \cdot n^3 + 2 n^2 + a \cdot n^3 + a \cdot b \cdot n^2 + 2 \cdot a \cdot n + 2 n^2 + 2 \cdot b \cdot n + 4$

$n^4 + 4$ plus stuff that needs to be zero: $(b + a)\cdot n^3 + (4 + a \cdot b)\cdot n^2 + (2 \cdot (a + b))\cdot n$

This means $a = -b$ and $a \cdot b = -4$. Great: $a = 2$ and $b = -2$.

$n^4 + 4 = (n^2 + 2n + 2)(n^2 - 2n + 2)\;\;\;\;\boxdot$

```python
def pf(n):
    pfn, i = [], 2
    while i * i < n:
        while n%i == 0: pfn.append(i); n = n / i
        i = i + 1
    pfn.append(int(n))
    return pfn

def npf(n): return len(pf(n))

def isprime(n):
    if npf(n) == 1: return True
    return False

for a in range(3):
    s = a * 10 + 5
    t = s*s*s*s + 4
    if isprime(t): print(str(t) + ' is prime')
    else: print(str(t) + ' factors are ' + str(pf(t)))
```

    629 factors are [17, 37]
    50629 factors are [197, 257]
    390629 factors are [577, 677]

...

...

(20) Let $d = (826, 1890)$. Use the Euclidean algorithm to compute $d$, then express $d$ as a linear combination of 826 and 1890

Solution

$1890 = 826 \cdot 2 + 238$

$826 = 238 \cdot 3 + 112$

$238 = 112 \cdot 2 + 14$

$112 = 14 \cdot 8 + 0$

$d = 14$

$d = u \cdot 826 + v \cdot 1890$ or equivalently $1 = u \cdot 59 + v \cdot 135$

Taking $u$ positive it can take on values ${ 4, 9, 14, 19, \dots }$.

*--a miracle occurs--*

$(d = 14) = 254 \cdot 826 - 111 \cdot 1890$

```python
254*826-111*1890
```

    14

...

(19) F = 1, 1, 2, 3, 5, 8, ... where $\;a_{n+1} = a_n + a_{n-1}$. Prove $(a_n, \; a_{n+1})\;=\;1$.

(20)

(21)

(22) Prove (a, b) = (a + b, [a, b]).

Theorem R1: For $a > 1$ and $b > 1$ if $(a, b) = 1$ then

$
\begin{align}
\frac{1}{a} + \frac{1}{b}
\end{align}
$
is not an integer.

Proof: As (2, 2) = 2 one of $a$ or $b$ must be greater than 2.
Therefore $a + b < ab$ and

$
\begin{align}
0 < \frac{1}{a} + \frac{1}{b} = \frac{a+b}{ab} < 1
\end{align}
$

$\boxdot$

Now suppose $(a,b) = d$ where $a=\alpha d$, $b = \beta d$ and $(\alpha, \beta) = 1$.
Consider $(a + b, [a,b])$:

$
\begin{align}
(a + b, [a,b]) = ((\alpha + \beta)d, \frac{ab}{(a,b)}) =
((\alpha + \beta)d, \frac{\alpha\beta d^2}{d}) = d(\alpha + \beta, \alpha\beta)
\end{align}
$

For any prime factor $p$ of $\beta$ we can write $\beta = pq$ where $p$ does not divide into
$\alpha$. What is $(\alpha + \beta)/(\alpha \beta)$?

$
\begin{align}
\frac{\alpha + \beta}{\alpha \beta} = \frac{1}{\beta} + \frac{1}{\alpha}
\end{align}
$

which is not an integer.

Left off here: Show (alpha + beta, alpha beta) = 1 by dividing alpha q p into alpha + beta...

(23) Find $a, b > 0 \;\; \ni \;\; a + b = 5264 \; and \; [a, b] = 200,340$.

...

(30) If $n > 1$ prove

$
\begin{align}
\sum_{k=1}^{n} \frac{1}{k}
\end{align}
$

is not an integer.

```python

```


# Linear Regression

## The problem

Linear regression assumes that the predictor function is:

$$\hat{y} = \mathbf{w}^\top \cdot \phi(\mathbf{x}) $$

With the basis function $\phi$ equal to the identity function:

$$\hat{y} = \mathbf{w}^\top \cdot \mathbf{x} $$

defined by parameters $\mathbf{w}$.

Given the data $D = \{(\mathbf{x_i}, y_i)\}_{i=1}^m$, we can compute the performance of an hypothesis with the Mean Square Error:

$$MSE_D = \frac{1}{m} \sum_i (\hat{y} - y)  $$

## Closed-form solution

$$\min MSE_D \implies \nabla_{\mathbf{w}} MSE_D = 0$$

$$ \implies $$

$$ \mathbf{w} = (\mathbf{X}^T \mathbf{X})^{-1}  \mathbf{X}^T \mathbf{y} $$

## Probabilistic interpretation (conditional log-likelihood maximization)

If we put the problem if linear regression in term of maximum likelihood estimation, that is, assuming the data are distributed as:

- $p(y | x) = \mathcal{N}(y; \hat{y}(\mathbf{w},\mathbf{x}), \sigma^2)$

We have that:

- MSE minimization $\equiv$ log-likelihood maximization

Indeed:
$$
\begin{align}
\mathbf{w}_{ML} &= \arg\max_\mathbf{w} p(Y | \mathbf{X}; \mathbf{w}) \\
&= \arg\max_\mathbf{w} \prod_i p(y_i | \mathbf{x_i}; \mathbf{w}) \\
&= \arg\max_\mathbf{w} \sum_i\log p(y_i | \mathbf{x_i}; \mathbf{w}) \\
&= \arg\max_\mathbf{w} - ... - ... - \sum_i \frac{||\hat{y}_i - y_i||^2}{2\sigma^2}
\end{align}
$$

## Bayesian LR

Adding a prior over the parameters and doing MAP estimation is equivalent to common regularization technique (e.g. Lasso/Ridge regression), when the prior is distributed as Gaussian or Laplacian, respectively.

## Univariate Linear Regression

[Gradient Descent based](#Gradient-Descent-based)
[Closed form](#Closed-form)
[Using Sklearn](#Using-sklearn)
[Using Tensorflow](#Using-sklearn)

### Gradient Descent based

```python
import numpy as np
import matplotlib.pyplot as plt

def generate_linear_data(w, N):
    """generate data"""
    x = np.random.uniform(low=0, high=20, size=(N, 1))
    # one column of ones for the bias
    x = np.hstack([np.ones((N, 1)), x])
    # linear data with uniform error
    y = x.dot(w) + (2 * np.random.random(N) -1)
    y = y.reshape((N, 1))
    return x, y

x, y = generate_linear_data(np.array([0, 1]), 20)
print(x.shape, y.shape)
plt.scatter(x[:, 1], y)
plt.show()
```

```python
# our hypothesis
def hypothesis(w):
    """In this case, our hypothesis is a line"""
    return lambda x: x.dot(w)

def cost(w, x, y):
    # Mean squared error
    N = len(x)
    h = hypothesis(w)
    hx = h(x).reshape(y.shape)
    # print(x.shape, y.shape)
    # print("h(x)", h(x).shape)
    # print("np.square(h(x)-y)", (h_x -y).shape)
    # print(np.sum(np.square(h(x)-y)))
    return 1/(2*N) * np.sum(np.square(hx-y))

def cost_derivative(w, x, y):
    N = len(x)
    h = hypothesis(w)
    hx = h(x).reshape(y.shape)
    s = (hx-y)*x
    # print(hx.shape)
    # print(s.shape)
    return 1/N * np.sum(s, axis=0)
```

```python
# find parameters that minimize the cost function: Gradient Descent

def gradient_descent(x, y, alpha=0.01, num_steps=1000):
    """Given data, return two parameters, a and b, for the best fit"""
    w = np.random.random(2)

    for i in range(num_steps):
        c = cost(w, x, y)
        new_w = w - alpha * cost_derivative(w, x, y)
        if i % 1000 == 0:
            print("Iteration:\t", i, "\tcost:\t", c, "\tweights:\t", w)
        w = new_w
    return w
```

```python
w = gradient_descent(x,y, alpha=0.01, num_steps=10000)
print("Final weights:", w)
```

    Iteration:	 0 	cost:	 35.4853701003202 	weights:	 [0.34663164 0.18260219]
    Iteration:	 1000 	cost:	 0.16418959105036107 	weights:	 [-0.01697571  0.98774108]
    Iteration:	 2000 	cost:	 0.16411069414476226 	weights:	 [-0.03883738  0.98943947]
    Iteration:	 3000 	cost:	 0.1641104840020508 	weights:	 [-0.03996565  0.98952712]
    Iteration:	 4000 	cost:	 0.16411048344233348 	weights:	 [-0.04002388  0.98953165]
    Iteration:	 5000 	cost:	 0.1641104834408426 	weights:	 [-0.04002688  0.98953188]
    Iteration:	 6000 	cost:	 0.16411048344083853 	weights:	 [-0.04002704  0.98953189]
    Iteration:	 7000 	cost:	 0.1641104834408387 	weights:	 [-0.04002704  0.98953189]
    Iteration:	 8000 	cost:	 0.16411048344083878 	weights:	 [-0.04002704  0.98953189]
    Iteration:	 9000 	cost:	 0.16411048344083856 	weights:	 [-0.04002704  0.98953189]
    Final weights: [-0.04002704  0.98953189]

```python
x_arr = np.arange(20).reshape((20, 1))
x_arr = np.hstack([np.ones((20, 1)), x_arr])
y_arr = hypothesis(w)(x_arr)

plt.scatter(x[:, 1], y)
plt.plot(x_arr[:, 1], y_arr, c="r")
plt.show()
```

### Closed-form

```python
w_closed_form = np.linalg.inv(x.T.dot(x)).dot(x.T).dot(y).reshape(w.size)
print("Closed-form weights:", w_closed_form)
y_closed_form = hypothesis(w_closed_form)(x_arr)

plt.scatter(x[:, 1], y)
plt.plot(x_arr[:, 1], y_arr, c="r")
plt.plot(x_arr[:, 1], y_closed_form, c="b")
plt.show()
```

### Using Sklearn

```python
from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(x, y)
y_sklearn = model.predict(x_arr)
print(model.coef_)
plt.scatter(x[:, 1], y)
plt.plot(x_arr[:, 1], y_arr, c="r")
plt.plot(x_arr[:, 1], y_closed_form, c="b")
plt.plot(x_arr[:, 1], y_sklearn, c="y")
plt.show()
```

### Using Tensorflow

```python
import tensorflow as tf

N = len(x)
X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

W = tf.Variable(tf.random_uniform((2, )), name="weights", expected_shape=(2, ))

Y_Pred = tf.reshape(tf.tensordot(X, W, 1), (N, 1))
error = tf.square(Y_Pred - Y)
f_error = tf.reduce_sum(error)/len(x)  # + tf.norm(W)
optimizer = tf.train.GradientDescentOptimizer(0.001).minimize(f_error)

epochs = 1000
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print("Initial weights: ", W.eval())

    loss_expected = 0
    for i in range(epochs):

        sess.run(optimizer, {X: x, Y: y})

        loss_per_epoch = sess.run(f_error, {X: x, Y: y})

        # Allow the training to quit if we've reached a minimum
        if np.abs(loss_expected - loss_per_epoch) < 0.0000001:
            break
        loss_expected = loss_per_epoch

        if i%100==0:
            print(loss_expected, "weights: ", W.eval(session=sess))

    print("final loss: ", loss_expected, "weights: ", W.eval(session=sess))
    y_tensorflow = Y_Pred.eval(feed_dict={X: x_arr}, session=sess)
    plt.figure()
    plt.scatter(x[:, 1], y)
    plt.plot(x_arr[:, 1], y_arr, c="r")
    plt.plot(x_arr[:, 1], y_closed_form, c="b")
    plt.plot(x_arr[:, 1], y_sklearn, c="y")
    plt.plot(x_arr[:, 1], y_tensorflow, c="g")
    plt.show()

```

## References

- [Chapter 5 of deeplearningbook.org](http://www.deeplearningbook.org/contents/ml.html)

```python

```


```python
import sympy as sy
```

```python
w, x, y, z = sy.symbols('w x y z')
```

```python
roll = sy.atan2(2*(w*x + y*z), w**2 - x**2 - y**2 + z**2)
pitch = sy.asin(2*(-w*y - x*z))
yaw = sy.atan2(2*(w*z + x*y), w**2 + x**2 - y**2 - z**2)
```

```python
roll
```

$\displaystyle \operatorname{atan_{2}}{\left(2 w x + 2 y z,w^{2} - x^{2} - y^{2} + z^{2} \right)}$

```python
pitch
```

$\displaystyle - \operatorname{asin}{\left(2 w y + 2 x z \right)}$

```python
yaw
```

$\displaystyle \operatorname{atan_{2}}{\left(2 w z + 2 x y,w^{2} + x^{2} - y^{2} - z^{2} \right)}$

```python
print(sy.Matrix([roll]).jacobian([w,x,y,z]))
sy.Matrix([roll]).jacobian([w,x,y,z])
```

    Matrix([[2*w*(-2*w*x - 2*y*z)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2) + 2*x*(w**2 - x**2 - y**2 + z**2)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2), 2*w*(w**2 - x**2 - y**2 + z**2)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2) - 2*x*(-2*w*x - 2*y*z)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2), -2*y*(-2*w*x - 2*y*z)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2) + 2*z*(w**2 - x**2 - y**2 + z**2)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2), 2*y*(w**2 - x**2 - y**2 + z**2)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2) + 2*z*(-2*w*x - 2*y*z)/((2*w*x + 2*y*z)**2 + (w**2 - x**2 - y**2 + z**2)**2)]])

$\displaystyle \left[\begin{matrix}\frac{2 w \left(- 2 w x - 2 y z\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}} + \frac{2 x \left(w^{2} - x^{2} - y^{2} + z^{2}\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}} & \frac{2 w \left(w^{2} - x^{2} - y^{2} + z^{2}\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}} - \frac{2 x \left(- 2 w x - 2 y z\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}} & - \frac{2 y \left(- 2 w x - 2 y z\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}} + \frac{2 z \left(w^{2} - x^{2} - y^{2} + z^{2}\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}} & \frac{2 y \left(w^{2} - x^{2} - y^{2} + z^{2}\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}} + \frac{2 z \left(- 2 w x - 2 y z\right)}{\left(2 w x + 2 y z\right)^{2} + \left(w^{2} - x^{2} - y^{2} + z^{2}\right)^{2}}\end{matrix}\right]$

```python
print(sy.Matrix([pitch]).jacobian([w,x,y,z]))
sy.Matrix([pitch]).jacobian([w,x,y,z])
```

    Matrix([[-2*y/sqrt(1 - (2*w*y + 2*x*z)**2), -2*z/sqrt(1 - (2*w*y + 2*x*z)**2), -2*w/sqrt(1 - (2*w*y + 2*x*z)**2), -2*x/sqrt(1 - (2*w*y + 2*x*z)**2)]])

$\displaystyle \left[\begin{matrix}- \frac{2 y}{\sqrt{1 - \left(2 w y + 2 x z\right)^{2}}} & - \frac{2 z}{\sqrt{1 - \left(2 w y + 2 x z\right)^{2}}} & - \frac{2 w}{\sqrt{1 - \left(2 w y + 2 x z\right)^{2}}} & - \frac{2 x}{\sqrt{1 - \left(2 w y + 2 x z\right)^{2}}}\end{matrix}\right]$

```python
print(sy.Matrix([yaw]).jacobian([w,x,y,z]))
sy.Matrix([yaw]).jacobian([w,x,y,z])
```

    Matrix([[2*w*(-2*w*z - 2*x*y)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2) + 2*z*(w**2 + x**2 - y**2 - z**2)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2), 2*x*(-2*w*z - 2*x*y)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2) + 2*y*(w**2 + x**2 - y**2 - z**2)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2), 2*x*(w**2 + x**2 - y**2 - z**2)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2) - 2*y*(-2*w*z - 2*x*y)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2), 2*w*(w**2 + x**2 - y**2 - z**2)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2) - 2*z*(-2*w*z - 2*x*y)/((2*w*z + 2*x*y)**2 + (w**2 + x**2 - y**2 - z**2)**2)]])

$\displaystyle \left[\begin{matrix}\frac{2 w \left(- 2 w z - 2 x y\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}} + \frac{2 z \left(w^{2} + x^{2} - y^{2} - z^{2}\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}} & \frac{2 x \left(- 2 w z - 2 x y\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}} + \frac{2 y \left(w^{2} + x^{2} - y^{2} - z^{2}\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}} & \frac{2 x \left(w^{2} + x^{2} - y^{2} - z^{2}\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}} - \frac{2 y \left(- 2 w z - 2 x y\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}} & \frac{2 w \left(w^{2} + x^{2} - y^{2} - z^{2}\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}} - \frac{2 z \left(- 2 w z - 2 x y\right)}{\left(2 w z + 2 x y\right)^{2} + \left(w^{2} + x^{2} - y^{2} - z^{2}\right)^{2}}\end{matrix}\right]$

```python

```


```python
from pygom import SimulateOde, Transition as T
```

```python
state = ['S_v', 'S_h', 'I_v', 'I_h']
```

```python
param = ['beta_v', 'beta_h', 'mu_v', 'mu_h',
'lambda_v', 'lambda_h', 'gamma']
```

```python
t1 = T('S_h', 'lambda_h-mu_h*S_h-beta_h*S_h*I_v+gamma*I_h')
```

```python
t2 = T('S_v', 'lambda_v-mu_v*S_v-beta_v*S_v*I_h')
```

```python
t3 = T('I_h', 'beta_h*S_h*I_v-(mu_h+gamma)*I_h')
```

```python
t4 = T('I_v', 'beta_v*S_v*I_h-mu_v*I_v')
```

```python
ode = SimulateOde(state, param, ode=[t1,t2,t3,t4])
```

```python
from pygom.model.epi_analysis import R0
```

```python
ode = ode.get_unrolled_obj()
```

    /usr/local/lib/python3.7/site-packages/sympy/__init__.py:676: SymPyDeprecationWarning:

    importing sympy.matrices.matrices with 'from sympy import *' has been
    deprecated since SymPy 1.6. Use import sympy.matrices.matrices
    instead. See https://github.com/sympy/sympy/issues/18245 for more
    info.

      deprecated_since_version="1.6").warn()

```python
R0(ode, ['I_v','I_h'])
```

$\displaystyle \sqrt{\frac{\beta_{h} \beta_{v} \lambda_{h} \lambda_{v}}{\mu_{h} \mu_{v}^{2} \left(\gamma + \mu_{h}\right)}}$

```python
ode.get_transition_graph()
```

```python

```


# Números complexos

Neste notebook exploramos alguns aspectos dos números complexos. Especialmente, vamos falar da interferência entre duas ondas da mesma frequência.

Vimos nas aulas passadas que uma função cossenoidal geral, expressa por:

\begin{equation}
x(t) = \mathrm{Re}\left\{A\mathrm{e}^{-\mathrm{j}\phi} \ \mathrm{e}^{\mathrm{j}\omega t}    \right\}
\end{equation}
em que $\tilde{A} = A\mathrm{e}^{-\mathrm{j}\phi}$ é a amplitude complexa do cosseno e contêm as informações de magnitude, $A$, e fase, $\phi$. Esta amplitude complexa pode ser representada no plano complexo por:

<div>

</div>

Imagine que temos duas ondas, ***de mesma frequência***, interferindo entre si. Queremos calcular a onda resultante. Então, podemos somá-las. Se cada uma das ondas é descrita por um número complexo $z_1$ e $z_2$, do tipo:

\begin{equation}
\tilde{z}_1 = a_1 + \mathrm{j} b_1 = |\tilde{z}_1| \mathrm{e}^{\mathrm{j}\phi_1}
\end{equation}

e

\begin{equation}
\tilde{z}_2 = a_2 + \mathrm{j} b_2 = |\tilde{z}_2| \mathrm{e}^{\mathrm{j}\phi_2}
\end{equation}

Vamos fazer algumas análises a seguir

```python
# importar as bibliotecas necessárias
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 14})
```

```python
# tempo e frequência
t = np.linspace(-2, 2, 1000) # vetor temporal
freq = 1
w = 2*np.pi*freq

# onda 1
M1 = 1
phi_1 = np.deg2rad(0)
z1 = M1*np.exp(1j*phi_1)
xt1 = np.real(z1*np.exp(1j*w*t))

# onda 2
M2 = 1
phi_2 = np.deg2rad(0)
z2 = M2*np.exp(1j*phi_2)
xt2 = np.real(z2*np.exp(1j*w*t))
```

Temos agora 2 opções para calcular a onda resultante. A primeira é somar os sinais no domínio do tempo

\begin{equation}
x(t) = x_1(t) + x_2(t).
\end{equation}

A segunda é somar as amplitudes complexas

\begin{equation}
z = z_1 + z_2
\end{equation}
e então construir o sinal resultante $x(t) = \tilde{z}\mathrm{e}^{\mathrm{j}\omega t}$

```python
# Sinal resultante pela soma das amplitudes complexas
z = z1+z2
xt = np.real(z*np.exp(1j*w*t))

# Figura
plt.figure(figsize=(12,8))

plt.subplot(2,1,2)
plt.title('Sinal resultante')
plt.plot(t, xt, '-r', linewidth = 2, label = 'resultante')
plt.plot(t, xt1, '--b', linewidth = 1, label = r'$x_1(t)$')
plt.plot(t, xt2, '--b', linewidth = 1, label = r'$x_2(t)$')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel(r'$x(t)$ [-]')
plt.ylim((-2, 2))
plt.xlim((t[0], t[-1]))

plt.subplot(2,2,1)
plt.plot(t, xt1, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel(r'$x_1(t)$ [-]')
plt.ylim((-2, 2))
plt.xlim((t[0], t[-1]))

plt.subplot(2,2,2)
plt.plot(t, xt2, '-b', linewidth = 2)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel(r'$x_2(t)$ [-]')
plt.ylim((-2, 2))
plt.xlim((t[0], t[-1]))
plt.tight_layout()
plt.show()

```

Aula 7 - Numeros complexos/.ipynb_checkpoints/Numeros complexos-checkpoint.ipynb


```
Problem 1
```

$f(x+h)=f(x)+f'(x)h+\frac{1}{2}f''(x)h^2+\frac{1}{6}f'''(x)h^3+\frac{1}{24}f''''(x)h^4+O(h^5)$                       （1）
  $f(x-h)=f(x)-f'(x)h+\frac{1}{2}f''(x)h^2-\frac{1}{6}f'''(x)h^3+\frac{1}{24}f''''(x)h^4-O(h^5)$                      （2）
  $f(x+2h)=f(x)+f'(x)2h+f''(x)2h^2+\frac{4}{3}f'''(x)h^3+\frac{2}{3}f''''(x)h^4+O(h^5)$                                （3）
  $f(x-2h)=f(x)-f'(x)2h+f''(x)2h^2-\frac{4}{3}f'''(x)h^3+\frac{2}{3}f''''(x)h^4-O(h^5)$                              （4）

  SO let $\frac{2}{3}((1)-(2))-\frac{1}{12}((3)-(4))$, We can get:

$O(h^4)=\left|f'(x)-\frac{\frac{2}{3}(f(x+h)-f(x-h))-\frac{1}{12}(f(x+2h)-f(x-2h))}{h}\right|$

So we get a difference approximation.

```
Problem 2
```

(1)

Since f(x) is an even function.

   So $f(x)=f(-x)$\
      $f'(x)=-f'(-x)$\
      $f'(0)=-f'(0)$

   That is $f'(0)=0$

(2)

Let f(x) be expanded at 0 by Tayler:
   $f(0+h)=f(0)+f'(0)h+\frac{1}{2}f''(0)h^2+\frac{1}{6}f'''(0)h^3+O(h^4)$

   That is: $f(h)=\frac{1}{2}f''(0)h^2+O(h^4)$

   So $O(h^2)=\left|\frac{2f(h)}{h^2}-f''(h)\right|$

(3)

$f(h)=\frac{1}{2}f''(0)h^2+\frac{1}{24}h^4f(0)+O(h^6)$

$f(2h)=f''(0)2h^2+\frac{2h^4}{3}f''''(0)+O(h^6)$

Since $f''(0)-\frac{C1f(h)+C2f(h)}{h^2}=O(h^2)$

So we can get the system of equations:

\begin{equation}
\left\{
             \begin{array}{lr}
            1-\frac{C1}{2}-2C2=0 &  \\
             \frac{C1}{24}+\frac{2C2}{3}=0 &
             \end{array}
\right.
\end{equation}

So $C1=\frac{8}{3},C2=-\frac{1}{6}$

(4)

Since f(x) is an odd function.
So the odd power of the function at zero is zero.
So we are able to use the same above way to find the $f''(x)$ efficiently.


# Objetivos

Implementar e testar a solução analítica para o problema de regressão linear

Este notebook depende de módulos auxiliares que estão na pasta util/ mais alguns imports do Python

# Regressão linear: equações normais

Dado um dataset $\{(\mathbf{x}_{1}, y_{1}), \dots ,(\mathbf{x}_{N}, y_{N})\}$ onde $\mathbf{x}_i \in \mathbb{R}^{d}$ e $y_i \in \mathbb{R}$, queremos aproximar a função desconhecida $f:\mathbb{R}^{d} \rightarrow \mathbb{R}$ (lembrando que $y_i =f(\mathbf{x}_i)$) por meio de um modelo linear $h$:
$$
h(\mathbf{x}_{i}; \mathbf{w}, b) = \mathbf{w}^\top  \mathbf{x}_{i} + b
$$

Note que $h(\mathbf{x}_{i}; \mathbf{w}, b)$ é na verdade uma [transformação afim](https://en.wikipedia.org/wiki/Affine_transformation) de $\mathbf{x}_{i}$. Como em outros lugares, vamos usar o termo "linear" também para caracterizar uma transformação afim.

A saída de $h$ é uma transformação linear de $\mathbf{x}_{i}$. Usamos a notação $h(\mathbf{x}_{i}; \mathbf{w}, b)$ para deixar claro que $h$ é um modelo parametrizado, i.e., a transformação $h$ é definida pelos parâmetros $\mathbf{w}$ e $b$. Podemos pensar no vetor $\mathbf{w}$ como um vetor de *pesos* controlando o efeito de cada *feature* na predição.

Adicionando uma feature a mais na obsevação $\mathbf{x}_{i}$ (com o valor 1) -- coordenada artificial -- podemos simplificar a notação do modelo:

$$
h(\mathbf{x}_{i}; \mathbf{w}) = \hat{y}_{i} = \mathbf{w}^\top  \mathbf{x}_{i}
$$

Gostaríamos de encontrar os melhores parâmetros $\mathbf{w}$ de modo que a predição $\hat{y}_{i}$ seja a mais próxima de $y_{i}$ de acordo com alguma métrica de erro. Usando o *erro quadrárico médio* como tal métrica podemos obter a seguinte função de custo:

\begin{equation}
J(\mathbf{w}) = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_{i} - y_{i})^{2}
\end{equation}

Desse modo, a tarefa de achar a função $h$ mais próxima de $f$ se torna a tarefa de encontrar os valores de $\mathbf{w}$ para minimizar $J(\mathbf{w})$.

**Aqui vamos começar a explorar esse modelo olhando para um dataset bem simples**

```python
# all imports
import numpy as np
import time
from util.util import get_housing_prices_data, r_squared
from util.plots import plot_points_regression

%matplotlib inline
```

### O dataset

Os dados que vamos trabalhar são dados artificiais. Iremos gerar 100 observações com apenas uma *feature* e um valor asociado a cada uma delas. Podemos interpretar essas observações como sendo um par *(metros quadrados de um imóvel, preço desse imóvel em $)*. Nossa tarefa é construir um modelo que consiga predizer o valor dos imóveis, dadas as suas áreas.

```python
X, y = get_housing_prices_data(N=100)
```

### Plotando os dados

Acima temos algumas informações sobre os dados. Podemos também visualizar cada ponto.

```python
plot_points_regression(X,
                       y,
                       title='Real estate prices prediction',
                       xlabel="m\u00b2",
                       ylabel='$')
```

### Equações normais

Dados $f:\mathbb{R}^{n\times m} \rightarrow \mathbb{R}$ e $\mathbf{A} \in \mathbb{R}^{n\times m}$, definimos o gradiente de $f$ com respeito a $\mathbf{A}$ como:

\begin{equation*}
\nabla_{\mathbf{A}}f = \frac{\partial f}{\partial \mathbf{A}} =  \begin{bmatrix}
\frac{\partial f}{\partial \mathbf{A}_{1,1}} & \dots & \frac{\partial f}{\partial \mathbf{A}_{1,m}} \\
\vdots &  \ddots & \vdots \\
\frac{\partial f}{\partial \mathbf{A}_{n,1}} &  \dots & \frac{\partial f}{\partial \mathbf{A}_{n,m}}
\end{bmatrix}
\end{equation*}

Seja $\mathbf{X} \in \mathbb{R}^{N\times m}$ a matriz cujas linhas são as observações do dataset (também chamada de *design matrix*) e seja $\mathbf{y} \in \mathbb{R}^{N}$ o vetor contendo todos os valores de $y_{i}$ (i.e., $\mathbf{X}_{i,:} = \mathbf{x}_{i}$ e $\mathbf{y}_{i} =y_{i}$). É fácil checar que:

\begin{equation}
J(\mathbf{w}) = \frac{1}{N}(\mathbf{X}\mathbf{w} - \mathbf{y})^{T}(\mathbf{X}\mathbf{w} - \mathbf{y})
\end{equation}

Usando certos conceitos básicos de derivada com matrizes podemos chegar no gradiente de $J(\mathbf{w})$ com respeito a $\mathbf{w}$:

\begin{equation}
\nabla_{\mathbf{w}}J(\mathbf{w}) = \frac{2}{N} (\mathbf{X}^{T}\mathbf{X}\mathbf{w} -\mathbf{X}^{T}\mathbf{y})
\end{equation}

Assim, quando $\nabla_{\mathbf{w}}J(\mathbf{w}) = 0$ temos que

\begin{equation}
\mathbf{X}^{T}\mathbf{X}\mathbf{w} = \mathbf{X}^{T}\mathbf{y}
\end{equation}

Desse modo,

\begin{equation}
\mathbf{w} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}
\end{equation}

A solução dada por essas equações são conhecidas como **equações normais**. Note que esse tipo de solução tem um custo, pois conforme cresce o número de variáveis, o tempo da inversão da matriz fica proibitivo. Vale a pena ler [esse material](http://cs229.stanford.edu/notes/cs229-notes1.pdf) para ver o argumento com mais detalhes.

### Exercício
Usando apenas a biblioteca **NumPy** (uma introdução rápida a certas funcionalidades dessa biblioteca pode ser encontrada [aqui](http://cs231n.github.io/python-numpy-tutorial/)), complete as duas funções abaixo. Lembre que $\mathbf{X} \in \mathbb{R}^{N\times d}$; assim, será preciso adicionar uma componente com valor 1 a cada observação em $\mathbf{X}$ para realizar a computação descrita acima.

```python
def normal_equation_weights(X, y):
    """
    Calculates the weights of a linear function using the normal equation method.
    You should add into X a new column with 1s.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param y: regression targets
    :type y: np.ndarray(shape=(N, 1))
    :return: weight vector
    :rtype: np.ndarray(shape=(d+1, 1))
    """

    # START OF YOUR CODE:
    raise NotImplementedError("Falta implementar normal_equation_weights()")
    # END YOUR CODE

    return w
```

```python
# teste da função normal_equation_weights()

w = 0  # isto é desnecessário
w = normal_equation_weights(X, y)
print("Estimated w = ", w)
```

```python
def normal_equation_prediction(X, w):
    """
    Calculates the prediction over a set of observations X using the linear function
    characterized by the weight vector w.
    You should add into X a new column with 1s.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param w: weight vector
    :type w: np.ndarray(shape=(d+1, 1))
    :param y: regression prediction
    :type y: np.ndarray(shape=(N, 1))
    """

    # START OF YOUR CODE:
    raise NotImplementedError("Falta implementar normal_equation_prediction()")
    # END YOUR CODE

    return prediction
```

Você pode usar a métrica [$R^2$](https://pt.wikipedia.org/wiki/R%C2%B2) para ver o quão bem o modelo linear está se ajustando aos dados.

**Nesse caso $𝑅^2$ tem que estar próximo de 0.5.**

```python
# teste da função normal_equation_prediction()
prediction = normal_equation_prediction(X, w)
r_2 = r_squared(y, prediction)
plot_points_regression(X,
                       y,
                       title='Real estate prices prediction',
                       xlabel="m\u00b2",
                       ylabel='$',
                       prediction=prediction,
                       legend=True,
                       r_squared=r_2)
```

## Testes adicionais

Vamos fazer a predição para $x=650$

```python
# usando a função de predição
x = np.asarray([650]).reshape(1,1)
prediction = normal_equation_prediction(x, w)
print("Area = %.2f  Predicted price = %.4f" %(x[0], prediction))

# de forma mais direta
y = np.dot(np.asarray((1,x)), w)
print("Area = %.2f  Predicted price = %.4f" %(x, y))

```

## Efeito do número de amostras e dimensão dos dados

Varie o número de amostras $N$ e veja como varia o tempo de processamento.

Teste o seu código para dados nos quais $𝑑>1$.

```python
# Teste para diferentes valores de N
X, y = get_housing_prices_data(N=1000000)
init = time.time()
w = normal_equation_weights(X, y)
prediction = normal_equation_prediction(X,w)
init = time.time() - init

print("Tempo de execução = {:.8f}(s)".format(init))
```

```python
# Teste para dados de dimensão d>1
```