# Quaternion Triple Products and Distance

by Doug Sweetser, sweetser@alum.mit.edu - please feel free to email

In this IPython notebook, efforts will be made to understand quaternion triple products and how they are related to distances in space and intervals in space-time as seen in special relativity. Rather than follow a historical story, I will try a more abstract approach. Initialize a few tools.

```python
%%capture
%matplotlib inline
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt

# To get equations the look like, well, equations, use the following.
from sympy.interactive import printing
printing.init_printing(use_latex=True)
from IPython.display import display

# Tools for manipulating quaternions.
import Q_tools as qt;
```

## Spatial Rotations

Define a triple product function modeled on what it takes to do a spatial rotation, $P R P^*$, where $R$ is a quaternion to be spatially rotated and $P$ is a quaternion parameter to do said rotation.

```python
def triple_sandwich(r, p=qt.QH([1, 0, 0, 0])):
    """A function that takes 2 quaternions but does a triple product. The default value for P leaves R unchanged."""

    return p.product(r.product(p.conj()))
```

```python
t, x, y, z = sp.symbols("t x y z")
s, u, v, w = sp.symbols("s u v w")

R = qt.QH([t, x, y, z])
P = qt.QH([s, u, v, w])
RP_sandwich = triple_sandwich(R, P)
sp.simplify(RP_sandwich.t)
```

The first term is just the norm of the parameter $P$ times the scalar value of $R$, how simple! Rotating a value is complicated.

```python
sp.simplify(RP_sandwich.x)
```

Show the interval of $R$ is unchanged up to the norm of the parameter $P$:

```python
sp.simplify(sp.factor(RP_sandwich.square().t))
```

The interval will be invariant so long as the norm of the parameter $P$ is equal to one. A common way to do this is to use sine and cosine functions due to the trig identity $\sin^2(\theta) + \cos^2(\theta) = 1$.

```python
def triple_trig_z(r, a):
    """A rotation around the z axis only by the double angle of a."""

    return triple_sandwich(r, qt.QH([sp.cos(a), 0, 0, sp.sin(a)]))

def is_quadratic(r):
    """Tests if the the first term of the square of a quaternion is equal to t^2 - x^2 - y^2 - z^2."""

    r2 = r.square()
    simple_r2 = sp.simplify(r2.t)
    it_is = ((simple_r2 == 1.0*t**2 - 1.0*x**2 - 1.0*y**2 - 1.0*z**2)
             or (simple_r2 == t**2 - x**2 - y**2 - z**2))

    if it_is:
        display(t**2 - x**2 - y**2 - z**2)
    else:
        display(simple_r2)

    return it_is
```

```python
a = sp.Symbol('a')
display(sp.simplify(triple_trig_z(R, a).t))
display(sp.simplify(triple_trig_z(R, a).x))
display(sp.simplify(triple_trig_z(R, a).y))
display(sp.simplify(triple_trig_z(R, a).z))
is_quadratic(triple_trig_z(R, a))
```

An important thing to notice is that rotations work for arbitrarily small values of an angle.

```python
display(sp.simplify(triple_trig_z(R, 0.01).t))
display(sp.simplify(triple_trig_z(R, 0.01).x))
display(sp.simplify(triple_trig_z(R, 0.01).y))
display(sp.simplify(triple_trig_z(R, 0.01).z))
is_quadratic(triple_trig_z(R, 0.01))
```

This is relevant to the fact that the group $SO(3)$ is a compact group. It is easy to visualize the example above: it is a circle in the $xy$ plane with $t$ and $z$ unaltered. Circles are sets of points where the "next" point is an arbitrarily short distance away.

Can we create a function that can take _any_ quaternion parameter $P$ yet still always generate another member of the group $SO(3)$? This can be done using the inverse of a quaternion which is the conjugate of a quaternion divided by the norm squared. Groups are about binary operations on a set. The binary operation can be a composite function, where the results of one rotation are fed into another.

```python
def next_rotation(r, p=qt.QH([1, 0, 0, 0])):
    """Generates another member of the rotation group given a quaternion parameter P."""

    return p.product(r.product(p.invert()))

def composite_rotation(r, p1=qt.QH([1, 0, 0, 0]), p2=qt.QH([1, 0, 0, 0])):
    """A composite function of next_rotation."""

    return next_rotation(next_rotation(r, p1), p2)
```

```python
display(sp.simplify(composite_rotation(R, qt.QH([s, u, v, w])).t))
display(sp.simplify(composite_rotation(R, qt.QH([s, u, v, w])).x))
is_quadratic(composite_rotation(R, qt.QH([s, u, v, w])))
```

The next_rotation function can use any quaternion parameter $P$ as input and create another member of the group. This does not mean that rotations have four degrees of freedom. There is an equivalence relation involved since the product of a quaternion with its inverse has a norm of one. This algebraic constraint means the composite_rotation function has $4-1=3$ degrees of freedom.

The composite_rotation function could be used to show that there is a real-valued quaternion representation of the compact Lie group $SO(3)$. Since it is well known quaternions can do this, such an effort will be skipped.

## Other Triple Products Lead to More Than Just Rotations

Other triple products are possible. For example, the two quaternions could be on the same side. A number of years ago, a search for a real-valued quaternion function that could do a Lorentz boost turned up this difference between two one-sided triples, $ \frac{1}{2}((P P R)^* - (P^* P^* R)^*)$:

```python
def triple_2_on_1(r, p=qt.QH([1, 0, 0, 0])):
    """The two are on one side, minus a different two on one side."""

    ppr = p.product(p.product(r)).conj()
    pcpcr = p.conj().product(p.conj().product(r)).conj()
    pd = ppr.dif(pcpcr)
    pd_ave = pd.product(qt.QH([1/2, 0, 0, 0]))
    return pd_ave
```

```python
rq_321 = triple_2_on_1(R, P)
display(sp.simplify(rq_321.t))
display(sp.simplify(rq_321.x))
display(sp.simplify(rq_321.y))
display(sp.simplify(rq_321.z))
```

If $s=0$, then triple_2_on_1 would contribute nothing.

Explore the hyperbolic sine and cosines:

```python
phx = qt.QH([sp.cosh(a), sp.sinh(a), 0, 0])
ppr = triple_2_on_1(R, phx)
display(sp.simplify(ppr.t))
```

This is promising for doing a Lorentz boost. There is a direct link between hyperbolic trig functions and the relativistic velocity $\beta$ and stretch factor $\gamma$ of special relativity.

$$\gamma = \cosh(\alpha)$$
$$\beta \gamma = \sinh(\alpha)$$

The trig functions are based on a circle in the plane, while the hyperbolic trig functions start with hyperbolas. The definitions are remarkably similar:

$$\sin(\alpha) = \frac{e^{i \alpha} - e^{-i \alpha}}{2 i}$$

$$\cos(\alpha) = \frac{e^{i \alpha} + e^{-i \alpha}}{2 i}$$

$$\sinh(\alpha) = \frac{e^{\alpha} - e^{-\alpha}}{2}$$

$$\cosh(\alpha) = \frac{e^{\alpha} + e^{-\alpha}}{2}$$

The hyperbolic trig functions oddly are "more real", never needing an imaginary factor. The hyperbola of the hyperbolic cosine does touch the unit circle at its minimum, suggesting a solitary link to the trig functions.

Combine the three triples and test if they do all the work of a Lorentz boost:
$$\rm{triple-triple}(R, P) \equiv P R P^* + \frac{1}{2}((P P R)^* - (P^* P^* R)^*)$$

```python
def triple_triple(r, p=qt.QH([1, 0, 0, 0])):
    """Use three triple products for rotations and boosts."""

    # Note: 'qtype' provides a record of what algrabric operations were done to create a quaternion.
    return triple_sandwich(r, p).add(triple_2_on_1(r, p), qtype="triple_triple")
```

Can this function do a rotation? If the first value of $P$ is equal to zero, then the two one-sided triple terms, $PPR$, will make no contribution, leaving the triple sandwich $PRP^*$. So long as the norm is equal to unity, then spatial rotations result. Do a rotation:

```python
jk = qt.QH([0, 0, 3/5, 4/5])
display(sp.simplify(triple_triple(R, jk).t))
display(sp.simplify(triple_triple(R, jk).x))
display(sp.simplify(triple_triple(R, jk).y))
display(sp.simplify(triple_triple(R, jk).z))
is_quadratic(triple_triple(R, jk))
```

Something important has changed going from the regular trig functions to these hyperbolic functions for rotations. The requirements that the first term must be zero while the other three terms are normalized to unity means that one cannot go an arbitrarily small distance away and find another transformation. If one wants a product of rotations, those rotations must be at right angles to each other.

```python
Qi, Qj, Qk = qt.QH([0, 1, 0, 0]), qt.QH([0, 0, 1, 0]), qt.QH([0, 0, 0, 1])
print(triple_triple(triple_triple(R, Qi), Qj))
print(triple_triple(R, Qi.product(Qj)))
```

    (t, -x, -y, z) triple_triple
    (t, -x, -y, z) triple_triple

The fact that one cannot find a super close neighbor is a big technical change.

What is so special about setting the first term equal to zero? Is there a more general form? Perhaps all that is needed is for the first term of the square to be equal to negative one. Test this out:

```python
minus_1 = qt.QH([2, 2, 1, 0])
print(minus_1.square().t)
display((triple_triple(R, minus_1).t, triple_triple(R, minus_1).x, triple_triple(R, minus_1).y, triple_triple(R, minus_1).z))
is_quadratic(triple_triple(R, minus_1))
```

To be honest, this came as a surprise to me. Notice that the value for time changes, so a rotation is getting mixed in with a boost. This sort of mixing of rotations and boosts is known to happen when one does two boosts, one say along $x$, the other along $y$. Now we can say a similar thing is possible for rotations. If there scalar is zero then one gets a pure spatial rotation. When that is not the case, there is a mixture of rotations and boosts.

Demonstrate that a boost along the $x$ axis works.

```python
bx = qt.QH([sp.cosh(a), sp.sinh(a), 0, 0])
display(sp.simplify(bx.square().t))
display(sp.simplify(triple_triple(R, bx).t))
display(sp.simplify(triple_triple(R, bx).x))
display(sp.simplify(triple_triple(R, bx).y))
display(sp.simplify(triple_triple(R, bx).z))
is_quadratic(triple_triple(R, bx))
```

Perfect. It was this result that began my investigation of triple_triple quaternion products. This is what the boost looks like using gammas and betas: $$(\gamma t - \gamma \beta x, \gamma x - \gamma \beta t, y, z)$$

The first term of the square of the hyperbolic parameter $P=bx$ is equal to positive one. So long as the triple_triple function is fed a quaternion parameter $P$ whose first term of the square has an absolute value of one, the interval is invariant. That is surprisingly simple.

Note the double angle in the hyperbolic trig function that appeared earlier for rotations.

## Spatial Reflection and Time Reversal

For a spatial reflection, just one spatial term flips signs. The first term of the square will not be altered. Yet the triple_triple function cannot flip only one sign. It can flip two terms. Thus, using just the triple_triple function one can go from all positive, to two positive-two negative, to all negative terms, but never one or three negative terms starting from an all positive quaternion $R$. The conjugate operator can do odd sign changes. Do a spatial reflection on $x$ only by rotating using $i$ and using the conjugate operator like so:

```python
x_reflection = triple_triple(R, Qi).conj()
print(x_reflection)
is_quadratic(x_reflection)
```

Time reversal also cannot be done using triple_triple. The parameter $P$ is used twice, so its sign is of no consequence for the scalar in $R$. The entire quaternion $R$ must be multiplied by $-1$ then take a conjugate like so:

```python
t_reversal = triple_triple(R).conj().product(qt.QH([-1, 0, 0, 0], qtype="sign_flip"))
print(t_reversal)
is_quadratic(t_reversal)
```

Rotations and boosts do not do the work of time reversal. Time reversal requires different algebraic tricks.

## Fixing the Limitations of the Triple_Triple Function

The triple_triple function must be fed quaternions whose square is either exactly equal to plus or minus one. Create a function that can take in _any_ quaternion as a parameter and generate the next quadratic. The function must be scaled to the square root of the first term of the quaternion parameter $P$ squared. Expand the parameters so both spatial reflections and time reversals can be done.

If the parameter $P$ is light-like, it cannot be used to do a boost. Feed the triple_triple function a light-like quaternion and it will always return zero. Light-like quaternions can do rotations. The next_rotation function is up to the task.

```python
def next_quadratic(r, p=qt.QH([1, 0, 0, 0]), conj=False, sign_flip=False):
    """Generates another quadratic using a quaternion parameter p,
    if given any quaternion and whether a conjugate or sign flip is needed."""

    pt_squared = p.square().t

    # Avoid using sp.Abs() so equations can be simplified.
    if isinstance(pt_squared, (int, float)):
        if pt_squared < 0:
            pt_squared *= -1
    else:
        if pt_squared.is_negative:
            pt_squared *= -1

    sqrt_pt_squared = sp.sqrt(pt_squared)

    # A light-like parameter P can rotate but not boost R.
    if sqrt_pt_squared == 0:
        rot_calc = next_rotation(r, p)
    else:
        p_normalized = p.product(qt.QH([1/sqrt_pt_squared, 0, 0, 0]))
        rot_calc = triple_triple(r, p_normalized)

    if conj:
        conj_calc = rot_calc.conj()
    else:
        conj_calc = rot_calc

    if sign_flip:
        sign_calc = conj_calc.product(qt.QH([-1, 0, 0, 0]))
    else:
        sign_calc = conj_calc

    calc_t = sp.simplify(sp.expand(sign_calc.t))
    calc_x = sp.simplify(sp.expand(sign_calc.x))
    calc_y = sp.simplify(sp.expand(sign_calc.y))
    calc_z = sp.simplify(sp.expand(sign_calc.z))

    return qt.QH([calc_t, calc_x, calc_y, calc_z], qtype="L")
```

```python
display(sp.simplify(next_quadratic(R, P, True, True).t))
display(sp.simplify(next_quadratic(R, P, True, True).x))
is_quadratic(next_quadratic(R, P, True, True))
```

No matter what values are used for the parameter $P$, the next_quadratic function will preserve the interval of $R$. Even a light-like interval works:

```python
print(next_quadratic(R, qt.QH([s, s, 0, 0])))
is_quadratic(next_quadratic(R, qt.QH([s, s, 0, 0])))
```

Notice how the $y$ and $z$ terms flip positions, but the squaring process will put both into their proper spots in the first term of the square.

## The Lorentz Group and Functional Composition with the next_quadratic Function

The Lorentz group is all possible ways to transform an event in space-time yet preserve the quadratic form:
$$(t, x, y, z) \rightarrow t^2 - x^2 - y^2 - z^2$$
The elements of the group are the tuples (t, x, y, z) but not the rotation angles, boost velocities, conjugation and sign flips.

A group is defined as a binary operation on a set of elements that has 4 qualities:
1. Closure
1. An inverse exists
1. There is an identity element
1. Associative

The next_quadratic function acts on one element of the group. The binary operation is a composite function built from two next_quadratic functions. Take the result of one action of the next_quadratic function, and have that result go into another round of the next_quadratic function.

```python
def composite_quadratic(r, p1=qt.QH([1, 0, 0, 0]), p2=qt.QH([1, 0, 0, 0]), conj1=False, conj2=False, sign_flip1=False, sign_flip2=False):
    """A composite function for the next_quadratic function."""

    return next_quadratic(next_quadratic(r, p1, conj1, sign_flip1), p2, conj2, sign_flip2)
```

```python
print(composite_quadratic(R))
is_quadratic(composite_quadratic(R))
print(composite_quadratic(R, Qi, Qj, True, True, True, False))
is_quadratic(composite_quadratic(R, Qi, Qj, True, True, True, False))
print(composite_quadratic(R, minus_1, Qj, False, True, False, True))
is_quadratic(composite_quadratic(R, minus_1, Qj, False, True, False, True))
print(composite_quadratic(R, bx, P, True, False, True, False))
is_quadratic(composite_quadratic(R, bx, P, True, False, True, False))
print(composite_quadratic(composite_quadratic(R, bx, bx)))
is_quadratic(composite_quadratic(composite_quadratic(R, bx, bx)))
```

Each of these composite functions generates exactly the same quadratic as required to be part of the Lorentz group. These five examples argue for closure: every possible choice for what one puts in the composite_quadratic function will have the same quadratic. I don't have the math skills to prove closure (unless one thinks the earlier general case is enough).

Quaternions are a division algebra. As such, it is reasonable to expect an inverse to exist. Look for one for the $Qi$, $Qk$ parameter case:

```python
print(composite_quadratic(R, Qi, Qj, True, True, True, False))
print(composite_quadratic(composite_quadratic(R, Qi, Qj, True, True, True, False), Qk))
```

    (-t, x, y, -z) L
    (-t, -x, -y, -z) L

Close, but not quite. Add a sign_flip.

```python
print(composite_quadratic(composite_quadratic(R, Qi, Qj, True, True, True, False), Qk, sign_flip1=True))
```

    (t, x, y, z) L

The is back where we started with the quaternion $R$. Again, this is just an example and not a proof. Some inverses are easier to find than others like pure rotations or pure boosts with a rotation or opposite velocity.

The identity composition was shown to do its fine work in the first composite_quadratic(R) example.

Composite functions are associative, at least according to wikipedia.

## The Difference Between composite_rotation and composite_quadratic

Both of these composite functions call another function twice, next_rotation and next_quadratic respectively. Both functions do a normalization. The next_rotation normalizes to the norm squared which can be zero if the parameter $P$ is zero, otherwise it is positive. The next_rotation function always does one thing, $P R P^{-1}$. The next_quadratic normalizes to the first term of the square of parameter $P$. That value can be positive, negative, or zero. When the first term of the square is positive or negative, the next_quadratic function treats both cases identically. Three triple quaternion products are used, $P R P^* + \frac{1}{2}((P P R)^* - (P^* P^* R)^*)$. The first term is identical to a rotation so long as the norm is equal to one. Otherwise, it is off just by a scaling factor. The difference happens when it is zero which indicates the properties of light come into play. It is the lightcone that separates time-like events from space-like events. For a time-like value of the parameter $P$, the triple-triple returns zero which is not a member of the group. If one uses the first triple, no matter what its norm of light-like parameter $P$ happens to be, the resulting $R->R'$ remains in the group. The rotation group $SO(3)$ is compact, while the Lorentz group $O(1, 3)$ is not. The change in algebra needed for light-light parameter $P$ may be another way to view this difference.

## Degrees of Freedom

The typical representation of the Lorentz group $O(1, 3)$ says there are six independent variables needed to represent the Lorentz group: three for rotations and three for boosts. Yet when one does two boosts in different directions, it is a mix between a boost and a rotation. This suggests there is no such thing as a completely separate notion of rotations and boosts, that they have a capacity to mix. If true, that decreases the degrees of freedom.

Two spacial rotations will result in spacial rotation:

```python
print(composite_quadratic(R, qt.QH([0, 1,0,1]), qt.QH([0, 1,1,0])))
is_quadratic(composite_quadratic(R, qt.QH([0, 1,0,1]), qt.QH([0, 1,1,0])))
```

Notice that the value of the first squared term is negative. That value gets normalized to negative one in the composite_quadratic function (via the next_quadratic function that gets called twice). What makes these rotations be only spacial is the zero in the first position of the parameter $P$. It is easy enough to look at situations where the first term of the square is negative, and the first term of the parameter is not equal to zero:

```python
print(composite_quadratic(R, qt.QH([4, 5,0,0])))
is_quadratic(composite_quadratic(R, qt.QH([4, 5,0,0])))
```

This is both a boost and a rotation. The boost effect can be seen in the first and second terms where there is a positve and negative term (the negative being the term that "doesn't belong", seeing the $x$ in the first term and $t$ in the second). The rotation appears in the sign flips for $y$ and $z$. If the 4 and 5 are switched, there is no rotation of these terms:

```python
print(composite_quadratic(R, qt.QH([5, 4,0,0])))
```

    (4.55555555555556*t - 4.44444444444444*x, -4.44444444444444*t + 4.55555555555556*x, y, z) L

The first two terms are exactly the same. Now the last two terms don't flip signs because there is no rotation. Both the (4, 5) and (5, 4) parameter composites will have the same first term for the square. This real-valued quaternion representation makes it possible to see.

At first blush, one looks into the next_quadratic function and sees six degrees of freedom: four for the quaternion parameter $P$, one for the conjugate operator and one for the sign_flip. These last two are needed to generate spatial reflection and time reversal. The quaternion parameter $P$ normalizes to the first term of the square of the quaternion parameter $P$. This means that once three of the values are chosen, then the value of the fourth one is set by this algebraic constraint. The same thing happens with the composite_rotation function defined earlier: a 4D quaternion may go in, but they way it gets normalized means there is an equivalence class to those quaternions that have a norm of one, and thus only 3 degrees of freedom. Representing the Lorentz group with only five degrees of freedom with this real-valued quaternion representation would be an interesting result if it can be rigorously proved.


# Fast-Slow系
\begin{equation}
\begin{array}{ll}
\dot{x}&=\varepsilon (y+x-\frac{x^3}{3})\\
\dot{y}&=-\frac{1}{\varepsilon}x
\end{array}
\label{fastslow_vdp}
\end{equation}

```python
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set('poster', 'whitegrid', 'dark', rc={"lines.linewidth": 2, 'grid.linestyle': '--'})
```

```python
def vdp(t, x, eps):
    return [eps * (x[1] + x[0] - x[0]**3/3), -x[0]/eps]
```

```python
eps=10
t0 = 0.0
```

```python
t1 = 100.0
x0 = [0.0, 1.5]
s0 = solve_ivp(vdp, [t0, t1], x0, args=([eps]),dense_output=True)
```

```python
T = np.linspace(t0, t1, 10000)
sol = s0.sol(T)
fig = plt.figure(figsize=(9,6))

ax = fig.add_subplot(111)
ax.set_xlabel("$x$")
ax.set_ylabel("$y$")
ax.set_xlim(-3,3)
ax.set_ylim(-2,2)
ax.set_yticks([-2,-1,0,1,2])
X = np.linspace(-3,3,256)
ax.plot(np.zeros(2), np.linspace(-2,2,2), '-', color='gray')
ax.plot(X, -X + X**3/3, '--', color='gray')
ax.plot(sol.T[:,0], sol.T[:,1], '-k')
# plt.savefig("fastslow.pdf", bbox_inches='tight')
```


# SymPy: Open Source Symbolic Mathematics

This notebook uses the [SymPy](http://sympy.org) package to perform symbolic manipulations,
and combined with numpy and matplotlib, also displays numerical visualizations of symbolically
constructed expressions.

We first load sympy printing extensions, as well as all of sympy:

```python
from IPython.display import display

from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')

import sympy as sym
x, y, z = sym.symbols("x y z")
k, m, n = sym.symbols("k m n", integer=True)
f, g, h = map(sym.Function, 'fgh')
```

Elementary operations

```python
sym.Rational(3,2)*sym.pi + sym.exp(sym.I*x) / (x**2 + y)
```

```python
sym.exp(sym.I*x).subs(x,sym.pi).evalf()
```

```python
e = x + 2*y
```

```python
sym.srepr(e)
```

```python
sym.exp(sym.pi * sym.sqrt(163)).evalf(50)
```

Algebra

```python
eq = ((x+y)**2 * (x+1))
eq
```

```python
sym.expand(eq)
```

```python
a = 1/x + (x*sym.sin(x) - 1)/x
a
```

```python
sym.simplify(a)
```

```python
eq = sym.Eq(x**3 + 2*x**2 + 4*x + 8, 0)
eq
```

```python
sym.solve(eq, x)
```

```python
a, b = sym.symbols('a b')
sym.Sum(6*n**2 + 2**n, (n, a, b))
```

Calculus

```python
sym.limit((sym.sin(x)-x)/x**3, x, 0)
```

```python
(1/sym.cos(x)).series(x, 0, 6)
```

```python
sym.diff(sym.cos(x**2)**2 / (1+x), x)
```

```python
sym.integrate(x**2 * sym.cos(x), (x, 0, sym.pi/2))
```

```python
eqn = sym.Eq(sym.Derivative(f(x),x,x) + 9*f(x), 1)
display(eqn)
sym.dsolve(eqn, f(x))
```

# Illustrating Taylor series

We will define a function to compute the Taylor series expansions of a symbolically defined expression at
various orders and visualize all the approximations together with the original function

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```

```python
# You can change the default figure size to be a bit larger if you want,
# uncomment the next line for that:
#plt.rc('figure', figsize=(10, 6))
```

```python
def plot_taylor_approximations(func, x0=None, orders=(2, 4), xrange=(0,1), yrange=None, npts=200):
    """Plot the Taylor series approximations to a function at various orders.

    Parameters
    ----------
    func : a sympy function
    x0 : float
      Origin of the Taylor series expansion.  If not given, x0=xrange[0].
    orders : list
      List of integers with the orders of Taylor series to show.  Default is (2, 4).
    xrange : 2-tuple or array.
      Either an (xmin, xmax) tuple indicating the x range for the plot (default is (0, 1)),
      or the actual array of values to use.
    yrange : 2-tuple
      (ymin, ymax) tuple indicating the y range for the plot.  If not given,
      the full range of values will be automatically used.
    npts : int
      Number of points to sample the x range with.  Default is 200.
    """
    if not callable(func):
        raise ValueError('func must be callable')
    if isinstance(xrange, (list, tuple)):
        x = np.linspace(float(xrange[0]), float(xrange[1]), npts)
    else:
        x = xrange
    if x0 is None: x0 = x[0]
    xs = sym.Symbol('x')
    # Make a numpy-callable form of the original function for plotting
    fx = func(xs)
    f = sym.lambdify(xs, fx, modules=['numpy'])
    # We could use latex(fx) instead of str(), but matploblib gets confused
    # with some of the (valid) latex constructs sympy emits.  So we play it safe.
    plt.plot(x, f(x), label=str(fx), lw=2)
    # Build the Taylor approximations, plotting as we go
    apps = {}
    for order in orders:
        app = fx.series(xs, x0, n=order).removeO()
        apps[order] = app
        # Must be careful here: if the approximation is a constant, we can't
        # blindly use lambdify as it won't do the right thing.  In that case,
        # evaluate the number as a float and fill the y array with that value.
        if isinstance(app, sym.numbers.Number):
            y = np.zeros_like(x)
            y.fill(app.evalf())
        else:
            fa = sym.lambdify(xs, app, modules=['numpy'])
            y = fa(x)
        tex = sym.latex(app).replace('$', '')
        plt.plot(x, y, label=r'$n=%s:\, %s$' % (order, tex) )

    # Plot refinements
    if yrange is not None:
        plt.ylim(*yrange)
    plt.grid()
    plt.legend(loc='best').get_frame().set_alpha(0.8)
```

With this function defined, we can now use it for any sympy function or expression

```python
plot_taylor_approximations(sym.sin, 0, [2, 4, 6], (0, 2*sym.pi), (-2,2))
```

```python
plot_taylor_approximations(sym.cos, 0, [2, 4, 6], (0, 2*sym.pi), (-2,2))
```

This shows easily how a Taylor series is useless beyond its convergence radius, illustrated by
a simple function that has singularities on the real axis:

```python
# For an expression made from elementary functions, we must first make it into
# a callable function, the simplest way is to use the Python lambda construct.
plot_taylor_approximations(lambda x: 1/sym.cos(x), 0, [2,4,6], (0, 2*sym.pi), (-5,5))
```

```python

```


Traçar um esboço do gráfico e obter uma equação da parábola que satisfaça as condições dadas.

17. Vértice: $V(0,0)$; Eixo $y=0$; Passa pelo ponto $(4,5)$

Como a parábola é paralela ao eixo $x$ a equação que a representa é dada por $y^2 = 2px$

Substituindo os pontos dados na equação temos:

$5^2 = 2\cdot p \cdot 4$

$25 = 8p$

$\frac{25}{8} = p$

Encontrando o valor do foco

$F = \frac{p}{2}$

$F = \frac{\frac{25}{8}}{2}$

$F = \frac{25}{8} \cdot \frac{1}{2}$

$F = \frac{25}{16}$

$F(\frac{25}{16},0)$

Encontrando o valor da diretriz

$D = -\frac{p}{2}$

$D = -\frac{25}{16}$

$D : x = \frac{25}{16}$

Montando a equação

$y^2 = 2 \cdot \frac{25}{8} \cdot x$

$y^2 = \frac{50}{8}x$

$y^2 = \frac{25}{4}x$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((y-0)**2, 25/4*(x+0)), (x,-10,10), (y,-10,10),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/17.ipynb


# Exercise 2
Write a function to compute the roots of a mathematical equation of the form
\begin{align}
  ax^{2} + bx + c = 0.
\end{align}
Your function should be sensitive enough to adapt to situations in which a user might accidentally set $a=0$, or $b=0$, or even $a=b=0$.  For example, if $a=0, b\neq 0$, your function should print a warning and compute the roots of the resulting linear function.  It is up to you on how to handle the function header: feel free to use default keyword arguments, variable positional arguments, variable keyword arguments, or something else as you see fit.  Try to make it user friendly.

Your function should return a tuple containing the roots of the provided equation.

**Hint:** Quadratic equations can have complex roots of the form $r = a + ib$ where $i=\sqrt{-1}$ (Python uses the notation $j=\sqrt{-1}$).  To deal with complex roots, you should import the `cmath` library and use `cmath.sqrt` when computing square roots.  `cmath` will return a complex number for you.  You could handle complex roots yourself if you want, but you might as well use available libraries to save some work.

```python
import cmath

def compute_roots(a, b, c):
    """ Returns roots for quadratic equation of form ax^2 + bx + c = 0 """
    if a == 0:
        if b == 0:
            if c == 0:
                raise ValueError('Infititely many solutions!')
            else:
                raise ValueError('No solutions!')
        else:
            return (-c/b,)

    descriminant = (b^2) - (4 * a * c)

    sol1 = -b + cmath.sqrt(descriminant) / (2 * a)
    sol2 = -b - cmath.sqrt(descriminant) / (2 * a)
    return sol1, sol2

a = 0
b = 2
c = 4
try:
    solution = compute_roots(0, 2, 4)
    print('Roots = {}'.format(solution))
except Exception as e:
    print(e)
```

    Roots = (-2.0,)

```python

```


# Modeling data 2

## Building a model

Recall that in notebook 3, we saw that we could use a mathematical function to classify an image as an apple or a banana, based on the average amount of green in an image:

A common function for performing this kind of **classification** is the sigmoid that we saw in the last notebook, and that we will now extend by adding two **parameters**, $w$ and $b$:

$$\sigma(x; w, b) := \frac{1}{1 + \exp(-wx + b)}$$

$$ x = \mathrm{data} $$

\begin{align}
\sigma(x;w,b) &\approx 0 \implies \mathrm{apple} \\
\sigma(x;w,b) &\approx 1 \implies \mathrm{banana}
\end{align}

In our mathematical notation above, the `;` in the function differentiates between the **data** and the **parameters**. `x` is the data and is determined from the image. The parameters, `w` and `b`, are numbers which we choose to make our function match the results it should be modeling.

Note that in the code below, we don't distinguish between data and parameters - both are just inputs to our function, σ!

```julia
using Images, Statistics

apple = load("data/10_100.jpg")
banana = load("data/104_100.jpg")

apple_green_amount = mean(Float64.(green.(apple)))
banana_green_amount = mean(Float64.(green.(banana)))

println("Average green for apple = $apple_green_amount")
println("Average green for banana = $banana_green_amount")
```

    Average green for apple = 0.3382027450980393
    Average green for banana = 0.8807972549019609

```julia
σ(x, w, b) = 1 / (1 + exp(-w * x + b))
```

    σ (generic function with 1 method)

What we want is that when we give σ as input the average green for the apple, roughly `x = 0.3385`, it should return as output something close to 0, meaning "apple". And when we give σ the input `x = 0.8808`, it should output something close to 1, meaning "banana".

By changing the parameters of the function, we can change the shape of the function, and hence make it represent, or **fit**, the data better!

## Data fitting by varying parameters

We can understand how our choice of `w` and `b` affects our model by seeing how our values for `w` and `b` change the plot of the $\sigma$ function.

```julia
using Plots; gr()   # GR works better for interactive manipulations
```

    Plots.GRBackend()

Run the code in the next cell. You should see two "sliders" appear, one for `w` and one for `b`.

**Game**:
Change w and b around until the blue curve, labeled "model", which is the graph of the `\sigma` function, passes through *both* of the data points at the same time.

```julia
w = 10.0 # try manipulating w between -10 and 30
b = 10.0 # try manipulating b between 0 and 20

plot(x -> σ(x, w, b), xlim=(-0,1), ylim=(-0.1,1.1), label="model", legend=:topleft, lw=3)

scatter!([apple_green_amount],  [0.0], label="apple", ms=5)   # marker size = 5
scatter!([banana_green_amount], [1.0], label="banana", ms=5)
```

Notice that the two parameters do two very different things. The **weight**, `w`, determines *how fast* the transition between 0 and 1 occurs. It encodes how trustworthy we think our data  actually is, and in what range we should be putting points between 0 and 1 and thus calling them "unsure". The **bias**, `b`, encodes *where* on the $x$-axis the switch should take place. It can be seen as shifting the function left-right. We'll come to understand these *parameters* more in notebook 6.

Here are some parameter choices that work well:

```julia
w = 25.58; b = 15.6

plot(x -> σ(x, w, b), xlim=(0,1), ylim=(-0.1,1.1), label="model", legend=:topleft, lw=3)

scatter!([apple_green_amount], [0.0], label="apple")
scatter!([banana_green_amount],[1.0], label="banana")
```

(Note that in this problem there are many combinations of `w` and `b` that fit the data well.)

Once we have a model, we have a computational representation for how to choose between "apple" and "banana". So let's pull in some new images and see what our model says about them!

```julia
apple2 = load("data/107_100.jpg")
```

```julia
green_amount = mean(Float64.(green.(apple2)))
@show green_amount

scatter!([green_amount], [0.0], label="new apple")
```

    green_amount = 0.4687666666666668

Our model successfully says that our new image is an apple! Pat yourself on the back: you've actually just trained your first neural network!

#### Exercise 1

Load the image of a banana in `data/8_100.jpg` as `mybanana`. Edit the code below to calculate the amount of green in `mybanana` and to overlay data for this image with the existing model and data points.

#### Solution

To get the desired overlay, the code we need is

```julia
mybanana = load("data/8_100.jpg")
mybanana_green_amount = mean(Float64.(green.(banana)))
scatter!([mybanana_green_amount], [1.0], label="my banana")
```

## Closing remarks: bigger models, more data, more accuracy

That last apple should start making you think: not all apples are red; some are yellow. "Redness" is one attribute of being an apple, but isn't the whole thing. What we need to do is incorporate more ideas into our model by allowing more inputs. However, more inputs would mean more parameters to play with. Also, we would like to have the computer start "learning" on its own, instead of modifying the parameters ourselves until we think it "looks right". How do we take the next step?

The first thing to think about is, if you wanted to incorporate more data into the model, how would you change the sigmoid function? Play around with some ideas. But also, start thinking about how you chose parameters. What process did you do to finally end up at good parameters? These two problems (working with models with more data and automatically choosing parameters) are the last remaining step to understanding deep learning.


# Information Retrieval in High Dimensional Data
## Lab #6, 23.11.2017

## Principal Component Analysis

### Task 1

In this task we will once again work with the MNIST training set as provided on Moodle. Chose three digit classes, e.g. 1, 2 and 3 and load N=1000 images from each of the clsses to the workspace. Store the data in a normalized matrix $X$ of type double and size(784, 3\*N). Furthermore, generate  color label matrix $C$ of dimensions (3\*N, 3). Each row of $C$ assigns an RGB color vector to the repective column of $X$ as an indicator of the digit class. Choose [0, 0, 1], [0, 1, 0] and [1, 0, 0] for the three digit classes.

```python
import os
import numpy as np
import matplotlib.pyplot as plt
import imageio
```

```python
N = 1000
X = np.zeros((784, 3*N), dtype='float64')
for i in range(1, 4):
    path = 'mnist/d{}/'.format(i)
    filenames = sorted((fn for fn in os.listdir(path) if fn.endswith('.png')))
    for idx, fn in enumerate(filenames):
        im = imageio.imread(path + fn)
        X[:, idx + N*(i-1)] = np.reshape(im, 784)
        if idx == 999:
            break
```

```python
labels = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0]])
C = np.zeros((3*N, 3))
for i in range(3):
    for j in range(1000):
        C[N*i + j, :] = labels[i]
C
```

    array([[ 0.,  0.,  1.],
           [ 0.,  0.,  1.],
           [ 0.,  0.,  1.],
           ...,
           [ 1.,  0.,  0.],
           [ 1.,  0.,  0.],
           [ 1.,  0.,  0.]])

a) Compute the row-wise mean mu of $X$ and substract it from each column of $X$.
Save the results as X_c.

```python
mu = np.mean(X, axis=1)
X_c = X-np.expand_dims(mu, axis=1)
```

b) Use np.linalg.svd with full_matrices=False to compute the singular value decomposition [U, SIgma, VT] of X_c. Make sure the matrices are sorted in descending order with respect to the singular values.

```python
U, Sigma, VT = np.linalg.svd(X_c, full_matrices=False)
```

c) Use reshape in order to convert mu and the first three columns of U to (28, 28)-matrices. Plot the resulting images. What do you see?

```python
plt.subplot(141)
plt.imshow(np.reshape(mu, (28,28)))
plt.subplot(142)
plt.imshow(np.reshape(U[:, 0], (28,28)))
plt.subplot(143)
plt.imshow(np.reshape(U[:, 1], (28,28)))
plt.subplot(144)
plt.imshow(np.reshape(U[:, 2], (28,28)))

plt.show()
```

d) Compute the matrix S=np.dot(np.diag(Sigma), VT)). Note that ths yields the same result s S=np.dot(U.T, X_c). The S matrix contains the 3\*N scores for the principal components 1 to 784. Create a 2D scatter plot with C as its color parameter in order to plot the scores for the first $two$ rincipal components of the data.

```python
S = np.dot(np.diag(Sigma), VT)
plt.scatter(S[:,:2], X_c[:, :2], c=C)
plt.show()
```

### Task 2

In this task we consider the problem of choosing he number of principal vectors. Assuming that $\mathbf{X} \in \mathbb{R}^{p\times N}$ is the centered data matrix and $ \mathbf{P} = \mathbf{U}_k \mathbf{U}_k^T $ is the projector onto the $k$-dimensional principal subspace, the dimension $k$ is chosen such that the fraction of overall energy contained in the projection error does not exceed $\epsilon$, i.e.

\begin{equation} \frac{\|\mathbf{X} - \mathbf{PX}\|_F^2}{\| \mathbf{X}\|_F^2} = \frac{\sum_{i=1}^M \|\mathbf{x}_i - \mathbf{Px}_i\|^2}{\sum_{i=1}^N \| \mathbf{x}_i\|^2} \leq \epsilon ,\end{equation}

where $\epsilon$ is usually chosen to be between $0.01$ and $0.2$.

The MIT VisTex database as provided on Moodle consists of a set of 167 RGB texture images of sizes (512, 512, 3). Download the ZIP file, unpack it and make yourself familiar with the directory structure.

a) After preprocessing the entire image set (converting to normalized grayscale matrices), divide the images into non overlapping tiles of sizes (64, 64) and  create a centered data matrix X_c of size (p, N) from them, where P=64\*64 and N=167\*(512/64)\*(512/64).

```python

```

b) Compute the SVD of X_c and make sure the singular values are sorted in descending order.

```python

```

c) Plot the fraction of energy contained in the projection error for the principal subspace dimensions 0 to p. How many principal vectors do you need to retain 80%, 90%, 95% or 99% of the original data energy?

```python

```

d) Discuss: Can you imagine a scenario, where energy is a bad measure of useful information?

```python

```


# Interpolation and Differentiation

All algorithms use the baryentric weights as described in the book "Implementing Spectral Methods for PDEs" by David Kopriva.

- Interpolation can be performed either with `interpolate(dest, u, basis)` or via an
  `interpolation_matrix(dest, basis)`.
- In a nodal basis, the derivative matrix is available as `basis.D`. Alternatively, `derivative_at(x, u, basis)`
  can be used.

```julia
using Revise
using PolynomialBases
using LaTeXStrings, Plots

# define nodal bases
p = 5 # polynomial degree
basis1 = LobattoLegendre(p)
basis2 = GaussLegendre(p)

# the function that will be interpolated
ufunc(x) = sinpi(x); uprim(x) = π*cospi(x)
#ufunc(x) = 1 / (1 + 25x^2); uprim(x) = -ufunc(x)^2*50x

for basis in (basis1, basis2)
    u = ufunc.(basis.nodes)

    xplot = range(-1, stop=1, length=500)
    uplot = interpolate(xplot, u, basis)

    fig1 = plot(xplot, ufunc.(xplot), label="u", xguide=L"x", yguide=L"u")
    plot!(fig1, xplot, uplot, label=L"\mathrm{I}(u)")

    fig2 = plot(xplot, uprim.(xplot), label="u'", xguide=L"x", yguide=L"u'")
    plot!(fig2, xplot, interpolate(xplot, basis.D*u, basis), label=L"\mathrm{I}(u)'")

    display(basis)
    display(plot(fig1, fig2))
end
```

# Integration

The nodes and weights are from [FastGaussQuadrature.jl](https://github.com/ajt60gaibb/FastGaussQuadrature.jl).

```julia
using Revise
using PolynomialBases
using LaTeXStrings, Plots; pyplot()

ufunc(x) = sinpi(x)^6

function compute_error(p, basis_type)
    basis = basis_type(p)
    u = ufunc.(basis.nodes)
    abs(5/8 - integrate(u, basis))
end

ps = 1:23
scatter(ps, compute_error.(ps, LobattoLegendre), label="Lobatto", xguide=L"p", yguide="Error", yaxis=:log10)
scatter!(ps, compute_error.(ps, GaussLegendre), label="Gauss")
```

# Evaluation of Orthogonal Polynomials

## Legendre Polynomials

Legendre poylnomials $P_p$ are evaluated as `legendre(x, p)` using the three term recursion formula.

```julia
using Revise
using PolynomialBases
using LaTeXStrings, Plots; pyplot()

x = range(-1, stop=1, length=10^3)
fig = plot(xguide=L"x")
for p in 0:5
    plot!(fig, x, legendre.(x, p), label="\$ P_$p \$")
end
fig
```

## Gegenbauer Polynomials

Gegenbauer poylnomials $C_p^{(\alpha)}$ are evaluated as `gegenbauer(x, p, α)` using the three term recursion formula.

```julia
using Revise
using PolynomialBases
using LaTeXStrings, Plots; pyplot()

α = 0.5
x = range(-1, stop=1, length=10^3)
fig = plot(xguide=L"x")
for p in 0:5
    plot!(fig, x, gegenbauer.(x, p, α), label="\$ C_$p^{($α)} \$")
end
fig
```

## Jacobi Polynomials

Jacobi poylnomials $P_p^{\alpha,\beta}$ are evaluated as `jacobi(x, p, α, β)` using the three term recursion formula.

```julia
using Revise
using PolynomialBases
using LaTeXStrings, Plots; pyplot()

α, β = -0.5, -0.5
#α, β = 0, 0
x = range(-1, stop=1, length=10^3)
fig = plot(xguide=L"x")
for p in 0:5
    plot!(fig, x, jacobi.(x, p, α, β), label="\$ P_$p^{$α, $β} \$")
end
fig
```

## Hermite Polynomials

Hermite poylnomials $H_p$ are evaluated as `hermite(x, p)` using the three term recursion formula.

```julia
using Revise
using PolynomialBases
using LaTeXStrings, Plots; pyplot()

x = range(-2, stop=2, length=10^3)
fig = plot(xguide=L"x")
for p in 0:4
    plot!(fig, x, hermite.(x, p), label="\$ H_$p \$")
end
fig
```

## Hahn Polynomials

```julia
using Revise
using PolynomialBases
using LaTeXStrings, Plots; pyplot()

α, β, N = -0.5, -0.5, 20
x = range(0, stop=N, length=10^3)
fig = plot(xguide=L"x")
for p in 0:5
    plot!(fig, x, hahn.(x, p, α, β, N), label="\$ Q_$p(x; $α, $β, $N) \$")
end
fig
```

# Symbolic Computations

Symbolic computations using [SymPy.jl](https://github.com/JuliaPy/SymPy.jl) and [SymEngine.jl](https://github.com/symengine/symengine) are supported.

```julia
using Revise
using PolynomialBases
import SymPy
import SymEngine

x_sympy = SymPy.symbols("x")
x_symengine = SymEngine.symbols("x")

legendre(x_sympy, 6) |> SymPy.expand |> display
legendre(x_symengine, 6) |>  SymEngine.expand |> display

GaussLegendre(2, SymPy.Sym).D |> display
GaussLegendre(2, SymEngine.Basic).D |> display
```

# Modal Matrices

The Vandermonde matrix $V$ can be computed as `legendre_vandermonde(basis)` and used to transform coeficients in a modal basis of Legendre polynomials to a nodal basis.

```julia
using Revise
using PolynomialBases
using LinearAlgebra

p = 7
basis = GaussLegendre(p)
V = legendre_vandermonde(basis)

# the modal derivative matrix
Dhat = legendre_D(p)
# they should be equal
norm( basis.D - V * Dhat / V)
```

Similarly, the Vandermonde matrix with respect to the Jacobi polynomials is given as `jacobi_vandermonde(basis, α, β)`.

```julia
using Revise
using PolynomialBases

p = 3
α, β = -0.5, -0.5
basis = GaussJacobi(p, α, β)
V = jacobi_vandermonde(basis, α, β)
```


## Exercise 10.1 (search)

We want to find the largest and smallest values in a long list of numbers. Implement
two algorithms, based on:

1. Iterating over the list entries; and
1. First applying a built-in sort operation to the list.

Encapsulate each algorithm in a function. To create lists of numbers for testing use, for example:
```python
x = np.random.rand(1000)
```

### Solution

We first create the list of random numbers

```python
import numpy as np
x = np.random.rand(1000)
```

#### Approach 1

```python
def min_max1(x):
    # YOUR CODE HERE
    raise NotImplementedError()
    return  x_min, x_max

print(min_max1(x))
```

#### Approach 2

```python
def min_max2(x):
    # YOUR CODE HERE
    raise NotImplementedError()

print(min_max2(x))
```

```python
assert min_max1(x) == min_max2(x)
```

In practice, we would use the the NumPy function:

```python
print(np.min(x), np.max(x))
```

## Exercise 10.2 (Newton's method for root finding)

### Background

Newton's method can be used to find a root $x$ of a function $f(x)$ such that
$$
f(x) = 0
$$
A Taylor series expansion of $f$ about $x_{i}$ reads:
$$
f(x_{i+1}) = f(x_{i}) + \left. f^{\prime} \right|_{x_{i}} (x_{i+1} - x_{i}) +  O((x_{i+1} - x_{i})^{2})
$$
If we neglect the higher-order terms and set $f(x_{i+1})$ to zero, we have Newton's method:
\begin{align}
x_{i + 1} &= - \frac{f(x_{i})}{f^{\prime}(x_{i})} + x_{i}
\\
x_{i} &\leftarrow x_{i+1}
\end{align}
In Newton's method, the above is applied iteratively until $\left|f(x_{i + 1})\right|$ is below a tolerance value.

### Task

Develop an implementation of Newton's method, with the following three functions in your implementation:
```python
def newton(f, df, x0, tol, max_it):
    # Implement here

    return x1  # return root
```
where `x0` is the initial guess, `tol` is the stopping tolerance, `max_it` is the maximum number
of iterations, and
```python
def f(x):
    # Evaluate function at x and return value

def df(x):
    # Evaluate df/dx at x and return value

```

Your implementation should raise an exception if the maximum number of iterations (`max_it`)
is exceeded.

Use your program to find the roots of:
$$
f(x) = \tan(x) - 2x
$$
between $-\pi/2$ and $\pi/2$. Plot $f(x)$ and $f^{\prime}(x)$ on the same graph,
and show the roots computed by Newton's method.

Newton's method can be sensitive to the starting value. Make sure you find the root around $x = 1.2$. What happens if you start at $x = 0.9$? It may help to add a print statement in the iteration loop, showing $x$ and $f$ at each iteration.

### Extension (optional)

For a complicated function we might not know how to compute the derivative, or it may be very complicated
to evaluate. Write a function that computes the *numerical derivative* of $f(x)$ by evaluating
$(f(x + dx) - f(x - dx)) / (2dx)$, where $dx$ is small. How should you choose $dx$?

### Solution

We first implement a Newton solver function:

```python
import numpy as np

def newton(f, df, x, tol=1e-8, max_it=20):
    """Find root of equation defined by function f(x) where df(x) is
    first derivative and x is the initial guess.Optional arguments tol
    (tolerance) and max_it (maximum number of iterations)"""

    # YOUR CODE HERE
    raise NotImplementedError()
```

We now provide implementations of `f` and `df`, and find the roots:

```python
def f(x):
    # YOUR CODE HERE
    raise NotImplementedError()

def df(x):
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# YOUR CODE HERE
raise NotImplementedError()
```

We can visualise the result:

```python
%matplotlib inline
import matplotlib.pyplot as plt

# Plot f and df/dx
x = np.linspace(-1.5, 1.5, 100)
plt.plot(x, f(x), label='$f(x)$')
plt.plot(x, df(x), label="$f^{\prime}(x)$")

# Add location of roots to plot
# YOUR CODE HERE
raise NotImplementedError()

plt.show()
```

For the extension, we can replace the function `df(x)` with a new version

```python
def df(x):
    # Try changing dx to 1e-15 or smaller
    dx = 1e-9
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# Find roots near -1.2, 0.1, and 1.2
xroots = np.array((newton(f, df, -1.2),
                   newton(f, df, 0.1),
                   newton(f, df, 1.2)))
assert np.isclose(xroots, [-1.16556119e+00, 2.08575213e-10, 1.16556119e+00]).all()
```

```python
# Plot f, f' and roots

# YOUR CODE HERE
raise NotImplementedError()
```

In practice, we could use the Newton function `scipy.optimize.newton` from SciPy (http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.newton.html) rather than implementing our own function.

## Exercise 10.3 (optional, low pass image filter)

Images files can be loaded and displayed with Matplotlib. An imported image is stored as a
three-dimensional NumPy array of floats. The shape of the array is `[0:nx, 0:ny, 0:3]`.
where `nx` is the number of pixels in the $x$-direction, `ny` is the number of pixels in the $y$-direction,
and the third axis is for the colour component (RGB: red, green and blue) intensity. See http://matplotlib.org/users/image_tutorial.html for more background.

Below we fetch an image and display it:

```python
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

# Import image
img = mpimg.imread('https://raw.githubusercontent.com/matplotlib/matplotlib.github.com/master/_images/stinkbug.png')

# Check type and shape
print(type(img))
print("Image array shape: {}".format(img.shape))

# Display image
plt.imshow(img);
```

The task is to write a *function* that applies a particular low-pass filter algorithm to an image array
and  returns the  filtered image. With this particular filter, the value of a pixel in the filtered image
is equal to the average value of the four neighbouring pixels in the original image. For the `[i, j, :]` pixel,
the neighbours are  `[i, j+1, :]`, `[i, j-1, :]`, `[i+1, j, :]` and  `[i-1, j, :]`.

Run the filter algorithm multiple times on the above image to explore the effect of the filter.

*Hint*: To create a NumPy array of zeros, `B`,  with the same shape as array `A`, use:
```python
import numpy as np
B = np.zeros_like(A)
```

```python
# YOUR CODE HERE
raise NotImplementedError()
```


```python
import loader
from sympy import *
init_printing()
from root.solver import *
```

Find the complementary and particular solution of the system of ode
$$
    \vec{\mathbf{x'}} = \begin{bmatrix}
        -2 & 1 \\
        1 & -2 \\
    \end{bmatrix} \vec{\mathbf{x}} + \begin{bmatrix}
        2 e^{-t} \\
        3 t
    \end{bmatrix}
$$
using variation of parameters

```python
xc, p = system([
    [-2, 1],
    [1, -2]
])
p.display()
_, p = nonhomo_system_variation_of_parameters(xc, [2 * exp(-t), 3*t])
p.display()
```

$\displaystyle \text{Characteristic equation: }$

$\displaystyle \left(\lambda + 2\right)^{2} - 1 = 0$

$\displaystyle \text{Eigenvalues and eigenvectors}$

$\displaystyle \lambda_1 = -3$

$\displaystyle \left[\begin{matrix}1 & 1 & 0\\1 & 1 & 0\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & 1 & 0\\0 & 0 & 0\end{matrix}\right]\Rightarrow v = \left[\begin{matrix}-1\\1\end{matrix}\right]$

$\displaystyle \lambda_2 = -1$

$\displaystyle \left[\begin{matrix}-1 & 1 & 0\\1 & -1 & 0\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & -1 & 0\\0 & 0 & 0\end{matrix}\right]\Rightarrow v = \left[\begin{matrix}1\\1\end{matrix}\right]$

$\displaystyle \text{General solution: }$

$\displaystyle \vec{\mathbf{x}} = C_{1}e^{- 3 t}\left[\begin{matrix}-1\\1\end{matrix}\right]+C_{2}e^{- t}\left[\begin{matrix}1\\1\end{matrix}\right]$

$\displaystyle \text{Fundamental matrix }\Psi$

$\displaystyle \left[\begin{matrix}- e^{- 3 t} & e^{- t}\\e^{- 3 t} & e^{- t}\end{matrix}\right]$

$\displaystyle \text{Calculate the inverse of the fundamental matrix }\Psi^{-1}$

$\displaystyle \Psi^{-1} = \left[\begin{matrix}- \frac{e^{3 t}}{2} & \frac{e^{3 t}}{2}\\\frac{e^{t}}{2} & \frac{e^{t}}{2}\end{matrix}\right]$

$\displaystyle \text{Compute }\Psi^{-1} g(t)$

$\displaystyle \Psi^{-1} g(t) = \left[\begin{matrix}\frac{3 t e^{3 t}}{2} - e^{2 t}\\\frac{3 t e^{t}}{2} + 1\end{matrix}\right]$

$\displaystyle \text{Compute the integral}$

$\displaystyle \int \Psi^{-1} g(t) =\left[\begin{matrix}\frac{t e^{3 t}}{2} - \frac{e^{3 t}}{6} - \frac{e^{2 t}}{2}\\\frac{3 t e^{t}}{2} + t - \frac{3 e^{t}}{2}\end{matrix}\right]$

$\displaystyle \text{Finally, }\vec{\mathbf{x_p}} = \Psi \int \Psi^{-1} g(t)$

$\displaystyle \vec{\mathbf{x_p}} =\left[\begin{matrix}t + t e^{- t} - \frac{4}{3} + \frac{e^{- t}}{2}\\2 t + t e^{- t} - \frac{5}{3} - \frac{e^{- t}}{2}\end{matrix}\right]$

Find a particular solution of the system
$$
    \vec{\mathbf{x'}} = \begin{bmatrix}
        10 & -5 \\
        20 & -10 \\
    \end{bmatrix} \vec{\mathbf{x}} + \begin{bmatrix}
        t^{-3} \\
        -t^{-2}
    \end{bmatrix}
$$
using variation of parameters

```python
xc, p = system([
    [10, -5],
    [20, -10]
])
p.display()
_, p = nonhomo_system_variation_of_parameters(xc, [t**(-3),-t**(-2)])
p.display()
```

$\displaystyle \text{Characteristic equation: }$

$\displaystyle \lambda^{2} = 0$

$\displaystyle \text{Eigenvalues and eigenvectors}$

$\displaystyle \lambda_1 = 0$

$\displaystyle \left[\begin{matrix}10 & -5 & 0\\20 & -10 & 0\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & - \frac{1}{2} & 0\\0 & 0 & 0\end{matrix}\right]\Rightarrow v = \left[\begin{matrix}\frac{1}{2}\\1\end{matrix}\right]$

$\displaystyle \text{Find the generalized eigenvector}\left( M - \lambda I \right) w = v $

$\displaystyle \left[\begin{matrix}10 & -5 & \frac{1}{2}\\20 & -10 & 1\end{matrix}\right]\text{ ~ }\left[\begin{matrix}1 & - \frac{1}{2} & \frac{1}{20}\\0 & 0 & 0\end{matrix}\right]\Rightarrow w = \left[\begin{matrix}\frac{1}{20}\\0\end{matrix}\right]$

$\displaystyle \text{General solution: }$

$\displaystyle \vec{\mathbf{x}} = C_{1}1\left[\begin{matrix}\frac{1}{2}\\1\end{matrix}\right]+C_{2}1\left(\left[\begin{matrix}\frac{1}{2}\\1\end{matrix}\right]t + \left[\begin{matrix}\frac{1}{20}\\0\end{matrix}\right]\right)$

$\displaystyle \text{Fundamental matrix }\Psi$

$\displaystyle \left[\begin{matrix}\frac{1}{2} & \frac{t}{2} + \frac{1}{20}\\1 & t\end{matrix}\right]$

$\displaystyle \text{Calculate the inverse of the fundamental matrix }\Psi^{-1}$

$\displaystyle \Psi^{-1} = \left[\begin{matrix}- 20 t & 10 t + 1\\20 & -10\end{matrix}\right]$

$\displaystyle \text{Compute }\Psi^{-1} g(t)$

$\displaystyle \Psi^{-1} g(t) = \left[\begin{matrix}- \frac{10}{t} - \frac{21}{t^{2}}\\\frac{10}{t^{2}} + \frac{20}{t^{3}}\end{matrix}\right]$

$\displaystyle \text{Compute the integral}$

$\displaystyle \int \Psi^{-1} g(t) =\left[\begin{matrix}- 10 \ln{\left(t \right)} + \frac{21}{t}\\- \frac{10}{t} - \frac{10}{t^{2}}\end{matrix}\right]$

$\displaystyle \text{Finally, }\vec{\mathbf{x_p}} = \Psi \int \Psi^{-1} g(t)$

$\displaystyle \vec{\mathbf{x_p}} =\left[\begin{matrix}- 5 \ln{\left(t \right)} - 5 + \frac{5}{t} - \frac{1}{2 t^{2}}\\- 10 \ln{\left(t \right)} - 10 + \frac{11}{t}\end{matrix}\right]$

```python

```


# 8.2 Regularized linear softening law for stable calculation in finite element code

```python
%matplotlib inline
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
```

Let us assume that the fracture energy $G_\mathrm{f}$ is known and  we want to improve the above model to deliver stable, mesh-independent results. Knowing that the energy dissipation is only happening in the one softening element, we can require that it alwas dissipates the same amount of energy. Recall that the fracture energy dissipated by a unit area of a stress free crack is evaluated as
\begin{align}
G_\mathrm{f} = \int_0^{\infty} f(w) \, \mathrm{d}w
\end{align}
where $w$ represents the crack opening.
In the studied case of linear softening with $f$ defined as
\begin{align}
f(w) = f_\mathrm{t}\left( 1 - \frac{1}{w_\mathrm{f}} w \right)
\end{align}
and the corresponding fracture energy
\begin{align}
G_\mathrm{f} = \int_0^{w_\mathrm{f}} f(w) \, \mathrm{d}w = \frac{1}{2} f_\mathrm{t} w_\mathrm{f}
\end{align}
This kind of model is working correctly and reproduces exactly the amount of fracture energy needed to produce the unit area of the stress free crack.

However, in a finite element model, the crack is not represented as a discrete line but is represented by a strain within the softening element of the size $L_s = L / N$. Thus, the softening is not related to crack opening displacement (COD) but to a strain in the softening zone using a softening function $\phi(\varepsilon_\mathrm{s})$. This model was used in the example above and delivered mesh-dependent results with varying amount of fracture energy.

To regularize the finite element model let us express the crack opening displacement as a product of the softening strain $\varepsilon_\mathrm{s}$ and the size of the softening zone $L_\mathrm{s}$:
\begin{align}
w = \varepsilon_\mathrm{s} L_\mathrm{s}
\end{align}
Now, the energy dissipated within the softening zone can be obtained as an integral over the history of the softening strain as
\begin{align}
G_\mathrm{f} = L_\mathrm{s} \int_0^{\infty} \phi(\varepsilon_\mathrm{s}) \, \mathrm{d}\varepsilon_\mathrm{s}
\end{align}
Comming back to the model with linear softening, the integral of the strain based softening function is expressed as
\begin{align}
\int_0^{\infty} \phi(\varepsilon_\mathrm{s})  \, \mathrm{d} \varepsilon_\mathrm{s} = \frac{1}{2} \varepsilon_\mathrm{f} f_\mathrm{t}
\end{align}
so that
\begin{align}
G_\mathrm{f} = \frac{1}{2} L_\mathrm{s} \varepsilon_\mathrm{f}
f_\mathrm{t}
\implies
\varepsilon_\mathrm{f} = \frac{2}{L_\mathrm{s}} \frac{G_\mathrm{f}}{ f_\mathrm{t}}
\end{align}

```python
L = 10.0
E = 20000.0
f_t = 2.4
G_f = 0.0125
n_E_list = [5,10,1000]
# run a loop over the different discretizations
for n in n_E_list:  # n: number of element
    L_s = L / n
    eps_f = 2 * G_f / f_t / L_s
    eps = np.array([0.0, f_t / E, eps_f / n])
    sig = np.array([0.0, f_t, 0.0])
    g_f = eps[-1] * sig[1] / 2
    print('Is the energy constant?', g_f)
    plt.plot(eps, sig, label='n=%i' % n)
    plt.legend(loc=1)

plt.xlabel('strain')
plt.ylabel('stress')
plt.show()
```

Consider a tensile test with the dimensions of a cross section $100 \times 100$ mm and length of $1000$ mm. The measured elongation of the beam during the test was as follows
Assume the stiffness of $E = 28000$ MPa and strength $f_t = 3$ MPa. The length of the cohesive zone at which the crack localized was measured using the digital image correlation and set to $L_s = 0.05$ mm.

## Task

Given the fracture energy, linear softening function, strength, E-modulus, bar length, manually calculate the force-displacement response of a tensile test.

tour6_energy/8_2_Regularized linear softening law for stable calculation in finite element code.ipynb


# Solvers

Boilerplate to make the doctester work.

```
import sys
import os
sys.path.insert(1, os.path.join(os.path.pardir, "ipython_doctester"))
from sympy import *
from ipython_doctester import test
# Work around a bug in IPython. This will disable the ability to paste things with >>>
def notransform(line): return line
from IPython.core import inputsplitter
inputsplitter.transform_classic_prompt = notransform
init_printing()
```

For each exercise, fill in the function according to its docstring. Execute the cell to see if you did it right.

```
a, b, c, d, x, y, z, t = symbols('a b c d x y z t')
f, g, h = symbols('f g h', cls=Function)
```

## Algebraic Equations

Write a function that computes the [quadratic equation](http://en.wikipedia.org/wiki/Quadratic_equation).

```
def quadratic():
    return ???
quadratic()
```

Write a function that computes the general solution to the cubic $x^3 + ax^2 + bx + c$.

```
def cubic():
    return ???
cubic()
```

## Differential Equations

A population that grows without bound is modeled by the differential equation

$$f'(t)=af(t)$$

Solve this differential equation using SymPy.

```

```

If the population growth is bounded, it is modeled by

$$f'(t) = f(t)(1 - f(t))$$

```

```


# The Variational Autoencoder

In this notebook we are interested in the problem of inference in a probabilistic model that contains both observed and latent variables, which can be represented as the following graphical model:

<div style="text-align:center;">

</div>

For this model the joint probability distribution factorizes as

$$
p(\mathbf{x}, \mathbf{z}\vert\boldsymbol{\theta}) = p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta})p(\mathbf{z}\vert\boldsymbol{\theta})
$$

For any distribution $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$, we can write the marginal log-likelihood as the sum of two terms: the evidence lower bound $\mathcal{L}(q, \boldsymbol{\theta},\boldsymbol{\phi})$ and the KL divergence between $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$ and the posterior $p(\mathbf{z}\vert\mathbf{x},\boldsymbol{\theta})$:

$$
\log p(\mathbf{x}\vert\boldsymbol{\theta}) = \mathcal{L}(q, \boldsymbol{\theta}, \boldsymbol{\phi}) + \text{KL}(q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})\Vert p(\mathbf{z\vert\mathbf{x}, \boldsymbol{\theta}}))
$$

Our goal is to find the values of $\boldsymbol{\theta}$ and $\boldsymbol{\phi}$ that maximize the marginal log-likelihood. In *variational inference*, we propose $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$ as an approximation to the posterior, so that the KL divergence is minimized. The KL divergence is minimized when we maximize the lower bound, defined as

$$
\begin{align}
\mathcal{L}(q, \boldsymbol{\theta}, \boldsymbol{\phi}) &= \mathbb{E}_q[\log p(\mathbf{x},\mathbf{z}\vert\boldsymbol{\theta}) - \log q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})]\\
&=
\mathbb{E}_q[\log p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta})] - \text{KL}(q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})\Vert p(\mathbf{z}\vert\boldsymbol{\theta}))
\end{align}
$$

We can think of $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$ as taking the variable $\mathbf{x}$ and producing a distribution over the latent variable $\mathbf{z}$. For this reason $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$ is also known as the **encoder**: the latent variable $\mathbf{z}$ acts as a code for the observation $\mathbf{x}$. The parameters $\boldsymbol{\phi}$ are known as the **variational parameters**, because they correspond to the distribution $q$ that we want to use as an approximation to the true posterior.

Similarly, we can see that our model for $p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta})$ does the opposite: given a latent representation, a distribution over the observation is produced. Therefore $p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta})$ is also known as the **decoder**, which takes the code $\mathbf{z}$ and reconstructs the observation $\mathbf{x}$. The parameters $\boldsymbol{\theta}$ are known as the **generative parameters**.

To maximize the lower bound, we can obtain its gradient with respect to the parameters and then update them in that direction:

$$
\nabla_{\boldsymbol{\theta},\boldsymbol{\phi}} \mathcal{L}(q, \boldsymbol{\theta}, \boldsymbol{\phi}) = \nabla_{\boldsymbol{\theta},\boldsymbol{\phi}}
\left[
\mathbb{E}_q[\log p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta})] - \text{KL}(q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})\Vert p(\mathbf{z}\vert\boldsymbol{\theta}))
\right]
$$

For some cases, the KL divergence can be calculated analytically, as well as its gradient with respect to both the generative and variational parameters. The expectation term can be approximated with a *Monte Carlo estimate*, by taking samples and averaging the result. However, how do we calculate the derivative with respect to $\boldsymbol{\phi}$ of a sampling operation from a distribution whose parameter is $\boldsymbol{\phi}$ itself?

## The reparameterization trick

Instead of using $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$ to obtain samples of $\mathbf{z}$, we will introduce an auxiliary random variable $\boldsymbol{\epsilon}$ with a corresponding, known distribution $p(\boldsymbol{\epsilon})$. We obtain a sample $\boldsymbol{\epsilon}$ from this distribution, and then we let $\mathbf{z}$ be a deterministic, differentiable function of it:

$$
\mathbf{z} = g(\mathbf{x},\boldsymbol{\epsilon},\boldsymbol{\phi})
$$

Given an appropriate choice of $p(\boldsymbol{\epsilon})$ and $g$, $\mathbf{z}$ would be as if we had sampled from $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$, which is what we wanted. The difference now is that the sample $\mathbf{z}$ was obtained from a differentiable function, and now we can obtain the gradient with respect to $\boldsymbol{\phi}$! We can take $L$ samples to obtain the Monte Carlo estimate of the expectation and then differentiate:

$$
\nabla_{\boldsymbol{\theta},\boldsymbol{\phi}} \mathcal{L}(q, \boldsymbol{\theta}, \boldsymbol{\phi}) \approx \nabla_{\boldsymbol{\theta},\boldsymbol{\phi}}\frac{1}{L}\sum_{i=1}^L \log p(\mathbf{x}\vert\mathbf{z}^{(i)},\boldsymbol{\theta})
$$

with $\mathbf{z}^{(i)}$ is obtained for each sample of $\boldsymbol{\epsilon}$.

## The algorithm

We now have an algorithm to optimize the lower bound, known as **Autoencoding Variational Bayes** [1]:

- Take an observation $\mathbf{x}$
- Take $L$ samples of $\boldsymbol{\epsilon}\sim p(\boldsymbol{\epsilon})$ and let $\mathbf{z}^{(i)}=g(\mathbf{x},\boldsymbol{\epsilon}^{(i)},\boldsymbol{\phi})$
- Calculate $\mathbf{g} = \nabla_{\boldsymbol{\theta},\boldsymbol{\phi}}\lbrace\frac{1}{L}\sum_{i=1}^L \log p(\mathbf{x}\vert\mathbf{z}^{(i)},\boldsymbol{\theta}) - \text{KL}(q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})\Vert p(\mathbf{z}\vert\boldsymbol{\theta}))\rbrace$
- Update $\boldsymbol{\theta}$ and $\boldsymbol{\phi}$ using $\mathbf{g}$ and an optimizer like Stochastic Gradient Descent or Adam.

## A practical example

As in the EM example, we will now define a generative model for the MNIST digits dataset. This time, however, we will assume the latent variables to be continuous instead of discrete, so that $\mathbf{z}\in\mathbb{R}^K$ where $K$ is a hyperparameter that indicates the dimension of the latent space. We choose the prior distribution as a Gaussian with zero mean and unit covariance,

$$
p(\mathbf{z}) = \mathcal{N}(\mathbf{z}\vert\mathbf{0}, \mathbf{I})
$$

Given an observation $\mathbf{x}$, the approximation of the posterior $p(\mathbf{z}\vert\mathbf{x},\boldsymbol{\theta})$ (the encoder) will be a Gaussian distribution with diagonal covariance:

$$
q(\mathbf{z}\vert\mathbf{x}, \boldsymbol{\phi}) = \mathcal{N}(\mathbf{z}\vert\boldsymbol{\mu}_e,\text{diag}(\boldsymbol{\sigma}_e))
$$

with

$$
\begin{align}
\boldsymbol{\mu}_e &= f_{\phi_\mu}(\mathbf{x})\\
\log(\boldsymbol{\sigma}_e^2) &= f_{\phi_\sigma}(\mathbf{x})
\end{align}
$$

where $e$ in the subscripts refer to the *encoder*, and $f_{\phi_\mu}$ and $f_{\phi_\sigma}$ are neural networks with weights $\boldsymbol{\phi}_\mu$ and $\boldsymbol{\phi}_\sigma$, respectively. These parameters form the parameters of the encoder: $\boldsymbol{\phi} = \lbrace\boldsymbol{\phi}_\mu,\boldsymbol{\phi}_\sigma\rbrace$. The reparameterization trick for this case is

$$
\begin{align}
\boldsymbol{\epsilon}&\sim\mathcal{N}(\mathbf{0}, \mathbf{I})\\
\mathbf{z} &= \boldsymbol{\mu}_e + \boldsymbol{\sigma}_e\odot\boldsymbol{\epsilon}
\end{align}
$$

where $\odot$ denotes element-wise multiplication.

For the prior and approximate posterior that we have defined, the KL divergence is

$$
\text{KL}(q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})\Vert p(\mathbf{z}\vert\boldsymbol{\theta})) = \frac{1}{2}\sum_{i=1}^K(\sigma_{ei}^2 + \mu_{ei}^2 - \log(\sigma_{ei}^2) - 1)
$$

Since the pixels in the images are binary, we model an observation $\mathbf{x}\in\mathbb{R}^D$, given the latent variable $\mathbf{z}$, as a multivariate Bernoulli random variable with mean $\boldsymbol{\mu}_d$. This corresponds to the *decoder*:

$$
p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta}) = \prod_{i=1}^D \mu_{di}^{x_i}(1-\mu_{di})^{(1-x_i)}
$$

with

$$
\boldsymbol{\mu}_d = f_\theta(\mathbf{z})
$$

where $f_\theta$ is a neural network with weights $\boldsymbol{\theta}$. Note that since the output of the decoder models a distribution over a multivariate Bernoulli, we must ensure that its values lie within 0 and 1. We do this with a sigmoid layer at the output.

Given this definition of the decoder, we have

$$
\log p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta}) = \sum_{i=1}^D x_i\log\mu_{di} + (1-x_i)\log(1-\mu_{di})
$$

which is the negative binary cross-entropy loss. We now have all the ingredients to implement and train the autoencoder, for which we will use PyTorch.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions.multivariate_normal import MultivariateNormal

class BernoulliVAE(nn.Module):
    def __init__(self, input_dim, latent_dim, enc_units, dec_units):
        super(BernoulliVAE, self).__init__()
        # Encoder parameters
        self.linear_enc = nn.Linear(input_dim, enc_units)
        self.enc_mu = nn.Linear(enc_units, latent_dim)
        self.enc_logvar = nn.Linear(enc_units, latent_dim)

        # Distribution to sample for the reparameterization trick
        self.normal_dist = MultivariateNormal(torch.zeros(latent_dim),
                                              torch.eye(latent_dim))

        # Decoder parameters
        self.linear_dec = nn.Linear(latent_dim, dec_units)
        self.dec_mu = nn.Linear(dec_units, input_dim)

        # Reconstruction loss: binary cross-entropy
        self.criterion = nn.BCELoss(reduction='sum')

    def encode(self, x):
        # Obtain the parameters of the latent variable distribution
        h = torch.relu(self.linear_enc(x))
        mu_e = self.enc_mu(h)
        logvar_e = self.enc_logvar(h)

        # Get a latent variable sample with the reparameterization trick
        epsilon = self.normal_dist.sample((x.shape[0],))
        z = mu_e + torch.sqrt(torch.exp(logvar_e)) * epsilon

        return z, mu_e, logvar_e

    def decode(self, z):
        # Obtain the parameters of the observation distribution
        h = torch.relu(self.linear_dec(z))
        mu_d = torch.sigmoid(self.dec_mu(h))

        return mu_d

    def forward(self, x):
        """ Calculate the negative lower bound for the given input """
        z, mu_e, logvar_e = self.encode(x)
        mu_d = self.decode(z)
        neg_cross_entropy = self.criterion(mu_d, x)
        kl_div = -0.5* (1 + logvar_e - mu_e**2 - torch.exp(logvar_e)).sum()

        # Since the optimizer minimizes, we return the negative
        # of the lower bound that we need to maximize
        return neg_cross_entropy, kl_div
```

We can now load the data, create a model and train it. We will choose 10 for the dimension of the latent space, and 300 units in the hidden layer for both the encoder and the decoder.

```python
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

input_dim = 28 * 28
batch_size = 128
epochs = 5

dataset = datasets.MNIST('data/', transform=transforms.ToTensor(), download=True)
loader = DataLoader(dataset, batch_size, shuffle=True)
model = BernoulliVAE(input_dim, latent_dim=10, enc_units=200, dec_units=200)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

log = '\r[{:d}/{:d}] NCE: {:.6f}  KL: {:.6f}  Total: {:.6f}'
for epoch in range(1, epochs + 1):
    print(f'Epoch {epoch}')
    avg_loss = 0
    for i, (data, _) in enumerate(loader):
        model.zero_grad()
        # Reshape data so each image is an array with 784 elements
        data = data.view(-1, input_dim)

        nce, kl_div = model(data)
        loss = nce + kl_div
        loss.backward()
        optimizer.step()

        avg_loss += loss.item()/len(dataset)

        if i % 100 == 0:
            # Print average loss per sample in batch
            print(log.format(i + 1, len(loader),
                             nce.item(), kl_div.item(), loss.item()),
                 end='')

    print(f'\nAverage loss: {avg_loss:.6f}'.format(avg_loss))
```

    Epoch 1
    [401/469] NCE: 14406.266602  KL: 1945.858276  Total: 16352.125000
    Average loss: 147.292228
    Epoch 2
    [401/469] NCE: 12620.369141  KL: 2087.734131  Total: 14708.103516
    Average loss: 123.565269
    Epoch 3
    [401/469] NCE: 13314.766602  KL: 2143.032471  Total: 15457.798828
    Average loss: 120.036630
    Epoch 4
    [401/469] NCE: 12593.125000  KL: 2143.898438  Total: 14737.023438
    Average loss: 117.831159
    Epoch 5
    [401/469] NCE: 13047.822266  KL: 2218.240967  Total: 15266.063477
    Average loss: 116.889384

Let's now use the autoencoder to take observations and reconstruct them from their latent representation.

```python
import matplotlib.pyplot as plt
%matplotlib inline

n_samples = 10
fig = plt.figure(figsize=(14, 3))
fig.suptitle('Observations (top row) and their reconstructions (bottom row)')
for i in range(n_samples):
    # Take a sample and view as mini-batch of size 1
    x = dataset[i][0].view(-1, input_dim)
    # Encode the observation
    z, mu_e, logvar_e = model.encode(x)
    # Get reconstruction
    x_d = model.decode(z)

    plt.subplot(2, n_samples, i + 1)
    plt.imshow(x.view(28, 28).data.numpy(), cmap='binary')
    plt.axis('off')
    plt.subplot(2, n_samples, i + 1 + n_samples)
    plt.imshow(x_d.view(28, 28).data.numpy(), cmap='binary')
    plt.axis('off')
```

We can see that the model effectively creates a representation of the latent variable $\mathbf{z}$ that contains enough information to reconstruct a digit very similar to the original observation.

The encoded representation acts as a low dimensional representation of the observation. The digit images have 784 pixels in total, with each pixel having values between 0 and 1. Therefore, the images lie in a certain region of $\mathbb{R}^{784}$. Their encoded representation, on the other hand, lies in a region of $\mathbb{R}^{10}$, which hopefully encodes a compact and meaningful representation of a digit. We can observed this with visualization techniques, such as t-SNE:

```python
from sklearn.manifold import TSNE
import numpy as np

# Select 1000 random samples
sample_idx = np.random.randint(0, len(dataset), size=1000)
X = torch.cat([dataset[i][0].view(-1, input_dim) for i in sample_idx])
labels = dataset.train_labels[sample_idx].data.numpy()

Z, _, _ = model.encode(X)
Z_vis = TSNE(n_components=2).fit_transform(Z.data.numpy())

plt.scatter(Z_vis[:, 0], Z_vis[:, 1], c=labels, cmap=plt.cm.get_cmap("jet", 10))
plt.colorbar();
```

As we expected, numbers of the same class cluster together in some regions of the space. This is possible thanks to the latent space discovered by the autoencoder.

The variational autoencoder is a powerful model for unsupervised learning that can be used in many applications like visualization, machine learning models that work on top of the compact latent representation, and inference in models with latent variables as the one we have explored. A particular example of this last application is reflected in the Bayesian Skip-gram [2], which I plan to explore in the near future.

### References

[1] Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).

[2] Bražinskas, Arthur, Serhii Havrylov, and Ivan Titov. "Embedding Words as Distributions with a Bayesian Skip-gram Model." arXiv preprint arXiv:1711.11027 (2017).

### Some final notes

1. In the Expectation Maximization algorithm, we get around the problem of inference in models with latent variables by calculating the posterior and simply setting $q(\mathbf{z}\vert\boldsymbol{\phi}) = p(\mathbf{z}\vert\mathbf{x},\boldsymbol{\theta})$, which effectively makes the KL divergence equal to zero. However, for some models calculating the posterior is not possible. Furthermore, the EM algorithm calculates updates using the complete dataset, which might not scale up well when we have millions of data points. The VAE addresses these issues by proposing an approximation to the posterior, and optimizing the parameters of the approximation with stochastic gradient descent.

2. Recall the expression for the evidence lower bound:
    $$
    \begin{align}
    \mathcal{L}(q, \boldsymbol{\theta}, \boldsymbol{\phi}) &= \mathbb{E}_q[\log p(\mathbf{x},\mathbf{z}\vert\boldsymbol{\theta}) - \log q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})]\\
    &=
    \mathbb{E}_q[\log p(\mathbf{x}\vert\mathbf{z},\boldsymbol{\theta})] - \text{KL}(q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})\Vert p(\mathbf{z}\vert\boldsymbol{\theta}))
    \end{align}
    $$

    This last formulation reveals what optimizing the lower bound does. Maximizing the first term finds parameters that given the latent variable $\mathbf{z}$, assign high probability to the observation $\mathbf{x}$. We can think of this as a negative reconstruction error, that is, a reconstruction from the latent variable $\mathbf{z}$ to the observation $\mathbf{x}$. Maximizing the second term (including the minus sign) minimizes the KL divergence between $q(\mathbf{z}\vert\mathbf{x},\boldsymbol{\phi})$ and the prior $p(\mathbf{z}\vert\boldsymbol{\theta})$, thus acting as a regularizer that enforces a prior structure that we have specified. Briefly stated, we then have

    $$
    \text{lower bound} = -\text{reconstruction error} - \text{regularization penalty}
    $$

    Therefore, values of $\boldsymbol{\theta}$ and $\boldsymbol{\phi}$ that maximize the lower bound will produce a low reconstruction error and a model that takes into account prior information.


### Suppose we have `n` features and `m` observations

| Index        | $X_{1}$       | $X_{2}$       | $X_{3}$       | .... | .... | $X_{n}$        | y        |
|--------------|---------------|---------------|---------------|------|------|----------------|----------|
| 1            | $x_{1}^{1} $  | $x_{2}^{1}$   | $x_{3}^{1}$   | ...  | ...  | $x_{n}^{1}$    | $y^{1}$  |
| 2            | $x_{1}^{2}$   | $x_{2}^{2}$   | $x_{3}^{2}$   | ...  | ...  | $x_{n}^{2}$    | $y^{2}$  |
| 3            | $x_{1}^{3}$   | $x_{2}^{3}$   | $x_{3}^{3}$   | ...  | ...  | $x_{n}^{3}$    | $y^{3}$  |
| .            | .             | .             | .             | ...  | ...  | .              |          |
| .            | .             | .             | .             | ...  | ...  | .              |          |
| .            | .             | .             | .             | ...  | ...  | .              |          |
| m            | $x_{1}^{m}$   | $x_{2}^{m}$   | $x_{3}^{m}$   | ...  | ...  | $x_{n}^{m}$    | $y^{m}$  |

Here `subscript` denotes feature and `superscript` denote observation number

Suppose the weights of matrix for n features is denoted by column vectors of shape [1, n]

$$ \beta = \begin{bmatrix} \beta_{1} & \beta_{2} & \beta_{3} & .... &  \beta_{n} \end{bmatrix} $$

### we need to calculate prediction for each observation

\begin{equation}
\hat{y^1} = \beta_{0} + \beta_{1}x_{1}^{1} +  \beta_{2}x_{2}^{1}  +  \beta_{3}x_{3}^{1}  +  \beta_{3}x_{3}^{1} + .... + \beta_{n}x_{n}^{1}
\end{equation}

\begin{equation}
\hat{y^2} = \beta_{0} + \beta_{1}x_{1}^{2} +  \beta_{2}x_{2}^{2}  +  \beta_{3}x_{3}^{2}  +  \beta_{3}x_{3}^{2} + .... + \beta_{n}x_{n}^{2}
\end{equation}

\begin{equation}
\hat{y^3} =  \beta_{0} +\beta_{1}x_{1}^{3} +  \beta_{2}x_{2}^{3}  +  \beta_{3}x_{3}^{3}  +  \beta_{3}x_{3}^{3} + .... + \beta_{n}x_{n}^{3}
\end{equation}

\begin{equation}
 ..................
\end{equation}

\begin{equation}
 ..................
\end{equation}

\begin{equation}
\hat{y^m} =  \beta_{0} +\beta_{1}x_{1}^{m} +  \beta_{2}x_{2}^{m}  +  \beta_{3}x_{3}^{m}  +  \beta_{3}x_{3}^{m} + .... + \beta_{n}x_{n}^{m}
\end{equation}

### In matrix form

\begin{equation}
\begin{bmatrix} \hat{y}^{1} \\ \hat{y}^{2} \\ \hat{y}^{3} \\ .. \\.. \\  \hat{y}^{m} \end{bmatrix} =
\begin{bmatrix} x_{1}^{1} & x_{2}^{1} & x_{3}^{1} & .... &  x_{n}^{1}
           \\ x_{1}^{2} & x_{2}^{2} & x_{3}^{2} & .... &  x_{n}^{2}
           \\ x_{1}^{3} & x_{2}^{3} & x_{3}^{3} & .... &  x_{n}^{3}
           \\.... &.... & ... &.... &....
           \\.... &.... & ... &.... &....
            \\ x_{1}^{m} & x_{2}^{m} & x_{3}^{m} & .... &  x_{n}^{m}
\end{bmatrix}
*
\begin{bmatrix} \beta_{1} \\ \beta_{2} \\ \beta_{3} \\ .. \\.. \\  \beta_{n} \end{bmatrix}
+ \beta_{0}
\end{equation}

\begin{equation} \hat{y} = X.\beta^{T} + \beta_{0} \end{equation}

#### Equivalent numpy implementation is:

 `y_hat = np.dot(X, B.T) + b`
Where
 $$ B = \beta $$
 $$ b = \beta_{0}$$

### The Mean Squared Error cost function is:

$$ J(\beta, \beta_{0}) = \frac{1}{2m}\sum_{n=1}^{m}   (y^{(i)} - \hat{y}^{(i)})^{2} $$

#### Equivalent numpy implementation is:

`cost = (1/(2*m))*np.sum((y-y_hat)**2)`

### Just for confirmation let us take an example

| y | $\hat{y}$ | (y - $\hat{y}$)^2 |
|---|---------|-----------------|
| 2 | 1       | 1               |
| 4 | 2       | 4               |
| 6 | 3       | 9               |
|  |        | 14               |

\begin{equation} 14/3 = 4.67 \end{equation}

```python
import numpy as np

y=np.array([[2],
           [4],
           [6]])
y_hat=np.array([[1],
           [2],
           [3]])
```

```python
np.sum((y-y_hat)**2)/3
```

    4.666666666666667

### Now lets calculate the gradient descent

### We have the loss function  defined as

$$ J(\beta, \beta_{0}) = \frac{1}{2m}\sum_{n=1}^{m}   (y - \hat{y})^{2}   $$

where: $$ \hat{y} = \beta_{0} + X*\beta^{T} $$

So:
$$ \frac {\partial J(\beta, \beta_{0})}{\partial \beta_{0}} =
\frac{-1}{m}\sum_{n=1}^{m}(y-\hat{y}) $$

Again: $$ J(\beta, \beta_{0}) = \frac{1}{2m}\sum_{n=1}^{m}   (y - \hat{y})^{2}   $$

$$ \frac {\partial J(\beta, \beta_{0})}{\partial \beta} = \frac{-1}{m}(y - \hat{y}) * \frac{\partial (\beta_{0} + X*\beta^{T})}{\partial \beta} $$

$$ \frac {\partial J(\beta, \beta_{0})}{\partial \beta} =
\frac{-1}{m}(y-\hat{y})*X * \frac{\partial \beta^{T}}{\partial \beta} $$

where $$ \beta = \begin{bmatrix} \beta_{1} & \beta_{2} & \beta_{3} &.... & \beta_{n} \end{bmatrix}$$

and

$$ \beta^{T} = \begin{bmatrix} \beta_{1} \\ \beta_{2} \\ \beta_{3} \\.... \\ \beta_{n} \end{bmatrix}$$

Since: $$ \frac{\partial \beta^{T}}{\partial \beta} = I $$

### Just a side note for calculation of $$ \frac{\partial \beta^{T}}{\partial \beta} $$
where $$ \beta = \begin{bmatrix} \beta_{1} & \beta_{2} & \beta_{3} &.... & \beta_{n} \end{bmatrix}$$

and

$$ \beta^{T} = \begin{bmatrix} \beta_{1} \\ \beta_{2} \\ \beta_{3} \\.... \\ \beta_{n} \end{bmatrix}$$

$$ \frac{\partial \beta^{T}}{\partial \beta} = \begin{bmatrix} \frac{\partial \beta^{T}}{\partial \beta_{1}} & \frac{\partial \beta^{T}}{\partial \beta_{2}} & \frac{\partial \beta^{T}}{\partial \beta_{3}} & ....  & \frac{\partial \beta^{T}}{\partial \beta_{n}}\end{bmatrix} $$

\begin{equation}
\frac{\partial \beta^{T}}{\partial \beta} =
\begin{bmatrix}
\frac{\partial \beta_{1}}{\partial \beta_{1}} & \frac{\partial \beta_{1}}{\partial \beta_{2}} & \frac{\partial \beta_{1}}{\partial \beta_{3}} & ..&..& \frac{\partial \beta_{1}}{\partial \beta_{n}}
\\
\frac{\partial \beta_{2}}{\partial \beta_{1}} & \frac{\partial \beta_{2}}{\partial \beta_{2}} & \frac{\partial \beta_{2}}{\partial \beta_{3}} &..&..& \frac{\partial \beta_{2}}{\partial \beta_{n}}
\\
\frac{\partial \beta_{2}}{\partial \beta_{1}} & \frac{\partial \beta_{3}}{\partial \beta_{2}} & \frac{\partial \beta_{3}}{\partial \beta_{3}} &..&..& \frac{\partial \beta_{3}}{\partial \beta_{n}}
\\
.. & .. & .. & ..  & .. &..
\\
.. & .. & .. & ..  & .. &..
\\
\frac{\partial \beta_{n}}{\partial \beta_{1}} & \frac{\partial \beta_{n}}{\partial \beta_{2}} & \frac{\partial \beta_{n}}{\partial \beta_{3}} & ..&..& \frac{\partial \beta_{n}}{\partial \beta_{n}}
\end{bmatrix}
\end{equation}

\begin{equation}
\frac{\partial \beta^{T}}{\partial \beta} =
\begin{bmatrix}
1 & 0 & 0 & ..&..& 0
\\
0 & 1 & 0 &..&..& 0
\\
0 & 0 & 1 &..&..& 0
\\
.. & .. & .. & ..  & .. &..
\\
.. & .. & .. & ..  & .. &..
\\
0 & 0 & 0 & ..&..& 1
\end{bmatrix}
\end{equation}

i.e $$  \frac{\partial \beta^{T}}{\partial \beta} = I_{n*n} $$

Since: $$ X_{m*n} * I_{n*n} = X $$

So:
 $$ \frac {\partial J(\beta, \beta_{0})}{\partial \beta} =
\frac{-1}{m}(y-\hat{y})*X  $$

### Shape of $$ (y-\hat{y}) $$  `m rows and 1 cols [m, 1]`

### Shape of `X is [m, n] ie m observations and n features`

### Required shape of `dB is [1, n] `

### [1, m] * [m, n] == [1, n]

## Equivalent Numpy Implementation is:

 `dB = (-1/m)* np.dot((y-y_hat).T, X)`

  `db = (-1/m)*np.sum(y-y_hat)`

```python
def propagate(B, b, X, Y):
    """
    params:
    B: weights of size [1, X.shape[1]]
    b: bias
    X: matrix of observations and features size [X.shape[0], X.shape[1]]
    Y: matrix of actual observation size [Y.shape[0], 1]

    returns:
    grads: dict of gradients, dB of shape same as B and db of shape [1, 1].
    cost: MSE cost
    """

    ## m is no of observations ie rows of X
    m = X.shape[0]

    #Calculate hypothesis
    y_hat = np.dot(X, B.T) + b

    y = Y.values.reshape(Y.shape[0],1)

    #Compute Cost
    cost = (1/(2*m))*np.sum((y-y_hat)**2)

    # BACKWARD PROPAGATION (TO FIND GRAD)
    dB = (-1/m)* np.dot((y-y_hat).T, X)

    db = -np.sum(y-y_hat)/m

    grads = {"dB": dB,
             "db": db}

    return grads, cost
```

```python
def optimize(B, b, X, Y, num_iterations, learning_rate):
    """
    params:
    B: weights of size [1, X.shape[1]]
    b: bias
    X: matrix of observations and features size [X.shape[0], X.shape[1]]
    Y: matrix of actual observation size [Y.shape[0], 1]
    num_iterations: number of iterations
    learning_rate: learning rate
    returns:
    params: parameters B of shape [1, X.shape[1]] and bias
    grads: dict of gradients, dB of shape same as B and db
    costs:  MSE cost
    """
    costs = []

    for i in range(num_iterations):

        # Cost and gradient calculation call function propagate
        grads, cost = propagate(B,b,X,Y)

        # Retrieve derivatives from grads
        dB = grads["dB"]
        db = grads["db"]

        # update parameters
        B = B - learning_rate * dB
        b = b - learning_rate * db

        costs.append(cost)

    params = {"B": B,
              "b": b}

    grads = {"dB": dB,
             "db": db}

    return params, grads, costs
```

```python
def predict(B, b, X):
    """:param
    B: weights
    b: bias
    X: matrix of observations and features
    """
  # Compute predictions for X
    Y_prediction = np.dot(X, B.T) + b
    return Y_prediction
```

```python
def model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5):
    """
    params:
    X_train: X_train
    Y_train: Y_train
    X_test: X_test
    Y_test: Y_test

    returns:
    d: dictionary
    """

    # initialize parameters with zeros
    B = np.zeros(shape=(1, X_train.shape[1]))
    b = 0

    # Gradient descent
    parameters, grads, costs = optimize(B, b, X_train, Y_train, num_iterations, learning_rate)

    # Retrieve parameters w and b from dictionary "parameters"
    B = parameters["B"]
    b = parameters["b"]

    # Predict test/train set examples
    Y_prediction_test = predict(B, b, X_test)
    Y_prediction_train = predict(B, b, X_train)

    Y_train = Y_train.values.reshape(Y_train.shape[0], 1)
    Y_test = Y_test.values.reshape(Y_test.shape[0], 1)

   # Print train/test Errors
    print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))
    print("test accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))

    d = {"costs": costs,
         "Y_prediction_test": Y_prediction_test,
         "Y_prediction_train" : Y_prediction_train,
         "B" : B,
         "b" : b,
         "learning_rate" : learning_rate,
         "num_iterations": num_iterations}

    return d
```

```python
import pandas as pd
df = pd.read_csv('../datasets/USA_Housing.csv')
```

```python
df
```

      Avg. Area Income
      Avg. Area House Age
      Avg. Area Number of Rooms
      Avg. Area Number of Bedrooms
      Area Population
      Price
      Address

      0
      79545.458574
      5.682861
      7.009188
      4.09
      23086.800503
      1.059034e+06
      208 Michael Ferry Apt. 674\nLaurabury, NE 3701...

      1
      79248.642455
      6.002900
      6.730821
      3.09
      40173.072174
      1.505891e+06
      188 Johnson Views Suite 079\nLake Kathleen, CA...

      2
      61287.067179
      5.865890
      8.512727
      5.13
      36882.159400
      1.058988e+06
      9127 Elizabeth Stravenue\nDanieltown, WI 06482...

      3
      63345.240046
      7.188236
      5.586729
      3.26
      34310.242831
      1.260617e+06
      USS Barnett\nFPO AP 44820

      4
      59982.197226
      5.040555
      7.839388
      4.23
      26354.109472
      6.309435e+05
      USNS Raymond\nFPO AE 09386

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      4995
      60567.944140
      7.830362
      6.137356
      3.46
      22837.361035
      1.060194e+06
      USNS Williams\nFPO AP 30153-7653

      4996
      78491.275435
      6.999135
      6.576763
      4.02
      25616.115489
      1.482618e+06
      PSC 9258, Box 8489\nAPO AA 42991-3352

      4997
      63390.686886
      7.250591
      4.805081
      2.13
      33266.145490
      1.030730e+06
      4215 Tracy Garden Suite 076\nJoshualand, VA 01...

      4998
      68001.331235
      5.534388
      7.130144
      5.44
      42625.620156
      1.198657e+06
      USS Wallace\nFPO AE 73316

      4999
      65510.581804
      5.992305
      6.792336
      4.07
      46501.283803
      1.298950e+06
      37778 George Ridges Apt. 509\nEast Holly, NV 2...

5000 rows × 7 columns

```python
df.drop(['Address'],axis=1,inplace=True)
```

```python
df.head()
```

      Avg. Area Income
      Avg. Area House Age
      Avg. Area Number of Rooms
      Avg. Area Number of Bedrooms
      Area Population
      Price

      0
      79545.458574
      5.682861
      7.009188
      4.09
      23086.800503
      1.059034e+06

      1
      79248.642455
      6.002900
      6.730821
      3.09
      40173.072174
      1.505891e+06

      2
      61287.067179
      5.865890
      8.512727
      5.13
      36882.159400
      1.058988e+06

      3
      63345.240046
      7.188236
      5.586729
      3.26
      34310.242831
      1.260617e+06

      4
      59982.197226
      5.040555
      7.839388
      4.23
      26354.109472
      6.309435e+05

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 5000 entries, 0 to 4999
    Data columns (total 6 columns):
    Avg. Area Income                5000 non-null float64
    Avg. Area House Age             5000 non-null float64
    Avg. Area Number of Rooms       5000 non-null float64
    Avg. Area Number of Bedrooms    5000 non-null float64
    Area Population                 5000 non-null float64
    Price                           5000 non-null float64
    dtypes: float64(6)
    memory usage: 234.5 KB

```python
#normalization of column values
df_norm = (df - df.mean()) / (df.max() - df.min())

# Putting feature variable to X
X = df_norm[['Avg. Area Income','Avg. Area House Age','Avg. Area Number of Rooms','Avg. Area Number of Bedrooms','Area Population']]

# Putting response variable to y
y = df_norm['Price']

#random_state is the seed used by the random number generator, it can be any integer.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7 ,test_size = 0.3, random_state=2)

```

```python
df_norm.head()
```

      Avg. Area Income
      Avg. Area House Age
      Avg. Area Number of Rooms
      Avg. Area Number of Bedrooms
      Area Population
      Price

      0
      0.121932
      -0.042817
      0.002844
      0.024149
      -0.188292
      -0.070538

      1
      0.118631
      0.003735
      -0.034156
      -0.198073
      0.057734
      0.111620

      2
      -0.081153
      -0.016194
      0.202692
      0.255260
      0.010348
      -0.070557

      3
      -0.058260
      0.176153
      -0.186228
      -0.160296
      -0.026685
      0.011636

      4
      -0.095667
      -0.136247
      0.113193
      0.055260
      -0.141246
      -0.245046

```python
X_train.shape
```

    (3500, 5)

```python
X_train.head()
```

      Avg. Area Income
      Avg. Area House Age
      Avg. Area Number of Rooms
      Avg. Area Number of Bedrooms
      Area Population

      2416
      0.129642
      -0.143456
      0.003923
      -0.169184
      -0.027249

      2417
      -0.094771
      -0.263002
      0.052597
      -0.164740
      0.132247

      2513
      -0.019134
      0.219268
      -0.100217
      -0.109184
      -0.300045

      1698
      -0.033811
      -0.295470
      0.058020
      0.533038
      -0.104025

      3322
      0.030541
      -0.015484
      -0.129778
      -0.329184
      -0.135710

```python
y_train.head()
```

    2416   -0.041265
    2417   -0.182448
    2513   -0.082988
    1698   -0.196576
    3322   -0.069286
    Name: Price, dtype: float64

```python
y_test.shape[0]
```

    1500

```python
model1 = model(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, num_iterations = 1000, learning_rate = 0.001)

model2 = model(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, num_iterations = 1000, learning_rate = 0.01)

model3 = model(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, num_iterations = 1000, learning_rate = 0.1)

model4 = model(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, num_iterations = 1000, learning_rate = 0.3)
```

    train accuracy: 88.62860037424024 %
    test accuracy: 88.8357850464578 %
    train accuracy: 90.16271542913721 %
    test accuracy: 90.33382459987604 %
    train accuracy: 95.97241738688837 %
    test accuracy: 96.00886188507282 %
    train accuracy: 96.6522103927996 %
    test accuracy: 96.73232357323806 %

```python
import matplotlib.pyplot as plt
plt.plot([i for i in range(1000)], model1['costs'])
plt.plot([i for i in range(1000)], model2['costs'])
plt.plot([i for i in range(1000)], model3['costs'])
plt.plot([i for i in range(1000)], model4['costs'])

plt.gca().legend(('alpha 0.001','alpha 0.01', 'alpha 0.1', 'alpha 0.3'))
plt.show()
```

```python
# model4 = model(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, num_iterations = 10, learning_rate = 0.01)

# model5 = model(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, num_iterations = 100, learning_rate = 0.01)

# model6 = model(X_train=X_train, Y_train=y_train, X_test=X_test, Y_test=y_test, num_iterations = 1000, learning_rate = 0.01)
```