```python
import numpy as np
import sympy as sy
import control.matlab as cm
```

```python
z = sy.symbols('z', real=False)
hh,r1,s0,s1, aa = sy.symbols('h,r1,s0,s1, a')
pc1 = -1.555-1j*1.555
pc2 = np.conjugate(pc1)
Tr = 1
omega0 = 2.2/Tr
#h = 0.2/omega0
h = Tr/10.0
a = -2*omega0
ad = sy.exp(h*a)
#ad = sy.symbols('a_d')
A2p = sy.simplify(sy.expand(sy.poly((z-np.exp(h*pc1))*(z-np.exp(h*pc2)), z))) # Desired closed loop poles
Acp = sy.simplify(sy.expand(sy.poly((z-np.exp(h*pc1))*(z-np.exp(h*pc2))*(z - ad), z))) # Desired charact polynomial
Ap = sy.poly((z-1)**2, z) # Plant denominator, double integrator
Bp = sy.poly(h**2/2*(z+1), z)
Rp = sy.poly(z+r1, z)
Sp = sy.poly(s0*z + s1, z)
dioph=(Ap*Rp+Bp*Sp-Acp).all_coeffs()
print A2p
print Acp
print dioph
```

    Poly(1.0*z**2 - 1.69131784560334*z + 0.732713875869332, z, domain='RR')
    Poly(1.0*z**3 - 2.33535426668648*z**2 + 1.82198416806576*z - 0.471894422292842, z, domain='RR')
    [1.0*r1 + 0.005*s0 + 0.335354266686485, -2.0*r1 + 0.005*s0 + 0.005*s1 - 0.821984168065758, 1.0*r1 + 0.005*s1 + 0.471894422292842]

```python
print sy.im(sy.exp(h*pc1))
print z-sy.exp(h*a)
print z-np.exp(h*pc1)
```

    -0.132570215943283
    z - 0.644036421083141
    z - 0.845658922801672 + 0.132570215943283*I

```python
sol=sy.solve(dioph, (r1,s0,s1))
print sol[r1]
print sol[s0]
print sol[s1]

t0 = A2p.evalf(subs={z:1})/Bp.evalf(subs={z:1,})
print t0
```

    -0.856007930368168*a_d + 0.143992069631831
    -28.7984139263664*a_d + 32.9380169529651
    24.6588108997672*a_d - 28.7984139263663
    4.13960302659890

```python
print A2p
1-1.69+0.737
```

    Poly(1.0*z**2 - 1.69131784560334*z + 0.732713875869332, z, domain='RR')

    0.04700000000000004

```python

G = Km * cm.tf([1], [tau, 1, 0])
Gd = Km * cm.tf([tau*(hpt-1+np.exp(-hpt)), tau*(1-(1+hpt)*np.exp(-hpt))], [1, -(1+np.exp(-hpt)), np.exp(-hpt)], h)
Gd2 = cm.c2d(G, h)
print Gd
print Gd2
```

     0.001339 z + 0.001247
    ----------------------
    z^2 - 1.807 z + 0.8071

    dt = 0.03

     0.001339 z + 0.001247
    ----------------------
    z^2 - 1.807 z + 0.8071

    dt = 0.03

```python
print A2p
print A2p.evalf(subs={z:1})
print Bp
print Bp.evalf(subs={z:1})

```

    Poly(z**2 - 1.58555290309441*z - (-0.792776451547206 + 0.168974310731771*I)*(0.792776451547206 + 0.168974310731771*I), z, domain='EX')
    0.0714939167206446
    Poly(0.00133942860759726*z + 0.00124712240506047, z, domain='RR')
    0.00258655101265772

```python
0.3/(5*np.sqrt(2))
```

    0.042426406871192847

```python
np.exp(-0.21)*np.sin(0.21)
```

    0.16897431073177133

```python
np.exp(0.03*(-14))
```

    0.65704681981505675

```python

```


# Exercise 1 2021/NJC/Functions/Q2 H2 Mathematics
For some unknown constants $a$ and $k$, the functions $f$, $g$ and $h$ are defined by
$$ f	:x\mapsto\left(2-x\right)\left(4+x\right),\,x<2,$$
$$ g	:x\mapsto\left(2-x\right)\left(4+x\right),\,x\leq k,$$
$$ h	:x\mapsto2^{x^{3}},\,x\in\left(-\infty,a\right).$$

- (i)	Find the range of $f$, and show that $f^{-1}$ does not exist.
- (ii)	Find the greatest value of $k$ such that $g^{-1}$  exists. Using this greatest value of $k$, define $g^{-1}$  in a similar form.
- (iii)	Find the range of values of $a$ such that the composite function $fh$ exists.

## Answer
- (i) The graph of $f$ looks like

The maximum turning point is at $(1,9)$. Thus, $R_f=(-\infty,9]$.

The horizontal line $y=1$ passes through the graph twice, thus, the function cannot be one-to-one and consequently, cannot have an inverse.

- (ii) From the graph drawn earlier, the greatest value of $k$ for $g^{-1}$ to exist is 1. With the value of $k$, let $y=(2-x)(4+x)$, we make $x$ the subject.

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('(2-x)*(4+x)')

print_inverse(f)
```

$\displaystyle x=- \sqrt{9 - y} - 1$

$\displaystyle x=\sqrt{9 - y} - 1$

Since $x\leq k$, $g^{-1}:x\mapsto -1-\sqrt{9-x}$.

- (iii) For $fh$ to exists, we need $R_h\subseteq D_f$ as such we need to solve $2^{a^3}<2$, so $a< (\frac{\ln 9}{\ln 2})^{\frac{1}{3}}=1$.

# Exercise 2 2021/NJC/Functions/Q2 H2 Mathematics
The functions $f$ and $g$ are defined by
$$ f:x\mapsto e^{\left|3-x\right|},\,x\in\mathbb{R},$$
$$ g:x\mapsto\left(x-1\right)^{2}+a,\,x\in\mathbb{R}\,\text{and\,}a\,\text{is a positive constant.}$$

- (i)	Sketch the graph of $y = f(x)$, and show that $f^{-1}$  does not exist.
- (ii)	The function $f$ has an inverse if its domain is restricted to $x\geq b$. State the smallest possible value of $b$ and define, in similar form, the inverse function $f^{-1}$  corresponding to this domain for $f$.
- (iii)	Using the value of $b$ in (ii), find the smallest possible value of $a$ such that the composite function $f^{-1}g$  exists. State the range of $f^{-1}g$  for this value of $a$.

## Answer
(i) We have that

The horizontal line $y=2$ passes through the graph twice, thus, the function cannot be one-to-one and consequently, cannot have an inverse.

(ii) From the graph, we see that that the smallest possible value of $b$ is 1.

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('(exp(1))**(x-3)')

print_inverse(f)
```

$\displaystyle x=\log{\left(y \right)} + 3$

Thus, $f^{-1}(x)=\ln(x)+3$.

- (iii) $f^{-1}g$ exists if $[a,\infty)=R_g \subseteq D_{f^{-1}}=R_f=[1,\infty)$. Therefore, the smallest possible value of $a$ is 1.

If $a=1$, $R_g=[1,\infty)$ and therefore, $R_{f^{-1}g}=[3,\infty)$ based on the graph drawn in part(i).

# Exercise 3 2021/NJC/Functions/Q3 H2 Mathematics
It is given that $\lambda$  is an unknown constant. The functions $f$ and $g$ are defined as follows:
$$ f:x\mapsto\frac{5-x}{1-x},\,x\in\mathbb{R},x\neq1,$$
$$ g:x\mapsto2x^{2}+4x+\lambda,\,x\in\mathbb{R},x>-2.$$

- (a) (i) Explain why $f^{-1}$ exists, and show that $f^{-1}(x)=f(x)$.
- (a) (ii) Hence, or otherwise, evaluate $f^{51}(4)$, where $f^{n}(x)$ denotes
$$ \underset{n\,\text{times}}{\underbrace{fff\cdots f}}(x).$$
- (b) (i)	Find the range of values of $\lambda$ such that $fg$ exists.
- (b) (ii)	Given that $fg$ exists, find the range of $fg$ in terms of $\lambda$.

## Answer
- (a) (i) We observe that the graph of $f(x)$ is a one-to-one function as any horizontal line $y=k$, $k\in \mathbb{R}$ cuts the graph at most once.

As such, the inverse function exist. Let $y=\frac{5-x}{1-x}$, we make $x$ the subject

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('(5-x)/(1-x)')

print_inverse(f)

```

$\displaystyle x=\frac{y - 5}{y - 1}$

Thus, $f^{-1}(x)=\frac{x-5}{x-1}=f(x)$.

- (a) (ii) From part (i), we see that $f^2(x)=x$, which further implies that $f^51(x) =f^{{50}}(f(x))=f(x)=\frac{5-x}{1-x}$. As such, $$f^{{51}}(4)=\frac{5-4}{1-4}=-\frac{1}{3}.$$

We verify our solution by bruteforcing our way to get the composite function.

```python
from sympy import *
from h2_math import *

x = symbols('x')

# Defining a function with name f
f = make_fn('(5-x)/(1-x)')

print('Checking the composition of function')
display(Math(f'f^{{51}}(x)={ latex(self_compose(f,51))}'))

print('Evaluating at x=4')
# We use .subs() method to do the substitution and evaluate them
display(Math(f'f^{{51}}(4)={ latex(self_compose(f,51).subs(x,4))}'))
```

    Checking the composition of function

$\displaystyle f^{51}(x)=\frac{x - 5}{x - 1}$

    Evaluating at x=4

$\displaystyle f^{51}(4)=- \frac{1}{3}$

- (b) (i) $fg$ exists when $R_g \subseteq D_f$. First, we note that by completing square $g(x)=2\left(x+1\right)^{2}+\left(\lambda-2\right)$ with minimum turning point point at $x=-1>-2$.

As such, $R_g=[\lambda -2,\infty)$. Thus, for $R_g$ to be a subset of $D_f$, we would then require

\begin{align*}
\lambda - 2 &> 1\\
\lambda &>3.
\end{align*}

- (b) (ii) We note that for $x>1$, if $x_0<x_1$, $f(x_0)<f(x_1)$ and $f(x)<1$ as $x=1$ is a vertical asympotote of $f$. Therefore,

\begin{align*}
R_{fg} &= [f(\lambda - 2),1)\\
&= [\frac{\lambda - 7}{\lambda -3},1).
\end{align*}

# Exercise 4 2021/NJC/Functions/Q4 H2 Mathematics
The function $f$ is defined by
$$f:x\mapsto\frac{ax}{bx-a},\,x\in\mathbb{R},x\neq\frac{a}{b},$$

where $a$ and $b$ are non-zero constants.

- (a) (i)	Determine if $f^{-1}$  exists. If yes, find $f^{-1}(x)$.
- (a) (ii)	Hence, or otherwise, find the rule of the composite function, $f^2(x)$  and state the range of $f^2$.
- (a) (iii)	Solve the equation $f^{-1}(x)=x$.

- (b) The function $g$ is defined by
		$$ g:x\mapsto\frac{1}{x},\text{for all real non-zero }x.$$
	State whether the composite function $fg$ exists, justifying your answer.

## Answer
- (a) (i) We observe that the graph of $f(x)$ is a one-to-one function as any horizontal line $y=k$, $k\in \mathbb{R}$ cuts the graph at most once.

As such, the inverse function exist. Let $y=\frac{ax}{bx-a}$, we make $x$ the subject

```python
# Modules for algebraic manipulation and pretty printing
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Since we have more symbols beyond x in our expression
# We also need to represent the real numbers a and b symbolically
a, b = symbols('a b', real=True)

# Defining a function with name f
class f(Function):
    @classmethod
    def eval(cls, x):
        # Return the rule of the function
        return (a*x)/(b*x-a)

print_inverse(f)
```

$\displaystyle x=\frac{a y}{- a + b y}$

Thus, $f^{-1}(x)=\frac{a x}{- a + b x}$. From the graph, we see that $R_f = (-\infty,0) \cup (0,\infty)$. Therefore, $D_{f^{-1}} = (-\infty,0) \cup (0,\infty)$ as well.

- (a) (ii) The smart way is to notice that $f(x)=f^{-1}(x)$ and as such,

\begin{align*}
f(x) &= f^{-1}(x)\\
f(f(x)) &= f(f^{-1}(x))\\
f^2(x) &= x.
\end{align*}

Altenatively, we can brute force it by evaluating the rule of the composite function via the definition given.

```python
# Modules for algebraic manipulation and pretty printing
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# We also need to represent the real numbers a and b symbolically
a, b = symbols('a b', real=True)

# Defining a function with name f
class f(Function):
    @classmethod
    def eval(cls, x):
        # Return the rule of the function
        return (a*x)/(b*x-a)

print('Checking the composition of function')
display(Math(f'f^2(x)={ latex(self_compose(f,2,simp=False))}'))

print('After simplification')
display(Math(f'f^2(x)={ latex(self_compose(f,2))}'))
```

    Checking the composition of function

$\displaystyle f^2(x)=\frac{a^{2} x}{\left(- a + b x\right) \left(\frac{a b x}{- a + b x} - a\right)}$

    After simplification

$\displaystyle f^2(x)=x$

Since $f^2(x)=x$ and $D_f=\mathbb{R}\backslash\left\{ \frac{a}{b}\right\} $, we have $R_f=\mathbb{R}\backslash\left\{ \frac{a}{b}\right\} $ as well.

- (a) (iii) We have that

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# We also need to represent the real numbers a and b symbolically
a, b = symbols('a b', real=True)

# Defining a function with name f
class f(Function):
    @classmethod
    def eval(cls, x):
        # Return the rule of the function
        return (a*x)/(b*x-a)

for x_vals in solve(f(x)-x,x):
    display(Math(f'x={ latex(x_vals) }.'))
```

$\displaystyle x=0.$

$\displaystyle x=\frac{2 a}{b}.$

- (b) Observe that the graph of $g$ looks like

As such $R_g=\mathbb{R}\backslash{0}$. Since $D_f=\mathbb{R}\backslash\left\{ \frac{a}{b}\right\} $, we have that $R_g \nsubseteq D_f$ as $\frac{a}{b}\in R_g$, but $\frac{a}{b} \notin D_f$. Thus, the function $fg$ does not exist.

## Exercise 5 2021/NJC/Functions/Q4 H2 Mathematics

It is given that

$$f\left(x\right)=\left|\frac{x-2a}{2}\right|\,\text{for}\,0\leq x<4a,\,\text{where }a\,\text{is a positive constant,}$$

and that $f(x)=f(x+4a)$ for all real values of $x$.

- **(i)**	Find the values of $f(-5a)$ and $f(8a)$ in terms of $a$.
- **(ii)**	Sketch the graph of $y=f(x)$ for $-6a\leq x\leq 10a$.
- **(iii)**	Hence or otherwise, find the exact value of ${\displaystyle \int_{-6a}^{10a}f\left(x\right)\,dx}$  in terms of $a$.

### Answer
- (i) Since $f(x) = f(x+4a)$ and $a$ is a positive constant, we have that
$$\begin{align*}
f(-5a) &= f(-a) \\
    &= f(3a) \\
    &= \left|\frac{3a-2a}{2}\right| \\
    &=\left|\frac{a}{2}\right| \\
    &=\frac{a}{2} \\
f(8a) &= f(4a) \\
    &= f(0) \\
    &= \left|\frac{0-2a}{2}\right| \\
    &=\left|\frac{-2a}{2}\right| \\
    &=\left|a\right| \\
    &=a \\
\end{align*}$$

- (ii) The graph looks like

- (iii) From the graph above, we can easily see that that ${\displaystyle \int_{-6a}^{10a}f\left(x\right)\,dx}$ is
    $$\begin{align}
    	4\left(\frac{1}{2}\left(2a-\left(-2\right)\right)\left(a\right)\right) = 8a^{2}.
    \end{align}$$

# Exercise 6 2021/NJC/Functions/Q4 H2 Mathematics
The function $f$ is defined as
$$ f:x\mapsto x^2-\lambda x+3, x\in \mathbb{R},$$

where $\lambda$ is a non-zero constant.
- **(i)** Find the range of $f$, giving your answer in terms of $\lambda$.

The function $f$ has an inverse function if its domain is restricted to $x\leq k$.
- **(ii)** State the greatest value of $k$ in terms of $\lambda$.

Using the result in **(ii)** for the case $\lambda=4$,

- **(iii)**	sketch, on the same diagram, the graphs of $y=f(x)$ and $y=f^{-1}(x)$, illustrating clearly the relationship between the two graphs, and labelling the axial intercept(s), if any.
- **(iv)** Find the exact solution of $f(x)=f^{-1}(x)$.

- (i) By completing the square, we see that
    $$ \begin{align*}
    x^2-\lambda x+3 &= \left(x-\frac{\lambda}{2}\right)^{2}+3-\left(\frac{\lambda}{2}\right)^{2} \\
                    &\geq 3-\frac{\lambda^2}{4}.
    \end{align*}$$
    Therefore, $R_f=[3-\frac{\lambda^2}{4},\infty)$.
- (ii) Since $f$ is a quadratic function with only one minimum turning point in the domain (\mathbb{R}) at $x=\frac{\lambda}{2}$, $k=\frac{\lambda}.{2}$.
- (iii) The graphs is given below.

- (iv) We see that the solution to $f\left(x\right)=f^{-1}(x)$ is also a solution $f(x)=x$ (observe the graph). Thus, we have that

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('x**2-4*x+3')

for x_vals in solve(f(x)-x,x):
    # The next line is for use in Markdown
    # print(latex(x_vals))
    display(Math(f'x={ latex(x_vals) }.'))
```

$\displaystyle x=\frac{5}{2} - \frac{\sqrt{13}}{2}.$

$\displaystyle x=\frac{\sqrt{13}}{2} + \frac{5}{2}.$

Since $x \leq \frac{\lambda}{2}=\frac{4}{2}=2$, we have that $x= \frac{5}{2} - \frac{\sqrt{13}}{2}$.

# Exercise 7 2021/NJC/Functions/Q7 H2 Mathematics
The function $f$ is defined by
$$f:x\mapsto (7+x)(1-x)-15, x\in [-2,\infty).$$

- (i)	Show that $f^{-1}$ exists, and find its domain.
- (ii)	Sketch, on the same diagram, the graphs of
	$$y =f(x), y=f^{-1}(x) \text{ and } y=f^{-1}f(x),$$
	showing clearly the relationship between the graphs.
- (iii)	Hence find the exact solution of $f^{-1}f(x)\leq f(x)$.

### Solution
- (i) From the following sketch of the graph of $y=f(x)$, any horizontal line intersects the graph of $y=f(x)$ at most once. Thus the function $f$ is one-one. It follows that $f^{-1}$ exists. Note that $D_{f^{-1}}=R_f=(-\infty,0]$.

- (ii) We have

- (iii) From the graph in (ii), we observe that the solution to $f^{-1}f(x)\leq f(x)$  is in the form $-2\leq x\leq k$  where $k$ is the $x$-coordinate of the point of intersection of the three curves: $y=f(x)$, $y=f^{-1}(x)$ and $y=f^{-1}f(x)=x$.
    Solving $f(x)=x$ for $k$:

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('(7+x)*(1-x)-15')

for x_vals in solve(f(x)-x,x):
    # The next line is for use in Markdown
    # print(latex(x_vals))
    display(Math(f'x={ latex(x_vals) }.'))
```

$\displaystyle x=- \frac{7}{2} - \frac{\sqrt{17}}{2}.$

$\displaystyle x=- \frac{7}{2} + \frac{\sqrt{17}}{2}.$

Since $-2\leq x$, $k=- \frac{7}{2} + \frac{\sqrt{17}}{2}$. As such, $-2\leq x \leq - \frac{7}{2} + \frac{\sqrt{17}}{2}$.

# Exercise 8 2021/NJC/Functions/Q8 H2 Mathematics
The functions $f$ and $g$ are defined by
$$\begin{align*}
f	:x &\mapsto\frac{3x}{x+2},\ &,x>-2\\
g	:x &\mapsto x^{2}+1,\ &,x\leq5
\end{align*}$$

Sketch, on separate diagrams, the graph of $y=f(x)$ and $y=g(x)$.

Determine, whether each of the following composite functions exists:
- (a) $fg$
- (b) $gf$

If any of the functions exists, then find its rule, domain and range.

### Solution
We have the graphs

- (a) From the graphs above, we see that $R_g=[1,\infty)$ and $D_f=(-2,\infty)$. As $R_g \subseteq D_f$, $fg$ exists. The composition is given by

```python
from sympy import *
from h2_math import *

# Defining the symbol for the parameters/arguments/input of the functions f,g
x = symbols('x')

# Defining functions f and g
f = make_fn('(3*x)/(x+2)')
g = make_fn('x**2+1')

print('The rule of fg')
#Compose f(g(x))
display(Math(f'fg(x)={ latex(f(g(x)))}'))

# print(latex(f(g(x)))) For Markdown use
```

    The rule of fg

$\displaystyle fg(x)=\frac{3 x^{2} + 3}{x^{2} + 3}$

By definition, $D_{fg}=D_g=(-\infty,5]$.
To find the range, observe that $y=f(x)$ is an increasing function, i.e as $x_1\leq x_2$, we have $f(x_1)\leq f(x_2)$. As such, as $R_g=[1,\infty)$, we have $R_{fg}=[1,3)$.

- (b) Similarly, from the graphs above, we see that $R_f=[-\infty,3)$ and $D_f=(-\infty,5)$. As $R_f \subseteq D_g$, $gf$ exists. The composition is given by

```python
from sympy import *
from h2_math import *

# Defining the symbol for the parameters/arguments/input of the functions f,g
x = symbols('x')

# Defining functions f and g
f = make_fn('(3*x)/(x+2)')
g = make_fn('x**2+1')

print('The rule of gf')
#Compose g(f(x))
display(Math(f'gf(x)={ latex(g(f(x)))}'))

#print(latex(g(f(x)))) #For Markdown use
```

    The rule of gf

$\displaystyle gf(x)=\frac{9 x^{2}}{\left(x + 2\right)^{2}} + 1$

By definition, $D_{gf}=D_f=(-2,\infty]$.
Unlike part (i), we can't $g$ isn't an increasing function (it has a local minimum turning point). As such, to find the range, we can just sketch the graph $gf$ with the domain $D_{gf}$ mentioned above. Observe that from the graph, $R_{gf}=[1,\infty)$.

# Exercise 9 2021/NJC/Functions/Q9 H2 Mathematics
The functions $f$ and $g$ are defined by
$$\begin{align*}
f	:x &\mapsto\frac{2}{x^{2}+1},\,x\leq k\,\text{ and }k\,\text{ is an unknown constant}, \\
g	:x &\mapsto\frac{1}{x-1},\,x\in[2,\infty)
\end{align*}$$

It is given that $f^{-1}$ exists. State the largest value of $k$, and use this value to

- (i)	define $f^{-1}$ and
- (ii)	show that the composite function $f^{-1}g$ exists. Also, find its range.

### Solution

We see that the graph of $y=\frac{2}{x^2+1}$ on the whole real number line looks like.

As such, if we want $f^{-1}$ to exists and $x\leq k$, then the largest value of $k$ is $0$.

- (i)

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('2/(x**2+1)')

print_inverse(f)
```

$\displaystyle x=\sqrt{\frac{2 - y}{y}}$

$\displaystyle x=- \sqrt{- \frac{y - 2}{y}}$

As $x\leq k = 0$, we have that $f^{-1}(x)=-\sqrt{-\frac{x-2}{x}}$, $D_f=(-\infty,0]$.

- (ii) We see that the graph of $y=g(x)$ is given below

    We have that $R_g=(0,1] \subseteq D_{f^{-1}}=R_f=(0,2]$. As such $f^{-1}g$ exists.

    To find the range of $f^{-1}g$, we plot the graph of $y=f^{-1}g(x)$ with $D_{f^{-1}g}=D_g=[2,\infty)$.

```python
from sympy import *
from h2_math import *

# Defining the symbol for the parameters/arguments/input of the functions f,g
x = symbols('x')

# Defining functions f and g
f_inv = make_fn('-sqrt((2-x)/x)')
g = make_fn('1/(x-1)')

print('The rule of f^{-1}g')
#Compose g(f(x))
display(Math(f'f^{{-1}}g(x)={ latex(simplify(f_inv(g(x))))}'))

#print(latex(g(f(x)))) #For Markdown use
```

    The rule of f^{-1}g

$\displaystyle f^{-1}g(x)=- \sqrt{2 x - 3}$

The graph look like

As such, $R_{f^{-1}g}=(-\infty,-1]$.

# Exercise 10 2021/NJC/Functions/Q10 H2 Mathematics
The function $f$ is defined by
$$f:x\mapsto x^2-4x+1, x\leq a \text{ and }a \text{ is an unknown constant}.$$

- (i)	Find the largest value of $a$ for which $f^{-1}$ exists, and define $f^{-1}$ for this case.
- (ii)	Using the value of $a$ in (i), sketch, on the same the diagram, the graphs of $$y=f(x), y=f^{-1}(x) \text{ and } ff^{-1}(x).$$
    Hence solve $f(x)=f^{-1}(x)$, leaving your answer in exact form.

### Solution
- (i) By completing the square, we see that
    $$ \begin{align*}
    x^2- 4x+1 &= \left(x-2\right)^{2}+1-\left(2)^{2}\right) \\
              &= \left(x-2\right)^{2}-3 \\
                    &\geq -3.
    \end{align*}$$
     Since $f$ is a quadratic function with only one minimum turning point in the $\mathbb{R}$ at $x=2$, $a=2$.

    Next, we find the rule of $f^{-1}$.

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('x**2-4*x+1')

print_inverse(f)
```

$\displaystyle x=2 - \sqrt{y + 3}$

$\displaystyle x=\sqrt{y + 3} + 2$

As $x\leq 2$, $f^{-1}(x)=2-\sqrt{x+3}$ with $D_{f^{-1}}=R_f=[-3,\infty)$.

- (ii) The graphs are given below

 We see that the solution to $f\left(x\right)=f^{-1}(x)$ is also a solution $f(x)=x$ (observe the graph). Thus, we have that

```python
from sympy import *
from h2_math import *

# Denote that x is a variable in the expression
x = symbols('x')

# Defining a function with name f
f = make_fn('x**2-4*x+1')

for x_vals in solve(f(x)-x,x):
    # The next line is for use in Markdown
    display(Math(f'x={ latex(x_vals) }.'))
    #print(latex(x_vals))
```

$\displaystyle x=\frac{5}{2} - \frac{\sqrt{21}}{2}.$

$\displaystyle x=\frac{\sqrt{21}}{2} + \frac{5}{2}.$

Since $x\leq 2$, $x= \frac{5}{2} - \frac{\sqrt{21}}{2}$.

# Exercise 11 2021/NJC/Functions/Q11 H2 Mathematics
The functions $f$ and $g$ are defined by
$$\begin{align*}
f&:x\mapsto\frac{1}{2}\left(x^{3}-3x+2\right),x\in\mathbb{R}\\
g&:x\mapsto\sqrt{x+1},x\in[-1,\infty)
\end{align*}$$

- (i)	Sketch the graphs of $y = f(x)$ and $y = g(x)$, labelling clearly the coordinates of stationary points and axial intercepts, if any.
- (ii)	Determine if $f^{-1}$ exists, justifying your answer.
- (iii)	The composite function $gf$ exists if the domain of $f$ is restricted to $x\geq k$.  Given that the range of $gf$ is $[1,\infty)$, find the range of possible values of $k$.

### Solution
- (i) We see that the graphs looked like

- (ii) The horizontal line $y=1$  intersects the graph of $y=f(x)$  more than once. Hence the function $f$ is not one-one. It follows that $f^{-1}$  does not exist.

- (iii) Observe from the graph of $y = g(x)$ that in order to obtain $R_{gf}=[1,\infty)$, we require $R_f=[0,\infty)$.

    Therefore from the graph of $y = f(x)$, required answer is $-2\leq k \leq 1$.

# Exercise 12 2021/NJC/Functions/Q12 H2 Mathematics
The function $f$ is defined by
$$f:x\mapsto \frac{e^x-1}{e-1} \text{ for }x\in \mathbb{R}.$$

Sketch the graph of $y=f(x)$ and state the range of $f$.

Another function $h$ is defined by

$$h:x\mapsto\begin{cases}
\left(x-1\right)^{2}+1 & ,\,x\le1\\
1-\frac{\left|1-x\right|}{2} & ,\,1<x\leq4
\end{cases}$$

Sketch the graph of $y=h(x)$ for $x\leq 4$ and explain why the composite function $f^{-1}h$ exists. Hence find the exact value of $(f^{-1}h)^{-1}(3)$.

### Solution
The graph of $f$ looks like

From the graph, we see that $R_h=[-\frac{1}{2},\infty)\subseteq (\frac{1}{1-e},\infty)=R_f=D_{f^{-1}}$. So, $f^{-1}h$ exists.

Let the value that we are asked to find be $a$. Thus, we have
$$\begin{align*}
\left(f^{-1}h\right)^{-1}\left(3\right)	&=a \\
f^{-1}h\left(\left(f^{-1}h\right)^{-1}\left(3\right)\right)	&=f^{-1}h\left(a\right) \\
3	&=f^{-1}h\left(a\right) \\
f\left(3\right)	&=f\left(f^{-1}h\left(a\right)\right) \\
\frac{e^{3}-1}{e-1}	&=h\left(a\right) \\
1+e+e^{2}	& =h\left(a\right)
\end{align*}$$

Since $1+e+e^2 > 0$, from the graph of $h$, we conclude that the piece of the piecewise function $h$ that can give rise to this value is $h(x)=(x-1)^2+1$ when $x<1$. As such,
$$1+e+e^2 = (a-1)^2+1.$$

```python
from sympy import *
from h2_math import *

# Denote that a is a variable in the expression
a = symbols('a')

# Defining a function with name h
h = make_fn('(x-1)**2+1')

for a_vals in solve(h(a)-(1+exp(1)+exp(2)),a):
    # The next line is for use in Markdown
    display(Math(f'a={ latex(a_vals) }.'))
    # print(latex(a_vals))
```

$\displaystyle a=1 + \sqrt{1 + e} e^{\frac{1}{2}}.$

$\displaystyle a=- \sqrt{1 + e} e^{\frac{1}{2}} + 1.$

As $a<1$, we have that $(f^{-1}h)^{-1}(3)=- e^{\frac{1}{2}}\sqrt{1 + e}  + 1 = 1-\sqrt{e^2+e}.$

# Exercise 13 2018/NYJC/JC1 MYE/Q6 H2 Mathematics
The functions $f$ and $g$ are defined by
$$\begin{align*}
f	:x &\mapsto\sin x+\cos x,\,-\frac{\pi}{4}\leq x\le\frac{5\pi}{4}, \\
g	:x &\mapsto x^{2}+1,\,x>-2
\end{align*}$$

- (i)	By writing $f(x)=R\sin (x+\alpha)$ where $R>0$ and $0<\alpha <\frac{\pi}{2}$, show that $gf$ exists and find the exact range of $gf$.
- (ii)	Explain why $f$ does not have an inverse.
- (iii)	The function $h$ is such that $h(x)=f(x)$ and the domain of $h$ is of the form $(a,\frac{5\pi}{4})$. State the exact value of the minimum value of $a$ for $h^{-1}$ to exist.
- (iv)	On a single diagram, sketch the graph of $y=h(x)$ and $y=h^{-1}(x)$. Your diagram should indicate the coordinates of the endpoints and the relationship between the two graphs.

### Solution
- (i) By R-formula, $f(x)=  \sqrt{2} \sin(x+\frac{\pi}{4})$. As such, $R_f=[-\sqrt{2},\sqrt{2}]$. Since $R_f \subseteq (-2,\infty)=D_g$, the composite function $gf$ exists.
    Next, we find the rule of $gf$.

```python
from sympy import *
from h2_math import *

# Defining the symbol for the parameters/arguments/input of the functions f,g
x = symbols('x')

# We define f,g
f = make_fn('sqrt(2)*sin(x+pi/4)')
g = make_fn('x**2+1')

print('The rule of gf')
#Compose g(f(x))
display(Math(f'gf(x)={ latex(simplify(g(f(x))))}'))

#print(latex(g(f(x)))) #For Markdown use
```

    The rule of gf

$\displaystyle gf(x)=\sin{\left(2 x \right)} + 2$

The graph of $gf$ looks like

And from the graph, we see that $R_{gf}=[1,3]$.

- (ii) We see that the graph of $f$ looks like

    The line $y=0$ cuts the graph at 2 points $x=-\frac{\pi}{4}$ and $x=\frac{3\pi}{4}$. As such, $f$ isn't a one-one function and consequently, $f$ doesn't have an inverse.

- (iii) From the graph, if we are to restrict the domain of $h$ such that $h^{-1}$, the value of $a$ must be the $x$-coordinate of the maximum turning point of the graph in the domain. This happens when
    $$\begin{align*}
    f\left(x\right)	&=\sqrt{2} \\
    \sqrt{2}\sin\left(x+\frac{\pi}{4}\right)	&=\sqrt{2}
    \end{align*}$$
    And, solving gives $x=\frac{\pi}{4}$. As such, $a=\frac{\pi}{4}$.

- (iv) The graph looks like


# pill 16 -  Kernel methods

Remember the setting in which we are. The problem in machine learning could be regarded as finding $f$ such that

$$
	f^*=\underset{f\in \mathcal{H}}{\operatorname{arg\,min}}\frac{1}{n}\sum_i{\mathcal{L}(f(x_i),y_i)}+\Omega(f)
	$$

where this problem balances the **fit to the data** and the **complexity penalty**.

We have seen that following Occam's razor is a good idea in terms of controlling overfitting: "simpler" hypotheses are better. In this case $\Omega(f)$} incorporates our simplicity assumptions. And, if we fail with the complexity penalty we still have $\lambda$ (fixed by validation).

The intuition tells us that complex functions "wiggle/oscillate" more. But...
**How do we measure the complexity/oscillation of a function?**

## Enter the Hilbert space!

+ Functional analysis is the branch of mathematics concerned with the study of spaces of functions.
+ In a nutshell, functions are seen as points in a vector space (usually of infinite dimension) on a domain $\mathcal{D}$.

$\mathcal{R}$ -- The space of functions from $\mathbb{R}$ to $\mathbb{R}.$

$C^n(\mathcal{D})$ -- The space of $n$ times differentiable functions on $\mathcal{D}$.

$L^1(\mathcal{D})$ -- The space of all absolutely integrable functions on $\mathcal{D}$.
(i.e. $\{f | \int_\mathcal{D} |f(x)| dx < \infty $)

$L^2(\mathcal{D})$ -- The space of all square integrable functions on $\mathcal{D}$. (i.e. $\{f | \int_\mathcal{D} |f(x)|^2 dx < \infty $)

All these spaces form a vector space over $\mathbb{R}$. Let $f,g \in \mathcal{R}$. The basic axioms of the vector space are the well known:

$(f+g)(x)=f(x)+g(x), \forall x\in \mathbb{R}$}

$(\alpha f)( x)=\alpha f(x), \forall x \in \mathbb{R}$

**"Not very serious" definition of functional**
A functional is a function that takes one or more functions as arguments, and which returns a scalar value.

Concepts of linear algebra such as *norm* and *inner product* can be defined for functions in function spaces.

**Normed vector space.**
A normed vector space is a pair $(V, \|\cdot\|)$ where $V$ is a vector space and $\|\cdot\|$ is the associated norm, satisfying the following properties for all $u, v \in V$ :

 $\|v\| \geq 0$

 $\|u + v\| \leq \|u\|+\|v\|$ (triangle inequality)

 $\| |\alpha| v \|= |\alpha|\|v\|$ (positive homogeneity)

 $\|v\|=0 \iff v=0$

**Inner product space**

An real inner product space is a pair $(V,\langle \cdot,\cdot\rangle)$, where V is a real vector space and $\langle\cdot,\cdot\rangle$ the associated inner product, satisfying the following properties for all $u, v, \in V$:

 $\langle u,v\rangle = \langle v,u\rangle$ (symmetry)

 $\langle\alpha u,v\rangle=\alpha \langle u,v\rangle, \quad \langle u,\alpha v\rangle=\alpha \langle u,v\rangle$ (bilinearity)

$\langle u+v,w\rangle=\langle u,w\rangle+\langle v,w\rangle, \quad \langle u,v+w\rangle=\langle u,w\rangle+\langle v,w\rangle $

 $\langle u,u\rangle\geq 0 \quad \mbox{and}\quad \langle u,u\rangle=0 \iff u=0$

The inner product introduces a norm by setting $\|v\|=\langle v,v\rangle^{1/2}$ . Furthermore,
the Cauchy-Schwarz inequality holds in real inner product spaces.
$$
|\langle u,v \rangle|\leq \|u\|\|v\|
$$

A Hilbert space is an inner product space that is complete with
respect to the induced metric.

Note that a Hilbert space does not depen on functions. (i.e. $\mathbb{R}^n$ is a Hilbert space)
Intuitively, a space is complete if there are no "points missing" from it (inside or at the boundary). A space M is complete (or a Cauchy space) if every Cauchy sequence of points in M has a limit that is also in M or, alternatively, if every Cauchy sequence in M converges in M.

A linear evaluation functional over the *Hilbert space of functions* $\mathcal{H}$ is a linear functional $\mathcal{F}_x : \mathcal{H} \rightarrow \mathbb{R}$ that evaluates each function in the space at the point $x$, or

$$
\mathcal{F}_x[f]=f(x)
$$

**Informal:** It is important to understand the difference between $f$ and $f(x)$. While $f$ represents the abstraction of a function, $f(x)$ corresponds to the evaluation of a function on a point $x$

A Hilbert space $\mathcal{H}$ is a *reproducing kernel Hilbert space*
(RKHS) if the evaluation functionals are bounded, i.e. if there exists a M such that
$$|\mathcal{F}_x[f]|=|f(x)|\leq M \|f\|_{\mathcal{H}} \forall f \in \mathcal{H}
$$

**Reproducing Kernel Hilbert Space
Property**
If $\mathcal{H}$ is a RKHS, then for each $x \in X$ there exists, by the
Riesz representation theorem a function $K_x$ of $\mathcal{H}$ (called
representer) with the reproducing property
$$
\mathcal{F}_x[f]=f(x)=\langle K_x, f\rangle=\langle K(x,\cdot ), f(\cdot) \rangle
$$

Observe that the evaluation of a funtion on a given point corresponds to the inner product between the function and the a Reproducing Kernel function evaluated at that point.

**The notion of complexity now can be defined in terms of the norm of a function in the Hilbert space**

Thus,
$$
	f^*=\underset{f\in \mathcal{H}}{\operatorname{arg\,min}}\frac{1}{n}\sum_i{\mathcal{L}(f(x_i),y_i)}+\lambda\|f\|_{\mathcal{H}}^2
	$$

By the **representer's theorem** the optimal solution after solving the minimization of the former problem is given by:
	 $$
	 f^*(\cdot)=\sum_i{\alpha_i K(x_i,\cdot)}
	 $$

This is a remarkable fact because it converts the problem of looking for a generic function in an infinite-dimensional space into that of finding the optimal $\alpha_i\in \mathbb{R}$ in a finite dimensional space.

**Sketch of proof**
Let $\mathcal{S}= span \{K(x_1,\cdot),\dots,K(x_n,\cdot)\}$
(Recall that this is the space spanned/generated by all linear combinations of the elements)

Claim: $f\in \mathcal{H}, f \perp \mathcal{S}$ if and only if $f(x_i)=0, \forall i$

Proof:
$f \perp \mathcal{S}$ if and only if $f$ is orthogonal to each $K(x_i,\cdot)$
i.e. $\langle f(\cdot),K(x_i,\cdot)\rangle_{\mathcal{H}}=0$

Considering the reproducing property the proof is clear $f(x_i)=\langle K(x_i,\cdot ), f(\cdot) \rangle_{\mathcal{H}}$

We can decompose the hypotheses space $\mathcal{H} = \mathcal{S}\bigoplus \mathcal{S}^{\perp}$
$$ f\in \mathcal{H}\qquad f=f_{\mathcal{S}}+f^{\perp}
$$

Claim:
$$f(x_i)=f_\mathcal{S}(x_i)$$
This is clear because of the claim in the former slide.

Claim:
$$\|f_\mathcal{S}\|_{\mathcal{H}}\leq\|f\|_{\mathcal{H}}$$
Proof:
Consider the triangle decomposition and remember that the $f^\perp$ and $f_\mathcal{S}$ are perpendicular (the cross term goes to zero), thus,
$$\|f_{\mathcal{S}}\|^2_{\mathcal{H}}+\|f^{\perp}\|^2_\mathcal{H}=\|f\|^2_{\mathcal{H}}$$

**Remember:** We want to find $f^*$, that is the solution of the following minimization problem:

$$f^*=\underset{f\in \mathcal{H}}{\operatorname{arg\,min}}\frac{1}{n}\sum_i{\mathcal{L}(f(x_i),y_i)}+\lambda\|f\|_{\mathcal{H}}^2$$

In the RKHS the function evaluation $f(x_i)$ corresponds to $f_{\mathcal{S}}(x_i)$. Thus the loss does not change value.

$$f^*=\underset{f\in \mathcal{H}}{\operatorname{arg\,min}}\frac{1}{n}\sum_i{\mathcal{L}(f_{\mathcal{S}}(x_i),y_i)}+\lambda\|f\|_{\mathcal{H}}^2$$

What about the other term?

Recall the last claim, $\|f^*_\mathcal{S}\|_{\mathcal{H}}^2\leq\|f^*\|_{\mathcal{H}}^2$

Thus, $f$ spaned by the basis in S has smaller (or equal) norm that the optimal solution.

In summary, the minimizer $f^*$ must lie in $\mathcal{S}$

Therefore
$$ f^*(x)=\sum_i{\alpha_i K(x_i,x)}$$
Important algorithmic implications!

Let us write down the value of $\|f^*\|^2_\mathcal{H}$,
$$\|f^*\|^2_\mathcal{H}=\langle f^*,f^* \rangle_\mathcal{H}=\langle \sum_j{\alpha_j K(x_j,x)},\sum_l{\alpha_l K(x_l,x)} \rangle_\mathcal{H}
$$ by linearity
$$
 \sum_j \sum_l \alpha_l \alpha_j \langle K(x_j,x), K(x_l,x)\rangle_\mathcal{H}
$$
using the reproducing property, $f(x)=\langle K(x,\cdot ), f(\cdot) \rangle$,
$$
 \sum_j \sum_l \alpha_l \alpha_j K(x_j,x_l)
$$
or equivalently in matrix form ($K(j,l)=K(x_j,x_l)$)
$$
\|f^*\|^2_\mathcal{H}=\alpha^T K \alpha
$$

## Kernel regularized least squares

Let us take one of the measures used to evaluate the fitting goodness of the function $f(x_i)$ i.e. $\mathcal{L}(f(x_i),y_i)=(f(x_i)-y_i)^2$, then the problem to solve is

$$f^*=\underset{f\in \mathcal{H}}{\operatorname{arg\,min}}\frac{1}{n}\frac{1}{2}\sum_i(f(x_i)-y_i)^2 +\frac{1}{2}\lambda\|f\|_{\mathcal{H}}^2 $$

The solution of the problem is given by the Representer's theorem:
$f^*(x)=\sum_j{\alpha_j K(x_j,x)}$

$$
	f^*=\underset{f\in \mathcal{H}}{\operatorname{arg\,min}}\frac{1}{2}\sum_i{(f(x_i)-y_i)^2}+\frac{\lambda}{2}\|f\|_{\mathcal{H}}^2
	$$

Replacing the first term,

$$
	\alpha^*=\underset{\alpha_i \in \mathbb{R}}{\operatorname{arg\,min}}\frac{1}{2}\sum_i{(\sum_j{\alpha_j K(x_j,x_i)}-y_i)^2}+\frac{\lambda}{2}\|f\|_{\mathcal{H}}^2
	$$

replacing the norm and using the matrix form
$$
	\alpha^*=\underset{\alpha \in \mathbb{R}^N}{\operatorname{arg\,min}}\frac{1}{2} \|K\alpha -y\|^2+\frac{\lambda}{2}\alpha^TK\alpha
	$$

$$
	\alpha^*=\underset{\alpha \in \mathbb{R}^N}{\operatorname{arg\,min}}\frac{1}{2} ( K\alpha -y)^T( K\alpha -y)+\frac{\lambda}{2}\alpha^TK\alpha
	$$

Now we find the extrema by setting the gradient with respect to $\alpha$ to $0$
\begin{align*}
-K(K\alpha -y)+\lambda K\alpha = 0\\
(K+\lambda I)\alpha = y\\
\alpha = (K+\lambda I)^{-1}y
\end{align*}

Thus, the solution is just a matrix inversion $\mathcal{O}(n^3)$ and a matrix multiplication $\mathcal{O}(n^2)$ that depends on the number of samples $n$ .
The matrix $K + \lambda I$ is symmetric positive definite, so the appropriate algorithm is Cholesky factorization ( or SVD ).

RKHS needs the definition of a positive semidefinite kernel function $K$:
$$
\begin{align*}
{\bf linear}: &K(x_i,x_j)=x_i^Tx_j\\
{\bf polynomial:}&K(x_i,x_j)=(x_i^Tx_j+1)^d\\
{\bf gaussian:}&K(x_i,x_j)=exp\big( -\frac{\|x_i-x_j\|^2}{\sigma^2} \big)
\end{align*}$$

```python
#example
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.datasets import make_moons

X,y=make_moons(noise = 0.1)

y = 2*y-1
%matplotlib inline
import matplotlib.pyplot as plt

plt.scatter(X[:, 0], X[:, 1], c=y,  alpha=0.7)
```

```python
import numpy as np
from scipy.spatial import distance

def gaussian_kernel(xa,xb,gamma):
    d = distance.cdist(xa,xb,metric='euclidean')
    return np.exp(-d*gamma)
```

```python
gamma = 1.
K = gaussian_kernel(X,X,gamma)
lamb = 1.
alpha = np.dot(np.linalg.inv(K+lamb*np.eye(K.shape[0])),y)
```

```python
xx,yy = np.meshgrid(np.linspace(-2,3,200),np.linspace(-2,3,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = np.dot(gaussian_kernel(viz,X,1.),alpha)

plt.scatter(X[:, 0], X[:, 1], c=y,  alpha=0.7)
plt.gca().set_xlim([-2.,3.])
plt.gca().set_ylim([-2.,3.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-2,3,-2,3),alpha=0.3, vmin=0, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)))

plt.gcf().set_size_inches((9,9))
```

## But ... what are kernels?

The Gram matrix shows the degree of shared information among training samples.

Gram matrix interpretation

 **Bad Kernel: ** Mostly diagonal $\implies$ most points are orthogonal to each other, no clusters, no structure.

 **Good Kernel: ** The matrix has structure and show clusters.

 **Ideal Kernel: ** ${\bf K_{\mbox{ideal}} = yy^T}$

```python
idx = np.argsort(y.ravel(),axis=0)
y_sort = y[idx, np.newaxis]
plt.imshow(np.dot(y_sort,y_sort.T),cmap="gray")
```

 **EXERCISE: ** Check the Gram matrix for different values of $\gamma$. Look for a good value of $\gamma$ and check its performance.

```python
gamma = 1e-9

X_sort = X[idx,:]
K = gaussian_kernel(X_sort,X_sort,gamma)

plt.imshow(K,interpolation="nearest")

K = gaussian_kernel(X,X,gamma)
lamb = 10.
alpha = np.dot(np.linalg.inv(K+lamb*np.eye(K.shape[0])),y)
xx,yy = np.meshgrid(np.linspace(-2,3,200),np.linspace(-2,3,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = np.dot(gaussian_kernel(viz,X,1.),alpha)
plt.figure()
plt.scatter(X[:, 0], X[:, 1], c=y,  alpha=0.7)
plt.gca().set_xlim([-2.,3.])
plt.gca().set_ylim([-2.,3.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-2,3,-2,3),alpha=0.3, vmin=0, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)))

plt.gcf().set_size_inches((9,9))
```

 **EXERCISE: ** What do we expect to happen when we increase $\lambda$? And decrease $\lambda$?

```python
#your code
```

**Designing your own kernel:**

Using the properties of kernels:
Let $K_1$ and $K_2$ be valid Mercer's kernels, i.e. symmetric and positive definite matrices; and $\alpha>0$, then the following are also valid kernels

 $K(x_i,x_j)=K_1(x_i,x_j)+K_2(x_i,x_j)$

 $K(x_i,x_j)=K_1(x_i,x_j)\cdot K_2(x_i,x_j)$

 $K(x_i,x_j)=\alpha K_1(x_i,x_j)$

**More ideas for designing your kernels**

Put your favorite distance $d(x_i,x_j)$ in $K(x_i,x_j)=e^{-d(x_i,x_j)}$. Done!

 Take your favorite nonlinear transform $\Phi(x)$ and write

$$K(x_i,x_j) = \Phi(x_i)^T\Phi(x_j)$$

## The Kernel trick

Consider the Regularized Least Squares problem using a linear model. But, instead of the original input space consider a mapping $\phi({\bf x}):\mathbb{R}^d \rightarrow \mathbb{R}^{d'}$ of the data into another feature space.

The problem now is
$$
	{\bf \alpha}^*=\underset{\alpha}{\operatorname{arg\,min}}\frac{1}{2}\sum_i{(y_i - \alpha_i \phi(x_i))^2}+\frac{\lambda}{2}\|\alpha \|^2 =
	$$

$$
= \underset{\alpha}{\operatorname{arg\,min}}\frac{1}{2}({\bf y} - \Phi\alpha)^T({\bf y} - \Phi\alpha ) +\frac{\lambda}{2} \alpha^T\alpha \implies
$$

$$
\implies \Phi^T \underset{\sigma}{\underbrace{({\bf y} - \Phi\alpha)}}  + \lambda \alpha = 0 \implies
$$
$$
\implies  \alpha = \frac{1}{\lambda} \Phi^T\sigma = \frac{1}{\lambda}\sum_{t=1}^N \sigma_t \phi({\bf x}_t)
$$
Observe that the optimal alpha lies in the span of the feature vectors corresponding to the training examples.

And the optimal $\sigma$ ?
$$\sigma=({\bf y} - \Phi\alpha ) = ({\bf y} - \frac{1}{\lambda}\Phi\Phi^T\sigma)$$

$$ \sigma = (I + \frac{1}{\lambda}\Phi \Phi^T)^{-1} {\bf y} = (\Phi \Phi^T + \lambda I)^{-1} \lambda {\bf y} $$

and the prediction model becomes
$$ \hat{y} = \Phi \alpha = \frac{1}{\lambda}\Phi\Phi^T \sigma$$

Observe that this is the same solution as in the kernel method where the Gram matrix is the inner product matrix of the data set casted in the mapped space $K = \Phi\Phi^T$. Using kernels can be seen as mapping data in an arbitrarily high dimensional space defined by the Gram matrix.

**For example:** Consider the case of the polynomial quadratic kernel and assume ${\bf x}\in \mathbb{R}^2$:
$$K({\bf x},{\bf x'})=({\bf x}^T{\bf x'}+1)^2 = $$
$$ = (1+ x_1x'_1 + x_2x'_2)^2=$$
$$1 + 2x_1x'_1 + 2x_2x'_2 + (x_1x'_1)^2+ (x_2x'_2)^2+ 2x_1x'_1x_2x'_2 \implies$$
$$\Phi({\bf x}) = [1,\sqrt{2}x_1,\sqrt{2}x_2,x_1^2,x_2^2,\sqrt{2}x_1x_2]^T \in \mathbb{R}^6$$

 Using kernels can be seen as mapping data in an arbitrarily high dimensional space spanned by $\Phi(x)$.

 The use of kernels allows a compact implicit description of the space generated by $\Phi(x)$ by means of the Gram matrix $K(x,\cdot) = \Phi(x)\Phi(\cdot)$.

 Observe that $K\in \mathbb{R}^{N\times N}$.

 The dimensionality of $\Phi(x)$ for a gaussian kernel is $\infty$.

```python
from IPython.display import HTML
HTML('')

```

## An example of the kernel trick: Kernel PCA

Remember Principal component analysis. Let us now suppose that we are looking for the principal components in a transformed space $\Phi(x)$.

The covariance matrix in this case corresponds to

$$C(i,j) = \sum_j\sum_i \Phi(x_i)\Phi(x_j)$$

Remember that in PCA we assume that data is centered.

Recall the problem we are trying to optimize

$$
\begin{align*}
\text{maximize}& \;  w^T C w\\
\text{subject to} &\;  w^Tw = 1\\
\end{align*}
$$

Taking the lagrangian we obtain that we want to find the extrema of $w^T C w - \lambda w^Tw$. Differentiating and seting the derivative equal to zero we have

$$C w - \lambda w = 0$$

The solution is given by the eigenvalues and eigen vectors solving the problem

$$Cw = \lambda w$$

the direction in the transformed spaced can be regarded as

$$w = \sum\alpha_i \Phi(x)$$

Solving the former equation is equivalent to solving

$$\Phi(x)Cw = \lambda\Phi(x) w$$

and replacing $w$ by its extension, we get,

$$\Phi(x)C \Phi(x)^T\alpha = \lambda\Phi(x) \Phi(x)^T\alpha$$

which is equivalent to

$$C^2\alpha = \lambda C \alpha$$

Thus the eigendecomposition of the problem

$$C\alpha = \lambda \alpha$$

satisfy the former problem.

Given a new data point $x_{test}$ we can project in that space in the standard way, i.e. First remember that the eigenvectors are given as $\Phi(x)^T\alpha$ (they are the $w$), then

$$\hat{x}=P x_{text}=\Phi(x_{test}) \Phi(x)^T\alpha = C(x_{test},x)\alpha$$

This would have been true as long as the data in the transformed space is centered. But how can we ensure that?

The centered covariance matrix is

$$\tilde{C} = \frac{1}{N}\sum_{i=1}^N \Big(\Phi(x_i)-\frac{1}{N}\sum_k \Phi(x_k)\Big)\Big(\Phi(x_i)-\frac{1}{N}\sum_l \Phi(x_l)\Big)^T$$

$$\tilde{C} = \frac{1}{N}\sum_{i=1}^N \Big(\Phi(x_i)\Phi(x_i)^T -\frac{1}{N}\sum_k \Phi(x_k)\Phi(x_i)^T - \Phi(x_i)\frac{1}{N}\sum_l \Phi(x_l)^T +\frac{1}{N^2}\sum_k \Phi(x_k)\sum_l \Phi(x_l)^T\Big) = $$

$$\tilde{C} = C - \frac{1}{N}C_{(\rightarrow)}{\bf 1}^T + - \frac{1}{N}{\bf 1}C_{(\downarrow)} + \frac{1}{N^2}C_{(\downarrow, \rightarrow)}{\bf 1}{\bf 1}^T$$

where $(\rightarrow)$ stands for the column-wise sum, and $(\downarrow)$ stands for row-wise sum.

Let us see this in action using the moons example.

```python
from sklearn.datasets import make_moons
import numpy as np

X,y=make_moons(noise = 0.1)

y = 2*y[:,np.newaxis]-1
X1 = X[np.where(y==1)[0],:]

print (X1.shape)
%matplotlib inline
import matplotlib.pyplot as plt

plt.scatter(X1[:, 0], X1[:, 1],  alpha=0.7)
```

Now we implement kernel PCA.

```python
import numpy as np
from scipy.spatial import distance

def gaussian_kernel(xa,xb,gamma):
    d = distance.cdist(xa,xb,metric='euclidean')
    return np.exp(-d*gamma)

gamma = 10.
K = gaussian_kernel(X1,X1,gamma)

N = X1.shape[0]
K_tilde = K - np.dot(np.sum(K,axis=1),np.ones((N,1)))/N - np.dot(np.ones((1,N)),np.sum(K,axis=0))/N + np.sum(K)*np.ones((N,N))/(N*N)

vaps,veps=np.linalg.eig(K_tilde)

```

```python
K_tilde
```

    array([[-4.0726905 , -5.0726878 , -5.05552981, ..., -5.07266963,
            -5.07268794, -4.93244409],
           [-5.0726878 , -4.0726905 , -5.0725366 , ..., -5.0726905 ,
            -4.96713292, -5.07267227],
           [-5.05552981, -5.0725366 , -4.0726905 , ..., -5.07268803,
            -5.07254482, -4.95423861],
           ...,
           [-5.07266963, -5.0726905 , -5.07268803, ..., -4.0726905 ,
            -5.07269049, -5.07267989],
           [-5.07268794, -4.96713292, -5.07254482, ..., -5.07269049,
            -4.0726905 , -5.0726723 ],
           [-4.93244409, -5.07267227, -4.95423861, ..., -5.07267989,
            -5.0726723 , -4.0726905 ]])

```python
xx,yy = np.meshgrid(np.linspace(-2,3,200),np.linspace(-2,3,200))
viz=np.c_[xx.ravel(),yy.ravel()]
print(viz.shape)

K_t= gaussian_kernel(viz,X1,gamma)
print (K_t.shape)
K = gaussian_kernel(X1,X1,gamma)

N = X1.shape[0]
N_test = viz.shape[0]

K_test = K_t - np.dot(np.sum(K_t,axis=1)[:,np.newaxis],np.ones((1,N)))/N - np.dot(np.ones((N_test,1)),np.sum(K,axis=0)[np.newaxis,:])/N + np.sum(K)*np.ones((N_test,N))/(N*N)

z = np.dot(K_test,veps[:,0])

plt.scatter(X1[:, 0], X1[:, 1],   alpha=0.7)
plt.gca().set_xlim([-2.,3.])
plt.gca().set_ylim([-2.,3.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-2,3,-2,3),alpha=0.3, vmin=-10, vmax=10)
plt.contour(xx,yy,z.reshape((200,200)))

plt.gcf().set_size_inches((9,9))
```

## Moving beyond the kernel trick: Kernels in the primal

The kernel trick replaces any inner product $<x_i,x_j>$ by a kernel $<x_i,x_j> = K(x_i,x_j)$. Historically, this was the only way of using kernels until 2006, when Oliver Chapelle considered the RKHS. This allowed to use this notion outside of the inner product reformulation.

Remember that in the RKHS $f(x) = <K(x,\cdot),f(\cdot)>$ and that we can replace $f(x) = \sum_i \alpha_i k(x,x_i),$ $\quad \forall x_i \in \mathcal{D}_{train}$

Let us start by introducing a simple technique and change it to a kernel technique.

### Another example: Online kernel perceptron

A **perceptron** is a linear model that solves in a very simple way the problem

$$
\begin{align}
\text{minimize}&\quad 1\\
\text{subject to}& \quad y_i(w^Tx_i) > 0
\end{align}
$$

Remember that his problem is not part of the disciplined convex optimization family since strict inequalities are not handled.

One way for solving this problem is to use **alternating projections**. This is, finding one violated constraint and project into the feasible set.

In this sense, the algorithm can be written as follows

Select a random training sample pair $(x_i,y_i)$

If $y_i(w^Tx_i)<0$ then $w_t \leftarrow w_{t-1} + \eta y_i x_i$

In standard perceptron, $\eta = 1$.

In order to understand this algorithm let us consider what is the projection of a sample in the feasible set:
if $w_{t-1}$ does not satisfy the constraint, then we have to move in the direction of the normal of the hyperplane, $y_i x_i$ in order to get into the positive side, thus,  $w_t \leftarrow w_{t-1} + \eta y_i x_i$ is closer to satisfy the constraint.

Let us change this to accomodate kernels in the primal.

**Kernels in the primal vs dual**

Observe that $w = \sum_{i=1}^l \alpha_i {\bf x}_i$

With respect to $\alpha$ the perceptron update rule becomes:

$$\sum_{i=1}^{l+1} \alpha_i {\bf x}_i=\sum_{i=1}^l \alpha_i {\bf x}_i + y_i {\bf x}_i \implies \alpha_i \leftarrow \alpha_i +y_i$$

Thus we can rewrite the algorithm as

Select a random training sample pair $(x_i,y_i)$

If $y_i \sum_{k=1}^l \alpha_k \langle{\bf x}_k, {\bf x}_i \rangle < 0: \quad  \alpha_i \leftarrow\alpha_i +y_i$\\

And the model becomes: $f(x) =\sum_{k=1}^l \alpha_k \langle{\bf x}_k, {\bf x}\rangle$

**Generalized algorithm: **

Select a random training sample pair $(x_i,y_i)$

 If $y_i f({\bf x}_i) < 0: \quad  \alpha_i \leftarrow \alpha_i + y_i$

**Model: $f(x)$**

Remember that $f(x) = K({\bf X},x)^T\alpha$ in RKHS. Thus,

Select a random training sample pair $(x_i,y_i)$

 If $y_i \sum\limits_{j=1}^N K(x_j,x_i) \alpha_j < 0: \quad  \alpha_i \leftarrow \alpha_i +y_i$

```python
#example
import numpy as np
from sklearn.datasets import make_moons

X,y=make_moons(noise = 0.1)

y = 2*y-1
print (np.unique(y))
%matplotlib inline
import matplotlib.pyplot as plt

plt.scatter(X[:, 0], X[:, 1], c=y,  alpha=0.7)
```

```python
import numpy as np
from scipy.spatial import distance

def gaussian_kernel(xa,xb,gamma):
    d = distance.cdist(xa,xb,metric='euclidean')
    return np.exp(-d*gamma)

gamma = 3.
K = gaussian_kernel(X,X,gamma)

```

We have to be carefull with the trivial solution in this case:

```python
def train_perceptron(X,y,gamma,n_iters):
    alpha = np.random.rand(X.shape[0],1)
    for i in range(n_iters):
        idx = np.random.randint(X.shape[0])
        xi,yi = X[idx,:],y[idx]
        if yi*np.dot(gaussian_kernel(xi[np.newaxis,:],X,gamma),alpha)<0:
            alpha[idx] = alpha[idx]+yi
    return alpha

def test_perceptron(xtest,X,alpha,gamma):
    return np.dot(gaussian_kernel(xtest,X,gamma),alpha)

```

```python
gamma = 2.

alpha = train_perceptron(X,y,gamma,10000)
```

```python
xx,yy = np.meshgrid(np.linspace(-2,3,200),np.linspace(-2,3,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = test_perceptron(viz,X,alpha,gamma)

plt.scatter(X[:, 0], X[:, 1], c=y,   alpha=0.7)
plt.gca().set_xlim([-2.,3.])
plt.gca().set_ylim([-2.,3.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-2,3,-2,3),alpha=0.3, vmin=0, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)))

plt.gcf().set_size_inches((9,9))
```

### Online kernel SVM

In a similar fashion we can extend support vector machines to the online case.

In this case we are going to use subgradient methods. Remember that the support vector machine formulation is a follows,

$$
\begin{align}
\text{minimize}& \quad\|w\|_2^2+C\sum_i (1-y_i w^Tx_i)_+
\end{align}
$$

we now replace $f(x_i) = w^Tx_i$ by $f(x_i) = \alpha_i k(x_i,{\bf X})$

$$
\begin{align}
\text{minimize}& \quad\|{\bf \alpha}\|_2^2+C\sum_i (1-y_i {\bf \alpha}^T k(x_i,{\bf X}))_+
\end{align}
$$

and its gradient:

$$\nabla_{\alpha} F_{objective}= \left\{ \begin{align} 2\alpha - C y_i k(x_i,{\bf X}) & \quad \text{if} \; 1-y_i {\bf \alpha}^T k(x_i,{\bf X})\geq 0\\2\alpha &\quad \text{otherwise} \end{align}\right.
$$

```python
#example
from sklearn.datasets import make_moons

X,y=make_moons(noise = 0.1)

y = 2*y-1
print (np.unique(y))
%matplotlib inline
import matplotlib.pyplot as plt

plt.scatter(X[:, 0], X[:, 1], c=y,  alpha=0.7)
```

```python
import numpy as np
from scipy.spatial import distance

def gaussian_kernel(xa,xb,gamma):
    d = distance.cdist(xa,xb,metric='euclidean')
    return np.exp(-d*gamma)

gamma = 2.
K = gaussian_kernel(X,X,gamma)

```

```python
def train_svm(X,y,gamma,C,eta,n_iters):
    alpha = np.zeros((X.shape[0],1))
    for i in range(n_iters):
        idx = np.random.randint(X.shape[0])
        xi,yi = X[idx,:],y[idx]
        if yi*np.dot(gaussian_kernel(xi[np.newaxis,:],X,gamma),alpha)<=1.:
            g = 2*alpha - C*yi*gaussian_kernel(xi[np.newaxis,:],X,gamma).T
        else:
            g = 2*alpha
        alpha = alpha - eta*g
    return alpha

def train_svm_stream(xi,yi,sv,gamma,alpha,C,eta,n_iters):
    if yi*np.dot(gaussian_kernel(xi[np.newaxis,:],sv,gamma),alpha)<=1.:
        g = 2*alpha - C*yi*gaussian_kernel(xi[np.newaxis,:],sv,gamma).T
        alpha = alpha - eta*g
        sv.append((xi,yi))
        alpha.append(yi)
    else:
        g = 2*alpha
        alpha = alpha - eta*g
    return alpha,sv

def test_svm(xtest,X,alpha,gamma):
    return np.dot(gaussian_kernel(xtest,X,gamma),alpha)
```

```python
gamma = 7.
C = 1.
eta = 0.01

alpha = train_svm(X,y,gamma,C,eta,10000)
```

```python
xx,yy = np.meshgrid(np.linspace(-2,3,200),np.linspace(-2,3,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = test_svm(viz,X,alpha,gamma)

plt.scatter(X[:, 0], X[:, 1], c=y,   alpha=0.7)
plt.gca().set_xlim([-2.,3.])
plt.gca().set_ylim([-2.,3.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-2,3,-2,3),alpha=0.3)
plt.contour(xx,yy,z.reshape((200,200)))

plt.gcf().set_size_inches((9,9))
```

### The real online SVM

The former online version of SVM is not correct, though it is the most found formulation. The correct formulation replaces $\ell_2$ norm by the Hilbert norm as follows,

$$
\begin{align}
\text{minimize}& \quad\|f\|_\mathcal{H}^2+C\sum_i (1-y_i f(x_i))_+
\end{align}
$$

```python
def train_svm(X,y,gamma,C,eta,n_iters):
    mom = 0.
    alpha = np.zeros((X.shape[0],1))
    K = gaussian_kernel(X,X,gamma)
    for i in range(n_iters):
        idx = np.random.randint(X.shape[0])
        xi,yi = X[idx,:],y[idx]
        if yi*np.dot(gaussian_kernel(xi[np.newaxis,:],X,gamma),alpha)<=1.:
            g = 2*np.dot(K,alpha) - C*yi*gaussian_kernel(xi[np.newaxis,:],X,gamma).T
        else:
            g = 2*np.dot(K,alpha)

        mom = 0.9*mom - g
        alpha = alpha + eta*mom
    return alpha

def test_svm(xtest,X,alpha,gamma):
    return np.dot(gaussian_kernel(xtest,X,gamma),alpha)
```

```python
gamma = 1e1
C = 1e-1
eta = 0.001

alpha = train_svm(X,y,gamma,C,eta,10000)
```

```python
xx,yy = np.meshgrid(np.linspace(-2,3,200),np.linspace(-2,3,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = test_svm(viz,X,alpha,gamma)

plt.scatter(X[:, 0], X[:, 1], c=y,   alpha=0.7)
plt.gca().set_xlim([-2.,3.])
plt.gca().set_ylim([-2.,3.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-2,3,-2,3),alpha=0.3, vmin=0, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)),[0])
print(z.max(),z.min())

plt.gcf().set_size_inches((9,9))
```

 **DELIVERABLE: ** Implement a polynomial and linear kernel and check the result on a `moons` dataset and on `checkerboard`.

```python

```


# Demo - Sparse Chebyshev-Petrov-Galerkin methods for differentiation

**Mikael Mortensen** (email: `mikaem@math.uio.no`), Department of Mathematics, University of Oslo.

Date: **October 26, 2021**

**Summary.** This demo explores how to use sparse Chebyshev-Petrov-Galerkin methods for finding Chebyshev coefficients of
the derivatives of smooth functions. We will compare the methods to the more commonly adopted
recursion methods that are found in most spectral textbooks.

## Introduction

The Chebyshev polynomials of the first kind can be defined as

$$
\begin{equation}
\label{eq:chebTU} \tag{1}
    T_k(x) = \cos(k\theta),
\end{equation}
$$

where $\theta = \cos^{-1} x$, $k$ is a positive integer and $x \in [-1, 1]$.
The Chebyshev polynomials span the discrete space $S_N = \text{span}\{T_k\}_{k=0}^{N-1}$,
and a function $u(x)$ can be approximated in this space as

$$
\begin{equation}
u_N(x) = \sum_{k=0}^{N-1} \hat{u}_k T_k(x).
\label{eq:uT} \tag{2}
\end{equation}
$$

Consider the expansion of the function $u(x)=\sin(\pi x)$, created in `shenfun` as

```
from shenfun import *
import sympy as sp
x = sp.Symbol('x')
ue = sp.sin(sp.pi*x)
N = 16
SN = FunctionSpace(N, 'C')
uN = Function(SN, buffer=ue)
uN
```

The Python Function `uN` represents the expansion ([2](#eq:uT)), and the printed
values represent $\boldsymbol{\hat{u}} = \{\hat{u}_k\}_{k=0}^{N-1}$. The expansion is fairly well resolved since
the highest values of $\{\hat{u}_k\}_{k=0}^{N-1}$ approach 0.
Note that the coefficients obtained are the discrete coefficients based on interpolation at
quadrature points and they do not agree completely with the coefficients truncated from a
series $u(x) = \sum_{k=0}^{\infty} \hat{u}_k T_k$.
Under the hood the coefficients are found by projection using quadrature for the integrals:
find $u_N \in S_N$ such that

$$
(u_N-u, v)_{\omega^{-1/2}} = 0, \quad \forall v \in S_N,
$$

where $\omega = (1-x^2)$ and the scalar product notation
$(a, b)_{\omega^{-1/2}} = \sum_{j=0}^{N-1} a(x_j)b(x_j)\omega_j \approx \int_{-1}^{1} a(x)b(x) \omega(x)^{-1/2} dx$,
where $\{\omega_j\}_{j=0}^{N-1}$ are the quadrature weights. The quadrature approach ensures
that $u(x_j) = u_N(x_j)$ for all quadrature points $\{x_j\}_{j=0}^{N-1}$.
In `shenfun` we compute the following under the hood: insert for $u_N = \sum_{j=0}^{N-1} \hat{u}_j T_j$,
$u=\sin(\pi x)$ and $v = T_k$ to get

$$
\sum_{j=0}^{N-1}(T_j, T_k)_{\omega^{-1/2}} \hat{u}_j = (\sin(\pi x), T_k)_{\omega^{-1/2}},
$$

This has now become a linear algebra problem, and we recognise the matrix $d^{(0)}_{kj} = (T_j, T_k)_{\omega^{-1/2}}=c_k \pi /2 \delta_{kj}$,
where $\delta_{kj}$ is the Kronecker delta function, and $c_0=2$ and $c_k=1$ for $k>0$.
The problem is solved trivially since $d^{(0)}_{kj}$ is diagonal,
and thus

$$
\hat{u}_k = \frac{2}{c_k \pi} (\sin(\pi x), T_k)_{\omega^{-1/2}}, \quad \forall \, k\in I^N,
$$

where $I^N = \{0, 1, \ldots, N-1\}$.
We can compare this to the exact coefficients, where the integral $(\sin(\pi x), T_k)_{\omega^{-1/2}}$
is computed with high precision. To this end we could use adaptive quadrature, or symbolic integration
with sympy, but it is sufficient to use a large enough number of polynomials to fully resolve the
function. Below we find this number to be 22 and we see that the absolute error in $\hat{u}_{N-1} \approx 10^{-11}$.

```
SM = FunctionSpace(0, 'C')
uM = Function(SM, buffer=ue, abstol=1e-16, reltol=1e-16)
print(uM[:N] - uN[:N])
print(len(uM))
```

## Differentiation

Let us now consider the $n$'th derivative of $u(x)$ instead, denoted here as $u^{(n)}$,
and attempt to find $u^{(n)}$ in the space $S_N$, i.e.,

$$
u_N^{(n)} = \sum_{k=0}^{N-1} \hat{u}^{(n)}_k T_k.
$$

We note that this is not the same as $(u_N)^{(n)}$, which is

$$
(u_N)^{(n)} = \sum_{k=0}^{N-1} \hat{u}_k T^{(n)}_k,
$$

where $T^{(n)}_k$ is the $n$'th derivative of $T_k$, a polynomial of order $k-n$.
Again use projection to find $u_N^{(n)} \in S_N$ such that

$$
(u_N^{(n)}-u^{(n)}, v)_{\omega^{-1/2}} = 0, \quad \forall v \in S_N.
$$

Inserting for $u_N^{(n)}$ and $u^{(n)} = (u_N)^{(n)}$ we get

$$
\begin{equation}
    \sum_{j=0}^{N-1}(T_j, T_k)_{\omega^{-1/2}} \hat{u}_j^{(n)} = (T_j^{(n)}, T_k)_{\omega^{-1/2}} \hat{u}_j,
\label{_auto1} \tag{3}
\end{equation}
$$

$$
\begin{equation}
    \sum_{j=0}^{N-1} d^{(0)}_{kj} \hat{u}_j^{(n)}  = \sum_{j=0}^{N-1} d^{(n)}_{kj} \hat{u}_j,
\label{_auto2} \tag{4}
\end{equation}
$$

where $d^{(n)}_{kj} = (T_j^{(n)}, T_k)_{\omega^{-1/2}}$.
We compute $\hat{u}_k^{(n)}$ by inverting the diagonal $d^{(0)_{kj}}$

$$
\begin{equation}
\hat{u}_k^{(n)} = \frac{2}{c_k \pi} \sum_{j=0}^{N-1} d^{(n)}_{kj} \hat{u}_j, \quad \forall \, k \in I^{N}.
\label{eq:fhat} \tag{5}
\end{equation}
$$

The matrix $d^{(n)}_{kj}$ is upper triangular, and the last $n$ rows are zero. Since $d^{(n)}_{kj}$ is
dense the matrix vector product $\sum_{j=0}^{N-1} d^{(n)}_{kj} \hat{u}_j$ is costly
and also susceptible to roundoff errors if the structure of the matrix is
not taken advantage of. But computing it in shenfun
is straightforward, for $n=1$ and $2$:

```
uN1 = project(Dx(uN, 0, 1), SN)
uN2 = project(Dx(uN, 0, 2), SN)
uN1
```

where `uN1` $=u_N^{(1)} $ and `uN2` $=u_N^{(2)}$.
Alternatively, doing all the work that goes on under the hood

```
u = TrialFunction(SN)
v = TestFunction(SN)
D0 = inner(u, v)
D1 = inner(Dx(u, 0, 1), v)
D2 = inner(Dx(u, 0, 2), v)
w0 = Function(SN) # work array
uN1 = Function(SN)
uN2 = Function(SN)
uN1 = D0.solve(D1.matvec(uN, w0), uN1)
uN2 = D0.solve(D2.matvec(uN, w0), uN2)
uN1
```

We can look at the sparsity patterns of $(d^{(1)}_{kj})$ and $(d^{(2)}_{kj})$

```
%matplotlib inline

import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
ax1.spy(D1.diags(), markersize=2, color='r')
ax2.spy(D2.diags(), markersize=2, color='b')
```

just to see that they are upper triangular. We now ask is there a better and faster
way to get `uN1` and `uN2`? A better approach would involve only sparse
matrices, like the diagonal $(d^{(0)}_{kj})$. But how do we get there?
Most textbooks on spectral methods use recursive methods to
find $\{\hat{u}_N^{(n)}\}$. Here we will show a Galerkin approach.

It turns out that a simple change of test space/function will be sufficient.
Let us first replace the test space $S_N$ with
the Dirichlet space $D_N=\{v \in S_N | v(\pm 1) = 0\}$ using basis
functions $v=T_k-T_{k+2}$ and see
what happens. Because of the two boundary conditions,
the number of degrees of freedom is reduced by two, and we need to use a
space with $N+2$ quadrature points in order to get a square matrix system.
The method now becomes classified as Chebyshev-Petrov-Galerkin, as we
wish to find $u_N^{(1)} \in S_N$ such that

$$
(u_N^{(n)}-u^{(n)}, v)_{\omega^{-1/2}} = 0, \quad \forall v \in D_{N+2}.
$$

The implementation is straightforward

```
SD = FunctionSpace(N+2, 'C', bc=(0, 0))
v = TestFunction(SD)
D0 = inner(u, v)
D1 = inner(Dx(u, 0, 1), v)
uN11 = Function(SN)
uN11 = D0.solve(D1.matvec(uN, w0), uN11)
print(uN11-uN1)
```

and since `uN11 = uN1` we see that we have achived the same result as in
the regular projection. However, the matrices in use now look like

```
fig, (ax1, ax2) = plt.subplots(1, 2)
ax1.spy(D0.diags(), markersize=2, color='r')
ax2.spy(D1.diags(), markersize=2, color='b')
```

So $(d^{(0)}_{kj})$ now contains two nonzero diagonals, whereas $(d^{(1)}_{kj})$ is
a matrix with one single diagonal. There is no longer a `full` differentiation
matrix, and we can easily perform this projection for millions of degrees of freedom.
What about $(d^{(2)}_{kj})$? We can now use biharmonic test functions that
satisfy four boundary conditions in the space $B_N = \{v \in S_N | v(\pm 1) = v'(\pm 1) =0\}$,
and continue in a similar fashion:

```
SB = FunctionSpace(N+4, 'C', bc=(0, 0, 0, 0))
v = TestFunction(SB)
D0 = inner(u, v)
D2 = inner(Dx(u, 0, 2), v)
uN22 = Function(SN)
uN22 = D0.solve(D2.matvec(uN, w0), uN22)
print(uN22-uN2)
```

We get that `uN22 = uN2`, so the Chebyshev-Petrov-Galerkin projection works. The matrices involved are now

```
fig, (ax1, ax2) = plt.subplots(1, 2)
ax1.spy(D0.diags(), markersize=2, color='r')
ax2.spy(D2.diags(), markersize=2, color='b')
```

So there are now three nonzero diagonals in $(d^{(0)}_{kj})$, whereas the differentiation matrix
$(d^{(2)}_{kj})$ contains only one nonzero diagonal.

Why does this work? The Chebyshev polynomials and their derivatives satisfy the following orthogonality relation

$$
\begin{equation}
\label{eq:orthon} \tag{6}
  \int_{-1}^{1} T^{(n)}_j T^{(n)}_k \omega^{n-1/2} dx = \alpha^{n}_k \delta_{kj}, \quad \text{for}\, n \ge 0,
\end{equation}
$$

where

$$
\begin{equation}
    \alpha^n_k = \frac{c_{k+n}\pi k (k+n-1)!}{2(k-n)!}.
\label{_auto3} \tag{7}
\end{equation}
$$

So when we choose a test function that is $\omega^n T^{(n)}_k$, we get the diagonal differentiation matrix

$$
\begin{equation}
  d^{(n)}_{kj} = \int_{-1}^{1} T^{(n)}_j (\omega^n T^{(n)}_k) \omega^{-1/2} dx = \alpha^{n}_k \delta_{kj}, \quad \text{for}\, n \ge 0.
\label{_auto4} \tag{8}
\end{equation}
$$

The two chosen test functions above are both proportional to $\omega^n T^{(n)}_k$. More precisely,
$T_k-T_{k+2} = \frac{2}{k+1} \omega T^{(1)}_{k+1}$ and the biharmonic test function is
$T_k-\frac{2(k+2)}{k+3}T_{k+2} + \frac{k+1}{k+3}T_{k+4} = \frac{4 \omega^2T^{(2)}_{k+2}}{(k+2)(k+3)}$.
Using these very specific test functions correponds closely to using the Chebyshev
recursion formulas that are found in most textbooks. Here they are adapted to
a Chebyshev-Petrov-Galerkin method, where we simply choose test and trial functions and everything
else falls into place in a few lines of code.

## Recursion

Let us for completion show how to
find $\hat{u}_N^{(1)}$ with a recursive approach. The Chebyshev polynomials
satisfy

$$
\begin{equation}
2T_k = \frac{1}{k+1}T'_{k+1}- \frac{1}{k-1} T'_{k-1}, \quad k \ge 1.
\label{eq:Trec1} \tag{9}
\end{equation}
$$

By using this and setting $u' = \sum_{k=0}^{\infty} \hat{u}^{(1)}_k T_k = \sum_{k=0}^{\infty} \hat{u}_k T'_k$
we get

$$
\begin{equation}
2k\hat{u}_k = c_{k-1}\hat{u}^{(1)}_{k-1} - \hat{u}^{(1)}_{k+1}, \quad k \ge 1.
\label{eq:Trec2} \tag{10}
\end{equation}
$$

Using this recursion together with $\hat{u}^{(1)}_{N-1} = 0$ we get ([[canuto]](#canuto))

$$
\begin{equation}
c_k \hat{u}^{(1)}_k = \hat{u}^{(1)}_{k+2} + 2(k+1)\hat{u}_{k+1}, \quad 0 \le k \le N-2,
\label{eq:Trec3} \tag{11}
\end{equation}
$$

which is easily implemented in a (slow) for-loop

```
f1 = np.zeros(N+1)
ck = np.ones(N); ck[0] = 2
for k in range(N-2, -1, -1):
    f1[k] = (f1[k+2]+2*(k+1)*uN[k+1])/ck[k]
print(f1[:-1]-uN1)
```

which evidently is exactly the same result. It turns out that this is not strange. If we multiply
([11](#eq:Trec3)) by $\pi/2$ and rearrange a little bit we get

$$
\begin{equation}
c_k \pi/2 \hat{u}^{(1)}_k -  \pi/2 \hat{u}^{(1)}_{k+2} = (k+1)\pi \hat{u}_{k+1}
\label{_auto5} \tag{12}
\end{equation}
$$

$$
\begin{equation}
\underbrace{(c_k \pi/2 \delta_{kj} - \pi/2 \delta_{k, j-2})}_{(D^0)_{kj}} \hat{u}^{(1)}_j = \underbrace{(k+1)\pi \delta_{k,j-1}}_{(D^1)_{kj}} \hat{u}_{j}
\label{_auto6} \tag{13}
\end{equation}
$$

$$
\begin{equation}
D^0 \boldsymbol{\hat{u}} = D^1 \boldsymbol{\hat{u}}
\label{_auto7} \tag{14}
\end{equation}
$$

$$
\begin{equation}
\boldsymbol{\hat{u}^{(1)}} = (D^0)^{-1} D^1 \boldsymbol{\hat{u}}
\label{_auto8} \tag{15}
\end{equation}
$$

which is exactly how $\boldsymbol{\hat{u}^{(1)}}$ was computed above with the Chebyshev-Petrov-Galerkin approach
(`uN11 = D0.solve(D1.matvec(uN, w0), uN11)`). Not convinced? Check that the matrices
`D0` and `D1` are truly as stated above. The matrices below are printed as dictionaries with diagonal
number as key (main is 0, first upper is 1 etc) and diagonal values as values:

```
import pprint
SD = FunctionSpace(N+2, 'C', bc=(0, 0))
v = TestFunction(SD)
D0 = inner(u, v)
D1 = inner(Dx(u, 0, 1), v)
pprint.pprint(dict(D0))
pprint.pprint(dict(D1))
```

In conclusion, we have shown that we can use an efficient Chebyshev-Petrov-Galerkin approach to obtain
the discrete Chebyshev coefficients for the derivatives
of a function. By inspection, it turns out that this approach is identical to the common methods based on
well-known Chebyshev recursion formulas.

1.  **C. Canuto, M. Hussaini, A. Quarteroni and J. T. A.**.
    *Spectral Methods in Fluid Dynamics*,
    *Scientific Computation*,
    Springer,
    2012.


### Imports

```python
import numpy as np
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from IPython import display
plt.style.use('seaborn-white')
```

### Read and process data

```python
import gdown
url = 'https://drive.google.com/uc?id=1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS'
output = 'input.txt'
gdown.download(url, output, quiet=False)
```

    Downloading...
    From: https://drive.google.com/uc?id=1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS
    To: /content/input.txt
    100%|██████████| 10.3k/10.3k [00:00<00:00, 2.70MB/s]

    'input.txt'

```python
data = open('input.txt', 'r').read()
```

Process data and calculate indexes

```python
chars = list(set(data))
data_size, X_size = len(data), len(chars)
print("data has %d characters, %d unique" % (data_size, X_size))
char_to_idx = {ch:i for i,ch in enumerate(chars)}
idx_to_char = {i:ch for i,ch in enumerate(chars)}
```

    data has 10223 characters, 75 unique

### Constants and Hyperparameters

```python
H_size = 100 # Size of the hidden layer
T_steps = 40 # Number of time steps (length of the sequence) used for training
learning_rate = 1e-1 # Learning rate
weight_sd = 0.1 # Standard deviation of weights for initialization
z_size = H_size + X_size # Size of concatenate(H, X) vector
```

### Activation Functions and Derivatives

#### Sigmoid

\begin{align}
\sigma(x) &= \frac{1}{1 + e^{-x}}\\
\frac{d\sigma(x)}{dx} &= \sigma(x) \cdot (1 - \sigma(x))
\end{align}

#### Tanh

\begin{align}
\frac{d\text{tanh}(x)}{dx} &= 1 - \text{tanh}^2(x)
\end{align}

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def dsigmoid(y):
    return y * (1 - y)

def tanh(x):
    return np.tanh(x)

def dtanh(y):
    return 1 - y * y
```

### Parameters

```python
class Param:
    def __init__(self, name, value):
        self.name = name
        self.v = value #parameter value
        self.d = np.zeros_like(value) #derivative
        self.m = np.zeros_like(value) #momentum for AdaGrad
```

We use random weights with normal distribution (`0`, `weight_sd`) for $tanh$ activation function and (`0.5`, `weight_sd`) for $sigmoid$ activation function.

Biases are initialized to zeros.

```python
class Parameters:
    def __init__(self):
        self.W_f = Param('W_f',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_f = Param('b_f',
                         np.zeros((H_size, 1)))

        self.W_i = Param('W_i',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_i = Param('b_i',
                         np.zeros((H_size, 1)))

        self.W_C = Param('W_C',
                         np.random.randn(H_size, z_size) * weight_sd)
        self.b_C = Param('b_C',
                         np.zeros((H_size, 1)))

        self.W_o = Param('W_o',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_o = Param('b_o',
                         np.zeros((H_size, 1)))

        #For final layer to predict the next character
        self.W_v = Param('W_v',
                         np.random.randn(X_size, H_size) * weight_sd)
        self.b_v = Param('b_v',
                         np.zeros((X_size, 1)))

    def all(self):
        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,
               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]

parameters = Parameters()
```

### Forward pass

*Operation $z$ is the concatenation of $x$ and $h_{t-1}$*

#### Concatenation of $h_{t-1}$ and $x_t$
\begin{align}
z & = [h_{t-1}, x_t] \\
\end{align}

#### LSTM functions
\begin{align}
f_t & = \sigma(W_f \cdot z + b_f) \\
i_t & = \sigma(W_i \cdot z + b_i) \\
\bar{C}_t & = tanh(W_C \cdot z + b_C) \\
C_t & = f_t * C_{t-1} + i_t * \bar{C}_t \\
o_t & = \sigma(W_o \cdot z + b_t) \\
h_t &= o_t * tanh(C_t) \\
\end{align}

#### Logits
\begin{align}
v_t &= W_v \cdot h_t + b_v \\
\end{align}

#### Softmax
\begin{align}
\hat{y_t} &= \text{softmax}(v_t)
\end{align}

$\hat{y_t}$ is `y` in code and $y_t$ is `targets`.

```python
def forward(x, h_prev, C_prev, p = parameters):
    assert x.shape == (X_size, 1)
    assert h_prev.shape == (H_size, 1)
    assert C_prev.shape == (H_size, 1)

    z = np.row_stack((h_prev, x))
    f = sigmoid(np.dot(p.W_f.v, z) + p.b_f.v)
    i = sigmoid(np.dot(p.W_i.v, z) + p.b_i.v)
    C_bar = tanh(np.dot(p.W_C.v, z) + p.b_C.v)

    C = f * C_prev + i * C_bar
    o = sigmoid(np.dot(p.W_o.v, z) + p.b_o.v)
    h = o * tanh(C)

    v = np.dot(p.W_v.v, h) + p.b_v.v
    y = np.exp(v) / np.sum(np.exp(v)) #softmax

    return z, f, i, C_bar, C, o, h, v, y
```

### Backward pass

#### Loss

\begin{align}
L_k &= -\sum_{t=k}^T\sum_j y_{t,j} log \hat{y_{t,j}} \\
L &= L_1 \\
\end{align}

#### Gradients

\begin{align}
dv_t &= \hat{y_t} - y_t \\
dh_t &= dh'_t + W_y^T \cdot dv_t \\
do_t &= dh_t * \text{tanh}(C_t) \\
dC_t &= dC'_t + dh_t * o_t * (1 - \text{tanh}^2(C_t))\\
d\bar{C}_t &= dC_t * i_t \\
di_t &= dC_t * \bar{C}_t \\
df_t &= dC_t * C_{t-1} \\
\\
df'_t &= f_t * (1 - f_t) * df_t \\
di'_t &= i_t * (1 - i_t) * di_t \\
d\bar{C}'_{t-1} &= (1 - \bar{C}_t^2) * d\bar{C}_t \\
do'_t &= o_t * (1 - o_t) * do_t \\
dz_t &= W_f^T \cdot df'_t \\
     &+ W_i^T \cdot di_t \\
     &+ W_C^T \cdot d\bar{C}_t \\
     &+ W_o^T \cdot do_t \\
\\
[dh'_{t-1}, dx_t] &= dz_t \\
dC'_t &= f_t * dC_t
\end{align}

* $dC'_t = \frac{\partial L_{t+1}}{\partial C_t}$ and $dh'_t = \frac{\partial L_{t+1}}{\partial h_t}$
* $dC_t = \frac{\partial L}{\partial C_t} = \frac{\partial L_t}{\partial C_t}$ and $dh_t = \frac{\partial L}{\partial h_t} = \frac{\partial L_{t}}{\partial h_t}$
* All other derivatives are of $L$
* `target` is target character index $y_t$
* `dh_next` is $dh'_{t}$ (size H x 1)
* `dC_next` is $dC'_{t}$ (size H x 1)
* `C_prev` is $C_{t-1}$ (size H x 1)
* $df'_t$, $di'_t$, $d\bar{C}'_t$, and $do'_t$ are *also* assigned to `df`, `di`, `dC_bar`, and `do` in the **code**.
* *Returns* $dh_t$ and $dC_t$

#### Model parameter gradients

\begin{align}
dW_v &= dv_t \cdot h_t^T \\
db_v &= dv_t \\
\\
dW_f &= df'_t \cdot z^T \\
db_f &= df'_t \\
\\
dW_i &= di'_t \cdot z^T \\
db_i &= di'_t \\
\\
dW_C &= d\bar{C}'_t \cdot z^T \\
db_C &= d\bar{C}'_t \\
\\
dW_o &= do'_t \cdot z^T \\
db_o &= do'_t \\
\\
\end{align}

```python
def backward(target, dh_next, dC_next, C_prev,
             z, f, i, C_bar, C, o, h, v, y,
             p = parameters):

    assert z.shape == (X_size + H_size, 1)
    assert v.shape == (X_size, 1)
    assert y.shape == (X_size, 1)

    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:
        assert param.shape == (H_size, 1)

    dv = np.copy(y)
    dv[target] -= 1

    p.W_v.d += np.dot(dv, h.T)
    p.b_v.d += dv

    dh = np.dot(p.W_v.v.T, dv)
    dh += dh_next
    do = dh * tanh(C)
    do = dsigmoid(o) * do
    p.W_o.d += np.dot(do, z.T)
    p.b_o.d += do

    dC = np.copy(dC_next)
    dC += dh * o * dtanh(tanh(C))
    dC_bar = dC * i
    dC_bar = dtanh(C_bar) * dC_bar
    p.W_C.d += np.dot(dC_bar, z.T)
    p.b_C.d += dC_bar

    di = dC * C_bar
    di = dsigmoid(i) * di
    p.W_i.d += np.dot(di, z.T)
    p.b_i.d += di

    df = dC * C_prev
    df = dsigmoid(f) * df
    p.W_f.d += np.dot(df, z.T)
    p.b_f.d += df

    dz = (np.dot(p.W_f.v.T, df)
         + np.dot(p.W_i.v.T, di)
         + np.dot(p.W_C.v.T, dC_bar)
         + np.dot(p.W_o.v.T, do))
    dh_prev = dz[:H_size, :]
    dC_prev = f * dC

    return dh_prev, dC_prev
```

### Forward Backward Pass

Clear gradients before each backward pass

```python
def clear_gradients(params = parameters):
    for p in params.all():
        p.d.fill(0)
```

Clip gradients to mitigate exploding gradients

```python
def clip_gradients(params = parameters):
    for p in params.all():
        np.clip(p.d, -1, 1, out=p.d)
```

Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.

* `input`, `target` are list of integers, with character indexes.
* `h_prev` is the array of initial `h` at $h_{-1}$ (size H x 1)
* `C_prev` is the array of initial `C` at $C_{-1}$ (size H x 1)
* *Returns* loss, final $h_T$ and $C_T$

```python
def forward_backward(inputs, targets, h_prev, C_prev):
    global paramters

    # To store the values for each time step
    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}
    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}
    v_s, y_s =  {}, {}

    # Values at t - 1
    h_s[-1] = np.copy(h_prev)
    C_s[-1] = np.copy(C_prev)

    loss = 0
    # Loop through time steps
    assert len(inputs) == T_steps
    for t in range(len(inputs)):
        x_s[t] = np.zeros((X_size, 1))
        x_s[t][inputs[t]] = 1 # Input character

        (z_s[t], f_s[t], i_s[t],
        C_bar_s[t], C_s[t], o_s[t], h_s[t],
        v_s[t], y_s[t]) = \
            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass

        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t

    clear_gradients()

    dh_next = np.zeros_like(h_s[0]) #dh from the next character
    dC_next = np.zeros_like(C_s[0]) #dh from the next character

    for t in reversed(range(len(inputs))):
        # Backward pass
        dh_next, dC_next = \
            backward(target = targets[t], dh_next = dh_next,
                     dC_next = dC_next, C_prev = C_s[t-1],
                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],
                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],
                     y = y_s[t])

    clip_gradients()

    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]
```

### Sample the next character

```python
def sample(h_prev, C_prev, first_char_idx, sentence_length):
    x = np.zeros((X_size, 1))
    x[first_char_idx] = 1

    h = h_prev
    C = C_prev

    indexes = []

    for t in range(sentence_length):
        _, _, _, _, C, _, h, _, p = forward(x, h, C)
        idx = np.random.choice(range(X_size), p=p.ravel())
        x = np.zeros((X_size, 1))
        x[idx] = 1
        indexes.append(idx)

    return indexes
```

## Training (Adagrad)

Update the graph and display a sample output

```python
def update_status(inputs, h_prev, C_prev):
    #initialized later
    global plot_iter, plot_loss
    global smooth_loss

    # Get predictions for 200 letters with current model

    sample_idx = sample(h_prev, C_prev, inputs[0], 200)
    txt = ''.join(idx_to_char[idx] for idx in sample_idx)

    # Clear and plot
    plt.plot(plot_iter, plot_loss)
    display.clear_output(wait=True)
    plt.show()

    #Print prediction and loss
    print("----\n %s \n----" % (txt, ))
    print("iter %d, loss %f" % (iteration, smooth_loss))
```

Update parameters

\begin{align}
\theta_i &= \theta_i - \eta\frac{d\theta_i}{\sum dw_{\tau}^2} \\
d\theta_i &= \frac{\partial L}{\partial \theta_i}
\end{align}

```python
def update_paramters(params = parameters):
    for p in params.all():
        p.m += p.d * p.d # Calculate sum of gradients
        #print(learning_rate * dparam)
        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))
```

To delay the keyboard interrupt to prevent the training
from stopping in the middle of an iteration

```python
import signal

class DelayedKeyboardInterrupt(object):
    def __enter__(self):
        self.signal_received = False
        self.old_handler = signal.signal(signal.SIGINT, self.handler)

    def handler(self, sig, frame):
        self.signal_received = (sig, frame)
        print('SIGINT received. Delaying KeyboardInterrupt.')

    def __exit__(self, type, value, traceback):
        signal.signal(signal.SIGINT, self.old_handler)
        if self.signal_received:
            self.old_handler(*self.signal_received)
```

```python
# Exponential average of loss
# Initialize to a error of a random model
smooth_loss = -np.log(1.0 / X_size) * T_steps

iteration, pointer = 0, 0

# For the graph
plot_iter = np.zeros((0))
plot_loss = np.zeros((0))
```

Training loop

```python
while True:
    try:
        with DelayedKeyboardInterrupt():
            # Reset
            if pointer + T_steps >= len(data) or iteration == 0:
                g_h_prev = np.zeros((H_size, 1))
                g_C_prev = np.zeros((H_size, 1))
                pointer = 0

            inputs = ([char_to_idx[ch]
                       for ch in data[pointer: pointer + T_steps]])
            targets = ([char_to_idx[ch]
                        for ch in data[pointer + 1: pointer + T_steps + 1]])

            loss, g_h_prev, g_C_prev = \
                forward_backward(inputs, targets, g_h_prev, g_C_prev)
            smooth_loss = smooth_loss * 0.999 + loss * 0.001

            # Print every hundred steps
            if iteration % 100 == 0:
                update_status(inputs, g_h_prev, g_C_prev)

            update_paramters()

            plot_iter = np.append(plot_iter, [iteration])
            plot_loss = np.append(plot_loss, [loss])

            pointer += T_steps
            iteration += 1
    except KeyboardInterrupt:
        update_status(inputs, g_h_prev, g_C_prev)
        break
```

### Gradient Check

Approximate the numerical gradients by changing parameters and running the model. Check if the approximated gradients are equal to the computed analytical gradients (by backpropagation).

Try this on `num_checks` individual paramters picked randomly for each weight matrix and bias vector.

```python
from random import uniform
```

Calculate numerical gradient

```python
def calc_numerical_gradient(param, idx, delta, inputs, target, h_prev, C_prev):
    old_val = param.v.flat[idx]

    # evaluate loss at [x + delta] and [x - delta]
    param.v.flat[idx] = old_val + delta
    loss_plus_delta, _, _ = forward_backward(inputs, targets,
                                             h_prev, C_prev)
    param.v.flat[idx] = old_val - delta
    loss_mins_delta, _, _ = forward_backward(inputs, targets,
                                             h_prev, C_prev)

    param.v.flat[idx] = old_val #reset

    grad_numerical = (loss_plus_delta - loss_mins_delta) / (2 * delta)
    # Clip numerical error because analytical gradient is clipped
    [grad_numerical] = np.clip([grad_numerical], -1, 1)

    return grad_numerical
```

Check gradient of each paramter matrix/vector at `num_checks` individual values

```python
def gradient_check(num_checks, delta, inputs, target, h_prev, C_prev):
    global parameters

    # To calculate computed gradients
    _, _, _ =  forward_backward(inputs, targets, h_prev, C_prev)

    for param in parameters.all():
        #Make a copy because this will get modified
        d_copy = np.copy(param.d)

        # Test num_checks times
        for i in range(num_checks):
            # Pick a random index
            rnd_idx = int(uniform(0, param.v.size))

            grad_numerical = calc_numerical_gradient(param,
                                                     rnd_idx,
                                                     delta,
                                                     inputs,
                                                     target,
                                                     h_prev, C_prev)
            grad_analytical = d_copy.flat[rnd_idx]

            err_sum = abs(grad_numerical + grad_analytical) + 1e-09
            rel_error = abs(grad_analytical - grad_numerical) / err_sum

            # If relative error is greater than 1e-06
            if rel_error > 1e-06:
                print('%s (%e, %e) => %e'
                      % (param.name, grad_numerical, grad_analytical, rel_error))
```

```python
gradient_check(10, 1e-5, inputs, targets, g_h_prev, g_C_prev)
```

    W_f (-1.208988e-05, -1.209118e-05) => 5.343509e-05
    W_f (4.853007e-06, 4.852757e-06) => 2.572533e-05
    W_f (9.209842e-05, 9.209955e-05) => 6.124582e-06
    W_f (-1.348610e-06, -1.348691e-06) => 3.005958e-05
    W_f (3.129259e-04, 3.129252e-04) => 1.074261e-06
    W_f (5.471179e-07, 5.475204e-07) => 3.674026e-04
    W_i (-5.409362e-06, -5.410217e-06) => 7.898553e-05
    W_i (8.868994e-06, 8.868956e-06) => 2.151003e-06
    W_i (2.248292e-04, 2.248300e-04) => 1.732387e-06
    W_i (7.403003e-05, 7.402941e-05) => 4.180942e-06
    W_C (-8.065157e-05, -8.065066e-05) => 5.693871e-06
    W_o (-2.047074e-06, -2.047184e-06) => 2.692733e-05
    W_v (-6.548362e-06, -6.547997e-06) => 2.788195e-05
    W_v (-1.100631e-06, -1.100822e-06) => 8.695271e-05
    W_v (-8.412826e-07, -8.409354e-07) => 2.062425e-04
    W_v (9.080736e-07, 9.074695e-07) => 3.325817e-04
    W_v (1.404814e-05, 1.404835e-05) => 7.604074e-06
    W_v (1.724487e-06, 1.724347e-06) => 4.062510e-05
    b_f (2.074707e-04, 2.074689e-04) => 4.230728e-06
    b_f (-1.285301e-05, -1.285210e-05) => 3.532601e-05
    b_f (-2.616147e-05, -2.616083e-05) => 1.230837e-05
    b_C (-5.878064e-04, -5.877075e-04) => 8.415461e-05
    b_C (-1.672656e-01, -1.672662e-01) => 1.868790e-06
    b_C (9.727998e-04, 9.727951e-04) => 2.420814e-06
    b_o (-1.124711e-03, -1.124715e-03) => 1.671310e-06
    b_o (-1.621325e-02, -1.621328e-02) => 1.142338e-06
    b_o (-1.672581e-02, -1.672594e-02) => 4.055322e-06


# 扩散映射(DiffusionMaps)

## 符号定义

|符号|概念|
|:--:|:--:|
|$\pmb{x}$|样本点|
|$X$|样本集合|
|$N$|样本总数|
|$G$|有限图|
|$S$|有限图元素集合|
|$W$|权重矩阵|
|$D$|度矩阵|
|$P$|转移矩阵|
|$M$|距离矩阵|
|$d$|距离|
|$m$|降维后维度|

## 概念
ISOMAP通过替换欧氏距离为最短路径距离实现了比较好的降维效果，但是ISOMAP有一个非常明显的缺陷：对噪声较敏感。噪声很有可能改变两个点之间的最短路径以至于影响相当多样本对的距离度量，从而得到错误的降维结果。

为了抵抗噪声的影响，一个非常简单的思路就是取多条路径的平均值（若两个点由多条短路径相连则可以断定这两个是相近的）。Diffusion Maps就有着类似的想法，不同的是，Diffusion Maps是以图的形式来审视所有数据。Diffusion Maps定义相应的距离度量用于衡量图中节点之间的连接程度，节点间相互连接的总量则作为两节点的相似程度。Difuusion Maps在随机游走以及扩散过程的基础上定义扩散距离，并找到了扩散距离与低维欧氏距离的联系，从而实现将高维空间的数据嵌入到低维空间。

Diffusion Maps主要有如下工作：

1. 定义一个有限图$G=(S, E)$，其中$S$为所有数据点构成的元素集合，$E$为所有边的集合
2. 在有限图的基础上，定义相应的转移矩阵$P$，并进行随机游走。
3. 寻找当随机游走步数$t\rightarrow \infty$时的平稳分布$\mu$
4. 定义扩散距离并实现数据的低维空间嵌入

## 推导

* **权重矩阵**

首先需要定义一个权重矩阵，权重矩阵的定义方式可以由实际情况决定，仅需要满足如下的条件：

1. 对称性
$$
\begin{equation}
    W = W^T
\end{equation}
$$
2. 元素非负
$$
\begin{equation}
    w(i, j) \geq 0
\end{equation}
$$

在下述推导中采用高斯核进行定义，即

$$
\begin{equation}
    \begin{split}
    W_{ij}
    &= w(i, j) \\
    &= \mathcal{k}(\pmb{x_i}, \pmb{x_j}) \\
    &= \exp(-\gamma||\pmb{x_i-\pmb{x_j}}||^2)
    \end{split}
\end{equation}
$$

为了调整节点空间位置的影响，对权重矩阵进行归一化。

为了方便计算，首先引入度矩阵$D$，度矩阵为对角阵

$$
\begin{equation}
    D_{ii} = \sum_{j=1}^NW(\pmb{x_i, x_j})
\end{equation}
$$

这里采用对称归一化，即$D^{-\frac{1}{2}}WD^{-\frac{1}{2}}$，这样处理后权重矩阵依然为对称矩阵。

* **随机游走**

上述使用核函数获得的两两点之间的相似性度量实际上仅描述了局部信息。为了进一步获得全局上的信息，Diffusion Maps引入了图上的随机游走。从某一个特定的节点$\pmb{x_i}$出发，经过一个时间步后，相较于较远的节点，显然更容易转移到较近的点。因此上述由高斯核函数定义的权重矩阵能够用于构造转移矩阵。

$$
\begin{equation}
    P = D^{-1}W
\end{equation}
$$

上述仅为一个时间步的转移，Diffusion Maps引入了多个时间步的转移。设$P^t$表示t个时间步的转移矩阵，其元素$P^t_{ij}$表示经过t个时间步从节点$\pmb{x_i}$转移到节点$\pmb{x_j}$的概率。显然多个时间步的转移能够体现节点间的内在联系。实际上，当$t\rightarrow\infty$时，会得到一个平稳分布。

$$
\begin{equation}
    \mu(\pmb{x_i}) = \frac{D_{ii}}{\sum_{j=1}^ND_{jj}}
\end{equation}
$$

详细证明可以参考参考资料[3]

* **扩散距离**

在随机游走的基础上，若两个节点的t个时间步的转移矩阵相似，我们可以认为这两个节点的相似度比较高，反之则可以认为这两个节点的相似度低，Diffusion Maps正是基于此定义了一个相当有效的节点相似度度量方法。其定义如下

$$
\begin{equation}
    d^2_t(\pmb{x_i}, \pmb{x_j}) = \sum_{z=1}^N\frac{(P^t_{iz}-P^t_{jz})^2}{\mu(\pmb{x_z})}
\end{equation}
$$

上式是一个带权重的$l^2$距离，节点分布密集的区域$\mu(\pmb{x_z})$较大，此时权重较小，而节点分布稀疏的区域$\mu(\pmb{x_z})$较小，此时权重较大。

* **转移矩阵的特征分解**

这里直接给结论，详细推导见参考资料[4]

对于$P^t$有如下分解

$$
\begin{equation}
    P^t = \sum_{i=1}^N\lambda_i^t\pmb{\varphi_i}\pmb{\psi_i}
\end{equation}
$$

其中$\lambda$为特征值，$\varphi$为左特征向量，$\psi$为右特征向量。左特征向量关于$\frac{1}{\mu}$归一化，右特征向量关于$\mu$归一化，并按照特征值的大小降序排列，可以得到

$$
\begin{equation}
    P^t_{ij} = \sum_{z=1}^N\lambda_z^t\varphi_z(i)\psi_z(j)
\end{equation}
$$

其中$P^t_{ij}$为矩阵$P^t$中位置为$ij$的值，$\varphi_z(i)$为第z个左特征向量的第i个位置的值

将式9带入式7可以得到

$$
\begin{equation}
    \begin{split}
    d^2_t(\pmb{x_i}, \pmb{x_j})
    &= \sum_{z=1}^N\frac{(P^t_{iz}-P^t_{jz})^2}{\mu(\pmb{x_z})} \\
    &= \sum_{z=1}^N\frac{(\sum_{l=1}^N\lambda_l^t\varphi_l(i)\psi_l(z)-\sum_{l=1}^N\lambda_l^t\varphi_l(j)\psi_l(z))^2}{\mu(\pmb{x_z})} \\
    &= \sum_{z=1}^N\frac{\sum_{l=1}^N\lambda_l^{2t}\psi_l^2(z)(\varphi_l(i)-\varphi_l(j))^2}{\mu(\pmb{x_z})} \\
    &= \sum_{l=1}^N\sum_{z=1}^N\frac{\psi_l^2(z)}{\mu(\pmb{x_z})}\lambda_l^{2t}(\varphi_l(i)-\varphi_l(j))^2 \\
    &= \sum_{l=1}^N\lambda_l^{2t}(\varphi_l(i)-\varphi_l(j))^2 \\
    &= \sum_{l=1}^N(\lambda_l^{t}\varphi_l(i) - \lambda_l^{t}\varphi_l(j))^2
    \end{split}
\end{equation}
$$

考虑到$\pmb{\varphi_1} \equiv \pmb{1}$因此忽略第一项，最终得到

$$
\begin{equation}
    d^2_t(\pmb{x_i}, \pmb{x_j}) = \sum_{l=2}^N(\lambda_l^{t}\varphi_l(i) - \lambda_l^{t}\varphi_l(j))^2
\end{equation}
$$

考虑到特征值的衰减，取前几项即可达到一定的精度，因此上述定义的距离可以由前$m$项近似表示

$$
\begin{equation}
    \begin{split}
    d^2_t(\pmb{x_i}, \pmb{x_j})
    &= \sum_{l=2}^N(\lambda_l^{t}\varphi_l(i) - \lambda_l^{t}\varphi_l(j))^2 \\
    &\simeq \sum_{l=2}^{m+1}(\lambda_l^{t}\varphi_l(i) - \lambda_l^{t}\varphi_l(j))^2
    \end{split}
\end{equation}
$$

上述定义的距离显然可以视为在$\mathcal{R^m}$上的欧式距离。在式-12的基础上定义如下映射

$$
\begin{equation}
    \Phi_t: \pmb{x_i} \longmapsto [\lambda_2^{t}\varphi_2(i), \lambda_3^{t}\varphi_3(i), \cdots, \lambda_{m+1}^{t}\varphi_{m+1}(i)]^T
\end{equation}
$$

可以得到
$$
\begin{equation}
    \begin{split}
    d^2_t(\pmb{x_i}, \pmb{x_j})
    &= \sum_{l=2}^N(\lambda_l^{t}\varphi_l(i) - \lambda_l^{t}\varphi_l(j))^2 \\
    &\simeq ||\Phi_t(\pmb{x_i}) - \Phi_t(\pmb{x_j})||^2
    \end{split}
\end{equation}
$$

式-14指出在原空间定义的扩散距离可以由低维空间$\mathcal{R^m}$中的欧式距离近似表示。这一低维表示由随机游走时间$t$以及转移矩阵决定。

## 算法步骤

## 参考资料

[1] https://en.wikipedia.org/wiki/Isomap

[2] https://www.cnblogs.com/EIPsilly/p/15732378.html

[3] https://www.cs.yale.edu/homes/spielman/561/lect10-18.pdf

[4] https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-262-discrete-stochastic-processes-spring-2011/video-lectures/lecture-8-markov-eigenvalues-and-eigenvectors/MIT6_262S11_lec08.pdf