# Exercises

## Problem 1: Linear algebra refresher

Look up the topic of *vector space* in your favorite linear algebra
book or search for the term at Wikipedia.

**a)**
Prove that vectors in the plane spanned by the vector $(a,b)$  form a vector space
by showing that all the axioms of a vector space are satisfied.

**Solution.**
The axioms of a vector space go as follows (see
[Wikipedia](https://en.wikipedia.org/wiki/Vector_space), but
we use slightly different notation below):

1. The sum of $u$ and $v$, denoted by $u + v$, is in V.

2. $u + v = v + u$.

3. $(u + v) + w = u + (v + w)$.

4. There is a *zero* vector $0$ in V such that $u + 0 = u$.

5. For each $u$ in V, there is a vector $-u$ in V such that $u + (-u) = 0$.

6. The scalar multiple of $u$ by $\gamma$, denoted by $\gamma u$, is in V.

7. $\gamma (u + v) = \gamma u + \gamma v$.

8. $(\gamma + \delta)u = \gamma u + \delta u$ for scalar $\gamma$ and $\delta$.

9. $\gamma(\delta u) = \gamma\delta u$.

10. $1u = u$.

We must show that each axiom is fulfilled by planar vectors and their
mathematical rules. Let $u=(a,b)$ and $v=(c,d)$.

Axiom 1:

$$
u+v = (a,b)+(c,d)=(a+c, b+d),
$$

is a planar vector and therefore in $V$.

Axiom 2:

$$
u+v = (a,b)+(c,d)=(a+c, b+d) = (c+a, d+b) = v + u{\thinspace .}
$$

Axiom 3:

$$
(u+v) + w = ((a,b) + (c,d)) + (e,f) = (a,b) + ((c,d) + (e,f)) = u + (v+w){\thinspace .}
$$

Axiom 4: The $(0,0)$ vector is the 0 element,

$$
u + 0 = (a,b) + (0,0) = (a+0, b+0) = (a,b) = u{\thinspace .}
$$

Axiom 5: Let $-u$ element is $(-a,-b)$, so

$$
u + (-u) = (a,b) + (-a,-b) = (0,0) = 0{\thinspace .}
$$

Axiom 6:

$$
\gamma u = \gamma\cdot(a,b)= (\gamma a, \gamma b),
$$

is also a planar vector and therefore in $V$.

Axiom 7:

$$
\gamma (u + v) = \gamma \cdot((a,b) + (c,d) = \gamma (a,b) + \gamma (c,d){\thinspace .}
$$

Axiom 8:

$$
(\gamma + \delta)u = (\gamma + \delta)(a,b) = \gamma (a,b) + \delta (a,b) = \gamma u + \delta u{\thinspace .}
$$

Axiom 9:

$$
\gamma (\delta u) = \gamma (\delta\cdot (a,b)) = \gamma (\delta a, \delta b) = (\gamma\delta a, \gamma\delta b) = \gamma\delta u{\thinspace .}
$$

Axiom 10:

$$
1u = 1(a,b) = (1\cdot a, 1\cdot b) = (a,b) = u{\thinspace .}
$$

**b)**
Prove that all linear functions of the form $ax+b$ constitute a vector space,
$a,b\in\mathbb{R}$.

**Solution.**
Let $u=ax+b$ and $v=cx+d$. We verify each axiom.

Axiom 1:

$$
u+v = ax + b + cx + d = (a+c)x + (b + d),
$$

is also a linear function and therefore in $V$.

Axiom 2:

$$
u+v = ax+b + cx+ d = cx+d + ax+ b = v + u{\thinspace .}
$$

Axiom 3:

$$
(u+v) + w = (ax+b + cx+ d) + ex + f = ax+b + cx+ d + ex + f
ax+b + (cx+ d + ex + f) = u + (v+w){\thinspace .}
$$

Axiom 4: The $0x+0=0$ function is the 0 element,

$$
u + 0 = ax + b + 0 = ax + b = u{\thinspace .}
$$

Axiom 5: Let $-u$ element is $-ax-b$, so

$$
u + (-u) = ax + b + (-ax - b) = 0{\thinspace .}
$$

Axiom 6:

$$
\gamma u = \gamma(ax +b)= \gamma ax + \gamma b,
$$

is also a linear function and therefore in $V$.

Axiom 7:

$$
\gamma (u + v) = \gamma (ax+b + cx+ d) = \gamma ax + \gamma b + \gamma cx + \gamma d = \gamma(ax + b) + \gamma(cd + d) = \gamma u + \gamma v{\thinspace .}
$$

Axiom 8:

$$
(\gamma + \delta)u = (\gamma + \delta)(ax + b) = \gamma (ax+b) + \delta (ax+b) = \gamma u + \delta u{\thinspace .}
$$

Axiom 9:

$$
\gamma (\delta u) = \gamma (\delta(ax+b))) = \gamma\delta u{\thinspace .}
$$

Axiom 10:

$$
1u = 1(ax+b) = ax+b = u{\thinspace .}
$$

**c)**
Show that all quadratic functions of the form $1 + ax^2 + bx$ *do not*
constitute a vector space.

**Solution.**
Let $u=ax^2+bx + 1$ and $v=cx^2 + dx + 1$. We try to verify each axiom.

Axiom 1:

$$
u+v = ax^2 + bx + 1 + cx^2 + dx + 1 = (a+c)x^2 + (b+d)x + 2,
$$

but this quadratic function is not in $V$ because the constant term is 2
and not 1. Consequently, quadratic functions of the particular
form $1 + ax^2 + bx$ do not constitute a vector space.
The more general form  $ax^2 + bx +c$ for arbitrary constants $a$, $b$, and
$c$ makes functions that span a function space.

**d)**
Check out
the topic of *inner product spaces*. Suggest a possible inner product
for the space of all linear functions of the form $ax+b$, $a,b\in\mathbb{R}$,
defined on some interval $\Omega=[A,B]$.
Show that this particular inner product satisfies the
general requirements of an inner product in a vector space.

**Solution.**
According to [Wikipedia](https://en.wikipedia.org/wiki/Inner_product_space),
an inner product space, with an inner product $(u,v)$, has three axioms
(we drop the possibility of complex numbers and assume everything is real):

1. Symmetry: $(u,v)$ = $(v,u)$

2. Linearity in the first argument: $\gamma u, v) = \gamma (u,v)$

3. Positive-definiteness: $(u,u)\geq 0$ and $(u,u)=0$ implies $u=0$

A possible inner product for linear functions on a domain $\Omega$ is

$$
(u,v) = \int_A^B uv{\, \mathrm{d}x} = \int_A^B (ax+b)(cx+d){\, \mathrm{d}x}{\thinspace .}
$$

Symmetry is obvious since $uv=vu$:

$$
(u,v) = \int_A^B uv{\, \mathrm{d}x} = \int_A^B vu d = (v,u){\thinspace .}
$$

Linearity in the first argument:

$$
(\gamma u,v) = \gamma\int_A^B(\gamma u)v{\, \mathrm{d}x} = \gamma\int_A^B uv{\, \mathrm{d}x} = \gamma (u,v){\thinspace .}
$$

Positive-definiteness:

$$
(u,u) = \int_A^B(ax+b)^2{\, \mathrm{d}x}\geq 0,
$$

since the integral of a function $f(x)\geq$ must be greater than or equal to
zero, and in particular,

$$
(u,u)=0\quad\Rightarrow\quad \int_A^B(ax+b)^2{\, \mathrm{d}x} = 0,
$$

implies $ax+b=0$.

Filename: `linalg1`.

## Problem 2: Approximate a three-dimensional vector in a plane

Given $\boldsymbol{f} = (1,1,1)$ in $\mathbb{R}^3$, find the best approximation vector
$\boldsymbol{u}$ in the plane spanned by the unit vectors $(1,0)$ and $(0,1)$.
Repeat the calculations using the vectors $(2,1)$ and $(1,2)$.

**Solution.**
We have the vector $\pmb{f} = (1,1,1)$.
Our aim is to approximate this with a vector in the vector-space spanned by the unit vectors $\pmb{\phi_0}$ and $\pmb{\phi_1}$. We seek a solution $u = c_0 \phi_0 + c_1 \phi_1$. To do this we use the least-square-method and solve the equation $\pmb{A c} = \pmb{b}$. Or written out:

$$
\left[\begin{array}{cc}
A_{0,0} & A_{0,1} \\A_{1,0} & A_{1,1}
\end{array}\right]
\left[ \begin{array}{c} c_0 \\ c_1 \end{array} \right]
=
\left[ \begin{array}{c} b_0 \\ b_1 \end{array} \right]
$$

where $A_{i,j} = ( \pmb{\phi_i}, \pmb{\phi_j} )$ and $b_i = ( \pmb{\phi_i}, \pmb{f} )$.

We start with $\pmb{\phi_0} = (0,1)$, $\pmb{\phi_1} = (1,0)$.
Calculations give

$$
\begin{align*}
A_{0,0} &= ([1,0],[1,0]) = 1,\\
A_{0,1} &= ([1,0],[0,1]) = 0,
A_{1,0} &= ([0,1],[1,0]) = 0,
A_{1,1} &= ([0,1],[0,1]) = 1,
b_0 &= ([1,0],[1,1,1]) = 1,
b_0 &= ([0,1],[1,1,1]) = 1.
\end{align*}
$$

The result becomes $ c_0 = c_1 =1$, and hence

$$
u = 1\cdot\pmb{\phi_0}  + 1\cdot\pmb{\phi_1}.
$$

Then we proceed with
$\pmb{\phi_0} = (2,1)$, $\pmb{\phi_1} = (1,2)$.
Calculations give

$$
\begin{align*}
A_{0,0} &= ([2,1],[2,1]) = 2\cdot 2 + 1\cdot 1 = 5,
A_{0,1} &= ([2,1],[1,2]) = 2\cdot 1 + 1\cdot 2 = 4,
A_{1,0} &= ([1,2],[2,1]) = 1\cdot 2 + 2\cdot 1 = 4,
A_{1,1} &= ([1,2],[1,2]) = 1\cdot 1 + 2\cdot 2 = 5,
b_0 &= ([2,1],[1,1,1]) = 2\cdot 1 + 1\cdot 1 = 3,
b_0 &= ([1,2],[1,1,1]) = 1\cdot 1 + 2\cdot 1 = 3.
\end{align*}
$$

Solving for the $c_i$ values results in $ c_0 = c_1 =\frac{1}{3}$ and
hence

$$
u = \frac{1}{3}\cdot \pmb{\phi_0}  + \frac{1}{3}\cdot \pmb{\phi_1} .
$$

Filename: `vec111_approx`.

## Problem 3: Approximate a parabola by a sine

Given the function $f(x)=1 + 2x(1-x)$ on $\Omega=[0,1]$, we want to
find an approximation in the function space

$$
V = \hbox{span}\{1, \sin(\pi x)\}{\thinspace .}
$$

**a)**
Sketch or plot $f(x)$. Think intuitively how an expansion in terms
of the basis functions of $V$, ${\psi}_0(x)=1$, ${\psi}_1(x)=\sin(\pi x)$,
can be constructed to yield a best approximation to $f$. Or
phrased differently, see if you can guess the coefficients $c_0$ and $c_1$
in the expansion

$$
u(x) = c_0{\psi}_0 + c_1{\psi}_1 = c_0 + c_1\sin (\pi x){\thinspace .}
$$

Compute the $L^2$ error $||f-u||_{L^2}=(\int_0^1(f-u)^2{\, \mathrm{d}x})^{1/2}$.

**Hint.**
If you make a mesh function `e` of the error
on some mesh with uniformly spaced coordinates in
the array `xc`, the integral can be approximated as `np.sqrt(dx*np.sum(e**2))`,
where `dx=xc[0]-xc[1]` is the mesh spacing and `np` is an alias for the
`numpy` module in Python.

**Solution.**
The function ${\psi}_0=1$ can be used to "take care of" of the
constant 1 in $f$, while ${\psi}_1=\sin(\pi x)$ can approximate
the parabola. The maximum of $f$ is 3/2, so we should use
$u(x) = 1\cdot{\psi}_0 + \frac{1}{2}{\psi}_1$.

Plotting and the computation of the $L^2$ error can be done by

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

xc = np.linspace(0, 1, 101)  # x coordinates for plotting

def f(x):
    return 1 + 2*x*(1-x)

import sympy as sym
x = sym.symbols('x')
psi_0 = 1
psi_1 = sym.sin(sym.pi*x)

half = sym.Rational(1,2)
u = 1*psi_0 + half*psi_1

# How to combine c_0*psi_0 + c_1*psi_1 to match f?
# Intuitively, c_0=c_1=1...
# Turn u to function so we can plot and compute with it
u = sym.lambdify([x], u, modules='numpy')

print('L2 error of intuitive approximation:', end=' ')
e = f(xc) - u(xc)
dx = xc[1] - xc[0]
print(np.sqrt(dx*np.sum(e**2)))

plt.plot(xc, f(xc), 'r--')
plt.plot(xc, u(xc), 'b-')
plt.legend(['exact', 'intuitive approximation'])
```

The $L^2$ error becomes $0.0179$.

**b)**
Perform the hand calculations for a least squares approximation.

**Solution.**
The least squares method ends up with a linear system where the
coefficient matrix has entries $A_{i,j}=\int_0^1{\psi}_i{\psi}_j{\, \mathrm{d}x}$
and the right-hand side has entries $b_i=\int_0^1{\psi}_i{\, \mathrm{d}x}$.
We can use `sympy` do carry out the integrals:

```python
# Do the calculations in the least squares or project method
A = sym.zeros(2, 2)
b = sym.zeros(2, 1)
A[0,0] = sym.integrate(psi_0*psi_0, (x, 0, 1))
A[0,1] = sym.integrate(psi_0*psi_1, (x, 0, 1))
A[1,0] = A[0,1]
A[1,1] = sym.integrate(psi_1*psi_1, (x, 0, 1))
b[0] = sym.integrate(f(x)*psi_0, (x, 0, 1))
b[1] = sym.integrate(f(x)*psi_1, (x, 0, 1))
print(('A:', A))
print(('b:', b))
c = A.LUsolve(b)
c = [sym.simplify(c[i,0]) for i in range(c.shape[0])]
print(('c:', c, [c_.evalf() for c_ in c]))
u = c[0]*psi_0 + c[1]*psi_1
print(('u:', u))
```

Looking at the results, we get the linear system

$$
\left(\begin{matrix}1 & \frac{2}{\pi}\\\frac{2}{\pi} & \frac{1}{2}\end{matrix}\right)
\left(\begin{bmatrix}
\frac{- 24 \pi^{2} - 96 + 4 \pi^{4}}{3 \pi^{2} \left(-8 + \pi^{2}\right)}\\
\frac{- 4 \pi^{2} + 48}{3 \pi \left(-8 + \pi^{2}\right)}
\end{bmatrix}\right) =
\left(\begin{matrix}
\frac{4}{3}\\
\frac{8}{\pi^{3}} + \frac{2}{\pi}
\end{matrix}\right)
$$

The resulting best approximation reads

$$
u(x) = \frac{4 \left(- \pi^{2} + 12\right) \sin{\left (\pi x \right )}}{3 \pi \left(-8 + \pi^{2}\right)} + \frac{- 24 \pi^{2} - 96 + 4 \pi^{4}}{3 \pi^{2} \left(-8 + \pi^{2}\right)}\approx 1.025 + 0.484\sin(\pi x)
$$

The $L^2$ error turns out to be $0.00876$.
To conclude, the least squares method is slightly better than the intuition
in this case.

Filename: `parabola_sin`.

## Problem 4: Approximate the exponential function by power functions

Let $V$ be a function space with basis functions $x^i$,
$i=0,1,\ldots,N$.  Find the best approximation to $f(x)=\exp(-x)$ on
$\Omega =[0,8]$ among all functions in $V$ for $N=2,4,6$. Illustrate
the three approximations in three separate plots.

**Hint.**
Apply the `lest_squares` and
`comparison_plot` functions in the `approx1D.py` module as these
make the exercise easier to solve.

**Solution.**
A suitable code is

```python
from approx1D import least_squares, comparison_plot
import matplotlib.pyplot as plt
import sympy as sym
from math import factorial
import numpy as np

x = sym.Symbol('x')
f = sym.exp(-x)

Omega = [0, 8]

for N in 2,4,6:
    psi = [x**i for i in range(N+1)]
    u, c = least_squares(f,psi,Omega)
    print((N, u))
    comparison_plot(f, u, Omega, filename='tmp_exp_%d' % N,
                    plot_title='N=%d' % N)
```

For the case $N=2$ the program prints the following, here edited for clearer
reading:

        A:
        Matrix([[8, 32, 512/3], [32, 512/3, 1024], [512/3, 1024, 32768/5]])
        b:
        Matrix([[-exp(-8) + 1], [-9*exp(-8) + 1], [-82*exp(-8) + 2]])

        f: exp(-x)
        u: x**2*(-465*exp(-8)/4096 + 105/4096) + x*(-141/512 +
           405*exp(-8)/512) - 111*exp(-8)/128 + 87/128

Filename: `exp_powers`.

## Problem 5: Approximate the sine function by power functions

In this exercise we want to approximate the sine function by polynomials
of order $N+1$. Consider two bases:

$$
\begin{align*}
V_1 &= \{x, x^3, x^5, \ldots, x^{N-2}, x^N \}, \\
V_2 &= \{1,x,x^2,x^3,\ldots, x^N\}{\thinspace .}
\end{align*}
$$

The basis $V_1$ is motivated by the fact that the Taylor polynomial
approximation to the sine function has only odd powers, while $V_2$
is motivated by the assumption that including the even powers could
improve the approximation in a least-squares setting.

Compute the best approximation to $f(x)=\sin(x)$ among all functions
in $V_1$ and $V_2$ on two domains of increasing sizes: $\Omega_{1,k} =
[0, k\pi]$, $k=2,3\ldots,6$ and $\Omega_{2,k} = [-k\pi /2, k\pi/2]$,
$k=2,3,4,5$.  Make plots for all combinations of $V_1$, $V_2$,
$\Omega_1$, $\Omega_2$, $k=2,3,\ldots,6$.

Add a plot of the $N$-th degree Taylor polynomial approximation of
$\sin(x)$ around $x=0$.

**Hint.**
You can make a loop over $V_1$ and $V_2$, a loop over
$\Omega_1$ and $\Omega_2$, and a loop over $k$. Inside the loops,
call the functions `least_squares` and
`comparison_plot` from the `approx1D` module.
$N=7$ is a suggested value.

**Solution.**
Suitable code is

```python
import sympy as sym
from approx1D import least_squares, comparison_plot
from math import pi
import matplotlib.pyplot as plt

x = sym.Symbol('x')
f = sym.sin(x)
N = 7
psi_bases = [[x**i for i in range(1, N+1, 2)],  # V_1
             [x**i for i in range(0, N+1)]]     # V_2
symbolic = False

for V, psi in enumerate(psi_bases):
    for domain_no in range(1, 3):
        for k in range(2, 6):
            if symbolic:
                Omega = [0, k*sym.pi] if domain_no == 1 else \
                        [-k*sym.pi/2, k*sym.pi/2]
            else:
                # cannot use sym.pi with numerical sympy computing
                Omega = [0, k*pi] if domain_no == 1 else \
                        [-k*pi/2, k*pi/2]

            u, c = least_squares(f, psi, Omega, symbolic=symbolic)

            comparison_plot(
                f, u, Omega,
                ymin=-2, ymax=2,
                filename='tmp_N%d_V%dOmega%dk%d' %
                (N, V, k, domain_no),
                plot_title='sin(x) on [0,%d*pi/2] by %s' %
                (k, ','.join([str(p) for p in psi])))
            # Need to kill the plot to proceed!
        for ext in 'png', 'pdf':
            cmd = 'doconce combine_images -2 ' + \
                  ' '.join(['tmp_N%d_V%dOmega%dk%d.' %
                            (N, V, k, domain_no) + ext
                            for k in range(2, 6)]) + \
                  ' sin_powers_N%d_V%d_Omega%d.' % (N, V, domain_no) + ext
            print(cmd)
            os.system(cmd)

# Show the standard Taylor series approximation
from math import factorial, pi
import time
Omega = [0, 12*pi/2.]
u = 0
for k in range(0,N+1):
    u = u + ((-1)**k*x**(1+2*k))/float(factorial(1+2*k))
# Shorter: u = sum(((-1)**k*x**(1+2*k))/float(factorial(1+2*k))
# for k in range(0,10))
comparison_plot(f, u, Omega, 'sin_taylor%d' % k,
                ymin=-1.5, ymax=1.5)
```

The odd powers ($V_1$ space) behave not so good on $\Omega_{1,k}$,
but better on $\Omega_{2,k}$:

$V_1$ space, $\Omega_{1,k}$ domain.

$V_1$ space, $\Omega_{2,k}$ domain.

Including also even powers ($V_2$ space) is clearly much better:

$V_2$ space, $\Omega_{1,k}$ domain.

$V_2$ space, $\Omega_{2,k}$ domain.

Comparison with a standard Taylor series shows that it is very inferior
as an approximation over the entire domain, but much more accurate
close to the origin (as expected, since the Taylor series is constructed
with this property, while the least squares method tries to find a good
approximation over the entire domain).

Filename: `sin_powers`.

## Problem 6: Approximate a steep function by sines

Find the best approximation of $f(x) = \tanh (s(x-\pi))$ on
$[0, 2\pi]$ in the space $V$ with basis
${\psi}_i(x) = \sin((2i+1)x)$, $i\in{\mathcal{I}_s} = \{0,\ldots,N\}$.
Make a movie showing how $u=\sum_{j\in{\mathcal{I}_s}}c_j{\psi}_j(x)$
approximates $f(x)$ as $N$ grows. Choose $s$ such that $f$ is
steep ($s=20$ is appropriate).

**Hint 1.**
One may naively call the `least_squares_orth` and `comparison_plot`
from the `approx1D` module in a loop and extend the basis with
one new element in each pass. This approach
implies a lot of recomputations.
A more efficient strategy is to let `least_squares_orth`
compute with only one basis function at a time and accumulate
the corresponding `u` in the total solution.

**Hint 2.**
`ffmpeg` or `avconv` may skip frames when plot files are combined to
a movie. Since there are few files and we want to see each of them,
use `convert` to make an animated GIF file (`-delay 200` is
suitable).

**Solution.**
The code may read

```python
import sympy as sym
from approx1D import least_squares_orth, comparison_plot
import matplotlib.pyplot as plt

x = sym.Symbol('x')

# Naive approach: (not utilizing the fact that i+1 computations can
# make use of i computations)
def naive(f, s, Omega, N=10):
    psi = []
    for i in range(N+1):
        psi.append(sym.sin((2*i+1)*x))
        u, c = least_squares_orth(f, psi, Omega, symbolic=False)
        comparison_plot(f, u, Omega, 'tmp_sin%02dx' % i,
                        legend_loc='upper left', show=True)

# Efficient approach: compute just the matrix diagonal
def efficient(f, s, Omega, N=10):
    u = 0
    for i in range(N+1):
        psi = [sym.sin((2*i+1)*x)]
        next_term, c = least_squares_orth(f, psi, Omega, False)
        u = u + next_term
        comparison_plot(f, u, Omega, 'tmp_sin%02dx' % i,
                        legend_loc='upper left', show=False,
                        plot_title='s=%g, i=%d' % (s, i))

if __name__ == '__main__':
    s = 20  # steepness
    f = sym.tanh(s*(x-sym.pi))
    from math import pi
    Omega = [0, 2*pi]  # sym.pi did not work here
    efficient(f, s, Omega, N=10)
    # Make movie
    # avconv/ffmpeg skips frames, use convert instead (few files)
    cmd = 'convert -delay 200 tmp_sin*.png tanh_sines_approx.gif'
    os.system(cmd)
    # Make static plots, 3 figures on 2 lines
    for ext in 'pdf', 'png':
        cmd = 'doconce combine_images %s -3 ' % ext
        cmd += 'tmp_sin00x tmp_sin01x tmp_sin02x tmp_sin04x '
        cmd += 'tmp_sin07x tmp_sin10x tanh_sines_approx'
        os.system(cmd)
    plt.show()
```

Filename: `tanh_sines`.

### Remarks

Approximation of a discontinuous (or steep) $f(x)$ by sines,
results in slow convergence and oscillatory behavior of the
approximation close to the abrupt changes in $f$.
This is known as the [Gibb's phenomenon](http://en.wikipedia.org/wiki/Gibbs_phenomenon).

## Problem 7: Approximate a steep function by sines with boundary adjustment

We study the same approximation problem as in
[Problem 6: Approximate a steep function by sines](#fem:approx:exer:tanh:sine1). Since ${\psi}_i(0)={\psi}_i(2\pi)=0$
for all $i$, $u=0$ at the boundary points $x=0$ and $x=2\pi$, while
$f(0)=-1$ and $f(2\pi)=1$. This discrepancy at the boundary can be
removed by adding a boundary function $B(x)$:

$$
u(x) = B(x) + \sum_{j\in{\mathcal{I}_s}} c_j{\psi}_j(x),
$$

where $B(x)$ has the right boundary values: $B(x_L)=f(x_L)$ and
$B(x_R)=f(x_R)$, with $x_L=0$ and $x_R=2\pi$ as the boundary points.
A linear choice of $B(x)$ is

$$
B(x) = \frac{(x_R-x)f(x_L) + (x-x_L)f(x_R)}{x_R-x_L}{\thinspace .}
$$

**a)**
Use the basis
${\psi}_i(x) = \sin((i+1)x)$, $i\in{\mathcal{I}_s} = \{0,\ldots,N\}$
and plot $u$ and $f$ for $N=16$. (It suffices to make plots for even $i$.)

**Solution.**
With a boundary term $B(x)$ we call `least_squares_orth` with
`f-B` as right-hand side function, and we must remember to add $B$ to $u$.

We can extend the code from [Problem 6: Approximate a steep function by sines](#fem:approx:exer:tanh:sine1) and
let the function `efficient` handle different choices of basis.
Appropriate code for all three subexercises is

```python
import sympy as sym
from approx1D import least_squares_orth, comparison_plot
import matplotlib.pyplot as plt
x = sym.Symbol('x')

def efficient(f, B, s, Omega, N=10, basis='a'):
    u = B
    for i in range(N+1):
        if basis == 'a':
            psi = [sym.sin((i+1)*x)]
        elif basis == 'b':
            psi = [sym.sin((2*i+1)*x)]
        elif basis == 'c':
            psi = [sym.sin(2*(i+1)*x)]
        next_term, c = least_squares_orth(f-B, psi, Omega, False)
        u = u + next_term
        # Make only plot for i even
        if i % 2 == 0:
            comparison_plot(f, u, Omega, 'tmp_sin%02dx' % i,
                            legend_loc='upper left', show=False,
                            plot_title='s=%g, i=%d' % (s, i))

if __name__ == '__main__':
    s = 20  # steepness
    f = sym.tanh(s*(x-sym.pi))
    from math import pi
    Omega = [0, 2*pi]  # sym.pi did not work here

    # sin((i+1)*x) basis
    xL = Omega[0]
    xR = Omega[1]
    B = ((xR-x)*f.subs(x, xL) + (x-xL)*f.subs(x, xR))/(xR-xL)
    for exercise in 'a', 'b', 'c':
        efficient(f, B, s, Omega, N=16, basis=exercise)
        # Make movie
        cmd = 'convert -delay 200 tmp_sin*.png '
        cmd += 'tanh_sines_boundary_term_%s.gif' % exercise
        os.system(cmd)
        # Make static plots, 3 figures on 2 lines
        for ext in 'pdf', 'png':
            cmd = 'doconce combine_images %s -3 ' % ext
            cmd += 'tmp_sin00x tmp_sin02x tmp_sin04x tmp_sin08x '
            cmd += 'tmp_sin12x tmp_sin16x '
            cmd += 'tanh_sines_boundary_term_%s' % exercise
            os.system(cmd)
```

**b)**
Use the basis from [Problem 6: Approximate a steep function by sines](#fem:approx:exer:tanh:sine1),
${\psi}_i(x) = \sin((2i+1)x)$, $i\in{\mathcal{I}_s} = \{0,\ldots,N\}$.
(It suffices to make plots for even $i$.)
Observe that the approximation converges to a piecewise
linear function!

**Solution.**

**c)**
Use the basis
${\psi}_i(x) = \sin(2(i+1)x)$, $i\in{\mathcal{I}_s} = \{0,\ldots,N\}$,
and observe that the approximation converges to a piecewise
constant function.

**Solution.**

Filename: `tanh_sines_boundary_term`.

### Remarks

The strange results in b) and c) are due to the choice of
basis. In b), ${\varphi}_i(x)$ is an odd function around
$x=\pi/2$ and $x=3\pi/2$. No combination of basis functions
is able to approximate the flat regions of $f$.
All basis functions in c) are even around $x=\pi/2$ and $x=3\pi/2$,
but odd at $x=0,\pi,2\pi$. With all the sines represented, as in a),
the approximation is not constrained with a particular symmetry
behavior.

## Exercise 8: Fourier series as a least squares approximation

**a)**
Given a function $f(x)$ on an interval $[0,L]$, look up the formula
for the coefficients $a_j$ and $b_j$ in the Fourier series of $f$:

$$
f(x) = \frac{1}{2}a_0 +
\sum_{j=1}^\infty a_j\cos \left(j\frac{2\pi x}{L}\right)
+ \sum_{j=1}^\infty b_j\sin \left(j\frac{2\pi x}{L}\right){\thinspace .}
$$

**Solution.**
From [Wikipedia](https://en.wikipedia.org/wiki/Fourier_series) we
have

$$
\begin{align*}
a_j &= \frac{2}{L}\int_0^P f(x)\cos\left(j\frac{2\pi x}{L}\right) {\, \mathrm{d}x},\\
b_j &= \frac{2}{L}\int_0^P f(x)\sin\left(j\frac{2\pi x}{L}\right) {\, \mathrm{d}x}{\thinspace .}
\end{align*}
$$

**b)**
Let an infinite-dimensional vector space $V$ have the basis functions
$\cos j\frac{2\pi x}{L}$ for $j=0,1,\dots,\infty$ and
$\sin j\frac{2\pi x}{L}$ for $j=1,\dots,\infty$.  Show that the least squares
approximation method from the section [Approximation principles](#fem:approx:global) leads to a
linear system whose solution coincides with the standard formulas for
the coefficients in a Fourier series of $f(x)$ (see also
the section [Fourier series](#fem:approx:global:Fourier)).

**Hint.**
You may choose

$$
\begin{equation}
{\psi}_{2i} = \cos\left( i\frac{2\pi}{L}x\right),\quad
{\psi}_{2i+1} = \sin\left( i\frac{2\pi}{L}x\right),
\label{fem:approx:exer:Fourier:basis} \tag{67}
\end{equation}
$$

for $i=0,1,\ldots,N\rightarrow\infty$.

**Solution.**
The entries in the linear system arising from the least squares method
are $A_{i,j}=\int_0^L{\psi}_i{\psi}_j{\, \mathrm{d}x}$ and $b_i=\int_0^L
f(x){\psi}_i {\, \mathrm{d}x}$. To avoid name clash between the right-hand side
components of the linear system and the $b_i$ coefficients in the
Fourier series, we use the symbol $q_i$ for the former.
With the basis functions in
([67](#fem:approx:exer:Fourier:basis)) we get four different types
of integrals:

$$
\begin{align*}
A_{2i,2j} &= \int_0^L \cos\left( i\frac{2\pi}{L}x\right)
\cos\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x} = A_{2j,2i},\\
A_{2i, 2j+1} &= \int_0^L \cos\left( i\frac{2\pi}{L}x\right)
\sin\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x},\\
A_{2i+1,2j} &= \int_0^L \sin\left( i\frac{2\pi}{L}x\right)
\cos\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x},\\
A_{2i+1, 2j+1} &= \int_0^L \sin\left( i\frac{2\pi}{L}x\right)
\sin\left( j\frac{2\pi}{L}x\right)dx,\\
q_{2i} &= \int_0^L f(x)\cos\left( i\frac{2\pi}{L}x\right){\, \mathrm{d}x},\\
q_{2i+1} &= \int_0^L f(x)\sin\left( i\frac{2\pi}{L}x\right){\, \mathrm{d}x}{\thinspace .}
\end{align*}
$$

Now, the sine and cosine basis functions are orthogonal on $[0,L]$.
We have in general

$$
\begin{align*}
\int_0^L
\cos\left( i\frac{2\pi}{L}x\right) \cos\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x}
&=0,\quad i\neq j,\\
\int_0^L
\cos\left( i\frac{2\pi}{L}x\right) \cos\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x}
&=\frac{L}{2},\quad i= j\neq 0,\\
\int_0^L
\cos\left( i\frac{2\pi}{L}x\right) \cos\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x}
&=L,\quad i= j= 0,\\
\int_0^L
\sin\left( i\frac{2\pi}{L}x\right) \sin\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x}
&=0,\quad i\neq j,\\
\int_0^L
\sin\left( i\frac{2\pi}{L}x\right) \sin\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x}
&=\frac{L}{2},\quad i= j,\\
\int_0^L
\cos\left( i\frac{2\pi}{L}x\right) \sin\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x}
&=0{\thinspace .}
\end{align*}
$$

These results imply that only diagonal terms in the coefficient matrix
are different from zero. We have

$$
\begin{align*}
A_{0,0} &= L,\\
A_{2i,2i} &= \frac{L}{2},\quad i>0,\\
A_{2i+1,2i+1} &= \frac{L}{2}{\thinspace .}
\end{align*}
$$

The unknown vector with components $c_i$ must be arranged as

$$
(a_0, b_1, a_1, b_2, a_2, b_3, \ldots){\thinspace .}
$$

We then get

$$
A_{0,0}a_0=q_0,\quad A_{1,1}b_1=q_1,\quad A_{2,2}a_1=q_2,\quad A_{3,3}b_2=q_3,\ldots
$$

These equations lead to the formulas

$$
\begin{align*}
a_0 &= \frac{1}{L}\int_0^P f(x){\, \mathrm{d}x},\\
b_1 &= \frac{2}{L}\int_0^P f(x)\sin\left( \frac{2\pi}{L}x\right){\, \mathrm{d}x},\\
a_1 &= \frac{2}{L}\int_0^P f(x)\cos\left( \frac{2\pi}{L}x\right){\, \mathrm{d}x},\\
b_2 &= \frac{2}{L}\int_0^P f(x)\sin\left( 2\frac{2\pi}{L}x\right){\, \mathrm{d}x},\\
a_2 &= \frac{2}{L}\int_0^P f(x)\cos\left( 2\frac{2\pi}{L}x\right){\, \mathrm{d}x}{\thinspace .}
\end{align*}
$$

which can be generalized to

$$
\begin{align*}
a_0 &= \frac{1}{L}\int_0^P f(x){\, \mathrm{d}x},\\
a_j &= \frac{2}{L}\int_0^P f(x)\cos\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x},\ j>0,\\
b_j &= \frac{2}{L}\int_0^P f(x)\sin\left( j\frac{2\pi}{L}x\right){\, \mathrm{d}x},\ j>0,
\end{align*}
$$

and these are the standard formulas for the Fourier coefficients in a)
if we recognize that the $a_0$ above is twice the $a_0$ in the
expressions in a).

**c)**
Choose $f(x) = H(x-\frac{1}{2})$ on $\Omega=[0,1]$, where $H$ is the
Heaviside function: $H(x)=0$ for $x<0$, $H(x)=1$ for $x>0$ and
$H(0)=\frac{1}{2}$. Find the coefficients $a_j$ and $b_j$ in the
Fourier series for $f(x)$. Plot the sum for $j=2N+1$, where $N=5$ and
$N=100$.

**Solution.**
The formulas give

$$
\begin{align*}
a_0 &= 2\int_0^1 f(x){\, \mathrm{d}x} = 2\int_{\frac{1}{2}}^1 {\, \mathrm{d}x},\\
a_j &= 2\int_0^1 f(x)\cos\left( 2j\pi x\right){\, \mathrm{d}x}
= 2\int_{\frac{1}{2}}^1 \cos\left( 2j\pi x\right){\, \mathrm{d}x},\\
b_j &= 2\int_{\frac{1}{2}}^1 \sin\left( 2j\pi x\right){\, \mathrm{d}x}{\thinspace .}
\end{align*}
$$

The integrals are readily computed by `sympy`:

```python
import sympy as sym
j = sym.symbols('k', integer=True)
x = sym.symbols('x', real=True)
I = integrate(cos(2*j*pi*x), (x,Rational(1,2),1))
I
```

```python
I = integrate(cos(2*0*pi*x), (x,Rational(1,2),1))
I
```

```python
I = integrate(sin(2*j*pi*x), (x,Rational(1,2),1))
I
```

This means that we have the series

$$
u(x) = \frac{1}{2} + 2\sum_{j=1}^\infty \frac{(-1)^j - 1}{2\pi j}
\sin\left( 2j\pi x\right){\thinspace .}
$$

We only get a nonzero coefficient for $j$ odd:

$$
u(x) = \frac{1}{2} -2\sum_{k=1}^\infty \frac{1}{(2k+1)\pi}
\sin\left( 2(2k+1)\pi x\right){\thinspace .}
$$

Appropriate computer code for visualizing the series goes like

```python
import numpy as np
import matplotlib.pyplot as plt
from math import pi
from numpy import sin

def Heaviside_series(x, N):
    s = 0.5
    for k in range(N):
        s += -2.0/((2*k+1)*pi)*sin(2*(2*k+1)*pi*x)
    return s

x = np.linspace(0, 1, 1001)
for N in 5, 100:
    H = Heaviside_series(x, N)
    plt.figure()
    plt.plot(x, H)
    plt.legend(['$N=%d$' % N], loc='upper left')
    plt.savefig('tmp_%d.png' % N)
    plt.savefig('tmp_%d.pdf' % N)
plt.show()
```

We clearly see the Gibbs' phenomenon: oscillations and overshoot around
the point of discontinuity in the function we try to approximate.

Filename: `Fourier_ls`.

## Problem 9: Approximate a steep function by Lagrange polynomials

Use interpolation with uniformly distributed
points and Chebychev nodes to approximate

$$
f(x) = -\tanh(s(x-\frac{1}{2})),\quad x\in [0,1],
$$

by Lagrange polynomials for $s=5$ and $s=20$, and $N=3,7,11,15$.
Combine $2\times 2$ plots of the approximation for the four
$N$ values, and create such figures for the four combinations of
$s$ values and point types.

**Solution.**
The following code does the work (symbolically):

```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from approx1D import interpolation, comparison_plot
from Lagrange import Lagrange_polynomials
import sympy as sym

x = sym.Symbol('x')
Omega = [0,1]
N_values = 3, 7, 11, 15

for s in 5, 20:
    f = -sym.tanh(s*(x-0.5))  # sympy expression
    for distribution in 'uniform', 'Chebyshev':
        for N in N_values:
            phi, points = Lagrange_polynomials(
                x, N, Omega,
                point_distribution=distribution)

            u, c = interpolation(f, phi, points)
            filename = 'tmp_tanh_%d_%d_%s' % (N, s, distribution)
            comparison_plot(f, u, Omega, filename,
                            plot_title='s=%g, N=%d, %s points' %
                            (s, N, distribution))
        # Combine plot files (2x2)
        for ext in 'png', 'pdf':
            cmd = 'doconce combine_images ' + ext + ' '
            cmd += ' '.join([
                'tmp_tanh_%d_%d_%s' % (N, s, distribution)
                for N in N_values])
            cmd += ' tanh_Lagrange_%s_s%s' % (distribution, s)
            os.system(cmd)
```

For a smooth function ($s=5$), the difference between uniform points and
Chebyshev nodes is not substantial:

However, for a steep function ($s=20$) the overshoot and oscillations
associated with uniform points must be considered unacceptable for
larger $N$ values:

Switching to Chebyshev points does give a great improvement, but
we still have oscillatory approximations:

Filename: `tanh_Lagrange`.

## Problem 10: Approximate a steep function by Lagrange polynomials and regression

Redo [Problem 9: Approximate a steep function by Lagrange polynomials](#fem:approx:exer:tanh:Lagrange), but apply a regression
method with $N$-degree Lagrange polynomials and $2N+1$
data points. Recall that
[Problem 9: Approximate a steep function by Lagrange polynomials](#fem:approx:exer:tanh:Lagrange) applies $N+1$ points and
the resulting approximation interpolates $f$ at these points, while
a regression method with more points does not interpolate $f$ at the
data points.
Do more points and a regression method help reduce
the oscillatory behavior of Lagrange polynomial approximations?

**Solution.**
We start out with the program from
[Problem 9: Approximate a steep function by Lagrange polynomials](#fem:approx:exer:tanh:Lagrange). This time we need
to call `Lagrange_polynomails` twice: first to compute the ${\psi}(x)$
functions (of degree $N$) and then to compute the data points corresponding
to a uniform or Chebyshev distribution of $2N+1$ nodes.

```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from approx1D import regression, comparison_plot
from Lagrange import Lagrange_polynomials
import sympy as sym
import numpy as np

x = sym.Symbol('x')
Omega = [0,1]
N_values = 3, 7, 11, 15

for s in 5, 20:
    f = -sym.tanh(s*(x-0.5))  # sympy expression
    for distribution in 'uniform', 'Chebyshev':
        for N in N_values:
            # Compute the points from a 2*N Lagrange polynomial
            dummy, points = Lagrange_polynomials(
                x, 2*N, Omega,
                point_distribution=distribution)
            # Compute phi from N points Lagrange polynomial
            phi, dummy = Lagrange_polynomials(
                x, N, Omega,
                point_distribution=distribution)
            points = np.array(points, dtype=float)
            point_values = -np.tanh(s*(points-0.5))

            u, c = regression(f, phi, points)
            filename = 'tmp_tanh_%d_%d_%s' % (N, s, distribution)
            comparison_plot(f, u, Omega, filename,
                            plot_title='s=%g, N=%d, %s points' %
                            (s, N, distribution),
                            points=points, point_values=point_values,
                            points_legend='%s points' % (2*N))
        # Combine plot files (2x2)
        for ext in 'png', 'pdf':
            cmd = 'doconce combine_images ' + ext + ' '
            cmd += ' '.join([
                'tmp_tanh_%d_%d_%s' % (N, s, distribution)
                for N in N_values])
            cmd += ' tanh_Lagrange_regr_%s_s%s' % (distribution, s)
            os.system(cmd)
```

An important point is to convert `points` to a `numpy` array using
`dtype=float`. Leaving out this second argument makes an array
of objects of symbolic expressions, and we cannot apply `tanh` to it.

The oscillatory behavior is much reduced using more points and a
regression method, and the difference between uniform and Chebyshev
points is minor, even in the steep case $s=20$:

Filename: `tanh_Lagrange_regression`.

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/02_APPROX_EXERCICES.ipynb


# 1. Set up the notebook

## 1.1 Import modules

```python
import numpy as np
import sympy as sym
import json
import matplotlib.pyplot as plt
from scipy import linalg
from scipy.interpolate import interp1d
```

## 1.2 Define data loaders

Here is a function to load data from simulation.

```python
def load_simulation_data(filename, dronename):
    # load data for all drones
    with open(filename, 'r') as f:
        data = json.load(f)

    # get data for one drone only
    data = data[dronename]

    # convert lists to numpy arrays
    for key in data.keys():
        if isinstance(data[key], dict):
            for subkey in data[key].keys():
                data[key][subkey] = np.array(data[key][subkey])
        else:
            data[key] = np.array(data[key])

    # return the data
    return data
```

Here is a function to load data from hardware (and resample it at 100 Hz using linear interpolation).

```python
def load_hardware_data(filename, t_min_offset=0, t_max_offset=0):
    # load raw data
    with open(filename, 'r') as f:
        data = json.load(f)

    # convert lists to numpy arrays
    for val in data.values():
        for key in val.keys():
            val[key] = np.array(val[key])

    # create an array of times at which to subsample
    t_min = -np.inf
    t_max = np.inf
    for key, val in data.items():
        t_min = max(t_min, val['time'][0])
        t_max = min(t_max, val['time'][-1])
    t_min += t_min_offset * 1000
    t_max -= t_max_offset * 1000
    nt = int(1 + np.floor((t_max - t_min) / 10.))
    t = np.arange(0, 10 * nt, 10) / 1000.
    resampled_data = {'time': t}

    # resample raw data with linear interpolation
    for k, v in data.items():
        f = interp1d((v['time'] - t_min) / 1000., v['data'])
        resampled_data[k] = f(t)

    # return the resampled data
    return resampled_data
```

## 1.3 Define autoencoders

Here is a function to print code that implements linear state feedback.

```python
def export_controller(K, s, i, s_with_des, i_eq,
                      decimals=8,
                      suffix='',
                      line_ending=''):
    """
    K is a gain matrix, of size m x n
    s is a list of states as symbolic variables, of length n
    i is a list of inputs as symbolic variables, of length m
    s_with_des is a list of states that have desired values, as
        symbolic variables - if there are no such states, then
        this should be an empty list []
    i_eq is a list of equilibrium values of inputs, of length m
    decimals is the number of decimals to include when printing
        each value
    suffix is the character (if any) to print after each number,
        for example 'f' to indicate a "float" when exporting to C
    line_ending is the character (if any) to print after each
        line, for example ';' when exporting to C
    """

    s_name = [scur.name for scur in s]
    i_name = [icur.name for icur in i]
    for row in range(len(i_name)):
        input_string = ''
        for col in range(len(s_name)):
            k = K[row, col]
            if not np.isclose(k, 0.):
                if (k < 0) and input_string:
                    input_string += ' +'
                if s[col] in s_with_des:
                    n = f'({s_name[col]} - {s_name[col]}_des)'
                else:
                    n = s_name[col]
                input_string += f' {-k:.{decimals}f}{suffix} * {n}'
        if not np.isclose(i_eq[row], 0.):
            if (i_eq[row] > 0) and input_string:
                input_string += ' +'
            input_string += f' {i_eq[row]:.{decimals}f}{suffix}'
        print(f'{i_name[row]} ={input_string}{line_ending}')
```

Here is a function to print code that implements a method of power distribution.

```python
def export_power_distribution(Pinv,
                              limiter='self.limitUint16',
                              decimals=1,
                              suffix='',
                              line_ending=''):
    """
    Pinv is a 4 x 4 matrix that maps inputs (tau_x, tau_y, tau_z, f_z)
        to motor power commands (m_1, m_2, m_3, m_4)
    limiter is the name of the function to apply that ensures each
        motor power command is valid (i.e., an integer within bounds),
        for example "limitUint16" when exporting to C
    decimals is the number of decimals to include when printing
        each value
    suffix is the character (if any) to print after each number,
        for example 'f' to indicate a "float" when exporting to C
    line_ending is the character (if any) to print after each
        line, for example ';' when exporting to C
    """

    i_name = ['tau_x', 'tau_y', 'tau_z', 'f_z']
    m_name = ['m_1', 'm_2', 'm_3', 'm_4']
    for row in range(len(m_name)):
        input_string = ''
        for col in range(len(i_name)):
            k = Pinv[row, col]
            if not np.isclose(k, 0.):
                if (k > 0) and input_string:
                    input_string += ' +'
                n = i_name[col]
                input_string += f' {k:.{decimals}f}{suffix} * {n}'
        print(f'{m_name[row]} = {limiter}({input_string} ){line_ending}')
```

## 1.5 Define an LQR solver

Here is a function that solves the linear quadratic regulator (LQR) problem - i.e., that finds the matrix $K$ for which

$$u(t) = - K x(t)$$

is the optimal solution to

$$
\begin{align*}
\underset{u_{[t_{0}, \infty)}}{\text{minimize}} &\qquad \int_{t_{0}}^{\infty}\left( x(t)^{T}Qx(t)+u(t)^{T}Ru(t)\right)dt \\
\text{subject to} &\qquad \dot{x}(t) = Ax(t)+Bu(t), \quad x(t_{0})=x_{0}.
\end{align*}
$$

```python
def lqr(A, B, Q, R):
    P = linalg.solve_continuous_are(A, B, Q, R)
    K = linalg.inv(R) @  B.T @ P
    return K
```

# 2. Derive equations of motion

## 2.1 Define symbolic variables

Define states.

```python
# components of position (meters)
o_x, o_y, o_z = sym.symbols('o_x, o_y, o_z')

# yaw, pitch, and roll angles (radians)
psi, theta, phi = sym.symbols('psi, theta, phi')

# components of linear velocity (meters / second)
v_x, v_y, v_z = sym.symbols('v_x, v_y, v_z')

# components of angular velocity (radians / second)
w_x, w_y, w_z = sym.symbols('w_x, w_y, w_z')
```

Define inputs.

```python
# components of net rotor torque
tau_x, tau_y, tau_z = sym.symbols('tau_x, tau_y, tau_z')

# net rotor force
f_z = sym.symbols('f_z')
```

Define parameters.

```python
m, J_x, J_y, J_z, g = sym.symbols('m, J_x, J_y, J_z, g')
```

Create linear and angular velocity vectors (in coordinates of the body frame).

```python
v_01in1 = sym.Matrix([[v_x], [v_y], [v_z]])
w_01in1 = sym.Matrix([[w_x], [w_y], [w_z]])
```

Create moment of inertia matrix (in coordinates of the body frame).

```python
J_in1 = sym.diag(J_x, J_y, J_z)
```

## 2.2 Define kinematics of orientation

### 2.2.1 Rotation matrix in terms of yaw, pitch, roll angles

Define individual rotation matrices.

```python
Rz = sym.Matrix([[sym.cos(psi), -sym.sin(psi), 0],
                 [sym.sin(psi), sym.cos(psi), 0],
                 [0, 0, 1]])

Ry = sym.Matrix([[sym.cos(theta), 0, sym.sin(theta)],
                 [0, 1, 0],
                 [-sym.sin(theta), 0, sym.cos(theta)]])

Rx = sym.Matrix([[1, 0, 0],
                 [0, sym.cos(phi), -sym.sin(phi)],
                 [0, sym.sin(phi), sym.cos(phi)]])
```

Apply sequential transformation to compute the rotation matrix that describes the orientation of the drone (i.e., of frame 1 in the coordinates of frame 0).

```python
R_1in0 = Rz * Ry * Rx
```

```python
R_1in0
```

### 2.2.2 Map from angular velocity to angular rates

Recall that

$$\begin{bmatrix} \dot{\psi} \\ \dot{\theta} \\ \dot{\phi} \end{bmatrix} = N w_{0, 1}^{1}$$

for some matrix $N$. Here is how to compute that matrix for a ZYX (yaw, pitch, roll) Euler angle sequence.  First, we compute its inverse:

```python
Ninv = sym.Matrix.hstack((Ry * Rx).T * sym.Matrix([[0], [0], [1]]),
                              (Rx).T * sym.Matrix([[0], [1], [0]]),
                                       sym.Matrix([[1], [0], [0]]))
```

Then, we compute $N$ by taking the inverse of $N^{-1}$:

```python
N = sym.simplify(Ninv.inv())
```

```python
N
```

## 2.3 Define equations of motion

Forces.

```python
f_in1 = R_1in0.T * sym.Matrix([[0], [0], [-m * g]]) + sym.Matrix([[0], [0], [f_z]])
```

Torques.

```python
tau_in1 = sym.Matrix([[tau_x], [tau_y], [tau_z]])
```

```python
f_in1
```

```python
tau_in1
```

Create equations of motion.

```python
f_sym = sym.Matrix.vstack(R_1in0 * v_01in1,
                          N * w_01in1,
                          (1 / m) * (f_in1 - w_01in1.cross(m * v_01in1)),
                          J_in1.inv() * (tau_in1 - w_01in1.cross(J_in1 * w_01in1)))
```

Show equations of motion, which have the form

$$\dot{s} = f(s, i, p)$$

for states

$$
s = \begin{bmatrix} o_x \\ o_y \\ o_z \\ \psi \\ \theta \\ \phi \\ v_x \\ v_y \\ v_z \\ w_x \\ w_y \\ w_z \end{bmatrix},
$$

inputs
$$
i = \begin{bmatrix} \tau_x \\ \tau_y \\ \tau_z \\ f_z \end{bmatrix},
$$

and parameters
$$
p = \begin{bmatrix} m \\ J_x \\ J_y \\ J_z \\ g \end{bmatrix}.
$$

```python
f_sym
```

# 3. Derive state-space model

## 3.1 Choose equilibrium point

An equilibrium point of the nonlinear system is a choice of states $s_\text{eq}$ and inputs $i_\text{eq}$ - along with constant parameters $p_\text{eq}$ - for which

$$0 = f(s_\text{eq}, i_\text{eq}, p_\text{eq}).$$

Create a list of states, inputs, and parameters as symbolic variables.

```python
s = [o_x, o_y, o_z, psi, theta, phi, v_x, v_y, v_z, w_x, w_y, w_z]
i = [tau_x, tau_y, tau_z, f_z]
p = [m, J_x, J_y, J_z, g]
```

Create a list of states to track as symbolic variables. These are states whose desired values will be specified by a client.

```python
s_with_des = [o_x, o_y, o_z]
```

Create a function that evaluates $f(\cdot)$ at particular values of $s$, $i$, and $p$.

```python
f = sym.lambdify(s + i + p, f_sym)
```

Define constants.

```python
# Mass
m = 0.032    # <-- FIXME

# Principle moments of inertia
J_x = 1e-05  # <-- FIXME
J_y = 1e-05  # <-- FIXME
J_z = 2e-05  # <-- FIXME

# Acceleration of gravity
g = 9.81
```

Create a list of parameter values in the **same order** as the symbolic list. These are the parameter estimates we found in our experiments. They are not choices. (We use the subscript `_eq` to be consistent with what follows, and could say "parameter values *at equilibrium*," but don't be misled. These parameter values are *given* and are *constant* - again, they aren't choices.)

```python
p_eq = [m, J_x, J_y, J_z, g]
```

Create a list of state and input values at equilibrium in the **same order** as the symbolic lists.

```python
s_eq = [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.] # <-- FIXME
i_eq = [0., 0., 0., 0.]                                 # <-- FIXME
```

Evaluate the equations of motion at the equilibrium point - if it actually *is* an equilibrium point, then the result should be an array of zeros:

```python
print(f(*s_eq, *i_eq, *p_eq))
```

Note that this equilibrium point would remain an equilibrium point for any choice of `o_x`, `o_y`, and `o_z` - that is one thing that allows the controller to track desired values of these variables.

## 3.2 Compute A and B

We want to find

$$
A = \frac{\partial f}{\partial s}\biggr\vert_{(s, i, p) = (s_\text{eq}, i_\text{eq}, p_\text{eq})}
\qquad\text{and}\qquad
B = \frac{\partial f}{\partial i}\biggr\vert_{(s, i, p) = (s_\text{eq}, i_\text{eq}, p_\text{eq})}.
$$

First, we compute each Jacobian (i.e., each matrix of partial derivatives) in symbolic form.

```python
A_sym = f_sym.jacobian(s)
B_sym = f_sym.jacobian(i)
```

Then, we create functions that allow us to evaluate these Jacobians at particular values of $s$, $i$, and $p$.

```python
A_num = sym.lambdify(s + i + p, A_sym)
B_num = sym.lambdify(s + i + p, B_sym)
```

Finally, we plug in our equilibrium point.

```python
A = A_num(*s_eq, *i_eq, *p_eq)
B = B_num(*s_eq, *i_eq, *p_eq)
```

Show $A$ (formatted nicely).

```python
A_str = np.array2string(A,
                        formatter={'float_kind': lambda x: f'{x:5.2f}'},
                        prefix='    ',
                        max_line_width=np.inf)

print(f'A = {A_str}')
```

Show $B$ (formatted nicely).

```python
B_str = np.array2string(B,
                        formatter={'float_kind': lambda x: f'{x:10.2f}'},
                        prefix='    ',
                        max_line_width=np.inf)

print(f'B = {B_str}')
```

The state-space system is described by

$$ \dot{x} = Ax + Bu $$

where

$$ x = s - s_\text{eq} $$

and

$$ u = i - i_\text{eq}. $$

Note that $A$ and $B$ would remain the same for any choice of `o_x`, `o_y`, and `o_z` - that is the other thing that allows the controller to track desired values of these variables.

# 4. Design method of power distribution

It should only be necessary to do this once.

Define constants.

```python
k_F = 2e-6 # <-- FIXME
k_M = 1e-8 # <-- FIXME
l = 0.035  # <-- FIXME
```

Define the matrix $P$ that maps motor power commands ($m_1$, $m_2$, $m_3$, $m_4$) to inputs ($\tau_x$, $\tau_y$, $\tau_z$, $f_z$).

```python
P = np.array([[ -l * k_F, -l * k_F,  l * k_F,  l * k_F  ],
              [ -l * k_F, l * k_F,   l * k_F,  -l * k_F ],
              [ -k_M,     k_M,       -k_M,     k_M      ],
              [ k_F,      k_F,       k_F,      k_F      ]])
```

Compute the matrix $P^{-1}$ that maps inputs to motor power commands.

```python
Pinv = linalg.inv(P)
```

Print code that implements the method of power distribution in python (for simulation).

```python
export_power_distribution(Pinv)
```

Print code that implements the method of power distribution in C (for hardware).

```python
export_power_distribution(
    Pinv,
    limiter='limitUint16',
    suffix='f',
    line_ending=';',
)
```

# 5. Design, implement, and test a sequence of controllers

To get a working controller in hardware, you will likely need to repeat the following steps many times:

* Design a controller
* Implement and test this controller in simulation
* Implement and test this controller in hardware

Here is a template for these three steps. Please duplicate this template for each flight test. Please also modify this template as appropriate (e.g., doing more than one flight test with the same controller, or doing only hardware flight tests when fine-tuning performance after you get a working controller).

## 5.x Flight test (template)

Each time you duplicate this section, replace the title with a brief description of your current flight test.

### 5.x.1 Control design

For your controller, we suggest you use linear state feedback of the form

$$u = - K (x - x_\text{des})$$

where the gain matrix $K$ is chosen by solving an infinite-horizon LQR problem (e.g., with the helper function `lqr`) and where

$$x_\text{des} = \begin{bmatrix} o_{x, \text{des}} \\ o_{y, \text{des}} \\ o_{z, \text{des}} \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \end{bmatrix}.$$

Remember that the LQR problem is defined by the choice of weights $Q$ and $R$. Both must be square matrices, $Q$ must be positve semidefinite, and $R$ must be positive definite. Both $Q$ and $R$ are almost always chosen to be diagonal. Remember that $Q$ penalizes non-zero states (i.e., non-zero values of $x$) and that $R$ penalizes non-zero inputs (i.e., non-zero values of $u$). If you want smaller states, increase $Q$. If you want smaller inputs, increase $R$.

**Replace this text** to say why and how you chose $Q$ and $R$ for the controller used in this flight test.

```python
Q = np.diag([
    1.,
    1.,
    1.,
    1.,
    1.,
    1.,
    1.,
    1.,
    1.,
    1.,
    1.,
    1.,
])

R = np.diag([
    1.,
    1.,
    1.,
    1.,
])
```

Find $K$ for the chosen $Q$ and $R$.

```python
K = lqr(A, B, Q, R)
```

Show $K$ (formatted nicely).

```python
K_str = np.array2string(K,
                        formatter={'float_kind': lambda x: f'{x:6.3f}'},
                        prefix='    ',
                        max_line_width=np.inf)

print(f'K = {K_str}')
```

Print code that implements the controller in python (for simulation).

```python
export_controller(
    K,               # the gain matrix
    s,               # list of states as symbolic variables
    i,               # list of inputs as symbolic variables
    s_with_des,      # list of states that have desired values as symbolic variables
    i_eq,            # list of equilibrium values of inputs
)
```

Print code that implements the controller in C (for hardware).

```python
export_controller(
    K,               # the gain matrix
    s,               # list of states as symbolic variables
    i,               # list of inputs as symbolic variables
    s_with_des,      # list of states that have desired values as symbolic variables
    i_eq,            # list of equilibrium values of inputs
    suffix='f',      # character to print after each number (indicates a "float")
    line_ending=';'  # character to print after each line
)
```

### 5.x.2 Simulation results

#### Describe the flight test

Replace this cell with the following information, at minimum:

* The name of the client you used (e.g., `RobotClient_Hover`).
* A description in words of the trajectory specified by this client.
* The initial state (i.e., as specified by `simulator.set_state`).

It may also be helpful to include a snippet of code from your client. You can include code in markdown with a "code block" that is delimited by three backslashes above and below - for example, this...

``````
```python
o_x = 0.
o_y = 0.
o_z = 0.3
psi = 0.
```
``````
...is rendered like this:
```python
o_x = 0.
o_y = 0.
o_z = 0.3
psi = 0.
```

#### Show a video of the flight test

Replace this cell with a video of your flight. Here are two ways to do this. (What you should *not* do is drag-and-drop your video into this notebook.)

##### Markdown

Put your video in the same directory as this notebook. Suppose this video is called `simulation_x_video.mov`. Then put the following code in a cell of type `Markdown` and evaluate it:
```

```

##### HTML
Put your video in the same directory as this notebook. Suppose this video is called `simulation_x_video.mov`. Then put the following code in a cell of type `Code` and evaluate it:
```
%%HTML

```
You can change the `width` parameter to resize your video.

#### Plot results from the flight test

Load data from simulation.

```python
data = load_simulation_data(
    'simulation_x_data.json', # <-- replace with name of file with simulation data
    'my_drone',               # <-- replace with name of drone whose data you want to load
)
```

Parse data to get:

* time
* position
* desired position
* orientation
* motor power commands

```python
# time
t = data['t']

# position
o_x = data['state']['o_x']
o_y = data['state']['o_y']
o_z = data['state']['o_z']

# desired position
o_x_des = data['setpoint']['o_x']
o_y_des = data['setpoint']['o_y']
o_z_des = data['setpoint']['o_z']

# orientation
psi = data['state']['psi']
theta = data['state']['theta']
phi = data['state']['phi']

# motor power commands
m_1 = data['power']['m_1']
m_2 = data['power']['m_2']
m_3 = data['power']['m_3']
m_4 = data['power']['m_4']
```

Compute error in position (this is a function of time).

```python
o_x_err = o_x - o_x_des
o_y_err = o_y - o_y_des
o_z_err = o_z - o_z_des
```

Compute root mean squared error in position (this is a single number).

```python
o_x_rmse = np.mean(o_x_err**2)
o_y_rmse = np.mean(o_y_err**2)
o_z_rmse = np.mean(o_z_err**2)
```

Plot position error (annotated with RMSE), orientation, and motor power commands.

```python
fig, (ax_pos, ax_ori, ax_pow) = plt.subplots(3, 1, figsize=(12, 12), sharex=True)
ax_pos.plot(t, o_x_err, label=f'x position error (RMSE {o_x_rmse:.2e})')
ax_pos.plot(t, o_y_err, label=f'y position error (RMSE {o_y_rmse:.2e})')
ax_pos.plot(t, o_z_err, label=f'z position error (RMSE {o_z_rmse:.2e})')
ax_pos.legend()
ax_pos.grid()
ax_ori.plot(t, psi, label='psi')
ax_ori.plot(t, theta, label='theta')
ax_ori.plot(t, phi, label='phi')
ax_ori.legend()
ax_ori.grid()
ax_pow.plot(t, m_1, label='m_1')
ax_pow.plot(t, m_2, label='m_2')
ax_pow.plot(t, m_3, label='m_3')
ax_pow.plot(t, m_4, label='m_4')
ax_pow.legend()
ax_pow.grid()
ax_pow.set_xlabel('time (s)')
```

#### Discuss the flight test

Replace this cell with a description of your results. Note any differences between what you expected and what you saw, in particular trying to explain the cause of any failures (e.g., crashed drone). Say what might be done to further improve performance.

### 5.x.3 Hardware results

#### Describe the flight test

Replace this cell with the following information, at minimum:

* A description of the flight trajectory, both in words and (if you like) a snippet of code from `flight.py`.
* A description of the flight conditions (e.g., where was the flight conducted, did you power cycle the drone just before flying, were you using the positioning system or only the onboard sensors, etc.).

#### Show a video of the flight test

Replace this cell with a video of your flight. Here are two ways to do this. (What you should *not* do is drag-and-drop your video into this notebook.)

##### Markdown

Put your video in the same directory as this notebook. Suppose this video is called `hardware_x_video.mov`. Then put the following code in a cell of type `Markdown` and evaluate it:
```

```

##### HTML
Put your video in the same directory as this notebook. Suppose this video is called `hardware_x_video.mov`. Then put the following code in a cell of type `Code` and evaluate it:
```
%%HTML

```
You can change the `width` parameter to resize your video.

#### Plot results from the flight test

Load data from hardware.

```python
data = load_hardware_data(
    'hardware_x_data.json', # <-- replace with name of file with hardware data
    t_min_offset=0.,      # <-- (optional) replace with how many seconds of data to ignore at start
    t_max_offset=0.,      # <-- (optional) replace with how many seconds of data to ignore at end
)
```

Parse data to get:

* time
* position
* desired position
* orientation
* motor power commands

```python
# time
t = data['time']

# position
o_x = data['ae483log.o_x']
o_y = data['ae483log.o_y']
o_z = data['ae483log.o_z']

# desired position
o_x_des = data['ae483log.o_x_des']
o_y_des = data['ae483log.o_y_des']
o_z_des = data['ae483log.o_z_des']

# orientation
psi = data['ae483log.psi']
theta = data['ae483log.theta']
phi = data['ae483log.phi']

# motor power commands
m_1 = data['ae483log.m_1']
m_2 = data['ae483log.m_2']
m_3 = data['ae483log.m_3']
m_4 = data['ae483log.m_4']
```

Compute error in position (this is a function of time).

```python
o_x_err = o_x - o_x_des
o_y_err = o_y - o_y_des
o_z_err = o_z - o_z_des
```

Compute root mean squared error in position (this is a single number).

```python
o_x_rmse = np.mean(o_x_err**2)
o_y_rmse = np.mean(o_y_err**2)
o_z_rmse = np.mean(o_z_err**2)
```

Plot position error (annotated with RMSE), orientation, and motor power commands.

```python
fig, (ax_pos, ax_ori, ax_pow) = plt.subplots(3, 1, figsize=(12, 12), sharex=True)
ax_pos.plot(t, o_x_err, label=f'x position error (RMSE {o_x_rmse:.2e})')
ax_pos.plot(t, o_y_err, label=f'y position error (RMSE {o_y_rmse:.2e})')
ax_pos.plot(t, o_z_err, label=f'z position error (RMSE {o_z_rmse:.2e})')
ax_pos.legend()
ax_pos.grid()
ax_ori.plot(t, psi, label='psi')
ax_ori.plot(t, theta, label='theta')
ax_ori.plot(t, phi, label='phi')
ax_ori.legend()
ax_ori.grid()
ax_pow.plot(t, m_1, label='m_1')
ax_pow.plot(t, m_2, label='m_2')
ax_pow.plot(t, m_3, label='m_3')
ax_pow.plot(t, m_4, label='m_4')
ax_pow.legend()
ax_pow.grid()
ax_pow.set_xlabel('time (s)')
```

#### Discuss the flight test

Replace this cell with a description of your results. Note any differences between what you expected and what you saw, in particular trying to explain the cause of any failures (e.g., crashed drone). Say what might be done to further improve performance.

# 6. Summarize and discuss the results

**Modify the text in this cell** to answer the following questions:

* What level of performance were you able to achieve?
* What are the remaining limitations of your controller?
* How might the control architecture be changed to address these limitations or to further improve performance?

```python

```


Computational Seismology

The Chebyshev Pseudospectral Method - Elastic Waves in 1D

---

This notebook is part of the supplementary material
to [Computational Seismology: A Practical Introduction](https://global.oup.com/academic/product/computational-seismology-9780198717416?cc=de&lang=en&#),
Oxford University Press, 2016.

##### Authors:
* David Vargas ([@dvargas](https://github.com/davofis))
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))
---

## Basic Equations
This notebook presents the numerical solution for the 1D elastic wave equation using the Chebyshev Pseudospectral Method. We depart from the equation

\begin{equation}
\rho(x) \partial_t^2 u(x,t) = \partial_x (\mu(x) \partial_x u(x,t)) + f(x,t),
\end{equation}

and use a standard 3-point finite-difference operator to approximate the time derivatives. Then, the displacement field is extrapolated as

\begin{equation}
\rho_i\frac{u_{i}^{j+1} - 2u_{i}^{j} + u_{i}^{j-1}}{dt^2}= \partial_x (\mu(x) \partial_x u(x,t))_{i}^{j} + f_{i}^{j}
\end{equation}

An alternative way of performing space derivatives of a function defined on the Chebyshev collocation points is to define a derivative matrix $D_{ij}$

\begin{equation}
D_{ij} =
  \begin{cases}
    -\frac{2 N^2 + 1}{6} \hspace{1.5cm} \text{for  i  = j = N}\\
    -\frac{1}{2} \frac{x_i}{1-x_i^2} \hspace{1.5cm} \text{for i = j = 1,2,...,N-1}\\
    \frac{c_i}{c_j} \frac{(-1)^{i+j}}{x_i - x_j} \hspace{1.5cm} \text{for i $\neq$ j = 0,1,...,N}
  \end{cases}
\end{equation}

where $N+1$ is the number of Chebyshev collocation points $ \ x_i = cos(i\pi / N)$, $ \ i=0,...,N$ and the $c_i$ are given as

$$ c_i = 2 \hspace{1.5cm} \text{for  i  = 0 or N} $$
$$ c_i = 1 \hspace{1.5cm} \text{otherwise} $$

This differentiation matrix allows us to write the derivative of the function $f_i = f(x_i)$ (possibly depending on time) simply as

$$\partial_x u_i = D_{ij} \ u_j$$

where the right-hand side is a matrix-vector product, and the Einstein summation convention applies.

```
# This is a configuration step for the exercise. Please run it before calculating the derivative!
import numpy as np
import matplotlib
# Show Plot in The Notebook
matplotlib.use("nbagg")
import matplotlib.pyplot as plt
from ricker import ricker
```

### 1. Chebyshev derivative method

#### Exercise
Define a python function call "get_cheby_matrix(nx)" that initializes the Chebyshev derivative matrix $D_{ij}$, call this function and display the Chebyshev derivative matrix.

```
#################################################################
# IMPLEMENT THE CHEBYSHEV DERIVATIVE MATRIX METHOD HERE!
#################################################################

# Call the chebyshev differentiation matrix
# ---------------------------------------------------------------
#D_ij =

# ---------------------------------------------------------------
# Display Differentiation Matrix
# ---------------------------------------------------------------
```

```

```

### 2. Initialization of setup

```
# Basic parameters
# ---------------------------------------------------------------
#nt = 5000        # number of time steps
tmax  = 0.0006
eps   = 1.4       # stability limit
isx   = 100
lw    = 0.7
ft    = 10
f0    = 100000    # dominant frequency
iplot = 20        # Snapshot frequency

# material parameters
rho = 2500.
c   = 3000.
mu  = rho*c**2

# space domain
nx = 100     # number of grid points in x 199
xs = np.floor(nx/2)      # source location
xr = np.floor(nx*0.8)
x  = np.zeros(nx+1)

# initialization of pressure fields
p = np.zeros(nx+1)
pnew = np.zeros(nx+1)
pold = np.zeros(nx+1)
d2p  = np.zeros(nx+1)

for ix in range(0,nx+1):
    x[ix] = np.cos(ix * np.pi / nx)
dxmin = min(abs(np.diff(x)))
dxmax = max(abs(np.diff(x)))

dt = eps*dxmin/c    # calculate time step from stability criterion
nt = int(round(tmax/dt))
```

### 3. Source Initialization

```
# source time function
# ---------------------------------------------------------------
t = np.arange(1, nt+1)*dt  # initialize time axis
T0 = 1./f0
tmp = ricker(dt, T0)
isrc = tmp
tmp = np.diff(tmp)
src = np.zeros(nt)
src[0:np.size(tmp)] = tmp

#spatial source function
# ---------------------------------------------------------------
sigma = 1.5*dxmax
x0 = x[int(xs)]
sg = np.exp(-1/sigma**2*(x-x0)**2)
sg = sg/max(sg)
```

### 4. Time Extrapolation

Now we time extrapolate using the previously defined get_cheby_matrix(nx) method to call the differentiation matrix. The discrete values of the numerical simulation are indicated by dots in the animation, they represent the Chebyshev collocation points. Observe how the wavefield near the domain center is less dense than towards the boundaries.

```
# Initialize animated plot
# ---------------------------------------------------------------
plt.figure(figsize=(10,6))
line = plt.plot(x, p, 'k.', lw=2)
plt.title('Chebyshev Method - 1D Elastic wave', size=16)
plt.xlabel(' x(m)', size=14)
plt.ylabel(' Amplitude ', size=14)

plt.ion() # set interective mode
plt.show()
# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
# Differentiation matrix
D = get_cheby_matrix(nx)
for it in range(nt):
    # Space derivatives
    dp = np.dot(D, p.T)
    dp = mu/rho * dp
    dp = D @ dp

    # Time extrapolation
    pnew = 2*p - pold + np.transpose(dp) * dt**2

    # Source injection
    pnew = pnew + sg*src[it]*dt**2/rho

    # Remapping
    pold, p = p, pnew
    p[0] = 0; p[nx] = 0 # set boundaries pressure free

    # --------------------------------------
    # Animation plot. Display solution
    if not it % iplot:
        for l in line:
            l.remove()
            del l

        # --------------------------------------
        # Display lines
        line = plt.plot(x, p, 'k.', lw=1.5)
        plt.gcf().canvas.draw()
```

notebooks/Computational Seismology/The Pseudospectral Method/ps_cheby_elastic_1d.ipynb


## Model Based Submodular Selection
Author: Jacob Schreiber <jmschreiber91@gmail.com>

Submodular selection is the task of identifying a representative subset of samples from a large set, and apricot focuses on the use of these algorithms to identify a good subset of data that can be used for the purpose of training machine learning models. However, submodular functions can also be coupled with a trained machine learning model and a feature attribution algorithm in order to identify the subset of features that the machine learning model thinks are the most important for training.

In order to understand how this works, let's review the feature based submodular algorithm that's implemented in apricot. These functions greedily add samples to the growing subset that utilize a diversity of features. The equation that the feature based functions optimize is as follows:

\begin{equation}
f(X) = \sum\limits_{u \in U} w_{u} \phi_{u} \left( \sum\limits_{x \in X} m_{u}(x_{u}) \right)
\end{equation}

In this equation, $U$ is the set of features, or dimensions, of a sample, and $u$ refers to a specific feature. $X$ refers to the original data set that we are selecting from and $x$ refers to a single sample from that data set. $w$ is a vector of weights that indicate how important each feature is, with $w_{u}$ being a scalar referring to how important feature $u$ is. Frequently these weights are uniform. $\phi$ refers to a set of saturating functions, such as $sqrt(X)$ or $log(X + 1)$, that have diminishing returns the larger X gets.

The default implementation in apricot for $m_{u}$ is the identify function $m_{u}(x_{u}) = x_{u}$, meaning that a sample will have the most gain when it has a high feature value in a feature that is not well represented with high values in the already included samples. What if, instead of using the raw feature value, we used the feature ~attribution~ as determined by some attribution algorithm? This would give us a subset of samples that gave a diversity of importanes.

```python
%pylab inline
import seaborn; seaborn.set_style('whitegrid')
import shap

from sklearn.linear_model import LogisticRegression
from shap import LinearExplainer
from apricot import FeatureBasedSelection
```

    Populating the interactive namespace from numpy and matplotlib

Let's start off with creating two clusters of data and seeing what happens when we use a feature based function off of them natively.

```python
X = numpy.concatenate([numpy.random.normal((7, 8), 1, size=(100, 2)),
                       numpy.random.normal((10, 5), 1, size=(100, 2))])

y = numpy.concatenate([numpy.zeros(100), numpy.ones(100)])

Xi, yi = FeatureBasedSelection(20).fit_transform(X, y)
```

```python
plt.figure(figsize=(8, 6))
plt.title("Submodular selection on Gaussian blobs", fontsize=16)
plt.scatter(X[:100, 0], X[:100, 1], marker='+', label='Negative Class')
plt.scatter(X[100:, 0], X[100:, 1], marker='+', label='Positive Class')
plt.scatter(Xi[yi == 0, 0], Xi[yi == 0, 1], label='Selected Negative Samples')
plt.scatter(Xi[yi == 1, 0], Xi[yi == 1, 1], label="Selected Positive Samples")

plt.legend(fontsize=14, loc=(1, 0.3))
plt.show()
```

That's unfortunate. It looks like the selected samples are just those with the highest feature values. This is not entirely true due to the non-linearity that's applied during selection, but an approximation is that as you run a diagonal line with slope y = -x down from the top right corner of the plot, it selected the first 20 points that it hits. This is a downside of using a feature based function natively.

Well, our goal here was not to select on the feature naively, but rather to identify features that a trained machine learning model thought were important. However, this term "important" can have two meanings. It can be the samples that are the most obvious samples from one or another class, perhaps the most representative, or it can be those that are on the boundary. Let's look at the first case first. In order to calculate feature importances we will use the package shap.

The first thing to do is to train our model. In our case let's train a logistic regression model.

```python
model = LogisticRegression().fit(X, y)
```

Now we need to gt the attributions. These values sum up to the final prediction and atribute to each feature the importance of that measurement.

```python
X_shap = LinearExplainer(model, X).shap_values(X)

plt.figure(figsize=(12, 4))
plt.subplot(121)
plt.title("Original Data", fontsize=14)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=10)
plt.scatter(X[y == 1, 0], X[y == 1, 1], s=10)

plt.subplot(122)
plt.title("Attributions", fontsize=14)
plt.scatter(X_shap[y == 0, 0], X_shap[y == 0, 1], s=10)
plt.scatter(X_shap[y == 1, 0], X_shap[y == 1, 1], s=10)
plt.xlabel("x-axis attribution", fontsize=14)
plt.ylabel("y-axis attribution", fontsize=14)
plt.show()
```

Unfortunately these values correspond to the log odds and so can be negative when they are predicting the negative class. We can correct for this by taking the absolute value of the attribution. That will ensure that the values are entirely positive and that the higher in magnitude they are, the most important that feature was for the prediction.

```python
X_shap = numpy.abs(X_shap)

plt.scatter(X_shap[y == 0, 0], X_shap[y == 0, 1], s=10, label="Negative Samples")
plt.scatter(X_shap[y == 1, 0], X_shap[y == 1, 1], s=10, label="Positive Samples")
plt.xlabel("|x-axis attribution|", fontsize=14)
plt.ylabel("|y-axis attribution|", fontsize=14)
plt.legend(fontsize=14, loc=(1.01, 0.4))
plt.show()
```

In this absolute attribution space we will still be selecting values as if you were running a line with slope y = -x down from the top of the plot, but these transformed values should be more amenable to that selection in that manner. Now, let's select the 20 most important samples according to the model. We can compare this to the samples that the model is most confident in, as ranked by maximum predicted probability.

```python
Xi1, yi1 = FeatureBasedSelection(20).fit_transform(X, y)

selector = FeatureBasedSelection(20)
selector.fit_transform(X_shap)
Xi2 = X[selector.ranking]
yi2 = y[selector.ranking]

y_pred = model.predict_proba(X).max(axis=1)
idx = numpy.argsort(y_pred)[::-1][:20]
Xi3 = X[idx]
yi3 = y[idx]

plt.figure(figsize=(14, 4))
plt.subplot(131)
plt.title("Selection in the original space", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=10, label="Negative Samples")
plt.scatter(X[y == 1, 0], X[y == 1, 1], s=10, label="Positive Samples")
plt.scatter(Xi1[yi1 == 0, 0], Xi1[yi1 == 0, 1], s=20, label="Negative Selected Samples")
plt.scatter(Xi1[yi1 == 1, 0], Xi1[yi1 == 1, 1], s=20, label="Positive Selected Samples")

plt.subplot(132)
plt.title("Selection in the transformed space", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=10, label="Negative Samples")
plt.scatter(X[y == 1, 0], X[y == 1, 1], s=10, label="Positive Samples")
plt.scatter(Xi2[yi2 == 0, 0], Xi2[yi2 == 0, 1], s=20, label="Negative Selected Samples")
plt.scatter(Xi2[yi2 == 1, 0], Xi2[yi2 == 1, 1], s=20, label="Positive Selected Samples")

plt.subplot(133)
plt.title("Selection in the transformed space", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=10, label="Negative Samples")
plt.scatter(X[y == 1, 0], X[y == 1, 1], s=10, label="Positive Samples")
plt.scatter(Xi3[yi3 == 0, 0], Xi3[yi3 == 0, 1], s=20, label="Negative Selected Samples")
plt.scatter(Xi3[yi3 == 1, 0], Xi3[yi3 == 1, 1], s=20, label="Positive Selected Samples")
plt.legend(fontsize=14, loc=(1, 0.5))
plt.show()
```

It certainly looks like using the absolute attributions in this manner is yielding samples that are more representative of the most confident samples. However, it's unclear what the differences between using attributions versus using the model predictions alone correspond to, and whether one is better than the other in this simple case.

Another way that one might want to select samples is based on the *least confident samples*, i.e., identifying those samples that lie near the decision boundary. This can be particularly useful for the identification of samples that a model would be uncertain about. We can use a similar procedure, but after taking the absolute value of the attribution values, we multiply all attributions for a sample by the *minimum predicted class probability*. This has the effect of reducing values for samples where the model is very confident, while preserving values when the model is uncertain. The submodular selection algorithm will then preferentially select samples that have a diversity of features being relevant for the prediction, but only for uncertain samples.

```python
X_shap = LinearExplainer(model, X).shap_values(X)
X_shap = (numpy.abs(X_shap).T * model.predict_proba(X).min(axis=1)).T

Xi1, yi1 = FeatureBasedSelection(20).fit_transform(X, y)

selector = FeatureBasedSelection(20)
selector.fit_transform(X_shap)
Xi2 = X[selector.ranking]
yi2 = y[selector.ranking]

y_pred = model.predict_proba(X).min(axis=1)
idx = numpy.argsort(y_pred)[::-1][:20]
Xi3 = X[idx]
yi3 = y[idx]

plt.figure(figsize=(14, 4))
plt.subplot(131)
plt.title("Selection on feature values", fontsize=14)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=10, label="Negative Samples")
plt.scatter(X[y == 1, 0], X[y == 1, 1], s=10, label="Positive Samples")
plt.scatter(Xi1[yi1 == 0, 0], Xi1[yi1 == 0, 1], s=20, label="Negative Selected Samples")
plt.scatter(Xi1[yi1 == 1, 0], Xi1[yi1 == 1, 1], s=20, label="Positive Selected Samples")

plt.subplot(132)
plt.title("Selection on feature attributions", fontsize=14)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=10, label="Negative Samples")
plt.scatter(X[y == 1, 0], X[y == 1, 1], s=10, label="Positive Samples")
plt.scatter(Xi2[yi2 == 0, 0], Xi2[yi2 == 0, 1], s=20, label="Negative Selected Samples")
plt.scatter(Xi2[yi2 == 1, 0], Xi2[yi2 == 1, 1], s=20, label="Positive Selected Samples")

plt.subplot(133)
plt.title("Least confident samples", fontsize=16)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=10, label="Negative Samples")
plt.scatter(X[y == 1, 0], X[y == 1, 1], s=10, label="Positive Samples")
plt.scatter(Xi3[yi3 == 0, 0], Xi3[yi3 == 0, 1], s=20, label="Negative Selected Samples")
plt.scatter(Xi3[yi3 == 1, 0], Xi3[yi3 == 1, 1], s=20, label="Positive Selected Samples")
plt.legend(fontsize=14, loc=(1, 0.3))
plt.show()
```

It looks like, again, using the attributions is much more meaningful tha running submodular selection on the original feature values. One could argue that it looks like performing submodular selection on the attributions yields a more diverse sampling near the decision boundary than using the those samples that the model is the least confident about, but it's a bit subjective to say that.

Let's try to solidify a situation in which submodular selection on the attribution values is better than selecting the least confident samples. In this case, we'll build a data set where samples overlap significant in one part of the decision boundary and not in another portion.

```python
numpy.random.seed(2)

X = numpy.concatenate([numpy.random.uniform((5.6, 5), (8, 6.3), size=(100, 2)),
                       numpy.random.uniform((4, 5), (5.8, 8), size=(200, 2)),
                       numpy.random.uniform(6, 8.1, size=(300, 2))])

y = numpy.concatenate([numpy.zeros(300), numpy.ones(300)])

plt.scatter(X[y == 0, 0], X[y == 0, 1], s=5, label="Negative Samples")
plt.scatter(X[y == 1, 0], X[y == 1, 1], c='r', s=5, label="Positive Samples")
plt.legend(fontsize=14, loc=(1, 0.4))
plt.xlim(4, 8)
plt.ylim(5, 8)
plt.show()
```

On this data set we can see that the blue and red samples overlap on the portion of the decision boundary where the y-axis separates the two classes, but that the samples don't overlap on the portion where the x-axis separates the two classes. A machine learning model built on this data will be less sure about the overlapping samples

```python
from sklearn.svm import SVC
from shap import KernelExplainer

model = SVC(probability=True).fit(X, y)
explainer = KernelExplainer(model.predict_proba, X)
X_shap = explainer.shap_values(X, l1_reg="num_features(10)")
X_shap = (numpy.abs(X_shap[1] - X_shap[0]).T * model.predict_proba(X).min(axis=1)).T

selector = FeatureBasedSelection(50)
selector.fit_transform(X_shap)
Xi2 = X[selector.ranking]

y_pred = model.predict_proba(X).min(axis=1)
idx = numpy.argsort(y_pred)[::-1]

xx, yy = np.meshgrid(np.arange(4, 8.1, 0.1), np.arange(5, 9.1, 0.1))
Z = model.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
```

    Using 600 background data samples could cause slower run times. Consider using shap.sample(data, K) or shap.kmeans(data, K) to summarize the background as K samples.

    HBox(children=(FloatProgress(value=0.0, max=600.0), HTML(value='')))

Now let's plot the attributions for each sample, the decision boundaries, and the selected samples using submodular selection versus the least confident samples.

```python
plt.figure(figsize=(14, 4))
plt.subplot(131)
plt.title("Feature Attributions", fontsize=16)
plt.scatter(X_shap[y == 0, 0], X_shap[y == 0, 1], s=5)
plt.scatter(X_shap[y == 1, 0], X_shap[y == 1, 1], c='r', s=5)
plt.scatter(X_shap[selector.ranking][:, 0], X_shap[selector.ranking][:, 1], c='g', s=20)
plt.xlabel("x-axis attribution", fontsize=12)
plt.ylabel("y-axis attribution", fontsize=12)

plt.subplot(132)
plt.title("Selection on Attribution", fontsize=16)
plt.contourf(xx, yy, Z, cmap='RdBu_r', linewidths=0.3, alpha=0.25)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=5)
plt.scatter(X[y == 1, 0], X[y == 1, 1], c='r', s=5)
plt.scatter(Xi2[:, 0], Xi2[:, 1], s=20, color='g', label="Selected Samples")
plt.xlim(4, 8)
plt.ylim(5, 8)

plt.subplot(133)
plt.title("Least confident samples", fontsize=16)

plt.contourf(xx, yy, Z, cmap='RdBu_r', linewidths=0.3, alpha=0.25)
plt.scatter(X[y == 0, 0], X[y == 0, 1], s=5)
plt.scatter(X[y == 1, 0], X[y == 1, 1], c='r', s=5)
plt.scatter(X[idx[:50], 0], X[idx[:50], 1], s=20, color='#FF6600', label="Selected Samples")
plt.xlim(4, 8)
plt.ylim(5, 8)
plt.savefig("img/attributionselection.png")
plt.show()
```

It looks like here that submodular selection identifies six samples from the x-axis dominated portion of the selection whereas the least confident samples selects only a single one.

```python

```