# 13 Linear Algebra: Singular Value Decomposition

One can always decompose a matrix $\mathsf{A}$

\begin{gather}
\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}\\
\mathsf{U}^T \mathsf{U} = \mathsf{U} \mathsf{U}^T = 1\\
\mathsf{V}^T \mathsf{V} = \mathsf{V} \mathsf{V}^T = 1
\end{gather}

where $\mathsf{U}$ and $\mathsf{V}$ are orthogonal matrices and the $w_j$ are the _singular values_ that are assembled into a diagonal matrix $\mathsf{W}$.

$$
\mathsf{W} = \text{diag}(w_j)
$$

The inverse (if it exists) can be directly calculated from the SVD:

$$
\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T
$$

## Solving ill-conditioned coupled linear equations

```python
import numpy as np
```

### Non-singular matrix
Solve the linear system of equations

$$
\mathsf{A}\mathbf{x} = \mathbf{b}
$$

Using the standard linear solver in numpy:

```python
A = np.array([
        [1, 2, 3],
        [3, 2, 1],
        [-1, -2, -6],
    ])
b = np.array([0, 1, -1])
```

```python
np.linalg.solve(A, b)
```

    array([ 0.83333333, -0.91666667,  0.33333333])

Using the inverse from SVD:

$$
\mathbf{x} = \mathsf{A}^{-1} \mathbf{b}
$$

```python
U, w, VT = np.linalg.svd(A)
print(w)
```

    [7.74140616 2.96605874 0.52261473]

First check that the SVD really factors $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python
U.dot(np.diag(w).dot(VT))
```

    array([[ 1.,  2.,  3.],
           [ 3.,  2.,  1.],
           [-1., -2., -6.]])

```python
np.allclose(A, U.dot(np.diag(w).dot(VT)))
```

    True

Now calculate the matrix inverse $\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T$:

```python
inv_w = 1/w
print(inv_w)
```

    [0.1291755  0.33714774 1.91345545]

```python
A_inv = VT.T.dot(np.diag(inv_w)).dot(U.T)
print(A_inv)
```

    [[-8.33333333e-01  5.00000000e-01 -3.33333333e-01]
     [ 1.41666667e+00 -2.50000000e-01  6.66666667e-01]
     [-3.33333333e-01 -1.08335035e-16 -3.33333333e-01]]

Check that this is the same that we get from `numpy.linalg.inv()`:

```python
np.allclose(A_inv, np.linalg.inv(A))
```

    True

Now, *finally* solve (and check against `numpy.linalg.solve()`):

```python
x = A_inv.dot(b)
print(x)
np.allclose(x, np.linalg.solve(A, b))
```

    [ 0.83333333 -0.91666667  0.33333333]

    True

```python
A.dot(x)
```

    array([-7.77156117e-16,  1.00000000e+00, -1.00000000e+00])

```python
np.allclose(A.dot(x), b)
```

    True

### Singular matrix

If the matrix $\mathsf{A}$ is *singular* (i.e., its rank (linearly independent rows or columns) is less than its dimension and hence the linear system of equation does not have a unique solution):

For example, the following matrix has the same row twice:

```python
C = np.array([
     [ 0.87119148,  0.9330127,  -0.9330127],
     [ 1.1160254,   0.04736717, -0.04736717],
     [ 1.1160254,   0.04736717, -0.04736717],
    ])
b1 = np.array([ 2.3674474,  -0.24813392, -0.24813392])
b2 = np.array([0, 1, 1])
```

```python
np.linalg.solve(C, b1)
```

NOTE: failure is not always that obvious: numerically, a matrix can be *almost* singular.

Try solving the linear system of equations

$$
\mathsf{D}\mathbf{x} = \mathbf{b}_1
$$
with matrix $\mathsf{D}$ below:

```python
D = C.copy()
D[2, :] = C[0] - 3*C[1]
D
```

    array([[ 0.87119148,  0.9330127 , -0.9330127 ],
           [ 1.1160254 ,  0.04736717, -0.04736717],
           [-2.47688472,  0.79091119, -0.79091119]])

```python
np.linalg.solve(D, b1)
```

    array([1.61493184e+00, 2.69013663e+16, 2.69013663e+16])

Note that some of the values are huge, and suspiciously like the inverse of machine precision? Sign of a nearly singular matrix.

**Note**: *Just because a function did not throw an exception it does not mean that the answer is correct.* **Always check your output!**

Now back to the example with $\mathsf{C}$:

#### SVD for singular matrices
If a matrix is *singular* or *near singular* then one can *still* apply SVD.

One can then compute the *pseudo inverse*

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

i.e., any singular $w_j = 0$ is being "augmented" by setting

$$
\frac{1}{w_j} \rightarrow 0 \quad\text{if}\quad w_j = 0
$$

in $\text{diag}(1/w_j)$.

Perform the SVD for the singular matrix $\mathsf{C}$:

```python
U, w, VT = np.linalg.svd(C)
print(w)
```

    [1.99999999e+00 1.00000000e+00 2.46519033e-32]

Note the third value $w_2 \approx 0$: sign of a singular matrix.

Test that the SVD really decomposes $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python
U.dot(np.diag(w).dot(VT))
```

    array([[ 0.87119148,  0.9330127 , -0.9330127 ],
           [ 1.1160254 ,  0.04736717, -0.04736717],
           [ 1.1160254 ,  0.04736717, -0.04736717]])

```python
np.allclose(C, U.dot(np.diag(w).dot(VT)))
```

    True

There are the **singular values** (let's say, $|w_i| < 10^{-12}$):

```python
singular_values = np.abs(w) < 1e-12
print(singular_values)
```

    [False False  True]

#### Pseudo-inverse

Calculate the **pseudo-inverse** from the SVD

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

Augment:

```python
inv_w = 1/w
inv_w[singular_values] = 0
print(inv_w)
```

    [0.5 1.  0. ]

```python
C_inv = VT.T.dot(np.diag(inv_w)).dot(U.T)
print(C_inv)
```

    [[-0.04736717  0.46650635  0.46650635]
     [ 0.5580127  -0.21779787 -0.21779787]
     [-0.5580127   0.21779787  0.21779787]]

#### Solution for $\mathbf{b}_1$

Now solve the linear problem with SVD:

```python
x1 = C_inv.dot(b1)
print(x1)
```

    [-0.34365138  1.4291518  -1.4291518 ]

```python
C.dot(x1)
```

    array([ 2.3674474 , -0.24813392, -0.24813392])

```python
np.allclose(C.dot(x1), b1)
```

    True

Thus, using the pseudo-inverse $\mathsf{C}^{-1}$ we can obtain solutions to the equation

$$
\mathsf{C} \mathbf{x}_1 = \mathbf{b}_1
$$

However, $\mathbf{x}_1$ is not the only solution: there's a whole line of solutions that are formed by the special solution and a combination of the basis vectors in the *null space* of the matrix:

The (right) *kernel* or *null space* contains all vectors $\mathbf{x^0}$ for which

$$
\mathsf{C} \mathbf{x^0} = 0
$$

(The dimension of the null space corresponds to the number of singular values.) You can find a basis that spans the null space. Any linear combination of null space basis vectors will also end up in the null space when $\mathbf{A}$ is applied to it.

Specifically, if $\mathbf{x}_1$ is a special solution and $\lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots$ is a vector in the null space then

$$
\mathbf{x} = \mathbf{x}_1 + ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots )
$$

is **also a solution** because

$$
\mathsf{C} \mathbf{x} = \mathsf{C} \mathbf{x_1} + \mathsf{C} ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots ) = \mathsf{C} \mathbf{x_1} + 0 = \mathbf{b}_1 + 0 = \mathbf{b}_1
$$

The $\lambda_i$ are arbitrary real numbers and hence there is an infinite number of solutions.

In SVD:

* The columns $U_{\cdot, i}$ of $\mathsf{U}$ (i.e. `U.T[i]` or `U[:, i]`) corresponding to non-zero $w_i$, i.e. $\{i : w_i \neq 0\}$, form the basis for the _range_ of the matrix $\mathsf{A}$.
* The columns $V_{\cdot, i}$ of $\mathsf{V}$ (i.e. `V.T[i]` or `V[:, i]`) corresponding to zero $w_i$, i.e. $\{i : w_i = 0\}$, form the basis for the _null space_ of the matrix $\mathsf{A}$.

```python
x1
```

    array([-0.34365138,  1.4291518 , -1.4291518 ])

The rank space comes from $\mathsf{U}^T$:

```python
U.T
```

    array([[-7.07106782e-01, -4.99999999e-01, -4.99999999e-01],
           [ 7.07106780e-01, -5.00000001e-01, -5.00000001e-01],
           [-2.47010760e-16, -7.07106781e-01,  7.07106781e-01]])

The basis vectors for the rank space (``~ bool_array`` applies a logical ``NOT`` operation to the entries in the boolean array so that we can pick out "not singular values"):

```python
U.T[~singular_values]
```

    array([[-0.70710678, -0.5       , -0.5       ],
           [ 0.70710678, -0.5       , -0.5       ]])

The null space comes from $\mathsf{V}^T$:

```python
VT
```

    array([[-0.8660254 , -0.35355339,  0.35355339],
           [-0.5       ,  0.61237244, -0.61237244],
           [-0.        , -0.70710678, -0.70710678]])

The basis vector for the null space:

```python
VT[singular_values]
```

    array([[-0.        , -0.70710678, -0.70710678]])

The component of $\mathbf{x}_1$ along the basis vector of the null space of $\mathsf{C}$ (here a 1D space) – note that this component is zero, i.e., the special solution lives in the rank space:

```python
x1.dot(VT[singular_values][0])
```

    2.220446049250313e-16

We can create a family of solutions by adding vectors in the null space to the special solution $\mathbf{x}_1$, e.g. $\lambda_1 = 2$:

```python
lambda_1 = 2
x1_1 = x1 + lambda_1 * VT[2]
print(x1_1)

np.allclose(C.dot(x1_1), b1)
```

    [-0.34365138  0.01493824 -2.84336536]

    True

Thus, **all** solutions are
```
x1 + lambda * VT[2]
```

#### Solution for  $\mathbf{b}_2$

The solution vector $x_2$ solves
$$
\mathsf{C}\mathbf{x}_2 = \mathbf{b}_2
$$

```python
b2
```

    array([0, 1, 1])

```python
x2 = C_inv.dot(b2)
print(x2)
print(C.dot(x2))
np.allclose(C.dot(x2), b2)
```

    [ 0.9330127  -0.43559574  0.43559574]
    [-4.4408921e-16  1.0000000e+00  1.0000000e+00]

    True

... and the general solution will again be obtained by adding any multiple of the null space basis vector.

#### Null space

The Null space is spanned by the following basis vectors (just one in this example):

```python
null_basis = VT[singular_values]
null_basis
```

    array([[-0.        , -0.70710678, -0.70710678]])

Show that
$$
\mathsf{C}\mathbf{x}^0 = 0
$$

```python
C.dot(null_basis.T)
```

    array([[ 0.0000000e+00],
           [-6.9388939e-18],
           [-6.9388939e-18]])

## SVD for fewer equations than unknowns
$N$ equations for $M$ unknowns with $N < M$:

* no unique solutions (underdetermined)
* $M-N$ dimensional family of solutions
* SVD: at least $M-N$ zero or negligible $w_j$: columns of $\mathsf{V}$ corresponding to singular $w_j$ span the solution space when added to a particular solution.

Same as the above [**Solving ill-conditioned coupled linear equations**](#Solving-ill-conditioned-coupled-linear-equations).

## SVD for more equations than unknowns
$N$ equations for $M$ unknowns with $N > M$:

* no exact solutions in general (overdetermined)
* but: SVD can provide best solution in the least-square sense
  $$
  \mathbf{x} = \mathsf{V}\, \text{diag}(1/w_j)\, \mathsf{U}^{T}\, \mathbf{b}
  $$
  where

  * $\mathbf{x}$ is a $M$-dimensional vector of the unknowns (parameters of the fit),
  * $\mathsf{V}$ is a $M \times N$ matrix
  * the $w_j$ form a square $M \times M$ matrix,
  * $\mathsf{U}$ is a $M \times N$ matrix (and $\mathsf{U}^T$ is a $N \times M$ matrix), and
  * $\mathbf{b}$ is the $N$-dimensional vector of the given values (data)

It can be shown that $\mathbf{x}$ minimizes the residual

$$
\mathbf{r} := |\mathsf{A}\mathbf{x} - \mathbf{b}|.
$$

where the matrix $\mathsf{A}$ will be described below and will contain the evaluation of the fit function for each data point in $\mathbf{b}$.

(For a $N \le M$, one can find $\mathbf{x}$ so that $\mathbf{r} = 0$ – see above.)

(In the following, we will switch notation and denote the vector of $M$ unknown parameters of the model as $\mathbf{a}$; this $\mathbf{a}$ corresponds to $\mathbf{x}$ above. $N$ is the number of observations.)

### Linear least-squares fitting

This is the *linear least-squares fitting problem*: Given $N$ data points $(x_i, y_i)$ (where $1 \le i \le N$), fit to a linear model $y(x)$, which can be any linear combination of $M$ functions of $x$.

For example, if we have $N$ functions $x^k$ with parameters $a_k$

$$
y(x) = a_1 + a_2 x + a_3 x^2 + \dots + a_M x^{M-1}
$$

or in general
$$
y(x) = \sum_{k=1}^M a_k X_k(x)
$$

The goal is to determine the $M$ coefficients $a_k$.

Define the **merit function**
$$
\chi^2 = \sum_{i=1}^N \left[ \frac{y_i - \sum_{k=1}^M a_k X_k(x_i)}{\sigma_i}\right]^2
$$
(sum of squared deviations, weighted with standard deviations $\sigma_i$ on the $y_i$).

Best parameters $a_k$ are the ones that *minimize $\chi^2$*.

*Design matrix* $\mathsf{A}$ ($N \times M$, $N \geq M$), vector of measurements $\mathbf{b}$ ($N$-dim) and parameter vector $\mathbf{a}$ ($M$-dim):

\begin{align}
A_{ij} &= \frac{X_j(x_i)}{\sigma_i}\\
b_i &= \frac{y_i}{\sigma_i}\\
\mathbf{a} &= (a_1, a_2, \dots, a_M)
\end{align}

The design matrix $\mathsf{A}$ contains the *predicted* values from the basis functions for all values $x_i$ of the independent variable $x$ for which we have measured data $y_i$.

Minimum occurs when the derivative vanishes:
$$
0 = \frac{\partial\chi^2}{\partial a_k} = \sum_{i=1}^N {\sigma_i}^{-2} \left[ y_i - \sum_{j=1}^M a_j X_j(x_i) \right] X_k(x_i), \quad 1 \leq k \leq M
$$
($M$ coupled equations)

To simplify the notation, define the $M \times M$ matrix
\begin{align}
\alpha_{kj} &= \sum_{i=1}^N \frac{X_k(x_i) X_j(x_i)}{\sigma_i^2}\\
\mathsf{\alpha} &= \mathsf{A}^T \mathsf{A}
\end{align}
and the vector of length $M$
\begin{align}
\beta_{k} &= \sum_{i=1}^N \frac{y_i X_k(x_i)}{\sigma_i^2}\\
\boldsymbol{\beta} &= \mathsf{A}^T \mathbf{b}
\end{align}

Then the $M$ coupled equations can be compactly written as
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \boldsymbol{\beta}
\end{align}

$\mathsf{\alpha}$ and  $\boldsymbol{\beta}$ are known, so we have to solve this matrix equation for the vector of the unknown parameters $\mathbf{a}$.

#### Error estimates for the parameters

The inverse of $\mathsf{\alpha}$ is related to the uncertainties in the parameters:
$$
\mathsf{C} := \mathsf{\alpha}^{-1}
$$
in particular
$$
\sigma(a_i) = C_{ii}
$$
(and the $C_{ij}$ are the co-variances).

#### Solution of the linear least-squares fitting problem with SVD
We need to solve the overdetermined system of $M$ coupled equations
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \boldsymbol{\beta}
\end{align}
We can solve the above equation with SVD.

SVD finds $\mathbf{a}$ that minimizes
$$
\chi^2 = |\mathsf{A}\mathbf{a} - \mathbf{b}|
$$
(proof in _Numerical Recipes_ Ch 2.) and so SVD is suitable to solve the equation above.

We can alternatively use SVD to directly solve the overdetermined system
$$
\mathsf{A}\mathbf{a} = \mathbf{b}
$$
and we should get the same answer.

The first approach might be preferrable because the matrix $\mathsf{\alpha}$ is a relatively small $M \times M$ matrix, i.e., its size depends on the number of parameters. The design matrix $\mathsf{A}$ is a $N \times M$ matrix and can be large (in one dimensions) because its size depends on the possibly large number $N$, the number of data points.

The errors are
$$
\sigma^2(a_j) = \sum_{i=1}^{M} \left(\frac{V_{ji}}{w_i}\right)^2
$$
(see also _Numerical Recipes_ Ch. 15) where $V_{ji}$ are elements of $\mathsf{V}$.

#### Example
Synthetic data

$$
y(x) = 3\sin x - 2\sin 3x + \sin 4x
$$

with noise $r$ added (uniform in range $-5 < r < 5$).

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import numpy as np

# use a seed for reproducible random numbers
rng = np.random.default_rng(seed=432)
```

```python
def signal(x, noise=0):
    r = rng.uniform(-noise, noise, len(x))
    return 3*np.sin(x) - 2*np.sin(3*x) + np.sin(4*x) + r
```

```python
X = np.linspace(-10, 10, 500)
Y = signal(X, noise=5)
```

```python
plt.plot(X, Y, 'r-', label="with noise")
plt.plot(X, signal(X, noise=0), 'k--', label="signal")
plt.xlabel("x")
plt.ylabel("y")
plt.legend(loc="best")
```

Define our fit function (the model) and the basis functions. We need the basis functions for setting up the problem and we will later use the fitfunction together with our parameter estimates to compare our fit to the true underlying function.

```python
def fitfunc(x, a):
    return a[0]*np.cos(x) + a[1]*np.sin(x) + \
           a[2]*np.cos(2*x) + a[3]*np.sin(2*x) + \
           a[4]*np.cos(3*x) + a[5]*np.sin(3*x) + \
           a[6]*np.cos(4*x) + a[7]*np.sin(4*x)

def basisfuncs(x):
    return np.array([np.cos(x), np.sin(x),
                     np.cos(2*x), np.sin(2*x),
                     np.cos(3*x), np.sin(3*x),
                     np.cos(4*x), np.sin(4*x)])
```

(Note that we could have used the `basisfuncs()` in `fitfunc()` – left as an exercise for the keen reader...)

Set up the $\mathsf{\alpha}$ matrix and the $\boldsymbol{\beta}$ vector (here we assume that all observations have the same error $\sigma = 1$):

```python
M = 8
sigma = 1.
alpha = np.zeros((M, M))
beta = np.zeros(M)
for x in X:
    Xk = basisfuncs(x)
    for k in range(M):
        for j in range(M):
            alpha[k, j] += Xk[k]*Xk[j]
for x, y in zip(X, Y):
    beta += y * basisfuncs(x)/sigma
```

Finally, solving the problem follows the same procedure as before:

Get the SVD:

```python
U, w, VT = np.linalg.svd(alpha)
V = VT.T
```

In this case, the singular values do not immediately show if any basis functions are superfluous (this would be the case for values close to 0).

```python
w
```

    array([296.92809624, 282.94804954, 243.7895787 , 235.7300808 ,
           235.15938555, 235.14838812, 235.14821093, 235.14821013])

... nevertheless, remember to routinely mask any singular values or close to singular values:

```python
w_inv = 1/w
w_inv[np.abs(w) < 1e-12] = 0
alpha_inv = V.dot(np.diag(w_inv)).dot(U.T)
```

Solve the system of equations with the pseudo-inverse:

```python
a_values = alpha_inv.dot(beta)
print(a_values)
```

    [-0.22257673  3.10015845 -0.03028998  0.22929961  0.14870762 -1.96639174
      0.24356561  1.24631138]

Compare the fitted values to the original parameters $a_j = 0, +3, 0, 0, 0, -2, 0, +1$.

The original parameters show up as 3.10, -1.97 and 1.24 but the other parameters also have appreciable values. Given that the noise was sizable, this is not unreasonable.

Compare the plot of the underlying true function ("signal", dashed line) to the model ("fit", solid line):

```python
plt.plot(X, fitfunc(X, a_values), 'b-', label="fit")
plt.plot(X, signal(X, noise=0), 'k--', label="signal")
plt.legend(loc="best", fontsize="small")
```

We get some spurious oscillations but overall the result looks reasonable.

```python

```

#### Direct calculation
Instead of solving the compact $M \times M$ matrix equation $\mathsf{\alpha}\mathbf{a} = \boldsymbol{\beta}$, we can try to directly solve the overdetermined $M \times N$ equation

$$
\mathsf{A}\mathbf{a} = \mathbf{b}
$$

Creating the design matrix is straight-forward (but because of the way that `basisfuncs()` returns values, we need to transpose the output to get the proper $N \times M$ matrix $\mathsf{A}$):

```python
A = np.transpose(basisfuncs(X))
b = Y
```

```python
A.shape
```

    (500, 8)

Calculate the pseudo-inverse $\mathsf{A}^{-1}$.

Note that we need to explicitly construct the matrix with the inverses of the singular values by filling a $M \times N$ matrix with $\text{diag}(w_i)$.

```python
U, w, VT = np.linalg.svd(A)
V = VT.T
singular_values = np.abs(w) < 1e-12
winv = 1/w
winv[singular_values] = 0
winvmat = np.zeros((V.shape[0], U.shape[0]))
winvmat[:len(winv), :len(winv)] = np.diag(winv)
Ainv = V.dot(winvmat).dot(U.T)
```

The singular values are all well behaved:

```python
w
```

    array([17.23160167, 16.8210597 , 15.61376248, 15.35350386, 15.33490742,
           15.33454884, 15.33454306, 15.33454304])

```python
V.shape, winvmat.shape, U.T.shape
```

    ((8, 8), (8, 500), (500, 500))

```python
Ainv.shape
```

    (8, 500)

```python
A.shape
```

    (500, 8)

Now solve directly
$$
\mathsf{A}^{-1} \mathbf{b} = \mathbf{a}
$$

```python
a = Ainv.dot(b)
a
```

    array([-0.22257673,  3.10015845, -0.03028998,  0.22929961,  0.14870762,
           -1.96639174,  0.24356561,  1.24631138])

The parameter estimates are the same as above.

```python
a_values - a
```

    array([-7.21644966e-16,  8.88178420e-16, -7.21644966e-16, -2.19269047e-15,
           -1.60982339e-15, -3.33066907e-15,  0.00000000e+00,  2.44249065e-15])

and hence the plot looks the same:

```python
plt.plot(X, fitfunc(X, a_values), 'b-', label="fit")
plt.plot(X, signal(X, noise=0), 'k--', label="signal")
plt.legend(loc="best", fontsize="small")
```

```python

```


## Actuator Control Allocation Example

When controlling physical systems like spacecraft or automobiles, the controller commands a desired control input $w \in \mathbb{R}^m$, which might be a wrench vector, containg forces and torques ($m=6$).

Usually, multiple atuators are available to produce this control input. The vector $u \in \mathbb{R}^n$ contains the single actuator input values in its components. If $n > m$ and the actuators are in general position, we call the system "over-actuated", since there are many realizations of $u$ that result in the same value of $w$ via the linear mapping $A$.

Having this freedom of choice, we want to minimize energy consumption, modeled as $\kappa^T | u |$ (with $\kappa \geq 0$), while discouraging rapid changes of the actuation values, i.e., $\lambda^\mathrm{sm} \Vert u-u^\mathrm{prev} \Vert_2^2$ with $\lambda^\mathrm{sm} \geq 0$ and $u^\mathrm{prev}$ being the actuation of the previous time step. Given the bounds $u^\mathrm{min}$ and $u^\mathrm{max}$ $(u^\mathrm{min} \leq u^\mathrm{max})$ on $u$, there might be cases when the desired control input $w$ is infeasible. Hence, we only softly penalize deviations between desired and actual control input with the cost term $\Vert A u - w \Vert_2^2$. We solve the optimization problem

\begin{equation}
\begin{array}{ll}
\text{minimize} \quad &\Vert A u - w \Vert_2^2  + \lambda^\mathrm{sm} \Vert u-u^\mathrm{prev} \Vert_2^2 + \kappa^T | u |\\
\text{subject to} \quad &u^\mathrm{min} \leq u \leq u^\mathrm{max},
\end{array}
\end{equation}

with variable $u \in \mathbb{R}^n$. To make the problem [DPP-compliant](https://www.cvxpy.org/tutorial/advanced/index.html#disciplined-parametrized-programming), we introduce the additional variable $\Delta u = u - u^\mathrm{prev}$ and solve

\begin{equation}
\begin{array}{ll}
\text{minimize} \quad &\Vert A u - w \Vert_2^2  + \lambda^\mathrm{sm} \Vert \Delta u \Vert_2^2 + \kappa^T | u |\\
\text{subject to} \quad &u^\mathrm{min} \leq u \leq u^\mathrm{max} \\
&\Delta u = u-u^\mathrm{prev}.
\end{array}
\end{equation}

Let's define the corresponding CVXPY problem.

```python
import cvxpy as cp

# define dimensions
n, m = 8, 3

# define variables
u = cp.Variable(n, name='u')
delta_u = cp.Variable(n, name='delta_u')

# define parameters
A = cp.Parameter((m, n), name='A')
w = cp.Parameter(m, name='w')
lamb_sm = cp.Parameter(nonneg=True, name='lamb_sm')
kappa = cp.Parameter(n, nonneg=True, name='kappa')
u_prev = cp.Parameter(n, name='u_prev')
u_min = cp.Parameter(n, name='u_min')
u_max = cp.Parameter(n, name='u_max')

# define objective
objective = cp.Minimize(cp.sum_squares(A@u-w) + lamb_sm*cp.sum_squares(delta_u) + kappa@cp.abs(u))

# define constraints
constraints = [u_min <= u, u <= u_max, delta_u == u-u_prev]

# define problem
problem = cp.Problem(objective, constraints)
```

Assign parameter values and solve the problem. In this case, the wrench vector $w = \left[f^T t\right]^T$ consists of force $f \in \mathbb{R}^2$ and torque $t \in \mathbb{R}$ in two-dimensional space. The control input $u \in \mathbb{R}^8$ contains four pairs of horizontal and vertical forces at 4 positions in the plane.

```python
import numpy as np

A.value = np.array([[1, 0, 1, 0, 1, 0, 1, 0],
                    [0, 1, 0, 1, 0, 1, 0, 1],
                    [1, -1, 1, 1, -1, 1, -1, -1]])

w.value = np.array([1, 1, 1])
lamb_sm.value = 0.5
kappa.value = 0.1*np.ones(8)
u_prev.value = np.zeros(8)
u_min.value = -np.ones(8)
u_max.value = np.ones(8)

val = problem.solve()
```

Generating C source for the problem is as easy as:

```python
from cvxpygen import cpg

cpg.generate_code(problem, code_dir='actuator_code')
```

Now, you can use a python wrapper around the generated code as a custom CVXPY solve method.

```python
from actuator_code.cpg_solver import cpg_solve
import numpy as np
import pickle
import time

# load the serialized problem formulation
with open('actuator_code/problem.pickle', 'rb') as f:
    prob = pickle.load(f)

# assign parameter values
prob.param_dict['A'].value = np.array([[1, 0, 1, 0, 1, 0, 1, 0],
                                       [0, 1, 0, 1, 0, 1, 0, 1],
                                       [1, -1, 1, 1, -1, 1, -1, -1]])

prob.param_dict['w'].value = np.array([1, 1, 1])
prob.param_dict['lamb_sm'].value = 0.5
prob.param_dict['kappa'].value = 0.1*np.ones(8)
prob.param_dict['u_prev'].value = np.zeros(8)
prob.param_dict['u_min'].value = -np.ones(8)
prob.param_dict['u_max'].value = np.ones(8)

# solve problem conventionally
t0 = time.time()
# CVXPY chooses eps_abs=eps_rel=1e-5, max_iter=10000, polish=True by default,
# however, we choose the OSQP default values here, as they are used for code generation as well
val = prob.solve(eps_abs=1e-3, eps_rel=1e-3, max_iter=4000, polish=False)
t1 = time.time()
print('\nCVXPY\nSolve time: %.3f ms' % (1000 * (t1 - t0)))
print('Objective function value: %.6f\n' % val)

# solve problem with C code via python wrapper
prob.register_solve('CPG', cpg_solve)
t0 = time.time()
val = prob.solve(method='CPG')
t1 = time.time()
print('\nCVXPYgen\nSolve time: %.3f ms' % (1000 * (t1 - t0)))
print('Objective function value: %.6f\n' % val)
```

Optimization results look as follows:

```python
from visualization.actuator import create_animation
from IPython.display import Image

create_animation(prob, 'actuator_animation')

with open('actuator_animation.gif', 'rb') as f:
    display(Image(f.read()))
```


# Continuous Signals

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Elementary Operations

Operations like superposition, temporal shifting and scaling are used to construct signals with a more complex structure than the previously introduced [standard signals](standard_signals.ipynb). A set of elementary operations are introduced that are frequently used in signal processing.

### Superposition

The weighted superposition $x(t)$ of two signals $x_\text{A}(t)$ and $x_\text{B}(t)$ is given as

\begin{equation}
x(t) = A \cdot x_\text{A}(t) + B \cdot x_\text{B}(t)
\end{equation}

with the complex weights $A, B \in \mathbb{C}$.

**Example**

The following example illustrates the superposition of two harmonic signals $x_\text{A}(t) = A \cdot \cos(\omega_\text{A} t)$ and  $x_\text{B}(t) = B \cdot \cos(\omega_\text{B} t)$ with weights $A$, $B$ and angular frequencies $\omega_\text{A}$ and $\omega_\text{B}$.

```python
import sympy as sym
%matplotlib inline
sym.init_printing()

t = sym.symbols('t', real=True)

A = .3
omA = 3
B = .5
omB = 5

x = A*sym.cos(omA*t) + B*sym.cos(omB*t)

sym.plot(x, (t, -5, 5), ylim=[-1.2, 1.2], ylabel=r'$x(t)$')
```

    <sympy.plotting.plot.Plot at 0x10fdbc0f0>

### Temporal Shift

The temporal shift of a signal $x(t)$ by the time $\tau$ is a frequently applied operation in signal processing. For instance, to model the propagation of signals from an actuator to a sensor.

The temporally shifted signal $x(t)$ is defined as

\begin{equation}
y(t) = x(t-\tau)
\end{equation}
with $\tau \in \mathbb{R}$. The signal $x(t)$ is

* shifted to the right (*delayed*) for $\tau > 0$
* shifted to the left (*leading*) for $\tau < 0$

**Example**

In order to illustrate the temporal shifting of signals, the construction of a staircase signal by a superposition of shifted [rectangular signals](standard_signals.ipynb#Rectangular-Signal) is considered

\begin{equation}
x(t) = \text{rect}\left(t - \frac{1}{2} \right) + \frac{2}{3} \cdot \text{rect}\left(t-\frac{3}{2} \right) + \frac{1}{3} \cdot \text{rect} \left(t-\frac{5}{2} \right)
\end{equation}

```python
rect = sym.Heaviside(t + 1/2) - sym.Heaviside(t - 1/2)
x = rect.subs(t, t-1/2) + 2/3*rect.subs(t, t-3/2) + 1/3*rect.subs(t, t-5/2)

sym.plot(x, (t, -1, 5), ylim=[-0.2, 1.2], ylabel='$x(t)$');
```

**Exercise**

* Add another step to the beginning of the staircase signal by modifying above example.

### Temporal Scaling

The temporal scaling of a signal $x(t)$ is defined as

\begin{equation}
y(t) = x(a \cdot t)
\end{equation}

with $a \in \mathbb{R}$. The signal $x(t)$ is

* stretched for $0 < a < 1$
* compressed for $a > 1$
* time-reversed and scaled for $a < 0$

An application of temporal scaling in signal processing is the adaption of the time scale for standard signals and the modeling of the [Doppler effect](https://en.wikipedia.org/wiki/Doppler_effect).

**Example**

The following example illustrates the temporal scaling of the staircase signal $y(t) = x(a \cdot t)$ introduced in the previous example. The original $x(t)$ is plotted in gray, the scaled signal $y(t)$ in blue. Here stretching is realized, such that $y(t)$ is twice as long as $x(t)$.

```python
a = .5
y = x.subs(t, a*t)

px = sym.plot(x, (t, -3, 7), ylim=[-0.2, 1.2],
              ylabel=r'$y(t)$', show=False, line_color='gray')
py = sym.plot(y.subs(a, 1/2), (t, -3, 7),
              ylim=[-0.2, 1.2], ylabel=r'$y(t)$', show=False)
px.extend(py)
px.show()
```

**Exercise**

* Modify above example such that the signal is compressed.
* Modify above example such that the signal is scaled and time reversed. What scaling factors `a` lead to  stretching/compression in this context?

### Temporal Flipping

The temporal flipping of a signal $x(t)$ is defined as

\begin{equation}
y(t) = x(\tau - t)
\end{equation}

for $\tau \in \mathbb{R}$. As $x(\tau - t) = x(- (t - \tau))$ the flipping operation can also be represented as a time-reversal of the signal $x(t)$ followed by a shift of $\tau$ of the reversed signal. For $\tau = 0$ this results in only a time-reversal of the signal.

The temporal flipping operation can be interpreted geometrically as a mirroring of the signal $x(t)$ at the vertical axis $t=\frac{\tau}{2}$.

**Example**

The following example illustrates the temporal flipping $y(t) = x(\tau - t)$ of the staircase signal $x(t)$ introduced before.

```python
tau = -1
y = x.subs(t, tau - t)

px = sym.plot(x, (t, -5, 5), ylim=[-0.2, 1.2],
              ylabel=r'$y(t)$', show=False, line_color='gray')
py = sym.plot(y, (t, -5, 5), ylim=[-0.2, 1.2], ylabel=r'$y(t)$', show=False)
px.extend(py)
px.show()
```

**Excercise**

* For what value $\tau$ does the signal start at $t=0$?
* Realize the temporal flipping by splitting it into two consecutive operations: (i) time-reversal and (ii) temporal shift.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 Sigurd Angenent.

# The logistic equation and the leap frog method

The logistic differential equation is a diffeq of the form
$$
\frac{dx}{dt} = f(x)
$$
where
$$
f(x) = x(1-x).
$$
The leap frog method for approximating solutions to this diffeq begins by replacing the derivative $dx/dt$ by a centered difference:
$$
\frac{dx}{dt} \approx \frac{x(t+\Delta t) - x(t-\Delta t)}{2\Delta t}.
\tag{1}
$$
If we write $x_n = x(n\Delta t)$ for the successive approximations, then the $x_n$ satisfy

\begin{equation}
\frac{x_{n+1} - x _ {n-1}}{2\Delta t} = f(x_n)
\implies
x _ {n+1} = x _ {n-1} + 2f(x_n).
\tag{2}
\label{eq:leapfrog}
\end{equation}
If we know $x_0$ and $x_1$ then this method tells us how to find $ x_2, x_3, \dots$.  Here $x_0$ should be a given as initial data.  We have to somehow come up with $x_1$.  Below we will use the midpoint rule to approximate $x_1$.

The symmetric difference in (1) is a better approximation of the derivative than the forward difference
$$
\frac{dx}{dt} \approx \frac{ x(t+\Delta t) - x(t)}{\Delta t}
$$
which is used in the derivation of Euler's method.  So you would think that the leap frog method is more accurate than the Euler method.  But the leap frog method turns out to be problematic.  In this notebook we compare the solutions of the leap frog method with those provided by the mid point rule.

### The simulation

```
#  To begin the simulation we import the necessary pyplot stuff
import numpy
import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

Next, we define a function that returns the right hand side of the logistic ODE, and also a function that performs one step of the leapfrog method.

```
def f(x):
    """the right hand side in the logistic differential equation"""
    return 3*x*(1.0-x)
function_formula="$3x-x^2$"  #  this string goes in the title of our figure

def leapfrog(u0,u1,f,dt):
    """ Compute the solution at the next time step
    by applying the leap frog method to the previous two known solution points. """
    return u0 + 2*dt*f(u1)
```

We need the midpoint method to estimate $x_1$, and we will also use it to compare the leapfrog solution with a solution we trust.

```
def midpoint_step(u, f, dt):
    """Returns the solution at the next time-step using the midpoint method.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
    u_n_plus_1 : array of float
        approximate solution at the next time step.
    """
    umid = u+dt*f(u)/2
    return u + dt * f(umid)
```

At this point we begin the computation.  We will compute the solution over a time interval $0\lt t\lt T$, taking $N$ steps.

We store the results of the leap frog and midpoint methods in two arrays ```ulp``` and ```umdp```

```
# the parameters
N=200
T=20.0
dt = T/N
u0 = 0.02  # initial value of the solution

# create arrays with indices 0, 1, 2, ..., N:
# they must have N+1 entries since we include 0.
times = numpy.linspace(0.0, T, N+1)
ulp = numpy.zeros(N+1)
umdp = numpy.zeros(N+1)

# assign initial data
ulp[0] = u0
umdp[0] = u0

```

```
# Compute umdp[1] and use it as u1 for the leap frog method

umdp[1] = midpoint_step(u0, f, dt)
ulp[1] = umdp[1]

# Compute the remaining solution points
for i in range(2,N+1):
    umdp[i] = midpoint_step(umdp[i-1], f, dt)
    ulp[i] = leapfrog(ulp[i-2], ulp[i-1], f, dt)
```

```
# now we plot
# visualization of the path
plt.figure(figsize=(12,6))
plt.grid(True)
plt.xlabel(r't', fontsize=18)
plt.ylabel(r'u', fontsize=18)
plt.title("Approximate solutions of $3x-x^2$.     Time step: $\Delta t=$%.3f "%(dt))
plt.plot(times, ulp, 'r-', lw=1, label='Leap frog')
plt.plot(times, umdp, 'g-', lw=2, label='Mid point rule')
# uncomment the following two lines to plot the
# even entries and the odd entries in the leapfrog approximation
#plt.plot(times[0::2], ulp[0::2], 'y-', lw=3)
#plt.plot(times[1::2], ulp[1::2], 'm-', lw=3)
plt.legend(loc='upper left');
```

### What is going on?

While the solution of the leapfrog method oscillates a lot, it looks like it oscillates between two nice graphs (you can see them by uncommenting two lines in the code above.)

To see what is happening, take the sequence of leap-frog values
$$
\dots, x _ {n-2}, x _ {n-1}, x _ {n}, x _ {n+1}, x _ {n+2}, \dots
$$
and separate the even from the odd numbered entries:  let
$$
p _ k = x _ {2k}, \qquad q _ k = x _ {2k+1}.
$$
So,
$$
x_0, x_1, x_2, x_3, x_4, \dots =
p_0, q_0, p_1, q_2, p_3, \dots
$$
Write the leap frog equation \eqref{eq:leapfrog} twice: once for $n$ even, and once for $n$ odd.  You get
\begin{align*}
x _ {2k} &= x _ {2k-2} + 2\Delta t f(x _ {2k-1}) &
\implies p _ k &= p _ {k-1} + 2\Delta t f(q _ {k-1})\\
x _ {2k+1} &= x _ {2k-1} + 2\Delta t f(x _ {2k}) &
\implies q _ k &= q _ {k-1} + 2\Delta t f(p _ k)
\end{align*}
The equations for $p _ k$  and $q_k$ are almost the Euler method for the system of differential equations
\begin{align*}
p' &= f(q)\\
q' &= f(p)
\end{align*}
where $p_k\approx p(2k\Delta t)$ and $q_k\approx q(2k\Delta t)$.  The leap frog method isn't solving one differential equation $x'=f(x)$, but a system of two diffeqs $p'=f(q)$, $q'=f(p)$!  As long as $p=q$ the system of diffeqs reduces to the original single equation, but apparently there are points where a small difference between $p$ and $q$ gets amplified, and the leap frog method shows us a solution of $p'=f(q), q'=f(p)$ with $p\neq q$.

### Suggestions

Play with the parameters: what happens when you double $N$?  If you change the initial value?

Try different functions $f$.

---

######The cell below loads the style of the notebook.

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>