# Solving systems of linear equations

Consider a general system of $m$ linear equations with $n$ unknowns (variables):

$$ a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n = b_1 \\
a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n = b_2 \\
    \vdots \qquad \qquad \vdots \\
a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n = b_m, $$

where $x_i$ are unknowns, $a_{ij}$ are the coefficients of the system and $b_i$ are the RHS terms. These can be real or complex numbers.

Almost every problem in linear algebra will come down to solving such a system. But what does it mean to solve a system of equations?

**Solution of the system.** Solving a system of equations means to find *all* n-tuples $(x_1, x_2, ..., x_n)$ such that substituting each one back into the system gives exactly those values given on the RHS, in the correct order. We call each such tuple a solution of the system of equations.

To solve such a system we will use basic arithmetic operations to transform our problem into a simpler one. That is, we will transform our system into another **equivalent system**. We say that two systems of equations are equivalent if they have the same set of solutions. In other words, the transformed system is equivalent to the original one if the transformation does not cause a solution to be lost or gained.

Three such transformations (sometimes called *elementary row operations*) of a system of linear equations that result in an equivalent system are:

1. Swapping any two equations of the system
2. Multiplying an equation of the system by any number different from 0
3. Adding an equation of the system multiplied by a scalar to another equation of the system

We will aim to use these transformations to eliminate certain unknowns from some equations. Ideally, we would like to reduce one equation to have only one unknown, which we can then simply solve for. Then we plug this value in other equations and so on. Let us demonstrate this on a couple of simple examples.

### Example: Unique solution

Consider the following system of 3 linear equations involving 3 unknowns $x, y, z$:

$$ x + z = 0 \\
y - z = 1 \\
x + 2y + z = 1 $$

Being a system of 3 equations and 3 unknowns, we should be able to solve it. We start by noticing that if we subtract the 1st equation from the 3rd we would be left with an equation involving only $y$. That is, we need to use the transformation rule number 3. After doing that, we get the following equivalent system:

$$ x + z = 0 \\
y - z = 1 \\
2y = 1. $$

Now we can easily see from the 3rd equation that $y = 1/2$. Now we plug this value of $y$ into the 2nd equation and solve it for $z$. Then we plug the value of $z$ into the 1st equation and solve it for $x$. After doing that, we find that the only solution to the problem is a triplet $(1/2, 1/2, -1/2)$.

## Matrix equation

We can represent any system of linear equation in matrix form. The general $m \times n$ system from the beginning of this notebook can be represented as:

$$ A \mathbf{x} = \mathbf{b}, $$

where $A \in \mathbb{c}^{m \times n}$ is called a **coefficient matrix** with coefficients as entries $a_{ij}$ and $\mathbf{x}$ and $\mathbf{b}$ are vectors $\in \mathbb{R}^n$.

The same transformation rules from before still apply to a systems represented in matrix form, where each row is one equation. Since these transformations are performed on both the LHS and RHS of equations it is convenient to write the system using an **augmented matrix** which is obtained by appending $\mathbf{b}$ to $A$:

$$ (A | \mathbf{b}) =
\left ( \begin{array}{cccc|c}
a_{11} & a_{12} & \cdots & a_{1n} & b_1 \\
a_{21} & a_{22} & \cdots & a_{2n} & b_2 \\
    & \vdots & & \vdots & \\
a_{m1} & a_{m2} & \cdots & a_{mn} & b_m \end{array} \right ).$$

To avoid confusing $A$ and $\mathbf{b}$, they are often separated by a straight line, as shown above.

### Example: No solution

Consider a very similar problem to the one in the previous example, which only differs in the sign of $a_{33}$ in the 3rd equation:

$$ x + z = 0 \\
y - z = 1 \\
x + 2y - z = 1 $$

Let us first write it in matrix-form $A \mathbf{x} = \mathbf{b}$:

$$ \begin{pmatrix}
1 & 0 & 1 \\
0 & 1 & -1 \\
1 & 2 & -1 \end{pmatrix}
\begin{pmatrix} x \\ y \\ z \end{pmatrix} =
\begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} $$

Or using an augmented matrix:

$$ \left ( \begin{array}{ccc|c}
1 & 0 & 1 & 0 \\
0 & 1 & -1 & 1 \\
1 & 2 & -1 & 1
\end{array} \right ) $$

We will again aim to eliminate certain coefficients from equations. For example, we could eliminate the first coefficient in the 3rd row by subtracting the 1st row from the 3rd row. By doing that, we get the equivalent system:

$$\begin{aligned}
    \left ( \begin{array}{ccc|c}
1 & 0 & 1 & 0 \\
0 & 1 & -1 & 1 \\
1 & 2 & -1 & 1
\end{array} \right )
    \hspace{-0.5em}
    \begin{align}
        &\phantom{I}\\
        &\phantom{II} \\
        &L_3 - L_1 \to L_3
    \end{align}
    \Rightarrow
\left ( \begin{array}{ccc|c}
1 & 0 & 1 & 0 \\
0 & 1 & -1 & 1 \\
0 & 2 & -2 & 1
\end{array} \right )
\end{aligned} $$

Now we want to eliminate the second coefficient in the 3rd equation. To do that, we use the third transformation rule again to subtract $2 \times$ 2nd equation from the 3rd:

$$\begin{aligned}
\left ( \begin{array}{ccc|c}
1 & 0 & 1 & 0 \\
0 & 1 & -1 & 1 \\
0 & 2 & -2 & 1
\end{array} \right )
    \hspace{-0.5em}
    \begin{align}
        &\phantom{I}\\
        &\phantom{II} \\
        & L_3 - 2L_1 \to L_3
    \end{align}
    \Rightarrow
\left ( \begin{array}{ccc|c}
1 & 0 & 1 & 0 \\
0 & 1 & -1 & 1 \\
0 & 0 & 0 & -1
\end{array} \right )
\end{aligned} $$

Let us look at the third equation: $0 = -1$. What this equation is telling us is that if a solution exists, that solution would be such that $0 = -1$. Since this is obviously not true, we conclude that there is no solution of this system of equations. Or, more precisely, we found that the solution set of this system is an **[empty set](https://en.wikipedia.org/wiki/Empty_set)**.

## Vector equation

Remember that a product of matrix-vector multiplication is a linear combination of the columns of the matrix. We can therefore write $A\mathbf{x} = \mathbf{b}$ as:

$$ x_1 \begin{pmatrix} \\ a_1 \\ \\ \end{pmatrix}
+ x_2 \begin{pmatrix} \\ a_2 \\ \\ \end{pmatrix}
+ \cdots + x_n \begin{pmatrix} \\ a_n \\ \\ \end{pmatrix}
= \begin{pmatrix} \\ b \\ \\ \end{pmatrix} $$

Therefore, solving $A \mathbf{x} = \mathbf{b}$ can be thought of as finding weights $x_1, ..., x_n$ such that the above is true.

## Triangular systems

Some special types of systems of linear equations can be solved very easily. An especially important type is a triangular system.

A square $n \times n$ matrix $A$ is a **lower triangular matrix** if $a_{ij} = 0$ for $i < j$. Similarly, we say that it is **upper triangular** if $a_{ij} = 0$ for $i > j$. For example,

$$\begin{pmatrix} 1 & 0 & 0 \\ 1 & 1 & 0 \\ 1 & 1 & 1 \end{pmatrix}, \quad
\begin{pmatrix} 0 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix}, \quad
\begin{pmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 1 \end{pmatrix}.$$

The first two matrices above are lower-triangular because $a_{12}=a_{13}=a_{23} = 0$ and the third one is upper-triangular because $a_{21}=a_{31}=a_{32}=0$.

A system of linear equations which has a triangular coefficient matrix is called a *triangular system*. If you look back at the examples above, you will see that the transformations performed were actually helping us reach a triangular form of the coefficient matrix. Indeed, often the easiest way to solve a system of linear equations will be to transform it into an equivalent triangular system.

### Example: Upper-triangular system

Here we will demonstrate why triangular systems of equations are very simple to solve. Consider the following upper-triangular system:

$$ \begin{pmatrix}
1 & -1 & 2 \\
0 & 2 & -1 \\
0 & 0 & 2 \end{pmatrix}
\begin{pmatrix} x \\ y \\ z \end{pmatrix} =
\begin{pmatrix} -1 \\ 3 \\ 2 \end{pmatrix} $$

Solving an $n \times n$ triangular system comes down to solving, in $n$ steps, one equation with one unknown. So we should be able to solve the above system in just 3 steps. We begin solving an upper-triangular system by solving the last equation, which in our case is the following equation with one unknown: $ 2z = 2 $. Clearly, $z=1$ and $z$ is no longer an unknown variable. Now we work our way up and solve the 2nd equation, plugging in our unique solution of $z$: $2y -z = 2y - 1 = 3$ which is again an equation with one unknown. We find that $y = 2$ which we then plug in the first equation: $ x - y + 2z = x - 2 + 2 = x = -1 $. We have successfully solved the system and we found that it has a unique solution $\mathbf{x} = (-1, 2, 1)$.

If the system is lower-triangular, we  start solving it from the 1st equation and work our way down to the last one.

### Trapezoidal matrix

We can generalise the idea of triangular matrices to non-square matrices. A non-square matrix with zero entries below or above the diagonal is called an upper or lower **trapezoidal matrix**. For example:

$$ \begin{pmatrix} 1 & 2 & 2 & 2 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 2\end{pmatrix}, \quad
\begin{pmatrix} 1 & 0 & 0 & 0 & 0 \\ 2 & 1 & 0 & 0 & 0 \\ 2 & 2 & 1 & 0 & 0 \end{pmatrix} $$

What can we say about such system of equations $A \mathbf{x} = \mathbf{b}$ where $A \in \mathbb{C}^{m \times n}$ and $\mathbf{x}$, $\mathbf{b} \in \mathbb{C}^n$? If $m < n$ there are fewer equations than there are unknowns so the system is **undertermined**. If $m > n$ there are more equations than unknown and the system is **overdetermined**.

#### Example: Underdetermined system

Let us consider the following system of 2 equations and 3 unknowns:

$$ x - y + 2z = -1 \\
2y - z = 3 $$

We begin from the 2nd equation since it involves less unknowns than the 1st equation: $ 2y - z = 3$. Here we have a choice of which variable to solve for, but we shall solve it for $y$ since it is the *leading variable* (first non-zero in the row). We find $y = (z + 3)/2$. $z$ is a *free variable*, which we introduce formally bellow, but it essentially means that $z$ can be any number $z \in \mathbb{C}$, say $z = t$. Substituting $y = (t + 3)/2$ into the 1st equation:

$$ x = y - 2z - 1 = (t + 3)/2 - 2t - 1 = -\frac{3t}{2} + \frac{1}{2} $$

Therefore our solution set in terms of $z$ is $\{(-\frac{3t}{2} + \frac{1}{2}, \frac{t + 3}{2}, t), t \in \mathbb{c}\}$.

## Existence and uniqueness of a solution

Let us think geometrically what it means for a system $A \mathbf{x} = \mathbf{b}$ to have a solution. Consider a simple system of linear equations:

$$ \begin{bmatrix} 2 & 3 \\ 1 & -4 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 7 \\ 3 \end{bmatrix}, $$

or, equivalently,

$$ 2x + 3y = 7 \\ x - 4y = 3. $$

Let us plot these two lines using Python:

```python
import numpy as np
import matplotlib.pyplot as plt

x1 = np.linspace(0, 8, 10)
y1 = (7 - 2 * x1) / 3

x2 = np.linspace(0, 8, 10)
y2 = (x2 - 3) / 4

plt.plot(x1, y1, label=r"$2x + 3y = 7$")
plt.plot(x2, y2, label=r"$x - 4y = 3$")
plt.xlim(1.5, 5)
plt.ylim(-1, 1.5)
plt.xlabel('x')
plt.ylabel('y')
plt.legend(loc='best')
plt.show()
```

The two lines cross at only one point, at point $ (37/11, 1/11)$. This point is the *unique solution* of the system, $\mathbf{x} = (37/11, 1/11)$.

Let's now consider two systems of two equations whose graphs are parallel lines:

$$ 2x + 3y = 7  \qquad 2x + 3y = 7 \\ 2x + 3y = 5 \qquad 4x + 6y = 14$$

and let us plot them.

```python
x2 = np.linspace(0, 2.5, 10)
y2 = (5 - 2 * x2) / 3

fig, ax = plt.subplots(1, 2, figsize=(10, 4), sharey=True)

ax[0].plot(x1, y1, label=r"$2x + 3y = 7$")
ax[0].plot(x2, y2, label=r"$2x + 3y = 5$")
ax[0].set_aspect('equal')
ax[0].set_xlabel('x')
ax[0].set_ylabel('y')
ax[0].legend(loc='best')

y2 = (14 - 4 * x1) / 6
ax[1].plot(x1, y1, label=r"$2x + 3y = 7$")
ax[1].plot(x1, y2, 'y--', label=r"$4x + 6y = 14$")
ax[1].set_aspect('equal')
ax[1].set_xlabel('x')
ax[1].legend(loc='best')

plt.setp(ax, xlim=(0, 3.7), ylim=(0, 2.5))
plt.show()
```

In the first case, the lines never cross because they are parallel. Therefore, the system of equations has no solution. We can write $\mathbf{x} \in \emptyset$ (empty set).

The lines are parallel in the second case as well, but now they are on top of each other. There are an infinite number of solutions to that system - every point on the line $ y = (7-2x)/3 $ is a solution. We can then write that the solution set is $ \{ (t, \frac{7-2t}{3}), t \in \mathbb{R})\} $.

We can conclude that the existence and uniqueness of the solution of a linear system depends on whether the lines are parallel or not.

Let us test this by finding the cross products of the row-vectors of the coefficient matrices. The row-vectors are normal to the lines given by the equations, so if the equation graphs are parallel, their normals will be too.

```python
from matplotlib.patches import Polygon

x2 = np.linspace(0, 8, 10)
y2 = (x2 - 3) / 4

fig, ax = plt.subplots(1, 2, figsize=(10, 6))

ax[0].plot(x1, y1, label=r"$2x + 3y = 7$")
ax[0].plot(x2, y2, label=r"$x - 4y = 3$")
ax[0].quiver(37/11, 1/11, 2, 3, scale=15, angles='xy', color='b')
ax[0].quiver(37/11, 1/11, 1, -4, scale=15, angles='xy', color='orange')
vertices = np.array([[0, 0], [2, 3], [3, -1], [1, -4]])/4.3 + np.array([37/11, 1/11])
ax[0].add_patch(Polygon(vertices , facecolor='lightblue', alpha=0.7))
ax[0].set_xlim(1.5, 5)
ax[0].set_ylim(-1, 1.5)
ax[0].set_aspect('equal')
ax[0].set_xlabel('x')
ax[0].set_ylabel('y')
ax[0].legend(loc='best')

y2 = (5 - 2 * x2) / 3
ax[1].plot(x1, y1, label=r"$2x + 3y = 7$")
ax[1].plot(x2, y2, label=r"$2x + 3y = 5$")
ax[1].quiver(1.3, 4.4/3, 2, 3, scale=15, angles='xy', color='b')
ax[1].quiver(1.5, 2/3, 2, 3, scale=15, angles='xy', color='orange')
ax[1].set_xlim(0, 3.7)
ax[1].set_ylim(0, 2.5)
ax[1].set_aspect('equal')
ax[1].set_xlabel('x')
ax[1].legend(loc='best')

plt.show()
```

If $\mathbf{a_1} = (a_{11}, a_{12})$ is the first row-vector of a coefficient matrix and $\mathbf{a_2} = (a_{21}, a_{22})$, we can express their cross product as:

$$ ( \mathbf{a_1} \times \mathbf{a_2} ) = |\mathbf{a_1}| |\mathbf{a_2}| \sin(\vartheta) \hat{n}, $$

where $|\cdot|$ denotes the magnitude of the vector, $\vartheta$ is the angle between $\mathbf{a_1}$ and $\mathbf{a_2}$ and $\hat{n}$ is the unit normal vector to both vectors. We can see that the magnitude of the vector is equal to the area of a parallelogram with sides $|\mathbf{u}|$ and $|\mathbf{v}|$, with $\vartheta$ controlling how skewed it is. Such a parallelogram is marked by light-blue on the left figure. We also know from the previous notebook that the magnitude of the cross product of the two rows or columns of the matrix is given by the determinant of the matrix:

$$ |(a_{11}, a_{12}) \times (a_{21}, a_{22}) | = |(a_{11}, a_{21}) \times (a_{12}, a_{22}) | = \det \begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix}.$$

Let us then conclude, and we will justify this in the next notebooks, that a system of linear equations $A \mathbf{x} = \mathbf{b}$ will have a unique solution iff $\det A \neq 0$. If $\det A = 0$, the system either has infinitely-many solutions or no solutions at all.

# Gaussian elimination

Let us finally formally introduce what we have been trying to achieve in most examples above. **Gaussian elimination** (or row reduction) uses the three transformations mentioned before to reduce a system to **row echelon form**. A matrix is in row echelon form if:

- all zero rows (if they exist) are below all non-zero rows
- the **leading coefficient** (or **pivot**; the first non-zero entry in a row) is always strictly to the right of the leading coefficient of the row above it. That is, for two leading elements $a_{ij}$ and $a_{kl}$: if $i < k$ then it is required that $j < l$.

Notice that these conditions require reduced echelon form to be an upper-trapezoidal matrix. For example:

$$ \begin{pmatrix} 1 & 2 & 0 & 1 \\ 0 & 1 & 0 & 5 \\ 0 & 0 &2 & 2 \end{pmatrix}, \quad
\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}, \quad
\begin{pmatrix} 1 & -2 & 0 & 1 \\ 0 & 0 & 1 & 3 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0\end{pmatrix}$$

The motivation behind performing Gaussian elimination is therefore clear, as we have demonstrated in the upper-triangular example why triangular systems are most convenient for solving systems of linear equations.

Let us consider again a general coefficient matrix $A \in \mathbb{C}^{m \times n}$ in $A \mathbf{x} = \mathbf{b}$:

$$ \begin{pmatrix} a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
a_{21} & a_{22} & a_{23} & \cdots & a_{2n} \\
a_{31} & a_{32} & a_{33} & \cdots & a_{3n} \\
 & \vdots & & \vdots & \\
a_{m1} & a_{m2} & a_{m3} & \cdots & a_{mn} \end{pmatrix} $$

What we want to achieve is to have all entries in the 1st column be 0 except for the first one. Then in the 2nd column all entires below the second entry should be 0. In the 3rd column all entries below the third entry should be 0, and so on. That means that we will have to use the 3rd transformation rule and subtract one equation from every one below it, multiplied such that the column entries will cancel:

$$
\begin{aligned}
    &\begin{pmatrix} a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
    a_{21} & a_{22} & a_{23} & \cdots & a_{2n} \\
    a_{31} & a_{32} & a_{33} & \cdots & a_{3n} \\
     & \vdots & & \vdots & \\
    a_{m1} & a_{m2} & a_{m3} & \cdots & a_{mn} \end{pmatrix}
    \hspace{-0.5em}
    \begin{align}
        &\phantom{L_1}\\
        &L_2 - ^{a_{21}}/_{a_{11}}L_1 \to L_2 \\
        &L_3 - ^{a_{31}}/_{a_{11}}L_1 \to L_3 \\
        &\qquad \cdots \\
        &L_m - ^{a_{m1}}/_{a_{11}}L_1 \to L_1
    \end{align} \\ \\
    \Rightarrow \quad
    &\begin{pmatrix} a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
    0 & \tilde{a_{22}} & \tilde{a_{23}} & \cdots & \tilde{a_{2n}} \\
    0 & \tilde{a_{32}} & \tilde{a_{33}} & \cdots & \tilde{a_{3n}} \\
     & \vdots & & \vdots & \\
    0 & \tilde{a_{m2}} & \tilde{a_{m3}} & \cdots & \tilde{a_{mn}} \end{pmatrix}
    \hspace{-0.5em}
    \begin{aligned}
        &\phantom{L_1}\\
        &\phantom{L_2} \\
        &L_3 - ^{a_{32}}/_{a_{12}}L_1 \to L_3 \\
        &\qquad \cdots \\
        &L_m - ^{a_{m2}}/_{a_{12}}L_1 \to L_1
    \end{aligned} \\ \\
    \Rightarrow \quad
    &\begin{pmatrix} a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
    0 & \tilde{a_{22}} & \hat{a_{23}} & \cdots & \hat{a_{2n}} \\
    0 & 0 & \hat{a_{33}} & \cdots & \hat{a_{3n}} \\
     & \vdots & & \vdots & \\
    0 & 0 & \hat{a_{m3}} & \cdots & \hat{a_{mn}} \end{pmatrix} \\
    & \qquad \dots
\end{aligned} $$

And so on. Notice that these operations (transformations) are performed on entire rows, so the entries in the entire row change (here denoted by tilde and hat). That is why we need to be careful what row we add to other row since, for example, adding the first row to other rows later on would re-introduce non-zero values in the first column which we previously eliminated.

### Example

Consider the following system of 3 equations and 3 unknowns $x, y, z$:

$$ x - y + 2z = -1 \\ x + 2y - z = 2 \\ -x + y + z = 0 $$

Let us write it in augmented-matrix form and perform Gaussian eliminations.

$$\begin{aligned}
    &\left ( \begin{array}{ccc|c}
        1 & -1 & 2 & -1 \\
        1 & 2 & -1 & 2 \\
        -1 & 1 & 1 & 0 \end{array} \right )
    \hspace{-0.5em}
    \begin{align}
        &\phantom{L_1}\\
        &L_2 - L_1 \to L_2 \\
        &L_3 + L_1 \to L_3 \\
    \end{align} \\ \\
    \Rightarrow \quad
    &\left ( \begin{array}{ccc|c}
        1 & -1 & 2 & -1 \\
        0 & 3 & -3 & 3 \\
        0 & 0 & 3 & -1 \end{array} \right )
\end{aligned}$$

The one transformation on the 3rd equation eliminated both $x$ and $y$ unknowns from it so we did not have to perform another transformation. Now the system is reduced to an upper-triangular one, which we have encountered before and know how to solve. We begin from the last equation and back-substitute found values as we work our way up. We leave it to the reader to confirm that there is a unique solution $\mathbf{x} = (1/3, 2/3, -1/3)$.

## Transformations as matrices

Recall the example on permutations a few notebooks ago, where we wrote each permutation of rows or columns as a permutation matrix multiplying the original matrix. We can do the same thing with elementary row transformations.

Consider the same example from above, where we performed the following 2 transformations on the square $3 \times 3$ matrix $A$:

1. subtracted row 1 from row 2; $L_2 - L_1 \to L_2$
2. added row 1 to row 3; $L_3 + L_1 \to L_3$

We can write both of these transformations using **elementary matrices**:

$$ E_1 = \begin{pmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}, \quad E_2 = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{pmatrix} $$

As in the example of permutations, left-multiplication by an elementary matrix $E$ represents elementary row operations, while right-multiplication represents elementary column operations. Our transformations are row operations, so we left multiply our original matrix:

$$ E_2 E_1 A =
\begin{pmatrix} 1 & -1 & 2 \\ 0 & 3 & -3 \\ 0 & 0 & 3 \end{pmatrix},$$

resulting in the same upper-triangular matrix as before. Note that the order in which we multiply is, in general, important.

# Using the inverse matrix to solve a linear system

Remember that an inverse of a square matrix $A$ is $A^{-1}$ such that $AA^{-1} = A^{-1}A = I$. Therefore, if we have a matrix equation $A \mathbf{x} = \mathbf{b}$, finding an inverse $A^{-1}$ would allow us to solve for all unknowns simultaneously, rather than in steps like in the examples above. Here is the idea:

$$ A\mathbf{x} = \mathbf{b} \\
\mbox{multiply both sides by} A^{-1} \\
I \mathbf{x} = A^{-1}\mathbf{b} \\
\mathbf{x} = A^{-1}\mathbf{b} $$

Finding the solution $\mathbf{x}$ is then reduced to a matrix-vector multiplication $A^{-1}\mathbf{b}$.

## Gauss-Jordan elimination

**Gauss-Jordan elimination** is a type of Gaussian elimination that we can use to find the inverse of a matrix. This process is based on reducing a matrix whose inverse we want to find to **reduced row echelon form**. For a matrix to be in reduced row echelon form it must satisfy the two conditions written above for row echelon forms and an additional condition:

- Each leading (pivot) element is 1 and all other entries in their columns are 0.

Examples of such matrices are:

$$ \begin{pmatrix} 1 & 0 & 0\\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}, \quad
\begin{pmatrix} 1 & 0 & 4\\ 0 & 1 & 3 \end{pmatrix}, \quad
\begin{pmatrix} 0 & 0 & 1 & 5 & 0 & 2 & 0 \\ 0 & 0 & 0 & 0 & 1 & 6 & 0 \\ 0 & 0 & 0 &0 &0 & 0 & 1 \end{pmatrix} $$

Now, let $A \in \mathbb{R}^{n \times n}$ be a square matrix whose inverse we want to find. We use it to form an augmented block matrix $[A | I]$. We perform elementary row transformations on this matrix such that we reduce $A$ to reduced row echelon form, which we will denote as $A_R$. In this process, the identity matrix $I$ on the right will be transformed to some new matrix $B$. If $A$ is invertible, we will have:

$$ [A | I] \Rightarrow [A_R | B] = [I | A^{-1}] $$

Let us think why this is. We showed above that elementary row operations can be represented as elementary matrices. Let there be $k$ such transformations needed to reduce $A$ to $A_R$. Then we can write:

$$ [A_R | B] = E_k E_{k-1} \dots E_2 E_1[A | I] $$

and let us denote $S = E_k E_{k-1} \dots E_2 E_1$, the product of all $k$ elementary matrices. Now,

$$ [A_R | B] = S[A | I] = [SA | SI] = [SA | S]$$

Therefore, if $A_R = I \Rightarrow I = SA$, meaning that $S = B$ is indeed the inverse of $A$.

If $A$ is not invertible, remember that it means that it is not full-rank. What that means is that $A_R$ will have at least one zero-row. In that case, $B \neq A^{-1}$.

### Example: Calculating an inverse of a matrix

Let us find the inverse matrix of:

$$ A = \begin{bmatrix} 2 & 1 & 3 \\ 0 & 2 & -1 \\ 3 & -1 & 2 \end{bmatrix}. $$

We begin by forming an augmented matrix $[A|I]$ and begin reducing $A$ to reduced row echelon form.

$$\begin{aligned}
    {[A | I] =} \quad &\left [ \begin{array}{ccc|ccc}
        2 & 1 & 3 & 1 & 0 & 0 \\
        0 & 2 & -1 & 0 & 1 & 0\\
        3 & -1 & 2 & 0 & 0 & 1 \end{array} \right ]
    \hspace{-0.5em}
    \begin{aligned}
        &^1/_2 L_1 \to L_1 \\
        &\phantom{L} \\
        &\phantom{L} \\
    \end{aligned} \\ \\
    \sim \quad
    &\left [ \begin{array}{ccc|ccc}
        1 & ^1/_2 & ^3/_2 & ^1/_2 & 0 & 0 \\
        0 & 2 & -1 & 0 & 1 & 0\\
        3 & -1 & 2 & 0 & 0 & 1 \end{array} \right ]
    \hspace{-0.5em}
    \begin{aligned}
        &\phantom{L} \\
        &\phantom{L} \\
        &L_3 -3L_1 \to L_3 \\
    \end{aligned} \\ \\
    \sim \quad
    &\left [ \begin{array}{ccc|ccc}
        1 & ^1/_2 & ^3/_2 & ^1/_2 & 0 & 0 \\
        0 & 2 & -1 & 0 & 1 & 0\\
        0 & -^5/_2 & -^5/_2 & -^3/_2 & 0 & 1 \end{array} \right ]
    \hspace{-0.5em}
    \begin{aligned}
        &\phantom{L} \\
        & ^1/_2 L_2 \to L_2\\
        &\phantom{L} \\
    \end{aligned} \\ \\
    \sim \quad
    &\left [ \begin{array}{ccc|ccc}
        1 & ^1/_2 & ^3/_2 & ^1/_2 & 0 & 0 \\
        0 & 1 & -^1/_2 & 0 & ^1/_2 & 0\\
        0 & -^5/_2 & -^5/_2 & -^3/_2 & 0 & 1 \end{array} \right ]
    \hspace{-0.5em}
    \begin{aligned}
        & L_1 -^1/_2 L_2 \to L_1 \\
        & \phantom{L}\\
        & L_3 +^5/_2 L_2 \to L_3 \\
    \end{aligned} \\ \\
    \sim \quad
    &\left [ \begin{array}{ccc|ccc}
        1 & 0 & ^7/_4 & ^1/_2 & -^1/_4 & 0 \\
        0 & 1 & -^1/_2 & 0 & ^1/_2 & 0\\
        0 & 0 & -^{15}/_4 & -^3/_2 & ^5/_4 & 1 \end{array} \right ]
    \hspace{-0.5em}
    \begin{aligned}
        & \phantom{L} \\
        & \phantom{L}\\
        & -^4/_{15}L_3 \to L_3 \\
    \end{aligned} \\ \\
    \sim \quad
    &\left [ \begin{array}{ccc|ccc}
        1 & 0 & ^7/_4 & ^1/_2 & -^1/_4 & 0 \\
        0 & 1 & -^1/_2 & 0 & ^1/_2 & 0\\
        0 & 0 & 1 & ^2/_5 & -^1/_3 & -^4/_{15} \end{array} \right ]
    \hspace{-0.5em}
    \begin{aligned}
        & L_1 -^7/_4 L_3 \to L_1 \\
        & L_2 +^1/_2 L_3 \to L_2\\
        & \phantom{L} \\
    \end{aligned} \\ \\
    \sim \quad
    &\left [ \begin{array}{ccc|ccc}
        1 & 0 & 0 & -^1/_5 & ^1/_3 & ^7/_{15} \\
        0 & 1 & 0 & ^1/_5 & ^1/_3 & -^2/_{15}\\
        0 & 0 & 1 & ^2/_5 & -^1/_3 & -^4/_{15} \end{array} \right ]
    = [I | B]
\end{aligned}$$

We successfully found the inverse $A^{-1} = B$! Let us now use it to solve the system $A \mathbf{x} = \mathbf{b}$, where $\mathbf{b} = (2, 0, 1)^T$. As explained before, $A$ on the LHS is eliminated by multiplying both sides by $A^{-1}$ on the left, leaving:

$$ \mathbf{x} = A^{-1}\mathbf{b} =
\begin{pmatrix}
-^1/_5 & ^1/_3 & ^7/_{15} \\
^1/_5 & ^1/_3 & -^2/_{15} \\
^2/_5 & -^1/_3 & -^4/_{15}
\end{pmatrix}
\begin{pmatrix} 2 \\ 0 \\ 1 \end{pmatrix} =
\begin{pmatrix} 1/15 \\ 4/15 \\ 8/15 \end{pmatrix} $$

The reader is encouraged to confirm this is the correct solution either through substitution into the original system or by using another solution method.

```python

```


# Grafico funciones simbolicas

En este tutorial graficaremos una función simbolica sencilla

```python
# Importar paquetes
import sympy as sym
import matplotlib.pyplot as plt
import numpy as np
```

```python
# Configurar funcion #1
a = 0.2
b = 0.2
x = sym.Symbol('x')
f_x = sym.sin(a*x) + sym.cos(b*x)
print("Funcion:")
f_x
```

    Funcion:

$\displaystyle \sin{\left(0.2 x \right)} + \cos{\left(0.2 x \right)}$

```python
# Configurar funcion #2
# Curva S-Shape

t = sym.Symbol('t')
f_t = 1/(1 + sym.exp(-t))
print("Funcion Sigmoide: ")
f_t
```

    Funcion Sigmoide:

$\displaystyle \frac{1}{1 + e^{- t}}$

## Grafico con Matplotlib

A continuación computaremos los valores numericos de la función simbolica
para crear un grafico usando matplotlib

```python
# Convertir la función simbolica para usar Numpy
num_f_x = sym.lambdify(x, f_x, modules=['numpy'])
# Crear un conjunto de valores entre [0, 100] en intervalos de 10
x_vals = np.linspace(-np.pi, np.pi, 100)
# Computar los valores de la función y
y_vals = num_f_x(x_vals)

# Funcion #2
num_f_t = sym.lambdify(t, f_t, modules=['numpy'])
# Crear un conjunto de valores entre [0, 100] en intervalos de 10
x2_vals = np.linspace(-100, 100)
# Computar los valores de la función y
y2_vals = num_f_t(x2_vals)
```

```python
# Configurar el layout
fig = plt.figure() # Figura principal
layout = fig.add_gridspec(1,2)

# Configuración del primer plano
trig = fig.add_subplot(layout[0,0])
trig.plot(x_vals, y_vals) # Graficar valores x, f(x)
trig.set_ylabel(f_x) # Configurar la leyenda del grafico
trig.set_xlabel('radianes')
trig.set_title('Funciones trigonmetricas')

# Configuración del segundo plano
trig2 = fig.add_subplot(layout[0,1])
trig2.plot(x2_vals, y2_vals) # Graficar valores x, f(x)
trig2.set_ylabel(f_t) # Configurar la leyenda del grafico
trig2.set_xlabel('Valores')
trig2.set_title('S-Shape')

# Mostrar graficas
plt.show()
```


## CS164 ASSIGNMENT 2: CONSTRAINED OPTIMIZATION & THE KKT CONDITIONS

#### Vinícius Miranda

The assignment instructions are available [here](https://course-resources.minerva.kgi.edu/uploaded_files/mke/00087691-7466/assignment2.pdf).

We are concerned with the linear program

$$
\begin{equation*}
\begin{aligned}
& \underset{x}{\text{min}}
& & c^Tx \\
& \text{subject to:}
& & Ax \leq b
\end{aligned}
\end{equation*}
$$

where $A \in \mathbb{R}^{m\times n}, b \in \mathbb{R}^{m\times 1}$ and $c\in \mathbb{R}^{n\times 1}$. Its KKT conditions are

$$
\begin{align}
\text{Primal Feasibility : } &A\textbf{x}^* \preceq b \\
\text{Dual Feasibility : } &\begin{cases} c + A^T\lambda = \textbf{0} \\
                            \lambda \succeq \textbf{0} \end{cases}  \\
\text{Complementary Slackness : } &\lambda^T(A\textbf{x}^*-b) = \textbf{0}
\end{align}
$$

where dual feasilibity is also known as the stationary condition and $\lambda \in \mathbb{R}^{m\times 1}$. Note that $\preceq$ and $\succeq$ are component-wise (vectorized) versions of $\leq$ and $\geq$. There are three possibilities for the solution of this LP (Calafiore XXXX). They are
1. If the intersection among the half-spaces defined by each of the $m$ constraints is empty, then the feasible set is the empty set and there is no solution.
2. If the feasible set is nonempty and unbounded, then the problem may or may not attain a finite solution. The geometric interpretation is that if one direction of descent of the objective function is unbounded in the feasible set, the minimum is never found as the function decreases infinitely.
3. If the feasible set is nonempty and bounded, the feasible is a polyhedron (or polytope) and the solution is attained at one of its vertices. The proof below follows from Calafiore (XXXX).

Let $\mathcal{X} = \{x\in\mathbb{R}^n : Ax\leq b\}$. A point $x^*$ is a solution if and only if $c^Tx\geq c^Tx^*, \forall x \in \mathcal{X}$. In other words, if all other points in the feasible set have higher objectives. Similarly, we may say that if $x^*$ is optimal, then $c^T(x-x^*)\geq0, \forall x \in \mathcal{X}$, which is to say that all directions in the feasible set are non-descent directions. Notice that for $f(x)=c^Tx, \nabla f=c$. If $x^*$ were not a boundary point of the feasible set,  one would need only step in the direction $-c$ to reduce the objective. Furthermore, when $\exists x \in \mathcal{X}$ such that $x \neq x^*$ and $c^T(x-x^*)=0$, there is a collection of points that extremize the objective. These points are all in a face (or facets) of the polyhedron (or polytope), which occurs when the level curves (or sets) of the objective function are parallel to the face.

## $l_1$ and $l_\infty$ Regression

The $l_1$ and $l_\infty$ norms can be used as cost functions to find the line-of-best-fit for a regression problem. We can thus define the problems

$$\min_{\Theta}||Y-X\Theta||_1$$

and

$$\min_{\Theta}||Y-X\Theta||_{\infty}$$

where $X \in \mathbb{R}^{N\times2}$ for a dataset of $N$ points in $\mathbb{R}^2$, $Y \in \mathbb{R}^{N\times1}$, and $\Theta \in \mathbb{R}^{2\times1}$.

### $l_1$ norm

The $l_1$ norm computes the sum of the absolute values of each component of a vector. In other words,

$$\min_{\Theta}||Y-X\Theta||_1=\min_{\Theta}\sum_{i=1}^N|Y_i-X_i\Theta|.$$

We can then introduce slack variables $u_i$ which will provide bounds to the absolute values. By minimizing over the slack variables, we will always find the tighest nonnegative bound and hence the linear program remains the same. The LP then becomes

$$
\begin{equation*}
\begin{aligned}
& \underset{\Theta, u}{\text{min}}
& & \sum_{i=1}^N u_i \\
& \text{subject to:}
& & |Y_i-X_i\Theta| \leq u_i, i=1,\dots,N
\end{aligned}
\end{equation*}
$$

which we can express in standard form as

$$
\begin{equation*}
\begin{aligned}
& \underset{\Theta, u}{\text{min}}
& & \textbf{1}^T\textbf{u} \\
& \text{subject to:}
& & Y_i-X_i\Theta -u_i \leq 0, i=1,\dots,N \\
& & & X_i\Theta-Y_i- u_i \leq 0, i=1,\dots,N
\end{aligned}
\end{equation*}
$$

We thus introduce $N$ slack variables and $2\times N$ constraints.

### $l_\infty$ norm

We proceed similarly with the $l_\infty$ norm. However, we are now interested only in the largest absolute value of all the components of the vector over which we minimize. Therefore, only one slack variable is necessary to provide a bound on this value. As before, as we minimize the slack variable, we find the tighest bound possible, equivalent to the value of the infinity norm, and thus the LPs are equivalent. Therefore,

$$
\begin{equation*}
\begin{aligned}
& \underset{\Theta, t}{\text{min}}
& & t\\
& \text{subject to:}
& & ||Y-X\Theta||_{\infty} \leq t
\end{aligned}
\end{equation*}
$$

Since $t$ bounds the largest component of the vector, we also bounds all the other components. In other words,

$$||Y-X\Theta||_{\infty} \leq t \iff \max_{i=1,\dots,N} |Y_i-X_i\Theta|\leq t \iff |Y_i-X_i\Theta|\leq t, i = 1,\dots, N$$

In standard form, the LP then becomes

$$
\begin{equation*}
\begin{aligned}
& \underset{\Theta, t}{\text{min}}
& & t\\
& \text{subject to:}
& & Y_i-X_i\Theta -t \leq 0, i=1,\dots,N \\
& & & X_i\Theta-Y_i- t \leq 0, i=1,\dots,N
\end{aligned}
\end{equation*}
$$

Thus, we introduce one slack variable $t$ and $2\times N$ constraints. This section is also heavily based on Calafiore (XXXX).

Some code is given below to generate a synthetic dataset.  Using CVX, solve two linear programs for computing the regression line for $l_1$ and $l_\infty$ regression. Plot the lines over the data to evaluate the fit.

```python
# l_1 and l_infinity regression using cvxpy
import numpy as np
import cvxpy as cvx
import matplotlib.pyplot as plt

# generate a synthetic dataset

# actual parameter values
theta1_act = 2
theta2_act = 5

# Number of points in dataset
N = 200

# Noise magnitude
mag = 30

# datapoints
x = np.arange(0,N)
y = theta1_act * x + theta2_act *np.ones([1,N]) + np.random.normal(0,mag,N)

plt.figure()
# Scatter plot of data
plt.scatter(x,y)
plt.show()

```

### $l_1$ norm

```python
# Arranging the variables and checking proper dimensionality
X = np.array([x, np.ones_like(x)]).T
Y = y.T
assert X.shape == (N, 2)
assert Y.shape == (N, 1)

# HINT: you will first want to declare a variable for the parameters of the line
theta_1 = cvx.Variable(2)

# The slack variables
u = cvx.Variable(N)

# The constraints.
constraints = [Y.flatten() - X@theta_1 - u <= np.zeros(N),
               X@theta_1 - Y.flatten() - u <= np.zeros(N)]

# Form objective.
obj = cvx.Minimize(np.ones(N).T@u)

# Form and solve problem.
prob = cvx.Problem(obj, constraints)
prob.solve("ECOS")  # Returns the optimal value.
print("status:", prob.status)
print("optimal value:", prob.value)
print("optimal Θ:", theta_1.value)
```

    status: optimal
    optimal value: 5034.734558966026
    optimal Θ: [1.97202379 9.75376663]

### $l_\infty$ norm

```python
# HINT: you will first want to declare a variable for the parameters of the line
theta_infty = cvx.Variable(2)

# The slack variables
t = cvx.Variable()

# The constraints.
constraints = [Yi - Xi@theta_infty - t <= 0 for Yi, Xi in zip(y.flatten(), X)] +\
              [Xi@theta_infty - Yi - t <= 0 for Yi, Xi in zip(y.flatten(), X)]

                #[Y.flatten() - X@theta_infty - t <= np.zeros(N),
               #X@theta_infty - Y.flatten() - t <= np.zeros(N)]

# Form objective.
obj = cvx.Minimize(t)

# Form and solve problem.
prob = cvx.Problem(obj, constraints)
prob.solve("ECOS")  # Returns the optimal value.
print("status:", prob.status)
print("optimal value:", prob.value)
print("optimal Θ:", theta_infty.value)
```

    status: optimal
    optimal value: 84.00028631278876
    optimal Θ: [ 1.92832652 19.26261364]

#### Visualizing the solutions...

```python
plt.figure(figsize=[12, 6])
plt.scatter(x,y, alpha=0.5)
plt.plot(x, theta_1.value[0] * x + theta_1.value[1] *np.ones(N),
         color='red', linewidth=3, alpha=0.7, label=r"$l_1$ norm")
Theta2 = np.linalg.inv(X.T@X)@X.T@Y
plt.plot(x, Theta2[0] * x + Theta2[1] *np.ones(N),
         color='green', linewidth=3, alpha=0.5, label=r"$l_2$ norm")
plt.plot(x, theta_infty.value[0] * x + theta_infty.value[1] *np.ones(N),
         color='orange', linewidth=3, alpha=0.7, label=r"$l_\infty$ norm")

plt.title("Line of best fit for different lp-norm cost functions")
plt.legend()
plt.show()
```

Assignment 2 - Constrained Optimization & The KKT Conditions.ipynb


## Conditional Independence

Two random variable $X$ and $Y$ are conditiaonly independent given $Z$, denoted by $X \perp \!\! \perp Y \mid Z$ if

$$p_{X,Y\mid Z} (x,y\mid z) = p_{X\mid Z}(x\mid z) \, p_{Y\mid Z}(y\mid z)$$

In general Marginal independence doesn't imply conditional independence and vice versa.

### Example

R: Red Sox Game

A: Accident

T: Bad Traffic

Find the following probability

(a) $\mathbb{P}(R=1) = 0.5$

(b) $\mathbb{P}(R=1 \mid T=1)$

$$\begin{align}p_{R,A}(r,a\mid 1)
&= \frac{p_{T\mid R,A}(1 \mid r, a)\, p_R(r) \, p_A(a)}{p_T(1)}\\
&= c\cdot p_{T\mid R,A}(1 \mid r, a)\end{align}$$

(c) $\mathbb{P}(R=1 \mid T=1, A=1)= \mathbb{P}(R=1 \mid T=1)$

### Practice Problem: Conditional Independence

Suppose $X_0, \dots , X_{100}$ are random variables whose joint distribution has the following factorization:

$$p_{X_0, \dots , X_{100}}(x_0, \dots , x_{100}) = p_{X_0}(x_0) \cdot \prod _{i=1}^{100} p_{X_ i | X_{i-1}}(x_ i | x_{i-1})$$

This factorization is what's called a Markov chain. We'll be seeing Markov chains a lot more later on in the course.

Show that $X_{50} \perp \!\! \perp X_{52} \mid X_{51}$.

**Answer:**
$$
\begin{eqnarray}
		p_{X_{50},X_{51},X_{52}}(x_{50},x_{51},x_{52})
        &=& \sum_{x_{0} \dots x_{49}} \sum_{x_{53} \dots x_{100}} p_{X_0, \dots , X_{100}}(x_0, \dots , x_{100})  \\
		&=& \sum_{x_{0} \dots x_{49}} \sum_{x_{53} \dots x_{100}} \left[p_{X_0}(x_{0}) \prod_{i=0}^{50} p_{X_i\mid X_{i-1}}(x_{i}|x_{i-1})\right] \\
        && \cdot \prod_{i=51}^{52} p_{X_i\mid X_{i-1}}(x_{i}|x_{i-1})\cdot \prod_{i=53}^{100}  p_{X_i\mid X_{i-1}}(x_{i}|x_{i-1}) \\
        &=& \underbrace{\sum_{x_{0} \dots x_{49}}  \left[p_{X_0}(x_{0}) \prod_{i=0}^{50} p_{X_i\mid X_{i-1}}(x_{i}|x_{i-1})\right]}_{=p_{X_{50}}(x_{50})} \\
        && \cdot \prod_{i=51}^{52} p_{X_i\mid X_{i-1}}(x_{i}|x_{i-1})\cdot \underbrace{\sum_{x_{53} \dots x_{100}}\prod_{i=53}^{100}  p_{X_i\mid X_{i-1}}(x_{i}|x_{i-1})}_{=1} \\[2ex]
		&=& p_{X_{50}}(x_{50}) \cdot p_{X_{51}\mid X_{50}}(x_{51}|x_{50}) \cdot p_{X_{52}\mid X_{51}}(x_{52}|x_{51}) \\[2ex]
        &=&  p_{X_{50}\mid X_{51}}(x_{50}|x_{51}) \cdot p_{X_{52}\mid X_{51}}(x_{52}|x_{51}) \\[2ex]
        \frac{p_{X_{50},X_{51},X_{52}}(x_{50},x_{51},x_{52})}{p_{X_{51}}(x_{51})}
        &=& p_{X_{50}\mid X_{51}}(x_{50}|x_{51}) \cdot p_{X_{52}\mid X_{51}}(x_{52}|x_{51}) \\[2ex]
        p_{X_{50},X_{52}\mid X_{51}}(x_{50},x_{52}\mid x_{51})
        &=& p_{X_{50}\mid X_{51}}(x_{50}|x_{51}) \cdot p_{X_{52}\mid X_{51}}(x_{52}|x_{51})
\end{eqnarray}
$$

```python

```


# 16 PDEs: Waves

(See *Computational Physics* Ch 21 and *Computational Modeling* Ch 6.5.)

## Background: waves on a string

Assume a 1D string of length $L$ with mass density per unit length $\rho$ along the $x$ direction. It is held under constant tension $T$ (force per unit length). Ignore frictional forces and the tension is so high that we can ignore sagging due to gravity.

### 1D wave equation
The string is displaced in the $y$ direction from its rest position, i.e., the displacement $y(x, t)$ is a function of space $x$ and time $t$.

For small relative displacements $y(x, t)/L \ll 1$ and therefore small slopes $\partial y/\partial x$ we can describe $y(x, t)$ with a *linear* equation of motion:

Newton's second law applied to short elements of the string with length $\Delta x$ and mass $\Delta m = \rho \Delta x$: the left hand side contains the *restoring force* that opposes the displacement, the right hand side is the acceleration of the string element:

\begin{align}
\sum F_{y}(x) &= \Delta m\, a(x, t)\\
T \sin\theta(x+\Delta x) - T \sin\theta(x) &= \rho \Delta x \frac{\partial^2 y(x, t)}{\partial t^2}
\end{align}

The angle $\theta$ measures by how much the string is bent away from the resting configuration.

Because we assume small relative displacements, the angles are small ($\theta \ll 1$) and we can make the small angle approximation

$$
\sin\theta \approx \tan\theta = \frac{\partial y}{\partial x}
$$

and hence

\begin{align}
T \left.\frac{\partial y}{\partial x}\right|_{x+\Delta x} - T \left.\frac{\partial y}{\partial x}\right|_{x}  &= \rho \Delta x \frac{\partial^2 y(x, t)}{\partial t^2}\\
\frac{T \left.\frac{\partial y}{\partial x}\right|_{x+\Delta x} - T \left.\frac{\partial y}{\partial x}\right|_{x}}{\Delta x} &= \rho \frac{\partial^2 y}{\partial t^2}
\end{align}

or in the limit $\Delta x \rightarrow 0$ a linear hyperbolic PDE results:

\begin{gather}
\frac{\partial^2 y(x, t)}{\partial x^2} = \frac{1}{c^2} \frac{\partial^2 y(x, t)}{\partial t^2}, \quad c = \sqrt{\frac{T}{\rho}}
\end{gather}

where $c$ has the dimension of a velocity. This is the (linear) **wave equation**.

### General solution: waves

General solutions are propagating waves:

If $f(x)$ is a solution at $t=0$ then

$$
y_{\mp}(x, t) = f(x \mp ct)
$$

are also solutions at later $t > 0$.

Because of linearity, any linear combination is also a solution, so the most general solution contains both right and left propagating waves

$$
y(x, t) = A f(x - ct) + B g(x + ct)
$$

(If $f$ and/or $g$ are present depends on the initial conditions.)

In three dimensions the wave equation is

$$
\boldsymbol{\nabla}^2 y(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2 y(\mathbf{x}, t)}{\partial t^2} = 0\
$$

### Boundary and initial conditions

* The boundary conditions could be that the ends are fixed

  $$y(0, t) = y(L, t) = 0$$

* The *initial condition* is a shape for the string, e.g., a Gaussian at the center

  $$
  y(x, t=0) = g(x) = y_0 \frac{1}{\sqrt{2\pi\sigma}} \exp\left[-\frac{(x - x_0)^2}{2\sigma^2}\right]
  $$

  at time 0.
* Because the wave equation is *second order in time* we need a second initial condition, for instance, the string is released from rest:

  $$
  \frac{\partial y(x, t=0)}{\partial t} = 0
  $$

  (The derivative, i.e., the initial displacement velocity is provided.)

### Analytical solution
Solve (as always) with *separation of variables*.

$$
y(x, t) = X(x) T(t)
$$

and this yields the general solution (with boundary conditions of fixed string ends and initial condition of zero velocity) as a superposition of normal modes

$$
y(x, t) = \sum_{n=0}^{+\infty} B_n \sin k_n x\, \cos\omega_n t,
\quad \omega_n = ck_n,\ k_n = n \frac{2\pi}{L} = n k_0.
$$

(The angular frequency $\omega$ and the wave vector $k$ are determined from the boundary conditions.)

The coefficients $B_n$ are obtained from the initial shape:

$$
y(x, t=0) = \sum_{n=0}^{+\infty} B_n \sin n k_0 x = g(x)
$$

In principle one can use the fact that $\int_0^L dx \sin m k_0 x \, \sin n k_0 x = \pi \delta_{mn}$ (orthogonality) to calculate the coefficients:

\begin{align}
\int_0^L dx \sin m k_0 x \sum_{n=0}^{+\infty} B_n \sin n k_0 x &= \int_0^L dx \sin(m k_0 x) \, g(x)\\
\pi \sum_{n=0}^{+\infty} B_n \delta_{mn} &= \dots \\
B_m &= \pi^{-1} \dots
\end{align}

(but the analytical solution is ugly and I cannot be bothered to put it down here.)

## Numerical solution

1. discretize wave equation
2. time stepping: leap frog algorithm (iterate)

Use the central difference approximation for the second order derivatives:

\begin{align}
\frac{\partial^2 y}{\partial t^2} &\approx \frac{y(x, t+\Delta t) + y(x, t-\Delta t) - 2y(x, t)}{\Delta t ^2} = \frac{y_{i, j+1} + y_{i, j-1} - 2y_{i,j}}{\Delta t^2}\\
\frac{\partial^2 y}{\partial x^2} &\approx \frac{y(x+\Delta x, t) + y(x-\Delta x, t) - 2y(x, t)}{\Delta x ^2} = \frac{y_{i+1, j} + y_{i-1, j} - 2y_{i,j}}{\Delta x^2}
\end{align}

and substitute into the wave equation to yield the *discretized* wave equation:

$$
\frac{y_{i+1, j} + y_{i-1, j} - 2y_{i,j}}{\Delta x^2} = \frac{1}{c^2} \frac{y_{i, j+1} + y_{i, j-1} - 2y_{i,j}}{\Delta t^2}
$$

Re-arrange so that the future terms $j+1$ can be calculated from the present $j$ and past $j-1$ terms:

$$
y_{i,j+1} = 2(1 - \beta^2)y_{i,j} - y_{i, j-1} + \beta^2 (y_{i+1,j} + y_{i-1,j}), \quad
\beta := \frac{c}{\Delta x/\Delta t}
$$

This is the time stepping algorithm for the wave equation.

## Numerical implementation

```python
# if you have plotting problems, try
# %matplotlib inline
%matplotlib notebook

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

plt.style.use('ggplot')
```

### Student version

```python
L = 0.5  # m
Nx = 50
Nt = 100

Dx = L/Nx
Dt = 1e-4  # s

rho = 1.5e-2   # kg/m
tension = 150  # N

c = np.sqrt(tension/rho)

# TODO: calculate beta
beta = c*Dt/Dx
beta2 = beta**2

print("c = {0} m/s".format(c))
print("Dx = {0} m,  Dt = {1} s, Dx/Dt = {2} m/s".format(Dx, Dt, Dx/Dt))
print("beta = {}".format(beta))

X = np.linspace(0, L, Nx+1)  # need N+1!

def gaussian(x, y0=0.05, x0=L/2, sigma=0.1*L):
    return y0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2/(2*sigma**2))

# displacements at j-1, j, j+1
y0 = np.zeros_like(X)
y1 = np.zeros_like(y0)
y2 = np.zeros_like(y0)

# save array
y_t = np.zeros((Nt+1, Nx+1))

# boundary conditions
y0[0] = y0[-1] = y1[0] = y1[-1] = 0
y2[:] = y0

# initial conditions: velocity 0, i.e. no difference between y0 and y1
y0[1:-1] = y1[1:-1] = gaussian(X)[1:-1]

# save initial
t_index = 0
y_t[t_index, :] = y0
t_index += 1
y_t[t_index, :] = y1

for jt in range(2, Nt):
    y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
    y0[:], y1[:] = y1, y2

    t_index += 1
    y_t[t_index, :] = y2
    print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

### Fancy version
Package as a function and can use `step` to only save every `step` time steps.

```python
def wave(L=0.5, Nx=50, Dt=1e-4, Nt=100, step=1,
        rho=1.5e-2, tension=150.):

    Dx = L/Nx

    #rho = 1.5e-2   # kg/m
    #tension = 150  # N

    c = np.sqrt(tension/rho)

    beta = c*Dt/Dx
    beta2 = beta**2

    print("c = {0} m/s".format(c))
    print("Dx = {0} m,  Dt = {1} s, Dx/Dt = {2} m/s".format(Dx, Dt, Dx/Dt))
    print("beta = {}".format(beta))

    X = np.linspace(0, L, Nx+1)  # need N+1!

    def gaussian(x, y0=0.05, x0=L/2, sigma=0.1*L):
        return y0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2/(2*sigma**2))

    # displacements at j-1, j, j+1
    y0 = np.zeros_like(X)
    y1 = np.zeros_like(y0)
    y2 = np.zeros_like(y0)

    # save array
    y_t = np.zeros((int(np.ceil(Nt/step)) + 1, Nx+1))

    # boundary conditions
    y0[0] = y0[-1] = y1[0] = y1[-1] = 0
    y2[:] = y0

    # initial conditions: velocity 0, i.e. no difference between y0 and y1
    y0[1:-1] = y1[1:-1] = gaussian(X)[1:-1]

    # save initial
    t_index = 0
    y_t[t_index, :] = y0
    if step == 1:
        t_index += 1
        y_t[t_index, :] = y1

    for jt in range(2, Nt):
        y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
        y0[:], y1[:] = y1, y2

        if jt % step == 0 or jt == Nt-1:
            t_index += 1
            y_t[t_index, :] = y2
            print("Iteration {0:5d}".format(jt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

    return y_t, X, Dx, Dt, step

```

```python
y_t, X, Dx, Dt, step = wave()
```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

### 1D plot
Plot the output in the save array `y_t`. Vary the time steps that you look at with `y_t[start:end]`.

We indicate time by color changing.

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis(i) for i in np.linspace(0, 1, len(y_t))])
ax.plot(X, y_t[:40].T, alpha=0.5);
```

    <IPython.core.display.Javascript object>

### 1D Animation
For 1D animation to work in a Jupyter notebook, use

```python
%matplotlib notebook
```

If no animations are visible, restart kernel and execute the `%matplotlib notebook` cell as the very first one in the notebook.

We use `matplotlib.animation` to look at movies of our solution:

```python
import matplotlib.animation as animation
```

Generate one full period:

```python
y_t, X, Dx, Dt, step = wave(Nt=100)
```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

The `update_wave()` function simply re-draws our image for every `frame`.

```python
y_limits = 1.05*y_t.min(), 1.05*y_t.max()

fig1 = plt.figure(figsize=(5,5))
ax = fig1.add_subplot(111)
ax.set_aspect(1)

def update_wave(frame, data):
    global ax, Dt, y_limits
    ax.clear()
    ax.set_xlabel("x (m)")
    ax.set_ylabel("y (m)")
    ax.plot(X, data[frame])
    ax.set_ylim(y_limits)
    ax.text(0.1, 0.9, "t = {0:3.1f} ms".format(frame*Dt*1e3), transform=ax.transAxes)

wave_anim = animation.FuncAnimation(fig1, update_wave, frames=len(y_t), fargs=(y_t,),
                                    interval=30, blit=True, repeat_delay=100)

```

    <IPython.core.display.Javascript object>

If you have ffmpeg installed then you can export to a MP4 movie file:

```python
wave_anim.save("string.mp4", fps=30, dpi=300)
```

The whole video can be incoroporated into an html page (uses the HTML5 `<video>` tag).

```python
 with open("string.html", "w") as html:
    html.write(wave_anim.to_html5_video())
```

```python

```

### 3D plot
(Uses functions from previous lessons.)

```python
def plot_y(y_t, Dx, Dt, step=1):
    X, Y = np.meshgrid(range(y_t.shape[0]), range(y_t.shape[1]))
    Z = y_t.T[Y, X]
    fig = plt.figure()
    ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(Y*Dx, X*Dt*step, Z)
    ax.set_ylabel(r"time $t$ (s)")
    ax.set_xlabel(r"position $x$ (m)")
    ax.set_zlabel(r"displacement $y$ (m)")
    fig.tight_layout()
    return ax

def plot_surf(y_t, Dt, Dx, step=1, filename=None, offset=-1, zlabel=r'displacement',
             elevation=40, azimuth=20, cmap=plt.cm.coolwarm):
    """Plot y_t as a 3D plot with contour plot underneath.

    Arguments
    ---------
    y_t : 2D array
          displacement y(t, x)
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    """

    t = np.arange(y_t.shape[0])
    x = np.arange(y_t.shape[1])
    T, X = np.meshgrid(t, x)
    Y = y_t.T[X, T]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    surf = ax.plot_surface(X*Dx, T*Dt*step, Y, cmap=cmap, rstride=1, cstride=1, alpha=1)
    cset = ax.contourf(X*Dx, T*Dt*step, Y, 20, zdir='z', offset=offset+Y.min(), cmap=cmap)

    ax.set_xlabel('x')
    ax.set_ylabel('t')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Y.min(), Y.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

```python
plot_y(y_t, Dx, Dt)
```

    <IPython.core.display.Javascript object>

    <matplotlib.axes._subplots.Axes3DSubplot at 0x115b5e470>

```python
plot_surf(y_t, Dt, Dx, offset=-0.04, cmap=plt.cm.coolwarm)
```

    <IPython.core.display.Javascript object>

    <matplotlib.axes._subplots.Axes3DSubplot at 0x116361c88>

## von Neumann stability analysis: Courant condition

Assume that the solutions of the discretized equation can be written as normal modes

$$
y_{m,j} = \xi(k)^j e^{ikm\Delta x}, \quad t=j\Delta t,\ x=m\Delta x
$$

The time stepping algorith is stable if

$$
|\xi(k)| < 1
$$

Insert normal modes into the discretized equation

$$
y_{i,j+1} = 2(1 - \beta^2)y_{i,j} - y_{i, j-1} + \beta^2 (y_{i+1,j} + y_{i-1,j}), \quad
\beta := \frac{c}{\Delta x/\Delta t}
$$

and simplify (use $1-\cos x = 2\sin^2\frac{x}{2}$):

$$
\xi^2 - 2(1-2\beta^2 s^2)\xi + 1 = 0, \quad s=\sin(k\Delta x/2)
$$

The characteristic equation has roots

$$
\xi_{\pm} = 1 - 2\beta^2 s^2 \pm \sqrt{(1-2\beta^2 s^2)^2 - 1}.
$$

It has one root for

$$
\left|1-2\beta^2 s^2\right| = 1,
$$

i.e., for

$$
\beta s = 1
$$

We have two real roots for

$$
\left|1-2\beta^2 s^2\right| < 1 \\
\beta s > 1
$$

but one of the roots is always $|\xi| > 1$ and hence these solutions will diverge and not be stable.

For

$$
\left|1-2\beta^2 s^2\right| ≥ 1 \\
\beta s ≤ 1
$$

the roots will be *complex conjugates of each other*

$$
\xi_\pm = 1 - 2\beta^2s^2 \pm i\sqrt{1-(1-2\beta^2s^2)^2}
$$

and the *magnitude*

$$
|\xi_{\pm}|^2 =  (1 - 2\beta^2s^2)^2 - (1-(1-2\beta^2s^2)^2) = 1
$$

is unity: Thus the solutions will not grow and will be *stable* for

$$
\beta s ≤ 1\\
\frac{c}{\frac{\Delta x}{\Delta t}} \sin\frac{k \Delta x}{2} ≤ 1
$$

Assuming the "worst case" for the $\sin$ factor (namely, 1), the **condition for stability** is

$$
c ≤ \frac{\Delta x}{\Delta t}
$$

or

$$
\beta ≤ 1.
$$

This is also known as the **Courant condition**. When written as

$$
\Delta t ≤ \frac{\Delta x}{c}
$$

it means that the time step $\Delta t$ (for a given $\Delta x$) must be *smaller than the time that the  wave takes to travel one grid step*.

## Simplified simulation
The above example used real numbers for a cello string. Below is bare-bones where we just set $\beta$.

```python
L = 10.0
Nx = 100
Nt = 200
step = 1

Dx = L/Nx

beta2 = 1.0

X = np.linspace(0, L, Nx+1)  # need N+1!

def gaussian(x):
    return np.exp(-(x-5)**2)

# displacements at j-1, j, j+1
y0 = np.zeros_like(X)
y1 = np.zeros_like(y0)
y2 = np.zeros_like(y0)

# save array
y_t = np.zeros((int(np.ceil(Nt/step)) + 1, Nx+1))

# boundary conditions
y0[0] = y0[-1] = y1[0] = y1[-1] = 0
y2[:] = y0

# initial conditions: velocity 0, i.e. no difference between y0 and y1
y0[:] = y1[:] = gaussian(X)

# save initial
t_index = 0
y_t[t_index, :] = y0
if step == 1:
    t_index += 1
    y_t[t_index, :] = y1

for jt in range(2, Nt):
    y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
    y0[:], y1[:] = y1, y2

    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        y_t[t_index, :] = y2
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

```

    Completed   199 iterations: t=199 s

```python
%matplotlib inline
```

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis_r(i) for i in np.linspace(0, 1, len(y_t))])
ax.plot(X, y_t.T);
```


# Overview of Python’s Bitwise Operators
Bitwise operators look virtually the same across different programming languages:

|Operator	| Example	| Meaning|
|:---|:---|:---|
|&	|a & b	|Bitwise AND|
|\|	|a \| b	|Bitwise OR|
|^	|a ^ b	|Bitwise XOR (exclusive OR)|
|~	|~a	|Bitwise NOT|
|<<	|a << n	|Bitwise left shift|
|>>	|a >> n	|Bitwise right shift|

All binary bitwise operators have a corresponding **compound operator** that performs an augmented assignment:

|Operator	|Example	|Equivalent to|
|---|---|---|
|&=	|a &= b	|a = a & b|
|\|=	|a \|= b	|a = a \| b|
|^=	|a ^= b	|a = a ^ b|
|<<=	|a <<= n	|a = a << n|
|>>=	|a >>= n	|a = a >> n|

# Why Use Binary?

Most modern civilizations use **positional notation(进位制)**, which is efficient, flexible, and well suited for doing arithmetic.

- base-two numeral system (binary system), for example 0x11101
- base-eight numeral system,
- base-ten numeral system， also known as the decimal system
- base-sixteen numeral system

Pros and Cons of Binary system vs decimal system

- The binary system requires more storage space than the decimal system but is much less complicated to implement in hardware
- the binary system is perfect for electronic devices, which translate digits into different voltage levels.
- By employing only two states, you make the system more reliable and resistant to noise

```python
len("€uro".encode("utf-8"))
```

    6

```python
(42).bit_length()
```

    6

```python
for char in "€uro":
    print(char, char.encode("utf-8"), len(char.encode("utf-8")))
```

    € b'\xe2\x82\xac' 3
    u b'u' 1
    r b'r' 1
    o b'o' 1

# Bitwise Logical Operators

## Evaluating Boolean Expressions With Bitwise Operators

Although this expression is syntactically correct, there are a few problems with it.
1. it’s arguably less readable
2. it doesn’t work as expected for all groups of data

```python
# The expression made of the bitwise operators evaluates to True, while the same expression built
# from the logical operators evaluates to False. That’s because bitwise operators take precedence
# over the comparison operators, changing how the whole expression is interpreted:
# so the bit expression is interpreted as: age >= (18 & ~is_self_excluded)
age = 18
is_self_excluded = True
print(age >= 18 & ~is_self_excluded)  # Bitwise logical operators

print(age >= 18 and not is_self_excluded)  # Logical operators
```

    True
    False

```python
# It’s as if someone put implicit parentheses around the wrong operands. To fix this,
# you can put explicit parentheses, which will enforce the correct order of evaluation:
(age >= 18) & ~is_self_excluded
```

    0

However, you no longer get a Boolean result. Python bitwise operators were designed primarily to work with integers, so their operands automatically get casted if needed.

Unlike their logical counterparts, bitwise operators are evaluated eagerly
1. short-circuit evaluation of Boolean expressions: Expressions using logical operators are evaluated lazily from left to right, it will stop as soon as the result of the entire expression is known

2. For bitwise operators, even though knowing the left operand is sufficient to determine the value of the entire expression, all operands are always evaluated unconditionally.

### Bitwise AND

1. The bitwise AND operator (&) performs **logical conjunction** on the corresponding bits of its operands. The resulting bit pattern is an intersection of the operator’s arguments
2. Arithmetically, this is equivalent to a **product** of two bit values
\begin{aligned}
(a \& b)_i = a_i \times b_i
\end{aligned}
3. Notice that when operands have unequal bit-lengths, the shorter one is automatically padded with zeros to the left.
4. Alternatively, you can take the **minimum** of the two bits in each pair.

```python
a = 156
b = 52
c = a & b
print(f"a {bin(a)}, {a}")
print(f"b {bin(b)}, {b}")
print(f"c {bin(c)}, {c}")
```

    a 0b10011100, 156
    b 0b110100, 52
    c 0b10100, 20

### Bitwise OR

1. The bitwise OR operator (|) performs **logical disjunction**. For each corresponding pair of bits, it returns a one if at least one of them is switched on.
2. The resulting bit pattern is a **union** of the operator’s arguments.
3. The arithmetic behind it is a combination of a **sum** and a **product** of the bit values.
\begin{equation}
(a | b)_i = a_i + b_i - (a_i \times b_i)
\end{equation}

```python
c = a | b
print(f"a {bin(a)}, {a}")
print(f"b {bin(b)}, {b}")
print(f"c {bin(c)}, {c}")
```

    a 0b10011100, 156
    b 0b110100, 52
    c 0b10111100, 188

### Bitwise XOR
1. Unlike bitwise AND, OR, and NOT, the bitwise XOR operator (^) doesn’t have a logical counterpart in Python.
2. The name XOR stands for “exclusive or” since it performs exclusive disjunction on the bit pairs. In other words, every bit pair must contain opposing bit values to produce a one
3. Similarly to the bitwise OR operator, the arithmetic of XOR involves a sum. However, while the bitwise OR clamps values at one, the XOR operator wraps them around with a sum modulo two
\begin{equation}
(a^b)_i = (a_i + b_i) \; mod \; 2
\end{equation}

```python
def xor(a, b):
    return (a and not b) or (not a and b)
```

```python
c = a ^ b
print(f"a {bin(a)}, {a}")
print(f"b {bin(b)}, {b}")
print(f"c {bin(c)}, {c}")
```

    a 0b10011100, 156
    b 0b110100, 52
    c 0b10101000, 168

### Bitwise NOT

1. It performs **logical negation** on a given number by flipping all of its bits.
2. It can be expressed arithmetically as the subtraction of individual bit values from one
\begin{equation}
\sim a_i = 1 - a_i
\end{equation}

```python
c = ~ a
print(f"a {bin(a)}, {a}")
print(f"c {bin(c)}, {c}")
```

    a 0b10011100, 156
    c -0b10011101, -157

### Bitwise Shift Operators

1. In general, shifting bits to the left corresponds to multiplying the number by a power of two, with an exponent equal to the number of places shifted
\begin{equation}
a << n = a \times 2^n
\end{equation}
2. The left shift used to be a popular optimization technique because bit shifting is a single instruction and is cheaper to calculate than exponent or product.

```python
a = 56
print (f"{a << 1} == {a * 2^1}")
print (f"{a << 2} == {a * 2^2}")
print (f"{a << 3} == {a * 2^3}")
```

    112 == 113
    224 == 114
    448 == 115

```python
print(f"{39 << 3}")
print(f"{(39 << 3) & 0xFF}")
```

    312
    56

3. the right shift operator automatically floors the result. It’s virtually the same as a floor division by a power of two
\begin{equation}
a >> n = \lfloor \frac{a}{2^n} \rfloor
\end{equation}

```python
a = 157
print(f"a: {a}")
print(f"a >> 1: {a >> 1}")
print(f"a >> 2: {a >> 2}")
print(f"a >> 2: {a >> 2}")
```

    a: 157
    a >> 1: 78
    a >> 2: 39
    a >> 2: 39

In the following transformations:
1. The bitwise right shift operator and the floor division operator both work the same way, even for negative numbers.
2. However, the floor division lets you choose any divisor and not just a power of two.
3. Using the bitwise right shift was a common way of improving the performance of some arithmetic divisions.

```python
print(f"5 >> 1: {5 >> 1}")  # Bitwise right shift
print(f"5 // 2: {5 // 2}")  # Floor division (integer division)
print(f"5 / 2 : {5 / 2 }")  # Floating-point division
```

    5 >> 1: 2
    5 // 2: 2
    5 / 2 : 2.5

```python

```

# Reference

1. [Bitwise Operators in Python](https://realpython.com/python-bitwise-operators/)

```python

```


Consider the equation $f(x) = 0$, which $f(x)$ is one-time differentiable. Given a reasonable starting point $x_0$, we may approximate the solution of $f(x) = 0$ by linearizing $f(x)$:
$$f(x) \sim f(x_0) + f'(x_0)(x-x_0)$$
If $f(x) = 0$ then
$$f(x_0) + f'(x_0)(x-x_0) \sim 0 \Rightarrow x \sim x_0 - \frac{f(x_0)}{f'(x_0)}$$
Thus we may apply this iteratively to obtain a better solution. i.e. set
$$x_{n+1} \sim x_n - \frac{f(x_n)}{f'(x_n)},  n \geq 0$$

## Preparation

```python
%matplotlib inline
%pylab
```

    Using matplotlib backend: Qt5Agg
    Populating the interactive namespace from numpy and matplotlib

```python
import sympy as sp
x = sp.symbols('x')
f = sp.Function('f')
Df = sp.Function('Df')
```

## Code

```python
def newton_raphson(f, x0, max_iteration, tolerance):

    # Initialization
    y = x0
    ylist = [y]
    Df = sp.diff(f,x)

    # Loop
    for n in range(max_iteration):
        ynew = y - f.evalf(subs={x: y})/Df.evalf(subs={x: y})
        ylist.append(ynew)
        if abs(ynew-y) < tolerance:
            break
        else:
            y = ynew

    actual_iteration=len(ylist)-1
    print(actual_iteration)

    return ylist
```

## Testing Cell

```python
# For obtaining estimates
f = x**3-2*x-5
yran = newton_raphson(f, 2.5, 30, 10**(-16))
print(yran)
```

    6
    [2.5, 2.16417910447761, 2.09713535581055, 2.09455523239045, 2.09455148155025, 2.09455148154233, 2.09455148154233]

```python
# For seeing accuracy
estzero = [f.evalf(subs={x: i}) for i in yran]
plot(estzero)
```

M1C (Python)/M1C-Numerical Solutions to Equations I/Newton Method.ipynb


Chapter 14 of [A Guided Tour of Mathematical Methods for the Physical Sciences](https://www.cambridge.org/core/books/guided-tour-of-mathematical-methods-for-the-physical-sciences/3C5AAC644C2B09F3F17837E703516920) introduces Fourier Analysis. In it, (well-behaved) functions can be transformed into a sum of trigonometric function, whereas discrete series can be represented in so-called Fourier series. The former application helps us solve differential equations, for example, and the latter is used to explore the frequency content in a time series, among other things. Let us focus in this notebook on an application of the latter. First, we load some libraries we are going to use:

```python
import matplotlib.pyplot as plt
%matplotlib notebook
import numpy as np
import pandas as pd
```

Next, we download two years of measurements of the sea height in the Auckland Harbour. The interval between measurements is 1000 seconds:

```python
df = pd.read_csv('https://auckland.figshare.com/ndownloader/files/21944535',squeeze=True)
sea_height = df.values
dt = 1000 # in seconds
t = np.arange(len(sea_height))*dt # in seconds
```

And we plot these data:

```python
days = t/3600/24
%matplotlib notebook
plt.figure()
plt.plot(days,sea_height)
plt.xlabel('time (days)')
plt.ylabel('sea level (m)')
plt.show()
```

    <IPython.core.display.Javascript object>

Fluctuations at this time scale are for a large part due to the tides. Zoom in to explore this time series in more detail. By eye, it appears the tide is dominated by a diurnal signal with a significant amplitude variation.

### Fourier

Let us have a closer look at the signal, with Fourier analysis. We can decompose the time series $f(t)$ of length $L$ as

\begin{equation}
    f(t) = \sum_{n=-\infty }^{\infty }c_{n}\;e^{in\pi t/L},  \label{Four.16}
\end{equation}
and the coeffients $c_n$ can be calculated with
\begin{equation}
    c_{n}=\frac{1}{2L}\int_{-L}^{L}f(x)e^{-in\pi t/L}dx.
    \label{Four.18}
\end{equation}
With $f(t)$ being a time series, the $c_n$ coefficients represent its frequency spectrum, and the coefficients are spaced by $1/L$. This means that in theory the longer the time series ($L$), the greater spectral resolution.

In practice, we'll approximate $f(t)$ with L coefficients in a discrete fourier transform algorithm called the Fast Fourier Transform. In python, we have a library for that in the numpy.fft suite:

```python
cn = np.fft.fft(sea_height) # array of the (complex) Fourier coefficients
freq = np.fft.fftfreq(t.shape[-1],dt) # array of the frequencies associated with each coefficient

period = 1/freq/3600 # The period in hours (never mind the warning when it comes to the divisiont when freq=0)
```

    /home/kvan637/miniconda3/lib/python3.7/site-packages/ipykernel_launcher.py:4: RuntimeWarning: divide by zero encountered in true_divide
      after removing the cwd from sys.path.

The absolute value of the complex Fourier coefficients $c_n$ present information about the power in the signal at each frequency:

```python
plt.figure()
plt.plot(period,np.abs(cn), color='black')
plt.xlim(0,25) # zooming in on periods between 0 and 25 hours
plt.xlabel("Period (h)")
plt.show()
```

    <IPython.core.display.Javascript object>

From the absolute value of the Fourier transform of two years of sea level measurements in the Auckland Harbour, we confirm the main period in the signal is roughly 12 hours. We say "rougly", because have a closer look by zooming in on the largest peak. What do you observe?

Tides on Earth are the result of the gravitational attraction of (large) bodies of mass. The biggest contributor is the Moon, followed by the Sun. The mass of the moon is much smaller than the Sun, but its greater proximity to Earth more than makes up for that. On average, the Sun contributes 40% of the total tidal energy. It may be obvious that the Earth experiences high tide when it faces the Sun or the Moon, but there is also a high tide associated on the opposite side of Earth. The Earth spins around its axis in 24 hours, and this means we can see spikes at 12 and 24 hours due to the Sun (named $S_2$ and $S_1$, respectively). However, because our Moon rotates around the Earth in 29.5 days, the period of diurnal lunar contribution is 12.4 hours. This is called the $M_2$ period. A smaller peak at 12 hours is the diurnal solar contribution, and the 24-hr peak is the daily contribution from the Sun. In the time domain, the original plot of our data, these close but different periods result in a beat pattern in the amplitudes of sea height.

### Multi-tapering
The absolute value of the output of the FFT of a single realisation of a noisy discrete and finite time series can be a poor representation of the true power spectrum. Several methods attempt to address the uncertainties in these estimates. One of those techniques is called the multi-taper method, and [free python code exists](https://github.com/krischer/mtspec).

### Homework
Install the multitaper [MTSPEC](https://github.com/krischer/mtspec) package either here on your server in the cloud, or on your local computer. Run the multitaper method on this time series to determine which peaks in the power spectral estimate are significant, and which are not. We discussed the ~12 hr and 24 hr periods, but how about the smaller bumps between 0 and 10 hr periodicity in the figure above. Do you think they are real?

```python

```


# Fail/Pass tests and the implied failure rate and confidence levels

Let's say we conduct a fail/pass test.
We subject $n_\mathrm{s}$ samples to an accelerated (representative) life test of $m=1$ lifetime equivalents.
The test is considered a success if $100\,\%$ of the $n_\mathrm{s}$ samples survive.
Yet, this leaves the important question of how certain can we be that the population as a whole (from which the $n_\mathrm{s}$ samples are a representative sub-set) will survive the $m=1$ lifetime equivalents?
And, apart from the implied confidence level, what fraction of the population is still expected to fail even if $100\,\%$ of the $n_\mathrm{s}$ samples did survive?

We consider a binomial distribution: the samples either survive (pass), or die (fail).
Each sample has a probability $p$ of dying.
Hence, starting with $n_\mathrm{s}$ samples, the probability of ending up with $k$ dead (failed) samples is

$$
B(k) = \frac{n_\mathrm{s}!}{k!(n_\mathrm{s}-k)!} p^k (1-p)^{(n_\mathrm{s}-k)}.
$$

The case of interest is with $k=0$ dead samples at the end of the test, i.e. all passed.
This leaves us with the special case of

\begin{equation}
B(0) = (1-p)^{n_\mathrm{s}}.
\label{eq:B0}
\end{equation}

With $p$ being the probability of dying, $(1-p)=R$ can be said to be a measure of the reliability (in the sense of the probability to survive the foreseen lifetime) of the devices being tested.
The probability $B(0)$ represents a measure of how often we expect to see this survival rate.
In other words, it is $B(0)=(1-C)$, with $C$ the confidence level.

In other words, we can rewrite Eq. (\ref{eq:B0}) as
\begin{equation}
(1-C) = R^{n_\mathrm{s}}.
\label{eq:CRn}
\end{equation}

And from Eq. (\ref{eq:CRn}) follows how $n_\mathrm{s}$ relates to $R$ and $C$:

\begin{equation}
n_\mathrm{s} = \frac{\ln(1-C)}{\ln(R)}.
\label{eq:n_lnCR}
\end{equation}

We can reuse already tested samples and expose them to the accelerated life test a second, third, ... , $m$ time.
Up to a certain point this would be equivalent a higher number of samples.
Written differently,

$$
n_\mathrm{s} = m \times n_\mathrm{actual\_samples},
$$

and thus

$$
n_\mathrm{s} = \frac{\ln(1-C)}{m \ln(R)}.
$$

Sidenote, it should be clear that a test conducted with only one sample but which was exposed to $m=100$ lifetimes would NOT be statistically equally meaningful as a test with $100$ samples exposed to $m=1$ lifetime.
In what context $m>1$ might be meaningful depends on various circumstances.

The confidence level $C$ is a critical parameter of a fail/pass test.
The purpose of a fail/pass test is to learn something
(that's the purpose of any test, of course, not only fail/pass).
Often this tool comes into play to confirm that a new product can be launched.
However, testing costs money (for the samples) and time (for the actual testing).
Hence, small sample sizes are typically preferred.

Yet, confirmation bias aside, what can we possibly learn from a passed test with few samples?
Let me assume an extreme case example: testing two samples, which are so inherently flawed that they fail half of the time.
Such a test will fail in $C=75\,\%$ of the time.
Upon failure the development team would likely investigate the failed sample and subsequently improve it.
This could be considered a good outcome.
On the other hand, starting off with an only $R=50\,\%$ reliable product left a lot of room for improvement.

And maybe even worse, there are still $B(0)=25\,\%$ of the time in which the two samples do NOT fail,
$25\,\%$ of the time in which the team would not investigate the failure mechanisms, and
$25\,\%$ of the time in which the product launch would continue according to schedule and being shipped.
It should be without saying, no customer is interested in a $R=50\,\%$ reliable product.

If the product was slightly more reliable (say, a still rather bad $R=70\,\%$),
the two samples would pass in $B(0)=49\,\%$ of the cases.
In other words, the fail/pass test would be only just slightly more informative than an independent coin-flip!

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.pyplot as plt
%matplotlib inline

```

## Analytical examples

```python
C = 0.9
R = 0.9
m = 1
n_s = np.log(1-C)/(m * np.log(R))

print("""
In the case we want to be $C=90\%$ confident that the whole population shows a $R=90\%$ reliability
(i.e. at least $90\%$ of the whole will survive duration $m$),
the minimum number of samples to test is

n_s = {:.2f}
""".format(n_s))
```

    In the case we want to be $C=90\%$ confident that the whole population shows a $R=90\%$ reliability
    (i.e. at least $90\%$ of the whole will survive duration $m$),
    the minimum number of samples to test is

    n_s = 21.85

```python
m = 1
n_s = 77
R = 0.95

# (1-C) = B(0) = (1-p)^n = R^n
C = 1 - R**n_s

print("""
Given m={}, n_s={}, and an assumed reliability of R={}%, the confidence level is

C = {:.2f}%.
""".format(m, n_s, R*100, C*100))
```

    Given m=1, n_s=77, and an assumed reliability of R=95.0%, the confidence level is

    C = 98.07%.

## Numerical examples

Given $n_\mathrm{s}$ samples which each have an (assumed) intrinsic reliability $R$, how often would we run a test without any sample failing?
This rate of at least one failure represents our confidence level that we would have encountered at least one failing sample if the underlying reliability was lower than the assumed $R$.

```python
class Device:
    def __init__(self, intrinsic_reliability):
        self._R = intrinsic_reliability

    def test(self):
        # test considered passed (i.e. device survived test) if
        # probability of intrinsic reliability is larger than outcome of random event
        return int( self._R > np.random.random() )

def conduct_test(test_battery):
    # return 1 = pass: all DUTs survived
    # return 0 = fail: at least 1 DUT died
    survival = 0
    for device in test_battery:
        survival += device.test()
    return int( survival==len(test_battery) )

# parameters to explore
reliabilities = [0.5, 0.8, 0.9, 0.95, 0.98, 0.99, 0.999]
n_samples = np.arange(2,100,5) # note: I want to cover n_s=22 and n_s=77 mentioned in the text above
N = 1000 # simulate N times, with N large to be statistically meaningful, but not too large to get the computer running forever
test_repetitions = np.arange(N)

# init results containers

# a multi-dimensional dictionary
# e.g. frequency_of_catching_subpar_samples[0.9][22] will contain the confidence level
# for a device with an intrinsic reliability R=0.9 when tested with n_s=22 samples.
frequency_of_catching_subpar_samples = {}
for R in reliabilities: frequency_of_catching_subpar_samples[R] = {}

repetition_passed = np.zeros(N)

# simulating the accelerated life tests

for R in reliabilities:
    for n_s in n_samples:
        test_battery = [Device(R) for _ in range(n_s)]

        for rep in test_repetitions:
            repetition_passed[rep] = conduct_test(test_battery)

        frequency_of_catching_subpar_samples[R][n_s] = 1-np.mean(repetition_passed)

print("Confidence level of detecting fact that population shows reliability of <R given n_s samples is")

def print_summary(R, n_s):
    print("R={}, n_s={}".format(R,n_s), "--> C=",frequency_of_catching_subpar_samples[R][n_s])

print_summary(0.9, 22)
print_summary(0.95, 77)
print_summary(0.99, max(n_samples))
```

    Confidence level of detecting fact that population shows reliability of <R given n_s samples is
    R=0.9, n_s=22 --> C= 0.9
    R=0.95, n_s=77 --> C= 0.979
    R=0.99, n_s=97 --> C= 0.6619999999999999

```python
plt.figure(figsize=(12,6))

linestyles = [(0,()),
              (0,(1,1,5,3)),
              (0,(4,1)),
              (0,(5,1,1,3)),
              (0,(1,1)),
              (0,(1,1,1,1,5,5)),
              (0,(5,1,1,1,1,5))]
linewidth = 3

for j, p in enumerate(reliabilities):
    confidence_of_catching_p_reliable_failures = [frequency_of_catching_subpar_samples[p][n_s] for n_s in n_samples]
    plt.plot(n_samples, confidence_of_catching_p_reliable_failures, label="R = {}".format(p),
             linestyle=linestyles[j], linewidth=linewidth)
    plt.plot(n_samples, 1-p**n_samples, linestyle='-', linewidth=0.5, color='black')
plt.grid()

plt.xlim([0,100])
plt.ylim([0,1])
fontsize = 'x-large'
plt.legend(loc='best', fontsize=fontsize)

plt.title("Numerical simulation vs analytical $C = 1 - R^{n_s}$ (thin black lines)", fontsize=fontsize)
plt.xlabel("Number of samples tested (-)", fontsize=fontsize)
plt.ylabel("Confidence level (-)", fontsize=fontsize)
plt.xticks(np.arange(0,101,10))
plt.yticks(np.linspace(0,1,11))

print("")
```

The numerical simulations are in line with the analytical solution derived above.

All the usual disclaimers regarding statistical explorations apply:
The "tested" samples are assumed to be identical and independent.
The analytical solution is true only in the limit of infinite repetitions.
And so on.

## Mapping out R vs C analytically

In the above sections we have looked at the question:
"how many samples are we supposed to test assuming a target $R$ and $C$?"
In the present section we approach the problem from the other side.
Assuming we have $n_\mathrm{s}$ samples available,
what can we learn from a fail/pass test?

As already mentioned above,
if all $n_\mathrm{s}$ samples pass the test,
we forego the opportunity to actually learn something
(meaning, by opening up a failing sample and learning about the root causes).
The only statement we can make given a $100\,\%$ pass rate is
that the underlying population is $R$ reliable with $C$ confidence,
related to each other through Eq. (\ref{eq:CRn}) and (\ref{eq:n_lnCR}), respectively.

As an example let's assume we have $n_\mathrm{s}=4$ samples available.
What is the underlying reliability $R$ of the population supposed to be
for us to have a reasonable chance of catching a failing sample
among these $n_\mathrm{s}=4$ samples?

One way to go about answering this question is to flip a coin:
tail means the population is impecable and can be shipped,
head means the population shows some not-further-specified flaws.
This strategy, stated in this extreme form, does not take the fail/pass test into account in any way whatsoever.
However, an unrelated coin flip is not actually worth less
than a test with expected frequency of finding a failing part of $C\leq0.5$.

For example, if none of the $n_\mathrm{s}=4$ samples failed,
but, if at the same time we were to expect the underlying population to show a reliability
of at least $R\geq0.84$,
then the fact that all $n_\mathrm{s}=4$ samples have passed
is actually less informative than a completely unrelated coin flip!

More generally, from Eq. (\ref{eq:CRn}) we find
\begin{equation}
R = \sqrt[\leftroot{-2}\uproot{2}n_\mathrm{s}]{1-C}.
\label{eq:nrootC}
\end{equation}

For a test with $n_\mathrm{s}=4$ to be meaningful -- say $C\geq0.9$ --
the underlying population has to show a reliability of at most $R=0.56$.

```python

R = lambda C, n_s: (1 - C)**(1/n_s)

def R_summary(C, n_s):
    print('Underlying reliability has to be at most R={:.2f}, given C={} with n_s={}'.format(R(C,n_s),C,n_s))

R_summary(0.5,4)
R_summary(0.9,4)
```

    Underlying reliability has to be at most R=0.84, given C=0.5 with n_s=4
    Underlying reliability has to be at most R=0.56, given C=0.9 with n_s=4

```python
plt.figure(figsize=(12,6))

linestyles = [(0,()),
              (0,(1,1,5,3)),
              (0,(4,1)),
              (0,(5,1,1,3)),
              (0,(1,1)),
              (0,(1,1,1,1,5,5)),
              (0,(5,1,1,1,1,5))]
linewidth = 3

sample_ns = [4, 10, 22, 77]
assumed_Rs = np.array(list(np.linspace(0.5,0.9,9)) + list(np.linspace(0.91,1,10)))

for j, n in enumerate(sample_ns):
    plt.plot(assumed_Rs, 1-assumed_Rs**n, label='{}'.format(n),
             linestyle=linestyles[j], linewidth=linewidth)

# caution: C<0.5 means confidence worse than a coin flip
# i.e. don't even think about ending up in this region!
warning_color = 'red'
plt.fill_between([0.5,1], [0.5,0.5], facecolor=warning_color, alpha=0.1)
plt.axhline(y=0.5, color='red', linewidth=0.5)
plt.text(0.575,0.325,'confidence worse than coin flip', fontsize=fontsize, color=warning_color)

# formating
plt.legend(loc='best')
plt.grid('on')

plt.xlabel("Assumed reliability (-)", fontsize=fontsize)
plt.ylabel("Confidence level (-)", fontsize=fontsize)
plt.xticks(np.linspace(0.5,1,11))
plt.xlim([0.5,1])
plt.yticks(np.linspace(0,1,11))
plt.ylim([0,1])

print("")
```

## Further reading

This notebook was originally inspired by an Accendo Reliability podcast on the topic
<https://accendoreliability.com/podcast/arw/making-use-reliability-statistics/>

Regarding what we can learn from a failed vs passed test (equally applicable for software and hardware),
I can recommend
K. Henney “A Test of Knowledge” <https://medium.com/@kevlinhenney/a-test-of-knowledge-78f4688dc9cb>


# Cyclical Systems: An Example of the Crank-Nicolson Method
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
import numpy as np
from numpy import *
# %matplotlib notebook
# %matplotlib nbagg
%matplotlib inline
%config InlineBackend.figure_format = 'svg'

# %matplotlib qt
import matplotlib.pyplot as plt
from scipy.optimize import fsolve
from scipy.integrate import odeint
```

```python
def forward_euler(rhs, f0, tend, dt):
    ''' Computes the forward_euler method '''
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        f[n+1] = f[n] + dt * rhs(f[n], time[n])
    return time, f

def forward_euler_system(rhsvec, f0vec, tend, dt):
    '''
    Solves a system of ODEs using the Forward Euler method
    '''
    nsteps = int(tend/dt)
    neqs = len(f0vec)
    f = np.zeros( (neqs, nsteps) )
    f[:,0] = f0vec
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        t = time[n]
        f[:,n+1] = f[:,n] + dt * rhsvec(f[:,n], t)
    return time, f

def be_residual(fnp1, rhs, fn, dt, tnp1):
    '''
    Nonlinear residual function for the backward Euler implicit time integrator
    '''
    return fnp1 - fn - dt * rhs(fnp1, tnp1)

def backward_euler(rhs, f0, tend, dt):
    '''
    Computes the backward euler method
    :param rhs: an rhs function
    '''
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        fn = f[n]
        tnp1 = time[n+1]
        fnew = fsolve(be_residual, fn, (rhs, fn, dt, tnp1))
        f[n+1] = fnew
    return time, f

def cn_residual(fnp1, rhs, fn, dt, tnp1, tn):
    '''
    Nonlinear residual function for the Crank-Nicolson implicit time integrator
    '''
    return fnp1 - fn - 0.5 * dt * ( rhs(fnp1, tnp1) + rhs(fn, tn) )

def crank_nicolson(rhs,f0,tend,dt):
    nsteps = int(tend/dt)
    f = np.zeros(nsteps)
    f[0] = f0
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        fn = f[n]
        tnp1 = time[n+1]
        tn = time[n]
        fnew = fsolve(cn_residual, fn, (rhs, fn, dt, tnp1, tn))
        f[n+1] = fnew
    return time, f
```

# Sharp Transient
Solve the ODE:
\begin{equation}
\frac{\text{d}y}{\text{d}t} = -1000 y + 3000 - 2000 e^{-t};\quad y(0) = 0
\end{equation}
The analytical solution is
\begin{equation}
y(t) = 3 - 0.998 e^{-1000t} - 2.002 e^{-t}
\end{equation}

We first plot the analytical solution

```python
y = lambda t : 3 - 0.998*exp(-1000*t) - 2.002*exp(-t)
t = np.linspace(0,1,500)
plt.plot(t,y(t))
plt.grid()
```

Now let's solve this numerically. We first define the RHS for this function

```python
def rhs_sharp_transient(f,t):
    return  3000 - 1000 * f - 2000* np.exp(-t)
```

Let's solve this using forward euler and backward euler

```python
y0 = 0
tend = 0.03
dt = 0.001
t,yfe = forward_euler(rhs_sharp_transient,y0,tend,dt)
t,ybe = backward_euler(rhs_sharp_transient,y0,tend,dt)
t,ycn = crank_nicolson(rhs_sharp_transient,y0,tend,dt)

plt.plot(t,y(t),label='Exact')
# plt.plot(t,yfe,'r.-',markevery=1,markersize=10,label='Forward Euler')
plt.plot(t,ybe,'k*-',markevery=2,markersize=10,label='Backward Euler')
plt.plot(t,ycn,'o-',markevery=2,markersize=2,label='Crank Nicholson')
plt.grid()
plt.legend()
```

    <matplotlib.legend.Legend at 0x151530fc88>

# Oscillatory Systems
Solve the ODE:
Solve the ODE:
\begin{equation}
\frac{\text{d}y}{\text{d}t} = r \omega \sin(\omega t)
\end{equation}
The analytical solution is
\begin{equation}
y(t) = r - r \cos(\omega t)
\end{equation}

First plot the analytical solution

```python
r = 0.5
ω = 0.02
y = lambda t : r - r * cos(ω*t)
t = np.linspace(0,100*pi)
plt.clf()
plt.plot(t,y(t))
plt.grid()
```

Let's solve this numerically

```python
def rhs_oscillatory(f,t):
    r = 0.5
    ω = 0.02
    return r * ω * sin(ω*t)
```

```python
y0 = 0
tend = 100*pi
dt = 10
t,yfe = forward_euler(rhs_oscillatory,y0,tend,dt)
t,ybe = backward_euler(rhs_oscillatory,y0,tend,dt)
t,ycn = crank_nicolson(rhs_oscillatory,y0,tend,dt)
plt.plot(t,y(t),label='Exact')
plt.plot(t,yfe,'r.-',markevery=1,markersize=10,label='Forward Euler')
plt.plot(t,ybe,'k*-',markevery=2,markersize=10,label='Backward Euler')
plt.plot(t,ycn,'o-',markevery=2,markersize=2,label='Crank Nicholson')
plt.grid()
plt.legend()
plt.savefig('cyclical-system-example.pdf')
```

```python
import urllib
import requests
from IPython.core.display import HTML
def css_styling():
    styles = requests.get("https://raw.githubusercontent.com/saadtony/NumericalMethods/master/styles/custom.css")
    return HTML(styles.text)
css_styling()

```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


# Penalised Regression

## YouTube Videos
1. **Scikit Learn Linear Regression:** https://www.youtube.com/watch?v=EvnpoUTXA0E
2. **Scikit Learn Linear Penalise Regression:** https://www.youtube.com/watch?v=RhsEAyDBkTQ

## Introduction
We often do not want the coefficients/ weights to be too large. Hence we append the loss function with a penalty function to discourage large values of $w$.

\begin{align}
\mathcal{L} & = \sum_{i=1}^N (y_i-f(x_i|w,b))^2 + \alpha \sum_{j=1}^D w_j^2 + \beta \sum_{j=1}^D |w_j|
\end{align}
where, $f(x_i|w,b) = wx_i+b$. The values of $\alpha$ and $\beta$ are positive (or zero), with higher values enforcing the weights to be closer to zero.

## Lesson Structure
1. The task of this lesson is to infer the weights given the data (observations, $y$ and inputs $x$).
2. We will be using the module `sklearn.linear_model`.

```python
import numpy as np
from sklearn.linear_model import LinearRegression

import matplotlib.pyplot as plt
%matplotlib inline

# In order to reproduce the exact same number we need to set the seed for random number generators:
np.random.seed(1)
```

A normally distributed random looks as follows:

```python
e = np.random.randn(10000,1)
plt.hist(e,100) #histogram with 100 bins
plt.ylabel('y')
plt.xlabel('x')
plt.title('Histogram of Normally Distributed Numbers')
plt.show()
```

Generate observations $y$ given feature (design) matrix $X$ according to:
$$
y = Xw + \xi\\
\xi_i \sim \mathcal{N}(0,\sigma^2)
$$

In this particular case, $w$ is a 100 dimensional vector where 90% of the numbers are zero. i.e. only 10 of the numbers are non-zero.

```python
# Generate the data
N = 40 # Number of observations
D = 100 # Dimensionality

x = np.random.randn(N,D) # get random observations of x
w_true = np.zeros((D,1)) # create a weight vector of zeros
idx = np.random.choice(100,10,replace=False) # randomly choose 10 of those weights
w_true[idx] = np.random.randn(10,1) # populate then with 10 random weights

e = np.random.randn(N,1) # have a noise vector
y = np.matmul(x,w_true) + e # generate observations

# create validation set:
N_test = 50
x_test = np.random.randn(50,D)
y_test_true = np.matmul(x_test,w_true)
```

```python
model = LinearRegression()
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.title('Estimated Weights')
plt.show()
```

One way of testing how good your model is to look at metrics. In the case of regression Mean Squared Error (MSE) is a common metric which is defined as:
$$ \frac{1}{N}\sum_{i=1}^N \xi_i^2$$ where, $\xi_i = y_i-f(x_i|w,b)$. Furthermore it is best to look at the MSE on a validation set, rather than on the training dataset that we used to train the model.

```python
y_est = model.predict(x_test)
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    6.79499954951

Ridge regression is where you penalise the weights by setting the $\alpha$ parameter right at the top. It penalises it so that the higher **the square of the weights** the higher the loss.

```python
from sklearn.linear_model import Ridge

model = Ridge(alpha=5.0,fit_intercept = False)
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.show()
```

This model is slightly better than without any penalty on the weights.

```python
y_est = model.predict(x_test)
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    6.42288072501

Lasso is a model that encourages weights to go to zero exactly, as opposed to Ridge regression which encourages small weights.

```python
from sklearn.linear_model import Lasso

model = Lasso(alpha=0.1,fit_intercept = False)
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.title('Lasso regression weight inference')
plt.show()
```

The MSE is significantly better than both the above models.

```python
y_est = model.predict(x_test)[:,None]
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    2.30600134084

Automated Relevance Determination (ARD) regression is similar to lasso in that it encourages zero weights. However, the advantage is that you do not need to set a penalisation parameter, $\alpha$, $\beta$ in this model.

```python
from sklearn.linear_model import ARDRegression

model = ARDRegression(fit_intercept = False)
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.show()
```

```python
y_est = model.predict(x_test)[:,None]
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    2.87016741296

### Note:
Rerun the above with setting N=400

## Inverse Problems
The following section is optional and you may skip it. It is not necessary for understanding Deep Learning.

Inverse problems are where given the outputs you are required to infer the inputs. A typical example is X-rays. Given the x-ray sensor readings, the algorithm needs to build an image of an individuals bone structure.

See [here](http://scikit-learn.org/stable/auto_examples/applications/plot_tomography_l1_reconstruction.html#sphx-glr-auto-examples-applications-plot-tomography-l1-reconstruction-py) for an example of l1 reguralisation applied to a compressed sensing problem (has a resemblance to the x-ray problem).

```python

```

deepschool.io/Lesson 01 - PenalisedRegression - Solutions.ipynb


### Problem 8
The four adjacent digits in the 1000-digit number that have the greatest product are 9 × 9 × 8 × 9 = 5832.

Find the thirteen adjacent digits in the 1000-digit number that have the greatest product. What is the value of this product?

$73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450$

```python
x = 7316717653133062491922511967442657474235534919493496983520312774506326239578318016984801869478851843858615607891129494954595017379583319528532088055111254069874715852386305071569329096329522744304355766896648950445244523161731856403098711121722383113622298934233803081353362766142828064444866452387493035890729629049156044077239071381051585930796086670172427121883998797908792274921901699720888093776657273330010533678812202354218097512545405947522435258490771167055601360483958644670632441572215539753697817977846174064955149290862569321978468622482839722413756570560574902614079729686524145351004748216637048440319989000889524345065854122758866688116427171479924442928230863465674813919123162824586178664583591245665294765456828489128831426076900422421902267105562632111110937054421750694165896040807198403850962455444362981230987879927244284909188845801561660979191338754992005240636899125607176060588611646710940507754100225698315520005593572972571636269561882670428252483600823257530420752963450
x = str(x)

def prod(x):
    """Computes product of a string of integers"""
    y = 1
    for c in x:
        y *= int(c)
    return y

biggest = 0
for i in range(1000):
    # x is 1000-digits long
    product = prod(x[i:i+13])
    biggest = product if product > biggest else biggest

print(biggest)

```

    23514624000

### Problem 9
A Pythagorean triplet is a set of three natural numbers, a < b < c, for which,

$$\begin{align}
a^2 + b^2 = c^2
\end{align}$$
For example, $32 + 42 = 9 + 16 = 25 = 52$.

There exists exactly one Pythagorean triplet for which $a + b + c = 1000$.

Find the product $abc$.

```python
import math
for a in range(1000):
    for b in range(1000):
        c = math.sqrt(a**2 + b**2)
        if a + b + c == 1000 and a < b < c:
            print(a,b,c)
            print(a*b*c)
```

    200 375 425.0
    31875000.0

```python

```