# Linear Programming: Simplex Method

## Karush-Kuhn-Tucker Conditions

We consider the linear program

\begin{equation}
\min \vec{c}^\mathsf{T}\vec x\\
\textrm{subject to} \begin{cases}
\mathbf{A}\vec x=\vec b\\
\vec x\ge\vec 0
\end{cases}
\end{equation}

and define the Lagrangian function

\begin{equation}
\mathcal L\left(\vec x,\vec \lambda,\vec s\right) = \vec{c}^\mathsf{T}\vec x - \vec \lambda^\mathsf{T}\left(\mathbf A\vec x-\vec b\right) - \vec s^\mathsf{T}\vec x
\end{equation}

where $\vec \lambda$ are the multipliers for the equality constraints and $\vec s$ are the multipliers for the bound constraints.

The Karush-Kuhn-Tucker condition states that to find the first-order necessary conditions for $\vec x^\star$ to be a solution of the problem, their exist $\vec \lambda^\star$ and $\vec s^\star$ such that

\begin{align}
\mathbf A^\mathsf{T}\vec \lambda^\star+\vec s^\star&=\vec c\\
\mathbf A\vec x^\star&=\vec b\\
\vec x^\star&\ge\vec 0\\
\vec s^\star&\ge \vec 0\\
\left(\vec x^\star\right)^\mathsf{T}\vec s^\star&=0
\end{align}

The first eqation states that the gradient of the Lagrangian with respect to $\vec x$ must be zero and the last equation that at least $x_i$ or $s_i$ must be zero for each $i=1,2,\dots,n$.

It can be shown that these conditions are also sufficient.

## The Simplex Method Explained

As mentioned in the previous lecture, all iterates of the simplex method are basic feasible points and therefore vertices of the feasible polytope. Most steps consists of a move from one vertex to an adjacent one. On most steps (but not all), the value of the objective function $\vec{c}^\mathsf{T}\vec x$ is decreased. Another type of step occurs when the problem is unbounded: the step is an edge along which the objective funtion is reduced, and along which we can move infinitely far without reaching a vertex.

The major issue at each simplex iteration is to decide which index to remove from the basic index set $\mathcal B$. Unless the step is a direction of unboundness, a single index must be removed from $\mathcal B$ and replaced by another from outside $\mathcal B$. We can gain some insight into how this decision is made by looking again at the KKT conditions.

First, define the nonbasic index set $\mathcal N = \left\{1,2,\dots,n\right\} \setminus \mathcal B$. Just as $\mathbf B$ is the basic matrix, whose columns are $\mathbf A_i$ for $i\in\mathcal B$, we use $\mathbf N$ to denote the nonbasic matrix $\mathbf N=\left[\mathbf A_i\right]_{i\in\mathcal N}$. We also partition the vectors $\vec x$, $\vec s$ and $\vec c$ according to the index sets $\mathcal B$  and $\mathcal N$, using the notation

\begin{align}
\vec x_\mathbf B=\left[\vec x_i \right]_{i\in\mathcal B},&\qquad\vec x_\mathbf N=\left[\vec x_i \right]_{i\in\mathcal N}\\
\vec s_\mathbf B=\left[\vec s_i \right]_{i\in\mathcal B},&\qquad\vec s_\mathbf N=\left[\vec s_i \right]_{i\in\mathcal N}\\
\vec c_\mathbf B=\left[\vec c_i \right]_{i\in\mathcal B},&\qquad\vec c_\mathbf N=\left[\vec c_i \right]_{i\in\mathcal N}
\end{align}

From the second KKT coniditions, we have that

\begin{equation}
\mathbf A \vec x= \mathbf B \vec x_\mathbf B + \mathbf N \vec x_\mathbf N=\vec b\,.
\end{equation}

The _primal_ variable $\vec x$ for this simplex iterate is defined as

\begin{equation}
\vec x_\mathbf B = \mathbf B^{-1}\vec b,\qquad \vec x_\mathbf N=\vec 0\,.
\end{equation}

Since we are dealing only with basic feasible points, we know that $\mathbf B$ is nonsingular and that $\vec x_\mathbf B\ge\vec0$, so this choice of $\vec x$ satisfies two of the KKT coniditions.

We choose $\vec s$ to satisfy the complimentary condition (the last one) by setting $\vec s_\mathbf B=\vec 0$. The remaining components $\vec \lambda$ and $\vec s_\mathbf N$ can be found by partitioning this condition into $\vec c_\mathbf B$ and $\vec c_\mathbf N$ components and using $\vec s_\mathbf B=\vec 0$ to obtain

\begin{equation}
\mathbf B^\mathsf{T}\vec \lambda=\vec c_\mathbf B,\qquad \vec N^\mathsf{T}\vec\lambda+\vec s_\mathbf N = \vec c_\mathbf N\,.
\end{equation}

Since $\mathbf B$ is square and nonsingular, the first equation uniquely defines $\vec \lambda$ as

\begin{equation}
\vec \lambda = \left(\mathbf B^\mathsf{T}\right)^{-1}\vec c_\mathbf B\,.
\end{equation}

The second equation implies a value for $\vec s_\mathbf N$:

\begin{equation}
\vec s_\mathbf N = \vec c_\mathbf N - \mathbf N^\mathsf{T}\vec \lambda=\vec c_\mathbf N -\left(\mathbf B ^{-1}\mathbf N\right)^\mathsf{T}\vec c_\mathbf B\,.
\end{equation}

Computation of the vector $\vec s_\mathbf N$ is often referred to as _pricing_. The components of $\vec s_\mathbf N$ are often called the _reduced costs_ of the nonbasic variables $\vec x_\mathbf N$.

The only KKT condition that we have not enforced explicitly is the nonnegativity condition $\vec s \ge \vec 0$. The basic components $\vec s_\mathbf B$ certainly satisfy this condition, by our choice $\vec s_\mathbf B = 0$. If the vector $\vec s_\mathbf N$ also satisfies $\vec s_\mathbf N \ge \vec 0$, we have found an optimal
vector triple $\left(\vec x^\star, \vec \lambda^\star, \vec s^\star\right)$, so the algorithm can terminate and declare success. Usually, however, one or more of the components of $\vec s_\mathbf N$ are negative. The new index to enter the basis index set $\mathcal B$ is chosen to be one of the indices $q \in \mathcal N$ for which $s_q < 0$. As we show below, the objective $\vec{c}^\mathsf{T}\vec x$ will decrease when we allow $x_q$ to become positive if and only if

1. $s_q < 0$ and
2. it is possible to increase $x_q$ away from zero while maintaining feasibility of $\vec x$.

Our procedure for altering $\mathcal B$ and changing $\vec x$ and $\vec s$ can be described accordingly as follows:

- allow $x_q$ to increase from zero during the next step;
- fix all other components of $\vec x_\mathbf N$ at zero, and figure out the effect of increasing $x_q$ on the current basic vector $\vec x_\mathbf B$, given that we want to stay feasible with respect to the equality constraints $\mathbf{A}\vec x=\vec b$;
- keep increasing $x_q$ until one of the components of $\vec x_\mathbf B$ ($x_p$, say) is driven to zero, or determining that no such component exists (the unbounded case);
- remove index $p$ (known as the leaving index) from $\mathcal B$ and replace it with the entering index $q$.

This process of selecting entering and leaving indices, and performing the algebraic operations necessary to keep track of the values of the variables $\vec x$, $\vec \lambda$, and $\vec s$, is sometimes known as _pivoting_.

We now formalize the pivoting procedure in algebraic terms. Since both the new iterate $\vec x^+$ and the current iterate $\vec x$ should satisfy $\mathbf A\vec x=\vec b$, and since $\vec x_\mathbf N=\vec 0$ and $\vec x_i^{+}=0$ for $i\in\mathcal N\setminus\left\{q\right\}$ we have

\begin{equation}
\mathbf A\vec x^+=\mathbf B\vec x_\mathbf B^+ +\vec A_q x_q^+=\vec b=\mathbf B\vec x_\mathbf B=\mathbf A\vec x\,.
\end{equation}

By multiplying this expression by $\mathbf B^{-1}$ and rearranging, we obtain

\begin{equation}
\vec x_\mathbf B^+=\vec x_\mathbf B-\mathbf B^{-1}\vec A_q x_q^+
\end{equation}

Geometrically speaking, we move along an edge of the feasible polytope that decreases $\vec{c}^\mathsf{T}\vec x$. We continue to move along this edge until a new vertex is encountered. At this vertex, a new constraint $x_p \ge 0$ must have become active, that is, one of the components $x_p$, $p \in \mathbf B$, has decreased to zero. We then remove this index $p$ from the basis index set $\mathcal B$ and replace it by $q$.

It is possible that we can increase $x_q^+$ to $\infty$ without ever encountering a new vertex. In other words, the constraint $x_\mathbf B^+=x_\mathbf B-\mathbf B^{-1}\vec A_q\vec x_q^+\ge 0$ holds for all positive values of $x_q+$. When this happens, the linear program is unbounded; the simplex method has identified a
ray that lies entirely within the feasible polytope along which the objective $\vec{c}^\mathsf{T}\vec x$ decreases to $−\infty$.

## One Step of Simplex Algorithm

Given $\mathcal B$, $\mathcal N$, $x_\mathbf B=\mathbf B^{-1}\vec b\ge 0$,$\vec x_\mathbf N=\vec 0$;

1. Solve $\vec \lambda = \left(\mathbf B^\mathsf{T}\right)^{-1}\vec c_\mathbf B$, and  compute $\vec s_\mathbf N =\vec c_\mathbf N - \mathbf N^\mathsf{T} \vec \lambda$ (pricing);
2. If $\vec s_\mathbf N \ge \vec 0$ stop (optimal point found);
3. Select $q\in\mathcal N$ with $s_q<0$ and solve $\vec d=\mathbf B^{-1}\vec A_q$;
4. If $\vec d\le\vec 0$ stop (problem is unbounded);
5. Calculate $x_q^+=\min_{i|d_i>0}\frac{\left(x_\mathbf B\right)_i}{d_i}$, and use $p$ to denote the minimizing $i$;
6. Update $\vec x_\mathbf B^+=\vec x_\mathbf B-x_q^+\vec d$;
7. Change $\mathcal B$ by adding $q$ and removing the basic variable corresponding to column $p$ of $\mathbf B$.

We illustrate this procedure with a simple example.

Consider the problem

\begin{equation}
\min -4x_1-2x_2\\
\textrm{subject to}
\begin{cases}
x_1+x_2+x_3&=5\\
2x_1+\frac{1}{2}x_2+x_4&=8\\
\vec x&\ge \vec 0
\end{cases}
\end{equation}

Suppose we start with the basis index set $\mathcal B=\left\{3,4\right\}$, for which we have

```julia
B = [1 0;0 1]
N = [1 1;2 0.5]
b = [5;8]
cB = [0;0]
cN = [-4;-2]
xB = inv(B)*b
λ = inv(transpose(B))*cB
sN = cN - transpose(N)*λ
@show xB λ sN;
```

\begin{equation}
\vec x_\mathbf B =
\begin{pmatrix}
x_3\\
x_4
\end{pmatrix}
=
\begin{pmatrix}
5\\
8
\end{pmatrix}
\,\qquad\vec\lambda =
\begin{pmatrix}
0\\
0
\end{pmatrix}
\,\qquad\vec s_\mathbf N =
\begin{pmatrix}
s_1\\
s_2
\end{pmatrix}
=
\begin{pmatrix}
-4\\
-2
\end{pmatrix}
\end{equation}

and an objective value of $\vec{c}^\mathsf{T}\vec x=0$. Since both elements of $\vec s_\mathbf N$ are negative, we could choose either 1 or 2 to be the entering variable. Suppose we choose $q=1$. We obtain

\begin{equation}
\vec d = \begin{pmatrix}
1\\
2
\end{pmatrix}\,,
\end{equation}

so we cannot (yet) conclude that the problem is unbounded. By performing the ratio calculation, we find that $p=2$ (corresponding to index 4) and $x_1^+=4$.

```julia
q = 1
Aq = [1;2]
d = inv(B)*Aq
ratio = xB./d
xq = minimum(ratio)
xB -= d * xq
@show d ratio xq xB;
```

We update the basic and nonbasic index sets to $\mathcal B=\left\{3,1\right\}$ and  $\mathcal N=\left\{4,2\right\}$, and move to the next iteration.

At the second iteration, we have

```julia
B = [1 1;0 2]
N = [0 1;1 0.5]
cB = [0;-4]
cN = [0;-2]
xB = inv(B)*b
λ = inv(transpose(B))*cB
sN = cN - transpose(N)*λ
@show xB λ sN;
```

\begin{equation}
\vec x_\mathbf B =
\begin{pmatrix}
x_3\\
x_1
\end{pmatrix}
=
\begin{pmatrix}
1\\
4
\end{pmatrix}
\,\qquad\vec\lambda =
\begin{pmatrix}
0\\
-2
\end{pmatrix}
\,\qquad\vec s_\mathbf N =
\begin{pmatrix}
s_4\\
s_2
\end{pmatrix}
=
\begin{pmatrix}
2\\
-1
\end{pmatrix}
\end{equation}

with an objective value of -16. We see that $\vec s_\mathbf N$ has one negative component, corresponding to the index $q=2$, se we select this index to enter the basis. We obtain

```julia
q = 2
Aq = [1;0.5]
d = inv(B)*Aq
ratio = xB./d
xq = minimum(ratio)
xB -= d * xq
@show d ratio xq xB;
```

\begin{equation}
\vec d = \begin{pmatrix}
\frac{3}{4}\\
\frac{1}{4}
\end{pmatrix}\,,
\end{equation}

so again we do not detect unboundedness. Continuing, we find that the minimum value of $x_2^+$ is $\frac{4}{3}$, and that $p=1$, which indicates that index 3 will leave the basic index set $\mathcal B$. We update the index sets to $\mathcal B=\left\{2,1\right\}$ and  $\mathcal N=\left\{4,3\right\}$ and continue.

At the start of the third iteration, we have

```julia
B = [1 1;0.5 2]
N = [0 1;1 0]
cB = [-2;-4]
cN = [0;0]
xB = inv(B)*b
λ = inv(transpose(B))*cB
sN = cN - transpose(N)*λ
@show xB λ sN;
```

\begin{equation}
\vec x_\mathbf B =
\begin{pmatrix}
x_2\\
x_1
\end{pmatrix}
=
\begin{pmatrix}
\frac{4}{3}\\
\frac{11}{3}
\end{pmatrix}
\,\qquad\vec\lambda =
\begin{pmatrix}
-\frac{4}{3}\\
-\frac{4}{3}
\end{pmatrix}
\,\qquad\vec s_\mathbf N =
\begin{pmatrix}
s_4\\
s_3
\end{pmatrix}
=
\begin{pmatrix}
\frac{4}{3}\\
\frac{4}{3}
\end{pmatrix}
\end{equation}

with an objective value of $-\frac{52}{3}$. We see that $\vec s_\mathbf N\ge\vec 0$, so the optimality test is satisfied, and we terminate.

We need to flesh out this procedure with specifics of three important aspects of the implementation:

- Linear algebra issues—maintaining an LU factorization of $\mathbf B$ that can be used to solve for $\vec \lambda$ and $\vec d$.
- Selection of the entering index $q$ from among the negative components of $\vec s_\mathbf N$. (In general, there are many such components.)
- Handling of degenerate bases and degenerate steps, in which it is not possible to choose a positive value of $x_q$ without violating feasibility.

Proper handling of these issues is crucial to the efficiency of a simplex implementation. We will use a software package handling these details.

## JuMP

```julia
#using Pkg
#pkg"add JuMP GLPK"

using JuMP
using GLPK
```

```julia
model = Model(with_optimizer(GLPK.Optimizer, method = GLPK.SIMPLEX))
@variable(model, 0 <= x1)
@variable(model, 0 <= x2)
@objective(model, Min, -4*x1 -2*x2)
@constraint(model, con1, x1 + x2 <= 5)
@constraint(model, con2, 2*x1 + 0.5*x2 <= 8)
optimize!(model)
```

```julia
termination_status(model)
```

```julia
primal_status(model)
```

```julia
objective_value(model)
```

```julia
value(x1)
```

```julia
value(x2)
```

```julia
using Plots
using LaTeXStrings

x = -1:5
plot(x, 5 .- x, linestyle=:dash, label=L"x_1+x_2=5")
plot!(x, (8 .- 2 .* x) ./ 0.5, linestyle=:dash, label=L"2x_1+0.5x_2=8")
plot!([0,4,11/3,0,0],[0,0,4/3,5,0], linewidth=2, label="constraints")
plot!(x, -4 .* x ./ 2, label=L"f\left(x_1,x_2\right)=-4x_1-2x_2=0")
plot!(x, (-16 .+ 4 .* x) ./ -2, label=L"f\left(x_1,x_2\right)=-4x_1-2x_2=-16")
plot!(x, (-52/3 .+ 4 .* x) ./ -2, label=L"f\left(x_1,x_2\right)=-4x_1-2x_2=-52/3")
```

## Where does the Simplex Method fit?

In linear programming, as in all optimization problems in which inequality constraints are present, the fundamental task of the algorithm is to determine which of these constraints are active at the solution and which are inactive. The simplex method belongs to a general class of algorithms for constrained optimization known as _active set methods_, which explicitly maintain estimates of the active and inactive index sets that are updated at each step of the algorithm. (At each iteration, the basis $\mathcal B$ is our current estimate of the inactive set, that is, the set of indices $i$ for which we suspect that $x_i > 0$ at the
solution of the linear program.) Like most active set methods, the simplex method makesonly modest changes to these index sets at each step; a single index is exchanged between $\mathcal B$ into $\mathcal N$.

One undesirable feature of the simplex method attracted attention from its earliest days. Though highly efficient on almost all practical problems (the method generally requires at most $2m$ to $3m$ iterations, where $m$ is the row dimension of the constraint matrix, there are pathological problems on which the algorithm performs very poorly. The complexity of the simplex method is _exponential_, roughly speaking, its running time may be an exponential function of the dimension of
the problem. For many years, theoreticians searched for a linear programming algorithm that has polynomial complexity, that is, an algorithm in which the running time is bounded by a polynomial function of the amount of storage required to define the problem.

In the mid-1980s, Karmarkar described a polynomial algorithm that approaches the solution through the interior of the feasible polytope rather than working its way around the boundary as the simplex method does.

## Interior Point Methods

In the 1980s it was discovered that many large linear programs could be solved efficiently by using formulations and algorithms from nonlinear programming and nonlinear equations. One characteristic of these methods was that they required all iterates to satisfy the inequality constraints in the problem _strictly_, so they became known as interior-point methods. By the early 1990s, a subclass of interior-point methods known as primal-dual methods had distinguished themselves as the most efficient practical approaches, and proved to be strong competitors to the simplex method on large problems.

Interior-point methods arose from the search for algorithms with better theoretical properties than the simplex method. The simplex method can be inefficient on certain pathological problems. Roughly speaking, the time required to solve a linear program may be exponential in the size of the problem, as measured by the number
of unknowns and the amount of storage needed for the problem data. For almost all practical problems, the simplex method is much more efficient than this bound would suggest, but its poor worst-case complexity motivated the development of new algorithms with better guaranteed performance.

Interior-point methods share common features that distinguish them from the simplex method. Each interior-point iteration is expensive to compute and can make significant progress towards the solution, while the simplex method usually requires a larger number of inexpensive iterations. Geometrically speaking, the simplex method works its way around the boundary of the feasible polytope, testing a sequence of vertices in turn until it finds the optimal one. Interior-point methods approach the boundary of the feasible set only in the limit. They may approach the solution either from the interior or the exterior of the feasible region, but they never actually lie on the boundary of this region.

```julia
model = Model(with_optimizer(GLPK.Optimizer, method = GLPK.INTERIOR))
@variable(model, 0 <= x1)
@variable(model, 0 <= x2)
@objective(model, Min, -4*x1 -2*x2)
@constraint(model, con1, x1 + x2 <= 5)
@constraint(model, con2, 2*x1 + 0.5*x2 <= 8)
optimize!(model)
```

```julia
termination_status(model)
```

```julia
primal_status(model)
```

```julia
objective_value(model)
```

```julia
value(x1)
```

```julia
value(x2)
```


# Module 2
## Module 2.1: Unit 3: Flow control
### Conditionals

#### Example: Heaviside function

Consider an implementation of the [Heaviside step function](http://mathworld.wolfram.com/HeavisideStepFunction.html)

$$
\Theta(x) = \begin{cases}
  0 & x < 0 \\
  \frac{1}{2} & x = 0\\
  1 & x > 0
  \end{cases}
$$

```python
x = float(input("x <- "))

theta = None
if x < 0:
    theta = 0
elif x == 0:
    theta = 0.5
else:
    theta = 1

print("theta({0}) = {1}".format(x, theta))
```

    x <- 2
    theta(2.0) = 1

#### Equality operator `==`

```python
3 == 3.0
```

    True

```python
"A" == "a"
```

    False

```python
0 == 0
```

    True

```python
"42" == 42
```

    False

```python

```

#### Comparison operators

```python
low = -2.3
high = 1000
```

```python
low < high
```

    True

```python
low >= high
```

    False

```python
low == high
```

    False

```python
middle = 0
```

```python
low < middle < high
```

    True

```python
low < middle > high
```

    False

```python
low < high > middle
```

    True

```python
low < middle <= 0 < high
```

    True

Works also for strings

```python
first = "aardvark"
last = "zebra"
```

```python
first < last
```

    True

```python
first == "Aardvark"
```

    False

```python
first > "Aardvark"
```

    True

... but why and how?

```python
"a" > "A"
```

    True

```python
ord("a")
```

    97

```python
ord("A")
```

    65

```python
chr(65)
```

    'A'

```python
(65, 65) < (97, 65)
```

    True

#### Boolean operators
* `and`
* `or`
* `not`

```python
low < middle and middle < high
```

    True

```python
low < middle or middle == high
```

    True

#### What is truth (in Python)?

Stop and try the following code and figure out why the following expressions return `True` or `False`:

```python
True
```

    True

```python
False
```

    False

```python
bool(True)
```

    True

```python
bool(False)
```

    False

```python
bool(0)
```

    False

```python
bool(1)
```

    True

```python
bool(2)
```

    True

```python
bool("True")
```

    True

```python
bool("true")
```

    True

```python
bool("False")
```

    True

```python
bool("")
```

    False

```python
bool(" ")
```

    True

```python
bool(None)
```

    False

#### Identity operator

**Warning**: Do not use `is` when testing for equality:

```python
a = 256
b = 256
```

```python
a is b
```

    True

```python
x = 257
y = 257
```

```python
x is y
```

    False

(see https://wsvincent.com/python-wat-integer-cache/ for more information and further links)

### Loops

#### `while`

```python
# countup.py

tmax = 10.
t, dt = 0, 2.

while t <= tmax:
   print("time " + str(t))
   t += dt
print("Finished")
```

    time 0
    time 2.0
    time 4.0
    time 6.0
    time 8.0
    time 10.0
    Finished

Fibonacci series
$$
F_n = F_{n−1} + F_{n−2} \quad \text{with}\ F_1 = F_2 = 1
$$

```python
# fibonacci.py

Fmax = 100
a, b = 0, 1

while b < Fmax:
   print(b, end=' ')
   a, b = b, a+b
print()
```

    1 1 2 3 5 8 13 21 34 55 89

##### Vertical throw

Throw a ball of mass $m$ vertically into the air from initial height $y_0 = 2$ m. What is its position $y(t)$ as a function of time $t$ and when does it hit the ground at $y=0$ if it has an initial upwards velocity $v_0 = 15\,\text{m/s}$ (a slow baseball pitch is $30\,\text{m/s}$)?

Kinematic equation of motion:
$$
y(t) = -\frac{1}{2} g t^2 + v_0 t + y_0
$$

```python
# parameters
g = 9.81  # acceleration due to gravity in m/s**2
v0 = 15   # initial velocity in m/s
y0 = 2    # initial position in m
y_ground = 0

dt = 0.1  # time step in s

# data containers
y_values = []
t_values = []

# initial conditions
y = y0
t = 0
while y > y_ground:
    y = -0.5*g*t**2 + v0*t + y0
    t_values.append(t)
    y_values.append(y)

    t += dt
```

```python
y_values[-1]
```

    -0.22720000000002472

```python
t_values[-1]
```

    3.2000000000000015

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.plot(t_values, y_values, 'o-')
plt.xlabel("time (s)")
plt.ylabel("y (m)");
```

#### `for` loop

Convert temperatures in Fahrenheit to Kelvin
$$
T/\text{K} = \frac{5}{9}(\theta/^\circ\text{F} - 32) + 273.15
$$

```python
# temperature conversion

temperatures_F = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]
for theta in temperatures_F:
   T = (5/9) * (theta - 32)  + 273.15
   print(T)

print("Conversion complete")
```

    288.76111111111106
    298.8722222222222
    310.26111111111106
    309.31666666666666
    311.65
    316.4833333333333
    Conversion complete

Store results:

```python
temperatures_F = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]
temperatures_K = []
for theta in temperatures_F:
   T = (5/9) * (theta - 32)  + 273.15
   temperatures_K.append(T)

print(temperatures_K)
```

    [288.76111111111106, 298.8722222222222, 310.26111111111106, 309.31666666666666, 311.65, 316.4833333333333]

Use `range()` to print side by side _after_ processing:

```python
temperatures_F = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]
temperatures_K = []
for theta in temperatures_F:
   T = (theta - 32) * (5/9) + 273.15
   temperatures_K.append(T)

# print results side by side
for i in range(len(temperatures_F)):
    T_F = temperatures_F[i]
    T_K = temperatures_K[i]
    print("{0:.2f} F = {1:.2f} K".format(T_F, T_K))
```

    60.10 F = 288.76 K
    78.30 F = 298.87 K
    98.80 F = 310.26 K
    97.10 F = 309.32 K
    101.30 F = 311.65 K
    110.00 F = 316.48 K

#### Vertical throw example with `for` and `break`

Fixed number of steps with condition
- run `Nsteps` or stop when ball hits ground
- Kinematic equation of motion:
  $$
  y(t) = -\frac{1}{2} g t^2 + v_0 t + y_0
  $$

```python
# parameters
g = 9.81  # acceleration due to gravity in m/s**2
v0 = 15   # initial velocity in m/s
y0 = 2    # initial position in m
y_ground = 0

dt = 0.1  # time step in s
Nsteps = 100

# data containers
y_values = []
t_values = []

# initial conditions
y = y0
t = 0

for i in range(Nsteps):
    t = i * dt
    y = -0.5*g*t**2 + v0*t + y0
    if y < y_ground:
        break
    y_values.append(y)
    t_values.append(t)
print("Computed", len(y_values), "positions")
```

    Computed 32 positions

```python
plt.plot(t_values, y_values, '.-')
```

## Module 2.2: Unit 4: Containers

*Data structures* are important for keeping track of data.
- organize many numbers
- Think of your key data structures *before* you start programming!

#### Examples in physics
  - vectors: $\mathbf{r} = \begin{pmatrix}x\\y\\z\end{pmatrix}$
  - matrices: $\mathsf{A} = \begin{pmatrix}
       A_{11} & A_{12} & A_{13}\\
       A_{21} & A_{22} & A_{23}
       \end{pmatrix}$
  - tensors: $\epsilon_{ijk}$

  - time series of measurements: $\big\{x(t_1), x(t_2), \dots, x(t_N)\big\}$
  - measurements at specific coordinates e.g.,
    - cartesian coordinate in space: $\rho(x,y,z)$
    - latitude/longitude on earth: $p(\phi, \theta)$

### Lists
Very versatile container (and for us the most important container)
* sequence
* ordered
* mutable

```python
temperatures = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]
stuff = ["dog", 42, -1.234, "cat", [3, 2, 1]]
empty = []
two = [[], []]
```

#### indexing

```python
temperatures = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]
```

```
temperatures = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]   elements
               |    |     |     |     |      |      |
               |  0 |   1 |   2 |   3 |    4 |    5 |   index
```

First element

```python
temperatures[0]
```

    60.1

Arbitrary elements

```python
temperatures[3]
```

    97.1

**Note**: Python indices are **0-based**.

```
temperatures = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]   elements
               |    |     |     |     |      |      |
               |  0 |   1 |   2 |   3 |    4 |    5 |   index
```
For example, the third element is at index 2:

```python
temperatures[2]
```

    98.8

Negative indices count from the last element to the first:
```
temperatures = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]   elements
               |    |     |     |     |      |      |
               |  0 |   1 |   2 |   3 |    4 |    5 |   index
               | -6 |  -5 |  -4 |  -3 |   -2 |   -1 |   neg.index

```

Last element

```python
temperatures[-1]
```

    110.0

Third element from end

```python
temperatures[-3]
```

    97.1

Python
[built-in function](https://docs.python.org/3.5/library/functions.html#built-in-functions)
to determine the *length of a list*:
[len()](https://docs.python.org/3.5/library/functions.html#len):

```python
len(temperatures)
```

    6

#### slicing
Slicing produces a new list by extracting a subset of elements as
determined by the "slice" _start:stop:step_. The general slicing syntax for a list `a`:
```python
a[start:stop:step]
```
where
- index `start` is *included* and
- `stop` is *excluded*;
- `start`, `stop`, `step` are each optional:
   - default for `start`: first element (index 0)
   - default for `stop`: after last element
   - default for `step` is 1, i.e., include every element.

Negative values are also allowed for indices and negative step counts backwards.

First 3 elements:

```
temperatures = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]   elements
               |    |     |     |     |      |      |
               |  0 |   1 |   2 |   3 |    4 |    5 |   index
               | -6 |  -5 |  -4 |  -3 |   -2 |   -1 |   neg.index

```

```python
temperatures[0:3]
```

    [60.1, 78.3, 98.8]

```python
temperatures[:3]
```

    [60.1, 78.3, 98.8]

(`start` defaults to 0 and can be omitted).

**`a[start:stop:step]`**

Omitting parameters
```python
temperatures[::2] == [60.1, 98.8, 101.3]
temperatures[2::2] == [98.8, 101.3]
temperatures[:2:2] == [60.1]
```

```
temperatures = [60.1, 78.3, 98.8, 97.1, 101.3, 110.0]   elements
               |    |     |     |     |      |      |
               |  0 |   1 |   2 |   3 |    4 |    5 |   index
               | -6 |  -5 |  -4 |  -3 |   -2 |   -1 |   neg.index

```

##### slicing example

```python
letters = ['A', 'B', 'C', 'D', 'E', 'F']
```

```
+---+---+---+---+---+---+
|'A'|'B'|'C'|'D'|'E'|'F'|  elements
+---+---+---+---+---+---+
| 0 | 1 | 2 | 3 | 4 | 5 |  index
+---+---+---+---+---+---+
|-6 |-5 |-4 |-3 |-2 |-1 |  index
+---+---+---+---+---+---+
```

```
+---+---+---+---+---+---+
|'A'|'B'|'C'|'D'|'E'|'F'|  elements
+---+---+---+---+---+---+
| 0 | 1 | 2 | 3 | 4 | 5 |  index
+---+---+---+---+---+---+
|-6 |-5 |-4 |-3 |-2 |-1 |  index
+---+---+---+---+---+---+
```

```python
letters[:3]
```

    ['A', 'B', 'C']

```python
letters[0:3]
```

    ['A', 'B', 'C']

```python
letters[1:3]
```

    ['B', 'C']

```
+---+---+---+---+---+---+
|'A'|'B'|'C'|'D'|'E'|'F'|  elements
+---+---+---+---+---+---+
| 0 | 1 | 2 | 3 | 4 | 5 |  index
+---+---+---+---+---+---+
|-6 |-5 |-4 |-3 |-2 |-1 |  index
+---+---+---+---+---+---+
```

```python
letters[-3]
```

    'D'

```python
letters[-3:-1]
```

    ['D', 'E']

```python
letters[-3:]
```

    ['D', 'E', 'F']

```python
letters[1::2]
```

    ['B', 'D', 'F']

#### Summary

```python
letters[:3] == ['A', 'B', 'C']
letters[0:3] == ['A', 'B', 'C']
letters[1:3] == ['B', 'C']
letters[-3] == 'D'
letters[-3:-1] == ['D', 'E']
letters[-3:] == ['D', 'E', 'F']
letters[1::2] == ['B', 'D', 'F']
```

```
+---+---+---+---+---+---+
|'A'|'B'|'C'|'D'|'E'|'F'|  elements
+---+---+---+---+---+---+
| 0 | 1 | 2 | 3 | 4 | 5 |  index
+---+---+---+---+---+---+
|-6 |-5 |-4 |-3 |-2 |-1 |  index
+---+---+---+---+---+---+
```

### Dictionaries

`dict` is a great data structure but used less often in numerical calculations.

We mostly use it to keep track of parameters, e.g. masses of elements:

```python
masses = {
    "H": 1.0079,  "He": 4.002602,
    "Li": 6.94,   "Be": 9.0121831,
    "B": 10.81,   "C": 12.011,
    "N": 14.007,  "O": 15.999,
    "F": 18.998403163,
    "Ne": 20.1797,
}
```

Calculate the mass of the H$_2$O molecule:

```python
m_water = 2*masses["H"] + 1*masses["O"]
print("Water mass", m_water, "u")
```

    Water mass 18.0148 u

### Nested list example: 2D harmonic oscillator trajectory

A mass $m$ held by two perpendicular identical springs in the x-y plane ([2D harmonic oscillator](https://farside.ph.utexas.edu/teaching/336k/Newtonhtml/node28.html)) moves in a potential
$$
U(x, y) = \frac{1}{2}k(x^2 + y^2)
$$
or with the vector $\mathbf{r} = (x,y)$ written as
$$
U(\mathbf{r}) = \frac{1}{2}k\mathbf{r}^2.
$$
Its trajectory (curve in space) $\mathbf{r}(t) = \big(x(t), y(t)\big)$ is

\begin{align}
x(t) &= A \cos(\omega t)\\
y(t) &= B \cos(\omega t + \phi)
\end{align}

where the amplitudes $A$, $B$ and the phase difference $\phi$ are determined by the initial conditions, and the frequency is $\omega = \sqrt{k/m}$.

Given $A=1$, $B=2$, $\phi=\pi/4$ and $\omega=1$, calculate the trajectory $\mathbf{r}(t)$.

```python
import math

# parameters
A = 1
B = 2
phi = math.pi/4
omega = 1

dt = 0.1
N = 100    # number of steps

r_values = []
for i in range(N):
    t = i*dt
    x = A*math.cos(omega*t)
    y = B*math.cos(omega*t + phi)
    r = [x, y]
    r_values.append(r)
```

`r_values` is a nested list with `N` entries:

```python
len(r_values)
```

    100

Each entry is a list of length 2, which is clear from looking at the first few elements:

```python
r_values[:3]
```

    [[1.0, 1.4142135623730951],
     [0.9950041652780257, 1.2659626133539166],
     [0.9800665778412416, 1.1050625843737085]]

We say that `r_values`
* has 2 dimensions (one needs two indices to get at a value), e.g. `r_values[3][1]` to get the y-coordinate at time index 3.
* has length `N`
* has *shape* `N x 2`

Unpack the values from the `N x 2` list into two `N` lists (using *list comprehensions*):

```python
x_values = [r[0] for r in r_values]
y_values = [r[1] for r in r_values]
```

Load the *matplotlib* library for plotting and make images appear inline in the notebook:

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.plot(x_values, y_values)
plt.xlabel("x")
plt.ylabel("y")
plt.gca().set_aspect(1)
```

##### Alternative ways to unpack:

Unpack the two-element list as loop variables:

```python
x_values = [x for x,y in r_values]
y_values = [y for x,y in r_values]
```

**Advanced (optional):** Very tricky use of the [zip()](https://docs.python.org/3/library/functions.html#zip) function and the [unpacking operator](https://docs.python.org/3/tutorial/controlflow.html#tut-unpacking-arguments) `*` :

```python
x_values, y_values = zip(*r_values)
```

(Understanding how this works is left as a challenge. I am happy to give more hints if you ask in the forum.)

By the way: you can also use `zip` to "merge" the `x_values` and `y_values` back into a `Nx2` list, as demonstrated by looking at first few elements:

```python
list(zip(x_values, y_values))[:3]
```

    [(1.0, 1.4142135623730951),
     (0.9950041652780257, 1.2659626133539166),
     (0.9800665778412416, 1.1050625843737085)]

```python
r_values[:3]
```

    [[1.0, 1.4142135623730951],
     [0.9950041652780257, 1.2659626133539166],
     [0.9800665778412416, 1.1050625843737085]]

```python

```


```python
import sympy as sym
from sympy.polys.multivariate_resultants import MacaulayResultant

sym.init_printing()
```

Macaulay Resultant
------------------

The Macauly resultant is a multivariate resultant. It is used for calculating the resultant of $n$ polynomials
in $n$ variables. The Macaulay resultant is calculated as the determinant of two matrices,

$$R = \frac{\text{det}(A)}{\text{det}(M)}.$$

Matrix $A$
-----------

There are a number of steps needed to construct matrix $A$. Let us consider an example from https://dl.acm.org/citation.cfm?id=550525 to
show the construction.

```python
x, y, z = sym.symbols('x, y, z')
```

```python
a_1_1, a_1_2, a_1_3, a_2_2, a_2_3, a_3_3 = sym.symbols('a_1_1, a_1_2, a_1_3, a_2_2, a_2_3, a_3_3')
b_1_1, b_1_2, b_1_3, b_2_2, b_2_3, b_3_3 = sym.symbols('b_1_1, b_1_2, b_1_3, b_2_2, b_2_3, b_3_3')
c_1, c_2, c_3 = sym.symbols('c_1, c_2, c_3')
```

```python
variables = [x, y, z]
```

```python
f_1 = a_1_1 * x ** 2 + a_1_2 * x * y + a_1_3 * x * z + a_2_2 * y ** 2 + a_2_3 * y * z + a_3_3 * z ** 2
```

```python
f_2 = b_1_1 * x ** 2 + b_1_2 * x * y + b_1_3 * x * z + b_2_2 * y ** 2 + b_2_3 * y * z + b_3_3 * z ** 2
```

```python
f_3 = c_1 * x + c_2 * y + c_3 * z
```

```python
polynomials = [f_1, f_2, f_3]
mac = MacaulayResultant(polynomials, variables)
```

**Step 1** Calculated $d_i$ for $i \in n$.

```python
mac.degrees
```

**Step 2.** Get $d_M$.

```python
mac.degree_m
```

**Step 3.** All monomials of degree $d_M$ and size of set.

```python
mac.get_monomials_set()
```

```python
mac.monomial_set
```

```python
mac.monomials_size
```

These are the columns of matrix $A$.

**Step 4** Get rows and fill matrix.

```python
mac.get_row_coefficients()
```

Each list is being multiplied by polynomials $f_1$, $f_2$ and $f_3$ equivalently. Then we fill the matrix
based on the coefficient of the monomials in the columns.

```python
matrix = mac.get_matrix()
matrix
```

Matrix $M$
-----------

Columns that are non reduced are kept. The rows which contain one if the $a_i$s is dropoed.
$a_i$s are the coefficients of $x_i ^ {d_i}$.

```python
mac.get_submatrix(matrix)
```

Second example
-----------------
This is from: http://isc.tamu.edu/resources/preprints/1996/1996-02.pdf

```python
x, y, z = sym.symbols('x, y, z')
```

```python
a_0, a_1, a_2 = sym.symbols('a_0, a_1, a_2')
b_0, b_1, b_2 = sym.symbols('b_0, b_1, b_2')
c_0, c_1, c_2,c_3, c_4 = sym.symbols('c_0, c_1, c_2, c_3, c_4')
```

```python
f = a_0 * y -  a_1 * x + a_2 * z
g = b_1 * x ** 2 + b_0 * y ** 2 - b_2 * z ** 2
h = c_0 * y - c_1 * x ** 3 + c_2 * x ** 2 * z - c_3 * x * z ** 2 + c_4 * z ** 3
```

```python
polynomials = [f, g, h]
```

```python
mac = MacaulayResultant(polynomials, variables=[x, y, z])
```

```python
mac.degrees
```

```python
mac.degree_m
```

```python
mac.get_monomials_set()
```

```python
mac.get_size()
```

```python
mac.monomial_set
```

```python
mac.get_row_coefficients()
```

```python
matrix = mac.get_matrix()
matrix
```

```python
matrix.shape
```

```python
mac.get_submatrix(mac.get_matrix())
```

```python

```

```python

```


```python
import numpy as np
import sympy
sympy.init_printing(use_unicode=True)
from sympy import *
from sympy.solvers import solve
from IPython.display import display

def simplified(exp):
    simp = simplify(exp)
    display(simp)
    return simp

def firstOrderCondition(exp, var, iSelectedSolution=None):
    diffExp = simplify(diff(exp, var))
    display(diffExp)
    solutions = solve(diffExp, var)
    display(solutions)
    if iSelectedSolution is not None:
        solution = solutions[iSelectedSolution]
        optimum = exp.subs(var, solution)
        return simplified(optimum)
    else:
        return solutions

x = symbols('x')
w,A,B = symbols('w A B', Integer=True, Positive=True)
n,p,q,d = symbols('n p q d', positive=True)
```

### Optimal Channel Initialization

```python
Xn = simplified(n/(2*p-1) - w/(2*p-1)*(1-(p/(1-p))**n)/(1-(p/(1-p))**w) )
```

```python
firstOrderCondition(Xn,n)
```

```python
Xn = simplified(n*(1-q**(-w/B)) - w*(1-q**(-n/B)))
firstOrderCondition(Xn,n)
```

```python
nopt.subs(B,1)
```

```python
nopt.subs(q,1/2)
```

```python
sympy.simplify(Xn.subs(n,nopt))
```

```python
cp = (A * x**(A+1) - (A+1)*x**A + 1)/(x-1)
cp
```

```python
sympy.simplify(cp.subs(A, 3))
```


# 基本
sympyは高性能ですが全般的に遅いです．
巨大な数式のsymplify（整理）等はどんでもない時間がかかります．

```python
import sympy as sy  # sympyのインポート
```

# シンボリック変数の定義
`変数 = sy.Symbol("変数")`
または
`変数1, 変数2 = sy.symbols("変数1, 変数2")`
で定義する．

```python
t = sy.Symbol('x')
x, y, z = sy.symbols('x, y, z')
```

シンボリック変数を使って数式を定義できる

```python
expr = x + y**2 + z
expr
```

$\displaystyle x + y^{2} + z$

sin関数などの一般的な関数も使える．

```python
expr2 = sy.sin(x) * sy.exp(y) * x**2
expr2
```

$\displaystyle x^{2} e^{y} \sin{\left(x \right)}$

## 微分
`sy.diff(数式, 変数)`を使う

```python
sy.diff(expr, y)
```

$\displaystyle 2 y$

```python
sy.diff(expr2, x)
```

$\displaystyle x^{2} e^{y} \cos{\left(x \right)} + 2 x e^{y} \sin{\left(x \right)}$

## 積分
`sy.integrate()`を使う．

```python
sy.integrate(expr, x)
```

$\displaystyle \frac{x^{2}}{2} + x \left(y^{2} + z\right)$

```python
sy.integrate(expr2, y)
```

$\displaystyle x^{2} e^{y} \sin{\left(x \right)}$

****
## ベクトル，行列の定義
numpyと似た形式

```python
X = sy.Matrix([
    [x],
    [y]
])
X
```

$\displaystyle \left[\begin{matrix}x\\y\end{matrix}\right]$

```python
Y = sy.Matrix([
    [x, x*y],
    [z**3, x*sy.sin(z)],
])
Y
```

$\displaystyle \left[\begin{matrix}x & x y\\z^{3} & x \sin{\left(z \right)}\end{matrix}\right]$

## ヤコビ行列
ヤコビ行列は
```python
ベクトルA.jacobian(ベクトルB)
```
で計算できる．

```python
V = sy.Matrix([
    [x**2 + z*sy.log(z)],
    [x + sy.sin(y)]
])
V.jacobian(X)
```

$\displaystyle \left[\begin{matrix}2 x & 0\\1 & \cos{\left(y \right)}\end{matrix}\right]$


Notes on [this litte paper](https://arxiv.org/abs/1807.03462).

Suppose we have a random variable $X$ with cumulative distribution function $F(x)$. Think about the following expectation.

\begin{align}
L_p(\theta) &= \mathbb{E}\left[\left| X-\theta \right|^p\right] \\
 &= \int_{-\infty}^{\theta} (\theta-x)^p dF + \int_{\theta}^{\infty} (x-\theta)^p dF
\end{align}

The minimum of $L_p$ gives us useful points in a distribution.

\begin{align}
\frac{\partial L_p}{\partial \theta} &= \int_{-\infty}^{\theta} p(\theta-x)^{p-1} dF - \int_{\theta}^{\infty} p(x-\theta)^{p-1} dF
\end{align}

\begin{align}
\frac{\partial L_p}{\partial \theta} = 0 \Rightarrow \int_{-\infty}^{\theta} (\theta-x)^{p-1} dF = \int_{\theta}^{\infty} (x-\theta)^{p-1} dF
\end{align}

For $p=2$:

\begin{align}
            &\int_{-\infty}^{\theta} (\theta-x) dF = \int_{\theta}^{\infty} (x-\theta) dF \\
\Rightarrow \qquad &\int_{-\infty}^{\infty} (x-\theta) dF = 0 \\
\Rightarrow \qquad &\theta = \int_{-\infty}^{\infty} x dF
\end{align}

This is the mean by definition.

For $p=1$:

\begin{align}
            &\int_{-\infty}^{\theta} dF = \int_{\theta}^{\infty} dF \\
\Rightarrow \qquad & F(\theta) = 1-F(\theta) \\
\Rightarrow \qquad & F(\theta) = \frac{1}{2}
\end{align}

This is the usual definition of the median.

Unlike the mean, this definition of the median is a bit troublesome, because there is not necessarily a single point which satisfies this equation. In particular, for an empirical distribution with an even number of samples, any point in the interval between the $(\frac{N}{2}-1)$th and $(\frac{N}{2}+1)$th point will do. This is the reason why at school we are taught the double definition for the sample median: The middle point if there are an odd number of points, and the midpoint of the two middle points if there are an even number of points. WTF!?

This paper has a way to generalize the definition of the median such that there is always a single unique value. Instead of minimizing $L_p$ with $p=1$, do it with $p=1+\epsilon$ for $\epsilon \rightarrow 0$.

\begin{align}
\int_{-\infty}^{\theta} (\theta-x)^{\epsilon} dF = \int_{\theta}^{\infty} (x-\theta)^{\epsilon} dF
\end{align}

Using Taylor:
\begin{align}
(x-\theta)^{\epsilon} &= \exp(\log(x-\theta))^{\epsilon} \\
                      &= \exp(\epsilon\log(x-\theta)) \\
                      &= 1 + \epsilon\log(x-\theta) + \mathcal{O}(\epsilon^2)
\end{align}

This gives:
\begin{align}
& \int_{-\infty}^{\theta} \left[ 1 + \epsilon\log(\theta-x) + \mathcal{O}(\epsilon^2) \right] dF = \int_{\theta}^{\infty} \left[ 1 + \epsilon\log(x-\theta) + \mathcal{O}(\epsilon^2) \right] dF \\
\Rightarrow \quad & F(\theta) + \epsilon \int_{-\infty}^{\theta} \log(\theta-x) dF + \mathcal{O}(\epsilon^2) = 1 - F(\theta) + \epsilon \int_{\theta}^{\infty} \log(x-\theta) dF + \mathcal{O}(\epsilon^2) \\
\Rightarrow \quad & 2F(\theta) - 1 + \epsilon \left[ \int_{-\infty}^{\theta} \log(\theta-x) dF - \int_{\theta}^{\infty} \log(x-\theta) dF \right]+ \mathcal{O}(\epsilon^2) = 0
\end{align}

If $2F(\theta) - 1 = 0$ has a unique solution then as $\theta$ will approach this in the limit as $\epsilon\rightarrow 0$. However, when ambiguity arises, the $\epsilon$ term acts as a tie-breaker.

In the empirical distribution case, $\theta$ will need to satisfy the following:

\begin{align}
\sum_{i \leq \frac{N}{2}} \log\left( \theta - x_i \right) = \sum_{i > \frac{N}{2}} \log\left( x_i - \theta \right)
\end{align}

We can try this out.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
from scipy import optimize

N = 8
seed = 1234

x_samples = stats.gamma.rvs(1, size=N, random_state=np.random.RandomState(seed))
x_samples.sort()

x_range = np.linspace(0, 4, 10000)
F = [(x_samples < x).sum() for x in x_range]

med_lower = x_samples[int(N/2-1)]
med_upper = x_samples[int(N/2)]

def Llog(theta):
    return np.log(theta-x_samples[:int(N/2)]).sum() - np.log(x_samples[int(N/2):]-theta).sum()

tiny_bit = 1E-10
med = optimize.brentq(Llog, med_lower+tiny_bit, med_upper-tiny_bit)

plt.plot(x_range, F)
plt.plot(med_lower*np.array([1,1]), [0,N], 'r')
plt.plot(med_upper*np.array([1,1]), [0,N], 'r')
plt.plot(med*np.array([1,1]), [0,N], 'g')
plt.show()

```

# Appendix

That derivative is not trivial. It relies on the following. Define:

\begin{align}
Q(\theta) &= \int_{-\infty}^{\theta} g(x,\theta) dF
\end{align}

Let's go.

\begin{align}
\frac{\partial Q}{\partial \theta} &= \lim_{\phi\rightarrow 0} \frac{1}{\phi}\left[ \int_{-\infty}^{\theta+\phi} g(x,\theta+\phi) dF - \int_{-\infty}^{\theta} g(x,\theta) dF \right] \\
&= \lim_{\phi\rightarrow 0} \frac{1}{\phi}\left[ \int_{-\infty}^{\theta+\phi} \left[ g(x,\theta) + \phi  h(x, \theta) + \mathcal{O}(\phi^2) \right] dF - \int_{-\infty}^{\theta} g(x,\theta) dF \right] \\
&= \lim_{\phi\rightarrow 0} \frac{1}{\phi}\left[ \int_{-\infty}^{\theta} \left[ g(x,\theta) + \phi h(x, \theta) + \mathcal{O}(\phi^2) \right] dF + \int_{\theta}^{\theta+\phi} \left[ g(x,\theta) + \phi h(x, \theta) + \mathcal{O}(\phi^2) \right] dF - \int_{-\infty}^{\theta} g(x,\theta) dF \right] \\
&= \lim_{\phi\rightarrow 0} \frac{1}{\phi}\left[ \phi \int_{-\infty}^{\theta} h(x, \theta) dF + \int_{\theta}^{\theta+\phi} \left[ g(x,\theta) + \phi h(x, \theta) \right] dF + \mathcal{O}(\phi^2) \right] \\
&= \lim_{\phi\rightarrow 0} \frac{1}{\phi}\left[ \phi \int_{-\infty}^{\theta} h(x, \theta) dF + \left[ g(\theta,\theta) + \phi h(\theta, \theta) \right] \int_{\theta}^{\theta+\phi}dF + \mathcal{O}(\phi^2) \right] \\
\end{align}

Where:

\begin{align}
h(x, \theta) &= \left.\frac{\partial g(x,y)}{\partial y}\right|_{y=\theta}
\end{align}

To finish off, notice that our $g$ has the convenient property that $g(\theta,\theta)=0$, so for this special case, we can carry on.

\begin{align}
\frac{\partial Q}{\partial \theta} &= \lim_{\phi\rightarrow 0} \frac{1}{\phi}\left[ \phi \int_{-\infty}^{\theta} h(x, \theta) dF + \mathcal{O}(\phi^2) \right] \\
&= \int_{-\infty}^{\theta} h(x, \theta) dF
\end{align}

Gosh, I can still do calculus.


# Final project

### Author: Roberto Corti

The Allen–Cahn equation (after John W. Cahn and Sam Allen) is a reaction–diffusion equation of mathematical physics which describes the process of phase separation in multi-component alloy systems, including order-disorder transitions.

The equation describes the time evolution of a scalar-valued state variable $\eta$  on a domain $\Omega=[0,1]$  during a time interval $[0,T]$, and is given (in one dimension) by:

$$
\frac{\partial \eta}{\partial t} - \varepsilon^2 \eta'' + f'(\eta) = 0, \qquad \eta'(0, t) = \eta'(1, t) = 0,\qquad\eta(x,0) = \eta_0(x)
$$

where $f$ is a double-well potential, $\eta_0$ is the initial condition, and $\varepsilon$ is the characteristic width of the phase transition.

This equation is the L2 gradient flow of the Ginzburg–Landau free energy functional, and it is closely related to the Cahn–Hilliard equation.

A typical example of double well potential is given by the following function

$$
f(\eta) = \eta^2(\eta-1)^2
$$

which has two minima in $0$ and $1$ (the two wells, where its value is zero), one local maximum in $0.5$, and it is always greater or equal than zero.

The two minima above behave like "attractors" for the phase $\eta$. Think of a solid-liquid phase transition (say water+ice) occupying the region $[0,1]$. When $\eta = 0$, then the material is liquid, while when $\eta = 1$ the material is solid (or viceversa).

Any other value for $\eta$ is *unstable*, and the equation will pull that region towards either $0$ or $1$.

Discretisation of this problem can be done by finite difference in time. For example, a fully explicity discretisation in time would lead to the following algorithm.

We split the interval $[0,T]$ in `n_steps` intervals, of dimension `dt = T/n_steps`. Given the solution at time `t[k] = k*dt`, it i possible to compute the next solution at time `t[k+1]` as

$$
\eta_{k+1} = \eta_{k} + \Delta t \varepsilon^2 \eta_k'' - \Delta t f'(\eta_k)
$$

Such a solution will not be stable. A possible remedy that improves the stability of the problem, is to treat the linear term $\Delta t \varepsilon^2 \eta_k''$ implicitly, and keep the term $-f'(\eta_k)$ explicit, that is:

$$
\eta_{k+1} - \Delta t \varepsilon^2 \eta_k'' = \eta_{k} - \Delta t f'(\eta_k)
$$

Grouping together the terms on the right hand side, this problem is identical to the one we solved in the python notebook number 9, with the exception of the constant $\Delta t \varepsilon^2$ in front the stiffness matrix.

In particular, given a set of basis functions $v_i$, representing $\eta = \eta^j v_j$ (sum is implied), we can solve the problem using finite elements by computing

$$
\big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} = \big((v_i, v_j) \eta^j_{k} - \Delta t (v_i, f'(\eta_k)\big)
$$
where a sum is implied over $j$ on both the left hand side and the right hand side. Let us remark that while writing this last version of the equation we moved from a forward Euler scheme to a backward Euler scheme for the second spatial derivative term: that is, we used $\eta^j_{k+1}$ instead of $\eta^j_{k}$.

This results in a linear system

$$
A x = b
$$

where

$$
A_{ij} = M_{ij}+ \Delta t \varepsilon^2 K_{ij} = \big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big)
$$

and

$$
b_i = M_{ij} \big(\eta_k^j - \Delta t f'(\eta_k^j)\big)
$$

where we simplified the integration on the right hand side, by computing the integral of the interpolation of $f'(\eta)$.

## Step 1

Write a finite element solver, to solve one step of the problem above, given the solution at the previous time step, using the same techniques used in notebook number 9.

In particular:

1. Write a function that takes in input a vector representing $\eta$, an returns a vector containing $f'(\eta)$. Call this function `F`.

2. Write a function that takes in input a vector of support points of dimension `ndofs` and the degree `degree` of the polynomial basis, and returns a list of basis functions (piecewise polynomial objects of type `PPoly`) of dimension `ndofs`, representing the interpolatory spline basis of degree `degree`

3. Write a function that, given a piecewise polynomial object of type `PPoly` and a number `n_gauss_quadrature_points`, computes the vector of global_quadrature_points and global_quadrature_weights, that contains replicas of a Gauss quadrature formula with `n_gauss_quadrature_points` on each of the intervals defined by `unique(PPoly.x)`

4. Write a function that, given the basis and the quadrature points and weights, returns the two matrices $M$ and $K$

## Step 2

Solve the Allen-Cahan equation on the interval $[0,1]$, from time $t=0$ and time $t=1$, given a time step `dt`, a number of degrees of freedom `ndofs`, and a polynomial degree `k`.

1. Write a function that takes the initial value of $\eta_0$ as a function, eps, dt, ndofs, and degree, and returns a matrix of dimension `(int(T/dt), ndofs)`  containing all the coefficients $\eta_k^i$ representing the solution, and the set of basis functions used to compute the solution

2. Write a function that takes all the solutions `eta`, the basis functions, a stride number `s`, and a resolution `res`, and plots on a single plot the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$, computed on `res` equispaced points between zero and one

## Step 3

Solve the problem for all combinations of

1. eps = [01, .001]

2. ndofs = [16, 32, 64, 128]

3. degree = [1, 2, 3]

3. dt = [.25, .125, .0625, .03125, .015625]

with $\eta_0 = \sin(2 \pi x)+1$.

Plot the final solution at $t=1$ in all cases. What do you observe? What happens when you increase ndofs and keep dt constant?

## Step 4 (Optional)

Instead of solving the problem explicitly, solve it implicitly, by using backward euler method also for the non linear term. This requires the solution of a Nonlinear problem at every step. Use scipy and numpy methods to solve the non linear iteration.

```python
%pylab inline
import sympy as sym
import scipy
from scipy.interpolate import *
from scipy.integrate import *
from scipy import optimize
```

    Populating the interactive namespace from numpy and matplotlib

## Step 1

1. The function `F`computes the first derivative of the double-well potential function, that is

$$ f'(\eta) = 2\eta(\eta-1)(2\eta-1) $$

```python
# Step 1.1

def F(eta):
    return 2*eta*(eta-1)*(2*eta-1)
```

2. The function `compute_basis_functions` computes the piecewise polynomial basis functions, representing the interpolatory spline basis of degree `degree`. As arguments it will require `support_points`, that is a list of points, and the `degree` of the piecewise polynomials.

```python
# Step 1.2

def compute_basis_functions(support_points, degree):
    '''
    Computes piecewise polynomial basis function.
    '''
    basis = []
    M = len(support_points)
    for i in range(M):
        c = support_points*0                                       # c = [0,0,....,0]
        c[i] = 1                                                   # c = [0,0,..1,.0]
        bi = PPoly.from_spline(splrep(support_points,c,k=degree))  # construct a piecewise polynomial from spline
        basis.append(bi)

    return basis
```

3. The function `compute_global_quadrature`, given as input the basis functions and an integer `n_gauss_quadrature_points`, computes global quadrature points and global quadrature weights used for integration.

```python
# Step 1.3

def compute_global_quadrature(basis, n_gauss_quadrature_points):
    '''
    Create a Gauss quadrature formula with n_gauss_quadrature_points, extract the intervals from basis,
    create len(x)-1 shifted and scaled Gauss quadrature formulas that can be used to integrate on each interval.
    Return the result
    '''

    intervals = unique(basis[0].x)  # Make sure every interval border is taken only once

    qp, w = numpy.polynomial.legendre.leggauss(n_gauss_quadrature_points+1)  #computes sample points and weights
                                                                             # for Gauss-Legendre quadrature for [-1,1]
    qp = (qp+1)/2      # Rescale the points and weights to work from zero to one
    w /= 2

    h = diff(intervals)
    gloabl_quadrature = array([intervals[i]+h[i]*qp for i in range(len(h))]).reshape((-1,))
    global_weights = array([w*h[i] for i in range(len(h))]).reshape((-1,))
    return gloabl_quadrature, global_weights

```

3. Given the basis, the global quadrature points and weights the function `compute_system_matrices` computes the two matrices $M=(v_i,v_j)$ and $K=(v_i',v_j')$:

```python
# Step 1.4

def compute_system_matrices(basis, gloabl_quadrature, global_weights):
    '''
    Compute the matrices M_ij = (v_i, v_j) and K_ij = (v_i', v_j') and return them
    '''
    M = len(basis)
    dbasis = []
    for i in range(M):
        dbasis.append(basis[i].derivative(1))
    Bq = array([basis[i](gloabl_quadrature) for i in range(M)]).T
    dBq = array([dbasis[i](gloabl_quadrature) for i in range(M)]).T
    M = einsum('qi, q, qj', Bq, global_weights, Bq)
    K = einsum('qi, q, qj', dBq, global_weights, dBq)
    return M, K

```

## Step 2

The function `solve_allen_cahan` returns the solutions $\eta_k^i$ at each time step $t_k$ and the set of basis functions used to compute the solution.
As initial function, I decide to use
$\eta_0 = \sin(2\pi x) +1$

```python
#initial function

def eta_0(x):
    return sin(2*pi*x)+1
```

```python
# Step 2.1

def solve_allen_cahan(eta_0_function, eps, dt, ndofs, degree):
    '''
    Forward Euler solution.
    Produce as result a matrix eta, containing the solution at all points, and basis.
    '''
    q = linspace(0,1, ndofs)                                # array of points
    basis = compute_basis_functions(q, degree)              # create basis from q
    Q, W = compute_global_quadrature(basis, degree+1)       # compute quadrature for having matrixes M and K
    M, K = compute_system_matrices(basis, Q, W)             # compute M,K matrixes
    A = M + (dt*eps**2)*K                                   # compute A matrix
    N_step = int(1/dt)                                      # compute number of steps to do
    time_interval= [i*dt for i in range(N_step+1)]          # creates list of times
    eta = zeros((len(time_interval), ndofs))                # creates matrix of eta where I'll store each eta_k(t_h)
    eta_k = eta_0_function(q)                               # initial function
    eta[0,:] = eta_k
    for t in range(1,len(time_interval)):                   # solving linear system Ax=b at each step (x=eta_k)
        b = M.dot((eta_k -dt*F(eta_k)))
        eta_k = linalg.solve(A, b)
        eta[t,:] = eta_k

    return eta, basis                                       # return eta matrix and basis functions

```

The function `plot_solution` produces a plot of the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$ computed on `resolution` equispaced points between $0$ and $1$:

```python
# Step 2.2

def plot_solution(eta, basis, stride, resolution):
    # plot eta[::stride], on x = linspace(0,1,resolution)
    x = linspace(0,1,resolution)
    B = zeros((resolution, len(basis)))
    for i in range(len(basis)):
        B[:,i] = basis[i](x)

    n_t = shape(eta)[0]
    t = ['t ='+str(round((i/(n_t-1)),2)) for i in range(n_t)]
    for eta,label_t in zip(eta[::-stride],t[::-stride]):
        plot(x, eta.dot(B.T),label=label_t)

    _= legend(fontsize='x-large')
    _= title('Allen–Cahn equation solution $\eta(x,t)$', fontsize=25)
    _= xlabel('$x$', fontsize=20)
    _= ylabel('$\eta$', fontsize=20)
```

```python
figure(figsize=(12,6))

eta, basis = solve_allen_cahan(eta_0, eps=0.01, dt=0.1, ndofs=64, degree=1)

_= plot_solution(eta, basis, stride=4, resolution=1025)
```

## Step 3

The functions `plot_increasing_ndofs`, `plot_increasing_dt`, `plot_increasing_degree`, `plot_increasing_eps` will plot the final solution $\eta(x, t=1)$ as increasing, for each associate function, `ndofs`, `dt`, `degree` and `eps`. I decide to vary the parameters at the following values:

```
eps = [01, .001]

ndofs = [16, 32, 64, 128]

degree = [1, 2, 3]

dt = [.25, .125, .0625, .03125, .015625]
```

```python
#parameters

resolution = 1025
eps = [.01, .001]
ndofs = [16, 32, 64, 128]
degree = [1, 2, 3]
dt = [.25, .125, .0625, .03125, .015625]
```

```python
# function for increase ndofs and fixed dt,eps,degree

def plot_increasing_ndofs(eps, degree, dt):
    ndofs = [16, 32, 64, 128]
    fig = figure(figsize=(22,11))
    for i in range(len(ndofs)):
        eta, basis = solve_allen_cahan(eta_0, eps, dt, ndofs[i], degree)
        subplot(2,2,i+1)
        plot_solution(eta, basis, int(1/dt+1), 1025)
        title('ndofs = '+ str(ndofs[i]), fontsize=18)
        legend(fontsize='x-large')
        xlabel('x', fontsize=10)
        ylabel('$\eta$', fontsize=10)

    print('eps = '+str(eps)+', deg = '+str(degree)+', dt = '+ str(dt))

def plot_increasing_dt(eps, degree, ndofs):
    dt = [.25, .125, .0625, .03125, .015625]
    fig = figure(figsize=(18,28))
    for i in range(len(dt)):
        eta, basis = solve_allen_cahan(eta_0, eps, dt[i], ndofs, degree)
        subplot(5,3,i+1)
        plot_solution(eta, basis, int(1/dt[i]+1), 1025)
        title('eps = '+str(eps)+', degree = '+str(degree)+', ndofs = '+ str(ndofs)
              +', dt = '+ str(dt[i]), fontsize=12)
        legend(fontsize='x-large')
        xlabel('x', fontsize=10)
        ylabel('$\eta$', fontsize=10)

def plot_increasing_eps(dt, degree, ndofs):
    eps = [.1, .01, .001]
    fig = figure(figsize=(16,7))
    for i in range(len(eps)):
        eta, basis = solve_allen_cahan(eta_0, eps[i], dt, ndofs, degree)
        subplot(1,3,i+1)
        plot_solution(eta, basis, int(1/dt+1), 1025)
        title('eps = '+ str(eps[i]), fontsize=15)
        legend(fontsize='x-large')
        xlabel('x', fontsize=10)
        ylabel('$\eta$', fontsize=10)

    print('dt = '+str(dt)+', degree = '+str(degree)+', ndofs = '+ str(ndofs))
```

```python
plot_increasing_ndofs(eps[1], degree[0], dt[3])
```

As first observation, I note that the solution becomes smoother when increasing the number of degrees of freedom (`ndofs`) and keeping `dt`, `eps` and `degree` constant. This behaviour can be explained by the fact that as we are increasing the `support_points` in `compute_basis_function` our polynomial aproximation becomes more close to the solution $\eta(x,t=1)$

```python
for deg in degree:
    plot_increasing_dt(eps[1], deg, ndofs[2])
```

From these plots it is possible to notice that the solution tends to $0$ for $x<0.5$ and to $1$ for $x>0.5$.  However, for large `dt` (`dt=0.25` and `dt=0.125`) the solution is unstable since the forward Euler method is conditionally stable.

## Step 4

By applying Backward Euler method we consider the following equation

 $$ \frac{\partial \eta}{\partial t} \simeq \frac{(\eta_{k+1} - \eta_k)}{\Delta t}  = \varepsilon^2 \eta_{k+1} '' - f'(\eta_{k+1}). $$

 Then, using the same discretiazation procedure done in **Step 2** and with our basis representation $\eta=\eta^{j}v_j$ we end up with the following non linear equation

 $$ \big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} + \Delta t (v_i, f'(\eta^j_{k+1})) - (v_i, v_j) \eta^j_{k} = 0.  $$

 Defining the matrix $A_{ij} = M_{ij} + \Delta t \varepsilon^2 K_{ij}$, where $ M_{ij} = (v_i, v_j) $ and $ K_{ij} = (v_i', v_j') $, the equation can be written in this compact form

 $$ \boxed {F_i(\eta^j_{k+1}) := A_{ij}\eta^j_{k+1} - M_{ij}\eta^j_k + \Delta t M_{ij} f'(\eta^j_{k+1}) = 0} $$

 Thus, in order to implement a numerical method I have to find the value of $\eta^j_{k+1}$ that satisfies $F_i(\eta^j_{k+1}) =0$. This is equivalent to find for every time step the root of the given function.

 In my implementation `solve_solve_allen_cahan_non_linear` I decide to use methods provided by `scipy`. After using several methods I find out that the method [`optimize.root`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.root.html) results the most efficient one. This function requires as arguments the function $F_i$ and its realative jacobian matrix $\partial_j F_i$. As did in `solve_solve_allen_cahan` the outputs are the solution, that is stored in a matrix of dimension (`time_steps`, `ndofs`), and the `basis` functions.

```python
# Step 4

def F_prime(eta):
    return 12*eta*(eta-1) + 2       #second derivative of f. I need for Jacobian matrix for solving non linear eq.

def solve_allen_cahan_non_linear(eta_0_function, eps, dt, ndofs, degree):
    '''
    Solution with backward Euler.
    This requires the solution of a Nonlinear problem at every step.
    Used scipy and numpy methods to solve the non linear iteration.
    '''
    q = linspace(0,1, ndofs)                            # array of points
    basis = compute_basis_functions(q, degree)          # create basis from q
    Q, W = compute_global_quadrature(basis, degree+1)   # compute quadrature for having matrixes M and K
    M, K = compute_system_matrices(basis, Q, W)         # compute M,K matrixes
    A = M + (dt*eps**2)*K                               # compute A matrix
    N_step = int(1/dt)                                  # compute number of steps to do
    time_interval= [i*dt for i in range(N_step+1)]      # create list of time steps
    eta = zeros((len(time_interval), ndofs))            # create eta_matrix
    eta_k = eta_0_function(q)                           # initial_solution
    eta[0,:] = eta_k

    for t in range(1,len(time_interval)):

        def f(z):                                       # F_i
            return A.dot(z)+dt*M.dot(F(z))-M.dot(eta_k)

        def Jacobian(z):                                # Ð_j F_i
            S = zeros((len(z), len(z)))
            for i in range(len(z)):
                for j in range(len(z)):
                    S[i,j] = dt*M[i,j]*F_prime(z[j])
            return A + S

        eta_k = (optimize.root(f, eta_k,jac=Jacobian, method='hybr')).x   # non-linear method solutor call
        eta[t,:] = eta_k                                                  # store eta solution for time-step t

    return eta, basis
```

```python
figure(figsize=(12,6))

eta, basis = solve_allen_cahan_non_linear(eta_0, eps=0.01, dt=.03125, ndofs=32, degree=1)

_= plot_solution(eta, basis, 12, 1025)
```

```python
figure(figsize=(16,5))

stride = 2
resolution = 1025

plt.subplot(1, 2, 1)
eta, basis = solve_allen_cahan(eta_0, eps=0.01, dt=0.25, ndofs=32, degree=1)
_= plot_solution(eta, basis, stride, resolution)
_= title("Forward Euler", fontsize=17)

plt.subplot(1, 2, 2)
eta, basis = solve_allen_cahan_non_linear(eta_0, eps=0.01, dt=0.25, ndofs=32, degree=1)
_ = plot_solution(eta, basis, stride, resolution)
_ = title("Backward Euler",  fontsize=17)
```

For large values of `dt`(i.e. `0.25`) the linear solution is unstable while the nonlinear solution converges. This result is expected, since we know that backward Euler is stable for any time-step value (see [Ordinary Differential Equations](https://people.sissa.it/~grozza/wp-content/uploads/2020/01/Slides_ODE_EN.pdf) )


```python
# symbols and expressions
import sympy
```

```python
x*2
```

```python
# we have to say that x is a symbol
x = sympy.var('x')
```

```python
x*x
```

    x**2

```python
from sympy import *
```

```python
x = symbols('x')
```

```python
f = x*x*x
```

```python
print(f)
```

    x**3

```python
# will cover in detail later
diff(f,x)
```

    3*x**2

```python
x, y, z, t = symbols('x y z t')
```

```python
expr = x*x+y*z+t*x*z
```

```python
diff(expr, z)
```

    t*x + y

```python
# caveat - possible to confuse and go mad if not careful
x = symbols('y')
```

```python
x*x
```

    y**2

```python
x = symbols('areyoucrazy')
```

```python
x*x
```

    areyoucrazy**2

```python
# == signs compares the expressions exactly, not symbolically.
# For comparisons use either equals or subtraction
x = symbols('x')
(x+2)**2 == x**2 + 4*x + 4
```

    False

```python
a = (x+2)**2
b = x**2 + 4*x + 4
```

```python
c = a-b
```

```python
print(c)
```

    -x**2 - 4*x + (x + 2)**2 - 4

```python
simplify(c)
```

    0

```python
a.equals(b)
```

    True

```python
# numerals
x + 1/10
```

    x + 0.1

```python
1/10
```

    0.1

```python
# constructing rational objects explicitly
Rational(1,10)
```

    1/10

```python
x + Rational(1,20)
```

    x + 1/20

```python
expr = x**y
```

```python
expr_x2 = expr.subs(x,2)
```

```python
print(expr_x2)
```

    2**y

```python
expr_yx = expr.subs(y, x)
```

```python
print(expr_yx)
```

    x**x

```python
expr_x2_y3 = expr.subs([(x,2),(y,3)])
```

```python
print(expr_x2_y3)
```

    8

```python
# evaluating expressions

```

```python
expr = sqrt(8)
```

```python
print(expr)
```

    2*sqrt(2)

```python
expr.evalf()
```

    2.82842712474619

```python
pi
```

    pi

```python
print(pi)
```

    pi

```python
pi.evalf(10)
```

    3.141592654

```python
pi.evalf(100)
```

    3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117068

```python
# simplifying expression
# by default the expressions entered by the user are not simplified because
# the user might want the expressions displayed in different formats.
# For example, some times it is useful to have the polynomials in terms of factors.

```

```python
(x+2)**2 - 4*x
```

    -4*x + (x + 2)**2

```python
# we can see the get the polynomial form by using expand
expand((x+2)**2)
```

```python
simplify((x+2)**2 - 4*x)
```

```python
# pretty printing for powers, exponentials, integrals, derivatives etc.
# IPython's QTConsole uses LATEX if installed. Otherwise, uses matplotlib engine.
```

```python
# use init_printing() for enabling pretty printing.
# This will load the default printing available
# unicode type printing is used on my laptop
from sympy import init_printing
init_printing()
```

```python
(x+2)**2 - 4*x
```

```python
Integral(sqrt(1/x), x)
```

```python
# symbols and functions
x, y, z = symbols('x y z', integer=True)
```

```python
f, g = symbols('f g', cls=Function)
```

```python
type(x)
```

    sympy.core.symbol.Symbol

```python
type(f)
```

    sympy.core.function.UndefinedFunction

```python
f = 2*x*y**2 + x**t
```

```python
type(f)
```

    sympy.core.add.Add

```python
g = 2*x*y*z**t
```

```python
type(g)
```

    sympy.core.mul.Mul

```python

```


# Public-Key Encryption (Optional)

**CS1302 Introduction to Computer Programming**
___

This notebook will introduce the idea of public-key (asymmetric key) cryptography called the [RSA algorithm](https://en.wikipedia.org/wiki/RSA_(cryptosystem)).

## Motivation

**What is public key encryption?**

Suppose there is one receiving trying to receive private messsages from multiple senders.

---

**Definition** (Asymmetric key cipher)

Every sender uses the same public key $k_e$ to encrypt a plaintext $x$ to a ciphertext $y$ as

$$
y = E(x, k_e).
$$ (encrypt)

The receiver with the private key $k_d$ decrypts the ciphertext back to the plaintext as

$$
x = D(y, k_d).
$$ (decrypt)

$k_e, k_d, E, D$ is chosen to ensure

- Decryptability: The receiver can recover the plaintext.
- Secrecy: Anyone with only the public key but not the private key cannot recover the plaintext.

---

**Exercise** (Optional) What is the benefit of asymmetric key cipher over symmetric key cipher?

- Easy to share keys: Encryption key can be shared to all senders publicly.
- Easy to store keys: Only the receiver needs to know the private key.

**Is public key encryption possible?**

Unfortunately, public key encryption is an invertible function given a public key:

---

**Proposition**

The plaintext can be recovered from the ciphertext using the public key, even without the private key.

---

**Exercise** (Optional) Prove the above proposition.

YOUR ANSWER HERE

**How to make public key encryption secure?**

We can make it computationally infeasible to invert $E(\cdot, k_e)$ unless with the private key $k_d$ is available. Such a function is called the [trapdoor function](https://en.wikipedia.org/wiki/Trapdoor_function#:~:text=A%20trapdoor%20function%20is%20a,are%20widely%20used%20in%20cryptography.). An example of a trapdoor function is:

---

**Proposition** (Integer factorization)

Computing the product of of two prime numbers $p$ and $q$, i.e.,

$$
(p,q)\mapsto n,
$$

is a trapdoor function because integer factorization (computing $p$ and $q$ from $n$) is [co-NP](https://en.wikipedia.org/wiki/Integer_factorization) (difficult).

---

As the size of $p$ and $q$ increases, the time required to factor $n$ increases dramatically as illustrated [here](https://www.khanacademy.org/computing/computer-science/cryptography/modern-crypt/pi/time-complexity-exploration).

## RSA Algorithm

**How to encrypt/decrypt?**

The encryption and decryption use modulo exponentiation instead of addition:

$$
\begin{align}
E(x, k_e) &:= x^e \bmod n && \text{where }k_e:=(e,n)\\
D(c, k_d) &:= c^d \bmod n && \text{where }k_d:=(d,n),
\end{align}
$$

and $e$, $d$, and $n$ are positive integers.

Computing exponentiation can be fast using [repeated squaring](https://en.wikipedia.org/wiki/Exponentiation_by_squaring). The built-in function `pow` already has an efficient implementation:

```python
%%timeit
x, e, n = 3, 2 ** 1000000, 4
pow(x, e, n)
```

**Exercise** (Optional) Implement you own `modulo_power` using a recusion.

```python
%%timeit
def modulo_power(x, e, n):
    # YOUR CODE HERE
    raise NotImplementedError()

x, e, n = 3, 2 ** 1000000, 4
pow(x, e, n)
```

**How to ensure decryptability?**

For $x = D(E(x, k_e), k_d) = x^{ed} \bmod n$, we need $0\leq x < n$ and

$$
x^{ed} \equiv x \mod n.
$$ (decryptability)

**Exercise** (Optional) Derive the above condition using $(a^c \bmod b) = (a\bmod b)^c \bmod b$.

$$
\begin{align}
x &= D(E(x, k_e), k_d) \\
&= (x^{e} \bmod n)^{d} \bmod n \\
&= x^{ed} \bmod n.
\end{align}
$$

RSA makes use of the following result to choose $(e, d, n)$:

---

**Theorem** (Fermat's little Theorem)

If $p$ is prime, then

$$
x^{p-1}\equiv 1 \mod p
$$ (fermat)

for any integer $x$.

---

There are elegant and elementary [combinatorial proofs](https://en.wikipedia.org/wiki/Proofs_of_Fermat%27s_little_theorem#Combinatorial_proofs).

Since {eq}`fermat` implies $x^p = x \bmod p$, can we choose
- $n=p$ and
- $ed=p$

to satisfies {eq}`decryptability`?

No because the private key can then be easily computed from the public key: $d = n/e$.

Alternatively, by raising {eq}`fermat` to the power of any integer $m$,

$$
x^{m(p-1)} \equiv 1 \mod p.
$$ (fermat-m)

Can we have $n=p$ and $ed \equiv 1 \bmod p-1$?

No because $d$ is the modular multiplicative inverse of $e$, which is [easy to compute](https://en.wikipedia.org/wiki/Modular_multiplicative_inverse#Computation), e.g., using `pow` with an exponent of `-1`. In particular, for prime modulus here, the inverse is $d=e^{p-2}\bmod p$:

```python
e, n = 3, 7
d = pow(e, -1, n)
d, e * d % n == 1 and d == e ** (n - 2) % n
```

**How to make it difficult to compute $d$?**

RSA makes use of the hardness of factoring a product of large primes to create the desired trapdoor function.

In particular, with $m(p-1)$ in {eq}`fermat-m` being the least common multiple $\operatorname{lcm}(p-1,q-1)$ for another prime number $q$, we have

$$
\begin{align}
x^{\operatorname{lcm}(p-1, q-1)} &\equiv 1 \mod p && \text{and}\\
x^{\operatorname{lcm}(p-1, q-1)} &\equiv 1 \mod q && \text{by symmetry.}
\end{align}
$$

This implies $x^{\operatorname{lcm}(p-1, q-1)} - 1$ is divisible by both $p$ and $q$, and so

$$
x^{\operatorname{lcm}(p-1, q-1)} \equiv 1 \mod p q.
$$

Raising both sides to the power of any positive integer $m$ give:

---

**Proposition** (RSA)

If $p$ and $q$ are prime, then

$$
x^{\overbrace{m \operatorname{lcm}(p-1, q-1)}^{ed - 1}} \equiv 1 \mod \overbrace{p q}^n
$$ (rsa)

for any integer $x$. This implies {eq}`decryptability` with by choosing $n = pq$ and

$$
ed \equiv 1 \mod \operatorname{lcm}(p-1, q-1).
$$ (ed)

---

Although $d$ is still the modulo multiplicative inverse of $e$, it is with respect to $\operatorname{lcm}(p-1, q-1)$, which is not easy to compute without knowing the factors of $n$, namely $p$ and $q$. It can be shown that computing $d$ is [as hard as](https://crypto.stackexchange.com/questions/16036/is-knowing-the-private-key-of-rsa-equivalent-to-the-factorization-of-n) computing $\operatorname{lcm}(p-1, q-1)$ or factoring $n$.

**How to generate the public key and private key?**

By {eq}`ed`, we can compute $d$ as the modulo multiplicative inverse of $e$. How to choose $e$ then?

We can choose any $e \in \{1, \dots, \operatorname{lcm}(p-1, q-1)\}$ such that $e$ does not divide $\operatorname{lcm}(p-1, q-1)$.

**Exercise** (Optional) For $e$ to have the modulo multiplicative inverse, it should not divide $\operatorname{lcm}(p-1, q-1)$. Why?

YOUR ANSWER HERE

The following function randomly generate the `e, d, n` for some given prime numbers `p` and `q`:

```python
from math import gcd
from random import randint

def get_rsa_keys(p, q):
    n = p * q
    lcm = (p - 1) * (q - 1) // gcd(p - 1, q - 1)
    while True:
        e = randint(1, lcm - 1)
        if gcd(e, lcm) == 1:
            break
    d = pow(e, -1, lcm)
    return e, d, n, lcm
```

Note that

$$
\operatorname{lcm}(p-1, q-1) = \frac{(p-1)(q-1)}{\operatorname{gcd}(p-1, q-1)}.
$$

As an example, if we choose two prime numbers $p=17094589121$ and $q=1062873761$:

```python
e, d, n, lcm = get_rsa_keys(17094589121, 1062873761)
e, d, n, e * d % lcm == 1
```

The integer $1302$ can be encrypted as follows:

```python
x = 1302
c = pow(x, e, n)
print(f'The plain text "{x}" has been encrypted into "{c}".')
```

With the private key $k_d$, the ciphertext can be decrypted easily as

```python
output = pow(c, d, n)
print(f'The cipher "{c}" has been decrypted into "{output}".')
```

**Exercise** (optional) Using the `rsa` module, [generate a RSA key pair](https://stuvel.eu/python-rsa-doc/usage.html#generating-keys) with suitable length and then use it to encrypt and decrypt your own message. You can install the module as follows:

```python
!pip install rsa
```


# Métodos Iterativos para sistemas de ecuaciones lineales

```python
import numpy as np
from scipy.linalg import solve_triangular
import matplotlib.pyplot as plt
from time import time
```

```python
def jacobi(A, b, n_iter=50, x_0=None):
    """
    Solve Ax=b using Jacobi method

    Parameters
    -----------
    A      : (n, n) array
             A matrix
    b      : (n, ) array
             RHS vector
    n_iter : int
             Number of iterations
    x_0    : (n, ) array
             Initual guess

    Returns
    -------
    X      : (n_iter + 1, n) array
             Matrix with approximation at each iteration
    """
    n = A.shape[0] # Matrix size
    X = np.zeros((n_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    D = np.diag(A) # Diagonal of A (only keep a vector with diagonal)
    # Inverse of D. Compute reciprocal of vector elements and then fill diagonal matrix
    D_inv = np.diag(1 / D) # This avoid inverse computation "O(n) instead O(n^3)"
    LU = (A - np.diag(D)) # A - D = L + U (here D is a matrix) - Rembember LU != LU of PA=LU or A=LU
    # Jacobi iteration
    for k in range(n_iter):
        X[k+1] = np.dot(D_inv, (b - np.dot(LU, X[k])))
    return X
```

```python
def gaussSeidel(A, b, n_iter=50, x_0=None):
    """
    Solve Ax=b using Gauss-Seidel method

    Parameters
    -----------
    A      : (n, n) array
             A matrix
    b      : (n, ) array
             RHS vector
    n_iter : int
             Number of iterations
    x_0    : (n, ) array
             Initual guess

    Returns
    -------
    X      : (n_iter + 1, n) array
             Matrix with approximation at each iteration
    """
    n = A.shape[0] # Matrix size
    X = np.zeros((n_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    LD = np.tril(A) # Get lower triangle with main diagonal (L + D)
    U = A - LD # Upper triangle
    # Gauss-Seidel iteration
    for k in range(n_iter):
        X[k+1] = solve_triangular(LD, b - np.dot(U, X[k]), lower=True)
    return X
```

```python
def SOR(A, b, w=1.05, n_iter=50, x_0=None):
    """
    Solve Ax=b using SOR(w) method

    Parameters
    -----------
    A      : (n, n) array
             A matrix
    b      : (n, ) array
             RHS vector
    w      : float
             Omega parameter
    n_iter : int
             Number of iterations
    x_0    : (n, ) array
             Initual guess

    Returns
    -------
    X      : (n_iter + 1, n) array
             Matrix with approximation at each iteration
    """
    n = A.shape[0] # Matrix size
    X = np.zeros((n_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    L = np.tril(A, k=-1) # Get lower triangle
    U = np.triu(A, k=1) # Get Upper triangle
    D = A - U - L
    # SOR
    for k in range(n_iter):
        X[k+1] = solve_triangular(w * L + D, w * b + np.dot((1 - w) * D - w * U, X[k]), lower=True)
    return X
```

```python
def error(X, x):
    """
    Compute error of approximation at each iteration

    Parameters
    ----------
    X       : (m, n) array
             Matrix with approximation at each iteration
    x       : (n, ) array
             Solution of system

    Returns
    -------
    X_err   : (m, ) array
             Error vector
    """
    X_err = np.linalg.norm(X - x, axis=1, ord=np.inf)
    return X_err
```

# Ejemplo Apunte

Resolver

\begin{equation}
    \begin{split}
        u + 3v & = -1 \\
        5u + 4v & = 6
    \end{split}
\end{equation}

Primero, lo expresamos de forma matricial:
\begin{equation}
    \begin{bmatrix}
        5 & 4 \\
        1 & 3
    \end{bmatrix}
    \begin{bmatrix} u \\ v \end{bmatrix}
    =
    \begin{bmatrix} 6 \\ -1 \end{bmatrix}.
\end{equation}

Notar que se intercambiaron filas para asegurar que $A$ sea **estrictamente diagonal dominante**. Comprobemos que se llega a la solución (o cerca) en la iteración que se indica.

```python
A_ap = np.array([[5, 4], [1, 3]])
b_ap = np.array([6, -1])
```

## Solución Analítica

```python
x_ap = np.linalg.solve(A_ap, b_ap)
x_ap
```

    array([ 2., -1.])

## Jacobi

```python
x_ap_j = jacobi(A_ap, b_ap, 50)
x_ap_j[-1]
```

    array([ 2., -1.])

## Gauss-Seidel

```python
x_ap_g = gaussSeidel(A_ap, b_ap, 17)
x_ap_g[-1]
```

    array([ 2., -1.])

## SOR($\omega$)

```python
x_ap_s = SOR(A_ap, b_ap, 1.09, 9)
x_ap_s[-1]
```

    array([ 2., -1.])

Efectivamente *SOR* obtiene la solución en menos iteraciones.

# Otro ejemplo

```python
A_2 = np.array([
        [3, -1, 0, 0, 0, 0.5],
        [-1, 3, -1, 0, 0.5, 0],
        [0, -1, 3, -1, 0, 0],
        [0, 0, -1, 3, -1, 0],
        [0, 0.5, 0, -1, 3, -1],
        [0.5, 0, 0, 0, -1, 3]
])
b_2 = np.array([2.5, 1.5, 1., 1., 1.5, 2.5])
```

## Solución de referencia

```python
x_2 = np.linalg.solve(A_2, b_2)
x_2
```

    array([1., 1., 1., 1., 1., 1.])

## Jacobi

```python
X_2_jac = jacobi(A_2, b_2)
X_2_jac[-1]
```

    array([1., 1., 1., 1., 1., 1.])

## Gauss-Seidel

```python
X_2_gss = gaussSeidel(A_2, b_2)
X_2_gss[-1]
```

    array([1., 1., 1., 1., 1., 1.])

## SOR($\omega$)

```python
def error(X, x):
    return np.linalg.norm(X - x, axis=1, ord=np.inf)
```

Podemos buscar el parámetro $\omega$, analizando el error para un par de iteraciones

```python
n_w = 20
sor_err_2 = np.zeros(n_w)
w_s = np.linspace(1, 1.3, n_w)
for i in range(n_w):
    X_sor_tmp = SOR(A_2, b_2, w_s[i], 5)
    err_tmp_2 = error(X_sor_tmp, x_2)
    sor_err_2[i] = err_tmp_2[-1]
    print("i: %d \t w: %f \t error: %f" % (i, w_s[i], sor_err_2[i]))
```

    i: 0 	 w: 1.000000 	 error: 0.013793
    i: 1 	 w: 1.015789 	 error: 0.012247
    i: 2 	 w: 1.031579 	 error: 0.010732
    i: 3 	 w: 1.047368 	 error: 0.009437
    i: 4 	 w: 1.063158 	 error: 0.008973
    i: 5 	 w: 1.078947 	 error: 0.008508
    i: 6 	 w: 1.094737 	 error: 0.008036
    i: 7 	 w: 1.110526 	 error: 0.007551
    i: 8 	 w: 1.126316 	 error: 0.007048
    i: 9 	 w: 1.142105 	 error: 0.006524
    i: 10 	 w: 1.157895 	 error: 0.005972
    i: 11 	 w: 1.173684 	 error: 0.005390
    i: 12 	 w: 1.189474 	 error: 0.004774
    i: 13 	 w: 1.205263 	 error: 0.006131
    i: 14 	 w: 1.221053 	 error: 0.007827
    i: 15 	 w: 1.236842 	 error: 0.009566
    i: 16 	 w: 1.252632 	 error: 0.011351
    i: 17 	 w: 1.268421 	 error: 0.013182
    i: 18 	 w: 1.284211 	 error: 0.015062
    i: 19 	 w: 1.300000 	 error: 0.016993

```python
plt.plot(w_s, sor_err_2, 'bd')
plt.yscale('log')
plt.grid(True)
plt.show()
```

Mirando los valores del gráfico obtenemos que $\omega=1.189474$

```python
min_pos_2 = np.argmin(sor_err_2)
X_2_sor = SOR(A_2, b_2, w_s[min_pos_2])
X_2_sor[-1]
```

    array([1., 1., 1., 1., 1., 1.])

## Convergencia de métodos

```python
# Error
e_jac = error(X_2_jac, x_2)
e_gss = error(X_2_gss, x_2)
e_sor = error(X_2_sor, x_2)
```

```python
n_jac = np.arange(e_jac.shape[-1])
n_gss = np.arange(e_gss.shape[-1])
n_sor = np.arange(e_sor.shape[-1])
plt.plot(n_jac, e_jac, 'ro', label="Jacobi")
plt.plot(n_gss, e_gss, 'bo', label="Gauss-Seidel")
plt.plot(n_sor, e_sor, 'go', label="SOR")
plt.yscale('log')
plt.grid(True)
plt.legend()
plt.show()
```

# Ejemplo aleatorio

```python
def ddMatrix(n):
    """
    Randomly generates an n x n strictly diagonally dominant matrix A.

    Parameters
    ----------
    n       : int
              Matrix size

    Returns
    -------
    A       : (n, n) array
             Strictly diagonally dominant matrix
    """
    A = np.random.random((n,n))
    deltas = 0.5 * np.random.random(n)
    row_sum = A.sum(axis=1) - np.diag(A)
    np.fill_diagonal(A, row_sum+deltas)
    return A
```

```python
n = 50
A_r = ddMatrix(n)
b_r = np.random.rand(n)
```

## Solución Analítica

```python
x_r = np.linalg.solve(A_r, b_r)
```

## Jacobi

```python
X_r_jac = jacobi(A_r, b_r, n_iter=100)
```

## Gauss-Seidel

```python
X_r_gss = gaussSeidel(A_r, b_r, n_iter=50)
```

## SOR($\omega$)

Para buscar $\omega$ de *SOR*, se realiza un par de iteraciones y se analiza el error. Vamos a elegir el que tenga menor error...

```python
n_w = 20
w_s = np.linspace(1, 1.3, n_w)
sor_err_r = np.zeros(n_w)
for i in range(n_w):
    X_sor_tmp = SOR(A_r, b_r, w_s[i], 5)
    sor_err_r[i] = np.linalg.norm(X_sor_tmp[-1] - x_r, np.inf)
    print("i: %d \t w: %f \t error: %f" % (i, w_s[i], sor_err_r[i]))
```

    i: 0 	 w: 1.000000 	 error: 0.000014
    i: 1 	 w: 1.015789 	 error: 0.000016
    i: 2 	 w: 1.031579 	 error: 0.000019
    i: 3 	 w: 1.047368 	 error: 0.000027
    i: 4 	 w: 1.063158 	 error: 0.000038
    i: 5 	 w: 1.078947 	 error: 0.000049
    i: 6 	 w: 1.094737 	 error: 0.000063
    i: 7 	 w: 1.110526 	 error: 0.000078
    i: 8 	 w: 1.126316 	 error: 0.000101
    i: 9 	 w: 1.142105 	 error: 0.000126
    i: 10 	 w: 1.157895 	 error: 0.000155
    i: 11 	 w: 1.173684 	 error: 0.000188
    i: 12 	 w: 1.189474 	 error: 0.000226
    i: 13 	 w: 1.205263 	 error: 0.000267
    i: 14 	 w: 1.221053 	 error: 0.000314
    i: 15 	 w: 1.236842 	 error: 0.000365
    i: 16 	 w: 1.252632 	 error: 0.000423
    i: 17 	 w: 1.268421 	 error: 0.000486
    i: 18 	 w: 1.284211 	 error: 0.000555
    i: 19 	 w: 1.300000 	 error: 0.000631

```python
min_pos_r = np.argmin(sor_err_r)
X_r_sor = SOR(A_r, b_r, w_s[min_pos_r], n_iter=50)
```

## Comparación de Error

```python
e_r_jac = error(X_r_jac, x_r)
e_r_gss = error(X_r_gss, x_r)
e_r_sor = error(X_r_sor, x_r)
```

```python
n_r_jac = np.arange(e_r_jac.shape[-1])
n_r_gss = np.arange(e_r_gss.shape[-1])
n_r_sor = np.arange(e_r_sor.shape[-1])
plt.plot(n_r_jac, e_r_jac, 'ro', label="Jacobi")
plt.plot(n_r_gss, e_r_gss, 'bo', label="Gauss-Seidel")
plt.plot(n_r_sor, e_r_sor, 'go', label="SOR")
plt.yscale('log')
plt.grid(True)
plt.legend()
plt.show()
```

Para analizar la convergencia de los métodos vamos a utilizar el *radio espectral* $\rho(A)$.

```python
def spectralRadius(A):
    """
    Compute spectral radius of A

    Parameters
    ----------
    A       : (n, n) array
             A Matrix

    Returns
    -------
    rho     : float
             Spectral radius

    """
    ev = np.linalg.eigvals(A) # Compute eigenvalues
    rho = np.max(np.abs(ev)) # Largest eigenvalue in magnitude
    return rho
```

```python
L = np.tril(A_r, k=-1)
U = np.triu(A_r, k=1)
D = A_r - L - U
```

```python
M_r_jac = np.dot(np.linalg.inv(D), L + U)
M_r_gss = np.dot(np.linalg.inv(L + D), U)
```

```python
sr_jac = spectralRadius(M_r_jac)
sr_gss = spectralRadius(M_r_gss)
print(sr_jac, sr_gss)
```

    0.989112355268985 0.21169440485989235

En este caso vemos que *Gauss-Seidel* converge mucho más rápido que Jacobi.

# Teorema de los círculos de Gershgorin

```python
def diskGershgorin(A):
    """
    Compute Gershgorin disks.

    Parameters
    ----------
    A       : (n, n) array
              Matrix

    Returns
    -------
    disks   : (n, 2) array
             Gershgorin disks.
             First column is the center = |a_{ii}| and second column is radius = \sum_{i\neq j} |a_{ij}|.
    """
    n = A.shape[0]
    disks = np.zeros((n, 2)) # First column is center and second is radius
    for i in range(n):
        c = A[i, i] # Center
        R = np.sum(np.abs(A[i])) - np.abs(c) # Sum of absolute values of rows without diagonal
        disks[i, 0] = c
        disks[i, 1] = R
    return disks
```

```python
def circles(disks):
    """
    Return circles.

    Parameters
    ----------
    disks   : (n, 2) array
             Gershgorin disks.

    Returns
    -------
    C       : (n, 100, 2) array
             Circles to plot.

    """
    n = disks.shape[0]
    N = 100
    theta = np.linspace(0, 2*np.pi, N)
    C = np.zeros((n, N, 2))
    for i in range(n):
        C[i, :, 0] = disks[i, 0] + disks[i, 1] * np.cos(theta)
        C[i, :, 1] = disks[i, 1] * np.sin(theta)
    return C
```

```python
def plotCircles(A):
    """
    Plot Gershgorin disks and eigenvalues of A.

    Parameters
    ----------
    A       : (n, n) array
              Matrix

    Returns
    -------
    None
    """
    disks = diskGershgorin(A)
    circs = circles(disks)
    ev = np.linalg.eigvals(A)
    for i in range(disks.shape[0]):
        plt.fill(circs[i, :, 0], circs[i, :, 1], alpha=.5)
        plt.plot(ev[i], 0, 'bo')
    plt.grid(True)
    plt.axis('equal')
    plt.show()
```

```python
A1 = np.array([
    [8, 1, 0],
    [1, 4, 0.1],
    [0, 0.1, 1]
])
```

```python
A2 = np.array([
    [10, -1, 0, 1],
    [0.2, 8, 0.2, 0.2],
    [1, 1, 2, 1],
    [-1, -1, -1, -11]
])
```

```python
plotCircles(A1)
```

```python
plotCircles(A2)
```

## Comentario sobre convergencia

Para un método iterativo de la forma $\mathbf{x}_{k+1}=M\mathbf{x}_k + \mathbf{\hat{b}}$, podemos usar el *radio espectral* $\rho(M)$ y así estudiar la convergencia del método. Esto implica resolver el problema
\begin{equation}
    M\mathbf{v} = \lambda \mathbf{v},
\end{equation}

para obtener los valores de $\lambda$, con una complejidad aproximada de $\sim  O(n^3)$. Si utilizamos el *Teorema de los círculos de Gershgorin*, podríamos obtener una *cota* de los valores propios con aproximadamente $\sim O(n^2)$ operaciones.

# Complejidad

Para ver los tiempos de cómputo y comparar el número de iteraciones, utilizaremos la tercera forma de representar los métodos.

```python
def jacobi2(A, b, n_iter=50, tol=1e-8, x_0=None):
    """
    Solve Ax=b using Jacobi method

    Parameters
    -----------
    A      : (n, n) array
             A matrix
    b      : (n, ) array
             RHS vector
    n_iter : int
             Number of iterations
    tol    : float
             Tolerance
    x_0    : (n, ) array
             Initual guess

    Returns
    -------
    X      : (n_iter + 1, n) array
             Matrix with approximation at each iteration
    """
    n = A.shape[0] # Matrix size
    X = np.zeros((n_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    D = np.diag(A) # Diagonal of A (only keep a vector with diagonal)
    D_inv = np.diag(1 / D) # Inverse of D
    r = b - np.dot(A, X[0]) # Residual vector
    # Jacobi iteration
    for k in range(n_iter):
        X[k+1] = X[k] + np.dot(D_inv, r)
        r = b - np.dot(A, X[k+1]) # Update residual
        if np.linalg.norm(r) < tol: # Stop criteria
            X = X[:k+2]
            break
    return X
```

```python
def gaussSeidel2(A, b, n_iter=50, tol=1e-8, x_0=None):
    """
    Solve Ax=b using Gauss-Seidel method

    Parameters
    -----------
    A      : (n, n) array
             A matrix
    b      : (n, ) array
             RHS vector
    n_iter : int
             Number of iterations
    tol    : float
             Tolerance
    x_0    : (n, ) array
             Initual guess

    Returns
    -------
    X      : (n_iter + 1, n) array
             Matrix with approximation at each iteration
    """
    n = A.shape[0] # Matrix size
    X = np.zeros((n_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    LD = np.tril(A) # Get lower triangle (L + D)
    # Get inverse in O(n^2) instead of np.linalg.inv(LD) O(n^3)
    LD_inv = solve_triangular(LD, np.eye(n), lower=True)
    r = b - np.dot(A, X[0]) # Residual
    # Gauss-Seidel iteration
    for k in range(n_iter):
        X[k+1] = X[k] + np.dot(LD_inv, r)
        r = b - np.dot(A, X[k+1]) # Residual update
        if np.linalg.norm(r) < tol: # Stop criteria
            X = X[:k+2]
            break
    return X
```

```python
def SOR2(A, b, w=1.05, n_iter=50, tol=1e-8, x_0=None):
    """
    Solve Ax=b using SOR(w) method

    Parameters
    -----------
    A      : (n, n) array
             A matrix
    b      : (n, ) array
             RHS vector
    w      : w
             Omega parameter.
    n_iter : int
             Number of iterations
    tol    : float
             Tolerance
    x_0    : (n, ) array
             Initual guess

    Returns
    -------
    X      : (n_iter + 1, n) array
             Matrix with approximation at each iteration
    """
    n = A.shape[0] # Matrix size
    X = np.zeros((n_iter + 1, n)) # Matrix with solution at each iteration
    # Initial guess
    if x_0 is not None:
        X[0] = x_0
    L = np.tril(A, k=-1) # Get lower triangle
    Dw = np.diag(np.diag(A) / w)
    # Get inverse in O(n^2) instead of np.linalg.inv(L+Dw) O(n^3)
    LDw_inv = solve_triangular(L+Dw, np.eye(n), lower=True)
    r = b - np.dot(A, X[0]) # Residual
    # SOR iteration
    for k in range(n_iter):
        X[k+1] = X[k] + np.dot(LDw_inv, r)
        r = b - np.dot(A, X[k+1]) # Residual update
        if np.linalg.norm(r) < tol: # Stop criteria
            X = X[:k+2]
            break
    return X
```

A continuación se resuelven sistemas parra distintos valores de $n$, y calculamos los tiempos.

```python
Ne = 5 # Number of experiments
N = 2 ** np.arange(7, 10) # N = [2^7, 2^{10}]
Nn = N.shape[-1]
# For times
times_jac = np.zeros(Nn)
times_gss = np.zeros(Nn)
times_sor = np.zeros(Nn)
```

```python
for i in range(Nn):
    n = N[i]
    A = ddMatrix(n)
    b = np.random.random(n)
    # Time Jacobi
    start_time= time()
    for j in range(Ne):
        x = jacobi2(A, b)
    end_time = time()
    times_jac[i] = (end_time - start_time) / Ne
    # Time G-S
    start_time = time()
    for j in range(Ne):
        x = gaussSeidel2(A, b)
    end_time = time()
    times_gss[i] = (end_time - start_time) / Ne
    # Time SOR
    start_time = time()
    for j in range(Ne):
        x = SOR2(A, b)
    end_time = time()
    times_sor[i] = (end_time - start_time) / Ne
```

```python
plt.figure(figsize=(12, 6))
plt.plot(N, times_jac, 'rx', label="Jacobi")
plt.plot(N, times_gss, 'bd', label="Gauss-Seidel")
plt.plot(N, times_sor, 'go', label="SOR")
# Deben adaptar el coeficiente que acompaña a N**k según los tiempos que obtengan en su computador
plt.plot(N, 1e-7 * N ** 2, 'g--', label=r"$O(n^2)$")
plt.plot(N, 1e-8 * N ** 3, 'r--', label=r"$O(n^3)$")
plt.grid(True)
plt.yscale('log')
plt.xscale('log')
plt.xlabel(r"$n$")
plt.ylabel("Time [s]")
plt.legend()
plt.show()
```

Del gráfico podemos confirmar que la complejidad de estos métodos es $\sim I n^2$, donde $I$ es el número de iteraciones. El valor de $I$ puede ser diferente en cada método.

# Referencias

* Sauer, T. (2006). Numerical Analysis Pearson Addison Wesley.
* https://github.com/tclaudioe/Scientific-Computing/tree/master/SC1/05_linear_systems_of_equations.ipynb

```python

```


## Deriving the transfer function of virtual analog first order filters.

HTML output built with: jupyter nbconvert --to html one_pole_z_domain_tf.ipynb

Source:
http://www.willpirkle.com/Downloads/AN-4VirtualAnalogFilters.pdf

We will derive the algorithm from the block diagram found on page 5, but we will follow the style of [Andrew Simper's SVF paper](https://cytomic.com/files/dsp/SvfLinearTrapOptimised2.pdf).

Sympy can't (very easily) be bent to display transfer functions in terms of $z^{-1}, z^{-2}, ...$ which is the convention. Plain $z$ will be used here instead - keep in mind it actually means $z^{-1}$.

```python
from sympy import *
init_printing()

z = symbols("z")
```

Start with the parameters.

```
g = Tan[π * cutoff / samplerate];
a1 = g / (1.0 + g);
```

The other coefficients defining the shape of the filter (`m0, m1`) will be ignored for now, as they are only used to "mix" the output.

```python
g = symbols("g")
a1 = g / (1.0 + g)

a1
```

Then the computation.

The variable `v0` represents the input signal - we will consider it to represent the z-transform of the input over time. `v1` and `v2` represent two other nodes in the block diagram.

The state variable `ic1eq` will be defined as unknown first, and then we will solve it using its equations.

The relevant lines of the algorithm are:

```
v1 = a1 * (v0 - ic1eq);
v2 = v1 + ic1eq;
```

Notice that `ic1eq` actually refers to the _previous_ value of these samples. This corresponds to multiplying by $z$ (contrary to convention!) in the z-domain.

```python
v0, ic1eq = symbols("v0 ic_1")

v1 = a1 * (v0 - ic1eq * z)
v2 = ic1eq * z + v1

(v1, v2)
```

The "new" value for `ic1eq` is computed as follows:

```
ic1eq = v2 + v1;
```

depending on the current values of `v1, v2`, and the previous value of `ic1eq`.

Consider this equation, and solve it:

```python
equation = [
    v2 + v1 - ic1eq, # = 0
]
solution = solve(equation, (ic1eq))

solution
```

We may now subsitute the solution into `v2` to obtain the transfer function

$$
\begin{aligned}
H_0(z) &= \frac {v_0(z)} {v_0(z)} = 1  \\
H_1(z) &= \frac {v_2(z)} {v_0(z)} \\
\end{aligned}
$$

```python
H0 = 1
H1 = v2.subs(solution) / v0
H1 = collect(simplify(H1), z)

(H1)
```

We can now assemble the complete transfer function, taking into account the mix coefficients `m0, m1`.

$$
H(z) = m_0 H_0(z) + m_1 H_1(z)
$$

```python
m0, m1 = symbols("m0 m1")

H = m0 * H0 + m1 * H1

print(H)
H
```

## Sanity check: High pass filter

```python
from sympy.functions import tan, exp

samplerate = 40_000
cutoff = sqrt(samplerate/2)

f = symbols("f")

H_hp_f = H.subs({
    g: tan(pi * cutoff / samplerate),
    m0: 1,
    m1: -1,
    z: exp(2*I*pi * f / samplerate)**-1,
})

plot(abs(H_hp_f), (f, 1, samplerate/2), xscale='log', yscale='log')
```

```python

```