```python
from sympy import *
x, y, z, t = symbols('x y z t')
```

## Mechanics

The module called [`sympy.physics.mechanics`](http://pyvideo.org/video/2653/dynamics-and-control-with-python)
contains elaborate tools for describing mechanical systems,
manipulating reference frames, forces, and torques.
These specialized functions are not necessary for a first-year mechanics course.
The basic `SymPy` functions like `solve`,
and the vector operations you learned in the previous sections are powerful enough for basic Newtonian mechanics.

### Dynamics

The net force acting on an object is the sum of all the external forces acting on it $\vec{F}_{\textrm{net}} = \sum \vec{F}$.
Since forces are vectors,
we need to use vector addition to compute the net force.

Compute
$\vec{F}_{\textrm{net}}=\vec{F}_1 + \vec{F}_2$,
where $\vec{F}_1=4\hat{\imath}[\mathrm{N}]$ and $\vec{F}_2 = 5\angle 30^\circ[\mathrm{N}]$:

```python
F_1 =  Matrix( [4,0] )
F_2 =  Matrix( [5*cos(30*pi/180), 5*sin(30*pi/180) ] )
F_net = F_1 + F_2
F_net                                    # in Newtons
```

```python
F_net.evalf()                            # in Newtons
```

To express the answer in length-and-direction notation,
use `norm` to find the length of $\vec{F}_{\textrm{net}}$
and `atan2` (The function `atan2(y,x)` computes the correct direction
for all vectors $(x,y)$, unlike `atan(y/x)` which requires corrections for angles in the range $[\frac{\pi}{2}, \frac{3\pi}{2}]$.) to find its direction:

```python
F_net.norm().evalf()                     # |F_net| in [N]
```

```python
(atan2( F_net[1],F_net[0] )*180/pi).n()  # angle in degrees
```

The net force on the object is $\vec{F}_{\textrm{net}}= 8.697\angle 16.7^\circ$[N].

### Kinematics

Let $x(t)$ denote the position of an object,
$v(t)$ denote its velocity,
and $a(t)$ denote its acceleration.
Together $x(t)$, $v(t)$, and $a(t)$ are known as the *equations of motion* of the object.

The equations of motion are related by the derivative operation:

$$
  a(t) \overset{\frac{d}{dt} }{\longleftarrow} v(t) \overset{\frac{d}{dt} }{\longleftarrow} x(t).
$$

Assume we know the initial position $x_i\equiv x(0)$ and the initial velocity $v_i\equiv v(0)$ of the object
and we want to find $x(t)$ for all later times.
We can do this starting from the dynamics of the problem—the forces acting on the object.

Newton's second law $\vec{F}_{\textrm{net}} = m\vec{a}$ states that a net force $\vec{F}_{\textrm{net}}$
applied on an object of mass $m$ produces acceleration $\vec{a}$.
Thus, we can obtain an objects acceleration if we know the net force acting on it.
Starting from the knowledge of $a(t)$, we can obtain $v(t)$ by integrating
then find $x(t)$ by integrating $v(t)$:

$$
a(t) \ \ \ \overset{v_i+ \int\!dt }{\longrightarrow} \ \ \ v(t) \ \ \ \overset{x_i+ \int\!dt }{\longrightarrow} \ \ \ x(t).
$$

The reasoning follows from the fundamental theorem of calculus:
if $a(t)$ represents the change in $v(t)$,
then the total of $a(t)$ accumulated between $t=t_1$ and $t=t_2$
is equal to the total change in $v(t)$ between these times: $\Delta v = v(t_2) - v(t_1)$.
Similarly, the integral of $v(t)$ from $t=0$ until $t=\tau$ is equal to $x(\tau) - x(0)$.

### Uniform acceleration motion (UAM)

Let's analyze the case where the net force on the object is constant.
A constant force causes a constant acceleration $a = \frac{F}{m} = \textrm{constant}$.
If the acceleration function is constant over time $a(t)=a$.
We find $v(t)$ and $x(t)$ as follows:

```python
t, a, v_i, x_i = symbols('t a v_i x_i')
v = v_i + integrate(a, (t, 0,t) )
v
```

```python
x = x_i + integrate(v, (t, 0,t) )
x
```

You may remember these equations from your high school physics class.
They are the *uniform accelerated motion* (UAM) equations:

\begin{align*}
 a(t) &= a,                                  \\
 v(t) &= v_i  + at,                          \\[-2mm]
 x(t) &= x_i + v_it + \frac{1}{2}at^2.
\end{align*}

In high school, you probably had to memorize these equations.
Now you know how to derive them yourself starting from first principles.

For the sake of completeness, we'll now derive the fourth UAM equation,
which relates the object's final velocity to the initial velocity,
the displacement, and the acceleration, without reference to time:

```python
(v*v).expand()
```

```python
((v*v).expand() - 2*a*x).simplify()
```

The above calculation shows $v_f^2 - 2ax_f = -2ax_i + v_i^2$.
After moving the term $2ax_f$ to the other side of the equation, we obtain

\begin{align*}
 (v(t))^2 \ = \ v_f^2 =  v_i^2  + 2a\Delta x \ = \  v_i^2  + 2a(x_f-x_i).
\end{align*}

The fourth equation is important for practical purposes
because it allows us to solve physics problems in a time-less manner.

#### Example

Find the position function of an object at time $t=3[\mathrm{s}]$,
if it starts from $x_i=20[\mathrm{m}]$ with $v_i=10[\mathrm{m/s}]$ and undergoes
a constant acceleration of $a=5[\mathrm{m/s^2}]$.
What is the object's velocity at $t=3[\mathrm{s}]$?

```python
x_i = 20  # initial position
v_i = 10  # initial velocity
a   = 5   # acceleration (constant during motion)
x = x_i + integrate( v_i+integrate(a,(t,0,t)), (t,0,t) )
x
```

```python
x.subs({t:3}).n()          # x(3) in [m]
```

```python
diff(x,t).subs({t:3}).n()  # v(3) in [m/s]
```

If you think about it,
physics knowledge combined with computer skills is like a superpower!

### General equations of motion

The procedure
$a(t) \ \overset{v_i+ \int\!dt }{\longrightarrow} \ v(t) \ \overset{x_i+ \int\!dt }{\longrightarrow} \ x(t)$
can be used to obtain the position function $x(t)$ even when the acceleration is not constant.
Suppose the acceleration of an object is $a(t)=\sqrt{k t}$;
what is its $x(t)$?

```python
t, v_i, x_i, k = symbols('t v_i x_i k')
a = sqrt(k*t)
x = x_i + integrate( v_i+integrate(a,(t,0,t)), (t, 0,t) )
x
```

### Potential energy

Instead of working with the kinematic equations of motion $x(t)$, $v(t)$, and $a(t)$ which depend on time,
we can solve physics problems using *energy* calculations.
A key connection between the world of forces and the world of energy is the concept of *potential energy*.
If you move an object against a conservative force (think raising a ball in the air against the force of gravity),
you can think of the work you do agains the force as being stored in the potential energy of the object.

For each force $\vec{F}(x)$ there is a corresponding potential energy $U_F(x)$.
The change in potential energy associated with the force $\vec{F}(x)$ and displacement $\vec{d}$
is defined as the negative of the work done by the force during the displacement: $U_F(x) = - W = - \int_{\vec{d}} \vec{F}(x)\cdot d\vec{x}$.

The potential energies associated with gravity $\vec{F}_g = -mg\hat{\jmath}$
and the force of a spring $\vec{F}_s = -k\vec{x}$ are calculated as follows:

```python
x, y = symbols('x y')
m, g, k, h = symbols('m g k h')
F_g = -m*g  # Force of gravity on mass m
U_g = - integrate( F_g, (y,0,h) )
U_g         # Grav. potential energy
```

```python
F_s = -k*x  # Spring force for displacement x
U_s = - integrate( F_s, (x,0,x) )
U_s         # Spring potential energy
```

Note the negative sign in the formula defining the potential energy.
This negative is canceled by the negative sign of the dot product $\vec{F}\cdot d\vec{x}$:
when the force acts in the direction opposite to the displacement,
the work done by the force is negative.

### Simple harmonic motion

The force exerted by a spring is given by the formula $F=-kx$.
If the only force acting on a mass $m$ is the force of a spring,
we can use Newton's second law to obtain the following equation:

$$
  F=ma
  \quad \Rightarrow \quad
  -kx = ma
  \quad \Rightarrow \quad
  -kx(t) = m\frac{d^2}{dt^2}\Big[x(t)\Big].
$$

The motion of a mass-spring system is described by the *differential equation* $\frac{d^2}{dt^2}x(t) + \omega^2 x(t)=0$,
where the constant $\omega = \sqrt{\frac{k}{m}}$ is called the angular frequency.
We can find the position function $x(t)$ using the `dsolve` method:

```python
t = Symbol('t')                 # time t
x = Function('x')               # position function x(t)
w = Symbol('w', positive=True)  # angular frequency w
sol = dsolve( diff(x(t),t,t) + w**2*x(t), x(t) )
sol
```

```python
x = sol.rhs
x
```

Note the solution $x(t)=C_1\sin(\omega t)+C_2 \cos(\omega t)$ is equivalent to $x(t) = A\cos(\omega t + \phi)$,
which is more commonly used to describe simple harmonic motion.
We can use the `expand` function with the argument `trig=True` to convince ourselves of this equivalence:

```python
A, phi = symbols("A phi")
(A*cos(w*t - phi)).expand(trig=True)
```

If we define $C_1=A\sin(\phi)$ and $C_2=A\cos(\phi)$,
we obtain the form $x(t)=C_1\sin(\omega t)+C_2 \cos(\omega t)$ that `SymPy` found.

### Conservation of energy

We can verify that the total energy of the mass-spring system is conserved by showing
$E_T(t) = U_s(t) + K(t) = \textrm{constant}$:

```python
x = sol.rhs.subs({"C1":0,"C2":A})
x
```

```python
v = diff(x, t)
v
```

```python
E_T = (0.5*k*x**2 + 0.5*m*v**2).simplify()
E_T
```

```python
E_T.subs({k:m*w**2}).simplify()     # = K_max
```

```python
E_T.subs({w:sqrt(k/m)}).simplify()  # = U_max
```


```python
%matplotlib inline
```

```python
# Write your imports here
import numpy as np
import numpy.polynomial.polynomial as p
import matplotlib.pyplot as plt
import math
import time

from sympy.ntheory import factorint
from Crypto.Util import number

```

# Basic Algebra Exercise
## Functions, Polynomials, Complex Numbers. Applications of Abstract Algebra

### Problem 1. Polynomial Interpolation
We know that if we have a set of $n$ data points with coordinates $(x_1; y_1), (x_2; y_2), \dots, (x_n; y_n)$, we can try to figure out what function may have generated these points.

Please note that **our assumptions about the data** will lead us to choosing one function over another. This means that our results are as good as our data and assumptions. Therefore, it's extremely important that we write down our assumptions (which sometimes can be difficult as we sometimes don't realize we're making them). It will be better for our readers if they know what those assumptions and models are.

In this case, we'll state two assumptions:
1. The points in our dataset are generated by a polynomial function
2. The points are very precise, there is absolutely no error in them. This means that the function should pass **through every point**

This method is called *polynomial interpolation* (*"polynomial"* captures assumption 1 and *"interpolation"* captures assumption 2).

It can be proved (look at [Wikipedia](https://en.wikipedia.org/wiki/Polynomial_interpolation) for example) that if we have $n$ data points, there is only one polynomial of degree $n-1$ which passes through them. In "math speak": "the vector spaces of $n$ points and polynomials of degree $n-1$ are isomorphic (there exists a bijection mapping one to the other)".

There are a lot of ways to do interpolation. We can also write the function ourselves if we want but this requires quite a lot more knowledge than we already covered in this course. So we'll use a function which does this for us. `numpy.polyfit()` is one such function. It accepts three main parameters (there are others as well, but they are optional): a list of $x$ coordinates, a list of $y$ coordinates, and a polynomial degree.

Let's say we have these points:
```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
```

First, we need to "extract" the coordinates:
```python
x = points[:, 0]
y = points[:, 1]
```

Then, we need to calculate the interpolating polynomial. For the degree, we'll set $n-1$:
```python
coefficients = np.polyfit(x, y, len(points) - 1)
poly = np.poly1d(coefficients)
```

After that, we need to plot the function. To do this, we'll create a range of $x$ values and evaluate the polynomial at each value:
```python
plot_x = np.linspace(np.min(x), np.max(x), 1000)
plot_y = poly(plot_x)
```

Finally, we need to plot the result. We'll plot both the fitting polynomial curve (using `plt.plot()`) and the points (using `plt.scatter`). It's also nice to have different colors to make the line stand out from the points.
```python
plt.plot(plot_x, plot_y, c = "green")
plt.scatter(x, y)
plt.xlabel("x")
plt.ylabel("y")
plt.show()
```
Don't forget to label the axes!

Your task now is to **wrap the code in a function**. It should accept a list of points, the polynomial degree, min and max value of $x$ used for plotting. **Be extremely careful to ensure that the function uses its parameters!**

We'll use this function to try some other cases.

```python

def interpolate_polynomial(points, degree, min_x, max_x):
    """
    Interpolates a polynomial of the specified degree through the given points and plots it
    points - a list of points (x, y) to plot
    degree - the polynomial degree
    min_x, max_x - range of x values used to plot the interpolating polynomial
    """
    x = points[:,0]
    y = points[:,1]

    coef = np.polyfit(x,y, degree)
    poly = np.poly1d(coef)

    plot_x = np.linspace(min_x, max_x, 1000)
    plot_y = poly(plot_x)

    plt.plot(plot_x, plot_y, c="yellow")
    plt.scatter(x, y)
    plt.xlabel("x")
    plt.ylabel("y")
    plt.show()

```

```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

We see this is a very nice fit. This is expected, of course. Let's try to expand our view a little. Let's try to plot other values of $x$, further than the original ones. This is **extrapolation**.

```python
interpolate_polynomial(points, len(points) - 1, -5, 10)
```

Hmmm... it seems our polynomial goes a little wild outside the original range. This is to show how **extrapolation can be quite dangerous**.

Let's try a lower polynomial degree now. We used 4, how about 3, 2 and 1?
**Note:** We can add titles to every plot so that we know what exactly we're doing. Te title may be passed as an additional parameter to our function.

```python
interpolate_polynomial(points, 3, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 2, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

We see the fitting curves (or line in the last case) struggle more and more and they don't pass through every point. This breaks our assumptions but it can be very useful.

Okay, one more thing. How about increasing the degree? Let's try 5, 7 and 10. Python might complain a little, just ignore it, everything is fine... sort of :).

```python
interpolate_polynomial(points, 5, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 7, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 10, np.min(points[:, 0]), np.max(points[:, 0]))
```

Those graphs look pretty much the same. But that's the point exactly. I'm being quite sneaky here. Let's try to expand our view once again and see what our results really look like.

```python
interpolate_polynomial(points, 5, -10, 10)
interpolate_polynomial(points, 7, -10, 10)
interpolate_polynomial(points, 10, -10, 10)
```

Now we see there are very wild differences. Even though the first two plots look quite similar, look at the $y$ values - they're quite different.

So, these are the dangers of interpolation. Use a too high degree, and you get "the polynomial wiggle". These are all meant to represent **the same** data points but they look insanely different. Here's one more comparison.

```python
interpolate_polynomial(points, len(points) - 1, -2, 7)
interpolate_polynomial(points, len(points) + 1, -2, 7)
```

Now we can see what big difference even a small change in degree can make. This is why we have to choose our interpolating functions very carefully. Generally, a lower degree means a simpler function, which is to be preferred. See [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor).

And also, **we need to be very careful about our assumptions**.

```python
points = np.array([(-5, 0.03846), (-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882), (5, 0.03846)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

```python
points = np.array([(-5, 0.03846), (-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882), (5, 0.03846)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0])-3, np.max(points[:, 0])+3)
```

This one definitely looks strange. Even stranger, if we remove the outermost points... ($x = \pm 5$), we get this

```python
points = np.array([(-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882)])
interpolate_polynomial(points, len(points - 1), np.min(points[:, 0]), np.max(points[:, 0]))
```

This is because the generating function is not a polynomial. It's actually:
$$ y = \frac{1}{1 + x^2} $$

Plot the polynomial interpolation and the real generating function **on the same plot**. You may need to modify the original plotting function or just copy its contents.

```python
def interpolate_polynomial_poly(points, degree, min_x, max_x):
    """
    Interpolates a polynomial of the specified degree through the given points and plots it
    points - a list of points (x, y) to plot
    degree - the polynomial degree
    min_x, max_x - range of x values used to plot the interpolating polynomial
    """
    x = points[:,0]
    y = points[:,1]

    coef = np.polyfit(x,y, degree)
    poly = np.poly1d(coef)
    plot_x = np.linspace(min_x, max_x, 1000)
    plot_y = poly(plot_x)
    plt.plot(plot_x, plot_y, c="yellow")
    plt.plot(plot_x, poly, c="green")
    plt.scatter(x, y)
    plt.xlabel("x")
    plt.ylabel("y")
    plt.show()
```

```python
points = np.array([(-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882)])
interpolate_polynomial_poly(points, len(points - 1), np.min(points[:, 0]), np.max(points[:, 0]))
```

### Problem 2. Complex Numbers as Vectors
We saw that a complex number $z = a + bi$ is equivalent to (and therefore can be represented as) the ordered tuple $(a; b)$, which can be plotted in a 2D space. So, complex numbers and 2D points are equivalent. What is more, we can draw a vector from the origin of the coordinate plane to our point. This is called a point's **radius-vector**.

Let's try plotting complex numbers as radius vectors. Don't forget to label the real and imaginary axes. Also, move the axes to the origin. Hint: These are called "spines"; you'll need to move 2 of them to the origin and remove the other 2 completely. Hint 2: You already did this in the previous lab.

We can use `plt.quiver()` to plot the vector. It can behave a bit strangely, so we'll need to set the scale of the vectors to be the same as the scale on the graph axes:
```python
plt.quiver(0, 0, z.real, z.imag, angles = "xy", scale_units = "xy", scale = 1)
```

Other than that, the main parameters are: $x_{begin}$, $y_{begin}$, $x_{length}$, $y_{length}$ in that order.

Now, set the aspect ratio of the axes to be equal. Also, add grid lines. Set the axis numbers (called ticks) to be something like `range(-3, 4)` for now.
```python
plt.xticks(range(-3, 4))
plt.yticks(range(-3, 4))

```

If you wish to, you can be a bit more clever with the tick marks. Find the minimal and maximal $x$ and $y$ values and set the ticks according to them. It's a good practice not to jam the plot too much, so leave a little bit of space. That is, if the actual x-range is $[-2; 2]$, set the plotting to be $[-2.5; 2.5]$ for example. Otherwise, the vector heads (arrows) will be "jammed" into a corner or side of the plot.

```python
def plot_complex_number(z):
    """
    Plots the complex number z as a radius vector in the 2D space
    """
    rel = z.real
    img = z.imag
    #print(rel)
    #print(img)
    plt.quiver(0,0,rel, img, angles = "xy", scale_units = "xy", scale = 1)
    plt.xticks(range(0, math.ceil(rel)+1))
    plt.yticks(range(0, math.ceil(img)+1))
    plt.show()

```

```python
plot_complex_number(2 + 3j)
```

How about many numbers? We'll need to get a little bit more creative. First, we need to create a 2D array, each element of which will be a 4-element array: `[0, 0, z.real, z.imag]`. Next, `plt.quiver()` can accept a range of values. Look at [this StackOverflow post](https://stackoverflow.com/questions/12265234/how-to-plot-2d-math-vectors-with-matplotlib) for details and adapt your code.

```python
def plot_complex_numbers(numbers, colors):
    """
    Plots the given complex numbers as radius vectors in the 2D space
    """
    real_part = []
    imag_part = []

    for i in numbers:
        real_part.append(i.real)
        imag_part.append(i.imag)

    #vectors =[0]*2*len(real_part)+ real_part + imag_part
    min_x = int(min(real_part)) - 2
    max_x = int(max(real_part)) + 2
    min_y = int(min(imag_part)) - 2
    max_y = int(max(imag_part)) + 2
    X = [0] * len(real_part)
    Y = [0] * len(real_part)
    #print(X)
    #print(Y)
    #print(real_part)
    #print(imag_part)

    plt.quiver(X, Y, real_part, imag_part, angles = "xy", scale_units = "xy", scale = 1, color= colors)

    plt.xticks(range(min_x, max_x))
    plt.yticks(range(min_y, max_y))
    plt.gca().set_aspect("equal")
    plt.show()

```

```python
plot_complex_numbers([2 + 3j, -2 - 1j, -3, 2j], ["green", "red", "blue", "orange"])
```

```python
x = ', '.join(map(str,[1,4,5,6]))
print(x)
```

    1, 4, 5, 6

Now let's see what the operations look like. Let's add two numbers and plot the result.

```python
z1 = 2 + 3j
z2 = 1 - 1j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

We can see that adding the complex numbers is equivalent to adding vectors (remember the "parallelogram rule"). As special cases, let's try adding pure real and pure imaginary numbers:

```python
z1 = 2 + 3j
z2 = 2 + 0j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

```python
z1 = 2 + 3j
z2 = 0 + 2j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

How about multiplication? First we know that multiplying by 1 gives us the same vector and mulpiplying by -1 gives us the reversed version of the same vector. How about multiplication by $\pm i$?

```python
z = 2 + 3j
plot_complex_numbers([z, z * 1], ["red", "blue"])
plot_complex_numbers([z, z * -1], ["red", "blue"])
plot_complex_numbers([z, z * 1j], ["red", "blue"])
plot_complex_numbers([z, z * -1j], ["red", "blue"])
```

So, multiplication by $i$ is equivalent to 90-degree rotation. We can actually see the following equivalence relationships between multiplying numbers and rotation about the origin:

| Real | Imaginary | Result rotation |
|------|-----------|-----------------|
| 1    | 0         | $0^\circ$       |
| 0    | 1         | $90^\circ$      |
| -1   | 0         | $180^\circ$     |
| 0    | -1        | $270^\circ$     |

Once again, we see the power of abstraction and algebra in practice. We know that complex numbers and 2D vectors are equivalent. Now we see something more: addition and multiplication are equivalent to translation (movement) and rotation!

Let's test the multiplication some more. We can see the resulting vector is the sum of the original vectors, but *scaled and rotated*:

```python
z1 = 2 + 3j
z2 = 1 - 2j
plot_complex_numbers([z1, z2, z1 * z2], ["red", "blue", "green"])
```

### Problem 3. Recursion and Fractals

> "To understand recursion, you first need to understand recursion."

There are three main parts to a recursive function:
1. Bottom - when the recursion should finish
2. Operation - some meaningful thing to do
3. Recursive call - calling the same function
4. Clean-up - returning all data to its previous state (this reverses the effect of the operation)

Let's do one of the most famous recursion examples. And I'm not talking about Fibonacci here. Let's draw a tree using recursive functions.

The figure we're going to draw is called a **fractal**. It's self-similar, which means that if you zoom in on a part of it, it will look the same. You can see fractals everywhere in nature, with broccoli being one of the prime examples. Have a look:

First, we need to specify the recursive part. In order to draw a tree, we need to draw a line of a given length (which will be the current branch), and then draw two more lines to the left and right. By "left" and "right", we should mean "rotation by a specified angle".

So, this is how to draw a branch: draw a line and prepare to draw two more branches to the left and right. This is going to be our recursive call.

To make things prettier, more natural-looking (and have a natural end to our recursion), let's draw each "sub-branch" a little shorter. If the branch becomes too short, it won't have "child branches". This will be the bottom of our recursion.

There's one more important part of recursion, and this is **"the clean-up"**. After we did something in the recursive calls, it's very important to return the state of everything as it was **before** we did anything. In this case, after we draw a branch, we go back to our starting position.

Let's first import the most import-ant (no pun intended...) Python drawing library: `turtle`! In order to make things easier, we'll import all methods directly.
```python
from turtle import *
```

You can look up the docs about turtle if you're more interested. The basic things we're going to use are going forward and backward by a specified number of pixels and turning left and right by a specified angle (in degrees).

Let's now define our recursive function:
```python
def draw_branch(branch_length, angle):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle)
        left(2 * angle)
        draw_branch(branch_length - 15, angle)
        right(angle)
        backward(branch_length)
```

And let's call it:
```python
draw_branch(100, 20)
```

We need to start the tree not at the middle, but toward the bottom of the screen, so we need to make a few more adjustments. We can wrap the setup in another function and call it. Let's start one trunk length below the center (the trunk length is the length of the longest line).

**Note:** It's important to call `done()` after the drawing is finished. If you miss it, the window with the turtle drawing will freeze and throw an exception.

```python
def draw_tree(trunk_length, angle):
    speed("fastest")
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

Note that the graphics will show in a separate window. Also note that sometimes you might get bugs. If you do, go to Kernel > Restart.

```python
from turtle import *

```

```python
def draw_branch(branch_length, angle):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle)
        left(2 * angle)
        draw_branch(branch_length - 15, angle)
        right(angle)
        backward(branch_length)
```

```python
def draw_tree(trunk_length, angle):
    speed("fastest")
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

```python
draw_branch(100, 20)
```

```python
draw_tree(100, 20)
```

Experiment with different lengths and angles. Especially interesting angles are $30^\circ$, $45^\circ$, $60^\circ$ and $90^\circ$.

```python
draw_tree(100, 30)
```

```python
draw_tree(100, 45)
```

```python
draw_tree(100, 90)
```

Now modify the original function a little. Draw the lines with different thickness. Provide the trunk thickness at the initial call. Similar to how branches go shorter, they should also go thinner.

```python
# Write your code here
```

#### * Optional problem
Try to draw another kind of fractal graphic using recursion and the `turtle` library. Two very popular examples are the "Koch snowflake" and the "Sierpinski triangle". You can also modify the original tree algorithm to create more natural-looking trees. You can, for example, play with angles, number of branches, lengths, and widths. The Internet has a lot of ideas about this :). Hint: Look up **"L-systems"**.

### Problem 4. Run-length Encoding
One application of algebra and basic math can be **compression**. This is a way to save data in less space than it originally takes. The most basic form of compression is called [run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding).

Write a function that encodes a given text. Write another one that decodes.

We can see that RLE is not very useful in the general case. But it can be extremely useful if we have very few symbols. An example of this can be DNA and protein sequences. DNA code, for example, has only 4 characters.

Test your encoding and decoding functions on a DNA sequence (you can look up some on the Internet). Measure how much your data is compressed relative to the original.

```python
def encode(text):
    if len(text) == 0 :
        return ''

    result = []
    counter = 1
    last_symbol = text[0]

    for index in range(1,len(text)):
        current_symbol = text[index]
        if current_symbol == last_symbol:
            counter += 1
        else:
            counter_value = str(counter) if counter > 1 else ''
            value_to_add = last_symbol + str(counter_value)
            result.append(value_to_add)
            last_symbol = current_symbol
            counter = 1

    counter_value = str(counter) if counter > 1 else ''
    value_to_add = last_symbol + str(counter_value)
    result.append(value_to_add)

    return ''.join(result)

def decode(text):
    if len(text) == 0 :
        return ''

    result =[]
    counter = []
    last_symbol = text[0]
    for index in range (1, len(text)):
        current_symbol = text[index]

        if current_symbol.isdigit():
            counter.append(current_symbol)
        else:
            if len(counter) == 0:
                result.append(last_symbol)
                last_symbol = current_symbol
            else:
                symbol_count = int(''.join(counter))
                result.append(last_symbol*symbol_count)
                counter = []
                last_symbol = current_symbol

    if len(counter) == 0:
        result.append(last_symbol)
        last_symbol = current_symbol
    else:
        symbol_count = int(''.join(counter))
        result.append(last_symbol*symbol_count)
        counter = []
        last_symbol = current_symbol

    return ''.join(result)
```

```python
# Tests
# Test that the functions work on their own
assert encode("AABCCCDEEEE") == "A2BC3DE4"
assert decode("A2BC3DE4") == "AABCCCDEEEE"

# Test that the functions really invert each other
assert decode(encode("AABCCCDEEEE")) == "AABCCCDEEEE"
assert encode(decode("A2BC3DE4")) == "A2BC3DE4"
```

### * Problem 5. Function Invertibility and Cryptography
As we already saw, some functions are able to be inverted. That is, if we know the output, we can see what input generated it directly. This is true if the function is **one-to-one correspondence** (bijection).

However, not all functions are created the same. Some functions are easy to compute but their inverses are extremely difficult. A very important example is **number factorization**. It's relatively easy (computationally) to multiply numbers but factoring them is quite difficult. Let's run an experiment.

We'll need a function to generate random n-bit numbers. One such number can be found in the `Crypto` package
```python
from Crypto.Util import number
random_integer = number.getRandomNBitInteger(n_bits)
```

We could, of course, write our factorization by hand but we'll use `sympy`
```python
from sympy.ntheory import factorint
factorint(1032969399047817906432668079951) # {3: 2, 79: 1, 36779: 1, 7776252885493: 1, 5079811103: 1}
```

This function returns a `dict` where the keys are the factors, and the values - how many times they should be multiplied.

We'll also need a tool to accurately measure performance. Have a look at [this one](https://docs.python.org/3/library/time.html#time.time) for example.

Specity a sequence of bit lengths, in increasing order. For example, you might choose something like `[10, 20, 25, 30, 32, 33, 35, 38, 40]`. Depending on your computer's abilities you can go as high as you want. For each bit length, generate a number. See how much time it takes to factor it. Then see how much time it takes to multiply the factors. Be careful how you measure these. You shouldn't include the number generation (or any other external functions) in your timing.

In order to have better accuracy, don't do this once per bit length. Do it, for example, five times, and average the results.

Plot all multiplication and factorization times as a function of the number of bits. You should see that factorization is much, much slower. If you don't see this, just try larger numbers :D.

```python
#random_prime = number.getPrime(n_bits)

def number_factorization_and_revers_time(n_bits):

    fact_time = []
    calc_initial_time = []

    for i in range(5):
        random_integer = number.getRandomNBitInteger(n_bits)
        start = time.time()
        facto_nums = factorint(random_integer)
        stop = time.time()
        fact_time.append(stop - start)

        initial_number = 1
        start_rev = time.time()
        for numb, occurencies in facto_nums.items():
            initial_number *= numb ** occurencies
        stop_rev = time.time()
        calc_initial_time.append(stop_rev - start_rev)

    print(f'time needed to decompose of prime factors of {n_bits} bits numbers')
    print(fact_time)
    print('Time needed to calculate the initial number')
    print(calc_initial_time)

```

```python
number_factorization_and_revers_time(64)
```

    time needed to decompose of prime factors of 64 bits numbers
    [0.036978960037231445, 0.004996061325073242, 0.00099945068359375, 0.0019989013671875, 0.010992765426635742]
    Time needed to calculate the initial number
    [0.0, 0.0, 0.0, 0.0, 0.0]

```python
number_factorization_and_revers_time(128)
```

    time needed to decompose of prime factors of 128 bits numbers
    [0.001004934310913086, 0.000993967056274414, 3.4900031089782715, 2.310678482055664, 0.001998424530029297]
    Time needed to calculate the initial number
    [0.0, 0.0, 0.0, 0.0, 0.0]

```python
number_factorization_and_revers_time(130)
```

    time needed to decompose of prime factors of 130 bits numbers
    [0.00099945068359375, 0.0019989013671875, 0.0039975643157958984, 0.2608451843261719, 0.00499725341796875]
    Time needed to calculate the initial number
    [0.0, 0.00099945068359375, 0.0, 0.0, 0.0]

### * Problem 6. Diffie - Hellman Simulation
As we already saw, there are functions which are very easy to compute in the "forward" direction but really difficult (computationally) to invert (that is, determine the input from the output). There is a special case: the function may have a hidden "trap door". If you know where that door is, you can invert the function easily. This statement is at the core of modern cryptography.

Look up **Diffie - Hellman key exchange** (here's a [video](https://www.youtube.com/watch?v=cM4mNVUBtHk) on that but feel free to use anything else you might find useful).

Simulate the algorithm you just saw. Generate large enough numbers so the difference is noticeable (say, factoring takes 10-15 seconds). Simulate both participants in the key exchange. Simulate an eavesdropper.

First, make sure after both participants run the algotihm, they have *the same key* (they generate the same number).

Second, see how long it takes for them to exchange keys.

Third, see how long it takes the eavesdropper to arrive at the correct shared secret.

You should be able to see **the power of cryptography**. In this case, it's not that the function is irreversible. It can be reversed, but it takes a really long time (and with more bits, we're talking billions of years). However, if you know something else (this is called a **trap door**), the function becomes relatively easy to invert.

```python
keys_bits = 22
base_bits = keys_bits // 2

num_mod =  number.getRandomNBitInteger(keys_bits)
base = number.getRandomNBitInteger(keys_bits)

print(f'Public number to perform mod devide is: {num_mod}')
print(f'Public base: {base}')

a_secret = number.getRandomNBitInteger(keys_bits)
print(f'Alice secret number is: {a_secret}')

b_secret = number.getRandomNBitInteger(keys_bits)
print(f'Bob secret number is: {b_secret}')

start = time.time()
a_num_to_sent = (base ** a_secret) % num_mod
print(f'Alice sent to Bob publically: {a_num_to_sent}')

b_num_to_sent = (base ** b_secret) % num_mod
print(f'Bob sent to Alice publically: {b_num_to_sent}')

a_key = (b_num_to_sent ** a_secret) % num_mod
b_key = (a_num_to_sent ** b_secret) % num_mod
end = time.time()

print(f'Time needd to calculate the keys is {round(end - start,2)} seconds')
print(f'Alice key is calculated to: {a_key}, and Bob key is: {b_key}')
```

    Public number to perform mod devide is: 3462015
    Public base: 3840224
    Alice secret number is: 4177708
    Bob secret number is: 2116548
    Alice sent to Bob publically: 2442361
    Bob sent to Alice publically: 2503201
    Time needd to calculate the keys is 121.75 seconds
    Alice key is calculated to: 1, and Bob key is: 1

### ** Problem 7. The Galois Field in Cryptography
Research about the uses of the Galois field. What are its properties? How can it be used in cryptography? Write a simple cryptosystem based on the field.

You can use the following questions to facilitate your research:
* What is a field?
* What is GF(2)? Why is it an algebraic field?
* What is perfect secrecy? How does it relate to the participants in the conversation, and to the outside eavesdropper?
* What is symmetrical encryption?
* How to encrypt one-bit messages?
* How to extend the one-bit encryption system to many buts?
* Why is the system decryptable? How do the participants decrypt the encrypted messages?
* Why isn't the eavesdropper able to decrypt?
* What is a one-time pad?
    * How does the one-time pad achieve perfect secrecy?
* What happens if we try to use a one-time pad many times?
    * Provide an example where you break the "many-time pad" security
* What are some current enterprise-grade applications of encryption over GF(2)?
* Implement a cryptosystem based on GF(2). Show correctness on various test cases

### ** Problem 8. Huffman Compression Algorithm
Examine and implement the **Huffman algorithm** for compressing data. It's based on information theory and probiability theory. Document your findings and provide your implementation.

This algorithm is used for **lossless compression**: compressing data without loss of quality. You can use the following checklist:

* What is the difference betwenn lossless and lossy compression?
* When can we get away with lossy compression?
* What is entropy?
* How are Huffman trees constructed?
    * Provide a few examples
* How can we get back the uncompressed data from the Huffman tree?
* How and where are Huffman trees stored?
* Implement the algorithm. Add any other formulas / assumptions / etc. you might need.
* Test the algorithm. A good meaure would be percentage compression: $$\frac{\text{compressed}}{\text{uncompressed}} * 100\%$$
* How well does Huffman's algorithm perform compared to other compression algorithms (e.g. LZ77)?


# Multiclass classification

The multiclass classification problem is a regression problem from an input $x \in {\cal X}$ to discrete labels $y\in {\cal Y}$, where ${\cal Y}$ is a discrete set of size $C$ bigger than two (for $C=2$ it is the more usual binary classification).

Labels are encoded in a one-hot fashion, that is if $C=4$ and $y=2$, we note $\bar{y} = [0,1,0,0]$.

The generative model for this problem consists of:

* $C$ latent functions $\mathbf{f} = [f_1,...,f_C]$ with an independent Gaussian Process prior
* a deterministic function that builds a discrete distribution $\pi(\mathbf{f}) = [\pi_1(f_1),...,\pi_C(f_C)]$ from the latents such that $\sum_c \pi_c(f_c) = 1$
* a discrete likelihood $p(y|\mathbf{f}) = Discrete(y;\pi(\mathbf{f})) = \prod_c \pi_c(f_c)^{\bar{y}_c}$

A typical example of $\pi$ is the softmax function:

\begin{equation}
\pi_c (f_c) \propto \exp( f_c)
\end{equation}

Another convenient one is the robust max:
\begin{equation}
\pi_c(\mathbf{f}) = \begin{cases} 1 - \epsilon, & \mbox{if } c = \arg \max_c f_c \\
 \epsilon /(C-1), & \mbox{ otherwise} \end{cases}
\end{equation}

```python
import numpy as np
import tensorflow as tf

import warnings

warnings.filterwarnings("ignore")  # ignore DeprecationWarnings from tensorflow

import matplotlib.pyplot as plt

%matplotlib inline

import gpflow

from gpflow.utilities import print_summary, set_trainable
from gpflow.ci_utils import ci_niter

from multiclass_classification import plot_posterior_predictions, colors

# reproducibility:
np.random.seed(0)
tf.random.set_seed(123)
```

## Sampling from the GP multiclass generative model

### Declaring model parameters and input

```python
# Number of functions and number of data points
C = 3
N = 100

# Lengthscale of the SquaredExponential kernel (isotropic -- change to `[0.1] * C` for ARD)
lengthscales = 0.1

# Jitter
jitter_eye = np.eye(N) * 1e-6

# Input
X = np.random.rand(N, 1)
```

### Sampling

```python
# SquaredExponential kernel matrix
kernel_se = gpflow.kernels.SquaredExponential(lengthscales=lengthscales)
K = kernel_se(X) + jitter_eye

# Latents prior sample
f = np.random.multivariate_normal(mean=np.zeros(N), cov=K, size=(C)).T

# Hard max observation
Y = np.argmax(f, 1).reshape(-1,).astype(int)

# One-hot encoding
Y_hot = np.zeros((N, C), dtype=bool)
Y_hot[np.arange(N), Y] = 1

data = (X, Y)
```

### Plotting

```python
plt.figure(figsize=(12, 6))
order = np.argsort(X.reshape(-1,))

for c in range(C):
    plt.plot(X[order], f[order, c], ".", color=colors[c], label=str(c))
    plt.plot(X[order], Y_hot[order, c], "-", color=colors[c])

plt.legend()
plt.xlabel("$X$")
plt.ylabel("Latent (dots) and one-hot labels (lines)")
plt.title("Sample from the joint $p(Y, \mathbf{f})$")
plt.grid()
plt.show()
```

## Inference

Inference here consists of computing the posterior distribution over the latent functions given the data $p(\mathbf{f}|Y, X)$.

You can use different inference methods. Here we perform variational inference.
For a treatment of the multiclass classification problem using MCMC sampling, see [Markov Chain Monte Carlo (MCMC)](../advanced/mcmc.ipynb).

### Approximate inference: Sparse Variational Gaussian Process

#### Declaring the SVGP model (see [GPs for big data](../advanced/gps_for_big_data.ipynb))

```python
# sum kernel: Matern32 + White
kernel = gpflow.kernels.Matern32() + gpflow.kernels.White(variance=0.01)

# Robustmax Multiclass Likelihood
invlink = gpflow.likelihoods.RobustMax(C)  # Robustmax inverse link function
likelihood = gpflow.likelihoods.MultiClass(3, invlink=invlink)  # Multiclass likelihood
Z = X[::5].copy()  # inducing inputs

m = gpflow.models.SVGP(
    kernel=kernel,
    likelihood=likelihood,
    inducing_variable=Z,
    num_latent_gps=C,
    whiten=True,
    q_diag=True,
)

# Only train the variational parameters
set_trainable(m.kernel.kernels[1].variance, False)
set_trainable(m.inducing_variable, False)
print_summary(m, fmt="notebook")
```

name                               	class    	transform  	prior  	trainable  	shape  	dtype  	value

SVGP.kernel.kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	1.0

SVGP.kernel.kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	1.0

SVGP.kernel.kernels[1].variance    	Parameter	Softplus   	       	False      	()     	float64	0.009999999999999998

SVGP.likelihood.invlink.epsilon    	Parameter	Sigmoid    	Beta   	False      	()     	float64	0.0010000000000000005

SVGP.inducing_variable.Z           	Parameter	Identity   	       	False      	(20, 1)	float64	[[0.5488135...

SVGP.q_mu                          	Parameter	Identity   	       	True       	(20, 3)	float64	[[0., 0., 0....

SVGP.q_sqrt                        	Parameter	Softplus   	       	True       	(20, 3)	float64	[[1., 1., 1....

#### Running inference

```python
opt = gpflow.optimizers.Scipy()

opt_logs = opt.minimize(
    m.training_loss_closure(data), m.trainable_variables, options=dict(maxiter=ci_niter(1000))
)
print_summary(m, fmt="notebook")
```

name                               	class    	transform  	prior  	trainable  	shape  	dtype  	value

SVGP.kernel.kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	198597.6324751772

SVGP.kernel.kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.1743317435395522

SVGP.kernel.kernels[1].variance    	Parameter	Softplus   	       	False      	()     	float64	0.009999999999999998

SVGP.likelihood.invlink.epsilon    	Parameter	Sigmoid    	Beta   	False      	()     	float64	0.0010000000000000005

SVGP.inducing_variable.Z           	Parameter	Identity   	       	False      	(20, 1)	float64	[[0.5488135...

SVGP.q_mu                          	Parameter	Identity   	       	True       	(20, 3)	float64	[[-0.23244898, 0.61406508, -0.3816161...

SVGP.q_sqrt                        	Parameter	Softplus   	       	True       	(20, 3)	float64	[[0.08260954, 0.07948762, 0.12074051...

```python
plot_posterior_predictions(m, X, Y)
```


```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
#sym.init_printing(use_unicode=False, wrap_line=True)
```

```python
s = sym.Symbol('alpha')
k = sym.Symbol('self%k')
asym  = (1-s)**2/(1+(k-1)*(1-1-s)**2)
print('a:  ',asym)
dasym = sym.diff(asym, s)
print("a': ",dasym)
d2asym = sym.diff(dasym, s)
print('a": ',d2asym)

def a(s,k):
    return (1 - s)**2/(s**2*(k - 1) + 1)
def da(s,k):
    return -2*s*(1 - s)**2*(k - 1)/(s**2*(k - 1) + 1)**2 + (2*s - 2)/(s**2*(k - 1) + 1)
def d2a(s,k):
    return 8*s**2*(1 - s)**2*(k - 1)**2/(s**2*(k - 1) + 1)**3 - 4*s*(k - 1)*(2*s - 2)/(s**2*(k - 1) + 1)**2 - 2*(1 - s)**2*(k - 1)/(s**2*(k - 1) + 1)**2 + 2/(s**2*(k - 1) + 1)
def w(s):
    return 1-(1-s)**2
```

    a:   (1 - alpha)**2/(alpha**2*(self.k - 1) + 1)
    a':  -2*alpha*(1 - alpha)**2*(self.k - 1)/(alpha**2*(self.k - 1) + 1)**2 + (2*alpha - 2)/(alpha**2*(self.k - 1) + 1)
    a":  8*alpha**2*(1 - alpha)**2*(self.k - 1)**2/(alpha**2*(self.k - 1) + 1)**3 - 4*alpha*(2*alpha - 2)*(self.k - 1)/(alpha**2*(self.k - 1) + 1)**2 - 2*(1 - alpha)**2*(self.k - 1)/(alpha**2*(self.k - 1) + 1)**2 + 2/(alpha**2*(self.k - 1) + 1)
    cw:  Integral(Piecewise((I*(1 - alpha)*(alpha - 1)**3/(2*((alpha - 1)**2 - 1)**(3/2)) - I*(1 - alpha)*(alpha - 1)/(2*((alpha - 1)**2 - 1)**(3/2)) + 3*I*(alpha - 1)**2/(2*sqrt((alpha - 1)**2 - 1)) - I/sqrt((alpha - 1)**2 - 1), (alpha - 1)**2 > 1), ((1 - alpha)*(alpha - 1)/(2*sqrt(1 - (alpha - 1)**2)) + sqrt(1 - (alpha - 1)**2)/2 + 1/(2*sqrt(1 - (alpha - 1)**2)), True)), (alpha, 0, 1))

```python
alpha = np.linspace(0,1,101)[1:-1]
fig, ax = plt.subplots(figsize=(8,6))
for k in (0,0.25,0.5,0.75,1.):
    ax.plot(alpha,a(alpha,k),label=r'$s=${0}'.format(k))
ax.set_xlabel(r'$\alpha$')
ax.legend(loc=0)
ax.set_title(r'$a(\alpha)$')
ax.plot()
```

    <IPython.core.display.Javascript object>

    []

```python
alpha = np.linspace(0,1,101)[1:-1]
fig, ax = plt.subplots(figsize=(8,6))
for k in (0,0.25,0.5,0.75,1.):
    ax.plot(alpha,da(alpha,k),label=r'$s=${0}'.format(k))
ax.set_xlabel(r'$\alpha$')
ax.legend(loc=0)
ax.set_title(r"$a'(\alpha)$")
ax.plot()
```

    <IPython.core.display.Javascript object>

    []

```python
alpha = np.linspace(0,1,101)[1:-1]
fig, ax = plt.subplots(figsize=(8,6))
for k in (0,0.25,0.5,0.75,1.):
    ax.plot(alpha,d2a(alpha,k),label=r'$s=${0}'.format(k))
ax.set_xlabel(r'$\alpha$')
ax.legend(loc=0)
ax.set_title(r'$a"(\alpha)$')
ax.plot()
```

    <IPython.core.display.Javascript object>

    []

```python
alpha = np.linspace(0,1,101)[1:-1]
fig, ax = plt.subplots(figsize=(8,6))
ax.plot(alpha,w(alpha),label=r'$w$')
ax.set_xlabel(r'$\alpha$')
#ax.legend(loc=0)
ax.set_title(r"$w(\alpha)$")
ax.plot()
```

    <IPython.core.display.Javascript object>

    []

```python

```


# SiouxFalls - (Parametric) Price of Anarchy

## 1. Imports and data readin

First we load the paminco package and read the SiouxFalls data.
The data was taken from [here](https://github.com/bstabler/TransportationNetworks) and converted to our format of choice: ``XML``.
The data for SiouxFalls comes with paminco and can be easily loaded:

```python
import paminco

sioux = paminco.load_sioux()
```

By default the edge cost equal the link travel time: $F_e = l_e$.
The link travel time is defined as

$$
\begin{equation*}
l_e(x_e) = \text{fft}_e \cdot \left( 1 + B_e \cdot \left(\frac{x}{\text{cap}_e}\right) ^ {p_e} \right)
\end{equation*}
$$

## 2. Fixed Price of Anarchy with Frank-Wolfe

We find minimum cost flows that coincide with user equilibrium and system optimum if we transform the edge cost by:

$$
\begin{align}
    \text{User equilibrium:} \quad F_e &= \int_0^{x_e} l_e(s) ds \\
    \text{System optimum:} \quad F_e &= x_e \cdot l_e \\
\end{align}
$$

```python
import copy

# Calculate user equilibrium -> F_e = integral_0^(x_e) l_e(s)ds
sioux_ue = copy.deepcopy(sioux)
sioux_ue.cost.integrate(inplace=True)
fw_ue = paminco.NetworkFW(sioux_ue)
fw_ue.run(max_iter=500)

# Calculate system optimum -> F_e = x_e * l_e
sioux_so = copy.deepcopy(sioux)
sioux_so.cost.times_x(inplace=True)
fw_so = paminco.NetworkFW(sioux_so)
fw_so.run(max_iter=500)
```

The [Price of Annarchy](https://en.wikipedia.org/wiki/Price_of_anarchy) (PoA) measures the inefficency of the network utilization due to selfish behaviour of the participants.
It is calculated as the ratio of the total (or equivalently average) travel time for all particpants in the user equilbrium to that of the system optimum:

$$
\begin{equation*}
    \text{PoA} = \frac{C(\text{UE})}{C(\text{SO})},
\end{equation*}
$$

where $C$ is the total system travel time (TSTT):

$$
\begin{equation*}
    \text{TTST}(\mathbf{x}) = x_e \cdot l_e,
\end{equation*}
$$

i.e., the objective function of the system optimal minimum cost flow.

```python
def TTST(x):
    return (x * sioux.cost(x)).sum()
```

The Price of Anarchy for SiouxFalls is about ``3.8 percent``:

```python
poa = TTST(fw_ue.x) / TTST(fw_so.x)
poa
```

    1.0388035864499006

## 3. Paramtric Price of Anarchy with MCA

The above example allowed to compute system inefficiency for a fixed demand $\mathbf{B}$.

However, it is of interest how the POA varies by the demand multiplier $\lambda$.
We can achieve this with the ``MCA`` algorithm by calculating parametric user equilibira flows and parametric system optima:

```python
sioux2 = copy.deepcopy(sioux)
sioux2.set_demand(("20", "3", 100000))

sioux_ue = copy.deepcopy(sioux2)
sioux_ue.cost.integrate(inplace=True)
mca_ue = paminco.MCA(sioux_ue)
mca_ue.run()

sioux_so = copy.deepcopy(sioux2)
sioux_so.cost.times_x(inplace=True)
mca_so = paminco.MCA(sioux_so)
mca_so.run()
```

For any demand multiplier in $\lambda \in [0, 1]$, we can now compute a price of anarchy:

```python
import numpy as np

lambdas = np.linspace(1e-5, 1, 100)

cost_user_equilibira = np.array([TTST(mca_ue.flow_at(l)) for l in lambdas])
cost_system_optima = np.array([TTST(mca_so.flow_at(l)) for l in lambdas])
paramtric_poa = cost_user_equilibira / cost_system_optima
```

```python
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(10, 6))
ax.plot(lambdas, paramtric_poa, color="black")
ax.set_xlabel("$\lambda$", fontsize=16)
ax.set_ylabel("PoA", fontsize=16)
ax.set_title("Parametric Price of Anarchy", fontsize=16)
```


# 09 Strain Gage

This is one of the most commonly used sensor.  It is used in many transducers.  Its fundamental operating principle is fairly easy to understand and it will be the purpose of this lecture.

A strain gage is essentially a thin wire that is wrapped on film of plastic.

The strain gage is then mounted (glued) on the part for which the strain must be measured.

## Stress, Strain
When a beam is under axial load, the axial stress, $\sigma_a$, is defined as:
\begin{align*}
\sigma_a = \frac{F}{A}
\end{align*}
with $F$ the axial load, and $A$ the cross sectional area of the beam under axial load.

Under the load, the beam of length $L$ will extend by $dL$, giving rise to the definition of strain, $\epsilon_a$:
\begin{align*}
\epsilon_a = \frac{dL}{L}
\end{align*}
The beam will also contract laterally: the cross sectional area is reduced by $dA$.  This results in a transverval strain $\epsilon_t$.  The transversal and axial strains are related by the Poisson's ratio:
\begin{align*}
\nu = - \frac{\epsilon_t }{\epsilon_a}
\end{align*}
For a metal the Poission's ratio is typically $\nu = 0.3$, for an incompressible material, such as rubber (or water), $\nu = 0.5$.

Within the elastic limit, the axial stress and axial strain are related through Hooke's law by the Young's modulus, $E$:
\begin{align*}
\sigma_a = E \epsilon_a
\end{align*}

## Resistance of a wire

The electrical resistance of a wire $R$ is related to its physical properties (the electrical resistiviy, $\rho$ in $\Omega$/m) and its geometry: length $L$ and cross sectional area $A$.

\begin{align*}
R = \frac{\rho L}{A}
\end{align*}

Mathematically, the change in wire dimension will result inchange in its electrical resistance.  This can be derived from first principle:
\begin{align}
\frac{dR}{R} = \frac{d\rho}{\rho} + \frac{dL}{L} - \frac{dA}{A}
\end{align}
If the wire has a square cross section, then:
\begin{align*}
A & = L'^2 \\
\frac{dA}{A} & = \frac{d(L'^2)}{L'^2} = \frac{2L'dL'}{L'^2} = 2 \frac{dL'}{L'}
\end{align*}
We have related the change in cross sectional area to the transversal strain.
\begin{align*}
\epsilon_t = \frac{dL'}{L'}
\end{align*}
Using the Poisson's ratio, we can relate then relate the change in cross-sectional area ($dA/A$) to axial strain $\epsilon_a = dL/L$.
\begin{align*}
\epsilon_t &= - \nu \epsilon_a \\
\frac{dL'}{L'} &= - \nu \frac{dL}{L} \; \text{or}\\
\frac{dA}{A} & = 2\frac{dL'}{L'} = -2 \nu \frac{dL}{L}
\end{align*}
Finally we can substitute express $dA/A$ in eq. for $dR/R$ and relate change in resistance to change of wire geometry, remembering that for a metal $\nu =0.3$:
\begin{align}
\frac{dR}{R} & = \frac{d\rho}{\rho} + \frac{dL}{L} - \frac{dA}{A} \\
& = \frac{d\rho}{\rho} + \frac{dL}{L} - (-2\nu \frac{dL}{L}) \\
& = \frac{d\rho}{\rho} + 1.6 \frac{dL}{L} = \frac{d\rho}{\rho} + 1.6 \epsilon_a
\end{align}
It also happens that for most metals, the resistivity increases with axial strain.  In general, one can then related the change in resistance to axial strain by defining the strain gage factor:
\begin{align}
S = 1.6 + \frac{d\rho}{\rho}\cdot \frac{1}{\epsilon_a}
\end{align}
and finally, we have:
\begin{align*}
\frac{dR}{R} =  S \epsilon_a
\end{align*}
$S$ is materials dependent and is typically equal to 2.0 for most commercially availabe strain gages. It is dimensionless.

Strain gages are made of thin wire that is wraped in several loops, effectively increasing the length of the wire and therefore the sensitivity of the sensor.

_Question:

Explain why a longer wire is necessary to increase the sensitivity of the sensor_.

Most commercially available strain gages have a nominal resistance (resistance under no load, $R_{ini}$) of 120 or 350 $\Omega$.

Within the elastic regime, strain is typically within the range $10^{-6} - 10^{-3}$, in fact strain is expressed in unit of microstrain, with a 1 microstrain = $10^{-6}$.  Therefore, changes in resistances will be of the same order.  If one were to measure resistances, we will need a dynamic range of 120 dB, whih is typically very expensive.  Instead, one uses the Wheatstone bridge to transform the change in resistance to a voltage, which is easier to measure and does not require such a large dynamic range.

## Wheatstone bridge:

The output voltage is related to the difference in resistances in the bridge:
\begin{align*}
\frac{V_o}{V_s} = \frac{R_1R_3-R_2R_4}{(R_1+R_4)(R_2+R_3)}
\end{align*}

If the bridge is balanced, then $V_o = 0$, it implies: $R_1/R_2 = R_4/R_3$.

In practice, finding a set of resistors that balances the bridge is challenging, and a potentiometer is used as one of the resistances to do minor adjustement to balance the bridge.  If one did not do the adjustement (ie if we did not zero the bridge) then all the measurement will have an offset or bias that could be removed in a post-processing phase, as long as the bias stayed constant.

If each resistance $R_i$ is made to vary slightly around its initial value, ie $R_i = R_{i,ini} + dR_i$.  For simplicity, we will assume that the initial value of the four resistances are equal, ie $R_{1,ini} = R_{2,ini} = R_{3,ini} = R_{4,ini} = R_{ini}$.  This implies that the bridge was initially balanced, then the output voltage would be:

\begin{align*}
\frac{V_o}{V_s} = \frac{1}{4} \left( \frac{dR_1}{R_{ini}} - \frac{dR_2}{R_{ini}} + \frac{dR_3}{R_{ini}} - \frac{dR_4}{R_{ini}} \right)
\end{align*}

Note here that the changes in $R_1$ and $R_3$ have a positive effect on $V_o$, while the changes in $R_2$ and $R_4$ have a negative effect on $V_o$.  In practice, this means that is a beam is a in tension, then a strain gage mounted on the branch 1 or 3 of the Wheatstone bridge will produce a positive voltage, while a strain gage mounted on branch 2 or 4 will produce a negative voltage.  One takes advantage of this to increase sensitivity to measure strain.

### Quarter bridge
One uses only one quarter of the bridge, ie strain gages are only mounted on one branch of the bridge.

\begin{align*}
\frac{V_o}{V_s} = \pm \frac{1}{4} \epsilon_a S
\end{align*}

### Half bridge
One uses half of the bridge, ie strain gages are mounted on two branches of the bridge.

\begin{align*}
\frac{V_o}{V_s} = \pm \frac{1}{2} \epsilon_a S
\end{align*}

### Full bridge

One uses of the branches of the bridge, ie strain gages are mounted on each branch.

\begin{align*}
\frac{V_o}{V_s} = \pm \epsilon_a S
\end{align*}

Therefore, as we increase the order of bridge, the sensitivity of the instrument increases.  However, one should be carefull how we mount the strain gages as to not cancel out their measurement.

_Exercise_

1- Wheatstone bridge

> How important is it to know \& match the resistances of the resistors you employ to create your bridge?
> How would you do that practically?
> Assume $R_1=120\,\Omega$, $R_2=120\,\Omega$, $R_3=120\,\Omega$, $R_4=110\,\Omega$, $V_s=5.00\,\text{V}$.  What is $V_\circ$?

```python
Vs = 5.00
Vo = (120**2-120*110)/(230*240) * Vs
print(Vo)
```

    0.10869565217391304

> How important is it to know \& match the resistances of the resistors you employ to create your bridge?
> How would you do that practically?
> Assume $R_1= R_2 =R_3=120\,\Omega$, $R_4=120.01\,\Omega$, $V_s=5.00\,\text{V}$.  What is $V_\circ$?

```python
Vs = 5.00
Vo = (120**2-120*120.01)/(240.01*240) * Vs
print(Vo)
```

    -0.00010416232656978944

2- Strain gage 1:

One measures the strain on a bridge steel beam.  The modulus of elasticity is $E=190$ GPa.  Only one strain gage is mounted on the bottom of the beam; the strain gage factor is $S=2.02$.

> a) What kind of electronic circuit will you use?  Draw a sketch of it.

> b) Assume all your resistors including the unloaded strain gage are balanced and measure $120\,\Omega$, and that the strain gage is at location $R_2$.  The supply voltage is $5.00\,\text{VDC}$.  Will $V_\circ$ be positive or negative when a downward load is added?

> c) For a loading in which $V_\circ = -1.25\,\text{mV}$, calculate the strain $\epsilon_a$ in units of microstrain.

> d) Calculate the axial stress (in MPa) in the beam under this load.

> e) You now want more sensitivity in your measurement, you install a second strain gage on top of the beam.  Which resistor should you use for this second active strain gage?

> f) With this new setup and the same applied load than previously, what should be the output voltage?

3- Strain Gage with Long Lead Wires

A quarter bridge strain gage Wheatstone bridge circuit is constructed with $120\,\Omega$ resistors and a $120\,\Omega$ strain gage.  For this practical application, the strain gage is located very far away form the DAQ station and the lead wires to the strain gage are $10\,\text{m}$ long and the lead wire have a resistance of $0.080\,\Omega/\text{m}$.  The lead wire resistance can lead to problems since $R_{lead}$ changes with temperature.

> Design a modified circuit that will cancel out the effect of the lead wires.

## Homework


## 1. Parallel and Distributed Optimization

This notebook provides concise BlueFog demos to concepts or algorithms introduced in the tutorial.

### 1.1 Finite-sum optimizaiton example: distributed least square

Suppose $m$ computing nodes collaborate to solve the following problem:

\begin{align*}
\min_{x\in \mathbb{R}^d}\ \sum_{i=1}^n h_i(x) \quad \mbox{where} \quad h_i(x) = \frac{1}{2}\|A_i x - b_i\|^2 \hspace{1cm} \mbox{(Opt-Problem)}
\end{align*}

where $h_i(x): \mathbb{R}^d \to \mathbb{R}$ is a local cost function held by node $i$ and $\{A_i, b_i\}$ are local data. Each node $i$ can evaluate its own data and gradient, but it has to communicate to achieve information from the other node. We let $x^\star$ denote the global solution to the above problem

### 1.2 All-reduce

BlueFog supports the Ring-Allreduce operation as follows:
```
x_bar = bf.allreduce(x_local)
```

In the following code, we activate 8 computing nodes (CPUs) and label them as $0,1,\cdots,7$. We target to let all nodes collaborate to compute the global average of their labels $(\sum_{i=0}^7 i)/8 = 3.5$.

```python
%%writefile Allreduce.py

import bluefog.torch as bf
import torch

bf.init()
x_local = torch.ones(1) * bf.rank()
x_bar = bf.allreduce(x_local)
print('Node {} achieved the global average {}'.format(bf.rank(), x_bar[0]))
```

    Overwriting Allreduce.py

```python
# You can change NUM_PROC into any value smaller than the number of CPUs you have.
import os
NUM_PROC = 2 if os.getenv("TEST_ENV") else 8
NUM_PROC
```

    8

```python
! bfrun -np $NUM_PROC python Allreduce.py
```

    Node 0 achieved the global average 3.5
    Node 1 achieved the global average 3.5
    Node 2 achieved the global average 3.5
    Node 4 achieved the global average 3.5
    Node 5 achieved the global average 3.5
    Node 6 achieved the global average 3.5
    Node 7 achieved the global average 3.5
    Node 3 achieved the global average 3.5

### 1.3 Distributed Gradient Descent

The distributed gradient descent is $x^{k+1} = x^k - \frac{\alpha}{n}\sum_{i=1}^n \nabla h_i(x^k)$. This process can be implemented as

```python
# core distributed gradient descent snippet
grad = A.T.mm(A.mm(x) - b)
next_x = x - alpha * bf.allreduce(grad)
```

```python
%%writefile DistributedGD.py

import bluefog.torch as bf
import torch

bf.init()
# Make sure different agent has different random seed.
torch.manual_seed(12345 * bf.rank())

def generate_data(m, d):
    A = torch.randn(m, d).to(torch.double)
    ns = 0.1*torch.randn(m, 1).to(torch.double)
    x_o = torch.rand(d,1).to(torch.double)
    b = A.mm(x_o) + ns

    return A, b

def check_opt_cond(x, A, b):

    grad_local = A.t().mm(A.mm(x) - b)
    grad = bf.allreduce(grad_local, name='gradient')  # global gradient

    # the norm of global gradient is expected to be 0 (optimality condition)
    global_grad_norm = torch.norm(grad, p=2)
    print("[Distributed Grad Descent] Rank {}: global gradient norm: {}".format(bf.rank(), global_grad_norm))

    return

def distributed_grad_descent(A, b, maxite=5000, alpha=1e-1):

    m, d = A.shape

    x_opt = torch.zeros(d, 1, dtype=torch.double)

    for _ in range(maxite):
        # calculate local gradient
        grad_local = A.t().mm(A.mm(x_opt) - b)

        # global gradient
        grad = bf.allreduce(grad_local, name='gradient')

        # distributed gradient descent
        x_opt = x_opt - alpha*grad

    check_opt_cond(x_opt, A, b)

    return x_opt

if __name__ == "__main__":
    m, d = 20, 5 # dimension of A
    A, b = generate_data(m, d)
    x_opt = distributed_grad_descent(A, b, maxite=200, alpha=1e-2)
```

    Overwriting DistributedGD.py

If $x^k\to x^\star$, it holds that $\sum_{i=1}^m \nabla h_i(x^k) \to 0$. We can use $\|\sum_{i=1}^m \nabla h_i(x)\|$ as a metric to gauge whether distributed gradient descent converge or not.

```python
! bfrun -np $NUM_PROC python DistributedGD.py
```

    [Distributed Grad Descent] Rank 2: global gradient norm: 7.786811775375865e-15
    [Distributed Grad Descent] Rank 3: global gradient norm: 7.786811775375865e-15
    [Distributed Grad Descent] Rank 4: global gradient norm: 7.786811775375865e-15
    [Distributed Grad Descent] Rank 5: global gradient norm: 7.786811775375865e-15
    [Distributed Grad Descent] Rank 6: global gradient norm: 7.786811775375865e-15
    [Distributed Grad Descent] Rank 7: global gradient norm: 7.786811775375865e-15
    [Distributed Grad Descent] Rank 0: global gradient norm: 7.786811775375865e-15
    [Distributed Grad Descent] Rank 1: global gradient norm: 7.786811775375865e-15

### 1.4 Primal Decomposition

We consider the following linearly-constrained resource sharing problem:

\begin{align*}
\min_{\{x_i\}, y}&\quad \|y\|_1 + \frac{1}{2}\sum_{i=1}^n \|A_i x_i - b_i\|^2 \\
\mathrm{s.t.} &\quad\sum_{i=1}^n B_i x_i = y
\end{align*}

where $x_i \in \mathbb{R}^d$, $A_i \in \mathbb{R}^{m\times d}$, and $B_i \in \mathbb{R}^{d\times d}$. For simplicity, we assume each $B_i$ is a invertible matrix. Each node $i$ has local data $\{A_i, B_i, b_i\}$. To solve the above problem in a distributed manner, we introduce $y_i = B_i x_i$ so that $y=\sum_{i=1}^n y_i$. Since $B_i$ is invertible, we have $x_i = B_i^{-1} y_i$. Substituting these facts into the above problem, we achieve

\begin{align*}
\min_{\{y_i\}}&\quad \|y_1 + \cdots + y_n\|_1 + \frac{1}{2}\sum_{i=1}^n \|A_i B_i^{-1} y_i - b_i\|^2
\end{align*}

which can be solved in the following distributed manner. For notation simplicity, we let $C_i = A_i B_i^{-1}$ and $g(y_1,\cdots, y_n) = \|y_1 + \cdots + y_n\|_1$. Following proximal gradient descent, we have

\begin{align*}
z_i^{k+1} &= y_i^k - \alpha C_i^T(C_i y_i^k - b_i), \quad \forall\ i=1,\cdots, n\\
y_i^{k+1} &= [\mathrm{Prox}_{\alpha g}(z_1^{k+1},\cdots, z_n^{k+1})]_{i} =z_i^{k+1} - \alpha v^{k+1}, \quad \forall\ i=1,\cdots, n \\
x_i^{k+1} &= B_i^{-1} y_i^{k+1}, \quad \forall\ i=1,\cdots, n
\end{align*}

where
$$v^{k+1} = \frac{1}{n\alpha^2}\big(\alpha(z^{k+1}_1 + \cdots + z^{k+1}_n) - \mathrm{Prox}_{n \alpha^2 \|\cdot\|_1}(\alpha z^{k+1}_1 + \cdots + \alpha z^{k+1}_n)\big) \quad \mbox{(Need Allreduce Communication)}$$

```python
%%writefile PrimalDecompose.py

import bluefog.torch as bf
import torch

bf.init()
# Make sure different agent has different random seed.
torch.manual_seed(12345 * bf.rank())

def generate_data(m, d):
    A = torch.randn(m, d).to(torch.double)
    B_inv = torch.randn(d, d).to(torch.double)
    ns = 0.1*torch.randn(m, 1).to(torch.double)
    x_o = torch.rand(d,1).to(torch.double)
    b = A.mm(x_o) + ns

    return A, B_inv, b

def soft_threshold(x, kappa):

    zeros = torch.zeros(d,1).to(torch.double)
    x = torch.max(x - kappa, zeros)
    x = torch.min(x + kappa, zeros)

    return x

def check_opt_cond(y, A, B_inv, b, alpha):

    m, d = A.shape
    n = bf.size()
    C = A.mm(B_inv)

    # update z
    grad_local = C.t().mm(C.mm(y) - b)
    z = y - alpha*grad_local
    z_bar = bf.allreduce(z)
    z_sum = z_bar * n

    # update v
    v = (1/(n*alpha*alpha)) * (alpha*z_sum - soft_threshold(alpha*z_sum, n*alpha*alpha))

    # update y
    y_next = z - alpha*v

    # the norm of global gradient is expected to be 0 (optimality condition)
    global_grad_norm = torch.norm((y - y_next), p=2)
    print("[Primal Decomposition] Rank {}: optimality metric norm: {}".format(bf.rank(), global_grad_norm))

    return

def primal_decomposition(A, B_inv, b, maxite=5000, alpha=1e-1):

    m, d = A.shape
    n = bf.size()
    C = A.mm(B_inv)

    y = torch.zeros(d, 1, dtype=torch.double)

    for _ in range(maxite):
        # update z
        grad_local = C.t().mm(C.mm(y) - b)
        z = y - alpha*grad_local
        z_bar = bf.allreduce(z)
        z_sum = z_bar * n

        # update v
        v = (1/(n*alpha*alpha)) * (alpha*z_sum - soft_threshold(alpha*z_sum, n*alpha*alpha))

        # update y
        y = z - alpha*v

        # update x
        x = B_inv.mm(y)

    check_opt_cond(y, A, B_inv, b, alpha)

    return y

if __name__ == "__main__":
    m, d = 20, 5 # dimension of A
    maxite = 5000 if os.getenv("TEST_ENV") else 100
    A, B_inv, b = generate_data(m, d)
    y = primal_decomposition(A, B_inv, b, maxite=maxite, alpha=3e-3)
```

    Overwriting PrimalDecompose.py

We define $G^k_i = \frac{1}{\alpha}\big(y_i^k - [\mathrm{Prox}_{\alpha g}(z_1^{k+1},\cdots, z_n^{k+1})]_i\big)$ and use $\|G^k_i\|$ as the metric to evaluate the optimality of $y_i^k$.

```python
! bfrun -np $NUM_PROC python PrimalDecompose.py
```

    [Primal Decomposition] Rank 2: optimality metric norm: 1.526888906516414e-05
    [Primal Decomposition] Rank 3: optimality metric norm: 0.00010111695583238024
    [Primal Decomposition] Rank 5: optimality metric norm: 2.66226578246676e-06
    [Primal Decomposition] Rank 6: optimality metric norm: 3.1143558672548934e-05
    [Primal Decomposition] Rank 4: optimality metric norm: 1.2605012098981551e-05
    [Primal Decomposition] Rank 1: optimality metric norm: 0.00013532193550425833
    [Primal Decomposition] Rank 0: optimality metric norm: 6.13283876701975e-05
    [Primal Decomposition] Rank 7: optimality metric norm: 4.335475665174773e-06

### 1.5 Distributed ADMM

Suppose we want to optimize the following consensus problem:
$$
    \min_{x} \sum_{i=1}^n h_i(x)
$$
Each machine $i$ can access the function $h_i$ only. Note the problem has a shared $x$ cross all machines. To relax this condition, we can reformat the problem into the following equivalent form:
\begin{align}
    \min_{\{x_i\}, y} & \;\;\; \sum_{i=1}^n h_i(x_i),\\
    {\rm subject\ to} & \;\;\;  x_i = y,\;\; \forall i.
\end{align}

Applying ADMM, we obtain:

\begin{align}
    x^{k+1}_i =&\;\; {\rm argmin}_{x_{i}} \left\{ h_i(x_i) + \langle u_i^k, x_i - y^k\rangle + \frac{\alpha}{2}  \left\|x_i - y^k \right\|^2\right\} \\
    y^{k+1} =&\;\; \frac{1}{n} \left(x_i^{k+1} + \frac{1}{\alpha} u^k_i\right)\\
    u^{k+1}_i = &\;\; u^k_i + \alpha\left(x_i^{k+1} - y^{k+1} \right)
\end{align}

We can further simplify it by noticing that $u^k_1,\ldots,u^k_n$ has mean 0. Here we provide the code snippet of a simple quadratic cost function again.

``` python
# For each agent, it owns data A and b differently.
def CentralizedADMMStepL2(A, b, x, y, u, alpha):
    next_x = ProximalStepL2(A, b, x, y, u, alpha)
    # We use allreduce to mimic the centralized behavior
    # It should be based on PS architecture and using gather and broadcast.
    next_y = bf.allreduce(next_x)   # Without u is okay since allreudce(u) == 0
    next_u = u + alpha * (next_x - next_y)
    return next_x, next_y, next_u
```

```python
! bfrun -np $NUM_PROC python CentralizedADMM.py
```

    [Centralized ADMM] Rank 5: ADMM residue gradient norm: 5.6263947243994325e-15
    [Centralized ADMM] Rank 0: ADMM residue gradient norm: 5.6263947243994325e-15
    [Centralized ADMM] Rank 1: ADMM residue gradient norm: 5.6263947243994325e-15
    [Centralized ADMM] Rank 2: ADMM residue gradient norm: 5.6263947243994325e-15
    [Centralized ADMM] Rank 4: ADMM residue gradient norm: 5.6263947243994325e-15
    [Centralized ADMM] Rank 6: ADMM residue gradient norm: 5.6263947243994325e-15
    [Centralized ADMM] Rank 3: ADMM residue gradient norm: 5.6263947243994325e-15
    [Centralized ADMM] Rank 7: ADMM residue gradient norm: 5.6263947243994325e-15
    Last three entries of x_ar:
     tensor([[0.4883],
            [0.6776],
            [0.6447]], dtype=torch.float64)
    Last three entries of x_admm:
     tensor([[0.4883],
            [0.6776],
            [0.6447]], dtype=torch.float64)

```python
import matplotlib.pyplot as plt
import scipy.io as sio
import os
%matplotlib inline
mse_records_dict = sio.loadmat('results/CentralizedADMM.mat')

plt.semilogy(mse_records_dict["mse"][0], label="Centralized ADMM")
plt.title("Centralized ADMM")
plt.xlabel("iteration")
plt.ylabel("Mean-Square-Error")
plt.grid("on")
plt.legend()
dirname = 'images'
if not os.path.exists(dirname):
    os.makedirs(dirname)
plt.savefig(os.path.join(dirname, 'centralized_admm.png'))
```

You can find the full code in the `CentralizedADMM.py`

**Exercise**: Modify the loss function, (sub-)gradient, and proximal step for $\ell_1 + \ell_2$ cases.


# Laboratório 2: Mofo e Fungicida

### Referente ao capítulo 6

Nesse laboratório, seja $x(t)$ a concentração de mofo que queremos reduzir em um período de tempo fixo. Assumiremos que $x$ tenha crescimento com taxa $r$ e capacidade de carga $M.$ Seja $u(t)$ o fungicida que reduz a população em $u(t)x(t).$ Assim

$$
x'(t) = r(M - x(t)) - u(t)x(t), x(0) = x_0 > 0
$$

O efeito do mofo e do fungicida são negativos para as pessoas ao redor e, por isso, queremos minimizar ambos. Assim, nosso objetivo será

$$
\min_u \int_0^T Ax(t)^2 + u(t)^2 dt
$$

tal que $A$ é o parâmetro que balanceia a importância dos termos do funcional, isto é, quanto mais o seu valor, mais importância em minimizar $x$.

**Resultado de existência:** $f(t,x,u) = Ax^2 + u^2 \implies f_{xx}(t,x,u) = 2A, f_{uu}(t,x,u) = 2$, logo $f$ é continuamente diferenciável nas três variáveis e convexa em $x$ e $u$. $g(t,x,u) = r(M - x(t)) - u(t)x(t) \implies g_{xx} = g_{uu} = 0$ e, de mesma forma é continuamente diferenciável e convexa em $x$ e $u$.

Assim, após encontrarmos $\lambda$ que satisfaça as condições necessárias, temos um resultado de existência, caso a integral seja finita.

## Condições Necessárias

### Hamiltoniano

$$
H = Ax^2 + u^2 + \lambda(r(M - x) - ux)
$$

### Condição de otimalidade

$$
0 = H_u = 2u - \lambda x \implies u^{*}(t) = \frac{1}{2}\lambda(t)x(t)
$$

### Equação adjunta

$$
\lambda '(t) = - H_x = -2Ax(t) + \lambda(t)(r + u(t))
$$

### Condição de transversalidade

$$
\lambda(T) = 0
$$

Deveríamos verificar que $\lambda(t) \ge 0$, mas isso não é feito aqui. Observe que formamos um sistema não linear de equações diferenciais, que tornam a solução analítica muito mais complexa. Por isso, vamos resolver esse problema de forma iterativa.

### Importanto as bibliotecas

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
import sympy as sp

import sys
sys.path.insert(0, '../pyscripts/')

from optimal_control_class import OptimalControl
```

Com a biblioteca `sympy`, de matemática simbólica, é possível obter as condições necessárias sem fazer contas. Para isso, vamos escrever o Hamiltoniano com uma expressão simbólica.

```python
x_sp,u_sp,lambda_sp, r_sp, A_sp, M_sp = sp.symbols('x u lambda r A M')
H = A_sp*x_sp**2 + u_sp**2 + lambda_sp*(r_sp*(M_sp - x_sp) - u_sp*x_sp)
H
```

$\displaystyle A x^{2} + \lambda \left(r \left(M - x\right) - u x\right) + u^{2}$

Assim podemos conseguir suas derivadas e, portanto, as condições necessárias

```python
print('H_x = {}'.format(sp.diff(H,x_sp)))
print('H_u = {}'.format(sp.diff(H,u_sp)))
print('H_lambda = {}'.format(sp.diff(H,lambda_sp)))
```

    H_x = 2*A*x + lambda*(-r - u)
    H_u = -lambda*x + 2*u
    H_lambda = r*(M - x) - u*x

Podemos resolver a equação $H_u = 0$, mas esse passo é importante conferir manualmente também.

```python
eq = sp.Eq(sp.diff(H,u_sp), 0)
sp.solve(eq,u_sp)
```

    [lambda*x/2]

Dessa vez vamos utilizar uma classe escrita em Python que codifica o algoritmo apresentado no Capítulo 5 e no Laboratório 1.

Primeiro precisamos definir as equações importantes das condições necessárias. É importante escrever no formato descrito nesse notebook. `par` é um dicionário com os parâmetros específicos do modelo.

```python
parameters = {'r': None, 'M': None, 'A': None}

diff_state = lambda t, x, u, par: par['r']*(par['M'] - x) - u*x                    # Derivada de x
diff_lambda = lambda t, x, u, lambda_, par: -2*par['A']*x + lambda_*(par['r'] + u) # Derivada de lambda_
update_u = lambda t, x, lambda_, par: 0.5*lambda_*x                                # Atualiza u através de H_u = 0
```

## Aplicando a classe ao exemplo

Vamos fazer algumas exeperimentações. Sinta-se livre para variar os parâmetros ao final do notebook.

```python
problem = OptimalControl(diff_state, diff_lambda, update_u)
```

```python
x0 = 1
T = 5
parameters['r'] = 0.3
parameters['M'] = 10
parameters['A'] = 1
```

```python
t,x,u,lambda_ = problem.solve(x0, T, parameters, h = 0.001)
ax = problem.plotting(t,x,u,lambda_)
```

O controle inicialmente aumenta até atingir um valor constante, da mesma forma que o estado. Dizemos que eles estão em equilíbrio. Eventualmente o controle decresce a 0. Note que o estado não decresce e isso acontece pela ponderação equivalente que damos aos efeitos negativo do mofo e do fungicida. Por isso, podemos sugerir um aumento em $A$.

```python
parameters['A'] = 10
```

```python
t,x,u,lambda_ = problem.solve(x0, T, parameters, h = 0.001)
ax = problem.plotting(t,x,u,lambda_)
```

Aqui o uso de fungicida é muito maior. Como gostaríamos, a quantidade de mofo também é menor em sua constante, porém no final do intervalo, quando a quantidade do fungicida diminui, o nível de mofo cresce consideravelmente. Para fins de comparação, podemos visualizar a diferença de quando $u \equiv 0$ do controle ótimo. Para isso, temos que fazer a integração da derivada de $x$ no intervalo.

```python
integration = solve_ivp(fun = diff_state,
                        t_span = (0,T),
                        y0 = (x0,),
                        t_eval = np.linspace(0,T,len(u)),
                        args = (0,parameters))
```

```python
plt.plot(t, x, integration.t, integration.y[0])
plt.title('Comparação da quantidade mofo')
plt.legend(['Controle ótimo', 'Sem controle'])
plt.grid(alpha = 0.5)
```

## Experimentação

Descomente a célula a seguir e varie os parâmetros para ver seus efeitos:

1. Aumentar $r$ para ver o crescimento mais rápido do mofo. Como o controle vai se comportar?
2. Variar $T$ faz alguma diferença? Como é essa diferença?
3. Variar $M$, a capacidade de carga, gera que efeito no estado?

```python
#x0 = 1
#T = 5
#parameters['r'] = 0.3
#parameters['M'] = 10
#parameters['A'] = 1
#
#t,x,u,lambda_ = problem.solve(x0, T, parameters, h = 0.001)
#roblem.plotting(t,x,u,lambda_)
```

### Este é o final do notebook


```python
from pylab import *
import sympy as sym
from sympy.physics.quantum import Commutator, Dagger, Operator
from sympy.physics.quantum import Bra, Ket, InnerProduct
import numpy as np
init_printing(use_unicode=True)
```

```python
def spinx(s):

    n = int(2.0*s+1)
    sx = np.matrix(zeros((n,n)))
    for a in range(0,n):
        for b in range(0,n):
            if (a==b+1):
                sx[a,b] = sx[a,b] + 0.5*sqrt((s+1)*(a+b+1)-(a+1)*(b+1))
            elif (a==b-1):
                sx[a,b] = sx[a,b] + 0.5*sqrt((s+1)*(a+b+1)-(a+1)*(b+1))

    return sx

def spiny(s):
    n = int(2.0*s+1)
    sy = np.matrix(zeros((n,n),dtype='complex'))

    for a in range(0,n):
        for b in range(0,n):
            if (a==b+1):
                sy[a,b] = sy[a,b] + 0.5j*sqrt((s+1)*(a+b+1)-(a+1)*(b+1))
            elif (a==b-1):
                sy[a,b] = sy[a,b] - 0.5j*sqrt((s+1)*(a+b+1)-(a+1)*(b+1))

    return sy

def spinz(s):

    n = int(2.0*s+1)
    sz = np.matrix(zeros((n,n)))

    for a in range(0,n):
        for b in range(0,n):

            if (a==b):
                sz[a,b] = (s+1-b-1)

    return sz
```

```python
s = 0.5
Sx = spinx(s)
Sy = spiny(s)
Sz = spinz(s)
```

```python
ahat = np.matrix([0,1,0])
bhat = np.matrix([np.cos(np.pi/6),np.sin(-np.pi/6),0])
chat = np.matrix([np.cos(-np.pi/6),np.sin(-np.pi/6),0])
print(ahat,bhat,chat,norm(ahat),norm(bhat),norm(chat))
```

    [[0 1 0]] [[ 0.8660254 -0.5        0.       ]] [[ 0.8660254 -0.5        0.       ]] 1.0 1.0 1.0

```python
Nm = 1000000 #number of measurements
Na = 3 #number of axes

thet = linspace(0,(Na-1.0)*2.0*pi/Na,Na) #array of three values of theta
phi = zeros(3) #set phi = 0 for all three axes

#singlet in z-basis
psi0 = (1.0/sqrt(2.0))*np.matrix([0,1.0,-1.0,0])
```

```python
#single spin operators
Ss = zeros((2,2,Na),dtype='complex')

for i in range(0,Na):
    Ss[:,:,i] = 0.5*matrix([[cos(thet[i]),sin(thet[i])*exp(-1j*phi[i])],[sin(thet[i])*exp(1j*phi[i]),-cos(thet[i])]])

#two-spin operators
Sab = zeros((2**2,2**2,Na**2),dtype='complex')
Vab = zeros((2**2,2**2,Na**2),dtype='complex') #eigenvectors
Eab = zeros((2**2,Na**2),dtype='complex')      #eigenvalues

for i in range(0,Na):
    for j in range(0,Na):
        Sab[:,:,i*Na+j] = kron(Ss[:,:,i],Ss[:,:,j])
        Eab[:,i*Na+j],Vab[:,:,i*Na+j] = eigh(Sab[:,:,i*Na+j])
```

```python
#MEASUREMENT FUNCTION

def doublemeasurement(Na,psi0,Vab,Eab):
    #select pair of axes at random
    m = randint(0,Na**2)

    #coefficients of psi in measurement basis
    psi = zeros(2**2,dtype='complex')

    for i in range(0,2**2):
        psi[i] = dot(conj(Vab[:,i,m]),psi0)

    r = rand()
    csum = abs(psi[0])**2

    n = 0
    while(r>csum):
        n = n + 1
        csum = csum + abs(psi[n])**2

    #MEASURED STATE is indexed by n;
    #returns sign of the product (+ -> same, - -> different)
    return sign(Eab[n,m])
```

```python
#counters
Nsame = 0
Ndiff = 0

#repeat measurement Nm times
for i in range(0,Nm):
    #if
    if (doublemeasurement(Na,psi0,Vab,Eab)>0):
        Nsame = Nsame + 1

    else:
        Ndiff = Ndiff+1

print('The probability of same measurement is ' + str(Nsame/Nm))
print('The probability of different measurement is ' + str(Ndiff/Nm))

if Nsame>4.0*Nm/9.0:
    print('Bell inequality VIOLATED!')
else:
    print('Bell inequality satisfied.')
```

    The probability of same measurement is 0.500589
    The probability of different measurement is 0.499411
    Bell inequality VIOLATED!


For linear initial value problems, stability analysis can be performed by:

1.  Computing the eigenvalues $\lambda_j$ of the ODE system.
2.  Determining the maximal timestep $h$ such that $h \lambda_j$ lies inside the absolute stability region of the time stepping method.

This is particularly useful when applying the method of lines to a PDE system; in this case, the eigenvalues come from the semi-discretization of the PDE.  Here we show how this can be done in NodePy with some simple examples.

```python
from matplotlib import pyplot as plt
from nodepy import rk, semidisc, lm
import numpy as np
```

First we load the classical method of Kutta and the simple first-order upwind semi-discretization of the advection equation (with periodic boundary conditions.

```python
rk4 = rk.loadRKM('RK44')

alpha = [-1, 1]
beta = [0, -1]
alpha.reverse()
beta.reverse()
alpha = np.array(alpha)
beta = np.array(beta)
lm1 = lm.LinearMultistepMethod(alpha=alpha, beta=beta)

alpha = [-1, 1, 0]
beta = [0, -1.5, 0.5]
alpha.reverse()
beta.reverse()
alpha = np.array(alpha)
beta = np.array(beta)
lm2 = lm.LinearMultistepMethod(alpha=alpha, beta=beta)

alpha = [-1, 1/3, 5/9, 1/9]
beta = [0, -16/9, 0, 0]
alpha.reverse()
beta.reverse()
alpha = np.array(alpha)
beta = np.array(beta)
lm3 = lm.LinearMultistepMethod(alpha=alpha, beta=beta)

alpha = [-1, 1.49323762497707, -0.574370781405754, 0.0855838379295368, -0.00445068150085398]
beta = [0, -2.10313656405320, 2.80393876806197, -1.68484817541425, 0.400601121454727 ]
alpha.reverse()
beta.reverse()
alpha = np.array(alpha)
beta = np.array(beta)

lm4 = lm.LinearMultistepMethod(alpha=alpha, beta=beta)
# ab3=lm.Adams_Bashforth(3)
upwind = semidisc.upwind_advection_matrix(100,dx=0.1)
```

```python
lm1.plot_stability_region()
print('lm1.is_zero_stable()', lm1.is_zero_stable())
print('lm1.order()',lm1.order())

lm2.plot_stability_region()
print('lm2.is_zero_stable()', lm2.is_zero_stable())
print('lm2.order()',lm2.order())

lm3.plot_stability_region()
print('lm3.is_zero_stable()', lm3.is_zero_stable())
print('lm3.order()',lm3.order(tol=0.1))

lm4.plot_stability_region()
print('lm4.is_zero_stable()', lm4.is_zero_stable())
print('lm4.order()',lm4.order(tol=0.1))

```

```python
lm4.plot_stability_region()
lm4.is_zero_stable()
```

```python
print('lm1.latex()', lm1.latex())
print('lm3.latex()', lm3.latex())
print('lm3.latex()', lm4.latex())

```

    lm1.latex() \begin{align} y_{n + 1} - y_{n} = h f(y_{n})\end{align}
    lm3.latex() \begin{align} y_{n + 3} -0.333 y_{n + 2} -0.556 y_{n + 1} -0.111 y_{n} = 1.778h f(y_{n + 2})\end{align}
    lm3.latex() \begin{align} y_{n + 4} -1.493 y_{n + 3} + 0.574 y_{n + 2} -0.086 y_{n + 1} + 0.004 y_{n} = 2.103h f(y_{n + 3}) -2.804h f(y_{n + 2}) + 1.685h f(y_{n + 1}) -0.401h f(y_{n})\end{align}

Now we can use the method `linearly_stable_step_size` to automatically compute the maximum absolutely stable step size for this combination.  As a bonus, the function shows a plot of the absolute stability region and the (maximally) scaled spectrum.

```python
# rk.linearly_stable_step_size(rk4,upwind)
ab3.plot_stability_region()
```

We can do the same for another linear spatial discretization.

```python
spectral = semidisc.load_semidisc('spectral difference advection',order=6)
rk.linearly_stable_step_size(rk4,spectral.L)
```

For a nonlinear discretization, like WENO, a simple approach is to consider the eigenvalues of one or more linearizations.  Here we use the linearization given in the limit of a very smooth function.

```python
weno5 = semidisc.weno5_linearized_matrix(100)
rk.linearly_stable_step_size(rk4,weno5)
```

The next example uses a semi-discretization of an advection-diffusion PDE.  For this stiff PDE, RK4 requires a very small step size and would be inefficient.

```python
centered_diff = semidisc.centered_advection_diffusion_matrix(10.,1.,N=100)
rk.linearly_stable_step_size(rk4,centered_diff)
```

A better choice would be a method with good real-axis stability, like the Runge-Kutta-Chebyshev methods.  Let's try one.

```python
rkc = rk.RKC2(10)
rk.linearly_stable_step_size(rkc,centered_diff)
```

Unfortunately, this doesn't work as well as hoped because the PDE in question has a small but non-zero convective component.  We can modify the RKC method in order to accommodate this:

```python
rkc_damped = rk.RKC2(10, epsilon=0.2)
rk.linearly_stable_step_size(rkc_damped,centered_diff)
```

Notice that the step size allowed is 10x larger than with the undamped RKC method, and over 20x larger than with the RK4 method.

How does the real-axis stability compare between the damped and undamped RKC methods?

```python
rkc.real_stability_interval()
```

    22.802433359971374

```python
rkc_damped.real_stability_interval()
```

    64.37373670307043

As a last example, here's an interactive widget that lets you design an RKC method (choosing the number of stages and the damping parameter) and see the resulting allowed step size.  Experiment with it.  How much damping do you need?  Does it depend on the number of stages?  What method is most efficient?

```python
from ipywidgets import interact, IntSlider, FloatSlider
```

```python
def plot_rkc2_stability_region(s=5,epsilon=0.):
    rkc = rk.RKC2(s, epsilon)
    h = rk.linearly_stable_step_size(rkc,centered_diff, tol=1.e-13)
    print('Maximum stable step size: {:.3e}'.format(h))
    plt.show()
```

```python
interact(plot_rkc2_stability_region,
         s=IntSlider(min=1,max=20,value=5),
         epsilon=FloatSlider(min=0.,max=2.,step=0.01,value=0.));
```

    interactive(children=(IntSlider(value=5, description='s', max=20, min=1), FloatSlider(value=0.0, description='…


# Price and Markdown Optimization for Multiple Time Intervals

This model optimizes prices for N consecutive time periods
each of which has its own demand function. It is assumed that
the stock level of the product is limited and the goal
is to sell out the stock in time maximizing the revenue.

### Detailed Description
See [Introduction to Algorithmic Marketing](https://algorithmicweb.wordpress.com/ ) book.

### Data
We use a simulator. No external datasets are used.

### Papers
Ferreira2017 (see [papers](https://github.com/ikatsov/tensor-house/blob/master/resources/papers.md))

```python
%matplotlib inline
import sympy as sy
import numpy as np
from tabulate import tabulate
import matplotlib.pyplot as plt
from scipy.optimize import linprog

def tabprint(msg, A):
    print(msg)
    print(tabulate(A, tablefmt="fancy_grid", floatfmt=".2f"))
```

```python
plevels = [89, 79, 69, 59, 49] # allowed price levels
C = 700                        # stock level

price = sy.symbols("price")

def rectified(f):
    return sy.Piecewise( (0, f < 0), (f, True))

# Demand functions estimated for each week of the sale
demands = [rectified(1500 - 10*price), # week 1
           rectified(1300 - 15*price), # week 2
           rectified(1200 - 15*price), # week 3
           rectified(1100 - 18*price)] # week 4
```

```python
# Evaluate values of demand functions for each price level
D = np.array([[q.subs(price, p) for p in plevels] for q in demands])
tabprint("D =", D)

# Evaluate revenue for each demand function and each price level
R = np.array([[p*q.subs(price, p) for p in plevels] for q in demands])
tabprint("R =", R)
```

    D =
    ╒════════╤════════╤════════╤════════╤═════════╕
    │ 610.00 │ 710.00 │ 810.00 │ 910.00 │ 1010.00 │
    ├────────┼────────┼────────┼────────┼─────────┤
    │   0.00 │ 115.00 │ 265.00 │ 415.00 │  565.00 │
    ├────────┼────────┼────────┼────────┼─────────┤
    │   0.00 │  15.00 │ 165.00 │ 315.00 │  465.00 │
    ├────────┼────────┼────────┼────────┼─────────┤
    │   0.00 │   0.00 │   0.00 │  38.00 │  218.00 │
    ╘════════╧════════╧════════╧════════╧═════════╛
    R =
    ╒══════════╤══════════╤══════════╤══════════╤══════════╕
    │ 54290.00 │ 56090.00 │ 55890.00 │ 53690.00 │ 49490.00 │
    ├──────────┼──────────┼──────────┼──────────┼──────────┤
    │     0.00 │  9085.00 │ 18285.00 │ 24485.00 │ 27685.00 │
    ├──────────┼──────────┼──────────┼──────────┼──────────┤
    │     0.00 │  1185.00 │ 11385.00 │ 18585.00 │ 22785.00 │
    ├──────────┼──────────┼──────────┼──────────┼──────────┤
    │     0.00 │     0.00 │     0.00 │  2242.00 │ 10682.00 │
    ╘══════════╧══════════╧══════════╧══════════╧══════════╛

```python
# Now we solve the following optimization problem:
# (q is demand, P is price, T is the number of time periods, and K is the number of price levels)
```

$\text{max} \quad \sum_{t=1}^{T} \sum_{i=1}^{K} z_{it} \cdot P_{i} \cdot q\left(P_{i},t\right)$

$\text{subject to}$

$\quad \sum_{t=1}^{T} \sum_{i=1}^{K} z_{it} \cdot q\left(P_{i},t\right) \le C $

$\quad \sum_{i=1}^{K} z_{it} = 1, \quad \text{for}\ t=1,\ldots,T $

$\quad z_{it}\ge 0 $

```python
L = len(demands)*len(plevels)

# First, we generate a binary mask to ensure that all z's
# in one time interval sum up to 1.0, that is z.M = B
M = np.array([[
    1 if i >= len(plevels)*j and i < len(plevels)*(j+1) else 0
    for i in range(L)
] for j in range(len(demands))])

tabprint("M = ", M)

B = [1 for i in range(len(demands))]

# Second, we ensure that the sum of all demands is less than the availbale stock level,
# that is z.Df <= C
Df = np.array(D).reshape(1, L)

res = linprog(-np.array(R).flatten(),
              A_eq=M,
              b_eq=B,
              A_ub=Df,
              b_ub=np.array([C]),
              bounds=(0, None))

print("Revenue value: $", -res.fun)

# Each column of the solution matrix corresponds to a time period (one week).
# Each row corresponds to z value that can be interpreted as the percentage
# of time z't price level should be used in the corresponding time period.
tabprint("Price schedule:", np.array(res.x).reshape(len(demands), len(plevels)).T)
```

    M =
    ╒══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╤══════╕
    │ 1.00 │ 1.00 │ 1.00 │ 1.00 │ 1.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │
    ├──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
    │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 1.00 │ 1.00 │ 1.00 │ 1.00 │ 1.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │
    ├──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
    │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 1.00 │ 1.00 │ 1.00 │ 1.00 │ 1.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │
    ├──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
    │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 0.00 │ 1.00 │ 1.00 │ 1.00 │ 1.00 │ 1.00 │
    ╘══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╧══════╛
    Revenue value: $ 61400.000000123204
    Price schedule:
    ╒══════╤══════╤══════╤══════╕
    │ 1.00 │ 0.29 │ 0.47 │ 0.33 │
    ├──────┼──────┼──────┼──────┤
    │ 0.00 │ 0.71 │ 0.53 │ 0.33 │
    ├──────┼──────┼──────┼──────┤
    │ 0.00 │ 0.00 │ 0.00 │ 0.33 │
    ├──────┼──────┼──────┼──────┤
    │ 0.00 │ 0.00 │ 0.00 │ 0.00 │
    ├──────┼──────┼──────┼──────┤
    │ 0.00 │ 0.00 │ 0.00 │ 0.00 │
    ╘══════╧══════╧══════╧══════╛

```python

```