# Playing with coupled oscillators - v3

This notebook builds on previous notebooks such as Orbit_games.ipynb to implement the solution of linear coupled differential equations using `solve_ivp` as well as with more analytic methods.  The formalism and notation is based on Taylor, Chapter 11: *Coupled Oscillators and Normal Modes*.

The general form of the equations to solve is compactly expressed in matrix form:

$\begin{align}
  \mathbf{M} \mathbf{\ddot q} = - \mathbf{K} \mathbf{q}
\end{align}$

where $\mathbf{M}$ and $\mathbf{K}$ are $N \times N$ symmetric matrices and $\mathbf{q}$ is a vector of generalized coordinates (this is $\mathbf{x}$ in the beginning discussion in Taylor).

We can solve this directly as differential equations or solve for normal modes.  For the former, we map the problem onto linear differential equations and the vectors $\mathbf{y}$ and $d\mathbf{y}/dt$, which are each of length $2N$.  Here is the $N=2$ mapping:

$\begin{align}
  \mathbf{y} = \left( \begin{array}{c}
                         q_1 \\
                         q_2 \\
                         \dot q_1 \\
                         \dot q_2
                      \end{array}
                       \right)
 \qquad\qquad
\frac{d\mathbf{y}}{dt} = \left( \begin{array}{c}
                         \dot q_1 \\
                         \dot q_2 \\
                         \ddot q_1 \\
                         \ddot q_2
                      \end{array}
                       \right)
\end{align}$

Then for the $\ddot q$ parts we use the matrix solution of our equation:

$\begin{align}
  \mathbf{\ddot q} = - \mathbf{M}^{-1} \mathbf{K} \mathbf{q}
\end{align}$

where $\mathbf{M}^{-1}$ means the matrix inverse.  Check out the implementation below!

To solve for normal modes, we solve the generalized eigenvalue problem

$\begin{align}
  (\mathbf{K} - \omega^2 \mathbf{M}) \mathbf{a} = 0
\end{align}$

using functions from `scipy.linalg` to find the eigenvalues and eigenvectors.  We use these below together with the numerical solutions to make plots of the normal modes.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
from scipy import linalg as LA
```

```python
# Change the common font size
font_size = 14
plt.rcParams.update({'font.size': font_size})
```

```python
class CoupledOscillators:
    """
    Solve the equations for linear coupled oscillators in matrix formulation.
    """

    def __init__(self, M_matrix, K_matrix, b_matrix=np.zeros(1)):
        self.M_matrix = M_matrix
        self.K_matrix = K_matrix
        self.q_len = len(self.M_matrix)
        if b_matrix.any():
            self.b_matrix = b_matrix
        else:
            self.b_matrix = np.zeros((self.q_len))
        self.full_matrix = np.zeros((self.q_len, self.q_len))

    def dy_dt(self, t, y):
        """
        This function returns the right-hand side of the diffeq:
        [dq_vec/dt d^2q_vec/dt^2]

        Parameters
        ----------
        t : float
            time
        y : float
            vector with y[:q_len] = q_vec, y[q_len:] = dqvec/dt

        """
        dy_dt_vec = np.zeros(2*self.q_len)   # Initialize to twice q_vec size
        # Set the upper half [:self.q_len] of dy_dt to the lower half of y
        dy_dt_vec[:self.q_len] = y[self.q_len:]
        # Set the lower half [self.q_len:] of dy_dt to the vector that
        #  results from M^{-1} K y where @ is matrix multiplication
        dy_dt_vec[self.q_len:] = -1.* LA.inv(self.M_matrix) @ \
                    ( self.K_matrix @ y[:self.q_len] + \
                      self.b_matrix @ y[self.q_len:] )
        return dy_dt_vec

    def solve_ode(self, t_pts, q_vec_0, q_vec_dot_0,
                  method='RK45',
                  abserr=1.0e-10, relerr=1.0e-10):
        """
        Solve the ODE given initial conditions.
        Use solve_ivp with the option of specifying the method.
        Specify smaller abserr and relerr to get more precision.
        """
        y = np.concatenate((q_vec_0, q_vec_dot_0))
        solution = solve_ivp(self.dy_dt, (t_pts[0], t_pts[-1]),
                             y, t_eval=t_pts, method=method,
                             atol=abserr, rtol=relerr)
        q_vec, q_vec_dot = np.array_split(solution.y, 2)
        return q_vec, q_vec_dot

    def find_eigenmodes(self):
        """
        Find the normal modes.
        """
        eig_vals, eig_vecs = LA.eigh(K_matrix, M_matrix)
        self.frequencies = np.sqrt(eig_vals)
        return eig_vals, eig_vecs

    def plot_eigenmodes(self, t_pts, output_file=None):
        """
        plot the normal modes separately
        """
        from itertools import cycle
        colors = ['blue', 'green', 'red', 'purple']

        eig_vals, eig_vecs = self.find_eigenmodes()
        for i in np.arange(self.q_len):
            q_vec_0 = eig_vecs[:,i]   # this is the i'th eigenvector
            q_vec_dot_0 = np.zeros(self.q_len)  # start with all qdots = 0
            q_vec, q_vec_dot = self.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

            fig, axes = plt.subplots(self.q_len, 1,
                                     figsize=(10, 2.*self.q_len))
            color_list = cycle(colors)
            for j in np.arange(self.q_len):
               axes[j].plot(t_pts, q_vec[j], color=next(color_list))
               axes[j].set_xlabel(r'$t$')
               axes[j].set_ylabel(fr'$q_{j+1:d}$')
               axes[j].axvline(2.*np.pi/np.sqrt(eig_vals[i]),
                               color='black', linestyle=':')
            overall_title = fr'Normal mode {i+1:d}, ' + \
                            fr'$\omega = {self.frequencies[i]:.2f}$'
            fig.suptitle(overall_title, va='center')
            fig.tight_layout()
            if output_file:
                 output_file_name = output_file.replace('.', f'_{i:d}.', 1)
                 fig.savefig(output_file_name, dpi=200, bbox_inches='tight')

```

```python
def start_stop_indices(t_pts, plot_start, plot_stop):
    start_index = (np.fabs(t_pts-plot_start)).argmin()  # index in t_pts array
    stop_index = (np.fabs(t_pts-plot_stop)).argmin()  # index in t_pts array
    return start_index, stop_index
```

## Section 11.2

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, 2.*k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  2.]]

    omega^2 = [1. 3.]
    frequencies = [1.         1.73205081]

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.70710678 -0.70710678]
    [-0.70710678  0.70710678]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
co1.plot_eigenmodes(t_pts, 'Section11p2.png')
```

### Figure 11.6

This has $A_1 = 1$, $A_2 = 0.7$, $\delta_1 = 0$ and $\delta_2 = \pi/2$.

```python
t_start = 0.
t_end = 50.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A_i -> -A_i to match convention in Figure 11.6
A_1 = -1.
delta_1 = 0.
A_2 = -0.7
delta_2 = np.pi/2.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = A_1 * eig_vecs[:,0] * np.cos(-delta_1) + \
          A_2 * eig_vecs[:,1] * np.cos(-delta_2)
q_vec_dot_0 = omega_1 * A_1 * eig_vecs[:,0] * np.sin(delta_1) + \
              omega_2 * A_2 * eig_vecs[:,1] * np.sin(delta_2)
q_vec_11p6, q_vec_dot_11p6 = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p6[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'General solution: ' + fr'$A_1 = {A_1:.2f}$, ' + \
                                       fr'$\delta_1 = {delta_1:.2f}$, ' + \
                                       fr'$A_2 = {A_2:.2f}$, ' + \
                                       fr'$\delta_2 = {delta_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

## Problem 11.5

Almost the same as section 11.2, but mass m2 is not attached by a spring to the wall.

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  1.]]

    omega^2 = [0.38196601 2.61803399]
    frequencies = [0.61803399 1.61803399]

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.52573111 -0.85065081]
    [-0.85065081  0.52573111]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
co1.plot_eigenmodes(t_pts)
```

## Problem 11.7

Writing the general solution to the cart problem in Section 11.2 in a different form and plot two solutions.

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, 2.*k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  2.]]

    omega^2 = [1. 3.]
    frequencies = [1.         1.73205081]

```python
t_start = 0.
t_end = 30.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A -> -A to match convention in Taylor
A = -1.
B_1 = A
B_2 = 0.
C_1 = 0.
C_2 = 0.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = B_1 * eig_vecs[:,0] + \
          B_2 * eig_vecs[:,1]
q_vec_dot_0 = omega_1 * C_1 * eig_vecs[:,0] + \
              omega_2 * C_2 * eig_vecs[:,1]
q_vec_11p7a, q_vec_dot_11p7a = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p7a[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'Specific solution 1: ' + fr'$B_1 = {B_1:.2f}$, ' + \
                                          fr'$C_1 = {C_1:.2f}$, ' + \
                                          fr'$B_2 = {B_2:.2f}$, ' + \
                                          fr'$C_2 = {C_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A -> -A to match convention in Taylor
A = -1.
B_1 = A / 2.
B_2 = A / 2.
C_1 = 0.
C_2 = 0.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = B_1 * eig_vecs[:,0] + \
          B_2 * eig_vecs[:,1]
q_vec_dot_0 = omega_1 * C_1 * eig_vecs[:,0] + \
              omega_2 * C_2 * eig_vecs[:,1]
q_vec_11p7a, q_vec_dot_11p7a = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p7a[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'Specific solution 1: ' + fr'$B_1 = {B_1:.2f}$, ' + \
                                          fr'$C_1 = {C_1:.2f}$, ' + \
                                          fr'$B_2 = {B_2:.2f}$, ' + \
                                          fr'$C_2 = {C_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

## Double pendulum with small oscillations

```python
m = 1.
L = 1.
M_matrix = m * L**2 * np.array([ [2., 1.], [1., 1.] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
omega_0 = 1.
K_matrix = m * L**2 * np.array([ [2.*omega_0**2, 0.], [0., omega_0**2]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

double_pendulum = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = double_pendulum.find_eigenmodes()

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {double_pendulum.frequencies}')
print(f'{2. + np.sqrt(2.):.4f}  {2. - np.sqrt(2.):.4f}')
```

    M matrix:
     [[2. 1.]
     [1. 1.]]

    K matrix:
     [[2. 0.]
     [0. 1.]]

    omega^2 = [0.58578644 3.41421356]
    frequencies = [0.76536686 1.84775907]
    3.4142  0.5858

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.38268343 -0.5411961 ]
    [-0.92387953  1.30656296]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
double_pendulum.plot_eigenmodes(t_pts)
```

```python

```


```python
from sympy import *
init_printing()
%matplotlib inline
```

```python
x, y, theta, u, v = symbols("x y theta u v", real=True)
g = symbols("g", real=True, positive=True)
```

```python
# we want to find the minimum distance from an edge pixel to the grid with size g (pixels),
# offset u, v (pixels), and angle theta

# distance to grid will be the minimum of distances to the nearest gridline
# (rounding (x-u, y-v) % g toward zero) after counter-rotation
S, C = sin(theta), cos(theta)
R = Matrix([[C, -S], [S, C]])
Dxy = Abs((R*Matrix([[x], [y]]) - Matrix([[u - g/2], [v - g/2]])) % g - Matrix([[g/2], [g/2]]))
D = Min(Dxy[0], Dxy[1])
```

```python
plot(D.subs([(g, 1), (u, 0), (v, 0), (theta, 0.1), (y, 0)]))
```

```python
cse(diff(D, theta))
```

$$\left ( \left [ \left ( x_{0}, \quad \cos{\left (\theta \right )}\right ), \quad \left ( x_{1}, \quad \sin{\left (\theta \right )}\right ), \quad \left ( x_{2}, \quad x x_{0} - x_{1} y\right ), \quad \left ( x_{3}, \quad \frac{g}{2}\right ), \quad \left ( x_{4}, \quad x x_{1}\right ), \quad \left ( x_{5}, \quad x_{0} y\right ), \quad \left ( x_{6}, \quad - x_{10} + x_{3} + x_{4} + x_{5}\right ), \quad \left ( x_{7}, \quad x_{3} - \left(x_{6}\bmod{g}\right)\right ), \quad \left ( x_{8}, \quad - u + x_{2} + x_{3}\right ), \quad \left ( x_{9}, \quad x_{3} - \left(x_{8}\bmod{g}\right)\right ), \quad \left ( x_{11}, \quad \left|{x_{9}}\right|\right ), \quad \left ( x_{12}, \quad \left|{x_{7}}\right|\right )\right ], \quad \left [ \left(- x_{2} \theta\left(x_{11} - x_{12}\right) \operatorname{sign}{\left (x_{7} \right )} \left(\left. \frac{\partial}{\partial \xi_{1}} \xi_{1}\bmod{g} \right|_{\substack{ \xi_{1}=x_{6} }}\right)\right) - \left(\left(- x_{4} - x_{5}\right) \theta\left(- x_{11} + x_{12}\right) \operatorname{sign}{\left (x_{9} \right )} \left(\left. \frac{\partial}{\partial \xi_{1}} \xi_{1}\bmod{g} \right|_{\substack{ \xi_{1}=x_{8} }}\right)\right)\right ]\right )$$

```python
dx, dy, ds = symbols("dx dy ds", real=True)
diff(abs(dx * cos(theta) + dy * sin(theta)), theta)
```

```python
print_python(diff((dx * cos(theta) + dy * sin(theta))**2, theta))
```

    dx = Symbol('dx')
    theta = Symbol('theta')
    dy = Symbol('dy')
    e = (-2*dx*sin(theta) + 2*dy*cos(theta))*(dx*cos(theta) + dy*sin(theta))

```python
Ix, Iy = Function("Ix", real=True), Function("Iy", real=True)  # x and y image gradients
diff(abs(Ix(x, y) - Ix(x + ds*cos(theta), y + ds*sin(theta))), theta)
```

$$\left(ds \sin{\left (\theta \right )} \left. \frac{\partial}{\partial \xi_{1}} \operatorname{Ix}{\left (\xi_{1},ds \sin{\left (\theta \right )} + y \right )} \right|_{\substack{ \xi_{1}=ds \cos{\left (\theta \right )} + x }} - ds \cos{\left (\theta \right )} \left. \frac{\partial}{\partial \xi_{2}} \operatorname{Ix}{\left (ds \cos{\left (\theta \right )} + x,\xi_{2} \right )} \right|_{\substack{ \xi_{2}=ds \sin{\left (\theta \right )} + y }}\right) \operatorname{sign}{\left (\operatorname{Ix}{\left (x,y \right )} - \operatorname{Ix}{\left (ds \cos{\left (\theta \right )} + x,ds \sin{\left (\theta \right )} + y \right )} \right )}$$

```python
obj = Min(Abs(dx * cos(theta) + dy * sin(theta)),
         Abs(dy * cos(theta) - dx * sin(theta)))
simplify(diff(obj, theta))
```

```python
print_python(cse(Out[30]))
```

    x0 = Symbol('x0')
    theta = Symbol('theta')
    x1 = Symbol('x1')
    x2 = Symbol('x2')
    dx = Symbol('dx')
    dy = Symbol('dy')
    x3 = Symbol('x3')
    x4 = Symbol('x4')
    x5 = Symbol('x5')
    e = ([(x0, cos(theta)), (x1, sin(theta)), (x2, dx*x0 + dy*x1), (x3, dx*x1 - dy*x0), (x4, Abs(x2)), (x5, Abs(x3))], [x2*Heaviside(x4 - x5)*sign(x3) - x3*Heaviside(-x4 + x5)*sign(x2)])

```python
simplify(diff(obj, theta, theta))
```

```python
vs, es = cse([obj, diff(obj, theta), diff(obj, (theta, 2))])
for v, e in vs:
    print(v, '=', e)
for e in es:
    print('=', e)
```

    x0 = cos(theta)
    x1 = sin(theta)
    x2 = dx*x0 + dy*x1
    x3 = Abs(x2)
    x4 = dx*x1
    x5 = dy*x0
    x6 = x4 - x5
    x7 = Abs(x6)
    x8 = sign(x2)
    x9 = -x3 + x7
    x10 = Heaviside(x9)
    x11 = x10*x8
    x12 = Heaviside(x3 - x7)
    x13 = sign(x6)
    x14 = x13*x2
    x15 = x6*x8
    x16 = (x14 + x15)*DiracDelta(x9)
    = Min(x3, x7)
    = x11*(-x4 + x5) + x12*x14
    = 2*x10*x6**2*DiracDelta(x2) - x11*x2 - x12*x13*x6 + 2*x12*x2**2*DiracDelta(x6) - x14*x16 - x15*x16

```python
obj = Min((dx * cos(theta) + dy * sin(theta))**2,
         (dy * cos(theta) - dx * sin(theta))**2)

vs, es = cse([obj, diff(obj, theta)])
for v, e in vs:
    print(v, '=', e)
for e in es:
    print('=', e)
```

    x0 = cos(theta)
    x1 = dx*x0
    x2 = sin(theta)
    x3 = dy*x2
    x4 = x1 + x3
    x5 = x4**2
    x6 = dy*x0
    x7 = dx*x2
    x8 = x6 - x7
    x9 = x8**2
    x10 = -x5
    x11 = (-x6 + x7)**2
    x12 = Heaviside(-x11 + x5)
    x13 = x10 + x11
    x14 = Heaviside(x13)
    = Min(x5, x9)
    = x4*(2*x6 - 2*x7)*Heaviside(x10 + x9) + x8*(-2*x1 - 2*x3)*Heaviside(x5 - x9)
    = -2*x11*x12 + 2*x11*x14 - 16*x11*x5*DiracDelta(x13) + 2*x12*x5 - 2*x14*x5

```python
delta = symbols("delta")  # huber loss

```

```python
# p(Ix(x) | x, u, g) = Ix*cos((x-u)*2pi/g)**2
plot(cos(x)**2)
```

```python
# f = 2*pi/g  - frequency in rad/pix instead of period in pixels
f = symbols("f", real=True, positive=True)
obj2 = Ix(x)*cos((x-u)*f)**2
vs, es = cse([obj2, diff(obj2, u), diff(obj2, f)])
for v, e in vs:
    print(v, '=', e)
for e in es:
    print('=', e)
```

    x0 = Ix(x)
    x1 = -u + x
    x2 = f*x1
    x3 = cos(x2)
    x4 = 2*x0*x3*sin(x2)
    = x0*x3**2
    = f*x4
    = -x1*x4

```python
# hang on, this is just weighted least squares, residual is sin(f*(x-u)), weight is Ix(x)
r = sin(f*(x-u))
J = Matrix([r]).jacobian(Matrix([u, f]))
r, J
```

$$\left ( \sin{\left (f \left(- u + x\right) \right )}, \quad \left[\begin{matrix}- f \cos{\left (f \left(- u + x\right) \right )} & \left(- u + x\right) \cos{\left (f \left(- u + x\right) \right )}\end{matrix}\right]\right )$$

```python
vs, es = cse([r, J])
for v, e in vs:
    print(v, '=', e)
for e in es:
    print('=', e)
```

    x0 = -u + x
    x1 = f*x0
    x2 = cos(x1)
    = sin(x1)
    = Matrix([[-f*x2, x0*x2]])

```python
# problem: the incentive is for f to = 0, and then everything is solved
# we need to encourage a match better
r = sin(f*(x-u))
J = Matrix([r]).jacobian(Matrix([u]))
r, J
```

$$\left ( \sin{\left (f \left(- u + x\right) \right )}, \quad \left[\begin{matrix}- f \cos{\left (f \left(- u + x\right) \right )}\end{matrix}\right]\right )$$

```python
pu, pv = symbols("p_u p_v", real=True)

S, C = sin(theta), cos(theta)
R*Matrix([pu, pv]) + Matrix([x, y])
```

$$\left[\begin{matrix}p_{u} \cos{\left (\theta \right )} - p_{v} \sin{\left (\theta \right )} + x\\p_{u} \sin{\left (\theta \right )} + p_{v} \cos{\left (\theta \right )} + y\end{matrix}\right]$$

```python
(R*Matrix([pu, pv]) + Matrix([x, y])).jacobian(Matrix([theta, x, y]))
```


# Set Theory Cheatsheet

集合论 (set theory) 的 cheatsheet.

## 1. 集合代数

### 定义 1.1

设 $A, B$ 为集合, 如果 $B$ 中的每个元素都是 $A$ 中的元素, 则称 $B$ 是 $A$ 的
*子集合*, 简称 *子集*.  这时也称 *B 被 A 包含*, 或 *A 包含 B*, 记作
$B \subseteq A$.

如果 $B$ 不被 $A$ 包含, 则记作 $B \nsubseteq A$.

包含的符号化表示为
$$B \subseteq A \Leftrightarrow \forall x(x \in B \rightarrow x \in A)$$

### 定义 1.2

设 $A, B$ 为集合, 如果 $A \subseteq B$ 且 $B \subseteq A$, 则称 $A$ 与 $B$
*相等, 记作 $A = B$, 如果不相等, 则记作 $A \ne B$.

相等的符号化表示为
$$A = B \Leftrightarrow A \subseteq B \land B \subseteq A$$

### 定义 1.3

设 $A, B$ 为集合, 如果 $B \subseteq A$ 且 $B \ne A$, 则称 $B$ 是 $A$ 的
*真子集*, 记作 $B \subset A$, 如果 $B$ 不是 $A$ 的真子集, 则记作
$A \not\subset B$.

真子集的符号化表示为
$$B \subset A \Leftrightarrow B \subseteq A \land B \ne A$$

### 定义 1.4

不含任何元素的集合叫做 *空集*, 记作 $\emptyset$.

空集可以符号化表示为 $$\emptyset = \{ x | x \ne x \}$$

### 定理 1.1

空集是一切集合的子集.

**推论:** 空集是唯一的.

### 定义 1.5

设 $A$ 为集合, 把 $A$ 的全体子集构成的集合叫做 $A$ 的 *幂集*, 记作 $P(A)$ (或
$\mathscr{P} A, 2^A$).

幂集的符号化表示为 $$P(A) = \{ x | x \subseteq A \}$$

**推论:** 若 $A$ 是 n 元集, 则 $P(A)$ 有 $2^n$ 个元素.

### 定义 1.6

在一个具体问题中, 如果所涉及的集合都是某个集合的子集, 则称这个集合为 *全集*, 记作
$E$.

### 定义 1.7

设 $A, B$ 为集合, $A$ 与 $B$ 的 *并集* $A \cup B$, *交集* $A \cap B$, $B$
对 $A$ 的 *相对补集* $A - B$ 分别定义如下:
$$A \cup B = \{ x | x \in A \lor x \in B \}$$
$$A \cap B = \{ x | x \in A \land x \in B \}$$
$$A - B = \{ x | x \in A \land x \notin B \}$$

### 定义 1.8

设 $A, B$ 为集合, $A$ 与 $B$ 的 *对称差集* $A \oplus B$ 定义为
$$A \oplus B = (A - B) \cup (B - A)$$

### 定义 1.9

$$\sim A = E - A = \{ x | x \in E \land x \notin A \}$$

### 定义 1.10

设 $A$ 为集合, $A$ 的元素的元素构成的集合称为 $A$ 的 *广义并*, 记为 $\cup A$,
符号化表示为 $$\cup A = \{ x | \exists z(z \in A \land x \in z) \}$$

空集的广义并是空集.

### 定义 1.11

设 $A$ 为非空集合, $A$ 的所有元素的公共元素构成的集合称为 $A$ *广义交*, 记为
$\cap A$.  符号化表示为
$$\cap A = \{ x | \forall z(z \in A \rightarrow x \in z) \}$$

### 定理 1.2 包含排斥原理

设 $S$ 为有穷集, $P_1, P_2, \dots, P_n$ 是 $n$ 个性质.  $S$ 中的任何元素
$x$ 或者具有性质 $P_i$, 或者不具有性质 $P_i$, 两种情况必居其一.  令 $A_i$ 表示
$S$ 中具有性质 $P_i$ 的元素构成的子集, 则 $S$ 中不具有性质
$P_1, P_2, \dots, P_n$ 的元素数为
$$|\overline{A_1} \cap \overline{A_2} \cap \dots \cap \overline{A_n}|$$
$$= |S| - \sum_{i = 1}^n |A_i| + \sum_{1 \le i \lt j \le n} |A_i \cap A_j|$$
$$- \sum_{1 \le i \lt j \lt k \le n} |A_i \cap A_j \cap A_k| + \dots + (-1)^n |A_1 \cap A_2 \cap \dots \cap A_n|$$

**推论:** $S$ 中至少具有一条性质的元素数为
$$|A_1 \cup A_2 \cup \dots \cup A_n|$$
$$= \sum_{i = 1}^n |A_i| - \sum_{1 \le i \lt j \le n} |A_i \cap A_j|$$
$$+ \sum_{1 \le i \lt j \lt k \le n} | A_i \cap A_j \cap A_k| - \dots + (-1)^n|A_1 \cap A_2 \cap \dots \cap A_n|$$

### 集合恒等式

1. 幂等律
$$A \cup A = A$$
$$A \cap A = A$$
2. 结合律
$$(A \cup B) \cup C = A \cup (B \cup C)$$
$$(A \cap B) \cap C = A \cap (B \cap C)$$
3. 交换律
$$A \cup B = B \cup A$$
$$A \cap B = B \cap A$$
4. 分配律
$$A \cup (B \cap C) = (A \cup B) \cap (A \cup C)$$
$$A \cap (B \cup C) = (A \cap B) \cup (A \cap C)$$
5. 同一律
$$A \cup \emptyset = A$$
$$A \cap E = A$$
6. 零律
$$A \cup E = E$$
$$A \cap \emptyset = \emptyset$$
7. 排中律
$$A \cup \sim A = E$$
8. 矛盾律
$$A \cap \sim A = \emptyset$$
9. 吸收律
$$A \cup (A \cap B) = A$$
$$A \cap (A \cup B) = A$$
10. 德摩根律
$$A - (B \cup C) = (A - B) \cap (A - C)$$
$$A - (B \cap C) = (A - B) \cup (A - C)$$
$$\sim (B \cup C) = \sim B \cap \sim C$$
$$\sim (B \cap C) = \sim B \cup \sim C$$
$$\sim \emptyset = E$$
$$\sim E = \emptyset$$
11. 双重否定律
$$\sim \sim A = A$$

### 其它运算性质的重要结果

$$A \cap B \subseteq A, A \cap B \subseteq B$$
$$A \subseteq A \cup B, B \subseteq A \cup B$$
$$A - B \subseteq A$$
$$A - B = A \cap \sim B$$
$$A \cup B = B \Leftrightarrow A \subseteq B \Leftrightarrow A \cap B = A \Leftrightarrow A - B = \emptyset$$
$$A \oplus B = B \oplus A$$
$$(A \oplus B) \oplus C = A \oplus (B \oplus C)$$
$$A \oplus \emptyset = A$$
$$A \oplus A = \emptyset$$
$$A \oplus B = A \oplus C \Rightarrow B = C$$

## 2. 二元关系

### 定义 2.1

由两个元素 $x$ 和 $y$ (允许 $x = y$) 按一定顺序排列成的二元组叫做一个 *有序对* 或
*序偶*, 记作 $<x, y>$, 其中 $x$ 是它的第一元素, $y$ 是它的第二元素.

### 定义 2.2

设 $A, B$ 为集合, 用 $A$ 中元素为第一元素, $B$ 中元素为第二元素构成有序对.
所有这样的有序对组成的集合叫做 $A$ 和 $B$ 的 *笛卡尔积*, 记作 $A \times B$.

笛卡尔积的符号化表示为 $$A \times B = \{ <x, y> | x \in A \land y \in B \}$$

### 定义 2.3

如果一个集合满足一下条件之一:

1. 集合非空, 且它的元素都是有序对.
2. 集合是空集.

则称该集合为一个 *二元关系*, 记作 $R$.  二元关系也可简称为 *关系*. 对于二元关系 $R$,
如果 $<x, y> \in R$, 可记作 $x\mathrel{R}y$, 如果 $<x, y> \notin R$, 则记作
$x\not\mathrel{R}y$.

### 定义 2.4

设 $A, B$ 为集合, $A \times B$ 的任何子集所定义的二元关系叫做 *从 A 到 B 的二元关系*,
特别当 $A = B$ 时叫做 $A$ 上的 *二元关系*.

### 定义 2.5

对任何集合 $A$, 定义
$$E_A = \{ <x, y> | x \in A \land y \in A \} = A \times A$$
$$I_A = \{ <x, x> | x \in A \}$$

### 定义 2.6

设 $R$ 是二元关系.

1. $R$ 中所有有序对的第一元素构成的集合称为 $R$ 的 *定义域*, 记作 $dom R$,
形式化表示为 $$dom R = \{ x | \exists y(<x, y> \in R) \}$$
2. $R$ 中所有有序对的第二元素构成的集合称为 $R$ 的 *值域*, 记作 $ran R$.
形式化表示为 $$ran R = \{ y | \exists x(<x, y> \in R) \}$$
3. $R$ 的定义域和值域的并集称为 $R$ 的 *域*, 记作 $fld R$.  形式化表示为
$$fld R = dom R \cup ran R$$

### 定义 2.7

设 $R$ 为二元关系, $R$ 的 *逆关系*, 简称 $R$ 的 *逆*, 记作 $R^-1$, 其中
$$R^-1 = \{ <x, y> | <y, x> \in R \}$$

### 定义 2.8

设 $F, G$ 为二元关系, $G$ 对 $F$ 的 *右复合* 记作 $F \circ G$, 其中
$$F \circ G = \{ <x, y> | \exists t(<x, t> \in F \land <t, y> \in G) \}$$
*左复合* $F \circ G$ 即
$$F \circ G = \{ <x, y> | \exists t(<x, t> \in G \land <t, y> \in F) \}$$

### 定义 2.9

设 $R$ 为二元关系, $A$ 是集合.

1. $R$ 在 $A$ 上的 *限制* 记作 $R \upharpoonright A$, 其中
$$R \upharpoonright A = \{ <x, y> | x\mathrel{R}y \land x \in A \}$$
2. $A$ 在 $R$ 下的 *像* 记作 $R[A]$, 其中
$$R[A] = ran(R \upharpoonright A)$$

**推论:** $R$ 在 $A$ 上的限制 $R \upharpoonright A$ 是 $R$ 的子关系.  $A$
在 $R$ 下的像 $R[A]$ 是 $ran R$ 的子集.

### 定理 2.1

设 $F$ 是任意的关系, 则:

1. $(F^{-1})^{-1} = F$.
2. $dom F^{-1} = ran F, ran F^{-1} = dom F$.

### 定理 2.2

设 $F, G, H$ 是任意的关系, 则:

1. $(F \circ G) \circ H = F \circ (G \circ H)$.
2. $(F \circ G)^{-1} = G^{-1} \circ F^{-1}$.

### 定理 2.3

设 $R$ 为 $A$ 上的关系, 则 $$R \circ I_A = I_A \circ R = R$$

### 定理 2.4

设 $F, G, H$ 为任意关系, 则:

1. $F \circ (G \cup H) = F \circ G \cup F \circ H$.
2. $(G \cup H) \circ F = G \circ F \cup H \circ F$.

### 定理 2.5

设 $F$ 为关系, $A, B$ 为集合, 则:

1. $F \upharpoonright (A \cup B) = F \upharpoonright A \cup F \upharpoonright B$.
2. $R[A \cup B] = F[A] \cup F[B]$.
3. $F \upharpoonright (A \cap B) = F \upharpoonright A \cap F \upharpoonright B$.
4. $F[A \cap B] \subseteq F[A] \cap F[B]$.

### 定义 2.10

设 $R$ 为 $A$ 上的关系, $n$ 为自然数, 则 $R$ 的 *n 次幂 $R^n$* 定义为:

1. $R^0 = \{ <x, x> | x \in A \} = I_A$.
2. $R^{n + 1} = R^n \circ R$.

### 定理 2.6

设 $A$ 为 n 元集, $R$ 是 $A$ 上的关系, 则存在自然数 $s$ 和 $t$, 使得 $R^s = R^t$.

### 定理 2.7

设 $R$ 为 $A$ 上的关系, $m, n \in \mathbb{N}$, 则:

1. $R^m \circ R^n = R^{m + n}$.
2. $(R^n)^n = R^{mn}$.

### 定理 2.8

设 $R$ 为 $A$ 上的关系, 若存在自然数 $s, t (s \lt t)$ 使得 $R^s = R^t$, 则:

1. 对任何 $k \in \mathbb{N}$ 有 $R^{s + k} = R^{t + k}$.
2. 对任何 $k, i \in \mathbb{N}$ 有 $R^{s + kp + i} = R^{s + i}$, 其中 $p = t - s$.
3. 令 $S = \{ R^0, R^1, \dots, R^{t - 1} \}$, 则对于任意的 $q \in \mathbb{N}$ 有
$R^q \in S$.

### 定义 2.11

设 $R$ 为 $A$ 上的关系.

1. 若 $\forall x(x \in A \rightarrow <x, x> \in R)$, 则称 $R$ 在 $A$ 上是 *自反* 的.
1. 若 $\forall x(x \in A \rightarrow <x, x> \notin R)$, 则称 $R$ 在 $A$ 上是
*反自反* 的.

### 定义 2.12

设 $R$ 为 $A$ 上的关系.

1. 若 $\forall x \forall y(x, y \in A \land <x, y> \in R \rightarrow <y, x> \in R)$,
则称 $R$ 为 $A$ 上 *对称* 的关系.
2. 若 $\forall x \forall y(x, y \in A \land <x, y> \in R \land <y, x> \in R \rightarrow x = y)$,
则称 $R$ 为 $A$ 上 *反对称* 的关系.

### 定义 2.13

设 $R$ 为 $A$ 上的关系, 若
$$\forall x \forall y \forall z(x, y, z \in A \land <x, y> \in R \land <y, z> \in R \rightarrow <x, z> \in R)$$
则称 $R$ 为 $A$ 上 *传递* 的关系.

### 定理 2.9

设 $R$ 为 $A$ 上的关系, 则:

1. $R$ 在 $A$ 上自反当且仅当 $I_A \subseteq R$.
2. $R$ 在 $A$ 上反自反当且仅当 $R \cap I_A = \emptyset$.
3. $R$ 在 $A$ 上对称当且仅当 $R = R^{-1}$.
4. $R$ 在 $A$ 上反对称当且仅当 $R \cap R^{-1} \subseteq I_A$.
5. $R$ 在 $A$ 上传递当且仅当 $R \circ R \subseteq R$.

### 五种性质的特点

||自反性|反自反性|对称性|反对称性|传递性|
|:-:|:-:|:-:|:-:|:-:|:-:|
|集合表达式|$I_A \in R$|$R \cap I_A = \emptyset$|$R = R^{-1}$|$R \cap R^{-1} \subseteq I_A$|$R \circ R \subseteq R$|
|关系矩阵|主对角线元素全是 1|主对角线元素全是 0|矩阵是对称矩阵|若 $r_{ij} = 1$ 且 $i \ne j$, 则 $r_{ji} = 0$|对 $M^2$ 中 1 所在的位置, $M$ 中相应的位置都是 1|
|关系图|每个顶点都有环|每个顶点都没有环|如果两个顶点之间有边, 一定是一对方向相反的边 (无单边)|如果两个顶点之间有边, 一定是一条有向边 (无双向边)|如果顶点 $x_i$ 到 $x_j$ 有边, $x_j$ 到 $x_k$ 有边, 则从 $x_i$ 到 $x_k$ 也有边|

### 原有性质与运算的关系

||自反性|反自反性|对称性|反对称性|传递性|
|:-:|:-:|:-:|:-:|:-:|:-:|
|$R_1^{-1}$|能保持|能保持|能保持|能保持|能保持|
|$R_1 \cap R_2$|能保持|能保持|能保持|能保持|能保持|
|$R_1 \cup R_2$|能保持|能保持|能保持|不一定能保持|不一定能保持|
|$R_1 - R_2$|不一定能保持|能保持|能保持|能保持|不一定能保持|
|$R_1 \circ R_2$|能保持|不一定能保持|不一定能保持|不一定能保持|不一定能保持|

### 定义 2.14

设 $R$ 是非空集合 $A$ 上的关系, $R$ 的自反 (对称或传递) 闭包是 $A$ 上的关系 $R\prime$,
使得 $R\prime$ 满足以下条件:

1. $R\prime$ 是自反的 (对称或传递的).
2. $R \subseteq R\prime$.
3. 对 $A$ 上任何包含 $R$ 的自反 (对称或传递) 关系 $R\prime\prime$ 有
$R\prime \subseteq R\prime\prime$.

一般将 $R$ 的自反闭包记作 $r(R)$, 对称闭包记作 $s(R)$, 传递闭包记作 $t(R)$.

### 定理 2.10

设 $R$ 为 $A$ 上的关系, 则有:

1. $r(R) = R \cup R^0$.
2. $s(R) = R \cup R^{-1}$.
3. $t(R) = R \cup R^2 \cup R^3 \cup \dots$.

**推论:** 设 $R$ 为有穷集 $A$ 上的关系, 则存在正整数 $r$ 使得
$$t(R) = R \cup R^2 \cup R^3 \cup \dots \cup R^r$$

### 定理 2.11

设 $R$ 是非空集合 $A$ 上的关系, 则:

1. $R$ 是自反的当且仅当 $r(R) = R$.
2. $R$ 是对称的当且仅当 $s(R) = R$.
3. $R$ 是传递的当且仅当 $t(R) = R$.

### 定理 2.12

设 $R_1$ 和 $R_2$ 是非空集合 $A$ 上的关系, 且 $R_1 \subseteq R_2$, 则:

1. $r(R_1) \subseteq r(R_2)$.
2. $s(R_1) \subseteq s(R_2)$.
3. $t(R_1) \subseteq t(R_2)$.

### 定理 2.13

设 $R$ 是非空集合 $A$ 上的关系, 则:

1. 若 $R$ 是自反的, 则 $s(R)$ 与 $t(R)$ 也是自反的.
2. 若 $R$ 是对称的, 则 $r(R)$ 与 $t(R)$ 也是对称的.
3. 若 $R$ 是传递的, 则 $r(R)$ 是传递的.

### 定义 2.15

设 $R$ 为非空集合 $A$ 上的关系, 如果 $R$ 是自反的, 对称的和传递的, 则称 $R$
为 $A$ 上的 *等价关系*.  设 $R$ 是一个等价关系, 若 $<x, y> \in R$, 则称
*x 等价于 y*, 记作 $x \sim y$.

### 定义 2.16

设 $R$ 为非空集合 $A$ 上的等价关系, $\forall x \in A$, 令
$$[x]_R = \{ y | y \in A \land x \mathrel{R} y \}$$
称 $[x]_R$ 为 *x 关于 R 的等价类*, 简称为 $x$ 的 *等价类*, 简记为 $[x]$ 或
$\overline{x}$.

### 定理 2.14

设 $R$ 为非空集合 $A$ 上的等价关系, 则:

1. $\forall x \in A$, $[x]$ 是 $A$ 的非空子集.
2. $\forall x, y \in A$, 如果 $x \mathrel{R} y$, 则 $[x] = [y]$.
3. $\forall x, y \in A$, 如果 $x \not\mathrel{R} y$, 则 $[x]$ 与 $[y]$
不交.
4. $\cup \{ [x] | x \in A \} = A$.

### 定义 2.17

设 $R$ 为非空集合 $A$ 上的等价关系, 以 $R$ 的所有等价类作为元素的集合称为 $A$
关系于 $R$ 的 *商集*, 记作 $A / R$, 即 $$A / R = \{ [x]_R | x \in A \}$$

### 定义 2.18

设 $A$ 为非空集合, 若 $A$ 的子集族 $\pi$ ($\pi \subseteq P(A)$, 是 $A$
的子集构成的集合) 满足下面的条件:

1. $\emptyset \notin \pi$.
2. $\forall x \forall y(x, y \in \pi \land x \ne y \rightarrow x \cap y = \emptyset)$.
3. $\cup \pi = A$.

则称 $\pi$ 是 $A$ 的一个 *划分*, 称 $\pi$ 中的元素为 $A$ 的 *划分块*.

### 定义 2.19

设 $R$ 为非空集合 $A$ 上的关系.  如果 $R$ 是自反的, 反对称的和传递的, 则称 $R$ 为
$A$ 上的 *偏序关系*, 记作 $\preccurlyeq$.  设 $\preccurlyeq$ 为偏序关系, 如果
$<x, y> \in \preccurlyeq$, 则记作 $x \preccurlyeq y$, 读作 x "小于或等于" y.

### 定义 2.20

设 $preccurlyeq$ 为非空集合 $A$ 上的偏序关系, 定义:

1. $\forall x, y \in A, x \prec y \Leftrightarrow x \preccurlyeq y \land x \ne y$.
2. $\forall x, y \in A, x 与 y 可比 \Leftrightarrow x \preccurlyeq y \lor y \preccurlyeq x$.

### 定义 2.21

设 $R$ 为非空集合 $A$ 上的偏序关系, 如果 $\forall x, y \in A$, $x$ 与 $y$
都是可比的, 则称 $R$ 为 $A$ 上的 *全序关系* (或 *线序关系*).

### 定义 2.22

集合 $A$ 和 $A$ 上的偏序关系 $\preccurlyeq$ 一起叫做 *偏序集*,
记作 $<A, \preccurlyeq>$.

### 定义 2.23

设 $<A, \preccurlyeq>$ 为偏序集, $\forall x, y \in A$, 如果 $x \prec y$
且不存在 $z \in A$ 使得 $x \prec z \prec y$, 则称 *y 覆盖 x*.

### 定义 2.24

设 $<A, \preccurlyeq>$ 为偏序集, $B \subseteq A, y \in B$.

1. 若 $\forall x(x \in B \rightarrow y \preccurlyeq x)$ 成立, 则称 $y$ 为
$B$ 的 *最小元*.
2. 若 $\forall x(x \in B \rightarrow x \preccurlyeq y)$ 成立, 则称 $y$ 为
$B$ 的 *最大元*.
3. 若 $\forall x(x \in B \land x \preccurlyeq y \rightarrow x = y)$ 成立,
则称 $y$ 为 $B$ 的 *极小元*.
4. 若 $\forall x(x \in B \land y \preccurlyeq x \rightarrow x = y)$ 成立,
则称 $y$ 为 $B$ 的 *极大元*.

### 定义 2.25

设 $<A, \preccurlyeq>$ 为偏序集, $B \subseteq A, y \in A$.

1. 若 $\forall x(x \in B \rightarrow x \preccurlyeq y)$ 成立, 则称 $y$ 为
$B$ 的 *上界*.
2. 若 $\forall x(x \in B \rightarrow y \preccurlyeq x)$ 成立, 则称 $y$ 为
$B$ 的 *下界*.
3. 令 $C = \{ y | y 为 B 的上界 \}$, 则称 $C$ 的最小元为 $B$ 的 *最小上界* 或
*上确界*.
4. 令 $D = \{ y | y 为 B 的下界 \}$, 则称 $D$ 的最大元为 $B$ 的 *最大下界* 或
*下确界*.

## 3. 函数

### 定义 3.1

设 $F$ 为二元关系, 若 $\forall x \in dom F$ 都存在唯一的 $y \in ran F$
使 $x \mathrel{F} y$ 成立, 则称 $F$ 为 *函数* (也可称作 *映射*).  对于函数 $F$,
如果有 $x \mathrel{F} y$, 则记作 $y = F(x)$, 并称 $y$ 为 $F$ 在 $x$ 的值.

### 定义 3.2

设 $F, G$ 为函数, 则
$$F = G \Leftrightarrow F \subseteq G \land G \subseteq F$$

### 定义 3.3

设 $A, B$ 为集合, 如果 $f$ 为函数, 且 $dom f = A, ran f \subseteq B$, 则 $f$
称为 *从 A 到 B 的函数*, 记作 $f: A \rightarrow B$.

### 定义 3.4

所有从 $A$ 到 $B$ 的函数的集合记作 $B^A$, 读作 "B 上 A", 符号化表示为
$$B^A = \{ f | f: A \rightarrow B \}$$

### 定义 3.5

设函数 $f: A \rightarrow B, A_1 \subseteq A, B_1 \subseteq B$.

1. 令 $f(A_1) = \{ f(x) | x \in A_1 \}$, 称 $f(A_1)$ 为 $A_1$ 在 $f$ 下的
*像*.  特别地, 当 $A_1 = A$ 时称 $f(A)$ 为 *函数的像*.
2. 令 $f^{-1}(B_1) = \{ x | x \in A \land f(x) \in B_1 \}$, 称 $f^{-1}(B_1)$
为 $B_1$ 在 $f$ 下的 *完全原像*.

### 定义 3.6

设 $f: A \rightarrow B$.

1. 若 $ran f = B$, 则称 $f: A \rightarrow B$ 是 *满射* 的.
2. 若 $\forall y \in ran f$ 都存在唯一的 $x \in A$ 使得 $f(x) = y$, 则称
$f: A \rightarrow B$ 是 *单射* 的.
3. 若 $f: A \rightarrow B$ 既是满射又是单射的, 则称 $f: A \rightarrow B$ 是
*双射* 的 (或一一映像).

### 定义 3.7

1. 设 $f: A \rightarrow B$, 如果存在 $c \in B$ 使得对所有的 $x \in A$ 都有
$f(x) = c$, 则称 $f: A \rightarrow B$ 是 *常函数*.
2. 称 $A$ 上的恒等关系 $I_A$ 为 $A$ 上的 *恒等函数*.  对所有的 $x \in A$ 都有
$I_A(x) = x$.
3. 设 $<A, \preccurlyeq>$, $<B, \preccurlyeq>$ 为偏序集,
$f: A \rightarrow B$, 如果对任意的 $x_1, x_2 \in A, x_1 \prec x_2$, 就有
$f(x_1) \preccurlyeq f(x_2)$, 则称 $f$ 为 *单调递增* 的, 如果对任意的
$x_1, x_2 \in A, x_1 \prec x_2$, 就有 $f(x_1) \prec f(x_2)$, 则称 $f$ 为
*严格单调递增* 的.  类似地也可以定义 *单调递减* 的和 *严格单调递减* 的函数.
4. 设 $A$ 为集合, 对于任意的 $A\prime \subseteq A$, $A\prime$ 的 *特征函数*
$\chi_{A\prime}: A \rightarrow \{0, 1\}$ 定义为:
\begin{equation}
    \chi_{A\prime}(a) = \begin{cases}
        1, a \in A\prime \\
        0, a \in A - A\prime
    \end{cases}
\end{equation}
5. 设 $R$ 是 $A$ 上的等价关系, 令
$$g: A \rightarrow A / R$$
$$g(a) = [a], \forall a \in A$$
称 $g$ 是从 $A$ 到商集 $A / R$ 的 *自然映射*.

### 定理 3.1

设 $F, G$ 是函数, 则 $F \circ G$ 也是函数, 且满足:

1. $dom (F \circ G) = \{ x | x \in dom F \land F(x) \in dom G \}$.
2. $\forall x \in dom (F \circ G)$ 有 $F \circ G(x) = G(F(x))$.

**推论1:** 设 $F, G, H$ 为函数, 则 $(F \circ G) \circ H$ 和
$F \circ (G \circ H)$ 都是函数, 且
$$(F \circ G) \circ H = F \circ (G \circ H)$$

**推论2:** 设 $f: A \rightarrow B, g: B \rightarrow C$, 则
$f \circ g: A \rightarrow C$, 且 $\forall x \in A$ 都有
$f \circ g(x) = g(f(x))$.

### 定理 3.2

设 $f: A \rightarrow B, g: B \rightarrow C$.

1. 如果 $f: A \rightarrow B, g: B \rightarrow C$ 都是满射的, 则
$f \circ g: A \rightarrow C$ 也是满射的.
2. 如果 $f: A \rightarrow B, g: B \rightarrow C$ 都是单射的, 则
$f \circ g: A \rightarrow C$ 也是单射的.
3. 如果 $f: A \rightarrow B, g: B \rightarrow C$ 都是双射的, 则
$f \circ g: A \rightarrow C$ 也是双射的.

### 定理 3.3

设 $f: A \rightarrow B$, 则有 $$f = f \circ I_B = I_A \circ f$$

### 定理 3.4

设 $f: A \rightarrow B$ 是双射的, 则 $f^{-1}: B \rightarrow A$
也是双射的.

### 定理 3.5

设 $f: A \rightarrow B$ 是双射的, 则
$$f^{-1} \circ f = I_B, f \circ f^{-1} = I_A$$

### 定义 3.8

设 $A, B$ 是集合, 如果存在着从 $A$ 到 $B$ 的双射函数, 就称 $A$ 和 $B$
是 *等势* 的, 记作 $A \approx B$, 如果 $A$ 不与 $B$ 等势, 则记作
$A \not\approx B$.

### 定理 3.6

设 $A, B, C$ 是任意集合.

1. $A \approx A$.
2. 若 $A \approx B$, 则 $B \approx A$.
3. 若 $A \approx B, B \approx C$, 则 $A \approx C$.

### 定理 3.7 康托定理

1. $\mathbb{N} \not\approx \mathbb{R}$.
2. 对任意集合 $A$ 都有 $A \not\approx P(A)$.

### 定义 3.9

1. 设 $A, B$ 是集合, 如果存在从 $A$ 到 $B$ 的单射函数, 就称 $B$ *优势于*
$A$, 记作 $A \preccurlyeq\bullet B$.  如果 $B$ 不是优势于 $A$, 则记作
$A \not\preccurlyeq\bullet B$.
2. 设 $A, B$ 是集合, 若 $A \preccurlyeq\bullet B$ 且 $A \not\approx B$,
则称 *B 真优势于 A*, 记作 $A \prec\bullet B$.  如果 $B$ 不是真优势于 $A$,
则记作 $A \nprec\bullet B$.

### 定理 3.8

设 $A, B, C$ 是任意的集合, 则:

1. $A \preccurlyeq\bullet A$.
2. 若 $A \preccurlyeq\bullet B$ 且 $B \preccurlyeq\bullet A$, 则
$A \approx B$.
3. 若 $A \preccurlyeq\bullet B$ 且 $B \preccurlyeq\bullet C$, 则
$A \preccurlyeq\bullet C$.

### 定义 3.10

用空集和 *后继* $n^+$ (紧跟在 $n$ 后面的自然数) 可以把所有的自然数定义为集合, 即
$$0 = \emptyset$$
$$n^+ = n \cup \{ n \}, \forall n \in \mathbb{N}$$

**推论 (自然数的 *三歧性*):** 对任何自然数 $n$ 和 $m$, 以下三个式子:
$$m \in n, m \approx n, n \in m$$
必成立其一且仅成立其一.

### 定义 3.11

一个集合是 *有穷* 的当且仅当它与某个自然数等势.  如果一个集合不是有穷的, 就称作
*无穷集*.

### 定义 3.12

1. 对于有穷集合 $A$, 称与 $A$ 等势的那个唯一的自然数为 $A$ 的 *基数*, 记作 $card A$
(或 $|A|$), 即 $$card A = n \Leftrightarrow A \approx n$$
2. 自然数集合 $\mathbb{N}$ 的基数记作 $\aleph_0$, 即 $$card \mathbb{N} = \aleph_0$$
3. 实数集 $\mathbb{R}$ 的基数记作 $\aleph$, 即 $$card \mathbb{R} = \aleph$$

### 定义 3.13

设 $A, B$ 为集合, 则:

1. $card A = card B \Leftrightarrow A \approx B$.
2. $card A \le card B \Leftrightarrow A \preccurlyeq\bullet B$.
3. $card A \lt card B \Leftrightarrow card A \le card B \land card A \ne card B$.

### 定义 3.14

设 $A$ 为集合, 若 $card A \le \aleph_0$, 则称 $A$ 为 *可数集* 或 *可列集*.


[Table of contents](../toc.ipynb)

# Ordinary differential equations

We will compare different discretization schemes to solve ordinary differential equations (ODE) in this mini project. The goal is that you have gathered practical experience in
 * explicit Euler,
 * and Runge-Kutta solver for ODEs.

We will solve and compare the solution of the basic ODE

$y'(t) = y(t)$

with exact solution, and numerical ODE solvers like it was done in section 1.2.6 [[Burg2013]](./references.bib) for Matlab.

```python
%matplotlib inline
from matplotlib import pyplot as plt

import sympy as sp
import numpy as np
```

## Exact solution

The exact solution is likely very simple to solve for you and you do not need Python for it, but let us compute with SymPy briefly.

First, we need to define a symbol and a function, which should result in the given function $y'(t) = y(t)$.

```python
y = sp.Function('y')
t = sp.symbols('t')

dgl = sp.Eq(y(t).diff(t), y(t))
dgl
```

$\displaystyle \frac{d}{d t} y{\left(t \right)} = y{\left(t \right)}$

We can compute now with SymPy the exact solution with and without initial condition.

```python
sp.dsolve(dgl, y(t))
```

$\displaystyle y{\left(t \right)} = C_{1} e^{t}$

The solution with initial condition becomes:

```python
sp.dsolve(dgl, y(t), ics={y(0): 3})
```

$\displaystyle y{\left(t \right)} = 3 e^{t}$

It is obvious that the integration constant $C_1$ from the solution without initial condition has always the value
$C_1 = y(0)$.

Therefore, we can define a function to compute the exact solution on different initial conditions.

```python
def exact_solution(time, init):
    return init * np.exp(time)
```

And a loop computes the solutions for each initial condition.

```python
time = np.arange(0, 3, 0.1)
init = np.arange(0, 4, 0.5)

dgl_exact = np.zeros((init.size, time.size))

for idx, value in enumerate(init):
    dgl_exact[idx,:] = exact_solution(time, value)
```

The next plot shows the solutions.

```python
plt.figure()
for row in np.ndindex(dgl_exact.shape[0]):
    plt.plot(time, dgl_exact[row], 'k-')
plt.ylim([0, 10])
plt.xlabel("time")
plt.ylabel("$y$")
plt.legend(["Exact solutions"])

plt.show()
```

## Numerical solution

### Runge-Kutta

Now we want to apply SciPy to compute the numerical solution with Runge-Kutta numerical solver. SciPy uses the `RK45`, which is an explicit Runge-Kutta of order four, where the step size is approximated from order five method. The classical Runge-Kutta of order four follows

$\begin{align}
r_1 &= f(t_i, y_i)\\
r_2 &= f(t_i + \frac{1}{2} \Delta t, y_i + \frac{1}{2} \Delta t r_1)\\
r_3 &= f(t_i + \frac{1}{2} \Delta t, y_i + \frac{1}{2} \Delta t r_2)\\
r_4 &= f(t_i + \Delta t, y_i + \Delta t r_3)\\
y_{i+1} &= y_i + \frac{1}{6} \Delta t (r_1 + 2 r_2 + 2 r_3 + r_4),
\end{align}$

see equation (1.105) in [[Burg2013]](./references.bib).

First, we need to define the function which should be solved.

```python
def equation(t, y):
    """Return right hand side of d/dt y(t) = y(t)"""
    return [1 * y[0]]
```

Now, we can use `scipy.intergrate` to solve the solution with Runge-Kutta solver.

```python
from scipy.integrate import solve_ivp

time_span = [0, 3]
init = [1]
dgl_RK45 = solve_ivp(equation, time_span, init, method='RK45', t_eval=time)
```

And now we plot this solution for initial value $y(0)=1$.

```python
plt.figure()
plt.plot(time, dgl_RK45.y[0, :])
plt.legend(["$y$ Runge-Kutta"])
plt.xlabel("Time")
plt.ylabel("$y$")
plt.ylim([0, 10])
plt.show()
```

### Explicit Euler

The explicit Euler method (forward Euler) is the simplest one step numerical solver we can implement by ourself.

It is based on the approximation

$y'(t_i) \approx \dfrac{y(t_{i+1}) - y(t_i)}{\Delta t}$,

$y(t_{i+1}) - y(t_i) \approx \Delta t f(t_i, y(t_i))$,

and the solution becomes,

$y_{i+1} = y_i + \Delta t f(t_i, y_i)$,

where $\Delta t$ is the step size and $f(t_i, y_i)$ is right hand side of $y'(t) = f(t, y(t))$, see section 1.2.6 [[Burg2013]](./references.bib).

```python
def solve_eeuler(func, time_vec, delta_t, inc):
    """Forwar Euler for DGL of type d/dt y = f(t, y(t))"""
    y = np.zeros(time_vec.size)
    y[0] = inc

    for idx, t in enumerate(time_vec[1:]):
        f = func(t, np.array([y[idx]]))[0]
        y[idx + 1] = y[idx] + delta_t * f
    return y
```

```python
dgl_eeuler = solve_eeuler(func=equation, time_vec=time, delta_t=0.1, inc=1.0)
```

```python
time2 = np.arange(0, 3, 0.3)
dgl_eeuler2 = solve_eeuler(func=equation, time_vec=time2, delta_t=0.3, inc=1.0)
```

### Implicit Euler

The implicit Euler or backward Euler solver looks very similar to implicit Euler. However, here the computation $f(t_{i+1}, y_{i+1})$ is required which makes this method computational more expensive but ensure stability. The implicit Euler is

$y_{t+1} = y_i + \Delta t f(t_{i+1}, y_{i+1})$.

However, we will skip the implementation of implicit Euler herein.

### Results

The next plot shows all solutions together. The large step size in `dgl_eeuler2` causes large discretization error. Add to this, Euler method can also cause unstable solutions if the step size is to large for the given problem.

```python
plt.figure()
plt.plot(time, dgl_exact[2], 'k-')
plt.plot(time, dgl_RK45.y[0, :], '--')
plt.plot(time, dgl_eeuler, '--')
plt.plot(time2, dgl_eeuler2, '--')
plt.legend(["Exact solution", "$y$ Runge-Kutta", "$y$ Forward Euler", "$y$ Forward Euler2"])
plt.xlabel("Time")
plt.ylabel("$y$")
plt.ylim([0, 10])
plt.show()
```

Finally, let us compute the error of Runge-Kutta and forward Euler solver compared with exact solution.

```python
plt.figure()
plt.plot(time, dgl_exact[2] - dgl_RK45.y[0, :], '--')
plt.legend(["Runge-Kutta"])
plt.xlabel("Time")
plt.ylabel("Discretization error")
plt.show()
```

```python
plt.figure()
plt.plot(time, dgl_exact[2] - dgl_eeuler, '--')
plt.legend(["Forward Euler"])
plt.xlabel("Time")
plt.ylabel("Discretization error")
plt.show()
```

It is clearly visible that Runge-Kutta provides much better accuracy than explicit Euler. For more ODE solvers and their accuracy consult pages 75-80 in [[Burg2013]](./references.bib).

## Exercise:  ODE mini project

Please find the exact and Runge-Kutta solution for the mechanical oscillator system

$m\ddot{x}(t) + r \dot{x}(t) + k x(t) = K(t)$,

where $m$ denotes mass, $r$ damping constant, $k$ spring constant, and $K(t)$ is a time varying external force. We are looking for the displacement $x(t)$.

You can assume that $K(t) = K_0 \cos \omega t$.


# Task for lecture 20

2. Regarding the distribution problem for the elbow muscles presented in this text:
    a. Test different initial values for the optimization.
    b. Test other values for the elbow angle where the results are likely to change.

3. In an experiment to estimate forces of the elbow flexors, through inverse dynamics it was found an elbow flexor moment of 10 Nm.
Consider the following data for maximum force (F0), moment arm (r), and pcsa (A) of the brachialis, brachioradialis, and biceps brachii muscles: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A (cm$^2$): 33, 8, 23, respectively (data from Robertson et al. (2013)).
    a. Use static optimization to estimate the muscle forces.
    b. Test the robustness of the results using different initial values for the muscle forces.
    c. Compare the results for different cost functions.

```python
# import Python libraries
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

### Muscle force estimation

Consider the following main flexors of the elbow joint (see figure below): biceps long head, biceps short head, and brachialis. Suppose that the elbow net joint torque determined using inverse dynamics is 20 Nm (flexor). How much each of these muscles contributed to the net torque?

Figure. A view in OpenSim of the arm26 model showing three elbow flexors (Biceps long and short heads and Brachialis).

For the optimization, we will need experimental data for the moment arm, maximum moment, and *pcsa* of each muscle. Let's import these data from the OpenSim arm26 model:

```python
# time elbow_flexion BIClong BICshort BRA
r_ef = np.loadtxt('./../data/r_elbowflexors.mot', skiprows=7)
f_ef = np.loadtxt('./../data/f_elbowflexors.mot', skiprows=7)
```

The maximum isometric force of these muscles are defined in the arm26 model as: Biceps long head: 624.3 N, Biceps short head: 435.56 N, and Brachialis: 987.26 N. Let's compute the mamimum torques that each muscle could produce considering a static situation at the different elbow flexion angles:

```python
m_ef = r_ef*1
m_ef[:, 2:] = r_ef[:, 2:]*f_ef[:, 2:]
```

These data don't have the *pcsa* value of each muscle. We will estimate the *pcsa* considering that the amount of maximum muscle force generated per area is constant and equal to 50N/cm$^2$. Consequently, the *pcsa* (in cm$^2$) for each muscle is:

```python
a_ef = np.array([624.3, 435.56, 987.26])/50  # 50 N/cm2
print(a_ef)
```

    [12.486   8.7112 19.7452]

### Solution of the optimization problem

For the present case, we are dealing with a problem of minimization, multidimensional (function of several variables), nonlinear, constrained, and we can't assume that the cost function is convex. Numerical optimization is hardly a simple task. There are many different algorithms and public and commercial software for performing optimization. For instance, look at [NEOS Server](http://www.neos-server.org/neos/), a free internet-based service for solving numerical optimization problems.
We will solve the present problem using the [scipy.optimize](http://docs.scipy.org/doc/scipy/reference/optimize.html#module-scipy.optimize) package which provides several optimization algorithms. We will use the function `minimize`:

```python
scipy.optimize.minimize(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None)
"""Minimization of scalar function of one or more variables."""
```

Now, let's write Python functions for each cost function:

```python
from scipy.optimize import minimize
```

```python
def cf_f1(x):
    """Cost function: sum of forces."""
    return x[0] + x[1] + x[2]

def cf_f2(x):
    """Cost function: sum of forces squared."""
    return x[0]**2 + x[1]**2 + x[2]**2

def cf_fpcsa2(x, a):
    """Cost function: sum of squared muscle stresses."""
    return (x[0]/a[0])**2 + (x[1]/a[1])**2 + (x[2]/a[2])**2

def cf_fmmax3(x, m):
    """Cost function: sum of cubic forces normalized by moments."""
    return (x[0]/m[0])**3 + (x[1]/m[1])**3 + (x[2]/m[2])**3
```

Let's also define the Jacobian for each cost function (which is an optional parameter for the optimization):

```python
def cf_f1d(x):
    """Derivative of cost function: sum of forces."""
    dfdx0 = 1
    dfdx1 = 1
    dfdx2 = 1
    return np.array([dfdx0, dfdx1, dfdx2])

def cf_f2d(x):
    """Derivative of cost function: sum of forces squared."""
    dfdx0 = 2*x[0]
    dfdx1 = 2*x[1]
    dfdx2 = 2*x[2]
    return np.array([dfdx0, dfdx1, dfdx2])

def cf_fpcsa2d(x, a):
    """Derivative of cost function: sum of squared muscle stresses."""
    dfdx0 = 2*x[0]/a[0]**2
    dfdx1 = 2*x[1]/a[1]**2
    dfdx2 = 2*x[2]/a[2]**2
    return np.array([dfdx0, dfdx1, dfdx2])

def cf_fmmax3d(x, m):
    """Derivative of cost function: sum of cubic forces normalized by moments."""
    dfdx0 = 3*x[0]**2/m[0]**3
    dfdx1 = 3*x[1]**2/m[1]**3
    dfdx2 = 3*x[2]**2/m[2]**3
    return np.array([dfdx0, dfdx1, dfdx2])
```

Perform the optimization using the previous initial condition looked in class

2. Regarding the distribution problem for the elbow muscles presented in this text:
    a. Test different initial values for the optimization.
    b. Test other values for the elbow angle where the results are likely to change.

```python
M = 20  # desired torque at the elbow
iang = 69  # which will give the closest value to 90 degrees
#iang = 34
r  = r_ef[iang, 2:]
f0 = f_ef[iang, 2:]
a  = a_ef
m  = m_ef[iang, 2:]
x0 = f_ef[iang, 2:]/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 20
    x0 = [57.51311369 36.29974032 89.6470056 ]
    r * x0 = 6.622004446068322

Inequality constraints (such as boundaries in our problem) can be entered with the parameter `bounds` to the `minimize` function:

```python
bnds = ((0, f0[0]), (0, f0[1]), (0, f0[2]))
```

Equality constraints (such as the sum of torques should equals the desired torque in our problem), as well as inequality constraints, can be entered with the parameter `constraints` to the `minimize` function (and we can also opt to enter the Jacobian of these constraints):

```python
# use this in combination with the parameter bounds:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)})
```

```python
# to enter everything as constraints:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[0]-x[0],
         'jac' : lambda x, r, f0, M: np.array([-1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[1]-x[1],
         'jac' : lambda x, r, f0, M: np.array([0, -1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[2]-x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, -1]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[0],
         'jac' : lambda x, r, f0, M: np.array([1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[1],
         'jac' : lambda x, r, f0, M: np.array([0, 1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, 1]), 'args': (r, f0, M)})
```

Although more verbose, if all the Jacobians of the constraints are also informed, this alternative seems better than informing bounds for the optimization process (less error in the final result and less iterations).

Given the characteristics of the problem, if we use the function `minimize` we are limited to the SLSQP (Sequential Least SQuares Programming) solver.

Finally, let's run the optimization for the four different cost functions and find the optimal muscle forces:

```python
f1r = minimize(fun=cf_f1, x0=x0, args=(), jac=cf_f1d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 409.59266009953336
                Iterations: 7
                Function evaluations: 7
                Gradient evaluations: 7

```python
f2r = minimize(fun=cf_f2, x0=x0, args=(), jac=cf_f2d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 75657.38479127169
                Iterations: 5
                Function evaluations: 6
                Gradient evaluations: 5

```python
fpcsa2r = minimize(fun=cf_fpcsa2, x0=x0, args=(a,), jac=cf_fpcsa2d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 529.9639777695737
                Iterations: 11
                Function evaluations: 11
                Gradient evaluations: 11

```python
fmmax3r = minimize(fun=cf_fmmax3, x0=x0, args=(m,), jac=cf_fmmax3d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 1075.1388931664733
                Iterations: 12
                Function evaluations: 12
                Gradient evaluations: 12

```python
# Storing the results of the first optimization using previous initial conditions
opt1 = np.around(np.vstack((f1r.x, f2r.x, fpcsa2r.x, fmmax3r.x)), 1)
```

```python
dat = np.vstack((np.around(r*100,1), np.around(a,1), np.around(f0,0), np.around(m,1)))
er = ['-', '-', '-', '-',
      np.sum(r*f1r.x)-M, np.sum(r*f2r.x)-M, np.sum(r*fpcsa2r.x)-M, np.sum(r*fmmax3r.x)-M]
data = np.vstack((np.vstack((dat, opt1)).T, er)).T

rows = ['$\text{Moment arm}\;[cm]$', '$pcsa\;[cm^2]$', '$F_{max}\;[N]$', '$M_{max}\;[Nm]$',
        '$\sum F_i$', '$\sum F_i^2$', '$\sum(F_i/pcsa_i)^2$', '$\sum(F_i/M_{max,i})^3$']
cols = ['Biceps long head', 'Biceps short head', 'Brachialis', 'Error in M']
df = pd.DataFrame(data, index=rows, columns=cols)
print('\Solving the distribution problem for a desired torque of 20Nm and at 90 elbow flexion')
df
```

    \Solving the distribution problem for a desired torque of 20Nm and at 90 elbow flexion

      Biceps long head
      Biceps short head
      Brachialis
      Error in M

      $\text{Moment arm}\;[cm]$
      4.9
      4.9
      2.3
      -

      $pcsa\;[cm^2]$
      12.5
      8.7
      19.7
      -

      $F_{max}\;[N]$
      575.0
      363.0
      896.0
      -

      $M_{max}\;[Nm]$
      28.1
      17.7
      20.4
      -

      $\sum F_i$
      215.4
      194.2
      0.0
      0.0

      $\sum F_i^2$
      184.7
      184.7
      86.1
      0.0

      $\sum(F_i/pcsa_i)^2$
      201.7
      98.2
      235.2
      0.0

      $\sum(F_i/M_{max,i})^3$
      241.1
      120.9
      102.0
      -3.552713678800501e-15

## 3. Regarding the distribution problem for the elbow muscles presented in this text:
    a. Test different initial values for the optimization.
    b. Test other values for the elbow angle where the results are likely to change.
Set different initial conditions such as consider a desired torque at the elbow of 40 Nm and a slightly higher initial sum of torques.

```python
M = 40  # desired torque at the elbow
iang = 69  # which will give the closest value to 90 degrees
r1  = r_ef[iang, 2:]
f01 = f_ef[iang, 2:]
a1  = a_ef
m1  = m_ef[iang, 2:]
x01 = f_ef[iang, 2:]/5  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 40
    x0 = [57.51311369 36.29974032 89.6470056 ]
    r * x0 = 6.622004446068322

Inequality constraints (such as boundaries in our problem) can be entered with the parameter `bounds` to the `minimize` function:

```python
bnds = ((0, f0[0]), (0, f0[1]), (0, f0[2]))
```

Equality constraints (such as the sum of torques should equals the desired torque in our problem), as well as inequality constraints, can be entered with the parameter `constraints` to the `minimize` function (and we can also opt to enter the Jacobian of these constraints):

```python
# use this in combination with the parameter bounds:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)})
```

```python
# to enter everything as constraints:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[0]-x[0],
         'jac' : lambda x, r, f0, M: np.array([-1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[1]-x[1],
         'jac' : lambda x, r, f0, M: np.array([0, -1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[2]-x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, -1]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[0],
         'jac' : lambda x, r, f0, M: np.array([1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[1],
         'jac' : lambda x, r, f0, M: np.array([0, 1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, 1]), 'args': (r, f0, M)})
```

Although more verbose, if all the Jacobians of the constraints are also informed, this alternative seems better than informing bounds for the optimization process (less error in the final result and less iterations).

Given the characteristics of the problem, if we use the function `minimize` we are limited to the SLSQP (Sequential Least SQuares Programming) solver.

Finally, let's run the optimization for the four different cost functions and find the optimal muscle forces:

```python
f1r = minimize(fun=cf_f1, x0=x0, args=(), jac=cf_f1d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 819.1853201990796
                Iterations: 10
                Function evaluations: 10
                Gradient evaluations: 10

```python
f2r = minimize(fun=cf_f2, x0=x0, args=(), jac=cf_f2d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 302704.8453888565
                Iterations: 5
                Function evaluations: 6
                Gradient evaluations: 5

```python
fpcsa2r = minimize(fun=cf_fpcsa2, x0=x0, args=(a,), jac=cf_fpcsa2d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 2119.855911078297
                Iterations: 11
                Function evaluations: 11
                Gradient evaluations: 11

```python
fmmax3r = minimize(fun=cf_fmmax3, x0=x0, args=(m,), jac=cf_fmmax3d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 8601.111145156132
                Iterations: 11
                Function evaluations: 11
                Gradient evaluations: 11

```python
opt2 = np.around(np.vstack((f1r.x, f2r.x, fpcsa2r.x, fmmax3r.x)), 1)
```

Displaying the results for the **second** optimization using different initial conditions basically changing the desired torque (twice as the previous) and initial sum of torques.

```python
dat = np.vstack((np.around(r*100,1), np.around(a,1), np.around(f0,0), np.around(m,1)))
er = ['-', '-', '-', '-',
      np.sum(r*f1r.x)-M, np.sum(r*f2r.x)-M, np.sum(r*fpcsa2r.x)-M, np.sum(r*fmmax3r.x)-M]
data = np.vstack((np.vstack((dat, opt2)).T, er)).T

rows = ['$\text{Moment arm}\;[cm]$', '$pcsa\;[cm^2]$', '$F_{max}\;[N]$', '$M_{max}\;[Nm]$',
        '$\sum F_i$', '$\sum F_i^2$', '$\sum(F_i/pcsa_i)^2$', '$\sum(F_i/M_{max,i})^3$']
cols = ['Biceps long head', 'Biceps short head', 'Brachialis', 'Error in M']
df = pd.DataFrame(data, index=rows, columns=cols)
print('\nSolving the distribution problem for a desired torque of 40Nm')
df
```

    Solving the distribution problem for a desired torque of 40Nm

      Biceps long head
      Biceps short head
      Brachialis
      Error in M

      $\text{Moment arm}\;[cm]$
      4.9
      4.9
      2.3
      -

      $pcsa\;[cm^2]$
      12.5
      8.7
      19.7
      -

      $F_{max}\;[N]$
      575.0
      363.0
      896.0
      -

      $M_{max}\;[Nm]$
      28.1
      17.7
      20.4
      -

      $\sum F_i$
      456.2
      363.0
      0.0
      -7.105427357601002e-15

      $\sum F_i^2$
      374.7
      363.0
      174.7
      0.0

      $\sum(F_i/pcsa_i)^2$
      403.4
      196.4
      470.5
      0.0

      $\sum(F_i/M_{max,i})^3$
      482.2
      241.8
      204.1
      -7.105427357601002e-15

## 3. Regarding the distribution problem for the elbow muscles presented in this text:
    b. Test other values for the elbow angle where the results are likely to change.

Coinsider and angle of 45 degrees.

```python
M = 20  # desired torque at the elbow
iang = 34  # which will give the closest value to 45 degrees
#iang = 34
r  = r_ef[iang, 2:]
f0 = f_ef[iang, 2:]
a  = a_ef
m  = m_ef[iang, 2:]
x0 = f_ef[iang, 2:]/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 20
    x0 = [62.38587639 43.36183466 98.2329904 ]
    r * x0 = 4.799197847242849

Inequality constraints (such as boundaries in our problem) can be entered with the parameter `bounds` to the `minimize` function:

```python
bnds = ((0, f0[0]), (0, f0[1]), (0, f0[2]))
```

Equality constraints (such as the sum of torques should equals the desired torque in our problem), as well as inequality constraints, can be entered with the parameter `constraints` to the `minimize` function (and we can also opt to enter the Jacobian of these constraints):

```python
# use this in combination with the parameter bounds:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)})
```

```python
# to enter everything as constraints:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[0]-x[0],
         'jac' : lambda x, r, f0, M: np.array([-1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[1]-x[1],
         'jac' : lambda x, r, f0, M: np.array([0, -1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[2]-x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, -1]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[0],
         'jac' : lambda x, r, f0, M: np.array([1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[1],
         'jac' : lambda x, r, f0, M: np.array([0, 1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, 1]), 'args': (r, f0, M)})
```

Although more verbose, if all the Jacobians of the constraints are also informed, this alternative seems better than informing bounds for the optimization process (less error in the final result and less iterations).

Given the characteristics of the problem, if we use the function `minimize` we are limited to the SLSQP (Sequential Least SQuares Programming) solver.

Finally, let's run the optimization for the four different cost functions and find the optimal muscle forces:

```python
f1r = minimize(fun=cf_f1, x0=x0, args=(), jac=cf_f1d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 590.5532154884442
                Iterations: 7
                Function evaluations: 7
                Gradient evaluations: 7

```python
f2r = minimize(fun=cf_f2, x0=x0, args=(), jac=cf_f2d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 163425.69129493315
                Iterations: 5
                Function evaluations: 6
                Gradient evaluations: 5

```python
fpcsa2r = minimize(fun=cf_fpcsa2, x0=x0, args=(a,), jac=cf_fpcsa2d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 1227.8544785495337
                Iterations: 11
                Function evaluations: 11
                Gradient evaluations: 11

```python
fmmax3r = minimize(fun=cf_fmmax3, x0=x0, args=(m,), jac=cf_fmmax3d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 7770.433925704175
                Iterations: 10
                Function evaluations: 10
                Gradient evaluations: 10

```python
opt3 = np.around(np.vstack((f1r.x, f2r.x, fpcsa2r.x, fmmax3r.x)), 1)
```

**Results of the optimization problem considering a desired torque of 20Nm and 45 degrees elbow flexion**

```python
dat = np.vstack((np.around(r*100,1), np.around(a,1), np.around(f0,0), np.around(m,1)))
er = ['-', '-', '-', '-',
      np.sum(r*f1r.x)-M, np.sum(r*f2r.x)-M, np.sum(r*fpcsa2r.x)-M, np.sum(r*fmmax3r.x)-M]
data = np.vstack((np.vstack((dat, opt3)).T, er)).T

rows = ['$\text{Moment arm}\;[cm]$', '$pcsa\;[cm^2]$', '$F_{max}\;[N]$', '$M_{max}\;[Nm]$',
        '$\sum F_i$', '$\sum F_i^2$', '$\sum(F_i/pcsa_i)^2$', '$\sum(F_i/M_{max,i})^3$']
cols = ['Biceps long head', 'Biceps short head', 'Brachialis', 'Error in M']
df = pd.DataFrame(data, index=rows, columns=cols)
print('\nSolving the distribution problem for a desired torque of 20Nm and at 45 elbow flexion')
df
```

    Solving the distribution problem for a desired torque of 20Nm and at 45 elbow flexion

      Biceps long head
      Biceps short head
      Brachialis
      Error in M

      $\text{Moment arm}\;[cm]$
      3.4
      3.4
      1.2
      -

      $pcsa\;[cm^2]$
      12.5
      8.7
      19.7
      -

      $F_{max}\;[N]$
      624.0
      434.0
      982.0
      -

      $M_{max}\;[Nm]$
      21.1
      14.7
      12.2
      -

      $\sum F_i$
      304.8
      285.8
      0.0
      0.0

      $\sum F_i^2$
      276.7
      276.7
      101.3
      0.0

      $\sum(F_i/pcsa_i)^2$
      324.1
      157.8
      296.7
      -3.552713678800501e-15

      $\sum(F_i/M_{max,i})^3$
      352.3
      204.1
      93.3
      0.0

## 4. In an experiment to estimate forces of the elbow flexors, through inverse dynamics it was found an elbow flexor moment of 10 Nm.
Consider the following data for maximum force (F0), moment arm (r), and pcsa (A) of the brachialis, brachioradialis, and biceps brachii muscles: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A (cm$^2$): 33, 8, 23, respectively (data from Robertson et al. (2013)).
    a. Use static optimization to estimate the muscle forces.

```python
M = 10  # desired torque at the elbow
iang = 34  # which will give the closest value to 45 degrees
r  = np.array([0.02, 0.05, 0.04])
f0 = np.array([1000, 250, 700])
a  = np.array([33, 8, 23])
m  = r*f0
x0 = f0/20  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 10
    x0 = [50.  12.5 35. ]
    r * x0 = 3.0250000000000004

Inequality constraints (such as boundaries in our problem) can be entered with the parameter `bounds` to the `minimize` function:

```python
bnds = ((0, f0[0]), (0, f0[1]), (0, f0[2]))
```

Equality constraints (such as the sum of torques should equals the desired torque in our problem), as well as inequality constraints, can be entered with the parameter `constraints` to the `minimize` function (and we can also opt to enter the Jacobian of these constraints):

```python
# use this in combination with the parameter bounds:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)})
```

```python
# to enter everything as constraints:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[0]-x[0],
         'jac' : lambda x, r, f0, M: np.array([-1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[1]-x[1],
         'jac' : lambda x, r, f0, M: np.array([0, -1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[2]-x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, -1]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[0],
         'jac' : lambda x, r, f0, M: np.array([1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[1],
         'jac' : lambda x, r, f0, M: np.array([0, 1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, 1]), 'args': (r, f0, M)})
```

Although more verbose, if all the Jacobians of the constraints are also informed, this alternative seems better than informing bounds for the optimization process (less error in the final result and less iterations).

Given the characteristics of the problem, if we use the function `minimize` we are limited to the SLSQP (Sequential Least SQuares Programming) solver.

Finally, let's run the optimization for the four different cost functions and find the optimal muscle forces:

```python
f1r = minimize(fun=cf_f1, x0=x0, args=(), jac=cf_f1d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 200.00000000000276
                Iterations: 12
                Function evaluations: 12
                Gradient evaluations: 12

```python
f2r = minimize(fun=cf_f2, x0=x0, args=(), jac=cf_f2d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 22222.22222222222
                Iterations: 5
                Function evaluations: 6
                Gradient evaluations: 5

```python
fpcsa2r = minimize(fun=cf_fpcsa2, x0=x0, args=(a,), jac=cf_fpcsa2d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 69.34812760055478
                Iterations: 11
                Function evaluations: 11
                Gradient evaluations: 11

```python
fmmax3r = minimize(fun=cf_fmmax3, x0=x0, args=(m,), jac=cf_fmmax3d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 267.80139010440996
                Iterations: 13
                Function evaluations: 13
                Gradient evaluations: 13

```python
opt4 = np.around(np.vstack((f1r.x, f2r.x, fpcsa2r.x, fmmax3r.x)), 1)
```

**Results of the optimization problem considering a desired torque of 20Nm and 45 degrees elbow flexion**

```python
dat = np.vstack((np.around(r*100,1), np.around(a,1), np.around(f0,0), np.around(m,1)))
er = ['-', '-', '-', '-',
      np.sum(r*f1r.x)-M, np.sum(r*f2r.x)-M, np.sum(r*fpcsa2r.x)-M, np.sum(r*fmmax3r.x)-M]
data = np.vstack((np.vstack((dat, opt4)).T, er)).T

rows = ['$\text{Moment arm}\;[cm]$', '$pcsa\;[cm^2]$', '$F_{max}\;[N]$', '$M_{max}\;[Nm]$',
        '$\sum F_i$', '$\sum F_i^2$', '$\sum(F_i/pcsa_i)^2$', '$\sum(F_i/M_{max,i})^3$']
cols = ['Biceps long head', 'Biceps short head', 'Brachialis', 'Error in M']
df = pd.DataFrame(data, index=rows, columns=cols)
print('\nSolving the distribution problem for a desired torque of 20Nm and at 45 elbow flexion')
df
```

    Solving the distribution problem for a desired torque of 20Nm and at 45 elbow flexion

      Biceps long head
      Biceps short head
      Brachialis
      Error in M

      $\text{Moment arm}\;[cm]$
      2.0
      5.0
      4.0
      -

      $pcsa\;[cm^2]$
      33.0
      8.0
      23.0
      -

      $F_{max}\;[N]$
      1000.0
      250.0
      700.0
      -

      $M_{max}\;[Nm]$
      20.0
      12.5
      28.0
      -

      $\sum F_i$
      0.0
      200.0
      0.0
      -1.7763568394002505e-15

      $\sum F_i^2$
      44.4
      111.1
      88.9
      0.0

      $\sum(F_i/pcsa_i)^2$
      151.0
      22.2
      146.7
      0.0

      $\sum(F_i/M_{max,i})^3$
      65.5
      51.1
      153.3
      0.0

  ### b. Test the robustness of the results using different initial values for the muscle forces.
  ### c. Compare the results for different cost functions.

```python
M = 20  # desired torque at the elbow
iang = 34  # which will give the closest value to 45 degrees
r  = np.array([0.02, 0.05, 0.04])
f0 = np.array([1000, 250, 700])
a  = np.array([33, 8, 23])
m  = r*f0
x0 = f0/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 20
    x0 = [100.  25.  70.]
    r * x0 = 6.050000000000001

Inequality constraints (such as boundaries in our problem) can be entered with the parameter `bounds` to the `minimize` function:

```python
bnds = ((0, f0[0]), (0, f0[1]), (0, f0[2]))
```

Equality constraints (such as the sum of torques should equals the desired torque in our problem), as well as inequality constraints, can be entered with the parameter `constraints` to the `minimize` function (and we can also opt to enter the Jacobian of these constraints):

```python
# use this in combination with the parameter bounds:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)})
```

```python
# to enter everything as constraints:
cons = ({'type': 'eq',
         'fun' : lambda x, r, f0, M: np.array([r[0]*x[0] + r[1]*x[1] + r[2]*x[2] - M]),
         'jac' : lambda x, r, f0, M: np.array([r[0], r[1], r[2]]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[0]-x[0],
         'jac' : lambda x, r, f0, M: np.array([-1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[1]-x[1],
         'jac' : lambda x, r, f0, M: np.array([0, -1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: f0[2]-x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, -1]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[0],
         'jac' : lambda x, r, f0, M: np.array([1, 0, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[1],
         'jac' : lambda x, r, f0, M: np.array([0, 1, 0]), 'args': (r, f0, M)},
        {'type': 'ineq', 'fun' : lambda x, r, f0, M: x[2],
         'jac' : lambda x, r, f0, M: np.array([0, 0, 1]), 'args': (r, f0, M)})
```

Although more verbose, if all the Jacobians of the constraints are also informed, this alternative seems better than informing bounds for the optimization process (less error in the final result and less iterations).

Given the characteristics of the problem, if we use the function `minimize` we are limited to the SLSQP (Sequential Least SQuares Programming) solver.

Finally, let's run the optimization for the four different cost functions and find the optimal muscle forces:

```python
f1r = minimize(fun=cf_f1, x0=x0, args=(), jac=cf_f1d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 437.4999999999541
                Iterations: 8
                Function evaluations: 8
                Gradient evaluations: 8

```python
f2r = minimize(fun=cf_f2, x0=x0, args=(), jac=cf_f2d,
               constraints=cons, method='SLSQP',
               options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 88888.88888888888
                Iterations: 8
                Function evaluations: 10
                Gradient evaluations: 8

```python
fpcsa2r = minimize(fun=cf_fpcsa2, x0=x0, args=(a,), jac=cf_fpcsa2d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 277.3925104022191
                Iterations: 11
                Function evaluations: 11
                Gradient evaluations: 11

```python
fmmax3r = minimize(fun=cf_fmmax3, x0=x0, args=(m,), jac=cf_fmmax3d,
                   constraints=cons, method='SLSQP',
                   options={'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 2142.4111183696405
                Iterations: 12
                Function evaluations: 12
                Gradient evaluations: 12

```python
opt5 = np.around(np.vstack((f1r.x, f2r.x, fpcsa2r.x, fmmax3r.x)), 1)
```

**Results of the optimization problem considering a desired torque of 20Nm and 45 degrees elbow flexion**

```python
dat = np.vstack((np.around(r*100,1), np.around(a,1), np.around(f0,0), np.around(m,1)))
er = ['-', '-', '-', '-',
      np.sum(r*f1r.x)-M, np.sum(r*f2r.x)-M, np.sum(r*fpcsa2r.x)-M, np.sum(r*fmmax3r.x)-M]
data = np.vstack((np.vstack((dat, opt5)).T, er)).T

rows = ['$\text{Moment arm}\;[cm]$', '$pcsa\;[cm^2]$', '$F_{max}\;[N]$', '$M_{max}\;[Nm]$',
        '$\sum F_i$', '$\sum F_i^2$', '$\sum(F_i/pcsa_i)^2$', '$\sum(F_i/M_{max,i})^3$']
cols = ['Biceps long head', 'Biceps short head', 'Brachialis', 'Error in M']
df = pd.DataFrame(data, index=rows, columns=cols)
print('\nSolving the distribution problem for a desired torque of 20Nm and at 45 elbow flexion')
df
```

    Solving the distribution problem for a desired torque of 20Nm and at 45 elbow flexion

      Biceps long head
      Biceps short head
      Brachialis
      Error in M

      $\text{Moment arm}\;[cm]$
      2.0
      5.0
      4.0
      -

      $pcsa\;[cm^2]$
      33.0
      8.0
      23.0
      -

      $F_{max}\;[N]$
      1000.0
      250.0
      700.0
      -

      $M_{max}\;[Nm]$
      20.0
      12.5
      28.0
      -

      $\sum F_i$
      -0.0
      250.0
      187.5
      0.0

      $\sum F_i^2$
      88.9
      222.2
      177.8
      0.0

      $\sum(F_i/pcsa_i)^2$
      302.1
      44.4
      293.5
      0.0

      $\sum(F_i/M_{max,i})^3$
      130.9
      102.3
      306.7
      0.0

## Comments

The results show that the estimations for the muscle forces depend on the cost function used in the optimization. Which one is correct? This is a difficult question and it's dependent on the goal of the actual task being modeled. Glitsch and Baumann (1997) investigated the effect of different cost functions on the optimization of walking and running and the predicted muscles forces were compared with the electromyographic activity of the corresponding muscles of the lower limb. They found that, among the analyzed cost functions, the minimization of the sum of squared muscle stresses resulted in the best similarity with the actual electromyographic activity.

In general, one should always test different algorithms and different initial values before settling for the solution found. Downey (2011), Kitchin (2013), and Kiusalaas (2013) present more examples on numerical optimization. The [NEOS Guide](http://neos-guide.org/) is a valuable source of information on this topic and [OpenOpt](http://openopt.org/) is a good alternative software for numerical optimization in Python.

## Exercises

1. Find the extrema in the function $f(x)=x^3-7.5x^2+18x-10$ analytically and determine if they are minimum or maximum.
2. Find the minimum in the $f(x)=x^3-7.5x^2+18x-10$ using the gradient descent algorithm.
2. Regarding the distribution problem for the elbow muscles presented in this text:
    a. Test different initial values for the optimization.
    b. Test other values for the elbow angle where the results are likely to change.

3. In an experiment to estimate forces of the elbow flexors, through inverse dynamics it was found an elbow flexor moment of 10 Nm.
Consider the following data for maximum force (F0), moment arm (r), and pcsa (A) of the brachialis, brachioradialis, and biceps brachii muscles: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A (cm$^2$): 33, 8, 23, respectively (data from Robertson et al. (2013)).
    a. Use static optimization to estimate the muscle forces.
    b. Test the robustness of the results using different initial values for the muscle forces.
    c. Compare the results for different cost functions.

## References

- Bergstra B, Bengio Y (2012) [Random Search for Hyper-Parameter Optimization](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf). Journal of Machine Learning Research, 13, 281-305.
- Crowninshield RD, Brand RA (1981) [A physiologically based criterion of muscle force prediction in locomotion](http://www.ncbi.nlm.nih.gov/pubmed/7334039). Journal of Biomechanics, 14, 793–801.
- Downey AB (2014) [Physical Modeling in MATLAB](http://greenteapress.com/wp/physical-modeling-in-matlab-2e/). 2nd edition. Green Tea Press.
- Herzog W (1987) [Individual muscle force estimations using a non-linear optimal design](http://www.ncbi.nlm.nih.gov/pubmed/3682873). J Neurosci Methods, 21, 167-179.
- Glitsch U, Baumann W (1997) [The three-dimensional determination of internal loads in the lower extremity](http://www.ncbi.nlm.nih.gov/pubmed/9456380). Journal of Biomechanics, 30, 1123–1131.
- Kitchin J (2013) [pycse - Python Computations in Science and Engineering](http://kitchingroup.cheme.cmu.edu/pycse/).
- Kiusalaas (2013) [Numerical methods in engineering with Python 3](http://books.google.com.br/books?id=aJkXoxxoCoUC). 3rd edition. Cambridge University Press.
- Nigg BM and Herzog W (2006) [Biomechanics of the Musculo-skeletal System](https://books.google.com.br/books?id=hOIeAQAAIAAJ&dq=editions:ISBN0470017678). 3rd Edition. Wiley.
- Robertson G, Caldwell G, Hamill J, Kamen G (2013) [Research Methods in Biomechanics](http://books.google.com.br/books?id=gRn8AAAAQBAJ). 2nd Edition. Human Kinetics.
- Seireg A, Arvikar RJ (1973) [A mathematical model for evaluation of forces in lower extremeties of the musculo-skeletal system](http://www.ncbi.nlm.nih.gov/pubmed/4706941). Journal of Biomechanics, 6,  313–322, IN19–IN20, 323–326.


# Test case: pivot irrigation kinematics

Consider a pivot irrigation system with following points:

* A: pivot
* B: attachment of pipe to rolling support
* C: wheel center
* D: wheel-ground contact point

The pivot rotates at an angular rate of $\omega_z$,
the wheel center is a distance $R$ from the pivot,
the wheel has a radius $r$,
and the wheel rolls without slip with angular rate $\omega_x$,
which implies the following relationship:

$ R \omega_z = r \omega_x $

The linear acceleration at points B and C is purely centripetal, pointing towards the pivot.
The linear acceleration at point D is a bit more complicated, so an analytical solution is derived here.

The position of point C:

$C = \\
R [\cos \omega_z t \\
   \sin \omega_z t\\
   0]^T$

and has linear acceleration:

$\ddot{C} = \\
-R \omega_z^2 [\cos \omega_z t \\
               \sin \omega_z t\\
               0]^T$

$\ddot{C} = -R \omega_z^2 C$

The position of point D relative to C:

$D - C= \\
r[ \sin \omega_x t \sin \omega_z t \\
  -\sin \omega_x t \cos \omega_z t \\
  -\cos \omega_z t]^T$

```
from sympy import *
R = Symbol('R')
r = Symbol('r')
t = Symbol('t')
wx = Symbol('wx')
wz = Symbol('wz')
```

```
# Position of D relative to C
Dx =  r * sin(wx*t) * sin(wz*t)
Dy = -r * sin(wx*t) * cos(wz*t)
Dz = -r * cos(wx*t)
```

```
# Acceleration of D relative to C
print(simplify(diff(diff(Dx, t), t)))
print(simplify(diff(diff(Dy, t), t)))
print(simplify(diff(diff(Dz, t), t)))
```

    r*(-wx**2*sin(t*wx)*sin(t*wz) + 2*wx*wz*cos(t*wx)*cos(t*wz) - wz**2*sin(t*wx)*sin(t*wz))
    r*(wx**2*sin(t*wx)*cos(t*wz) + 2*wx*wz*sin(t*wz)*cos(t*wx) + wz**2*sin(t*wx)*cos(t*wz))
    r*wx**2*cos(t*wx)

and has linear acceleration:

$\ddot{D} - \ddot{C} = \\
r [-(\omega_x^2 + \omega_z^2) \sin \omega_x t \sin \omega_z t + 2 \omega_x \omega_z \cos \omega_x t \cos \omega_z t \\
   (\omega_x^2 + \omega_z^2) \sin \omega_x t \cos \omega_z t + 2 \omega_x \omega_z \cos \omega_x t \sin \omega_z t \\
   \omega_x^2 \cos \omega_x t]^T$

$\ddot{D} - \ddot{C} = \\
r [-(\omega_x^2 + \omega_z^2) \sin \omega_x t \sin \omega_z t \\
   (\omega_x^2 + \omega_z^2) \sin \omega_x t \cos \omega_z t \\
   \omega_x^2 \cos \omega_x t]^T \\
+ 2r \omega_x \omega_z \cos \omega_x t
[\cos \omega_z t \\
 \sin \omega_z t \\
 0]^T$

which has a relative centripetal acceleration of $2r \omega_x \omega_z t$
when point D is at the contact point ($cos \omega_x t = 1$).
Using the no-slip condition, this is equivalent to $2R \omega_z^2 t$.
The xy centripetal acceleration components of D towards the pivot are then

$\ddot{D}_{xy} = R \omega_z^2 C_{xy}$

which is the exact opposite of the centripetal acceleration at $C$.

```

```


# A Cournot competition model with product differentiation
## Model Project
### Group: Anders&Frederik
#### Group members: Frederik Andresen, rjv586. Anders Meelby, zpw286.

**The model**
Consider two firms who compete in the same market i.e. a duopoly. The market is characterized by Cournot competetion:

- The firms are competing in quantities they produce;
- The firms decide the quantities independently of each other and simultaneously;
- The firms do not coorperate (no collusion);
- The firms have market power why each firm's output decision affects the price of each firm's good;
- The firms are economically rational and act strategically to maximize their profit given their competitors' decisions.

**Demand/price**

We define the inverse demand (price) for firm $i$ in such a market:
$p_i=p(x_i+x_j)=1-x_i+b\cdot x_j,$ for each $i\neq j \in \{1,2\},$
where $x_1$ is production by firm 1, $x_2$ is production by firm 2, and $b$ is a positive constant determining the degree of substitutability between the two goods. Letting $b=1$ makes the goods perfect substitutes and $b<1$ makes the goods differentiated.

```python
# 1.1 Define price/demand
def demand(x1,x2,b):
    return 1-x1-b*x2
```

**Cost**
Both firms have the cost functions, $C(x_i)=cx_i$ for $i=1,2$ such that the marginal costs are constant, $\dfrac{\partial C(x_i)}{\partial x_i}=c_i.$

```python
# 1.2 Define costs
def cost(x,c): #c(x==0)=0
    if x == 0:
     cost = 0
    else:
     cost = c*x
    return cost
```

**Profit**
The profit of firm $i$ can be stated as the following:
$\pi_i(x_i,x_j) = p_i(x_i,x_j)x_i-C(x_i)$   for each $i\neq j \in \{1,2\}$

```python
# 1.3 Define profits
def profit(x1,x2,c1,b):
    return demand(x1,x2,b)*x1 - cost(x1,c1)
```

**Best response**
Firm $i$ chooses its quantity, $x_i$, as to maximize profits, taking $x_j$ as given. $(x_i^*,x_j^*)$ is therefore a Nash equillibrium if $x_i^*=\underset{x_i}{\textrm{arg max}} \pi_i(x_i,x_j^*)$ for each $i\neq j \in{1,2}$.
If the profit is concave in output, the Nash equillibrium $(x_i^*,x_j^*)$ is the solution to $\frac{\partial \pi_i(x_i,x_j}{\partial x_i}|_{x_i=x_i^*,x_j=x_j^*} = 0$.

Isolating the profit maximizing quantity $x_i^*$ yields the best response function, $x_i^*=R_{x_i}(x_j^*)$.
To define this function, we use scipy brute force where the optimization problem per default is a minimization problem. Thus, if we want to maximize the profit of firm i, $\pi_i(x_i,x_j)$, we need to minimize $-\pi_i(x_i,x_j)$.
The method evaluates $-\pi_i(x_i,x_j)$ for each value of $x_i$ in the given range in order to find the global minimum. We set this particular range of 0 to 1 for our results to be well-defined.
The best response functions:

```python
from scipy import optimize

# 1.4 Define best response function of both firms using lambda and optimize.brute
def reaction(x2,c,b):
    x1 = optimize.brute(lambda x: -profit(x,x2,c,b),
                        ranges=((0,1),), #x1,x2 can take on values between 0 and 1.
                        finish=optimize.fmin)
    return x1[0]
```

**Finding the equilibrium by numerical optimization**
Having defined the best repsonse function as a function of the other firms choice of quantity and the parameters, we need to set up a system of equations, which ensures that both firms are responding optimally given the other firms choice of quantity. We define two vectors in order to satisfy this condition.
The first vector, $x^* = \begin{pmatrix}x_1^* \\x_2^* \end{pmatrix}$, is the Nash equillibrium, and the other being each firms reaction function given the
other firms choice of quantity, $f(x^*) = \begin{pmatrix}r_1(x_2^*) \\r_2(x_1^*) \end{pmatrix}$.
The Nash equillibrium, $x^*$ must satisfy that $x^* = f(x^*) \Leftrightarrow x^* - f(x^*) = 0$.
This is just another way of stating that both firms are responding optimally and therefore have no incentive to deviate. We define a function, vector_reaction, which is passed an initial quantity for each firm as well as a set of parameters. vector_reaction keeps changing quantities untill both firms has maximized profits i.e when $x^* - f(x^*) = 0$.
The equilibrium condition in vectorized form (array):

```python
from numpy import array

# 1.5 Define best responses as an array
def vector_reaction(x,param): # vector param = (b,c1,c2)
    return array(x)-array([reaction(x[1],param[1],param[0]),  #x1 = reaction(x2,c1,b)
                           reaction(x[0],param[2],param[0])]) #x2 = reaction(x1,c2,b)
```

We can solve for the profit maximizing quantity using Scipy's "fsolve", which can solve a vectorized system of equations using an initial guess and the parameter values.
The first case is the one with perfect substitutes or homogeneous products (b=1), the next is imperfect substitutes (b=0.5) and the last is perfect complements  (b=0).
Finally, we print the Nash equilibrium quantity and -profit.

```python
from IPython.display import Markdown, display #Creating Markdown-like printing for better visuals
def printmd(string):
    display(Markdown(string))

b_list=[1,0.5,0]
x0_list=[[0.3,0.3],[0.6,0.6],[0.5,0.5]]
titles =['**Perfect substitutes (b=1)**','**Imperfect substitutes (b=0.5)**','**Perfect complements  (b=0)**']
# 1.6 Print results by looping through the above lists
for x0, b, t in zip(x0_list, b_list, titles):
    x_opt = optimize.fsolve(vector_reaction, x0, args = ([b,0,0]))
    printmd(t)
    printmd(f'The Nash equilibrium quantity for both firms, $x^* =[x_1^*,x_2^*]$ = {x_opt}')
    printmd(f'The Nash equilibrium price for firm 1 and 2, $p^*$= {demand(x_opt[0],x_opt[1],b)}')
    printmd(f'The Nash equilibrium profit for firm 1 and 2, $\pi^*$= {profit(x_opt[0],x_opt[1],0,b)}')

```

**Perfect substitutes (b=1)**

The Nash equilibrium quantity for both firms, $x^* =[x_1^*,x_2^*]$ = [0.33332648 0.33332648]

The Nash equilibrium price for firm 1 and 2, $p^*$= 0.3333470347232139

The Nash equilibrium profit for firm 1 and 2, $\pi^*$= 0.11111339458222716

**Imperfect substitutes (b=0.5)**

The Nash equilibrium quantity for both firms, $x^* =[x_1^*,x_2^*]$ = [0.4 0.4]

The Nash equilibrium price for firm 1 and 2, $p^*$= 0.39999999997022395

The Nash equilibrium profit for firm 1 and 2, $\pi^*$= 0.15999999999602987

**Perfect complements  (b=0)**

The Nash equilibrium quantity for both firms, $x^* =[x_1^*,x_2^*]$ = [0.49995888 0.49995888]

The Nash equilibrium price for firm 1 and 2, $p^*$= 0.5000411184210527

The Nash equilibrium profit for firm 1 and 2, $\pi^*$= 0.24999999830927547

Each firm sets a higher output (and charges a higher price) at the monopoly outcome (b=0) with independent products than if products are homogeneous (b=1).
Generally, the equilibrium quantity, -price and profit increase as the products become more differentiated.

The analytical solution is symmetrical as well (c=0):    $x^*=\dfrac{1}{2+b}=p^*$, and, $\pi^*=\dfrac{1+2b}{(2+b)^2}$,
and checks out with the results above.

**Vizualizing the best response functions**
We plot the best response functions for the two firms and visualize the equilibrium quantities with three different parameter values of b.

```python
import sympy as sy
import matplotlib.pyplot as plt
import numpy as np
plt.style.use('seaborn-poster')

# 2.1 Create list of three parameter values and figure titles
b_list = [1, 0.5, 0.001]
title_list = ["The best response functions in quantity-space with perfect substitutes (b=1)", "The best response functions in quantity-space with imperfect substitutes (b=0.5)","The best response functions in quantity-space with close complements (b=0.001)"]

# 2.2 Two loops through two lists for the parameter b and figure titles, respectively
for b, t in zip(b_list,title_list):

    # 2.3 Define array of quantities and storage vectors
    N=1000
    q_vec = np.linspace(0, 1, N)
    q1_vec = np.empty(N)
    q2_vec = np.empty(N)

    # 2.4 Loop through the 1000 quantities in q_vec and get R_i(x_j) for each firm
    for i,x2 in enumerate(q_vec):
        cord1 = q1_vec[i] = reaction(x2,0,b)
    for i,x1 in enumerate(q_vec):
        cord2 = q2_vec[i] = reaction(x1,0,b)

    # 2.5 Plot equilibrium quantity given b
    plt.title(t)
    plt.xlabel('$x_2$')
    plt.ylabel('$x_1$')
    plt.plot(q1_vec, q_vec, '-', label='$R_1(x_2)$')
    plt.plot(q_vec, q2_vec, '-', label='$R_2(x_1)$')
    plt.legend(loc='best')
    plt.xlim(0,1)
    plt.ylim(0,1)
    p = np.argwhere(np.diff(np.sign(q_vec - q1_vec))).flatten()
    plt.plot(q_vec[p], q_vec[p], 'ro')
    printmd(f'Intersection, $x^*$=({q_vec[p]},{q_vec[p]})')
    plt.show()
```

In the two first graphs both best responses are decreases in the rival firm's quantity: If one firm raises its output it is optimal for the other firm to reducer its output ("Strategic substitutes").
At (near) perfect complements, the optimal output is (near) independent of the rival firm's output.

**Plotting the parameter $b$ against equilibrium quantity, -price and profit.**
* We initially define a set of storage arrays for each variable to be used with a length of N=1000.
* We loop through each value of b in an equal spaced array using enumerate.
* For each b in the loop, we solve for the equilibrium quantity for either firm using Scipy's optimize.solve.
* We define the resulting eq. price and -profit.
* We get that the equilibrium quantity, -price and profit get lower as the products become less differentiated.

```python
# 3.1 Define data for b and storage vectors
b_vec = np.linspace(0, 1, N)
x1_vec = np.empty(N)
x2_vec = np.empty(N)
price_vec = np.empty(N)
pi_vec = np.empty(N)

# 3.2 Define solver from the vectorized equilibrium condition for b to loop through
def solver(b):
    return optimize.fsolve(vector_reaction, [0.5,0.5], args = ([b,0,0]))

# 3.3 Let b loop through the solver and get optimal quantity, -price and -profit of firm 1 for each b (price and profit of firm 2 are commented out)
for i,b in enumerate(b_vec):
    cord = solver(b)
    x1_vec[i] = cord[0]
    x2_vec[i] = cord[1]
    price_vec[i] = demand(x1_vec[i],x2_vec[i],b)
    pi_vec[i] = profit(x1_vec[i],x2_vec[i],0,b)

# 3.4 Plot equilibrium quantity given b
fig, (ax_0, ax_1, ax_2) = plt.subplots(3, sharex=True)

ax_0.plot(b_vec,x1_vec)

ax_0.set_title('Equilibrium quantity given b')
ax_0.set_ylabel('$x^\star$')
ax_0.grid(True)

# 3.5 Plot equilibrium price given b

ax_1.plot(b_vec,price_vec)

ax_1.set_title('Equilibrium price given b')
ax_1.set_ylabel('$p^\star$')
ax_1.grid(True)

# 3.5 Plot equilibrium profit given b

ax_2.plot(b_vec,pi_vec)

ax_2.set_title('Equilibrium profit given b')
ax_2.set_xlabel('$b$')
ax_2.set_ylabel('$\pi^\star$')
ax_2.grid(True)

plt.show
```

The equilibrium quantity and price are equal as previously stated.
**Conclusion**
The equilibrium quantity, -price and profit decrease as the products become less differentiated.
Thus, having two monopolists with heterogeneous products is preferable to a duopoly with homogeneous products from a social welfare point-of-view.


```python
import numpy as np
import scipy as sp
from scipy import special as sp
import matplotlib.pyplot as plt
```

# Gaussians

Mainly the ideas expressed in the tutorial: https://www.youtube.com/watch?v=4vGiHC35j9s&t=422s

Gaussian processes can be used for a variety of things. This Notebook will provide an overview of Gaussian Processes. We start with the basics of Gaussian functions. Gaussian processes are defined by covarieneces. Some important definitions follow:

1. The covariance

\begin{align}
cov( \mathbf x, \mathbf y) = \sum_i { (x_i - \mu_x)(y_i - \mu_y) } = \mathbb E ( \mathbf x \mathbf y )
\end{align}

2. The coorrelation coefficient

\begin{align}
\rho( \mathbf x, \mathbf y) = \frac { cov( \mathbf x, \mathbf y ) } { \sqrt{ cov( \mathbf x, \mathbf x ) cov( \mathbf y, \mathbf y ) }}
\end{align}

3. The dot product represents similarity between vectors: $\mathbf x \cdot \mathbf y$ = `x @ y.T`

4. $\mathcal N ( 0, 1 ) $ Gaussians are very useful because they can be shifted and scaled to any place.

5. Gaussian vectors can be sampled from Gaussians:

\begin{align}
\begin{bmatrix}
x_1 \\
x_2
\end{bmatrix} &=
\mathcal N \Big({
\begin{bmatrix}
0 \\
0
\end{bmatrix},
\begin{bmatrix}
1                   & \mathbb E (x_1x_2) \\
\mathbb E (x_2x_1)  & 1
\end{bmatrix}}\big)                                \\
&= \mathcal N \Big({
\begin{bmatrix}
0 \\
0
\end{bmatrix},
\mathbf \Sigma
}\big)
\end{align}

Thus evevrything about Gaussian processes revolves around the covariance matrix $\mathbf \Sigma$

## Samppling form a distribution

Sampling can be performed by sampling from a uniform distribution, and then projecting it throough the CDF of the Gaussian function in the reverse order. This can be later shifted and scaled as necessary with $\sigma$ and $\mu$

```python
dist = sp.erfinv((np.random.rand(100000)-0.5)*2)

plt.figure(figsize=(15,5))
plt.subplot(131)
_ = plt.hist(dist, bins=100)
plt.subplot(132)
_ = plt.hist(dist+5, bins=100)
plt.subplot(133)
_ = plt.hist(dist*2 + 5, bins=100)
plt.show()
plt.close()
```

```python

```