# Eigenvalue problems

"Eigenvalue" means characteristic value. These types of problems show up in many areas involving boundary-value problems, where we may not be able to obtain an analytical solution, but we can identify certain characteristic values that tell us important information about the system: the eigenvalues.

## Example: beam buckling

Let's consider deflection in a simply supported (static) vertical beam: $y(x)$, with boundary conditions $y(0) = 0$ and $y(L) = 0$. To get the governing equation, start with considering the sum of moments around the upper pin:
\begin{align}
\sum M &= M_z + P y = 0 \\
M_z &= -P y
\end{align}

We also know that $M_z = E I y''$, so we can obtain
\begin{align}
M_z = E I \frac{d^2 y}{dx^2} &= -P y \\
y'' + \frac{P}{EI} y &= 0
\end{align}
This equation governs the stability of a beam, considering small deflections.
To simplify things, let's define $\lambda^2 = \frac{P}{EI}$, which gives us the ODE
\begin{equation}
y'' + \lambda^2 y = 0
\end{equation}
We can get the general solution to this:
\begin{equation}
y(x) = A \cos (\lambda x) + B \sin (\lambda x)
\end{equation}

To find the coefficients, let's apply the boundary conditions, starting with $x=0$:
\begin{align}
y(x=0) &= 0 = A \cos 0 + B \sin 0 \\
\rightarrow A &= 0 \\
y(x=L) &= 0 = B \sin (\lambda L)
\end{align}
Now what? $B \neq 0$, because otherwise we would have the trivial solution $y(x) = 0$. Instead, to satisfy the boundary condition, we need
\begin{align}
B \neq 0 \rightarrow \sin (\lambda L) &= 0 \\
\text{so} \quad \lambda L &= n \pi \quad n = 1, 2, 3, \ldots, \infty \\
\lambda &= \frac{n \pi}{L} \quad n = 1, 2, 3, \ldots, \infty
\end{align}
$\lambda$ give the the **eigenvalues** for this problem; as you can see, there are an infinite number, that correspond to **eigenfunctions**:
\begin{equation}
y_n = B \sin \left( \frac{n \pi x}{L} \right) \quad n = 1, 2, 3, \ldots, \infty
\end{equation}

The eigenvalues and associated eigenfunctions physically represent different modes of deflection.
For example, consider the first three modes (corresponding to $n = 1, 2, 3$):

```matlab
clear all; clc

L = 1.0;
x = linspace(0, L);
subplot(1,3,1);
y = sin(pi * x / L);
plot(y, x); title('n = 1')
subplot(1,3,2);
y = sin(2 * pi * x / L);
plot(y, x); title('n = 2')
subplot(1,3,3);
y = sin(3* pi * x / L);
plot(y, x); title('n = 3')
```

Here we see different modes of how the beam will buckle. How do we know when this happens?

Recall that the eigenvalue is connected to the physical properties of the beam:
\begin{gather}
\lambda^2 = \frac{P}{EI} \rightarrow \lambda = \sqrt{\frac{P}{EI}} = \frac{n \pi}{L} \\
P = \frac{EI}{L} n^2 \pi^2
\end{gather}
This means that when the combination of load force and beam properties match certain values, the beam will deflect—and buckle—in one of the modes corresponding to the associated eigenfunction.

In particular, the first mode ($n=1$) is interesting, because this is the first one that will be encountered if a load starts at zero and increases. This is the **Euler critical load** of buckling, $P_{cr}$:
\begin{gather}
\lambda_1 = \frac{\pi}{L} \rightarrow \lambda_1^2 = \frac{P}{EI} = \frac{\pi^2}{L^2} \\
P_{cr} = \frac{\pi^2 E I}{L^2}
\end{gather}

## Example: beam buckling with different boundary conditions

Let's consider a slightly different case, where at $x=0$ the beam is supported such that $y'(0) = 0$. How does the beam buckle in this case?

The governing equation and general solution are the same:
\begin{align}
y'' + \lambda^2 y &= 0 \\
y(x) &= A \cos (\lambda x) + B \sin (\lambda x)
\end{align}
but our boundary conditions are now different:
\begin{align}
y'(0) = 0 = -\lambda A \sin(0) + \lambda B\cos(0) \\
\rightarrow B &= 0 \\
y &= A \cos (\lambda x) \\
y(L) &= 0 = A \cos (\lambda L) \\
A \neq 0 \rightarrow \cos(\lambda L) &= 0 \\
\text{so} \quad \lambda L &= \frac{(2n-1) \pi}{2} \quad n = 1,2,3,\ldots, \infty \\
\lambda &= \frac{(2n-1) \pi}{2 L} \quad n = 1,2,3,\ldots, \infty
\end{align}

Then, the critical buckling load, again corresponding to $n=1$, is
\begin{equation}
P_{cr} = \frac{\pi^2 EI}{4 L^2}
\end{equation}

## Getting eigenvalues numerically

We can only get the eigenvalues analytically if we can obtain an analytical solution of the ODE, but we might want to get eigenvalues for more complex problems too. In that case, we can use an approach based on *finite differences* to find the eigenvalues.

Consider the same problem as above, for deflection of a simply supported beam:
\begin{equation}
y'' + \lambda^2 y = 0
\end{equation}
with boundary conditions $y(0) = 0$ and $y(L) = 0$. Let's represent this using finite differences, for a case where $L=3$ and $\Delta x = 1$, so we have four points in our solution grid.

The finite difference representation of the ODE is:
\begin{align}
\frac{y_{i-1} - 2y_i + y_{i+1}}{\Delta x^2} + \lambda^2 y_i &= 0 \\
y_{i-1} + \left( \lambda^2 \Delta x^2 - 2 \right) y_i + y_{i+1} &= 0
\end{align}
However, in this case, we are not solving for the values of deflection ($y_i$), but instead the **eigenvalues** $\lambda$.

Then, we can write the system of equations using the above recursion formula and our two boundary conditions:
\begin{align}
y_1 &= 0 \\
y_1 + y_2 \left( \lambda^2 \Delta x^2 - 2 \right) + y_3 &= 0 \\
y_2 + y_3 \left( \lambda^2 \Delta x^2 - 2 \right) + y_4 &= 0 \\
y_4 &= 0
\end{align}
which we can simplify down to two equations by incorporating the boundary conditions into the equations for the two middle points, and also letting $k = \lambda^2 \Delta x^2$:
\begin{align}
y_2 (k-2) + y_3 &= 0 \\
y_2 + y_3 (k-2) &= 0
\end{align}
Let's modify this once more by multiplying both equations by $-1$:
\begin{align}
y_2 (2-k) - y_3 &= 0 \\
-y_2 + y_3 (2-k) &= 0
\end{align}

Now we can represent this system of equations as a matrix equation $A \mathbf{y} = \mathbf{b} = \mathbf{0}$:
\begin{equation}
\begin{bmatrix} 2-k & -1 \\ -1 & 2-k \end{bmatrix}
\begin{bmatrix} y_2 \\ y_3 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}
\end{equation}
$\mathbf{y} = \mathbf{0}$ is a trivial solution to this, so instead $\det(A) = 0$ satisfies this equation.
For our $2\times 2$ matrix, that looks like:
\begin{align}
\det(A) = \begin{vmatrix} 2-k & -1 \\ -1 & 2-k \end{vmatrix} = (2-k)^2 - 1 &= 0 \\
k^2 - 4k + 3 &= 0 \\
(k-3)(k-1) &= 0
\end{align}
so the roots of this equation are $k_1 = 1$ and $k_2 = 3$. Recall that $k$ is directly related to the eigenvalue: $k = \lambda^2 \Delta x^2$, and $\Delta x = 1$ for this case, so we can calculate the two associated eigenvalues:
\begin{align}
k_1 &= \lambda_1^2 \Delta x^2 = 1 \rightarrow \lambda_1 = 1 \\
k_2 &= \lambda_2^2 \Delta x^2 = 3 \rightarrow \lambda_2 = \sqrt{3} = 1.732
\end{align}

Our work has given us approximations for the first two eigenvalues. We can compare these against the exact values, given in general by $\lambda = n \pi / L$ (which we determined above):
\begin{align}
n=1: \quad \lambda_1 &= \frac{\pi}{L} = \frac{\pi}{3} = 1.0472 \\
n=2: \quad \lambda_2 &= \frac{2\pi}{L} = \frac{2\pi}{3} = 2.0944
\end{align}
So, our approximations are close, but with some obvious error. This is because we used a fairly crude step size of $\Delta x = 1$, dividing the domain into just three segments. By using a finer resolution, we can get more-accurate eigenvalues and also more of them (remember, there are actually an infinite number!).

To do that, we will need to use Matlab, which offers the `eig()` function for calculating eigenvalues---essentially it is finding the roots to the polynomial given by $\det(A) = 0$. We need to modify this slightly, though, to use the function:
\begin{align}
\det(A) &= 0 \\
\det \left( A^* - k I \right) = 0
\end{align}
where the new matrix is
\begin{equation}
A^* = \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}
\end{equation}
Then, `eig(A*)` will provide the values of $k$, which we can use to find the $\lambda$s:

```matlab
clear all; clc

dx = 1.0;
L = 3.0;

Astar = [2 -1; -1 2];
k = eig(Astar);

lambda = sqrt(k) / dx;

fprintf('lambda_1: %6.3f\n', lambda(1));
fprintf('lambda_2: %6.3f', lambda(2));
```

    lambda_1:  1.000
    lambda_2:  1.732

As expected, this matches with our manual calculation above. But, we might want to calculate these eigenvalues more accurately, so let's generalize this a bit and then try using $\Delta x= 0.1$:

```matlab
clear all; clc

dx = 0.1;
L = 3.0;
x = 0 : dx : L;
n = length(x) - 2;

Astar = zeros(n,n);
for i = 1 : n
    if i == 1
        Astar(1,1) = 2;
        Astar(1,2) = -1;
    elseif i == n
        Astar(n,n-1) = -1;
        Astar(n,n) = 2;
    else
        Astar(i,i-1) = -1;
        Astar(i,i) = 2;
        Astar(i,i+1) = -1;
    end
end
k = eig(Astar);

lambda = sqrt(k) / dx;

fprintf('lambda_1: %6.3f\n', lambda(1));
fprintf('lambda_2: %6.3f\n\n', lambda(2));

err = abs(lambda(1) - (pi/L)) / (pi/L);
fprintf('Error in lambda_1: %5.2f%%\n', 100*err);
```

    lambda_1:  1.047
    lambda_2:  2.091

    Error in lambda_1:  0.05%

## Example: mass-spring system

Let's analyze the motion of masses connected by springs in a system:

:::{figure-md} fig-mass-spring

System with two masses connected by springs
:::

First, we need to write the equations of motion, based on doing a free-body diagram on each mass:
\begin{align}
m_1 \frac{d^2 x_1}{dt^2} &= -k x_1 + k(x_2 - x_1) \\
m_2 \frac{d^2 x_2}{dt^2} &= -k (x_2 - x_1) - k x_2
\end{align}
We can condense these equations a bit:
\begin{align}
x_1^{\prime\prime} - \frac{k}{m_1} \left( -2 x_1 + x_2 \right) &= 0 \\
x_2^{\prime\prime} - \frac{k}{m_2} \left( x_1 - 2 x_2 \right) &= 0
\end{align}

To proceed, we can assume that the masses will move in a sinusoidal fashion, with a shared frequency but separate amplitude:
\begin{align}
x_i &= A_i \sin (\omega t) \\
x_i^{\prime\prime} &= -A_i \omega^2 \sin (\omega t)
\end{align}
We can plug these into the ODEs:
\begin{align}
\sin (\omega t) \left[ \left( \frac{2k}{m_1} - \omega^2 \right) A_1 - \frac{k}{m_1} A_2 \right] &= 0 \\
\sin (\omega t) \left[ -\frac{k}{m_2} A_1 + \left( \frac{2k}{m_2} - \omega^2 \right) A_2 \right] &= 0
\end{align}
or
\begin{align}
\left( \frac{2k}{m_1} - \omega^2 \right) A_1 - \frac{k}{m_1} A_2 &= 0 \\
-\frac{k}{m_2} A_1 + \left( \frac{2k}{m_2} - \omega^2 \right) A_2 &= 0
\end{align}
Let's put some numbers in, and try to solve for the eigenvalues: $\omega^2$.
Let $m_1 = m_2 = 40 $ kg and $k = 200$ N/m.

Now, the equations become
\begin{align}
\left( 10 - \omega^2 \right) A_1 - 5 A_2 &= 0 \\
-5 A_1 + \left( 10 - \omega^2 \right) A_2 &= 0
\end{align}
or $A \mathbf{y} = \mathbf{0}$, which we can represent as
\begin{equation}
\begin{bmatrix} 10-\omega^2 & -5 \\ -5 & 10-\omega^2 \end{bmatrix}
\begin{bmatrix} A_1 \\ A_2 \end{bmatrix} =
\begin{bmatrix} 0 \\ 0 \end{bmatrix}
\end{equation}
Here, $\omega^2$ are the eigenvalues, and we can find them with $\det(A) = 0$:
\begin{align}
\det(B) &= 0 \\
\det (B^* - \omega^2 I) &= 0
\end{align}

```matlab
clear all; clc

Bstar = [10 -5; -5 10];
omega_squared = eig(Bstar);
omega = sqrt(omega_squared);

fprintf('omega_1 = %5.2f rad/s\n', omega(1));
fprintf('omega_2 = %5.2f rad/s\n', omega(2));
```

    omega_1 =  2.24 rad/s
    omega_2 =  3.87 rad/s

We find there are two modes of oscillation, each associated with a different natural frequency. Unfortunately, we cannot calculate independent and unique values for the amplitudes, but if we insert the values of $\omega$ into the above equations, we can find relations connecting the amplitudes:
\begin{align}
\omega_1: \quad A_1 &= A_2 \\
\omega_2: \quad A_1 &= -A_2
\end{align}

So, for the first mode, we have the two masses moving in sync with the same amplitude. In the second mode, they move with opposite (but equal) amplitude. With the two different frequencies, they also have two different periods:

```matlab
t = linspace(0, 3);
subplot(1,5,1)
plot(sin(omega(1)*t), t); hold on
plot(0,0, 's');
set (gca, 'ydir', 'reverse' )
box off; set(gca,'Visible','off')

subplot(1,5,2)
plot(sin(omega(1)*t), t); hold on
plot(0,0, 's');
set (gca, 'ydir', 'reverse' )
text(-2.5,-0.2, 'First mode')
box off; set(gca,'Visible','off')

subplot(1,5,4)
plot(-sin(omega(2)*t), t); hold on
plot(0,0, 's');
set (gca, 'ydir', 'reverse' )
box off; set(gca,'Visible','off')

subplot(1,5,5)
plot(sin(omega(2)*t), t); hold on
plot(0,0, 's');
set (gca, 'ydir', 'reverse' )
box off; set(gca,'Visible','off')
text(-2.7,-0.2, 'Second mode')
```

We can confirm that the system would actually behave in this way by setting up the system of ODEs and integrating based on initial conditions matching the amplitudes of the two modes.

For example, let's use $x_1 (t=0) = x_2(t=0) = 1$ for the first mode, and $x_1(t=0) = 1$ and $x_2(t=0) = -1$ for the second mode. We'll use zero initial velocity for both cases.

Then, we can solve by converting the system of two 2nd-order ODEs into a system of four 1st-order ODEs:

```matlab
%%file masses.m
function dxdt = masses(t, x)
% this is a function file to calculate the derivatives associated with the system

m1 = 40;
m2 = 40;
k = 200;

dxdt = zeros(4,1);

dxdt(1) = x(2);
dxdt(2) = (k/m1)*(-2*x(1) + x(3));
dxdt(3) = x(4);
dxdt(4) = (k/m2)*(x(1) - 2*x(3));
```

    Created file '/Users/niemeyek/projects/ME373-book/content/bvps/masses.m'.

```matlab
clear all; clc

% this is the integration for the system in the first mode
[t, X] = ode45('masses', [0 3], [1.0 0.0 1.0 0.0]);
subplot(1,5,1)
plot(X(:,1), t);
ylabel('displacement (m)'); xlabel('time (s)')
set (gca, 'ydir', 'reverse' )
%box off; set(gca,'Visible','off')

subplot(1,5,2)
plot(X(:,3), t); xlabel('time (s)')
set (gca, 'ydir', 'reverse' )
text(-4,-0.2, 'First mode')

% this is the integration for the system in the second mode
[t, X] = ode45('masses', [0 3], [1.0 0.0 -1.0 0.0]);
subplot(1,5,4)
plot(X(:,1), t);
ylabel('displacement (m)'); xlabel('time (s)')
set (gca, 'ydir', 'reverse' )
%box off; set(gca,'Visible','off')

subplot(1,5,5)
plot(X(:,3), t); xlabel('time (s)')
set (gca, 'ydir', 'reverse' )
text(-4,-0.2, 'Second mode')
```

This shows that we get either of the pure modes of motion with the appropriate initial conditions.

What about if the initial conditions *don't* match either set of amplitude patterns?

```matlab
[t, X] = ode45('masses', [0 3], [0.25 0.0 0.75 0.0]);
subplot(1,5,1)
plot(X(:,1), t);
%plot(0,0, 's');
set (gca, 'ydir', 'reverse' )
%box off; set(gca,'Visible','off')

subplot(1,5,2)
plot(X(:,3), t);
set (gca, 'ydir', 'reverse' )

```

In this case, the resulting motion will be a complicated superposition of the two modes.


$ x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} $

```python
from sympy import *
from IPython.display import display
from sympy.printing.mathml import mathml
from IPython.display import display, Math, Latex

x, y, z = symbols('x y z')
init_printing(use_unicode=True)
```

```python
def mprint(e):
    display(Math(latex(e)))
```

```python
pprint(diff(2*x**4))
```

       3
    8⋅x

```python
expr = simplify((x**3 + x**2 - x - 1)/(x**2 + 2*x + 1))
```

```python
expr = (x**3 + x**2 - x - 1)/(x**2 + 2*x + 1)
display(Math(latex(expr)))
expr = simplify(expr)
print(type(expr))
print(latex(expr))
display(Math(latex(expr)))
```

$$\frac{x^{3} + x^{2} - x - 1}{x^{2} + 2 x + 1}$$

    <class 'sympy.core.add.Add'>
    x - 1

$$x - 1$$

```python
from IPython.display import display, Math, Latex
display(Math('\\frac{1}{2}'))
```

$$\frac{1}{2}$$

```python
print(expr.subs(x,5))
```

    4

```python
eql = Eq(3*x+5,10)
```

```python
z = solveset(eql,x)
display(Math(latex(z)))
```

```python

```

$$\left\{\frac{5}{3}\right\}$$

```python
from sympy import *
x, y, z = symbols('x y z')
init_printing(use_unicode=True)
expr = diff(sin(x)/x**2, x)
mprint(expr)
```

$$\frac{1}{x^{2}} \cos{\left (x \right )} - \frac{2}{x^{3}} \sin{\left (x \right )}$$

```python
expr_i = integrate(expr,x)
mprint(expr_i)
```

$$\frac{1}{x^{2}} \sin{\left (x \right )}$$


```python
from sympy import *
```

Copyright - Michael Corley

# **Printing and graphing functions**
*   This tutorial should be useful for those looking for a Mathematica or Matlab-like tool in Python.
*   This tutorial is designed to bring you up to speed as quickly as possible on how to use the tool, not to teach you what functions are.
> *SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python.*

---

**Let** *f(x) = e^x* and *g(x) = sqrt(x-1)*
*   Lets work through how to initiate these functions

```python
x = Symbol('x')
f = Function('f')
g = Function('g')(x)

class f(Function):
  @classmethod
  def eval(cls, x):
    return exp(x)

class g(Function):
  @classmethod
  def eval(cls, x):
    return sqrt(x-1)
```

**First:** sympy allows you to print symbolic functions
*   You can either print it directly, or you can use the **latex()** function to output in Latex

```python
print(g(f(x)))
print(f(g(x)))

print('\n Latex output:')
print(latex(f(g(x))))
```

    sqrt(exp(x) - 1)
    exp(sqrt(x - 1))

     Latex output:
    e^{\sqrt{x - 1}}

**Second:** sympy allows you to graph functions automatically, but is build on commonly used Python libraries.
*   Normally this would required that you input some values for the function
*   You would have to write custom code to do this

```python
plot(f(g(x)))
```

```python
plot(g(f(x)))
```

---

# Solving for variables and simplifying expressions

Solving algebraic expressions using **solve()**
*   Simplfication is automatically set to true
*   **solve()** works with linear and non-linear equations

```python
z = Symbol('z')
t = Symbol('t')
equation = Equality(3*z-t,5)

#solve for z
print('z=')
print(solve(equation,z))

#solve for t
print('t=')
print(solve(equation,t))
```

    z=
    [t/3 + 5/3]
    t=
    [3*z - 5]

Non-linear function

```python
x = Symbol('x')
y = Symbol('y')
z = Symbol('z')

#solve for x
print('x=')
print(solve(z*x**2-y**2, x))

print('y=')
print(solve(z*x**2-y**2, y))
```

    x=
    [-y*sqrt(1/z), y*sqrt(1/z)]
    y=
    [-x*sqrt(z), x*sqrt(z)]

---
**Integrals:** probability density function example
$p(x) = \frac{\sqrt{2} e^{- \frac{0.5 \left(- \mu + x\right)^{2}}{s^{2}}}}{2 \sqrt{\pi} s}$

```python
mu = Symbol('mu')
s = Symbol('s')
x = Symbol('x')
pr = Function('pr')

#probability density function
class pr(Function):
  @classmethod
  def eval(cls, x, mu, s):
    return (1/(s*sqrt(2*pi)))*exp((-1/2)*((x-mu)/s)**2)

print(latex(pr(x, mu, s)))
```

    \frac{\sqrt{2} e^{- \frac{0.5 \left(- \mu + x\right)^{2}}{s^{2}}}}{2 \sqrt{\pi} s}

Plot the probability density function

```python
plot(pr(x,0,1)) #standard normal distribution has a mean of 0 and a sdev of 1
```

p(a≤x≤b) = ∫f(x)dx between b an a. What is the probability that x falls between 1 and 4?
*  **Solution per Excel: 0.158624**
* N() will make the numerical solution display
* integrate() can be used to evaluate definate and indefinate integrals

```python
a = 1
b = 4
print(N(integrate(pr(x, 0, 1), (x, a, b))))
```

    0.158623582689624

**Statistics:** probability density function
* The Normal() function can be used to define a normal distribution.
* The P() function can be used to compute the probability.

```python
from sympy.stats import *
Z=Normal('Z', 0, 1)
print(N(P(Z<4)-P(Z<1)))
```

    0.158623582689624

# Resources
* Latest official documentation: https://docs.sympy.org/latest/index.html
* Unofficial Tutorial: https://www.tutorialspoint.com/sympy/


# Binomial Distribution - Coin Toss

> This document is written in *R*.
>
> ***GitHub***: https://github.com/czs108

## Background

> Suppose you conduct a series of experiments, each of which consists of tossing a fair coin **6** times. Let $X$ be the number of *heads*.

## Question A

> What are the probabilities of each value of $X$?

\begin{equation}
P(X = r) =\, ^{6}C_{r} \cdot 0.5^{r} \cdot 0.5^{6 - r}
\end{equation}

```R
heads <- c(0:6)

probs <- choose(n=6, k=heads) * (0.5 ^ heads) * (0.5 ^ (6 - heads))
probs
```

0.015625

0.09375

0.234375

0.3125

0.234375

0.09375

0.015625

Use the `dbinom` function directly.

```R
dbinom(x=heads, prob=0.5, size=6)
```

0.015625

0.09375

0.234375

0.3125

0.234375

0.09375

0.015625

```R
barplot(probs ~ c(0:6), xlab="Number of heads", ylab="Probability")
```

## Question B

\begin{equation}
\begin{split}
P(1st = head \mid X = 4)
    &= \frac{P(1st = head \cap X = 4)}{P(X = 4)} \\
    &= \frac{^{5}C_{3}}{^{6}C_{4}} \\
    &= 0.6667
\end{split}
\end{equation}

## Question C

\begin{equation}
\begin{split}
P(X = 4 \mid 1st = head)
    &= \frac{P(1st = head \cap X = 4)}{P(1st = head)} \\
    &= \frac{^{5}C_{3}}{2^{5}} \\
    &= 0.3125
\end{split}
\end{equation}

```R
dbinom(x=3, prob=0.5, size=5)
```

0.3125

## Question D

> Suppose the event

\begin{equation}
A = \text{The 1st trial where X = 6 is 10th.}
\end{equation}

> Find out

\begin{equation}
P(A)
\end{equation}

Its's a *Geometric Distribution*.

\begin{equation}
p = P(X = 6) = \frac{1}{2^{6}}
\end{equation}

\begin{equation}
\begin{split}
P(A) &= (1 - p)^{9} \cdot p \\
    &= (1 - \frac{1}{2^{6}})^{9} \times \frac{1}{2^{6}} \\
    &= 0.0136
\end{split}
\end{equation}

## Question E

\begin{equation}
\begin{split}
P(X > 4) &= P(X = 5) + P(X = 6) \\
    &= \frac{^{6}C_{5} +\, ^{6}C_{6}}{2^6} \\
    &= 0.1094
\end{split}
\end{equation}

Use the `pbinom` function.

```R
1 - pbinom(q=4, prob=0.5, size=6)
```

0.109375

## Question F

> What is the *mean* of $X$?

Suppose

\begin{equation}
N = \text{The number of trials needed to get the first head.}
\end{equation}

The the *expectation* is

\begin{equation}
E(N) = \frac{1}{0.5} = 2
\end{equation}

There are **6** trials.

\begin{equation}
E(X) = \frac{6}{E(N)} = 3
\end{equation}

```R
mean <- sum(heads * probs)
mean
```

3

exercises/Binomial Distribution - Coin Toss.ipynb


# pH-Absorbance Calibration with PyTorch

In this notebook, we will be revisiting the data I collected in my UCLA undergrad Bioengineering capstone project. This data involves an absorbance based pH sensor (using an Arduino, LED, and phenol red indicator solution) for noninvasive monitoring of cell culture. By normalizing the voltage reading to that of phosphate buffered saline as blank, we obtained the Absorbance: $A = -\log \frac{I}{I_{PBS}}$.

The theoretical equation relating pH to absorbance is then given by:

\begin{equation}

A = f(pH) = \frac{A_{max}}{1 + 10^{pK_{a} - pH}}

\end{equation}

This corresponds to a sigmoid curve from $0$ to $A_{max}$. We choose to add in an extra shape parameter $\phi$ to account for deviations from the theory and use the natura exponential:

\begin{equation}

A = f(pH) = \frac{A_{max}}{1 + e^{(pK_{a} - pH)/\phi}}

\end{equation}

Unlike say a typical logistic regression sigmoid, this sigmoid has parameters that need to be found via nonlinear least square optimization methods. The loss to be minimized is the mean squared error:

\begin{equation}

Loss(A_{max},pK_{a},\phi) = \frac{1}{n} \sum^{n}_{i=1} (A_i - \frac{A_{max}}{1 + e^{(pK_{a} - pH_{i})/\phi}})^{2}

\end{equation}

We also have some prior information from theory. It can be shown with algebra that Equation (2) simplifies to Equation (1) when $\phi = \frac{1}{\log(10)} \approx 0.4343$. Additionally the theoretical pKa of phenol red is $pK_{a} = 7.6$. In a frequentist sense, this prior knowledge can be used to add regularization terms. For $A_{max}$ we do not necessarily have prior information, but we do not want the maximum absorbance to be extremely high, and thus can regularize it toward 0. An L1 penalty (in this case it will simplify to absolute values) will be used to regularize these parameters and will penalize the deviation from these prior values:

\begin{equation}

Penalty(A_{max},pK_{a},\phi) = \lambda_{A_{max}} |A_{max}| + \lambda_{pK_{a}} |pK_{a} - 7.6| + \lambda_{\phi}|\phi - \frac{1}{\log(10)}|

\end{equation}

The minimization problem, with $\theta = (A_{max},pK_{a},\phi)$ then becomes:

\begin{equation}

\underset{\theta}{\arg\min} (Loss(\theta) + Penalty(\theta))

\end{equation}

## Nonlinear Least Squares and Nonlinear Mixed Model

This dataset consists of 4 Trials, and during the trial, the solution pH was adjusted by adding very small drops of concentrated HCl or NaOH to neglect volume changes. The absorbance was measured and calibrated to a standard pH sensor. However, the nature of the experiment leads to correlated data points within a given trial. **In this first section, we will investigate the dataset with standard built in methods**.

We will fit NLS models from a wrapper calling R's nls() and (for comparison) scipy least_squares(). These do not account for correlation. To account for correlation, a nonlinear mixed model (NLMM) must be used. This is done through a wrapper that calls R's nlmer() function from lme4 package.

It is assumed that the only random effect is for $A_{max}$ and is normally distributed:

\begin{equation}

A_{max,Trial} \sim N(A_{max},\sigma_{A_{max}}^{2})

\end{equation}

The rpy2 package is used to communicate with R in order to use the wrappers found in the pHAbs_NLSNLMM.R file

All of these are unregularized (beyond the trial-specific regularization toward the mean induced by random effects in the NLMM from nlmer())

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from plotnine import ggplot, geom_point,geom_line, aes
from scipy.stats import truncnorm
from scipy.optimize import least_squares
```

```python
import rpy2.robjects as ro
from rpy2.robjects.packages import importr
from rpy2.robjects import pandas2ri
from rpy2.robjects.conversion import localconverter
```

```python

base = importr('base')
stats = importr('stats')
lme4 = importr('lme4')
ro.r['source']('pHAbs_NLSNLMM.R')
```

ListVector with 2 elements.

    value

    <rpy2.rinterface.SexpClosure object at 0x7f85cc0dad80> [RTYPES.CLOSXP]

    visible

    <rpy2.rinterface.BoolSexpVector object at 0x7f85b17995c0> [RTYPES.LGLSXP]

```python
data = pd.read_csv("Full_pHAbsdata.csv")
data.sample(frac=1) #randomize row order for later
```

      pH
      ALED
      Trial

      11
      7.74
      0.313864
      1

      19
      4.08
      -0.001617
      2

      27
      6.68
      0.055223
      2

      67
      5.36
      0.005238
      4

      39
      7.96
      0.290464
      2

      ...
      ...
      ...
      ...

      41
      8.51
      0.335765
      2

      64
      11.56
      0.461222
      3

      72
      6.67
      0.050556
      4

      89
      10.66
      0.413847
      4

      30
      6.97
      0.096303
      2

91 rows × 3 columns

```python
pH_data = data.pH.to_numpy()
ALED_data = data.ALED.to_numpy()
```

```python

with localconverter(ro.default_converter + pandas2ri.converter):
  NLSresult = ro.r.Fit_NLS(data)
  NLMMresult = ro.r.Fit_NLMM(data)

data["Ahat_NLS"] = np.array(stats.predict(NLSresult))
data["Ahat_NLMM"] = np.array(stats.predict(NLMMresult))

(ggplot(data,aes('pH','ALED',color ='factor(Trial)'))
+ geom_point() + geom_line(aes('pH','Ahat_NLMM',color='factor(Trial)'))
+ geom_line(aes('pH','Ahat_NLS'),inherit_aes=False))
```

The data and the fitted values from R's nls() and nlmer() (colored) are seen above. The dark curve represents the overall average relationship based on nls() while the different colored curves are the Trial-specific fits as calculated by nlmer() with a random effect on $A_{max}$. The differences in $A_{max}$ can be caused by differing optics between the trials, which would affect how the light enters the cuvette.

## Nonlinear Least Squares Results (R nls())

```python
print(base.summary(NLSresult))
```

    Formula: ALED ~ Amax/(1 + exp((pKa - pH)/phi))

    Parameters:
         Estimate Std. Error t value Pr(>|t|)
    Amax 0.423757   0.006912   61.30   <2e-16 ***
    pKa  7.470071   0.027519  271.45   <2e-16 ***
    phi  0.453836   0.023565   19.26   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.02612 on 88 degrees of freedom

    Number of iterations to convergence: 5
    Achieved convergence tolerance: 2.001e-06

According to R nls(), we find that $\hat{\theta} = (\hat{A_{max}},\hat{pK_{a}},\hat{\phi}) = (0.42,7.47,0.45)$

The pKa is in agreement with the theory, although to assess this rigorously (and trust the SEs) we should use the mixed model approach. Before that, we will try scipy least_squares() next.

```python

def pHAbsfun(theta,pH,Aobs):
    A = theta[0]/(1+np.exp((theta[1]-pH)/(theta[2])))
    res = A-Aobs
    return res

pHAbsdata_fun = lambda theta: pHAbsfun(theta,pH_data,ALED_data)

ls_result = least_squares(pHAbsdata_fun,[0.5,7.6,0.4])

```

```python
ls_result.x, ls_result.cost
```

    (array([0.42375714, 7.47007174, 0.45383623]), 0.03002231818680512)

The results between R's nls() and scipy least_squares() are in agreement for the coefficient values.

## Nonlinear Mixed Effect Model (R nlmer())

```python
print(base.summary(NLMMresult))
```

    Nonlinear mixed model fit by maximum likelihood  ['nlmerMod']
    Formula: ALED ~ nfunall(pH, Amax, pKa, phi) ~ Amax | Trial
       Data: data

         AIC      BIC   logLik deviance df.resid
      -497.3   -484.7    253.7   -507.3       86

    Scaled residuals:
        Min      1Q  Median      3Q     Max
    -2.9346 -0.6216 -0.0852  0.5278  4.2925

    Random effects:
     Groups   Name Variance  Std.Dev.
     Trial    Amax 0.0015279 0.03909
     Residual      0.0001855 0.01362
    Number of obs: 91, groups:  Trial, 4

    Fixed effects:
         Estimate Std. Error t value
    Amax  0.42990    0.01990   21.60
    pKa   7.46974    0.01498  498.76
    phi   0.46458    0.01289   36.04

    Correlation of Fixed Effects:
        Amax  pKa
    pKa 0.133
    phi 0.092 0.497

Based on the above, we can compute a z-score for $pK_{a}$ and $\phi$ to compare them to 7.6 and 1/log(10) respectively:

\begin{equation}

|z_{pKa}| = |\frac{7.469-7.6}{0.015}| =  8.69 \\

|z_{\phi}| = |\frac{0.4646 - 0.4343}{0.013}| = 2.33

\end{equation}

With a bonferroni correction for 2 tests assuming overall familywise error rate of $\alpha = 0.05$, the critical value for each test (per test $\alpha = 0.025$) occurs at $z_{crit} = 2.24$. Thus we reject both null hypotheses, and there is a significant difference obtained in our experiment vs the theoretical curve. However, this difference may not be practically significant, and as long as the results from our device are consistent, that is all that matters for calibrating the sensor.

Based on the above parameters for the NLMM, we can also simulate more values to obtain a larger dataset for the later parts involving PyTorch:

## pH-Absorbance Simulation Functions

```python
def generate_pHAbs(n,Amax=0.43,pKa=7.47,phi=0.46,sd_e=0.025):
    mean_pH,sd_pH = 7.6, 2.2
    min_pH, max_pH = 0, 14
    a,b = (min_pH - mean_pH)/sd_pH , (max_pH-mean_pH)/sd_pH
    pH = truncnorm.rvs(a,b,loc=mean_pH,scale=sd_pH,size=n)
    e = np.random.normal(loc=0,scale=sd_e,size=n)
    A = Amax / (1+(np.exp(pKa-pH))/phi) + e
    simdf = pd.DataFrame({'pH': pH,'ALED': A})
    return simdf

def generate_pHAbs_Trials(Trials,n,Amax=0.43,Asd=0.04,pKa=7.47,phi=0.46,sd_e=0.025):
    Amaxes = np.random.normal(Amax,Asd,Trials)
    simdfall = []
    for i in range(Trials):
        simdf = generate_pHAbs(n=n,Amax=Amaxes[i],pKa=pKa,phi=phi,sd_e=sd_e)
        simdf['Trial'] = i+1
        simdfall.append(simdf)
    simdfall = pd.concat(simdfall)
    return simdfall

```

# PyTorch pH-Absorbance Analysis

## pHAbsorbance Custom Layer

Below, we implement a custom layer that contains the 3 parameters and outputs the absorbance values. A random initialization is used as follows for the parameters (we set reasonable values as if we have not seen the above standard analysis):

\begin{equation}

A_{max} \sim N(1,0.2^{2}) \\

pK_{a} \sim N(7.6,0.5^{2}) \\

\phi \sim N(0.5,0.1^{2}) \\

\end{equation}

Notice that nn.Parameter() needs to be used on the weights so that PyTorch optimizer later on knows these are the custom parameters of the layer. Additionally, in the pHAbsLayer custom layer we initialize regularizers to 0, and instead choose to configure them when the pHAbsModel containing the layer is instantiated.

```python
import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader

```

```python
class pHAbsLayer(nn.Module):
    """Custom pHAbs Layer: Amax/(1+e^(pKa-pH)/phi)"""
    def __init__(self):
        super().__init__()
        weights = np.random.normal([1,7.6,0.5],[0.2,0.5,0.1]) #[Amax,pKa,phi]
        weights = torch.from_numpy(weights)
        self.weights = nn.Parameter(weights)
        self.regularizer = torch.zeros(3,dtype=torch.float64)

    def forward(self,x):
        y = self.weights[0]/(1+torch.exp((self.weights[1]-x)/self.weights[2]))
        return y
```

## pHAbsModel Model Class

Now that the pHAbsLayer() custom layer is created, we can use it like any other layer within the actual model class. In this class, we will also leave the option to set hyperparameters.

```python

class pHAbsModel(nn.Module):
    def __init__(self,lam_Amax=0,lam_pKa=0,lam_phi=0):
        super().__init__()
        self.f_pH = pHAbsLayer()
        self.f_pH.regularizer[0] = lam_Amax
        self.f_pH.regularizer[1] = lam_pKa
        self.f_pH.regularizer[2] = lam_phi

    def forward(self,x):
        return self.f_pH(x)
```

## pHAbs Dataset

Below, we create the Dataset class for the data. In this case it is relatively simple, the __getitem__ method should return the features (just pH) and label at a certain index and the __len__ method should return the total length of the dataset.

```python
class pHAbsDataset(Dataset):
    def __init__(self,pH,Abs):
        self.pH=pH.reshape(-1,1)
        self.Abs = Abs.reshape(-1,1)

    def __len__(self):
        return len(self.pH)

    def __getitem__(self,idx):
        return self.pH[idx],self.Abs[idx]

```

## Loss Penalty

As mentioned earlier in this notebook, we will be using an L1 penalty on the parameters' $\theta = (A_{max},pK_{a},\phi)$ deviations from $(0, 7.6, 0.43)$ respectively

```python
def penalty(model):
    weights = model.f_pH.weights
    regularizer = model.f_pH.regularizer
    prior = torch.Tensor([0,7.6,1/np.log(10)])
    penalty = (weights-prior).abs().dot(regularizer)
    return penalty
```

## Train and Test Loop

Below we define the training and testing loop. On the original dataset, we will use the full data to compare results to the first part and then later on will simulate data to compare train/test curves and effect of regularizers, etc.

```python

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        pred = model(X)
        loss = loss_fn(pred, y)
        pen = penalty(model)

        pen_loss = loss + pen
        # Backpropagation
        optimizer.zero_grad()
        pen_loss.backward()
        #loss.backward()
        optimizer.step()

        if batch % 10 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")
        return(loss)

def test_loop(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    test_loss, correct = 0, 0

    with torch.no_grad():
        for X, y in dataloader:
            pred = model(X)
            test_loss += loss_fn(pred, y).item()

    test_loss /= size
    print(f"Avg loss: {test_loss:>8f} \n")
    return(test_loss)

```

## Train Model on full original data

We now train the model on the full original data, and, since this dataset is small, we use a batch size of 91 which is all of the data.

Additionally, the Adam optimizer with a learning rate of 0.01 is used below, the model is trained for 1000 epochs. No regularization is applied for this first time on the full data.

For reference, we can extract the mean square error from the R nls() fit, which appears to be 0.00066

```python
residNLS = np.array(stats.residuals(NLSresult))
np.mean(np.square(residNLS))
```

    0.0006598311689431394

```python
origdataset = pHAbsDataset(pH_data,ALED_data)

origdataloader = DataLoader(origdataset,batch_size=91,shuffle=True)
```

```python

origmodel = pHAbsModel()
learning_rate = 0.01

loss_fn = nn.MSELoss()

optimizer = torch.optim.Adam(origmodel.parameters(), lr=learning_rate)

```

```python
%%capture

epochs = 1000
loss_orig = np.zeros(epochs)
Amax_orig = np.zeros(epochs)
pKa_orig = np.zeros(epochs)
phi_orig = np.zeros(epochs)

for i in range(epochs):
    print(f"Epoch {i+1}\n-------------------------------")
    loss_orig[i] = train_loop(origdataloader, origmodel, loss_fn, optimizer)
    Amax_orig[i] = origmodel.f_pH.weights[0]
    pKa_orig[i] = origmodel.f_pH.weights[1]
    phi_orig[i] = origmodel.f_pH.weights[2]

```

```python
plt.plot(loss_orig,"r-")
plt.title("Loss vs Epochs")
plt.ylabel("MSE Loss")
plt.xlabel("Epochs")
plt.show()
```

```python
loss_orig[-1]
```

    0.0006598443647888356

The above final loss is the same as the loss obtained for R's nls() function on this dataset. The parameter weights are also almost exactly the same as obtained via nls(), and thus solving the NLS problem via PyTorch tools was a success. Below we can examine the parameter traces vs epochs as well:

```python
origmodel.f_pH.weights
```

    Parameter containing:
    tensor([0.4240, 7.4712, 0.4544], dtype=torch.float64, requires_grad=True)

```python
plt.plot(Amax_orig,"y-")
plt.title("Amax vs Epochs")
plt.ylabel("Amax")
plt.xlabel("Epochs")
plt.show()
```

```python
plt.plot(pKa_orig,"m-")
plt.title("pKa vs Epochs")
plt.ylabel("pKa")
plt.xlabel("Epochs")
plt.show()
```

The pKa trace is interesting in that at first it started at a low value 7.35 and increased for some time until 7.85 before it started decreasing.

```python
plt.plot(phi_orig,"r-")
plt.title("phi vs Epochs")
plt.ylabel("phi")
plt.xlabel("Epochs")
plt.show()
```

The trace for the $\phi$ parameter shows an peak as well, although shorter indicating that at first this parameter was increasing briefly before it settled on the final value. This whole time, the loss was still decreasing, however.

## Experiment with regularization on original data

Below, we experiment with some regularization on the original data. The regularization parameters are $\lambda = (0.0001,0.001,0.01)$ for $(A_{max},pK_{a},\phi)$ respectively

```python
%%capture

origmodelreg = pHAbsModel(lam_Amax=0.0001,lam_pKa=0.001,lam_phi=0.01)
learning_rate = 0.01

loss_fn = nn.MSELoss()

optimizer = torch.optim.Adam(origmodelreg.parameters(), lr=learning_rate)

epochs = 1000
loss_origreg = np.zeros(epochs)

for i in range(epochs):
    print(f"Epoch {i+1}\n-------------------------------")
    loss_origreg[i] = train_loop(origdataloader, origmodelreg, loss_fn, optimizer)

```

```python
print(loss_origreg[-1])
origmodelreg.f_pH.weights
```

    0.0006840412471735585

    Parameter containing:
    tensor([0.4267, 7.4964, 0.4337], dtype=torch.float64, requires_grad=True)

As seen above, the parameters are closer to the prior values that were mentioned in the beginning of this notebook. Thus the regularization has worked.

We now move on to simulated data where we can also investigate the train-val curves to investigate phenomenon such as early stopping.

## Simulated Data

Below, we simulate 100 Trials with 100 points each for both a Training and Validation set. The true parameters in the training set are set to $A_{max,true} = 0.43,~~ pK_{a,true} = 7.47,~~ \phi_{true} =  0.46$.

To examine how distribution shift may affect the training/val curves, the true parameters in the validation set are set to $A_{max,true} = 0.40,~~ pK_{a,true} = 7.52,~~ \phi_{true} =  0.48$.

The noise in the absorbance value is $\epsilon \sim N(0, 0.025^{2})$

```python

np.random.seed(100)
TrainSim = generate_pHAbs_Trials(Trials=100,n=100)
np.random.seed(10)
ValSim = generate_pHAbs_Trials(Trials=100,n=100,Amax=0.40,pKa=7.52,phi=0.48)

```

```python
pH_Train, Abs_Train = TrainSim.pH.to_numpy(), TrainSim.ALED.to_numpy()
pH_Val,Abs_Val = ValSim.pH.to_numpy(), ValSim.ALED.to_numpy()
```

```python
TrainDS = pHAbsDataset(pH_Train,Abs_Train)
ValDS = pHAbsDataset(pH_Val,Abs_Val)

TrainLoader = DataLoader(TrainDS,batch_size=100,shuffle=True)
ValLoader = DataLoader(ValDS,batch_size=100,shuffle=True)
```

```python
%%capture

sim_model = pHAbsModel()
learning_rate = 0.01

loss_fn_train = nn.MSELoss()
loss_fn_val = nn.MSELoss(reduction="sum") #because test loop divides in the end

optimizer = torch.optim.Adam(sim_model.parameters(), lr=learning_rate)

epochs = 1000
loss_simtrain = np.zeros(epochs)
loss_simval = np.zeros(epochs)

for i in range(epochs):
    print(f"Epoch {i+1}\n-------------------------------")
    loss_simtrain[i] = train_loop(TrainLoader, sim_model, loss_fn_train, optimizer)
    loss_simval[i] = test_loop(ValLoader,sim_model,loss_fn_val)
```

```python
plt.plot(loss_simtrain,"b-")
plt.plot(loss_simval,"r-")
plt.legend(["Train","Val"])
plt.title("Loss vs Epochs")
plt.ylabel("MSE Loss")
plt.xlabel("Epochs")
plt.show()
```

```python
sim_model.f_pH.weights
```

    Parameter containing:
    tensor([0.4332, 8.2639, 0.9997], dtype=torch.float64, requires_grad=True)

```python
final_losstrain = loss_simtrain[-1]
final_lossval = loss_simval[-1]

print(f"The final training Loss is: {final_losstrain:.5f} and final validation Loss is: {final_lossval:.5f}")
```

    The final training Loss is: 0.00108 and final validation Loss is: 0.00124

This time, the $pK_{a} = 8.27,~~\phi = 1.01$ which are far from the true parameter values. We can check the answer with the wrapper from R's nls(), which confirms that this is just a result of the data obtained. The good news is that the validation loss and training loss are still about the same. The slight distribution shift did not appear to affect the results too much in this case.

```python
with localconverter(ro.default_converter + pandas2ri.converter):
  NLSTrainresult = ro.r.Fit_NLS(TrainSim)

print(base.summary(NLSTrainresult))

```

    Formula: ALED ~ Amax/(1 + exp((pKa - pH)/phi))

    Parameters:
         Estimate Std. Error t value Pr(>|t|)
    Amax 0.427711   0.001322   323.6   <2e-16 ***
    pKa  8.255887   0.009396   878.6   <2e-16 ***
    phi  1.002325   0.006636   151.1   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.0322 on 9997 degrees of freedom

    Number of iterations to convergence: 6
    Achieved convergence tolerance: 8.089e-08

## With Regularization

Now we will try the same thing as above with regularization and determine whether this ends up having a better test error. The same regularization parameters as earlier will be used. Ideally, cross validation or other hyperparameter selection methods would be used.

```python
%%capture

sim_modelreg = pHAbsModel(lam_Amax=0.0001,lam_pKa=0.001,lam_phi=0.01)
learning_rate = 0.01

loss_fn_train = nn.MSELoss()
loss_fn_val = nn.MSELoss(reduction="sum") #because test loop divides in the end

optimizer = torch.optim.Adam(sim_modelreg.parameters(), lr=learning_rate)

epochs = 1000
loss_simtrain = np.zeros(epochs)
loss_simval = np.zeros(epochs)

for i in range(epochs):
    print(f"Epoch {i+1}\n-------------------------------")
    loss_simtrain[i] = train_loop(TrainLoader, sim_modelreg, loss_fn_train, optimizer)
    loss_simval[i] = test_loop(ValLoader,sim_modelreg,loss_fn_val)
```

```python
plt.plot(loss_simtrain,"b-")
plt.plot(loss_simval,"r-")
plt.legend(["Train","Val"])
plt.title("Loss vs Epochs")
plt.ylabel("MSE Loss")
plt.xlabel("Epochs")
plt.show()
```

```python
final_losstrain = loss_simtrain[-1]
final_lossval = loss_simval[-1]

print(f"The final training Loss is: {final_losstrain:.5f} and final validation Loss is: {final_lossval:.5f}")
```

    The final training Loss is: 0.00233 and final validation Loss is: 0.00237

```python

```

In this case, the regularization resulted in both worse training and test error, this indicates we are over-regularizing the parameters. In the next run, the regularizers will be decreased

```python
%%capture

sim_modelreg = pHAbsModel(lam_Amax=1e-5,lam_pKa=1e-5,lam_phi=1e-3)
learning_rate = 0.01

loss_fn_train = nn.MSELoss()
loss_fn_val = nn.MSELoss(reduction="sum") #because test loop divides in the end

optimizer = torch.optim.Adam(sim_modelreg.parameters(), lr=learning_rate)

epochs = 1000
loss_simtrain = np.zeros(epochs)
loss_simval = np.zeros(epochs)

for i in range(epochs):
    print(f"Epoch {i+1}\n-------------------------------")
    loss_simtrain[i] = train_loop(TrainLoader, sim_modelreg, loss_fn_train, optimizer)
    loss_simval[i] = test_loop(ValLoader,sim_modelreg,loss_fn_val)
```

```python
final_losstrain = loss_simtrain[-1]
final_lossval = loss_simval[-1]

print(f"The final training Loss is: {final_losstrain:.5f} and final validation Loss is: {final_lossval:.5f}")
```

    The final training Loss is: 0.00112 and final validation Loss is: 0.00124

With a new less conservative choice of hyperparameters after experimenting, the training loss is higher as expected, and the validation loss is ever so slightly lower.


# Modeling and Simulation in Python

Case study

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter so figures appear in the notebook
%matplotlib inline

# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import functions from the modsim.py module
from modsim import *
```

## Yo-yo

Suppose you are holding a yo-yo with a length of string wound around its axle, and you drop it while holding the end of the string stationary.  As gravity accelerates the yo-yo downward, tension in the string exerts a force upward.  Since this force acts on a point offset from the center of mass, it exerts a torque that causes the yo-yo to spin.

This figure shows the forces on the yo-yo and the resulting torque.  The outer shaded area shows the body of the yo-yo.  The inner shaded area shows the rolled up string, the radius of which changes as the yo-yo unrolls.

In this model, we can't figure out the linear and angular acceleration independently; we have to solve a system of equations:

$\sum F = m a $

$\sum \tau = I \alpha$

where the summations indicate that we are adding up forces and torques.

As in the previous examples, linear and angular velocity are related because of the way the string unrolls:

$\frac{dy}{dt} = -r \frac{d \theta}{dt} $

In this example, the linear and angular accelerations have opposite sign.  As the yo-yo rotates counter-clockwise, $\theta$ increases and $y$, which is the length of the rolled part of the string, decreases.

Taking the derivative of both sides yields a similar relationship between linear and angular acceleration:

$\frac{d^2 y}{dt^2} = -r \frac{d^2 \theta}{dt^2} $

Which we can write more concisely:

$ a = -r \alpha $

This relationship is not a general law of nature; it is specific to scenarios like this where there is rolling without stretching or slipping.

Because of the way we've set up the problem, $y$ actually has two meanings: it represents the length of the rolled string and the height of the yo-yo, which decreases as the yo-yo falls.  Similarly, $a$ represents acceleration in the length of the rolled string and the height of the yo-yo.

We can compute the acceleration of the yo-yo by adding up the linear forces:

$\sum F = T - mg = ma $

Where $T$ is positive because the tension force points up, and $mg$ is negative because gravity points down.

Because gravity acts on the center of mass, it creates no torque, so the only torque is due to tension:

$\sum \tau = T r = I \alpha $

Positive (upward) tension yields positive (counter-clockwise) angular acceleration.

Now we have three equations in three unknowns, $T$, $a$, and $\alpha$, with $I$, $m$, $g$, and $r$ as known parameters.  It is simple enough to solve these equations by hand, but we can also get SymPy to do it for us.

```python
from sympy import init_printing, symbols, Eq, solve

init_printing()
```

```python
T, a, alpha, I, m, g, r = symbols('T a alpha I m g r')
```

```python
eq1 = Eq(a, -r * alpha)
```

```python
eq2 = Eq(T - m * g, m * a)
```

```python
eq3 = Eq(T * r, I * alpha)
```

```python
soln = solve([eq1, eq2, eq3], [T, a, alpha])
```

```python
soln[T]
```

```python
soln[a]
```

```python
soln[alpha]
```

The results are

$T      = m g I / I^*   $

$a      = -m g r^2 / I^* $

$\alpha = m g r / I^*    $

where $I^*$ is the augmented moment of inertia, $I + m r^2$.

You can also see [the derivation of these equations in this video](https://www.youtube.com/watch?v=chC7xVDKl4Q).

To simulate the system, we don't really need $T$; we can plug $a$ and $\alpha$ directly into the slope function.

```python
radian = UNITS.radian
m = UNITS.meter
s = UNITS.second
kg = UNITS.kilogram
N = UNITS.newton
```

newton

**Exercise:**  Simulate the descent of a yo-yo.  How long does it take to reach the end of the string?

I provide a `Params` object with the system parameters:

* `Rmin` is the radius of the axle.  `Rmax` is the radius of the axle plus rolled string.

* `Rout` is the radius of the yo-yo body.  `mass` is the total mass of the yo-yo, ignoring the string.

* `L` is the length of the string.

* `g` is the acceleration of gravity.

```python
params = Params(Rmin = 8e-3 * m,
                Rmax = 16e-3 * m,
                Rout = 35e-3 * m,
                mass = 50e-3 * kg,
                L = 1 * m,
                g = 9.8 * m / s**2,
                t_end = 1 * s)
```

      values

      Rmin
      0.008 meter

      Rmax
      0.016 meter

      Rout
      0.035 meter

      mass
      0.05 kilogram

      L
      1 meter

      g
      9.8 meter / second ** 2

      t_end
      1 second

Here's a `make_system` function that computes `I` and `k` based on the system parameters.

I estimated `I` by modeling the yo-yo as a solid cylinder with uniform density ([see here](https://en.wikipedia.org/wiki/List_of_moments_of_inertia)).

In reality, the distribution of weight in a yo-yo is often designed to achieve desired effects.  But we'll keep it simple.

```python
def make_system(params):
    """Make a system object.

    params: Params with Rmin, Rmax, Rout,
                              mass, L, g, t_end

    returns: System with init, k, Rmin, Rmax, mass,
                         I, g, ts
    """
    L, mass = params.L, params.mass
    Rout, Rmax, Rmin = params.Rout, params.Rmax, params.Rmin

    init = State(theta = 0 * radian,
                 omega = 0 * radian/s,
                 y = L,
                 v = 0 * m / s)

    I = mass * Rout**2 / 2
    k = (Rmax**2 - Rmin**2) / 2 / L / radian

    return System(params, init=init, I=I, k=k)
```

Testing `make_system`

```python
system = make_system(params)
```

      values

      Rmin
      0.008 meter

      Rmax
      0.016 meter

      Rout
      0.035 meter

      mass
      0.05 kilogram

      L
      1 meter

      g
      9.8 meter / second ** 2

      t_end
      1 second

      init
      theta               0 radian
omega    0.0 radi...

      I
      3.0625000000000006e-05 kilogram * meter ** 2

      k
      9.6e-05 meter / radian

```python
system.init
```

      values

      theta
      0 radian

      omega
      0.0 radian / second

      y
      1 meter

      v
      0.0 meter / second

Write a slope function for this system, using these results from the book:

$ r = \sqrt{2 k y + R_{min}^2} $

$ T      = m g I / I^*  $

$ a      = -m g r^2 / I^* $

$ \alpha  = m g r / I^*  $

where $I^*$ is the augmented moment of inertia, $I + m r^2$.

```python
# Solution

def slope_func(state, t, system):
    """Computes the derivatives of the state variables.

    state: State object with theta, omega, y, v
    t: time
    system: System object with Rmin, k, I, mass

    returns: sequence of derivatives
    """
    theta, omega, y, v = state
    g, k, Rmin = system.g, system.k, system.Rmin
    I, mass = system.I, system.mass

    r = sqrt(2*k*y + Rmin**2)
    alpha = mass * g * r / (I + mass * r**2)
    a = -r * alpha

    return omega, alpha, v, a
```

Test your slope function with the initial paramss.

```python
# Solution

slope_func(system.init, 0*s, system)
```

    (0.0 <Unit('radian / second')>,
     180.54116292458264 <Unit('1 / radian ** 0.5 / second ** 2')>,
     0.0 <Unit('meter / second')>,
     -2.888658606793322 <Unit('meter / radian / second ** 2')>)

Write an event function that will stop the simulation when `y` is 0.

```python
# Solution

def event_func(state, t, system):
    """Stops when y is 0.

    state: State object with theta, omega, y, v
    t: time
    system: System object with Rmin, k, I, mass

    returns: y
    """
    theta, omega, y, v = state
    return y
```

Test your event function:

```python
# Solution

event_func(system.init, 0*s, system)
```

1 meter

Then run the simulation.

```python
# Solution

results, details = run_ode_solver(system, slope_func, events=event_func, max_step=0.05)
details
```

      values

      sol
      None

      t_events
      [[0.879217870162702]]

      nfev
      134

      njev
      0

      nlu
      0

      status
      1

      message
      A termination event occurred.

      success
      True

Check the final state.  If things have gone according to plan, the final value of `y` should be close to 0.

```python
# Solution

results.tail()
```

      theta
      omega
      y
      v

      0.706147
      44.0182
      121.32
      0.327291
      -1.76573

      0.756147
      50.268
      128.609
      0.236982
      -1.84499

      0.806147
      56.8724
      135.496
      0.142962
      -1.91405

      0.856147
      63.8093
      141.885
      0.0457628
      -1.97198

      0.879218
      67.1147
      144.631
      9.02056e-17
      -1.99469

Plot the results.

`theta` should increase and accelerate.

```python
def plot_theta(results):
    plot(results.theta, color='C0', label='theta')
    decorate(xlabel='Time (s)',
             ylabel='Angle (rad)')
plot_theta(results)
```

`y` should decrease and accelerate down.

```python
def plot_y(results):
    plot(results.y, color='C1', label='y')

    decorate(xlabel='Time (s)',
             ylabel='Length (m)')

plot_y(results)
```

Plot velocity as a function of time; is the yo-yo accelerating?

```python
# Solution

v = results.v # m / s
plot(v)
decorate(xlabel='Time (s)',
         ylabel='Velocity (m/s)')
```

Use `gradient` to estimate the derivative of `v`.  How does the acceleration of the yo-yo compare to `g`?

```python
# Solution

a = gradient(v)
plot(a)
decorate(xlabel='Time (s)',
         ylabel='Acceleration (m/$s^2$)')
```

```python

```


# Regression

Regression is the process of approximating noisy data with functions.  The same numerical methods can also be used to approximate a complicated function with a simpler function.  We'll begin by looking at some limited cases in terms of linear algebra.

## Polynomial regression

Long ago (in the Linear Algebra notebook), we solved an over-determined linear system to compute a polynomial approximation of a function.
Sometimes we make approximations because we are interested in the polynomial coefficients.
That is usually only for very low order (like linear or quadratic fits).
Inferring higher coefficients is ill-conditioned (as we saw with Vandermonde matrices) and probably not meaningful.
We now know that Chebyshev bases are good for representing high-degree polynomials, but if the points are arbitrarily spaced, how many do we need for the Chebyshev approximation to be well-conditioned?

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')

def cosspace(a, b, n=50):
    return (a + b)/2 + (b - a)/2 * (
        np.cos(np.linspace(-np.pi, 0, n)))

def vander_chebyshev(x, n=None):
    if n is None:
        n = len(x)
    T = np.ones((len(x), n))
    if n > 1:
        T[:,1] = x
    for k in range(1,n-1):
        T[:,k+1] = 2 * x * T[:,k] - T[:,k-1]
    return T

def runge1(x):
    return 1 / (1 + 10*x**2)
```

```python
def chebyshev_regress_eval(x, xx, n):
    V = vander_chebyshev(x, n)
    Q, R = np.linalg.qr(V)
    return vander_chebyshev(xx, n) @ np.linalg.solve(R, Q.T)

xx = np.linspace(-1, 1, 100)
x = np.linspace(-1, 1, 80)
plt.plot(x, runge1(x), '.k')
plt.plot(xx, chebyshev_regress_eval(x, xx, 40) @ runge1(x), label='p(x)')
plt.plot(xx, runge1(xx), '-.', label='runge1(x)')
plt.legend(loc='upper left');
```

* What is the degree $k$ of the polynomial that is used?
* What distribution of points is used for the data?
* Would this have artifacts if we used a degree $k$ polynomial to interpolate noise-free samples at $k+1$ points using the same distribution?

```python
ns = np.geomspace(5, 1000, dtype=int)
conds = [np.linalg.cond(vander_chebyshev(np.linspace(-1, 1, n),
                                         int(n**(.5))))
         for n in ns]
plt.semilogy(ns, conds)
plt.xlabel('$n$')
plt.ylabel('cond');
```

* If we have $n$ data points, can we use a polynomial of degree $k = \lfloor n/5 \rfloor$?
* What about $k = n^{3/4}$?
* What expression $k = ?(n)$ appears to be sufficient?

## Noisy data

Regression really comes into its own when the data is noisy.

```python
def runge1_noisy(x, sigma):
    return runge1(x) + np.random.randn(*x.shape)*sigma

x = np.linspace(-1, 1, 500)
y = runge1_noisy(x, 0.2)
plt.plot(x, y, '.')
plt.plot(x, runge1(x), 'k', label='runge1(x)')
plt.plot(x, chebyshev_regress_eval(x, x, 7) @ y, label='regress(x)')
plt.legend(loc='upper left');
```

### Probability distributions

In order to interpret real data, we need a model for the noise.  We have chosen the most common and computationally convenient choice when creating the synthetic data above.
The function `randn` draws samples from the "standard normal" or Gaussian distribution.

$$ p(t) = \frac{1}{\sqrt{2\pi}} e^{-t^2/2} . $$

```python
def stdnormal(t):
    return np.exp(-t**2/2) / np.sqrt(2*np.pi)

n = 1000
w = np.random.randn(n)
plt.hist(w, bins=40, range=(-3,3), density=True)
t = np.linspace(-3, 3)
plt.plot(t, stdnormal(t))
plt.xlabel('$t$')
plt.ylabel('$p(t)$');
w.mean()
```

### Regression on noisy data

We can just go run our regression algorithm on the noisy data.

```python
x = np.linspace(-1, 1, 500)
y = runge1_noisy(x, sigma=0.1)
```

```python
plt.plot(x, y, '.')
plt.plot(xx, chebyshev_regress_eval(x, xx, 9) @ y, label='p(x)')
plt.plot(xx, runge1(xx), label='runge1(x)')
plt.legend(loc='upper left');
```

## What problem are we solving?

### Why do we call it a linear model?

We are currently working with algorithms that express the regression as a linear function of the model parameters.  That is, we search for coefficients $c = [c_0, c_1, \dotsc]^T$ such that

$$ V(x) c \approx y $$

where the left hand side is linear in $c$.  In different notation, we are searching for a predictive model

$$ f(x_i, c) \approx y_i \text{ for all $(x_i, y_i)$} $$

that is linear in $c$.

### Assumptions

1. The independent variables $x$ are error-free
1. The prediction (or "response") $f(x,c)$ is linear in $c$
1. The noise in the measurements $y$ is independent (uncorrelated)
1. The noise in the measurements $y$ has constant variance

There are reasons why all of these assumptions may be undesirable in practice, thus leading to more complicated methods.

### Loss functions

The error in a single prediction $f(x_i,c)$ of an observation $(x_i, y_i)$ is often measured as
$$ \frac 1 2 \big( f(x_i, c) - y_i \big)^2, $$
which turns out to have a statistical interpretation when the noise is normally distributed.
It is natural to define the error over the entire data set as
\begin{align} L(c; x, y) &= \sum_i \frac 1 2 \big( f(x_i, c) - y_i \big)^2 \\
&= \frac 1 2 \lVert f(x, c) - y \rVert^2
\end{align}
where I've used the notation $f(x,c)$ to mean the vector resulting from gathering all of the outputs $f(x_i, c)$.
The function $L$ is called the "loss function" and is the key to relaxing the above assumptions.

### Partial derivatives

Let's step back from optimization and consider how to differentiate a function of several variables.  Let $f(\boldsymbol x)$ be a function of a vector $\boldsymbol x$.  For example,

$$ f(\boldsymbol x) = x_0^2 + \sin(x_1) e^{3x_2} . $$

We can evaluate the **partial derivative** by differentiating with respect to each component $x_i$ separately (holding the others constant), and collect the result in a vector,

\begin{align}
\frac{\partial f}{\partial \boldsymbol x} &= \begin{bmatrix} \frac{\partial f}{\partial x_0} & \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} \end{bmatrix} \\
&= \begin{bmatrix} 2 x_0 & \cos(x_1) e^{3 x_2} & 3 \sin(x_1) e^{3 x_2} \end{bmatrix}.
\end{align}

Now let's consider a vector-valued function $\boldsymbol f(\boldsymbol x)$, e.g.,

$$ \boldsymbol f(\boldsymbol x) = \begin{bmatrix} x_0^2 + \sin(x_1) e^{3x_2} \\ x_0 x_1^2 / x_3 \end{bmatrix} . $$

and write the derivative as a matrix,

\begin{align}
\frac{\partial \boldsymbol f}{\partial \boldsymbol x} &=
\begin{bmatrix} \frac{\partial f_0}{\partial x_0} & \frac{\partial f_0}{\partial x_1} & \frac{\partial f_0}{\partial x_2} \\
\frac{\partial f_1}{\partial x_0} & \frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} \\
\end{bmatrix} \\
&= \begin{bmatrix} 2 x_0 & \cos(x_1) e^{3 x_2} & 3 \sin(x_1) e^{3 x_2} \\
x_1^2 / x_3 & 2 x_0 x_1 / x_3 & -x_0 x_1^2 / x_3^2
\end{bmatrix}.
\end{align}

#### Geometry of derivatives

* Handy resource on partial derivatives for matrices and vectors: https://explained.ai/matrix-calculus/index.html#sec3

#### Derivative of dot product

Let $f(\boldsymbol x) = \boldsymbol y^T \boldsymbol x = \sum_i y_i x_i$ and compute the derivative

$$ \frac{\partial f}{\partial \boldsymbol x} = \begin{bmatrix} y_0 & y_1 & \dotsb \end{bmatrix} = \boldsymbol y^T . $$

Note that $\boldsymbol y^T \boldsymbol x = \boldsymbol x^T \boldsymbol y$ and we have the product rule,

$$ \frac{\partial \lVert \boldsymbol x \rVert^2}{\partial \boldsymbol x} = \frac{\partial \boldsymbol x^T \boldsymbol x}{\partial \boldsymbol x} = 2 \boldsymbol x^T . $$

Also,
$$ \frac{\partial \lVert \boldsymbol x - \boldsymbol y \rVert^2}{\partial \boldsymbol x} = \frac{\partial (\boldsymbol x - \boldsymbol y)^T (\boldsymbol x - \boldsymbol y)}{\partial \boldsymbol x} = 2 (\boldsymbol x - \boldsymbol y)^T .$$

### Optimization
Given data $(x,y)$ and loss function $L(c; x,y)$, we wish to find the coefficients $c$ that minimize the loss, thus yielding the "best predictor" (in a sense that can be made statistically precise).  I.e.,
$$ \bar c = \arg\min_c L(c; x,y) . $$

It is usually desirable to design models such that the loss function is differentiable with respect to the coefficients $c$, because this allows the use of more efficient optimization methods.  Recall that our forward model is given in terms of the Vandermonde matrix,

$$ f(x, c) = V(x) c $$

and thus

$$ \frac{\partial f}{\partial c} = V(x) . $$

We can now differentiate our loss function
$$ L(c; x, y) = \frac 1 2 \lVert f(x, c) - y \rVert^2 = \frac 1 2 \sum_i (f(x_i,c) - y_i)^2 $$
term-by-term as
\begin{align} \nabla_c L(c; x,y) = \frac{\partial L(c; x,y)}{\partial c} &= \sum_i \big( f(x_i, c) - y_i \big) \frac{\partial f(x_i, c)}{\partial c} \\
&= \sum_i \big( f(x_i, c) - y_i \big) V(x_i)
\end{align}
where $V(x_i)$ is the $i$th row of $V(x)$.
Alternatively, we can take a more linear algebraic approach to write the same expression is
\begin{align} \nabla_c L(c; x,y) &= \big( f(x,c) - y \big)^T V(x) \\
&= \big(V(x) c - y \big)^T V(x) \\
&= V(x)^T \big( V(x) c - y \big) .
\end{align}
A necessary condition for the loss function to be minimized is that $\nabla_c L(c; x,y) = 0$.

* Is the condition sufficient for general $f(x, c)$?
* Is the condition sufficient for the linear model $f(x,c) = V(x) c$?
* Have we seen this sort of equation before?

##### Uniqueness
We can read the expression $ V(x)^T \big( V(x) c - y \big) = 0$ as saying that the residual $V(x) c - y$ is orthogonal to the range of $V(x)$.  Suppose $c$ satisfies this equation and $c' \ne c$ is some other value of the coefficients.  Then
\begin{align}
V(x)^T \big( V(x) c' - y \big) &= V(x)^T V(x) c' - V(x)^T y \\
&= V(x)^T V(x) c' - V(x)^T V(x) c \\
&= V(x)^T V(x) (c' - c) \ne 0
\end{align}
whenever $V(x)^T V(x)$ is nonsingular, which happens any time $V(x)$ has full column rank.

##### Gradient descent
Instead of solving the least squares problem using linear algebra (QR factorization), we could solve it using gradient descent.  That is, on each iteration, we'll take a step in the direction of the negative gradient.

```python
def grad_descent(loss, grad, c0, gamma=1e-3, tol=1e-5):
    """Minimize loss(c) via gradient descent with initial guess c0
    using learning rate gamma.  Declares convergence when gradient
    is less than tol or after 500 steps.
    """
    c = c0.copy()
    chist = [c.copy()]
    lhist = [loss(c)]
    for it in range(500):
        g = grad(c)
        c -= gamma * g
        chist.append(c.copy())
        lhist.append(loss(c))
        if np.linalg.norm(g) < tol:
            break
    return c, np.array(chist), np.array(lhist)

class quadratic_loss:
    """Test problem to give example of gradient descent."""
    def __init__(self):
        self.A = np.array([[1, 1], [1, 4]])
    def loss(self, c):
        return .5 * c @ self.A @ c
    def grad(self, c):
        return self.A @ c

def test(gamma):
    q = quadratic_loss()
    c, chist, lhist = grad_descent(q.loss, q.grad, .9*np.ones(2), gamma=gamma)
    plt.semilogy(lhist)
    plt.ylabel('loss')
    plt.xlabel('cumulative iterations')

    plt.figure()
    l = np.linspace(-1, 1)
    x, y = np.meshgrid(l, l)
    z = [q.loss(np.array([x[i,j], y[i,j]])) for i in range(50) for j in range(50)]
    plt.contour(x, y, np.reshape(z, x.shape))
    plt.plot(chist[:,0], chist[:,1], 'o-')
    plt.title('gamma={}'.format(gamma))

test(.45)
```

```python
class chebyshev_regress:
    def __init__(self, x, y, n):
        self.V = vander_chebyshev(x, n)
        self.y = y
        self.n = n

    def init(self):
        return np.zeros(self.n)

    def loss(self, c):
        r = self.V @ c - y
        return 0.5 * (r @ r)

    def grad(self, c):
        r = self.V @ c - y
        return self.V.T @ r

reg = chebyshev_regress(x, y, 6)
c, _, lhist = grad_descent(reg.loss, reg.grad, reg.init(), gamma=2e-3)
plt.semilogx(np.arange(1, 1+len(lhist)), lhist)
plt.ylabel('loss')
plt.xlabel('cumulative iterations')
plt.ylim(bottom=0);
```

```python
np.linalg.cond(reg.V.T @ reg.V)
```

    8.108087894474187

* How does changing the "learning rate" `gamma` affect convergence?
* How does changing the number of basis functions affect convergence rate?

#### Did we find the same solution?
We intend to solve the same problem as we previously solved using QR, so we hope to find the same solution.

```python
plt.plot(xx, chebyshev_regress_eval(x, xx, 6) @ y, '-k', label='QR')
plt.plot(xx, vander_chebyshev(xx, 6) @ c, '--', label='grad_descent')
plt.plot(x, y, '.')
plt.legend();
```

#### Observations

* This algorithm is kind of finnicky:
 * It takes many iterations to converge
 * The "learning rate" needs to be empirically tuned
 * When not tuned well, the algorithm can diverge
* It could be made more robust using a line search
* The $QR$ algorithm is a more efficient and robust way to solve these problems

## Nonlinear regression

Instead of the linear model
$$ f(x,c) = V(x) c = c_0 + c_1 \underbrace{x}_{T_1(x)} + c_2 T_2(x) + \dotsb $$
let's consider a rational model with only three parameters
$$ f(x,c) = \frac{1}{c_0 + c_1 x + c_2 x^2} = (c_0 + c_1 x + c_2 x^2)^{-1} . $$
We'll use the same loss function
$$ L(c; x,y) = \frac 1 2 \lVert f(x,c) - y \rVert^2 . $$

We will also need the gradient
$$ \nabla_c L(c; x,y) = \big( f(x,c) - y \big)^T \nabla_c f(x,c) $$
where
\begin{align}
\frac{\partial f(x,c)}{\partial c_0} &= -(c_0 + c_1 x + c_2 x^2)^{-2} = - f(x,c)^2 \\
\frac{\partial f(x,c)}{\partial c_1} &= -(c_0 + c_1 x + c_2 x^2)^{-2} x = - f(x,c)^2 x \\
\frac{\partial f(x,c)}{\partial c_2} &= -(c_0 + c_1 x + c_2 x^2)^{-2} x^2 = - f(x,c)^2 x^2 .
\end{align}

```python
class rational_regress:
    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.n = 3

    def init(self):
        return np.ones(self.n)

    def f(self, c):
        x = self.x
        return 1 / (c[0] + c[1]*x + c[2]*x**2)

    def df(self, c):
        x = self.x
        f2 = self.f(c)**2
        return np.array([-f2, -f2*x, -f2*x**2]).T

    def loss(self, c):
        r = self.f(c) - self.y
        return 0.5 * (r @ r)

    def grad(self, c):
        r = self.f(c) - self.y
        return r @ self.df(c)

reg = rational_regress(x, y)
c, _, lhist = grad_descent(reg.loss, reg.grad, reg.init(), gamma=2e-2)
plt.semilogx(np.arange(1, 1+len(lhist)), lhist)
plt.ylabel('loss')
plt.xlabel('cumulative iterations')
plt.ylim(bottom=0)
plt.title('rational c={}'.format(c));
```

```python
x = np.linspace(-1, 1, 500)
y = runge1_noisy(x, sigma=0.1)

c0 = np.array([1, 0, 1.])
c, _, lhist = grad_descent(reg.loss, reg.grad, c0, gamma=2e-2)
plt.semilogx(np.arange(1, 1+len(lhist)), lhist)
plt.ylabel('loss')
plt.xlabel('cumulative iterations')
plt.ylim(bottom=0)
plt.title('rational c={}'.format(c));
```

```python
plt.plot(x, y, '.')
plt.plot(xx, runge1(xx), label='runge1(x)')
plt.plot(xx, chebyshev_regress_eval(x, xx, 40) @ y, label='Polynomial')
plt.plot(xx, 1/(c[0] + c[1]*xx + c[2]*xx**2), '--k', label='Rational')
plt.legend();
```

#### Observations

* There can be local minima or points that look an awful lot like local minima.
* Convergence is sensitive to learning rate.
* It takes a lot of iterations to converge.
* A well-parametrized model (such as the rational model above) can accurately reconstruct the mean even with very noisy data.

### Outlook

* The optimization problem can be solved using a Newton method.  It can be onerous to implement the needed derivatives.
* The [Gauss-Newton method](https://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm) (see activity) is often more practical than Newton while being faster than gradient descent, though it lacks robustness.
* The [Levenberg-Marquardt method](https://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm) provides a sort of middle-ground between Gauss-Newton and gradient descent.
* Many globalization techniques are used for models that possess many local minima.
* One pervasive approach is stochastic gradient descent, where small batches (e.g., 1 or 10 or 20) are selected randomly from the corpus of observations (500 in our current example), and a step of gradient descent is applied to that reduced set of observations.  This helps to escape saddle points and weak local minima.
* Among expressive models $f(x,c)$, some may converge much more easily than others.  Having a good optimization algorithm is essential for nonlinear regression with complicated models, especially those with many parameters $c$.
* Classification is a very similar problem to regression, but the observations $y$ are discrete, thus

 * models $f(x,c)$ must have discrete output
 * the least squares loss function is not appropriate.
* [Why momentum really works](https://distill.pub/2017/momentum/)


# Homework 5

1. A group of $n \ge 4$ people are comparing their birthdays (as usual, assume their birthdays are independent, are not February 29, etc.).

    (a) Let $I_{ij}$ be the indicator r.v. of $i$ and $j$ having the same birthday (for $i \lt j$). Is $I_{12}$ independent of $I_{34}$? Are the $I_{ij}$'s independent?

    (b) Explain why the Poisson Paradigm is applicable here even for moderate $n$, and use it to get a good approximation to the probability of at least 1 match when $n=23$.

    (c) About how many people are needed so that there is a 50% chance (or better) that two either have the same birthday or are only 1 day apart? (Note that this is much harder than the birthday problem to do exactly, but the Poisson Paradigm makes it possible to get fairly accurate approximations quickly.)

#### Solution

##### 1 (a)

$I_{12}$ is independent of $I_{34}$, since knowing that persons 1 and 2 having the same birthday provides absolutely no information about persons 3 and 4.

However, $I_{ij}$ are not entirely independent, as knowing that $I_{12}$ and $I_{23}$ does tell us that person 1 and 3 _must_ have the same birthday.

##### 1 (b)

Checklist for the applying the Poisson Paradigm:

1. <input type=checkbox checked disabled> events $I_{ij}$ where $i \lt j$, $P(I_{ij}) = \frac{1}{365}$
2. <input type=checkbox checked disabled> large number of trials $\binom{n}{2}$; when $n=23$, $\binom{23}{2} = 253$
3. <input type=checkbox checked disabled> small $p = \frac{1}{365}$

As the events $I_{ij}$ are _weakly independent_, we can approximate this r.v. with a Poisson distribution.

Let $X \sim Pois(\mu)$.

\begin{align}
  \mu &= \binom{n}{2} ~ p \\
      &= \frac{n(n-1)}{2} ~ \frac{1}{365} \\
      &= \frac{n(n-1)}{730} \\
      &\approx 0.693 & \quad \text{when } n = 23
      \\\\
  \therefore P(X \ge 1) &= 1 - P(X = 0) \\
      &= 1 - e^{-0.693} ~ \frac{0.693^0}{0!} \\
      &= 1 - e^{-0.693} \\
      &\approx 0.500
\end{align}

##### 1 (c)

We are interested in the case where $I_{ij}$ is where 2 people have the _same_ birthday, or _1 day apart_.

This means that $I_{ij} = 1$, and so $P(I_{ij}=1) = \frac{3}{365}$.

We can approximate this with $Pois(\mu = \frac{n(n-1)}{2} ~ \frac{3}{365})$, and so $\mu = \frac{3n(n-1)}{730}$.

\begin{align}
 P(X \ge 1) &= 1 - P(X=0) \\
            &= 1 - e^{-\frac{3n(n-1)}{730}} ~ \frac{\left(\frac{3n(n-1)}{730}\right)^0}{0!} \\
            &= 1 - e^{\frac{-3n^2+3n}{730}}
            \\\\
            1 - e^{\frac{-3n^2+3n}{730}} &= \frac{1}{2} \\
            \frac{-3n^2+3n}{730} &= log\left(\frac{1}{2}\right) \\
            &= - log ~ 2 \\
            3n^2 -3n - 730 ~ log ~ 2 &= 0
\end{align}

----

```python
import math
math.log(2)
```

    0.6931471805599453

```python

```


# Tipos compósitos

[1] (i) ¿Cómo representarías a una partícula en 1D con posición, velocidad y masa en Julia?

(ii) ¿Cómo moverías la partícula en un paso $\delta t$?

(iii) ¿Si necesitas otra partícula con las mismas propiedades, qué harías?

(iv) Para $N$ tales partículas, ¿qué podrías hacer?

El problema aquí es que la representación del concepto "partícula" está repartida en distintas variables. Julia provee una manera de recolectar la información de un "objeto", al definir un *tipo compósito* ("composite type"):

```julia
type MiTipo
    a
    b::Int
end
```

Esto define un tipo de objeto llamado `MiTipo`. Cada objeto de este tipo tendrá *adentro* su propia copia de una variable llamada `a` y otra llamada `b`. En este caso, no hemos especificado ningún tipo para `a`, mientras que `b` está forzado a tener el tipo `Int`.

[En general, en Julia, para *anotar* a una variable con un tipo dado, usamos esta notación con `::`.]

[2] Define un tipo que se llama `Particula`, que tiene variables para la posición, velocidad y masa en una dimensión.

[3] Experimenta para ver cómo crear un objeto de tipo `Particula`.  [Pista: piensa en funciones]

[4] ¿Cómo podemos definir una función `mover` que mueve la partícula en un paso de tiempo $\delta t$? [Pista: Para especificar que un objeto `t` es de tipo `MiTipo`, usamos la sintaxis `t::MiTipo`.]

[5] Define un objeto `Gas` que representa $N$ partículas, así como una función `mover` que mueve el gas.

[6] Considera una estructura compuesta que denotaremos
${\overline v} = (f_v, d_v)$, que consta de dos campos `f_v` y `d_v`, que son flotantes. Esta estructura está definida de tal manera que se cumplen las siguientes propiedades:

\begin{align}
{\overline c} &= (c,\,0), &\textrm{ para toda constante $c$},\\
{\overline x} &= (x_0,\,1), &\textrm{para toda variable independiente $x = x_0$},\\
c {\overline v} &= {\overline c}\, {\overline v} = (c f_v,\, c d_v), \\
{\overline v} \pm {\overline w} & =  (f_v \pm f_w,\, d_v \pm d_w),\\
{\overline v} \cdot {\overline w} & =  (f_v \cdot f_w,\, f_v \cdot d_w + d_v \cdot f_w),\\
\frac{{\overline v}}{{\overline w}} & =
\left( \frac{f_v}{f_w},\, \frac{d_v \cdot f_w - f_v \cdot d_w}{f_w^2} \right),\\
{\overline u}^\alpha &= (f_u^\alpha,\, \alpha f_u^{\alpha-1} d_u), &\textrm{donde $\alpha$
es un número real}.
\end{align}

i. Implementa esto usando Julia.

ii. Define un polinomio $p(x)$ cuya variable independiente es $x$. Evalúa el polinomio en ${\overline x}$ (variable independiente $x$), en $x_0=0$. ¿Qué interpretación tiene *el valor* obtenido para $d_x$? Y si en lugar de un polinomio utilizas un cociente de polinomios $r({\overline x}) = p({\overline x}) / q({\overline x})$?

iii. Pensando en la interpretación que le diste a $d_x$, cómo definirías la acción sobre ${\overline x}$ de las siguientes funciones:
- $\exp(\,{\overline x}\,)$
- $\log(\,{\overline x}\,)$
- $\sin(\,{\overline x}\,)$
- $\cos(\,{\overline x}\,)$
- $\tan(\,{\overline x}\,)$

iv. ¿Cómo podemos definir las cosas en Julia de tal manera que ${\overline v} + c$, y las demás posibles operaciones
entre una variable compósita y un flotante $c$, tengan sentido?

[7] En el resto del curso, trataremos con *aritmética de intervalos*. En este nuevo tipo de aritmética, ocupamos intervalos $[a,b]$ de la recta real, que es el conjunto

$$[a, b] := \{x : a \le x \le b \}$$

(i) Define un tipo composito para representar un intervalo de dos números reales.

(ii) ¿Cómo podríamos tener operaciones sensatas sobre los intervalos? La idea básica es que el resultado de la operación sobre dos intervalos contenga los valores posibles resultantes de operar con los miembros de los dos intervalos respectivos.

(iii) Implementa estas operaciones, sin tomar en cuenta cuestiones de redondeo.

(iv) ¿Cómo nos puede ayudar el redondeo? Impleméntalo.

notebooks/06. Tipos compositos en Julia.ipynb