# Hartree-Fock Theory

**This assignment is due on Friday, January 24 at 5PM by email.**

This partial walkthrough assumes you have already familiar with the math of Hatree-Fock theory.

The Hartree-Fock method is typically used to solve the time-independent Schrodinger equation for a multi-electron atom or molecule as described in the Born-Oppenheimer approximation. Since there are no known solutions for many-electron systems (hydrogenic atoms and the diatomic hydrogen cation being notable one-electron exceptions), the problem is solved numerically.

## Theory
### The Closed-Shell Hartree-Fock Energy
The electronic energy in the MO basis
\begin{align}
E_{elec}^{clscf}  = &  E_{one} \ +\  E_{two}   \\
 = &  2 \ \sum_i^n \  h_{ii} \ +\  \sum_{ij}^n \  \biggl\{ 2(ii|jj) \ -\  (ij|ij) \biggr\} \\
 = &  2 \ \sum_i^n \ h_{ii} \ + \ \sum_{ij}^n \  \biggl\{ 2J_{ij} \ -\  K_{ij} \biggr\} \\
 = &  \sum_i^n \  \biggl( h_{ii} \ +\  F_{ii} \biggr) \\
\end{align}

The density matrix
\begin{equation}
D_{\mu\nu} = \sum_i^{\mathrm{d.o.}} C_\mu^i C_\nu^i
\end{equation}

The one-electron energy
\begin{align}
E_{one} = &   2 \ \sum_i^n \  h_{ii} \\
 = &  2 \ \sum_i^n \  \left\{ \sum_{ \mu \nu }^{AO} \  C_{\mu}^i C_{\nu}^i\ h_{ \mu \nu } \right\} \\
 = &  2 \ \sum_{ \mu \nu }^{AO} \  \left\{ \sum_i^n \  C_{\mu}^i C_{\nu}^i  \right\}  h_{ \mu \nu } \\
 = &  2 \ \sum_{ \mu \nu }^{AO} \  D_{ \mu \nu }   h_{ \mu \nu }
\end{align}

The two-electron energy
\begin{align}
E_{two} = &   \sum_{ij}^n \  \biggl\{ 2 (ii|jj) \ -\  (ij|ij) \biggr\} \\
 = &  \sum_{ij}^n \  \Biggl[ \sum_{ \mu \nu \rho \sigma }^{AO} \  \biggl\{ 2 C_{\mu}^i C_{\nu}^i C_{\rho}^j
C_{\sigma}^j\ ( \mu \nu | \rho \sigma )
\ -\  C_{\mu}^i C_{\nu}^j C_{\rho}^i C_{\sigma}^j\ ( \mu \nu | \rho \sigma ) \biggr\}  \Biggr]
 \nonumber  \\
   \\
 = &  \sum_{ \mu \nu \rho \sigma }^{AO} \  \Biggl[ \biggl\{ \sum_i^n \  C_{\mu}^i C_{\nu}^i \biggr\}
\biggl\{ \sum_j^n \  C_{\rho}^j C_{\sigma}^j \biggr\}
   \biggl\{  2 ( \mu \nu | \rho \sigma ) \ -\  ( \mu \rho | \nu \sigma ) \biggr\}
\Biggr] \\
 = & \sum_{ \mu \nu \rho \sigma }^{AO} \  D_{ \mu \nu } D_{ \rho \sigma }
    \biggl\{ 2 ( \mu \nu | \rho \sigma )
\ -\  ( \mu \rho | \nu \sigma ) \biggr\}
\end{align}

The electronic energy in the AO basis
\begin{align}
E_{elec}^{clscf} = & \ 2 \ \sum_{ \mu \nu }^{AO} \  D_{ \mu \nu } h_{ \mu \nu }
\ +\  \sum_{ \mu \nu \rho \sigma }^{AO} \  D_{ \mu \nu } D_{ \rho \sigma }
\biggl\{ 2 ( \mu \nu | \rho \sigma ) \ -\  ( \mu \rho | \nu \sigma ) \biggr\} \nonumber \\
   \\
 = &  \sum_{ \mu \nu }^{AO} \  D_{ \mu \nu }  \Biggl[   2 h_{ \mu \nu }
\ +\  \sum_{ \rho \sigma }^{AO} \  D_{ \rho \sigma }
  \biggl\{ 2 ( \mu \nu | \rho \sigma ) \ -\  ( \mu \rho | \nu \sigma ) \biggr\} \Biggr] \\
 = &  \sum_{ \mu \nu }^{AO} \  D_{ \mu \nu }   \biggl(   h_{ \mu \nu } \ +\  F_{ \mu \nu } \biggr)
\end{align}
\begin{align}
F_{ \mu \nu } = \ h_{ \mu \nu }
\ +\  \sum_{ \rho \sigma }^{AO} \  D_{ \rho \sigma }
  \biggl\{ 2 ( \mu \nu | \rho \sigma ) \ -\  ( \mu \rho | \nu \sigma ) \biggr\} \Biggr]
\end{align}

## Load Psi4 and NumPy
You will use Psi4 to compute integrals and NumPy to perform the linear algebra.

Input options will be provided by using a separate Python file `input.py`. Sample contents:

    Settings = dict()

    Settings["basis"] = "STO-3G"
    Settings["molecule"] = """
      0 1
      O
      H 1 R
      H 1 R 2 A
      R = 1.0
      A = 104.5
      symmetry c1
    """
    Settings["nalpha"] = 5
    Settings["nbeta"] = 5
    Settings["scf_max_iter"] = 50

```python
import psi4
import numpy as np
from input import Settings

print(Settings)

np.set_printoptions(suppress=True, linewidth=120)
```

    {'basis': 'STO-3G', 'molecule': '\n  0 1\n  O\n  H 1 R\n  H 1 R 2 A\n  R = 1.0\n  A = 104.5\n  symmetry c1\n', 'nalpha': 5, 'nbeta': 5, 'scf_max_iter': 50}

## Setup your initial conditions
- Construct a new molecule in C1 symmetry
- Load in the basis set
- Create a `MintsHelper` object to will compute the integrals. Refer to [this](http://www.psicode.org/psi4manual/master/api/psi4.core.MintsHelper.html) for grossly incomplete documentation.

```python
molecule = psi4.geometry(Settings['molecule'])
molecule.update_geometry()

ndocc = Settings['nalpha']

scf_max_iter = Settings['scf_max_iter']

print(molecule.nuclear_repulsion_energy())

basis = psi4.core.BasisSet.build(molecule, 'BASIS', Settings['basis'])

mints = psi4.core.MintsHelper(basis)
```

    8.801465564567374

## Compute the needed integrals

```python
# Overlap
S = mints.ao_overlap().np

# Kinetic
T = mints.ao_kinetic().np

# Potential
V = mints.ao_potential().np

# Two-electron repulsion
I = mints.ao_eri().np

print(I.shape)
```

    (7, 7, 7, 7)

## Form the one-electron Hamiltonian (H)
\begin{equation}
H_{\mu\nu} = T_{\mu\nu} + V_{\mu\nu}
\end{equation}

```python
H = T + V

print(H)
```

    [[-32.67360938  -7.60160704  -0.01741536   0.           0.          -1.57818038  -1.57818038]
     [ -7.60160704  -9.2914758   -0.20781331   0.          -0.          -3.47334591  -3.47334591]
     [ -0.01741536  -0.20781331  -7.50695983   0.           0.          -1.55452724  -1.55452724]
     [  0.           0.           0.          -7.42214005   0.           0.           0.        ]
     [  0.          -0.           0.           0.          -7.56362101   1.92474988  -1.92474988]
     [ -1.57818038  -3.47334591  -1.55452724   0.           1.92474988  -4.90357867  -1.42265626]
     [ -1.57818038  -3.47334591  -1.55452724   0.          -1.92474988  -1.42265626  -4.90357867]]

## Construct the orthogonalizer $S^{-1/2}$
The steps to complete this are:
- Diagonalize the S matrix
\begin{align}
& S L_S = L_S \Lambda_S \\
& L_S \tilde L_S = L_S L_S^{-1} = 1
\end{align}
- Form the $S^{-1/2}$ matrix
\begin{equation}
S^{-1/2} = L_S \Lambda_S^{-1/2} \tilde L_S
\end{equation}

Or use the Psi4 function  ```power``` which does all of this for you.

```python
A = mints.ao_overlap()
A.power(-0.5, 1.e-16)
A = A.np
print(A)
```

    [[ 1.02403592 -0.14128825 -0.00938732  0.          0.          0.02117023  0.02117023]
     [-0.14128825  1.21591378  0.09833518  0.         -0.         -0.27163956 -0.27163956]
     [-0.00938732  0.09833518  1.04984867  0.         -0.         -0.1381764  -0.1381764 ]
     [ 0.          0.          0.          1.         -0.         -0.         -0.        ]
     [ 0.         -0.         -0.         -0.          1.10025748  0.2134062  -0.2134062 ]
     [ 0.02117023 -0.27163956 -0.1381764  -0.          0.2134062   1.1823526  -0.08078394]
     [ 0.02117023 -0.27163956 -0.1381764  -0.         -0.2134062  -0.08078394  1.1823526 ]]

## Construct the initial density matrix
In this sample, we will be using a core Hamiltonian guess.  Perform the following steps:

- Form an initial (transformed) $F_0^\tau$ matrix using the H matrix.
\begin{equation}
F_0^\tau = \tilde{S}^{-1/2} H S^{-1/2}
\end{equation}
- Diagonalize the $F_0^\tau$ matrix using a standard eigenvalue routine.
\begin{equation}
F_0^\tau C_0^\tau = C_0^\tau \epsilon
\end{equation}
- Form the SCF eigenvector matrix
\begin{equation}
C = S^{-1/2} C_0^\tau
\end{equation}
- Form the first density matrix (D) using the doubly occupied orbitals from the eigenvector matrix
\begin{equation}
D_{\mu\nu} = \sum_{m}^{\mathrm{d.o.}} C_\mu^m C_\nu^m
\end{equation}

```python
Ft = A.dot(H).dot(A)
_, C = np.linalg.eigh(Ft)
C = A.dot(C)
Cocc = C[:, :ndocc]
D = np.einsum('pi,qi->pq', Cocc, Cocc)

print(D)
```

    [[ 1.06661727 -0.29755998 -0.02536219  0.          0.          0.04007352  0.04007352]
     [-0.29755998  1.32589965  0.15348429  0.         -0.         -0.17684186 -0.17684186]
     [-0.02536219  0.15348429  1.08573619 -0.         -0.         -0.11682694 -0.11682694]
     [ 0.          0.         -0.          1.         -0.         -0.          0.        ]
     [ 0.         -0.         -0.         -0.          1.17091368  0.1776736  -0.1776736 ]
     [ 0.04007352 -0.17684186 -0.11682694 -0.          0.1776736   0.05924138  0.00532125]
     [ 0.04007352 -0.17684186 -0.11682694  0.         -0.1776736   0.00532125  0.05924138]]

## SCF Iterations
Within the SCF iteration we must perform the following steps.
- Construct a new Fock matrix (F) include the two-electron repulsion integrals
\begin{equation}
F_{\mu\nu} = H_{\mu\nu} + \sum_{\rho\sigma}^{AO} D_{\rho\sigma} \left[ 2 (\mu\nu \mid \rho\sigma) - (\mu\rho \mid \nu\sigma) \right]
\end{equation}
- Calculate this iteration's electronic and total energies
\begin{equation}
E_{elec} = \sum_{\mu\nu}^{AO} D_{\mu\nu} \left( H_{\mu\nu} + F_{\mu\nu} \right)
\end{equation}
\begin{equation}
E_{total} = E_{elec} + E_{nuc}
\end{equation}
- Transform the Fock matrix
\begin{equation}
F^\tau = \tilde{S}^{-1/2} F S^{-1/2}
\end{equation}
- Diagonalize the Fock matrix
\begin{equation}
F^\tau C^\tau = C^\tau \epsilon
\end{equation}
- Construct the new SCF eigenvector matrix
\begin{equation}
C = S^{-1/2} C^\tau
\end{equation}
- Form the new density matrix
\begin{equation}
D_{\mu\nu} = \sum_{m}^{d.o.} C_\mu^m C_\nu^m
\end{equation}
- Test convergency of the density matrix and energy
\begin{equation}
rms = \left[ \sum_{\mu\nu}^{AO} \left( D_{\mu\nu}^n - D_{\mu\nu}^{n-1} \right)^2 \right]^{1/2} < \delta_1
\end{equation}
\begin{equation}
\Delta E = E_{SCF}^{n} - E_{SCF}^{n-1} < \delta_2
\end{equation}

```python
J = np.einsum('pqrs,rs->pq', I, D)
K = np.einsum('prqs,rs->pq', I, D)
F = H + J * 2 - K
print(F)
```

    [[-18.85371222  -4.88435988  -0.01396825  -0.          -0.          -1.02412265  -1.02412265]
     [ -4.88435988  -1.86077282  -0.21349587  -0.           0.          -0.71030031  -0.71030031]
     [ -0.01396825  -0.21349587   0.1682888    0.           0.          -0.21769452  -0.21769452]
     [ -0.          -0.           0.           0.25498122   0.           0.          -0.        ]
     [ -0.           0.           0.           0.           0.10983997   0.19510018  -0.19510018]
     [ -1.02412265  -0.71030031  -0.21769452   0.           0.19510018  -0.21971495  -0.25918679]
     [ -1.02412265  -0.71030031  -0.21769452  -0.          -0.19510018  -0.25918679  -0.21971495]]

With my code the ouput is the following:

    RHF iteration   1: energy   -73.25301168397348  dE -7.32530E+01
    RHF iteration   2: energy   -74.93149651086453  dE -1.67848E+00
    RHF iteration   3: energy   -74.96316771179171  dE -3.16712E-02
    RHF iteration   4: energy   -74.96447365205802  dE -1.30594E-03
    RHF iteration   5: energy   -74.96462787384583  dE -1.54222E-04
    RHF iteration   6: energy   -74.96465575231846  dE -2.78785E-05
    RHF iteration   7: energy   -74.96466118823379  dE -5.43592E-06
    RHF iteration   8: energy   -74.96466226898241  dE -1.08075E-06
    RHF iteration   9: energy   -74.96466248501822  dE -2.16036E-07
    RHF iteration  10: energy   -74.96466252827092  dE -4.32527E-08
    RHF iteration  11: energy   -74.96466253693451  dE -8.66359E-09
    RHF iteration  12: energy   -74.96466253867011  dE -1.73560E-09
    RHF iteration  13: energy   -74.96466253901779  dE -3.47683E-10
    RHF iteration  14: energy   -74.96466253908747  dE -6.96758E-11

## Test against Psi4

```python
# The energy that you obtain must match the energy that Psi4 produces (at least
# to the extent that you converged the energy). If you do not match then check
# your code for bugs!
psi4.set_options({'basis': 'sto-3g',
                  'scf_type': 'pk',
                  'e_convergence': 1e-10,
                 'reference': 'uhf'})
psi4.energy('scf')
```

    -74.96466253910498

```python

```


$$ \LaTeX \text{ command declarations here.}
\newcommand{\N}{\mathcal{N}}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\norm}[1]{\|#1\|_2}
\newcommand{\d}{\mathop{}\!\mathrm{d}}
\newcommand{\qed}{\qquad \mathbf{Q.E.D.}}
\newcommand{\vx}{\mathbf{x}}
\newcommand{\vy}{\mathbf{y}}
\newcommand{\vt}{\mathbf{t}}
\newcommand{\vb}{\mathbf{b}}
\newcommand{\vw}{\mathbf{w}}
\newcommand{\vm}{\mathbf{m}}
\newcommand{\I}{\mathbb{I}}
\newcommand{\th}{\text{th}}
$$

# EECS 445:  Machine Learning
## Hands On 08:  More on Naive Bayes (saving SVM for later...)
* Instructor:  **Ben Bray, Chansoo Lee, Jia Deng, Jake Abernethy**
* Date:  October 5, 2016

## Estimating the probability of a lead pipes Flint

Officials are investing \$27Million to dig up lead pipes in Flint MI. Before they spend \$5k on digging up the pipes for a home, they want a better estimate whether the pipe is lead. We have two observable variables, whether the home is *Old* (i.e. built before 1950) or not (built after 1950), and what the messy city records suggest. Keep in mind the city records are often wrong.

We make the "naive bayes" assumption that, given the target HasLead(X), the events IsOld(X) and RecordsSayLead(X) are independent of each other. Initially, the city believes the following parameters are roughly true:

\begin{align}
P(HasLead(X)) &= 0.4\\
P(IsOld(X) \mid HasLead(X)) &= 0.7\\
P(IsOld(X) \mid Not HasLead(X)) &= 0.3\\
P(RecordsSayLead(X) \mid HasLead(X)) &= 0.8\\
P(RecordsSayLead(X) \mid Not HasLead(X)) &= 0.5
\end{align}

Compute the probabilty:
$$P(HasLead(X) \mid IsOld(X), RecordsSayLead(X))$$
Now do the same for the other three conditions (i.e. conditioning on $ IsOld(X) \& Not RecordsSayLead(X)$, etc.)

## Solution:
We use Bayes Rule, then use the **Naive Bayes assumption**.

\begin{align}
P(HasLead(X) \mid IsOld(X), RecordsSayLead(X)) &=
\frac{P(IsOld(X), RecordsSayLead(X) \mid HasLead(X)) \cdot P(HasLead(X))}{P(IsOld(X), RecordsSayLead(X))} \\
&=
\frac{P(IsOld(X) \mid HasLead(X)) \cdot P(RecordsSayLead(X) \mid HasLead(X)) \cdot P(HasLead(X))}{P(IsOld(X), RecordsSayLead(X))} \\
&=
\frac{0.7 \cdot 0.8 \cdot 0.4}{P(IsOld(X), RecordsSayLead(X))}
\end{align}

We also have to marginalize to compute the denominator.

\begin{align}
P(IsOld(X), RecordsSayLead(X)) &= P(IsOld(X), RecordsSayLead(X), HasLead(X)) \\
& \quad \quad + P(IsOld(X), RecordsSayLead(X), Not HasLead(X))\\
&= P(IsOld(X), RecordsSayLead(X) \mid HasLead(X)) \cdot P(HasLead(X)) \\
& \quad \quad + P(IsOld(X), RecordsSayLead(X) \mid Not HasLead(X)) \cdot P(Not HasLead(X))\\
&= P(IsOld(X) \mid HasLead(X)) \cdot P(RecordsSayLead(X) \mid HasLead(X)) \cdot P(HasLead(X)) \\
& \quad \quad + P(IsOld(X) \mid Not HasLead(X)) \cdot P(RecordsSayLead(X) \mid Not HasLead(X)) \cdot P(Not HasLead(X))\\
&= 0.7 \cdot 0.8 \cdot 0.4 + 0.3 \cdot 0.5 \cdot (1-0.4)
\end{align}

To the final answer is $\frac{0.7 \cdot 0.8 \cdot 0.4}{0.7 \cdot 0.8 \cdot 0.4 + 0.3 \cdot 0.5 \cdot (1-0.4)} \approx 0.713$.

## Flint Starts Gathering Data, wants to update parameters

Over the past month, Flint has dug up about 200 service lines, and they've observed the pipe materials for several of these. They are starting to believe their initial estimates are incorrect.

They want to update these values, still assuming the Naive Bayes model. Here are the necessary parameters of this model:
\begin{align}
P(HasLead(X)) &= \pi_{\text{HasLead}} =  ?\\
P(IsOld(X) \mid HasLead(X)) & = \theta_{\text{HasLead}, \text{IsOld}} = ?\\
P(IsOld(X) \mid Not HasLead(X)) &= \theta_{\text{NotHasLead}, \text{IsOld}} = ?\\
P(RecordsSayLead(X) \mid HasLead(X)) &= \theta_{\text{HasLead}, \text{RecordsSayLead}} = ?\\
P(RecordsSayLead(X) \mid Not HasLead(X)) &= \theta_{\text{NotHasLead}, \text{RecordsSayLead}} = ?
\end{align}

Load the dataset and compute the **maximum likelihood** estimate for the above parameters.

```python
%pylab inline
import numpy as np
import pandas as pd
data = pd.read_csv('estimating_lead.csv')

# Run this to see a printout of the data
data
```

    Populating the interactive namespace from numpy and matplotlib

      Lead
      IsOld
      RecordSaysLead

      0
      Lead
      True
      True

      1
      Lead
      True
      True

      2
      Lead
      True
      True

      3
      Lead
      True
      True

      4
      Lead
      True
      True

      5
      NotLead
      True
      True

      6
      Lead
      True
      True

      7
      NotLead
      False
      False

      8
      Lead
      True
      True

      9
      Lead
      True
      True

      10
      Lead
      True
      True

      11
      NotLead
      False
      False

      12
      NotLead
      True
      False

      13
      Lead
      True
      False

      14
      Lead
      True
      True

      15
      Lead
      True
      True

      16
      Lead
      True
      True

      17
      Lead
      True
      True

      18
      Lead
      True
      False

      19
      Lead
      True
      True

      20
      Lead
      True
      True

      21
      NotLead
      False
      True

      22
      Lead
      True
      False

      23
      Lead
      True
      False

      24
      Lead
      True
      False

      25
      NotLead
      False
      True

      26
      NotLead
      False
      False

      27
      Lead
      True
      True

      28
      Lead
      True
      True

      29
      Lead
      False
      True

      ...
      ...
      ...
      ...

      170
      NotLead
      True
      True

      171
      NotLead
      True
      True

      172
      NotLead
      False
      False

      173
      Lead
      True
      True

      174
      Lead
      True
      True

      175
      Lead
      False
      True

      176
      NotLead
      True
      True

      177
      Lead
      True
      True

      178
      Lead
      True
      True

      179
      NotLead
      False
      False

      180
      Lead
      True
      True

      181
      Lead
      False
      True

      182
      NotLead
      True
      False

      183
      Lead
      True
      True

      184
      Lead
      True
      True

      185
      Lead
      True
      True

      186
      Lead
      False
      True

      187
      Lead
      True
      True

      188
      NotLead
      False
      False

      189
      Lead
      True
      True

      190
      NotLead
      False
      False

      191
      Lead
      True
      True

      192
      Lead
      True
      True

      193
      Lead
      True
      False

      194
      NotLead
      True
      False

      195
      Lead
      True
      True

      196
      Lead
      False
      True

      197
      Lead
      False
      True

      198
      Lead
      False
      True

      199
      Lead
      True
      True

200 rows × 3 columns

```python
# The object 'data' is a pandas DataFrame
# Don't worry if you don't know what that is, we can turn it into a numpy array
datamatrix = data.as_matrix()
```

## Solution

For the Naive Bayes model, computing the MLE estimate for the parameters is pretty easy because it can be reduced to estimating empirical frequencies (i.e. you just need to compute the "count" of how many times something occured in our dataset). Let's say we have $N$ examples in our dataset, and I'll use the notation $\#\{ \}$ to mean "size of set".

\begin{align}
\pi_{\text{HasLead}}^{\text{MLE}} &= \frac{\#\{HasLead(X_i)\}}{N} \\
\theta_{\text{HasLead},\text{IsOld}}^{\text{MLE}} &= \frac{\#\{HasLead(X_i) \wedge IsOld(X_i)\}}{\#\{HasLead(X_i)\}} \\
\theta_{\text{Not HasLead},\text{IsOld}}^{\text{MLE}} &= \frac{\#\{Not HasLead(X_i) \wedge IsOld(X_i)\}}{\#\{Not HasLead(X_i)\}} \\
\theta_{\text{HasLead},\text{RSL}}^{\text{MLE}} &= \frac{\#\{HasLead(X_i) \wedge RecordSaysLead(X_i)\}}{\#\{HasLead(X_i)\}} \\
\theta_{\text{Not HasLead},\text{RSL}}^{\text{MLE}} &= \frac{\#\{Not HasLead(X_i) \wedge RecordSaysLead(X_i)\}}{\#\{Not HasLead(X_i)\}} \\
\end{align}

```python
# We can use some pandas magic to do these counts quickly

N = data.shape[0]
params = {
    'pi_mle': data[data.Lead == 'Lead'].shape[0] / N,
    'theta_haslead_isold': data[(data.Lead == 'Lead') & (data.IsOld == True) ].shape[0] / \
                           data[data.Lead == 'Lead'].shape[0],
    'theta_nothaslead_isold': data[(data.Lead != 'Lead') & (data.IsOld == True) ].shape[0] / \
                           data[data.Lead != 'Lead'].shape[0],
    'theta_haslead_rsl': data[(data.Lead == 'Lead') & (data.RecordSaysLead == True) ].shape[0] / \
                           data[data.Lead == 'Lead'].shape[0],
    'theta_nothaslead_rsl': data[(data.Lead != 'Lead') & (data.RecordSaysLead == True) ].shape[0] / \
                           data[data.Lead != 'Lead'].shape[0],
}
print(pd.Series(params))
```

    pi_mle                    0.700000
    theta_haslead_isold       0.778571
    theta_haslead_rsl         0.900000
    theta_nothaslead_isold    0.383333
    theta_nothaslead_rsl      0.433333
    dtype: float64

## Putting a Prior on $\pi_{\text{HasLead}}$

For the case of the discreet event, such as material=Lead or =NoLead, we are working with a categorical distribution, i.e. a distrbution on one of $C$ things occuring. The parameters of this distribution are a probability vector $\pi \in \Delta_C$. (That is, $\pi_c \geq 0$ for all $c$, and $\sum_c \pi_c = 1$.)

Often when we have limited data, we want to add a prior distribution on our parameters. The standard prior to use is a *Dirichlet* with parameters $\alpha_1, \ldots, \alpha_C$. That is, we assume that $\pi \sim \mathrm{Dirichlet}(\alpha_1, \dots, \alpha_C)$. Recall that the Dirichlet has PDF $f(\pi_1, \dots, \pi_C) = \frac{1}{B(\alpha)} \prod_{c=1}^C \pi_c^{\alpha_c-1}$, where $B(\cdot)$ is just the normalizing term.

For our Flint problem, assume that the parameters $(\pi_{\text{HasLead}}, 1-\pi_{\text{HasLead}}) \sim \mathrm{Dirichlet}(3,3)$. Compute the MAP estimate of $\pi_{\text{HasLead}}$ for this distribution using the above dataset.

## Solution

For the Naive Bayes model, computing the MAP estimate is very similar to the MLE, but you have to add the Dirichlet prior parameters as "pseudocounts" to the frequency calculation. In this case we have two prior parameters $\alpha_{\text{HasLead}}$ and $\alpha_{\text{NotHasLead}}$, which we are choosing to be the value 3.

\begin{align}
\pi_{\text{HasLead}}^{\text{MAP}} &= \frac{\#\{HasLead(X_i)\} + \alpha_{\text{HasLead}} - 1 }{N + \alpha_{\text{HasLead}} - 1 + \alpha_{\text{NotHasLead}} - 1}
= \frac{\#\{HasLead(X_i)\} + 2 }{N + 4}
\end{align}


# Rudder equations

```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

```python
import sympy as sp
from sympy.plotting import plot as plot
from sympy.plotting import plot3d as plot3d
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
sp.init_printing()
from IPython.core.display import HTML,Latex
```

```python
import seaman_symbol as ss
from rudder_equations import *
from bis_system import BisSystem

from seaman_symbols import *
import seaman_symbol as ss

import sys
sys.path.append("../")
import seaman
```

## Coordinate system

## Symbols

```python
#HTML(ss.create_html_table(symbols=equations.total_sway_hull_equation_SI.free_symbols))
```

## Rudder equation
The rudder forces consist of mainly two parts, one that is
depending on the ship axial speed and one that is depending on the thrust.

The stalling effect is represented by a third degree term with a stall coefficient s.
The total expression for the rudder force is thus written as:

```python

rudder_equation_no_stall
```

If we also consider stall

```python
rudder_equation
```

## Effective rudder angle

```python
effective_rudder_angle_equation
```

```python
delta_e_expanded
```

### Speed dependent part

```python
Latex(sp.latex(rudder_u_equation))
```

### Thrust dependent part
This part is assumed to be proportional to the propeller thrust

```python
rudder_T_equation
```

```python
sp.latex(rudder_total_sway_equation)
```

```python
rudder_total_sway_equation_SI
```

## Rudder resistance
The rudder resistance is taken to be proportional to the rudder side force (without stall) and the
rudder angle, thus:

```python
rudder_drag_equation
```

```python
sp.latex(rudder_drag_equation_expanded)
```

```python
rudder_drag_equation_expanded_SI
```

## Rudder yawing moment

```python
rudder_yaw_equation
```

```python
rudder_yaw_equation_expanded_SI
```

## Rudder roll moment

```python
rudder_roll_equation
```

```python
rudder_roll_equation_expanded_SI = ss.expand_bis(rudder_roll_equation_expanded)
rudder_roll_equation_expanded_SI
```

## Lambda functions

```python
from rudder_lambda_functions import *
```

## Plotting effective rudder angle equation

```python
df = pd.DataFrame()
V = 5.0
beta = np.deg2rad(np.linspace(-10,10,20))
df['u_w'] = V*np.cos(beta)
df['v_w'] = -V*np.sin(beta)
df['delta'] = np.deg2rad(5)
df['r_w'] = 0.0
df['L'] = 50.0
df['k_r'] = 0.5
df['k_v'] = -1.0
df['g'] = 9.81
df['xx_rud'] = -1
df['l_cg'] = 0

result = df.copy()
result['delta_e'] = effective_rudder_angle_function(**df)
result['delta_e_deg'] = np.rad2deg(result['delta_e'])
result['beta_deg'] = np.rad2deg(beta)

result.plot(x = 'beta_deg',y = 'delta_e_deg');
```

```python
df = pd.DataFrame()
V = 5.0
beta = np.deg2rad(np.linspace(-10,10,20))
df['u_w'] = V*np.cos(beta)
df['v_w'] = -V*np.sin(beta)
df['delta'] = np.deg2rad(5)
df['r_w'] = 0.0
df['L'] = 50.0
df['k_r'] = 0
df['k_v'] = 0
df['g'] = 9.81
df['xx_rud'] = -1
df['l_cg'] = 0

result = df.copy()
result['delta_e'] = effective_rudder_angle_function(**df)
result['delta_e_deg'] = np.rad2deg(result['delta_e'])
result['beta_deg'] = np.rad2deg(beta)

result.plot(x = 'beta_deg',y = 'delta_e_deg');
```

```python
df = pd.DataFrame()
df['r_w'] = np.linspace(-0.3,0.3,20)
df['delta'] = 0.1
df['u_w'] = 5.0
df['v_w'] = 0.0
df['L'] = 50.0
df['k_r'] = 0.5
df['k_v'] = 0.5
df['g'] = 9.81
df['xx_rud'] = -1
df['l_cg'] = 0

result = df.copy()
result['delta_e'] = effective_rudder_angle_function(**df)

result.plot(x = 'r_w',y = 'delta_e');
```

## Plotting the total sway rudder equation

```python
df = pd.DataFrame()
df['delta'] = np.linspace(-0.3,0.3,10)
df['T_prop'] = 1.0
df['n_prop'] = 1.0
df['u_w'] = 5.0
df['v_w'] = 0.0
df['r_w'] = 0.0
df['rho'] = 1025
df['L'] = 1.0
df['k_r'] = 1.0
df['k_v'] = 1.0
df['g'] = 9.81
df['disp'] = 23.0
df['s'] = 0
df['Y_Tdelta'] = 1.0
df['Y_uudelta'] = 1.0
df['xx_rud'] = -1
df['l_cg'] = 0

result = df.copy()
result['fy'] = rudder_total_sway_function(**df)

result.plot(x = 'delta',y = 'fy');
```

### Plotting with coefficients from a real seaman ship model

```python
import generate_input
ship_file_path='test_ship.ship'
shipdict = seaman.ShipDict.load(ship_file_path)
```

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-35,35,20))
df['T_prop'] = 10*10**6
df['n_prop'] = 1
df['u_w'] = 5.0
df['v_w'] = 0.0
df['r_w'] = 0.0
df['rho'] = 1025
df['g'] = 9.81

df_input = generate_input.add_shipdict_inputs(lambda_function=rudder_total_sway_function,
                                              shipdict = shipdict,
                                              df = df,)
df_input
```

```python
result = df_input.copy()
result['fy'] = rudder_total_sway_function(**df_input)
```

```python
result.plot(x = 'delta',y = 'fy');
```

## Plotting the total rudder drag equation

```python
df = pd.DataFrame()
df['delta'] = np.linspace(-0.3,0.3,20)
df['T'] = 1.0
df['u_w'] = 5.0
df['v_w'] = 0.0
df['r_w'] = 0.0
df['rho'] = 1025
df['L'] = 1.0
df['k_r'] = 1.0
df['k_v'] = 1.0
df['g'] = 9.81
df['disp'] = 23.0
df['s'] = 0
df['Y_Tdelta'] = 1.0
df['Y_uudelta'] = 1.0
df['X_Yrdelta'] = -1.0
df['xx_rud'] = -1
df['l_cg'] = 0

result = df.copy()
result['fx'] = rudder_drag_function(**df)

result.plot(x = 'delta',y = 'fx');
```

Real seaman has a maximum effective rudder angle 0.61 rad for the rudder drag, which is why seaman gives different result for really large drift angles or yaw rates:

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-45,45,50))
df['T'] = 10*10**6
df['u_w'] = 5.0
df['v_w'] = 0.0
df['r_w'] = 0.0
df['rho'] = 1025
df['g'] = 9.81

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=rudder_drag_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label='fx',
                                                        seaman_function = run_real_seaman.calculate_static_ship_rudder)

fig,ax = plt.subplots()
result_comparison.plot(x = 'delta',y = ['fx','fx_seaman'],ax = ax)
ax.set_title('Rudder angle variation');
```

```python
df = pd.DataFrame()
df['v_w'] = (np.linspace(-10,10,20))
df['delta'] = 0
df['T'] = 10*10**6
df['u_w'] = 5.0
df['r_w'] = 0.0
df['rho'] = 1025
df['g'] = 9.81

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=rudder_drag_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label='fx',
                                                        seaman_function = run_real_seaman.calculate_static_ship_rudder)

fig,ax = plt.subplots()
result_comparison.plot(x = 'v_w',y = ['fx','fx_seaman'],ax = ax)
ax.set_title('Rudder drift angle variation');
```

```python
df = pd.DataFrame()
df['r_w'] = (np.linspace(-0.05,0.05,20))
df['delta'] = 0
df['T'] = 10*10**6
df['u_w'] = 5.0
df['v_w'] = 0.0
df['rho'] = 1025
df['g'] = 9.81

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=rudder_drag_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label='fx',
                                                        seaman_function = run_real_seaman.calculate_static_ship_rudder)

fig,ax = plt.subplots()
result_comparison.plot(x = 'r_w',y = ['fx','fx_seaman'],ax = ax)
ax.set_title('Rudder yaw rate variation');
```

## Plotting the rudder yawing moment equation

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-35,35,20))
df['T'] = 10*10**6
df['u_w'] = 5.0
df['v_w'] = 0.0
df['r_w'] = 0.0
df['rho'] = 1025
df['g'] = 9.81

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=rudder_yawing_moment_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label='mz',
                                                        seaman_function = run_real_seaman.calculate_static_ship)

fig,ax = plt.subplots()
result_comparison.plot(x = 'delta',y = ['mz','mz_seaman'],ax = ax)
ax.set_title('Rudder angle variation');
```

## Plotting the rudder roll moment equation

```python
df = pd.DataFrame()
df['delta'] = np.deg2rad(np.linspace(-35,35,20))
df['T'] = 10*10**6
df['u_w'] = 5.0
df['v_w'] = 0.0
df['r_w'] = 0.0
df['rho'] = 1025
df['g'] = 9.81

result_comparison = run_real_seaman.compare_with_seaman(lambda_function=rudder_roll_moment_function,
                                                        shipdict = shipdict,
                                                        df = df,
                                                        label='mx',
                                                        seaman_function = run_real_seaman.calculate_static_ship)

fig,ax = plt.subplots()
result_comparison.plot(x = 'delta',y = ['mx','mx_seaman'],ax = ax)
ax.set_title('Rudder angle variation');
```

```python
shipdict.rudder_particulars
```

```python
%connect_info
```

```python

```


```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np
```

### Extracting information about sequence quality and enrichment

### Enrichment

+ Often want to compare two datasets (tissue 1 vs. tissue 2; -drug vs. +drug; etc.)
+ Done by taking ratio of counts for sequences between data sets

$$f_{seq} = \frac{C_{seq}}{C_{total}}$$

The normalized frequency of a sequence $f_{seq}$ is determined by the number of counts of that sequence relative to all counts in the data set.

The enrichment of the sequence in dataset 2 vs. dataset 1 is given by:

$$E_{seq} = \frac{f_{seq,2}}{f_{seq,1}}$$

where $f_{seq,1}$ and $f_{seq,2}$ are the normalized frequencies of the sequence in dataset 1 and 2.

### How do we decide which sequences are high enough quality to include?

### For each cluster, you get a sequence of colors representing the sequence

+ Some bases are read well, others are ambiguous.

### The "Phred" score measures confidence in the base "call":

\begin{equation}
Q = -10log_{10}(p) \qquad \qquad \text{$(1)$}
\end{equation}

where $p$ is the probability that the call is **wrong**.

By rearranging Eq. $(1)$ above, we get:
\begin{equation}
p = 10^{- \frac{Q}{10}} \qquad \qquad \text{$(2)$}
\end{equation}

+ Create a plot of Q vs. p.  Is a high "Q" good or bad?

```python
p = np.arange(0.001,1,0.001)
plt.plot(p,-10*np.log10(p))
plt.title("High Q score is good")
```

#### Phred scores are encoded in last line:

```
@SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=60
GGGTGATGGCCGCTGCCGATGGCGTCAAATCCCACCAAGTTACCCTTAACAACTTAAGGG
+SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=60
IIIIIIIIIIIIIIIIIIIIIIIIIIIIII9IG9ICIIIIIIIIIIIIIIIIIIIIDIII
```

### Encoding goes like

|Letter | ASCII | $Q$ | $p$     |
|:-----:|:-----:|:---:| -------:|
|  `!`  | 33    | 0   | 1.00000 |
|  `"`  | 34    | 1   | 0.79433 |
|  `#`  | 35    | 2   | 0.63096 |
|  ...  | ...   | ... | ...     |
|  `J`  | 74    | 41  | 0.00008 |
|  `K`  | 75    | 42  | 0.00006 |

### python `chr` command converts integer ASCII to character

```python
print(chr(33))
print(chr(34))
print(chr(35))
print("...")
print(chr(74))
print(chr(75))
```

    !
    "
    #
    ...
    J
    K

### Can create dictionary that converts letters to quality scores

Recall that
\begin{equation}
p = 10^{- \frac{Q}{10}} \qquad \qquad \text{$(2)$}
\end{equation}

```python
Q_dict = {}
p_dict = {}
for i in range(33,76):
    Q_dict[chr(i)] = i-33
    p_dict[chr(i)] = 10**(-(Q_dict[chr(i)])/10.)

p_dict["K"]
```

    6.309573444801929e-05

### Example

$$p_{correct} = \prod_{i=1}^{L} (1-p_{incorrect})$$

where $i$ indexes along sequence and $L$ is sequence length.

```python
qual_string = "IIIIIIIIIIIIIIIIIIIIIIIIIIIIII9IG9ICIIIIIIIIIIIIIIIIIIIIDIII"

p_correct = 1.0
for q in qual_string:
    p_correct = p_correct*(1-p_dict[q])

print(p_correct)
```

    0.9857512130454114

### Modify this code to pull out the $p_{correct}$ each the sequence

```python
import gzip

get_line = False
seqs = {}
with gzip.open("files/example.fastq.gz") as f:
    for l in f:
        l_ascii = l.decode("ascii")
        if l_ascii[0] == "@":
            get_line = True
            continue
        if get_line:
            try:
                seqs[l_ascii.strip()] += 1
            except KeyError:
                seqs[l_ascii.strip()] = 1
            get_line = False
```

### Does sequence quality vary along the length of your reads?

```python
import gzip

Q_dict = {}
p_dict = {}
for i in range(33,76):
    Q_dict[chr(i)] = i-33
    p_dict[chr(i)] = 10**(-(Q_dict[chr(i)])/10.)

def prob_correct(qual_string):

    p_correct = 1.0
    out = []
    for q in qual_string:
        p_correct = p_correct*(1-p_dict[q])
        out.append(1-p_dict[q])

    return out, p_correct

get_line = 100
seqs = {}
x = np.zeros(75,dtype=float)
j = 0
with gzip.open("files/example.fastq.gz") as f:
    for l in f:
        l_ascii = l.decode("ascii")
        if l_ascii[0] == "@":
            get_line = 0
            j += 1
            continue
        if get_line == 1:
            try:
                seqs[l_ascii.strip()] += 1
            except KeyError:
                seqs[l_ascii.strip()] = 1
            get_line += 1
        elif get_line == 2:
            out, p = prob_correct(l_ascii.strip())
            x += np.array(out)
            #plt.plot(out)
            get_line = 100

        else:
            get_line += 1

plt.plot(x/j)
plt.ylim(0,1)
```

```python

```


# Non-linear Gaussian filtering and smoothing (discrete)

Provided is an example of a nonlinear state-space model on which one can perform Bayesian filtering and smoothing in order to obtain
a posterior distribution over a latent state trajectory based on noisy observations.
In order to understand the theory behind these methods in detail we refer to [1] and [2].

In order to perform Bayesian Filtering and Smoothing on non-linear models, we show how to use functionalities provided by ProbNum to
either linearise the respective model (Extended Kalman Filter, EKF) or to use an Unscented Transform (Unscented Kalman Filter, UKF) which does not require a Jacobian of the nonlinear model.

**References**:
> [1] Särkkä, Simo, and Solin, Arno. Applied Stochastic Differential Equations. Cambridge University Press, 2019.
>
> [2] Särkkä, Simo. Bayesian Filtering and Smoothing. Cambridge University Press, 2013.

```python
import numpy as np

import probnum as pn
from probnum import filtsmooth, randvars, randprocs
from probnum.problems import TimeSeriesRegressionProblem
```

```python
rng = np.random.default_rng(seed=123)
```

```python
# Make inline plots vector graphics instead of raster graphics
%matplotlib inline
from IPython.display import set_matplotlib_formats

set_matplotlib_formats("pdf", "svg")

# Plotting
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec

plt.style.use("../../probnum.mplstyle")
```

    /tmp/ipykernel_17168/236124620.py:5: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`
      set_matplotlib_formats("pdf", "svg")

## **Non-Linear Discrete** State-Space Model: Pendulum

---

For nonlinear models, we assume that at least one of the components of our state-space model (i.e. either the dynamics, the measurement model, or both) is a nonlinear function of the latent state.

Consider nonlinear functions $f: \mathbb{R}^d \rightarrow \mathbb{R}^d$ and $h: \mathbb{R}^d \rightarrow \mathbb{R}^m$ where $d$ is the dimension of the latent state and
$m$ is the dimension of the measurements.

For $k = 1, \dots, K$ and $x_0 \sim \mathcal{N}(\mu_0, \Sigma_0)$:

$$
\begin{align}
    \boldsymbol{x}_k &\sim \mathcal{N}(f(\boldsymbol{x}_{k-1}), \boldsymbol{Q}) \\
    \boldsymbol{y}_k &\sim \mathcal{N}(h(\boldsymbol{x}_k), \boldsymbol{R})
\end{align}
$$

This defines a dynamics model that assumes a state $x_k$ in a **discrete** sequence of states arising from a nonlinear map of the previous state $x_{k-1}$ corrupted with additive Gaussian noise under a **process noise** covariance matrix $Q$.
Similarly, the measurements $y_k$ are assumed to be nonlinear transformations of the latent state under additive Gaussian noise according to a **measurement noise** covariance $R$.
In the described case, we cannot use the Kalman Filtering equations since these assume linear relationships. One method is to linearise the nonlinear parts of the model by a first-order Taylor Expansion (Extended Kalman Filter, EKF) or employing the Unscented Kalman Filter (UKF) which is based on the unscented transform and does not require the computation of a Jacobian.
The `probnum` packages provides both methods by wrapping linearizing "Components" around nonlinear dynamcis and/or measurement models, as we will see below.

Note that this can be generalized to a nonlinear time-varying state-space model, as well. Then we would have $f: \mathbb{T} \times \mathbb{R}^d \rightarrow \mathbb{R}^d$ and $h: \mathbb{T} \times \mathbb{R}^d \rightarrow \mathbb{R}^m$ where $\mathbb{T}$ is the "time dimension".

### Define State-Space Model

#### I. Discrete Dynamics Model: Nonlinear Gaussian Transitions

```python
state_dim = 2
observation_dim = 1
```

```python
# approx. gravitational constant
g = 9.81
# dt
delta_t = 0.0075

def pendulum_rhs(state):
    """Right-hand side of an ODE that defines pendulum dynamics"""
    x1, x2 = state
    y1 = x1 + x2 * delta_t
    y2 = x2 - g * np.sin(x1) * delta_t
    return np.array([y1, y2])

def pendulum_jacobian(state):
    """Jacobian of the pendulum ODE"""
    x1, x2 = state
    dy1_dx = [1.0, delta_t]
    dy2_dx = [-g * np.cos(x1) * delta_t, 1.0]
    return np.array([dy1_dx, dy2_dx])

dynamics_transition_function = lambda t, state: pendulum_rhs(state)
dynamics_transition_jacobian_function = lambda t, state: pendulum_jacobian(state)

dynamics_diffusion_matrix = 1.0 * (
    np.diag(np.array([delta_t ** 3 / 3, delta_t]))
    + np.diag(np.array([delta_t ** 2 / 2]), 1)
    + np.diag(np.array([delta_t ** 2 / 2]), -1)
)
```

We here define the non-linear dynamics of a pendulum.

```python
noise = randvars.Normal(
    mean=np.zeros(dynamics_diffusion_matrix.shape[0]), cov=dynamics_diffusion_matrix
)

# Create discrete, non-linear Gaussian dynamics model
dynamics_model = randprocs.markov.discrete.NonlinearGaussian(
    input_dim=state_dim,
    output_dim=state_dim,
    transition_fun=dynamics_transition_function,
    noise_fun=lambda t: noise,
    transition_fun_jacobian=dynamics_transition_jacobian_function,
)
```

#### II. Discrete Measurement Model: Nonlinear Gaussian Measurements

```python
def pendulum_measurement(state):
    x1, x2 = state
    return np.array([np.sin(x1)])

def pendulum_measurement_jacobian(state):
    x1, x2 = state
    return np.array([[np.cos(x1), 0.0]])

measurement_function = lambda t, state: pendulum_measurement(state)
measurement_jacobian_function = lambda t, state: pendulum_measurement_jacobian(state)

measurement_variance = 0.32 ** 2
measurement_covariance = measurement_variance * np.eye(observation_dim)
```

```python
measurement_jacobian_function(0, np.ones(2)) @ np.ones(2)
```

    array([0.54030231])

```python
# Create discrete, non-linear Gaussian measurement model
measurement_noise = randvars.Normal(mean=np.zeros(measurement_covariance.shape[0]), cov=measurement_covariance)
measurement_model = randprocs.markov.discrete.NonlinearGaussian(
    input_dim=state_dim,
    output_dim=observation_dim,
    transition_fun=measurement_function,
    noise_fun=lambda t: measurement_noise,
    transition_fun_jacobian=measurement_jacobian_function,
)

```

#### III. Initial State Random Variable

```python
mu_0 = np.ones(state_dim)
sigma_0 = measurement_variance * np.eye(state_dim)
initial_state_rv = randvars.Normal(mean=mu_0, cov=sigma_0)
```

### Kalman Filtering

#### I. Linearize model (Extended Kalman Filter)

Since we could easily derive the Jacobian of our nonlinear model components above, we can use the Extended Kalman Filter (EKF) to obtain a filtering posterior over the states. Therefore, in `probnum` we just wrap the dynamics model and the measurement model in a `DiscreteEKFComponent` object.
In cases in which the Jacobian is not as easy to write down, `probnum` also provides the `DiscreteUKFComponent` class that implements the linearization by means of an
_unscented Kalman filter_ that does not require a Jacobian.
Alternatively, the Jacobian can be calculated via automatic differentiation.

```python
linearised_dynamics_model = filtsmooth.gaussian.approx.DiscreteEKFComponent(
    dynamics_model
)

```

```python
prior_process = randprocs.markov.MarkovSequence(
    transition=linearised_dynamics_model, initrv=initial_state_rv, initarg=0.0
)
```

#### II. Kalman Filter

Then, we simply pass these linearized models to the Kalman Filter (exactly the same interface as in the linear case) and proceed.

```python
kalman_filter = filtsmooth.gaussian.Kalman(prior_process)
```

#### III. Generate Data for the State-Space Model

`statespace.generate_artificial_measurements()` is used to sample both latent states and noisy observations from the specified state space model.

```python
time_grid = np.arange(0.0, 5.0, step=delta_t)
```

```python
latent_states, observations = randprocs.markov.utils.generate_artificial_measurements(
    rng=rng,
    prior_process=prior_process,
    measmod=measurement_model,
    times=time_grid,
)
regression_problem = TimeSeriesRegressionProblem(
    observations=observations,
    locations=time_grid,
    measurement_models=[
        filtsmooth.gaussian.approx.DiscreteEKFComponent(measurement_model)
    ]
    * len(time_grid),
)
```

#### IV. Perform Kalman Filtering + Rauch-Tung-Striebel Smoothing

```python
state_posterior, _ = kalman_filter.filtsmooth(regression_problem)
```

The method `filtsmooth` returns a `KalmanPosterior` object which provides convenience functions for e.g. sampling and interpolation.
We can also extract the just computed posterior smoothing state variables.
This yields a list of Gaussian Random Variables from which we can extract the statistics in order to visualize them.

```python
posterior_state_rvs = (
    state_posterior.states
)  # List of <num_time_points> Normal Random Variables
posterior_state_means = posterior_state_rvs.mean  # Shape: (num_time_points, state_dim)
posterior_state_covs = (
    posterior_state_rvs.cov
)  # Shape: (num_time_points, state_dim, state_dim)
```

### Visualize Results

```python
state_fig = plt.figure()
state_fig_gs = gridspec.GridSpec(ncols=2, nrows=1, figure=state_fig)

ax_00 = state_fig.add_subplot(state_fig_gs[0, 0])
ax_01 = state_fig.add_subplot(state_fig_gs[0, 1])

# Plot means
mu_x_1, mu_x_2 = [posterior_state_means[:, i] for i in range(state_dim)]

ax_00.plot(time_grid, mu_x_1, label="posterior mean")
ax_01.plot(time_grid, mu_x_2)

# Plot marginal standard deviations
std_x_1, std_x_2 = [np.sqrt(posterior_state_covs[:, i, i]) for i in range(state_dim)]

ax_00.fill_between(
    time_grid,
    mu_x_1 - 1.96 * std_x_1,
    mu_x_1 + 1.96 * std_x_1,
    alpha=0.2,
    label="1.96 marginal stddev",
)
ax_01.fill_between(
    time_grid, mu_x_2 - 1.96 * std_x_2, mu_x_2 + 1.96 * std_x_2, alpha=0.2
)

# Plot 5 samples on a subsampled grid
samples = state_posterior.sample(rng=rng, t=state_posterior.locations[::5], size=5)
for smp in samples:
    ax_00.plot(
        state_posterior.locations[::5],
        smp[:, 0],
        color="gray",
        alpha=0.75,
        linewidth=1,
        linestyle="dashed",
        label="sample",
    )
    ax_01.plot(
        state_posterior.locations[::5],
        smp[:, 1],
        color="gray",
        alpha=0.75,
        linewidth=1,
        linestyle="dashed",
        label="sample",
    )

# Plot groundtruth
ax_00.scatter(time_grid, observations, marker=".", label="measurements")

# Add labels etc.
ax_00.set_xlabel("t")
ax_01.set_xlabel("t")

ax_00.set_title(r"$x_1$")
ax_01.set_title(r"$x_2$")

# The following two lines are just to remove duplicate labels (caused by samples) from the legend
handles, labels = ax_00.get_legend_handles_labels()
by_label = dict(zip(labels, handles))

state_fig.legend(
    by_label.values(), by_label.keys(), loc="center left", bbox_to_anchor=(1, 0.5)
)

state_fig.tight_layout()
```


```python
import time
import itertools
from importlib import reload
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import gen_exp
import utils as ut
```

```python
np.random.seed(42)
```

```python
# matplotlib parameters for figures
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.rcParams.update({'font.size': 10, 'font.serif': ['Bookman']})
```

# 1. Binary classification
Let $(X,Y)$ be a tuple of random variables on $\mathcal{X} \times \mathcal{Y}$ with  $\mathcal{Y} = \left \{ -1, +1\right \}$ that follow a probability law $\mathbb{P}_{X,Y}$. The **Bayes optimal scorer** is defined as $ \eta : x \mapsto \mathbb{P} \left (Y=+1 \; | \; X=x \right )$. Define $\mathbb{P}_X$ as the marginal law of the variable $X$ and $f$ the density associated to this law. Then the tuple $(f,\eta)$ completely determines our density $\mathbb{P}_{X,Y}$.

In binary classification, the loss of a classifier $g$ is defined as $\mathcal{L} (g) := \mathbb{P}\left ( g(X) \ne Y \right ) $.
The best classifier according to this loss is $g^*: x \mapsto  2 \mathbb{I}_{\eta(x) > 1/2 }-1$. It is called the **Bayes classifier**.
Given $n$ i.i.d. realizations $(x_i,y_i)_{i=1}^n$ of $(X,Y)$ we learn a classifier $\hat{g}$ that minimize the empirical loss $\widehat{\mathcal{L}} (g) := \sum_{i=1}^n \mathbb{I} \left \{ g(X_i) \ne Y_i \right \}$ on a function class $\mathcal{G}$. The **generalization error** is defined as $R(\hat{g})$ where $R(g) = \mathcal{L}(g ) - \mathcal{L}(g^*) \ge 0$ is the **regret**.

Let $\alpha \in [0,1)$, we introduce the **Mammen-Tsybakov condition**:
$$
\begin{align}
\exists B > 0, \; \forall t \ge 0, \; \mathbb{P} \left ( |2  \eta (X) - 1 | \le t  \right )  \le B t^{\frac{\alpha}{1-\alpha}}.
\end{align}
$$
In the general case, the generalization error can be show to be of order $n^{-1/2}$ with great probability.
Under this condition, the generalization error can be shown to be of the order $n^{\frac{-1}{2-\alpha}}$ with great probability.

We assume that $\mathcal{X} = [0,1]$ and that $f = 1$ and finally that $\eta$ is increasing. Then
$$
\begin{align}
\mathbb{P} \left ( |2  \eta (X) - 1 | \le t  \right ) &= \mathbb{P} \left ( (1-t)/2 \le \eta (X) \le (1+t)/2  \right )\\
 &=
\int_\mathcal{X} \mathbb{I} \left \{ (1-t)/2 \le \eta(x) \le (1+t)/2  \right \} dx \\
&= \eta^{-1} \left ( [1+t]/2\right)  - \eta^{-1} \left( [1-t]/2\right ) \\
&\approx t \left(\eta^{-1}  \right )' \left ( 1/2\right) +
\frac{t^3}{12} \left(\eta^{-1}  \right )^{(3)} \left ( 1/2\right)  + o_0(t^3) \\
\end{align}
$$
The last equality is only useful close to zero and true if $\eta^{-1}$ is sufficiently regular.

This decomposition basically shows that to have fast generalization speeds, one needs to choose smartly the derivative of $\eta$ close to $1/2$.

Introducing
* $\eta_a(x) =
\begin{cases}
 \frac{1}{2}  + \frac{1}{2} \left ( 2x - 1 \right )^{\frac{1-a}{a}} \quad & \text{if } x \in [1/2,1], \\
 \frac{1}{2}  - \frac{1}{2} \left ( 1 - 2x \right )^{\frac{1-a}{a}} \quad & \text{otherwise.}
\end{cases}$

then

* $\eta_a^{-1}(x) =
\begin{cases}
 \frac{1}{2}  + \frac{1}{2} \left ( 2x - 1 \right )^{\frac{a}{1-a}} \quad & \text{if } x \in [1/2,1], \\
 \frac{1}{2}  - \frac{1}{2} \left ( 1 - 2x \right )^{\frac{a}{1-a}} \quad & \text{otherwise.}
\end{cases}$

and we have that
* $\mathbb{P} \left ( |2  \eta_a (X) - 1 | \le t  \right ) = t^{\frac{a}{1-a}}.$

which shows that $(1.,\eta_a)$ determines our distribution, then the Mammen-Tsybakov condition is verified.

```python
a_collection = np.linspace(0., 1., 6)[1:-1]
ut.plot_eta(a_collection, gen_exp.eta_a)
```

```python
ut.plot_distribution(a_collection, gen_exp.datagen_eta_a)
```

We define $g_a^*$ as the classifier based on $\eta_a$, i.e. $g_a^*(x) = I_{\eta_a(x) > 1/2}$, then, with $L_a$ the risk computed when the data is generated with $\eta = \eta_a$,
$$
\begin{align*}
L_a(g_a) &=\frac{1-a}{2}  \\
\end{align*}
$$

Since our data is 1-dimensional, we assume that we learn a frontier $f$, such that if a data point $x \in \mathbb{R}$ verifies $x>f$ it is positive and negative otherwise.
The classifier $g_{f}$ is defined this way, then:
$$
\begin{align*}
L_a(g_{f})  &= L_a(g_a) + \frac{a}{2} |2f-1 |^{a^{-1}}.
\end{align*}
$$

```python
n_list = [int(np.power(2, i)) for i in range(6, 10)]
```

```python
df_classif = gen_exp.classification_experiments(n_list, a_collection, n_exps=1000, verbose=True)
```

    Done   0 %
    Done   5 %
    Done  10 %
    Done  15 %
    Done  20 %
    Done  25 %
    Done  30 %
    Done  35 %
    Done  40 %
    Done  45 %
    Done  50 %
    Done  55 %
    Done  60 %
    Done  65 %
    Done  70 %
    Done  75 %
    Done  80 %
    Done  85 %
    Done  90 %
    Done  95 %

```python
ut.boxplot_slopes(df_classif)
```

```python
ut.slopes_quants(df_classif)
```

```python
df_classif.to_csv("results_classif.csv", index = False)
```

The value of the experiment slope is obtained from regression on quantiles obtained with 1000 experiences.
These points are not random and show that by designing a function that satisfies the Mammen-Tsybakov function, we confirm the theoretical results.

# 2. Pointwise ROC optimization in bipartite ranking

**It is advised to have read and understood section 5.2 of:** * *

Clémençon, S. & Vayatis, N.*Overlaying classifiers: a practical approach for optimal ranking.*
Constructive Approximation, Advances in Neural Information Processing Systems 21.

** before reading this section. **

Bipartite ranking seeks to learn a score function that orders observations from the most likely to the least likely to be positive. An optimal solution is to take the order induced by the Bayes optimal scorer $\eta$.

Pointwise ROC optimization consists in findind a set $R$ such that
$$
\sup_{R \in \mathcal{R}} G(R) \qquad \text{subject to}\qquad H (R) \le \alpha + \phi,
$$
where $G(R) = \mathbb{P} (X \in R | Y=+1)$ and $H(R) = \mathbb{P} (X \in R | Y=-1)$.

Let $G_n$, $H_n$ be the empirical counterparts of $G$, $H$.
The purpose of this section will be to study the generalization error associated to the following problem, where $\phi$ is a tolerance parameter:
$$
\sup_{R \in \mathcal{R}} \hat{G}_n(R) \qquad \text{subject to}\qquad \hat{H}_n (R) \le \alpha + \phi.
$$

The noise condition in this case is that there exist constants $a \in (0,1)$ and $D > 0$ such that $\forall t \ge 0 $,
$$
\mathbb{P} \left \{ | \eta(X) - Q^*(\alpha) | \le t  \right \} \le D \cdot t^{\frac{a}{1-a}},
$$
where $Q^*(\alpha) = \bar{H}^{*-1}(\alpha)$ with $H^*(t) = \mathbb{P} \{ \eta(X) < t \mid Y = -1\}$.

Bayes' formula gives us that:
$$
\bar{H}^*(t) = \frac{1}{1-p} \int_{1-t > 1 - \eta(x)} (1-\eta(x)) f(x) dx.
$$

For $\alpha$ fixed, we wish to find a collection of $\eta$'s such that $\bar{H}^*(1/2) = \alpha$, and that are equal to the $\eta_a$'s of section 1 around $x=1/2$.

This collection will enable us to illustrate the fast speeds for pointwise ROC optimization.
The condition $\bar{H}^*(1/2) = \alpha$ is equivalent to:
$$
\frac{\int_{\eta(x)>0.5} (1-\eta(x)) dx}{\int (1-\eta(x)) dx} = \alpha.
$$

Introducing, with $C<0.5, m< 0.5$
* $\eta_{a,C,m}(x) =
\begin{cases}
C  \quad & \text{if } x \in [0,l], \\
 \frac{1}{2}  - \frac{1}{2} \left ( 1 - 2x \right )^{\frac{1-a}{a}} \quad & \text{if } x \in ]m,0.5], \\
 \frac{1}{2}  + \frac{1}{2} \left ( 2x - 1 \right )^{\frac{1-a}{a}} \quad & \text{if } x \in [0.5,1-m[, \\
1-C  \quad & \text{if } x \in [1-m,1]. \\
\end{cases}$

We have that for $t$ small enough, $\mathbb{P} \left ( |2 \eta_{a,C,m} (X) - 1 | \le t  \right ) = t^{\frac{a}{1-a}}$
and thus the noise condition is satisfied with parameter $a$.

With this family of $\eta$'s the condition $\bar{H}^*(1/2) = \alpha$ becomes:
$$
\frac{1}{2}+ m(1-2C)  + \frac{a (1-2m)^{a^{-1}}}{2} = 1-\alpha.
$$
We fix $m$ and $\alpha$ and solve in $C$,
$$
C = \frac{1}{2} + \frac{2\alpha-1}{4m} + \frac{a(1-2m)^{a^{-1}}}{4m} .
$$

There are conditions on the possibles values of $\alpha$ imposed by the restrictions on the value $C$
$$
C<1/2 \Leftrightarrow \alpha < \frac{1}{2} - \frac{a}{2} (1-2m)^{a^{-1}},
$$
$$
C>0 \Leftrightarrow  \alpha > \frac{1}{2} - m - \frac{a}{2} (1-2m)^{a^{-1}}.
$$

If we wish for $\eta$ to be increasing, we need to have that $C < \frac{1}{2} - \frac{1}{2} (1-2m)^{\frac{1-a}{a}}$,
but it is not very important in practice.

```python
alpha = 0.18
a = 0.80
m = 0.35
```

```python
ut.plot_possible_alpha_values(gen_exp.bipart_interval_alpha, a_collection, alpha=alpha, m=m)
```

```python
C = gen_exp.bipart_C(a, alpha, m=m)
```

```python
gendatafun = lambda n, a: gen_exp.datagen_eta_aCm(n, a, C, m)
etafun = lambda x, a: gen_exp.eta_aCm(x, a, C, m)
ut.plot_emp_mammen(gendatafun, etafun, a_collection, n_obs = 200)
```

```python
eta_fun = lambda x, a: gen_exp.eta_aCm(x, a, gen_exp.bipart_C(a, alpha,m), m)
ut.plot_eta(a_collection, eta_fun)
```

```python
gen_datafun = lambda n, a : gen_exp.datagen_eta_aCm(n, a, C=gen_exp.bipart_C(a, alpha, m=m))
ut.plot_distribution(a_collection, gen_datafun)
```

We need to know the primitive of $\eta_{a,C,m}$ to compute $G$ and $H$ on the region defined by our frontier.

To compute the integral of $\eta_{a,C,m}$ we fist compute the integral of $\eta_{a}$:
$$
\int_0^f \eta_a(x) dx = \frac{2f-a}{4} + \frac{a|2f-1|^{a^{-1}}}{4}.
$$

```python
n_list = [int(np.power(2., i)) for i in range(6, 10)]
```

```python
n_exps = 1000
```

```python
n_list, a_collection, alpha, m, n_exps
```

    ([64, 128, 256, 512], array([ 0.2,  0.4,  0.6,  0.8]), 0.18, 0.35, 1000)

```python
df_bipart = gen_exp.exp_bi_ranking(n_list, n_exps, a_collection, alpha=alpha, m=m)
```

    Done   0 %
    Done   5 %
    Done  10 %
    Done  15 %
    Done  20 %
    Done  25 %
    Done  30 %
    Done  35 %
    Done  40 %
    Done  45 %
    Done  50 %
    Done  55 %
    Done  60 %
    Done  65 %
    Done  70 %
    Done  75 %
    Done  80 %
    Done  85 %
    Done  90 %
    Done  95 %

```python
ut.boxplot_slopes(df_bipart, quant=0.8)
```

```python
ut.slopes_quants(df_bipart)
```

```python
df_bipart.to_csv("results_bipart.csv", index = False)
```

# 3. Pointwise ROC optimization in similarity ranking

**It is advised to have read and understood:** * *

Vogel, R.; Bellet, A. & Clémençon, S.
*A Probabilistic Theory of Supervised Similarity Learning for Pointwise ROC Curve Optimization.*
Proceedings of the 35th International Conference on Machine Learning, PMLR, 2018, 80, 5062-5071

** before reading this section. **

In a two classes setting, assume that we have two classes ${-1, +1}$ of equal probability and $X \in [0,1]$ such that $X \sim \mathcal{U} ([0,1])$.

Then if $g$ and $f$ are respectively the densities of $X|Y=+1$ and $X | Y=-1$, we have that $1 = \frac{1}{2}g + \frac{1}{2}f$.

We introduce the following density $g_{a,C,m}$ for the first class, with $C<0.5, m< 0.5$
$$
\frac{1}{2}g (x) = \frac{1}{2} g_{a,C,m}(x) =
\begin{cases}
C  \quad & \text{if } x \in [0,m], \\
 \frac{1}{2}  - \frac{1}{2} \left ( 1 - 2x \right )^{\frac{1-a}{a}} \quad & \text{if } x \in ]m,0.5], \\
 \frac{1}{2}  + \frac{1}{2} \left ( 2x - 1 \right )^{\frac{1-a}{a}} \quad & \text{if } x \in [0.5,1-m[, \\
1-C  \quad & \text{if } x \in [1-m,1]. \\
\end{cases}
$$

Our optimal similarity function satisfies:
$$
\eta(x,x') = \eta_{a,C,m}(x,x') = \frac{1}{4}g(x)g(x') + \frac{1}{4}(2-g(x))(2-g(x'))
= \frac{1}{2} + \frac{1}{2} \left ( g(x) - 1 \right )\left( g(x') - 1 \right ),
$$

where

$$
g (x) - 1 =
\begin{cases}
2C - 1 \quad & \text{if } x \in [0,m], \\
 \text{sgn}(2x-1)  \left | 2x - 1 \right |^{\frac{1-a}{a}} \quad & \text{if } x \in ]m,1-m], \\
1-2C  \quad & \text{if } x \in [1-m,1]. \\
\end{cases}
$$

As before, we wish to have $Q^*(\alpha)=1/2$. When $t$ is small enough,

$$
\begin{align}
\mathbb{P} \left( |\eta(x,x') - Q^*(\alpha)| \le t \right)
&= \mathbb{P} \left( |\eta(x,x')-1/2|  \le  t \right), \\
&= 2 (4t)^{\frac{a}{1-a}} \left [ 1 - \log(2) -  \frac{a}{1-a} \log \left ( 4t \right ) \right ],\\
\end{align}
$$
which satisfies our noise condition. Indeed, if $t$ is small enough, we have something of order $t^{\frac{a}{1-a}} \log(t)$.

It is necessary to have once again that:
$$
\int_{\eta(x, x')>0.5} \eta(x, x') dx= \frac{1-\alpha}{2}.
$$

We note by
* $A_1$ the value of the integral of $\eta(x,x')$ restricted to $[1-m,1]\times [1-m,1]$,
* $A_2$ the value of the integral of $\eta(x,x')$ restricted to $[1-m,1]\times [0.5,1-m]$,
* $A_3$ the value of the integral of $\eta(x,x')$ restricted to $[0.5,1-m]\times [0.5,1-m]$,
see drawing below.

```python
gen_exp.illustrate_As()
```

Then
$$
\begin{align}
\int_{\eta(x, x')>0.5} \eta(x, x') dx &=
2 \times A_1 + 4 \times A_2 + 2\times A_3,\\
\int_{\eta(x, x')>0.5} \eta(x, x') dx &\\
=
2 \times \frac{m^2}{2} & \left ( 1 +  \left(1-2C\right )^2   \right ) +
4 \times \frac{m}{4} \left [ (1-2m) + a(1-2C)(1-2m)^{a^{-1}}  \right ]  +
2 \times  \frac{1}{8} \left[ \left(1 -2m \right)^2 + a^2 \left ( 1-2m \right )^{2a^{-1}} \right].\\
\end{align}
$$

We solve the following equation in $C$
$$
2A_1 + 4A_2 + 2A_3 =  \frac{1-\alpha}{2},\\
$$
with a symbolic calculus software and obtain
$$
C = \frac{1}{2}- \frac{\sqrt{1-2\alpha}}{4m} + \frac{a(1-2m)^{a^{-1}}}{4m} .
$$

The interval in which $\alpha$ is supposed to be to satisfy $C \ge 0$ (lhs constraint) and $C \le 1/2$ (rhs constraint) is
$$
\left [\frac{1}{2} - \frac{ \left (2m + a(1-2m)^{a^{-1}} \right )^2}{2} ;
\frac{1}{2} - \frac{  a^2(1-2m)^{2a^{-1}}}{ 8m^2} \right]
$$

```python
alpha = 0.26
m = 0.35
a_collection = np.linspace(0.1, 0.9, 9)
```

```python
ut.plot_possible_alpha_values(gen_exp.simrank_interval_alpha, a_collection, alpha=alpha, m=m, sup_inc_simrank=True)
```

```python
gendata_fun_of_a = lambda n, a: gen_exp.datagen_eta_aCm(n, a, C=gen_exp.simrank_C(a, alpha, m), m=m)
eta_fun_of_a = lambda x, xp, a: gen_exp.simrank_eta(x, xp, a, alpha=alpha, m=m)
ut.plot_emp_mammen(gendata_fun_of_a, eta_fun_of_a, [a for i, a in enumerate(a_collection) if i%2 == 0], simrank=True)
```

```python
n_list = [int(np.power(2., i)) for i in range(6, 10)] # 11
a_collection = np.linspace(0.1, 0.9, 9)
m = 0.36
n_exps = 1000
m, alpha, n_list, a_collection, len(n_list)*len(a_collection)*n_exps
```

    (0.36,
     0.26,
     [64, 128, 256, 512],
     array([ 0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9]),
     36000)

```python
# df_simrank = gen_exp.exp_sim_ranking(n_list, n_exps, a_collection, m=m, alpha=alpha)
# Very long since we use complete estimators, should be ran on a powerful machine.
# You can run the script gen_exp.py and load the results from there.

df_simrank = pd.read_csv("results_simrank.csv")
```

```python
sel_as = [a for i, a in enumerate(df_simrank["a"].unique()) if i % 3 == 0]
```

```python
ut.boxplot_slopes(df_simrank[[a in sel_as for a in df_simrank["a"]]], quant=0.8)
```

```python
ut.slopes_quants(df_simrank)
```


# Asymptotic solutions in long-times

Projectile motion in a non-homogenous potential field with drag is described by the equation

$$y_{\tau \tau} + \beta \epsilon y_{\tau}^2 + (1 + \phi)\frac{1}{(1 + \epsilon y)^2} = 0,$$

with $y(0) = \epsilon$ and $y_{\tau}(0)=1$, and where $\epsilon \ll 1$ is expected.

```python
import sympy as sym
from sympy import init_printing
init_printing(order='rev-lex')
```

```python
y, eps, a, b, tau, t, beta, gamma, phi = sym.symbols('y, epsilon, a, b, tau, t, beta, gamma, phi')
y0 = sym.Function('y0')(t)
y1 = sym.Function('y1')(t)
y2 = sym.Function('y2')(t)
y3 = sym.Function('y3')(t)
y4 = sym.Function('y4')(t)
```

```python
y = sym.Eq(y0 + eps*y1 + eps**2*y2 + eps**3*y3 + eps**4*y4) # naive expansion

class f(sym.Function):

    @classmethod
    def eval(cls, y):
        return y.lhs.diff(t,t) + beta*eps*(y.lhs.diff(t))**2 + (1 + phi)/(1 + eps*y.lhs)**2
        #return y.lhs.diff(tau, tau) + eps/y.lhs**2
y
```

```python
the_series = sym.series(f(y), eps, x0=0, n=5)
by_order = sym.collect(the_series, eps, evaluate=False)
the_series
```

### $\mathcal{O} \left( 1 \right) \mbox{Solution}$

```python
sym.Eq(by_order[1].removeO())
```

```python
eqn = sym.Eq(by_order[1].removeO())    #1 + y0(tau).diff(tau, tau))
soln0 = sym.dsolve(eqn, y0)
constants = sym.solve([soln0.rhs.subs(t,0) - 0, \
                      soln0.rhs.diff(t).subs(t,0) - 1])
C1, C2 = sym.symbols('C1 C2')
soln0 = soln0.subs(constants)
soln0
```

### $\mathcal{O} \left( \epsilon \right) \mbox{Solution}$

```python
by_order[eps]
```

```python
try:
    eqn = sym.Eq(by_order[eps].replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps])

soln1 = sym.dsolve(eqn, y1)
constants = sym.solve([soln1.rhs.subs(t,0) - 0, \
                      soln1.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln1 = soln1.subs(constants)
soln1

```

### $\mathcal{O} \left( \epsilon^2 \right) \mbox{Solution}$

```python
by_order[eps**2]
```

```python
try:
    eqn = sym.Eq(by_order[eps**2].replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**2].replace(y1, soln1.rhs))

soln2 = sym.dsolve(eqn, y2)
constants = sym.solve([soln2.rhs.subs(t,0) - 0, \
                      soln2.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln2 = soln2.subs(constants)
sym.factor(soln2)
```

### $\mathcal{O} \left( \epsilon^3 \right) \mbox{Solution}$

```python
by_order[eps**3]
```

```python
try:
    eqn = sym.Eq(by_order[eps**3].replace(y2, soln2.rhs).replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**3].replace(y2, soln2.rhs))

soln3 = sym.dsolve(eqn, y3)
constants = sym.solve([soln3.rhs.subs(t,0) - 0, \
                      soln3.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln3 = soln3.subs(constants)
sym.factor(soln3)
```

### $\mathcal{O} \left( \epsilon^4 \right) \mbox{Solution}$

```python
by_order[eps**4]
```

```python
try:
    eqn = sym.Eq(by_order[eps**4].replace(y3, soln3.rhs).replace(
        y2, soln2.rhs).replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**4].replace(y3, soln3.rhs))

soln4 = sym.dsolve(eqn, y4)
constants = sym.solve([soln4.rhs.subs(t,0) - 0, \
                      soln4.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln4 = soln4.subs(constants)
sym.factor(soln4)
```

### $\mbox{Composite Solution}$

```python
y_comp = sym.symbols('y_{comp}', cls=sym.Function)
try:
    y_comp = sym.Eq(y_comp, soln0.rhs + eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)
except NameError:
    y_comp = sym.Eq(y_comp, eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)

#print(sym.latex(y_comp))
#print(str(y_comp.rhs.subs(beta,6.5E-4)))
print(str(y_comp.rhs.subs(beta,6.5E-4).subs(t, 1)))
```

    epsilon**4*(-0.00782733896808962*phi**5 + 0.0391366948404481*phi**4 - 0.0377019044211757*phi**3 - 0.0473119647406747*phi**2 + 0.0473251558176387*phi + 0.00997118232863578) + epsilon**3*(-0.0144932237746701*phi**4 + 0.0579728950986804*phi**3 - 0.0226467929133676*phi**2 - 0.0770225675465816*phi + 0.0180903441714802) + epsilon**2*(-0.0305736205*phi**3 + 0.0917208615*phi**2 + 0.0249999436666668*phi - 0.0972943974999999) + epsilon*(-0.0833875*phi**2 + 0.166775*phi + 0.2498375) - phi/2 + 1/2

### $\mbox{The Trajectory}$

```python
def savefig(filename, pics):
    if pics == True:
        plt.savefig('../doc/figures/{}.pgf'.format(filename), bbox_inches='tight', dpi=400)
    else:
        pass
pics = True
```

```python
import matplotlib.pyplot as plt
import matplotlib
import numpy as np
import scipy as sp

%config InlineBackend.figure_format = 'retina'
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.rcParams['figure.dpi'] = 300
matplotlib.rcParams.update(
{   'text.color': 'k',
    'xtick.color': 'k',
    'ytick.color': 'k',
    'axes.labelcolor': 'k'
})
plt.rc('font', size=14)
eps_val = [.1, .5, 1.][::-1]
linestyle = ['rs--', 'bo-', 'cv-.', 'k+:', 'm']
tt = sp.arange(0,1,0.001)
bet = 6.5E-4
plt.figure(figsize=(6, 4))#, dpi=100)
for keys, vals in enumerate(eps_val):
    y_compP = sym.lambdify(t, y_comp.rhs.subs(eps, vals).subs(beta, bet), 'numpy')
    plt.plot(tt, y_compP(tt), linestyle[keys],label=r'$\phi \mathbf{E}\mbox{u}$'+ ' = {}'.format(vals).rstrip('0').rstrip('.')
             , markevery=100)

plt.ylim(ymin=0., ymax=0.7)
plt.xlim(xmax=1)
plt.ylabel(r'$y^*$')
plt.xlabel(r'$t^*$')
leg = plt.legend(title = r'$\mathbf{D}\mbox{g}$' +' = {:1.0E}'.format(bet)[:-4] + r'$\times 10^{-4}$')
leg.get_frame().set_linewidth(0.0)
savefig('long_times',pics)
plt.show()
```

## Time aloft

```python
y2 = sym.symbols('y2', cls=sym.Function)
y2 = sym.Function('y2')(t)
try:
    y2 = sym.Eq(y2, soln0.rhs + eps*soln1.rhs + eps**2*soln2.rhs
                + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)
except NameError:
    y2 = sym.Eq(y2, eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs)
y2.rhs
#y2.diff(t)
```

```python
tau0, tau1, tau2, tau3 = sym.symbols('tau0 tau1 tau2 tau3')
tau = sym.Eq(tau0 + eps*tau1 + eps**2*tau2 + eps**3*tau3)
y3 = y2.rhs.subs(t, tau.lhs).series(eps)
col = sym.collect(y3, eps, evaluate=False)
```

### $\mathcal{O} \left( 1 \right) \mbox{Solution}$

```python
#tau0 = 2
sym.Eq(col[1].removeO())
```

Two roots, lets look at $\tau_0 = 2$.

### $\mathcal{O} \left( \epsilon \right) \mbox{Solution}$

```python
order_eps = col[eps].subs(tau0, 2)
order_eps
```

```python
soln_eps = sym.solve(order_eps, tau1)
```

### $\mathcal{O} \left( \epsilon^2 \right) \mbox{Solution}$

```python
order_eps2 = col[eps**2].subs(tau0, 2).subs(tau1, soln_eps[0])
order_eps2
```

```python
soln_eps2 = sym.solve(order_eps2, tau2)
```

### $\mathcal{O} \left( \epsilon^3 \right) \mbox{Solution}$

```python
order_eps3 = col[eps**3].subs(tau0, 2).subs(tau1, soln_eps[0]).subs(tau2, soln_eps2[0])
order_eps3
soln_eps3 = sym.solve(order_eps3, tau3)
```

### Composite solution

```python
0.2*8*np.pi
```

```python
tau0, tau1, tau2, tau3 = sym.symbols('tau0 tau1 tau2 tau3')
tau = sym.Eq(tau0 + eps*tau1 + eps**2*tau2 + eps**3*tau3)
tau = tau.subs(tau0, 2).subs(tau1, soln_eps[0]).subs(tau2, soln_eps2[0]).subs(tau3, soln_eps3[0])
print(tau.subs(beta, 6E-4).lhs*1.31)
print(str(tau.subs(beta, 6E-4).lhs))
tau_soln = sym.lambdify(eps, tau.subs(beta, 6E-4).lhs*1.31, 'numpy')
print(tau_soln(5.026548245743669))
tau
```

```python
a=y_comp.rhs.subs(t, tau.lhs.subs(eps, gamma)).subs(beta,6.5E-4)
(sym.collect(a, eps, evaluate=False)[1])
```

```python
ttt = np.arange(0.01, 1,0.001)
betas = [bet]
linestyle = ['k','k--', 'bo-', 'cv-.', 'k+:', 'm']
plt.figure(figsize=(6, 4))#, dpi=100)
for keys, vals in enumerate(betas):
    taun = tau.subs(beta, vals)
    tau_soln = sym.lambdify(eps, taun.lhs, 'numpy')
    label=['{:1.0E}'.format(vals)[:-4] + r'$\times 10^{-4}$']
    plt.semilogx(ttt, tau_soln(ttt), linestyle[keys],
                 label=r'$\mathbf{D}\mbox{g}$ = '+ label[keys], markevery=100)
plt.xlabel(r'$\phi \mathbf{E}\mbox{u}$')
plt.ylabel(r'$t_f$')
plt.legend()
savefig('drag', pics)
plt.show();
```

## Drag vs. inertia

```python
t, V = sym.symbols('t V', real=True, positive=True)
u = sym.symbols('u', cls=sym.Function)
eq = sym.diff(u(t), t) + u(t)**2
sol = sym.dsolve(eq, u(t))
print(sol.rhs)
u = sol.rhs
```

    1/(C1 + t)

```python
C1, C2 = sym.symbols('C1 C2')
eq = sym.Eq(u.subs(t, 0), 1)
sol = sym.solve(eq, C1)[0]
print(sol)
print(u.subs(C1, sol))
```

    1
    1/(t + 1)

```python
tt = np.arange(0.01,1.,0.001)
plt.figure(figsize=(6, 4), dpi=100)
u_soln = sym.lambdify(t, u.subs(C1, sol).subs(V,1.), 'numpy')
plt.semilogy(tt, u_soln(tt), 'k')
plt.show();
```

```python

```


#  Taylor problem 5.32

last revised: 12-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

**Replace ### by appropriate expressions.**

The equation for an underdamped oscillator, such as a mass on the end of a spring, takes the form

$\begin{align}
  x(t) = e^{-\beta t} [B_1 \cos(\omega_1 t) + B_2 \sin(\omega_1 t)]
\end{align}$

where

$\begin{align}
  \omega_1 = \sqrt{\omega_0^2 - \beta^2}
\end{align}$

and the mass is released from rest at position $x_0$ at $t=0$.

**Goal: plot $x(t)$ for $0 \leq t \leq 20$, with $x_0 = 1$, $\omega_0=1$, and $\beta = 0.$, 0.02, 0.1, 0.3, and 1.**

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def underdamped(t, betas, omega_0=1, x_0=1):
    """Solution x(t) for an underdamped harmonic oscillator."""
    omega_1 = np.sqrt(omega_0**2 - betas**2)
    B_1 = x
    B_2 = 5
    return np.exp(-betas*t) \
             * ( B_1 * np.cos(omega_1*t) + B_2 * np.sin(omega_1*t) )
```

```python
t_pts = np.arange(0., 20., .01)
betas = [0., 0.02, 0.1, 0.3, 0.9999]

fig = plt.figure(figsize=(10,6))
# look up "python enumerate" to find out how this works!
for i, beta in enumerate(betas):
    ax = fig.add_subplot(2, 3, i+1)
    ax.plot(t_pts, underdamped(t_pts, beta), color='blue')
    ax.set_title(rf'$\beta = {beta:.2f}$')
    ax.set_xlabel('t')
    ax.set_ylabel('x(t)')
    ax.set_ylim(-1.1,1.1)
    ax.axhline(0., color='black', alpha=0.3)  # lightened black zero line

fig.tight_layout()
plt.show()
```

## Bonus: Widgetized!

```python
from ipywidgets import interact, fixed
import ipywidgets as widgets

omega_0 = 1.

def plot_beta(beta):
    """Plot function for underdamped harmonic oscillator."""
    t_pts = np.arange(0., 20., .01)

    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(t_pts, underdamped(t_pts, beta), color='blue')
    ax.set_title(rf'$\beta = {beta:.2f}$')
    ax.set_xlabel('t')
    ax.set_ylabel('x(t)')
    ax.set_ylim(-1.1,1.1)
    ax.axhline(0., color='black', alpha=0.3)

    fig.tight_layout()

max_value = omega_0 - 0.0001
interact(plot_beta,
         beta=widgets.FloatSlider(min=0., max=max_value, step=0.01,
                                  value=0., readout_format='.2f',
                                  continuous_update=False));

```

    interactive(children=(FloatSlider(value=0.0, continuous_update=False, description='beta', max=0.9999, step=0.0…

Now let's allow for complex numbers!  This will enable us to take $\beta > \omega_0$.

```python
# numpy.lib.scimath version of sqrt handles complex numbers.
#  numpy exp, cos, and sin already can.
import numpy.lib.scimath as smath

def all_beta(t, beta, omega_0=1, x_0=1):
    """Solution x(t) for damped harmonic oscillator, allowing for overdamped
        as well as underdamped solution.
    """
    omega_1 = smath.sqrt(omega_0**2 - beta**2)
    return np.real( x_0 * np.exp(-beta*t) \
               * (np.cos(omega_1*t) + (beta/omega_1)*np.sin(omega_1*t)) )
```

```python
from ipywidgets import interact, fixed
import ipywidgets as widgets

omega_0 = 1.

def plot_all_beta(beta):
    """Plot of x(t) for damped harmonic oscillator, allowing for overdamped
       as well as underdamped cases."""
    t_pts = np.arange(0., 20., .01)

    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(t_pts, all_beta(t_pts, beta), color='blue')
    ax.set_title(rf'$\beta = {beta:.2f}$')
    ax.set_xlabel('t')
    ax.set_ylabel('x(t)')
    ax.set_ylim(-1.1,1.1)
    ax.axhline(0., color='black', alpha=0.3)

    fig.tight_layout()

interact(plot_all_beta,
         beta=widgets.FloatSlider(min=0., max=2, step=0.01,
                                  value=0., readout_format='.2f',
                                  continuous_update=False));

```

    interactive(children=(FloatSlider(value=0.0, continuous_update=False, description='beta', max=2.0, step=0.01),…

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def underdamped(t, betas, omega_0=1, x_0=1):
    """Solution x(t) for an underdamped harmonic oscillator."""
    omega_1 = np.sqrt(omega_0**2 - betas**2)
    B_1 = x_0
    B_2 = x_0*betas/omega_1
    return np.exp(-betas*t) \
             * ( B_1 * np.cos(omega_1*t) + B_2 * np.sin(omega_1*t) )
```

```python
t_pts = np.arange(0., 20., .01)
betas = [0., 0.02, 0.1, 0.3, 0.9999]

fig = plt.figure(figsize=(10,6))
# look up "python enumerate" to find out how this works!
for i, beta in enumerate(betas):
    ax = fig.add_subplot(2, 3, i+1)
    ax.plot(t_pts, underdamped(t_pts, beta), color='blue')
    ax.set_title(rf'$\beta = {beta:.2f}$')
    ax.set_xlabel('t')
    ax.set_ylabel('x(t)')
    ax.set_ylim(-1.1,1.1)
    ax.axhline(0., color='black', alpha=0.3)  # lightened black zero line

fig.tight_layout()
plt.show()
```

```python

```


```python
# For reading data sets from the web.
import pandas
# For lots of great things.
import numpy as np
# To make our plots.
import matplotlib.pyplot as plt
%matplotlib inline
# Because sympy and LaTeX make
# everything look wonderful!
from sympy import *
init_printing(use_latex=True)
from IPython.display import display
# We will use this to check our implementation...
from sklearn.decomposition import PCA
# We will grab another data set using Keras
# after we finish up with Iris...
import keras
# Need this for LabelEncoder
from sklearn import preprocessing
```

```python

```

```python
student_data = np.array(pandas.read_table("./student-por.csv",
delimiter=";", header=0))
```

```python
student_data.shape
```

```python
le = preprocessing.LabelEncoder()
```

# Encoding non-numeric data to integers

```python
# School (binomial)
le.fit(student_data[:,0])
student_data[:,0] = le.transform(student_data[:,0])
```

```python
# Sex (binomial)
le.fit(student_data[:,1])
student_data[:,1] = le.transform(student_data[:,1])
```

```python
# address (binomial)
le.fit(student_data[:,3])
student_data[:,3] = le.transform(student_data[:,3])
```

```python
# Famsize (binomial)
le.fit(student_data[:,4])
student_data[:,4] = le.transform(student_data[:,4])
```

```python
# Pstatus (binomial)
le.fit(student_data[:,5])
student_data[:,5] = le.transform(student_data[:,5])
```

```python
# Mjob (nominal)
le.fit(student_data[:,8])
student_data[:,8] = le.transform(student_data[:,8])
```

```python
# Fjob (nominal)
le.fit(student_data[:,9])
student_data[:,9] = le.transform(student_data[:,9])
```

```python
# Reason (nominal)
le.fit(student_data[:,10])
student_data[:,10] = le.transform(student_data[:,10])
```

```python
# Guardian (nominal)
le.fit(student_data[:,11])
student_data[:,11] = le.transform(student_data[:,11])
```

```python
# schoolsup (binomial)
le.fit(student_data[:,15])
student_data[:,15] = le.transform(student_data[:,15])
```

```python
# famsup (binomial)
le.fit(student_data[:,16])
student_data[:,16] = le.transform(student_data[:,16])
```

```python
# paid (binomial)
le.fit(student_data[:,17])
student_data[:,17] = le.transform(student_data[:,17])
```

```python
# activities (binomial)
le.fit(student_data[:,18])
student_data[:,18] = le.transform(student_data[:,18])
```

```python
# nursery (binomial)
le.fit(student_data[:,19])
student_data[:,19] = le.transform(student_data[:,19])
```

```python
# higher (binomial)
le.fit(student_data[:,20])
student_data[:,20] = le.transform(student_data[:,20])
```

```python
# internet (binomial)
le.fit(student_data[:,21])
student_data[:,21] = le.transform(student_data[:,21])
```

```python
# romantic (binomial)
le.fit(student_data[:,22])
student_data[:,22] = le.transform(student_data[:,22])
```

```python
student_data
```

    array([[0, 0, 18, ..., 0, 11, 11],
           [0, 0, 17, ..., 9, 11, 11],
           [0, 0, 15, ..., 12, 13, 12],
           ...,
           [1, 0, 18, ..., 11, 12, 9],
           [1, 1, 17, ..., 10, 10, 10],
           [1, 1, 18, ..., 10, 11, 11]], dtype=object)

# Encoding 0's to -1 for binomial data.

### Surely there's a one line method to do this... right, Keras?

```python
school = student_data[:,0]
school = np.where(school==0, -1, school)
student_data[:,0] = school
```

```python
sex = student_data[:,1]
sex = np.where(sex==0, -1, sex)
student_data[:,1] = sex
```

```python
address = student_data[:,3]
address = np.where(address==0, -1, address)
student_data[:,3] = address
```

```python
famsize = student_data[:,4]
famsize = np.where(famsize==0, -1, famsize)
student_data[:,4] = famsize
```

```python
Pstatus = student_data[:,5]
Pstatus = np.where(Pstatus==0, -1, Pstatus)
student_data[:,5] = Pstatus
```

```python
schoolsup = student_data[:,15]
schoolsup = np.where(schoolsup==0, -1, schoolsup)
student_data[:,15] = schoolsup
```

```python
famsup = student_data[:,16]
famsup = np.where(famsup==0, -1, famsup)
student_data[:,16] = famsup
```

```python
paid = student_data[:,17]
paid = np.where(paid==0, -1, paid)
student_data[:,17] = paid
```

```python
activities = student_data[:,18]
activities = np.where(activities==0, -1, activities)
student_data[:,18] = activities
```

```python
nursery = student_data[:,19]
nursery = np.where(nursery==0, -1, nursery)
student_data[:,19] = nursery
```

```python
higher = student_data[:,20]
higher = np.where(higher==0, -1, higher)
student_data[:,20] = higher
```

```python
internet = student_data[:,21]
internet = np.where(internet==0, -1, internet)
student_data[:,21] = internet
```

```python
romantic = student_data[:,22]
romantic = np.where(romantic==0, -1, romantic)
student_data[:,22] = romantic
```

# Standardizing the nominal and numerical data.

```python
scaler = preprocessing.StandardScaler()
```

### I don't think this warning is THAT concerning

```python
temp = student_data[:,[2,14,29,30,31]]
Standardized = scaler.fit_transform(temp)
```

    /nfshome/apps/python-3.6.7/lib/python3.6/site-packages/sklearn/utils/validation.py:590: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
      warnings.warn(msg, DataConversionWarning)
    /nfshome/apps/python-3.6.7/lib/python3.6/site-packages/sklearn/utils/validation.py:590: DataConversionWarning: Data with input dtype object was converted to float64 by StandardScaler.
      warnings.warn(msg, DataConversionWarning)

### Not sure about the -0.0 for the mean...

```python
print('Mean:', round(Standardized.mean()))
print('Standard deviation:', Standardized.std())
```

    Mean: 0.0
    Standard deviation: 1.0

```python
student_data[:,[2,14,29,30,31]] = Standardized
```

## 1-of-C encode nominal data

```python
# Mother's job
## Teacher, Health Care, Civil Services, At Home, Other

# index of current feature
index = 8

# unique values this feature can inhibit
uniqueVals = Matrix(np.unique(student_data[:,index])).T
display(uniqueVals)

# convert numerical data into categories
shape = uniqueVals.shape[1]
categoricalArr = keras.utils.to_categorical(student_data[:,index], shape)
for i in range (649):
    student_data[i,index] = [categoricalArr[i,:]]
display(student_data[:,index].shape)
display(student_data[:5,index])
```

```python
# Father's Job
## Teacher, Health Care, Civil Services, At Home, Other
index = 9
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
categoricalArr = keras.utils.to_categorical(student_data[:,index], shape)
for i in range (649):
    student_data[i,index] = [categoricalArr[i,:]]
```

```python
# Reason for Choosing This School
## Close to Home, School Reputation, Course Prefrence, Other
index = 10
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
categoricalArr = keras.utils.to_categorical(student_data[:,index], shape)
for i in range (649):
    student_data[i,index] = [categoricalArr[i,:]]
```

```python
# Gaurdian
## Mother, Father, Other
index = 11
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
categoricalArr = keras.utils.to_categorical(student_data[:,index], shape)
for i in range (649):
    student_data[i,index] = [categoricalArr[i,:]]
```

# Thermometer Encode Nominal Data

```python
# Mother's Education
## None, Primary (4th grade), 5th to 9th grade, Secondary, Higher
index = 6
display(student_data[0:5,index])
```

    array([4, 1, 1, 4, 3], dtype=object)

```python
uniqueVals = Matrix(np.unique(student_data[:,index])).T
display(uniqueVals)
shape = uniqueVals.shape[1]
```

$$\left[\begin{matrix}0 & 1 & 2 & 3 & 4\end{matrix}\right]$$

```python
# https://stackoverflow.com/questions/49080613/numpy-thermometer-encoding
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
display(thermArr[0:5,:])
display(thermArr[0:5,:shape-1])
```

    array([[1, 1, 1, 1, 0, 0, 0, 0],
           [1, 0, 0, 0, 0, 0, 0, 0],
           [1, 0, 0, 0, 0, 0, 0, 0],
           [1, 1, 1, 1, 0, 0, 0, 0],
           [1, 1, 1, 0, 0, 0, 0, 0]], dtype=uint8)

    array([[1, 1, 1, 1],
           [1, 0, 0, 0],
           [1, 0, 0, 0],
           [1, 1, 1, 1],
           [1, 1, 1, 0]], dtype=uint8)

```python
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]
display(student_data[0:5,index])
```

    array([list([array([1, 1, 1, 1], dtype=uint8)]),
           list([array([1, 0, 0, 0], dtype=uint8)]),
           list([array([1, 0, 0, 0], dtype=uint8)]),
           list([array([1, 1, 1, 1], dtype=uint8)]),
           list([array([1, 1, 1, 0], dtype=uint8)])], dtype=object)

```python
# Father's Education
## None, Primary (4th grade), 5th to 9th grade, Secondary, Higher
index = 7
display(student_data[0:5,index])
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]
display(student_data[0:5,index])
```

    array([4, 1, 1, 2, 3], dtype=object)

    array([list([array([1, 1, 1, 1], dtype=uint8)]),
           list([array([1, 0, 0, 0], dtype=uint8)]),
           list([array([1, 0, 0, 0], dtype=uint8)]),
           list([array([1, 1, 0, 0], dtype=uint8)]),
           list([array([1, 1, 1, 0], dtype=uint8)])], dtype=object)

```python
# Travel Time
## <15min, 15min - 30min, 30min - 1hour, >1hour
index = 12
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]

# Study Time
## <2hours, 2hours - 5hours, 5hours - 10hours, >10hours
index = 13
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]

# Failures
## 0, 1, 2, 3, 4+
index = 14
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]

# Free Time
## 1 - very low to 5 - very high
index = 24
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]

# Going Out With Friends
## 1 - very low to 5 - very high
index = 25
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]

# Weekend Alcohol Consumption
## 1 - very low to 5 - very high
index = 26
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]

# Workday Alcohol Consumption
## 1 - very low to 5 - very high
index = 27
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]

# Current Health Status
## 1 - very low to 5 - very high
index = 28
uniqueVals = Matrix(np.unique(student_data[:,index])).T
shape = uniqueVals.shape[1]
thermArr = np.array([student_data[:,index]], dtype=np.uint8)
thermArr = np.fliplr(np.unpackbits((1 << thermArr) - 1).reshape(-1,8))
for i in range (649):
    student_data[i,index] = [thermArr[i,:shape-1]]
```

### Convert results to one-hot encoding
#### the actual output vector doesn't include at least one example of each possible grade so I made a vector of all possibilities for to_categorical

```python
possible_results = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
```

## Getting a weird error in the training below
## am I storing the data wrong here?

```python
results = student_data[:,32]
y = keras.utils.to_categorical(results, len(possible_results))
x = student_data[:,:32]

```

# Data is now preprocessed

```python
model = keras.Sequential()
```

```python
input_size = x.shape[1]
output_size = y.shape[1]
```

```python
# create the hidden layer
model.add(keras.layers.Dense(400,input_dim=input_size,activation='relu'))
```

```python
model.add(keras.layers.Dense(200,activation='relu'))
```

```python
# output layer
model.add(keras.layers.Dense(output_size,activation='softmax'))
```

```python
# compile
model.compile(loss='categorical_crossentropy',optimizer='adam', metrics=['accuracy'])
```

```python
model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_14 (Dense)             (None, 400)               13200
    _________________________________________________________________
    dense_15 (Dense)             (None, 200)               80200
    _________________________________________________________________
    dense_16 (Dense)             (None, 20)                4020
    =================================================================
    Total params: 97,420
    Trainable params: 97,420
    Non-trainable params: 0
    _________________________________________________________________

# Model Training

```python
history = model.fit(x,y,batch_size = 32, epochs = 160, verbose = 0, validation_split = 0.25)
```

```python
#plt.plot(history.history['loss'])
#plt.title('model loss')
#plt.ylabel('loss')
#plt.xlabel('epoch')
#plt.legend(['train'], loc='upper left')
#plt.show()

```

```python
score = model.evaluate(x,y, verbose=0)
```

```python
plt.figure(1)

# summarize history for accuracy

plt.subplot(211)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train','test'], loc ='upper left')

# summarize history for loss

plt.subplot(212)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train','test'], loc ='upper left')

plt.tight_layout()
plt.show()
```

```python
print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

```python
## PCAs of Data
```

```python
#def func(X):
 #   return X
#xFunc = np.apply_along_axis(func,0,x)

#U,S,V = np.linalg.svd(xFunc,full_matrices=True)

# Plot the percent of variance accounted for by each feature
#plt.plot(100.0*S/np.sum(S))
#plt.ylabel("Percent Variance")
#plt.xlabel("Singular Value")
#plt.show()
```

```python
# https://www.youtube.com/watch?v=Lsue2gEM9D0
pca=PCA()
pca.fit(x)
pca_data = pca.transform(x)
```

```python
per_var = np.round(pca.explained_variance_ratio_* 100, decimals = 1)
labels = [str(i) for i in range (1, len(per_var)+1)]

plt.bar(x=range(1,len(per_var)+1), height=per_var, tick_label=labels)
plt.ylabel("% var")
plt.xlabel("PC")
plt.title("Scree Plot")
plt.show()
```

```python

```

```python

```


**This notebook is an exercise in the [Computer Vision](https://www.kaggle.com/learn/computer-vision) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/convolution-and-relu).**

---

# Introduction #

In this exercise, you'll work on building some intuition around feature extraction. First, we'll walk through the example we did in the tutorial again, but this time, with a kernel you choose yourself. We've mostly been working with images in this course, but what's behind all of the operations we're learning about is mathematics. So, we'll also take a look at how these feature maps can be represented instead as arrays of numbers and what effect convolution with a kernel will have on them.

Run the cell below to get started!

```python
# Setup feedback system
from learntools.core import binder
binder.bind(globals())
from learntools.computer_vision.ex2 import *

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

plt.rc('figure', autolayout=True)
plt.rc('axes', labelweight='bold', labelsize='large',
       titleweight='bold', titlesize=18, titlepad=10)
plt.rc('image', cmap='magma')

tf.config.run_functions_eagerly(True)
```

# Apply Transformations #

The next few exercises walk through feature extraction just like the example in the tutorial. Run the following cell to load an image we'll use for the next few exercises.

```python
image_path = '../input/computer-vision-resources/car_illus.jpg'
image = tf.io.read_file(image_path)
image = tf.io.decode_jpeg(image, channels=1)
image = tf.image.resize(image, size=[400, 400])

img = tf.squeeze(image).numpy()
plt.figure(figsize=(6, 6))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.show();
```

You can run this cell to see some standard kernels used in image processing.

```python
import learntools.computer_vision.visiontools as visiontools
from learntools.computer_vision.visiontools import edge, bottom_sobel, emboss, sharpen

kernels = [edge, bottom_sobel, emboss, sharpen]
names = ["Edge Detect", "Bottom Sobel", "Emboss", "Sharpen"]

plt.figure(figsize=(12, 12))
for i, (kernel, name) in enumerate(zip(kernels, names)):
    plt.subplot(1, 4, i+1)
    visiontools.show_kernel(kernel)
    plt.title(name)
plt.tight_layout()
```

# 1) Define Kernel #

Use the next code cell to define a kernel. You have your choice of what kind of kernel to apply. One thing to keep in mind is that the *sum* of the numbers in the kernel determines how bright the final image is. Generally, you should try to keep the sum of the numbers between 0 and 1 (though that's not required for a correct answer).

In general, a kernel can have any number of rows and columns. For this exercise, let's use a $3 \times 3$ kernel, which often gives the best results. Define a kernel with `tf.constant`.

```python
# YOUR CODE HERE: Define a kernel with 3 rows and 3 columns.
kernel = tf.constant([
    [-1, -1, -1],
    [-1,  8, -1],
    [-1, -1, -1],
])
# Uncomment to view kernel
# visiontools.show_kernel(kernel)

# Check your answer
q_1.check()
```

```python
# Lines below will give you a hint or solution code
#q_1.hint()
q_1.solution()
```

Now we'll do the first step of feature extraction, the filtering step. First run this cell to do some reformatting for TensorFlow.

```python
# Reformat for batch compatibility.
image = tf.image.convert_image_dtype(image, dtype=tf.float32)
image = tf.expand_dims(image, axis=0)
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

# 2) Apply Convolution #

Now we'll apply the kernel to the image by a convolution. The *layer* in Keras that does this is `layers.Conv2D`. What is the *backend function* in TensorFlow that performs the same operation?

```python
# YOUR CODE HERE: Give the TensorFlow convolution function (without arguments)
conv_fn = tf.nn.conv2d

# Check your answer
q_2.check()
```

```python
# Lines below will give you a hint or solution code
#q_2.hint()
q_2.solution()
```

Once you've got the correct answer, run this next cell to execute the convolution and see the result!

```python
image_filter = conv_fn(
    input=image,
    filters=kernel,
    strides=1, # or (1, 1)
    padding='SAME',
)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_filter)
)
plt.axis('off')
plt.show();
```

Can you see how the kernel you chose relates to the feature map it produced?

# 3) Apply ReLU #

Now detect the feature with the ReLU function. In Keras, you'll usually use this as the activation function in a `Conv2D` layer. What is the *backend function* in TensorFlow that does the same thing?

```python
# YOUR CODE HERE: Give the TensorFlow ReLU function (without arguments)
relu_fn = tf.nn.relu

# Check your answer
q_3.check()
```

```python
# Lines below will give you a hint or solution code
#q_3.hint()
q_3.solution()
```

Once you've got the solution, run this cell to detect the feature with ReLU and see the result!

The image you see below is the feature map produced by the kernel you chose. If you like, experiment with some of the other suggested kernels above, or, try to invent one that will extract a certain kind of feature.

```python
image_detect = relu_fn(image_filter)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_detect)
)
plt.axis('off')
plt.show();
```

In the tutorial, our discussion of kernels and feature maps was mainly visual. We saw the effect of `Conv2D` and `ReLU` by observing how they transformed some example images.

But the operations in a convolutional network (like in all neural networks) are usually defined through mathematical functions, through a computation on numbers. In the next exercise, we'll take a moment to explore this point of view.

Let's start by defining a simple array to act as an image, and another array to act as the kernel. Run the following cell to see these arrays.

```python
# Sympy is a python library for symbolic mathematics. It has a nice
# pretty printer for matrices, which is all we'll use it for.
import sympy
sympy.init_printing()
from IPython.display import display

image = np.array([
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 1, 1, 1],
    [0, 1, 0, 0, 0, 0],
])

kernel = np.array([
    [1, -1],
    [1, -1],
])

display(sympy.Matrix(image))
display(sympy.Matrix(kernel))
# Reformat for Tensorflow
image = tf.cast(image, dtype=tf.float32)
image = tf.reshape(image, [1, *image.shape, 1])
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

# 4) Observe Convolution on a Numerical Matrix #

What do you see? The image is simply a long vertical line on the left and a short horizontal line on the lower right. What about the kernel? What effect do you think it will have on this image? After you've thought about it, run the next cell for the answer.

```python
# View the solution (Run this code cell to receive credit!)
q_4.check()
```

Now let's try it out. Run the next cell to apply convolution and ReLU to the image and display the result.

```python
image_filter = tf.nn.conv2d(
    input=image,
    filters=kernel,
    strides=1,
    padding='VALID',
)
image_detect = tf.nn.relu(image_filter)

# The first matrix is the image after convolution, and the second is
# the image after ReLU.
display(sympy.Matrix(tf.squeeze(image_filter).numpy()))
display(sympy.Matrix(tf.squeeze(image_detect).numpy()))
```

Is the result what you expected?

# Conclusion #

In this lesson, you learned about the first two operations a convolutional classifier uses for feature extraction: **filtering** an image with a **convolution** and **detecting** the feature with the **rectified linear unit**.

# Keep Going #

Move on to [**Lesson 3**](https://www.kaggle.com/ryanholbrook/maximum-pooling) to learn the final operation: **condensing** the feature map with **maximum pooling**!

---

*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/computer-vision/discussion) to chat with other learners.*