# Project 2: Solution
Copyright © 2017 Oliver Beckstein. All Rights Reserved.

## Goal and Objectives

Simulate the movements of planets near the star TRAPPIST-1

1. Simulate movement of the 6 closest planets (b, c, d, e, f, g) for 1000 earth days
2. From which planets can an observer see the most other planets so close that one could resolve features of size 1000 km by naked eye?

The Space Tourism Bureau suggests TRAPPIST-1e (see poster below). Are they right?

(From [NASA Exoplanet Exploration: Galleries: TRAPPIST-1](https://exoplanets.nasa.gov/resources/2159/?linkId=34784370).)

## Outline of a Solution

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')
```

Most of the code is in the module `simulation`. First load system parameters from `parameters`.

```python
import parameters
import simulation
import analysis

from parameters import planets, star
```

```python
from importlib import reload
reload(simulation)
reload(analysis)
```

    <module 'analysis' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2017/Projects/P2/solution/analysis.py'>

## Orbits with star fixed

### Initial conditions

```python
planets_x = simulation.rmax(planets['semi-major'], planets['eccentricity'])
planets_vy = simulation.v_aphelion(planets['semi-major'], planets['eccentricity'], M=star['mass'])

r0 = np.array([planets_x, np.zeros_like(planets_x)]).T
v0 = np.array([np.zeros_like(planets_vy), planets_vy]).T
planet_masses = planets['mass']
masses = np.concatenate(([star['mass']], planet_masses))
```

### Simulation

Test that the integration is (likely) correct and reproduces the period for the innermost planet: Check visually that after 1.5 d, TRAPPIST-1b has almost completed one revolution (period 1.51 d).

```python
t, r, v = simulation.orbits(r0, v0, planet_masses, t_max=1.5, dt=0.01)
analysis.plot_orbits(r)
```

```python
t, r, v = simulation.orbits(r0, v0, planet_masses, t_max=5, dt=0.01)
analysis.plot_orbits(r)
```

Run the simulation for 1000 d.

```python
t, r, v = simulation.orbits(r0, v0, planet_masses, t_max=1000, dt=0.01)
analysis.plot_orbits(r)
```

### Energy conservation
Test energy conservation

$$
E = T + U = \text{const}
$$

for the 1000-d run.

The `kinetic_energy()` function compute the kinetic energy for each planet and time step. To get the system energies, we have to sum over all bodies (`axis=-1`) but we also need to compute the potential energy for *all* $N$-body interactions

$$
U = \frac{1}{2} \sum_{i=1}^N \sum_{j=1, j\neq i}^N \left( - \frac{G m_i m_j}{r_{ij}} \right)
$$

(as implemented in `simulation.U_tot()`). The total energy as a function of time

$$
E(t) = T(t) + U(t)
$$

or in code:

```python
T = np.sum(analysis.kinetic_energy(v, masses), axis=-1)
U = np.array([simulation.U_tot(rt, masses) for rt in r])
E_tot = T + U
```

(`U_tot()` can only operate on individual frames of the trajectory, so we need to loop and put the results into an array.)

Note that for the analysis we provide all masses, not just the planet masses:

```python
analysis.analyze_energies(t, r, v, simulation.U_tot, m=masses)
```

Energy conservation is acceptable, given the time step of `dt = 0.01`. (The unit of time is 1 so for the step of $10^{-2}$ we expect an error on the order of $\Delta t^2$, i.e., $10^{-4}$ as seen.)

## BONUS: Star wobble

Allow the central star to move, too, but set initial conditions so that the total momentum is 0. (Total momentum is conserved by velocity Verlet, and thus the center of mass of our simulation will remain fixed during the simulation.)

```python
# planets
r0 = np.array([planets_x, np.zeros_like(planets_x)]).T
v0 = np.array([np.zeros_like(planets_vy), planets_vy]).T

# prepend star
r0_system = np.vstack((np.array([[0, 0]]), r0))

# choose star velocity to make total momentum 0
p_planets = np.sum(planet_masses[:, np.newaxis] * v0, axis=0)
v0_star = -p_planets / star['mass']
v0_system = np.vstack((v0_star, v0))
```

NOTE: perhaps we should put the center of mass at 0 and move the star to

\begin{equation}
\mathbf{r}_{star} = - \frac{\sum_{i=1}^N m_i \mathbf{r}_i}{M_{star}},
\end{equation}

### Trajectories with star wobbling

1000 d trajectory with very fine time step (as a base line); later we go back to 0.01d time step.

```python
t, r, v = simulation.orbits2(r0_system, v0_system, masses, t_max=1000, dt=0.001)
analysis.plot_orbits2(r)
```

```python
analysis.analyze_energies(t, r, v, simulation.U_tot, m=masses)
```

Simulate for 1000 d with standard $\Delta t = 0.01\,\text{d}$:

```python
t, r, v = simulation.orbits2(r0_system, v0_system, masses, t_max=1000, dt=0.01)
analysis.plot_orbits2(r)
```

Note that the larger time step of 0.01 d introduces some inaccuracies in the inner orbits, compared to the trajectories for the 0.001 time step.

### Energy conservation with wobble

```python
analysis.analyze_energies(t, r, v, simulation.U_tot, m=masses)
```

Qualitatively, the movement of the central star does not seem to affect the observed orbits and it also does not negatively affect the stability of the integrator. The relative accuracy is about $10^{-4}$, as expected for velocity Verlet's accuracy $\mathcal{O}(\Delta t^2/T)$ (where $T$ is the shortest time scale for an orbit, namely the orbital period of TRAPPIST-1b, 1.51 d) with a time step of $\Delta t/T \approx 10^{-2}$ and hence $\epsilon \approx (10^{-2})^2 = 10^{-4}$.

### Total momentum conservation
Analyze conservation of total system momentum:

$$
\mathbf{P}(t) = \sum_{i=1}^N m_i \mathbf{v}_i(t)
$$

Because we start with $P = 0$ we expect $P(t)$ to be close to 0 during the whole trajectory.

```python
P = np.sum(v * masses[np.newaxis, :, np.newaxis], axis=1)   # total momentum vector P(t)
p = np.sqrt(np.sum(P*P, axis=-1))   # length of momentum vector
plt.plot(t, p)
plt.xlabel("time (d)")
plt.ylabel(r"total momentum $|\mathbf{P}(t)|$")
```

```python
p.max()
```

    7.1329132750160884e-17

The largest deviation is on the order of the machine precision. Therefore, momentum *is conserved* in our simulation to an excellent degree.

### Movement of the star

Plot the movement of the center of the star for the first 5 days; the star's size is shown by an orange circle (which is much bigger than the area covered by the movement).

```python
r_star = r[:int(5/0.01), 0][:, np.newaxis, :]
ax = analysis.plot_orbits_fancy2(r_star)
ax.set_xlabel("x (mAU)")
ax.set_ylabel("y (mAU)")
```

Movement of the star over 1000 d:

```python
r_star = r[:, 0][:, np.newaxis, :]
ax = analysis.plot_orbits_fancy2(r_star)
ax.set_xlabel("x (mAU)")
ax.set_ylabel("y (mAU)")
```

Compare the extend of the movement to the size of the star itself (in mAU):

```python
parameters.star_radius_localunits
```

    0.544182879201903

The orbit of the star TRAPPIST-1a is inside the star itself. Thus the common center of mass of the system is inside the star itself and probably no clear wobble is observable. But we will check.

### Velocity analysis of TRAPPIST-1: Is the wobble detectable with Dopple spectroscopy?

Could the wobble be observed by [Doppler spectroscopy](https://en.wikipedia.org/wiki/Doppler_spectroscopy) measurements? Third generation instruments are sensitive to 0.1 m/s.

Assume ideal observation geometry (namely, observation side on). We approximate the [radial velocity](https://en.wikipedia.org/wiki/Radial_velocity) by the y-component of the velocity, $v_y$. (In principle it is the projection of the velocity on the observation direction but the change in angle is going to be very small and not much different from 0 because the observation distances is so much larger than the extend of the star motion. We could have also picked $v_x$ or any other viewing direction to get a sense for the magnitudes involved.)

Compute the time series of star velocities and look at the maximum differences.

Transform from center-of-mass coordinates (simulation) to lab frame (observer) coordinates:

$$
\mathbf{v}_{*} = -\frac{\sum_{i=1}^{N_{\text{planets}}}
        m_{i}}{M_{\text{star}} + \sum_{i=1}^{N_{\text{planets}}} m_{i}}\mathbf{v}
$$

```python
def v_observer(v_com, M_star, masses_planets):
    return - v_com * masses_planets.sum() / (M_star + masses_planets.sum())
```

```python
v_star_com = v[:, 0]
v_star = v_observer(v_star_com, star['mass'], planet_masses)
vy_star = v_star[:, 1]  # in 1e-3 au/d
vy_star_ms = analysis.au2km(vy_star * 1e-3) * 1e3 / (24*3600)  # in m/s
dt = 0.01
time = dt * np.arange(0, len(vy_star))
```

```python
ax1 = plt.subplot(121)
ax1.plot(time, vy_star_ms)
ax1.set_xlabel(r"time $t$ (d)")
ax1.set_ylabel(r"radial $v_y$ (m/s)")
ax2 = plt.subplot(122)
ax2.plot(time, vy_star_ms)
ax2.set_xlabel(r"time $t$ (d)")
ax2.set_xlim(250, 280)
ax1.figure.tight_layout()
```

```python
vy_max, vy_min = vy_star_ms.max(), vy_star_ms.min()
print("Delta v_radial = {0:.3f} m/s\nmax v_r = {1:.3f} m/s\nmin v_r = {2:.3f} m/s".format(vy_max-vy_min, vy_max, vy_min))
```

    Delta v_radial = 0.004 m/s
    max v_r = 0.002 m/s
    min v_r = -0.002 m/s

Current Doppler spectroscopy can resolve velocities on the order of 1 m/s (and latest generation instruments can do perhaps 0.1 m/s) but the radial velocities for TRAPPIST-1 are still 1-2 orders of magnitude smaller. Therefore, the wobble cannot be detected with Doppler spectroscopy. (Instead, light curver measurements were used.)

## Analyze close planets

We want to count only planets as *nearby* by if we can resolve features of size `feature_size` with the nake eye:

```python
feature_size = 1000 # km
```

Given the angular resolution of the naked eye ($\theta_\text{eye} = 1'$), we look for planets within a cutoff distance $d$ where features of size $y$ are still resolved:

$$
d(y) ≤ \frac{y}{\theta_\text{eye}}
$$

The critical distance is

```python
d_max = analysis.dmax(feature_size)
print("{0:g} km".format(d_max))
print("{0:.1f} mAU".format(analysis.km2au(d_max) * 1e3))
```

    3.43775e+06 km
    23.0 mAU

Analyze the whole trajectory (here we are looking at the trajectory with the star allowed to wobble and so we remove the star from the trajectory with `r[:, 1:]`; qualitatively similar results are obtained when the star is fixed).

```python
counts = analysis.find_nearby_planets(r[:, 1:], ymin=feature_size)
```

Plot the time series for TRAPPIST-1e (index 3 in the `counts` time series):

```python
plt.plot(t, counts[:, 3], color="gray")
plt.xlabel("time (days)")
plt.ylabel(r"number of close planets")
```

Probability distribution for observing close planets:

```python
plt.hist(counts, bins=6, range=(-0.5, 5.5), normed=True, label=planets['name'])
plt.legend(loc="best")
plt.xlabel("number of other nearby planets")
plt.ylabel("probability")
plt.title("Features of size $y ≥ {}$ km are visible".format(feature_size))
plt.xlim(-0.5, 5.5);
```

In general, the inner planets (b, c, d) provide more opportunities to see other planets up close. However TRAPPIST-1e is the planet where one is most likely to see *four* other planets up close and it still has the third highest probability of any planet to have a close up view on *three* planets.

Only TRAPPIST-1d and TRAPPIST-1e see *five* planets at all:

```python
np.sum(counts >= 5, axis=0)
```

    array([  0,   0,  28, 266,   0,   0])

although it happens rarely. Still TRAPPIST-1e is

```python
266/28
```

    9.5

about 9 times more likely to see five planets than the only other contender.

The probability of observing *five* planets is small, but non-zero: the table shows the probability that one can see $N$ other planets from a given planet:

```python
# make probability density (histogram) from the time series of each planet separately
histograms = [np.histogram(counts[:, i], bins=6, range=(-0.5, 5.5), density=True) for i in range(counts.shape[1])]
_, edges = histograms[0]
n = 0.5*(edges[1:] + edges[:-1])  # number of planets

# formatting as a table (using only python)
row_format = "{:6s}" + "{:>8}" * len(n)
rule = "-" * len(row_format.format("x", *n))
print(rule + "\n" +
      row_format.format("planet", *n.astype(int)) + "\n" +
      rule)
for name, (probability, _) in zip(planets['name'], histograms):
    print(row_format.format(name, *np.round(probability * 100, decimals=2)))
print(rule)
print("Probability in % to observe N planets.")
```

    ------------------------------------------------------
    planet       0       1       2       3       4       5
    ------------------------------------------------------
    b        12.85   41.22   36.88    9.06     0.0     0.0
    c        13.11   38.77   35.93   11.51    0.68     0.0
    d        16.67   40.62   33.39     8.2    1.09    0.03
    e         26.6   38.64    23.1    9.49    1.91    0.27
    f        50.24   35.98   12.65    0.87    0.26     0.0
    g        72.11    25.1     2.8     0.0     0.0     0.0
    ------------------------------------------------------
    Probability in % to observe N planets.

Only about 0.27% of the time are 5 planets visible from TRAPPIST-1e, but this is still more than any other; and the probability to observe 4 planets 1.9% is the highest for all planets.

Thus, the poster advertising TRAPPIST-1e speaks the truth: TRAPPIST-1e **is the best destination for planet watching**.

(even though the poster depicts a very rare sight)

```python

```


# Linear elasticity and notations

## TL;DR

In engineering mechanics there are various way how to present equations and theorems. In this post, I'll go through some linear elasticity using index, tensor and matrix notation. All example are made with Julia programming language. Most of the formulas used here can be found from [here](https://en.wikipedia.org/wiki/Hooke%27s_law).

## Linear elasticity

Introduction to linear elasticity from [Wikipedia](https://en.wikipedia.org/wiki/Linear_elasticity):

`Linear elasticity is the mathematical study of how solid objects deform and become internally stressed due to prescribed loading conditions. Linear elasticity models materials as continua. Linear elasticity is a simplification of the more general nonlinear theory of elasticity and is a branch of continuum mechanics. The fundamental "linearizing" assumptions of linear elasticity are: infinitesimal strains or "small" deformations (or strains) and linear relationships between the components of stress and strain. In addition linear elasticity is valid only for stress states that do not produce yielding. These assumptions are reasonable for many engineering materials and engineering design scenarios. Linear elasticity is therefore used extensively in structural analysis and engineering design, often with the aid of finite element analysis.`

Applying force to a continuum leads to deformation of the medium. This results in changes in shape and size. Hooke's Law is equations, used to descibe this relation:

\begin{equation}
F = -kX
\end{equation}

in which F is force, k is the slope and X is the elongation. The analogous of Hooke's spring law for continuous media is then

\begin{equation}
\sigma = − c \epsilon
\end{equation}

in which $\sigma$ is second order stress tensor, c is a fourth-order stiffness tensor end $\epsilon$ is a strain second order strain tensor. In a Cartesian coordinate system, the stress and strain tensors can be represented as:

<div align="center">
$
\sigma = \begin{bmatrix}
       \sigma_{11} & \sigma_{12} & \sigma_{13} \\[0.3em]
       \sigma_{21} & \sigma_{22} & \sigma_{23} \\[0.3em]
       \sigma_{31} & \sigma_{32} & \sigma_{33}
     \end{bmatrix}
$,

$
\epsilon = \begin{bmatrix}
       \epsilon_{11} & \epsilon_{12} & \epsilon_{13} \\[0.3em]
       \epsilon_{21} & \epsilon_{22} & \epsilon_{23} \\[0.3em]
       \epsilon_{31} & \epsilon_{32} & \epsilon_{33}
     \end{bmatrix}
$
</div>

stiffness tensor $c$ can be represented as a 3x3x3x3 matrix, which has a total of 81 individual elements. In this post, we'll be only looking at isotropic media (which have the same physical properties in any direction). When isotropic media is used, $c$ can be constructed from two numbers: $K$ (the bulk modulus ) and $G$ (the shear modulus). [Wikipedia](https://en.wikipedia.org/wiki/Linear_elasticity)

## 1D example using Hooke's Law

Before dive into tensors and matrices, we'll calculate a small 1D example with Hooke's law. Equation for stress:

\begin{equation}
\sigma = E\epsilon
\end{equation}

in which E is the Young's modulus. Strain can be calculated using using engineering strain( https://en.wikipedia.org/wiki/Deformation_(mechanics) ):
\begin{equation}
\epsilon = \frac{\Delta L}{L}
\end{equation}

```julia
# 1D
using Plots

L = 1000.0               # Initial length [mm]
ΔL = linspace(0, 2, 100) # Change of length [mm]

ϵ = ΔL / L               # Strain

E = 200000.0             # Young's modulus for steel [MPa]
σ = E * ϵ                # Stress [MPa]

plot(ϵ, σ)
xlabel!("Strain")
ylabel!("Stress [MPa]")
```

    [Plots.jl] Initializing backend: pyplot

This is the plot, which we want to archive in each example. In the following examples we'll be using the same $L$ and $\Delta L$ also. But, looking good. Now we can just into the real deal.

## Index notation

Index notation (or in relativity theory, Einstein notation) is a hand way to deal with arrays, vector and matrices. Using the isotropic material (media), stress can be defined as:

\begin{equation}
\sigma_{ij} = \lambda \epsilon_{kk}\delta_{ij} + 2\mu \epsilon_{ij}
\end{equation}

in which $\delta$ is the [Kronecker's delta](https://en.wikipedia.org/wiki/Kronecker_delta) and both $\lambda$ and $\mu$ are the [Lamé's constant](# https://en.wikipedia.org/wiki/Lam%C3%A9_parameters), which can be calculated with:
\begin{equation}
\lambda = K - \frac{2}{3}G,\hspace{1cm}\mu = G
\end{equation}

Lastly, we'll define the bulk modulus ($K$) and shear modulus ($G$):

\begin{equation}
K = \frac{E}{3 (1 - 2\nu)},\hspace{1cm}G = \frac{E}{2 (1 + \nu)}
\end{equation}

in which $\nu$ is a poisson's ratio and $E$ is Young's modulus.

```julia
n = 100                      # Number of steps
E = 200000.0                 # Young's modulus for steel [MPa]
ν = 0.3                      # Poisson's ratio for steel

K = E / (3 *(1 - 2*ν))       # Bulk modulus
G = E / (2 * (1 + ν))        # Shear modulus

λ = K - 2/3 * G              # Lamé constant
μ = G                        # Lamé constant
L = 1000.0                   # Initial length [mm]
ΔL = linspace(0, 2, n)       # Change of length [mm]

ϵ_total = ΔL / L             # Total strain

σ = zeros(n, 3, 3)           # Empty stress history

δ = (i, j) -> i == j ? 1 : 0 # Kronecker delta

for t=1:n
    ϵ = zeros(3,3)

    # Collect strain and add poisson effect
    ϵ[1, 1] = ϵ_total[t]
    ϵ[2, 2] = -ϵ_total[t] * ν
    ϵ[3, 3] = -ϵ_total[t] * ν

    # Calculate stress
    for i=1:3
        for j=1:3
            ϵ_kk = sum([ϵ[k, k] for k=1:3])
            σ[t, i, j] += λ * ϵ_kk * δ(i, j) + 2 * μ  * ϵ[i, j]
        end
    end

end

σ_11 = vec(σ[:, 1, 1]);
plot(ϵ_total, σ_11)
xlabel!("Strain")
ylabel!("Stress [MPa]")
```

Plot remains the same, as it should be.

## Tensor notation

Now that we're comfortable using indeces, we'll proceed to tensor notation. Here's a quotation from [wikipedia]() from tensors:

`Tensors are geometric objects that describe linear relations between geometric vectors, scalars, and other tensors. Elementary examples of such relations include the dot product, the cross product, and linear maps. Euclidean vectors, often used in physics and engineering applications, and scalars themselves are also tensors. A more sophisticated example is the Cauchy stress tensor T, which takes a direction v as input and produces the stress T(v) on the surface normal to this vector for output, thus expressing a relationship between these two vectors.
`

Sound intimidating, but tensor are marely a generalization of scalars and vectors. As said in the quotation, scalars are rank 0 tensors and vectors are rank 1 tensors. One of the nice properties of tensors is that they are independent of any chosen frame of reference. Stress can now be defined in tensor notation:

\begin{equation}
\sigma = \lambda \rm{tr}(\epsilon) \bf{I} + \mu \epsilon
\end{equation}

in which $\lambda$ and $\mu$ are Lamé constants, tr is [trace operator](https://en.wikipedia.org/wiki/Trace_%28linear_algebra%29), $\epsilon$ is strain and $\bf{I}$ is an identity tensor:

\begin{equation}
\bf{I}\rm = \begin{bmatrix}
       1 & 0 & 0 \\[0.3em]
       0 & 1 & 0 \\[0.3em]
       0 & 0 & 1
     \end{bmatrix}
\end{equation}

```julia
n = 100                  # Number of steps
E = 200000.0             # Young's modulus [GPa]
ν = 0.3                  # Poisson's ratio

K = E / (3 *(1 - 2*ν))   # Bulk modulus
G = E / (2 * (1 + ν))    # Shear modulus

λ = K - 2/3 * G          # Lamé constant
μ = G                    # Lamé constant

L = 1000.0               # Initial length [mm]
ΔL = linspace(0, 2, n)   # Change of length [mm]

ϵ_total = ΔL / L
I = eye(3)
σ = zeros(n, 3, 3)

for i=1:n
    ϵ = zeros(3,3)
    ϵ[1, 1] = ϵ_total[i]
    ϵ[2, 2] = -ϵ_total[i] * ν
    ϵ[3, 3] = -ϵ_total[i] * ν

    # The most general form of Hooke's law for isotropic materials
    σ[i, :, :] = λ * trace(ϵ) * I + 2 * μ * ϵ
end

σ_11 = vec(σ[:, 1, 1])
plot(ϵ_total, σ_11)
xlabel!("Strain")
ylabel!("Stress [MPa]")
```

Usually (at least what I've studied during these years) the stress equation is presented as:

\begin{equation}
\sigma = C : \epsilon
\end{equation}

in which : is [double dot product](https://en.wikipedia.org/wiki/Dyadics) operation, C is fourth order stiffness tensor and $\epsilon$ is second order strain tensor. Stiffness tensor can be presented as:

\begin{equation}
C = \lambda \bf{I} \otimes \bf{I} + 2\mu \bf{|}
\end{equation}

in which $\bf{I}$ is second order identity tensor, $\otimes$ is an [outer product operation](https://en.wikipedia.org/wiki/Outer_product) and $\bf{|}$ is [fourth order identity tensor](http://www.ce.berkeley.edu/~sanjay/ce231mse211/symidentity.pdf). Equation for fourth order identity tensor:

\begin{equation}
\bf{|_{\rm ijkl}}\rm = \frac{1}{2}(\delta_{i,k} \delta_{j,l} + \delta_{i,l}\delta_{j,k})
\end{equation}

```julia
δ = (i, j) -> i == j ? 1 : 0 # Kronecker delta function

""" Double dot product operation. This implementation works
only for (3,3) matrices """
function double_dot_product(E, ϵ, i, j)
    stress = 0.0
    for k=1:3
        for l=1:3
            stress += E[i, j, k, l] * ϵ[k, l]
        end
    end
    stress
end

""" Function for creating fourth order identity tensor """
function create_4th_symmetric_identity_tensor()
    II = (i,j,k,l) -> 1/2 * (δ(i,k)*δ(j,l) + δ(i,l)*δ(j,k))
    sym_fourth_eye = zeros(3,3,3,3)
    for i=1:3
        for j=1:3
            for k=1:3
                for l=1:3
                    sym_fourth_eye[i,j,k,l] = II(i,j,k,l)
                end
            end
        end
    end
    sym_fourth_eye
end

""" Outer product operation. This implementation works only
for (3,3) matrices """
function outer_product(A, B)
    tensor = zeros(3,3,3,3)
    for i=1:3
        for j=1:3
            for k=1:3
                for l=1:3
                    tensor[i,j,k,l] = A[i, j] * B[k, l]
                end
            end
        end
    end
    tensor
end

n = 100                  # Number of steps
E = 200000.0             # Young's modulus [GPa]
ν = 0.3                  # Poisson's ratio

K = E / (3 *(1 - 2*ν))   # Bulk modulus
G = E / (2 * (1 + ν))    # Shear modulus

λ = K - 2/3 * G          # Lamé constant
μ = G                    # Lamé constant
L = 1000.0               # Initial length [mm]
ΔL = linspace(0, 2, n)   # Change of length [mm]

ϵ_total = ΔL / L
I = eye(3)
σ = zeros(n, 3, 3)

I = eye(3)
sym_fourth_eye = create_4th_symmetric_identity_tensor()

C = λ * outer_product(I, I) + 2*μ*sym_fourth_eye

for t=1:n
    ϵ = zeros(3,3)
    ϵ[1, 1] = ϵ_total[t]
    ϵ[2, 2] = -ϵ_total[t] * ν
    ϵ[3, 3] = -ϵ_total[t] * ν

    # Calculating each element for stress
    for i=1:3
        for j=1:3
            σ[t, i, j] = double_dot_product(C, ϵ, i, j)
        end
    end

end

σ_11 = vec(σ[:, 1, 1])
plot(ϵ_total, σ_11)
xlabel!("Strain")
ylabel!("Stress [MPa]")
```

## Matrix notation

There are justified reasons presenting equations with tensor notation but the
practical implementation of equations is faster using matrix notation. In matrix notation rank of the tensors is so to speak, dropped. Due to symmetricity, $C$ can be presented as a 2D matrix:

\begin{equation}
C = \frac{E}{(1+\nu)(1-2\nu)}\begin{bmatrix}
       1-\nu & \nu & \nu & 0 & 0 & 0 \\[0.3em]
       \nu & 1-\nu & \nu & 0 & 0 & 0 \\[0.3em]
       \nu & \nu & 1-\nu & 0 & 0 & 0 \\[0.3em]
       0 & 0 & 0 & \frac{1-2\nu}{2} & 0 & 0 \\[0.3em]
       0 & 0 & 0 & 0 & \frac{1-2\nu}{2} & 0 \\[0.3em]
       0 & 0 & 0 & 0 & 0 & \frac{1-2\nu}{2}
     \end{bmatrix}
\end{equation}

and the strain tensor can be reduced into vector:

\begin{equation}
\epsilon = \begin{bmatrix}
       \epsilon_{11} \\[0.3em]
       \epsilon_{22} \\[0.3em]
       \epsilon_{33} \\[0.3em]
       2\epsilon_{12} \\[0.3em]
       2\epsilon_{23} \\[0.3em]
       2\epsilon_{13}
     \end{bmatrix}
\end{equation}

The double dot product, used in tensor notation can be now changed into dot product and the stress can be defined as:

\begin{equation}
\sigma = C \cdot \epsilon
\end{equation}

```julia
n = 100                  # Number of steps
E = 200000.0             # Young's modulus [GPa]
ν = 0.3                  # Poisson's ratio

L = 1000.0               # Initial length [mm]
ΔL = linspace(0, 2, n) # Change of length [mm]

ϵ_total = ΔL / L            # Strain in x-direction
I = eye(3)
σ = zeros(n, 6)

C = E / ((1 + ν)*(1 - 2*ν)) * [
    1-ν   ν   ν         0         0         0
      ν 1-ν   ν         0         0         0
      ν   ν 1-ν         0         0         0
      0   0   0 (1-2*ν)/2         0         0
      0   0   0         0 (1-2*ν)/2         0
      0   0   0         0         0 (1-2*ν)/2]

for i=1:n
    ϵ = zeros(6)
    ϵ[1] = ϵ_total[i]
    ϵ[2] = -ϵ_total[i] * ν
    ϵ[3] = -ϵ_total[i] * ν

    # The most general form of Hooke's law for isotropic materials
    σ[i, :, :] = C * ϵ
end

σ_11 = vec(σ[:, 1])
plot(ϵ_total, σ_11)
xlabel!("Strain")
ylabel!("Stress [MPa]")
```

## Discussion

Juggling with tensors, matrices and indeces can be troublesome but in the end, each of them has their own place. Tensors are much handier when presenting the equations and formulas, matrix formulation is easier (at least in my opinion) to implement.


from http://docs.sympy.org/dev/modules/solvers/solvers.html

```python
from sympy import Symbol,nsolve,init_printing,Eq
from IPython.display import display
init_printing(use_latex='mathjax')
import sympy
import mpmath
mpmath.mp.dps = 15
```

```python
Vout = Symbol('V_out')
Vin  = Symbol('V_in')
i    = Symbol('i')
R1   = Symbol('R_1')
R2   = Symbol('R_2')

e1 = Eq(i, Vin/(R1+R2))
e2 = Eq(Vout,i*R2)
display(e1,e2)

Vin  = 10
R1   = 2000
Vout = 7.5

e1 = Eq(i, Vin/(R1+R2))
e2 = Eq(Vout,i*R2)
display(e1,e2)

_a1, _a2 = nsolve((e1,e2),(R2,i),(10000,0.01))

print
print "R2 =",_a1
print "i  =",_a2
```

$$i = \frac{V_{in}}{R_{1} + R_{2}}$$

$$V_{out} = R_{2} i$$

$$i = \frac{10}{R_{2} + 2000}$$

$$7.5 = R_{2} i$$

    R2 = 6000.0
    i  = 0.00125

<hr>
### Triode Modeling

see [Triode Modeling Notebook](https://github.com/holla2040/valvestudio/blob/master/experiments/02-modeling/triode/triode-modeling.ipynb) on the ValveStudio github page

The [Child-Langmuir](http://www.john-a-harper.com/tubes201/) equation for the next sympy exampe

$$I_a = K (V_{gk} + D_aV_{ak})^{3/2}$$

```python
K     = Symbol("K")
Da    = Symbol("D_a")
n     = 3.0/2.0

Vbatt = Symbol("V_batt")
Pdiss = Symbol("P_diss")
Va    = Symbol("V_a")
Vk    = Symbol("V_k")
Vg    = Symbol("V_g")

Ia    = Symbol("I_a")
Vgk   = Symbol("V_gk")
Vak   = Symbol("V_ak")

Rk    = Symbol("R_k")
Rload = Symbol("R_load")

Vbatt = 250
Va    = Vbatt/2

K     = 0.00126098
Da    = 0.01179251
Pdiss = 0.25

e1 = Eq(Va, Vbatt-Rload*Ia)
e2 = Eq(Ia, K*(Vgk + Da*Vak)**n)
e3 = Eq(Vk, Ia * Rk)
e4 = Eq(Vgk, -Vk)
e5 = Eq(Vak,Va - Vk)
e6 = Eq(Pdiss,Ia*Vak)
display(e1,e2,e3,e4,e5,e6)

ans = nsolve((e1,e2,e3,e4,e5,e6),(Rload,Rk,Ia,Vak,Vk,Vgk),(100000,100.0,0.001,100.0,1.0,-1.0))
print ans

```

$$125 = - I_{a} R_{load} + 250$$

$$I_{a} = 0.00126098 \left(0.01179251 V_{ak} + V_{gk}\right)^{1.5}$$

$$V_{k} = I_{a} R_{k}$$

$$V_{gk} = - V_{k}$$

$$V_{ak} = - V_{k} + 125$$

$$0.25 = I_{a} V_{ak}$$

    [   62444.0481110815]
    [   55.9017990964864]
    [0.00200179206475592]
    [   124.888096222163]
    [  0.111903777836926]
    [ -0.111903777836926]

```python

```


```python
from sympy import *
init_printing(pretty_print=True,use_latex=True)
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
f,t,T,G = symbols('f t T G')
```

```python
f = G*t
print('f = ', f)
```

    f =  G*t

```python
plot((f.subs(G,25),(t,0,1)),((f-25).subs(G,25),(t,1,2)),((f-50).subs(G,25),(t,2,3)), ylabel='f = 25 t [V]',xlabel='t [sec]',xlim = (0,3))
```

```python
c0 = integrate(f,(t,0,T))*(1/T)
print(c0)
print(c0.subs([(G,25),(T,1)]))
```

    G*T/2
    25/2

```python
a1 = (2/T)*integrate(f*sin(2*pi*t),(t,0,T))
```

```python
print(a1)
print(a1.subs([(T,1),(pi, 3.14),(G,25)]))
```

    2*G*(-T*cos(2*pi*T)/(2*pi) + sin(2*pi*T)/(4*pi**2))/T
    -7.96178343949045

```python
a,b = [],[]
for n in range(10):
    a.append((2/T)*integrate(f*sin(n*pi*t),(t,0,T)))
    b.append((2/T)*integrate(f*cos(n*pi*t),(t,0,T)))
```

```python
a[9], b[9]
```

```python

c = []
for n in range(1,10):
    print(a[n])
    expr = (a[n]**2 + b[n]**2)
    expr = expr.subs([(T,1),(pi, 3.14),(G,25)])
    # print expr
    # val = lambdify(t,expr,'numpy')
    c.append(expr)
```

    2*G*(-T*cos(pi*T)/pi + sin(pi*T)/pi**2)/T
    2*G*(-T*cos(2*pi*T)/(2*pi) + sin(2*pi*T)/(4*pi**2))/T
    2*G*(-T*cos(3*pi*T)/(3*pi) + sin(3*pi*T)/(9*pi**2))/T
    2*G*(-T*cos(4*pi*T)/(4*pi) + sin(4*pi*T)/(16*pi**2))/T
    2*G*(-T*cos(5*pi*T)/(5*pi) + sin(5*pi*T)/(25*pi**2))/T
    2*G*(-T*cos(6*pi*T)/(6*pi) + sin(6*pi*T)/(36*pi**2))/T
    2*G*(-T*cos(7*pi*T)/(7*pi) + sin(7*pi*T)/(49*pi**2))/T
    2*G*(-T*cos(8*pi*T)/(8*pi) + sin(8*pi*T)/(64*pi**2))/T
    2*G*(-T*cos(9*pi*T)/(9*pi) + sin(9*pi*T)/(81*pi**2))/T

```python
plt.figure()
plt.bar(range(1,10),c)
plt.xlabel('n')
plt.ylabel('Fourier coefficients $a_n^2 + b_n^2$')
```

```python

```


# Logistic Regression
Logistic Regression is a popular and powerful machine learning algorithm that learns the relationship between a target binary variable and one or more input variables. Despite its name, Logistic Regression is in modern terminology a form of classification algorithm rather than regression algorithm. Its name is based on the fact that the core of the algorithm is founded on fundamentally the same mechanisms as Linear Regression.

In Logistic Regression our target variable $Y$ is assumed to be a binary variable that represents some class. For example, the class could be **Sex** with two constituent categories **Male** and **Female**. However we model the problem with a binary target variable. By convention our target $Y \in \{1,0\}$ where:
 * 0 represents one class
 * 1 represents the other class

For a model with 1 predictor feature and our target variable we can assume a training set of $m$ cases:

\begin{equation}
\{ (X^{1},Y^{1}),(X^{2},Y^{2}),(X^{3},Y^{3})\dots(X^{M},Y^{M})               \}
\end{equation}

where, as before, the superscript notation is used to denote a specific training instance.

To demonstrate the form and use of the Logistic Function in the case of a single predictor feature, let us introduce an example based around the relationship between Type II Diabetes and Average Daily Sugar Consumption.

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt

# Target Variable - diagnosed onset of Type II diabetes
Y = np.array([0,0,0,0,1,1,1,1])
# Input Feature - Average daily consumption of added sugars in grammes
X = np.array([5,10,15,20,30,40,45,60])

plt.plot(X,Y, "x")
plt.xlabel('Averae Daily Sugar Consumption')
plt.ylabel('Diagnosis of Type II Diabetes')
plt.ylim([-0.5,1.5])
plt.xlim([0,80])
plt.show()
```

As can be seen there is a clear relationship between the amount of additional sugar consumed daily and the onset of diabetes (it should be noted that this is idealised data and not taken from a specific study). The question is how do we best model this data.

## Modelling Binary Targets with Linear Regression
Before introducing Logistic Regression we first investigate the application of Linear Regression to our example.

We first import two important function definitions that we previously established in the Linear Regression notes. One notable difference is that we separate out the calculation of the hypothesis for linear regression into a separate function. This just makes it easier for us to swap from Linear Regression to Logistic Regression later.

```python
# Standardization over 2D arrays
def standardize(x):
    col_means = np.mean(x,axis=0)
    col_std = np.std(x,axis=0)
    return (x - col_means) / col_std , col_means, col_std

# Linear Regression Function definition
def hLinear(theta_in,X_in):
    return np.matmul(theta_in,X_in)

# Cost function for Linear regression
def cost_linear(theta,xi,yi):
    # note - this function calculate the cost for all variations of theta. A vector is returned.
    h = np.matmul(theta,xi)
    rv = ((h - yi)**2) / 2
    return rv

# The outer cost function 'J' calculator. This iterates over our training cases.
def J(theta,x,y,f):
    s = 0
    m = len(x)
    for i in range(0,m):
        v = f(theta,x[i],y[i])
        s+= v
    return s / m

# implementation of the gradient descent algorithm
def gd(theta_in,min_delta_in,a_in,X_in,Y_in,h,maxiterations=-1,cost=cost_linear):
    m = len(Y_in) # number of training examples
    _ , n = np.shape(X_in) # number of features

    # define a list used to store the costs - these do not have to be calculated but are useful for debugging
    c = []

    # Create a theta variable which we will use to keep track of the current value of theta
    theta = theta_in

    # Create a list which we will use for storing theta results during optimization
    thetas = np.array(theta_in,ndmin=2)

    # Create a counter which is used for limiting the number of iterations used if maxiterations has been defined
    k = 0

    # Begin main search loop
    while maxiterations == -1 or k < maxiterations:
        k+=1
        # log the cost - useful for visualisation and later analysis
        c.append(J(theta_in,X_in,Y_in,cost))

        # save a copy of the parameters -- this is the set we will read from
        temp = np.copy(theta)
        # iterate over all features -- remember though that we apply changes in parallel
        for j in range(0,n):
            # making calculations for jth parameter
            s = 0
            # calculate our gradient update over all training cases
            for i in range(0,m):
                s+=  (h(temp.T,np.ravel(X_in[i,:])) - Y_in[i])  * X_in[i,j]
            theta[j] = temp[j] - a_in * (1/m) * s

        # log the changed parameters
        thetas = np.vstack((thetas,theta))

        # not lets check to see if we are still making progress -- i.e., that our cost is reducing
        # we check to make sure that at least one of our parameters is changing by a minimum threshold
        progressing = False
        for j in range(0,n):
            if(abs(temp[j] - theta[j]) > min_delta_in):
                progressing = progressing | True
        if not progressing: break

    # if we have completed a maximum number of iterations -- or if we are no longer improving -- return
    return thetas, c
```

We begin by first normalizing our data with z-score standardization in the usual way.

```python
X, means, stds = standardize(X)

plt.plot(X,Y, "x")
plt.xlabel('Averae Daily Sugar Consumption (standardised)')
plt.ylabel('Diagnosis of Type II Diabetes')
plt.ylim([-0.5,1.5])
plt.xlim([-2.5,2.5])
plt.show()
```

We can then directly apply Linear Regression to our training data set.

```python
# define initial values for theta
theta = np.ones(2)
theta[0] = np.mean(Y) # remember - we directly calculate t0

# set min_delta and a
md = 0.01
a = 0.75

# define the new matrix X based around our standardized vector for X previously obtained
# remember that x_0 are fake features that we always set to 0 as it makes the calculations easier
x_0 = np.array([1]*len(X))
X = np.vstack((x_0,X)).T

# Run gradient descent
thetas,costs = gd(theta,md,a,X,Y,hLinear)

# plot a number of new points based on the returned function parameters
x1 = np.arange(-2,2,0.2)
x0 = np.array([1]*len(x1))
x = np.vstack((x0,x1)).T
y = np.matmul(thetas[-1,:],x.T)

plt.plot(X[:,1],Y, "x")
plt.xlabel('Averae Daily Sugar Consumption (standardised)')
plt.ylabel('Diagnosis of Type II Diabetes')
plt.ylim([-0.5,1.5])
plt.xlim([-2.5,2.5])
plt.plot(x[:,1],y)
plt.show()
```

Looking at the output of our Linear Regression function we are reminded that the range of the function is $\in R$ and as such  is not a good fit for our classification task. However we can fix this easily by adopting a thresholding policy such that if:
\begin{equation}
  h_{\theta}(x) \geq 0.5 \; then \; y = 1
\end{equation}
\begin{equation}
  h_{\theta}(x) \leq 0.5 \; then \; y = 0
\end{equation}

which can be coded up as follows:

```python
def threshold(x):
    if x >= 0.5: return 1
    else: return 0
threshold = np.vectorize(threshold)

# plot a number of new points based on the returned function parameters
x1 = np.arange(-2,2,0.1)
x0 = np.array([1]*len(x1))
x = np.vstack((x0,x1)).T
y = np.matmul(thetas[-1,:],x.T)
y = threshold(y)

plt.plot(X[:,1],Y, "x")
plt.xlabel('Average Daily Sugar Consumption (normalised)')
plt.ylabel('Diagnosis of Type II Diabetes')
plt.ylim([-0.5,1.5])
plt.xlim([-2.5,2.5])
plt.plot(x[:,1],y)
plt.show()
```

#### Problems with Linear Regression

Looking at the output of our function it seems to have given us a relatively good approximation of our data. All data points seem aligned with the function output. However this simple thresholded function has two major limitations:
 * First, our thresholded function is not a smooth function. This is important when it comes to training algorithms which often depend on having a continuous function derivative.
 * Second, the function performs poorly when the distributions of the data are not so ideal

To understand this second point lets extend our diabetes example with a number of other data points.

```python
# define data and standardize
Y = np.array([0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1])
X = np.array([0,1,2,3,4,7,9,10,15,30,40,45,60,100,120,124,124,140])
X, means, stds = standardize(X)

# define initial values for theta
theta = np.ones(2)
theta[0] = np.mean(Y) # remember - we directly calculate t0

# set min_delta and a
md = 0.01
a = 0.75

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*len(X))
X = np.vstack((x_0,X)).T

# Run gradient descent
thetas,costs = gd(theta,md,a,X,Y,hLinear)

# plot a number of new points based on the returned function parameters
x1 = np.arange(-2,2,0.2)
x0 = np.array([1]*len(x1))
x = np.vstack((x0,x1)).T
y = np.matmul(thetas[-1,:],x.T)
y = threshold(y)

plt.plot(X[:,1],Y, "x")
plt.xlabel('Average Daily Sugar Consumption')
plt.ylabel('Diagnosis of Type II Diabetes')
plt.ylim([-0.5,1.5])
plt.xlim([-2.5,2.5])
plt.plot(x[:,1],y)
plt.show()
```

In this case we can see that a number of instances of label 1 are now incorrectly going to be labeled as 0. This is in spite of the fact that there is still a very clear boundary between cases of 0 and cases of 1.

In summary we can use Linear Regression to solve problems like this, but it is not a particularly good solution. We will see that Logistic Regression gives us a solution that is specifically tailored to the binary label problem.

## The Logistic Function
Logistic Regression is based around the idea that a particular type of function, the Logistic Function, provides a better fit to our data than a straightforward linear function or even a thresholded linear function. The logistic function is of the form:

\begin{equation}
 h_{\theta}(x) = \frac{1}{1+e^{-z}}
\end{equation}

where $z$ can be any real number but in our case is usually taken to be $\theta^{T}X$, i.e., the result of a linear function as we have used in Linear Regression. This value z is often called the logit.

We can gain an intuition about the structure of the Logistic Function by visualizing its components.

```python
x1 = np.arange(-5,5,0.1)
x0 = np.array([1]*len(x1))
x = np.vstack((x0,x1))
z = np.matmul(np.array([0,1]),x)

plt.plot(x1,1+np.exp(-z))
plt.xlim([-5,5])
plt.ylim([-3,50])
plt.title('Function f(x) = 1 + exp(-(0 + x))')
plt.ylabel("f(x)")
plt.xlabel("x")
plt.show()
plt.plot(x1,1/(1 + np.exp(-z)))
plt.xlim([-5,5])
plt.ylim([-2,2])
plt.title('Function f(x) = 1 / (1 + exp(-(0 + x)))')
plt.ylabel("f(x)")
plt.xlabel("x")
plt.show()
```

We see that the function $f(x) = 1+e^{-x}$ is a vertically displaced inverse exponential function which has a limit of $1$ in the positive x direction and a limit of infinity in the negative x direction. Our Logistic Function is the reciprocal value of this function. Considering the limits just mentioned, the limits of the reciprocal are therefore $\frac{1}{\infty} = 0$ in the negative x direction and $\frac{1}{1} = 1$ in the positive x direction. This gives us a convenient s-shaped function that is bounded between 0 and 1.

Using the vector notation introduced for Linear Regression we can re-write the Logistic Function as follows:

\begin{equation}
 h_{\theta}(x) = g(\theta^{T}x)
\end{equation}

where

\begin{equation}
 g(z) = \frac{1}{1+e^{-z}}
\end{equation}

This is the general form of the Logistic Function that we will use throughout this section. We see that the parameter $z$ is taken to be the product of $\theta$ and $x$ as used in the linear regression equation. At this stage it is worth noting that this is a very important function in Deep Learning and is worth paying attention to.

The severity of the transition between 0 and 1 is controlled by the parameter $\theta_{1}$. We can illustrate this with a number of examples:

```python
x1 = np.arange(-3,3,0.1)
x0 = np.array([1]*len(x1))
x = np.vstack((x0,x1))
z0 = np.matmul(np.array([0,0.5]),x)
z1 = np.matmul(np.array([0,1]),x)
z2 = np.matmul(np.array([0,2]),x)
z3 = np.matmul(np.array([0,4]),x)

plt.plot(x1,1/(1 + np.exp(-z0)))
plt.plot(x1,1/(1 + np.exp(-z1)))
plt.plot(x1,1/(1 + np.exp(-z2)))
plt.plot(x1,1/(1 + np.exp(-z3)))
plt.xlim([-3,3])
plt.ylim([-2,2])
plt.ylabel("f(x)")
plt.xlabel("x")
plt.legend(['theta1 = 0.5','theta1 = 1', 'theta1 = 2', 'theta1 = 4'], loc='lower right')
plt.show()
```

Similarly we can see that the logistic function is displaced along the x axis through the parameter $\theta_{0}$.

```python
x1 = np.arange(-5,5,0.1)
x0 = np.array([1]*len(x1))
x = np.vstack((x0,x1))
z1 = np.matmul(np.array([0,1]),x)
z2 = np.matmul(np.array([1,1]),x)
z3 = np.matmul(np.array([2,1]),x)
z4 = np.matmul(np.array([3,1]),x)

plt.plot(x1,1/(1 + np.exp(-z1)))
plt.plot(x1,1/(1 + np.exp(-z2)))
plt.plot(x1,1/(1 + np.exp(-z3)))
plt.plot(x1,1/(1 + np.exp(-z4)))
plt.xlim([-5,5])
plt.ylim([-2,2])
plt.ylabel("f(x)")
plt.xlabel("x")
plt.legend(['theta0 = 0', 'theta0 = 1', 'theta0 = 2', 'theta0 = 3'], loc='lower right')
plt.show()
```

### Aside: Logistic Functions & Sigmoid Functions

The Logistic Function is one instance of the class of Sigmoid Functions which all broadly have an s-shape and are bounded between certain limits. Some other important examples of sigmoid functions include:

\begin{equation}
f(x) = tanh(x)
\end{equation}

\begin{equation}
f(x) = \frac{x}{1 + |x|}
\end{equation}

We illustrate these below alongside the logistic function. Note that not all logistic functions are naturally bounded between 0 and 1.

```python
plt.plot(x1,1/(1 + np.exp(-z1)))
plt.plot(x1,np.tanh(x1))
plt.plot(x1,x1 / (1 + abs(x1)))

plt.xlim([-3,3])
plt.ylim([-2,1.5])
plt.ylabel("f(x)")
plt.xlabel("x")
plt.title("3 Sigmoid Functions")
plt.legend(['f(x) = 1/(1 + np.exp(-z1))', 'f(x) = tanh(x)', 'f(x) = x1 / (1 + abs(x1))'], loc='lower right')
plt.show()
```

### Interpreting the Logistic Function
As noted the Logistic Function gives an output in [0,1]. This helps us to give our logistic function based model $h_{\theta}$ a  probabilistic interpretation. Namely we interpret $h_{\theta}(x)$ as the estimated probability that $y=1$ given an input $x$. More formally

\begin{equation}
h_{\theta}(x) = P(Y=1 \;|\; x_{i},\theta)
\end{equation}

This gives us a good mechanism for balancing out the fact that we know $Y\in\{0,1\}$ while $h_{\theta}(x) \in [0,1]$.   From this we can also obtain:

\begin{equation}
P(Y=0 \;|\; x_{i},\theta) = 1 - h_{\theta}(x)
\end{equation}

We can also interpret our hypothesis function as defining a **Decision Boundary** between cases where we expect $Y=1$ and $Y=0$. In general we expect $Y=1$ where $h_{\theta}(x)>=0.5$ and $Y=0$ where $h_{\theta}(x)<0.5$.

### Applying Logistic Function to the Diabetes Dataset
Given our definition of the Logistic function, we can manually fit a logistic function to our Diabetes data from above.

```python
plt.plot(X[:,1],Y, "x")
plt.xlabel('Average Daily Sugar Consumption')
plt.ylabel('Diagnosis of Type II Diabetes')

x1 = np.arange(-2.5,3,0.1)
x0 = np.array([1]*len(x1))
x = np.vstack((x0,x1))
z1 = np.matmul(np.array([15,30]),x)

plt.plot(x1,1/(1 + np.exp(-z1)))
plt.ylim([-0.5,1.5])
plt.xlim([-2.5,2.5])
plt.show()
```

We can see that as hoped this provides a significantly better fit to our data than our original thresholded linear function. The step between g(z) = 0 and g(z) = 1 has now been displaced to better fit where this transition occurs in our data. The function itself is now also smooth. Of course, we have manually selected our parameters, now we need to consider how to achieve this automatically through optimization.

## Logistic Cost Function
Optimization of the Logistic Function requires that we find the set of parameters $\theta$ that produces a function that best matches our training data. We will see that we don't however use the Squared Error cost function for this, instead we will use a cost function specific to Logistic Regression.

That said, before we introduce this new cost function, let us first consider the option of directly using the Squared Error cost function. The code below plots the sum of square error cost for a range of values for X and Y.

```python
from matplotlib import cm

def cost(theta,xi,yi):
    # note - this function calculate the cost for all variations of theta. A vector is returned.
    z = np.matmul(theta,xi)
    h = 1 / (1 + np.exp(-z))
    rv = ((h - yi)**2) / 2
    return rv

def J(theta,x,y,f):
    s = 0
    m = len(x)
    for i in range(0,m):
        v = f(theta,x[i],y[i])
        s+= v
    return s / m

theta0_range = np.arange(-20,20,0.5)
theta1_range = np.arange(-20,20,0.5)

theta0s, theta1s = np.meshgrid(theta0_range, theta1_range)
mesh_rows, mesh_cols = theta0s.shape

theta0s_id = np.ravel(theta0s)
theta1s_id = np.ravel(theta1s)
theta = np.vstack((theta0s_id,theta1s_id)).T   # 10000 x 2 10000 different theta sets

costs = J(theta,X,Y,cost)

# reshape result to have the same organization as result of meshgrid \n",
costs = np.reshape(costs,(mesh_rows,mesh_cols))

plt.figure()
plt.contour(theta0s,theta1s,costs)
plt.title("cost space for theta0 and theta1")
plt.show()

theta1_range = np.arange(-20,20,0.5)
theta0_range = np.array([0.0]*len(theta1_range))
theta = np.vstack((theta0_range,theta1_range)).T   # 10000 x 2 10000 different theta sets
costs = J(theta,X,Y,cost)

plt.figure()
plt.plot(theta1_range,costs)
plt.title("Cost space for theta1 where theta0 = 0.5")
plt.show()
```

We can see that in this case the error space now looks a lot more complex than for linear regression. Our bowl-like convex function has been replaced with a non-convex function. While this may not strictly speaking be a problem for the current example the non-convex cost function can lead to non-optimal solutions in more complex examples.

### Cross Entropy Loss Function

For Logistic Regression we base our cost function on the following cost heuristic:

\begin{equation}
cost(h_{\theta}(x^{(i)}),y^{(i)}) = -y^{(i)} log(h_{\theta}(x^{(i)}))   - (1-y) log(1 - h_{\theta}(x^{(i)}))
\end{equation}

which is referred to as the Cross Entropy Cost Function. This function may look a bit intimidating at first, but it is actually very straightforward and almost as easy to interpret as the Sum of Squared Error Cost Function.

To begin with, there are two interesting notes about this cost function. First, the cost function is calculated differently depending on whether our target value is 0 or 1. In other words, the function has two distinct halves, and only one of these is in play depending on whether our label is 0 or 1.

Second, the cost function heavily penalizes cases where the hypothesis function gives the opposite of what is expected.

We can illustrate these two properties by plotting our cost function for the cases of Y=1 and Y=0.

```python
h = np.arange(0.001,0.999,0.01)
plt.figure()
plt.xlabel('h(x)')
plt.ylabel('Cost')
plt.plot(h,-np.log(h))
plt.title("Logistic Cost Function where Y=1")
plt.show()

plt.xlabel('h(x)')
plt.ylabel('Cost')
plt.plot(h,-np.log(1-h))
plt.title("Logistic Cost Function where Y=0")
plt.show()
```

An important technical point to note is that the cost function is dependent on calculating logs of the sigmoid function. This calculation is dependent on the fact that the sigmoid function should in the strict mathematical sense converge towards but never reach 0 and 1. If when y=1 h is exactly 0 then the log is undefined. Similarly when y=0 and z=1 the log is similarly not defined. This explains our reason for selecting bounds of 0.001 and 0.999 in our illustration of the logistic cost functions above.

While the sigmoid function should not produce either an output that is exactly equal to 0 or 1, rounding will in practice lead to this occurring. For this reason we implement the cost function with an if else construct to avoid calculating the log where a singularity would occur.

### Logistic Cost Function Formal Definition
We can take our cost heuristic defined above and build it in to a full formal cost definition for $J(\theta)$ as follows.

Looking again at a formal definition of cost for multivariate Linear Regression we have

\begin{equation}
J(\theta) = \frac{1}{2m}     \sum_{i=0}^{m}  (h_{\theta}(x^{(i)}) - y^{(i)})^{2}
\end{equation}

where
 * $\theta$ is the vector of parameters of our model
 * $m$ is the number of training examples
 * $y^{i}$ is the ith target value
 * $x^{i}$ is the ith feature, and
 * $h_{\theta}$ is our model / function of $x$ that we wish to evaluate.
 * 2, or rather $\frac{1}{2}$, is a constant we introduce to allow the calculation of the derivative of J to be more straightforward.

We can generalize this cost function $J(\theta)$ by moving our derivative constant inside the summation

\begin{equation}
J(\theta) = \frac{1}{m}     \sum_{i=0}^{m} \frac{1}{2} (h_{\theta}(x^{(i)}) - y^{(i)})^{2}
\end{equation}

And then separating out the summation argument into a separate function as follows:

\begin{equation}
J(\theta) = \frac{1}{m}     \sum_{i=0}^{m} cost_{linear}(h_{\theta}(x^{(i)}),y^{(i)})
\end{equation}

where

\begin{equation}
cost_{linear}(h_{\theta}(x^{(i)}),y^{(i)})  = \frac{1}{2} (h_{\theta}(x^{(i)}) - y^{(i)})^{2}
\end{equation}

Using the general template for $J(\theta)$ we can substitute our heuristic for Logistic Regression to obtain the following function:

 \begin{equation}
J(\theta) = \frac{1}{m} \sum_{i=1}^{m} cost_{logistic}(h_{\theta}(x^{(i)}),y^{(i)})
\end{equation}

where

\begin{equation}
cost_{logistic}(h_{\theta}(x^{(i)}),y^{(i)}) = -y^{(i)} log(h_{\theta}(x^{(i)}))   - (1-y^{(i)}) log(1 - h_{\theta}(x^{(i)}))
\end{equation}

Note that our general template for $J(\theta)$ simply states that we average over individual costs for each data point. Costs for individual data points are calculated very differently between our original Linear Regression cost function and our Logistic Regression Cost Function.

We can illustrate the properties of our Logistic Cost function directly with a number of examples with respect to our Diabetes data set.

```python
def cost_logistic(theta,xi,yi):
    # note - this function calculate the cost for all variations of theta. A vector is returned.
    z = np.matmul(theta,xi)
    h = 1 / (1 + np.exp(-z))
    if yi == 1:
        return -np.log(h)
    elif yi == 0:
        return -np.log(1-h)
    else: raise ValueError('yi was neither 0 nor 1: ' + str(yi))

costs = J(np.array([15.0,30.0]),X,Y,cost_logistic)
print("Costs : " + str(costs))

# shift to the left - all y=0 cases should fail
costs = J(np.array([30.0,30.0]),X,Y,cost_logistic)
print("Costs : " + str(costs))

# shift to the right - some y=1 cases should fail
costs = J(np.array([0.0,30.0]),X,Y,cost_logistic)
print("Costs : " + str(costs))

# shift to the right - some y=1 cases should fail
costs = J(np.array([1.0,1.0]),X,Y,cost_logistic)
print("Costs : " + str(costs))

# shift to the right - some y=1 cases should fail
costs = J(np.array([0.0,0.0]),X,Y,cost_logistic)
print("Costs : " + str(costs))

```

    Costs : 0.0015778308789776054
    Costs : 2.7755862143275576
    Costs : 0.8243068625612893
    Costs : 0.4914370094254397
    Costs : 0.6931471805599453

Similarly we can plot a contour map of the logistic regression cost function over a range of parameter values.

```python
theta0_range = np.arange(-10,10,0.2)
theta1_range = np.arange(-10,10,0.2)

theta0s, theta1s = np.meshgrid(theta0_range, theta1_range)
mesh_rows, mesh_cols = theta0s.shape

theta0s_id = np.ravel(theta0s)
theta1s_id = np.ravel(theta1s)
theta = np.vstack((theta0s_id,theta1s_id)).T   # 10000 x 2 10000 different theta sets

costs = J(theta,X,Y,cost_logistic)

# reshape result to have the same organization as result of meshgrid \n",
costs = np.reshape(costs,(mesh_rows,mesh_cols))

plt.figure()
plt.contour(theta0s,theta1s,costs)
plt.xlabel('Theta0')
plt.ylabel('Theta1')
plt.title("cost space for theta0 and theta1")
plt.show()
```

We can see from the above that we once again have our conventional convex cost function with a bowl shape.

### Gradient Descent for Logistic Regression

The Gradient Descent algorithm as defined in the notes for Linear Regression was dependent on the partial derivative of the linear regression Cost Function $J(\theta)$ rather than the Cost Function itself. The same is true here for Logistic Regression. In order to implement gradient descent for Logistic Regression we must calculate the partial derivatives of the Logistic Regression Cost Function.

Fortunately the partial derivatives of the Logistic Regression Cost Function is identical to that for Linear Regression with the exception that we now use the Logistic Regression definition of our hypothesis function rather than the Linear Regression form. Specifically:

\begin{equation}
\frac{\partial}{\partial \theta_{j}}J_{logistic}(\theta) =  \frac{1}{m} \sum_{i=0}^{m} (h_{logistic,\theta}(x^{(i)}) - y^{(i)}).x^{(i)}
\end{equation}

Given this we can easily repurpose our Gradient Descent algorithm to perform search of the Logistic Function space by providing a new definition for our hypothesis function.

```python
def hLogistic(theta_in,X_in):
    z = np.matmul(theta_in,X_in)
    return 1 / (1 + np.exp(-z))
```

We can test this new definition by applying it to our normalized training data.

```python
# define initial values for theta
rows, cols = X.shape
theta = np.array([1.0,1.0])

print("Theta is : " + str(theta))
a = 10
md = 0.01
# Run gradient descent
print("md is : " + str(md))
print("a is  : " + str(a))
thetas, c = gd(theta,md,a,X,Y,hLogistic,cost=cost_logistic)
print("t is  : " + str(thetas[-1]))

plt.plot(c, 'ro')
plt.xlabel('Iterations')
plt.ylabel('Cost')
plt.title("Cost Values at each iteration")
plt.show()
```

## Multivariate Logistic Regression
We can easily extend our method above to create a multivariate example with two features. In this case our data is idealized to show a linear decision boundary over two dimensions.

```python
from mpl_toolkits.mplot3d import Axes3D

# Define the training data
Y = np.array([0,0,0,1,1,1])
X1 = np.array([0,1,0,4,3,4])
X2 = np.array([0,0,1,1,2,2])
X = np.vstack((X1,X2)).T

# Plot the training data in our usual way where the Target variable occupies the z-axis
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X1, X2, Y, c='r', marker='o')
plt.show()

# standardize our input features
X, means, stds = standardize(X)

# define initial values for theta
rows, cols = X.shape
theta = np.ones(1+cols)*1

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*rows)
X = np.hstack((x_0[:, np.newaxis],X))

# Perform Gradient Descent
thetas,c = gd(theta,md,a,X,Y,hLogistic)

# plot a surface based on the returned function parameters

# initialize values for features x1 and x2
x_range = np.arange(-2,2,0.1)

x1s, x2s = np.meshgrid(x_range, x_range) # remember - plt.surface likes this meshgrid output
mesh_rows, mesh_cols = x1s.shape

# construct 1D arrays for x1, x2 and x0
x1 = np.ravel(x1s)
x2 = np.ravel(x2s)
x0 = np.array([1]*len(x2))

# merge 1D arrays into a single x array for calculations",
x = np.matrix([x0,x1,x2])

# perform calculations",
z = hLogistic(thetas[-1,:],x)

# reshape result to have the same organization as result of meshgrid \n",
zs = np.reshape(z,(mesh_rows,mesh_cols))

# plot the surface along with our training data points
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(x1s, x2s, zs, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
surf = ax.scatter(X[:,2], X[:,1], Y, c='r', marker='o')
ax.set_xlabel('Average Rainfall')
ax.set_ylabel('Temperature')
ax.set_zlabel('Growth')
ax.set_zlim(-0.2, 1.2)
plt.show()
```

## Non-Linear Logistic Regression
As with Linear Regression we can extend the method to provide non-linear decision boundaries simply by introducing extra non-linear functions of our data.

For example, lets consider the case where positive examples are clustered together and negative examples form a ring around that cluster as illustrated below.

```python
# Define the training data
Y = np.array([ 0,0,0,0,0,       1,1,1,  0,0,    1,1,1,     0,0,0,0,0])
X1 = np.array([4,6,1,9,1,       4,5,6,  9,1,    4,5,6,     9,1,9,4,6])
X2 = np.array([0,0,1,1,2,       2,2,2,  2,3,    3,3,3,     3,4,4,5,5])
X = np.vstack((X1,X2)).T

# Plot the training data in our usual way where the Target variable occupies the z-axis
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X1, X2, Y, c='r', marker='o')
plt.show()
```

Or with normalized data:

```python
# standardize our input features
X, means, stds = standardize(X)

# Plot the training data in our usual way where the Target variable occupies the z-axis
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X[:,0], X[:,1], Y, c='r', marker='o')
plt.show()
```

If we attempt to model this data distribution directly we see that the results are poor. We have to define a max number of iterations as otherwise the gradient descent algorithm will fail to find a solution that is useful and will instead bounce backwards and forward around the parameter space.

```python
# define initial values for theta
rows, cols = X.shape
theta = np.ones(1+cols)*1

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*rows)
X = np.hstack((x_0[:, np.newaxis],X))

# Perform Gradient Descent
thetas,c = gd(theta,md,a,X,Y,hLogistic,maxiterations=1000)

# plot a surface based on the returned function parameters

# initialize values for features x1 and x2
x_range = np.arange(-2,2,0.1)

x1s, x2s = np.meshgrid(x_range, x_range) # remember - plt.surface likes this meshgrid output
mesh_rows, mesh_cols = x1s.shape

# construct 1D arrays for x1, x2 and x0
x1 = np.ravel(x1s)
x2 = np.ravel(x2s)
x0 = np.array([1]*len(x2))

# merge 1D arrays into a single x array for calculations",
x = np.matrix([x0,x1,x2])

# perform calculations",
z = hLogistic(thetas[-1,:],x)

# reshape result to have the same organization as result of meshgrid \n",
zs = np.reshape(z,(mesh_rows,mesh_cols))

# plot the surface along with our training data points
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(x1s, x2s, zs, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
surf = ax.scatter(X[:,2], X[:,1], Y, c='r', marker='o')
ax.set_xlabel('Average Rainfall')
ax.set_ylabel('Temperature')
ax.set_zlabel('Growth')
ax.set_zlim(-0.2, 1.2)
plt.show()
```

The surface when viewed in this way may give the impression that a separation has been achieved. However that is not in practice what is happening. Essentially the plane that has been returned is stating that for all locations the function s giving a roughly 50% probability that the value at that point will be a 1. This is not what we want. We want the function to closely match our data such that the function predicts 1 where we have positive cases and 0s where we have negative cases.

An circular decision boundary should give us a good approximation for this data. We would expect such a function to be of the form:

\begin{equation}
X^{2} + Y^{2} + aX + bY +c = 0
\end{equation}

We can achieve this by adding in two additional features which are squares of our original data.

```python
# Define the training data
Y = np.array([ 0,0,0,0,0,       1,1,1,  0,0,    1,1,1,     0,0,0,0,0])
X1 = np.array([4,6,1,9,1,       4,5,6,  9,1,    4,5,6,     9,1,9,4,6])
X2 = np.array([0,0,1,1,2,       2,2,2,  2,3,    3,3,3,     3,4,4,5,5])
X3 = X1**2
X4 = X2**2
X = np.vstack((X1,X2,X3,X4)).T

# standardize our input features
X, means, stds = standardize(X)

# define initial values for theta
rows, cols = X.shape
theta = np.ones(1+cols)*1

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*rows)
X = np.hstack((x_0[:, np.newaxis],X))

# Perform Gradient Descent
a = 5
thetas,c = gd(theta,md,a,X,Y,hLogistic,maxiterations=1000)

# plot a surface based on the returned function parameters

# initialize values for features x1 and x2
x_range = np.arange(0.,10.,0.1)
y_range = np.arange(0.,6.,0.1)

x1s, x2s = np.meshgrid(x_range, y_range) # remember - plt.surface likes this meshgrid output
mesh_rows, mesh_cols = x1s.shape

# construct 1D arrays for x1, x2 and x0
x1 = np.ravel(x1s)
x2 = np.ravel(x2s)
x3 = x1**2
x4 = x2**2
x = np.vstack((x1,x2,x3,x4)).T

# rescale our data based on our normalization parameters
x =  (x-means)/stds

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*len(x2))
x = np.hstack((x_0[:, np.newaxis],x))

# perform calculations
z = hLogistic(thetas[-1,:],x.T)

# reshape result to have the same organization as result of meshgrid
zs = np.reshape(z,(mesh_rows,mesh_cols))
x1s = np.reshape(x[:,1],(mesh_rows,mesh_cols))
x2s = np.reshape(x[:,2],(mesh_rows,mesh_cols))

# plot the surface along with our training data points
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(x1s, x2s, zs, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
surf = ax.scatter(X[:,1], X[:,2], Y, c='g', marker='o')
ax.set_zlim(-0.2, 1.2)
plt.show()
```

Rather than using up an axis dimension to explicitly visualize the Target variable as in the case above we can instead collapse this dimension and instead represent different classes with different icons. For example we can re-visualize our results below in 2D by plotting the points and the decision space as a contour plot.

```python
for l in range(len(Y)):
    if Y[l] == 0:
        plt.plot(X[l,1],X[l,2],'o',c='r')
    elif Y[l] == 1:
        plt.plot(X[l,1],X[l,2],'o',c='b')
    else: print(Y[l])

plt.contour(x1s, x2s, zs)
plt.ylim([-2,2])
plt.show()
```

Rather than printing the entire decision surface as a contour it is typically enough just to print the decision boundary, i.e., the line at which $P(Y=1)=0.5$.

```python
for l in range(len(Y)):
    if Y[l] == 0:
        plt.plot(X[l,1],X[l,2],'o',c='r')
    elif Y[l] == 1:
        plt.plot(X[l,1],X[l,2],'o',c='b')
    else: print(Y[l])

plt.contour(x1s, x2s, zs, 1)
plt.ylim([-2,2])
plt.show()
```

## Multinomial Logistic Regression
In the example to this point our Target Variable has been a single binary variable. In many practical examples we will instead have a target variable that is multinomial, e.g., 3 different colours, or 4 different disease types.

Logistic Regression cannot directly cope with more than two class types. However we can easily adopt our usage of Logistic Regression to handle more than two classes by applying **one hot encoding** to the target variable. **One Hot Encoding** is the rewriting of a single variable with greater than 2 classes into a number of individual binary variables where each binary variable indicates the presence of one of the original 3 classes.

To illustrate consider the multinomial target Y defined as a Python list alongside the two input features X1 and X2.

```python
# Define the data - r for red, g for green, and b for blue
Y = ['b','b','r','b','g','r','g','r','g']
X1 = [1,1,4,2,2,3,1,4,2]
X2 = [1,2,4,2,3,3,4,2,4]

# Visualize as a multinomial problem
for i in range(len(Y)):
    plt.plot(X1[i],X2[i],'o',c=Y[i])
plt.ylim([0,5])
plt.xlim([0,5])
plt.show()
```

To recode this case we create 3 new target binary variables $Yr$, $Yg$, and $Yb$, where each variable captures where $Y$ was $r$, $g$, or $b$ respectively.

```python
Yr = [1 if a =='r' else 0 for a in Y]
Yg = [1 if a =='g' else 0 for a in Y]
Yb = [1 if a =='b' else 0 for a in Y]

# Visualize as a 3 binary problem
for i in range(len(Yr)):
    if(Yr[i] == 1):
        plt.plot(X1[i],X2[i],'o',c='r')
    else:
        plt.plot(X1[i],X2[i],'o',c='c')
plt.ylim([0,5])
plt.xlim([0,5])
plt.show()

for i in range(len(Yg)):
    if(Yg[i] == 1):
        plt.plot(X1[i],X2[i],'o',c='g')
    else:
        plt.plot(X1[i],X2[i],'o',c='c')
plt.ylim([0,5])
plt.xlim([0,5])
plt.show()

for i in range(len(Yb)):
    if(Yb[i] == 1):
        plt.plot(X1[i],X2[i],'o',c='b')
    else:
        plt.plot(X1[i],X2[i],'o',c='c')
plt.ylim([0,5])
plt.xlim([0,5])
plt.show()
```

Rather than training one single model we end up training n distinct models - one for each of our new target variables. For a given model $i$ we are computing:

\begin{equation}
h_{\theta}^{(i)} (x) = P(y=i \;|\ x,\theta)
\end{equation}

i.e., the interpreted likelihood of the data $x$ resulting in the activation of the model with the specific parameters $\theta$. For a new data point we then simply want to produce the hypothesis for each trained model and select the maximum of these models:

\begin{equation}
\max_i h_{\theta}^{(i)}
\end{equation}

```python
X = np.vstack((X1,X2)).T

# standardize our input features
X, means, stds = standardize(X)

# define initial values for theta
rows, cols = X.shape
theta = np.ones(1+cols)*1

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*rows)
X = np.hstack((x_0[:, np.newaxis],X))

# Perform Gradient Descent
a = 5
thetas_r,c_r = gd(theta,md,a,X,Yr,hLogistic,maxiterations=1000)
thetas_g,c_g = gd(theta,md,a,X,Yg,hLogistic,maxiterations=1000)
thetas_b,c_b = gd(theta,md,a,X,Yb,hLogistic,maxiterations=1000)

# Define some new data point
x_1 = [0,4]
x_2 = [0,3]

x = np.vstack((x_1,x_2)).T

# rescale our data based on our normalization parameters
x =  (x-means)/stds

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*len(x_2))
x = np.hstack((x_0[:, np.newaxis],x))

# perform calculations
z_r = hLogistic(thetas_r[-1,:],x.T)
print(z_r)

z_g = hLogistic(thetas_g[-1,:],x.T)
print(z_g)

z_b = hLogistic(thetas_b[-1,:],x.T)
print(z_b)

```

    [2.32043769e-10 9.99996921e-01]
    [2.93381016e-11 3.46136429e-09]
    [1.0000000e+00 3.7122262e-09]

Therefore we can see that for multinomial logistic regression we need to train n different models for n different target labels. We will take a different approach in neural networks where we will train one single model with shared weights even in the case of a multinomial target.

## Logistic Regression the Easy Way
We can use scipy to perform Logistic Regression using the standard model interface.

```python
from sklearn import linear_model

# Create the Logistic Regression model instance
logreg = linear_model.LogisticRegression(C=1e5)

# Fit the data
model = logreg.fit(X, Y)

print(model)
```

    LogisticRegression(C=100000.0, class_weight=None, dual=False,
                       fit_intercept=True, intercept_scaling=1, l1_ratio=None,
                       max_iter=100, multi_class='auto', n_jobs=None, penalty='l2',
                       random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                       warm_start=False)

See SKLearn Logistic Regression 3-class Classifier for more detailed example.

Complete Modules/Deep Learning/week 3/Logistic Regression.ipynb


```python
# Double resolution figures for MacOSX retina displays
%config InlineBackend.figure_format = 'retina'
```

# NumPy
* Makes working with N-dimensional arrays (e.g. vectors, matrices, tensors) effecient.
* NumPy functions are written in C, so they're fast. In fact, Python itself is written in C.

```python
import numpy as np  # import numpy module
```

## N-dimensional arrays

```python
a = np.array([1, 2, 3, 4, 5])
print(a)
print()
print("a.shape = ", a.shape)
print("type(a) = ", type(a))
print("a.dtype = ", a.dtype)
```

    [1 2 3 4 5]

    arr.shape =  (5,)
    type(arr) =  <class 'numpy.ndarray'>
    arr.dtype =  int64

```python
a = np.zeros((3, 3))  # 3x3 random matrix
print(a)
print()
print("a.shape = ", a.shape)
print("type(a) = ", type(a))
print("a.dtype = ", a.dtype)
```

    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]

    arr.shape =  (3, 3)
    type(arr) =  <class 'numpy.ndarray'>
    arr.dtype =  float64

```python
a = np.zeros((3, 3), dtype=int)  # 3x3 random matrix
print(a)
print()
print("a.shape = ", a.shape)
print("type(a) = ", type(a))
print("a.dtype = ", a.dtype)
```

    [[0 0 0]
     [0 0 0]
     [0 0 0]]

    arr.shape =  (3, 3)
    type(arr) =  <class 'numpy.ndarray'>
    arr.dtype =  int64

```python
a = np.random.rand(2, 3, 3)  # 3x3 random matrix of floats in [0, 1)
print(a)
print()
print("a.shape = ", a.shape)
print("type(a) = ", type(a))
print("a.dtype = ", a.dtype)
```

    [[[0.03111165 0.97138943 0.19205285]
      [0.42724477 0.02948653 0.46096785]
      [0.00898452 0.99232939 0.7564268 ]]

     [[0.81428869 0.52431353 0.04443435]
      [0.53670176 0.86850834 0.31947024]
      [0.56926624 0.45408431 0.73173848]]]

    arr.shape =  (2, 3, 3)
    type(arr) =  <class 'numpy.ndarray'>
    arr.dtype =  float64

```python
def show_array_info(a):
    print(arr)
    print()
    print("arr.shape = ", arr.shape)
    print("type(arr) = ", type(arr))
    print("arr.dtype = ", arr.dtype)
```

```python
a = np.random.randint(1, 10, size=(5, 5))  # 3x3 random matrix of ints between 1 and 10
show_array_info(a)
```

    [[8 6 7]
     [4 5 5]
     [7 9 4]]

    arr.shape =  (3, 3)
    type(arr) =  <class 'numpy.ndarray'>
    arr.dtype =  int64

## Array indexing

```python
m = np.random.randint(1, 10, size=(5, 5))  # 3x3 random matrix of ints between 1 and 10
print(m)

m[0,0], m[1,0], m[3,4]  # [row,col] indexes
```

    [[4 9 7 2 9]
     [8 2 4 3 6]
     [8 3 8 6 7]
     [7 6 3 6 2]
     [6 4 2 7 7]]

    (4, 8, 2)

```python
m[4,4] = 100
m
```

    array([[  4,   4,   5,   7,   4],
           [  9,   7,   9,   6,   6],
           [  1,   6,   9,   4,   5],
           [  1,   1,   1,   6,   2],
           [  1,   9,   9,   7, 100]])

## Subarrays

```python
m[0,:]  # 1st row
```

    array([5, 5, 1, 4, 6])

```python
m[:,3]  # 4th col
```

    array([4, 2, 9, 1, 4])

```python
m[1:3,3]  # 2nd-3rd elements in 4th col
```

    array([2, 9])

## Reductions

```python
m = np.random.randint(1, 10, size=(5, 5))  # 3x3 random matrix of ints between 1 and 10
print(m)
m.min(), m.max(), m.mean(), m.var(), m.std()  # min, max, mean, variance, standard deviation
```

    [[9 8 2 3 8]
     [8 4 7 4 7]
     [3 7 2 3 8]
     [3 4 5 5 4]
     [1 7 6 8 4]]

    (1, 9, 5.2, 5.28, 2.2978250586152114)

## Partial reductions

```python
m.max(axis=0), m.max(axis=1)  # max across rows (axis=0) and cols (axis=1), respectively
```

    (array([9, 8, 7, 8, 8]), array([9, 8, 8, 5, 8]))

## Array multiplication

```python
A = np.random.randint(1, 10, size=(2, 2))  # 2x2 random matrix of ints between 1 and 10
b = np.array([2, 3])  # length 2 row vector
c = np.reshape(b, (2, 1))  # b as a col vector instead of row vector
print(A)
print()
print(b)
print()
print(c)
```

    [[9 7]
     [7 8]]

    [2 3]

    [[2]
     [3]]

### Element-wise multiplication

```python
print(A)
print()
print(A * A)
```

    [[9 7]
     [7 8]]

    [[81 49]
     [49 64]]

```python
print(b)
print()
print(b / 2)
```

    [2 3]

    [1.  1.5]

### Broadcasting

```python
print(A)
print()
print(b)
print()

print(A * b)  # element-wise multiplication of b with each row of A
```

    [[6 5]
     [6 7]]

    [2 3]

    [[12 15]
     [12 21]]

```python
print(b)
print()
print(c)
```

    [2 3]

    [[2]
     [3]]

```python
print(A)
print()
print(c)
print()

print(A * c)  # element-wise multiplication of c with each col of A
```

    [[5 2]
     [8 8]]

    [[2]
     [3]]

    [[10  4]
     [24 24]]

## Matrix multiplication

\begin{equation}\label{eq:matrixeqn}
     \begin{pmatrix}
       m_{00} & m_{01} & m_{02} \\
       m_{10} & m_{11} & m_{12} \\
       m_{20} & m_{21} & m_{22}
     \end{pmatrix}
     \cdot
     \begin{pmatrix}
       v_{0} \\
       v_{1} \\
       v_{2}
     \end{pmatrix}
     =
     \begin{pmatrix}
       m_{00} * v_{0} + m_{01} * v_{1} + m_{02} * v_{2} \\
       m_{10} * v_{0} + m_{11} * v_{1} + m_{12} * v_{2} \\
       m_{20} * v_{0} + m_{21} * v_{1} + m_{22} * v_{2}
     \end{pmatrix}
\end{equation}

(3 x 3) . (3 x 1) = (3 x 1)

```python
print(b)
print()
print(c)
print()
print(b.dot(c))
```

    [2 3]

    [[2]
     [3]]

    [13]

```python
c.dot(b)
```

```python
print(A)
print()
print(b)
print()
print(A.dot(b))
```

    [[6 5]
     [6 7]]

    [2 3]

    [27 33]

```python
print(A)
print()
print(c)
print()
print(A.dot(c))
```

    [[6 5]
     [6 7]]

    [[2]
     [3]]

    [[27]
     [33]]

## Speed and timing
* Basically do everything in numpy that you possibly can because it's *much much* faster than native python code.

```python
import timeit
import time

start = timeit.default_timer()  # timestamp in sec
time.sleep(2)  # sleep 2 sec
stop = timeit.default_timer()  # timestamp in sec

stop - start  # elapsed time
```

    2.000289000999601

```python
a = np.linspace(1, 1000000, num=1000000, endpoint=True)  # 1-1000000
start = timeit.default_timer()
for i in range(len(a)):
    a[i] = a[i]**2
stop = timeit.default_timer()
stop - start
```

    0.418272782999793

```python
a = np.linspace(1, 1000000, num=1000000, endpoint=True)  # 1-1000000
start = timeit.default_timer()
a = a**2
stop = timeit.default_timer()
stop - start
```

    0.002637679000144999

# Plot with matplotlib
Many more types of plots than those shown below can be made, such as histograms, contours, etc. Lot's of info on these is available online.

```python
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d  # for 3d plots
```

```python
x = np.random.rand(100)
y = np.random.rand(100)
z = np.random.rand(100)

fig = plt.figure()  # make a figure
plt.plot(x, y, 'o')  # plot x vs. y in current figure using circle markers
plt.plot(x, z, 'o')  # plot x vs. y in current figure using circle markers
plt.xlabel('x')
plt.ylabel('y or z')
plt.title('2d plot')
plt.legend(['x', 'y']);  # last semicolon suppresses title object output
```

```python
fig = plt.figure()  # make a figure
ax = plt.axes(projection='3d')  # set axes of current figure to 3d axes (this requires having imported mplot3d from mpl_toolkits)
ax.scatter(x, y, z)  # 3d scatter plot of x vs. y vs. z
ax.scatter(x, z, y)  # 3d scatter plot of x vs. z vs. y
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
plt.title('3d plot')
plt.legend(['x', 'y']);
```

## Subplots

```python
fig, ax = plt.subplots(nrows=2, ncols=3)

ax[0,0].scatter(x, y)
ax[0,0].set_xlabel('x')
ax[0,0].set_ylabel('y');

ax[1,1].scatter(x, z, marker='s', color='r')
ax[1,1].set_xlabel('x')
ax[1,1].set_ylabel('z')

ax[0,2].plot(range(len(y)), y, linestyle='-', color='c')
ax[0,2].set_ylabel('y')

fig.tight_layout();  # helps improve margins between plots
```

## Interactive plots
You may need to install the following for interactive plots in JupyterLab:

    > conda install -c conda-forge ipympl
    > conda install -c conda-forge widgetsnbextension
    > conda install nodejs
    > jupyter labextension install @jupyter-widgets/jupyterlab-manager
    > jupyter labextension install jupyter-matplotlib

`%matplotlib widget` will start using interactive plots

`%matplotlib inline` will go back to using non-interactive plots

```python
# interactive plot mode
%matplotlib widget

fig1 = plt.figure()  # make a figure
plt.plot(x, y, 'o')  # plot x vs. y in current figure using circle markers

fig2 = plt.figure()  # make a figure
ax = plt.axes(projection='3d')  # set axes of current figure to 3d axes
ax.scatter(x, y, z);  # 3d scatter plot of x vs. y vs. z
```

    FigureCanvasNbAgg()

    FigureCanvasNbAgg()

```python
# back to non-interactive plot mode
%matplotlib inline
```

```python

```


# Linear roll motion analytical solution
As decribed by Journee, Peter Piel etc.

```python
%matplotlib inline
%load_ext autoreload
%autoreload 2
```

```python
#from jupyterthemes import jtplot
#jtplot.style(theme='chesterish', context='notebook', ticks=True, grid=False)
```

```python
import sympy as sp
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

from sympy.physics.mechanics import dynamicsymbols

from rolldecay.symbols import *
from rolldecay import equations
import rolldecayestimators.equations
from rolldecayestimators.substitute_dynamic_symbols import lambdify
from rolldecayestimators import symbols
from rolldecayestimators import direct_linear_estimator

from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from rolldecay import paper_writing
```

## General Linear ordinary differential equation:

```python
Math(vlatex(equations.diff_eq))
```

```python
equations.equation_D
```

```python
Math(vlatex(equations.analytical_solution_general))
```

```python
Math(vlatex(equations.analytical_solution))
```

```python
Math(vlatex(equations.analytical_phi1d))
```

```python
Math(vlatex(equations.analytical_phi2d))
```

```python
Math(vlatex(rolldecayestimators.equations.extinction_equation))
```

```python
Math(vlatex(rolldecayestimators.equations.xeta_equation))
```

```python
analytical_solution_lambda = lambdify(sp.solve(equations.analytical_solution,phi)[0])
analytical_solution_phi1d_lambda = lambdify(sp.solve(equations.analytical_phi1d,phi_dot)[0])
analytical_solution_phi2s_lambda = lambdify(sp.solve(equations.analytical_phi2d,phi_dot_dot)[0])
extintion_equation_lambda = lambdify(sp.solve(rolldecayestimators.equations.extinction_equation,symbols.phi_a)[0])
```

```python
time = np.linspace(0,100,1000)
df = pd.DataFrame(index=time)
df['t']=time
df['zeta'] = 0.1
df['omega0'] = 0.5
df['phi_0'] = np.deg2rad(10)
df['phi_01d'] = 0.0

result = df.copy()
result['phi'] = analytical_solution_lambda(**df,)
result['phi1d'] = analytical_solution_phi1d_lambda(**df,)
result['phi2d'] = analytical_solution_phi2s_lambda(**df,)
result['phi_a'] = extintion_equation_lambda(omega0=df['omega0'], t=df['t'], zeta=df['zeta'],
                                            phi_0=df['phi_0'])
```

```python
fig,ax = plt.subplots()
result['phi_deg']=np.rad2deg(result['phi'])
result['phi_a_deg']=np.rad2deg(result['phi_a'])

result.plot(y='phi_deg', label='motion', ax=ax);
result.plot(y='phi_a_deg', label='amplitude', ax=ax);

ax.grid(True)
ax.set_xlabel('Time [s]')
ax.set_ylabel('Ship roll anlge [deg]')
ax.set_title('Ship roll decay test');

```

```python
with plt.style.context('grayscale'):

    fig,ax = plt.subplots()
    result['phi_deg']=np.rad2deg(result['phi'])
    result['phi_a_deg']=np.rad2deg(result['phi_a'])

    result.plot(y='phi_deg', label='motion', ax=ax);
    result.plot(y='phi_a_deg', label='amplitude', ax=ax);

    ax.grid(True)
    ax.set_xlabel('Time [s]')
    ax.set_ylabel('Ship roll anlge [deg]')
    ax.set_title('Ship roll-decay test')
    fig.savefig(fname='../../../roll-decay-logo.png',dpi=100)
```

### Numerical solution

```python
s0 = df.iloc[0]
linear_estimator = direct_linear_estimator.DirectLinearEstimator.load(omega0=s0['omega0'],
                                       zeta=s0['zeta'])

time=np.array(df['t'])
result_sim = linear_estimator.simulate(t=time,phi0=s0['phi_0'],
                                       phi1d0=s0['phi_01d'],
                                       )
result_sim = result_sim.combine_first(df)
```

```python
fig,axes = plt.subplots(nrows = 3)
fig.set_size_inches(15,10)
keys = ['phi','phi1d','phi2d']
labels = ['$\Phi$ [rad]','$\dot{\Phi}$ [rad/s]','$\ddot{\Phi}$ $[rad/s^2]$']
for ax, key, label in zip(axes,keys, labels):
    result.plot(y=key, ax=ax, label='analytical')
    result_sim.plot(y=key, ax=ax, label='numerical',style='--')

    ax.set_ylabel(label)

ax.set_xlabel('Time [s]');
axes[0].set_title('Solutions to linear roll motion ($\omega_0$:%0.1f [rad/s], $\zeta$:%0.1f [-])' %
             (df.iloc[0]['omega0'], df.iloc[0]['zeta']));

paper_writing.save_fig(fig=fig, name='analytical')
```

```python
from ipywidgets import interactive
```

```python
time = np.linspace(0,100,300)
df = pd.DataFrame(index=time)
df['t']=time
```

```python
def f(zeta=0.1,omega0=0.5,phi_0=10):
    fig,ax = plt.subplots()
    ax.set_xlim(0,time[-1])
    ax.set_ylim(-10,10)
    fig.set_size_inches(15,5)

    df['zeta'] = zeta
    df['omega0'] = omega0
    df['phi_0'] = np.deg2rad(phi_0)
    df['phi_01d'] = 0

    result = df.copy()
    result['phi'] = analytical_solution_lambda(**df,)
    result['phi']=np.rad2deg(result['phi'])
    ax.set_ylabel('Roll angle $\Phi$ [deg]')
    ax.set_xlabel('Time [s]')
    result.plot(y='phi', ax=ax)

    plt.show()

interactive_plot = interactive(f, zeta=(0.05,0.30, 0.01), omega0=(0.1,1,0.05), phi_0=(1.0,10.0,0.5))
output = interactive_plot.children[-1]
output.layout.height = '350px'
interactive_plot
```

```python

```

```python

```

```python

```