# PCA and DMD for the complex Ginzburg-Landau equation

## Arnold D. Kim
*Department of Applied Mathematics, University of California, Merced*

```python
import  time
print( 'Last updated: %s' %time.strftime('%d/%m/%Y') )
```

    Last updated: 22/09/2019

```python
%matplotlib qt
```

```python
# defaults for the codes below

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
```

## Description

We seek to apply principal component analysis (PCA) and dynamic mode decomposition (DMD) to time series data corresponding to the numerical solution of the complex Ginzburg-Landau equation,
\begin{equation}
   u_{t} = q^{2} ( \mathrm{i} + c_{0} ) u_{xx} + \rho u + ( \mathrm{i} - \rho ) u |u|^{2}.
\end{equation}
This spatio-temporal system exhibits diffusion, dispersion, linear growth, and nonlinearity. Consequently, it has a rich set of complex dynamics.

We first compute the numerical solution to an initial-boundary value problem for the complex Ginzburg-Landau equation with periodic boundary conditions in a chaotic regime. We collect that data and use it as an example time series. We then apply PCA to that time series to reveal its low-dimensional structure. Then we apply DMD to also reveal its low-dimensional structure and to characterize its dynamics.

## Numerical solution of the complex Ginzburg-Landau equation

To compute the numerical solution of the complex Ginzburg-Landau equation, we write it as
\begin{equation}
   u_{t} = L_{1}[u] + L_{2}[u],
\end{equation}
with $L_{1}$ denoting the linear operator,
\begin{equation}
   L_{1}[u] = q^{2} ( \mathrm{i} + c_{0} ) u_{xx} + \rho u,
\end{equation}
and $L_{2}$ denoting the nonlinear operator
\begin{equation}
   L_{2}[u] = ( \mathrm{i} - \rho ) u | u |^{2}.
\end{equation}

We use Strang splitting to compute the numerical solution, which is given as follows.

1. Solve the linear equation, $u_{t} = q^{2} ( \mathrm{i} + c_{0} ) u_{xx} + \rho u$ over a half time step, $\Delta t/2$, using a Fourier spectral method using the present value as the initial data.

2. Solve the nonlinear equation $u_{t} = ( \mathrm{i} - \rho ) u |u|^{2}$ over a full time step, $\Delta t$, using a second-order Runge-Kutta method using the result from Step 1 as the initial data.

3. Solve the linear equation, $u_{t} = q^{2} ( \mathrm{i} + c_{0} ) u_{xx} + \rho u$ over a half time step, $\Delta t/2$, using a Fourier spectral method using the result from Step 2 as the initial data.

The code below implements this method and collects the resulting numerical solution as an $M \times N$ matrix where $M$ is the number of spatial grid points and $N$ is the number of time steps.

```python
# numerical solution of the complex Ginzburg-Landau equation

# set the parameter values

c0 = 0.25
ρ  = 0.25
q  = 0.95

# compute the spatial grid

M  = 1000
Δx = 2 * np.pi / M
x  = np.arange( M ) * Δx

# compute the spectral wavenumbers

xk = np.fft.fftfreq( M, 1/M )

# set the time step

Δt = 0.05

# compute the forward propagator for the linear operator at half a time step

L1 = np.exp( ( ρ - xk**2 * q**2 * ( 1j + c0 ) ) * Δt / 2 )

# set the number of time steps

N = 10000

# allocate memory for the solution data

U = np.full( ( M, N ), 'nan', dtype = 'complex' )

# set the initial condition

U[:,0] = 1 + 0.02 * np.cos( x )

# loop over remaining time steps

for it in range( 1, N ):

    # first Strang-splitting step (linear)

    V1 = np.fft.ifft( L1 * np.fft.fft( U[:,it-1] ) )

    # second Strang-splitting step (nonlinear)

    k1   = Δt * ( 1j - ρ ) * V1 * np.abs( V1 ) ** 2
    Vtmp = V1 + 0.5 * k1
    V2   = V1 + Δt * ( 1j - ρ ) * Vtmp * np.abs( Vtmp ) ** 2

    # third Strang-splitting step (linear)

    U[:,it] = np.fft.ifft( L1 * np.fft.fft( V2 ) )
```

```python
# plot the real part of the solution

fig = plt.figure()
ax  = fig.add_subplot(111, projection='3d')

# set the figure size

plt.rcParams['figure.figsize'] = [12,8]

# create a meshgrid for plotting

[ Tplot, Xplot ] = np.meshgrid( np.arange(N) * Δt, x )

# plot the solution as a surface

ax.plot_surface( Tplot, Xplot, np.real(U), rstride=1, cstride=10, cmap = 'viridis' )
ax.view_init( 40, 210 )

plt.show()
```

## Prepare the data for analysis

In the code that follows, we take a portion of the computed solution as the data to analyze using PCA and DMD.

```python
# prepare the data

index = 200
xdata = x
tdata = np.arange( N-index, N ) * Δt
Data  = U[:,N-index:N]

print(Data.shape)

np.save('cgle-1000x200.npy',Data)
```

    (1000, 200)

## Principal component analysis (PCA)

We now perform PCA on the data using the singular value decomposition.

```python
# compute the singular-value decomposition of the data

U_PCA, σ_PCA, VH_PCA = np.linalg.svd( Data )

# compute the "energy"

E = σ_PCA ** 2 / np.sum( σ_PCA ** 2 )

# plot the energy

plt.rcParams['figure.figsize'] = [12,6]

plt.subplot(1,2,1)
plt.plot( E, 'o' )
plt.ylabel( 'energy' )

plt.subplot(1,2,2)
plt.plot( np.cumsum( E ), 'o' )
plt.ylabel( 'cumulative sum of energy' )

plt.show()
```

```python
# set the dimension reduction

k = 2

σ_tilde = σ_PCA[0:k]

# form the PCA reconstruction of the data

Data_PCA = U_PCA[:,:k] @ np.diag( σ_tilde ) @ VH_PCA[:k,:]
```

## Dynamic mode decomposition (DMD)

We now perform DMD on the data using the singular value decomposition.

```python
# define the X1 and X2 matrices

X1 = Data[:,:index-1]
X2 = Data[:,1:index]

# compute the singular-value decomposition of the X1 matrix

U_DMD, σ_DMD, VH_DMD = np.linalg.svd( X1 )

# truncate the SVD consistent with what was used for PCA

U_k  = U_DMD[:,:k]
σ_k  = σ_DMD[:k]
V_k = VH_DMD[:k,:].conj().T

# compute the DMD system matrix

A_DMD = U_k.conj().T @ X2 @ V_k @ np.diag( 1 / σ_k )

# compute the spectrum of the DMD system matrix

λ, W = np.linalg.eig( A_DMD )

# compute the continuous time frequencies (add 0*1j to ensure it is treated as a complex number)

ω = np.log( λ + 1j * 0 ) / Δt

# compute the Φ matrix

Φ = X2 @ V_k @ np.diag( 1 / σ_k ) @ W

# compute the DMD mode amplitudes b

b, residuals, rank, s = np.linalg.lstsq( Φ, X1[:,0], rcond = None )

# form the DMD reconstruction of the data

Dr = np.full( ( k, index ), 'nan', dtype = 'complex' )

for i in range( index ):

    Dr[:,i] = b * np.exp( ω * ( tdata[i] - tdata[0] ) )

Data_DMD = Φ @ Dr
```

## Compare the reconstructions

Below, we plot the reconstructions of the time series.

```python
# plot the reconstructions of the spatio-temporal dynamics

fig = plt.figure()
ax1 = fig.add_subplot(1, 3, 1, projection='3d')
ax2 = fig.add_subplot(1, 3, 2, projection='3d')
ax3 = fig.add_subplot(1, 3, 3, projection='3d')

# set the figure size

plt.rcParams['figure.figsize'] = [20,8]

# create a meshgrid for plotting

[ Tdata, Xdata ] = np.meshgrid( tdata, xdata )

# plot the solution as a surface

ax1.plot_surface( Tdata, Xdata, np.real(Data), rstride=1, cstride=10, cmap = 'viridis' )
ax1.view_init( 40, 210 )
ax1.set_xlabel( r'$t$', fontsize = 14 )
ax1.set_ylabel( r'$x$', fontsize = 14 )
ax1.set_title( 'time series', fontsize = 14 )

ax2.plot_surface( Tdata, Xdata, np.real(Data_PCA), rstride=1, cstride=10, cmap = 'viridis' )
ax2.view_init( 40, 210 )
ax2.set_xlabel( r'$t$', fontsize = 14 )
ax2.set_ylabel( r'$x$', fontsize = 14 )
ax2.set_title( 'PCA reconstruction', fontsize = 14 )

ax3.plot_surface( Tdata, Xdata, np.real(Data_DMD), rstride=1, cstride=10, cmap = 'viridis' )
ax3.view_init( 40, 210 )
ax3.set_xlabel( r'$t$', fontsize = 14 )
ax3.set_ylabel( r'$x$', fontsize = 14 )
ax3.set_title( 'DMD reconstruction', fontsize = 14 )

plt.show()
```

```python
# plot the reconstruction of the time series at a fixed value of x

# set the value of x to be plotted

xndx = 0

# plot the time series and its PCA and DMD reconstructions

plt.rcParams['figure.figsize'] = [10,6]

plt.plot( tdata, np.real(Data[xndx,:]), tdata, np.real(Data_PCA[xndx,:]), '--', tdata, np.real(Data_DMD[xndx,:]), '-.' )
plt.xlabel( r'$t$', fontsize = 14 )
plt.ylabel( r'$u(x=0,t)$', fontsize = 14  )
plt.legend( ( 'time series', 'PCA', 'DMD' ), fontsize = 14  )
plt.show()
```

```python

```

PCA and DMD for the complex Ginzburg-Landau equation.ipynb


### OpenGL
- 默认使用右手坐标系
- 默认使用列主序矩阵
- 归一化的CVV (*Canonical View Volume*)的z范围是[-1, 1]
***

### DirectX
- 默认使用左手坐标系
- 默认使用行主序矩阵
- 归一化的CVV的z范围是[0, 1]
***

- (x, y, z, 0) 向量
- (x, y, z, 1) 点
***

***

### DirectX透视投影矩阵推导

  - $P(x, y, z) => P’(x’, y’, z’)$
  - $P(N\dfrac{x}{z}, N\dfrac{y}{z}, N)$, 投影后的$x’$和$y’$与$z’$不是线性关系, 与$\dfrac {1}{z'}$才是线性关系, 用$\dfrac{az+b}{z}$表示
  - 当:
\begin{equation}
\left\{
    \begin{array}{**lr**}
    \dfrac{az+b}{z}=0, z=N  &  \\
    \dfrac{az+b}{z}=1, z=F  &
    \end{array}
\right.
\end{equation}
  - 可得:
\begin{equation}
\left\{
    \begin{array}{**lr**}
    a=\dfrac{F}{F-N}   &  \\
    b=-\dfrac{NF}{F-N}  &
    \end{array}
\right.
\end{equation}
  - 反推投影矩阵, 可得:
$$(x, y, z, 1) *
\begin{bmatrix}N & 0 & 0 & 0\\0 & N & 0 & 0\\0 & 0 & a & 1\\0 & 0 & b & 0\end{bmatrix}=
(Nx, Ny, az+b, Z)
$$

  - 给定$x$属于$[a, b]$, 对应于$y$属于$[c, d]$, 使得$x$与$a$的距离比上$ab$等于$y$与$c$的距离比上$cd$, 即$\dfrac{x-a}{b-a} = \dfrac{y-c}{d-c}$, 此时, $[a, b]$上任一点都能在$[c, d]$中找到对应点, 同理, 将$(x’, y’)$映射到$[-1, 1]$ (线性插值), 可得
\begin{equation}
\left\{
    \begin{array}{**lr**}
    \dfrac{\dfrac{Nx}{z}-left}{right-left}=\dfrac{X_{cvv}-(-1)}{1-(-1)}   &  \\
    \dfrac{\dfrac{Ny}{z}-bottom}{top-bottom}=\dfrac{Y_{cvv}-(-1)}{1-(-1)}  &
    \end{array}
\right.
\end{equation}

  - 上式可写成
\begin{equation}
\left\{
    \begin{array}{**lr**}
    \dfrac{\dfrac{Nx}{z}}{right-left}-{\dfrac{left}{right-left}}=\dfrac{X_{cvv}}{2}+\dfrac{1}{2}   &  \\
    \dfrac{\dfrac{Ny}{z}}{top-bottom}-{\dfrac{bottom}{top-bottom}}=\dfrac{Y_{cvv}}{2}+\dfrac{1}{2}  &
    \end{array}
\right.
\end{equation}

  - 当投影平面在$x$方向上居中, 且在$y$方向上居中, 即
\begin{equation}
\left\{
    \begin{array}{**lr**}
    left=-right, \dfrac{left}{right-left}=\dfrac{1}{2}   &  \\
    bottom=-top, \dfrac{bottom}{top-bottom}=\dfrac{1}{2}  &
    \end{array}
\right.
\end{equation}

  - 则上式可写成
\begin{equation}
\left\{
    \begin{array}{**lr**}
    \dfrac{\dfrac{Nx}{z}}{right-left}=\dfrac{X_{cvv}}{2}   &  \\
    \dfrac{\dfrac{Ny}{z}}{top-bottom}=\dfrac{Y_{cvv}}{2}   &
    \end{array}
\right.
\end{equation}
\begin{equation}
\left\{
    \begin{array}{**lr**}
    X_{cvv}=\dfrac{2Nx}{(right-left)z}   &  \\
    Y_{cvv}=\dfrac{2Ny}{(top-bottom)z}   &
    \end{array}
\right.
\end{equation}

  - 反推投影矩阵, 可得
$$(x, y, z, 1) *
\begin{bmatrix}\dfrac{2N}{right-left} & 0 & 0 & 0\\0 & \dfrac{2N}{top-bottom} & 0 & 0\\0 & 0 & a & 1\\0 & 0 & b & 0\end{bmatrix}=
(\dfrac{2Nx}{right-left}, \dfrac{2Ny}{top-bottom}, az+b, Z)
$$

  - 也可根据FOV来求

  - 一般情况下的矩阵
$$
\begin{bmatrix}\dfrac{2N}{right-left} & 0 & \dfrac{right+left}{right-left} & 0\\0 & \dfrac{2N}{top-bottom} & \dfrac{top+bottom}{top-bottom} & 0\\0 & 0 & a & b\\0 & 0 & -1 & 0\end{bmatrix}
$$


# Curve Fitting

## Objective and Prerequisites

Try this Jupyter Notebook Modeling Example to learn how you can fit a function to a set of observations. We will formulate this regression problem as a linear programming problem using the Gurobi Python API and then solve it with the Gurobi Optimizer.

This model is example 11 from the fifth edition of Model Building in Mathematical Programming, by H. Paul Williams on pages 266 and 319-320.

This modeling example is at the beginner level, where we assume that you know Python and that you have some knowledge about building mathematical optimization models. The reader should also consult the
[documentation](https://www.gurobi.com/resources/?category-filter=documentation)
of the Gurobi Python API.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

**Gurobi License**

In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EDU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_CURVE_FITTING_COM_EVAL_GitHub&utm_term=Curve_Fitting&utm_content=C_JPM) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EDU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_CURVE_FITTING_ACADEMIC_EVAL_GitHub&utm_term=Curve_Fitting&utm_content=C_JPM) as an *academic user*.

## Model Formulation

### Sets and Indices

$i \in \text{Observations}=\{1, .. ,n\}$.

### Parameters

$x_{i} \in \mathbb{R}$: Independent variable value at observation $i$.

$y_{i} \in \mathbb{R}$: Dependent variable value at observation $i$.

### Decision Variables

$a \in \mathbb{R}$: Value of the constant term in the function that explains the values of $y$ in terms of the values of $x$.

$b \in \mathbb{R}$: Coefficient of the linear term in the function that explains the values of $y$ in terms of the values of $x$.

$u_{i} \in \mathbb{R}^+$: Positive deviation of the proposed function of x with respect to the value of y at observation $i$.

$v_{i} \in \mathbb{R}^+$: Negative deviation of the proposed function of x with respect to the value of y at observation $i$.

$z$: Value of the maximum deviation.

We model the problem for the first goal:

* Fit a line $y=a+bx$ to the given data set in order to minimize the sum of absolute deviations of each observed value of $y$ from the value predicted by the linear relationship.

### Constraints Problem 1

**Deviation**: Each pair of corresponding data values $(x_{i},y_{i})$ gives rise to the following constraint.

\begin{equation}
bx_{i} + a + u_{i} - v_{i} = y_{i}  \quad \forall i \in \text{Observations}
\end{equation}

Where $x_{i}$  and $y_{i}$  are  the given values in the set of observations, $b$, $a$, $u_{i}$ and $v_{i}$ are variables.
The positive deviation $u_{i}$ and the negative deviation $v_{i}$ give the amounts by which the values of $y_{i}$ proposed by the linear expression differ from the observed values.

### Objective Function Problem 1

**Total deviation**: The objective is to minimize the total positive and negative deviations.

\begin{equation}
\text{Minimize} \quad \sum_{i \in \text{Observations}} (u_{i} + v_{i})
\end{equation}

We now provide a model formulation for the second goal:

* Fit a line $y=a+bx$ to the given data set in order to  minimize the maximum deviation of all the observed values of $y$ from the value predicted by the linear relationship.

For this new formulation, in addition to the "Deviation constraints", we need to include the following constraints.

### Constraints Problem 2

**Maximum deviation**: The following constraints ensure that the decision variable $z$ takes the value of the maximum deviation.

\begin{equation}
z \geq u_{i}  \quad \forall i \in \text{Observations}
\end{equation}

\begin{equation}
z \geq v_{i}  \quad \forall i \in \text{Observations}
\end{equation}

### Objective Function Problem 2

**Minimum/Maximum deviation**: The objective is to minimize the maximum deviation.

\begin{equation}
\text{Minimize} \quad z
\end{equation}

## Python Implementation

We import the Gurobi Python Module.

```python
import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.1.0
```

## Input data

We define the corresponding values for $x$ and $y$ in the set of observations.

```python
# Sample data: values of independent variable x and dependent variable y

observations, x, y = gp.multidict({
    ('1'): [0,1],
    ('2'): [0.5,0.9],
    ('3'): [1,0.7],
    ('4'): [1.5,1.5],
    ('5'): [1.9,2],
    ('6'): [2.5,2.4],
    ('7'): [3,3.2],
    ('8'): [3.5,2],
    ('9'): [4,2.7],
    ('10'): [4.5,3.5],
    ('11'): [5,1],
    ('12'): [5.5,4],
    ('13'): [6,3.6],
    ('14'): [6.6,2.7],
    ('15'): [7,5.7],
    ('16'): [7.6,4.6],
    ('17'): [8.5,6],
    ('18'): [9,6.8],
    ('19'): [10,7.3]
})
```

## Model Deployment

We create a model and the variables. The variables of the model are the constant term and coefficient of the linear term of the function f(x), the positive and negative deviations, and the maximum deviation.

```python
model = gp.Model('CurveFitting')

# Constant term of the function f(x). This is a free continuous variable that can take positive and negative values.
a = model.addVar(lb=-GRB.INFINITY, ub=GRB.INFINITY, vtype=GRB.CONTINUOUS, name="a")

# Coefficient of the linear term of the function f(x). This is a free continuous variable that can take positive
# and negative values.
b = model.addVar(lb=-GRB.INFINITY, ub=GRB.INFINITY, vtype=GRB.CONTINUOUS, name="b")

# Non-negative continuous variables that capture the positive deviations
u = model.addVars(observations, vtype=GRB.CONTINUOUS, name="u")

# Non-negative continuous variables that capture the negative deviations
v = model.addVars(observations, vtype=GRB.CONTINUOUS, name="v")

# Non-negative continuous variables that capture the value of the maximum deviation
z = model.addVar(vtype=GRB.CONTINUOUS, name="z")
```

    Using license file c:\gurobi\gurobi.lic

Each pair of corresponding data values $x_{i}$ and $y_{i}$ gives rise to a constraint.

```python
# Deviation constraints

deviations = model.addConstrs( (b*x[i] + a + u[i] - v[i] == y[i] for i in observations), name='deviations')
```

The objective function of problem 1 is to minimize the total positive and negative deviations.

```python
# Objective function of problem 1

model.setObjective(u.sum('*') + v.sum('*'))
```

```python
# Verify model formulation

model.write('CurveFitting.lp')

# Run optimization engine

model.optimize()
```

    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 19 rows, 41 columns and 75 nonzeros
    Model fingerprint: 0x0bec2f7b
    Coefficient statistics:
      Matrix range     [5e-01, 1e+01]
      Objective range  [1e+00, 1e+00]
      Bounds range     [0e+00, 0e+00]
      RHS range        [7e-01, 7e+00]
    Presolve removed 0 rows and 1 columns
    Presolve time: 0.01s
    Presolved: 19 rows, 40 columns, 75 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0      handle free variables                          0s
          20    1.1466250e+01   0.000000e+00   0.000000e+00      0s

    Solved in 20 iterations and 0.01 seconds
    Optimal objective  1.146625000e+01

```python
# Output report

print("\n\n_________________________________________________________________________________")
print(f"The best straight line that minimizes the absolute value of the deviations is:")
print("_________________________________________________________________________________")
print(f"y = {b.x:.4f}x + ({a.x:.4f})")
```

    _________________________________________________________________________________
    The best straight line that minimizes the absolute value of the deviations is:
    _________________________________________________________________________________
    y = 0.6375x + (0.5813)

For Problem 2, it is necessary to introduce another variable $z$ to capture the value of the maximum deviations

```python
# Maximum deviation constraints

maxPositive_deviation = model.addConstrs( (z >= u[i] for i in observations), name='maxPositive_deviation')

maxNegative_deviation = model.addConstrs( (z >= v[i] for i in observations), name='maxNegative_deviation')
```

The objective function for Problem 2 is to minimize the maximum deviation.

```python
# Objective function for Problem 2

model.setObjective(z)

# Run optimization engine

model.optimize()
```

    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 57 rows, 41 columns and 151 nonzeros
    Coefficient statistics:
      Matrix range     [5e-01, 1e+01]
      Objective range  [1e+00, 1e+00]
      Bounds range     [0e+00, 0e+00]
      RHS range        [7e-01, 7e+00]
    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    0.0000000e+00   1.146625e+01   0.000000e+00      0s
          11    1.7250000e+00   0.000000e+00   0.000000e+00      0s

    Solved in 11 iterations and 0.01 seconds
    Optimal objective  1.725000000e+00

```python
# Output report

print("\n\n_________________________________________________________________________________")
print(f"The best straight line that minimizes the maximum deviation is:")
print("_________________________________________________________________________________")
print(f"y = {b.x:.4f}x + ({a.x:.4f})")
```

    _________________________________________________________________________________
    The best straight line that minimizes the maximum deviation is:
    _________________________________________________________________________________
    y = 0.6250x + (-0.4000)

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC

```python

```


# The minimum jerk hypothesis

Marcos Duarte

Hogan and Flash (1984, 1985), based on observations of voluntary movements in primates, suggested that movements are performed (organized) with the smoothest trajectory possible. In this organizing principle, the endpoint trajectory is such that the mean squared-jerk across time of this movement is minimum.

Jerk is the derivative of acceleration and the observation of the minimum-jerk trajectory is for the endpoint in the extracorporal coordinates (not for joint angles) and according to Flash and Hogan (1985), the minimum-jerk trajectory of a planar movement is such that minimizes the following objective function:

$$ C=\frac{1}{2} \int\limits_{t_{i}}^{t_{f}}\;\left[\left(\frac{d^{3}x}{dt^{3}}\right)^2+\left(\frac{d^{3}y}{dt^{3}}\right)^2\right]\:\mathrm{d}t $$

Hogan (1984) found that the solution for this objective function is a fifth-order polynomial trajectory (see Shadmehr and Wise (2004) for a simpler proof):

$$ \begin{array}{l l}
x(t) = a_0+a_1t+a_2t^2+a_3t^3+a_4t^4+a_5t^5 \\
y(t) = b_0+b_1t+b_2t^2+b_3t^3+b_4t^4+b_5t^5
\end{array} $$

With the following boundary conditions for $ x(t) $ and $ y(t) $: initial and final positions are $ (x_i,y_i) $ and $ (x_f,y_f) $ and initial and final velocities and accelerations are zero.

Let's employ [Sympy](http://sympy.org/en/index.html) to find the solution for the minimum jerk trajectory using symbolic algebra.

```python
# import the necessary libraries
from __future__ import division, print_function
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
#sns.set()
sns.set_context("notebook", font_scale=1.2, rc={"lines.linewidth": 2, "lines.markersize": 10})
from IPython.display import display, Math, Latex
from sympy import symbols, Matrix, latex, Eq, collect, solve, diff, simplify
from sympy.utilities.lambdify import lambdify
```

Using Sympy, the equation for minimum jerk trajectory for x is:

```python
# declare the symbolic variables
x, xi, xf, y, yi, yf, d, t = symbols('x, x_i, x_f, y, y_i, y_f, d, t')
a0, a1, a2, a3, a4, a5 = symbols('a_0:6')
x = a0 + a1*t + a2*t**2 + a3*t**3 + a4*t**4 + a5*t**5
display(Math(latex('x(t)=') + latex(x)))
```

$$x(t)=a_{0} + a_{1} t + a_{2} t^{2} + a_{3} t^{3} + a_{4} t^{4} + a_{5} t^{5}$$

Without loss of generality, consider $ t_i=0 $ and let's use $ d $ for movement duration ($ d=t_f $). The system of equations with the boundary conditions for $ x $ is:

```python
# define the system of equations
s = Matrix([Eq(x.subs(t,0)          , xi),
            Eq(diff(x,t,1).subs(t,0),  0),
            Eq(diff(x,t,2).subs(t,0),  0),
            Eq(x.subs(t,d)          , xf),
            Eq(diff(x,t,1).subs(t,d),  0),
            Eq(diff(x,t,2).subs(t,d),  0)])
display(Math(latex(s, mat_str='matrix', mat_delim='[')))
```

$$\left[\begin{matrix}a_{0} = x_{i}\\a_{1} = 0\\2 a_{2} = 0\\a_{0} + a_{1} d + a_{2} d^{2} + a_{3} d^{3} + a_{4} d^{4} + a_{5} d^{5} = x_{f}\\a_{1} + 2 a_{2} d + 3 a_{3} d^{2} + 4 a_{4} d^{3} + 5 a_{5} d^{4} = 0\\2 a_{2} + 6 a_{3} d + 12 a_{4} d^{2} + 20 a_{5} d^{3} = 0\end{matrix}\right]$$

Which gives the following solution:

```python
# algebraically solve the system of equations
sol = solve(s, [a0, a1, a2, a3, a4, a5])
display(Math(latex(sol)))
```

$$\left \{ a_{0} : x_{i}, \quad a_{1} : 0, \quad a_{2} : 0, \quad a_{3} : \frac{1}{d^{3}} \left(10 x_{f} - 10 x_{i}\right), \quad a_{4} : \frac{1}{d^{4}} \left(- 15 x_{f} + 15 x_{i}\right), \quad a_{5} : \frac{1}{d^{5}} \left(6 x_{f} - 6 x_{i}\right)\right \}$$

Substituting this solution in the fifth order polynomial trajectory equation, we have the actual displacement trajectories:

```python
# substitute the equation parameters by the solution
x2 = x.subs(sol)
x2 = collect(simplify(x2, ratio=1), xf-xi)
display(Math(latex('x(t)=') + latex(x2)))
y2 = x2.subs([(xi, yi), (xf, yf)])
display(Math(latex('y(t)=') + latex(y2)))
```

$$x(t)=x_{i} + \left(x_{f} - x_{i}\right) \left(\frac{10 t^{3}}{d^{3}} - \frac{15 t^{4}}{d^{4}} + \frac{6 t^{5}}{d^{5}}\right)$$

$$y(t)=y_{i} + \left(y_{f} - y_{i}\right) \left(\frac{10 t^{3}}{d^{3}} - \frac{15 t^{4}}{d^{4}} + \frac{6 t^{5}}{d^{5}}\right)$$

And for the velocity, acceleration, and jerk trajectories in x:

```python
# symbolic differentiation
vx = x2.diff(t, 1)
display(Math(latex('v_x(t)=') + latex(vx)))
ax = x2.diff(t, 2)
display(Math(latex('a_x(t)=') + latex(ax)))
jx = x2.diff(t, 3)
display(Math(latex('j_x(t)=') + latex(jx)))
```

$$v_x(t)=\left(x_{f} - x_{i}\right) \left(\frac{30 t^{2}}{d^{3}} - \frac{60 t^{3}}{d^{4}} + \frac{30 t^{4}}{d^{5}}\right)$$

$$a_x(t)=\frac{60 t}{d^{3}} \left(x_{f} - x_{i}\right) \left(1 - \frac{3 t}{d} + \frac{2 t^{2}}{d^{2}}\right)$$

$$j_x(t)=\frac{60}{d^{3}} \left(x_{f} - x_{i}\right) \left(1 - \frac{6 t}{d} + \frac{6 t^{2}}{d^{2}}\right)$$

Let's plot the minimum jerk trajectory for x and its velocity, acceleration, and jerk considering $x_i=0,x_f=1,d=1$:

```python
# substitute by the numerical values
x2  = x2.subs([(xi, 0), (xf, 1), (d, 1)])
#create functions for calculation of numerical values
xfu = lambdify(t, diff(x2, t, 0), 'numpy')
vfu = lambdify(t, diff(x2, t, 1), 'numpy')
afu = lambdify(t, diff(x2, t, 2), 'numpy')
jfu = lambdify(t, diff(x2, t, 3), 'numpy')
#plots using matplotlib
ts = np.arange(0, 1.01, .01)
font = {'weight':'normal', 'size':12}
plt.matplotlib.rc('font', **font)
fig, axs = plt.subplots(1, 4, figsize=(10, 4), sharex=True, squeeze=True)
axs[0].plot(ts, xfu(ts), linewidth=3)
axs[0].set_title('Displacement [$\mathrm{m}$]')
axs[1].plot(ts, vfu(ts), linewidth=3)
axs[1].set_title('Velocity [$\mathrm{m/s}$]')
axs[2].plot(ts, afu(ts), linewidth=3)
axs[2].set_title('Acceleration [$\mathrm{m/s^2}$]')
axs[3].plot(ts, jfu(ts), linewidth=3)
axs[3].set_title('Jerk [$\mathrm{m/s^3}$]')

for axi in axs:
    axi.set_xlabel('Time [s]', fontsize=14)
    axi.grid(True)

fig.suptitle('Minimum jerk trajectory kinematics', fontsize=20, y=1.03)
fig.tight_layout()
plt.show()
```

At the minimum jerk trajectory, initial and final values of both velocity and acceleration are zero, but not for the jerk.

Read more about the minimum jerk trajectory hypothesis in the [Shadmehr and Wise's book companion site](http://www.shadmehrlab.org/book/minimum_jerk/minimumjerk.htm) and in [Paul Gribble's website](http://www.gribblelab.org/compneuro/4_Computational_Motor_Control_Kinematics.html#sec-5-1).

## Problems

1. Calculate and plot the position, velocity, acceleration, and jerk trajectories for different movement speeds (for example, consider always a displacement of 1 m and movement durations of 0.5, 1, and 2 s).
2. For the data in the previous item, calculate the ratio peak speed to average speed. Shadmehr and  Wise (2004) argue that psychophysical experiments show that reaching movements with the hand have this ratio equals to 1.75. Compare with the calculted values.

## References

- Flash T, Hogan N (1985) [The coordination of arm movements: an experimentally confirmed mathematical model](http://www.jneurosci.org/cgi/reprint/5/7/1688.pdf). Journal of Neuroscience, 5, 1688-1703.
- Hogan N (1984) [An organizing principle for a class of voluntary movements](http://www.jneurosci.org/content/4/11/2745.full.pdf). Journal of Neuroscience, 4, 2745-2754.
- Shadmehr R, Wise S (2004) [The Computational Neurobiology of Reaching and Pointing: A Foundation for Motor Learning](http://www.shadmehrlab.org/book/). A Bradford Book. [Companion site](http://www.shadmehrlab.org/book/).
- Zatsiorsky VM (1998) [Kinematics of Human Motion](http://books.google.com.br/books/about/Kinematics_of_Human_Motion.html?id=Pql_xXdbrMcC&redir_esc=y). Champaign, Human Kinetics.


```
# default_exp
```

```
#hide
#all_slow
```

# 2-D prior toy example

> Demonstration of prior error bounds for a 2-D toy example

In this notebook we will demonstrate the error bounds for the statFEM prior for the toy example introduced in twoDim. We first import some required packages.

```
from dolfin import *
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (10,6)
# import required functions from twoDim
from statFEM.twoDim import mean_assembler, kernMat, cov_assembler
from scipy.linalg import sqrtm
import sympy; sympy.init_printing()
from tqdm.notebook import tqdm
# code for displaying matrices nicely
def display_matrix(m):
    display(sympy.Matrix(m))
```

Since we do not have ready access to a solution operator for this problem we will utilise a different approach to demonstrate the appropriate rate for the error bounds. This approach involves comparing the statFEM distributions on finer and finer meshes as follows. We first assume we have the following error bound:

$$W(\nu_{\star},\nu_{h})\leq\mathcal{O}(h^{p}) \text{ as } h\rightarrow 0$$

From this we have:

$$W(\nu_{\star},\nu_{h})=Ch^{p}+\mathcal{O}(h^{p+1})$$

The triangle inequality then yields:

$$W(\nu_{h},\nu_{h/2})\leq W(\nu_{h},\nu_{\star}) + W(\nu_{\star},\nu_{h/2})$$

And so with the above we obtain:

$$W(\nu_{h},\nu_{h/2})=Ch^{p}+C(h/2)^{p}+\mathcal{O}(h^{p+1})$$

Similarly we have:

$$W(\nu_{h/2},\nu_{h/4})=C(h/2)^{p}+C(h/4)^{p}+\mathcal{O}(h^{p+1})$$

Dividing the two above equations yields:

$$\frac{W(\nu_{h},\nu_{h/2})}{W(\nu_{h/2},\nu_{h/4})}=\frac{Ch^{p}+C(h/2)^{p}+\mathcal{O}(h^{p+1})}{C(h/2)^{p}+C(h/4)^{p}+\mathcal{O}(h^{p+1})}=\frac{1-2^{-p}+\mathcal{O}(h)}{2^{-p}-2^{-2p}+\mathcal{O}(h)}=2^{p}+\mathcal{O}(h)$$

From which it follows:

$$\operatorname{LR}(h):=\log_{2}\frac{W(\nu_{h},\nu_{h/2})}{W(\nu_{h/2},\nu_{h/4})}=p+\mathcal{O}(h)$$

Note that this ratio above does not rely on having access to the true prior $\nu_{\star}$.

We will thus compare the errors on a succesively refined mesh and take the base-2 log of the above ratio. We hope to see these logarithms converge to $p=2$.

We now set up the mean and kernel functions for the random forcing term $f$.

```
# set up mean and kernel functions
f_bar = Constant(1.0)

l_f = 0.4
σ_f = 0.1

def k_f(x):
    return (σ_f**2)*np.exp(-(x**2)/(2*(l_f**2)))
```

We now set up a function to get the statFEM prior for a FE mesh size $h$, using functions from twoDim.

```
def fem_prior(h,f_bar,k_f,grid):
    J = int(np.round(1/h))
    μ = mean_assembler(h,f_bar)
    Σ = cov_assembler(J,k_f,grid,False,True)
    return μ,Σ
```

We now set up a reference grid on which we will compare the covariance matrices.

```
N = 41
x_range = np.linspace(0,1,N)
grid = np.array([[x,y] for x in x_range for y in x_range])
```

We now set up a function to compare the covariance functions on the reference grid.

```
def compute_cov_diff(C1,C2,tol=1e-10):
    N = np.sqrt(C1.shape[0])
    C1_sqrt = np.real(sqrtm(C1))
    rel_error_1 = np.linalg.norm(C1_sqrt @ C1_sqrt - C1)/np.linalg.norm(C1)
    assert rel_error_1 < tol

    C12 = C1_sqrt @ C2 @ C1_sqrt
    C12_sqrt = np.real(sqrtm(C12))
    rel_error_12 = np.linalg.norm(C12_sqrt @ C12_sqrt - C12)/np.linalg.norm(C12)
    assert rel_error_12 < tol

    hSq = (1/(N-1))**2
    return hSq*(np.trace(C1) + np.trace(C2) - 2*np.trace(C12_sqrt))
```

We now set up a function to compute the Wasserstein distance between the statFEM priors.

```
def W(μ_1,μ_2,Σ_1,Σ_2):
    mean_error = errornorm(μ_1,μ_2,'L2')
    cov_error = compute_cov_diff(Σ_1,Σ_2)
    cov_error = np.sqrt(np.abs(cov_error))
    error = mean_error + cov_error
    return error
```

In the interests of memory efficiency we will now create a function which will compute the ratios of errors mentioned above by succesively refining the FE mesh.

```
def refine(h,n,f_bar,k_f,grid):
    # set up empty lists to hold h-values and errors (this being the ratios)
    h_range = []
    errors = []
    # get the statFEM prior for h and h/2
    μ_1, Σ_1 = fem_prior(h,f_bar,k_f,grid)
    μ_2, Σ_2 = fem_prior(h/2,f_bar,k_f,grid)
    # compute the distance between these and store in numerator variable
    numerator = W(μ_1,μ_2,Σ_1,Σ_2)
    # succesively refine the mesh by halving and do this n times
    for i in tqdm(range(n)):
        # store mean and cov for h/2 in storage for h
        μ_1, Σ_1 = μ_2, Σ_2
        # in storage for h/2 store mean and cov for h/4
        μ_2, Σ_2 = fem_prior(h/4,f_bar,k_f,grid)
        # compute the distance between the priors for h/2 and h/4
        # and store in denominator variable
        denominator = W(μ_1,μ_2,Σ_1,Σ_2)
        # compute the ratio and store in error
        error = numerator/denominator
        # append the current value of h and the ratio
        h_range.append(h)
        errors.append(error)
        # store denominator in numerator and halve h
        numerator = denominator
        h = h/2
    # return the list of h-values together with the ratios for these values
    return h_range,errors
```

> Note: The above function will compute the ratio $\frac{W(\nu_{h},\nu_{h/2})}{W(\nu_{h/2},\nu_{h/4})}$ for the values $h,h/2,\cdots,h/2^{n-1}$.

We now set up a list of starting $h$ values and number of refinements $n$ to get a decent number of ratios to approximate $p$.

```
my_list = [(0.25,4),(0.2,3),(0.175,3),(0.22,4),(0.176,3),(0.177,3),(0.178,3),(0.179,3),(0.18,3),(0.21,3),(0.215,3),(0.1,2),(0.3,4),(0.31,4),(0.315,4),(0.24,4),(0.245,4),(0.25,4),(0.14,2),(0.16,2),(0.15,2),(0.18,2)]
```

> Warning: As we are now in 2-D the matrices needed to compute the statFEM prior get very large in size as $h$ decreases. Due to memory constraints we must thus make sure $h$ does not get too small.

#hide

The following cells use `pickle` to load the results, avoiding having to wait for the results to compute.

```
#hide
# import pickle
```

#hide

If this next cell is run the dictionary `results` will be filled in from the file `2D_prior_results`. The code below can then be run to produce the final figure without having to wait for the results to be computed.

```
#hide
# with open('2D_prior_results','rb') as f:
#     results = pickle.load(f)
# h_range = results['h']
# errors = results['ratios']
```

We now compute the results:

#hide

This next cell shouldn't be run if you load the results.

```
%%time
h_range = []
errors = []
for h,n in tqdm(my_list,desc = 'Outer loop'):
    h_range_tmp, errors_tmp = refine(h,n,f_bar,k_f,grid)
    h_range.extend(h_range_tmp)
    errors.extend(errors_tmp)
```

    Outer loop:   0%|          | 0/22 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

    CPU times: user 1h 15min 42s, sys: 43min 30s, total: 1h 59min 12s
    Wall time: 31min 30s

#hide

Run the following cell to save the results.

```
#hide
# results = {'h':h_range,'ratios':errors}
# with open('2D_prior_results','wb') as f:
#     pickle.dump(results,f)
```

We now sort these results by $h$ and get rid of duplicates (from repeats in $h$-values).

```
h_range_array = np.array(h_range)
errors_array = np.array(errors)
```

```
argInd = np.argsort(h_range_array)
hs = h_range_array[argInd]
es = errors_array[argInd]
hs,hs_ind = np.unique(hs,return_index=True)
es = es[hs_ind]
```

We now plot the base-2 logarithm of the ratios against $h$ below:

```
#hide_input
plt.plot(hs,np.log2(es),c='r')
plt.xlabel('$h$')
plt.ylabel('$\operatorname{LR}(h)$')
plt.title('Base-2 log plot of the ratios against $h$')
plt.hlines(2.0,hs[0],hs[-1],colors='black',linestyles='--')
plt.xlim(hs[-1],hs[0])
plt.grid()
plt.savefig('2D_prior_results.png',dpi=300,bbox_inches='tight',facecolor='w')
plt.show()
```

We can see from the above plot that the logarithms seems to be approaching $p=2$ as $h$ gets small just as predicted. However, the results aren't that smooth and they haven't seemed to settle on $p=2$ yet. This can be due to memory constraints meaning we cannot use small enough $h$. We thus smooth the above results by taking a cumulative average of the ratios and then applying $\log_2$. We take the rolling average starting with large $h$. We choose a cut-off point of $h=0.15$, i.e. we discard any results for $h>0.15$.

The results are shown below:

```
#hide_input
# get the cumulative average of es:
i = -19
ES = es[:i]
HS = hs[:i]
pos = np.arange(0,40,step=5)
es_av = np.log2(np.cumsum(ES[::-1])/np.arange(1,len(ES)+1))
plt.plot(es_av,label='log_of_cum_av',c='r')
plt.xticks(pos,HS[::-1][pos])
plt.hlines(2.0,0,len(es_av),colors='black',linestyles='--')
plt.title('Base-2 logarithms of the cumulative averages of the ratios for $h\leq 0.15$')
plt.xlabel('min $h$')
plt.ylabel('smoothed $\operatorname{LR}(h)$')
plt.grid()
plt.savefig('2D_prior_results_smoothed.png',dpi=300,bbox_inches='tight',facecolor='w')
plt.show()
```

From the smoothed results above we can see more clearly that the ratios are converging to around $p=2$. In fact, discarding the values corresponding to $h>0.15$ seems to result in the the rolling average converging to a value slightly greater than $2$.


# Plotting vector fields

In this tutorial we are going to plot 2D and 3D vector fields with the `vector_plot` function. At this stage of development, only a few backends support this functionality.

```python
from spb.backends.plotly import PB
from spb.backends.bokeh import BB
from spb.backends.k3d import KB
from spb import *
from sympy import *
init_printing(use_latex=True)
```

The plotting interface is basically the same of any other plotting function. We need to specify:
* `vector, range1, range2` if we are plotting a single vector field.
* we can use tuples of the form `(vector1, range1, range2, label1), (vector2, range1, range2, label2), ...` to plot multiple vector fields simultaneously.

```python
help(vector_plot)
```

## Plotting vectors created with sympy.vector module

Let's create simple vector:

```python
from sympy.vector import CoordSys3D
N = CoordSys3D("N")
i, j, k = N.base_vectors()
x, y, z = N.base_scalars()
v1 = -sin(y) * i + cos(x) * j
v1
```

```python
vector_plot(v1, (x, -5, 5), (y, -5, 5), backend=PB, xlabel="x", ylabel="y",
           quiver_kw=dict(scale=0.3))
```

Here, we used Plotly. A few things to note:
* we need to specify the x-y-labels.
* by default, the x and y axis will use an equal aspect ratio. We can disable it by setting the keyword argument `aspect=None`.
* by default, a contour plot of the magnitude of the vector field is shown (more on this later).
* solid color is used for the arrows (or quivers), whose lengths are proportional to the local magnitude value. Note that Plotly doesn't support gradient coloring for quivers.
* We use the `quiver_kw` dictionary to control the appearance of the quivers, where we write the keyword arguments targeting the specific backend's quiver function. In this case, the quiver function is [Plotly's `create_quiver`](https://plotly.com/python/quiver-plots/). Here, we used `scale=0.3` to set a decent size for the quivers.

Let's say we are not interested in showing the contour plot representing the magnitude. We can disable it by setting the keyword argument `scalar=None`:

```python
vector_plot(v1, (x, -5, 5), (y, -5, 5), backend=PB, xlabel="x", ylabel="y",
           quiver_kw=dict(scale=0.3), scalar=None)
```

Alternatively, we can set `scalar` to any scalar field, for example:

```python
x2, y2 = symbols("x, y")
vector_plot(v1, (x, -5, 5), (y, -5, 5), backend=PB, xlabel="x", ylabel="y",
           quiver_kw=dict(scale=0.3), scalar=x2*y2)
```

Instead of visualizing quivers, we can plot streamlines by setting `streamlines=True`:

```python
vector_plot(v1, (x, -5, 5), (y, -5, 5), backend=PB, xlabel="x", ylabel="y",
            streamlines=True, stream_kw=dict(density=2, arrow_scale=0.2))
```

A few things to note:
* computing and visualizing streamlines is usually computationally more expensive than plotting quivers, so the function may takes longer to produce the plot.
* We use the `stream_kw` dictionary to control the appearance of the streamlines, where we write the keyword arguments targeting the specific backend's quiver function. In this case, the quiver function is [Plotly's `create_streamline`](https://plotly.com/python/streamline-plots/). Here, we increased the density and set an appropriate arrow size.

## Quick way to plot vectors

In the previous section we used `sympy.vector` module to define vectors. However, if we are in a hurry we can avoid using that module, passing in to the function a list containing the components of the vector. For example:

```python
x, y = symbols("x, y")
vector_plot([-sin(y), cos(x)], (x, -5, 5), (y, -5, 5), backend=BB, xlabel="x", ylabel="y",
           quiver_kw=dict(scale=0.5))
```

Here, we used Bokeh. A few things to note:
* by switching backend, the user experience will be overall quite similar. Unfortunately, it is hardly possible to have one-one-one correspondance between colors and color maps.
* Bokeh doesn't automatically support contour plots. If we zoom in, we will see that the scalar field is using square "pixels" to be rendered, leading to an unpleasant result. We can "fix" this problem by bumping up the number of discretization points for the contour plot by setting the keyword argument `nc=250` (or some other number).

Let's try to increase the number of discretization points for the contour plot and decrease the number of discretization points for the quivers:

```python
vector_plot([-sin(y), cos(x)], (x, -5, 5), (y, -5, 5), backend=BB, xlabel="x", ylabel="y",
           quiver_kw=dict(scale=0.5), nc=250, n=20)
```

Note that by increasing `nc`, the plot is slower to render. Having discovered that Bokeh doesn't handle that well a contour plot, let's disable the scalar field:

```python
vector_plot([-sin(y), cos(x)], (x, -5, 5), (y, -5, 5), backend=BB, xlabel="x", ylabel="y",
           quiver_kw=dict(scale=0.5), scalar=None)
```

By default, a color map will be applied to the quivers based on the local magnitude value. We can further customize the color of the quivers by using the `quiver_kw`:

```python
vector_plot([-sin(y), cos(x)], (x, -5, 5), (y, -5, 5), backend=BB, xlabel="x", ylabel="y",
           quiver_kw=dict(scale=0.5, line_color="red", line_width=2), scalar=None)
```

Finally, Bokeh also "supports" streamlines:

```python
vector_plot([-sin(y), cos(x)], (x, -5, 5), (y, -5, 5), backend=BB, xlabel="x", ylabel="y",
           streamlines=True)
```

## 3D Vector Fields

As always, Bokeh doesn't support 3D plots, so we are left with Plotly and K3D. The principle of operation is the same as 2D vector fields.

```python
x, y, z = symbols("x:z")
vector_plot(Matrix([z, y, x]), (x, -5, 5), (y, -5, 5), (z, -5, 5), n=7,
           quiver_kw=dict(sizeref=10), backend=PB, xlabel="x", ylabel="y", zlabel="z")
```

A few things to note:
* we used a matrix, `Matrix([z, y, x])`, to represent a vector. When dealing with 3D vectors, some components may be numbers: in that case the internal algorithm might get confused, thinking of the vector as a range. In order to avoid this ambiguity, we wrap the 3D vector into a matrix of three elements and away we go.
* plotting 3D vector fields is computationally more expensive, hence we reduced the number of discretization points to `n=7` in each direction.
* 3D quivers are colored by the local value of the magnitude of the vector field.
* With the usual `quiver_kw` dictionary, we can provide backend-specific keyword arguments to control the appearance of the quivers. Here, we choose an appropriate size. Refer to [Plotly's Cone function](https://plotly.com/python/cone-plot/) for more information.

It is usually difficult to understand a 3D vector field by using quivers. Therefore, we might get a better idea by using streamlines:

```python
vector_plot(Matrix([z, y, x]), (x, -5, 5), (y, -5, 5), (z, -5, 5), n=10,
           streamlines=True, backend=PB, xlabel="x", ylabel="y", zlabel="z")
```

In order to generate streamlines, the internal algorithm automatically computed optimal starting points (seeds) at the boundaries of the domain, where the vectors are pointing inward toward the domain. This tends to produce regularly spaced streamlines. We will see later how to change the seeds.

Usually, the tricky part is chosing the size of the streamlines. This is an iterative process. Note that the streamlines are coloured according to the local magnitude value.

Keep in mind that Plotly uses a different technology to compute streamlines than Matplotlib and K3D-Jupyter. Therefore, it may become slower and slower as we increase the number of discretization points.

Now, let's change a little bit the vector for illustrative purposes:

```python
p1 = vector_plot(Matrix([y, z, x]), (x, -5, 5), (y, -5, 5), (z, -5, 5), n=5,
           backend=PB, xlabel="x", ylabel="y", zlabel="z", show=False,
                quiver_kw=dict(sizeref=10))

p2 = vector_plot(Matrix([y, z, x]), (x, -5, 5), (y, -5, 5), (z, -5, 5), n=10,
           streamlines=True, backend=PB, xlabel="x", ylabel="y", zlabel="z", show=False)
p1.extend(p2)
p1.show()
```

A few things to note here:
* We created two separates plots of the same vector field and later merged the second (the streamlines) into the first (the quivers).
* At this moment of development, if `stream_kw` was provided in the second plot, it would have been lost during the merging operation. Hopefully, this _bug_ will be fixed in the future.

Now, just for fun, let’s visualize the original vector field with K3D:

```python
x, y, z = symbols("x:z")
vector_plot(Matrix([z, y, x]), (x, -5, 5), (y, -5, 5), (z, -5, 5),
    n=10, quiver_kw=dict(scale=0.2), backend=KB,
    xlabel="x", ylabel="y", zlabel="z")
```

Note that we used different keyword argument to customize the size of the quivers.

Let's now try to plot streamlines with K3DBackend. We can set the keyword argument `starts` in the `stream_kw` dictionary to one of the following values:
* `starts=None` (or do not provide it at all): this is the default value, with which the algorithm will automatically chose the seeds points on the surfaces of the discretized volume based on the direction of the vectors.
* `starts={"x": x_list, "y": y_list, "z": z_list}`: a dictionary containing lists of coordinates of the seeds is passed in.
* `starts=True`: the algorithm will randomly chose the seeds points inside the discretized volume. In this case we can also specify the number of points to be generated by setting `npoints`: usually, the number of computed streamlines will be much lower than `npoints`.

Let's use the [Arnold–Beltrami–Childress flow](https://en.wikipedia.org/wiki/Arnold%E2%80%93Beltrami%E2%80%93Childress_flow) (velocity field) to explore the different seeds options.

First, the default one:

```python
vector_plot(
    Matrix([
        sin(z) + sqrt(S(1) / 3) * cos(y),
        sqrt(S(2) / 3) * sin(x) + cos(z),
        sqrt(S(1) / 3) * sin(y) + sqrt(S(2) / 3) * cos(y)
    ]), (x, -5, 5), (y, -5, 5), (z, -5, 5),
    backend=KB, streamlines=True, n=15,
    xlabel="x", ylabel="y", zlabel="z"
)
```

Now, let's try a random seeds:

```python
vector_plot(
    Matrix([
        sin(z) + sqrt(S(1) / 3) * cos(y),
        sqrt(S(2) / 3) * sin(x) + cos(z),
        sqrt(S(1) / 3) * sin(y) + sqrt(S(2) / 3) * cos(y)
    ]), (x, -5, 5), (y, -5, 5), (z, -5, 5),
    backend=KB, streamlines=True, n=15,
    stream_kw={
        "starts": True,
        "npoints": 500
    },
    xlabel="x", ylabel="y", zlabel="z"
)
```

Note how the streamlines are randomly distributed in the volume. The user might want to increase `n`, the number of discretization points, in order to get smoother streamlines.

Finally, let's generate the streamlines from a plane parallel to the xy plane:

```python
ranges = [(x, -5, 5), (y, -5, 5), (z, -5, 5)]
n = 10
xx, yy, zz = get_plot_data(Plane((0, 0, 0), (0, 0, 1)), *ranges, n1=n, n2=n)
vector_plot(
    Matrix([
        sin(z) + sqrt(S(1) / 3) * cos(y),
        sqrt(S(2) / 3) * sin(x) + cos(z),
        sqrt(S(1) / 3) * sin(y) + sqrt(S(2) / 3) * cos(y)
    ]), *ranges,
    backend=KB, streamlines=True, n=15,
    stream_kw={
        "starts": {
            "x": xx,
            "y": yy,
            "z": zz,
        }
    },
    xlabel="x", ylabel="y", zlabel="z"
)
```

Note how the streamlines are packed around the provided plane.

## Interactive-Parametric Vector Plots

We can also use `iplot` to play with parametric vector fields, all we have to remember is to set `is_vector = True`:

```python
from spb.interactive import iplot
a, b, x, y, z = symbols("a, b, x:z")

iplot(
    ([-a * sin(y), b * cos(x)], (x, -5, 5), (y, -3, 3)),
    params = {
        a: (1, 0, 2),
        b: (1, 0, 2),
    },
    xlabel = "x",
    ylabel = "y",
    backend = PB,
    n = 10,
    quiver_kw = dict(
        scale = 0.4
    ),
    is_vector = True
)
```

In contrast to `vector_plot`, the `iplot` function:
* We need to specify the number of discretization point, `n=10`. Alternatively, we can set `n1, n2, n3` to specify the number of discretization points in the three directions. **Remembert to set `n` to a sufficiently low number**. Since `n` will be used on every direction, the internal algorithm will create 4 `n x n` matrices for 2D vector fields, and 6 `n x n x n` matrices for 3D vector fields, hence a lot more memory will be used as we increase `n`!!!
* A few other keyword arguments have been set to customize the appearance.

Let's try plotting streamlines with `BokehBackend`. Remember: streamlines are always more computationally expensive to compute, so expect a delay of a few seconds from when you interact with the slider to the moment you will see the updated plot:

```python
iplot(
    ([-a * sin(y), b * cos(x)], (x, -5, 5), (y, -3, 3)),
    params = {
        a: (1, 0, 2),
        b: (1, 0, 2),
    },
    xlabel = "x",
    ylabel = "y",
    backend = BB,
    n = 20,
    streamlines = True,
    stream_kw = dict(
        line_color = "red"
    )
)
```

Let's now try to plot 3D vector fields. We are going to use Plotly and K3D:

```python
iplot(
    ([a * z, b * y, x], (x, -5, 5), (y, -3, 3), (z, -4, 4)),
    params = {
        a: (1, 0, 2),
        b: (1, 0, 2),
    },
    xlabel = "x",
    ylabel = "y",
    zlabel = "z",
    backend = PB,
    n = 8,
    quiver_kw = dict(
        sizeref = 4
    )
)
```

```python
iplot(
    ([a * z, b * y, x], (x, -5, 5), (y, -3, 3), (z, -4, 4)),
    params = {
        a: (1, 0, 2),
        b: (1, 0, 2),
    },
    xlabel = "x",
    ylabel = "y",
    zlabel = "z",
    backend = KB,
    n = 8,
    quiver_kw = dict(
        scale = 0.25
    )
)
```

At the time of writing this tutorial, Plotly and K3D do not support `iplot` for streamlines:

```python
iplot(
    ([a * z, b * y, x], (x, -5, 5), (y, -3, 3), (z, -4, 4)),
    params = {
        a: (1, 0, 2),
        b: (1, 0, 2),
    },
    xlabel = "x",
    ylabel = "y",
    zlabel = "z",
    backend = KB,
    n = 8,
    streamlines = True
)
```

```python

```


disclaimer: To ensure that the notebook can be run from (more or less) any point, I try to load the relevant functions or modules whenever I use them in a cell. This is generally not good practice as it adds unneccesary overhead

# 0.  Image representation as numerical arrays
### We start by importing numpy and creating and printing a simple 9x9 checkerboard array

```python
%matplotlib inline
%load_ext autoreload
%autoreload
import numpy as np

#make a 9x9 checkerboard
checkBoard = #

print(checkBoard)
```

### Then, we import [pyplot](https://matplotlib.org/api/pyplot_api.html) and [image](https://matplotlib.org/api/image_api.html) modules from the ploting library [matplotlib](https://matplotlib.org/3.1.1/api/index.html). Using it, we can display our checkerboard array in image form:

```python
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
plt.imshow(checkBoard, cmap='gray', interpolation='nearest')
plt.show()
```

### As another simple example, we will import the [data](https://scikit-image.org/docs/dev/api/skimage.data.html) module image processing library [scikit-image](https://scikit-image.org/) and load a small image of a bush.

#### First, we want to print the pixel values:

```python
from skimage import data
image_of_a_bush = data.lfw_subset()
image_of_a_bush = image_of_a_bush[0,:,:]

#print the #of dimentions, the shape, and the pixel values of the image
print("The number of dimensions of the image is: ", image_of_a_bush.ndim)
print("The size of the image is: ", image_of_a_bush.shape)
print(image_of_a_bush)
```

### Can you see the bush?

#### Next, show the image:

```python
plt.figure(figsize=(1,1))

# display the image
plt.#
```

# 1. Pixel-level operations
### Now that we have a sense of what a digital image is, let's start manipulating it. We'll begin with simple pixel-level operations

## 1.1 Basic pixel-level operations
### Let's look at a more interesting image. From scikit-image data we'll open a example IHC image, and plot it using pyplot.

```python
from skimage import data
import matplotlib.pyplot as plt
import numpy as np

image_hist = data.immunohistochemistry()
#check the size of the image
print("The number of dimensions of the image is: ", image_hist.ndim)
print("The size of the image is: ", image_hist.shape)
plt.imshow(image_hist, cmap=plt.cm.gray)
```

### Seems like we have an RGB image. Let's look at every channel independently.

```python
plt.figure(figsize=(15,5))
plt.subplot(131)
plt.gca().set_title('Red channel')
plt.imshow(, cmap='Reds', interpolation='nearest')
plt.subplot(132)
plt.gca().set_title('Green channel')
plt.imshow(, cmap='Greens', interpolation='nearest')
plt.subplot(133)
plt.gca().set_title('Blue channel')
plt.imshow(, cmap='Blues', interpolation='nearest')

plt.show()
```

```python
#for the moment let's look at only the first color channel
image_hist = image_hist[:,:,0]
plt.gca().set_title('First channel')
plt.imshow(image_hist, cmap=plt.cm.gray)

```

### We can invert the image using the *invert* function from [scikit-images utilities module](https://scikit-image.org/docs/dev/api/skimage.util.html):

```python
from skimage.util import invert

inverted_image = #

plt.figure(figsize=(15,5))
plt.subplot(121)
plt.gca().set_title('original image')
plt.imshow(image_hist, cmap=plt.cm.gray)
plt.subplot(122)
plt.gca().set_title('inverted image')
plt.imshow(inverted_image, cmap=plt.cm.gray)

```

### Let's try some other pixel-level operations. We'll use the [Exposure module](https://scikit-image.org/docs/dev/api/skimage.exposure.html#skimage.exposure.adjust_sigmoid) from scikit image.

1. A gamma correction applies the nonlinear transform $V_{out} = V_{in}^\gamma$.

2. A log transform applies $V_{out} = log(V_{in}+1)$.

3. A sigmoid transform applies $V_{out} = \frac{1}{1+e^{gain\cdot(\text{cutoff}-V_{in})}}$.

4. Equalization transforms the intensity histogram of an image to a uniform distribution. It often enhances the contrast of the image

5. Contrast Limited Adaptive Histogram Equalization (CLAHE) works similarly to equalization thats applied separately to different regions of the image.

Try to apply these by calling the relevant function from skimage.exposure, or by direct calculation.

Play with the different parameters and see how they change the output.

```python
from skimage import exposure

# apply gamma scaling with gamma=2
gamma=
gamma_corrected =

# apply logarithmic scaling
logarithmic_corrected =

# apply sigmoidal scaling with cutoff=0.4
cutoff =
sigmoid_corrected =

# equalize
equalize_corrected =

# apply Contrast Limited Adaptive Histogram Equalization (CLAHE)
CLHA_corrected =

plt.figure(figsize=(15,10))

plt.subplot(231)
plt.gca().set_title('original')
plt.imshow(image_hist, cmap=plt.cm.gray)

plt.subplot(232)
plt.gca().set_title('gamma corrected')
plt.imshow(gamma_corrected, cmap=plt.cm.gray)

plt.subplot(233)
plt.gca().set_title('log corrected')
plt.imshow(logarithmic_corrected, cmap=plt.cm.gray)

plt.subplot(234)
plt.gca().set_title('sigmoid')
plt.imshow(sigmoid_corrected, cmap=plt.cm.gray)

plt.subplot(235)
plt.gca().set_title('equalized')
plt.imshow(equalize_corrected, cmap=plt.cm.gray)

plt.subplot(236)
plt.gca().set_title('CLHA corrected')
plt.imshow(CLHA_corrected, cmap=plt.cm.gray)
```

## 1.2 Image filtering

### Spatial filtering is an image processing technique for changing the intensities of a pixel according to the intensities of some neighborhood of pixels.

### The *Kernel* of the filter defines the neighborhood and the weights asigned to each pixel in the neighborhood:

This procedure is formally a convolution and is marked by an asterisk: $I_o = I_i\ast f$.

*side note: since a convolution in the spatial domain is equivalent to multiplication in the frequency domain. Sometimes it is more computationally reasonable to calculate these in fourier space.*

*side side note: filtering can also be performed in the frequency domain by directly removing a set of frequencies from an image.*

### The kernel can be of any shape/size, it is applied to each pixel in the image, and the output is a new, filtered, image. The output image is often called the *response* to the given filter.
Example, local average:

#### Filtering is an incredibly versatile tool with which you can emphasize certain features or remove other features.
#### Image processing operations implemented with filtering include smoothing, sharpening, and edge enhancement.

### To implement different image filters, we will use the [filters module from scikit-image](https://scikit-image.org/docs/dev/api/skimage.filters.html)

### 1.2.1 Smoothing

#### Smoothing, aka low-pass filtering, is used for removing high-frequency noise from images. Most commonly, a gaussian kernel is used, but others (e.g. local mean/median) work too. We'll see the effect of gaussian filtering.

Try to change the value of sigma (width of the gaussian) to see how the output changes.

```python
import matplotlib.pyplot as plt
import numpy as np

from skimage import filters

image_hist = data.immunohistochemistry()

sigma = 2

gauss_filtered_img =

plt.figure(figsize=(15,8))
plt.subplot(121)
plt.gca().set_title('original image')
plt.imshow(image_hist, cmap=plt.cm.gray)
plt.subplot(122)
plt.gca().set_title('response, gaussian smoothing')
plt.imshow(gauss_filtered_img, cmap=plt.cm.gray)

```

### 1.2.2 Sharpening

#### sharpening is sometimes used to enhance a blurry (i.e. crappy) image.

1. Start with input image
2. Apply gaussian filter with very narrow kernel
3. Subtract filtered image from input image to get only high frequency components
3. Amplify (alpha) and add high frequency components to original input image

```python
filter_blurred_f = filters.gaussian(gauss_filtered_img, sigma=0.5, multichannel=False)
alpha = 3
sharpened = gauss_filtered_img + alpha * (gauss_filtered_img - filter_blurred_f)

plt.figure(figsize=(15,8))
plt.subplot(121)
plt.gca().set_title('input - blury image')
plt.imshow(gauss_filtered_img, cmap=plt.cm.gray)
plt.subplot(122)
plt.gca().set_title('sharpened')
plt.imshow(sharpened, cmap=plt.cm.gray)

```

#### Direct application of edge detectors often results in somewhat noisy responses. A way to overcome this is by first smoothing the image with a gaussian filter and then applying the edge filter. The width of the gaussian kernel will determine the size of edges the filter detects.

### 1.2.3 Edge enhancement

#### Edge detecting filters work by measuring the local spatial gradient of an image. Common types are the Sobel, Prewitt and Roberts.

#### The filters are usually applied to each direction individually and then the total magnitude of the gradient is calculated.
$|\nabla| = \sqrt{\nabla_x^2+\nabla_y^2}$

#### Sobel:

#### Prewitt:

#### Roberts:

```python
from skimage import data
image_hist = data.immunohistochemistry()
image_hist = image_hist[:,:,2]

# sobel magnitude
filtered_img =
# sobel horizontal
filtered_img_h =
# sobel vertical
filtered_img_v =

plt.figure(figsize=(15,16))
plt.subplot(221)
plt.gca().set_title('input image')
plt.imshow(image_hist[:,:,2], cmap=plt.cm.gray)
plt.subplot(222)
plt.gca().set_title('sobel filter response - magnitude')
plt.imshow(filtered_img, cmap=plt.cm.gray)
plt.subplot(223)
plt.gca().set_title('sobel filter response - horizontal edges')
plt.imshow(np.abs(filtered_img_h), cmap=plt.cm.gray)
plt.subplot(224)
plt.gca().set_title('sobel filter response - vertical edges')
plt.imshow(np.abs(filtered_img_v), cmap=plt.cm.gray)
```

#### Direct application of edge detectors often results in somewhat noisy responses. A way to overcome this is by first smoothing the image with a gaussian filter and then applying the edge filter. The width of the gaussian kernel will determine the size of edges the filter detects.

```python
from skimage import data
from skimage import feature

image_hist = data.immunohistochemistry()
image_hist = image_hist[:,:,2]

# sobel magnitude
filtered_img = filters.sobel(image_hist)

#apply a gaussian filter, followed by a sobel filter with sigma=3
sigma=
DoG_img =

plt.figure(figsize=(15,16))
plt.subplot(221)
plt.gca().set_title('input image')
plt.imshow(image_hist, cmap=plt.cm.gray)

plt.subplot(223)
plt.gca().set_title('sobel filter response - magnitude')
plt.imshow(filtered_img, cmap=plt.cm.gray)
plt.subplot(224)
plt.gca().set_title('DoG filter response - magnitude')
plt.imshow(DoG_img, cmap=plt.cm.gray)
```

### 1.2.4 Gaussian derivatives
Now let's remember that the application of filters is an associative operation (because convolution is linear!). It's equivalent to apply the gaussian and then the gradient filter to the image and to apply the gradient filter to the gaussian, then apply the result to the image, i.e. $\nabla*(G*I) = (\nabla*G)*I$ where $\nabla$ is the gradient (derivative) filter, $G$ is the gaussian filter, and $I$ is the image.

We can generalize this idea and apply the derivative multiple times, to get a family of interesting filters:

$\nabla^n*G$:

Notice that all of the even orders are edge detectors, while all the odd orders are ridge detectors!

### Let's use a second order derivative (aka a Laplacian) and make a ridge detector:
#### We'll look at a retinal photo where vasculature is an interesting feature. We invert the image so that the vasculature appears as bright lines on a dark background

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg
from skimage.color import rgb2gray
from skimage.util import invert

#this is how we load an image from the hard drive
image_retina = ((img_as_float(mpimg.imread("../Data/RetinalPhoto.png"))))
image_ridges = invert(rgb2gray(image_retina))

plt.figure(figsize=(15,5))
plt.subplot(121)
plt.gca().set_title('retinal photo')
plt.imshow(image_retina, cmap=plt.cm.gray)
plt.subplot(122)
plt.gca().set_title('inverted grayscale image')
plt.imshow(image_ridges, cmap=plt.cm.gray)

```

**Apply a gaussian filter, followed by a second order derivative (laplacian).**

```python
sigma=3
LoG_ridges =

plt.imshow(LoG_ridges,cmap='gray')
plt.gca().set_title('retinal photo - response to LoG')
```

**This commonly used ridge detector is called a Laplacian of Gaussian (LoG)!**

(We'll get back to this image in a bit)

## 1.3 Masking

### A mask is a binary image (0s and 1s) that typically separates a given input image into Foreground (interesting) and Background (boring) regions, or for picking a region-of-interest (ROI). A mask is *applied* to an image by element-wise multiplication. The size of a mask must be *identical* to the size of the image it's applied to.

### Let's begin by creating a simple circular mask. We'll create an array where the value at each point is it's distance from the center of the image, and display it as an image:

```python
import matplotlib.pyplot as plt
import numpy as np

#dimensions in x and y
y = 512
x = 512

#position of center
centY = np.ceil(y/2)
centX = np.ceil(x/2)

#create the grid
yy,xx = np.indices((y,x))

#create radial distance map
radialDist =

#display
plt.gca().set_title('Radial distance')
plt.imshow(radialDist, cmap='gray', interpolation='nearest')
plt.show()
```

### Of these points, we'll pick a circle of radius 100 and display it as an image:

```python
circ1 =

plt.show()
plt.gca().set_title('Circle with radius 100')
plt.imshow(circ1, cmap='inferno', interpolation='nearest')
```

#### This object is a **mask**. If you multiply this matrix of 0s and 1s with an image of the same size, only the parts that are ==1 will remain

### Let's apply this mask to our histology image.

```python
plt.gca().set_title('Masked first channel')
plt.imshow(, cmap=plt.cm.gray)

```

### What happens if we invert the mask?

```python
inverted_mask =

plt.figure(figsize=(15,5))
plt.subplot(121)
plt.gca().set_title('inverted mask')
plt.imshow(inverted_mask, cmap=plt.cm.gray)
plt.subplot(122)
plt.gca().set_title('inverted masked image')
plt.imshow(image_hist[:,:,2]*, cmap=plt.cm.gray)
```

**Just for closure, let's see what happens when we look at the full RGB image and try to apply the mask**

```python
image = data.immunohistochemistry()

masked_image = image*circ1
plt.imshow(masked_image, cmap=plt.cm.gray)
```

**Whoops. Seems like something is wrong. Our problem is that numpy didn't know how to multiply a 512x512x3 with a 512x512 mask. Numpy makes solving this very easy by adding a singleton dimension (look up broadcasting in your spare time).**

```python
image = data.immunohistochemistry()
plt.gca().set_title('Masked image')
masked_image = mage*np.expand_dims(circ1,2)
plt.imshow(masked_image, cmap=plt.cm.gray)

```

## 1.4 Thresholding
## 1.4.1 Simple thresholding
### Thresholding an image is the process of setting an intensity (or intensities) for separating the different components of an image.

#### In simplest case, the foreground and background have very different intensities. In that case thresholding is just clustering pixels by their intensity levels.

```python
#this function from skimage converts images of integer types into floats, which are easier to work with.
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
from skimage import data

# First, let's create a noisy image of blobs
image_blobs = img_as_float(data.binary_blobs(length=512, seed=1))
sigma = 0.22
image_blobs += np.random.normal(loc=0, scale=sigma, size=image_blobs.shape)

print("The number of dimensions of the image is: ", image_blobs.ndim)
print("The size of the image is: ", image_blobs.shape)
plt.imshow(image_blobs, cmap=plt.cm.gray)
```

### To find the right threshold, let's examine a histogram of pixel intensity values

```python
plt.hist(image_blobs.flatten(),bins=250)
plt.show()
```

Pick an appropriate threshold, by eye, and see if you can remove the background.
What happens when you increase or decrease the threshold?

```python
thresh =

mask =

masked_image =

plt.figure(figsize=(15,5))
plt.subplot(131)
plt.gca().set_title('original')
plt.imshow(image_blobs, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(132)
plt.gca().set_title('mask')
plt.imshow(mask, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(133)
plt.gca().set_title('masked image')
plt.imshow(masked_image, interpolation='nearest', cmap=plt.cm.gray)
```

### Our mask looks ok, but it has a lot of salt & pepper speckle noise. Why is that?
We can try and use what we learned before about filtering to clean up our results. What filter should we use?

```python
from skimage import filters

thresh =

mask =

masked_image =

plt.figure(figsize=(15,5))
plt.subplot(131)
plt.gca().set_title('original')
plt.imshow(image_blobs, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(132)
plt.gca().set_title('mask')
plt.imshow(mask, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(133)
plt.gca().set_title('masked image')
plt.imshow(masked_image, interpolation='nearest', cmap=plt.cm.gray)
```

It's usually a good idea before creating a mask to despeckle an image using a narrow gaussian filter!

## 1.4.2 Morphological operations
Morphology is a broad set of image processing operations that process images based on shapes. In a morphological operation, each pixel in the image is adjusted based on the value of other pixels in its neighborhood. By choosing the size and shape of the neighborhood, you can construct a morphological operation that is sensitive to specific shapes in the input image. (explanation from Mathworks)

Morphological operations are based around a *structuring element*, which is a small binary image, often of a disk or a square. The structuring element is positioned at all possible locations in the image and it is compared with the corresponding neighbourhood of pixels. Some operations test whether the element "fits" within the neighbourhood, while others test whether it "hits" or intersects the neighbourhood.

Common operations for image processing

Erosion - output image =1 wherever the structuring element **fits** (erodes the mask)

Dilation - output image =1 wherever the structuring element **hits** (expands the mask)

Opening - Erosion followed by dilation (opens gaps in spots where the mask is weakly connected)

Closing - Dilation followed by erosion (closes holes in the mask)

A very thorough explanation of morphological operationscould be found [here](https://www.cs.auckland.ac.nz/courses/compsci773s1c/lectures/ImageProcessing-html/topic4.htm)

```python
from skimage.morphology import erosion, dilation, opening, closing
from skimage.morphology import disk

#define a "disk" structuring element with radius 10
selem =

#apply erosion, dilation, opening, and closing
erosion_mask =

dilation_mask =

opening_mask =

closing_mask =

plt.figure(figsize=(15,10))
plt.subplot(231)
plt.gca().set_title('mask')
plt.imshow(mask, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(232)
plt.gca().set_title('erosion')
plt.imshow(erosion_mask, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(233)
plt.gca().set_title('dilation')
plt.imshow(dilation_mask, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(235)
plt.gca().set_title('opening')
plt.imshow(opening_mask, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(236)
plt.gca().set_title('closing')
plt.imshow(closing_mask, interpolation='nearest', cmap=plt.cm.gray)

```

## 1.4.3 Masking actual data

### We'll repeat the thresholding procedure using an actual microscopy image of fluorescent nuclei

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg

#this is how we load an image from the hard drive
image_nuclei = img_as_float(mpimg.imread("../Data/xy040-1.png"))

fig = plt.figure(num=None, figsize=(7.1, 4.6), dpi=80, facecolor='w', edgecolor='k')
print("The number of dimensions of the image is: ", image_nuclei.ndim)
print("The size of the image is: ", image_nuclei.shape)
plt.imshow(image_nuclei, cmap=plt.cm.gray, vmin=0, vmax=0.01)
```

### Again, let's plot a histogram of intensity values

```python
plt.
plt.xlim((0, 0.02))
plt.show()

```

### And again we'll pick a value by eye:

```python
thresh =

#remember to despeckle before creating a mask!
mask =

plt.figure(figsize=(8,15))
plt.subplot(311)
plt.gca().set_title('original')
plt.imshow(image_nuclei, interpolation='nearest', cmap=plt.cm.gray, vmin=0, vmax=0.01)
plt.subplot(312)
plt.gca().set_title('mask')
plt.imshow(mask, interpolation='nearest', cmap=plt.cm.gray)
plt.subplot(313)
plt.gca().set_title('masked image')
plt.imshow(image_nuclei*mask, interpolation='nearest', cmap=plt.cm.gray, vmin=0, vmax=0.01)
```

### Not bad! But also not very scalable. If we have 100s of images we can't look at them one-by-one and find thresholds by eye. Next, we'll look at some methods for automatically finding the thresholds.

## 1.5 Automated threshold calculation

### There is a very large list of algorithms for threshold calculation out there that are optimized for different situations. We will briefly review a few of the most common ones.

### 1.5.1 Iterative mean thresholding
Algorithm:
1. Start with some threshold $T_i$
2. Compute the means $m_0$ and $m_1$ of the BG and FG
3. Update $T_{i+1} = \frac{m_0+m_1}{2}$
4. Repeat until it converges

### 1.5.2 Otsu thresholding
The algorithm exhaustively searches for the threshold that minimizes the intra-class variance, defined for a given threshold $T$ as a weighted sum of variances of the two classes:
$\sigma^2_w(T)=\omega_0(T)\sigma^2_0(T)+\omega_1(T)\sigma^2_1(T)$

For 2 classes, minimizing the intra-class variance is equivalent to maximizing inter-class variance, which is much easier to calculate:
\begin{align}
\sigma^2_b(T) & =\sigma^2-\sigma^2_w(T)=\omega_0(\mu_0-\mu_T)^2+\omega_1(\mu_1-\mu_T)^2 \\
& =\omega_0(T) \omega_1(T) \left[\mu_0(T)-\mu_1(T)\right]^2
\end{align}

### 1.5.3 Triangle thresholding
Algorithm:
1. Draw a straight line between the histogram peak and the brightest value.
2. From every point on that line, draw the shortest connecting line to the histogram.
3. Find longest of these connecting lines.
4. Threshold is set at the intersection of that line and the curve.

*note: Triangle thresholding is good for situations where the image is mostly background, and there is no clear "peak" of bright pixels.*

### [scikit-image's filters module](https://scikit-image.org/docs/dev/api/skimage.filters.html) implements a large variety of thresholding algorithms. Let's apply the ones we just learned about.

```python
from skimage import filters

#calculate iterative mean threshold
meanThresh =
print(meanThresh)

#calculate otsu threshold
OtsuThresh =
print(OtsuThresh)

#calculate triangle
TriThresh =
print(TriThresh)

```

### Let's look at the resulting masks we get with each of these thresholds

```python
fig = plt.figure(num=None, figsize=(12, 8), dpi=80)
ax1 = fig.add_axes([0.1,0.6,0.4,0.4])

ax1.hist(image_nuclei.flatten(),bins=250)

ax1.axvline(meanThresh, color='g', linestyle='--')
ax1.axvline(OtsuThresh, color='r', linestyle='--')
ax1.axvline(TriThresh, color='k', linestyle='--')

ax1.legend(['mean' ,'otsu', 'triangle'])
ax1.set_title('histogram')

ax2 = fig.add_axes([0.6,0.6,0.4,0.4])
#get iterative mean mask (remember to despeckle)
mask_mean =

ax2.imshow(mask_mean)
ax2.set_title('Iterative mean')
ax2.set_axis_off()

ax2 = fig.add_axes([0.1,0.1,0.4,0.4])
#get otsu mask
mask_otsu =

ax2.imshow(mask_otsu)
ax2.set_title('Otsu')
ax2.set_axis_off()

ax2 = fig.add_axes([0.6,0.1,0.4,0.4])
#get triangle mask
mask_tri =

ax2.imshow(mask_tri)
ax2.set_title('Triangle')
ax2.set_axis_off()

```

## Let's briefly look back at the retinal images and try to mask only the vasculature:

**as before, load the image, invert, and apply LoG filter:**

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg
from skimage.color import rgb2gray

#this is how we load an image from the hard drive
image_ridges = invert(rgb2gray(img_as_float(mpimg.imread("../Data/RetinalPhoto.png"))))

plt.imshow(image_ridges, cmap=plt.cm.gray)
plt.gca().set_title('retinal photo')
sigma = 3
LoG_ridges = filters.laplace(filters.gaussian(image_ridges, sigma=sigma))
ax2.set_axis_off()
```

**Now, let's do the same procedure of automatically finding thresholds:**

```python
from skimage import filters

meanThresh = filters.threshold_mean(LoG_ridges)
print(meanThresh)

OtsuThresh = filters.threshold_otsu(LoG_ridges)
print(OtsuThresh)

TriThresh = filters.threshold_triangle(LoG_ridges)
print(TriThresh)

fig = plt.figure(num=None, figsize=(12, 8), dpi=80)
ax1 = fig.add_axes([0.1,0.6,0.4,0.4])

ax1.hist(image_nuclei.flatten(),bins=250)

ax1.axvline(meanThresh, color='g', linestyle='--')
ax1.axvline(OtsuThresh, color='r', linestyle='--')
ax1.axvline(TriThresh, color='k', linestyle='--')
ax1.legend(['mean' ,'otsu', 'triangle'])
ax1.set_title('histogram')

ax1 = fig.add_axes([0.2,0.65,0.3,0.3])

plt.imshow(image_ridges, cmap=plt.cm.gray)
ax1.set_axis_off()

ax2 = fig.add_axes([0.6,0.6,0.4,0.4])
mask_mean =

ax2.imshow(mask_mean)
ax2.set_title('Iterative mean')
ax2.set_axis_off()

ax2 = fig.add_axes([0.1,0.1,0.4,0.4])
mask_otsu =

ax2.imshow(mask_otsu)
ax2.set_title('Otsu')
ax2.set_axis_off()

ax2 = fig.add_axes([0.6,0.1,0.4,0.4])
mask_tri =

ax2.imshow(mask_tri)
ax2.set_title('Triangle')
ax2.set_axis_off()
```

**Play around with the width of the gaussian. Which thresholding algorithm works best in this case?**

### 1.5.4 Local thresholding
#### All of the methods we saw so far are *global* in the sense that the same threshold is applied to the whole picture. Sometimes we can have an image with vastly different intensity distributions at different locations. Using local thresholding, we can overcome such cases.

Let's compare the results from a global (Otsu) and a local threshold.

```python
from skimage import data

image = data.page()
fig = plt.figure(num=None, figsize=(12, 8), dpi=80)

#global thresholding
threshGlobal = filters.threshold_otsu(image)

ax1 = fig.add_axes([0.1,0.6,0.4,0.4])
ax1.set_title('mask - Otsu threshold')
plt.imshow(image ,cmap='gray')

ax2 = fig.add_axes([0.6,0.6,0.4,0.4])
ax2.set_title('mask - Otsu threshold')
plt.imshow(image>threshGlobal,cmap='gray')

#local thresholding

#Try and change this number and see what happens
block_size  = 81

#calculate local threshold map
threshLocal =

ax1 = fig.add_axes([0.1,0.2,0.4,0.4])
ax1.imshow(threshLocal,cmap='gray')
ax1.set_title('local threshold map')

ax2 = fig.add_axes([0.6,0.2,0.4,0.4])
ax2.set_title('mask - Local threshold')
plt.imshow(image>threshLocal,cmap='gray')

```

# 2. Image segmentation
### Image segmentation is the process of partitioning a digital image into multiple segments. The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.

## 2.1 Connected components
### After we generate a mask, the simplest segmentation is achieved by taking regions in the mask that are connected and labeling each one as a separate object.

#### We begin by generating a simple mask using the triangle threshold method:

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg
from skimage import filters

image_nuclei = img_as_float(mpimg.imread("../Data/xy040-1.png"))
TriThresh = filters.threshold_triangle(image_nuclei)
#despeckle
mask = filters.gaussian(image_nuclei, sigma=1)>TriThresh
```

### [scikit-image's measure module](https://scikit-image.org/docs/dev/api/skimage.measure.html) implements a  variety of useful methods for segmentation. the *label* function returns a *labeled image* of connected components (CCs). Each CC is uniquely numbered by an integer.

$\begin{bmatrix}
1 & 1 & 0 & 0 & 2\\
1 & 1 & 0 & 2 & 2\\
0 & 0 & 0 & 0 & 0\\
0 & 3 & 0 & 4 & 4\\
0 & 0 & 0 & 4 & 4\\
\end{bmatrix}$

```python
from skimage import measure
#generate a labeled matrix of connected components
labels =

plt.figure(figsize=(12,5))

plt.subplot(121)
plt.imshow(mask, cmap='gray')
plt.subplot(122)
plt.imshow(labels, cmap='nipy_spectral')
```

#### We can easily generate a *mask* for a **specific** CC using the binary operation *labels==i*

```python
i=43
mask_of_CC_i =

plt.figure(figsize=(10,5))
plt.imshow(mask_of_CC_i, cmap='gray')
```

## Problem with simple CC segmentation : overlapping objects

### We often really care about having only a single object per label. Using CC, any overlapping objects will merge into one blob:

```python
i=87
mask_of_CC_i = labels==i
plt.imshow(mask_of_CC_i, cmap='gray')
```

### These problems can be partially resolved using morphological operations, but there's no silver bullet

```python
from skimage.morphology import erosion, dilation, opening, closing
from skimage.morphology import disk

#define a "disk" structuring element
selem1 = disk(10)
selem2 = disk(7)

plt.figure(figsize=(15,10))
plt.subplot(121)
plt.gca().set_title('original')
plt.imshow(mask, cmap='nipy_spectral')
plt.subplot(122)
plt.gca().set_title('opening')
plt.imshow(dilation(erosion(mask, selem1),selem2), interpolation='nearest', cmap='nipy_spectral')

```

## 2.2 Watershed based segmentation

### 2.2.1 The watershed algorithm
 The watershed transformation treats the image it operates upon like a topographic map, with the brightness of each point representing its height, and finds the lines that run along the tops of ridges.

More precisely, the algorithm goes as follows:
1. *Label* local minima (i.e. $S_1$, $S_2$)
2. Move to next higher intensity level
3. Assign to each point the label of it's closest label set.
**By passing the argument *watershed_line = 1* Points equidistant to multiple sets are labeled as boundaries and intensities set to 0**
4. Repeat until all points are labeled

Let's start with a very naive application. We will invert the image, and then simply apply the *watershed* function from the [scikit-image morphology module](https://scikit-image.org/docs/dev/api/skimage.morphology). The function returns a labeled image. We'll plot the edges of that image using a Sobel filter.

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg
from skimage import filters

from skimage.util import invert
from skimage.morphology import watershed

image_nuclei = img_as_float(mpimg.imread("../Data/xy040-1.png"))

#invert image
image_to_watershed =

#Calculate watershed transform, remember to pass the  watershed_line = 1 argument
labels_naive =
plt.figure(figsize=(15,10))

#let's look at all the boundaries
plt.figure(figsize=(12,20))
plt.subplot(211)
plt.gca().set_title('image fed to watershed')
plt.imshow(image_to_watershed, cmap='gray')
plt.subplot(212)
plt.gca().set_title('watershed result')
plt.imshow(filters.sobel(labels_naive), cmap='nipy_spectral')

```

### So this clearly didn't work. Why? How do we fix it?

Noise generates a ton of local minima. Each gets its own basin. This leads to massive oversegmentation.

#### Watershed segmentation is only a *part* of a segmentation pipeline. Preprocessing (denoising, smoothing, seeding minima) of the image is CRUCIAL for it to work well.

The first thing we'll do is to apply the mask that we found before. This is simply done by adding a *mask* argument to the watershed function.

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg
from skimage import filters

from skimage.util import invert
from skimage.morphology import watershed

image_nuclei = img_as_float(mpimg.imread("../Data/xy040-1.png"))

#calculate mask
mask = filters.gaussian(image_nuclei, sigma=1)>TriThresh

#apply mask and invert image
masked image =

inverted_masked_image =

image_to_watershed = inverted_masked_image

#Calculate watershed transform
#Now, also pass the mask to the watershed function so it avoids segmenting the BG
labels_masked =

#let's look at all the boundaries
plt.figure(figsize=(12,20))
plt.subplot(211)
plt.gca().set_title('image fed to watershed')
plt.imshow(image_to_watershed, cmap='gray')
plt.subplot(212)
plt.gca().set_title('watershed result')
plt.imshow(filters.sobel(labels_masked), cmap='nipy_spectral')

```

#### So we got rid of all the BG regions, but we are still oversegmenting. Why?

Let's try to smoothen the image and get rid of the many local minima. How wide should the gaussian kernel be?

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg
from skimage import filters

from skimage.util import invert
from skimage.morphology import watershed

image_nuclei = img_as_float(mpimg.imread("../Data/xy040-1.png"))

#Calculate mask
mask = filters.gaussian(image_nuclei, sigma=1)>TriThresh

#mask, smooth, and invert the image
masked_image = image_nuclei*mask

sigma_for_smoothing =
smoothed_masked_image =

inverted_smoothed_masked_image = invert(smoothed_masked_image)

image_to_watershed = inverted_smoothed_masked_image
#Calculate watershed transform
#pass the mask to the watershed function so it avoids segmenting the BG
labels_masked_smooth = watershed(image_to_watershed, watershed_line = 1, mask=mask)

#let's look at all the boundaries
plt.figure(figsize=(12,20))
plt.subplot(211)
plt.gca().set_title('image fed to watershed')
plt.imshow(image_to_watershed, cmap='gray')
plt.subplot(212)
plt.gca().set_title('watershed result')
plt.imshow(filters.sobel(labels_masked_smooth), cmap='nipy_spectral')

```

### We're starting to get somewhere!! Can we do better?

#### We can do more to help the algorithm by providing local markers (seeds) from which to start the process

#### We will find seeds by calculating local maxima over areas that are larger than 30x30 pixels using the footprint argument for the function peak_local_max

```python
import matplotlib.pyplot as plt
import numpy as np
from skimage import img_as_float
import matplotlib.image as mpimg
from skimage import filters
from skimage import measure

from skimage.util import invert
from skimage.morphology import watershed

from skimage.feature import peak_local_max

image_nuclei = img_as_float(mpimg.imread("../Data/xy040-1.png"))

mask = filters.gaussian(image_nuclei, sigma=1)>TriThresh

#mask, smooth, and invert the image
masked_image = image_nuclei*mask
sigma_for_smoothing = 4
smoothed_masked_image = filters.gaussian(masked_image, sigma=sigma_for_smoothing)
inverted_smoothed_masked_image = invert(smoothed_masked_image)

image_to_watershed = inverted_smoothed_masked_image

#find local peaks to use as seeds
#focus on this function. Look at the effect of different arguments!
#Specifically. look at the footprint argument
MaskedImagePeaks =

#This is for presentation of our markers
#create disk structuring element of radius 5
selem = disk(5)
#dilate local peaks so that close ones merge
peakMask = dilation(MaskedImagePeaks,selem)
# label local peak regions to find initial markers
markers = measure.label(peakMask)

#pass the *markers* argument to the watershed function
labels_localmax_markers = watershed(image_to_watershed,markers, watershed_line = 1, mask=mask)

#let's look at all the boundaries
plt.figure(figsize=(12,20))
plt.subplot(211)
plt.gca().set_title('image fed to watershed')
plt.imshow(image_to_watershed-peakMask, cmap='gray')
plt.clim((0.95, 1))
plt.subplot(212)
plt.gca().set_title('watershed result')
plt.imshow(filters.sobel(labels_localmax_markers), cmap='nipy_spectral')

```

#### This is pretty good! We're still getting a few errors here and there, but there's no big systematic over- or under- segmentation. This is a typical good result when dealing with real data.

# 3. Feature extraction
### Feature extraction is a process of dimensionality reduction by which an initial raw image is reduced to a list of objects and attributes

## 3.1 Extracting region properties

### [scikit-image's measure module](https://scikit-image.org/docs/dev/api/skimage.measure.html)  implements a  method called *regionprops* that accepts a labeled mask of connected components, and, optionally, a corresponding image, and returns a list.  Each object on the list contains useful data about the size, shape, position, and intensity ([see the full list here](https://scikit-image.org/docs/dev/api/skimage.measure.html#skimage.measure.regionprops)) of a specific component.

The length of the list is equal to the total number of objects detected.

#### We'll start by extracting the number of CC we found and the area of each CC

```python
from skimage import measure

#We use regionprops to extract properties on all the CCs
props = measure.regionprops(labels_localmax_markers,image_nuclei)
#how many total connected components did we get?
print(len(props))
props[1].perimeter
```

```python
#This is how we make a list of a specific property for each CC
areas = [r.area for r in props]

#Do the same for the "mean_intensity" property
intensities = [r.mean_intensity for r in props]

#let's look at all the boundaries
plt.figure(figsize=(12,6))
plt.subplot(121)
plt.gca().set_title('areas')
plt.hist(areas)
plt.subplot(122)
plt.gca().set_title('intensities')
plt.hist(intensities)
```

## 3.2 Some options for data presentation

**We can look at individual objects we found**

```python
i=2
plt.imshow(props[i].intensity_image)
plt.gca().set_title('Single cell')
```

**Let's use a scatter plot to compare our results to the image**

```python
intensities = np.array([r.mean_intensity for r in props])
centroids = np.array([r.centroid for r in props])

fig = plt.figure(figsize=(12,15))
fig.add_axes([0.1,0.6,0.4,0.25])
plt.gca().set_title('original')
plt.imshow(image_nuclei, interpolation='nearest', cmap=plt.cm.gray, vmin=0, vmax=0.02)
fig.add_axes([0.6,0.6,0.4,0.25])
plt.scatter(centroids[:,1],centroids[:,0], c=intensities)
plt.axis('equal')
plt.gca().invert_yaxis()
```

**Or even nicer scatter plots!**

```python
intensities = np.array([r.mean_intensity for r in props])
areas = np.array([r.area for r in props])

centroids = np.array([r.centroid for r in props])

fig = plt.figure(figsize=(12,15))
fig.add_axes([0.1,0.6,0.4,0.25])
plt.gca().set_title('original')
plt.imshow(image_nuclei, interpolation='nearest', cmap=plt.cm.gray, vmin=0, vmax=0.02)
fig.add_axes([0.6,0.6,0.4,0.25])
plt.scatter(centroids[:,1],centroids[:,0], c=intensities, s=areas/20)
plt.axis('equal')
plt.gca().invert_yaxis()

plt.text(centroids[10,1],centroids[10,0],props[10].label)
```

**You can even draw your points directly on the image!**

```python
intensities = np.array([r.mean_intensity for r in props])
areas = np.array([r.area for r in props])

centroids = np.array([r.centroid for r in props])

fig = plt.figure(figsize=(12,15))
fig.add_axes([0.1,0.6,0.8,0.5])
plt.gca().set_title('original')
plt.imshow(image_nuclei, interpolation='nearest', cmap=plt.cm.gray, vmin=0, vmax=0.02)
plt.gca().patch.set_alpha(0.5)
plt.scatter(centroids[:,1],centroids[:,0], c=intensities, alpha=1)

```

## 3.3 Converting regionprops to a table (Dataframe)

#### Let's define some useful functions for converting a list of props into a pandas dataframe. These should become obsolete soon since the new version of scikit-image will have this functionality #### These are some useful functions for converting a list of props into a pandas dataframe

```python
import pandas as pd

def scalar_attributes_list(im_props):
    """
    Makes list of all scalar, non-dunder, non-hidden
    attributes of skimage.measure.regionprops object
    """

    attributes_list = []

    for i, test_attribute in enumerate(dir(im_props[0])):

        #Attribute should not start with _ and cannot return an array
        #does not yet return tuples
        if test_attribute[:1] != '_' and not\
                isinstance(getattr(im_props[0], test_attribute), np.ndarray):
            attributes_list += [test_attribute]

    return attributes_list

def regionprops_to_df(im_props):
    """
    Read content of all attributes for every item in a list
    output by skimage.measure.regionprops
    """

    attributes_list = scalar_attributes_list(im_props)

    # Initialise list of lists for parsed data
    parsed_data = []

    # Put data from im_props into list of lists
    for i, _ in enumerate(im_props):
        parsed_data += [[]]

        for j in range(len(attributes_list)):
            parsed_data[i] += [getattr(im_props[i], attributes_list[j])]

    # Return as a Pandas DataFrame
    return pd.DataFrame(parsed_data, columns=attributes_list)
```

### Now, to get all the properties in table form we simply run:

```python
props_df = regionprops_to_df(props)
props_df
```

### Finally, if we imaged our cells in multiple channels, we would want to use the same segmented nuclei and measure intensities of other channels.

```python
from skimage import measure
from skimage import img_as_float
import matplotlib.image as mpimg

image_2ndChannel = img_as_float(mpimg.imread("../Data/xy040-2.png"))

# extract regionprops using labels_localmax_markers mask from image_2ndChannel
props_other_channel = measure.regionprops(labels_localmax_markers,image_2ndChannel)

plt.figure(figsize=(12,6))
plt.subplot(121)
plt.gca().set_title('Nuclei')
plt.imshow(image_nuclei, cmap='gray')
plt.subplot(122)
plt.gca().set_title('other channel')
plt.imshow(image_2ndChannel, cmap='gray')
```

### Extract only the intensity related features

```python
mean_2nd_channel = [r.mean_intensity for r in props_other_channel]
max_2nd_channel = [r.max_intensity for r in props_other_channel]
min_2nd_channel = [r.min_intensity for r in props_other_channel]

plt.gca().set_title('intensities of 2nd channel')
plt.hist(mean_2nd_channel)

```

### Add these new features to the pandas dataframe

```python
props_df['mean_intensity_ch2'] = mean_2nd_channel
props_df['max_intensity_ch2'] = max_2nd_channel
props_df['min_intensity_ch2'] = min_2nd_channel

props_df
```

**Sometimes it's easier to see a bimodal distribution in log scale**

```python
fig = plt.figure(figsize=(12,6))
fig.add_axes([0.1,0.1,0.4,0.4])
plt.gca().set_title('Histogram of intensities')
plt.hist(props_df.mean_intensity_ch2,20)
fig.add_axes([0.6,0.1,0.4,0.4])
plt.gca().set_title('Histogram of log of intensities')

plt.hist(np.log(props_df.mean_intensity_ch2),20)
```

**We can compare distributions of different channels**

```python
plt.figure(figsize=(12,6))

plt.gca().set_title('scatter plot of intensities')
plt.scatter(np.log(props_df['max_intensity_ch2']), np.log(props_df['max_intensity']))
plt.xlabel('Ch2')
plt.ylabel('Ch1')
```

### And so, we've successfully implemented a simple image segmentation pipeline for multicolor microscopy data.

#### Fin.


```
from sympy import *
```

```
n_u, n_b, m_p, m_r = symbols("n_u n_b m_p m_r")
F = MatrixSymbol("F", n_u, n_u)
M = MatrixSymbol("M", n_b, n_b)
C = MatrixSymbol("C", n_b, n_u)
B = MatrixSymbol("B", m_p, n_u)
D = MatrixSymbol("D", m_r, n_b)
```

```
B.T.shape
```

    (n_u, m_p)

```
A = BlockMatrix([[F,B.T,C.T,ZeroMatrix(n_u,m_r)],
                 [B,ZeroMatrix(m_p,m_p),ZeroMatrix(m_p,n_b),ZeroMatrix(m_p,m_r)],
                 [-C,ZeroMatrix(n_b,m_p),M,D.T],
                 [ZeroMatrix(m_r,n_u),ZeroMatrix(m_r,m_p),D,ZeroMatrix(m_r,m_r)]])
A
```

    [     F, B', C',  0]
    [     B,  0,  0,  0]
    [(-1)*C,  0,  M, D']
    [     0,  0,  D,  0]

```
n_u, n_b, m_p, m_r = symbols("n_u n_b m_p m_r")
F = Symbol("F")
M = Symbol("M")
C = Symbol("C")
Ct = Symbol("Ct")
B = Symbol("B")
Bt = Symbol("Bt")
D = Symbol("D")
Dt = Symbol("Dt")

```

```
A = Matrix([[F,Bt,Ct,0],[B,0,0,0],[-C,0,M,Dt],[0,0,D,0]])
A
```

    [ F, Bt, Ct,  0]
    [ B,  0,  0,  0]
    [-C,  0,  M, Dt]
    [ 0,  0,  D,  0]

```
A.LUdecomposition()
```

    ([   1,    0,   0, 0]
    [ B/F,    1,   0, 0]
    [-C/F, -C/B,   1, 0]
    [   0,    0, D/M, 1],
     [F,      Bt,      Ct,       0]
    [0, -B*Bt/F, -B*Ct/F,       0]
    [0,       0,       M,      Dt]
    [0,       0,       0, -D*Dt/M],
     [])


Euler Problem 187
=================

A composite is a number containing at least two prime factors. For example, 15 = 3 × 5; 9 = 3 × 3; 12 = 2 × 2 × 3.

There are ten composites below thirty containing precisely two, not necessarily distinct, prime factors: 4, 6, 9, 10, 14, 15, 21, 22, 25, 26.

How many composite integers, n < 108, have precisely two, not necessarily distinct, prime factors?

```python
from sympy import sieve, primepi
print(sum(primepi(10**8 / p) - i for i, p in enumerate(sieve.primerange(1, 10**4))))
```

    17427258

```python

```