## slot waveguides, MIMIM waveguides etc.
We want to analyze leakage for more complex geometries, so we use something from Prof. Fan's EE236 class

### Slot Waveguide
We analyze the TM mode (Hy, Ex, Ez). The Ez field is discontinuous
We also need the Hy field.

The transcendental dispersion being solved is:
\begin{equation}
    \epsilon_r \bigg(\frac{\alpha}{k_x}\bigg)\tanh(\alpha a) = \tan(k_x(b-a)-\phi)
\end{equation}

where:
\begin{equation}
    \phi = \tan^{-1}\bigg(\epsilon_r\frac{\alpha}{k_x}\bigg)
\end{equation}

In this particular situation, we are assuming solutions of the form:
\begin{equation}
    H_y = \begin{cases}
        Ce^{-\alpha(x-b)} & x>b\\
        A\cosh(\alpha x)  & 0 < x< a \\
        B_1 \cos(k_x(x-a)) + B_2\sin(k_x(x-a)) & a<x<b
    \end{cases}
\end{equation}

where 0 to a is the slot region, a to b is the waveguide region and beyond b is the infinite half space.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python

L0 = 1e-6;
wavelength = 1.55*L0;
eps0 = 8.854e-12;
mu0 = 4*np.pi*10e-7;
c0 = 1/np.sqrt(mu0*eps0);

omega = 2*np.pi*c0/wavelength;

# waveguide parameters
a = 0.05*L0;##0.025*L0; #a gap size
b = a+0.25*L0; #waveguide size
delta = b-a;

n_c = 1; n_s = 1; #cover and substrate
n_film = 3.48;
eps_film = n_film**2;
kx_max = np.sqrt(omega**2/c0**2*(eps_film-1)); # = kx^2+alpha^2;

beta = np.linspace(1.4*omega/c0, 1.65*omega/c0, 1e7);

## dispersion relations
kx = np.sqrt(eps_film*omega**2/c0**2 - beta**2);
#as we have -alpha^2, we must have beta larger than omega/c0
alpha = np.sqrt(beta**2 - omega**2/c0**2);

# %what we have now are circles, but we need to know what alphas
# % and what kx's are actually allowed
# % part c) in the homework gives us relationship between kx and alpha

a_kx_ratio = alpha/kx;
RHS = eps_film*(a_kx_ratio)*np.tanh(alpha*a);
# figure()
# plot(kx, RHS, 'linewidth', 2)
# hold on;
phi = np.arctan(eps_film*(a_kx_ratio)); #phase shift
LHS = np.tan(kx*delta - phi) ;
# plot(kx, LHS, 'linewidth', 2)
# xlabel('kx (1/m)')
# legend('RHS of dispersion', 'LHS of dispersion')
# title('Graphical Solution to Dispersion Relation')

# %get the allowed kx and alpha (this is where RHS = LHS
match = 0;
for i in range(len(LHS)):
    if(abs(LHS[i] - RHS[i])/abs(LHS[i])<1e-7):
        match = i;

## plot the Ex field given kx and alpha information and beta
kx = kx[i];
alpha = alpha[i];
beta = (eps_film*omega**2/c0**2 - kx**2)**.5;

# test the dispersion relations at this point
x1 = np.linspace(0, a, 1000);
x2 = np.linspace(a, b, 1000);
x3 = np.linspace(b, 500e-9, 1000);

A = 1; #free parameter A
coef= A; #we don't even need beta/omega

a_kx = alpha/kx;

B1 = np.cosh(alpha*a);
B2 = (alpha*eps_film/kx)*np.sinh(alpha*a);
C =  (np.cosh(alpha*a)*np.cos(kx*delta) +(eps_film*alpha/kx)*np.sinh(alpha*a)*np.sin(kx*delta));

## Hy field
Hz1 = np.cosh(alpha*x1);
Hz2= B1*np.cos(kx*(x2-a))+B2*np.sin(kx*(x2-a));
Hz3 = C*np.exp(-alpha*(x3-b));

## directly model the electric field in half of the structure (structure is symmetric about x =0)
E1 = np.cosh(alpha*x1);
E2 = ((np.cosh(alpha*a)/eps_film)*np.cos(kx*(x2-a)) + \
    (alpha/kx)*np.sinh(alpha*a)*np.sin(kx*(x2-a)));
E3 = (np.cosh(alpha*a)*np.cos(kx*delta) + \
    (eps_film*alpha/kx)*np.sinh(alpha*a)*np.sin(kx*delta))*np.exp(-alpha*(x3-b));

E1_max = max(abs(E1));E2_max = max(abs(E2));
E3_max = max(abs(E3));

E_max = max([E1_max, E2_max, E3_max]);
E1 = E1/E_max; E2 = E2/E_max; E3 = E3/E_max;
#normalize by max field intensity

print(a,b)
```

    D:\Anaconda3\lib\site-packages\ipykernel_launcher.py:20: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

    5e-08 3e-07

```python
plt.plot(x1/L0, E1, '.b')
plt.plot(x2/L0, E2, '.r')
plt.plot(x3/L0, E3, '.c')
plt.axvline(a/L0, linestyle = '--');
plt.axvline(b/L0, linestyle = '--')
plt.axvline(-a/L0, linestyle = '--');
plt.axvline(-b/L0, linestyle = '--')

plt.plot(-x1/L0, E1, '.b')
plt.plot(-x2/L0, E2, '.r')
plt.plot(-x3/L0, E3, '.c')
plt.ylabel('Ex-field (normalized)')
plt.xlabel('x-direction (microns)')
plt.title('Ex field of fundamental TM mode')
plt.text(0.2,0.6,'$\lambda_0$=1.55 microns')
print(alpha*L0)

plt.figure();
plt.plot(x1/L0, Hz1, '.b')
plt.plot(x2/L0, Hz2, '.r')
plt.plot(x3/L0, Hz3, '.c')
plt.axvline(a/L0, linestyle = '--');
plt.axvline(b/L0, linestyle = '--')
plt.axvline(-a/L0, linestyle = '--');
plt.axvline(-b/L0, linestyle = '--')
plt.plot(-x1/L0, Hz1, '.b')
plt.plot(-x2/L0, Hz2, '.r')
plt.plot(-x3/L0, Hz3, '.c')
```

## Application to Adjacent air core (MIMIM) waveguides
In this situation, we should be able to modify the existing solution for the dielectric slot waveguide. Let's start with the out of plane H field in the TM polarization:
In this particular situation, we are assuming solutions of the form:
\begin{equation}
    H_y = \begin{cases}
        Ce^{-\alpha(x-b)} & x>b\\
        A\cosh(\alpha x)  & 0 < x< a \\
        B_1 \cos(k_x(x-a)) + B_2\sin(k_x(x-a)) & a<x<b
    \end{cases}
\end{equation}
In fact, it is not even clear to me that it is necessary to change the form of this solution. AS it stands, in a metal, we have a decaying field and in the separator wall, we also have a decaying field, which must be even.

In order to evaluate the correct solutions using Newton's algorith, we will also need to give the analytic first derivative of the dispersion, as such:
\begin{equation}
    \epsilon_r \bigg(\frac{\alpha}{k_x}\bigg)\tanh(\alpha a) = \tan(k_x(b-a)-\phi)
\end{equation}

where:
\begin{equation}
    \phi = \tan^{-1}\bigg(\epsilon_r\frac{\alpha}{k_x}\bigg)
\end{equation}

We are solving for $k_x$ in this equation, so the derivative looks like it will be quite complicated.

```python
from scipy.optimize import newton
## need newton for this...since we have to probe complex space

L0 = 1e-6;
wavelength = 1.55*L0;
eps0 = 8.854e-12;
mu0 = 4*np.pi*10e-7;
c0 = 1/np.sqrt(mu0*eps0);

omega = 2*np.pi*c0/wavelength;

# waveguide parameters
a = 0.05*L0;##0.025*L0; #a gap size
b = a+0.1*L0; #waveguide size
delta = b-a;

n_c = 1; n_s = 1; #cover and substrate
n_film = 1;
eps_film = n_film**2;
n_c = -1;
n_s = -1;
kx_max = np.sqrt(omega**2/c0**2*(eps_film-1)); # = kx^2+alpha^2;

beta = np.linspace(1.4*omega/c0, 1.65*omega/c0, 1e7);

def mimim_even(beta, a= a, n_c = n_c, n_s = n_s, eps_film = eps_film):
    '''
    uses same transcendental dispersion used for dielectric slot waveguides
    '''
    alpha = np.sqrt(beta**2 - omega**2/c0**2);
    kx = np.sqrt(eps_film*omega**2/c0**2 - beta**2);
    a_kx_ratio = alpha/kx;
    RHS = eps_film*(a_kx_ratio)*np.tanh(alpha*a);
    phi = np.arctan(eps_film*(a_kx_ratio)); #phase shift
    LHS = np.tan(kx*delta - phi) ;
    return RHS-LHS;

def mimim_even_derivative():
    return;

initial_guess = 20*1j; #does initial have to be real?
kans = newton(mimim_even,  x0 = initial_guess, maxiter = 1000)
print('kappa_i: '+str(kans)); #want it to be imaginary for large core sizes

beta = (eps_film*omega**2/c0**2 - kx**2)**.5;

# test the dispersion relations at this point
x1 = np.linspace(0, a, 1000);
x2 = np.linspace(a, b, 1000);
x3 = np.linspace(b, 500e-9, 1000);

A = 1; #free parameter A
coef= A; #we don't even need beta/omega

a_kx = alpha/kx;

B1 = np.cosh(alpha*a);
B2 = (alpha*eps_film/kx)*np.sinh(alpha*a);
C =  (np.cosh(alpha*a)*np.cos(kx*delta) +(eps_film*alpha/kx)*np.sinh(alpha*a)*np.sin(kx*delta));

## Hy field
Hz1 = np.cosh(alpha*x1);
Hz2= B1*np.cos(kx*(x2-a))+B2*np.sin(kx*(x2-a));
Hz3 = C*np.exp(-alpha*(x3-b));

## directly model the electric field in half of the structure (structure is symmetric about x =0)
E1 = np.cosh(alpha*x1);
E2 = ((np.cosh(alpha*a)/eps_film)*np.cos(kx*(x2-a)) + \
    (alpha/kx)*np.sinh(alpha*a)*np.sin(kx*(x2-a)));
E3 = (np.cosh(alpha*a)*np.cos(kx*delta) + \
    (eps_film*alpha/kx)*np.sinh(alpha*a)*np.sin(kx*delta))*np.exp(-alpha*(x3-b));

E1_max = max(abs(E1));E2_max = max(abs(E2));
E3_max = max(abs(E3));

E_max = max([E1_max, E2_max, E3_max]);
E1 = E1/E_max; E2 = E2/E_max; E3 = E3/E_max;
#normalize by max field intensity

print(a,b)
print(E2, E3)
```

```python
plt.plot(x1/L0, E1, '.b')
plt.plot(x2/L0, E2, '.r')
plt.plot(x3/L0, E3, '.c')
plt.axvline(a/L0, linestyle = '--');
plt.axvline(b/L0, linestyle = '--')
plt.axvline(-a/L0, linestyle = '--');
plt.axvline(-b/L0, linestyle = '--')

plt.figure();
plt.plot(x1/L0, Hz1, '.b')
plt.plot(x2/L0, Hz2, '.r')
plt.plot(x3/L0, Hz3, '.c')
plt.axvline(a/L0, linestyle = '--');
plt.axvline(b/L0, linestyle = '--')
plt.axvline(-a/L0, linestyle = '--');
plt.axvline(-b/L0, linestyle = '--')
plt.plot(-x1/L0, Hz1, '.b')
plt.plot(-x2/L0, Hz2, '.r')
plt.plot(-x3/L0, Hz3, '.c')

```

## Impedance Transformation Method, finite thickness metal slab
Does this method provide a way to study an MIM waveguide? It should as the finite thickness metal slab is just an IMI waveguide.

It should also be able to provide a way to study arbitrary numbers of layers AS A WAVEGUIDE type system

```python
c0 = 3e8; #m/s
mu0 = 4*np.pi*10e-7;
e0 = 8.854e-12;

## we'll define our own plasma frequency
omega_plasma = 2.7e15; #Hz;

omega1 = np.linspace(0, omega_plasma-1, 1000);
omega2 = np.linspace(omega_plasma+1, 2*omega_plasma, 1000);

#convert omegas into lambdas
lambda1 = 2*np.pi*c0/omega1;
lambda2 = 2*np.pi*c0/omega2;

kappa = (omega1/c0)*(omega_plasma**2/omega1**2-1)**.5;
kz = (omega2/c0)*(1 - omega_plasma**2/omega2**2)**.5;

d = 3*2*np.pi*c0/omega_plasma; # scale of microns

#onset of plasmons occurs at omega_p/sqrt(2);

#calculate alpha in the slab
alpha = kappa;

#k is the wavevector pointing down
Z02 = 1j*omega1*mu0/alpha; #inside the slab has a special form
Z01 = np.sqrt(mu0/e0)*1;      #normal incidence;
Z03 = Z01;

## construct the reflection coefficient when omega < omega_p (negative situation)
r_coef = np.tanh(alpha*d)*(Z02**2 - Z01**2)/(2*Z02*Z01 + np.tanh(alpha*d)*(Z01**2 + Z02**2));
R = abs(r_coef)**2;

## ow construct the reflection coefficient when omega > omega_p real situation
#in this case Z02 has a new form
Z02 = np.sqrt(mu0/e0*(1-omega_plasma**2/omega2**2) +0*1j);
Z = Z02*(Z01 - 1j*Z02*np.tan(-kz*d))/(Z02 - 1j*Z01*np.tan(-kz*d));
r_coef_2 = (Z - Z01)/(Z + Z01);

R2 = abs(r_coef_2)**2;
plt.figure();
plt.plot(omega1/omega_plasma,R)
#hold on;
#line([1,1], [0,1]);
plt.plot(omega2/omega_plasma, R2);
plt.title('Reflection Spectrum of a Drude Metal')
plt.xlabel('\omega/\omega_p (1/s)')
plt.ylabel('emittance')
plt.show();
```

```python

```


# Diffusion with Chemical Reaction in a One-Dimensional Slab
This is the seventh problem of the famous set of [Ten Problems in Chemical Engineering](https://www.polymath-software.com/ASEE/Tenprobs.pdf).  Here, the goal is to solve second order ordinary differential equations with two point boundary conditions

Jacob Albrecht, 2019

# Problem Setup

For a first order irreversible reaction ($A \rightarrow B$) occuring in a 1D domain, the concentration of species $A$ is governed by the differential equation:

$$\frac{d^2C_A}{dz^2} = \frac{k}{D_{AB}}C_A$$

at location $z = 0$, the boundary condition is:

$$C_A = C_{A0}$$

at the other edge of the domain, $z = L$, the condition is:

$$\frac{dC_A}{dz} = 0$$

# Problem Tasks

a)Numerically solve the differential equation with the boundary conditions with

$C_{A0} = 0.2 kg\cdot mol/m^3$

$k = 10^{-3} s^{-1}$

$D_{AB} = 1.2\cdot10^{-9} m^2/s$

$L = 10^{-3} m$

This solution should  utilize an  ODE  solver  with  a  shooting  technique  and  employ  Newton’s  method  or some other technique for converging on the boundary condition.

b) Compare the concentration profiles over the thickness as predicted by the numerical solution of (a) with the analytical solution of the differential equation.

# Solutions
## Solution to part a)

Rewriting the second-order differential equation and boundary conditions as a set of first-order equations will allow for the use of the scipy integration functions, following [the approach here](https://polymath-software.com/ASEE/Matlab/Matlab.pdf).

$$y_2 = \frac{dC_A}{dz}$$

At $z = 0$, $y_2$ will have an unknown value $\alpha$.  Define new quantities $y_3$ and $y_4$ to incorporate the boundary conditions and to solve for $\alpha$ by the shooting technique

$$y_3 = \frac{\partial C_A}{\partial \alpha}$$

$$y_4 = \frac{dy_3}{dz}$$

$$\frac{dy_4}{dz} = \frac{ky_3}{D_{AB}}$$

The initial conditions for $C_A$, $y_2$, $y_3$, and $y_4$ are $C_{A0}$, $\alpha$, $0$, and $1$, respectively

```python
import numpy as np
from scipy.optimize import newton
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# define function for solving the expanded ode:
def odefun(t,y):
    dy = np.zeros(4)
    dy[0] = y[1]
    dy[1] = k*y[0]/Dab
    dy[2] = y[3]
    dy[3] = k*y[2]/Dab
    return(dy)

# define function for optimizing α
def objfun(alpha):
    y0 = np.array([CA0,alpha,0,1])
    sol = solve_ivp(odefun,tspan,y0)
    err = sol.y[1,-1]/sol.y[3,-1]
    return(err)

# define constants:
L=1e-3
CA0 = 0.2
k = 1e-2
Dab = 1.210e-9
alpha=0

# set integration range:
tspan = [0,L]

# run optimization
opt_α = newton(objfun,alpha)

# use optimal α to calculate profile:
y0 = np.array([CA0,opt_α,0,1])
t_eval = np.linspace(0,L,num=25)
numerical_sol = solve_ivp(odefun,tspan,y0,t_eval=t_eval)
err = numerical_sol.y[1,-1]/numerical_sol.y[3,-1]

print('Optimal \u03B1: {:.4}'.format(opt_α)) # unicode is ok with python
print('Final error: {:.2}'.format(err))
```

    Optimal α: -571.3
    Final error: 5e-14

```python
plt.plot(numerical_sol.t,numerical_sol.y[0,:],'o')
plt.xlabel('z')
plt.ylabel('$C_A$');
```

## Solution to part b)
Instead of using the analytical expression given in the problem statement, we will use `sympy` to calculate the analytical solution from the differential equation and boundary conditions.

```python
from sympy import symbols, Function, Eq, dsolve, Derivative, lambdify

k, D_AB, z, L, C_A0 = symbols('k D_AB z L C_A0')

C = Function('C')

deq = Eq(Derivative(C(z),(z,2))-k/D_AB*C(z))
```

Inspect the differential equation from sympy:

```python
deq
```

$\displaystyle \frac{d^{2}}{d z^{2}} C{\left(z \right)} - \frac{k C{\left(z \right)}}{D_{AB}} = 0$

Solve the differential equation with the boundary conditions and view the output:

```python
analytical_sol = dsolve(deq,ics={C(0):C_A0,
                      Derivative(C(L),L):0})
```

```python
analytical_sol
```

$\displaystyle C{\left(z \right)} = \frac{C_{A0} e^{2 L \sqrt{\frac{k}{D_{AB}}}} e^{- z \sqrt{\frac{k}{D_{AB}}}}}{e^{2 L \sqrt{\frac{k}{D_{AB}}}} + 1} + \frac{C_{A0} e^{z \sqrt{\frac{k}{D_{AB}}}}}{e^{2 L \sqrt{\frac{k}{D_{AB}}}} + 1}$

Substitute in the constant values from the problem statement:

```python
C = analytical_sol.subs({C_A0:0.2,L:1e-3,D_AB:1.210e-9,k:1e-2})
C
```

$\displaystyle C{\left(z \right)} = 0.00063479232011981 e^{2874.79787288035 z} + 0.19936520767988 e^{- 2874.79787288035 z}$

```python
# convert the sympy expression to numpy for plotting
lam_C = lambdify(z,C.rhs, 'numpy')
lam_z = np.linspace(0,1e-3,num=100)
```

```python
plt.plot(lam_z,lam_C(lam_z),label= 'Analytical')
plt.scatter(numerical_sol.t,numerical_sol.y[0,:],facecolors='none',edgecolors='b',label='Numeric')
plt.xlabel('z')
plt.ylabel('$C_A$')
plt.legend()
plt.show()

```

The analytical and numeric solutions match perfectly.  Calculus FTW.

# Reference
“The Use of Mathematical Software packages in Chemical Engineering”, Michael B. Cutlip, John J. Hwalek, Eric H.
Nuttal, Mordechai Shacham, Workshop Material from Session 12, Chemical Engineering Summer School, Snowbird,
Utah, Aug., 1997.

```python
%load_ext watermark
%watermark -v -p scipy,matplotlib,numpy,sympy
```

    CPython 3.7.3
    IPython 7.6.1

    scipy 1.3.0
    matplotlib 3.1.0
    numpy 1.16.4
    sympy 1.4

```python

```


Solution to: [Day 0: Mean, Median, and Mode](https://www.hackerrank.com/challenges/s10-basic-statistics/problem)

Table of Contents

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

- Table of Contents
    - Wiki definition
    - Equations
    - Mean
    - Median
    - Mode
- Solution
    - Imports
    - Constants
    - Input
    - Measures of Central Tendency
    - Scale function
    - Main

# Wiki definition
In statistics, a central tendency (or measure of central tendency) is a central or typical value for a probability distribution.

The most common measures of central tendency are the arithmetic mean, the median, and the mode.
A middle tendency can be calculated for either a finite set of values or for a theoretical distribution, such as the normal distribution.
Occasionally authors use central tendency to denote "the tendency of quantitative data to cluster around some central value."

The central tendency of a distribution is typically contrasted with its dispersion or variability;
dispersion and central tendency are the often characterized properties of distributions.
Analysis may judge whether data has a strong or a weak central tendency based on its dispersion.

## Equations

### Mean
\begin{equation}
\Large
\mu = \frac{\sum_{i=1}^{n}x_{i}}{n}
\end{equation}

### Median

$$
\normalsize
\mathrm{Med}(X) = \begin{cases}
X[\frac{n}{2}] & \text{if n is even} \\
\Large
\frac{X[\frac{n-1}{2}] + X[\frac{n+1}{2}]}{2} & \text{if n is odd}
\end{cases}
$$

### Mode

\begin{equation}
\normalsize
\mathrm{Mode}(X) = \mathrm{max}(\mathrm{frequency})
\end{equation}

# Solution
	- Print on separate lines (mean, median, mode)
	- Scale of 1 decimal place
	- With multiple mode values, select min.

## Imports

```python
from collections import Counter  # used to count instances for mode
from typing import List, Tuple, Union
```

## Constants

```python
SCALE = 1	# scale refers to number of decimal places
```

## Input

```python
def get_input() -> Tuple[int, list]:
    """Returns tuple of input:
    n: int, length of list.
    nums: list, nums to use.
    """
    n = int(input())
    nums = sorted(int(num) for num in input().split())
    return (n, nums)
```

## Measures of Central Tendency

```python
def calc_mean(nums: List[int], n: int) -> Union[int, float]:
	"""Returns mean of nums list with n elements."""
	return format_scale( sum(nums) / n)
```

```python
def calc_median(nums: List[int], n: int) -> Union[int, float]:
	"""Returns median of nums list with n elements."""
	mid, is_odd = divmod(len(nums), 2)
	if is_odd:
		median = nums[mid]
	else:
		median = calc_mean( nums[mid - 1: mid + 1], 2)
	return format_scale(median)
```

```python
def calc_mode(nums: List[int], n: int) -> int:
	"""Returns mode of nums list.

	Uses auxiliary get_mode_ties func.
	"""
	def get_mode_ties(num_counts: dict, max_freq: int) -> int:
		"""Generator function to return contenders for mode in dict."""
		for num, count in num_counts.items():
			if count == max_freq:
				yield num

	num_counts = Counter(nums)
	max_freq = num_counts[ max( num_counts, key = num_counts.get)]

	return min(list(get_mode_ties(num_counts, max_freq)))
```

## Scale function

```python
def format_scale(num: Union[int, float]) -> float:
	"""Returns float number ."""
	if isinstance(num, int):
		return num
	return round(num, SCALE)
```

## Main

```python
def main():
	measure_central_tendency = (
		calc_mean,
		calc_median,
		calc_mode
	)
	n, nums = get_input()
	for measure in measure_central_tendency:
		print(measure(nums, n))
```

```python
if __name__ == "__main__":
	main()
```

    10
    64630 11735 14216 99233 14470 4978 73429 38120 51135 67060
    43900.6
    44627.5
    4978


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *
Image(url= "https://i.imgur.com/kvcC7at.png")
```

```python
L  = Function("L")
f = Function("f")
x,a, y = symbols('x a y')
L(x)
```

$\displaystyle L{\left(x \right)}$

```python
f = Eq(L(x) , f(a) + Derivative(f(a),a)*(x-a))
f
```

$\displaystyle L{\left(x \right)} = \left(- a + x\right) \frac{d}{d a} f{\left(a \right)} + f{\left(a \right)}$

```python
dfa = 4
fa = 76
a = 120
y = 116
print(f.rhs)
```

    (-a + x)*Derivative(f(a), a) + f(a)

```python
f = (-a + x)*dfa + fa
f

```

$\displaystyle 4 x - 404$

```python
f.subs(x,y)
```

$\displaystyle 60$

```python
Image(url= "https://i.imgur.com/wEWyASn.png")
```


# An Introduction to Bayesian Statistical Analysis

Before we jump in to model-building and using MCMC to do wonderful things, it is useful to understand a few of the theoretical underpinnings of the Bayesian statistical paradigm. A little theory (and I do mean a *little*) goes a long way towards being able to apply the methods correctly and effectively.

## What *is* Bayesian Statistical Analysis?

Though many of you will have taken a statistics course or two during your undergraduate (or graduate) education, most of those who have will likely not have had a course in *Bayesian* statistics. Most introductory courses, particularly for non-statisticians, still do not cover Bayesian methods at all, except perhaps to derive Bayes' formula as a trivial rearrangement of the definition of conditional probability. Even today, Bayesian courses are typically tacked onto the curriculum, rather than being integrated into the program.

In fact, Bayesian statistics is not just a particular method, or even a class of methods; it is an entirely different paradigm for doing statistical analysis.

> Practical methods for making inferences from data using probability models for quantities we observe and about which we wish to learn.
*-- Gelman et al. 2013*

A Bayesian model is described by parameters, uncertainty in those parameters is described using probability distributions.

All conclusions from Bayesian statistical procedures are stated in terms of *probability statements*

This confers several benefits to the analyst, including:

- ease of interpretation, summarization of uncertainty
- can incorporate uncertainty in parent parameters
- easy to calculate summary statistics

## Bayesian vs Frequentist Statistics: What's the difference?

Any statistical paradigm, Bayesian or otherwise, involves at least the following:

1. Some **unknown quantities** about which we are interested in learning or testing. We call these *parameters*.
2. Some **data** which have been observed, and hopefully contain information about (1).
3. One or more **models** that relate the data to the parameters, and is the instrument that is used to learn.

### The Frequentist World View

- The data that have been observed are considered **random**, because they are realizations of random processes, and hence will vary each time one goes to observe the system.
- Model parameters are considered **fixed**. The parameters' values are unknown, but they are fixed, and so we *condition* on them.

In mathematical notation, this implies a (very) general model of the following form:

\\[f(y | \theta)\\]

Here, the model \\(f\\) accepts data values \\(y\\) as an argument, conditional on particular values of \\(\theta\\).

Frequentist inference typically involves deriving **estimators** for the unknown parameters. Estimators are formulae that return estimates for particular estimands, as a function of data. They are selected based on some chosen optimality criterion, such as *unbiasedness*, *variance minimization*, or *efficiency*.

> For example, lets say that we have collected some data on the prevalence of autism spectrum disorder (ASD) in some defined population. Our sample includes \\(n\\) sampled children, \\(y\\) of them having been diagnosed with autism. A frequentist estimator of the prevalence \\(p\\) is:

>

> \\[\hat{p} = \frac{y}{n}\\]
>

> Why this particular function? Because it can be shown to be unbiased and minimum-variance.

It is important to note that new estimators need to be derived for every estimand that is introduced.

### The Bayesian World View

- Data are considered **fixed**. They used to be random, but once they were written into your lab notebook/spreadsheet/IPython notebook they do not change.
- Model parameters themselves may not be random, but Bayesians use probability distribtutions to describe their uncertainty in parameter values, and are therefore treated as **random**. In some cases, it is useful to consider parameters as having been sampled from probability distributions.

This implies the following form:

\\[p(\theta | y)\\]

This formulation used to be referred to as ***inverse probability***, because it infers from observations to parameters, or from effects to causes.

Bayesians do not seek new estimators for every estimation problem they encounter. There is only one estimator for Bayesian inference: **Bayes' Formula**.

# Computational Methods in Bayesian Analysis

The process of conducting Bayesian inference can be broken down into three general steps (Gelman *et al.* 2013):

### Step 1: Specify a probability model

As was noted above, Bayesian statistics involves using probability models to solve problems. So, the first task is to *completely specify* the model in terms of probability distributions. This includes everything: unknown parameters, data, covariates, missing data, predictions. All must be assigned some probability density.

This step involves making choices.

- what is the form of the sampling distribution of the data?
- what form best describes our uncertainty in the unknown parameters?

### Step 2: Calculate a posterior distribution

The mathematical form \\(p(\theta | y)\\) that we associated with the Bayesian approach is referred to as a **posterior distribution**.

> posterior /pos·ter·i·or/ (pos-tēr´e-er) later in time; subsequent.

Why posterior? Because it tells us what we know about the unknown \\(\theta\\) *after* having observed \\(y\\).

This posterior distribution is formulated as a function of the probability model that was specified in Step 1. Usually, we can write it down but we cannot calculate it analytically. In fact, the difficulty inherent in calculating the posterior distribution for most models of interest is perhaps the major contributing factor for the lack of widespread adoption of Bayesian methods for data analysis. Various strategies for doing so comprise this tutorial.

**But**, once the posterior distribution is calculated, you get a lot for free:

- point estimates
- credible intervals
- quantiles
- predictions

### Step 3: Check your model

Though frequently ignored in practice, it is critical that the model and its outputs be assessed before using the outputs for inference. Models are specified based on assumptions that are largely unverifiable, so the least we can do is examine the output in detail, relative to the specified model and the data that were used to fit the model.

Specifically, we must ask:

- does the model fit data?
- are the conclusions reasonable?
- are the outputs sensitive to changes in model structure?

## Example: binomial calculation

Binomial model is suitable for data that are generated from a sequence of exchangeable Bernoulli trials. These data can be summarized by $y$, the number of times the event of interest occurs, and $n$, the total number of trials. The model parameter is the expected proportion of trials that an event occurs.

$$p(Y|\theta) = \frac{n!}{y! (n-y)!} \theta^{y} (1-\theta)^{n-y}$$

where $y \in \{0, 1, \ldots, n\}$ and $p \in [0, 1]$.

To perform Bayesian inference, we require the specification of a prior distribution. A reasonable choice is a uniform prior on [0,1] which has two implications:

1. makes all probability values equally probable *a priori*
2. makes calculation of the posterior easy

The second task in performing Bayesian inference is, given a fully-specified model, to calculate a posterior distribution. As we have specified the model, we can calculate a posterior distribution up to a proportionality constant (that is, a probability distribution that is **unnormalized**):

$$P(\theta | n, y) \propto P(y | n, \theta) P(\theta) = \theta^y (1-\theta)^{n-y}$$

We can present different posterior distributions as a function of different realized data.

We can also calculate posterior estimates for $\theta$ by maximizing the unnormalized posterior using optimization.

### Exercise: posterior estimation

Write a function that returns posterior estimates of a binomial sampling model using a uniform prior on the unknown probability. Plot the posterior densities for each of the following datasets:

1. n=5, y=3
2. n=20, y=12
3. n=100, y=60
4. n=750, y=450

what type of distribution do these plots look like?

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

n = [5, 20, 100, 750]
y = [3, 12, 60, 450]
```

```python
# Write your answer here
```

# Evaluating Hypotheses with Bayes

Statistical inference is a process of learning from incomplete or imperfect (error-contaminated) data. Can account for this "imperfection" using either a sampling model or a measurement error model.

### Statistical hypothesis testing

The *de facto* standard for statistical inference is statistical hypothesis testing. The goal of hypothesis testing is to evaluate a **null hypothesis**. There are two possible outcomes:

- reject the null hypothesis
- fail to reject the null hypothesis

Rejection occurs when a chosen test statistic is higher than some pre-specified threshold valuel; non-rejection occurs otherwise.

Notice that neither outcome says anything about the quantity of interest, the **research hypothesis**.

Setting up a statistical test involves several subjective choices by the user that are rarely justified based on the problem or decision at hand:

- statistical test to use
- null hypothesis to test
- significance level

Choices are often based on arbitrary criteria, including "statistical tradition" (Johnson 1999). The resulting evidence is indirect, incomplete, and typically overstates the evidence against the null hypothesis (Goodman 1999).

Most importantly to applied users, the results of statistical hypothesis tests are very easy to misinterpret.

### Estimation

Instead of testing, a more informative and effective approach for inference is based on **estimation** (be it frequentist or Bayesian). That is, rather than testing whether two groups are different, we instead pursue an estimate of *how different* they are, which is fundamentally more informative.

Additionally, we include an estimate of **uncertainty** associated with that difference which includes uncertainty due to our lack of knowledge of the model parameters (*epistemic uncertainty*) and uncertainty due to the inherent stochasticity of the system (*aleatory uncertainty*).

## One Group

Before we compare two groups using Bayesian analysis, let's start with an even simpler scenario: statistical inference for one group.

For this we will use Gelman et al.'s (2007) radon dataset. In this dataset the amount of the radioactive gas radon has been measured among different households in all counties of several states. Radon gas is known to be the highest cause of lung cancer in non-smokers. It is believed to be more strongly present in households containing a basement and to differ in amount present among types of soil.

>  the US EPA has set an action level of 4 pCi/L. At or above this level of radon, the EPA recommends you take corrective measures to reduce your exposure to radon gas.

Let's import the dataset:

```python
import pandas as pd
import seaborn as sns
sns.set_context('notebook')

RANDOM_SEED = 20090425
```

```python
radon = pd.read_csv('../data/radon.csv', index_col=0)
radon.head()
```

Let's focus on the (log) radon levels measured in a single county (Hennepin).

Suppose we are interested in:

- whether the mean log-radon value is greater than 4 pCi/L in Hennepin county
- the probability that any randomly-chosen household in Hennepin county has a reading of greater than 4

```python
hennepin_radon = radon.query('county=="HENNEPIN"').log_radon
sns.distplot(hennepin_radon);
```

### The model

Recall that the first step in Bayesian inference is specifying a **full probability model** for the problem.

This consists of:

- a likelihood function(s) for the observations
- priors for all unknown quantities

The measurements look approximately normal, so let's start by assuming a normal distribution as the sampling distribution (likelihood) for the data.

$$y_i \sim N(\mu, \sigma^2)$$

(don't worry, we can evaluate this assumption)

This implies that we have 2 unknowns in the model; the mean and standard deviation of the distribution.

#### Prior choice

How do we choose distributions to use as priors for these parameters?

There are several considerations:

- discrete vs continuous values
- the support of the variable
- the available prior information

While there may likely be prior information about the distribution of radon values, we will assume no prior knowledge, and specify a **diffuse** prior for each parameter.

Since the mean can take any real value (since it is on the log scale), we will use another normal distribution here, and specify a large variance to allow the possibility of very large or very small values:

$$\mu \sim N(0, 10^2)$$

For the standard deviation, we know that the true value must be positive (no negative variances!). I will choose a uniform prior bounded from below at zero and from above at a value that is sure to be higher than any plausible value the true standard deviation (on the log scale) could take.

$$\sigma \sim U(0, 10)$$

We can encode these in a Python model, using the PyMC3 package, as follows:

```python
from pymc3 import Model, Normal, Uniform

with Model() as radon_model:

    μ = Normal('μ', mu=0, sd=5)
    σ = Uniform('σ', 0, 5)
```

> ## Software
> Today there is an array of software choices for Bayesians, including both open source software (*e.g.*, Stan, PyMC, JAGS, emcee) and commercial (*e.g.*, SAS, Stata). These examples can be replicated in any of these environments.

All that remains is to add the likelihood, which takes $\mu$ and $\sigma$ as parameters, and the log-radon values as the set of observations:

```python
with radon_model:

    dist = Normal('dist', mu=μ, sd=σ, observed=hennepin_radon)
```

Before we go ahead and estimate the model paramters from the data, its a good idea to perform a **prior predictive check**. This involves sampling from the model before data are incorporated, and gives you an idea of the range of observations that would be considered reasonable within the scope of the modeling assumptions (including choice of priors). If the simnulations generate too many extreme observations relative to our expectations based on domain knowledge, then it can be an indication of problems with model formulation.

```python
from pymc3 import sample_prior_predictive

with radon_model:

    prior_sample = sample_prior_predictive(1000)
```

```python
plt.hist(prior_sample['dist'].ravel(), bins=30);
```

```python
plt.hist(radon.log_radon, bins=30);
```

Now, we will fit the model using **Markov chain Monte Carlo (MCMC)**, which will be covered in detail in an upcoming section. This will draw samples from the posterior distribution (which cannot be calculated exactly).

```python
from pymc3 import sample

with radon_model:

    samples = sample(1000, tune=1000, cores=2, random_seed=RANDOM_SEED)
```

```python
from arviz import plot_posterior

plot_posterior(samples, var_names=['μ'], ref_val=np.log(4), credible_interval=0.95, kind='kde');
```

The plot shows the posterior distribution of $\mu$, along with an estimate of the 95% posterior **credible interval**.

The output

    85% < 1.38629 < 15%

informs us that the probability of $\mu$ being less than log(4) is 85% and the corresponding probability of being greater than log(4) is 15%.

> The posterior probability that the mean level of household radon in Henneprin County is greater than 4 pCi/L is 0.15.

### Prediction

What is the probability that a given household has a log-radon measurement larger than one? To answer this, we make use of the **posterior predictive distribution**.

$$p(z |y) = \int_{\theta} p(z |\theta) p(\theta | y) d\theta$$

where here $z$ is the predicted value and y is the data used to fit the model.

We can estimate this from the posterior samples of the parameters in the model.

```python
mus = samples['μ']
sigmas = samples['σ']
```

```python
radon_samples = Normal.dist(mus, sigmas).random()
```

```python
(radon_samples > np.log(4)).mean()
```

> The posterior probability that a randomly-selected household in Henneprin County contains radon levels in excess of 4 pCi/L is about 0.46.

### Model checking

But, ***how do we know this model is any good?***

Its important to check the fit of the model, to see if its assumptions are reasonable. One way to do this is to perform **posterior predictive checks**. This involves generating simulated data using the model that you built, and comparing that data to the observed data.

One can choose a particular statistic to compare, such as tail probabilities or quartiles, but here it is useful to compare them graphically.

We already have these simulations from the previous exercise!

```python
sns.distplot(radon_samples, label='simulated')
sns.distplot(hennepin_radon, label='observed')
plt.legend()
```

### Prior sensitivity

Its also important to check the sensitivity of your choice of priors to the resulting inference.

Here is the same model, but with drastically different (though still uninformative) priors specified:

```python
from pymc3 import Flat, HalfCauchy

with Model() as prior_sensitivity:

    μ = Flat('μ')
    σ = HalfCauchy('σ', 5)

    dist = Normal('dist', mu=μ, sd=σ, observed=hennepin_radon)

    sensitivity_samples = sample(1000, tune=1000)
```

```python
plot_posterior(sensitivity_samples, var_names=['μ'], ref_val=np.log(4));
```

Here is the original model for comparison:

```python
plot_posterior(samples, var_names=['μ'], ref_val=np.log(4));
```

## Two Groups with Continiuous Outcome

To illustrate how this Bayesian estimation approach works in practice, we will use a fictitious example from Kruschke (2012) concerning the evaluation of a clinical trial for drug evaluation. The trial aims to evaluate the efficacy of a "smart drug" that is supposed to increase intelligence by comparing IQ scores of individuals in a treatment arm (those receiving the drug) to those in a control arm (those recieving a placebo). There are 47 individuals and 42 individuals in the treatment and control arms, respectively.

```python
drug = pd.DataFrame(dict(iq=(101,100,102,104,102,97,105,105,98,101,100,123,105,103,100,95,102,106,
        109,102,82,102,100,102,102,101,102,102,103,103,97,97,103,101,97,104,
        96,103,124,101,101,100,101,101,104,100,101),
                         group='drug'))
placebo = pd.DataFrame(dict(iq=(99,101,100,101,102,100,97,101,104,101,102,102,100,105,88,101,100,
           104,100,100,100,101,102,103,97,101,101,100,101,99,101,100,100,
           101,100,99,101,100,102,99,100,99),
                            group='placebo'))

trial_data = pd.concat([drug, placebo], ignore_index=True)
trial_data.hist('iq', by='group');
```

Since there appear to be extreme ("outlier") values in the data, we will choose a Student-t distribution to describe the distributions of the scores in each group. This sampling distribution adds **robustness** to the analysis, as a T distribution is less sensitive to outlier observations, relative to a normal distribution.

The three-parameter Student-t distribution allows for the specification of a mean $\mu$, a precision (inverse-variance) $\lambda$ and a degrees-of-freedom parameter $\nu$:

$$f(x|\mu,\lambda,\nu) = \frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})} \left(\frac{\lambda}{\pi\nu}\right)^{\frac{1}{2}} \left[1+\frac{\lambda(x-\mu)^2}{\nu}\right]^{-\frac{\nu+1}{2}}$$

the degrees-of-freedom parameter essentially specifies the "normality" of the data, since larger values of $\nu$ make the distribution converge to a normal distribution, while small values (close to zero) result in heavier tails.

Thus, the likelihood functions of our model are specified as follows:

$$\begin{align}
y^{(drug)}_i &\sim T(\nu, \mu_1, \sigma_1) \\
y^{(placebo)}_i &\sim T(\nu, \mu_2, \sigma_2)
\end{align}$$

As a simplifying assumption, we will assume that the degree of normality $\nu$ is the same for both groups.

### Prior choice

Since the means are real-valued, we will apply normal priors. Since we know something about the population distribution of IQ values, we will center the priors at 100, and use a standard deviation that is more than wide enough to account for plausible deviations from this population mean:

$$\mu_k \sim N(100, 10^2)$$

```python
with Model() as drug_model:

    μ_0 = Normal('μ_0', 100, sd=10)
    μ_1 = Normal('μ_1', 100, sd=10)
```

Similarly, we will use a uniform prior for the standard deviations, with an upper bound of 20.

```python
with drug_model:
    σ_0 = Uniform('σ_0', lower=0, upper=20)
    σ_1 = Uniform('σ_1', lower=0, upper=20)
```

For the degrees-of-freedom parameter $\nu$, we will use an **exponential** distribution with a mean of 30; this allocates high prior probability over the regions of the parameter that describe the range from normal to heavy-tailed data under the Student-T distribution.

```python
from pymc3 import Exponential

with drug_model:
    ν = Exponential('ν_minus_one', 1/29.) + 1

```

```python
sns.distplot(Exponential.dist(1/29).random(size=10000), kde=False);
```

```python
from pymc3 import StudentT

with drug_model:

    drug_like = StudentT('drug_like', nu=ν, mu=μ_1, lam=σ_1**-2, observed=drug.iq)
    placebo_like = StudentT('placebo_like', nu=ν, mu=μ_0, lam=σ_0**-2, observed=placebo.iq)
```

Now that the model is fully specified, we can turn our attention to tracking the posterior quantities of interest. Namely, we can calculate the difference in means between the drug and placebo groups.

As a joint measure of the groups, we will also estimate the "effect size", which is the difference in means scaled by the pooled estimates of standard deviation. This quantity can be harder to interpret, since it is no longer in the same units as our data, but it is a function of all four estimated parameters.

```python
from pymc3 import Deterministic

with drug_model:

    diff_of_means = Deterministic('difference of means', μ_1 - μ_0)

    effect_size = Deterministic('effect size',
                        diff_of_means / np.sqrt((σ_1**2 + σ_0**2) / 2))

```

```python
with drug_model:

    drug_trace = sample(1000, random_seed=RANDOM_SEED)
```

```python
plot_posterior(drug_trace[100:], var_names=['μ_0', 'μ_1', 'σ_0', 'σ_1', 'ν_minus_one']);
```

```python
plot_posterior(drug_trace[100:],
          var_names=['difference of means', 'effect size'],
          ref_val=0);
```

> The posterior probability that the mean IQ of subjects in the treatment group is greater than that of the control group is 0.99.

## Exercise: Two Groups with Binary Outcome

Now that we have seen how to generalize normally-distributed data to another distribution, try our hand with another data type. Binary outcomes are common in clinical research:

- survival/death
- true/false
- presence/absence
- positive/negative

In practice, binary outcomes are encoded as ones (for event occurrences) and zeros (for non-occurrence). A single binary variable is distributed as a **Bernoulli** random variable:

$$f(x \mid p) = p^{x} (1-p)^{1-x}$$

In terms of inference, we are typically interested in whether $p$ is larger or smaller in one group relative to another.

To demonstrate the comparison of two groups with binary outcomes using Bayesian inference, we will use a sample pediatric dataset. Data on 671 infants with very low (<1600 grams) birth weight from 1981-87 were collected at Duke University Medical Center. Of interest is the relationship between the outcome intra-ventricular hemorrhage (IVH) and predictor such as birth weight, gestational age, presence of pneumothorax and mode of delivery.

```python
vlbw = pd.read_csv('../data/vlbw.csv', index_col=0).dropna(axis=0, subset=['ivh', 'pneumo'])
vlbw.head()
```

To demonstrate binary data analysis, we will try to estimate the difference between the probability of an intra-ventricular hemorrhage for infants with and without a pneumothorax.

```python
pd.crosstab(vlbw.ivh, vlbw.pneumo)
```

We will create a binary outcome by combining `definite` and `possible` into a single outcome.

```python
ivh = vlbw.ivh.isin(['definite', 'possible']).astype(int).values
x = vlbw.pneumo.astype(int).values
```

Fit a model that evaluates the association of a pneumothorax with the probability of IVH.

```python
# Write your answer here
```

---

# References

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis, Third Edition. CRC Press.

Pilon, Cam-Davidson. [Probabilistic Programming and Bayesian Methods for Hackers](http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/)


# Algebraic differentiators: A detailed introduction
This notebook includes a detailed introduction into the theoretical background of algebraic differentiators and shows how to use the proposed implementation.
## Content of this notebook
\textbf{Theoretical background}: Time-domain and frequency-domain analysis

\textbf{Numerical differentiation of sample signals}: The numerical estimation of derivatives of simulation and experimental data.
## Theoretical background
### Time-domain analysis

The algebraic derivative estimation methods  were initially derived using differential algebraic manipulations of truncated Taylor series \cite{mboup2007b,mboup2009a}. Later works \cite{liu2011a,kiltz2017} derived these filters using an approximation theoretical approach that yields a straightforward analysis of the filter characteristics, especially the estimation delay. Using this approach, the estimate of the $n$-th order derivative of a signal ${t\mapsto y(t)}$ denoted $\hat{y}^{(n)}$ can be approximated as
\begin{equation}
\hat{{y}}^{(n)}(t)=\int_{0}^{T}g^{(n)}_T(\tau)y(t-\tau)\textrm{d}\tau,
\end{equation}
with the filter kernel
\begin{equation}
g(t)=\frac{2}{T}\sum_{i=0}^{N}\frac{P_i^{(\alpha,\beta)}(\vartheta)}{\big\lVert{P_i^{(\alpha,\beta)}\big\rVert}^2}w^{(\alpha,\beta)}\left(\nu(t)\right)P_i^{(\alpha,\beta)}\left(\nu(t)\right).
\end{equation}
In the latter equation $\nu(t)=1-\frac{2}{T}t$, $\big\lVert{x}\big\rVert=\sqrt{\langle x,x\rangle}$ is the norm induced by the inner product
    \begin{equation}
        \langle x,y\rangle=\int_{-1}^{1}w^{(\alpha,\beta)}(\tau)x(\tau){y(\tau)}\textrm{d}\tau,
    \end{equation}
    with the weight function
    \begin{equation}
        w^{(\alpha,\beta)}(\tau)=\begin{cases}
        (1-\tau)^\alpha(1+\tau)^\beta,&\quad\tau\in[-1,1],\\
        0,&\quad\text{otherwise},
        \end{cases}
    \end{equation}
    $N$ is the degree of the polynomial approximating the signal $y^{(n)}$ in the time window $[t-T,t]$, $T$ is the filter window length, and $\vartheta$ parameterizes the estimation delay as described below.

 This approach yields a straightforward analysis of the estimation delay $\delta_t$ and the degree of exactness $\gamma$. The degree of exactness was introduced in \cite{kiltz2017} as the polynomial degree up to which the derivative estimation is exact. If $\gamma=2$ for example, the first and second time derivatives of a polynomial signal of degree two are exact up to an estimation delay.  The delay and the degree of exactness  are given as
\begin{align}
\gamma&=\left\{\begin{matrix}
n+N+1,\quad &\text{if } N=0\vee\vartheta=p_{N+1,k}^{(\alpha,\beta)}\\[4pt]
n+N,\quad&\text{otherwise,}
\end{matrix}\right.\\
\delta_t&=\left\{\begin{matrix}
\frac{\alpha+1}{\alpha+\beta+2}T,\quad&\text{if } N=0\\[4pt]
\frac{1-\vartheta}{2}T,\quad&\text{otherwise,}
\end{matrix}\right.
\end{align}
with $p_{N+1,k}^{(\alpha,\beta)}$ the $k$-th zero of the Jacobi polynomial of degree $N+1$.

#### Initializing an algebraic differentiator and performing a time-domain analysis

The parameter $\vartheta$ is initialized by default as the largest zero of the Jacobi polynomial of degree $N+1$. This can be changed using the appropriate class member function (see documentation).

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import sys
sys.path.append("..")
from algebraicDifferentiator import AlgebraicDifferentiator
import numpy as np
ts = 0.001

# Initialize two different differentiators: For the first one the window length is specified
# while the cutoff frequency is specified for the second
diffA = AlgebraicDifferentiator(N=0,alpha=4.,beta=4,T=0.1, ts=ts)
diffB = AlgebraicDifferentiator(N=1,alpha=4.,beta=4.,T=None,wc = 100, ts=ts)
```

    The differentiator has the parameters:
    Alpha: 4.000000
    Beta: 4.000000
    Window length in s: 0.100000
    Polynomial degree: 0
    Estimation delay in s: 0.050000
    Cutoff Frequency: 68.534675
    Discrete window length: 100
    The differentiator has the parameters:
    Alpha: 4.000000
    Beta: 4.000000
    Window length in s: 0.091000
    Polynomial degree: 1
    Estimation delay in s: 0.031781
    Cutoff Frequency: 100.901547
    Discrete window length: 91

```python
# Plot the impulse and step response of the filter
t = np.arange(0,0.2,ts/10)
n = 1
gA = diffA.evalKernelDer(t,n)
gB = diffB.evalKernelDer(t,n)
fig, ax = plt.subplots(nrows=1, ncols=2,sharex=True, figsize=(10, 5))
fig.suptitle("Impulse and step response of two algebraic differentiators")
ax[0].plot(t/diffA.get_T(),diffA.evalKernel(t),label=r"diff. A")
ax[0].plot(t/diffB.get_T(),diffB.evalKernel(t),label=r"diff. A")
ax[1].plot(t/diffA.get_T(),diffA.get_stepResponse(t),label=r"\int_{0}^{t}g_{\mathrm{A}}(\tau)\mathrm{d}\tau")
ax[1].plot(t/diffB.get_T(),diffB.get_stepResponse(t),label=r"\int_{0}^{t}g_{\mathrm{B}}(\tau)\mathrm{d}\tau")
ax[0].set_xlabel(r"$\frac{t}{T}$")
ax[1].set_xlabel(r"$\frac{t}{T}$")
ax[0].set_ylabel(r"impulse responses")
ax[1].set_ylabel(r"step responses")
ax[0].legend()
ax[0].grid()
ax[1].grid()
plt.grid()
plt.show()
```

    <IPython.core.display.Javascript object>

```python
# Plot the first derivative of the kernel

t = np.arange(-0.01,0.2,ts/10)
n = 1
gA = diffA.evalKernelDer(t,n)
gB = diffB.evalKernelDer(t,n)
fig, ax_g = plt.subplots(nrows=1, ncols=1,
                                       sharex=True, figsize=(10, 5))
fig.suptitle("First derivatives of the kernels of two algebraic differentiators")
ax_g.plot(t/diffA.get_T(),gA*diffA.get_T()**(1+n)/2**n,label="diff. A$")
ax_g.plot(t/diffB.get_T(),gB*diffA.get_T()**(1+n)/2**n,label="$diff. B")
ax_g.set_xlabel(r"$\frac{t}{T}$")
ax_g.set_ylabel(r"first derivative of the kernels")
ax_g.legend()
plt.grid()
plt.show()
```

    <IPython.core.display.Javascript object>

## Frequency-domain analysis
The fourier transform of an algebraic differentiator is given as
\begin{equation}
\mathcal{G}(\omega)=\sum_{i=0}^{N}\frac{\left(\alpha+\beta+2i+1\right)\mathrm{P}_i^{(\alpha,\beta)}}{\alpha+\beta+i+1}\sum_{k=0}^{i}(-1)^{i-k}\left(\begin{array}ii\\k\end{array}\right)\mathrm{M}_{i,k}^{(\alpha,\beta)}(-\iota\omega T)
\label{eq:FourierTranformation}
\end{equation}
where $\mathrm{M}_{i,k}^{(\alpha,\beta)}$ denotes the hypergeometric Kummer M-Function.

An approximation of the amplitude spectrum of the algebraic differentiator is
\begin{equation}
\left|\mathcal{G}(\omega)\right|\approx\begin{cases}
1,&\quad \left|{\omega}\right|\leq\omega_{\mathrm{c}},\\
\left|{\frac{\omega_{\mathrm{c}}}{\omega}}\right|^{\mu},&\quad \text{otherwise},
\end{cases}
\label{eq:lowPassFilterApp}
\end{equation}
with $\omega_{\mathrm{c}}$ the cutoff frequency of the algebraic differentiator and $\mu=1+\min\{\alpha,\beta\}$.

Lower and upper bounds can be computed for the amplitude spectrum of the algebraic differentiators. If $N=0$ and $\alpha=\beta$, the amplitude reaches $0$ and the lower bound is $0$.

```python
omega = np.linspace(1,800,4*10**3)
omegaH = np.linspace(1,800,4*10**3)

# Get phase and amplitude of Fourier transform of filter A
ampA,phaseA = diffA.get_ampAndPhaseFilter(omega)

# Get upper and lower bound for the amplitude of Fourier transform of filter A
uA, lA, mA = diffA.get_asymptotesAmpFilter(omegaH)

# Get phase and amplitude of fourier transform of filter B
ampB,phaseB = diffB.get_ampAndPhaseFilter(omega)

# Get upper and lower bound for the amplitude of Fourier transform of filter B
uB, lB, mB = diffB.get_asymptotesAmpFilter(omegaH)

# Plot results
## PLEASE NOTE: Python will give a warning in the conversion to dB for the differentiato A
## since the amplitude spectrum reaches zero!
fig, ax = plt.subplots(nrows=1, ncols=2,sharex=False, figsize=(10, 5))
fig.suptitle("Frequency analysis of two algebraic differentiators")
ax[0].plot(omega/diffA.get_cutoffFreq(),20*np.log10(ampA),label=r"$|\mathcal{G}_{\mathrm{A}}(\omega)|$")
ax[0].plot(omegaH/diffA.get_cutoffFreq(),20*np.log10(uA),label=r"$|\mathcal{G}_{\mathrm{A}}^+(\omega)|$")
ax[0].plot(omegaH/diffA.get_cutoffFreq(),20*np.log10(lA),label=r"$|\mathcal{G}_{\mathrm{A}}^-(\omega)|$")
ax[0].plot(omegaH/diffA.get_cutoffFreq(),20*np.log10(mA),label=r"$|\mathcal{G}_{\mathrm{A}}^{\Delta}(\omega)|$")
ax[0].set_ylim(top=1)
ax[0].set_xlabel(r"$\frac{\omega}{\omega_{\mathrm{c,A}}}$")
ax[0].set_ylabel(r"amplitudes in dB")
ax[0].legend()
ax[0].grid()

ax[1].plot(omega/diffB.get_cutoffFreq(),20*np.log10(ampB),label=r"$|\mathcal{G}_{\mathrm{B}}(\omega)|$")
ax[1].plot(omegaH/diffB.get_cutoffFreq(),20*np.log10(uB),label=r"$|\mathcal{G}_{\mathrm{B}}^+(\omega)|$")
ax[1].plot(omegaH/diffB.get_cutoffFreq(),20*np.log10(mB),label=r"$|\mathcal{G}_{\mathrm{B}}^{\Delta}(\omega)|$")
ax[1].set_xlabel(r"$\frac{\omega}{\omega_{\mathrm{c,B}}}$")
ax[1].set_ylim(top=1)
ax[1].legend()
plt.grid()
fig.show()
```

    <IPython.core.display.Javascript object>

    <ipython-input-4-09f842cd3212>:23: RuntimeWarning: divide by zero encountered in log10
      ax[0].plot(omegaH/diffA.get_cutoffFreq(),20*np.log10(lA),label=r"$|\mathcal{G}_{\mathrm{A}}^-(\omega)|$")

## Numerical differentiation of sample signals
### Simulated signal
The derivative of a signal $y:t\rightarrow y(t)$ without any disturbance is estimated: The first differentiator has a delay. The second differentiator is parametrized for a delay free approximation.

```python
import sympy as sp

#######################################
# Compute signals and its derivatives
#######################################
a = sp.symbols('a_0:3')
t = sp.symbols('t')
y = a[0]*sp.exp(-a[1]*t)*sp.sin(a[2]*t)

# Derivative to be estimated
dy = sp.diff(a[0]*sp.exp(-a[1]*t)*sp.sin(a[2]*t),t,1)
d2y = sp.diff(a[0]*sp.exp(-a[1]*t)*sp.sin(a[2]*t),t,2)
aeval = {'a_0':1,'a_1':0.1,'a_2':4}

# Evaluate signal and true derivative
teval = np.arange(0,20,ts)
for ai in a:
    y = y.subs({ai:aeval[repr(ai)]})
    dy = dy.subs({ai:aeval[repr(ai)]})
    d2y = d2y.subs({ai:aeval[repr(ai)]})

yeval = sp.lambdify(t, y, "numpy")
yeval = yeval(teval)
dyeval = sp.lambdify(t, dy, "numpy")
dyeval = dyeval(teval)
d2yeval = sp.lambdify(t, d2y, "numpy")
d2yeval = d2yeval(teval)

#######################################
# Parametrize diffB to be delay-free
#######################################
# Set the parameter \vartheta
diffB.set_theta(1,False)
# Print the characteristics of the differentiator
diffB.printParam()

#######################################
# Estimate derivatives
#######################################
# Filter the signal y
xAppA = diffA.estimateDer(0,yeval)
xAppB = diffB.estimateDer(0,yeval)

# Estimate its first derivative
dxAppA = diffA.estimateDer(1,yeval)
dxAppB = diffB.estimateDer(1,yeval)

# Estimate its 2nd derivative
d2xAppA = diffA.estimateDer(2,yeval)
d2xAppB = diffB.estimateDer(2,yeval)

#######################################
# Plot results
#######################################
fig, (fy,fdy,f2dy) = plt.subplots(nrows=1, ncols=3,sharex=True, figsize=(10, 5))
fig.subplots_adjust( wspace=0.5)
fy.plot(teval,yeval,label='true signal')
fy.plot(teval,xAppA,label='diff A')
fy.plot(teval,xAppB,label='diff B')
fy.set_xlabel(r'$t$')
fy.set_ylabel(r'signals')

fdy.plot(teval,dyeval,label='true signal')
fdy.plot(teval,dxAppA,label='diff A')
fdy.plot(teval,dxAppB,label='diff B')
fdy.set_xlabel(r'$t$')
fdy.set_ylabel(r'first der. of signals')

f2dy.plot(teval,d2yeval,label='true signal')
f2dy.plot(teval,d2xAppA,label='diff A')
f2dy.plot(teval,d2xAppB,label='diff B')
f2dy.set_xlabel(r'$t$')
f2dy.set_ylabel(r' second der. of signal')

plt.legend()

plt.show()
```

    The differentiator has the parameters:
    Alpha: 4.000000
    Beta: 4.000000
    Window length in s: 0.111000
    Polynomial degree: 1
    Estimation delay in s: 0.000000
    Cutoff Frequency: 101.490088
    Discrete window length: 111

    <IPython.core.display.Javascript object>

### Estimation of derivative of a measured signal
The measurements are loaded from .dat file. The signal is filtered and the first and second derivatives are estimated using an algebraic differentiator.

```python
from os.path import dirname, join as pjoin
import scipy.io as sio

# Load measurements
data = np.loadtxt('data90.dat')
tmeas = data[:,0]
ts = tmeas[2]-tmeas[1]
xmeas = data[:,2]

# Estimate derivatives
diffA = AlgebraicDifferentiator(N=0,alpha=4,beta=4,T=None,wc=25, ts=ts,display=True)
xAppA = diffA.estimateDer(0,xmeas)
dxAppA = diffA.estimateDer(1,xmeas)
d2xAppA = diffA.estimateDer(2,xmeas)

# Plot results
fig, (fy,fdy, fd2y) = plt.subplots(nrows=1, ncols=3,sharex=True, figsize=(10, 5))
fig.subplots_adjust( wspace=0.5)
fy.plot(tmeas,xmeas,label='$y(t)$')
fy.plot(tmeas,xAppA,label='$\hat y(t)$')
fy.set_xlabel(r'$t$ in seconds')
fy.set_ylabel(r'signals')
fy.legend()

fdy.plot(tmeas,dxAppA,label='$\dot\hat y(t)$')
fdy.set_xlabel(r'$t$ in seconds')
fdy.legend()

fd2y.plot(tmeas,d2xAppA,label='$\ddot\hat y(t)$')
fd2y.set_xlabel(r'$t$ in seconds')
plt.legend()

plt.show()
```

    The differentiator has the parameters:
    Alpha: 4.000000
    Beta: 4.000000
    Window length in s: 0.270000
    Polynomial degree: 0
    Estimation delay in s: 0.135000
    Cutoff Frequency: 25.383213
    Discrete window length: 54

    <IPython.core.display.Javascript object>

# References

[1] M. Mboup, C. Join and M. Fliess, ``_A revised look at numerical differentiation with an application to nonlinear feedback control_'', 15th Mediterranean Conf. on Control $\&$ Automation, June 2007.  [online](https://ieeexplore.ieee.org/document/4433728)

[2] Mboup M., Join C. and Fliess M., ``_Numerical differentiation with annihilators in noisy environment_'', Numerical Algorithms, vol. 50, number 4, pp. 439--467,  2009.  [online](https://link.springer.com/article/10.1007/s11075-008-9236-1)

[3] Liu D.-Y., Gibaru O. and Perruquetti W., ``_Differentiation by integration with Jacobi polynomials_'', Journal of Computational and Applied Mathematics, vol. 235, number 9, pp. 3015 - 3032,  2011.  [online](http://www.sciencedirect.com/science/article/pii/S0377042710006734)

[4] L. Kiltz, ``_Algebraische Ableitungsschätzer in Theorie und Anwendung_'',  2017.  [online](https://scidok.sulb.uni-saarland.de/handle/20.500.11880/26974)


Table of Contents

$$
\begin{align}
\max_{v,p,t,e,a} \quad & f^T v \\
\text{subject to} \quad & Sv = 0, \\
   & l \leq v \leq u, \\
   & \sum p \leq P, && \text{P: total protein mass} \\
   & C e = p,  && \text{C: protein complex stoichiometry}  \\
   & v_j \leq \sum_{i\in r(j)} k^\text{eff}_{ij} e_i, && \text{r(j): enzymes catalyzing reaction j} \\
   & t_i = p_i / w_i && \text{Scaling RNA (transcript, $t$)-to-protein by a factor $w$. Simplest case: uniform scaling.} \\
   & \log_2(t_i) = \log_2(t^0_i) + R_i a,  && \text{R: log-linear regulatory constraints, $R_i$: row $i$} \\
   & t \geq \delta
\end{align}
$$
where $p^0$ is a constant reference vector (log_2 transformed), and $\delta$ is a small number (e.g., 0.01 / P), and $w_i$ are RNA-to-protein scaling weights, $p$ is protein, $e$ is enzyme complex, $t$ is transcript, $a$ is regulatory component activity.

This problem has the general form:

$$
\begin{align}
\max_{x} \quad & f^T x \\
\text{subject to} \quad & l \leq Ax \leq u, \\
   & \log(Cx) + Dx = b
\end{align}
$$

$$
\begin{align}
\max_{x,y} \quad & f^T x \\
\text{subject to} \quad & l \leq Ax \leq u, \\
   & y + Dx = b \\
   & y_i = \log(x_j)
\end{align}
$$

Moving $y_i=\log(x_j)$ into objective with augmented Lagrangian should make the constraint be solved exactly.  Interior point also appropriate?

The log constraint comes from:

ICA-equivalent form:
$$
\begin{align}
\log_2(p) - \log_2(p^0) &= Ma
\end{align}
$$

$$
\begin{align}
\log_2(X) - \log_2(X^0) &= SA
\end{align}
$$

Note: $\log(x)$ is quasiconvex on $x > 0$.

Can use the augmented Lagrangian of MINOS.

```python

```

```python
import pandas as pd
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
```

```python
%matplotlib inline
```

ICA values

```python
df_X0 = pd.read_csv('../data/data_files/log_tpm.csv', index_col=0)
df_X = pd.read_csv('../data/data_files/log_tpm_norm.csv', index_col=0)
df_M = pd.read_csv('../data/data_files/M.csv', index_col=0)
df_A = pd.read_csv('../data/data_files/A.csv', index_col=0)
```

```python
X0 = df_X0.to_numpy()
X = df_X.to_numpy()
A = df_A.to_numpy()
M = df_M.to_numpy()
MA = np.dot(M,A)
```

```python
print('X not norm:', df_X0.min().min(), df_X0.max().max())
print('X norm:', df_X.min().min(), df_X.max().max())
```

    X not norm: 0.0 16.4019957054125
    X norm: -11.336737814951801 12.831169252022299

df_A.head()

```python
### X as fold change:
m = X0.shape[0]
xref = np.log2(df_X0.iloc[:,:2].mean(1)+1.).to_numpy()
xref = xref.reshape((m,1))
ncond = X0.shape[1]
Xref = np.outer(xref, np.ones((1,ncond)))
Xf = np.log2(X0+1) - Xref
```

```python
plt.hist((Xf - MA).flatten(), 50)
```


# Introducing Scikit-Learn

### Basics of the API

The steps of one Scikit-Learn estimator:

1. Choose a model and import it (The models are class).
2. Make an instance and choose the hyperparameters.
3. Arrange data into a features matrix and target vector.
4. Fit the model to your data by calling the ``fit()`` method of the model instance.
5. Apply the Model to new data:
   - For supervised learning, often we predict labels for unknown data using the ``predict()`` method.
   - For unsupervised learning, we often transform or infer properties of the data using the ``transform()`` or ``predict()`` method.

```python
class Estimator(object):
    def fit(self, X, y=None):
        """Fits estimator to data."""
        # set state of ``self``
        return self
```

```python
%matplotlib inline
import numpy as np
import pylab as plt
import pandas as pd
from sklearn import svm
from sklearn.metrics import classification_report,confusion_matrix,accuracy_score
```

```python
np.random.seed(0)
X = np.r_[np.random.randn(20, 2) - [2, 2], np.random.randn(20, 2) + [2, 2]]
Y = [0] * 20 + [1] * 20
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.coolwarm, s=25)
```

```python
fig, ax = plt.subplots()
clf2 = svm.LinearSVC(C=1).fit(X, Y)

# get the separating hyperplane
w = clf2.coef_[0]
a = -w[0] / w[1]
xx = np.linspace(-5, 5)
yy = a * xx - (clf2.intercept_[0]) / w[1]

# create a mesh to plot in
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx2, yy2 = np.meshgrid(np.arange(x_min, x_max, .2),
                     np.arange(y_min, y_max, .2))
Z = clf2.predict(np.c_[xx2.ravel(), yy2.ravel()])

Z = Z.reshape(xx2.shape)
ax.contourf(xx2, yy2, Z, cmap=plt.cm.coolwarm, alpha=0.3)
ax.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.coolwarm, s=25)
ax.plot(xx,yy)

ax.axis([x_min, x_max,y_min, y_max])
plt.show()
```

**Supervised learning**:

  - Linear models (Ridge, Lasso, Elastic Net, ...)
  - Support Vector Machines
  - Tree-based methods (Random Forests, Bagging, GBRT, ...)
  - Nearest neighbors
  - Neural networks (basics)

**Unsupervised learning**:

  - Clustering (KMeans, Ward, ...)
  - Outlier detection

## Accuracy and precision

1. precision:
The fraction of relevant instances among the retrieved instances,
1. recall:
The fraction of relevant instances that have been retrieved over the total amount of relevant instances.
1. F-score

* The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.

* The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.

* The F-beta score can be interpreted as a weighted harmonic mean of the precision and recall, where an F-beta score reaches its best value at 1 and worst score at 0. The F-beta score weights recall more than precision by a factor of beta. beta == 1.0 means recall and precision are equally important.

* The support is the number of occurrences of each class in y_true.

Take a look at [HERE](https://en.wikipedia.org/wiki/F1_score) or [HERE](https://en.wikipedia.org/wiki/Precision_and_recall).

```python
# Generate data
from sklearn.datasets import make_blobs,make_circles,make_moons
X, y = make_blobs(n_samples=1000, centers=2,
                  cluster_std=1.5,
                  center_box=(-4.0, 4.0))
# X, y = make_circles(n_samples=1000, noise=.05, factor=.5)
# X,y = make_moons(n_samples=1000, noise=.05)

plt.scatter(X[:,0],X[:,1],c=y,)

```

```python
print(X[:5])
```

    [[ 0.21046649 -2.71428506]
     [ 2.22501727 -1.27184575]
     [ 2.34093076 -0.91781464]
     [-2.43273456 -0.25884974]
     [-0.97239553 -1.44875417]]

```python
print(y[:5])
```

    [0 1 1 0 0]

```python
# X is a 2 dimensional array, with 1000 rows and 2 columns
print(X.shape)

# y is a vector of 1000 elements
print(y.shape)
```

    (1000, 2)
    (1000,)

```python
X_train, y_train = X[:700], y[:700]
X_test, y_test = X[700:], y[700:]
```

## K-Nearest Neighbours

```python
# K-Nearest Neighbours
from sklearn.neighbors import KNeighborsClassifier

Model = KNeighborsClassifier(n_neighbors=8)
Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)
```

```python
y_pred[:5],y_test[:5]
```

    (array([0, 0, 0, 1, 1]), array([1, 0, 0, 0, 1]))

```python
# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
# print(confusion_matrix(y_test, y_pred))
# Accuracy score

print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.88      0.89      0.89       154
               1       0.88      0.88      0.88       146

        accuracy                           0.88       300
       macro avg       0.88      0.88      0.88       300
    weighted avg       0.88      0.88      0.88       300

    accuracy is 0.8833333333333333

```python
# Compute (approximate) class probabilities
print(Model.predict_proba(X_test[:5]))
```

    [[0.875 0.125]
     [0.    1.   ]
     [1.    0.   ]
     [0.    1.   ]
     [0.    1.   ]]

##  Radius Neighbors Classifier

```python
from sklearn.neighbors import  RadiusNeighborsClassifier
Model=RadiusNeighborsClassifier(radius=8.0)
Model.fit(X_train,y_train)
y_pred=Model.predict(X_test)
#summary of the predictions made by the classifier
print(classification_report(y_test,y_pred))
# print(confusion_matrix(y_test,y_pred))
#Accouracy score
print('accuracy is ', accuracy_score(y_test,y_pred))
```

                  precision    recall  f1-score   support

               0       0.98      0.34      0.51       154
               1       0.59      0.99      0.74       146

        accuracy                           0.66       300
       macro avg       0.79      0.67      0.62       300
    weighted avg       0.79      0.66      0.62       300

    accuracy is  0.66

## Naive Bayes

\begin{align}\begin{aligned}P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)\\\Downarrow\\\hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i \mid y),\end{aligned}\end{align}

```python
# Naive Bayes
from sklearn.naive_bayes import GaussianNB
Model = GaussianNB()
Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)

# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
# print(confusion_matrix(y_test, y_pred))
# Accuracy score
print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.91      0.88      0.89       154
               1       0.87      0.90      0.89       146

        accuracy                           0.89       300
       macro avg       0.89      0.89      0.89       300
    weighted avg       0.89      0.89      0.89       300

    accuracy is 0.89

## SVM

```python
# Support Vector Machine
from sklearn.svm import SVC

Model = SVC(kernel='linear')
# Model = svm.LinearSVC(C=1)

Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)

# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
# print(confusion_matrix(y_test, y_pred))
# Accuracy score

print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.91      0.88      0.89       154
               1       0.88      0.91      0.89       146

        accuracy                           0.89       300
       macro avg       0.89      0.89      0.89       300
    weighted avg       0.89      0.89      0.89       300

    accuracy is 0.8933333333333333

```python
plt.scatter(X_test[:,0],X_test[:,1],c=y_pred,)
```

## Decision Tree

```python
# Decision Tree's
from sklearn.tree import DecisionTreeClassifier

Model = DecisionTreeClassifier()

Model.fit(X_train, y_train)

y_pred = Model.predict(X_test)

# Summary of the predictions made by the classifier
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
# Accuracy score
print('accuracy is',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.84      0.86      0.85       154
               1       0.85      0.83      0.84       146

        accuracy                           0.84       300
       macro avg       0.84      0.84      0.84       300
    weighted avg       0.84      0.84      0.84       300

    [[132  22]
     [ 25 121]]
    accuracy is 0.8433333333333334

## RandomForest

```python
from sklearn.ensemble import RandomForestClassifier
Model= RandomForestClassifier(max_depth=2)
Model.fit(X_train,y_train)
y_pred=Model.predict(X_test)
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_pred,y_test))
#Accuracy Score
print('accuracy is ',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.91      0.88      0.89       154
               1       0.87      0.90      0.89       146

        accuracy                           0.89       300
       macro avg       0.89      0.89      0.89       300
    weighted avg       0.89      0.89      0.89       300

    [[135  14]
     [ 19 132]]
    accuracy is  0.89

## Neural network

```python
from sklearn.neural_network import MLPClassifier
Model=MLPClassifier()
Model.fit(X_train,y_train)
y_pred=Model.predict(X_test)
# Summary of the predictions
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_test,y_pred))
#Accuracy Score
print('accuracy is ',accuracy_score(y_pred,y_test))
```

                  precision    recall  f1-score   support

               0       0.92      0.87      0.89       154
               1       0.87      0.92      0.89       146

        accuracy                           0.89       300
       macro avg       0.89      0.89      0.89       300
    weighted avg       0.89      0.89      0.89       300

    [[134  20]
     [ 12 134]]
    accuracy is  0.8933333333333333

```python

```

```python

```

```python

```

```python

```


```python
import numpy as np
import ipyvolume as ipv
import sympy as sy
from sympy.geometry import Plane as syPlane, Point3D as syPoint3D
import tqdm
```

```python
xyz = np.array((2 * np.random.random(1000) - 1, 2 * np.random.random(1000) - 1, 2 * np.random.random(1000) - 1))
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
ipv.show()
```

Now, similarly to the [req2.1 notebook](req2.1_filter_plane.ipynb), we will want to filter and plot here a thick cone, i.e. basically two  the same cones where their surfaces are separatated by some thickness.

The equation for a generically oriented and located regular (non-slanted) cone is given on [Wikipedia](https://en.wikipedia.org/wiki/Cone) as $F(u)=0$ where:

$$
F(u) = u \cdot d - |d| |u| \cos \theta
$$

where $u=(x,y,z)$, $d=(d,e,f)$ is the vector to which the cone axis is parallel and the aperture is $2\theta$. The vertex of this cone is at the origin, so to displace it to $(a,b,c)$ you need to subtract an additional vector $o = (a,b,c)$ from u:

$$
F(u) = (u - o) \cdot d - |d| |u - o| \cos \theta
$$

Alternatively, one can apparently use the equation

$$
F(u) = ((u - o) \cdot d)^2 - (d \cdot d) ((u - o) \cdot (u - o)) (\cos \theta)^2
$$

This is actually easier, because you can skip the square roots necessary for the norms.

```python
x, y, z, d, e, f, a, b, c, theta = sy.symbols('x, y, z, d, e, f, a, b, c, theta')
```

```python
cone_eqn = ((x-a)*d + (y-b)*e + (z-c)*f)**2 \
           - sy.cos(theta)**2 * (d * d + e * e + f * f) \
           * ((x-a) * (x-a) + (y-b) * (y-b) + (z-c) * (z-c))
```

To solve, we need to fill out the parameters we want. Let's try some simple things first.

```python
res = sy.solve(cone_eqn.subs(theta, sy.pi/4).subs(a, 0).subs(b, 0).subs(c, 0).subs(d, 0).subs(e, 1).subs(f, 0), y)
```

```python
value = res[0].subs(x, 0.1).subs(z, 0.1)
```

```python
float(value)
```

```python
res0 = res[0]
```

```python
cone_eqn.subs
```

Ok, that seems reasonable actually!

```python
def plot_cone(res, x_sy, z_sy):
    """
    Draw a cone.
    """
    # get box limits in two dimensions
    x_lim = ipv.pylab.gcf().xlim
    z_lim = ipv.pylab.gcf().zlim
    x_steps = np.linspace(*x_lim, 10)
    z_steps = np.linspace(*z_lim, 10)
    x, z = np.meshgrid(x_steps, z_steps)

    # find corresponding y coordinates
    y1, y2 = [], []
    for xi, zi in zip(x.flatten(), z.flatten()):
        y1.append(float(res[0].evalf(subs={x_sy: xi, z_sy: zi})))
        y2.append(float(res[1].evalf(subs={x_sy: xi, z_sy: zi})))

    # plot
    ipv.plot_surface(x, np.array(y1), z)
    ipv.plot_surface(x, np.array(y2), z)
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone(res, x, z)
ipv.show()
```

Nice! Could be better though.

- The origin point must be included.
- It looks a bit odd that the edges are cut off. It makes sense if you want to fill the whole box, but probably most people will prefer a circular ending. This we could add as an option.
- Just pick one of the two solutions, probably the one that is actually in the same direction as the $d$ vector.

Let's also try to make it a bit more general and try out some different examples to test other corner cases.

```python
def plot_cone_2(axis, origin, half_aperture):
    """
    Draw a cone.

    axis: three-component vector along which the cone axis lies
    origin: location of the apex of the cone
    half_aperture: the aperture of the cone is 2*half_aperture
    """
    x, y, z, d, e, f, a, b, c, theta = sy.symbols('x, y, z, d, e, f, a, b, c, theta')
    cone_eqn = ((x-a)*d + (y-b)*e + (z-c)*f)**2 \
               - sy.cos(theta)**2 * (d * d + e * e + f * f) \
               * ((x-a) * (x-a) + (y-b) * (y-b) + (z-c) * (z-c))

    res = sy.solve(cone_eqn.subs({d: axis[0], e: axis[1], f: axis[2],
                                  a: origin[0], b: origin[1], c: origin[2],
                                  theta: half_aperture}), y)

#     print(res)

    # get box limits in two dimensions
    x_lim = ipv.pylab.gcf().xlim
    z_lim = ipv.pylab.gcf().zlim
    x_steps = np.linspace(*x_lim, 20)
    z_steps = np.linspace(*z_lim, 20)
    x_array, z_array = np.meshgrid(x_steps, z_steps)

    y_lim = ipv.pylab.gcf().ylim

    # find corresponding y coordinates
    y1, y2 = [], []
    for xi, zi in zip(x_array.flatten(), z_array.flatten()):
        y1.append(float(res[0].evalf(subs={x: xi, z: zi})))
        y2.append(float(res[1].evalf(subs={x: xi, z: zi})))

    # plot
    ipv.plot_surface(x_array, np.array(y1), z_array)
    ipv.plot_surface(x_array, np.array(y2), z_array)

    ipv.ylim(*y_lim)
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_2((0, 1, 0), (0, 0, 0), sy.pi/4)
ipv.show()
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_2((0.25, 0.5, 0), (0, 0.5, 0), sy.pi/8)
ipv.show()
```

Arrgh, what's going on here?

Oh, wait, this must be what happens when the x and z parameters are, in fact, outside of where the cone is going to be.

Is there a more reliable way to find points x, y, z that are actually on the cone surface instead of just randomly (or rather grid-wise) probing x and z and hoping there's a y point there?

The trick is to find a parametric form, I think. Wikipedia has this form for a cone along the z-axis, so we could use that, but then we'd still have to displace it and rotate it.

Maybe we can find our own parametric form though.

One dimension that we would probably want is the same as the $u$ one of the Wikipedia parameterization, i.e. a coordinate that runs along the length of the axis up to its height $h$, i.e. $u \in [0,h]$. I think this means it should be that

$$
u^2 = x^2 + y^2 + z^2
$$

so that $u$ is the square-root of the sum of the three components. This indeed makes sense, because it has two solutions, a positive and a negative one, which technically the cone equations indeed must have. However, we only use the positive range.

The remaining two parameters $s$ and $t$ must describe the remaining two

```python
...
```

Ok... this is going to be too much work, and won't generalize nicely to other surfaces anyway. Let's try the easier approach of just rotating the simple parametric form. This will also automatically fix all three "niceness" issues mentioned above.

Wikipedia's parametric description for a cone along the z-axis:

$$
F(s,t,u) = \left(u \tan s \cos t, u \tan s \sin t, u \right)
$$

where $s,t,u$ range over $[0,\theta)$, $[0,2\pi)$, and $[0,h]$, respectively. Actually, Wolfram Mathworld has a simpler parameterization with only two parameters:

$$
x = \frac{h-u}{h} r \cos\theta	\\
y = \frac{h-u}{h} r \sin\theta	\\
z = u
$$

for $u$ in $[0,h]$ and $\theta$ in $[0,2\pi)$.

```python
def plot_cone_3(height, radius, N_steps=20):
    """
    Draw a cone.

    height: height along the z-axis
    radius: radius of the circle
    """
    h, r, u, theta = sy.symbols('h, r, u, theta')
    x_eqn = (h - u) / h * r * sy.cos(theta)
    y_eqn = (h - u) / h * r * sy.sin(theta)
    z_eqn = u

    # get box limits in two dimensions
#     x_lim = ipv.pylab.gcf().xlim
#     z_lim = ipv.pylab.gcf().zlim
#     x_steps = np.linspace(*x_lim, 20)
#     z_steps = np.linspace(*z_lim, 20)
#     x_array, z_array = np.meshgrid(x_steps, z_steps)
    u_steps = np.linspace(0, height, N_steps)
    theta_steps = np.linspace(0, 2 * np.pi, N_steps)
    u_array, theta_array = np.meshgrid(u_steps, theta_steps)

    y_lim = ipv.pylab.gcf().ylim

    # find corresponding y coordinates
    x, y, z = [], [], []
    for ui, thetai in zip(u_array.flatten(), theta_array.flatten()):
        x.append(float(x_eqn.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))
        y.append(float(y_eqn.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))
        z.append(float(z_eqn.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))

    # plot
    ipv.plot_surface(np.array(x).reshape(u_array.shape),
                     np.array(y).reshape(u_array.shape),
                     np.array(z).reshape(u_array.shape))

#     ipv.ylim(*y_lim)
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_3(0.5, 0.5)
ipv.show()
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_3(1, 0.5)
ipv.show()
```

```python
def plot_cone_4(height, radius, rot_x=2*np.pi, rot_y=2*np.pi, N_steps=20):
    """
    Draw a cone.

    height: height along the z-axis
    radius: radius of the circle
    rot_y: rotation angle about the y axis (radians)
    rot_z: rotation angle about the z axis (radians)
    N_steps: number of steps in the parametric range used for drawing
    """
    h, r, u, theta = sy.symbols('h, r, u, theta')
    x_eqn = (h - u) / h * r * sy.cos(theta)
    y_eqn = (h - u) / h * r * sy.sin(theta)
    z_eqn = u

    x_rot_x = x_eqn
    y_rot_x = y_eqn * sy.cos(rot_x) + z_eqn * sy.sin(rot_x)
    z_rot_x = - y_eqn * sy.sin(rot_x) + z_eqn * sy.cos(rot_x)

    x_rot_y = x_rot_x * sy.cos(rot_y) + z_rot_x * sy.sin(rot_y)
    y_rot_y = y_rot_x
    z_rot_y = - x_rot_x * sy.sin(rot_y) + z_rot_x * sy.cos(rot_y)

    # get box limits in two dimensions
#     x_lim = ipv.pylab.gcf().xlim
#     z_lim = ipv.pylab.gcf().zlim
#     x_steps = np.linspace(*x_lim, 20)
#     z_steps = np.linspace(*z_lim, 20)
#     x_array, z_array = np.meshgrid(x_steps, z_steps)
    u_steps = np.linspace(0, height, N_steps)
    theta_steps = np.linspace(0, 2 * np.pi, N_steps)
    u_array, theta_array = np.meshgrid(u_steps, theta_steps)

    y_lim = ipv.pylab.gcf().ylim

    # find corresponding y coordinates
    x, y, z = [], [], []
    for ui, thetai in zip(u_array.flatten(), theta_array.flatten()):
        x.append(float(x_rot_y.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))
        y.append(float(y_rot_y.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))
        z.append(float(z_rot_y.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))

    # plot
    ipv.plot_surface(np.array(x).reshape(u_array.shape),
                     np.array(y).reshape(u_array.shape),
                     np.array(z).reshape(u_array.shape))

#     ipv.ylim(*y_lim)
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_4(1, 0.5, rot_y=np.pi/4)#, rot_z=np.pi/2)
ipv.show()
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_4(1, 0.5, rot_x=np.pi/9)#, rot_z=np.pi/2)
ipv.show()
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_4(1, 0.5, rot_x=np.pi/9, rot_y=np.pi/4)#, rot_z=np.pi/2)
ipv.show()
```

Ok, this we can work with, it rotates as expected. Probably, we should also add a version with a direction vector, but this will do for now. And we need a version in which you specify two positions: the base center and the apex position.

Now just add a translation vector.

```python
def plot_cone_5(height, radius, rot_x=2*np.pi, rot_y=2*np.pi, base_pos=(0, 0, 0), N_steps=20, **kwargs):
    """
    Draw a cone.

    height: height along the z-axis
    radius: radius of the circle
    rot_y: rotation angle about the y axis (radians)
    rot_z: rotation angle about the z axis (radians)
    base_pos: translation of base of cone to this position, iterable of three numbers
    N_steps: number of steps in the parametric range used for drawing
    """
    h, r, u, theta = sy.symbols('h, r, u, theta')
    x_eqn = (h - u) / h * r * sy.cos(theta)
    y_eqn = (h - u) / h * r * sy.sin(theta)
    z_eqn = u

    x_rot_x = x_eqn
    y_rot_x = y_eqn * sy.cos(rot_x) - z_eqn * sy.sin(rot_x)
    z_rot_x = y_eqn * sy.sin(rot_x) + z_eqn * sy.cos(rot_x)

    x_rot_y = x_rot_x * sy.cos(rot_y) - z_rot_x * sy.sin(rot_y) + base_pos[0]
    y_rot_y = y_rot_x + base_pos[1]
    z_rot_y = x_rot_x * sy.sin(rot_y) + z_rot_x * sy.cos(rot_y) + base_pos[2]

    # get box limits in two dimensions
    u_steps = np.linspace(0, height, N_steps)
    theta_steps = np.linspace(0, 2 * np.pi, N_steps)
    u_array, theta_array = np.meshgrid(u_steps, theta_steps)

    y_lim = ipv.pylab.gcf().ylim

    # find corresponding y coordinates
    x, y, z = [], [], []
    for ui, thetai in zip(u_array.flatten(), theta_array.flatten()):
        x.append(float(x_rot_y.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))
        y.append(float(y_rot_y.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))
        z.append(float(z_rot_y.evalf(subs={h: height, r: radius, u: ui, theta: thetai})))

    # plot
    ipv.plot_surface(np.array(x).reshape(u_array.shape),
                     np.array(y).reshape(u_array.shape),
                     np.array(z).reshape(u_array.shape), **kwargs)
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_cone_5(0.5, 0.5, rot_x=np.pi/9, rot_y=np.pi/4, base_pos=(0, 0.5, 0))#, rot_z=np.pi/2)
ipv.show()
```

Ok, that's it, now a thick cone...

```python
plot_cone = plot_cone_5
```

```python
def cone_axis_from_rotation(rot_x, rot_y):
    # z-unit vector (0, 0, 1) rotated twice
    cone_axis = (0, -np.sin(rot_x), np.cos(rot_x))  # rotation around x-axis
    cone_axis = np.array((-np.sin(rot_y) * cone_axis[2],
                          cone_axis[1],
                          np.cos(rot_y) * cone_axis[2]))  # around y
    return cone_axis
```

```python
def thick_cone_base_positions(height, radius, thickness, rot_x, rot_y, base_pos):
    thickness = abs(thickness)
    base_distance = thickness / radius * height * np.sqrt(1 + radius**2 / height**2)  # trigonometry

    cone_axis = cone_axis_from_rotation(rot_x, rot_y)

    base_pos_1 = np.array(base_pos) - cone_axis * 0.5 * base_distance
    base_pos_2 = np.array(base_pos) + cone_axis * 0.5 * base_distance

    return base_pos_1, base_pos_2
```

```python
def plot_thick_cone(height, radius, thickness,
                    rot_x=2*np.pi, rot_y=2*np.pi, base_pos=(0, 0, 0),
                    **kwargs):
    """
    Plot two cones separated by a distance `thickness`.

    Parameters: same as plot_cone, plus `thickness`.
    """
    base_pos_1, base_pos_2 = thick_cone_base_positions(height, radius, thickness, rot_x, rot_y, base_pos)
    plot_cone(height, radius, rot_x=rot_x, rot_y=rot_y, base_pos=base_pos_1, **kwargs)
    kwargs.pop('color', None)
    plot_cone(height, radius, rot_x=rot_x, rot_y=rot_y, base_pos=base_pos_2, color='blue', **kwargs)
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_thick_cone(0.5, 0.5, 0.2, rot_x=np.pi/9, rot_y=np.pi/4, base_pos=(0, 0.5, 0))
ipv.show()
```

Nice!

# Filtering

This is going to be a bit more difficult. We need a closest distance of each point to both surfaces. How to do this?

Maybe we can try using Sympy to solve the system of equations describing the minimum distance of a point $P=(p_1, p_2, p_3)$ to a cone surface, like e.g. here: https://math.stackexchange.com/a/880971/258876

```python
h, r, u, theta, p1, p2, p3 = sy.symbols('h, r, u, theta, p1, p2, p3')
x_eqn = (h - u) / h * r * sy.cos(theta)
y_eqn = (h - u) / h * r * sy.sin(theta)
z_eqn = u

distance_eqn = (x_eqn - p1)**2 + (y_eqn - p2)**2 + (z_eqn - p3)**2

system = [sy.Derivative(distance_eqn, u),
          sy.Derivative(distance_eqn, theta)]

res = sy.solve(system, distance_eqn, subs={h: 0.5, r: 0.5, p1: 0, p2: 0, p3: 0})
```

```python
res
```

Not sure how to fix this.

Another possibility may be to first filter by looking whether the angle of the vector from apex to $P$ with the vector along the cone axis is larger than $90^\circ + \theta/2$ where $\theta$ is the opening angle.

```python
def cone_apex_position(height, rot_x=2*np.pi, rot_y=2*np.pi, base_pos=(0, 0, 0)):
    cone_axis = cone_axis_from_rotation(rot_x, rot_y)
    return np.array(base_pos) + cone_axis * height
```

```python
def cone_opening_angle(height, radius):
    """
    Twice the opening angle is the maximum angle between directrices
    """
    return np.arctan(radius / height)
```

```python
def angle_between_two_vectors(a, b):
    return np.arccos(np.sum(a * b) / np.sqrt(np.sum(a**2)) / np.sqrt(np.sum(b**2)))
```

```python
def point_distance_to_cone(P, height, radius,
                           rot_x=2*np.pi, rot_y=2*np.pi, base_pos=(0, 0, 0),
                           return_extra=False):
    """
    Check whether for a point P, the shortest path to the cone is
    perpendicular to the cone surface (and if so, return it). If
    not, it is either "above" the apex and the shortest path is simply
    the line straight to the apex, or it is "below" the base, and the
    shortest path is the shortest path to the directrix (the base
    circle).

    This function returns a second value depending on which of the
    three above cases is true for point P. If we're using the
    perpendicular, it is True, if we're above the apex it is False and
    if it is below the base, it is None.

    Extra values can be returned to be reused outside the function by
    setting return_extra to True.
    """
    cone_axis = cone_axis_from_rotation(rot_x, rot_y)
    apex_pos = cone_apex_position(height, rot_x=rot_x, rot_y=rot_y, base_pos=base_pos)
    point_apex_vec = np.array(P) - apex_pos
    point_apex_angle = np.pi - angle_between_two_vectors(cone_axis, point_apex_vec)
    opening_angle = cone_opening_angle(height, radius)

    # for the second conditional, we need the length of the component of the
    # difference vector between P and apex along the closest generatrix
    point_apex_generatrix_angle = point_apex_angle - opening_angle
    point_apex_distance = np.sqrt(np.sum(point_apex_vec**2))
    point_apex_generatrix_component = point_apex_distance * np.cos(point_apex_generatrix_angle)
    generatrix_length = np.sqrt(radius**2 + height**2)

    returnees = {}
    if return_extra:
        returnees['opening_angle'] = opening_angle
        returnees['point_apex_angle'] = point_apex_angle

    if point_apex_angle > opening_angle + np.pi / 2:
        # "above" the apex
        return point_apex_distance, False, returnees
    elif point_apex_generatrix_component > generatrix_length:
        # "below" the directrix
        # use cosine rule to find length of third side
        return np.sqrt(point_apex_distance**2 + generatrix_length**2
                       - 2 * point_apex_distance * generatrix_length
                       * np.cos(point_apex_generatrix_angle)), None, returnees
    else:
        # "perpendicular" to a generatrix
        return point_apex_distance * np.sin(point_apex_generatrix_angle), True, returnees
```

Ok, that should do as well, an all-in-one distance function, a bit complicated, but I see no way around that.

```python
def filter_points_cone(points_xyz, height, radius, thickness,
                       rot_x=2*np.pi, rot_y=2*np.pi, base_pos=(0, 0, 0), verbose=False):
    base_pos_1, base_pos_2 = thick_cone_base_positions(height, radius, thickness, rot_x, rot_y, base_pos)

    p_filtered = []
    for ix, p_i in tqdm.tqdm(enumerate(points_xyz.T)):
        d_cone1, flag_cone1, vals1 = point_distance_to_cone(p_i, height, radius,
                                                            rot_x=rot_x, rot_y=rot_y,
                                                            base_pos=base_pos_1, return_extra=True)
        d_cone2, flag_cone2, _ = point_distance_to_cone(p_i, height, radius,
                                                        rot_x=rot_x, rot_y=rot_y,
                                                        base_pos=base_pos_2, return_extra=True)
        if flag_cone2 is False or flag_cone1 is None:
            # it is definitely outside of the cones' range
            pass
            if verbose: print(f"case 1: {p_i} was ignored")
        elif flag_cone1 is False:
            # the first condition is logically enclosed in the second, but the
            # first is faster and already covers a large part of the cases/volume:
            if d_cone1 <= thickness or \
               d_cone1 <= thickness / np.cos(vals1['point_apex_angle'] - vals1['opening_angle'] - np.pi/2):
                p_filtered.append(p_i)
                if verbose: print(f"case 2: {p_i} was added")
            else:
                pass
                if verbose: print(f"case 3: {p_i} was ignored")
        elif d_cone1 <= thickness and d_cone2 <= thickness:
            p_filtered.append(p_i)
            if verbose: print(f"case 4: {p_i} was added")
    return p_filtered
```

```python
p_filtered = filter_points_cone(xyz, 0.5, 0.5, 0.2, rot_x=np.pi/9, rot_y=np.pi/4, base_pos=(0, 0.5, 0))
```

```python
len(p_filtered)
```

```python
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_thick_cone(0.5, 0.5, 0.2, rot_x=np.pi/9, rot_y=np.pi/4, base_pos=(0, 0.5, 0))
ipv.scatter(*np.array(p_filtered).T, marker='circle_2d', color='blue')
ipv.show()
```

First try: That makes no sense...

Second try (modified code, added `np.pi - ` in `point_distance_to_cone`): ok, better, at least the points are now in one of the cones, but there are still some weird things; false negatives (between the cones) and false positives (in the red cone).

Third try (modified code, was using opening angle as if it was twice what it is, so now without divisions by 2): awesome!

## Debugging

For debugging, I used some simple points like these:

```python
_xyz = np.array([[0, 0, 0.5], # inside blue, above red
                 [0, 0, 0.7], # above blue (i.e. above both), within "thickness" from blue
                 [0, 0, 1],   # far above both
                 [0, 0, 0.3], # inside red
                 [0, 0, 0.1], # inside red, "below" blue
                 [0, 0, -0.1], # below both
                ]).T

ipv.clear()
ipv.scatter(*_xyz, color="black")
p_filtered = filter_points_cone(_xyz, 0.5, 0.5, 0.2, verbose=True)
plot_thick_cone(0.5, 0.5, 0.2)
ipv.scatter(*np.array(p_filtered).T, color='green')
ipv.show()
```

~~Ok, so the ones above are correct and the one inside is correct, but the ones inside the red cone are wrong. Let's go through the wrong cases one by one.~~

~~Why is `[0, 0, 0.3]` added? It's in case 4, so `d_cone1 <= thickness and d_cone2 <= thickness` is True. What are the values?~~

Before, the ones in the red cone were also added, which is what I'm debugging below (but makes no sense anymore now, i.e. now it makes sense, because it's no longer wrong).

```python
import ectopylasm.geometry
```

```python
ectopylasm.geometry.filter_points_cone(_xyz, 0.5, 0.5, 0.2)
```

```python
filter_points_cone(_xyz, 0.5, 0.5, 0.2)
```

Arrgh

```python
p_filtered = ectopylasm.geometry.filter_points_cone(xyz, 0.5, 0.5, 0.2, rot_x=np.pi/9, rot_y=np.pi/4, base_pos=(0, 0.5, 0))
ipv.clear()
ipv.scatter(*xyz, marker='circle_2d')
plot_thick_cone(0.5, 0.5, 0.2, rot_x=np.pi/9, rot_y=np.pi/4, base_pos=(0, 0.5, 0))
ipv.scatter(*np.array(p_filtered).T, marker='circle_2d', color='blue')
ipv.show()
```

```python
height = 0.5
radius = 0.5
thickness = 0.2
rot_x = 0
rot_y = 0
base_pos = (0, 0, 0)

p_i = [0, 0, 0.3]

base_pos_1, base_pos_2 = thick_cone_base_positions(height, radius, thickness, rot_x, rot_y, base_pos)

d_cone1, flag_cone1, vals1 = point_distance_to_cone(p_i, height, radius,
                                                    rot_x=rot_x, rot_y=rot_y,
                                                    base_pos=base_pos_1, return_extra=True)
d_cone2, flag_cone2, _ = point_distance_to_cone(p_i, height, radius,
                                                rot_x=rot_x, rot_y=rot_y,
                                                base_pos=base_pos_2, return_extra=True)
```

```python
d_cone1, d_cone2
```

```python

```