#  The 2-Body Problem and Binary Stars

*Colin Leach, October 2018*

## Contents

- [The Gravitational 2-Body Problem](#2body)
- [Interactive plot of the 2-body problem](#2bodyint)
- [Binary Stars](#binary)
- [Binary simulation](#sim)
- [Run simulation](#runsim)
- [Plot simulation](#plotsim)
- [Eclipse?](#eclipse)
- [References](#refs)

## The Gravitational 2-Body Problem

The 1-body problem assumes a *fixed* central mass with a much lighter body in a Keplerian orbit around it. What if both bodies are free to move? In particular, what if they are of comparable mass, so that the joint center of mass (CoM) is in free space between the bodies?

Fortunately, by working in the CoM frame (i.e. the CoM is stationary at the origin) we can state some fairly simple principles about the resulting mutual orbit:

- The bodies are in elliptical orbits sharing a common focus at the CoM
- At any time, the line joining the two bodies must pass through this common focus
- The 2-body problem is *mathematically* (not physically) equivalent to a 1-body problem with a *reduced mass* $\mu$ in a Keplerian orbit around a stationary *total mass* $M$ at the CoM.

For bodies with masses $m_1$ and $m_2$, the total mass is: $$M = m_1 + m_2$$

The reduced mass is: $$\mu = \frac{m_1 m_2}{M}$$

From Kepler's Third Law, the semi-major axis of the binary system is
$$a = \sqrt[3]{\frac{P^2 G M}{4 \pi^2}}$$

The individual stars have semi-major axes about the center of mass:
$$a_1 = \frac{\mu}{m_1} a \qquad a_2 = \frac{\mu}{m_2} a \quad \Rightarrow \quad a_1 + a_2 = a$$

The 1-body problem has the usual solution: $$r = \frac{a(1-e^2)}{1 + e \cos \phi}$$
The individual position vectors are (note opposite signs):
$$\mathbf{r}_1 = -\frac{\mu}{m_1} \mathbf{r} \qquad \mathbf{r}_2 = \frac{\mu}{m_2} \mathbf{r}$$

The individual orbits are thus scaled-down versions of the 1-body orbit, with the same eccentricity $e$ and phase angle $\phi$.

## Interactive plot of the 2-body problem

I find this hard to visualize just from reading about it, so let's set up some interactive graphics. First the usual definitions:

```python
%matplotlib inline

import time

import numpy as np
import matplotlib.pyplot as plt

from IPython.display import Image, HTML, display, clear_output
from matplotlib.offsetbox import AnchoredText

from ipywidgets import interact, interactive, fixed, interact_manual, Layout, Output
import ipywidgets as w

from astropy import units as u
from astropy.constants import G, sigma_sb, L_sun
```

Next define a plotting function:

```python
def plot2body(mratio, e, phi=0):
    """
    Here we use arbitrary units, assuming P = 1, a = 1
    m_2 = 1, m_1 = mratio >= 1
    phi in degrees
    """

    m_2 = 1
    m_1 = mratio
    M = 1 + mratio
    mu = mratio/M
    a_1 = mu/m_1
    a_2 = mu/m_2

    # get some evenly-spaced phase angles
    phis = np.linspace(0, 2*np.pi, 100)
    # 1-body orbit, polar coords
    rs = (1-e**2)/(1 + e*np.cos(phis))
    # individual orbits
    r1s = -mu/m_1*rs
    r2s = mu/m_2*rs

    # now convert to cartesian coords
    xs = rs*np.cos(phis)
    ys = rs*np.sin(phis)
    x1s = r1s*np.cos(phis)
    y1s = r1s*np.sin(phis)
    x2s = r2s*np.cos(phis)
    y2s = r2s*np.sin(phis)

    # current positions
    phi_rad = phi*np.pi/180 # deg to rad conversion
    r = (1-e**2)/(1 + e*np.cos(phi_rad))
    r1 = -mu/m_1*r
    r2 = mu/m_2*r
    r_curr = np.array([r, r1, r2])
    x_curr = r_curr*np.cos(phi_rad)
    y_curr = r_curr*np.sin(phi_rad)

    # the plot
    figwidth = 10
    fig2body = plt.figure(figsize=(figwidth, figwidth))
    plt.plot(xs, ys, 'b-', label='1-body')
    plt.plot(x1s, y1s, 'r-', label='$m_1$')
    plt.plot(x2s, y2s, 'g-', label='$m_2$')
    plt.axis('equal')
    plt.plot(0,0,'y+',markersize=20)
    plt.plot(x_curr, y_curr, 'ko')
    plt.xlabel('x', fontsize=18)
    plt.ylabel('y', fontsize=18)
    plt.legend(fontsize=16)
    plt.title('Orbits with e = {:.2f}, mass ratio {:.1f}, phi {}'
              .format(e, mratio, phi), fontsize=20)
    plt.show()
```

Set up the slider controls for interaction with the plot. Continuous update probably needs to be turned off, so the plot will redraw once you release the slider (mouse button up).

```python
style = {'description_width': 'initial'} # to avoid the labels getting truncated
interact(plot2body,
            mratio = w.FloatSlider(description="mass ratio (>1)", style=style,
                                        layout=Layout(width='80%'),
                                        continuous_update=False, # otherwise flicker is horrible
                                        min=1.0, max=10,
                                        value=3.0),
            e = w.FloatSlider(description="Eccentricity", style=style,
                                        layout=Layout(width='80%'),
                                        continuous_update=False,
                                        min=0.0, max=0.99, step=0.01,
                                        value=0.3),
            phi = w.IntSlider(description="current angle (deg)", style=style,
                                        layout=Layout(width='80%'),
                                        continuous_update=False,
                                        min=0, max=359,
                                        value=30));

```

    interactive(children=(FloatSlider(value=3.0, continuous_update=False, description='mass ratio (>1)', layout=La…

Notice the limiting cases:
- if $m_1 \gg m_2$, $m_2$ approaches $\mu$ and $m_1$ hardly moves: we have a 1-body approximation
- if $e = 0$, all orbits are concentric circles

## Binary Stars

After considering the 2-body problem in the abstract, now let's focus on a common astronomical example. The aim is a simple binary star simulation, giving position, velocity and light curve information.

This part of the code borrows from the method of Carroll and Ostlie, Appendix K. I have simplified it somewhat, particularly by omitting proper motions of the CoM, but added more interactivity and visuals.

Obviously, this is far from being research-grade software. Simplifying assumptions include:
- the two stars are spherically symmetrical, no tidal distortions or centrifugal effects
- they do not collide
- surface fluxes are uniform and depend only on stellar radius

Features used by the simulation include:
- elliptical orbits
- arbitrary inclination
- arbitrary periastron angle
- a simple model of limb darkening

In addition to the 2-body motions, we will also want to simulate light curves. We know the temperatures $T_e$ for each star, so the luminosities come from the Stefan-Boltzmann equation:
$$ L = 4 \pi R^2 \sigma T_e^4 $$

All the input parameters and results will be stored in a single dictionary called `stars`. Putting this in the top scope is maybe a bit questionable, but it saves a lot of parameter passing.

```python
stars = {}
```

It is useful to create some reasonable initial values for the simulation to work with. These can be changed later, and we will set up interactive controls for this.

Starting parameters include two angles, both in degrees. Inclination `i` ranges from $0^{\circ}$ for a face-on view to $90^{\circ}$ for an edge-on view. Edge-on is good, it gives us eclipses and a lot more useful information.

Periastron is the point of closest approach to the center of mass. Orientation of periastron $\phi$ is the angle between periastron for star 1 and the positive $x$ axis.

```python
def setDefaults():
    # star #1
    m1_Msun = 1
    R1_Rsun = 1
    T1 = 5777  # that's our sun

    # star #2
    m2_Msun = 2.9
    R2_Rsun = 2.2
    T2 = 9800 # an A0 star

    # orbit
    P_day = 1500 # period (days)
    e = 0.5      # eccentricity
    i = 90       # inclination to the line of sight (deg)
    phi = 60     # orientation of periastron (deg)

    setInput(m1_Msun, R1_Rsun, T1, m2_Msun, R2_Rsun, T2, P_day, e, i, phi)
```

```python
def setStars():

    M = (stars['m1'] + stars['m2'])
    a = ((stars['P']**2 * G * M)/(4*np.pi**2))**(1/3)
    mu = (stars['m1'] * stars['m2'])/M
    a1 = (mu/stars['m1'])*a
    a2 = (mu/stars['m2'])*a

    if a < stars['R1'] + stars['R2']:
        print("Error: your two stars are in contact!")
        print()
        stars['Error'] = "Your two stars are in contact"
        return False
    else:
        stars['M'] = M
        stars['mu'] = mu
        stars['a'] = a
        stars['a1'] = a1
        stars['a2'] = a2

        # Calculate luminosities from the Stefan-Boltzmann equation
        stars['L1'] = 4*np.pi * stars['R1']**2 * sigma_sb * stars['T1']**4
        stars['L2'] = 4*np.pi * stars['R2']**2 * sigma_sb * stars['T2']**4
        stars['L'] = stars['L1'] + stars['L2']

```

Two coordinates systems are relevant. The stars orbit in the $(x,y)$ plane. This is tilted at an inclination angle $i$ to the $(y',z')$ plane, the plane of the sky. The observer is on the positive $x'$ axis.

The coordinate transformations are:
$$ \begin{align}
   x' &= z \cos i + x \sin i \\
   y' &= y \\
   z' &= z \sin i - x \cos i
   \end{align}$$

Some functions used in calculating flux are defined in a separate file:

```python
from StellarBinary_suppl import transformation
from StellarBinary_suppl import limbDarkening, F, fluxIntegral, eclipse
```

Next, define how to identify eclipses and calculate dips in the light curve.

This is probably the most confusing part of the code (as I discovered during debugging). It is also where we get the benefit of assuming both stars are perfectly spherical, as this makes the geometry of the problem much more symmetrical than in the real-world case and saves significant computation.

The left diagram shows the basic geometry, with the larger, redder star in front of and overlapping the smaller yellow star. There is a line of symmetry passing through both centers (background $bc$ and foreground $fc$). This line makes an angle $\theta'_0$ with the $y'$ axis.

The algorithm goes something like this:
- For the current position in the Keplerian orbit $(x,y)$, get the corresponding plane-of-sky coordinates $(x',y',z')$
- Decide which star is in front of which. The centers are $fc$ and $bc$ respecively.
- If the distance between star centers in these coordinates is less than the sum of radii, we have an eclipse
- Find the line of symmetry passing through both star centers. This is at angle
$$ \theta'_0 = \tan^{-1} \left( \frac{z'_{fc} - z'_{bc}}{y'_{fc} - y'_{bc}} \right) $$
- Find the maximum radius $r$ on the background star where there is overlap. As drawn, this is simply $R_b$, but for a small foreground star its disk could be entirely within the background disk.
- Integrate over arcs of radius $r'$ and thickness $dr'$, working inwards towards $bc$.
- Within each arc, integrate over asub-arc of width $d\theta'$ at angle $\theta'$, working anticlockwise from the symmetry line at $\theta'_0$. Stop when there is no more overlap or $\theta'$ has covered a full semicircle.
- For each area $dA = r' dr' d\theta'$ calculate the flux drop $dS = 2 \times F(dA)$, relative to non-overlapping disks. The factor of 2 covers the other side of the symmetry line, which we avoided calculating explicitly.
- Return $dS$ for theses star positions, along with the primed coordinates for both star centers.

## The simulation!

Define the actual simulation run. The mathematical bit is fairly concise code but there are a lot of lines just handling storage of the results.

Calculating motions in the orbital plane is similar to the KeplerOrbit notebook, except that here we make no assumptions about the relative masses of the two objects. Instead we calculate the orbit of a reduced mass $\mu$ in the center of mass reference frame, then translate from this to the positions of each star.

The angular momentum is
$$ L_{ang} = \mu \sqrt{G M a (1-e^2)} $$
and
$$ \frac{dA}{dt} = \frac{1}{2} r^2 \frac{d \theta}{dt} = \frac{1}{2} \frac{L}{\mu} \quad \Rightarrow \quad d \theta = \frac{dA}{dt} \frac{1}{r^2} dt $$

```python
from StellarBinary_suppl import runSim, saveData
```

Writing data to disk is not often useful, but saveData() is a function that can be called if necessary.

For no adequate scientic reason (just playing), let's estimate what color the stars would appear to us:

```python
from StellarBinary_suppl import lookup_temp
```

Define a graphing function that will be called later. This will output a $2 \times 3$ grid of different plots, so there is a lot of rather boring housekeeping code.

```python
from StellarBinary_suppl import plotOrbit
```

For each star we want mass `m` (units of $M_{\odot}$), radius `R` ($R_{\odot}$), temperature `T` ($K$).

For the orbit we want period `P` (days), eccentricity `e`, orbital inclination `i` (deg) and orientation of periastron `phi` (deg). The inclination needs to be fairly near $90^{\circ}$ if you want eclipses.

```python
def setInput(m1_Msun, R1_Rsun, T1, m2_Msun, R2_Rsun, T2, P_day, e, i, phi):

    # star #1
    m1_Msun = m1_Msun * u.M_sun
    R1_Rsun = R1_Rsun * u.R_sun
    T1 = T1 * u.K # that's our sun

    # star #2
    m2_Msun = m2_Msun * u.M_sun
    R2_Rsun = R2_Rsun * u.R_sun
    T2 = T2 * u.K # an A0 star

    # orbit
    P_day = P_day * u.day
    i = i * u.deg
    phi = phi * u.deg

    # empty the data dictionary for a fresh start
    stars.clear()

    # create variables in SI units for calculations and put everything in a single dictionary
    stars['m1'] = m1_Msun.to(u.kg)
    stars['R1'] = R1_Rsun.to(u.m)
    stars['T1'] = T1
    stars['m2'] = m2_Msun.to(u.kg)
    stars['R2'] = R2_Rsun.to(u.m)
    stars['T2'] = T2
    stars['P'] = P_day.to(u.s)
    stars['e'] = e
    stars['i'] = i
    stars['phi'] = phi

    setStars()
```

## Go...

With all the functions defined, now we can start running things.

```python
setDefaults()
```

There are a lot of inputs to this simulation and setting them interactively could get complicated.

Let's start by creating some useful widgets and storing them for later.

```python
from StellarBinary_suppl import widgetDict

wd = widgetDict(stars)
```

The `update()` function will be called whenever the interactive controls change. It updates the `stars` dictionary with latest starting values, but does not re-run the simulation or plot (by default).

```python
def update(m1_Msun, R1_Rsun, T1, m2_Msun, R2_Rsun, T2, P_day, e, i, phi):
    setInput(m1_Msun, R1_Rsun, T1, m2_Msun, R2_Rsun, T2, P_day, e, i, phi)

    display("a1: {:.2f}, L1: {:2g}, a2: {:.2f}, L2: {:2g}, mu: {:f}".
        format(stars['a1'].to(u.AU), stars['L1'], stars['a2'].to(u.AU), stars['L2'], stars['mu'].to(u.M_sun)))
#     return "a1: {:.2f}, L1: {:2g}, a2: {:.2f}, L2: {:2g}, mu: {:f}".format(
#         stars['a1'].to(u.AU), stars['L1'], stars['a2'].to(u.AU), stars['L2'], stars['mu'].to(u.M_sun))

#     removing the previous line of display would be good, but clear_output() also removed the control widgets
#         clear_output(wait=True)
#     I don't have a better approach right now

#     plotOrbit can take a few seconds per update
#     uncomment this line if you are patient or have a very fast machine
#     plotOrbit()

#     Otherwise, run plotOrbit() manually from the last cell in this notebook
```

As of mid-2018, Matplotlib `ipywidgets.interact()` does not support layouts other than a single vertical set of controls. This has been discussed by the developers, e.g. [issue 1731](https://github.com/jupyter-widgets/ipywidgets/issues/1731), and may be fixed in a future release. Meanwhile this `interact_hookup()` hack is adapted from [issue 910](https://github.com/jupyter-widgets/ipywidgets/issues/910).

Displaying an update of calculated values (distances and luminosities) is still ugly. Any suggestions for fixes?

This is the default `interact()` setup. Maybe something to go back to if the more complex hack breaks?

```python
# # style = {'description_width': 'initial'} # to avoid the labels getting truncated
# interact(update, m1_Msun = wd['m1_Msun'], R1_Rsun = wd['R1_Rsun'], T1 = wd['T1'],
#                 m2_Msun = wd['m2_Msun'], R2_Rsun = wd['R2_Rsun'], T2 = wd['T2'],
#                 P_day = wd['P_day'], e = wd['e'], i = wd['i'], phi = wd['phi'],
#                 vcmxp = wd['vcmxp'], vcmyp = wd['vcmyp'], vcmzp = wd['vcmzp'],
#                 );
```

The control layout hack...

```python
def interact_hookup(f, controls):
    def observer(change):
        kwargs = {k:v.value for k,v in controls.items()}
        f(**kwargs)
    for k,w in controls.items():
        w.observe(observer, 'value')
```

```python
UI = w.VBox([
        w.VBox([
                w.HBox([wd['m1_Msun'], wd['R1_Rsun'], wd['T1'],], padding='100pt', margin='100pt', width='100%'),
                w.HBox([wd['m2_Msun'], wd['R2_Rsun'], wd['T2'],], width='100%'),
                w.HBox([wd['P_day'], wd['e'],], width='100%'),
                wd['i'],
                wd['phi'],
            ]),
    ])

interact_hookup(update, wd)
display(UI)
```

    VBox(children=(VBox(children=(HBox(children=(FloatText(value=1.0, description='star 1 mass ($M_{\\odot}$)', la…

## Now run the simulation and plot it

Doing this automatically from the interactive controls is a bit tedious, even on a reasonably fast machine. Just run the cell below and it will use current values for the various parameters.

Note that the "plane of sky" plot can look odd. It includes proper motions, so the stars may not return to their starting positions. Also, keep an eye on the axis scaling: the $x$ and $y$ axes may or may not be equal, to avoid everything disappearing into a horizontal line for an edge-on orbit.

Both orbit plots show the (almost) first position of the stars, $t=0$, with some indication of their relative sizes and colors.

The blue cross is the center of mass at $t=0$. If it's not visible, it's probably covered by the larger star.

_UI glitch:_ sometimes the orbit plots will collapse into a scrollable box, which is confusing. Clicking on the blank area to the left of the plots (below "In []") should fix this.

```python
plotOrbit(stars)
```

## Find the eclipse (if any)

You can see from the above light curve if there are dips, but the code needs to do this more precisely.

The `argrelextrema()` function seems to be a convenient way to find local maxima. For real data you would want to apply smoothing first, but our simulated data is already artificially clean.

Note that this function always returns a tuple with $N$ entries, for an $N$-dimensional data set. We have a 1-D array so take the first (and only) element from the tuple.

```python
from scipy.signal import argrelextrema

def star_in_front(n):
    " compare coords along line of sight and return # of front star"
    if stars['xp_diff'][b] > 0:
        return 1
    else:
        return 2

blips = argrelextrema(stars['dS'], np.greater)[0]

blipcount = len(blips)
print("There are {} eclipses during the orbit\n".format(blipcount))
if blipcount > 0:
    print("{0:>5} {1:>15} {2:>9} {3:>18}".format("t/P","elapsed time","dS","Star in front"))
    for b in blips:
        print("{0:#6.3g} {1:#10.4g} {2:#11.2g} {3:10n}".
              format(stars['t_P'][b], (stars['t_P'][b]*stars['P']).to(u.year), stars['dS'][b], star_in_front(b)))
else:
    print("If you want to play with eclipse data, try running a simulation with i close to 90 deg")
```

    There are 2 eclipses during the orbit

      t/P    elapsed time        dS      Star in front
     0.170     0.6982 yr     6.0e+24 W          2
     0.945      3.881 yr     5.5e+23 W          1

What would each eclipse look like from our line of sight, if we had an impossibly perfect telescope? We have enough data to draw this (very crudely) with SVG.

```python
from svgscene import SVGScene # expect a .py file in same directory

if blipcount > 0:
    scene = SVGScene()

    # TODO - making everything automatically fit nicely needs more effort!
    #        meanwhile, each sim may need these values tweaked
    x = 200
    y = 200
    scale_factor = 40

    for b in blips:
        front = star_in_front(b)
        if front == 1:
            R_f = stars['R1'].to(u.R_sun).value * scale_factor
            color_f = lookup_temp(stars['T1'].value)
            yp_f = stars['y1p'][b].to(u.R_sun).value * scale_factor
            zp_f = stars['z1p'][b].to(u.R_sun).value * scale_factor
            R_b = stars['R2'].to(u.R_sun).value * scale_factor
            color_b = lookup_temp(stars['T2'].value)
            yp_b = stars['y2p'][b].to(u.R_sun).value * scale_factor
            zp_b = stars['z2p'][b].to(u.R_sun).value * scale_factor
        else:
            R_f = stars['R2'].to(u.R_sun).value * scale_factor
            color_f = lookup_temp(stars['T2'].value)
            yp_f = stars['y2p'][b].to(u.R_sun).value * scale_factor
            zp_f = stars['z2p'][b].to(u.R_sun).value * scale_factor
            R_b = stars['R1'].to(u.R_sun).value * scale_factor
            color_b = lookup_temp(stars['T1'].value)
            yp_b = stars['y1p'][b].to(u.R_sun).value * scale_factor
            zp_b = stars['z1p'][b].to(u.R_sun).value * scale_factor

        scene.text((x-70, 25), "Star {} in front".format(front))
        scene.circle((x+yp_b, y), R_b, color_b)
        scene.circle((x+yp_f, y+zp_f-zp_b), R_f, color_f)

        # move right for showing the next eclipse
        x = x + 3*(R_f + R_b)

    display(scene)
```

# References

```"An Introduction to Modern Astrophysics", Appendix K
Bradley W. Carroll and Dale A. Ostlie
Second Edition, Addison Wesley, Copyright 2007.```


```python

```

```python
text = """Under the hood, these functions simply create a pattern object for you and
call the appropriate method on it. They also store the compiled object in a cache, so future calls using the same RE won’t need to parse the pattern again and again."""
```

# Count Lines

```python
def countLines(text):
    n = text.split("\n")
    return len(n)
```

```python
countLines(text)
```

    2

# Count Sentense

```python
import re
```

```python
def countSentense(text):
    l = re.compile("[.]\s[A-Z]").split(text)
    return len(l)
```

```python
countSentense(text)
```

    2

# Count Words

```python
def countWords(text):
    l = text.split(" ")
    s = set(l)
    return len(l),len(s)
```

```python
countWords(text)
```

    (44, 36)

# Frequncy of word The

```python
def countGivenWord(text,word):
    #return text.lower().count(word.lower())
    c = list(1 for match in re.finditer(word.lower(),text.lower()))
    return len(c)
```

```python
countGivenWord(text,"The")
```

    7

# Scalar

```python
import numpy as np
import math
import sympy

```

```python
w = np.array([2, 3, 4])
v1 = np.array([5, 6, 8])
v2 = np.array([3, 3, 4])

r = (np.column_stack((v1,v2,w)))

r[1] = np.divide(r[1],3)

r[2] = np.divide(r[2],3)

r[2] = np.subtract(r[1],r[2])

print(r)
l1 = [[*r[0,0:2]], [*r[1,0:2]]]
l2= [r[0,2],r[1,2]]
print("--------------")
print(l1)
print("--------------")
print(l2)
l2 = list(filter(lambda x:x != 0, l2))
ans = np.linalg.solve(l1,l2)
a = np.ceil(ans[0])
b=np.floor(ans[1])

print("Value of K1,K2 is {0} and {1}".format(a,b))
```

    [[5 3 2]
     [2 1 1]
     [0 0 0]]
    --------------
    [[5, 3], [2, 1]]
    --------------
    [2, 1]
    Value of K1,K2 is 1.0 and -1.0

```python
w = np.array([2, 3, 4])
v1 = np.array([5, 6, 8])
v2 = np.array([3, 3, 4])

a=np.column_stack([v1,v2])
b=np.linalg.lstsq(a,w)

c,d=b[0]

```


# Valorización con Colateral

**Ametrano, Ferdinando M. and Bianchetti, Marco**, Everything You Always Wanted to Know About Multiple Interest Rate Curve Bootstrapping but Were Afraid to Ask (April 2, 2013). Available at SSRN: https://ssrn.com/abstract=2219548 or http://dx.doi.org/10.2139/ssrn.2219548

Se puede suponer que las cantidades captadas o colocadas por una contraparte que participa en el mercado OTC están asociadas a una cuenta de financiamiento (*funding account*) con el valor $B_{\alpha}\left(t\right)$ en $t$. El índice $\alpha$ denota la fuente específica de financiamiento.

Se asume la siguiente dinámica:

$$
\begin{equation}
dB_{\alpha}\left(t\right)=r_{\alpha}\left(t\right)B_{\alpha}\left(t\right)dt
\end{equation}
$$

$$
\begin{equation}
B_{\alpha}\left(0\right)=1
\end{equation}
$$

$$
\begin{equation}
B_{\alpha}\left(t\right)=\exp\int_{0}^{t}r_{\alpha}\left(u\right)du
\end{equation}
$$

Hemos ya identificado dos fuentes de financiamiento para el mercado OTC:

- La fuente de financiamiento general (o de tesorería), denotada con $B_f$ asociada a la operativa usual en mercados monetarios y de bonos. Podemos pensar que, en este caso, $r_f=Libor+spread,\space SOFR+spread$

- La cuenta de colateral, $B_c$, asociada a un CSA para la cual típicamente $r_c=ON$ (overnight).

## Definición

Un CSA perfecto es un CSA con las siguientes características:

- margen inicial igual a 0
- totalmente simétrico (las condiciones aplican a ambas contrapartes por igual)
- colateral sólo en *cash*
- threshold igual a 0
- MTA igual a 0
- periodicidad de cálculo continua
- tasa de colateral $r_c\left(t\right)$
- traspaso instantáneo de colateral
- sin reinversión de colateral

Como consecuencia se tiene que:

$$
\begin{equation}
B_c\left(t\right)=\Pi\left(t\right)
\end{equation}
$$

donde $\Pi\left(t\right)$ es el valor de la cartera de derivados asociada al CSA.

## Teorema (valorización con colateral)

Sea $\Pi$ un instrumento derivado que vence en $T$ escrito sobre el activo $X$ que obedece la siguiente ecuación diferencial estocástica:

$$
\begin{equation}
dX\left(t\right)=\mu^P\left(t,X\right)X\left(t\right)dt+\sigma\left(t,X\right)X\left(t\right)dW^P\left(t\right),
\end{equation}
$$

$$
\begin{equation}
X\left(0\right)=X_0
\end{equation}
$$

donde $t\in\mathbb{R}_+$, $X\in\mathbb{R}$, $\mu^P\in\mathbb{R}_+\times\mathbb{R}\rightarrow\mathbb{R}$, $\sigma\in\mathbb{R}_+\times\mathbb{R}\rightarrow\mathbb{R}_+$, $W^P\in\mathbb{R}$ es un movimiento browniano de dimension 1 sobre $\left(\Omega,\mathscr{F},P\right)$ y $P$ es la medida histórica (u objetiva). Asumiendo colateral perfecto, el precio del derivado $\Pi\left(t\right)$ en $t\lt T$ satisface:

$$
\begin{equation}
\overset{\wedge}{\mathcal{D}}_{rf}\Pi\left(t,X\right)=r_c\left(t\right)\Pi\left(t,X\right)
\end{equation}
$$

$$
\begin{equation}
\overset{\wedge}{\mathcal{D}}_{rf}=\frac{\partial}{\partial t}+r_f\left(t\right)X\left(t\right)\frac{\partial}{\partial X}+\frac{1}{2}\sigma^2\left(t\right)X^2\left(t\right)\frac{\partial^2}{\partial X^2}
\end{equation}
$$

y está dado también por:

$$
\begin{equation}
\Pi\left(t,X\right)=\mathbb{E}_t^{Q_f}\left[D_c\left(t,T\right)\Pi\left(T,X\right)\right]
\end{equation}
$$

$$
\begin{equation}
D_c\left(t,T\right)=\exp\left[-\int_t^Tr_c\left(u\right)du\right]
\end{equation}
$$

donde $Q_f$ es la medida de probabilidad asociada a la cuenta de financiamiento $B_f$ de modo que:

$$
\begin{equation}
dX\left(t\right)=r_f\left(t\right)X\left(t\right)dt+\sigma\left(t,X\right)X\left(t\right)dW^{Q_f}\left(t\right),
\end{equation}
$$

### Corolario (en la medida forward)

Vale la expresión:

$$
\begin{equation}
\Pi\left(t,X\right)=P_c\left(t,T\right)\mathbb{E}_{t}^{Q_f^T}\left[\Pi\left(T,X\right)\right]
\end{equation}
$$

$$
\begin{equation}
P_c\left(t,T\right)=\mathbb{E}^{Q_f}\left[D_c\left(t,T\right)\right]
\end{equation}
$$

donde $Q_f^T$ es la probabilidad asociada a $P_c\left(t,T\right)$ (la medida $T$-forward).


```python
%load_ext lab_black
# Import important tools
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
from scipy.io import loadmat, savemat
from mpl_toolkits.mplot3d import Axes3D
from scipy.optimize import curve_fit, minimize, least_squares
from scipy.integrate import trapz
from scipy.stats import norm, kurtosis
from matplotlib.ticker import ScalarFormatter
from matplotlib import rc
```

```python

```

```python
mpl.rcParams["xtick.direction"] = "in"
mpl.rcParams["ytick.direction"] = "in"
mpl.rcParams["lines.markeredgecolor"] = "k"
mpl.rcParams["lines.markeredgewidth"] = 0.1
mpl.rcParams["figure.dpi"] = 130
rc('font', family='serif')
rc('text', usetex=True)
rc('xtick', labelsize='x-small')
rc('ytick', labelsize='x-small')
def cm2inch(value):
    return value/2.54
```

We load the data

```python
raw_data = loadmat('fit_result_dur_27052020_n_r_fix_0p0513_wav_532_r_1p516_n_1.597.mat')["data"][:,0:3]
r = 1.516*1e-6
n_part =  1.597
fps = 60
time = np.arange(0,np.shape(raw_data)[0])/fps
dataset={}
dataset["r"] = r
dataset["n"] = n_part
dataset["fps"] = fps
dataset["time"] = time

```

# Data exploration

```python
# We put everything in microns
raw_data_m = raw_data
raw_data_m[:,0:3] = raw_data_m[:,0:3] * 0.0513
plt.plot(time/fps, raw_data_m[:,2])
x = raw_data_m[:,0]
y = raw_data_m[:,1]
z = raw_data_m[:,2]

plt.xlabel("$t$ (minutes)")
plt.ylabel("$z$ ($\mathrm{\mu m}$)")
plt.show()
```

```python
plt.plot(time/fps, raw_data_m[:,0], label="x")
plt.plot(time/fps, raw_data_m[:,1], label="y")
plt.xlabel("$t$ (minutes)")
plt.ylabel("$x$ ($\mathrm{\mu m}$)")
plt.show()
```

# MSD

We compute the MSD using the formula:

\begin{equation}
	\langle\Delta r_i(t)^2 \rangle_t = \langle[r_i(t+\Delta t) - r_i(t)]^2\rangle_t\ .
	\label{MSDdef}
\end{equation}

```python
def MSD(x, t):
    MSD = np.zeros(len(t))
    for n, i in enumerate(t):
        MSD[n] = np.nanmean((x[0:-i] - x[i:]) ** 2)
    return MSD

```

```python
t = np.array(
    [
        *np.arange(1, 10, 1),*np.arange(10, 100, 10),
        *np.arange(100, 1000, 100),
        *np.arange(1000, 40000, 1000),
    ]
)
MSD_x = MSD(x * 1e-6, t)  # m² conversion
MSD_y = MSD(y * 1e-6, t)
MSD_z = MSD(z * 1e-6, t)

plt.loglog(time[t], MSD_x, "o", label="MSDx")
plt.plot(time[t], MSD_y, "o", label="MSDy")
plt.plot(time[t], MSD_z, "o", label="MSDz")
plt.ylabel("MSD ($\mathrm{m^2}$)")
plt.xlabel("$\Delta t$ (s)")

plt.legend()

dataset["MSD_x_tot"] = MSD_x
dataset["MSD_y_tot"] = MSD_y
dataset["MSD_z_tot"] = MSD_z
dataset["MSD_time_tot"] = time[t]

```

We fit the short time MSD with and average diffusion coefficient such as:

\begin{equation}
	\langle\Delta r_i(t)^2 \rangle_t = 2 \langle D_i \rangle \Delta t\ ,
	\label{averagediff}
\end{equation}

```python
Do = 4e-21/(6 * np.pi * 0.001 * r)
f = lambda x,a,noiselevel : 2 * Do * a * x + (noiselevel * 1e-9) ** 2
popt_1 , pcov_1 = curve_fit(f,time[t[0:5]],MSD_x[0:5], p0 = [1, 30])
popt_2 , pcov_1 = curve_fit(f,time[t[0:5]],MSD_y[0:5], p0 = [1, 30])
popt_3 , pcov_1 = curve_fit(f,time[t[0:5]],MSD_z[0:5], p0 = [1, 30])

dataset["x_MSD_fit"] = time[t[0:5]]

dataset["MSD_x"] = MSD_x[0:5]
dataset["MSD_y"] = MSD_y[0:5]
dataset["MSD_z"] = MSD_z[0:5]
```

```python
print("We measure a reduced mean diffusion coefficient of {:.3f} for the perpendicular motion and of {:.3f} for the parallel motion".format((popt_1[0]+popt_2[0])/2, popt_3[0]))
```

    We measure a reduced mean diffusion coefficient of 0.522 for the perpendicular motion and of 0.243 for the parallel motion

# Displacement distributions

## $\Delta x$ distributions

```python
def pdf(data, bins = 10, density = True):
    """
    function to automatize the computations of experimental probability density functions.
    """

    pdf, bins_edge = np.histogram(data, bins = bins, density = density)
    bins_center = (bins_edge[0:-1] + bins_edge[1:]) / 2

    return pdf, bins_center
```

```python
I = [2, 5 , 10, 50 ,100,500, 1000,2000]

for i in I:

    Dezs = x[0:-i] - x[i:]
    hist, bins_center = pdf(Dezs, bins = 50)

    plt.plot(bins_center, hist, label = " t = {:.2f} s".format(time[i]))

plt.legend()
plt.ylabel("$P(\Delta x)$ [a.u.]")
plt.xlabel("$\Delta x$ [$\mathrm{\mu m}$]")
```

If we now normalize by the standard deviation

```python
def gauss_function(x, a, x0, sigma):
    return a*np.exp(-(x-x0)**2/(2*sigma**2))
```

```python

for n,i in enumerate(I):

    Dezs = (x[0:-i] - x[i:])
    Dezs = Dezs / np.sqrt(2 * Do * time[i])
    hist, bins_center = pdf(Dezs, bins = 30)

    #if i == I[0]:
    #    popt, pcov = curve_fit(gauss_function, bins_center/np.max(bins_center), hist, p0 = [1, np.mean(hist), np.std(hist)])
    #    plt.plot(bins_center/np.max(bins_center), gauss_function(bins_center, *popt), label = "fit at t = {:.2f} s".format(time[i]))
    #    plt.plot(bins_center/np.max(bins_center), hist, "x",label = " t = {:.2f} s".format(time[i]),color = "tab:blue")
    #    continue

    plt.plot(bins_center/np.max(bins_center), hist, ".",label = " $Delta$t = {:.2f} s".format(time[i]))

plt.ylabel("$P(\Delta x)$ [a.u.]")
plt.xlabel("normalized $\Delta x$ [a.u.]")
```

```python
(3.5e-22)**(1/3)
```

    7.047298732064899e-08

We can see a clear change but we would need to average on different trajectectories to have consitant results.

## $\Delta z$ distributions

```python
I = [2, 5 , 10, 50 ,100,500, 1000, 2000, 5000, 10000]

for i in I:

    Dezs = z[0:-i] - z[i:]
    hist, bins_center = pdf(Dezs[~np.isnan(Dezs)], bins = 50)

    plt.plot(bins_center, hist, label = " t = {:.2f} s".format(time[i]))

plt.legend()
plt.ylabel("$P(\Delta x)$ [a.u.]")
plt.xlabel("$\Delta x$ [$\mathrm{\mu m}$]")

```

```python
for i in I:

    Dezs = (z[0:-i] - z[i:])
    Dezs = Dezs / np.nanstd(Dezs)
    hist, bins_center = pdf(Dezs[~np.isnan(Dezs)], bins = 30)

    plt.plot(bins_center, hist, label = " t = {:.2f} s".format(time[i]))

plt.legend()
plt.ylabel("$P(\Delta x)$ [a.u.]")
plt.xlabel("$\Delta x$ [$\mathrm{\mu m}$]")
```

### Short time distributions

```python
I = [1,2,5,6,9,10]

for i in I:

    Dezs = (z[0:-i] - z[i:])
    Dezs = Dezs / np.std(Dezs)
    hist, bins_center = pdf(Dezs[~np.isnan(Dezs)], bins = 100)

    if i == I[0]:
        popt, pcov = curve_fit(gauss_function, bins_center, hist, p0 = [1, np.mean(hist), np.std(hist)])
        plt.plot(bins_center, gauss_function(bins_center, *popt))
        plt.plot(bins_center, hist, ".",label = " t = {:.2f} s".format(time[i]),color = "tab:blue")
        continue
    plt.semilogy(bins_center, hist, "." ,label = " t = {:.2f} s".format(time[i]))

plt.legend()
plt.ylabel("$P(\Delta x)$ [a.u.]")
plt.xlabel("$\Delta x$ [$\mathrm{\mu m}$]")
axes = plt.gca()
axes.set_ylim([1e-5,1])
```

```python
popt
```

    array([0.44423242, 0.00136534, 0.86463018])

The non Gaussianity is due to the fact that the diffusion coefficient vary as a funtion of the height, thus it vary during the diffusion (diffusing diffusivity) process knowing that one can write :
\begin{equation} P(\Delta z , \Delta t) = \int_0 ^\infty \mathrm{d}DP(D) \frac{1}{\sqrt{4 \pi D \Delta t}} \exp\left[ \frac{-\Delta z ^2}{4D\Delta t} \right]~. \end{equation}

### Long time distributions

```python
I = [2000, 5000, 10000]

color_long_time = ["tab:gray","tab:olive","tab:cyan"]
for n,i in enumerate(I):

    Dezs = z[0:-i] - z[i:]
    hist, bins_center = pdf(Dezs[~np.isnan(Dezs)],  bins = 10)

    plt.semilogy(bins_center, hist, "o",label = " t = {:.2f} s".format(time[i]), color = color_long_time[n])

plt.legend()

plt.ylabel("$P(\Delta x)$ [a.u.]")
plt.xlabel("$\Delta x$ [$\mathrm{\mu m}$]")
```

Indeed at long time it becomes exponential and it's no longer dependent on $\Delta t$

At very long time intervals $\Delta t$ each position measurment can be seen as random measurment on the le Boltzman distribution. Thus, one can write the probability distribution as : \begin{equation} P(\Delta z) = \int ^ \infty _{- \infty} dz P_B (z)P_B (z + \Delta z), \end{equation}

with :
\begin{equation} P_B(z) = Ae^{\left(Bexp\left(-\frac{z}{l_d}\right) - \frac{z}{l_b}\right)} \end{equation}

Also, $P_B(z < 0)$

giving at long time step : \begin{equation} P(\Delta z) = A'exp \left[Bexp\left[-\frac{z}{l_d}\right] (1 + exp[- \frac{\Delta z}{ l_d}]) - \frac{ 2z + \Delta z}{l_b} \right] \end{equation}

## Analysis of pdf of the $\Delta z$ at large time step

To have a better data set we are going to measure the pdf of the $\Delta z$ for a lot of different time step and we arge going to average them. But first of all we need to get rid of the drifts at long time. The best way to do that is to a moving average, taking a box long enough to assume that the mean value should be equal to the equilibrium mean value. We can estimate the time over wich we have to average with the MSD of z and the time it takes to reach the plateau. Here we will look at times > at 30s

## Dedrifting the z trajectory

```python
def movmin(datas, k):
    result = np.empty_like(datas)
    start_pt = 0
    end_pt = int(np.ceil(k / 2))

    for i in range(len(datas)):
        if i < int(np.ceil(k / 2)):
            start_pt = 0
        if i > len(datas) - int(np.ceil(k / 2)):
            end_pt = len(datas)
        result[i] = np.min(datas[start_pt:end_pt])
        start_pt += 1
        end_pt += 1

    return result
```

```python
z_dedrift = z - movmin(z,10000)
```

```python

# Fig for comparing the two

plt.plot(time,z-np.min(z), label = "raw")
plt.plot(time,z_dedrift, label = "dedrifted")
plt.legend()

plt.xlabel("t [s]")
plt.ylabel("z [$\mathrm{\mu m}$]")
```

###  Measuring pdf at large $\Delta t$  with the dedrifted trajectory and analysing it

```python

t_start = 25
t_end = 30
I = np.arange(t_start*fps,t_end*fps)
bins = 50

hists = np.zeros((bins,len(I)))
bins_centers = np.zeros((bins,len(I)))

for n,i in enumerate(I):

    Dezs = z_dedrift[0:-i] - z_dedrift[i:]
    hist, bins_center = pdf(Dezs[~np.isnan(Dezs)],  bins = bins)

    hists[:,n] = hist
    bins_centers[:,n] = bins_center

pdf_long_t = np.mean(hists, axis = 1)
bins_centers_long_t = np.mean(bins_centers, axis = 1)
err_long_t = np.std(hists, axis = 1)
err_bins_centers = np.std(bins_centers, axis = 1)
```

```python
plt.plot(bins_centers_long_t,pdf_long_t)
plt.fill_between(bins_centers_long_t, pdf_long_t-err_long_t, pdf_long_t+err_long_t, alpha = 0.3)
plt.ylabel("$P(\Delta x)$ ($m^{-1}$)")
plt.xlabel("$\Delta x$ ($\mathrm{\mu m}$)")
```

We are now going to code the function
\begin{equation} P(\Delta z) = \int ^ \infty _{- \infty}A'exp \left[Bexp\left[-\frac{z}{l_d}\right] (1 + exp[- \frac{\Delta z}{ l_d}]) - \frac{ 2z + \Delta z}{l_b} \right] \end{equation}

Noting that coding the form :
\begin{equation} P(\Delta z) = \int ^ \infty _{- \infty} dz P_B (z)P_B (z + \Delta z), \end{equation}

Will be easier and $P_B$ will be reused later on. Also since $P_B(z < 0) = 0$ :
\begin{equation} P(\Delta z) = \int ^ \infty _{0} dz P_B (z)P_B (z + \Delta z), \end{equation}

with :
\begin{equation} P_B(z) = Ae^{\left(Bexp\left(-\frac{z}{l_d}\right) - \frac{z}{l_b}\right)} \end{equation}

```python
def P_b(z, A, B, ld, lb):
    P_b = A * np.exp(-B * np.exp(-z / (ld)) - z / lb)
    P_b[z < 0] = 0
    return P_b

def dPdeltaz_long(z, DZ, A, B, ld, lb):
    return P_b(z, A, B, ld, lb) * P_b(z + DZ, A, B, ld, lb)

def P_computation(DZ, A, B, ld, lb):
    z = np.linspace(0, 20e-6, 1000)
    dP = dPdeltaz_long(z, DZ, A, B, ld, lb)
    P = trapz(dP,z)
    return P

def Pdeltaz_long(DZ, B, ld, lb):
    if type(DZ) == float:
        return P_computation(i, 1, B, ld, lb)

    pdf = np.array([P_computation(i, 1, B, ld*1e-9, lb*1e-9) for i in DZ])

    # normalisation of the PDF to not use A

    A = trapz(pdf,DZ*1e6)

    return np.array([P_computation(i, 1, B, ld*1e-9, lb*1e-9) for i in DZ]) / A

```

```python
A = 0.14e8
B = 4
ld = 70
lb = 500
p1 = [B, ld, lb]

# Normalisation fo the pdf

pdf_long_t = pdf_long_t / trapz(pdf_long_t,bins_centers_long_t)

popt, pcov = curve_fit(Pdeltaz_long, bins_centers_long_t * 1e-6,pdf_long_t,p0 = p1)
dataset["pdf_longtime"] = pdf_long_t
dataset["x_pdf_longtime"] = bins_centers_long_t * 1e-6

```

    <ipython-input-25-da5e4a6111f0>:2: RuntimeWarning: overflow encountered in exp
      P_b = A * np.exp(-B * np.exp(-z / (ld)) - z / lb)

```python
A = 0.14e8
B = 400
ld = 70
lb = 500
p0 = [B, ld, lb]

plt.semilogy(bins_centers_long_t,pdf_long_t, label = "experimantal pdf")
plt.fill_between(bins_centers_long_t, pdf_long_t-err_long_t, pdf_long_t+err_long_t, alpha = 0.3)

plt.plot(bins_centers_long_t,Pdeltaz_long(bins_centers_long_t*1e-6, *popt), label = "fit")
plt.ylabel("$P(\Delta z)$")
plt.xlabel("$\Delta z$ [$\mathrm{\mu m}$]")
```

```python
print("We measure, B = {:.2f}, ld = {:.2f} nm, lb = {:.2f} nm".format(*popt))
B, ld, lb = popt

```

    We measure, B = 20.71, ld = 71.84 nm, lb = 504.78 nm

## Analyse of the MSD z plateau

```python
t = np.concatenate((np.arange(1,10, 1),np.arange(10,100, 10), np.arange(100,1000, 100),np.arange(1000,10000, 1000),np.arange(10000,50000, 5000)))

MSD_z_dedrift = MSD(z_dedrift*1e-6, t)

plt.loglog(time[t],MSD_z_dedrift,"o", label = "MSDz undrifted", color = "tab:green")
plt.legend()
plt.ylabel("MSD (m$^2$)")
plt.xlabel("$\Delta$t (s)")

```

```python
def func_plateau(x,P,tau):
    return P * (1 - np.exp(-x/tau))

popt, pcov = curve_fit(func_plateau, time[t], MSD_z_dedrift, p0 = [1e-12, 100])
```

```python
plt.loglog(time[t],MSD_z_dedrift,"o", label = "MSDz undrifted", color = "tab:green")
plt.plot(time[t], func_plateau(time[t], *popt))
plt.legend()
plt.ylabel("MSD (m$^2$)")
plt.xlabel("$\Delta$t (s)")

```

```python
np.mean(MSD_z_dedrift[time[t]>1e2])
```

    5.348018604759325e-13

```python

#dataset["plateau_MSD"] = popt[0]
dataset["plateau_MSD"] =np.mean(MSD_z_dedrift[time[t]>1e2])
print("Measured plateau : {:e}".format(popt[0]))
```

    Measured plateau : 5.387309e-13

The MSD plateau is theoritically given by:

\begin{equation}
Plateau = \int _{-\infty} ^{+\infty} \Delta z ^ 2 P_{\Delta z, t \rightarrow +\infty}(\Delta z,B,l_d,l_b) d\Delta z
\end{equation}

```python
x_Th_Plateau = bins_centers_long_t*1e-6

def Theoritical_Plateau(B,ld,lb):
    x = dataset["x_pdf_longtime"]
    P = Pdeltaz_long(x, B, ld, lb) / trapz(Pdeltaz_long(x, B, ld, lb),x)

    res = trapz((x ** 2) * P,x)
    return res

```

```python
def minimize_plateau(x):
    B = x[0]
    ld = x[1]
    lb = x[2]
    return (np.log(Theoritical_Plateau(B,ld,lb)) - np.log(dataset["plateau_MSD"])) ** 2 / np.log(Theoritical_Plateau(B,ld,lb))**2
```

```python
res_plateau = minimize(minimize_plateau,x0=[B,ld,lb])
print("We measure, B = {:.2f}, ld = {:.2f} nm, lb = {:.2f} nm".format(*res_plateau.x))

```

    We measure, B = 20.71, ld = 71.84 nm, lb = 504.78 nm

## PDF of heights

```python
def logarithmic_hist(data,begin,stop,num = 50,base = 2):

    if begin == 0:
        beg = stop/num
        bins = np.logspace(np.log(beg)/np.log(base), np.log(stop)/np.log(base), num-1, base=base)
        widths = (bins[1:] - bins[:-1])
        bins = np.cumsum(widths[::-1])
        bins = np.concatenate(([0],bins))
        widths = (bins[1:] - bins[:-1])

    else:
        bins = np.logspace(np.log(begin)/np.log(base), np.log(stop)/np.log(base), num, base=base)
        widths = (bins[1:] - bins[:-1])

    hist,bins = np.histogram(data, bins=bins,density=True)
    # normalize by bin width
    bins_center = (bins[1:] + bins[:-1])/2

    return bins_center,widths, hist
```

```python
pdf_z, bins_center_pdf_z = pdf(z_dedrift[z_dedrift < 1.5],  bins = 150)
plt.plot(bins_center_pdf_z,pdf_z, "o")
plt.xlabel("z ($\mathrm{\mu m}$)")
plt.ylabel("P(z) [a.u.]")

```

The idea now is to find where the substrate is, to do this we will use a first method which consist to adjust the PDF with an offset to make it fit with the measured mean Diffusion coefficient. With :
\begin{equation}
< D_i > = \int _{-\infty} ^{\infty} dz D_i(z)P(z)
\end{equation}

For z we are going to use the Padé approx :
\begin{equation} D_z (z) \approx D_0 \left( \frac{6z^2 + 2rz}{6z^2 + 9rz + 2r^2} \right) \end{equation}

For x we are going to use the Faxen formula :
\begin{equation} D_x (z) \approx D_0 \left[ 1 - \frac{9}{16} \left(\frac{r}{z}\right) + \frac{1}{8} \left(\frac{r}{z}\right)^3 - \frac{45}{236} \left(\frac{r}{z}\right)^4 - \frac{1}{16} \left(\frac{r}{z}\right)^5 \right] \end{equation}

To do this we will fit the PDF with an offset, adjust it with the mean value of z. Let's first do it over z

```python

def P_b_off(z,z_off,  B, ld, lb):
    z_off = z_off * 1e-6
    lb = lb * 1e-9
    ld = ld * 1e-9
    z = z - z_off
    P_b = np.exp(-B * np.exp(-z / (ld)) - z / lb)
    P_b[z < 0] = 0

    # Normalization of P_b

    A = trapz(P_b,z * 1e6)
    P_b = P_b / A

    return P_b

```

```python
#Normalization of the PDF

pdf_z = pdf_z / trapz(pdf_z,bins_center_pdf_z)

p2 = [0,B, ld, lb]

popt, pcov = curve_fit(P_b_off, bins_center_pdf_z * 1e-6,pdf_z, p0 = p2)

```

    <ipython-input-39-635051e60521>:6: RuntimeWarning: overflow encountered in exp
      P_b = np.exp(-B * np.exp(-z / (ld)) - z / lb)
    <ipython-input-39-635051e60521>:12: RuntimeWarning: invalid value encountered in true_divide
      P_b = P_b / A

```python

plt.semilogy(bins_center_pdf_z,pdf_z, "o")
plt.plot(bins_center_pdf_z,P_b_off(bins_center_pdf_z*1e-6,*popt))

plt.xlabel("z $\mathrm{(\mu m)}$")
plt.ylabel("P(z)")
plt.grid()

```

```python

mean_Dx = (popt_1[0]+popt_2[0])/2
mean_Dz = popt_3[0]
print("We measure a mean diffusion coefficient of {:.3f}D0 for the perpendicular motion and of {:.3f}D0 for the parallel motion".format((popt_1[0]+popt_2[0])/2, popt_3[0]))

dataset["D_para"] = mean_Dx
dataset["D_perp"] = mean_Dz
```

    We measure a mean diffusion coefficient of 0.522D0 for the perpendicular motion and of 0.243D0 for the parallel motion

```python
Do = 4e-21/(6*np.pi*0.001*r)

def Dz_z(z):
    result = ((6*z*z + 2*r*z) / (6*z*z + 9*r*z + 2*r*r))
    return result

def Dx_z(z):
    result = (1 - 9/16*(r/(z+r)) + 1/8*(r/(z+r))**3 - 45/256*(r/(z+r))**4 - 1/16*(r/(z+r))**5)
    return result

```

```python
def minimizer(z_off):
    Dx_pdf = trapz(Dx_z(bins_center_pdf_z*1e-6) * P_b_off(bins_center_pdf_z*1e-6,z_off,*popt[1:]),bins_center_pdf_z)
    Dz_pdf = trapz(Dz_z(bins_center_pdf_z*1e-6) * P_b_off(bins_center_pdf_z*1e-6,z_off,*popt[1:]),bins_center_pdf_z)

    return np.abs((1 - mean_Dx/Dx_pdf) + (1 - mean_Dx/Dx_pdf))

res = minimize(minimizer, 0, method='nelder-mead')

```

```python
offset = res
```

```python

offset = np.mean(res["final_simplex"][0])
print("From the measurement of the mean diffusion coefficient, we measure an offset of {:.3f} um".format(offset))

```

    From the measurement of the mean diffusion coefficient, we measure an offset of 0.005 um

```python
def logarithmic_hist(data,begin,stop,num = 50,base = 2):
    """
    Function to make logarithmic histograms to have more points
    near the surface and where the particle spend the most of its time.
    """
    if begin == 0:
        beg = stop/num
        bins = np.logspace(np.log(beg)/np.log(base), np.log(stop)/np.log(base), num-1, base=base)
        widths = (bins[1:] - bins[:-1])
        bins = np.cumsum(widths[::-1])
        bins = np.concatenate(([0],bins))
        widths = (bins[1:] - bins[:-1])

    else:
        bins = np.logspace(np.log(begin)/np.log(base), np.log(stop)/np.log(base), num, base=base)
        widths = (bins[1:] - bins[:-1])

    hist,a= np.histogram(data, bins=bins,density=True)
    # normalize by bin width
    bins_center = (bins[1:] + bins[:-1])/2

    return bins_center,widths, hist

bins_center_pdf_z,widths, pdf_z = logarithmic_hist(z_dedrift,0.01,2,num = 50,base = 12)

p2 = [0, B, ld, lb]
popt_pdf, pcov_pdf = curve_fit(P_b_off, bins_center_pdf_z * 1e-6,pdf_z, p0 = p2)
dataset["pdf_z"] = pdf_z
dataset["x_pdf_z"] = bins_center_pdf_z * 1e-6

plt.semilogy(bins_center_pdf_z,pdf_z, "o")
plt.plot(bins_center_pdf_z,P_b_off(bins_center_pdf_z*1e-6,*popt_pdf), color = "black")

plt.xlabel("z ($\mathrm{\mu m}$)")
plt.ylabel("P(z)  ($m^{-1}$)")
```

```python
offset_pdf, B_pdf, ld_offset, lb_offset = popt_pdf
```

We write the diffusion function.

```python
def Dz_z(z,off):
    off = off * 1e-6
    z = z - off
    result = ((6*z*z + 2*r*z) / (6*z*z + 9*r*z + 2*r*r))
    return result

def Dx_z_off(z,offset):
    offset = offset * 1e-6
    z = z + offset
    result = (1 - 9/16*(r/(z+r)) + 1/8*(r/(z+r))**3 - 45/256*(r/(z+r))**4 - 1/16*(r/(z+r))**5)
    return result
```

## Measuring the diffusion coefficient using the Frishman and Ronceray's method

```python
from scipy.io import loadmat
D = loadmat("../data/diffusionAnalysis.mat")["diffusion"]
dataset["z_D"] = D[:,0] + 17.94*1e-9
dataset["z_D_x"] = (D[:,1] + D[:,2])/2
dataset["z_D_z"] = D[:,3]

```

```python
def c_P_D(B,ld,lb,offset=None):
    if offset == None:
        offset = 0

    z = np.linspace(1e-9,15e-6,1000)

    P_D = Dz_z(z,offset) * Do * P_b_off(z, offset, B, ld, lb)

    return Dz_z(z,offset) * Do, P_D/np.trapz(P_D,z)

def _P_Dz_short_time(Dz,Dt,B,ld,lb,offset=None):
    if offset == None:
        offset = 0

    D_z, P_D = c_P_D(B,ld,lb,offset)

    P = np.trapz(P_D / np.sqrt(4 * np.pi * D_z * Dt) * np.exp(- Dz**2 / (4 * D_z*Dt)),D_z)

    return P

def P_Dz_short_time(Dz,Dt,B,ld,lb,offset=None):
    if offset == None:
        offset = 0

    P = [_P_Dz_short_time(i,Dt,B,ld,lb,offset=offset) for i in Dz]
    P = np.array(P)
    P = P / np.trapz(P,Dz)

    return P

```

# Fit everything in the same time !

Finaly we can fit everything in the same time to recap we have :

   - MSD x and MSD y => $< D >$
   - MSD z => $< D >$
   - mean $< D >$ with the pdf
   - Long time pdf $\Delta z$ => $l_d,l_b,B$
   - Pdf z => $offset,l_d,l_b, B$
   - D parallel, perp => offset

The minimizer $\chi ^2$ we are going to optimize can be written as :
\begin{equation} \chi^2 = \sum _{n = 1} ^{N} \chi_n^2 \end{equation}\begin{equation} \chi_n^2 = \sum^A(n) _i = 1 \frac{1}{\sigma_{ni}}(y_{ni} - y_n(x_{ni},\boldsymbol{a}))^2 \end{equation}

with $\sigma_{ni}$ the uncertainty (can be set to 1), A the number of point in the dataset for each function, $y_n$, nth equation, $\boldsymbol{a}$ the fit parameters

We have nonlinear functions so we can use the Marquardt to optimize or Nelder-Mead methods to optimize the minimizer.

```python
def minimizer_diffusion_coeff(mean_D_para, mean_D_perp, z_off, B, ld, lb):
    #minimization of the mean diffusion coefficient measurement with the PDF and MSD
    a = trapz(Dx_z_off(bins_center_pdf_z*1e-6, z_off) * P_b_off(bins_center_pdf_z*1e-6,z_off, B, ld, lb),bins_center_pdf_z)
    b = trapz(Dz_z(bins_center_pdf_z*1e-6, z_off) * P_b_off(bins_center_pdf_z*1e-6,z_off, B, ld, lb),bins_center_pdf_z)
    at = mean_Dx; bt = mean_Dz
    return (a-at)**2 / at**2 + (b-bt)**2 / bt**2

dataset["z"] = z_dedrift
dataset["x"] = x
dataset["y"] = y

def minimizer_Dz_small_t(B,ld,lb):
    xi = 0

    for n,i in enumerate([1,2,3]):
        Dezs = (dataset["z"][0:-i] - dataset["z"][i:]) * 1e-6
        Dezs = Dezs# - np.mean(Dezs)

        hist, bins_center = pdf(Dezs[~np.isnan(Dezs)], bins = 30)
        hist = hist/np.trapz(hist,bins_center)

        Dz_th = bins_center
        PPP = P_Dz_short_time(Dz_th,time[i],B,ld,lb)

        #xi = xi + np.nanmean((((np.abs(hist) - (PPP) ) ) ** 2) / ((np.abs(hist)**2)))
        xi = xi + np.nanmean(((hist[hist>0]-PPP[hist>0]) ** 2) / hist[hist>0]**2)
    return xi
```

```python
dataset["D_para"] = mean_Dx
dataset["D_perp"] = mean_Dz

def minimizer(x, *args):
    data = dataset
    ld = x[0]
    lb = x[1]
    B = x[2]
    offset_dif = x[3]
    offset_boltz = 0

    chi_mean_D_pdf = minimizer_diffusion_coeff(dataset["D_para"], dataset["D_perp"],offset_boltz, B, ld, lb)
    chi_MSD_plateau = minimize_plateau([B,ld,lb])

    E_longtime_pdf = (Pdeltaz_long(data["x_pdf_longtime"], B, ld, lb)) - (data["pdf_longtime"])
    chi_longtime_pdf = np.mean((E_longtime_pdf[E_longtime_pdf > -np.inf] ** 2) / (((Pdeltaz_long(data["x_pdf_longtime"], B, ld, lb)))**2))

    E_chi_pdf_z = (P_b_off(data["x_pdf_z"], offset_boltz, B, ld, lb) - data["pdf_z"])
    chi_pdf_z = np.nanmean((E_chi_pdf_z[E_chi_pdf_z > -np.inf] ** 2) / ((P_b_off(data["x_pdf_z"], offset_boltz, B, ld, lb))**2))

    E_D_z = (Dz_z(data["z_D"], offset_dif)) - (data["z_D_z"] / Do)
    chi_D_z = np.mean((E_D_z[E_D_z > -np.inf] ** 2) / ((Dz_z(data["z_D"], offset_dif))**2))

    E_D_x = (Dx_z_off(data["z_D"], offset_dif)) - (data["z_D_x"] / Do)
    chi_D_x = np.mean((E_D_x[E_D_x > -np.inf]  ** 2) / ((Dx_z_off(data["z_D"], offset_dif))**2))

    chi_Dz_small_t = minimizer_Dz_small_t(B,ld,lb)

    summ = chi_mean_D_pdf + chi_MSD_plateau + chi_longtime_pdf + chi_pdf_z + chi_D_z + chi_D_x + chi_Dz_small_t

    return  summ
```

```python
B = 5
ld = ld_offset
x0 = [ld,550,B,0,offset_pdf]

```

```python
from scipy.optimize import leastsq

options={
    'maxc1or': 30,
    'ftol': 2.2e-10,
    'gtol': 1e-5,
    'eps': 1e-08,
    'maxfun': 15000,
    'maxiter': 15000,
    'maxls': 20,
    'finite_diff_rel_step': None,
}

res = minimize(minimizer,
                   x0,
                   method = "BFGS",
                   tol = 1e-1,
                  )
```

```python
res.x
results = {
    "ld":res.x[0],
    "lb":res.x[1],
    "B":res.x[2],
    "offset_diffusion":res.x[3],
}

results
```

    {'ld': 25.470469609361395,
     'lb': 549.9705309805663,
     'B': 4.405291114790512,
     'offset_diffusion': 0.019878479897997275}

This final result has been used to plot theories along the manuscript.

```python

```


```python
import sys
import os
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath('./introduction.ipynb'))))
```

```python
import networkx as nx
from ylearn.causal_model import model, graph
```

## The first test example of the id method.

### Initialization.

```python
# Construct the causal graph for the causal model
causation = {
    'X': ['W'],
    'Z': ['X', 'W'],
    'W': [],
    'Y': ['Z'],
}
arc = [('X', 'Y')]
cg = graph.CausalGraph(causation, latent_confounding_arcs=arc)
```

```python
cm = model.CausalModel(causal_graph=cg)
```

### id method.

```python
# Test the identification method
p = cm.id({'Y'}, {'X'})
```

```python
causation = {
    'X': ['W'],
    'Z': ['X', 'W'],
    'W': [],
    'Y': ['Z'],
}
arc = [('X', 'Y')]
cg = graph.CausalGraph(causation, latent_confounding_arcs=arc)
p = cm.id({'Y'}, {'X'})
p.show_latex_expression()
```

$\sum_{Z, W}\left[P(W)\right]\left[\left[P(Z|X, W)\right]\right]\left[\sum_{X}\left[P(Y|Z, X, W)\right]\left[P(X|W)\right]\right]$

```python

```

$P(y|do(x))$

$y=g(z,w,x,\epsilon) = f(z,x,w) + \epsilon = xh(z,w) + \epsilon$

$E(y|z,w,x)$

ATE

$E(Y(x)) = \sum_{y}yP^*(y)$

\begin{align*}
\partial_x E[Y(x)] = \sum_{y}y\partial_x \left(\sum_{Z}\left[\sum_{X}\left[P(Y|Z, X)\right]\left[P(X)\right]\right]\left[P(Z|X)\right]\right)
\end{align*}

$P(y|x,z)P(z)$

$E_zE(Y|x,z)$

```python
print(p.parse())
```

    \sum_{Z, W}\left[P(W)\right]\left[\left[P(Z|X, W)\right]\right]\left[\sum_{X}\left[P(Y|Z, X, W)\right]\left[P(X|W)\right]\right]

$\sum_{W, Z}\left[\sum_{X}\left[P(X|W)\right]\left[P(Y|X, Z, W)\right]\right]\left[\left[P(Z|X, W)\right]\right]\left[P(W)\right]$

### Final result.

The above expressions are combined to give the final result:

\begin{equation}
P(y|do(x)) = \sum_{w, z} P(w)P(z|x, w)\sum_{x} P(x|w)P(y|x,w,z)
\end{equation}

Exactly the desired one.

## The second test example of the id method.

```python
causation1 = {
    'X': ['Z1'],
    'Z1': [],
    'Z2': ['X'],
    'Y': ['Z2'],
}
arcs1 = [('X', 'Z1'), ('Z1', 'Z2'), ('Z1', 'Y'), ('X', 'Y')]
cg1 = graph.CausalGraph(causation1, latent_confounding_arcs=arcs1)
cm1 = model.CausalModel(cg1)
```

```python
cm1.id({'Y'}, {'X'})
```

#### The causal effect in the above example can not be identified. The id method gives us exactly what we want.

### Test another one.

```python
causation5 = {
    'X': ['Z2'],
    'Y': ['Z1', 'Z3'],
    'Z1': ['X', 'Z2'],
    'Z2': [],
    'Z3': ['Z2']
}
arcs5 = [('X', 'Z2'), ('X', 'Y'), ('X', 'Z3'), ('Z2', 'Y')]
cg5 = graph.CausalGraph(causation5, latent_confounding_arcs=arcs5)
cm5 = model.CausalModel(cg5)
p5 = cm5.id({'Y'}, {'X'})
```

```python
p5.show_latex_expression()
```

$\sum_{Z2, Z3, Z1}\left[\left[P(Z1|Z2, X)\right]\right]\left[\left[P(Z2)\right]\left[P(Y|Z2, Z3)\right]\right]\left[\left[P(Z3|Z2)\right]\right]$

#### The next example can be done with the frontdoor adjustment, we check wether the result returned by the id method will conincide with that returned by the frontdoor adjustment.

```python
causation6 = {
    'X': [],
    'Y': ['Z'],
    'Z': ['X'],
}
arcs6 = [('X', 'Y')]
cg6 = graph.CausalGraph(causation6, latent_confounding_arcs=arcs6)
cm6 = model.CausalModel(cg6)
p6 = cm6.id('Y', 'X')
p6
```

    <ylearn.causal_model.prob.Prob at 0x7fc93c775370>

```python
p6.show_latex_expression()
```

$\sum_{Z}\left[\sum_{X}\left[P(Y|Z, X)\right]\left[P(X)\right]\right]\left[\left[P(Z|X)\right]\right]$

```python
p.marginal
```

    {'W', 'Z'}

```python
print(p.parse())
```

    \sum_{Z, W}\left[P(W)\right]\left[\left[P(Z|X, W)\right]\right]\left[\sum_{X}\left[P(Y|Z, X, W)\right]\left[P(X|W)\right]\right]

$\sum_{Z}\left[\left[P(Z|X)\right]\right]\left[\sum_{X}\left[P(Y|X, Z)\right]\left[P(X)\right]\right]$

```python
for a in p.product:
    print(f'var {a.variables} ')
    print(f'mar {a.marginal}')
    print(f'cond {a.conditional}')
    if a.product:
        for b in a.product:
           print(f'sub var {b.variables}')
           print(f'sub mar {b.marginal}')
           print(f'sub cond {b.conditional}')
```

    var {'W'}
    mar set()
    cond set()
    var set()
    mar set()
    cond set()
    sub var {'Z'}
    sub mar set()
    sub cond {'X', 'W'}
    var set()
    mar {'X'}
    cond set()
    sub var {'Y'}
    sub mar set()
    sub cond {'Z', 'X', 'W'}
    sub var {'X'}
    sub mar set()
    sub cond {'W'}

The above result will be converted to the following probability expression

\begin{align*}
    P(y|do(x)) & = \sum_{z} \sum_x P(x)P(y|x, z) P(z|x)\\
                & = \sum_{z} P(y|z) P(z|x)
\end{align*}
This is correct.

## Now we run one more example.

```python
causation2 = {
    'W1': [],
    'W2': [],
    'X': ['W1'],
    'Y1': ['X'],
    'Y2': ['W2']
}
arcs2 = [('W1', 'Y1'), ('W1', 'W2'), ('W1', 'Y2'), ('W1', 'Y1')]
cg2 = graph.CausalGraph(causation2, latent_confounding_arcs=arcs2)
cm2 = model.CausalModel(cg2)
```

```python
p2 = cm2.id({'Y1', 'Y2'}, {'X'})
```

```python
p2.show_latex_expression()
```

$\sum_{W2}\left[\left[P(Y2|W2)\right]\right]\left[\sum_{W1}\left[P(W1)\right]\left[P(Y1|X, W1)\right]\right]\left[P(W2)\right]$

```python
print(p2.parse())
```

    \sum_{W2}\left[\left[P(Y2|W2)\right]\right]\left[\sum_{W1}\left[P(W1)\right]\left[P(Y1|X, W1)\right]\right]\left[P(W2)\right]

$\sum_{W2}\left[P(W2)\right]\left[\sum_{W1}\left[P(Y1|X, W1)\right]\left[P(W1)\right]\right]\left[\left[P(Y2|W2)\right]\right]$

```python
p2.variables
p2.marginal
```

    {'W2'}

```python
cm2.causal_graph.remove_incoming_edges({'X'}, new=True).ancestors({'Y1', 'Y2'})
```

    {'W2', 'X', 'Y1', 'Y2'}

```python
list(cm2.causal_graph.remove_nodes({'X', 'W1'}, new=True).c_components)
```

    [{'Y1'}, {'W2'}, {'Y2'}]

```python
for p_ in p2.product:
    print(f'var {p_.variables}')
    print(f'cond {p_.conditional}')
    print(f'mar {p_.marginal}')
    for p__ in p_.product:
        print(f'sub var {p__.variables}')
        print(f'sub cond {p__.conditional}')
        print(f'sub mar {p__.marginal}')
        if p__.product:
            for z in p__.product:
                print(f'subsub var {z.variables}')
                print(f'subsub maar {z.arginal}')
                print(f'subsub prod {z.product}')
```

    var set()
    cond set()
    mar set()
    sub var {'Y2'}
    sub cond {'W2'}
    sub mar set()
    var set()
    cond set()
    mar {'W1'}
    sub var {'W1'}
    sub cond set()
    sub mar set()
    sub var {'Y1'}
    sub cond {'X', 'W1'}
    sub mar set()
    var {'W2'}
    cond set()
    mar set()

The above result gives us the following expression

\begin{align*}
    P(y_1, y_2 |do(x)) &= \sum_{w_2} P(y_2|w_2)\sum_{w_1}P(w_1)P(y_1|w_1, x)P(w_2) \\
                        & = \sum_{w_2}P(y_2, w_2)\sum_{w_1}P(w_1)P(y_1|w_1, x)
\end{align*}

Also correct.

# We now test other methods.

## Start from the backdoor.

```python
import networkx as nx
from ylearn.causal_model import model, graph
causation3 = {
    'X1': [],
    'X2': [],
    'X3': ['X1'],
    'X4': ['X1', 'X2'],
    'X5': ['X2'],
    'X6': ['X'],
    'X': ['X3', 'X4'],
    'Y': ['X6', 'X4', 'X5']
}
cg3 = graph.CausalGraph(causation3)
cm3 = model.CausalModel(cg3)
```

```python
ad, p3 = list(cm3.identify({'X'}, {'Y'}, identify_method=('backdoor', 'simple')).values())[0]
```

```python
p3.show_latex_expression()
```

$\sum_{X3, X4}\left[P(X3, X4)\right]\left[P(Y|X3, X, X4)\right]$

$\sum_{X3, X4}\left[P(X3, X4)\right]\left[P(Y|X4, X3, X)\right]$

```python
cm3.identify({'X'}, {'Y'}, identify_method=('backdoor', 'all'))
```

    {'backdoor': ([{'X3', 'X4'},
       {'X1', 'X4'},
       {'X2', 'X4'},
       {'X4', 'X5'},
       {'X1', 'X3', 'X4'},
       {'X2', 'X3', 'X4'},
       {'X3', 'X4', 'X5'},
       {'X1', 'X2', 'X4'},
       {'X1', 'X4', 'X5'},
       {'X2', 'X4', 'X5'},
       {'X1', 'X2', 'X3', 'X4'},
       {'X1', 'X3', 'X4', 'X5'},
       {'X2', 'X3', 'X4', 'X5'},
       {'X1', 'X2', 'X4', 'X5'},
       {'X1', 'X2', 'X3', 'X4', 'X5'}],
      <ylearn.causal_model.prob.Prob at 0x7fc93c7fbdc0>)}

```python
cm3.identify({'X'}, {'Y'}, identify_method=('backdoor', 'minimal'))
```

    {'backdoor': ([{'X3', 'X4'}],
      <ylearn.causal_model.prob.Prob at 0x7fc93c82e220>)}

### Verified. Now test other methods related to backdoor adjustment.

```python
cm3.is_valid_backdoor_set({'X1', 'X4'}, {'X'}, {'Y'})
```

    True

```python
cm3.is_valid_backdoor_set({'X4'}, 'X', 'Y')
```

    False

```python
cm3.get_backdoor_path('X', 'Y')
```

    [['X', 'X3', 'X1', 'X4', 'Y'],
     ['X', 'X3', 'X1', 'X4', 'X2', 'X5', 'Y'],
     ['X', 'X4', 'Y'],
     ['X', 'X4', 'X2', 'X5', 'Y']]

```python
cm3.has_collider(['X', 'X3', 'X1', 'X4', 'X2', 'X5', 'Y'])
```

    True

```python
cm3.is_connected_backdoor_path(['X', 'X4', 'X2', 'X5', 'Y'])
```

    True

## Now test methods related to frontdoor adjustment.

```python
causation4 = {
    'X': [],
    'Z': ['X'],
    'Y': ['Z']
}
arcs4 = [('X', 'Y')]
cg4 = graph.CausalGraph(causation4, latent_confounding_arcs=arcs4)
cm4 = model.CausalModel(cg4)
```

```python
cm4.get_backdoor_path('X', 'Z')
```

    [['X', 'U0', 'Y', 'Z']]

```python
cm4.causal_graph.dag.nodes
```

    NodeView(('X', 'Z', 'Y'))

```python
cm4.has_collider(['Z', 'X', 'U0', 'Y'])
```

    False

```python
list(cm4.causal_graph.dag.nodes)
```

    ['X', 'Z', 'Y']

```python
cm4.is_connected_backdoor_path(['Z', 'X', 'U0', 'Y'])
```

    True

```python
cm4.is_valid_backdoor_set({'X'}, {'Z'}, {'Y'})
```

    True

```python
cm4.is_frontdoor_set({'Z'}, 'X', 'Y')
```

    True

```python
z, p = cm4.get_frontdoor_set('X', 'Y')
```

```python
p.show_latex_expression()
```

$\sum_{Z}\left[\sum_{X}\left[P(X)\right]\left[P(Y|Z, X)\right]\right]\left[P(Z|X)\right]$

```python
print(p.parse())
```

    \sum_{Z}\left[\sum_{X}\left[P(X)\right]\left[P(Y|Z, X)\right]\right]\left[P(Z|X)\right]

$\sum_{Z}\left[P(Z|X)\right]\left[\sum_{X}\left[P(Y|Z, X)\right]\left[P(X)\right]\right]$

```python
p.show_latex_expression()
```

$\sum_{Z}\left[\sum_{X}\left[P(X)\right]\left[P(Y|Z, X)\right]\right]\left[P(Z|X)\right]$

## Let's see if our tool can be used in a previous paper (https://arxiv.org/pdf/2009.13000.pdf).

```python
cau = {
    'D': [],
    'C': ['D', 'X'],
    'X': ['D'],
    'Y': ['C', 'X']
}
cgg = graph.CausalGraph(causation=cau)
cmm = model.CausalModel(cgg)
```

```python
ppp = cmm.id({'Y'}, {'X'})
```

```python
ppp.show_latex_expression()
```

$\sum_{C, D}\left[\left[P(C|X, D)\right]\right]\left[\left[P(Y|C, X, D)\right]\right]\left[P(D)\right]$

## Exactly what we want.

### Now we test methods related to iv

```python
causation7 = {
    'p':[],
    't': ['p'],
    'l': ['p'],
    'g': ['t', 'l']
}
arc7 = [('t', 'g')]
cg7 = graph.CausalGraph(causation=causation7, latent_confounding_arcs=arc7)
cm7 = model.CausalModel(causal_graph=cg7)
```

```python

```

```python
cm7.get_iv('t', 'g')
```

    No valid instrument variable has been found.

```python
causation8 = {
    'p':[],
    't': ['p'],
    'l': [],
    'g': ['t', 'l']
}
cg8 = graph.CausalGraph(causation=causation8, latent_confounding_arcs=arc7)
cm8 = model.CausalModel(causal_graph=cg8)
cm8.get_iv('t', 'g')
```

    {'p'}


# Table of Contents

1  Generating dataset

2  Estimate Parameters of NOE and NARX models

3  Plot Histogram

```python
import os.path
import numpy as np
import matplotlib.pyplot as plt
from oeshoot import narx
import multiprocessing
from joblib import Parallel, delayed
import matplotlib as mpl
num_cores = multiprocessing.cpu_count()
%matplotlib inline
```

# Generating dataset

The data set used for identification will be generated using the following equation:
\begin{equation}
y[n] = 0.5y[n-1]+0.8u[n-2]+u^2[n-1]-0.05y^2[n-2]+0.5
\end{equation}
The system input $u[n]$ is generated such that each value is hold for ten samples and have Gaussian distribution with zero  mean and unitary variance.

```python
# Define System
def system_equation(y, u):
    ynext = 0.5*y[0]+0.8*u[1]+u[0]**2-0.05*y[1]**2+0.5
    return ynext

# System Parameters
N = 2
M = 2
Ny = 1
Nu = 1
```

```python
# Defining parameters
rept = 2
random_samples = 200
total_time = rept*random_samples

# Time vector
n = np.arange(0, total_time)

# Generating input
u = np.random.normal(size=(random_samples,)).repeat(rept)
```

```python
# Plot
label_size = 17
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size
_, ax = plt.subplots(figsize=(12, 6))
ax.step(n, u, color='black')
ax.set_xlabel("$k$", fontsize=26)
ax.set_ylabel("$u[k]$", fontsize=26)
ax.grid()
plt.tight_layout()
plt.show()
```

```python
# Initialize input
y = np.empty_like(u)

# Initial Conditions
y0 = [0, 0]

# Simulate it
y[0:2] = y0
for i in range(2, len(y)):
    y[i] = system_equation([y[i-1], y[i-2]], [u[i-1], u[i-2]])+np.random.normal(scale=0.00001)

# Noise free data
ynoise_free = y

# Generate noise
v = np.random.normal(scale=0.5, size=y.shape)

# Data corrupted with noise
ynoisy = ynoise_free + v

# Divide between validation and training set
Ni = 600
Nv = len(y) - Ny

# Identification data
yi = ynoisy[:Ni].reshape([-1, 1])
ui = u[:Ni].reshape([-1, 1])

# Validation data
yv = ynoise_free[Ni:].reshape([-1, 1])
uv = u[Ni:].reshape([-1, 1])

# Plot
_, ax = plt.subplots(figsize=(12, 6))
ax.plot(n, ynoisy, label='Noisy data', color='black')
ax.set_xlabel("$k$", fontsize=26)
ax.set_ylabel("$y[k]$", fontsize=26)
ax.grid()
plt.tight_layout()
plt.show()
```

# Estimate Parameters of NOE and NARX models

```python
# Polynomial model
m1 = narx.Monomial(1, 0, [1], [])
m2 = narx.Monomial(0, 1, [], [2])
m3 = narx.Monomial(0, 1, [], [1], [], [2])
m4 = narx.Monomial(1, 0, [2], [], [2], [])
m5 = narx.Monomial(0, 0, [], [], [], [])
mdl = narx.Polynomial([m1, m2, m3, m4, m5])

# Define error objects
sim_error = narx.SimulationError(mdl, ys, ui, maxlength=np.Inf)
predict_error = narx.PredictionError(mdl, yi, ui)

initial_guess = np.zeros_like(theta)

# Estimate params using prediction error
params_predict, info = predict_error.lsq_estimate_parameters(initial_guess)

# Estimate params using simulation error
params_sim, info = sim_error.lsq_estimate_parameters(initial_guess, use_sparse=True, use_jacobian=True)

# Show estimated parameter
params_sim
```

    array([ 0.5 ,  0.8 ,  1.  , -0.05,  0.5 ])

# Plot Histogram

Repeat the same experiment multiple times

```python
# Noise levels
noise_levels = np.linspace(0.01, 1.2, 10)

# Numer of trials
ntrials = 100

def evaluate_estimator(i, nn):
    np.random.seed(i+int(10*nn))
    # Generating output
    y = narx.simulate(mdl, y0, ui, theta)
    y = np.vstack((y0, y[:-1]))

    # Add noise
    y += np.random.normal(scale=nn, size=y.shape)

    # Define errors
    sim_error = narx.SimulationError(mdl, y, ui, maxlength=np.Inf)
    predict_error = narx.PredictionError(mdl, y, ui)

    # Estimate params using prediction error
    params_predict, info = predict_error.lsq_estimate_parameters(initial_guess)

    # Estimate params using simulation error
    params, info = sim_error.lsq_estimate_parameters(initial_guess, use_sparse=True)

    return [params, params_predict]
```

```python
# Only do the computation when necessary
if os.path.isfile('polynomial_results_ms.npy'):
    results=np.load('polynomial_results_ms.npy')
else:
    results = Parallel(n_jobs=num_cores)(delayed(evaluate_estimator)(i, nn) for nn in noise_levels for i in range(ntrials))
    results = np.array(results)
    results = results.reshape((len(noise_levels), ntrials, 2, len(theta)))
    np.save('polynomial_results_ms.npy', results)
```

```python
# Compute mean and standard deviation
mean = np.mean(results, axis=1)
std = np.std(results, axis=1)

label_size = 15
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size

fig, ax = plt.subplots(2, 1,figsize= (8, 3*2))

for pn in range(2):
    (_, caps, _) = ax[pn].errorbar(10*np.log10(np.var(ynoise_free)/noise_levels**2), mean[:, 0, pn], yerr=std[:, 0, pn], fmt='--o', ms=6, elinewidth=2.5, linewidth=3)
    for cap in caps:
        cap.set_markeredgewidth(2.5)
    (_, caps, _) = ax[pn].errorbar(10*np.log10(np.var(ynoise_free)/noise_levels**2), mean[:, 1, pn], yerr=std[:, 1, pn], fmt='--o', ms=6, elinewidth=2.5, linewidth=3)
    for cap in caps:
        cap.set_markeredgewidth(2.5)
    ax[pn].axhline(theta[pn], color='black', linestyle='-', lw=1)
    ax[pn].set_ylabel(r'$\Theta_'+str(pn+1)+'$', fontsize=30)
    ax[pn].set_xlabel('SNR', fontsize=20)
    ax[pn].set_xlim(ax[pn].get_xlim()[::-1])
    if pn==3:
        ax[pn].set_yticks([-0.05, -0.04, -0.03, -0.02])

plt.tight_layout()
plt.savefig("single_shooting1.pdf")
plt.show()
```


### Convex optimization

+ It is not easy to find practical problems that are just smooth (with Lipschitz continuous gradients only), in order to show sublinear convergence in practice (even logistic regression, as we will see later on, has linear convergence under assumptions).

+ In the following example, we will highlight the importance of the condition number in optimization

### Well-conditioned linear regression

```python
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import random
from scipy import stats
from scipy.optimize import fmin

from matplotlib import rc
#rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
## for Palatino and other serif fonts use:
rc('font',**{'family':'serif','serif':['Palatino']})
rc('text', usetex=True)

from PIL import Image

import random
from numpy import linalg as la
from scipy.sparse.linalg import svds, eigs
import math

p = 100  # Ambient dimension
n = 100  # Number of samples

# Generate a p-dimensional zero vector
x_star = np.random.randn(p)
# Normalize
x_star = (1 / la.norm(x_star, 2)) * x_star

# Well-conditioned matrix A
A = np.random.randn(n, p)
[U, S, V] = la.svd(A)
S = np.linspace(1, 2, n)
S = np.diag(S)
A = U.dot(S.dot(V.T))

y = A.dot(x_star)

# Print condition number of A
print(la.cond(A))

# Compute largest singular value of A
u, s, v = svds(A, k=1)
eta = 1/s**2
print(eta)
```

    1.9999999999999991
    [0.25]

```python
def GD(y, A, eta, iters, epsilon, verbose, x_star):
    p = A.shape[1]   # Length of original signal
    n = A.shape[0]   # Length of measurement vector

    x_new = np.zeros(p)    # Initial estimate
    At = np.transpose(A)   # Transpose of A

    x_list, f_list = [1], [f(y, A, x_new)]

    for i in range(iters):
        x_old = x_new

        # Compute gradient
        grad = -At @ (y - A @ x_new)

        # Perform gradient step
        x_new = x_old - eta * grad

        if (la.norm(x_new - x_old, 2) / la.norm(x_new, 2)) < epsilon:
            break

        # Keep track of solutions and objective values
        x_list.append(la.norm(x_new - x_star, 2))
        f_list.append(f(y, A, x_new))

        if verbose:
            print("iter# = "+ str(i) + ", ||x_new - x_old||_2 = " + str(la.norm(x_new - x_old, 2)))

    print("Number of steps:", len(f_list))
    return x_new, x_list, f_list

def f(y, A, x):
    return 0.5 * math.pow(la.norm(y - A @ x, 2), 2)
```

```python
# Run algorithm
epsilon = 1e-6                # Precision parameter
iters = 100

x_IHT, x_list, f_list = GD(y, A, eta, iters, epsilon, True, x_star)

# Plot
plt.rc('text', usetex=True)
plt.rc('font', family='serif')

xs = range(len(x_list))
plt.plot(xs, x_list, '-o', color = '#3399FF', linewidth = 4, alpha = 0.7, markerfacecolor = 'b')
plt.yscale('log')
plt.xlabel('Iterations')
plt.ylabel(r"$\|x^\star - \widehat{x}\|_2$")

# Make room for the ridiculously large title.
plt.subplots_adjust(top=0.8)
plt.show()
```

```python
# Ill-conditioned matrix A
A = np.random.randn(n, p)
[U, S, V] = la.svd(A)
S = np.linspace(1, 100, n)
S = np.diag(S)
A = U.dot(S.dot(V.T))

y = A.dot(x_star)

# Print condition number of A
print(la.cond(A))

# Compute largest singular value of A
u, s, v = svds(A, k=1)
eta = 2/(100 + 1)**2
print(eta)
```

    100.00000000000009
    0.00019605920988138416

```python
# Run algorithm
epsilon = 1e-6                # Precision parameter
iters = 1000

x_IHT, x_list, f_list = GD(y, A, eta, iters, epsilon, True, x_star)

# Plot
plt.rc('text', usetex=True)
plt.rc('font', family='serif')

xs = range(len(x_list))
plt.plot(xs, x_list, '-o', color = '#3399FF', linewidth = 4, alpha = 0.7, markerfacecolor = 'b')
plt.yscale('log')
plt.xlabel('Iterations')
plt.ylabel(r"$\|x^\star - \widehat{x}\|_2$")

# Make room for the ridiculously large title.
plt.subplots_adjust(top=0.8)
plt.show()
```

**Problem definition: Sparse linear regression**.

\begin{align}
  y = A x^\star + w
\end{align}

+ $A \in \mathbb{R}^{n \times p}$, but now $n \ll p$
+ $x^\star \in \mathbb{R}^p$ but $k$-sparse, where $k \ll p$
+ $w \in \mathbb{R}^n$

Assume $n = p$, and $A$ is in general position.
Given $y$ and $A$:

\begin{equation*}
\begin{aligned}
& \underset{x \in \mathbb{R}^p}{\text{min}}
& & f(x) \triangleq \|y - A x\|_2^2 \\
& \text{subject to}
& & \|x\|_1 \leq \lambda
\end{aligned}
\end{equation*}

+ This is the famous Lasso approach, that was analyzed and popularized by R. Tibshirani in 1996.

Tibshirani, Robert. "Regression shrinkage and selection via the lasso." Journal of the Royal Statistical Society. Series B (Methodological) (1996): 267-288. ---> 25K citations!

```python
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import random
from scipy import stats
from scipy.optimize import fmin

from matplotlib import rc
#rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
## for Palatino and other serif fonts use:
rc('font',**{'family':'serif','serif':['Palatino']})
rc('text', usetex=True)

from PIL import Image

import random
from numpy import linalg as la

p = 1000  # Ambient dimension
n = 600   # Number of samples
k = 50    # Sparsity level

# Generate a p-dimensional zero vector
x_star = np.zeros(p)
# Randomly sample k indices in the range [1:p]
x_star_ind = random.sample(range(p),  k)
# Set x_star_ind with k random elements from Gaussian distribution
x_star[x_star_ind] = np.random.randn(k)
# Normalize
x_star = (1 / la.norm(x_star, 2)) * x_star

# Plot
xs = range(p)
markerline, stemlines, baseline = plt.stem(xs, x_star, '-.')
plt.setp(markerline, 'alpha', 0.3, 'ms', 6)
plt.setp(markerline, 'markerfacecolor', 'b')
plt.setp(baseline, 'color', 'r', 'linewidth', 1, 'alpha', 0.3)
plt.xlabel('Index')
plt.ylabel('Amplitude')
plt.show()
```

### Hyper-parameter tuning

+ Key component in Lasso is the proper guess/tuning of the hyper-parameter $\lambda$.
+ Let's see how it works in practice

```python
import math

# Generate sensing matrix
A = (1 / math.sqrt(n)) * np.random.randn(n, p)

# Observation model
y = A @ x_star

# Compute largest singular value of A
u, s, v = svds(A, k=1)
eta = 1/s**2
print(eta)
```

    [0.19089884]

### Projection onto the $\ell_1$ norm ball

+ We first need to figure out how the projection onto the $\ell_1$-norm ball can be efficiently completed. Is it possible? Or should we depend on approximation schemes?

+ It turns out that the projection onto the $\ell_1$ norm ball is efficiently and accurately computed as:

1. Sort input vector $x$ into $\mu$: $\mu_1 \geq \mu_2 \geq \cdots \geq \mu_p$.

2. Find $\rho = \max \left\{ j \in [p]~:~ \mu_j - \tfrac{1}{j} \left( \sum_{r = 1}^j \mu_r - \lambda \right) > 0 \right\}$

3. Define $\theta = \tfrac{1}{\rho} \left( \sum_{i=1}^\rho \mu_i - \lambda\right)$

4. Output $w$ such that $w_i = \max\{ x_i - \theta, 0 \}$

```python
from numpy import linalg as la

# Projection onto l1 norm ball
def l1normProj(x, l):
    assert l > 0, "Radius lambda must be strictly positive (%d <= 0)" % l
    n, = x.shape

    u = np.abs(x)                               # compute the vector of absolute values

    if u.sum() <= l:                            # check if v is already a solution
        return x

    u = np.sort(x)[::-1]                        # get the array of cumulative sums of a sorted (decreasing) copy of v
    cssv = np.cumsum(u)
    rho = np.nonzero(u * np.arange(1, n+1) > (cssv - l))[0][-1]   # get the number of > 0 components of the optimal solution
    theta = (float(cssv[rho] - l) / rho).clip(min=0)                 # compute the projection by thresholding v using theta          # compute the Lagrange multiplier associated to the simplex constraint
    w = (np.abs(x) - theta).clip(min=0)                 # compute the projection by thresholding v using theta
    w *= np.sign(x)

    return w

# Returns the value of the objecive function
def f(y, A, x):
    return 0.5 * math.pow(la.norm(y - A @ x, 2), 2)
```

```python
def Lasso(y, A, l, eta, iters, epsilon, verbose, x_star):
    p = A.shape[1]   # Length of original signal
    n = A.shape[0]   # Length of measurement vector

    x_new = np.zeros(p)    # Initial estimate
    At = np.transpose(A)   # Transpose of A

    x_list, f_list = [1], [f(y, A, x_new)]

    for i in range(iters):
        x_old = x_new

        # Compute gradient
        grad = -At @ (y - A @ x_new)

        # Perform gradient step
        x_temp = x_old - eta * grad

        # Perform hard thresholding step
        x_new = l1normProj(x_temp, l)

        if (la.norm(x_new - x_old, 2) / la.norm(x_new, 2)) < epsilon:
            break

        # Keep track of solutions and objective values
        x_list.append(la.norm(x_new - x_star, 2))
        f_list.append(f(y, A, x_new))

        if verbose:
            print("iter# = "+ str(i) + ", ||x_new - x_old||_2 = " + str(la.norm(x_new - x_old, 2)))

    print("Number of steps:", len(f_list))
    return x_new, x_list, f_list

```

```python
# Run algorithm
epsilon = 1e-6                # Precision parameter
iters = 1000

l = la.norm(x_star, 1)
l = 2

x_Lasso, x_list, f_list = Lasso(y, A, l, eta, iters, epsilon, True, x_star)

# Plot
plt.rc('text', usetex=True)
plt.rc('font', family='serif')

xs = range(p)
markerline, stemlines, baseline = plt.stem(xs, x_Lasso, '-.x')
plt.setp(markerline, 'alpha', 0.3, 'ms', 6)
plt.setp(markerline, 'markerfacecolor', 'b')
plt.setp(baseline, 'linewidth', 1, 'alpha', 0.3)
plt.xlabel('Index')
plt.ylabel('Amplitude')
#plt.title(r"$\|x^\star - \widehat{x}\|_2 = %s$" %(la.norm(x_star - x_IHT, 2)), fontsize=16)

# Make room for the ridiculously large title.
plt.subplots_adjust(top=0.8)
plt.show()
print(l)
```

```python
# Plot
plt.rc('text', usetex=True)
plt.rc('font', family='serif')

xs = range(len(x_list))
plt.plot(xs, x_list, '-o', color = '#3399FF', linewidth = 4, alpha = 0.7, markerfacecolor = 'b')
plt.yscale('log')
plt.xlabel('Iterations')
plt.ylabel(r"$\|x^\star - \widehat{x}\|_2$")

# Make room for the ridiculously large title.
plt.subplots_adjust(top=0.8)
plt.show()
```

### Logistic regression (based on Aurélien Geron's post)

Let us first re-define the problem of logistic regression. Similar to linear regression, we are interested in finding a vector $\theta$ in $p$ dimensions that linearly interacts with the input features $x_i$:

$$
\theta^\top x_i
$$

However, in contrast to linear regression where we compare the output $\theta^\top x_i$ with the known response $y_i$ via the $\ell_2$-norm cost function:

$$
\left(y_i - \theta^\top x_i\right)^2,
$$

in logistic regression, we first pass the inner product via the sigmoid function $\sigma(\cdot)$:

$$
\sigma(\theta^\top x_i).
$$

Before, we proceed let's recall how a sigmoid function looks like:

```python
t = np.linspace(-10, 10, 100)
sig = 1 / (1 + np.exp(-t))
plt.figure(figsize=(9, 3))
plt.plot([-10, 10], [0, 0], "k-")
plt.plot([-10, 10], [0.5, 0.5], "k:")
plt.plot([-10, 10], [1, 1], "k:")
plt.plot([0, 0], [-1.1, 1.1], "k-")
plt.plot(t, sig, "b-", linewidth=2, label=r"$\sigma(t) = \frac{1}{1 + e^{-t}}$")
plt.xlabel("t")
plt.legend(loc="upper left", fontsize=20)
plt.axis([-10, 10, -0.1, 1.1])
# save_fig("logistic_function_plot")
plt.show()
```

I.e., the sigmoid function is a function that takes values in the real line and squeezes them between the values $[0, 1]$. Thus, it works as a mapping onto probabilities: if the input argument is positive then, we get values towards 1; if the input argument is negative, we get values towards 0.

Thus, the logistic/sigmoid function works perfectly (at least in intuition) for binary classification: if in a way we can find a way to map input $x_i$ into a single number, which in sequence will be "classified" by the sigmoid function, then we can build a classifier.

This is the notion of logistic regression:

$$
\hat{y}_i := h_{\theta}(x_i) = \sigma(\theta^\top x_i)
$$

where:

$$
\begin{cases}
\text{class } 0, & \text{ if } \hat{y}_i < 0.5, \\
\text{class } 1, & \text{ if } \hat{y}_i \geq 0.5,
\end{cases}
$$

But, how we train such a model? We want to penalize cases where, when the true $y = 0$, the estimated $\hat{y}$ should not be close to 1; and vice versa, when the true $y = 1$, we want the estimated $\hat{y}$ not be close to 0. This is well-represented via the following cost function:

$$
c(\theta) = \begin{cases}
-\log(\hat{y}),     &\text{ if } y = 1, \\
-\log(1 - \hat{y}), &\text{ if } y = 0.
\end{cases}
$$

Taking the maximum likelihood estimation approach (out of the scope of this course, but should be described in an ML course), we obtain the cost function, over $n$ samples:

$$
\mathcal{L}(\theta) = -\tfrac{1}{n} \sum_{i = 1}^n \left(y_i \cdot \log(\hat{y}_i) + (1 - y_i) \log (1 - \hat{y}_i) \right),
$$

where $\theta$, our parameters to optimize on, are hidden in $\hat{y}_i$.

It turns out that the logistic regression objective is convex, so one can rely on convex optimization ideas in order to solve the problem. See the Notes of the course for a description of the above procedure (with probably different notation on purpose), and how we compute gradients and Hessians. Then, one can use Gradient descent to solve the problem.

#### Example with real dataset

```python
from sklearn import datasets
iris = datasets.load_iris()
print(iris.DESCR)
```

    Iris Plants Database
    ====================

    Notes
    -----
    Data Set Characteristics:
        :Number of Instances: 150 (50 in each of three classes)
        :Number of Attributes: 4 numeric, predictive attributes and the class
        :Attribute Information:
            - sepal length in cm
            - sepal width in cm
            - petal length in cm
            - petal width in cm
            - class:
                    - Iris-Setosa
                    - Iris-Versicolour
                    - Iris-Virginica
        :Summary Statistics:

        ============== ==== ==== ======= ===== ====================
                        Min  Max   Mean    SD   Class Correlation
        ============== ==== ==== ======= ===== ====================
        sepal length:   4.3  7.9   5.84   0.83    0.7826
        sepal width:    2.0  4.4   3.05   0.43   -0.4194
        petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
        petal width:    0.1  2.5   1.20  0.76     0.9565  (high!)
        ============== ==== ==== ======= ===== ====================

        :Missing Attribute Values: None
        :Class Distribution: 33.3% for each of 3 classes.
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    This is a copy of UCI ML iris datasets.
    http://archive.ics.uci.edu/ml/datasets/Iris

    The famous Iris database, first used by Sir R.A Fisher

    This is perhaps the best known database to be found in the
    pattern recognition literature.  Fisher's paper is a classic in the field and
    is referenced frequently to this day.  (See Duda & Hart, for example.)  The
    data set contains 3 classes of 50 instances each, where each class refers to a
    type of iris plant.  One class is linearly separable from the other 2; the
    latter are NOT linearly separable from each other.

    References
    ----------
       - Fisher,R.A. "The use of multiple measurements in taxonomic problems"
         Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
         Mathematical Statistics" (John Wiley, NY, 1950).
       - Duda,R.O., & Hart,P.E. (1973) Pattern Classification and Scene Analysis.
         (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
       - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
         Structure and Classification Rule for Recognition in Partially Exposed
         Environments".  IEEE Transactions on Pattern Analysis and Machine
         Intelligence, Vol. PAMI-2, No. 1, 67-71.
       - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
         on Information Theory, May 1972, 431-433.
       - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
         conceptual clustering system finds 3 classes in the data.
       - Many, many more ...

```python
X = iris["data"][:, 3:]  # petal width
y = (iris["target"] == 2).astype(np.int)  # 1 if Iris-Virginica, else 0

from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression(solver="liblinear", random_state=42)

# {‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’}

log_reg.fit(X, y)

X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
y_proba = log_reg.predict_proba(X_new)
decision_boundary = X_new[y_proba[:, 1] >= 0.5][0]

plt.figure(figsize=(8, 3))
plt.plot(X[y==0], y[y==0], "bs")
plt.plot(X[y==1], y[y==1], "g^")
plt.plot([decision_boundary, decision_boundary], [-1, 2], "k:", linewidth=2)
plt.plot(X_new, y_proba[:, 1], "g-", linewidth=2, label="Iris-Virginica")
plt.plot(X_new, y_proba[:, 0], "b--", linewidth=2, label="Not Iris-Virginica")
plt.text(decision_boundary+0.02, 0.15, "Decision  boundary", fontsize=14, color="k", ha="center")
plt.arrow(decision_boundary, 0.08, -0.3, 0, head_width=0.05, head_length=0.1, fc='b', ec='b')
plt.arrow(decision_boundary, 0.92, 0.3, 0, head_width=0.05, head_length=0.1, fc='g', ec='g')
plt.xlabel("Petal width (cm)", fontsize=14)
plt.ylabel("Probability", fontsize=14)
plt.legend(loc="center left", fontsize=14)
plt.axis([0, 3, -0.02, 1.02])
# save_fig("logistic_regression_plot")
plt.show()
```

```python
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = (iris["target"] == 2).astype(np.int)

log_reg = LogisticRegression(solver="liblinear", C=10**10, random_state=42)
log_reg.fit(X, y)

x0, x1 = np.meshgrid(
        np.linspace(2.9, 7, 500).reshape(-1, 1),
        np.linspace(0.8, 2.7, 200).reshape(-1, 1),
    )
X_new = np.c_[x0.ravel(), x1.ravel()]

y_proba = log_reg.predict_proba(X_new)

plt.figure(figsize=(10, 4))
plt.plot(X[y==0, 0], X[y==0, 1], "bs")
plt.plot(X[y==1, 0], X[y==1, 1], "g^")

zz = y_proba[:, 1].reshape(x0.shape)
contour = plt.contour(x0, x1, zz, cmap=plt.cm.brg)

left_right = np.array([2.9, 7])
boundary = -(log_reg.coef_[0][0] * left_right + log_reg.intercept_[0]) / log_reg.coef_[0][1]

plt.clabel(contour, inline=1, fontsize=12)
plt.plot(left_right, boundary, "k--", linewidth=3)
plt.text(3.5, 1.5, "Not Iris-Virginica", fontsize=14, color="b", ha="center")
plt.text(6.5, 2.3, "Iris-Virginica", fontsize=14, color="g", ha="center")
plt.xlabel("Petal length", fontsize=14)
plt.ylabel("Petal width", fontsize=14)
plt.axis([2.9, 7, 0.8, 2.7])
#save_fig("logistic_regression_contour_plot")
plt.show()
```

(It is a cool exercise to design your own GD solver for multiclass / softmax regression)