Table of Contents

1  Pooling coefficients measured in saline

1.1  Measure the amplitude of the sine signal on each site under different conditions

1.2  Compute the pooling coefficients

1.3  Identify sites with defective switches

1.4  Plot the pooling coefficients at the good sites

2  Electrical characteristics measured in saline

2.1  Load raw Neuropixel recordings and compute power spectra

2.2  Fit a 4-parameter circuit model to the noise spectra

2.3  Examples of noise measurements and fits from the circuit model

2.3.1  Figure 8C left

2.3.2  Figure 8C right

2.4  Noise in the highest saline, bank 1 vs bank 2

2.4.1  Figure 8D

2.5  Histograms of the parameters in the circuit model

2.5.1  Figure 8E

2.5.2  Figure 4C

3  Revision

3.1  Thermal noise

3.1.1  Figure 4B

3.2  Pooling coefficients

3.2.1  In vitro

3.2.1.1  Figure 4A

3.2.2  In vivo

3.2.2.1  Figure 4E

3.3  Biological noise

3.3.1  In the two split conditions

3.3.1.1  Figure 4D

3.3.2  Pooled, predicted from independence vs measured

3.3.2.1  Figure 4F

```python
# Imports
from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
import pickle
from scipy.optimize import minimize
import sys

module_path = 'code/'
if module_path not in sys.path:
    sys.path.append(module_path)
from MM_Plot_Utils import plot, hist

%matplotlib inline
%reload_ext autoreload
%autoreload 2
```

# Pooling coefficients measured in saline

## Measure the amplitude of the sine signal on each site under different conditions
- Neuropixels probe 2 or 3
- Frequency 400 Hz or 1000 Hz
- Gradient from bottom or top
- switch settings: split 0, split 1, pooled 01

This code is provided for reference only. The raw data are not included because too voluminous. The routine simply measures the Fourier amplitude of the signal at the frequency of stimulation and saves that to file. The output files for probe 2 are included with this repo.

```python
# # Compute peak amplitude of fourier transform for all sites from raw data
# # Load data directly from the external drive
# def getFourierAmplitude(probe,freq,grad,bank,dur=5):
#     '''
#     probe in ['2','3']
#     freq in ['400','1000']
#     grad in ['bot','top']
#     bank in ['0','1','01']
#     dur = duration in seconds
#     '''
#     file=getFileName(probe,freq,grad,bank)
#     binFullPath=Path(file)
#     meta=readMeta(binFullPath) # get metadata
#     rawData = makeMemMapRaw(binFullPath, meta)
#     sRate=SampRate(meta) # get sampling rate (in Hz)

#     tStart=10        # in seconds
#     tEnd=tStart+dur
#     m=int(freq)*(tEnd-tStart) # location of the frequency of stimulation in the fourier transform

#     fa=np.zeros(384)
#     for i in range(384):
#         chanList = [i] # channels to load
#         selectData = rawData[chanList,int(tStart*sRate):int(tEnd*sRate)]
#         if meta['typeThis'] == 'imec':
#             convData = 1e3*GainCorrectIM(selectData, chanList, meta) # apply gain correction and convert to mV
#         x=convData[0]
#         fa[i]=np.abs(np.fft.rfft(x))[m]
#     with open('Probe'+probe+'/FA_p'+probe+'_f'+freq+'_g'+grad+'_b'+bank, "wb") as f:
#         pickle.dump(fa,f)
#     return fa
```

```python
# # Analyze and save all fourier amplitudes
# for probe in ['2','3']:
#     for freq in ['400','1000']:
#         for grad in ['bot','top']:
#             for bank in ['0','1','01']:
#                 getFourierAmplitude(probe,freq,grad,bank,dur=5)
```

## Compute the pooling coefficients

We applied an oscillating electric field (1000 Hz) along the electrode array with a pair of annular electrodes (Figure 4A). From the recorded waveform we estimated the signal amplitude by the Fourier coefficient at the stimulus frequency. Two different field gradients (called A and B) yielded two sets of measurements, each in the two split modes ($U_{0,\rm{A}},U_{1,\rm{A}},U_{0,\rm{B}},U_{1,\rm{B}}$) and the pooled mode ($U_{\rm{P},\rm{A}},U_{\rm{P},\rm{B}}$). For each of the 383 wires we estimated the pooling coefficients of its two electrodes by solving

\begin{align*}\label{eq:saline_pc}
    \begin{bmatrix}
        U_{0,\textrm{A}} & U_{1,\textrm{A}}\\
        U_{0,\textrm{B}} & U_{1,\textrm{B}}
    \end{bmatrix}
    \begin{bmatrix}
        k_0\\
        k_1
    \end{bmatrix}
    =
    \begin{bmatrix}
        U_{\textrm{P},\textrm{A}}\\
        U_{\textrm{P},\textrm{B}}
    \end{bmatrix}
\end{align*}

These mixing coefficients $k_0$ and $k_1$ express the recorded amplitude $U_{\textrm{P}}$ in terms of the recorded amplitudes $U_0$ and $U_1$,

\begin{align}\label{eq:V_pool_saline}
U_{\rm{P}}=k_0 U_0 + k_1 U_1
\end{align}

```python
def FAname(probe,freq,grad,bank):
    '''
    returns the file name for fourier amplitudes
    probe in ['2','3']
    freq in ['400','1000']
    grad in ['bot','top']
    bank in ['0','1','01']
    '''
    return 'data/Probe'+probe+'/FA_p'+probe+'_f'+freq+'_g'+grad+'_b'+bank
```

```python
# compute pooling coefficients for a given probe, frequency
def getPC(probe,freq):
    with open(FAname(probe,freq,'top','0'), "rb") as f:
        dt0=pickle.load(f)
    with open(FAname(probe,freq,'top','1'), "rb") as f:
        dt1=pickle.load(f)
    with open(FAname(probe,freq,'top','01'), "rb") as f:
        dt01=pickle.load(f)
    with open(FAname(probe,freq,'bot','0'), "rb") as f:
        db0=pickle.load(f)
    with open(FAname(probe,freq,'bot','1'), "rb") as f:
        db1=pickle.load(f)
    with open(FAname(probe,freq,'bot','01'), "rb") as f:
        db01=pickle.load(f)
    sp=np.stack((dt01,db01),axis=-1) # (384,2) pooled signal with the 2 gradients
    A=np.stack((np.stack((abs(dt0),abs(dt1)),axis=-1),
                np.stack((abs(db0),abs(db1)),axis=-1)),axis=-2) # (384,2,2) split signals
    spe=np.expand_dims(sp,axis=-1) # add an empty dimension for proper multiplication
    Ainv=np.linalg.inv(A) # (384,2,2) 384 inverses, each (2,2)
    kpe=np.matmul(Ainv,spe) # multiply inverse * vector 384 times
    kf=np.squeeze(kpe) # remove the empty dimension
    return kf # (384,2) pooling coefficients
```

```python
# compute and plot all pooling coeffs
kf2=getPC('2','1000')
ax=plot([kf2[:,0],kf2[:,1]],fmts=['r-','g-'],linewidth=1,ylim=[0,1.1],
    figsize=(10,4),title='Probe 2, 1000 Hz, Pooling coefficients',legend=['bank0','bank1'],
    xlabel='Wire',ylabel='Pooling coefficient');
```

## Identify sites with defective switches
We found that this probe contained a few defective switches that did not repond correctly to the control signal. Those sites are recognized by the grossly abnormal pooling coefficients in the graph above.

Here we identify the "good" sites on Probe 2:

```python
# Mark the sites with pooling coefficients inside [0.4,0.6]
P2_OK=np.logical_and(np.logical_and(kf2[:,0]>0.4,kf2[:,0]<0.6),np.logical_and(kf2[:,1]>0.4,kf2[:,1]<0.6))
print(len(np.where(P2_OK==True)[0]),'good sites')
```

    356 good sites

## Plot the pooling coefficients at the good sites

```python
# Plot pooling coeffs at good sites
kf2=getPC('2','1000')
wire=np.arange(384)
fig,axes=plt.subplots(1,2,figsize=(8,2.5),gridspec_kw={'width_ratios':[6, 1]})
plot(wire[P2_OK],[kf2[P2_OK,0],kf2[P2_OK,1]],fmts=['r.','g.'],markersize=2,
    xlabel='Wire',ylabel='Pooling coefficient',
    legend=['bank0','bank1'],loc='upper right',xlim=[-5,389],ylim=[0.0,1.0],
    yticks=[0,0.5,1],axes=axes[0]);
hist([kf2[P2_OK,0],kf2[P2_OK,1]],bins=np.arange(0,1.1,0.02),
    orientation='horizontal',xhide=True,yhide=False,
    ylim=[0.0,1.0],yticks=[0,0.5,1],axes=axes[1]);
axes[1].tick_params(labelleft=False)
fig.subplots_adjust(wspace=0.02)
plt.savefig('figs/PoolCoeffsProbe2.pdf')
```

# Electrical characteristics measured in saline

## Load raw Neuropixel recordings and compute power spectra
This routine is included for reference only. It simply applies a power spectral analysis to each channel of the Neuropixels output.

The input to this routine is not included in this repo because too voluminous.

The output power spectra are provided.

```python
# Load raw data and compute spectrum for each site. This will not run, provided for reference only!
# def pow_all_P2(conc,bank):
#     '''
#     compute power spectra for all sites on Probe 2
#     conc=concentration
#     bank=bank
#     '''
#     binFullPath=Path(getFileNameP2(conc,bank)) # probe 2
#     meta=readMeta(binFullPath) # get metadata
#     rawData = makeMemMapRaw(binFullPath, meta)
#     sRate=SampRate(meta) # get sampling rate (in Hz)
#     tStart=10        # in seconds
#     tEnd=60
#     pa=[]
#     for i in range(384):
#         chanList = [i] # channels to load
#         selectData=rawData[chanList,int(tStart*sRate):int(tEnd*sRate)]
#         if meta['typeThis'] == 'imec':
#             convData=GainCorrectIM(selectData, chanList, meta) # apply gain correction and convert to units of V
#         x=convData[0]
#         fr,po=signal.welch(x,sRate,nperseg=4096) # set of frequencies and power spectral density
#         pa+=[po]
#     with open('Probe2/NoiseSpectrum_p2_c'+conc+'_b'+bank, "wb") as f:
#         pickle.dump((fr,pa),f)
```

```python
# # compute spectra for all concentrations and banks, one file each
# for conc in ['-3','-2','-1','0','1']: # log_10 of saline concentration
#     for bank in ['0','1','01']: # split bank 0, split bank 1, pooled
#         pow_all_P2(conc,bank)
```

## Fit a 4-parameter circuit model to the noise spectra

The goal here is to infer the values of the components in the above circuit from the RMS noise measurements observed at the output ($U$) under the different conditions: 5 saline concentrations and 3 switch settings.

The assumptions are:
- The source of the noise is thermal noise in $R_0$ and $R_1$ and electronic noise $N_\rm{amp}$.
- $R_0$ and $R_1$ vary in inverse proportion to the saline concentration.
- $C_0$ and $C_1$ are such that at normal saline the electrode impedance is the spec value of 170 kohm.
- $R_\rm{S}$ is infinite, i.e. the shunt impedance is entirely capacitive.

With these assumptions there are only 4 free parameters of the model: $R_0$ and $R_1$ in normal saline, $C_\rm{S}$ and $N_\rm{amp}$. Those 4 parameters are optimized to fit the 15 noise measurements by minimizing the squared error.

```python
# Some useful global numbers
c=np.array([0.001,0.01,0.1,1,10]) # concentrations
fLo=300 # low end of frequency band to be fit, in Hz
fHi=10000 # high end of frequency band to be fit, in Hz
re=1.5e5 # spec impedance of each electrode at f = 1 kHz, in ohm
om1kHz=2*np.pi*1e3 # radial frequency at f = 1 kHz, in s^-1
kT=4.1e-21 # kT in J
```

```python
# Load noise power spectra for probe 2 and integrate to get RMS noise
no=np.zeros((5,3,384)) # 5 concentrations, 3 switch conditions, 384 sites
for i,conc in enumerate(['-3','-2','-1','0','1']):
    for j,bank in enumerate(['0','1','01']):
        with open('data/Probe2/NoiseSpectrum_p2_c'+conc+'_b'+bank, "rb") as f: # load spectra from files
            fd,p=pickle.load(f)
        p=np.array(p)
        no[i,j]=np.sqrt(np.sum(p[:,np.logical_and(fd>=fLo,fd<fHi)],axis=-1)*(fd[1]-fd[0])) # integrate over the AP band to get RMS noise
```

```python
# More useful global numbers
fr=fd[np.logical_and(fd>=fLo,fd<fHi)] # frequencies where to simulate the power (same as in the measured spectra)
om=2*np.pi*fr # corresponding radial frequencies
df=fd[1]-fd[0] # delta f between frequencies
```

```python
def par(x,y):
    '''
    return impedance of parallel impedances x and y
    '''
    return x*y/(x+y)
```

```python
def noi6(r0,r1,cs,c0,c1):
    '''
    Simulates the 5-component circuit for one set of the parameters
    Computes the Johnson noise for the two split modes and the pooled mode
    Inputs in ohm and F
    Outputs in V
    '''
    z0=r0+1/(1j*om*c0) # complex impedance electrode 0
    z1=r1+1/(1j*om*c1) # complex impedance electrode 1
    zs=1/(1j*om*cs) # complex impedance shunt
    k00=1/(1+z0/zs) # gain for input to R0 in split 0 recording
    k11=1/(1+z1/zs) # gain for input to R1 in split 1 recording
    k0p=1/(1+z0/par(z1,zs)) # gain for input to R0 in pooled recording
    k1p=1/(1+z1/par(z0,zs)) # gain for input to R1 in pooled recording
    v0=np.sqrt(sum(4*kT*(r0*abs(k00)**2))*df) # integrate Johnson noise spectrum in mode split 0
    v1=np.sqrt(sum(4*kT*(r1*abs(k11)**2))*df) # integrate Johnson noise spectrum in mode split 1
    v01=np.sqrt(sum(4*kT*(r0*abs(k0p)**2+r1*abs(k1p)**2))*df) # integrate Johnson noise spectrum in pooled mode
    return v0,v1,v01 # return the RMS noise for each mode

def eval6(x):
    '''
    Evaluates a 4-parameter circuit model.
    Computes the RMS noise in the two split modes and in pooled mode for all saline concentrations.
    x = [r0,r1,cs,vc] = the model parameters
    Input: resistors in Mohm, capacitors in pF, common noise in uV
    Output: 5x3 array with RMS noise in V vs concentration and mode
    '''
    r0=x[0]*1e6 # R_0 at concn = 1, scale to SI units
    r1=x[1]*1e6 # R_1 at concn = 1
    cs=x[2]*1e-12 # C_s, the shunt capacitance
    vc=x[3]*1e-6 # N_amp, the electronic noise
    c0=1/om1kHz/np.sqrt(re**2-r0**2) # choose C0 in F to make R_ele0 = re
    c1=1/om1kHz/np.sqrt(re**2-r1**2) # choose C1 in F to make R_ele1 = re
    v=np.zeros((5,3))
    for i,co in enumerate(c): # for each concentration
        r0c=r0/co # compute the resistors
        r1c=r1/co
        v[i]=noi6(r0c,r1c,cs,c0,c1) # get the Johnson noise
        v[i]=np.sqrt(v[i]**2+vc**2) # add the electronic noise in quadrature
    return v # return one (5,3) array of RMS noise

def fit6(x,j):
    '''
    Returns the squared error between predicted and observed noise
    x = [r0,r1,cs,vc] = the model parameters
    j = site number on the array
    Input: resistors in Mohm, common noise in uV
    Output: error in uV**2
    '''
    v=eval6(x) # (5,3) array of predicted RMS noise
    err=np.sum((no[:,:,j]-v)**2) # compare to measured noise
    return err*1e12 # error in µV^2
```

```python
# Perform 4-parameter circuit fits to all 384 sites. This takes a little while to run...
# [r0,r1,cs,vc] = the model parameters
# r in Mohm, c in pF, vc in uV
par6=np.zeros((384,4)) # store best-fit parameters in this array
err6=np.zeros(384) # store errors
for j in range(384):
    res=minimize(fun=fit6, x0=(0.01,0.01,8,6.0), # minimize squared error
             args=(j), method = 'L-BFGS-B',jac=None, hess=None, hessp=None,
             bounds=[(1e-8,re/1e6-1e-8),(1e-8,re/1e6-1e-8),(1e-8,None),(1e-8,None)],
             constraints=(), tol=None, callback=None, options=None)
    par6[j]=res.x
    err6[j]=res.fun
    print(j,end=',')
with open('data/Probe2/2R1Crms_P2', 'wb') as f: # save parameters and fit errors to file
        pickle.dump((par6,err6),f)
```

    0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,

## Examples of noise measurements and fits from the circuit model

```python
# Load 2R1C circuit parameters for Probe 2
with open('data/Probe2/2R1Crms_P2', 'rb') as f:
    pa,er=pickle.load(f)
```

```python
# Load data for rms noise probe 2
no=np.zeros((5,3,384))
for i,conc in enumerate(['-3','-2','-1','0','1']):
    for j,bank in enumerate(['0','1','01']):
        with open('data/Probe2/NoiseSpectrum_p2_c'+conc+'_b'+bank, "rb") as f:
            fd,p=pickle.load(f)
        p=np.array(p)
        no[i,j]=np.sqrt(np.sum(p[:,np.logical_and(fd>=fLo,fd<fHi)],axis=-1)*(fd[1]-fd[0])) # rms noise
fr=fd[np.logical_and(fd>=fLo,fd<fHi)] # frequencies where to evaluate the power
om=2*np.pi*fr
df=fd[1]-fd[0]
```

```python
def plotFitFig(j):
    '''
    Plots RMS noise vs saline concentration for site number j, in the 2 split conditions and pooled
    ALso plots predictions from the 4-parameter circuit model for that site
    assumes no[:,:,j] = RMS noise data, pa[j] = parameters, and eval6(pa[j]) = the fit, c = concentrations
    '''
    fdata=['ro','g+','bs']
    ffit=['r-','g-','b-']
    td=no[:,:,j]*1e6
    tf=eval6(pa[j])*1e6
    ax=plot(c,[td[:,0],td[:,1],td[:,2]],fmts=fdata,xscale='log',figsize=(3,3),fillstyle='none');
    plot(c,[tf[:,0],tf[:,1],tf[:,2]],fmts=ffit,xscale='log',axes=ax,
        xlabel='Saline concentration',ylabel='RMS noise (µV)',
        legend=['split 0','split 1','pooled'],xticks=[1e-3,1e-2,1e-1,1,10],
        ylim=[0,25]);
    plt.minorticks_off()
    plt.savefig('figs/PowerVsConc{:03d}.pdf'.format(j))
```

### Figure 8C left

```python
plotFitFig(19)
```

### Figure 8C right

```python
plotFitFig(353)
```

## Noise in the highest saline, bank 1 vs bank 2

### Figure 8D

```python
# Noise in 10x saline split 1 vs split 0. Limit to sites with OK switches.
ax=plot(no[4,0,P2_OK]*1e6,no[4,2,P2_OK]*1e6,fmts=['bo'],markersize=2,equal=True);
x=[0,15]
plot(x,x,fmts=['k:'],linewidth=1,axes=ax,
     xlabel='Split 0 RMS noise (µV)',ylabel='Split 1 RMS noise (µV)',xlim=[0,15],ylim=[0,15],
    xticks=[0,5,10,15],yticks=[0,5,10,15]);
plt.savefig('figs/Noise10xSaline.pdf')
```

## Histograms of the parameters in the circuit model

Histogram the 4 parameters of the circuit model for Probe 2.

Limit this to sites with
- functioning switches and
- a good fit with small residual error.

### Figure 8E

```python
# Probe 2, rms noise, 2R1C circuit model, sites with err<4 and OK switches
with open('data/Probe2/2R1Crms_P2', 'rb') as f:
    p,e=pickle.load(f)
OK=np.logical_and(P2_OK,e<4)
print('Histograms over {} sites'.format(len(np.where(OK==True)[0])))
hist(p[OK,0]*1000,bins=np.arange(100)*0.3,xlabel='$R_0$ (kohm)',color='red',xlim=[0,25],figsize=(2,2),yhide=True);
plt.savefig('figs/P2_R0_Hist.pdf')
plt.show()
hist(p[OK,1]*1000,bins=np.arange(100)*0.3,xlabel='$R_1$ (kohm)',color='green',xlim=[0,25],figsize=(2,2),yhide=True);
plt.savefig('figs/P2_R1_Hist.pdf')
plt.show()
hist(p[OK,2],bins=np.arange(100)*0.15,xlabel='$C_{\mathrm{s}}$ (pF)',color='blue',xlim=[0,15],figsize=(2,2),
     yhide=True,xticks=[0,5,10,15]);
plt.savefig('figs/P2_Cs_Hist.pdf')
plt.show()
hist(p[OK,3],bins=np.arange(100)*0.1,xlabel='$N_{\mathrm{amp}}$ (µV)',color='magenta',xlim=[0,15],figsize=(2,2),yhide=True);
plt.savefig('figs/P2_Na_Hist.pdf')
plt.show()
```

### Figure 4C

```python
hist(p[OK,3],bins=np.arange(150)*0.1,xlabel='$N_{\mathrm{amp}}$ (µV)',color='magenta',xlim=[0,15],figsize=(2,2),yhide=True);
plt.savefig('figs/P2_Na_Hist2.pdf')
plt.show()
```

```python
np.max(p[OK,3])
```

    14.358126381876733

# Revision

## Thermal noise
Use the measured bath resistors to compute the resulting noise from thermal noise alone. Do in split 0, split 1, and pool condition. Report the results of split 0 and split 1

```python
# Probe 2, rms noise, 2R1C circuit model, sites with err<4 and OK switches
# [r0,r1,cs,vc] = the model parameters
# r in Mohm, c in pF, vc in uV
with open('data/Probe2/2R1Crms_P2', 'rb') as f: # load circuit parameters for this probe
    p,e=pickle.load(f)
OK=np.logical_and(P2_OK,e<4) # "good" sites

fLo=300 # low end of frequency band to be fit, in Hz
fHi=10000 # high end of frequency band to be fit, in Hz
re=1.5e5 # spec impedance of each electrode at f = 1 kHz, in ohm
om1kHz=2*np.pi*1e3 # radial frequency at f = 1 kHz, in s^-1
kT=4.1e-21 # kT in J

fr=np.arange(fLo,fHi+10,10) # range of frequencies with 10 Hz sampling
om=2*np.pi*fr # corresponding radial frequencies
df=fr[1]-fr[0] # delta f between frequencies

r0=p[:,0]*1e6 # R_0 at concn = 1, scale to SI units
r1=p[:,1]*1e6 # R_1 at concn = 1
cs=p[:,2]*1e-12 # C_s, the shunt capacitance
vc=p[:,3]*1e-6 # V_c, the common noise voltage
c0=1/om1kHz/np.sqrt(re**2-r0**2) # choose C0 in F to make R_ele0 = re
c1=1/om1kHz/np.sqrt(re**2-r1**2) # choose C1 in F to make R_ele1 = re

no=np.array([noi6(*x) for x in zip(r0,r1,cs,c0,c1)]) # compute noise [split0, split1, pool]
```

### Figure 4B

```python
# histogram thermal noise, separately on banks 0 and 1
hist([no[OK,0]*1e6,no[OK,1]*1e6],
     bins=np.arange(100)*0.02,
     xlabel=r'$N_{\rm{the}}$ (µV)',
     color=['red','green'],
     xlim=[0,2],
     figsize=(2,2),yhide=True,
#      legend=['bank 0','bank 1'],
#      loc='upper left',
);
plt.savefig('figs/P2_Nthe_Hist.pdf')
```

```python
print('Thermal noise = {:.2f} ± {:.2f} µV'.format(
    np.mean(np.concatenate((no[OK,0],no[OK,1]))*1e6),
    np.std(np.concatenate((no[OK,0],no[OK,1]))*1e6)))
```

    Thermal noise = 1.45 ± 0.10 µV

```python
# histogram thermal noise, combine banks 0 and 1
hist(np.concatenate((no[OK,0],no[OK,1]))*1e6,bins=np.arange(100)*0.02,xlabel=r'$N_{\rm{the}}$ (µV)',
     color=['red'],xlim=[0,2],figsize=(3,2),yhide=True,
    loc='upper left');
plt.savefig('figs/P2_Nthe_Hist_Both.pdf')
```

## Pooling coefficients

### In vitro

#### Figure 4A

```python
# Plot pooling coeffs at good sites
kf2=getPC('2','1000')
hist([kf2[P2_OK,0],kf2[P2_OK,1]],bins=np.arange(0,1.1,0.02),yhide=True,
    xticks=[0,0.5,1],xlim=[0,1],figsize=(2.8,1.95),legend=['bank 0','bank 1'],loc='upper left',
    xlabel=r'$c_{\rm{0}},c_{\rm{1}}$');
plt.savefig('figs/P2_PC_Hist.pdf')
```

### In vivo

```python
# Plot pooling coeffs at good sites
wi=np.genfromtxt('data/Invivo/channels_with_signal_on_both_banks.csv',delimiter=',',dtype='int')
cv0=np.genfromtxt('data/Invivo/invivo_bank0_pooling_coeff.csv',delimiter=',')
cv1=np.genfromtxt('data/Invivo/invivo_bank1_pooling_coeff.csv',delimiter=',')
cs0=np.genfromtxt('data/Invivo/saline_bank0_pooling_coeff.csv',delimiter=',')
cs1=np.genfromtxt('data/Invivo/saline_bank1_pooling_coeff.csv',delimiter=',')
```

#### Figure 4E

```python
fig,ax=plt.subplots(2,2,
                    figsize=(4.2,4.2),
                    gridspec_kw={'width_ratios':[3,1],'height_ratios':[1,3]},
#                     sharex='col',
#                     sharey='row',
)
plot([cs0,cs1,[0,1]],[cv0,cv1,[0,1]],
     fmts=['r.','g.','k:'],
     linewidth=1,
#      equal=True,
     xlim=[0,1],
     ylim=[0,1],
     xticks=[0,0.5,1],
     yticks=[0,0.5,1],
     xlabel=r'$c_{\rm{0}},c_{\rm{1}}$ in saline',
     ylabel=r'$c_{\rm{0}},c_{\rm{1}}$ in brain',
#      legend=['Bank 0','Bank 1'],
     axes=ax[1,0],
);
hist([cs0,cs1],
     bins=np.arange(0,1.025,0.025),
     yhide=True,
     xlim=[0,1],
     xticks=[0,0.5,1],
     axes=ax[0,0]);
ax[0,0].tick_params(labelbottom=False)

hist([cv0,cv1],
     bins=np.arange(0,1.025,0.025),
     xhide=True,
     ylim=[0,1],
     yticks=[0,0.5,1],
     orientation='horizontal',
     axes=ax[1,1]);
ax[1,1].tick_params(labelleft=False)

ax[0,1].axis('off')
fig.subplots_adjust(wspace=0.05,hspace=0.05)
# fig.tight_layout()
plt.savefig('figs/PCsInvivoVsSaline.pdf')

```

## Biological noise

```python
nb0=np.genfromtxt('data/Invivo/bank0_invivo_noise.csv',delimiter=',')
nb1=np.genfromtxt('data/Invivo/bank1_invivo_noise.csv',delimiter=',')
nb01=np.genfromtxt('data/Invivo/bank01_invivo_noise.csv',delimiter=',')
np01=np.genfromtxt('data/Invivo/bank01_invivo_noise_predicted.csv',delimiter=',')
```

### In the two split conditions

#### Figure 4D

```python
hist([nb0,nb1],
#      legend=['bank 0','bank 1'],
     figsize=(2.5,2),xlabel=r'$N_{\rm{bio}}$ split (µV)',
     xlim=[0,22],
    yhide=True,
);
plt.savefig('figs/Nbio_Hist.pdf')
```

### Pooled, predicted from independence vs measured

#### Figure 4F

```python
x=np.array([5,15])
y=x*np.sqrt(2)
ax=plot([x,x],[x,y],
    fmts=['k:','r:'],
    linewidth=1,
    figsize=(4,4),
);
plot(np01,nb01,
        fmts=['b.'],
        xlim=[5,16],
        ylim=[5,16],
        xlabel=r'$N_{\rm{bio}}$ pooled predicted (µV)',
        ylabel=r'$N_{\rm{bio}}$ pooled measured (µV)',
        legend=['1 x',r'$\sqrt{2}$ x'],
        equal=True,
        axes=ax,
);
plt.savefig('figs/Nbio_Pred.pdf')
```

```python

```


```python
# weights can be downloaded from
# https://github.com/duanzhiihao/RAPiD/releases/download/v0.1/pL1_MWHB608_Mar11_4500.ckpt
detector = Detector(model_name='rapid',
                    backbone='yolov5n',
                    weights_path='./weights/rapid_pL1_yolov5n_COCO608_Dec20_60000.ckpt',
                    use_cuda=False)
```

    Using backbone yolov5s. Loading COCO pre-trained weights...

                     from  n    params  module                                  arguments
      0                -1  1      1760  models.backbones.Conv                   [3, 16, 6, 2, 2]
      1                -1  1      4672  models.backbones.Conv                   [16, 32, 3, 2]
      2                -1  1      4800  models.backbones.C3                     [32, 32, 1]
      3                -1  1     18560  models.backbones.Conv                   [32, 64, 3, 2]
      4                -1  2     29184  models.backbones.C3                     [64, 64, 2]
      5                -1  1     73984  models.backbones.Conv                   [64, 128, 3, 2]
      6                -1  3    156928  models.backbones.C3                     [128, 128, 3]
      7                -1  1    295424  models.backbones.Conv                   [128, 256, 3, 2]
      8                -1  1    296448  models.backbones.C3                     [256, 256, 1]
      9                -1  1    164608  models.backbones.SPPF                   [256, 256, 5]
     10                -1  1     33024  models.backbones.Conv                   [256, 128, 1, 1]
     11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
     12           [-1, 6]  1         0  models.backbones.Concat                 [1]
     13                -1  1     90880  models.backbones.C3                     [256, 128, 1, False]
     14                -1  1      8320  models.backbones.Conv                   [128, 64, 1, 1]
     15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
     16           [-1, 4]  1         0  models.backbones.Concat                 [1]
     17                -1  1     22912  models.backbones.C3                     [128, 64, 1, False]
     18                -1  1     36992  models.backbones.Conv                   [64, 64, 3, 2]
     19          [-1, 14]  1         0  models.backbones.Concat                 [1]
     20                -1  1     74496  models.backbones.C3                     [128, 128, 1, False]
     21                -1  1    147712  models.backbones.Conv                   [128, 128, 3, 2]
     22          [-1, 10]  1         0  models.backbones.Concat                 [1]
     23                -1  1    296448  models.backbones.C3                     [256, 256, 1, False]

    Number of parameters in backbone: 1757152
    Successfully initialized model rapid. Total number of trainable parameters: 2138006
    Successfully loaded weights: ./weights/rapid_pL1_yolov5n_COCO608_Dec20_60000.ckpt
    Using CPU instead of CUDA...

```python
# Single Image Detection
im = detector.detect_one(img_path='./images/COCO/val2017/000000581062.jpg', return_img=True,
                         input_size=608, conf_thres=0.3, test_aug=None, visualize=False)

plt.figure(figsize=(12,12))
plt.imshow(im)
plt.show()
```

```python
import sympy as sp
x
```

```python

```


```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))
from formats import load_style
load_style(plot_style = False)
```

```python
os.chdir(path)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. a ipython magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, roc_curve, auc
from sklearn.metrics import precision_recall_curve, average_precision_score

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    Ethen 2017-07-31 14:28:35

    CPython 3.5.2
    IPython 5.4.1

    numpy 1.13.1
    pandas 0.20.2
    matplotlib 2.0.2
    sklearn 0.18.1

# ROC/AUC for Binary Classification

For this documentation, we'll be working with a human resource dataset. Our goal is to find out the employees that are likely to leave in the future and act upon our findings, i.e. retain them before they choose to leave. This dataset contains 12000 observations and 7 variables, each representing :

- `S` The satisfaction level on a scale of 0 to 1
- `LPE` Last project evaluation by a client on a scale of 0 to 1
- `NP` Represents the number of projects worked on by employee in the last 12 month
- `ANH` Average number of hours worked in the last 12 month for that employee
- `TIC` Amount of time the employee spent in the company, measured in years
- `Newborn` This variable will take the value 1 if the employee had a newborn within the last 12 month and 0 otherwise
- `left` 1 if the employee left the company, 0 if they’re still working here. This is our response variable

```python
filename = 'HR.csv'
data = pd.read_csv(filename)
print('dimensions: ', data.shape)
data.head()
```

    dimensions:  (12000, 7)

      S
      LPE
      NP
      ANH
      TIC
      Newborn
      left

      0
      0.38
      0.53
      2
      157
      3
      0
      1

      1
      0.80
      0.86
      5
      262
      6
      0
      1

      2
      0.11
      0.88
      7
      272
      4
      0
      1

      3
      0.72
      0.87
      5
      223
      5
      0
      1

      4
      0.37
      0.52
      2
      159
      3
      0
      1

To train and evaluate the model, we’ll perform a simple train/test split. 80 percent of the dataset will be used to actually train the model, while the rest will be used to evaluate the accuracy of this model, i.e. out of sample error. Note that the best practice is to split it in three ways train/validation/test split.

```python
label_col = 'left'
label = data[label_col].values
data = data.drop(label_col, axis = 1)
print('labels distribution:', np.bincount(label) / label.size)

test_size = 0.2
random_state = 1234
data_train, data_test, y_train, y_test = train_test_split(
    data, label, test_size = test_size, random_state = random_state, stratify = label)
```

    labels distribution: [ 0.83333333  0.16666667]

This probability table tells you that around 16 percent of the employees who became a staff member of yours have left! If those employees are all the ones that are performing well in the company, then this is probabliy not a good sign. We'll leave out the exploratory analysis part to you ...

## Sklearn Transformer

We then convert perform some generic data preprocessing including standardizing the numeric columns and one-hot-encode the categorical columns (the "Newborn" variable is treated as a categorical variable) and convert everything into a numpy array that sklearn expects. This generic preprocessing step is written as a custom sklearn Transformer. You don't have to follow this structure if you prefer your way of doing it.

To roll out our own Transformer a adheres to the sklearn API, we need to

- Ensure that all arguments to the `__init__` method should be explicit: i.e. `*args` or `**kwargs` should be avoided, as they will not be correctly handled within cross-validation routines
- Subclass/Inherit [`BaseEstimator`](http://scikit-learn.org/stable/modules/generated/sklearn.base.BaseEstimator.html) to get some free stuff. It will give us class representations that are more informative when printing the class object. And provides us a `get_params` and `set_params` functions. These functionalities are used in sklearn's methods such as GridSearch and RandomSearch.
- Subclass/Inherit an appropriate class for your task (one of ClassifierMixin, RegressorMixin, ClusterMixin, TransformerMixin). In our case, we will be implementing a Transformer, thus we'll be subclassing [`TransformerMixin`](http://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html). For transformer, we need to implement a `.fit` method which fits some stuff on the training data and a `.transform` method that can perform transformation on both the training and test data. Note that we don't need to subclass [`TransformerMixin`](http://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html) this to work, but it does give the end-user the idea that this is a Transformer and we get the `.fit_transform` method that does the fitting and transformer on the training data in one shot for free
- In the fit implementation, you'll notice results that were learned during the `.fit` method is stored with a trailing underscore (e.g., self.colnames_). This is a convention used in sklearn so that we can quickly scan the members of an estimator and distinguish which members are fitting during training time

If you would like to read more on this topic. The following two link might be of interest to you.

- [Blog: Creating your own estimator in scikit-learn](http://danielhnyk.cz/creating-your-own-estimator-scikit-learn/)
- [scikit-learn Documentation: Rolling your own estimator](http://scikit-learn.org/dev/developers/contributing.html#rolling-your-own-estimator)

```python
from collections import defaultdict
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler

class Preprocess(BaseEstimator, TransformerMixin):
    """
    Generic data preprocessing including:
    - standardize numeric columns
    - one-hot encode categorical columns

    Parameters
    ----------
    num_cols : list
        Numeric column's name

    cat_cols : list
        Categorical column's name

    Attributes
    ----------
    colnames_ : list
        Column name of the transformed numpy array

    label_encode_dict_ : dict of sklearn's LabelEncoder
        LabelEncoder that was used to encode the value
        of the categorical columns into with value between
        0 and n_classes-1. Categorical columns will go through
        this encoding process before being one-hot encoded

    cat_encode_ : sklearn's OneHotEncoder
        OneHotEncoder that was used to one-hot encode the
        categorical columns

    scaler_ : sklearn's StandardScaler
        Scaler that was used to standardize the numeric columns
    """
    def __init__(self, num_cols = None, cat_cols = None):
        self.num_cols = num_cols
        self.cat_cols = cat_cols

    def fit(self, data):
        """
        Fit the Preprocess

        Parameters
        ----------
        data : DataFrame
        """
        data = data.copy()

        # Label encoding across multiple columns in scikit-learn
        # https://stackoverflow.com/questions/24458645/label-encoding-across-multiple-columns-in-scikit-learn
        if self.cat_cols is not None:
            self.label_encode_dict_ = defaultdict(LabelEncoder)
            label_encoded = (data[self.cat_cols]
                             .apply(lambda x: self.label_encode_dict_[x.name].fit_transform(x)))

            self.cat_encode_ = OneHotEncoder(sparse = False)
            self.cat_encode_.fit(label_encoded)

        if self.num_cols is not None:
            self.scaler_ = StandardScaler().fit(data[self.num_cols])

        # store the column names (numeric columns comes before the
        # categorical columns) so we can refer to them later
        if self.num_cols is not None:
            colnames = self.num_cols.copy()
        else:
            colnames = []

        if self.cat_cols is not None:
            for col in self.cat_cols:
                cat_colnames = [col + '_' + str(classes)
                                for classes in self.label_encode_dict_[col].classes_]
                colnames += cat_colnames

        self.colnames_ = colnames
        return self

    def transform(self, data):
        """
        Trasform the data using the fitted Preprocess

        Parameters
        ----------
        data : DataFrame
        """
        if self.cat_cols is not None:
            label_encoded = (data[self.cat_cols]
                             .apply(lambda x: self.label_encode_dict_[x.name].transform(x)))
            cat_encoded = self.cat_encode_.transform(label_encoded)

        if self.num_cols is not None:
            scaled = self.scaler_.transform(data[self.num_cols])

        # combine encoded categorical columns and scaled numerical
        # columns, it's the same as concatenate it along axis 1
        if self.cat_cols is not None and self.num_cols is not None:
            X = np.hstack((scaled, cat_encoded))
        elif self.num_cols is None:
            X = cat_encoded
        else:
            X = scaled

        return X
```

```python
num_cols = ['S', 'LPE', 'NP', 'ANH', 'TIC']
cat_cols = ['Newborn']

preprocess = Preprocess(num_cols, cat_cols)
X_train = preprocess.fit_transform(data_train)
X_test = preprocess.transform(data_test)

print('colnames', preprocess.colnames_)
X_train
```

    colnames ['S', 'LPE', 'NP', 'ANH', 'TIC', 'Newborn_0', 'Newborn_1']

    array([[ 0.24997745,  0.61402599,  0.16885833, ...,  0.72182646,
             1.        ,  0.        ],
           [ 0.12545568, -0.27568766,  1.02655144, ..., -0.22007024,
             1.        ,  0.        ],
           [ 0.58203549, -1.34334405,  0.16885833, ..., -0.22007024,
             1.        ,  0.        ],
           ...,
           [-1.32729826, -0.75020161, -0.68883478, ..., -0.22007024,
             1.        ,  0.        ],
           [ 0.58203549, -0.21637342, -0.68883478, ..., -1.16196694,
             0.        ,  1.        ],
           [ 0.45751372,  0.25814053, -0.68883478, ..., -0.22007024,
             1.        ,  0.        ]])

```python
# pick your favorite classfication model
tree = RandomForestClassifier(max_depth = 4)
tree.fit(X_train, y_train)
```

    RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                max_depth=4, max_features='auto', max_leaf_nodes=None,
                min_impurity_split=1e-07, min_samples_leaf=1,
                min_samples_split=2, min_weight_fraction_leaf=0.0,
                n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
                verbose=0, warm_start=False)

After training our model, we need to evaluate whether its any good or not and probably the most straightforward and intuitive metric for a supervised classifier's performance is accuracy. Unfortunately, there are circumstances where simple accuracy does not work well. For example, with a disease that only affects 1 in a million people, a completely bogus screening test that always reports "negative" will be 99.9999% accurate. Unlike accuracy, ROC curves are less sensitive to class imbalance; the bogus screening test would have an AUC of 0.5, which is like not having a test at all.

## ROC curves

**ROC curve (Receiver Operating Characteristic)** is a commonly used way to visualize the performance of a binary classifier and AUC (Area Under the ROC Curve) is used to summarize its performance in a single number. Most machine learning algorithms have the ability to produce probability scores that tells us the strength in which it thinks a given observation is positive. Turning these probability scores into yes or no predictions requires setting a threshold; cases with scores above the threshold are classified as positive, and vice versa. Different threshold values can lead to different result:

- A higher threshold is more conservative about labelling a case as positive; this makes it less likely to produce false positive (an observation that has a negative label but gets classified as positive by the model) results but more likely to miss cases that are in fact positive (lower true positive rate)
- A lower threshold produces positive labels more liberally, so it creates more false positives but also generate more true positives

A quick refresher on terminology:

\begin{align}
[\text{true positive rate}]
&= \frac{[\text{# positive data points with positive predicitions}]}{\text{[# all positive data points]}} \\
&= \frac{[\text{# true positives}]}{[\text{# true positives}] + [\text{# false negatives}]}
\end{align}

true positive rate is also known as **recall** or **sensitivity**

\begin{align}
[\text{false positive rate}]
&= \frac{[\text{# positive data points with positive predicitions}]}{\text{[# all negative data points]}} \\
&= \frac{[\text{# false positives}]}{[\text{# false positives}] + [\text{# true negatives}]}
\end{align}

The ROC curve is created by plotting the true positive rate (when it's actually a yes, how often does it predict yes?) on the y axis against the false positive rate (when it's actually a no, how often does it predict yes?) on the x axis at various cutoff settings, giving us a picture of the whole spectrum of the tradeoff we're making between the two measures.

If all these true/false positive terminology is confusing to you, consider reading the material at the following link. [Blog: Simple guide to confusion matrix terminology](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/)

### Implementation

There are commonly used packages to plot ROC curves and to compute metrics from them, but it can still be worthwhile to work through how these curves are calculated from scratch to try to understand better what they show us.

```python
def _binary_clf_curve(y_true, y_score):
    """
    Calculate true and false positives per binary classification
    threshold (can be used for roc curve or precision/recall curve);
    the calcuation makes the assumption that the positive case
    will always be labeled as 1

    Parameters
    ----------
    y_true : 1d ndarray, shape = [n_samples]
        True targets/labels of binary classification

    y_score : 1d ndarray, shape = [n_samples]
        Estimated probabilities or scores

    Returns
    -------
    tps : 1d ndarray
        True positives counts, index i records the number
        of positive samples that got assigned a
        score >= thresholds[i].
        The total number of positive samples is equal to
        tps[-1] (thus false negatives are given by tps[-1] - tps)

    fps : 1d ndarray
        False positives counts, index i records the number
        of negative samples that got assigned a
        score >= thresholds[i].
        The total number of negative samples is equal to
        fps[-1] (thus true negatives are given by fps[-1] - fps)

    thresholds : 1d ndarray
        Predicted score sorted in decreasing order

    References
    ----------
    Github: scikit-learn _binary_clf_curve
    - https://github.com/scikit-learn/scikit-learn/blob/ab93d65/sklearn/metrics/ranking.py#L263
    """

    # sort predicted scores in descending order
    # and also reorder corresponding truth values
    desc_score_indices = np.argsort(y_score)[::-1]
    y_score = y_score[desc_score_indices]
    y_true = y_true[desc_score_indices]

    # y_score typically consists of tied values. Here we extract
    # the indices associated with the distinct values. We also
    # concatenate a value for the end of the curve
    distinct_indices = np.where(np.diff(y_score))[0]
    end = np.array([y_true.size - 1])
    threshold_indices = np.hstack((distinct_indices, end))

    thresholds = y_score[threshold_indices]
    tps = np.cumsum(y_true)[threshold_indices]

    # (1 + threshold_indices) = the number of positives
    # at each index, thus number of data points minus true
    # positives = false positives
    fps = (1 + threshold_indices) - tps
    return tps, fps, thresholds
```

```python
# we'll work with some toy data so it's easier to
# show and confirm the calculated result
y_true = np.array([1, 0, 1, 0, 1])
y_score = np.array([0.45, 0.4, 0.35, 0.35, 0.8])

tps, fps, thresholds = _binary_clf_curve(y_true, y_score)
print('thresholds:', thresholds)
print('true positive count:', tps)
print('false positive count:', fps)
```

    thresholds: [ 0.8   0.45  0.4   0.35]
    true positive count: [1 2 2 3]
    false positive count: [0 0 1 2]

From the result above, we can see that the function will compute the true/false positive count for all unique threshold in the predicted score `y_score`. We can validate the result by hand to confirm that the calculation this in fact correct.

Recall that ROC curve plots that true positive rate on the y-axis and false positive rate on the x-axis. Thus all we need to do is to convert the count into rate and we have our ROC curve.

```python
# convert count to rate, append 0 to
# both true positive and false positive
# so the visualization will start from origin (0, 0)
tpr = np.hstack((0, tps / tps[-1]))
fpr = np.hstack((0, fps / fps[-1]))
print('true positive rate:', tpr)
print('false positive rate:', fpr)

plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

fig = plt.figure()
plt.plot(fpr, tpr, marker = 'o', lw = 1)
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.show()
```

Now to calculate the AUC (Area Under the Curve) for the ROC curve, we need sum up the rectangular area and the triangular area under the curve. Depicted by the visualization below:

- For the rectangular area (the plot on the left illustrates one of them), the height are the TPR (true positive rate) and widths are in the difference in the FPR (false positive rate), so the total area of all the rectangles is the dot product of TPR and FPR's differences
- For the triangluar area (the plot on the right illustrates one of them), the height are the difference in TPR (true positive rate) and widths are in the difference in the FPR (false positive rate), so the total area of all the rectangles is the dot product of both TPR's and FPR's difference. But only half the area of each rectangle is below its segment of the ROC curve, thus we divide the rectangle by 2 to obtain the triangular area

```python
import matplotlib.patches as patches

fig, ax = plt.subplots(1, 2, figsize = (10, 6))
fig.suptitle('Receiver Operator Characteristic', y = 1.02)

# this part is hard-coded for illustration purpose
fpr_diff = fpr[3] - fpr[0]
tpr_diff = tpr[3] - tpr[0]
rect1 = patches.Rectangle(xy = (0, 0), width = fpr_diff,
                          height = tpr_diff, alpha = 0.3)
ax[0].add_patch(rect1)

fpr_diff = fpr[-1] - fpr[-2]
tpr_diff = tpr[-1] - tpr[-2]
rect2 = patches.Rectangle(xy = (fpr[-2], tpr[-2]), width = fpr_diff,
                          height = tpr_diff, alpha = 0.3)
ax[1].add_patch(rect2)

for i in range(len(ax)):
    ax[i].plot(fpr, tpr, marker = 'o', lw = 1)
    ax[i].set_xlabel('false positive rate')
    ax[i].set_ylabel('true positive rate')

plt.tight_layout()
plt.show()
```

```python
def _roc_auc_score(y_true, y_score):
    """
    Compute Area Under the Curve (AUC) from prediction scores

    Parameters
    ----------
    y_true : 1d ndarray, shape = [n_samples]
        True targets/labels of binary classification

    y_score : 1d ndarray, shape = [n_samples]
        Estimated probabilities or scores

    Returns
    -------
    auc : float
    """

    # ensure the target is binary
    if np.unique(y_true).size != 2:
        raise ValueError('Only two class should be present in y_true. ROC AUC score '
                         'is not defined in that case.')

    tps, fps, _ = _binary_clf_curve(y_true, y_score)

    # convert count to rate
    tpr = tps / tps[-1]
    fpr = fps / fps[-1]

    # compute AUC using the trapezoidal rule;
    # appending an extra 0 is just to ensure the length matches
    zero = np.array([0])
    tpr_diff = np.hstack((np.diff(tpr), zero))
    fpr_diff = np.hstack((np.diff(fpr), zero))
    auc = np.dot(tpr, fpr_diff) + np.dot(tpr_diff, fpr_diff) / 2
    return auc
```

```python
auc_score = _roc_auc_score(y_true, y_score)
print('auc score:', auc_score)

# confirm with scikit-learn's result
auc_score = roc_auc_score(y_true, y_score)
print('package auc socre:', auc_score)
```

    auc score: 0.75
    package auc socre: 0.75

After working through the implementation of ROC curve and AUC score from sratch, we now pull back and visualize:

- The ROC curve of our original model
- Dotted line represents the ROC curve of a purely random classifier and a perfect classifier

```python
# calling the roc_curve, extract the probability of
# the positive class from the predicted probability
tree_test_pred = tree.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, tree_test_pred, pos_label = 1)

# AUC score that summarizes the ROC curve
roc_auc = auc(fpr, tpr)

plt.plot(fpr, tpr, lw = 2, label = 'ROC AUC: {:.2f}'.format(roc_auc))
plt.plot([0, 1], [0, 1],
         linestyle = '--',
         color = (0.6, 0.6, 0.6),
         label = 'random guessing')
plt.plot([0, 0, 1], [0, 1, 1],
         linestyle = ':',
         color = 'black',
         label = 'perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.legend(loc = "lower right")
plt.tight_layout()
plt.show()
```

The goal of visualizing the  ROC curve is to let us know how well can our classifier be expected to perform in general, at a variety of different baseline probabilities (percentage of the majority class)?

The diagnoal line depicts a completely random classifier and ideally our model's ROC curve should be toward the top-left corner and stay as far away from the diagnoal line as possible.

Side note: Apart from comparing the model's ROC curve against the ROC curve of a classifier that does random guessing, it's also useful to plot the ROC curve of different classifiers to compare performance against each other.

### AUC probabilistic interpretation

The probabilistic interpretation of the AUC metric is that if we randomly choose a positive case and a negative case, the probability that the positive case outranks the negative case according to the classifier is given by the AUC. Hopefully, this is evident from the ROC curve figure, where plot is enumerating all possible combinations of positive and negative cases, and the fraction under the curve comprises of the area where the positive case outranks the negative one. I personally find this interpretation extremely useful when conveying what AUC is measuring to a non-technical audience.

```python
def auc_probability(y_true, y_score, size = 100000):
    """probabilistic interpretation of AUC"""
    labels = y_true.astype(np.bool)
    pos = np.random.choice(y_score[labels], size = size, replace = True)
    neg = np.random.choice(y_score[~labels], size = size, replace = True)
    auc = np.sum(pos > neg) + np.sum(pos == neg) / 2
    auc /= size
    return auc

# we want this be close the score returned
# by roc_auc_score
auc_probability(y_true = y_test, y_score = tree_test_pred)
```

    0.98211499999999996

## Precision Recall Curve

Apart from ROC curve, there is also the **precision recall curve**. Instead of plotting true positive rate (a.k.a recall) versus false positive rate. We now plot precision versus recall.

\begin{align}
[\text{precision}]
&= \frac{[\text{# positive data points with positive predicitions}]}{\text{[# all data points with positive predictions]}} \\
&= \frac{[\text{# true positives}]}{[\text{# true positives}] + [\text{# false positives}]}
\end{align}

```python
tree_test_pred = tree.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(
    y_test, tree_test_pred, pos_label = 1)

# AUC score that summarizes the precision recall curve
avg_precision = average_precision_score(y_test, tree_test_pred)

label = 'Precision Recall AUC: {:.2f}'.format(avg_precision)
plt.plot(recall, precision, lw = 2, label = label)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision Recall Curve')
plt.legend()
plt.tight_layout()
plt.show()
```

A classifier with high recall but low precision flags many positive results, but most of its predicted labels are incorrect when compared to its correponding labels. On the other hand, a classifier with high precision but low recall is just the opposite, returning very few results, but most of its predicted labels are correct when compared to the training labels. An ideal system with high precision and high recall will return many results, with all results labeled correctly.

Precision recall curve answers a fundamentally different question compared to ROC curve. By defition, precision directly answers the question, "What is the probability that this is a real hit given my classifier says it is? Thus it is useful in practice for needle-in-haystack type problems or problems where the "positive" class is more interesting than the negative class.

Although ROC curve is presumably the more popular choice when evaluating binary classifiers, it is highly recommended to use precision recall curve as a supplement to ROC curves to get a full picture when evaluating and comparing classifiers.

# Thresholding via Cost

Lots of real world binary classification needs a threshold to convert the model into a business decision. i.e. All cases with a model score above the threshold get some sort of special treatment. For example:

- **Fraud Prevention**: We're a social network company and we'd like to delete fake accounts. We build a classifier that assigns a "fraud score" between 0 and 1 to each account and, after some research, decide that all accounts whose score is above 0.9 should be sent to our fraud team, which will review each case and delete the accounts that are actually fake
- **Response/Propensity Modeling**: We're at an enterprise software company and we'd like to improve our outbound sales program. We buy a large database of potential customers and build a classifier to predict which ones are likely buy our product if contacted by our sales team. We decide that all customers with a "response score" of above threshold 0.7 should get a call
- **Shopping Cart Abandonment**: We're at an e-commerce company, and we'd like to email a 10% off coupon to users who abandoned their shopping carts and won't return organically. (We don't want to give a 10% off coupon to users who are going to return anyway, since then you'd be losing 10% of the sale price.) We build a classifier to predict which users will never return to their carts, and decide that all users with a "abandonment score" above 0.85 should get the 10% off coupon
- Etc.

Thresholding is popular because of its simplicity and ease of implementation: We translate a continuous score to a binary yes/no decision, and act on it in a predetermined way. The biggest question for the thresholding pattern is: Where should I set the threshold point?

Up until this point, we've been using AUC to give us a single-number summary of classifier performance. This might be suitable in some circumstances, but for binary classifiers, evaluation metrics that take into account the actual costs of false positive and false negative errors may be much more appropriate than AUC. If we know these costs, we can use them not only to tie the evaluation metric more directly to the business value but also choose an appropriate final cutoff threhold for the classifier.

In real world application, the cost that comes along with making these two mistakes (false positive and false negative) are usually a whole lot different. Take our case for example, a false negative (FN) means an employee left our company but our model failed to detect that, while a false positive (FP) means an employee is still currently working at our company and our model told us that they will be leaving. The former mistake would be a tragedy, since, well, the employee left and we didn’t do anything about it! As for conducting the latter mistake, we might be wasting like 20 minutes of a HR manager's time when we arrange a face to face interview with a employee, questioning about how the company can do better to retain him/her, while he/she is perfectly fine with the current situation.

In the code cell below, we assign a cost for a false negative (FN) and false positive (FP) to be 100 and 1000 respectively, given the cost associated with the two mistakes we will multiply them with the false negative and false positive rate at each threshold to figure out where's the best cutoff value. Note that the cost associated with the mistake can just be a back of the envelope number as long as we're sure about which one is more "expensive" than the other.

```python
fpr, tpr, thresholds = roc_curve(y_test, tree_test_pred, pos_label = 1)
fnr = tpr[-1] - tpr

# fn = false negative, meaning the employee left and we
# didn't do anything about it, in our case this will be
# substantially more expensive than a false positive
fp_cost = 100
fn_cost = 1000

costs = (fpr * fp_cost + fnr * fn_cost) * y_test.size
```

```python
best = np.argmin(costs)
best_cost = np.round(costs[best])
best_threshold = np.round(thresholds[best], 2)

plt.plot(thresholds, costs, label = 'cost ($)')
label = 'best cost: {} at threshold {}'.format(best_cost, best_threshold)
plt.axvline(best_threshold, linestyle = '--', lw = 2, color = 'black', label = label)
plt.xlabel('threshold')
plt.ylabel('$')
plt.title('Cost as a Function of Threshold')
plt.legend()
plt.tight_layout()
plt.show()
```

Just to hit the notion home, when executing on a project, if we are able to compute expected cost for each mistake, consider maximizing that directly instead of AUC or another general-purpose metrics.

# Reference

- [Scikit-learn Documentation: Precision Recall](http://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html)
- [StackExchange: ROC vs precision-and-recall curves](https://stats.stackexchange.com/questions/7207/roc-vs-precision-and-recall-curves)
- [Blog: Calculating AUC: the area under a ROC Curve](http://blog.revolutionanalytics.com/2016/11/calculating-auc.html)
- [Blog: Machine Learning Meets Economics](http://blog.mldb.ai/blog/posts/2016/01/ml-meets-economics/)
- [Blog: Visualizing Machine Learning Thresholds to Make Better Business Decisions](http://blog.insightdatalabs.com/visualizing-classifier-thresholds/)