# the $k$-nearest neighbors (kNN) supervised learning algorithm

* can be used for regression or classification
* non-parametric
* imposes only mild structural assumptions about the data

let's explore $k$-nearest neighbor classification.

training data examples: $\{(\mathbf{x}_1, y_1), (\mathbf{x}_2, y_2), ..., (\mathbf{x}_n, y_n)\}$ where $\mathbf{x}_i$ is the feature vector and $y_i \in \{0, 1\}$ is the label on data point $i$. This is a binary classification since each data point is labeled with $0$ or $1$.

the $k$-NN algorithm uses the training data to classify new data points. say we have a new data point $\mathbf{x}$ but we don't know its label. the $k$-NN algorithm predicts the class of this data point as:
\begin{equation}
    \hat{y}(\mathbf{x})= \frac{1}{k} \displaystyle \sum_{\mathbf{x}_i \in N_k(\mathbf{x})} y_i
\end{equation}
where $N_k(x)$ is the *neighborhood* of $\mathbf{x}$ defined as the $k$ "closest" points in the training data set. For "closest" to be mathematically defined, we need a distance metric. One such distance metric is Euclidean distance. In words, the $k$ data points closest to $\mathbf{x}$ vote on whether we should classify this point as a 0 or 1 with their labels.

```julia
using CSV
using DataFrames
using PyPlot
using ScikitLearn # machine learning package
using StatsBase
using Random
using LaTeXStrings # for L"$x$" to work instead of needing to do "\$x\$"
using Printf

# (optional)change settings for all plots at once, e.g. font size
rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16

# (optional) change the style. see styles here: https://matplotlib.org/3.1.1/gallery/style_sheets/style_sheets_reference.html
PyPlot.matplotlib.style.use("seaborn-white")
```

## classifying breast tumors as malignant or benign

source: [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic))

> Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.

The mean radius and smoothness of the cell nuclei (the two features) and the outcome (M = malignant, B = benign) of the tumor are in the `breast_cancer_data.csv`.

```julia
df = CSV.read("breast_cancer_data.csv")
first(df, 5)
```

	mean_radius	mean_smoothness	outcome

	Float64	Float64	String

5 rows × 3 columns

1	13.85	1.495	B

2	9.668	2.275	B

3	9.295	2.388	B

4	19.69	4.585	M

5	9.755	1.243	B

```julia
by(df, :outcome, total=:outcome => length)
```

	outcome	total

	String	Int64

2 rows × 2 columns

1	B	180

2	M	120

let's map the outcomes to a number, 0 or 1, to facilitate the NN algo.

benign (B) : 0

malignant (M) : 1

```julia
df[!, :class] = map(tumor -> tumor == "M" ? 1 : 0, df[!, :outcome])
first(df, 5)
```

	mean_radius	mean_smoothness	outcome	class

	Float64	Float64	String	Int64

5 rows × 4 columns

1	13.85	1.495	B	0

2	9.668	2.275	B	0

3	9.295	2.388	B	0

4	19.69	4.585	M	1

5	9.755	1.243	B	0

let's also have a color scheme for the class labels.

```julia
color_scheme = Dict("M" => "r", "B" => "g")
```

    Dict{String,String} with 2 entries:
      "B" => "g"
      "M" => "r"

since this is a two-dimensional feature space, we have the luxury of visualizing how the classes are distributed in feature space. in practice, we do not have this luxury. I choose a 2D feature space for pedagogical purposes :)

```julia
figure()
xlabel("mean radius")
ylabel("mean smoothness")
for df_o in groupby(df, :outcome)
    outcome = df_o[1, :outcome]
    scatter(
        df_o[!, :mean_radius],
        df_o[!, :mean_smoothness],
        alpha=0.25,
        color=color_scheme[outcome],
        label=outcome
    )
end
legend(frameon=false)
```

### using Scikitlearn for $k$-nearest neighbor classification

scikitlearn documentation for `KNeighborsClassifier` [here](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)

```julia
@sk_import neighbors : KNeighborsClassifier
@sk_import model_selection : train_test_split
@sk_import metrics : confusion_matrix
```

    ┌ Warning: Module model_selection has been ported to Julia - try `import ScikitLearn: CrossValidation` instead
    └ @ ScikitLearn.Skcore /home/mick/.julia/packages/ScikitLearn/bo2Pt/src/Skcore.jl:140

    PyObject <function confusion_matrix at 0x7fb852243ae8>

scikitlearn takes as input:
* a feature matrix `X`, which must be `n_samples` by `n_features`
* a target vector `y`, which must be `n_samples` long (of course)

```julia
n_tumors = nrow(df)
X = zeros(n_tumors, 2)
y = zeros(n_tumors)

for (i, tumor) in enumerate(eachrow(df))
    X[i, 1] = tumor[:mean_radius]
    X[i, 2] = tumor[:mean_smoothness]
    y[i] = tumor[:class]
end
```

construct a nearest neighbor object in scikitlearn

```julia
knn = KNeighborsClassifier(n_neighbors=10)
knn.fit(X, y)
```

    PyObject KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
               metric_params=None, n_jobs=None, n_neighbors=10, p=2,
               weights='uniform')

pass the `KNeighborsClassifier` object our training data. this will be used to make predictions on future, unseen data.

```julia

```

we can now make predictions on new, unseen data. that is, if we know the mean radius and mean smoothness of a new tumor, we can make a prediction about whether this tumor is malignant or benign. let's imagine we take measurements on a new tumor and it has the feature vector `x_new` below. we aim to predict whether it is benign or malignant.

```julia
# new tumor. unknown if malignant or benign...
# x_new = [10.0 1.5] # should be B
x_new = [20.0 5.0] # should be M
```

    1×2 Array{Float64,2}:
     20.0  5.0

first, let's plot where it falls in feature space

```julia
figure()
xlabel("mean radius")
ylabel("mean smoothness")
for df_o in groupby(df, :outcome)
    outcome = df_o[1, :outcome]
    scatter(
        df_o[!, :mean_radius],
        df_o[!, :mean_smoothness],
        alpha=0.25,
        color=color_scheme[outcome],
        label=outcome
    )
end
scatter(x_new[1], x_new[2], color="k")
legend(frameon=false)
```

predict whether this new tumor is benign or malignant

```julia
knn.predict(x_new)
```

    1-element Array{Float64,1}:
     1.0

visualize the decision boundary

```julia
radius = range(5, stop=30, length=100)
smoothness = range(0, stop=18, length=100)

knn_prediction = zeros(length(smoothness), length(radius)) * NaN

for i in 1:length(radius)
    for j in 1:length(smoothness)
        # build the feature vector for this point
        X_here = [radius[i] smoothness[j]]
        knn_prediction[j, i] = knn.predict(X_here)[1]
    end
end
```

```julia
figure()
xlabel("mean radius")
ylabel("mean smoothness")
contourf(radius, smoothness, knn_prediction, [-1, 0.5, 2], colors=["g", "r"], alpha=0.7)

```

we can compute the accuracy of the prediction on the training data. but when $k=1$ we get 100% accuracy by construction!

Pro tip: simpler decision boundaries tend to work better on unseen data

```julia

```

### the test/train paradigm

randomly split your data set into a training and test set. train the $k$-NN algo on the training data, then use the trained model to make predictions on data in the test set. we can compare the predicted label to the true label. the test set error is a quality prediction of generalization error on *unseen* data. we could write our own code for this, but scikitlearn provides `test_train_split` for us.

```julia
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4)

println("size of test set:\t", length(y_test))
println("size of training set:\t", length(y_train))
```

    size of test set:	120
    size of training set:	180

to fully assess the performance of our model, trained on the training set, and tested on the test set, we can plot a so-called confusion matrix. it tells us about false positives, true positives, false negatives, and true negatives.

```julia
# see https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py
function plot_confusion_matrix(y_true::Array{Float64}, y_pred::Array{Float64}, classes::Array{String})
    # calculate confusion matrix from scikitlearn
    cm = confusion_matrix(y_true, y_pred)

    fig, ax = plt.subplots()
    ax.imshow(cm, interpolation="nearest", cmap=plt.cm.Greens)
    ax.set(
        xlim=[-0.5, 1.5],
        ylim=[-0.5, 1.5],
        xticks=[0, 1],
        yticks=[0, 1],
        xticklabels=classes,
        yticklabels=classes,
        xlabel="predicted label",
        ylabel="true label"
    )

    for i = 1:2
        for j = 1:2
            ax.text(j-1, i-1, @sprintf("%d", cm[i, j]), ha="center", va="center")
        end
    end
    tight_layout()
end

knn = KNeighborsClassifier(n_neighbors=8)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
plot_confusion_matrix(y_test, y_pred, ["B", "M"])
```

With a confusion matrix, we want the cross terms (white boxes above) to be small, while the other terms are large. Decide relative importance (false + vs false -) and adjust accordingly

to find the best $k$, we can scan over all $k$, train the model in the training set, then test the model on the test set (unseen data) to see how it performs. we take the "best" $k$ as the one that yields the lowest test set error, as we expect it to have the lowest generalization error on unseen data.

```julia
ks = 1:20
test_accuracy = zeros(length(ks)) * NaN
train_accuracy = zeros(length(ks)) * NaN
for (i, k) in enumerate(ks)
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    test_accuracy[i] = knn.score(X_test, y_test)
    train_accuracy[i] = knn.score(X_train, y_train)
end

fig, ax = subplots(1, 1, figsize=(18, 8))
ax.plot(ks, test_accuracy, label="test")
ax.plot(ks, train_accuracy, label="train")
ax.legend(frameon=false)
ax.set_xlabel("k")
ax.set_ylabel("Accuracy")
ax.set_xlim([1, 20])
```

the optimal $k$ is around 8. So we should train our $k$-NN algo with $k=8$, then deploy this model, since it should result in the best accuracy on unseen data.

Since this changes depending on which data are sampled for train or test sets, use k-fold cross validation.

in this method, k is the number of bins to divide your data into. then the above process is run through k times, and each bin acts as the test set once. results are averaged.

```julia
using ScikitLearn.CrossValidation: KFold
```

```julia
K = 5 # number of folds
kf = KFold(n_tumors, n_folds=K, shuffle=true); # this gives [[train indices, test indices]_i]
```

```julia
ks = 1:20
test_accuracy = [NaN for _ in ks]
for k in ks
    test_accuracy[k] = 0.0
    for (train_ids, test_ids) in kf
        knn = KNeighborsClassifier(n_neighbors=k)
        knn.fit(X[train_ids, :], y[train_ids])
        # this is just returning the percentage of correct estimates, which is easy
        # to code
        test_accuracy[k] += knn.score(X[test_ids, :], y[test_ids])
    end
    test_accuracy[k] /= K
    println("Test set accuracy: ", test_accuracy[k], " (K = $k)")
end
plot(ks, test_accuracy)
k_max_test_accuracy = argmax(test_accuracy)
axvline(k_max_test_accuracy)
```

For small datasets, it's probably a good idea to repeat this analysis a bunch of times, since the optimum k varies pretty widely with the train/test split

**special case: K (num folds) = n (# data points)**
you're only testing on one data point. This is called leave one out cross-validation
* most amount of data in your training set, but...
* you need to train the model n times
* high variance on accuracy estimate

note: testing with only one data point is okay because you're averaging a shitload of accuracies... almost like PWM

In-Class Notes/K Nearest Neighbors/kNN_algo_sparse.ipynb


# <center> Single compartment model using double exponentials</center>

## Summary

### 1. Setup and testing
The model is trying to simulate a single compartment,
$$ C_m \frac{dV_m}{dt} = g_{leak}(V_m - E_{leak}) + g_{exc}(V_m - E_{AMPA}) + g_{inh}(V_m - E_{GABA})$$
Here $E$'s are reversal potentials, $V_m$ is membrane potential, $g$'s are conductances, and $C_m$ is the membrane capacitance.
The synaptic conductances are modeled as double exponentials: $$g(t) = \bar{g}\frac{( e^\frac{\delta_{onset} - t }{\tau_{decay}} - e^\frac{\delta_{onset} - t }{\tau_{rise}})}{- \left(\frac{\tau_{rise}}{\tau_{decay}}\right)^{\frac{\tau_{decay}}{\tau_{decay} - \tau_{rise}}} + \left(\frac{\tau_{rise}}{\tau_{decay}}\right)^{\frac{\tau_{rise}}{\tau_{decay} - \tau_{rise}}}}$$
Here $\bar{g}$ is maximum conductance, $\tau$ and $\delta$ are time course and onset delay parameters respectively. The denominator is to normalize the term to 1. All the quantities in the model have been set up with sympy and have units.

In this notebook, later the model is tested by plotting excitatory and inhibitory PSPs with the parameters.

### 2. Exploring hypotheses

After setting up the model with reasonable parameters from literature or data, the following hypotheses are explored:

1. What can give rise to divisive normalization?
    1. No inhibition?
    2. Proportional excitation and inhibition?
    3. Divisive recruitment kinetics of inhibition? (Here g_i recruitment is a nonlinear function in g_e).
    4. Proportional excitation and inhibition, with delays as a function of excitation?

### 3. Still to be done:
1. Put a thresholding non-linearity and check spike times as a function of g_e.

___

```python
from IPython.display import display, Markdown
```

```python
# Parallelizing over the cores
import os
# import ipyparallel as ipp
# clients = ipp.Client()
# dview = clients.direct_view()
# print(clients.ids)
```

```python
#with dview.sync_imports():
from sympy import symbols, exp, solve, logcombine, simplify, Piecewise, lambdify, N, init_printing, Eq
import numpy
#from sympy import *
from sympy.physics.units import seconds, siemens, volts, farads, amperes, milli, micro, nano, pico, ms, s, kg, meters
from matplotlib import pyplot as plt
#%px plt = pyplot
```

```python
#dview.block=True
```

```python
init_printing()
```

```python
#matplotlib notebook
%matplotlib inline
import matplotlib
matplotlib.rcParams['figure.figsize'] = (10,6)
import matplotlib.pyplot as plt
plt.style.use('seaborn-notebook')
```

#### Simulation time

```python
samplingRate = 20  # kHz, to get milliseconds
sample_every = 1 # ms
timeStep, maxTime = (sample_every*1.)/ samplingRate, 100.  # ms
trange = numpy.arange(
    0., maxTime, timeStep)  # We will always use 100. ms timecourse of PSPs.
```

#### Range of $g_e$ explored

```python
emax = 4
e_step = 0.1
erange = numpy.arange(0., emax, e_step)
```

#### Range of proportionality ($P$) between $E$ and $I$

```python
prop_array = numpy.arange(1, 5, 1)
```

```python
# dview.push(dict(trange=trange, erange=erange, prop_array=prop_array))
```

## Setting up the variables, parameters and units for simulation

```python
t, P, e_r, e_d, delta_e, rho_e, g_e, i_r, i_d, delta_i, rho_i, g_i, b, Cm, g_L = symbols(
    't P \\tau_{er} \\tau_{ed} \\delta_e \\rho_e \\bar{g}_e \\tau_{ir} \\tau_{id} \\delta_i \\rho_i \\bar{g}_i \\beta C_m \\bar{g}_L',
    positive=True,
    real=True)
```

```python
leak_rev, e_rev, i_rev, Vm = symbols(
    'Leak_{rev} Exc_{rev} Inh_{rev} V_m', real=True)
```

```python
SymbolDict = {
    t: "Time (ms)",
    P: "Proportion of $g_i/g_e$",
    e_r: "Excitatory Rise (ms)",
    e_d: "Excitatory Fall (ms)",
    delta_e: "Excitatory onset time (ms)",
    rho_e: "Excitatory $tau$ ratio (fall/rise)",
    g_e: "Excitatory max conductance",
    i_r: "Inhibitory Rise (ms)",
    i_d: "Inhibitory Fall(ms)",
    delta_i: "Inhibitory onset time(ms)",
    rho_i: "Inhibitory $tau$ ratio (fall/rise)",
    g_i: "Inhibitory max conductance",
    b: "Inhibitory/Excitatory $tau$ rise ratio"
}
```

```python
unitsDict = {
    's': seconds,
    'exp': exp,
    'S': siemens,
    'V': volts,
    'A': amperes,
    'm': meters,
    'kg': kg
}  # This is for lamdify
```

```python
nS, pF, mV, pA = nano * siemens, pico * farads, milli * volts, pico*amperes
```

### Estimates from data and averaging them to get a number

```python
estimateDict = {
    P: (1.9, 2.1),
    e_r: (1.5 * ms, 5 * ms),
    e_d: (8. * ms, 20. * ms),
    delta_e: (0. * ms, 0. * ms),
    rho_e: (2., 7.),
    g_e: (0.02 * nS, 0.25 * nS),
    i_r: (1.5 * ms, 5. * ms),
    i_d: (14. * ms, 60. * ms),
    delta_i: (2. * ms, 4. * ms),
    rho_i: (5., 20.),
    g_i: (0.04 * nS, 0.5 * nS),
    b: (0.5, 5.)
}
```

```python
averageEstimateDict = {
    key: value[0] + value[1] / 2
    for key, value in estimateDict.items()
}
```

```python
# Customized some numbers.
```

```python
averageEstimateDict[i_r], averageEstimateDict[e_r] = 3.6 * ms, 6. * ms
```

```python
print ("| Variable  |  Meaning |  Range |")
print ("|---|---|---|")
print ("|$)t$|Time (ms)|0-100|")
for i in [P, e_r, e_d, delta_e, rho_e, g_e, i_r, i_d, delta_i, rho_i, g_i, b]:
    print ("|${}$|{}|{}-{}|".format(i, SymbolDict[i], estimateDict[i][0],
                                   estimateDict[i][1]))
```

    | Variable  |  Meaning |  Range |
    |---|---|---|
    |$)t$|Time (ms)|0-100|
    |$P$|Proportion of $g_i/g_e$|1.9-2.1|
    |$\tau_{er}$|Excitatory Rise (ms)|0.0015*s-s/200|
    |$\tau_{ed}$|Excitatory Fall (ms)|0.008*s-0.02*s|
    |$\delta_e$|Excitatory onset time (ms)|0-0|
    |$\rho_e$|Excitatory $tau$ ratio (fall/rise)|2.0-7.0|
    |$\bar{g}_e$|Excitatory max conductance|2.0e-11*A**2*s**3/(kg*m**2)-2.5e-10*A**2*s**3/(kg*m**2)|
    |$\tau_{ir}$|Inhibitory Rise (ms)|0.0015*s-0.005*s|
    |$\tau_{id}$|Inhibitory Fall(ms)|0.014*s-0.06*s|
    |$\delta_i$|Inhibitory onset time(ms)|0.002*s-0.004*s|
    |$\rho_i$|Inhibitory $tau$ ratio (fall/rise)|5.0-20.0|
    |$\bar{g}_i$|Inhibitory max conductance|4.0e-11*A**2*s**3/(kg*m**2)-5.0e-10*A**2*s**3/(kg*m**2)|
    |$\beta$|Inhibitory/Excitatory $tau$ rise ratio|0.5-5.0|

| Variable  |  Meaning |  Range |
|---|---|---|
|$t$|Time (ms)|0-100|
|$P$|Proportion of $g_i/g_e$|1.9-2.1|
|$\tau_{er}$|Excitatory Rise (ms)|1.5-5|
|$\tau_{ed}$|Excitatory Fall (ms)|8-20|
|$\delta_e$|Excitatory onset time (ms)|0-0|
|$\rho_e$|Excitatory $tau$ ratio (fall/rise)|2-7|
|$\bar{g}_e$|Excitatory max conductance|0.02-0.25|
|$\tau_{ir}$|Inhibitory Rise (ms)|1.5-5|
|$\tau_{id}$|Inhibitory Fall(ms)|14-60|
|$\delta_i$|Inhibitory onset time(ms)|3-15|
|$\rho_i$|Inhibitory $tau$ ratio (fall/rise)|5-20|
|$\bar{g}_i$|Inhibitory max conductance|0.04-0.5|
|$\beta$|Inhibitory/Excitatory $tau$ rise ratio|0.5-5|

### Approximating the rest from literature

```python
approximateDict = {
    g_L: 10 * nS,
    e_rev: 0. * mV,
    i_rev: -70. * mV,
    leak_rev: -65. * mV,
    Cm: 100 * pF
}
sourceDict = {
    g_L: "None",
    e_rev: "None",
    i_rev: "None",
    leak_rev: "None",
    Cm: "Neuroelectro.org"
}
```

```python
print ("| Variable  |  Meaning | Source | Value |")
print ("|---|---|---|")
print ("|$g_L$|Leak conductance|Undefined| 10 nS |")
print ("|$Exc_{rev}$|Excitatory reversal|Undefined| 0 mV|")
print ("|$Inh_{rev}$|Inhibitory reversal |Undefined| -70 mV |")
print ("|$Leak_{rev}$|Leak reversal |Undefined| -65 mV |")
print ("|$C_m$|Membrane capacitance |neuroelectro.org| 100 pF|")
```

    | Variable  |  Meaning | Source | Value |
    |---|---|---|
    |$g_L$|Leak conductance|Undefined| 10 nS |
    |$Exc_{rev}$|Excitatory reversal|Undefined| 0 mV|
    |$Inh_{rev}$|Inhibitory reversal |Undefined| -70 mV |
    |$Leak_{rev}$|Leak reversal |Undefined| -65 mV |
    |$C_m$|Membrane capacitance |neuroelectro.org| 100 pF|

| Variable  |  Meaning | Source | Value |
|---|---|---|
|$g_L$|Leak conductance|Undefined| 10 nS |
|$Exc_{rev}$|Excitatory reversal|Undefined| 0 mV|
|$Inh_{rev}$|Inhibitory reversal |Undefined| -70 mV |
|$Leak_{rev}$|Leak reversal |Undefined| -65 mV |
|$C_m$|Membrane capacitance |neuroelectro.org| 100 pF|

## Functions

### Check spike times

```python
def find_spike_time(voltage, threshold= 25*mV ):
    ''' Returns time at spike'''
    return numpy.argmax(voltage > threshold) * timeStep * ms
```

---

### Double exponential to explain the net synaptic conductance.

```python
alpha = exp(-(t - delta_e) / e_d) - exp(-(t - delta_e) / e_r)
```

```python
alpha
```

```python
#alpha = alpha.subs(e_d, (rho_e*e_r)).doit()
```

```python
alpha_prime = alpha.diff(t)
```

```python
alpha_prime
```

```python
theta_e = solve(alpha_prime, t)  # Time to peak
```

```python
theta_e = logcombine(theta_e[0])
```

```python
theta_e
```

```python
simplify(theta_e.subs(averageEstimateDict))
```

```python
alpha_star = simplify(alpha.subs(t, theta_e).doit())
```

```python
alpha_star
```

```python
#alpha_star = simplify(alpha) # Replacing e_d/e_r with tau_e
```

### Finding maximum of the curve and substituting ratio of taus

```python
alpha_star
```

```python
g_E = Piecewise((0. * nS, t / ms < delta_e / ms), (g_e * (alpha / alpha_star),
                                                   True))
```

### Final equation for Excitation normalized to be maximum at $g_e$

```python
g_E
```

### Verifying that E Behaves

```python
E_check = g_E.subs(averageEstimateDict).evalf()
```

```python
E_check
```

```python
E_check.free_symbols
```

```python
f = lambdify(
    (t), simplify(E_check / nS), modules=("sympy", "numpy", unitsDict))
```

```python
fig, ax = plt.subplots()
ax.plot(trange, [f(dt * ms) for dt in trange], label="Excitation")
ax.set_xlabel("Time (in ms)")
ax.set_ylabel("Conductance (in nS)")
ax.legend()
```

```python
plt.close(fig)
```

### Doing the same with inhibition

```python
g_I = g_E.xreplace({
    g_e: g_i,
    rho_e: rho_i,
    e_r: i_r,
    e_d: i_d,
    delta_e: delta_i
})
```

```python
alpha_I = alpha.xreplace({e_r: i_r, e_d: i_d, delta_e: delta_i})
alpha_star_I = alpha_star.xreplace({e_r: i_r, e_d: i_d})
```

```python
g_I = Piecewise((0. * nS, t / ms < delta_i / ms),
                (g_i * (alpha_I / alpha_star_I), True))
```

```python
g_I
```

### Verifying that I Behaves

```python
I_check = simplify(g_I.subs(averageEstimateDict).evalf())
```

```python
N(I_check.subs({t: 4.9 * ms}))
```

```python
f = lambdify(t, simplify(I_check / nS), modules=("sympy", "numpy", unitsDict))
```

```python
f(5 * ms)
```

    array(0.0868746200026281, dtype=object)

```python
plt.plot(trange, [-f(dt * ms) for dt in trange], label="Inhibition")
plt.xlabel("Time (in ms)")
plt.ylabel("Conductance (in nS)")
plt.legend()
plt.show()
```

### Now finding the control peak using difference of these double-exponentials

```python
compartment = Eq((1 / Cm) * (g_E * (Vm - e_rev) + g_I * (Vm - i_rev) + g_L *
                             (Vm - leak_rev)), Vm.diff(t))
```

```python
compartment
```

```python
Vm_t = solve(compartment, Vm, rational=False, simplify=True)
```

```python
check_vm_t = Vm_t[0].subs(averageEstimateDict).subs(approximateDict)/mV
vm_change = [check_vm_t.subs({t:dt*ms}) for dt in trange]
```

```python
plt.plot(trange, vm_change)
plt.show()
```

### Sending to clusters

```python
# symbols_to_pass = dict(t=t, P=P, e_r=e_r, e_d=e_d, delta_e=delta_e, rho_e=rho_e, g_e=g_e, i_r=i_r, i_d=i_d, delta_i=delta_i, rho_i=rho_i, g_i=g_i, b=b, Cm=Cm, g_L=g_L, leak_rev=leak_rev, e_rev=e_rev, i_rev=i_rev, Vm=Vm)
# units_to_pass = dict(nS=nS, pF=pF, mV=mV)
# expressions_to_pass = dict(Vm_t=Vm_t)
```

```python
# dview.push(symbols_to_pass)
# dview.push(units_to_pass)
# dview.push(expressions_to_pass)
# dview.push(dict(unitsDict=unitsDict))
```

---

# Testing hypotheses

### Varying $g_e$ and proportionality $P$

```python
check_vm_t = Vm_t[0].subs({ i: averageEstimateDict[i] for i in averageEstimateDict if i not in [g_e, g_i, P] }).subs(approximateDict).subs({ g_i: P * g_e })
```

```python
check_vm_t
```

```python
f = lambdify((g_e, P, t), check_vm_t/mV, (unitsDict, "numpy"))
```

#### Case 1: No inhibition

```python
prop = 0 # No inhibition
norm = matplotlib.colors.Normalize(
    vmin=numpy.min(erange),
    vmax=numpy.max(erange))
c_m = matplotlib.cm.viridis
s_m = matplotlib.cm.ScalarMappable(cmap=c_m, norm=norm)
s_m.set_array([])
```

```python
fig, ax = plt.subplots()
excitation_only = [] # Excitation only vector
for e in erange:
    e_t =  [float(f(e * nS, prop, dt * ms)) for dt in trange]
    ax.plot(trange, e_t, color=s_m.to_rgba(e))
    excitation_only.append(e_t)
plt.xlabel("Time")
plt.ylabel("$V_m(mV)$")
plt.title("No Inhibition")
plt.colorbar(s_m, label="g_e(nS)")
```

```python
numpy.savetxt('excitation_only.txt', excitation_only, header="0-{}:{}".format(emax, e_step))
```

#### Case 2: With inhibition proportional to excitation, or $g_i = P \times g_e$

```python
prop = 3.  # Proportionality
fig, ax = plt.subplots()
proportional_e_i = []
for e in erange:
#     dview.push(dict(e=e, prop=prop))
#     v_t = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)), trange)
    v_t =  [float(f(e * nS, prop, dt * ms)) for dt in trange]
    proportional_e_i.append(v_t)
    ax.plot(trange, v_t, color=s_m.to_rgba(e))
plt.xlabel("Time")
plt.ylabel("$V_m(mV)$")
plt.title("$g_i = {:.2f} \\times g_e$".format(prop))
plt.colorbar(s_m, label="g_e(nS)")
plt.legend()
```

```python
numpy.savetxt('excitation_inhibition_proportional_only.txt', proportional_e_i, header="0-{}:{}, P={}".format(emax, e_step, prop))
```

```python
fig, ax = plt.subplots()
for prop in prop_array:
    v_max = []
    e_max = []
    for e, e_t in zip(erange, excitation_only):
        v_t =  [float(f(e * nS, prop, dt * ms)) for dt in trange]
        v_max.append(max(v_t) - float(approximateDict[leak_rev]/mV))
        e_max.append(max(e_t) - float(approximateDict[leak_rev]/mV))
#         dview.push(dict(e=e, prop=prop))
#         vm_change = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)),
#                                    trange)
#         em_change = dview.map_sync(lambda dt: float(f(e * nS, 0., dt * ms)),
#                                    trange)
#         v_max.append(
#             max(vm_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
#         e_max.append(
#             max(em_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
    ax.scatter(e_max, v_max, label="$P= {:.2f}$".format(prop))
    numpy.savetxt('excitation_inhibition_proportional_{}_only_scatter.txt'.format(prop), (v_max, e_max), header="0-{}:{}, P={}".format(emax, e_step, prop))
ax.set_xlabel("Excitation $V_{max}$")
ax.set_ylabel("Control  $V_{max}$")
ax.legend()
```

#### Case 3: With inhibition recruited, $I = \frac{l}{1+\frac{1}{kE}}\times E$

```python
p = lambda k,l: (k*l*e)/(1.+ (k*e))
```

```python
k, l = 1, 2
fig, ax = plt.subplots()
for e in erange:
    prop = p(k, l)
#     dview.push(dict(e=e, prop=prop))
    v_t =  [float(f(e * nS, prop, dt * ms)) for dt in trange]
#     v_t = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)), trange)
    ax.plot(
        trange, v_t, color=s_m.to_rgba(e) )
plt.colorbar(s_m, label="g_e(nS)")
plt.legend()
```

```python
l = 2
fig, ax = plt.subplots()
for k in numpy.arange(0., 5., 1.):
    v_max = []
    e_max = []
    for e, e_t in zip(erange, excitation_only):
        prop = p(k, l)
        v_t =  [float(f(e * nS, prop, dt * ms)) for dt in trange]
        v_max.append(max(v_t) - float(approximateDict[leak_rev]/mV))
        e_max.append(max(e_t) - float(approximateDict[leak_rev]/mV))

#         dview.push(dict(e=e, prop=prop))
#         vm_change = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)),
#                                    trange)
#         em_change = dview.map_sync(lambda dt: float(f(e * nS, 0., dt * ms)),
#                                    trange)
#         v_max.append(
#             max(vm_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
#         e_max.append(
#             max(em_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
    ax.scatter(e_max, v_max, label="$P= {:.2f}$".format(prop))
    ax.set_xlabel("Excitation $V_{max}$")
    ax.set_ylabel("Control  $V_{max}$")
    numpy.savetxt('excitation_inhibition_divisive_recruited_only.txt', (e_max, v_max), header="0-{}:{}, k={},l={}".format(emax, e_step, k, l))
plt.legend()
```

## Changing kinetics as functions of the excitation.

### Changing $\delta_i$ = $\delta_{min}( 1 + \frac{1}{k\times{g_e}})$

```python
d = lambda minDelay,k,e: minDelay*(1 + (1./(k*e)))
```

```python
k, minDelay = 10, 1.5*ms
```

```python
fig, ax = plt.subplots()
ax.scatter(erange, [d(minDelay, k, e) / ms for e in erange])
ax.set_xlabel("$g_e$")
ax.set_ylabel("$\\delta_i$")
```

### Changing $\delta_i$ = $\delta_{min} + me^{-k\times{g_e}}$

```python
d = lambda minDelay,k,e: minDelay + m*exp(-(k*e))
```

```python
nS = nano*siemens
k, m, minDelay = 2./nS, 9*ms, 1.*ms
```

```python
ax = plt.subplot()
ax.scatter(erange, [d(minDelay,k,e*nS)/ms for e in erange])
plt.xlabel("$g_e$")
plt.ylabel("$\\delta_i$")
plt.show()
```

```python
check_vm = simplify(Vm_t[0].subs({i:averageEstimateDict[i] for i in averageEstimateDict if i not in [g_e, g_i, delta_i]}).subs(approximateDict).subs({g_i: P*g_e, delta_i: d(minDelay,k,g_e)}).evalf())
```

```python
f = lambdify((g_e, P, t), check_vm/mV, (unitsDict, "numpy"))
```

```python
prop = 3.
ax = plt.subplot()
for e in erange:
#     dview.push(dict(e=e, prop=prop))
    v_t =  [float(f(e * nS, prop, dt * ms)) for dt in trange]
    #[float(f(e * nS, prop, dt * ms)) for dt in trange]
    delay = float(d(minDelay,k,e*nS)/ms)
#     v_t = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)), trange)
#     delay = float(dview['float(d(minDelay,k,e*nS)/ms)'][0])
    ax.plot(
        trange, v_t, color=s_m.to_rgba(e))
plt.colorbar(s_m, label="g_e(nS)")
plt.legend()
```

```python
fig, ax = plt.subplots(1, 2)
for prop in prop_array:
    v_max = []
    e_max = []
    v_ttp = []
    e_ttp = []
    for e, e_t in zip(erange, excitation_only):
#         dview.push(dict(e=e, prop=prop))
        v_t = [float(f(e * nS, prop, dt * ms)) for dt in trange]
        v_max.append(max(v_t) - float(approximateDict[leak_rev]/mV))
        e_max.append(max(e_t) - float(approximateDict[leak_rev]/mV))

#         vm_change = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)),
#                                    trange)
#         em_change = dview.map_sync(lambda dt: float(f(e * nS, 0., dt * ms)),
#                                    trange)
#         v_max.append(
#             max(vm_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
#         e_max.append(
#             max(em_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
        v_ttp.append(numpy.argmax(v_t) * timeStep)
        e_ttp.append(numpy.argmax(e_t) * timeStep)
    e_max, v_max = numpy.array(e_max), numpy.array(v_max)
    ax[0].scatter(e_max, v_max, label=str(prop))
    ax[1].scatter(e_max, e_max - v_max, label=str(prop))
    #ax[1].scatter(e_ttp, v_ttp, label=str(p))
    numpy.savetxt('excitation_inhibition_proportional_{}_delay_scatter.txt'.format(prop), (v_max, e_max), header="0-{}:{}, P={}".format(emax, e_step, prop))
maxCoord = max(ax[0].get_xlim()[1], ax[0].get_ylim()[1])
ax[0].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[1].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[0].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[1].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[0].set_xlabel("Excitation $V_{max}$")
ax[0].set_ylabel("Control  $V_{max}$")
ax[1].set_xlabel("Excitation $V_{max}$")
ax[1].set_ylabel("Excitation - Control  $V_{max}$")
#ax[1].set_xlabel("Excitation $t_{peak}$")
#ax[1].set_ylabel("Control  $t_{peak}$")
ax[0].legend()
ax[1].legend()
plt.show()
```

```python
plt.scatter( e_max[1:], v_ttp[1:])
plt.title("Time to peak with excitation max")
plt.xlabel("Excitation Max")
plt.ylabel("Time to peak")
plt.show()
```

---

## Robustness testing of Divisive Normalization

### Divisive normalization is robust to 10% noise in proportionality between E and I.

```python
prop = 3.
delta_prop = numpy.linspace(-1,1,len(erange))
numpy.random.shuffle(delta_prop) # Randomizing order of change
ax = plt.subplot()
for index, e in enumerate(erange):
#     dview.push(dict(e=e, prop=prop))
    v_t = [float(f(e * nS, prop + delta_prop[index], dt * ms)) for dt in trange]
    delay = float(d(minDelay,k,e*nS)/ms)
#     v_t = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)), trange)
#     delay = float(dview['float(d(minDelay,k,e*nS)/ms)'][0])
    ax.plot(
        trange, v_t, color=s_m.to_rgba(e))
plt.colorbar(s_m, label="g_e(nS)")
plt.legend()
```

```python
fig, ax = plt.subplots(1, 2)
delta_prop = numpy.linspace(0.9,1.1,len(erange))
for prop in prop_array:
    v_max = []
    e_max = []
    v_ttp = []
    e_ttp = []
    numpy.random.shuffle(delta_prop) # Randomizing order of change
    for index, e in enumerate(erange):
#         dview.push(dict(e=e, prop=prop))
        v_t = [float(f(e * nS, prop * delta_prop[index], dt * ms)) for dt in trange]
        e_t = excitation_only[index]
        v_max.append(max(v_t) - float(approximateDict[leak_rev]/mV))
        e_max.append(max(e_t) - float(approximateDict[leak_rev]/mV))

#         vm_change = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)),
#                                    trange)
#         em_change = dview.map_sync(lambda dt: float(f(e * nS, 0., dt * ms)),
#                                    trange)
#         v_max.append(
#             max(vm_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
#         e_max.append(
#             max(em_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
        v_ttp.append(numpy.argmax(v_t) * timeStep)
        e_ttp.append(numpy.argmax(e_t) * timeStep)
    e_max, v_max = numpy.array(e_max), numpy.array(v_max)
    ax[0].scatter(e_max, v_max, label=str(prop))
    ax[1].scatter(e_max, e_max - v_max, label=str(prop))
    #ax[1].scatter(e_ttp, v_ttp, label=str(p))
maxCoord = max(ax[0].get_xlim()[1], ax[0].get_ylim()[1])
ax[0].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[1].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[0].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[1].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[0].set_xlabel("Excitation $V_{max}$")
ax[0].set_ylabel("Control  $V_{max}$")
ax[1].set_xlabel("Excitation $V_{max}$")
ax[1].set_ylabel("Excitation - Control  $V_{max}$")
#ax[1].set_xlabel("Excitation $t_{peak}$")
#ax[1].set_ylabel("Control  $t_{peak}$")
ax[0].legend()
ax[1].legend()
plt.show()
```

```python
plt.scatter( e_max[1:], v_ttp[1:])
plt.title("Time to peak with excitation max")
plt.xlabel("Excitation Max")
plt.ylabel("Time to peak")
plt.show()
```

### Permutation Test shows that Divisive Normalization cannot be achieved without E-I balance with just delays

```python
check_vm = Vm_t[0].subs({ i: averageEstimateDict[i] for i in averageEstimateDict if i not in [g_e, g_i, P] }).subs(approximateDict)
```

```python
f = lambdify((g_e, g_i, t), check_vm/mV, (unitsDict, "numpy"))
```

```python
fig, ax = plt.subplots(1, 2)
for prop in prop_array:
    v_max = []
    e_max = []
    v_ttp = []
    e_ttp = []
    irange = prop*erange
    numpy.random.shuffle(irange)
    for index, e in enumerate(erange):
#         dview.push(dict(e=e, prop=prop))
        v_t = [float(f(e * nS, irange[index]* nS, dt * ms)) for dt in trange]
        e_t = excitation_only[index]
        v_max.append(max(v_t) - float(approximateDict[leak_rev]/mV))
        e_max.append(max(e_t) - float(approximateDict[leak_rev]/mV))

#         vm_change = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)),
#                                    trange)
#         em_change = dview.map_sync(lambda dt: float(f(e * nS, 0., dt * ms)),
#                                    trange)
#         v_max.append(
#             max(vm_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
#         e_max.append(
#             max(em_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
        v_ttp.append(numpy.argmax(v_t) * timeStep)
        e_ttp.append(numpy.argmax(e_t) * timeStep)
    e_max, v_max = numpy.array(e_max), numpy.array(v_max)
    ax[0].scatter(e_max, v_max, label=str(prop))
    ax[1].scatter(e_max, e_max - v_max, label=str(prop))
    #ax[1].scatter(e_ttp, v_ttp, label=str(p))
maxCoord = max(ax[0].get_xlim()[1], ax[0].get_ylim()[1])
ax[0].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[1].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[0].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[1].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[0].set_xlabel("Excitation $V_{max}$")
ax[0].set_ylabel("Control  $V_{max}$")
ax[1].set_xlabel("Excitation $V_{max}$")
ax[1].set_ylabel("Excitation - Control  $V_{max}$")
#ax[1].set_xlabel("Excitation $t_{peak}$")
#ax[1].set_ylabel("Control  $t_{peak}$")
ax[0].legend()
ax[1].legend()
plt.show()
```

```python
plt.scatter( e_max[1:], v_ttp[1:])
plt.title("Time to peak with excitation max")
plt.xlabel("Excitation Max")
plt.ylabel("Time to peak")
plt.show()
```

### Divisive normalization is robust to ± 10% change in delay.

```python
check_vm = simplify(Vm_t[0].subs({i:averageEstimateDict[i] for i in averageEstimateDict if i not in [g_e, g_i, delta_i]}).subs(approximateDict).subs({g_i: P*g_e}).evalf())
```

```python
f = lambdify((g_e, P, delta_i, t), check_vm/mV, (unitsDict, "numpy"))
```

```python
fig, ax = plt.subplots(1, 2)
for prop in prop_array:
    v_max = []
    e_max = []
    v_ttp = []
    e_ttp = []
    for index, e in enumerate(erange):
#         dview.push(dict(e=e, prop=prop))
        randomFactor = numpy.random.uniform(0.9,1.1)
        v_t = [float(f(e * nS, prop, d(minDelay,k,e*nS) * randomFactor, dt * ms)) for dt in trange]
        e_t = excitation_only[index]
        v_max.append(max(v_t) - float(approximateDict[leak_rev]/mV))
        e_max.append(max(e_t) - float(approximateDict[leak_rev]/mV))

#         vm_change = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)),
#                                    trange)
#         em_change = dview.map_sync(lambda dt: float(f(e * nS, 0., dt * ms)),
#                                    trange)
#         v_max.append(
#             max(vm_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
#         e_max.append(
#             max(em_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
        v_ttp.append(numpy.argmax(v_t) * timeStep)
        e_ttp.append(numpy.argmax(e_t) * timeStep)
    e_max, v_max = numpy.array(e_max), numpy.array(v_max)
    ax[0].scatter(e_max, v_max, label=str(prop))
    ax[1].scatter(e_max, e_max - v_max, label=str(prop))
    #ax[1].scatter(e_ttp, v_ttp, label=str(p))
maxCoord = max(ax[0].get_xlim()[1], ax[0].get_ylim()[1])
ax[0].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[1].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[0].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[1].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[0].set_xlabel("Excitation $V_{max}$")
ax[0].set_ylabel("Control  $V_{max}$")
ax[1].set_xlabel("Excitation $V_{max}$")
ax[1].set_ylabel("Excitation - Control  $V_{max}$")
#ax[1].set_xlabel("Excitation $t_{peak}$")
#ax[1].set_ylabel("Control  $t_{peak}$")
ax[0].legend()
ax[1].legend()
plt.show()
```

```python
plt.scatter( e_max[1:], v_ttp[1:])
plt.title("Time to peak with excitation max")
plt.xlabel("Excitation Max")
plt.ylabel("Time to peak")
plt.show()
```

### Permutation test shows that Divisive Normalization does not work with permuted delays for large values of excitation.

```python
fig, ax = plt.subplots(1, 2)
for prop in prop_array:
    v_max = []
    e_max = []
    v_ttp = []
    e_ttp = []
    delayRange = [d(minDelay,k,e*nS) for e in erange]
    numpy.random.shuffle(delayRange)
    for index, e in enumerate(erange):
#         dview.push(dict(e=e, prop=prop))
        v_t = [float(f(e * nS, prop, delayRange[index], dt * ms)) for dt in trange]
        e_t = excitation_only[index]
        v_max.append(max(v_t) - float(approximateDict[leak_rev]/mV))
        e_max.append(max(e_t) - float(approximateDict[leak_rev]/mV))

#         vm_change = dview.map_sync(lambda dt: float(f(e * nS, prop, dt * ms)),
#                                    trange)
#         em_change = dview.map_sync(lambda dt: float(f(e * nS, 0., dt * ms)),
#                                    trange)
#         v_max.append(
#             max(vm_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
#         e_max.append(
#             max(em_change) - dview['float(approximateDict[leak_rev]/mV)'][0])
        v_ttp.append(numpy.argmax(v_t) * timeStep)
        e_ttp.append(numpy.argmax(e_t) * timeStep)
    e_max, v_max = numpy.array(e_max), numpy.array(v_max)
    ax[0].scatter(e_max, v_max, label=str(prop))
    ax[1].scatter(e_max, e_max - v_max, label=str(prop))
    #ax[1].scatter(e_ttp, v_ttp, label=str(p))
maxCoord = max(ax[0].get_xlim()[1], ax[0].get_ylim()[1])
ax[0].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[1].plot([0, 1], [0, 1], '--', transform=ax[1].transAxes)
ax[0].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[1].set(xlim=(0, maxCoord), ylim=(0, maxCoord))
ax[0].set_xlabel("Excitation $V_{max}$")
ax[0].set_ylabel("Control  $V_{max}$")
ax[1].set_xlabel("Excitation $V_{max}$")
ax[1].set_ylabel("Excitation - Control  $V_{max}$")
#ax[1].set_xlabel("Excitation $t_{peak}$")
#ax[1].set_ylabel("Control  $t_{peak}$")
ax[0].legend()
ax[1].legend()
plt.show()
```

```python
plt.scatter( e_max[1:], v_ttp[1:])
plt.title("Time to peak with excitation max")
plt.xlabel("Excitation Max")
plt.ylabel("Time to peak")
plt.show()
```

---

## Voltage clamping to see current responses.

```python
I_t = (g_E * (Vm - e_rev) + g_I * (Vm - i_rev) + g_L *
                             (Vm - leak_rev))
```

```python
I_t_check = I_t.subs({i:averageEstimateDict[i] for i in averageEstimateDict if i not in [g_e, g_i, delta_i]}).subs(approximateDict).subs({g_i: P*g_e})
```

```python
I_t_check
```

```python
f = lambdify((Vm,P,g_e, delta_i,t), I_t_check/pA, (unitsDict, "numpy"))
```

### Clamping voltage at -40mv

```python
clamp_voltage = -40
prop = 3
delayRange = [d(minDelay,k,e*nS) for e in erange[::10]]
ax = plt.subplot()
for index, e in enumerate(erange[::10]):
    I_clamped_t = [float(f(clamp_voltage *mV, prop, e*nS, delayRange[index], dt * ms)) for dt in trange]
    ax.plot(trange, I_clamped_t)
    ax.set_xlabel("Time (ms)")
    ax.set_ylabel("Current (pA)")
```

### Clamping at -70

```python
clamp_voltage = -70
prop = 3
delayRange = [d(minDelay,k,e*nS) for e in erange[::10]]
ax = plt.subplot()
for index, e in enumerate(erange[::10]):
    I_clamped_t = [float(f(clamp_voltage *mV, prop, e*nS, delayRange[index], dt * ms)) for dt in trange]
    ax.plot(trange, I_clamped_t)
    ax.set_xlabel("Time (ms)")
    ax.set_ylabel("Current (pA)")
```

### Clamping at 0 mV

```python
clamp_voltage = 0
prop = 3
delayRange = [d(minDelay,k,e*nS) for e in erange[::10]]
ax = plt.subplot()
for index, e in enumerate(erange[::10]):
    I_clamped_t = [float(f(clamp_voltage *mV, prop, e*nS, delayRange[index], dt * ms)) for dt in trange]
    ax.plot(trange, I_clamped_t)
    ax.set_xlabel("Time (ms)")
    ax.set_ylabel("Current (pA)")
```

### Overplotting 0 and -40

```python
prop = 3
delayRange = [d(minDelay,k,e*nS) for e in erange[::5]]
ax = plt.subplot()
baseCurrent_0 = float(f(clamp_voltage *mV, 0, 0, 0, 0 * ms))
for index, e in enumerate(erange[::5]):
    clamp_voltage = 0
    delay = d(minDelay,k,e*nS)
    I_clamped_t = [float(f(clamp_voltage *mV, prop, e*nS, delay, dt * ms)) for dt in trange]
    ax.plot(trange, I_clamped_t, color=s_m.to_rgba(e))
    maxCurrent = numpy.max(I_clamped_t)
    clamp_voltage = -40
    I_clamped_t = [float(f(clamp_voltage *mV, prop, e*nS, delay, dt * ms)) for dt in trange]
    ax.plot(trange, I_clamped_t, color=s_m.to_rgba(e))
    minCurrent = numpy.min(I_clamped_t)
    ax.vlines(delay/ms, minCurrent,maxCurrent, color=s_m.to_rgba(e), linestyle='--')
ax.set_xlabel("Time (ms)")
ax.set_ylabel("Current (pA)")
```

---

## <span style="color:red">  Try out the following: </span>
1. Estimate how many more synapses can be accommodated because of Divisive normalization before spiking.
2. Check out spike time changes because of DN.

## Trying out other kinetic changes.

### Changing $\tau_{id} = k\times{g_e}$

```python
tp = lambda rise, decay: ((decay*rise)/(decay-rise))*log(decay/rise)
```

```python
k, rise = 10 * ms / nS, 3
plt.scatter(erange, [tp(rise, k * e * nS / ms) for e in erange])
plt.xlabel("$g_e$")
plt.ylabel("$t_{peak}$")
plt.show()
```

```python
check_vm = simplify(Vm_t[0].subs({
    i: averageEstimateDict[i]
    for i in averageEstimateDict if i not in [g_e, g_i, i_d]
}).subs(approximateDict).subs({
    g_i: P * g_e,
    i_d: k * g_e
})).evalf()
```

```python
f = lambdify((g_e, P, t), check_vm/mV, (unitsDict, "numpy"))
```

```python
for e in erange:
    plt.plot(
        trange, [f(e * nS, 1., dt * ms) for dt in trange],
        label="$g_e={}nS, \\delta_i={:.2f}ms$".format(e, k * e / nS))
plt.legend()
plt.show()
```

```python
fig, ax = plt.subplots(1, 2)
for p in np.arange(1, 4, 1):
    v_max = []
    e_max = []
    v_ttp = []
    e_ttp = []
    for e in erange:
        vm_change = [f(e * nS, p, dt * ms) for dt in trange]
        em_change = [f(e * nS, 0., dt * ms) for dt in trange]
        v_max.append(max(vm_change) - approximateDict[leak_rev] / mV)
        e_max.append(max(em_change) - approximateDict[leak_rev] / mV)
        v_ttp.append(np.argmax(vm_change) * timeStep)
        e_ttp.append(np.argmax(em_change) * timeStep)
    ax[0].scatter(e_max, v_max, label=str(p))
    ax[1].scatter(e_ttp, v_ttp, label=str(p))
ax[0].set_xlabel("Excitation $V_{max}$")
ax[0].set_ylabel("Control  $V_{max}$")
ax[1].set_xlabel("Excitation $t_{peak}$")
ax[1].set_ylabel("Control  $t_{peak}$")
ax[0].legend()
ax[1].legend()
plt.show()
```

```python
f(10,1,50)
```

```python
for new_P in np.arange(1, 3, 0.5):
    v_max = []
    for e in erange:
        f = lambdify(g_e, P, vm_change, "numpy")
        trange = np.arange(0, 200, 0.1)
        vm_change = [
            check_vm_t.subs({
                t: dt,
                g_e: new_g_e,
                P: new_P
            }) for dt in trange
        ]
        v_max.append(max(vm_change) - leak_rev.subs(approximateDict))
    plt.scatter(erange, v_max)
    plt.show()
```

```python

```

```python
compartment.subs(averageEstimateDict).subs(approximateDict)
```

### Now trying to change parameters and look at how $V_m$ changes

```python
# 1. Get rid of the normalization factor.
# 2. The functions must approximate each other when close to 0.
# 3. The precision of the parameters in the equations must increase with g_e.
```

```python
g_e.subs(averageEstimateDict)
```

```python
A = Piecewise((0, t < delta_i), (exp(t/(2*ms)), True))
```

```python
A = A.subs({delta_i:10*ms})
```

```python
f = lambdify(t,A, {'s':seconds, 'exp':exp})
```


```
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```
xy = [[1,1],[0,0],[2,2],[1.5,1.5],[1,1.5],[4.5,4.5],[5,4],[4.5,5],[5,5],[5.5,6]]
```

```
xy = np.array(xy)
print(len(xy))
```

    10

```
from sklearn.datasets import make_blobs
```

```
plt.figure(figsize=(80, 80))
plt.subplots_adjust(bottom=.05, top=.9, left=.05, right=.95)
```

    <matplotlib.figure.Figure at 0x7f579b57e9e8>

```
def plot_kmeans(xy, c12):
    plt.scatter(xy[:,0], xy[:,1])
    plt.scatter(c12[:,0], c12[:,1], marker="x", color='r')
    plt.show()

def find_distance(xy,c12):
    dist0 = np.linalg.norm(xy-c12[0],axis=1,ord=2)
    dist1 = np.linalg.norm(xy-c12[1],axis=1,ord=2)
    return (dist0, dist1)

def find_groups(D):
    S = D[0]<=D[1]
    return S.astype('float')

def update_centroids(xy, S):
    S0 = np.where(S==0)
    S1 = np.where(S==1)
    c12[0] = (np.sum(xy[S0],axis=0))/len(S0[0])
    c12[1] = (np.sum(xy[S1],axis=0))/len(S1[0])
    return c12

```

```
plt.subplot(111)
plt.title("Three blobs", fontsize='small')
X1, Y1 = make_blobs(n_features=2, centers=2)
plt.scatter(X1[:, 0], X1[:, 1], marker='o',
            s=25, edgecolor='k')

```

```
xy = X1
```

```
plt.scatter(xy[:,0], xy[:,1])
```

```
c12 = np.array([[0.1,0.1],[0.3,0.4]])
```

```
xy
```

    array([[-5.98552153,  1.10083778],
           [ 6.15039215,  8.90953492],
           [ 5.35665914,  9.0910911 ],
           [ 4.6042203 ,  8.91352351],
           [ 4.53175404, 10.59751617],
           [-4.33433448,  1.9429219 ],
           [ 4.45962568,  7.70591648],
           [ 4.12910695,  9.37129784],
           [ 4.45053566,  9.54600667],
           [ 2.85530716,  8.31066207],
           [-3.63825247,  2.24387085],
           [-5.14941688,  1.19500984],
           [-2.9822534 ,  0.15111023],
           [-4.60613923,  2.18337533],
           [-5.22103376,  2.85677731],
           [ 5.50289441, 10.05942164],
           [ 4.54712127,  9.55950665],
           [-5.06868523,  3.85904361],
           [-4.56534422,  2.01953067],
           [-4.86138034,  1.66443329],
           [ 3.56282222,  9.94687024],
           [ 4.83895483,  9.38991621],
           [-4.47296449,  0.04569408],
           [-6.03396241,  1.00672275],
           [-4.7043615 ,  1.37025107],
           [ 5.91271306,  7.90647646],
           [ 4.36116331,  8.07381589],
           [ 7.16424637,  8.94847638],
           [ 6.65257377,  8.16541789],
           [ 7.41335119,  7.29020561],
           [-5.99293167, -0.7143299 ],
           [-5.98462669,  2.64272068],
           [-3.64713185,  3.09579813],
           [-4.03826746,  1.4183213 ],
           [-4.49759756,  1.53368305],
           [-3.9083969 ,  2.27754467],
           [ 7.31120537,  8.60962912],
           [ 5.63167547,  9.78482006],
           [-3.54029157,  0.77806601],
           [ 4.78023149,  9.14341112],
           [ 6.26072749,  8.67388592],
           [ 6.53184304,  8.11899944],
           [ 5.25750631,  7.41303217],
           [ 6.59412703,  9.03235156],
           [ 4.39212436,  7.74862078],
           [-3.18515656,  2.13374273],
           [-5.08676032,  3.22079237],
           [-5.64525102,  1.55331984],
           [-4.99869331,  0.96259152],
           [-5.77992368,  2.8741915 ],
           [ 5.32776006,  8.75831674],
           [-4.09242291,  2.84999526],
           [-4.18811361,  2.27307789],
           [ 5.82448288,  8.36221744],
           [ 5.59733875,  9.11301418],
           [-4.68872483,  0.98874539],
           [ 4.65148858,  9.30311004],
           [ 5.2564886 ,  7.49779862],
           [ 5.64800084,  9.63463326],
           [ 5.15568613, 10.18866991],
           [-5.25668489,  2.14748686],
           [ 6.38837256,  7.83554381],
           [ 7.06379343, 10.71927195],
           [-4.70780818,  2.76192369],
           [ 6.585827  ,  8.6395715 ],
           [-4.57492718,  2.77064404],
           [-4.59084892,  1.35769311],
           [-6.21861938,  0.69743312],
           [-3.33176226,  1.02641479],
           [-5.13774495,  2.01284055],
           [ 5.21390084,  8.55028039],
           [ 7.20152797, 10.48149493],
           [-5.73672611,  2.99584402],
           [ 7.19524756, 10.08922852],
           [-3.71832086,  1.60698633],
           [-3.14264371,  2.63317088],
           [-3.765537  , -1.00149139],
           [-5.58844091,  2.27352254],
           [-3.68910326,  2.59570052],
           [ 4.90527853,  9.51859501],
           [-5.79278445,  1.63470408],
           [-5.82961177,  0.15667849],
           [ 7.18617845,  7.92999582],
           [-4.8521674 ,  1.73945939],
           [ 5.88408126,  9.50084489],
           [ 6.42359373, 10.42418634],
           [ 4.49049826,  7.47774918],
           [ 5.89953152,  8.34106368],
           [-5.22005168,  1.2922353 ],
           [-4.29100622,  1.33900177],
           [ 3.30252075, 10.21755089],
           [-3.85326611,  0.8383688 ],
           [-4.73663221,  0.47313076],
           [ 5.02064176,  8.53536357],
           [ 3.10776831,  9.09636456],
           [-4.86158041, -0.10908537],
           [ 3.52495096,  8.05238334],
           [ 6.15697584,  8.47342656],
           [ 5.45218697,  9.12043298],
           [-5.40340294,  0.19698647]])

```
plot_kmeans(xy, c12)
```

```
print(min(min(xy[0]), min(xy[1])))
print(max(max(xy[0]), max(xy[1])))

min_v = min(min(xy[0]), min(xy[1]))
max_v = max(max(xy[0]), max(xy[1]))
```

    -5.985521525420311
    8.909534919888202

```
from collections import defaultdict
```

```
C = defaultdict(list)
```

```
for i in range(10):
    D = find_distance(xy,c12)
    G = find_groups(D)
    c12 = update_centroids(xy, G)
    print(c12)
    C[str(i)] = c12
    print(D,G,c12)
    plot_kmeans(xy, c12)
```

```
c12
```

    array([[-4.70395221,  1.61934976],
           [ 5.43434007,  8.92343028]])

```
print(np.mean(c12,axis=0))
```

    [0.36519393 5.27139002]

\begin{equation}
ax+by=c\\
by = -ax+c\\
y = 1/b*(-ax+c)
\end{equation}

```
p1, p2 = c12
print(p1)
print(p2)
```

    [-4.70395221  1.61934976]
    [5.43434007 8.92343028]

```

```

```
def find_slope(p1, p2):
    x1, y1 = p1
    x2, y2 = p2
#     print(y2, y1)
#     print(x2, x1)
    m = (y2-y1)/(x2-x1)
    return m
```

```
def perpendicular_slope(m1):
    return -(1/m1)
```

```
# needed to find the line passing throgh the points
# needed to find the perpendicular line passing through a point

def get_c(slope, point):
    c = point[1] - slope*point[0]
    return c

```

```
def line_passing_through(point, slope):
    return get_c(slope, point)
```

```
plt.gcf().clear()
m = find_slope(c12[0], c12[1])

c = line_passing_through(c12[0], m)
c = line_passing_through(c12[1], m)

x = np.linspace(min_v, max_v, 100, dtype='float')
y = m*x+c
plt.plot(x,y)
plt.hold(True)

m1 = m
m2 = perpendicular_slope(m1)
c1 = get_c(m2, c12[0])
y = m2*x+c1
plt.plot(x,y)
plt.hold(True)

c2 = get_c(m2, c12[1])
y = m2*x+c2
plt.plot(x,y)
plt.hold(True)

cm = get_c(m2, np.mean(c12, axis=0))
y = m2*x+cm
plt.plot(x,y)
plt.hold(True)

plot_kmeans(xy, c12)

```

```

```


# Semi-Monocoque Theory

```python
from pint import UnitRegistry
import sympy
import networkx as nx
import numpy as np
import matplotlib.pyplot as plt
import sys
%matplotlib inline
from IPython.display import display
```

Import **Section** class, which contains all calculations

```python
from Section import Section
```

Initialization of **sympy** symbolic tool and **pint** for dimension analysis (not really implemented rn as not directly compatible with sympy)

```python
ureg = UnitRegistry()
sympy.init_printing()
```

Define **sympy** parameters used for geometric description of sections

```python
A, A0, t, t0, a, b, h, L = sympy.symbols('A A_0 t t_0 a b h L', positive=True)
```

We also define numerical values for each **symbol** in order to plot scaled section and perform calculations

```python
values = [(A, 400 * ureg.millimeter**2),(A0, 250  * ureg.millimeter**2),(a, 400 * ureg.millimeter), \
          (b, 300 * ureg.millimeter),(h, 150 * ureg.millimeter),(L, 650 * ureg.millimeter),(t, 3 * ureg.millimeter)]
datav = [(v[0],v[1].magnitude) for v in values]
```

# Multiconnected Section

Define graph describing the section:

1) **stringers** are **nodes** with parameters:
- **x** coordinate
- **y** coordinate
- **Area**

2) **panels** are **oriented edges** with parameters:
- **thickness**
- **lenght** which is automatically calculated

```python
stringers = {1:[(4*a,2*a),A],
             2:[(a,2*a),A],
             3:[(sympy.Integer(0),a),A],
             4:[(a,sympy.Integer(0)),A],
             5:[(2*a,a),A],
             6:[(4*a,sympy.Integer(0)),A]}

panels = {(1,2):t,
          (2,3):t,
          (3,4):t,
          (4,5):t,
          (5,2):t,
          (4,6):t,
          (6,1):t}
```

Define section and perform first calculations

```python
S3 = Section(stringers, panels)
```

As we need to compute $x_{sc}$, we have to perform

$$A \cdot q_{ext} = T$$

where:
- **A** is a matrix with **number of nodes + number of loops** rows and **number of edges +1** columns (it is square)
- **q** is a column vector of unknowns: **#edges fluxes** and **shear center** coordinate
- **T** is the vector of known terms: $-\frac{T_y}{J_x} \cdot S_{x_i}$ or $-\frac{T_x}{J_y} \cdot S_{y_i}$ for *n-1* nodes and the rest are 0

**Expression of A**

```python
sympy.simplify(S3.A)
```

**Expression of T**

```python
sympy.simplify(S3.T)
```

**Resulting fluxes and coordinate**

```python
sympy.simplify(S3.tempq)
```

## Plot of **S3** section in original reference frame

```python
start_pos={ii: [float(S3.g.node[ii]['ip'][i].subs(datav)) for i in range(2)] for ii in S3.g.nodes() }
```

```python
plt.figure(figsize=(12,8),dpi=300)
nx.draw(S3.g,with_labels=True, arrows= True, pos=start_pos)
plt.arrow(0,0,20,0)
plt.arrow(0,0,0,20)
#plt.text(0,0, 'CG', fontsize=24)
plt.axis('equal')
plt.title("Section in starting reference Frame",fontsize=16);
```

Expression of **Inertial properties** wrt Center of Gravity in with original rotation

```python
S3.Ixx0, S3.Iyy0, S3.Ixy0, S3.α0
```

## Plot of **S3** section in inertial reference Frame

Section is plotted wrt **center of gravity** and rotated (if necessary) so that *x* and *y* are principal axes.
**Center of Gravity** and **Shear Center** are drawn

```python
positions={ii: [float(S3.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S3.g.nodes() }
```

```python
x_ct, y_ct = S3.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S3.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);
```

Expression of **inertial properties** in *principal reference frame*

```python
S3.Ixx, S3.Iyy, S3.Ixy, S3.θ
```

## **Shear center** expression

```python
S3.ct
```

## Loops detection

```python
S3.cycles
```

## Torsional moment of inertia

```python
Tx, Ty, Nz, Mx, My, Mz, F, ry, ry, mz = sympy.symbols('T_x T_y N_z M_x M_y M_z F r_y r_x m_z')
S3.set_loads(_Tx=0, _Ty=1, _Nz=0, _Mx=Mx, _My=0, _Mz=0)
S3.compute_Jt()
S3.Jt
```

```python

```


# Lightning Review of Class Imbalance Techniques

There is no definitive answer to the question: What should one do when dealing with imbalanced data?
The answer will depend on the data in question.
Here are a few approaches one might try to improve performance on an unbalanced dataset.
It is common to discuss these topics in a job interview.

## Metrics to use

Using the accuracy of a classifier as a metric can be misleading because a model could predict that every sample belongs to the majority class and still achieve high accuracy if the data is imbalanced enough.
For this reason accuracy is not a recommended metric in this setting.
The ROC curve is a good general purpose metric, providing information about the true and false positive rates across a range of thresholds, and the area under the ROC curve (AUC) is a good general purpose, single number metric.
The AUC score focuses on true negatives, which are less informative than false positives and false negatives when the data is imbalanced.
Therefore we recommend the precision-recall curve as the metric to use on imbalanced data, or the average precision score if a single number if preferred.

## Balance the training set

One might also try to balance the training set either by undersampling the majority class, oversampling the minority class, or by synthesizing new minority examples.
Oversampling runs the risk of overfitting, and training with oversampling takes longer because of the additional data.
For these reasons we will focus on undersampling in this work.
In particular, we will use random undersampling to create a balanced random forest, and balanced batches to feed into a neural network.
The algorithm for how the balanced random forest makes classifications is as follows:

Image Credit: [Tom Fawcett, Silicon Valley Data Science](https://www.svds.com/learning-imbalanced-classes/).
It is the second step of this process that is absence in a standard random forest.
Even if this does not lead to a gain in performance training is faster with this approach because less data is used.

## Change the algorithm

Lastly, one might consider making changes to the algorithms being used.
A simple example of this is if a metric such as precision, recall, or F1 score is being used, its decision threshold can be optimized to maximize performance.
Another approach is to adjust the class weights in the loss function, creating a larger penalty for misclassifying an example from the minority class.
Consider the cross entropy loss function used for binary classification
\begin{equation}
L = - y \log(p) - (1 - y) \log(1 - p) ,
\end{equation}
where $y$ is the ground-truth class with $y = 1$ for the LL class, and $p$ is the model's estimated probability that a given event belong to the LL class.
Weighting the loss function can be implemented as follows
\begin{equation}
L = - \alpha\, y \log(p) - (1 - \alpha) (1 - y) \log(1 - p) ,
\end{equation}
where the class weight hyperparameter, $\alpha$, takes values between 0 and 1 in this notation.
(For tree based models, such as random forest, our baseline loss function is the Gini impurity not the cross entropy. However the logic of modifying the baseline loss function is the same.)
[arxiv:1708.02002](https://arxiv.org/abs/1708.02002) took this one step further, introducing the _focal loss_ function
\begin{equation}
L = - \alpha\, y (1 - p)^{\gamma} \log(p) - (1 - \alpha) (1 - y) p^{\gamma} \log(1 - p) ,
\end{equation}
where the focusing parameter, $\gamma > 0$, puts the focus on hard, misclassified examples.
In doing so it down weights the easy-to-classify negatives that can overwhelm a model during training in the presence of class imbalance.

# imbalanced-learn

There is a software package, [imbalanced-learn](https://imbalanced-learn.org/en/stable/index.html), dedicated to the task of learning from imbalanced data.

```
conda install -c conda-forge imbalanced-learn
```

# Case Study

```python
from imblearn.datasets import fetch_datasets
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from imblearn.ensemble import BalancedRandomForestClassifier
```

Use a utility from `imblearn` to fetch the 27 datasets used in this demonstration

```python
datasets = fetch_datasets('datasets/')
```

We consider 3 models:
- Random Forest
- Weighted Random Forest - loss function has weights associated with each class. We use the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))
- Balanced Random Forest - randomly under-samples each boostrap sample to balance it.

Each forest has 100 trees with a maximum depth of 5. No hyperparameter tuning has been performed. Below are dictionaries to store the results for each model

```python
rfc_results = dict()
wrfc_results = dict()
brfc_results = dict()
```

Run our three models on the 27 datasets with a five-fold cross validation, scoring them on average precision

```python
def cv_analysis(dataset):
    name = dataset['DESCR']
    X = dataset['data']
    y = dataset['target']
    print(name + '\n----------')

    rfc = RandomForestClassifier(n_estimators=100,
                                 max_depth=5)
    rfc_results[name] = cross_validate(rfc,
                                       X,
                                       y,
                                       scoring='average_precision',
                                       cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=10),
                                       return_train_score=False)
    print('Random Forest - average precision: %0.3f +/- %0.3f'
          %(np.mean(rfc_results[name]['test_score']), np.std(rfc_results[name]['test_score'])))

    wrfc = RandomForestClassifier(n_estimators=100,
                                  max_depth=5,
                                  class_weight='balanced')
    wrfc_results[name] = cross_validate(wrfc,
                                        X,
                                        y,
                                        scoring='average_precision',
                                        cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=10),
                                        return_train_score=False)
    print('Weighted Random Forest - average precision: %0.3f +/- %0.3f'
          %(np.mean(wrfc_results[name]['test_score']), np.std(wrfc_results[name]['test_score'])))

    brfc = BalancedRandomForestClassifier(n_estimators=100,
                                          max_depth=5)
    brfc_results[name] = cross_validate(brfc,
                                        X,
                                        y,
                                        scoring='average_precision',
                                        cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=10),
                                        return_train_score=False)
    print('Balanced Random Forest - average precision: %0.3f +/- %0.3f'
          %(np.mean(brfc_results[name]['test_score']), np.std(brfc_results[name]['test_score'])))
    print('\n')
```

Printing mean +/- standard deviation of the cross validation

```python
for dataset in datasets:
    cv_analysis(datasets[dataset])
```

    ecoli
    ----------
    Random Forest - average precision: 0.734 +/- 0.070
    Weighted Random Forest - average precision: 0.631 +/- 0.135
    Balanced Random Forest - average precision: 0.651 +/- 0.106

    optical_digits
    ----------
    Random Forest - average precision: 0.936 +/- 0.014
    Weighted Random Forest - average precision: 0.927 +/- 0.023
    Balanced Random Forest - average precision: 0.931 +/- 0.027

    satimage
    ----------
    Random Forest - average precision: 0.709 +/- 0.028
    Weighted Random Forest - average precision: 0.694 +/- 0.026
    Balanced Random Forest - average precision: 0.691 +/- 0.030

    pen_digits
    ----------
    Random Forest - average precision: 0.988 +/- 0.003
    Weighted Random Forest - average precision: 0.989 +/- 0.002
    Balanced Random Forest - average precision: 0.990 +/- 0.004

    abalone
    ----------
    Random Forest - average precision: 0.341 +/- 0.034
    Weighted Random Forest - average precision: 0.334 +/- 0.029
    Balanced Random Forest - average precision: 0.342 +/- 0.048

    sick_euthyroid
    ----------
    Random Forest - average precision: 0.839 +/- 0.040
    Weighted Random Forest - average precision: 0.850 +/- 0.053
    Balanced Random Forest - average precision: 0.859 +/- 0.051

    spectrometer
    ----------
    Random Forest - average precision: 0.887 +/- 0.101
    Weighted Random Forest - average precision: 0.911 +/- 0.078
    Balanced Random Forest - average precision: 0.896 +/- 0.047

    car_eval_34
    ----------
    Random Forest - average precision: 0.936 +/- 0.022
    Weighted Random Forest - average precision: 0.961 +/- 0.024
    Balanced Random Forest - average precision: 0.959 +/- 0.017

    isolet
    ----------
    Random Forest - average precision: 0.837 +/- 0.035
    Weighted Random Forest - average precision: 0.811 +/- 0.042
    Balanced Random Forest - average precision: 0.841 +/- 0.031

    us_crime
    ----------
    Random Forest - average precision: 0.569 +/- 0.072
    Weighted Random Forest - average precision: 0.551 +/- 0.034
    Balanced Random Forest - average precision: 0.554 +/- 0.061

    yeast_ml8
    ----------
    Random Forest - average precision: 0.090 +/- 0.017
    Weighted Random Forest - average precision: 0.101 +/- 0.021
    Balanced Random Forest - average precision: 0.105 +/- 0.019

    scene
    ----------
    Random Forest - average precision: 0.300 +/- 0.045
    Weighted Random Forest - average precision: 0.279 +/- 0.021
    Balanced Random Forest - average precision: 0.308 +/- 0.036

    libras_move
    ----------
    Random Forest - average precision: 0.878 +/- 0.115
    Weighted Random Forest - average precision: 0.891 +/- 0.167
    Balanced Random Forest - average precision: 0.820 +/- 0.187

    thyroid_sick
    ----------
    Random Forest - average precision: 0.879 +/- 0.038
    Weighted Random Forest - average precision: 0.852 +/- 0.019
    Balanced Random Forest - average precision: 0.842 +/- 0.039

    coil_2000
    ----------
    Random Forest - average precision: 0.177 +/- 0.014
    Weighted Random Forest - average precision: 0.175 +/- 0.016
    Balanced Random Forest - average precision: 0.167 +/- 0.015

    arrhythmia
    ----------
    Random Forest - average precision: 0.710 +/- 0.105
    Weighted Random Forest - average precision: 0.669 +/- 0.065
    Balanced Random Forest - average precision: 0.577 +/- 0.134

    solar_flare_m0
    ----------
    Random Forest - average precision: 0.224 +/- 0.082
    Weighted Random Forest - average precision: 0.212 +/- 0.077
    Balanced Random Forest - average precision: 0.243 +/- 0.088

    oil
    ----------
    Random Forest - average precision: 0.559 +/- 0.114
    Weighted Random Forest - average precision: 0.629 +/- 0.132
    Balanced Random Forest - average precision: 0.521 +/- 0.170

    car_eval_4
    ----------
    Random Forest - average precision: 0.886 +/- 0.044
    Weighted Random Forest - average precision: 0.983 +/- 0.023
    Balanced Random Forest - average precision: 0.950 +/- 0.033

    wine_quality
    ----------
    Random Forest - average precision: 0.292 +/- 0.033
    Weighted Random Forest - average precision: 0.263 +/- 0.015
    Balanced Random Forest - average precision: 0.256 +/- 0.049

    letter_img
    ----------
    Random Forest - average precision: 0.911 +/- 0.012
    Weighted Random Forest - average precision: 0.926 +/- 0.005
    Balanced Random Forest - average precision: 0.920 +/- 0.006

    yeast_me2
    ----------
    Random Forest - average precision: 0.439 +/- 0.130
    Weighted Random Forest - average precision: 0.392 +/- 0.081
    Balanced Random Forest - average precision: 0.368 +/- 0.050

    webpage
    ----------
    Random Forest - average precision: 0.723 +/- 0.043
    Weighted Random Forest - average precision: 0.511 +/- 0.052
    Balanced Random Forest - average precision: 0.458 +/- 0.037

    ozone_level
    ----------
    Random Forest - average precision: 0.288 +/- 0.076
    Weighted Random Forest - average precision: 0.328 +/- 0.101
    Balanced Random Forest - average precision: 0.340 +/- 0.101

    mammography
    ----------
    Random Forest - average precision: 0.701 +/- 0.023
    Weighted Random Forest - average precision: 0.645 +/- 0.039
    Balanced Random Forest - average precision: 0.650 +/- 0.053

    protein_homo
    ----------
    Random Forest - average precision: 0.793 +/- 0.013
    Weighted Random Forest - average precision: 0.851 +/- 0.015
    Balanced Random Forest - average precision: 0.844 +/- 0.013

    abalone_19
    ----------
    Random Forest - average precision: 0.069 +/- 0.056
    Weighted Random Forest - average precision: 0.051 +/- 0.021
    Balanced Random Forest - average precision: 0.040 +/- 0.018

```python
rfc_mean = np.array([np.mean(rfc_results[dataset]['test_score']) for dataset in datasets])
rfc_std = np.array([np.std(rfc_results[dataset]['test_score']) for dataset in datasets])
wrfc_mean = np.array([np.mean(wrfc_results[dataset]['test_score']) for dataset in datasets])
wrfc_std = np.array([np.std(wrfc_results[dataset]['test_score']) for dataset in datasets])
brfc_mean = np.array([np.mean(brfc_results[dataset]['test_score']) for dataset in datasets])
brfc_std = np.array([np.std(brfc_results[dataset]['test_score']) for dataset in datasets])
```

Plotting the results:
- Fig.1 (9 of 27 cases) - datasets where both weighting and balancing improve the average precision
- Fig.2 (12 of 27 cases) - datasets where neither weighting nor balancing improve the average precision
- Fig.3 (6 of 27 cases) - datasets where each weighting or balancing improve the average precision

The improvement is determined based on the mean value of the cross validation. The black error bars give the standard deviation of the cross validation

```python
import matplotlib.pyplot as plt
%matplotlib notebook
```

```python
plt.figure(1)

improves = (wrfc_mean > rfc_mean) & (brfc_mean > rfc_mean)

plt.bar(range(0, 4*np.sum(improves)+0, 4),
        rfc_mean[improves],
        yerr=rfc_std[improves],
        label='Random Forest')
plt.bar(range(1, 4*np.sum(improves)+1, 4),
        wrfc_mean[improves],
        yerr=wrfc_std[improves],
        label='Weighted Random Forest')
plt.bar(range(2, 4*np.sum(improves)+2, 4),
        brfc_mean[improves],
        yerr=brfc_std[improves],
        label='Balanced Random Forest')

plt.xticks(range(1, 4*np.sum(improves)+1, 4),
           np.array(list(rfc_results.keys()))[improves],
           rotation=20)
plt.ylabel('Average Precision')
plt.ylim(0.0, 1.3)
plt.legend()
plt.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

```python
plt.figure(2)

no_impro = (wrfc_mean < rfc_mean) & (brfc_mean < rfc_mean)

plt.bar(range(0, 4*np.sum(no_impro)+0, 4),
        rfc_mean[no_impro],
        yerr=rfc_std[no_impro],
        label='Random Forest')
plt.bar(range(1, 4*np.sum(no_impro)+1, 4),
        wrfc_mean[no_impro],
        yerr=wrfc_std[no_impro],
        label='Weighted Random Forest')
plt.bar(range(2, 4*np.sum(no_impro)+2, 4),
        brfc_mean[no_impro],
        yerr=brfc_std[no_impro],
        label='Balanced Random Forest')

plt.xticks(range(1, 4*np.sum(no_impro)+1, 4),
           np.array(list(rfc_results.keys()))[no_impro],
           rotation=20)
plt.ylabel('Average Precision')
plt.ylim(0.0, 1.3)
plt.legend()
plt.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

```python
plt.figure(3)

no_impro = (wrfc_mean > rfc_mean) ^ (brfc_mean > rfc_mean)

plt.bar(range(0, 4*np.sum(no_impro)+0, 4),
        rfc_mean[no_impro],
        yerr=rfc_std[no_impro],
        label='Random Forest')
plt.bar(range(1, 4*np.sum(no_impro)+1, 4),
        wrfc_mean[no_impro],
        yerr=wrfc_std[no_impro],
        label='Weighted Random Forest')
plt.bar(range(2, 4*np.sum(no_impro)+2, 4),
        brfc_mean[no_impro],
        yerr=brfc_std[no_impro],
        label='Balanced Random Forest')

plt.xticks(range(1, 4*np.sum(no_impro)+1, 4),
           np.array(list(rfc_results.keys()))[no_impro],
           rotation=20)
plt.ylabel('Average Precision')
plt.ylim(0.0, 1.3)
plt.legend()
plt.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

```python

```

```python

```