# Neuromatch Academy: Week 3, Day 2, Tutorial 1
# Neuronal Network Dynamics: Neural Rate Models

__Content creators:__ Qinglong Gu, Songtin Li, Arvind Kumar, John Murray, Julijana Gjorgjieva

__Content reviewers:__  Maryam Vaziri-Pashkam, Ella Batty, Lorenzo Fontolan, Richard Gao, Spiros Chavlis, Michael Waskom

---
# Tutorial Objectives

The brain is a complex system, not because it is composed of a large number of diverse types of neurons, but mainly because of how neurons are connected to each other. The brain is indeed a network of highly specialized neuronal networks.

The activity of a neural network constantly evolves in time. For this reason, neurons can be modeled as dynamical systems. The dynamical system approach is only one of the many modeling approaches that computational neuroscientists have developed (other points of view include information processing,  statistical models, etc.).

How the dynamics of neuronal networks affect the representation and processing of information in the brain is an open question. However, signatures of altered brain dynamics present in many brain diseases (e.g., in epilepsy or Parkinson's disease) tell us that it is crucial to study network activity dynamics if we want to understand the brain.

In this tutorial, we will simulate and study one of the simplest models of biological neuronal networks. Instead of modeling and simulating individual excitatory neurons (e.g., LIF models that you implemented yesterday), we will treat them as a single homogeneous population and approximate their dynamics using a single one-dimensional equation describing the evolution of their average spiking rate in time.

In this tutorial, we will learn how to build a firing rate model of a single population of excitatory neurons.

**Steps:**
- Write the equation for the firing rate dynamics of a 1D excitatory population.
- Visualize the response of the population as a function of parameters such as threshold level and gain, using the frequency-current (F-I) curve.
- Numerically simulate the dynamics of the excitatory population and find the fixed points of the system.
- Investigate the stability of the fixed points by linearizing the dynamics around them.

---
# Setup

```python
# Imports
import matplotlib.pyplot as plt
import numpy as np
import scipy.optimize as opt       # root-finding algorithm
```

```python
# @title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper functions

def plot_fI(x, f):
  plt.figure(figsize=(6, 4))  # plot the figure
  plt.plot(x, f, 'k')
  plt.xlabel('x (a.u.)', fontsize=14)
  plt.ylabel('F(x)', fontsize=14)
  plt.show()

def plot_dr_r(r, drdt, x_fps=None):
  plt.figure()
  plt.plot(r, drdt, 'k')
  plt.plot(r, 0. * r, 'k--')
  if x_fps is not None:
    plt.plot(x_fps, np.zeros_like(x_fps), "ko", ms=12)
  plt.xlabel(r'$r$')
  plt.ylabel(r'$\frac{dr}{dt}$', fontsize=20)
  plt.ylim(-0.1, 0.1)

def plot_dFdt(x, dFdt):
  plt.figure()
  plt.plot(x, dFdt, 'r')
  plt.xlabel('x (a.u.)', fontsize=14)
  plt.ylabel('dF(x)', fontsize=14)
  plt.show()
```

---
# Section 1: Neuronal network dynamics

```python
# @title Video 1: Dynamic networks
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="p848349hPyw", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=p848349hPyw

## Section 1.1: Dynamics of a single excitatory population

Individual neurons respond by spiking. When we average the spikes of neurons in a population, we can define the average firing activity of the population. In this model, we are interested in how the population-averaged firing varies as a function of time and network parameters. Mathematically, we can describe the firing rate dynamic as:

\begin{align}
\tau \frac{dr}{dt} &= -r + F(w\cdot r + I_{\text{ext}})  \quad\qquad (1)
\end{align}

$r(t)$ represents the average firing rate of the excitatory population at time $t$, $\tau$ controls the timescale of the evolution of the average firing rate, $w$ denotes the strength (synaptic weight) of the recurrent input to the population, $I_{\text{ext}}$ represents the external input, and the transfer function $F(\cdot)$ (which can be related to f-I curve of individual neurons described in the next sections) represents the population activation function in response to all received inputs.

To start building the model, please execute the cell below to initialize the simulation parameters.

```python
# @markdown *Execute this cell to set default parameters for a single excitatory population model*

def default_pars_single(**kwargs):
  pars = {}

  # Excitatory parameters
  pars['tau'] = 1.     # Timescale of the E population [ms]
  pars['a'] = 1.2      # Gain of the E population
  pars['theta'] = 2.8  # Threshold of the E population

  # Connection strength
  pars['w'] = 0.  # E to E, we first set it to 0

  # External input
  pars['I_ext'] = 0.

  # simulation parameters
  pars['T'] = 20.       # Total duration of simulation [ms]
  pars['dt'] = .1       # Simulation time step [ms]
  pars['r_init'] = 0.2  # Initial value of E

  # External parameters if any
  pars.update(kwargs)

  # Vector of discretized time points [ms]
  pars['range_t'] = np.arange(0, pars['T'], pars['dt'])

  return pars

```

You can now use:
- `pars = default_pars_single()` to get all the parameters, and then you can execute `print(pars)` to check these parameters.
- `pars = default_pars_single(T=T_sim, dt=time_step)` to set new simulation time and time step
- To update an existing parameter dictionary, use `pars['New_para'] = value`

Because `pars` is a dictionary, it can be passed to a function that requires individual parameters as arguments using `my_func(**pars)` syntax.

## Section 1.2: F-I curves
In electrophysiology, a neuron is often characterized by its spike rate output in response to input currents. This is often called the **F-I** curve, denoting the output spike frequency (**F**) in response to different injected currents (**I**). We estimated this for an LIF neuron in yesterday's tutorial.

The transfer function $F(\cdot)$ in Equation $1$ represents the gain of the population as a function of the total input. The gain is often modeled as a sigmoidal function, i.e., more input drive leads to a nonlinear increase in the population firing rate. The output firing rate will eventually saturate for high input values.

A sigmoidal $F(\cdot)$ is parameterized by its gain $a$ and threshold $\theta$.

$$ F(x;a,\theta) = \frac{1}{1+\text{e}^{-a(x-\theta)}} - \frac{1}{1+\text{e}^{a\theta}}  \quad(2)$$

The argument $x$ represents the input to the population. Note that the second term is chosen so that $F(0;a,\theta)=0$.

Many other transfer functions (generally monotonic) can be also used. Examples are the rectified linear function $ReLU(x)$ or the hyperbolic tangent $tanh(x)$.

### Exercise 1: Implement F-I curve

Let's first investigate the activation functions before simulating the dynamics of the entire population.

In this exercise, you will implement a sigmoidal **F-I** curve or transfer function $F(x)$, with gain $a$ and threshold level $\theta$ as parameters.

```python
def F(x, a, theta):
  """
  Population activation function.

  Args:
    x (float): the population input
    a (float): the gain of the function
    theta (float): the threshold of the function

  Returns:
    float: the population activation response F(x) for input x
  """

  # Define the sigmoidal transfer function f = F(x)
  f = (1 + np.exp(-a * (x - theta)))**-1 - (1 + np.exp(a * theta))**-1

  return f

pars = default_pars_single()  # get default parameters
x = np.arange(0, 10, .1)      # set the range of input

# Uncomment below to test your function
f = F(x, pars['a'], pars['theta'])
plot_fI(x, f)
```

### Interactive Demo: Parameter exploration of F-I curve
Here's an interactive demo that shows how the F-I curve changes for different values of the gain and threshold parameters. How do the gain and threshold parameters affect the F-I curve?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def interactive_plot_FI(a, theta):
  """
  Population activation function.

  Expecxts:
    a     : the gain of the function
    theta : the threshold of the function

  Returns:
    plot the F-I curve with give parameters
  """

  # set the range of input
  x = np.arange(0, 10, .1)
  plt.figure()
  plt.plot(x, F(x, a, theta), 'k')
  plt.xlabel('x (a.u.)', fontsize=14)
  plt.ylabel('F(x)', fontsize=14)
  plt.show()

_ = widgets.interact(interactive_plot_FI, a=(0.3, 3, 0.3), theta=(2, 4, 0.2))
```

    interactive(children=(FloatSlider(value=1.5, description='a', max=3.0, min=0.3, step=0.3), FloatSlider(value=3…

```python
"""
Discussion:

For the function we have chosen to model the F-I curve (eq 2),
- a determines the slope (gain) of the rising phase of the F-I curve
- theta determines the input at which the function F(x) reaches its mid-value (0.5).
That is, theta shifts the F-I curve along the horizontal axis.

For our neurons we are using in this tutorial:
- a controls the gain of the neuron population
- theta controls the threshold at which the neuron population starts to respond
""";
```

## Section 1.3: Simulation scheme of E dynamics

Because $F(\cdot)$ is a nonlinear function, the exact solution of Equation $1$ can not be determined via analytical methods. Therefore, numerical methods must be used to find the solution. In practice, the derivative on the left-hand side of Equation $1$ can be approximated using the Euler method on a time-grid of stepsize $\Delta t$:

\begin{align}
&\frac{dr}{dt} \approx \frac{r[k+1]-r[k]}{\Delta t}
\end{align}
where $r[k] = r(k\Delta t)$.

Thus,

$$\Delta r[k] = \frac{\Delta t}{\tau}[-r[k] + F(w\cdot r[k] + I_{\text{ext}}(k;a,\theta))]$$

Hence, Equation (1) is updated at each time step by:

$$r[k+1] = r[k] + \Delta r[k]$$

```python
# @markdown *Execute this cell to enable the single population rate model simulator: `simulate_single`*

def simulate_single(pars):
  """
  Simulate an excitatory population of neurons

  Args:
    pars   : Parameter dictionary

  Returns:
    rE     : Activity of excitatory population (array)

  Example:
    pars = default_pars_single()
    r = simulate_single(pars)
  """

  # Set parameters
  tau, a, theta = pars['tau'], pars['a'], pars['theta']
  w = pars['w']
  I_ext = pars['I_ext']
  r_init = pars['r_init']
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # Initialize activity
  r = np.zeros(Lt)
  r[0] = r_init
  I_ext = I_ext * np.ones(Lt)

  # Update the E activity
  for k in range(Lt - 1):
      dr = dt / tau * (-r[k] + F(w * r[k] + I_ext[k], a, theta))
      r[k+1] = r[k] + dr

  return r

help(simulate_single)
```

    Help on function simulate_single in module __main__:

    simulate_single(pars)
        Simulate an excitatory population of neurons

        Args:
          pars   : Parameter dictionary

        Returns:
          rE     : Activity of excitatory population (array)

        Example:
          pars = default_pars_single()
          r = simulate_single(pars)

### Interactive Demo: Parameter Exploration of single population dynamics

Note that $w=0$, as in the default setting, means no recurrent input to the neuron population in Equation (1). Hence, the dynamics are entirely determined by the external input $I_{\text{ext}}$. Explore these dynamics in this interactive demo.

How does $r_{\text{sim}}(t)$ change with different $I_{\text{ext}}$ values? How does it change with different $\tau$ values? Investigate the relationship between $F(I_{\text{ext}}; a, \theta)$ and the steady value of $r(t)$.

Note that, $r_{\rm ana}(t)$ denotes the analytical solution - you will learn how this is computed in the next section.

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

# get default parameters
pars = default_pars_single(T=20.)

def Myplot_E_diffI_difftau(I_ext, tau):
  # set external input and time constant
  pars['I_ext'] = I_ext
  pars['tau'] = tau

  # simulation
  r = simulate_single(pars)

  # Analytical Solution
  r_ana = (pars['r_init']
           + (F(I_ext, pars['a'], pars['theta'])
           - pars['r_init']) * (1. - np.exp(-pars['range_t'] / pars['tau'])))

  # plot
  plt.figure()
  plt.plot(pars['range_t'], r, 'b', label=r'$r_{\mathrm{sim}}$(t)', alpha=0.5,
           zorder=1)
  plt.plot(pars['range_t'], r_ana, 'b--', lw=5, dashes=(2, 2),
           label=r'$r_{\mathrm{ana}}$(t)', zorder=2)
  plt.plot(pars['range_t'],
           F(I_ext, pars['a'], pars['theta']) * np.ones(pars['range_t'].size),
           'k--', label=r'$F(I_{\mathrm{ext}})$')
  plt.xlabel('t (ms)', fontsize=16.)
  plt.ylabel('Activity r(t)', fontsize=16.)
  plt.legend(loc='best', fontsize=14.)
  plt.show()

_ = widgets.interact(Myplot_E_diffI_difftau, I_ext=(0.0, 10., 1.),
                     tau=(1., 5., 0.2))
```

    interactive(children=(FloatSlider(value=5.0, description='I_ext', max=10.0, step=1.0), FloatSlider(value=3.0, …

```python
# exponential increase/decay in activity with time-scale determined by time constant tau
```

```python
"""
Discussion:

Given the choice of F-I curve (eq 2) and dynamics of the neuron population (eq. 1)
the neurons have two fixed points or steady-state responses irrespective of the input.
- Weak inputs to the neurons eventually result in the activity converging to zero
- Strong inputs to the neurons eventually result in the activity converging to max value

The time constant tau, does not affect the steady-state response but it determines
the time the neurons take to reach to their fixed point.
""";
```

## Think!
Above, we have numerically solved a system driven by a positive input and that, if $w_{EE} \neq 0$, receives an excitatory recurrent input (**extra challenge: try changing the value of $w_{EE}$ to a positive number and plotting the results of simulate_single**). Yet, $r_E(t)$ either decays to zero or reaches a fixed non-zero value.
- Why doesn't the solution of the system "explode" in a finite time? In other words, what guarantees that $r_E$(t) stays finite?
- Which parameter would you change in order to increase the maximum value of the response?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

# get default parameters
pars = default_pars_single(T=20.)

def Myplot_E_diffI_difftau(I_ext, tau, w):
  # set external input and time constant
  pars['I_ext'] = I_ext
  pars['tau'] = tau
  pars['w'] = w

  # simulation
  r = simulate_single(pars)

  # Analytical Solution
  r_ana = (pars['r_init']
           + (F(I_ext, pars['a'], pars['theta'])
           - pars['r_init']) * (1. - np.exp(-pars['range_t'] / pars['tau'])))

  # plot
  plt.figure()
  plt.plot(pars['range_t'], r, 'b', label=r'$r_{\mathrm{sim}}$(t)', alpha=0.5,
           zorder=1)
  plt.plot(pars['range_t'], r_ana, 'b--', lw=5, dashes=(2, 2),
           label=r'$r_{\mathrm{ana}}$(t)', zorder=2)
  plt.plot(pars['range_t'],
           F(I_ext, pars['a'], pars['theta']) * np.ones(pars['range_t'].size),
           'k--', label=r'$F(I_{\mathrm{ext}})$')
  plt.xlabel('t (ms)', fontsize=16.)
  plt.ylabel('Activity r(t)', fontsize=16.)
  plt.legend(loc='best', fontsize=14.)
  plt.show()

_ = widgets.interact(Myplot_E_diffI_difftau, I_ext=(0.0, 10., 1.),
                     tau=(1., 5., 0.2), w=(0.0, 1, 0.05))
```

    interactive(children=(FloatSlider(value=5.0, description='I_ext', max=10.0, step=1.0), FloatSlider(value=3.0, …

```python
"""
Discussion:

1) As the F-I curve is bounded between zero and one, the system doesn't explode.
The f-curve guarantees this property

2) One way to increase the maximum response is to change the f-I curve. For
example, the ReLU is an unbounded function, and thus will increase the overall maximal
response of the network.
""";

```

---
# Section 2: Fixed points of the single population system

```python
# @title Video 2: Fixed point
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="Ox3ELd1UFyo", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=Ox3ELd1UFyo

As you varied the two parameters in the last Interactive Demo, you noticed that, while at first the system output quickly changes, with time, it reaches its maximum/minimum value and does not change anymore. The value eventually reached by the system is called the **steady state** of the system, or the **fixed point**. Essentially, in the steady states the derivative with respect to time of the activity ($r$) is zero, i.e. $\displaystyle \frac{dr}{dt}=0$.

We can find that the steady state of the Equation. (1) by setting $\displaystyle{\frac{dr}{dt}=0}$ and solve for $r$:

$$-r_{\text{steady}} + F(w\cdot r_{\text{steady}} + I_{\text{ext}};a,\theta) = 0, \qquad (3)$$

When it exists, the solution of Equation. (3) defines a **fixed point** of the dynamical system in Equation (1). Note that if $F(x)$ is nonlinear, it is not always possible to find an analytical solution, but the solution can be found via numerical simulations, as we will do later.

From the Interactive Demo, one could also notice that the value of $\tau$ influences how quickly the activity will converge to the steady state from its initial value.

In the specific case of $w=0$, we can also analytically compute  the solution of Equation (1) (i.e., the thick blue dashed line) and deduce the role of $\tau$ in determining the convergence to the fixed point:

$$\displaystyle{r(t) = \big{[}F(I_{\text{ext}};a,\theta) -r(t=0)\big{]} (1-\text{e}^{-\frac{t}{\tau}})} + r(t=0)$$ \\

We can now numerically calculate the fixed point with a root finding algorithm.

## Exercise 2: Visualization of the fixed points

When it is not possible to find the solution for Equation (3) analytically, a graphical approach can be taken. To that end, it is useful to plot $\displaystyle{\frac{dr}{dt}}$ as a function of $r$. The values of $r$ for which the plotted function crosses zero on the y axis correspond to fixed points.

Here, let us, for example, set $w=5.0$ and $I^{\text{ext}}=0.5$. From Equation (1), you can obtain

$$\frac{dr}{dt} = [-r + F(w\cdot r + I^{\text{ext}})]\,/\,\tau $$

Then, plot the $dr/dt$ as a function of $r$, and check for the presence of fixed points.

```python
def compute_drdt(r, I_ext, w, a, theta, tau, **other_pars):
  """Given parameters, compute dr/dt as a function of r.

  Args:
    r (1D array) : Average firing rate of the excitatory population
    I_ext, w, a, theta, tau (numbers): Simulation parameters to use
    other_pars : Other simulation parameters are unused by this function

  Returns
    drdt function for each value of r
  """
  # Calculate drdt
  drdt = (-r + F(w * r + I_ext, a, theta)) / tau

  return drdt

# Define a vector of r values and the simulation parameters
r = np.linspace(0, 1, 1000)
pars = default_pars_single(I_ext=0.5, w=5)

drdt = compute_drdt(r, **pars)

plot_dr_r(r, drdt)
```

## Exercise 3: Fixed point calculation

We will now find the fixed points numerically. To do so, we need to specif initial values ($r_{\text{guess}}$) for the root-finding algorithm to start from. From the line $\displaystyle{\frac{dr}{dt}}$ plotted above in Exercise 2, initial values can be chosen as a set of values close to where the line crosses zero on the y axis (real fixed point).

The next cell defines three helper functions that we will use:

- `my_fp_single(r_guess, **pars)` uses a root-finding algorithm to locate a fixed point near a given initial value
- `check_fp_single(x_fp, **pars)`, verifies that the values of $r_{\rm fp}$ for which $\displaystyle{\frac{dr}{dt}} = 0$ are the true fixed points
- `my_fp_finder(r_guess_vector, **pars)` accepts an array of initial values and finds the same number of fixed points, using the above two functions

```python
# @markdown *Execute this cell to enable the fixed point functions*

def my_fp_single(r_guess, a, theta, w, I_ext, **other_pars):
  """
  Calculate the fixed point through drE/dt=0

  Args:
    r_guess  : Initial value used for scipy.optimize function
    a, theta, w, I_ext : simulation parameters

  Returns:
    x_fp    : value of fixed point
  """
  # define the right hand of E dynamics
  def my_WCr(x):
    r = x
    drdt = (-r + F(w * r + I_ext, a, theta))
    y = np.array(drdt)

    return y

  x0 = np.array(r_guess)
  x_fp = opt.root(my_WCr, x0).x.item()

  return x_fp

def check_fp_single(x_fp, a, theta, w, I_ext, mytol=1e-4, **other_pars):
  """
   Verify |dr/dt| < mytol

  Args:
    fp      : value of fixed point
    a, theta, w, I_ext: simulation parameters
    mytol   : tolerance, default as 10^{-4}

  Returns :
    Whether it is a correct fixed point: True/False
  """
  # calculate Equation(3)
  y = x_fp - F(w * x_fp + I_ext, a, theta)

  # Here we set tolerance as 10^{-4}
  return np.abs(y) < mytol

def my_fp_finder(pars, r_guess_vector, mytol=1e-4):
  """
  Calculate the fixed point(s) through drE/dt=0

  Args:
    pars    : Parameter dictionary
    r_guess_vector  : Initial values used for scipy.optimize function
    mytol   : tolerance for checking fixed point, default as 10^{-4}

  Returns:
    x_fps   : values of fixed points

  """
  x_fps = []
  correct_fps = []
  for r_guess in r_guess_vector:
    x_fp = my_fp_single(r_guess, **pars)
    if check_fp_single(x_fp, **pars, mytol=mytol):
      x_fps.append(x_fp)

  return x_fps

help(my_fp_finder)
```

    Help on function my_fp_finder in module __main__:

    my_fp_finder(pars, r_guess_vector, mytol=0.0001)
        Calculate the fixed point(s) through drE/dt=0

        Args:
          pars    : Parameter dictionary
          r_guess_vector  : Initial values used for scipy.optimize function
          mytol   : tolerance for checking fixed point, default as 10^{-4}

        Returns:
          x_fps   : values of fixed points

```python
r = np.linspace(0, 1, 1000)
pars = default_pars_single(I_ext=0.5, w=5)
drdt = compute_drdt(r, **pars)

r_guess_vector = [0, .4, .9]

x_fps = my_fp_finder(pars, r_guess_vector)
print('%.3f '*3 % tuple(x_fps))

plot_dr_r(r, drdt, x_fps)
```

## Interactive Demo: fixed points as a function of recurrent and external inputs.

You can now explore how the previous plot changes when the recurrent coupling $w$ and the external input $I_{\text{ext}}$ take different values. How does the number of fixed points change?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def plot_intersection_single(w, I_ext):
  # set your parameters
  pars = default_pars_single(w=w, I_ext=I_ext)

  # find fixed points
  r_init_vector = [0, .4, .9]
  x_fps = my_fp_finder(pars, r_init_vector)
  print('%.3f '*len(x_fps) % tuple(x_fps))

  # plot
  r = np.linspace(0, 1., 1000)
  drdt = (-r + F(w * r + I_ext, pars['a'], pars['theta'])) / pars['tau']

  plot_dr_r(r, drdt, x_fps)

_ = widgets.interact(plot_intersection_single, w=(0, 7, 0.2),
                     I_ext=(0, 3, 0.1))
```

    interactive(children=(FloatSlider(value=3.0, description='w', max=7.0, step=0.2), FloatSlider(value=1.0, descr…

```python
"""
Discussion:

The fixed points of the single excitatory neuron population are determined by both
recurrent connections w and external input I_ext. In a previous interactive demo
we saw how the system showed two different steady-states when w = 0. But when w
doe not equal 0, for some range of w the system shows three fixed points (the middle
one being unstable) and the steady state depends on the initial conditions (i.e.
r at time zero.).

More on this will be explained in the next section.
""";
```

---
# Summary

In this tutorial, we have investigated the dynamics of a rate-based single population of neurons.

We learned about:
- The effect of the input parameters and the time constant of the network on the dynamics of the population.
- How to find the fixed point(s) of the system.

Next, we have two Bonus, but important concepts in dynamical system analysis and simulation. If you have time left, watch the next video and proceed to solve the exercises. You will learn:

- How to determine the stability of a fixed point by linearizing the system.
- How to add realistic inputs to our model.

---
# Bonus 1: Stability of a fixed point

```python
# @title Video 3: Stability of fixed points
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="KKMlWWU83Jg", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=KKMlWWU83Jg

#### Initial values and trajectories

Here, let us first set $w=5.0$ and $I_{\text{ext}}=0.5$, and investigate the dynamics of $r(t)$ starting with different initial values $r(0) \equiv r_{\text{init}}$. We will plot the trajectories of $r(t)$ with $r_{\text{init}} = 0.0, 0.1, 0.2,..., 0.9$.

```python
# @markdown Execute this cell to see the trajectories!

pars = default_pars_single()
pars['w'] = 5.0
pars['I_ext'] = 0.5

plt.figure(figsize=(8, 5))
for ie in range(10):
  pars['r_init'] = 0.1 * ie  # set the initial value
  r = simulate_single(pars)  # run the simulation

  # plot the activity with given initial
  plt.plot(pars['range_t'], r, 'b', alpha=0.1 + 0.1 * ie,
           label=r'r$_{\mathrm{init}}$=%.1f' % (0.1 * ie))

plt.xlabel('t (ms)')
plt.title('Two steady states?')
plt.ylabel(r'$r$(t)')
plt.legend(loc=[1.01, -0.06], fontsize=14)
plt.show()
```

## Interactive Demo: dynamics as a function of the initial value

Let's now set $r_{\rm init}$ to a value of your choice in this demo. How does the solution change? What do you observe?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

pars = default_pars_single(w=5.0, I_ext=0.5)

def plot_single_diffEinit(r_init):
  pars['r_init'] = r_init
  r = simulate_single(pars)

  plt.figure()
  plt.plot(pars['range_t'], r, 'b', zorder=1)
  plt.plot(0, r[0], 'bo', alpha=0.7, zorder=2)
  plt.xlabel('t (ms)', fontsize=16)
  plt.ylabel(r'$r(t)$', fontsize=16)
  plt.ylim(0, 1.0)
  plt.show()

_ = widgets.interact(plot_single_diffEinit, r_init=(0.1, 1, 0.02))
```

    interactive(children=(FloatSlider(value=0.54, description='r_init', max=1.0, min=0.1, step=0.02), Output()), _…

```python
"""
Discussion:

To better appreciate what is happening here, you should go back to the previous
interactive demo. Set the w = 5 and I_ext = 0.5.

You will find that there are three fixed points of the system for these values of
w and I_ext. Now, choose the initial value in this demo and see in which direction
the system output moves. When r_init is in the vicinity of the leftmost fixed points
it moves towards the left most fixed point. When r_init is in the vicinity of the
rightmost fixed points it moves towards the rightmost fixed point.
""";
```

### Stability analysis via linearization of the dynamics

Just like Equation $1$ in the case ($w=0$) discussed above, a generic linear system
$$\frac{dx}{dt} = \lambda (x - b),$$
has a fixed point for $x=b$. The analytical solution of such a system can be found to be:
$$x(t) = b + \big{(} x(0) - b \big{)} \text{e}^{\lambda t}.$$
Now consider a small perturbation of the activity around the fixed point: $x(0) = b+ \epsilon$, where $|\epsilon| \ll 1$. Will the perturbation $\epsilon(t)$ grow with time or will it decay to the fixed point? The evolution of the perturbation with time can be written, using the analytical solution for $x(t)$, as:
 $$\epsilon (t) = x(t) - b = \epsilon \text{e}^{\lambda t}$$

- if $\lambda < 0$, $\epsilon(t)$ decays to zero, $x(t)$ will still converge to $b$ and the fixed point is "**stable**".

- if $\lambda > 0$, $\epsilon(t)$ grows with time, $x(t)$ will leave the fixed point $b$ exponentially, and the fixed point is, therefore, "**unstable**" .

### Compute the stability of Equation $1$

Similar to what we did in the linear system above, in order to determine the stability of a fixed point $r^{*}$ of the excitatory population dynamics, we perturb Equation (1) around $r^{*}$ by $\epsilon$, i.e. $r = r^{*} + \epsilon$. We can plug in Equation (1) and obtain the equation determining the time evolution of the perturbation $\epsilon(t)$:

\begin{align}
\tau \frac{d\epsilon}{dt} \approx -\epsilon + w F'(w\cdot r^{*} + I_{\text{ext}};a,\theta) \epsilon
\end{align}

where $F'(\cdot)$ is the derivative of the transfer function $F(\cdot)$. We can rewrite the above equation as:

\begin{align}
\frac{d\epsilon}{dt} \approx \frac{\epsilon}{\tau }[-1 + w F'(w\cdot r^* + I_{\text{ext}};a,\theta)]
\end{align}

That is, as in the linear system above, the value of

$$\lambda = [-1+ wF'(w\cdot r^* + I_{\text{ext}};a,\theta)]/\tau \qquad (4)$$

determines whether the perturbation will grow or decay to zero, i.e., $\lambda$ defines the stability of the fixed point. This value is called the **eigenvalue** of the dynamical system.

## Exercise 4: Compute $dF$

The derivative of the sigmoid transfer function is:
\begin{align}
\frac{dF}{dx} & = \frac{d}{dx} (1+\exp\{-a(x-\theta)\})^{-1}  \\
& = a\exp\{-a(x-\theta)\} (1+\exp\{-a(x-\theta)\})^{-2}. \qquad (5)
\end{align}

Let's now find the expression for the derivative $\displaystyle{\frac{dF}{dx}}$ in the following cell and plot it.

```python
def dF(x, a, theta):
  """
  Population activation function.

  Args:
    x     : the population input
    a     : the gain of the function
    theta : the threshold of the function

  Returns:
    dFdx  : the population activation response F(x) for input x
  """

  # Calculate the population activation
  dFdx = a * np.exp(-a * (x - theta)) * (1 + np.exp(-a * (x - theta)))**-2

  return dFdx

pars = default_pars_single()  # get default parameters
x = np.arange(0, 10, .1)      # set the range of input

df = dF(x, pars['a'], pars['theta'])

with plt.xkcd():
  plot_dFdt(x, df)
```

## Exercise 5: Compute eigenvalues

As discussed above, for the case with $w=5.0$ and $I_{\text{ext}}=0.5$, the system displays **three** fixed points. However, when we simulated the dynamics and varied the initial conditions $r_{\rm init}$, we could only obtain **two** steady states. In this exercise, we will now check the stability of each of the three fixed points by calculating the corresponding eigenvalues with the function `eig_single`. Check the sign of each eigenvalue (i.e., stability of each fixed point). How many of the fixed points are stable?

Note that the expression of the eigenvalue at fixed point $r^*$
$$\lambda = [-1+ wF'(w\cdot r^* + I_{\text{ext}};a,\theta)]/\tau$$

**SAMPLE OUTPUT**

```
Fixed point1 at 0.042 with Eigenvalue=-0.583
Fixed point2 at 0.447 with Eigenvalue=0.498
Fixed point3 at 0.900 with Eigenvalue=-0.626
```

```python
def eig_single(fp, tau, a, theta, w, I_ext, **other_pars):
  """
  Args:
    fp   : fixed point r_fp
    tau, a, theta, w, I_ext : Simulation parameters

  Returns:
    eig : eigevalue of the linearized system
  """
  # Compute the eigenvalue
  eig = (-1. + w * dF(w * fp + I_ext, a, theta)) / tau

  return eig

# Find the eigenvalues for all fixed points of Exercise 2
pars = default_pars_single(w=5, I_ext=.5)
r_guess_vector = [0, .4, .9]
x_fp = my_fp_finder(pars, r_guess_vector)

for fp in x_fp:
  eig_fp = eig_single(fp, **pars)
  print(f'Fixed point1 at {fp:.3f} with Eigenvalue={eig_fp:.3f}')
```

    Fixed point1 at 0.042 with Eigenvalue=-0.583
    Fixed point1 at 0.447 with Eigenvalue=0.498
    Fixed point1 at 0.900 with Eigenvalue=-0.626

## Think!
Throughout the tutorial, we have assumed $w> 0 $, i.e., we considered a single population of **excitatory** neurons. What do you think will be the behavior of a population of inhibitory neurons, i.e., where $w> 0$ is replaced by $w< 0$?

```python
"""
Discussion:

You can check this by going back the second last interactive demo and set the
weight to w<0. You will notice that the system has only one fixed point and that
is at zero value. For this particular dynamics, the system will eventually converge
to zero. But try it out.
""";
```

---
# Bonus 2: Noisy input drives the transition between two stable states

## Ornstein-Uhlenbeck (OU) process

As discussed in several previous tutorials, the OU process is usually used to generate a noisy input into the neuron. The OU input $\eta(t)$ follows:

$$\tau_\eta \frac{d}{dt}\eta(t) = -\eta (t) + \sigma_\eta\sqrt{2\tau_\eta}\xi(t)$$

Execute the following function `my_OU(pars, sig, myseed=False)` to generate an OU process.

```python
# @title OU process `my_OU(pars, sig, myseed=False)`

# @markdown Make sure you execute this cell to visualize the noise!

def my_OU(pars, sig, myseed=False):
  """
  A functions that generates Ornstein-Uhlenback process

  Args:
    pars       : parameter dictionary
    sig        : noise amplitute
    myseed     : random seed. int or boolean

  Returns:
    I          : Ornstein-Uhlenbeck input current
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size
  tau_ou = pars['tau_ou']  # [ms]

  # set random seed
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # Initialize
  noise = np.random.randn(Lt)
  I_ou = np.zeros(Lt)
  I_ou[0] = noise[0] * sig

  # generate OU
  for it in range(Lt - 1):
    I_ou[it + 1] = (I_ou[it]
                    + dt / tau_ou * (0. - I_ou[it])
                    + np.sqrt(2 * dt / tau_ou) * sig * noise[it + 1])

  return I_ou

pars = default_pars_single(T=100)
pars['tau_ou'] = 1.  # [ms]
sig_ou = 0.1
I_ou = my_OU(pars, sig=sig_ou, myseed=2020)
plt.figure(figsize=(10, 4))
plt.plot(pars['range_t'], I_ou, 'r')
plt.xlabel('t (ms)')
plt.ylabel(r'$I_{\mathrm{OU}}$')
plt.show()
```

## Example: Up-Down transition

In the presence of two or more fixed points, noisy inputs can drive a transition between the fixed points! Here, we stimulate an E population for 1,000 ms applying OU inputs.

```python
# @title Simulation of an E population with OU inputs

# @markdown Make sure you execute this cell to spot the Up-Down states!

pars = default_pars_single(T=1000)
pars['w'] = 5.0
sig_ou = 0.7
pars['tau_ou'] = 1.  # [ms]
pars['I_ext'] = 0.56 + my_OU(pars, sig=sig_ou, myseed=2020)

r = simulate_single(pars)

plt.figure(figsize=(10, 4))
plt.plot(pars['range_t'], r, 'b', alpha=0.8)
plt.xlabel('t (ms)')
plt.ylabel(r'$r(t)$')
plt.show()
```


# The Python ecosystem - The scikit-learn library

Scikit-learn is probably the most used Machine Learning Library in Python. It is built on NumPy, SciPy, and matplotlib. The library offers a simple and efficient tools for data mining and data analysis. Scikit-learn offers a consistent API across different model and applications and hence is one of the best tools in Python for shallow learning algorithms.

## Modeling pipeline in scikit learn

* Loading The Data
* Training And Test Data
* Preprocessing The Data
* Create the Model
* Model Fitting
* Prediction
* Evaluate the Model's Performance
* Tune the Model

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

**Add the `src` directory as one where we can import modules**

```python
import os
import sys

# add the 'src' directory as one where we can import modules
src_dir = os.path.abspath(os.path.join(os.getcwd(), os.pardir, 'src'))
sys.path.append(src_dir)
print(src_dir)
```

```python
import helper_funcs as hf
```

## Loading The Data

Your data needs to be numeric and stored as NumPy arrays or SciPy sparse matrices. Other types that are convertible to numeric arrays, such as Pandas `DataFrame`, are also acceptable.

**Data for a Machine Learning model may look like this...**

```python
import numpy as np
X = np.random.random((10,5))
X[X < 0.3] = 0
print("Features array: {}\n".format(X.shape), X)
y = np.array(['M','M','F','F','F','M','M','F','F','F'])
print("Labels: {}\n".format(y.shape),y)
```

Further scikit-learn provides a rich [dataset loading utilities](http://scikit-learn.org/stable/datasets/index.html). It comes with easy to load toy datasets, sample images and sample generators.

**Toy datasets:**

Boston house-prices dataset (regression)

    load_boston()

Iris dataset (classification)

    load_iris()

Diabetes dataset (regression)

    load_diabetes()

Digits dataset (classification)

    load_digits()

Linnerud dataset (multivariate regression)

    load_linnerud()

Wine dataset (classification)

    load_wine()

Breast cancer wisconsin dataset (classification)

    load_breast_cancer()

```python
from sklearn.datasets import load_boston
ds = load_boston()
```

```python
print(ds.DESCR)
```

```python
print("Features array: {}\n".format(ds.data.shape), ds.data)
print("Labels: {}\n".format(ds.target.shape),y)
```

**Sample generators:**

```python
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=25, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           random_state=33)
```

```python
print("Features array: {}\n".format(X.shape), X)
print("Labels: {}\n".format(y.shape),y)
```

```python
fig, ax = plt.subplots()
ax.scatter(X[:, 0], X[:, 1], marker='o', c=y, s=25, edgecolor='k');
```

## Training And Test Data

```python
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           weights=[0.096],
                           random_state=33)
fig, ax = plt.subplots()
ax.scatter(X[:, 0], X[:, 1], marker='o', c=y, s=25, edgecolor='k');
```

```python
np.bincount(y)
```

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,
                                                    shuffle=True, random_state=42)
```

```python
print("Train set size:" , X_train.shape, "\nTest set size:", X_test.shape)
```

```python
print("Train label size:" , y_train.shape, "\nTest label size:", y_test.shape)
```

```python
pd.Series(y_train).value_counts()
```

```python
pd.Series(y_test).value_counts()
```

This `stratify` parameter makes a split so that the proportion of values in the sample produced will be the same as the proportion of values provided to parameter `stratify`.

```python
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,
                                                    test_size=0.4, random_state=42)

print(pd.Series(y_train).value_counts())
print(pd.Series(y_test).value_counts())
```

## Preprocessing The Data

In general, __learning algorithms benefit from standardization of the data set__. The `sklearn.preprocessing` package provides several utility functions and transformer classes to change raw feature vectors.

```python
from sklearn import preprocessing
```

```python
X_train = np.array([[ 1., -1.,  2.],
                    [ 2.,  0.,  0.],
                    [ 0.,  1., -1.]])
```

```python
scaler = preprocessing.StandardScaler()
scaler
```

```python
scaler.fit(X_train)
```

```python
X_train_scaled = scaler.transform(X_train)
X_train_scaled
```

```python
print("Mean: ",np.mean(X_train_scaled, axis=0))
print("\nStandard deviation: ", np.std(X_train_scaled, axis=0))
```

An alternative standardization is __aling features to lie between a given minimum and maximum value, often between zero and one, or so that the maximum absolute value of each feature is scaled to unit size__

```python
X_train
```

```python
min_max_scaler = preprocessing.MinMaxScaler()
```

```python
min_max_scaler.fit(X_train)
```

```python
X_train_scaled = min_max_scaler.transform(X_train)
X_train_scaled
```

```python
print("Min: ", np.min(X_train_scaled, axis=0))
print("\nMax: ", np.max(X_train_scaled, axis=0))
```

###  Encoding categorical features

Often features are not given as continuous values but categorical. For example a person could have features `"male"`, `"female"` or `"high"`, `"low"`, `"medium"`. Such features can be efficiently coded as integers, such as `0`, `1` (_nominal_)  or `2`, `0`, `1` (_ordinal_).

```python
X_train = np.array([['Male', 1, 0.76], ['Female', 3, 0.22], ['Female', 2, 0.57]])
X_train
```

```python
X_train[:,0]
```

#### Label encoding

```python
le = preprocessing.LabelEncoder()
le.fit(X_train[:,0])
```

```python
list(le.classes_)
```

```python
le.transform(X_train[:,0])
```

```python
X_encoded = np.copy(X_train)
X_encoded[:,0] = le.transform(X_train[:,0])
X_encoded = X_encoded.astype(float)
X_encoded
```

```python
le.inverse_transform(X_encoded[:,0].astype(int))
```

#### One-Hot encoding

```python
X_encoded[:,0].reshape(-1,1)
```

```python
ohe = preprocessing.OneHotEncoder()
ohe.fit(X_encoded[:,0].reshape(-1,1))
```

```python
X_train[:,0]
```

```python
ohe.transform(X_encoded[:,0].reshape(-1,1)).toarray()
```

_Note that starting with scikit-learn version 20.0 there will be a `CategoricalEncoder` available to convert categorical features to by __one-hot encoding__ or __ordinal encoding__ in one step_.

```python
import sklearn
sklearn.__version__
```

### Imputation of missing values

Many real world datasets contain missing values, often encoded as blanks, `NaN`s or other placeholders. A basic strategy to use incomplete datasets is to discard entire rows and/or columns containing missing values. Another strategy is to impute the missing values, i.e., to infer them from the known part of the data.

IN scikit-learn the `Imputer` class provides basic strategies for imputing missing values, either using __the mean__, __the median__ or __the most frequent value__ of the row or column in which the missing values are located.

```python
X_train = np.array([[1, 2], [np.nan, 3], [7, 6]])
X_train
```

```python
imp = preprocessing.Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(X_train)
```

```python
np.nanmean(X_train, axis=0)
```

```python
imp.transform(X_train)
```

## Create the Model

Scikit-learn ships with many different supervised and unsupervised models as well as some basic neural network models; among those are:

* ### Linear models
  * Ordinary Least Squares
  * Polynomial regression
  * Ridge Regression
  * Lasso
  * Elastic Net
  * Logistic regression

* ### Support Vector Machines

* ### Nearest Neighbors

* ### Gaussian Processes

* ### Decision Trees

* ### Ensemble methods
  * Random Forest
  * Gradient Boosting

Visit the [scikit-learn documentation website](http://scikit-learn.org/stable/index.html) for a comprehensive list of avaiake models and techniques.

### Loading a toy data set

For  the sake of this tutorial we load the _wine dataset_ provided by scikit learn.

```python
from sklearn.datasets import load_wine
```

```python
ds = load_wine()
print(ds.DESCR)
```

```python
ds.data
```

```python
ds.target
```

### Subsetting the data set to two classes

```python
y = ds.target[ds.target < 2]
y.shape
```

```python
X = ds.data[ds.target < 2]
X.shape
```

### Train-test split

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=42)
```

```python
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)
```

### Creation of a model instance

For the purpose of this tutorial we introduce [**Logistic regression**](https://en.wikipedia.org/wiki/Logistic_regression), also known as **logit regression**, or **logit model**, a probabilistic linear model for dichotomous data. The response variable is a binary variable (nominal variable), which means the variable has two categories or two values; Class 1 vs. Class 2, True vs. False, or $1$ vs. $0$, or success vs. failure, with the probabilities of $\pi$ and $1-\pi$, respectively.

The output of a logistic regression is a probability $(\pi)$, thus a value between $0$ and $1$. Moreover, this output is a linear function of known covariates $x_i$, which is just another word for the features in our data set.
$$\pi =\beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k$$

However, the right term of the equation can take any real value, whereas the left term of the equation is a probability, on the scale $0$ to $1$. In order to transform the scale of the data (right term) into a probability between $0$ and $1$ we apply a so-called **link function**.

For the logistic regression model this link function is the [**logit function**](https://en.wikipedia.org/wiki/Logit). The logit function maps probabilities from the range $(0, 1)$ to the entire real number range $(-\infty, \infty)$. It is written as

$$\eta = logit(\pi)\text{,}$$

where $\pi$ is the probability.

To understand the logit we first introduce the [**odds ratio**](https://en.wikipedia.org/wiki/Odds_ratio) or in short **odds**. The odds (o) can be written as

$$o = \frac{\pi}{1-\pi}\text{,}$$

where $\pi$ is the probability that an event occurs. If the probability of an event is a $0.5$, the odds are one-to-one or even $\left(\frac{0.5}{1-0.5}=1\right)$. We further define the or **log-odds**, which is the logarithm of the odds:
$$\eta = logit(\pi)= log \left( \frac{\pi}{1-\pi}\right)$$

This logarithmic function has the effect of removing the floor restriction, thus the function, the [**logit function**](https://en.wikipedia.org/wiki/Logit), our link function, transforms values in the range $0$ to $1$ to values over the entire real number range $(-\infty, \infty)$. If the probability is $1/2$ the odds are even and the logit is zero. Negative logits represent probabilities below one half and positive logits correspond to probabilities above one half.

The inverse form of the logit function is also called the [logistic function](https://en.wikipedia.org/wiki/Logistic_function), sometimes simply abbreviated as [**sigmoid function**](https://en.wikipedia.org/wiki/Sigmoid_function) due to its characteristic S-shape. Is allows us to go back from logits to probabilities.

$$\pi =logit^{-1}(\eta)= \frac{e^{\eta}}{1+e^{\eta}}=\frac{1}{1+e^{-\eta}}=\frac{1}{1+e^{-\beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k}}$$

The logistic function for the interval $[-6,6]$ is shown below. For values of $\eta$ in the range from $-\infty$ to $\infty$ $\pi$ is in the range of $0$ to $1$.

```python
def logit(interval=[-6,6]):
    x = np.linspace(interval[0], interval[1])
    y = 1 / (1 + np.exp(-x))
    return (x,y)

fig, ax = plt.subplots(figsize=(8,4))
ax.plot(logit()[0], logit()[1])
ax.axvline(0, color="k", linestyle="dashed", linewidth=0.5)
ax.axhline(0.5, color="k", linestyle="dashed", linewidth=0.5)
ax.set_title("The logistic function", size=18);
```

The logit function maps probabilities to values over the entire real number range. Thus, the probability of an event/outcome/success to be true $(y=1)$, given the set of predictors $x_i$, which is our data, is written as

$$logit(P(y=1|x_i))= \beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k\text{,}$$
For a matter of simplification we express the inverse of the function above as

$$\phi(\eta) = \frac{1}{1+e^{-\eta}}\text{,}$$

where $\eta$ is the linear combination of coefficients $(\beta_i)$ and predictor variables $(x_i)$, calculated as $\eta = \beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k$.

The parameters $(\beta_i)$ of the logit model are estimated by the [**method of maximum likelihood**](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation). However, there is no closed-form solution, so the maximum likelihood estimates are obtained by using iterative algorithms such as [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent), among others.

The output of the sigmoid function is interpreted as the probability of a particular observation belonging to class 1. It is written as  $\phi(\eta)=P(y=1|x_i,\beta_i)$, the probability of success $(y=1)$ given the predictor variables $x_i$ parameterized by the coefficients $\beta_i$. For example, if we compute $\phi(\eta)=0.65$ for a particular observation, this means that the chance that this observation belongs to class 1 is 65%. Similarly, the probability that this observation belongs to class 2 is calculated as $\phi(\eta)=P(y=0|x_i,\beta_i)= 1 - P(y=1|x_i,\beta_i)=1-0.65=0.35$ or 35%. For class assignment the predicted probability is then converted into a binary outcome via a unit step function:

$$
\hat y =
\begin{cases}
1,  & \text{if $\phi(\eta) \ge$ 0.5} \\
0, & \text{otherwise}
\end{cases}
$$

```python
from sklearn import linear_model
logistic = linear_model.LogisticRegression()
logistic
```

## Model Fitting

```python
logistic.fit(X_train, y_train)
```

## Prediction

### In-sample prediction

```python
logistic.predict(X_train)
```

### Out of sample prediction

```python
y_pred = logistic.predict(X_test)
y_pred
```

```python
logistic.predict_proba(X_test)[:5]
```

##  Evaluate the Model's Performance

In scikit-learn there are 3 different APIs for evaluating the quality of a model’s predictions:

* __Estimator score method__: Estimators have a score method providing a default evaluation criterion for the problem they are designed to solve.
* __Scoring parameter__: Model-evaluation tools using cross-validation rely on an internal scoring strategy.
* __Metric functions__: The metrics module implements functions assessing prediction error for specific purposes. Functions ending with `_score` return a value to maximize, the higher the better, and Functions ending with `_error` or `_loss` return a value to minimize, the lower the better.

### Estimator score method

Returns the mean accuracy on the given test data and labels.

$$\text{accuracy}(y, \hat y) = \frac{1}{n_{\text{samples}}} \sum_{i=0}^{n_{\text{samples}}-1}\mathbb1(\hat y_i = y_i)$$

```python
logistic.score(X_test, y_test)
```

```python
np.sum(y_pred == y_test)/len(y_test)
```

### Metric functions

```python
from sklearn.metrics import accuracy_score
accuracy_score(y_pred, y_test)
```

Note that there are many more [model metrics](http://scikit-learn.org/stable/modules/model_evaluation.html) available in scikit-learn.

```python
dir(sklearn.metrics)[11:]
```

## Tune the Model

Be aware that there are two types of parameters:
* __model parameters__ and
* __hyperparameters__.

Models parameters (i.e. regression coefficients) are learned from the data. Hyperparameters however, are parameters whose values are set before the learning process begins. Different model training algorithms require different hyperparameters, some simple algorithms  require none.

The optimal hyperparameter configuration for a particular modeling task is unknown. Hence, we apply different techniques, such as grid search, random search or Baysian optimization to approximate the best hyperparamter configuration (referred to as [hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization).

During the process of model optimization we want to avoid [overfitting](https://en.wikipedia.org/wiki/Overfitting). Hence, in machine learning a technique referred to as [k-fold cross-validation][1] is applied. Cross-validation is a process for reliably estimating the performance of a method for building a model by training and evaluating your model multiple times using the same method.

#### K-fold cross-validation (Source: [Wikipedia][1])

[1]: https://en.wikipedia.org/wiki/Cross-validation_(statistics)

### Model creation

For the purpose of this tutor ail we apply [support vector machines]() (SVMs) for classification of the subseted _wine data set_ (see above). An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible.

The advantages of support vector machines are:

* Effective in high dimensional spaces.
* Still effective in cases where number of dimensions is greater than the number of samples.
* Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
* Versatile: different Kernel functions can be specified for the decision function.

The disadvantages of support vector machines include:

* If the number of features is much greater than the number of samples, avoid over-fitting in choosing Kernel functions and regularization term is crucial.
* SVMs do not directly provide probability estimates.
* SVM do not scale well.

The mathematics of the algorithm are beyond the scope of this tutorial, however if you are interested we suggest to dive into the [scikit-learn documentation](http://scikit-learn.org/stable/modules/svm.html) or watch the informative [video](https://www.youtube.com/watch?v=-Z4aojJ-pdg) by Brandon Rohrer.

```python
from sklearn import svm
svc = svm.SVC()
svc
```

### Hyperparameter optimization using `GridSearchCV`

* `C`: Penalty parameter `C` of the error term. The parameter allows one to trade off training error vs. model complexity. Hence, the `C` parameter trades off misclassification of training examples against simplicity of the decision surface. A low `C` makes the decision surface smooth, while a high `C` aims at classifying all training examples correctly by giving the model freedom to select more samples as support vectors.

* `kernel`:  Specifies the kernel type to be used in the algorithm. It must be one of `linear`, `poly`, `rbf`, `sigmoid`, `precomputed` or a callable.

* `gamma`: Kernel coefficient for `rbf`, `poly` and `sigmoid`. Intuitively, the `gamma` parameter defines how far the influence of a single training example reaches, with low values meaning _far_ and high values meaning _close_. The `gamma` parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.

```python
from sklearn.model_selection import GridSearchCV

param_grid = [{'C': [0.1, 1, 10, 100],
               'gamma': [0.01, 0.001, 0.0001],
               'kernel': ['linear', 'rbf']}]

clf = GridSearchCV(svc, param_grid, cv=5, verbose=1, return_train_score=True)
```

### Model fitting

```python
clf.fit(X_train, y_train)
```

### Cross validation results

```python
pd.DataFrame(clf.cv_results_ )
```

`best_params_`: Parameter setting that gave the best results on the hold out data.

```python
clf.best_params_
```

`best_estimator_`: Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data.

```python
clf.best_estimator_
```

`best_score_`: Mean cross-validated score of the best_estimator

```python
clf.best_score_
```

### Model preditction

```python
y_pred = clf.best_estimator_.predict(X_test)
y_pred
```

### Model evaluation

**Accuracy**

```python
from sklearn.metrics import accuracy_score
accuracy_score(y_pred, y_test)
```

**Confusion matrix**

```python
from sklearn.metrics import confusion_matrix
cnf_matrix = confusion_matrix(y_test, y_pred)
cnf_matrix
```

```python
# Plot non-normalized confusion matrix
hf.plot_confusion_matrix(cnf_matrix, classes=["class 0", "class 1"],
                      title='Confusion matrix, without normalization')
```

**Seclected classification metrics**

\begin{equation}
\text{precision} = \frac{TP}{TP+FP}
\end{equation}

\begin{equation}
\text{recall} = \frac{TP}{TP+FN}
\end{equation}

\begin{equation}
\text{F1-score} = 2 \times \frac{\text{precision} \times \text{recall}}{\text{precision}+\text{recall}}
\end{equation}

```python
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
```

notebooks/02h - The Python Ecosystem - The scikit-learn library.ipynb


```python
!pip3 install qiskit
```

    Requirement already satisfied: qiskit in /home/r0x6f736f646f/.local/lib/python3.6/site-packages
    Collecting qiskit-aer==0.3.2 (from qiskit)
      Using cached https://files.pythonhosted.org/packages/18/2f/63ee86a48d131f0d6772ed9aba947d2e8457d17408add32c1d0d44822c84/qiskit_aer-0.3.2-cp36-cp36m-manylinux1_x86_64.whl
    Collecting qiskit-ibmq-provider==0.3.3 (from qiskit)
    Requirement already satisfied: qiskit-ignis==0.2.0 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit)
    Collecting qiskit-terra==0.10.0 (from qiskit)
      Using cached https://files.pythonhosted.org/packages/09/97/1be77cc1b22f3bd341024b1bae89d71ca9ee147ac047f793364cadc873a8/qiskit_terra-0.10.0-cp36-cp36m-manylinux1_x86_64.whl
    Requirement already satisfied: qiskit-aqua==0.6.1 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit)
    Requirement already satisfied: numpy>=1.13 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aer==0.3.2->qiskit)
    Collecting nest-asyncio==1.0.0 (from qiskit-ibmq-provider==0.3.3->qiskit)
      Using cached https://files.pythonhosted.org/packages/62/18/4c6a895c925e8822e72535d774861e9ca07089c4b938ed20002b1aaab66a/nest_asyncio-1.0.0-py3-none-any.whl
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: requests-ntlm>=1.1.0 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-ibmq-provider==0.3.3->qiskit)
    Collecting websockets<8,>=7 (from qiskit-ibmq-provider==0.3.3->qiskit)
      Using cached https://files.pythonhosted.org/packages/43/71/8bfa882b9c502c36e5c9ef6732969533670d2b039cbf95a82ced8f762b80/websockets-7.0-cp36-cp36m-manylinux1_x86_64.whl
    Requirement already satisfied: scipy!=0.19.1,>=0.19 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-ignis==0.2.0->qiskit)
    Requirement already satisfied: setuptools>=40.1.0 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-ignis==0.2.0->qiskit)
    Requirement already satisfied: sympy>=1.3 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: ply>=3.10 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: psutil>=5 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: networkx>=2.2 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: marshmallow-polyfield<6,>=5.7 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: jsonschema>=2.6 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: marshmallow<4,>=3 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: docplex in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: pyscf; sys_platform != "win32" in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: cvxopt in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: quandl in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: fastdtw in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: dlx in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: scikit-learn>=0.20.0 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: h5py in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: certifi>=2017.4.17 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: chardet<4,>=3.0.2 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from requests>=2.19->qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: ntlm-auth>=1.0.2 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: cryptography>=1.3 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: mpmath>=0.19 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from sympy>=1.3->qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: decorator>=4.3.0 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from networkx>=2.2->qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from marshmallow-polyfield<6,>=5.7->qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: pyrsistent>=0.14.0 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from jsonschema>=2.6->qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: attrs>=17.4.0 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from jsonschema>=2.6->qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from jsonschema>=2.6->qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: docloud>=1.0.375 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from docplex->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: pyOpenSSL in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from quandl->qiskit-aqua==0.6.1->qiskit)
    Collecting more-itertools<=5.0.0 (from quandl->qiskit-aqua==0.6.1->qiskit)
      Cache entry deserialization failed, entry ignored
      Using cached https://files.pythonhosted.org/packages/a4/a6/42f17d065bda1fac255db13afc94c93dbfb64393eae37c749b4cb0752fc7/more_itertools-5.0.0-py3-none-any.whl
    Requirement already satisfied: inflection>=0.3.1 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from quandl->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: pandas>=0.14 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from quandl->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: pyasn1 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from quandl->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: python-dateutil in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from quandl->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: ndg-httpsclient in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from quandl->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: joblib>=0.11 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: cffi!=1.11.3,>=1.8 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.3->qiskit)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->jsonschema>=2.6->qiskit-terra==0.10.0->qiskit)
    Requirement already satisfied: pytz>=2017.2 in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from pandas>=0.14->quandl->qiskit-aqua==0.6.1->qiskit)
    Requirement already satisfied: pycparser in /home/r0x6f736f646f/.local/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.3.3->qiskit)
    Installing collected packages: qiskit-aer, nest-asyncio, qiskit-terra, websockets, qiskit-ibmq-provider, more-itertools
      Found existing installation: qiskit-aer 0.3.4
        Uninstalling qiskit-aer-0.3.4:
          Successfully uninstalled qiskit-aer-0.3.4
      Rolling back uninstall of qiskit-aer
    Exception:
    Traceback (most recent call last):
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
        prefix=options.prefix_path,
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
        **kwargs
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
        self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
        isolated=self.isolated,
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/wheel.py", line 345, in move_wheel_files
        clobber(source, lib_dir, True)
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/wheel.py", line 316, in clobber
        ensure_dir(destdir)
      File "/home/r0x6f736f646f/.local/lib/python3.6/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
        os.makedirs(path)
      File "/usr/lib/python3.6/os.py", line 210, in makedirs
        makedirs(head, mode, exist_ok)
      File "/usr/lib/python3.6/os.py", line 210, in makedirs
        makedirs(head, mode, exist_ok)
      File "/usr/lib/python3.6/os.py", line 220, in makedirs
        mkdir(name, mode)
    PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/qiskit'
    You are using pip version 9.0.3, however version 20.0.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.

## Getting started with qiskit
Here we will get the overview of qiskit and build a fundamental unit of qiskit, quantum circuit.
We need to build the execute

```python
import numpy as np
from qiskit import *
%matplotlib inline
```

```python
# We create q quantum circuit on a quantum register with 4 qubits
circuit = QuantumCircuit(4)
```

We can now add gates to manipulate the register.
eg Hadamard, Controlled not

```python
# we have added a Hadamard gate to the first qubit (Superposition)
circuit.h(0)
# we add a CNOT gate on 0 and 1 (Bell state)
circuit.cx(0,1)
# CNOT on 0 and 2
circuit.cx(0,2)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f61cb4d6278>

```python
# Lets see what we have made
circuit.draw()
```

Note: If you don't have matplotlib set up as your default in '~/.qiskit/settings.conf' it will use a text-based drawer over matplotlib. To set the default to matplotlib, use the following in the settings.conf
[default]
circuit_drawer = mpl

## Simulating circuit using qiskit Aer which provides many backends
We will use `statevector_simulator` which is the most common which returns quantum states

```python
from qiskit import Aer

# Run the circuit
backend = Aer.get_backend('statevector_simulator')

#Create execution
job = execute(circuit, backend)
```

```python
#Monitor our job with
job.status()
```

    <JobStatus.DONE: 'job has successfully run'>

```python
# get results
result = job.result()
```

```python
outputs = result.get_statevector(circuit)
print(outputs)
```

    [0.70710678+0.j 0.        +0.j 0.        +0.j 0.        +0.j
     0.        +0.j 0.        +0.j 0.        +0.j 0.70710678+0.j
     0.        +0.j 0.        +0.j 0.        +0.j 0.        +0.j
     0.        +0.j 0.        +0.j 0.        +0.j 0.        +0.j]

## OpenQASM backend
It provides information about the state output by the ideal circuit and the matrix representation of the circuit

```python
circuit1 = QuantumCircuit(3,3)
circuit1.h(0)
circuit1.cx(0,1)
circuit1.cx(1,2)
circuit1.barrier(range(3))
# map quantum measurements to the classical bits
circuit1.measure(range(3), range(3))
circuit1.draw()
```

```python
backend2 = Aer.get_backend('qasm_simulator')
# we repeate the circuit 2048 times
job2 = execute(circuit1, backend2, shots= 2048)
result2 = job2.result()
```

```python
from qiskit.visualization import plot_histogram
plot_histogram(result2.get_counts(circuit1))
```

## Running on IBMQ

```python
from qiskit import IBMQ
```

After generating your API token, call: IBMQ.save_account('MY_TOKEN').

This will store your IBM Q credentials in a local file.

```python
IBMQ.load_account()
```

    <AccountProvider for IBMQ(hub='ibm-q', group='open', project='main')>

```python
IBMQ.providers()
```

    [<AccountProvider for IBMQ(hub='ibm-q', group='open', project='main')>]

```python
provider = IBMQ.get_provider(group='open')
provider.backends()
```

    [<IBMQSimulator('ibmq_qasm_simulator') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmqx2') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_16_melbourne') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_vigo') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_ourense') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_london') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_burlington') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_essex') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_armonk') from IBMQ(hub='ibm-q', group='open', project='main')>]

```python
backend3 = provider.get_backend('ibmq_16_melbourne')
```

```python
from qiskit.tools.monitor import job_monitor

job3 = execute(circuit1, backend=backend3)
job_monitor(job3)
```

    Job Status: job has successfully run

```python
result3 = job3.result()
```

```python
counts3= result3.get_counts(circuit1)
plot_histogram([counts3,result3.get_counts(circuit1)], legend=['Device', 'Simulator'])
```

```python
backend4 = provider.get_backend('ibmq_qasm_simulator')
```

```python
job4 = execute(circuit1, backend=backend4)
```

```python
result4 = job4.result()
```

```python
counts4 = result4.get_counts(circuit1)
plot_histogram(counts4)
```

```python

```


```python
import numpy as np

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score

from kyle.calibration import ModelCalibrator
from kyle.models import CalibratableModel
from kyle.metrics import ECE
from kyle.calibration.calibration_methods import TemperatureScaling
from kyle.sampling.fake_clf import DirichletFC
from kyle.transformations import MaxComponentSimplexAut
from kyle.evaluation import EvalStats
```

# What is calibration?

When we talk about how good a machine learning model is, what we (generally) mean to ask is: How accurate is the model?
While this is a good enough metric in many cases, we are, in fact, leaving out important information about the model.
One such piece of information is concerned with whether the confidence of the model is in line with its accuracy.
If it is, we say the model is calibrated.

To explain this concept in detail, let's begin with an example. Suppose we want to predict whether a patient has cancer.
We can simulate data with two classes i.e. $y \in \{0, 1\}$ where $y=0$ denotes a healthy patient and $y=1$ denotes a
patient who has cancer.

```python
n_samples = 2000
n_classes = 2
```

```python
X, y = datasets.make_classification(
        n_samples=n_samples,
        n_features=20,
        n_informative=7,
        n_redundant=10,
        n_classes=n_classes,
        random_state=42,
    )
X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42
    )
```

We can then train a neural network on our data:

```python
model = MLPClassifier(hidden_layer_sizes=(50, 50, 50))
model.fit(X_train, y_train)
```

and make predictions on new samples. Let's see how our model performs on unseen examples:

```python
y_pred = model.predict(X_test)
model_accuracy = accuracy_score(y_test, y_pred)

f"Model accuracy: {model_accuracy*100}%"
```

That seems pretty good! One might think our job here is done: After all, the model predicts whether a person has cancer
or not with decent accuracy.
Unfortunately, accuracy of a model does not tell us the full story. This is so due to the fact that at inference time,
for a given sample a model outputs confidence scores for each class. We then take the class with the highest confidence
and interpret that as the prediction of the model.

This conversion of continuous (probability) to discrete (label) values can hide certain properties of the model.
To illustrate this, let's take two models -- $A$ and $B$ -- trained on the same data. Let's further assume they have
similar accuracy. Suppose we test both models with 10 healthy samples. $A$ assigns probabilities $(0.49, 0.51)$ to all
samples, whereas $B$ assigns $(0.1, 0.9)$. While $A$ & $B$ will be wrong 100% of the time, notice $A$ being much closer
to classifying the samples as belonging to the correct class compared to $B$.

Continuing with our previous example: Imagine that on all examples where the model was $95$% confident that the subject
has cancer, it was correct $70$% of the time. Intuitively, it seems there's something not quite right with the model:
the model is over-confident in its predictions. This notion is formalized by the concept of calibration.
We say a model is (strongly) calibrated when, for any confidence value $p \in [0, 1]$,
prediction of a class with confidence $p$ is correct with probability $p$:

\begin{equation}
P(\widehat{y}=y|\widehat{p}=p) = p \quad \forall p \in [0, 1]
\end{equation}

So, is our model calibrated? As we can see in the equation above, $\widehat{p}$ is continuous, which means we cannot
compute the equation with finite data. We can, however, develop empirical measures that approximate the true measure
of (mis)calibration.

One simple way to get an empirical estimate of the model's accuracy and confidence is to discretize the probability
space. This is done by slicing $p$ into $K$ equal-sized bins. We can then calculate the accuracy and confidence for each
bin:

\begin{equation}
accuracy_{B_k} = \frac{1}{|B_k|} \sum_{m=1}^{|B_k|}1(\widehat{p}_m=p_m)
\end{equation}

\begin{equation}
confidence_{B_k} = \frac{1}{|B_k|} \sum_{m=1}^{|B_k|}\widehat{p}_m
\end{equation}

We can now simply calculate the weighted average difference between the accuracy and confidence of the model over all bins:

\begin{equation}
\sum_{k=1}^{K} \frac{|B_k|}{n} \Big|\:accuracy_{B_k} - confidence_{B_k} \Big|
\end{equation}

This is known as the **Expected Calibration Error** $(ECE).$ As can be seen, $ECE=0$ if a model is perfectly calibrated.
Let's calculate the $ECE$ for our model with $10$ bins:

```python
ece = ECE(bins=10)
```

```python
# Evaluate uncalibrated predictions
uncalibrated_confidences = model.predict_proba(X_test)

pre_calibration_ece = ece.compute(uncalibrated_confidences, y_test)

f"ECE before calibration: {pre_calibration_ece}"
```

We can also visualize the extent of miscalibration by plotting the model's confidence *(x-axis)* vs. the ground truth
probability *(y-axis)*. For a perfectly calibrated model, the plot should be $y=x$. Let's see how our model fares:

```python
eval_stats = EvalStats(y_test, uncalibrated_confidences)
class_labels = [i for i in range(n_classes)]

eval_stats.plot_reliability_curves(class_labels)
```

Okay, so our model is not calibrated as $ECE>0$. Can we do anything to remedy the situation?

# Model calibration

Indeed, we can improve the calibration of our model using various techniques. What's more, we don't need to train our
model again; many calibration techniques are post-processing methods i.e. operating on the trained model's output
confidence scores. The output scores for calibration are typically obtained on a validation set.

In `kyle`, we have provided a `CalibratableModel` class which takes a model and, as the name suggests, makes it possible
to calibrate that model. By default, we use a technique called [*Temperature scaling*](https://arxiv.org/abs/1706.04599)
for calibration.

```python
# Create calibratable model
calibration_method = TemperatureScaling()
calibratable_model = CalibratableModel(model, calibration_method)
```

We also provide a `ModelCalibrator` class which holds the data to calibrate models:

```python
# Create model calibrator and calibrate model
calibrator = ModelCalibrator(X_calibrate=X_test,
                             y_calibrate=y_test,
                             X_fit=X_train,
                             y_fit=y_train)
```

We now have everything ready to calibrate our model:

```python
calibrator.calibrate(calibratable_model)
```

Let's see if calibrating the model improved the $ECE$ score

```python
# Passing X_test instead of X_calibrate in predict_proba() to make comparison with pre-calib model clear,
# same reasong for y_test in ece.compute()
calibrated_confidences = calibratable_model.predict_proba(X_test)

post_calibration_ece = ece.compute(calibrated_confidences, y_test)

f"ECE before calibration: {pre_calibration_ece}, ECE after calibration: {post_calibration_ece}"
```

Great! $ECE$ has improved. Let's also plot a reliability curve to visually confirm the improvement in calibration.

```python
eval_stats = EvalStats(y_test, calibrated_confidences)

eval_stats.plot_reliability_curves(class_labels)
```

Wonderful! We have successfully improved our model's calibration.

# Model-agnostic calibration

You may have noticed that to evaluate (mis)calibration of a model, we don't require the model itself.
Rather, it is sufficient to have the confidence scores predicted by the model.
This means we can abstract away the model and generate both the ground truth and confidence scores via sampling processes.

In `kyle` we have provided samplers that simulate different kinds of calibration properties.
One such sampler is the `DirichletFC` class which provides calibrated ground truth and confidences by default.

```python
sampler = DirichletFC(num_classes=2)

# Get 1000 calibrated fake confidence scores
calibrated_samples = sampler.get_sample_arrays(1000)
ground_truth, confidences = calibrated_samples
```

Let's evaluate the $ECE$ for these samples:

```python
ece.compute(confidences, ground_truth)
```

Wait, the $ECE>0$, how can we say that the samples are calibrated?

As mentioned earlier, we only have finite samples so true miscalibration can only be measured asymptotically.
This means that the more samples we have, the more accurate would $ECE$'s estimate become.
We can test this by generating *5x* as many samples as before and evaluating $ECE$ again:

```python
calibrated_samples = sampler.get_sample_arrays(5000)
ground_truth, confidences = calibrated_samples

ece.compute(confidences, ground_truth)
```

As expected, $ECE$ goes down with more samples.

We can also systematically generate uncalibrated samples. For instance, the `ShiftingSimplexAutomorphism` shifts the
confidence scores by adding a fixed vector with positive entries to the input and normalizing the result.

```python
def overestimating_max(x: np.ndarray):
    x = x.copy()
    mask = x > 1/2
    x[mask] = x[mask] - (1/4 - (1-x[mask])**2)
    return x

automorphism = MaxComponentSimplexAut(overestimating_max)
shifted_sampler = DirichletFC(num_classes=2, simplex_automorphism=automorphism)

# Get 1000 uncalibrated fake confidence scores
uncalibrated_samples = shifted_sampler.get_sample_arrays(10000)
ground_truth, confidences = uncalibrated_samples
```

Let's see if the uncalibrated nature of the samples is validated by $ECE$:

```python
ece.compute(confidences, ground_truth)
```

Once again, to verify that miscalibration will indeed increase with more samples, let's sample *5x* as many samples as
before and measure $ECE$ again:

```python
uncalibrated_samples = shifted_sampler.get_sample_arrays(1000)
ground_truth, confidences = uncalibrated_samples

ece.compute(confidences, ground_truth)
```

Great! Calibration error goes up as we sample more instances.


# SoftwareDesign 2020年10月号第一特集 Pythonではじめる統計学

公式のサンプルコードの写経

## 第2章 平均から始める記述統計

```
# 基本ライブラリ
import pandas as pd
import numpy as np
import numpy.random as rd
import scipy.stats as st
from tqdm import tqdm
from IPython.display import display

# グラフ描画系
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import animation as ani
import seaborn as sns

# plot setting
plt.rcParams["patch.force_edgecolor"] = True
sns.set(style="whitegrid", palette="muted", color_codes=True)
sns.set_style("whitegrid", {'grid.linestyle':'--'})

```

```
!pip install japanize_matplotlib
```

    Collecting japanize_matplotlib
      Downloading https://files.pythonhosted.org/packages/2c/aa/3b24d54bd02e25d63c8f23bb316694e1aad7ffdc07ba296e7c9be2f6837d/japanize-matplotlib-1.1.2.tar.gz (4.1MB)
         |████████████████████████████████| 4.1MB 2.8MB/s
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from japanize_matplotlib) (3.2.2)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->japanize_matplotlib) (2.8.1)
    Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/dist-packages (from matplotlib->japanize_matplotlib) (1.18.5)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->japanize_matplotlib) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->japanize_matplotlib) (2.4.7)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->japanize_matplotlib) (1.2.0)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib->japanize_matplotlib) (1.15.0)
    Building wheels for collected packages: japanize-matplotlib
      Building wheel for japanize-matplotlib (setup.py) ... done
      Created wheel for japanize-matplotlib: filename=japanize_matplotlib-1.1.2-cp36-none-any.whl size=4120193 sha256=c456b72fecea1b4bfbca1c444ba8966b77859fc220a3a9b779acb99ac229e0cb
      Stored in directory: /root/.cache/pip/wheels/9c/f9/fc/bc052ce743a03f94ccc7fda73d1d389ce98216c6ffaaf65afc
    Successfully built japanize-matplotlib
    Installing collected packages: japanize-matplotlib
    Successfully installed japanize-matplotlib-1.1.2

```
import japanize_matplotlib
```

    /usr/local/lib/python3.6/dist-packages/japanize_matplotlib/japanize_matplotlib.py:15: MatplotlibDeprecationWarning:
    The createFontList function was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use FontManager.addfont instead.
      font_list = font_manager.createFontList(font_files)

```
np.random.seed(71)
n_data = 1000
height_men = np.random.normal(loc=171.4, scale=5.8, size=n_data)
height_women = np.random.normal(loc=157.5, scale=5.4, size=n_data)
```

* 図1 身長データのヒストグラム

```
# コード1: pythonでヒストグラム描画
plt.figure(figsize=(8,4))
plt.hist(height_men, bins=np.arange(150, 195, 2.5),)
plt.xlim(150, 195)
plt.ylabel("度数")
plt.xlabel("身長")
plt.show()
```

```
plt.figure(figsize=(8,4))
plt.hist(height_women, bins=np.arange(135, 185, 2.5),)
plt.xlim(135, 185)
plt.ylabel("度数")
plt.xlabel("身長")
plt.show()
```

* 図2 2つのデータセットのヒストグラム

```
# コード2: pythonでヒストグラム描画
plt.figure(figsize=(8,4))
plt.hist(height_men, bins=np.arange(140, 200, 2.5), alpha=0.7, label="男性")
plt.hist(height_women, bins=np.arange(135, 180, 2.5), alpha=0.7, label="女性")
plt.xlim(135, 195)
plt.ylabel("度数")
plt.xlabel("身長")
plt.show()
```

* 図３ 総和記号への書き換え

1000人分の合計
$$
\begin{align}
 &= x_1 + x_2 + x_3 + \dotsc + x_{999} + x_{1000} \\
 &= \sum_{i=1}^{1000} x_i
 \end{align}
$$

```
# コード3: 総和記号がやっていることをpythonコードで表す
height_sum = 0
for i in range(len(height_men)):
  height_sum += height_men[i]
print(height_sum)
```

    171409.2829305438

```
# exp関数
print(np.e)
```

    2.718281828459045

### exp関数

* 図５ exp関数の描画コードおよびグラフ

```
# コード4: exp(x)のグラフ描画
x = np.linspace(-2, 2, 301)
plt.plot(x, np.exp(x))
plt.ylabel("exp(x)")
plt.xlabel("x")
plt.show()
```

### 平均

$
\begin{align}
  \bar{x}
  &= \frac{x_1 + x_2 + \dotsc + x_n}{n} \\
  &= \frac{1}{n} \sum_{i=1}^{n} x_i
\end{align}
$

```
# コード5: 平均の計算のpython コード
sum_val = 0
n = len(x)
for i in range(n):
  sum_val += x[i]
mean_val = sum_val / n
print(mean_val)
```

    1.0327656043024712e-17

### 平均値の特性

```
def plot_mean(x, xlim):
  m = np.mean(x)
  plt.figure(figsize=(13,2))
  plt.scatter(x, np.zeros_like(x), s=50, zorder=100)
  plt.scatter(m, [0], s=100, c="r", zorder=100)

  for i in range((len(x))):
    plt.text(x[i], -0.3, f"x{i+1}", size=12)
    plt.text(x[i], -0.5, f"={x[i]}", size=12)

  plt.text(m, -0.3, "average", size=12)
  plt.text(m, -0.6, f"={m:.1f}", size=12)
  plt.plot([0,100], [0,0], c="K", zorder=10)

  plt.yticks([])
  plt.xlim(xlim)
  plt.ylim(-1,1)
  plt.xlabel("test score")
  plt.show()
```

* 図６ 平均値の特性

```
x = [96, 63, 85, 66, 91, 89, 77]
plot_mean(x, xlim=(60, 100))
```

### 中央値

* 図７ 平均値の位置（外れ値がある場合）

```
x = [98, 23, 14, 26, 20, 11, 17, ]
plot_mean(x, xlim=(10, 100))
```

* 図８ 中央値の位置

```
x = [98, 23, 14, 26, 20, 11, 17, ]
xlim = (10, 100)

m = np.median(x)
plt.figure(figsize=(13,2))
plt.scatter(x, np.zeros_like(x), s=50, zorder=100)
plt.scatter(m, [0], s=100, c="r", zorder=100)

for i in range(len(x)):
  plt.text(x[i], -0.3, f"x{i+1}", size=12)
  plt.text(x[i], -0.5, f"={x[i]}", size=12)

plt.text(m, -0.7, "median", size=12)
plt.text(m, -0.9, f"={m:.1f}", size=12)
plt.plot([0, 100], [0, 0], c="k", zorder=10)

plt.yticks([])
plt.xlim(xlim)
plt.ylim(-1, 1)
plt.xlabel("test score")
plt.show()
```

### 最頻値

```
# 収入を模した疑似データの生成
np.random.seed(71)
n_data = 6000
income = np.random.lognormal(
    mean=np.log(427), sigma=0.85, size=n_data)
```

* 図９ 収入データのヒストグラム

```
# 階級の幅の単位
class_interval_unit = 100

plt.figure(figsize=(20,7))
# bins = np.arange(0, 3000, class_interval_unit).tolist() + [4000, 5000]
bins = np.arange(0, 5000, class_interval_unit)
class_freq, class_interval, _ = plt.hist(income, bins=bins, density=False)

# for i in range(len(class_freq)):
#   if class_freq[i] != 0:
#     plt.text(class_interval[i+1]-100, class_freq[i], f"{class_freq[i]/n_data*100: .1f}%")

# 代表値の計算
mean_val = np.mean(income)
median_val = np.median(income)
mode_class_idx = np.argmax(class_freq)
mode_val = class_interval[mode_class_idx] + class_interval_unit/2

# 代表値の描画
max_freq_round = np.round(np.max(class_freq), -3)
plt.vlines(mean_val, 0, max_freq_round, color="r")
plt.text(mean_val-50, max_freq_round, "平均")
plt.vlines(median_val, 0, max_freq_round, color="r")
plt.text(median_val-100, max_freq_round, "中央値")
plt.vlines(mode_val, 0, max_freq_round, color="r")
plt.text(mode_val-100, max_freq_round, "最頻値")

plt.xlabel("収入")
plt.ylabel("度数")
print(max(income))

```

```
n_over_mean_val = np.sum([income > mean_val])
n_income = len(income)
n_over_mean_val, n_income, n_over_mean_val / n_income
```

    (1949, 6000, 0.3248333333333333)

### 分位点

* 図１０ 四分位を計算するpythonコード

```
height_men_25 = np.percentile(height_men, 25)
height_men_75 = np.percentile(height_men, 75)
print(f"男性の身長の第1四分位点： {height_men_25:.3f}, 第3四分位点： {height_men_75:.3f}")
```

    男性の身長の第1四分位点： 167.632, 第3四分位点： 175.085

♯平均偏差

* 図１１ ２つのグループの身長の分布

```
np.random.seed(71)
n_data = 1000
x1 = np.random.normal(loc=171.4, scale=3.8, size=n_data)
x2 = np.random.normal(loc=171.4, scale=7.8, size=n_data)

x1_mean = np.mean(x1)
x2_mean = np.mean(x2)

plt.figure(figsize=(8,4))
plt.hist(x1, bins=np.arange(140, 200, 2.5), alpha=0.7, label="グループ1")
plt.hist(x2, bins=np.arange(140, 200, 2.5), alpha=0.7, label="グループ2")

plt.vlines(x1_mean, 0, 290, "b")
plt.vlines(x2_mean, 0, 290, "orange")
plt.xlim(145, 200)
plt.ylabel("度数")
plt.xlabel("身長")
plt.legend(loc="best")
plt.show()

print(x1_mean, x2_mean)

```

### 平均偏差

```
# テストの点数データ
x = [96, 63, 85, 66, 91, 89, 77]
m = np.mean(x)
deviation_x = np.array(x) - m
deviation_x
```

    array([ 15., -18.,   4., -15.,  10.,   8.,  -4.])

* 図１２ 偏差を可視化して理解する

```
m = np.mean(x)
plt.figure(figsize=(11,4))
plt.scatter(x, np.zeros_like(x), s=50, zorder=100)
plt.scatter(m, [0], s=100, c="r", zorder=100)

for i in range(len(x)):
  plt.text(x[i]+0.2, -0.8, f"x{i+1}\n={x[i]}", size=12)

plt.text(m+0.1, -0.8, f"average\n{m}", size=12)
plt.plot([m, m], [-5,10], "k--", zorder=10)
for i, p, sd in zip(range(len(x)), [1.5, 4.5, 3, 4, 2, 2.5, 3.5], deviation_x):
  plt.plot([x[i], x[i]], [-5, 10], "k--", zorder=10)
  plt.plot([x[i], m], [p, p], "r", zorder=10)
  plt.text(x[i]+0.1, p-0.3, f"{sd}", size=12)

plt.yticks([])
plt.xlim(60, 100)
plt.ylim(-1, 5)
plt.xlabel("test score")
plt.show()

```

```
deviation_x_abs = np.abs(deviation_x)
deviation_x_abs, np.mean(deviation_x)
```

    (array([15., 18.,  4., 15., 10.,  8.,  4.]), 0.0)

* 図１３ 偏差の絶対値をとった値を可視化

```
m = np.mean(x)
plt.figure(figsize=(11,4))

plt.plot([m, m], [-5, 10], "k--", zorder=10)
for i, p, sd in zip(range(len(x)), [1.5, 4.5, 3, 4, 2, 2.5, 3.5], deviation_x_abs):
  plt.plot([x[i], x[i]], [-5, 10], "k--", zorder=10)
  plt.plot([m+sd, m], [p, p], "r", zorder=10)
  plt.text(m+sd+0.1, p-0.1, f"{sd:.0f}", size=12)

plt.yticks([])
plt.xlim(60, 100)
plt.ylim(1, 5)
plt.xlabel("test score")
plt.show()
```

* 図１４ 平均偏差を計算するPythonコードと、実際の計算式

平均偏差 $ = \frac{1}{n} \sum_{i=1}^{n} | x_i - \bar{x} | $

```
# コード5:平均偏差を計算するpythonコード
def average_deviation(x):
  mean_val = np.mean(x)
  n_data = len(x)
  sum_val = 0
  for i in range(len(x)):
    sum_val += np.abs(x[i] - mean_val)
  return sum_val / n_data
```

```
average_deviation(x1)
```

    2.9132362036922173

```
average_deviation(x2)
```

    6.310475113543013

# 分散

```
from matplotlib.patches import Rectangle
```

```
deviation_x
```

    array([ 15., -18.,   4., -15.,  10.,   8.,  -4.])

* 図１６ 偏差の２乗を可視化

```
y_max_for_plot = 90

m = np.mean(x)
plt.figure(figsize=(11,11))
ax = plt.subplot(111)
plt.scatter(x, np.zeros_like(x), s=50, zorder=100)
plt.scatter(m, [0], s=100, c="r", zorder=100)

for i in range(len(x)):
  plt.text(x[i]+0.2, -5, f"x{i+1}\n={x[i]}", size=12)

plt.text(m+0.1, -5, f"average\n{m}", size=12, zorder=300)
plt.plot([0, 130], [0, 0], c="k", zorder=10)

plt.plot(
    [m, m], [-10, y_max_for_plot], "gray", linewidth=1,
    linestyle="--", lw=0.5, zorder=-10
)
for i, p, sd in zip(range(len(x)), [5, 66, 41, 50, 21, 32, 45], deviation_x):
  abs_sd = np.abs(sd)
  plt.plot([x[i], x[i]], [-10, y_max_for_plot], "gray", linewidth=1, linestyle="--", zorder=-10)
  plt.plot([x[i], m], [p, p], "r", linewidth=2, zorder=200)
  plt.text(x[i]+0.1, p-0.3, f"{sd}", size=12, zorder=300)
  r = Rectangle(xy=(x[i], p), width=-sd, height=abs_sd, fill=True, color="red", alpha=0.3)
  ax.add_patch(r)

plt.yticks([])
plt.xlim(30,130)
plt.ylim(-10, y_max_for_plot)
plt.xlabel("test score")
plt.show()
```

```
np.mean(deviation_x ** 2)
```

    138.57142857142858

* 図１５ 分散を計算するPythonコードと、実際の計算式

分散 $ = \frac{1}{n} \sum_{i=1}^{n} ( x_i - \bar{x} )^2 $

```
# コード６：分散を計算するpythonコード
def variance(x):
  mean_val = np.mean(x)
  n_data = len(x)
  sum_val = 0
  for i in range(len(x)):
    sum_val += (x[i] - mean_val) ** 2
  return sum_val / n_data
```

```
def standard_deviation(x):
  return np.sqrt(variance(x))
```

```
variance(x1), variance(x2)
```

    (13.369184608430686, 62.79554529772076)

```
standard_deviation(x1), standard_deviation(x2)
```

    (3.656389559173186, 7.924364031120779)

### 標準偏差

* 図１９ 標準偏差の式

標準偏差 $ = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} ( x_i - \bar x ) ^2 } $

* 図２０ 身長データの分布と標準偏差

```
np.random.seed(71)
n_data = 1000
x1 = np.random.normal(loc=171.4, scale=3.8, size=n_data)
x2 = np.random.normal(loc=171.4, scale=7.8, size=n_data)

x1_mean = np.mean(x1)
x2_mean = np.mean(x2)
x1_std = standard_deviation(x1)
x2_std = standard_deviation(x2)

plt.figure(figsize=(8,4))
plt.hist(x1, bins=np.arange(140, 200, 2.5), alpha=0.7, label="グループ1")
plt.hist(x2, bins=np.arange(140, 200, 2.5), alpha=0.7, label="グループ2")

plt.vlines(x1_mean, 0, 290, "b")
plt.vlines(x2_mean, 0, 290, "darkorange")
plt.hlines(150, x1_mean, x1_mean + x1_std, "b")
plt.hlines(50, x2_mean, x2_mean + x2_std, "darkorange")

plt.xlim(145, 200)
plt.ylabel("度数")
plt.xlabel("身長")
plt.legend(loc="best")
plt.show()

print(x1_mean, x2_mean)
```

＃ pandasを使った統計値の算出

* 表３ pandasにより算出した身長データの統計値

```
height_df = pd.DataFrame({"men": height_men, "women":height_women})
display(height_df.head())

height_df.describe()
```

      men
      women

      0
      168.902502
      156.303818

      1
      164.475217
      157.181695

      2
      168.823066
      155.049801

      3
      174.238591
      167.162850

      4
      168.780969
      155.546082

      men
      women

      count
      1000.000000
      1000.000000

      mean
      171.409283
      157.409554

      std
      5.583598
      5.488843

      min
      157.374672
      139.473176

      25%
      167.632105
      153.853533

      50%
      171.231193
      157.356507

      75%
      175.084984
      161.109156

      max
      187.664545
      175.292621