# PHY321: Harmonic Oscillations, Damping, Resonances and time-dependent Forces
**[Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/)**, Department of Physics and Astronomy and Facility for Rare Ion Beams (FRIB), Michigan State University, USA and Department of Physics, University of Oslo, Norway

Date: **Mar 3, 2022**

Copyright 1999-2022, [Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/). Released under CC Attribution-NonCommercial 4.0 license

## Aims and Overarching Motivation

### Monday, February 28

Damped oscillations. Analytical and numerical solutions.
1. [Video of lecture](https://youtu.be/kt4LYyJc_aE)

2. [Handwritten notes](https://github.com/mhjensen/Physics321/blob/master/doc/HandWrittenNotes/Spring2022/NotesFebruary28.pdf)

**Reading suggestion**: Taylor sections 5.4-5.5. See also slides from last week.

### Wednesday, March 2

Driven oscillations and resonances with examples. Discussion of first midterm. First part of Wednesday is a regular lecture. The second half is dedicated to the midterm.

1. [Video of lecture](https://youtu.be/jQ-l6P3qW18)

2. [Handwritten notes](https://github.com/mhjensen/Physics321/blob/master/doc/HandWrittenNotes/Spring2022/NotesMarch2.pdf)

**Reading suggestion**: Taylor sections 5.5-5.6.

### Friday, March 4

Work on first midterm. Then spring break!

## Damped Oscillators

We consider only the case where the damping force is proportional to
the velocity. This is counter to dragging friction, where the force is
proportional in strength to the normal force and independent of
velocity, and is also inconsistent with wind resistance, where the
magnitude of the drag force is proportional the square of the
velocity. Rolling resistance does seem to be mainly proportional to
the velocity. However, the main motivation for considering damping
forces proportional to the velocity is that the math is more
friendly. This is because the differential equation is linear,
i.e. each term is of order $x$, $\dot{x}$, $\ddot{x}\cdots$, or even
terms with no mention of $x$, and there are no terms such as $x^2$ or
$x\ddot{x}$. The equations of motion for a spring with damping force
$-b\dot{x}$ are

$$
\begin{equation}
m\ddot{x}+b\dot{x}+kx=0.
\label{_auto1} \tag{1}
\end{equation}
$$

## Harmonic Oscillator,  Damping

Just to make the solution a bit less messy, we rewrite this equation as

$$
\begin{equation}
\label{eq:dampeddiffyq} \tag{2}
\ddot{x}+2\beta\dot{x}+\omega_0^2x=0,~~~~\beta\equiv b/2m,~\omega_0\equiv\sqrt{k/m}.
\end{equation}
$$

Both $\beta$ and $\omega$ have dimensions of inverse time. To find solutions (see appendix C in the text) you must make an educated guess at the form of the solution. To do this, first realize that the solution will need an arbitrary normalization $A$ because the equation is linear. Secondly, realize that if the form is

$$
\begin{equation}
x=Ae^{rt}
\label{_auto2} \tag{3}
\end{equation}
$$

that each derivative simply brings out an extra power of $r$. This
means that the $Ae^{rt}$ factors out and one can simply solve for an
equation for $r$. Plugging this form into Eq. ([2](#eq:dampeddiffyq)),

$$
\begin{equation}
r^2+2\beta r+\omega_0^2=0.
\label{_auto3} \tag{4}
\end{equation}
$$

## Harmonic Oscillator,  Solutions of Damped Motion

Because this is a quadratic equation there will be two solutions,

$$
\begin{equation}
r=-\beta\pm\sqrt{\beta^2-\omega_0^2}.
\label{_auto4} \tag{5}
\end{equation}
$$

We refer to the two solutions as $r_1$ and $r_2$ corresponding to the
$+$ and $-$ roots. As expected, there should be two arbitrary
constants involved in the solution,

$$
\begin{equation}
x=A_1e^{r_1t}+A_2e^{r_2t},
\label{_auto5} \tag{6}
\end{equation}
$$

where the coefficients $A_1$ and $A_2$ are determined by initial
conditions.

The roots listed above, $\sqrt{\omega_0^2-\beta_0^2}$, will be
imaginary if the damping is small and $\beta<\omega_0$. In that case,
$r$ is complex and the factor $\exp{(rt)}$ will have some oscillatory
behavior. If the roots are real, there will only be exponentially
decaying solutions. There are three cases:

## Underdamped: $\beta<\omega_0$

$$
\begin{eqnarray}
x&=&A_1e^{-\beta t}e^{i\omega't}+A_2e^{-\beta t}e^{-i\omega't},~~\omega'\equiv\sqrt{\omega_0^2-\beta^2}\\
\nonumber
&=&(A_1+A_2)e^{-\beta t}\cos\omega't+i(A_1-A_2)e^{-\beta t}\sin\omega't.
\end{eqnarray}
$$

Here we have made use of the identity
$e^{i\omega't}=\cos\omega't+i\sin\omega't$. Because the constants are
arbitrary, and because the real and imaginary parts are both solutions
individually, we can simply consider the real part of the solution
alone:

$$
\begin{eqnarray}
\label{eq:homogsolution} \tag{7}
x&=&B_1e^{-\beta t}\cos\omega't+B_2e^{-\beta t}\sin\omega't,\\
\nonumber
\omega'&\equiv&\sqrt{\omega_0^2-\beta^2}.
\end{eqnarray}
$$

## Critical dampling: $\beta=\omega_0$

In this case the two terms involving $r_1$ and $r_2$ are identical
because $\omega'=0$. Because we need to arbitrary constants, there
needs to be another solution. This is found by simply guessing, or by
taking the limit of $\omega'\rightarrow 0$ from the underdamped
solution. The solution is then

$$
\begin{equation}
\label{eq:criticallydamped} \tag{8}
x=Ae^{-\beta t}+Bte^{-\beta t}.
\end{equation}
$$

The critically damped solution is interesting because the solution
approaches zero quickly, but does not oscillate. For a problem with
zero initial velocity, the solution never crosses zero. This is a good
choice for designing shock absorbers or swinging doors.

## Overdamped: $\beta>\omega_0$

$$
\begin{eqnarray}
x&=&A_1\exp{-(\beta+\sqrt{\beta^2-\omega_0^2})t}+A_2\exp{-(\beta-\sqrt{\beta^2-\omega_0^2})t}
\end{eqnarray}
$$

This solution will also never pass the origin more than once, and then
only if the initial velocity is strong and initially toward zero.

Given $b$, $m$ and $\omega_0$, find $x(t)$ for a particle whose
initial position is $x=0$ and has initial velocity $v_0$ (assuming an
underdamped solution).

The solution is of the form,

$$
\begin{eqnarray*}
x&=&e^{-\beta t}\left[A_1\cos(\omega' t)+A_2\sin\omega't\right],\\
\dot{x}&=&-\beta x+\omega'e^{-\beta t}\left[-A_1\sin\omega't+A_2\cos\omega't\right].\\
\omega'&\equiv&\sqrt{\omega_0^2-\beta^2},~~~\beta\equiv b/2m.
\end{eqnarray*}
$$

From the initial conditions, $A_1=0$ because $x(0)=0$ and $\omega'A_2=v_0$. So

$$
x=\frac{v_0}{\omega'}e^{-\beta t}\sin\omega't.
$$

## Harmonic Oscillator,  Solutions

Consider a single solution with no arbitrary constants, which we will
call a **particular solution**, $x_p(t)$. It should be emphasized
that this is **A** particular solution, because there exists an
infinite number of such solutions because the general solution should
have two arbitrary constants. Now consider solutions to the same
equation without the driving term, which include two arbitrary
constants. These are called either **homogenous solutions** or
**complementary solutions**, and were given above,
e.g. Eq. ([7](#eq:homogsolution)) for the underdamped case. The
homogenous solution already incorporates the two arbitrary constants,
so any sum of a homogenous solution and a particular solution will
represent the **general solution** of the equation. The general
solution incorporates the two arbitrary constants $A$ and $B$ to
accommodate the two initial conditions. One could have picked a
different particular solution, i.e. the original particular solution
plus any homogenous solution with the arbitrary constants $A_p$ and
$B_p$ chosen at will. When one adds in the homogenous solution, which
has adjustable constants with arbitrary constants $A'$ and $B'$, to
the new particular solution, one can get the same general solution by
simply adjusting the new constants such that $A'+A_p=A$ and
$B'+B_p=B$. Thus, the choice of $A_p$ and $B_p$ are irrelevant, and
when choosing the particular solution it is best to make the simplest
choice possible.

## Harmonic Oscillator,  Particular Solution

To find a particular solution, one first guesses at the form,

$$
\begin{equation}
\label{eq:partform} \tag{9}
x_p(t)=D\cos(\omega t-\delta),
\end{equation}
$$

and rewrite the differential equation as

$$
\begin{equation}
D\left\{-\omega^2\cos(\omega t-\delta)-2\beta\omega\sin(\omega t-\delta)+\omega_0^2\cos(\omega t-\delta)\right\}=\frac{F_0}{m}\cos(\omega t).
\label{_auto6} \tag{10}
\end{equation}
$$

One can now use angle addition formulas to get

$$
\begin{eqnarray}
D\left\{(-\omega^2\cos\delta+2\beta\omega\sin\delta+\omega_0^2\cos\delta)\cos(\omega t)\right.&&\\
\nonumber
\left.+(-\omega^2\sin\delta-2\beta\omega\cos\delta+\omega_0^2\sin\delta)\sin(\omega t)\right\}
&=&\frac{F_0}{m}\cos(\omega t).
\end{eqnarray}
$$

Both the $\cos$ and $\sin$ terms need to equate if the expression is to hold at all times. Thus, this becomes two equations

$$
\begin{eqnarray}
D\left\{-\omega^2\cos\delta+2\beta\omega\sin\delta+\omega_0^2\cos\delta\right\}&=&\frac{F_0}{m}\\
\nonumber
-\omega^2\sin\delta-2\beta\omega\cos\delta+\omega_0^2\sin\delta&=&0.
\end{eqnarray}
$$

After dividing by $\cos\delta$, the lower expression leads to

$$
\begin{equation}
\tan\delta=\frac{2\beta\omega}{\omega_0^2-\omega^2}.
\label{_auto7} \tag{11}
\end{equation}
$$

## Solving with Driven Oscillations

Using the identities $\tan^2+1=\csc^2$ and $\sin^2+\cos^2=1$, one can also express $\sin\delta$ and $\cos\delta$,

$$
\begin{eqnarray}
\sin\delta&=&\frac{2\beta\omega}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}},\\
\nonumber
\cos\delta&=&\frac{(\omega_0^2-\omega^2)}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}
\end{eqnarray}
$$

Inserting the expressions for $\cos\delta$ and $\sin\delta$ into the expression for $D$,

$$
\begin{equation}
\label{eq:Ddrive} \tag{12}
D=\frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}.
\end{equation}
$$

For a given initial condition, e.g. initial displacement and velocity,
one must add the homogenous solution then solve for the two arbitrary
constants. However, because the homogenous solutions decay with time
as $e^{-\beta t}$, the particular solution is all that remains at
large times, and is therefore the steady state solution. Because the
arbitrary constants are all in the homogenous solution, all memory of
the initial conditions are lost at large times, $t>>1/\beta$.

The amplitude of the motion, $D$, is linearly proportional to the
driving force ($F_0/m$), but also depends on the driving frequency
$\omega$. For small $\beta$ the maximum will occur at
$\omega=\omega_0$. This is referred to as a resonance. In the limit
$\beta\rightarrow 0$ the amplitude at resonance approaches infinity.

## Alternative Derivation for Driven Oscillators

Here, we derive the same expressions as in Equations ([9](#eq:partform)) and ([12](#eq:Ddrive)) but express the driving forces as

$$
\begin{eqnarray}
F(t)&=&F_0e^{i\omega t},
\end{eqnarray}
$$

rather than as $F_0\cos\omega t$. The real part of $F$ is the same as before. For the differential equation,

$$
\begin{eqnarray}
\label{eq:compdrive} \tag{13}
\ddot{x}+2\beta\dot{x}+\omega_0^2x&=&\frac{F_0}{m}e^{i\omega t},
\end{eqnarray}
$$

one can treat $x(t)$ as an imaginary function. Because the operations
$d^2/dt^2$ and $d/dt$ are real and thus do not mix the real and
imaginary parts of $x(t)$, Eq. ([13](#eq:compdrive)) is effectively 2
equations. Because $e^{\omega t}=\cos\omega t+i\sin\omega t$, the real
part of the solution for $x(t)$ gives the solution for a driving force
$F_0\cos\omega t$, and the imaginary part of $x$ corresponds to the
case where the driving force is $F_0\sin\omega t$. It is rather easy
to solve for the complex $x$ in this case, and by taking the real part
of the solution, one finds the answer for the $\cos\omega t$ driving
force.

We assume a simple form for the particular solution

$$
\begin{equation}
x_p=De^{i\omega t},
\label{_auto8} \tag{14}
\end{equation}
$$

where $D$ is a complex constant.

From Eq. ([13](#eq:compdrive)) one inserts the form for $x_p$ above to get

$$
\begin{eqnarray}
D\left\{-\omega^2+2i\beta\omega+\omega_0^2\right\}e^{i\omega t}=(F_0/m)e^{i\omega t},\\
\nonumber
D=\frac{F_0/m}{(\omega_0^2-\omega^2)+2i\beta\omega}.
\end{eqnarray}
$$

The norm and phase for $D=|D|e^{-i\delta}$ can be read by inspection,

$$
\begin{equation}
|D|=\frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}},~~~~\tan\delta=\frac{2\beta\omega}{\omega_0^2-\omega^2}.
\label{_auto9} \tag{15}
\end{equation}
$$

This is the same expression for $\delta$ as before. One then finds $x_p(t)$,

$$
\begin{eqnarray}
\label{eq:fastdriven1} \tag{16}
x_p(t)&=&\Re\frac{(F_0/m)e^{i\omega t-i\delta}}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}\\
\nonumber
&=&\frac{(F_0/m)\cos(\omega t-\delta)}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}.
\end{eqnarray}
$$

This is the same answer as before.
If one wished to solve for the case where $F(t)= F_0\sin\omega t$, the imaginary part of the solution would work

$$
\begin{eqnarray}
\label{eq:fastdriven2} \tag{17}
x_p(t)&=&\Im\frac{(F_0/m)e^{i\omega t-i\delta}}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}\\
\nonumber
&=&\frac{(F_0/m)\sin(\omega t-\delta)}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}.
\end{eqnarray}
$$

## Damped and Driven Oscillator

Consider the damped and driven harmonic oscillator worked out above. Given $F_0, m,\beta$ and $\omega_0$, solve for the complete solution $x(t)$ for the case where $F=F_0\sin\omega t$ with initial conditions $x(t=0)=0$ and $v(t=0)=0$. Assume the underdamped case.

The general solution including the arbitrary constants includes both the homogenous and particular solutions,

$$
\begin{eqnarray*}
x(t)&=&\frac{F_0}{m}\frac{\sin(\omega t-\delta)}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}
+A\cos\omega't e^{-\beta t}+B\sin\omega't e^{-\beta t}.
\end{eqnarray*}
$$

The quantities $\delta$ and $\omega'$ are given earlier, $\omega'=\sqrt{\omega_0^2-\beta^2},
\delta=\tan^{-1}(2\beta\omega/(\omega_0^2-\omega^2)$. Here, solving
the problem means finding the arbitrary constants $A$ and
$B$. Satisfying the initial conditions for the initial position and
velocity:

$$
\begin{eqnarray*}
x(t=0)=0&=&-\eta\sin\delta+A,\\
v(t=0)=0&=&\omega\eta\cos\delta-\beta A+\omega'B,\\
\eta&\equiv&\frac{F_0}{m}\frac{1}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}.
\end{eqnarray*}
$$

The problem is now reduced to 2 equations and 2 unknowns, $A$ and $B$. The solution is

$$
\begin{eqnarray}
A&=& \eta\sin\delta ,~~~B=\frac{-\omega\eta\cos\delta+\beta\eta\sin\delta}{\omega'}.
\end{eqnarray}
$$

## Resonance Widths; the $Q$ factor

From above, the particular solution for a driving force, $F=F_0\cos\omega t$, is

$$
\begin{eqnarray}
x_p(t)&=&\frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}\cos(\omega_t-\delta),\\
\nonumber
\delta&=&\tan^{-1}\left(\frac{2\beta\omega}{\omega_0^2-\omega^2}\right).
\end{eqnarray}
$$

If one fixes the driving frequency $\omega$ and adjusts the
fundamental frequency $\omega_0=\sqrt{k/m}$, the maximum amplitude
occurs when $\omega_0=\omega$ because that is when the term from the
denominator $(\omega_0^2-\omega^2)^2+4\omega^2\beta^2$ is at a
minimum. This is akin to dialing into a radio station. However, if one
fixes $\omega_0$ and adjusts the driving frequency one minimize with
respect to $\omega$, e.g. set

$$
\begin{equation}
\frac{d}{d\omega}\left[(\omega_0^2-\omega^2)^2+4\omega^2\beta^2\right]=0,
\label{_auto10} \tag{18}
\end{equation}
$$

and one finds that the maximum amplitude occurs when
$\omega=\sqrt{\omega_0^2-2\beta^2}$. If $\beta$ is small relative to
$\omega_0$, one can simply state that the maximum amplitude is

$$
\begin{equation}
x_{\rm max}\approx\frac{F_0}{2m\beta \omega_0}.
\label{_auto11} \tag{19}
\end{equation}
$$

$$
\begin{eqnarray}
\frac{4\omega^2\beta^2}{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}=\frac{1}{2}.
\end{eqnarray}
$$

For small damping this occurs when $\omega=\omega_0\pm \beta$, so the $FWHM\approx 2\beta$. For the purposes of tuning to a specific frequency, one wants the width to be as small as possible. The ratio of $\omega_0$ to $FWHM$ is known as the [quality factor](https://en.wikipedia.org/wiki/Q_factor), or $Q$ factor,

$$
\begin{equation}
Q\equiv \frac{\omega_0}{2\beta}.
\label{_auto12} \tag{20}
\end{equation}
$$

## Numerical Studies of Driven Oscillations

Solving the problem of driven oscillations numerically gives us much
more flexibility to study different types of driving forces. We can
reuse our earlier code by simply adding a driving force. If we stay in
the $x$-direction only this can be easily done by adding a term
$F_{\mathrm{ext}}(x,t)$. Note that we have kept it rather general
here, allowing for both a spatial and a temporal dependence.

Before we dive into the code, we need to briefly remind ourselves
about the equations we started with for the case with damping, namely

$$
m\frac{d^2x}{dt^2} + b\frac{dx}{dt}+kx(t) =0,
$$

with no external force applied to the system.

Let us now for simplicty assume that our external force is given by

$$
F_{\mathrm{ext}}(t) = F_0\cos{(\omega t)},
$$

where $F_0$ is a constant (what is its dimension?) and $\omega$ is the frequency of the applied external driving force.
**Small question:** would you expect energy to be conserved now?

Introducing the external force into our lovely differential equation
and dividing by $m$ and introducing $\omega_0^2=\sqrt{k/m}$ we have

$$
\frac{d^2x}{dt^2} + \frac{b}{m}\frac{dx}{dt}+\omega_0^2x(t) =\frac{F_0}{m}\cos{(\omega t)},
$$

Thereafter we introduce a dimensionless time $\tau = t\omega_0$
and a dimensionless frequency $\tilde{\omega}=\omega/\omega_0$. We have then

$$
\frac{d^2x}{d\tau^2} + \frac{b}{m\omega_0}\frac{dx}{d\tau}+x(\tau) =\frac{F_0}{m\omega_0^2}\cos{(\tilde{\omega}\tau)},
$$

Introducing a new amplitude $\tilde{F} =F_0/(m\omega_0^2)$ (check dimensionality again) we have

$$
\frac{d^2x}{d\tau^2} + \frac{b}{m\omega_0}\frac{dx}{d\tau}+x(\tau) =\tilde{F}\cos{(\tilde{\omega}\tau)}.
$$

Our final step, as we did in the case of various types of damping, is
to define $\gamma = b/(2m\omega_0)$ and rewrite our equations as

$$
\frac{d^2x}{d\tau^2} + 2\gamma\frac{dx}{d\tau}+x(\tau) =\tilde{F}\cos{(\tilde{\omega}\tau)}.
$$

This is the equation we will code below using the Euler-Cromer method.

```
%matplotlib inline

# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

from pylab import plt, mpl
plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'

DeltaT = 0.001
#set up arrays
tfinal = 20 # in dimensionless time
n = ceil(tfinal/DeltaT)
# set up arrays for t, v, and x
t = np.zeros(n)
v = np.zeros(n)
x = np.zeros(n)
# Initial conditions as one-dimensional arrays of time
x0 =  1.0
v0 = 0.0
x[0] = x0
v[0] = v0
gamma = 0.2
Omegatilde = 0.5
Ftilde = 1.0
# Start integrating using Euler-Cromer's method
for i in range(n-1):
    # Set up the acceleration
    # Here you could have defined your own function for this
    a =  -2*gamma*v[i]-x[i]+Ftilde*cos(t[i]*Omegatilde)
    # update velocity, time and position
    v[i+1] = v[i] + DeltaT*a
    x[i+1] = x[i] + DeltaT*v[i+1]
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots()
ax.set_ylabel('x[m]')
ax.set_xlabel('t[s]')
ax.plot(t, x)
fig.tight_layout()
save_fig("ForcedBlockEulerCromer")
plt.show()
```

In the above example we have focused on the Euler-Cromer method. This
method has a local truncation error which is proportional to $\Delta t^2$
and thereby a global error which is proportional to $\Delta t$.
We can improve this by using the Runge-Kutta family of
methods. The widely popular Runge-Kutta to fourth order or just **RK4**
has indeed a much better truncation error. The RK4 method has a global
error which is proportional to $\Delta t$.

Let us revisit this method and see how we can implement it for the above example.

## Differential Equations, Runge-Kutta methods

Runge-Kutta (RK) methods are based on Taylor expansion formulae, but yield
in general better algorithms for solutions of an ordinary differential equation.
The basic philosophy is that it provides an intermediate step in the computation of $y_{i+1}$.

To see this, consider first the following definitions

$$
\begin{equation}
\frac{dy}{dt}=f(t,y),
\label{_auto13} \tag{21}
\end{equation}
$$

and

$$
\begin{equation}
y(t)=\int f(t,y) dt,
\label{_auto14} \tag{22}
\end{equation}
$$

and

$$
\begin{equation}
y_{i+1}=y_i+ \int_{t_i}^{t_{i+1}} f(t,y) dt.
\label{_auto15} \tag{23}
\end{equation}
$$

To demonstrate the philosophy behind RK methods, let us consider
the second-order RK method, RK2.
The first approximation consists in Taylor expanding $f(t,y)$
around the center of the integration interval $t_i$ to $t_{i+1}$,
that is, at $t_i+h/2$, $h$ being the step.
Using the midpoint formula for an integral,
defining $y(t_i+h/2) = y_{i+1/2}$ and
$t_i+h/2 = t_{i+1/2}$, we obtain

$$
\begin{equation}
\int_{t_i}^{t_{i+1}} f(t,y) dt \approx hf(t_{i+1/2},y_{i+1/2}) +O(h^3).
\label{_auto16} \tag{24}
\end{equation}
$$

This means in turn that we have

$$
\begin{equation}
y_{i+1}=y_i + hf(t_{i+1/2},y_{i+1/2}) +O(h^3).
\label{_auto17} \tag{25}
\end{equation}
$$

However, we do not know the value of   $y_{i+1/2}$. Here comes thus the next approximation, namely, we use Euler's
method to approximate $y_{i+1/2}$. We have then

$$
\begin{equation}
y_{(i+1/2)}=y_i + \frac{h}{2}\frac{dy}{dt}=y(t_i) + \frac{h}{2}f(t_i,y_i).
\label{_auto18} \tag{26}
\end{equation}
$$

This means that we can define the following algorithm for
the second-order Runge-Kutta method, RK2.

$$
\begin{equation}
k_1=hf(t_i,y_i),
\label{_auto19} \tag{27}
\end{equation}
$$

$$
\begin{equation}
k_2=hf(t_{i+1/2},y_i+k_1/2),
\label{_auto20} \tag{28}
\end{equation}
$$

with the final value

$$
\begin{equation}
y_{i+i}\approx y_i + k_2 +O(h^3).
\label{_auto21} \tag{29}
\end{equation}
$$

The difference between the previous one-step methods
is that we now need an intermediate step in our evaluation,
namely $t_i+h/2 = t_{(i+1/2)}$ where we evaluate the derivative $f$.
This involves more operations, but the gain is a better stability
in the solution.

The fourth-order Runge-Kutta, RK4, has the following algorithm

$$
k_1=hf(t_i,y_i) \hspace{0.5cm}   k_2=hf(t_i+h/2,y_i+k_1/2)
$$

$$
k_3=hf(t_i+h/2,y_i+k_2/2)\hspace{0.5cm}   k_4=hf(t_i+h,y_i+k_3)
$$

with the final result

$$
y_{i+1}=y_i +\frac{1}{6}\left( k_1 +2k_2+2k_3+k_4\right).
$$

Thus, the algorithm consists in first calculating $k_1$
with $t_i$, $y_1$ and $f$ as inputs. Thereafter, we increase the step
size by $h/2$ and calculate $k_2$, then $k_3$ and finally $k_4$. The global error goes as $O(h^4)$.

However, at this stage, if we keep adding different methods in our
main program, the code will quickly become messy and ugly. Before we
proceed thus, we will now introduce functions that enbody the various
methods for solving differential equations. This means that we can
separate out these methods in own functions and files (and later as classes and more
generic functions) and simply call them when needed. Similarly, we
could easily encapsulate various forces or other quantities of
interest in terms of functions. To see this, let us bring up the code
we developed above for the simple sliding block, but now only with the simple forward Euler method. We introduce
two functions, one for the simple Euler method and one for the
force.

Note that here the forward Euler method does not know the specific force function to be called.
It receives just an input the name. We can easily change the force by adding another function.

```
def ForwardEuler(v,x,t,n,Force):
    for i in range(n-1):
        v[i+1] = v[i] + DeltaT*Force(v[i],x[i],t[i])
        x[i+1] = x[i] + DeltaT*v[i]
        t[i+1] = t[i] + DeltaT
```

```
def SpringForce(v,x,t):
#   note here that we have divided by mass and we return the acceleration
    return  -2*gamma*v-x+Ftilde*cos(t*Omegatilde)
```

It is easy to add a new method like the Euler-Cromer

```
def ForwardEulerCromer(v,x,t,n,Force):
    for i in range(n-1):
        a = Force(v[i],x[i],t[i])
        v[i+1] = v[i] + DeltaT*a
        x[i+1] = x[i] + DeltaT*v[i+1]
        t[i+1] = t[i] + DeltaT
```

and the Velocity Verlet method (be careful with time-dependence here, it is not an ideal method for non-conservative forces))

```
def VelocityVerlet(v,x,t,n,Force):
    for i in range(n-1):
        a = Force(v[i],x[i],t[i])
        x[i+1] = x[i] + DeltaT*v[i]+0.5*a*DeltaT*DeltaT
        anew = Force(v[i],x[i+1],t[i+1])
        v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
        t[i+1] = t[i] + DeltaT
```

Finally, we can now add the Runge-Kutta2 method via a new function

```
def RK2(v,x,t,n,Force):
    for i in range(n-1):
# Setting up k1
        k1x = DeltaT*v[i]
        k1v = DeltaT*Force(v[i],x[i],t[i])
# Setting up k2
        vv = v[i]+k1v*0.5
        xx = x[i]+k1x*0.5
        k2x = DeltaT*vv
        k2v = DeltaT*Force(vv,xx,t[i]+DeltaT*0.5)
# Final result
        x[i+1] = x[i]+k2x
        v[i+1] = v[i]+k2v
	t[i+1] = t[i]+DeltaT
```

Finally, we can now add the Runge-Kutta2 method via a new function

```
def RK4(v,x,t,n,Force):
    for i in range(n-1):
# Setting up k1
        k1x = DeltaT*v[i]
        k1v = DeltaT*Force(v[i],x[i],t[i])
# Setting up k2
        vv = v[i]+k1v*0.5
        xx = x[i]+k1x*0.5
        k2x = DeltaT*vv
        k2v = DeltaT*Force(vv,xx,t[i]+DeltaT*0.5)
# Setting up k3
        vv = v[i]+k2v*0.5
        xx = x[i]+k2x*0.5
        k3x = DeltaT*vv
        k3v = DeltaT*Force(vv,xx,t[i]+DeltaT*0.5)
# Setting up k4
        vv = v[i]+k3v
        xx = x[i]+k3x
        k4x = DeltaT*vv
        k4v = DeltaT*Force(vv,xx,t[i]+DeltaT)
# Final result
        x[i+1] = x[i]+(k1x+2*k2x+2*k3x+k4x)/6.
        v[i+1] = v[i]+(k1v+2*k2v+2*k3v+k4v)/6.
        t[i+1] = t[i] + DeltaT
```

The Runge-Kutta family of methods are particularly useful when we have a time-dependent acceleration.
If we have forces which depend only the spatial degrees of freedom (no velocity and/or time-dependence), then energy conserving methods like the Velocity Verlet or the Euler-Cromer method are preferred. As soon as we introduce an explicit time-dependence and/or add dissipitave forces like friction or air resistance, then methods like the family of Runge-Kutta methods are well suited for this.
The code below uses the Runge-Kutta4 methods.

```
DeltaT = 0.001
#set up arrays
tfinal = 20 # in dimensionless time
n = ceil(tfinal/DeltaT)
# set up arrays for t, v, and x
t = np.zeros(n)
v = np.zeros(n)
x = np.zeros(n)
# Initial conditions (can change to more than one dim)
x0 =  1.0
v0 = 0.0
x[0] = x0
v[0] = v0
gamma = 0.2
Omegatilde = 0.5
Ftilde = 1.0
# Start integrating using Euler's method
# Note that we define the force function as a SpringForce
RK4(v,x,t,n,SpringForce)

# Plot position as function of time
fig, ax = plt.subplots()
ax.set_ylabel('x[m]')
ax.set_xlabel('t[s]')
ax.plot(t, x)
fig.tight_layout()
save_fig("ForcedBlockRK4")
plt.show()
```


# Linear Classifier

```
!git clone https://github.com/martin-fabbri/colab-notebooks.git
```

    Cloning into 'colab-notebooks'...
    remote: Enumerating objects: 29, done.
    remote: Counting objects: 100% (29/29), done.
    remote: Compressing objects: 100% (26/26), done.
    remote: Total 881 (delta 10), reused 16 (delta 3), pack-reused 852
    Receiving objects: 100% (881/881), 122.56 MiB | 10.33 MiB/s, done.
    Resolving deltas: 100% (460/460), done.
    Checking out files: 100% (165/165), done.

```
import os
os.chdir('colab-notebooks/')
!pwd
```

    /content/colab-notebooks

```
!pip install alibi
```

    Collecting alibi
      Downloading https://files.pythonhosted.org/packages/00/e7/54214fcf84a65339d6c993121da52edea52b56d39e6ec87ad30c755d665a/alibi-0.3.2-py3-none-any.whl (81kB)
         |████████████████████████████████| 92kB 2.5MB/s
    Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.6/dist-packages (from alibi) (4.6.3)
    Requirement already satisfied: spacy in /usr/local/lib/python3.6/dist-packages (from alibi) (2.1.9)
    Requirement already satisfied: tensorflow<2.0 in /usr/local/lib/python3.6/dist-packages (from alibi) (1.15.0)
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from alibi) (0.25.3)
    Requirement already satisfied: Pillow in /usr/local/lib/python3.6/dist-packages (from alibi) (6.2.2)
    Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from alibi) (2.21.0)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from alibi) (0.22.1)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from alibi) (1.17.5)
    Requirement already satisfied: scikit-image in /usr/local/lib/python3.6/dist-packages (from alibi) (0.16.2)
    Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (1.0.1)
    Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (0.2.4)
    Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (0.6.0)
    Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (2.0.1)
    Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (1.0.2)
    Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (2.0.3)
    Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (0.9.6)
    Requirement already satisfied: thinc<7.1.0,>=7.0.8 in /usr/local/lib/python3.6/dist-packages (from spacy->alibi) (7.0.8)
    Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (0.8.1)
    Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (3.10.0)
    Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (0.1.8)
    Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.1.0)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.12.0)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (3.1.0)
    Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (0.9.0)
    Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.0.8)
    Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.15.1)
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.15.0)
    Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (0.2.2)
    Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.15.0)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (0.33.6)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.1.0)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow<2.0->alibi) (1.11.2)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->alibi) (2018.9)
    Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->alibi) (2.6.1)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->alibi) (2019.11.28)
    Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->alibi) (2.8)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->alibi) (3.0.4)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->alibi) (1.24.3)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->alibi) (0.14.1)
    Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->alibi) (1.4.1)
    Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->alibi) (2.4.1)
    Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->alibi) (3.1.2)
    Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->alibi) (2.4)
    Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image->alibi) (1.1.1)
    Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.6/dist-packages (from thinc<7.1.0,>=7.0.8->spacy->alibi) (4.28.1)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow<2.0->alibi) (42.0.2)
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow<2.0->alibi) (2.8.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2.0->alibi) (0.16.0)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2.0->alibi) (3.1.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->alibi) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->alibi) (2.4.6)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->alibi) (0.10.0)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.0->scikit-image->alibi) (4.4.1)
    Installing collected packages: alibi
    Successfully installed alibi-0.3.2

```
import nltk
import en_core_web_sm
import re
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import spacy
import string
import mlxtend

from alibi.explainers import AnchorText
from alibi.datasets import fetch_movie_sentiment
from alibi.utils.download import spacy_model
from mlxtend.plotting import plot_decision_regions
# from nltk.corpus import stopwords
from scipy.optimize import minimize
from sklearn import datasets
from sklearn.base import TransformerMixin
from sklearn.datasets import load_digits
from sklearn.datasets import load_wine
from sklearn.datasets import load_breast_cancer
from sklearn.datasets import load_boston
from sklearn.datasets import fetch_openml
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
from sklearn.svm import SVC
from sklearn.utils import check_random_state
from spacy.lang.en import English
from plot_classifier import plot_classifier

from spacy.lang.en.stop_words import STOP_WORDS
nltk.download('stopwords')
nlp= en_core_web_sm.load()
nltk_stopwords = nltk.corpus.stopwords
stopwords = list(STOP_WORDS)
punctuations = string.punctuation
parser = English()
print(f'Inported stop words: {len(stopwords)}')
```

    [nltk_data] Downloading package stopwords to /root/nltk_data...
    [nltk_data]   Package stopwords is already up-to-date!
    Inported stop words: 326

```
def custom_tokenizer(sentence):
  custom_tokens = parser(sentence)
  custom_tokens = [ word.lemma_.lower().strip() if word.lemma_ != "-PRON-" else word.lower_ for word in custom_tokens ]
  custom_tokens = [ word for word in custom_tokens if word not in stopwords and word not in punctuations ]
  return custom_tokens
```

```
custom_tokenizer("This is just a test. But it's quite important to get it done.")
```

    ['test', 'important']

```
def spacy_tokenizer(doc):
    return [x.orth_ for x in nlp(doc)]

def clean_text(text):
  return text.strip().lower()

class predictors(TransformerMixin):
  def transform(self, X, **transform_params):
    return [clean_text(text) for text in X]

  def fit(self, X, y, **fit_params):
    return self

  def get_params(self, deep=True):
    return {}
```

## Warmup

```
imdb_movies_url = 'https://raw.githubusercontent.com/martin-fabbri/colab-notebooks/master/data/imdb_dataset.csv'
imdb_df = pd.read_csv(imdb_movies_url)
print(imdb_df.shape)
imdb_df.head(3)
```

    (50000, 2)

      review
      sentiment

      0
      One of the other reviewers has mentioned that ...
      positive

      1
      A wonderful little production. <br /><br />The...
      positive

      2
      I thought this was a wonderful way to spend ti...
      positive

```
imdb_df.describe()
```

      review
      sentiment

      count
      50000
      50000

      unique
      49582
      2

      top
      Loved today's show!!! It was a variety and not...
      negative

      freq
      5
      25000

```
imdb_df['sentiment'].value_counts()
```

    negative    25000
    positive    25000
    Name: sentiment, dtype: int64

```
le = LabelEncoder()
imdb_df['target'] = le.fit_transform(imdb_df['sentiment'])
imdb_df.head(3)
```

      review
      sentiment
      target

      0
      One of the other reviewers has mentioned that ...
      positive
      1

      1
      A wonderful little production. <br /><br />The...
      positive
      1

      2
      I thought this was a wonderful way to spend ti...
      positive
      1

```
from spacy.lang.en.stop_words import STOP_WORDS
X_arr = imdb_df['review'].tolist()

#Removing Stop Words
set(nltk_stopwords.words('english'))
STOPWORDS = '<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});'

cleanr = re.compile(STOPWORDS)

for i in range(0,len(X_arr)):
    X_arr[i] = re.sub(cleanr,"",X_arr[i])

print(X_arr[1])

stopwords_english = set(nltk_stopwords.words('english'))

for st in stopwords_english:
    for i in range(0,len(X_arr)):
        if st in X_arr[i]:
            X_arr[i].replace(st,'')
```

    A wonderful little production. The filming technique is very unassuming- very old-time-BBC fashion and gives a comforting, and sometimes discomforting, sense of realism to the entire piece. The actors are extremely well chosen- Michael Sheen not only "has got all the polari" but he has all the voices down pat too! You can truly see the seamless editing guided by the references to Williams' diary entries, not only is it well worth the watching but it is a terrificly written and performed piece. A masterful production about one of the great master's of comedy and his life. The realism really comes home with the little things: the fantasy of the guard which, rather than use the traditional 'dream' techniques remains solid then disappears. It plays on our knowledge and our senses, particularly with the scenes concerning Orton and Halliwell and the sets (particularly of their flat with Halliwell's murals decorating every surface) are terribly well done.

```
input_len = 2000
X = X_arr[0:2000]

#Lets use TFIDF to get the numerical values for the text
tfidf = TfidfVectorizer(max_df=0.5, min_df=2,
                                   ngram_range=(1,2),
                                   stop_words='english')

features = tfidf.fit_transform(X)
```

```
df = pd.DataFrame(features.todense(),
                 columns=tfidf.get_feature_names())
print(df.shape)
print(df.head())
```

    (2000, 25469)
        00  000  000 looks  007        10  ...  zooming  zooms   zu  zu warriors  zucker
    0  0.0  0.0        0.0  0.0  0.000000  ...      0.0    0.0  0.0          0.0     0.0
    1  0.0  0.0        0.0  0.0  0.000000  ...      0.0    0.0  0.0          0.0     0.0
    2  0.0  0.0        0.0  0.0  0.000000  ...      0.0    0.0  0.0          0.0     0.0
    3  0.0  0.0        0.0  0.0  0.055016  ...      0.0    0.0  0.0          0.0     0.0
    4  0.0  0.0        0.0  0.0  0.000000  ...      0.0    0.0  0.0          0.0     0.0

    [5 rows x 25469 columns]

```
X_train, y_train, X_test, y_test = (df[0:1000], imdb_df['target'][0:1000],
                                    df[1000:2000], imdb_df['target'][1000:2000])
print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)
```

    (1000, 25469) (1000,) (1000, 25469) (1000,)

```
knn = KNeighborsClassifier()
knn.fit(X_train, y_train)
pred = knn.predict([X_test.iloc[0]])
print('Prediction for test example 0:', pred)
```

    Prediction for test example 0: [0]

```
pred = knn.predict([X_test.iloc[4]])
print('Prediction for test example 4:', pred)
```

    Prediction for test example 4: [1]

## Running LogisticRegression and SVC

```
digits = datasets.load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target)

lr = LogisticRegression()
lr.fit(X_train, y_train)
print(lr.score(X_train, y_train))
print(lr.score(X_test, y_test))

svm = SVC()
svm.fit(X_train, y_train)
print(svm.score(X_train, y_train))
print(svm.score(X_test, y_test))

```

    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    1.0
    0.96
    0.9977728285077951
    0.9777777777777777

## Sentiment analysis for movie reviews

```
from spacy.lang.en.stop_words import STOP_WORDS
movies = fetch_movie_sentiment()
print(f'Movies dataset components: {movies.keys()}')

data, labels, target_names = (movies.data, movies.target, movies.target_names)
print(f'Number of movies loaded: {len(data)}')
print(f'Number labels: {len(labels)}')
print(f'Label names: {target_names}')
```

    Movies dataset components: dict_keys(['data', 'target', 'target_names'])
    Number of movies loaded: 10433
    Number labels: 10433
    Label names: ['negative', 'positive']

```
X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.3,
                                                    random_state=42)
y_train = np.array(y_train)
y_test = np.array(y_test)
```

```
vectorizer = CountVectorizer(min_df=1)
vectorizer.fit(X_train)
```

    CountVectorizer(analyzer='word', binary=False, decode_error='strict',
                    dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
                    lowercase=True, max_df=1.0, max_features=None, min_df=1,
                    ngram_range=(1, 1), preprocessor=None, stop_words=None,
                    strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
                    tokenizer=None, vocabulary=None)

```
np.random.seed(0)
clf = LogisticRegression(solver='liblinear')
clf.fit(vectorizer.transform(X_train), y_train)
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='auto', n_jobs=None, penalty='l2',
                       random_state=None, solver='liblinear', tol=0.0001, verbose=0,
                       warm_start=False)

```
predict_fn = lambda x: clf.predict(vectorizer.transform(x))
```

```
preds_train = predict_fn(X_train)
preds_test = predict_fn(X_test)
print('Train accuracy', accuracy_score(y_train, preds_train))
print('Test accuracy', accuracy_score(y_test, preds_test))
```

    Train accuracy 0.981651376146789
    Test accuracy 0.7667731629392971

```
review1 = 'LOVED IT! This movie was amazing. Top 10 this year.'
review1_features = vectorizer.transform([review1])
print('Probability of positive review:', clf.predict_proba(review1_features)[0,1])
```

    Probability of positive review: 0.6881337693954368

```
review2 = "Total junk! I'll never watch a film by that director again, no matter how good the reviews."
review2_features = vectorizer.transform([review2])
print('Probability of positive review:', clf.predict_proba(review2_features)[0,1])
```

    Probability of positive review: 0.23357442695010272

## Linear decision boundaries

```
wine = load_wine()
X, y = (wine.data[:,:2], wine.target)
models = [LogisticRegression(), LinearSVC(), SVC(), KNeighborsClassifier()]
models = (c.fit(X, y) for c in models)
```

```
def make_meshgrid(x, y, h=.02):
    """Create a mesh of points to plot in

    Parameters
    ----------
    x: data to base x-axis meshgrid on
    y: data to base y-axis meshgrid on
    h: stepsize for meshgrid, optional

    Returns
    -------
    xx, yy : ndarray
    """
    x_min, x_max = x.min() - 1, x.max() + 1
    y_min, y_max = y.min() - 1, y.max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    return xx, yy

def plot_contours(ax, clf, xx, yy, **params):
    """Plot the decision boundaries for a classifier.

    Parameters
    ----------
    ax: matplotlib axes object
    clf: a classifier
    xx: meshgrid ndarray
    yy: meshgrid ndarray
    params: dictionary of params to pass to contourf, optional
    """
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    out = ax.contourf(xx, yy, Z, **params)
    return out
```

```
titles = ('LogisticRegression',
          'LinearSVC',
          'SVC',
          'KNeighborsClassifier')

# Set-up 2x2 grid for plotting.
fig, sub = plt.subplots(2, 2)
plt.subplots_adjust(wspace=0.4, hspace=0.4)

X0, X1 = X[:, 0], X[:, 1]
xx, yy = make_meshgrid(X0, X1)

for clf, title, ax in zip(models, titles, sub.flatten()):
    plot_contours(ax, clf, xx, yy,
                  cmap=plt.cm.coolwarm, alpha=0.8)
    ax.scatter(X0, X1, c=y, cmap=plt.cm.coolwarm, s=20, edgecolors='k')
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    ax.set_title(title);

```

## How LogisticRegression makes predictions?

```
ds = load_breast_cancer()
X = ds.data
y = ds.target
```

```
lr = LogisticRegression()
lr.fit(X, y)
lr.predict(X)[10], lr.predict(X)[20]
```

    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    (0, 1)

```
# raw model output
print('lr.coef_ shape', lr.coef_.shape)
print('X[10] shaoe', X[10].shape)
print('Raw output', lr.coef_ @ X[10] + lr.intercept_)
print('since the raw output is negative, we predict the Negative class')
```

    lr.coef_ shape (1, 30)
    X[10] shaoe (30,)
    Raw output [-6.58983057]
    since the raw output is negative, we predict the Negative class

```
print('Raw output', lr.coef_ @ X[20] + lr.intercept_)
print('since the raw output is positive, we predict the Positive class')
```

    Raw output [5.42097797]
    since the raw output is positive, we predict the Positive class

## Minimizing a loss

Optimize:

\begin{equation}
\min x_1x_4 (x_1+x_3+x_3)+x_3
\end{equation}
\begin{equation}
s.t. x_1x_2x_3x_4>=25
\end{equation}
\begin{equation}
x_1^2+x_2^2+x_3^2+x_4^2=40
\end{equation}
\begin{equation}
1<=x_1,x_2,x_3,x_4<=5
\end{equation}
\begin{equation}
x_0=(1,5,5,1)
\end{equation}

```
def objective(x):
  x1, x2, x3, x4 = x
  return x1 * x4 * (x1 + x2 +x3) + x3

def constraint1(x):
  x1, x2, x3, x4 = x
  return x1 * x2 * x3 * x4 - 25

def constraint2(x):
  x1, x2, x3, x4 = x
  return 40 - x1**2 - x2**2 - x3**2 - x4**2

```

```
x0= [1, 5, 5, 1]
print(objective(x0))
```

    16

```
b = (1.0, 5.0)
bnds = (b,b,b,b)
con1 = {'type': 'ineq', 'fun': constraint1}
con2 = {'type': 'eq', 'fun': constraint2}
cons = [con1, con2]
```

```
sol = minimize(objective, x0, method='SLSQP', bounds=bnds, constraints=cons)
sol
```

         fun: 17.01401724563517
         jac: array([14.57227015,  1.37940764,  2.37940764,  9.56415057])
     message: 'Optimization terminated successfully.'
        nfev: 30
         nit: 5
        njev: 5
      status: 0
     success: True
           x: array([1.        , 4.7429961 , 3.82115462, 1.37940765])

```
sol.x
```

    array([1.        , 4.7429961 , 3.82115462, 1.37940765])

```
boston = load_boston()
X = boston.data
y = boston.target
```

```
def my_loss(w):
  l = 0
  for i in range(y.size):
    y_i_true = y[i]
    y_i_pred = w @ X[i]
    l = l + (y_i_true - y_i_pred) ** 2
  return l
```

```
w_fit = minimize(my_loss, X[0]).x
w_fit
```

    array([-9.28968320e-02,  4.87152411e-02, -4.05738481e-03,  2.85398463e+00,
           -2.86830574e+00,  5.92815252e+00, -7.26948711e-03, -9.68512138e-01,
            1.71156671e-01, -9.39666264e-03, -3.92188278e-01,  1.49054651e-02,
           -4.16304241e-01])

```
lr = LinearRegression(fit_intercept=False)
lr.fit(X, y)
lr.coef_
```

    array([-9.28965170e-02,  4.87149552e-02, -4.05997958e-03,  2.85399882e+00,
           -2.86843637e+00,  5.92814778e+00, -7.26933458e-03, -9.68514157e-01,
            1.71151128e-01, -9.39621540e-03, -3.92190926e-01,  1.49056102e-02,
           -4.16304471e-01])

## Comparing the logistic and hinge losses

```
def log_loss(raw_model_output):
  return np.log(1 + np.exp(-raw_model_output))
```

```
def hinge_loss(raw_model_output):
  return np.maximum(0, 1 - raw_model_output)
```

```
grid = np.linspace(-2, 2, 1000)
plt.plot(grid, log_loss(grid), label='logistic');
plt.plot(grid, hinge_loss(grid), label='hinge');
```

## Implementing logistic regression

```
ds = load_breast_cancer()
X = ds.data
y = np.where(ds.target == 0, -1, ds.target)
```

```
def my_loss(w):
  l = 0
  for i in range(y.size):
    raw_model_output = w @ X[i]
    l += log_loss(raw_model_output * y[i])
  return l

w_fit = minimize(my_loss, X[0]).x
print(w_fit)

lr = LogisticRegression(fit_intercept=False, C=1000000)
lr.fit(X, y)
print(lr.coef_)
```

    [1.799e+01 1.038e+01 1.228e+02 1.001e+03 1.184e-01 2.776e-01 3.001e-01
     1.471e-01 2.419e-01 7.871e-02 1.095e+00 9.053e-01 8.589e+00 1.534e+02
     6.399e-03 4.904e-02 5.373e-02 1.587e-02 3.003e-02 6.193e-03 2.538e+01
     1.733e+01 1.846e+02 2.019e+03 1.622e-01 6.656e-01 7.119e-01 2.654e-01
     4.601e-01 1.189e-01]
    [[ 1.0660862   0.50001866  0.26312059 -0.01646808 -0.04026022 -0.18847567
      -0.26373564 -0.11119512 -0.05574705 -0.0111655   0.04664792  0.42558937
       0.17422577 -0.13293594 -0.00362103 -0.04034038 -0.05625094 -0.01444271
      -0.0134644  -0.00375437  1.13345877 -0.57228181 -0.24533118 -0.01458381
      -0.07286688 -0.58662035 -0.73014074 -0.21369178 -0.17629518 -0.05654333]]

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: RuntimeWarning: overflow encountered in exp

    /usr/local/lib/python3.6/dist-packages/scipy/optimize/optimize.py:697: RuntimeWarning: invalid value encountered in double_scalars
      df = (f(*((xk + d,) + args)) - f0) / d[k]
    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

```
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)
random_state = check_random_state(0)
permutation = random_state.permutation(X.shape[0])
X = X[permutation]
y = y[permutation]
X = X.reshape((X.shape[0], -1))
X.shape, y.shape
```

    ((70000, 784), (70000,))

```
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=10000)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
```

```
train_errs = list()
valid_errs = list()
C_values = [0.001, 0.01, 0.1, 1, 10, 100, 1000]
for C_value in C_values:
  lr = LogisticRegression(C=C_value)
  lr.fit(X_train, y_train)
  train_errs.append(1.0 - lr.score(X_train, y_train))
  valid_errs.append(1.0 - lr.score(X_test, y_test))

plt.semilogx(C_values, train_errs, C_values, valid_errs)
plt.legend(('train', 'validation'));
```

```
# # Train and validaton errors initialized as empty list
# train_errs = list()
# valid_errs = list()

# # Loop over values of C_value
# for C_value in [0.001, 0.01, 0.1, 1, 10, 100, 1000]:
#     # Create LogisticRegression object and fit
#     lr = ____
#     lr.fit(____)

#     # Evaluate error rates and append to lists
#     train_errs.append( 1.0 - lr.score(____) )
#     valid_errs.append( 1.0 - lr.score(____) )

# # Plot results
# plt.semilogx(C_values, train_errs, C_values, valid_errs)
# plt.legend(("train", "validation"))
# plt.show()
```

## Logistic regression and feature

```
from spacy.lang.en.stop_words import STOP_WORDS
movies = fetch_movie_sentiment()
data, labels, target_names = (movies.data, movies.target, movies.target_names)
X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.3,
                                                    random_state=42)
y_train = np.array(y_train)
y_test = np.array(y_test)
vectorizer = CountVectorizer(min_df=1)
vectorizer.fit(X_train)
```

    CountVectorizer(analyzer='word', binary=False, decode_error='strict',
                    dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
                    lowercase=True, max_df=1.0, max_features=None, min_df=1,
                    ngram_range=(1, 1), preprocessor=None, stop_words=None,
                    strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
                    tokenizer=None, vocabulary=None)

```
lr = LogisticRegression(penalty='l2')
searcher = GridSearchCV(lr, {'C':[0.001, 0.01, 0.1, 1, 10]}, cv=3)
searcher.fit(vectorizer.transform(X_train), y_train)
print('Best CV params', searcher.best_params_)
best_lr = searcher.best_estimator_
coefs = best_lr.coef_
print('Total number of features:', coefs.size)
print('Number of selected features:', np.count_nonzero(coefs))
```

    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    Best CV params {'C': 1}
    Total number of features: 14978
    Number of selected features: 14978

    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

```
best_lr.coef_
```

    array([[-0.07044221, -0.13594933, -0.18873476, ..., -0.12842599,
            -0.0437782 , -0.54910657]])

## Regularization and probabilities

```
X = np.array([[ 1.78862847,  0.43650985],
     [ 0.09649747, -1.8634927 ],
     [-0.2773882 , -0.35475898],
     [-3.08274148,  2.37299932],
     [-3.04381817,  2.52278197],
     [-1.31386475,  0.88462238],
     [-2.11868196,  4.70957306],
     [-2.94996636,  2.59532259],
     [-3.54535995,  1.45352268],
     [ 0.98236743, -1.10106763],
     [-1.18504653, -0.2056499 ],
     [-1.51385164,  3.23671627],
     [-4.02378514,  2.2870068 ],
     [ 0.62524497, -0.16051336],
     [-3.76883635,  2.76996928],
     [ 0.74505627,  1.97611078],
     [-1.24412333, -0.62641691],
     [-0.80376609, -2.41908317],
     [-0.92379202, -1.02387576],
     [ 1.12397796, -0.13191423]])
y = np.array([-1, -1, -1,  1,  1, -1,  1,  1,  1, -1, -1,  1,  1, -1,  1, -1, -1,
     -1, -1, -1])

```

```
lr = LogisticRegression(C=1)
lr.fit(X, y)
plot_decision_regions(X, y, clf=lr);
```

```
lr = LogisticRegression(C=0.1)
lr.fit(X, y)
plot_decision_regions(X, y, clf=lr);
```

```
lr = LogisticRegression(C=1)
lr.fit(X, y)
plot_classifier(X, y, lr, proba=True);
```

```
lr = LogisticRegression(C=0.1)
lr.fit(X, y)
plot_classifier(X, y, lr, proba=True);
```

## Least and Most confident predictions

```
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)
random_state = check_random_state(0)
permutation = random_state.permutation(X.shape[0])
X = X[permutation]
y = y[permutation].astype('int')
X = X.reshape((X.shape[0], -1))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=10000)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
```

```
lr = LogisticRegression()
lr.fit(X, y)
# proba_inds = np.argsort(np)
```

    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='auto', n_jobs=None, penalty='l2',
                       random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                       warm_start=False)

```
proba = lr.predict_proba(X)
z = np.max(proba, axis=1)
```

```
z
```

    array([0.98221517, 0.99850338, 0.99790959, ..., 0.99756821, 0.98554044,
           0.99406461])

```
z.shape
```

    (70000,)

```
X.shape
```

    (70000, 784)

```
np.argsort(np.max(proba,axis=1))
```

    array([57753, 11040, 53809, ..., 48742, 51728, 20777])

## Fitting multi-class logistic regression

```
lr_ovr = LogisticRegression()
lr_ovr.fit(X_train, y_train)

print("OVR training accuracy:", lr_ovr.score(X_train, y_train))
print("OVR test accuracy    :", lr_ovr.score(X_test, y_test))

# Fit softmax classifier
lr_mn = LogisticRegression(multi_class='multinomial', solver='lbfgs')
lr_mn.fit(X_train, y_train)

print("Softmax training accuracy:", lr_mn.score(X_train, y_train))
print("Softmax test accuracy    :", lr_mn.score(X_test, y_test))
```

    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    OVR training accuracy: 0.9435833333333333
    OVR test accuracy    : 0.915
    Softmax training accuracy: 0.9435833333333333
    Softmax test accuracy    : 0.915

    /usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

```
# svm_class_1 = SVC()
# svm_class_1.fit(X_train, y_train)
# plot_classifier(X_train, y_train, svm_class_1)
```

```
# svm_class_1 = SVC()
# svm_class_1.fit(X_train, y_train==1)
# plot_classifier(X_train, y_train==1, svm_class_1)
```

## Support Vector Machines

```
wine = load_wine()
X, y = (wine.data[:,:2], wine.target)
```

```
svm = SVC(kernel='linear')
svm.fit(X, y)
plot_classifier(X, y, svm, lims=(11,15,0,6));
```

```
print('Number of original exmaples:', len(X))
print('Number of support vectors:', len(svm.support_))
```

    Number of original exmaples: 178
    Number of support vectors: 81

```
X_small = X[svm.support_]
y_small = y[svm.support_]
svm_small = SVC(kernel='linear')
svm_small.fit(X_small, y_small)
plot_classifier(X_small, y_small, svm_small, lims=(11,15,0,6));
```

# Kernell SVMs

```
digits = load_digits()
X = digits.data
y = digits.target == 2
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=43)
```

```
svm = SVC()
parameters = {'gamma' : [0.00001, 0.0001, 0.001, 0.01, 0.1]}
searcher = GridSearchCV(svm, parameters, cv=3)
searcher.fit(X_train, y_train)
print('Best CV params', searcher.best_params_)
```

    Best CV params {'gamma': 0.001}

```
svm = SVC()
parameters =  {'C' : [0.0001, 0.001, 0.1, 0.25, 0.5, 1, 10],
               'gamma' : [0.00001, 0.0001, 0.001, 0.01, 0.1]}
searcher = GridSearchCV(svm, parameters)
searcher.fit(X_train, y_train)
print('Best CV params', searcher.best_params_)
print('Best CV accuracy', searcher.best_score_)

print('Test accuracy of best grid search hypers:', searcher.score(X_test, y_test))
```

    Best CV params {'C': 10, 'gamma': 0.0001}
    Best CV accuracy 1.0
    Test accuracy of best grid search hypers: 0.9977753058954394

## SGDClassifier

```
linear_classifier = SGDClassifier(random_state=0)
parameters = {
    'alpha' : [0.00001, 0.0001, 0.001, 0.01, 0.1, 1],
    'loss' : ['hinge', 'log'],
    'penalty': ['l1', 'l2']
}
searcher = GridSearchCV(linear_classifier, parameters, cv=10)
searcher.fit(X_train, y_train)
print('Best CV params', searcher.best_params_)
print('Best CV accuracy', searcher.best_score_)
print('Test accuracy of best grid search hypers:', searcher.score(X_test, y_test))
```

    Best CV params {'alpha': 0.01, 'loss': 'hinge', 'penalty': 'l2'}
    Best CV accuracy 0.9977777777777778
    Test accuracy of best grid search hypers: 0.9977753058954394

```

```

```

```

```

```

```

```

```

```


```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl

from keras.models import Sequential
from keras.layers import Dense
from keras import regularizers
from keras.layers import Conv2D, Dropout, Flatten

from pandas import DataFrame

%matplotlib inline

## Set some values globally
# Ticks label size
label_size = 10
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size
# Ticks width
ticks_width = 0.5
mpl.rcParams['xtick.major.width'] = ticks_width
mpl.rcParams['ytick.major.width'] = ticks_width
# Line width
mpl.rcParams['axes.linewidth'] = 0.5
```

    Using TensorFlow backend.

### 2D Ising Gauge Hamiltonian:
\begin{equation}
H = -J \sum_p \prod_{i\in p} S^z_i
\end{equation}

```python
def plot_configuration(spins, dual=False):
    '''
    Helper function to plot the configuration of spins given by `spins`.

    Note: (i,j) denotes a vertex coordinate, such that the location of
    the plaquette center is at (i+0.5, j+0.5) and thus, the spin is
    at (i+1, j+0.5).

    Arguments:
        spins -- a spin configuration; type: numpy array of shape (N, N, 2).
        dual -- whether to plot the spin configuration in dual space;
                type: bool; default: False.
    '''
    N = np.shape(spins)[1]

    fig, ax = plt.subplots(dpi=100, figsize=(5, 5))
    fig.add_axes()
    ax = fig.axes[0]

    # Plot lattice (grid).
    for i in range(N+1):
        ax.plot([i, i], [0, N], 'k', lw=0.5)
        ax.plot([0, N], [i, i], 'k', lw=0.5)

    # Standard plot
    if not dual:
        # Plot spins (circles).
        colors = ['C0', 'C1']  # blue: down, orange: up.
        for i in range(N):
            fig.gca().add_patch(plt.Circle((0,i+0.5), radius=0.2, fc=colors[int((spins[-1, i, 0]+1)/2.)],
                                          ec='k', lw=0.3))
            fig.gca().add_patch(plt.Circle((i+0.5,0), radius=0.2, fc=colors[int((spins[i, -1, 1]+1)/2.)],
                                          ec='k', lw=0.3))
            for j in range(N):
                fig.gca().add_patch(plt.Circle((i+1,j+0.5), radius=0.2, fc=colors[int((spins[i, j, 0]+1)/2.)],
                                              ec='k', lw=0.3))
                fig.gca().add_patch(plt.Circle((i+0.5,j+1), radius=0.2, fc=colors[int((spins[i, j, 1]+1)/2.)],
                                              ec='k', lw=0.3))

    # Dual space plot
    if dual:
        excitation = []
        for i in range(N):
            if spins[-1, i, 0] == 1:
                ax.plot([-0.5, 0.5], [i+0.5, i+0.5], 'C2', lw=2)
            if spins[i, -1, 1] == 1:
                ax.plot([i+0.5, i+0.5], [-0.5, 0.5], 'C2', lw=2)
            for j in range(N):
                j_up = (N+j-1)%N
                i_left = (i+N-1)%N
                if spins[i, j, 0] == 1:
                    ax.plot([i+0.5, i+1.5], [j+0.5, j+0.5], 'C2', lw=2)
                if spins[i, j, 1] == 1:
                    ax.plot([i+0.5, i+0.5], [j+0.5, j+1.5], 'C2', lw=2)
                if spins[i, j, 0]*spins[i_left, j, 0]*spins[i, j, 1]*spins[i, j_up, 1] == -1:
                    excitation.append([i+0.5, j+0.5])

        if len(excitation) > 0:
            plt.scatter(np.array(excitation)[:, 0], np.array(excitation)[:, 1], color='C3', s=100, marker=(5,1))

    ax.set_ylim(-1,N+1)
    ax.set_xlim(-1,N+1)
    ax.set_aspect('equal')

    plt.xlabel("x")
    plt.ylabel("y")

    plt.axis('off')
#     fig.show()
    if dual==True:
        plt.savefig('ising_gauge_dual.png', bbox_inches='tight', dpi=600)
    else:
        plt.savefig('ising_gauge.png', bbox_inches='tight', dpi=600)
```

First we need to create training and test configurations with the script **create_configurations.py**, then run the next cells.

```python
# Training set.
train_configs = np.loadtxt("configs/train_configs.txt")
train_labels = np.loadtxt("configs/train_labels.txt")
N = int(np.sqrt(np.shape(train_configs[0])[0]/2))
print("Loaded %i configurations of size %ix%i for training."%(len(train_configs), N,N))
```

    Loaded 2000 configurations of size 16x16 for training.

```python
# Test set.
test_config = np.loadtxt("configs/test_configs.txt")
test_labels = np.loadtxt("configs/test_labels.txt")
N = int(np.sqrt(np.shape(test_config[0])[0]/2))
print("Loaded %i test configurations of size %ix%i for testing."%(len(test_config), N,N))
```

    Loaded 1000 test configurations of size 16x16 for testing.

```python
# Prepare data for training
x = train_configs
y = np.array([(i%2,(i+1)%2) for i in train_labels])
```

```python
# Standard plot
plot_configuration(np.reshape(train_configs[0], (N, N, 2)))
```

```python
# Dual space plot
plot_configuration(np.reshape(train_configs[0], (N, N, 2)), dual=True)
```

```python
# Show how excitations/defects are shown in 'dual' space
single_excitation = np.copy(train_configs[0])
i = 11
j = 5
single_excitation[i*N+j] *= -1
plot_configuration(np.reshape(single_excitation, (N, N, 2)), dual=True)
```

## Dense network - 1st try

```python
# Create simple model.
# Network architecture: input layer, hidden layer with 32 neurons, and output layer with 2 neurons.

model = Sequential()
model.add(Dense(32, input_dim=2*N**2, kernel_initializer='glorot_normal', activation='sigmoid',
                kernel_regularizer=regularizers.l2(0.02)))
model.add(Dense(2, kernel_initializer='glorot_normal', activation='softmax',
                kernel_regularizer=regularizers.l2(0.01)))

# Compile model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
```

```python
model.summary()
```

    Model: "sequential_3"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_5 (Dense)              (None, 32)                16416
    _________________________________________________________________
    dense_6 (Dense)              (None, 2)                 66
    =================================================================
    Total params: 16,482
    Trainable params: 16,482
    Non-trainable params: 0
    _________________________________________________________________

```python
# Train model.
model.fit(x, y, epochs=25, batch_size=32)
```

    Epoch 1/25
    2000/2000 [==============================] - 0s 105us/step - loss: 1.4993 - accuracy: 0.5035
    Epoch 2/25
    2000/2000 [==============================] - 0s 51us/step - loss: 0.9708 - accuracy: 0.5940
    Epoch 3/25
    2000/2000 [==============================] - 0s 51us/step - loss: 0.8036 - accuracy: 0.6415
    Epoch 4/25
    2000/2000 [==============================] - 0s 48us/step - loss: 0.7426 - accuracy: 0.6455
    Epoch 5/25
    2000/2000 [==============================] - 0s 48us/step - loss: 0.7163 - accuracy: 0.6540
    Epoch 6/25
    2000/2000 [==============================] - 0s 56us/step - loss: 0.7027 - accuracy: 0.6615
    Epoch 7/25
    2000/2000 [==============================] - 0s 59us/step - loss: 0.6965 - accuracy: 0.6600
    Epoch 8/25
    2000/2000 [==============================] - 0s 68us/step - loss: 0.6920 - accuracy: 0.6620
    Epoch 9/25
    2000/2000 [==============================] - 0s 56us/step - loss: 0.6877 - accuracy: 0.6555
    Epoch 10/25
    2000/2000 [==============================] - 0s 60us/step - loss: 0.6847 - accuracy: 0.6680
    Epoch 11/25
    2000/2000 [==============================] - 0s 71us/step - loss: 0.6839 - accuracy: 0.6590
    Epoch 12/25
    2000/2000 [==============================] - 0s 73us/step - loss: 0.6822 - accuracy: 0.6685
    Epoch 13/25
    2000/2000 [==============================] - 0s 76us/step - loss: 0.6807 - accuracy: 0.6625
    Epoch 14/25
    2000/2000 [==============================] - 0s 74us/step - loss: 0.6800 - accuracy: 0.6725
    Epoch 15/25
    2000/2000 [==============================] - 0s 72us/step - loss: 0.6793 - accuracy: 0.6610
    Epoch 16/25
    2000/2000 [==============================] - 0s 74us/step - loss: 0.6790 - accuracy: 0.6690
    Epoch 17/25
    2000/2000 [==============================] - 0s 65us/step - loss: 0.6773 - accuracy: 0.6640
    Epoch 18/25
    2000/2000 [==============================] - 0s 55us/step - loss: 0.6775 - accuracy: 0.6645
    Epoch 19/25
    2000/2000 [==============================] - 0s 49us/step - loss: 0.6751 - accuracy: 0.6760
    Epoch 20/25
    2000/2000 [==============================] - 0s 47us/step - loss: 0.6755 - accuracy: 0.6725
    Epoch 21/25
    2000/2000 [==============================] - 0s 47us/step - loss: 0.6757 - accuracy: 0.6730
    Epoch 22/25
    2000/2000 [==============================] - 0s 47us/step - loss: 0.6745 - accuracy: 0.6700
    Epoch 23/25
    2000/2000 [==============================] - 0s 51us/step - loss: 0.6743 - accuracy: 0.6750
    Epoch 24/25
    2000/2000 [==============================] - 0s 50us/step - loss: 0.6732 - accuracy: 0.6675
    Epoch 25/25
    2000/2000 [==============================] - 0s 48us/step - loss: 0.6726 - accuracy: 0.6725

    <keras.callbacks.callbacks.History at 0x637ff4fd0>

```python
# Prepare data for testing.
test_y = np.array([(i%2, (i+1)%2) for i in test_labels])
```

```python
# Evaluate model.
predictions = model.predict(test_config)
preds = model.evaluate(x=test_config, y=test_y)

print ("Loss = {0:.3f}".format(preds[0]))
print ("Test Accuracy = {0:.3f}".format(preds[1]))
```

    1000/1000 [==============================] - 0s 42us/step
    Loss = 0.787
    Test Accuracy = 0.487

The simple, fully-connected NN doesn't perform better than random chance (coin toss)...

## Larger/deeper dense network - 2nd try

```python
# Create bigger and deeper model.
# Network architecture: input layer, hidden layer #1 with 64 neurons,
# hidden layer #2 with 16 neurons, and output layer with 2 neurons.

model2 = Sequential()
model2.add(Dense(64, input_dim=2*N**2, kernel_initializer='glorot_normal', activation='sigmoid',
                 kernel_regularizer=regularizers.l2(0.02)))
model2.add(Dense(16, kernel_initializer='glorot_normal', activation='sigmoid',
                 kernel_regularizer=regularizers.l2(0.01)))
model2.add(Dense(2, kernel_initializer='glorot_normal', activation='softmax',
                 kernel_regularizer=regularizers.l2(0.01)))

# Compile model
model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
```

```python
# Train model.
model2.fit(x, y, epochs=25, batch_size=32)
```

    Epoch 1/25
    2000/2000 [==============================] - 0s 110us/step - loss: 2.0633 - accuracy: 0.4820
    Epoch 2/25
    2000/2000 [==============================] - 0s 64us/step - loss: 1.0510 - accuracy: 0.5315
    Epoch 3/25
    2000/2000 [==============================] - 0s 66us/step - loss: 0.8478 - accuracy: 0.5785
    Epoch 4/25
    2000/2000 [==============================] - 0s 63us/step - loss: 0.7954 - accuracy: 0.5750
    Epoch 5/25
    2000/2000 [==============================] - 0s 63us/step - loss: 0.7718 - accuracy: 0.5805
    Epoch 6/25
    2000/2000 [==============================] - 0s 64us/step - loss: 0.7588 - accuracy: 0.5735
    Epoch 7/25
    2000/2000 [==============================] - 0s 60us/step - loss: 0.7486 - accuracy: 0.5845
    Epoch 8/25
    2000/2000 [==============================] - 0s 66us/step - loss: 0.7410 - accuracy: 0.6030
    Epoch 9/25
    2000/2000 [==============================] - 0s 65us/step - loss: 0.7383 - accuracy: 0.5395
    Epoch 10/25
    2000/2000 [==============================] - 0s 60us/step - loss: 0.7329 - accuracy: 0.5435
    Epoch 11/25
    2000/2000 [==============================] - 0s 64us/step - loss: 0.7318 - accuracy: 0.5320
    Epoch 12/25
    2000/2000 [==============================] - 0s 62us/step - loss: 0.7267 - accuracy: 0.5855
    Epoch 13/25
    2000/2000 [==============================] - 0s 66us/step - loss: 0.7245 - accuracy: 0.5395
    Epoch 14/25
    2000/2000 [==============================] - 0s 59us/step - loss: 0.7239 - accuracy: 0.5040
    Epoch 15/25
    2000/2000 [==============================] - 0s 61us/step - loss: 0.7212 - accuracy: 0.5245
    Epoch 16/25
    2000/2000 [==============================] - 0s 61us/step - loss: 0.7188 - accuracy: 0.5255
    Epoch 17/25
    2000/2000 [==============================] - 0s 59us/step - loss: 0.7188 - accuracy: 0.5230
    Epoch 18/25
    2000/2000 [==============================] - 0s 60us/step - loss: 0.7160 - accuracy: 0.5190
    Epoch 19/25
    2000/2000 [==============================] - 0s 60us/step - loss: 0.7151 - accuracy: 0.5080
    Epoch 20/25
    2000/2000 [==============================] - 0s 63us/step - loss: 0.7136 - accuracy: 0.5130
    Epoch 21/25
    2000/2000 [==============================] - 0s 61us/step - loss: 0.7132 - accuracy: 0.4910
    Epoch 22/25
    2000/2000 [==============================] - 0s 67us/step - loss: 0.7108 - accuracy: 0.5080
    Epoch 23/25
    2000/2000 [==============================] - 0s 60us/step - loss: 0.7102 - accuracy: 0.5095
    Epoch 24/25
    2000/2000 [==============================] - 0s 62us/step - loss: 0.7091 - accuracy: 0.4905
    Epoch 25/25
    2000/2000 [==============================] - 0s 65us/step - loss: 0.7075 - accuracy: 0.5025

    <keras.callbacks.callbacks.History at 0x638e77f60>

```python
# Evaluate model.
predictions2 = model2.predict(test_config)
preds2 = model2.evaluate(x=test_config, y=test_y)

print ("Loss = {0:.3f}".format(preds2[0]))
print ("Test Accuracy = {0:.3f}".format(preds2[1]))
```

    1000/1000 [==============================] - 0s 59us/step
    Loss = 0.708
    Test Accuracy = 0.500

Again, pure random chance!

## Convolutional NN (CNN) - 3rd try

```python
# Reshape the configurations to fit the input layer of a CNN.
x = np.array([np.reshape(config, (N, N, 2)) for config in train_configs])
```

```python
def create_periodic_padding(configs, kernel_size):
    """
    Periodic `padding` for the spin configurations.

    Arguments:
        configs -- spin configuration; shape: NxNx2; type: (int, int, 2).
        kernel_size -- filter size; type: int.

    Returns:
        x -- padded spin configurations.
    """

    N = np.shape(configs)[1]
    padding = kernel_size-1  # Padding size

    x = []

    for config in configs:
        # Tensor to store padded configurations
        padded = np.zeros((N+2*padding, N+2*padding, 2))
        # Lower left corner
        padded[:padding, :padding, :] = config[N-padding:, N-padding:, :]
        # Lower middle
        padded[padding:N+padding, :padding, :] = config[:, N-padding:, :]
        # Lower right corner
        padded[N+padding:, :padding, :] = config[:padding, N-padding:, :]
        # Left side
        padded[:padding, padding:N+padding, :] = config[N-padding:, :, :]
        # Center
        padded[padding:N+padding, padding:N+padding, :] = config[:, :, :]
        # Right side
        padded[N+padding:, padding:N+padding, :] = config[:padding, :, :]
        # Top left corner
        padded[:padding, N+padding:, :] = config[N-padding:, :padding, :]
        # Top middle
        padded[padding:N+padding, N+padding:, :] = config[:, :padding, :]
        # Top right corner
        padded[N+padding:, N+padding:, :] = config[:padding, :padding, :]

        x.append(padded)

    return np.array(x)
```

```python
# Create CNN model.
# Network architecture: input layer, hidden conv. layer (#1) with 64 filters,
# hidden fully connected layer (#2) with 64 neurons, and output layer with 2 neurons.

def convolutional_model(kernel_size):
    input_size = 16+2*(kernel_size-1)

    model = Sequential()
    # [Conv. + Relu] block
    model.add(Conv2D(64, (kernel_size, kernel_size), strides=1, input_shape=(input_size, input_size, 2),
                     activation='relu'))
    # [Flatten (i.e. convert to vector) + fully connected] block
    model.add(Flatten())
    model.add(Dense(64, activation='relu'))
    # Dropout regularization
    model.add(Dropout(0.2))
    # Softmax output layer
    model.add(Dense(2, activation='softmax'))

    # Compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model
```

```python
# Prepare data for testing.
test_x = np.array([np.reshape(config, (N, N, 2)) for config in test_config])
```

```python
# Train models (this will take a while).
kernel_sizes = [2, 3, 4, 5, 6]
accuracy = []
runs = 10  # Number of runs per kernel size

for kernel_size in kernel_sizes:
    print("Kernel size is %ix%i"%(kernel_size, kernel_size))
    print('-------------------\n')

    x_n = create_periodic_padding(x, kernel_size)
    test_x_n = create_periodic_padding(test_x, kernel_size)

    for i in range(runs):
        print("run %i of %i"%(i+1, runs))

        # Create CNN model.
        model = convolutional_model(kernel_size)
        # Train model.
        model.fit(x_n, y, epochs=20, batch_size=32, verbose=0)

        # Evaluate model.
        preds = model.evaluate(test_x_n, test_y, verbose=0)
        acc = preds[1]
        accuracy.append(acc)
        print("Test accuracy = {0:.3f}".format(acc))
    print(' ')
```

    Kernel size is 2x2
    -------------------

    run 1 of 10
    Test accuracy = 1.000
    run 2 of 10
    Test accuracy = 1.000
    run 3 of 10
    Test accuracy = 0.996
    run 4 of 10
    Test accuracy = 1.000
    run 5 of 10
    Test accuracy = 1.000
    run 6 of 10
    Test accuracy = 1.000
    run 7 of 10
    Test accuracy = 1.000
    run 8 of 10
    Test accuracy = 1.000
    run 9 of 10
    Test accuracy = 1.000
    run 10 of 10
    Test accuracy = 1.000

    Kernel size is 3x3
    -------------------

    run 1 of 10
    Test accuracy = 1.000
    run 2 of 10
    Test accuracy = 1.000
    run 3 of 10
    Test accuracy = 1.000
    run 4 of 10
    Test accuracy = 1.000
    run 5 of 10
    Test accuracy = 0.992
    run 6 of 10
    Test accuracy = 1.000
    run 7 of 10
    Test accuracy = 1.000
    run 8 of 10
    Test accuracy = 1.000
    run 9 of 10
    Test accuracy = 1.000
    run 10 of 10
    Test accuracy = 0.999

    Kernel size is 4x4
    -------------------

    run 1 of 10
    Test accuracy = 0.893
    run 2 of 10
    Test accuracy = 0.869
    run 3 of 10
    Test accuracy = 0.514
    run 4 of 10
    Test accuracy = 0.911
    run 5 of 10
    Test accuracy = 0.743
    run 6 of 10
    Test accuracy = 0.585
    run 7 of 10
    Test accuracy = 0.500
    run 8 of 10
    Test accuracy = 0.833
    run 9 of 10
    Test accuracy = 0.506
    run 10 of 10
    Test accuracy = 0.974

    Kernel size is 5x5
    -------------------

    run 1 of 10
    Test accuracy = 0.635
    run 2 of 10
    Test accuracy = 0.484
    run 3 of 10
    Test accuracy = 0.489
    run 4 of 10
    Test accuracy = 0.489
    run 5 of 10
    Test accuracy = 0.496
    run 6 of 10
    Test accuracy = 0.516
    run 7 of 10
    Test accuracy = 0.498
    run 8 of 10
    Test accuracy = 0.503
    run 9 of 10
    Test accuracy = 0.497
    run 10 of 10
    Test accuracy = 0.493

    Kernel size is 6x6
    -------------------

    run 1 of 10
    Test accuracy = 0.500
    run 2 of 10
    Test accuracy = 0.500
    run 3 of 10
    Test accuracy = 0.510
    run 4 of 10
    Test accuracy = 0.490
    run 5 of 10
    Test accuracy = 0.504
    run 6 of 10
    Test accuracy = 0.488
    run 7 of 10
    Test accuracy = 0.493
    run 8 of 10
    Test accuracy = 0.513
    run 9 of 10
    Test accuracy = 0.504
    run 10 of 10
    Test accuracy = 0.486

```python
acc = np.reshape(np.array(accuracy), (len(kernel_sizes), runs))
```

```python
accur = {}

# Get accuracies per kernel size
for j, k in enumerate(kernel_sizes):
    accur['{}'.format(k)] = acc[j]
```

```python
results = DataFrame(accur)
print(results.describe())
```

                   2          3          4          5          6
    count  10.000000  10.000000  10.000000  10.000000  10.000000
    mean    0.999600   0.999100   0.732800   0.510000   0.498800
    std     0.001265   0.002514   0.188457   0.044803   0.009283
    min     0.996000   0.992000   0.500000   0.484000   0.486000
    25%     1.000000   1.000000   0.531750   0.490000   0.490750
    50%     1.000000   1.000000   0.788000   0.496500   0.500000
    75%     1.000000   1.000000   0.887000   0.501750   0.504000
    max     1.000000   1.000000   0.974000   0.635000   0.513000

```python
fig = plt.figure(dpi=100, figsize=(5, 3))

plt.errorbar(results.columns, results.mean(), yerr=results.std(),
             ls='--', marker='o', markeredgecolor='k', elinewidth=1.2, capsize=4)

plt.xlabel("Kernel size")
plt.ylabel("Accuracy")
plt.ylim(0, 1)

# plt.show()
plt.savefig("kernel_sizes_acc.png", bbox_inches='tight', dpi=600)
```

```python

```