# Lab 08 - Unsupervised learning

(Adapted from [one of the 2019 ICFA tutorial notebooks](https://github.com/radiasoft/icfa_ml/blob/master/tutorials_2019/unsupervized_learning/unsupervised_learning.ipynb))

## Homework
- 8-1: K-means clustering comparison
- 8-2 B: DBSCAN algorithm

# Environment

```python
%reset -f

# This will import np, plt
%pylab --no-import-all inline

# For high DPI displays
%config InlineBackend.figure_format = 'retina'
```

    Populating the interactive namespace from numpy and matplotlib

# Introduction

As opposed to supervised learning, in unsupervised learning the goal is to infer a model which describes an unknown structure from unlabeled data. We'll consider two classes of unsupervised learning algorithms: clustering and anomaly detection.

We'll also see how unsupervised techniques can be used to reduce the dimensionality of the input feature set.

# 1. Clustering

Clustering is the task of grouping a set of items together in such a way that items in one group (a cluster) are more similar to each other than to those in other groups. There are several types of clustering algorithms:

*   Hierarchical clustering (e.g. Linkage clustering)
*   Centroid-based clustering (e.g. k-means)
*   Distribution-based clustering (e.g. Expectation-Maximization)
*   Density-based clustering (e.g. DBSCAN)

We will look at one of the more simpler approaches: centroid-based clustering.

## K-Means Algorithm

Suppose we have a dataset {$x_1$, $x_2$, $x_3$, ..., $x_N$} consisting of $N$ observations of $D$ dimensional vectors **x** (i.e. $D$ features).

The goal is to partition the dataset into $K$ clusters. Therefore, the number of clusters in our dataset needs to be known a priori.

A cluster is a group of data points whose distances between one another in $D$-dimensional space are small compared to points outside the cluster.

This can be formalized by introducing a $D$-dimensional mean vector $\mu_k$, where $k = 1,2,3,...K$. This represents the center of the cluster.

The K-means clustering algorithm assigns a vector $x_{i,j}$ to the cluster which minimizes the following distortion measure: $J_k = ||x_{i,j} - \mu_k||^2$

The mean vector is then updated by computing the mean intensity value of the considered cluster such that

\begin{equation}
\mu_k = \frac{\sum\limits_i \sum\limits_j r_{i,j,k}x_{i,j}}{\sum\limits_i \sum\limits_j r_{i,j}}
\end{equation}

where

\begin{equation}
r_{i,j,k}=\begin{cases}
    1, & \text{if $k = \arg_k \min(||x_{i,j} - \mu_k||^2)$}\\
    0, & \text{otherwise}.
  \end{cases}
\end{equation}

(see also Lecture 13.2 — Clustering | KMeans Algorithm — [ Machine Learning | Andrew Ng ] https://www.youtube.com/watch?v=hDmNF9JG3lo for more details)

## Dataset

Let's generate a two-dimensional dataset that we can easily visualize.

Note that the first 7 points are similar to each other (therefore they would belong to cluster 0) while the next 7 points are also similar to each other (they would belong to cluster 1)

```python
# Generate a dataset
data = np.array([[2,2], [2,3], [3,2], [3,3], [5,4], [1,1], [6,4], [15,12], [14,14], [18,14], [25,30], [21,17], [18,16], [12,18]])

# Inspect the shape
data.shape
```

    (14, 2)

```python
# Plot the data
## YOUR CODE
```

## Clustering by distances

**Task** Complete the implementation of the functions `distance`, `predict_cluster` and `predict_clusters`, which are used within the k-means algorithm.

```python
# Define a distance function as the Euclidean distance
def distance(pt1, pt2):

    ## YOUR CODE

    return ## YOUR CODE

# Test
distance(data[0], data[1])
```

    1.0

```python
# Return a cluster index of a single data point, given a list of cluster centers
def predict_cluster(pt1, centers):

    ## YOUR CODE

    return label

# Test. This should return 0
predict_cluster(data[0], [ np.array([1,2]), np.array([20,20]) ] )
```

    0

```python
# Return a list of cluster indexes (cluster_labels)
def predict_clusters(data, centers):
    ## YOUR CODE
    return cluster_labels

# Test. This should return an array of labels: array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1])
predict_clusters(data, [ np.array([1,2]), np.array([20, 20]) ] )
```

    array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1])

## Visualization

The code below assigns colors to each cluster: Cluster 0 will be blue, Cluster 1 will be red, etc.

```python
def plot_clusters(data, centers):

    # Get the cluster indices
    cluster_labels = predict_clusters(data, centers)

    fig, ax = plt.subplots()
    colors = ['blue', 'red', 'green', 'orange']
    for ix, center in enumerate(centers):
        cdata = data[cluster_labels == ix]
        ax.scatter(cdata[:,0], cdata[:,1], color=colors[ix], label=f'Cluster {ix}')
        ax.scatter(center[0], center[1], marker='x', color=colors[ix], label=f'Cluster center {ix}')
    ax.legend()

plot_clusters(data,  [ np.array([4,2]), np.array([20, 20]) ] )
```

## Algorithm

**Task**

Complete the implementation of the function below, which performs one iteration of k-means, i.e.:

- Finding the center for each data point
- Calculating the mean of the data points associated with each cluster.

```python
# This will perform one iteration of the algorithm
def calculate_new_centers(data, centers):
    """
    Args:
        data (array-like): points with shape (N points, D features)
        centers (list): initial cluster centers

    Returns:
        new_centers (list): new centers

    """

    ## YOUR CODE

    return new_centers

calculate_new_centers(data, data[2:4])
```

    [array([2.        , 1.66666667]), array([12.63636364, 12.27272727])]

We'll iterate the algorithm 4 times

```python
# Randomly initialize the means to some value (x1, x2) within our two-dimensional space
means0 = np.array([[np.random.random()*30,np.random.random()*30], [np.random.random()*30,np.random.random()*30]])

for it in range(4):
    # Plot the current solution
    plot_clusters(data, means0)
    plt.title(f'Iteration: {it}')

    # Next solution
    means0 = calculate_new_centers(data, means0)
```

```python
# Final result
means0
```

    [array([3.14285714, 2.71428571]), array([17.57142857, 17.28571429])]

```python
# Final cluster assignments
predict_clusters(data, means0)
```

    array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1])

**Task**

Repeat the above iterations a few times. What are potential problems with the algorithm?

## K-Means with sklearn

In practice we often use algorithms from well-developed packages.

```python
# We now import the KMeans module in sklearn
from sklearn.cluster import KMeans
```

```python
# We can use a one-liner to fit our data using 2 clusters
kmeans = KMeans(n_clusters=2, random_state=0).fit(data)
```

```python
# The final means of each cluster
kmeans.cluster_centers_
```

    array([[17.57142857, 17.28571429],
           [ 3.14285714,  2.71428571]])

```python
# Label per data point indicating the final cluster it has been assigned to:
kmeans.labels_
```

    array([1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], dtype=int32)

```python
# We can predict new values
kmeans.predict([[0, 0], [17, 15]] )
```

    array([1, 0], dtype=int32)

```python
# Use the same plotting function as before
plot_clusters(data, kmeans.cluster_centers_)
```

**Task**

Try K-means on a smoother dataset.

```python
data2 = np.random.multivariate_normal([5, 5], np.array([[2, 1], [1,2]]), size=1000)
plt.scatter(data2[:,0], data2[:,1])
```

```python
kmeans2 = ## YOUR CODE
```

```python
# Use the same plotting function as before
plot_clusters(data2, kmeans2.cluster_centers_)
```

# 2. Anomaly Detection

Anomaly detection is the identification of outliers, which do not resemble the bulk of the dataset. Several techniques are used in anomaly detection, including:

*   k Nearest Neighbours
*   DBSCAN
*   Isolation trees

## k Nearest Neighbours Distance

The k Nearest Neighbours distance anomaly detection algorithm assigns a score to each point in the dataset based on its distance from its kth nearest neighbour.

The larger the distance score, the more likely the point is an anomaly.

```python
# Randomly generate points
data = np.random.randint(10, size=(15,2)) + 10

# Add outliers
data = np.vstack((data, np.array([5,5])))

# Inspect the shape
data.shape
```

    (16, 2)

```python
# Visualize the data
plt.scatter(data[:,0], data[:,1])
```

**Task**

Complete the implementation of the function `compute_distances` below.

```python
# Compute distances of each point wrt all other points

def compute_distances(data):
    """"
    Input: array of shape (N points, D features)
    Output: array of shape (N,N)
    """

    ## YOUR CODE
    return distances

distances = compute_distances(data)

#visalize the distances
plt.matshow(distances)
plt.colorbar()
```

```python
# For each point, sort the distances and check which is the (k + 1)th nearest neighbour

def generate_knn_distances(k, distances):

    knn_distances = []

    for i in range(len(data)):

        sorted_distances = np.sort(distances[i,:])

        knn_distances.append(sorted_distances[k+1])

    return np.array(knn_distances)
```

```python
# Plot scatter plot with heat map

knn_distances = generate_knn_distances(3, distances)

plt.scatter(data[:,0], data[:,1], c=knn_distances, edgecolors='black')
plt.colorbar(label='Outlier Score')
```

### Try on some other data

```python
# Randomly generate points

data = np.random.randint(10, size=(15,2)) + 10

# Add outliers

data = np.vstack((data, np.array([[5,5], [6,6]])))

distances = compute_distances(data)

knn_distances = generate_knn_distances(3, distances)

plt.scatter(data[:,0], data[:,1], c=knn_distances, edgecolors='black')
plt.colorbar(label='Outlier Score')
```

```python
# Randomly generate points

data_1 = np.random.randint(10, size=(15,2)) + 10

data_2 = np.random.randint(10, size=(15,2)) + 20

# Add outliers

data = np.vstack((data_1, data_2))

data = np.vstack((data, np.array([[5,5], [6,6]])))

distances = compute_distances(data)

knn_distances = generate_knn_distances(3, distances)

plt.scatter(data[:,0], data[:,1], c=knn_distances, edgecolors='black')
plt.colorbar(label='Outlier Score')
```

# 3. Feature dimensionality reduction

Principal Components Analysis (PCA) is an unsupervised learning technique which reduces the dimensionality of the features.

```python
import pandas as pd
import seaborn as sns
```

```python
# load make_blobs to simulate data
from sklearn.datasets import make_blobs
```

```python
X1, Y1 = make_blobs(n_features=10,
         n_samples=100,
         centers=4, random_state=4,
         cluster_std=2)

# Inspect the shape
X1.shape, Y1.shape
```

    ((100, 10), (100,))

```python
# Check the labels
Y1
```

    array([2, 1, 0, 3, 0, 2, 1, 2, 3, 3, 1, 3, 0, 2, 0, 0, 3, 2, 3, 2, 1, 0,
           0, 1, 0, 0, 2, 0, 0, 2, 0, 1, 2, 3, 3, 0, 2, 1, 3, 2, 1, 3, 1, 2,
           2, 0, 0, 1, 1, 0, 2, 0, 1, 0, 3, 2, 3, 1, 2, 3, 2, 1, 3, 3, 0, 1,
           3, 3, 1, 2, 3, 2, 3, 0, 3, 3, 1, 1, 3, 1, 3, 1, 1, 0, 2, 1, 0, 2,
           0, 2, 1, 2, 0, 3, 2, 0, 1, 3, 1, 2])

```python
# These are the unique ones
set(Y1)
```

    {0, 1, 2, 3}

# Try to visualize
Visualizing this 10-dimensional space is difficult. Try various 2D and 3D projections

```python
# Plot setup
from matplotlib import colors
cmap = plt.cm.rainbow
norm = colors.BoundaryNorm(np.arange(-.5, 4, 1), cmap.N)
```

**Task:**
   Try various projections: 0,1,2, or 3,6,9

```python
plt.scatter(X1[:,2], X1[:,4], c=Y1, cmap=cmap, norm=norm)
plt.colorbar(ticks=[0,1,2,3,4])
```

# PCA decomposition

Principal Component Analysis (see https://en.wikipedia.org/wiki/Principal_component_analysis) can be useful to project a high-dimensional space onto a lower dimensional space.

```python
from sklearn import decomposition
```

```python
# Get the documentation
# ?decomposition.PCA
```

```python
pca = decomposition.PCA(n_components=4)

pc = pca.fit_transform(X1)

pc.shape
```

    (100, 4)

```python
# Organize the data with pandas

pc_df = pd.DataFrame(data = pc,
        columns = ['PC1', 'PC2','PC3','PC4'])
pc_df['Cluster'] = Y1
pc_df
```

      PC1
      PC2
      PC3
      PC4
      Cluster

      0
      -8.133443
      -0.302138
      9.984672
      0.423878
      2

      1
      18.931381
      0.193838
      0.045462
      -3.894042
      1

      2
      -6.571463
      -14.188490
      -3.120115
      1.278588
      0

      3
      -7.533948
      14.439427
      -5.443487
      3.358252
      3

      4
      -4.591760
      -11.315284
      -9.130630
      -1.420151
      0

      ...
      ...
      ...
      ...
      ...
      ...

      95
      -4.734792
      -15.677706
      -4.013717
      1.638843
      0

      96
      14.474724
      -1.170496
      1.650520
      -6.619600
      1

      97
      -6.305432
      13.098273
      -3.843662
      3.060215
      3

      98
      22.169496
      -3.863984
      1.128840
      -0.480688
      1

      99
      -8.071378
      -1.455127
      9.638617
      -2.078444
      2

100 rows × 5 columns

```python
pca.explained_variance_ratio_
```

    array([0.41594854, 0.3391866 , 0.1600729 , 0.02016822])

```python
df = pd.DataFrame({'var':pca.explained_variance_ratio_,
             'PC':['PC1','PC2','PC3','PC4']})
sns.barplot(x='PC',y="var",
           data=df, color="c");
```

```python
sns.lmplot( x="PC1", y="PC2",
  data=pc_df,
  fit_reg=False,
  hue='Cluster', # color by cluster
  legend=True,
  scatter_kws={"s": 80}) # specify the point size
```

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(pc_df['PC1'], pc_df['PC2'], pc_df['PC3'], c=pc_df['Cluster'])
#ax.legend()
ax.set_xlabel("PC1")
ax.set_ylabel("PC2")
ax.set_zlabel("PC3")
```

# Homework

**Homework 8-1**

Perform K-means clustering on both the original and the PCA-reduced blob dataset. Do you get the same results?

Note: the specific cluster index assigned by K-means to a given cluster (e.g. `0` or `1` or `2`) may differ (even between repeated calls of k-means on the same dataset), but the value of these specific indices do not carry any meaning: the important point is whether K-means finds the same separate clusters. (Irrespective of how they are labeled.)

```python
## YOUR CODE
```

### PCA-reduced dataset

**Homework 8-2**

In K-means, we need to know the number of clusters in advance. Try using the DBSCAN algorithm instead, once again on both the original and the PCA-reduced dataset, and see if you get the same results.

(Note: DBSCAN requires the eps parameter to be set, i.e. the maximum distance between two samples for them to be considered as in the same neighborhood)

```python
## YOUR CODE
```


# Integración numérica

**Integrar, al contrario de derivar, no es una operación *algorítmica*,** es decir, no existe una manera de, dada una función, calcular su integral definida en un intervalo.

Debido a esto, integrar analíticamente es mucho más dificil que derivar analíticamente. Más aún, existen funciones cuyas integrales, aunque existan, no se pueden expresar de manera simple.

Para atacar todos estos problemas, en análisis numérico se han desarrollado distintos métodos para calcular la integral de una función.

## Planteamiento del problema

Dada una función $f:[a,b] \to \mathbb{R}$ continua a pedazos (tiene a lo más un número numerable de discontinuidades), quiero calcular $\int_{a}^{b} f(s) ds$, que corresponde al área algebraica entre la función y el eje x

## ¿Cómo calculo una integral numéricamente?

Primero debemos recordar la definición de una integral sobre el intervalo $[a,b]$.

### Particiones regulares
Sea $P = \{x_0=a, x_1, \ldots, x_{n-1}, x_n = b\}$. Decimos que $P$ es una **partición regular** de $[a,b]$ si y solo si, para toda $i$ se cumple que

1. $x_{i} < x_{i+1}$

2.  $x_{i+1} - x_{i} = h = \frac{b-a}{n}$

En este ejemplo, la partición tiene $n+1$ puntos y $n$ subintervalos. La función `range(a,stop=b,length=n+1)` nos construye un rango que corresponde a una partición regular de $[a,b]$ con $n$ subintervalos.

### Sumas de Riemann

Primero la integral se aproxima mediante la suma de las áreas de varios rectángulos cuyas bases están sobre una partición regular del invervalo $[a,b]$

$$
\int_{a}^{b} f(s) ds \approx \sum_{i=1}^{n} f(\xi_i)(x_{i} - x_{i-1}) = h \sum_{i=1}^{n}  f(\xi_i)
$$

con $\xi_i \in [x_{i-1},x_{i}]$. Notemos que la altura del rectángulo, dada por $f(\xi_i)$, es arbitraria. Para tener la integral, simplemente tomamos el límite cuando el número de puntos de la partición tiende a infinito:

$$
\int_{a}^{b} f(s) ds = \lim_{n \to \infty} \sum_{i=1}^{n} f(\xi_i)(x_{i} - x_{i-1}) = \lim_{n \to \infty} h \sum_{i=1}^{n}  f(\xi_i)
$$

con $\xi_i \in [x_{i-1},x_{i}]$

### Sumas importantes

Si **$f$ es continua**, sabemos que en cualquier intervalo cerrado alcanza su máximo y su mínimo. Así, podemos tomar como alturas de los rectángulos los valores máximos y mínimos de la función y así obtener:

#### Suma superior:

$$\xi_i = \text{argmax}_{x\in[x_{i-1},x_i]}{f(x)} \quad , \quad f(\xi_i) = \max_{x \in [x_{i-1},x_i]}{f(x)}$$

#### Suma inferior:

$$\xi_i = \text{argmin}_{x\in[x_{i-1},x_i]}{f(x)} \quad , \quad f(\xi_i) = \min_{x \in [x_{i-1},x_i]}{f(x)}$$

## ¿Podemos implementar una suma superior o inferior en la computadora?

Generalmente **NO**: como ya vimos, encontrar minimos y maximos no es sencillo y eso se tendría que hacer para cada subintervalo de la partición ($n$ veces), lo que haría la integración muy tardada.

## ¿Cómo calculamos entonces la integral?

Ya que no podemos hacer sumas superiores o inferiores, lo más sencillo es tomar $\xi_i$ de manera que sea fácil calcular la integral en la computadora:

Recordamos que $\xi_i \in [x_{i-1},x_i]$

### Primera opción: $\xi_i = x_{i}$ (Integral rectangular hacia adelante)

$$
\int_{a}^{b} f(s) ds \approx h \sum_{i=1}^{n}  f(\xi_i) =  h \sum_{i=1}^{n}  f(x_i)
$$

Tomar el punto $\xi_i = x_{i}$ corresponde a aproximar la integral como se muestra en la siguiente figura

## Preludio computacional: diferencia entre los indices de la partición y de la lista de Julia

Muchas veces, cuando planteamos un problema de manera matemática, la notación del problema puede incluir índices que nosotros también quisiéramos utilizar. Sin embargo, es importante recordar que **los índices en un objeto iterable siempre empiezan en 1**. Para el caso de la partición, se cumple que:

Partición: $xs = \{x_0, x_1, x_2 \ldots, x_{n-1},x_n\}$

Si queremos representar todos los puntos de la partición en un arreglo, debemos notar que tendrán la siguiente correspondencia

$xs[1] = x_0$

$xs[2] = x_1$

.
.
.

$xs[i] = x_{i-1}$

$xs[i+1] = x_{i}$

Es fundamental que dicha correspondencia se respete para no tener problemas entre lo teórico y lo computacional

```julia
function intAdelante(f,a,b,n)
    # hacer la particion regular con n+1 puntos, es decir, n subintervalos
    xs = range(a,stop = b, length = n+1)
    # hacer la suma de Riemann
    integral = 0
    for i in 1:n
        # xs[i+1] es el x_i de la partición
        integral = integral + f(xs[i+1])
    end
    # multiplicamos por h = (b-a)/n
    integral = (b-a)/n * integral
    return integral
end
```

    intAdelante (generic function with 1 method)

Podemos probar nuestra implementación en la función $f(x) = 3x^2$

Analíticamente, sabemos que

$$
\int_0^2 3x^2 dx= x^3 \mid_0^2 = 2^3 - 0^3 = 8
$$

```julia
function prueba(x)
    return 3*x^2
end
```

    prueba (generic function with 1 method)

```julia
# con 10 subintervalos, o rectángulos, la aproximación es mala
println(intAdelante(prueba,0,2,10))
# tomando más mejora
println(intAdelante(prueba,0,2,100))
println(intAdelante(prueba,0,2,1000))
```

    9.24
    8.120399999999998
    8.012004000000006

```julia
using Plots
```

```julia
# seleccionamos ns espaciadas logaritmicamente
# expns guarda los exponentes de las ns
expns = range(1,stop=4,length=30)
println("exponentes: ")
println(collect(expns))
# ns va a guardar mis ns (los distintos numeros de subintervalos)
# todos los elementos de n deben ser enteros, por lo que debemos redondearlos hacia arriba con la función
# ceil (función techo) y luego convertirlos a enteros con la función Int64()
ns = [Int64(ceil(10^k)) for k in expns]
println("numeros de  subintervalos: ")
println(collect(ns))
ys = [abs(intAdelante(prueba,0,2,n)-8) for n in ns]
scatter(ns,ys,title="Error absoluto",label="Integral hacia adelante",xscale=:log10,yscale=:log10)
```

    exponentes:
    [1.0, 1.103448275862069, 1.206896551724138, 1.3103448275862069, 1.4137931034482758, 1.5172413793103448, 1.6206896551724137, 1.7241379310344827, 1.8275862068965518, 1.9310344827586208, 2.0344827586206895, 2.1379310344827585, 2.2413793103448274, 2.3448275862068964, 2.4482758620689653, 2.5517241379310347, 2.6551724137931036, 2.7586206896551726, 2.8620689655172415, 2.9655172413793105, 3.0689655172413794, 3.1724137931034484, 3.2758620689655173, 3.3793103448275863, 3.4827586206896552, 3.586206896551724, 3.689655172413793, 3.793103448275862, 3.896551724137931, 4.0]
    numeros de  subintervalos:
    [10, 13, 17, 21, 26, 33, 42, 53, 68, 86, 109, 138, 175, 222, 281, 357, 453, 574, 728, 924, 1173, 1488, 1888, 2396, 3040, 3857, 4894, 6211, 7881, 10000]

## Ejercicio 1

Resuelve las siguientes dos integrales utilizando la regla de integración hacia adelante:

1. $$\int_{0}^{1} \frac{\arctan{\sqrt{x^2+2}}}{\left(\sqrt{x^2+2}\right)(x^2+1)} \; dx = \frac{5\pi^2}{96}$$

2. $$\int_{0}^{\pi} \ln{\left(1 - 2 e \cos{x}+ e^2 \right) } \; dx = 2\pi$$

$e$ es el número de Euler.

Usando el resultado que se muestra, grafica el error absoluto como función del número de puntos de la partición $n$ y discute cuál es la velocidad de convergencia del error.

## Preludio matemático: integrales como funciones

Aunque sabemos que la integral de una función es un número real, podemos definir una función a partir de una integral de la siguiente forma:

$$
F(x) = \int_{a}^{x} f(s) \; ds
$$

Así, podemos utilizar nuestra función para integrar numéricamente también para construir una función cuya definición esté dada por una integral. Por ejemplo:

$$
\ln{x} = \int_{1}^x \frac{1}{s} \; ds
$$

```julia
# definimos una función auxiliar para nuestro logaritmo integrado numéricamente
function inv(s)
    return 1/s
end
# calculamos el logaritmo como una integral
function lnIntegral(x,n)
    return intAdelante(inv,1,x,n)
end
```

    lnIntegral (generic function with 1 method)

```julia
print("Valor exacto: ")
println(log(exp(1)))
print("valor aproximado: ")
println(lnIntegral(exp(1),100))
```

    Valor exacto: 1.0
    valor aproximado: 0.9945904672391623

```julia
# dominio de la gráfica
xs = range(1,stop=10,length=100)
# subintervalos de la integral
n = 100
ys1 = [log(x) for x in xs]
ys2 = [lnIntegral(x,n) for x in xs]
plot(xs,ys1,label="Logaritmo original",title="Integrales como funciones")
plot!(xs,ys2,label="Definición con integral numérica")
```

## Ejercicio 2
La función $\text{erf}(x)$ se define de la siguiente forma
    \begin{equation}
        \text{erf}(x) = \frac{2}{\sqrt{\pi}} \int_0^x e^{-t^2} dt
    \end{equation}
 Define una función `miErf(x,n)` que regresa el valor la función $\text{erf}(x)$ al calcular la integral utilizando la regla de integración rectangular hacia adelante con $n$ subintervalos.

## Otras opciones para definir integrales mediante sumas de Riemman:

### Segunda opción: $\xi_i = x_{i-1}$ (Integral rectangular hacia atras)

$$
\int_{a}^{b} f(s) ds \approx h \sum_{i=1}^{n}  f(\xi_i) =  h \sum_{i=1}^{n}  f(x_{i-1})
$$

## Ejercicio 3

Implementa una función `intAtrás(f,a,b,n)` regrese el valor la integral $\int_a^b f(s) \; ds$ calculado utilizando la fórmula de la integral rectangular hacia atrás con $n$ subintervalos. Prueba tu función integrando la función  `prueba`.

### Tercera opción: tomar el punto medio $\xi_i = \frac{x_{i-1} + x_{i}}{2}$ (Integral rectangular Centrada o regla rectangular)

$$
\int_{a}^{b} f(s) ds \approx h \sum_{i=1}^{n}  f(\xi_i) =  h \sum_{i=1}^{n}  f\left( \frac{x_{i-1} + x_{i}}{2} \right)
$$

## Ejercicio 4

Implementa una función `intCentrada(f,a,b,n)` regrese el valor la integral $\int_a^b f(s) \; ds$ calculado utilizando la fórmula de la regla centrada con $n$ subintervalos. Prueba tu función integrando la función  `prueba`.

## Ejercicio 5

Repite el ejercicio 1 pero ahora utilizando las reglas de integración del ejercicio 3 y 4. ¿la tasa de convergencia mejora con las nuevas reglas?

## Ejercicio 6

Repite el ejercicio 2 pero ahora utilizando las reglas de integración del ejercicio 3 y 4.

## Aproximación más compleja: regla trapezoidal

Notemos que todas las aproximaciones rectangulares de la integral, por su construcción a partir de sumas de Riemann, aproximaban a la función $f(x)$ en cada intervalo $[x_{i-1},x_{i}]$ como una función constante, es decir, un **polinomio de grado 0**.

Una aproximación más compleja para la función en dicho intervalo sería aproximar a la función no como una constante si no como una recta (o un **polinomio de grado 1**) que pasa por los puntos $(x_{i-1},f(x_{i-1}))$ y $(x_{i},f(x_{i}))$. Así, el area de la integral se aproxima como la siguiente figura

## Ejercicio 7

Deduce la fórmula para el valor de $\int_{x_{i-1}}^{x_{i}} f(s) \; ds$ cuando aproximamos la función como una recta. Es decir, da la fórmula para el área del trapecio en un intervalo $[x_{i-1},x_{i}]$ de la figura anterior. La fórmula solo debe depender de $x_{i-1},f(x_{i-1}), x_{i},f(x_{i})$

**Sugerencia:** Las áreas siempre son valores positivos, por lo que debes de tratar por separado los casos $f(x_{i-1}) \geq f(x_{i})$ y   $f(x_{i-1}) \leq f(x_{i})$.

Aunque trates ambos casos por separado, la fórmula obtenida **debe de ser la misma para ambos**.

### Planteamiento de la regla

Podemos explotar el ejercicio para crear una nueva regla de integración. Recordando que, dada una partición regular de orden $n$, se cumple que

$$
\int_{a}^b f(s) \; ds = \sum_{i=1}^{n} \int_{x_{i-1}}^{x_{i}} f(s) \; ds
$$

Podemos entonces utilizar la fórmula de $\int_{x_{i-1}}^{x_{i}} f(s) \; ds$ que encontramos en el ejercicio anterior para calcular la integral en todo el intervalo $[a,b]$. A este método de cálculo de la integral se le llama la **regla trapezoidal**

## Ejercicio 8

Escribe una función `intTrapezoidal(f,a,b,n)` que regresa el valor de la integral $\int_{a}^b f(s) \; ds$ calculado utilizando la regla trapezoidal. Es decir, que para cada integral $\int_{x_{i-1}}^{x_{i}} f(s) \; ds$ se calcula como el area de un trapecio.

## Ejercicio 9

Repite el ejercicio 1 utilizando ahora la regla de integración trapezoidal. ¿Mejora la tasa de convergencia?

## Aproximación avanzada: regla de Simpson

Una regla todavía más avanzada para integrar consiste en ahora aproximar la función $f(x)$ como un **polinomio de grado 2** en un subintervalo de la partición. Sin embargo, para aproximar dicha función por un polinomio, como vimos en la clase de interpolación, necesitamos conocer tres puntos por los que pasa la función.

Para todo lo que sigue, supondremos que el orden $n$ de la partición tomada para aproximar la integral es un número **par**. Recordando el notebook de interpolación, sabemos que el polinomio de grado 2 $P_i(x)$ que pasa por los puntos $(x_{i-1},f(x_{i-1}))$, $(x_i,f(x_i))$ y $(x_{i+1},f(x_{i+1}))$ esta dado por la ecuación

\begin{align}
P_i(x) &= f(x_{i-1}) \frac{(x-x_{i})(x-x_{i+1})}{(x_{i-1}-x_{i})(x_{i-1}-x_{i+1})} \\
 &+ f(x_{i}) \frac{(x-x_{i-1})(x-x_{i+1})}{(x_{i}-x_{i-1})(x_{i}-x_{i+1})} \\
 &+ f(x_{i+1}) \frac{(x-x_{i-1})(x-x_{i})}{(x_{i+1}-x_{i-1})(x_{i+1}-x_{i})} \\
\end{align}

Así, podemos entonces aproximar la integral en el subintervalo $[x_{i-1},x_{i+1}]$ como

$$
\int_{x_{i-1}}^{x_{i+1}} f(s) \; ds \approx \int_{x_{i-1}}^{x_{i+1}} P_i(s) \;ds
$$

### Ejercicio 10

Demuestra, realizando la integración analíticamente, que

$$
\int_{x_{i-1}}^{x_{i+1}} P_i(s) \; ds = h \left(\frac{f(x_{i-1})}{3} + \frac{4 f(x_{i})}{3} + \frac{f(x_{i+1})}{3} \right)
$$

### Planteamiento de la regla

Podemos ahora, después de demostrar la fórmula del ejercicio 9, utilizar dicha fórmula para implementar una regla de integración, notando que

$$
\int_{a}^b f(s) \; ds = \sum_{i=0}^{(n-2)/2} \int_{x_{2i}}^{x_{2i+2}} f(s) \; ds
$$

A esta regla de integración se le conoce como **regla de Simpson**

## Ejercicio 11

Implementa una función `intSimpson(f,a,b,n)` que regrese el valor de la integral $\int_{a}^b f(s) \; ds $ calculado mediante regla de Simpson.

## Ejercicio 12

Repite el ejercicio 1 ahora utilizando la regla de Simpson y compara con los otros resultados. ¿Qué tanto mejora la tasa de convergencia?

## La visión general

Integrar funciones es un problema que surge todo el tiempo a la hora de hacer cómputo científico. Muchísimos problemas se descomponen, desde el punto de vista teórico, en hacer integrales (electromagnetismo, por ejemplo). Existen, debido a eso, una infinidad de reglas y aproximaciones para hacer integrales de funciones de manera numérica.

El mayor desafío es cuando tenemos ya no integrales en una dimensión, sino integrales en varias dimensiones (a veces más de 20). En esos casos, las aproximaciones aquí mencionadas se pueden volver inútiles.

Las reglas rectangulares no son muy usadas en aplicaciones reales pues convergen a una velocidad muy pequeña. Las regla de Simpson y la trapezoidal, por el contrario, son mucho más usadas.

Existe una manera formal de estimar el error de las reglas de integración mostradas aquí. No quise enseñárselos por que, aunque sea importante, es más largo y complicado que los errores de derivación numérica. Pueden consultar como calcular dichos errores en el libro de Stickler (*Basic Concepts in Computational Physics*)

```julia

```


```python
from sympy import *
```

```python
s=10
```

```python
x = Symbol("x")
```

```python
legpols = []
```

```python
for k in range(0,s+1):
    legpol = 0
    for j in range(0, k+1):
        legpol += (-1)**(j+k) * binomial(k,j) * binomial(j+k,j) * x**j
    legpols.append(sqrt(2*k+1)*legpol)
```

```python
legpols
```

    [1,
     sqrt(3)*(2*x - 1),
     sqrt(5)*(6*x**2 - 6*x + 1),
     sqrt(7)*(20*x**3 - 30*x**2 + 12*x - 1),
     210*x**4 - 420*x**3 + 270*x**2 - 60*x + 3,
     sqrt(11)*(252*x**5 - 630*x**4 + 560*x**3 - 210*x**2 + 30*x - 1),
     sqrt(13)*(924*x**6 - 2772*x**5 + 3150*x**4 - 1680*x**3 + 420*x**2 - 42*x + 1),
     sqrt(15)*(3432*x**7 - 12012*x**6 + 16632*x**5 - 11550*x**4 + 4200*x**3 - 756*x**2 + 56*x - 1),
     sqrt(17)*(12870*x**8 - 51480*x**7 + 84084*x**6 - 72072*x**5 + 34650*x**4 - 9240*x**3 + 1260*x**2 - 72*x + 1),
     sqrt(19)*(48620*x**9 - 218790*x**8 + 411840*x**7 - 420420*x**6 + 252252*x**5 - 90090*x**4 + 18480*x**3 - 1980*x**2 + 90*x - 1),
     sqrt(21)*(184756*x**10 - 923780*x**9 + 1969110*x**8 - 2333760*x**7 + 1681680*x**6 - 756756*x**5 + 210210*x**4 - 34320*x**3 + 2970*x**2 - 110*x + 1)]

```python

```


Chapter 9

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

9.5 Discriminative Classification

\tab In discriminative classification, classification boundaries are modeled without creating class density estimations. Nearest neghbor classifications (9.4) are one example of this type of classification. Recall, for a set of two classes $y \in \{0,1\}$, the discriminant function is $g(x) = p(y=1|x)$. The rule for classification is
$$\hat{y} = \begin{cases} 1 & g(x) \gt 1/2, \\ 0 & \text{otherwise}, \end{cases}$$
which can be used regardless of how we get $g(x)$.

9.5.1 Logistic Regression

While logistic regression can be in binomial or multinomial cases, we will be looking at the binomial case for simplicity. We use the linear model
\begin{align}
p(y=1|x) &= \frac{\exp[\sum_j \theta_j x^j]}{1+\exp[\sum_j \theta_j x^j]},\\
&= p(\boldsymbol{\theta})
\end{align}
and define the logit function as
$$logit(p_i) = log \left(\frac{p_i}{1-p_i}\right) = \displaystyle\sum_j \theta_j x_i^j.$$

Since we only have two classes (y is binary), we can use a Binomial distribution as a model, specifically the Bernoulli distribution [see Eq (3.50) in $\S$ 3.3.3]. This has a likelihood function of
$$L(\beta) = \displaystyle\prod_{i=1}^N p_i(\beta)^{y_i} (1-p(\beta))^{1-y_i}.$$

Logistic regression is related to other linear models, including linear discriminant analysis ($\S$ 9.3.4). In LDA, the assumptions give rise to
\begin{align}
\log\left(\frac{p(y=1|x)}{p(y=0|x)}\right) &= -\frac{1}{2}(\mu_0+\mu_1)^T \Sigma^{-1}(\mu_1-\mu_0) \\
&\quad + \log\left(\frac{\pi_0}{\pi_1}\right) + x^T\Sigma^{-1}(\mu_1-\mu_0) \\
&= \alpha_0 + \alpha^Tx.
\end{align}

In logistic regression, this is the model by assumption:
$$\log\left(\frac{p(y=1|x)}{p(y=0|x)}\right) = \beta_0 + \beta^Tx.$$
The difference is in how the parameters are estimated. Logistic regression chooses parameters to minimize classification errors, not density errors.

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import colors

from sklearn.linear_model import LogisticRegression
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)

N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# perform Classification
classifiers = []
predictions = []
Ncolors = np.arange(1, X.shape[1] + 1)

for nc in Ncolors:
    clf = LogisticRegression(class_weight='auto')
    clf.fit(X_train[:, :nc], y_train)
    y_pred = clf.predict(X_test[:, :nc])

    classifiers.append(clf)
    predictions.append(y_pred)

completeness, contamination = completeness_contamination(predictions, y_test)

print "completeness", completeness
print "contamination", contamination

#------------------------------------------------------------
# Compute the decision boundary
clf = classifiers[1]
xlim = (0.7, 1.35)
ylim = (-0.15, 0.4)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 71),
                     np.linspace(ylim[0], ylim[1], 81))

print clf.intercept_
print clf.raw_coef_

Z = clf.predict_proba(np.c_[yy.ravel(), xx.ravel()])[:, 1]
Z = Z.reshape(xx.shape)

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.binary, zorder=2)
im.set_clim(-0.5, 1)

im = ax.imshow(Z, origin='lower', aspect='auto',
               cmap=plt.cm.binary, zorder=1,
               extent=xlim + ylim)
im.set_clim(0, 2)

ax.contour(xx, yy, Z, [0.5], colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# plot completeness vs Ncolors
ax = fig.add_subplot(222)

ax.plot(Ncolors, completeness, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))
ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

9.2 Support Vector Machines

The idea behind SVM is to find a hyperplane that maximizes the distance between the plane and the closest points of either class. This distance is referred to as the *margin* and data points on the margin are called *support vectors*. We begin with the assumption that the classes $y \in \{-1,1\}$ are linearly seperable.

To find the hyperplane that maximizes the margin one must maximize
$$\max_{\beta_0,\beta}(m)\quad \text{subject to} \quad \frac{1}{\|\beta\|}y_i(\beta_0 + \beta^T x_i) \geq m \quad \forall i.$$
For any $\beta_0$ and $\beta$ that satisfy the above inequality, any positive scaled multiple works as well, so we can set $\|\beta\|=1/m$ and rewrite the problem as minimizing
$$\frac{1}{2}\|\beta\|\quad \text{subject to} \quad y_i(\beta_0 + \beta^T x_i) \geq 1 \quad \forall i.$$

In realistic problems, the assumption that classes are linearly seperable must be relaxed. We add in *slack variables* $\xi_i$ and minimize
$$\frac{1}{2}\|\beta\|\quad \text{subject to} \quad y_i(\beta_0 + \beta^T x_i) \geq 1-\xi_i \quad \forall i.$$
We limit the slack by imposing constraints to effectively bound the number of misclassifications:
$$\xi_i \geq 0 \quad \text{and} \quad \displaystyle\sum_i \xi_i \leq C.$$

SVM optimization is equivalent to minimizing
$$\displaystyle\sum_{i=1}^N \max(0,1-y_ig(x_i)) + \lambda \|\beta\|^2,$$
where $\lambda$ is related to the tuning parameter C.

There is an equivalent way to write this problem using the inner products $\langle x_i,x_{i'}\rangle$ of $x_i$ and $x_{i'}$, which can be replaced with kernel functions $K(x_i,x_{i'})$ to make this method nonlinear [See Eq (9.36), (9.37), (9.38), and (9.42)].

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.svm import SVC
from astroML.decorators import pickle_results
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results

# SVM takes several minutes to run, and is order[N^2]
#  truncating the dataset can be useful for experimentation.
#X = X[::5]
#y = y[::5]

(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)

N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# Fit SVM
Ncolors = np.arange(1, X.shape[1] + 1)

@pickle_results('SVM_rrlyrae.pkl')
def compute_SVM(Ncolors):
    classifiers = []
    predictions = []

    for nc in Ncolors:
        # perform support vector classification
        clf = SVC(kernel='linear', class_weight='auto')
        clf.fit(X_train[:, :nc], y_train)
        y_pred = clf.predict(X_test[:, :nc])

        classifiers.append(clf)
        predictions.append(y_pred)

    return classifiers, predictions

classifiers, predictions = compute_SVM(Ncolors)

completeness, contamination = completeness_contamination(predictions, y_test)

print "completeness", completeness
print "contamination", contamination

#------------------------------------------------------------
# compute the decision boundary
clf = classifiers[1]
w = clf.coef_[0]
a = -w[0] / w[1]
yy = np.linspace(-0.1, 0.4)
xx = a * yy - clf.intercept_[0] / w[1]

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
ax.plot(xx, yy, '-k')
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.binary, zorder=2)
im.set_clim(-0.5, 1)

ax.set_xlim(0.7, 1.35)
ax.set_ylim(-0.15, 0.4)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.svm import SVC
from sklearn import metrics

from astroML.datasets import fetch_rrlyrae_mags
from astroML.decorators import pickle_results
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # re-order the colors for better 1-color results

# SVM takes several minutes to run, and is order[N^2]
#  truncating the dataset can be useful for experimentation.
#X = X[::5]
#y = y[::5]

(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)

N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# Fit Kernel SVM
Ncolors = np.arange(1, X.shape[1] + 1)

@pickle_results('kernelSVM_rrlyrae.pkl')
def compute_SVM(Ncolors):
    classifiers = []
    predictions = []

    for nc in Ncolors:
        # perform support vector classification
        clf = SVC(kernel='rbf', gamma=20.0, class_weight='auto')
        clf.fit(X_train[:, :nc], y_train)
        y_pred = clf.predict(X_test[:, :nc])

        classifiers.append(clf)
        predictions.append(y_pred)

    return classifiers, predictions

classifiers, predictions = compute_SVM(Ncolors)

completeness, contamination = completeness_contamination(predictions, y_test)

print "completeness", completeness
print "contamination", contamination

#------------------------------------------------------------
# compute the decision boundary
clf = classifiers[1]
xlim = (0.7, 1.35)
ylim = (-0.15, 0.4)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 101),
                     np.linspace(ylim[0], ylim[1], 101))

Z = clf.predict(np.c_[yy.ravel(), xx.ravel()])
Z = Z.reshape(xx.shape)

# smooth the boundary
from scipy.ndimage import gaussian_filter
Z = gaussian_filter(Z, 2)

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.binary, zorder=2)
im.set_clim(-0.5, 1)

ax.contour(xx, yy, Z, [0.5], colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination, 'o-k', ms=6)

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

9.7 Decision Trees

Decision trees set up a heirarchical set of decision boundaries to classify data. Each node splits the data points it includes into two subsets, and this branching continues until predetermined stopping criteria are met. Branching decision boundaries are often based on one feature (axis aligned). Terminal (or leaf) nodes record the fraction of points they contain of each class, with the largest deciding which class is associated with that node.

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.tree import DecisionTreeRegressor
from astroML.datasets import fetch_sdss_specgals

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#------------------------------------------------------------
# Fetch data and prepare it for the computation
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 separate points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the cross-validation scores for several tree depths
depth = np.arange(1, 21)
rms_test = np.zeros(len(depth))
rms_train = np.zeros(len(depth))
i_best = 0
z_fit_best = None

for i, d in enumerate(depth):
    clf = DecisionTreeRegressor(max_depth=d, random_state=0)
    clf.fit(mag_train, z_train)

    z_fit_train = clf.predict(mag_train)
    z_fit = clf.predict(mag_test)
    rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
    rms_test[i] = np.mean(np.sqrt((z_fit - z_test) ** 2))

    if rms_test[i] <= rms_test[i_best]:
        i_best = i
        z_fit_best = z_fit

best_depth = depth[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# first panel: cross-validation
ax = fig.add_subplot(121)
ax.plot(depth, rms_test, '-k', label='cross-validation')
ax.plot(depth, rms_train, '--k', label='training set')
ax.set_xlabel('depth of tree')
ax.set_ylabel('rms error')
ax.yaxis.set_major_locator(plt.MultipleLocator(0.01))
ax.set_xlim(0, 21)
ax.set_ylim(0.009,  0.04)
ax.legend(loc=1)

# second panel: best-fit results
ax = fig.add_subplot(122)
ax.scatter(z_test, z_fit_best, s=1, lw=0, c='k')
ax.plot([-0.1, 0.4], [-0.1, 0.4], ':k')
ax.text(0.04, 0.96, "depth = %i\nrms = %.3f" % (best_depth, rms_test[i_best]),
        ha='left', va='top', transform=ax.transAxes)
ax.set_xlabel(r'$z_{\rm true}$')
ax.set_ylabel(r'$z_{\rm fit}$')

ax.set_xlim(-0.02, 0.4001)
ax.set_ylim(-0.02, 0.4001)
ax.xaxis.set_major_locator(plt.MultipleLocator(0.1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.1))

plt.show()
```

9.7.1 Defining the Split Criteria

We can consider a simple split criteria based on the entropy of the data defined (in $\S$5.2.2) as
$$E(x)=-\displaystyle\sum_i p_i(x)\ln(p_i(x)).$$
Here $p_i$ is the probability of class $i$ given the training data. WE can define the reduction of entropy due to branching as the information gain (also known as Kullback-Leibler divergence). With a binary split where $i=0$ for points below the decision boundary and $i=1$ for those above it, the information gain is
$$IG(x|x_i)=E(x)-\displaystyle\sum_{i=0}^1\frac{N_i}{N}E(x_i),$$
where you have $N_i$ points $x_i$ in the $i$th class, ans $E(x_i)$ is the entropy associated with that class.

Other commonly used loss functions are the Gini coefficient ($\S4.7.2$) and the misclassification error. The Gini coefficient for a $k$-class sample is
$$G=\displaystyle\sum_i^k p_i(1-p_i).$$
Here $p_i$ is the probability of finding a point with class $i$ within a data set. The misclassification error is
$$MC=1-\displaystyle\max_i(p_i).$$

9.7.2 Building the Tree

Where do you stop? There are several ways to find stopping criteria so as to capture trends while avoiding capturing the noise of the training set. Common options include the following:
    - If a node contains only one class of object
    - If splitting does not supply information gain or reduce misclassifications
    - If a node contains a predefined number of points
Overall tree depth can be determined by using a cross validation set to determine when the complexity of the model starts overfitting data.

Another option is pruning: method where the tree grows until nodes contain a small predefined number of points and then cross validation is used to determine where to remove branching nodes in the tree.

9.7.3 Bagging and Random Forests

Ensemble learning (the idea of taking outputs of multiple models and combining them) gives us two methods of improving our classification. In bagging, the predictive results of a series of bootstrap samples (see $\S4.5$) are averaged. For a sample of $N$ points in a training set, bagging generates $K$ bootstrap samples of equal size to estimate $f_i(x)$. The final bagging estimator is
$$f(x)=\frac{1}{K} \displaystyle\sum_i^K f_i(x).$$

Random forests generate a decision tree for each in a series of bootstrap samples. To generate a forest we need to define the number of trees $n$ to use and the number of features $m$ to decide on a boundary at each node. Features on which to split are randomly decided upon at each node. Random forests address both the overfitting of deep trees and the nonlinear boundaries in data sets that single decision trees do not.

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.ensemble import RandomForestRegressor
from astroML.datasets import fetch_sdss_specgals
from astroML.decorators import pickle_results

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#------------------------------------------------------------
# Fetch and prepare the data
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 distinct points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the results
#  This is a long computation, so we'll save the results to a pickle.
@pickle_results('photoz_forest.pkl')
def compute_photoz_forest(depth):
    rms_test = np.zeros(len(depth))
    rms_train = np.zeros(len(depth))
    i_best = 0
    z_fit_best = None

    for i, d in enumerate(depth):
        clf = RandomForestRegressor(n_estimators=10,
                                    max_depth=d, random_state=0)
        clf.fit(mag_train, z_train)

        z_fit_train = clf.predict(mag_train)
        z_fit = clf.predict(mag_test)
        rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
        rms_test[i] = np.mean(np.sqrt((z_fit - z_test) ** 2))

        if rms_test[i] <= rms_test[i_best]:
            i_best = i
            z_fit_best = z_fit

    return rms_test, rms_train, i_best, z_fit_best

depth = np.arange(1, 21)
rms_test, rms_train, i_best, z_fit_best = compute_photoz_forest(depth)
best_depth = depth[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# left panel: plot cross-validation results
ax = fig.add_subplot(121)
ax.plot(depth, rms_test, '-k', label='cross-validation')
ax.plot(depth, rms_train, '--k', label='training set')
ax.legend(loc=1)

ax.set_xlabel('depth of tree')
ax.set_ylabel('rms error')

ax.set_xlim(0, 21)
ax.set_ylim(0.009,  0.04)
ax.yaxis.set_major_locator(plt.MultipleLocator(0.01))

# right panel: plot best fit
ax = fig.add_subplot(122)
ax.scatter(z_test, z_fit_best, s=1, lw=0, c='k')
ax.plot([-0.1, 0.4], [-0.1, 0.4], ':k')
ax.text(0.03, 0.97, "depth = %i\nrms = %.3f" % (best_depth, rms_test[i_best]),
        ha='left', va='top', transform=ax.transAxes)

ax.set_xlabel(r'$z_{\rm true}$')
ax.set_ylabel(r'$z_{\rm fit}$')

ax.set_xlim(-0.02, 0.4001)
ax.set_ylim(-0.02, 0.4001)
ax.xaxis.set_major_locator(plt.MultipleLocator(0.1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.1))

plt.show()
```

9.7.4 Boosting Classification

Boosting is another ensemble method that iterates over a sample, changing the weights of data points after each iteration in an attempt to fix errors in the previous attempt. If we have a "weak" classifier $h(x)$ we can make a "strong" classifier $f(x)$, such that
$$f(x)=\displaystyle\sum_m^K \theta_mh_m(x),$$
where $m$ indicates the iteration and $\theta_m$ is the weight of that iteration of the classifier.
For a data set of $N$ points with known classifications $y$, we can assign a weight $w_m(x)$ to each point (With the initial weight being uniform $1/N$). After application of the weak classifier, we estimate the classification error as
$$e_m = \displaystyle\sum_{i=1}^N w_m(x_i) \times \begin{cases} 1 & h_m(x_i) \neq y_i, \\ 0 & \text{otherwise.}\end{cases}$$
We then define the weight of that iteration as
$$\theta_m = \frac{1}{2}\log{\left(\frac{1-e_m}{e_m}\right)},$$
and update the weight of each point:
$$w_{m+1}(x_i) = w_m(x_i) \times \begin{cases} e^{-\theta_m} & h_m(x_i) = y_i \\ e^{\theta_m} & h_m(x_i) \neq y_i\end{cases}
= \frac{w_m(x_i)e^{-\theta_m y_i h_m(x_i)}}{\sum_{i=1}^N w_m(x_i) e^{-\theta_m y_i h_m(x_i)}}.$$

Since each iteration depends on the previous iteration, parallelization is not possible here. This can lead to comutation time problems with large data sets. An alternative to adaptive boosting is gradient boosting, which may scale better to larger data.

9.8 Evaluating Classifiers: ROC Curves

Receiver operating characteristic (ROC) curves can be useful for visualizing how well classification methods work. They also conveniently show the trade offs between completeness and efficiency. The graphs can also be true positives vs, false positives, though this may not be the most useful when your data includes a large number of background data points.

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from sklearn.naive_bayes import GaussianNB
from sklearn.lda import LDA
from sklearn.qda import QDA
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from astroML.classification import GMMBayes

from sklearn.metrics import precision_recall_curve, roc_curve

from astroML.utils import split_samples, completeness_contamination
from astroML.datasets import fetch_rrlyrae_combined

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
y = y.astype(int)
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)

#------------------------------------------------------------
# Fit all the models to the training data
def compute_models(*args):
    names = []
    probs = []
    for classifier, kwargs in args:
        print classifier.__name__
        clf = classifier(**kwargs)
        clf.fit(X_train, y_train)
        y_probs = clf.predict_proba(X_test)[:, 1]

        names.append(classifier.__name__)
        probs.append(y_probs)

    return names, probs

names, probs = compute_models((GaussianNB, {}),
                              (LDA, {}),
                              (QDA, {}),
                              (LogisticRegression,
                               dict(class_weight='auto')),
                              (KNeighborsClassifier,
                               dict(n_neighbors=10)),
                              (DecisionTreeClassifier,
                               dict(random_state=0, max_depth=12,
                                    criterion='entropy')),
                              (GMMBayes, dict(n_components=3, min_covar=1E-5,
                                              covariance_type='full')))

#------------------------------------------------------------
# Plot ROC curves and completeness/efficiency
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(left=0.1, right=0.95, bottom=0.15, top=0.9, wspace=0.25)

# ax2 will show roc curves
ax1 = plt.subplot(121)

# ax1 will show completeness/efficiency
ax2 = plt.subplot(122)

labels = dict(GaussianNB='GNB',
              LDA='LDA',
              QDA='QDA',
              KNeighborsClassifier='KNN',
              DecisionTreeClassifier='DT',
              GMMBayes='GMMB',
              LogisticRegression='LR')

thresholds = np.linspace(0, 1, 1001)[:-1]

# iterate through and show results
for name, y_prob in zip(names, probs):
    fpr, tpr, thresh = roc_curve(y_test, y_prob)

    # add (0, 0) as first point
    fpr = np.concatenate([[0], fpr])
    tpr = np.concatenate([[0], tpr])

    ax1.plot(fpr, tpr, label=labels[name])

    comp = np.zeros_like(thresholds)
    cont = np.zeros_like(thresholds)
    for i, t in enumerate(thresholds):
        y_pred = (y_prob >= t)
        comp[i], cont[i] = completeness_contamination(y_pred, y_test)
    ax2.plot(1 - cont, comp, label=labels[name])

ax1.set_xlim(0, 0.04)
ax1.set_ylim(0, 1.02)
ax1.xaxis.set_major_locator(plt.MaxNLocator(5))
ax1.set_xlabel('false positive rate')
ax1.set_ylabel('true positive rate')
ax1.legend(loc=4)

ax2.set_xlabel('efficiency')
ax2.set_ylabel('completeness')
ax2.set_xlim(0, 1.0)
ax2.set_ylim(0.2, 1.02)

plt.show()
```

9.9 Which Classifier Should I Use?

Here's how the different methods perform in different areas.

Accuracy:

In general, it can't be stated what will be most accurate, but there are a few rules of thumb.
    - More parameters -> potentially more accurate
    - Ensemble methods are often more accurate
    - Nonparametric are usually more accurate for large data sets
    - Parametric can be more accurate for small data sets, or high dimension sets

Interpretability:

Parametric models are often the methods that will be interpretable, though some parametric models (KNN, Decision trees) are still interpretable.

Scalability:

Naive Bayes (and variants) is by far the easiest to compute, and logistic regression and linear support vector machines are next on the list. Scalability usually means that the method gives a simpler model for classification.

Simplicity:

Naive Beyes takes the cake here. Other simple methods include logistic regression, decision trees, and K nearest neighbors.

*Table 9.1 in the book has a nice comparison of the classification methods*.


### Python Library

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style="darkgrid")
#from scipy.optimize import curve_fit
from lmfit import Model
from sympy import symbols, diff, exp, sqrt, lambdify
```

### Definitions of functions which will be used for fitting

#### Bi-exponential
$I(t) = I_{0,a}(1-\exp(-t/T_{1,a})) + I_{0,b}(1-\exp(-t/T_{1,b}))$

#### Mono-exponential
$I(t) = I_{0}(1-\exp(-t/T_{1}))$

#### Stretched exponential
$I(t) = I_{0}(1-\exp(-t/T_{1})^\beta)$

```python
def twoexpbuild(x, ampl1, tau1, tau2):
    exponential = ampl1*(1-np.exp(-x/tau1))+(1-ampl1)*(1-np.exp(-x/tau2))
    return exponential

def monoexpbuild(x, tau):
    exponential = 1-np.exp(-x/tau)
    return exponential

def stretchedexp(x, T1, beta):
    exponential = (1-np.exp(-np.power((x/T1),beta)))
    return exponential
```

### Calculation of the optimum build up time by finding the minimum in:

$|\frac{dI(t)}{dt}|$

```python
def optimumrecycledelaytwoexp(x, ampl1, tau1, tau2):
    t, b, a, Ta, Tb = symbols ('t b a Ta Tb')
    b = 1-a
    s=a*(1-exp(-t/Ta))/sqrt(t)+b*(1-exp(-t/Tb))/sqrt(t)
    k=diff(s,t)
    y=k.subs({Ta:tau1,Tb:tau2,a:ampl1})
    lam_x = lambdify(t, y, modules=['numpy'])
    x_vals = np.linspace(1, 40, 100000)
    y_vals = lam_x(x_vals)
    return x_vals[np.argmin(abs(y_vals))]

def optimumrecycledelayoneexp(x, tau1):
    t, Ta = symbols ('t Ta')
    s=(1-exp(-t/Ta))/sqrt(t)
    k=diff(s,t)
    y=k.subs({Ta:tau1})
    lam_x = lambdify(t, y, modules=['numpy'])
    x_vals = np.linspace(1, 40, 100000)
    y_vals = lam_x(x_vals)
    return x_vals[np.argmin(abs(y_vals))]

def optimumrecycledelaystrexp(x, tau1, beta):
    t, b, Ta = symbols ('t b Ta')
    s=(1-exp(-np.power((t/Ta),b)))/sqrt(t)
    k=diff(s,t)
    y=k.subs({Ta:tau1,b:beta})
    lam_x = lambdify(t, y, modules=['numpy'])
    x_vals = np.linspace(1, 40, 100000)
    y_vals = lam_x(x_vals)
    return x_vals[np.argmin(abs(y_vals))]
```

### Read the processed Bruker data in Pandas dataframe

```python
colspecs=[(0, 10),(10, 21), (22, 35), (36, 48), (48, 61)]

df=pd.read_fwf('D:/OneDrive/Academics/Data/2022-01-CP-1H-13C-PAVLOT-Indomethacin/1/pdata/1/ct1t2.txt', skiprows=21, skipfooter=3, colspecs=colspecs, names=['Point','Tau','Expt','Calc','Difference'])

p=0

int_expt=df['Expt']
time=df['Tau']
int_calc=df['Calc']

time_array=np.zeros((time.size,1))
for i in time:
    if i[-1] in 'm':
        time_array[p]=float(i[:-1])*0.001
    else:
        time_array[p]=float(i[:-1])*1
    p+=1
```

## Ask user for the method to fit the data.

```python
#plt.plot(time_array,int_calc, label='Norm. Topspin Calc. Data')

ydata=int_expt.to_numpy()
xdata=time_array

choice=int(input("1 for monoexp, 2 for biexp, 3 for stretched: "))
```

    1 for monoexp, 2 for biexp, 3 for stretched:  3

```python
if choice == 1:
    exp_build=Model(monoexpbuild)
    exp_build.set_param_hint('tau', value=20, min=0.001, max=256.0)
    result=exp_build.fit(ydata.flatten(),x=xdata.flatten(),method='nelder')
    #plt.plot(time_array,result.best_fit,'--')
    t1=result.best_values.get("tau")
    optimumd1=optimumrecycledelayoneexp(xdata.flatten(), t1)
    #plt.plot(time_array,result.best_fit,'--')
    print('T1 is '+ str(round(t1,2))+ 's')
    print('The optimum recycle delay should be: '+ str(round(optimumd1,2))+'s')

elif choice == 2:
    exp_build=Model(twoexpbuild)
    exp_build.set_param_hint('ampl1', value=0.5, min=0.01, max=1)
    exp_build.set_param_hint('tau1', value=0.3, min=0.001, max=256.0)
    exp_build.set_param_hint('tau2', value=100, min=10, max=256.0)
    result=exp_build.fit(ydata.flatten(),x=xdata.flatten(),method='nelder')

    a=result.best_values.get("ampl1")
    b=1-a
    t1a=result.best_values.get("tau1")
    t1b=result.best_values.get("tau2")

    optimumd1=optimumrecycledelaytwoexp(xdata.flatten(), a, t1a, t1b)
    #plt.plot(time_array,result.best_fit,'--')
    print('Component 1 ' + str(round(a,2)) +'\n'+'Component 2 ' + str(round(b,2))+ '\n'+ 'T1 of 1 '+str(round(t1a,2))+ '\n'+ 'T1 of 2 '+str(round(t1b,2)))
    print('The optimum recycle delay should be: '+ str(round(optimumd1,2))+'s')
elif choice == 3:
    exp_build=Model(stretchedexp)
    #exp_build.set_param_hint('a1', value=1, min=0.01, max=1)
    exp_build.set_param_hint('T1', value=10, min=0.001, max=256.0)
    exp_build.set_param_hint('beta', value=0.5, min=0, max=1)
    result=exp_build.fit(ydata.flatten(),x=xdata.flatten(),method='nelder')

    #a=result.best_values.get("a1")
    t1=result.best_values.get("T1")
    beta=result.best_values.get("beta")
    optimumd1=optimumrecycledelaystrexp(xdata.flatten(), t1, beta)
    #plt.plot(time_array,result.best_fit,'--')
    print('T1 is '+str(round(t1,2)) + '\n' + 'Stretching factor ' + str(round(beta,2)))
    print('The optimum recycle delay should be: '+ str(round(optimumd1,2))+'s')
else:
    print('Invalid method')
```

    T1 is 7.81
    Stretching factor 0.79
    The optimum recycle delay should be: 6.31s

## Plot the data and best fit

```python
plt.plot(time_array,int_expt)
plt.plot(time_array,result.best_fit,'--')
```


# Descripción del modelo

## Modelo Markoviano SEAIHRF en tiempo discreto para el esparcimiento del COVID-19

El espárcimiento de epidemias ha sido exitosamente explicado con modelos matemáticos que consideran las interacciones en una población entre indiviudos suceptibles, infectados y removidos (ver, por ejemplo, [[1]](https://dx.doi.org/10.1017%2FS0950268810002165), [[2]](https://doi.org/10.1111/j.1541-0420.2006.00609.x), [[3]](https://doi.org/10.1038/nature03063) ).

Para nuestras proyecciones seguimos el trabajo de Arenas y sus colaboradores [[4]](https://covid-19-risk.github.io/map/model.pdf), que propusieron una aproximación microscópica Markoviana a tiempo discreto para modelar el esparcimiento de epidemias. En su modelo, cada paso temporal representa un día, y clasifican a una población de $N$ individuos en las siguientes categorías: población susceptible ($S$), expuesta al virus ($E$), asintomática ($A$), infectada ($I$), hospitalizada a UCI ($H$), recuperada ($R$), y fallecida ($F$), donde $S + E + A + I + H + R + F = N$ en todo momento. Adicionalmente, subdividen a la población en 1) el grupo de edad al que pertenecen --específicamente jóvenes (<25), adultos (25-65), y viejos (65+)-- y 2) la región en donde viven, donde cada región está acoplada con las demás usando los patrones de movilidad entre ellas. Presentamos una breve explicación de cada una de las categorías en una lista al final del texto.

Nuestro modelo es una simplificación del de Arenas ya que agregamos a los grupos de edad en una sola categoría y, aunque sí tomamos regiones separadas --en nuestro caso estados de la república--, la dinámica de cada estado es independiente de las demás. Recalcamos que, aunque [nuestro equipo](https://mexicovid19.github.io/Mexico/about.html) está conformado por físicos con experiencia en redes complejas, no somos epidemiólogos, y éste o cualquier otro modelo podría mejorar con la opinión sensible de un experto en el tema.  Tomamos como referencia el modelo de Arenas dado la precisión que tuvo para explicar el espárcimiento del virus en las diferentes regiones autónomas de España.

Para cada estado $i$, considere una población de $N_i$ personas. Definimos la dinámica de esparcimiento de la epidemia en $i$ como

\begin{align}
  \begin{aligned}
    S^i(t+1) &= S^i(t)( 1 - \Pi^i(t) ) \\
    E^i(t+1) &= S^i(t) \Pi^i(t) + (1 - \eta) E^i(t) \\
    A^i(t+1) &= \eta E^i(t) + (1 - \alpha) A^i(t) \\
    I^i(t+1) &= \alpha A^i(t) + ( 1 - (\gamma_I + \mu_I + \nu) ) I^i(t) \\
    H^i(t+1) &= \nu I^i(t) + ( 1 - (\gamma_H + \mu_H) ) H^i(t) \\
    R^i(t+1) &= \gamma_I I^i(t) + \gamma_H H^i(t) + R^i(t) \\
    F^i(t+1) &= \mu_I I^i(t) + \mu_H H^i(t) + F^i(t),
  \end{aligned}
  \label{eq:seaihrd}
\end{align}

cuya formulación se explica a continuación. En un día, $t$, la población susceptible se infecta con probabilidad $\Pi^i(t)$, quedando expuesta al virus. Los individuos que quedan expuestos incuban el virus y, en un promedio de $\eta^{-1}$ días, se consideran como asintomáticos que, en un promedio de $\alpha^{-1}$ días, presentan síntomas y se consideran infectados. La diferencia entre un individuo expuesto, uno asintomático y uno infectado es que los primeros no presentan síntomas ni son contagiosos, los segundos tampoco presentan síntomas pero sí son contagiosos y los últimos ya presentan síntomas notables. Los individuos infectados pueden recuperarse naturalmente, a una tasa $\gamma_I$, ir al hospital a una tasa $\nu$, o, en el peor de los casos, fallecer sin ser tratados, a una tasa $\mu_I$. De los pacientes hospitalizados, unos se recuperaran, a una tasa $\gamma_H$, y otros fallecen, a una tasa $\mu_H$. Dichas tasas consideran 1) el tiempo de transición entre una categoría y otra, y 2) la probabilidad de un individuo de cambiar de categoría.

Cuantificamos la probabilidad de que un individuo quede expuesto al virus de la siguiente manera

\begin{equation}
    \Pi^i(t) = 1 - (1-\beta)^{ \langle k \rangle \left( \frac{ I^i(t) + A^i(t) }{ N^i } \right) },
    \label{eq:prob_desease}
\end{equation}

donde $\Pi^i(t)$ depende de la proporción de gente contagiosa en el día $t$. Elaborando mejor, tome $\beta$ como la probabilidad que un individuo suceptible se infecte al ponerse en contacto con uno contagioso (ya sea asintomático o infectado) y tome $\langle k \rangle$ como el número de contactos promedio que alguien tiene durante el día. Asumiendo que la población está bien mezclada y que cada contacto es independiente, el último término en la ecuación (\ref{eq:prob_desease}) nos da la probabilidad de que alguien **no** se contagie el día $t$.

Ajustamos los parámetros de nuestro modelo con los presentados en [[4]](https://covid-19-risk.github.io/map/model.pdf) y las referencias ahí expuestas. La información sobre la estructura poblacional de México fue tomada de los datos oficiales del INEGI [[5]](https://www.inegi.org.mx/). Para estimar la infectividad $\beta$ en nuestro modelo, tomamos el número básico de reproducción, $R_0$, y notamos que, simplificando el apéndice en [[4]](https://covid-19-risk.github.io/map/model.pdf), se obtiene que $R_0 = \langle k \rangle \log \big( (1-\beta)^{\alpha^{-1} + \eta^{-1}}\big)$. La incertidumbre de nuestro modelo proviene de la incertidumbre de $R_0$. Tomamos el rango de $R_0$ como el intervalo de confianza de $95 \%$ reportado en [[6]](https://www.nejm.org/doi/full/10.1056/NEJMoa2001316), lo que nos permite hacer un cono de proyecciones del número de casos totales.

Con el modelo (\ref{eq:seaihrd}), podemos calcular la evolución nacional del virus sumando para cada estado. Esto es,

\begin{aligned}
  m(t) &= \sum_{i=1}^{32} m^i(t), \\
  m &\in \big\{ S, E, A, I, H, R, D \big\}.
\end{aligned}

Nuestras proyecciones presentan el número total de casos registrados de COVID-19 en México, es decir

$$\text{Casos Totales}(t) = I(t) + H(R) + R(t) + D(t).$$

## Modelo con restricciones de movilidad

La epidemia que estamos viviendo se está esparciendo rápidamente en todo el mundo. Por eso, es importante tomar serias medidas para `#aplanarlacurva` y hacer que haya menos gente infectada al mismo tiempo. En concreto, una medida efectiva se da cuando el gobierno declara contención a nivel nacional, como se estableció por el gobierno mexicano el 23 de marzo de 2020 cuando inició de la [Jornada Nacional de Sana Distancia](https://www.gob.mx/cms/uploads/attachment/file/541687/Jornada_Nacional_de_Sana_Distancia.pdf).

Llamamos $t_c$ al día cuando se hace efectiva dicha contención. A partir de $t_c$, cada individuo disminuye, en promedio, su número de contactos diarios. Sin embargo, estos individuos volverán a sus hogares, donde conviven con la gente que vive ahí. Sea $\kappa_0$ el porcentaje de gente que se queda en casa a partir de la contención, y sea $\sigma$ el número promedio de habitantes por casa. El número promedio de contactos $\langle k \rangle$ disminuye a partir de $t_c$ de la siguiente manera

\begin{equation}
    \langle k \rangle \to \langle k \rangle(t) = (1 - \kappa_0 \Theta( t - t_c ) )\langle k \rangle + \kappa_0(\sigma - 1)\Theta( t - t_c ),
    \label{eq:k_containtment}
\end{equation}

donde $\Theta(x)$ es la función de Heaviside, donde $\Theta(x) = 1$ si $x \geq 0$ y $\Theta(x) = 0$ si $x < 0$. Viendo el lado derecho de la ecuación (\ref{eq:k_containtment}), su primer término representa el número de contactos que un individuo tiene fuera de casa, mientras que el segundo representa los contactos que tiene dentro de ella.

De las familias que se autoaislan, hay un porcentaje de ellas que no tienen ningún infectado y, por tanto, no serán suceptibles al virus mientras sigan en cuarentena. La cantidad de individuos que viven en el estado $i$ que dejan de ser suceptibles por contención es

\begin{equation}
    C^i(t_c) = \left( S^i(t_c) + R^i(t_c) \right)^\sigma.
    \label{eq:contained}
\end{equation}

Quitamos a la población definida en (\ref{eq:contained}) de la categoría de susceptibles y los derivamos a una nueva: los individuos contenidos ($C$). Así, podemos redefinir al modelo (\ref{eq:seaihrd}) tomando en cuenta la contención del gobierno. Las ecuaciones resultantes son

\begin{align}
  \begin{aligned}
    S^i(t+1)  &= S^i(t)( 1 - \Pi^i(t) )( 1 - \delta_{t,t_c} \kappa_0 C^i(t_c) ) \\
    E^i(t+1)  &= S^i(t) \Pi^i(t) ( 1 - \delta_{t,t_c} \kappa_0 C^i(t_c) ) + (1 - \eta) E^i(t) \\
    C^i(t+1) &= \Theta( t - t_c ) \kappa_0 C^i(t_c) S^i(t_c) \\
    A^i(t+1) &= \eta E^i(t) + (1 - \alpha) A^i(t) \\
    I^i(t+1) &= \alpha A^i(t) + ( 1 - (\gamma_I + \mu_I + \nu) ) I^i(t) \\
    H^i(t+1) &= \nu I^i(t) + ( 1 - (\gamma_H + \mu_H) ) H^(t) \\
    R^i(t+1) &= \gamma_I I^i(t) + \gamma_H H^i(t) + R^i(t) \\
    F^i(t+1) &= \mu_I I^i(t) + \mu_H H^i(t) + F^i(t),
  \end{aligned}
  \label{eq:seaihrd_containtment}
\end{align}

donde $\delta_{t,t_c} = 1$ si $t = t_c$ y  $\delta_{t,t_c} = 0$ si $t \neq t_c$ es la delta de Kronecker.

<hr>

Descripción de cada categoría en la que se divide la población
- **Susceptibles** - Las personas que no han contraído el virus, pero es posible que se contagien al entrar en contacto con algún portador.
- **Expuestos** - Las personas que sí han contraído el virus pero no son contagiosas.
- **Asintomáticos** - Las personas que sí han contraído el virus, sí son contagiosas, pero no presentan síntomas.
- **Infectados** - Las personas que sí presentan síntomas y sí son contagiosas. Lo más probable es que se recuperen pronto en casa.
- **Hospitalizados** - presentan síntomas graves y requieren de cuidados intensivos.
- **Contenidos** - Las personas sanas que se aislaron en casa a partir de la instrucción del gobierno.
- **Recuperados** - Las personas que se recuperan de coronavirus, ya sea en casa o en el hospital.
- **Fallecidos** - Las personas que han fallecido por coronavirus después de estar en cuidados intensivos.

# Referencias

1. Metcalf, C. J. E., Bjørnstad, O. N., Ferrari, M. J., Klepac, P., Bharti, N., **Lopez-Gatell, H.**, & Grenfell, B. T. (2011). The epidemiology of rubella in Mexico: seasonality, stochasticity and regional variation. Epidemiology & Infection, 139(7), 1029-1038.

1. Lekone, P. E., & Finkenstädt, B. F. (2006). Statistical inference in a stochastic epidemic SEIR model with control intervention: Ebola as a case study. Biometrics, 62(4), 1170-1177.

1. Mills, C. E., Robins, J. M., & Lipsitch, M. (2004). Transmissibility of 1918 pandemic influenza. Nature, 432(7019), 904-906.

1. Arenas, Alex, et al. "A mathematical model for the spatiotemporal epidemic spreading of COVID19." medRxiv (2020).

1. Instituto Nacional de Estadística y Geografía (INEGI) https://www.inegi.org.mx/

1. Li, Q., Guan, X., Wu, P., Wang, X., Zhou, L., Tong, Y., ... & Xing, X. (2020). Early transmission dynamics in Wuhan, China, of novel coronavirus–infected pneumonia. New England Journal of Medicine.