# Modelos con variables latentes y repaso del algoritmo K-Means

> **¿Qué significa variable latente?**

> Para responder a esta pregunta, nos remontamos hacia la raiz *latina* etimológica de la palabra latente. Esta palabra viene la palabra en Latín **latens** que significa escondido u oculto.

> En el contexto del modelado probabilístico nos referimos con variables latentes a variables que nunca observamos, pero que (inferimos) están ahí.

## 1. Variables latentes

### ¿Porqué consideramos variables latentes?

Hay diversas razones por las que permitirnos incluir variables latentes en nuestros modelos cobra muchísima importancia. Algunas de ellas:

1. **No porque no observemos una variable significa que no exista.**

2. **Muchas veces nos permiten conseguir modelos más simples.**

**Ejemplo:**

Una empresa acaba de abrir una posición en el equipo de ciencia de datos. En este sentido, el departamento de RH está interesado en entrevistar a varios candidatos para encontrar a alguien idóneo para la posición.

Para ello, ya se tiene un tabulador que involucra varias variables:
- Grado académico.
- Promedio de calificaciones del último grado académico.
- Entrevista telefónica.
- Entrevista en vivo.

Sin embargo, la entrevista en vivo es un evento que puede llegar a involucrar muchos recursos económicos, y por experiencia, hay varios candidatos que se pueden descartar con solo el conocimiento de las otras variables. La idea es desarrollar un modelo usando datos históricos del departamento de RH:

| Candidato | Grado | Promedio | E. Telefónica | E. Vivo |
| --------- | ----- | -------- | ------------- | ------- |
| 1         | Lic   | 8.4      | 7             | 5       |
| 2         | Maes  | 8.0      | 7             | 6       |
| 3         | Lic   | 9.5      | 8             | 9       |
| 4         | Doc   | 8.9      | 9             | 10      |

Si intentamos establecer un modelo que relacione estas variables, después de revisarlo un poco, llegaríamos a que todas estas variables están relacionadas entre sí, obteniendo un modelo completamente conectado,

con lo que no tendríamos ni una sola pista de la estructura del modelo, y tendríamos que definir la probabilidad conjunta sobre todas las variables (número exponencial de parámetros).

*Alternativa 1*: considerar un modelo estructurado del tipo

$$
p(x_1, x_2, x_3, x_4) = \frac{\exp\{-w^T x\}}{Z},
$$

con lo que solo tendríamos 5 parámetros $w_0, w_1, w_2, w_3, w_4$. Sin embargo, $Z$ es una constante de normalización que involucra una suma sobre todos los posibles valores de las cuatro variables aleatorias.

*Alternativa 2*: considerar una variable latente de Inteligencia

con lo que el modelo sería:

$$
p(x_1, x_2, x_3, x_4) = \sum_{I} p(x_1, x_2, x_3, x_4 | I) p(I) = \sum_{I} p(x_1 | I)p(x_2 | I)p(x_3 | I)p(x_4 | I) p(I)
$$

Con esto reducimos notablemente la complejidad del modelo.

3. **Aplicaciones prácticas -> Clustering -> Segmentación de clientes, Motores de búsqueda, Sistemas de recomendación, ...**

En aplicaciones clustering pretendemos descubrir segmentaciones en los datos. Esta segmentación la podemos entender como una variable latente.

## 2. Clustering

Probablemente ya hayan escuchado hablar de clustering. Como tal, es de las aplicaciones más importantes en aprendizaje **no supervisado**, y seguramente en un proyecto de análisis de datos no va a pasar más de un mes para cuando necesiten utilizar este tipo de técnicas. Así que vamos a estudiar un par de ellas.

**Ejemplo:**

Los departamentos de créditos en general (personas, pymes, empresarial) normalmente estudian la relación entre ingresos y deuda del candidato a crédito para decidir si acreditan o no a la persona.

Hay varias heurísticas que se utilizan. Por ejemplo:

- Si las deudas rebasan el 40% del apalancamiento (capital social + ingresos + pasivos), es una empresa de alto riesgo.
- Si las deudas rebasan 3 meses de ingresos, es una empresa de alto riesgo.

Se puede estudiar esta relación, y segmentar a los clientes de acuerdo a su perfil en estas variables.

```python
# Importamos función para generar datos
from bank_customer_data import generate_bank_customer_data
# Importamos pyplot
from matplotlib import pyplot as plt
```

```python
# Generamos datos
data = generate_bank_customer_data()
```

```python
data.head(10)
```

      income
      debt
      labels

      0
      2.784242
      1.362889
      0.0

      1
      1.592602
      2.562608
      0.0

      2
      3.480014
      1.563046
      0.0

      3
      4.254610
      2.204767
      2.0

      4
      1.000312
      5.724966
      1.0

      5
      2.051364
      1.143760
      0.0

      6
      3.715297
      2.682284
      0.0

      7
      2.142184
      2.812753
      0.0

      8
      6.087734
      4.938509
      2.0

      9
      1.761891
      6.742030
      1.0

```python
# Datos
plt.scatter(data["income"], data["debt"], c="gray", alpha=0.6)
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

Lo que queremos en clustering es identificar los grupos a los que pertenecen cada uno de los clientes.

```python
# Grupos "reales"
plt.scatter(data["income"], data["debt"], c=data["labels"], cmap="Accent", alpha=0.6)
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

Esta idea se conoce como **Hard clustering**; bajo este esquema, identificamos para cada punto un único grupo al que pertenece, es decir:

$$
\text{cluster_id}_x = f(x)
$$

- Los puntos verdes son 100% verdes.
- Los puntos azules son 100% azules.
- Los puntos grises son 100% grises.

Sin embargo, fijemos un momento nuestra atención en los recuadros a continuación ($[2, 3] \times [4, 5]$ y $[4, 5] \times [3, 4]$):

```python
plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.scatter(data["income"], data["debt"], c=data["labels"], cmap="Accent", alpha=0.6)
plt.axis([1, 3, 4, 5])
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")

plt.subplot(1, 2, 2)
plt.scatter(data["income"], data["debt"], c=data["labels"], cmap="Accent", alpha=0.6)
plt.axis([3.5, 5, 2, 4])
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

Desde un punto de vista intuitivo, los puntos en estos sectores no es muy claro a qué grupo pertenecen. Estaríamos tentados a decir que pertenecen a un grupo con cierta probabilidad y con cierta probabilidad a otro.

Esta idea se conoce como **Soft clustering**, y está estrechamente relacionado a **clustering probabilístico**.

$$
p(\text{cluster_id}_x |x)
$$

Al darle un enfoque probabilístico, tenemos varias ventajas colaterales:

- Sintonización de hiperparámetros.
- Modelo generativo.

## 3. Un algoritmo de hard clustering: K-Means

Aunque el K-Means es uno de los algoritmos de hard clustering más conocidos y usados, lo veremos en un par de sesiones más desde una perspectiva probabilística.

De manera que nos conviene estudiarlo antes.

**Problema:** dado un conjunto de observaciones $x_1, x_2, \dots, x_N \in \mathbb{R}^d$, se debe particionar las $N$ observaciones en $k$ ($\leq N$) clusters $\{1, 2, \dots, k\}$, de manera que se minimice la suma de distancias al cuadrado (varianza).

**Algoritmo:**

1. Inicializar los parámetros $\theta = \{\mu_1, \dots, \mu_k\}$ de manera aleatoria.
2. Repetir hasta la convergencia (hasta que los parámetros no varíen):
   1. Para cada punto calcule el centroide más cercano:
      $$
      c_i = \arg \min_{c} ||x_i - \mu_c||.
      $$

   2. Actualizar centroides:
      $$
      \mu_c = \frac{\sum_{i: c_i = c} x_i}{\sum_{i: c_i = c} 1}
      $$

**Tarea:** Programar el algoritmo K-Means.

Nosotros usaremos sklearn durante la clase:

```python
# Importamos sklearn.cluster.KMeans
from sklearn.cluster import KMeans
```

```python
# Algoritmo de sklearn
KMeans?
```

```python
# Instanciamos el algoritmo
kmeans = KMeans(n_clusters=3)
```

```python
# Entrenamos
kmeans.fit(X=data[["income", "debt"]])
```

    KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,
           n_clusters=3, n_init=10, n_jobs=None, precompute_distances='auto',
           random_state=None, tol=0.0001, verbose=0)

```python
# Gráfico
plt.scatter(data["income"], data["debt"], c=kmeans.labels_, cmap="Accent", alpha=0.6)
plt.plot(kmeans.cluster_centers_[0, 0], kmeans.cluster_centers_[0, 1], "*g", ms=20)
plt.plot(kmeans.cluster_centers_[1, 0], kmeans.cluster_centers_[1, 1], "*b", ms=20)
plt.plot(kmeans.cluster_centers_[2, 0], kmeans.cluster_centers_[2, 1], "*k", ms=20)
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

Todo se ve bien hasta acá. **¿Qué pasa si aumentamos el número de clusters?**

Una métrica que podemos usar para ver qué tan bueno está siendo el agrupado es la suma de las distancias al cuadrado de cada punto a su centroide respectivo:

$$
\frac{1}{N}\sum_{i=1}^N ||x_i - \mu_{c_i}||^2.
$$

```python
from sklearn.model_selection import train_test_split
import numpy as np
```

```python
def msd(X, cluster_id, centroids):
    """
    Mean squared distance.
    :param data: Data.
    :param centroids: Centroids.
    :return: Mean squared distance.
    """
    # Number of clusters
    k = centroids.shape[0]
    # Number of points
    N = X.shape[0]

    # Distances initialization
    distances = np.zeros(N)

    # Compute distances to corresponding cluster
    for j in range(k):
        distances[cluster_id == j] = np.linalg.norm(X[cluster_id == j] - centroids[j, :], axis=1)

    return (distances**2).mean()
```

```python
X_train, X_test = train_test_split(data[["income", "debt"]], test_size=0.2)

msd_train = []
msd_test = []
for k in range(2, 20):
    # Instanciamos el algoritmo
    kmeans = KMeans(n_clusters=k)

    # Entrenamos
    kmeans.fit(X=X_train)

    # Métrica con datos de entrenamiento
    msd_train.append(msd(X_train, kmeans.labels_, kmeans.cluster_centers_))

    # Métrica con datos de prueba
    msd_test.append(msd(X_test, kmeans.predict(X_test), kmeans.cluster_centers_))
```

```python
plt.plot(range(2, 20), msd_train, label="train")
plt.plot(range(2, 20), msd_test, label="test")
plt.legend()
plt.xlabel("Número de clusters")
plt.ylabel("Suma de distancias cuadradas")
```

```python
dist_orig = np.array(msd_test)**2 + np.arange(2, 20)**2
dist_orig
```

    array([ 12.12205427,  12.47908258,  17.78525849,  26.3824619 ,
            36.99025887,  49.80146088,  64.54630547,  81.42108952,
           100.34572121, 121.31073854, 144.29311666, 169.21797374,
           196.2439913 , 225.19173974, 256.15941281, 289.14232662,
           324.14097165, 361.14044157])

```python
np.arange(2, 20)[dist_orig.argmin()]
```

    2

```python
# Gráfico
plt.scatter(X_train["income"], X_train["debt"], c=kmeans.labels_, cmap="inferno", alpha=0.6)
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

Como observamos, esta métrica siempre decrece con la cantidad de clusters, lo que hace bastante complejo elegir un número de clusters adecuado cuando este es desconocido.

# 4. Modelo de mezcla Gaussiana (GMM)

Como vimos, el K-Means (y en general los algoritmos de hard clustering) tienen varios detalles:

- No es claro cómo elegir el número de clusters.
- Hay puntos que podrían estar en una frontera entre dos o más clusters, y el hard clustering no nos permite tener incertidumbre en la pertenencia.

Para lidiar con estos problemas, podemos plantear un modelo probablístico de nuestros datos.

Hasta ahora, conocemos algunas distribuciones. Entre ellas la Gaussiana, para la cual ya sabemos como estimar sus parámetros.

¿Qué pasa si intentamos ajustar una **distribución Gaussiana** a los datos? Es decir, si modelamos los datos con

$$
p(x|\theta) = \mathcal{N}(x|\mu, \Sigma), \qquad \theta=\{\mu, \Sigma\}
$$

```python
# Datos
plt.scatter(data["income"], data["debt"], c="gray", alpha=0.6)
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

```python
from scipy.stats import multivariate_normal
import numpy as np
```

```python
# Ajustamos parámetros
mu = data[["income", "debt"]].mean()
cov = data[["income", "debt"]].cov()
# Definimos VA
X = multivariate_normal(mean=mu, cov=cov)
```

```python
# Datos
plt.scatter(data["income"], data["debt"], c="gray", alpha=0.6)

# Gaussiana
x = np.linspace(0, 8, 100)
y = np.linspace(0, 9, 100)
x, y = np.meshgrid(x, y)
z = X.pdf(np.dstack([x, y]))
plt.contour(x, y, z)
plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

```python
mu
```

    income    2.913434
    debt      4.104847
    dtype: float64

Sin embargo, este modelo no parece corresponder con nuestros datos. La región de máxima probabilidad (media) cae en un punto medio entre los clusters, y allí no hay muchos datos.

### ¿Qué pasa si usamos varias Gaussianas?

```python
# Ajustamos parámetros
mu1 = data.loc[data["labels"] == 0, ["income", "debt"]].mean()
mu2 = data.loc[data["labels"] == 1, ["income", "debt"]].mean()
mu3 = data.loc[data["labels"] == 2, ["income", "debt"]].mean()
cov1 = data.loc[data["labels"] == 0, ["income", "debt"]].cov()
cov2 = data.loc[data["labels"] == 1, ["income", "debt"]].cov()
cov3 = data.loc[data["labels"] == 2, ["income", "debt"]].cov()
# Definimos VA
X1 = multivariate_normal(mean=mu1, cov=cov1)
X2 = multivariate_normal(mean=mu2, cov=cov2)
X3 = multivariate_normal(mean=mu3, cov=cov3)
```

```python
# Datos
plt.scatter(data["income"], data["debt"], c="gray", alpha=0.6)

# Gaussiana 1
x = np.linspace(0, 6, 100)
y = np.linspace(0, 5, 100)
x, y = np.meshgrid(x, y)
z = X1.pdf(np.dstack([x, y]))
plt.contour(x, y, z)

# Gaussiana 2
x = np.linspace(0, 4, 100)
y = np.linspace(4, 9, 100)
x, y = np.meshgrid(x, y)
z = X2.pdf(np.dstack([x, y]))
plt.contour(x, y, z)

# Gaussiana 3
x = np.linspace(3, 8, 100)
y = np.linspace(2, 8, 100)
x, y = np.meshgrid(x, y)
z = X3.pdf(np.dstack([x, y]))
plt.contour(x, y, z)

plt.xlabel("Ingresos mensuales (x100k MXN)")
plt.ylabel("Deuda (x100k MXN)")
```

**¡Mucho mejor!**

Cada Gaussiana explica un cluster de puntos, y el modelo general sería una suma ponderada de estas densidades Gaussianas:

$$
p(x | \theta) = \sum_{c=1}^{3} \pi_c \mathcal{N}(x | \mu_c, \Sigma_c), \qquad \theta = \{\pi_1, \pi_2, \pi_3, \mu_1, \mu_2, \mu_3, \Sigma_1, \Sigma_2, \Sigma_3\}
$$

**¿Y esto cómo lo interpretamos?**

Bueno, pues si logramos encontrar los parámetros $\pi_1, \pi_2, \pi_3, \mu_1, \mu_2, \mu_3, \Sigma_1, \Sigma_2, \Sigma_3$ para este conjunto de datos, habremos resuelto el problema de (soft) clustering, ya que encontraremos para cada punto la probabilidad de que venga de cada una de las Gaussianas.

**¿Qué ventajas tenemos?**

Como ventaja respecto a usar una sola Gaussiana, hemos añadido flexibilidad a nuestro modelo. Es decir, con esta estuctura podemos representar conjuntos de datos complejos.

> En efecto, podemos aproximar casi cualquier distribución continua con una mezcla de Gaussianas con precisión arbitraria, dado que incluyamos un número suficiente de Gaussianas en la mezcla.

**¿A qué costo?**

La cantidad de parámetros que debemos estimar se multiplica por la cantidad de Gaussianas en la mezcla.

### ¿Cómo encontramos (entrenamos) los parámetros?

Podemos maximizar la función de verosimilitud (suposición de independencia):

$$
\max_{\theta} p(X | \theta) = \prod_{i=1}^N p(x_i | \theta) = \prod_{i=1}^N \sum_{c=1}^{3} \pi_c \mathcal{N}(x_i | \mu_c, \Sigma_c)
$$

sujeto a:

\begin{align}
\sum_{c=1}^3 \pi_c & = 1 \\
\pi_c & \geq 0 \quad \text{for } c=1,2,3\\
\Sigma_c & \succ 0 \quad \text{for } c=1,2,3
\end{align}

Es decir, las matrices de covarianza deben ser definidas positivas (¿por qué?).

**Complejidades numéricas:**

Este problema de optimización puede ser resuelto numéricamente con un algoritmo como el gradiente descendiente. Sin embargo,

1. La restricción sobre la matriz de covarianzas hace el problema de optimización muy complejo de resolver numéricamente hablando.

   Una simplificación para poder trabajar con esta restricción es suponer que las matrices de covarianza son diagonales:

$$
\Sigma_c = \text{diag}(\sigma_{c1}, \sigma_{c2}, \dots, \sigma_{cn}),
$$

2. La suma dentro del producto también hace bastante complejo el cálculo de los gradientes. Comúnmente, para evitar el producto se toma logaritmo de la verosimilitud:

   $$
   \log p(X | \theta) = \log \left(\prod_{i=1}^N p(x_i | \theta) \right)= \sum_{i=1}^N \log\left(\sum_{c=1}^{3} \pi_c \mathcal{N}(x_i | \mu_c, \Sigma_c)\right)
   $$

   y con esto, podemos observar que permanece una suma ponderada dentro del logaritmo.

### ¿Y entonces?

Afortunadamente, existe un algoritmo alternativo con base probabilística llamado **algoritmo de maximización de la esperanza**, el cual estaremos estudiando en las próximas clases no solo para el problema de mezclas Gaussianas, sino para entrenar cualquier modelo con **variables latentes**.

**¿Variables latentes?**

Recordamos que propusimos el siguiente modelo:

$$
p(x | \theta) = \sum_{c=1}^{3} \pi_c \mathcal{N}(x | \mu_c, \Sigma_c), \qquad \theta = \{\pi_1, \pi_2, \pi_3, \mu_1, \mu_2, \mu_3, \Sigma_1, \Sigma_2, \Sigma_3\}
$$

Este modelo en realidad, lo podemos pensar como un modelo con una variable latente $t$ que determina a cuál Gaussiana pertenece cada punto:

Entonces, razonablemente podemos atribuirle a $t$ tres posibles valores (1, 2, y 3), que nos dicen de qué Gaussiana viene el punto. Recordamos que $t$ es una variable latente, nunca la observamos.

Sin embargo, razonando probabilísticamente, después de entrenar nuestra mezcla Gaussiana, podríamos preguntarle al modelo, ¿Cuál es el valor más probable de $t$ dado el punto $x$? --> **Clustering**

Con este modelo, podemos asignar las siguientes probabilidades:

- Previa:

  $$
  p(t=c | \theta) = \pi_c
  $$

- Verosimilitud:

  $$
  p(x | t=c, \theta) = \mathcal{N}(x | \mu_c, \Sigma_c)
  $$

Razonable, ¿no?

Y con lo anterior,

$$
p(x | \theta) = \sum_{c=1}^3 p(x, t=c | \theta) = \sum_{c=1}^3 \underbrace{p(x | t=c, \theta)}_{\mathcal{N}(x | \mu_c, \Sigma_c)} \underbrace{p(t=c | \theta)}_{\pi_c},
$$

justo como el modelo intuitivo que habíamos propuesto.

### Algoritmo de maximización de la esperanza para mezclas Gaussianas - Intuición

Supongamos que tenemos los siguientes puntos de tamaños de playeras, y queremos definir cuáles son talla chica y cuales son talla grande:

```python
from shirts_size_data import generate_shirts_data
from scipy.stats.distributions import norm
```

```python
shirts_data = generate_shirts_data()
shirts_data.head()
```

      size
      labels

      0
      31.554606
      0.0

      1
      31.222931
      0.0

      2
      30.678384
      0.0

      3
      42.104057
      1.0

      4
      40.355446
      1.0

```python
plt.scatter(shirts_data["size"], np.zeros(len(shirts_data)), c="gray", alpha=0.6)
plt.ylim([-0.01, 0.3])
plt.xlabel("Tamaño (cm)")
```

¿Cómo estimamos los parámetros de nuestro modelo de variable latente?

Analicemos varios casos:

1. Si de entrada supiéramos cuáles playeras son chicas y cuáles grandes:

```python
plt.scatter(shirts_data.loc[shirts_data["labels"] == 0, "size"],
            np.zeros((shirts_data["labels"] == 0).sum()),
            alpha=0.6)
plt.scatter(shirts_data.loc[shirts_data["labels"] == 1, "size"],
            np.zeros((shirts_data["labels"] == 1).sum()),
            alpha=0.6)
x = shirts_data["size"].copy().values
x.sort()
mu1 = shirts_data.loc[shirts_data["labels"] == 0, "size"].mean()
mu2 = shirts_data.loc[shirts_data["labels"] == 1, "size"].mean()
s1 = shirts_data.loc[shirts_data["labels"] == 0, "size"].std()
s2 = shirts_data.loc[shirts_data["labels"] == 1, "size"].std()
plt.plot(x, norm.pdf(x, loc=mu1, scale=s1))
plt.plot(x, norm.pdf(x, loc=mu2, scale=s2))
plt.ylim([-0.01, 0.3])
plt.xlabel("Tamaño (cm)")
```

$$
p(x | t=1, \theta) = \mathcal{N}(x | \mu_1, \sigma_1)
$$

Con lo cual:

$$
\mu_1 = \frac{\sum_{i: t_i = 1} x_i}{\sum_{i: t_i = 1} 1}, \qquad \sigma_1^2 = \frac{\sum_{i: t_i = 1} (x_i - \mu_1)^2}{\sum_{i: t_i = 1} 1},
$$

y

$$
\mu_2 = \frac{\sum_{i: t_i = 2} x_i}{\sum_{i: t_i = 2} 1}, \qquad \sigma_2^2 = \frac{\sum_{i: t_i = 2} (x_i - \mu_1)^2}{\sum_{i: t_2 = 1} 1}
$$

2. Como sablemos, en el algoritmo de mezclas Gaussianas nunca sabremos si un punto pertenece a cierto cluster o no, sino que conoceremos las probabilidades de que pertenezca a cada cluster.

   De modo que si conocemos la posterior $p(t | x, \theta)$, entonces ponderamos lo anterior por esta probabilidad:

   $$
\mu_1 = \frac{\sum_{i} p(t_i=1 | x_i, \theta)x_i}{\sum_{i} p(t_i=1 | x_i, \theta)}, \qquad \sigma_1^2 = \frac{\sum_{i} p(t_i=1 | x_i, \theta) (x_i - \mu_1)^2}{\sum_{i} p(t_i=1 | x_i, \theta)}.
   $$

3. ¿Y cómo conocemos la posterior $p(t | x, \theta)$?

   Bueno, pues si conocemos los parámetros, es bastante fácil:

   $$
   p(t=c | x, \theta) = \frac{p(x | t=c, \theta) p(t=c | \theta)}{Z} = \frac{\pi_c \mathcal{N}(x | \mu_c, \sigma_c)}{Z}.
   $$

Tenemos un razonamiento circular (un problema del tipo, ¿Qué fue primero?, ¿El huevo?, ¿O la gallina?).

**¿Cómo lo resolvemos? Iterando...**

**Algoritmo de maximización de la esperanza:**

1. Inicializamos los parámetros de cada Gaussiana aleatoriamente.

2. Repetir hasta la convergencia:

   - Calcular para cada punto la probabilidad posterior $p(t_i=c | x_i, \theta)$.
   - Actualizar los parámetros de las Gaussianas con las probabildades calculadas.

Created with Jupyter by Esteban Jiménez Rodríguez.


# A neural netowork to test the primality of a number.

## Install libraries on the VM

```
!uname -a
!lsb_release -a
!apt install graphviz
!pip install -q pydot
```

## Import python libraries

```
import tensorflow as tf
print("Tensorflow version: {}".format(tf.__version__))
```

    Tensorflow version: 1.12.0-rc2

```
print(tf.__version__)
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
```

    1.12.0-rc2
    Found GPU at: /device:GPU:0

```
import keras
import numpy as np
import math
from sympy import primerange, sieve, composite, prime

import pickle
import timeit

from PIL import Image, ImageColor
%matplotlib inline
import os

np.random.seed(1)
```

    Using TensorFlow backend.

## Data preparation and utility functions

```
def convert_dec2bin(num_list,feature_size):
  p_x = None
  for p in num_list:
    pb = bin(p)[2:].zfill(32)
    li = [ int(i) for i in pb]
    a1 = np.array(li).reshape((feature_size,1))
    if p_x is None:
      p_x = a1
    else:
      p_x = np.concatenate((p_x, a1), axis=1)
  return p_x

def generate_sequence_primes(between):
  start, end = between
  p_num = [i for i in primerange(start, end)]
  return p_num, len(p_num)

def generate_random_primes(size, primes):
  le = len(primes)
  p_num = [primes[i] for i in np.random.randint(le , size=size)]
  return p_num, size

def generate_composite(size, primes_set):
  c_num = np.random.randint(10000000, size=size)
  c_num = set(c_num)
  c_num = c_num - primes_set
  return list(c_num), len(c_num)

def generate_x_y(primes=None, composites=None, feature_size=32, ):
  c_examples_size = 0
  c_x = None
  c_y = None
  p_examples_size = 0
  p_x = None
  p_y = None
  X = None
  Y = None

  if primes is not None :
    p_examples_size = len(primes)
    p_x = convert_dec2bin(primes, feature_size)
    p_y = np.ones(shape=(1,p_examples_size), dtype=np.int32)

  if composites is not None:
    c_examples_size = len(composites)
    c_x = convert_dec2bin(composites, feature_size)
    c_y = np.zeros(shape=(1,c_examples_size), dtype=np.int32)

  if c_x is not None and p_x is not None:
    X = np.concatenate((c_x, p_x), axis=1)
    size = p_examples_size + c_examples_size
    Y = np.append(c_y, p_y).reshape((1, size))
  elif p_x is not None:
    X = p_x
    Y = p_y
    size = p_examples_size
  elif c_x is not None:
    X = c_x
    Y = c_y
    size = c_examples_size

  return X, Y, size

def one_hot_matrix(labels, C):
    """
    Creates a matrix where the i-th row corresponds to the ith class number and the jth column
                     corresponds to the jth training example. So if example j had a label i. Then entry (i,j)
                     will be 1.

    Arguments:
    labels -- vector containing the labels
    C -- number of classes, the depth of the one hot dimension

    Returns:
    one_hot -- one hot matrix
    """

    C = tf.constant(dtype=tf.int32, name="C", value=C)
    one_hot_matrix = tf.one_hot(labels,depth=C, axis=0)
    with tf.Session() as sess:
      one_hot = sess.run(one_hot_matrix)

    return one_hot

def save_primes(file, start, end):
  p_num = [i for i in primerange(start, end)]
  pickle.dump( p_num, open(file, "wb" ))
  return p_num

def load_primes(file):
 return pickle.load(open(file, "rb" ))

```

## Generate the Primes

```
%%time
start = 2147483647 - 10000000
end = 2147483647
PRIMES_FILENAME = "primes1.p"
if not os.path.exists(PRIMES_FILENAME):
  PRIMES = save_primes(PRIMES_FILENAME, start, end)
else:
  PRIMES = load_primes(PRIMES_FILENAME)
print(len(PRIMES))
```

    465816
    CPU times: user 36.3 s, sys: 633 ms, total: 36.9 s
    Wall time: 36.9 s

## Generate the Training and Test Datasets

```
examples_size = 5000
primes_set = set(PRIMES)
c, size = generate_composite(examples_size, primes_set)
p, size = generate_random_primes(examples_size,PRIMES)
s_p_train = set(p)
X_train, Y_train, training_size = generate_x_y(p,c)

examples_size = 100
c, size = generate_composite(examples_size, primes_set)
p, size = generate_random_primes(examples_size,PRIMES)

# remove the primes common for both test and train
s_p_test = set(p)
s_p_test = s_p_test - s_p_train
X_test, Y_test, test_size = generate_x_y(list(s_p_test),c)

Y_train = one_hot_matrix(Y_train.reshape(training_size), 2)
Y_test = one_hot_matrix(Y_test.reshape(test_size), 2)
```

```
print(X_train.T.shape)
```

    (9999, 32)

## Model Creation and Training

```
# Create your first MLP in Keras
from keras.models import Sequential
from keras.layers import Dense

X_train.shape

# create model
model = Sequential()
model.add(Dense(64, input_dim=32, activation='relu',))
model.add(Dense(2, input_dim=64,activation='sigmoid'))
#model.add(Dense(2,input_dim=, activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
history = model.fit(X_train.T, Y_train.T, validation_split=0.25, epochs=5, batch_size=10, verbose=1)
# evaluate the model
scores = model.evaluate(X_train.T, Y_train.T)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
```

    Train on 7499 samples, validate on 2500 samples
    Epoch 1/5
    7499/7499 [==============================] - 6s 818us/step - loss: 0.0640 - acc: 0.9871 - val_loss: 0.0010 - val_acc: 1.0000
    Epoch 2/5
    7499/7499 [==============================] - 5s 705us/step - loss: 4.9445e-04 - acc: 1.0000 - val_loss: 1.9856e-04 - val_acc: 1.0000
    Epoch 3/5
    7499/7499 [==============================] - 5s 730us/step - loss: 1.5551e-04 - acc: 1.0000 - val_loss: 9.5011e-05 - val_acc: 1.0000
    Epoch 4/5
    7499/7499 [==============================] - 5s 708us/step - loss: 7.0929e-05 - acc: 1.0000 - val_loss: 4.5775e-05 - val_acc: 1.0000
    Epoch 5/5
    7499/7499 [==============================] - 5s 703us/step - loss: 3.7805e-05 - acc: 1.0000 - val_loss: 2.6371e-05 - val_acc: 1.0000
    9999/9999 [==============================] - 1s 73us/step

    acc: 100.00%

## Visualize the Model

```
from keras.utils import plot_model
plot_model(model, to_file='model.png', show_shapes=True, show_layer_names=False, rankdir='LR' )
from IPython.display import Image
Image(filename='model.png')
```

## Visualize Model Accuracy

```
import matplotlib.pyplot as plt

# Plot training & validation accuracy values
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()

# Plot training & validation loss values
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()
```

## Evaluate the Test Datasets

```
scores = model.evaluate(X_test.T, Y_test.T)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
```

    200/200 [==============================] - 0s 161us/step

    acc: 100.00%

## Inference - Sequence test.

Check all to see if the model can accurately test for prime for all the PRIMES

```
p = np.array(PRIMES)
print(p.shape)
size = p.shape[0]

width = 500
for i in range(0,1000,width):
  x = p[i: i+width]
  X, Y, size = generate_x_y(x,None)
  Y = one_hot_matrix(Y.reshape(size), 2)
  scores = model.evaluate(X.T, Y.T)
  print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

```

    (465816,)
    500/500 [==============================] - 0s 95us/step

    acc: 100.00%
    500/500 [==============================] - 0s 94us/step

    acc: 100.00%


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, C. Cooper. Based on [CFDPython](https://github.com/barbagroup/CFDPython), (c)2013 L.A. Barba, also under CC-BY license.

# Space & Time

## Burgers' Equation

Hi there! We have reached the final lesson of the series *Space and Time — Introduction to Finite-difference solutions of PDEs*, the second module of ["Practical Numerical Methods with Python"](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/about).

We have learned about the finite-difference solution for the linear and non-linear convection equations and the diffusion equation. It's time to combine all these into one: *Burgers' equation*. The wonders of *code reuse*!

Before you continue, make sure you have completed the previous lessons of this series, it will make your life easier. You should have written your own versions of the codes in separate, clean Jupyter Notebooks or Python scripts.

You can read about Burgers' Equation on its [wikipedia page](http://en.wikipedia.org/wiki/Burgers'_equation).
Burgers' equation in one spatial dimension looks like this:

$$
\begin{equation}
\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = \nu \frac{\partial ^2u}{\partial x^2}
\end{equation}
$$

As you can see, it is a combination of non-linear convection and diffusion. It is surprising how much you learn from this neat little equation!

We can discretize it using the methods we've already detailed in the previous notebooks of this module.  Using forward difference for time, backward difference for space and our 2nd-order method for the second derivatives yields:

$$
\begin{equation}
\frac{u_i^{n+1}-u_i^n}{\Delta t} + u_i^n \frac{u_i^n - u_{i-1}^n}{\Delta x} = \nu \frac{u_{i+1}^n - 2u_i^n + u_{i-1}^n}{\Delta x^2}
\end{equation}
$$

As before, once we have an initial condition, the only unknown is $u_i^{n+1}$. We will step in time as follows:

$$
\begin{equation}
u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)
\end{equation}
$$

### Initial and Boundary Conditions

To examine some interesting properties of Burgers' equation, it is helpful to use different initial and boundary conditions than we've been using for previous steps.

The initial condition for this problem is going to be:

$$
\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi(t=0) = \phi_0 &=& \exp \bigg(\frac{-x^2}{4 \nu} \bigg) + \exp \bigg(\frac{-(x-2 \pi)^2}{4 \nu} \bigg)
\end{eqnarray}
$$

This has an analytical solution, given by:

$$
\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi &=& \exp \bigg(\frac{-(x-4t)^2}{4 \nu (t+1)} \bigg) + \exp \bigg(\frac{-(x-4t -2 \pi)^2}{4 \nu(t+1)} \bigg)
\end{eqnarray}
$$

The boundary condition will be:

$$
\begin{equation}
u(0) = u(2\pi)
\end{equation}
$$

This is called a *periodic* boundary condition. Pay attention! This will cause you a bit of headache if you don't tread carefully.

### Saving Time with SymPy

The initial condition we're using for Burgers' Equation can be a bit of a pain to evaluate by hand.  The derivative $\frac{\partial \phi}{\partial x}$ isn't too terribly difficult, but it would be easy to drop a sign or forget a factor of $x$ somewhere, so we're going to use SymPy to help us out.

[SymPy](http://sympy.org/en/) is the symbolic math library for Python.  It has a lot of the same symbolic math functionality as Mathematica with the added benefit that we can easily translate its results back into our Python calculations (it is also free and open source).

Start by loading the SymPy library, together with our favorite library, NumPy.

```python
import numpy
import sympy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

We're also going to tell SymPy that we want all of its output to be rendered using $\LaTeX$. This will make our Notebook beautiful!

```python
sympy.init_printing()
```

Start by setting up symbolic variables for the three variables in our initial condition.  It's important to recognize that once we've defined these symbolic variables, they function differently than "regular" Python variables.

If we type `x` into a code block, we'll get an error:

```python
x
```

`x` is not defined, so this shouldn't be a surprise.  Now, let's set up `x` as a *symbolic* variable:

```python
x = sympy.symbols('x')
```

Now let's see what happens when we type `x` into a code cell:

```python
x
```

The value of `x` is $x$.  Sympy is also referred to as a computer algebra system -- normally the value of `5*x` will return the product of `5` and whatever value `x` is pointing to.  But, if we define `x` as a symbol, then something else happens:

```python
5 * x
```

This will let us manipulate an equation with unknowns using Python!  Let's start by defining symbols for $x$, $\nu$ and $t$ and then type out the full equation for $\phi$.  We should get a nicely rendered version of our $\phi$ equation.

```python
x, nu, t = sympy.symbols('x nu t')
phi = (sympy.exp(-(x - 4 * t)**2 / (4 * nu * (t + 1))) +
       sympy.exp(-(x - 4 * t - 2 * sympy.pi)**2 / (4 * nu * (t + 1))))
phi
```

It's maybe a little small, but that looks right.  Now to evaluate our partial derivative $\frac{\partial \phi}{\partial x}$ is a trivial task.  To take a derivative with respect to $x$, we can just use:

```python
phiprime = phi.diff(x)
phiprime
```

If you want to see the non-rendered version, just use the Python print command.

```python
print(phiprime)
```

    -(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 4*pi)*exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1)))/(4*nu*(t + 1))

### Now what?

Now that we have the Pythonic version of our derivative, we can finish writing out the full initial condition equation and then translate it into a usable Python expression.  For this, we'll use the *lambdify* function, which takes a SymPy symbolic equation and turns it into a callable function.

```python
from sympy.utilities.lambdify import lambdify

u = -2 * nu * (phiprime / phi) + 4
print(u)
```

    -2*nu*(-(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 4*pi)*exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)))/(exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1))) + exp(-(-4*t + x)**2/(4*nu*(t + 1)))) + 4

### Lambdify

To lambdify this expression into a usable function, we tell lambdify which variables to request and the function we want to plug them into.

```python
u_lamb = lambdify((t, x, nu), u)
print('The value of u at t=1, x=4, nu=3 is {}'.format(u_lamb(1, 4, 3)))
```

    The value of u at t=1, x=4, nu=3 is 3.49170664206445

### Back to Burgers' Equation

Now that we have the initial conditions set up, we can proceed and finish setting up the problem.  We can generate the plot of the initial condition using our lambdify-ed function.

```python
# Set parameters.
nx = 101  # number of spatial grid points
L = 2.0 * numpy.pi  # length of the domain
dx = L / (nx - 1)  # spatial grid size
nu = 0.07  # viscosity
nt = 100  # number of time steps to compute
sigma = 0.1  # CFL limit
dt = sigma * dx**2 / nu  # time-step size

# Discretize the domain.
x = numpy.linspace(0.0, L, num=nx)
```

We have a function `u_lamb` but we need to create an array `u0` with our initial conditions.  `u_lamb` will return the value for any given time $t$, position $x$ and $nu$.  We can use a `for`-loop to cycle through values of `x` to generate the `u0` array.  That code would look something like this:

```Python
u0 = numpy.empty(nx)

for i, x0 in enumerate(x):
    u0[i] = u_lamb(t, x0, nu)
```

But there's a cleaner, more beautiful way to do this -- *list comprehension*.

We can create a list of all of the appropriate `u` values by typing

```Python
[u_lamb(t, x0, nu) for x0 in x]
```

You can see that the syntax is similar to the `for`-loop, but it only takes one line.  Using a list comprehension will create... a list.  This is different from an *array*, but converting a list to an array is trivial using `numpy.asarray()`.

With the list comprehension in place, the three lines of code above become one:

```Python
u = numpy.asarray([u_lamb(t, x0, nu) for x0 in x])
```

```python
# Set initial conditions.
t = 0.0
u0 = numpy.array([u_lamb(t, xi, nu) for xi in x])
u0
```

    array([4.        , 4.06283185, 4.12566371, 4.18849556, 4.25132741,
           4.31415927, 4.37699112, 4.43982297, 4.50265482, 4.56548668,
           4.62831853, 4.69115038, 4.75398224, 4.81681409, 4.87964594,
           4.9424778 , 5.00530965, 5.0681415 , 5.13097336, 5.19380521,
           5.25663706, 5.31946891, 5.38230077, 5.44513262, 5.50796447,
           5.57079633, 5.63362818, 5.69646003, 5.75929189, 5.82212374,
           5.88495559, 5.94778745, 6.0106193 , 6.07345115, 6.136283  ,
           6.19911486, 6.26194671, 6.32477856, 6.38761042, 6.45044227,
           6.51327412, 6.57610598, 6.63893783, 6.70176967, 6.76460125,
           6.82742866, 6.89018589, 6.95176632, 6.99367964, 6.72527549,
           4.        , 1.27472451, 1.00632036, 1.04823368, 1.10981411,
           1.17257134, 1.23539875, 1.29823033, 1.36106217, 1.42389402,
           1.48672588, 1.54955773, 1.61238958, 1.67522144, 1.73805329,
           1.80088514, 1.863717  , 1.92654885, 1.9893807 , 2.05221255,
           2.11504441, 2.17787626, 2.24070811, 2.30353997, 2.36637182,
           2.42920367, 2.49203553, 2.55486738, 2.61769923, 2.68053109,
           2.74336294, 2.80619479, 2.86902664, 2.9318585 , 2.99469035,
           3.0575222 , 3.12035406, 3.18318591, 3.24601776, 3.30884962,
           3.37168147, 3.43451332, 3.49734518, 3.56017703, 3.62300888,
           3.68584073, 3.74867259, 3.81150444, 3.87433629, 3.93716815,
           4.        ])

Now that we have the initial conditions set up, we can plot it to see what $u(x,0)$ looks like:

```python
# Plot the initial conditions.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.title('Initial conditions')
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, color='C0', linestyle='-', linewidth=2)
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0);
```

This is definitely not the hat function we've been dealing with until now. We call it a "saw-tooth function".  Let's proceed forward and see what happens.

### Periodic Boundary Conditions

We will implement Burgers' equation with *periodic* boundary conditions.  If you experiment with the linear and non-linear convection notebooks and make the simulation run longer (by increasing `nt`) you will notice that the wave will keep moving to the right until it no longer even shows up in the plot.

With periodic boundary conditions, when a point gets to the right-hand side of the frame, it *wraps around* back to the front of the frame.

Recall the discretization that we worked out at the beginning of this notebook:

$$
\begin{equation}
u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)
\end{equation}
$$

What does $u_{i+1}^n$ *mean* when $i$ is already at the end of the frame?

Think about this for a minute before proceeding.

```python
# Integrate the Burgers' equation in time.
u = u0.copy()
for n in range(nt):
    un = u.copy()
    # Update all interior points.
    u[1:-1] = (un[1:-1] -
               un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
               nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
    # Update boundary points.
    u[0] = (un[0] -
            un[0] * dt / dx * (un[0] - un[-1]) +
            nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
    u[-1] = (un[-1] -
            un[-1] * dt / dx * (un[-1] - un[-2]) +
            nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
```

```python
# Compute the analytical solution.
u_analytical = numpy.array([u_lamb(nt * dt, xi, nu) for xi in x])
```

```python
# Plot the numerical solution along with the analytical solution.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u, label='Numerical',
            color='C0', linestyle='-', linewidth=2)
pyplot.plot(x, u_analytical, label='Analytical',
            color='C1', linestyle='--', linewidth=2)
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0);
```

Let's now create an animation with the `animation` module of Matplotlib to observe how the numerical solution changes over time compared to the analytical solution.
We start by importing the module from Matplotlib as well as the special `HTML` display method.

```python
from matplotlib import animation
from IPython.display import HTML
```

We create a function `burgers` to computes the numerical solution of the 1D Burgers' equation over time.
(The function returns the history of the solution: a list with `nt` elements, each one being the solution in the domain at a time step.)

```python
def burgers(u0, dx, dt, nu, nt=20):
    """
    Computes the numerical solution of the 1D Burgers' equation
    over the time steps.

    Parameters
    ----------
    u0 : numpy.ndarray
        The initial conditions as a 1D array of floats.
    dx : float
        The grid spacing.
    dt : float
        The time-step size.
    nu : float
        The viscosity.
    nt : integer, optional
        The number of time steps to compute;
        default: 20.

    Returns
    -------
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    u_hist = [u0.copy()]
    u = u0.copy()
    for n in range(nt):
        un = u.copy()
        # Update all interior points.
        u[1:-1] = (un[1:-1] -
                   un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
                   nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
        # Update boundary points.
        u[0] = (un[0] -
                un[0] * dt / dx * (un[0] - un[-1]) +
                nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
        u[-1] = (un[-1] -
                un[-1] * dt / dx * (un[-1] - un[-2]) +
                nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
        u_hist.append(u.copy())
    return u_hist
```

```python
# Compute the history of the numerical solution.
u_hist = burgers(u0, dx, dt, nu, nt=nt)
```

```python
# Compute the history of the analytical solution.
u_analytical = [numpy.array([u_lamb(n * dt, xi, nu) for xi in x])
                for n in range(nt)]
```

```python
fig = pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
u0_analytical = numpy.array([u_lamb(0.0, xi, nu) for xi in x])
line1 = pyplot.plot(x, u0, label='Numerical',
                    color='C0', linestyle='-', linewidth=2)[0]
line2 = pyplot.plot(x, u0_analytical, label='Analytical',
                    color='C1', linestyle='--', linewidth=2)[0]
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 10.0)
fig.tight_layout()
```

```python
def update_plot(n, u_hist, u_analytical):
    """
    Update the lines y-data of the Matplotlib figure.

    Parameters
    ----------
    n : integer
        The time-step index.
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    u_analytical : list of numpy.ndarray objects
        The history of the analytical solution.
    """
    fig.suptitle('Time step {:0>2}'.format(n))
    line1.set_ydata(u_hist[n])
    line2.set_ydata(u_analytical[n])
```

```python
# Create an animation.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(u_hist, u_analytical),
                               interval=100)
```

```python
# Display the video.
HTML(anim.to_html5_video())
```

## Array Operation Speed Increase

Coding up discretization schemes using array operations can be a bit of a pain.  It requires much more mental effort on the front-end than using two nested `for` loops.  So why do we do it?  Because it's fast.  Very, very fast.

Here's what the Burgers code looks like using two nested `for` loops.  It's easier to write out, plus we only have to add one "special" condition to implement the periodic boundaries.

At the top of the cell, you'll see the decorator `%%timeit`.
This is called a "cell magic".  It runs the cell several times and returns the average execution time for the contained code.

Let's see how long the nested `for` loops take to finish.

```python
%%timeit
# Set initial conditions.
u = numpy.array([u_lamb(t, x0, nu) for x0 in x])
# Integrate in time using a nested for loop.
for n in range(nt):
    un = u.copy()
    # Update all interior points and the left boundary point.
    for i in range(nx - 1):
        u[i] = (un[i] -
                un[i] * dt / dx *(un[i] - un[i - 1]) +
                nu * dt / dx**2 * (un[i + 1] - 2 * un[i] + un[i - 1]))
    # Update the right boundary.
    u[-1] = (un[-1] -
             un[-1] * dt / dx * (un[-1] - un[-2]) +
             nu * dt / dx**2 * (un[0]- 2 * un[-1] + un[-2]))
```

    23.1 ms ± 344 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Less than 50 milliseconds.  Not bad, really.

Now let's look at the array operations code cell.  Notice that we haven't changed anything, except we've added the `%%timeit` magic and we're also resetting the array `u` to its initial conditions.

This takes longer to code and we have to add two special conditions to take care of the periodic boundaries.  Was it worth it?

```python
%%timeit
# Set initial conditions.
u = numpy.array([u_lamb(t, xi, nu) for xi in x])
# Integrate in time using array operations.
for n in range(nt):
    un = u.copy()
    # Update all interior points.
    u[1:-1] = (un[1:-1] -
               un[1:-1] * dt / dx * (un[1:-1] - un[:-2]) +
               nu * dt / dx**2 * (un[2:] - 2 * un[1:-1] + un[:-2]))
    # Update boundary points.
    u[0] = (un[0] -
            un[0] * dt / dx * (un[0] - un[-1]) +
            nu * dt / dx**2 * (un[1] - 2 * un[0] + un[-1]))
    u[-1] = (un[-1] -
            un[-1] * dt / dx * (un[-1] - un[-2]) +
            nu * dt / dx**2 * (un[0] - 2 * un[-1] + un[-2]))
```

    2.52 ms ± 64.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Yes, it is absolutely worth it.  That's a nine-fold speed increase.  For this exercise, you probably won't miss the extra 40 milliseconds if you use the nested `for` loops, but what about a simulation that has to run through millions and millions of iterations?  Then that little extra effort at the beginning will definitely pay off.

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


# Halo model power spectra
This example illustrates how to use CCL to calculate power spectra using the halo model.

```python
import numpy as np
import pyccl as ccl
import pylab as plt
import matplotlib.cm as cm
from scipy.special import erf
%matplotlib inline
```

## Preliminaries
Generate a cosmology object and a couple of $k$ (comoving wavenumber) and $a$ (scale factor) arrays.

```python
# Cosmology
cosmo = ccl.Cosmology(Omega_c=0.27, Omega_b=0.045, h=0.67, sigma8=0.83, n_s=0.96)

# Wavenumbers and scale factors
k_arr = np.geomspace(1E-4,1E1,256)
a_arr = np.linspace(0.1,1,32)

```

Let us also initialize a series of halo model quantities. We will need:
- A halo mass definition.
- A concentration-mass relation.
- A halo mass function parametrization.
- A halo bias parametrization.
- Halo profiles corresponding to the quantities we want to compute the power spectrum of.

```python
# We will use a mass definition with Delta = 200 times the matter density
hmd_200m = ccl.halos.MassDef200m()

# The Duffy 2008 concentration-mass relation
cM = ccl.halos.ConcentrationDuffy08(hmd_200m)

# The Tinker 2008 mass function
nM = ccl.halos.MassFuncTinker08(cosmo, mass_def=hmd_200m)

# The Tinker 2010 halo bias
bM = ccl.halos.HaloBiasTinker10(cosmo, mass_def=hmd_200m)

# The NFW profile to characterize the matter density around halos
pM = ccl.halos.profiles.HaloProfileNFW(cM)
```

Additionally, in order to showcase the flexibility of the current halo model implementation, let us create our own halo profile characterizing the galaxy overdensity using a Halo Occupation Distribution model. We will use the same parametrization used in Nicola et al. https://arxiv.org/abs/1912.08209.

```python
class HaloProfileHOD(ccl.halos.HaloProfileNFW):
    def __init__(self, c_M_relation,
                 lMmin=12.02, lMminp=-1.34,
                 lM0=6.6, lM0p=-1.43,
                 lM1=13.27, lM1p=-0.323):
        self.lMmin=lMmin
        self.lMminp=lMminp
        self.lM0=lM0
        self.lM0p=lM0p
        self.lM1=lM1
        self.lM1p=lM1p
        self.a0 = 1./(1+0.65)
        self.sigmaLogM = 0.4
        self.alpha = 1.
        super(HaloProfileHOD, self).__init__(c_M_relation)
        self._fourier = self._fourier_analytic_hod

    def _lMmin(self, a):
        return self.lMmin + self.lMminp * (a - self.a0)

    def _lM0(self, a):
        return self.lM0 + self.lM0p * (a - self.a0)

    def _lM1(self, a):
        return self.lM1 + self.lM1p * (a - self.a0)

    def _Nc(self, M, a):
        # Number of centrals
        Mmin = 10.**self._lMmin(a)
        return 0.5 * (1 + erf(np.log(M / Mmin) / self.sigmaLogM))

    def _Ns(self, M, a):
        # Number of satellites
        M0 = 10.**self._lM0(a)
        M1 = 10.**self._lM1(a)
        return np.heaviside(M-M0,1) * ((M - M0) / M1)**self.alpha

    def _fourier_analytic_hod(self, cosmo, k, M, a, mass_def):
        M_use = np.atleast_1d(M)
        k_use = np.atleast_1d(k)

        Nc = self._Nc(M_use, a)
        Ns = self._Ns(M_use, a)
        # NFW profile
        uk = self._fourier_analytic(cosmo, k_use, M_use, a, mass_def) / M_use[:, None]

        prof = Nc[:, None] * (1 + Ns[:, None] * uk)

        if np.ndim(k) == 0:
            prof = np.squeeze(prof, axis=-1)
        if np.ndim(M) == 0:
            prof = np.squeeze(prof, axis=0)
        return prof

    def _fourier_variance(self, cosmo, k, M, a, mass_def):
        # Fourier-space variance of the HOD profile
        M_use = np.atleast_1d(M)
        k_use = np.atleast_1d(k)

        Nc = self._Nc(M_use, a)
        Ns = self._Ns(M_use, a)
        # NFW profile
        uk = self._fourier_analytic(cosmo, k_use, M_use, a, mass_def) / M_use[:, None]

        prof = Ns[:, None] * uk
        prof = Nc[:, None] * (2 * prof + prof**2)

        if np.ndim(k) == 0:
            prof = np.squeeze(prof, axis=-1)
        if np.ndim(M) == 0:
            prof = np.squeeze(prof, axis=0)
        return prof

pg = HaloProfileHOD(cM)
```

## Halo model power spectra

The halo-model power spectrum for two quantities defined by their halo profiles $u$ and $v$ is given by
\begin{equation}
   P_{uv}(k) = \int dM\,n(M)\,\langle u(k|M)\,v(k|M)\rangle + \left[\int dM\,n(M)\,b(M)\,\langle u(k|M)\rangle\right]\,\left[\int dM\,n(M)\,b(M)\,\langle v(k|M)\rangle\right]\,P_{\rm lin}(k).
\end{equation}

The first part, the so-called "one-halo" term, involves the Fourier-space two-point cumulant of the two profiles ($\langle u(k|M)\,v(k|M)\rangle$). This is often approximated by the product of the profile means ($\langle u(k|M)\rangle\,\langle v(k|M)\rangle$), but this is not correct in general (e.g. for the HOD profile). For this reason, the computation of the halo model power spectrum in CCL requires three main ingredients:
- The two profiles to be correlated.
- The linear matter power spectrum. Together with the profile means it controls the 2-halo term.
- The two-point cumulant of the two profiles, controlling the 1-halo term. By default we will set this to the product of the profile means, but we provide ways to define your own two-points.

Let's start by computing the different auto- and cross-correlations between matter and galaxy overdensities at $z=0$.

The first step is creating a `HMCalculator` object that essentially takes care of the mass integrals.

```python
hmc = ccl.halos.HMCalculator(cosmo, nM, bM, hmd_200m)
```

Now let's start computing the different power spectra. Note that, because we are interested in overdensities, we normalize by the means of both quantities. This is done by setting the `normprof` parameters of all method calls to `True`.

### 1. Matter-matter

```python
pk_MM = ccl.halos.halomod_power_spectrum(cosmo, hmc, k_arr, 1.,
                                         pM, normprof1=True)
```

### 2. Galaxy-matter

```python
pk_gM = ccl.halos.halomod_power_spectrum(cosmo, hmc, k_arr, 1.,
                                         pg, prof2=pM,
                                         normprof1=True, normprof2=True)
```

### 3. Galaxy-galaxy

In this case we have a non-trivial two-point cumulant (see Eq. 3.9 of https://arxiv.org/abs/1909.09102). To deal with this we first create a `ProfileCovar` object that implements this.

```python
class Profile2ptHOD(ccl.halos.Profile2pt):
    def fourier_2pt(self, prof, cosmo, k, M, a,
                      prof2=None, mass_def=None):
        return prof._fourier_variance(cosmo, k, M ,a, mass_def)
HOD2pt = Profile2ptHOD()

pk_gg = ccl.halos.halomod_power_spectrum(cosmo, hmc, k_arr, 1.,
                                         pg, prof_2pt=HOD2pt,
                                         normprof1=True)
```

OK, great, now let's plot the power spectra!

```python
plt.figure()
plt.plot(k_arr, pk_MM, 'r-', label='$P_{MM}(k)$')
plt.plot(k_arr, pk_gM, 'b-', label='$P_{gM}(k)$')
plt.plot(k_arr, pk_gg, 'y-', label='$P_{gg}(k)$')
plt.xscale('log')
plt.yscale('log')
plt.legend(loc='lower left', fontsize=15)
plt.ylabel(r'$P(k)\,\,[{\rm Mpc}^3]$', fontsize=15)
plt.xlabel(r'$k\,\,[{\rm Mpc}^{-1}]$', fontsize=15);
```

## 2D power spectra and $C_\ell$s

The `HMCalculator` objects can also provide `Pk2D` power spectrum objects that can then be used to perform Limber integrals and compute angular power spectra. Let's do so for the 3D cross correlations we just looked at.

First, let's create two CCL tracers, a `NumberCountsTracer` that we will associate with the galaxy overdensity, and a `WeakLensing` tracer that we will associate with the matter overdensity.

```python
z_arr = np.linspace(0,0.5,256)
nz_arr = np.exp(-((z_arr - 0.25) / 0.05)**2)

# We give it unit bias because the HOD P(k) is already biased
t_g = ccl.NumberCountsTracer(cosmo, False, (z_arr, nz_arr),
                             bias=(z_arr, np.ones_like(z_arr)))
t_M = ccl.WeakLensingTracer(cosmo, (z_arr, nz_arr))
```

Now let's create `Pk2D` objects for the three cross correlations

```python
pk_MMf = ccl.halos.halomod_Pk2D(cosmo, hmc, pM,
                                normprof1=True,
                                lk_arr=np.log(k_arr), a_arr=a_arr)
pk_gMf = ccl.halos.halomod_Pk2D(cosmo, hmc, pg, prof2=pM,
                                normprof1=True, normprof2=True,
                                lk_arr=np.log(k_arr), a_arr=a_arr)
pk_ggf = ccl.halos.halomod_Pk2D(cosmo, hmc, pg, prof_2pt=HOD2pt,
                                normprof1=True,
                                lk_arr=np.log(k_arr), a_arr=a_arr)
```

Now let's use these to compute $C_\ell$s

```python
l_arr = np.unique(np.geomspace(2, 1000, 64).astype(int))

cl_MM = ccl.angular_cl(cosmo, t_M, t_M, l_arr, p_of_k_a=pk_MMf)
cl_gM = ccl.angular_cl(cosmo, t_g, t_M, l_arr, p_of_k_a=pk_gMf)
cl_gg = ccl.angular_cl(cosmo, t_g, t_g, l_arr, p_of_k_a=pk_ggf)

```

Alright, plotting time!

```python
plt.figure()
lfac = l_arr * (l_arr + 1) / (2 * np.pi)
plt.plot(l_arr, lfac * cl_MM, 'r-', label='$\gamma\gamma$')
plt.plot(l_arr, lfac * cl_gM, 'b-', label='$g\gamma$')
plt.plot(l_arr, lfac * cl_gg, 'y-', label='$gg$')
plt.xscale('log')
plt.yscale('log')
plt.legend(loc='upper left', fontsize=15, ncol=2, frameon=False)
plt.ylabel(r'$\ell(\ell+1)C_\ell/(2\pi)$', fontsize=15)
plt.xlabel(r'$\ell$', fontsize=15);
```

```python

```


# Vigenere cipher cryptoanalysis

This notebook studies the cryptoanalysis of the [Vigenere][wiki] cipher, which is polyalphabetic, namely two occurrences  $a_{1}, a_{2}$ of character $a$ belonging to the plaintext are associated with occurrences $c_{1}, c_{2}$ in the ciphertext, such that $c_{1}\neq c_{2}$ with very high probability.

Our implementation closely follows a class lecture given by Prof. [Orazio Puglisi][puglisi] within Cryptography course at the University of Florence. In order to fully understand the last part, where the concept of *mutual coincidence index* is crucial, we rest on the explanation at pag. 20 of [this notes][notes].

[wiki]:https://en.wikipedia.org/wiki/Vigen%C3%A8re_cipher
[puglisi]:http://web.math.unifi.it/users/puglisi/
[notes]:http://iml.univ-mrs.fr/~ritzenth/cours/crypto.general.pdf

```python
import itertools
from itertools import *
from copy import copy, deepcopy
from heapq import *
from random import *
import matplotlib.pyplot as plt
from collections import Counter
from sympy import *

init_printing()

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 10.0)
```

---

### alphabet

Let $\mathcal{A}$ be our alphabet composed of standard English characters plus the `space` character. It will be sufficient to encode simple and (not so quite) short messages. To define it as a group $\frac{\mathbb{Z}}{n\mathbb{Z}}$ in the Python language we use a `dict` object, which can be reversed because it is a bijection.

```python
def make_alphabet_entry(i):
    alpha = i + ord('a')
    return chr(alpha),i

A = dict(map(make_alphabet_entry, range(26)))
A.update({' ':26})
inverse_A = {v:k for k,v in A.items()}

A
```

    {'a': 0,
     'b': 1,
     'c': 2,
     'd': 3,
     'e': 4,
     'f': 5,
     'g': 6,
     'h': 7,
     'i': 8,
     'j': 9,
     'k': 10,
     'l': 11,
     'm': 12,
     'n': 13,
     'o': 14,
     'p': 15,
     'q': 16,
     'r': 17,
     's': 18,
     't': 19,
     'u': 20,
     'v': 21,
     'w': 22,
     'x': 23,
     'y': 24,
     'z': 25,
     ' ': 26}

### string encoding and the *plaintext*

We define a function `encode` that consumes a string and produces a list of integer in our field. In parallel, function `decode` goes backwards, it consumes a list of integers and return a string. Such functions are useful in order to use the cipher and analyze it using `str` objects instead of the coded version using lists of integers.

```python
def encode(s, alphabet=A): return list(map(lambda i: alphabet[i], s))

def decode(e, inverse_alphabet=inverse_A): return "".join(map(lambda i: inverse_alphabet[i], e))
```

The following plaintext is a prose taken from [here]; before using it we have to swipe out punctuation marks:

[here]:http://www.bartleby.com/209/2.html

```python
def clean_text(text):
    remove_chars = [',', '.', ';', ':', '-', '(', ')', "'", '"']
    for rc in remove_chars: text = text.replace(rc, '')
    text = text.replace('\n', ' ')

    return "".join(filter(lambda c: not c.isdigit(), text))

with open('rest_plain_text.txt', 'r') as f:
    plain_text = clean_text(f.read().lower())

encoded_plain_text = encode(plain_text)

chunk = 500
"{}...".format(plain_text[:chunk])
```

    'the attraction of medieval literature comes perhaps more strongly from some other countries than from england in france and provence in germany and iceland there were literary adventurers more daring and achievements more distinguished it was not in england that the most wonderful things were produced there is nothing in old english that takes hold of the mind with that masterful and subduing power which still belongs to the lyrical stanzas of the troubadours and minnesingers to welsh romance or...'

With the following assert we ensure that function `decode` is the inverse of function `encode`:

```python
assert decode(encode(plain_text)) == plain_text
```

### key

Let $\textbf{k}=(k_{0},\ldots,k_{m-1}) \in \left(\frac{\mathbb{Z}}{n\mathbb{Z}}\right)^{m}$ be a *key* of length $m\in\mathbb{N}$. In order to have a meaningful analysis we build a function `generate_random_key` which generate a random key of random length, keeping it safe...we will uncover it only at the end to check our work:

```python
def generate_random_key(given_key=None, required_length=None, max_length=None, alphabet=A):

    if given_key is not None: return given_key

    if required_length is None and max_length is None: max_length = len(alphabet)

    # the minimum length of the key is 3 to build interesting cases
    length = required_length if required_length else randint(3, max_length)
    key = [0] * length
    # -1 in the following max limit because it is inclusive in the sense of `randint`.
    for i in range(length):
        key[i] = randint(0, len(alphabet)-1)
    return key

#key = encode("ericsmullyan")
secret_key = generate_random_key(required_length=17)
```

### encryption and the *ciphertext*

Now we are in the position to define `encrypt` and `decrypt` functions, both of them consumes an *encoded* message, namely a list of integers, and a key. The encryption work by repeating the key as long as necessary to match the same length of the input $\textbf{x}=(x_{0},\ldots,x_{h})$, where $h > m-1$ otherwise the cipher is a *OneTimePad* which is unbreakable,
$$(\underbrace{x_{0},\ldots,x_{m-1}}_{k_{0},\ldots,k_{m-1}}
\underbrace{x_{m},\ldots,x_{m-1}}_{k_{0},\ldots,k_{m-1}}
\ldots
\underbrace{x_{lm},\ldots,x_{h}}_{k_{0},\ldots,k_{h \mod m}}) =
(y_{0},\ldots,y_{h}) = \textbf{y}$$
truncating the very right block to match plaintext suffix. At last, the ciphertext $\textbf{y}$ is obtained by addition modulo $n$ of corresponding symbols, where $n$ is the length of the alphabet. Decryption follows the same scheme, using modular subtraction.

```python
def encrypt(message, key, alphabet=A):
    n = len(alphabet)
    return [(p+v)%n for p,v in zip(message, cycle(key))]

def decrypt(cipher, key, alphabet=A):
    n = len(alphabet)
    return [(c-v)%n for c,v in zip(cipher, cycle(key))]
```

the following is the *ciphertext* produced by the application of function `encrypt` to the plaintext:

```python
cipher_text = encrypt(encoded_plain_text, secret_key)
'{}...'.format(decode(cipher_text)[:chunk])
```

    ' ksy cnzajpnqxqupgpsbhopilghnvohgcausybygmsgljtrrvbgpbpdjmarvjlnhqlavpnqnwzho djtjtuduwegdbuahhjehaesjlquj xyhiuwktkdiogdabzzlwvljhgjztjnhek xshau ekmvrkugnrgolm cawgjbocluopxjvlvljywavxbgpbpdjyirpjlbkdjjhfvgdezueupxbwexoggwqsshoupomgnqocgzoqzxia pjegxxrkugnrgknhtoaeoyizjcrabda blgpmkxxyjcheczzlwdazjfjjnoyhngrjhwtoesijztjvorydxatizdevrrzj dycrjbwlkwthjjnogpwlcjqqtowyjkjfwhvgcqpot hjibbkhnalaezzideywajstnjzwwjkjwmlvjlujjuj ksykhlqchheucrtihvnmejnpegpwqdsgnvxeqzkhl tespoioxnheqzcihwlhxjjhuwhqqczyl...'

Assuming to know the `secret_key`, we ensure that function `decrypt` is the inverse of function `encrypt`:

```python
assert decode(decrypt(cipher_text, secret_key)) == plain_text
```

### coincidence index

Let $I_{c}$ be the *coincidence index* of a sequence $\alpha$, over an alphabet $A$, defined as
$$ I_{c}(\alpha) = \sum_{i=0}^{n}{\frac{f_{a_{i}}^{(\alpha)}(f_{a_{i}}^{(\alpha)}-1)}{m(m-1)}} $$ where $n$ is the length of the alphabet, $m$ is the length of $\alpha$ and $f_{a_{i}}^{^{(\alpha)}}$ is the frequency of symbol $a_{i}\in A$, namely the number of occurrences of $a_{i}$, in $\alpha$. In other words, $I_{c}$ is the probability to sample two occurrences of the same symbol $a$, forall $a\in A$, from the sequence $\alpha$.

Index $I_{c}$ is invariant to shifting by the same constant $v$. Let $\alpha,\beta$ be two sequences of integers of length $l$ such that $\beta_{i} \equiv_{n} \alpha_{i} + v$, forall $i\in\lbrace0,\ldots,l\rbrace$; moreover, let $q_{\gamma_{i}}^{(\gamma)}$ be the probability to sample *two* occurrences of $\gamma_{i}$ from a sequence $\gamma$, we can state the relations
$$
    I_{c}(\beta) = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}}^{(\beta)}}
                = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}-v}^{(\alpha)}}
                = \sum_{\hat{a}_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{\hat{a}_{i}}^{(\alpha)}}
                = I_{c}(\alpha)
$$
where $\hat{a}_{i}\equiv_{n}a_{i}-v$, proving the invariance of $I_{c}$ when a sequence is produced by another one shifted by a constant $v\in\frac{\mathbb{Z}}{n\mathbb{Z}}$.

```python
def frequencies(lst, alphabet=A, inverse_alphabet=inverse_A):
    """ Produces a `dict` counting occcurrences of each object of the alphabet within the iterable.

    `frequencies` consumes an iterable $lst$ and an alphabet $A$,
    produces a dictionary of entries $(k,v)$ where $k$ is a character
    in $A$ and $v$ is the number of occurrences of $k$ in $lst$
    """
    counter = Counter(lst)
    return {k:counter[v] for k,v in alphabet.items()} # Counter handles the case of missing key returning 0

def length_from_frequencies(freqs):
    """ Returns the length of the original sequence by summation of symbols frequencies. """
    return sum(freqs.values())

def coincidence_index(freqs, alphabet=A):
    """ Produces the I_{c} relative to frequencies matched against an alphabet. """
    denom = length_from_frequencies(freqs)
    if denom in range(2): return None
    def mapper(a):
        v = freqs[a] if a in freqs else 0
        return v*(v-1)
    return sum(map(mapper, alphabet.keys()))/(denom*(denom-1))

def draw_frequencies_histogram(seq, alphabet=A, y_maxlimit=None, normed=None):

    #plaintext_length = len(plain_text)
    #freqs = [plaintext_frequencies[inverse_A[ia]] for ia in sorted(inverse_A.keys())]
    n, bins, patches = plt.hist(seq, len(alphabet),facecolor='green', alpha=0.5)

    plt.xlabel('alphabet symbols')
    plt.ylabel('frequencies' + (', normed respect: {}'.format(str(normed)) if normed else ''))
    #plt.xticks(range(-1, len(alphabet)), sorted(alphabet.keys()))
    if y_maxlimit: plt.ylim([0, y_maxlimit])
    plt.grid(True)
    plt.show()

    return None
```

### frequencies comparison

The following are frequencies of alphabet symbols in the *plaintext*: in the analysis they are used as a `dict` produced by function `frequencies`, moreover it is possible to draw an histogram with the relative function.

```python
plaintext_frequencies = frequencies(encoded_plain_text)
draw_frequencies_histogram(encoded_plain_text, y_maxlimit=1800)
```

```python
plaintext_frequencies
```

    {'a': 571,
     'b': 90,
     'c': 207,
     'd': 272,
     'e': 1001,
     'f': 208,
     'g': 162,
     'h': 422,
     'i': 594,
     'j': 2,
     'k': 46,
     'l': 312,
     'm': 184,
     'n': 624,
     'o': 633,
     'p': 131,
     'q': 8,
     'r': 529,
     's': 490,
     't': 800,
     'u': 158,
     'v': 71,
     'w': 129,
     'x': 13,
     'y': 129,
     'z': 2,
     ' ': 1620}

The following histogram shows frequencies in the *ciphertext*: using the same `y_maxlimit` value, we see that they are spread "uniformly" over symbols.

```python
draw_frequencies_histogram(cipher_text, y_maxlimit=1800)
```

A first approach to encryption finishes computing the coincidence indexes of both *plaintext* and *ciphertext*:

```python
print("coincidence index of *plaintext*: {}\ncoincidence index of *ciphertext*: {}".format(
        coincidence_index(plaintext_frequencies), coincidence_index(frequencies(cipher_text))))
```

    coincidence index of *plaintext*: 0.07720140650073147
    coincidence index of *ciphertext*: 0.03876412502919739

### finding the key *length* by spreading

The following set of functions allows us to probe the key *length* by repeatedly fixing a candidate length $l$, then spreading the *ciphertext* in a matrix with $l$ columns, then computing $I_{c}$ of each column and, finally, report $l$ if the majority of $I_{c}$ scores is greater than a threshold (.06 for English).

```python
def spread(message, block_length):
    return [message[i:i+block_length] for i in range(0, len(message), block_length)]

def col(spreaded, c, join=False, joiner=lambda c: ''.join(decode(c))):
    column = [lst[c] if c < len(lst) else None for lst in spreaded]
    ready = list(filter(lambda i: i is not None, column))
    return joiner(ready) if join else ready

def decode_spreaded(spreaded, join_as_str=False):
    decoded_spread = list(map(decode, spreaded))
    return '\n'.join(decoded_spread) if join_as_str else decoded_spread

def analyze(cipher_text, max_key_length=None):
    res = {}
    # we discard the case where the key length equals the
    # length of the cipher text, since it is the case of
    # OneTimePad cipher, which is unbreakable!
    for d in range(2, len(cipher_text) if max_key_length is None else max_key_length + 1):
        spreaded = spread(cipher_text, d)
        res[d] = []
        for c in range(d):
            ci = coincidence_index(frequencies(col(spreaded, c)))
            if ci: res[d].append(ci)

    return res

def guess_key_length(analysis, threshold=0.06):
    candidates = {}
    for k,v in analysis.items():
        cs = list(filter(lambda i: i > threshold, v))
        if cs and len(cs) > ceiling(k/2): candidates[k] = cs
    return candidates
```

here are $I_{c}$ scores witnesses for the candidate key length after spreading the *ciphertext* by columns:

```python
analysis = analyze(cipher_text, max_key_length=20)
guess = guess_key_length(analysis)
guess
```

```python
probing_key_length = 17
```

The following is the *ciphertext* spread over a matrix with a number of column equals the candidate length found in the previous cell.

```python
spreaded = decode_spreaded(spread(cipher_text, probing_key_length))
print("{}\n...\n{}".format("\n".join(spreaded[:10]), "\n".join(spreaded[-3:])))
```

     ksy cnzajpnqxqup
    gpsbhopilghnvohgc
    ausybygmsgljtrrvb
    gpbpdjmarvjlnhqla
    vpnqnwzho djtjtud
    uwegdbuahhjehaesj
    lquj xyhiuwktkdio
    gdabzzlwvljhgjztj
    nhek xshau ekmvrk
    ugnrgolm cawgjboc
    ...
     kolzmiuezwscckxk
    solyhxuxrpinvslkj
    lvfzxbu

### mutual coincidence index

Once the length $m$ of key $\textbf{k}$ has been "established", we're left with finding actual key symbols $k_{i}$, for $i\in\{0,\ldots,m-1\}$. In order to fullfil this step, we need another object, which resembles the index of mutual coincidence $I_{c}$ but it is more general, in the sense that sampling occurs on two given sequences instead of the same one. Formally, let $I_{mc}(\alpha, \beta)$ be the *index of mutual coincidence* of sequences $\alpha$ and $\beta$, defined as
$$
I_{mc}(\alpha,\beta) = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}}^{(\alpha)}q_{a_{i}}^{(\beta)}}
$$
where $q_{a_{i}}^{(\nu)}$ is the probability to draw an occurrence of symbol $a_{i}$ in sequence $\nu$.

Let $\eta$ and $\gamma$ be two sequences of length $l$, produced by adding shifts $v_{\alpha}$ and $v_{\beta}$ to sequences $\alpha$ and $\beta$, respectively (formally, $\eta_{i} \equiv_{n} \alpha_{i} + v_{\alpha}$ and $\gamma_{i} \equiv_{n} \beta_{i} + v_{\beta}$, forall $i\in\lbrace0,\ldots,l-1\rbrace$). The inequality
$$
    I_{mc}(\eta,\gamma)
                = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}}^{(\eta)}q_{a_{i}}^{(\gamma)}}
                = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}-v_{\alpha}}^{(\alpha)}q_{a_{i}-v_{\beta}}^{(\beta)}}
                = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}}^{(\alpha)}q_{a_{i}+v_{\alpha}-v_{\beta}}^{(\beta)}}
                \neq \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}}^{(\alpha)}q_{a_{i}}^{(\beta)}}
                = I_{mc}(\alpha,\beta)
$$
holds unless factor $v_{\alpha}-v_{\beta}$ in each subscript. We can define an even more general version of $I_{mc}$ as
$$
    I_{mc}(\eta,\gamma,g)
                = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}-g}^{(\eta)}q_{a_{i}}^{(\gamma)}}
                = \sum_{a_{i}\in\frac{\mathbb{Z}}{n\mathbb{Z}}}{q_{a_{i}}^{(\alpha)}q_{a_{i}+v_{\alpha}-v_{\beta}+g}^{(\beta)}}
$$
where sequence $\eta$ is shifted back by $g\in\mathbb{N}$. Therefore we can state the equality with the usual definition according to
$$
    I_{mc}(\eta,\gamma,g) = I_{mc}(\alpha,\beta) \leftrightarrow v_{\beta}-v_{\alpha}=g
$$
proving the invariance of $I_{mc}$ when two sequences are produced by shifting two other ones by constant values $q,w$ respectively, providing that $g = w-q$.

```python
def mutual_coincidence_index(fst_freqs, snd_freqs, offset=0,
                             alphabet=A, inverse_alphabet=inverse_A):
    fst_len = length_from_frequencies(fst_freqs)
    snd_len = length_from_frequencies(snd_freqs)
    n = len(alphabet)
    return sum(fst_freqs[k] * snd_freqs[inverse_alphabet[(v+offset) % n]]
               for k,v in alphabet.items())/(fst_len * snd_len)
```

Previous generalization allows us to set the basis to break the cipher. It works as follows: for each pair $(\eta,\gamma)$ of different columns in the spread-matrix form of the *ciphertext*, fix a $g\in\frac{\mathbb{Z}}{n\mathbb{Z}}$ to compute $I_{mc}(\eta,\gamma,g)$: if such index value is close to $I_{c}(\nu)$, where $\nu$ is a structured sequence using the English language, namely close to $0.065$, then collect equation $k_{i(\beta)}-k_{i(\alpha)}=g$, where function $i(\nu)$ return the zero-based index of column $\nu$ in the matrix spread. Such equations are important since state difference relations over symbols of the key.

```python
def build_offsets_eqs(cipher_text, key_length, indexed_sym, threshold=.06, alphabet=A):

    n = len(alphabet)
    eqs = {c:{} for c in range(key_length)}
    spreaded = spread(cipher_text, key_length)
    for c,a in itertools.product(range(key_length), repeat=2):

        if a == c: continue

        eqs[c][a]=[]

        for g in range(1,n):
            column_freqs = frequencies(col(spreaded, c))
            another_freqs = frequencies(col(spreaded, a))
            mci = mutual_coincidence_index(column_freqs, another_freqs, g)
            if mci > threshold:
                eqs[c][a].append(tuple([Eq(indexed_sym[a]-indexed_sym[c],g,evaluate=True), mci]))

    return eqs
```

```python
k_sym=IndexedBase('k')

eqs_dict = build_offsets_eqs(cipher_text, key_length=probing_key_length, indexed_sym=k_sym)
```

The following cell report the set of difference equation collected respect the *first* column of the spread matrix: we observe that such set fails to instantiate $k_{6}$ because no likely equation gives a relation for it. On the contrary, it can be the case that collecting equations for a pair of columns with indexes $(c,a)$ yields one or more equations, namely $\{k_{a}-k_{c}=g_{u_{0}},\ldots,k_{a}-k_{c}=g_{u_{v}}\}$, for some $v\in\mathbb{N}$: to properly explore all keys space, we have to consider the product of each list of "equally-likely" equations.

```python
eqs_dict[0]
```

The following function implements the last observation, namely it produces the complete keys space where we've to look for the good one.

```python
def explode_key_space(eqs_dict):

    res = {}
    for c, eq_dict in eqs_dict.items():
        eqs_list = []
        for a, eqs in eq_dict.items():
            if not eqs:
                res[c] = [] # no equations in `eqs` causes the product to be empty as well
                break
            eqs_list.append([[a] + list(eq_pair) for eq_pair in eqs])
        else: # if no empty eqs was found then it is meaningful to cross product them
            res[c] = list(itertools.product(*eqs_list))
    return res
```

```python
eqs_dict_pure = explode_key_space(eqs_dict)
eqs_dict_pure
```

```python
print(eqs_dict_pure)
```

    {0: [], 1: [([0, Eq(k[0] - k[1], 4), 0.07966023276727183], [2, Eq(-k[1] + k[2], 11), 0.07773136623701599], [3, Eq(-k[1] + k[3], 22), 0.08098632850682272], [4, Eq(-k[1] + k[4], 23), 0.07777698132388015], [5, Eq(-k[1] + k[5], 7), 0.07751632368465639], [6, Eq(-k[1] + k[6], 18), 0.08160213217948885], [7, Eq(-k[1] + k[7], 5), 0.07663483069049033], [8, Eq(-k[1] + k[8], 24), 0.07533571395930305], [9, Eq(-k[1] + k[9], 4), 0.08045710629908409], [10, Eq(k[10] - k[1], 20), 0.08440341817849473], [11, Eq(k[11] - k[1], 2), 0.07514313132829789], [12, Eq(k[12] - k[1], 26), 0.07765649786853461], [13, Eq(k[13] - k[1], 7), 0.07829952800934842], [14, Eq(k[14] - k[1], 14), 0.07823750987393997], [15, Eq(k[15] - k[1], 3), 0.07887727590236387], [16, Eq(k[16] - k[1], 7), 0.079944640653867])], 2: [([0, Eq(k[0] - k[2], 20), 0.07727521536837441], [1, Eq(k[1] - k[2], 16), 0.07773136623701599], [3, Eq(-k[2] + k[3], 11), 0.07670176856208213], [4, Eq(-k[2] + k[4], 12), 0.07599473471568768], [5, Eq(-k[2] + k[5], 23), 0.07530725019223501], [6, Eq(-k[2] + k[6], 7), 0.07744790105436015], [7, Eq(-k[2] + k[7], 21), 0.07558052238854689], [8, Eq(-k[2] + k[8], 13), 0.07440544192817647], [9, Eq(-k[2] + k[9], 20), 0.07780991115086075], [10, Eq(k[10] - k[2], 9), 0.08161586619750491], [11, Eq(k[11] - k[2], 18), 0.07366122430327521], [12, Eq(k[12] - k[2], 15), 0.07579921791867138], [13, Eq(k[13] - k[2], 23), 0.07608645980898414], [14, Eq(k[14] - k[2], 3), 0.07650100208250371], [15, Eq(k[15] - k[2], 19), 0.07666747181438952], [16, Eq(k[16] - k[2], 23), 0.0756392764115654])], 3: [([0, Eq(k[0] - k[3], 9), 0.07921711478059143], [1, Eq(k[1] - k[3], 5), 0.08098632850682272], [2, Eq(k[2] - k[3], 16), 0.07670176856208213], [4, Eq(-k[3] + k[4], 1), 0.07748374147975341], [5, Eq(-k[3] + k[5], 12), 0.07601428403862946], [6, Eq(-k[3] + k[6], 23), 0.08063769891436093], [7, Eq(-k[3] + k[7], 10), 0.0768502621082249], [8, Eq(-k[3] + k[8], 2), 0.07454906287333285], [9, Eq(-k[3] + k[9], 9), 0.07961822941487522], [10, Eq(k[10] - k[3], 25), 0.08357759774384552], [11, Eq(k[11] - k[3], 7), 0.07412146415025363], [12, Eq(k[12] - k[3], 4), 0.07712771166136792], [13, Eq(k[13] - k[3], 12), 0.07806777602966425], [14, Eq(k[14] - k[3], 19), 0.07739536887734118], [15, Eq(k[15] - k[3], 8), 0.07830932034651818], [16, Eq(k[16] - k[3], 12), 0.07879893720500585])], 4: [([0, Eq(k[0] - k[4], 8), 0.07754238944857876], [1, Eq(k[1] - k[4], 4), 0.07777698132388015], [2, Eq(k[2] - k[4], 15), 0.07599473471568768], [3, Eq(k[3] - k[4], 26), 0.07748374147975341], [5, Eq(-k[4] + k[5], 11), 0.07458066702289877], [6, Eq(-k[4] + k[6], 22), 0.0781581931212449], [7, Eq(-k[4] + k[7], 9), 0.07591999007709833], [8, Eq(-k[4] + k[8], 1), 0.07368407309000463], [9, Eq(-k[4] + k[9], 8), 0.0778360240499801], [10, Eq(k[10] - k[4], 24), 0.08226216045070864], [11, Eq(k[11] - k[4], 6), 0.07356982915635751], [12, Eq(k[12] - k[4], 3), 0.07617785495590185], [13, Eq(k[13] - k[4], 11), 0.07654669965596256], [14, Eq(k[14] - k[4], 18), 0.0761517420567825], [15, Eq(k[15] - k[4], 7), 0.07650100208250371], [16, Eq(k[16] - k[4], 11), 0.07624640131609012])], 5: [], 6: [([0, Eq(k[0] - k[6], 13), 0.07994369794992767], [1, Eq(k[1] - k[6], 9), 0.08160213217948885], [2, Eq(k[2] - k[6], 20), 0.07744790105436015], [3, Eq(k[3] - k[6], 4), 0.08063769891436093], [4, Eq(k[4] - k[6], 5), 0.0781581931212449], [5, Eq(k[5] - k[6], 16), 0.07685164670463579], [7, Eq(-k[6] + k[7], 14), 0.0780579836924945], [8, Eq(-k[6] + k[8], 6), 0.07515618777785757], [9, Eq(-k[6] + k[9], 13), 0.08055502967078162], [10, Eq(k[10] - k[6], 2), 0.08512478701666656], [11, Eq(k[11] - k[6], 11), 0.0747971354149666], [12, Eq(k[12] - k[6], 8), 0.07855086466337209], [13, Eq(k[13] - k[6], 16), 0.07891644525104288], [14, Eq(k[14] - k[6], 23), 0.078315848571298], [15, Eq(k[15] - k[6], 12), 0.07935057219890196], [16, Eq(k[16] - k[6], 16), 0.0800752051494637])], 7: [([0, Eq(k[0] - k[7], 26), 0.07737252009061175], [1, Eq(k[1] - k[7], 22), 0.07663483069049033], [2, Eq(k[2] - k[7], 6), 0.07558052238854689], [3, Eq(k[3] - k[7], 17), 0.0768502621082249], [4, Eq(k[4] - k[7], 18), 0.07591999007709833], [5, Eq(k[5] - k[7], 2), 0.07397457909270733], [6, Eq(k[6] - k[7], 13), 0.0780579836924945], [8, Eq(-k[7] + k[8], 19), 0.07368324673243756], [9, Eq(-k[7] + k[9], 26), 0.07728353318574666], [10, Eq(k[10] - k[7], 15), 0.08146261228413813], [11, Eq(k[11] - k[7], 24), 0.07246026114339342], [12, Eq(k[12] - k[7], 21), 0.07591993695411188], [13, Eq(k[13] - k[7], 2), 0.07610632780591807], [14, Eq(k[14] - k[7], 9), 0.07572700607241775], [15, Eq(k[15] - k[7], 25), 0.07659683004751332], [16, Eq(k[16] - k[7], 2), 0.07579240637129711])], 8: [([0, Eq(k[0] - k[8], 7), 0.07471226849282875], [1, Eq(k[1] - k[8], 3), 0.07533571395930305], [2, Eq(k[2] - k[8], 14), 0.07440544192817647], [3, Eq(k[3] - k[8], 25), 0.07454906287333285], [4, Eq(k[4] - k[8], 26), 0.07368407309000463], [5, Eq(k[5] - k[8], 10), 0.07298881715095214], [6, Eq(k[6] - k[8], 21), 0.07515618777785757], [7, Eq(k[7] - k[8], 8), 0.07368324673243756], [9, Eq(-k[8] + k[9], 7), 0.075082813128456], [10, Eq(k[10] - k[8], 23), 0.07829396780343287], [11, Eq(k[11] - k[8], 5), 0.07217576984326818], [12, Eq(k[12] - k[8], 2), 0.0736799767174936], [13, Eq(k[13] - k[8], 10), 0.07407564852571376], [14, Eq(k[14] - k[8], 17), 0.07396446801761884], [15, Eq(k[15] - k[8], 6), 0.07445824027415805], [16, Eq(k[16] - k[8], 10), 0.07409526861537757])], 9: [], 10: [([0, Eq(k[0] - k[10], 11), 0.08352537194560683], [1, Eq(-k[10] + k[1], 7), 0.08440341817849473], [2, Eq(-k[10] + k[2], 18), 0.08161586619750491], [3, Eq(-k[10] + k[3], 2), 0.08357759774384552], [4, Eq(-k[10] + k[4], 3), 0.08226216045070864], [5, Eq(-k[10] + k[5], 14), 0.08020576964506042], [6, Eq(-k[10] + k[6], 25), 0.08512478701666656], [7, Eq(-k[10] + k[7], 12), 0.08146261228413813], [8, Eq(-k[10] + k[8], 4), 0.07829396780343287], [9, Eq(-k[10] + k[9], 11), 0.0840753542243688], [11, Eq(-k[10] + k[11], 9), 0.07843130843107952], [12, Eq(-k[10] + k[12], 6), 0.0816424631060564], [13, Eq(-k[10] + k[13], 14), 0.08213296534765163], [14, Eq(-k[10] + k[14], 21), 0.0820544849889964], [15, Eq(-k[10] + k[15], 10), 0.08242072666272085], [16, Eq(-k[10] + k[16], 14), 0.08236840642361735])], 11: [([0, Eq(k[0] - k[11], 2), 0.07398110731748715], [1, Eq(-k[11] + k[1], 25), 0.07514313132829789], [2, Eq(-k[11] + k[2], 9), 0.07366122430327521], [3, Eq(-k[11] + k[3], 20), 0.07412146415025363], [4, Eq(-k[11] + k[4], 21), 0.07356982915635751], [5, Eq(-k[11] + k[5], 5), 0.07258080310221242], [6, Eq(-k[11] + k[6], 16), 0.0747971354149666], [7, Eq(-k[11] + k[7], 3), 0.07246026114339342], [8, Eq(-k[11] + k[8], 22), 0.07217576984326818], [9, Eq(-k[11] + k[9], 2), 0.07506973306868013], [10, Eq(k[10] - k[11], 18), 0.07843130843107952], [12, Eq(-k[11] + k[12], 24), 0.07358514628411852], [13, Eq(-k[11] + k[13], 5), 0.07363092649333408], [14, Eq(-k[11] + k[14], 12), 0.07316985438623455], [15, Eq(-k[11] + k[15], 1), 0.0734641557311917], [16, Eq(-k[11] + k[16], 5), 0.07346742574613566])], 12: [([0, Eq(k[0] - k[12], 5), 0.07752919748532781], [1, Eq(-k[12] + k[1], 1), 0.07765649786853461], [2, Eq(-k[12] + k[2], 12), 0.07579921791867138], [3, Eq(-k[12] + k[3], 23), 0.07712771166136792], [4, Eq(-k[12] + k[4], 24), 0.07617785495590185], [5, Eq(-k[12] + k[5], 8), 0.07488526644949439], [6, Eq(-k[12] + k[6], 19), 0.07855086466337209], [7, Eq(-k[12] + k[7], 6), 0.07591993695411188], [8, Eq(-k[12] + k[8], 25), 0.0736799767174936], [9, Eq(-k[12] + k[9], 5), 0.07802582657802747], [10, Eq(k[10] - k[12], 21), 0.0816424631060564], [11, Eq(k[11] - k[12], 3), 0.07358514628411852], [13, Eq(-k[12] + k[13], 8), 0.07700231190056539], [14, Eq(-k[12] + k[14], 15), 0.07575316619196949], [15, Eq(-k[12] + k[15], 4), 0.07659683004751332], [16, Eq(-k[12] + k[16], 8), 0.07626001850828458])], 13: [], 14: [([0, Eq(k[0] - k[14], 17), 0.07743127411363028], [1, Eq(-k[14] + k[1], 13), 0.07823750987393997], [2, Eq(-k[14] + k[2], 24), 0.07650100208250371], [3, Eq(-k[14] + k[3], 8), 0.07739536887734118], [4, Eq(-k[14] + k[4], 9), 0.0761517420567825], [5, Eq(-k[14] + k[5], 20), 0.07529654461062403], [6, Eq(-k[14] + k[6], 4), 0.078315848571298], [7, Eq(-k[14] + k[7], 18), 0.07572700607241775], [8, Eq(-k[14] + k[8], 10), 0.07396446801761884], [9, Eq(-k[14] + k[9], 17), 0.07796369629409207], [10, Eq(k[10] - k[14], 6), 0.0820544849889964], [11, Eq(k[11] - k[14], 15), 0.07316985438623455], [12, Eq(k[12] - k[14], 12), 0.07575316619196949], [13, Eq(k[13] - k[14], 20), 0.0762992586876122], [15, Eq(-k[14] + k[15], 16), 0.07664915028661681], [16, Eq(-k[14] + k[16], 20), 0.07643659931525887])], 15: [([0, Eq(k[0] - k[15], 1), 0.07849211064035357], [1, Eq(-k[15] + k[1], 24), 0.07887727590236387], [2, Eq(-k[15] + k[2], 8), 0.07666747181438952], [3, Eq(-k[15] + k[3], 19), 0.07830932034651818], [4, Eq(-k[15] + k[4], 20), 0.07650100208250371], [5, Eq(-k[15] + k[5], 4), 0.07550544780357878], [6, Eq(-k[15] + k[6], 15), 0.07935057219890196], [7, Eq(-k[15] + k[7], 2), 0.07659683004751332], [8, Eq(-k[15] + k[8], 21), 0.07445824027415805], [9, Eq(-k[15] + k[9], 1), 0.07841168834141572], [10, Eq(k[10] - k[15], 17), 0.08242072666272085], [11, Eq(k[11] - k[15], 26), 0.0734641557311917], [12, Eq(k[12] - k[15], 23), 0.07659683004751332], [13, Eq(k[13] - k[15], 4), 0.07712984248338015], [14, Eq(k[14] - k[15], 11), 0.07664915028661681], [16, Eq(-k[15] + k[16], 4), 0.07720178281214746])], 16: []}

In order to instantiate candidates keys, for each index column $c$ we use the set of equations $\{k_{a}-k_{c}=g_{u}\}$, for some $u\in\mathbb{N}$ and $a\in\{0,\ldots,m-1\}\setminus\{c\}$ as follows: for each equation $k_{a}-k_{c}=g_{u}$, instantiate $k_{c} = s$, for $s\in\frac{\mathbb{Z}}{n\mathbb{Z}}$, and solve it respect $k_{a}$. Therefore for each column index $c$ we have a candidate key if each symbol $k_{i}$ has been instantiated.

```python
def candidate_keys(eqs_dict, indexed_sym, alphabet=A):
    key_length = len(eqs_dict)
    n = len(alphabet)
    candidates=set()
    for c, eqs_tuples in eqs_dict.items():
        for d in range(len(alphabet)):
            for eq_tuple in eqs_tuples:
                key = [indexed_sym[i] for i in range(key_length)]
                key[c] = d
                for a, eq, mci in eq_tuple:
                    subs_eq = eq.subs(indexed_sym[c],d)
                    key[a] = solve(subs_eq, indexed_sym[a])[0]
                    key[a] = key[a] % n

                for k in key:
                    if isinstance(k, Indexed): break
                else:
                    candidates.add(tuple(key)) # `tuple` application to make `key` hashable
    return candidates
```

```python
possible_keys = candidate_keys(eqs_dict_pure, k_sym)
```

The last step to filter out mistaken keys, we use each candidate key to perform a decryption, checking it against frequencies of an arbitrary English prose:

```python
def arbitrary_frequencies(length, filename='rest_plain_text.txt'):
    with open(filename, 'r') as f:
        text = clean_text(f.read().lower())
    text = text[:length]
    return frequencies(encode(text))

def attempt_keys(candidate_keys, cipher_text, threshold=.06, arbitrary_freqs=None):
    sols = set()
    for key in candidate_keys:
        decrypted = decrypt(cipher_text, key)
        freqs = frequencies(decrypted)
        if arbitrary_freqs:
            good = True
            for k,v in arbitrary_freqs.items():
                if 1-abs((freqs[k]-v)/v) < .6: good = False
            if good: sols.add(decode(key))
        else:
            ci = coincidence_index(freqs)
            if ci > threshold: sols.add(decode(key))#sols.add((ci, decode(key)))
    return sols
```

```python
sols = attempt_keys(possible_keys, cipher_text, arbitrary_freqs=arbitrary_frequencies(len(cipher_text)))
```

```python
len(sols), sols
```

    (1, {'hdoz kviahxfckrgk'})

```python
for sol in sols:
    key = sol
    print("key:({})\nplaintext:\n{}\n\n".format(
            key, decode(decrypt(cipher_text, encode(key)))[:chunk]))
```

    key:(hdoz kviahxfckrgk)
    plaintext:
    the attraction of medieval literature comes perhaps more strongly from some other countries than from england in france and provence in germany and iceland there were literary adventurers more daring and achievements more distinguished it was not in england that the most wonderful things were produced there is nothing in old english that takes hold of the mind with that masterful and subduing power which still belongs to the lyrical stanzas of the troubadours and minnesingers to welsh romance or

```python
decode(secret_key)
```

    'hdoz kviahxfckrgk'

**:)**

<!-- -->

<center>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">

</a>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">

</a>
<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">

</a>
<br />
This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.</center>