# Condensación estática en Cálculo Matricial de Estructuras

Análisis de Estructuras. Grado en Ingeniería Civil.

Autor

- Alejandro E. Martínez-Castro, amcastro@ugr.es
- Departamento de Mecánica de Estructuras e Ingeniería Hidráulica. Universidad de Granada.

Condensación estática en Cálculo Matricial de Estructuras by Alejandro Martínez Castro is licensed under a Creative Commons Reconocimiento-NoComercial-CompartirIgual 4.0 Internacional License.

## Concepto

¿Qué es la __condensación estática__?. La palabra "condensar" se introduce en el argot del análisis estructural, para hacer referencia a un recurso bien conocido por el alumno. En efecto, cuando se estudiaron **sistemas de ecuaciones lineales**, una técnica de resolución (método de sustitución) consistía en **despejar una variable en función de las demás**, y sustituir la relación obtenida en el resto de ecuaciones. En consecuencia, el sistema pasaba a tener un orden menor, y sólo aparecían como variables a resolver el resto de variables, excepto la que se había despejado.

### Ejemplo: Condensación en sistemas de ecuaciones lineales

Consideremos el sistema de ecuaciones lineal:

\begin{align}
x - y + z = 2 \\
2\,x + 3 \,y - 2\,z = 2 \\
x - 3\,y + 4 \, z = 1 \\
\end{align}

En la primera, se puede despejar la variable $x$ en función de las variables $y$, $z$. En efecto, es fácil comprobar que

$$x = 2 + y - z$$

Esta ecuación se puede sustituir en las dos ecuaciones del sistema, quedando:

\begin{align}
2\,(2 + y - z) + 3\,y - 2\, z = 2 \\
(2 + y - z) - 3\,y + 4\,z = 1
\end{align}

Simplificando, se obtiene un sistema de 2 ecuaciones con 2 incógnitas:

\begin{align}
5\,y - 4\,z = -2 \\
- 2\,y + 3\,z = -1
\end{align}

Se ha obtenido un sistema, en el cual la variable $x$ no está presente.

Este ejemplo muestra **cuatro aspectos clave**:

+ La variable $x$ se ha despejado, **no se ha "convertido en cero"**.
+ El sistema de ecuaciones, sólo ya en las variables $y$, $z$, tiene unos **coeficientes diferentes** para estas dos variables, y son diferentes que los coeficientes que tenían en el sistema original 3x3.
+ Los **términos de la derecha** son diferentes.
+ Una vez despejados los valores de $y$, $z$ (que se pueden resolver por el método de Cramer, por ejemplo), el valor de $x$ se puede calcular, considerando la ecuación $x = 2 + y - z$ que se planteó inicialmente.

## Relación con los sistemas de ecuaciones de Cálculo Matricial

En este ejemplo, conocido por el alumno en álgebra, sirve de base para plantear qué es esto de la "condensación estática". Sobre este ejemplo, diremos ahora en el argot propio de Cálculo Matricial que la variable $x$ del sistema original **se ha condensado**, y el sistema para las variables $y$ y $z$ es el resultante de haber condensado la variable $x$.

**Condensar por tanto equivale a "despejar un grado de libertad en función de los demás"**.

"Condensar" por tanto es algo que ya sabemos hacer, y que el hecho de llamarlo de otra forma no debe confundirnos.

Por supuesto, se mantienen las mismas conclusiones que las vistas en sistemas de ecuaciones. Al condensar una variable:

+ La variable condensada no se anula. Simplemente, existe una ecuación que permite calcular el grado de libertad que no está en el nuevo sistema de ecuaciones.
+ El nuevo sistema de ecuaciones tiene una matriz de coeficientes del sistema diferente.
+ El nuevo sistema de ecuaciones tiene un vector de la derecha diferente.
+ El grado de libertad condensado puede calcularse, una vez resuelto el sistema de ecuaciones, retomando la ecuación que se empleó para la condensación.

### Ejemplo: condensación en formato matricial.

El ejemplo anterior puede plantearse en formato matricial. En efecto, el sistema:

\begin{equation}
\begin{bmatrix}
1 & -1 & 1 \\
2 & 3 & -2 \\
1 & -3 & 4 \\
\end{bmatrix} \cdot
\begin{pmatrix}
x \\
y \\
z
\end{pmatrix} =
\begin{pmatrix}
2 \\
2 \\
1
\end{pmatrix}
\end{equation}

Puede plantearse por bloques como sigue:

\begin{equation}
\begin{bmatrix}
\mathbf{K}_{xx} & \mathbf{K}_{xu} \\
\mathbf{K}_{ux}& \mathbf{K}_{uu}
\end{bmatrix} =
\begin{pmatrix}
x \\
\mathbf{u}
\end{pmatrix} =
\begin{pmatrix}
2 \\
\mathbf{b}
\end{pmatrix}
\end{equation}

siendo:

\begin{equation}
\mathbf{u} = \begin{pmatrix} y \\ z \end{pmatrix}; \,\,\,\mathbf{b} = \begin{pmatrix} 2 \\ 1 \end{pmatrix}
\end{equation}

y

\begin{equation}
\mathbf{K}_{xx} = 2; \,\,\,\mathbf{K}_{xu} = \begin{pmatrix} -1 & 1 \end{pmatrix}; \,\,\,\mathbf{K}_{ux} = \begin{pmatrix} 2 \\ 1 \end{pmatrix}; \,\,\, \mathbf{K}_{uu} = \begin{pmatrix} 3 & -2 \\ -3 & 4 \end{pmatrix}
\end{equation}

La primera fila permite despejar la $x$, en formato matricial:

\begin{equation}
\mathbf{K}_{xx}\cdot x + \mathbf{K}_{xu}\cdot \mathbf{u} = 2; \Rightarrow x = \mathbf{K}_{xx}^{-1} \cdot \left( 2 - \mathbf{K}_{xu}\cdot \mathbf{u}\right)
\end{equation}

En la segunda fila, multiplicando por bloques, se obtiene:

\begin{align}
\mathbf{K}_{ux}\cdot x + \mathbf{K_{uu}} \cdot \mathbf{u} = \mathbf{b}
\end{align}

Y sustituyendo la expresión de $x$, se obtiene:
\begin{equation}
\mathbf{K_{ux}}\cdot \mathbf{K_{xx}}^{-1} \cdot \left( 2 - \mathbf{K_{xu}}\cdot \mathbf{u}\right) + \mathbf{K_{uu}} \cdot \mathbf{u} = \mathbf{b}
\end{equation}

Operando, y agrupando, se obtiene el siguiente sistema:

\begin{equation}
\left( \mathbf{K}_{uu} - \mathbf{K}_{ux} \cdot \mathbf{K}_{xx}^{-1}\,\cdot \mathbf{K}_{xu} \right) \cdot \mathbf{u} = \mathbf{b} - \mathbf{K}_{ux}\cdot \mathbf{K}_{xx}^{-1}\cdot 2
\end{equation}

Este sistema puede verse como:

\begin{equation}
\mathbf{K}^q \, \mathbf{u}  = \mathbf{F}^q
\end{equation}

Nótese que:

+ El hecho de que no aparezca la $x$ no implica que hayamos hecho $x=0$. ¡ Sería absurdo pensar tal cosa !
+ La matriz $\mathbf{K}^q$ es distinta de la matriz del sistema original, y se obtiene mediante operaciones matriciales. No se obtiene meramente eliminando o haciendo ceros filas o columnas en la matriz original del sistema completo. ¡También sería absurdo pensar tal cosa!.
+ El vector de la derecha, $\mathbf{F}^q$ se ha obtenido mediante operaciones matriciales. Es completamente diferente del vector de la derecha original, y pensar que $\mathbf{F}^q$ procede de quedarse con posiciones del vector de la derecha original, ¡también sería absurdo!.

Estos conceptos, tan bien asentados en álgebra lineal, deben seguir los razonamientos que se hagan en los aspectos de condensación estática en cálculo matricial. Ya que formalmente, aunque se introduce una palabra nueva, "condensación", nada cambia.

Una vez resuelto el nuevo sistema, donde las únicas incógnitas están en el vector $\mathbf{u}$, es decir, $y$, y $z$, la reconstrucción del valor de $x$ es posible, simplemente, retomando la ecuación que permitió despejar $x$ en función de $y$ y de $z$.

\begin{equation}
x = \mathbf{K_{xx}}^{-1} \cdot \left( 2 - \mathbf{K}_{xu}\cdot \mathbf{u}\right)
\end{equation}

La formulación anterior puede verse con números:

```python
import numpy as np

matA = np.matrix([[1,-1,1],
                  [2,3,-2],
                  [1,-3,4]])
matA
```

    matrix([[ 1, -1,  1],
            [ 2,  3, -2],
            [ 1, -3,  4]])

```python
vecB = np.matrix([2,1]).T
vecB
```

    matrix([[2],
            [1]])

Definamos las matrices que intervienen:

```python
Kxx = matA[0,0]
Kxu = matA[0,1:3]
Kux = matA[1:3,0]
Kuu = matA[1:3,1:3]
```

```python
print ("Kxx = ", Kxx)
print ("Kxu = ", Kxu)
print ("Kux = ", Kux)
print ("Kuu = ", Kuu)
```

    Kxx =  1
    Kxu =  [[-1  1]]
    Kux =  [[2]
     [1]]
    Kuu =  [[ 3 -2]
     [-3  4]]

A continuación calculemos la matriz $\mathbf{K}$ y el vector $\mathbf{F}$ del nuevo sistema. Para ello, aplicamos las sumas y productos matriciales anteriores. Puesto que $\mathbf{K_{xx}} = 1$, la inversa se toma explícitamente.

```python
matK = Kuu - 1./Kxx * Kux.dot(Kxu)
print ("K = ", matK)
```

    K =  [[ 5. -4.]
     [-2.  3.]]

```python
vecF = vecB - 1/Kxx *Kux* 2
print ("vecF = ", vecF)
```

    vecF =  [[-2.]
     [-1.]]

Obsérvese que el sistema que se ha obtenido es igual al que inicialmente se planteó para la $y$ y $z$. Se ha obtenido mediante operaciones matriciales.

## Condensación estática

La condensación estática es una técnica que puede extenderse a sistemas de ecuaciones de manera general. Es posible despejar más de un grado de libertad, y sustituirlo en función del resto de grados de libertad. De esta forma, se genera un sistema de ecuaciones reducido.

Vamos a considerar un sistema matricial de la siguiente forma:

\begin{equation}
\begin{pmatrix}
\mathbf{F}_A \\ \mathbf{F}_B \end{pmatrix}
- \begin{pmatrix} \mathbf{\tilde{F}}_A \\ \mathbf{\tilde{F}}_B
\end{pmatrix} =
\begin{pmatrix}
\mathbf{K}_{AA} & \mathbf{K}_{AB} \\
\mathbf{K}_{BA} & \mathbf{K}_{BB}
\end{pmatrix} \cdot
\begin{pmatrix}
\mathbf{u}_A \\
\mathbf{u}_B
\end{pmatrix}
\end{equation}

vamos a condensar los grados de libertad del vector $\mathbf{u}_B$ (grados de libertad *esclavos*) en función de $\mathbf{u}_A$ (grados de libertad *maestros*). Tomamos para ello las filas de estos grados,

\begin{equation}
\mathbf{F}_B - \mathbf{\tilde{F}}_B = \mathbf{K}_{BA}\,\mathbf{u}_A + \mathbf{K}_{BB}\,\mathbf{u}_B
\end{equation}

y despejamos $\mathbf{u}_B$

$$\mathbf{u}_B = \mathbf{K}_{BB}^{-1}\,\mathbf{F}_B - \mathbf{K}_{BB}^{-1}\,\mathbf{\tilde{F}}_B - \mathbf{K}_{BB}^{-1}\,\mathbf{K}_{BA} \,\mathbf{u}_A $$

En la fila de los grados de libertad que no vamos a condensar, $\mathbf{u}_A$, quedará:

\begin{equation}
\mathbf{F}_A - \mathbf{\tilde{F}}_A = \mathbf{K}_{AA}\,\mathbf{u}_A + \mathbf{K}_{AB}\,\mathbf{u}_B
\end{equation}

y sustituyendo la expresión de $\mathbf{u}_B quedará:

\begin{equation}
\mathbf{F}_A - \mathbf{\tilde{F}}_A = \mathbf{K}_{AA}\,\mathbf{u}_A + \mathbf{K}_{AB}\,\left( \mathbf{K}_{BB}^{-1}\,\mathbf{F}_B - \mathbf{K}_{BB}^{-1}\,\mathbf{\tilde{F}}_B - \mathbf{K}_{BB}^{-1}\,\mathbf{K}_{BA} \,\mathbf{u}_A \right)
\end{equation}

Agrupando términos,

$$\left( \mathbf{F}_A - \mathbf{K}_{AB}\,\mathbf{K}_{BB}^{-1}\,\mathbf{F}_B \right) - \left( \mathbf{\tilde{F}}_A - \mathbf{K}_{AB}\,\mathbf{K}_{BB}^{-1}\,\mathbf{\tilde{F}}_B \right) = \left( \mathbf{K}_{AA}-\mathbf{K}_{AB}\,\mathbf{K}_{BB}^{-1}\,\mathbf{K}_{BA} \right)\,\mathbf{u}_A $$

Esta última ecuación puede escribirse como:

$$ \mathbf{F}^q - \mathbf{\tilde{F}}^q = \mathbf{K}^q \,\mathbf{u}^q$$

Donde:

+ $\mathbf{F}^q = \mathbf{F}_A - \mathbf{K}_{AB}\,\mathbf{K}_{BB}^{-1}\,\mathbf{F}_B$
+ $\mathbf{\tilde{F}}^q = \mathbf{\tilde{F}}_A - \mathbf{K}_{AB}\,\mathbf{K}_{BB}^{-1}\,\mathbf{\tilde{F}}_B$
+ $\mathbf{K}^q = \mathbf{K}_{AA} - \mathbf{K}_{AB} \,\mathbf{K}_{BB}^{-1}\,\mathbf{K}_{BA}$
+ $\mathbf{u}^q = \mathbf{u}$

Este sistema equivalente tiene como grados de libertad, los mismos $\mathbf{u}_A$ que se mantienen del sistema original (grados maestros). Pero cambian tanto la matriz de rigidez como los vectores de fuerzas. En dicha matriz y vectores se "oculta" el efecto de los grados de libertad que, aunque no aparecen explícitamente, sí generan un efecto en el sistema de ecuaciones.

## Condensación estática del giro en barra con extremo articulado

En este apartado se va a calcular la matriz de rigidez de una barra en configuración empotrado-articulado, en coordenadas locales. Nótese que para que se presente esta configuración, la barra debe tener en la parte de la derecha una rótula, que suponga una condición de contorno de momento flector nulo en dicho extremo, y en la izquierda, los 3 grados de libertad.

El término "empotrado" o "apoyado" no supone:
+ Que en la parte izquierda exista un empotramiento que bloquee los 3 grados de libertad.
+ Que en la parte de la derecha exista un extremo apoyado, con un apoyo simple.

Eso son condiciones de contorno, las cuales se aplican sobre los grados de libertad de la barra condensada.

La condensación se realizará sobre el grado de libertad $\theta_2$.

Se considerará que sobre la barra existe una carga uniforme, hacia abajo, de valor $q$ (carga distribuida).

En esta ocasión se utilizará el paquete SymPy.

```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.9 (Python 3.9.7-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.9/

En primer lugar se definen los símbolos que se van a emplear

```python
EA,EI, L, q = symbols('EA EI L q')
```

```python
matrig = Matrix([[EA/L, 0, 0, -EA/L, 0, 0],
                [0, 12*EI/L**3, 6*EI/L**2, 0, -12*EI/L**3, 6*EI/L**2],
                [0, 6*EI/L**2, 4*EI/L, 0, -6*EI/L**2, 2*EI/L],
                [-EA/L, 0, 0, EA/L, 0, 0],
                [0, -12*EI/L**3, -6*EI/L**2, 0, 12*EI/L**3, -6*EI/L**2],
                [0, 6*EI/L**2, 2*EI/L, 0, -6*EI/L**2, 4*EI/L]])
```

```python
matrig
```

    C:\Users\34657\anaconda3\lib\site-packages\IPython\lib\latextools.py:126: MatplotlibDeprecationWarning:
    The to_png function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    C:\Users\34657\anaconda3\lib\site-packages\IPython\lib\latextools.py:126: MatplotlibDeprecationWarning:
    The to_rgba function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    C:\Users\34657\anaconda3\lib\site-packages\IPython\lib\latextools.py:126: MatplotlibDeprecationWarning:
    The to_mask function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    C:\Users\34657\anaconda3\lib\site-packages\IPython\lib\latextools.py:126: MatplotlibDeprecationWarning:
    The MathtextBackendBitmap class was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)

$\displaystyle \left[\begin{matrix}\frac{EA}{L} & 0 & 0 & - \frac{EA}{L} & 0 & 0\\0 & \frac{12 EI}{L^{3}} & \frac{6 EI}{L^{2}} & 0 & - \frac{12 EI}{L^{3}} & \frac{6 EI}{L^{2}}\\0 & \frac{6 EI}{L^{2}} & \frac{4 EI}{L} & 0 & - \frac{6 EI}{L^{2}} & \frac{2 EI}{L}\\- \frac{EA}{L} & 0 & 0 & \frac{EA}{L} & 0 & 0\\0 & - \frac{12 EI}{L^{3}} & - \frac{6 EI}{L^{2}} & 0 & \frac{12 EI}{L^{3}} & - \frac{6 EI}{L^{2}}\\0 & \frac{6 EI}{L^{2}} & \frac{2 EI}{L} & 0 & - \frac{6 EI}{L^{2}} & \frac{4 EI}{L}\end{matrix}\right]$

Verificamos que es simétrica: al restar la matriz a su traspuesta debe salir la matriz nula

```python
matrig.T - matrig
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right]$

El vector de fuerzas de empotramiento es:

```python
Femp = Matrix([[0, q*L/2, q*L**2/12, 0, q*L/2, -q*L**2/12]]).T
Femp
```

$\displaystyle \left[\begin{matrix}0\\\frac{L q}{2}\\\frac{L^{2} q}{12}\\0\\\frac{L q}{2}\\- \frac{L^{2} q}{12}\end{matrix}\right]$

A continuación se definen las submatrices necesarias para las operaciones matriciales. Puesto que se condensa el giro en el extremo 2, se ha de considerar la fila sexta.

```python
Kbb = matrig[5,5]
```

```python
Kbb
```

```python
Kaa = matrig[0:5,0:5]
Kaa
```

$\displaystyle \left[\begin{matrix}\frac{EA}{L} & 0 & 0 & - \frac{EA}{L} & 0\\0 & \frac{12 EI}{L^{3}} & \frac{6 EI}{L^{2}} & 0 & - \frac{12 EI}{L^{3}}\\0 & \frac{6 EI}{L^{2}} & \frac{4 EI}{L} & 0 & - \frac{6 EI}{L^{2}}\\- \frac{EA}{L} & 0 & 0 & \frac{EA}{L} & 0\\0 & - \frac{12 EI}{L^{3}} & - \frac{6 EI}{L^{2}} & 0 & \frac{12 EI}{L^{3}}\end{matrix}\right]$

```python
Kab = matrig[0:5,5]
Kab
```

$\displaystyle \left[\begin{matrix}0\\\frac{6 EI}{L^{2}}\\\frac{2 EI}{L}\\0\\- \frac{6 EI}{L^{2}}\end{matrix}\right]$

```python
Kba = matrig[5,0:5]
Kba
```

$\displaystyle \left[\begin{matrix}0 & \frac{6 EI}{L^{2}} & \frac{2 EI}{L} & 0 & - \frac{6 EI}{L^{2}}\end{matrix}\right]$

### Cálculo de la matriz de rigidez condensada

En esta ocasión se tiene en cuenta que la matriz $\mathbf{K_{BB}}$ es un escalar.

```python
Krig_condensada_giro = Kaa-Kab.multiply(Kba)/Kbb
```

```python
Krig_condensada_giro
```

$\displaystyle \left[\begin{matrix}\frac{EA}{L} & 0 & 0 & - \frac{EA}{L} & 0\\0 & \frac{3 EI}{L^{3}} & \frac{3 EI}{L^{2}} & 0 & - \frac{3 EI}{L^{3}}\\0 & \frac{3 EI}{L^{2}} & \frac{3 EI}{L} & 0 & - \frac{3 EI}{L^{2}}\\- \frac{EA}{L} & 0 & 0 & \frac{EA}{L} & 0\\0 & - \frac{3 EI}{L^{3}} & - \frac{3 EI}{L^{2}} & 0 & \frac{3 EI}{L^{3}}\end{matrix}\right]$

Se observa que esta matriz coincide con la que se ha definido en la asignatura para la barra empotrada-articulada. Esta matriz aparece en el apartado B.2. del libro de la asignatura, "Cálculo Matricial de Estructuras", de E. Puertas, A. Martínez, F. Ávila y R. Gallego, de la Universidad de Granada".

### Cálculo del vector de fuerzas de empotramiento

El nuevo vector de fuerzas de empotramiento ha de obtenerse mediante operaciones matriciales:

```python
Fa = Femp[0:5,0]
```

```python
Fa
```

$\displaystyle \left[\begin{matrix}0\\\frac{L q}{2}\\\frac{L^{2} q}{12}\\0\\\frac{L q}{2}\end{matrix}\right]$

```python
Fb = Femp[5,0]
Fb
```

```python
Femp_condensada = Fa - Kab.multiply(Fb)/Kbb
```

```python
Femp_condensada
```

$\displaystyle \left[\begin{matrix}0\\\frac{5 L q}{8}\\\frac{L^{2} q}{8}\\0\\\frac{3 L q}{8}\end{matrix}\right]$

Que resulta una expresión idéntica a la obtenida para las fuerzas de empotramiento.

En general, esta matriz de rigidez y vector de fuerzas de empotramiento deben definirse en el **sistema de coordenadas globales**. Para la matriz de rigidez, existe una matriz de cambio de base a coordenadas globales para cada caso de barra condensada. (Ver libro de la asignatura).

##  Condensación estática del desplazamiento vertical en barra con deslizadera en el extremo

En esta ocasión calcularemos la expresión simbólica de la matriz de rigidez de una barra en configuración empotrada-deslizadera, en coordenadas locales. El grado de libertad que se condensa es el desplazamiento vertical en el nudo $2$. Esta condensación es posible porque existe un dato en la fuerza vertical en el nudo "2", que es una fuerza vertical conocida en dicho nudo, de valor nulo (pues existe una deslizadera vertical).

Partiremos de la matriz de rigidez de la barra completa:

```python
matrig
```

$\displaystyle \left[\begin{matrix}\frac{EA}{L} & 0 & 0 & - \frac{EA}{L} & 0 & 0\\0 & \frac{12 EI}{L^{3}} & \frac{6 EI}{L^{2}} & 0 & - \frac{12 EI}{L^{3}} & \frac{6 EI}{L^{2}}\\0 & \frac{6 EI}{L^{2}} & \frac{4 EI}{L} & 0 & - \frac{6 EI}{L^{2}} & \frac{2 EI}{L}\\- \frac{EA}{L} & 0 & 0 & \frac{EA}{L} & 0 & 0\\0 & - \frac{12 EI}{L^{3}} & - \frac{6 EI}{L^{2}} & 0 & \frac{12 EI}{L^{3}} & - \frac{6 EI}{L^{2}}\\0 & \frac{6 EI}{L^{2}} & \frac{2 EI}{L} & 0 & - \frac{6 EI}{L^{2}} & \frac{4 EI}{L}\end{matrix}\right]$

Definiremos las matrices $\mathbf{K}_{AA}$,$\mathbf{K}_{AB}$ y $\mathbf{K}_{BB}$. En esta ocasión, la fila 5 es la fila que permite despejar el desplazamiento vertical en 2. Por tanto:

```python
Kbb = matrig[4,4]
Kbb
```

A continuación vamos a extraer de la matriz de rigidez las filas y columnas que forman la matriz $\mathbf{K}_{BB}$. Para eso, vamos a usar funciones que borran la fila y columna 5 (para Python es la 4, pues empieza a indexar en 0 en vez de en 1)

```python
Kaa = matrig.copy() # Creamos una copia para que no se borre la original
```

```python
Kaa.col_del(4)
Kaa.row_del(4)
Kaa
```

$\displaystyle \left[\begin{matrix}\frac{EA}{L} & 0 & 0 & - \frac{EA}{L} & 0\\0 & \frac{12 EI}{L^{3}} & \frac{6 EI}{L^{2}} & 0 & \frac{6 EI}{L^{2}}\\0 & \frac{6 EI}{L^{2}} & \frac{4 EI}{L} & 0 & \frac{2 EI}{L}\\- \frac{EA}{L} & 0 & 0 & \frac{EA}{L} & 0\\0 & \frac{6 EI}{L^{2}} & \frac{2 EI}{L} & 0 & \frac{4 EI}{L}\end{matrix}\right]$

```python
Kba = matrig[4,:]
Kba.col_del(4)
Kab = Kba.T
Kab
```

$\displaystyle \left[\begin{matrix}0\\- \frac{12 EI}{L^{3}}\\- \frac{6 EI}{L^{2}}\\0\\- \frac{6 EI}{L^{2}}\end{matrix}\right]$

```python
Krig_condensada_desp = Kaa-Kab.multiply(Kba)/Kbb
Krig_condensada_desp
```

$\displaystyle \left[\begin{matrix}\frac{EA}{L} & 0 & 0 & - \frac{EA}{L} & 0\\0 & 0 & 0 & 0 & 0\\0 & 0 & \frac{EI}{L} & 0 & - \frac{EI}{L}\\- \frac{EA}{L} & 0 & 0 & \frac{EA}{L} & 0\\0 & 0 & - \frac{EI}{L} & 0 & \frac{EI}{L}\end{matrix}\right]$

Esta matriz coincide con la mostrada en el apartado B.4. del libro de la asignatura. Puede expresarse en el sistema de coordenadas global, mediante las matrices de cambio de base (ver expresión final en B.4).

# Conclusión

En este cuaderno se muestra que el concepto de "condensación" sólo aporta una palabra nueva, pero ya era un recurso  conocido: el método de sustitución en la resolución de sistemas de ecuaciones lineales.

Se insiste una vez más en que **condensar no equivale a "hacer cero" ningún grado de libertad**. El hecho de que se eliminen esos grados de libertad en el sistema de ecuaciones final no debe llevar a pensar al alumno que dichos grados son nulos. Al contrario, existe una ecuación que permite reconstruir dichos grados de libertad.

Aplicaremos de forma directa la condensación estática a una barra en cálculo matricial de tipo P2D, reduciendo de 6x6, a una matriz 5x5. Y lo haremos con dos configuraciones:

+ Barra empotrada-articulada (o viceversa, articulada-empotrada). Se condensa el giro en la articulación.
+ Barra empotrada-deslizadera (o viceversa, deslizadera-empotrada). Se condensa el desplazamiento vertical en la deslizadera.

Al condensar, debe quedar claro que:

+ El grado de libertad que se condensa NO APARECE en el vector de grados de libertad.
+ La matriz de rigidez de la barra condensada es DIFERENTE. Debe tomarse de forma adecuada, y es una matriz 5x5 distinta y escrita en el libro de la asignatura.
+ El vector de fuerzas de empotramiento es DIFERENTE al que se obtenía en la barra de 6 grados de libertad. Debe seleccionarse el caso de apoyos adecuado, en el prontuario de casos de fuerzas de empotramiento. Debe ser coherente con el tipo de barra condensada.


```python
import numpy as np
import sympy as sy
from sympy import I, exp
from sympy.utilities.codegen import codegen
```

```python
z = sy.symbols("z", real=False)
r1,r2,r3, r4, r5, s0,s1, s2,s3, s4, s5 = sy.symbols("r1,r2,r3,r4, r5, s0,s1,s2,s3,s4, s5", real=True)
a, h = sy.symbols("a, h", real=True)
```

```python
pd1 = 0.4
pd2 = pd1
hh = 0.2
aa = 0
# Will need characteristic polynomial of order 9, since in the left hand side the order of
# the polynomial will be n_A + n_R = 5+4 = 9. R and S will then have 4+5=9 unknowns
Ac = sy.poly(z**3*(z - pd1)*(z-pd2), z )
#Ao = sy.poly(((z-a)**4).subs(a, aa), z )
Ao = sy.poly(((z-a)**4), z )
Acl = Ac*Ao

# Assume anti-aliasing filter corresponding to a delay of two sample periods + extra delay of 1 sample period
A = sy.poly( (z**3*(z-1)*(z-exp(h))).subs(h,hh), z )
B = sy.poly( ((-1+exp(h)-h)*z + 1-(1-h)*exp(h)).subs(h,hh), z)

print A
print B
print Ao
print Ac
```

    Poly(1.0*z**5 - 2.22140275816017*z**4 + 1.22140275816017*z**3, z, domain='RR')
    Poly(0.0214027581601699*z + 0.022877793471864, z, domain='RR')
    Poly(z**4 - 4*a*z**3 + 6*a**2*z**2 - 4*a**3*z + a**4, z, domain='ZZ[a]')
    Poly(1.0*z**5 - 0.8*z**4 + 0.16*z**3, z, domain='RR')

```python
R = sy.poly(z**4 + r1*z**3 + r2*z**2 + r3*z**1 + r4, z)
S = sy.poly(s0*z**4 + s1*z**3 + s2*z**2 + s3*z**1 + s4, z)
print A*R+B*S
print Acl

dioph=(A*R+B*S-Acl).all_coeffs()
```

    Poly(1.0*z**9 + (1.0*r1 - 2.22140275816017)*z**8 + (-2.22140275816017*r1 + 1.0*r2 + 1.22140275816017)*z**7 + (1.22140275816017*r1 - 2.22140275816017*r2 + 1.0*r3)*z**6 + (1.22140275816017*r2 - 2.22140275816017*r3 + 1.0*r4 + 0.0214027581601699*s0)*z**5 + (1.22140275816017*r3 - 2.22140275816017*r4 + 0.022877793471864*s0 + 0.0214027581601699*s1)*z**4 + (1.22140275816017*r4 + 0.022877793471864*s1 + 0.0214027581601699*s2)*z**3 + (0.022877793471864*s2 + 0.0214027581601699*s3)*z**2 + (0.022877793471864*s3 + 0.0214027581601699*s4)*z + 0.022877793471864*s4, z, domain='RR[r1,r2,r3,r4,s0,s1,s2,s3,s4]')
    Poly(1.0*z**9 + (-4.0*a - 0.8)*z**8 + (6.0*a**2 + 3.2*a + 0.16)*z**7 + (-4.0*a**3 - 4.8*a**2 - 0.64*a)*z**6 + (1.0*a**4 + 3.2*a**3 + 0.96*a**2)*z**5 + (-0.8*a**4 - 0.64*a**3)*z**4 + 0.16*a**4*z**3, z, domain='RR[a]')

```python
sol=sy.solve(dioph, (r1,r2, r3, r4, r5, s0,s1,s2, s3, s4, s5))
print sol
```

    {s3: 0.0, s4: 0.0, r4: 0.455360670818637*a**4 - 2.32869913240506*a**3 + 5.38239533770142*a**2 - 5.12669902254084*a + 1.75143708540282, s0: 25.447156161159*a**4 - 156.844826966403*a**3 + 336.13593650405*a**2 - 306.22482671798*a + 101.635942011644, r3: -4.0*a**3 + 8.52841654896102*a**2 - 8.38442099709322*a + 2.92018873285642, s2: 0.0, s1: -17.3171761421304*a**4 + 124.324906890288*a**3 - 287.356056389879*a**2 + 273.704906641866*a - 93.5059619926153, r1: -4.0*a + 1.42140275816017, r2: 6.0*a**2 - 5.68561103264068*a + 2.0961052492733}

```python
print len(dioph)
print dioph
```

    11
    [1.00000000000000, 1.0*r1 - 2.22140275816017, -2.22140275816017*r1 + 1.0*r2 + 0.22140275816017, 1.22140275816017*r1 - 2.22140275816017*r2 + 1.0*r3 + 0.8, 1.22140275816017*r2 - 2.22140275816017*r3 + 1.0*r4 + 0.0214027581601699*s0 - 0.16, 1.22140275816017*r3 - 2.22140275816017*r4 + 1.0*r5 + 0.022877793471864*s0 + 0.0214027581601699*s1, 1.22140275816017*r4 - 2.22140275816017*r5 + 0.022877793471864*s1 + 0.0214027581601699*s2, 1.22140275816017*r5 + 0.022877793471864*s2 + 0.0214027581601699*s3, 0.022877793471864*s3 + 0.0214027581601699*s4, 0.022877793471864*s4 + 0.0214027581601699*s5, 0.022877793471864*s5]

```python
# Reorganize solution expression for matlab code generation
sol_expr = ('RST_example_aliasing', [sol[r1], sol[r2], sol[r3], sol[r4], sol[s0], sol[s1], sol[s2], sol[s3], sol[s4], Ac.subs(z, 1)/B.subs(z,1), a])
```

```python
# Export to matlab code
[(m_name, m_code)] = codegen(sol_expr, 'octave')
```

```python
print m_code
with open("/home/kjartan/Dropbox/undervisning/tec/MR2007/matlab/PS4_ship_heading_RST_example_aliasing_3h.m", "w") as text_file:
    text_file.write(m_code)
```

    function [out1, out2, out3, out4, out5, out6, out7, out8, out9, out10, out11] = RST_example_aliasing(a)
      %RST_EXAMPLE_ALIASING  Autogenerated by sympy
      %   Code generated with sympy 0.7.6
      %
      %   See http://www.sympy.org/ for more information.
      %
      %   This file is part of 'project'

      out1 = -4.0*a + 1.42140275816017;
      out2 = 6.0*a.^2 - 5.68561103264068*a + 2.0961052492733;
      out3 = -4.0*a.^3 + 8.52841654896102*a.^2 - 8.38442099709322*a + 2.92018873285642;
      out4 = 0.455360670818637*a.^4 - 2.32869913240506*a.^3 + 5.38239533770142*a.^2 - 5.12669902254084*a + 1.75143708540282;
      out5 = 25.447156161159*a.^4 - 156.844826966403*a.^3 + 336.13593650405*a.^2 - 306.22482671798*a + 101.635942011644;
      out6 = -17.3171761421304*a.^4 + 124.324906890288*a.^3 - 287.356056389879*a.^2 + 273.704906641866*a - 93.5059619926153;
      out7 = 0.0;
      out8 = 0.0;
      out9 = 0.0;
      out10 = 8.1299800190286;
      out11 = a;

    end

```python
sy.expand((z-a)**3)
```

    -a**3 + 3*a**2*z - 3*a*z**2 + z**3

```python

```


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Discussion: Week 7

## Exercise: The Baseline RBC Model without Labor

The equilibrium conditions for the RBC model without labor are:

\begin{align}
\frac{1}{C_t} & = \beta E_t \left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1} +1-\delta }{C_{t+1}}\right]\\
K_{t+1} & = I_t + (1-\delta) K_t\\
Y_t & = A_t K_t^{\alpha}\\
Y_t & = C_t + I_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$.

The objective is use `linearsolve` to create a stochastic simulation of the RBC model. USe the following parameter values for the simulation:

| $\rho$ | $\sigma$ | $\beta$ | $\alpha$ | $\delta $ |
|--------|----------|---------|----------|-----------|
| 0.75   | 0.006    | 0.99    | 0.35     |  0.025    |

### Model Preparation

Before proceding, let's recast the model in the form required for `linearsolve`. Write the model with all variables moved to the lefthand side of the equations and dropping the expecations operator $E_t$ and the exogenous shock $\epsilon_{t+1}$:

\begin{align}
0 & = \beta\left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1} +1-\delta }{C_{t+1}}\right] - \frac{1}{C_t}\\
0 & = A_t K_t^{\alpha} - Y_t\\
0 & = I_t + (1-\delta) K_t - K_{t+1}\\
0 & = C_t + I_t - Y_t\\
0 & = \rho \log A_t - \log A_{t+1}
\end{align}

Remember, capital and TFP are called *state variables* because they're $t+1$ values are predetermined. Output, consumption, and investment are called a *costate* or *control* variables. Note that the model as 5 equations in 5 endogenous variables.

### Initialization, Approximation, and Solution

The next several cells initialize the model in `linearsolve` and then approximate and solve it.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series. CELL PROVIDED
parameters = pd.Series()
parameters['rho'] = .75
parameters['sigma_squared'] = 0.006**2
parameters['beta'] = 0.99
parameters['alpha'] = 0.35
parameters['delta'] = 0.025

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first
var_names = ['a','k','y','c','i']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable.
shock_names = ['e_a','e_k']

# Define a function that evaluates the equilibrium conditions of the model solved for zero.
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Euler equation
    euler_equation = p.beta*(p.alpha*fwd.a*fwd.k**(p.alpha-1)+1-p.delta)/fwd.c - 1/cur.c

    # Production function
    production_function = cur.a*cur.k**p.alpha - cur.y

    # Capital evolution
    capital_evolution = cur.i + (1 - p.delta)*cur.k - fwd.k

    # Market clearing
    market_clearing = cur.c+cur.i - cur.y

    # Exogenous tfp
    tfp_process = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        euler_equation,
        production_function,
        capital_evolution,
        market_clearing,
        tfp_process
        ])

# Initialize the model into a variable named 'rbc_model'
rbc_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters=parameters)

# Compute the steady state numerically using .compute_ss() method of rbc_model
guess = [1,4,1,1,1]
rbc_model.compute_ss(guess)

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of rbc_model
rbc_model.approximate_and_solve()
```

### Stochastic Simulation

Compute a 201 period stochastic simulation of the model. Set the seed for the simulation to 126.

Recall that `linearsolve` handles the model *as if* there is an exogenous shock for every state variable. In this case, that means two exogenous shocks $\epsilon_t^a$ and $\epsilon^k_t$. For the stochastic simulation, we need to specify the *covariance matrix* for the two shocks. The covariance matrix has the variance of each shock in the diagonal elements and covariances on the off-diagonal elements. Since there is no shock to capital in our model,

\begin{align}
\text{Covariance matrix} & = \left[\begin{array}{cc}\sigma^2 & 0\\ 0 & 0\end{array}\right]
\end{align}

```python
# Compute the stochastic simulation using the .stoch_sim() method of rbc_model

```

```python
# Print the first 5 rows of rbc_model.simulated

```

```python
# Print the first 5 rows of simulated output

```

## Exercise: Analyze Simulation Results

On a single axis, plot the simulated values for output, consumption and investment.

```python
# Plot simulated results

```

**Questions**

1. What is the exogenous *cause* of the fluctuations in output, consumption and investment in your simulation?
2. Looking at the plot of your simulated data, which quantity — output, consumption or investment — has the smallest fluctuations from the steady state? What feature of the model leads to this?
3. Which quantity — output, consumption or investment — has the largest fluctuations from the steady state? What feature of the model leads to this?

**Answers**

1.

2.

3.


# Introduction

Understanding the behavior of neural networks and why they generalize has been a central pursuit of the theoretical deep learning community.
Recently, [Valle-Perez et al. (2019)](http://arxiv.org/abs/1805.08522) observed that neural networks have a certain "simplicity bias" and proposed this as a solution to the generalization question.
One of the ways with which they argued that this bias exists is the following experiment:
they drew a large sample of boolean functions $\{\pm1\}^7 \to \{\pm 1\}$ by randomly initializing neural networks and thresholding the output.
They observed that there is a bias toward some "simple" functions which get sampled disproportionately more often.
However, their experiments were only done for 2 layer relu networks.
Can one expect this "simplicity bias" to hold universally, for any architecture?

# A Quick Replication of Valle-Perez et al.'s Probability-vs-Rank Experiment

```python
import numpy as np
import scipy as sp
from scipy.special import erf as erf
from collections import OrderedDict as OD
import matplotlib.pyplot as plt
from itertools import product
import seaborn as sns
sns.set()
from mpl_toolkits.axes_grid1 import ImageGrid
def tight_layout(plt):
    plt.tight_layout(rect=[0, 0.03, 1, 0.95])
# our functions for sampling boolean functions belong here
from sample_boolean import *
np.random.seed(0)
_ = torch.manual_seed(0)
```

We sample $10^4$ random neural networks on the 7-dimensional boolean cube $\{\pm 1\}^7$ and threshold the results to get $10^4$ boolean functions.
Here, we sample 2 layer relu networks with 40 neurons each, with weights $W_{ij} \sim \mathcal N(0, \sigma_w^2/40) = \mathcal N(0, 2/40)$ and biases $b_i \sim \mathcal N(0, \sigma_b^2) = \mathcal N(0, 2)$, following [Valle-Perez et al. (2019)](http://arxiv.org/abs/1805.08522).

```python
WIDTHSPEC = (7, 40, 40, 1)
nsamples = 10**4
funcounters = {'relu': {}}
funfreq = {'relu': {}}
# vb = sigma_b^2
vb = 2
# vw = \sigma_w^2
for vw in [2]:
    # `funcounters` holds a dictionary (more precisely, a `Counter` object)
    # of boolean function (as a string of length 2^7 = 128) to its frequency
    funcounters['relu'][vw] = sample_boolean_fun(MyNet(nn.ReLU, WIDTHSPEC), vw, vb, nsamples, outformat='counter')
    # `funfreq` just has a list of frequencies
    funfreq['relu'][vw] = OD(funcounters['relu'][vw].most_common()).values()
```

Sort the functions according to frequency and then plot its rank in this order versus its empirical probability.

```python
plt.plot(np.array(list(funfreq['relu'][2]), dtype='float')/ nsamples, '--', label='relu | 2 | 2')
plt.loglog()
plt.xlabel('rank')
plt.ylabel('probability')
plt.title('relu network simplicity bias')
plt.show()
```

Indeed, some functions are *way more* likely than others.

For example, what are the top 4 most frequent boolean functions? They are either constant functions or a single value different from one.

```python
for boolfun, freq in funcounters['relu'][2].most_common()[:4]:
    print('function as a binary string:')
    print('\t', boolfun)
    print('frequency')
    print('\t', freq)
```

    function as a binary string:
    	 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
    frequency
    	 1802
    function as a binary string:
    	 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
    frequency
    	 1762
    function as a binary string:
    	 11111111111111111111111111111111111111111111111111111111111111111111110111111111111111111111111111111111111111111111111111111111
    frequency
    	 9
    function as a binary string:
    	 11101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
    frequency
    	 5

Hmm this is pretty interesting!
Would this phenomenon hold for architectures other than a 2 layer relu network?
For example, let's try some networks with sigmoid activations.
Here we will use `erf`, since we can do some spectral calculations for it later.

```python
nsamples = 10**4
funcounters['erf'] = {}
funfreq['erf'] = {}
vb = 0
for vw in [1, 2, 4]:
    # `funcounters` holds a dictionary (more precisely, a `Counter` object) of boolean function (as a string) to its frequency
    funcounters['erf'][vw] = sample_boolean_fun(MyNet(Erf, WIDTHSPEC), vw, vb, nsamples, outformat='counter')
    # `funfreq` just has a list of frequencies
    funfreq['erf'][vw] = OD(funcounters['erf'][vw].most_common()).values()
```

```python
plt.plot(np.array(list(funfreq['relu'][2]), dtype='float')/ nsamples, '--', label='relu | 2 | 2')
plt.plot(np.array(list(funfreq['erf'][1]), dtype='float')/ nsamples, label='erf | 1 | 0')
plt.plot(np.array(list(funfreq['erf'][2]), dtype='float')/ nsamples, label='erf | 2 | 0')
plt.plot(np.array(list(funfreq['erf'][4]), dtype='float')/ nsamples, label='erf | 4 | 0')
plt.loglog()
plt.xlabel('rank')
plt.ylabel('probability')

plt.title(u'probability vs rank of $10^4$ random networks on $\{\pm1\}^7$')
plt.legend(title='$\phi$ | $\sigma_w^2$ | $\sigma_b^2$')
plt.show()
```

Looks like this "simplicity bias" is diminished when we use `erf`, and then goes away when we increase $\sigma_w^2$!
So it doesn't look like this "simplicity bias" is universal.
How can we understand this phenomenon better?
When can we expect "simplicity bias"?

# A Spectral Perspective on Simplicity Bias

*A priori*, the nonlinear nature seems to present an obstacle in reasoning about the distribution of random networks.
However, this question turns out to be more easily treated if we allow the *width to go to infinity*.
A long line of works starting with [Neal (1995)](http://www.cs.toronto.edu/~radford/bnn.book.html) and extended recently by [Lee et al. (2018)](https://openreview.net/forum?id=B1EA-M-0Z), [Novak et al. (2019)](https://arxiv.org/abs/1810.05148), and [Yang (2019)](https://arxiv.org/abs/1902.04760) has shown that randomly initialized, infinite-width networks are distributed as Gaussian processes.
These Gaussian processes also describe finite width random networks well as confirmed by [Valle-Pereze et al.](http://arxiv.org/abs/1805.08522) themselves.
We will refer to the corresponding kernels as the *Conjugate Kernels* (CK), following the terminology of [Daniely et al. (2017)](http://papers.nips.cc/paper/6427-toward-deeper-understanding-of-neural-networks-the-power-of-initialization-and-a-dual-view-on-expressivity.pdf).
Given the CK $K$, the simplicity bias of a wide neural network can be read off quickly from the *spectrum of $K$*:
If the largest eigenvalue of $K$ accounts for most of its trace, then a typical random network looks like a function from the top eigenspace of $K$.

More precisely, if we have the eigendecomposition
\begin{equation}
K = \sum_{i \ge 1} \lambda_i u_i\otimes u_i
\label{eqn:eigendecomposition}
\end{equation}
with eigenvalues $\lambda_i$ in decreasing order and corresponding eigenfunctions $u_i$, then each sample (i.e. wide neural network) from this GP can be obtained as
$$
\sum_{i \ge 1} \sqrt{\lambda_i} \omega_i u_i,\quad
\omega_i \sim \mathcal N(0, 1).
$$
If, for example, $\lambda_1 \gg \sum_{i \ge 2}\lambda_i$, then a typical sample function is just a very small perturbation of $u_1$.

This motivates us to take a look at the spectrum of the CK.

## A brief summary of the spectral theory of CK

Now, if the CK has spectra difficult to compute, then this perspective is not so useful.
But in idealized settings, where the data distribution is uniform over the boolean cube, the sphere, or from the standard Gaussian, a complete (or almost complete in the Gaussian case) eigendecomposition of the kernel can be obtained, thanks to the symmetry of the domain.
Of course, the probability-vs-rank experiment of Valle-Parez et al. is in this exact setting.
Here and in the paper, we focus on the boolean cube, since in high dimensions, all three distributions are very similar, and the boolean cube eigenvalues are much easier to compute (see paper for more details).

We briefly summarize the spectral theory of CK and NTK (of multilayer perceptrons, or MLPs) on the boolean cube.
First, these kernels are always diagonalized by the *boolean Fourier basis*, which are just monomial functions like $x_1 x_3 x_{10}$.
These Fourier basis functions are naturally graded by their *degree*, ranging from 0 to the dimension $d$ of the cube.
Then the kernel has $d+1$ unique eigenvalues,
$$\mu_0, \ldots, \mu_d$$
corresponding to each of the degrees, so that the eigenspace associated to $\mu_k$ is a $\binom d k$ dimensional space of monomials with degree $k$.
These eigenvalues are simple linear functions of a small number of the kernel values, and can be easily computed.

So let's compute the eigenvalues of the CK correponding to the architectures we've used above!

# Computing Eigenvalues over a Grid of Hyperparameters

Our methods for doing the theoretical computations lie in the `theory` module.

```python
from theory import *
```

First, let's compute the eigenvalues of erf CK and NTK over these hyperparameters:

- $\sigma_w^2 \in \{1, 2, 4\}$
- $\sigma_b^2 = 0$
- dimension 7 boolean cube
- depth up to 100
- degree $k \le 7$.

```python
erfvwrange = [1, 2, 4]
erfvbrange = [0]
s_erfvws, s_erfvbs = np.meshgrid([1, 2, 4], [0], indexing='ij')
dim = 7
depth = 100
maxdeg = 7
```

As mentioned in the paper, any CK or NTK $K$ of multilayer perceptrons (MLPs) takes the form
$$K(x, y) = \Phi\left(\frac{\langle x, y \rangle}{\|x\|\|y\|}, \frac{\|x\|^2}d, \frac{\|y\|^2}d\right)$$
for some function $\Phi: \mathbb R^3 \to \mathbb R$.
On the boolean cube $\{1, -1\}^d$, $\|x\|^2 = d$ for all $x$, and $\langle x, y \rangle / d$ takes value in a discrete set $\{-1, -1+2/d, \ldots, 1-2/d, 1\}$.
Thus $K(x, y)$ only takes a finite number of different values as well.
We first compute these values (see paper for the precise formulas).

```python
# `erfkervals` has two entries, with keys `cks` and `ntks`, but the `ntks` entry is not relevant to us here
# Each entry is an array with shape (`depth`, len(erfvwrange), len(erfvbrange), `dim`+1)
# The last dimension carries the entries $\Phi(-1), \Phi(-1 + 2/d), ..., \Phi(1)$
s_erfkervals = boolcubeFgrid(dim, depth, s_erfvws, s_erfvbs, VErf, VDerErf)
```

The eigenvalues $\mu_k, k = 0, 1, \ldots, d$, can be expressed a simple linear function of $\Phi$'s values, as hinted before.
However, a naive evaluation would lose too much numerical precision because the number of alternating terms.
Instead, we do something more clever, resulting in the following algorithm:

- For $\Delta = 2/d$, we first evaluate $\Phi^{(a)}(x) = \frac 1 2 \left(\Phi^{(a-1)}(x) - \Phi^{(a-1)}(x - \Delta)\right)$ with base case $\Phi^{(0)} = \Phi$, for $a = 0, 1, \ldots$, and for various values of $x$.
- Then we just sum a bunch of nonnegative terms to get the eigenvalue $\mu_k$ associated to degree $k$ monomials
$$\mu_k = \frac 1{2^{d-k}} \sum_{r=0}^{d-k}\binom{d-k}r \Phi^{(k)}(1 - r \Delta).$$

Note that, here we will compute *normalized eigenvalues*, normalized by their trace.
So these normalized eigenvalues, with multiplicity, should sum up to 1.

```python
s_erfeigs = {}
# `erfeigs['ck']` is an array with shape (`maxdeg`, `depth`+1, len(erfvwrange), len(erfvbrange))
# `erfeigs['ck'][k, L] is the matrix of eigenvalue $\mu_k$ for a depth $L$ erf network,
# as a function of the values of $\sigma_w^2, \sigma_b^2$ in `erfvwrange` and `erfvbrange`
# Note that these eigenvalues are normalized by the trace
# (so that all normalized eigenvalues sum up to 1)
s_erfeigs['ck'] = relu(boolCubeMuAll(dim, maxdeg, s_erfkervals['cks'], twostep=False))
```

This computes all we need for the erf kernels.
Now let's do the relu one.

```python
s_reluvws, s_reluvbs = np.meshgrid([2], [1], indexing='ij')
dim = 7
depth = 2
maxdeg = 7

s_relukervals = boolcubeFgrid(dim, depth, s_reluvws, s_reluvbs, VReLU, VStep)

s_relueigs = {}
s_relueigs['ck'] = relu(boolCubeMuAll(dim, maxdeg, s_relukervals['cks'], twostep=False))
```

# A Spectral Explanation of the Simplicity Bias

```python
def prunesmall(s, thr=1e-14):
    t = np.array(s)
    t[t<thr] = 0
    return t
```

```python
plt.figure(figsize=(12, 4.25))

ax0 = plt.subplot(121)
plt.plot(np.array(list(funfreq['relu'][2]), dtype='float')/ nsamples, '--', label='relu | 2 | 2')
plt.plot(np.array(list(funfreq['erf'][1]), dtype='float')/ nsamples, label='erf | 1 | 0')
plt.plot(np.array(list(funfreq['erf'][2]), dtype='float')/ nsamples, label='erf | 2 | 0')
plt.plot(np.array(list(funfreq['erf'][4]), dtype='float')/ nsamples, label='erf | 4 | 0')
plt.loglog()
plt.xlabel('rank')
plt.ylabel('probability')
ax0.text(-.15, -.15, '(a)', fontsize=24, transform=ax0.axes.transAxes)

plt.title(u'probability vs rank of $10^4$ random networks on $\{\pm1\}^7$')
plt.legend(title='$\phi$ | $\sigma_w^2$ | $\sigma_b^2$')

ax1 = plt.subplot(122)
plt.plot(prunesmall(s_relueigs['ck'][:, -1, 0, 0]), marker='x', linestyle='None', label=r'relu | 2 | 2 | 2')
for i in range(3):
    plt.plot(prunesmall(s_erfeigs['ck'][:, 2, i, 0]), marker='o', linestyle='None',
             label=r'erf | {} | 0 | 2'.format(2**i))
plt.plot(prunesmall(s_erfeigs['ck'][:, 32, -1, 0]), marker='*', linestyle='None', label=r'erf | 4 | 0 | 32')

plt.legend(title=r'$\phi$ | $\sigma_w^2$ | $\sigma_b^2$ | depth', loc='lower left')
plt.xlabel('degree $k$')
plt.ylabel(r'normalized eigenvalue $\tilde{\mu}_k$')
plt.title('erf networks lose simplicity bias for large $\sigma_w^2$ and depth')
plt.semilogy()
ax1.text(-.15, -.15, '(b)', fontsize=24, transform=ax1.axes.transAxes)

tight_layout(plt)
```

In **(a)**, we have reproduced the plot from above.
In **(b)** we have plotted the 8 unique (normalized) eigenvalues for the CK of each architecture given in the legend.

Immediately, we see that for relu and $\sigma_w^2 = \sigma_b^2 = 2$, the degree 0 eigenspace, corresponding to constant functions, accounts for more than $80\%$ of the variance.
This means that a typical infinite-width relu network of 2 layers is expected to be almost constant, and this should be even more true after we threshold the network to be a boolean function.
Indeed, this is exactly what we saw in [Section 2](#A-Quick-Replication-of-Valle-Perez-et-al.'s-Frequency-vs-Rank-Experiment)

On the other hand, for erf and $\sigma_b = 0$, the even degree $\mu_k$s all vanish, and most of the variance comes from degree 1 components (i.e. linear functions).
This concentration in degree 1 also lessens as $\sigma_w^2$ increases.
But because this variance is spread across a dimension 7 eigenspace, we don't see duplicate function samples nearly as much as in the relu case.
As $\sigma_w$ increases, we also see the eigenvalues become more equally distributed, which corresponds to the flattening of the probability-vs-rank curve in (a).

Finally, we observe that a 32-layer erf network with $\sigma_w^2 = 4$ has all its nonzero eigenvalues (associated to odd degrees) all equal (see points marked by $*$ in (b)).
This means that its distribution is a "white noise" on the space of *odd* functions, and the distribution of boolean functions obtained by thresholding the Gaussian process samples is the *uniform distribution* on *odd* functions.
This is the complete lack of simplicity bias modulo the oddness constraint.
Therefore, the simplicity bias is *really far away* from being universal to all neural networks, and seems more like a particular (nice) property of relu.

However, from the spectral perspective, there is a weak sense in which a simplicity bias holds for all neural network-induced CKs and NTKs.
We prove the following theorem in the paper.

**Theorem (Weak Spectral Simplicity Bias).**
Let $K$ be the CK of an MLP (with any nonlinearity) on a boolean cube $\{\pm1\}^d$.
Then the eigenvalues $\mu_k, k = 0, \ldots, d,$ satisfy
\begin{equation}
\mu_0 \ge \mu_2 \ge \cdots \ge \mu_{2k} \ge \cdots,\quad
\mu_1 \ge \mu_3 \ge \cdots \ge \mu_{2k+1} \ge \cdots.
\label{eqn:weaksimplicitybias}
\end{equation}

Even though it's not true that the fraction of variance contributed by the degree $k$ eigenspace is decreasing with $k$, the eigenvalue themselves will be in a nonincreasing pattern across even and odd degrees.
Of course, as we have seen, this is a *very weak* sense of simplicity bias, as it doesn't prevent "white noise" behavior as in the case of erf CK with large $\sigma_w^2$ and large depth.

# Conclusion

We have clarified the extent of "simplicity bias" in neural networks from the angle of eigendecomposition of the associated infinite-width conjugate kernel.
While this bias does not seem universal, it could still be that, architectures benefiting from a simplicity bias also do generalize better.
This would require some knowledge of the training of neural networks though.
Coincidentally, recent advances in deep learning theory have revealed that a different kernel, the *Neural Tangent Kernel*, in fact governs the evolution of NN gradient descent dynamics.
We discuss training and generalization from a spectral analysis of the NTK in the notebook *[Neural Network Generalization](NeuralNetworkGeneralization.ipynb)*, and more thoroughly in our full paper [*A Fine-Grained Spectral Perspective on Neural Networks*](https://arxiv.org/abs/1907.10599).

# Appendix

## The $\{0, 1\}^d$ Boolean Cube vs the $\{\pm 1 \}^d$ Boolean Cube

[Valle-Perez et al.]() actually did their experiments on the $\{0, 1\}^d$ boolean cube, whereas in the paper and the notebook here, we have focused on the $\{\pm 1\}^d$ boolean cube.
As datasets are typically centered before feeding into a neural network (for example, using `torchvision.transform.Normalize`), $\{\pm 1\}^d$ is much more natural.
In comparison, using the $\{0, 1\}^d$ cube is equivalent to adding a bias in the input of a network and reducing the weight variance in the input layer, since any $x \in \{\pm 1\}^d$ corresponds to $\frac 1 2 (x + 1) \in \{0, 1\}^d$.

Nevertheless, here we verify that the main point of the paper and of the examples above still holds over the $\{0, 1\}^d$ cube.
Let's do the same experiments as the beginning for $\{0, 1\}^d$.

```python
WIDTHSPEC = (7, 40, 40, 1)
nsamples = 10**4
funcounters = {'relu': {}}
funfreq = {'relu': {}}
# vb = sigma_b^2
vb = 2
# vw = \sigma_w^2
for vw in [2]:
    # `funcounters` holds a dictionary (more precisely, a `Counter` object)
    # of boolean function (as a string of length 2^7 = 128) to its frequency
    funcounters['relu'][vw] = sample_boolean_fun(MyNet(nn.ReLU, WIDTHSPEC), vw, vb, nsamples, outformat='counter',
                                                bit=[0, 1]) # this allows us to sample over the {0, 1} cube
    # `funfreq` just has a list of frequencies
    funfreq['relu'][vw] = OD(funcounters['relu'][vw].most_common()).values()
```

```python
nsamples = 10**4
funcounters['erf'] = {}
funfreq['erf'] = {}
vb = 0
for vw in [1, 2, 4]:
    # `funcounters` holds a dictionary (more precisely, a `Counter` object) of boolean function (as a string) to its frequency
    funcounters['erf'][vw] = sample_boolean_fun(MyNet(Erf, WIDTHSPEC), vw, vb, nsamples, outformat='counter',
                                               bit=[0, 1]) # this allows us to sample over the {0, 1} cube
    # `funfreq` just has a list of frequencies
    funfreq['erf'][vw] = OD(funcounters['erf'][vw].most_common()).values()
```

Let's also try a 32 layer erf network with $\sigma_w^2 = 4$, which gives a "white noise" distribution over $\{\pm 1\}^d$

```python
nsamples = 10**4
vw = 4
vb = 0
widthspec = [7] + [40] * 32 + [1]
funcounters['deeperf'] = sample_boolean_fun(MyNet(Erf, widthspec), vw, vb, nsamples, outformat='counter',
                                           bit=[0, 1])  # this allows us to sample over the {0, 1} cube
funfreq['deeperf'] = OD(funcounters['deeperf'].most_common()).values()
```

Plot them as before...

```python
plt.plot(np.array(list(funfreq['relu'][2]), dtype='float')/ nsamples, '--', label='relu | 2 | 2 | 2')
plt.plot(np.array(list(funfreq['erf'][1]), dtype='float')/ nsamples, label='erf | 1 | 0 | 2')
plt.plot(np.array(list(funfreq['erf'][2]), dtype='float')/ nsamples, label='erf | 2 | 0 | 2')
plt.plot(np.array(list(funfreq['erf'][4]), dtype='float')/ nsamples, label='erf | 4 | 0 | 2')
plt.plot(np.array(list(funfreq['deeperf']), dtype='float')/ nsamples, label='erf | 4 | 0 | 32')
plt.loglog()
plt.xlabel('rank')
plt.ylabel('probability')

plt.title(u'probability vs rank of $10^4$ random networks on $\{0, 1\}^7$')
plt.legend(title='$\phi$ | $\sigma_w^2$ | $\sigma_b^2$ | depth')
```

Just like over the $\{\pm 1\}^d$ cube, the relu network biases significantly toward certain functions, but with erf, and with increasing $\sigma_w^2$, this lessens.
With depth 32 and $\sigma_w^2$, the boolean functions obtained from erf network see no bias at all.


# MAP inference

> In classes 6 and 7 we focused our attention to inference methods the find conditional probability queries. In this class, we turn our attention to MAP queries, which are also very useful in several applications (recall the motivating example of the first class).
>
> As a reminder, a MAP query aims to find the most likely assignment to all of the non-evidence variables. Similarly, a marginal MAP query aims to find the most likely assignment to a subset of the variables.

> **Objetives:**
> - To reformulate the Variable Elimination method for MAP inference.
> - To reformulate the Belief Propagation algorithm for MAP inference.

> **References:**
> - Probabilistic Graphical Models: Principles and Techniques, By Daphne Koller and Nir Friedman. Ch. 13.
> - Mastering Probabilistic Graphical Models Using Python, By Ankur Ankan and Abinash Panda. Ch. 3.
> - Probabilistic Graphical Models Specialization, offered through Coursera. Prof. Daphne Koller.

 Imagen recuperada de: https://upload.wikimedia.org/wikipedia/commons/e/ed/Bayes_icon.svg.

___

# 1. Variable Elimination

We begin studying the most fundamental inference algorithm: variable elimination.

## 1.1. Max-product variable elimination

Let's introduce this problem though a simple example.

**Example.** Consider the following BN

```python
from IPython.display import Image
```

```python
Image("figures/simpleExample.png")
```

Or goal is to compute:

$$\begin{align}
\max_{a,b} P(A=a, B=b)  & = \max_{a,b} P(A=a)P(B=b | A=a) \\
                        & = \max_{a} \max_{b} P(A=a)P(B=b | A=a).
\end{align}$$

From this last form, we would want to choose $a$ so that

$$\max_{b} P(A=a)P(B=b | A=a) = P(A=a) \max_{b}P(B=b | A=a)$$

is maximum. Hence,

- a **necessary condition** of the joint MAP assignment $a,b$ is that $b$ is selected to maximize $P(B=b | A=a)$.

- This is not sufficient, however, since we must choose $a$ appropriately.

For instance, consider the following CPDs

```python
Image("figures/simpleExample2.png")
```

Let's denote $\max_{b}P(B=b | A=a) = \phi_1(a)$. Then

- $\phi_1(a^0)=0.9$
- $\phi_1(a^1)=0.55$

Finally,

$$\max_{a} P(a) \phi_1(a) = \max_{a} \phi_2(a)=0.36$$

| $A$ | $\phi_2$           |
| --- | ------------------ |
|$a^0$|$0.9\times0.4=0.36$ |
|$a^1$|$0.55\times0.6=0.33$|

### Important comments:

- One can note that the operation $\max_{b}P(B=b | A=a) = \phi_1(a)$ corresponds to a *max-marginalization* of the $P(B|A)$ factor.

- Just like in conditional probability queries, this step marginalizes out the variable $B$, not by summing, but by maximizing it.

- One key point is: Assume we are given two factors $\phi_1,\phi_2$ and that $X\notin \mathrm{scope}[\phi_1]$, then

  $$\max_{X} \phi_1 \phi_2 = \phi_1 \max_{X} \phi_2.$$

  This is, we can push in the maximization operations (just as we push in the summation operations) over factors that don't involve the variable being maximized. Similarly,

  $$\max_{X} (\phi_1 + \phi_2) = \phi_1 + \max_{X} \phi_2.$$

The final comment directly leads to the **max-product and max-sum variable elimination algorithms** which is completely dual to the **sum-prod variable elimination algorithm** that we studied in class 6.

### Max-product variable elimination algorithm

We will describe the steps for eliminating some variable $Z$ from $\bar{\Phi}$: *Eliminate Var-$Z$ from $\bar{\Phi}$*.

1. Determine the set of factors that involve $Z$:

   $$\bar{\Phi}' = \left\{\phi_i \in \bar{\Phi} : Z  \in \mathrm{scope}[\phi_i]\right\}$$

2. Compute:

   $$\psi = \prod_{\phi_i \in \bar{\Phi}'} \phi_i$$

3. Compute:

   $$\tau = \max_Z \psi$$

   Store the value $z^\ast$ that maximizes $\psi$.

4. Overwrite:

   $$\bar{\Phi} := \left(\bar{\Phi}\setminus \bar{\Phi}'\right) \cup \{\tau\}$$

Thus, the complete algorithm can be described as:

1. The first step is to reduce all factors in $\bar{\Phi}$ acording to the given evidence, if any.

2. For each non-query variable $Z$:
   - Eliminate Var-$Z$ from $\bar{\Phi}$.

3. Multiply all the remaining factors.

## 1.2. Max-sum variable elimination algorithm

Often, the values of some factors are so small that, when multipled together, they may produce values that are less than the processor/memory resolution: **numerical underflow**.

One effective way of dealing with this situation is to consider the $\log$ of the factors.

We have that:

$$P_{\bar{\Phi}} (\bar{X}) \propto \prod_{k} \phi_{k} (\bar{X}_k)$$

and we want,

$$\arg \max_{X} \prod_{k} \phi_{k} (\bar{X}_k) = \arg \max_{X} \log\left(\prod_{k} \phi_{k} (\bar{X}_k)\right) = \arg \max_{X} \sum_{k} \log\left(\phi_{k} (\bar{X}_k)\right),$$

since the $\log$ is an increasing function.

Thus, converting factors with the logarithm allows us to interchange products by sum, obtaining some **practical robustness against numerical underflow**.

The algorithm is completely analogous:

### Max-sum variable elimination algorithm

We will describe the steps for eliminating some variable $Z$ from $\bar{\Phi}$: *Eliminate Var-$Z$ from $\bar{\Phi}$*.

0. Take the $log$ of all the factors:

   $$\theta_k(\bar{X}_k) = \log{\phi_k}(\bar{X}_k).$$

   $$\bar{\Theta} = \{\theta_{1}(\bar{X}_1), \dots, \theta_{m}(\bar{X}_m)\}.$$

1. Determine the set of factors that involve $Z$:

   $$\bar{\Theta}' = \left\{\theta_i \in \bar{\Theta} : Z  \in \mathrm{scope}[\theta_i]\right\}$$

2. Compute:

   $$\psi = \sum_{\theta_i \in \bar{\Theta}'} \phi_i$$

3. Compute:

   $$\tau = \max_Z \psi$$

   Store the value $z^\ast$ that maximizes $\psi$.

4. Overwrite:

   $$\bar{\Theta} := \left(\bar{\Theta}\setminus \bar{\Theta}'\right) \cup \{\tau\}$$

How is this done in `pgmpy`?

```python
Image("figures/restaurant.png")
```

```python
# Import pgmpy.models.BayesianModel
from pgmpy.models import BayesianModel
# Import pgmpy.factors.discrete.TabularCPD
from pgmpy.factors.discrete import TabularCPD
```

```python
# Define model skeleton
restaurant_model = BayesianModel([("L", "C"),
                                  ("Q", "C"),
                                  ("L", "N"),
                                  ("C", "N")])
```

```python
# Define CPDs
cpd_L = TabularCPD(variable="L",
                   variable_card=2,
                   values=[[0.4],
                           [0.6]])
cpd_Q = TabularCPD(variable="Q",
                   variable_card=3,
                   values=[[0.2],
                           [0.5],
                           [0.3]])
cpd_C = TabularCPD(variable="C",
                   variable_card=2,
                   evidence=["L", "Q"],
                   evidence_card=[2, 3],
                   values=[[0.95, 0.4, 0.4, 0.9, 0.4, 0.2],
                           [0.05, 0.6, 0.6, 0.1, 0.6, 0.8]])
cpd_N = TabularCPD(variable="N",
                   variable_card=2,
                   evidence=["L", "C"],
                   evidence_card=[2, 2],
                   values=[[0.4, 0.9, 0.2, 0.4],
                           [0.6, 0.1, 0.8, 0.6]])
```

```python
# Attach CPDs to the model
restaurant_model.add_cpds(cpd_L, cpd_Q, cpd_C, cpd_N)
```

```python
# Check if the model is correctly defined
restaurant_model.check_model()
```

    True

```python
# Import pgmpy.inference.VariableElimination
from pgmpy.inference import VariableElimination
```

```python
# Create VariableElimination inference object
inference = VariableElimination(restaurant_model)
```

```python
# Map query
print(inference.map_query(variables=["N"], evidence={"L": 0, "C": 1}))
```

    Finding Elimination Order: : : 0it [00:00, ?it/s]
    0it [00:00, ?it/s]

    {'N': 0}

```python
# Max marginal
print(inference.max_marginal(variables=["N"], evidence={"L": 0, "C": 1}))
```

    Finding Elimination Order: : : 0it [00:00, ?it/s]
    0it [00:00, ?it/s]

    0.9

# 2. Max-sum Belief Propagation

We start by a simple example

### Max-sum elimination in chains

Consider the following chain

```python
from IPython.display import Image
```

```python
Image("figures/chain.png")
```

We want:

$$\begin{align}
\max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) & = \max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) \\
                                                      & = \max_{D} \max_{C} \max_{B} \max_{A} \left(\theta_1(A,B) + \theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E)\right) \\
                                                      & = \max_{D} \max_{C} \max_{B} \left(\theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E) + \max_{A}\theta_1(A,B)\right)
\end{align}$$

___
**1. Sum of factors**

$$\phi_1(A, B)$$

|   A   |   B   | $\phi_1$ |
| ----- | ----- | -------- |
| $a^0$ | $b^0$ | 5        |
| $a^0$ | $b^1$ | 8        |
| $a^1$ | $b^0$ | 1        |
| $a^1$ | $b^1$ | 0        |
| $a^2$ | $b^0$ | 3        |
| $a^2$ | $b^1$ | 9        |

$$\phi_2(B, C)$$

|   B   |   C   | $\phi_2$ |
| ----- | ----- | -------- |
| $b^0$ | $c^0$ | 5        |
| $b^0$ | $c^1$ | 7        |
| $b^1$ | $c^0$ | 1        |
| $b^1$ | $c^1$ | 2        |

$$\phi_3(A, B, C) = \phi_1(A, B) + \phi_2(B, C)$$

|   A   |   B   |   C   | $\phi_3$     |
| ----- | ----- | ----- | ------------ |
| $a^0$ | $b^0$ | $c^0$ | $5 + 5 = 10$ |
| $a^0$ | $b^0$ | $c^1$ | $5 + 7 = 12$ |
| $a^0$ | $b^1$ | $c^0$ | $8 + 1 = 9$  |
| $a^0$ | $b^1$ | $c^1$ | $8 + 2 = 10$ |
| $a^1$ | $b^0$ | $c^0$ | $1 + 5 = 6$  |
| $a^1$ | $b^0$ | $c^1$ | $1 + 7 = 8$  |
| $a^1$ | $b^1$ | $c^0$ | $0 + 1 = 1$  |
| $a^1$ | $b^1$ | $c^1$ | $0 + 2 = 2$  |
| $a^2$ | $b^0$ | $c^0$ | $3 + 5 = 8$  |
| $a^2$ | $b^0$ | $c^1$ | $3 + 7 = 10$ |
| $a^2$ | $b^1$ | $c^0$ | $9 + 1 = 10$ |
| $a^2$ | $b^1$ | $c^1$ | $9 + 2 = 11$ |

**2. Max reduction**

$$\phi_3(A, B, C)$$

|   A   |   B   |   C   | $\phi_3$     |
| ----- | ----- | ----- | ------------ |
| $a^0$ | $b^0$ | $c^0$ | $5 + 5 = 10$ |
| $a^0$ | $b^0$ | $c^1$ | $5 + 7 = 12$ |
| $a^0$ | $b^1$ | $c^0$ | $8 + 1 = 9$  |
| $a^0$ | $b^1$ | $c^1$ | $8 + 2 = 10$ |
| $a^1$ | $b^0$ | $c^0$ | $1 + 5 = 6$  |
| $a^1$ | $b^0$ | $c^1$ | $1 + 7 = 8$  |
| $a^1$ | $b^1$ | $c^0$ | $0 + 1 = 1$  |
| $a^1$ | $b^1$ | $c^1$ | $0 + 2 = 2$  |
| $a^2$ | $b^0$ | $c^0$ | $3 + 5 = 8$  |
| $a^2$ | $b^0$ | $c^1$ | $3 + 7 = 10$ |
| $a^2$ | $b^1$ | $c^0$ | $9 + 1 = 10$ |
| $a^2$ | $b^1$ | $c^1$ | $9 + 2 = 11$ |

$$\phi_4(A, B) = \max_{C}\phi_3(A, B, C)$$

|   A   |   B   | $\phi_3$     |
| ----- | ----- | ------------ |
| $a^0$ | $b^0$ | $5 + 7 = 12$ |
| $a^0$ | $b^1$ | $8 + 2 = 10$ |
| $a^1$ | $b^0$ | $1 + 7 = 8$  |
| $a^1$ | $b^1$ | $0 + 2 = 2$  |
| $a^2$ | $b^0$ | $3 + 7 = 10$ |
| $a^2$ | $b^1$ | $9 + 2 = 11$ |

___

See in the whiteboard then show:

$$\begin{align}
\max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) & = \max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) \\
                                                      & = \max_{D} \max_{C} \max_{B} \max_{A} \left(\theta_1(A,B) + \theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E)\right) \\
                                                      & = \max_{D} \max_{C} \max_{B} \left(\theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E) + \max_{A}\theta_1(A,B)\right) \\
                                                      & = \max_{D} \max_{C} \max_{B} \left(\theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E) + \tau_1(B)\right) \\
                                                      & = \max_{D} \max_{C} \left(\theta_3(C,D) + \theta_4(D,E) + \max_{B} (\theta_2(B,C)+ \tau_1(B))\right) \\
                                                      & = \max_{D} \left(\theta_4(D,E) + \max_{C} (\theta_3(C,D) + \tau_2(C))\right) \\
                                                      & = \max_{D} \left(\theta_4(D,E) + \tau_3(D)\right) \\
                                                      & = \tau_4(E)
\end{align}$$

### Max-sum in clique trees

Consider the clique tree of the above chain:

```python
Image("figures/clusterGraphChain.png")
```

The message passing has the same idea in this context:

```python
Image("figures/clusterGraphChain2.png")
```

Then, the forward-backward algorithm we defined for conditional probability queries in clique trees is exactly the same for MAP queries.

We only have to change products by sums/products, and reduction by max-reductions.

How is this done in `pgmpy`?

```python
# Import pgmpy.inference.BeliefPropagation
from pgmpy.inference import BeliefPropagation
```

```python
# Create BeliefPropagation inference object
inference = BeliefPropagation(restaurant_model)
```

```python
# MAP query
print(inference.map_query(variables=["N"], evidence={"L": 0, "C": 1}))
```

    Eliminating: Q: 100%|██████████| 1/1 [00:00<00:00, 332.78it/s]

    {'N': 0}

# 3. Application - Classification

Let's use the example in the first class

```python
# Library imports (sklearn.datasets.load_iris, pandas y numpy)
from sklearn.datasets import load_iris
import pandas as pd
import numpy as np
```

```python
# Data loading (integer parts, for simplicity)
iris = load_iris()
X = iris['data'][:, :2].astype(int)
y = iris['target']
df = pd.DataFrame(X, columns=['sepal_length', 'sepal_width'])
df['target'] = y
df
```

      sepal_length
      sepal_width
      target

      0
      5
      3
      0

      1
      4
      3
      0

      2
      4
      3
      0

      3
      4
      3
      0

      4
      5
      3
      0

      ...
      ...
      ...
      ...

      145
      6
      3
      2

      146
      6
      2
      2

      147
      6
      3
      2

      148
      6
      3
      2

      149
      5
      3
      2

150 rows × 3 columns

| Type of flower | label  |
| -------------- | ------ |
| Setosa         | 0      |
| Versicolor     | 1      |
| Virginica      | 2      |

```python
# Import matplotlib.pyplot
import matplotlib.pyplot as plt
```

```python
# Scatter plot (largo vs. ancho), color regarding type of flower. Add a bit of noise
l = df['sepal_length'] + np.random.normal(0, 0.2, (len(df), ))
w = df['sepal_width'] + np.random.normal(0, 0.2, (len(df), ))
plt.figure(figsize=(8, 5))
plt.scatter(l, w, c=['bgr'[c] for c in df['target']])
plt.show()
```

```python
# Import sklearn.model_selection.train_test_split
from sklearn.model_selection import train_test_split
```

```python
train_test_split?
```

```python
# Train/Test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1)
```

```python
Image('figures/iris.png')
```

```python
# Flower model
iris_model = BayesianModel([("T", "W"),
                            ("T", "L")])
```

```python
X_train
```

    array([[4, 3],
           [6, 3],
           [5, 4],
           [5, 2],
           [4, 2],
           [6, 2],
           [5, 2],
           [5, 2],
           [5, 2],
           [5, 2],
           [6, 3],
           [6, 3],
           [5, 2],
           [5, 4],
           [6, 3],
           [7, 3],
           [5, 2],
           [4, 3],
           [6, 3],
           [5, 2],
           [6, 3],
           [4, 3],
           [6, 2],
           [5, 2],
           [4, 3],
           [6, 2],
           [6, 3],
           [5, 2],
           [5, 2],
           [5, 3],
           [6, 3],
           [5, 3],
           [4, 2],
           [4, 3],
           [6, 2],
           [6, 3],
           [5, 2],
           [6, 2],
           [6, 3],
           [6, 2],
           [5, 3],
           [4, 3],
           [5, 3],
           [4, 3],
           [5, 3],
           [5, 3],
           [5, 2],
           [5, 2],
           [5, 2],
           [6, 2],
           [5, 3],
           [5, 3],
           [6, 2],
           [5, 3],
           [6, 2],
           [5, 3],
           [7, 2],
           [7, 3],
           [5, 3],
           [6, 2],
           [6, 3],
           [5, 3],
           [6, 2],
           [6, 2],
           [5, 3],
           [5, 3],
           [5, 3],
           [4, 3],
           [5, 3],
           [6, 2],
           [6, 2],
           [5, 3],
           [5, 3],
           [5, 3],
           [5, 2],
           [6, 3],
           [5, 2],
           [6, 3],
           [7, 3],
           [6, 3],
           [4, 3],
           [4, 2],
           [5, 2],
           [4, 3],
           [4, 3],
           [6, 3],
           [5, 3],
           [7, 3],
           [5, 3],
           [5, 2],
           [6, 3],
           [5, 3],
           [5, 3],
           [5, 2],
           [6, 2],
           [5, 2],
           [6, 3],
           [6, 3],
           [6, 2],
           [6, 3],
           [5, 4],
           [6, 3],
           [5, 3],
           [6, 3],
           [7, 3],
           [5, 3],
           [5, 4],
           [4, 3],
           [5, 3],
           [6, 2],
           [7, 2],
           [5, 2],
           [6, 3],
           [5, 3],
           [4, 3],
           [6, 2],
           [6, 3],
           [7, 2],
           [6, 3],
           [6, 3],
           [4, 3],
           [4, 3],
           [4, 3],
           [6, 3],
           [6, 3],
           [6, 2],
           [6, 2],
           [5, 3],
           [7, 3],
           [4, 3],
           [6, 3],
           [5, 2],
           [5, 3],
           [7, 2],
           [7, 3]])

```python
# Generate train frame
# iris_model.fit?
df_train = pd.DataFrame({"L": X_train[:, 0],
                         "W": X_train[:, 1],
                         "T": y_train})
df_train.head()
```

      L
      W
      T

      0
      4
      3
      0

      1
      6
      3
      2

      2
      5
      4
      0

      3
      5
      2
      1

      4
      4
      2
      2

```python
# Train model
iris_model.fit(df_train)
```

```python
# Generate test frame
df_test = pd.DataFrame({"L": X_test[:, 0],
                        "W": X_test[:, 1],
                        "T": y_test})
df_test.head()
```

      L
      W
      T

      0
      7
      3
      2

      1
      5
      3
      0

      2
      6
      3
      2

      3
      5
      3
      0

      4
      4
      2
      1

```python
# Predict (MAP Assignment)
y_pred = iris_model.predict(df_test[["L", "W"]])
```

    100%|██████████| 7/7 [00:00<00:00, 522.60it/s]

```python
y_test, y_pred.values.reshape(-1,)
```

    (array([2, 0, 2, 0, 1, 1, 1, 0, 1, 0, 2, 0, 1, 1, 2]),
     array([2, 0, 2, 0, 0, 2, 1, 0, 2, 0, 2, 0, 2, 2, 2]))

```python
# Accuracy
(y_test == y_pred.values.reshape(-1,)).mean()
```

    0.6666666666666666

# Announcements

## 1. Quiz.

Created with Jupyter by Esteban Jiménez Rodríguez.


# Otimização

## Introdução

Problemas de otimização (POs) são encontrados em diversas situações da Engenharia, em particular na Engenharia de Produção. Em uma linha de produção, por exemplo, a otimização de custos com logística, recursos humanos, matéria-prima são exemplos de onde podemos empregar métodos computacionais para obter soluções _ótimas_. Entretanto, princípios de otimização são a base de muitos algoritmos e aplicações de inteligência artificial, em particular, no aprendizado de máquina. Máquinas de vetor de suporte (_support vector machines_) são um exemplo de onde se usa otimização, já que podem ser formuladas como problemas convexos quadráticos.

Problemas de otimização são comumente tratados como *problemas de minimização*, onde se busca o _mínimo global_ de uma _função objetivo_ (FO) escalar $f(x)$, visto que maximizar $f(x)$ é equivalente a minimizar $-f(x)$.

Entretanto, esses problemas são acompanhados de _restrições_, que podem ser representadas por uma igualdade ou por uma desigualdade. Quando uma restrição é escrita na forma $g(x) = 0$, dizemos que $g(x)$ é uma _restrição de igualdade_; quando escrita na forma $h(x) \leq 0$, dizemos que $h(x)$ é uma _restrição de desigualdade_.

Neste capítulo, faremos uma breve explanação sobre otimização tomando o cálculo de derivadas e pontos críticos como elementos fundamentais. Utilizaremos recursos de computação simbólica para resolver um problema unidimensional e revisitaremos conceitos aprendidos nas disciplinas de Cálculo.

### Classificação de problemas de otimização

Problemas de otimização (PO) são classificados com base nas propriedades das funções $f(x)$, $g(x)$ e $h(x)$. Em linhas gerais, um PO pode ser:

- _univariado_ (ou _unidimensional_), se $x$ é escalar, i.e. $x \in \mathbb{R}$;
- _multivariado_ (ou _multidimensional_), se $x$ é um vetor, i.e. $x \in \mathbb{R}^n$.
- _linear_: se a FO e as restrições são funções lineares. Neste caso, por razões históricas, diz-se que o problema é de _programação linear_.
- _não-linear_: se a FO e as restrições são funções não-lineares. Neste caso, diz-se que o problema é de _programação não-linear_.

Com respeito às restrições, um PO pode ainda ser:

- _irrestrito_: quando não se assumem limites para os valores de $x$.
- _restrito_: quando limites para os valores de $x$ são impostos.

Aqui trataremos apenas de casos em que $x \in \mathbb{R}$.

### Problemas convexos

Sabe-se que problemas não-lineares são muito mais difíceis de resolver do que problemas lineares porque eles podem admitir uma ampla variedade de comportamentos. Um PO não-linear pode ter tanto _mínimos locais_ quanto _mínimos globais_. Logo, encontrar o _mínimo global_ de uma função $f(x)$ não-linear exige técnicas aperfeiçoadas.

Neste sentido, uma subclasse de problemas não-lineares que pode ser resolvida eficientemente são os chamados _convexos_. Em problemas convexos, a função $f(x)$ é _convexa_. Mas o que é uma _função convexa_?

Uma função convexa definida em um intervalo $[a,b]$ é aquela em que todos os seus valores estão abaixo da reta secante que passa pelos pontos $(a,f(a))$ e $(b,f(b)$. Isto, por sua vez, garante que ela contenha _somente_ um mínimo global.

Importaremos os seguintes módulos:

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sy
sy.init_printing()
```

**Exemplo**: a função $f(x) = 3x^2 - 0.36x - 11.2$ é convexa em $[-2,3]$.

```python
# domínio
a,b = -2,3
x = np.linspace(a,b,100)

# função e valores nos extremos
f = lambda x: 5*x**2 - 10.36*x - 11.2
fa,fb = f(a),f(b)

# reta secante
s = fa + (fb - fa)/(b - a)*(x - a)

# ponto de mínimo: -b/(2a)
xmin = 10.36/10

# plotagem de funções
plt.figure(figsize=(5,3))
plt.plot(x,f(x))
plt.plot(x,s,color='#ffa500')

# pontos da secante
plt.plot(a,f(a),'o',color='#ffa500')
plt.plot(b,f(b),'o',color='#ffa500')

# ponto de mínimo
plt.plot(xmin,f(xmin),'*r',ms=10);

plt.title('Exemplo de função convexa');
```

**Exemplo**: a função $p(x) = 10x^2\textrm{sen}(6x) - 10.36\exp(x/8) - 11.2$ não é convexa em $[-2,3]$.

```python
# função
p = lambda x: 10*x**2*np.sin(6*x) - 10.36*x*np.exp(x/8) - 11.2

# extremos
pa,pb = p(a),p(b)

# secante
t = pa + (pb - pa)/(b - a)*(x - a)

# plotagem de funções
plt.figure(figsize=(5,3))
plt.plot(x,p(x))
plt.plot(x,t,color='#ffa500')

# pontos da secante
plt.plot(a,p(a),'o',color='#ffa500')
plt.plot(b,p(b),'o',color='#ffa500')

# mínimos locais
xloc = [-1.33868618,0.88811853,1.87451904]

for xl in xloc:
    plt.plot(xl,p(xl),'or');

# mínimo global
xmin2 = 2.90547127

plt.plot(xmin2,p(xmin2),'*r',ms=10);

plt.title('Exemplo de função não convexa');
```

Como vemos acima, a função $p(x)$ admite 3 mínimos locais e um mínimo global. Pense um pouco sobre que estratégia computacional você utilizaria para encontrar os mínimos locais. Mais adiante mostraremos como localizar o mínimo global para funções univariadas contínuas (quando possível).

### Pontos de sela

Como vimos acima, a convexidade de uma função é uma propriedade muito importante para que um mínimo global seja localizado. Como sabemos do Cálculo, pontos de máximo ou mínimo identificam-se como _pontos críticos_ de uma função nos quais a primeira derivada da função se anula.

Casos particulares onde a derivada de uma FO anula-se mas o ponto não pode ser definido como de mínimo ou máximo podem ocorrer. Tais situações implicam a existência dos chamados _pontos de sela_. Uma função com um único ponto de sela, por exemplo, não admitirá mínimo global nem mínimo local. Para testarmos se um ponto crítico é um ponto de sela, devemos verificar o sinal da segunda derivada da função. Uma das seguintes situações deve ser obtida em um ponto crítico $x^*$:

- _ponto de mínimo:_ $f''(x^*) > 0$
- _ponto de máximo:_ $f''(x^*) < 0$
- _ponto de sela:_ $f''(x^*) = 0$

**Exemplo:** qualquer função quadrática admite ou um ponto de mínimo ou de máximo. A função $f(x) = x^3$ possui um ponto de sela em $x^* = 0$.

```python
x = np.linspace(-1,1)

plt.figure(figsize=(10,3))

plt.subplot(131)
plt.plot(x,x**2 + 1)
plt.plot(0,1,'r*',ms=10)
plt.title('mínimo global')

plt.subplot(132)
plt.plot(x,-x**2 + 1)
plt.plot(0,1,'r*',ms=10)
plt.title('máximo global')

plt.subplot(133)
plt.plot(x,x**3 + 1)
plt.plot(0,1,'r*',ms=10)
plt.title('ponto de sela');
```

## Otimização univariada

Como dissemos anteriormente, a otimização univariada visa resolver um problema de minimização tomando uma FO que depende apenas de uma variável. Matematicamente, podemos descrever este problema da seguinte forma:

$$\text{Encontre } x^{*} = \min f(x), \, \text{sujeito a} \, g(x) = 0, h(x) \leq 0.$$

Em geral, $x$ é uma _variável de decisão_, isto é, uma quantidade que pode ser ajustada livremente (ex. comprimentos, áreas, ângulos etc.).

As técnicas utilizadas para a resolução de um problema desse tipo são baseadas em métodos analíticos (busca pelos zeros das derivadas) ou em métodos computacionais (determinação de raízes por processos iterativos). Métodos chamados de _root finding_ são estudados em um curso introdutório de Métodos Numéricos.

Para exemplificar, usaremos uma abordagem analítica por meio de computação simbólica (módulo `sympy`) para resolver um problema que pode ser exibido como de otimização univariada.

### Problema resolvido

Consideremos o seguinte problema: _maximizar a área do retângulo inscrito em uma elipse._

### Resolução

Em primeiro lugar, escreveremos este problema em linguagem matemática. Sabemos que a área de um retângulo com vértice esquerdo inferior na origem da elipse e com vértice direito superior no ponto $(x,y)$ da elipse que está no primeiro quadrante é dada por $A_r = xy$. Logo, a área do retângulo inscrito na elipse será $A = 4xy$.

A área $A$ pode ser escrita em termos de $y$. Uma vez que a equação da elipse (centrada na origem) é dada por

$$\frac{x^2}{a^2} + \frac{y^2}{b^2} = 1,$$

podemos resolver a equação da elipse para $x$ (ou $y$) e substituir esta solução na expressão da área para ter uma função $A(x)$ (ou $A(y)$). Se escolhermos $x$, o problema de otimização pode ser escrito como:

$$\text{Encontre } x^{*} = \min \,( -A(x) ), \, \text{sujeito a} \, x > 0.$$

Notemos que maximizar $A(x)$ equivale a minimizar $-A(x)$.

Na busca do ponto de mínimo $x^{*}$, usaremos computação simbólica.

Primeiramente, criamos variáveis simbólicas que representem as variáveis de interesse do problema e a expressão da área total.

```python
# cria variáveis simbólicas
x,y,a,b = sy.symbols('x,y,a,b')

# área do retângulo no 1o. quadrante é xy
# logo, área total é 4xy
A = -4*x*y
A
```

Em seguida, resolvemos a equação da elipse para a variável $y$ utilizando a função `sympy.solve`.

```python
# resolve equação da elipse para y
sol = sy.solve(x**2/a**2 + y**2/b**2 - 1,y)
sol[0],sol[1]
```

Duas soluções são possíveis para $y$. Porém, como o nosso ponto de referência sobre a elipse está no primeiro quadrante, tomamos a expressão para $y > 0$ e a substituímos na expressão da área de forma a obter uma expressão univariada $A(x)$.

```python
# substitui expressão de y positivo em A para ter -A(x)
A = A.subs({'y':sol[1]})
A
```

Localizaremos o ponto crítico da função a partir da derivada $A'(x)$. Derivando $A$ em relação a $x$, obtemos:

```python
# deriva -A(x) com a,b constantes
dAdx = A.diff(x)
dAdx
```

Em seguida, buscamos $x^{*}$ tal que $A'(x^{*}) = \frac{dA}{dx}(x^{*}) = 0$.

```python
# resolve A'(x*) = 0
sol_x = sy.solve(dAdx,x)
sol_x
```

Duas soluções, são possíveis, porém, podemos verificar qual ponto de crítico, de fato, é o que minimizará $-A(x)$ através da análise da concavidade. Então, calculamos $A''(x)$, para cada ponto crítico.

```python
# testa A''(x) para os dois pontos
dAdx2 = dAdx.diff(x)
dAdx2.subs(x,sol_x[0]).simplify(),dAdx2.subs(x,sol_x[1]).simplify()
```

Uma vez que a segunda solução verifica a concavidade positiva, temos que o ponto crítico $x^{*}$ é:

```python
# concavidade para cima => ponto de mínimo
xs = sol_x[1]
xs
```

Usando este valor na equação da elipse, obtemos a ordenada correspondente:

```python
# resolve para y > 0
ys = sy.solve(xs**2/a**2 + y**2/b**2 - 1,y)[1]
ys
```

Por fim, substituindo $x^{*}$ na expressão da área, temos que $A_{max}$ é:

```python
# área máxima
A_max = A.subs(x,xs)
A_max
```

ou, de forma, simplificada,

```python
# simplificando
A_max.simplify()
```

### Conclusão

A área do retângulo inscrito na elipse será máxima quando $x = \frac{\sqrt{2}}{2}a$ e $y = \frac{\sqrt{2}}{2}b$. Portanto, $A_{max} = 2ab$, para comprimentos $a$ e $b$ de semi-eixo maior e menor.

## Estudo paramétrico de geometria

No gráfico abaixo, plotamos a variação das áreas de retângulos inscritos em uma elipse arbitrária com semi-eixos $a$ e $b$ em função do comprimento $x$ da meia-base do retângulo até o limite da meia-base do retângulo de área máxima. Adicionalmente, plotamos a variação do comprimento da diagonal do retângulo. A constante $A_{elip}$ é a área da elipse.

Você pode alterar os parâmetros de construção de elipse, o número de valores para $x$ e realizar uma nova análise dos parâmetros.

```python
# semi-eixos da elipse
a,b = 10,2

# no. de retângulos inscritos
nx = 40

# base variável do retângulo
X = np.linspace(0,np.sqrt(2)/2*a,nx)

# área da elipse
e = np.pi*a*b

# áreas dos retângulos
R = []
H = []
for x in X:
    y = b*np.sqrt(1 - x**2/a**2)
    r = 4*x*y
    h = np.hypot(2*x,2*y) # diagonal do retângulo
    R.append(r)
    H.append(h)

# plotagem
fig,ax1 = plt.subplots(figsize=(6,4))
ax1.plot(X,R,'sb',mec='w',alpha=0.8,label='$A_{ret}(x)$')
ax1.plot(X,np.full(X.shape,2*a*b),'--r',alpha=0.8,label='$A_{max}$')
ax1.plot(X,np.full(X.shape,e),'-',alpha=0.8,label='$A_{elip}$')
ax1.legend(fontsize=10)
# labels
plt.xlabel('$x$ [compr. base ret. inscrito]')
plt.ylabel('$A$ [áreas]');

ax2 = ax1.twinx()
ax2.plot(X,H,'og',mec='w',alpha=0.8,label='$h_{ret}(x)$')
ax2.legend(loc=5,ncol=1,fontsize=10)
plt.ylabel('$h$ [compr. diag ret.]');

plt.suptitle('Variação de áreas e diagonais: elipse x retângulo inscrito\n');
plt.title(f'Elipse: $x^2/({a:.1f})^2 + y^2/({b:.1f})^2 = 1$',fontsize=10);
```


# Modelling Afterpay's customer growth
> How can we model the growth in Afterpay's customer base?

- toc: true
- badges: true
- comments: true
- categories: [Bayesian,PyMC3,Finance,Monte Carlo Simulation,Afterpay]

- image: images/2020-08-13-Modelling-Afterpays-Customer-Growth/header.png

## The Context

> Important:  Obviously I need a disclaimer. If you use anything I say as the basis for any decision, financial or otherwise, you are an idiot.

## The Model

First off, let's load in a bunch of libraries.

```python
%matplotlib inline
import pymc3 as pm
import matplotlib.pyplot as plt
import numpy as np

from io import StringIO
import pandas as pd

import math

plt.rcParams["figure.figsize"] = (10,10)
from warnings import filterwarnings
filterwarnings('ignore')

```

While reading through Afterpay's releases to the markets, I came across this chart, which appears on page 3 of [this](https://www.afterpaytouch.com/images/APT_ASX-Announcement_Q2-FY18_16-Jan-Final-2.pdf) release.

csv_data = StringIO('''afterpay_rev,date,month_count
            0.0,30-sep-2015,0
            0.2,31-dec-2015,3
            0.3,31-mar-2016,6
            0.8,30-jun-2016,9
            1.7,30-sep-2016,12
            4.3,31-dec-2016,15
            6.0,31-mar-2017,18
            10.9,30-jun-2017,21
            14.2,30-sep-2017,24
            22.2,31-dec-2017,27
            ''')

#Underlying sales

csv_data = StringIO('''anz_underlying_sales_value,date,month_count
0,FY15,0
37.3,FY16,12
561.2,FY17,24
2184.6,FY18,36
4314.1,FY19,48
6566.9,FY20,60''')

df = pd.read_csv(csv_data, sep=",")
plt.plot(df.month_count,df.anz_underlying_sales_value,color='r')
plt.ylabel('Millions of customers')
plt.xlabel('Months after launch')
plt.grid()
plt.show()

```python
#Underlying sales
csv_data = StringIO('''anz_underlying_sales_value,date,month_count
0,FY15,0
37.3,FY16,12
561.2,FY17,24
2184.6,FY18,36
4314.1,FY19,48
6566.9,FY20,60''')

df = pd.read_csv(csv_data, sep=",")
plt.plot(df.month_count,df.anz_underlying_sales_value,color='r')
plt.ylabel('Millions of customers')
plt.xlabel('Months after launch')
plt.grid()
plt.show()
```

from scipy.optimize import curve_fit

def sigmoid(x, a, b, l):
     y = l / (1 + np.exp(-b*(x-a)))
     return y

xdata = df.month_count.values
ydata = df.anz_underlying_sales_value.values

popt, pcov = curve_fit(sigmoid, xdata, ydata)

x = np.linspace(0,60, 61)

y = sigmoid(x, *popt)

plt.plot(xdata, ydata, 'o', label='data')
plt.plot(x,y, label='fit')

#section = np.linspace(20,40, 100)
#plt.fill_between(section,sigmoid(section, *popt))

plt.title('ANZ Sales by preceding Financial Year')
plt.xlabel('Months after launch')
plt.ylabel('ANZ Sales ($M AUD)')
plt.legend(loc='best')
plt.show()

```python
from scipy.optimize import curve_fit

def quadratic(x, a, b, c):
     y = a * x**2 + b * x + c
     return y

xdata = df.month_count.values
ydata = df.anz_underlying_sales_value.values

popt, pcov = curve_fit(quadratic, xdata, ydata)

print(popt)
x = np.linspace(0,60, 61)
y = quadratic(x, *popt)

plt.plot(xdata, ydata, 'o', label='data')
plt.plot(x,y, label='fit')

#section = np.linspace(20,40, 100)
#plt.fill_between(section,sigmoid(section, *popt))

plt.title('ANZ Sales by preceding Financial Year')
plt.xlabel('Months after launch')
plt.ylabel('ANZ Sales ($M AUD)')
plt.legend(loc='best')
plt.show()
```

Ok, let's pause for a moment, and actually think about what this curve represents.

It's the sum, or integral of the total sales for the preceding 1 year period.

Now, let's find the derivative of this function, which will give us the instantanious rate of sales, at any given point in time.

```python
xdata = df.month_count.values
ydata = df.anz_underlying_sales_value.values

popt, pcov = curve_fit(quadratic, xdata, ydata)
print(popt)

x = np.linspace(0,60, 61)

y = sigmoid(x, *popt)

plt.plot(xdata, ydata, 'o', label='data')
plt.plot(x,y, label='fit')

section = np.linspace(20,40, 100)

plt.fill_between(section,sigmoid(section, *popt))
plt.title('ANZ Sales by preceding Financial Year')
plt.xlabel('Months after launch')
plt.ylabel('ANZ Sales ($M AUD)')
plt.legend(loc='best')
plt.show()

```

```python
from sympy import *
a,b,c, x,y = symbols('a b c x y')
init_printing(use_unicode=True)
#expr = diff(l / (1 + exp(-b*(x-a))),x)

expr = diff(a*x**2 + b*x + c ,x)
print(expr)

f = lambdify([a,b,x], expr, "numpy")
#print(expr)

```

    2*a*x + b

```python
x = np.linspace(0,60, 61)
y = f(4.59234416e+01,1.09078695e-01,7.94167267e+03,x)
plt.plot(x,y)
plt.show()

plt.plot(np.cumsum(y))
plt.show()

```

## Bayesian

```python
import numpy as np
import scipy

x = df.month_count.values
y = df.anz_underlying_sales_value
f2 = scipy.interpolate.interp1d(x, y, kind='cubic')

z = np.polyfit(x, y, 3)
print(z)

xnew = np.linspace(24, 60, num=61, endpoint=True)

plt.plot(x, y, 'o', xnew, f2(xnew), '--')
plt.legend(['Data','Cubic Fit'], loc='best')
plt.title('ANZ Sales by preceding Financial Year')
plt.xlabel('Months after launch')
plt.ylabel('ANZ Sales ($M AUD)')
plt.show()

```

```python

```

```python
with pm.Model() as model:
    millions_customers = df.anz_underlying_sales_value.values
    x = df.month_count.values.astype(np.float64)
    L =  pm.Uniform('L', lower = 0, upper = 200_000)
    k =  pm.Uniform('k', lower=0, upper=1)
    x0 = pm.Uniform('x0', lower=0, upper=100)

    customers_predicted = L/(1+math.e**(-k*(x-x0)))
    customers = pm.Normal('customers', mu = customers_predicted, sigma = 100, observed = millions_customers)

```

```python
with model:
    trace = pm.sample(draws=10_000,tune=5_000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [x0, k, L]
    Sampling 2 chains, 0 divergences: 100%|██████████| 30000/30000 [00:20<00:00, 1470.55draws/s]
    The acceptance probability does not match the target. It is 0.6988053991948783, but should be close to 0.8. Try to increase the number of tuning steps.
    The number of effective samples is smaller than 25% for some parameters.

```python
pm.traceplot(trace);
```

```python
pm.plot_posterior(trace);
```

```python
pm.summary(trace)
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      L
      8017.789
      405.424
      7310.574
      8803.450
      6.689
      4.757
      3674.0
      3632.0
      3829.0
      3979.0
      1.0

      k
      0.109
      0.007
      0.095
      0.122
      0.000
      0.000
      4266.0
      4266.0
      4233.0
      4638.0
      1.0

      x0
      46.148
      1.254
      43.871
      48.554
      0.020
      0.015
      3759.0
      3734.0
      3897.0
      3852.0
      1.0

## Posterior Predictive Checks
Let's see if our model makes sense. By drawing samples from our model, we can further investigate the potential growth rates of Afterpay.

```python
for i in range(0,1_000):
    x = np.arange(0,61)
    plt.plot(x,trace['L'][i]/(1+math.e**(-trace['k'][i]*(x-trace['x0'][i]))),color='k',alpha=0.01)
plt.plot(df.month_count,df.anz_underlying_sales_value,color='r',label='Reported Customers')

plt.legend()
plt.ylabel('Millions of customers')
plt.xlabel('Months after launch')
plt.show()
```

```python

for i in range(0,10_000):
    x = np.arange(0,61)
    rev = trace['L'][i]/(1+math.e**(-trace['k'][i]*(x-trace['x0'][i])))
    losses = 0.008 * trace['L'][i]/(1+math.e**(-trace['k'][i]*(x-trace['x0'][i])))

    plt.plot(x,rev,color='k',alpha=0.01)

plt.plot(df.month_count,df.anz_underlying_sales_value,color='r',label='Reported Customers')

plt.ylabel('REV')
plt.xlabel('Months after launch')
plt.show()
```

```python

```

    0 0.8 9.078746534844535e-17
    1 0.8768924792462777 0.0053325080670845815
    2 0.9596556382575269 0.011491015297085
    3 1.0484388151895756 0.018512270974545605
    4 1.1433363710266302 0.026419431764464006
    5 1.2443776390617562 0.03521892179569219
    6 1.351517319435952 0.04489751495489178
    7 1.4646268658138089 0.05541989380646474
    8 1.5834874649479926 0.06672696101080594
    9 1.7077852243254332 0.07873517444965368
    10 1.8371091466871534 0.09133713942373317
    11 1.9709523752419522 0.10440361685716969
    12 2.1087170383921188 0.11778699755165799

```python
for shift in range(0,13):
    loss_rates = []
    for i in range(0,10_000):
        x = np.arange(0,31)
        rev = df.anz_underlying_sales_value.values
        losses = 0.008 * df.anz_underlying_sales_value.values

        loss_rate = 100*losses/rev
        loss_rates.append(loss_rate[-1])

    loss_rates = np.array(loss_rates)

    print(shift,np.mean(loss_rates),np.std(loss_rates))

#plt.hist(loss_rates)
#plt.show()
```

    0 0.8000000000000002 2.220446049250313e-16
    1 0.8000000000000002 2.220446049250313e-16
    2 0.8000000000000002 2.220446049250313e-16
    3 0.8000000000000002 2.220446049250313e-16
    4 0.8000000000000002 2.220446049250313e-16
    5 0.8000000000000002 2.220446049250313e-16
    6 0.8000000000000002 2.220446049250313e-16
    7 0.8000000000000002 2.220446049250313e-16
    8 0.8000000000000002 2.220446049250313e-16
    9 0.8000000000000002 2.220446049250313e-16
    10 0.8000000000000002 2.220446049250313e-16
    11 0.8000000000000002 2.220446049250313e-16
    12 0.8000000000000002 2.220446049250313e-16


# 基礎数学

## 集合

- **集合**:
    - 範囲がはっきりした数学的対称の集まり
    - 集合を構成する1つの1つの値を集合の **要素** と呼ぶ

### 集合の表記方法
集合は `{}` の中に要素を並べて書くことがある

$$
A = \{1, 3, 6, 8\}
$$

集合内の要素の順序は意味を持たないため、以下の式が成り立つ

$$
\{1, 3, 6, 8\} = \{6, 3, 1, 8\}
$$

$x$ が $X$ の要素であることを以下のように表す

$$
x\in{X}
$$

逆に $x$ が $X$ の要素でないことを以下のように表す

$$
x\notin{X}
$$

上記 $A$ の例で言えば、以下の式が成り立つ

$$
1,3\in{A} \\
2,4\notin{A}
$$

集合の要素の個数は必ずしも有限である必要はないが、要素を列挙する書き方では無限個の要素を表現できない

そのため、次のような集合の表記を用いることがある

$$
X = \{x | P(x)\}
$$

$P(x)$ は $x$ についての条件であり、この集合 $X$ は $P(x)$ を満たす要素 $x$ で構成されていることを意味する

例として、

$$
A = \{x | x は偶数\}
$$

とすると、

$$
-2,6,8\in{A}
$$

となる

### 空集合
- **空集合**:
    - 要素が一つもない集合
    - 通常 $\theta$ で表す

例えば、以下のような式が成り立つ

$$
\{x | x > 1 かつ x < 0\} = \theta
$$

### 集合の包含関係
集合 $A$ が集合 $B$ に含まれる（$A$ の要素が $B$ の要素でもある）ことを $A\subset{B}$ と表記する

このとき「$A$ は $B$ の **部分集合** である」と呼ぶが、$A\subset{B}$ には $A = B$ の可能性もあることに気をつける

これはつまり、任意の集合について $A\subset{A}$ であるとも言える

また、空集合は任意の集合の部分集合であると言えるため $\theta\subset{A}$ が成り立つ

#### 積集合・和集合・差集合
- **積集合**:
    - 集合 $A$ と $B$ の両方に含まれている要素の集合
    - $A\cap{B}$ と記述する
    - 定義: $A\cap{B} = \{x | x\in{A}\ かつ\ x\in{B}\}$
- **和集合**:
    - 集合 $A$ または $B$ の少なくとも一方に含まれている要素の集合
    - $A\cup{B}$ と記述する
    - 定義: $A\cup{B} = \{x | x\in{A}\ または\ x\in{B}\}$
- **差集合**:
    - 集合 $A$ に含まれるが $B$ には含まれない要素の集合
    - $A - B$ もしくは $ A\setminus{B}$ と記述する
    - 定義: $A - B = \{x | x\in{A}\ かつ\ x\notin{B}\}$

### よく使われる集合記号
- $\mathbb{Z}$
    - 整数の全体を表す集合
    - $\mathbb{Z} = \{x|xは整数\}$
- $\mathbb{N}$
    - 自然数（1以上の整数）の全体を表す集合
    - $\mathbb{N} = \{x|xは自然数\}$
- $\mathbb{Q}$
    - 有理数（整数 $p$, $q$ ($q\neq{0}$) について $\frac{p}{q}$ のように分数で表される数）全体を表す集合
    - $\begin{align}\mathbb{Q} &= \{x|xは整数\} \\ &= \{\frac{p}{q}\ |\ p\in{\mathbb{Z}},\ q\in{\mathbb{Z}-\{0\}}\}\end{align}$
- $\mathbb{R}$
    - 実数（数直線上で表現可能な数）の全体を表す集合
        - ※有理数ではない実数（$\mathbb{R} - \mathbb{Q}$）を無理数と呼ぶ
    - $\mathbb{R} = \{x|xは実数\}$
- $\mathbb{C}$
    - 複素数（2乗すると -1 になるような数を含む数: $u + v\sqrt{-1} \quad (u, v \in{R})$）の全体を表す集合
    - $\begin{align}\mathbb{C} &= \{x|xは複素数\} \\ &= \{u + v\sqrt{-1}\ |\ u, v \in{R}\} \end{align}$

## 数列

- **数列**:
    - 2, 4, 6, 8, ... のように数を並べたもの
    - 数列の $n$ 番目の数を $a_n$ のように表現し、この右下に記述される小さな数を **添字** と呼ぶ
- **一般項**:
    - $a_n$ を $n$ の式で表現したもの
    - 例えば $a_1 = 2, a_2 = 4, a_3 = 6, a_4 = 8, \dots$ のような数列は $a_n = 2n$ という一般項で表される

数列の和を考える時は、以下のような式で表現する

$$
\sum_{n=1}^{N}{a_n} = a_1 + a_2 + \ \dots \ + a_N
$$

同様に、数列の積は以下のように表現される

$$
\prod_{n=1}^{N}{a_n} = a_1 * a_2 * \ \dots \ * a_N
$$

数列の和については、以下の公式が成り立つ（証明は省略）

$$
\begin{align}
&\sum_{i=1}^n{(a_i + b_i)} = \sum_{i=1}^n{a_i} + \sum_{i=1}^n{b_i} \\
&\sum_{i=1}^n{ka_i} = k\sum_{i=1}^n{a_i} \\
&\sum_{i=1}^n{i} = \frac{1}{2}n(n+1) \\
&\sum_{i=1}^n{i^2} = \frac{1}{6}n(n+1)(2n+1)
\end{align}
$$

## 写像と関数

### 写像
- **写像**:
    - 集合 $X$, $Y$ について、集合 $X$ の要素を1つ選んだときに、それに対して $Y$ の要素を1つ対応付ける規則のこと
    - 写像は $f$ などの記号で表し、$x\in{X}$ に対応付けられる $Y$ の要素を $f(x)$ と記述する
    - 定義: $f\ :\ X \rightarrow Y$
        - このとき写される元の集合 $X$ は写像 $f$ の **定義域** と呼ぶ

$x$ に対して $y=f(x)$ が対応付けられる時、$y$ は $x$ の **像** であると言い、$x$ は $y$ の **現像** であると言う

$x$ が $y$ に写される（$y=f(x)$ である）時、以下のように表現することもある

$$
f\ :\ x \mapsto y
$$

$X$ の部分集合 $A$ に対してそれが写像 $f$ で写される先の全体を $f(A)$ と書く

これはつまり、以下のような関係が成り立つことを示す

$$
f(A) = \{f(x)\ |\ x\in{A}\}
$$

このとき $f(A)$ を $A$ の **像** と呼び、$f(A)$ は関数 $f$ の **値域** と呼ばれる

特に $X$ 全体の像 $f(X)$ を考えると、それは必ずしも $Y$ に一致する必要はなく、一般には $f(X)\subset{Y}$ となる

### 関数
本稿では、写像の写り先が数の集合（$\mathbb{R}$ や $\mathbb{C}$）であるものを特に **関数** と呼ぶ

ここで、

$$
\begin{align}
&f(\mathbb{R}) \rightarrow \mathbb{R} \\
&f\ :\ x \mapsto y
\end{align}
$$

という写像を考えるとき、関数 $f$ は数式で表現されることがある

例えば、

$$
y = f(x) = x^2
$$

という数式で表されると考える

これは $x\in{\mathbb{R}}$ が与えられると、それに対して $x^2\in{\mathbb{R}}$ を対応付けるという意味になる

このような数式による表現はよく用いられるため、「$\mathbb{R} \rightarrow \mathbb{R}$ の関数である」ことを明記せず、暗黙に仮定することが多い

特に本稿では複素数を扱わないため、関数の定義域は $\mathbb{R}$ またはその部分集合とする

ここで $x\mapsto{y}$ であることも暗黙に仮定し、次のような関数を考える

$$
y = \frac{1}{x}
$$

この関数の定義域として $x=0$ は定義されない（$\frac{1}{0}$ という計算が定義されていないため）

そのため、この関数の定義域は暗黙的に $\mathbb{R}-\{0\}$ であると仮定される

定義域を明記する場合は、以下のように区間を表す記号を定義する

$$
[a,b] = \{x|a\leq{x}\leq{b}\}
$$

上記区間は、以下のような表記の仕方をすることもある

$$
\begin{align}
&(a,b) = \{x|a\lt{x}\lt{b}\} \\
&[a,b) = \{x|a\leq{x}\lt{b}\} \\
&(a,b] = \{x|a\lt{x}\leq{b}\}
\end{align}
$$

## SymPy.jl

記号計算プログラムとして有名なものに、Python 用ライブラリの **SymPy** がある

これを PyCall 経由でラッピングした Julia 用パッケージとして　SymPy.jl もある

SymPy は速度的にはあまり速くないため、計算速度が欲しい場合は SymEngine.jl や ModelingTool.jl 等のパッケージを使うほうが良い

しかし、使える機能の充実度の高さから、それほど速度を求めない場合は SymPy.jl を選択しておくのが無難である

```julia
using Pkg

# パッケージインストール時のログ出力先を stdout に変更
Pkg.DEFAULT_IO[] = stdout

# SymPy.jl, Plots.jl パッケージインストール
## Plots.jl: グラフ描画用パッケージ
Pkg.add(["SymPy", "Plots"])
```

        Updating registry at `~/.julia/registries/General`
       Resolving package versions...
      No Changes to `~/github/julia_ml-tuto/Project.toml`
      No Changes to `~/github/julia_ml-tuto/Manifest.toml`

SymPy.jl を用いて $\sum_{i=1}^n{i^2} = \frac{1}{6}n(n+1)(2n+1)$ を記述してみる

```julia
# SymPy.jl を使う
using SymPy

# 変数 i, N を記号処理の対象（シンボル）に
@vars i
@vars N

# i = (1..N)^2 の数列を展開
summation(i^2, (i, 1, N))
```

$\begin{equation*}\frac{N^{3}}{3} + \frac{N^{2}}{2} + \frac{N}{6}\end{equation*}$

```julia
# i = (1..N)^2 の数列を展開し、簡単化する
summation(i^2, (i, 1, N)) |> simplify
```

$\begin{equation*}\frac{N \left(2 N^{2} + 3 N + 1\right)}{6}\end{equation*}$

```julia

```