# Programación Entera: Heurísticas para el Problema de Localización de Servicios

## Descripción

En este trabajo se estudia el problema de localización de servicios de manera detallada, en concreto localización con costos fijos, analizando el problema de programación lineal además de la heurística *ADD*. Todo ello se ha hecho apoyandose en el lenguaje *R* y el solver *glpk*. Por último se ha comprobado la validez de los métodos sobre distintos problemas a modo de ejemplo.

Este documento forma parte de un grupo de trabajos relacionados con la heuristicas para la resolución de problemas de programación entera. Existen otros tres trabajos referidos al estudio del [problema de la asignación](https://nbviewer.jupyter.org/github/garciparedes/integer-programming-heuristics/blob/master/integer-programming-assignment-heuristics.ipynb), al [problema de la mochila](https://nbviewer.jupyter.org/github/garciparedes/integer-programming-heuristics/blob/master/integer-programming-knapsack-heuristics.ipynb) y a [problemas de redes](https://nbviewer.jupyter.org/github/garciparedes/integer-programming-heuristics/blob/master/integer-programming-network-heuristics.ipynb). El contenido completo del trabajo está publicado en el siguiente repositorio: https://github.com/garciparedes/integer-programming-heuristics

## Autor

  * Sergio García Prado - [garciparedes.me](https://garciparedes.me)

## Fecha

  * Mayo de 2018

## Contenidos

  * [Introducción](#Introducción)
  * [Problemas de Localización de Servicios](#Problemas-de-Localización-de-Servicios)
    * [Localización con Costos Fijos](#Localización-con-Costos-Fijos)
  * [Conclusiones](#Conclusiones)

## Introducción

Los problemas de localización de servicios son una clase de problemas de programación lineal mixta (variables enteras y continuas) que permite la resolución de una gran cantidad de problemas de optimización que se dan en nuestra sociedad. A grandes rasgos, estos modelos permiten el estudio de aquellos problemas de decisión basados en la búsqueda del lugar óptimo donde colocar un servicio, de tal manera que se minimice una determinada métrica. Es por ello que su ámbito de aplicación es tan extenso.

El caso prototípico es aquel en que una determinada empresa va a llevar a cabo una ampliación de sucursales. Es por ello que necesita seleccionar la alternativa que mayor beneficio le proporcione. Para ello, puede tratar de definir el problema de distintas maneras, como la de cubrir la zona en que se encuentran todos sus clientes, o más modestamente, una proporción de la misma, así como reducir la distancia según función de error u otras sitituaciones. Las más habituales son las siguientes:

  * Cubrimiento Total (Set Covering).
  * Cubrimiento Máximo.
  * Localización de Medianas.
  * Localización de Centros de Emergencia.
  * Localización con Costos Fijos.
  * Distritos.

En este trabajo, vamos a centrarnos en el problema de *Localización con Costos Fijos*, el cual se basa en la búsqueda de la configuración óptima entre un conjunto de puntos de demanda y un conjunto de puntos de abastecimiento posibles, entre los cuales se debe escoger un subconjunto de los mismos para satisfacer las necesidades de los puntos de demanda.

El resto del trabajo se distribuye de la siguiente manera: Inicialmente se han definido distintas constantes y funciones que se utilizarán en el resto de apartados del trabajo. Posteriormente, se describirá más en detalle el modelo de los problemas de localización de servicios, para después centrarnos en el caso de *Localización de Costes Fijos*, describiendo el modelo en detalle y analizando la heurística *ADD* (muy usada en entornos reales). Después se resolverá el problema *WebBook* describiendo paso a paso las restricciones que vamos añadiendo al problema. Finalmente se discuten distintos aspectos de este tipo de problemas en el apartado de conclusiones.

#### Configuración del Entorno

```R
rm(list = ls())
```

```R
library(slam, warn.conflicts = FALSE)
library(Rglpk, warn.conflicts = FALSE)
library(purrr, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
library(magrittr, warn.conflicts = FALSE)
library(ggplot2, warn.conflicts = FALSE)
library(ggthemes, warn.conflicts = FALSE)
library(scales, warn.conflicts = FALSE)
```

    Using the GLPK callable library version 4.65

```R
options(repr.matrix.max.rows = 600, repr.matrix.max.cols = 600)
```

#### Constantes y Funciones de Apoyo

```R
Solve <- function(...) {
    Rglpk_solve_LP(...)
}
```

```R
FormatUncapacitedFacicilityLocationSolution <- function(s, p) {
    list(
        min.cost = s$optimum,
        ratios = matrix(round(s$solution[1:(p$m * p$n)], digits = 3), nrow = p$n),
        services = matrix(s$solution[(p$m * p$n + 1):(p$n * (1 + p$m))], nrow = 1),
        amount = round(t(t(matrix(s$solution[1:(p$m * p$n)], nrow = p$n)) * p$d), digits = 3)
    )
}
```

## Problemas de Localización de Servicios

Tal y como se ha indicado anteriormente, los problema de localización de servicios tratan de resolver el problema de buscar el lugar óptimo en que colocar puntos de abastecimiento de entre un conjunto de posibles puntos, para cumplir las necesidades de un conjunto de puntos de demanda, tratando de reducir al máximo los costes.

### Localización con Costos Fijos

#### Descripción

El problema de localización con costos fijos se basa en lo siguiente: suponemos que tenemos un conjunto de $m$ puntos de demanda, los cuales tienen una determinada cantidad de demanda denotada por $d_i$ (para representar la demanda del punto $i$-ésimo). Además, existe un conjunto de $n$ posibles puntos de abastecimiento, los cuales se definen por una tupla $(f_j, s_j)$, que representa el coste de fijo de utilización, y la capacidad máxima para el punto $j$-ésimo. Entonces, el problema se basa en decidir la planificación óptima entre puntos de demanda y abastecimiento, teniendo en cuenta los costes fijos de utilización. Además, existe una matriz $C$ de $m$ filas y $n$ columnas, que almacena los costos de utilización del punto de abastecimiento $j$-ésimo por el punto de demanda $i$-ésimo.

Este problema se puede modelizar de distintas maneras, sin embargo una de las más utilizadas utiliza las siguientes variables de decisión: Un vector $X$ de longitud $n$ y naturaleza binaria que indicará si es o no utilizado un determinado punto de abastecimiento (lo cual se utilizará para contabilidar los costos fijos), junto con una matriz $Y$ de $m$ filas y $n$ columnas que de naturaleza continua, que indica el ratio de utilización del punto de abastecimiento $j$-ésimo por el punto de demanda $i$-ésimo. Por tanto, estas variables tomarán valores únicamente en el intervalo $[0, 1]$.

#### Modelo

A continuación se incluye la modelización como problema de programación lineal para el problema de localización con costos fijos. Para ellos, nos hemos apoyado en la notación descrita en el anterior apartado.

\begin{equation}
    \begin{array}{ll@{}ll}
      \text{Minimizar} & \displaystyle \sum\limits_{j=1}^{n} f_{j} \cdot x_{j} + \sum\limits_{i=1}^{m}\sum\limits_{j=1}^{n} c_{ij} \cdot d_{i} \cdot y_{ij} \\
      \text{sujeto a}  & \sum\limits_{j=1}^{n} y_{ij} = 1, & \forall i \in \{1,...,m\}\\
                       & \sum\limits_{i=1}^{m} d_{i} \cdot y_{ij} - s_j \cdot x_{j} \leq 0, & \forall j \in \{1,...,n\}\\
                       & y_{ij} \geq 0, & \forall i \in \{1,...,m\}, \forall j \in \{1,...,n\} \\
                       & x_{j} \in \{0, 1\}, & \forall j \in \{1,...,n\}
    \end{array}
\end{equation}

```R
SolveUncapacitedFacicilityLocationExact <- function(p, filtered = NULL) {

    conditions <- matrix(0, nrow = p$n + p$m, ncol = (1 + p$m) * p$n)

    for (i in 1:p$m) {
        conditions[i, seq(p$n * (i - 1) + 1, length.out = p$n)] <- 1
    }

    for (i in 1:p$n) {
        conditions[p$m + i, seq(i, length.out = p$m, by = p$n)] <- p$d
        if (is.null(filtered) || i %in% filtered) {
            conditions[p$m + i, p$m * p$n + i] <- -p$s[i]
        }
    }

    rhs <- rep(c(1, 0), c(p$m, p$n))
    obj <- c(t(t(p$c) * p$d), p$f)

    dir <- rep(c("==", "<="), c(p$m, p$n))
    types <- rep(c("C", "B"), c(p$m, 1) * p$n)

    s <- Solve(obj = obj,
               mat = conditions,
               dir = dir,
               rhs = rhs,
               types = types)

    return(s)
}

```

#### Heurísticas

Para tratar de resolver el problema de costos fijos utilizando un menor coste computacional, existen distintas heurísticas que proporcionan resultados razonables utilizando un menor coste computacional. En este caso, se va a estudiar la heurística ADD para el problema de *localización de servicios con costos fijos*.

##### ADD

TODO(@garciparedes): Añadir Descripción de la Heurística ADD.

```R
SolveUncapacitedFacicilityLocationADD <- function(p, ...) {
    stop("SolveUncapacitedFacicilityLocationADD() unimplemented yet!")
}
```

#### Ejemplo: WebBook

Web Book vende libros por internet. La administración quiere saber en qué sitios se deben ubicar almacenes de distribución, de entre $5$ sitios posibles, que denotaremos por *A1*, *A2*, *A3*, *A4* y *A5*. La región de ventas está dividida en $5$ zonas indicadas por *NO*, *SO*, *MO*, *SE* y *NE*. Se proporcionan los siguientes datos: la demanda semanal promedio de cada región (en número de libros), el costo promedio de envío de un libro (Euros/libro), el costo fijo semanal (en Euros) si el almacén está en funcionamiento, y la capacidad de cada almacén (en número de libros).

Los datos necesarios para la resolución del problema se definen a continuación, siendo `n` el número de puntos de demanda, `m` el de posibles puntos de abastecimiento, `c` la matriz de costes, `f` los costes fijos, `s` las capacidades de cada punto de abastecimiento y `d` la demanda de cada punto de demanda.

```R
p <- list(n = 5,
          m = 5,
          c = matrix(c(2.40, 3.50, 4.80, 6.80, 5.75,
                       3.25, 2.30, 3.40, 5.25, 6.00,
                       4.05, 3.25, 2.85, 4.30, 4.75,
                       5.25, 6.05, 4.30, 3.25, 2.75,
                       6.95, 5.85, 4.80, 2.10, 3.50),
                     nrow = 5, byrow = TRUE),
          f = c(4000, 3000, 2500, 4000, 3000),
          s = c(20000, 20000, 15000, 25000, 15000),
          d = c(8000, 12000, 9000, 14000, 17000)
)
```

##### WebBook: Cuestión 1

En este apartado se resuelve el problema de manera exacta, proporcionando los siguientes resultados (nótese que la solución obtenida para la matriz $Y$ (o de ratios) es binaria)

```R
s <- SolveUncapacitedFacicilityLocationExact(p)
FormatUncapacitedFacicilityLocationSolution(s, p)
```

$min.cost

165100

$ratios

1	0	0	0	0

0	1	0	0	0

0	0	1	0	0

0	0	0	0	1

0	0	0	1	0

$services

1	1	1	1	1

$amount

8000 	    0	   0 	    0	    0

   0 	12000	   0 	    0	    0

   0 	    0	9000 	    0	    0

   0 	    0	   0 	    0	17000

   0 	    0	   0 	14000	    0

##### WebBook: Cuestión 2

*"Un directivo de la compañía piensa que la mejor opción es abrir los almacenes *A1*, *A2* y *A4*. Para esta opción, calcular manualmente su correspodiente costo."*

En este caso, se restringe el espacio de posibles soluciones a las que se basen en los almacenes 1, 2 y 4. La solución obtenida en este caso es la siguiente (nótese que en este caso las soluciones obtenidas en la matriz $Y$ (o de ratios) no son binarias).

```R
s <- SolveUncapacitedFacicilityLocationExact(p, c(1, 2, 4))
FormatUncapacitedFacicilityLocationSolution(s, p)
```

$min.cost

199850

$ratios

1    	0.083	0    	0    	0.353

0    	0.917	1    	0    	0.000

0    	0.000	0    	0    	0.000

0    	0.000	0    	1    	0.647

0    	0.000	0    	0    	0.000

$services

1	1	0	1	0

$amount

8000 	 1000	   0 	    0	 6000

   0 	11000	9000 	    0	    0

   0 	    0	   0 	    0	    0

   0 	    0	   0 	14000	11000

   0 	    0	   0 	    0	    0

##### WebBook: Cuestión 3

*"Añadir las siguientes restricciones:*
  * *Por proximidad geográfica, los almacenes *A1* y *A3* no deberían estar abiertos simultáneamente.*
  * *Para mantener un equilibrio entre los distribuidores, si al almacén *A4* se le asignan más de $22.000$ libros semanales, los almacenes *A1* y *A2* deben tener asignados, en conjunto, al menos $15.000$ libros.*"

En este caso, vamos a resolver el problema paso a paso, construyendo de manera apropiada la matriz de restricciones. A continuación se construye la matriz de condiciones inicial:

```R
conditions <- matrix(0, nrow = p$n + p$m, ncol = (1 + p$m) * p$n)
for (i in 1:p$m) {
    conditions[i, seq(p$n * (i - 1) + 1, length.out = p$n)] <- 1
}
for (i in 1:p$n) {
    conditions[p$m + i, seq(i, length.out = p$m, by = p$n)] <- p$d
    conditions[p$m + i, p$m * p$n + i] <- -p$s[i]
}
rhs <- rep(c(1, 0), c(p$m, p$n))
obj <- c(t(t(p$c) * p$d), p$f)
dir <- rep(c("==", "<="), c(p$m, p$n))
types <- rep(c("C", "B"), c(p$m, 1) * p$n)

cbind(conditions, dir, rhs)
```

																														dir	rhs

1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	==    	1

8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-20000                                        	0     	0     	0     	0     	<=                                         	0

0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-20000                                        	0     	0     	0     	<=                                         	0

0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-15000                                        	0     	0     	<=                                         	0

0     	0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-25000                                        	0     	<=                                         	0

0     	0     	0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-15000                                        	<=                                         	0

El siguiente paso es añadir la restricción de que únicamente puedan construirse el Almacén A1 o el A3, obteniendo la siguiente matriz de restricciones:

```R
a1 <- rep(0, (1 + p$m) * p$n)
a1[p$m * p$n + c(1, 3)] <- 1
conditions <- rbind(conditions, a1)
dir <- c(dir, "<=")
rhs <- c(rhs, 1)

rownames(conditions) <- NULL
cbind(conditions, dir, rhs)
```

																														dir	rhs

1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	==    	1

8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-20000                                        	0     	0     	0     	0     	<=                                         	0

0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-20000                                        	0     	0     	0     	<=                                         	0

0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-15000                                        	0     	0     	<=                                         	0

0     	0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-25000                                        	0     	<=                                         	0

0     	0     	0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-15000                                        	<=                                         	0

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	0     	1     	0     	0     	<=                                         	1

Finalmente, se añadirán las restricciones necesarias para indicar que si el número de libros almacenados en *A4* supera los $22000$, entonces *A1* y *A2* deberán superar $15000$ libros en conjunto. En este caso es necesario añadir una variable binaria adiccional para modelizar la nueva lógica de implicación. La matriz de restricciones se muestra a continuación:

```R
obj <- c(obj, 0)
conditions <- cbind(conditions, rep(0, nrow(conditions)))
types <- c(types, "B")

a2 <- conditions[4, ]
a2[length(a2)] <- -1
conditions <- rbind(conditions, a2)
dir <- c(dir, "<=")
rhs <- c(rhs, 22000 / p$s[4])

a3 <- colSums(conditions[c(1, 2), ])
conditions <- rbind(conditions, a3)
dir <- c(dir, ">=")
rhs <- c(rhs, 15000/sum(p$s[c(1, 2)]))

rownames(conditions) <- NULL
cbind(conditions, dir, rhs)
```

																															dir	rhs

1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	==    	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	==    	1

8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-20000                                        	0     	0     	0     	0     	0                                             	<=                                         	0

0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-20000                                        	0     	0     	0     	0                                             	<=                                         	0

0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-15000                                        	0     	0     	0                                             	<=                                         	0

0     	0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-25000                                        	0     	0                                             	<=                                         	0

0     	0     	0     	0     	8000                                          	0     	0     	0     	0     	12000                                         	0     	0     	0     	0     	9000                                          	0     	0     	0     	0     	14000                                         	0     	0     	0     	0     	17000                                         	0     	0     	0     	0     	-15000                                        	0                                             	<=                                         	0

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	0     	1     	0     	0     	0                                             	<=                                         	1

0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	-1                                            	<=                                         	0.88

1     	1     	1     	1     	1     	1     	1     	1     	1     	1     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0     	0                                             	>=                                         	0.375

Por último, resolvemos el problema con la matriz de restricciones indicada, obteniendo la siguiente solución, cuya matriz $Y$ es binaria, a pesar de que esta no se ha restringido de tal manera. Por tanto, esta solución también será la que se obtenga modelizando el problema como de fuente única. Los resultados se muestran a continuación

```R
s <- Solve(obj   = obj,
           mat   = conditions,
           dir   = dir,
           rhs   = rhs,
           types = types)
FormatUncapacitedFacicilityLocationSolution(s, p)
```

$min.cost

167900

$ratios

0	0	0	0	0

1	1	0	0	0

0	0	1	0	0

0	0	0	0	1

0	0	0	1	0

$services

0	1	1	1	1

$amount

   0 	    0	   0 	    0	    0

8000 	12000	   0 	    0	    0

   0 	    0	9000 	    0	    0

   0 	    0	   0 	    0	17000

   0 	    0	   0 	14000	    0

##### WebBook: Cuestión 4

*¿Cómo se debe modificar el modelo de distribución si se exige que cada región de ventas debe ser asignada a un único almacén?*

Para modelizar el problema como de fuente única basta con indicar que todas las variables de decisión deberán ser de tipo binario, lo cual se hace a partir de la siguiente orden:

```R
types.unique <- rep("B", (p$m + 1) * p$n)
```

Lo siguiente es resolver el problema, que como vemos, proporciona los mismos resultados que el anterior, ya que este ya era de fuente única:

```R
s <- Solve(obj = obj,
          mat = conditions,
          dir = dir,
          rhs = rhs,
          types = types.unique)
FormatUncapacitedFacicilityLocationSolution(s, p)
```

$min.cost

167900

$ratios

0	0	0	0	0

1	1	0	0	0

0	0	1	0	0

0	0	0	0	1

0	0	0	1	0

$services

0	1	1	1	1

$amount

   0 	    0	   0 	    0	    0

8000 	12000	   0 	    0	    0

   0 	    0	9000 	    0	    0

   0 	    0	   0 	    0	17000

   0 	    0	   0 	14000	    0

## Conclusiones

Los problemas de localización de servicios mediante la programación lineal representan un método de modelización muy enriquecedor, que permite estudiar el problema desde el punto de vista matemático, de manera relativamente sencilla. Estos problemas, presentan una gran complejidad a nivel computacional debido a su carácter de programación binaria, donde la explosión combinatoria hace que para situaciones en que hay que modelizar un gran número de puntos de demanda y abastecimiento, encontrar soluciones en tiempos razonables se convierte en una tarea muy complicada.

Por dichas razones, la literatura ha propuesto distintas heurísticas que tratan de hacer frente a estas situaciones tratando de proporcionar un contrato razonable entre los resultados óptimos que se obtiene a través de métodos exactos, y el uso reducido de recursos computacionales. Tal y como se ha indicado, las heurísticas greedy más básicas son *ADD* y *DROP*, pero a pesar de ello, proporcionan resultados eficientes. Es por ello que estas técnicas deben ser tenidas en cuenta.

## Referencias

  * [TRC13] Team, R.C., 2013. R: A language and environment for statistical computing.
  * [GP18] Sergio García Prado. Programación Entera: Heurísticas, 2018. [github.com/garciparedes/integer-programming-heuristics](https://github.com/garciparedes/integer-programming-heuristics).
  * [SA18] Jesús Sáez Aguado. Programación Entera, 2017/18. Facultad de Ciencias: Departamento de Estadística e Investigación Operativa.
  * [THBSST17] Theussl, S., Hornik, K., Buchta, C., Schwendinger, F., Schuchardt, H. and Theussl, M.S., 2017. Package ‘Rglpk’.


## Truss2D

The problem was proposed in <cite data-cite="truss2d"></cite> for the purpose of investigating rules on Pareto-front.

**Definition**

\begin{align}
\begin{split}
\end{split}
\end{align}

**Optimum**

$$$$

**Contour**

```python
from pymoo.factory import get_problem
from pymoo.visualization.scatter import scatter

pf = get_problem("truss2d").pareto_front()

plot = scatter(title="Pareto-front")
plot.add(pf, s=80, facecolors='none', edgecolors='r')
plot.add(pf, plot_type="line", color="black", linewidth=2)
plot.show()
```

```python
plot.reset()
plot.do()
plot.apply(lambda ax: ax.set_yscale("log"))
plot.apply(lambda ax: ax.set_xscale("log"))
plot.show()
```

```python

```

```python

```


# Resolviendo ecuaciones diferenciales con FEniCS

Juan Luis Cano Rodríguez <juanlu@pybonacci.org>

Python Madrid, 2015-01-28

# Índice

1. Introducción
2. Arquitectura
3. Instalación
4. Ejemplo básico
5. Problemas más complejos
6. Recursos y alternativas
7. Conclusiones
8. Q & A

# 1. Introducción

## ¿Qué es FEniCS?

* FEniCS es una colección de bibliotecas libres para la **solución de ecuaciones diferenciales por el método de los elementos finitos**

* La ecuación se especifica en forma variacional utilizando un lenguaje simbólico
* FEniCS automatiza la discretización de la ecuación diferencial
* Para la resolución del sistema utiliza *solvers* establecidos como UMFPACK o PETSc
* Enfoque muy matemático: artesanal y potente

## Método de los elementos finitos

* Método numérico para la **aproximación de ecuaciones diferenciales**
* Se particiona el espacio en subdominios llamados **elementos**, y en ellos se aproximan las ecuaciones

# 2. Arquitectura

* DOLFIN 0.2.0 en 2002: inicialmente una sola biblioteca en C++
* En 2003 nace el proyecto FEniCS y se empieza a diversificar
* Se van introduciendo componentes, *algunos se retiran*
* Se libera FEniCS 1.0 en diciembre de 2011
* **FEniCS 1.5 el 12 de enero de 2015**

## Componentes

## Unified Form Language

* En FEniCS, las formas variacionales se expresan mediante un **lenguaje simbólico**
* **UFL** (Unified Form Language) es este lenguaje
* Se pretende compilar esas formas para que se evalúen más rápido
* **UFC** (Unified Form Compiler) es el código C++ generado
* **FFC** (FEniCS Form Compiler) es quien se encarga de esta compilación

## FFC

* FFC toma la forma variacional expresada en UFL
* **FIAT** (FInite Element Tabulator) genera los elementos finitos sobre el mallado
* **Instant** es una interfaz simplificada para incrustar expresiones C++ en Python que se compilan con SWIG

# 3. Instalación

* Los desarrolladores proporcionan binarios actualizados para Debian/Ubuntu y OS X

* Si tienes otro sistema...

Tras buscar en Google, aparecen algunas opciones que no conducen a ninguna parte.

* Dorsal ya no está soportado: no funcionará en todos los sistemas ni con versiones nuevas
* **fenics-install.sh**: nuevo script basado en Hashdist, de creación muy reciente

`curl -s http://fenicsproject.org/fenics-install.sh | bash`

* Compilar todos los paquetes a mano: **que tengas suerte**

* Posible solución: **Anaconda**

## Instalación con conda

He creado un **paquete conda** con la mayoría de las dependencias para que FEniCS sea lo más fácil de instalar posible:

https://github.com/Juanlu001/fenics-recipes

```
$ bash
$ conda create -n fenics27 python=2.7
$ source activate fenics27
(fenics27) $ conda install fenics mkl --channel juanlu001
```

Estos comandos instalan binarios precompilados, enlazados con MKL de Intel

## Compilación con conda-build

Además, conda funciona como un build system:

```
$ conda install conda-build
$ conda build boost eigen3 petsc petsc4py instant ufl fiat ffc dolfin fenics --python 27
$ conda install fenics mkl --use-local
```

Suficiente rollo, ¡vamos a meternos en harina!

# 4. Ejemplo básico

Resolver **la ecuación de Poisson**

$$ -\nabla^2 u = f \quad u \in \Omega,$$

en el dominio $\Omega = [0, 1] \times [0, 1] \in \mathbb{R}^2$, siendo $f = 100\exp(-((x - 0.5)^2 + (y - 0.5)^2) / 0.02)$ el término forzante

y sometida a la condición de contorno $u = \sin{\left(2 \pi x^2\right)}$ en $\partial\Omega$.

## Calentando motores

```
from fenics import *
#from dolfin import *
```

## Dominio y mallado

* En FEniCS tenemos varias funciones para crear dominios sencillos
* En nuestro caso, queremos `UnitSquareMesh`
* Tenemos que especificar el número de elementos de la partición

```
mesh = UnitSquareMesh(8, 6)
mesh
```

## Espacios de funciones

* En FEM, las soluciones a las ecuaciones se buscan en **espacios de Sobolev**
* Su elección es importante: condiciona el error de la solución
* En FEniCS hay soporte completo para varios:

```
supported_elements
```

    ['Brezzi-Douglas-Fortin-Marini',
     'Brezzi-Douglas-Marini',
     'Crouzeix-Raviart',
     'Discontinuous Lagrange',
     'Discontinuous Raviart-Thomas',
     'Lagrange',
     'Nedelec 1st kind H(curl)',
     'Nedelec 2nd kind H(curl)',
     'Raviart-Thomas']

Para nuestro problema concreto, vamos a utilizar los clásicos **elementos polinómicos de Lagrange de orden 1**:

```
V = FunctionSpace(mesh, 'Lagrange', 1)

u = TrialFunction(V)
v = TestFunction(V)
```

    DEBUG:FFC:Reusing form from cache.

## Condiciones de contorno

* De dos tipos: **naturales** y **esenciales**
* Tenemos que definir sobre qué contorno las aplicamos
* Las primeras se crean usando `DirichletBC`:

```
def boundary(x, on_boundary):
    """El parámetro on_boundary es verdadero si el punto está
    sobre el contorno. Podríamos hacer otro tipo de comprobaciones,
    pero en este caso basta con devolver este mismo valor.

    """
    return on_boundary

u0 = Expression('sin(2 * pi * x[0] * x[0])')
bc = DirichletBC(V, u0, boundary)
```

## Formulación variacional

* Reescribir la ecuación en forma débil es el paso más importante
* Buscamos expresiones de este tipo:

$$a(u, v) = L(v)$$

* Gracias a UFL, la traducción es directa

En nuestro caso:

$$-\nabla^2 u = f$$

$$\begin{align}
-\int_\Omega (\nabla^2 u) v dx & = \int_\Omega \nabla u \cdot \nabla v dx - \oint_{\partial\Omega} \frac{\partial u}{\partial n} v ds \\
& = \int_\Omega \nabla u \cdot \nabla v dx
\end{align}$$

$$\boxed{\displaystyle \int_\Omega \nabla u \cdot \nabla v dx = \int_\Omega f v dx}$$

Y ahora, la escribimos en Python:

```
f = Expression("100 * exp(-(pow(x[0] - 0.5, 2) + pow(x[1] - 0.5, 2)) / 0.02)")
a = inner(nabla_grad(u), nabla_grad(v)) * dx  # Miembro izquierdo
L = f * v * dx
```

### Resolvemos el sistema

* FEniCS tiene interfaces a multitud de solvers
* Necesitamos una función sobre la que escribir la solución
* En el caso más sencillo, llamamos simplemente a la función `solve`:

```
u = Function(V)
solve(a == L, u, bc)

print max(abs(u.vector().array()))  # Array de valores numéricos
```

    DEBUG:FFC:Reusing form from cache.
    DEBUG:FFC:Reusing form from cache.

    1.68671201474

## Los colorines

```
fig = plot(u, interactive=False)
fig.write_png("poisson")
```

```
from IPython.display import Image
Image("poisson.png")
```

# 5. Problemas más complejos

* La formulación variacional puede ser bastante complicada\*

$$ \begin{split}\sum_{K \in \mathcal{T}} \int_{K} \nabla^{2} u \nabla^{2} v \, dx \
- \int_{\Gamma} \left<\nabla^{2} u \right>  [\!\![ \nabla v ]\!\!] \, ds
- \int_{\Gamma}[\!\![  \nabla u ]\!\!]  \left<\nabla^{2} v \right>  \, ds
+ \int_{\Gamma} \frac{\alpha}{h}[\!\![ \nabla u ]\!\!]  [\!\![ \nabla v ]\!\!]  \, ds
= \int_{\Omega} vf \, dx \quad \forall \ v \in V\end{split} $$

\*[Formulación con funciones *penalty* para la ecuación biharmónica](http://fenicsproject.org/documentation/dolfin/1.5.0/python/demo/documented/biharmonic/python/documentation.html)

* Para formulaciones mixtas, se pueden crear espacios de funciones combinando varios

* Se puede trocear el dominio y aplicar condiciones naturales y esenciales

* A veces es necesario especificar los parámetros del *solver* para mejorar el proceso de la solución

* Para geometrías más complicadas hay que acudir a programas externos, comom **Gmsh**

# 6. Recursos y alternativas

* Introducción en Pybonacci: http://pybonacci.org/2015/01/20/fenics-resolucion-de-ecuaciones-diferenciales-en-python/
* Libro de FEniCS: http://fenicsproject.org/book/
* FEniCS Q&A: http://fenicsproject.org/qa/

Existen otras soluciones para resolver ecuaciones diferenciales con Python:

* **SfePy**, también por elementos finitos, enfoque más estructural
* **FiPy**, volúmenes finitos
* **Clawpack**, volúmenes finitos orientado a ecuaciones hiperbólicas

# Conclusiones

* FEniCS es un proyecto muy ambicioso y sus bibliotecas muy potentes
* Instalarlo ya no es un problema ;)
* La dificultad está en la parte matemática
* Libro un poco desactualizado, por lo demás muy detallado
* ¡Esto es solo la punta del iceberg!

Muchas gracias :)

# Q & A

2015-01_fenics/Resolviendo ecuaciones diferenciales con FEniCS.ipynb


```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
from pyodesys.symbolic import PartiallySolvedSystem, ScaledSys, symmetricsys, get_logexp
from pyodesys.plotting import plot_result as _plot_res
from pycvodes import fpes
from chempy import ReactionSystem, Substance
from chempy.kinetics.ode import get_odesys
sp.init_printing()
%matplotlib inline
```

```python
rsys = ReactionSystem.from_string(
    """
    A -> B; 0.04
    B + C -> A + C; 1e4
    2 B -> B + C; 3e7
    """,
    substance_factory=lambda formula: Substance(formula, composition={1: 1}),
    substance_keys='ABC')
```

```python
rsys.composition_balance_vectors()
```

```python
dep_scaling = 1e8
orisys, extra = get_odesys(rsys, description='original')
scaledsys = ScaledSys.from_other(orisys, dep_scaling=dep_scaling, description='scaled')
```

```python
orisys.linear_invariants, orisys.names, orisys.latex_names
```

```python
indep_end = 1e18
init_conc = {'A': 1, 'B': 0, 'C': 0}
integrate_kw = dict(integrator='cvode', atol=1e-6, rtol=1e-6, nsteps=5000, record_rhs_xvals=True, record_jac_xvals=True)

def integrate_systems(systems, **kwargs):
    for k, v in integrate_kw.items():
        if k not in kwargs:
            kwargs[k] = v
    return [odesys.integrate(indep_end, init_conc, record_order=True, record_fpe=True,
                             return_on_error=True, **kwargs) for odesys in systems]

def descr_str(info):
    keys = 'n_steps nfev time_cpu success'.split()
    if isinstance(info, dict):
        nfo = info
    else:
        nfo = {}
        for d in info:
            for k in keys:
                if k in nfo:
                    if k == 'success':
                        nfo[k] = nfo[k] and d[k]
                    else:
                        nfo[k] += d[k]
                else:
                    nfo[k] = d[k]
    return ' (%d steps, %d rhs evals., %.5g s CPUt)' % tuple([nfo[k] for k in keys[:3]]) + (
        '' if nfo['success'] else ' - failed!')

def plot_result(description, res, ax=None, vline_post_proc=None, colors=('k', 'r', 'g'), xlim=None):
    if ax is None:
        ax = plt.subplot(1, 1, 1)
    vk = 'steps rhs_xvals jac_xvals'.split()  # 'fe_underflow fe_overflow fe_invalid fe_divbyzero'
    res.plot(xscale='log', yscale='log', ax=ax, c=colors,
             info_vlines_kw=dict(vline_keys=vk, post_proc=vline_post_proc))
    lines = ax.get_lines()
    for idx, val in enumerate([1 - 0.04*1e-7, 0.04*1e-7]):
        ax.plot(1e-7, val, marker='o', c=colors[idx])
    for idx, val in enumerate([0.2083340149701255e-7, 0.8333360770334713e-13, 0.9999999791665050]):
        ax.plot(1e11, val, marker='o', c=colors[idx])
    ax.legend(loc='best')
    ax.set_title(description + descr_str(res.info))
    if xlim is not None:
        ax.set_xlim(xlim)

def plot_results(systems, results, axes=None, **kwargs):
    if axes is None:
        _fig, axes = plt.subplots(2, 2, figsize=(14, 14))
    for idx, (odesys, res) in enumerate(zip(systems, results)):
        plot_result(odesys.description, res, ax=axes.flat[idx], **kwargs)
```

```python
plot_results([orisys, scaledsys], integrate_systems([orisys, scaledsys], first_step=1e-23),
             axes=plt.subplots(1, 2, figsize=(14, 7))[1], xlim=(1e-23, indep_end))
```

```python
psys = [PartiallySolvedSystem.from_linear_invariants(scaledsys, [k], description=k) for k in 'ABC']
scaled_linear = [scaledsys] + psys
[cs.dep for cs in psys]
```

Switch formulation using roots:

```python
orisys['A']
```

```python
rssys_A = PartiallySolvedSystem.from_linear_invariants(
    scaledsys, ['A'], description='A root finding',
    roots=[10*scaledsys['A'] - scaledsys['C']])
```

```python
res_A = rssys_A.integrate(indep_end, init_conc, return_on_root=True, **integrate_kw)
res_C = psys[2].integrate(indep_end - res_A.xout[-1], res_A.yout[-1, :], **integrate_kw)
```

```python
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
plt_kw = dict(xscale='log', yscale='log')
res_A.plot(**plt_kw, ax=axes[0])
res_C.plot(**plt_kw, ax=axes[1])
_plot_res(np.concatenate((res_A.xout, res_C.xout[1:] + res_A.xout[-1])),
          np.concatenate((res_A.yout, res_C.yout[1:])),
          ax=axes[2], names='ABC', **plt_kw)
for descr, ax, info in zip(['A', 'C', 'A/C'], axes, [res_A.info, res_C.info, (res_A.info, res_C.info)]):
    ax.legend(loc='best', prop={'size': 11})
    ax.set_title(descr + descr_str(info))
    ax.set_xlim([1e-9, indep_end])
```

Not switching:

```python
linresults = integrate_systems(scaled_linear)
```

```python
plot_results(scaled_linear, linresults, xlim=(1e-9, indep_end))
```

```python
plot_result('scaled A/C ', rssys_A.integrate(
        indep_end, init_conc, return_on_root=True, **integrate_kw).extend_by_integration(
            indep_end, odesys=scaled_linear[3], **integrate_kw), xlim=(1e-9, indep_end))
```

## Logarithmic

```python
indep0 = 1e-26
def symmetric_factory(exprs_process_cb=lambda exprs: [sp.powsimp(expr.expand(), force=True) for expr in exprs]):
    return symmetricsys(get_logexp(1, 1e-20, b2=0), get_logexp(1, indep0, b2=0),
                        exprs_process_cb=exprs_process_cb, check_transforms=False)
LogLogSys_together = symmetric_factory()
stlogsystems = [LogLogSys_together.from_other(ls, description='stl ' + ls.description) for ls in scaled_linear]
```

```python
stlogresults = integrate_systems(stlogsystems, nsteps=500)
```

```python
plot_results(stlogsystems, stlogresults, vline_post_proc=lambda x: np.abs(np.exp(x) - indep0), xlim=[indep0*10, indep_end])
```

```python
stlogsystems[0].exprs
```

```python
unscaled_linear = [orisys] + [PartiallySolvedSystem.from_linear_invariants(
        orisys, [k], description=k) for k in 'ABC']
utlogsystems = [LogLogSys_together.from_other(ls, description='utl ' + ls.description) for ls in unscaled_linear]
utlogresults = integrate_systems(utlogsystems, nsteps=500)
plot_results(utlogsystems, utlogresults, vline_post_proc=lambda x: np.abs(np.exp(x) - indep0), xlim=[indep0*10, indep_end])
```

```python
utlogsystems[0].exprs
```

```python
LogLogSys_apart = symmetric_factory(None)
salogsystems = [LogLogSys_apart.from_other(ls, description='sal ' + ls.description) for ls in scaled_linear]
salogresults = integrate_systems(salogsystems, nsteps=500)
plot_results(salogsystems, salogresults, vline_post_proc=lambda x: np.abs(np.exp(x) - indep0), xlim=[indep0*10, indep_end])
```

```python
ualogsystems = [LogLogSys_apart.from_other(ls, description='ual ' + ls.description) for ls in unscaled_linear]
ualogresults = integrate_systems(ualogsystems, nsteps=500)
plot_results(ualogsystems, ualogresults, vline_post_proc=lambda x: np.abs(np.exp(x) - indep0), xlim=[indep0*10, indep_end])
```

```python
rsalogsys_A = LogLogSys_apart.from_other(rssys_A, autonomous_interface=True)
assert rsalogsys_A.autonomous_interface
rsalogsys_A.roots, rsalogsys_A.exprs
```

```python
res_log_switch = rsalogsys_A.integrate(indep_end, init_conc, return_on_root=True, **integrate_kw)
res_log_switch.extend_by_integration(indep_end, odesys=salogsystems[3], return_on_error=True,
                                     autonomous=True, **integrate_kw)
fig, ax = plt.subplots(1, 1, figsize=(14, 4))
ax.axvline(res_log_switch.xout[res_log_switch.info['root_indices'][0]], ls='--')
res_log_switch.plot(ax=ax, xscale='log', yscale='log')
ax.legend(loc='best', prop={'size': 11})
ax.set_title('log A/C ' + descr_str(res_log_switch.info))
_ = ax.set_xlim([1e-9, indep_end])
```

```python
fig, ax = plt.subplots(1, 1, figsize=(16, 4))
plot_result('log A/C', res_log_switch, ax, vline_post_proc=lambda x: np.abs(np.exp(x) - indep0))
_ = ax.set_xlim([1e-20, 1e12])
```


# Neuromatch Academy: Week 1, Day 3, Tutorial 4
# Model Fitting: Multiple linear regression and polynomial regression

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith, Ella Batty

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault, Michael Waskom

---
#Tutorial Objectives

This is Tutorial 4 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will generalize the regression model to incorporate multiple features.
- Learn how to structure inputs for regression using the 'Design Matrix'
- Generalize the MSE for multiple features using the ordinary least squares estimator
- Visualize data and model fit in multiple dimensions
- Fit polynomial regression models of different complexity
- Plot and evaluate the polynomial regression fits

```python
#@title Video 1: Multiple Linear Regression and Polynomial Regression
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV11Z4y1u7cf', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video
```

    Video available at https://www.bilibili.com/video/BV11Z4y1u7cf

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure Settings
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper Functions

def plot_fitted_polynomials(x, y, theta_hat):
  """ Plot polynomials of different orders

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    theta_hat (dict): polynomial regression weights for different orders
  """

  x_grid = np.linspace(x.min() - .5, x.max() + .5)

  plt.figure()

  for order in range(0, max_order + 1):
    X_design = make_design_matrix(x_grid, order)
    plt.plot(x_grid, X_design @ theta_hat[order]);

  plt.ylabel('y')
  plt.xlabel('x')
  plt.plot(x, y, 'C0.');
  plt.legend([f'order {o}' for o in range(max_order + 1)], loc=1)
  plt.title('polynomial fits')
  plt.show()

```

---
# Section 1: Multiple Linear Regression

Now that we have considered the univariate case and how to produce confidence intervals for our estimator, we turn to the general linear regression case, where we can have more than one regressor, or feature, in our input.

Recall that our original univariate linear model was given as

\begin{align}
y = \theta x + \epsilon
\end{align}

where $\theta$ is the slope and $\epsilon$ some noise. We can easily extend this to the multivariate scenario by adding another parameter for each additional feature

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_1 x_2 + ... +\theta_d x_d + \epsilon
\end{align}

where $\theta_0$ is the intercept and $d$ is the number of features (it is also the dimensionality of our input).

We can condense this succinctly using vector notation for a single data point

\begin{align}
y_i = \boldsymbol{\theta}^{\top}\mathbf{x}_i + \epsilon
\end{align}

and fully in matrix form

\begin{align}
\mathbf{y} = \mathbf{X}\boldsymbol{\theta} + \mathbf{\epsilon}
\end{align}

where $\mathbf{y}$ is a vector of measurements, $\mathbf{X}$ is a matrix containing the feature values (columns) for each input sample (rows), and $\boldsymbol{\theta}$ is our parameter vector.

This matrix $\mathbf{X}$ is often referred to as the "[design matrix](https://en.wikipedia.org/wiki/Design_matrix)".

For this tutorial we will focus on the two-dimensional case ($d=2$), which allows us to easily visualize our results. As an example, think of a situation where a scientist records the spiking response of a retinal ganglion cell to patterns of light signals that vary in contrast and in orientation. Then contrast and orientation values can be used as features / regressors to predict the cells response.

In this case our model can be writen as:

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \epsilon
\end{align}

or in matrix form where

\begin{align}
\mathbf{X} =
\begin{bmatrix}
1 & x_{1,1} & x_{1,2} \\
1 & x_{2,1} & x_{2,2} \\
\vdots & \vdots & \vdots \\
1 & x_{n,1} & x_{n,2}
\end{bmatrix},
\boldsymbol{\theta} =
\begin{bmatrix}
\theta_0 \\
\theta_1 \\
\theta_2 \\
\end{bmatrix}
\end{align}

For our actual exploration dataset we shall set $\boldsymbol{\theta}=[0, -2, -3]$ and draw $N=40$ noisy samples from $x \in [-2,2)$. Note that setting the value of $\theta_0 = 0$ effectively ignores the offset term.

```python
#@title
#@markdown Execute this cell to simulate some data

# Set random seed for reproducibility
np.random.seed(1234)

# Set parameters
theta = [0, -2, -3]
n_samples = 40

# Draw x and calculate y
n_regressors = len(theta)
x0 = np.ones((n_samples, 1))
x1 = np.random.uniform(-2, 2, (n_samples, 1))
x2 = np.random.uniform(-2, 2, (n_samples, 1))
X = np.hstack((x0, x1, x2))
noise = np.random.randn(n_samples)
y = X @ theta + noise

ax = plt.subplot(projection='3d')
ax.plot(X[:,1], X[:,2], y, '.')

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()

```

Now that we have our dataset, we want to find an optimal vector of paramters $\boldsymbol{\hat\theta}$. Recall our analytic solution to minimizing MSE for a single regressor:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}.
\end{align}

The same holds true for the multiple regressor case, only now expressed in matrix form

\begin{align}
\boldsymbol{\hat\theta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}.
\end{align}

This is called the [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator.

### Exercise 1: Ordinary Least Squares Estimator

In this exercise you will implement the OLS approach to estimating $\boldsymbol{\hat\theta}$ from the design matrix $\mathbf{X}$ and measurement vector $\mathbf{y}$. You can use the `@` symbol for matrix multiplication, `.T` for transpose, and `np.linalg.inv` for matrix inversion.

```python
def ordinary_least_squares(X, y):
  """Ordinary least squares estimator for linear regression.

  Args:
    x (ndarray): design matrix of shape (n_samples, n_regressors)
    y (ndarray): vector of measurements of shape (n_samples)

  Returns:
    ndarray: estimated parameter values of shape (n_regressors)
  """
  ######################################################################
  ## TODO for students: solve for the optimal parameter vector using OLS
  # Fill out function and remove
  raise NotImplementedError("Student exercise: solve for theta_hat vector using OLS")
  ######################################################################

  # Compute theta_hat using OLS
  theta_hat = ...

  return theta_hat

# Uncomment below to test your function
# theta_hat = ordinary_least_squares(X, y)
# print(theta_hat)
```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_3dc448db.py)

After filling in this function, you should see that $\hat{\theta}$ = [ 0.13861386, -2.09395731, -3.16370742]

Now that we have our $\mathbf{\hat\theta}$, we can obtain $\mathbf{\hat y}$ and thus our mean squared error.

```python
# Compute predicted data
theta_hat = ordinary_least_squares(X, y)
y_hat = X @ theta_hat

# Compute MSE
print(f"MSE = {np.mean((y - y_hat)**2):.2f}")
```

    MSE = 0.91

Finally, the following code will plot a geometric visualization of the data points (blue) and fitted plane. You can see that the residuals (green bars) are orthogonal to the plane.

Extra: how would you check that the residuals are orthogonal to the fitted plane? (this is sometimes called the orthogonality principle, see [least squares notes](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf))

```python
#@title
#@markdown Execute this cell to visualize data and predicted plane

theta_hat = ordinary_least_squares(X, y)
xx, yy = np.mgrid[-2:2:50j, -2:2:50j]
y_hat_grid = np.array([xx.flatten(), yy.flatten()]).T @ theta_hat[1:]
y_hat_grid = y_hat_grid.reshape((50, 50))

ax = plt.subplot(projection='3d')
ax.plot(X[:, 1], X[:, 2], y, '.')
ax.plot_surface(xx, yy, y_hat_grid, linewidth=0, alpha=0.5, color='C1',
                cmap=plt.get_cmap('coolwarm'))

for i in range(len(X)):
  ax.plot((X[i, 1], X[i, 1]),
          (X[i, 2], X[i, 2]),
          (y[i], y_hat[i]),
          'g-', alpha=.5)

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()

```

---
# Section 2:  Polynomial Regression

So far today, you learned how to predict outputs from inputs by fitting a linear regression model. We can now model all sort of relationships, including in neuroscience!

One potential problem with this approach is the simplicity of the model. Linear regression, as the name implies, can only capture a linear relationship between the inputs and outputs. Put another way, the predicted outputs are only a weighted sum of the inputs. What if there are more complicated computations happening? Luckily, many more complex models exist (and you will encounter many more over the next 3 weeks). One model that is still very simple to fit and understand, but captures more complex relationships, is **polynomial regression**, an extension of linear regression.

Since polynomial regression is an extension of linear regression, everything you learned so far will come in handy now! The goal is the same: we want to predict the dependent variable $y_{n}$ given the input values $x_{n}$. The key change is the type of relationship between inputs and outputs that the model can capture.

Linear regression models predict the outputs as a weighted sum of the inputs:

$$y_{n}= \theta_0 + \theta x_{n} + \epsilon_{n}$$

With polynomial regression, we model the outputs as a polynomial equation based on the inputs. For example, we can model the outputs as:

$$y_{n}= \theta_0 + \theta_1 x_{n} + \theta_2 x_{n}^2 + \theta_3 x_{n}^3 + \epsilon_{n}$$

We can change how complex a polynomial is fit by changing the order of the polynomial. The order of a polynomial refers to the highest power in the polynomial. The equation above is a third order polynomial because the highest value x is raised to is 3. We could add another term ($+ \theta_4 x_{n}^4$) to model an order 4 polynomial and so on.

First, we will simulate some data to practice fitting polynomial regression models. We will generate random inputs $x$ and then compute y according to $y = x^2 - x - 2 $, with some extra noise both in the input and the output to make the model fitting exercise closer to a real life situation.

```python
#@title
#@markdown Execute this cell to simulate some data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)
n_samples = 30
x = np.random.uniform(-2, 2.5, n_samples)  # inputs uniformly sampled from [-2, 2.5)
y =  x**2 - x - 2   # computing the outputs

output_noise = 1/8 * np.random.randn(n_samples)
y += output_noise  # adding some output noise

input_noise = 1/2 * np.random.randn(n_samples)
x += input_noise  # adding some input noise

fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');
```

## Section 2.1: Design matrix for polynomial regression

Now we have the basic idea of polynomial regression and some noisy data, let's begin! The key difference between fitting a linear regression model and a polynomial regression model lies in how we structure the input variables.

For linear regression, we used $X = x$ as the input data. To add a constant bias (a y-intercept in a 2-D plot), we use $X = \big[ \boldsymbol 1, x \big]$, where $\boldsymbol 1$ is a column of ones.  When fitting, we learn a weight for each column of this matrix. So we learn a weight that multiples with column 1 - in this case that column is all ones so we gain the bias parameter ($+ \theta_0$). We also learn a weight for every column, or every feature of x, as we learned in Section 1.

This matrix $X$ that we use for our inputs is known as a **design matrix**. We want to create our design matrix so we learn weights for $x^2, x^3,$ etc. Thus, we want to build our design matrix $X$ for polynomial regression of order $k$ as:

$$X = \big[ \boldsymbol 1 , x^1, x^2 , \ldots , x^k \big],$$

where $\boldsymbol{1}$ is the vector the same length as $x$ consisting of of all ones, and $x^p$ is the vector or matrix $x$ with all elements raised to the power $p$. Note that $\boldsymbol{1} = x^0$ and $x^1 = x$

### Exercise 2: Structure design matrix

Create a function (`make_design_matrix`) that structures the design matrix given the input data and the order of the polynomial you wish to fit. We will print part of this design matrix for our data and order 5.

```python
def make_design_matrix(x, order):
  """Create the design matrix of inputs for use in polynomial regression

  Args:
    x (ndarray): input vector of shape (n_samples)
    order (scalar): polynomial regression order

  Returns:
    ndarray: design matrix for polynomial regression of shape (samples, order+1)
  """
  ########################################################################
  ## TODO for students: create the design matrix ##
  # Fill out function and remove
  raise NotImplementedError("Student exercise: create the design matrix")
  ########################################################################

  # Broadcast to shape (n x 1) so dimensions work
  if x.ndim == 1:
    x = x[:, None]

  #if x has more than one feature, we don't want multiple columns of ones so we assign
  # x^0 here
  design_matrix = np.ones((x.shape[0], 1))

  # Loop through rest of degrees and stack columns (hint: np.hstack)
  for degree in range(1, order + 1):
      design_matrix = ...

  return design_matrix

# Uncomment to test your function
order = 5
# X_design = make_design_matrix(x, order)
# print(X_design[0:2, 0:2])
```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_35b1e60c.py)

You should see that the printed section of this design matrix is `[[ 1.         -1.51194917]
 [ 1.         -0.35259945]]`

## Section 2.2: Fitting polynomial regression models

Now that we have the inputs structured correctly in our design matrix, fitting a polynomial regression is the same as fitting a linear regression model! All of the polynomial structure we need to learn is contained in how the inputs are structured in the design matrix. We can use the same least squares solution we computed in previous exercises.

### Exercise 3: Fitting polynomial regression models with different orders

Here, we will fit polynomial regression models to find the regression coefficients ($\theta_0, \theta_1, \theta_2,$ ...) by solving the least squares problem. Create a function `solve_poly_reg` that loops over different order polynomials (up to `max_order`), fits that model, and saves out the weights for each. You may invoke the `ordinary_least_squares` function.

We will then qualitatively inspect the quality of our fits for each order by plotting the fitted polynomials on top of the data. In order to see smooth curves, we evaluate the fitted polynomials on a grid of $x$ values (ranging between the largest and smallest of the inputs present in the dataset).

```python
def solve_poly_reg(x, y, max_order):
  """Fit a polynomial regression model for each order 0 through max_order.

  Args:
    x (ndarray): input vector of shape (n_samples)
    y (ndarray): vector of measurements of shape (n_samples)
    max_order (scalar): max order for polynomial fits

  Returns:
    dict: fitted weights for each polynomial model (dict key is order)
  """

  # Create a dictionary with polynomial order as keys, and np array of theta_hat
  # (weights) as the values
  theta_hats = {}

  # Loop over polynomial orders from 0 through max_order
  for order in range(max_order + 1):

    ##################################################################################
    ## TODO for students: Create design matrix and fit polynomial model for this order
    # Fill out function and remove
    raise NotImplementedError("Student exercise: fit a polynomial model")
    ##################################################################################

    # Create design matrix
    X_design = ...

    # Fit polynomial model
    this_theta = ...

    theta_hats[order] = this_theta

  return theta_hats

# Uncomment to test your function
max_order = 5
#theta_hats = solve_poly_reg(x, y, max_order)
#plot_fitted_polynomials(x, y, theta_hats)

```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_54b1e4e3.py)

*Example output:*

## Section 2.4: Evaluating fit quality

As with linear regression, we can compute mean squared error (MSE) to get a sense of how well the model fits the data.

We compute MSE as:

$$ MSE = \frac 1 N ||y - \hat y||^2 = \sum_{i=1}^N (y_i - \hat y_i)^2 $$

where the predicted values for each model are given by $ \hat y = X \hat \theta$.

*Which model (i.e. which polynomial order) do you think will have the best MSE?*

### Exercise 4: Compute MSE and compare models

We will compare the MSE for different polynomial orders with a bar plot.

```python
mse = np.zeros((max_order + 1))
for order in range(0, max_order + 1):

  X_design = make_design_matrix(x, order)

  ############################################################################
  ## TODO for students: Compute MSE
  #############################################################################

  ## Uncomment below and fill in with your code

  # Get prediction for the polynomial regression model of this order
  #y_hat = ...

  # Compute the residuals
  #residuals = ...

  # Compute the MSE
  #mse[order] = ...

  pass

fig, ax = plt.subplots()

# Uncomment once above exercise is complete
#   ax.bar(range(max_order + 1), mse);

ax.set(title='Comparing Polynomial Fits', xlabel='Polynomial order', ylabel='MSE')
```

[*Click for solution*](https://github.com/erlichlab/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial4_Solution_d8d00e00.py)

*Example output:*

---
# Summary

* Linear regression generalizes naturally to multiple dimensions
* Linear algebra affords us the mathematical tools to reason and solve such problems beyond the two dimensional case

* To change from a linear regression model to a polynomial regression model, we only have to change how the input data is structured

* We can choose the complexity of the model by changing the order of the polynomial model fit

* Higher order polynomial models tend to have lower MSE on the data they're fit with

**Note**: In practice, multidimensional least squares problems can be solved very efficiently (thanks to numerical routines such as LAPACK).

**Suggested readings**

[Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares](http://vmls-book.stanford.edu/)
Stephen Boyd and Lieven Vandenberghe


# Tutorial on Active Inference with `pymdp`

This set of 3 tutorial notebooks aims to be an accessible introduction to discrete-state-space active inference modelling with the `pymdp` package. We assume no prerequisites other than a good grasp of Python and some basic mathematical knowledge (specifically some familiarity with probability and linear algebra). We assume no prior knowledge of active inference. Hopefully, by the end of this series of notebooks, you will understand active inference well enough to understand the recent literature, as well as implement your own agents!

These tutorials will walk you through the specification, and construction of an active inference agent which can solve a simple navigation task in a 2-dimensional grid-world environment. The goal here is to implement the agent 'from scratch'. Specifically, instead of just using magical library functions, we will show you an example of how these functions could be implemented from pure Python and numpy code. The goal at the end of these tutorials is that you understand at a fairly detailed level how active inference works in discrete state spaces and how to apply it, as well as how you could implement a simple agent without using the `pymdp` package. Once you understand what the `pymdp` package aims to abstract, we will go through the structure and functionality offered by the package, and show how you can construct complex agents in a simple and straightforward way.

# What is Active Inference?

Fundamentally, the core contention of active inference is that the brain (and agents in general) can be thought of as fundamentally performing (Bayesian) inference about the world. Specifically, an agent performs two functions.

1.) **Perception**. An agent does not necessarily know the true state of the world, but instead must infer it from a limited set of (potentially ambiguous) observations.

2.) **Action**. Typically, the agent can also perform the actions which change the state of the world. The agent can use these actions to drive the world towards a set of states that it desires.

The theory of Active Inference argues that **both** perception and action can be represented and solved as Bayesian inference problems.

# What is Bayesian Inference?

Bayesian inference provides a recipe for performing *optimal* inference. That is, if you have some set of Hypotheses $H$ and some set of data $D$, then Bayesian inference allows you to compute the *best possible* update to your hypotheses given the data that you have. In other words, the best explanation $H$ that accounts for your actual data $D$.

For instance, suppose you are a trader and you want to know whether some stock will go up tomorrow. And you have a set of information about that stock (for instance earnings reports, sales data, rumours a friend of a friend told you etc). Then Bayesian inference provides the *optimal way* to estimate the probability that the stock will go up. In this scenario, our "hypotheses" $H$ is that the stock will go up, or it will go down and our "data" $D$ is the various pieces of information we hold.

The fundamental equation in Bayesian inference is **Bayes Rule**. Which is
$$
\begin{align}
p(H | D) = \frac{p(D | H)p(H)}{p(D)}
\end{align}
$$
Here $p(H)$ etc are *probability distributions*. All a probability distribution is is a function that assigns a probability value (between 0 and 1) to a specific outcome. A probability distribution then represents the probability of that outcome for every possible outcome. For instance, take p(H). This is the probability distribution over our *hypothesis space*, which you can think of as our baseline assumptions about whether the stock tends to go up or down i.e. $p(H) = [p(stock\_goes\_up), p(stock\_goes\_down)]$, before we've encountered any data that provides evidence for/against either hypothesis.

If we assume we have no idea whether the stock will go up or down, we can say that the probability in each case is 0.5 so that $p(H) = [0.5, 0.5]$. When there is a discrete set of possible outcomes or states, probability distributions can be simply represented as vectors with one element for each outcome - where the element itself is simply the probability of seeing that outcome. The sum of all the elements of a probability distribution must equal 1. The vector's elements encode the probabilities of *all possible events*, so one of them *must* occur -- i.e. the probability of seeing *some* outcome is 100%.

There are three fundamental quantities in Bayesian inference: the **posterior**, the **likelihood** and the **prior**. The posterior is ultimately the goal of inference, it is $p(H | D)$. What this represents is the probability of each hypothesis in the hypothesis space *given the data $D$*. You can think of as you best guesses at the truth of each hypothesis after optimally integrating the data. Next is the **prior** $p(H)$ which represents your assumptions about how likely each hypothesis is *prior to seeing the data*. Finally, there is the likelihood $p(D | H)$ which quantifies how likely each outcome you see is, given the different hypotheses. The likelihood distribution can also be thought of as a *model* for how the data relates to the hypotheses.

The key insight of Bayes rule is simply that the posterior probability -- how likely the hypothesis is, given the data -- is simply the likelihood times the prior. This multiplication can be expressed as computing the following: how likely is the data you actually saw, given the different hypotheses ($P(D = d | H)$), multiplied by the prior probability you assign to each hypothesis ($P(H)$). So the full posterior is a distribution over the different hypotheses - in the case of discrete / Categorical distributions, your posterior $p(H |D)$ will also be a vector of probabilities, e.g. $p(H | D) = [0.75, 0.25]$.

Effectively, hypotheses are more likely if they can predict the data well, but are also weighted by their a-priori probability. The marginal likelihood $p(D)$ is basically just there to normalize the posterior (i.e. make sure it equals 1).

# Generative Models

In Active Inference we typically talk about *generative models* as a core component. But what are generative models? Technically a generative model is simply the product of a likelihood and a prior, for all the possible data points $D$ and hypotheses $H$. This is also known as a *joint distribution* $p(H,D) = p(D | H)p(H)$. The generative model, then, is simply the numerator of Bayes rule, and if we normalize it we can compute posterior probabilities. It is called a generative model because it allows you to *generate* samples of the data. To do so, we follow the following steps:

1.) Sample a hypothesis $h_i$ from the prior -- i.e. $p(H)$

2.) Sample a datum $d_i$ from the likelihood distribution, given the particular hypothesis $h_i$ that you sampled i.e. sample $d_i$ from $p(D | H = h_i)$.

Another way to think about a generative model is that it is simply a model, or set of beliefs/assumptions, of how observed data are generated. This is often a very helpful way to think about inference problems, since it aligns with our notion of causality -- i.e. there are unknown processes in the world which generate data. If we can imagine a process and then imagine the data generated by this process, then we are imagining a generative model. Inference of the posterior, on the other hand, is more difficult because it goes in the *reverse* direction -- i.e. you have some set of observations and want to reconstruct the process that gave rise to them. Fundamentally, all of Bayesian statistics can be broken down into two steps:

1.) Make a mathematical model of the data-generating process - the sort of environmental / world structures you think could give rise to the sort of data you have (i.e. come up with a generative model). Generative models are classically written down using a set of unknown parameters (e.g. parameters that describe probability distributions, like sufficient statistics). You want to "fit" (read: infer the values of) these parameters, given some data.

2.) Given your generative model and some data, compute the posterior distribution over the unknown parameters using Bayes rule, or some approximation of Bayes rule.

3.) Be happy!

All the methods in Bayesian statistics essentially fall into two classes. Coming up with more expressive and powerful generative models and then figuring out algorithms to perform inference on them.

# Why is Bayesian Inference hard?

At this point, you may be wondering: Bayesian inference seems pretty easy. We known Bayes rule. We can invent generative models easily enough. Computing the posterior is just taking the generative model and dividing it by $p(D)$. Why all this fuss? Why do we need to make a whole academic field out of this anyway? Luckily this has a straightforward answer. Bayesian inference is hard for essentially just one reason: that computing $p(D)$, the normalizing constant, is hard.

Let's think about why. $p(D)$, which is known as the **marginal likelihood**, is fundamentally just the probability of the data. What does that mean? How is there just some free-floating probability of the data? Fundamentally there isn't. $p(D)$ is the probability of the data *averaged over all possible hypotheses*. We can write this as,
$$
\begin{align}
p(D) = \sum_h p(D | H)p(H)
\end{align}
$$

Effectively, $p(D)$ is the sum over all possible hypotheses of the probability of the data given that hypothesis, weighted by the prior probability of that hypothesis. This is challenging to compute in practice because you are often using really large (or indeed often infinite) hypothesis spaces. For instance, suppose your trader doesn't just want to know whether the stock will go up or down but *how much* it will go up or down. Now, with this simple change, you have an *infinite* amount of hypotheses: $p(H) = [p(stock\_goes\_up\_0.00001), p(stock\_goes\_up\_0.00002), p(stock\_goes\_up\_0.00003) ...]$. Then, if we want to compute the posterior in this case, we need to sum over every one of this infinite amount of hypotheses. You can see why this ends up being quite challenging in practice.

Because of this intrinsic difficulty, there are a large number of special case algorithms which can solve (or usually approximate) the Bayesian posterior in various cases, and a large amount of work in statistics goes into inventing new ones or improving existing methods. Active Inference agents use one special class of approximate Bayesian inference methods called *variational methods* or *variational inference*. This will be discussed in much more detail in notebook 2.

Beyond merely the difficulty of performing inference, another reason why Bayesian statistics is hard is that you often *don't know* the generative model. Or at least you are uncertain about some aspects of it. There is also a wide class of methods which let you perform inference and *learning* simultaneously, including for active inference, although they won't be covered in this tutorial.

# A Generative Model for an Agent

In Active Inference, we typically go a step beyond the simple cases of Bayesian inference described above, where we have a static set of hypotheses and some static data. We are instead interested in the case of an *agent* interacting with a *dynamic environment*. The key thing we need to add to the formalism to accomodate this is a notion of *time*. We consider an environment consisting of a set of *states* $[x_1, x_2, \dots x_t]$ evolving over time. Moreover, there is an agent which is in the environment over time. This agent receives a set of *observations* $[o_1, o_2 \dots o_t]$, which are a function of the state of the environment, and it can emit *actions* $[a_1, a_2 \dots a_t]$ which can change the state of the environment. The key step to get a handle on this situation mathematically is to define a *generative model* of it.

To start, we must make some assumptions to simplify the problem. Specifically, we assume that the *state* of the environment $x_t$ only depends on the state at the previous timestep $x_{t-1}$ and the action emitted by the agent at the previous timestep $a_{t-1}$. Then, we assume that the observation $o_t$ at a given time-step is only a function of the environmental state at the current timestep $x_t$. Together, these assumptions are often called the **Markov Assumptions** and if the environment adheres to them it is often called a **Markov Decision Process**.

The general computational "flow" of a Markov decision process can be thought of as following a sequence of steps.

1.) The state of the environment is $x_t$.

2.) The environment state $x_t$ generates an observation $o_t$.

3.) The agent receives an observation $o_t$, and based on it (or inferences derived thereof) decides to take some action $a_t$, which it executes in the environment.

4.) Given the current state $x_t$ and the agent's action $a_t$, the environment updates its own state to produce $x_{t+1}$

5.) Go back to step 1.

Now that we have this series of steps, we can try to define what it means mathematically. Specifically, we need to define two quantities.

a.) We need to know how the state of the environment $x_t$ is reflected in the observation sent to the agent $o_t$. In Bayesian terms from the agent's perspective, the true state of the environment is unknown, and its various possible states can be thought of as "hypotheses", while the observations it receives are "data". The agent's generative model encodes some prior assumptions about each possible state $x_t$ (i.e. each hypothesis) relates to the probability of seeing each possible outcome $o_t$. This relationship (from the agent's perspective) is a function known as the **likelihood distribution** $p(D | H)$ or, in our new notation $p(o_t | x_t)$.

b.) We need to know how the environment updates itself to form its new state $x_{t+1}$, given the old one $x_t$ and the action $a_t$. This can be thought of (from the perspective of the agent who does not observe it) as the **prior** since it can be thought of as the default expectation the agent can have about the state of the environment prior to receiving any observations. It can be written as $p(x_t | x_{t-1}, a_{t-1})$. This distribution is also known as the **transition distribution** since it specifies (the agent's assumptions about) how the environment transitions from one state to another.

These two distributions $p(o_t | x_t)$ and $p(x_t | x_{t-1}, a_{t-1})$ are all that is needed to specify the evolution of the environment. At this point, it is necessary to make a distinction between the *actual* evolution of the environment -- known as the **generative process** -- and the *agent's model* of the evolution of the environment, which is known as the **generative model**. These are not necessarily the same, although in some sense the goal of the agent is to figure out a generative model that is as close to the true generative process as possible. In this example, we will consider a scenario in which the agent knows the true model, so the generative model and the generative process are the same, although this is not always the case.

To make this concrete, all that is necessary to do is to specify precisely what the **likelihood** and **transition** distributions actually are.

In the case of discrete state spaces, where it is possible to explicitly enumerate all states, a very generic way of representing these distributions is as *matrices*. Specifically, we can represent the likelihood distribution as a matrix denoted $\textbf{A}$, which is of shape $dimension\_of\_observation \times dimension\_of\_state$. The element $A_{i,j}$ of $\textbf{A}$ represents the probability that observation $i$ is generated by state $j$.

Secondly, we can represent the transition distribution by a matrix $\textbf{B}$ of shape $state\_dim \times state\_dim \times action\_dim$ where element $\textbf{B}_{i,j,k}$ represents the probability of the environment moving to state $i$ given that it was previously in state $j$ and that action $k$ was taken.

In the rest of this notebook, we will explicitly write down the generative process / generative model for a simple grid-world environment in code, to get a better handle on how the environment and the agent's model is specified. In the next notebook, we will turn to inference and action selection and discuss how active inference solves these two tricky problems.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
```

## Constructing a Generative Model

For the rest of this notebook, we will construct a simple generative model for an active inference agent navigating a 3x3 grid world environment. The agent can perform one of 5 movement actions at each time step: `LEFT, RIGHT, UP, DOWN, STAY`. The goal of the agent is to navigate to its preferred position.

We will create matrices for both the environment as well as the agent itself. As we go up levels of abstraction, these environment and generative model matrices will be imported from classes - but this notebook is the lowest level representation of construction, to show how everything is built from the ground up.

## Understanding the state space

The first thing to note is that we are in a 3x3 grid world which means we have 9 states in total. We can define the following mapping to better understand the space.

```python
state_mapping = {0: (0,0), 1: (1,0), 2: (2,0), 3: (0,1), 4: (1,1), 5:(2,1), 6: (0,2), 7:(1,2), 8:(2,2)}
```

All we're doing with this mapping dictionary is assigning a particular index (`0`, `1`, `2`, ..., `8`) to each grid position, which is defined as a pair of `(x, y)` coordinates ( `(0, 0), (1, 0)`, ..., `(2, 2)`). We will use the linear indices to refer to the grid positions in our probability distributions (e.g. `P(o_t | x_t = 5)`), so this `state_mapping` will allow us to easily move between these linear indices and the grid world indices. These kinds of mappings are very handy for intuition and visualization.

And the following heatmap just represents how the coordinates map to the real grid space

```python
grid = np.zeros((3,3))
for linear_index, xy_coordinates in state_mapping.items():
    x, y = xy_coordinates
    grid[y,x] = linear_index # rows are the y-coordinate, columns are the x-coordinate -- so we index into the grid we'll be visualizing using '[y, x]'
fig = plt.figure(figsize = (3,3))
sns.set(font_scale=1.5)
sns.heatmap(grid, annot=True,  cbar = False, fmt='.0f', cmap='crest')
```

## Likelihood Matrix: A

The likelihood matrix represents $P(o_t | x_t)$ , the probability of an observation given a state. In a grid world environment, the likelihood matrix of the agent is identical to that of the environment. It is simply the identity matrix over all states (in this case 9 states, for a 3x3 grid world) which represents the fact that the agent has probability 1 of observing that it is occupying any state x, given that it is in state x. This just means that the agent has full transparency over its own location in the grid.

```python
A = np.eye(9)
```

```python
A
```

We can also plot the likelihood matrix as follows:

```python
labels = [state_mapping[i] for i in range(A.shape[1])]
def plot_likelihood(A):
    fig = plt.figure(figsize = (6,6))
    ax = sns.heatmap(A, xticklabels = labels, yticklabels = labels, cbar = False)
    plt.title("Likelihood distribution (A)")
    plt.show()

```

```python
plot_likelihood(A)
```

## Transition matrix: B

The transition matrix determines how the agent can move around the gridworld given each of the 5 available actions (UP, DOWN, LEFT, RIGHT, STAY).
So the transition matrix will be a 9x9x5 matrix, where each entry corresponds to an end state, a starting state, and the action that defines that specific transition.

To construct this matrix, we have to understand that when the agent is at the edges of the grid, it cannot move outward, so trying to move right at the right wall will cause the agent to stay still.

We will start by constructing a dictionary which we call P, which maps each state to its next state given an action

```python
state_mapping
```

```python
P = {}
dim = 3
actions = {'UP':0, 'RIGHT':1, 'DOWN':2, 'LEFT':3, 'STAY':4}

for state_index, xy_coordinates in state_mapping.items():
    P[state_index] = {a : [] for a in range(len(actions))}
    x, y = xy_coordinates

    '''if your y-coordinate is all the way at the top (i.e. y == 0), you stay in the same place -- otherwise you move one upwards (achieved by subtracting 3 from your linear state index'''
    P[state_index][actions['UP']] = state_index if y == 0 else state_index - dim

    '''f your x-coordinate is all the way to the right (i.e. x == 2), you stay in the same place -- otherwise you move one to the right (achieved by adding 1 to your linear state index)'''
    P[state_index][actions["RIGHT"]] = state_index if x == (dim -1) else state_index+1

    '''if your y-coordinate is all the way at the bottom (i.e. y == 2), you stay in the same place -- otherwise you move one down (achieved by adding 3 to your linear state index)'''
    P[state_index][actions['DOWN']] = state_index if y == (dim -1) else state_index + dim

    ''' if your x-coordinate is all the way at the left (i.e. x == 0), you stay at the same place -- otherwise, you move one to the left (achieved by subtracting 1 from your linear state index)'''
    P[state_index][actions['LEFT']] = state_index if x == 0 else state_index -1

    ''' Stay in the same place (self explanatory) '''
    P[state_index][actions['STAY']] = state_index
```

```python
P
```

From here, we can easily construct the transition matrix

```python
num_states = 9
B = np.zeros([num_states, num_states, len(actions)])
for s in range(num_states):
    for a in range(len(actions)):
        ns = int(P[s][a])
        B[ns, s, a] = 1

```

B is a very large matrix, we can see its shape below, which is as expected:

```python
B.shape
```

We can also visualize B on the plots below. The x axis is the starting state, and the y axis is the ending state, and each plot corresponds to an action given by the title.

```python
fig, axes = plt.subplots(2,3, figsize = (15,8))
a = list(actions.keys())
count = 0
for i in range(dim-1):
    for j in range(dim):
        if count >= 5:
            break
        g = sns.heatmap(B[:,:,count], cmap = "OrRd", linewidth = 2.5, cbar = False, ax = axes[i,j], xticklabels=labels, yticklabels=labels)
        g.set_title(a[count])
        count +=1
fig.delaxes(axes.flatten()[5])
plt.tight_layout()
plt.show()

```

Now our generative model and environment are set up, and we can move on to Notebook 2, where we go through the core mechanics of how to perform inference and planning on this environment with this generative model.