# Ecuaciones Diferenciales Ordinarias

Como es sabido en la [historia de la computación](https://youtu.be/jWum9IJPyks), uno de los problemas que motivo más fuertemente la construcción de computadoras digitales programables fue la solucionar las ecuaciones diferenciales ordinarias que describen la trayectoria de misiles balísticos (esto, claro, durante la segunda guerra mundial).

Dichas ecuaciones, así como muchas otras ecuaciones diferenciales ordinarias, son imposibles de resolver analíticamente, por lo que la única manera de resolverlas es acudir a métodos numéricos que nos permitan obtener una solución aproximada al problema.

Existen [libros enteros](https://books.google.com.mx/books?id=lZysESfSWwgC&printsec=frontcover&dq=leveque+finite+differences&hl=en&sa=X&ved=2ahUKEwi9hJaQ-arrAhVN5awKHXjgDnkQ6AEwAHoECAUQAg#v=onepage&q=leveque%20finite%20differences&f=false), y cursos también, dedicados al estudio de la solución numérica de ecuaciones diferenciales tanto ordinarias como parciales.

## Preludio Computacional 1: alocación de memoria para arreglos

Durante el curso, hemos, en muchas ocasiones, construido arreglos que representan a los elementos de una sucesión, **iniciando con un arreglo que contiene un solo elemento y añadiéndole elementos uno a uno**. Esto se puede ya que, como ya mencionamos anteriormente, los arreglos en Julia son **dinámicos**.

En general, es mala idea tratar con arreglos dinámicos pues es computacionalmente costoso ir asignando o  **alocando** memoria a elementos nuevos. Lo más conveniente, de ser posible, es construir un arreglo de la longitud que necesitemos desde un inicio. Así, podemos tratar al arreglo de manera **estática**: solo cambiamos sus elementos a nuestro gusto pero no cambiamos su longitud

En Julia, existen varias funciones para crear arreglos:
* La función `zeros(n)` crea un arreglo 1D de longitud $n$ en la que cada elemento es un $0$.

* La función `ones(n)` crea un arreglo 1D de longitud $n$ en la que cada elemento es un $1$.

* La función `rand(n)` crea un arreglo 1D de longitud $n$ en la que cada elemento es un número aleatorio en el intervalo $[0,1]$

```julia
arr1 = zeros(10)
println("ceros")
println(arr1)
arr2 = ones(10)
println("unos")
println(arr2)
arr3 = rand(10)
println("aleatorios")
println(arr3)
```

    ceros
    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
    unos
    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
    aleatorios
    [0.7034387397539563, 0.612325513973325, 0.8391447008501354, 0.010999890722896533, 0.4387715344354042, 0.47784317549624356, 0.7138346812026295, 0.6255369959626298, 0.1659882222723219, 0.09009913705543471]

Noten también que `rand()` te genera un solo número aleatorio

```julia
println(rand())
```

    0.10447819693445637

Por default, los arreglos creados con `zeros` y `ones` tienen elementos de tipo `Float64`. Sin embargo, podemos especificar que sean de otro tipo utilizando la sintaxis `zeros(T,n)` o `ones(T,n)` con `T` el tipo de interés

```julia
arr4 = zeros(5)
println(arr4)
println(typeof(arr4))
arr5 = zeros(Int32,5)
println(arr5)
println(typeof(arr5))
```

    [0.0, 0.0, 0.0, 0.0, 0.0]
    Array{Float64,1}
    Int32[0, 0, 0, 0, 0]
    Array{Int32,1}

Podemos usar la prealocación a la hora de construir sucesiones recursivas como las que usamos en la mayoría de métodos numéricos

```julia
# crea $n$ términos de la sucesión de fibonacci
function fibonacciPrealocado(n)
    # creamos arreglo para guardar los $n$ términos
    final = zeros(Int64,n)
    # definimos los primeros elementos
    final[1] = 1
    final[2] = 1
    # creamos los demás, iniciando en el tercero
    for i in 3:n
        final[i] = final[i-1] + final[i-2]
    end
    return final
end
```

    fibonacciPrealocado (generic function with 1 method)

```julia
fibonacciPrealocado(30)
```

    30-element Array{Int64,1}:
          1
          1
          2
          3
          5
          8
         13
         21
         34
         55
         89
        144
        233
          ⋮
       4181
       6765
      10946
      17711
      28657
      46368
      75025
     121393
     196418
     317811
     514229
     832040

## Preludio Computacional II: tuplas

Hasta ahora, hemos trabajado con objetos iterables como listas, arreglos y rangos, los cuales se pueden describir a grandes rasgos como sucesiones ordenadas de objetos. Sabemos que, en caso de las listas y los arreglos podemos cambiar sus elementos

```julia
nombres = ["aldo","david","crhistian"]
nombres[1] = "sayeg"
println(nombres)
```

    ["sayeg", "david", "crhistian"]

Podemos cambiar sus entradas por que las listas y los arreglos son **objetos mutables**, es decir, que pueden cambiar sus elementos. Existen objetos **no mutables** que también representan sucesiones de objetos pero que no podemos cambiar sus elementos. Dichos objetos son las **tuplas** (de tipo `Tuple` en Julia). Una tupla se define con la siguiente sintaxis

```julia
tup1 = (obj1,obj2,obj3)
```

```julia
tupNombres = ("aldo","david","crhistian")
println(tupNombres)
println(typeof(tupNombres))
```

    ("aldo", "david", "crhistian")
    Tuple{String,String,String}

```julia
# al intentar cambiar un elemento arrojará un error
tupNombres[1] = "sayeg"
```

`length` funciona de igual manera con las tuplas

```julia
println(length(tupNombres))
```

    3

Nuevamente, por ser inmutable, no podemos añadirle objetos y `push!` fracasará

```julia
push!(tupNombres,"miguel")
```

### Preludio Computacional III: productos cartesianos

En las siguientes clases trabajaremos con objetos como campos vectoriales y otros análogos, los cuales viven en espacios de varias dimensiones.

Muchas veces nos va a interesar poder obtener el producto cartesiano de de dos arreglos, es decir, un objeto de la forma

$$\{(x,y) \; \mid \; x \in xs \; , \; y \in ys\}$$

Podemos utilizar comprensión de arreglos en Julia para crear ese arreglo con la siguiente sintaxis

```julia
prodCart = [[x,y] for x in xs for y in ys]
```

```julia
# Ejemplo simple
xs = [1,2,3]
ys = [8,9]
# se empieza a iterar por el úlimo arreglo
prodCart = [[x,y] for x in xs for y in ys]
display(prodCart)
```

    6-element Array{Array{Int64,1},1}:
     [1, 8]
     [1, 9]
     [2, 8]
     [2, 9]
     [3, 8]
     [3, 9]

Podemos hacer esto mismo con tuplas:

```julia
nombres = ["aldo","david","miguel"]
apellidos = ["perez","lopez","medrano"]
# ahora tendremos tuplas en el producto cartesiano
prodCar = [(x,y) for x in nombres for y in apellidos]
display(prodCar)
```

    9-element Array{Tuple{String,String},1}:
     ("aldo", "perez")
     ("aldo", "lopez")
     ("aldo", "medrano")
     ("david", "perez")
     ("david", "lopez")
     ("david", "medrano")
     ("miguel", "perez")
     ("miguel", "lopez")
     ("miguel", "medrano")

## Preludio Matemático: Ecuciones Diferenciales I

Sea $x(t): I \subseteq \mathbb{R} \to \mathbb{R}$ una función de clase $C^{1}$, es decir, una función derivable en todo $I$ con derivada continua en $I$. Una ecuación diferencial de primer orden es una ecuación **funcional**  de la siguiente forma

$$
\dot{x}(t) = f(x(t),t)
$$

Donde $\dot{x}(t) = \frac{dx}{dt} (t)$ y $f$ una función continua de dos variables. Notemos que, a diferencia de una ecuación **algebraica** donde las variables representan números o miembros de algún otra estructura algebraica "simple" (vectores, matrices, etc) que solo se operan con operadores algebraicos como suma, producto etc, en una ecuación diferencial algunas variables representan **funciones**.

Más aún, las operaciones que se les realizan a nuestros objetos no son operaciones algebraicas simples si no operaciones más delicadas una derivada o la aplicación de una función arbitraria.

La técnica más estándar para resolver una ecuación diferencial consiste en utilizar el teorema fundamental del cálculo para integrar ambos lados de la ecuación respecto al tiempo. Sin embargo, a la hora de realizar dicha integración, es probable que surgan **constantes de integración**. Para poder determinar dichas constantes, será necesario tener una **condición** sobre la función $x$, es decir, conocer un punto en el que sepamos su valor.

$$
x(t_i) = C
$$

### Problemas de valores iniciales

Aunque el punto $t_i$ puede ser un punto arbitrario del dominio de la función, en general, lo que tenemos que es que ese tiempo es un tiempo que llamamos **inicial** y denotamos $t_0$, y la solución $x(t)$ nos interesa conocerla para tiempos $t \geq t_0$. Cuando ese es el caso, decimos que tenemos un **problema de valores iniciales** descrito por las dos ecuaciones

$$
\begin{cases}
\dot{x}(t) = f(x(t),t) \\
x(t_0) = x_0 \\
\end{cases}
$$

### Campo de direcciones o "retrato fase"

Antes de resolver una ecuación diferencial, podemos darnos una idea del comportamiento que tendrán las soluciones si graficamos el campo de direcciones o retrato fase de la curva: el campo vectorial de las pendientes de las funciones que son solución a la ecuación diferencial pero nonecesariamente a a la condición inicial. Dicho campo corresponde al campo vectorial $P:\mathbf{R}^2 \to \mathbf{R}^2$ definido por

$$
P(t,x) = (1,f(x,t))
$$

Recordando el tutorial de Plots para Julia, podemos graficar un campo vectorial utilizando la función `quiver` del paquete Plots. La sintaxis es

```julia
quiver(xs,ys,vectorfield=(us,vs))
```

La función hace una grafica consistente en flechas entre los puntos `(xs[i],ys[i])` y `(xs[i]+us[i],ys[i]+vs[i])`

### Ejemplo 1:

Sea $\dot{x}(t) = \alpha x(t) $. El campo vectorial entonces es

$$
P(t,x) = ( 1,\alpha x(t) )
$$

```julia
using Plots, LinearAlgebra
```

```julia
function campoVecEjemplo1(X,alpha)
    return [1,X[2]*alpha]
end
function campoDireccionesEjemplo1(alpha)
    # definimos nuestros puntos ts yxs donde van a empezar las flechas
    ts = range(-1,stop=1,length=15)
    xs = range(0,stop=2,length=15)
    # hacemos el producto carteasiano ts x xs para tener puntos en R^2 donde poner flechas
    # la siguiente sintaxis nos construye el producto cartesiano de dos listas en otro
    dom =[[t,x] for t in ts for x in xs]
    # aplicamos el campo vectorial al los elementos del dominio
    imagen = [campoVecEjemplo1(X,alpha) for X in dom]
    # normalizamos cada elemento del campo vectorial para que tenga norma 0.1
    # normalize(arr) es una funcion de LinearAlgebra que nos regresa el vector `arr` normalizado
    imagen = [0.1*normalize(ima) for ima in imagen]
    # hacemos la gráfica del campo vectorial
    quiver([d[1] for d in dom],[d[2] for d in dom],vectorfield=([ima[1] for ima in imagen],[ima[2] for ima in imagen]))
    # otra manera de poner títulos
    # poner `$(var)` en un string remplaza ese pedazo por lo que haya en la variable `var`
    title!("Campo de direcciones para alpha = $(alpha)")
    # otra manera de poner título a los ejes
    xlabel!("t")
    ylabel!("x")
end
```

    campoDireccionesEjemplo1 (generic function with 1 method)

```julia
campoDireccionesEjemplo1(2)
```

## El método de Euler

El método más sencillo para resolver numéricamente un problema de valores iniciales es el **método de Euler**, que consiste en aproximar la derivada $\dot{x}$ con una diferencia finita hacia adelante. Así, obtenemos

$$
\frac{x(t+h) - x(t)}{h} = f(x(t),t)
$$

Despejando encontramos que

$$
x(t+h) = x(t) + h \; f(x(t),t)
$$

Es claro que, si conocemos $x(t_0) = x_0$ y $t_0$, esta regla nos permite calcular $x(t_0 +h)$ y, al tener $x(t_0+h)$ y también $t_0 + h$, posteriormente podemos calcular $x(t_0 + 2h)$ y así sucesivamente para obtener $x(t_0 + k h)$  para cualquier natural $k$. Planteandolo en términos de una sucesión recursiva, sea $t_k = t+ k h$ y entonces $x_{k} := x(t_k)$, entonces la sucesión recursiva del método de Euler es

$$
x_k = \begin{cases}
x_0 & k = 0\\
x_{k-1} + h f(x_{k-1},t_{k-1}) & k > 0\\
\end{cases}
$$

### Ejercicio 1

implementa una función `primerEuler(f,x_0,t_0,t_f,h)` con `f` la función **vectorial** tal que $\dot{x}(t) = f(x(t),t)$, `x_0`, `t_0` la condición inicial, `t_f` el tiempo máximo al que queremos obtener la solución y `h` el tamaño de paso. La función debe de implementar el método de euler y debe de regresar el arreglo de los tiempos `[t_0,t_1, t_2,....,t_n]` con $t_{n}< t_f < t_{n} + h$ y el de la solución a la ecuación diferencial `[x_0,x_1,...,x_n]`.

**Sugerencia:** prealoquen los arreglos para guardar los tiempos y la solución, y también utilicen una tupla para regresar ambos arreglos.

### Ejercicio 2

Utiliza tu función `primerEuler` para resolver las siguientes ecuaciones diferenciales (con las condiciones inciales y tiempo final de tu preferencia):

1. $\dot{x}(t) = 0.3 x(t) $

2. $\dot{x}(t) = 3t^2 - 5t $

3. $\dot{x}(t) = x(t)(1-x(t))$

Grafica el campo de direcciones para cada ecuación. Realiza una gráfica de la solución obtenida y compárala con la solución analítica en la misma gráfica

**Sugerencia**: ten cuidado pues no las condiciones iniciales que pueden utilizarse no son arbitrarias. (Por ejemplo, en la ecuación 1, no pueden tomar $t_0 = 0$, $x_0 = 0$)

### Ejercicio 3

Podemos definir el error absoluto de la solución numérica como la diferencia entre el último valor de la función y su correspondiente analítico, es decir

$$
\text{error} = \mid x_n - x_{\text{analítica}}(t_n) \mid
$$

Para cada una de las funciones del ejercicio anterior, analiza el error absoluto como función del parámetro $h$. Realiza su gráfica y menciona algo sobre su comportamiento

## Ecuaciones de orden mayor: reducción

En la práctica del día a día, en particular en la física, por lo general tratamos con ecuaciones diferenciales de orden mayor a uno. En general, una ecuación diferencial de orden $l$ es una ecuación funcional de la forma

$$
y^{(l)} (t) = g(y^{(l-1)} (t),y^{(l-2)} (t), \ldots, y^{(1)}(t), y(t),t)
$$

Donde $y^{(l)} (t) = \frac{d^l y}{dt^l}$ es la $l$-ésima derivada de $y(t)$. Aunque podríamos desarrollar un método para atacar ecuaciones diferenciales de ese orden, existe un [**truco**](http://mathonline.wikidot.com/converting-nth-order-odes-to-systems-of-n-first-order-odes) para convertir una ecuación de orden $n$ en un sistema de ecuaciones de primer orden con $n$ variables. Definimos

$$
\begin{align}
 x_1(t) &= y(t) \\
 x_2(t) &= y^{(1)}(t) \\
 &\vdots \\
 x_i(t) &= y^{(i-1)}(t) \\
 &\vdots \\
 x_l(t) &= y^{(l-1)}(t)
\end{align}
$$

Entonces la ecuación diferencial se reduce al siguiente sistema

$$
\begin{align}
 \dot{x_1}(t) &= y^{(1)}(t) = x_2(t) \\
 \dot{x_2}(t) &= y^{(2)}(t) = x_3(t) \\
 &\vdots \\
 \dot{x_i}(t) &= y^{(i)}(t) = x_{i+1}(t) \\
 &\vdots \\
 \dot{x}_{l-1}(t) &= y^{(l-1)}(t) = x_{l}(t) \\
\dot{x_l}(t) &= y^{(l)}(t)
\end{align}
$$

Pero

$$
y^{(l)}(t) = g(y^{(l-1)} (t),y^{(l-2)} (t), \ldots, y^{(1)}(t),y(t),t) = g(x_{l}(t),x_{l-1}(t),\ldots,x_1(t),t)
$$

Por lo que el sistema se vuelve

$$
\begin{align}
 \dot{x_1}(t) &= x_2(t) \\
 \dot{x_2}(t) &= x_3(t) \\
 &\vdots \\
 \dot{x_i}(t) &= x_{i+1}(t) \\
 &\vdots \\
 \dot{x}_{l-1}(t) &= x_{l}(t) \\
\dot{x_l}(t) &= g(x_{l}(t),\ldots,x_1(t),t)
\end{align}
$$

### Ejercicio 4

Analíticamente, convierte las siguientes ecuaciones diferenciales de orden $n$ en sistemas de $n$ ecuaciones diferenciales:

1. $y''(t) = -\frac{10}{m}$

2. $y'''(t) = t + 2y(t) + 3y''(t)$

3. $y''(t) = -\frac{k}{m} y(t) - \frac{c}{m} y'(t)$

## Método de euler vectorial

Sea entonces **el vector de funciones**  $\mathbf{x}(t) : \mathbb{R} \to \mathbb{R}^{l}$

$$\mathbf{x}(t) := (x_1(t),\ldots,x_{l}(t))$$

y la función vectorial $\mathbf{f}(\mathbf{x}(t),t) : \mathbb{R}^{l+1} \to \mathbb{R}^{l}$

$$\mathbf{f}(\mathbf{x}(t),t) = (x_2(t),\ldots,x_{l}(t),g(x_{l}(t),x_{l-1}(t),\ldots,x_1(t),t))$$

Podemos escribir el sistema de ecuaciones diferenciales de forma vectorial como

$$
\dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t),t)
$$

Que nuevamente admite una solución mediante el método de euler al aproximar la derivada del vector $\mathbf{x}(t)$ con una diferencia hacia adelante:

$$
\mathbf{x}(t+h) = \mathbf{x}(t) + h \; \mathbf{f}(\mathbf{x}(t),t)
$$

### Ejercicio 5

Implementa una función `eulerVec(f,x_0,t_0,t_f,h)` con `f` la función **vectorial** tal que $\dot{\mathbf{x}}(t) = \mathbf{f}(\mathbf{x}(t),t)$, `x_0` (un vector de $n$ entradas) y `t_0` la condición inicial, `t_f` el tiempo máximo al que queremos obtener la solución y `h` el tamaño de paso. La función debe de implementar el método de euler y debe de regresar el arreglo de los tiempos `[t_0,t_1, t_2,....,t_n]` con $t_{n}< t_f < t_{n} + h$ y el de la solución a la ecuación diferencial `[x_0,x_1,...,x_n]`.

**Sugerencia 1:** utiliza una tupla para regresar ambos valores.

**Sugerencia 2** puedes utilizar `zeros(n,l)` para prealocar una matriz de zeros de $n \times l$ en la cual puedes guardar como renglones cada vector $\mathbf{x}_i$

### Ejercicio 6

Utiliza tu función `eulerVec` para resolver los sistemas de ecuaciones obtenidos en el ejercicio 4. Nuevamente, puedes escoger arbitrariamente las condiciones iniciales, el tiempo final y las constantes de las ecucaiones ($m$,$k$,etc).

Grafica  el **espacio fase** de tus soluciones, es decir, el conjunto

$$\{ \mathbf{x}(t) \; \mid \; t_0 \leq t \leq t_f\} \subset \mathbb{R}^2, \mathbb{R}^3$$

Si es posible, compara con la solución analítica

### Ejercicio 7

El inciso 1 y 3 del ejercicio 4, la variable $y(t)$ corresponde a la posición como función del tiempo para un objeto en **caída libre** o en un **oscilador armónico amortiguado**. Para unas condiciones iniciales, tiempo final y valor de los parámetros, soluciona el sistema y realiza una **animación** del movimiento del objeto (i.e animar $y(t)$ para distintos tiempos). Verifica que coincida con las soluciones que has visto en los cursos de mecánica.

```julia

```


# The Signature Method with Sktime

The ‘signature method’ refers to a collection of feature extraction techniques for multimodal sequential data, derived from the theory of controlled differential equations. In recent years, a large number of modifications have been suggested to the signature method so as to improve some aspect of it.

In the paper ["A Generalised Signature Method for Time-Series"](https://arxiv.org/abs/2006.00873) [1] the authors collated the vast majority of these modifications into a single document and ran a large hyper-parameter study over the multivariate UEA datasets to build a generic signature algorithm that is expected to work well on a wide range of datasets. We implement the best practice results from this study as the default starting values for our hyperparameters in the `SignatureClassifier` module.

## The Path Signature
At the heart of the signature method is the so-called "signature transform".

A path $X$ of finite length in $\textit{d}$ dimensions can be described by the mapping $X:[a, b]\rightarrow\mathbb{R}$ $\!\!^d$, or in terms of co-ordinates $X=(X^1_t, X^2_t, ...,X^d_t)$, where each coordinate $X^i_t$ is real-valued and parameterised by $t\in[a,b]$.

The **signature transform** $S$ of a path $X$ is defined as an infinite sequence of values:
\begin{equation}
    S(X)_{a, b} = (1, S(X)_{a, b}^1, S(X)_{a, b}^2, ..., S(X)_{a, b}^d, S(X)_{a,b}^{1, 1}, S(X)_{a,b}^{1, 2}, ...),
    \label{eq:path_signature}
\end{equation}
where each term is a $k$-fold iterated integral of $X$ with multi-index $i_1,...,i_k$:
\begin{equation}
    S(X)_{a, b}^{i_1,...,i_k} = \int_{a<t_k<b}...\int_{a<t_1<t_2} \mathrm{d}X_{t_1}^{i_1}...\mathrm{d}X_{t_k}^{i_k}.
    \label{eq:sig_moments}
\end{equation}
This defines a graded sequence of numbers associated with a path which is known to characterise it up to a generalised form of reparameterisation [2]. One can think of the signature as a collection of summary statistics that determine a path (almost) uniquely. Furthermore, any continuous function on the path $X$ can be approximated arbitrarily well as a linear function on its signature [3]; the signature unravels the non-linearities on functions on the space of unparameterised paths.

### A Visualisation
To give an idea of what the signature terms represent physically, we consider a patient in an ICU where we are tracking their systolic blood pressure (SBP) and heart rate (HR) changing in time. This can be represented as a path in $\mathbb{R}^3$ (assuming time is included as a channel).

The plot above sketches two scenarios of how such a path might look. We are assuming here an implicit time dimension for each plot such that the path is traversed from left to right along the blue line.

#### Depth 1:
The signature terms to depth 1 are simply the changes of each of the variables over the interval, in the image this is the $\Delta \text{HR}$ and $\Delta \text{SBP}$ terms. Note that these values are the same in each case.

#### Depth 2:
The second level gives us the signed areas (the shaded orange regions), where the orientation of the left most plot is such that the negatively signed area is produced whereas the second gives the positive value, and thus, at order 2 in the signature we now have sufficient information to discriminate between these two situations where in the first rise in heart rate occurs before (or at least, initially faster than) the rise in blood pressure, and vice versa.

#### Depth > 2:
Depths larger than 2 become more difficult to visualise graphically, however the idea is similar to that of the depth 2 case where we saw that the signature produced information on whther the increase in HR or SBP appeared to be happening first, along with some numerical quantification of how much this was happening. At higher orders the signature is doing something similar, but now with three events, rather than two. The signature picks out structural information regarding the order in which events occur.

### The Signature in Time-Series Analysis
The signature is a natural tool to apply in problems related to time-series analysis. As described above it can convert multi-dimensional time-series data into static features that represent information about the sequential nature of the time-series, that can be fed through a standard machine learning model.

A simplistic view of how this works is as follows:
\begin{equation}
    \text{Model}(\text{Signature}(\text{Sequential data}))) = \text{Predictions}
\end{equation}

## Considered Signature Variations
Again, following the work in [1] we group the variations on the siganture method conceptually into:

- **Augmentations** - Transformation of an input sequence or time series into one or more new sequences, so that the signature will return different information about the path.
- **Windows** - Windowing operations, so that the signature can act with some locality.
- **Transforms** - The choice between the signature or the logsignature transformation.
- **Rescalings** - Method of signature rescaling.

This is neatly represented in the following graphic, where $\phi$ represents the augmentation, $W^{i, j}$ the windowing operation, $S^N$ the signature, and $\rho_{\text{pre}/\text{post}}$ the rescalig method.

Please refer to the full paper for a more comprehensive exploration into what each of these groupings means.

## The Sktime Modules
We now give an introduction to the classification and transformation modules included in th sktime interface, along with an example to show how to perform efficient hyperparameter optimisation that was found to give good results in [1].

```python
# Some additional imports we will use
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

from sktime.datasets import load_gunpoint
```

```python
# Load an example dataset
train_x, train_y = load_gunpoint(split="train", return_X_y=True)
test_x, test_y = load_gunpoint(split="test", return_X_y=True)
```

### Overview
We provide the following:
- **sktime.transformers.panel.signature_based.SignatureTransformer** - An sklearn transformer that provides the functionality to apply the signature method with some choice of variations as noted above.
- **sktime.classification.feature_based.SignatureClassifier** - This provides a simple interface to append a classifier to the SignatureTransformer class.

```python
from sktime.classification.feature_based import SignatureClassifier
from sktime.transformations.panel.signature_based import SignatureTransformer
```

### Example 1: Sequential Data -> Signature Features.
Here we will give a very simple example of converting the sequential 3D GunPoint data of shape [num_batch, series_length, num_features] -> [num_batch, signature_features].

```python
# First build a very simple signature transform module
signature_transform = SignatureTransformer(
    augmentation_list=("addtime",),
    window_name="global",
    window_depth=None,
    window_length=None,
    window_step=None,
    rescaling=None,
    sig_tfm="signature",
    depth=3,
)

# The simply transform the stream data
print("Raw data shape is: {}".format(train_x.shape))
train_signature_x = signature_transform.fit_transform(train_x)
print("Signature shape is: {}".format(train_signature_x.shape))
```

    Raw data shape is: (50, 1)
    Signature shape is: (50, 14)

It then becomes extremely easy to build a time-series classification model. For example:

```python
# Train
model = RandomForestClassifier()
model.fit(train_signature_x, train_y)

# Evaluate
test_signature_x = signature_transform.transform(test_x)
test_pred = model.predict(test_signature_x)
print("Accuracy: {:.3f}%".format(accuracy_score(test_y, test_pred)))
```

    Accuracy: 0.747%

### Example 2: Fine Tuning the Generalised Model
As previously mentioned, in [1] the authors performed a large hyperparameter search over the signature variations on the full UEA archive to develop a 'Best Practices' approach to building a model. This required some fine tuning over the following parameters, as they were found to be very dataset specific:
- `depth` over [1, 2, 3, 4, 5, 6]
- `window_depth` over [2, 3, 4]
- `RandomForestClassifier` hyperparamters.

Here we show how this is easily done using the sktime framework.

```python
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold

# Some params
n_cv_splits = 5
n_gs_iter = 20

# Random forests found to perform very well in general
classifier = RandomForestClassifier()

# The grid to be passed to an sklearn gridsearch
signature_grid = {
    # Signature params
    "depth": [1, 2, 3, 4, 5],
    "window_name": ["dyadic"],
    "augmentation_list": [["basepoint", "addtime"]],
    "window_depth": [1, 2, 3, 4],
    "rescaling": ["post"],
    # Classifier and classifier params
    "classifier": [classifier],
    "classifier__n_estimators": [50, 100, 500],
    "classifier__max_depth": [2, 4, 6, 8, 12, 16, 24, 32, 45, 60],
}

# Initialise the estimator
estimator = SignatureClassifier()

# Run a random grid search and return the gs object
cv = StratifiedKFold(n_splits=n_cv_splits)
gs = RandomizedSearchCV(estimator, signature_grid, cv=n_cv_splits, n_iter=n_gs_iter)
gs.fit(train_x, train_y)

# Get the best classifier
best_classifier = gs.best_estimator_

# Evaluate
train_preds = best_classifier.predict(train_x)
test_preds = best_classifier.predict(test_x)
train_score = accuracy_score(train_y, train_preds)
test_score = accuracy_score(test_y, test_preds)
print(
    "Train acc: {:.3f}%  |  Test acc: {:.3f}%".format(
        train_score * 100, test_score * 100
    )
)
```

    Train acc: 100.000%  |  Test acc: 94.667%

## A Full Description of the Parameters
We conclude by giving further explanation of each of the parameters in the `SignatureClassifier` module and what values they can take.

Parameters
----------
Below we list each parameter and the values that they can take. For further details about what the options mean refer to [1].

**classifier** Needs to be any sklearn estimator. Defaults to `RandomForestClassifier()`.

**augmentation_list**: list of tuple of strings, List of augmentations to be applied before the signature transform is applied. These can be any from:
- 'addtime' - Add an equally spaced time channel.
- 'leadlag' - The leadlag transform.
- 'ir' - Perform the invisibility reset transform.
- 'cumsum' - Perform a cumulative sum transform.
- 'basepoint' - Append zero to the start of the path to remove translational invariance.

**window_name** str, The name of the window transform to apply. Can be any of:
- 'global' - A single window over all the data.
- 'expanding' - Multiple windows starting at the first datapoint that extend over the data (increasing width).
- 'sliding' - Multiple windows that slide along the data (fixed width).
- 'dyadic' - Partition the data into dyadic windows.

**window_depth**: int, The depth of the dyadic window. (Active only if `window_name == 'dyadic']`).

**window_length**: int, The length of the sliding/expanding window. (Active only if `window_name in ['sliding, 'expanding'].`)

**window_step**: int, The step of the sliding/expanding window. (Active only if `window_name in ['sliding, 'expanding'].`)

**rescaling**: str, The method of signature rescaling. Any of:
- 'pre' - Rescale the path before the signature transform.
- 'post' - Rescale the path after the signature transform.
- None - No rescaling.

**sig_tfm**: str, String to specify the type of signature transform. Either of: ['signature', 'logsignature'].

**depth**: int, Signature truncation depth.

**random_state**: int, Random state initialisation.

## References
[1] Morrill, James, Adeline Fermanian, Patrick Kidger, and Terry Lyons. "A Generalised Signature Method for Time Series." arXiv preprint arXiv:2006.00873 (2020).

[2] Hambly, B., Lyons, T.: Uniqueness for the signature of a path of bounded variation and the reduced pathgroup. Annals of Mathematics171(1), 109–167 (2010). doi:10.4007/annals.2010.171.10913.

[3] Lyons, T.J.: Differential equations driven by rough signals. Revista Matem ́atica Iberoamericana14(2), 215–310(1998)


# Find worst cases

\begin{equation}
\begin{array}{rl}
    \mathcal{F}_L =& \dfrac{4 K I H r}{Q_{in}(1+f)}\\
    u_{c} =& \dfrac{-KI\mathcal{F}_L}{\theta \left(\mathcal{F}_L+1\right)}\\
    \tau =& -\dfrac{r}{|u_{c}|}\\
    C_{\tau,{\rm decay}}=& C_0 \exp{\left(-\lambda \tau \right)}\\
    C_{\tau,{\rm filtr}}=& C_0 \exp{\left(-k_{\rm att} \tau \right)}\\
    C_{\tau,{\rm dilut}} =& C_{in} \left( \dfrac{Q_{in}}{u_c \Delta y \Delta z} \right)\\
    C_{\tau,{\rm both}} =& \dfrac{C_{\rm in}Q_{\rm in}}{u_c \Delta y H \theta} \exp{\left(-\lambda\dfrac{r}{|u_c|}\right)}
\end{array}
\end{equation}

```python
%reset -f
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
from os import system
import os
from matplotlib.gridspec import GridSpec
from drawStuff import *

import jupypft.attachmentRateCFT as CFT
import jupypft.plotBTC as BTC

''' GLOBAL CONSTANTS '''
PI = 3.141592
THETA = 0.35
```

```python
def flowNumber():
    return (4.0*K*I*H*r) / (Qin*(1+f))

def uChar():
    '''Interstitial water velocity'''
    return -(K*I*flowNumber())/(THETA*(flowNumber() + 1))

def tChar():
    return -r/uChar()

def cDecay():
    return C0 * np.exp(-decayRate * tChar())

def cAttach():
    return C0 * np.exp(-attchRate * tChar())

def cDilut():
    return (C0 * Qin) / (-uChar() * delY * delZ * THETA)

def cBoth():
    return (C0 * Qin) / (-uChar() * delY * delZ * THETA) * np.exp(-decayRate * tChar())

def cTrice():
    return (C0 * Qin) / (-uChar() * delY * delZ * THETA) * np.exp(-(decayRate+attchRate) * tChar())

def findSweet():
    deltaConc = np.abs(cBoth() - np.max(cBoth()))
    return np.argmin(deltaConc)

def findSweet2():
    deltaConc = np.abs(cTrice() - np.max(cTrice()))
    return np.argmin(deltaConc)
```

```python
K = 10**-2
Qin = 0.24/86400
f = 10
H = 20
r = 40
I = 0.001
C0 = 1.0
qabs = np.abs(uChar()*THETA)

kattDict = dict(
    dp = 1.0E-7,
    dc = 2.0E-3,
    q = qabs,
    theta = THETA,
    visco = 0.0008891,
    rho_f = 999.79,
    rho_p = 1050.0,
    A = 5.0E-21,
    T = 10. + 273.15,
    alpha = 0.01)

decayRate = 3.5353E-06
attchRate,_ = CFT.attachmentRate(**kattDict)
delY,delZ = 1.35,H
```

```python
print("Nondim Flow  = {:.2E}".format(flowNumber()))
print("Charac. Vel  = {:.2E} m/s".format(uChar()))
print("Charac. time = {:.2E} s".format(tChar()))
```

    Nondim Flow  = 1.05E+03
    Charac. Vel  = -2.85E-05 m/s
    Charac. time = 1.40E+06 s

```python
print("Rel concenc. due decay = {:.2E}".format(cDecay()))
print("Rel conc. due dilution = {:.2E}".format(cDilut()))
print("Rel conc. due attachmt = {:.2E}".format(cAttach()))
print("Rel conc. due both eff = {:.2E}".format(cBoth()))
print("Rel conc. due three ef = {:.2E}".format(cTrice()))
```

    Rel concenc. due decay = 7.05E-03
    Rel conc. due dilution = 1.03E-02
    Rel conc. due attachmt = 4.42E-05
    Rel conc. due both eff = 7.26E-05
    Rel conc. due three ef = 3.21E-09

## v.1
K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet()

        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$",
                     1: r"$\bf{I}$ (%)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

threeHeatplots(data={"I":Ii.T,"C":Ci.T,"FL":FLi.T},\
               ylabel=rarray,xlabel=np.round(Qin_array*86400,decimals=2),myLabels=myLabels);K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 1.119E-5

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet()

        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$",
                     1: r"$\bf{I}$ (%)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

threeHeatplots(data={"I":Ii.T,"C":Ci.T,"FL":FLi.T},\
               ylabel=rarray,xlabel=np.round(Qin_array*86400,decimals=2),myLabels=myLabels);
## PLOTRAN SIMULATION RESULTS
minI = np.array([0.00046667, 0.0013    , 0.0048    , 0.012     , 0.00046667,
       0.0013    , 0.0048    , 0.012     , 0.00053333, 0.0013    ,
       0.0048    , 0.012     , 0.00056667, 0.0021    , 0.0053    ,
       0.012     ])

minC = np.array([2.2514572 , 2.62298917, 3.14213329, 3.51421485, 1.64182175,
       2.00913676, 2.52461269, 2.89537637, 0.74130696, 1.0754177 ,
       1.55071976, 1.90646243, 0.18705258, 0.39222131, 0.73428991,
       1.00387133])Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])

i = 0
for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        worstC = minC[i]
        worstGradient = minI[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient

        i += 1

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$",
                     1: r"$\bf{I}$ (%)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

threeHeatplots(data={"I":Ii.T,"C":Ci.T,"FL":Ii.T},\
               ylabel=rarray,xlabel=np.round(Qin_array*86400,decimals=2),myLabels=myLabels);
# v.2

```python
K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])
I = Iarray

## Predictions
for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet()
        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

## Pflotran results
minI = np.array([0.00046667, 0.0013    , 0.0048    , 0.012     , 0.00046667,
       0.0013    , 0.0048    , 0.012     , 0.00053333, 0.0013    ,
       0.0048    , 0.012     , 0.00056667, 0.0021    , 0.0053    ,
       0.012     ])

minC = np.array([2.2514572 , 2.62298917, 3.14213329, 3.51421485, 1.64182175,
       2.00913676, 2.52461269, 2.89537637, 0.74130696, 1.0754177 ,
       1.55071976, 1.90646243, 0.18705258, 0.39222131, 0.73428991,
       1.00387133])

PFLO_I = np.reshape(minI,Ii.shape).T
PFLO_C = np.reshape(minC,Ci.shape).T

## Build annotations
annot_C = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]
annot_I = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        pflC,pflI = (0, PFLO_C[ri,qi]) [float(PFLO_C[ri,qi])>0.0] ,PFLO_I[ri,qi]*100
        zC,zI = (0, Ci[ri,qi]) [float(Ci[ri,qi])>0.0] , Ii[ri,qi]*100
        annot_C[ri][qi] = "{:.1f}\n({:.1f})".format(pflC,zC)
        annot_I[ri][qi] = "{:.2f}\n({:.2f})".format(pflI,zI)

#########

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$"+"\nConsidering only inactivation",
                     1: r"${\bf I} \times 10^{-2}$ (m/m)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

fig, axs = plt.subplots(1,2,sharex=False, sharey=False,\
              figsize=(8,8),gridspec_kw={"width_ratios":[1.75,1],"hspace":0.04,"wspace":0.1})

font = dict(size=12)

ax = axs[0]
im = ax.imshow(PFLO_C.T,cmap="autumn",vmin=0,vmax=4,alpha=0.75)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["{:.0f}".format(i) for i in rarray])
ax.set_yticklabels(["{:.2f}".format(i*86400) for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_C[j][i],
                       ha="center", va="center", color="k",
                       fontweight="bold",fontsize=12,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.set_title(myLabels["Title"][0],fontdict=font)
############################
ax = axs[1]
im = ax.imshow(PFLO_I.T*100,vmin = 0, vmax = 1, cmap="winter_r",alpha=0.5)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["".format(i) for i in rarray])
ax.set_yticklabels(["" for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_I[j][i],
                       ha="center", va="center", color="k",
                       fontweight="normal",fontsize=10,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.yaxis.tick_right()
ax.yaxis.set_label_position("right")
ax.set_title(myLabels["Title"][1],fontdict=font)

# Turn spines off and create white grid.
for ax in axs:
    for edge, spine in ax.spines.items():
        spine.set_visible(False)
    ax.grid(which="minor", color="w", linestyle='-', linewidth=3)
    ax.tick_params(which="minor", bottom=False, left=False)
    ax.set_xlabel(myLabels["X"],fontdict=font)
    ax.set_ylabel(myLabels["Y"],fontdict=font)

# Legend
import matplotlib.lines as mlines

legend = r"$\bf{0.0}$" + "    PFLOTRAN"+"\n"\
       + r"$\bf{(0.0)}$" + "    PF model"
ax.text(4,5.5,legend,fontsize=12,
        ha='right',va='center',weight='normal',\
        bbox=dict(boxstyle='round', facecolor='mintcream', alpha=0.90))

plt.savefig("HeatmapComplete_OnlyInactivation.png",dpi=200,bbox_inches='tight', pad_inches=0)

plt.show()
```

```python
K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])
I = Iarray

## Predictions
for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet2()
        worstC = -np.log10(cTrice()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

## Pflotran results

minI = np.array([0.0014, 0.0039, 0.0144, 0.036 , 0.0014, 0.0039, 0.0144, 0.036 ,
       0.0016, 0.0039, 0.0144, 0.036 , 0.0017, 0.0063, 0.0159, 0.036 ])

minC = np.array([2.70966103, 3.09070854, 3.61937381, 3.99433636, 2.0934136 ,
       2.47294713, 3.00034689, 3.37487213, 1.13490091, 1.49712873,
       2.00931935, 2.37863042, 0.39427172, 0.66484608, 1.08376174,
       1.41433393])

PFLO_I = np.reshape(minI,Ii.shape).T
PFLO_C = np.reshape(minC,Ci.shape).T

## Build annotations
annot_C = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]
annot_I = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        pflC,pflI = (0, PFLO_C[ri,qi]) [float(PFLO_C[ri,qi])>0.0] ,PFLO_I[ri,qi]*100
        zC,zI = (0, Ci[ri,qi]) [float(Ci[ri,qi])>0.0] , Ii[ri,qi]*100
        annot_C[ri][qi] = "{:.1f}\n({:.1f})".format(pflC,zC)
        annot_I[ri][qi] = "{:.2f}\n({:.2f})".format(pflI,zI)

#########

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$" + "\nConsidering inactivation and filtration",
                     1: r"${\bf I} \times 10^{-2}$ (m/m)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

fig, axs = plt.subplots(1,2,sharex=False, sharey=False,\
              figsize=(8,8),gridspec_kw={"width_ratios":[1.75,1],"hspace":0.04,"wspace":0.1})

font = dict(size=12)

ax = axs[0]
im = ax.imshow(PFLO_C.T,cmap="autumn",vmin=0,vmax=4,alpha=0.75)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["{:.0f}".format(i) for i in rarray])
ax.set_yticklabels(["{:.2f}".format(i*86400) for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_C[j][i],
                       ha="center", va="center", color="k",
                       fontweight="bold",fontsize=12,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.set_title(myLabels["Title"][0],fontdict=font)
############################
ax = axs[1]
im = ax.imshow(PFLO_I.T*100,vmin = 0, vmax = 1, cmap="winter_r",alpha=0.5)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["".format(i) for i in rarray])
ax.set_yticklabels(["" for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_I[j][i],
                       ha="center", va="center", color="k",
                       fontweight="normal",fontsize=10,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.yaxis.tick_right()
ax.yaxis.set_label_position("right")
ax.set_title(myLabels["Title"][1],fontdict=font)

# Turn spines off and create white grid.
for ax in axs:
    for edge, spine in ax.spines.items():
        spine.set_visible(False)
    ax.grid(which="minor", color="w", linestyle='-', linewidth=3)
    ax.tick_params(which="minor", bottom=False, left=False)
    ax.set_xlabel(myLabels["X"],fontdict=font)
    ax.set_ylabel(myLabels["Y"],fontdict=font)

# Legend
import matplotlib.lines as mlines

legend = r"$\bf{0.0}$" + "    PFLOTRAN"+"\n"\
       + r"$\bf{(0.0)}$" + "    PF model"
ax.text(4,5.5,legend,fontsize=12,
        ha='right',va='center',weight='normal',\
        bbox=dict(boxstyle='round', facecolor='mintcream', alpha=0.90))
plt.savefig("HeatmapComplete_WithFiltration.png",dpi=200,bbox_inches='tight', pad_inches=0)
plt.show()
```

# v.3
K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])
I = Iarray

## Predictions
for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet()
        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

## Pflotran results ONLY DECAY
minI = np.array([0.00046667, 0.0013    , 0.0048    , 0.012     , 0.00046667,
       0.0013    , 0.0048    , 0.012     , 0.00053333, 0.0013    ,
       0.0048    , 0.012     , 0.00056667, 0.0021    , 0.0053    ,
       0.012     ])

minC = np.array([2.2514572 , 2.62298917, 3.14213329, 3.51421485, 1.64182175,
       2.00913676, 2.52461269, 2.89537637, 0.74130696, 1.0754177 ,
       1.55071976, 1.90646243, 0.18705258, 0.39222131, 0.73428991,
       1.00387133])

PFLO_I = np.reshape(minI,Ii.shape).T
PFLO_C = np.reshape(minC,Ci.shape).T

## Build annotations
annot_C = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]
annot_I = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        pflC,pflI = (0, PFLO_C[ri,qi]) [float(PFLO_C[ri,qi])>0.0] ,PFLO_I[ri,qi]*100
        #zC,zI = (0, Ci[ri,qi]) [float(Ci[ri,qi])>0.0] , Ii[ri,qi]*100
        annot_C[ri][qi] = "{:.1f}".format(pflC)
        annot_I[ri][qi] = "{:.2f}".format(pflI)

#########

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$"+"\nConsidering only inactivation",
                     1: r"${\bf I} \times 10^{-2}$ (m/m)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

fig, axs = plt.subplots(1,1,figsize=(4,4))
font = dict(size=12)

ax = axs
im = ax.imshow(PFLO_C.T,cmap="autumn",vmin=0,vmax=4,alpha=0.75)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["{:.0f}".format(i) for i in rarray])
ax.set_yticklabels(["{:.2f}".format(i*86400) for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_C[j][i],
                       ha="center", va="center", color="k",
                       fontweight="bold",fontsize=13,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.set_title(myLabels["Title"][0],fontdict=font)

# Turn spines off and create white grid.
for edge, spine in ax.spines.items():
    spine.set_visible(False)

ax.grid(which="minor", color="w", linestyle='-', linewidth=3)
ax.tick_params(which="minor", bottom=False, left=False)
ax.set_xlabel(myLabels["X"],fontdict=font)
ax.set_ylabel(myLabels["Y"],fontdict=font)

plt.show()K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06
attchRate = 7.2E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])
I = Iarray

## Predictions
for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet2()
        worstC = -np.log10(cTrice()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

## Pflotran results WITH FILTRATION
minI = np.array([0.0014, 0.0039, 0.0144, 0.036 , 0.0014, 0.0039, 0.0144, 0.036 ,
       0.0016, 0.0039, 0.0144, 0.036 , 0.0017, 0.0063, 0.0159, 0.036 ])

minC = np.array([2.70966103, 3.09070854, 3.61937381, 3.99433636, 2.0934136 ,
       2.47294713, 3.00034689, 3.37487213, 1.13490091, 1.49712873,
       2.00931935, 2.37863042, 0.39427172, 0.66484608, 1.08376174,
       1.41433393])

PFLO_I = np.reshape(minI,Ii.shape).T
PFLO_C = np.reshape(minC,Ci.shape).T

## Build annotations
annot_C = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]
annot_I = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        pflC,pflI = (0, PFLO_C[ri,qi]) [float(PFLO_C[ri,qi])>0.0] ,PFLO_I[ri,qi]*100
        #zC,zI = (0, Ci[ri,qi]) [float(Ci[ri,qi])>0.0] , Ii[ri,qi]*100
        annot_C[ri][qi] = "{:.1f}".format(pflC)
        annot_I[ri][qi] = "{:.2f}".format(pflI)

#########

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$"+"\nConsidering inactivation and filtration",
                     1: r"${\bf I} \times 10^{-2}$ (m/m)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

fig, axs = plt.subplots(1,1,figsize=(4,4))
font = dict(size=12)

ax = axs
im = ax.imshow(PFLO_C.T,cmap="autumn",vmin=0,vmax=4,alpha=0.75)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["{:.0f}".format(i) for i in rarray])
ax.set_yticklabels(["{:.2f}".format(i*86400) for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_C[j][i],
                       ha="center", va="center", color="k",
                       fontweight="bold",fontsize=13,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.set_title(myLabels["Title"][0],fontdict=font)

# Turn spines off and create white grid.
for edge, spine in ax.spines.items():
    spine.set_visible(False)

ax.grid(which="minor", color="w", linestyle='-', linewidth=3)
ax.tick_params(which="minor", bottom=False, left=False)
ax.set_xlabel(myLabels["X"],fontdict=font)
ax.set_ylabel(myLabels["Y"],fontdict=font)

plt.show()

```python
K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])
I = Iarray

## Predictions
for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet()
        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

## Pflotran results ONLY DECAY
minI = np.array([0.00046667, 0.0013    , 0.0048    , 0.012     , 0.00046667,
       0.0013    , 0.0048    , 0.012     , 0.00053333, 0.0013    ,
       0.0048    , 0.012     , 0.00056667, 0.0021    , 0.0053    ,
       0.012     ])

minC = np.array([2.2514572 , 2.62298917, 3.14213329, 3.51421485, 1.64182175,
       2.00913676, 2.52461269, 2.89537637, 0.74130696, 1.0754177 ,
       1.55071976, 1.90646243, 0.18705258, 0.39222131, 0.73428991,
       1.00387133])

PFLO_I = np.reshape(minI,Ii.shape).T
PFLO_C = np.reshape(minC,Ci.shape).T

## Build annotations
annot_C = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]
annot_I = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        pflC,pflI = (0, PFLO_C[ri,qi]) [float(PFLO_C[ri,qi])>0.0] ,PFLO_I[ri,qi]*100
        #zC,zI = (0, Ci[ri,qi]) [float(Ci[ri,qi])>0.0] , Ii[ri,qi]*100
        annot_C[ri][qi] = "{:.1f}".format(pflC)
        annot_I[ri][qi] = "{:.2f}".format(pflI)

#########

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$"+"\nConsidering only inactivation",
                     1: r"${\bf I} \times 10^{-2}$ (m/m)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

fig, axs = plt.subplots(1,2,figsize=(10,5))
font = dict(size=12)

ax = axs[0]
im = ax.imshow(PFLO_C.T,cmap="autumn",vmin=0,vmax=4,alpha=0.75)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["{:.0f}".format(i) for i in rarray])
ax.set_yticklabels(["{:.2f}".format(i*86400) for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_C[j][i],
                       ha="center", va="center", color="k",
                       fontweight="bold",fontsize=13,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.set_title(myLabels["Title"][0],fontdict=font)

# Turn spines off and create white grid.
for edge, spine in ax.spines.items():
    spine.set_visible(False)

ax.grid(which="minor", color="w", linestyle='-', linewidth=3)
ax.tick_params(which="minor", bottom=False, left=False)
ax.set_xlabel(myLabels["X"],fontdict=font)
ax.set_ylabel(myLabels["Y"],fontdict=font)

K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06
attchRate = 7.2E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])
I = Iarray

## Predictions
for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet2()
        worstC = -np.log10(cTrice()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber

## Pflotran results WITH FILTRATION
minI = np.array([0.0014, 0.0039, 0.0144, 0.036 , 0.0014, 0.0039, 0.0144, 0.036 ,
       0.0016, 0.0039, 0.0144, 0.036 , 0.0017, 0.0063, 0.0159, 0.036 ])

minC = np.array([2.70966103, 3.09070854, 3.61937381, 3.99433636, 2.0934136 ,
       2.47294713, 3.00034689, 3.37487213, 1.13490091, 1.49712873,
       2.00931935, 2.37863042, 0.39427172, 0.66484608, 1.08376174,
       1.41433393])

PFLO_I = np.reshape(minI,Ii.shape).T
PFLO_C = np.reshape(minC,Ci.shape).T

## Build annotations
annot_C = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]
annot_I = [ [ "None" for y in range(len(Qin_array)) ] for x in range(len(rarray)) ]

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        pflC,pflI = (0, PFLO_C[ri,qi]) [float(PFLO_C[ri,qi])>0.0] ,PFLO_I[ri,qi]*100
        #zC,zI = (0, Ci[ri,qi]) [float(Ci[ri,qi])>0.0] , Ii[ri,qi]*100
        annot_C[ri][qi] = "{:.1f}".format(pflC)
        annot_I[ri][qi] = "{:.2f}".format(pflI)

#########

myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$"+"\nConsidering inactivation and filtration",
                     1: r"${\bf I} \times 10^{-2}$ (m/m)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

#fig, axs = plt.subplots(1,1,figsize=(4,4))
font = dict(size=12)

ax = axs[1]
im = ax.imshow(PFLO_C.T,cmap="autumn",vmin=0,vmax=4,alpha=0.75)
#cbar = ax.figure.colorbar(im, ax=ax, orientation="horizontal",\
#                         shrink=0.75)

# We want to show all ticks...
ax.set_xticks(np.arange(len(rarray)))
ax.set_yticks(np.arange(len(Qin_array)))
# ... and label them with the respective list entries
ax.set_xticklabels(["{:.0f}".format(i) for i in rarray])
ax.set_yticklabels(["{:.2f}".format(i*86400) for i in Qin_array])

# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=0, ha="center",
         rotation_mode="anchor")

# Loop over data dimensions and create text annotations.
for i in range(len(Qin_array)):
    for j in range(len(rarray)):
        text = ax.text(j, i, annot_C[j][i],
                       ha="center", va="center", color="k",
                       fontweight="bold",fontsize=13,
                       bbox= dict(facecolor='w', alpha=0,edgecolor='w'))

ax.set_title(myLabels["Title"][0],fontdict=font)

# Turn spines off and create white grid.
for edge, spine in ax.spines.items():
    spine.set_visible(False)

ax.grid(which="minor", color="w", linestyle='-', linewidth=3)
ax.tick_params(which="minor", bottom=False, left=False)
ax.set_xlabel(myLabels["X"],fontdict=font)
ax.set_ylabel(myLabels["Y"],fontdict=font)

#INDICATORS A-B
axs[0].text(-0.10,1.05,"A",transform=axs[0].transAxes,\
              fontsize=18,c='dimgray',fontweight='bold')
axs[1].text(-0.10,1.05,"B",transform=axs[1].transAxes,\
              fontsize=18,c='dimgray',fontweight='bold')

plt.savefig("HeatmapsSimple.png",dpi=300,bbox_inches='tight', pad_inches=0)
plt.show()
```

```python

```


# Μια επίδειξη των δυνατοτήτων της Python με εφαρμογές στον επιστημονικό προγραμματισμό και την ανάλυση δεδομένων

Η γλώσσα προγραμματισμού Python είναι μια πολύ υψηλού επιπέδου Interpreted γλώσσα (δηλαδή δεν χρειάζεται Compilation αλλά τρέχει "γραμμή-γραμμή" (Script)).

Μπορεί να χρησιμοποιηθεί και για Συναρτησιακό αλλα και για Αντικειμενοστραφή προγραμματισμό ενώ η ευκολοδιάβαστει δομή της είναι ιδανική για code reading.

Είναι απολύτως δωρεάν και ανοιχτού κώδικα (open source) και υποστηρίζεται από μια μεγάλη κοινότητα προγραμματιστών και χρηστών (Community) δίνοντας της μεγαλύτερη ευελιξία/προσαρμοστικότητα και ταχύτητα ανάπτυξης σε σχέση με "ανταγωνιστικές γλώσσες".

Λόγω της απουσίας compilation ο ίδιος κώδικας λειτουργεί πανομοιότυπα σε κάθε μηχάνημα ανεξαρτητώς αρχιτεκτονικής και λειτουργικού συστήματος. Βέβαια ακριβώς για τον ίδιο λόγο είναι αρκετά πιο άργη από compiled γλώσσες (C, Fortran κλπ)

Παρακάτω θα παρουσιάσουμε

### Τύποι αριθμητικών μεταβλητών
* Ακέραιος **int**
* Floating-point number **float** (αντίστοιχως της **double** στη C)
* Μιγαδικός **complex**
* Λογικός **boolean**

>Στη Python δεν χρειάζεται να δηλώσουμε προηγουμένως τις μεταβλητές μας (με εξαίρεση τις **global**). Για να δηλώσουμε οτι μια μεταβλητή ειναι ακέραια απλά της δίνουμε μια αρχική ακέραια τιμή (βλ. παρακάτω τη μεταβλητή **a**)

>Για να δηλώσουμε μια float μεταβλητή δίνουμε μια αρχική τιμή σε δεκαδική μορφή (βλ. παρακάτω τη μεταβλητή **b**).

>Ο φανταστικός αριθμός i συμβολίζεται με **J**.

```python
a=2 #integer
b=2. #float
c=1.+3.J
d= 3>2
print(a)
print(b)
print(c)
print(d)
```

    2
    2.0
    (1+3j)
    True

### Containers
Επίσης υπάρχουν και "συνδυαστικές(?!)" δομές δεδομένων (sequences). Η βασική δομή δεδομένων είναι η λίστα (**List**) δηλαδή η συλλογή διαφορετικών (και είδους) μεταβλητών. Η λίστα δηλώνεται μέσω του συμβολισμού **[ ]**.
Μία λέξιλογική προταση (**string**) είναι ουσιαστικά μια λίστα γραμμάτων.

Μπορούμε να ανατρέξουμε σε μια λίστα μέσω του αριθμου **index** δηλαδή ζητώντας το σημείο εκείνο όπου είναι αποθηκευμένη μεταβλητή μας ξεκινώντας απο το **0**. Δηλαδή αν η λίστα είναι η L, ζητάμε τη μεταβλητή που έχει αποθηκευθεί εντός της λίστα στο σημείο 4 μέσω της εντολής ```L[4]```. (*βλ. παράδειγμα παρακάτω*)

Ένα "λεξικό" (**Dictionary**) είναι μία λίστα όπου αντί για index χρησιμοποιούμε ένα string. Αυτό είναι πολύ χρησίμο όταν καταχωρούμε πολλά (και διαφορετικά) δεδομένα σε μια λίστα και θέλουμε να τα καλέσουμε μέσω μιας λέξης κλειδί. πχ: ```H['time']```

```python
L = [1.,301.,a,'a string',1.e-4]
L[0],L[3],L[-1],L[-2],L[1:4]
```

    (1.0, 'a string', 0.0001, 'a string', [301.0, 2, 'a string'])

```python
print L[-2]
```

    a string

```python
G=[L[:2],L[2:]]
G
```

    [[1.0, 301.0], [2, 'a string', 0.0001]]

```python
G[0][0]
```

    1.0

```python
L[3][2:5]
```

    'str'

```python
P=[2.]
```

```python
2*P

```

    [2.0, 2.0]

**Προσοχή** Μία λίστα δεν είναι ένας "πίνακας"

```python
K=[2,4,5,6,7]
2*K
```

    [2, 4, 5, 6, 7, 2, 4, 5, 6, 7]

Ένα *string* είναι μια λίστα.

```python
s='a bigger string'
s[3:5]
```

    'ig'

```python
D = {'a':1,'b':2,'c':3}
D['b']
```

    2

```python
D['a']
```

    1

```python
tup =(2.,3.,5.)
tup[2]=4
```

```python
K
```

    [2, 4, 5, 6, 7]

```python
K[1:4]=[2,3,9]
```

```python
K[0]=9.
K
```

    [9.0, 2, 3, 9, 7]

```python
K
```

    [9.0, 2, 3, 9, 7]

```python
K[:2]=[0,'a']
K
```

### Control Flow

```python
a
```

    2

```python
d=10 if a>4 else 4
d
```

    4

```python
a=2
```

```python
if a == 1:
    print(a)
elif a==2:
    print(a**2)
```

    4

```python
K[1]=2.
K
```

    [9.0, 2.0, 3, 9, 7]

```python
for i,k in enumerate(K):
    print i,k
```

    0 9.0
    1 2
    2 3
    3 9
    4 7

```python
len(K)
```

    5

```python
for i in range(10):
    print(i)
```

    0
    1
    2
    3
    4
    5
    6
    7
    8
    9

```python
range(len(K))
```

    [0, 1, 2, 3, 4]

```python
for i in range(len(K)):
    print i,K[i]

```

    0 9.0
    1 2.0
    2 3
    3 9
    4 7

```python
enumerate(K)
```

    <enumerate at 0x7f26d467ffa0>

```python
for index,value in enumerate(K):
    print index,value
```

    0 9.0
    1 2.0
    2 3
    3 9
    4 7

### List Comprehensions

```python
lc=[i**2 for i in range(len(K))]
lc=[i**2 for i in K]
print(lc)
```

    [81.0, 4, 9, 81, 49]

### Functions

```python
def f(x):
    return 3.*x**2
```

```python
[f(i) for i in K]
```

    [243.0, 12.0, 27.0, 243.0, 147.0]

```python
[0.,3.]**3
```

```python
f([488,5])
```

```python
f(K)
```

# Βιβλιοθήκες/Πακέτα της Python

## Εγκατάσταση Πακέτων
### Package manager
* Παράδειγμα για debian based systems (ubuntu, mint, elementary κλπ)
``` bash
sudo apt-get install numpy
```

### pip (Linux)
* The PyPA (Python Packaging User Guide) recommended tool for installing Python packages.
* Ανεξάρτητο σύστημα άρα και ανεξάρτητο από τους package manager της εκάστοτε διανομής.
* Latest version
* Compile from source
* Παράδειγμα:
``` bash
sudo pip install numpy
```

## Numpy  (http://www.numpy.org/)

NumPy is the fundamental package for scientific computing with Python. It contains among other things:
* a powerful N-dimensional array object
* sophisticated (broadcasting) functions
* tools for integrating C/C++ and Fortran code
* useful linear algebra, Fourier transform, and random number capabilities

Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.
``` python
import numpy as np
import numpy.random as rnd
```

```python
import numpy as np
a= np.array(K)
a
```

    array([ 9.,  2.,  3.,  9.,  7.])

```python
a[0]='string'
```

```python
2.*a
```

    array([ 18.,   4.,   6.,  18.,  14.])

```python
def f(x):
    """
    Docstring
    Usage
    Description of the function (Input and Output)
    """
    return x**2
```

```python
f(a)
```

    array([ 81.,   4.,   9.,  81.,  49.])

```python
np.pi
```

    3.141592653589793

```python
from numpy import tan
```

```python
K
```

    [9.0, 2, 3, 9, 7]

```python
tan(np.pi)
```

    -1.2246467991473532e-16

```python
np.arctan(K)
```

    array([ 1.46013911,  1.10714872,  1.24904577,  1.46013911,  1.42889927])

```python
a=np.ones((4,2))
a
```

    array([[ 1.,  1.],
           [ 1.,  1.],
           [ 1.,  1.],
           [ 1.,  1.]])

```python
a.shape
```

    (4, 2)

```python
a[2,1]=4.
a
```

    array([[ 1.,  1.],
           [ 1.,  1.],
           [ 1.,  4.],
           [ 1.,  1.]])

```python
g=np.random.rand(4,2)
g
```

    array([[ 0.58305839,  0.06789145],
           [ 0.08276288,  0.83649712],
           [ 0.45219536,  0.41127844],
           [ 0.51657028,  0.3546244 ]])

```python
g[:,0]
```

    array([ 0.58305839,  0.08276288,  0.45219536,  0.51657028])

```python
a[:,0]=[0.,0.,0.,0.]
```

```python
a
```

    array([[ 0.,  1.],
           [ 0.,  1.],
           [ 0.,  4.],
           [ 0.,  1.]])

```python
aa=np.ones((4,3))*2.
print aa
print aa[1:3,1]
```

    [[ 2.  2.  2.]
     [ 2.  2.  2.]
     [ 2.  2.  2.]
     [ 2.  2.  2.]]
    [ 2.  2.]

```python
aa[1]=np.array([1,2,3])
```

```python
aa
```

    array([[ 2.,  2.,  2.],
           [ 1.,  2.,  3.],
           [ 2.,  2.,  2.],
           [ 2.,  2.,  2.]])

```python
from IPython.display import Image
Image('numpy_indexing.png')
```

```python
aa[3]=np.array([0,1,3])
aa
```

    array([[ 2.,  2.,  2.],
           [ 1.,  2.,  3.],
           [ 2.,  2.,  2.],
           [ 0.,  1.,  3.]])

```python
aa>2
```

    array([[False, False, False],
           [False, False,  True],
           [False, False, False],
           [False, False,  True]], dtype=bool)

```python
aa[aa>2]
```

    array([ 3.,  3.])

```python
aa[aa>2]=0
aa
```

    array([[ 2.,  2.,  2.],
           [ 1.,  2.,  0.],
           [ 2.,  2.,  2.],
           [ 0.,  1.,  0.]])

```python
aa>2 && aa
```

```python
np.logical_and(aa>2,aa<1)
```

    array([[False, False, False],
           [False, False, False],
           [False, False, False],
           [False, False, False]], dtype=bool)

```python
aa[np.logical_or(aa>2,aa<1)]
```

    array([ 0.,  0.,  0.])

```python
f(np.array(K))
```

    array([ 81.,   4.,   9.,  81.,  49.])

```python
vectorized_f=np.vectorize(f)
vectorized_f(K)
```

    array([ 81.,   4.,   9.,  81.,  49.])

## Scipy (http://www.scipy.org/)

It provides many user-friendly and efficient numerical routines such as routines for numerical integration and optimization.

```python
from scipy.optimize import curve_fit
```

## Matplotlib (http://matplotlib.org/)
matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. matplotlib can be used in python scripts, the python and ipython shell (ala MATLAB®* or Mathematica®†), web application servers, and six graphical user interface toolkits.

* http://www.labri.fr/perso/nrougier/teaching/matplotlib/matplotlib.html

```python
%matplotlib inline
#%matplotlib notebook
import matplotlib.pyplot as plt
```

```python
np.arange(0,7,0.1)
```

    array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ,
            1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  1.7,  1.8,  1.9,  2. ,  2.1,
            2.2,  2.3,  2.4,  2.5,  2.6,  2.7,  2.8,  2.9,  3. ,  3.1,  3.2,
            3.3,  3.4,  3.5,  3.6,  3.7,  3.8,  3.9,  4. ,  4.1,  4.2,  4.3,
            4.4,  4.5,  4.6,  4.7,  4.8,  4.9,  5. ,  5.1,  5.2,  5.3,  5.4,
            5.5,  5.6,  5.7,  5.8,  5.9,  6. ,  6.1,  6.2,  6.3,  6.4,  6.5,
            6.6,  6.7,  6.8,  6.9])

```python
X=np.linspace(0,7,4)
X
```

    array([ 0.        ,  2.33333333,  4.66666667,  7.        ])

```python
#fig = plt.figure(figsize=(10,10))
plt.plot(X,X**2)
plt.plot(X,X)
```

```python
X=np.linspace(-1.5,1.5,2000)
Y=np.linspace(-2,2,2000)
X,Y=np.meshgrid(X,Y)
c=X+1J*Y
ms=c*0.
z=ms

for i in range(20):
    z=z**2+c
ms=np.abs(z)<50
fig = plt.figure()
plt.imshow(np.abs(ms),cmap='viridis')
```

## Pandas (http://pandas.pydata.org/)
pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

```python
import pandas as pd
```

## Sympy (http://www.sympy.org/)
SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python.
```python
import sympy as sm
sm.init_printing()
```

## Jupyter Notebook (iPython)
The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text.

### Code cell
παρακάτω
### Markdown Cell
* Latex support
$$
    \begin{equation}
    \rho \left(\frac{\partial u_x}{\partial t} + u_x \frac{\partial u_x}{\partial x} + u_y \frac{\partial u_x}{\partial y} + u_z \frac{\partial u_x}{\partial z}\right) = -\frac{\partial p}{\partial x} + \mu \left(\frac{\partial^2 u_x}{\partial x^2} + \frac{\partial^2 u_x}{\partial y^2} + \frac{\partial^2 u_x}{\partial z^2}\right) - \mu \frac{\partial}{\partial x} \left( \frac{\partial u_x}{\partial x} + \frac{\partial u_y}{\partial y} + \frac{\partial u_z}{\partial z} \right) + \rho g_x
    \end{equation}
$$

$$
    \begin{equation}
    \rho \left(\frac{\partial u_y}{\partial t} + u_x \frac{\partial u_y}{\partial x} + u_y \frac{\partial u_y}{\partial y} + u_z \frac{\partial u_y}{\partial z}\right) = -\frac{\partial p}{\partial y} + \mu \left(\frac{\partial^2 u_y}{\partial x^2} + \frac{\partial^2 u_y}{\partial y^2} + \frac{\partial^2 u_y}{\partial z^2}\right) - \mu \frac{\partial}{\partial y} \left( \frac{\partial u_x}{\partial x} + \frac{\partial u_y}{\partial y} + \frac{\partial u_z}{\partial z} \right) + \rho g_y
    \end{equation}
$$

### Rich Display System
```python
from IPython.display import display, Image,YouTubeVideo

```
### Magic
``` python
%magic
%lsmagic
```

```python

```

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.gridspec as gridspec
import seaborn
from tqdm import tqdm,tqdm_gui
import sympy as sm
import sympy.physics.vector as vc
sm.init_printing()
```

```python
%magic
#%lsmagic
```

```python
%edit
```

    IPython will make a temporary file named: /tmp/ipython_edit_x1i0yK/ipython_edit_zvHx__.py

```python
r,u,pr,pu,G,Ms,a,e=sm.symbols('r,\phi,p_r,p_\phi,G,M_\odot,a,e')
H=pr**2/2.+pu**2/(2.*r**2)-G*Ms/r
H
```

```python
H.subs({G:u**2})
```

```python
dotr=H.diff(pr)
dotu=H.diff(pu)
dotpr=-H.diff(r)
dotpu=-H.diff(u)
dotr,dotu,dotpr,dotpu
```

```python
pr0=sm.Float(0)
u0=sm.Float(0)
pu0=sm.sqrt(Ms*G*a*(1.-e**2))
r0=a*(1.-e)
r0,u0,pr0,pu0
```

```python
sub1={G*Ms:0.0172**2,a:1.0,e:0.0167}
```

```python
Y0=np.array([r0.subs(sub),
             u0.subs(sub),
             pr0.subs(sub),
             pu0.subs(sub)
            ]).astype('float')
Y0
```

    array([ 0.9833   ,  0.       ,  0.       ,  0.0171976])

```python
dr = sm.lambdify((r,u,pr,pu),dotr.subs(sub1),"numpy")
du = sm.lambdify((r,u,pr,pu),dotu.subs(sub1),"numpy")
dpr = sm.lambdify((r,u,pr,pu),dotpr.subs(sub1),"numpy")
dpu = sm.lambdify((r,u,pr,pu),dotpu.subs(sub1),"numpy")
def dY(y,t):
    a= np.array([
            dr(y[0],y[1],y[2],y[3]),
            du(y[0],y[1],y[2],y[3]),
            dpr(y[0],y[1],y[2],y[3]),
            dpu(y[0],y[1],y[2],y[3])
        ])
    return a
```

```python
dr(0,1,2,0)
```

```python
from scipy.integrate import odeint
```

```python
%%time
t=np.arange(0.,50.*365.,3./365.)
sol=odeint(dY,Y0,t,rtol=1e-12,atol=1e-13,printmessg=True)
rt,ut=sol[:,0],sol[:,1]
prt,put=sol[:,2],sol[:,3]
```

    CPU times: user 844 ms, sys: 44 ms, total: 888 ms
    Wall time: 891 ms

    /usr/local/lib/python2.7/dist-packages/scipy/integrate/odepack.py:221: ODEintWarning: Integration successful.
      warnings.warn(warning_msg, ODEintWarning)

```python
x=rt*np.cos(ut)
y=rt*np.sin(ut)

#True Orbit
ee=0.0167
phi=np.linspace(0.,2.*np.pi,len(x))
rr=(1.-ee**2)/(1.+ee*np.cos(phi))
xx=rr*np.cos(phi)
yy=rr*np.sin(phi)

fig=plt.figure(figsize=(16,12))
gs=gridspec.GridSpec(3,4)
ax1=plt.subplot(gs[0:2,0:2])
ax1.plot(x,y,label='Earth Orbit (numerical)')
ax1.plot([0],[0],'o',label='Sun Position')
ax1.axis(xmin=-1.3,xmax=1.3,ymin=-1.3,ymax=1.3)
ax1.set_ylabel('Y Position (au)')
ax1.set_xlabel('X Position (au)')
ax1.plot(xx,yy,label='Earth Orbit (true)',linestyle='--')
ax1.legend(loc='best')
ax2=plt.subplot(gs[2,:])
#ax2.plot(T,vx,label='X Velocity')
#ax2.plot(T,vy,label='Y Velocity')
ax2.plot(t,np.abs(rt-rr),label='Error')
ax2.set_ylabel('Difference (au)')
ax2.set_xlabel('Time (days)')
ax2.legend(loc='best')
```

```python

```


```python
import scipy.io as sio
import numpy as np
import math
import matplotlib.pyplot as plt
from scipy.sparse import csc_matrix
data = sio.loadmat('XwindowsDocData.mat')
```

```python
xtrain = data['xtrain']; xtest = data['xtest']
ytrain = data['ytrain']; ytest = data['ytest']
vocab = data['vocab']
```

### Naive Bayes classifiers
- To classify vectors of discrete-valued features, $\mathbf{x}\in\{1,...,K\}^D$, where K is the number of values for each feature, and D is the number of features. A generative approach requires to specify the class conditional distribution, $p(\mathbf{x}|y=c)$. The simplest approach is to assume the features are conditionally independent given the class label. The class conditional density as a product of one dimensional densities:

$$p(\mathbf{x}|y=c,\mathbf{\theta})= \prod^D_{j=1}p(x_j|y=c,\theta_{jc})$$

The resulting model is called a naive Bayes classifier (NBC) $\sim O(CD)$, for C classes and D features.

- In the case of **real-valued features**, the Gaussian distribution can be used. $p(\mathbf{x}|y=c,\theta)=\prod^D_{j=1}\mathcal{N}(x_j|\mu_{jc},\sigma^2_{jc})$, where $\mu_{jc}$ the mean of feature $j$ in objects of class $c$, and $\sigma^2_{jc}$ its variance.
- In the case of **binary features**, $x_j\in\{0,1\}$, the Bernoulli distribution $p(\mathbf{x}|y=c,\theta)=\prod^D_{j=1}Ber(x_j|\mu_{jc})$, where $\mu_{jc}$ the probability that feature $j$ occurs in class $c$. (**multivariate Bernoulli naive Bayes**)
- In the case of **categorical features**, $x_j\in\{1,...,K\}$, the multinoulli distribution is used, $p(\mathbf{x}|y=c,\theta)=\prod^D_{j=1}Cat(x_j|\mu_{jc})$, where $\mu_{jc}$ is a histogram over the K possible values for x_j in class c.

Train a naive Bayes classifier = computing the MLE or the MAP estimate for the parameters.

### MLE for NBC
The probability for a single data case is given by,

$$
p(\mathbf{x_i},y_i|\theta) = p(y_i|\pi)\prod_j p(x_{ij}|\theta_j) = \prod_c \pi_c^{\mathbb{I}(y_i=c)}\prod_j\prod_c p(x_{ij}|\theta_{jc})^{\mathbb{I}(y_i=c)}
$$

where $p(y_i|\pi)$ the likelihood over a class $c$, and $p(x_{ij}|\theta_j)$ the probability of feature $j$ in the text document $i$ over class $c$.

\begin{align}
\log p(\mathcal{D}|\theta) &= \sum_i\log p(\mathbf{x}_i,y_i|\theta) = \sum_i\log\prod_c \pi_c^{\mathbb{I}(y_i=c)}\prod_j\prod_c p(x_{ij}|\theta_{jc})^{\mathbb{I}(y_i=c)} \\
&= \sum_i\sum^C_{c=1}\mathbb{I}(y_i=c)\log\pi_c+\sum_i\sum_{j=1}^D\sum^C_{c=1}\mathbb{I}(y_i=c)\log p(x_{ij}|\theta_{jc}) \\
\log p(\mathcal{D}|\theta) &= \sum^C_{c=1}N_c\log\pi_c + \sum_{j=1}^D\sum^C_{c=1}\sum_{i:y_i=c}\log p(x_{ij}|\theta_{jc})
\end{align}

where $N_c\triangleq\sum_i\mathbb{I}(y_i=c)$ is the number of examples in class c.

To enforce the constraints that $\sum_c\pi_c = 1$, a **Lagrange multiplier** is used. The **constrained objective function (Lagrangian)** is given by the **log likelihood** + the **constraint**:

$$\mathcal{l}(\theta,\lambda) = \sum^C_{c=1}N_c\log\pi_c + \sum_{j=1}^D\sum^C_{c=1}\sum_{i:y_i=c}\log p(x_{ij}|\theta_{jc})+\lambda(1-\sum_c\pi_c)$$

Taking derivatives with respect to $\lambda$ yields the original constraint $\sum_c\pi_c = 1$.

Taking derivatives with respect to $\pi_c$ yields

\begin{align}
\frac{\partial \mathcal{l}}{\partial \pi_c} &= \frac{N_c}{\pi_c}-\lambda =0 \Longrightarrow N_c = \lambda\pi_c \\
\Longrightarrow \sum_c N_c &= \lambda\sum_c\pi_c \Longrightarrow
\lambda = \sum_c N_c = N
\end{align}
Therefore, $$\hat{\pi}_c = \frac{N_c}{\lambda}=\frac{N_c}{N}$$

### Classifying documents using bag of words
**Document classification** is the problem of classifying text documents into different categories. One simple approach is to represent each document as a binary vector, which records whether each word is present or not, so $x_{ij}=1$ iff (if or only if) word $j$ occurs in document $i$, otherwise $x_{ij}=0$.
Suppose all features are binary (sparse matrix), so $x_i|y = c \sim Ber(\theta_{jc})$ and $p(x_{ij}|\theta_{jc}) = Ber(x_{ij}|\theta_{jc}) = \theta_{jc}^{\mathbb{I}(x_{ij})}(1-\theta_{jc})^{\mathbb{I}(1-x_{ij})}$.

Class conditional density:
$$p(\mathbf{x}_i|y_i=c,\theta)=\prod^D_{j=1}Ber(x_{ij}|\theta_{jc})=\prod^D_{j=1}\theta_{jc}^{\mathbb{I}(x_{ij})}(1-\theta_{jc})^{\mathbb{I}(1-x_{ij})}$$

To enforce the constraints that $\sum_c\sum_j\theta_{jc} = 1$, a **Lagrange multiplier** is used. The **constrained objective function (Lagrangian)** is given by the **log likelihood** + the **constraint**:

$$\mathcal{l}(\theta,\lambda) = \sum^C_{c=1}N_c\log\pi_c + \sum_{j=1}^D\sum^C_{c=1}\sum_{i:y_i=c}\log p(x_{ij}|\theta_{jc})+\lambda(1-\sum_j\sum_c\theta_{jc})$$

Therefore, $$\hat{\theta}_{jc}=\frac{N_{jc}}{N}$$.

### Fitting a naive Bayes classifier to binary features
$$\hat{\pi}_c= \frac{N_c}{N}\text{ , }\hat{\theta}_{jc}=\frac{N_{jc}}{N}$$

```python
def naiveBayesFit(xtrain,ytrain):
    pC = 1
    c = np.unique(ytrain)
    Ntrain,D = xtrain.shape
    theta = np.zeros((len(c),D))
    Nclass = []
    for i in c:
        ndx = np.where(ytrain==i)[0]
        Xtr = xtrain[ndx,:]
        Non = np.sum(Xtr==1,axis=0)
        Noff = np.sum(Xtr==0,axis=0)
        theta[i-1,:] = (Non+pC)/(Non+Noff+2*pC)
        Nclass.append(len(ndx))
    classPrior = Nclass/np.sum(Nclass)
    return theta,classPrior
```

```python
theta,classPrior = naiveBayesFit(xtrain,ytrain)
```

    C:\Users\44754\Anaconda3\lib\site-packages\ipykernel_launcher.py:1: SparseEfficiencyWarning: Comparing a sparse matrix with 0 using == is inefficient, try using != instead.
      """Entry point for launching an IPython kernel.
    C:\Users\44754\Anaconda3\lib\site-packages\ipykernel_launcher.py:1: SparseEfficiencyWarning: Comparing a sparse matrix with 0 using == is inefficient, try using != instead.
      """Entry point for launching an IPython kernel.

### Predicting with a naive bayes classifier for binary features

$$p(y=c|\mathbf{x},\mathcal{D})\propto p(y=c|\mathcal{D})\prod^D_{j=1}p(x_j|y=c,\mathcal{D})$$

The correct Bayesian procedure is to integrate out the unknwon parameters,

$$p(y=c|\mathbf{x},\mathcal{D})\propto\bigg[\int Cat(y=c|\pi)p(\pi|\mathcal{D})d\pi\bigg]\prod^D_{j=1}\int Ber(x_j|y=c,\theta_{jc})p(\theta_{jc}|\mathcal{D})$$

The posterior predictive density is given by,

\begin{align}
p(y=c|\mathbf{x},\mathcal{D})&\propto \bar{\pi}_c\prod^D_{j=1}\bar{\theta}_{jc}^{\mathbb{I}(x_j=1)}(1-\bar{\theta}_{jc})^{\mathbb{I}(x_j=0)} \\
\bar{\theta}_{jk} &= \frac{N_{jc}+\beta_1}{N_c+\beta_0+\beta_1} \\
\bar{\pi}_c &= \frac{N_c+\alpha_c}{N+\alpha_0}
\end{align}

To avoid numerical underflow, the log-sum-exp trick is also used.

\begin{align}
\log p(y=c|\mathbf{x}) &= b_c - \log\sum^C_{c'=1}\exp(b_{c'}) = \log \frac{p(\mathbf{x}|y=c)p(y=c)}{p(\mathbf{x})}\\
b_c &\triangleq \log p(\mathbf{x}|y=c)+\log p(y=c) \\
\log\sum_{c'} \exp(b_{c'}) &= \log\sum_{c'}p(y=c',\mathbf{x})=\log p(\mathbf{x})
\end{align}

In general,
$$\log\sum_c\exp(b_c) = \log\bigg[(\sum_c\exp(b_c-B))\exp B\bigg] = \log\bigg[(\sum_c\exp(b_c-B))\bigg] + B$$

\begin{align}
p_{ic} &= \exp(L_{ic}-\log\sum\exp L_{i,:}) \\
\hat{y}_i &= \arg\max_c p_{ic}
\end{align}

```python
def naiveBayesPredict(theta,classPrior,xtest):
    Ntest = xtest.shape[0]
    C = theta.shape[0]
    logPrior = np.log(classPrior)
    logPost = np.zeros((Ntest,C))
    #logPost = []
    logT = np.log(theta)
    logTnot = np.log(1-theta)
    xtestnot = csc_matrix((xtest.todense()==0)*1)

    xtesttmp = xtest.todense()
    xtestnottmp = xtestnot.todense()
    for i in np.array([1,2]):
        tmpT = np.tile(logT[i-1,:],(Ntest,1))
        tmpTnot = np.tile(logTnot[i-1,:],(Ntest,1))
        L1 = csc_matrix(np.multiply(tmpT,xtesttmp))
        L0 = csc_matrix(np.multiply(tmpTnot,xtestnottmp))
        logPost[:,i-1]=np.sum(L0+L1,axis=1).squeeze()
    yhat = np.argmax(logPost,axis=1)
    return yhat
def zeroOneLossFn(y,ypred):
    err = y!=ypred
    return err
```

```python
ypred_train = naiveBayesPredict(theta,classPrior,xtrain)
err_train = np.mean(zeroOneLossFn(ytrain.squeeze(),ypred_train+1))
ypred_test = naiveBayesPredict(theta,classPrior,xtest)
err_test = np.mean(zeroOneLossFn(ytest.squeeze(),ypred_test+1))
```

```python
print('Misclassification rate on train: '+str(err_train))
print('Misclassification rate on test: '+str(err_test))
```

    Misclassification rate on train: 0.08333333333333333
    Misclassification rate on test: 0.18666666666666668

```python
plt.bar(range(theta[0,:].shape[0]),theta[0,:])
```

```python
plt.bar(range(theta[1,:].shape[0]),theta[1,:])
```

### TO DO: Feature selection using mutual information
Since an NBC is fitting a joint distribution over potentially many features, it can suffer from overfitting. In addition, the run-time cost is $O(D)$, which may be too high for some applications.

One common approach to tackling both of these problems is to perform feature selection, to remove 'irrelevant' features that do not help much with the classification problem. The simplest approach to feature selection is to evaluate the relevance of each feature separately, and then take the top K, where K is chosen based on some tradeoff between accuracy and complexity. The approach is known as variable **ranking, filtering or screening**.

#### Mutual information
To measure relevance, the mutual information (MI) between feature $X_j$ and the class label $Y$.

$$I(X,Y) = \sum_{x_j}\sum_y p(x_j,y)\log\frac{p(x_j,y)}{p(x_j)p(y)}$$

If the features are binary, MI can be computed as follows,

$$I_j = \sum_c\bigg[\theta_{jc}\pi_c\log\frac{\theta_{jc}}{\theta_j}+(1-\theta_{jc})\pi_c\log\frac{1-\theta_{jc}}{1-\theta_j}\bigg]$$

where $\pi_c = p(y=c)$, $\theta_{jc}=p(x_j=1|y=c)$ and $\theta_j=p(x_j=1)=\sum_c\pi_c\theta_{jc}$.

```python

```

Machine Learning A Probabilistic Perspective/3GMDD/F3.8/.ipynb_checkpoints/3.8naiveBayesBowDemo-checkpoint.ipynb