Métodos Numéricos

Capítulo 1: Error y Representación de números en el computador

2021/02

MEDELLÍN - COLOMBIA

 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license.(c) Carlos Alberto Alvarez Henao

***

***Docente:*** Carlos Alberto Álvarez Henao, I.C. D.Sc.

***e-mail:*** carlosalvarezh@gmail.com

***skype:*** carlos.alberto.alvarez.henao

***Linkedin:*** https://www.linkedin.com/in/carlosalvarez5/

***github:*** https://github.com/carlosalvarezh/Metodos_Numericos

***Herramienta:*** [Jupyter](http://jupyter.org/)

***Kernel:*** Python 3.8

***

Tabla de Contenidos

1  Fuentes de error

1.1  Error en el modelo y los datos

1.2  Error de truncamiento

1.3  Error de  representación de punto fotante

1.4  Definiciones básicas

1.4.1  Error absoluto

1.4.2  Error relativo

2  Notación $\text{Big}-\mathcal{O}$

2.1  Reglas para el error de propagación basado en la notación $\text{Big}-\mathcal{O}$

3  Error de truncamiento

3.1  Laboratorio Numérico 1

4  Error de punto flotante

4.1  Aritmética de punto flotante

4.2  Propiedades de los sistemas de punto flotante

4.3  Sistema Binario

4.3.1  Numero y distribución de números

4.4  Sistema real - IEEE 754 sistema binario de punto flotante

4.4.1  Precisión simple

4.4.2  Precisión doble

4.5  Acceso de Python a números de la IEEE

4.6  Calculo "manual" del $\epsilon_{mach}$

5  Por qué debería importarnos esto?

5.1  Ejemplo 1: Aritmética simple

5.2  Ejemplo 2: Cancelación catastrófica

5.3  Ejemplo 3: Evaluación de una función

5.4  Ejemplo 4: Evaluación de un Polinomio

5.5  Ejemplo 5: Evaluación de una función racional

6  Combinación de error

6.1  Ejemplo 1:

6.2  Ejemplo 2:

6.3  Ejemplo 3: Error relativo

7  Operaciones de conteo

7.1  Ejemplo 1: Multiplicación matriz - vector

7.2  Ejemplo 2: Método de Horner para evaluar polinomios

7.3  Algoritmo

***Comentario:*** este capítulo está basado en parte de las  notas del curso del profesor [Kyle T. Mandli](https://github.com/mandli/intro-numerical-methods) (en inglés)

 Fuente: Ariane 5 Explosion

```python
#Bibliotecas a ser utilizadas en el Notebook

import numpy as np
import matplotlib.pyplot as plt
import sympy
import scipy.special

```

## Fuentes de error

Los cálculos numéricos, que involucran el uso de máquinas (análogas o digitales) presentan una serie de errores que provienen de diferentes fuentes:

- del Modelo

- de los datos

- de truncamiento

- de representación de los números (punto flotante)

- $ \ldots$

***Meta:*** Categorizar y entender cada tipo de error y explorar algunas aproximaciones simples para analizarlas.

[Volver a la Tabla de Contenido](#TOC)

### Error en el modelo y los datos

Errores en la formulación fundamental

- Error en los datos: imprecisiones en las mediciones o incertezas en los parámetros

Infortunadamente no tenemos control de los errores en los datos y el modelo de forma directa pero podemos usar métodos que pueden ser más robustos en la presencia de estos tipos de errores.

[Volver a la Tabla de Contenido](#TOC)

### Error de truncamiento

Los errores surgen de la expansión de funciones con una función simple, por ejemplo, $sin(x) \approx x$ para $|x|\approx0$.

[Volver a la Tabla de Contenido](#TOC)

### Error de  representación de punto fotante

Los errores surgen de aproximar números reales con la representación  en precisión finita de números en el computador.

[Volver a la Tabla de Contenido](#TOC)

### Definiciones básicas

Dado un valor verdadero de una función $f$ y una solución aproximada $F$, se define:

#### Error absoluto

$$e_a=|f-F|$$

***Ejemplo:*** se realiza una medición y se obtiene un valor aproximado de $29.99$ *m*. Asumiendo que el valor exacto de dicha medición debería ser de $30.00$ *m*, ¿cuál es el error absoluto obtenido? Cuál sería el error absoluto si se disminuye un orden de magnitud las cantidades?

```python
f = 30.0
F = 29.9
```

```python
ea = abs(f - F)
print("{0:6.4f}".format(ea))
```

```python
# reduciendo un órden de magnitud las cantidades

f = 3.0
F = 2.9
```

```python
ea = abs(f - F)
print("{0:6.4f}".format(ea))
```

Se observa que el valor del error absoluto es igual ($\approx 0.1$), independiente de la magnitud de las cantidades.

[Volver a la Tabla de Contenido](#TOC)

#### Error relativo

$$e_r (\%)= \frac{e_a}{|f|}=\frac{|f-F|}{|f|} \times 100 \%$$

***Ejemplo:*** Repetir el ejemplo anterior, pero calculando el error relativo porcentual.

```python
f = 30.0
F = 29.9
```

```python
er = abs(f - F) / f * 100
print("{0:6.4f}%".format(er))
```

```python
# reduciendo un órden de magnitud las cantidades

f = 3.0
F = 2.9
```

```python
er = abs(f - F) / f * 100
print("{0:6.4f}%".format(er))
```

Se observa que los resultados son diferentes y es mayor cuando las cantidades medidas son menores.

Entre las dos formas de representar el error, la relativa es más consistente con la magnitud de lo que se está midiendo.

[Volver a la Tabla de Contenido](#TOC)

## Notación $\text{Big}-\mathcal{O}$

sea $$f(x)= \mathcal{O}(g(x)) \text{ cuando } x \rightarrow a$$

si y solo si

$$|f(x)|\leq M|g(x)| \text{ cuando } |x-a| < \delta \text{ donde } M, a > 0$$

En la práctica, usamos la notación $\text{Big}-\mathcal{O}$ para decir algo sobre cómo se pueden comportar los términos que podemos haber dejado fuera de una serie. Veamos el siguiente ejemplo de la aproximación de la serie de Taylor:

***Ejemplo:***

sea $f(x) = \sin(x)$ con $x_0 = 0$ entonces

$$T_N(x) = \sum^N_{n=0} (-1)^{n} \frac{x^{2n+1}}{(2n+1)!}$$

Podemos escribir $f(x)$ como

$$f(x) = x - \frac{x^3}{6} + \frac{x^5}{120} + \mathcal{O}(x^7)$$

Esto se vuelve más útil cuando lo vemos como lo hicimos antes con $\Delta x$:

$$f(x) = \Delta x - \frac{\Delta x^3}{6} + \frac{\Delta x^5}{120} + \mathcal{O}(\Delta x^7)$$

[Volver a la Tabla de Contenido](#TOC)

### Reglas para el error de propagación basado en la notación $\text{Big}-\mathcal{O}$

En general, existen dos teoremas que no necesitan prueba y se mantienen cuando el valor de $x$ es grande:

Sea

$$\begin{aligned}
    f(x) &= p(x) + \mathcal{O}(x^n) \\
    g(x) &= q(x) + \mathcal{O}(x^m) \\
    k &= \max(n, m)
\end{aligned}$$

Entonces

$$
    f+g = p + q + \mathcal{O}(x^k)
$$

y

\begin{align}
    f \cdot g &= p \cdot q + p \mathcal{O}(x^m) + q \mathcal{O}(x^n) + O(x^{n + m}) \\
    &= p \cdot q + \mathcal{O}(x^{n+m})
\end{align}

De otra forma, si estamos interesados en valores pequeños de $x$, $\Delta x$, la expresión puede ser modificada como sigue:

\begin{align}
    f(\Delta x) &= p(\Delta x) + \mathcal{O}(\Delta x^n) \\
    g(\Delta x) &= q(\Delta x) + \mathcal{O}(\Delta x^m) \\
    r &= \min(n, m)
\end{align}

entonces

$$
    f+g = p + q + O(\Delta x^r)
$$

y

\begin{align}
    f \cdot g &= p \cdot q + p \cdot \mathcal{O}(\Delta x^m) + q \cdot \mathcal{O}(\Delta x^n) + \mathcal{O}(\Delta x^{n+m}) \\
    &= p \cdot q + \mathcal{O}(\Delta x^r)
\end{align}

***Nota:*** En este caso, supongamos que al menos el polinomio con $k=max(n,m)$ tiene la siguiente forma:

$$
    p(\Delta x) = 1 + p_1 \Delta x + p_2 \Delta x^2 + \ldots
$$

o

$$
    q(\Delta x) = 1 + q_1 \Delta x + q_2 \Delta x^2 + \ldots
$$

para que $\mathcal{O}(1)$

de modo que hay un término $\mathcal{O}(1)$ que garantiza la existencia de $\mathcal{O}(\Delta x^r)$ en el producto final.

Para tener una idea de por qué importa más la potencia en $\Delta x$ al considerar la convergencia, la siguiente figura muestra cómo las diferentes potencias en la tasa de convergencia pueden afectar la rapidez con la que converge nuestra solución. Tenga en cuenta que aquí estamos dibujando los mismos datos de dos maneras diferentes. Graficar el error como una función de $\Delta x$ es una forma común de mostrar que un método numérico está haciendo lo que esperamos y muestra el comportamiento de convergencia correcto. Dado que los errores pueden reducirse rápidamente, es muy común trazar este tipo de gráficos en una escala log-log para visualizar fácilmente los resultados. Tenga en cuenta que si un método fuera realmente del orden $n$, será una función lineal en el espacio log-log con pendiente $n$.

```python
dx = np.linspace(1.0, 1e-4, 100)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2.0)
axes = []
axes.append(fig.add_subplot(1, 2, 1))
axes.append(fig.add_subplot(1, 2, 2))

for n in range(1, 5):
    axes[0].plot(dx, dx**n, label="$\Delta x^%s$" % n)
    axes[1].loglog(dx, dx**n, label="$\Delta x^%s$" % n)

axes[0].legend(loc=2)
axes[1].set_xticks([10.0**(-n) for n in range(5)])
axes[1].set_yticks([10.0**(-n) for n in range(16)])
axes[1].legend(loc=4)
for n in range(2):
    axes[n].set_title("Crecimiento del Error vs. $\Delta x^n$")
    axes[n].set_xlabel("$\Delta x$")
    axes[n].set_ylabel("Error Estimado")
    axes[n].set_title("Crecimiento de las diferencias")
    axes[n].set_xlabel("$\Delta x$")
    axes[n].set_ylabel("Error Estimado")

plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

## Error de truncamiento

***Teorema de Taylor:*** Sea $f(x) \in C^{m+1}[a,b]$ y $x_0 \in [a,b]$, para todo $x \in (a,b)$ existe un número $c = c(x)$ que se encuentra entre $x_0$ y $x$ tal que

$$ f(x) = T_N(x) + R_N(x)$$

donde $T_N(x)$ es la aproximación del polinomio de Taylor

$$T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\times(x-x_0)^n}{n!}$$

y $R_N(x)$ es el residuo (la parte de la serie que obviamos)

$$R_N(x) = \frac{f^{(n+1)}(c) \times (x - x_0)^{n+1}}{(n+1)!}$$

Otra forma de pensar acerca de estos resultados consiste en reemplazar $x - x_0$ con $\Delta x$. La idea principal es que el residuo $R_N(x)$ se vuelve mas pequeño cuando $\Delta x \rightarrow 0$.

$$T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\times \Delta x^n}{n!}$$

y $R_N(x)$ es el residuo (la parte de la serie que obviamos)

$$ R_N(x) = \frac{f^{(n+1)}(c) \times \Delta x^{n+1}}{(n+1)!} \leq M \Delta x^{n+1}$$

***Ejemplo 1:***

$f(x) = e^x$ con $x_0 = 0$

Usando esto podemos encontrar expresiones para el error relativo y absoluto en función de $x$ asumiendo $N=2$.

Derivadas:
$$\begin{aligned}
    f'(x) &= e^x \\
    f''(x) &= e^x \\
    f^{(n)}(x) &= e^x
\end{aligned}$$

Polinomio de Taylor:
$$\begin{aligned}
    T_N(x) &= \sum^N_{n=0} e^0 \frac{x^n}{n!} \Rightarrow \\
    T_2(x) &= 1 + x + \frac{x^2}{2}
\end{aligned}$$

Restos:
$$\begin{aligned}
    R_N(x) &= e^c \frac{x^{n+1}}{(n+1)!} = e^c \times \frac{x^3}{6} \quad \Rightarrow \\
    R_2(x) &\leq \frac{e^1}{6} \approx 0.5
\end{aligned}$$

Precisión:
$$
    e^1 = 2.718\ldots \\
    T_2(1) = 2.5 \Rightarrow e \approx 0.2 ~~ r \approx 0.1
$$

¡También podemos usar el paquete `sympy` que tiene la capacidad de calcular el polinomio de *Taylor* integrado!

```python
x = sympy.symbols('x')
f = sympy.symbols('f', cls=sympy.Function)

f = sympy.exp(x)
f.series(x0=0, n=11)
```

```python
a = 1.1**500

print(a)
```

Graficando

```python
x = np.linspace(-5, 5, 100)
T_N = 1.0 + x + x**2 / 2.0 + x**3 / 6.0 + x**4 / 24.0 + x**5 / 120.0 + x**6 / 720.0 + x**7 / 5040.0 + x**8 / 40320.0 + x**9 / 362880
R_N = np.exp(1) * x**10 / 3628800.0

plt.plot(x, T_N, 'r', x, np.exp(x), 'k', x, R_N, 'b')
plt.plot(0.0, 1.0, 'o', markersize=10)
plt.grid(True)
plt.xlabel("x")
plt.ylabel("$f(x)$, $T_N(x)$, $R_N(x)$")
plt.legend(["$T_N(x)$", "$f(x)$", "$R_N(x)$"], loc=2)
plt.show()
```

```python
R_N
```

***Ejemplo 2:***

Aproximar

$$ f(x) = \frac{1}{x} \quad x_0  = 1,$$

usando $x_0 = 1$ para el tercer termino de la serie de Taylor.

$$\begin{aligned}
    f'(x) &= -\frac{1}{x^2} \\
    f''(x) &= \frac{2}{x^3} \\
    f^{(n)}(x) &= \frac{(-1)^n n!}{x^{n+1}}
\end{aligned}$$

$$\begin{aligned}
    T_N(x) &= \sum^N_{n=0} (-1)^n (x-1)^n \Rightarrow \\
    T_2(x) &= 1 - (x - 1) + (x - 1)^2
\end{aligned}$$

$$\begin{aligned}
    R_N(x) &= \frac{(-1)^{n+1}(x - 1)^{n+1}}{c^{n+2}} \Rightarrow \\
    R_2(x) &= \frac{-(x - 1)^{3}}{c^{4}}
\end{aligned}$$

```python
x = np.linspace(0.8, 2, 100)
T_N = 1.0 - (x-1) + (x-1)**2
R_N = -(x-1.0)**3 / (1.1**4)

plt.plot(x, T_N, 'r', x, 1.0 / x, 'k', x, R_N, 'b')
plt.plot(1.0, 1.0, 'o', markersize=10)
plt.grid(True)
plt.xlabel("x")
plt.ylabel("$f(x)$, $T_N(x)$, $R_N(x)$")

plt.legend(["$T_N(x)$", "$f(x)$", "$R_N(x)$"], loc=8)
plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

### Laboratorio Numérico 1

$\color{red}{\textbf{Ejercicio:}}$ Realice la expansión de la serie de Taylor para los dos ejemplos anteriores con 3, 4 y 5 términos. Cuál es el error que se tiene a medida que se adicionan más términos? Realice una gráfica comparativa del Residuo que se obtiene para cada término adicional. Haga un análisis de lo que sucede. Si extendemos hasta el "infinito" dicho residuo qué pueden concluir?

[Volver a la Tabla de Contenido](#TOC)

## Error de punto flotante

Errores surgen de aproximar números reales con números de precisión finita

$$\pi \approx 3.14$$

o $\frac{1}{3} \approx 0.333333333$ en decimal, los resultados forman un número finito de registros para representar cada número.

***Ej.:*** considere la representación de $\sqrt{2}=1.4142 \ldots$. Como sabemos, éste es un número irracional, es decir, tiene una cantidad infinita de dígitos decimales. El computador almacena de forma incompleta la represenación de ese valor empleando cierta cantidad de números decimales

$$2 - (\sqrt{2})^2$$

```python
a = np.sqrt(2)
print("a: ", a)
```

    a:  1.4142135623730951

```python
b = abs(2-a**2)
print("|2-a^2| = ", b)
```

    |2-a^2| =  4.440892098500626e-16

[Volver a la Tabla de Contenido](#TOC)

### Aritmética de punto flotante

Los números en sistemas de [punto flotante](https://en.wikipedia.org/wiki/Floating-point_arithmetic "Floating point arithmetic") se representan como una serie de bits que representan diferentes partes de un número. En los sistemas de punto flotante normalizados, existen algunas convenciones estándar para el uso de estos bits. En general, los números se almacenan dividiéndolos en la forma

$$fl(x) = \pm (0.d_1 d_2 d_3 \ldots d_p)_\beta \times \beta^E$$

donde los digitos $\{d_i\}_{i=1}^p$ son enteros tales que $0\leq d_i \leq \beta-1$ y $d_1 \neq 0$

El sistema se caracteriza por cuatro números enteros:

- la *base* $\beta>1$. Para el sistema binario $\beta = 2$, para decimal $\beta = 10$, etc.

- La precisión $p \geq 1$, que representa la cantidad de dígitos significativos, y

- el *exponente* $E$, que es un entero en el rango $[E_{\min}, E_{\max}]$

$\pm$ es un bit único y representa el signo del número.

Los puntos importantes en cualquier sistema de punto flotante son:

1. Existe un conjunto discreto y finito de números representables.

2. Estos números representables no están distribuidos uniformemente en la línea real

3. La aritmética en sistemas de punto flotante produce resultados diferentes de la aritmética de precisión infinita (es decir, matemática "real")

[Volver a la Tabla de Contenido](#TOC)

### Notación de punto flotante

Es común encontrar la siguiente notación para representar un conjunto de números de punto flotante:

$$F(\beta, p, E_{min}, E_{max})$$

[Volver a la Tabla de Contenido](#TOC)

### Propiedades de los sistemas de punto flotante

Todos los sistemas de punto flotante se caracterizan por varios números importantes

- Número normalizado reducido ([*underflow*](https://en.wikipedia.org/wiki/Arithmetic_underflow) si está por debajo, relacionado con números sub-normales alrededor de cero)

- Número normalizado más grande ([*overflow*](https://en.wikipedia.org/wiki/Integer_overflow))

- Cero

- $\epsilon$ o $\epsilon_{mach}$

- `Inf` y `nan`

***Ejemplo: Sistema de juguete***

Considere el sistema decimal de 2 digitos de  precisión (normalizado)

$$F(10,2,-2,0)$$

$$f = \pm 0.d_1d_2 \times 10^E$$

con $E \in [-2, 0]$.

**Numero y distribución de números**

1. Cuántos números pueden representarse con este sistema?

2. Cuál es la distribución en la línea real?

3. Cuáles son los límites underflow y overflow?

Cuántos números pueden representarse con este sistema?

$$f = \pm 0.d_1d_2 \times 10^E ~~~ \text{con} ~~~ E \in [-2, 0]$$

$$2 \times 9 \times 10 \times 3 + 1 = 541$$

Cuál es la distribución en la recta "real"?

```python
d_1_values = [1, 2, 3, 4, 5, 6, 7, 8, 9]
d_2_values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
E_values = [0, -1, -2]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, '+', markersize=20)
axes.plot([-10.0, 10.0], [0.0, 0.0], 'k')

axes.set_title("Distribución de Valores")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-0.1, 0.1])
plt.show()
```

Cuáles son los límites superior (overflow) e inferior (underflow)?

- El menor número que puede ser representado (underflow) es: $1.0 \times 10^{-2} = 0.01$

- El mayor número que puede ser representado (overflow) es:  $9.9 \times 10^0 = 9.9$

[Volver a la Tabla de Contenido](#TOC)

### Sistema Binario

Considere el sistema en base 2 de 2 dígitos de precisión

$$F(2,2,-1,1)$$

$$F=\pm 0.d_1d_2 \times 2^E \quad \text{con} \quad E \in [-1, 1]$$

#### Numero y distribución de números

1. Cuántos números pueden representarse con este sistema?

2. Cuál es la distribución en la línea real?

3. Cuáles son los límites underflow y overflow?

Cuántos números pueden representarse en este sistema?

$$f=\pm 0.d_1d_2 \times 2^E ~~~~ \text{con} ~~~~ E \in [-1, 1]$$

$$ 2 \times 1 \times 2 \times 3 + 1 = 13$$

Cuál es la distribución en la línea real?

```python
d_1_values = [1]
d_2_values = [0, 1]
E_values = [1, 0, -1]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.5) * 2**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.5) * 2**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, 'r+', markersize=20)
axes.plot([-4.5, 4.5], [0.0, 0.0], 'k')

axes.set_title("Distribución de Valores")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-3.5, 3.5])
plt.show()
```

Cuáles son los límites superior (*overflow*) e inferior (*underflow*)?

- El menor número que puede ser representado (*underflow*) es: $1.0 \times 2^{-1} = 0.5$

- El mayor número que puede ser representado (*overflow*) es:  $1.1 \times 2^1 = 3$

Observe que estos números son en sistema binario.

Una rápida regla de oro:

$$2^3 2^2 2^1 2^0 . 2^{-1} 2^{-2} 2^{-3}$$

corresponde a

8s, 4s, 2s, 1s . mitades, cuartos, octavos, $\ldots$

[Volver a la Tabla de Contenido](#TOC)

***Ejercicio:*** Cuál sería la representación en punto flotante del siguiente conjunto de números:

$$F(2,3,-1,3)$$

- Cuántos números se pueden representar?

- Cuál sería el menor número representable?

- Cuál sería el mayor número?

[Volver a la Tabla de Contenido](#TOC)

### Sistema real - [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) sistema binario de punto flotante

#### Precisión simple

- Almacenamiento total es de 32 bits

- Exponente de 8 bits $\Rightarrow E \in [-126, 127]$

- Fracción 23 bits ($p = 24$)

```
s EEEEEEEE FFFFFFFFFFFFFFFFFFFFFFF
0 1      8 9                     31
```

Overflow $= 2^{127} \approx 3.4 \times 10^{38}$

Underflow $= 2^{-126} \approx 1.2 \times 10^{-38}$

$\epsilon_{\text{machine}} = 2^{-23} \approx 1.2 \times 10^{-7}$

[Volver a la Tabla de Contenido](#TOC)

#### Precisión doble

- Almacenamiento total asignado es 64 bits

- Exponenete de 11 bits $\Rightarrow E \in [-1022, 1024]$

- Fracción de 52 bits ($p = 53$)

```
s EEEEEEEEEE FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FF
0 1       11 12                                                      63
```
Overflow $= 2^{1024} \approx 1.8 \times 10^{308}$

Underflow $= 2^{-1022} \approx 2.2 \times 10^{-308}$

$\epsilon_{\text{machine}} = 2^{-52} \approx 2.2 \times 10^{-16}$

[Volver a la Tabla de Contenido](#TOC)

### Acceso de Python a números de la IEEE

Accede a muchos parámetros importantes, como el epsilon de la máquina

```python
import numpy as np
np.finfo(float).eps
```

```python
np.finfo(float).eps

print(np.finfo(np.float16))
print(np.finfo(np.float32))
print(np.finfo(float))
```

[Volver a la Tabla de Contenido](#TOC)

### Calculo "manual" del $\epsilon_{mach}$

La determinación "manual" del $\epsilon_{mach}$ es muy simple. Veamos el siguiente algoritmo

```python
eps = 1.0

while 1.0 + eps > 1.0:
    eps = eps / 2.0

print(eps)
```

Si lo comparamos con el valor obtenido en el numeral anterior para `float64`, `2.2204460492503131e-16`, se observa que es del orden de dos veces menor, por qué?

[Volver a la Tabla de Contenido](#TOC)

## Por qué debería importarnos esto?

- Aritmética de punto flotante no es conmutativa o asociativa

- Errores de punto flotante compuestos, No asuma que la precisión doble es suficiente

- Mezclar precisión es muy peligroso

***EL ORDEN DE LOS FACTORES NO ALTERA EL PRODUCTO???***

$$2 \times 3 = 3 \times 2 = 6$$

$$ 10^{300} \times 10^{50} \times 10^{-60} = 10^{300} \times 10^{-60} \times 10^{50} ??$$

```python
a = 10**300
b = 10**10
c = 10**-60

```

```python
d1 = a * b * c
print("d1: ", d1)
```

```python
d2 = b * c * a
print("d2: ", d2)

```

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 1: Aritmética simple

Aritmética simple $\delta < \epsilon_{\text{machine}}$

   $$(1+\delta) - 1 = 1 - 1 = 0$$

   $$1 - 1 + \delta = \delta$$

```python
delta = 1.0000000001 * eps

value = (1 + delta) - 1
print(value)
```

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 2: Cancelación catastrófica

Miremos qué sucede cuando sumamos dos números $x$ y $y$ cuando $x+y \neq 0$. De hecho, podemos estimar estos límites haciendo un análisis de error. Aquí necesitamos presentar la idea de que cada operación de punto flotante introduce un error tal que

$$
    \text{fl}(x ~\text{op}~ y) = (x ~\text{op}~ y) (1 + \delta)
$$

donde $\text{fl}(\cdot)$ es una función que devuelve la representación de punto flotante de la expresión encerrada, $\text{op}$ es alguna operación (ex. $+, -, \times, /$), y $\delta$ es el error de punto flotante debido a  $\text{op}$.

De vuelta a nuestro problema en cuestión. El error de coma flotante debido a la suma es

$$\text{fl}(x + y) = (x + y) (1 + \delta).$$

Comparando esto con la solución verdadera usando un error relativo tenemos

$$\begin{aligned}
    \frac{(x + y) - \text{fl}(x + y)}{x + y} &= \frac{(x + y) - (x + y) (1 + \delta)}{x + y} = \delta.
\end{aligned}$$

entonces si $\delta = \mathcal{O}(\epsilon_{\text{machine}})$ no estaremos muy preocupados.

Que pasa si consideramos un error de punto flotante en la representación de $x$ y $y$, $x \neq y$, y decimos que $\delta_x$ y $\delta_y$ son la magnitud de los errores en su representación. Asumiremos que esto constituye el error de punto flotante en lugar de estar asociado con la operación en sí.

Dado todo esto, tendríamos

$$\begin{aligned}
    \text{fl}(x + y) &= x (1 + \delta_x) + y (1 + \delta_y) \\
    &= x + y + x \delta_x + y \delta_y \\
    &= (x + y) \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right)
\end{aligned}$$

Calculando nuevamente el error relativo, tendremos

$$\begin{aligned}
    \frac{x + y - (x + y) \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right)}{x + y} &= 1 - \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right) \\
    &= \frac{x}{x + y} \delta_x + \frac{y}{x + y} \delta_y \\
    &= \frac{1}{x + y} (x \delta_x + y \delta_y)
\end{aligned}$$

Lo importante aquí es que ahora el error depende de los valores de $x$ y $y$, y más importante aún, su suma. De particular preocupación es el tamaño relativo de $x + y$. A medida que se acerca a cero en relación con las magnitudes de $x$ y $y$, el error podría ser arbitrariamente grande. Esto se conoce como ***cancelación catastrófica***.

```python
dx = np.array([10**(-n) for n in range(1, 16)])
x = 1.0 + dx
y = -np.ones(x.shape)
error = np.abs(x + y - dx) / (dx)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1)
axes.loglog(dx, x + y, 'o-')
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("$x + y$")
axes.set_title("$\Delta x$ vs. $x+y$")

axes = fig.add_subplot(1, 2, 2)
axes.loglog(dx, error, 'o-')
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("$|x + y - \Delta x| / \Delta x$")
axes.set_title("Diferencia entre $x$ y $y$ vs. Error relativo")

plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 3: Evaluación de una función

Considere la función

$$
    f(x) = \frac{1 - \cos x}{x^2}
$$

con $x\in[-10^{-4}, 10^{-4}]$.

Tomando el límite cuando $x \rightarrow 0$ podemos ver qué comportamiento esperaríamos ver al evaluar esta función:

$$
    \lim_{x \rightarrow 0} \frac{1 - \cos x}{x^2} = \lim_{x \rightarrow 0} \frac{\sin x}{2 x} = \lim_{x \rightarrow 0} \frac{\cos x}{2} = \frac{1}{2}.
$$

¿Qué hace la representación de punto flotante?

```python
f = (1-np.cos(0))/0**2
```

```python
x = np.linspace(-1e-3, 1e-3, 100, dtype=np.float32)
error = (0.5 - (1.0 - np.cos(x)) / x**2) / 0.5

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, error, 'o')
axes.set_xlabel("x")
axes.set_ylabel("Error Relativo")
```

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 4: Evaluación de un Polinomio

 $$f(x) = x^7 - 7x^6 + 21 x^5 - 35 x^4 + 35x^3-21x^2 + 7x - 1$$

```python
x = np.linspace(0.988, 1.012, 1000, dtype=np.float16)
y = x**7 - 7.0 * x**6 + 21.0 * x**5 - 35.0 * x**4 + 35.0 * x**3 - 21.0 * x**2 + 7.0 * x - 1.0

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, y, 'r')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_ylim((-0.1, 0.1))
axes.set_xlim((x[0], x[-1]))
plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 5: Evaluación de una función racional

Calcule $f(x) = x + 1$ por la función $$F(x) = \frac{x^2 - 1}{x - 1}$$

¿Cuál comportamiento esperarías encontrar?

```python
x = np.linspace(0.5, 1.5, 101, dtype=np.float16)
f_hat = (x**2 - 1.0) / (x - 1.0)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, np.abs(f_hat - (x + 1.0)))
axes.set_xlabel("$x$")
axes.set_ylabel("Error Absoluto")
plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

## Combinación de error

En general, nos debemos ocupar de la combinación de error de truncamiento con el error de punto flotante.

- Error de Truncamiento: errores que surgen de la aproximación de una función, truncamiento de una serie.

$$\sin x \approx x - \frac{x^3}{3!} + \frac{x^5}{5!} + O(x^7)$$

- Error de punto flotante: errores derivados de la aproximación de números reales con números de precisión finita

$$\pi \approx 3.14$$

o $\frac{1}{3} \approx 0.333333333$ en decimal, los resultados forman un número finito de registros para representar cada número.

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 1:

Considere la aproximación de diferencias finitas donde $f(x) = e^x$ y estamos evaluando en $x=1$

$$f'(x) \approx \frac{f(x + \Delta x) - f(x)}{\Delta x}$$

Compare el error entre disminuir $\Delta x$ y la verdadera solucion $f'(1) = e$

```python
delta_x = np.linspace(1e-20, 5.0, 100)
delta_x = np.array([2.0**(-n) for n in range(1, 60)])
x = 1.0
f_hat_1 = (np.exp(x + delta_x) - np.exp(x)) / (delta_x)
f_hat_2 = (np.exp(x + delta_x) - np.exp(x - delta_x)) / (2.0 * delta_x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.loglog(delta_x, np.abs(f_hat_1 - np.exp(1)), 'o-', label="Unilateral")
axes.loglog(delta_x, np.abs(f_hat_2 - np.exp(1)), 's-', label="Centrado")
axes.legend(loc=3)
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("Error Absoluto")
plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 2:

Evalúe $e^x$ con la serie de *Taylor*

$$e^x = \sum^\infty_{n=0} \frac{x^n}{n!}$$

podemos elegir $n< \infty$ que puede aproximarse $e^x$ en un rango dado $x \in [a,b]$ tal que el error relativo $E$ satisfaga $E<8 \cdot \varepsilon_{\text{machine}}$?

¿Cuál podría ser una mejor manera de simplemente evaluar el polinomio de Taylor directamente por varios $N$?

```python
def my_exp(x, N=10):
    value = 0.0
    for n in range(N + 1):
        value += x**n / scipy.special.factorial(n)

    return value

x = np.linspace(-2, 2, 100, dtype=np.float32)
for N in range(1, 50):
    error = np.abs((np.exp(x) - my_exp(x, N=N)) / np.exp(x))
    if np.all(error < 8.0 * np.finfo(float).eps):
        break

print(N)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, error)
axes.set_xlabel("x")
axes.set_ylabel("Error Relativo")
plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 3: Error relativo

Digamos que queremos calcular el error relativo de dos valores $x$ y $y$ usando $x$ como valor de normalización

$$
    E = \frac{x - y}{x}
$$
y
$$
    E = 1 - \frac{y}{x}
$$

son equivalentes. En precisión finita, ¿qué forma pidría esperarse que sea más precisa y por qué?

Ejemplo tomado de [blog](https://nickhigham.wordpress.com/2017/08/14/how-and-how-not-to-compute-a-relative-error/) posteado por Nick Higham*

Usando este modelo, la definición original contiene dos operaciones de punto flotante de manera que

$$\begin{aligned}
    E_1 = \text{fl}\left(\frac{x - y}{x}\right) &= \text{fl}(\text{fl}(x - y) / x) \\
    &= \left[ \frac{(x - y) (1 + \delta_+)}{x} \right ] (1 + \delta_/) \\
    &= \frac{x - y}{x}  (1 + \delta_+) (1 + \delta_/)
\end{aligned}$$

Para la otra formulación tenemos

$$\begin{aligned}
    E_2 = \text{fl}\left( 1 - \frac{y}{x} \right ) &= \text{fl}\left(1 - \text{fl}\left(\frac{y}{x}\right) \right) \\
    &= \left(1 - \frac{y}{x} (1 + \delta_/) \right) (1 + \delta_-)
\end{aligned}$$

Si suponemos que todos las $\text{op}$s tienen magnitudes de error similares, entonces podemos simplificar las cosas dejando que

$$
    |\delta_\ast| \le \epsilon.
$$

Para comparar las dos formulaciones, nuevamente usamos el error relativo entre el error relativo verdadero $e_i$ y nuestras versiones calculadas $E_i$

Definición original

$$\begin{aligned}
    \frac{e - E_1}{e} &= \frac{\frac{x - y}{x} - \frac{x - y}{x}  (1 + \delta_+) (1 + \delta_/)}{\frac{x - y}{x}} \\
    &\le 1 - (1 + \epsilon) (1 + \epsilon) = 2 \epsilon + \epsilon^2
\end{aligned}$$

Definición manipulada:

$$\begin{aligned}
    \frac{e - E_2}{e} &= \frac{e - \left[1 - \frac{y}{x}(1 + \delta_/) \right] (1 + \delta_-)}{e} \\
    &= \frac{e - \left[e - \frac{y}{x} \delta_/) \right] (1 + \delta_-)}{e} \\
    &= \frac{e - \left[e + e\delta_- - \frac{y}{x} \delta_/ - \frac{y}{x} \delta_/ \delta_-)) \right] }{e} \\
    &= - \delta_- + \frac{1}{e} \frac{y}{x} \left(\delta_/ + \delta_/ \delta_- \right) \\
    &= - \delta_- + \frac{1 -e}{e} \left(\delta_/ + \delta_/ \delta_- \right) \\
    &\le \epsilon + \left |\frac{1 - e}{e}\right | (\epsilon + \epsilon^2)
\end{aligned}$$

Vemos entonces que nuestro error de punto flotante dependerá de la magnitud relativa de $e$

```python
# Based on the code by Nick Higham
# https://gist.github.com/higham/6f2ce1cdde0aae83697bca8577d22a6e
# Compares relative error formulations using single precision and compared to double precision

N = 501    # Note: Use 501 instead of 500 to avoid the zero value
d = numpy.finfo(numpy.float32).eps * 1e4
a = 3.0
x = a * numpy.ones(N, dtype=numpy.float32)
y = [x[i] + numpy.multiply((i - numpy.divide(N, 2.0, dtype=numpy.float32)), d, dtype=numpy.float32) for i in range(N)]

# Compute errors and "true" error
relative_error = numpy.empty((2, N), dtype=numpy.float32)
relative_error[0, :] = numpy.abs(x - y) / x
relative_error[1, :] = numpy.abs(1.0 - y / x)
exact = numpy.abs( (numpy.float64(x) - numpy.float64(y)) / numpy.float64(x))

# Compute differences between error calculations
error = numpy.empty((2, N))
for i in range(2):
    error[i, :] = numpy.abs((relative_error[i, :] - exact) / numpy.abs(exact))

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(y, error[0, :], '.', markersize=10, label="$|x-y|/|x|$")
axes.semilogy(y, error[1, :], '.', markersize=10, label="$|1-y/x|$")

axes.grid(True)
axes.set_xlabel("y")
axes.set_ylabel("Error Relativo")
axes.set_xlim((numpy.min(y), numpy.max(y)))
axes.set_ylim((5e-9, numpy.max(error[1, :])))
axes.set_title("Comparasión Error Relativo")
axes.legend()
plt.show()
```

Algunos enlaces de utilidad con respecto al punto flotante IEEE:

- [What Every Computer Scientist Should Know About Floating-Point Arithmetic](http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)

- [IEEE 754 Floating Point Calculator](http://babbage.cs.qc.edu/courses/cs341/IEEE-754.html)

- [Numerical Computing with IEEE Floating Point Arithmetic](http://epubs.siam.org/doi/book/10.1137/1.9780898718072)

[Volver a la Tabla de Contenido](#TOC)

## Operaciones de conteo

- ***Error de truncamiento:*** *¿Por qué no usar más términos en la serie de Taylor?*

- ***Error de punto flotante:*** *¿Por qué no utilizar la mayor precisión posible?*

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 1: Multiplicación matriz - vector

Sea $A, B \in \mathbb{R}^{N \times N}$ y $x \in \mathbb{R}^N$.

1. Cuenta el número aproximado de operaciones que tomará para calcular $Ax$

2. Hacer lo mismo para $AB$

***Producto Matriz-vector:*** Definiendo $[A]_i$ como la $i$-ésima fila de $A$ y $A_{ij}$ como la $i$,$j$-ésima entrada entonces

$$
    A x = \sum^N_{i=1} [A]_i \cdot x = \sum^N_{i=1} \sum^N_{j=1} A_{ij} x_j
$$

Tomando un caso en particular, siendo $N=3$, entonces la operación de conteo es

$$
    A x = [A]_1 \cdot v + [A]_2 \cdot v + [A]_3 \cdot v = \begin{bmatrix}
        A_{11} \times v_1 + A_{12} \times v_2 + A_{13} \times v_3 \\
        A_{21} \times v_1 + A_{22} \times v_2 + A_{23} \times v_3 \\
        A_{31} \times v_1 + A_{32} \times v_2 + A_{33} \times v_3
    \end{bmatrix}
$$

Esto son 15 operaciones (6 sumas y 9 multiplicaciones)

Tomando otro caso, siendo $N=4$, entonces el conteo de operaciones es:

$$
    A x = [A]_1 \cdot v + [A]_2 \cdot v + [A]_3 \cdot v = \begin{bmatrix}
        A_{11} \times v_1 + A_{12} \times v_2 + A_{13} \times v_3 + A_{14} \times v_4 \\
        A_{21} \times v_1 + A_{22} \times v_2 + A_{23} \times v_3 + A_{24} \times v_4 \\
        A_{31} \times v_1 + A_{32} \times v_2 + A_{33} \times v_3 + A_{34} \times v_4 \\
        A_{41} \times v_1 + A_{42} \times v_2 + A_{43} \times v_3 + A_{44} \times v_4 \\
    \end{bmatrix}
$$

Esto lleva a 28 operaciones (12 sumas y 16 multiplicaciones).

Generalizando, hay $N^2$ mutiplicaciones y $N(N-1)$ sumas para un total de

$$
    \text{operaciones} = N (N - 1) + N^2 = \mathcal{O}(N^2).
$$

***Producto Matriz-Matriz ($AB$):*** Definiendo $[B]_j$ como la $j$-ésima columna de $B$ entonces

$$
    (A B)_{ij} = \sum^N_{i=1} \sum^N_{j=1} [A]_i \cdot [B]_j
$$

El producto interno de dos vectores es representado por

$$
    a \cdot b = \sum^N_{i=1} a_i b_i
$$

conduce a $\mathcal{O}(3N)$ operaciones. Como hay $N^2$ entradas en la matriz resultante, tendríamos $\mathcal{O}(N^3)$ operaciones

Existen métodos para realizar la multiplicación matriz - matriz más rápido. En la siguiente figura vemos una colección de algoritmos a lo largo del tiempo que han podido limitar el número de operaciones en ciertas circunstancias
$$
    \mathcal{O}(N^\omega)
$$

[Volver a la Tabla de Contenido](#TOC)

### Ejemplo 2: Método de Horner para evaluar polinomios

Dado

$$P_N(x) = a_0 + a_1 x + a_2 x^2 + \ldots + a_N x^N$$

o

$$P_N(x) = p_1 x^N + p_2 x^{N-1} + p_3 x^{N-2} + \ldots + p_{N+1}$$

queremos encontrar la mejor vía para evaluar $P_N(x)$

Primero considere dos vías para escribir $P_3$

$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$

y usando multiplicación anidada

$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$

Considere cuántas operaciones se necesitan para cada...

$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$

$$P_3(x) = \overbrace{p_1 \cdot x \cdot x \cdot x}^3 + \overbrace{p_2 \cdot x \cdot x}^2 + \overbrace{p_3 \cdot x}^1 + p_4$$

Sumando todas las operaciones, en general podemos pensar en esto como una pirámide

podemos estimar de esta manera que el algoritmo escrito de esta manera tomará aproximadamente $\mathcal{O}(N^2/2)$ operaciones para completar.

Mirando nuetros otros medios de evaluación

$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$

Aquí encontramos que el método es $\mathcal{O}(N)$ (el 2 generalmente se ignora en estos casos). Lo importante es que la primera evaluación es $\mathcal{O}(N^2)$ y la segunda $\mathcal{O}(N)$!

[Volver a la Tabla de Contenido](#TOC)

### Algoritmo

Complete la función e implemente el método de *Horner*

```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """
    pass
```

```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """
    ### ADD CODE HERE
    pass
```

```python
# Scalar version
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """

    y = p[0]
    for coefficient in p[1:]:
        y = y * x + coefficient

    return y

# Vectorized version
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x can by a NumPy ndarray.
    """

    y = numpy.ones(x.shape) * p[0]
    for coefficient in p[1:]:
        y = y * x + coefficient

    return y

p = [1, -3, 10, 4, 5, 5]
x = numpy.linspace(-10, 10, 100)
plt.plot(x, eval_poly(p, x))
plt.show()
```

[Volver a la Tabla de Contenido](#TOC)

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

```python

```


```python
from IPython.display import display, Image

```

# Introduction to Gradient Boosting Methods (GBMs)

We note that the following content mainly builds upon [Introduction to Boosted Trees](https://xgboost.readthedocs.io/en/stable/tutorials/model.html).

The technique of gradient boosting which has attracted significantly increasing attention in recent years due to its superior for solving
tabular data problems. The term **Gradient Boosting** originates from the paper *Greedy Function Approximation: A Gradient Boosting Machine*, by Friedman. This tutorial aims to provide a clear explanation on typical gradient boosting methods, such as gradient boosting decision trees (GBDT), in a self-contained and principled way using the elements of supervised learning.

## 1 Elements of Supervised Learning

First, we introduce the notations used throughout this tutorial as follows:

Given the training data $X=\{\mathbf{x}_i\}_{i=1}^{n}$, and the target $Y=\{y_i\}_{i=1}^{n}$, where $\mathbf{x}_i$ denotes the feature vector with respect to the $i$-th data instance, which can be either continuous or categorical features. $\mathbf{x}_{ij}$ denotes the $j$-th feature of $\mathbf{x}_i$.

### 1.1 Model and Parameters
The **model** in supervised learning usually refers to the mathematical structure by which the prediction $\hat{y}_{i}$ is made given the input $\mathbf{x}_i$. A common example is a **linear model**, where the prediction is given as $\hat{y}_i = \sum_j \theta_j \mathbf{x}_{ij}$, namely a linear combination of weighted input features. The prediction value can have different interpretations, depending on the task, i.e., regression or classification. For example, it can be logistic transformed to get the probability of positive class in logistic regression, and it can also be used as a ranking score when we want to rank the outputs.

The **parameters** are the undetermined part that we need to learn from data. In linear regression problems, the parameters are the coefficients $\theta$. Usually we will use $\theta$ to denote the parameters.

### 1.2 Objective Function: Training Loss + Regularization
With judicious choices for $y_i$, we may express a variety of tasks, such as regression, classification, and ranking.
The task of **training** the model amounts to finding the best parameters $\theta$ that best fit the training data $\mathbf{x}_i$ and labels $y_i$. In order to train the model, we need to define the **objective function**
to measure how well the model fit the training data.

A salient characteristic of objective functions is that they consist two parts: **training loss** and **regularization term**:

\begin{equation}
\text{obj}(\theta) = L(\theta) + \Omega(\theta)
\end{equation}

where $L$ is the training loss function, and $\Omega$ is
the **regularization term**. The training loss measures how *predictive* our model is with respect to the training data. A common choice of $L$ is the *mean squared error*, which is given by

$L(\theta) = \sum_i (y_i-\hat{y}_i)^2$

Another commonly used loss function is logistic loss, to be used for logistic regression:

$$L(\theta) = \sum_i[ y_i\ln (1+e^{-\hat{y}_i}) + (1-y_i)\ln (1+e^{\hat{y}_i})]$$

The **regularization term** is what people usually forget to add. The regularization term controls the complexity of the model, which helps us to avoid overfitting.

### 1.3 Why introduce the general principle?
The elements introduced above form the basic elements of supervised learning, and they are natural building blocks of machine learning toolkits. For example, you should be able to describe the differences and commonalities between gradient boosted trees and random forests. Understanding the process in a formalized way also helps us to understand the objective that we are learning and the reason behind the heuristics such as pruning and smoothing.

## 2 Gradient Boosting Decision Trees (GBDT)

### 2.1 Tree Ensembles
Now that we have introduced the elements of supervised learning, let us get started with real trees. The tree ensemble model consists of a set of classification and regression trees (CART). Here's a simple example of a CART that classifies whether someone will like a hypothetical computer game X.

Fig. A toy example for CART

We classify the members of a family into different leaves, and assign them the score on the corresponding leaf.
A CART is a bit different from decision trees, in which the leaf only contains decision values. In CART, a real score
is associated with each of the leaves, which gives us richer interpretations that go beyond classification.
This also allows for a principled, unified approach to optimization, as we will see in a later part of this tutorial.

Usually, a single tree is not strong enough to be used in practice. What is actually used is the ensemble model,
which sums the prediction of multiple trees together.

Fig. A toy example for tree ensemble, consisting of two CARTs

Here is an example of a tree ensemble of two trees. The prediction scores of each individual tree are summed up to get the final score.
If you look at the example, an important fact is that the two trees try to **complement** each other.
Mathematically, we can write our model in the form

$$\hat{y}_i = \sum_{k=1}^K f_k(x_i), f_k \in \mathcal{F}$$

where $K$ is the number of trees, $f$ is a function in the functional space $\mathcal{F}$, and $\mathcal{F}$ is the set of all possible CARTs. The objective function to be optimized is given by

$$\text{obj}(\theta) = \sum_i^n l(y_i, \hat{y}_i) + \sum_{k=1}^K \Omega(f_k)$$

Now here comes a trick question: what is the **model** used in random forests? Tree ensembles! So random forests and boosted trees are really the same models; the difference arises from how we train them. This means that, if you write a predictive service for tree ensembles, you only need to write one and it should work for both random forests and gradient boosted trees. (See [Treelite](https://treelite.readthedocs.io/en/latest/index.html) for an actual example.) One example of why elements of supervised learning rock.

### 2.2 Tree Boosting

Now that we introduced the model, let us turn to training: How should we learn the trees?
The answer is, as is always for all supervised learning models: **define an objective function and optimize it**!

Let the following be the objective function (remember it always needs to contain training loss and regularization):

$$\text{obj} = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t)}) + \sum_{i=1}^t\Omega(f_i)$$

In particular, $t$ denotes the training step, each step also corresponds to a member function $f$, i.e., a tree.

### 2.3 Additive Training

The first question we want to ask: what are the **parameters** of trees? You can find that what we need to learn are those functions $f_i$, **each containing the structure of the tree and the leaf scores**. Learning tree structure is much harder than traditional optimization problem where you can simply take the gradient. **It is intractable to learn all the trees at once**.
Instead, we use an **additive strategy: fix what we have learned, and add one new tree at a time**. In other words, the functions $f_1$ ... $f_{t-1}$ would be viewed as learned functions when we learn $f_t$. We write the prediction value at **step** $t$ as $\hat{y}_i^{(t)}$. Then we have

\begin{equation}
\begin{split}
\hat{y}_i^{(0)} &= 0\\
  \hat{y}_i^{(1)} &= f_1(x_i) = \hat{y}_i^{(0)} + f_1(x_i)\\
  \hat{y}_i^{(2)} &= f_1(x_i) + f_2(x_i)= \hat{y}_i^{(1)} + f_2(x_i)\\
  &\dots\\
  \hat{y}_i^{(t)} &= \sum_{k=1}^t f_k(x_i)= \hat{y}_i^{(t-1)} + f_t(x_i)
\end{split}
\end{equation}

It remains to ask: which tree do we want at each step?  A natural thing is to add the one that optimizes our objective.

\begin{equation}
\begin{split}
  \text{obj}^{(t)} & = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t)}) + \sum_{i=1}^t\Omega(f_i) \\
            & = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega(f_t) + \mathrm{constant}
\end{split}
\end{equation}

If we consider using mean squared error (MSE) as our loss function, the objective becomes

\begin{equation}
\begin{split}
  \text{obj}^{(t)} & = \sum_{i=1}^n (y_i - (\hat{y}_i^{(t-1)} + f_t(x_i)))^2 + \sum_{i=1}^t\Omega(f_i) \\
            & = \sum_{i=1}^n [2(\hat{y}_i^{(t-1)} - y_i)f_t(x_i) + f_t(x_i)^2] + \Omega(f_t) + \mathrm{constant}
\end{split}
\end{equation}

where the terms without $f_t$ are aggregated as a constant since the functions $f_1$ ... $f_{t-1}$ are learned functions in previous steps.

> In calculus, Taylor's theorem gives an approximation of a k-times
differentiable function around a given point by a polynomial of degree
k, called the kth-order Taylor polynomial. For a smooth function,
the Taylor polynomial is the truncation at the order k of the Taylor
series of the function.
>
> \begin{equation}
f(x)=\sum_{n=0}^{\infty}\frac{f^{(n)}(x_{0})}{n!}(x-x_{0})^{n}
\end{equation}
>
> The first-order Taylor polynomial is the linear approximation of the
function,
>
> $f(x)\approx f(x_{0})+f^{'}(x_{0})(x-x_{0})$
>
>The second-order Taylor polynomial is often referred to as the quadratic
approximation,
>
>$f(x)\approx f(x_{0})+f^{'}(x_{0})(x-x_{0})+f^{''}(x_{0})\frac{(x-x_{0})^{2}}{2}$

The form of MSE is friendly, with a first order term (usually called the residual) and a quadratic term.
For other losses of interest (for example, logistic loss), it is not so easy to get such a nice form.
So in the general case, we take the **Taylor expansion of the loss function up to the second order**:

\begin{equation}
\begin{split}
  \text{obj}^{(t)} = \sum_{i=1}^n [l(y_i, \hat{y}_i^{(t-1)}) + g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)] + \Omega(f_t) + \mathrm{constant}
\end{split}
\end{equation}

where the $g_i$ and $h_i$ are defined as

\begin{equation}
\begin{split}
  g_i &= \partial_{\hat{y}_i^{(t-1)}} l(y_i, \hat{y}_i^{(t-1)})\\
  h_i &= \partial_{\hat{y}_i^{(t-1)}}^2 l(y_i, \hat{y}_i^{(t-1)})
\end{split}
\end{equation}

> We note that the $f$ in the description on Taylor's theorem is different from $f_{t}$ in the loss function. Put another way, $g_i$ corresponds to $f^{'}(x_{0})$, $h_i$ corresponds to $f^{''}(x_{0})$, $f_t(x_i)$ corresponds to $x-x_{0}$, $\sum_{i=1}^n l(y_i, \hat{y}_i^{(t-1)})$ corresponds to $f(x_{0})$.

After we remove all the constants, the specific objective at step $t$ becomes

\begin{equation}
\begin{split}
  \sum_{i=1}^n [g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)] + \Omega(f_t)
\end{split}
\end{equation}

**This becomes our optimization goal for the new tree**. One important advantage of this definition is that
the value of the objective function only depends on $g_i$ and $h_i$. This is how the popular packages, such as **XGBoost** and **LightGBM**, support custom loss functions.
**We can optimize every loss function, including logistic regression and pairwise ranking, using exactly the same solver that takes $g_i$ and $h_i$ as input**!

### 2.4 Model Complexity
We have introduced the training step, but wait, there is one important thing, the **regularization term**!
We need to define the complexity of the tree $\Omega(f)$. In order to do so, let us first refine the definition of the tree $f(x)$ as

\begin{equation}
\begin{split}
  f_t(x) = w_{q(x)}, w \in R^T, q:R^d\rightarrow \{1,2,\cdots,T\} .
\end{split}
\end{equation}

Here $w$ is the vector of scores on leaves, $q$ is a function assigning each data point to the corresponding leaf, and $T$ is the number of leaves.
In XGBoost, the complexity is defined as

\begin{equation}
\begin{split}
  \Omega(f) = \gamma T + \frac{1}{2}\lambda \sum_{j=1}^T w_j^2
\end{split}
\end{equation}

Of course, there is more than one way to define the complexity, but this one works well in practice. The regularization is one part most tree packages treat
less carefully, or simply ignore. This was because the traditional treatment of tree learning only emphasized improving impurity, while the complexity control was left to heuristics.
By defining it formally, we can get a better idea of what we are learning and obtain models that perform well in the wild.

### 2.5 The Structure Score
Here is the magical part of the derivation. After re-formulating the tree model, we can write the objective value with the $t$-th tree as:

\begin{equation}
\begin{split}
  \text{obj}^{(t)} &\approx \sum_{i=1}^n [g_i w_{q(x_i)} + \frac{1}{2} h_i w_{q(x_i)}^2] + \gamma T + \frac{1}{2}\lambda \sum_{j=1}^T w_j^2\\
  &= \sum^T_{j=1} [(\sum_{i\in I_j} g_i) w_j + \frac{1}{2} (\sum_{i\in I_j} h_i + \lambda) w_j^2 ] + \gamma T
\end{split}
\end{equation}

where $I_j = \{i|q(x_i)=j\}$ is the set of indices of data points assigned to the $j$-th leaf.
Notice that in the second line we have changed the index of the summation because all the data points on the same leaf get the same score.
We could further compress the expression by defining $G_j = \sum_{i\in I_j} g_i$ and $H_j = \sum_{i\in I_j} h_i$:

\begin{equation}
\begin{split}
  \text{obj}^{(t)} = \sum^T_{j=1} [G_jw_j + \frac{1}{2} (H_j+\lambda) w_j^2] +\gamma T
\end{split}
\end{equation}

In this equation, $w_j$ are independent with respect to each other, the form $G_jw_j+\frac{1}{2}(H_j+\lambda)w_j^2$ is quadratic and the best $w_j$ for a given structure $q(x)$ and the best objective reduction we can get is:

\begin{equation}
\begin{split}
  w_j^\ast &= -\frac{G_j}{H_j+\lambda}\\
  \text{obj}^\ast &= -\frac{1}{2} \sum_{j=1}^T \frac{G_j^2}{H_j+\lambda} + \gamma T
\end{split}
\end{equation}

The last equation measures *how good* a tree structure $q(x)$ is.

Fig. An illustration of structure score (fitness)

If all this sounds a bit complicated, let's take a look at the picture, and see how the scores can be calculated.
Basically, for a given tree structure, we push the statistics $g_i$ and $h_i$ to the leaves they belong to,
sum the statistics together, and use the formula to calculate how good the tree is.
This score is like the impurity measure in a decision tree, except that it also takes the model complexity into account.

### 2.6 Learn the tree structure
Now that we have a way to measure how good a tree is, ideally we would enumerate all possible trees and pick the best one.
In practice this is intractable, so we will try to optimize one level of the tree at a time.
Specifically we try to split a leaf into two leaves, and the score it gains is

\begin{equation}
\begin{split}
  Gain = \frac{1}{2} \left[\frac{G_L^2}{H_L+\lambda}+\frac{G_R^2}{H_R+\lambda}-\frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right] - \gamma
\end{split}
\end{equation}

This formula can be decomposed as: 1) the score on the new left leaf, 2) the score on the new right leaf, 3) the score on the original leaf, 4) regularization on the additional leaf.
We can see an important fact here: if the gain is smaller than $\gamma$, we would do better not to add that branch. This is exactly the **pruning** techniques in tree based models! By using the principles of supervised learning, we can naturally come up with the reason these techniques work :)

### 2.7 Approximate Split Finding Using Feature Histograms

It is vital to find the optimal split of a tree node efficiently, as enumerating every possible split in a brute-force manner is impractical. Current works generally adopt a histogram-based algorithm for
fast and accurate split finding, like the following picture.

```python
path_img_his = "../img/histogram_split.png"
img_ltr_perqdata = Image(path_img_his, width = 800, height = 100)
display(img_ltr_perqdata)
```

Specifically, the algorithm considers only $k$ values (i.e., number of bins) for each feature as candidate splits rather than all possible splits (e.g., all feature values). The most common approach to propose the candidates is using the **quantile sketch** to approximate the feature distribution. After candidate splits are prepared, we enumerate
all instances on a tree node and accumulate their gradient statistics into two histograms, first- and second-order gradients, respectively. The histogram consists of $k$ bins, each of which sums the first- or second-order gradients of instances whose $j$-th feature values fall into that bin. In this way, each feature is summarized by two histograms. We find the best split of $j$-th feature upon the histograms that achieve the maximum gain value and the global best split is the best split over all features.

Another advantage of the histogram-based algorithm is that we can accelerate the algorithm by a histogram subtraction technique. The instances on two children nodes are **non-overlapping and mutual exclusive**, since **an instance will be classified onto either left or right child node when the parent node gets split** (since the bins or histograms are naturally ordered). Considering the basic operation of histogram is adding gradients, therefore, for a specific feature, the element-wise sum of first or second-order histograms of children nodes equals to that of parent.

- Example case: using local bins

    Motivated by this, we can significantly accelerate training by first constructing the histograms of the one child node with fewer instances, and then getting those of the sibling node via histogram subtraction (histograms of parent node are persist in memory). By doing so, we can skip at least one half of the instances. Since histogram construction usually dominates the computation cost, such subtraction technique can speed up the training process considerably.

> Limitation of additive tree learning

  Since it is intractable to enumerate all possible tree structures, we add one split at a time. This approach works well most of the time, but there are some edge cases that fail due to this approach. For those edge cases, training results in a degenerate model because we consider only one feature dimension at a time. See [Can Gradient Boosting Learn Simple Arithmetic?](<http://mariofilho.com/can-gradient-boosting-learn-simple-arithmetic/>) for an example.


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Toggle cell visibility <a href="javascript:code_toggle()">here</a>.''')
display(tag)

# Hide the code completely

# from IPython.display import HTML
# tag = HTML('''<style>
# div.input {
#     display:none;
# }
# </style>''')
# display(tag)
```

Toggle cell visibility here.

```python
# Examples:
# Factored form: 1/(x**2*(x**2 + 1))
# Expanded form: 1/(x**4+x**2)

import sympy as sym
from IPython.display import Latex, display, Markdown, Javascript, clear_output
from ipywidgets import widgets, Layout # Interactivity module
```

## Razcep na parcialne ulomke

Ob uporabi Laplaceove transformacije za analizo sistema, dobimo Laplaceovo transformacijo izstopnega signala z množenjem prenosne funkcije in Laplaceove transformacije vstopnega signala. Rezultat tega množenja je pogosto težak za razumevanje. Z namenom izvedbe inverzne Laplaceove transformacije je najprej potrebno izvesti razcep na parcialne ulomke. Ta interaktivni primer prikazuje način izvedbe razcepa.

---

### Kako upravljati s tem interaktivnim primerom?
Preklapljaš lahko med opcijama *Vnos funkcije* ali *Vnos koeficientov polinoma*.

1. *Vnos funkcije*:
* Primer: Če želiš vnesti funkcijo $\frac{1}{x^2(x^2 + 1)}$ (faktorizirana oblika) vnesi 1/(x\*\*2\*(x\*\*2 + 1)); če želiš vnesti isto funkcijo a v razširjeni obliki ($\frac{1}{x^4+x^2}$) type 1/(x\*\*4+x\*\*2).

2. *Vnos koeficientov polinoma*:
* Z uporabo drsnikov izberi stopnji števca in imenovalca izbrane racionalne funkcije.
* Vnesi vrednost koeficientov števa in imenovalca v ustrezna besedilna polja; za potrditev klikni na gumb *Potrdi*.

```python
## System selector buttons
style = {'description_width': 'initial'}
typeSelect = widgets.ToggleButtons(
    options=[('Vnos funkcije', 0), ('Vnos koeficientov polinoma', 1),],
    description='Izberi: ',style={'button_width':'230px'})

btnReset=widgets.Button(description="Ponastavi")

# function
textbox=widgets.Text(description=('Vnesi funkcijo:'),style=style)
btnConfirmFunc=widgets.Button(description="Potrdi") # ex btnConfirm

# poly
btnConfirmPoly=widgets.Button(description="Potrdi") # ex btn

display(typeSelect)

def on_button_clickedReset(ev):
    display(Javascript("Jupyter.notebook.execute_cells_below()"))

def on_button_clickedFunc(ev):
    eq = sym.sympify(textbox.value)

    if eq==sym.factor(eq):
        display(Markdown('Vnešena funkcija $%s$ je zapisana v faktorizirani obliki. ' %sym.latex(eq) + 'Njena razširjena oblika je enaka $%s$.' %sym.latex(sym.expand(eq))))

    else:
        display(Markdown('Vnešena funkcija $%s$ je zapisana v razširjeni obliki. ' %sym.latex(eq) + 'Njena faktorizirana oblika je enaka $%s$.' %sym.latex(sym.factor(eq))))

    display(Markdown('Rezultat razcepa na parcialne ulomke: $%s$' %sym.latex(sym.apart(eq)) + '.'))
    display(btnReset)

def transfer_function(num,denom):
    num = np.array(num, dtype=np.float64)
    denom = np.array(denom, dtype=np.float64)
    len_dif = len(denom) - len(num)
    if len_dif<0:
        temp = np.zeros(abs(len_dif))
        denom = np.concatenate((temp, denom))
        transferf = np.vstack((num, denom))
    elif len_dif>0:
        temp = np.zeros(len_dif)
        num = np.concatenate((temp, num))
        transferf = np.vstack((num, denom))
    return transferf

def f(orderNum, orderDenom):
    global text1, text2
    text1=[None]*(int(orderNum)+1)
    text2=[None]*(int(orderDenom)+1)
    display(Markdown('2. Vnesi koeficiente polinoma v števcu.'))
    for i in range(orderNum+1):
        text1[i]=widgets.Text(description=(r'a%i'%(orderNum-i)))
        display(text1[i])
    display(Markdown('3. Vnesi koeficiente polinoma v imenovalcu.'))
    for j in range(orderDenom+1):
        text2[j]=widgets.Text(description=(r'b%i'%(orderDenom-j)))
        display(text2[j])
    global orderNum1, orderDenom1
    orderNum1=orderNum
    orderDenom1=orderDenom

def on_button_clickedPoly(btn):
    clear_output()
    global num,denom
    enacbaNum=""
    enacbaDenom=""
    num=[None]*(int(orderNum1)+1)
    denom=[None]*(int(orderDenom1)+1)
    for i in range(int(orderNum1)+1):
        if text1[i].value=='' or text1[i].value=='Vnesi koeficient':
            text1[i].value='Vnesi koeficient'
        else:
            try:
                num[i]=int(text1[i].value)
            except ValueError:
                if text1[i].value!='' or text1[i].value!='Vnesi koeficient':
                    num[i]=sym.var(text1[i].value)

    for i in range (len(num)-1,-1,-1):
        if i==0:
            enacbaNum=enacbaNum+str(num[len(num)-i-1])
        elif i==1:
            enacbaNum=enacbaNum+"+"+str(num[len(num)-i-1])+"*x+"
        elif i==int(len(num)-1):
            enacbaNum=enacbaNum+str(num[0])+"*x**"+str(len(num)-1)
        else:
            enacbaNum=enacbaNum+"+"+str(num[len(num)-i-1])+"*x**"+str(i)

    for j in range(int(orderDenom1)+1):
        if text2[j].value=='' or text2[j].value=='Vnesi koeficient':
            text2[j].value='Vnesi koeficient'
        else:
            try:
                denom[j]=int(text2[j].value)
            except ValueError:
                if text2[j].value!='' or text2[j].value!='Vnesi koeficient':
                    denom[j]=sym.var(text2[j].value)

    for i in range (len(denom)-1,-1,-1):
        if i==0:
            enacbaDenom=enacbaDenom+"+"+str(denom[len(denom)-i-1])
        elif i==1:
            enacbaDenom=enacbaDenom+"+"+str(denom[len(denom)-i-1])+"*x"
        elif i==int(len(denom)-1):
            enacbaDenom=enacbaDenom+str(denom[0])+"*x**"+str(len(denom)-1)
        else:
            enacbaDenom=enacbaDenom+"+"+str(denom[len(denom)-i-1])+"*x**"+str(i)

    funcSym=sym.sympify('('+enacbaNum+')/('+enacbaDenom+')')

    DenomSym=sym.sympify(enacbaDenom)
    NumSym=sym.sympify(enacbaNum)
    DenomSymFact=sym.factor(DenomSym);
    funcFactSym=NumSym/DenomSymFact;

    if DenomSym==sym.expand(enacbaDenom):
        if DenomSym==DenomSymFact:
            display(Markdown('Vnešena funkcija je enaka $%s$. Števca ni moč razcepiti.' %sym.latex(funcSym)))
        else:
            display(Markdown('Vnešena funkcija je enaka $%s$. Števca ni moč razcepiti. Isto funkcijo lahko zapišemo v faktorizirani obliki kot $%s$.' %(sym.latex(funcSym), sym.latex(funcFactSym))))

    if sym.apart(funcSym)==funcSym:
        display(Markdown('Razcepa na parcialne ulomke ni možno izvesti.'))
    else:
        display(Markdown('Rezultat razcepa na parcialne ulomke je enak $%s$' %sym.latex(sym.apart(funcSym)) + '.'))

    btnReset.on_click(on_button_clickedReset)
    display(btnReset)

def partial_frac(index):

    if index==0:
        x = sym.Symbol('x')
        display(widgets.HBox((textbox, btnConfirmFunc)))
        btnConfirmFunc.on_click(on_button_clickedFunc)
        btnReset.on_click(on_button_clickedReset)

    elif index==1:
        display(Markdown('1. Določi stopnji polinomov v števcu (orderNum) in imenovalcu (orderDenom).'))
        widgets.interact(f, orderNum=widgets.IntSlider(min=0,max=10,step=1,value=0),
                 orderDenom=widgets.IntSlider(min=0,max=10,step=1,value=0));
        btnConfirmPoly.on_click(on_button_clickedPoly)
        display(btnConfirmPoly)

input_data=widgets.interactive_output(partial_frac,{'index':typeSelect})
display(input_data)
```

    ToggleButtons(description='Izberi: ', options=(('Vnos funkcije', 0), ('Vnos koeficientov polinoma', 1)), style…

    Output()


# Example of DOV search methods for CPT measurements (sonderingen)

[](https://mybinder.org/v2/gh/DOV-Vlaanderen/pydov/master?filepath=docs%2Fnotebooks%2Fsearch_sonderingen.ipynb)

## Use cases explained below
* Get CPT measurements in a bounding box
* Get CPT measurements with specific properties
* Get CPT measurements in a bounding box based on specific properties
* Select CPT measurements in a municipality and return depth
* Get CPT measurements based on fields not available in the standard output dataframe
* Get CPT measurements data, returning fields not available in the standard output dataframe
* Get CPT measurements in a municipality and where groundwater related data are available

```python
%matplotlib inline
import inspect, sys
```

```python
import pydov
```

## Get information about the datatype 'Sondering'

```python
from pydov.search.sondering import SonderingSearch
sondering = SonderingSearch()
```

A description is provided for the 'Sondering' datatype:

```python
sondering.get_description()
```

    'In DOV worden de resultaten van sonderingen ter beschikking gesteld. Bij het uitvoeren van de sondering wordt een sondeerpunt met conus bij middel van buizen statisch de grond ingedrukt. Continu of met bepaalde diepte-intervallen wordt de weerstand aan de conuspunt, de plaatselijke wrijvingsweerstand en/of de totale indringingsweerstand opgemeten. Eventueel kan aanvullend de waterspanning in de grond rond de conus tijdens de sondering worden opgemeten met een waterspanningsmeter. Het op diepte drukken van de sondeerbuizen gebeurt met een indrukapparaat. De nodige reactie voor het indrukken van de buizen wordt geleverd door een verankering en/of door het gewicht van de sondeerwagen. De totale indrukcapaciteit varieert van 25 kN tot 250 kN, afhankelijk van apparaat en opstellingswijze.'

The different fields that are available for objects of the 'Sondering' datatype can be requested with the get_fields() method:

```python
fields = sondering.get_fields()

# print available fields
for f in fields.values():
    print(f['name'])
```

    id
    sondeernummer
    pkey_sondering
    weerstandsdiagram
    meetreeks
    x
    y
    start_sondering_mtaw
    gemeente
    diepte_sondering_van
    diepte_sondering_tot
    datum_aanvang
    uitvoerder
    conus
    sondeermethode
    apparaat
    informele_stratigrafie
    formele_stratigrafie
    hydrogeologische_stratigrafie
    opdrachten
    datum_gw_meting
    diepte_gw_m
    lengte
    diepte
    qc
    Qt
    fs
    u
    i

You can get more information of a field by requesting it from the fields dictionary:
* *name*: name of the field
* *definition*: definition of this field
* *cost*: currently this is either 1 or 10, depending on the datasource of the field. It is an indication of the expected time it will take to retrieve this field in the output dataframe.
* *notnull*: whether the field is mandatory or not
* *type*: datatype of the values of this field

```python
fields['diepte_sondering_tot']
```

    {'name': 'diepte_sondering_tot',
     'definition': 'Maximumdiepte van de sondering ten opzichte van het aanvangspeil, in meter.',
     'type': 'float',
     'notnull': False,
     'query': True,
     'cost': 1}

Optionally, if the values of the field have a specific domain the possible values are listed as *values*:

```python
fields['conus']['values']
```

    {'E': None, 'M1': None, 'M2': None, 'M4': None, 'U': None, 'onbekend': None}

## Example use cases

### Get CPT measurements in a bounding box

Get data for all the CPT measurements that are geographically located within the bounds of the specified box.

The coordinates are in the Belgian Lambert72 (EPSG:31370) coordinate system and are given in the order of lower left x, lower left y, upper right x, upper right y.

```python
from pydov.util.location import Within, Box

df = sondering.search(location=Within(Box(152999, 206930, 153050, 207935)))
df.head()
```

    [000/001] c

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      lengte
      diepte
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100 kN
      NaN
      NaN
      0.2
      NaN
      1.6
      2.06
      NaN
      NaN
      NaN

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100 kN
      NaN
      NaN
      0.4
      NaN
      3.6
      4.26
      NaN
      NaN
      NaN

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100 kN
      NaN
      NaN
      0.6
      NaN
      2.6
      3.46
      NaN
      NaN
      NaN

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100 kN
      NaN
      NaN
      0.8
      NaN
      4.0
      5.66
      NaN
      NaN
      NaN

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100 kN
      NaN
      NaN
      1.0
      NaN
      3.0
      6.53
      NaN
      NaN
      NaN

The dataframe contains one CPT measurement where multiple measurement points. The available data are flattened to represent unique attributes per row of the dataframe.

Using the *pkey_sondering* field one can request the details of this borehole in a webbrowser:

```python
for pkey_sondering in set(df.pkey_sondering):
    print(pkey_sondering)
```

    https://www.dov.vlaanderen.be/data/sondering/1973-016812

### Get CPT measurements with specific properties

Next to querying CPT based on their geographic location within a bounding box, we can also search for CPT measurements matching a specific set of properties. For this we can build a query using a combination of the 'Sondering' fields and operators provided by the WFS protocol.

A list of possible operators can be found below:

```python
[i for i,j in inspect.getmembers(sys.modules['owslib.fes'], inspect.isclass) if 'Property' in i]
```

    ['PropertyIsBetween',
     'PropertyIsEqualTo',
     'PropertyIsGreaterThan',
     'PropertyIsGreaterThanOrEqualTo',
     'PropertyIsLessThan',
     'PropertyIsLessThanOrEqualTo',
     'PropertyIsLike',
     'PropertyIsNotEqualTo',
     'PropertyIsNull',
     'SortProperty']

In this example we build a query using the *PropertyIsEqualTo* operator to find all CPT measuremetns that are within the community (gemeente) of 'Herstappe':

```python
from owslib.fes import PropertyIsEqualTo

query = PropertyIsEqualTo(propertyname='gemeente',
                          literal='Elsene')
df = sondering.search(query=query)

df.head()
```

    [000/029] ccccccccccccccccccccccccccccc

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      lengte
      diepte
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25 kN
      NaN
      1.97
      1.0
      NaN
      3.3
      NaN
      NaN
      NaN
      NaN

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25 kN
      NaN
      1.97
      1.1
      NaN
      2.9
      NaN
      NaN
      NaN
      NaN

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25 kN
      NaN
      1.97
      1.2
      NaN
      2.7
      NaN
      NaN
      NaN
      NaN

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25 kN
      NaN
      1.97
      1.3
      NaN
      2.4
      NaN
      NaN
      NaN
      NaN

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25 kN
      NaN
      1.97
      1.4
      NaN
      3.6
      NaN
      NaN
      NaN
      NaN

Once again we can use the *pkey_sondering* as a permanent link to the information of these CPT measurements:

```python
for pkey_sondering in set(df.pkey_sondering):
    print(pkey_sondering)
```

    https://www.dov.vlaanderen.be/data/sondering/1980-024719
    https://www.dov.vlaanderen.be/data/sondering/1980-024720
    https://www.dov.vlaanderen.be/data/sondering/1971-022776
    https://www.dov.vlaanderen.be/data/sondering/1971-023322
    https://www.dov.vlaanderen.be/data/sondering/1976-030128
    https://www.dov.vlaanderen.be/data/sondering/1971-023323
    https://www.dov.vlaanderen.be/data/sondering/1976-013899
    https://www.dov.vlaanderen.be/data/sondering/1992-000338
    https://www.dov.vlaanderen.be/data/sondering/1976-013900
    https://www.dov.vlaanderen.be/data/sondering/1971-023091
    https://www.dov.vlaanderen.be/data/sondering/1975-014064
    https://www.dov.vlaanderen.be/data/sondering/1971-022777
    https://www.dov.vlaanderen.be/data/sondering/1992-000339
    https://www.dov.vlaanderen.be/data/sondering/1971-023321
    https://www.dov.vlaanderen.be/data/sondering/1976-030150
    https://www.dov.vlaanderen.be/data/sondering/1992-000336
    https://www.dov.vlaanderen.be/data/sondering/1974-016927
    https://www.dov.vlaanderen.be/data/sondering/1975-014063
    https://www.dov.vlaanderen.be/data/sondering/1971-023319
    https://www.dov.vlaanderen.be/data/sondering/1976-013898
    https://www.dov.vlaanderen.be/data/sondering/1992-000335
    https://www.dov.vlaanderen.be/data/sondering/1971-022775
    https://www.dov.vlaanderen.be/data/sondering/1976-014638
    https://www.dov.vlaanderen.be/data/sondering/1971-023320
    https://www.dov.vlaanderen.be/data/sondering/1974-016926
    https://www.dov.vlaanderen.be/data/sondering/1992-000337
    https://www.dov.vlaanderen.be/data/sondering/1976-014640
    https://www.dov.vlaanderen.be/data/sondering/1976-030148
    https://www.dov.vlaanderen.be/data/sondering/1976-030140

### Get CPT measurements in a bounding box based on specific properties

We can combine a query on attributes with a query on geographic location to get the CPT measurements within a bounding box that have specific properties.

The following example requests the CPT measurements with a depth greater than or equal to 2000 meters within the given bounding box.

(Note that the datatype of the *literal* parameter should be a string, regardless of the datatype of this field in the output dataframe.)

```python
from owslib.fes import PropertyIsGreaterThanOrEqualTo

query = PropertyIsGreaterThanOrEqualTo(
            propertyname='diepte_sondering_tot',
            literal='20')

df = sondering.search(
    location=Within(Box(200000, 211000, 205000, 214000)),
    query=query
    )

df.head()
```

    [000/021] ccccccccccccccccccccc

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      lengte
      diepte
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200 kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.30
      1.30
      1.22
      NaN
      1.0
      NaN
      0.8

      1
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200 kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.35
      1.35
      3.19
      NaN
      2.0
      NaN
      1.0

      2
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200 kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.40
      1.40
      7.21
      NaN
      63.0
      NaN
      1.2

      3
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200 kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.45
      1.45
      12.75
      NaN
      138.0
      NaN
      1.2

      4
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200 kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.50
      1.50
      15.26
      NaN
      143.0
      NaN
      1.4

We can look at one of the CPT measurements in a webbrowser using its *pkey_sondering*:

```python
for pkey_sondering in set(df.pkey_sondering):
    print(pkey_sondering)
```

    https://www.dov.vlaanderen.be/data/sondering/2010-062407
    https://www.dov.vlaanderen.be/data/sondering/2007-049200
    https://www.dov.vlaanderen.be/data/sondering/2008-077556
    https://www.dov.vlaanderen.be/data/sondering/2015-054999
    https://www.dov.vlaanderen.be/data/sondering/2008-077592
    https://www.dov.vlaanderen.be/data/sondering/2009-000054
    https://www.dov.vlaanderen.be/data/sondering/2008-077565
    https://www.dov.vlaanderen.be/data/sondering/2008-077579
    https://www.dov.vlaanderen.be/data/sondering/2015-055496
    https://www.dov.vlaanderen.be/data/sondering/2009-000052
    https://www.dov.vlaanderen.be/data/sondering/2008-077566
    https://www.dov.vlaanderen.be/data/sondering/2008-077581
    https://www.dov.vlaanderen.be/data/sondering/2008-077564
    https://www.dov.vlaanderen.be/data/sondering/2008-077557
    https://www.dov.vlaanderen.be/data/sondering/2008-077580
    https://www.dov.vlaanderen.be/data/sondering/2015-054995
    https://www.dov.vlaanderen.be/data/sondering/2007-049201
    https://www.dov.vlaanderen.be/data/sondering/2008-077577
    https://www.dov.vlaanderen.be/data/sondering/2008-077545
    https://www.dov.vlaanderen.be/data/sondering/2008-077591
    https://www.dov.vlaanderen.be/data/sondering/2009-000053

### Select CPT measurements in a municipality and return depth

We can limit the columns in the output dataframe by specifying the *return_fields* parameter in our search.

In this example we query all the CPT measurements in the city of Ghent and return their depth:

```python
query = PropertyIsEqualTo(propertyname='gemeente',
                          literal='Gent')
df = sondering.search(query=query,
                      return_fields=('diepte_sondering_tot',))
df.head()
```

      diepte_sondering_tot

      0
      2.7

      1
      1.4

      2
      7.6

      3
      11.5

      4
      18.6

```python
df.describe()
```

      diepte_sondering_tot

      count
      3628.000000

      mean
      18.562031

      std
      8.481631

      min
      1.000000

      25%
      11.400000

      50%
      18.800000

      75%
      24.600000

      max
      52.600000

```python
ax = df.boxplot()
ax.set_title('Distribution depth CPT measurements in Ghent');
ax.set_ylabel("depth (m)")
```

### Get CPT measurements based on fields not available in the standard output dataframe

To keep the output dataframe size acceptable, not all available WFS fields are included in the standard output. However, one can use this information to select CPT measurements as illustrated below.

For example, make a selection of the CPT measurements in municipality the of Antwerp, using a conustype 'U':

```python
from owslib.fes import And

query = And([PropertyIsEqualTo(propertyname='gemeente',
                               literal='Antwerpen'),
             PropertyIsEqualTo(propertyname='conus',
                               literal='U')]
            )
df = sondering.search(query=query,
                      return_fields=('pkey_sondering', 'sondeernummer', 'x', 'y', 'diepte_sondering_tot', 'datum_aanvang'))
df.head()
```

      pkey_sondering
      sondeernummer
      x
      y
      diepte_sondering_tot
      datum_aanvang

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      29.70
      1993-03-02

      1
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-02/111-S1
      150347.3
      214036.4
      29.95
      2002-12-17

      2
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-04/123-SKD4-E
      146437.7
      222317.5
      4.45
      2004-07-12

      3
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-04/123-SKD6-E
      146523.9
      222379.7
      7.40
      2004-07-14

      4
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-04/123-SKD5-E
      146493.4
      222298.8
      1.65
      2004-07-16

### Get CPT data, returning fields not available in the standard output dataframe

As denoted in the previous example, not all available fields are available in the default output frame to keep its size limited. However, you can request any available field by including it in the *return_fields* parameter of the search:

```python
query = And([PropertyIsEqualTo(propertyname='gemeente', literal='Gent'),
             PropertyIsEqualTo(propertyname='conus', literal='U')])

df = sondering.search(query=query,
                      return_fields=('pkey_sondering', 'sondeernummer', 'diepte_sondering_tot',
                                     'conus', 'x', 'y'))

df.head()
```

      pkey_sondering
      sondeernummer
      diepte_sondering_tot
      conus
      x
      y

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SV
      33.80
      U
      110241.6
      204692.2

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SI
      15.65
      U
      110062.5
      205051.4

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SII
      26.50
      U
      110107.0
      204965.3

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIII
      16.50
      U
      110152.4
      204876.1

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIV
      16.70
      U
      110197.8
      204787.0

```python
df
```

      pkey_sondering
      sondeernummer
      diepte_sondering_tot
      conus
      x
      y

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SV
      33.80
      U
      110241.6
      204692.2

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SI
      15.65
      U
      110062.5
      205051.4

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SII
      26.50
      U
      110107.0
      204965.3

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIII
      16.50
      U
      110152.4
      204876.1

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIV
      16.70
      U
      110197.8
      204787.0

      5
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIX
      27.60
      U
      110479.5
      205240.7

      6
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SVI
      16.80
      U
      110288.5
      204608.8

      7
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SVII
      26.70
      U
      110334.3
      204519.8

      8
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SX
      27.50
      U
      110685.0
      204845.5

      9
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SXI
      25.60
      U
      109941.5
      204346.9

      10
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SXII
      26.50
      U
      110412.2
      204398.1

      11
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/096-SIX(CPT9)
      17.60
      U
      105018.0
      190472.0

      12
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/096-SVII(CPT7)
      26.05
      U
      105046.0
      190550.0

      13
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/096-SVIII(CPT8)
      24.75
      U
      104997.0
      190521.0

      14
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-97/002-S2
      29.90
      U
      105376.6
      189104.3

      15
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-97/002-S3
      5.90
      U
      105391.3
      189083.7

      16
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-97/002-S1
      30.60
      U
      105399.3
      189065.2

      17
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S1
      18.05
      U
      106104.1
      188699.4

      18
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S2
      17.30
      U
      106045.3
      188708.4

      19
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S3
      18.70
      U
      106100.5
      188743.8

      20
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S5
      17.30
      U
      106130.0
      188712.0

      21
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S4
      17.00
      U
      106077.5
      188686.0

## Resistivity plot

The data for the reporting of resistivity plots with the online application, see for example [this report](https://www.dov.vlaanderen.be/zoeken-ocdov/proxy-sondering/sondering/1993-001275/rapport/identifygrafiek?outputformaat=PDF), is also accessible with the pydov package. Querying the data for this specific _sondering_:

```python
query = PropertyIsEqualTo(propertyname='pkey_sondering',
                          literal='https://www.dov.vlaanderen.be/data/sondering/1993-001275')
df_sond = sondering.search(query=query)

df_sond.head()
```

    [000/001] c

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      lengte
      diepte
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200 kN
      NaN
      NaN
      0.6
      NaN
      11.60
      NaN
      130.0
      69.0
      NaN

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200 kN
      NaN
      NaN
      0.7
      NaN
      6.30
      NaN
      100.0
      29.0
      NaN

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200 kN
      NaN
      NaN
      0.8
      NaN
      6.22
      NaN
      120.0
      -4.0
      NaN

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200 kN
      NaN
      NaN
      0.9
      NaN
      4.92
      NaN
      120.0
      -48.0
      NaN

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200 kN
      NaN
      NaN
      1.0
      NaN
      4.40
      NaN
      80.0
      -35.0
      NaN

We have the depth (`length`) available, together with the measured values for each depth of the variables (in dutch):

* `qc`: Opgemeten waarde van de conusweerstand, uitgedrukt in MPa.
* `Qt`: Opgemeten waarde van de totale weerstand, uitgedrukt in kN.
* `fs`: Opgemeten waarde van de plaatelijke kleefweerstand uitgedrukt in kPa.
* `u`: Opgemeten waarde van de porienwaterspanning, uitgedrukt in kPa.
* `i`: Opgemeten waarde van de inclinatie, uitgedrukt in graden.

To recreate the resistivity plot, we also need the `resistivity number` (wrijvingsgetal `rf`), see [DOV documentation](https://www.dov.vlaanderen.be/page/sonderingen).

\begin{equation}
R_f = \frac{f_s}{q_c}
\end{equation}

**Notice:** $f_s$ is provide in kPa and $q_c$ in MPa.

Adding `rf` to the dataframe:

```python
df_sond["rf"] = df_sond["fs"]/df_sond["qc"]/10
```

Recreate the resistivity plot:

```python
import matplotlib.pyplot as plt
```

```python
def make_patch_spines_invisible(ax):
    ax.set_frame_on(True)
    ax.patch.set_visible(False)
    for sp in ax.spines.values():
        sp.set_visible(False)
```

```python
# Determine Lengte or Depth
# If diepte is available, the y-axis will be Diepte
# Else the y-axis will be Lengte
if df_sond['diepte'].isnull().values.any():
    # IsNan
    y_type = "lengte"
    y_axis = "Length (m)"
else:
    y_type = "diepte"
    y_axis = "Depth (m)"

fig, ax0 = plt.subplots(figsize=(8, 12))

# Prepare the individual axis
ax_qc = ax0.twiny()
ax_fs = ax0.twiny()
ax_u = ax0.twiny()
ax_rf = ax0.twiny()

for i, ax in enumerate([ax_qc, ax_fs, ax_u]):
    ax.spines["top"].set_position(("axes", 1+0.05*(i+1)))
    make_patch_spines_invisible(ax)
    ax.spines["top"].set_visible(True)

# Plot the data on the axis
df_sond.plot(x="rf", y=y_type, label="rf", ax=ax_rf, color='purple', legend=False)
df_sond.plot(x="qc", y=y_type, label="qc (MPa)", ax=ax_qc, color='black', legend=False)
df_sond.plot(x="fs", y=y_type, label="fs (kPa)", ax=ax_fs, color='green', legend=False)
df_sond.plot(x="u", y=y_type, label="u (kPa)", ax=ax_u, color='red',
        legend=False, xlim=(-100, 300)) # ! 300 is hardocded here for the example

# styling and configuration
ax_rf.xaxis.label.set_color('purple')
ax_fs.xaxis.label.set_color('green')
ax_u.xaxis.label.set_color('red')

ax0.axes.set_visible(False)
ax_qc.axes.yaxis.set_visible(False)
ax_fs.axes.yaxis.set_visible(False)
for i, ax in enumerate([ax_rf, ax_qc, ax_fs, ax_u, ax0]):
    ax.spines["right"].set_visible(False)
    ax.spines["bottom"].set_visible(False)
    ax.xaxis.label.set_fontsize(15)
    ax.xaxis.set_label_coords(-0.05, 1+0.05*i)
    ax.spines['left'].set_position(('outward', 10))
    ax.spines['left'].set_bounds(0, 30)
ax_rf.set_xlim(0, 46)

ax_u.set_title("Resistivity plot CPT measurement GEO-93/023-SII-E", fontsize=12)

ax0.invert_yaxis()
ax_rf.invert_xaxis()
ax_u.set_ylabel(y_axis, fontsize=12)
fig.legend(loc='lower center', ncol=4)
fig.tight_layout()
```

## Visualize locations

Using Folium, we can display the results of our search on a map.

```python
# import the necessary modules (not included in the requirements of pydov!)
import folium
from folium.plugins import MarkerCluster
from pyproj import Transformer
```

```python
# convert the coordinates to lat/lon for folium
def convert_latlon(x1, y1):
    transformer = Transformer.from_crs("epsg:31370", "epsg:4326", always_xy=True)
    x2,y2 = transformer.transform(x1, y1)
    return x2, y2

df['lon'], df['lat'] = zip(*map(convert_latlon, df['x'], df['y']))
# convert to list
loclist = df[['lat', 'lon']].values.tolist()
```

```python
# initialize the Folium map on the centre of the selected locations, play with the zoom until ok
fmap = folium.Map(location=[df['lat'].mean(), df['lon'].mean()], zoom_start=11)
marker_cluster = MarkerCluster().add_to(fmap)
for loc in range(0, len(loclist)):
    folium.Marker(loclist[loc], popup=df['sondeernummer'][loc]).add_to(marker_cluster)
fmap

```

```python

```