# Exercise 5 - Classification Algorithms (BPNN & SVM)

## a) Back Propogation Neural Network

### Program 1 - Implementing BPNN from scratch

#### AIM
To implement Back Propogation Neural Network from scratch in python.

#### ALGORITHM
##### Forward propogation
###### Single input
$$
\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}}
\newcommand{\matrix}[1]{\begin{bmatrix}{#1}\end{bmatrix}}
\begin{align*}
\vect{o}_{-1} &= \vect{x}_i\\
\vect{o}_k &= \vect{\phi}_k\left(
\vect{W}_k
\matrix{ 1 \\ \vect{o}_{k-1}}
\right) &\text{for }k = 0,..,n-1
\end{align*}
$$

###### Batch input
$$
\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}}
\begin{align*}
\vect{O}_{-1} &= \vect{X}^T\\
\vect{O}_k &= \vect{\phi}_k\left(
\vect{W}_k^T
\matrix{ \vect{1} \\ \vect{O}_{k-1}}
\right) &\text{for }k = 0,..,n-1
\end{align*}
$$

##### Error
###### Single Input
$$
E = \frac{1}{2}||\vect{o}_{n-1}-\vect{t}||_2
$$
###### Batch Input
$$
E = \frac{1}{2m} \sum_{i=1}^{m} ||\vect{o}^{(i)}_{n-1}-\vect{t}||_2
$$

##### Backward Propogation

###### Single input
$$
\newcommand{\diff}[2]{\frac{\mathrm{d}{#1}}{\mathrm{d}{#2}}}
\begin{align}
\vect{\delta}_{k} &= \diff{E}{\vect{o}_{k}}\circ \vect{\phi}'(\text{net}_{k} )\\
\diff{E}{\vect{o}_{k}} &= \begin{cases}
(\vect{o}_{n-1}-\vect{t}) & k=n-1\\
\vect{W}^T_{k+1}\vect{\delta}_{k+1} & k = n-2,...,-1 \\
\end{cases}\\
\vect{\phi}'(\text{net}_{k}) &= \vect{o}_k \circ (\vect{1}-\vect{o}_k)  \text{ for }   k = n-1,...,0 \\
\end{align}
$$

###### Batch input
$$
\newcommand{\diff}[2]{\frac{\mathrm{d}{#1}}{\mathrm{d}{#2}}}
\begin{align}
\vect{\Delta}_{k} &= \diff{E}{\vect{O}_{k}}\circ \vect{\phi}'(\text{net}_{k} )\\
\diff{E}{\vect{O}_{k}} &= \begin{cases}
(\vect{O}_{n-1}-\vect{T}) & k=n-1\\
\vect{W}_{k+1}^T \vect{\Delta}_{k+1} & k = n-2,...,-1 \\
\end{cases}\\
\vect{\phi}'(\text{net}_{k}) &= \vect{O}_k \circ (\vect{1}-\vect{O}_k)  \text{ for }   k = n-1,...,0 \\
\end{align}
$$

##### Weight Update
###### Single Input
$$
\vect{W}_k := \vect{W}_k - \alpha \diff{E}{\vect{W}_k}\\
\diff{E}{\vect{W}_k} = \vect{\delta}_{k}
\matrix{1 \\ \vect{o}_{k-1}}^T\ \text{for } k = n-1,...,0 \\
$$

###### Batch Input
$$
\vect{W}_k := \vect{W}_k - \alpha \diff{E}{\vect{W}_k}\\
\diff{E}{\vect{W}_k} = \frac{1}{m}\vect{\Delta}_{k}
\matrix{1 \\ \vect{O}_{k-1}}^T \text{for } k = n-1,...,0 \\
$$

where
$\vect{x}_i$ - the input vector of the  $i$th data point
$\vect{o}_k$ - the output vector of the $k$th layer
$\vect{\delta}_k$ - the delta vector of the $k$th layer
$\vect{W}_k$ - the weight matrix of the $k$th layer
$\vect{X}$ - the batch input matfix
$\vect{O}_k$ - the batch output matrix of the $k$th layer
$\vect{\Delta}_k$ - the batch delta matrix of the $k$th layer

#### Part 1 - Defining class for BPNN

```python
import numpy as np
import pandas as pd
```

```python
class BPNN:
    def __init__(
        self,hidden_layer_sizes,
        l_rate=.001,
        n_epoch=20,
        batch_size=20,
        random_state=0
    ):
        self.hidden_layer_sizes = hidden_layer_sizes
        self.l_rate= float(l_rate)
        self.n_epoch = int(n_epoch)
        self.batch_size = int(batch_size)
        self.random_state = random_state

        self.n = len(hidden_layer_sizes)+1
        self.outputs = [None]*(self.n+1)
        self.delta = [None]*self.n

    def activation(self,x):
        return 1/(1+np.exp(-x))

    def d_activation(self,x):
         return x * (1 - x)

    def pad_ones(X):
        pad_width = [(1,0),(0,0)]
        return np.pad(X,pad_width=pad_width,constant_values=1)

    def inputs(self,i):
        return BPNN.pad_ones(self.outputs[i-1])

    def initialize_random_weights(self):
        np.random.seed(self.random_state)
        self.weights = [
            np.random.standard_normal((o,i+1))*.001
            for i,o in zip(
                [self.n_inputs]+self.hidden_layer_sizes,
                self.hidden_layer_sizes+[self.n_outputs]
            )
        ]

    def forward_propagate(self, inputs):
        assert (self.weights is not None),"weights not given"
        self.outputs[-1] = inputs
        for i in range(self.n):
            self.outputs[i] = self.activation(
                self.weights[i] @ self.inputs(i)
            )
        return self.outputs[self.n-1]

    def backward_propagate_error(self, target):
        for i in range(self.n-1, -1, -1):
            if i == self.n-1:
                errors = self.outputs[i] - target
            else:
                errors = self.weights[i+1][:,1:].T @ self.delta[i+1]
            self.delta[i] = errors * self.d_activation(self.outputs[i])

    def update_weights(self):
        for i in range(self.n):
            self.weights[i] -= self.l_rate * self.delta[i] @ self.inputs(i).T/self.batch_size

    def error(self,actual,predicted):
        error = actual-predicted
        return np.sum(error * error)

    def fit(self, X_train,y_train,verbose=False):
        m = X_train.shape[0]
        input_matrix = X_train.T
        classes = np.unique(y_train)
        target_matrix = (y_train.reshape(-1,1) == classes).T
        self.n_inputs = input_matrix.shape[0]
        self.n_outputs = target_matrix.shape[0]
        self.initialize_random_weights()
        if verbose:
            print("Initial Weights:")
            print(*self.weights,sep="\n")
            print("Training:")

        for epoch in range(self.n_epoch):
            sum_error = 0
            fs = range(m+self.batch_size)
            for f,t in zip(fs,fs[1:]):
                outputs = self.forward_propagate(input_matrix[:,f:t])
                sum_error += self.error(target_matrix[:,f:t],outputs)
                self.backward_propagate_error(target_matrix[:,f:t])
                self.update_weights()
            if verbose:
                print(f'> epoch={epoch+1}, lrate={self.l_rate:.3}, error={sum_error/m:.5f}')
        if verbose:
            print("Trained Weights:")
            print(*self.weights,sep="\n")

    def predict(self, inputs):
        outputs = self.forward_propagate(inputs.T).T.argmax(axis=-1)
        return outputs

    def accuracy(self,X_test,y_test):
        return (self.predict(X_test)==y_test).mean()
```

#### Part 2 - Loading and processing dataset

```python
titanic_df = pd.read_csv("datasets/titanic_processed.csv")
X = titanic_df.drop('Survived',axis = 1).values
y = titanic_df['Survived'].values
split_ratio = 0.8
s = int(split_ratio*X.shape[0])
X_train, X_test = X[:s],X[s:]
y_train, y_test = y[:s],y[s:]
```

#### Part 3 - Implementing BPNN

```python
b = BPNN(
    hidden_layer_sizes = [5,3],
    l_rate=30, n_epoch=20, batch_size=25,random_state=10
)
b.fit(X_train,y_train,verbose=True)
print("Evaluation:")
print(f"Acuracy of the classifier: {b.accuracy(X_test,y_test)*100:.2f}%")
```

    Initial Weights:
    [[ 1.33158650e-03  7.15278974e-04 -1.54540029e-03 -8.38384993e-06
       6.21335974e-04 -7.20085561e-04  2.65511586e-04  1.08548526e-04
       4.29143093e-06 -1.74600211e-04]
     [ 4.33026190e-04  1.20303737e-03 -9.65065671e-04  1.02827408e-03
       2.28630130e-04  4.45137613e-04 -1.13660221e-03  1.35136878e-04
       1.48453700e-03 -1.07980489e-03]
     [-1.97772828e-03 -1.74337230e-03  2.66070164e-04  2.38496733e-03
       1.12369125e-03  1.67262221e-03  9.91492158e-05  1.39799638e-03
      -2.71247988e-04  6.13204185e-04]
     [-2.67317189e-04 -5.49309014e-04  1.32708296e-04 -4.76142015e-04
       1.30847308e-03  1.95013279e-04  4.00209988e-04 -3.37632337e-04
       1.25647226e-03 -7.31969502e-04]
     [ 6.60231551e-04 -3.50871891e-04 -9.39433360e-04 -4.89337217e-04
      -8.04591142e-04 -2.12697639e-04 -3.39140246e-04  3.12169936e-04
       5.65152670e-04 -1.47420258e-04]]
    [[-2.59053368e-05  2.89094204e-04 -5.39879071e-04  7.08160020e-04
       8.42224738e-04  2.03580797e-04]
     [ 2.39470366e-03  9.17458938e-04 -1.12272471e-04 -3.62180447e-04
      -2.32182256e-04 -5.01728900e-04]
     [ 1.12878515e-03 -6.97810030e-04 -8.11221838e-05 -5.29296081e-04
       1.04618286e-03 -1.41855603e-03]]
    [[-0.0003625  -0.00012191  0.00031936  0.0004609 ]
     [-0.00021579  0.00098907  0.00031475  0.00246765]]
    Training:
    > epoch=1, lrate=30.0, error=0.49515
    > epoch=2, lrate=30.0, error=0.49066
    > epoch=3, lrate=30.0, error=0.44759
    > epoch=4, lrate=30.0, error=0.32836
    > epoch=5, lrate=30.0, error=0.32012
    > epoch=6, lrate=30.0, error=0.31767
    > epoch=7, lrate=30.0, error=0.31607
    > epoch=8, lrate=30.0, error=0.31484
    > epoch=9, lrate=30.0, error=0.31382
    > epoch=10, lrate=30.0, error=0.31292
    > epoch=11, lrate=30.0, error=0.31207
    > epoch=12, lrate=30.0, error=0.31125
    > epoch=13, lrate=30.0, error=0.31043
    > epoch=14, lrate=30.0, error=0.30963
    > epoch=15, lrate=30.0, error=0.30886
    > epoch=16, lrate=30.0, error=0.30816
    > epoch=17, lrate=30.0, error=0.30752
    > epoch=18, lrate=30.0, error=0.30695
    > epoch=19, lrate=30.0, error=0.30645
    > epoch=20, lrate=30.0, error=0.30598
    Trained Weights:
    [[ 2.4976328  -3.27137878 -1.35600823  0.52673885  0.52304233 -3.19704645
       0.39883285 -0.40978078 -0.51246783 -3.54894486]
     [ 2.49858236 -3.27054553 -1.35717467  0.52721097  0.5224186  -3.19489909
       0.39699621 -0.40938148 -0.51440629 -3.54895058]
     [ 2.50036793 -3.27256362 -1.36248268  0.52726938  0.51917641 -3.17240322
       0.39203713 -0.40863187 -0.52741317 -3.5374918 ]
     [ 2.49684424 -3.26991156 -1.35703243  0.52480005  0.52215121 -3.1869823
       0.39638606 -0.41104905 -0.51872859 -3.54117576]
     [ 2.49974609 -3.27394878 -1.35582883  0.52807662  0.52134853 -3.19668184
       0.39856944 -0.4084107  -0.51188295 -3.55207963]]
    [[ 0.97395648 -2.1637305  -2.16450945 -2.15728669 -2.15967589 -2.16477422]
     [ 1.70663229 -2.05598908 -2.05676708 -2.04797014 -2.05198676 -2.05873351]
     [ 1.40632024 -2.09847304 -2.09768519 -2.09035198 -2.09225785 -2.1003679 ]]
    [[-2.60724793  1.55437491  1.68003918  1.61398662]
     [ 2.60724975 -1.55498627 -1.68088901 -1.61254007]]
    Evaluation:
    Acuracy of the classifier: 85.96%

## b) Support Vector Machine

### Program 1 - Implementing SVM from scratch using CVXOPT

#### AIM
To implement SVM from scratch using CVXOPT.

#### Formula
##### SVM statement in dual form
$$
\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}}
\begin{aligned}
    & \min_{\vect{\alpha}}  \frac{1}{2}  \vect{\alpha}^T  \left( \vect{y}\vect{y}^T \circ (\Phi(\vect{X})\Phi(\vect{X})^T \right)  \vect{\alpha} - \vect{1}^T \vect{\alpha}\\
s.t.&  - \alpha_i \leq 0 \\
    & \alpha_i \leq C\\
    & y^T \vect{\alpha} = 0
\end{aligned}
$$

##### Standard form of a quadratic program
$$
\begin{aligned}
    & \min_{\vect{x}}  \frac{1}{2}  \vect{x}^T  \vect{P}  \vect{x} - \vect{q}^T\vect{x}\\
s.t.&  \vect{G}\vect{x} \leq \vect{h} \\
    & A \vect{x} = \vect{b}
\end{aligned}
$$

##### Formulization
$$
\begin{align*}
\vect{P}&= \vect{y}\vect{y}^T \circ \Phi(\vect{X})\Phi(\vect{X})^T
& \vect{q}&= - \vect{1}_n\\
\vect{G}&= \begin{bmatrix}
    \vect{-I}_n\\
    \vect{I}_n
\end{bmatrix}
& \vect{h}&=\begin{bmatrix}
    C \cdot \vect{1}_n^T\\
    \vect{0}_n^T
\end{bmatrix}\\
\vect{A} &= [\vect{y}]
&\vect{b} &= [0]\\
\end{align*}
$$

##### Decision Rule
$$
\vect{\hat y} = \mathrm{sign}\left( (\vect{\alpha}^T \circ \vect{y}^T) \Phi(\vect{X})\Phi(\vect{X})^T  + b \right)
$$

#### Part 1 - Defining class for SVM

```python
import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

from cvxopt import solvers
from cvxopt import matrix

from scipy.spatial.distance import cdist
```

```python
class SVM:
    def __init__(self,C,kernel):
        self.C = C
        self.kernel = kernel

    def fit(self,X_train,y_train):
        self.scaler = StandardScaler()
        self.X = self.scaler.fit_transform(X_train)
        self.y = y_train.reshape(-1,1)

        n=self.X.shape[0]
        I_n = np.eye(n)
        P=(self.y@self.y.T)*self.kernel(self.X,self.X)
        q=np.full(n,-1)
        G=np.vstack((I_n,-1*I_n))
        h=np.hstack((np.full(n,self.C),np.zeros(n)))
        A=y_train.reshape(1,-1)
        b=np.zeros(1)

        P,q,G,h,A,b = map(lambda x : matrix(x,tc="d"),(P,q,G,h,A,b))

        solution = solvers.qp(P, q, G, h, A, b)
        self.a = np.asarray(solution['x']).squeeze()

        support_indices = np.logical_and(self.a>=1e-10, self.a<self.C)
        X_S = self.X[support_indices]
        self.b = np.mean(self.y - self.a*self.y.T @ self.kernel(self.X, X_S))

    def predict(self,X_test):
        X_test=self.scaler.transform(X_test)
        return np.sign(self.a*self.y.T @ self.kernel(self.X, X_test) + self.b)

```

#### Part 2 - Defining Radial Basis Function(RBF) Kernel

```python
def rbf_kernel(X1,X2,sigma):
    return np.exp(-cdist(X1, X2, 'sqeuclidean') / (2*sigma**2))
```

#### Part 3 - Loading and Processing Dataset

```python
titanic_df = pd.read_csv('datasets/titanic_processed.csv')
X = titanic_df.drop('Survived',axis = 1).values
y = titanic_df['Survived'].values
y[y==0] = -1
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
```

#### Part 4 - Implementing SVM

```python
from functools import partial
C = .35
sigma = 2.5
kernel = partial(rbf_kernel,sigma=sigma)

svm_classifier = SVM(C,kernel)
svm_classifier.fit(X_train,y_train)
y_pred = svm_classifier.predict(X_test)
print(f'Accuracy of the classifer: {(y_test == y_pred).mean()*100:.2f}%')

```

         pcost       dcost       gap    pres   dres
     0: -2.1166e+02 -5.8558e+02  6e+03  8e+00  3e-15
     1: -8.8848e+01 -4.9225e+02  6e+02  4e-01  3e-15
     2: -8.4608e+01 -1.3669e+02  5e+01  2e-03  2e-15
     3: -9.2566e+01 -1.1007e+02  2e+01  5e-04  2e-15
     4: -9.5567e+01 -1.0239e+02  7e+00  1e-04  2e-15
     5: -9.6721e+01 -9.9922e+01  3e+00  6e-05  2e-15
     6: -9.7365e+01 -9.8738e+01  1e+00  8e-06  2e-15
     7: -9.7606e+01 -9.8266e+01  7e-01  2e-06  2e-15
     8: -9.7781e+01 -9.7964e+01  2e-01  2e-07  2e-15
     9: -9.7826e+01 -9.7894e+01  7e-02  4e-08  2e-15
    10: -9.7850e+01 -9.7861e+01  1e-02  5e-09  2e-15
    11: -9.7852e+01 -9.7857e+01  4e-03  7e-10  2e-15
    12: -9.7854e+01 -9.7855e+01  1e-03  1e-10  2e-15
    13: -9.7854e+01 -9.7855e+01  2e-04  2e-11  2e-15
    14: -9.7854e+01 -9.7854e+01  4e-05  3e-12  2e-15
    Optimal solution found.
    Accuracy of the classifer: 75.28%


```python
import numpy as np
import cv2
from matplotlib import pyplot as plt
import math
import sys
```

### Optical Flow
$$\min_\mathbf{d} E(\mathbf{d}) = \|\mathbf{J}(\mathbf{x}+\mathbf{d})-\mathbf{I}(\mathbf{x})\|^2$$

Three steps for solving this minimization problem:

- Solve for $\frac{\delta E}{\delta \mathbf{d}}|_{\mathbf{d^*}}=0$

\begin{align}
\frac{\delta E}{\delta \mathbf{d}}\bigg|_{\mathbf{d^*}} &= 2\frac{\delta \mathbf{J(x+d)}^\top}{\delta \mathbf{d}}(\mathbf{J}(\mathbf{x}+\mathbf{d})-\mathbf{I}(\mathbf{x})) = 0 \\
\frac{\delta E}{\delta \mathbf{d}}\bigg|_{\mathbf{d^*}} &= 2\frac{\delta \mathbf{J(x)}^\top}{\delta \mathbf{x}}(\mathbf{J}(\mathbf{x}+\mathbf{d})-\mathbf{I}(\mathbf{x})) = 0
\end{align}
where $\frac{\delta \mathbf{J}}{\delta \mathbf{x}} = \left[\begin{array}{cc}\frac{\delta \mathbf{J}}{\delta x}&\frac{\delta \mathbf{J}}{\delta y}\end{array}\right]^\top$.
- Taylor expansion on $\mathbf{J}(\mathbf{x}+\mathbf{d})=\mathbf{J}(\mathbf{x})+\frac{\delta \mathbf{J}(\mathbf{x})}{\delta \mathbf{x}}\mathbf{d}$,

- Solve for $\mathbf{d}$, warp image, iterate

```python
num_frames = 166
video_imgs = []
readdir = 'D:/44754/Documents/robotics/Perception/RoboticsPerceptionWeek3Code/data/apriltagims/image{:03d}.jpg'
for i in range(num_frames):
    tmp = cv2.imread(readdir.format(i+1))
    video_imgs.append(tmp)
video_imgs = np.array(video_imgs)
print(video_imgs.shape)
```

    (166, 480, 640, 3)

```python
initial_points = 100 * np.array([
   [1.981631469726562,3.165294189453125],
   [3.786268920898438,3.424402770996094],
   [4.036800842285157,1.686005859375000],
   [2.333528289794922,1.491907043457031] ]);
```

```python
plt.imshow(video_imgs[0,:,:,:])
plt.plot(initial_points[:,0],initial_points[:,1],'r.')
```

```python
gray = cv2.cvtColor(video_imgs[0,:,:,:],cv2.COLOR_BGR2GRAY)
# params for ShiTomasi corner detection
feature_params = dict( maxCorners = 5,
                       qualityLevel = .1,
                       minDistance = 75,
                       blockSize = 3 )

corners = cv2.goodFeaturesToTrack(gray, mask = None, **feature_params)
# cv2.goodFeaturesToTrack(gray,30,0.01,10)
corners = np.int0(corners)
newcopy = video_imgs[0,:,:,:].copy()

for i in corners:
    x,y = i.ravel()
    cv2.circle(newcopy,(x,y),5,255,-1)

plt.imshow(newcopy)
```

```python
import cv2 as cv
%matplotlib inline
%load_ext autoreload
%autoreload 2
%matplotlib notebook

fig = plt.figure(figsize=(20,10))
ax = plt.gca()
plt.ion()
fig.show()
fig.canvas.draw()

# Parameters for lucas kanade optical flow
lk_params = dict( winSize  = (31,31),
                  maxLevel = 3,
                  criteria = (cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 5, 0.03))
# Create some random colors
color = np.random.randint(0,255,(4,3))
# Take first frame and find corners in it
old_frame = video_imgs[0,:,:,:]
old_gray = cv.cvtColor(old_frame, cv.COLOR_BGR2GRAY)
p0 = np.floor(initial_points).reshape(-1,1,2).astype(np.float32)

# Create a mask image for drawing purposes
mask = np.zeros_like(old_frame)
for i in range(1,num_frames):
    frame = video_imgs[i,:,:,:]
    frame_gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
    # calculate optical flow
    p1, st, err = cv.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)
    # Select good points
    good_new = p1[st==1]
    good_old = p0[st==1]
    # draw the tracks
    for i,(new,old) in enumerate(zip(good_new, good_old)):
        a,b = new.ravel()
        c,d = old.ravel()
        mask = cv.line(mask, (a,b),(c,d), color[i].tolist(), 2)
        frame = cv.circle(frame,(a,b),5,color[i].tolist(),-1)
    img = cv.add(frame,mask)

    ax.clear()
    ax.imshow(img)
    fig.canvas.draw()

    k = cv.waitKey(5) & 0xff
    if k == 27:
        break
    # Now update the previous frame and previous points
    old_gray = frame_gray.copy()
    p0 = good_new.reshape(-1,1,2)

#     if i == num_frames-1:
    plt.savefig('result.png')
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```python

```


(page_topic1)=
Errors in Numerical Differentiation
=======================

In order for the finite difference formulas derived in the previous section to be useful, we need to have some idea of the errors involved in using these formulas.  As mentioned, we are replacing $f(x)$ with its polynomial interpolant $p_n(x)$ and then taking the derivatives, so that

$$ f'(x) = p_n'(x) + \epsilon'(x),\qquad f''(x) = p_n''(x) + \epsilon''(x).$$

We need to $\epsilon'(x)$ and \epsilon''(x) to determine the errors in the formulas we derived.

First recall [the polynomial interpolation error](../InterpFit/InterpErrors) is

$$\begin{equation} \epsilon(x)= f(x)-p_n(x) = \frac{f^{n+1}(\xi)}{(n+1)!}\prod_{j=0}^n (x-x_j). \end{equation}$$ (inter_err)

Taking the derivative of this using the product rule yields

$$
\begin{equation} \epsilon'(x) = \frac{f^{n+1}(\xi)}{(n+1)!}\left[\sum_{k=0}^n \prod_{j=0,j\neq k}^n (x-x_j)\right] + \frac{f^{n+2}(\xi)\xi'}{(n+1)!}\prod_{j=0}^n (x-x_j), \end{equation}$$ (errprime)

where the second term arises from the fact that $\xi$ may depend on $x$.  Evaluating this at one of the gridpoints, say $x_i$, simplifies the expression considerably as all but one of the terms contain the factor $(x_i-x_i)$,

$$
\begin{equation} \epsilon'(x_i) = \frac{f^{n+1}(\xi)}{(n+1)!}\prod_{j=0,j\neq i}^n (x_i-x_j). \end{equation}$$ (pprimeerr)

We can now use this expression for the errors in our first derivative finite difference formulas.  Before we do, let's derive the error formula for the second derivatives.

Taking the derivative {eq}`errprime` yields

$$
\begin{align} \epsilon''(x) = & \frac{f^{n+1}(\xi)}{(n+1)!}\left[ \sum_{k=0}^n \sum_{m=0, m\neq k}^n \prod_{j=0,j\neq k,m}^n (x-x_j)\right]\\
& + \frac{f^{n+2}(\xi)\xi''+f^{n+3}(\xi)\xi'}{(n+1)!}\prod_{j=0}^n (x-x_j) \\
&+ \frac{f^{n+2}(\xi)\xi'}{(n+1)!}\left[\sum_{k=0}^n \prod_{j=0,j\neq k}^n (x-x_j)\right],\end{align}$$

Evaluating this at one of the gridpoints, say $x_i$, simplifies the expression to,

$$
\begin{align}
\epsilon''(x_i) &=\frac{f^{n+1}(\xi)}{(n+1)!}\left[\sum_{m=0,m\neq i}^n \prod_{j=0, j\neq i,m} (x_i-x_j) + \sum_{k=0,k\neq i}^n \prod_{j=0, j\neq i,k} (x_i-x_j) \right] +\frac{f^{n+2}(\xi)\xi'}{(n+1)!}\prod_{j=0,j\neq i}^n (x_i-x_j),\\
&=\frac{2f^{n+1}(\xi)}{(n+1)!}\left[\sum_{k=0,k\neq i}^n \prod_{j=0, j\neq i,k} (x_i-x_j) \right] +\frac{f^{n+2}(\xi)\xi'}{(n+1)!}\prod_{j=0,j\neq i}^n (x_i-x_j).
\end{align}$$ (ppperr)

This is a bit messier than the error for the first derivative {eq}`pprimeerr`.  In most cases one can argue that the second term is small compared to the first as it contains one additional factor of the difference of grid points and one expects $\xi'$ to be small.

**Example: Error in the two point formulas**  In the previous section, for two points $x_0$ and $x_1=x_0+h$ we derived the forward/backwards difference formulae and we would now like to know the error in using these formulae.  Using {eq}`pprimeerr` with $n=1$ we straightforwardly get

$$ \epsilon'(x_0) = \frac{f''(\xi)}{2}(-h),\quad \text{and}\quad  \epsilon'(x_1) = \frac{f''(\xi)}{2}(h),$$

noting that $\xi$ is not likely to be the same number in the two cases.  So, in summary, we now have

$$
\begin{equation}
\left[{\begin{array}{c}
f'(x_0) \\
f'(x_1) \\
\end{array}} \right] =
\left[{\begin{array}{c}
\frac{f(x_1)-f(x_0)}{h} \\
\frac{f(x_1)-f(x_0)}{h} \\
\end{array}} \right] +
\left[{\begin{array}{c}
  -\frac{h}{2}f''(\xi) \\
   \frac{h}{2}f''(\xi) \\
\end{array}} \right].
\end{equation}
$$ (twopoint)

**Example: Error in the three point formulae**  In the previous section, for three points $x_0,\,x_1=x_0+h,$ and $x_2=x_0+2h,$ $x_0$ we derived finite difference formulae for first and second derivatives and we would now like to know the error in using these formulae.  Using {eq}`pprimeerr` with $n=2$ we straightforwardly get the result for the first derivatives:

$$
\begin{equation}
\left[{\begin{array}{c}
f'(x_0) \\
f'(x_1) \\
f'(x_2)
\end{array}} \right] =
\left[{\begin{array}{c}
\frac{-3f(x_0)+4f(x_1)-f(x_2)}{2h} \\
\frac{f(x_2)-f(x_0)}{2h} \\
\frac{3f(x_2)-4f(x_1)+f(x_0)}{2h}
\end{array}} \right] +
\left[{\begin{array}{c}
  \frac{h^2}{3}f^{(3)}(\xi) \\
  -\frac{h^2}{6}f^{(3)}(\xi) \\\
  \frac{h^2}{3}f^{(3)}(\xi) \\
\end{array}} \right].
\end{equation}
$$

First note that the errors here are $\mathcal{O}(h^2)$ compared to {eq}`twopoint` for the two-point formulae which are $\mathcal{O}(h)$.  So, assuming $h$ is small, these do indeed have a higher order accuracy.  Further, the error in the *central* difference formula is *half* that of the backward/forward differences.  As a result, the central difference formula is the one most widely used.

We now turn to the second derivative errors.  At $x_0$ {eq}`ppperr` gives

$$
\begin{align}
\epsilon''(x_0) &= 2\frac{f^{(3)}(\xi)}{3!}\left[(x_0-x_1)+(x_0-x_2)\right]+\frac{f^{(4)}(\xi)\xi'}{3!}(x_0-x_1)(x_0-x_2)\\
&= -f^{(3)}(\xi) h + \frac{f^{(4)}(\xi)\xi'}{3}h^2.
\end{align}
$$

At $x_1$ we get

$$
\begin{align}
\epsilon''(x_1) &= 2\frac{f^{(3)}(\xi)}{3!}\left[(x_1-x_0)+(x_1-x_2)\right]+\frac{f^{(4)}(\xi)\xi'}{3!}(x_1-x_0)(x_1-x_2)\\
&= -\frac{f^{(4)}(\xi)\xi'}{6} h^2.
\end{align}
$$

These, together with the result at $x_2$ gives

$$
\begin{equation}
\left[{\begin{array}{c}
f''(x_0) \\
f''(x_1) \\
f''(x_2)
\end{array}} \right] =
\left[{\begin{array}{c}
\frac{f(x_0)-2f(x_1)+f(x_2)}{h^2} \\
\frac{f(x_0)-2f(x_1)+f(x_2)}{h^2} \\
\frac{f(x_0)-2f(x_1)+f(x_2)}{h^2}
\end{array}} \right] +
\left[{\begin{array}{c}
  -h f^{(3)}(\xi) \\
  -\frac{h^2}{6}f^{(4)}(\xi)\xi' \\\
  h f^{(3)}(\xi) \\
\end{array}} \right],
\end{equation}
$$

where we have dropped the higher order error terms.  Note that even through the approximation formula is the same for all 3 points, the error in this formula is much worse at the endpoints ($\mathcal{O}(h)$ versus $\mathcal{O}(h^2)$).  In homework you will see that you can also derive the central difference formula using a Taylor series expansion (including errors) about $x_1$ and then using the mean value theorem to combine the errors at the points $x_1\pm h$ at a new, also unknown, point $\eta$ to get

$$ f''(x_1) \approx \frac{f(x_0)-2f(x_1)+f(x_2)}{h^2} -\frac{h^2}{12}f^{(4)}(\eta). $$

### Stability

The central difference formula we derived

$$
f'(x_1)=\frac{f(x_2)-f(x_0)}{2h}-\frac{h^2}{6}f^{(3)}(\xi),
$$

involves two things we warned about when we talked about [roundoff errors](../ErrorsModule/RoundoffAmplification).  Namely, subtraction of two similar numbers, the $f(x_2)-f(x_0)$, and division by a small number $h$.  As a result, attempting to obtain the "exact" value of $f'(x_1)$ by taking ever smaller $h$ is **not** a numerically stable operation as we will now demonstrate.

As in our discussion of [roundoff errors](../ErrorsModule/HornersAlgorithm) let $\mathcal{fl}(.)$ denote the finite precision arithmetic on the computer, i.e.

$$
\mathcal{fl}(f(x))=f(x) + e_r(x)
$$

where $e_r$ is the roundoff error.  The error in evaluation of the central difference formula on a computer is then

$$
\begin{align}
E(h) &= \left|f'(x_1)-\mathcal{fl}\left(\frac{f(x_2)-f(x_0)}{2h} \right)\right|, \\
&= \left|-\frac{h^2}{6}f^{(3)}(\xi) + \frac{e_r(x_2)-e_r(x_0)}{2h}\right|,
\end{align}
$$

where the second term comes from the fact that we cannot evaluate the central difference exactly using finite precision arithemetic.  The roundoff errors here are unlikely to be correlated and, in particular, are as likely to be positive as negative.  We do expect it to be bounded so let's assume $|e_r(x)|<\beta$.  Putting this together, along with use of the triangle inequality gives

$$
E(h) \leq \frac{\beta}{h} + \frac{h^2}{6}|f^{(3)}(\xi)|.
$$

Note that while the truncation error (the second term) decreases as $h$ gets smaller, the roundoff error *increases* as $h$ gets smaller.  As a result, there is actually an *optimal* $h$ to minimize the error.  To find this we take the derivative of $E(h)$ and set it to zero to get

$$
\frac{dE}{dh}\approx -\frac{\beta}{h^2}+\frac{h}{3}|f^{(3)}(\xi)| = 0.
$$

Solving this for the optimal $h_{op}$ gives

$$
h_{op}=  \sqrt[3]{\frac{3 \beta}{|f^{(3)}(\xi)|}}.
$$

If $f(x)\sim 1$ we would expect the roundoff to be of order of the machine epsilon $\epsilon_r$.  For $f(x)$ not of order unity, we would expect $\beta \approx \epsilon_r |f(x)|$. So around a given $x$ we expect roughly

$$
h_{op}\approx  \sqrt[3]{\frac{3 \epsilon_r |f(x)|}{|f^{(3)}(x)|}}.
$$ (hop)

**Example** Suppose we use the central difference formula to evaluate the derivative of $f(x)=e^x$.  What is the error and what is the optimal $h_{op}$?

First note that we can work out the analytic derivative ($f'(x)=e^x$) and the third derivative here ($f^{(3)}(x)=e^x$).  In this case, {eq}`hop` then gives,

$$
h_{op}=6.7\times 10^{-6}.
$$

We can test this assertion with a short python script:

```python
from matplotlib import pyplot as plt
import numpy as np
x = np.arange(2, 12, 0.1)
h = pow(10,-x)
err = abs(np.exp(1.)-(np.exp(1.+h)-np.exp(1.-h))/(2.*h))
plt.loglog(h,err)
plt.xlabel(r'$h$')
plt.ylabel(r'absolute error')
plt.show()
```

Indeed, we do see the minimum in the error plot is around our calculated $h_{op}$.  Interestingly, the error using $h=10^{-12}$ is comparable to the error using $h=10^{-2}$.  Also, the part of the curve dominated by roundoff errors ($h<h_{op}$) is quite noisy.  This is simply a refection of the somewhat unpreditable nature or roundoff errors.  Importantly, our arguments correctly predict the trends here.  In practise, though, one should keep in mind that roundoff errors may fluctuate an order or magnitude around these trends.

Note that similar problems arise when attempting to apply numerical differentiation to noisy data (data with stochastic noise).  A similar construction, with the errors being from stochastic noise rather than roundoff, limits how small $h$ can be before the results for the derivative become less accurate.  In this case, a presmoothing operation to reduce the noise amplitude should be applied before attempting to construct a finite difference for the derivative.


$\newcommand{\RR}{\mathbb{R}}$
$\newcommand{\Nn}{\mathcal{N}}$
$\newcommand{\EE}{\mathbb{E}}$

```python
%matplotlib inline
from ipywidgets import interactive
import numpy as np
from numpy import linalg as la
import matplotlib.pyplot as plt
import os.path
```

```python
# Uncoment the pip command below if you are using Google Colab.
# In case of errors, use the option --upgrade --force-reinstall

#!pip install git+https://github.com/Guillaume-Garrigos/invprob.git  # For Google Colab
import sys  # For local launch
sys.path.append('..')  # For local launch
import invprob.sparse as sparse
from invprob.optim import fb_lasso
```

```python
np.random.seed(seed=74)  # Seed for np.random
dpi = 100  # Resolution for plotting (230 for small screen, 100 for large one)
plt.ion()
```

# Notes on Inverse Problems
(c) Guillaume Garrigos. This notebook is rendered as a static page on [nbviewer](https://nbviewer.jupyter.org/github/Guillaume-Garrigos/invprob/blob/master/notebook/lasso_2-noise_parameter_selection.ipynb), and as an interactive notebook on [Google Colab](https://colab.research.google.com/github/Guillaume-Garrigos/invprob/blob/master/notebook/lasso_2-noise_parameter_selection.ipynb)

## II. Sparse Regularisation with the L1 norm (LASSO) : noise and parameter selection

We still try to recover a ground truth signal $x_0\in \RR^N$, which is sparse, in the sense that it is the solution of the following optimisation problem

\begin{equation}\label{Eq:true problem}
\min\limits_{y = \Phi x} \ \Vert x \Vert_1 , \hspace{1cm} \text{ where } \Vert x \Vert_1 := \sum_{i=1}^N \vert x_i \vert.
\tag{$P_0$}
\end{equation}

But instead of being given the linear measurment $y$, we assume here that we are given a *noisy* data $\hat{y}$, such that $\Vert y - \hat y \Vert \leq \sigma$.
In other words, $\hat y = \Phi x_0 + \varepsilon$, where $\Vert \varepsilon \Vert \leq \delta$.

### 1. Presentation of the problem

We define $\Phi$, $x_0$, $y$ and a unitary noisy vector $\varepsilon$.

```python
# We start by defining the characteristics of the problem
data_size = 100
data_number = round(data_size / 2)
sparsity_level = 10

# We define the main components of our problem
Phi = np.random.randn(data_number, data_size)
x0 = np.sign(sparse.randn(data_size, 1, sparsity_level))
y = Phi@x0
noisy_vector = np.random.randn(data_number, 1)
```

Given a noisy measurement $\hat{y}$, we can try to find (approximatively) a solution of

\begin{equation}
\min\limits_{\hat{y} = \Phi x} \ \Vert x \Vert_1
\tag{$\hat{P}_0$}
\end{equation}

by taking a small value of $\lambda$ and computing a solution $\hat x_\lambda$ of

\begin{equation}
\min\limits_{x \in \RR^N} \hat{f}_\lambda(x):= \Vert x \Vert_1 + \frac{1}{2 \lambda}\Vert \Phi x - \hat{y} \Vert^2.
\tag{$\hat{P}_\lambda$}
\end{equation}

We will do that by running an instance of the Forward-Backward algorithm introduced in the previous notebook.

```python
noise_level = 1
reg_param = 0.1
iter_nb = 40000
x_reg, details = fb_lasso(Phi, y + noise_level*noisy_vector, reg_param, iter_nb, verbose=True)
```

```python
plt.figure(dpi=dpi, figsize=(12, 4))

plt.subplot(1, 2, 1)
sparse.stem(x0, "C0", "ground truth")
sparse.stem(x_reg, "C1", "reg solution")

plt.subplot(1, 2, 2)
plt.title(r"Evolution of #supp$(\hat{x}^k)$")
plt.plot(details.get("iterate_support"))
plt.show()
```

We see that the algorithm converges towards a solution $\hat{x}_\lambda$ which is far from being close to $x_0$.
This is normal, since information has been lost about $x_0$ when noise contaminated the measurements.

Nevertheless, we are going to see that even though we have no hope to recover exactly $x_0$ in presence of noise, we can do *better* than what we just did.

### 2. The regularization path, again

Given noisy data, remember that the regularization path is

$$\{x_\lambda \ | \ x_\lambda \text{ is a solution of } (P_\lambda), \lambda > 0 \}.$$

We saw in the noiseless case that an interesting feature of the regularization path is that $\hat{x}_\lambda \to \hat{x}_0$ when $\lambda \to 0$.
But this is useless here, since we are not interested at all in the solutions of the noisy problem $(\hat{P}_0)$.

Actually, the regularization path has an other interesting feature: it is *stable* with respect to noise on $y$.
To try to see this, we are going to compute the regularization path and try to vizualise it.
A way to do so is to project these high-dimensional vectors onto a 2D plane.
Here, we select this 2D plane by extracting the two first eigenvectors of the covariance matrix of the path.

```python
def compute_reg_path(Phi, y, reg_param_grid):
    ''' Given a matrix Phi and (noisy) data y, returns the regularision
    path: we compute solutions $x_\lambda$ of each problem $(P_\lambda)$
    for $\lambda$ being taken in the list reg_param_grid
    '''
    reg_path = np.empty((Phi.shape[1], 0), int)
    x_ini = np.zeros((Phi.shape[1], 1))
    for reg_param in reg_param_grid:
        ''' We choose the number of iterations to do depending on the reg_param.
        This is a completely custom choice, it seems to work quite well
        in practice.
        '''
        if reg_param < 0.1:
            iter_nb = 60000
        elif reg_param < 1:
            iter_nb = 4000
        elif reg_param < 10:
            iter_nb = 1000
        else:
            iter_nb = 200
        # We use a warm restart approach:
        # for each problem we use the solution of the previous problem
        # as a starting point
        x_reg = fb_lasso(Phi, y, reg_param, iter_nb, x_ini=x_ini)
        x_ini = x_reg
        reg_path = np.concatenate((reg_path, x_reg), axis=1)
    return reg_path
```

```python
# We put a grid on the space of reg_param
# We taking it on a log scale and decreasing, between 10^2 and 10^-2
regp_number = 100
reg_param_grid = np.round(np.logspace(-2, 2, regp_number), 3)

# We compute the regularisation path for different values of noise:
# sigma = 10^-2, 10^-1, 10^0 and 0
# This can take some time!
reg_path_2 = compute_reg_path(Phi, y + 1e-2 * noisy_vector, reg_param_grid)
reg_path_1 = compute_reg_path(Phi, y + 1e-1 * noisy_vector, reg_param_grid)
reg_path_0 = compute_reg_path(Phi, y + 1e-0 * noisy_vector, reg_param_grid)
reg_path = compute_reg_path(Phi, y, reg_param_grid)
```

```python
def get_pca_projector(data):
    ''' Given a set of data points (stored in columns of a 2D np.array)
    returns the projection matrix corresponding to the 2 more relevant
    directions
    '''
    centered_points = data - np.mean(data, axis=1).reshape(data.shape[0], 1)
    V = np.cov(centered_points)
    _, vectors = la.eig(V)
    return np.real(vectors[:, 0:2]).T

def scatter_reg_path(path, limit=None, title=None):
    ''' Given a 2D path, plots it using scatter
    '''
    cm = plt.cm.get_cmap('RdBu')
    if limit is not None:
        plt.scatter(limit[0, :], limit[1, :], c='r', marker='x', s=150)
    if title is not None:
        _ = plt.title(title)
    plt.plot(path[0, :], path[1, :], c='black', linewidth=0.5)
    fig = plt.scatter(path[0, :], path[1, :], c=range(path.shape[1]),
                      vmin=0, vmax=path.shape[1], s=35, cmap=cm)
    plt.colorbar(fig)
```

```python
# We plot the (projected) regularisation paths and x0
P_svd = get_pca_projector(reg_path_0)

plt.figure(dpi=dpi, figsize=(12, 8))
plt.subplot(2, 2, 1)
scatter_reg_path(P_svd@reg_path, title="$\delta$=0", limit=P_svd@x0)
plt.subplot(2, 2, 2)
scatter_reg_path(P_svd@reg_path_2, title="$\delta$=0.01", limit=P_svd@x0)
plt.subplot(2, 2, 3)
scatter_reg_path(P_svd@reg_path_1, title="$\delta$=0.1", limit=P_svd@x0)
plt.subplot(2, 2, 4)
scatter_reg_path(P_svd@reg_path_0, title="$\delta$=1", limit=P_svd@x0)
plt.show()
```

We see that:
- in the noiseless case ($\delta=0$), the regularisation path goes from $x_{100}=0 $ to $x_{0.01} \sim x_0$.
- the more noise there is, the more the regularisation path tends to diverge from $x_0$ (in each case represented by a red cross). This is normal since it converges to $\hat{x}_0$.
- in presence of noise, the "extremity" of the path might be far from $x_0$ but other points on the path might be closer! So we should not take $\lambda \sim 0$, but instead choose it carefully so that $\hat{x}_\lambda$ is as close as possible from $x_0$.

In inverse problems, the step of choosing $\lambda$ is commonly referred as the *regularisation parameter selection* (or *hyperparameter selection* in the machine learning community).

### 3. Regularisation parameter selection (given the ground truth)

In this section we are interested in, given the regularisation path, finding the best possible regularised solution.
Here "best" is to be understand in the $L^2$ sense, meaning that we will look for the parameter $\lambda(\delta)$ minimizing $\Vert \hat{x}_\lambda - x_0 \Vert_2$.

```python
def reg_param_selection(path, ground_truth):
    '''Given a regularization path and a ground truth,
    returns the best regularized solution (in L2 sense)
    '''
    path_length = path.shape[1]
    path_dimension = path.shape[0]
    reg_sol = path[:, 0, None]
    reg_param_index = 0
    for k in np.arange(path_length):
        if la.norm(ground_truth - path[:, k, None]) \
                < la.norm(ground_truth - reg_sol):
            reg_sol = path[:, k, None]
            reg_param_index = k
    return reg_sol, reg_param_index
```

```python
# We compute the best parameter given the ground truth
reg_sol, reg_param_index = reg_param_selection(reg_path, x0)
reg_sol_2, reg_param_index_2 = reg_param_selection(reg_path_2, x0)
reg_sol_1, reg_param_index_1 = reg_param_selection(reg_path_1, x0)
reg_sol_0, reg_param_index_0 = reg_param_selection(reg_path_0, x0)
```

```python
# We represent the selected solutions
_ = plt.figure(dpi=dpi, figsize=(12, 8))
plt.subplot(2, 2, 1)
sparse.stem(x0, "C0", "ground truth")
sparse.stem(reg_sol, "C1", "reg solution")
plt.title(r"$\delta$=0, $\lambda$="+str(reg_param_grid[reg_param_index]))
plt.subplot(2, 2, 2)
sparse.stem(x0, "C0", "ground truth")
sparse.stem(reg_sol_2, "C1", "reg solution")
plt.title(r"$\delta$=0.01, $\lambda$="+str(reg_param_grid[reg_param_index_2]))
plt.subplot(2, 2, 3)
sparse.stem(x0, "C0", "ground truth")
sparse.stem(reg_sol_1, "C1", "reg solution")
plt.title(r"$\delta$=0.1, $\lambda$="+str(reg_param_grid[reg_param_index_1]))
plt.subplot(2, 2, 4)
sparse.stem(x0, "C0", "ground truth")
sparse.stem(reg_sol_0, "C1", "reg solution")
plt.title(r"$\delta$=1, $\lambda$="+str(reg_param_grid[reg_param_index_0]))
plt.show()
```

In the case $\delta = 0.1$, the solution we obtain is *much* better than $\hat{x}_0$ that we computed at the begining!

We can visualise these selected solutions on a 2D plane as well:

```python
plt.figure(dpi=dpi, figsize=(12, 8))
plt.subplot(2, 2, 1)
scatter_reg_path(P_svd@reg_path,
                 title="$\delta$=0, $\lambda$="+str(reg_param_grid[reg_param_index]),
                 limit=np.concatenate((P_svd@x0, P_svd@reg_sol), axis=1))
plt.subplot(2, 2, 2)
scatter_reg_path(P_svd@reg_path_2,
                 title="$\delta$=0.01, $\lambda$="+str(reg_param_grid[reg_param_index_2]),
                 limit=np.concatenate((P_svd@x0, P_svd@reg_sol_2), axis=1))
plt.subplot(2, 2, 3)
scatter_reg_path(P_svd@reg_path_1,
                 title="$\delta$=0.1, $\lambda$="+str(reg_param_grid[reg_param_index_1]),
                 limit=np.concatenate((P_svd@x0, P_svd@reg_sol_1), axis=1))
plt.subplot(2, 2, 4)
scatter_reg_path(P_svd@reg_path_0,
                 title="$\delta$=1, $\lambda$="+str(reg_param_grid[reg_param_index_0]),
                 limit=np.concatenate((P_svd@x0, P_svd@reg_sol_0), axis=1))
plt.show()
```

### 4. Convergence rates (in the "inverse problem" sense)

We just saw that, by choosing carefully $\lambda$, we obtain regularised solutions $\hat{x}_\lambda$ which approximates quite well $x_0$.
But how good this approximation can be? Of course, the more the data is corrupted by noise, the worse the approximation will be.
But we would like to know how much the error $\Vert \hat{x}_{\lambda(\delta)} - x_0 \Vert$ is affected by the level of noise $\delta$.

In general we try to obtain upper bounds of the form

$$ \Vert \hat{x}_{\lambda(\delta)} - x_0 \Vert = O(\delta^\alpha),$$

where $\alpha >0$ is a certain constant that we hope to be as large as possible.

```python
# We import precomputed data (see a few blocks below for why we do that)
# Be careful, relative paths can be tricky depending on the platform ..
import_data = np.load('../output/L1_reg/reg_path_rates_examples.npy')
noise_level_grid = import_data[:, :, 0]
param_value = import_data[:, :, 1]
dist_to_truth = import_data[:, :, 2]
```

```python
# We plot the rates
plt.figure(dpi=dpi, figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.title(r"Evolution of $\Vert x_{\lambda_\delta} - x_0 \Vert$ in function of $\delta$")
plt.loglog(noise_level_grid, dist_to_truth)
plt.subplot(1, 2, 2)
plt.title(r"Evolution of $\lambda(\delta)$ in function of $\delta$")
plt.loglog(noise_level_grid, param_value)
plt.show()
```

We can see from the graphs above that for $\lambda(\delta) \sim \delta$, we have $\Vert \hat{x}_{\lambda(\delta)} - x_0 \Vert \sim \delta$.

To produce the above figures, we computed 20 regularisation paths for 20 different levels of noise, after what we selected the best regularisation parameter.
If you want to reproduce this experiment, with maybe different parameters, here is the code:

```python
wanna_compute = False  # Just to prevent the computation of the next block by accident
batch_nb = 20  # number of paths we want to compute
noise_level_grid = np.round(np.logspace(-2, 0, batch_nb), 3).reshape(batch_nb, 1)
```

```python
# This loop is *very* expensive
if wanna_compute:
    param_value = np.empty([batch_nb, 1])
    dist_to_truth = np.zeros((batch_nb, 1))
    for k in np.arange(batch_nb):
        print("Compute the regularisation path "+str(k+1)+" over "+str(batch_nb))
        reg_path_temp = compute_reg_path(Phi,
                                         y + noise_level_grid[k] * noisy_vector,
                                         reg_param_grid)
        reg_sol_temp, reg_param_index_temp = reg_param_selection(reg_path_temp, x0)
        param_value[k] = reg_param_grid[reg_param_index_temp]
        dist_to_truth[k] = la.norm(reg_sol_temp - x0)
```

```python

```


 ILI285/INF285 Computación Científica

 Aritmética de punto flotante y pérdida de precisión

```python
import numpy as np
import bitstring as bs
import matplotlib.pyplot as plt
```

# Números binarios

**Número binario**: Son aquellos números escritos en función te potencias de $2$. Se representan como:

$$
B = \cdots b_{2}\,b_{1}\,b_{0}.b_{-1}\,b_{-2}\cdots
$$

donde cada dígito o bit es 0 o 1.

Al igual que con los decimales, se escriben como potencias de 2:

$$
\begin{align}
(1001.11)_{2} &= 1\,2^{3}+0\,2^{2}+0\,2^{1}+1\,2^{0}+1\,2^{-1}+1\,2^{-2} \\
&= 8 + 0 + 0 + 1 + 0.5 + 0.25\\
&= 9 + 0.75\\
&= (9.75)_{10}
\end{align}
$$

Para transformar:
**Decimal a binario**: Separar parte entera y decimal.

Luego:

* Dividir la parte entera por 2 sucesivamente y almacenar el resto. El número binario se escibe en sentido contrario a como aparecieron.

* La parte fraccionaria se multplica por 2, y la parte entera son los digitos binarios.

**Ejemplo**: Transformemos $(53.7)_{10}$.

Comenzamos separando la parte entera y decimal. $(53.7)_{10} = 53 + 0.7$. La parte entera queda como:

$$
\begin{align}
53/2 &= 26,\,R = 1\\
26/2 &= 13,\,R = 0 \\
13/2 &= 6, \,R = 1 \\
6/2 &= 3,\, R= 0\\
3/2 &= 1,\, R = 1\\
1/2 &= 0, \,R = 1
\end{align}
$$

Por lo tanto la parte entera corresponde ser a $110101$ en binario. La parte decimal queda como:

$$
\begin{align}
0.7\times 2 &= 1+0.4,\,R = 1\\
0.4\times 2&=  0+0.8\,R = 0 \\
0.8\times 2 &= 1+0.6, \,R = 1 \\
0.6\times 2 &= 1+0.2,\, R= 1\\
0.2\times 2 &= 0+0.4,\, R = 0\\
&\vdots
\end{align}
$$

Por lo tanto la parte decimal corresponde ser a $0.1\overline{0110}$. Finalmente el numero final $(53.7)_{10} = (110101.1\overline{0110})_{2}$

El algoritmo anterior funciona debido a:

* Sea $x$ el número, se separa en $x = n + d$, con $n >1$, $d \in (0,1)$.
* $n$ se divide por 2 y se toma el resto, es decir: $n = 2n_{0} + r_{0}$. Luego se realiza lo mismo con $n_{o}$:

$$
n_{0} = 2n_{1}+r_{1} \rightarrow n = 2(2n_{1} + r_{1}) + r_{0} \\
n = 2^{2} n_{1} + 2^{1}r_{1} + 2^{0}r_{0}
$$

Para la parte decirmal es al revés: $2d = d_{-1} + r_{-1} \rightarrow d = 2^{-1}d_{-1}+2^{-1}r_{-1}$, pero $2d_{-1} = d_{-2} + r_{-2}$, entonces:

$$
\begin{align}
d = 2^{-1}d_{-1} + 2^{-1}r_{-1} &= 2^{-1}(2^{-1}d^{-2} + 2^{-1}r^{-2}) + 2^{-1}r_{-1}\\
=& 2^{-2}d_{-2} + 2^{-2}r_{-2} + 2^{-1}r_{-1}
\end{align}
$$

y así sucesivamente.

**De binario a decimal**: Se separa la parte entera de la decimal. Luego:

* La parte entera se calcula con potencias de 2.
* La parte fraccionaria, si es finita, se puede usar potencias de 2 con exponente negativos. Si es periodica, se puede despalzar la parte periodica o aplicar serie geometrica.

**Ejemplo**:

Para la parte periódica, como se mencionó, se puede:
**Desplazar**: Tomemos $x = (0.\overline{1011})_{2} \rightarrow 2^{4} x = (1011.\overline{1011})_{2}$ Asi:

$$
2^{4}x -x = (1011.\overline{1011})_{2} - (0.\overline{1011})_{2} = (1011)_{2} = 11 \\
\therefore (2^{4} -1 )x = 11\\
\rightarrow x = \frac{11}{2^{4}-1} = \big(\frac{11}{15}\big)_{10}
$$
**Serie geométrica**:

Recordar que:

$$
1 + r + r^{2} + \cdots = \displaystyle \sum_{i=0}^{\infty} r^{i} = \frac{1}{1-r}, \quad |r| < 1
$$

Así, por ejemplo: $x = (0.\overline{10})_{2}$ es:

$$
x = (0.101010\cdots)_{2} = \displaystyle \sum_{i=0}^{\infty} 2^{-(2i+1)} = \displaystyle \sum_{i=0}^{\infty} 2^{-2i} 2^{-1} = \frac{1}{2} \displaystyle \sum_{i=0}^{\infty} (2^{-2})^{i},\,\,\,\, r = 2^{-2} = \frac{1}{4}\\
\therefore x = \frac{1}{2} \frac{1}{1-\frac{1}{4}} = \frac{1}{2} \frac{1}{\frac{3}{4}} = \frac{1}{2} \frac{4}{3} = \Big(\frac{2}{3}\Big)_{10}
$$

# Punto flotante

* El standart actual es el IEEE754, el cual permite representar números reales.
* El números se componete de tres partes:
    * El signo (+/-)
    * Mantisa: bits significativos
    * Exponente de la base 2
* Su forma es:
$$
b = \underbrace{\pm}_{a}1.\underbrace{b_{1}b_{2}\cdots b_{n}}_{b} \times 2^{\overbrace{p}^{c}}
$$

(a) Signo
(b) Mantisa
(c) Exponente

a todo lo anterior se le llama word/palabra

**Ejemplos**:
* $9 = (1001.0)_{2} = 1.001 \times 2^{3}$
* $0.95 = (0.11)_{2} = 11 \times 2^{-1}$

Los tamaños usuales son:
* Single (32): 1 bit signo, 8 exponente, 23 mantisa
* Double (64): 1 bit signo, 11 exponente, 52 mantisa
* Long double(80): 1 bit signo, 15 exponente, 64 mantisa

Cada forma funciona de manera equivalente. Todas funcionan en formanto **normalizado**, es decir, el bit entero es 1:

$$
b = \pm 1.b_{1}b_{2}\cdots b_{n} \times 2^{p}
$$

El bit más significativo se deja en 1 moviendo el exponente.

Notar que el $1$, tomando 64 bits es:

$$
1_{10} = \pm 1.00 \cdots 0 \times 2^{0} \rightarrow b_{i} = 0, \forall i \in \{1 \cdots 52\}
$$

El siguiente flotante más grande de 1 es:

$$
+ 1.00 \cdots 001 \times 2^{0} = 1\times 2^{0} + 2^{-52} \times 2^{0} = 1 + 2^{-52}
$$

**Machine epsilon**: Denotado como $\epsilon_{mach}$, se define como la distancia entre el 1 y el siguiente número flotante mayor que 1. Para el IEEE double precision, su valor es:

$$
\epsilon_{mach} = 2^{-52} \approx 2·10^{-16}
$$

Cuando un número binario tiene mas de 52 bits, los bits que no son almacenados pueden **truncarse** (descartar bits sobrantes) o **Redondearse** (aproximar de alguna forma, lo que hace IEEE754).

**Regla del redondeo al más cercano de IEEE**: Para doble precisión, si el bit 53 después del punto decimal:

* Es cero, se redondea hacia abajo (truncar).
* Si es uno, entonces:
    * Redondear hacia arriba (sumar 1 al bit 52) si los números después del bit 53 no son todos cero.
    * Si todos los bits después del bit 53 son cero, entonces se suma uno al bit 52 si y solo si es 1.

Denotaremos por $fl(x)$ el valor en punto flotante con **IEEE754**, con regla de redoneo al más cercano de un número real $x$.

#### Ejemplos

\begin{equation}
    −1.\underbrace{0001 \dots 0001}_{52 ~ \text{bits}}|1001\dots\cdot 2^8 \implies −1.\underbrace{0001 \dots 0010}_{52 ~ \text{bits}} \cdot 2^8
\end{equation}

Como los números a la derecha del bit $52$ no son todos $0$, simplemente se suma $1$ al bit $52$. Se debe considerar el acarreo correspondiente.

\begin{equation}
    +1.\underbrace{0001 \dots 0001}_{52 ~ \text{bits}}|1000\dots \cdot 2^5 \implies +1.\underbrace{0001 \dots 0010}_{52 ~ \text{bits}} \cdot 2^5
\end{equation}

En este caso todos los bits después del bit $53$ son $0$, pero como el bit $52$ es $1$ entonces se le suma $1$.

\begin{equation}
    +1.\underbrace{1001 \dots 0100}_{52 ~ \text{bits}}|1000\dots\cdot 2^3 \implies +1.\underbrace{1001 \dots 0100}_{52 ~ \text{bits}} \cdot 2^3
\end{equation}

Para este ejemplo no se le suma $1$ al bit $52$ debido a que los que aparecen a la derecha del bit $53$ son puros $0$.

**Error absoluto y relativo**: Sea $x_{c}$ la versión comptutada de un valor real $x$, entonces:

* Se define el **error absoluto** como $|x_{c} - x|$
* Se define el **error relativo** como $\frac{|x_{c}-x|}{|x|}$, si existe dicho valor.

**Cota de redondeo**: Sea $fl(x)$ el valor aproximado para un modelo de punto flotante de $x$. Entonces:

\begin{equation}
    \frac{|fl(x) - x|}{|x|} \leq \frac{1}{2} \cdot \epsilon_{mach}
\end{equation}

o de manera análoga

\begin{equation}
    |fl(x) - x| \leq \frac12 \cdot \epsilon_{mach} |x|
\end{equation}

Esto quiere decir que la distancia entre la representación de punto flotante y el número $x$ es proporcional a la magnitud del real $x$.

**Demostración**: Tomemos $x=1.b_{1}b_{2}\dots b_{52} b_{53} \dots \times 2^{p}$ y $fl(x) = 1.b_{1}b_{2}\dots b_{52} \times 2^{p}$. Tenemos:

* Si $b_{53} = 0 \rightarrow b{i} = e_{i}$ y $x > fl(x)$:

$$
\begin{align}
0 < x - fl(x) &= 1.b_{1}b_{2}\dots b_{52} \dots 2^{p} - 1.b_{1}b_{2}\dots b_{52} \times 2^{p}\\
&=0.000\dots 0 b_{53} \dots \times 2^{p}\\
&=0.b_{53} \dots \times 2^{p-52}\\
&= b_{53}.b_{54} \dots \times 2^{p-1} \epsilon_{mach}\text{, como } b_{53} < 1 \\
&\leq 2^{p-1} \epsilon_{mach}
\end{align}
$$

* Si $b_{53} = 1 \rightarrow$ Redondeo: $fl(x) = 1.b_{1}b_{2} \dots b_{52} \times 2^{p} + \epsilon_{mach} \times 2^{p}$

$$
\begin{align}
0 < fl(x) - x &= 1.b_{1}b_{2}\dots b_{52} \times 2^{p} + \epsilon_{mach} \times 2^{p} - 1.b_{1}b_{2}\dots b_{52} \dots \times 2^{p}  \\
&= (e_{mach} - 0.00 \dots 0 b_{53} \dots ) \times 2^{p}\\
&= (e_{mach} - 0. b_{53} \dots \times 2^{-52}) \times 2^{p} \\
&= (1-0.b_{53} \dots) \times 2^{p} \epsilon_{mach}
\end{align}
$$

$$
\begin{align}
\text{Como } 0.b_{53}\dots \geq 0.5 \iff -0.5 &\geq -0.b_{53}\dots\,\,\, /+1\\
1 - \frac{1}{2} &\geq 1 -0.b_{53}\dots\\
\frac{1}{2} &\geq 1 -0.b_{53}\dots
\end{align}
$$

$$
\therefore 0 < fl(x) - x \leq \frac{1}{2} 2^{p} \epsilon_{mach} = 2^{p-1} \epsilon_{mach}
$$

La definición de valor absoluto nos dice:

$$
|a|\geq b\iff a\leq -b\ \lor a \geq b
$$

En nuestro caso

$$
\begin{align}
x - fl(x) \leq  2^{p-1} \epsilon_{mach} \lor fl(x) - x &\leq  2^{p-1} \epsilon_{mach} \\
x - fl(x) \leq  2^{p-1} \epsilon_{mach} \lor  - 2^{p-1} \epsilon_{mach}  &\leq  x - fl(x)
\end{align}
$$
Por lo que obtenemos:

$$
|x-fl(x)| \leq 2^{p-1} \epsilon_{mach}
$$

Además: $|x| = 1. b_{1} \dots b_{52} \dots \times 2^{p} \geq 2^{p} \iff \frac{1}{|x|} \leq \frac{1}{2^{p}}$.

Finalmente obtenemos:

$$
\frac{|x-fl(x)|}{|x|} \leq \frac{2^{p-1} \epsilon_{mach}}{2^{p}} = \frac{1}{2} \epsilon_{mach}
$$

**La estructura de un número de punto flotante**, considerando 64 bits u 8 bytes es:

**Word**:

| $\text{Signo}$ | $\text{Exponente}$   | $\text{Mantisa}$ |
|------|------| ----|
| 1 bit| 11 bits | 52 bits |
|   $s$  | $e_{1}$ ... $e_{n}$ | $b_{1}$ ...  $b_{52}$ |

La estructura de cada parte es:

* **Signo**: Solo es un bit, por lo que es 0 o 1. Se toma que 0 es positivo y 1 es negativo
* **Exponente**: Como son 11 bits, hay $2^{11} = 2048$ combinaciones sin embargo, el 0 y el 2047 son usados para propósitos especiales. El rango disponible es [1,2046]. Se usa **Exponent bias**, que mapea el rango disponible (positivo, para almacenar en binario) a un rango con valores positivos y negativos.

La regla para encontrar el bias mediante el standart IEEE-754 es:

$$K = 2^{n-1} - 1,$$

donde $n$ es el número de bits.

Exponent bias para **Double**:

$2^{11-1} - 1 = 2^{10} - 1 = 1024 - 1 = 1023 $

$p \in [ 1- 1023, 2046-1023] = [-1022,1023]$

```python
y = lambda x: x - 1023
points = np.linspace(1,2026,100)

plt.figure(figsize=(5, 5))
plt.plot(points,y(points),label = r"y = x - 1023")
plt.grid(True)
plt.legend()
plt.show()
```

```python
#n = '000000000000'
n = '11111111101'
print("Exponente sin shift",int(n,2))
if int(n,2) == 2047 or int(n,2) == 0:
    print("Caso especial!")
else:
    print("Exponente con shift",int(n,2)-1023)
    print("2**"+str(int(n,2)-1023))
```

    Exponente sin shift 2045
    Exponente con shift 1022
    2**1022

Además, es importante recalcular que los bits del exponente se almacenan en orden inverso, es decir:

$p: e_{1}e_{2}\cdots e_{11} \rightarrow \overline{p} = e_{1}2^{10} + e_{2}2^{9} + \cdots + e_{11} 2^{0} - 1023$

**Nota**: Como el número es almacenado en potencias de 2, los intervalos y la cantidad de números representable en estos intervalos son más grandes.

**Casos especiales del exponente**:

* **P=2047**: es decir, p = $11111111111$. En este caso, se designan números o valores "especiales":

| $s$ | $e_{1}\,e_{2}\,e_{3}$ ... $e_{11}$   |  $b_{1}\,b_{2}\,b_{3}$ ... $b_{52}$ | Lo que representa|
|------|------| ----| ---- |
| 0| 1 1 1 ... 1  | 0 0 0 ... 0  | $+ \infty$ |
| 1| 1 1 1 ... 1  | 0 0 0 ... 0  | $- \infty$ |
| 1| 1 1 1 ... 1  | x x x ... x  | NaN |

* **P=0**: Se utilza el formato **Non-normalized floating point number**, que son los números:

$$
\pm 0.b_{1} b_{2} \cdots{...} b_{52} \times 2^{-1022}
$$

Así, con esto el número 0 se representa de dos formas:

$$
+0: S = 0, e_{i} = 0, b_{i} = 0 \\
-0: S = 1, e_{i} = 0, b_{i} = 0
$$

```python
def next_float(f):
    #packing double-precision foat
    b = bs.pack('>d', f)

    #extracting mantisa as unsigned int
    #and adding up 1
    m = b[12:].uint
    m += 1

    #putting the result in his place
    b[12:] = m

    return b.float

def epsilon(f):
    next_f = next_float(f)
    return next_f - f

def to_binary(f):
    b = bs.pack('>d', f)
    b = b.bin
    #show sign + exponent + mantisa
    print(b[0]+' '+b[1:12]+ ' '+b[12:])
```

```python
int('01111111111',2) - 1023
```

    0

```python
to_binary(2**53+4)
```

    0 10000110100 0000000000000000000000000000000000000000000000000010

```python
"0 10000110100 0000000000000000000000000000000000000000000000000001|1"
```

    '0 10000110100 0000000000000000000000000000000000000000000000000001|1'

**Ejemplo 1**:

\begin{align}
1.00\dots0 \times 2^{0} + 1.00\dots0 \times 2^{-53} &=  \\
&= 1.0000000000000000000000000000000000000000000000000000|0 \times 2^{0}\\
&+ 0.0000000000000000000000000000000000000000000000000000|1 \times 2^{0}\\
&----------------------------------\\
&= 1.0000000000000000000000000000000000000000000000000000|1 \times 2^{0}
\end{align}

El resultado es igual a $1$ y esperabamos $1 + 2^{-53}$

**Ejemplo 2**:

\begin{align}
1.00\dots0 \times 2^{0} + 1.10\dots0 \times 2^{-52} &=  \\
&= 1.0000000000000000000000000000000000000000000000000000|0 \times 2^{0}\\
&+ 0.0000000000000000000000000000000000000000000000000001|1 \times 2^{0}\\
&----------------------------------\\
&= 1.0000000000000000000000000000000000000000000000000001|1 \times 2^{0}\\
&= 1.0000000000000000000000000000000000000000000000000010 \times 2^{0}
\end{align}

El resultado es igual a $1+2^{-51}$ y el valor esperado era $1+2^{-52}+2^{-53}$

**Perdida de significancia**: Ocurre cuando se restan valores muy cercano, pero debido a la representación flotante su diferencia da cero. Esto se corrige con una manipulación algebraica.

**Ejemplo**:

Veamos que sucede con:

$$
E1 = \frac{1-cos(x)}{sin(x)^{2}}
$$

```python
np.cos(10**-8)
```

    1.0

```python
def E1(x):
    return((1-np.cos(x))/np.sin(x)**2)
```

```python
E1(10**-8)
```

    0.0

Volvamos a escribir el problema:

$$
\frac{1-cos(x)}{sin(x)^{2}} \frac{1+cos(x)}{1+cos(x)} = \frac{1^{2} - cos(x)^{2}}{sin(x)^{2}(1+cos(x))}= \frac{sin(x)^{2}}{sin(x)^{2}(1+cos(x))} = \frac{1}{1+cos(x)} = E2
$$

```python
def E2(x):
    return(1/(1+np.cos(x)))
```

```python
E2(10**-9)
```

    0.5

**Ejemplo**:

$$
f(x) = \frac{1-(1-x)^{3}}{x}
$$

```python
def f(x):
    return((1-(1-x)**3)/x)
```

```python
f(10**-17)
```

    0.0

$$
f(x) = \frac{x^3 - 3 x^2 + 3 x}{x} = \frac{x(x^2 - 3 x + 3)}{x} = (x^2 - 3 x + 3)
$$

```python
def f2(x):
    return(x**2 - 3*x + 3)
```

```python
f2(10**-18)
```

    3.0

## Ejemplo

Generar la línea recta con la representación de todos los números considerando:
* 1 bit para el signo
* 2 bits para la mantisa
* 3 bits para el exponente

En este caso tenemos la siguiente representación:

| $\text{Signo}$ | $\text{Exponente}$   | $\text{Mantisa}$ |
|------|---------| ----|
| 1 bit| 2 bits | 2 bits |
|   $s$ | $e_{1}$$e_{2}$$e_3$ | $b_{1}$$b_{2}$ |

Con esto podemos representar números como:

\begin{equation}
    b = \pm 1.b_1b_2\times 2^p
\end{equation}

Para los valores exponente podemos representar $2^3=8$ valores, omitiendo el $0$ y $7$ para los casos especiales, solo podemos usar $[1, 2, \dots, 6]$. Calculando el bias obtenemos:

\begin{equation}
    \begin{split}
        K & = 2^{n-1} - 1 \\
         & = 2^2 - 1 \\
         & = 3,
    \end{split}
\end{equation}

es decir, $p\in[1-3, 6-3] = [-2,3]$

* ¿Cuál es el primer entero no representable para este caso? ¿En double precisión?

En el caso del primer entero no representable para el **ejemplo** tenemos:

\begin{equation}
    +1.11 \cdot 2^p,
\end{equation}
que es igual a

\begin{equation}
    (2^{0} + 2^{-1} + 2^{-2}) \cdot 2^p.
\end{equation}

Para $p=2$ tenemos:

\begin{equation}
    (2^{0} + 2^{-1} + 2^{-2}) \cdot 2^2 \\
    2^2 + 2^1 + 2^0,
\end{equation}

hasta acá todo bien, pero ahora para el caso $p=3$

\begin{equation}
    (2^{0} + 2^{-1} + 2^{-2}) \cdot 2^3 \\
    2^3 + 2^2 + 2^1,
\end{equation}

o sea no alcanzamos a representar $\mathbf{2^3+1}$.

En el caso de **doble precisión**, utilizando el mismo argumento anterior, veamos la siguiente situación:
\begin{equation}
    +1.1111\dots 1111\cdot 2^p,
\end{equation}

esto es
\begin{equation}
    (2^{0} + 2^{-1} + 2^{-2} + \dots + 2^{-52}) \cdot 2^p.
\end{equation}

Para $p=52$, tenemos

\begin{equation}
        (2^{0} + 2^{-1} + 2^{-2} + \dots + 2^{-52}) \cdot 2^{52} \\
        2^{52} + 2^{51} + 2^{50} + \dots + 2^{0}
\end{equation}

Hasta acá no tenemos problemas para representar enteros consecutivos gracias a $2^0$, pero veamos el siguiente caso. Para $p=53$, tenemos

\begin{equation}
        (2^{0} + 2^{-1} + 2^{-2} + \dots + 2^{-52}) \cdot 2^{53} \\
        2^{53} + 2^{52} + 2^{51} + \dots + 2^{1}
\end{equation}

En este último caso ya no podemos representar números consecutivos, puesto a que para el bit $b_{52}$ siempre podremos sumar como mínimo $2$. De esta forma el primer valor entero que no podríamos representar sería $\mathbf{2^{53} + 1}$.

* ¿Cuál es $\epsilon_{\text{mach}}$ para este caso?

El número $1$ se puede representar como $1.00 \times 2^0$ y el siguiente valor es $1.01 \times 2^0$, esto quiere decir que:

\begin{equation}
    \epsilon_{mach}=1.01 \times 2^0 - 1.00 \times 2^0 = 0.01 \times 2^0 = 1.00 \times 2^{-2},
\end{equation}

o sea $\epsilon_{mach}=2^{-2}$

* ¿Cuál es el menor número representable?

Utilizando esta representación, el número más pequeño que se puede representar con $p=-2$ es:

\begin{equation}
    \begin{split}
        b & = 0.01\times 2^{-2} \\
        b & = 2^{-4}
    \end{split}
\end{equation}

Recta de números representados por el ejemplo.

## Ejemplos experimentales

```python
def next_float(f):
    #packing double-precision foat
    b = bs.pack('>d', f)

    #extracting mantisa as unsigned int
    #and adding up 1
    m = b[12:].uint
    m += 1

    #putting the result in his place
    b[12:] = m

    return b.float

def epsilon(f):
    next_f = next_float(f)
    return next_f - f

def to_binary(f):
    b = bs.pack('>d', f)
    b = b.bin
    #show sign + exponent + mantisa
    print(b[0]+' '+b[1:12]+ ' '+b[12:])
```

```python
to_binary(53.7)
```

    0 10000000100 1010110110011001100110011001100110011001100110011010

```python
epsilon(1)
```

    2.220446049250313e-16

```python
#values between 10**-32 and 10**+32
values = np.array([10**i for i in range(-32,32)]).astype(float)

#corresponding epsilons
vepsilon = np.vectorize(epsilon)
eps = vepsilon(values)
```

```python
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(values, eps,'.',markersize=20)
plt.xlabel('Values')
plt.ylabel('Corresponding Epsilons')
plt.title('Epsilons v/s Values')
plt.grid(True)
plt.subplot(1, 2, 2)
plt.loglog(values, eps,'.')
plt.xlabel('Values')
plt.ylabel('Corresponding Epsilons')
plt.title('Epsilons v/s Values')
plt.grid(True)
plt.tight_layout()
plt.show()
```

* a) Si trabajamos con 11 bits en el exponente tenemos: $2^{11}$ = 2048 combinaciones de números. Por lo tanto trabajariamos entre [0,2047]. Pero como existen casos especiales para el 0 y el 2047 el rango en el que nos podriamos mover sería de [$1\dots2046$]. Calculando el shift utilizando la formula $2^{n-1} - 1$ obtenemos que es : 1023 quedandonos así el el rango de los exponentes igual a [$-1022\dots1023$].

    Realizando el mismo estudio pero para 6 bits dada la perdida de 5 bits trabajariamos con un total de $2^{6}$ combinaciones de números. Por lo tanto el rango en que se mueve el exponente es igual a [0,63]. Descartando los casos especiales obtenemos que el rango es [1,62]. Calculando el shift ($2^{6-1}-1$) = 31 obtenemos que el rango de los exponentes queda como [-30,31]. Este esta dentro del rango que necesita tony stark por lo que recomiendo asignar los bits sobrantes a la mantisa para tener mas presición.

* b) $2^{-47}$. Ya que el ultimo bit de la mantisa es el 47. ($1.00\dots\underbrace{1}_{47} x 2^{0}- 1.00\dots\underbrace{1}_{47} x 2^{0}$)

* c) Utilizando notación subnormal el menor número representable menor que 0 es el $2^{-47} 2^{-1022}$. Ya que este se representa como $0.00\dots\underbrace{0}_{47} x 2^{-1022}$

* d) El ($2^{31} + 2^{30} \dots 2^{0}$) + 1. Esto es debido a que cuando estamos trabajando con el exponente 31 que es el maximo exponente que podemos representar el ultimo entero exacto que podemos representar es $1.\underbrace{1111111111111111111111111111111}_{31\,1s}\underbrace{00000000000000000000}_{20\,0s} x 2^{31}$. Notar que todos los bits que estan en 0 representan números decimales, en especifico representan la suma de las potencias ($2^{-1}+2^{-2}\dots 2^{-21})$. Esto equivale a una serie geometrica no infinita.

$$
\displaystyle \sum_{k=0}^{n} r^{k} = \frac{1-r^{n+1}}{1-r} = \frac{1-(\frac{1}{2})^{21}}{1-\frac{1}{2}} = 1.9999999
$$

    A este hay que restarle 1 por que no estoy considerando la suma desde 1 sino que desde $2^{-1}$ por lo tanto el resultado es $0.9999\neq 1$. Esto significa que para poder representar $2^{31} + 2^{30} \dots 2^{0}$) + 1 debemos hacer uso de $2^{32}$ pero no es posible representar.

e) Utilizando notación subnormal el menor número representable menor que 0 es el $2^{-52} 2^{-31}$. Ya que este se representa como $0.00\dots\underbrace{1}_{52} x 2^{-31}$

| $\text{Signo}$ | $\text{Exponente}$   | $\text{Mantisa}$ |
|------|---------| ----|
| 1 bit| 3 bits | 4 bits |
|   $s$ | $e_{1}$$e_{2}$$e_3$ | $b_{1}$$b_{2}$$b_{3}$ $b_{4}$  |

1) $2^{-4}$. Ya que el ultimo bit de la mantisa es el 4. ($1.000\underbrace{1}_{4} x 2^{0}- 1.000\underbrace{0}_{4} x 2^{0}$)

2) Realizando el shift al itervalo [1,6] obtenemos [-2,3]. Por lo tanto el menor número representable es $2^{-4}\,2^{-2} = 2^{-6}$

3) Primero lo llevaremos a la forma normal.

$$
1.1100|1100 \dots \times 2^{-1}
$$

Utilizando las reglas de la IEEE tenemos:

$$
1.1101 \times 2^{-1}
$$

Ese número en decimal equivale a ($2^{-1}+2^{-2}+2^{-3}+2^{-5}$) = $0.90625$.

Calculando el error relativo tenemos:

$$
\frac{|0.90625-0.9|}{0.9} = 0.00694444444444442 < \frac{1}{2} 2^{-4} = 0.03125
$$

4) **Correción**

Caso 1:

$$
l1 = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4} \\
l2 = \frac{1}{16} \cdot \frac{1}{16} = \frac{1}{2^{8}}\\
$$

$$
l3  =
\begin{align*}
1&.0000 \times 2^{-2}\\
+1&.0000 \times 2^{-8}\\
\end{align*}
$$

Eso es equivalente a:

$$
\begin{align*}
1&.000000000 \times 2^{-2}\\
+0&.000001000 \times 2^{-2}\\
-&--------\\
1.&000001000 \times 2^{-2}
\end{align*}
$$

Pero como podemos almacenar solo 4 digitos en la mantisa tenemos como resultado:

$$
1.0000 \times 2^{-2} = \frac{1}{4}
$$
Finalmente out es $\frac{1}{2}$ luego de aplicar la raiz.

Caso 2:
$$
M = \frac{1}{2}\\
m = \frac{1}{16}\\
\frac{m}{M} = \frac{\frac{1}{16}}{\frac{1}{2}} = \frac{1}{8}\\
l4 = \frac{1}{8}\cdot\frac{1}{8} = \frac{1}{64} = \frac{1}{2^6}
$$

$$
l5 =
\begin{align*}
1&.0000\times2^{0}\\
+0&.0001\times2^{-2}
\end{align*}
$$

Eso es equivalente a:
$$
\begin{align}
1&.000000\times2^{0}\\
+0&.000001\times2^{0}\\
-&------\\
1&.000001 \times2^{0}
\end{align}
$$

Pero como podemos almacenar solo 4 digitos en la mantisa tenemos como resultado:

$$
1.0000 \times 2^{0}
$$
Luego:
$$
l6 = \sqrt{1} = 1
$$

Finalmente out corresponde ser a $M \times l6 = \frac{1}{2}$

```python

```

```python
2**3 + 2**-3
```

    8.125

```python
2**-3
```

    0.125

```python

```


# Module 9: Anomaly Detection

The following tutorial contains Python examples for detecting anomalies (outliers) from data. You should refer to Chapters 9 of the "Introduction to Data Mining" book to understand some of the concepts introduced in this tutorial. The notebook can be downloaded from http://www.cse.msu.edu/~ptan/dmbook/tutorials/tutorial9/tutorial9.ipynb.

Anomaly detection is the task of identifying instances whose characteristics differ significantly from the rest of the data. In this tutorial, we will provide examples of applying different anomaly detection techniques using Python and its library packages.

Read the step-by-step instructions below carefully. To execute the code, click on the corresponding cell and press the SHIFT-ENTER keys simultaneously.

## 9.1 Using Parametric Models

This approach assumes that the majority of the data instances are governed by some well-known probability distribution, e.g., Binomial or Gaussian distribution. Anomalies can then detected by seeking for observations that do not fit the overall distribution of the data.

In this example, our goal is to detect anomalous changes in the daily closing prices of various stocks. The input data *stocks.csv* contains the historical closing prices of stocks for 3 large corporations (Microsoft, Ford Motor Company, and Bank of America).

```python
import pandas as pd

stocks = pd.read_csv('datasets/stocks.csv', header='infer' )
stocks.index = stocks['Date']
stocks = stocks.drop(['Date'],axis=1)
stocks.head()
```

      MSFT
      F
      BAC

      Date

      1/3/2007
      29.860001
      7.51
      53.330002

      1/4/2007
      29.809999
      7.70
      53.669998

      1/5/2007
      29.639999
      7.62
      53.240002

      1/8/2007
      29.930000
      7.73
      53.450001

      1/9/2007
      29.959999
      7.79
      53.500000

We can compute the percentage of changes in the daily closing price of each stock as follows:
\begin{equation}
\Delta(t) = 100 \times \frac{x_t - x_{t-1}}{x_{t-1}}
\end{equation}

where $x_t$ denotes the price of a stock on day $t$ and $x_{t-1}$ denotes the price on its previous day, $t-1$.

```python
import numpy as np

N,d = stocks.shape
delta = pd.DataFrame(100*np.divide(stocks.iloc[1:,:].values-stocks.iloc[:N-1,:].values, stocks.iloc[:N-1,:].values),
                    columns=stocks.columns, index=stocks.iloc[1:].index)
delta.head()
```

      MSFT
      F
      BAC

      Date

      1/4/2007
      -0.167455
      2.529960
      0.637532

      1/5/2007
      -0.570278
      -1.038961
      -0.801185

      1/8/2007
      0.978411
      1.443570
      0.394438

      1/9/2007
      0.100231
      0.776197
      0.093543

      1/10/2007
      -1.001332
      -0.770218
      0.149536

We can plot the distribution of the percentage daily changes in stock price.

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
%matplotlib inline

fig = plt.figure(figsize=(8,5)).gca(projection='3d')
fig.scatter(delta.MSFT,delta.F,delta.BAC)
fig.set_xlabel('Microsoft')
fig.set_ylabel('Ford')
fig.set_zlabel('Bank of America')
plt.show()
```

Assuming the data follows a multivariate Gaussian distribution, we can compute the mean and covariance matrix of the 3-dimensional data as follows

```python
meanValue = delta.mean()
covValue = delta.cov()
print(meanValue)
print(covValue)
```

    MSFT    0.045003
    F       0.061374
    BAC     0.033351
    dtype: float64
              MSFT         F        BAC
    MSFT  3.191674  2.136351   2.788870
    F     2.136351  8.524944   4.997405
    BAC   2.788870  4.997405  13.770761

To determine the anomalous trading days, we can compute the Mahalanobis distance between the percentage of price change on each day against the mean percentage of price change.
\begin{equation}
\textrm{Mahalanobis}(x) = (x - \bar{x}) \Sigma^{-1}(x - \bar{x})^T
\end{equation}
where $x$ is assumed to be a row vector.

See Equation 9.4 in Section 9.3.1 for more information about using Mahalanobis distance for detecting anomalies in multivariate Gaussian distribution.

```python
from numpy.linalg import inv

X = delta.values
S = covValue.values
for i in range(3):
    X[:,i] = X[:,i] - meanValue[i]

def mahalanobis(row):
    return np.matmul(row,S).dot(row)

anomaly_score = np.apply_along_axis( mahalanobis, axis=1, arr=X)

fig = plt.figure(figsize=(10,6))
ax = fig.add_subplot(111, projection='3d')
p = ax.scatter(delta.MSFT,delta.F,delta.BAC,c=anomaly_score,cmap='jet')
ax.set_xlabel('Microsoft')
ax.set_ylabel('Ford')
ax.set_zlabel('Bank of America')
fig.colorbar(p)
plt.show()
```

The top-2 anomalies are shown as a brown point in the figure above. The highest anomaly corresponds to the day in which the prices for all 3 stocks increase significantly whereas the second highest anomaly corresponds  to the day in which all 3 stocks suffer a large percentage drop in their closing prices. We can examine the dates associated with the top-2 highest anomaly scores as follows.

```python
anom = pd.DataFrame(anomaly_score, index=delta.index, columns=['Anomaly score'])
result = pd.concat((delta,anom), axis=1)
result.nlargest(2,'Anomaly score')
```

      MSFT
      F
      BAC
      Anomaly score

      Date

      4/9/2009
      2.456295
      7.280398
      35.235771
      20691.465032

      10/7/2008
      -6.789282
      -20.928583
      -26.259300
      20471.095209

Note that the sharp drop in the stock prices on October 7, 2008 coincide with the beginning of the global financial crisis (https://en.wikipedia.org/wiki/Global_financial_crisis_in_October_2008) while the increase in the stock prices on April 9, 2009.

```python
fig, (ax1,ax2) = plt.subplots(nrows=1, ncols=2, figsize=(15,6))

ts = delta[440:447]
ts.plot.line(ax=ax1)
ax1.set_xticks(range(7))
ax1.set_xticklabels(ts.index)
ax1.set_ylabel('Percent Change')

ts = delta[568:575]
ts.plot.line(ax=ax2)
ax2.set_xticks(range(7))
ax2.set_xticklabels(ts.index)
ax2.set_ylabel('Percent Change')
```

## 9.2 Using Distance-based Approach

This is a model-free anomaly detection approach as it does not require constructing an explicit model of the normal class to determine the anomaly score of data instances. The example code shown below employs the k-nearest neighbor approach to calculate anomaly score. Specifically, a normal instance is expected to have a small distance to its k-th nearest neighbor whereas an anomaly is likely to have a large distance to its k-th nearest neighbor. In the example below, we apply the distance-based approach with k=4 to identify the anomalous trading days from the stock market data described in the previous section.

```python
from sklearn.neighbors import NearestNeighbors
import numpy as np
from scipy.spatial import distance

knn = 4
nbrs = NearestNeighbors(n_neighbors=knn, metric=distance.euclidean).fit(delta.values)
distances, indices = nbrs.kneighbors(delta.values)

anomaly_score = distances[:,knn-1]

fig = plt.figure(figsize=(10,6))
ax = fig.add_subplot(111, projection='3d')
p = ax.scatter(delta.MSFT,delta.F,delta.BAC,c=anomaly_score,cmap='jet')
ax.set_xlabel('Microsoft')
ax.set_ylabel('Ford')
ax.set_zlabel('Bank of America')
fig.colorbar(p)
plt.show()
```

The results are slightly different than the one shown in Section 9.1 since we have used Euclidean distance (instead of Mahalanobis distance) to detect the anomalies. We can examine the dates associated with the top-5 highest anomaly scores as follows.

```python
anom = pd.DataFrame(anomaly_score, index=delta.index, columns=['Anomaly score'])
result = pd.concat((delta,anom), axis=1)
result.nlargest(5,'Anomaly score')
```

      MSFT
      F
      BAC
      Anomaly score

      Date

      10/13/2008
      18.559648
      20.039128
      9.166457
      15.642827

      11/26/2008
      2.456248
      29.456698
      4.223406
      14.212749

      10/7/2008
      -6.789282
      -20.928583
      -26.259300
      13.751302

      11/28/2008
      -1.362724
      25.054905
      5.280972
      13.139586

      9/30/2008
      6.672314
      24.638866
      15.669129
      12.599739

```python
fig = plt.figure(figsize=(10,4))

ax = fig.add_subplot(111)
ts = delta[445:452]
ts.plot.line(ax=ax)
ax.set_xticks(range(7))
ax.set_xticklabels(ts.index)
ax.set_ylabel('Percent Change')
```

## 9.3 Summary

This tutorial illustrates examples applying an anomaly detection approach to a multivariate time series data. We consider two approaches, one based on a parametric statistical approach using multivariate Gaussian while the other is a nonparametric distance-based approach using k-nearest neighbor.


# Prior definition

In this notebook we demonstrate how to include prior knowledge into a parameter inference problem, in particular how to define (log-)priors for parameters. If you want to maximize your posterior distribution, you need to define

* A (negative log-)likelihood
* A (log-)prior

The posterior is then built as an `AggregatedObjective`. If you import a problem via `PEtab` and the priors are contained in the parameter table, the definition of priors is done automatically.

**CAUTION**: The user needs to specify the **negative** _log-likelihood_, while the _log-prior_ is internally mulitplied by -1.

```python
# install if not done yet
# %pip install pypesto --quiet
```

```python
import numpy as np
import scipy as sp

import pypesto
```

## Example: Rosenbrock Banana

We will use the Rosenbrock Banana

\begin{align}
f(x, \theta) = \sum_{i=1}^{N} \underbrace{100 \cdot(x_{i}-x_{i-1}^2)^2}_{\text{"negative log-likelihood"}} + \underbrace{(x_{i-1}-1)^2}_{\text{"Gaussian log-prior"}}
\end{align}

as an example. Here we interpret the first term as the _negative log-likelihood_ and the second term as Gaussian _log-prior_ with mean $1$ and standard deviation $1/\sqrt{2}$.

Note that the second term is only equivalent to the negative log-distribution of a Gaussian up to a constant.

### Define the negative log-likelihood

```python
n_x = 5

def rosenbrock_part_1(x):
    """
    Calculate obj. fct + gradient of the "likelihood" part.
    """
    obj = sum(100.0 * (x[1:] - x[:-1] ** 2.0) ** 2.0)

    grad = np.zeros_like(x)
    grad[:-1] += -400 * (x[1:] - x[:-1] ** 2.0) * x[:-1]
    grad[1:] += 200 * (x[1:] - x[:-1] ** 2.0)

    return (obj, grad)

neg_log_likelihood = pypesto.Objective(fun=rosenbrock_part_1, grad=True)
```

### Define the log-prior

A prior on an individual parameter is defined in a `prior_dict`, which contains the following key-value pairs:

* `index`: Index of the parameter
* `density_fun`: (Log-)posterior. (Scalar function!)
* `density_dx`: d/dx (Log-)posterior (optional)
* `density_ddx`: d^2/dx^2 (Log-)posterior (optional)

A `prior_dict` can be either obtained by `get_parameter_prior_dict` for several common priors, or defined by the user.

```python
from pypesto.objective.priors import get_parameter_prior_dict

# create a list of prior dicts...
prior_list = []
mean = 1
std_dev = 1 / np.sqrt(2)

for i in range(n_x - 1):
    prior_list.append(get_parameter_prior_dict(i, "normal", [mean, std_dev]))

# create the prior
neg_log_prior = pypesto.objective.NegLogParameterPriors(prior_list)
```

### Define the negative log-posterior and the problem

The negative log-posterior is defined as an `AggregatedObjective`. Since optimization/visualization is not the main focus of this notebook, the reader is referred to other examples for a more in-depth presentation of these.

```python
neg_log_posterior = pypesto.objective.AggregatedObjective(
    [neg_log_likelihood, neg_log_prior]
)

lb = -5 * np.ones((n_x, 1))
ub = 5 * np.ones((n_x, 1))

problem = pypesto.Problem(
    objective=neg_log_posterior,
    lb=lb,
    ub=ub,
)
```

### Optimize

```python
import pypesto.optimize as optimize

result = optimize.minimize(problem=problem, n_starts=10, filename=None)
```

    100%|█████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 41.26it/s]

### Some basic visualizations

```python
import pypesto.visualize as visualize

visualize.waterfall(result, size=(15, 6))

# parallel coordinates plot for best 5 fits
visualize.parameters(result, start_indices=5);
```


```python
from IPython.display import Image
```

# Labeling: Fixed Horizon

_*Fig. 1:*_ Distribution of labels for standardized returns on closing prices of SPY in the time period from Jan 2008 to July 2016 using a 20 day rolling window for the standard deviation.

## Abstract

Fixed Horizon is a classification labelling technique in which time-indexed data is labeled according to whether it exceeds, falls in between, or is less than a threshold. This method is most commonly used with time bars, but also be applied to any time-indexed data such as dollar or volume bars. The subsequent labelled data can then be used as training and test data for ML algorithms.

The following paper outlines the technique in greater depth, and is followed closely in this notebook tutorial: ["Classification-based Financial Markets Prediction using Deep Neural Networks"](https://arxiv.org/abs/1603.08604) by _Dixon et al._ (2016).

## How it works

Fixed time horizon is a common method used in labeling financial data, usually applied on time bars. The forward rate of return relative
to $t_0$ over time horizon $h$ is calculated as follows:

   $$r_{t0,t1} = \frac{p_{t1}}{p_{t0}} - 1$$

Where $t_1 = t_0 + h$ is the time bar index after a fixed horizon of $h$ ticks have passed, and $p_{t0}, p_{t1}$
are prices at times $t_0, t_1$. This method assigns a label based on comparison of rate of return to a threshold $\tau$
\[Lopez de Prado 2018\]:

$$
     \begin{equation}
     \begin{split}
       L_{t0, t1} = \begin{cases}
       -1 &\ \text{if} \ \ r_{t0, t1} < -\tau\\
       0 &\ \text{if} \ \ -\tau \leq r_{t0, t1} \leq \tau\\
       1 &\ \text{if} \ \ r_{t0, t1} > \tau
       \end{cases}
     \end{split}
     \end{equation}
$$

Though time bars are the most common format for financial data, there can be potential problems with over reliance on time bars. First,
time bars exhibit high seasonality, as trading behavior may be quite different at the open or close versus midday; thus it will not be
informative to apply the same thershold on non-uniform distribution. Solutions include applying the fixed horizon method to tick or
volume bars instead of time bars, using data sampled at the same time every day (e.g. closing prices) or inputting a dynamic threshold
as a pd.Series corresponding to the times in the dataset. Another solution is to apply the labels to data based on standardized returns \[Lopez de Prado, 2020: p.66\]. In this case, the data is scaled by mean and standard deviation based on a given rolling window before labels are applied.

---
## Examples of use

```python
import numpy as np
import pandas as pd
import yfinance as yf

from mlfinlab.labeling import fixed_time_horizon

import matplotlib.pyplot as plt
```

```python
# Load price data
msft = yf.Ticker('MSFT')
msft_df = msft.history(start='2010-1-1', end ='2020-5-18')

# Subset closing prices
close = msft_df['Close']
close.head()
```

    Date
    2010-01-04    24.23
    2010-01-05    24.23
    2010-01-06    24.09
    2010-01-07    23.84
    2010-01-08    24.00
    Name: Close, dtype: float64

### Labeling Financial Time Series

```python
# Getting labels for a constant threshold of 1%. Will return 1 if the daily return is greater than 1%,
# -1 if less than -1%, and  0 if in between.
bars1 = fixed_time_horizon(close, 0.01, look_forward=1)
bars1.value_counts()
```

     0.0    1596
     1.0     560
    -1.0     453
    dtype: int64

```python
ax1 = bars1.value_counts().reindex([-1, 0, 1]).plot(kind='bar', x='label', y='frequency')
ax1.set_xlabel('Label')
ax1.set_ylabel('Frequency')
```

Predictably, lowering the threshold results in fewer 0 labels.

```python
bars2 = fixed_time_horizon(close, 0.001, look_forward=1)
ax2 = bars2.value_counts().reindex([-1, 0, 1]).plot(kind='bar', x='label', y='frequency')
ax2.set_xlabel('Label')
ax2.set_ylabel('Frequency')
```

### Labeling With Dynamic Threshold

A major problem with the fixed time horizon method is the __seasonality__ of the data. Time bars in the middle of the trading day would look very different than those at the open or close. One way to get around this is to only use data which has been collected at the same time per day, such as close prices. Another method is to apply a dynamic threshold which can adjust for such differences.

We will demonstrate two methods of setting a dynamic threshold:
1. Using a rolling standard deviation
2. Using SPY as the market return

```python
# Get rolling standard deviation
msft_returns = close.pct_change(periods=1).shift(-1)
rolling_std = msft_returns.rolling(10).std()
```

```python
bars3 = fixed_time_horizon(close, rolling_std, look_forward=1)
bars3.value_counts()
```

     0.0    1780
     1.0     463
    -1.0     357
    dtype: int64

```python
bars3 = fixed_time_horizon(close, 0.001, look_forward=1)
ax3 = bars3.value_counts().reindex([-1, 0, 1]).plot(kind='bar', x='label', y='frequency')
ax3.set_xlabel('Label')
ax3.set_ylabel('Frequency')
```

```python
# Threshold based on market returns
spy = yf.Ticker('SPY')
spy_df = spy.history(start='2010-1-1', end ='2020-5-18')
market_close = spy_df['Close']

market_returns = market_close.pct_change(1).shift(-1)
market_returns.head()
```

    Date
    2010-01-04    0.002602
    2010-01-05    0.000757
    2010-01-06    0.004213
    2010-01-07    0.003335
    2010-01-08    0.001394
    Name: Close, dtype: float64

```python
close.pct_change(periods=1).shift(-1)
```

    Date
    2010-01-04    0.000000
    2010-01-05   -0.005778
    2010-01-06   -0.010378
    2010-01-07    0.006711
    2010-01-08   -0.012917
                    ...
    2020-05-11   -0.022661
    2020-05-12   -0.015110
    2020-05-13    0.004351
    2020-05-14    0.014553
    2020-05-15         NaN
    Name: Close, Length: 2610, dtype: float64

```python
bars4 = fixed_time_horizon(close, market_returns, look_forward=1)
bars4.value_counts()
```

     1.0    1319
    -1.0     837
     0.0     453
    dtype: int64

```python
ax4 = bars4.value_counts().reindex([-1, 0, 1]).plot(kind='bar', x='label', y='frequency')
ax4.set_xlabel('Label')
ax4.set_ylabel('Frequency')
```

### Labeling with Standardization

Scaling each return by the mean and standard deviation is also a potential solution in getting around the non-homogeneaity of the data, as it could adjust for volatility differences between different bars. If standardization is desires, a rolling window must be given.

```python
bars5 = fixed_time_horizon(close, 1, look_forward=1, standardized=True, window=5)
bars5.value_counts()
```

     0.0    1787
     1.0     415
    -1.0     403
    dtype: int64

```python
ax5 = bars5.value_counts().reindex([-1, 0, 1]).plot(kind='bar', x='label', y='frequency')
ax5.set_xlabel('Label')
ax5.set_ylabel('Frequency')
```

### Error Handling

Warnings will be raised if look_forward or window is greater than the length of the data. In this case, the function will still run, but all labels will be NaN. An error is raised if standardized is set to True but the window is not specified correctly (as an int).

```python
fixed_time_horizon(close, 0.01, look_forward=99999999)
```

    c:\users\ruifan\mlfinlab\mlfinlab\labeling\fixed_time_horizon.py:45: UserWarning: look_forward period is greater than the length of the Series. All labels will be NaN.
      UserWarning)

    Date
    2010-01-04   NaN
    2010-01-05   NaN
    2010-01-06   NaN
    2010-01-07   NaN
    2010-01-08   NaN
                  ..
    2020-05-11   NaN
    2020-05-12   NaN
    2020-05-13   NaN
    2020-05-14   NaN
    2020-05-15   NaN
    Length: 2610, dtype: float64

```python
fixed_time_horizon(close, 0.01, look_forward=1, standardized=True, window=99999999)
```

    c:\users\ruifan\mlfinlab\mlfinlab\labeling\fixed_time_horizon.py:53: UserWarning: window is greater than the length of the Series. All labels will be NaN.
      warnings.warn('window is greater than the length of the Series. All labels will be NaN.', UserWarning)

    Date
    2010-01-04   NaN
    2010-01-05   NaN
    2010-01-06   NaN
    2010-01-07   NaN
    2010-01-08   NaN
                  ..
    2020-05-11   NaN
    2020-05-12   NaN
    2020-05-13   NaN
    2020-05-14   NaN
    2020-05-15   NaN
    Length: 2610, dtype: float64

```python
fixed_time_horizon(close, 0.01, look_forward=1, standardized=True)
```

---
## Conclusion

This notebook presents the fixed horizon method, a simple method of labeling data for later use in machine learning algorithms. In this process:
 - Forward rates of return for assets are calculated based on tick data, usually in the form of time bars.
 - The forward return rate is compared to a threshold for the return. The threshold can be a constant, or a dynamic series corresponding to every timestamp in the data.
 - Each observation is compared to the threshold, and is labeled -1 if it is below -threshold, 0 if it is between -threshold and +threshold, and 1 if it is greater than +threshold.

In Dixon's paper, the data is labelled using this method. Subsequently a neural network is trained to to predict the label based on the data. A simple trading strategy in which a long position is taken if a ticker is classified as 1, and short if it's classified at -1, can then be applied.

## References

1. Dixon, M., Klabjan, D. and Jin Hoon, B., 2020. Classification-based Financial Markets Prediction using Deep Neural Networks. arXiv, [online] Available at: https://arxiv.org/abs/1603.08604

2. López de Prado, M., 2018. Advances In Financial Machine Learning. pp.43-44.

3. López de Prado, M., 2020. Machine Learning For Asset Managers. pp.65-66.

jupyter-notebooks/hudson-and-thames-quant/Labelling/Labels Fixed Horizon/Fixed Time Horizon.ipynb