# **Save this file as studentid1_studentid2_lab2.ipynb**, please check this suffix when you upload your lab, especially when you have multiple copy's in the same folder!
(Your student-id is the number shown on your student card.)

E.g. if you work with 3 people, the notebook should be named:
12301230_3434343_1238938934_lab2.ipynb.

**This will be parsed by a regexp, so please double check your filename.**

Before you turn this problem in, please make sure everything runs correctly. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All). Note, that **you are not allowed to use Google Colab**.

**Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your names and email adresses below.**

```python
NAME = "Radu Sibechi"
NAME2 = "Nicu-Traian Vidrascu"
EMAIL = "ntvidrascu@gmail.com"
EMAIL2 = "radu.sibechi@yahoo.com"

```

# Lab 2: Classification

### Machine Learning 1, November 2018

Notes on implementation:

* You should write your code and answers in this IPython Notebook: http://ipython.org/notebook.html. If you have problems, please contact your teaching assistant.
* Please write your answers right below the questions.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Use the provided test cells to check if your answers are correct
* **Make sure your output and plots are correct before handing in your assignment with Kernel -> Restart & Run All**

* **If possible, all your implementations should be vectorized and rely on loops as little as possible. Therefore for some questions, we give you a maximum number of loops that are necessary for an efficient implementation. This number refers to the loops in this particular function and does not count the ones in functions that are called from the function. You should not go above this number for the maximum number of points.**

$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bm}{\mathbf{m}}$
$\newcommand{\bb}{\mathbf{b}}$
$\newcommand{\bS}{\mathbf{S}}$
$\newcommand{\ba}{\mathbf{a}}$
$\newcommand{\bz}{\mathbf{z}}$
$\newcommand{\bv}{\mathbf{v}}$
$\newcommand{\bq}{\mathbf{q}}$
$\newcommand{\bp}{\mathbf{p}}$
$\newcommand{\bh}{\mathbf{h}}$
$\newcommand{\bI}{\mathbf{I}}$
$\newcommand{\bX}{\mathbf{X}}$
$\newcommand{\bT}{\mathbf{T}}$
$\newcommand{\bPhi}{\mathbf{\Phi}}$
$\newcommand{\bW}{\mathbf{W}}$
$\newcommand{\bV}{\mathbf{V}}$

```python
%pylab inline
plt.rcParams["figure.figsize"] = [9,5]

import time
start = time.time()
```

    Populating the interactive namespace from numpy and matplotlib

```python
# This cell makes sure that you have all the necessary libraries installed

import sys
import platform
from importlib.util import find_spec, module_from_spec

def check_newer_version(version_inst, version_nec):
    version_inst_split = version_inst.split('.')
    version_nec_split = version_nec.split('.')
    for i in range(min(len(version_inst_split), len(version_nec_split))):
        if int(version_nec_split[i]) > int(version_inst_split[i]):
            return False
        elif int(version_nec_split[i]) < int(version_inst_split[i]):
            return True
    return True

module_list = [('jupyter', '1.0.0'),
               ('matplotlib', '2.0.2'),
               ('numpy', '1.13.1'),
               ('python', '3.6.2'),
               ('sklearn', '0.19.0'),
               ('scipy', '0.19.1'),
               ('nb_conda', '2.2.1')]

packages_correct = True
packages_errors = []

for module_name, version in module_list:
    if module_name == 'scikit-learn':
        module_name = 'sklearn'
    if module_name == 'pyyaml':
        module_name = 'yaml'
    if 'python' in module_name:
        python_version = platform.python_version()
        if not check_newer_version(python_version, version):
            packages_correct = False
            error = f'Update {module_name} to version {version}. Current version is {python_version}.'
            packages_errors.append(error)
            print(error)
    else:
        spec = find_spec(module_name)
        if spec is None:
            packages_correct = False
            error = f'Install {module_name} with version {version} or newer, it is required for this assignment!'
            packages_errors.append(error)
            print(error)
        else:
            x =__import__(module_name)
            if hasattr(x, '__version__') and not check_newer_version(x.__version__, version):
                packages_correct = False
                error = f'Update {module_name} to version {version}. Current version is {x.__version__}.'
                packages_errors.append(error)
                print(error)

try:
    from google.colab import drive
    packages_correct = False
    error = """Please, don't use google colab!
It will make it much more complicated for us to check your homework as it merges all the cells into one."""
    packages_errors.append(error)
    print(error)
except:
    pass

packages_errors = '\n'.join(packages_errors)
```

# Part 1. Multiclass logistic regression

Scenario: you have a friend with one big problem: she's completely blind. You decided to help her: she has a special smartphone for blind people, and you are going to develop a mobile phone app that can do _machine vision_ using the mobile camera: converting a picture (from the camera) to the meaning of the image. You decide to start with an app that can read handwritten digits, i.e. convert an image of handwritten digits to text (e.g. it would enable her to read precious handwritten phone numbers).

A key building block for such an app would be a function `predict_digit(x)` that returns the digit class of an image patch $\bx$. Since hand-coding this function is highly non-trivial, you decide to solve this problem using machine learning, such that the internal parameters of this function are automatically learned using machine learning techniques.

The dataset you're going to use for this is the MNIST handwritten digits dataset (`http://yann.lecun.com/exdb/mnist/`). You can download the data with scikit learn, and load it as follows:

```python
from sklearn.datasets import fetch_mldata
import os
# Fetch the data
try:
    mnist = fetch_mldata('MNIST original', data_home='.')
except Exception:
    raise FileNotFoundError('Please download mnist-original.mat from Canvas and put it in %s/mldata' % os.getcwd())
data, target = mnist.data, mnist.target.astype('int')
# Shuffle
indices = np.arange(len(data))
np.random.seed(123)
np.random.shuffle(indices)
data, target = data[indices].astype('float32'), target[indices]

# Normalize the data between 0.0 and 1.0:
data /= 255.

# Split
x_train, x_valid, x_test = data[:50000], data[50000:60000], data[60000: 70000]
t_train, t_valid, t_test = target[:50000], target[50000:60000], target[60000: 70000]
```

MNIST consists of small 28 by 28 pixel images of written digits (0-9). We split the dataset into a training, validation and testing arrays. The variables `x_train`, `x_valid` and `x_test` are $N \times M$ matrices, where $N$ is the number of datapoints in the respective set, and $M = 28^2 = 784$ is the dimensionality of the data. The second set of variables `t_train`, `t_valid` and `t_test` contain the corresponding $N$-dimensional vector of integers, containing the true class labels.

Here's a visualisation of the first 8 digits of the trainingset:

```python
def plot_digits(data, num_cols, targets=None, shape=(28,28)):
    num_digits = data.shape[0]
    num_rows = int(num_digits/num_cols)
    for i in range(num_digits):
        plt.subplot(num_rows, num_cols, i+1)
        plt.imshow(data[i].reshape(shape), interpolation='none', cmap='Greys')
        if targets is not None:
            plt.title(int(targets[i]))
        plt.colorbar()
        plt.axis('off')
    plt.tight_layout()
    plt.show()

plot_digits(x_train[0:40000:5000], num_cols=4, targets=t_train[0:40000:5000])
```

In _multiclass_ logistic regression, the conditional probability of class label $j$ given the image $\bx$ for some datapoint is given by:

$ \log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$

where $\log q_j = \bw_j^T \bx + b_j$ (the log of the unnormalized probability of the class $j$), and $Z = \sum_k q_k$ is the normalizing factor. $\bw_j$ is the $j$-th column of $\bW$ (a matrix of size $784 \times 10$) corresponding to the class label, $b_j$ is the $j$-th element of $\bb$.

Given an input image, the multiclass logistic regression model first computes the intermediate vector $\log \bq$ (of size $10 \times 1$), using $\log q_j = \bw_j^T \bx + b_j$, containing the unnormalized log-probabilities per class.

The unnormalized probabilities are then normalized by $Z$ such that $\sum_j p_j = \sum_j \exp(\log p_j) = 1$. This is done by $\log p_j = \log q_j - \log Z$ where $Z = \sum_i \exp(\log q_i)$. This is known as the _softmax_ transformation, and is also used as a last layer of many classifcation neural network models, to ensure that the output of the network is a normalized distribution, regardless of the values of second-to-last layer ($\log \bq$)

**Warning**: when computing $\log Z$, you are likely to encounter numerical problems. Save yourself countless hours of debugging and learn the [log-sum-exp trick](https://www.xarg.org/2016/06/the-log-sum-exp-trick-in-machine-learning/ "Title").

The network's output $\log \bp$ of size $10 \times 1$ then contains the conditional log-probabilities $\log p(t = j \;|\; \bx, \bb, \bW)$ for each digit class $j$. In summary, the computations are done in this order:

$\bx \rightarrow \log \bq \rightarrow Z \rightarrow \log \bp$

Given some dataset with $N$ independent, identically distributed datapoints, the log-likelihood is given by:

$ \mathcal{L}(\bb, \bW) = \sum_{n=1}^N \mathcal{L}^{(n)}$

where we use $\mathcal{L}^{(n)}$ to denote the partial log-likelihood evaluated over a single datapoint. It is important to see that the log-probability of the class label $t^{(n)}$ given the image, is given by the $t^{(n)}$-th element of the network's output $\log \bp$, denoted by $\log p_{t^{(n)}}$:

$\mathcal{L}^{(n)} = \log p(t = t^{(n)} \;|\; \bx = \bx^{(n)}, \bb, \bW) = \log p_{t^{(n)}} = \log q_{t^{(n)}} - \log Z^{(n)}$

where $\bx^{(n)}$ and $t^{(n)}$ are the input (image) and class label (integer) of the $n$-th datapoint, and $Z^{(n)}$ is the normalizing constant for the distribution over $t^{(n)}$.

## 1.1 Gradient-based stochastic optimization
### 1.1.1 Derive gradient equations (20 points)

Derive the equations for computing the (first) partial derivatives of the log-likelihood w.r.t. all the parameters, evaluated at a _single_ datapoint $n$.

You should start deriving the equations for $\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}$ for each $j$. For clarity, we'll use the shorthand $\delta^q_j = \frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}$.

For $j = t^{(n)}$:
$$
\delta^q_j
= \frac{\partial \log q_{t^{(n)}}}{\partial \log q_j}
-
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
= 1
-
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
$$

For $j \neq t^{(n)}$:
$$
\delta^q_j
= \frac{\partial \log q_{t^{(n)}}}{\partial \log q_j}
-
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
=0 - \frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
$$

Complete the above derivations for $\delta^q_j$ by furtherly developing $\frac{\partial \log Z}{\partial Z}$ and $\frac{\partial Z}{\partial \log q_j}$. Both are quite simple. For these it doesn't matter whether $j = t^{(n)}$ or not.

YOUR ANSWER HERE
For $j = t^{(n)}$:
\begin{align}
\delta^q_j
&=1-Z^{-1}\frac{\partial\sum_kq_k}{\partial\log q_j}=1-Z^{-1}\frac{\partial\sum_ke^{\log(q_k)}}{\partial\log q_j} =1-Z^{-1}e^{\log(q_j)}
\end{align}
For $j \neq t^{(n)}$:
\begin{align}
\delta^q_j
&=-Z^{-1}e^{\log(q_j)}
\end{align}

Given your equations for computing the gradients $\delta^q_j$ it should be quite straightforward to derive the equations for the gradients of the parameters of the model, $\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}}$ and $\frac{\partial \mathcal{L}^{(n)}}{\partial b_j}$. The gradients for the biases $\bb$ are given by:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial b_j}
= \frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial b_j}
= \delta^q_j
\cdot 1
= \delta^q_j
$

The equation above gives the derivative of $\mathcal{L}^{(n)}$ w.r.t. a single element of $\bb$, so the vector $\nabla_\bb \mathcal{L}^{(n)}$ with all derivatives of $\mathcal{L}^{(n)}$ w.r.t. the bias parameters $\bb$ is:

$
\nabla_\bb \mathcal{L}^{(n)} = \mathbf{\delta}^q
$

where $\mathbf{\delta}^q$ denotes the vector of size $10 \times 1$ with elements $\mathbf{\delta}_j^q$.

The (not fully developed) equation for computing the derivative of $\mathcal{L}^{(n)}$ w.r.t. a single element $W_{ij}$ of $\bW$ is:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}} =
\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial W_{ij}}
= \mathbf{\delta}_j^q
\frac{\partial \log q_j}{\partial W_{ij}}
$

What is $\frac{\partial \log q_j}{\partial W_{ij}}$? Complete the equation above.

If you want, you can give the resulting equation in vector format ($\nabla_{\bw_j} \mathcal{L}^{(n)} = ...$), like we did for $\nabla_\bb \mathcal{L}^{(n)}$.

YOUR ANSWER HERE

$
\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}} = \mathbf{\delta}_j^q
\frac{\partial \log q_j}{\partial W_{ij}}
= \mathbf{\delta}_j^q \frac{\partial W_j^TX + b_j}{\partial W_{ij}}
= \mathbf{\delta}_j^q \frac{\partial \sum_kw_{kj}x_k + b_j}{\partial W_{ij}}
= \mathbf{\delta}_j^q x_i
$

$
\nabla_{w_j}\mathcal{L}^{(n)} = (\mathbf{\delta}_j^q*x^{(\mathbf{n})})^T
$

### 1.1.2 Implement gradient computations (15 points)

Implement the gradient calculations you derived in the previous question. Write a function `logreg_gradient(x, t, w, b)` that returns the gradients $\nabla_{\bw_j} \mathcal{L}^{(n)}$ (for each $j$) and $\nabla_{\bb} \mathcal{L}^{(n)}$, i.e. the first partial derivatives of the log-likelihood w.r.t. the parameters $\bW$ and $\bb$, evaluated at a single datapoint (`x`, `t`).
The computation will contain roughly the following intermediate variables:

$
\log \bq \rightarrow Z \rightarrow \log \bp\,,\, \mathbf{\delta}^q
$

followed by computation of the gradient vectors $\nabla_{\bw_j} \mathcal{L}^{(n)}$ (contained in a $784 \times 10$ matrix) and $\nabla_{\bb} \mathcal{L}^{(n)}$ (a $10 \times 1$ vector).

For maximum points, ensure the function is numerically stable.

```python

```

```python
# 1.1.2 Compute gradient of log p(t|x;w,b) wrt w and b
def logreg_gradient(x, t, w, b):
    log_q = np.dot(x,w) + b

    a = np.max(log_q)

    log_Z = a + np.log(np.sum(np.exp(log_q-a)))

    logp = log_q - log_Z

    delta = np.zeros_like(log_q)
    delta[:,t] = 1
    delta -= np.exp(log_q)/np.exp(log_Z)

    dL_db = delta

    dL_dw = x.T.dot(delta)

    # here the statement contains logp[:,t] where logp is meant tas a matrix of shape 1x10

    return logp[:,t].squeeze(), dL_dw, dL_db.squeeze()

```

```python
# Hidden tests for efficiency
```

```python
np.random.seed(123)
# scalar, 10 X 768  matrix, 10 X 1 vector
w = np.random.normal(size=(28*28,10), scale=0.001)
# w = np.zeros((784,10))
b = np.zeros((10,))

# test gradients, train on 1 sample
logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)

print("Test gradient on one point")
print("Log Likelihood:\t", logpt)
print("\nGrad_W_ij\t",grad_w.shape,"matrix")
print("Grad_W_ij[0,152:158]=\t", grad_w[152:158,0])
print("\nGrad_B_i shape\t",grad_b.shape,"vector")
print("Grad_B_i=\t", grad_b.T)
print("i in {0,...,9}; j in M")

assert logpt.shape == (), logpt.shape
assert grad_w.shape == (784, 10), grad_w.shape
assert grad_b.shape == (10,), grad_b.shape

```

    Test gradient on one point
    Log Likelihood:	 -2.2959726720744777

    Grad_W_ij	 (784, 10) matrix
    Grad_W_ij[0,152:158]=	 [-0.04518971 -0.06758809 -0.07819784 -0.09077237 -0.07584012 -0.06365855]

    Grad_B_i shape	 (10,) vector
    Grad_B_i=	 [-0.10020327 -0.09977827 -0.1003198   0.89933657 -0.10037941 -0.10072863
     -0.09982729 -0.09928672 -0.09949324 -0.09931994]
    i in {0,...,9}; j in M

```python
# It's always good to check your gradient implementations with finite difference checking:
# Scipy provides the check_grad function, which requires flat input variables.
# So we write two helper functions that provide the gradient and output with 'flat' weights:
from scipy.optimize import check_grad

np.random.seed(123)
# scalar, 10 X 768  matrix, 10 X 1 vector
w = np.random.normal(size=(28*28,10), scale=0.001)
# w = np.zeros((784,10))
b = np.zeros((10,))

def func(w):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w.reshape(784,10), b)
    return logpt
def grad(w):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w.reshape(784,10), b)
    return grad_w.flatten()
finite_diff_error = check_grad(func, grad, w.flatten())
print('Finite difference error grad_w:', finite_diff_error)
assert finite_diff_error < 1e-3, 'Your gradient computation for w seems off'

def func(b):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)
    return logpt
def grad(b):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)
    return grad_b.flatten()
finite_diff_error = check_grad(func, grad, b)
print('Finite difference error grad_b:', finite_diff_error)
assert finite_diff_error < 1e-3, 'Your gradient computation for b seems off'

```

    Finite difference error grad_w: 6.3612946893e-07
    Finite difference error grad_b: 5.23511748609e-08

```python
# DO NOT REMOVE THIS CELL!
# It contains hidden tests

```

### 1.1.3 Stochastic gradient descent (15 points)

Write a function `sgd_iter(x_train, t_train, w, b)` that performs one iteration of stochastic gradient descent (SGD), and returns the new weights. It should go through the trainingset once in randomized order, call `logreg_gradient(x, t, w, b)` for each datapoint to get the gradients, and update the parameters **using a small learning rate of `1e-6`**. Note that in this case we're maximizing the likelihood function, so we should actually performing gradient ___ascent___... For more information about SGD, see Bishop 5.2.4 or an online source (i.e. https://en.wikipedia.org/wiki/Stochastic_gradient_descent)

```python
def sgd_iter(x_train, t_train, W, b):
    mu = 1e-4
    logp_train = 0
    indexes = np.array([x for x in range(0,len(x_train))])
    np.random.shuffle(indexes)
    for i in np.nditer(indexes):
        logpt, grad_w, grad_b = logreg_gradient(x_train[i:i+1,:], t_train[i:i+1], W,b)
        W += mu*(grad_w)
        b += mu*(grad_b)
        logp_train += logpt

    return logp_train, W, b
```

```python
# Hidden tests for efficiency
```

```python
# Sanity check:
np.random.seed(1243)
w = np.zeros((28*28, 10))
b = np.zeros(10)

logp_train, W, b = sgd_iter(x_train[:5], t_train[:5], w, b)

```

## 1.2. Train

### 1.2.1 Train (12 points)
Perform SGD on the training set. Plot (in one graph) the conditional log-probability of the training set and validation set after each iteration. (6 points)

Instead of running SGD for a fixed number of steps, run it until convergence. Think of a reasonable criterion for determining convergence. As a reference: choose a criterion such that the algorithm terminates in less than 15 iterations over the training set. (2 points)

Make sure your implementation (in particular, the output of the conditional log-probability of the training set and validation set) is independent of the size of the dataset. (2 points)

```python
def plot(traning, validation, itter):

    plt.plot(itter, traning, label='trainning')
    plt.plot(itter, validation, label='validation')

    plt.xlabel('itterations')
    plt.ylabel('log p probabilities')

    plt.title("Probabilites over itterations")

    plt.legend()

    plt.show()

def test_sgd(x_train, t_train, x_valid, t_valid, w, b):
    epsilon = 10.0
    diffrence = 1000.0
    i = 1
    t_log_p = []
    v_log_p = []
    itter = []
    while diffrence > epsilon:
        old_w = w.copy()
        old_b = b.copy()
        iter_logp, w, b = sgd_iter(x_train, t_train, w, b)
        iter_logp_val, _, _ = sgd_iter(x_valid, t_valid, w, b)

        t_log_p.append(iter_logp)
        v_log_p.append(iter_logp_val)
        itter.append(i)

        diffrence = np.sum(np.abs(old_w-w))
        diffrence += np.sum(np.abs(old_b-b))
        i+=1

    plot(t_log_p, v_log_p, itter)
    return w,b

np.random.seed(1243)
w = np.zeros((28*28, 10))
b = np.zeros(10)
w,b = test_sgd(x_train, t_train, x_valid, t_valid, w, b)
```

```python

```

```python
# Hidden tests for efficiency
```

### 1.2.2 Visualize weights (10 points)
Visualize the resulting parameters $\bW$ after a few iterations through the training set, by treating each column of $\bW$ as an image. If you want, you can use or edit the `plot_digits(...)` above.

```python
def plot_digits(data, num_cols, targets=None, shape=(28,28)):
    num_digits = data.shape[0]
    num_rows = int(num_digits/num_cols)
    for i in range(num_digits):
        plt.subplot(num_rows, num_cols, i+1)
        plt.imshow(data[i].reshape(shape), interpolation='none', cmap='Greys')
        if targets is not None:
            plt.title(int(targets[i]))

        plt.axis('off')
    plt.tight_layout()
    plt.show()

plot_digits(w.T, num_cols=5)
```

**Describe in less than 100 words why these weights minimize the loss**

These weights are minimizing the loss as they are greater in similar places as would written digits be. Thus,
they would be learning where handwritten digits would be. Sometimes we can see clearly which digits they try to fit
as in the case of 0 or 3, but as the digit could be written in diffrent maners it would have to fit all that writting space
hence making a bit unclear which digit they try to fit, still this minimizes the loss as the model tries to fill all the possible given writtings based uupon their occurences in the training set.

### 1.2.3. Visualize the 8 hardest and 8 easiest digits (10 points)
Visualize the 8 digits in the validation set with the highest probability of the true class label under the model.
Also plot the 8 digits that were assigned the lowest probability.

```python
# YOUR CODE HERE
from operator import itemgetter

def log_p(x,W,b):
    log_q = np.dot(x,w) + b

    a = np.max(log_q)

    log_Z = a + np.log(np.sum(np.exp(log_q-a)))

    logp = log_q - log_Z
    return logp

def visiualize(x_valid, t_valid, w, b):
    prob = []
    true_classes = []
    for i in range(0, len(x_valid)):
        logp = log_p(x_valid[i:i+1,:],W,b)
        prob.append((logp[:,t_valid[i]],t_valid[i:i+1],x_valid[i:i+1,:]))

    worst = sorted(prob,key= lambda x:x[0])[0:8]
    best = sorted(prob,key= lambda x:x[0], reverse = True)[0:8]

    best_digits = np.array([x[2][0] for x in best])
    best_targets = np.array([x[1][0] for x in best])
    plot_digits(best_digits,4,targets=best_targets)

    worst_digits = np.array([x[2][0] for x in worst])
    worst_targets = np.array([x[1][0] for x in worst])
    plot_digits(worst_digits,4,targets=worst_targets)

visiualize(x_valid,t_valid, w, b)
```

Ask yourself if these results make sense. Explain in no more then two sentences what it means that a digit is hard to classify.

YOUR ANSWER HERE
A digit is hard to classify if it resembeles to another digit, these results make sens since 7 may have common lines to the way we write 1 or 4. We can see that we don't have this problem with 2 as it does not resemble any other digit. Briefly, the more the digits have in common of their way of writting, the darder they are to classify.

# Part 2. Multilayer perceptron

You discover that the predictions by the logistic regression classifier are not good enough for your application: the model is too simple. You want to increase the accuracy of your predictions by using a better model. For this purpose, you're going to use a multilayer perceptron (MLP), a simple kind of neural network. The perceptron will have a single hidden layer $\bh$ with $L$ elements. The parameters of the model are $\bV$ (connections between input $\bx$ and hidden layer $\bh$), $\ba$ (the biases/intercepts of $\bh$), $\bW$ (connections between $\bh$ and $\log q$) and $\bb$ (the biases/intercepts of $\log q$).

The conditional probability of the class label $j$ is given by:

$\log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$

where $q_j$ are again the unnormalized probabilities per class, and $Z = \sum_j q_j$ is again the probability normalizing factor. Each $q_j$ is computed using:

$\log q_j = \bw_j^T \bh + b_j$

where $\bh$ is a $L \times 1$ vector with the hidden layer activations (of a hidden layer with size $L$), and $\bw_j$ is the $j$-th column of $\bW$ (a $L \times 10$ matrix). Each element of the hidden layer is computed from the input vector $\bx$ using:

$h_j = \sigma(\bv_j^T \bx + a_j)$

where $\bv_j$ is the $j$-th column of $\bV$ (a $784 \times L$ matrix), $a_j$ is the $j$-th element of $\ba$, and $\sigma(.)$ is the so-called sigmoid activation function, defined by:

$\sigma(x) = \frac{1}{1 + \exp(-x)}$

Note that this model is almost equal to the multiclass logistic regression model, but with an extra 'hidden layer' $\bh$. The activations of this hidden layer can be viewed as features computed from the input, where the feature transformation ($\bV$ and $\ba$) is learned.

## 2.1 Derive gradient equations (20 points)

State (shortly) why $\nabla_{\bb} \mathcal{L}^{(n)}$ is equal to the earlier (multiclass logistic regression) case, and why $\nabla_{\bw_j} \mathcal{L}^{(n)}$ is almost equal to the earlier case.

Like in multiclass logistic regression, you should use intermediate variables $\mathbf{\delta}_j^q$. In addition, you should use intermediate variables $\mathbf{\delta}_j^h = \frac{\partial \mathcal{L}^{(n)}}{\partial h_j}$.

Given an input image, roughly the following intermediate variables should be computed:

$
\log \bq \rightarrow Z \rightarrow \log \bp \rightarrow \mathbf{\delta}^q \rightarrow \mathbf{\delta}^h
$

where $\mathbf{\delta}_j^h = \frac{\partial \mathcal{L}^{(n)}}{\partial \bh_j}$.

Give the equations for computing $\mathbf{\delta}^h$, and for computing the derivatives of $\mathcal{L}^{(n)}$ w.r.t. $\bW$, $\bb$, $\bV$ and $\ba$.

You can use the convenient fact that $\frac{\partial}{\partial x} \sigma(x) = \sigma(x) (1 - \sigma(x))$.

YOUR ANSWER HERE

$
\mathbf{\delta}_j^h = \frac{\partial \mathcal{L}^{(n)}}{\partial \bh_j} =
\frac{\partial\log q_j}{\partial \bh_j } - \frac{\partial\log Z}{\partial \bh_j } =
\frac{\partial\bw_j^T \bh + b_j}{\partial \bh_j } - \frac{ \log Z}{\partial Z} \frac{\partial Z}{\partial \bh_j} =
w_{jj} - Z^{-1}\frac{\partial Z}{\partial \bh_j}=
w_{jj} - Z^{-1}\frac{\partial\sum_i e^{\log q_i}}{\partial \log q_j}\frac{\log q_j}{\partial \bh_j}=
w_{jj}(1 - Z^{-1} e^{\log q_j}) =
w_{jj}\mathbf{\delta}_j^q
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial b_k} =
\frac{\partial\log q_j}{\partial  b_k } - \frac{\partial\log Z}{\partial  b_k } =
\frac{\partial\bw_j^T \bh + b_j}{\partial b_k } - \frac{\partial\log Z}{\partial Z}\frac{\partial Z}{\partial b_k} =
(j==k) - Z^{-1} \frac{\sum_i e^{log q_i}}{b_k}=
(j==k) - Z^{-1} e^{log q_k} =
\delta^q_k
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial b} = \delta^q
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}} =
\frac{\partial\log q_j}{\partial  W_{ij} } - \frac{\partial\log Z}{\partial  W_{ij} } =
\frac{\partial\bw_j^T \bh + b_j}{\partial W_{ij} } - \frac{\partial\log Z}{\partial Z}\frac{\partial Z}{\partial W_{ij}}=
h_i - Z^{-1}e^{\ln q_j}h_i =
h_i(1 - Z^{-1}e^{\ln q_j}) =
h_i\mathbf{\delta}_j^q
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial w} = h^T\mathbf{\delta}^q
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial V_{j}} =
\frac{\partial\bw_j^T \bh + b_j}{\partial V_{j} } - \frac{\partial\log Z}{\partial Z}\frac{\partial Z}{\partial V_{j}}=
w_{jj}h_j(1-h_j)x_j(1 - Z^{-1} e^{log q_j}) =
w_{jj}h_j(1-h_j)x_j\mathbf{\delta}_j^q
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial V} = w_{j}h(1-h)x\mathbf{\delta}^q
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial a_j} =
\frac{\partial\bw_j^T \bh + b_j}{\partial a_j } - \frac{\partial\log Z}{\partial Z}\frac{\partial Z}{\partial a_j}=
w_{jj}h_j(1-h_j)(1 - Z^{-1} e^{log q_j}) =
w_{jj}h_j(1-h_j)\mathbf{\delta}_j^q
$

$
\frac{\partial \mathcal{L}^{(n)}}{\partial a} =
w_{j}h(1-h)\mathbf{\delta}^q
$

## 2.2 MAP optimization (10 points)

You derived equations for finding the _maximum likelihood_ solution of the parameters. Explain, in a few sentences, how you could extend this approach so that it optimizes towards a _maximum a posteriori_ (MAP) solution of the parameters, with a Gaussian prior on the parameters.

For determining the maximum a posteriori solution we must also determine the parameters for the gaussian prior which would
tell us how likely each class it is. Thus, would to find the optimal mean and variance for this prior. Also we would do these calculations in the logaritmic space for making them simpler. The only diffrence this time would be that we will not derivate only after $\mathcal{L}^{(n)}$ but by $\mathcal{L}^{(n)}$ times the gaussian prior. We would not take account of the evidence as it is not influenced by any of this parameters.

## 2.3. Implement and train a MLP (15 points)

Implement an MLP model with a single hidden layer of **20 neurons**.
Train the model for **10 epochs**.
Test your implementation for learning rates of 1e-2, 1e-3 and 1e-4 and plot (in one graph) the conditional log-probability of the trainingset and validation set.

For the best model plot the weights of the first layer for in epoch 0,4 and 9.

- 10 points: Working MLP that learns with plots
- +5 points: Fast, numerically stable, vectorized implementation

```python
def mlp_gradient(x, t, W, b, V, a):
    h = 1 / (1 + np.exp(-x.dot(V) - a))
    log_q = h.dot(W) + b

    a = np.max(log_q)
    log_z = a + np.log(np.sum(np.exp(log_q - a)))
    z = np.exp(log_z)

    grad_b = np.zeros(log_q.shape)
    grad_b[0][t[0]] = 1
    grad_b -= np.exp(log_q) / z
    grad_W = np.matmul(np.transpose(h), grad_b)

    grad_h = np.matmul(grad_b, np.transpose(W))
    grad_a = grad_h * h * (1 - h)

    grad_V = np.matmul(np.transpose(x), grad_a)

    log_p = log_q - log_z

    return log_p[np.arange(len(t)), t], grad_V, grad_a.squeeze(), grad_W, grad_b.squeeze()
```

```python
# Hidden tests for efficiency
```

```python
# Write training code here:
# Plot the conditional loglikelihoods for the train and validation dataset after every iteration.
# Plot the weights of the first layer.

def sgd_mlp_iter(x_train, t_train, v, a, W, b, lr):
    idx = list(range(len(x_train)))
    np.random.shuffle(idx)

    logp_train = 0
    for i in idx:
        logpt, grad_v, grad_a, grad_w, grad_b = mlp_gradient(x_train[i:i+1, :], t_train[i:i+1], W, b, v, a)
        logp_train += logpt
        v += lr  * grad_v
        a += lr  * grad_a
        W += lr  * grad_w
        b += lr  * grad_b

    return logp_train, v, a, W, b
```

```python
def plot_weights(halves, num_cols, targets=None, shape=(28,28)):
    num_digits = halves[0].shape[1]
    num_rows = int(num_digits/num_cols)

    for w in halves:
        for i in range(num_digits):
            plt.subplot(2, int(num_digits/2), i+1)
            plt.imshow(w[:,i].reshape(shape), interpolation='None', cmap='YlGnBu')
            if targets is not None:
                plt.title("Weight " + str(i),fontsize=11,fontweight="bold")
            plt.colorbar()
            plt.axis('off')
            i +=1
        plt.tight_layout()
        plt.show()

def mlp_compute_loss(x, t, v, a, w, b):
    h = 1/(1+np.exp(-(np.dot(x,v)+a)))
    log_q = np.dot(h,w)+b
    max_log_q = np.max(log_q,axis = 1)

    log_Z = max_log_q[:,np.newaxis] + np.log(np.sum(np.exp(log_q - max_log_q[:,np.newaxis]),axis = 1))[:,np.newaxis]

    logp = log_q - log_Z
    return logp[np.arange(len(t)),t]

def test_mlp(x_train, t_train, x_valid, t_valid, v, a, w, b, lr, nr_iters):
    train_loss = np.zeros((nr_iters))
    val_loss = np.zeros((nr_iters))
    for i in range(nr_iters):
        loss, v, a, w, b = sgd_mlp_iter(x_train, t_train, v, a, w, b, lr)
        train_loss[i] = loss/len(x_train)
        val_loss[i] = np.sum(mlp_compute_loss(x_valid,t_valid,v,a,w,b)) / len(x_valid)

        if lr == 1e-2 and (i == 0 or i== 4 or i==9):
            print('Weights plots of iteration: ' + str(i))
            halves = np.split(v, 2, 1)
            plot_weights(halves, 5)

    return train_loss, val_loss

def run_test_mlp(lrs, nr_iters, classes, hidden):
    best_params = []
    best_loss = 200
    nr_runs = len(lrs)
    train_loss = np.empty([nr_runs,nr_iters])
    val_loss = np.empty([nr_runs,nr_iters])
    for idx_lr, lr in enumerate(lrs):
        i = idx_lr
        print('\n\n---starting run lr '+str(lr))
        np.random.seed(123)
        v = np.random.normal(size=(28*28,hidden), scale=0.001)
        a = np.zeros((hidden,))
        w = np.random.normal(size=(hidden,classes), scale=0.001)
        b = np.zeros((classes,))
        train_loss[i], val_loss[i] = test_mlp(x_train, t_train, x_valid, t_valid, v, a, w, b, lr, nr_iters)
        if val_loss[i][-1]<best_loss:
            best_loss = val_loss[i][-1]
            best_params = [v, a, w , b]

    fig, ax = plt.subplots(nrows=len(lrs), ncols=1,figsize=(15,15))
    for idx_lr, lr in enumerate(lrs):
        i = idx_lr
        ax = plt.subplot(len(lrs),1, i+1)
        ax.plot(list(range(nr_iters)), train_loss[i])
        ax.plot(list(range(nr_iters)), val_loss[i])
        ax.legend(['train ', 'validation '])
        ax.set_title('sgd_iter' +' lr= '+str(lr))
    #plt.tight_layout()
    plt.show()
    return best_params

classes = 10
lrs = [1e-2,1e-3,1e-4]
iters = 10
hidden = 20
best = run_test_mlp(lrs, iters, classes, hidden = hidden)
```

### 2.3.1. Explain the learning curves (5 points)
In less than 80 words, explain the observed behaviour for the different learning rates.

We can observe from these graphs that a learning rate of $1e^{-5}$ results in very small steps and does not converge within the 10 iterations. Bigger learning steps are needed in the beginning, as it is the case for $1e^{-4}$ and $1e^-5$, which ensures that indeed the algorithm does converge

### 2.3.2. Explain the weights (5 points)
In less than 80 words, explain how and why the weights of the hidden layer of the MLP differ from the logistic regression model, and relate this to the stronger performance of the MLP.

```python
halves = np.split(best[0], 2, 1)
plot_weights(halves, 5)
```

The model is able to capture more complex dependencies between the input and the output with the help of the hidden layer.

### 2.3.2. Different activation functions (10 points)
In the task above we use a sigmoid as an activation function.
Two other popular choices for activation functions are tanh and the rectified linear unit (ReLU). The ReLU is defined as:

$$f(x) = \max(0.,x)$$

You already derived the derivative of the softmax function above. Here, write down the derivative for both the tanh and the ReLU function. Furthermore, for all three, plot the function and its derivative in a range $x\in[-3,3]$

Write down the derivative of ReLU and tanh w.r.t. their respective argument:

The ReLU derivative is 0 for x <0, 1 for x > 0, and undefined at 0.

Tanh is defined as $\frac{e^a  - e^{-a}}{e^a + e ^{-a}}$.

$\frac{\partial \tanh{a}}{\partial a} = 1 - \tanh{(a)}^2$

Name two properties that you would like your activation function to have (one sentence each). Why are they important?

1) Centered around 0. This ensures that the algorithm usually convergences faster. Otherwise, maybe bias on gradient direction, which might slow down learning.

2) Continuously differentiable (almost) everywhere. Necessary for enabling gradient-based optimization methods

3) Non linearity as the purpose of the activation function is to introduce non-linearity into the network.

```python
# plot the function and the derivative for the activations sigmoid, tanh and ReLU.

def plot(title, value1, value1_label, value2, value2_label, value3, value3_label):
    fontsize = 18
    plt.plot(x, value1, 'g', label=value1_label)
    plt.plot(x, value2, 'b', label=value2_label)
    plt.plot(x, value3, 'm', label=value3_label)
    plt.xlabel("x")
    plt.ylabel("y")
    plt.title(title,fontsize=fontsize,fontweight="bold")
    plt.legend()
    plt.show()

def tanh_derivative(x):
    return 1 / (np.cosh(x)**2)

def sigmoid(x):
    return 1/(1 + np.exp(-x))

def sigmoid_derivative(x):
    return sigmoid(x)*(1 - sigmoid(x))

def relu(x):
    return np.maximum(0, x)

def relu_derivative(x):
    return np.where(x <= 0, 0, 1)

x = np.linspace(-3, 3, 200)

tanh_values = np.tanh(x)
tanh_derivative_values = tanh_derivative(x)

sigmoid_values = sigmoid(x)
sigmoid_derivative_values = sigmoid_derivative(x)

relu_values = relu(x)
relu_derivative_values = relu_derivative(x)

plot("Activation functions", tanh_values, "tanh", sigmoid_values, "sigmoid", relu_values, "relu")
plot("Activation functions derivatives", tanh_derivative_values, "tanh derivative", sigmoid_derivative_values, "sigmoid derivatives", relu_derivative_values, "relu derivative")

```

Now that you plotted the activations and derivatives, which activation do you think is the best? Why would you choose this activation function? For your answer consider what you named as essential properties for an activation function above. Keep your answer short at no more then 3 sentences.

Sigmoid is useful for binary classification, however it suffers from vanishing gradient problem, and it is not zero centered. Tanh is zero centered however it still suffers from the vanishing gradient problem. ReLU   and its variants (leakyReLU) have become very popular as they avoid and rectify the vanishing gradient problem. It suffers from problems such as dead neurons, however its variants such as LeakyReLU solve this issue

```python
print('Notebook ran in {:2.3} minutes.'.format((time.time()-start)/60))
```


# Computer Vision, Lab 05: Cameras and lenses

Today we'll learn about cameras, lenses, and how to calibrate a camera.
Some references for this material:

 - https://docs.opencv.org/master/d4/d94/tutorial_camera_calibration.html
 - Halcon Manual
 - https://www.baslerweb.com/
 - https://www.baslerweb.com/en/products/tools/lens-selector/
 - https://docs.opencv.org/master/dc/dbb/tutorial_py_calibration.html
 - https://learnopencv.com/camera-calibration-using-opencv/

## Camera Systems

The camera system is an important module in any machine vision system. The camera sytem integrates with an overall machine vision system as follows.

The important parameters in a camera system are:

 1. Resolution: size of relevant aspect of object of interest.
 2. Field of View (FOV): The area the camera captures. HFOV and VFOV indicate the horizontal and vertical FOVs.
 3. Working Distance (WD): Distance between the camera and object. Fixed in an industrial system or variable in other applications.
 4. Sensor Size: Size of the CCD or CMOS image sensor chip on the digital camera's circuit board.
 5. Depth of Field (DOF): Distance between the nearest and farthest objects that are in focus.
 6. Image Resolution: Size of image, $(w \times h)$. Standard sizes such as 1920$\times$1080 HD are referred to by the vertical resolution,
    e.g. 1080p.
 7. Pixel: Grayscale, RGB, multispectral, etc. For a fixed working distance, a pixel's backprojection has a fixed size, e.g., 2 mm per pixel.
 8. Object Resolution: size of object of interest when imaged at the minimum size (e.g., faces at 20$\times$20 or
    humans at 64$\times$128).
 9. Focal Length: Distance between the image plane and the optical center of the camera, measured in pixels or mm.
 10. Magnification: Ratio between sensor size and FOV.

DOF varies depending on aperture:

### Image Sensors

When an image is captured by the image sensor, the state of each photoreceptor
is saved then transferred to host memory. Image sensors in digital cameras and
industrial sensors may be one of two types:

 - CCDs convert photoreceptor charges to analog voltages then digitized off-chip
 - CMOS sensors have separate digitization circuitry for every pixel

Early fabrication techniques in the 1970s and 1980s were more suitable for CCDs, so available CCD sensors delivered much higher quality images and therefore
dominated machine vision. In the 1990s, mobile phone commoditization started pushing CMOS technology, and now the available CMOS sensors are equal to or greater
than CCDs in quality and much cheaper, though CCDs still deliver higher quality NIR imaging than CMOS sensors.

Image sensor size is the main way to adjust the quality of the image output. Early on, most sensors had 4:3 aspect ratios, but now HD resolutions with 16:9
ratios are becoming more common.

Depending on the size and density of the photoreceptors on the image sensor, we get different image resolutions.

### How to know if resolution is suitable for your work?

The International Standards Organization has standard 12233, which
defines various standardized charts that can be used to assess the
quality of an image. Here is the "legacy" (pre-2000) chart:

You can purchase charts from the ISO or print one yourself. See
[Stephen Westin's Cornell Graphics web page on the ISO resolution chart](https://www.graphics.cornell.edu/~westin/misc/res-chart.html)
for more detail.

A rough idea can be obtained by considering the physical size of the smallest detail you want to analyze, considering how accurately
you want to analyze it (the minimum number of pixels that should span the detail), then using that to calculate the necessary image
resolution, lens focal length, and working distance. For example, suppose

1. You want to detect cracks in a surface object that are a minimum 0.5. mm wide.
2. You believe your crack detection algorithm requires at least 4 pixels of width
   to ensure accurate detection (8 pixels per mm).
3. You like the image quality and price of a 2 MP HD camera with a 50 degree HFOV.

Since object size in pixels is $fX/Z$, we can write $$4 \text{pixels} = f \text{pixels} \times 0.5 \text{mm} / D.$$
To get $f$, we know that $$\tan^{-1}(25^\circ) = 960 / f,$$ so $f = 2333.3$ and $D = 0.2916$ m. If a 29 cm working distance
is OK for our application, the last step would be to ensure that the camera can focus on objects at that distance and make sure
the DOF around a focus of 29 cm is sufficient.

### Frame rate

Another parameter that is important in most applications is the camera's frame rate. It is measured in frames per second.
When you are imaging objects in motion, you should consider the maximum object velocity at the minimum working distance. That
will tell you the maximum object displacement to expect per frame. For object tracking, you would want to ensure a fast enough
frame rate that successive images of the moving object overlap sufficiently to have high confidence when associating the object
appearance in a new frame with the object's appearance in the previous frame.

Three other factors are related to frame rate: exposure time, shutter type, and interlacing.

A short exposure time will minimize motion blur but will also decrease contrast. A longer exposure time will allow more light
to reach the sensor, improving contrast but also increasing motion blur.

The shutter can be global (CCD sensors and some CMOS sensors) or rolling (most CMOS sensors). A global shutter captures the entire
image simultaneously, while a rolling shutter exposes sensor elements sequentially. If the shutter is fast enough relative to object motion,
the shutter type is unimprotant. But if object motion (or camera motion) is high, a global shutter is preferred.

Interlacing means that multiple passes are required to piece together the image. For example, even-numbered rows may be imaged on the
first pass and odd-numbered rows may be imaged on the second pass. Interlacing is not common on modern sensors.

## Understanding Lens Specifications

Lens specifications have two important measurements: Focal Length and Aperture (F-Stop)

### Aperture (F-Stop)

Aperture (called "F-Stop" in commercial camera specifications)
indicates the width of the opening allowing light into the lens.
Aperture (A) is measured relative to focal length:

$$A = \frac{f}{S},$$

where $S$ is the "aperture scale." Smaller aperture scales mean more light than
larger aperture scales.

When illumination is low, you may need larger apertures (smaller aperture scale numbers):

And conversely, when light is especially bright,
you may need smaller apertures. Aperture has interesting effects
on especially bright light sources like the sun:

But more importantly, when you have good illumination, you can afford to reduce the aperture
(increase the aperture number), which has the benefit of giving you a larger DOF:

Obviously, you would like to have a large DOF for a mobile robot, so you might want a very small
aperture, but on the other hand, you also want a wide field of view (shorter focal length),
which reduces your relative aperture size.

### Focal length

Focal length is the distance between the image sensor and the center of projection. In camera
specifications, it is measured in mm, whereas in machine vision calculations, it is measured in pixels.
Longer focal lengths give more zoom and smaller FOV.

A small focal length is great for imaging a wide field of view, but spreading the image out so much
also typically requires a more convex lens, which increases the radial distortion.

### Planar Perspective Correction

When your camera's principal axis is not orthogonal to the surface you're imaging, you will experience perspective distortion. As
we've seen in class, if the object is truly planar, we can correct the image using a 4-point homography.

### Radial Distortion

The word "distortion" might be used to describe any kind of warping of an object due to depth, non-orthogonal principal axis, or
"bending" of straight lines. In computer vision, we normally reserve the word "distortion" to refer to non-pinhole effects arising
from the use of a lens to achieve the desired projection onto the image sensor. Non-pinhole distortion is either *radial* (warping
along a vector emanting from a center of distortion), which is very common, especially with wide angle lenses, and *tangential*
(warping orthogonal to the radial direction), which is less common and normally arises due to imprecise lens mounting. Getting precise
measurements from a camera system requires either taking distortion into account or eliminating it. Distortion
can be eliminated using optical techniques (using compound lenses and other hardware techniques), but this is expensive and possibly bulky.
A more practical solution to eliminate distortion is to model the distortion function of your camera system then either use that model
in your calculations or rectify every image before further processing.

The modeling process is called *spatial calibration* and uses a known calibration pattern:

## Geometric Camera Parameters

Knowing the camera fundamentals above, we can use a calibration
process to calculate camera parameters in geometric form. We'll assume
the camera's optical center as the origin of a 3D coordinate reference frame
and the principal point of the camera as the point in the image at which the
camera's principal axis and image plane intersect.

### Types of parameters

There are two types of parameters to calculate to reconstruct the 3D structure of a scene from the pixel coordinates of image points:

1. Extrinsic camera parameters: Parameters defining location and orientation of the camera reference frame in a world reference frame.
2. Intrinsic camera parameters: Parameters linking 3D points in the camera reference frame to image points.

As the figure shows, we can express a camera coordinate point in terms of a world coordinate using the intrinsic and extrinsic matrices as:

\begin{equation}
\mathbf{x} = \begin{bmatrix}
x \\
y \\
1
\end{bmatrix} \propto \mathtt{K} \mathbf{X}_c = \mathtt{K} \begin{bmatrix} \mathtt{R} \mid \mathbf{t} \end{bmatrix} \begin{bmatrix}
X \\
Y \\
Z \\
1
\end{bmatrix}
\end{equation}

Where
 - $\mathbf{X}_c$ is a 3D camera point
 - $\mathbf{X}_w = \begin{bmatrix} X & Y & Z & 1 \end{bmatrix}^\top$ is a 3D world point in homogeneous form
 - $\mathtt{K}$ is the intrinsic calibration matrix

### Extrinsic Camera Parameters

The parameters $\mathtt{R}$ and $\mathbf{t}$ mean

1. The origin of the world coordiante system in the camera reference frame.
2. The rotation matrix that brings the corresponding axes of the two frames into alignment (i.e., onto each other).

Using the extrinsic camera parameters, we can find the relation between the coordinates of a point P in world ($P_w$) and image plane ($P_{im}$) coordinates:

$$
\mathbf{x} \propto \mathtt{K}\mathtt{R}(\tilde{\mathbf{X}}_w - \mathbf{C}) = \mathtt{K} \begin{bmatrix} \mathtt{R} \mid \mathbf{t} \end{bmatrix} \mathbf{X}_w,
$$
where
\begin{equation}
[R | t] =
\begin{bmatrix}
r_{11} & r_{12} & r_{13} & t_1\\
r_{21} & r_{22} & r_{23} & t_2\\
r_{31} & r_{32} & r_{33} & t_3
\end{bmatrix}
\end{equation}
If
$\mathbf{X}_c =
\begin{bmatrix}
X_c\\
Y_c\\
Z_c
\end{bmatrix}$ and
$\mathbf{X}_w =
\begin{bmatrix}
X_w\\
Y_w\\
Z_w\\
1,
\end{bmatrix}$
then
\begin{equation}
\begin{bmatrix}
X_c\\
Y_c\\
Z_c
\end{bmatrix} =
\begin{bmatrix}
r_{11} & r_{12} & r_{13} & t_1\\
r_{21} & r_{22} & r_{23} & t_2\\
r_{31} & r_{32} & r_{33} & t_3
\end{bmatrix}
\begin{bmatrix}
X_w\\
Y_w\\
Z_w\\
1
\end{bmatrix}
\end{equation}

### Intrinsic camera parameters

These are the parameters that characterize the optical, geometric, and digital characteristics of the camera:
1. Perspective projection (focal length $f$).
2. The transformation between image plane coordinates and pixel coordinates.
3. The geometric distortion introduced by lens optics.

#### From Camera Coordinates to Image Plane Coordinates

We apply the following perspective projection:

$x = f\frac{X_c}{Z_c}$, $y = f\frac{X_c}{Z_c}$.

#### From Image Plane Coordinates to Pixel coordinates

In matrix notation:
\begin{equation}
\begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \propto
\begin{bmatrix} x Z_c \\ y Z_c \\ Z_c \end{bmatrix} =
\begin{bmatrix}
\alpha_x & s & x_0\\
0 & \alpha_y & y_0\\
0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
X_c\\
Y_c\\
Z_c
\end{bmatrix}
\end{equation}

### From world coordinates to pixel coordinates

\begin{equation}
\begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \propto
\begin{bmatrix} x Z_c \\ y Z_c \\ Z_c \end{bmatrix} =
\begin{bmatrix}
\alpha_x & s & x_0\\
0 & \alpha_y & y_0\\
0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
r_{11} & r_{12} & r_{13} & t_1\\
r_{21} & r_{22} & r_{23} & t_2\\
r_{31} & r_{32} & r_{33} & t_3
\end{bmatrix}
\begin{bmatrix}
X_w\\
Y_w\\
Z_w\\
1
\end{bmatrix}
\end{equation}

Finally,
\begin{equation}
\begin{bmatrix}
x\\
y\\
1
\end{bmatrix} \propto
\begin{bmatrix}
m_{11} & m_{12} & m_{13} & m_{14}\\
m_{21} & m_{22} & m_{23} & m_{24}\\
m_{31} & m_{32} & m_{33} & m_{34}
\end{bmatrix}
\begin{bmatrix}
X_w\\
Y_w\\
Z_w\\
1
\end{bmatrix}
\end{equation}

### Image Distortion Due to Lens Optics

As the image input might get curved by the lens, we must rectify pixel coordinates before
attempting to calculate camera coordinate points using $\mathtt{K}^{-1}$.
We assume that $(x_d, y_d)$ is the pixel coordinate position of a point after distortion by the lens
before rectifying the image. The parameters used in OpenCV are
\begin{equation}
x = (x_d - x_c)(1+k_1r^2+k_2r^4 + k_3r^6)
\end{equation}
\begin{equation}
y = (y_d-y_c)(1+k_1r^2+k_2r^4 + k_3r^6)
\end{equation}

When $r^2 = (x_d - x_c)^2+(y_d - y_c)^2$, and $k_1$ and $k_2$ are intrinsic parameters.

The presence of the radial distortion manifests in form of the "barrel" or "fish-eye" effect.

Tangential distortion occurs because the image taking lenses are not perfectly parallel to the imaging plane. It can be represented via the formulas:

\begin{equation}
x = x_d+(2p_1(x_d - x_c)(y_d - x_c) + p_2(r^2+2(x_d-x_c)^2))
\end{equation}
\begin{equation}
y = y_d+(2p_1(r^2+2(y_d - y_c)^2)+2p_2(x_d - x_c)(y_d - y_c))
\end{equation}

finally,

\begin{equation}
x = x_d + (x_d - x_c)(1+k_1r^2+k_2r^4 + k_3r^6) +(2p_1(x_d - x_c)(y_d - x_c) + p_2(r^2+2(x_d-x_c)^2))
\end{equation}
\begin{equation}
y = y_d + (y_d-y_c)(1+k_1r^2+k_2r^4 + k_3r^6) +(2p_1(r^2+2(y_d - y_c)^2)+2p_2(x_d - x_c)(y_d - y_c))
\end{equation}

Thus we have 5 distortion parameters which in OpenCV are represented as a 5-element vector:

\begin{equation}\mathbf{C} = \begin{bmatrix}k1 & k2 & p1 & p2 & k3 \end{bmatrix}^\top\end{equation}

We can make C matrix by:

\begin{equation}
\begin{bmatrix} x \\ y \end{bmatrix} = \mathtt{A} \mathbf{C}'
\end{equation}

Where $\mathbf{C}'$ is
\begin{equation}\mathbf{C}' = \begin{bmatrix} k1 & k2 & p1 & p2 & k3 & 1\end{bmatrix}\end{equation}

and $\mathtt{A}$ is

\begin{equation}
\mathtt{A} =
\begin{bmatrix}
r^2(x_d-x_c) & r^4(x_d-x_c) & 2(x_d - x_c)(y_d - x_c) & (r^2+2(x_d-x_c)^2) & r^6(x_d-x_c) & (2x_d - x_c - x) \\
r^2(y_d-y_c) & r^4(y_d-y_c) & (r^2+2(y_d-y_c)^2) & 2(x_d - x_c)(y_d - x_c) & r^6(y_d-y_c) & (2y_d - y_c - y)
\end{bmatrix}
\end{equation}

## Camera Calibration

The calibration process is explained by a flowchart given below.

### Chessboard calibration

#### 0. Firs of all, save and print the image (Chessboard) for use in camera.

There are many patterns for calibration. However, openCV supports 3 patterns:
- Classical black-white chessboard (Image above)
- Symmetrical circle pattern

- Asymmetrical circle pattern

#### 1. Define real world coordinates with checkerboard pattern

In the process of calibration we calculate the camera parameters by a set of know 3D points $(X_w, Y_w, Z_w)$ and their corresponding pixel location $(u,v)$ in the image.

For the 3D points we photograph a checkerboard pattern with known dimensions at many different orientations. The world coordinate is attached to the checkerboard and since all the corner points lie on a plane, we can arbitrarily choose Z_w for every point to be 0. Since points are equally spaced in the checkerboard, the $(X_w, Y_w)$ coordinates of each 3D point are easily defined by taking one point as reference $(0, 0)$ and defining remaining with respect to that reference point.

Write a program to get the image from camera or video.

If you use web camera, code example is at below:

### C++

```python
#include <opencv2/opencv.hpp>
#include <iostream>

using namespace cv;
using namespace std;
int main()
{
    int frameAdd = 0;
    Mat frame;
    int iKey = -1;
    //--- INITIALIZE VIDEOCAPTURE
    VideoCapture cap;
    // open the default camera using default API
    // cap.open(0);
    // OR advance usage: select any API backend
    int deviceID = 0;             // 0 = open default camera
    int apiID = cv::CAP_ANY;      // 0 = autodetect default API
    // open selected camera using selected API
    cap.open(deviceID, apiID);
    // check if we succeeded
    if (!cap.isOpened()) {
        cerr << "ERROR! Unable to open camera\n";
        return -1;
    }
    //--- GRAB AND WRITE LOOP
    cout << "Start grabbing" << endl
        << "Press s to save images and q to terminate" << endl;
    for (;;)
    {
        // wait for a new frame from camera and store it into 'frame'
        cap.read(frame);
        // check if we succeeded
        if (frame.empty()) {
            cerr << "ERROR! blank frame grabbed\n";
            break;
        }
        // show live and wait for a key with timeout long enough to show images
        imshow("Live", frame);
        iKey = waitKey(5);
        if (iKey == 's' || iKey == 'S')
        {
            imwrite("./images/frame" + to_string(frameAdd) + ".jpg", frame);
            wantFrame[frameAdd] = frame.clone();
            frameAdd++;
            count << "Frame: " << frameAdd << " has been saved." << endl;
        }
        else if (iKey == 'q' || iKey == 'Q')
        {
            break;
        }
    }
    // the camera will be deinitialized automatically in VideoCapture destructor
    return 0;
}
```

### Python

```python
import cv2
import numpy as np
import os
import glob

cap = cv2.VideoCapture()
cap.open(0, cv2.CAP_ANY);
if not cap.isOpened():
    print("ERROR! Unable to open camera\n")
    exit()
print("Start grabbing")
print("Press s to save images and q to terminate")
frameAdd = 0
while True:
    _, frame = cap.read()
    if frame is None:
        print("ERROR! blank frame grabbed\n")
        exit()
    cv2.imshow("Live", frame)
    iKey = cv2.waitKey(5)
    if iKey == ord('s') or iKey == ord('S'):
        cv2.imwrite("./images/frame" + str(frameAdd) + ".jpg", frame)
        frameAdd += 1
        print("Frame: ", frameAdd, " has been saved.")
    elif iKey == ord('q') or iKey == ord('Q'):
        break
```

Run the web camera, you will see the camera with chessboard as below:

####  2. Take several pictures for the checkerboard (generally more than 10)

Save and put them to <code>./images</code>

There are two cases here

    (1) To calibrate the distortion coefficient and camera internal parameters, the photographs need to contain a complete
    chessboard, and at the same time require different distances, different orientations, and different tilt angles of the
    chess board.

    (2) Calibration distortion coefficient, camera internal parameters and camera external parameters. The picture contains
    the above requirements. At the same time, each photo in the calibration program generated results will calculate an
    external camera parameter. Therefore, according to actual needs, add a few photos of the chessboard at the working
    position. (It is recommended to use the solvePnP function to obtain external camera parameters)

#### 3. 1. Find checkerboard corners

OpenCV provides a builtin function called findChessboardCorners that looks for a checkerboard and returns the coordinates of the corners. Let’ see the usage in the code block below. Its usage is given by

### C++

```python
bool findChessboardCorners(InputArray image,
                           Size patternSize,
                           OutputArray corners,
                           int flags=CALIB_CB_ADAPTIVE_THRESH+CALIB_CB_NORMALIZE_IMAGE );
```

### Python

```python
retval, corners = cv2.findChessboardCorners(image, patternSize, flags)
```

| Variable | Meaning |
| :--- | :--- |
| **image** | Source chessboard view. It must be an 8-bit grayscale or color image. |
| **patternSize** | Number of inner corners per a chessboard row and column ( patternSize = cvSize (points_per_row, points_per_colum) = cvSize(columns,rows) ). |
| **corners** | Output array of detected corners. |
| **flags** | Various operation flags. You have to worry about these only when things do not work well. Go with the default. |

####  3. 2. Refine checkerboard corners

Good calibration is all about precision. To get good results it is important to obtain the location of corners with sub-pixel level of accuracy.

OpenCV’s function cornerSubPix takes in the original image, and the location of corners, and looks for the best corner location inside a small neighborhood of the original location. The algorithm is iterative in nature and therefore we need to specify the termination criteria ( e.g. number of iterations and/or the accuracy )

### C++

```python
bool findChessboardCorners(InputArray image,
                           Size patternSize,
                           OutputArray corners,
                           int flags = CALIB_CB_ADAPTIVE_THRESH + CALIB_CB_NORMALIZE_IMAGE );
```

### Python

```python
retval, corners = cv2.findChessboardCorners(image, patternSize, flags)
```

| Variable | Meaning |
| :--- | :--- |
| **image** | Input image. |
| **corners** | Initial coordinates of the input corners and refined coordinates provided for output. |
| **winSize** | Half of the side length of the search window. |
| **zeroZone** | Half of the size of the dead region in the middle of the search zone over which the summation in the formula below is not done. It is used sometimes to avoid possible singularities of the autocorrelation matrix. The value of (-1,-1) indicates that there is no such a size. |
|**criteria** | Criteria for termination of the iterative process of corner refinement. That is, the process of corner position refinement stops either after criteria.maxCount iterations or when the corner position moves by less than criteria.epsilon on some iteration.|

#### 4. Calibrate Camera

The final step of calibration is to pass the 3D points in world coordinates and their 2D locations in all images to OpenCV’s calibrateCamera method. The implementation is based on a paper by Zhengyou Zhang. The math is a bit involved and requires a background in linear algebra.

Let’s look at the syntax for calibrateCamera

### C++

```python
double calibrateCamera(InputArrayOfArrays objectPoints,
                       InputArrayOfArrays imagePoints,
                       Size imageSize,
                       InputOutputArray cameraMatrix,
                       InputOutputArray distCoeffs,
                       OutputArrayOfArrays rvecs,
                       OutputArrayOfArrays tvecs);
```

### Python

```python
retval, cameraMatrix, distCoeffs, rvecs, tvecs = cv2.calibrateCamera(objectPoints, imagePoints, imageSize)
```

| Variable | Meaning |
| :--- | :--- |
| **objectPoints** | A vector of vector of 3D points. The outer vector contains as many elements as the number of the pattern views. |
| **imagePoints** | A vector of vectors of the 2D image points. |
| **imageSize** | Size of the image |
| **cameraMatrix** | Intrinsic camera matrix |
|**distCoeffs** | Lens distortion coefficients. These coefficients will be explained in a future post.|
| **rvecs** | Rotation specified as a 3×1 vector. The direction of the vector specifies the axis of rotation and the magnitude of the vector specifies the angle of rotation. |
| **tvecs** | 3×1 Translation vector. |

### Full code of camera calibration

### C++

```python
#include <opencv2/opencv.hpp>
#include <opencv2/calib3d/calib3d.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <stdio.h>
#include <iostream>

// Defining the dimensions of checkerboard
int CHECKERBOARD[2]{8,11}; // width, height

int main()
{
  // Creating vector to store vectors of 3D points for each checkerboard image
  std::vector<std::vector<cv::Point3f> > objpoints;

  // Creating vector to store vectors of 2D points for each checkerboard image
  std::vector<std::vector<cv::Point2f> > imgpoints;

  // Defining the world coordinates for 3D points
  std::vector<cv::Point3f> objp;
  for(int i = 0; i<CHECKERBOARD[1]; i++)
  {
    for(int j = 0; j<CHECKERBOARD[0]; j++)
      objp.push_back(cv::Point3f(j,i,0));
  }

  // Extracting path of individual image stored in a given directory
  std::vector<cv::String> images;
  // Path of the folder containing checkerboard images
  std::string path = "./images/*.jpg";

  cv::glob(path, images);

  cv::Mat frame, gray;
  // vector to store the pixel coordinates of detected checker board corners
  std::vector<cv::Point2f> corner_pts;
  bool success;

  // Looping over all the images in the directory
  for(int i{0}; i<images.size(); i++)
  {
    frame = cv::imread(images[i]);
    cv::cvtColor(frame,gray,cv::COLOR_BGR2GRAY);

    // Finding checker board corners
    // If desired number of corners are found in the image then success = true
    success = cv::findChessboardCorners(gray, cv::Size(CHECKERBOARD[0], CHECKERBOARD[1]), corner_pts, CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_FAST_CHECK | CV_CALIB_CB_NORMALIZE_IMAGE);

    /*
     * If desired number of corner are detected,
     * we refine the pixel coordinates and display
     * them on the images of checker board
    */
    if(success)
    {
      cv::TermCriteria criteria(CV_TERMCRIT_EPS | CV_TERMCRIT_ITER, 30, 0.001);

      // refining pixel coordinates for given 2d points.
      cv::cornerSubPix(gray,corner_pts,cv::Size(11,11), cv::Size(-1,-1),criteria);

      // Displaying the detected corner points on the checker board
      cv::drawChessboardCorners(frame, cv::Size(CHECKERBOARD[0], CHECKERBOARD[1]), corner_pts, success);

      objpoints.push_back(objp);
      imgpoints.push_back(corner_pts);
    }

    cv::imshow("Image",frame);
    cv::waitKey(0);
  }

  cv::destroyAllWindows();

  cv::Mat cameraMatrix,distCoeffs,R,T;

  /*
   * Performing camera calibration by
   * passing the value of known 3D points (objpoints)
   * and corresponding pixel coordinates of the
   * detected corners (imgpoints)
  */
  cv::calibrateCamera(objpoints, imgpoints, cv::Size(gray.rows,gray.cols), cameraMatrix, distCoeffs, R, T);

  std::cout << "cameraMatrix : " << cameraMatrix << std::endl;
  std::cout << "distCoeffs : " << distCoeffs << std::endl;
  std::cout << "Rotation vector : " << R << std::endl;
  std::cout << "Translation vector : " << T << std::endl;

  return 0;
}
```

### Python

```python
#!/usr/bin/env python

import cv2
import numpy as np
import os
import glob

# Defining the dimensions of checkerboard
CHECKERBOARD = (8,11)
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# Creating vector to store vectors of 3D points for each checkerboard image
objpoints = []
# Creating vector to store vectors of 2D points for each checkerboard image
imgpoints = []

# Defining the world coordinates for 3D points
objp = np.zeros((1, CHECKERBOARD[0] * CHECKERBOARD[1], 3), np.float32)
objp[0,:,:2] = np.mgrid[0:CHECKERBOARD[0], 0:CHECKERBOARD[1]].T.reshape(-1, 2)
prev_img_shape = None

# Extracting path of individual image stored in a given directory
images = glob.glob('./images/*.jpg')
for fname in images:
    img = cv2.imread(fname)
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    # Find the chess board corners
    # If desired number of corners are found in the image then ret = true
    ret, corners = cv2.findChessboardCorners(gray, CHECKERBOARD, cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_FAST_CHECK + cv2.CALIB_CB_NORMALIZE_IMAGE)

    """
    If desired number of corner are detected,
    we refine the pixel coordinates and display
    them on the images of checker board
    """
    if ret == True:
        objpoints.append(objp)
        # refining pixel coordinates for given 2d points.
        corners2 = cv2.cornerSubPix(gray, corners, (11,11),(-1,-1), criteria)

        imgpoints.append(corners2)

        # Draw and display the corners
        img = cv2.drawChessboardCorners(img, CHECKERBOARD, corners2, ret)

    cv2.imshow('img',img)
    cv2.waitKey(0)

cv2.destroyAllWindows()

h,w = img.shape[:2]

"""
Performing camera calibration by
passing the value of known 3D points (objpoints)
and corresponding pixel coordinates of the
detected corners (imgpoints)
"""
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)

print("Camera matrix : \n")
print(mtx)
print("dist : \n")
print(dist)
print("rvecs : \n")
print(rvecs)
print("tvecs : \n")
print(tvecs)

# Show images of undistorted
for fname in images:
    img = cv2.imread(fname)
    res = cv2.undistort(img, mtx, dist)
    cv2.imshow('img',img)
    cv2.imshow('res',res)
    cv2.waitKey(0)
```

### Result of the images

You can see the result like this.

Source image

Chessboard detection

Rectified image

### More fancier in undistorted

After undistorted, the result of rectified image may need to be crops.

### Python

```python
# Way 1: This is the easiest way. Just call the function and use ROI obtained above to crop the result.
for fname in images:
    img = cv2.imread(fname)
    h,  w = img.shape[:2]
    newcameramtx, roi = cv2.getOptimalNewCameraMatrix(mtx, dist, (w,h), 1, (w,h))

    res = cv2.undistort(img, mtx, dist)

    # crop the image
    x, y, w, h = roi
    res = res[y:y+h, x:x+w]

    cv2.imshow('img',img)
    cv2.imshow('res',res)
    cv2.waitKey(0)
```

```python
# Way 2: This way is a little bit more difficult.
# First, find a mapping function from the distorted image to the undistorted image.
# Then use the remap function.

for fname in images:
    img = cv2.imread(fname)
    h,  w = img.shape[:2]
    newcameramtx, roi = cv2.getOptimalNewCameraMatrix(mtx, dist, (w,h), 1, (w,h))

    mapx, mapy = cv2.initUndistortRectifyMap(mtx, dist, None, newcameramtx, (w,h), 5)
    res = cv2.remap(img, mapx, mapy, cv.INTER_LINEAR)

    # crop the image
    x, y, w, h = roi
    res = res[y:y+h, x:x+w]

    cv2.imshow('img',img)
    cv2.imshow('res',res)
    cv2.waitKey(0)
```

### Re-projection Error

Re-projection error gives a good estimation of just how exact the found parameters are. The closer the re-projection error is to zero, the more accurate the parameters we found are. Given the intrinsic, distortion, rotation and translation matrices, we must first transform the object point to image point using cv.projectPoints(). Then, we can calculate the absolute norm between what we got with our transformation and the corner finding algorithm. To find the average error, we calculate the arithmetical mean of the errors calculated for all the calibration images.

### Python

```python
mean_error = 0
for i in range(len(objpoints)):
    imgpoints2, _ = cv.projectPoints(objpoints[i], rvecs[i], tvecs[i], mtx, dist)
    error = cv.norm(imgpoints[i], imgpoints2, cv.NORM_L2)/len(imgpoints2)
    mean_error += error
print( "total error: {}".format(mean_error/len(objpoints)) )
```

### Save calibration file

As the result, save the calibration file into yml file.

### Full code of Calibration from scratch

Please look at https://github.com/opencv/opencv/blob/master/samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp
There is the source code in C++ from scratch. It may useful when you need to write it as commercial product.

### C++

```python
#include <iostream>
#include <sstream>
#include <string>
#include <ctime>
#include <cstdio>

#include <opencv2/core.hpp>
#include <opencv2/core/utility.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/calib3d.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>

using namespace cv;
using namespace std;

class Settings
{
public:
    Settings() : goodInput(false) {}
    enum Pattern { NOT_EXISTING, CHESSBOARD, CIRCLES_GRID, ASYMMETRIC_CIRCLES_GRID };
    enum InputType { INVALID, CAMERA, VIDEO_FILE, IMAGE_LIST };

    void write(FileStorage& fs) const                        //Write serialization for this class
    {
        fs << "{"
                  << "BoardSize_Width"  << boardSize.width
                  << "BoardSize_Height" << boardSize.height
                  << "Square_Size"         << squareSize
                  << "Calibrate_Pattern" << patternToUse
                  << "Calibrate_NrOfFrameToUse" << nrFrames
                  << "Calibrate_FixAspectRatio" << aspectRatio
                  << "Calibrate_AssumeZeroTangentialDistortion" << calibZeroTangentDist
                  << "Calibrate_FixPrincipalPointAtTheCenter" << calibFixPrincipalPoint

                  << "Write_DetectedFeaturePoints" << writePoints
                  << "Write_extrinsicParameters"   << writeExtrinsics
                  << "Write_gridPoints" << writeGrid
                  << "Write_outputFileName"  << outputFileName

                  << "Show_UndistortedImage" << showUndistorted

                  << "Input_FlipAroundHorizontalAxis" << flipVertical
                  << "Input_Delay" << delay
                  << "Input" << input
           << "}";
    }
    void read(const FileNode& node)                          //Read serialization for this class
    {
        node["BoardSize_Width" ] >> boardSize.width;
        node["BoardSize_Height"] >> boardSize.height;
        node["Calibrate_Pattern"] >> patternToUse;
        node["Square_Size"]  >> squareSize;
        node["Calibrate_NrOfFrameToUse"] >> nrFrames;
        node["Calibrate_FixAspectRatio"] >> aspectRatio;
        node["Write_DetectedFeaturePoints"] >> writePoints;
        node["Write_extrinsicParameters"] >> writeExtrinsics;
        node["Write_gridPoints"] >> writeGrid;
        node["Write_outputFileName"] >> outputFileName;
        node["Calibrate_AssumeZeroTangentialDistortion"] >> calibZeroTangentDist;
        node["Calibrate_FixPrincipalPointAtTheCenter"] >> calibFixPrincipalPoint;
        node["Calibrate_UseFisheyeModel"] >> useFisheye;
        node["Input_FlipAroundHorizontalAxis"] >> flipVertical;
        node["Show_UndistortedImage"] >> showUndistorted;
        node["Input"] >> input;
        node["Input_Delay"] >> delay;
        node["Fix_K1"] >> fixK1;
        node["Fix_K2"] >> fixK2;
        node["Fix_K3"] >> fixK3;
        node["Fix_K4"] >> fixK4;
        node["Fix_K5"] >> fixK5;

        validate();
    }
    void validate()
    {
        goodInput = true;
        if (boardSize.width <= 0 || boardSize.height <= 0)
        {
            cerr << "Invalid Board size: " << boardSize.width << " " << boardSize.height << endl;
            goodInput = false;
        }
        if (squareSize <= 10e-6)
        {
            cerr << "Invalid square size " << squareSize << endl;
            goodInput = false;
        }
        if (nrFrames <= 0)
        {
            cerr << "Invalid number of frames " << nrFrames << endl;
            goodInput = false;
        }

        if (input.empty())      // Check for valid input
                inputType = INVALID;
        else
        {
            if (input[0] >= '0' && input[0] <= '9')
            {
                stringstream ss(input);
                ss >> cameraID;
                inputType = CAMERA;
            }
            else
            {
                if (isListOfImages(input) && readStringList(input, imageList))
                {
                    inputType = IMAGE_LIST;
                    nrFrames = (nrFrames < (int)imageList.size()) ? nrFrames : (int)imageList.size();
                }
                else
                    inputType = VIDEO_FILE;
            }
            if (inputType == CAMERA)
                inputCapture.open(cameraID);
            if (inputType == VIDEO_FILE)
                inputCapture.open(input);
            if (inputType != IMAGE_LIST && !inputCapture.isOpened())
                    inputType = INVALID;
        }
        if (inputType == INVALID)
        {
            cerr << " Input does not exist: " << input;
            goodInput = false;
        }

        flag = 0;
        if(calibFixPrincipalPoint) flag |= CALIB_FIX_PRINCIPAL_POINT;
        if(calibZeroTangentDist)   flag |= CALIB_ZERO_TANGENT_DIST;
        if(aspectRatio)            flag |= CALIB_FIX_ASPECT_RATIO;
        if(fixK1)                  flag |= CALIB_FIX_K1;
        if(fixK2)                  flag |= CALIB_FIX_K2;
        if(fixK3)                  flag |= CALIB_FIX_K3;
        if(fixK4)                  flag |= CALIB_FIX_K4;
        if(fixK5)                  flag |= CALIB_FIX_K5;

        if (useFisheye) {
            // the fisheye model has its own enum, so overwrite the flags
            flag = fisheye::CALIB_FIX_SKEW | fisheye::CALIB_RECOMPUTE_EXTRINSIC;
            if(fixK1)                   flag |= fisheye::CALIB_FIX_K1;
            if(fixK2)                   flag |= fisheye::CALIB_FIX_K2;
            if(fixK3)                   flag |= fisheye::CALIB_FIX_K3;
            if(fixK4)                   flag |= fisheye::CALIB_FIX_K4;
            if (calibFixPrincipalPoint) flag |= fisheye::CALIB_FIX_PRINCIPAL_POINT;
        }

        calibrationPattern = NOT_EXISTING;
        if (!patternToUse.compare("CHESSBOARD")) calibrationPattern = CHESSBOARD;
        if (!patternToUse.compare("CIRCLES_GRID")) calibrationPattern = CIRCLES_GRID;
        if (!patternToUse.compare("ASYMMETRIC_CIRCLES_GRID")) calibrationPattern = ASYMMETRIC_CIRCLES_GRID;
        if (calibrationPattern == NOT_EXISTING)
        {
            cerr << " Camera calibration mode does not exist: " << patternToUse << endl;
            goodInput = false;
        }
        atImageList = 0;

    }
    Mat nextImage()
    {
        Mat result;
        if( inputCapture.isOpened() )
        {
            Mat view0;
            inputCapture >> view0;
            view0.copyTo(result);
        }
        else if( atImageList < imageList.size() )
            result = imread(imageList[atImageList++], IMREAD_COLOR);

        return result;
    }

    static bool readStringList( const string& filename, vector<string>& l )
    {
        l.clear();
        FileStorage fs(filename, FileStorage::READ);
        if( !fs.isOpened() )
            return false;
        FileNode n = fs.getFirstTopLevelNode();
        if( n.type() != FileNode::SEQ )
            return false;
        FileNodeIterator it = n.begin(), it_end = n.end();
        for( ; it != it_end; ++it )
            l.push_back((string)*it);
        return true;
    }

    static bool isListOfImages( const string& filename)
    {
        string s(filename);
        // Look for file extension
        if( s.find(".xml") == string::npos && s.find(".yaml") == string::npos && s.find(".yml") == string::npos )
            return false;
        else
            return true;
    }
public:
    Size boardSize;              // The size of the board -> Number of items by width and height
    Pattern calibrationPattern;  // One of the Chessboard, circles, or asymmetric circle pattern
    float squareSize;            // The size of a square in your defined unit (point, millimeter,etc).
    int nrFrames;                // The number of frames to use from the input for calibration
    float aspectRatio;           // The aspect ratio
    int delay;                   // In case of a video input
    bool writePoints;            // Write detected feature points
    bool writeExtrinsics;        // Write extrinsic parameters
    bool writeGrid;              // Write refined 3D target grid points
    bool calibZeroTangentDist;   // Assume zero tangential distortion
    bool calibFixPrincipalPoint; // Fix the principal point at the center
    bool flipVertical;           // Flip the captured images around the horizontal axis
    string outputFileName;       // The name of the file where to write
    bool showUndistorted;        // Show undistorted images after calibration
    string input;                // The input ->
    bool useFisheye;             // use fisheye camera model for calibration
    bool fixK1;                  // fix K1 distortion coefficient
    bool fixK2;                  // fix K2 distortion coefficient
    bool fixK3;                  // fix K3 distortion coefficient
    bool fixK4;                  // fix K4 distortion coefficient
    bool fixK5;                  // fix K5 distortion coefficient

    int cameraID;
    vector<string> imageList;
    size_t atImageList;
    VideoCapture inputCapture;
    InputType inputType;
    bool goodInput;
    int flag;

private:
    string patternToUse;

};

static inline void read(const FileNode& node, Settings& x, const Settings& default_value = Settings())
{
    if(node.empty())
        x = default_value;
    else
        x.read(node);
}

enum { DETECTION = 0, CAPTURING = 1, CALIBRATED = 2 };

bool runCalibrationAndSave(Settings& s, Size imageSize, Mat&  cameraMatrix, Mat& distCoeffs,
                           vector<vector<Point2f> > imagePoints, float grid_width, bool release_object);

int main(int argc, char* argv[])
{
    const String keys
        = "{help h usage ? |           | print this message            }"
          "{@settings      |default.xml| input setting file            }"
          "{d              |           | actual distance between top-left and top-right corners of "
          "the calibration grid }"
          "{winSize        | 11        | Half of search window for cornerSubPix }";
    CommandLineParser parser(argc, argv, keys);
    parser.about("This is a camera calibration sample.\n"
                 "Usage: camera_calibration [configuration_file -- default ./default.xml]\n"
                 "Near the sample file you'll find the configuration file, which has detailed help of "
                 "how to edit it. It may be any OpenCV supported file format XML/YAML.");
    if (!parser.check()) {
        parser.printErrors();
        return 0;
    }

    if (parser.has("help")) {
        parser.printMessage();
        return 0;
    }

    //! [file_read]
    Settings s;
    const string inputSettingsFile = parser.get<string>(0);
    FileStorage fs(inputSettingsFile, FileStorage::READ); // Read the settings
    if (!fs.isOpened())
    {
        cout << "Could not open the configuration file: \"" << inputSettingsFile << "\"" << endl;
        parser.printMessage();
        return -1;
    }
    fs["Settings"] >> s;
    fs.release();                                         // close Settings file
    //! [file_read]

    //FileStorage fout("settings.yml", FileStorage::WRITE); // write config as YAML
    //fout << "Settings" << s;

    if (!s.goodInput)
    {
        cout << "Invalid input detected. Application stopping. " << endl;
        return -1;
    }

    int winSize = parser.get<int>("winSize");

    float grid_width = s.squareSize * (s.boardSize.width - 1);
    bool release_object = false;
    if (parser.has("d")) {
        grid_width = parser.get<float>("d");
        release_object = true;
    }

    vector<vector<Point2f> > imagePoints;
    Mat cameraMatrix, distCoeffs;
    Size imageSize;
    int mode = s.inputType == Settings::IMAGE_LIST ? CAPTURING : DETECTION;
    clock_t prevTimestamp = 0;
    const Scalar RED(0,0,255), GREEN(0,255,0);
    const char ESC_KEY = 27;

    //! [get_input]
    for(;;)
    {
        Mat view;
        bool blinkOutput = false;

        view = s.nextImage();

        //-----  If no more image, or got enough, then stop calibration and show result -------------
        if( mode == CAPTURING && imagePoints.size() >= (size_t)s.nrFrames )
        {
          if(runCalibrationAndSave(s, imageSize,  cameraMatrix, distCoeffs, imagePoints, grid_width,
                                   release_object))
              mode = CALIBRATED;
          else
              mode = DETECTION;
        }
        if(view.empty())          // If there are no more images stop the loop
        {
            // if calibration threshold was not reached yet, calibrate now
            if( mode != CALIBRATED && !imagePoints.empty() )
                runCalibrationAndSave(s, imageSize,  cameraMatrix, distCoeffs, imagePoints, grid_width,
                                      release_object);
            break;
        }
        //! [get_input]

        imageSize = view.size();  // Format input image.
        if( s.flipVertical )    flip( view, view, 0 );

        //! [find_pattern]
        vector<Point2f> pointBuf;

        bool found;

        int chessBoardFlags = CALIB_CB_ADAPTIVE_THRESH | CALIB_CB_NORMALIZE_IMAGE;

        if(!s.useFisheye) {
            // fast check erroneously fails with high distortions like fisheye
            chessBoardFlags |= CALIB_CB_FAST_CHECK;
        }

        switch( s.calibrationPattern ) // Find feature points on the input format
        {
        case Settings::CHESSBOARD:
            found = findChessboardCorners( view, s.boardSize, pointBuf, chessBoardFlags);
            break;
        case Settings::CIRCLES_GRID:
            found = findCirclesGrid( view, s.boardSize, pointBuf );
            break;
        case Settings::ASYMMETRIC_CIRCLES_GRID:
            found = findCirclesGrid( view, s.boardSize, pointBuf, CALIB_CB_ASYMMETRIC_GRID );
            break;
        default:
            found = false;
            break;
        }
        //! [find_pattern]
        //! [pattern_found]
        if ( found)                // If done with success,
        {
              // improve the found corners' coordinate accuracy for chessboard
                if( s.calibrationPattern == Settings::CHESSBOARD)
                {
                    Mat viewGray;
                    cvtColor(view, viewGray, COLOR_BGR2GRAY);
                    cornerSubPix( viewGray, pointBuf, Size(winSize,winSize),
                        Size(-1,-1), TermCriteria( TermCriteria::EPS+TermCriteria::COUNT, 30, 0.0001 ));
                }

                if( mode == CAPTURING &&  // For camera only take new samples after delay time
                    (!s.inputCapture.isOpened() || clock() - prevTimestamp > s.delay*1e-3*CLOCKS_PER_SEC) )
                {
                    imagePoints.push_back(pointBuf);
                    prevTimestamp = clock();
                    blinkOutput = s.inputCapture.isOpened();
                }

                // Draw the corners.
                drawChessboardCorners( view, s.boardSize, Mat(pointBuf), found );
        }
        //! [pattern_found]
        //----------------------------- Output Text ------------------------------------------------
        //! [output_text]
        string msg = (mode == CAPTURING) ? "100/100" :
                      mode == CALIBRATED ? "Calibrated" : "Press 'g' to start";
        int baseLine = 0;
        Size textSize = getTextSize(msg, 1, 1, 1, &baseLine);
        Point textOrigin(view.cols - 2*textSize.width - 10, view.rows - 2*baseLine - 10);

        if( mode == CAPTURING )
        {
            if(s.showUndistorted)
                msg = cv::format( "%d/%d Undist", (int)imagePoints.size(), s.nrFrames );
            else
                msg = cv::format( "%d/%d", (int)imagePoints.size(), s.nrFrames );
        }

        putText( view, msg, textOrigin, 1, 1, mode == CALIBRATED ?  GREEN : RED);

        if( blinkOutput )
            bitwise_not(view, view);
        //! [output_text]
        //------------------------- Video capture  output  undistorted ------------------------------
        //! [output_undistorted]
        if( mode == CALIBRATED && s.showUndistorted )
        {
            Mat temp = view.clone();
            if (s.useFisheye)
            {
                Mat newCamMat;
                fisheye::estimateNewCameraMatrixForUndistortRectify(cameraMatrix, distCoeffs, imageSize,
                                                                    Matx33d::eye(), newCamMat, 1);
                cv::fisheye::undistortImage(temp, view, cameraMatrix, distCoeffs, newCamMat);
            }
            else
              undistort(temp, view, cameraMatrix, distCoeffs);
        }
        //! [output_undistorted]
        //------------------------------ Show image and check for input commands -------------------
        //! [await_input]
        imshow("Image View", view);
        char key = (char)waitKey(s.inputCapture.isOpened() ? 50 : s.delay);

        if( key  == ESC_KEY )
            break;

        if( key == 'u' && mode == CALIBRATED )
           s.showUndistorted = !s.showUndistorted;

        if( s.inputCapture.isOpened() && key == 'g' )
        {
            mode = CAPTURING;
            imagePoints.clear();
        }
        //! [await_input]
    }

    // -----------------------Show the undistorted image for the image list ------------------------
    //! [show_results]
    if( s.inputType == Settings::IMAGE_LIST && s.showUndistorted && !cameraMatrix.empty())
    {
        Mat view, rview, map1, map2;

        if (s.useFisheye)
        {
            Mat newCamMat;
            fisheye::estimateNewCameraMatrixForUndistortRectify(cameraMatrix, distCoeffs, imageSize,
                                                                Matx33d::eye(), newCamMat, 1);
            fisheye::initUndistortRectifyMap(cameraMatrix, distCoeffs, Matx33d::eye(), newCamMat, imageSize,
                                             CV_16SC2, map1, map2);
        }
        else
        {
            initUndistortRectifyMap(
                cameraMatrix, distCoeffs, Mat(),
                getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize, 1, imageSize, 0), imageSize,
                CV_16SC2, map1, map2);
        }

        for(size_t i = 0; i < s.imageList.size(); i++ )
        {
            view = imread(s.imageList[i], IMREAD_COLOR);
            if(view.empty())
                continue;
            remap(view, rview, map1, map2, INTER_LINEAR);
            imshow("Image View", rview);
            char c = (char)waitKey();
            if( c  == ESC_KEY || c == 'q' || c == 'Q' )
                break;
        }
    }
    //! [show_results]

    return 0;
}

//! [compute_errors]
static double computeReprojectionErrors( const vector<vector<Point3f> >& objectPoints,
                                         const vector<vector<Point2f> >& imagePoints,
                                         const vector<Mat>& rvecs, const vector<Mat>& tvecs,
                                         const Mat& cameraMatrix , const Mat& distCoeffs,
                                         vector<float>& perViewErrors, bool fisheye)
{
    vector<Point2f> imagePoints2;
    size_t totalPoints = 0;
    double totalErr = 0, err;
    perViewErrors.resize(objectPoints.size());

    for(size_t i = 0; i < objectPoints.size(); ++i )
    {
        if (fisheye)
        {
            fisheye::projectPoints(objectPoints[i], imagePoints2, rvecs[i], tvecs[i], cameraMatrix,
                                   distCoeffs);
        }
        else
        {
            projectPoints(objectPoints[i], rvecs[i], tvecs[i], cameraMatrix, distCoeffs, imagePoints2);
        }
        err = norm(imagePoints[i], imagePoints2, NORM_L2);

        size_t n = objectPoints[i].size();
        perViewErrors[i] = (float) std::sqrt(err*err/n);
        totalErr        += err*err;
        totalPoints     += n;
    }

    return std::sqrt(totalErr/totalPoints);
}
//! [compute_errors]
//! [board_corners]
static void calcBoardCornerPositions(Size boardSize, float squareSize, vector<Point3f>& corners,
                                     Settings::Pattern patternType /*= Settings::CHESSBOARD*/)
{
    corners.clear();

    switch(patternType)
    {
    case Settings::CHESSBOARD:
    case Settings::CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; ++i )
            for( int j = 0; j < boardSize.width; ++j )
                corners.push_back(Point3f(j*squareSize, i*squareSize, 0));
        break;

    case Settings::ASYMMETRIC_CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; i++ )
            for( int j = 0; j < boardSize.width; j++ )
                corners.push_back(Point3f((2*j + i % 2)*squareSize, i*squareSize, 0));
        break;
    default:
        break;
    }
}
//! [board_corners]
static bool runCalibration( Settings& s, Size& imageSize, Mat& cameraMatrix, Mat& distCoeffs,
                            vector<vector<Point2f> > imagePoints, vector<Mat>& rvecs, vector<Mat>& tvecs,
                            vector<float>& reprojErrs,  double& totalAvgErr, vector<Point3f>& newObjPoints,
                            float grid_width, bool release_object)
{
    //! [fixed_aspect]
    cameraMatrix = Mat::eye(3, 3, CV_64F);
    if( !s.useFisheye && s.flag & CALIB_FIX_ASPECT_RATIO )
        cameraMatrix.at<double>(0,0) = s.aspectRatio;
    //! [fixed_aspect]
    if (s.useFisheye) {
        distCoeffs = Mat::zeros(4, 1, CV_64F);
    } else {
        distCoeffs = Mat::zeros(8, 1, CV_64F);
    }

    vector<vector<Point3f> > objectPoints(1);
    calcBoardCornerPositions(s.boardSize, s.squareSize, objectPoints[0], s.calibrationPattern);
    objectPoints[0][s.boardSize.width - 1].x = objectPoints[0][0].x + grid_width;
    newObjPoints = objectPoints[0];

    objectPoints.resize(imagePoints.size(),objectPoints[0]);

    //Find intrinsic and extrinsic camera parameters
    double rms;

    if (s.useFisheye) {
        Mat _rvecs, _tvecs;
        rms = fisheye::calibrate(objectPoints, imagePoints, imageSize, cameraMatrix, distCoeffs, _rvecs,
                                 _tvecs, s.flag);

        rvecs.reserve(_rvecs.rows);
        tvecs.reserve(_tvecs.rows);
        for(int i = 0; i < int(objectPoints.size()); i++){
            rvecs.push_back(_rvecs.row(i));
            tvecs.push_back(_tvecs.row(i));
        }
    } else {
        int iFixedPoint = -1;
        if (release_object)
            iFixedPoint = s.boardSize.width - 1;
        rms = calibrateCameraRO(objectPoints, imagePoints, imageSize, iFixedPoint,
                                cameraMatrix, distCoeffs, rvecs, tvecs, newObjPoints,
                                s.flag | CALIB_USE_LU);
    }

    if (release_object) {
        cout << "New board corners: " << endl;
        cout << newObjPoints[0] << endl;
        cout << newObjPoints[s.boardSize.width - 1] << endl;
        cout << newObjPoints[s.boardSize.width * (s.boardSize.height - 1)] << endl;
        cout << newObjPoints.back() << endl;
    }

    cout << "Re-projection error reported by calibrateCamera: "<< rms << endl;

    bool ok = checkRange(cameraMatrix) && checkRange(distCoeffs);

    objectPoints.clear();
    objectPoints.resize(imagePoints.size(), newObjPoints);
    totalAvgErr = computeReprojectionErrors(objectPoints, imagePoints, rvecs, tvecs, cameraMatrix,
                                            distCoeffs, reprojErrs, s.useFisheye);

    return ok;
}

// Print camera parameters to the output file
static void saveCameraParams( Settings& s, Size& imageSize, Mat& cameraMatrix, Mat& distCoeffs,
                              const vector<Mat>& rvecs, const vector<Mat>& tvecs,
                              const vector<float>& reprojErrs, const vector<vector<Point2f> >& imagePoints,
                              double totalAvgErr, const vector<Point3f>& newObjPoints )
{
    FileStorage fs( s.outputFileName, FileStorage::WRITE );

    time_t tm;
    time( &tm );
    struct tm *t2 = localtime( &tm );
    char buf[1024];
    strftime( buf, sizeof(buf), "%c", t2 );

    fs << "calibration_time" << buf;

    if( !rvecs.empty() || !reprojErrs.empty() )
        fs << "nr_of_frames" << (int)std::max(rvecs.size(), reprojErrs.size());
    fs << "image_width" << imageSize.width;
    fs << "image_height" << imageSize.height;
    fs << "board_width" << s.boardSize.width;
    fs << "board_height" << s.boardSize.height;
    fs << "square_size" << s.squareSize;

    if( !s.useFisheye && s.flag & CALIB_FIX_ASPECT_RATIO )
        fs << "fix_aspect_ratio" << s.aspectRatio;

    if (s.flag)
    {
        std::stringstream flagsStringStream;
        if (s.useFisheye)
        {
            flagsStringStream << "flags:"
                << (s.flag & fisheye::CALIB_FIX_SKEW ? " +fix_skew" : "")
                << (s.flag & fisheye::CALIB_FIX_K1 ? " +fix_k1" : "")
                << (s.flag & fisheye::CALIB_FIX_K2 ? " +fix_k2" : "")
                << (s.flag & fisheye::CALIB_FIX_K3 ? " +fix_k3" : "")
                << (s.flag & fisheye::CALIB_FIX_K4 ? " +fix_k4" : "")
                << (s.flag & fisheye::CALIB_RECOMPUTE_EXTRINSIC ? " +recompute_extrinsic" : "");
        }
        else
        {
            flagsStringStream << "flags:"
                << (s.flag & CALIB_USE_INTRINSIC_GUESS ? " +use_intrinsic_guess" : "")
                << (s.flag & CALIB_FIX_ASPECT_RATIO ? " +fix_aspectRatio" : "")
                << (s.flag & CALIB_FIX_PRINCIPAL_POINT ? " +fix_principal_point" : "")
                << (s.flag & CALIB_ZERO_TANGENT_DIST ? " +zero_tangent_dist" : "")
                << (s.flag & CALIB_FIX_K1 ? " +fix_k1" : "")
                << (s.flag & CALIB_FIX_K2 ? " +fix_k2" : "")
                << (s.flag & CALIB_FIX_K3 ? " +fix_k3" : "")
                << (s.flag & CALIB_FIX_K4 ? " +fix_k4" : "")
                << (s.flag & CALIB_FIX_K5 ? " +fix_k5" : "");
        }
        fs.writeComment(flagsStringStream.str());
    }

    fs << "flags" << s.flag;

    fs << "fisheye_model" << s.useFisheye;

    fs << "camera_matrix" << cameraMatrix;
    fs << "distortion_coefficients" << distCoeffs;

    fs << "avg_reprojection_error" << totalAvgErr;
    if (s.writeExtrinsics && !reprojErrs.empty())
        fs << "per_view_reprojection_errors" << Mat(reprojErrs);

    if(s.writeExtrinsics && !rvecs.empty() && !tvecs.empty() )
    {
        CV_Assert(rvecs[0].type() == tvecs[0].type());
        Mat bigmat((int)rvecs.size(), 6, CV_MAKETYPE(rvecs[0].type(), 1));
        bool needReshapeR = rvecs[0].depth() != 1 ? true : false;
        bool needReshapeT = tvecs[0].depth() != 1 ? true : false;

        for( size_t i = 0; i < rvecs.size(); i++ )
        {
            Mat r = bigmat(Range(int(i), int(i+1)), Range(0,3));
            Mat t = bigmat(Range(int(i), int(i+1)), Range(3,6));

            if(needReshapeR)
                rvecs[i].reshape(1, 1).copyTo(r);
            else
            {
                //*.t() is MatExpr (not Mat) so we can use assignment operator
                CV_Assert(rvecs[i].rows == 3 && rvecs[i].cols == 1);
                r = rvecs[i].t();
            }

            if(needReshapeT)
                tvecs[i].reshape(1, 1).copyTo(t);
            else
            {
                CV_Assert(tvecs[i].rows == 3 && tvecs[i].cols == 1);
                t = tvecs[i].t();
            }
        }
        fs.writeComment("a set of 6-tuples (rotation vector + translation vector) for each view");
        fs << "extrinsic_parameters" << bigmat;
    }

    if(s.writePoints && !imagePoints.empty() )
    {
        Mat imagePtMat((int)imagePoints.size(), (int)imagePoints[0].size(), CV_32FC2);
        for( size_t i = 0; i < imagePoints.size(); i++ )
        {
            Mat r = imagePtMat.row(int(i)).reshape(2, imagePtMat.cols);
            Mat imgpti(imagePoints[i]);
            imgpti.copyTo(r);
        }
        fs << "image_points" << imagePtMat;
    }

    if( s.writeGrid && !newObjPoints.empty() )
    {
        fs << "grid_points" << newObjPoints;
    }
}

//! [run_and_save]
bool runCalibrationAndSave(Settings& s, Size imageSize, Mat& cameraMatrix, Mat& distCoeffs,
                           vector<vector<Point2f> > imagePoints, float grid_width, bool release_object)
{
    vector<Mat> rvecs, tvecs;
    vector<float> reprojErrs;
    double totalAvgErr = 0;
    vector<Point3f> newObjPoints;

    bool ok = runCalibration(s, imageSize, cameraMatrix, distCoeffs, imagePoints, rvecs, tvecs, reprojErrs,
                             totalAvgErr, newObjPoints, grid_width, release_object);
    cout << (ok ? "Calibration succeeded" : "Calibration failed")
         << ". avg re projection error = " << totalAvgErr << endl;

    if (ok)
        saveCameraParams(s, imageSize, cameraMatrix, distCoeffs, rvecs, tvecs, reprojErrs, imagePoints,
                         totalAvgErr, newObjPoints);
    return ok;
}
//! [run_and_save]
```

## Lab and independent exercises

1. Write code to save and read the calibration file.
2. Get a video of your own chessboard, calibrate your camera, and re-calculate the homography again. Then write code to rectify images in a video and show them side
   by side.
3. Using calibration images for the wide-angle Raspberry Pi camera on Matt's home robot, calibrate the camera then use the calibration to undistort the
   videos provided from the camera thus far.
4. Re-run your bird's-eye rectification from Lab 02 using the undistored images. Do you get a better result?

## Report

As always, turn in a report on your experiments and results by next lab.

```python

```