# Assignment 02 Companion Notebook

This notebook contains some exercises to walk you through implementing the linear regression algorithm.  We'll pay special attention to debugging and visualization as we go along.

## A Toy Linear Regression Problem Revisited

As we discovered in the last assignment, the idea of a toy problem is very useful for validating that a machine learning algorithm is working as it is intended to.  Recall the following basic setup and role for a toy problem:

> Suppose you are given a learning algorithm designed to estimate some model parameters $\textbf{w}$ from some training data.
>
> 1.  Generate values for the model parameters $\mathbf{w}$ (e.g., set them to some known values or generate them randomly).  If you were applying your algorithm to real data, you would of course not know these parameters, but instead estimate them from data.  For our toy problem we'll proceed with values that we generate so we can test our algorithm more easily.
>
> 2.  Generate some training input data, $\mathbf{X}$, (random numbers work well for this).  Generate the training output data, $\mathbf{y}$, by applying the model with parameters $\mathbf{w}$.  For example, in a linear regression problem since $\mathbf{w}$ represents the regression coefficients, then we can generate each training label, $y_i$ as $y_i = \mathbf{x_i}^\top \mathbf{w}$.
>
> 3.  Run your learning algorithm on the synthesized training data $\mathbf{X}, \mathbf{y}$ to arrive at estimated values of the model parameters, $\hat{\mathbf{w}}$.
>
> 4.  Compare $\mathbf{w}$ and $\hat{\mathbf{w}}$ as a way of understanding whether your learning algorithm is working.

In the next code block, you'll see an example of a toy regression problem where we set $\mathbf{w} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ and generate some training data.  To make the data a little more interesting, we'll add some noise to the training outputs (you saw this in the last assignment).  We'll also visualize the training data.

```python
%matplotlib inline
from mpl_toolkits.mplot3d import Axes3D
import numpy as np
import matplotlib.pyplot as plt

n_points = 50
X = np.random.randn(n_points,2)
w_true = np.array([1, 2])
# we'll apply a Gaussian noise with a standard deviation of 0.5 to the outputs to make it more interesting
y = X.dot(w_true) + np.random.randn(n_points,)*0.5

fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111, projection='3d')

ax.scatter(X[:,0], X[:,1], y)
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('y')

plt.show()
```

### *Notebook Exercise 1 (40 minutes)*

Before implementing the algorithm you derived for computing $\mathbf{w}^\star$, let's create a visualization of the sum of squared errors as a function of the entries of $\mathbf{w}$. You should recall from exercises in the assignment document that the sum of squared errors for a particular value of $\mathbf{w}$ can be written as $\left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left (\mathbf{X}\mathbf{w} - \mathbf{y} \right)$

(a) Write a function called `sum_of_squared_errors` that takes the parameters `X`, `y`, and `w` and returns the sum of squared errors that this particular value of `w` incurs on the training data `X`, `y`.  We have included a skeletal outline of the function along with a unit test (SoftDes flashback!!).

(b) Run the visualization code in the cell below and interpret the resulting output.  What do the contour lines represent in the generated plot?  Based on the visualization, where is the optimal value of `w` (the one that minizes the squared error)?  Does this agree with the setup of the toy problem?  If not, why doesn't it match?

```python
def sum_of_squared_errors(X, y, w):
    """
    Return the sum of squared errors for the given training data (X, y) and
    model parameters w.

    >>> sum_of_squared_errors(np.array([[1, 4, 3],\
                                        [2, -1, 4]]),\
                              np.array([3, 4]),\
                              np.array([1, 2, 3]))
    289
    """
    err = X.dot(w) - y
    return err.dot(err)

import doctest
doctest.testmod()
```

    TestResults(failed=0, attempted=1)

```python
w1 = np.linspace(-2, 4, 50)
w2 = np.linspace(-2, 4, 50)

W1, W2 = np.meshgrid(w1, w2)
E = np.array([[sum_of_squared_errors(X, y, np.array([W1[i, j], W2[i, j]])) \
               for j in range(W1.shape[1])] \
              for i in range(W1.shape[0])])
fig, ax = plt.subplots(figsize=(8,8))
CS = ax.contour(W1, W2, E, colors='black', levels=20)
ax.clabel(CS, inline=1, fontsize=10)
plt.xlabel('$w_1$')
plt.ylabel('$w_2$')
plt.title('Sum of Squared Errors')
plt.show()
```

The above is a contour plot of the sum of squared errors for different weight arrays. W_1 and W_2 form a meshgrid from which an array of weights is chosen - for each different weight array, the sum of squared errors is calculated. These errors are displayed together in a contour plot, with the error values determining the 'elevation' of the contour plot.

#### *Expand for Solution*

```python
# ***Solution***
def sum_of_squared_errors(X, y, w):
    """
    Return the sum of squared errors for the given training data (X, y) and
    model parameters w.

    >>> sum_of_squared_errors(np.array([[1, 4, 3],\
                                        [2, -1, 4]]),\
                              np.array([3, 4]),\
                              np.array([1, 2, 3]))
    289
    """
    e = X.dot(w) - y
    return e.dot(e)

import doctest
doctest.testmod()
```

```python
# ***Solution***
w1 = np.linspace(-2, 4, 50)
w2 = np.linspace(-2, 4, 50)

W1, W2 = np.meshgrid(w1, w2)
E = np.array([[sum_of_squared_errors(X, y, np.array([W1[i, j], W2[i, j]])) \
               for j in range(W1.shape[1])] \
              for i in range(W1.shape[0])])
fig, ax = plt.subplots(figsize=(8,8))
CS = ax.contour(W1, W2, E, colors='black', levels=20)
ax.clabel(CS, inline=1, fontsize=10)
plt.xlabel('$w_1$')
plt.ylabel('$w_2$')
plt.title('Sum of Squared Errors')
plt.show()
```

***Solution***

(b) The contour lines represent values of $\mathbf{w}$ that incur equal squared error on the training set.  The optimal value of $\mathbf{w}$ (the one that minimizes the error) occurs near $w_1 = 1, w_2 = 2$, which is what we'd expect given the setup of the toy problem.

## Computing the Optimal Weights

Now you're ready to implement the formula that you derived in the assignment document.  In that document you should have arrived at the following formula for the optimal weights:

$$\mathbf{w^\star} = \left ( \mathbf{X}^\top \mathbf{X} \right)^{-1} \mathbf{X}^\top \mathbf{y}$$

### Notebook Exercise 2 (20 minutes)

Fill in the body of the function `optimal_weights` below.  You've done the hard work to derive this beautiful expression, translating it to code is the last step to glory!  Hint: `np.linalg.inv` computes the inverse of a specified matrix.  We've included code that will run your code on the training data.  Does your code compute sensible values of $\mathbf{w}$ given the setup of the toy problem?

```python
def optimal_weights(X, y):
     """ Returns the optimal weights in the least squares sense for the specified
         training inputs (X) and training outputs (y) """
     return np.linalg.inv(X.T.dot(X)).dot(X.T.dot(y))

optimal_weights(X, y)
```

    array([1.04718477, 1.97927105])

Yes, the solution makes sense, as the optimal weights are very close to the true weights. The difference between the values is also acceptable, as a certain amount of noise was added to the input.

#### Expand for Solution

```python
# ***Solution***

def optimal_weights(X, y):
     """ Returns the optimal weights in the least squares sense for the specified
         training inputs (X) and training outputs (y) """
     return np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)

optimal_weights(X, y)
```

    array([1.04718477, 1.97927105])

***Solution***

The output makes sense since it is close to the values used to generate the data, but not identical.  We don't expect it to be identical since we added noise to the training outputs.

## Sanity Checking your Implementation

On the first day we talked a lot about evaluating ML models.  For instance, we talked about running experiments to see how well they work for some problem.  When we are implementing the algorithm ourselves a different and more basic thing we'd like to evaluate is whether we've implemented the algorithm correctly.

While you are probably feeling pretty confident right now that your implementation of linear regression is correct, for more complicated algorithms there have been cases when implementations of algorithms (even in published works) turned out to be incorrect (i.e., they didn't accurately reflect the algorithm that had been derived in the paper).  The story round the campfire (by which I mean I heard this from one of my professors in grad school, but I can't seem to find a link online verifying it) is that the initial implementation of the backpropagation algorithm (a foundational algorithm for machine learning in neural networks that we'll be learning about in the coming weeks) was wrong.  The experimental results presented in the paper were based on a flawed implementation (although clearly it wasn't so flawed that the results were garbage).

### *Notebook Exercise 3 (30 minutes)*

Let's check out a few strategies that we can use to verify that an implementation of an algorithm is correct.

a. ***Strategy 1 check for local optimality.***  If the machine learning algorithm involves optimizing some function (for example in linear regression you are optimizing squared error), you can verify that the solution you compute is locally optimal.  What does it mean for the solution to be locally optimal?  One very basic thing we can check is to see whether the value of the error gets strictly higher as we perturb the solution (e.g., add a small delta to the weights computed by your implementation of linear regression).  The following not very elegant, but illustrative code provides an implementation of this optimality check.

As a quick diagnostic of your understanding, what should be true of the output below in order for an implementation to pass the optimality check?  Why is it important to test each of the four perturbations below?

```python
w_star = optimal_weights(X, y)
w_star_err = sum_of_squared_errors(X, y, w_star)
perturbation = 10**-5
print(sum_of_squared_errors(X, y, w_star + np.array([perturbation, 0])) - w_star_err)
print(sum_of_squared_errors(X, y, w_star - np.array([perturbation, 0])) - w_star_err)
print(sum_of_squared_errors(X, y, w_star + np.array([0, perturbation])) - w_star_err)
print(sum_of_squared_errors(X, y, w_star - np.array([0, perturbation])) - w_star_err)
```

    5.081698617459551e-09
    5.081698617459551e-09
    3.6463116970253395e-09
    3.6463116970253395e-09

It is important to perturb it in all four directions, as essentially that means we are perturbing the weights in both directions of all dimensions in the weight array (in this case, there are only two dimensions).

b. ***Strategy 2: check the gradient.***  For many machine learning algorithms that involve optimizing some function (linear regression is a great example) a second sanity check is to verify that the gradient is 0 at a potential solution.  Since it is not necessarily straightforward to calculate the gradient of the function we are optimizing, we can instead check that a numerical approximation of the gradient is close to 0.  We will use the finite differences method to approximate the gradient.

To help you understand what we mean by finite differences, here is the definition of the derivative of a single variable function.

$$f'(x) = \lim_{h \rightarrow 0} \frac{f(x+h) - f(x)}{h}$$

This definition suggests that we can approximate the derivative using the finite difference method as $f'(x) \approx \frac{f(x+h) - f(x)}{h}$ for some small value of $h$.  What typically works even better is to use the method of central differences where we estimate the drivative as $f'(x) \approx \frac{f(x+h) - f(x - h)}{2h}$.

In the code below, we'll apply this idea to estimating the gradient (which consists of two partial derivatives) at the optimal solution returned by your implementation of linear regression.

As a quick check of your understanding, what should be true of the output below in order for an implementation to pass the gradient check?

```python
estimate_partial_w_1 = (sum_of_squared_errors(X, y, w_star + np.array([perturbation, 0])) - sum_of_squared_errors(X, y, w_star - np.array([perturbation, 0])))/(2*perturbation)
estimate_partial_w_2 = (sum_of_squared_errors(X, y, w_star + np.array([0, perturbation])) - sum_of_squared_errors(X, y, w_star - np.array([0, perturbation])))/(2*perturbation)
print(estimate_partial_w_1, estimate_partial_w_2)
```

    0.0 0.0

The output should be very close to 0, if not actually 0. As it happens above, it is essentially 0, so we've passed this check.

c. ***Strategy 3: compare to a known working implementation.***  Perhaps the most direct approach to validating your implementation would be to compare it to a known working implementation (assuming you have access to one).  In the cell below, we call `numpy`'s implementation of linear regression and compare it with your solution.

As a quick check of your understanding, what should be true of the output below in order for an implementation to pass the *compare to a known working implementation* check?

```python
w_known_working, _, _, _ = np.linalg.lstsq(X, y, rcond=-1)
print(w_known_working - w_star)
```

    [-2.22044605e-16  4.44089210e-16]

For the same reasons in strategy two, it should be zero. And like in strategy two, it passed.

#### Expand for Solution

***Solution***

In order to pass the test all of the differences printed out should be positive since this corresponds to the error increasing as we move away from the optimal solution we computed.  It's important to test all four direction since you want to look along each dimension and in both the positive and negative directions.

The gradient should be close to 0.

The difference between the two vectors should be small.

## Training Test Splits: Bikeshare Revisited

In this next section of the notebook we're going to revisit the dataset that we met in the first assignment.  Our goals in this activity are twofold.

1.  We will introduce the notion of a train / test split for validating machine learning algorithms.
2.  We will motivate, derive, and implement an extension to linear regression called [ridge regression](https://en.wikipedia.org/wiki/Tikhonov_regularization).

For your convenience, here is the text from the previous notebook that we used to introduce the dataset.
> The [Bikeshare](https://archive.ics.uci.edu/ml/datasets/bike%20sharing%20dataset) dataset contains daily usage data over a roughly two year period. Along with each record of user counts, there are independent variables that measure various characteristics of the day in question (e.g., whether it was a weekday or a weekend, the air temperature, the wind speed).

The code below loads the dataset and produces hexplots that show various characteristics of the day versus ridership.

```python
import pandas as pd
bikeshare = pd.read_csv('https://raw.githubusercontent.com/kylecho/nd101_p1_neural_network/master/Bike-Sharing-Dataset/day.csv')

X_bikeshare = bikeshare.drop(columns=['instant', 'dteday', 'cnt', 'registered', 'casual'])
y_bikeshare = bikeshare['cnt']
X_bikeshare['bias'] = 1

plt.figure(figsize=(30, 18))
for idx, col in enumerate(X_bikeshare):
    plt.subplot(3, 4, idx+1)
    plt.hexbin(X_bikeshare[col], y_bikeshare, gridsize=25, cmap='jet')
    plt.colorbar()
    plt.xlabel(col)
    plt.ylabel('rider count')

plt.subplots_adjust(wspace=.2)
plt.show()
```

### Training and Testing Sets

One of the most fundamental ideas in evaluating machine learning algorithm involves [partitioning data into a training set (used for fitting a model) and a testing set (used for estimating the performance of the model).  There is a pretty comprehensive article on [training, validation, and testing sets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets) on Wikipedia, but for now we are not going to be talking about the validation set.  Feel free to follow along with our presentation here and keep the link handy for future reference (no need to read the linked article now).

Remember the basic supervised learning problem setup where we are given a training data consisting of inputs $\mathbf{x_1}, \mathbf{x_2}, \ldots, \mathbf{x_n}$ and outputs $y_1, y_2, \ldots, y_n$.  So far we have been applying our learning algorithms to *all* $n$ of the training data instances.  We might be then tempted to estimate how well the resultant model would work on new data bytes computing the average squared error on these $n$ training instances.  It turns out that this approach can wildly overestimate how well the model will work on new data.  The reason is that the model parameters (e.g., the weights in linear regression) have been tuned to the training data.  Some of these model parameters will reflect genuine relationships between the inputs and outputs, and other model parameters may largely reflect particular quirks of the training data (e.g., noise) that are not applicable to new data.

In order to get an unbiased estimate of the performance of the model on new data we reserve a portion of the training data as a ***testing set***.  This testing set is not used to fit the model parameters and is only used to estimate model performance *after the model has been created*.

To clarify what we mean, in the next cell is some code that partitions the Bikeshare data into training and test sets.  We will fit the parameters of a linear regression model (using your code!) on the training set and calculate mean squared error on the testing set.  To make the code less cluttered, we will be using a helper function from the [`scikit-learn`](https://scikit-learn.org/stable/) library that creates a [training set / testing set split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html#sklearn.model_selection.train_test_split).  This function will ***randomly*** partition the given data into two disjoint sets: the training set and the testing set.  The parameter `test_size` controls the fraction of data assigned to the testing set versus the training set.  You'll also notice that we divide the `sum_of_squared_errors` by `y_train.shape[0]`, which is the number of training data instances.  Divided the sum of squared error by the number of training data instances gives us the *mean squared error.*  The mean squared error is more interpretable than the sum of squared errors since it controls for the number of data instances.

#### Notebook Exercise 4 (30 minutes)

Run the code below several times in order to answer the following questions.

What causes the results to change from run to run?

As you run the code multiple times, does there seem to be a trend that the performance on the training set is better (i.e. has lower mean squared error) than the performance on the testing set?

Since the training set was used to fit the model parameters, we might expect the training set to always have better performance than the testing set.  It appears that this is not always the case.  How is this possible?

```python
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X_train, X_test, y_train, y_test = train_test_split(X_bikeshare, y_bikeshare, test_size=0.5)
w = optimal_weights(X_train, y_train)

print("training set mean squared error=%f" % (sum_of_squared_errors(X_train, y_train, w)/y_train.shape[0]))
print("testing set mean squared error=%f" % (sum_of_squared_errors(X_test, y_test, w)/y_test.shape[0]))
```

    training set mean squared error=743086.767617
    testing set mean squared error=809804.756786

1. As the division between training set and test set is decided randomly, the actual training set and test set used in each cell run differs, and hence the results will also differ.
2. The training set usually has a lower mean squared error than the test set, which is understandable, as the training set was used to train the model parameters - i.e. the parameters were optimized for the training set, and not the testing set.
3. This could be because some random test sets could actually be more linearly fit than the training set.

##### Expand for Solution

***Solution***

The changes in output are driven by randomness in the `train_test_split` function.  Depending on which instances are assigned to the training versus the testing set, the output will differ.

It does appear that, on average, the performance on the testing set is worse.

On any given run it does not follow that performance on the testing set will always be worse.  It could be the case that the testing set happened to contain a lot of easy to predict instances and the training set contained a high number of outliers.

### Ridge Regression

So far we've been working with this Bikeshare dataset in cases where we have a relatively high number of training instances compared with the dimensionality of the data.  To make this more precise, the shape of `X_train` is 365 by 12, which means we have ratio of roughly 30:1 training instances to inputs features.  While there are no hard and fast rules about this, a 30:1 ratio is considered pretty good for coming up with good esitmates of model parameters.

Suppose instead that we faced a situation where we had very little training data.  To simulate this case, below we rerun our experiment with the BikeShare dataset but set the `test_size` to 0.95.  You should notice two things when running this code.

1.  The performance on the training set is markedly better than the testing set.
2.  Occasionally you will get an error message about a singular matrix.

```python
X_train, X_test, y_train, y_test = train_test_split(X_bikeshare, y_bikeshare, test_size=0.95)
print("number of training points %d, number of testing points %d" % (y_train.shape[0], y_test.shape[0]))

w = optimal_weights(X_train, y_train)
print("training set mean squared error=%f" % (sum_of_squared_errors(X_train, y_train, w)/y_train.shape[0]))
print("testing set mean squared error=%f" % (sum_of_squared_errors(X_test, y_test, w)/y_test.shape[0]))
```

    number of training points 36, number of testing points 695
    training set mean squared error=935543.071127
    testing set mean squared error=2164611.460142

The first observation (that the performance on the training set is markedly better than the testing set) is perhaps not very surprising since we now have much less training data to use to reliably estimate the model parameters.  To understand the second observation, we need remind ourselves of the formula for the optimal weights in linear regression.

$$\mathbf{w^\star} = \left ( \mathbf{X}^\top \mathbf{X} \right)^{-1} \mathbf{X}^\top \mathbf{y}$$

The error regarding a singular matrix is coming from the fact that we are computing the inverse of the matrix $\mathbf{X}^\top \mathbf{X}$.  One property of a [singular matrix](http://mathworld.wolfram.com/SingularMatrix.html) is that it is not invertible, hence the error message.  The reason it is not invertible is that the matrix $\mathbf{X}^\top \mathbf{X}$ is not full rank.  This happens when the training data does not properly span the space of the features.  This usually happens for a combination of the following reasons:

1.  There is too little training data
2.  There are features that are defined as linear combinations of each other.

In order to solve this problem, a common approach is to modify the linear regression problem to prefer solutions that have small weights.  We do this by penalizing the sum of the squares of the weights themselves.  This is called ridge regression (or Tikhonov regularization).  Below, we show the original version of ordinary least squares along with ridge regression.

Ordinary least squares:
$$\begin{align}
\mathbf{w^\star} &= \arg\min_\mathbf{w} \sum_{i=1}^n \left ( \mathbf{w}^\top \mathbf{x_i} - y_i \right)^2 \\
&= \arg\min_\mathbf{w} \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)
\end{align}$$

Ridge regression (note that $\lambda$ is a non-negative parameter that controls how much the algorithm cares about fitting the data and how much it cares about having small weights):
$$\begin{align}
\mathbf{w^\star} &= \arg\min_\mathbf{w} \sum_{i=1}^n \left ( \mathbf{w}^\top \mathbf{x_i} - y_i \right)^2 + \lambda\sum_{i=1}^d w_i^2 \\
&= \arg\min_\mathbf{w} \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left ( \mathbf{X}\mathbf{w} -  \mathbf{y} \right) + \lambda \mathbf{w}^\top \mathbf{w}
\end{align}$$

The penalty term may seem a little arbitrary, but it can be motivated on a conceptual level pretty easily.  The basic idea is that in the absence of sufficient training data to suggest otherwise, we should try to make the weights small.  Small weights have the property that changes to the input result in minor changes to our predictions, which is a good default behavior.

### Notebook Exercise 5 (60 minutes)
*Note: this one is really a math problem, but we didn't want to send you back to the other document and then back here again.  Let us know if you like this or not via NB.*

Derive an expression to compute the optimal weights, $\mathbf{w^\star}$, to the ridge regression problem.

* Hint 1: This is very, very similar to exercise 5 in the assignment document.
* Hint 2: If you follow the same steps as you did in exercise 5, you'll arrive at an expression that looks like this (note: $\mathbf{I}_{d \times d}$ is the $d$ by $d$ identity matrix).

$$\mathbf{w^\star} = \arg\min_\mathbf{w} \mathbf{w}^\top \mathbf{X}^\top \mathbf{X} \mathbf{w} - 2\mathbf{w}^\top \mathbf{X}^\top \mathbf{y} + \mathbf{y}^\top \mathbf{y} + \lambda \mathbf{w}^\top  \mathbf{I}_{d \times d} \mathbf{w}$$

* Hint 3: to get $\mathbf{w^\star}$, take the gradient, set it to 0 and solve for $\mathbf{w}$.

#### Expand for Solution

***Solution***

$$\begin{align}\mathbf{w^\star} &= \arg\min_\mathbf{w} \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left ( \mathbf{X}\mathbf{w} -  \mathbf{y} \right) + \lambda \mathbf{w}^\top \mathbf{w} \\
& = \arg\min_\mathbf{w} \mathbf{w}^\top \mathbf{X}^\top \mathbf{X} \mathbf{w} - 2\mathbf{w}^\top \mathbf{X}^\top \mathbf{y} + \mathbf{y}^\top \mathbf{y} + \lambda \mathbf{w}^\top  \mathbf{I}_{d \times d} \mathbf{w}\\
&= \arg\min_\mathbf{w} \mathbf{w}^\top \left ( \mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I_{d \times d}} \right )\mathbf{w} - 2\mathbf{w}^\top \mathbf{X}^\top \mathbf{y} + \mathbf{y}^\top \mathbf{y} \\
2 \left (  \mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I_{d \times d}} \right ) \mathbf{w^\star} - 2 \mathbf{X}^\top \mathbf{y} &=0 ~~\mbox{take the gradient and set to 0} \\
\mathbf{w}^\star &= \left ( \mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I_{d \times d}} \right)^{-1} \mathbf{X}^\top \mathbf{y}
\end{align}$$

### Notebook Exercise 6 (20 minutes)

Now we'll be revisiting the Bikeshare dataset and see if ridge regression can help.  If you'd like to implement the algorithm yourself, feel free.  Since it is a relatively small change from the implementation that you created earlier, we have gone ahead and provided you with implementation below.  Here are some questions to test your understanding of the effects of applying ridge regression to the bike share dataset.

Run the code below with the default setting of the input `lam`.  You should notice that singular matrix error does not arise anymore.  Make the value of `lam` really large (search over different orders of magnitude to find a value that is really large).  What happens to the training and test set errors?

Does there seem to be a value of `lam` that is best (we advise you to search over different orders of magnitude)?  How do you define best?  What would be a good process for determining a good value of `lam` (we'll be learning about this in much more detail coming up, but we wanted to get you thinking about some possibilities)?

```python
def optimal_weights_ridge(X, y, lam):
     """ Returns the optimal weights in the least squares sense for the specified
         training inputs (X) and training outputs (y) with ridge term `lam` """
     return np.linalg.inv(X.T.dot(X) + lam*np.eye(X.shape[1])).dot(X.T).dot(y)

X_train, X_test, y_train, y_test = train_test_split(X_bikeshare, y_bikeshare, test_size=0.95)
print("number of training points %d, number of testing points %d" % (y_train.shape[0], y_test.shape[0]))

w = optimal_weights_ridge(X_train, y_train, 0.0001)
print("training set mean squared error=%f" % (sum_of_squared_errors(X_train, y_train, w)/y_train.shape[0]))
print("testing set mean squared error=%f" % (sum_of_squared_errors(X_test, y_test, w)/y_test.shape[0]))
```

    number of training points 36, number of testing points 695
    training set mean squared error=945547.759100
    testing set mean squared error=1498693.705417

1. When `lam` is very large, the model is more interested in making the weights small than it is in fitting the data, so the errors become quite large as well.
2. A small value of `lam` seems to be doing the best - I found it to be something like 0.01

#### Expand for Solution

***Solution***

(a) When you make lambda really big, the errors become very large.  This is because the model is underfitting to the data.  In other words, it cares more about making the weights small than it does about fitting the data.

(b) It's pretty hard to tell, but it seems like a value around $0.0001$ seems to work pretty well.  We defined best by mentally averaging over the testing mean squared error measured across a few runs.  In order to do this more rigorously you'd want to have a defined space of values you'd search over, repeat the experiment a number of times, and then choose the best average performance.  We'll dig into this more systematically soon, but this is a good amount answer to get to with the current tools we have discussed.


```python
%matplotlib inline
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
```

# Ejemplo : Conteo de Fotones
Se quiere saber el flujo de una estrella contando los fotones, pero antes se hacen las siguientes suposiciones:

- Flujo constante en el tiempo
- Se ignora el ruido presente en las mediciones.
- Hicimos $N$ medidas con nuestro telescopio, obteniendo observaciones $F_i$ de el flujo de fotones con un error $e_i$
- Asumimos que los errores $e_i$ se distribuyen normalmente.

Dado que tenemos entonces los datos $D = (F_i, e_i)$, queremos estimar el valor de $F_{true}$

##  🔭 Observaciones
Debido a que lo que queremos es contar, una buena distribución para hacer la simulación es Poisson.

ver: [Scipy - Poisson](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.poisson.html)

```python
# Generamos datos de conteo de fotones
np.random.seed(1)  # Reprodubilicidad

F_true = 1000  # Flujo verdadero (n fotones/tiempo)
N = 50 # Cantidad de observaciones
F = stats.poisson.rvs(mu = F_true, size = N)  # Observaciones
e = np.sqrt(F)  # Errores (Poisson)
```

```python
F,e
```

    (array([ 992,  962,  968,  991,  993,  970,  992,  961,  983, 1042,  947,
             966,  951, 1079, 1018, 1017,  979,  953, 1050,  980,  971,  999,
            1007, 1008,  953, 1018,  928, 1015, 1067, 1049,  961,  991, 1058,
            1024, 1044, 1024,  978,  993, 1015,  956,  995,  992, 1049,  984,
            1044, 1050,  966, 1060,  939, 1020]),
     array([31.4960315 , 31.01612484, 31.11269837, 31.48015248, 31.51190251,
            31.144823  , 31.4960315 , 31.        , 31.35283081, 32.28002478,
            30.77336511, 31.08054054, 30.83828789, 32.84813541, 31.90611227,
            31.89043744, 31.28897569, 30.87069808, 32.40370349, 31.30495168,
            31.1608729 , 31.60696126, 31.73326331, 31.74901573, 30.87069808,
            31.90611227, 30.46309242, 31.85906464, 32.66496594, 32.38826948,
            31.        , 31.48015248, 32.52691193, 32.        , 32.31098884,
            32.        , 31.27299154, 31.51190251, 31.85906464, 30.91924967,
            31.54362059, 31.4960315 , 32.38826948, 31.36877428, 32.31098884,
            32.40370349, 31.08054054, 32.55764119, 30.64310689, 31.93743885]))

Visualizamos las observaciones

```python
fig, ax = plt.subplots(figsize = (6,6))
ax.errorbar(F, np.arange(N), xerr=e, fmt='ok', ecolor='gray', alpha=0.5)
ax.vlines([F_true], 0, N, linewidth=5, alpha=0.2)
ax.set_xlabel("Flujo");ax.set_ylabel("Número de Observación");
```

**Propondremos un modelo Gaussiano para $F_{\rm true}$**

**Dadas las observaciones,  Cual es el mejor valor estimado para el flujo $F_{true}$**

## Enfoque Frecuentista
Un enfoque clásico frecuentista es obtener el estimador de máxima verosimilitud, o sea, dada una observación $D_i$, podemos calcular la probabilidad de hacer esa observación dado que existe un valor de flujo constante $F_{true}$, dado el supuesto de errores que se distribuyen iid normal tenemos.

\begin{equation}
  P(D_i~|~F_{\rm true}) = \frac{1}{\sqrt{2\pi e_i^2}} \exp{\left[\frac{-(F_i - F_{\rm true})^2}{2 e_i^2}\right]}
\end{equation}

Que es una distribución normal de media $F_{true}$ y desviación estándar $e_i$. Luego se quuere calcular la probabilidad de los sucesos independientes:

\begin{equation}
\mathcal{L}(D~|~F_{\rm true}) = \prod_{i=1}^N P(D_i~|~F_{\rm true})
\end{equation}

Y para facilidad de la optimización, se utiliza las propiedades del logaritmo para tener una formulación equivalente: Log-Likelihood

\begin{equation}
\log\mathcal{L} = -\frac{1}{2} \sum_{i=1}^N \left[ \log(2\pi  e_i^2) + \frac{(F_i - F_{\rm true})^2}{e_i^2} \right]
\end{equation}

Al maximizar la expesión de arriba se tiene
\begin{equation}
F_{\rm est} = \frac{\sum w_i F_i}{\sum w_i};~~w_i = 1/e_i^2
\end{equation}

Y si todos los errores fuesen iguales

\begin{equation}
F_{\rm est} = \frac{1}{N}\sum_{i=1}^N F_i
\end{equation}

Lo que es simplemente la media de las observaciones, esto coincide con el enfoque frecuentista, para los cuales las probabilidades están fundamentalmente relacionadas con la frecuencia de los eventos.

```python
w = 1. / e ** 2
print("""
      F_true = {0}
      F_est  = {1:.0f} +/- {2:.0f} (basado en {3} observaciones)
      """.format(F_true, (w * F).sum() / w.sum(), w.sum() ** -0.5, N))
```

          F_true = 1000
          F_est  = 998 +/- 4 (basado en 50 observaciones)

```python

```

## Enfoque Bayesiano
Se busca computar $P(F_{true}|D)$, o sea, una vez se observan los datos, lo que ajustamos son los modelos posibles. Es importante además destacar que en este enfoque se piensa en $F$ como una variable aleatoria con distribución $P(F)$ -> Prior.

Para esto, se utiliza la ley de Bayes resultando:
\begin{equation}
P(F_{\rm true}~|~D) = \frac{P(D~|~F_{\rm true})~P(F_{\rm true})}{P(D)}
\end{equation}

De acá podemos observar
- $P(F_{\rm true}| D)$ Distribución **posterior**, o distribución de los parámetros del modelo dado los datos.
- $P(D| F_{\rm true})$: **Likelihood**, distribución proporcional a $\mathcal{L}(D~|~F_{\rm true})$
- $P(F{\rm true})$: **Prior** , Es nuestro conocimiento sobre el comportamiendo del modelo sobre los datos
- $P(D)$: **Data Probability**, Constante de renormalización.

Se puede observar que en algunos casos simples en que esojamos una distribución a priori plana: $P(F_{\rm true}) \propto 1$ se tiene que el enfoque bayesiano es equivalente al frequentista (se maximizan en el mismo punto).
\begin{equation}
P(F_{\rm true}|D) \propto \mathcal{L}(D|F_{\rm true})
\end{equation}

```python
# theta = parámetro a estimar (es mejor pensarlo como un vector para generalizar)
def log_prior(theta):
    return 1  # prior plana

def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * e ** 2)
                         + (F - theta[0]) ** 2 / e ** 2)

def log_posterior(theta, F, e): # Omitimos la constante de normalización
    return log_prior(theta) + log_likelihood(theta, F, e)
```

Al aumentar el número de parámetros del modelo, la distribución a posterior se vuelve muy dificil de computar, para sobrellevar esto se hace uso de los métodos de sampleo como Marcov Chain Monte Carlo (Vimos esto la ayudantía pasada).

> 🕰 Breve Recuerdo : Los métodos de sampleo MCMC buscan estimar una distribución de probabilidad sampleando puntos con este comportamiento utilizando las cadenas de marcov. [MCMC](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo)

Si bien nuestro caso es simple, se aplicará de todas formas una MCMC para ejemplificar un caso con mayores parámetros.

**Desafío** Computar la distribución posterior directamente.

> 👀 Se utilizará la libreria [EMCEE](https://emcee.readthedocs.io/en/stable/) para la aplicación de las MCMC.

```python
!pip install emcee
```

    Collecting emcee
      Downloading https://files.pythonhosted.org/packages/97/f4/00151f5f843088337c6a53edd6cbb2df340f1044d23080c662f95219cc3f/emcee-3.0.2-py2.py3-none-any.whl (41kB)
         |████████████████████████████████| 51kB 2.1MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from emcee) (1.19.5)
    Installing collected packages: emcee
    Successfully installed emcee-3.0.2

```python
ndim = 1  # parametros del modelo (en nuestro caso sólo es F true)
nwalkers = 50  # número de walkers del MCMC
nburn = 1000  # Burn-in (Los primeros sampleos son quemados ya que no siguen adecuadamente la distribución de probabilidad)
nsteps = 2000  # número de pasos de la MCMC

# Punto inicial
starting_guesses = 2000 * np.random.rand(nwalkers, ndim)

import emcee
# construcción de los samples
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F, e])
sampler.run_mcmc(starting_guesses, nsteps)

sample = sampler.chain  # shape = (nwalkers, nsteps, ndim)
sample = sampler.chain[:, nburn:, :].ravel()  # se queman los puntos burn-in

```

```python
# 50000 puntos sampleados desde la distribución posterior
len(sample)
```

    50000

```python
# Gráficamos el histograma
plt.hist(sample, bins=50, histtype="stepfilled", alpha=0.3, density = True)

# graficamos una gaussiana
F_fit = np.linspace(975, 1025)
pdf = stats.norm(np.mean(sample), np.std(sample)).pdf(F_fit)

plt.plot(F_fit, pdf, '-k')
plt.xlabel("F"); plt.ylabel("P(F)")

```

```python
print("""
      F_true = {0}
      F_est  = {1:.0f} +/- {2:.0f} (based on {3} measurements)
      """.format(F_true, np.mean(sample), np.std(sample), N))
```

          F_true = 1000
          F_est  = 998 +/- 4 (based on 50 measurements)

Como nota al margen, es posible que también haya notado un pequeño juego de manos: ¡al final, usamos un enfoque frecuentista para caracterizar nuestras muestras posteriores! Cuando calculamos la media muestral y la desviación estándar anteriores, estábamos empleando una técnica claramente frecuentista para caracterizar la distribución posterior. El resultado bayesiano puro para un problema como este sería informar la distribución posterior en sí (es decir, su muestra representativa) y dejarlo así. Es decir, en el bayesianismo puro la respuesta a una pregunta no es un solo número con barras de error; la respuesta es la distribución posterior sobre los parámetros del modelo.

## Resumen :
- El frecuentismo considera que las probabilidades están relacionadas con las frecuencias de eventos reales o hipotéticos.
- El bayesianismo considera probabilidades para medir grados de conocimiento.
- Los análisis frecuentistas generalmente proceden mediante el uso de estimaciones puntuales y enfoques de máxima verosimilitud.
- Los análisis bayesianos generalmente calculan el posterior ya sea directamente o mediante alguna versión del muestreo MCMC.

```python

```

#Cuando los enfoques difieren: Juego de Billar

Uno de los factores que hace que difieran los enfoques, es la presencia de parámetros "molestos".

Los parámetros molestos son aquellos que no nos interesa estudiar, pero aún así son vitales para poder estimar algun parámetro de interés.

## Juego del Billar
Alice y Bob entran en una habitación. Detrás de una cortina hay una mesa de billar, que no pueden ver, pero su amiga Carol sí. Carol hace rodar una pelota por la mesa y marca dónde cae. Una vez que esta marca está en su lugar, Carol comienza a hacer rodar nuevas bolas por la mesa. Si la bola cae a la izquierda de la marca, Alice obtiene un punto; si aterriza a la derecha de la marca, Bob obtiene un punto. Podemos suponer, por ejemplo, que los lanzamientos de Carol son imparciales: es decir, las bolas tienen la misma probabilidad de terminar en cualquier lugar de la mesa. La primera persona en llegar a seis puntos gana el juego.

Aquí la ubicación de la marca (determinada por la primera tirada) puede considerarse un **parámetro molesto** ($\hat{p}$): se desconoce y no es de interés inmediato, pero claramente debe tenerse en cuenta al predecir el resultado de las tiradas posteriores. Si la primera tirada se coloca muy a la derecha, las tiradas posteriores favorecerán a Alice. Si se asienta muy a la izquierda, Bob será el favorito en su lugar.

Pregunta:
> En un juego en particular, después de ocho tiradas, Alice tiene cinco puntos y Bob tiene tres puntos. ¿Cuál es la probabilidad de que Bob gane el juego?

```python

```

## Un enfoque Frecuentista

Se puede estimar el parametro molesto $\hat{p}$, según MLE de la distribución binomial para estimar $p$ como :
\begin{equation}
\hat{p} = 5/8
\end{equation}

[MLE Binomial Distribution](https://en.wikipedia.org/wiki/Binomial_distribution#Estimation_of_parameters).

Luego, se puede computar la probabilidad de que bob gane el juego como:
\begin{equation}
P(B) = (1 - \hat{p})^3
\end{equation}

```python
p_hat = 5. / 8.
freq_prob = (1 - p_hat) ** 3
print("Naïve Frequentist Probability of Bob Winning: {0:.2f}".format(freq_prob))
```

    Naïve Frequentist Probability of Bob Winning: 0.05

## Enfoque Bayesiano
- B : Gana Bob
- D : Data observada $D = (5,3)$
- $p$ : probabilidad de que la bola caiga en el lado de alicia durante el juego.

En resumen, el enfoque bayesiano lidia con los parámetros molestos marginalizandolos, o sea, sumando su probabilidad conjunta sobre todo su dominio:

\begin{equation}
P(B~|~D) \equiv \int_{-\infty}^\infty P(B,p~|~D) {\mathrm d}p
\end{equation}

Aplicando la ley de bayes y la definición de probabilidad condicional queda :
\begin{equation}
P(B~|~D) = \frac{\int P(B~|~p,D) P(D~|~p) P(p) dp}{\int P(D~|~p)P(p) dp}
\end{equation}

En donde
- $P(B~|~p,D) = (1 - p) ^ 3$  -> Likelihood
- $P(D~|~p) \propto p^5 (1-p)^3$ -> Binomial
- $P(p) \propto 1$ -> Prior Plana

Resultando

\begin{equation}
P(B~|~D) = \frac{\int_0^1 (1 - p)^6 p^5 dp}{\int_0^1 (1 - p)^3 p^5 dp}
\end{equation}
Lo cual es un caso especial de la Función Beta:
\begin{equation}
\beta(n, m) = \int_0^1 (1 - p)^{n - 1} p^{m - 1}
\end{equation}

```python
from scipy.special import beta
bayes_prob = beta(6 + 1, 5 + 1) / beta(3 + 1, 5 + 1)

print("P(B|D) = {0:.2f}".format(bayes_prob))
```

    P(B|D) = 0.09

## Que enfoque es el correcto? : Simulación de Monte Carlo
Para poder descubrir esto, simplemente nos queda simular. Debido a que muchas de las distribuciones involucradas son uniformes, es sencillo poder simular la situación dada.

En resumen, lo que se hace es simular muchos juegos que cumplan las condiciones dadas $D = (5,3)$, y ver en cuantos de ellos Bob gana.

```python
import numpy as np
np.random.seed(0)

# play 100000 games with randomly-drawn p, between 0 and 1
p = np.random.random(100000)

# each game needs at most 11 rolls for one player to reach 6 wins
rolls = np.random.random((11, len(p)))

# count the cumulative wins for Alice and Bob at each roll
Alice_count = np.cumsum(rolls < p, 0)
Bob_count = np.cumsum(rolls >= p, 0)

# sanity check: total number of wins should equal number of rolls
total_wins = Alice_count + Bob_count
assert np.all(total_wins.T == np.arange(1, 12))
print("(Sanity check passed)")

# determine number of games which meet our criterion of (A wins, B wins)=(5, 3)
# this means Bob's win count at eight rolls must equal 3
good_games = Bob_count[7] == 3
print("Number of suitable games: {0}".format(good_games.sum()))

# truncate our results to consider only these games
Alice_count = Alice_count[:, good_games]
Bob_count = Bob_count[:, good_games]

# determine which of these games Bob won.
# to win, he must reach six wins after 11 rolls.
bob_won = np.sum(Bob_count[10] == 6)
print("Number of these games Bob won: {0}".format(bob_won.sum()))

# compute the probability
mc_prob = bob_won.sum() * 1. / good_games.sum()
print("Monte Carlo Probability of Bob winning: {0:.2f}".format(mc_prob))
```

    (Sanity check passed)
    Number of suitable games: 11068
    Number of these games Bob won: 979
    Monte Carlo Probability of Bob winning: 0.09

Podemos ver que el enfoque Bayesiano logra coincidir con este resultado, pero esto no quiere decir que el enfoque frecuentista sea incorrecto, de hecho se fue un poco injusto, ya que existen métodos frecuentistas que pueden dar un correcto resultado ya que manejan bien los parámetros molestosos (en este caso utilizamos un método nativo frecuentista)

# Referencia:
- http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/

```python

```


# 3D Reconstruction and Structure from Motion

Suppose we are given a series of images from a disaster situation. How can we begin to make sense of the contents of the image? Typically this involves answering two questions:

- _What_ is in an image (e.g. debris, buildings, etc.)?
- _Where_ are these things located _in 3D space_ ?

We are first going to set the first question aside and focus entirely on the second. This will be the first in two lectures on so called "structure from motion", which is the idea that you can recreate a 3D scene from a series of 2D images by taking those images from a moving camera. Today's lecture will focus on the theory of 3D reconstruction and how to implement it. Tomorrow's lecture will focus on a real world example of structure from motion, and we will work to assign GPS coordinates to pixels on an image.

We will first start with two images and work our way up to multiple images.

## The Camera Model:

Let's first introduce the model that we will be basing our analysis on. For this session, we will be using the _pinhole model_ of a camera, depicted in the following picture:

In this model of the camera, light rays are reflected off of an object (in this case a kettle), they pass through an infinitesimally small pinhole, and arrive at the opposite wall of the camera, creating a photo-negative (think upside down) image of the object. The pinhole is located at a point called the _center of projection_. Because the image on the actual camera is a photo-negative, we typically instead center our reference frame on the center of projection and work as if the image were on the opposite side, which we will call the _image plane_. The distance _f_ between the center of projection and the image plane is called the _focal length_ which is usually measured in millimeters. On the image plane, the center of the plane is called the _principal point_ and the line that connects the center of projection and the principal point is called the _principal axis_.

We can align our coordinate frame as follows: let's center our coordinate system such that the origin is the center of projection, let's align the Z-axis such that it passes through principal axis, and let's align the X- and Y-axis such that the X-axis goes along the horizonal direction of the image plane and the Y-axis goes along the vertical direction of the image plane. Note that X increases to the right, but Y increases *downward*. This is shown in the following image:

The image plane is given by a _uv-plane_ that is centered on the principal point and is orthogonal to the principal axis. Note that cameras typically measure coordinates starting from the upper-left corner, and increasing to the left and downwards (which motivates our decision to align our Y axis downwards). Furthermore, note that cameras cannot capture an image with perfect precision. Rather, they are composed of small elements called _pixels_ that vary in intensity in order to capture the scene. Therefore, we would rather operate on the discrete pixel plane, which we will call the lowercase xy-plane. You can compare both planes in the image below:

All objects create a projection on the xy-plane, which is what we actually see on the image. For example, the point $M$ on the kettle is projected onto the xy-plane on the point _m_. Suppose we have the coordinates ($X$, $Y$, $Z$) of the point $M$. **What are the coordinates on the xy-plane of the point $m$?** State your answer in terms of $\alpha$ (the focal length measured in pixels, not millimeters), $p_x$ and $p_y$ (the coordinates of the principal point $p$ in the xy-plane), and $X$, $Y$ and $Z$, the coordinates of the point $M$.

    ANSWER
    We can use geometry to arrive at the answer. Note that $C$, $p$ and $m$ form a similar triangle with $C$, $Z$ and $M$. Therefore, $\frac{x-p_x}{\alpha} = \frac{X}{Z}$ and $\frac{y-p_y}{\alpha} = \frac{Y}{Z}$. Now it's a matter of solving: $x = \alpha\frac{X}{Z} + p_x$ and $y = \alpha\frac{Y}{Z} + p_y$

### List of variables used in this lesson
| Variable |                                               Meaning                                              |
|:--------:|:--------------------------------------------------------------------------------------------------:|
|     M    |                                         A point in 3D space                                        |
|  m1, m2  |                                  Projection of M on image 1 and 2                                  |
|  C1, C2  |                             Center of camera 1, 2 (a point in 3D space)                            |
|     p    |         A point on the z axis of C (directly in front of it) at the center of a given photo        |
|     a    |                                      Focal length of a camera                                      |
|     R    |                                     Rotation of a camera image                                     |
|     t    |                                    Translation of a camera image                                   |
|     K    | A intrinsic matrix used to calculate a camera's offset, a function of its focal length and point p |

## The Problem: Depth Ambiguity

Scene reconstruction requires us to go the other way around: we know the coordinates ($x$, $y$) of $m$ and we want the ($X$, $Y$, $Z$) coordinates of the point $M$. This presents a problem: we have two equations and three unknowns. Specifically, there is an ambiguity in the depth $Z$ that is impossible to resolve with just one image. Therefore, the point $M$ can be anywhere along the ray connecting the center of projection and the point $m$. This is shown in the image below:

However, this can be remediated by adding a second image into the equation. This is because, assuming we have perfect cameras*, the rays connecting both centers of projection to the point $M$ should only intersect at one point, as is shown in the image below:

Let's assume that $C_1$ and $C_2$ are both from the same camera but different locations. Furthermore, let's center our coordinate system such that $C_1$ is at the origin. What do we need in order to find the coordinates of point $M$ ?
- The corresponding points $m_1$ and $m_2$
- The focal length $\alpha$ and the coordinates $p_x$ and $p_y$. Because these are internal to the camera, we call these the _intrinsic parameters_.
- The relative rotation $R$ and translation $t$ of $C_2$ with respect to $C_1$. Here, $R$ is a 3x3 matrix and $t$ is a 3x1 vector. We call $R$ and $t$ the _extrinsic parameters_.

\* The perfect cameras assumption is actually critical. To see why, imagine if by some measurement error (imperfect lens, pixel accuracy, etc), $m_2$ is slightly off. **What would happen in the previous example?**

    ANSWER
    The rays corresponding to points $m_1$ and $m_2$ will not intersect, so there is no solution to this problem!

In practice, no camera is actually perfect. There are tools that work under the hood to mitigate this. We will briefly discuss them, but will not go in depth.

Let's get started!

```python
import cv2
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd
import os
import sys
```

```python
%matplotlib inline
# Load and display the two images
img1 = cv2.imread("kitchen_example/images/img1.jpg", cv2.IMREAD_GRAYSCALE)
plt.imshow(img1, cmap='gray')
plt.show()

img2 = cv2.imread("kitchen_example/images/img2.jpg", cv2.IMREAD_GRAYSCALE)
plt.imshow(img2, cmap='gray')
plt.show()
```

## Corresponding points

Our first task is to find the points $m_1$ and $m_2$. Note that it's not just a matter of finding arbitrary points, but also of ensuring the $m_1$ is actually pointing to the same object $M$ as $m_2$. Of course, one way to do this is to manually go through both images and match points on one image with points on the other. This is not a scalable solution as you increase the number of images. To this end, experts in computer vision work with the concept of _features_ which represent points of interest within an image. To understand how features work, consider the image below. **Which of the options, A through F are most easily identifiable in the image?**

    ANSWER
    A and B are the hardest because they could be anywhere on the surface, so surfaces are not good. C and D are a bit easier, because edges are more immediately recognizable. However, the points can still be anywhere along the edge. E and F are the easiest, since there is only one corner that looks like them. What this shows is that corners are good points to track, followed by edges and finally surfaces.

Features, therefore, are typically corners that are automatically extracted based on color differences around the corner. Features are composed of both the _coordinates_ of the point of interest as well as a _descriptor_ of what those points look like.

There are all sorts of features that will do the job. Here, we will show ORB features, which were created by the makers of OpenCV. We first extract the features and then match them with a feature in the opposite image that has the closest descriptor. It's worth noting that most of these matches will not be very good, so we only want to keep the best matches we have.

### Feature extraction

```python
MAX_FEATURES = 5000

# creating the ORB feature extractor
orb = cv2.ORB_create(MAX_FEATURES)

# creating feature points for first and second images
kp1, dc1 = orb.detectAndCompute(img1, None)
kp2, dc2 = orb.detectAndCompute(img2, None)

img1_ORB = cv2.drawKeypoints(img1, kp1, None, flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
plt.imshow(img1_ORB)
plt.show()
```

### Feature matching

```python
# most matches are not very good, so we want to keep only the best ones.
# here, we keep only the top 20%
GOOD_MATCH_PERCENT = 0.2

# create BFMatcher object
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck = True)

# match descriptors
matches = bf.match(dc1, dc2)

# Sort matches by score
matches.sort(key=lambda x: x.distance, reverse=False)

# Remove worst matches
numGoodMatches = int(len(matches) * GOOD_MATCH_PERCENT)
matches = matches[:numGoodMatches]

img_match = cv2.drawMatches(img1,kp1,img2,kp2,matches,None, flags=2)
plt.imshow(img_match)
plt.show()

pts1 = np.float32([ kp1[m.queryIdx].pt for m in matches ]).reshape(-1,1,2)
pts2 = np.float32([ kp2[m.trainIdx].pt for m in matches ]).reshape(-1,1,2)
```

## Intrinsic parameters

We now proceed to obtain the intrinsic parameters. We have already seen that we can gather important data about the image by looking at its metadata. We can do something similar here:

It might seem tempting to take the focal length as is, and to take the height and width of the image, divide by two and call that the coordinates of the principal point. While this might certainly be a start if you have no other options, there are some limitations to take into account. First, recall that the focal length here is measured in millimeters, but we need the length in pixels $\alpha = m*f$, where $m$ is the size in millimeters of each pixel. This information is rarely found in the metadata. Usually, if you want to get the focal length in pixels just from the metadata, you would have to find the camera specifications online and make the conversion of millimeters to pixels. Also, the focal length is typically rounded to the nearest integer in the metadata, further complicating matters.

Second, because of manufacturing constraints in positioning the sensor the principal point is always off from the middle of an image by a few pixels. While the middle of an image is most likely a good approximation, the error will continue to add up as you go farther away from the camera center.

Finally, even if you have the focal length and principal point coordinates, the reconstruction must account for _distortion_. Distortion typically comes from imperfect lenses. The most well known form of distortion is _radial distortion_. This has the effect of making straight lines bulge outwards the farther away you are from the center. The figure below shows an example of this:

Distortion is rarely in the image metadata. Sometimes cameras have distortion control that corrects for distortion when it's taking the image, so you don't have to worry about it. Still, this is relatively rare and it does not show up in the metadata whether distortion control was applied. So if you're not familiar with the camera you're using, you're still in the dark.

One way to get around this is to perform explicit camera calibration. A common way to do this is to map points with a known 3D-to-2D correspondence (like a chessboard) and perform an optimization to correct for the distortion. How this typically works is by assigning coordinates to the inner corners of the camera (e.g. the top left corner will be (0, 0, 0), the one on the right might be (1, 0, 0), the one below would be (0, 1, 0)), and then choosing the focal length, principal point and distortion parameters that best explain deviations in the expected 2D coordinates. Typically you have to take something on the order of 20-30 images of a chessboard from slightly varying locations for this procedure to work.

```python
import glob
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((6*9,3), np.float32)
objp[:,:2] = np.mgrid[0:6,0:9].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.

images = glob.glob('chessboard/*.jpg')

for fname in images:
    img = cv2.imread(fname)
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

    # Find the chess board corners
    ret, corners = cv2.findChessboardCorners(gray, (6,9),None)

    # If found, add object points, image points (after refining them)
    if ret == True:
        print("corners found!")
        objpoints.append(objp)

        corners2 = cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
        imgpoints.append(corners2)

        # Draw and display the corners
        img = cv2.drawChessboardCorners(img, (6,9), corners2,ret)
        #plt.imshow(img, cmap="gray")
        #plt.show()
    else:
        print("corners not found, trying next image.")

ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)
print(mtx)
np.save('camera_mat.npy', mtx)
print(dist)

```

    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    corners found!
    [[3.31666140e+03 0.00000000e+00 2.00701025e+03]
     [0.00000000e+00 3.33904291e+03 1.48586298e+03]
     [0.00000000e+00 0.00000000e+00 1.00000000e+00]]
    [[ 4.66900428e-01 -2.96560179e+00 -1.66952307e-03 -5.79841461e-05
       5.69842923e+00]]

```python

with open("chessboard/20200720_110201.jpg", 'rb') as image_file:
    tags = exifread.process_file(image_file, details=False)
    print(tags["EXIF FocalLength"])
    print(img1.shape)
```

    43/10
    (3000, 4000)

Usually the intrinsic parameters are stored as a matrix, called the *intrinsic matrix*. The form of the intrinsic matrix is as follows:

\begin{equation}
K = \begin{bmatrix}
\alpha_x & 0 & p_x\\
0 & \alpha_y & p_y\\
0 & 0 & 1
\end{bmatrix}
\end{equation}

I loaded the camera matrix from my phone below. Notice that the coordinates of the principal point are close to, but not quite, the middle of the image. Also notice that we introduced $\alpha_x$ and $\alpha_y$, even though we've only worked with $\alpha$ thus far. This is a more general notation, and it takes into account the possibility that cameras might not have square pixels. If one side of a pixel is larger than the other, it makes more sense to talk about focal length measured in pixel width or height. Still, cameras typically have square pixels, and this is reflected in the fact that $\alpha_x$ is very close to $\alpha_y$, so we can chalk up the discrepancy to measurement error.

As a final note, some very specialized cameras have non-rectangular pixels. It turns out $K$ can account for that by changing the value of $K_{1, 2}$, otherwise known as the skew. For most cameras, you can assume it equals 0.

```python
# the cell above takes a very long time to run. I already ran it and saved the matrix.
K = np.load('camera_mat.npy')
print(K)
K

```

    [[3.31666140e+03 0.00000000e+00 2.00701025e+03]
     [0.00000000e+00 3.33904291e+03 1.48586298e+03]
     [0.00000000e+00 0.00000000e+00 1.00000000e+00]]

    array([[3.31666140e+03, 0.00000000e+00, 2.00701025e+03],
           [0.00000000e+00, 3.33904291e+03, 1.48586298e+03],
           [0.00000000e+00, 0.00000000e+00, 1.00000000e+00]])

### Exercise
Print out the chessboard that was provided to you in chessboard_original/ (or use your own chessboard). Using your phone (or whatever other camera), take 20 or so photos of the chessboard lying on a flat surface and upload them to the chessboard/ directory.

Uncomment the cell with the callibration code and run the callibration with your images. Compare the values in the estimated camera matrix with the values you would have expected. How similar are they to the expected values? What is the size of a pixel in your camera in millimeters?

```python

```

## Extrinsic parameters

We now need to find the rotation and translation $R$ and $t$ of both cameras. To reiterate, $t$ is a 3x1 vector that denotes the distance from the origin in $X$, $Y$ and $Z$\*. $R$ is a 3x3 matrix that encodes the rotation of the camera. We made our lives a bit easier by centering the coordinate system on the first camera, such that $t_1 = [0, 0, 0]'$ and $R_1$ is the identity matrix. Remember that we also aligned the coordinate system such that the vector $[0, 0, 1]'$ is in the direction that the camera is pointing. One way to interpret the rotation matrix $R_2$ is by saying that it is the matrix that rotates the vector $[0, 0, 1]'$ into the vector that is aligned with the direction of the second camera.

One of the most amazing results in multiview geometry is that $R_2$ and $t_2$ can both be recovered (up to scale, in the case of $t_2$) just by knowing the corresponding points and the intrinsic parameters. At a very high level, this is a key tenet of something called _epipolar geometry_. Consider images (a) and (b) below. On the image you will find random corresponding points between the two images. On image (a), clearly the camera that took (b) is somewhere to the right. Imagine we could extend (a) so much that you could actually see the camera that took (b), and let's draw a line between the camera that took (b) and all of the corresponding points. What you would find is a series of lines, called epipolar lines, that all converge at the camera that took (b), whose point is called the epipole. This is what we see in (c) and (d).

(source: https://www.youtube.com/watch?v=QzYn0OPO0Yw)

It turns out there is a matrix called the _fundamental matrix_ that encodes this information. This matrix can be estimated using the corresponding points we got before. Furthermore, by combining it with the intrinsic parameters, we get another matrix called the _essential matrix_ which can then be decomposed into a relative rotation and translation $R_2$ and $t_2$, which is what we're after! This is of course a criminally high level summary of how this works. For more information, this is an excellent source: https://www.youtube.com/watch?v=QzYn0OPO0Yw.

\* This is actually a bit incorrect. $t$ represents the vector connecting the center of projection to the origin *from the point of view of the camera being considered*, not the first camera. Still, the general idea is that it denotes displacement.

```python
# converting to pixel coordinates
pts1 = np.int32(pts1)
pts2 = np.int32(pts2)

# Estimate the essential matrix
E, mask = cv2.findEssentialMat(pts1,pts2,K)

# We select only inlier points
pts1_inliers = (pts1[mask.ravel()==1])
pts2_inliers = (pts2[mask.ravel()==1])
```

```python
# Recover pose using what we've learned so far
retval, R, t, mask = cv2.recoverPose(E, pts1_inliers, pts2_inliers, K)
print(R)
print(t)
```

    [[ 0.94182083 -0.09896575  0.32121534]
     [ 0.08510747  0.99474354  0.05693862]
     [-0.32516186 -0.02628816  0.94529292]]
    [[-0.98203659]
     [-0.07243396]
     [ 0.17423392]]

## Finalizing the reconstruction

We now have everything we need to reconstruct the scene in 3D! We have corresponding points in two images, the intrinsic parameters of the camera and the extrinsic parameters as the camera took the two images. There is just one caveat. Remember that the value $t_2$ that we obtained was up to scale. It turns out that our entire scene is scaled such that the length of $t_2 = 1$. This means that while the relative distances between the reconstructed points are conserved, the absolute positions are not.

Because we want to take advantage of visualization software that currently exists, we will use a package called OpenSfM (https://github.com/mapillary/OpenSfM). OpenSfM automatically goes through the trouble of extracting the features, matching them, finding the poses of the cameras and doing the triangulation. One thing it does not do, however, is the camera calibration. We can either provide our own intrinsic camera parameters or we can let the algorithm take its best guess through a process called _bundle adjustment_. Bundle adjustment essentially works by identifying the _reprojection error_ (the error in reprojecting the reconstructed 3D features back onto the image) and slightly modifying the camera positions and parameters, as well as the positions of the features, so that the error is minimized.

As a final note, what do we do when we have more than three images? Clearly if you can find the camera pose of image 2 relative to image 1, then it's possible to find the pose of image 3 relative to image 2. Therefore, multiple images can take part in a reconstruction. Here, bundle adjustment is very important to ensure that errors in the reconstruction do not keep adding up.

Here are the steps to reconstruct a scene:
- Extract the focal length from the metadata
- Detect features from images (in this particular case, we are using so called HAHOG features)
- Match features across images
- Reconstruct the scene using the inferred rotations and translations
- Visualize the scene

OpenSfM has a very good command line interface, so we'll move to the terminal for this portion. However, the commands are on the following cell in case you want to refer back to them.

```python
# Take initial guess of intrinsic parameters through metadata
!opensfm extract_metadata kitchen_example/

# Detect features points
!opensfm detect_features kitchen_example/

# Match feature points across images
!opensfm match_features kitchen_example/

# This creates "tracks" for the features. That is to say, if a feature in image 1 is matched with one in image 2,
# and in turn that one is matched with one in image 3, then it links the matches between 1 and 3. In this case,
# it does not matter since we only have two images
!opensfm create_tracks kitchen_example/

# Calculates the essential matrix, the camera pose and the reconstructed feature points
!opensfm reconstruct kitchen_example/

# For visualization using Open3D
!opensfm export_ply kitchen_example/
```

    2020-07-20 19:56:09,774 INFO: Loading existing EXIF for img1.jpg
    2020-07-20 19:56:09,775 INFO: Loading existing EXIF for img2.jpg
    2020-07-20 19:56:11,803 INFO: Skip recomputing ROOT_HAHOG features for image img1.jpg
    2020-07-20 19:56:11,803 INFO: Skip recomputing ROOT_HAHOG features for image img2.jpg
    2020-07-20 19:56:13,836 INFO: Matching 1 image pairs
    2020-07-20 19:56:13,842 INFO: Computing pair matching with 1 processes
    2020-07-20 19:56:13,867 DEBUG: No segmentation for img2.jpg, no features masked.
    2020-07-20 19:56:13,868 DEBUG: No segmentation for img1.jpg, no features masked.
    2020-07-20 19:56:14,375 DEBUG: Matching img2.jpg and img1.jpg.  Matcher: FLANN (symmetric) T-desc: 0.506 T-robust: 0.001 T-total: 0.508 Matches: 396 Robust: 362 Success: True
    2020-07-20 19:56:14,375 DEBUG: Image img2.jpg matches: 1 out of 1
    2020-07-20 19:56:14,376 DEBUG: Image img1.jpg matches: 0 out of 0
    2020-07-20 19:56:14,376 INFO: Matched 1 pairs for 2 ref_images (perspective-perspective: 1) in 0.5394126999999571 seconds (0.5394131800021569 seconds/pair).
    2020-07-20 19:56:16,438 INFO: reading features
    2020-07-20 19:56:16,463 DEBUG: Merging features onto tracks
    2020-07-20 19:56:16,467 DEBUG: Good tracks: 362
    2020-07-20 19:56:18,502 INFO: Starting incremental reconstruction
    2020-07-20 19:56:18,505 INFO: Starting reconstruction with img1.jpg and img2.jpg
    2020-07-20 19:56:18,521 INFO: Two-view reconstruction inliers: 361 / 362
    2020-07-20 19:56:18,544 INFO: Triangulated: 362
    2020-07-20 19:56:18,551 DEBUG: Ceres Solver Report: Iterations: 4, Initial cost: 8.297278e+00, Final cost: 6.950114e+00, Termination: CONVERGENCE
    2020-07-20 19:56:18,581 DEBUG: Ceres Solver Report: Iterations: 3, Initial cost: 7.186157e+00, Final cost: 6.737168e+00, Termination: CONVERGENCE
    2020-07-20 19:56:18,622 DEBUG: Ceres Solver Report: Iterations: 10, Initial cost: 2.200140e+01, Final cost: 1.615703e+01, Termination: CONVERGENCE
    2020-07-20 19:56:18,624 INFO: Removed outliers: 0
    2020-07-20 19:56:18,624 INFO: -------------------------------------------------------
    2020-07-20 19:56:18,667 DEBUG: Ceres Solver Report: Iterations: 10, Initial cost: 1.938163e+01, Final cost: 1.615703e+01, Termination: CONVERGENCE
    2020-07-20 19:56:18,701 INFO: Removed outliers: 0
    2020-07-20 19:56:18,707 INFO: {'points_count': 362, 'cameras_count': 2, 'observations_count': 724, 'average_track_length': 2.0, 'average_track_length_notwo': -1}
    2020-07-20 19:56:18,707 INFO: Reconstruction 0: 2 images, 362 points
    2020-07-20 19:56:18,707 INFO: 1 partial reconstructions in total.

```python
!opensfm undistort kitchen_example/

!opensfm compute_depthmaps kitchen_example/
```

    2020-07-20 20:41:45,004 DEBUG: Undistorting the reconstruction
    2020-07-20 20:41:45,020 DEBUG: Undistorting image img1.jpg
    2020-07-20 20:41:47,625 DEBUG: Undistorting image img2.jpg
    2020-07-20 20:41:52,189 INFO: Computing neighbors
    2020-07-20 20:41:52,210 INFO: Computing depthmap for image img1.jpg with PATCH_MATCH_SAMPLE
    2020-07-20 20:42:14,751 INFO: Computing depthmap for image img2.jpg with PATCH_MATCH_SAMPLE
    2020-07-20 20:42:37,527 INFO: Cleaning depthmap for image img1.jpg
    2020-07-20 20:42:39,811 INFO: Cleaning depthmap for image img2.jpg
    2020-07-20 20:42:42,106 INFO: Pruning depthmap for image img1.jpg
    2020-07-20 20:42:43,652 INFO: Pruning depthmap for image img2.jpg
    2020-07-20 20:42:45,301 INFO: Merging depthmaps

```python
import open3d as o3d
from open3d import JVisualizer

pcd = o3d.io.read_point_cloud("kitchen_example/reconstruction.ply")
visualizer = JVisualizer()
visualizer.add_geometry(pcd)
visualizer.show()

# UNCOMMENT ONLY IF RUNNING LOCALLY
# o3d.visualization.draw_geometries([pcd])
```

    JVisualizer with 1 geometries

### Exercise
We will now let you try out 3D reconstruction for yourself using OpenSfM! First, we need to make sure you have the proper file structure to make this work. Create a directory called "workspace/" (or whatever else you want). In that directory, create a directory called "images/". From the kitchen_example/ directory, copy "config.yaml" and paste it in workspace/. Take two or more images of your work space and upload them to workspace/images. Now run the commands above to reconstruct the scene and visualize it. Is the reconstruction accurate? What about your scene might make it difficult for the reconstruction to work well?

Go to workspace/reports and examine the files there. They will give you a sense of how many features were extracted, how many features were matched, and whether some reconstructions are only partial. Do you have multiple partial reconstructions or one larger reconstruction? If you have partial reconstructions, why do you think that is?

```python
img1 = cv2.imread("workspace/images/img1.jpg", cv2.IMREAD_GRAYSCALE)
plt.imshow(img1, cmap='gray')
plt.show()

img2 = cv2.imread("workspace/images/img2.jpg", cv2.IMREAD_GRAYSCALE)
plt.imshow(img2, cmap='gray')
plt.show()
```

```python
MAX_FEATURES = 5000

# creating the ORB feature extractor
orb = cv2.ORB_create(MAX_FEATURES)

# creating feature points for first and second images
kp1, dc1 = orb.detectAndCompute(img1, None)
kp2, dc2 = orb.detectAndCompute(img2, None)

img1_ORB = cv2.drawKeypoints(img1, kp1, None, flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
plt.imshow(img1_ORB)
plt.show()
```

```python
GOOD_MATCH_PERCENT = 0.2

# create BFMatcher object
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck = True)

# match descriptors
matches = bf.match(dc1, dc2)

# Sort matches by score
matches.sort(key=lambda x: x.distance, reverse=False)

# Remove worst matches
numGoodMatches = int(len(matches) * GOOD_MATCH_PERCENT)
matches = matches[:numGoodMatches]

img_match = cv2.drawMatches(img1,kp1,img2,kp2,matches,None, flags=2)
plt.imshow(img_match)
plt.show()

pts1 = np.float32([ kp1[m.queryIdx].pt for m in matches ]).reshape(-1,1,2)
pts2 = np.float32([ kp2[m.trainIdx].pt for m in matches ]).reshape(-1,1,2)
```

```python
K = np.array([[3.31666140e+03, 0.00000000e+00, 2.00701025e+03],
       [0.00000000e+00, 3.33904291e+03, 1.48586298e+03],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00]])
```

```python
pts1 = np.int32(pts1)
pts2 = np.int32(pts2)

# Estimate the essential matrix
E, mask = cv2.findEssentialMat(pts1,pts2,K)

# We select only inlier points
pts1_inliers = (pts1[mask.ravel()==1])
pts2_inliers = (pts2[mask.ravel()==1])
```

```python
retval, R, t, mask = cv2.recoverPose(E, pts1_inliers, pts2_inliers, K)
print(R)
print(t)
```

    [[ 0.98359866 -0.12902517  0.12604038]
     [ 0.12829603  0.99163822  0.01392003]
     [-0.1267825   0.00247875  0.99192744]]
    [[-0.97485292]
     [-0.15737803]
     [ 0.15777813]]

```python
# Take initial guess of intrinsic parameters through metadata
!opensfm extract_metadata workspace/

# Detect features points
!opensfm detect_features workspace/

# Match feature points across images
!opensfm match_features workspace/

# This creates "tracks" for the features. That is to say, if a feature in image 1 is matched with one in image 2,
# and in turn that one is matched with one in image 3, then it links the matches between 1 and 3. In this case,
# it does not matter since we only have two images
!opensfm create_tracks workspace/

# Calculates the essential matrix, the camera pose and the reconstructed feature points
!opensfm reconstruct workspace/

# For visualization using Open3D
!opensfm export_ply workspace/

```

    2020-07-20 21:23:01,928 INFO: Loading existing EXIF for img1.jpg
    2020-07-20 21:23:01,929 INFO: Loading existing EXIF for img2.jpg
    2020-07-20 21:23:03,837 INFO: Skip recomputing ROOT_HAHOG features for image img1.jpg
    2020-07-20 21:23:03,837 INFO: Skip recomputing ROOT_HAHOG features for image img2.jpg
    2020-07-20 21:23:05,747 INFO: Matching 1 image pairs
    2020-07-20 21:23:05,753 INFO: Computing pair matching with 1 processes
    2020-07-20 21:23:05,777 DEBUG: No segmentation for img2.jpg, no features masked.
    2020-07-20 21:23:05,778 DEBUG: No segmentation for img1.jpg, no features masked.
    2020-07-20 21:23:06,235 DEBUG: Matching img2.jpg and img1.jpg.  Matcher: FLANN (symmetric) T-desc: 0.455 T-robust: 0.002 T-total: 0.457 Matches: 818 Robust: 778 Success: True
    2020-07-20 21:23:06,236 DEBUG: Image img2.jpg matches: 1 out of 1
    2020-07-20 21:23:06,236 DEBUG: Image img1.jpg matches: 0 out of 0
    2020-07-20 21:23:06,236 INFO: Matched 1 pairs for 2 ref_images (perspective-perspective: 1) in 0.4883900179993361 seconds (0.4883904779999284 seconds/pair).
    2020-07-20 21:23:08,128 INFO: reading features
    2020-07-20 21:23:08,152 DEBUG: Merging features onto tracks
    2020-07-20 21:23:08,159 DEBUG: Good tracks: 778
    2020-07-20 21:23:10,019 INFO: Starting incremental reconstruction
    2020-07-20 21:23:10,025 INFO: Starting reconstruction with img1.jpg and img2.jpg
    2020-07-20 21:23:10,058 INFO: Two-view reconstruction inliers: 778 / 778
    2020-07-20 21:23:10,104 INFO: Triangulated: 778
    2020-07-20 21:23:10,117 DEBUG: Ceres Solver Report: Iterations: 3, Initial cost: 1.095092e+01, Final cost: 8.994927e+00, Termination: CONVERGENCE
    2020-07-20 21:23:10,213 DEBUG: Ceres Solver Report: Iterations: 3, Initial cost: 9.416828e+00, Final cost: 8.694985e+00, Termination: CONVERGENCE
    2020-07-20 21:23:10,310 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 2.391270e+01, Final cost: 1.953910e+01, Termination: CONVERGENCE
    2020-07-20 21:23:10,315 INFO: Removed outliers: 0
    2020-07-20 21:23:10,316 INFO: -------------------------------------------------------
    2020-07-20 21:23:10,418 DEBUG: Ceres Solver Report: Iterations: 5, Initial cost: 2.176169e+01, Final cost: 1.953908e+01, Termination: CONVERGENCE
    2020-07-20 21:23:10,424 INFO: Removed outliers: 0
    2020-07-20 21:23:10,435 INFO: {'points_count': 778, 'cameras_count': 2, 'observations_count': 1556, 'average_track_length': 2.0, 'average_track_length_notwo': -1}
    2020-07-20 21:23:10,435 INFO: Reconstruction 0: 2 images, 778 points
    2020-07-20 21:23:10,435 INFO: 1 partial reconstructions in total.

```python

```

```python
!opensfm undistort workspace/

```

    2020-07-20 21:23:16,587 DEBUG: Undistorting the reconstruction
    2020-07-20 21:23:16,616 DEBUG: Undistorting image img1.jpg
    2020-07-20 21:23:18,852 DEBUG: Undistorting image img2.jpg

```python

!opensfm compute_depthmaps workspace/
```

    2020-07-20 21:23:25,163 INFO: Computing neighbors
    2020-07-20 21:23:25,196 INFO: Using precomputed raw depthmap img1.jpg
    2020-07-20 21:23:25,196 INFO: Using precomputed raw depthmap img2.jpg
    2020-07-20 21:23:25,196 INFO: Using precomputed clean depthmap img1.jpg
    2020-07-20 21:23:25,196 INFO: Using precomputed clean depthmap img2.jpg
    2020-07-20 21:23:25,196 INFO: Using precomputed pruned depthmap img1.jpg
    2020-07-20 21:23:25,196 INFO: Using precomputed pruned depthmap img2.jpg
    2020-07-20 21:23:25,196 INFO: Merging depthmaps

```python
import open3d as o3d
from open3d import JVisualizer

pcd = o3d.io.read_point_cloud("workspace/undistorted/depthmaps/merged.ply")
visualizer = JVisualizer()
visualizer.add_geometry(pcd)
visualizer.show()

# UNCOMMENT ONLY IF RUNNING LOCALLY
# o3d.visualization.draw_geometries([pcd])
```

    JVisualizer with 1 geometries

```python

```