# `DSML Workshop 08` - Multiple Linear Regression and Polynomial Regression

In this workshop continue with hands-on machine learning focusing mostly on the non-linear regression example (polynomial regression) covererd previously in the lecture.

This workshop is structured as follows:
1. Using multiple features in linear regression
1. Non-linear modeling and polynomial regression (by hand and in scikit learn)
1. Cross-validation techniques

```python
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import datetime

%matplotlib inline
```

## Example: predicting peak electrical power

We will again work with our electric power example from last week which we retieved from PJM from the following link [here](https://dataminer2.pjm.com/feed/hrl_load_metered/definition)). The files we are loading are the raw files we downloaded from this source. The final input data for our code is `Pittsburgh_load_data.csv`.

```python
df = pd.read_csv("Pittsburgh_load_data.csv")
df["Date"] = pd.to_datetime(df["Date"], format="%d.%m.%Y")
df["Month"] = df["Date"].apply(lambda x: x.month)
df.head()
```

      Date
      AVG
      MAX
      MIN
      Total
      High_temp
      Avg_temp
      Month

      0
      2013-01-01
      1.598524
      1.859947
      0.001599
      38.368031
      0.0
      -1.68
      1

      1
      2013-01-02
      1.809347
      2.054215
      0.001809
      43.428194
      -3.9
      -6.58
      1

      2
      2013-01-03
      1.832822
      2.049550
      0.001833
      43.991607
      0.6
      -6.12
      1

      3
      2013-01-04
      1.812699
      2.008168
      0.001813
      43.508609
      0.0
      -1.95
      1

      4
      2013-01-05
      1.662036
      1.838251
      0.001662
      39.892360
      1.7
      -1.47
      1

**Recall from last week's workshop**: We fitted a linear model to the summer period of our electricity data. To do so we used `scikit learn` and obtained the following result:

```python
# limit to summer month
df_summer = df[(df["Month"] > 5) & (df["Month"] < 9)]
```

```python
X = df_summer["High_temp"].values.reshape((-1,1)) # if we pass a 1-feature array we need to re-shape it!
y = df_summer["MAX"]
```

```python
# initialize model
lin_mod = LinearRegression()

#train model
lin_mod.fit(X, y)

#make predictions using model
model_pred = lin_mod.predict(X)

# plot resulzts
plt.figure(figsize = (8,6))
plt.scatter(X, y, marker="x")
plt.xlabel("High Temperature (°C)")
plt.ylabel("Peak Demand (GW)")
plt.plot(X, model_pred, c='C2')
plt.legend(['Squared loss fit','Observed days'])
print(lin_mod.coef_, lin_mod.intercept_)
```

## Multi-dimensional linear regression

How do we now incorporate multiple features into our model? Essentially all we nee to do is pass a larger feature verctor to our model. To see this let us again consider the example of electricity demand:

We assume that consumption behavior is differnt on weekends. Can you think of the key reasons why this might be the case?

Let us create a boolean indicator feature entitled `IsWeekday` that returns True if the day is a weekday and False if it falls on a weekend.

```python
df_summer.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 459 entries, 151 to 1702
    Data columns (total 8 columns):
     #   Column     Non-Null Count  Dtype
    ---  ------     --------------  -----
     0   Date       459 non-null    datetime64[ns]
     1   AVG        459 non-null    float64
     2   MAX        459 non-null    float64
     3   MIN        459 non-null    float64
     4   Total      459 non-null    float64
     5   High_temp  459 non-null    float64
     6   Avg_temp   459 non-null    float64
     7   Month      459 non-null    int64
    dtypes: datetime64[ns](1), float64(6), int64(1)
    memory usage: 32.3 KB

```python
# add the weekday column to our data frame
# note: we are using here the very powerful map() function, which returns a list of the results
# after applying the given function to each item of a given iterable (list, tuple etc.)

df_summer["Weekday"] = df_summer["Date"].apply(lambda dt: dt.weekday())
```

```python
df_summer.head(5)
```

      Date
      AVG
      MAX
      MIN
      Total
      High_temp
      Avg_temp
      Month
      Weekday

      151
      2013-06-01
      1.946586
      2.298662
      0.001947
      46.722317
      29.4
      24.79
      6
      5

      152
      2013-06-02
      1.787233
      2.026075
      0.001787
      42.897404
      26.7
      23.46
      6
      6

      153
      2013-06-03
      1.664323
      1.832544
      0.001664
      39.947248
      20.6
      17.11
      6
      0

      154
      2013-06-04
      1.581505
      1.827014
      0.001582
      37.959541
      22.2
      15.40
      6
      1

      155
      2013-06-05
      1.736095
      2.109310
      0.001736
      41.670127
      26.7
      19.91
      6
      2

```python
# alternatively you can also define you own function

def weekday_check (dt):

    day_number = dt.weekday()

    if day_number <=4:
        return 1
    else:
        return 0

df_summer["IsWeekday"] = df_summer["Date"].apply(lambda dt: weekday_check(dt))
```

```python
# alternative
df_summer["IsWeekday"] = df_summer["Weekday"].apply(lambda x: 1 if x<=4 else 0)
```

```python
df_summer.head(10)
```

      Date
      AVG
      MAX
      MIN
      Total
      High_temp
      Avg_temp
      Month
      Weekday
      IsWeekday

      151
      2013-06-01
      1.946586
      2.298662
      0.001947
      46.722317
      29.4
      24.79
      6
      5
      0

      152
      2013-06-02
      1.787233
      2.026075
      0.001787
      42.897404
      26.7
      23.46
      6
      6
      0

      153
      2013-06-03
      1.664323
      1.832544
      0.001664
      39.947248
      20.6
      17.11
      6
      0
      1

      154
      2013-06-04
      1.581505
      1.827014
      0.001582
      37.959541
      22.2
      15.40
      6
      1
      1

      155
      2013-06-05
      1.736095
      2.109310
      0.001736
      41.670127
      26.7
      19.91
      6
      2
      1

      156
      2013-06-06
      1.734597
      1.923417
      0.001735
      41.633988
      21.7
      19.28
      6
      3
      1

      157
      2013-06-07
      1.595062
      1.799999
      0.001595
      38.284876
      18.9
      16.73
      6
      4
      1

      158
      2013-06-08
      1.415416
      1.568676
      0.001415
      33.972964
      21.1
      16.87
      6
      5
      0

      159
      2013-06-09
      1.549465
      1.915678
      0.001549
      37.190627
      27.2
      20.94
      6
      6
      0

      160
      2013-06-10
      1.812117
      2.072227
      0.001812
      43.494687
      23.9
      21.28
      6
      0
      1

Let us plot the results (this time using seaborn for convenience)

```python
plt.figure(figsize = (8,6))

sns.scatterplot(x=df_summer["High_temp"], y=df_summer["MAX"],hue=df_summer["IsWeekday"], palette="mako")
plt.show()
```

What do you observe? Is this in line with you expectation?

Let us now train a new linear model with an additional feature for weekdays. As mentioned, all we need to do is to pass a larger feature vector to our linear model.

```python
#y
```

```python
X = df_summer[["High_temp", "IsWeekday"]]
y = df_summer[["MAX"]]
lin_mod_day = LinearRegression()
lin_mod_day.fit(X,y)
print(lin_mod_day.coef_, lin_mod_day.intercept_)
```

    [[0.08541718 0.22431882]] [-0.28201097]

```python
#X
```

**Question:** Note that we have now added a second element to our coefficient vector. Can you interpret the results?

Let us plot this

```python
X_weekday = X[X["IsWeekday"]==1]
X_weekend = X[X["IsWeekday"]==0]

# predict weekends and weekdays seperately (usning same model!)
y_pred_weekday = lin_mod_day.predict(X_weekday)
y_pred_weekend = lin_mod_day.predict(X_weekend)

plt.figure(figsize = (8,6))

plt.scatter(X_weekday["High_temp"],
            df_summer[df_summer["IsWeekday"]==True]["MAX"],
            marker="+", label="Observed Weekdays")

plt.scatter(X_weekend["High_temp"],
            df_summer[df_summer["IsWeekday"]==False]["MAX"],
            marker="x", label="Observed Weekends")

plt.plot(X_weekday["High_temp"],
         y_pred_weekday,
         label="Weekday Prediction")
plt.plot(X_weekend["High_temp"],
         y_pred_weekend,
         label="Weekend Prediction")

plt.xlabel("High Temperature (°C)")
plt.ylabel("Peak Demand (GW)")

plt.legend()
plt.show()
```

**Exercise**: How has the inclusion of an additional feature improved training performance? Report MAE, RMSE and $R^2$ of both model setups on the full dataset. Explain your results.

**Caution**: This is the training error! Do not confuse it with actual model predictive performance which can only be evaluated on unseen data!

```python
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
```

```python
# YOUR CODE HERE

# define X and y vectors
#target
y_true = df_summer["MAX"]
#features
X_1d = df_summer["High_temp"].values.reshape((-1,1)) # need to reshape 1d vector
X_2d = df_summer[["High_temp","IsWeekday"]]

#make predictions (models have already been fitted above!)
y_pred_1d = lin_mod.predict(X_1d)
y_pred_2d = lin_mod_day.predict(X_2d)

print("MAE 1D:", mean_absolute_error(y_true,y_pred_1d))
print("MAE 2D:", mean_absolute_error(y_true,y_pred_2d))

print("RMSE 1D:", mean_squared_error(y_true,y_pred_1d)**0.5)
print("RMSE 2D:", mean_squared_error(y_true,y_pred_2d)**0.5)

print("R2 1D:", r2_score(y_true,y_pred_1d))
print("R2 2D:", r2_score(y_true,y_pred_2d))
```

    MAE 1D: 0.1259610051963046
    MAE 2D: 0.09542776362321943
    RMSE 1D: 0.1567504679751341
    RMSE 2D: 0.1195172899949024
    R2 1D: 0.7301181597178452
    R2 2D: 0.8431019864973227

## `Non-linear modelling (polynomial regression)`

The relationship between high temperature and electrical demand is well-modelled by a linear function during the summer months, when (at least in Pittsburgh), electricity in summer is dominated by air conditioners (so with higher temperatures comes higher consumption).

However, this is clearly not the case of the entire year. Indeed, if our previous linear model is to be believed then with lower temperatures we would continue to have lower and lower consumption (until, at some point, we’d start generating electricity). Naturally, this is not the case, and if we instead consider the entire year of high temperature and peak dmeand, or average temperature and average demand then another picture emerges.

Let us define the following common notation:

- xp = High temperature
- yp = Peak demand
- xa = Average temperature
- ya = Average demand

```python
# note that we know use the full data (not just summer)

xp = df["High_temp"].values
yp = df["MAX"].values
xa = df["Avg_temp"].values
ya = df["AVG"].values
```

```python
plt.figure(figsize = (8,6))
plt.scatter(xp, yp, marker="x")
plt.xlabel("High Temperature (°C)")
plt.ylabel("Peak Demand (GW)")
plt.show()
```

This captures the effects we expect. Although peak demand increases sharply with consumption after a high temperature of 22 degrees or so (precisely the range where air conditioning usually starts), below this the demand actually starts to increase with lower temperature – though not with as steep a slope, due to the fact, for example, that most heating in Pittsburgh is done with gas rather than with electricity, and other loads that do increase with lower temperatures tend to be smaller in magnitude than air conditioning units.

### Linear regression
If we were to fit a linear model to this data, it would look like the following:

```python
# initialize, fit and predict
lr = LinearRegression()
lr.fit(xp.reshape(-1,1), yp)
model_pred_p = lr.predict(xp.reshape(-1,1))

# plot figure
plt.figure(figsize = (8,6))
plt.scatter(xp, yp, marker="x" )
plt.xlabel("High Temperature (°C)")
plt.ylabel("Peak Demand (GW)")
plt.plot(xp, model_pred_p, c='C2')
print ('The R^2 of linear regression is: ',r2_score(yp, model_pred_p))
```

### Polynomial regression (quadratic)
To capture data of this type, we clearly want some way of expressing nonlinear relationships in the data.  Fortunately, this is possible without actually leaving the domain of so-called "linear regresion".  The trick we are going to use is a simple one: rather than have features $x^{(i)}$ which only include the "raw" inputs such as temperatuare (plus other raw inputs such as weekday indicators that we saw previously), we are going to make build features that include _nonlinear_ functions of the underlying inputs.  For example, we could choose the following features
\begin{equation}
x^{(i)} = \left [ \begin{array}{c} (\mathrm{HighTemperature}^{(i)})^2 \\
\mathrm{HighTemperature}^{(i)} \\ 1 \end{array} \right ]
\end{equation}
which also include a quadratic function of the high temperature variable.  If we choose this representation, then our linear hypothesis function $h_\theta(x) = \theta^Tx$ is now given by
\begin{equation}
h_\theta(x) = \theta_1 (\mathrm{HighTemperature}^{(i)})^2 + \theta_2 \mathrm{HighTemperature}^{(i)} + \theta_3
\end{equation}
which is a _quadratic_ function of the high temperature.  Importantly, however, the hypothesis function is still a linear function of the parameters, and so the exact same solution methods work as before (including the exact solution), just by passing a different feature matrix.

Let us first do this by hand. We define a simple function entitled `plot_regression_poly` that takes features, target and the degree of the polynomial as input.

In this setting we're actually going to standardize features to the range $[-1,+1]$ even with the exactly least-squares solution for numerical reasons.  High polynomials get very large very quickly, and if we aren't careful it's easy to overload the range of double precision floating point values.

To standardize to the range $[-1,+1]$ we use the following formula:
\begin{equation}
2*\frac{x - min(x)}{max(x) - min(x)}-1
\end{equation}

```python
# x is the input variable
# y is the output vaiable
# d is degree pf polynomial regression

def plot_regression_poly(x, y, d):

    # Create polynomial features
    min_x, max_x = x.min(), x.max()
    xs = 2*(x - min_x)/(max_x - min_x)-1  # standardize to range [-1,1]
    X = np.array([xs**i for i in range(d,-1,-1)]).T

    # Implement polynomial regression using least squares (we use the nomal equations as derived in the lecture)
    theta = np.linalg.solve(X.T @ X, X.T @ y)
    xt0 = np.linspace(min_x-1, max_x+1, 400)

    # create scaled test dataset
    xt = 2* (xt0 - min_x)/(max_x - min_x) -1
    Xt = np.array([xt**i for i in range(d,-1,-1)]).T
    yt = Xt @ theta

    # Plot results
    plt.figure(figsize = (8,6))
    plt.scatter(x, y, marker="x")
    ylim = plt.ylim()
    plt.plot(xt0, yt, 'C1')
    plt.xlabel("Temperature (°C)")
    plt.ylabel("Demand (GW)")
    plt.xlim([min_x-2, max_x+2])
    plt.ylim(ylim)
    print(theta[:7]) # prints first 3 thetas
```

```python
plot_regression_poly(x = xp, y=yp, d = 2)
```

This looks better, but quadratic function is symmetric around its minimum point, and the data we're trying to fit is definitely not symmetric. Thus, we may want a cubic function.

### Polynomial regression (Cubic)

```python
# Plot of Peak demand vs. High temperature
plot_regression_poly(x = xp, y=yp, d = 3)
```

Still not bad. Let's keep going:

```python
# Try some other values of d --> What seems to be a good one?

plot_regression_poly(x = xp, y=yp, d = 6)
```

How about a 100-degree polynomial? Is this something we want? Also, what do you think about the value of the coefficients?

```python
plot_regression_poly(x = xp, y=yp, d = 100)
```

Something very bad has happened at this point. Especially at the tail ends of the data, where the data points are spaced less closely, we start to get very odd patterns for the data. But the important point is that this is actually a very good fit to the data from a least-squares perspective. As you can see from the figure, the lines pass exactly through many of the data point (most obvious on the left hand side of the plot), whereas for the "better" fits we had above, our function didn't pass exactly through those points, so actually suffered more loss. But there is an obvious way in which the degree 100 polynomial fit, despite having lower loss, is actually a worse approximation to the underlying data. This brings us to the second topic of these notes, which is the issue of generlization and overfitting.

We will see shortly how we manage these situations, but let us first implement the above example using `scikit learn`. Essentially all we need to do is to pre-process our feature vector and create polynomial features. To do so we use a built-in module called `PolynomialFeatures`.

```python
from sklearn.preprocessing import PolynomialFeatures
```

Let us first consider a toy example to understand how this module works

```python
Q = np.array([[2,3,4], [5,6,7]])

#initialize PolynomialFeatures
PF = PolynomialFeatures(degree = 4,interaction_only = False, include_bias = True)
Q_Poly = PF.fit_transform(Q.reshape(-1,1))
print(Q)
print(Q_Poly)
```

    [[2 3 4]
     [5 6 7]]
    [[1.000e+00 2.000e+00 4.000e+00 8.000e+00 1.600e+01]
     [1.000e+00 3.000e+00 9.000e+00 2.700e+01 8.100e+01]
     [1.000e+00 4.000e+00 1.600e+01 6.400e+01 2.560e+02]
     [1.000e+00 5.000e+00 2.500e+01 1.250e+02 6.250e+02]
     [1.000e+00 6.000e+00 3.600e+01 2.160e+02 1.296e+03]
     [1.000e+00 7.000e+00 4.900e+01 3.430e+02 2.401e+03]]

**Note**: This will generate a new feature matrix consisting of all polynomial combinations
of the features with degree less than or equal to the specified degree. For example, if an input sample is two dimensional and of the form [a, b], the degree-2 polynomial features are [1, a, b, a^2, ab, b^2]. This is slightly different from what we have done manually above but is a more common way of creating polynomial features.

```python
def plot_regression_poly_scikit(X,Y,d):

    # initialize PolynomialFeatures
    poly_reg = PolynomialFeatures(degree = d)

    # Polynomial transformation
    x_poly = poly_reg.fit_transform(X.reshape(-1,1))

    # Fitting linear regression to polynomial features
    lin_reg_Poly = LinearRegression()
    lin_reg_Poly.fit(x_poly, Y)
    model_pred = lin_reg_Poly.predict(x_poly)

    # Plotting the regression line and the data (we have to transform the inputs as well!)
    x_fit = np.arange(X.min(),X.max() ,1)[:, np.newaxis]
    y_pred = lin_reg_Poly.predict(poly_reg.fit_transform(x_fit.reshape(-1,1)))

    plt.figure(figsize = (8,6))
    plt.scatter(X,Y,marker="x", c='C2')
    ylim = plt.ylim()
    plt.plot(x_fit,y_pred, c='C1')
    plt.xlabel("Temperature (°C)")
    plt.ylabel("Demand (GW)")
    plt.xlim([X.min()-2,X.max()+2]) # leave some space before and after limits
    plt.ylim(ylim)
    print ('The R^2 for quadratic curve is: ',r2_score(Y, model_pred))
    #print(lin_reg_Poly.coef_)
```

```python
plot_regression_poly_scikit(X = xp, Y = yp, d = 8)
```

## `Generalization and overfitting`

Recall from our previous discussions that the canoncial machine learning problem that we solve with every algorithm is the following
\begin{equation}
\DeclareMathOperator*{\minimize}{minimize}
\minimize_\theta \; \frac{1}{m}\sum_{i=1}^m \ell \left(h_\theta(x^{(i)},y^{(i)} \right)
\end{equation}
i.e., to minimize the sum of losses on the data set.  However, in a more fundamental way, this is not really our goal.  We ultimately do not care about achieving low loss specifically on the points that we are feeding into the algorithm: we already _know_ the true output for each of these data points, and if we want to "predict" these points precisely, we could just look up each point in a database (assuming we have some way of referring to the points uniquely).  What we _really_ want from a machine learning algorithm is the ability to predict will on _new_ data points _of the same type_ as those we learned our model on.  We don't care about knowing was the peak demand _was_ on previous days we have already seen; we care about being able to predict what the peak demand will _be_ tomorrow given the high temperature as input.

**Generalization error**

This discussion leads us to the notion of _generalization error_.  Informally, the generalization error is just the error (or more generally, the loss) we would experience not on the training data itself, but on new data drawn from the "same distirbution" as the training data.  _This_ is really the quantity we want to minimize, not the error on the training data itself.  Because when we run our machine learning algorithm on new data, all that will matter is its generalization performance.

What the above example illustrated was an instance of _overfitting_, the situation where the training error is low, but the generalization error is high.  This occurs because we are explicitly trying to minimize the loss on the training set, and if the expressiveness of of hypothesis function is small enough, then we can make this loss arbitrarily small, usually by fitting to pecularities in the training data that provide no real benefit when looking at new data.

**Overfitting and model complexity**

In general, we expect something like the following behavior when it comes to overfitting.  We're showing the "cartoon" form here but we will shortly see what this looks like in practice.

When model complexity is low, both training and generalization loss are high (here model complexity can mean any type of representational power of the model, but since we have covered this so far, you can think of it just as the degree of the polynomial in our feature vector).  As we increase model complexity, the both training and generalization performance start to decrease with training loss usually slightly lower that generalization loss (due to the simple fact tha we explicitly optimized training loss).  As we futher increase model complexity, training loss will continue to only decrease: by adding additional representational power to our model, we will only fit the data better and better, since we are explicitly choosing parameters to minimize this loss.  But at a certain point, generalization loss will start to increase again.  Our goal when choosing the "right" model for a particular machine learning problem is to find the model with lowest generalization error, the minimum in the red line above.  However, we cannot do so using the training set alone, because performance on the training set (the blue line), gives us no clue at to the performance of the generalization loss.

### Standard cross validation

The idea of cross validation is simple.  Given the entirety of our training data, we take some fixed percentage, say 70% of the data, and call this our "new" training set.  We then take the remaining data (30% in this case), and call it the _holdout_ or _validation_ set.  The basic idea is that we will use the emprical error or loss on this holdout set as an approximation for the generalization error.  This works because, unlike the training set, we do _not_ choose the parameters of the model based upon the validation set.  This means that there is no way for the parameters to overfit to this data, and thus the validation set still provides a reasonable estimate of generalization error even _after_ the parameters have been trained.

Let's see how to generate these training and validation sets.

#### Train-test split by hand

```python
# create an array
X = df["High_temp"]

# create randomly shuffled list of indexes
np.random.seed(10) # This method is called when RandomState is initialized
perm = np.random.permutation(len(X))

# select first 70% indexes of shuffled list as train set
idx_train = perm[:int(len(perm)*0.7)]

# select last 30% indexes of shuffled list as train set
idx_cv = perm[int(len(perm)*0.7):]

x_train, y_train = df["High_temp"].iloc[idx_train].values, df["MAX"].iloc[idx_train].values
x_cv, y_cv = df["High_temp"].iloc[idx_cv].values, df["MAX"].iloc[idx_cv].values
```

```python
plt.figure(figsize = (8,6))
plt.scatter(x_train, y_train, marker='x', color="C0")
plt.scatter(x_cv, y_cv, marker='x', color="C2")
plt.xlabel("Temperature (°C)")
plt.ylabel("Demand (GW)")
plt.legend(['Training set', 'Holdout set'])
plt.show()
#plt.savefig('crossvalidation.png', dpi = 300)
```

```python
len(x_train)
```

    1277

```python
len(x_cv)
```

    548

#### Train-test split with `scikit learn`

We can also make our life earsier and use the `train_test_split` in `scikit learn`.

```python
from sklearn.model_selection import train_test_split
```

```python
# split the data with 70-30% split as above
x_train, x_test, y_train, y_test = train_test_split(df["High_temp"], df["MAX"], test_size=0.3,random_state=42)
```

This gives us a very similar random test split (see figure below).

```python
plt.figure(figsize = (8,6))
plt.scatter(x_train, y_train, marker='x', color="C0")
plt.scatter(x_test, y_test, marker='x', color="C2")
plt.xlabel("Temperature (°C)")
plt.ylabel("Demand (GW)")
plt.legend(['Training set', 'Holdout set'])
plt.show()
#plt.savefig('crossvalidation.pdf', dpi = 300)
```

```python
len(x_train)
```

    1277

```python
len(x_test)
```

    548

```python
def ls_poly(x, y, d): #ls=lest squares

    # Create polynomial features
    min_x, max_x = x.min(), x.max()
    xs = 2*(x - min_x)/(max_x - min_x)-1  # standardize to range [-1,1]
    X = np.array([xs**i for i in range(d,-1,-1)]).T

    # Implement polynomial regression using least squares (we use the nomal equations as derived in the lecture)
    theta = np.linalg.solve(X.T @ X, X.T @ y)

    return theta
```

```python
min_x_train, max_x_train = x_train.min(), x_train.max()
x_train = 2*(x_train - min_x_train)/(max_x_train - min_x_train) - 1
x_test = 2*(x_test - min_x_train)/(max_x_train - min_x_train) - 1

def poly_feat(x, degree):
    return np.array([x**i for i in range(degree,-1,-1)]).T

err_train = []
err_cv = []
for i in range(50):
    theta = ls_poly(x_train, y_train, i)
    err_train.append(((poly_feat(x_train,i) @ theta - y_train)**2).mean())
    err_cv.append(((poly_feat(x_test,i) @ theta - y_test)**2).mean())

plt.figure(figsize = (8,6))
plt.semilogy(range(50), err_train, range(50), err_cv)
plt.legend(["Training", "Validation"])
plt.xlabel("Polynomial degree")
plt.ylabel("Mean squared error")
```

Let's zoom in on the "good" region...

```python
min_x_train, max_x_train = x_train.min(), x_train.max()
x_train = 2*(x_train - min_x_train)/(max_x_train - min_x_train) - 1
x_test = 2*(x_test - min_x_train)/(max_x_train - min_x_train) - 1

def poly_feat(x, degree):
    return np.array([x**i for i in range(degree,-1,-1)]).T

err_train = []
err_cv = []
for i in range(20):
    theta = ls_poly(x_train, y_train, i)
    err_train.append(((poly_feat(x_train,i) @ theta - y_train)**2).mean())
    err_cv.append(((poly_feat(x_test,i) @ theta - y_test)**2).mean())

plt.figure(figsize = (8,6))
plt.semilogy(range(20), err_train, range(20), err_cv)
plt.legend(["Training", "Validation"])
plt.xlabel("Polynomial degree")
plt.ylabel("Mean squared error")
```

**Exercise**: Re-produce the above graph for the case of predicting **average electical load** from **average temperature** using `scikit learn`. To do so follow the steps below:

- Define train and test datasets using `train_test_split`
- Define a polynomial model with variable degree inputs using `PolynomialFeatures` and `LinearModel`
- Write a simple loop to train the model for degrees ranging from 0-50 on the training dataset
- Compute and store the results in two sperate arrays (as done above)
- Plot your results

```python
# YOUR CODE HERE

#define X and y vectors
xa = df["Avg_temp"]
ya = df["AVG"]

# train-test split
X_train, X_test, y_train, y_test = train_test_split(xa, ya, test_size=0.3,random_state=42)
```

```python
# YOUR CODE HERE
#define loop
err_train = []
err_cv = []
for deg in range(50): # we want to loop across 0-50 degrees

    # create poly features
    poly_feat = PolynomialFeatures(degree=deg) #initialize

    X_train_poly=poly_feat.fit_transform(X_train.values.reshape((-1,1)))
    X_test_poly=poly_feat.fit_transform(X_test.values.reshape((-1,1)))

    # initialize and fit model
    lin_mod_poly = LinearRegression()
    lin_mod_poly.fit(X_train_poly,y_train)

    # compute errors and append to err_train and err_cv
    err_train.append(mean_squared_error(y_train,lin_mod_poly.predict(X_train_poly)))
    err_cv.append(mean_squared_error(y_test,lin_mod_poly.predict(X_test_poly)))

# plot results
plt.figure(figsize = (8,6))
plt.semilogy(range(50), err_train, range(50), err_cv)
plt.legend(["Training", "Validation"])
plt.xlabel("Polynomial degree")
plt.ylabel("Mean squared error")
```

Note that the error curve naturaly looks slightly different from above since our targets and predictors are the average load and temperature respectively.

### Cross validation using train/holdout/test splits

As we have discussed in last week's lecture it is advisable to actually do a three-ways split of the dataset to avoid leakage. You would typically do the following:

1. Divide data into training
set, holdout set, and test
set
2. Train algorithm on training
set (i.e., to learn
parameters), use holdout
set to select
hyperparameters
3. (Optional) retrain system
on training + holdout
4. Evaluate performance on
test set

Naturally, as above, we can do the split manually. Here is the code:

```python
# create an array
X = df["High_temp"]

# create randomly shuffled list of indexes
np.random.seed(10) # This method is called when RandomState is initialized
perm = np.random.permutation(len(X))

# select first 70% indexes of shuffled list as train set
idx_train = perm[:int(len(perm)*0.5)]

# select last 30% indexes of shuffled list as train set
idx_hold= perm[int(len(perm)*0.5):int(len(perm)*0.7)]

# select last 30% indexes of shuffled list as train set
idx_test= perm[int(len(perm)*0.7):int(len(perm)*1)]

x_train, y_train = df["High_temp"].iloc[idx_train].values, df["MAX"].iloc[idx_train].values
x_hold, y_hold = df["High_temp"].iloc[idx_hold].values, df["MAX"].iloc[idx_hold].values
x_test, y_test = df["High_temp"].iloc[idx_test].values, df["MAX"].iloc[idx_test].values
```

```python
print(len(x_train),len(x_hold),len(x_test))
```

    912 365 548

But we can do it in `scikit learn` a bit more easily. Please have a go yourself.

**Exercise**: Create a train/holdout/test set of with respective proportions of 50%, 20% and 30% using `scikit learn's` `train_test_split` module consecutively. Visualize your results.

```python
# YOUR CODE HERE
#Do a 70-30 split first
X_train, X_test, y_train, y_test = train_test_split(xp, yp, test_size=0.3,random_state=34 )

# now split X_train to achive 50-20-30 split
X_train, X_hold, y_train, y_hold = train_test_split(X_train, y_train, test_size=(0.2/0.7),random_state=34 )
```

```python
print(len(X_train),len(X_hold),len(X_test))
```

    912 365 548

```python
min_x_train, max_x_train = X_train.min(), X_train.max()
x_train = 2*(X_train - min_x_train)/(max_x_train - min_x_train) - 1
x_hold = 2*(X_hold - min_x_train)/(max_x_train - min_x_train) - 1
x_test = 2*(X_test - min_x_train)/(max_x_train - min_x_train) - 1

def poly_feat(x, degree):
    return np.array([x**i for i in range(degree,-1,-1)]).T

err_train = []
err_hold = []
err_test = []
for i in range(15):
    theta = ls_poly(x_train, y_train, i)
    err_train.append(((poly_feat(x_train,i) @ theta - y_train)**2).mean())
    err_hold.append(((poly_feat(x_hold,i) @ theta - y_hold)**2).mean())
    err_test.append(((poly_feat(x_test,i) @ theta - y_test)**2).mean())
plt.semilogy(range(15), err_train, range(15), err_hold,range(15), err_test)
plt.legend(["Training", "Validation", "Test"])
plt.xlabel("Polynomial degree")
plt.ylabel("Mean squared error")
```

### Further validation techniques

There is a whole range of more or less evolved validation techniques. For most of these pre-built implementations in scikit learn exist. We refer you to the excellent scikit learn cross validation page for further reading. You can access it [here](https://scikit-learn.org/stable/modules/cross_validation.html). The most important technique to check out is k-fold cross-validation.

---


## Nonnegative Matrix Factorization

```python
%matplotlib inline
%load_ext Cython

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import NMF as NMF_sklearn
```

NMF is a method to factorize a nonnegative matrix $X_{m \times n}$ to nonnegative matrix factors $W_{m \times r}$ and $H_{r \times n}$ such that $X \approx WH$. Here, $r$ is a parameter to be determined and can be called the inner dimension.

### Iterative Algorithm for NMF
An iterative algorithm for NMF as proposed in [[1]](#references) using the multiplicate update rules. This implementation is only for practice; when you need to use NMF, please refer to [sklearn.decomposition.NMF](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html).

```cython
%%cython
import numpy as np
cimport numpy as np
cimport cython

DTYPE = np.float64
ctypedef np.float_t DTYPE_t

@cython.boundscheck(False)
@cython.wraparound(False)
def obj_func(np.ndarray[DTYPE_t, ndim=2] orig, np.ndarray[DTYPE_t, ndim=2] approx):
    """
    Compute the objective function value for NMF given the original
    and approximate matrices.

    Parameters
    ----------
    orig : numpy.array
        Original matrix to factorize.
    approx : numpy.array
        Current approximate matrix.

    Returns
    -------
    obj_value : float
        Current objective function value.
    """
    return np.sum(orig*np.log(approx) - approx)

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.cdivision(True)
def nmf(np.ndarray[DTYPE_t, ndim=2] X, int r):
    """
    Given a nonnegative matrix X of shape m x n, iteratively
    compute a nonnegative matrix factorization X = WH, and
    return the factors W and H.

    The returned factors have shapes (m, r) and (r, n),
    respectively.

    Parameters
    ----------
    X : numpy.array
        m x n nonnegative matrix.
    r : int
        Number of columns of factor W.

    Returns
    -------
    W : numpy.array
        m x r nonnegative matrix.
    H : numpy.array
        r x n nonnegative matrix.
    """
    cdef int m = X.shape[0]
    cdef int n = X.shape[1]
    cdef int prev_index = 0
    cdef int a, i, mu
    cdef DTYPE_t curr_obj, curr_mean
    cdef DTYPE_t prev_mean = 0
    cdef DTYPE_t coeff, col_sum
    cdef np.ndarray[DTYPE_t, ndim=2] W = np.random.rand(m, r)
    cdef np.ndarray[DTYPE_t, ndim=2] H = np.random.rand(r, n)
    cdef np.ndarray[DTYPE_t, ndim=1] prev_objs = np.zeros(10, dtype=DTYPE)
    cdef np.ndarray[DTYPE_t, ndim=2] XA

    while True:
        XA = W@H

        # Convergence check
        curr_obj = obj_func(X, XA)
        if abs(curr_obj - prev_mean) < 1e-6:
            break
        else:
            prev_mean = (10*prev_mean - prev_objs[prev_index] + curr_obj)/10
            prev_objs[prev_index] = curr_obj
            prev_index = (prev_index + 1)%10

        # Update each element of W while fixing H
        for i in range(m):
            for a in range(r):
                coeff = 0
                for mu in range(n):
                    coeff += X[i, mu]/XA[i, mu]*H[a, mu]
                W[i, a] *= coeff

        # Normalize columns of W st they sum to 1
        for a in range(r):
            col_sum = 0
            for i in range(m):
                col_sum += W[i, a]
            for i in range(m):
                W[i, a] /= col_sum

        # Update each element of H while fixing W
        for a in range(r):
            for mu in range(n):
                coeff = 0
                for i in range(m):
                    coeff += W[i, a]*X[i, mu]/XA[i, mu]
                H[a, mu] *= coeff

    return W, H
```

```python
X = np.random.randint(low=0, high=100, size=(10, 200)).astype(np.float)
```

```python
r = 4
W, H = nmf(X, r)

learner = NMF_sklearn(n_components=r, solver='mu', max_iter=1000)
W_sklearn = learner.fit_transform(X)
H_sklearn = learner.components_
```

```python
D = X - W@H
D_sklearn = X - W_sklearn@H_sklearn

print('Divergence of our impl.    : {}'.format(la.norm(D, ord='fro')))
print('Divergence of sklearn impl.: {}'.format(la.norm(D_sklearn, ord='fro')))
```

    Divergence of our impl.    : 976.5349142405947
    Divergence of sklearn impl.: 1009.184869947594

## Bayesian Inference For NMF

### $r = 1$ Case
In this case, we factorize $X$ into row and column vectors $W$ and $H$ whose outer product is the approximation $\tilde{X}$.

$$
\begin{align}
    \begin{pmatrix}
      x_{11} & \dots & x_{1n} \\
      \vdots & \ddots & \vdots \\
      x_{m1} & \dots & x_{mn}
    \end{pmatrix}
    \approx
    \begin{pmatrix}
      w_1 \\
      \vdots \\
      w_m
    \end{pmatrix}
    \begin{pmatrix}
      h_1 & \dots & h_n
    \end{pmatrix}
\end{align}
$$

```python
def rank_one_nmf(X):
    """
    Given a nonnegative matrix X, construct nonnegative vectors
    w and h whose outer product approximates X.
    """
    m, n = X.shape
    w = X[:, 0]
    h = np.ones(n)
    for i in range(1, n):
        w_unit = w/la.norm(w)
        proj = np.dot(X[:, i], w_unit)*w_unit
        h[i] = np.mean(proj/w)
    return w, h
```

```python
w = np.array([5, 3, 7])
h = np.array([0.1, 2, 3, 0.7])
X = np.outer(w, h)
#X = np.random.randint(low=0, high=100, size=(3, 3))
print('Original matrix\n', X)
w_found, h_found = rank_one_nmf(X)
print('Reconstructed matrix\n', np.outer(w_found, h_found))
print('w = ', w_found)
print('h = ', h_found)
```

    Original matrix
     [[  0.5  10.   15.    3.5]
     [  0.3   6.    9.    2.1]
     [  0.7  14.   21.    4.9]]
    Reconstructed matrix
     [[  0.5  10.   15.    3.5]
     [  0.3   6.    9.    2.1]
     [  0.7  14.   21.    4.9]]
    w =  [ 0.5  0.3  0.7]
    h =  [  1.  20.  30.   7.]

## References
1. D. Lee and H. S. Seung, “Learning the parts of objects with nonnegative matrix factorization,” Nature, vol. 401, pp. 788– 791, 1999.


# Derivatives

```python
import numpy as np
from itertools import product
import matplotlib.pylab as plt
import scipy.special as spec
```

# Finite difference dispersion

There is a one to one corespondence between a finite step derivative and the dispersion of the corresponding momentum operator on the lattice.

E.g. for a Laplace finite step derivative of the form

$$
  \frac{\partial^2}{\partial^2x}f(x) \mapsto
  \frac{1}{\epsilon^2}\sum_{n=-N_s}^{N_s} c_n^{N_s} f(\epsilon n_x + \epsilon n) \, ,
  x = \epsilon n_x \, ,
$$
where $\epsilon$ is the lattice spacing, corresponds to the momentum dispersion
$$
    p^2 \mapsto D^{N_s}(p) = - \frac{1}{\epsilon^2}\sum_{n=-N_s}^{N_s} c_n^{N_s} \cos(n p \epsilon) \, .
$$

In general, one wants to identify the coefficients $c_n^{N_s}$ such that $D^{N_s}(p) = p^2 + \mathcal O \left( (\epsilon p)^{2 N_s} \right)$.

If one now expands
$$
    \cos(n p \epsilon) = \sum_{m=0}^\infty \frac{(-)^m}{(2m)!} (n p \epsilon)^{2m}\, ,
$$
one finds that the coefficients $c_n$ are determined by the liner equation
\begin{align}
    v_m &= \sum_{n=0}^{N_s} A_{m n} \gamma_n
    \, , &
    A_{mn} &= \frac{(-)^m}{(2m)!} n^{2m} \, ,
    \\
    v_1 &= 1
    \, , &
    v_m &= 0 \, \forall \, m \neq 1
    \\
    c_0 &= - \gamma_0
    \, , &
    c_{\pm n} &= - \frac{\gamma_n}{2} \, \forall \, n > 0
\end{align}
where $n$ and $m$ run from $0$ to $N_s$.

```python
def derivative_coeffs(Nstep: int):
    """Computes the derivative coefficient for the lapace operator up to step range Nstep.

    The dispersion of the related momentum squared operator is equal to p**2 up to order P**(2*NStep).

    **Arguments**
        Nstep: int
            The number of momentum steps in each direction.
    """
    v = np.zeros(Nstep + 1)
    v[1] = 1

    nn, mm = np.meshgrid(*[np.arange(Nstep + 1)] * 2)
    A = 1 / spec.gamma(2 * mm + 1) * (-1) ** mm * nn ** (2 * mm)

    gamma = np.linalg.solve(A, v)

    coeffs = {}
    for nstep, coeff in enumerate(gamma):
        if nstep == 0:
            coeffs[nstep] = -coeff
        else:
            coeffs[+nstep] = -coeff / 2
            coeffs[-nstep] = -coeff / 2

    return coeffs

```

```python
fig, ax = plt.subplots(figsize=(3, 2), dpi=250)

p = np.linspace(-1, 1, 1000)

for nstep in range(2, 10, 2):
    coeffs = derivative_coeffs(nstep)
    Dp = np.sum([-cn * np.cos(n * p) for n, cn in coeffs.items()], axis=0)
    ax.plot(p, p ** 2 - Dp, label=f"$D^{{({nstep})}}(p)$", lw=1)

ax.legend(fontsize=6, frameon=False)
ax.set_xlabel("$p$")
ax.set_ylabel("$p^2 - D(p)$")
ax.set_yscale("log")
ax.set_ylim(1.0e-12, 1)

plt.show(fig)
```

```python
fig, ax = plt.subplots(figsize=(3, 2), dpi=250)

p = np.linspace(-1, 1, 1000)

for nstep in range(10, 30, 5):
    coeffs = derivative_coeffs(nstep)
    nrange = np.array(list(coeffs.keys()))
    crange = np.array(list(coeffs.values()))
    inds = np.argsort(nrange)
    ax.plot(nrange[inds], np.abs(crange[inds]), label=f"$N_s = {nstep}$", ls="-", marker=".", ms=1, lw=1)

ax.legend(fontsize=6, frameon=False)
ax.set_xlabel("$p$")
ax.set_ylabel("$|c_n|$")
ax.set_yscale("log")

plt.show(fig)
```

```python

```


# Quantum Cosine Classifier

This notebook describes how quantum cosine classifier works in practice.

Classically, cosine classifier is defined as follows:

\begin{align}\label{equation:cosine classifier}
    Pr\Big(y^{(test)} = y_{b}\Big) = \frac{1}{2}+\frac{\Big[d\big(x_{b}, x^{(test)}\big)\Big]^2}{2}
\end{align}

where $(x_{b}, y_{b})$ is a random training example, $x^{(test)}$ the test point and $d(\cdot, \cdot)$ the cosine distance between $x_{b}$ and $x^{(test)}$.

Since the probability of belonging to a class depends on the squared cosine distance between the two vectors, the maximum dissimilarity occurs when training and test observations are orthogonal. In this case, the cosine classifier  assigns a uniform probability distribution in the two classes for $y^{(test)}$. This means that the cosine classifier performs well only if the test point belongs to the same class of the training point.

## Quantum Implementation of the Cosine Classifier

The quantum circuit that implements the cosine classifier encodes data into three different registers: the training vector $x^{(i)}$, the training label $y^{(i)}$ and the test point $x^{(test)}$. One more qubit is used to store the prediction.

The algorithm is made of the following three steps.

### Step 1: State Preparation

Data are encoded into three different quantum registers. Since each set of data ($x$ and $y$) is stored separately, the state preparation routine can be performed independently for each qubit.

$$ \left|\Phi_1\right\rangle = \Big( S_{x_{b}} \otimes S_{x^{(test)}} \otimes S_{y_{b}} \otimes  \mathbb{1}  \Big) \left|0\right\rangle \left|0\right\rangle \left|0\right\rangle \left|0\right\rangle  = \left|x_{b}\right\rangle \left|x^{(test)}\right\rangle \left|y_{b}\right\rangle \left|0\right\rangle
$$

where $S_{(x)}$ is the routine which encodes in the amplitudes of a qubit a $2$-dimensional real vector $x$.

### Step 2: Execution of the swap test

In the second step, the swap-test transforms the amplitudes of the qubit $y^{(test)}$ as a function of the squared cosine distance:

\begin{align}
\left|\Phi_2\right\rangle & =
\big(\mathbb{1} \otimes \mathbb{1} \otimes \mathbb{1} \otimes H\big)
\big(\text{cswap} \otimes \mathbb{1} \otimes C \big)
\big(\mathbb{1} \otimes \mathbb{1} \otimes \mathbb{1} \otimes H\big)
\left|x_{b}\right\rangle \left|x^{(test)}\right\rangle \left|y_{b}\right\rangle \left|0\right\rangle \\
   & = \frac{1}{2}
   \Big(
   \big( \left|x_{b}\right\rangle \left|x^{(test)}\right\rangle + \left|x^{(test)}\right\rangle \left|x_{b}\right\rangle \big)\left|y_{b}\right\rangle  \left|0\right\rangle +
  \big(\left|x_{b}\right\rangle \left|x^{(test)}\right\rangle - \left|x^{(test)}\right\rangle \left|x_{b}\right\rangle \big)\left|y_{b}\right\rangle  \left|1\right\rangle \Big)
\end{align}

where $H$ is the Hadamard gate, cswap is the controlled-swap operation which uses the fourth qubit (position of gate *C*) as control  qubit to swap $\left|x_{b}\right\rangle$ and $\left|x^{(test)}\right\rangle$.
After the execution of the swap test the probability to redout the basis state $\left|0\right\rangle$, that is the probability for the *test* observation to be classified in class $0$ is:

\begin{align}
    P(y^{(test)} = \left|0\right\rangle) =
    \frac{1}{2}+\frac{|\langle x_{b}|x^{(test)} \rangle|^2}{2}
\end{align}

### Step 3: Controlled Pauli-*X* gate

The third step consists in applying a controlled Pauli-$X$ rotation using the control qubit that encodes the label of the training vector. This implies that $y^{(test)}$ is left untouched if  $x_{b}$ belongs to the class $0$. Otherwise,
the amplitudes of the $y^{(test)}$ qubit are exchanged,
and the probability $P(y^{(test)} = 1)$ is higher as the similarity between the two vectors increases.

$$
\left|\Phi_3\right\rangle = \big(\mathbb{1} \otimes \mathbb{1} \otimes \text{C-X} \big) \left|\Phi_2\right\rangle
$$

Thus, the probability amplitudes of the two basis states of $y^{(test)}$ lead to a classification rule that is equivalent to the classic cosine classifier.

# Quantum Implementation

To study the behaviour of the quantum cosine classifier $2000$ random datasets are generated. Each dataset is made  of a training point $(x_{b}, y_{b})$ and a test point $x^{(test)}$.

```python
#Import modules and packages
import sys
sys.path.insert(1, '../')

from Utils import *
from modeling import *
```

```python
n_shots = 1000
n=1000
x = []
x_err = []
P0 = []
P1 = []

for i in np.arange(n):
    '''Random generated dataset'''
    x_train = [random.uniform(-1, 1), random.uniform(0, 1)]
    x_train_norm = normalize_custom(x_train)
    x_test = [random.uniform(-1, 1), random.uniform(0, 1)]
    x_test_norm = normalize_custom(x_test)

    '''Compute cosine distance and append it to x'''
    d_cos = cosine_similarity([x_train], [x_test])[0][0]
    x.append(d_cos)

    '''If x_train belongs to class 0'''
    qc = quantum_cosine_classifier(x_train_norm, x_test_norm, [1,0])
    r = exec_simulator(qc, n_shots)
    P_q = r['0']/n_shots
    P0.append(P_q)
    d_cos_err = np.sqrt(2*P_q-1)
    x_err.append( d_cos_err )

P0 = np.array(P0)
x = np.array(x)
x_err = np.array(x_err)

order = x.argsort()

x = x[order[::-1]]
P0 = P0[order[::-1]]
x_err = x_err[order[::-1]]

err = [ abs(x1 - x2) for (x1, x2) in zip(abs(x), x_err)]
err = np.array(err)
P1 = 1-P0
```

```python
quantum_cos_random_data(x, P0, P1, err)
```

```python
qc.draw(output='mpl', scale=.7)
```


```python
# https://colab.research.google.com/github/kassbohm/tm-snippets/blob/master/ipynb/TM_A/TM_0/trafo_tensor/trafo_tensor_1.ipynb

from sympy.physics.units import *
from sympy import *

"""
    Find normal stress using quantities wrt two different frames:

    1) The (x,y)-frame and
    2) The (xbar, ybar)-frame, with the (xbar, ybar)-placement specified
       by an angle.
    3) phi: This angle, measured from x to xbar, counted positive about the z-axis.

    This program shows, that:

    * The stress tensor is a bilinear form.
    * The stress is found by applying this form to vectors.
    * The result is independent on which frame is used.

"""

# Shortcuts for cos(phi) and sin(phi):
C, S = var("C, S")

sxx, syy, txy = var("sigma_{xx}, sigma_{yy}, tau_{xy}")
vx, vy = var("vx, vy")

T = Matrix([[sxx, txy],[txy, syy]])

prec = 2
phi_deg = 90

sub_list=[
    (C, cos(phi_deg*pi/180)),
    (S, sin(phi_deg*pi/180)),
    (sxx, -1 *Pa),
    (syy,  5 *Pa),
    (txy, 4 *Pa),
    ]

# Rotation matrix:
R = Matrix([[C, S],[-S, C]])

pprint("\nphi in deg:")
tmp = phi_deg
pprint(tmp)

pprint("\n\nComponents wrt (x,y):")
pprint("---------------------")

pprint("\nStress tensor in Pa:")
tmp = T
tmp = tmp.subs(sub_list)
tmp /= Pa
pprint(tmp)

n = Matrix([C,S])
t = Matrix([-S,C])
nt = n.transpose()
tt = t.transpose()

pprint("\nNormal vector n:")
tmp = n
pprint(tmp)
pprint("\nTangent vector t:")
tmp = t
pprint(tmp)
pprint("\nTranspose n' of n:")
tmp = nt
pprint(tmp)
pprint("\nTranspose t' of t:")
tmp = tt
pprint(tmp)

pprint(u"\nσ = n' T n in Pa:")
tmp = nt*(T*n)
tmp = tmp[0]
tmp = tmp.subs(sub_list)
tmp /= Pa
pprint(N(tmp,prec))

pprint("\nτ = t' T n in Pa:")
tmp = tt*T*n
tmp = tmp[0]
tmp = tmp.subs(sub_list)
tmp /= Pa
pprint(N(tmp,prec))

# unicode utf overbar xbar ybar is \u0304:
pprint("\nComponents wrt (x\u0304,y\u0304):")
pprint("---------------------")

n = Matrix([1,0])
nt = n.transpose()
t = Matrix([0,1])
tt = t.transpose()
T = R*T*R.transpose()

pprint("\nStress tensor in Pa:")
tmp = T
tmp = tmp.subs(sub_list)
tmp /= Pa
pprint(tmp)

pprint("\nNormal vector n:")
tmp = n
pprint(tmp)
pprint("\nTranspose n' of n:")
tmp = nt
pprint(tmp)
pprint("\nTangent vector t:")
tmp = t
pprint(tmp)
pprint("\nTranspose t' of t:")
tmp = tt
pprint(tmp)

pprint(u"\nσ = n' T n in Pa:")
tmp = nt*(T*n)
tmp = tmp[0]
tmp = tmp.subs(sub_list)
tmp /= Pa
pprint(N(tmp,prec))

pprint("\nτ = t' T n in Pa:")
tmp = tt*T*n
tmp = tmp[0]
tmp = tmp.subs(sub_list)
tmp /= Pa
pprint(N(tmp,prec))

```


## Using curve fitting, we managed to translate the available data for the uncontroled reentry of Tiangon-1 into a mathematical expression.

See https://www.desmos.com/calculator/15mcnjggsr.

```python
from sympy import *
init_printing()

x, y, r, theta, t, a, E = symbols("x y r theta t a E", Variables = True)
A_l, D_l, S_l, A_s, F_s, phi, D_y = symbols('A_l D_l S_l A_s F_s phi D_y', Variables=True)

mu = symbols("mu", Contant = True) # The gravitational parameter
a = (A_l*ln(D_l - S_l*t) + A_s*sin(F_s*t + phi) + D_y) # Km ..... The semi-major axis (apogee of Tiangong-1)
E = sqrt(mu / a**3) * t # The eccentric anomally
```

Apogee of Tiangong-1:

```python
Eq(symbols('a'), a)
```

Where A_l, D_l, S_l, A_s, F_s, phi, D_y are just parameters the minipulate the shape of the curve.

For simpicity, we'll use a circular orbit.

```python
theta = E
r = a

x = r*cos(theta)
y = r*sin(theta)
```

The following x and y are the position of the satellite, specifically Tiangong-1, duo to the Earth's gravity and the atmospheric drag.

```python
Eq(symbols('x'), x)
```

```python
Eq(symbols('y'), y)
```

Now, using the values of the previously mentioned parameters, which can be found in https://www.desmos.com/calculator/15mcnjggsr, we get:

```python
expr_a = a.subs({A_l:38.4,
                D_l:122000,
                S_l:2.5,
                A_s:5.1,
                F_s:0.001,
                phi: 2.7,
                D_y: 6330})
Eq(symbols("a"), expr_a)
```

In order to be able to plug time in seconds, we multiply t by (1/(60\*60\*24\*30\*12))\*(48797):

```python
expr_a_sec = expr_a.subs({t:t*(1/(60*60*24*30*12))*(48797)})
Eq(symbols('a_sec'), expr_a_sec)
```

Plotting 'a' with these values:

```python
plot(expr_a_sec,
     (t, 0, 3.2*10**7),
     xlim=(0, 3.5*10**7),
     ylim=(6400, 6800),
     xlabel="\ntime in seconds",
     ylabel="apogee in meters")
```

We can now get x and y with everything substituted:

```python
Eq(symbols('x'), x.subs({a:expr_a_sec, mu:398600.4405}))
```

```python
Eq(symbols('y'), y.subs({a:expr_a_sec, mu:398600.4405}))
```

Now we can simulate Tiangong-1's uncontrolled reentry. See https://www.desmos.com/calculator/yix3q9nyyc

```python

```

Python Notebooks/Atmospheric Drag (Simulation of Tiangong-1's reentry).ipynb


# Naive Bayes classifiers

A naive Bayes classifier uses the Bayes theorem to classify data. It is frequently used to filter out SPAM documents. To classify a document as SPAM/NONSPAM (HAM) we want

\begin{equation}
    p(\text{SPAM}|\text{TEXT})
\end{equation}

using the Bayes theorem we get

\begin{equation}
    p(\text{SPAM}|\text{TEXT}) =
    \frac{p(\text{TEXT}|\text{SPAM})p(\text{SPAM})}{p(\text{TEXT})}
\end{equation}

The Bayes theorem allows us to express a classification problem as a generation problem: we will create a model for generating texts ($p(\text{TEXT}|\text{SPAM})$) and combine it with the prior probability of getting a spam ($p(\text{SPAM})$). However, we will not need  $p(\text{TEXT})$: the probability of ever seeing a given document.

To estimate $p(\text{TEXT}|\text{SPAM})$ we need to define a data generation model. A text is a sequence of words:
$$
\text{TEXT} = W_1, W_2, W_3,\ldots,W_n.
$$
Thus,
$$
p(\text{TEXT}|\text{SPAM}) = p(W_1|\text{SPAM})p(W_2|W_1,\text{SPAM})p(W_n|W_1, ..., W_{n-1}, \text{SPAM})
$$
We will further simply this by (naively) assuming that

\begin{equation}
    \begin{split}
 p(\text{TEXT}|\text{SPAM}) &=  (W_1|\text{SPAM})p(W_2|W_1,\text{SPAM})p(W_n|W_1, ..., W_{n-1}, \text{SPAM}) \\
 &\approx p(W_1|\text{SPAM})p(W_2|\text{SPAM})p(W_n|\text{SPAM}) \\
 &= \prod_{W_i \in \text{TEXT}}p(W_i|\text{SPAM})
\end{split}
\end{equation}

This corresponds to a generative models in which the sender first flips a biased coin to see if the generated document will be a spam or ham one. Then, he picks a box labeled spam or ham. Finally, the sender draws with replacement words from the appropriate box.

The full sampling model has the following parameters:
1. $\phi$ - the probability of generating a SPAM.
2. $\theta_{w,s}$ - the probability of generating word $w$ in a SPAM document, $\sum_w \theta_{w,s}=1$,
3. $\theta_{w,h}$ - the probability of generating word $w$ in a HAM document, $\sum_w \theta_{w,h}=1$.

All parameters are easy to estimate using maximum likelihood principle:
1. $\phi = p(\text{SPAM}=s)$ is just the fraction of all spams in our corpus.
2. $\theta_{w,s} = p(W=w|SPAM=s)$ is the fraction of the number of occurrences of word $w$ in all spams.
3. $\theta_{w,h} = p(W=w|SPAM=h)$ is the fraction of the number of occurrences of word $w$ in all non-spams.

The derivation is somewhat tedious, and requires the use of Langrange multipliers:

Example:

suppose our corpus has 4 documents:
1. "buy much now": SPAM
2. "many dollars gain": SPAM
3. "like you much": HAM
4. "do your nice homework": HAM

Then:
$\phi = p(\text{SPAM}=s) = 2/4 = 0.5$

$\theta_{w,h}$ is given by the following table

|       | buy | much | now | dollars | gain | like | you/your | do  | homework | nice |
|------|-----|------|-----|---------|------|------|----------|-----|----------|------|
| SPAM | 1/6 | 2/6  | 1/6 | 1/6     | 1/6  | 0/6  | 0/6      | 0/6 | 0/6      | 0/6  |
| HAM  | 0/7 | 1/7  | 0/7 | 0/7     | 0/7  | 1/7  | 2/7      | 1/7 | 1/7      | 1/7  |

To classify a new phrase "much much gain" we compute

$$
\begin{split}
p(\text{SPAM} = s | \text{"much much gain"}) &= p(\text{SPAM}=s) p(\text{much}|\text{SPAM}=s)p(\text{much}|\text{SPAM}=s)p(\text{gain}|\text{SPAM}=s) / p(\text{TEXT} = \text{"much much gain"}) = \\
&= 1/2 \cdot 2/6 \cdot 2/6  \cdot 1/6  \cdot 1/Z = 4/216 \cdot 1/Z
\end{split}
$$

$$
\begin{split}
p(\text{HAM} = s | \text{"much much gain"}) &= p(\text{HAM}=s) p(\text{much}|\text{HAM}=s)p(\text{much}|\text{HAM}=s)p(\text{gain}|\text{HAM}=s) / p(\text{TEXT} = \text{"much much gain"}) = \\
&= 1/2 \cdot 1/7 \cdot 1/7  \cdot 0/6  \cdot 1/Z = 0
\end{split}
$$

Thus the text is classified as SPAM.

 In fact all text with word "gain" will never be classified as non-spam, because "gain" never appeared in a non-spam document. This is a problem with modeling: for rare words, we are using maximum likelihood estimation to compute the frequencies. However MLE estimation doesn't work well with low data counts.

 Inspired by the Bayesian approach to polling (estimating counts) a common technique is called Laplace smoothing - assuming that each word in the vocabulary was seen at least once (or even a fraction of times) in each kind of document.

 With Laplace smoothing (assuming each word occurred 0.5 times in spam and 0.5 times in ham) the table becomes

|       | buy | much | now | dollars | gain | like | you/your | do  | homework | nice |
|------|-----|------|-----|---------|------|------|----------|-----|----------|------|
| SPAM | 1.5/11 | 2.5/11  | 1.5/11 | 1.5/11     | 1.5/11  | 0.5/11  | 0.5/11      | 0.5/11 | 0.5/11      | 0.5/11  |
| HAM  | 0.5/12 | 1.5/12  | 0.5/12 | 0.5/12     | 0.5/12  | 1.5/12  | 2.5/12      | 1.5/12 | 1.5/12      | 1.5/12  |

Now:

$$
\begin{split}
p(\text{SPAM} = s | \text{"much much gain"}) &= 1/2 \cdot 2.5/11 \cdot 2.5/11  \cdot 1.5/11  \cdot 1/Z = 9.375/2662 \cdot 1/Z \\
p(\text{SPAM} = h | \text{"much much gain"}) &= 1/2 \cdot 1.5/12 \cdot 1.5/12  \cdot 0.5/12  \cdot 1/Z = 1.125/3456 \cdot 1/Z
\end{split}
$$

Since $p(\text{SPAM} = s | \text{"much much gain"})  + p(\text{SPAM} = h | \text{"much much gain"})  = 1$ we can work out that $Z=0.00385$ and
$$
\begin{split}
p(\text{SPAM} = s | \text{"much much gain"}) &= 9.375/2662 \cdot 1/Z  = 0.915 = 91.5\%\\
p(\text{SPAM} = h | \text{"much much gain"}) &= 1.125/3456 \cdot 1/Z = 0.085 = 8.5\%
\end{split}
$$

Thus the modelpredicts that with $91.5\%$ the new text is SPAM.

```

```


# Cinemática III

Hasta el momento hemos usado solamente numeros para representar las transformaciones que describen la cinematica de los manipuladores, pero dentro del ambiente de IPython tenemos una libreria que nos va a ayudar mucho cuando queremos calcular una formula general.

## Sympy

La libreria de sympy nos ofrece herramientas de calculo analitico y numerico para analisis simbolico (por lo general nos referimos a el simplemente como calculo simbolico). Lo primero que tenemos que hacer es inicializar las funciones de sympy que vamos a necesitar.

```
# Importa la función para imprimir LaTeX en el notebook de IPython
from sympy.interactive import printing
printing.init_printing()
# Importamos funciones utiles para algebra
from sympy import var, simplify, collect, expand, solve, sin, cos, Matrix
```

Y ahora podemos inicializar las variables que vamos a ocupar, esto lo hacemos asi:

```
var("x y z")
```

Podemos representar cualquier tipo de expresión que queramos:

```
x**2 + sin(y) + 3*x + 1
```

Incluso resolver algunas formulas con algebra o trigonometria:

```
solve(x**2 + 2*x + 1, x)
```

```
solve(sin(y) + cos(y), y)
```

Podemos guardar estas expresiones en variables, y estas variables van a tener metodos especiales, como por ejemplo la derivada.

```
f = sin(x) + x
```

```
f.diff(x)
```

Y tambien podemos representar matrices.

```
Rz = Matrix([[cos(x), -sin(x), 0, 0],
             [sin(x),  cos(x), 0, 0],
             [     0,       0, 1, 0],
             [     0,       0, 0, 1]])
Rz
```

y usarla para operaciones:

```
Rz*Rz
```

Nuestras funciones de Python, tambien nos pueden regresar objetos de sympy:

```
def rotacionz(theta):
    '''
    Esta función devuelve una matriz de rotación con respecto al eje z
    con una magnitud de theta.

    Uso:

    >>> rotacionz(y)
    Matrix([
    [cos(y), -sin(y), 0, 0],
    [sin(y),  cos(y), 0, 0],
    [     0,       0, 1, 0],
    [     0,       0, 0, 1]])
    '''
    Rz = Matrix([[cos(theta), -sin(theta), 0, 0],
                 [sin(theta),  cos(theta), 0, 0],
                 [         0,           0, 1, 0],
                 [         0,           0, 0, 1]])
    return Rz
```

```
var("q_1")
rotacionz(q_1)
```

## Ejercicios

1. Implemente el codigo necesario para tener funciones que describan las rotaciones en $x$ y en $y$ y las traslaciones en $x$, $y$ y $z$.
2. Implemente una función que nos devuelva la transformación homogenea de un manipulador PUMA.

Practicas/Practica 3 - Cinematica III.ipynb


# Homework 2
*This notebook includes both coding and written questions. Please hand in this notebook file with all the outputs and your answers to the written questions.*

This assignment covers Canny edge detector and Hough transform.

```python
# Setup
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from time import time
from skimage import io

from __future__ import print_function

%matplotlib inline
plt.rcParams['figure.figsize'] = (15.0, 12.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading extenrnal modules
%load_ext autoreload
%autoreload 2
```

## Part 1: Canny Edge Detector (85 points)
In this part, you are going to implement a Canny edge detector. The Canny edge detection algorithm can be broken down in to five steps:
1. Smoothing
2. Finding gradients
3. Non-maximum suppression
4. Double thresholding
5. Edge tracking by hysterisis

### 1.1 Smoothing (10 points)
#### Implementation (5 points)
We first smooth the input image by convolving it with a Gaussian kernel. The equation for a Gaussian kernel of size $(2k+1)\times(2k+1)$ is given by:

$$h_{ij}=\frac{1}{2\pi\sigma^2}\exp{\Bigl(-\frac{(i-k)^2+(j-k)^2}{2\sigma^2}\Bigr)}, 0\leq i,j < 2k+1$$

Implement **`gaussian_kernel`** in `edge.py` and run the code below.

```python
from edge import conv, gaussian_kernel

# Define 3x3 Gaussian kernel with std = 1
kernel = gaussian_kernel(3, 1)
kernel_test = np.array(
    [[ 0.05854983, 0.09653235, 0.05854983],
     [ 0.09653235, 0.15915494, 0.09653235],
     [ 0.05854983, 0.09653235, 0.05854983]]
)

# Test Gaussian kernel
if not np.allclose(kernel, kernel_test):
    print('Incorrect values! Please check your implementation.')
```

Implement **`conv`** in `edge.py` and run the code below.

```python
# Test with different kernel_size and sigma
kernel_size = 5
sigma = 1.4

# Load image
img = io.imread('iguana.png', as_grey=True)

# Define 5x5 Gaussian kernel with std = sigma
kernel = gaussian_kernel(kernel_size, sigma)

# Convolve image with kernel to achieve smoothed effect
smoothed = conv(img, kernel)

plt.subplot(1,2,1)
plt.imshow(img)
plt.title('Original image')
plt.axis('off')

plt.subplot(1,2,2)
plt.imshow(smoothed)
plt.title('Smoothed image')
plt.axis('off')

plt.show()
```

#### Question (5 points)
What is the effect of changing kernel_size and sigma?

**Your Answer:** Write your solution in this markdown cell.
A larger kernel size makes the image more blurry (bigger filter considers more neighbours) but takes longer to compute. A larger sigma makes the image more blurry since the weights in the kernel vary less and the kernel becomes more of a moving average.

### 1.2 Finding gradients (15 points)
The gradient of a 2D scalar function $I:\mathbb{R}^2\rightarrow{\mathbb{R}}$ in Cartesian coordinate is defined by:

$$\nabla{I(x,y)}=\bigl[\frac{\partial{I}}{\partial{x}},\frac{\partial{I}}{\partial{y}}\bigr],$$

where

$$
\frac{\partial{I(x,y)}}{\partial{x}}=\lim_{\Delta{x}\to{0}}\frac{I(x+\Delta{x},y)-I(x,y)}{\Delta{x}} \\
\frac{\partial{I(x,y)}}{\partial{y}}=\lim_{\Delta{y}\to{0}}\frac{I(x,y+\Delta{y})-I(x,y)}{\Delta{y}}.
$$

In case of images, we can approximate the partial derivatives by taking differences at one pixel intervals:

$$
\frac{\partial{I(x,y)}}{\partial{x}}\approx{\frac{I(x+1,y)-I(x-1,y)}{2}} \\
\frac{\partial{I(x,y)}}{\partial{y}}\approx{\frac{I(x,y+1)-I(x,y-1)}{2}}
$$

Note that the partial derivatives can be computed by convolving the image $I$ with some appropriate kernels $D_x$ and $D_y$:

$$
\frac{\partial{I}}{\partial{x}}\approx{I*D_x}=G_x \\
\frac{\partial{I}}{\partial{y}}\approx{I*D_y}=G_y
$$

#### Implementation (5 points)
Find the kernels $D_x$ and $D_y$ and implement **`partial_x`** and **`partial_y`** using `conv` defined in `edge.py`.

*-Hint: Remeber that convolution flips the kernel.*

```python
from edge import partial_x, partial_y

# Test input
I = np.array(
    [[0, 0, 0],
     [0, 1, 0],
     [0, 0, 0]]
)

# Expected outputs
I_x_test = np.array(
    [[ 0, 0, 0],
     [ 0.5, 0, -0.5],
     [ 0, 0, 0]]
)

I_y_test = np.array(
    [[ 0, 0.5, 0],
     [ 0, 0, 0],
     [ 0, -0.5, 0]]
)

# Compute partial derivatives
I_x = partial_x(I)
I_y = partial_y(I)

# Test correctness of partial_x and partial_y
if not np.all(I_x == I_x_test):
    print('partial_x incorrect')

if not np.all(I_y == I_y_test):
    print('partial_y incorrect')
```

```python
# Compute partial derivatives of smoothed image
Gx = partial_x(smoothed)
Gy = partial_y(smoothed)

plt.subplot(1,2,1)
plt.imshow(Gx)
plt.title('Derivative in x direction')
plt.axis('off')

plt.subplot(1,2,2)
plt.imshow(Gy)
plt.title('Derivative in y direction')
plt.axis('off')

plt.show()
```

#### Question (5 points)
What is the reason for performing smoothing prior to computing the gradients?

**Your Answer:** Write your solution in this markdown cell.

Gradients respond very strongly to noise, so doing denoising with smoothing first helps achieve better results.

#### Implementation (5 points)
Now, we can compute the magnitude and direction of gradient with the two partial derivatives:

$$
G = \sqrt{G_{x}^{2}+G_{y}^{2}} \\
\Theta = arctan\bigl(\frac{G_{y}}{G_{x}}\bigr)
$$

Implement **`gradient`** in `edge.py` which takes in an image and outputs $G$ and $\Theta$.

```python
from edge import gradient

G, theta = gradient(smoothed)

if not np.all(G >= 0):
    print('Magnitude of gradients should be non-negative.')

if not np.all((theta >= 0) * (theta < 360)):
    print('Direction of gradients should be in range 0 <= theta < 360')

plt.imshow(G)
plt.title('Gradient magnitude')
plt.axis('off')
plt.show()
```

### 1.3 Non-maximum suppression (15 points)
You should be able to see that the edges extracted from the gradient of the smoothed image are quite thick and blurry. The purpose of this step is to convert the "blurred" edges into "sharp" edges. Basically, this is done by preserving all local maxima in the gradient image and discarding everything else. The algorithm is for each pixel (x,y) in the gradient image:
1. Round the gradient direction $\Theta[y,x]$ to the nearest 45 degrees, corresponding to the use of an 8-connected neighbourhood.

2. Compare the edge strength of the current pixel with the edge strength of the pixel in the positive and negative gradient directions. For example, if the gradient direction is south (theta=90), compare with the pixels to the north and south.

3. If the edge strength of the current pixel is the largest; preserve the value of the edge strength. If not, suppress (i.e. remove) the value.

Implement **`non_maximum_suppression`** in `edge.py`.

We provide the correct output and the difference between it and your result for debugging purposes.  If you see white spots in the Difference image, you should check your implementation.

```python
from edge import non_maximum_suppression

# Test input
g = np.array(
    [[0.4, 0.5, 0.6],
     [0.3, 0.5, 0.7],
     [0.4, 0.5, 0.6]]
)

# Print out non-maximum suppressed output
# varying theta
for angle in range(0, 180, 45):
    print('Thetas:', angle)
    t = np.ones((3, 3)) * angle # Initialize theta
    print(non_maximum_suppression(g, t))
```

    Thetas: 0
    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]
    Thetas: 45
    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]
    Thetas: 90
    [[0.  0.  0. ]
     [0.  0.5 0. ]
     [0.  0.  0. ]]
    Thetas: 135
    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]

```python
nms = non_maximum_suppression(G, theta)
plt.imshow(nms)
plt.title('Non-maximum suppressed')
plt.axis('off')
plt.show()

plt.subplot(1, 3, 1)
plt.imshow(nms)
plt.axis('off')
plt.title('Your result')

plt.subplot(1, 3, 2)
reference = np.load('references/iguana_non_max_suppressed.npy')
plt.imshow(reference)
plt.axis('off')
plt.title('Reference')

plt.subplot(1, 3, 3)
plt.imshow(nms - reference)
plt.title('Difference')
plt.axis('off')
plt.show()
```

### 1.4 Double Thresholding (20 points)

The edge-pixels remaining after the non-maximum suppression step are (still) marked with their strength pixel-by-pixel. Many of these will probably be true edges in the image, but some may be caused by noise or color variations, for instance, due to rough surfaces. The simplest way to discern between these would be to use a threshold, so that only edges stronger that a certain value would be preserved. The Canny edge detection algorithm uses double thresholding. Edge pixels stronger than the high threshold are marked as strong; edge pixels weaker than the low threshold are suppressed and edge pixels between the two thresholds are marked as weak.

Implement **`double_thresholding`** in `edge.py`

```python
from edge import double_thresholding

low_threshold = 0.02
high_threshold = 0.03

strong_edges, weak_edges = double_thresholding(nms, high_threshold, low_threshold)
assert(np.sum(strong_edges & weak_edges) == 0)

edges=strong_edges * 1.0 + weak_edges * 0.5

plt.subplot(1,2,1)
plt.imshow(strong_edges)
plt.title('Strong Edges')
plt.axis('off')

plt.subplot(1,2,2)
plt.imshow(edges)
plt.title('Strong+Weak Edges')
plt.axis('off')

plt.show()
```

### 1.5 Edge tracking (15 points)

Strong edges are interpreted as “certain edges”, and can immediately be included in the final edge image. Weak edges are included if and only if they are connected to strong edges. The logic is of course that noise and other small variations are unlikely to result in a strong edge (with proper adjustment of the threshold levels). Thus strong edges will (almost) only be due to true edges in the original image. The weak edges can either be due to true edges or noise/color variations. The latter type will probably be distributed independently of edges on the entire image, and thus only a small amount will be located adjacent to strong edges. Weak edges due to true edges are much more likely to be connected directly to strong edges.

Implement **`link_edges`** in `edge.py`.

We provide the correct output and the difference between it and your result for debugging purposes.  If you see white spots in the Difference image, you should check your implementation.

```python
from edge import get_neighbors, link_edges

test_strong = np.array(
    [[1, 0, 0, 0],
     [0, 0, 0, 0],
     [0, 0, 0, 0],
     [0, 0, 0, 1]],
    dtype=np.bool
)

test_weak = np.array(
    [[0, 0, 0, 1],
     [0, 1, 0, 0],
     [1, 0, 0, 0],
     [0, 0, 1, 0]],
    dtype=np.bool
)

test_linked = link_edges(test_strong, test_weak)

plt.subplot(1, 3, 1)
plt.imshow(test_strong)
plt.title('Strong edges')

plt.subplot(1, 3, 2)
plt.imshow(test_weak)
plt.title('Weak edges')

plt.subplot(1, 3, 3)
plt.imshow(test_linked)
plt.title('Linked edges')
plt.show()
```

```python
edges = link_edges(strong_edges, weak_edges)

plt.imshow(edges)
plt.axis('off')
plt.show()

plt.subplot(1, 3, 1)
plt.imshow(edges)
plt.axis('off')
plt.title('Your result')

plt.subplot(1, 3, 2)
reference = np.load('references/iguana_edge_tracking.npy')
plt.imshow(reference)
plt.axis('off')
plt.title('Reference')

plt.subplot(1, 3, 3)
plt.imshow(edges ^ reference)
plt.title('Difference')
plt.axis('off')
plt.show()
```

### 1.6 Canny edge detector
Implement **`canny`** in `edge.py` using the functions you have implemented so far. Test edge detector with different parameters.

Here is an example of the output:

We provide the correct output and the difference between it and your result for debugging purposes.  If you see white spots in the Difference image, you should check your implementation.

```python
from edge import canny

# Load image
img = io.imread('iguana.png', as_grey=True)

# Run Canny edge detector
edges = canny(img, kernel_size=5, sigma=1.4, high=0.03, low=0.02)
print (edges.shape)

plt.subplot(1, 3, 1)
plt.imshow(edges)
plt.axis('off')
plt.title('Your result')

plt.subplot(1, 3, 2)
reference = np.load('references/iguana_canny.npy')
plt.imshow(reference)
plt.axis('off')
plt.title('Reference')

plt.subplot(1, 3, 3)
plt.imshow(edges ^ reference)
plt.title('Difference')
plt.axis('off')
plt.show()
```

### 1.7 Question (10 points)

**(a)** Suppose that the Canny edge detector successfully detects an edge in an image. The edge (see the figure above) is then rotated by θ, where the relationship between a point on the original edge $(x, y)$ and a point on the rotated edge $(x', y')$ is defined as

$$
x'=x\cos{\theta}\\
y'=x\sin{\theta}
$$

Will the rotated edge be detected using the same Canny edge detector? Provide either a mathematical proof or a counter example.

*-Hint 1: The detection of an edge by the Canny edge detector depends only on the magnitude of its derivative. The derivative at point (x, y) is determined by its components along the x and y directions. Think about how these magnitudes have changed because of the rotation.*

*-Hint 2: You can assume that (x,y) lies on the x-axis, i.e., y = 0. *

*-Hint 3: You can also assume that G_x(x, y) = 0. In other words, the gradient which is perpendicular to the direction of the unrotated edge at (x, y) only has a vertical component and thus only consists of G_y(x, y).*

**Your Answer:** Write your solution in this markdown cell.

We know that $y=0$ and $G_x(x,y)=0$. So the magnitude of the old gradient is

\begin{align}
    G = \sqrt{G_x^2 + G_y^2} = |G_y|
\end{align}

Consider the new gradients ${G'}_x = G_x(x',y')$ and ${G'}_y = G_y(x',y')$

Using change of variables and the geometry of the gradient vectors, we see that

\begin{align}
    {G'}_x &= -\sin{\theta} \cdot G_y \\
    {G'}_y &= \cos{\theta} \cdot G_y
\end{align}

Since detection of an edge only depends on the magnitude of the derivative, let's calculate the magnitude of the new gradient $G'$ in terms of $G$

\begin{align}
    G' = \sqrt{{G'}_x^2 + {G'}_y^2} = \sqrt{\cos^2\theta \cdot G_y^2 + \sin^2\theta \cdot G_y^2} = |G_y |
\end{align}

Therefore the rotated edge will also be detected since its gradient has the same magnitude.

**(b)** After running the Canny edge detector on an image, you notice that long edges are broken into short segments separated by gaps. In addition, some spurious edges appear. For each of the two thresholds (low and high) used in hysteresis thresholding, explain how you would adjust the threshold (up or down) to address both problems. Assume that a setting exists for the two thresholds that produces the desired result. Briefly explain your answer.

**Your Answer:** Write your solution in this markdown cell.

The short segments separagted by gaps mean that some pixels in the edge are below the lower threshold. Thus decreasing the lower threshold will make these into weak edges and since they will connected to strong pixels, they will be recognized as edge pixels.

If spurious edges appear, then that means some noise is being identified as strong pixels, so we should increase the higher threshold.

### Extra Credit: Optimizing Edge Detector
One way of evaluating an edge detector is to compare detected edges with manually specified ground truth edges. Here, we use precision, recall and F1 score as evaluation metrics. We provide you 40 images of objects with ground truth edge annotations. Run the code below to compute precision, recall and F1 score over the entire set of images. Then, tweak the parameters of the Canny edge detector to get as high F1 score as possible. You should be able to achieve F1 score higher than 0.31 by carefully setting the parameters.

```python
from os import listdir
from itertools import product

# Define parameters to test
sigmas = [1]
highs = [20]
lows = [5]

for sigma, high, low in product(sigmas, highs, lows):

    print("sigma={}, high={}, low={}".format(sigma, high, low))
    n_detected = 0.0
    n_gt = 0.0
    n_correct = 0.0

    for img_file in listdir('images/objects'):
        img = io.imread('images/objects/'+img_file, as_grey=True)
        gt = io.imread('images/gt/'+img_file+'.gtf.pgm', as_grey=True)

        mask = (gt != 5) # 'don't' care region
        gt = (gt == 0) # binary image of GT edges

        edges = canny(img, kernel_size=5, sigma=sigma, high=high, low=low)
        edges = edges * mask

        n_detected += np.sum(edges)
        n_gt += np.sum(gt)
        n_correct += np.sum(edges * gt)

    p_total = n_correct / n_detected
    r_total = n_correct / n_gt
    f1 = 2 * (p_total * r_total) / (p_total + r_total)
    print('Total precision={:.4f}, Total recall={:.4f}'.format(p_total, r_total))
    print('F1 score={:.4f}'.format(f1))
```

## Part2: Lane Detection (15 points)

In this section we will implement a simple lane detection application using Canny edge detector and Hough transform.
Here are some example images of how your final lane detector will look like.

The algorithm can broken down into the following steps:
1. Detect edges using the Canny edge detector.
2. Extract the edges in the region of interest (a triangle covering the bottom corners and the center of the image).
3. Run Hough transform to detect lanes.

### 2.1 Edge detection
Lanes on the roads are usually thin and long lines with bright colors. Our edge detection algorithm by itself should be able to find the lanes pretty well. Run the code cell below to load the example image and detect edges from the image.

```python
from edge import canny

# Load image
img = io.imread('road.jpg', as_grey=True)

# Run Canny edge detector
edges = canny(img, kernel_size=5, sigma=1.4, high=0.03, low=0.02)

plt.subplot(211)
plt.imshow(img)
plt.axis('off')
plt.title('Input Image')

plt.subplot(212)
plt.imshow(edges)
plt.axis('off')
plt.title('Edges')
plt.show()
```

### 2.2 Extracting region of interest (ROI)
We can see that the Canny edge detector could find the edges of the lanes. However, we can also see that there are edges of other objects that we are not interested in. Given the position and orientation of the camera, we know that the lanes will be located in the lower half of the image. The code below defines a binary mask for the ROI and extract the edges within the region.

```python
H, W = img.shape

# Generate mask for ROI (Region of Interest)
mask = np.zeros((H, W))
for i in range(H):
    for j in range(W):
        if i > (H / W) * j and i > -(H / W) * j + H:
            mask[i, j] = 1

# Extract edges in ROI
roi = edges * mask

plt.subplot(1,2,1)
plt.imshow(mask)
plt.title('Mask')
plt.axis('off')

plt.subplot(1,2,2)
plt.imshow(roi)
plt.title('Edges in ROI')
plt.axis('off')
plt.show()
```

### 2.3 Fitting lines using Hough transform (15 points)
The output from the edge detector is still a collection of connected points. However, it would be more natural to represent a lane as a line parameterized as $y = ax + b$, with a slope $a$ and y-intercept $b$. We will use Hough transform to find parameterized lines that represent the detected edges.

In general, a straight line $y = ax + b$ can be represented as a point $(a, b)$ in the parameter space. However, this cannot represent vertical lines as the slope parameter will be unbounded. Alternatively, we parameterize a line using $\theta\in{[-\pi, \pi]}$ and $\rho\in{\mathbb{R}}$ as follows:

$$
\rho = x\cdot{cos\theta} + y\cdot{sin\theta}
$$

Using this parameterization, we can map every point in $xy$-space to a sine-like line in $\theta\rho$-space (or Hough space). We then accumulate the parameterized points in the Hough space and choose points (in Hough space) with highest accumulated values. A point in Hough space then can be transformed back into a line in $xy$-space.

*See [notes](http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/HoughTrans_lines_09.pdf) on Hough transform.*

Implement **`hough_transform`** in `edge.py`.

```python
from edge import hough_transform

# Perform Hough transform on the ROI
acc, rhos, thetas = hough_transform(roi)

# Coordinates for right lane
xs_right = []
ys_right = []

# Coordinates for left lane
xs_left = []
ys_left = []

for i in range(20):
    idx = np.argmax(acc)
    r_idx = idx // acc.shape[1]
    t_idx = idx % acc.shape[1]
    acc[r_idx, t_idx] = 0 # Zero out the max value in accumulator

    rho = rhos[r_idx]
    theta = thetas[t_idx]

    # Transform a point in Hough space to a line in xy-space.
    a = - (np.cos(theta)/np.sin(theta)) # slope of the line
    b = (rho/np.sin(theta)) # y-intersect of the line

    # Break if both right and left lanes are detected
    if xs_right and xs_left:
        break

    if a < 0: # Left lane
        if xs_left:
            continue
        xs = xs_left
        ys = ys_left
    else: # Right Lane
        if xs_right:
            continue
        xs = xs_right
        ys = ys_right

    for x in range(img.shape[1]):
        y = a * x + b
        if y > img.shape[0] * 0.6 and y < img.shape[0]:
            xs.append(x)
            ys.append(int(round(y)))

plt.imshow(img)
plt.plot(xs_left, ys_left, linewidth=5.0)
plt.plot(xs_right, ys_right, linewidth=5.0)
plt.axis('off')
plt.show()
```