# Assignment #1 - Linear Regression

 Abdullah Al Raqibul Islam

# INDEX

I. Introduction

II. Data

  II.I. Reading the data

  II.II. Preprocessing of the data

  II.III. Visualization of the data

    II.III.I. Correlation Heatmap

    II.III.II. Pie-chart

    II.III.III. Count Plots

    II.III.IV. Line Plots

    II.III.V. Linear Regression Plots

    II.III.VI. Box Plots

    II.III.VII. Preliminary Observation About Data

III. Method

  III.I. Super Classs Definition

  III.II. Least Squares

  III.III. Least Mean Squares

IV. Usage Examples

  IV.I. Usage Example: Least Squares

  IV.II. Usage Example: Least Mean Squares

V. Preliminary Test

VI. Experiments

  VI.I. Plotting Functions

  VI.II. Data Partitioning

  **VI.III. Experiments on Least Squares**

    VI.III.I. Feature Scaling: Unscaled Data

      VI.III.I.I. Feature Selection: All Features

      VI.III.I.II. Feature Selection: Significant Features

    VI.III.II. Feature Scaling: Normalizaed Data

      VI.III.II.I. Feature Selection: All Features

      VI.III.II.II. Feature Selection: Significant Features

    VI.III.III. Feature Scaling: Standarized Data

      VI.III.III.I. Feature Selection: All Features

      VI.III.III.II. Feature Selection: Significant Features

  **VI.IV. Experiments on Least Mean Squares (LMS)**

    VI.IV.I. Feature Scaling: Unscaled Data

      VI.IV.I.I. Feature Selection: All Features

      VI.IV.I.II. Feature Selection: Significant Features

    VI.IV.II. Feature Scaling: Normalized Data

      VI.IV.II.I. Feature Selection: All Features

      VI.IV.II.II. Feature Selection: Significant Features

    VI.IV.III. Feature Scaling: Standarized Data

      VI.IV.III.I. Feature Selection: All Features

      VI.IV.III.II. Feature Selection: Significant Features

VII. Conclusions

VIII. Extra Credit

IX. References

# I. Introduction

The goal of this assignment is to perform linear regression on a dataset to predict the price of house sale [[1]](https://www.kaggle.com/harlfoxem/housesalesprediction) based on the different features (i.e. size, condition, facility).

The features have been explained in the data section. I have implemented and used two linear regression models namely `Least Squares` and `Least Mean Squares`. The detail of the algorithm discussed in `section III`. Section `IV` and `V` contains the integraty testing of the implemented algorithms. In `section VI`, I discussed the performance evaluations in details of the implementation domain. This section also contains the extra credit works listed in `section VIII`.

# II. Data

In this expriment I used `House Sales in King County, USA` [[1]](https://www.kaggle.com/harlfoxem/housesalesprediction) dataset from `kaggle` [[2]](https://www.kaggle.com). This dataset contains house sale prices for King County, which includes Seattle. It includes homes sold between May 2014 and May 2015. The dataset have 21613 rows and 21 attributes. Here is the attribute list along with the data-type.

**Attribute Information:**
 0.   id (int64)
 1.   date (object)
 2.   price (float64)
 3.   bedrooms (int64)
 4.   bathrooms (float64)
 5.   sqft_living (int64)
 6.   sqft_lot (int64)
 7.   floors (float64)
 8.   waterfront (int64)
 9.   view (int64)
 10.  condition (int64)
 11.  grade (int64)
 12.  sqft_above (int64)
 13.  sqft_basement (int64)
 14.  yr_built (int64)
 15.  yr_renovated (int64)
 16.  zipcode (int64)
 17.  lat (float64)
 18.  long (float64)
 19.  sqft_living15 (int64)
 20.  sqft_lot15 (int64)

## II.I. Reading the data

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

# reading the data from csv datafile; using read_csv method from Pandas library
df_r = pd.read_csv("data/regression/kc_house_data.csv")

# displaying all the columns data of top 5 rows in the jupyter notebook
pd.set_option('max_columns', 28)
df_r.head(10)
```

```python
df_r.describe()
```

## II.II. Preprocessing of the data

```python
# get the metadata; getting familiarized with columns and data-types
df_r.info()
```

From the "Non-Null Count" column above, we can observe all the rows contain same number of `non-null` counts. From this count we can sense that there is no null values in this dataset.

To confirm this, let's check it using library function.

```python
# checking columns with null values
df_r.isna().any()
```

There is no null values in any particular columns. So we don't need to perform any data pre-processing here.

```python
# observe pairwise correlation of columns using library function
df_r.corr(method='pearson')
```

Pandas dataframe.corr() is used to find the pairwise correlation of all columns in the dataframe. Non-numeric data type columns from the dataframe is ignored (i.e. "date" column in this case).

This correlation matrix gives the measure of the strength of the association between the two columns. I used Pearson's Correlation Coefficient method to generate this matrix.

## II.III. Visualization of the data

### II.III.I. Correlation Heatmap

```python
f = plt.subplots(figsize=(10, 10))
sns.heatmap(df_r.corr(method='pearson'), annot=True, linewidths=.1,fmt= '.1f')
plt.title("Correlation Heatmap for HouseSales Data", color = 'green', fontsize = 20)
plt.show()
```

#### Observation:

1. The most correlated attributes with `price` are
    * bathrooms
    * sqft_living
    * view
    * grade
    * sqft_above
    * sqft_living15
2. Among them all, `sqft_living` and `grade` have the highest correlation coefficient (`0.7`).
3. This actually makes sense, as this data is suitable for `regrassion analysis` and house price mostly depends on the `size` and `quality`.

### II.III.II. Pie-chart

```python
# Plot number of records in the dataset for different condition

conditionList = []
conditionCount = []
for condition_name, subset in df_r.groupby('condition'):
    conditionList.append(condition_name)
    conditionCount.append(len(subset))
print(conditionList)
print(conditionCount)

plt.figure(figsize = (7, 7))
plt.pie(conditionCount, labels = conditionList)
plt.title("Pie Chart: Condition", color = 'green', fontsize = 20)
plt.show()
```

### II.III.III. Count Plots

```python
fig, ax = plt.subplots(figsize=(25,6))
sns.countplot(df_r.bathrooms)
plt.title("Count Plot: # of bathrooms", color = 'green', fontsize = 30)
```

```python
# Plot number of records in the dataset for different condition, view, waterfront, bedrooms, grade, zipcode

f, axes = plt.subplots(1, 3,figsize=(25,6))
sns.countplot(x= "condition", data=df_r, orient='v', ax=axes[0])
sns.countplot(x= "view", data=df_r, orient='v', ax=axes[1])
sns.countplot(x= "waterfront", data=df_r, orient='v', ax=axes[2])

f, axes = plt.subplots(1, 3,figsize=(25,6))
sns.countplot(x= "bedrooms", data=df_r, orient='v', ax=axes[0])
sns.countplot(x= "grade", data=df_r, orient='v', ax=axes[1])
sns.countplot(x= "zipcode", data=df_r, orient='v', ax=axes[2])
```

#### Observation:

1. The `pie-chart` and `count-plot` is used to compare parts to the whole. It helps picturing which types of data exist more in the dataset and can influence making any decision.
2. From the pie-chart we can observe that people buy more `condition 3` type house.
3. From the count-plot we can make simillar type of observations like we made in the pie-chart. That is majority people likes house with `2.5 bathrooms`.
4. Similarly, the group count-plot shows the prefered values for `condition`, `view`, `waterfront`, `bedrooms`, `grade`, and `zipcode`.

### II.III.IV. Line Plots

```python
# Plot the prices based on size

sns.set_theme(style="darkgrid")
sns.lineplot(x="sqft_living", y="price", data=df_r)
plt.title("Line Plot: Price Vs. Size", color = 'green', fontsize = 20)
```

### II.III.V. Linear Regression Plots

```python
# Plot linear regression on price based on size

g = sns.lmplot(data=df_r, x="sqft_living", y="price")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Linear Regression Plot: Price Vs. Size", color = 'green', fontsize = 20)
```

```python
# Plot linear regression on price based on size15

g = sns.lmplot(data=df_r, x="sqft_living15", y="price")
g.set_axis_labels("Living Space 15 (sqft)", "Price (usd)")
plt.title("Linear Regression Plot: Price Vs. Size_15", color = 'green', fontsize = 20)
```

```python
# Plot linear regression on price based on size_above

g = sns.lmplot(data=df_r, x="sqft_above", y="price")
g.set_axis_labels("Living Space Above (sqft)", "Price (usd)")
plt.title("Linear Regression Plot: Price Vs. Size_above", color = 'green', fontsize = 20)
```

```python
# Plot multiple linear regression on price based on size varying number of bedrooms

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="bedrooms")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t # of bedrooms)", color = 'green', fontsize = 20)
```

```python
# Plot multiple linear regression on price based on size varying waterfront

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="waterfront")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t waterfront)", color = 'green', fontsize = 20)
```

```python
# Plot multiple linear regression on price based on size varying view

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="view")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t waterfront)", color = 'green', fontsize = 20)
```

```python
# Plot multiple linear regression on price based on size varying condition

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="condition")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t condition)", color = 'green', fontsize = 20)
```

#### Observation:

1. The linear regression plots clearly show correlation between price and size.
2. The multiple linear regression plots indicate -
    * Price is higher with waterfront
    * House with view 4 have highest price within the same living space size
    * House with condition 5 price more

### II.III.VI. Box Plots

```python
# Plot price on different condition

sns.boxplot(x='condition', y='price', data=df_r)
plt.title("Box Plot: Price Vs. Condition", color = 'green', fontsize = 20)
```

```python
# Plot price on varying bedrooms

sns.boxplot(x='bedrooms', y='price', data=df_r)
plt.title("Box Plot: Price Vs. Bedrooms", color = 'green', fontsize = 20)
```

#### Observation:

1. The boxplot indicate -
    * Price range varies most for houses with 8 bedrooms

### II.III.VII. Preliminary Observation About Data

We made individual observations on the different plotting above. To summarize

* People mostly like houses with the following property -
    * Three bedroom
    * Condition 3
    * Seventh grade
    * 2.5 Bathrooms
    * Without waterfront
    * View 0
* This is actually true as we have observed from the linear regression graph that `with waterfront` price is prety high.

# III. Method

As I mentioned before, in this experiment I used two linear regression algorithms on the data described in `section II`. Given data, the goal of the linear regression algorithm is to find a best fit on all the data. The equation of a line can be written as

$$
f(x; a, b) = a x + b.
$$

Where we can replace the `a` and `b` by the weight symbol $w$,
$$
f(x; w) =  w_1 x + w_0.
$$

Considering multiple input features as $x$, we can extend the input $x$ to an input vector with bias input $x_0 = 1$:

$$
\begin{align}
f(x; w) &=  w_D x_D + \cdots + w_1 x_1 + w_0 \\
            &= \sum_{i=0}^{D} w_i x_i \quad\text{where } x_0 = 1\\
            &= w^\top x.
\end{align}
$$

Our goal here is to find the best value for $w$. So, this becomes an optimization problem. As this is a optimization problem, we have to define a cost function with respect whom we can figure out whether our choice is best or not. One possible solution is measuring the `Euclidean distances` between the target values and the predicted outputs. So we are defining our cost function like this:

$$
E(w) = \sum_{i=1}^N \Big( f(x_i; w_i) - t_i \Big)^2
$$

## III.I. Least Squares
Least squres is the simplest linear regression model. The parameter that gives best fit here will be

$$
w^* = \arg\min_w \sum_{i=1}^{N} \Big( f(x_i; w) - t_i \Big)^2
$$

Here the error funciton is quadratic. So this problem can be analytically solved by setting derivative with respect to $w$ to zero. Let's consider, the target values are collected in matrix $t$, and the input samples are in matrix $X$.

$$
\begin{align}
t &= [t_1, t_2, \cdots, t_N]^\top \\
\\
w &= [w_0, w_1, \cdots, w_D]^\top \\
\\
X &= \begin{bmatrix}
    x_{10} & x_{11} & x_{12} & \dots  & x_{1D} \\
    x_{20} & x_{21} & x_{22} & \dots  & x_{2D} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    x_{N0} & x_{N1} & x_{N2} & \dots  & x_{ND}
\end{bmatrix}
\end{align}
$$
where the first column is one's, $x_{*0} = [1, 1, \dots, 1]^\top$.

With this matrix, $f(x; w)$ can be written in matrix form as:
$$
f(x; w) = X w.
$$
Thus, the error function will become
$$
\begin{align}
E(w) &= \sum_{i=1}^N \Big(f(x_i; w_i) - t_i \Big)^2 \\
\end{align}
$$

Now, by doing the partial derivative w.r.t. $w$, we get -
$$
\begin{align}
\frac{\partial E(w)}{\partial w} &= 2 X^\top X w - 2 X^\top t \\
\end{align}
$$

By setting this to zero,
$$
\begin{align}
 2 X^\top X w - 2 X^\top t  &= 0\\
\\
w &= \big(X^\top X\big)^{-1} X^\top t
\end{align}
$$

So, given the feature matrix $X$ and our weight vector $w$, least squares find the best fitted line by this formula.

## III.II. Least Mean Squares
In `Least Squares` we observed that, it uses all the training data in finding the best fit. This is a computationally costly operation when dealing with learge features or number of entried. When data is sufficiently large, we can consider `Least Mean Squares` which learn *sequentially* instead of what doing by `Least Squares`. During the *sequential* learning process, $w$ is updated for a single data point in each iteration.

`Least Mean Squares` starts with an initial guess $w$ and changes it as it reads more data until it converges.

$$
w^{(k+1)} = w^{(k)} - \alpha \nabla E_k
$$

Here $k$ represents the steps for the repetition and $E_k$ is the error for the $k$'th sample and $\alpha$ is a learning rate. In the $k$'th iteration with the sample $x_k$, the gradient for the sum-of-squares error is

$$
\begin{align}
\nabla E_k = \frac{\partial E}{\partial w^{(k)}} &= \frac{\partial }{\partial w^{(k)}}\Big( f(x_k; w^{(k)}) - t_k \Big)^2 \\
&= 2\Big( {w^{(k)}}^\top x_k - t_k \Big) x_k.
\end{align}
$$

By replacing $\nabla E_k$, we get:

$$
w^{(k+1)} = w^{(k)} - \alpha \Big( {w^{(k)}}^\top x_k - t_k \Big) x_k.
$$

So, given the feature matrix $X$ and the randomly initialized weight vector $w$; in each iteration of data points, `Least Mean Squares` update the $w$ by this formula.

`Least Mean Squares` solves the potential computational issue of `Least Squares`, but it could be possible that this algorithm will never converge. For dealing this, we usually run this algorithm withing a fixed number of iterations.

## III.I. Super Classs Definition

These are the super class definitions for general linear model (i.e. `LinearModel`). This is the base model for the implementation of both the models (i.e. `Least Squares Model` and `Least Mean Square Model`). The abstract methods (i.e. `train` and `use`) will enable the unified interfaces for child classes to be overridden.

`LinearModel` also contains the implementations of two helper functions `_check_matrix` and `add_ones` which helps checking dimentional constraint and adding a basis to the featur matrix respectively.

```python
import numpy as np
from abc import ABC, abstractmethod

# Super class for machine learning models

class BaseModel(ABC):
    """ Super class for ITCS Machine Learning Class"""

    @abstractmethod
    def train(self, X, T):
        pass

    @abstractmethod
    def use(self, X):
        pass

class LinearModel(BaseModel):
    """
        Abstract class for a linear model

        Attributes
        ==========
        w       ndarray
                weight vector/matrix
    """

    def __init__(self):
        """
            weight vector w is initialized as None
        """
        self.w = np.zeros([1,1])
        # self.w = None

    # check if the matrix is 2-dimensional. if not, raise an exception
    def _check_matrix(self, mat, name):
        if len(mat.shape) != 2:
            raise ValueError(''.join(["Wrong matrix ", name]))

    # add a basis
    def add_ones(self, X):
        """
            add a column basis to X input matrix
        """
        self._check_matrix(X, 'X')
        return np.hstack((np.ones((X.shape[0], 1)), X))

    ####################################################
    #### abstract funcitons ############################
    @abstractmethod
    def train(self, X, T):
        """
            train linear model

            parameters
            -----------
            X     2d array
                  input data
            T     2d array
                  target labels
        """
        pass

    @abstractmethod
    def use(self, X):
        """
            apply the learned model to input X

            parameters
            ----------
            X     2d array
                  input data

        """
        pass
```

Inheriting the super class, we can define two difference classes for Least Squares and Least Mean Square algorithms.

## III.II. Least Squares

`LinearRegress` class implements the least square solutions. Here, I implemented the `train` and `use` functions. `train()` updates the weights using least squares solution and `use()` returns the predictions for the argument X by using the trained weight w.

```python
# Linear Regression Class for least squares
class LinearRegress(LinearModel):
    """
        LinearRegress class

        attributes
        ===========
        w    nd.array  (column vector/matrix)
             weights
    """
    def __init__(self):
        LinearModel.__init__(self)

    # train lease-squares model
    def train(self, X, T):
        # First creast X1 by adding 1's column to X
        N = X.shape[0]
        X1 = self.add_ones(X)
        # Next, using inverse, solve, lstsq function to get w*
        self.w = np.linalg.inv(X1.transpose().dot(X1)).dot(X1.transpose()).dot(T)

    # apply the learned model to data X
    def use(self, X):
        # First creast X1 by adding 1's column to X
        N = X.shape[0]
        X1 = self.add_ones(X)
        return X1.dot(self.w)
```

## III.III. Least Mean Squares

For online learning, I have implemented LMS algorithm here. `train_step()` function updats the weights for a single input vector $x$ and one target label. I update the weight vector with $0$ if it is not initialized or it mismatches the shape with the input vector $x$. As the name suggests, `train()` function simply call `train_step()` in a loop to learn incrementally for the batch data.

```python
import collections # for checking iterable instance

# LMS class
class LMS(LinearModel):
    """
        Lease Mean Squares. online learning algorithm

        attributes
        ==========
        w        nd.array
                 weight matrix
        alpha    float
                 learning rate
    """
    def __init__(self, alpha):
        LinearModel.__init__(self)
        self.alpha = alpha

    # batch training by using train_step function
    def train(self, X, T):
        for x, t in zip(X, T):
            self.train_step(x, t)

    # train LMS model one step
    # here the x is 1d vector
    def train_step(self, x, t):
        N = x.shape[0]
        X1 = np.hstack((np.ones(1),x))
        N = X1.shape[0]
        #print(self.w)
        if(len(self.w)!= N):
            self.w = np.zeros(N)
        y = self.w @ X1
        self.w -= self.alpha * (y - t) * X1

    # apply the current model to data X
    def use(self, X):
        N = X.shape[0]
        X1 = np.hstack((np.ones((N, 1)), X.reshape((X.shape[0], -1))))
        y = X1 @ self.w
        N = X1.shape[0]
        y.shape = (N,1)
        return y
```

# IV. Usage Examples

In this section I used some sample linear data to test the integrary of my implementations.

```python
# HERE follow are for my code tests.

import matplotlib.pyplot as plt
%matplotlib inline
```

## IV.I. Usage Example: Least Squares

First prepare the sample data and put it in $x$. Then call `LinearRegress` model to train and predict the data. Line plot indicate the integrary of the `LinearRegress` implementation.

```python
X = np.linspace(0,10, 11).reshape((-1, 1))
T = -2 * X + 3.2

ls = LinearRegress()

ls.train(X, T)

plt.plot(ls.use(X))
```

## IV.II. Usage Example: Least Mean Squares

Here I used the same sample data $x$ and call `LMS` model to train and predict the data. In the first plotting, I draw the prediction lines in each `train_step`. And the later one showing the final prediction line after calling the `train` function. The line plots indicate the integrary of the `LMS` implementation.

```python
lms = LMS(0.1)
for x, t in zip(X, T):
    lms.train_step(x, t)
    plt.plot(lms.use(X))
```

```python
lms = LMS(0.1)
lms.train(X, T)
plt.plot(lms.use(X))
```

# V. Preliminary Test

This section contains the integrary test of the implmentation of `LinearRegress` and `LMS` classes. As all the tests passed here, getting much confidence for running my implementations of linear regression models (i.e. `LinearRegress` and `LMS`) w.r.t. the `House Sales in King County, USA` data described in `section II`.

```python
##################### WHAT I WILL RELEASE ############

# Self-Test code for accuracy of your model - DO NOT MODIFY THIS
# Primilnary test data
X = np.array([[2,5],
              [6,2],
              [1,9],
              [4,5],
              [6,3],
              [7,4],
              [8,3]])
T = X[:,0, None] * 3 - 2 * X[:, 1, None] + 3
N = X.shape[0]

def rmse(T, Y):
    return np.sqrt(np.sum((T-Y)**2))

model_names = ['LS', 'LMS_All', 'LMS_1STEP']
models = [LinearRegress(), LMS(0.02), LMS(0.02)]
#train
for i, model in enumerate(models):
    print("training ", model_names[i], "...")
    if i == len(models) -1:
        # train only one step for LMS2
        model.train_step(X[0], T[0])
    else:
        model.train(X, T)

def check(a, b, eps=np.finfo(float).eps):
    if abs(a-b) > eps:
        print("failed.", a, b)
    else:
        print("passed.")

errors = [1.19e-13, 2.8753214702, 38.0584918251]
for i, model in enumerate(models):
    print("---- Testing ", model_names[i], "...", end=" ")

    # rmse test
    err = rmse(T, model.use(X))
    if check(err, errors[i], eps=1e-10):
        print ("check your weights: ", model.w)
        print ("oracle: ", )
```

# VI. Experiments

This section contains the experiment results of `LinearRegress` and `LMS` models (described in `section III`) w.r.t. the `House Sales in King County, USA` data (described in `section II`). Basically we will predict the `price` of the house considering the other features of the data (i.e. `sqft_living`, `view`, `grade`, etc.). My experiment domain is listed in the `Table 1` bellow.

      Algorithms
      Feature Selection
      Feature Scaling

      Least Squares

                           Least Mean Squares
      All Features

                           Significant Features
      Unscaled

                           Normalized

                           Standarized

      Table 1: Summary of the experiment domain.

**Feature Scaling**

Differentt scales of input variables may cause problem for the regression models. Having features on a similar scale can help the gradient descent converge more quickly towards the minima.

* Dominated by one large scale inputs!
* "Standardize" the range of independent input features

There are different ways of doing the scaling. In this experiment I considered the following two feature scaling technique,
* Min-Max Normalization (I used `MinMaxScaler` from the `sklearn.preprocessing` library)
* Standardization (I used `StandardScaler` from the `sklearn.preprocessing` library)

Before jumping to the experiment results, here is the detailed descriptions of the experiment subsections:

**VI.I. Plotting Functions**

Data visualization is crutial to understand the performance of ML models. In this section, I put general graph plotting functions that will help visualizing different performance matric. Particularly the `residual plotting` is a requirement (the third one) for the `Extra Credit` (please check `section VIII`).

A residual is a measure of how far away a point is vertically from the prediction line. Basically, it is the error between a predicted value and the corresponding observed actual value.

**VI.II. Data Partitioning**

First we randomly shuffle the whole dataset, which gives two benifits,

1. Improve the ML model quality
2. Improve the predictive performance

As we know, to test the performance of linear regession model we have to partition the data into `train-data` and `test-data`. Model training will be done on the `train-data` and then the performance of the model training is done on the `test-data`. As we have total `21613` data point, I considered `19000` data for training and the rest for testing.

Besides partitioning, feature selection is also very crutial for regression models. In feature selection we actually determine which features are importent predicting the output. As discussed in `section II.III.I`, the most correlated attributes with the target attribute (i.e. `price`) are
1. bathrooms
2. sqft_living
3. view
4. grade
5. sqft_above
6. sqft_living15

So considering this, I do two types of experiments,
1. Selecting all the features
2. Selecting the most significant features (I considered the features that have `correlation_val >= 0.5` as the significant features).

I believe this covers the first two requirements of `Extra Credit` listed in `section VIII`.

## VI.I. Plotting Functions

In this section, I put general graph plotting functions (snapshot of `Prediction Vs. Actual`, `feature scaling` comparator plotting, `residual plotting`) that will help visualizing different performance matric.

```python
"""
    Prediction comparator plotting

    attributes
    ==========
    predicted     array of predicted values
    actual        array of actual values
    feature_type  feature selection
"""
def predict_comp_plotting(predicted, actual, feature_type):
    fig = plt.figure(figsize=(10, 5), dpi = 150)
    plt.plot(predicted, label="Prediction")
    plt.plot(actual, label="Actual")

    plt.legend(bbox_to_anchor=(0.83, 0.98), loc='upper left', borderaxespad=0.)

    plt.xlabel('Test data index')
    plt.ylabel('Prediction Vs. Actual')
    plt.title('Snapshot of Prediction Vs. Actual plotting for ' + feature_type, color = 'green', fontsize = 20)

    plt.show()
```

```python
"""
    Feature scaling comparator plotting

    attributes
    ==========
    predicted   array of predicted values
    actual      array of actual values
    columns     xticks of the box plot (feature title)
    scale_type  feature scaling
"""

def feature_scal_comp_plotting(unscaled, scaled, columns, scale_type):
    f, axes = plt.subplots(1, 2, figsize=(15,6))

    plt_norm = sns.boxplot(data=pd.DataFrame(unscaled), orient='v', ax=axes[0])
    axes[0].set_xticklabels(columns, rotation=90)
    axes[0].set_title('Unscaled Data')

    sns.boxplot(data=pd.DataFrame(scaled), orient='v', ax=axes[1])
    axes[1].set_xticklabels(columns, rotation=90)
    axes[1].set_title(scale_type + ' Data')

    f.suptitle('Comparing Unscaled Vs. ' + scale_type + ' data', color = 'green', fontsize = 20)
```

```python
"""
    Residual plotting

    attributes
    ==========
    predicted     array of predicted values
    actual        array of actual values
    feature_type  feature selection
"""

def residual_plotting(predicted, actual, feature_type):
    diff = actual - predicted
    sample_count = np.arange(len(actual))
    fig=plt.figure(figsize=(10, 5), dpi= 144, facecolor='w', edgecolor='k')
    plt.scatter(sample_count, diff, marker='.')
    plt.plot([0, 2600],[0, 2600], 'r-')
    plt.title('Residual Plot for ' + feature_type, color = 'green', fontsize = 20)
```

## VI.II. Data Partitioning

First we randomly shuffle the whole dataset. And then partition the data into `train-data` and `test-data` samples. As we have total 21613 data points, I considered 19000 data for training and the rest for testing.

```python
# randomly shuffel all data
df_r.sample(frac=1)
```

```python
# constants

# list of all features
all_features = ['long', 'lat',
          'yr_renovated', 'yr_built',
          'sqft_basement', 'sqft_above',
          'sqft_lot', 'sqft_lot15',
          'sqft_living', 'sqft_living15',
          'condition', 'view', 'grade',
          'floors', 'waterfront',
          'zipcode',
          'bedrooms', 'bathrooms']

# list of significant features
sig_features = ['sqft_living', 'sqft_living15', 'sqft_above', 'view', 'grade', 'bathrooms']

# list of target features
target_features = ['price']

# feature scaling type
scale_type_norm = 'Normalized'
scale_type_stand = 'Standarized'

# feature selection type
feature_type_all = "All Features"
feature_type_sig = "Significant Features"

# training data partition threshold
train_data_th = 19000
snapshot_th = 100
```

### VI.II.I. Unscaled Data

```python
# partitioning unscaled-data of all possible features

X_All = df_r[all_features].copy()

X_All_Train = np.array(X_All.iloc[:train_data_th])
X_All_Test = np.array(X_All.iloc[train_data_th:])
```

```python
# partitioning unscaled-data of most significant co-related features (i.e. correlation_val >= 0.5)

X = df_r[sig_features].copy()
X_Train = np.array(X.iloc[:train_data_th])
X_Test = np.array(X.iloc[train_data_th:])
```

```python
# partitioning unscaled-data of target features

T = df_r[target_features].copy()
T_Train = np.array(T.iloc[:train_data_th])
T_Test = np.array(T.iloc[train_data_th:])
```

### VI.II.II. Normalized Data

```python
# partitioning normalized-data of all possible features

# data normalization with sklearn
from sklearn.preprocessing import MinMaxScaler

X_All_Train_Norm = np.array(X_All.iloc[:train_data_th])
X_All_Test_Norm = np.array(X_All.iloc[train_data_th:])

# normalize all possible feature data
norm = MinMaxScaler().fit(X_All_Train)
X_All_Train_Norm = norm.transform(X_All_Train)
X_All_Test_Norm = norm.transform(X_All_Test)

feature_scal_comp_plotting(X_All_Train, X_All_Train_Norm, all_features, scale_type_norm)
```

```python
# partitioning normalized-data of most significant co-related features (i.e. correlation_val >= 0.5)

# data normalization with sklearn
from sklearn.preprocessing import MinMaxScaler

X_Train_Norm = np.array(X.iloc[:train_data_th])
X_Test_Norm = np.array(X.iloc[train_data_th:])

# normalize significant feature data
norm = MinMaxScaler().fit(X_Train)
X_Train_Norm = norm.transform(X_Train)
X_Test_Norm = norm.transform(X_Test)

feature_scal_comp_plotting(X_Train, X_Train_Norm, sig_features, scale_type_norm)
```

### VI.II.II. Standardized Data

```python
# partitioning standarized-data of all possible features

# data standardization with  sklearn
from sklearn.preprocessing import StandardScaler

X_All_Train_Stand = np.array(X_All.iloc[:train_data_th])
X_All_Test_Stand = np.array(X_All.iloc[train_data_th:])

# define standard scaler
scaler = StandardScaler()
# standarized all possible feature data
X_All_Train_Stand = scaler.fit_transform(X_All_Train)
X_All_Test_Stand = scaler.fit_transform(X_All_Test)

feature_scal_comp_plotting(X_All_Train, X_All_Train_Stand, all_features, scale_type_stand)
```

```python
# partitioning standarized-data of most significant co-related features (i.e. correlation_val >= 0.5)

# data standardization with  sklearn
from sklearn.preprocessing import StandardScaler

X_Train_Stand = np.array(X.iloc[:train_data_th])
X_Test_Stand = np.array(X.iloc[train_data_th:])

# define standard scaler
scaler = StandardScaler()
# standarized significant feature data
X_Train_Stand = scaler.fit_transform(X_Train)
X_Test_Stand = scaler.fit_transform(X_Test)

feature_scal_comp_plotting(X_Train, X_Train_Stand, sig_features, scale_type_stand)
```

## VI.III. Experiments on Least Squares

This section contains the experiment results on `Least Squares` algorithm.

### VI.III.I. Feature Scaling: Unscaled Data

Here is the `Least Squares` algorithm's performance evaliation on unscaled data.

#### VI.III.I.I. Feature Selection: All Features

```python
lr = LinearRegress()
lr.train(X_All_Train, T_Train)
T_Predicted = lr.use(X_All_Test)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.III.I.II. Feature Selection: Significant Features

```python
lr = LinearRegress()
lr.train(X_Train, T_Train)
T_Predicted = lr.use(X_Test)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observations:**

From the rmse value (`464727090` for all the features Vs. `12318249` for significant features) and the residual plotting we can understand the importance of removing `insignificant features`. The rmse value gets better by `38x` when we considered only the significant features.

From the residual plot, we can observe that the data points get more clustered on the prediction line. Which also support the evidence we get from the change of rmse value.

### VI.III.II. Feature Scaling: Normalizaed Data

Here is the `Least Squares` algorithm's performance evaliation on normalized data.

#### VI.III.II.I. Feature Selection: All Features

```python
lr = LinearRegress()
lr.train(X_All_Train_Norm, T_Train)
T_Predicted = lr.use(X_All_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.III.II.II. Feature Selection: Significant Features

```python
lr = LinearRegress()
lr.train(X_Train_Norm, T_Train)
T_Predicted = lr.use(X_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observations:**

From the rmse value (`40098628` for all the features Vs. `12318249` for significant features) we again gets the proof of the importance of removing `insignificant features`. The residual plot also supports this.

Although, the performance gap is reduced comparing to the experiment result of the unscaled data (`38x` Vs. `3.25`). I believe this change occurs as we `normalized` the input feature data. Interestingly, we also can observe that for all the features `normalization` improves the performance by `11.5`, but there is no such observation found in the significant feature data. I believe this is because the impacts of normalization have been dominated by the signigicant features.

### VI.III.III. Feature Scaling: Standarized Data

#### VI.III.III.I. Feature Selection: All Features

```python
lr = LinearRegress()
lr.train(X_All_Train_Stand, T_Train)
T_Predicted = lr.use(X_All_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.III.III.II. Feature Selection: Significant Features

```python
lr = LinearRegress()
lr.train(X_Train_Stand, T_Train)
T_Predicted = lr.use(X_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observations:**

From the rmse value (`11513128` for all the features Vs. `12533302` for significant features) we again gets the proof of the importance of removing `insignificant features`. The residual plot also supports this. Interestingly, the performance gap indicate that the performance of the algorithm degrades while selecting significant features on the standarized data. I believe this change occurs as we `standarided` the input feature data and the impacts of standaraization have been dominated here due to feature selection.

## VI.IV. Experiments on Least Mean Squares (LMS)

### VI.IV.I. Feature Scaling: Unscaled Data

#### VI.IV.I.I. Feature Selection: All Features

```python
lms = LMS(0.02)
lms.train(X_All_Train, T_Train)
T_Predicted = lms.use(X_All_Test)

print("rmse value", rmse(T_Test, T_Predicted))
```

#### VI.IV.I.II. Feature Selection: Significant Features

```python
lms = LMS(0.02)
lms.train(X_Train, T_Train)
T_Predicted = lms.use(X_Test)

print("rmse value", rmse(T_Test, T_Predicted))
```

We got `overflow encountered in multiply` while running the `Least Mean Squares` on the unscaled data.

### VI.IV.II. Feature Scaling: Normalized Data

#### VI.IV.II.I. Feature Selection: All Features

```python
lms = LMS(0.02)
lms.train(X_All_Train_Norm, T_Train)
T_Predicted = lms.use(X_All_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.IV.II.II. Feature Selection: Significant Features

```python
lms = LMS(0.02)
lms.train(X_Train_Norm, T_Train)
T_Predicted = lms.use(X_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:100], T_Test[:100], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

### VI.IV.III. Feature Scaling: Standarized Data

#### VI.IV.III.I. Feature Selection: All Features

```python
lms = LMS(0.02)
lms.train(X_All_Train_Stand, T_Train)
T_Predicted = lms.use(X_All_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.IV.III.II. Feature Selection: Significant Features

```python
lms = LMS(0.02)
lms.train(X_Train_Stand, T_Train)
T_Predicted = lms.use(X_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:100], T_Test[:100], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observation**

* Normalization works better than standarization.
* Feature scaling makes important difference.

## VI.V. Experiment Results

In this section I like to give a recap about the overall experiment result. `Table-2` bellow gives the overall rmse values and `Table-3` represents the comparative performance (w.r.t. the rmse value).

        Table 2: Summary of the RMSE value within the experiment domain.
        Table 3: Comparison of RMSE value within the experiment domain.

                    Algorithms
                    Feature Selection
                    Feature Scaling

                    Unscaled
                    Normalized
                    Standarized

                    Least Squares
                    All Features
                    464727090
                    40098628
                    11513129

                    Significant Features
                    12318249
                    12318249
                    12533303

                    Least Mean Squares
                    All Features
                    N/A
                    11021941
                    3.25e+39

                    Significant Features
                    N/A
                    12703794
                    13333369

                    Algorithms
                    Feature Selection
                    Feature Scaling

                    Unscaled
                    Normalized
                    Standarized

                    Least Squares
                    All Features
                    37.70 X
                    3.25 X
                    0.94 X

                    Significant Features
                    X
                    X
                    1.02 X

                    Least Mean Squares
                    All Features
                    N/A
                    X
                    2.95e32 X

                    Significant Features
                    N/A
                    1.15 X
                    1.20 X

The above tables indicate that standarization within all the features gives the best result for `Least Squares`. On the other hand, normalization within all the features gives the best result for `Least Mean Squares`. It also indicates that the feature scaling improves the performance of `Least Squares` algorithm; but as I didn't able to get the result for `Least Mean Squares` on unscaled data, it was not possible to observe such behaviour. But it was surprising that instead of improving the performance, selecting significant features degrades the performance in `Least Mean Squares`. Also, the rmse value for `Least Mean Squares` on the standarized data for all the features are quite high, but the residual plotting indicates somewhat best performance. Due to the time constraint, I didn't go further to investigate these issues. Instead, I kept these as the future work for this assignment.

# VII. Conclusions

In this assignment I have performed linear regression on the `House Sales in King County, USA` using the `Linear Square` and `Linear Mean Square` methods. The `Linear Mean Square` is more sophisticated (as I get exception on the unscaled data) due to the greater detail in its computation compared to the `Linear Square` model.

# VIII. Extra Credit

If you want to work more for an extra credit, place your work here for additional analysis: weight and residual analysis.
Try to answer to the following questions:
1. what is the most and least significant features for your data. (covered in `section VI. Experiments`)
2. what are the consequences if you remove those features from the model? (covered in `section VI., VI.III.II., VI.III.III., VI.IV.II., VI.IV.III.`)
3. produce residual plots and observe the patterns for the goodness of fit (covered in `section VI., VI.III.II., VI.III.III., VI.IV.II., VI.IV.III.`)

# IX. References

1. harlfoxem (2016). House Sales in King County, USA, Version 1. Retrieved September 20, 2020 from https://www.kaggle.com/harlfoxem/housesalesprediction.

2. “Kaggle: Your Machine Learning and Data Science Community,” https://www.kaggle.com, accessed Oct. 1, 2020.


# Python and Jupyter notebooks: part 01

Last revised: 01-Jan-2020 by Dick Furnstahl [furnstahl.1@osu.edu]

This is a whirlwind tour of just the minimum we need to know about Python and Jupyter notebooks to get started doing mechanics problems.  We'll add more features and details as we proceed.

A Jupyter notebook is displayed on a web browser on a computer, tablet (e.g., IPad), or even your smartphone.  The notebook is divided into *cells*, of which two types are relevant for us:
* Markdown cells: These have headings, text, and mathematical formulas in $\LaTeX$ using a simple form of HTML (language of webpages) called markdown.
* Code cells: These have Python code (or other languages, but we'll stick to Python).

Either type of cell can be selected with your cursor and will be highlighted in color when active.  You evaluate an active cell with shift-return (as with Mathematica) or by pressing `Run` on the toolbar.  Some notes:
* When a new cell is inserted, by default it is a Code cell and will have `In []:` in front.  You can type Python expressions or entire programs in a cell.  How you break up code between cells is your choice and you can always put Markdown cells in between.  When you evaluate a cell it gets the next number, e.g., `In [5]:`.
* On the menu bar is a pulldown menu that lets you change back and forth between Code and Markdown cells.  Once you evaluate a Markdown cell, it gets formatted (and has a blue border).  To edit the Markdown cell, double click in it.

**Try double-clicking on this cell and then shift-return.**  You will see that a bullet list is created just with an asterisk and a space at the beginning of lines (without the space you get *italics* and with two asterisks you get **bold**).  **Double click on the title header above and you'll see it starts with a single #.**  Headings of subsections are made using ## or ###.  See this [Markdown cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) for a quick tour of the Markdown language (including how to add links!).

**Now try turning the next (empty) cell to a Markdown cell and type:** `Einstein says $E=mc^2$` **and then evaluate it.**  This is $\LaTeX$! (If you forget to convert to Markdown and get `SyntaxError: invalid syntax`, just select the cell and convert to Markdown with the menu.)

```python

```

The menus enable you to rename your notebook file (always ending in `.ipynb`) or `Save and Checkpoint` to save the changes to your notebook.  You can insert and delete cells (use the up and down arrows in the toolbar to easily move cells).  You will often use the `Kernel` menu to `Restart` the notebook (and possibly clear output).

## Ok, time to try out Python expressions and numpy

We can use the Jupyter notebook as a super calculator much like Mathematica and Matlab.  **Try some basic operations, modifying and evaluating the following cells, noting that exponentiation is with** `**` **and not** `^`.

```python
 1 + 1  # Everything after a number sign / pound sign / hashtag)
        #  is a comment
```

```python
3.2 * 4.713
```

If we want a floating point number (which will be the same as a `double` in C++), we *always* include a decimal point (even when we don't have to) while a number without a decimal point is an integer. (Note: in the Python we use there is no problem with 1/2 being evaluated as zero, as in C++, but we will still keep the convention of including decimal points.)

```python
3.**2
```

We can define integer, floating point, and string variables, perform operations on them, and print them.  Note that we don't have to predefine the type of a variable (unlike C++) and we can use underscores in the names (unlike Mathematica).  **Evaluate the following cells and then try your own versions.**

```python
x = 5.
print(x)
x   # If the last line of a cell returns a value, it is returned as output.
```

```python
y = 3.*x**2 - 2.*x + 7.
print('y = ', y)           # Strings delimited by ' 's (or " "s)
print(f'y = {y:.0f}')      # Just a preview: more on formatting later
print(f'y = {y:.2f}')      #  (note: this uses an "fstring")
```

```python
first_name = 'Dick'     # Strings delimited by ' 's
last_name = 'Furnstahl'
full_name = first_name + ' ' + last_name  # you can concatenate strings
print(full_name)
```

Ok, how about square roots and trigonometric functions and ...

```python
sqrt(2)
```

```python
sin(pi)
```

We need to `import` these functions through the numpy library. There are other choices, but numpy works with the arrays we will use.  Note: *Never* use `from numpy import *` instead of `import numpy as np`.  Here `np` is just a abbreviation for numpy (which we can choose to be anything, but `np` is conventional).

```python
import numpy as np
```

```python
print(np.cos(0.))
```

Now functions and constants like `np.sqrt` and `np.pi` will work.  **Go back and fix the square root and sine.**

### Debugging aside . . .

Suppose you try to import and it fails (**go ahead and evaluate the cell**):

```python
import numpie
```

When you get a `ModuleNotFoundError`, the first thing to check is whether you have misspelled the name. Try using Google, e.g., search for "python numpie". In this case (and in most others), Google will suggest the correct name (here it is numpy).  If the name does exist, check whether it sounds like the package you wanted.

If you have the correct spelling, check whether you have installed the relevant package.  If you installed Python with Anaconda (which we will assume you did -- if not, do it!), then use `conda list`, e.g., `conda list numpy` in a Terminal window (on a Mac or Linux box) or in an Anaconda Prompt window (on a Windows PC).

### numpy arrays

At first, we will most often use numpy arrays so we'll start with those.  They are *like* lists delimited by square brackets, i.e., `[]`s, and we will construct them with `np.arange(min, max, step)` to get an array from `min` to `max` in steps of `step`. Examples:

```python
t_pts = np.arange(0., 10., .1)
t_pts
```

If we give a numpy array to a function, each term in the list is evaluated with that function:

```python
x = np.arange(1., 5., 1.)
print(x)
print(x**2)
print(np.sqrt(x))
```

We can pick out elements of the list.  **Why does the last one fail?**

```python
print(x[0])
print(x[3])
print(x[4])
```

An alternative to `np.arange` is `np.linspace(min, max, number)` to get an array from `min` to `max` with `number` elements.  Example:

```python
u_pts = np.linspace(0., 10., 10)
u_pts
```

**Change the last statement for `u_pts` to get an array from 0 to 10 spaced by 1.**

## Getting help

You will often need help identifying the appropriate Python (or numpy or scipy or ...) command or you will need an example of how to do something or you may get an error message you can't figure out.  In all of these cases, Google (or equivalent) is your friend. Always include "python" in the search string (or "numpy" or "matplotlib" or ...) to avoid getting results for a different language. You will usually get an online manual as one of the first responses if you ask about a function; these usually have examples if you scroll down. Otherwise, answers from *Stack Overflow* queries are your best bet to find a useful answer.

## Functions

There are many Python language features that we will use eventually, but in the short term what we need first are functions.  Here we first see the role of *indentation* in Python in place of {}s or ()s in other languages.  We'll always indent four spaces (never tabs!).  We know a function definition is complete when the indentation stops.

To find out about a Python function or one you define, put your cursor on the function name and hit shift+Tab+Tab. **Go back and try it on `np.arange`.**

```python
# Use "def" to create new functions.
#  Note the colon and indentation (4 spaces).
def my_function(x):
    """This function squares the input.  Always include a brief description
        at the top between three starting and three ending quotes.  We will
        talk more about proper documentation later.
        Try shift+Tab+Tab after you have evaluated this function.
    """
    return x**2

print(my_function(5.))

# We can pass an array to the function and it is evaluated term-by-term.
x_pts = np.arange(1., 10., 1.)
print(my_function(x_pts))
```

```python
# Two variables, with a default for the second
def add(x, y=4.):
    """Add two numbers."""
    print("x is {} and y is {}".format(x, y))
    return x + y  # Return values with a return statement

# Calling functions with parameters
print('The sum is ', add(5, 6))  # => prints out "x is 5 and y is 6" and returns 11

# Another way to call functions is with keyword arguments
add(y=6, x=5)  # Keyword arguments can arrive in any order.

```

**How do you explain the following result?**

```python
add(2)
```

### Debugging aside . . .

There are two bugs in the following function.  **Note the line where an error is first reported and fix the bugs sequentially (so you see the different error messages).**

```python
def hello_function()
    msg = "hello, world!"
    print(msg)
     return msg
```

## Plotting with Matplotlib

Matplotlib is the plotting library we'll use, at least at first.  We'll follow convention and abbreviate the module we need as `plt`.  The `%matplotlib inline` line tells the Jupyter notebook to make inline plots (we'll see other possibilities later).

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
```

Procedure we'll use to make the skeleton plot:
0. Generate some data to plot in the form of arrays.
1. Create a figure;
2. add one or more subplots;
3. make a plot and display it.

```python
t_pts = np.arange(0., 10., .1)     # step 0.
x_pts = np.sin(t_pts)  # More often this would be from a function
                       #  *we* write.

my_fig = plt.figure()              # step 1.
my_ax = my_fig.add_subplot(1,1,1)  # step 2: rows=1, cols=1, 1st subplot
my_ax.plot(t_pts, x_pts)           # step 3: plot x vs. t
```

NOTE: When making just a single plot, you will more usually see steps 1 to 3 compressed into `plt.plot(t_pts, np.sin(t_pts))`.  *Don't do this.*  It saves a couple of lines but restricts your ability to easily extend the plot, which is what we want to make easy.

We can always go back and dress up the plot:

```python
my_fig = plt.figure()
my_ax = my_fig.add_subplot(1,1,1)  # nrows=1, ncols=1, first plot
my_ax.plot(t_pts, x_pts, color='blue', linestyle='--', label='sine')

my_ax.set_xlabel('t')
my_ax.set_ylabel(r'$\sin(t)$')  # use $...$ to get LaTeX and r to render it
my_ax.set_title('Sine wave')

# here we'll put the function in the call to plot!
my_ax.plot(t_pts, np.cos(t_pts), label='cosine')  # just label the plot

my_ax.legend();  # turn on legend (try removing the semicolon)

```

Now make two subplots:

```python
y_pts = np.exp(t_pts)         # another function for a separate plot

fig = plt.figure(figsize=(10,5))  # allow more room for two subplots

# call the first axis ax1
ax1 = fig.add_subplot(1,2,1)  # one row, two columns, first plot
ax1.plot(t_pts, x_pts, color='blue', linestyle='--', label='sine')
ax1.plot(t_pts, np.cos(t_pts), label='cosine')  # just label the plot
ax1.legend()

ax2 = fig.add_subplot(1,2,2)  # one row, two columns, second plot
ax2.plot(t_pts, np.exp(t_pts), label='exponential')
ax2.legend();

```

## Solving ordinary differential equations (ODEs)

Newton's 2nd Law for one particle:

$\begin{align}
  \mathbf{F} = m\mathbf{a} = m\frac{d\mathbf{v}}{dt} = m\frac{d^2\mathbf{x}}{dt^2}
\end{align}$

is a first-order differential equation in the velocity vector and a second-order differential equation in the position vector. Here we assume that $\mathbf{F} = \mathbf{F}(\mathbf{x}, \mathbf{v}, t)$ does not have higher derivatives. So we need to know how to solve such differential equations, usually with initial conditions (as opposed to boundary conditions).

Here is how to do it with the Scipy function `odeint`.  (Note that `solve_ivp` is now preferred to `odeint`; we'll switch to that later but it is easier to get started with `odeint`.)

### First-order ODE

```python
# Import the required modules
import numpy as np
import matplotlib.pyplot as plt

from scipy.integrate import odeint  # Get only odeint from scipy.integrate
# for the future:
from scipy.integrate import solve_ivp   # Now preferred to odeint
```

Let's try a one-dimensional first-order ODE, say:

$\begin{align}
\quad
\frac{dv}{dt} = -g, \quad \mbox{with} \quad v(0) = 10
\end{align}$

in some appropriate units (we'll use MKS units by default).  This ODE can be separated and directly integrated:

$\begin{align}
  \int_{v_0=10}^{v} dv' = - g \int_{0}^{t} dt'
  \quad\Longrightarrow\quad
    v - v_0 = - g (t - 0)
  \quad\Longrightarrow\quad
   v(t) = 10 - gt
\end{align}$

```python
# Define a function which calculates the derivative
def dv_dt(v, t, g=9.8):
    """Returns the right side of a simple first-order ODE with default g."""
    return -g

t_pts = np.linspace(0., 10., 101)     # 101 points from t=0 and t=10.
v_0 = 10.0  # the initial condition
v_pts = odeint(dv_dt, v_0, t_pts)  # odeint( function for rhs,
                                   #         initial value of v(t),
                                   #         array of t values )

```

```python

```

Let's check the output $v(t)$.  Does it make sense?

```python
print(v_pts[0:10])
v_pts.shape
```

Now plot $v(t)$ calculated numerically and the analytic solution:

```python
g = 9.8
v_pts_exact = v_0 - g*t_pts

fig = plt.figure(figsize=(8,4))

ax = fig.add_subplot(1,2,1)
ax.plot(t_pts, v_pts, label='numerical', color='blue', lw=4)
ax.plot(t_pts, v_pts_exact, label='exact', color='red')
ax.set_xlabel('t [sec]')
ax.set_ylabel('v(t) [meters/sec]')
ax.legend();

diff = v_pts_exact - v_pts.flatten()
ax2 = fig.add_subplot(1,2,2)
ax2.semilogy(t_pts, diff, label='difference', color='red')
ax2.set_xlabel('t [sec]')
ax2.set_ylabel('v(t) [meters/sec]')
ax2.legend();

fig.tight_layout()
```

We adjusted the colors and linewidths so you could see that the lines are on top of each other. **Why are the differences around $10^{-15}$?**

Try solving instead:

$\begin{align}
\quad
\frac{dx}{dt} = -x \quad \mbox{with} \quad x(0) = 1
\end{align}$

both analytically and numerically.

### Second-order ODE

Suppose we have a second-order ODE such as:

$$
\quad y'' + 2 y' + 2 y = \cos(2x), \quad \quad y(0) = 0, \; y'(0) = 0
$$

We can turn this into two first-order equations by defining a new dependent variable equal to $y'$ (we'll call it $z$):

$$
\quad z \equiv y' \quad \Rightarrow \quad z' + 2 z + 2y = \cos(2x)
  \quad\mbox{or}\quad z' = -2z -2y + \cos(2x), \quad\mbox{with}
\quad z(0)=y(0) = 0.
$$

Now introduce the vector

$$
  \mathbf{U}(x) = \left(\begin{array}{c}
                         y(x) \\
                         z(x)
                        \end{array}
                  \right)
        \quad\Longrightarrow\quad
    \frac{d\mathbf{U}}{dx} =
                             \left(\begin{array}{c}
                                    y' \\
                                    z'
                                   \end{array}
                             \right)
                           =
                             \left(\begin{array}{c}
                                    z \\
                                    -2 y' - 2 y + \cos(2x)
                                   \end{array}
                             \right)
              \quad\mbox{with}\quad
    \mathbf{U}(0) = \left(\begin{array}{c}
                         0 \\
                         0
                        \end{array}
                  \right)
$$

We can solve this system of ODEs using `odeint` with lists, as follows:

```python
# Define a function for the right side
def dU_dx(U, x):
    """Right side of the differential equation to be solved.
    U is a two-component vector with y=U[0] and z=U[1].
    Thus this function should return [y', z']
    """
    return [U[1], -2*U[1] - 2*U[0] + np.cos(2*x)]

# initial condition U_0 = [y(0)=0, z(0)=y'(0)=0]
U_0 = [0., 0.]

x_pts = np.linspace(0, 15, 200)  # Set up the mesh of x points
U_pts = odeint(dU_dx, U_0, x_pts)  # U_pts is a 2-dimensional array
y_pts = U_pts[:, 0]   # Ok, this is tricky.  For each x, U_pts has two
                      #  components.  We want the upper component for all
                      #  x, which is y(x).  The : means all of the first
                      #  index, which is x, and the 0 means the first
                      #  component in the other dimension.
                      # What is U_pts[:, 1]? (Try plotting it.)
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(x_pts, y_pts)
ax.set_xlabel('x')
ax.set_ylabel('y')
```

## Widgets!

A widget is an object such as a slider or a check box or a pulldown menu.  We can use them to make it easy to explore different parameter values in a problem we're solving, which is invaluable for building intuition.  They act on the argument of a function.  We'll look at a simple case here but plan to explore this much more as we proceed.

The set of widgets we'll use here (there are others!) is from `ipywidgets`; we'll conventionally import the module as `import ipywidgets as widgets` and we'll also often use `display` from `Ipython.display`.

```python
import numpy as np
import matplotlib.pyplot as plt

import ipywidgets as widgets
from IPython.display import display

%matplotlib inline
```

The simplest form is to use `interact`, which we pass a function name and the variables with ranges.  By default this makes a *slider*, which takes on integer or floating point values depending on whether you put decimal points in the range. **Try it! Then modify the function and try again.**

```python
# We can do this to any function
def test_f(x=5.):
    """Test function that prints the passed value and its square.
       Note that there is no return value in this case."""
    print ('x = ', x, ' and  x^2 = ', x**2)

widgets.interact(test_f, x=(0.,10.));
```

```python
# Explicit declaration of the widget (here FloatSlider) and details
def test_f(x=5.):
    """Test function that prints the passed value and its square.
       Note that there is no return value in this case."""
    print ('x = ', x, ' and  x^2 = ', x**2)

widgets.interact(test_f,
                 x = widgets.FloatSlider(min=-10,max=30,step=1,value=10));
```

Here's an example with some bells and whistles for a plot.  **Try making changes!**

```python
def plot_it(freq=1., color='blue', lw=2, grid=True, xlabel='x',
            function='sin'):
    """ Make a simple plot of a trig function but allow the plot style
        to be changed as well as the function and frequency."""
    t = np.linspace(-1., +1., 1000)  # linspace(min, max, total #)

    fig = plt.figure(figsize=(8,6))
    ax = fig.add_subplot(1,1,1)

    if function=='sin':
        ax.plot(t, np.sin(2*np.pi*freq*t), lw=lw, color=color)
    elif function=='cos':
        ax.plot(t, np.cos(2*np.pi*freq*t), lw=lw, color=color)
    elif function=='tan':
        ax.plot(t, np.tan(2*np.pi*freq*t), lw=lw, color=color)

    ax.grid(grid)
    ax.set_xlabel(xlabel)

widgets.interact(plot_it,
                 freq=(0.1, 2.), color=['blue', 'red', 'green'],
                 lw=(1, 10), xlabel=['x', 't', 'dog'],
                 function=['sin', 'cos', 'tan'])

```

```python

```

```python

```

```python

```


## Solution for $\mathbb{H}_{u,v}$

In this notebook we validate our recursive solution to the integral $\mathbb{H}$.

```python
import sympy
from sympy import *
import numpy as np
np.random.seed(0)
```

```python
# Initialize the session
init_session(quiet=True)
print("Using sympy version", sympy.__version__)

# Define our symbols
phi, lam1, lam2 = symbols('phi lambda_1 lambda_2')
lam = Array((lam1, lam2))
```

    Using sympy version 1.4

Here's the exact version of the integral, computed with `sympy`:

```python
def Hexact(u, v):
    return integrate(cos(phi) ** u * sin(phi) ** v, (phi, lam1, lam2))
```

Here's our recursive version:

```python
def Delta(x):
    return x[1] - x[0]

def H(u, v):
    if u == v == 0:
        return Delta(lam)
    elif u == 1 and v == 0:
        return Delta(lam.applyfunc(sin))
    elif u == 0 and v == 1:
        return Delta(lam.applyfunc(lambda x: -cos(x)))
    elif u == 1 and v == 1:
        return Delta(lam.applyfunc(lambda x: -cos(x) ** 2 / 2))
    elif u < 2 and v >= 2:
        return (
            Delta(lam.applyfunc(lambda x: -cos(x) ** (u + 1) * sin(x) ** (v - 1)))
            + (v - 1) * H(u, v - 2)
        ) / (u + v)
    else:
        return (
            Delta(lam.applyfunc(lambda x: cos(x) ** (u - 1) * sin(x) ** (v + 1)))
            + (u - 1) * H(u - 2, v)
        ) / (u + v)
```

`sympy` has a little trouble checking for the analytical equivalence of the expressions as `u` and `v` get large, so let's check that the expressions agree numerically for several different values of $\lambda$:

```python
for i in range(10):
    x1, x2 = np.random.random(2) * 2 * np.pi
    for u in range(5):
        for v in range(5):
            diff = (H(u, v) - Hexact(u, v)).replace(lam1, x1).replace(lam2, x2)
            assert abs(diff) < 1e-15
```


## Gráfica de T de rocío

$f(\psi)=\sum_{i=1}^{c}\frac{z_i[1-k_i]}{1+\psi[k_i-1]}$

Fracciones del problema realizado en clase:

$z_{n-Butano}=0.2$

$z_{n-Pentano}=0.25$

$z_{n-Hexano}=0.25$

$z_{n-Heptano}=0.3$

Constantes del problema realizado en clase:

$K_{n-Butano}=4$

$K_{n-Pentano}=1.9$

$K_{n-Hexano}=1$

$K_{n-Heptano}=0.5$

Desarrollando la expresión:

$f(\psi)=\frac{z_1[1-k_1]}{1+\psi[k_1-1]} + \frac{z_2[1-k_2]}{1+\psi[k_2-1]} + \frac{z_3[1-k_3]}{1+\psi[k_3-1]} + \frac{z_4[1-k_4]}{1+\psi[k_4-1]}$

```python
# Librerias
import numpy as np
from numpy import poly1d,polyfit
import matplotlib.pyplot as plt
from sympy import Symbol

# Para imprimir en formato LaTex
from sympy.interactive import printing
printing.init_printing(use_latex=True)

# Corremos nuestro script con las librerías anteriormente citadas
%run function_zoo.py
```

## Derivada de la función

```python
psi = Symbol('psi')

z_1 = Symbol('z_1')
z_2 = Symbol('z_2')
z_3 = Symbol('z_3')
z_4 = Symbol('z_4')

k_1 = Symbol('k_1')
k_2 = Symbol('k_2')
k_3 = Symbol('k_3')
k_4 = Symbol('k_4')

f_psi= ((z_1*(1-k_1)/(1+psi*(k_1-1))) + (z_2*(1-k_2)/(1+psi*(k_2-1))) + (z_3*(1-k_3)/(1+psi*(k_3-1))) + (z_4*(1-k_4)/(1+psi*(k_4-1))))
dpsi = f_psi.diff(psi)

print('Función:')
display(f_psi)

print('Derivada:')
display(dpsi)
```

```python
# Diseño de la gráfica por medio de Sumatoria
z = [0.2,0.25,0.25,0.3]
k = [4, 1.9, 1, 0.5]

[psi,f_psi] = Rachford_Rice_4(z,k,0,1,.1)

p2 = 0.0315
plt.plot(psi,f_psi, marker='*')
plt.grid()
plt.xlabel('$\psi$')
plt.ylabel('$f(\psi$)').set_rotation(0)
plt.axvline(x=1,color='r',ls='dotted')
plt.axvline(x=0,color='r',ls='dotted')
# Prueba con el resultado de la sumatoria
plt.axhline(y=0.0315,color='green',ls='dotted',label='Diferencial al valor 0')
plt.legend()
plt.show()
```

```python
# Diseño de la gráfica por forma vectorizada
psi = np.arange(0,1.1,.1)
f_psia= ((z[0]*(1-k[0])/(1+psi*(k[0]-1))) + (z[1]*(1-k[1])/(1+psi*(k[1]-1))) + (z[2]*(1-k[2])/(1+psi*(k[2]-1))) + (z[3]*(1-k[3])/(1+psi*(k[3]-1))))

plt.plot(psi,f_psia,marker='*')
plt.grid()
plt.xlabel('$\psi$')
plt.ylabel('$f(\psi$)').set_rotation(0)
plt.axvline(x=1,color='r',ls='dotted')
plt.axvline(x=0,color='r',ls='dotted')
# Prueba con el resultado de la sumatoria
plt.axhline(y=0.0315,color='green',ls='dotted',label='Diferencial al valor 0')
plt.legend()
plt.show()
```


# Skew G-Jensen-Shannon divergence

```python
import matplotlib.pyplot as plt
import numpy as np

from matplotlib import cm

from utils import PlotParams
```

```python
plotter = PlotParams()
plotter.set_params()
FIG_DIR = os.path.join(os.pardir, 'figs')
```

```python
def gaussian(x, mu, sig):
    return 1. / (np.sqrt(2. * np.pi) * sig) * np.exp(-np.power(x - mu, 2.) / (2 * np.power(sig, 2.)))
```

## One-dimensional normal distribution comparison

### Distributions

```python
x = np.linspace(-5, 5, 500)

example_num = 1
m_1, s_1 = -2, 1
m_2, s_2 = 2, 2

# example_num = 2
# m_1, s_1 = 0, 0.5
# m_2, s_2 = 0, 3

# example_num = 3
# m_1, s_1 = 2, 1
# m_2, s_2 = -2, 1

# example_num = 4
# m_1, s_1 = -5, 1
# m_2, s_2 = 5, 2

p = gaussian(x, m_1, s_1)
q = gaussian(x, m_2, s_2)

plt.plot(x, p, 'b', label='$p(x)\sim\mathcal{N}$'+f'({m_1},{s_1})')
plt.plot(x, q, 'r', label='$q(x)\sim\mathcal{N}$'+f'({m_2},{s_2})')
plt.xlim(-5, 5), plt.ylim(bottom=0)
plt.xlabel('$x$')
plt.legend()
plt.savefig(os.path.join(FIG_DIR, f'gaussians-1D_{example_num}.pdf'), bbox_inches='tight')
```

### Kullback-Leibler divergence
* Clearly asymmetric

```python
kl_pq = p * np.log(p / q)
plt.plot(x, kl_pq, 'b', label=r'KL$(p\parallel q)$')
kl_qp = q * np.log(q / p)
plt.plot(x, kl_qp, 'r', label=r'KL$(q\parallel p)$')
plt.xlim(-5, 5)
plt.xlabel('$x$')
_ = plt.legend()
```

### Jensen-Shannon divergence
Jenson-Shannon divergence is the average divergence to the **arithmetic** mean
\begin{align}
    \textrm{JS}(q\parallel p) = \frac{1}{2}\textrm{KL}\left(p\parallel \frac{p+q}{2}\right) + \frac{1}{2}\textrm{KL}\left(q\parallel\frac{p+q}{2}\right).
\end{align}

```python
_ = range_ = np.linspace(0, 1, 11)
cmap = cm.jet(range_)
for a, c in zip(range_, cmap):
    m = a * p + (1 - a) * q
    plt.plot(x, m, c=c, label=r'$\alpha=$'+f'{a:.1f}')
plt.xlim(-5, 5), plt.ylim(bottom=0)
plt.xlabel('$x$')
```

```python
mo = (p + q) / 2
```

```python
plt.plot(x, p, 'b')
plt.plot(x, q, 'r')
plt.plot(x, mo, 'g', label=r'$m(x)=\frac{1}{2}(p(x)+q(x))$')
plt.xlim(-5, 5), plt.ylim(bottom=0)
plt.xlabel('$x$')
_ = plt.legend()
plt.savefig(os.path.join(FIG_DIR, f'gaussians-1D_JS-0.5.pdf'), bbox_inches='tight')
```

KL divergence terms to **arithmetic** mean in JS divergence

```python
kl_pm = p * np.log(p / mo)
plt.plot(x, kl_pm, 'b', label=r'KL$(p\parallel m)$')
kl_qm = q * np.log(q / mo)
plt.plot(x, kl_qm, 'r', label=r'KL$(q\parallel m)$')
plt.xlim(-5, 5)
plt.xlabel('$x$')
_ = plt.legend()
```

Final (point-wise) JS divergence

```python
js_pq = (kl_pm + kl_qm) / 2
plt.plot(x, js_pq, 'k', label=r'JS$(p\parallel q)$')
plt.xlim(-5, 5), plt.ylim(bottom=0)
plt.xlabel('$x$')
_ = plt.legend()
```

## Skew geometric-Jensen-Shannon

In one dimension, the matrix harmonoic barycentre for $\Sigma_{\alpha}$
\begin{align}
    \Sigma_{\alpha} &= \left((1-\alpha)\Sigma_{1}^{-1} + \alpha\Sigma_{2}^{-1}\right)^{-1} \\
    \mu_{\alpha} &= \Sigma_{\alpha}\left((1-\alpha)\Sigma_{1}^{-1}\mu_{1}+\alpha\Sigma_{2}^{-1}\mu_{2}\right),
\end{align}
reduces to a weighted harmonic mean:
\begin{align}
        \sigma_{\alpha}^{2} &= \frac{1}{(1-\alpha)\frac{1}{\sigma_{1}^{2}}+\alpha\frac{1}{\sigma_{2}^{2}}} \\
        \mu_{\alpha}^{2} &= \sigma_{\alpha}^{2} \left((1-\alpha)\frac{\mu_{1}}{\sigma_{1}^{2}}+\alpha\frac{\mu_{2}}{\sigma_{2}^{2}}\right)
\end{align}

```python
def get_s_a(m_1, s_1, m_2, s_2, a=0.5):
    s_a = 1 / ((1 - a) / s_1 + a / s_2)
    m_a = s_a * ((1 - a) * m_1 / s_1 + a * m_2 / s_2)

    return m_a, s_a
```

```python
m_a, s_a = get_s_a(m_1, s_1, m_2, s_2, a=0.5)
m = gaussian(x, m_a, s_a)
```

Skew geometric-Jenson-Shannon divergence is the weighted average divergence to the **geometric** mean
\begin{align}
    \textrm{JS}(q\parallel p) = (1-\alpha)\textrm{KL}\left(p, G_{\alpha}(p,q)\right) + \alpha\textrm{KL}\left(q, G_{\alpha}(p,q)\right).
\end{align}

```python
plt.plot(x, p, 'b')
plt.plot(x, q, 'r')
plt.plot(x, mo, 'g', label=r'$\frac{1}{2}(p(z)+q(z|x))$')
plt.plot(x, m, 'g:', label=r'$p(z)^{\frac{1}{2}}q(z|x)^{\frac{1}{2}}$')
plt.xlim(-5, 5), plt.ylim(bottom=0)
_ = plt.legend()
plt.savefig(os.path.join(FIG_DIR, f'gaussians-1D_GJS-0.5.pdf'), bbox_inches='tight')
```

$\alpha=[0.0,\ldots,1.0]$

```python
_ = range_ = np.linspace(0, 1, 11)
cmap = cm.jet(range_)
for a, c in zip(range_, cmap):
    m_a, s_a = get_s_a(m_1, s_1, m_2, s_2, a=a)
    m = gaussian(x, m_a, s_a)
    plt.plot(x, m, c=c, label=r'$\alpha=$'+f'{a:.1f}')
plt.xlim(-5, 5), plt.ylim(bottom=0)
plt.xlabel('$x$')
plt.legend()
```

```python
m_a, s_a = get_s_a(m_1, s_1, m_2, s_2, a=0.5)
m = gaussian(x, m_a, s_a)
```

KL divergence terms to **geometric** mean in G-JS divergence

```python
kl_pm = p * np.log(p / m)
plt.plot(x, kl_pm, 'b', label=r'$KL(p\parallel m)$')
kl_qm = q * np.log(q / m)
plt.plot(x, kl_qm, 'r', label=r'$KL(q\parallel m)$')
plt.xlim(-5, 5)
plt.xlabel('$x$')
_ = plt.legend()
```

Final (point-wise) G-JS divergence

```python
gjs_pq = (kl_pm + kl_qm) / 2
plt.plot(x, gjs_pq, 'k', label=r'G$^{\alpha}$-JS$(p\parallel q)$')
plt.xlim(-5, 5)
plt.xlabel('$x$')
_ = plt.legend()
```

```python
plt.plot(x, p, 'b')
plt.plot(x, q, 'r')
plt.plot(x, js_pq, 'k', label=r'JS$(p\parallel q)$')
plt.plot(x, gjs_pq, 'k:', label=r'G$^{\alpha}$-JS$(p\parallel q)$')
plt.xlim(-5, 5)
plt.xlabel('$x$')
plt.legend()
plt.savefig(os.path.join(FIG_DIR, f'gaussians-divergences-1D_{example_num}.pdf'), bbox_inches='tight')
```

```python

```

```python

```


# Structured prediction

In this example$\newcommand{\reals}{\mathbf{R}}$$\newcommand{\ones}{\mathbf{1}}$, we fit a regression model to structured data, using an LLCP.
The training dataset $\mathcal D$ contains $N$ input-output pairs $(x, y)$,
where $x \in \reals^{n}_{++}$ is an input and $y \in \reals^{m}_{++}$ is an
outputs. The entries of each output $y$ are sorted in ascending order, meaning
$y_1 \leq y_2 \leq \cdots y_m$.

Our regression model $\phi : \reals^{n}_{++} \to \reals^{m}_{++}$ takes as
input a vector $x \in \reals^{n}_{++}$, and solves an LLCP to produce a
prediction $\hat y \in \reals^{m}_{++}$. In particular, the solution of the
LLCP is model's prediction. The model is of the form
$$
\begin{equation}
\begin{array}{lll}
\phi(x) = &
\mbox{argmin} & \ones^T (z/y + y / z) \\
& \mbox{subject to} &  y_i \leq y_{i+1}, \quad i=1, \ldots, m-1 \\
&& z_i = c_i x_1^{A_{i1}}x_2^{A_{i2}}\cdots x_n^{A_{in}}, \quad i = 1, \ldots, m.
\end{array}\label{e-model}
\end{equation}
$$
Here, the minimization is over $y \in \reals^{m}_{++}$ and an auxiliary
variable $z \in \reals^{m}_{++}$, $\phi(x)$ is the optimal value of $y$, and
the parameters are $c \in \reals^{m}_{++}$ and $A \in \reals^{m \times n}$. The
ratios in the objective are meant elementwise, as is the inequality $y \leq z$, and
$\ones$ denotes the vector of all ones. Given a vector $x$, this model finds a
sorted vector $\hat y$ whose entries are close to monomial functions of $x$
(which are the entries of $z$), as measured by the fractional error.

The training loss
$\mathcal{L}(\phi)$ of the model on the training set is the mean squared loss
$$
\mathcal{L}(\phi) = \frac{1}{N}\sum_{(x, y) \in \mathcal D} \|y - \phi(x)\|_2^2.
$$
We emphasize that $\mathcal{L}(\phi)$ depends on $c$ and $A$.
In this example, we fit the parameters $c$ and $A$ in the LLCP
to minimize the training loss $\mathcal{L}(\phi)$.

**Fitting.** We fit the parameters by an iterative projected
gradient descent method on $\mathcal L(\phi)$. In each iteration, we first
compute predictions $\phi(x)$ for each input in the training set; this requires
solving $N$ LLCPs. Next, we evaluate the training loss $\mathcal L(\phi)$. To
update the parameters, we compute the gradient $\nabla \mathcal L(\phi)$ of the
training loss with respect to the parameters $c$ and $A$. This
requires differentiating through the solution map of the LLCP. We can compute this gradient efficiently, using the ``backward`` method in CVXPY
(or CVXPY Layers). Finally, we subtract
a small multiple of the gradient from the parameters. Care must be taken to
ensure that $c$ is strictly positive; this can be done by clamping the entries
of $c$ at some small threshold slightly above zero. We run this method for
a fixed number of iterations.

This example is described in the paper [Differentiating through Log-Log Convex Programs](http://web.stanford.edu/~boyd/papers/pdf/diff_llcvx.pdf).

Shane Barratt formulated the idea of using an optimization layer to regress on sorted vectors.

**Requirements.**
This example requires PyTorch and CvxpyLayers >= v0.1.3.

```python
from cvxpylayers.torch import CvxpyLayer

import cvxpy as cp
import matplotlib.pyplot as plt
import numpy as np
import torch

torch.set_default_tensor_type(torch.DoubleTensor)
%matplotlib inline
```

### Data generation

```python
n = 20
m = 10

# Number of training input-output pairs
N = 100

# Number of validation pairs
N_val = 50
```

```python
torch.random.manual_seed(243)
np.random.seed(243)

normal = torch.distributions.multivariate_normal.MultivariateNormal(torch.zeros(n), torch.eye(n))
lognormal = lambda batch: torch.exp(normal.sample(torch.tensor([batch])))

A_true = torch.randn((m, n)) / 10
c_true = np.abs(torch.randn(m))
```

```python
def generate_data(num_points, seed):
    torch.random.manual_seed(seed)
    np.random.seed(seed)

    latent = lognormal(num_points)
    noise = lognormal(num_points)
    inputs = noise + latent

    input_cp = cp.Parameter(pos=True, shape=(n,))
    prediction = cp.multiply(c_true.numpy(), cp.gmatmul(A_true.numpy(), input_cp))
    y = cp.Variable(pos=True, shape=(m,))
    objective_fn = cp.sum(prediction / y + y/prediction)
    constraints = []
    for i in range(m-1):
        constraints += [y[i] <= y[i+1]]
    problem = cp.Problem(cp.Minimize(objective_fn), constraints)

    outputs = []
    for i in range(num_points):
        input_cp.value = inputs[i, :].numpy()
        problem.solve(cp.SCS, gp=True)
        outputs.append(y.value)
    return inputs, torch.stack([torch.tensor(t) for t in outputs])
```

```python
train_inputs, train_outputs = generate_data(N, 243)
plt.plot(train_outputs[0, :].numpy())
```

```python
val_inputs, val_outputs = generate_data(N_val, 0)
plt.plot(val_outputs[0, :].numpy())
```

## Monomial fit to each component

We will initialize the parameters in our LLCP model by fitting monomials to the training data, without enforcing the monotonicity constraint.

```python
log_c = cp.Variable(shape=(m,1))
theta = cp.Variable(shape=(n, m))
inputs_np = train_inputs.numpy()
log_outputs_np = np.log(train_outputs.numpy()).T
log_inputs_np = np.log(inputs_np).T
offsets = cp.hstack([log_c]*N)
```

```python
cp_preds = theta.T @ log_inputs_np + offsets
objective_fn = (1/N) * cp.sum_squares(cp_preds - log_outputs_np)
lstq_problem = cp.Problem(cp.Minimize(objective_fn))
```

```python
lstq_problem.is_dcp()
```

    True

```python
lstq_problem.solve(verbose=True)
```

    -----------------------------------------------------------------
               OSQP v0.6.0  -  Operator Splitting QP Solver
                  (c) Bartolomeo Stellato,  Goran Banjac
            University of Oxford  -  Stanford University 2019
    -----------------------------------------------------------------
    problem:  variables n = 1210, constraints m = 1000
              nnz(P) + nnz(A) = 23000
    settings: linear system solver = qdldl,
              eps_abs = 1.0e-05, eps_rel = 1.0e-05,
              eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
              rho = 1.00e-01 (adaptive),
              sigma = 1.00e-06, alpha = 1.60, max_iter = 10000
              check_termination: on (interval 25),
              scaling: on, scaled_termination: off
              warm start: on, polish: on, time_limit: off

    iter   objective    pri res    dua res    rho        time
       1   0.0000e+00   3.30e+00   1.22e+04   1.00e-01   3.06e-03s
      50   1.0014e-02   1.72e-07   1.64e-07   1.75e-03   7.37e-03s
    plsh   1.0014e-02   1.56e-15   1.17e-14   --------   9.68e-03s

    status:               solved
    solution polish:      successful
    number of iterations: 50
    optimal objective:    0.0100
    run time:             9.68e-03s
    optimal rho estimate: 8.77e-05

    0.010014212812318733

```python
c = torch.exp(torch.tensor(log_c.value)).squeeze()
lstsq_val_preds = []
for i in range(N_val):
    inp = val_inputs[i, :].numpy()
    pred = cp.multiply(c,cp.gmatmul(theta.T.value, inp))
    lstsq_val_preds.append(pred.value)
```

## Fitting

```python
A_param = cp.Parameter(shape=(m, n))
c_param = cp.Parameter(pos=True, shape=(m,))
x_slack = cp.Variable(pos=True, shape=(n,))
x_param = cp.Parameter(pos=True, shape=(n,))
y = cp.Variable(pos=True, shape=(m,))

prediction = cp.multiply(c_param, cp.gmatmul(A_param, x_slack))
objective_fn = cp.sum(prediction / y + y / prediction)
constraints = [x_slack == x_param]
for i in range(m-1):
    constraints += [y[i] <= y[i+1]]
problem = cp.Problem(cp.Minimize(objective_fn), constraints)
problem.is_dgp(dpp=True)
```

    True

```python
A_param.value = np.random.randn(m, n)
x_param.value = np.abs(np.random.randn(n))
c_param.value = np.abs(np.random.randn(m))

layer = CvxpyLayer(problem, parameters=[A_param, c_param, x_param], variables=[y], gp=True)
```

```python
torch.random.manual_seed(1)
A_tch = torch.tensor(theta.T.value)
A_tch.requires_grad_(True)
c_tch = torch.tensor(np.squeeze(np.exp(log_c.value)))
c_tch.requires_grad_(True)
train_losses = []
val_losses = []

lam1 = torch.tensor(1e-1)
lam2 = torch.tensor(1e-1)

opt = torch.optim.SGD([A_tch, c_tch], lr=5e-2)
for epoch in range(10):
    preds = layer(A_tch, c_tch, train_inputs, solver_args={'acceleration_lookback': 0})[0]
    loss = (preds - train_outputs).pow(2).sum(axis=1).mean(axis=0)

    with torch.no_grad():
        val_preds = layer(A_tch, c_tch, val_inputs, solver_args={'acceleration_lookback': 0})[0]
        val_loss = (val_preds - val_outputs).pow(2).sum(axis=1).mean(axis=0)

    print('(epoch {0}) train / val ({1:.4f} / {2:.4f}) '.format(epoch, loss, val_loss))
    train_losses.append(loss.item())
    val_losses.append(val_loss.item())

    opt.zero_grad()
    loss.backward()
    opt.step()
    with torch.no_grad():
        c_tch = torch.max(c_tch, torch.tensor(1e-8))
```

    (epoch 0) train / val (0.0018 / 0.0014)
    (epoch 1) train / val (0.0017 / 0.0014)
    (epoch 2) train / val (0.0017 / 0.0014)
    (epoch 3) train / val (0.0017 / 0.0014)
    (epoch 4) train / val (0.0017 / 0.0014)
    (epoch 5) train / val (0.0017 / 0.0014)
    (epoch 6) train / val (0.0016 / 0.0014)
    (epoch 7) train / val (0.0016 / 0.0014)
    (epoch 8) train / val (0.0016 / 0.0014)
    (epoch 9) train / val (0.0016 / 0.0014)

```python
with torch.no_grad():
    train_preds_tch = layer(A_tch, c_tch, train_inputs)[0]
    train_preds = [t.detach().numpy() for t in train_preds_tch]
```

```python
with torch.no_grad():
    val_preds_tch = layer(A_tch, c_tch, val_inputs)[0]
    val_preds = [t.detach().numpy() for t in val_preds_tch]
```

```python
fig = plt.figure()

i = 0
plt.plot(val_preds[i], label='LLCP', color='teal')
plt.plot(lstsq_val_preds[i], label='least squares', linestyle='--', color='gray')
plt.plot(val_outputs[i], label='true', linestyle='-.', color='orange')
w, h = 8, 3.5
plt.xlabel(r'$i$')
plt.ylabel(r'$y_i$')
plt.legend()
plt.show()
```


# Problem: Reflection unitaries

In the previous two problems, the output of a classical Boolean function is computed into an additional qubit. However, quantum computers can also compute Boolean functions into the phase of a quantum state, which we explore in this problem.

Suppose $f:\{0,1\}^n\rightarrow\{0,1\}$ is a Boolean function. For $\beta \in R$, define an $\textit{n}$-qubit unitary

$$
\large
\begin{equation*}
    U_f(\beta) = \sum_{x\in\{0,1\}^n} e^{i\beta f(x)}|x\rangle \langle x|
\tag{1}
\end{equation*}
$$
When $U_f$ is applied to computational basis state $|x\rangle$ it adds a phase $e^{i\beta}$ if $f(x) = 1$ and does  nothing otherwise.

Your goal in this problem is to implement $U_f$ for a variety of functions with $\textit{n}$= 3

\begin{align}
f_0(x_1x_2x_3)&=0,\quad f_1(x_1x_2x_3)=x_1x_2x_3,\quad f_2(x_1x_2x_3)=x_1x_2x_3+(1-x_1)(1-x_2)(1-x_3),\\\nonumber
f_3(x_1x_2x_3)&=x_1+x_2+x_3+x_1x_2x_3,\quad f_4(x_1x_2x_3)=x_1+x_2+x_3.
\tag{2}
\end{align}

Here, $f_m$ has m solutions to f(x) = 1. For instance,$f_2$ evaluates to 1 on the inputs x = 000 and x = 111.You may use $A \geq 0$ ancilla qubits in your implementation initialized in $∣0^A\rangle$ as long as they are returned to $∣0^A\rangle$ by the end of the computation. Two examples are shown in Fig.2 for n= 2 functions.

The "reflection unitaries" $U_f(\beta)$ are useful in some quantum algorithms. In particular, the famous Grover search algorithm uses these reflections to outperform classical search algorithms for unstructured lists.

    $f_m$ for m = 0

```python
# Importing the qiskit module
from qiskit import *

q0 =    # number of required qubits
a0 =    # number of ancilla qubit
beta = circuit.Parameter('beta') #Parameter for reflection

# Defining Quantum Circuit with the given circuits
def Circuit_0(q,a,beta):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a, 'ancilla')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q, 'input')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q)

    ##### Create you circuit below #########

    ########################################

    # Uncomment to draw quantum circuit
#     display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_3a = Circuit_0(q0,a0,beta)
```

    $f_m$ for m = 1

```python
# Importing the qiskit module
from qiskit import *

q1 =   # number of required qubits
a1 =   # number of ancilla qubit
beta = circuit.Parameter('beta') #Parameter for reflection

# Defining Quantum Circuit with the given circuits
def Circuit_1(q,a,beta):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a, 'ancilla')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q, 'input')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q)

    ##### Create you circuit below #########

    ########################################

    # Uncomment to draw quantum circuit
#     display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_3b = Circuit_1(q1,a1,beta)
```

    $f_m$ for m = 2

```python
# Importing the qiskit module
from qiskit import *

q2 =   # number of required qubits
a2 =   # number of ancilla qubit
beta = circuit.Parameter('beta') #Parameter for reflection

# Defining Quantum Circuit with the given circuits
def Circuit_2(q,a,beta):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a, 'ancilla')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q, 'input')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q)

    ##### Create you circuit below #########

    ########################################

    # Uncomment to draw quantum circuit
#     display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_3c = Circuit_2(q2,a2,beta)
```

    $f_m$ for m = 3

```python
# Importing the qiskit module
from qiskit import *

q3 =   # number of required qubits
a3 =   # number of ancilla qubit
beta = circuit.Parameter('beta') #Parameter for reflection

# Defining Quantum Circuit with the given circuits
def Circuit_3(q,a,beta):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a, 'ancilla')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q, 'input')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q)

    ##### Create you circuit below #########

    ########################################

    # Uncomment to draw quantum circuit
    display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_3d = Circuit_3(q3,a3,beta)
```

    $f_m$ for m = 4

```python
# Importing the qiskit module
from qiskit import *

q4 =   # number of required qubits
a4 =   # number of ancilla qubit
beta = circuit.Parameter('beta') #Parameter for reflection

# Defining Quantum Circuit with the given circuits
def Circuit_4(q, a, beta):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a, 'ancilla')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q, 'input')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q)

    ##### Create you circuit below #########

    ########################################

    # Uncomment to draw quantum circuit
    display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_3e = Circuit_4(q4,a4,beta)
```

### Grader

```python
all_circuits = [qc_3a, qc_3b, qc_3c, qc_3d, qc_3e]
from qc_grader import grade_ex3

grade_ex3(all_circuits)
```

# Circuit verification and Cost Metric

$\textbf{Costing metric:}$ For the $m^{th}$ circuit, let $G_m$ be the number of 2-qubit gates, $D_m$ the depth,  and $A_m$ the number of ancillas. Your solution quality will be judged on the following cost metric:
$$\text{Cost} = \sum_{m=0}^4 10G_m + D_m(n+A_m)$$ .
The reasoning behind this metric relates to the error rates in quantum circuits. Notice that $D_m(n+A_m)$ is roughly the spacetime volume of the circuit (the collective number of timesteps qubits participating in thecircuit must be kept alive). Also, two-qubit gates are roughly (optimistically) 10 times as likely to fail as idling qubits. Thus, $Cost_n$ is roughly proportional to error rate of all of your circuits combined.

### Submission for evaluation

```python
## Uncomment and run the following code to submit
# from qc_grader import submit_ex3

# submit_ex3(all_circuits)
```

Exercise 3/ICPC Question 3 Notebook.ipynb


```python
# https://colab.research.google.com/github/kassbohm/tm-snippets/blob/master/ipynb/TM_1/7_SG/1.7.W.ipynb

from sympy.physics.units import *
from sympy import *

# Given symbols:
q, b, beta = var("q, b, beta")

# Given in terms of these symbols:
h = beta * b

x = var("x")
z = 4 * beta/b *x**2

# --- a ---
zp = diff(z,x)
pprint("\n\nz':")
pprint(zp)

# --- b ---
pprint("\n\nSupport reactions Ah, Av, H:")
Ah, Av, H = var("Ah, Av, H")

R = q*b/2
eq1 = Eq(H, Ah)
eq2 = Eq(Av, R)
eq3 = Eq(h*H, b/4*R)

sol=solve([eq1, eq2, eq3], [Ah, Av, H])
pprint(sol)
H = sol[H]

# --- c ---
pprint("\n\nSection Forces/Moment N, Q, M:")
phi = atan(zp)
Rx = q*x
c, s = cos(phi), sin(phi)

N, Q, M = var("N, Q, M")

eq1 = Eq(N*c - Q*s - H)
eq2 = Eq(Q*c + N*s - Rx)
eq3 = Eq(Rx*x/2 - H*z - M)

sol = solve([eq1, eq2, eq3], [N, Q, M])
pprint(sol)
N = sol[N]

# --- d ---
pprint("\n\nN / H:")
pprint(N / H)

pprint("\n\nN / R:")
pprint(N / R)

# z':
# 8⋅β⋅x
# ─────
#   b
#
#
# Support reactions Ah, Av, H:
# ⎧    b⋅q      b⋅q     b⋅q⎫
# ⎨Ah: ───, Av: ───, H: ───⎬
# ⎩    8⋅β       2      8⋅β⎭
#
#
# Section Forces/Moment N, Q, M:
# ⎧                   ______________      ⎫
# ⎪                  ╱         2  2       ⎪
# ⎪                 ╱      64⋅β ⋅x        ⎪
# ⎪         b⋅q⋅   ╱   1 + ────────       ⎪
# ⎨               ╱            2          ⎬
# ⎪             ╲╱            b           ⎪
# ⎪M: 0, N: ────────────────────────, Q: 0⎪
# ⎪                   8⋅β                 ⎪
# ⎩                                       ⎭
#
#
# N / H:
#       ______________
#      ╱         2  2
#     ╱      64⋅β ⋅x
#    ╱   1 + ────────
#   ╱            2
# ╲╱            b
#
#
# N / R:
#       ______________
#      ╱         2  2
#     ╱      64⋅β ⋅x
#    ╱   1 + ────────
#   ╱            2
# ╲╱            b
# ────────────────────
#         4⋅β

```