```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.linear_model import LinearRegression

plt.style.use('seaborn')
```

# Gradient Descent

Imagine a blindfolded person sitting at the bottom of a hill (or anywhere on the hill really). Their goal is to get to the top of the hill. How would they do it?

* detect the slope of the hill that they're standing on
* take a step in an upwards direction
* repeat until slope feels flat

**Gradient Descent** is an algorithm that is used for finding the optimal values of a function (maximum or minimum). Although we described our hiker man as if he was trying to find a maximum value, Gradient Descent is typically used to find minimum values of a function. Let's learn a little more about how Gradient Descent uses an iterative mathematic approach (like our hiker) to find a function's minimum.

# "Hypothesis" Functions

In Linear Regression we estimate parameters $\beta_0, \beta_1, \ldots , \beta_n$. and plug them into our model. This version of our model with real coefficient values is a Hypothesis Function. However, it is not a requirement of hypothesis functions that they use the estimated parameters. They could contain any real values for the betas and still be a hypothesis function. Lets look at some examples. We'll go back to our ice cream sales example for some easy data to demonstrate with.

```python
# Read Data
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/'
                 'datasets/master/Ice_Cream_Sales.csv')

# Creat Scatterplot
plt.scatter(df.Farenheit, df.Dollars)

X = df[['Farenheit']].values
y = df[['Dollars']].values

# Create Hypothesis Function Lines
for theta_0, theta_1 in zip([-500, 0, 1000], [30, 10, -10]):
    plt.plot(X, [theta_0 + theta_1*x for x in X])

# Graph axes and title
plt.ylabel("Dollars")
plt.xlabel("Farenheit")
plt.title("Hypothesis Functions")

# Plot Hypothesis Functions
plt.show()
```

The above are all Hypothesis functions, they don't necessarily fit our data the best, but they are created using the same model, they just have different (arbitrary) parameters for $\beta_0$ and $\beta_1$. We'll represent hypothesis functions generally as $h(x)$.

The above hypotheis functions all follow the form:

\begin{align}
  h(x) = \theta_0 + \theta_1X
\end{align}

Basically we're assuming that the function that best fits the data is linear (a line) and hypothesizing the $\theta$ values that define that line.

## Given the hypothesis functions below, which is closest to the optimum?

How do you know? How do you measure the quality of a hypothesis?

```python
X = df[['Farenheit']].values
y = df[['Dollars']].values

# Creat Scatterplot
plt.scatter(X, y)

# Create Hypothesis Function Lines
for theta_0, theta_1 in zip(range(-350, -420, -10),
                            range(10, 26, 2)):
    plt.plot(X, [theta_0 + theta_1*x for x in X])

# Graph axes and title
plt.ylabel("Dollars")
plt.xlabel("Farenheit")
plt.title("Hypothesis Functions")

# Plot Hypothesis Functions
plt.show()
```

How can you show that one of these lines (and by extension its $\theta$ parameters) is closer to the optimal $\beta$ values?

# Cost Functions

Functions that gradient descent is trying to minimize are usually referred to as "cost functions" these functions are also referred to by names like: loss function, error function, or scoring function.

We can measure the quality of our hypotheses using a **cost function**. A cost function compares our hypothesis line to our actual y values and gives it a score based on how well our hypothesis function fits the data. How might we define a cost function to give a score to how well our hypotheses fit the data? We would find the sum of the squared error (or residuals) between our fitted values and our datapoints:

\begin{align}
SSE = \sum_{i=1}^{n}(y_i - \hat{y})^2
\end{align}

We might rewrite our SSE equation in terms of our hypothesis functions instead of our already optimized function like so:

\begin{align}
SSE = \sum_{i=1}^{n}(y_i - h(x_i))^2
\end{align}

The goal is to find the values for $\theta_0$ and $\theta_1$ (contained within $h(x)$) that minimize this cost function ("minimize the sum of the squared residuals").

## What does this cost function look like?

```python
# Define a function that can calculate SSE for any hypothesis function
def h_x(theta_0, theta_1, x):
    return theta_0 + theta_1 * x

def SSE(theta_0, theta_1, y, X, h_func=h_x):
    squared_residuals = []
    for x_i, y_i in zip(X, y):
        h_y = h_func(theta_0, theta_1, x_i)
        squared_residuals.append((h_y - y_i) ** 2)

    return sum(squared_residuals)

# Investigate various slopes
theta_0 = -600
theta_1s = list(range(17, 35, 2))
SSEs = []
for theta_1 in theta_1s:
    SSE_i = SSE(theta_0, theta_1, y, X)
    SSEs.append(SSE_i)

plt.scatter(theta_1s, SSEs)
plt.title("Cost Function h(x) holding theta_0 fixed")
plt.xlabel('theta_1')
plt.ylabel('SSE')
```

This parabolic shape is due to varying only the slope (theta_1) while holding the intercept constant. If we varied both values we would then have a mapping of a higher dimension. What would our cost function look like if we varied both $\theta_0$ and $\theta_1$? Something like this:

Where the x axis is $\theta_0$ the z axis is $\theta_1$ and the y axis is our $SSE(\theta_0, \theta_1)$

# Using Gradient Descent to Minimize Cost Functions

One use of gradient descent is to minimize cost functions. To our blindfolded hiker, that might look something like this:

If we took our 3D plot and turned it into a contour plot his travels might look something like this:

We'll first talk about using Gradient Descent to minimize an arbitrary cost function and then later we'll implement it specifically on our linear regression example.

First lets talk about the equation for gradient descent:

- $J$ represents our cost function.
- **gradient** is the derivative term in front of $J$ (the fraction part). In 2D space this is the "slope".
- $\alpha$ is our "Learning Rate" (how big of steps that we we initially take)
- $\theta_j$ represents our current position. So all of the RHS (right hand side) of this equation is calculated all at once and is then used to replace our current position with the new updated position.

1. Start off with initial guesses for $\theta_0$ and $\theta_1$. (typically between 0 and 1)
2. Change $\theta_0$ and $\theta_1$ a little bit (according to the above equation) and reevaluate our cost function.

Notice that our step size (alpha) is multiplied by our gradient (slope) such that we take smaller and smaller steps as the graph gets flatter (closer to a minimum hopefully).

# Optimizing The Learning Rate

This is part of a process that we call hyperparameter tuning. Hyperparameter tuning in its most crude form basically means try out different features of your parameters (like learning rate) and see what happens to your results. Keep on iterating until you converge on an optimal learning rate. (Maybe we can implement gradient descent to find the most optimal learning rate for our gradient descent. 🤔)

You can read up on different techniques for choosing an appropriate learning rate. We will explore more sophisticated Hyperparameter Tuning techniques later on in the class.

You'll usually be fine with a pretty small learning rate (.01 or .02) when you're solving linear regression using Gradient Descent (as long as you use a sufficient number of iterations)

# Gradient Descent Implementation for Bivariate Linear Regression

**X matrix must be standardized**

```python
# Read Data
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/'
                 'datasets/master/Ice_Cream_Sales.csv')

# Create X matrix and y vector
X = df[['Farenheit']].values
y = df['Dollars'].values

# z-standardize - if we don't do this our GD stuff will break
X = (X - X.mean()) / X.std()

# Matrix version makes cool linalg tricks possible
X_linalg = np.c_[np.ones(X.shape[0]), X]

print(X_linalg)
```

    [[ 1.         -1.61957156]
     [ 1.         -1.49352   ]
     [ 1.         -0.98931376]
     [ 1.         -0.65317627]
     [ 1.          0.10313309]
     [ 1.          0.31321903]
     [ 1.          0.43927059]
     [ 1.          0.64935652]
     [ 1.          0.77540808]
     [ 1.          0.81742527]
     [ 1.          1.65776901]]

### SciKit-Learn

```python
# Fit Model
model = LinearRegression()
model.fit(X, y)

# Assign coefficient and intercept to variables (beta_1, and beta_0)
beta_1 = model.coef_[0]
beta_0 = model.intercept_

print("beta_1:", beta_1)
print("beta_0:", beta_0)
```

    beta_1: 587.5808413073934
    beta_0: 1145.4545454545455

```python
# Plot the points
plt.scatter(X, y)

# Plot the line of best fit
y_hat = [beta_1*x + beta_0 for x in X]
plt.plot(X,y_hat)

# Add labels and title
plt.ylabel("Dollars")
plt.xlabel("Standardized Farenheit")
plt.title("Ice Cream Sales and Temperature")
plt.show()
```

### Minimal Gradient-Descent Implementation

```python
np.random.seed(42)

def gradient_descent(x, y, theta, iterations, alpha):
    n = y.size
    for i in range(iterations):
        prediction = np.dot(x, theta)
        error = prediction - y
        theta = theta - (alpha * (1/n)*np.dot(x.T, error))
    return theta

final_theta = gradient_descent(X_linalg, y,
                               theta=[0, 0], iterations=2000, alpha=0.02)
print('theta_0:', final_theta[0], 'theta_1:', final_theta[1], sep='\n')
```

    theta_0:
    1145.45454545454
    theta_1:
    587.5808413073904

```python
# Implementation with saved cost and theta values for graphing costs and animations
def gradient_descent(x, y, theta, iterations, alpha):
    n = x.shape[0]
    past_costs = []
    for i in range(iterations):
        prediction = np.dot(X_linalg, theta)
        error = prediction - y
        theta = theta - (alpha * (1/n)*np.dot(X_linalg.T, error))
        cost = np.dot(error.T, error)
        past_costs.append(cost)
    return theta, past_costs

final_theta, past_costs = gradient_descent(X_linalg, y, theta, iterations, alpha)
print('theta_0:', final_theta[0], 'theta_1:', final_theta[1], sep='\n')

plt.plot(past_costs)
plt.title('Cost Reduction by Iteration')
plt.xlabel('No. of iterations')
plt.ylabel('Cost')
plt.show()
```

## Gradient Descent *is* Machine Learning

What does Gradient Descent do? It finds model parameters that minimize a cost function. (That get the best results!) This cuts to the very heart of what machine learning is: solving uniquely specified optimization problems. I hope you can see how this iterative process of seeing an optimum is not just good for machines, but can be an elegant way of describing learning in general. For both humans and computers!

## Improvements to Gradient Descent

There are **many** Gradient-Descent based optimizers out there that seek improve upon the "vanilla" gradient descent that we have implemented here.

What if we had a cost function that wasn't convex? What if our cost function had multiple local minima?

But what if we had initialized our theta values just slightly differently. Where would the gradient have taken us?

Hopefully these two images help you see how with more complex cost functions it can be difficult to find the global minimum and even slightly different initial values can lead to completely different results. The question about how to overcome these challenges has lead to a plethora of gradient-descent based optimizers all seeking to overcome these challenges.

If you feel like you have a good grasp of what I'm calling "vanilla" gradient descent then start delving into understanding these different optimizers:

[Overview of GD-based optimizers](http://ruder.io/optimizing-gradient-descent/)

[Siraj Raval - Evolution of Gradient Descent-Based Optimizers](https://youtu.be/nhqo0u1a6fw)


# Mean-Variance Optimization

MPT solves for the optimal portfolio weights to minimize volatility for a given expected return, or maximize returns for a given level of volatility. The key requisite input are expected asset returns, standard deviations, and the covariance matrix.

Diversification works because the variance of portfolio returns depends on the covariance of the assets and can be reduced below the weighted average of the asset variances by including assets with less than perfect correlation. In particular, given a vector, ω, of portfolio weights and the covariance matrix, $\Sigma$, the portfolio variance, $\sigma_{\text{PF}}$ is defined as:
$$\sigma_{\text{PF}}=\omega^T\Sigma\omega$$

Markowitz showed that the problem of maximizing the expected portfolio return subject to a target risk has an equivalent dual representation of minimizing portfolio risk subject to a target expected return level, $μ_{PF}$. Hence, the optimization problem becomes:
$$
\begin{align}
\min_\omega & \quad\quad\sigma^2_{\text{PF}}= \omega^T\Sigma\omega\\
\text{s.t.} &\quad\quad \mu_{\text{PF}}= \omega^T\mu\\
&\quad\quad \lVert\omega\rVert =1
\end{align}
$$

We can calculate an efficient frontier using `scipy.optimize.minimize` and the historical estimates for asset returns, standard deviations, and the covariance matrix.

## Imports & Settings

```python
import warnings
warnings.filterwarnings('ignore')
```

```python
%matplotlib inline

import pandas as pd
import numpy as np
from numpy.random import random, uniform, dirichlet, choice
from numpy.linalg import inv

from scipy.optimize import minimize

import pandas_datareader.data as web
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter
import seaborn as sns
```

```python
sns.set_style('whitegrid')
np.random.seed(42)
```

```python
cmap = sns.diverging_palette(10, 240, n=9, as_cmap=True)
```

## Prepare Data

We select historical data for tickers included in the S&P500 (according to Wikipedia) from 1998-2017.

```python
with pd.HDFStore('../data/assets.h5') as store:
    sp500_stocks = store['sp500/stocks']
```

```python
sp500_stocks.head()
```

      name
      gics_sector
      gics_sub_industry
      location
      first_added
      cik
      founded

      ticker

      MMM
      3M Company
      Industrials
      Industrial Conglomerates
      St. Paul, Minnesota
      1976-08-09
      66740
      1902

      ABT
      Abbott Laboratories
      Health Care
      Health Care Equipment
      North Chicago, Illinois
      1964-03-31
      1800
      1888

      ABBV
      AbbVie Inc.
      Health Care
      Pharmaceuticals
      North Chicago, Illinois
      2012-12-31
      1551152
      2013 (1888)

      ABMD
      ABIOMED Inc
      Health Care
      Health Care Equipment
      Danvers, Massachusetts
      2018-05-31
      815094
      1981

      ACN
      Accenture plc
      Information Technology
      IT Consulting & Other Services
      Dublin, Ireland
      2011-07-06
      1467373
      1989

```python
with pd.HDFStore('../data/assets.h5') as store:
    prices = (store['quandl/wiki/prices']
              .adj_close
              .unstack('ticker')
              .filter(sp500_stocks.index)
              .sample(n=30, axis=1))
```

## Compute Inputs

### Compute Returns

```python
start = 2008
end = 2017
```

Create month-end monthly returns and drop dates that have no observations:

```python
weekly_returns = prices.loc[f'{start}':f'{end}'].resample('W').last().pct_change().dropna(how='all')
weekly_returns = weekly_returns.dropna(axis=1)
weekly_returns.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 521 entries, 2008-01-13 to 2017-12-31
    Freq: W-SUN
    Data columns (total 25 columns):
     #   Column  Non-Null Count  Dtype
    ---  ------  --------------  -----
     0   WYNN    521 non-null    float64
     1   FIS     521 non-null    float64
     2   VLO     521 non-null    float64
     3   MRK     521 non-null    float64
     4   DTE     521 non-null    float64
     5   REGN    521 non-null    float64
     6   TEL     521 non-null    float64
     7   YUM     521 non-null    float64
     8   HST     521 non-null    float64
     9   AES     521 non-null    float64
     10  XOM     521 non-null    float64
     11  NEE     521 non-null    float64
     12  HAL     521 non-null    float64
     13  TJX     521 non-null    float64
     14  AXP     521 non-null    float64
     15  IR      521 non-null    float64
     16  CERN    521 non-null    float64
     17  IVZ     521 non-null    float64
     18  XLNX    521 non-null    float64
     19  LOW     521 non-null    float64
     20  WEC     521 non-null    float64
     21  ETFC    521 non-null    float64
     22  XEL     521 non-null    float64
     23  CDNS    521 non-null    float64
     24  BXP     521 non-null    float64
    dtypes: float64(25)
    memory usage: 105.8 KB

### Set  Parameters

```python
stocks = weekly_returns.columns
```

```python
n_obs, n_assets = weekly_returns.shape
n_assets, n_obs
```

    (25, 521)

```python
NUM_PF = 100000 # no of portfolios to simulate
```

```python
x0 = uniform(0, 1, n_assets)
x0 /= np.sum(np.abs(x0))
```

### Annualization Factor

```python
periods_per_year = round(weekly_returns.resample('A').size().mean())
periods_per_year
```

    52.0

### Compute Mean Returns, Covariance and Precision Matrix

```python
mean_returns = weekly_returns.mean()
cov_matrix = weekly_returns.cov()
```

The precision matrix is the inverse of the covariance matrix:

```python
precision_matrix = pd.DataFrame(inv(cov_matrix), index=stocks, columns=stocks)
```

### Risk-Free Rate

Load historical 10-year Treasury rate:

```python
treasury_10yr_monthly = (web.DataReader('DGS10', 'fred', start, end)
                         .resample('M')
                         .last()
                         .div(periods_per_year)
                         .div(100)
                         .squeeze())
```

```python
rf_rate = treasury_10yr_monthly.mean()
```

## Simulate Random Portfolios

The simulation generates random weights using the Dirichlet distribution, and computes the mean, standard deviation, and SR for each sample portfolio using the historical return data:

```python
def simulate_portfolios(mean_ret, cov, rf_rate=rf_rate, short=True):
    alpha = np.full(shape=n_assets, fill_value=.05)
    weights = dirichlet(alpha=alpha, size=NUM_PF)
    if short:
        weights *= choice([-1, 1], size=weights.shape)

    returns = weights @ mean_ret.values + 1
    returns = returns ** periods_per_year - 1
    std = (weights @ weekly_returns.T).std(1)
    std *= np.sqrt(periods_per_year)
    sharpe = (returns - rf_rate) / std
    return pd.DataFrame({'Annualized Standard Deviation': std,
                         'Annualized Returns': returns,
                         'Sharpe Ratio': sharpe}), weights
```

```python
simul_perf, simul_wt = simulate_portfolios(mean_returns, cov_matrix, short=False)
```

```python
df = pd.DataFrame(simul_wt)
df.describe()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24

      count
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      ...
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000
      100000.000

      mean
      0.040
      0.040
      0.040
      0.040
      0.041
      0.040
      0.040
      0.040
      0.040
      0.040
      ...
      0.039
      0.039
      0.040
      0.040
      0.041
      0.040
      0.040
      0.040
      0.040
      0.040

      std
      0.131
      0.131
      0.130
      0.131
      0.133
      0.131
      0.130
      0.131
      0.132
      0.129
      ...
      0.130
      0.128
      0.130
      0.132
      0.132
      0.131
      0.131
      0.131
      0.132
      0.130

      min
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000

      25%
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000

      50%
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      ...
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000
      0.000

      75%
      0.002
      0.002
      0.002
      0.002
      0.002
      0.002
      0.003
      0.002
      0.002
      0.002
      ...
      0.002
      0.002
      0.002
      0.002
      0.003
      0.002
      0.002
      0.002
      0.002
      0.002

      max
      0.998
      0.998
      0.999
      1.000
      1.000
      0.999
      0.998
      0.999
      0.999
      1.000
      ...
      1.000
      1.000
      0.998
      0.999
      0.998
      0.998
      0.999
      0.999
      0.999
      0.999

8 rows × 25 columns

### Plot Simulated Portfolios

```python
ax = simul_perf.plot.scatter(x=0, y=1, c=2, cmap='Blues',
                             alpha=0.5, figsize=(14, 9), colorbar=True,
                             title=f'{NUM_PF:,d} Simulated Portfolios')

max_sharpe_idx = simul_perf.iloc[:, 2].idxmax()
sd, r = simul_perf.iloc[max_sharpe_idx, :2].values
print(f'Max Sharpe: {sd:.2%}, {r:.2%}')
ax.scatter(sd, r, marker='*', color='darkblue', s=500, label='Max. Sharpe Ratio')

min_vol_idx = simul_perf.iloc[:, 0].idxmin()
sd, r = simul_perf.iloc[min_vol_idx, :2].values
ax.scatter(sd, r, marker='*', color='green', s=500, label='Min Volatility')
plt.legend(labelspacing=1, loc='upper left')
plt.tight_layout()
```

## Compute Annualize PF Performance

Now we'll set up the quadratic optimization problem to solve for the minimum standard deviation for a given return or the maximum SR.

To this end, define the functions that measure the key metrics:

```python
def portfolio_std(wt, rt=None, cov=None):
    """Annualized PF standard deviation"""
    return np.sqrt(wt @ cov @ wt * periods_per_year)
```

```python
def portfolio_returns(wt, rt=None, cov=None):
    """Annualized PF returns"""
    return (wt @ rt + 1) ** periods_per_year - 1
```

```python
def portfolio_performance(wt, rt, cov):
    """Annualized PF returns & standard deviation"""
    r = portfolio_returns(wt, rt=rt)
    sd = portfolio_std(wt, cov=cov)
    return r, sd
```

## Max Sharpe PF

Define a target function that represents the negative SR for scipy's minimize function to optimize, given the constraints that the weights are bounded by [-1, 1], if short trading is permitted, and [0, 1] otherwise, and sum to one in absolute terms.

```python
def neg_sharpe_ratio(weights, mean_ret, cov):
    r, sd = portfolio_performance(weights, mean_ret, cov)
    return -(r - rf_rate) / sd
```

```python
weight_constraint = {'type': 'eq',
                     'fun': lambda x: np.sum(np.abs(x))-1}
```

```python
def max_sharpe_ratio(mean_ret, cov, short=False):
    return minimize(fun=neg_sharpe_ratio,
                    x0=x0,
                    args=(mean_ret, cov),
                    method='SLSQP',
                    bounds=((-1 if short else 0, 1),) * n_assets,
                    constraints=weight_constraint,
                    options={'tol':1e-10, 'maxiter':1e4})
```

## Compute Efficient Frontier

The solution requires iterating over ranges of acceptable values to identify optimal risk-return combinations

```python
def min_vol_target(mean_ret, cov, target, short=False):

    def ret_(wt):
        return portfolio_returns(wt, mean_ret)

    constraints = [{'type': 'eq',
                    'fun': lambda x: ret_(x) - target},
                   weight_constraint]

    bounds = ((-1 if short else 0, 1),) * n_assets
    return minimize(portfolio_std,
                    x0=x0,
                    args=(mean_ret, cov),
                    method='SLSQP',
                    bounds=bounds,
                    constraints=constraints,
                    options={'tol': 1e-10, 'maxiter': 1e4})
```

The mean-variance frontier relies on in-sample, backward-looking optimization. In practice, portfolio optimization requires forward-looking input. Unfortunately, expected returns are notoriously difficult to estimate accurately.

The covariance matrix can be estimated somewhat more reliably, which has given rise to several alternative approaches. However, covariance matrices with correlated assets pose computational challenges since the optimization problem requires inverting the matrix. The high condition number induces numerical instability, which in turn gives rise to Markovitz curse: the more diversification is required (by correlated investment opportunities), the more unreliable the weights produced by the algorithm.

## Min Volatility Portfolio

```python
def min_vol(mean_ret, cov, short=False):
    bounds = ((-1 if short else 0, 1),) * n_assets

    return minimize(fun=portfolio_std,
                    x0=x0,
                    args=(mean_ret, cov),
                    method='SLSQP',
                    bounds=bounds,
                    constraints=weight_constraint,
                    options={'tol': 1e-10, 'maxiter': 1e4})
```

```python
def efficient_frontier(mean_ret, cov, ret_range, short=False):
    return [min_vol_target(mean_ret, cov, ret) for ret in ret_range]
```

## Run Calculation

### Get random PF

```python
simul_perf, simul_wt = simulate_portfolios(mean_returns, cov_matrix, short=False)
```

```python
print(simul_perf.describe())
```

           Annualized Standard Deviation  Annualized Returns  Sharpe Ratio
    count                     100000.000          100000.000    100000.000
    mean                           0.273               0.177         0.676
    std                            0.077               0.055         0.214
    min                            0.140               0.020         0.056
    25%                            0.218               0.148         0.511
    50%                            0.261               0.174         0.666
    75%                            0.314               0.202         0.834
    max                            0.655               0.466         1.333

```python
simul_max_sharpe = simul_perf.iloc[:, 2].idxmax()
simul_perf.iloc[simul_max_sharpe]
```

    Annualized Standard Deviation   0.166
    Annualized Returns              0.222
    Sharpe Ratio                    1.333
    Name: 67110, dtype: float64

### Get Max Sharpe PF

```python
max_sharpe_pf = max_sharpe_ratio(mean_returns, cov_matrix, short=False)
max_sharpe_perf = portfolio_performance(max_sharpe_pf.x, mean_returns, cov_matrix)
```

```python
r, sd = max_sharpe_perf
pd.Series({'ret': r, 'sd': sd, 'sr': (r-rf_rate)/sd})
```

    ret   0.229
    sd    0.166
    sr    1.372
    dtype: float64

From simulated pf data

### Get Min Vol PF

```python
min_vol_pf = min_vol(mean_returns, cov_matrix, short=False)
min_vol_perf = portfolio_performance(min_vol_pf.x, mean_returns, cov_matrix)
```

### Get Efficent PFs

```python
ret_range = np.linspace(simul_perf.iloc[:, 1].min(), simul_perf.iloc[:, 1].max(), 50)
eff_pf = efficient_frontier(mean_returns, cov_matrix, ret_range, short=True)
eff_pf = pd.Series(dict(zip([p['fun'] for p in eff_pf], ret_range)))
```

### Plot Result

The simulation yields a subset of the feasible portfolios, and the efficient frontier identifies the optimal in-sample return-risk combinations that were achievable given historic data.

The below figure shows the result including the minimum variance portfolio and the portfolio that maximizes the SR and several portfolios produce by alternative optimization strategies. The efficient frontier

```python
fig, ax = plt.subplots()
simul_perf.plot.scatter(x=0, y=1, c=2, ax=ax, cmap='Blues',alpha=0.25,
                        figsize=(14, 9), colorbar=True)

eff_pf[eff_pf.index.min():].plot(linestyle='--', lw=2, ax=ax, c='k',
                                 label='Efficient Frontier')

r, sd = max_sharpe_perf
ax.scatter(sd, r, marker='*', color='k', s=500, label='Max Sharpe Ratio PF')

r, sd = min_vol_perf
ax.scatter(sd, r, marker='v', color='k', s=200, label='Min Volatility PF')

kelly_wt = precision_matrix.dot(mean_returns).clip(lower=0).values
kelly_wt /= np.sum(np.abs(kelly_wt))
r, sd = portfolio_performance(kelly_wt, mean_returns, cov_matrix)
ax.scatter(sd, r, marker='D', color='k', s=150, label='Kelly PF')

std = weekly_returns.std()
std /= std.sum()
r, sd = portfolio_performance(std, mean_returns, cov_matrix)
ax.scatter(sd, r, marker='X', color='k', s=250, label='Risk Parity PF')

r, sd = portfolio_performance(np.full(n_assets, 1/n_assets), mean_returns, cov_matrix)
ax.scatter(sd, r, marker='o', color='k', s=200, label='1/n PF')

ax.legend(labelspacing=0.8)
ax.set_xlim(0, eff_pf.max()+.4)
ax.set_title('Mean-Variance Efficient Frontier', fontsize=16)
ax.yaxis.set_major_formatter(FuncFormatter(lambda y, _: '{:.0%}'.format(y)))
ax.xaxis.set_major_formatter(FuncFormatter(lambda y, _: '{:.0%}'.format(y)))
sns.despine()
fig.tight_layout();
```


```python
from logicqubit.logic import *
from cmath import *
import numpy as np
import sympy as sp
from scipy.optimize import *
import matplotlib.pyplot as plt
```

    logicqubit-gpu version 1.5.4

```python
def qft(qr):
    for i in range(len(qr)):
        for j in range(i):
            qr[i].CU1(qr[j], pi/float(2**(i-j)))
        qr[i].H()

def iqft(qr): # transformada quântica de Fourier inversa
    for i in range(len(qr)):
        for j in range(i):
            qr[i].CU1(qr[j], -pi/float(2**(i-j)))
        qr[i].H()

def swap(s1, s2):
    s2.CX(s1)
    s1.CX(s2)
    s2.CX(s1)

#def ansatz(q, params):
#    return q.RY(params[0])

def _ansatz(reg, params):
    n_qubits = len(reg)
    depth = n_qubits
    for i in range(depth):
        reg[1].CNOT(reg[0])
        for j in range(n_qubits):
            reg[i].RY(params[j])

def ansatz(reg, params):
    n_qubits = len(reg)
    depth = n_qubits
    for i in range(depth):
        for j in range(n_qubits):
            if(j < n_qubits-1):
                reg[j+1].CNOT(reg[j])
            reg[i].RY(params[j])

def ansatz_2(q1, q2, params):
    q2.CNOT(q1)
    q1.RY(params[0])
    q2.RY(params[1])
    q1.CNOT(q2)
    q1.RY(params[0])
    q2.RY(params[1])
    q2.CNOT(q1)
    q1.RY(params[0])
    q2.RY(params[1])
```

```python
def expectation(params):
    # H = [[1,0,0,0],[0,-1,0,0],[0,0,-5,0],[0,0,0,1]]
    # <psi|H|psi> = <psi|00><00|psi> - <psi|01><01|psi> - 5*<psi|10><10|psi> + <psi|11><11|psi>
    logicQuBit  = LogicQuBit(2, first_left=True)
    #reg = QubitRegister(2)
    q1 = Qubit()
    q2 = Qubit()

    #_ansatz(reg.getQubits(), params)
    #ansatz([q2,q1], params)
    ansatz_2(q1,q2,params)

    res = logicQuBit.Measure([q1,q2])

    return -5*res[0]+res[1]-4*res[2]-10*res[3]
```

```python
expectation([0,0])
```

    -5.0

```python
minimum = minimize(expectation, [0,0], method='Nelder-Mead')
print(minimum)
```

     final_simplex: (array([[1.21267393, 0.32226075],
           [1.21263944, 0.32234993],
           [1.21262322, 0.32233524]]), array([-7.5783574, -7.5783574, -7.5783574]))
               fun: -7.578357400998107
           message: 'Optimization terminated successfully.'
              nfev: 124
               nit: 66
            status: 0
           success: True
                 x: array([1.21267393, 0.32226075])

```python

```


# The standard deb model

The standard DEB model, in the energy formulation, contains four dynamic state variables: reserve energy $E$, structure volume $V$, maturity energy $E_M$ and reproduction buffer energy $E_R$:

\begin{eqnarray}
    \frac{dE}{dt} &=& \dot{p}_A - \dot{p}_C\\
    \frac{dV}{dt} &=& \frac{\dot{p}_G}{[E_G]}\\
    \frac{dE_H}{dt} &=& \dot{p}_R (1 - H(E_H - E^p_H))\\
    \frac{dE_R}{dt} &=& \dot{p}_R H(E_H - E^p_H)
\end{eqnarray}

In this coupled set of ODEs, four fluxes appear:

\begin{eqnarray}
    \dot{p}_A &=& f(X)\{\dot{p}_{Am}\}V^{2/3}\\
    \dot{p}_C &=& E \left( \frac{[E_G]\dot{v}V^{2/3} + \dot{p}_S}{\kappa E + [E_G]V} \right)\\
    \dot{p}_G &=& \kappa\dot{p}_C - \dot{p}_S\\
    \dot{p}_R &=& (1-\kappa)\dot{p}_C - \dot{p}_J\\
\end{eqnarray}

Here, two additional loss fluxes appear, somatic maintenance $\dot{p}_S$ and maturity maintenance $\dot{p}_J$:

\begin{eqnarray}
    \dot{p}_S &=& [\dot{p}_M]V - \{\dot{p}_T\}V^{2/3}\\
    \dot{p}_J &=& \dot{k}_JE_H
\end{eqnarray}

```python
%matplotlib inline

#Import packages we need
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

#Import deb modules
import deb
import deb_compound_pars
import deb_aux

#Configure plotting
sns.set_context('notebook')
sns.set_style('white')
```

## The 12 primary parameters
Also include temperature parameters here for now

```python
deb.get_deb_params_pandas()
```

      Min
      Max
      Value
      Dimension
      Units
      Description

      Fm
      2.220446e-16
      NaN
      6.500000
      l L**-2 t**-1

      Specific searching rate

      kappaX
      2.220446e-16
      1.0
      0.800000
      -

      Assimilation efficiency

      pAm
      2.220446e-16
      NaN
      530.000000
      e L**-2 t**-1

      max specific assimilation rate

      v
      2.220446e-16
      NaN
      0.020000
      L t**-1
      cm/d
      Energy conductance

      kappa
      2.220446e-16
      1.0
      0.800000
      -

      Allocation fraction to soma

      kappaR
      2.220446e-16
      1.0
      0.950000
      -

      Reproduction efficiency

      pM
      2.220446e-16
      NaN
      18.000000
      e L**-3 t**-1
      J/d/cm**3
      Volume-specific somatic maintenance cost

      pT
      0.000000e+00
      NaN
      0.000000
      e L**-1 t**-1

      Surface-specific somatic maintenance cost

      kJ
      2.220446e-16
      NaN
      0.002000
      t**-1

      Maturity maintenance rate coefficient

      EG
      2.220446e-16
      NaN
      4184.000000
      e L**-3

      Specific cost for structure

      EbH
      2.220446e-16
      NaN
      0.000001
      e

      Maturity at birth

      EpH
      2.220446e-16
      NaN
      843.600000
      e

      Maturity at puberty

      TA
      2.220446e-16
      NaN
      6000.000000
      T
      K
      Arrhenius temperature

      Ts
      2.220446e-16
      NaN
      293.100000
      T
      K
      Reference temperature

## Forward solve test
Solve the standard DEB model equations (four state variables)

```python
#Instantiate DEB model with constant food level (must be a function though)
f = lambda t: 1.0
dm = deb.DEBStandard({'f': f})

#Integrate the DEB equations, 20000 days
y0 = [1000, 1, 0, 0]               # Initial state
t = np.linspace(0, 20000, 100)     # Time points
dm.predict(y0, t)                  # Predict

#Plot the state variable dynamics. Dashed line indicated implied maximum structural length.
fig = dm.plot_state()
```

## Real-world predictions with auxillary data and equations

Organism physical length [cm]

\begin{equation}
    L_w = \frac{L}{\delta_M} = \frac{\sqrt[3]{V}}{\delta_M}
\end{equation}

Organism total (dry) weight [g] is the sum of reserve, structure (and reproduction buffer) weights:

\begin{equation}
    W_W = w_VM_V + w_EM_E
\end{equation}

Organism physical volume

\begin{equation}
    V_w = V + (E + E_R)\frac{w_E}{d_E\bar{\mu}_E}
\end{equation}

### Supporting parameter relationships
See pps. 81 - 83 i DEB3.

Structural mass [mol]:

\begin{equation}
    M_V = [M_V]V = \frac{d_V}{w_V}V
\end{equation}

Reserve mass [mol]:

\begin{equation}
    M_E = \frac{E}{\bar{\mu}_E}
\end{equation}

Number of C-atoms per unit of structural volume [mol/cm$^3$]

\begin{equation}
    [M_V] = \frac{d_V}{w_V}
\end{equation}

#### Food function
Functional (food) response:

\begin{equation}
    f(X) = \frac{X}{K+X}
\end{equation}

Half saturation coefficient

\begin{equation}
    K = \frac{\{\dot{J}_{XAm}\}}{\{\dot{F}_m \}} = \frac{\{\dot{p}_{Am} \}}{\kappa_X\{\dot{F}_m \}\bar{\mu}_X}
\end{equation}

### Parameter values that must be specified
These equations requires the specification of three scalar parameters:

| Description | Unit | Symbol | Typical value |
|----------------------------------------|:-------:|:-------------:|:-------------------:|
| Specific chemical potential of reserve | J/mol   | $\bar{\mu}_E$ | 550 000 (addchem.m) |
| Specific chemical potential of food    | J/mol   | $\bar{\mu}_X$ | 525 000 (addchem.m) |
| C-molar weight of water-free structure | g/mol   | $w_V$         | 24.6 (DEB3 ex)      |
| C-molar weight of water-free reserve   | g/mol   | $w_E$         | 23.0 (get_pars2.m)  |
| Specific density of dry structure      | g/cm$^3$| $d_V$         | 0.1                 |
| Specific density of reserve            | g/cm$^3$| $d_V$         | dV (addchem.m)      |
| Shape factor                           | -       | $\delta_M$    | 0.9 (made up)       |

```python
aux = deb_aux.AuxPars(dm)
fig = aux.plot_observables()
plt.tight_layout()
```

## Implied compound parameter

```python
deb_compound_pars.calculate_compound_pars(dm.params)
```

      Min
      Max
      Value
      Dimension
      Unit
      Description

      Em
      0.0
      NaN
      26500.000000
      e L**-3
      J/m
      Max reserve density

      g
      0.0
      NaN
      0.197358
      -
      -
      Energy investment ration

      Lm
      0.0
      NaN
      23.555556
      l
      m
      Maximum structural length

      kM
      0.0
      NaN
      0.004302
      t**-1
      1/d
      Somatic maintenance rate

      kappaG
      0.0
      NaN
      0.800000
      -
      -
      Fraction of growth energy fixed in structure

# Testing food functions

```python
#Periodic food function
flist = [lambda t: 1/2*(1+np.sin(t/2000.)**2),
         lambda t: 0*t + 1.0]

fig, ax = plt.subplots(2, 2, figsize=(8, 8))
ax = ax.flatten()
ax2 = ax[0].twinx()
for f in flist:
    dm = deb.DEBStandard({'f': f})
    y0 = [0, 1, 0, 0]
    t = np.linspace(0, 20000, 100)
    dm.predict(y0, t)
    fig = dm.plot_state(fig=fig)
    ax2.plot(t, [f(ti) for ti in t], '--')

    fig2, ax = plt.subplots(1, 3, figsize=(12, 4))
    aux = deb_aux.AuxPars(dm)
    _ = aux.plot_observables(fig=fig2)
    fig2.tight_layout()

ax2.set_ylim(-1, 1.5)
fig.tight_layout()
```


```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from sympy import *
from scipy.integrate import quad
plt.rcParams.update({'font.size': 16, 'figure.figsize': [8.0, 6.0]})
init_printing()
```

```python
N,V,T,Q,L,q,k,h,m,beta = symbols("N V T Q Lambda q k h m beta", positive=True, real=True)
E,S,F,A = symbols("E S G A", real=True)
```

# Ideal monatomic gas

Here we show how to use `sympy` to symbolically manage expressions, take the derivative etc.

#### Partition function

```python
L = h/sqrt(2*pi*m/beta) # de Broglie wavelength
q = V/L**3              # translational partition function
Q = q**N / factorial(N) # molecular part. func.; independent particles
```

#### Energy

```python
E = -diff(ln(Q), beta)
E.subs(beta, 1/(k*T))
```

#### Free energy (experimental section)

```python
L = symbols('Lambda')
Q = (V/L**3)**N / factorial(N)
F = -k*T*ln(Q)
F = simplify(F)
F
```

```python
F = F.subs( log(factorial(N)), N*log(N)-N ) # stirling's approximation
logcombine(F)
```

# Rotational Partition Function

The following plots the rotational partition function and how each quantum number, $J$, contributions. We use `ipywidgets` to add an interactive slider to control the temperature.
The manual for `ipywidgets` can be found [here](https://ipywidgets.readthedocs.io/en/latest/index.html).

```python
from scipy.constants import h, c, k, N_A
from scipy.misc import derivative

B = 10.591 * 100 # 1/m # rotational wave-number (related to moment of inertia)
J = np.arange(0,15,1)  # quantum numbers (called "l" in Dill's book)

def q(J,beta):
    ''' rotational partition function for a given J '''
    return (2*J+1) * np.exp( -beta*h*c*B*J*(J+1) )

def qrot(beta):
    ''' partition function for "inifinite" number of quantum numbers '''
    J = np.arange(0,500,1)
    return np.sum( q(J,beta) )

def qrotHT(beta):
    ''' partition function in the high temperature limit, i.e. summation --> integral '''
    return 1/(beta*h*c*B)

def plot(T):
    ''' plots partition function at given temperature '''
    beta = 1/(k*T)
    plt.plot(J, q(J,beta).cumsum(), 'b--o', alpha=0.7, lw=2, ms=10,
             label=r'Explicit summation <$E$>={:.2f} kT'
             .format(-derivative( qrot, beta, dx=0.001*beta ) / qrot(beta) * beta) )
    plt.plot( [0,1000], [qrotHT(beta),qrotHT(beta)], 'r-', alpha=0.5, lw=4,
             label=r'High temperature limit <$E$>={:.2f} kT'
             .format(-derivative( qrotHT, beta, dx=0.001*beta ) / qrotHT(beta) * beta))
    plt.xlim( 0, 14 )
    plt.xlabel( 'Quantum number, $J$' )
    plt.ylabel( 'Summed partition function, $q$' )
    plt.legend(loc=0, frameon=False)
    plt.title('Rotational Partition Function, $q_r$')

# this part sets up a temperature slider to control the plot
i = interact(plot, T=(50, 1000), continuous_update=False)
i.widget.children[0].description=r'$T$ (Kelvin)';
```

    interactive(children=(IntSlider(value=525, description='T', max=1000, min=50), Output()), _dom_classes=('widge…

# Two-level system

Here we consider a system where two energy level are possible; the ground state (0) and the first excited state, $\epsilon$. The partition function is

$$
q = 1+e^{-\epsilon / k_BT}
$$

```python
from scipy.constants import h, c, k, N_A # import physical constants

def q(T): return 1+np.exp(-epsilon/(k*T))
def energy(T): return epsilon / (np.exp(epsilon/(k*T)+1) ) # atkins p625 (10th)
def Cv(T): return np.gradient( energy(T)/k, T[1]-T[0] )

fig, ax = plt.subplots()
epsilon=k*4
T=np.arange(0.1, 10, 0.5)
plt.plot(T,q(T), label=r'$q = 1+1 / e^{-\beta \epsilon}$', lw=4, alpha=0.7)
plt.plot(T, energy(T)/k, label=r'$\langle \epsilon \rangle = -\frac{1}{q}\frac{\partial q}{\partial \beta}$',
         lw=4, alpha=0.7)
plt.plot(T, Cv(T), label=r'$C_v=\partial \langle \epsilon \rangle / \partial T$', lw=4, alpha=0.7)
plt.legend(frameon=False, loc=0, fontsize='large', ncol=2)
plt.ylim(0, 2)
plt.xlabel('Temperature');
```

# Bragg-Williams Model for a Two-component System

This example demonstrates how the Bragg-Williams (BW) model predicts the partial pressure of compoments A and B over a liquid mixture of the two. The interaction parameter $\alpha$ includes the relative A-B neighbor interaction, compared to the A-B and B-B interactions, as well as the number of neighbors, $z$. The partial pressure is given by,
$$
p_A = p_A^0 \chi_A e^{\alpha \chi_B^2}
$$

where $p_A^0$ is the vapor pressure of pure A. If either $\alpha$ or the mole fraction, $\chi_B$, of the other component approach zero, ideal behavior, i.e. Rault's law, is obtained:
$$
p_A = p_A^0 \chi_A
$$

```python
chiA = np.arange(0,1.1,0.05) # mole fraction of A
chiB = 1-chiA                # mole fraction of B

def plotbw(alpha=1, p0A=0.7, p0B=0.2):
    plt.plot( chiA, chiA * p0A * np.exp(alpha*chiB**2),'b-', label='Bragg-Williams'  )
    plt.plot( chiA, chiA * p0A, 'b--', label="Rault's law")
    plt.plot( chiA, chiB * p0B * np.exp(alpha*chiA**2),'r-')
    plt.plot( chiA, chiB * p0B, 'r--')
    plt.xlim(0,1)

    plt.xlabel(r'$\chi_A=1-\chi_B$')
    plt.ylabel(r'$p$', color='k')

    plt.annotate(r'$p_A^0$', (1,p0A), color='blue')
    plt.annotate(r'$p_B^0$', (0,p0B), color='red')
    plt.legend(loc=0, frameon=False)

i = interact(plotbw, alpha=(-3,3, 0.5), p0A=(0,1,0.1), p0B=(0,1,0.1), continuous_update=False )

i.widget.children[0].description=r'$\alpha$'
i.widget.children[1].description=r'$p_A^0$'
i.widget.children[2].description=r'$p_B^0$'
```

    interactive(children=(FloatSlider(value=1.0, description='alpha', max=3.0, min=-3.0, step=0.5), FloatSlider(va…

# Equation of State using Generalized van der Waals Theory

This notebook illustrates the generalized van der Waals theory (gvdW) for the equation of state for interacting particles. Based on the lecture notes, _Properties of Molecular Fluids in Equilibrium_ by Sture Nordholm.

### Pair potential

The particles are here assumed to interact via a _Lennard-Jones_ potential,
$$
\beta w(r) =
4\beta \varepsilon_{LJ} \left ( \left ( \frac{\sigma}{r}\right )^{12} - \left ( \frac{\sigma}{r}\right )^{6}\right )
$$

Any potential may in principle be given and must return the energy in units of $k_BT=\beta^{-1}$.

### Interaction parameter

Here we integrate the above pair potential to get the average interaction energy per particle, assuming that the pair correlation function, $g(r)$, can be described by a simple step function, zero when $r<\sigma$, unity otherwise:
$$
\hat{a} = -\frac{1}{2} \int_{\sigma}^{\infty} 4\pi w(r) r^2 dr
$$

In this Notebook we simply do the integration numerically so that we can use arbitrary pair potentials.
From this we calculate the pressure, $p$, versus density, $n$, using,
$$
\beta p_{gvdW} = \frac{1}{v-v_0} - \frac{\hat{a}}{v^2}
$$

where $v=1/n$ and $v_0=2\pi\sigma^3/3$ is the particle volume.
For reference we'll also plot EOS for an ideal system (van 't Hoff), $\beta p_{ideal}=n$,
where $\beta = 1/k_BT$.

```python
def pairpotential(r, eps, sigma): # Lennard-Jones potential
    return 4 * eps * ( (sigma/r)**12 - (sigma/r)**6 )

def Pgvdw(n, eps, sigma):
    a  = -2*pi*quad(lambda r: pairpotential(r, eps, sigma)*r**2, sigma, np.infty, limit=20)[0]
    v0 = 2*pi*sigma**3 / 3
    v  = 1 / n
    return 1/(v-v0) - a/v**2

def Pideal(n): # ideal gas law
    return n

def plot( eps=0, sigma=4.0 ):
    plt.plot(n, Pideal(n),  'k--', label='ideal', lw=2)
    plt.plot(n, Pgvdw(n, eps, sigma),  'r-', label='gvdW', lw=2)
    plt.legend(loc=0, frameon=False)

    plt.xlabel(r'Density, $n$')
    plt.ylabel(r'Pressure, $\beta p$')

n = np.linspace(1e-7, 6e-3, 50)
i = interact(plot, eps=(0.0, 2.0, 0.1), sigma=(0, 5, 0.1), continuous_update=False )

i.widget.children[0].description=r'$\beta\varepsilon_{LJ}$'
i.widget.children[1].description=r'$\sigma_{LJ}$'
```

    interactive(children=(FloatSlider(value=0.0, description='eps', max=2.0), FloatSlider(value=4.0, description='…

## Variation of Dill problem 5.1 (page 91)

Let's assume that each corner (n, s, e, w) are associated with an energy, $\epsilon_i$. Specifically we set an attraction for north ($\beta\epsilon_n=-0.5$) and a repulsion for south ($\beta\epsilon_s = 0.5$).

```python
energy = np.array([-0.5, 0.5, 0.0, 0.0])
p = np.exp(-energy) / np.exp(-energy).sum()
entropy = -(p*np.log(p)).sum()
print("p_i =", p)
print("sum(p_i) =", p.sum())
print("S/k = {:.3f}".format(entropy))
```

    p_i = [0.38745562 0.14253696 0.23500371 0.23500371]
    sum(p_i) = 1.0
    S/k = 1.326

```python

```


Table of Contents

1  Matrix multiplication

2  The @ operator for matrix multiplication

3  Enrichment exercise

4  The transpose of a matrix

>All content is released under Creative Commons Attribution [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) and all source code is released under a [BSD-3 clause license](https://en.wikipedia.org/wiki/BSD_licenses).
>
>Please reuse, remix, revise, and reshare this content in any way, keeping this notice.
>
>**Are you viewing this on jupyter.org?** Then this notebook will be read-only.

>See how you can interactively run the code in this notebook by visiting our [instruction page about Notebooks](https://yint.org/notebooks).

## Matrix multiplication
We saw earlier examples of matrices; they can be considered like spreadsheets containing rows and observations.

But the real power of matrices come from some of the calculations we can do with them. We have already seen addition and subtraction in a [prior notebook on elementwise functions](./). In that notebook we also saw elementwise multiplication and division.

But there is a special matrix operation, called ***matrix multiplication*** which we will consider here.

Let us take a look at an example: we multiply one matrix, called $\mathbf{A}$ that has 4 rows and 2 columns, with a second matrix, called $\mathbf{B}$, which contains 2 rows and 3 columns. The product of $\mathbf{A}$ and $\mathbf{B}$, or also simply written as $\mathbf{AB}$ is yet another matrix, we have called it $\mathbf{X} = \mathbf{AB}$. That matrix $\mathbf{X}$ has 4 rows and 3 columns.

The value in first row and first column of $\mathbf{A}$ is given by the variable $a_{1,1}$; the value in row 1 and column 2 is $a_{1,2}$, and so on. The entry in row $i$ and column $j$ is therefore given by $a_{i,j}$. We use a similar notation for matrix $\mathbf{B}$ and $\mathbf{X}$. You can see these values in the figure shown here. Figure credit: [Wikipedia](https://en.wikipedia.org/wiki/Matrix_multiplication#/media/File:Matrix_multiplication_diagram_2.svg).

$$\begin{eqnarray}\mathbf{A}\mathbf{B} &=& \mathbf{X}\\
\begin{bmatrix}
{a_{1,1}} & {a_{1,2}} \\
{a_{2,1}} & {a_{2,2}} \\
{a_{3,1}} & {a_{3,2}} \\
{a_{4,1}} & {a_{4,2}} \\
\end{bmatrix}
\begin{bmatrix}
{b_{1,1}} & {b_{1,2}} & {b_{1,3}} \\
{b_{2,1}} & {b_{2,2}} & {b_{2,3}} \\
\end{bmatrix}
&=& \begin{bmatrix}
x_{1,1} & x_{1,2} & x_{1,3} \\
x_{2,1} & x_{2,2} & x_{2,3} \\
x_{3,1} & x_{3,2} & x_{3,3} \\
x_{4,1} & x_{4,2} & x_{4,3} \\
\end{bmatrix}\end{eqnarray}
$$

The values at the intersections marked with circles are:

$\begin{align}
x_{1,2} & = {{a_{1,1}}}{{b_{1,2}}} + {{a_{1,2}}}{{b_{2,2}}} \\
x_{3,3} & = {{a_{3,1}}}{{b_{1,3}}} + {{a_{3,2}}}{{b_{2,3}}}
\end{align}$

Notice how the dimensions are consistent: a vector of 1 row and 2 columns $[{{a_{1,1}}} \,\,\, {{a_{1,2}}}]$ is multiplied element-by-element with $[{{b_{1,2}}}\,\,\, {{b_{2,2}}}]$, and then the multiplied values are added. That process is repeated over and over, until all the values in the new matrix $\mathbf{X}$ is formed.

There is also a pattern: the value that goes into row 1 and column 2 of $\mathbf{X}$ comes from multiplying row 1 of $\mathbf{A}$ and column 2 of $\mathbf{B}$. This form of multiplying the vectors element-by-element and then adding up the result is called the ***dot product***. This explains why the function to do matrix multiplication in NumPy is called ``np.dot(...)``.

Just a quick mention: you cannot just multiply any two matrices $\mathbf{A}$ and $\mathbf{B}$. We do require that all values be present (no missing value) and that $\mathbf{A}$ must have the sum number of columns as the number of rows in matrix $\mathbf{B}$.

There is a lot of multiplying and adding happening here; 12 dot products. This is certainly not something you want to do by hand regularly. So ... let's try it with NumPy.

```python
import numpy as np
A = np.array([[8, 7], [6, 5], [4, 3], [2, 1]])
B = np.array([[1, 2, 3], [4, 5, 6]])

print('The values for matrix A = \n{}'.format(A))
print('The values for matrix B = \n{}'.format(B))

X = np.dot(A, B)
print('The values for matrix X = AB is \n{}'.format(X))

# The value of X can also be calculated from:
X = A.dot(B)

print(('The value in position (2,3) of X comes from row 2 '
       'of A [{}] and column 3 of B [{}] which corresponds '
       'to 6*3 + 5*6 = {}').format(A[1,:], B[:,2], 6*3 + 5*6))
```

    The values for matrix A =
    [[8 7]
     [6 5]
     [4 3]
     [2 1]]
    The values for matrix B =
    [[1 2 3]
     [4 5 6]]
    The values for matrix X = AB is
    [[36 51 66]
     [26 37 48]
     [16 23 30]
     [ 6  9 12]]
    The value in position (2,3) of X comes from row 2 of A [[6 5]] and column 3 of B [[3 6]] which corresponds to 6*3 + 5*6 = 48

## The `@` operator for matrix multiplication

Recently ([in 2014](https://www.python.org/dev/peps/pep-0465/)) Python introduced a new operator: `@` specifically defined for matrix multiplication. Depending on if your Python version is up to date, you do not need to write `np.dot(A, B)` anymore, but rather, more compactly: `A @ B` for matrix multiplication.

```python
# Requires a recent Python version
import numpy as np
A = np.array([[8, 7], [6, 5], [4, 3], [2, 1]])
B = np.array([[1, 2, 3], [4, 5, 6]])
X = A @ B
print(('Compact matrix multiplication: A@B = \n{}').format(X))

# which is the same result as given above.
```

    Compact matrix multiplication: A@B =
    [[36 51 66]
     [26 37 48]
     [16 23 30]
     [ 6  9 12]]

## Enrichment exercise

Try this important exercise:
> Let matrix $\mathbf{A}$ be a matrix with 4 rows and 3 columns.
> Let matrix $\mathbf{B}$ be a matrix with 3 rows and 2 columns.
>```python
import numpy as np
A = np.array([[8, 7, 3], [6, 5, 4], [4, 3, 5], [2, 1, 6]])
B = np.array([[1, 2], [4, 5], [7, 8]])
```
>1. Verify if $\mathbf{AB}$ is equal to $\mathbf{BA}$.
>
> *Hint*: we already know from the shapes of $\mathbf{A}$ and $\mathbf{B}$ that this is not true. Remember the rule about the number of rows and columns?
>
> Now if the rows and columns must match, try the following:
>```python
A = np.array([[8, 7], [6, 5]])
B = np.array([[1, 2], [3, 4]])
```

```python
# Try proving here in general if AB = BA.

# Part 1:
import numpy as np
A = np.array([[8, 7, 3], [6, 5, 4], [4, 3, 5], [2, 1, 6]])
B = np.array([[1, 2], [4, 5], [7, 8]])

# write your code here to test

# Part 2:
A = np.array([[8, 7], [6, 5]])
B = np.array([[1, 2], [3, 4]])

# write your code here to test
```

## The transpose of a matrix

In the above example you saw an error message when the dimensions of the matrix don't match. For example, you have two matrices:
* $\mathbf{A}$ with 4 rows and 3 columns
* $\mathbf{B}$ with 2 rows and 3 columns

We want to multiply them as $\mathbf{AB}$ to get a new matrix $\mathbf{X}=\mathbf{AB}$ with 4 rows and 2 columns. But because matrix $\mathbf{B}$ is not quite right [it should have 3 rows and 2 columns], we first need to ***transpose*** it.

To ***transpose*** a matrix, means to flip the rows and columns: rows become columns and columns become rows.

In this example once we have transposed matrix $\mathbf{B}$ it will have 3 rows now and 2 columns. And then it will be the right size to multiply as $\mathbf{AB}^T$ (since $\mathbf{A}$ has 3 columns and $\mathbf{B}$ has 3 rows).

There above you see the notation we use to indicate a transpose:
* $\mathbf{B}$ has 2 rows and 3 columns
* $\mathbf{B}^T$ has 3 rows and 2 columns
* $\mathbf{B}'$ is sometimes also used to indicate the transpose

Transposing happens so frequently that there is shortcut for it in NumPy: ``B.T`` will transpose matrix ``B``.

Try it below:

```python
import numpy as np
A = np.array([[8, 7, 3], [6, 5, 4], [4, 3, 5], [2, 1, 6]])
B = np.array([[1, 2, 3], [4, 5, 6]])
print('Matrix B has this shape: {}'.format(np.shape(B)))
print('The B matrix is:\n{}'.format(B))

B_transpose = B.T
print('The transposed matrix now has this shape: {}'.format(np.shape(B_transpose)))
print('B transposed is:\n{}'.format(B_transpose))
```

    Matrix B has this shape: (2, 3)
    The B matrix is:
    [[1 2 3]
     [4 5 6]]
    The transposed matrix now has this shape: (3, 2)
    B transposed is:
    [[1 4]
     [2 5]
     [3 6]]

We don't need to create an intermediate variable ``B_transpose`` to hold the transposed value.

We can simply write: ``np.dot(A, B.T)`` or even shorter ``A @ B.T``

```python
# Complete the code below to calculate the product: AB'
# (out loud you would say: "A times B transposed")

print('A times B transposed is {}'.format( ... ))
```

Notebooks/02.7 Matrix transpose and multiplication with NumPy.ipynb


$\Large{\text{Ensemble methods}}$

```python
import numpy as np #numpy package will be useful for most of the array operations in the code
# from sklearn.datasets import load_wine
```

```python
"""
Loading data set
"""
from sklearn.datasets import load_svmlight_file

myfeature, y = load_svmlight_file('diabetes')

"""
converting sparse matrix myfeature (which is in tuple form) to a dense matrix
"""

X = myfeature.todense()
X = np.array(X)
# print (y)
# print (X[0])
```

```python
#Let us print the contents of the X_y variable

print(type(X))
# print(X_y)
```

    <class 'numpy.ndarray'>

```python
#So we found that the X_y variable is of type sklearn.utils.bunch
#further, we see that the X_y variable consists of dictionary-type structures named "data",  "target" and "target_names"

# Let us now segregate the data attributes and class labels into separate variables

# X = X_y.data
# y = X_y.target

print('X shape:', X.shape, 'type(X):', type(X) )
print(' y shape:', y.shape, ' type(y):', type(y))
```

    X shape: (768, 8) type(X): <class 'numpy.ndarray'>
     y shape: (768,)  type(y): <class 'numpy.ndarray'>

```python
#Thus we see that X and y are numpy ndarrays.
#We can print the unique labels available in y
print(np.unique(y))
# print (y)
```

    [-1.  1.]

Consider an input space $\mathcal{X} \subseteq {\mathbb{R}}^d$ and the output space $\mathcal{Y} = \{+1,-1\}$. Assume a fixed (but unknown) distribution $P(X,Y)$ over $\mathcal{X} \times \mathcal{Y}$.

Let us assume that there are 15 base classifiers $C_1, C_2, \ldots, C_{15},$ where each classifier has an error rate of $\epsilon = 0.25$ on some sample from a fixed data distribution $P(X,Y)$.

To predict the label for a test sample $\hat{x}$, we adopt the following inference procedure (called $\textbf{Ensemble Classifier}(\textbf{EC})$):

1.   Predict $C_i(\hat{x})$ using each classifier $C_i, \ i \in \{1,2,\ldots,15\}$.
2.   Predict the final label $\hat{y} = \arg\max_{y \in \mathcal{Y}} \sum_{i=1}^{15} \delta(y==C_i(\hat{x}))$ where $\delta(p)$ is the indicator function given by:

  $
  \delta(p) =
  \begin{cases}
  1 \text{ if } p \text{ is true. } \\
  0 \text{ if } p \text{ is false. }
  \end{cases}
  $

$\textbf{Question:}$ What would be error rate of the classifier obtained from the above inference algorithm $\textbf{EC}$?

$\textbf{One possible answer:}$

Suppose the classifiers are assumed to be independent, then the $\textbf{EC}$ classifier would make an error only when more than half of the classifiers (i.e. more than 7 classifiers) make error in the prediction. Hence  we may write the error rate of $\textbf{EC}$ as:

$
\begin{align}
\text{error}_{\textbf{EC}} = \sum_{i=8}^{15} \begin{pmatrix} 15 \\  i \end{pmatrix} {\epsilon}^{i} (1-\epsilon)^{15-i}
\end{align}
$

which is approximately $0.017$. Note that this error rate is considerably smaller than the individual error rates of the classifiers.

```python
#compute the error rates of EC for different error rates of C_i
import math
import matplotlib.pyplot as plt

def comb(n, k):
    return math.factorial(n) // math.factorial(k) // math.factorial(n - k)

epsilons = np.linspace(0,1,11)
errors = np.zeros(epsilons.shape)

num_classifiers = 15
eps_idx=0
for epsilon in epsilons:
  error_EC = 0
  for j in np.arange(np.ceil(num_classifiers/2),num_classifiers+1):
    err = comb(num_classifiers,j)*math.pow(epsilon,j)*math.pow(1-epsilon,num_classifiers-j)
    error_EC += err
  errors[eps_idx] = error_EC
  eps_idx+=1

figure, ax = plt.subplots(1)

plt.plot(epsilons, errors, marker='o')
ax.set_xlabel('$\epsilon$')
ax.set_ylabel('$Error_{EC}$')
ax.set_xlim(0, 1)
ax.set_ylim(errors.min()-0.1,errors.max()+0.1)
ax.set_xticks(epsilons)
ax.set_yticks(np.linspace(0,1,11))
ax.set_title('Error rate of Ensemble Classifier vs $\epsilon$')
plt.show()

```

$\textbf{Important to note:}$

1.   The base classifiers $C_1, C_2, \ldots, C_{15}$ are assumed to be independent.
2.   The error rate $\epsilon$ of each base classifier must be less than $0.5$ for the ensemble classifier to behave better. $\textbf{What is meant by a base classifier having an error rate less than } 0.5$?

$\large{\text{Ways of building an ensemble classifier}}$

$\textbf{Create multiple data partitions from training data}$

1.   Resample the original training data $D$ (using sampling with replacement) and create different data paritions $D_1, D_2, \ldots, D_M$.
2.   Train different classifiers $C_i$ on respective data partition $D_i$, $i \in \{1,2,\ldots,M\}$.
3.   For a test data point $\hat{x}$ predict the label $\hat{y}=\text{MajorityVote}(C_1(\hat{x}),C_2(\hat{x}), \ldots, C_M(\hat{x}))$.

```python
from urllib.request import urlopen
from PIL import Image

img = Image.open(urlopen('https://github.com/balamurugan-palaniappan-CEP/AIML_CEP_2021/raw/main/images/ensemble_classifier.png'))
img
```

```python
np.random.seed(1000)
#Create an index array
num_samples = X.shape[0]
# print (num_samples)
indexarr = np.arange(num_samples) #index array
np.random.shuffle(indexarr) #shuffle the indices
#print('shuffled indices of samples:')
#print(indexarr)
```

```python
#Use the samples corresponding to first 60% of indexarr for training
num_train = int(0.8*num_samples)
#Use the remaining 20% samples for testing
num_test = num_samples-num_train
print('num_train: ',num_train, 'num_test: ', num_test)
```

    num_train:  614 num_test:  154

```python
#Use the first 80% of indexarr to create the train data features and train labels
train_X = X[indexarr[0:num_train]]
train_y = y[indexarr[0:num_train]]
print('shape of train data features:')
print(train_X.shape)
print('shape of train data labels')
print(train_y.shape)
```

    shape of train data features:
    (614, 8)
    shape of train data labels
    (614,)

```python
#Use remaining 20% of indexarr to create the test data and test labels
test_X = X[indexarr[num_train:num_samples]]
test_y = y[indexarr[num_train:num_samples]]
print('shape of test data features:')
print(test_X.shape)
print('shape of test data labels')
print(test_y.shape)
```

    shape of test data features:
    (154, 8)
    shape of test data labels
    (154,)

```python
#Let us now use five different base classifiers
from sklearn.svm import LinearSVC #import linear SVM
from sklearn.neighbors import KNeighborsClassifier
from sklearn import tree #decision tree from scikit learn
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
# creating an object of LogisticRegression class

clf_list = []

clf_linearsvc = LinearSVC(C=0.01)
clf_neigh = KNeighborsClassifier(n_neighbors=3, metric='manhattan') #weights='uniform' (default) or 'distance'
clf_svc = SVC(kernel='rbf', gamma=0.01)
clf_tree = tree.DecisionTreeClassifier(criterion='gini')
clf_logit = LogisticRegression(C=1.0) # C is set to be large number in order to remove the inbuilt regularization

clf_list.append(clf_linearsvc)
clf_list.append(clf_neigh)
clf_list.append(clf_svc)
clf_list.append(clf_tree)
clf_list.append(clf_logit)

clf_names = ['Linear SVM','Nearest Neighbors', 'Kernel SVM','Decision Tree','Logistic Regression']
num_classifiers = 5

for i in range(num_classifiers):
  print('classifier:',clf_list[i])
  indexarr = np.arange(train_X.shape[0]) #index array for train data
  np.random.shuffle(indexarr) #shuffle the indices
  #we shall choose 70% of the data
  partition_prop = 0.7
  num_samples_partition = int(partition_prop*train_X.shape[0])

  X_partition = train_X[indexarr[0:num_samples_partition]]
  y_partition = train_y[indexarr[0:num_samples_partition]]

  base_clf = clf_list[i]
  base_clf_model = base_clf.fit(X_partition,y_partition.ravel())

  #test accuracy
  from sklearn.metrics import accuracy_score
  test_y_predicted = base_clf_model.predict(test_X)
  test_acc = accuracy_score(test_y, test_y_predicted)
  print('test accuracy from classifier:',clf_names[i],' is:', test_acc)

  if i == 0:
    test_pred_all_clf = base_clf_model.predict(test_X)

  elif i ==1:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i==2:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i ==3:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i ==4:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

```

    classifier: LinearSVC(C=0.01, class_weight=None, dual=True, fit_intercept=True,
              intercept_scaling=1, loss='squared_hinge', max_iter=1000,
              multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,
              verbose=0)
    test accuracy from classifier: Linear SVM  is: 0.7077922077922078
    classifier: KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='manhattan',
                         metric_params=None, n_jobs=None, n_neighbors=3, p=2,
                         weights='uniform')
    test accuracy from classifier: Nearest Neighbors  is: 0.6753246753246753
    classifier: SVC(C=1.0, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
        decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',
        max_iter=-1, probability=False, random_state=None, shrinking=True,
        tol=0.001, verbose=False)
    test accuracy from classifier: Kernel SVM  is: 0.6428571428571429
    classifier: DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                           max_depth=None, max_features=None, max_leaf_nodes=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, presort='deprecated',
                           random_state=None, splitter='best')
    test accuracy from classifier: Decision Tree  is: 0.6363636363636364
    classifier: LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='auto', n_jobs=None, penalty='l2',
                       random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                       warm_start=False)
    test accuracy from classifier: Logistic Regression  is: 0.7532467532467533

    /usr/local/lib/python3.7/dist-packages/sklearn/svm/_base.py:947: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      "the number of iterations.", ConvergenceWarning)
    /usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

```python
test_pred_all_clf = np.array(test_pred_all_clf)
print(test_pred_all_clf.shape)
```

    (154, 5)

```python
## test predictions by 5 base classifiers. The last column contains actual test label.
print(np.column_stack( (test_pred_all_clf, test_y)) )
```

    [[ 1.  1.  1.  1. -1.  1.]
     [ 1. -1.  1. -1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1. -1.  1.]
     [ 1.  1.  1. -1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1. -1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1. -1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [-1. -1. -1. -1. -1.  1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [-1. -1.  1. -1. -1.  1.]
     [ 1. -1.  1. -1.  1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1. -1.  1. -1.  1.  1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1. -1.  1. -1.  1. -1.]
     [ 1. -1.  1. -1.  1.  1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1. -1.  1.  1.  1.]
     [ 1. -1.  1.  1. -1. -1.]
     [ 1.  1.  1.  1. -1.  1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1. -1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1. -1. -1.]
     [-1.  1.  1.  1. -1. -1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1. -1.  1.  1. -1.  1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1.  1.]
     [ 1. -1.  1. -1. -1.  1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1. -1. -1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1. -1. -1. -1. -1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1. -1. -1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1. -1. -1. -1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1. -1.  1.]
     [ 1.  1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1. -1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1. -1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1. -1.  1.]
     [ 1. -1.  1.  1.  1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1. -1. -1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1.  1.]
     [-1.  1. -1.  1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1. -1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1. -1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1. -1.  1. -1. -1. -1.]
     [-1.  1.  1.  1.  1.  1.]
     [-1. -1.  1. -1. -1.  1.]
     [ 1. -1. -1. -1. -1. -1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [-1. -1. -1. -1. -1. -1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1. -1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1. -1. -1.]
     [ 1. -1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1.  1.  1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1. -1. -1.  1.]
     [-1. -1.  1.  1. -1. -1.]
     [-1. -1. -1. -1. -1.  1.]
     [-1. -1.  1. -1. -1.  1.]
     [ 1.  1.  1.  1. -1. -1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1. -1.  1. -1. -1.  1.]
     [ 1.  1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1. -1. -1.]
     [ 1. -1. -1.  1.  1. -1.]
     [-1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1. -1.  1. -1. -1. -1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1. -1. -1.  1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [ 1.  1.  1. -1.  1.  1.]
     [ 1.  1.  1.  1.  1.  1.]
     [-1. -1. -1. -1. -1. -1.]]

```python

from scipy import stats

test_pred_all_clf = test_pred_all_clf.astype(int)

test_pred, counts = stats.mode(test_pred_all_clf, axis=1)
test_acc = accuracy_score(test_y, test_pred)
print('test accuracy from ensemble classifier is:', test_acc)

```

    test accuracy from ensemble classifier is: 0.6883116883116883

```python
## creating confusing matrix

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(test_y, test_pred)
print(cm)

```

    [[24 28]
     [19 83]]

```python
#Plotting the confusion matrix for better interpretation
import seaborn as sns
import matplotlib.pyplot as plt
ax = plt.subplot()
sns.heatmap(cm, annot=True, fmt='g', ax=ax)
# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
ax.xaxis.set_ticklabels(['-1', '1'])
ax.yaxis.set_ticklabels(['-1', '1'])
```

$\large{\text{Ways of building an ensemble classifier}}$

$\textbf{Create multiple attribute partitions from training data}$

1.   Resample the attributes from original training data $D$ (using sampling with replacement) and create different feature paritions $F_1, F_2, \ldots, F_M$. Note that the number of samples in these partitions might be same as that in $D$ or might be different.
2.   Train different classifiers $C_i$ on respective feature partition $F_i$, $i \in \{1,2,\ldots,M\}$.
3.   For a test data point $\hat{x}$ first create feature partitions based on $F_1, F_2, \ldots, F_M$ and predict the label $\hat{y}=\text{MajorityVote}(C_1(F_1(\hat{x})),C_2(F_2(\hat{x})), \ldots, C_M(F_M(\hat{x})))$.

```python
from urllib.request import urlopen
from PIL import Image

img = Image.open(urlopen('https://github.com/balamurugan-palaniappan-CEP/AIML_CEP_2021/raw/main/images/ensemble_classifier_RF.png'))
img
```

```python
#Let us now use three different base classifiers and check the decision boundary
from sklearn.ensemble import RandomForestClassifier

clf_list = []

clf_linearsvc = LinearSVC(C=1.0)
clf_neigh = KNeighborsClassifier(n_neighbors=2, metric='manhattan') #weights='uniform' (default) or 'distance'
clf_svc = SVC(kernel='rbf', gamma=0.01)
# clf_tree = tree.DecisionTreeClassifier(criterion='entropy')
clf_rf = RandomForestClassifier(n_estimators = 100,  random_state=0)
clf_logit = LogisticRegression(C=1.0) # C is set to be large number in order to remove the inbuilt regularization

clf_list.append(clf_linearsvc)
clf_list.append(clf_neigh)
clf_list.append(clf_svc)
clf_list.append(clf_logit)
clf_list.append(clf_rf)

clf_names = ['Linear SVM','Nearest Neighbors', 'Kernel SVM','Logistic Regression','Random Forest']
num_classifiers = 5

for i in range(num_classifiers):
  print('classifier:',clf_list[i])
  indexarr = np.arange(train_X.shape[0]) #index array for train data
  np.random.shuffle(indexarr) #shuffle the indices
  #we shall choose 70% of the data
  partition_prop = 0.7
  num_samples_partition = int(partition_prop*train_X.shape[0])

  X_partition = train_X[indexarr[0:num_samples_partition]]
  y_partition = train_y[indexarr[0:num_samples_partition]]

  base_clf = clf_list[i]
  base_clf_model = base_clf.fit(X_partition,y_partition.ravel())

  #test accuracy
  from sklearn.metrics import accuracy_score
  test_y_predicted = base_clf_model.predict(test_X)
  test_acc = accuracy_score(test_y, test_y_predicted)
  print('test accuracy from classifier:',clf_names[i],' is:', test_acc)

  if i == 0:
    test_pred_all_clf = base_clf_model.predict(test_X)

  elif i ==1:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i==2:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i ==3:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )

  elif i ==4:
    test_pred_all_clf = np.column_stack( (test_pred_all_clf,test_y_predicted) )
```

    classifier: LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,
              intercept_scaling=1, loss='squared_hinge', max_iter=1000,
              multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,
              verbose=0)
    test accuracy from classifier: Linear SVM  is: 0.6818181818181818
    classifier: KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='manhattan',
                         metric_params=None, n_jobs=None, n_neighbors=2, p=2,
                         weights='uniform')
    test accuracy from classifier: Nearest Neighbors  is: 0.6038961038961039
    classifier: SVC(C=1.0, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
        decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',
        max_iter=-1, probability=False, random_state=None, shrinking=True,
        tol=0.001, verbose=False)
    test accuracy from classifier: Kernel SVM  is: 0.6493506493506493
    classifier: LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='auto', n_jobs=None, penalty='l2',
                       random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                       warm_start=False)
    test accuracy from classifier: Logistic Regression  is: 0.7012987012987013
    classifier: RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                           criterion='gini', max_depth=None, max_features='auto',
                           max_leaf_nodes=None, max_samples=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, n_estimators=100,
                           n_jobs=None, oob_score=False, random_state=0, verbose=0,
                           warm_start=False)

    /usr/local/lib/python3.7/dist-packages/sklearn/svm/_base.py:947: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      "the number of iterations.", ConvergenceWarning)
    /usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

    test accuracy from classifier: Random Forest  is: 0.6818181818181818

```python
test_pred_all_clf = np.array(test_pred_all_clf)
print(test_pred_all_clf.shape)

from scipy import stats

test_pred_all_clf = test_pred_all_clf.astype(int)

test_pred, counts = stats.mode(test_pred_all_clf, axis=1)
test_acc = accuracy_score(test_y, test_pred)
print('test accuracy from ensemble classifier is:', test_acc)

```

    (154, 5)
    test accuracy from ensemble classifier is: 0.6883116883116883

```python
## creating confusing matrix

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(test_y, test_pred)
print(cm)

```

    [[25 27]
     [21 81]]

```python
#Plotting the confusion matrix for better interpretation
import seaborn as sns
import matplotlib.pyplot as plt
ax = plt.subplot()
sns.heatmap(cm, annot=True, fmt='g', ax=ax)
# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
ax.xaxis.set_ticklabels(['-1', '1'])
ax.yaxis.set_ticklabels(['-1', '1'])
```

#Exercise:
Use an ensemble method with classifiers as linear SVMs with different C values (C= 0.1, 0.01, 0.001, 0.0001) to train on diabetes dataset.


## COMP3670/6670 Programming Assignment 1 - Linear Algebra and Analytic Geometry
---

**Enter Your Student ID:** u6828533

**Your Name:** Yuxuan Lin

**Deadline:** 11:59pm, Aug 29th, 2021

**Submit:** Write your answers in this file, and submit a single Jupyter Notebook file (.ipynb) on Wattle. Rename this file with your student number as 'u6828533.ipynb'.

**Enter Discussion Partner IDs Below:**
You could add more IDs with the same markdown format above.

---
**Marking distribution for the homework/assignment**

- Task1 = 10%
- Task2 = 30%
- Task3 = 10%
- Task4 = 50%

## Task 0: Introduction
---

**NOTE:** *This part of the first assignment is by necessity somewhat tedious as its primary purpose is to introduce syntax, how to access and understand the Numpy documentation and some very basic concepts. If you are already familiar with Numpy, you can just read the **TASK** headings and complete the questions without worrying about all the additional information. This is designed for people who have never seen Numpy before, so it's a very easy 1st year style introduction to just introduce syntax.*

*As this is a third year subject, it is assumed you already know to to program well (but may be unfamiliar with Python and Numpy).*

---

Arguably the most fundamental tool needed to engage with machine learning in Python is Numpy *(np)*. To include Numpy in any project, simply type the following line at the top of your python file:

```python
# numpy
import numpy as np

!pip install sympy
import sympy as sp

# matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
```

A Jupyter Notebook is divided into cells, each of which works like a Python module or Latex file. When a cell is run, any cells that follow it will have access to its results. Running the above cell will give all following cells access to the Numpy library.

-----------

   **TASK 0.1:** To run the above cell, select it by clicking on it, hold **shift** and press **return**. If you have succeeded, then the cell will print "Done" just above.

-----------

Numpy is a library of common mathematical data structures and algorithms used in machine learning. For example:

- It allows you to declare vectors and matrices, with all the associated mathematical operations like matrix vector products, matrix addition / subtraction.
- It provides convenient, efficient implementations of algorithms to solve matrix equations, find the inverse of a matrix or perform Eigen decomposition. The implementation of these algorithms is compiled from C code, making Numpy much faster than programming these algorithms yourself in Python.

You'll need to know Numpy inside and out. We'll start by getting you familiar with the easy to access online documentation and performing a few basic operations.

Below I have declared the matrices:

$$A =
\begin{bmatrix}
2&3\\
0&1\\
\end{bmatrix}
\\
x =
\begin{bmatrix}
1\\
3\\
\end{bmatrix}
$$.

---
**TASK 0.2:** Run the cell below and observe what it prints.

---
(If it throws an error, you have the wrong version of python installed. This entire course will use Python 3, not Python 2)

```python
A = np.array([[2, 3], [0, 1]])
x = np.array([[1], [3]])

# Matrix Multiplication Example
b = A @ x
print("\nMatrix Multiplication")
print(b)
```

    Matrix Multiplication
    [[11]
     [ 3]]

The above code illustrates how to perform matrix multiplication. Memorise it. Below are some other basic operations you'll likely need over the coming semester:

```python
# Matrix Addition Example
b = A + x
print("\nMatrix Addition")
print(b)

# Elementwise Multiplication Example
b = A * x
print("\nElementwise Matrix Multiplication")
print(b)

# Extract a single element of a matrix:
print("\nSingle Element Extraction")
b = A[0, 0]
print(b)

# Extract an entire column of a matrix:
print("\nColumn Extraction")
b = A[:, 0]
print(b)

# Extract an entire row of a matrix:
print("\nRow Extraction")
b = A[0, :]
print(b)

# Transpose of a matrix:
print("\nTranspose")
A_Transpose = A.T
print(A_Transpose)
```

    Matrix Addition
    [[3 4]
     [3 4]]

    Elementwise Matrix Multiplication
    [[2 3]
     [0 3]]

    Single Element Extraction
    2

    Column Extraction
    [2 0]

    Row Extraction
    [2 3]

    Transpose
    [[2 0]
     [3 1]]

## Task1: Solving a system of linear equations
---

A vital part of linear algebra is to know how to solve a system of linear equations. For e.g.

$$a_{11}x_1+a_{12}x_2 \dots a_{1d}x_d=b_1$$
$$a_{21}x_1+a_{22}x_2 \dots a_{2d}x_d=b_2$$
$$\vdots$$
$$a_{n1}x_1+a_{n2}x_2 \dots a_{nd}x_d=b_n$$

The above system of linear equations can also be written down in a compact matrix form as follows:

$$AX = B$$

where,
$$A = \begin{bmatrix}
a_{11} & \dots & a_{1d}\\
\vdots & \ddots & \vdots \\
a_{n1} & \dots & a_{nd}
\end{bmatrix}, \quad
B = \begin{bmatrix}
b_1 \\ \vdots \\ b_n
\end{bmatrix}, \quad
X = \begin{bmatrix}
x_1 \\ \vdots \\ x_d
\end{bmatrix}.
$$

---
**Task 1.1**: Use numpy's solve function to compute X

---
**HINT**: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.solve.html

```python
A = np.array([[2, 3, 4], [0, 1, 3], [-1, 0, 2]])
B = np.array([[3], [5], [7]])

def solve_with_numpy(A, B):
    return np.linalg.solve(A, B)

X = solve_with_numpy(A, B)

print(X)
```

    [[-11.]
     [ 11.]
     [ -2.]]

A more hands on way for solving for X, involves computing first computing the inverse of the matrix $A$

---
**Task 1.2**: Use numpy's inbuilt method for computing the inverse of the matrix $A$

---

**HINT**: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.inv.html

```python
A = np.array([[2, 3, 4], [0, 1, 3], [-1, 0, 2]])

A_inv = np.linalg.inv(A)

print(A_inv)
```

    [[-2.  6. -5.]
     [ 3. -8.  6.]
     [-1.  3. -2.]]

---
**Task 1.3**: Use the inverse of matrix $A$, to solve for $X$

---

```python
A = np.array([[2, 3, 4], [0, 1, 3], [-1, 0, 2]])
B = np.array([[3], [5], [7]])

X = np.linalg.inv(A) @ B

print(X)
```

    [[-11.]
     [ 11.]
     [ -2.]]

But what if $A$ is not a square matrix?

---
**Task 1.4**: Compute the Moore-Penrose pseudo inverse for matrix $A$ and use it for computing $X$. Note that, you must not directly use `np.linalg.pinv` for computing the pseudo-inverse.

---

**HINT**: https://en.wikipedia.org/wiki/Moore–Penrose_inverse

```python
A = np.array([[1, 3], [2, 7], [5, 1]])
B = np.array([[13], [30], [9]])

A_pseudo_inverse = np.linalg.inv(A.T @ A) @ A.T
X = A_pseudo_inverse @ B

print(X)
```

    [[1.]
     [4.]]

## Task 2: Gaussian elimination

Lets practice Gaussian elimination without numpy!

---
**Task 2.1**:  Complete the following gaussian elimnation function to compute the **reduced row-echelon form** of matrix $A$. You must implement the gaussian elimination algorithm yourself, not merely call someone elses library function. You **MUST NOT** copy codes from any source. Your function needs to fit different shapes of matrices.

---

**HINT:** https://en.wikipedia.org/wiki/Gaussian_elimination

```python
"""
hint: Pseudocode for reduced row echelon form
url: https://en.wikipedia.org/wiki/Row_echelon_form
"""

def gaussian_elim(X):
    R, C = X.shape  # get the number of rows and columns in X
    X, p = (
        np.float64(X),
        0,
    )  # cast the elements matrix to float, initialize pivot element

    for r in range(R):
        if C > p:
            i = r
            while np.isclose(X[i, p], 0):
                i += 1
                if R == i:
                    i = r
                    p += 1
                    if C == p:
                        p -= 1
                        break
            if i != r:
                X[[i, r]] = X[[r, i]]  # swap rows i and r
            if not np.isclose(X[r, p], 0):
                X[r] /= X[r, p]  # divide row r by X[r, p]
            for i in range(R):
                if i != r:
                    X[i] -= (X[i, p] * X[r]
                            )  # subtract X[i, p] multiplied by row r from row i
            p += 1

    return X
```

```python
A = np.array(
    [[1, 0, 1, 1], [1, 0, 1, 1], [0, 1, 1, 1], [1, 1, 1, 0], [1, 1, 1, 0]],
    dtype=np.float64,
)
print(gaussian_elim(A))
# The result should look like
# [[ 1.,  0.,  0., -1.],
#  [ 0.,  1.,  0., -1.],
#  [-0., -0.,  1.,  2.],
#  [ 0.,  0.,  0.,  0.],
#  [ 0.,  0.,  0.,  0.]]
```

    [[ 1.  0.  0. -1.]
     [ 0.  1.  0. -1.]
     [-0. -0.  1.  2.]
     [ 0.  0.  0.  0.]
     [ 0.  0.  0.  0.]]

```python
B = np.array([[2, 0, 3, 1, 0, 2], [1, 3, 2, 0, 3, 1], [3, 2, 3, 1, 1, 2]])
print(gaussian_elim(B))
# The result should look like
# [[ 1.          0.          0.          0.36363636 -0.81818182  0.18181818]
#  [ 0.          1.          0.         -0.18181818  0.90909091 -0.09090909]
#  [ 0.          0.          1.          0.09090909  0.54545455  0.54545455]]
```

    [[ 1.          0.          0.          0.36363636 -0.81818182  0.18181818]
     [ 0.          1.          0.         -0.18181818  0.90909091 -0.09090909]
     [-0.         -0.          1.          0.09090909  0.54545455  0.54545455]]

We will evaluate your code with multiple different matrices.

---
**Task 2.2**:  If a square matrix is invertible, how would its reduced row-echelon form looks like? Use no more than 3 sentences in your answer.

---
**Answer**:

The reduced row-echelon form of an invertible square matrix will be an identity matrix of the same size.

## Task 3: Geometrical intuition for system of linear equations

---

In this task we will develop a geometrical intuition for solving the system of linear equations:

$$AX=B$$

where if you recall from before:

$$A = \begin{bmatrix}
a_{11} & \dots & a_{1d}\\
\vdots & \ddots & \vdots \\
a_{n1} & \dots & a_{nd}
\end{bmatrix}, \quad
B = \begin{bmatrix}
b_1 \\ \vdots \\ b_n
\end{bmatrix}, \quad
X = \begin{bmatrix}
x_1 \\ \vdots \\ x_d
\end{bmatrix}.
$$

This can also be written as:
$$
\begin{bmatrix}
a_{11} \\ \vdots \\ a_{n1}
\end{bmatrix} \ x_1
+
\begin{bmatrix}
a_{12} \\ \vdots \\ a_{n2}
\end{bmatrix} \ x_2
+ \dots
\begin{bmatrix}
a_{1d} \\ \vdots \\ a_{nd}
\end{bmatrix} \ x_d
= \begin{bmatrix}
b_1 \\ \vdots \\ b_n
\end{bmatrix}
$$

Thats is, solving the system of linear equations $AX=B$ is equivalent to finding a linear combination of the column vectors of matrix $A$, that result in $B$.

---
**Task 3.1**: Given a matrix $A$ of shape (2 X 2), generate a plot containing arrows representing the column vectors of $A$.

---
**HINTS:**

- For $A=I$, the plot should look something like:

- You'll need to use plt.quiver() with angles='xy', scale_units='xy' and scale=1.
    - https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.quiver.html

```python
origin = [0], [0]
A = np.array([[2, 1], [1, 4]])

x_pos, y_pos = origin

R_A, C_A = A.shape
for i in range(C_A):
    x_dir, y_dir = A[:, i]
    plt.quiver(x_pos,
               y_pos,
               x_dir,
               y_dir,
               angles="xy",
               scale_units="xy",
               scale=1)

plt.xlim([-5, 5])
plt.ylim([-5, 5])
plt.title("Visualising column vectors of a matrix")
plt.show()
```

---
**Task 3.2**: To the above plot, now add an arrow (with different color) representing vector $B$. While doing so remember to:
* Assign different labels to the arrows for matrix $A$ and $B$, using the keyword `label=A` or `label=B`.
* Display the appropriate legend, with proper labels. Read more: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.legend.html
* Assign colored arrows for matrix $A$ as `color=blue` and arrow for vector $B$ as `color=red`
---

```python
origin = [0], [0]

A = np.array([[2, 1], [1, 4]])
B = np.array([[2], [2]])

x_pos, y_pos = origin

R_A, C_A = A.shape
for i in range(C_A):
    x_dir, y_dir = A[:, i]
    plt.quiver(
        x_pos,
        y_pos,
        x_dir,
        y_dir,
        angles="xy",
        scale_units="xy",
        scale=1,
        color="blue",
        label="A",
    )

R_B, C_B = B.shape
for j in range(C_B):
    x_dir, y_dir = B[:, j]
    plt.quiver(
        x_pos,
        y_pos,
        x_dir,
        y_dir,
        angles="xy",
        scale_units="xy",
        scale=1,
        color="red",
        label="B",
    )

plt.xlim([-5, 5])
plt.ylim([-5, 5])
plt.legend()
plt.title("Visualising column vectors of a matrix")
plt.show()
```

The above plot should convey a more intuitve picture than the simply looking at $AX=B$. Let's test our intuition!

---
**Task 3.3**: Without calculating, for $X = [x_1 \ x_2]^T$, analytically comment on the magnitude of $x_1$ and $x_2$. That is first choose one of the following and then use no more than 3 sentences to explain your choice.
1. $|x_1| > |x_2|$
2. $|x_1| < |x_2|$
3. $|x_1| \approx |x_2|$

---

**Answer**:

    $|x_1| > |x_2|$.

    Because from the plot above, the magnitude of $B$'s orthogonal projection on $A$'s first column vector is greater than the magnitude of $B$'s orthogonal projection on $A$'s second column vector.

## Task 4: Gram-Schmidt orthogonalization

The Gram-Schmidt algorithm is used for finding an orthonormal basis for the subspace spanned by a set of vectors.

Given a linearly independent set of vectors $S = \lbrace \mathbf{v_1}, \mathbf{v_2}, ..., \mathbf{v_n} \rbrace$, an orthogonal set $S' = \lbrace \mathbf{u_1}, \mathbf{u_2}, ..., \mathbf{u_n} \rbrace $ can be generated under this procedure:

$\mathbf{u_1} = \mathbf{v_1}$\
$\mathbf{u_2} = \mathbf{v_2} - \textrm{proj}_{\mathbf{u_1}}(\mathbf{v_2})$\
$\mathbf{u_3} = \mathbf{v_3} - \textrm{proj}_{\mathbf{u_1}}(\mathbf{v_3}) - \textrm{proj}_{\mathbf{u_2}}(\mathbf{v_3})$\
...\
$\mathbf{u_n} = \mathbf{v_n} - \sum_{i=1}^{n-1}\textrm{proj}_{\mathbf{u_i}}(\mathbf{v_n})$,

where $\textrm{proj}_{\mathbf{u}}(\mathbf{v}) = \dfrac{\langle \mathbf{u}, \mathbf{v} \rangle}{\langle \mathbf{u}, \mathbf{u} \rangle}\mathbf{u}$

The orthogonal set is not normalized, you can normalize it by $\mathbf{e_i} = \dfrac{\mathbf{u_i}}{||\mathbf{u_i}||}$

---
**Read more:** https://en.wikipedia.org/wiki/Gram–Schmidt_process

---
**Task 4.1**: Implement a function that finds an orthonormal basis of column vectors for matrix $A$. The *gram_schmidt* function should be according to following specifications:
* Take as input any matrix $A$.
* Return a matrix with the orthonormal basis vectors as the columns.

---
**NOTE**: The resulting basis should be orthonormal, so each basis vector should be of norm 1.

```python
# Calculate the Euclidean norm of a column vector
# WARNING: The column vectors of input matrix might be linearly dependent
# We set the norm of zero vector as 1.0 for further division calculation
def euclidean_norm(V):
    foo = 0
    for v in V:
        foo += v**2
    norm = np.sqrt(foo)
    return 1.0 if np.isclose(norm, 0) else norm

# Projection operator
def proj(u, v):
    return np.dot(np.dot(np.transpose(u), v), u) / euclidean_norm(u)

# Find orthonormal basis using gram-schmidt
def gram_schmidt(A):
    A = np.float64(A)  # cast the elements matrix to float
    R, C = A.shape  # get the number of rows and columns in A
    B = np.zeros((R, C))  # initialize a matrix for storing result

    B[:, 0] = A[:, 0] / euclidean_norm(A[:, 0])

    for i in range(1, C):
        B[:, i] = A[:, i]
        for j in range(i):
            B[:, i] -= proj(B[:, j], B[:, i])
        B[:, i] = (np.zeros(R) if euclidean_norm(B[:, i]) == 1.0 else B[:, i] /
                   euclidean_norm(B[:, i]))

    r, c = B.shape  # get the number of rows and columns in B

    # Delete all the zero vector columns in the result matrix
    mask = np.array([[k] for k in range(c) if np.count_nonzero(B[:, k]) == 0])
    if mask.size > 0:
        B = np.delete(B, mask, axis=1)

    return B
```

---
**TASK 4.2:** Implement a function that accepts a matrix and returns TRUE if all the columns of that matrix are orthogonal to one another, and each one of them is of length 1 (the L2 norm of each column vector is 1).

Use this function to verify the output of your gram_schmidt function is an orthonormal basis for $A$.

Again, you must program this function yourself, rather than relying on a library function.

---

**HINT:**
- A pair of column vectors $v_1$ and $v_2$ are orthogonal to one another if and only if $v_1^Tv_2 = 0$. They form an orthonormal basis if they are all orthogonal to eachother, and they are each of length 1. https://en.wikipedia.org/wiki/Orthogonality
- You can check whether your algorithms work using the last 6 lines in the code given below.

```python
# Calculate the L2 norm (Euclidean norm) of a column vector
def l2_norm(V):
    foo = 0
    for v in V:
        foo += v**2
    norm = np.sqrt(foo)
    return norm

# Check if two vectors are orthogonal to one another
def if_orthogonal(v1, v2):
    return np.isclose(np.dot(np.transpose(v1), v2), 0)

# Check if the L2 norm of a vector is of length 1
def if_length(v):
    return np.isclose(l2_norm(v), 1)

# Check if all the column vectors are orthogonal to one another and if each vector is of length 1
def orthogonality_checker(A):
    R, C = A.shape  # get the number of rows and columns in A

    for i in range(C):
        if not if_length(A[:, i]):
            return False

        for j in range(C):
            if i != j and not if_orthogonal(A[:, i], A[:, j]):
                return False

    return True

true_test = np.eye(3, 3)
false_test = np.ones((3, 3))
print("\nCHECKING TRUE TEST (SHOULD RETURN TRUE):")
print(orthogonality_checker(true_test))
print("\nCHECKING FALSE TEST (SHOULD RETURN FALSE):")
print(orthogonality_checker(false_test))
```

    CHECKING TRUE TEST (SHOULD RETURN TRUE):
    True

    CHECKING FALSE TEST (SHOULD RETURN FALSE):
    False


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *

f, d, D, z, kx = symbols('f,d,D,Z, K_x')
z = symbols('Z')
Delta = Symbol("Delta")
eq_right = Delta * z
eq_left = Abs(1/d-1/(d+Delta*d))*f*D*kx
Eq(eq_right, eq_left)
```

$\displaystyle \Delta Z = D K_{x} f \left|{\frac{1}{\Delta d + d} - \frac{1}{d}}\right|$

```python
# intrinsics: [1137.952358814274, 1139.2455706836256, 1040.2907098789112, 764.4019425366827]
# resolution: [2048, 1536]
# assuming that first entry of intrinsics is the kx parameter [pixels/inches]

# error for given measured diameter d on image plane in meters
def error_function(d, delta_d):
    delta_Z = D*f*kx*abs(1/(d+delta_d) - 1/d)*10**(-3) # factor at the end converts it to meters from cm
    return delta_Z

kx = 1137.952358814274/2.54   # to turn [pixels/inches] to [pixels/cm]
ky = 1139.2455706836256/2.54  # to turn [pixels/inches] to [pixels/cm]
num_of_samples = 2048
d_d = 3
f = 1
D = 10
# d is diameter of ball in pixels (only integer numbers)
d = np.linspace(1, 2048, num_of_samples, endpoint=True)
delta_Z = error_function(d, d_d)

print(delta_Z.shape)

pprint(delta_Z)

fig, ax = plt.subplots(nrows=1, ncols=3)
fig.set_size_inches(h=14, w=21)
fig.subplots_adjust(left=0.05, bottom=0.05, right=0.95,
                        top=0.9, wspace=0.25, hspace=0.25)

distance = np.linspace(0,100, num_of_samples, endpoint=True) # different distances (camera frame) to scale x_error and y_error
# delta_x is in pixels
for delta_x in range(1,10):
    delta_X = distance*delta_x/(kx*f)
    ax[0].plot(distance, delta_X, label='with delta_x: {}'.format(delta_x))

for delta_y in range(1,10):
    delta_Y = distance*delta_y/(ky*f)
    ax[1].plot(distance, delta_Y, label='with delta_y: {}'.format(delta_y))

for delta_d in range(1,10):
    delta_Z = error_function(d[:30], delta_d)
    ax[2].plot(d[:30], delta_Z, label='with delta_d: {}'.format(delta_d))

ax[0].set_xlabel('distance in Z-direction in meters')
ax[0].set_ylabel('error in X-direction in meters')
ax[0].legend()
ax[1].set_xlabel('distance in Z-direction in meters')
ax[1].set_ylabel('error in Y-direction in meters')
ax[1].legend()
ax[2].set_xlabel('diameter d in pixels')
ax[2].set_ylabel('error in Z-direction in meters')
ax[2].legend()
plt.show()

```

```python

```