# Homework 3 - Linear and Logistic Regressions
## CSCI 4622 - Fall 2021
***
**Name**: Willem Scott
***

This assignment is due on Canvas by **11.59 PM on Monday, November 1st**.
Submit only this Jupyter notebook to Canvas.  Do not compress it using tar, rar, zip, etc.
Your solutions to analysis questions should be done in Markdown directly below the associated question.
Remember that you are encouraged to discuss the problems with your classmates and instructors,
but **you must write all code and solutions on your own**, and list any people or sources consulted.

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import pickle, gzip
%matplotlib inline
```

## Problem 1: Linear Regression (55 points)

We have seen that Ridge regression adds a regularization term to the least square using the L2 norm.
The objective is then to minimize:

\begin{align}
\frac{1}{N}\sum_{i=1}^{N} ||y_i-w^T x_i-b||^2 + \alpha||w||^2
\end{align}

where $\mathbf w$ denotes the set of parameters for the linear regression model,
 $\alpha$ is the trade-off regularization parameter, and $N$ the number of samples
The intercept $b$ can also be included in $(x_i)_{i\leq N}$ by appending a constant feature to the data.
You will be using the following data sets:

```python
class CloudData:
    def __init__(self):
        # Data from: https://archive.ics.uci.edu/ml/datasets/Cloud
        data = np.fromfile("data/cloud.data", sep = " ").reshape((1024, 10))
        y = data[:, 6]
        X = np.delete(data, 6, axis = 1)

        self.train_x = X[:800]
        self.train_y = y[:800]

        self.test_x = X[800:]
        self.test_y = y[800:]

class ForestData:
    def __init__(self):
        # Data from: http://archive.ics.uci.edu/ml/datasets/Forest+Fires
        data = pd.read_csv("data/forestfires.csv")
        data = data.sample(frac = 1).reset_index(drop = True).drop(columns = ["month", "day"])
        data["area"] = np.log(data["area"] + 1)
        X = data.drop(columns = "area").values
        y = data["area"].values

        self.train_x = X[:400]
        self.train_y = y[:400]

        self.test_x = X[400:]
        self.test_y = y[400:]
cloud_data = CloudData()
forest_data = ForestData()
```

To implement Ridge Regression, We'll use the solver from sklearn module `Ridge`
([read more](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html)).
- 1.1 [5 points] Complete the `fit` and `evaluate` methods following the docstring description.

```python
from sklearn import linear_model

class Ridge:
    def __init__(self, alpha):
        """
        Parameters
        ----------
        alpha:  regularization parameter
        """
        self.alpha = alpha # our regularization / penalty term for weights
        self._model = linear_model.Ridge(alpha, fit_intercept=True)  # Using sklearn module Ridge

    def fit(self, X, y):
        """
        Fit the ridge model, train it using the provided data
        Calculate the number of non-zero coefficients in the model weights.
        X: training features (n_samples, n_features)
        y: target values (n_samples)

        RETURN :
            num_nonzero_coeff : number of non-zero coefficients in the model weights
        """
        #Workspace 1.1.a
        #BEGIN
        #TODO: Fit the ridge model and return the number of non-zero coefficients in the model weights.

        #Code here
        self._model.fit(X, y)

        #END
        return self._model.coef_

    def evaluate(self, test_x, test_y):
        """
        Compute Mean square error (MSE) between the predicted values and the actual values  of the test data
        Parameters
        ----------
        test_x: test features
        test_y: test target

        Returns MSE
        -------
        """
        #Workspace 1.1.b
        #BEGIN
        #TODO: predict based on the test features and return the mean_squared_error

        #Code here
        res = self._model.predict(test_x)

        #END
        return self.mse(test_y, res)

    @staticmethod
    def mse(y_true, y_hat):
        return np.mean((y_true-y_hat)**2)
```

```python
# Tests cell
r = Ridge(15)
print(r.fit(cloud_data.train_x, cloud_data.train_y))

```

    [-3.19294965e-04  3.65307232e-03 -2.41868248e-03  3.32305831e-01
     -8.13531061e-03 -6.01887500e-01 -1.72318259e-03  9.95178025e-04
      1.19171413e-03]

1.2 [5 points] For each of the datasets, produce 2 plots using `Ridge` :
  - The number non-zero coefficients versus $\alpha$
  - Mean Squared Error (MSE) on test set versus $\alpha$

 Use $\alpha = 1, 50, 100, 200, 1000$

```python
alphas = [1, 50, 100, 200, 1000]
#BEGIN Workspace 1.2

#code here
datasets = [cloud_data, forest_data]
for dataset in datasets:
    aset = [1,50,100,200,1000]
    mses = []
    cfcount = []

    for a in aset:
        mod = Ridge(a)
        coeffs = mod.fit(dataset.train_x, dataset.train_y)
        mse = mod.evaluate(dataset.test_x, dataset.test_y)
        mses.append(mse)
        cfcount.append((coeffs != 0).sum())

    plt.plot(aset, mses)
    plt.xlabel("alpha")
    plt.ylabel("MSE")
    plt.title(f"Alpha vs MSE for {dataset.__class__.__name__}")
    plt.show()
    plt.plot(aset, cfcount)
    plt.xlabel("alpha")
    plt.ylabel("Nonzero coeffs")
    plt.title(f"Alpha vs Nonzero Coefficients for {dataset.__class__.__name__}")
    plt.show()

#END Workspace 1.2
```

Now we'll move to Lasso regression. Lasso uses $l_1$ norm in the regularization term and minimizes:
\begin{align}
\frac{1}{2N}\sum_i ||y_i-w^t x_i -b||^2 + \alpha||w||_1
\end{align}

We'll be using the `Lasso`solver from sklearn ([more details](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html)).

Complete the Lasso class in the same way as you completed the Ridge class. We'll use the solver from sklearn module `Lasso`
([read more](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html)).
- 1.3 [5 points] Complete the `fit` and `evaluate` methods following the docstring description.

```python
class Lasso:
    def __init__(self, alpha):
        """
        Parameters
        ----------
        alpha:  regularization parameter
        """
        self.alpha = alpha
        self._model = linear_model.Lasso(alpha=alpha, fit_intercept=True)

    def fit(self, X, y):
        """
        Fit the Lasso model, train it using the provided data
        Calculate the number of non-zero coefficients in the model weights.
        X: training features (n_samples, n_features)
        y: target values (n_samples)

        RETURN :
            num_nonzero_coeff : number of non-zero coefficients in the model weights
        """
        #Workspace 1.3.a
        #BEGIN
        #TODO: Fit the Lasso model and return the number of non-zero coefficients in the model weights.

        #Code here
        self._model.fit(X, y)

        #END
        return self._model.coef_

    def evaluate(self, test_x, test_y):
        """
        Compute Mean square error (MSE) between the predicted values and the actual values  of the test data
        Parameters
        ----------
        test_x: test features
        test_y: test target

        Returns MSE
        -------
        """
        #Workspace 1.3.b
        #BEGIN
        #TODO: predict based on the test features and return the mean_squared_error

        #Code here
        res = self._model.predict(test_x)

        #END
        return self.mse(test_y, res)

    @staticmethod
    def mse(y_true, y_hat):
        return np.mean((y_true-y_hat)**2)
```

```python
# Tests cell
r = Lasso(15)
print(r.fit(cloud_data.train_x, cloud_data.train_y))

```

    [-0.          0.          0.          0.          0.00569222 -0.
      0.          0.          0.        ]

1.4 [5 points] For each of the datasets, produce 2 plots using `Lasso` :
  - The number non-zero coefficients versus $\alpha$
  - Mean Squared Error (MSE) on test set versus $\alpha$

 Use $\alpha = 0.01, 0.05, 0.1, 0.2, 0.3$

```python
alphas = [0.01, 0.05, 0.1, 0.2, 0.3]
# Workspace 1.4
#BEGIN

#code here
datasets = [cloud_data, forest_data]
for dataset in datasets:
    aset =  [0.01, 0.05, 0.1, 0.2, 0.3]
    mses = []
    cfcount = []

    for a in aset:
        mod = Lasso(a)
        coeffs = mod.fit(dataset.train_x, dataset.train_y)
        mse = mod.evaluate(dataset.test_x, dataset.test_y)
        mses.append(mse)
        cfcount.append((coeffs != 0).sum())

    plt.plot(aset, mses)
    plt.xlabel("alpha")
    plt.ylabel("MSE")
    plt.title(f"Alpha vs MSE for {dataset.__class__.__name__}")
    plt.show()

    plt.plot(aset, cfcount)
    plt.xlabel("alpha")
    plt.ylabel("Nonzero coeffs")
    plt.title(f"Alpha vs Nonzero Coefficients for {dataset.__class__.__name__}")
    plt.show()

#END Workspace 1.2
```

1.5 [5 points] Compare the two algorithms on each data set:
- Compare the number of non-zero coordinates of Ridge vs Lasso and their MSE on each dataset.
- Which type of regression is better for each dataset?

# Workspace 1.5

- Write-up answer here

Elastic Net try to combine both types of regularization to get the best of both worlds by minimizing:

\begin{align}
\frac{1}{2N} \sum_i ||y_i - w^Tx_i -b||^2_2 + \alpha\beta||w||_1 + \frac{\alpha}{2}(1 - \beta)||w||^2_2
\end{align}

where $\beta\in[0,1]$ is the $l_1$ ratio ($\beta=1$ for Lasso and $\beta=0$ for Ridge).

Complete the Elastic class in the same way as you completed the Ridge and Lasso class. We'll use the solver from sklearn module `ElasticNet`
([read more](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html)).
- 1.6 [5 points] Complete the `fit` and `evaluate` methods following the docstring description.

```python
class Elastic:
    def __init__(self, alpha, beta=0.5):
        """
        Parameters
        ----------
        alpha:  First regularization parameter
        beta:   Second regularization parameter (default set to 0.5)
        """
        self.alpha = alpha
        self.beta = beta
        self._model = linear_model.ElasticNet(alpha=alpha, l1_ratio=beta, fit_intercept=True)

    def fit(self, X, y):
        """
        Fit the Elastic model, train it using the provided data
        Calculate the number of non-zero coefficients in the model weights.
        X: training features (n_samples, n_features)
        y: target values (n_samples)

        RETURN :
            num_nonzero_coeff : number of non-zero coefficients in the model weights
        """
        #Workspace 1.6.a
        #BEGIN
        #TODO: Fit the Elastic model and return the number of non-zero coefficients in the model weights.

        #Code here
        self._model.fit(X, y)

        #END
        return self._model.coef_

    def evaluate(self, test_x, test_y):
        """
        Compute Mean square error (MSE) between the predicted values and the actual values  of the test data
        Parameters
        ----------
        test_x: test features
        test_y: test target

        Returns MSE
        -------
        """
        #Workspace 1.6.b
        #BEGIN
        #TODO: predict based on the test features and return the mean_squared_error

        #Code here
        res = self._model.predict(test_x)

        #END
        return self.mse(test_y, res)

    @staticmethod
    def mse(y_true, y_hat):
        return np.mean((y_true-y_hat)**2)
```

```python
# Tests cell
r = Elastic(15)
print(r.fit(cloud_data.train_x, cloud_data.train_y))

```

    [-0.          0.          0.          0.          0.0067543  -0.
      0.          0.00148363  0.        ]

1.7 [5 points] For each of the datasets, produce 2 plots using `Elastic` :
  - The number non-zero coefficients versus $\alpha$
  - Mean Squared Error (MSE) on test set versus $\alpha$

Use $\alpha = 0.01, 0.05, 0.1, 0.2, 0.3, 0,5 , 1$  and  $\beta= 0.5$.

```python
alphas = [0.01, 0.05, 0.1, 0.2, 0.3]
#BEGIN Workspace 1.7

#code here
datasets = [cloud_data, forest_data]
for dataset in datasets:
    aset = [0.01, 0.05, 0.1, 0.2, 0.3]
    mses = []
    cfcount = []

    for a in aset:
        mod = Elastic(a)
        coeffs = mod.fit(dataset.train_x, dataset.train_y)
        mse = mod.evaluate(dataset.test_x, dataset.test_y)
        mses.append(mse)
        cfcount.append((coeffs != 0).sum())

    plt.plot(aset, mses)
    plt.xlabel("alpha")
    plt.ylabel("MSE")
    plt.title(f"Alpha vs MSE for {dataset.__class__.__name__}")
    plt.show()

    plt.plot(aset, cfcount)
    plt.xlabel("alpha")
    plt.ylabel("Nonzero coeffs")
    plt.title(f"Alpha vs Nonzero Coefficients for {dataset.__class__.__name__}")
    plt.show()

#END Workspace 1.2
```

1.8 [5 points] What are the pros and cons of each of three types of regressions we have implemented?

# Workspace 1.8
- Write-up answer here

## Problem 2: Logistic Regression for Binary Classification (45 points)
***
The second part of this assignment will be dealing with Logistic Regression. While the name "regression" suggests otherwise, Logistic Regression is actaully used for classification.
It's called regression because the model learns the continuous likelihood of multiple/a certain outcome.

Our dataset is a subset of the MNIST dataset, which is a higher resolution of the sklearn digits data seen in HW1.

In this problem you'll implement a Logistic Regression classifier to take drawings of either an eight
or a nine and output the corresponding label.

```python
class BinaryMNIST:
    """
    Class to store MNIST data for images of 9 and 8 only
    """
    def __init__(self, location='./data/mnist.pklz'):
        # You shouldn't have to modify this class, but you can if you'd like
        # Load the dataset
        with gzip.open(location, 'rb') as f:
            self.train_x, self.train_y, self.test_x, self.test_y = pickle.load(f)

        train_indices = np.where(self.train_y > 7)
        self.train_x, self.train_y = self.train_x[train_indices], self.train_y[train_indices]
        self.train_y = self.train_y - 8

        test_indices = np.where(self.test_y > 7)
        self.test_x, self.test_y = self.test_x[test_indices], self.test_y[test_indices]
        self.test_y = self.test_y - 8

binary_mnist = BinaryMNIST()
```

Logistic Regression is about minimizing the negative log likelihood objective is defined as:
\begin{align}
\textrm{NLL}(w) = -\frac{1}{N}\sum_{i=1}^N \left[y_i \log \sigma(w^T{x_i}) + (1-y_i)\log(1 - \sigma(w^Tx_i))\right]
\end{align}
where $\sigma$ is the *sigmoid function* seen in class.

Note that we're excluding the intercept since we'll be adding a constant column to the $(x_i)_i$.
We will call it the zero-th column and the intercept will be $w_0$.

The gradient of $NLL$ w.r.t $w$,  $\frac{\partial \textrm{NLL}}{\partial w}$

\begin{align}
\frac{\partial \textrm{NLL}}{\partial w} = \frac{1}{N} \sum_i \left[\sigma(w^Tx_i)-y_i)\right]x_i
\end{align}

- 2.1 [5 points] Finish the `calculate_score` function to return the output of applying
the dot product of the weights with the input parameter

- 2.2 [5 points] Finish the `sigmoid` function to return the output of applying the sigmoid function to the calculated score

- 2.3 [5 points] Finish the `compute_gradient` function to return the derivative of the cost w.r.t. the weights

- 2.4 [5 points] Finish the `batch_update` function so that it performs batch gradient descent using the provided batch data and updates the weight vector correspondingly

- 2.5 [5 points] Finish the `fit` function so that it iterates over the training epochs and returns the Recall score (in-built sklearn metrics) on the validation data at the end of each epoch.

```python
from sklearn.metrics import recall_score

class LogisticRegression:

    def __init__(self, eta = 0.1):
        """
        Create a Logistic regression classifier
        :param eta: Learning rate (the default is a constant value)
        """
        self.w = None      # our array of weights
        self.eta = eta # our learning rate
        self.X = None   # features matrix
        self.y = None   # labels / outcomes array

    def calculate_score(self, x):
        """
        :param x: This can be a single training example or it could be n training examples
        :return score: Calculate the score that you will plug into the logistic function
        """
        #Workspace 2.1
        # TODO: Compute the score to be fed to the sigmoid function
        #BEGIN
        #Code here
#         w.X + b

        return -self.w * x

        #END

    def sigmoid(self, score, threshold=25.0):
        """
        :param score: Either a real valued number or a vector to convert into a number between 0 and 1
        :param threshold : Capping activations at 25 prevent overflow of np.exp() function.
        :return sigmoid: Calculate the output of applying the sigmoid function to the score. This could be a single
        value or a vector depending on the input.
        """
        #BEGIN Workspace 2.2
        # TODO: Complete this function to return the output of applying the sigmoid function to the score
        #Code here
        return (1/(1 + np.exp(score))).round()

    def compute_gradient(self, x, h, y):
        """
        :param x: Feature vector
        :param h: the sigmoid of the scores computed from x
        :param y: real class label
        :return gradient: Return the derivative of the cost w.r.t to the weights
        """

        #Workspace 2.3
        # TODO: Finish this function to compute the gradient
        #BEGIN
        #Code here

        return np.dot(x.T, (h - y))/x.shape[0]

    def batch_update(self, batch_x, batch_y):
        """
        Single self.w update using the batch. We should average the gradient over the batch size
        :param batch_x: NumPy array of features (size : size of batch X features + 1 for the intercept)
        :param batch_y: Numpy array of class labels (size : size of batch )
        :return: nothing
        """

        #Workspace 2.4
        #BEGIN
        #Code here
        self.w = self.w - self.eta * self.compute_gradient(batch_x, self.sigmoid(batch_x), batch_y)

        #END

    def fit(self, X, y, epochs=1, batch_size=1, validation_X=None, validation_y=None):
        """
        :param X: training features
        :param y: training labels
        :param epochs: number of epochs
        :param batch_size: size of batch for gradient update
        :param validation_X: validation rows, should default to training data if not provided
        :param validation_y: validation labels
        :return: metric value at the end of each epoch on validation data
        """
        if validation_X is None:
            validation_X, validation_y = X, y
        metrics = []

        # Workspace  2.5
        # TODO: Process x to append the zero-th constant column
        # TODO: Compute average recall on the validation data at the end of each epoch
        # HINT: Don't forget to initialize your weights!

        #BEGIN
        #Code here
        self.w = np.zeros(X.shape)

        for epoch in range(epochs):
            for i in range((X.shape[0]-1)//batch_size + 1):
                # Defining batches. SGD.
                start_i = i*batch_size
                end_i = start_i + batch_size

                # Updating the parameters.
                self.batch_update(X[start_i:end_i], y[start_i:end_i])

        # Calculating loss and appending it in the list.
        l = loss(y, self.sigmoid(np.dot(X, self.w)))
        losses.append(l)

        #END

        return np.array(metrics)

    def predict(self, test_x):
        """
        :param test_x: n rows to predict on
        :return: n predicted labels
        """
        X = np.concatenate([np.ones((test_x.shape[0],1)), test_x], axis=1)
        return np.round(self.sigmoid(self.calculate_score(X))).astype(int)

    def optimize(self, batch_size):
        """
        Shuffle the training data (use permutation) and send batches to batch_update to update $self.w$
        Parameters
        :batch_size: size of batches to use for stochastic gradient
        :return: nothing
        """

        indices = np.random.permutation(len(self.X))
        for i in range(0, self.X.shape[0], batch_size):
            batch_x = self.X[indices[i:i+batch_size]]
            batch_y = self.y[indices[i:i+batch_size]]
            self.batch_update(batch_x, batch_y)
```

```python
# Tests cell
r = LogisticRegression(15)
print(r.fit(cloud_data.train_x, cloud_data.train_y))

```

After completing the class above, loop over the training data and perform batch training with `batch_size = 1` for `10 epochs`, and five different values of `eta`, i.e, [$.0001, .01, .1, .5, 1$]. Train your model and do the following:

2.6 [5 points] Create a new classifier object and Using the `fit` method, plot the recall trend for the different values of eta on the training data

```python
etas = [.0001,.001,.01, 1, 5]
np.random.seed(42)
for eta in etas:
#Workspace 2.6
#BEGIN
        #Code here
        raise NotImplementedError()

#END
```

- 2.7 [5 points] Create a new classifier object and Using the `fit` method, plot the values of recall versus eta

```python
etas = [.0001,.001,.01, 1, 5]
#Workspace 2.7
#BEGIN
        #Code here
        raise NotImplementedError()

#END
```

This time we want to analyze the effect of varying the batch size. We fix `eta = 0.1` and `epochs = 10` and we want to examine the recall on the test set at the end of the training for `batch_size` in [$1, 2, 4, 8, 12, 16$].

- 2.8 [5 points] Produce a plot of the recall at the end of the training as a function of the batch size

```python
batch_sizes= [1, 2, 4, 8, 12, 16]
recalls = []
#Workspace 2.8
#BEGIN
        #Code here
        raise NotImplementedError()

#END
```

2.9 [5 points] How does the learning rate (`eta`) and the number of epochs affect the performance of the classifier.

# Workspace 2.9
- Write-up answer here

2.BONUS [10 Points] Since we're done with the binary regression, we will try to add Ridge regularization:
\begin{align}
\textrm{NLL}(w) = -\frac{1}{N}\sum_{i=1}^N \left[y_i \log \sigma(w^T{x_i}) + (1-y_i)\log(1 - \sigma(w^Tx_i))\right] + \alpha {||w||^2}
\end{align}

We will need to add an `alpha` parameter in our `LogisticRegression` class (with a default value)

First, write the gradient formula in the cell below. Then, edit your `compute_gradient` to account for the regularization term. Ensure the rest of your functions still run correctly!

<2.BONUS Write your modified gradient formula here>


Sascha Spors,
Professorship Signal Theory and Digital Signal Processing,
Institute of Communications Engineering (INT),
Faculty of Computer Science and Electrical Engineering (IEF),
University of Rostock,
Germany

# Tutorial Digital Signal Processing

**FIR Filter**,
Winter Semester 2021/22 (Master Course #24505)

- lecture: https://github.com/spatialaudio/digital-signal-processing-lecture
- tutorial: https://github.com/spatialaudio/digital-signal-processing-exercises

Feel free to contact lecturer frank.schultz@uni-rostock.de

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.markers import MarkerStyle
from matplotlib.patches import Circle
from scipy import signal
```

```python
def zplane_plot(ax, z, p, k):
    """Plot pole/zero/gain plot of discrete-time, linear-time-invariant system.

    Note that the for-loop handling might be not very efficient
    for very long FIRs

    z...array of zeros in z-plane
    p...array of poles in z-zplane
    k...gain factor

    taken from own work
    URL = ('https://github.com/spatialaudio/signals-and-systems-exercises/'
           'blob/master/sig_sys_tools.py')

    currently we don't use the ax input parameter, we rather just plot
    in hope for getting an appropriate place for it from the calling function
    """
    # draw unit circle
    Nf = 2**7
    Om = np.arange(Nf) * 2*np.pi/Nf
    plt.plot(np.cos(Om), np.sin(Om), 'C7')

    try:  # TBD: check if this pole is compensated by a zero
        circle = Circle((0, 0), radius=np.max(np.abs(p)),
                        color='C7', alpha=0.15)
        plt.gcf().gca().add_artist(circle)
    except ValueError:
        print('no pole at all, ROC is whole z-plane')

    zu, zc = np.unique(z, return_counts=True)  # find and count unique zeros
    for zui, zci in zip(zu, zc):  # plot them individually
        plt.plot(np.real(zui), np.imag(zui), ms=8,
                 color='C0', marker='o', fillstyle='none')
        if zci > 1:  # if multiple zeros exist then indicate the count
            plt.text(np.real(zui), np.imag(zui), zci)

    pu, pc = np.unique(p, return_counts=True)  # find and count unique poles
    for pui, pci in zip(pu, pc):  # plot them individually
        plt.plot(np.real(pui), np.imag(pui), ms=8,
                 color='C0', marker='x')
        if pci > 1:  # if multiple poles exist then indicate the count
            plt.text(np.real(pui), np.imag(pui), pci)

    plt.text(0, +1, 'k={0:f}'.format(k))
    plt.text(0, -1, 'ROC for causal: white')
    plt.axis('square')
    plt.xlabel(r'$\Re\{z\}$')
    plt.ylabel(r'$\Im\{z\}$')
    plt.grid(True, which="both", axis="both",
             linestyle="-", linewidth=0.5, color='C7')

def bode_plot(b, N=2**10, fig=None):  # we use this here for FIRs only
    if fig is None:
        fig = plt.figure()

    a = np.zeros(len(b))  # some scipy packages need len(a)==len(b)
    a[0] = 1

    z, p, gain = signal.tf2zpk(b, a)
    W, Hd = signal.freqz(b, a, N, whole=True)

    print('number of poles:', len(p), '\npole(s) at:', p,
          '\nnumber of zeros:', len(z), '\nzero(s) at:', z)

    gs = fig.add_gridspec(2, 2)
    # magnitude
    ax1 = fig.add_subplot(gs[0, 0])
    ax1.plot(W/np.pi, np.abs(Hd), "C0",
             label=r'$|H(\Omega)|$)',
             linewidth=2)
    ax1.set_xlim(0, 2)
    ax1.set_xticks(np.arange(0, 9)/4)
    ax1.set_xlabel(r'$\Omega \,/\, \pi$', color='k')
    ax1.set_ylabel(r'$|H|$', color='k')
    ax1.set_title("Magnitude response", color='k')
    ax1.grid(True, which="both", axis="both",
             linestyle="-", linewidth=0.5, color='C7')

    # phase
    ax2 = fig.add_subplot(gs[1, 0])
    ax2.plot(W/np.pi, (np.angle(Hd)*180/np.pi), "C0",
             label=r'$\mathrm{angle}(H('r'\omega))$',
             linewidth=2)
    ax2.set_xlim(0, 2)
    ax2.set_xticks(np.arange(0, 9)/4)
    ax2.set_xlabel(r'$\Omega \,/\, \pi$', color='k')
    ax2.set_ylabel(r'$\angle(H)$ / deg', color='k')
    ax2.set_title("Phase response", color='k')
    ax2.grid(True, which="both", axis="both",
             linestyle="-", linewidth=0.5, color='C7')

    # zplane
    ax3 = fig.add_subplot(gs[0, 1])
    zplane_plot(ax3, z, p, gain)

    # impulse response
    N = 2**3  # here specially chosen for the examples below
    k = np.arange(N)
    x = np.zeros(N)
    x[0] = 1  # create a Dirac
    h = signal.lfilter(b, a, x)
    ax4 = fig.add_subplot(gs[1, 1])
    ax4.stem(k, h, linefmt='C0', markerfmt='C0o',
             basefmt='C0:', use_line_collection=True)
    ax4.set_xlabel(r'$k$')
    ax4.set_ylabel(r'$h[k]$')
    ax4.set_title('Impulse Response')
    ax4.grid(True, which="both", axis="both", linestyle="-",
             linewidth=0.5, color='C7')

def plot_windowed_FIR_design():
    hw = h*w
    W = np.arange(0, 2**10) * 2*np.pi / 2**10
    [_, H] = signal.freqz(h, a=1, worN=W)
    [_, Hw] = signal.freqz(hw, a=1, worN=W)
    plt.figure(figsize=(10, 10))
    plt.subplot(2, 1, 1)
    plt.plot(k, h, 'C3o-', label='rectangular windowed FIR h[k]')
    plt.plot(k, w, 'C7o-', label='Kaiser Bessel window w[k]')
    plt.plot(k, hw, 'C0o-', label='Kaiser-Bessel windowed FIR hw[k]')
    plt.xlabel('k')
    plt.title('Impulse responses and window')
    plt.legend()
    plt.grid(True)
    plt.subplot(2, 1, 2)
    plt.plot([W[0]/np.pi, W[-1]/np.pi], [0, 0], 'C7')
    plt.plot([W[0]/np.pi, W[-1]/np.pi], [-6, -6], 'C1')
    plt.plot([W[0]/np.pi, W[-1]/np.pi], [-21, -21], 'C3:')
    plt.plot([W[0]/np.pi, W[-1]/np.pi],
             [StopBandMaxLevel, StopBandMaxLevel], 'C0:')
    plt.plot([Wc/np.pi, Wc/np.pi], [StopBandMaxLevel, 0],
             'C1', label=r'-6dB @ $\Omega_c$')
    plt.plot(W/np.pi, 20*np.log10(np.abs(H)), color='C3',
             label='rectangular windowed FIR')
    plt.plot(W/np.pi, 20*np.log10(np.abs(Hw)), color='C0',
             label='Kaiser-Bessel windowed FIR')
    plt.xlim((0, 2))
    plt.yticks(np.arange(-6-12*8, 12, 12))
    plt.xlabel(r'$\Omega \,/\, \pi$')
    plt.ylabel(r'$20\lg|H(\Omega)|$ / dB')
    plt.title('Level response')
    plt.legend()
    plt.grid(True)

# some defaults for the upcoming code:
figsize = (12, 9)
```

# Filter Fundamentals

The transfer function of digital filters can be generally expressed in the $z$-domain as
\begin{equation}
H(z)=\frac{Y(z)}{X(z)} = \frac{\sum\limits_{m=0}^M b_mz^{-m}}{\sum\limits_{n=0}^N a_nz^{-n}}
=\frac{b_0z^0+b_1z^{-1}+b_2z^{-2}+...+b_Mz^{-M}}{a_0z^0+a_1z^{-1}+a_2z^{-2}+...+a_Nz^{-N}}
\end{equation}
with input $X(z)$ and output $Y(z)$.
Real input signals $x[k]$ that should end up as real output signals $y[k]$ (in terms of signal processing fundamentals this is a special case, though most often needed in practice) require real coefficients $b,a\in\mathbb{R}$.
This is only achieved with

- single or multiple **real** valued
- single or multiple **complex conjugate** pairs

of zeros and poles.

Furthermore, in practice we most often aim at (i) causal and (ii) bounded input, bound output (BIBO) stable LTI systems, which requires (i) $M \leq N$ and (ii) poles inside the unit circle.
If all poles **and** zeros are **inside** the unit circle then the system is **minimum-phase** and thus $H(z)$ is straightforwardly **invertible**.

Further concepts related to the transfer function are:

- Analysis of the transfer characteristics is done by the DTFT
$H(z=\mathrm{e}^{\mathrm{j}\Omega})$, i.e. evaluation on the unit circle.

- We use $a_0=1$ according to convention in many textbooks.

- The convention for arraying filter coefficients is straightforward with Python index starting at zero:
$b_0=b[0]$, $b_1=b[1]$, $b_2=b[2]$, ..., $a_0=a[0]=1$, $a_1=a[1]$, $a_2=a[2]$.

## Filtering Process

- A **non-recursive** system with $a_1,a_2,...,a_N=0$ always exhibits a **finite
impulse response** (FIR), note: $a_0=1$ for output though. Due to the finite length impulse response, a non-recursive system is always stable.

- The output signal of a **non-recursive** system in practice can be calculated by **linear
convolution**
\begin{equation}
y[k] = \sum\limits_{m=0}^{M} h[m] x[-m+k]
\end{equation}
of the finite impulse response $h[m]=[b_0, b_1, b_2,...,b_M]$ and the input signal $x[k]$.

- A **recursive system** exhibits at least one $a_{n\geq1}\neq0$. Because
of the feedback of the output into the system, a potentially **infinite impulse
response** (IIR) and a potentially non-stable system results.

- For a **recursive** system, in practice the **difference equation**
\begin{equation}
y[k] = b_0 x[k] + b_1 x[k-1] + b_2 x[k-2] + ... + b_M x[k-M] -a_1 y[k-1] - a_2 y[k-2] - a_3 y[k-3] - ... - a_N y[k-N]
\end{equation}
needs to be implemented.

- A **pure non-recursive** system is obtained by ignoring the feedback paths, i.e. setting $a_1,a_2,...,a_N=0$.
- A **pure recursive** system is obtained by ignoring the forward paths, i.e. setting $b_0,b_1,...,b_M=0$. Then, the values of the state variables $z^{-1}, z^{-2}, ..., z^{-M}$ alone determine how the system starts to perform at $k=0$, since the system has no input actually. This system type can be used to generate (damped) oscillations.

Please note: A recursive system can have a finite impulse response, but this is very rarely the case.
Therefore, literature usually refers to
- an FIR filter when dealing with a non-recursive system
- an IIR filter when dealing with a recursive system

## Signal Flow Chart of Direct Form I

For example, the signal flow for a **second order** ($M=N=2$), system with
- a non-recursive part (feedforward paths, left $z^{-}$-path)
- a recursive part (feedback paths, left $z^{-}$-path)

is depicted below (graph taken from Wikimedia Commons) as straightforward **direct form I**, i.e. directly following the difference equation.

Such as second order section is usually termed a biquad.

# FIR Filter

If all coefficients $a_{1,...,N}=0$, the feedback paths are not existent in the signal flow chart above.
This yields a non-recursive system and has transfer function

\begin{equation}
H(z) = \frac{Y(z)}{X(z)} = \sum\limits_{m=0}^M b_mz^{-m}
=b_0z^0+b_1z^{-1}+b_2z^{-2}+...+b_Mz^{-M}.
\end{equation}

with the difference equation

\begin{equation}
y[k] = b_0 x[k] + b_1 x[k-1] + b_2 x[k-2] + ... + b_M x[k-M],
\end{equation}

from which we can directly observe that the impulse response (i.e. for $x[k] = \delta[k]$) is

\begin{equation}
h[k] = b_0 \delta[k] + b_1 \delta[k-1] + b_2 \delta[k-2] + ... + b_M \delta[k-M].
\end{equation}

This constitutes $h[k]$ as the coefficients $b_k$ at sample instances $k$.

The impulse response for this non-recursive system has always finite length of $M+1$ samples.

Usually this filter type is referred to as finite impulse response (FIR) filter in literature.

Very special recursive systems/filters can produce FIRs as well. This is however so rare, that
the common link FIR filter == non-recursive system is predominantly made.

The filter **order** is $M$, the **number of coefficients** $b$ is $M+1$. Be cautious here and consistent with the naming, it sometimes gets confusing. Especially for linear phase filters (see below) it is really important if $M$ or $M+1$ is either even or odd.

Sometimes the **number of taps** $M+1$ is stated (rather than calling this number of coefficients). This refers to tapping $M+1$ delayed instances of the signal input signal $x$ to calculate the filtered output signal $y$. Note however, that tapping the signal for the first coefficient $b_0$ involves non-delayed $x[k]$.

For FIR filters the magnitude at $\Omega=0$ and $\Omega=\pi$ can be straightforwardly evaluated with the following equations.

The DC magnitude is obtained by

\begin{align}
g_0 = \sum_{k=0}^{M} h[k].
\end{align}

The magnitude at $\Omega=\pi$ (i.e. at half the sampling frequency) is obtained by

\begin{align}
g_\pi = \sum_{k=0}^{M} (-1)^k h[k].
\end{align}

## Poles / Zeros of FIR Filter

To calculate poles and zeros of the transfer function $H(z)$ it is meaningful to rewrite

\begin{equation}
H(z) = \sum\limits_{m=0}^M b_mz^{-m} \frac{z^M}{z^M}
=(b_0z^0+b_1z^{-1}+b_2z^{-2}+...+b_Mz^{-M}) \frac{z^M}{z^M}.
\end{equation}

This reveals that FIR filters have an $M$-fold pole at $z_\infty = 0$. This is always the case for non-recursive systems and besides the finite impulse response explains why these systems are always stable: poles in the origin are harmless, since they equally contribute to all frequencies.

For the zeros, the equation

\begin{equation}
\sum\limits_{m=0}^M b_m z^{-m} z^M = b_M z^M+b_1 z^{M-1}+b_2 z^{M-2}+...+b_M = 0
\end{equation}

needs to be solved. The $M$-th order polynomial has $M$ zeros. Recall from above, that for $b\in\mathbb{R}$ only real or complex conjugate zeros can occur, but never single complex zeros.

## Essence of FIR Filter Design

The fundamental concept (just as was it with window design for the DFT) is to place the $M$ available zeros in the $z$-plane such that a target magnitude **and** phase response results, which suits the desired filter characteristics.

It is important to note that (contrary to IIR filters) the **magnitude** and **phase** response can be **separately controlled** with FIR filters.

In the referenced [english monographs](../index.ipynb) we can find information on specific FIR design techniques such as

- FIR design with windowing method
- FIR design as minimax least-squares optimization problem
- FIR design of frequency sampling a DTFT spectrum

These are well covered in Python's Scipy and Matlab. We will later discuss the windowing method.

## A Note on FIR Filtering vs. DFT Windowing

Consider an input signal $x[k]$.

An FIR filter $h[k]$ is used for convolution (i.e. filtering process)

\begin{align}
x[k] \ast h[k] \circ-\bullet X(\mathrm{e}^{\mathrm{j}\Omega}) \cdot H(\mathrm{e}^{\mathrm{j}\Omega}),
\end{align}

whereas the DFT windowing process involves a multiplication with a window $w[k]$

\begin{align}
x[k] \cdot w[k] \circ-\bullet \frac{1}{2\pi} X(\mathrm{e}^{\mathrm{j}\Omega}) \circledast_{2\pi} W(\mathrm{e}^{\mathrm{j}\Omega}).
\end{align}

In the DTFT domain this results in multiplication and circular convolution, respectively.

So, for the finite-length sequences $h[k]$ and $w[k]$ the same design fundamentals hold: we must put zeros at suitable locations in the z-plane to realize a certain desired DTFT spectrum, either $H(\mathrm{e}^{\mathrm{j}\Omega})$ or $W(\mathrm{e}^{\mathrm{j}\Omega})$. Depending on the application, filtering or windowing, the DTFT design criteria might be very different, since the DTFT spectrum acts as multiplication or convolution onto the input signal's DTFT spectrum. However, the design concepts and algorithms itself are basically the same, this is sometimes not so obvious in textbooks.

# FIR Examples with M=1 and M=2

It is meaningful to discuss a very simple case of FIR in detail in the first place.
Once getting familiar with the underlying principles and concepts it is comparably easy to increase the FIR filter order and see how complexity evolves.

Almost all important issues on FIRs can be explained with

- the filter order $M=1$, thus number of coefficients is $M+1=2$ and with
- the filter order $M=2$, thus number of coefficients is $M+1=3$.

The calculus of zeros is not too tedious and can be still performed manually. Furthermore, the impact of $M$ zeros in the $z$-plain is comparably easy linked to the magnitude and phase response.

So, let's play around with some simple coefficient settings.

## Example FIR M=1, b0=1, b1=1

This is the most simple case of an FIR (actually the most simple FIR would be using only $b_0$, which is a pure gain/attenuation for input signal).

The squared magnitude response can be given analytically as (try yourself, make use of $\mathrm{e}^{-\mathrm{j}\Omega}=\cos(\Omega)-\mathrm{j}\sin(\Omega)$ )

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})|^2 = |1 + \mathrm{e}^{-\mathrm{j}\Omega}|^2 = 2 \cos(\Omega) + 2 = 4 \cos^2(\frac{\Omega}{2}).
\end{equation}

Thus the magnitude response is

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})| = 2 |\cos(\frac{\Omega}{2})|,
\end{equation}

which is confirmed by the below plot (left, top). The magnitude response exhibits lowpass characteristics.

The impulse response is simply

\begin{align}
h[k] = b_0 \delta[k] + b_1\delta[k-1],
\end{align}

confirmed by the plot (right, bottom).

For the $M$-th order polynomial the $M=1$ zero is easy to evaluate

\begin{equation}
b_0 z^1+b_1 z^{0} = z+1 = 0\to z_{0,1} = -1
\end{equation}

There is $M=1$ pole in the origin, i.e. $z_{\infty,1} = 0$. Zero and pole are shown in the $z$-plane (right, top).

This FIR has special characteristics on the phase response, namely it is linear phase (type II), see discussion below.

In the present case DC magnitude is $g_0=2$ and $f_s/2$-magnitude is $g_\pi=0$.

```python
b = [1, 1]  # linear phase FIR Type II
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example FIR M=1, b0=1, b1=-1

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})|^2 = 4 \sin^2(\frac{\Omega}{2}).
\end{equation}

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})| = 2 |\sin(\frac{\Omega}{2})|.
\end{equation}

$z_{\infty,1}=0$,
$z_{0,1}=1$

$g_0=0$,
$g_\pi=2$

Linear phase type IV

This is a simple **highpass**, it performs the difference between two adjacent samples (see the impulse response).

```python
b = [1, -1]  # linear phase FIR Type IV
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example FIR M=2, b0=1, b1=0, b2=1

Filter order $M=2$, number of coefficients $M+1=3$, although one coefficient is zero, namely $b_1=0$.

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})|^2 = 4 \cos^2(\Omega).
\end{equation}

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})| = 2 |\cos(\Omega)|.
\end{equation}

double pole in origin $z_{\infty,1,2}=0$

conjugate-complex pair $z_{0,1,2}=\pm \mathrm{j}$

$g_0=2$,
$g_\pi=2$

Linear phase type I

This is a simple **bandstop**.

```python
b = [1, 0, 1]  # linear phase FIR Type I, the zero in between counts as coeff
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example FIR M=2, b0=1, b1=0, b2=-1

Filter order $M=2$, number of coefficients $M+1=3$, although one coefficient is zero, namely $b_1=0$.

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})|^2 = 4 \sin^2(\Omega).
\end{equation}

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})| = 2 |\sin(\Omega)|.
\end{equation}

double pole in origin $z_{\infty,1,2}=0$

two single real zeros $z_{0,1,2}=\pm 1$

$g_0=0$,
$g_\pi=0$

Linear phase type III

This is a simple **bandpass**.

```python
b = [1, 0, -1]  # linear phase FIR Type III, the zero in between counts as coeff
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example FIR M=2, b0=1, b1=2, b2=1

Filter order $M=2$, number of coefficients $M+1=3$.

The manual derivation of analytic magnitude response starts to get tedious, however:

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})|^2 = 16 \cos^4(\Omega).
\end{equation}

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})| = 4 \cos^2(\Omega).
\end{equation}

double pole in origin $z_{\infty,1,2}=0$

double real zero $z_{0,1,2}=-1$

$g_0=4$,
$g_\pi=0$

Linear phase type I

This is a simple **lowpass**, with a little smoother characteristics.

```python
b = [1, 2, 1]  # linear phase FIR Type I
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example FIR M=2, b0=1, b1=-2, b2=1

By reversing the sign for $b_1$ compared to the just discussed lowpass we obtain a highpass.

Filter order $M=2$, number of coefficients $M+1=3$.

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})|^2 = 16 \sin^4(\Omega).
\end{equation}

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})| = 4 \sin^2(\Omega).
\end{equation}

double pole in origin $z_{\infty,1,2}=0$

double real zero $z_{0,1,2}=0$

$g_0=0$,
$g_\pi=4$

Linear phase type I

This **highpass** has also slightly smoother characteristics.

```python
b = [1, -2, 1]  # linear phase FIR Type I
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example FIR M=2, b0=1, b1=1, b2=1/2

So, far all zeros were aligned **on** the unit circle. We are not restricted to those locations, as long as we ensure positioning only real and complex-conjugate pairs in the $z$-plane.

However, recall that we discussed so called optimum window design for DFT-based spectral analysis.
There, zeros **on** the unit circle was a good idea, since the zero has most impact to shape the amplitude of the sidelobes. If a window has all zeros on the unit circle, we called this **optimum window**.

Back to our FIR design example:

Filter order $M=2$, number of coefficients $M+1=3$.

double pole in origin $z_{\infty,1,2}=0$

conjugate-complex pair $z_{0,1,2}=-\frac{1}{2}\pm \frac{1}{2}\mathrm{j}$

$g_0=\frac{5}{2}$,
$g_\pi=\frac{1}{2}$

This 2nd order FIR **lowpass** becomes a little more complicated and has a ripple in the stopband. Most important, since the zeros are **not** on the unit circle the magnitude response exhibits **no** exact zeros. Second morst important: the impulse response does not have a symmetry of linear-phase filter types I-IV, thus it is a **non-linear-phase** FIR. In fact, since poles and zeros are all within unit circle, the filter is minimum-phase.

Try to find and plot the inverse transfer function $H(z)^{-1}$. Why this must be a minimum phase filter as well and why is this an IIR filter. The code in Jupyter notebook `iir_filter.ipynb` might be helpful.

```python
b = [1, 1, 1/2]  # NON linear phase FIR, since there is no symmetry in the IR
# minimum phase since all poles and zeros are inside unit circle
# this system has a stable inverse (pole/zero reversal, 1/gain factor)
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example FIR M=2, b0=1/2, b1=1, b2=1

We can even align zeros outside the unit circle, since contrary to poles this has no impact on stability.

Filter order $M=2$, number of coefficients $M+1=3$.

double pole in origin $z_{\infty,1,2}=0$

conjugate-complex pair $z_{0,1,2}=-1\pm \mathrm{j}$

$g_0=\frac{5}{2}$,
$g_\pi=\frac{1}{2}$

This 2nd order FIR **lowpass** has the same magnitude response as the above lowpass. This is due to the fact that in both cases their zeros have same distance to the unit circle.

Obviously, this filter is also non-linear-phase, but also **not minimum-phase**. In fact, the filter is **maximum-phase**, i.e. the largest phase excess for the given magnitude response. Due to its maximum-phase, this FIR cannot simply be inverted, since then poles would lie outside the unit circle, yielding a non-stable system.

```python
b = [0.5, 1, 1]  # NON linear phase FIR, since there is no symmetry in the IR
# same magnitude as b = [1, 1, 1/2]
# but also NON minimum phase since zeros outside unit-circle
# rather maximum-phase
# this system has NO stable inverse
bode_plot(b, fig=plt.figure(figsize=figsize))
```

## Example Playground

Try some other coefficient settings by yourself, use larger $M$. Note that the plot of $h[k]$ is hard coded up to $k=7$ only.

For example, how about a lowpass filter where the coefficients come from the Kaiser-Bessel window?!?!

Do you see the link to the [Pole/Zeros Plots of Window Functions](https://nbviewer.jupyter.org/github/spatialaudio/digital-signal-processing-exercises/blob/outputs/dft/window_zplane_frequency_response.ipynb) Jupyter notebook?

```python
# 7 coeff, beta such that DC gain about 5
b = signal.kaiser(7, beta=2.15, sym=True)
bode_plot(b, fig=plt.figure(figsize=figsize))
np.sum(b)
```

Or another lowpass using the rectangular window. Yes, the rectangular window has lowpass characteristics, in literature this is known as the most simplest form of running average. In the example below, the last 7 samples are taken into account for averaging with equal weights.

```python
# 7 coeff, DC gain = 5
b = np.ones(7)/7*5
bode_plot(b, fig=plt.figure(figsize=figsize))
np.sum(b)
```

# FIR Filters with Linear Phase

In the above examples a very important concept of FIR filters was already included, which we did not discuss in detail so far. We do this here: the **linear phase**. In practice we can realize this only in digital signal processing, since analog circuits do not have the precision that would be required to shape the required impulse response characteristics (we talk about certain symmetries here as we will see below).

A linear-phase FIR filter exhibits the DTFT spectrum

\begin{equation}
H(\mathrm{e}^{\mathrm{j}\Omega}) = A(\mathrm{e}^{\mathrm{j}\Omega})\,\mathrm{e}^{-\mathrm{j}\,\alpha\,\Omega}\,\mathrm{e}^{\mathrm{j}\,\beta}
\end{equation}

with the magnitude spectrum $A(\mathrm{e}^{\mathrm{j}\Omega})\in\mathbb{R}$ and the phase $\mathrm{e}^{-\mathrm{j}\,\alpha\,\Omega}\,\mathrm{e}^{\mathrm{j}\,\beta}$ with $\alpha,\beta\in\mathbb{R}^+$.

There are four different basic types of linear-phase FIR filters that differ by the symmetry of the impulse response and the length of the finite impulse response.

The constant group delay in samples for all FIR filter types is $\frac{M}{2}=\text{const}$, which leads to half sample values for odd $M$.

Most books follow this numbering, although this is not a strict standardization, so please be careful and rather check the FIR characteristics rather than the type number!

## FIR Type I

- filter order $M$ even, odd number $M+1$ of filter coefficients $b$
- even symmetry of the impulse response $h[k]=h[M-k]$
- $\beta=0,\,\pi$
- No fixed zeros, therefore all filter types are possible

## FIR Type II

- filter order $M$ odd, even number $M+1$ of filter coefficients $b$
- even symmetry of the impulse response $h[k]=h[M-k]$
- $\beta=0,\,\pi$
- Fixed zero $H(z=-1)=0$, i.e. zero at $\Omega=\pi$, $f=\frac{f_s}{2}$.
- Therefore, only a lowpass or a bandpass can be realized properly.

## FIR Type III
- filter order $M$ even, odd number $M+1$ of filter coefficients $b$
- odd symmetry of the impulse response $h[k]=-h[M-k]$
- $\beta=\frac{\pi}{2},\,\frac{3}{2}\pi$
- Fixed zeros $H(z=1)=0$ and $H(z=-1)=0$, i.e. zeros at $\Omega=0$, $f=0$ and
$\Omega=\pi$, $f=\frac{f_s}{2}$.
- Therefore, only a bandpass can be realized properly.

## FIR Type IV
- filter order $M$ odd, even number $M+1$ of filter coefficients $b$
- odd symmetry of the impulse response $h[k]=-h[M-k]$
- $\beta=\frac{\pi}{2},\,\frac{3}{2}\pi$
- Fixed zero $H(z=1)=0$, i.e. zero at $\Omega=0$, $f=0$.
- Therefore, only a highpass or a bandpass can be realized properly.

Bandpasses are possible with all four FIR filter types.

Since FIR type I has no restrictions it might be the favorable choice, except you wish to place zeros explicitly at $\Omega=0,\pi$. Then the other types might be of interest.

# Windowed FIR Design Low-Pass

Now, let us discuss one potential FIR design method that is straightforward and comes typically first in teaching & learning DSP. It is still used in practice, especially when low computational complexity is aimed for.
The basic idea is to cut a suitable finite-length sequence out of an infinite impulse response of the ideal filter and to apply a window onto the finite-length sequence in order to reduce certain artifacts. The most simple case is using the rect window.

So, let's discuss the technique with the ideal lowpass filter: the ideal lowpass (i.e. zero-phase and infinite impulse response) with cutoff frequency $0 < \Omega_c < \pi$ is given by inverse DTFT as

\begin{equation}
h[k] = \frac{1}{2 \pi} \int\limits_{-\Omega_c}^{+\Omega_c} 1 \cdot \mathrm{e}^{+\mathrm{j} \Omega k} \mathrm{d}\Omega.
\end{equation}

The analytic solution is

\begin{equation}
h[k]=\frac{\sin\left(\Omega_c k \right)}{\pi k} = \frac{\Omega_c}{\pi}\frac{\sin\left(\Omega_c k \right)}{\Omega_c k},
\end{equation}

i.e. a weighted sinc-function. We should have expected a sinc-like shape, since rect and sinc correspond in terms of the Fourier transform.

In order to obtain a practical (finite order M) and causal (peak at M/2) we can shift this impulse response by $M/2$ (time delay)

\begin{equation}
h[k]=\frac{\sin\left(\Omega_c\left(k-\frac{M}{2}\right)\right)}{\pi\left(k-\frac{M}{2}\right)}
\end{equation}

and consider only the values for $0\leq k \leq M$, so filter order $M$, filter length $M+1$. Furthermore, we require that $M$ is even to obtain a linear-phase type I FIR filter. Then we can simplify

\begin{equation}
h[k]=\begin{cases}
\frac{\sin\left(\Omega_c\left(k-\frac{M}{2}\right)\right)}{\pi\left(k-\frac{M}{2}\right)} &\quad k\neq \frac{M}{2}\\
\frac{\Omega_c}{\pi} &\quad k=\frac{M}{2}.
\end{cases}
\end{equation}

The plain cut out (i.e. a rectangular window) towards a FIR yields

- on the one hand, the steepest roll-off for $\Omega>\Omega_c$ towards the first zero in the magnitude response
- but on the other hand also the worst stop band damping in the magnitude response (maximum stopband level is only about -21 dB)

Most often, larger maximum stopband level is desired under acceptance of less steep initial roll-off. This can be achieved with other window functions. A very well suited window for this filter design task is the Kaiser-Bessel window. Kaiser figured out that a certain maximum stopband level (usually this is here the first side lobe) can be controlled by the parameter $\beta$. This of course holds only if $M$ is chosen large enough to obtain such a damping at all.

For a maximum stopband level $\gamma<-50$ in dB, the approximation

\begin{equation}
\beta_\mathrm{Kaiser-Bessel-Window} = -0.1102\,(\gamma+8.7)
\end{equation}

was invented.

Then, with an accordingly designed window for $0\leq k \leq M$, the improved FIR is given as

\begin{equation}
h_w[k] = w[k] \cdot h[k],
\end{equation}

hence the design method's naming.

In the examples below we use a Kaiser-Bessel window to control that stopband level does not exceed -54 dB.

We design linear-phase type I FIR filters with the windowing method.

Note, that contrary to the typical -3dB cut-off frequency definition for analog filters, here the **cut-off frequency is defined at -6 dB**!

```python
# we require even! FIR order
# thus odd number of coefficients, linear-phase type I
M = 2**5  # -> 33 coeff
k = np.arange(M+1)
Wc = 2*np.pi * 1/4  # desired cut-off frequency
```

```python
StopBandMaxLevel = -54  # < -50
beta = -0.1102*(StopBandMaxLevel+8.7)
# beta = 0  # beta = 0 is equal to the rectangular window!
print('beta =', beta)
w = signal.kaiser(M+1, beta, sym=True)
```

```python
h = np.sin(Wc*(k-M//2)) / (np.pi*(k-M//2))
h[M//2] = Wc / np.pi
plot_windowed_FIR_design()
```

# Windowed FIR Design High-Pass

We can do the same approach for the ideal highpass filter:

The ideal highpass (i.e. zero-phase and infinite impulse response) with cutoff frequency $0 < \Omega_c < \pi$ is given by inverse DTFT as

\begin{equation}
h[k] = \frac{1}{2 \pi} \int\limits_{\Omega_c}^{2\pi-\Omega_c} 1 \cdot \mathrm{e}^{+\mathrm{j} \Omega k} \mathrm{d}\Omega.
\end{equation}

The analytic solution is

\begin{equation}
h[k]=\frac{\sin\left(\pi k \right)}{\pi k}-\frac{\sin\left(\Omega_c k\right)}{\pi k}.
\end{equation}

In order to obtain a practical (finite order M) and causal (peak at M/2) we can shift this impulse response by $M/2$ (time delay)

\begin{equation}
h[k]=\frac{\sin\left(\pi\left(k-\frac{M}{2}\right)\right)}{\pi\left(k-\frac{M}{2}\right)}-\frac{\sin\left(\Omega_c\left(k-\frac{M}{2}\right)\right)}{\pi\left(k-\frac{M}{2}\right)}
\end{equation}

and consider only the values for $0\leq k \leq M$, so filter order $M$, filter length $M+1$. Furthermore, we require that $M$ is even to obtain linear-phase type I. Then we can simplify

\begin{equation}
h[k]=\begin{cases}
-\frac{\sin\left(\Omega_c\left(k-\frac{M}{2}\right)\right)}{\pi\left(k-\frac{M}{2}\right)} &\quad k\neq \frac{M}{2}\\
1-\frac{\Omega_c}{\pi} &\quad k=\frac{M}{2}.
\end{cases}
\end{equation}

Again, with an accordingly designed window for $0\leq k \leq M$, the improved FIR is given as

\begin{equation}
h_w[k] = w[k] \cdot h[k].
\end{equation}

Kaiser-Bessel window is also perfectly suited for the highpass. So we use precisely the same window as for the lowpass.

```python
h = - np.sin(Wc*(k-M//2)) / (np.pi*(k-M//2))
h[M//2] = 1 - Wc/np.pi
plot_windowed_FIR_design()
```

Play around with $M$, $\Omega_c$, `StopBandMaxLevel` ($\beta$) and how they are linked. Then, you're ready to study FIR design methods in books in detail.

It is also worth to check the pole/zero plot of these lowpass / highpass filters.

# **Copyright**

The notebooks are provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebooks for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Frank Schultz, Digital Signal Processing - A Tutorial Featuring Computational Examples* with the URL https://github.com/spatialaudio/digital-signal-processing-exercises


```python
import numpy as np
import pandas as pd
import scipy.stats as st
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
```

## Basic HCM sampling

This notebook shows the HMC sampling method for a multivariate Gaussian distribution and compares the obtained samples to generated samples of the RWMH algorithm.

## Data

We will use 800 samples of a multivaraite Gaussian distribution with the following mean vector

\begin{equation}
\mu = \begin{pmatrix} 0 \\ 0\\ 0 \end{pmatrix}
\end{equation}

and covariance matrix

\begin{equation}
\Sigma = \begin{pmatrix} 1.5 & 0.85 & 0.55 \\ 0.85 & 1.5 & -0.5 \\ 0.55 & -0.5 & 1.0 \end{pmatrix}
\end{equation}

```python
μ = np.zeros(3)
Σ = np.array([[1.5,.85,0.55],[.85,1.5,-0.5],[0.55,-0.5,1.0]])
Y = pd.DataFrame(np.random.multivariate_normal(μ,Σ,size=800),columns=['x_1','x_2','x_3'])
Y.describe()
```

      x_1
      x_2
      x_3

      count
      800.000000
      800.000000
      800.000000

      mean
      -0.017733
      0.029691
      -0.039493

      std
      1.188286
      1.217572
      1.006855

      min
      -5.105062
      -4.250228
      -3.172033

      25%
      -0.814859
      -0.755122
      -0.734122

      50%
      0.057809
      0.045333
      -0.046366

      75%
      0.847174
      0.788344
      0.637828

      max
      3.449742
      3.985105
      4.043018

We define the multivariate normal density function $f(x|\mu,\Sigma)$, the energy function $U(x)$ and the gradient of the energy funtion.

We set the energy function to the negative log-likelihood
$$ U(x)=-\log{f(x)}$$

```python
def f(x,μ,Σ):
    n = μ.shape[0]
    mvn = (2*np.pi)**(-n/2)*np.linalg.det(Σ)**(-1/2)*np.exp(-0.5*(x-μ).T@np.linalg.inv(Σ)@(x-μ))
    return mvn

def U(x,μ,Σ):
    return -np.log(f(x,μ,Σ))

def grad_U(x,μ,Σ):
    return np.linalg.inv(Σ)@(x-μ)
```

## Implementation of HMC - can be found in https://arxiv.org/pdf/1206.1901.pdf

```python
def HMC(L,ϵ,mu,sigma,grad_U,U,init_pos,epochs=800, reproducible=True):
    counter = 0
    samples = [init_pos]
    #initially draw 800 mvn samples with given random state for reproducible results
    if reproducible:
        np.random.seed(404)
        P_draw = np.random.multivariate_normal(np.zeros(len(init_pos)),np.eye(len(init_pos)),size=800)

    for e in range(800):
        #current position
        q_current = np.copy(samples[-1])
        q_prop = np.copy(q_current)
        if reproducible:
            p_current = P_draw[e,:]
        else:
            #sample momentum variable
            p_current = np.random.multivariate_normal(np.zeros(len(init_pos)),np.eye(len(init_pos)))
        p_prop = np.copy(p_current)
        dU = grad_U(q_prop,mu,sigma)
        # half step for momentum
        p_prop-=ϵ*dU/2
        for s in range(L):
            #full step for position
            q_prop+= ϵ*p_prop
            if s !=(L-1):
                # full step for momentum execpt at end of trajectory
                p_prop-= ϵ*grad_U(q_prop,mu,sigma)
        #half step for momentum at end
        p_prop-=ϵ*grad_U(q_prop,mu,sigma)/2
        # negate momentum
        p_prop = -1*p_prop

        #Calculate U(q) and K(p)
        U_current = U(q_current,mu,sigma)
        U_prop = U(q_prop,mu,sigma)
        K_current = 0.5*p_current.T@p_current
        K_prop = 0.5*p_prop.T@p_prop
        #accept or reject
        event = np.random.uniform(0,1)
        if event <= np.exp(U_current-U_prop+K_current-K_prop):
            counter +=1
            samples.append(q_prop)
        else:
            samples.append(q_current)
    return (np.asarray(samples),counter/800)
```

## Trace plot of HMC samples

```python
samples,_ = HMC(15,0.40,μ,Σ,grad_U,U,np.array([-1,-1.55,1.0]))
print("Acceptance rate of {}%".format(_*100))
fig3, axes = plt.subplots(3,figsize=(10,14))
axes[0].plot(range(samples.shape[0]), samples[:,0],color="royalblue");
axes[0].set_ylabel('first coordinate')
axes[1].plot(range(samples.shape[0]), samples[:,1],color="royalblue");
axes[1].set_ylabel('second coordinate')
axes[2].plot(range(samples.shape[0]), samples[:,2], color="royalblue");
axes[2].set_ylabel('third coordinate');
axes[2].set_xlabel('iteration');
```

```python
def plot(samples):
    fig = plt.figure(figsize=(10,10))
    ax1=plt.subplot2grid( (3,3), [0,0], 1, 1 )
    sns.histplot(data=Y,x="x_1",ax=ax1,color='black',alpha=0.4,label='True values')
    sns.histplot(data=samples,x="x_1",color='royalblue',ax=ax1,alpha=0.6,label='Sampled values')
    ax1.set_xlabel('')
    #ax1.legend()
    ax2=plt.subplot2grid( (3,3), [1,0], 1, 1 )
    sns.kdeplot(data=Y,x='x_1',y='x_2',ax=ax2,color='black',alpha=0.6)
    sns.scatterplot(data=samples,x='x_1',y='x_2',ax=ax2,color='royalblue',linewidth=0.25)
    ax2.set_xlabel('')
    #plt.plot( x2, y2 )
    #plt.subplot2grid( (3,3), [1,0], 1, 1 )
    ax3=plt.subplot2grid( (3,3), [1,1], 1, 1 )
    sns.histplot(data=Y,x="x_2",ax=ax3,color='black',alpha=0.4)
    sns.histplot(data=samples,x="x_2",color='royalblue',ax=ax3,alpha=0.6)
    ax3.set_ylabel('')
    ax3.set_xlabel('')
    ax4=plt.subplot2grid( (3,3), [2,0], 1, 1 )
    sns.kdeplot(data=Y,x='x_1',y='x_3',ax=ax4,color='black',alpha=0.6)
    sns.scatterplot(data=samples,x='x_1',y='x_3',ax=ax4,color='royalblue',linewidth=0.25)
    ax5=plt.subplot2grid( (3,3), [2,1], 1, 1 )
    sns.kdeplot(data=Y,x='x_2',y='x_3',ax=ax5,color='black',alpha=0.6)
    sns.scatterplot(data=samples,x='x_2',y='x_3',ax=ax5,color='royalblue',linewidth=0.25)
    ax5.set_ylabel('')
    ax6=plt.subplot2grid( (3,3), [2,2], 1, 1 )
    sns.histplot(data=Y,x="x_3",ax=ax6,color='black',alpha=0.4)
    sns.histplot(data=samples,x="x_3",color='royalblue',ax=ax6,alpha=0.6,linewidth=1)
    ax6.set_ylabel('')
    handles, labels = ax1.get_legend_handles_labels()
    fig.legend(handles, labels,loc='center right',bbox_to_anchor=[0.82,0.59]);
    return fig
```

## Comparison of HMC samples to samples which were directly drawn from the target distribution

```python
samples_df = pd.DataFrame(samples,columns=['x_1','x_2','x_3'])
fig4 = plot(samples_df)
```

## RWMH algorithm

```python
init = np.array([-1,-1.55,1.0])
n= 800
MCMC = [init]
s = init
accepted = 0
for i in range(n):
    s = MCMC[-1]
    #generate proposal sample
    y = np.random.multivariate_normal(s,1.0*np.eye(3))
    #compute acceptance rate
    α = f(y,μ,Σ)/f(s,μ,Σ)
    #accepte or reject
    if α > np.random.uniform(0,1):
        #if accepted append proposal sample to list of samples
        MCMC.append(y)
        accepted +=1
    else:
        #if rejected append current state to list of samples
        MCMC.append(s)
print("Acceptance rate of {}%".format(accepted/n*100))
```

    Acceptance rate of 22.625%

```python
samples_hm = pd.DataFrame(np.asarray(MCMC),columns=['x_1','x_2','x_3'])
```

## Autocorrelation of RWMH and HMC samples

```python
fig3 ,ax = plt.subplots(2,figsize=(15,10))
for variable in samples_hm.columns:
    pd.plotting.autocorrelation_plot(samples_hm[variable], label = variable,ax=ax[0])
for variable in samples_df.columns:
    pd.plotting.autocorrelation_plot(samples_df[variable], label = variable,ax=ax[1])
```


```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import pandas as pd
import sympy as sy
import statsmodels.api as sm

from sympy import init_printing
init_printing()

plt.style.use('ggplot')
```

    E:\Anaconda\lib\site-packages\statsmodels\compat\pandas.py:65: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
      from pandas import Int64Index as NumericIndex

This chapter focuses on the features of sampling distribution of OLS estimator. We would like to know that if OLS can provide us with unbiased, consistent and efficient estimates.

#  Unbaisedness of OLS

In this chapter, we assume regression model has the form

$$
\boldsymbol{y} = \boldsymbol{X\beta} +\boldsymbol{u}, \quad \boldsymbol{u}\sim \text{IID}(\boldsymbol{0},\sigma^2\mathbf{I})
$$

The notation $\text{IID}$ means **identically independent distributed**, $\sigma^2\mathbf{I}$ is the **covariance matrix**, being a diagonal matrix means regressors are independently distributed. We also assume all diagonal elements are the same, i.e. $\sigma^2$, which means **homoscedasticity**.

If an estimator $\theta$ is unbiased, it should always satisfy the condition

$$
E(\hat{\theta}) -\theta_0 = 0
$$

To show that the OLS estimator is unbiased, we denote $\boldsymbol{\beta}_0$ as the true parameter in the _data generating process_ (DGP), that is to say, we would like to see $E(\boldsymbol{\hat{\beta}})=\boldsymbol{\beta}_0$.

To show the conditions that makes OLS unbiased, we substitute DGP back in OLS formula

\begin{align}
\hat{\boldsymbol{\beta}}&= (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\\
&=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T(\boldsymbol{X}\boldsymbol{\beta}_0+u)\\
& = \boldsymbol{\beta}_0 + (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{u}
\end{align}

First, we assume $\boldsymbol{X}$ to be _nonstochastic_, also with assumption $E(\boldsymbol{u}) = 0$, we obtain

\begin{align}
E(\boldsymbol{\hat{\beta}}) &= \boldsymbol{\beta}_0 + (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{u}\\
 &= \boldsymbol{\beta}_0 + (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^TE(\boldsymbol{u})\\
 &=\boldsymbol{\beta}_0
\end{align}
However nonstochastic assumption is limited mostly in cross section data.

Second, we can assume $\boldsymbol{X}$  **exogenous**, such that $E(\boldsymbol{u}|\boldsymbol{X}) = 0$, also proves $E(\boldsymbol{\hat{\beta}})=\boldsymbol{\beta}_0$. Exogeneity simply means that the randomness of $\boldsymbol{X}$ has nothing to do with $\boldsymbol{u}$.

We have even weaker version of exogeneity, $E(u_t|\boldsymbol{X}_t) = 0$, which excludes the possibility that $u_t$ might depend on $X_{t-1}$ or $X_{t-2}$, etc. It is called **predeterminedness condition**, which is suitable for time series data.

However, for time series data, OLS shall be used with cautions, it is widely known that OLS is not suitable for $\text{ARMA}$ model, because $\boldsymbol{y}$ has lagged dependent variables.

For $\text{VAR}$, OLS is a common practice, because it doesn't differentiate endogenous and exogenous variables.

Predetermined Variable vs Exogenous Variable

In strictly speaking, predetermined variables has two categories: exogenous variables and predetermined endogenous variables.

The former one can either be current or lagged, as long as it doesn't evolve with model's dynamics. The latter one is common in autoregressive models.

#  Consistency of OLS

Consistency means the estimates tends to the quantity of true parameters as sample tends to infinity. Now let's show why OLS estimator is consistent. Start from

$$
\boldsymbol{\hat{\beta}} = \boldsymbol{\beta}_0 + (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{u} \label{1}\tag{1}
$$

We would to like to know if $\text{plim}\boldsymbol{\hat{\beta}} = \boldsymbol{\beta}_0$, which boils down to verify if

$$
\text{plim}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{u} =0 \label{2}\tag{2}
$$

However $\boldsymbol{X}^T\boldsymbol{X}$ and $\boldsymbol{X}^Tu$ do not have probability limit even if $n \rightarrow \infty$.

Modify the probability limit we obtain

$$
\bigg(\text{plim}\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}\bigg)^{-1}\text{plim}\frac{1}{n}\boldsymbol{X}^T\boldsymbol{u} =0
$$

We assume a weaker version of exogeneity $E(u_t|X_t) = 0$, so is $E(X_t^Tu_t|X_t)=0$, law of iterated expectation tells that $E(X_t^Tu_t)=0$. To summarize

$$
\text{plim}\frac{1}{n}\boldsymbol{X}^T\boldsymbol{u}=\text{plim}\frac{1}{n}\sum_{t=1}^nX_t^T u_t = 0
$$

which proves the consistency of OLS.

##  Simulation of Consistency of OLS

Here we can demonstrate a simple Monte Carlo simulation with true parameters $\boldsymbol{\beta}_0 = [2, 3, 4, 5]^T$, we increase sample size from $5$ to $100$ with a unit increment in each loop, and also with each sample size we redraw the disturbance term $10$ times, you can experiment with codes below, they might not be concise but considerably intuitive. Pay attention to the plots how they approach the true parameters as sample sizes increase.

```python
sampl_size = np.arange(5, 100)
re_draw = 10

beta_array = np.array([2, 3, 4, 5])
beta_array = beta_array[np.newaxis, :].T

beta_hat_mean = []
for i in sampl_size:
    const = np.ones(i)
    const = const[np.newaxis, :]

    X_inde = np.random.randn(3, i)
    X = np.concatenate((const.T, X_inde.T), axis=1)

    beta_hat = []
    for j in range(re_draw):
        u = np.random.randn(i)
        u = u[np.newaxis, :].T
        y = X@beta_array + u
        beta_hat.append(np.linalg.inv(X.T@X)@X.T@y)
    beta_hat = np.array(beta_hat).T
    beta_hat_mean.append(np.mean(beta_hat[0], axis = 1))

beta0, beta1, beta2, beta3 = [], [], [], []
for i in beta_hat_mean:
    beta0.append(i[0])
    beta1.append(i[1])
    beta2.append(i[2])
    beta3.append(i[3])
```

```python
fig, ax = plt.subplots(figsize =(18, 8), nrows = 2, ncols=2)
fig.suptitle('Simulation of OLS Consistency', size = 18)
ax[0,0].plot(beta0)
ax[0,0].axhline(2, color ='r')
ax[0,0].set_xlabel(r'$\beta_1$ Sample Size')

ax[0,1].plot(beta1)
ax[0,1].axhline(3, color ='r')
ax[0,1].set_xlabel(r'$\beta_2$ Sample Size')

ax[1,0].plot(beta2)
ax[1,0].axhline(4, color ='r')
ax[1,0].set_xlabel(r'$\beta_3$ Sample Size')

ax[1,1].plot(beta3)
ax[1,1].axhline(5, color ='r')
ax[1,1].set_xlabel(r'$\beta_4$ Sample Size')
plt.show()
```

#  Efficiency of the OLS

The **efficiency** of estimator represents the capacity of utilizing the information efficiently. For instance, OLS is more efficient than many other estimators, because with the same sample size, OLS yields the estimates with highest precision.

##  Positive Definite Matrices

Before moving any further, we will refresh the definition of positive definite matrices.

A **quadratic form** is a function with form $Q(\mathbf{x})=\mathbf{x}^TA\mathbf{x}$, where $A$ is an $n\times n$ symmetric matrix, which is called the **the matrix of the quadratic form**. For more details, check linear algebra material.

Now some terms to need to be defined, a quadratic form $Q$ is:

positive definite if $Q(\mathbf{x})>0$ for all $\mathbf{x} \neq \mathbf{0}$

negative definite if $Q(\mathbf{x})<0$ for all $\mathbf{x} \neq \mathbf{0}$

positive semidefinite if $Q(\mathbf{x})\geq0$ for all $\mathbf{x} \neq \mathbf{0}$

negative semidefinite if $Q(\mathbf{x})\leq0$ for all $\mathbf{x} \neq \mathbf{0}$

indefinite if $Q(\mathbf{x})$ assumes both positive and negative values.

If $B$ is an $m\times n$ matrix, both $B^TB$ and $BB^T$ yield symmetric matrices. Here is an example of $B_{3\times 2}$

```python
B = sy.Matrix([[1, -1], [3, 4], [0, 2]]); B
```

$\displaystyle \left[\begin{matrix}1 & -1\\3 & 4\\0 & 2\end{matrix}\right]$

$B^TB$ yields a $2\times2$ matrix

```python
B.T*B
```

$\displaystyle \left[\begin{matrix}10 & 11\\11 & 21\end{matrix}\right]$

$BB^T$ yields a $3\times3$ matrix

```python
B*B.T
```

$\displaystyle \left[\begin{matrix}2 & -1 & -2\\-1 & 25 & 8\\-2 & 8 & 4\end{matrix}\right]$

Since $B^TB$ and $BB^T$ are both symmetric, $A$ can be replaced by either of them, technically the **Cholesky decomposition** can achieve $A = BB^T$ or $A = B^TB$.

Actually any matrix of form $B^TB$ or $BB^T$ is positive semidefinite, to see this, observe that
$$
x^TB^TBx = (Bx)^T(Bx) = \|Bx\|^2 \geq 0\\
xBB^Tx^T = xB (xB)^T = \|xB\|^2\geq 0
$$
where $x\neq0$.

In practice, the second form is quite rare, so we focus the on the first form $x^TB^TBx$, since $x\neq 0$, the only condition of $\|Bx\|^2 = 0$ is $B$ does not have **full rank**, in other words, if $B$ has full rank, then $B^TB$ is positive definite rather than positive semidefinite.

##  Covariance Matrix of the OLS Parameter Estimates

The covariance matrix of error term is

$$
\text{Var}(\boldsymbol{u}) = E(\boldsymbol{u}\boldsymbol{u}^T) = \sigma^2_0\mathbf{I}
$$
where $\sigma^2_0\mathbf{I}$ is a positive definite matrix
$$
\boldsymbol{x}^{T} \sigma_0^2\mathbf{I} \boldsymbol{x}=\sigma_0^2\sum_{i=1}^{k} x_{i}^{2}
$$
Now we can construct covariance matrix of $\boldsymbol{\hat{\beta}}$, from discussions above we know $\boldsymbol{\hat{\beta}}- \boldsymbol{\beta}_0 = (\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{u}$, therefore

\begin{align}
\text{Var}(\boldsymbol{\hat{\beta}}) = E\big[(\boldsymbol{\hat{{\beta}}}-\boldsymbol{\beta_{0}})(\boldsymbol{\hat{{\beta}}}-\boldsymbol{{\beta}}_{0})^{T}\big]&=
(\boldsymbol{X}^{\top} \boldsymbol{X})^{-1} \boldsymbol{X}^{T}E( \boldsymbol{u} \boldsymbol{u}^{T}) \boldsymbol{X}(\boldsymbol{X}^{T} \boldsymbol{X})^{-1}\\
&=(\boldsymbol{X}^{\top} \boldsymbol{X})^{-1} \boldsymbol{X}^{T}\sigma_0^2\mathbf{I} \boldsymbol{X}(\boldsymbol{X}^{T} \boldsymbol{X})^{-1}\\
& = \sigma_0^2 (\boldsymbol{X}^{\top} \boldsymbol{X})^{-1}
\end{align}
which is the standard result for the covariance matrix of $\boldsymbol{\hat{\beta}}$.

It is illuminating if we rewrite the last equation,

$$
\text{Var}(\boldsymbol{\hat{\beta}})  =\frac{1}{n}\sigma_0^2 \left(\frac{1}{n}\boldsymbol{X}^{\top} \boldsymbol{X}\right)^{-1}
$$

We need to make one extra assumption to interprete the equation properly, that is

$$
\text{plim}\left(\frac{1}{n}\boldsymbol{X}^{T} \boldsymbol{X}\right)^{-1} = \boldsymbol{S}_{\boldsymbol{X}^{T} \boldsymbol{X}}
$$

where $\boldsymbol{S}_{\boldsymbol{X}^{T} \boldsymbol{X}}$ is a nonstochastic matrix with full rank $k$.

Thus we get a different form of variance of OLS estimator

$$
\text{Var}(\boldsymbol{\hat{\beta}})  =\frac{\sigma_0^2}{n} \boldsymbol{S}_{\boldsymbol{X}^{T} \boldsymbol{X}}
$$

if we double the $\sigma_0^2$ or double the sample size $n$, the variance of $\boldsymbol{\hat{\beta}}$ will be doubled and halfed respectively.

And standard errors is

$$
\sigma_\hat{\beta}=\frac{\sigma_0}{\sqrt{n}}\sqrt{\boldsymbol{S}_{\boldsymbol{X}^{T} \boldsymbol{X}}}
$$

For your reference, in basic statistics, we use formula for variance of sample means

$$
\sigma_{\bar{x}}=\frac{\sigma_0}{\sqrt{n}}
$$

Apart from $n$ and $\sigma_0^2$, the third factor which influences $\text{Var}(\boldsymbol{\hat{\beta}})$ is  $\boldsymbol{X}$. To see this, we partition $\boldsymbol{\beta}$ into $[{\beta}_1\quad\boldsymbol{\beta}_2]$, where $\boldsymbol{\beta}_2$ denotes the $(k-1)$-vector. The linear regression model rewritten as

$$
\boldsymbol{y} = \boldsymbol{x}_1\beta_1+\boldsymbol{X}_2\boldsymbol{\beta}_2+\boldsymbol{u}
$$

Use $\text{FWL}$ regression

$$
\boldsymbol{M}_2\boldsymbol{y}= \boldsymbol{M}_2\boldsymbol{x}_1\beta_1 + \boldsymbol{v}
$$

where $\boldsymbol{M}_2$ means projection off $\boldsymbol{X}_2$.

According to OLS formula $\boldsymbol{\hat{\beta}}= \boldsymbol{(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}}$

$$
\hat{\beta}_1 = (\boldsymbol{x}_1^T\boldsymbol{M}_2\boldsymbol{x}_1)^{-1}\boldsymbol{x}_1^T\boldsymbol{M}_2\boldsymbol{y}=\frac{\boldsymbol{x}_1^T\boldsymbol{M}_2\boldsymbol{y}}{\boldsymbol{x}_1^T\boldsymbol{M}_2\boldsymbol{x}_1}
$$

And use variance formula $\text{Var}(\boldsymbol{\hat{\beta}}) = \sigma_0^2 (\boldsymbol{X}^{\top} \boldsymbol{X})^{-1}$, we get

$$
\text{Var}(\boldsymbol{\hat{\beta}}_1) = \sigma_0^2\left(\boldsymbol{x}_1^T\boldsymbol{M}_2\boldsymbol{x}_1\right)^{-1}=\frac{\sigma_0^2}{\boldsymbol{x}_1^T\boldsymbol{M}_2\boldsymbol{x}_1}
$$

Also note that denominator of $\text{FWL}$ $\text{Var}(\boldsymbol{\hat{\beta}}_1)$ is actually $\|\boldsymbol{M}_2\boldsymbol{x}_1\|^2$. The geometric interpretation of precision (the inverse of variance) is provided in next section.

##  Multicollinearity and Visualization

The formula of $\text{FWL}$ $\text{Var}(\boldsymbol{\hat{\beta}})$ raises the problem of multicollinearity, which means that one or some of regressors have strong correlation with rest.

Here we visualize the subspace $\text{span}({x}_1,\ \boldsymbol{X}_2)$ prepared for $\text{FWL}$ regression above.

```python
fig, ax = plt.subplots(figsize = (10, 10))

basis = np.array([[0, 0, 5, 7], [0, 0, 9, 0]])
X, Y, U, V = zip(*basis)
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)

M2x1 = np.array([[0, 0, 0, 7]])
X, Y, U, V = zip(*M2x1)
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1, color = 'red')

x1_apos = np.array([[0, 0, 7, 5]])
X, Y, U, V = zip(*x1_apos)
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1, color = 'gray')

M2x1_apos = np.array([[0, 0, 0, 5]])
X, Y, U, V = zip(*M2x1_apos)
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1, color = 'pink')

X2c = np.array([[0, 0, 5, 0]])
X, Y, U, V = zip(*X2c)
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1, color = 'aqua')

point1 = [7, 5]
point2 = [0, 5]
line = np.array([point1, point2])
ax.plot(line[:,0], line[:,1], c = 'k', lw = 3.5,alpha =0.5, ls = '--')

point1 = [5, 7]
point2 = [0, 7]
line = np.array([point1, point2])
ax.plot(line[:,0], line[:,1], c = 'k', lw = 3.5,alpha =0.5, ls = '--')

point1 = [5, 0]
point2 = [5, 7]
line = np.array([point1, point2])
ax.plot(line[:,0], line[:,1], c = 'aqua', lw = 3.5,alpha =0.5, ls = '--')

ax.set_xlim([-1, 10])
ax.set_ylim([-1, 10])
#plt.draw()
ax.grid()

ax.text(5, 7, r'$x_1$', size = 18)
ax.text(9, 0, r'$X_2$', size = 18)
ax.text(0, 7, r'$M_2 x_1$', size = 18)
ax.text(7, 5, r'$x_1^\prime$', size = 18)
ax.text(0, 5, r'$M_2 x_1^\prime$', size = 18)
ax.text(7, 7, r'$Var(\hat{\beta}_1)=\frac{\sigma_0^2}{\|\|M_2x_1\|\|^2}$', size = 18)
ax.text(.3, .1, r'$\vartheta$', size = 22)
ax.text(5, 0.1, r'$X_2c$', size = 18)
ax.text(4.85, 3, r'$\hat{u}$', size = 22)

style = 'Simple, tail_width=0.5, head_width=8, head_length=16'
kw = dict(arrowstyle=style, color='orange', ls = '-.', alpha = .5, lw = 3)
arrow = mpl.patches.FancyArrowPatch((9,2),(2,9), connectionstyle='arc3,rad=.35', **kw)
ax.add_patch(arrow)

plt.show()
```

As we can see from the graph, if the length $\|\boldsymbol{M}_2{x_1}\|$ is becoming longer means that the angle $\vartheta$ is increasing on condition that $\|x_1\|$ remain the same.

From linear algebra course, we know the formula

$$
\cos{\vartheta} = \frac{\boldsymbol{u}\cdot\boldsymbol{v}}{\|\boldsymbol{u}\|\|\boldsymbol{v}\|}
$$

which is exactly the correlation coefficient. The larger the angle, the lower the correlation, i.e. smaller $\cos{\vartheta}$.

In this example, if $\boldsymbol{x}_1$ is highly correlated with $\boldsymbol{X}_2$, then $\vartheta$ must be relatively small, therefore $\text{Var}(\hat{\beta})$ is relatively high. This is what we call multicollinearity in practice, though it is not really collinear, rather, approximately pointing to the same direction.

But remember this, multicollinearity always presents, but in a matter of degree, not yes or no.

We can also interpret the graph by using a standard regression model

$$
\boldsymbol{x}_1 = \boldsymbol{X}_2c + \hat{u}
$$

where $\hat{u} = \boldsymbol{M}_2\boldsymbol{x}_1$.

##  Comparing Precision of Estimators

If estimator $\boldsymbol{\hat{\beta}}$ is more precise than $\boldsymbol{\tilde{\beta}}$, the difference between them
$$
\text{Var}(\boldsymbol{\tilde{\beta}})- \text{Var}(\boldsymbol{\hat{\beta}})
$$
is a positive semidefinite matrix.

Also any linear combination of $w^T\boldsymbol{\hat{\beta}}$ is also more efficient than $w^T\boldsymbol{\tilde{\beta}}$, because $\text{Var}(w^T\beta)=w^T\text{Var}(\beta)w$
$$
\boldsymbol{w}^{\top} \operatorname{Var}(\tilde{\boldsymbol{\beta}}) \boldsymbol{w}-\boldsymbol{w}^{\top} \operatorname{Var}(\hat{\boldsymbol{\beta}}) \boldsymbol{w}=\boldsymbol{w}^{\top}(\operatorname{Var}(\tilde{\boldsymbol{\beta}})-\operatorname{Var}(\hat{\boldsymbol{\beta}})) \boldsymbol{w}\geq0
$$

##  The Gauss-Markov Theorem

The principal result on efficiency of OLS estimator is called the **Gauss-Markov Theorem**. The informal way of stating this theorem is to say that $\hat{\beta}$ is the **best linear unbiased estimator** (**BLUE**) .

Formally, if these two conditions are satisfied
1. $E(\boldsymbol{u}|\boldsymbol{X})=\boldsymbol{0}$
2. $E(\boldsymbol{u}\boldsymbol{u}^T|\boldsymbol{X})=\sigma^2\mathbf{I}$ (homoskedasticity and uncorrelated $u_t$)

OLS estimator is the BLUE.

The first condition holds because
$$
E\left[\begin{array}{c}
u_{1} \mid \boldsymbol{X} \\
u_{2} \mid \boldsymbol{X} \\
\vdots \\
u_{n} \mid \boldsymbol{X}
\end{array}\right]=\left[\begin{array}{c}
E\left(u_{1}\right) \\
E\left(u_{2}\right) \\
\vdots \\
E\left(u_{n}\right)
\end{array}\right]=\left[\begin{array}{c}
0 \\
0 \\
\vdots \\
0
\end{array}\right]
$$

The second condition can be shown
\begin{align}
E\left(\boldsymbol{u u}^T \mid \boldsymbol{X}\right)&=E\left[\begin{array}{cccc}
u_{1}^{2} \mid X & u_{1} u_{2} \mid X & \ldots & u_{1} u_{n} \mid X \\
u_{2} u_{1} \mid X & u_{2}^{2} \mid X & \ldots & u_{2} u_{n} \mid X \\
\vdots & \vdots & \vdots & \vdots \\
u_{n} u_{1} \mid X & u_{n} u_{2} \mid X & \ldots & u_{n}^{2} \mid X
\end{array}\right]\\
&=\left[\begin{array}{cccc}
E\left[u_{1}^{2} \mid X\right] & E\left[u_{1} u_{2} \mid X\right] & \ldots & E\left[u_{1} u_{n} \mid X\right] \\
E\left[u_{2} u_{1} \mid X\right] & E\left[u_{2}^{2} \mid X\right] & \ldots & E\left[u_{2} u_{n} \mid X\right] \\
\vdots & \vdots & \vdots & \vdots \\
E\left[u_{n} u_{1} \mid X\right] & E\left[u_{n} u_{2} \mid X\right] & \ldots & E\left[u_{n}^{2} \mid X\right]
\end{array}\right]\\
&= \left[\begin{array}{cccc}
\sigma^{2} & 0 & \ldots & 0 \\
0 & \sigma^{2} & \ldots & 0 \\
\vdots & \vdots & \vdots & \vdots \\
0 & 0 & \ldots & \sigma^{2}
\end{array}\right]
\end{align}

in the linear regression model, then OLS estimator $\boldsymbol{\hat{\beta}}$ is more efficient than any other $\boldsymbol{\tilde{\beta}}$, i.e. $\text{Var}(\boldsymbol{\hat{\beta}}) - \text{Var}(\boldsymbol{\tilde{\beta}})$ is a positive semidefinite matrix.

#  Residuals and Disturbance Terms

We have discussed geometric and numerical properties of $\hat{\boldsymbol{u}}$ in last chapter, here we will discuss the properties of $\hat{\boldsymbol{u}}$.

One important insight from geometric view of OLS is
$$
\|\boldsymbol{u}\| \geq \|\boldsymbol{\hat{u}}\|
$$
which is a direct deduction of $\boldsymbol{\hat{u}}=\boldsymbol{M_Xu}$.

##  Expectation of Residuals

Consider two matrices $\boldsymbol{A}_{n\times m}$ and $\boldsymbol{B}_{m \times k}$, then the $i^{th}$ row of $\boldsymbol{AB}$ is the product of $i^{th}$ row of $\boldsymbol{A}$ and entire $\boldsymbol{B}$, if we want to represent the $t^{th}$ row of $\boldsymbol{AB}$ we can use

$$
\boldsymbol{A}_t\boldsymbol{B}
$$

where $\boldsymbol{A}_t$ is the $t^{th}$ row of $\boldsymbol{A}$.

In general we can represent $t^{th}$ row of a product of matrices by picking out the $t^{th}$ row of the leftmost matrix.

Using this fact, we can denote the $t^{th}$ residual as

$$
\hat{u}_t = \boldsymbol{M_X}\boldsymbol{u} = (\mathbf{I}-\boldsymbol{P_X})\boldsymbol{u} = u_t - \boldsymbol{X}_t(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{u}
$$

Although in Gauss-Markov Theorem, we assume the elements in error terms are independent, but this is not the case among residuals.

It is clear $\boldsymbol{\hat{u}}$ is a linear combination of $\boldsymbol{u}$ i.e. $\boldsymbol{M_Xu}$, so the $E(\boldsymbol{\hat{u}}) = \boldsymbol{0}$.

##  Variance of Residuals

Use the variance formula on $\boldsymbol{\hat{u}}$

\begin{align}
\text{Var}(\boldsymbol{\hat{u}})&= \text{Var}(\boldsymbol{M_X}\boldsymbol{{u}}) = E[(\boldsymbol{M_X u}- E(\boldsymbol{M_X u}))(\boldsymbol{M_X u}- E(\boldsymbol{M_X u})^T)]\\
&= E(\boldsymbol{M_X}\boldsymbol{u}\boldsymbol{u}^T\boldsymbol{M_X}) = \boldsymbol{M_X}E(\boldsymbol{u}\boldsymbol{u}^T)\boldsymbol{M_X} = \boldsymbol{M_X}\text{Var}(\boldsymbol{u})\boldsymbol{M_X}\\
& = \boldsymbol{M_X}(\sigma_0^2\mathbf{I})\boldsymbol{M_X}\\
& = \sigma_0^2 \boldsymbol{M_X}
\end{align}

Obviously, $\boldsymbol{M_X}$ is not an identity matrix, which means that variances of $\hat{u}_t$ might not be homoscedasticity, also $E(u_tu_s)\neq 0$, i.e. the residuals are correlated.

We have denoted the $t^{th}$ diagonal element of $\boldsymbol{P_X}$ as $h_t$, then the $t^{th}$ diagonal element of $\boldsymbol{M_X}$ is $1-h_t$. In last chapter, we have shown that $0 \leq h_t < 1$, hence $0 \leq 1- h_t < 1$

$$
\text{Var}(\hat{u}_t)<\sigma_0^2
$$

The variance of any residual from OLS is always smaller than variance of disturbance term, this feature determined by the mechanism of OLS algorithm.

##  Estimating the Variance of Error Terms

We do not observe the disturbance terms $u_t$, neither does $\sigma^2_0$. But we are able to estimate it by method of moments.

The simplest estimator of $\sigma^2_0$ is

$$
\hat{\sigma}^2 = \frac{1}{n}\sum_{t=1}^n\hat{u}_t^2
$$

However we have shown $\text{Var}(\hat{u}_t)<\sigma_0^2$, so $\hat{\sigma}^2$ must be biased downward.

Before showing how to neutralize the bias, here is a side note about the trace of matrices.

###  The Trace of Matrices

The trace is the sum of the elements on the principal diagonal of a $n\times n$ matrix, denoted $\text{Tr}(\boldsymbol{A})$. And it can be a product of any two matrices $\boldsymbol{A}_{n\times m} \boldsymbol{B}_{m \times n}$. But there is a convenient property called **cyclic permutation**.

$$
\text{Tr}(\boldsymbol{ABC}) = \text{Tr}(\boldsymbol{CAB}) = \text{Tr}(\boldsymbol{BCA})
$$

Because $\boldsymbol{P_X}$ is a square matrix, we can use the formula onto $\boldsymbol{P_X}$ and we can show that

$$
\text{Tr}(\boldsymbol{P_X})= \text{Tr}(\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T) =   \text{Tr}(\boldsymbol{X}^T\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}) =  \text{Tr}(\mathbf{I}_k) = k
$$

###  Unbiased Estimator of Variance of Error Term

Let's process with $\hat{\sigma}^2$ by taking expectation

$$
E(\hat{\sigma}^2) = \frac{1}{n}\sum_{t=1}^nE(\hat{u}_t^2)
$$

Use the fact that $\text{Var}(\hat{u}_t) = E[(\hat{u}_t- E(\hat{u}_t))^2] = E[(\hat{u}_t)^2] = (1-h_t)\sigma_0^2$ and $\text{Tr}(\mathbf{I}_k)=k$

$$
\frac{1}{n}\sum_{t=1}^nE(\hat{u}_t^2) = \frac{1}{n}\sum_{t=1}^n(1-h_t)\sigma_0^2=\frac{n-k}{n}\sigma_0^2
$$

We have the unbiased estimator of $\sigma^2$

$$
s^2 = \frac{1}{n-k}\sum_{t=1}^nE(\hat{u}_t^2)= \frac{1}{n-k}\sum_{t=1}^n\hat{u}_t^2=\frac{\hat{u}^T\hat{u}}{n-k}
$$

And now we can replace the $\sigma_0^2$ in the $\text{Var}(\boldsymbol{\hat{\beta}}) = \sigma_0^2(\boldsymbol{X^TX})^{-1}$, the square root of principal diagonal holds the **standard error** of $\boldsymbol{\hat{\beta}}$.

$$
\widehat{\text{Var}}(\boldsymbol{\hat{\beta}}) = s^2(\boldsymbol{X^TX})^{-1}
$$

###  Numerical Example of Residuals

Use the class or functions that I wrote in ```linregfunc.py```, we can carry out a quick simulation comparing the modulus of residuals and disturbance term.

There are both class and functions in the ```linregfunc.py```, we will choose the whatever the most convenient.

The moment when an object is instantiated, all the randomized data for OLS will be generated accordingly.

```python
import linregfunc as lr
```

```python
ols_obj = lr.OLS_Simu(100, 3, [6, 7, 8])
```

```python
print('The moduduls of residual: {:.4f}'.format(np.linalg.norm(ols_obj.resid)))
print('The moduduls of disturbance term: {:.4f}'.format(np.linalg.norm(ols_obj.u)))
```

    The moduduls of residual: 9.6886
    The moduduls of disturbance term: 9.8263

It holds because theory guarantees
$$
\text{Var}(\hat{u}_t)<\sigma_0^2
$$

However simulate this in a manner of distribution?

The function, which generates disturbance terms, follows a standard normal distribution, i.e. $\sigma_0^2=1$. $\hat{u}_t$ is a specific term of the residual vector, for instance to estimate the variance of $\hat{u}_1$, we simulate the estimation process $10000$ times, then calculate the variance from those $10000$ $\hat{u}_1$'s.

Set sample size $n=100$, create projection matrix $\boldsymbol{M}$.

```python
proj_matM = ols_obj.proj_mat_M()
```

We have create $\boldsymbol{\beta} =  [6, 7, 8]^T$. Now add a placeholder array for residuals.

```python
resid_array = np.zeros([100, 1])
```

Generate $10000$ vectors of residuals and stack them horizontally by specifying ```axis=1```. ```gen_u``` is a function for generating disturbance term which is identical to the one used in the class ```OLS_Simu```, for convenience of loops we use the function rather than method.

```python
for i in range(10000):
    u = lr.gen_u(100)
    resid = proj_matM@u
    resid_array = np.concatenate((resid_array, resid), axis = 1)
resid_array = resid_array[:, 1:]
```

Because this is a sequence of simulations, the variance will be a frequency distribution rather than a single number. Here is the histogram, obviously the mean is less than $1$ as we have expected.

```python
fig, ax = plt.subplots(figsize=(10, 10))
ax.hist(np.var(resid_array, axis = 0), bins= 80)
ax.axvline(x = np.mean(np.var(resid_array, axis = 0)), color = 'blue')
ax.set_title('Variance Distribution of Sampled Residuals')
plt.show()
```

###  Standard Errors Simulation

Here we simply show a numerical example of covariance matrix and the square root of its principal diagonal which holds the standard errors of $\hat{\boldsymbol{\beta}}$.

```python
ols_obj = lr.OLS_Simu(100, 4, [6, 7, 8, 9])
cov_betahat, cov_betahat_diag = ols_obj.cov_beta_hat()
```

```python
np.sqrt(cov_betahat_diag)
```

    array([0.10323776, 0.11105161, 0.11171944, 0.0987561 ])

Compare with ```statsmodels``` results.

```python
ols_obj_fit = sm.OLS(ols_obj.y, ols_obj.X).fit()
print(ols_obj_fit.bse)
```

    [0.10323776 0.11105161 0.11171944 0.0987561 ]

Results are exactly the same.

Chapter 2 - Statistical Properties of OLS.ipynb


```python
import numpy as np
```

```python
vec=np.array([5,6,8,7,8])
```

```python
vec[0]
```

    5

```python
mat=np.array([[3,2],[1,4]])
```

```python
mat
```

    array([[3, 2],
           [1, 4]])

```python
mat[1,1]
```

    4

```python
nor=np.random.normal(10,5,50).reshape(10,5)
```

```python
nor
```

    array([[17.52605772, 10.49391801,  2.56245836,  9.37272696, 10.0897693 ],
           [15.8219556 , 16.12623198,  3.0979952 ,  2.07392566,  8.70725088],
           [10.67362134, 12.38694245, 10.3016678 , 12.85186214,  3.21795513],
           [19.38731991, 12.2794654 ,  6.10136969,  9.39494412,  8.15617831],
           [14.31018751,  1.69837832, 14.16075557, 16.25312941,  3.33344476],
           [19.03624129,  7.48235034,  9.10815773,  9.90733591,  6.18343445],
           [ 6.82358215,  6.20859119,  9.22478183, 12.80975449,  2.43029982],
           [19.82770733,  4.36095005,  4.89655891,  9.16833886, 14.36344395],
           [ 8.18323865, 14.27817019, 14.79842231, 12.17061264, 18.75810942],
           [12.74085621, 15.06889639,  3.22612138, 16.86317668,  7.73922489]])

```python
nor[:,0:0:2]
```

    array([], shape=(10, 0), dtype=float64)

```python
d=np.random.normal(10,5,6).reshape(3,2)
```

```python
a=np.random.normal(10,5,6).reshape(2,3)
```

```python
d
```

    array([[ 8.9128375 ,  4.89927704, 13.5372206 ],
           [11.40797596, 17.16768408,  8.8219575 ]])

```python
a
```

    array([[10.4177932 ,  5.76675217, 23.0212424 ],
           [ 5.73138704, 14.65483343,  0.32205845]])

```python
d
```

    array([[ 8.9128375 ,  4.89927704, 13.5372206 ],
           [11.40797596, 17.16768408,  8.8219575 ]])

```python
a
```

    array([[10.4177932 ,  5.76675217, 23.0212424 ],
           [ 5.73138704, 14.65483343,  0.32205845]])

```python
vec1=x=np.linspace(0,100,500)
```

```python
vec1
```

```python
e=np.random.normal(10,21,500)
```

```python
y=5+3.5*x +e
```

```python
import matplotlib.pyplot as plt

```

```python
plt.scatter(x,y)
```

```python
X=np.c_[np.ones(len(x)),x]
```

```python
vec[0]
```

    5

```python
plt.figure (figsize=(14,8))
plt.scatter(x,y)
plt.plot(x,b[0]+b[1]*x)
```

```python
import numpy as np
k=np.log(19999/((20000/345)-1))/50
```

```python
def c(t):
  return 20000 / (1+19999 * np.exp(-k*t))
```

```python
t = np.arange(0,175,0.01)
```

```python
c(t)
```

    array([1.00000000e+00, 1.00117281e+00, 1.00234699e+00, ...,
           1.99995048e+04, 1.99995053e+04, 1.99995059e+04])

```python
dcdt=k*c(t)*(1-(c(t)/20000))
```

```python
from sympy import *
from sympy import Eq
```

```python

```

```python
ec = Eq(c(t).diff(t),K*c(t)*(1-(c(t)/20000)))
```

```python
 import matplotlib.pyplot as plt
 plt.figure(figsize=(18,10))
 plt.plot(t, c(t),c="black",linestyle="--")
 plt.axvline(np.log(19999)/k)

```

```python
plt.plot(c(t),dcdt)
plt.axhline(590, linestyle="--")
```

```python

```