# **Lab 2: Naive Bayes Classification and Linear Regression**

CS 412, Introduction to Machine Learning

Department of Computer Science, University of Illinois at Chicago

***This is a group work for at most four students.***

This is your second lab work, and you will work on it with your teammates. You will learn how to apply the Naive Bayes model to filter spam SMS messages. You will also learn how to make real-valued predictions using a linear regression model.

***Deadline:***
This assignment is due **Oct 21** (Anywhere on Earth, [AoE](https://www.timeanddate.com/time/zones/aoe)). That is, you can resubmit as often as you like provided that anywhere on Earth is still on or before this date.

***How to submit:***
See bottom of the page

***Python version:***
The code should work on Python 3.7 or later, though it might work on earlier versions (not tested). There should be no version problem if you work on Colab.  See a more detailed introduction to Python and Colab at this [link](https://colab.research.google.com/github/cs231n/cs231n.github.io/blob/master/python-colab.ipynb#scrollTo=nxvEkGXPM3Xh). You may also find the setup instructions useful [here](https://cs231n.github.io/setup-instructions/).

**Please note before starting the lab:**

1. There are unit test cases provided after each function you need to implement. Make good use of them.

2. Since the experiments involve randomness, it is important to ensure that your results are replicable. To this end, your implementation should take one integer (or any numeric value) as a seed that is used to initialize the random number generators.
See, e.g. [random.seed](https://docs.python.org/3/library/random.html).
This has been done for you in the first code block below.

3. If you use Colab, ignore the following message when you open the notebook (if it shows up): \\
`Unrecognized runtime "python_defaultSpec_1600651579462"; defaulting to "python3"`

## Problem 1. Naive Bayes Classification **(62 points)** {-}

In this problem, you will implement the Naive Bayes classification method and use it for SMS message classifcation. The SMS dataset `SMSSpamCollection` has been provided in the assignment folder, which also can be downloaded from the [UCI Link](http://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection).  In case the repository gets offline occasionally, we have made local copies for the [dataset](https://www.cs.uic.edu/~zhangx/teaching/SMSSpamCollection.dat) and [readme](https://www.cs.uic.edu/~zhangx/teaching/readme.txt).

To help you to better understand this algorithm, you are **not** allowed to use any off-the-shelf naive Bayes implementations from third-party libraries. We will implement it with detailed step-by-step instructions.

### Recap of the Naive Bayes Algorithm  {-}
Naive Bayes classification is a fast and simple classification method. Its efficiency stems from some simplifications we make about the underlying probability distributions, namely, the assumption about the conditional independence of features. Suppose for any class $Y$, we have a probability distribution over all possible combinations of values for a feature vector $X$:
$$
P(X|Y).
$$
The main idea of Bayesian classification is to reverse the direction of dependence --- we want to predict the label based on the features:
$$
P(Y|X).
$$
This is made possible by the Bayes theorem:
\begin{equation}
P(Y|X) = \frac{P(X|Y)P(Y)}{P(X)}. \tag{1}
\end{equation}

To make it more concrete, let us consider the SMS message classification problem. Ignoring punctuations, each SMS message contains an ordered sequence of $T$ words (case-insensitive) $X = \{X_1, ...,X_T\}$. That is, $X$ corresponds to an SMS message, and $X_i$ corresponds to the $i$-th word in it. For each message from the training set, there is a corresponding label $Y\in\{spam, ham\}$.

**Model specification and key assumption.** The conditional distribution can be written as:
$$
P(X|Y) = P(X_1, ..., X_T|Y).
$$
Since this conditional probability is intractable, we simplify it in two steps:

1. **Assume** that all features $X_i$ are independent, conditional on the category $Y$. This leads to a naive Bayes model which writes formally as
\begin{equation}
P(X|Y) = P(X_1, ..., X_T|Y) = \prod_{i=1}^T P(X_i|Y). \tag{2}
\end{equation}

2. **Assume** that $P(X_i | Y) = P(X_j|Y)$ for all $i \neq j$.
In other words, given the label $Y$, the value of the $7$-th word has the same distribution as the value of the $10$-th word. Note this is not assumed by default in naive Bayes, and we make this additional assumption to significantly simplify our model.
It is often referred to as "tying" the probability $P(X_i|Y)$ over $i$.
As a result, the order of words no longer matters for $P(X|Y)$,
i.e.,
$$P(X=\text{'cat is cute'}|Y) \ \ = \ \ P(X=\text{'cute cat is'}|Y) \quad \text{for all } Y.
$$

Plugging Eq (2) into the Bayes theorem in Eq (1), we arrive at
$$
\begin{aligned}
P(Y|X) &= \frac{P(Y) P(X|Y)}{P(X)} = \frac{P(Y)\prod_{i=1}^T P(X_i|Y)}{P(X)} \\
&\propto P(Y)\prod_{i=1}^T P(X_i|Y),
\end{aligned}
$$
where $\propto$ denotes proportionality. Since the denominator $P(X)$ does not depend on $Y$, the prediction probability is proportional to the numerator.

**Making predictions.**
Naturally, given an SMS message $X$, we can first compute $P(Y|X)$ for all possible categories $Y$ (in this example, only two categories), and then make predictions by outputting the $Y$ that maximizes the probability. This can be expressed mathmatically as:
$$
\arg\max_Y P(Y)\prod_{i=1}^T P(X_i|Y) = \arg\max_Y \left\{\log P(Y) + \sum_{i=1}^T \log P(X_i|Y) \right\}. \tag{3}
$$
If there is a tie, we just break it arbitrarily.
Here the logarithm uses natural basis.

**Learning the model.**
To apply the prediction rule in Eq (3), we need to first figure out (formally termed "estimate") the value of $P(Y)$ and $P(X_i|Y)$ by using the training data. Recall that since we tie the conditional probabilities $P(X_i|Y)$ across all $i$,
the subscript $i$ can be dropped.
However, we still carry it just for clarity.

Firstly, $P(Y=y)$ can be estimated by computing the frequency of category $y$ in the whole training set ($y$ can be either "$spam$" or "$ham$"). Here, in order to avoid confusion, we have used the convention that capital letters denote random variables, and lowercase letters denote their possible instantiations.

Secondly, $P(X_i = w |Y=y)$ for a word $w$ (e.g., "cat") can be estimated by counting the frequency that it appears in the training message set for a given category $y$ (derivation not required in the course):
\begin{align}
P(X_i &= w|Y=y)
= \frac{Count(w, y)}{Count(y)}, \ \ where
\tag{4} \\
\notag
 Count(w, y) &= \text{total number of occurrence of $w$ in all SMS messages of category } y \\
\notag
Count(y) &= \text{total number of words appearing in SMS messages of category } y.
\end{align}

*Remark 1:* If $w$ appears in a single message for 3 times, then it contributes to $Count(w, y)$ by 3, not 1. Similarly, $Count(y)$ indeed equals the total length of all messages in category $y$.

*Remark 2:* Obviously, the right-hand side of Eq (4) does not depend on $i$. This is consistent with our previous note that we carry the subscript $i$ in $P(X_i|Y)$ only for clarity, while in fact different $i$ share the same $P(X_i|Y)$.

For example, suppose there are four messages
$$
\text{{'cat is cute', ham}, {'dog rocks', spam}, {'whatever is is right', ham}, {'hello', spam}.}
$$
Then $P(X_i = '\text{is}' | \text{ham}) = 3 / 7$ (**not** $2/7$),
and $P(X_i = '\text{is}' | \text{spam}) = 0 / 3$.

Quiz (no need to submit): compute $P(X_i = w | \text{ham})$, for $w = $ 'cat', 'is', 'cute', 'whatever', and 'right'. Check if their sum is 1. Now appreciate why in Remark 1, a word appearing for mulitple times in a single message should be counted multiple times.

You may have noticed that any word $w$ with $Count(w,y)=0$ leads to $P(X_i = w|Y=y) = 0$.
As a result, by Eq (2), any message $x$ has conditional probably $P(X = x |Y=y) = 0$ if $w$ appears in $x$.
Such a "veto" is not favorable, and can create significant problems when a word in the test data has never appeared in the training data (think why?).
To bypass this issue, we can add pseudo-count, a.k.a additive smoothing:
$$
\hat{P}(X_i = w|Y=y) = \frac{Count(w, y) + \alpha}{Count(y) + N\alpha}, \tag{5}
$$
where $\alpha$ is a smoothing parameter. $\alpha=0$ corresponds to no smoothing. In our experiment, let us set $\alpha = 1.0$. $N$ denotes the number of distinct words in the vocabulary, and let us set $N = 20,000$ in this lab.

Now Let's start with data preprocessing.

```python
# set up code for this experiment
import pandas as pd
import numpy as np
import string
from collections import Counter
import matplotlib.pyplot as plt

%matplotlib inline

np.random.seed(1)
```

### Data Preprocessing (not for grading) {-}

We will use `pandas` to import the dataset. Since `SMSSpamCollection` separates labels and text content in each message by a tab, we will use '\t' as the value for the `sep` argument and read raw data into a pandas dataframe. As a result, we store labels and SMS messages into two columns. To facilitate the subsequent steps, we also rename the columns by passing a list `['label', 'sms_message']` to the `names` argument of the `read_table()` method.

Let us print the first five rows of the dataframe to get a basic understanding of the dataset.

```python
# Download the dataset to the server
# Import the data using the read_csv() method from pandas

import urllib.request
import shutil

url = 'https://www.cs.uic.edu/~zhangx/teaching/SMSSpamCollection.dat'
file_name = 'SMSSpamCollection.dat'
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
    shutil.copyfileobj(response, out_file)

df = pd.read_csv(file_name,
                    sep='\t',
                    header=None,
                    names=['label', 'sms_message'])
df.head()
```

      label
      sms_message

      0
      ham
      Go until jurong point, crazy.. Available only ...

      1
      ham
      Ok lar... Joking wif u oni...

      2
      spam
      Free entry in 2 a wkly comp to win FA Cup fina...

      3
      ham
      U dun say so early hor... U c already then say...

      4
      ham
      Nah I don't think he goes to usf, he lives aro...

#### Step 1: Convert string labels to numerical labels (not for grading) {-}

As we can see, there are 2 columns. The first column, which is named `label`, takes two values `spam` (the message is spam) and `ham` (the message is not spam). The second column is the text content of the SMS message that is being classified.  It is a string in which words are separated by space.

Note that the string-typed labels are unwieldy for calculating performance metrices, e.g., when calculating precision and recall scores. Hence, let's convert the lables to binary variables, 0 for `ham` and 1 for `spam`.

```python
# Run the next line only once after running the previous code block
# Running it more than once will turn the labels into NaN
df['label'] = df.label.map({'ham':0, 'spam':1})
df.head()
```

      label
      sms_message

      0
      0
      Go until jurong point, crazy.. Available only ...

      1
      0
      Ok lar... Joking wif u oni...

      2
      1
      Free entry in 2 a wkly comp to win FA Cup fina...

      3
      0
      U dun say so early hor... U c already then say...

      4
      0
      Nah I don't think he goes to usf, he lives aro...

#### Step 2: Bag of words **(10 points)** {-}

What we have in our dataset is a large collection of text data (5,572 rows/messages). Most ML algorithms rely on numerical data to be fed into them as input, but SMS messages are usually text heavy.

To address this issue, we would like to introduce the concept of Bag of Words (BoW), which is designed for problems with a 'bag of words' or a collection of text data. The basic idea is to count the frequency of the words in the text. It is important to note that BoW treats each word individually, ignoring the order in which the words occur.

To count the frequency of the words in text, usually we need to process the input text data in four steps:

- Convert all strings into their lower case form
- Removing all punctuations
- Tokenization, i.e., split a sentence into individual words
- Count frequencies

Once this has been done, we are supposed to obtain a vocabulary dictionary with frequencies of each words for the given text data.

```python
def count_frequency(documents):
    """
    count occurrence of each word in the document set.
    Inputs:
    - documents: list, each entity is a string type SMS message
    Outputs:
    - frequency: a dictionary. The key is the unique words, and the value is the number of occurrences of the word
    """
    """
    Here is the pseudo-code
    # Step 1: covert all strings into their lower case form
    lower_case_doc = []
    for s in documents:
        lower_case_doc.append(???)

    # Step 2: remove all punctuations
    no_punc_doc = []
    for s in lower_case_doc:
        no_punc_doc.append(???)

    # Step 3: tokenize a sentence, i.e., split a sentence into individual words
    # using a delimiter. The delimiter specifies what character we will use to identify the beginning
    # and the end of a word.
    words_doc = []
    for s in no_punc_doc:
        words_doc.append(???)

    # Step 4: count frequencies. To count the occurrence of each word in the document set.
    # We can use the `Counter` method from the Python `collections` library for this purpose.
    # `Counter` counts the occurrence of each item in the list and returns a dictionary with
    # the key as the item being counted and the corresponding value being the count of that item in the list.
    all_words = []
    for s in words_doc:
        all_words.extend(???)
    frequency = "some function/constructor on all_words"
    """

    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    return frequency

# Unit test case:
# documents = ['Hello, how are you!',
#              'Win money, win from home.',
#              'Call me now.',
#              'Hello, Call hello you tomorrow?']
# sample outputs:
# Counter({'hello': 3, 'you': 2, 'win': 2, 'call': 2, 'how': 1, 'are': 1, 'money': 1, 'from': 1, 'home': 1,
# 'me': 1, 'now': 1, 'tomorrow': 1})
documents = ['Hello, how are you!',
            'Win money, win from home.',
            'Call me now.',
            'Hello, Call hello you tomorrow?']

freq = count_frequency(documents)
print(freq)
```

    Counter({'hello': 3, 'you': 2, 'win': 2, 'call': 2, 'how': 1, 'are': 1, 'money': 1, 'from': 1, 'home': 1, 'me': 1, 'now': 1, 'tomorrow': 1})

#### Step 3: Create training and test sets **(5 points)** {-}

We will partition the `SMSSpamCollection` dataset into training and test sets so that we can analyze the model's performance on data it has not witnessed during training. In Lab 1, we have implemented the `split_nfold()` method from scratch for data partition. In this Lab, we will learn to use the `scikit` library. `scikit` is a powerfull tool for machine learning and data mining, providing plenty of well-designed methods for data analysis. We'll use its `train_test_split()` method to create training and testing sets. In this experiment, we use 80% data for training and the remaining 20% data for testing. To ensure your results are replicable, you need to set the `random_state` argument of `train_test_split()` to **1**.

There is no cross validation in this Lab.

```python
from sklearn.model_selection import train_test_split

# learn to read API documentation
# you can get detailed instructions about this method through this link:
# https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

print(f'The original dataset contains {df.shape[0]} examples in total.')
print(f'The training set contains {X_train.shape[0]} examples.')
print(f'The testing set contains {X_test.shape[0]} examples.')
```

    The original dataset contains 5572 examples in total.
    The training set contains 4457 examples.
    The testing set contains 1115 examples.

### Implementing Naive Bayes method from scratch {-}

#### Step 1: training the Naive Bayes Model **(16 points)** {-}

Now that we know what Naive Bayes is, we can take a closer look at how to calculate the posterior probability
$$
P(Y|X) \propto P(Y)\prod_{i=1}^T P(X_i|Y).
$$

The goal of training is to learn the prior and conditional probability from data. The calculation of the prior $P(Y=y)$ is straightforward. It can be estimated via the frequency of messages in the training set that belong to class $y$, e.g.,
$$
P(Y=spam) = \frac{\# \text{training messages in the spam category}}{\# \text{training messages}}.
$$

The conditional probability given the class label --- $P(X_i|Y)$ --- can also be estimated from the data by using Eq (5). As we assumed above, it is indeed independent of $i$ (i.e., shared by all $i$). We will leave it to you to translate Eq (5) into a concrete computation scheme. No pseudo-code is provided because by now you should be able to do it. However, do make sure that your implementation complies with the input and output data types as specified in the code.

**Hint**:
- `count_frequency()` can be useful for computing the conditional probability.
- You need to apply the **pseudo-count** trick to handle with unseen words when computing the conditional probability for testing data.

```python
def train_NB_model(X_train, y_train):
    """
    training a naive bayes model from the training data.
    Inputs:
    - X_train: an array of shape (num_train,) which stores SMS messages. each entity is a string type SMS message
    - y_train: an array of shape (num_train,). the ground true label for each training data.
    Output:
    - prior: a dictionary, whose key is the class label, and value is the prior probability.
    - conditional: a dictionary whose key is the class label y, and value is another dictionary.
                   In the latter dictionary, the key is word w, and the value is the
                   conditional probability P(X_i = w | y).
    """

    # To make your code more readable, you can implement some auxiliary functions
    # such as `prior_prob` and `conditional_prob` outside of this train_NB_model function

    # compute the prior probability
    prior = prior_prob(y_train)

    # compute the conditional probability
    conditional = conditional_prob(X_train, y_train)

    return prior, conditional

# Start your auxiliary functions

def prior_prob(y_train):
    """
    compute the prior probability
    Inputs:
    - y_train: an array that stores ground true label for training data
    Outputs:
    - prior: a dictionary. key is the class label, value is the prior probability.
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return prior

def conditional_prob(X_train, y_train):
    """
    compute the conditional probability for a document set
    Inputs:
    - X_train: an array of shape (num_train,) which stores SMS messages. each entity is a string type SMS message
    - y_train: an array of shape (num_train,). the ground true label for each training data.
    Ouputs:
    - cond_prob: a dictionary. key is the class label, value is a dictionary in which the key is word, the              value is the conditional probability of feature x_i given y.
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return cond_prob

# unit test case:
# x_train = ['Hello, how are you!',
#           'Win money, win from home.',
#            'Call me now.',
#            'Hello, Call hello you tomorrow?']
# y_train = np.array([0,1,1,0])
#
# sample outputs:
# prior: {0: 0.5, 1: 0.5}
# conditional: {0: {'hello': 0.0001999100404817832, 'how': 9.99550202408916e-05,
#                   'are': 9.99550202408916e-05, 'you': 0.0001499325303613374,
#                   'call': 9.99550202408916e-05, 'tomorrow': 9.99550202408916e-05},
#               1: {'win': 0.00014994002399040384, 'money': 9.996001599360256e-05,
#                   'from': 9.996001599360256e-05, 'home': 9.996001599360256e-05,
#                   'call': 9.996001599360256e-05, 'me': 9.996001599360256e-05,
#                   'now': 9.996001599360256e-05}}
x_train = np.array(['Hello, how are you!',
            'Win money, win from home.',
            'Call me now.',
            'Hello, Call hello you tomorrow?'])
y_train_mini = np.array([0,1,1,0])

prior = prior_prob(y_train_mini)
cond_prob = conditional_prob(x_train, y_train_mini)
print(prior)
print(cond_prob)

```

    {0: 0.5, 1: 0.5}
    {0: {'hello': 0.0001999100404817832, 'how': 9.99550202408916e-05, 'are': 9.99550202408916e-05, 'you': 0.0001499325303613374, 'call': 9.99550202408916e-05, 'tomorrow': 9.99550202408916e-05}, 1: {'win': 0.00014994002399040384, 'money': 9.996001599360256e-05, 'from': 9.996001599360256e-05, 'home': 9.996001599360256e-05, 'call': 9.996001599360256e-05, 'me': 9.996001599360256e-05, 'now': 9.996001599360256e-05}}

#### Step 2: predict label for test data **(16 points)** {-}

Once we have the two models $P(Y)$ and $P(X_i|Y)$ from *training*, we can use them to predict the label for a given test message. To this end, we need to compute the probability of all possible labels, and then predict the one with maximum probability value:
$$
\arg\max_Y P(Y)\prod_{i=1}^T P(X_i|Y). \tag{6}
$$

**Avoid numerical underflow with log-trick.**
As shown in the above equation, the calculation involves multiplying many probabilities together. Since probabilities lie in $(0,1]$, multiplying many of them together can lead to numerical underflow (i.e., a floating point number close to 0 gets rounded down to 0 by a computer), especially when $T$ is large, i.e., the test message is long.

To overcome this problem, it is common to change the calculation from the product of probabilities to the sum of log probabilities.
That is, take the natual logarithm of the right-hand side of Eq (6) as
$$
g_Y(X) = \log P(Y) + \sum_{i=1}^T \log P(X_i|Y). \tag{7}
$$
It is much more numerically stable to compute $g_Y(X)$ and to take $\arg\max_Y g_Y(X)$ to find the most likely class label (as the output prediction).

With $g_Y(X)$, we can easily compute the posterior probability by
$$
P(Y|X) = \frac{\exp (g_Y(X) - m)}{\sum_y \exp (g_y(X)-m)},
\text{ where  }
m = \max_y g_y(X).
$$
In our message classification problem, the summation in the denominator is just over `positive` and `negative`.
Note we subtract by $m$, which does not change the result because the numerator and denominator cancel.
However it is numerically useful because sometimes all $g_Y(X)$ are overly small and can cause numerical underflow inside exponentiation.
By subtracting $m$, $g_Y(X) - m$ will be 0 (properly scaled) for at least one value of $Y$, and be negative for the other.  And even if another $y$ still suffers underflow in exponentiating $g_y(X)-m$, the posterior probabilities will still be correct.

Again, you are expected to implement Eq (7) and loop over all test examples by yourself with no pseudo-code given.

```python
from scipy.special import softmax
def predict_label(X_test, prior_prob, cond_prob):
    """
    predict the class labels for the testing set
    Inputs:
    - X_test: an array of shape (num_test,) which stores test data.
              Each entity is a string type SMS message.
    - prior_prob: a dictionary which stores the prior probability for all categories
    - cond_prob: a dictionary whose key is the class label y, and value is another dictionary.
                   In the latter dictionary, the key is word w, and the value is the
                   conditional probability P(X_i = w | y).
    Outputs:
    - predict: an array that stores predicted labels
    - test_prob: an array of shape (num_test, num_classes) which stores the posterior probability of each class
    """

    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    return predict, test_prob

def compute_test_prob(word_count, prior_cat, cond_cat):
    """
    predict the class label for one test example
    Inputs:
    - word_count: a dictionary which stores the frequencies of each word in a SMS message.
                  Key is the word, value is the number of its occurrence in that message
    - prior_cat: a scalar. prior probability of a specific category
    - cond_cat: a dictionary. conditional probability of a specific category
    Outputs:
    - prob: posterior probability of a specific category for the test example
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    return prob

# unit test case:
# x_test = np.array(['Hello, how are you!'])
# sample outputs:
# y_pred: [0]
# prob: [[0.92298104 0.07701896]]
x_test = np.array(['Hi, how are you today!'])
y_pred, test_prob = predict_label(x_test, prior, cond_prob)
print(y_pred, test_prob)
```

    [0] [[0.92298104 0.07701896]]

#### Step 3: compute performance metrics **(5 points)** {-}
You may have noticed that the classes are heavily imbalanced. There are only 747 `spam` messages, compared with 4827 `ham` messages. If a classifier simply predicts all messages as `ham`, it will get around 86% accuracy (pretty high). Therefore, accuracy is not a good metric in this case for evaluating the performance of the classifier. As we did before, we can use F-score metrics. But this time we will not implement it from the scratch. Instead, we will learn how to use the builtin methods from `scikit`. [Here](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics) is a summary of well-implemented and commonly used metrics for evaluating the quality of a model's predictions.

In this task, you need to **report** the testing accuracy, confusion matrix, and F1 score of the Naive Bayes method by choosing proper functions from `scikit` to compute those metrics with required arguments.

Hint: you need to import methods from `sklearn.metrics` before using them.

```python
def compute_metrics(y_pred, y_true):
    """
    compute the performance metrics
    Inputs:
    - y_pred: an array of predictions
    - y_true: an array of ground true labels
    Outputs:
    - acc: accuracy
    - cm: confusion matrix
    - f1: f1_score
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    return acc, cm, f1

# unit test case:
# y_pred = np.array([0,1,1,1,0,1])
# y_true = np.array([0,1,0,0,1,1])
#
# sample outputs:
# acc: 0.5
# cm: [[1 2]
#      [1 2]]
# f1: 0.5714285714285715
y_pred = np.array([0,1,1,1,0,1])
y_true = np.array([0,1,0,0,1,1])
acc, cm, f1 = compute_metrics(y_pred, y_true)
print(acc, cm, f1)
```

    0.5 [[1 2]
     [1 2]] 0.5714285714285715

#### Step 4: Plot ROC curve and print other results **(10 points)** {-}

ROC (Receiver Operating Characteristics) curve is one of the most commonly used metrics for evaluating the performance of machine learning algorithms, especially when the classes are imbalanced.

ROC is a probability curve for different classes. ROC tells us how good the model is for distinguishing the given classes, in term of the **predicted probability** (not the final hard label in pos/neg). A typical ROC curve has False Positive Rate (FPR) on the $x$-axis and True Positive Rate (TPR) on the $y$-axis. To obtain the FPR and TPR, you can use the `roc_curve` method from `scikit`. This `roc_curve` function takes two arguments: 1) the ground truth labels of the test examples, and 2) the predicted probability that each example is positive. It returns the FPR and TPR which can be used for plotting.

You can even compute the area under the curve (AUC) by calling `roc_auc_score` which takes the same arguments as `roc_curve` required.

In this task, **plot** the ROC curve and compute the AUC score.

```python
from sklearn.metrics import roc_curve, roc_auc_score

def plot_roc_curve(fpr, tpr):
  plt.plot(fpr, tpr, color='blue', label='ROC')
  plt.plot([0, 1], [0, 1], color='black', linestyle='--')
  plt.xlabel('FPR')
  plt.ylabel('TPR')
  plt.title('ROC Curve')
  plt.legend()
  plt.show()

# We now compute the test performance.
# X_train, X_test, y_train, y_test are the same as above

# training naive Bayes model
prior, cond = train_NB_model(X_train, y_train)

# evaluate on test set
y_pred, prob = predict_label(X_test, prior, cond)

# Implement the following:
#   1. compute the fpr and tpr for roc curve using the probability of being positive
#   2. compute the auc score
#   3. plot roc curve by calling the plot_roc_curve() method
#   4. print AUC score, test accuracy, F-score, and Confusion matrix

# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
```

# Problem 2: Linear regression **(38 points)** {-}

In this section, we will explore linear regression model. The dataset we will use for this section is Wine Qualuty, whose description can be found [here](http://archive.ics.uci.edu/ml/datasets/Wine+Quality). This dataset contains **4898** examples, each containing **11** features (the first 11 columns), and the **last** (12-th) column is the value we want to predict. The dataset can be downloaded here [`winequality-white.csv`](https://www.cs.uic.edu/~zhangx/teaching/winequality-white.csv) (our code will download it directly).

Different from classification models, a regression model is used to predict real values rather than the category an example belongs to. Linear regression is a linear approach to modeling the relationship between features and real value target. To perform supervised learning, we represent the hypothesis as a linear function of features ($x$) to predict the output ($y$).

\begin{equation}
f(x) = \theta_0 + \theta_1x_1 + ... + \theta_nx_n    \tag{8}
\end{equation}

Here $\theta_i$'s are the **parameters** parameterizing the space of linear functions mapping from $\mathcal{X}$ to $\mathcal{Y}$. Our goal is to **learn** these parameters so that we can find a linear function in this hypothesis space to estimate the output $y$.

To simplify the notation and ease the computation, we **pad** the input $x$ by letting $x_0=1$. That is, for an example with three features $x=[x_1, x_2, x_3]$, the padded feature vector will be $x=[1,x_1, x_2, x_3]$. Then, the linear function can be written as:

\begin{equation}
f_\theta(x) = \sum_{j=0}^n \theta_j x_j = \theta^\top x    \tag{9}
\end{equation}
where on the right-hand side above we are viewing $\theta$ and $x$ both as vectors, and here $n = 11$ is the number of features.

Given a training set, the way to learn these parameters is to make $f_\theta(x)$ close to $y$. To measure the closeness, we use Mean-Squared-Error (MSE) here. The loss function can therefore be defined as:

\begin{equation}
L(\theta) = \frac{1}{2m}\sum_{i=1}^m(f_\theta(x^{(i)})-y^{(i)})^2 = \frac{1}{2m}\sum_{i=1}^m(\theta^\top x^{(i)}-y^{(i)})^2,  \tag{10}
\end{equation}
where the superscript $(i)$ denotes the $i$-th example,
and $m$ is the total number of training samples. To learn the parameter $\theta$, our goal is to **minimize** the above loss function. In this lab, we will explore two different methods to learn the parameter:

1. Gradient descent
2. Closed-form solution (root of the gradient)

## 2.1 Data preprocessing {-}

Once we have received the dataset, we first need to preprocess it.  Very often, the features in a dataset are of very different scale, which can slow down the optimization for Eq (10). To accelerate it, we need to normalize each feature by substracting its mean value, and then dividing by its standard deviation (std). Assuming $X_i = [x_i^{(1)}, ... , x_i^{(m)}]$ is the $i$-th feature in the training set (across the $m$ examples), the normalized feature $i$ for the $j$-th training example can be computed by:
\begin{equation}
\hat{x}^j_i = \frac{x^j_i - m_i}{s_i},
\text{ where } m_i = mean(X_i), \text{ and } s_i = std(X_i).
  \tag{11}
\end{equation}

**Step 1: normalize the training set (3 points)** {-}

In the following code block, implement a function `featureNormalization`. The input is the training set. The output is the normalized training set, along with the mean and std of each features. You will need the mean and std to apply to the test set later.

```python
import numpy as np
import matplotlib.pyplot as plt
def featureNormalization(X):
    """
    Normalize each feature for the input set
    Input:
    - X: a 2-D numpy array of shape (num_train, num_features)
    Outputs:
    - X_normalized: a 2-D numpy array of shape (num_train, num_features)
    - X_mean: a 1-D numpy array of length (num_features)
    - X_std: a 1-D numpy array of length (num_features)
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    num_train = X.shape[0]
    num_features = X.shape[1]
    X_normalized = np.zeros_like(X)
    X_mean = np.mean(X, axis=0)
    X_std  = np.std(X, axis=0)

    for i in range(num_train):
        X_normalized[i] = (X[i] - X_mean)/ X_std

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return X_normalized, X_mean, X_std

"""
 Unit test case
 Should print
[[ 0.49109521 -0.40242644 -0.92156595 -0.1052653 ]
 [ 0.90298151  1.37532553  1.3897809   1.27398003]
 [-1.39407672 -0.97289909 -0.46821496 -1.16871473]]
[0.99 3.12 4.47 4.51]
[0.63124216 2.26128282 1.34553583 3.70492465]
"""
X = np.array([[1.30,2.21,3.23,4.12], [1.56, 6.23, 6.34, 9.23], [0.11, 0.92, 3.84, 0.18]])
X_normalized, X_mean, X_std = featureNormalization(X)
print(X_normalized)
print(X_mean)
print(X_std)
```

    [[ 0.49109521 -0.40242644 -0.92156595 -0.1052653 ]
     [ 0.90298151  1.37532553  1.3897809   1.27398003]
     [-1.39407672 -0.97289909 -0.46821496 -1.16871473]]
    [0.99 3.12 4.47 4.51]
    [0.63124216 2.26128282 1.34553583 3.70492465]

**Step 2: normalize the test set (3 points)** {-}

The above normalization function will be used for the training set. At test time, we will need to normalize the test data in the same way. However, we shouldn't compute new mean and std from the test set itself, because it may be inconsistent with the training data.  Instead, we will apply the mean $m_i$ and std $s_i$ computed from the training set.  Given a text example $[x_1, \ldots, x_m]$, we just transform $x_i$ into $(x_i - m_i)/s_i$,
where $m_i$ and $s_i$ are computed from the training data as in the *where* clause of Eq (11).

In the following code block, implement a function `applyNormalization`, which normalizes the test set for each feature using the provided mean and std.

```python
def applyNormalization(X, X_mean, X_std):
    """
    Normalize each feature for the input set X
    Input:
    - X: a 2-D numpy array of shape (num_test, num_features)
    - X_mean: a 1-D numpy array of length (num_features)
    - X_std: a 1-D numpy array of length (num_features)

    Output:
    - X_normalized: a 2-D numpy array of shape (num_test, num_features)
    """

    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    X_normalized = np.zeros_like(X)

    for i in range(X.shape[0]):
        X_normalized[i] = (X[i] - X_mean) / X_std
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return X_normalized

"""
Unit test case
Should print
[[ 3.00e-01  1.21e+00  6.15e-01  4.02e+01]
 [ 5.60e-01  5.23e+00  2.17e+00  9.13e+01]
 [-8.90e-01 -8.00e-02  9.20e-01  8.00e-01]]
"""
X =  np.array([[1.30,2.21,3.23,4.12], [1.56, 6.23, 6.34, 9.23], [0.11, 0.92, 3.84, 0.18]])
X_mean = np.array([1.0, 1.0, 2.0, 0.1])
X_std = np.array([1.0, 1.0, 2.0, 0.1])

X_normalized = applyNormalization(X, X_mean, X_std)
print(X_normalized)
```

    (4,)
    [[ 3.00e-01  1.21e+00  6.15e-01  4.02e+01]
     [ 5.60e-01  5.23e+00  2.17e+00  9.13e+01]
     [-8.90e-01 -8.00e-02  9.20e-01  8.00e-01]]

##2.2 Gradient Descent {-}

In this section, you will need to implement the gradient descent algorithm that trains the linear regression model. Some introductions to gradient descent can be found [here](https://ml-cheatsheet.readthedocs.io/en/latest/gradient_descent.html).

**Step 1: implement the loss function (3 points)** {-}

As introduced at the begining of this problem, we will use MSE to measure the loss. In the following code block, implement a function `computeMSE`. Follow Equation (10), and the function should compute the MSE for the input set with the given $\theta$.

```python
def computeMSE(X, y, theta):
    """
    Compute MSE for the input set (X,y) with theta
    Inputs:
    - X: a 2-D numpy array of shape (num_samples, num_features+1)
    - y: a 1-D numpy array of length (num_samples)
    - theta: a 1-D numpy array of length (num_features+1)
    Output:
    - error: MSE, a real number
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    error = [ 1/(2 *  X.shape[0]) *np.sum(np.square(np.dot(theta.T, X.T) - y))]

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return error[0]

# Unit test case:
# Should print 73.0
X = np.array([[2.0, 1.0, 3.0], [3.0, 6.0, 2.0]])
y =  np.array([1.0, 1.0])
theta = np.array([[1.0], [2.0],[1.0]])
error = computeMSE(X, y, theta)
print(error)
```

    73.0

**Step 2: compute the gradient of the loss function (7 points)** {-}

Recall that our goal is to find the parameter $\theta$ that can minimize the loss $L(\theta)$. To find the $\theta$ with gradient descent method, we start from some initial $\theta$, and then repeatedly perform the update:
\begin{equation}
\theta = \theta - \alpha\nabla_{\theta}L(\theta).    \tag{12}
\end{equation}

Here $\alpha > 0$ is a step size, a.k.a., learning rate.
To enable this update rule, we first need to compute the gardient in $\theta$.

1. Please derive the gradient of $\theta$ from Eq (10), and type the result in the following line:

$$
\nabla_{\theta}L(\theta) = X^\top(X\theta- y)
$$

Note for computational efficiency, your expression is not allowed to have $\sum_{i=1}^m$, and cannot have any multiplication of two matrices.  Multiplications of a matrix and a vector is allowed. Here is a hint.  Let $a_i$ and $b_i$ be vectors.  Then
$$
  \sum_i (\theta^\top b_i) a_i = \sum_i a_i (b_i^\top \theta) = \left(\sum_i a_i b_i^\top \right) \theta = (A B^\top) \theta = A (B^\top \theta),
$$
where $A = [a_1, ..., a_m]$ and $B = [b_1, ..., b_m]$.

2. Then, implement a function `computeGradient` to compute the gradient $\nabla_{\theta}L(\theta)$ by following the expression you just derived above.

```python
def computeGradient(X, y, theta):
    """
    Compute the gradient of theta
    Inputs:
    - X: A 2-D numpy array of shape (num_train, num_features+1)
    - y: A 1-D numpy array of length (num_train)
    - theta: A 1-D numpy array of length (num_features+1)
    Output:
    - gradient: A 1-D numpy array of length (num_features+1)
    """

    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    gradient = X.T.dot(X.dot(theta) - y.reshape(y.shape[0],1))/X.shape[0]
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    return gradient

"""
# Unit test case:
# Should return
[[30.]
 [51.]
 [25.]]
"""
X = np.array([[2.0, 1.0, 3.0], [3.0, 6.0, 2.0]])
y =  np.array([1.0, 1.0])
theta = np.array([[1.0], [2.0],[1.0]])
gradient = computeGradient(X, y, theta)
print(gradient)
```

    [[30.]
     [51.]
     [25.]]

**Step 3: implement the gradient descent algorithm (4 points)** {-}

Now we can use the update rule in Equation (12) to find the $\theta$ that minimizes $L(\theta)$. We start from some initial $\theta_0$, then repeatedly take a step in the direction of steepest decrease of $L$. The $\alpha$ in Equation (12) indicates how large the step we want to take at every update. We repeat the updates for a certain number of iterations, and the last updated $\theta$ will be the $\theta$ we find.
In the following code block, implement a function `gradientDescent`, which updates $\theta$ for `num_iters` times and records the loss value (MSE) at every iteration.

```python
def gradientDescent(X, y, theta, alpha, num_iters):
    """
    Update theta using equation (5) for num_iters times.
    Input:
    - X: a numpy array of shape (num_train, num_features+1)
    - y: a numpy array of shape (num_train, 1)
    - theta: a 1-D numpy array of length (num_features+1)
    - alpha: learning rate, a scalar
    - num_iters: an integer specifying how many steps to run the gradient descent
    Outputs:
    - theta: the final theta, a 1-D numpy array of length (num_features+1).
           You can directly overwrite the theta in the input argument, and return it.
    - Loss_record: a 1-D numpy array of length (num_iters),
          recording the loss value of Eq (10) at every iteration,
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    Loss_record = np.zeros((num_iters))
    m = len(y)
    for i in range(num_iters):
        #theta = theta - alpha*(1.0/m) * np.transpose(X).dot(X.dot(theta) - np.transpose([y]))
        theta = theta - alpha * computeGradient(X, y, theta)
        Loss_record[i] = computeMSE(X, y, theta)
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    return theta, Loss_record

"""
# Unit test case:
# Should return
[[0.3322825]
 [0.858839 ]
 [0.446925 ]]
[37.5778     19.36559064 10.00046345]
"""
X = np.array([[2.0, 1.0, 3.0], [3.0, 6.0, 2.0]])
y =  np.array([1.0, 1.0])
theta = np.array([[1.0], [2.0],[1.0]])
alpha = 0.01
num_iters = 3
theta, Loss_record = gradientDescent(X, y, theta, alpha, num_iters)
print(theta)
print(Loss_record)
```

    [[0.3322825]
     [0.858839 ]
     [0.446925 ]]
    [37.5778     19.36559064 10.00046345]

## 2.3 Train the linear regression model with gradient descent {-}

Now we are ready to chain all the above functions together to perform the linear regression training on the Wine Quality dataset.

**Step 1: load the data (not for grading)**

```python
# First load the data (this code block is not for grading)

import urllib.request
import shutil

url = 'https://www.cs.uic.edu/~zhangx/teaching/winequality-white.csv'
file_name = 'winequality-white.csv'
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
    shutil.copyfileobj(response, out_file)

data = np.genfromtxt(file_name, delimiter=";", skip_header=1)
print(data.shape)
```

    (4898, 12)

**Step 2: training and testing (7 points)**

After loading the dataset, split the dataset into training and test sets. Please split the first **4000** samples as training set and the rest as test set. Then perform the following:

*   Normalize training set features
*   Pad the normalized training features by a constant 1, as the new first feature
*   Initialize $\theta$ as a zero vector
*   Update $\theta$ using gradient descent (`num_iters` and `alpha` are provided)
*   **Plot** a figure where $x$-axis is the number of iterations, $y$-axis is the loss value (MSE).
*   Apply normalization to test set features, pad the features
*   Compute the test error (MSE) and **print** out the test error

```python

num_train = 4000
alpha = 0.01
num_iters = 500

# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
X_train = data[:4000, :11]
Y_train = data[:4000, 11]

X_test  = data[4000:, :11]
Y_test  = data[4000:, 11]

X_train_norm, X_mean, X_std = featureNormalization(X_train)
pad = np.ones((X_train.shape[0],1))
X_train_norm = np.hstack((pad, X_train_norm))
theta = np.zeros((X_train_norm.shape[1], 1))
theta, loss_record = gradientDescent(X_train_norm, Y_train, theta, alpha, num_iters)
plt.plot(np.arange(len(loss_record)), loss_record)
plt.xlabel("Number of iteration")
plt.ylabel("MSE")

pad = np.ones((X_test.shape[0],1))
X_test_norm = applyNormalization(X_test, X_mean, X_std)
X_test_norm = np.hstack((pad, X_test_norm))

error = computeMSE(X_test_norm, Y_test, theta)
print(f"test MSE using gradient descent is: {error}\n")

# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
```

## 2.4 Effect of different learning rate (4 points) {-}

To investigate the effect of learning rate, repeat the learning process (gradient descent) with different learning rate in $[1.0, 0.1, 0.01, 0.001]$.
**Plot** 4 figures corresponding to different learning rates, where the $x$-axis is the number of iterations,
and the $y$-axis is the loss value (MSE). **Print** the test error (MSE) respectively in the format "test MSE for using learning rate __ is __" (four lines in total).

```python
learning_rates = [1.0, 0.1, 0.01, 0.001]

# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
fig, axs = plt.subplots(2, 2)

m = np.arange(num_iters)

quadrant = [(0,0),(0,1),(1,0),(1,1)]

for i, l in enumerate(learning_rates):
    theta = np.zeros((X_train_norm.shape[1], 1))
    theta, loss_record = gradientDescent(X_train_norm, Y_train, theta, l, num_iters)
    error = computeMSE(X_test_norm, Y_test, theta)
    print(f"test MSE for using learning rate {i} is {error}\n")
    axs[quadrant[i][0], quadrant[i][1]].plot(m, loss_record)
    axs[quadrant[i][0], quadrant[i][1]].set_title(f'Alpha = {i}')

for ax in axs.flat:
    ax.label_outer()

# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
```

## 2.5 Closed-form solution  {-}

Gradient descent minimizes $L$ by updating $\theta$ iteratively. There is another way to find the $\theta$ explicitly. Indeed, by finding the root of the gradient $\nabla_\theta L(\theta)$ (i.e., the $\theta$ such that $\nabla_\theta L(\theta) = 0$), we can obtain a closed-form solution of $\theta$ that minimizes the loss $L$.

**Step 1: find the root of the gradient to obtain the closed-form solution of $\theta$  (5 points)**

Type your result in the following lines:

$$
\nabla_{\theta}L(\theta) = 0 \quad \Rightarrow \quad
\theta = (X^TX)^−1 X^Ty
$$
Then, implement a function `closeForm` to compute the closed-form solution of $\theta$ using the expression you have derived above.

```python
def closeForm(X, y):
    """
    Compute close form solution for theta
    Inputs:
    - X: a numpy array of shape (num_train, num_features+1)
    - y: a 1-D numpy array of length (num_train)
    Output:
    - theta: a 1-D numpy array of length (num_features+1)
    """
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    return np.expand_dims(theta, axis=1)

"""
# Unit test case:
# Should return
[[0.13812155]
 [0.01933702]
 [0.23480663]]
"""
# X = np.array([[2.0, 1.0, 3.0], [3.0, 6.0, 2.0]])
# y =  np.array([1.0, 1.0])
# theta = closeForm(X, y)

X = np.array([[2.0, 1.0, 3.0], [3.0, 6.0, 2.0], [-1, 0, 2.0]])
y = np.array([1.0, 1.0, -1.0])
theta = closeForm(X, y)

print(theta)
```

    [[ 0.76470588]
     [-0.17647059]
     [-0.11764706]]

**Step 2: evaluate the test error using closed-form solution (2 points)** {-}

Compute a new $\theta$ using the closed-form solution. Evaluate the new $\theta$ on test set by **printing** the test error (MSE) in the format: "test MSE using close form solution is : __".

```python
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
theta = closeForm(X_train_norm, Y_train)
test_error = computeMSE(X_test_norm, Y_test, theta)
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

print('test MSE using close form solution is : ', test_error)

```

    test MSE using close form solution is :  0.25606452665329565

# Submission Instruction {-}

You're almost done! Take the following steps to finally submit your work.

1. After executing all commands and completing this notebook, save your `Lab_2.ipynb` as a PDF file, named as `X_Y_UIN.pdf`, where `X` is your first name, `Y` is your last name, and `UIN` is your UIN. Make sure that your PDF file includes all parts of your solution, including the plots.

> * Print out all unit test case results before printing the notebook into a PDF.
* If you use Colab, open this notebook in Chrome.  Then File -> Print -> set Destination to "Save as PDF".  If the web page freezes when printing, close Chrome and reopen the page. If Chrome doesn't work, try Firefox.
* If you are working on your own computer, we recommend using the browser (not jupyter) for saving the PDF. For Chrome on a Mac, this is under *File->Print...->Open PDF in Preview*. When the PDF opens in Preview, you can use *Save...* to save it.
* Sometimes, a figure that appears near the end of a page can get cut.  In this case, try to add some new lines in the preceding code block so that the figure is pushed to the beginning of the next page.

2. Upload `X_Y_UIN.pdf` to Gradescope under `Lab_2_Written`.

3. A template of `Lab_2.py` has been provided.  For all functions in `Lab_2.py`, copy the corresponding code snippets you have written into it.  Do not copy any code of plotting figures. **Do not** change the function names.

4. Zip `Lab_2.py` and `Lab_2.ipynb` (**2 files**) into a zip file named `X_Y_UIN.zip`. Suppose the two files are in the folder `Lab_2`.  Then zip up the two files inside the `Lab_2` folder.  Do NOT zip up the folder `Lab_2`. Submit this zip file to Gradescope under `Lab_2_Code`.

5. The autograder on Gradscope will be open all the time. We designed some simple test cases to help you check wehther your functions is execuable. You will see the results of running autogrder once you submit your code. If you see 'test `function_name` completed', it means your function is execuable; otherwise please follow the error messages to debug. Since those simple test cases are designed for debugging, it does notguaranttee your solution will work well on real dataset. It is your responsibility to make your code logically correct.

6. Only one member of each team needs to submit.  But please specify **all** your teammates on Gradescope. After each submission, please click 'Add Group Member' (right up corner of your submission page) to claim all teammates.

You can submit to Gradescope as many times as you would like. We will only consider your last submission.


# Design of Digital Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Design of Non-Recursive Filters using the Window Method

The design of non-recursive filters with a finite-length impulse response (FIR) is a frequent task in practical applications. The designed filter should approximate a prescribed frequency response as close as possible. First, the design of causal filters is considered. For many applications the resulting filter should have a linear phase characteristic since this results in a constant (frequency independent) group delay. We therefore specialize the design to causal linear-phase filters in a second step.

### Causal Filters

Let's assume that the desired frequency characteristic of the discrete filter is given by its continuous frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ in the discrete-time Fourier domain. Its impulse response is given by inverse discrete-time Fourier transform (inverse DTFT) of the frequency response

\begin{equation}
h_\text{d}[k] = \frac{1}{2 \pi} \int\limits_{- \pi}^{\pi} H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \, \mathrm{e}^{\,\mathrm{j}\,\Omega\,k} \; \mathrm{d}\Omega
\end{equation}

In the general case, $h_\text{d}[k]$ will not be a causal FIR. The [Paley-Wiener theorem](https://en.wikipedia.org/wiki/Paley%E2%80%93Wiener_theorem) states, that the transfer function of a causal system may only have zeros at a countable number of single frequencies. This is not the case for idealized filters, like e.g. the [ideal low-pass filter](https://en.wikipedia.org/wiki/Low-pass_filter#Ideal_and_real_filters), were the transfer function is zeros over an interval of frequencies. The basic idea of the window method is to truncate the impulse response $h_\text{d}[k]$ in order to derive a causal FIR filter. This can be achieved by applying a window $w[k]$ of finite length $N$ to $h_\text{d}[k]$

\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

where $h[k]$ denotes the impulse response of the designed filter and $w[k] = 0$ for $k < 0 \land k \geq N$. Its frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given by the multiplication theorem of the discrete-time Fourier transform (DTFT)

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{2 \pi} \; H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \circledast W(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

where $W(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ denotes the DTFT of the window function $w[k]$. The frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the filter is given as the periodic convolution of the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the frequency response of the window function $W(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is equal to the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ only if $W(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 2 \pi \cdot \delta(\Omega)$. This would require that $w[k] = 1$ for $k = -\infty, \dots, \infty$. Hence for a window $w[k]$ of finite length, deviations from the desired frequency response are to be expected.

In order to investigate the effect of truncation on the frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, a particular window is considered. A straightforward choice is the rectangular window $w[k] = \text{rect}_N[k]$ of length $N$. Its DTFT is given as

\begin{equation}
W(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathrm{e}^{-\mathrm{j} \, \Omega \,\frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,\Omega}{2})}{\sin(\frac{\Omega}{2})}
\end{equation}

The frequency-domain properties of the rectangular window have already been discussed for the [leakage effect](../spectral_analysis_deterministic_signals/leakage_effect.ipynb). The rectangular window features a narrow main lobe at the cost of relative high sidelobe level. The main lobe gets narrower with increasing length $N$. The convolution of the desired frequency response with the frequency response of the window function effectively results in smoothing and ringing. While the main lobe will smooth discontinuities of the desired transfer function, the sidelobes result in undesirable ringing effects. The latter can be alleviated by using other window functions. Note that typical [window functions](../spectral_analysis_deterministic_signals/window_functions.ipynb) decay towards their ends and are symmetric with respect to their center. This may cause problems for desired impulse responses with large magnitudes towards their ends.

#### Example - Causal approximation of ideal low-pass

The design of an ideal low-pass filter using the window method is illustrated in the following. For $|\Omega| < \pi$ the transfer function of the ideal low-pass is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \begin{cases}
1 & \text{for } |\Omega| \leq \Omega_\text{c} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

where $\Omega_\text{c}$ denotes the cut frequency of the low-pass. An inverse DTFT of the desired transfer function yields

\begin{equation}
h_\text{d}[k] = \frac{\Omega_\text{c}}{\pi} \cdot \text{sinc}(\Omega_\text{c} \, k)
\end{equation}

The impulse response $h_\text{d}[k]$ is not causal nor FIR. In order to derive a causal FIR approximation, a rectangular window $w[k]$ of length $N$ is applied

\begin{equation}
h[k] = h_\text{d}[k] \cdot \text{rect}_N[k]
\end{equation}

The resulting magnitude and phase response is computed numerically in the following.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig
%matplotlib inline

N = 32  # length of filter
Omc = np.pi/2

# compute impulse response
k = np.arange(N)
hd = Omc/np.pi * np.sinc(k*Omc/np.pi)
# windowing
w = np.ones(N)
h = hd * w

# frequency response
Om, H = sig.freqz(h)

# plot impulse response
plt.figure(figsize=(10, 3))
plt.stem(h, use_line_collection=True)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot magnitude responses
plt.figure(figsize=(10, 3))
plt.plot([0, Omc, Omc], [0, 0, -100], 'r--', label='desired')
plt.plot(Om, 20 * np.log10(abs(H)), label='window method')
plt.title('Magnitude response')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -20, 3])
plt.grid()
plt.legend()
# plot phase responses
plt.figure(figsize=(10, 3))
plt.plot([0, Om[-1]], [0, 0], 'r--', label='desired')
plt.plot(Om, np.unwrap(np.angle(H)), label='window method')
plt.title('Phase')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi (\Omega)$ in rad')
plt.grid()
plt.legend()
```

    <matplotlib.legend.Legend at 0x1c1af92ed0>

**Exercises**

* Does the resulting filter have the desired phase?
* Increase the length `N` of the filter. What changes?

Solution: The desired filter has zero-phase for all frequencies, hence $\varphi_\text{d}(\Omega) = 0$. The phase of the resulting filter is not zero as can be concluded from the lower illustration. The small local variations (ripples) in the magnitude $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ of the transfer function of the resulting filter decrease with an increasing number `N` of filter coefficients. The achievable attenuation in the stop-band of the low-pass does not change.

### Zero-Phase Filters

Lets assume a general zero-phase filter with transfer function $H_d(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ with magnitude $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})  \in \mathbb{R}$. Due to the symmetry relations of the DTFT, its impulse response $h_d[k] = \mathcal{F}_*^{-1} \{ H_d(e^{j \Omega} \}$ is conjugate complex symmetric

\begin{equation}
h_d[k] = h_d^*[-k]
\end{equation}

A zero-phase filter of length $N > 1$ is not causal as a consequence. The anti-causal part could simply be removed by windowing with a heaviside signal. However, this will result in large deviations between the desired transfer function and the designed filter. This explains the findings from the previous example, that an ideal-low pass cannot be realized very well by the window method. The reason is that an ideal-low pass has zero-phase, as most of the idealized filters.

The impulse response of a stable system, in the sense of the bounded-input/bounded-output (BIBO) criterion, has to be absolutely summable. Which in general is given when its magnitude decays by tendency with increasing time-index $k$.
This observation motivates to shift the desired impulse response to the center of the window in order to limit the effect of windowing. This can be achieved by replacing the zero-phase with a linear-phase, as is illustrated below.

### Causal Linear-Phase Filters

The design of a non-recursive causal FIR filter with a linear phase is often desired due to its constant group delay. Let's assume a filter with generalized linear phase. For $|\Omega| < \pi$ its transfer function is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \mathrm{e}^{-\mathrm{j} \alpha \Omega + \mathrm{j} \beta}
\end{equation}

where $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \in \mathbb{R}$ denotes the amplitude of the filter, $\alpha$ the linear slope of the phase and $\beta$ a constant phase offset. Such a system can be decomposed into two cascaded systems: a zero-phase system with transfer function $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and an all-pass with phase $\varphi(\Omega) = - \alpha \Omega + \beta$. The linear phase term $- \alpha \Omega$ results in the constant group delay $t_g(\Omega) = \alpha$.

The impulse response $h[k]$ of a linear-phase system shows a specific symmetry which can be deduced from the symmetry relations of the DTFT for odd/even symmetry of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ as

\begin{equation}
h[k] = \pm h[N-1-k]
\end{equation}

for $k=0, 1, \dots, N-1$ where $N \in \mathbb{N}$ denotes the length of the (finite) impulse response. The transfer function of a linear phase filter is given by its DTFT

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k=0}^{N-1} h[k] \, \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k}
\end{equation}

Introducing the symmetry relations of the impulse response $h[k]$ into the DTFT and comparing the result with above definition of a generalized linear phase system reveals four different types of linear-phase systems. These can be discriminated with respect to their phase and magnitude characteristics

| Type | Length $N$ | Impulse Response $h[k]\;$ | Group Delay $\alpha$ in Samples| Constant Phase $\beta$ | Transfer Function $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ |
| :---:  | :---: | :---: | :---:| :---: | :---: |
| 1 | odd | $h[k] = h[N-1-k]$ | $\alpha = \frac{N-1}{2} \in \mathbb{N}$ | $\beta = \{0, \pi\}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})=A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, all filter characteristics|
| 2 | even| $h[k] = h[N-1-k]$ | $\alpha = \frac{N-1}{2} \notin \mathbb{N}$ | $\beta = \{0, \pi\}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})=A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,\pi}) = 0$, only lowpass or bandpass|
| 3 | odd | $h[k] = -h[N-1-k]$ | $\alpha = \frac{N-1}{2} \in \mathbb{N}$ | $\beta = \{ \frac{\pi}{2}, \frac{3 \pi}{2} \}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})=-A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,0}) = A(\mathrm{e}^{\,\mathrm{j}\,\pi}) =  0$, only bandpass|
| 4 | even | $h[k] = -h[N-1-k]$ | $\alpha = \frac{N-1}{2} \notin \mathbb{N}$ | $\beta = \{ \frac{\pi}{2}, \frac{3 \pi}{2} \}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})=-A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,0}) = 0$, only highpass or bandpass|

These relations have to be considered in the design of a causal linear phase filter. Depending on the desired magnitude characteristics $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ the suitable type is chosen. The odd/even length $N$ of the filter and the phase (or group delay) is chosen accordingly for the design of the filter.

#### Example - Causal linear-phase approximation of ideal low-pass

We aim at the design of a causal linear-phase low-pass using the window technique. According to the previous example, the desired frequency response has an even symmetry $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,-\mathrm{j}\,\Omega})$ with $A(\mathrm{e}^{\mathrm{j}\,0}) = 1$. This could be realized by a filter of type 1 or 2. We choose type 1 with $\beta = 0$, since the resulting filter exhibits an integer group delay of $t_g(\Omega) = \frac{N-1}{2}$ samples. Consequently the length of the filter $N$ has to be odd.

The impulse response $h_\text{d}[k]$ is given by the inverse DTFT of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ incorporating the linear phase

\begin{equation}
h_\text{d}[k] = \frac{\Omega_\text{c}}{\pi} \cdot \text{sinc} \left( \Omega_\text{c} \left(k-\frac{N-1}{2} \right) \right)
\end{equation}

The impulse response fulfills the desired symmetry for $k=0,1, \dots, N-1$. A causal FIR approximation is obtained by applying a window function of length $N$ to the impulse response $h_\text{d}[k]$

\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

Note that the window function $w[k]$ also has to fulfill the desired symmetries.

As already outlined, the chosen window determines the properties of the transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The [spectral properties of commonly applied windows](../spectral_analysis_deterministic_signals/window_functions.ipynb) have been discussed previously. The width of the main lobe will generally influence the  smoothing of the desired transfer function $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, while the sidelobes influence the typical ringing artifacts. This is illustrated in the following.

```python
N = 33  # length of filter
Omc = np.pi/2

# compute impulse response
k = np.arange(N)
hd = Omc/np.pi * np.sinc((k-(N-1)/2)*Omc/np.pi)
# windowing
w1 = np.ones(N)
w2 = np.blackman(N)
h1 = hd * w1
h2 = hd * w2

# frequency responses
Om, H1 = sig.freqz(h1)
Om, H2 = sig.freqz(h2)

# plot impulse response
plt.figure(figsize=(10, 3))
plt.stem(h1, use_line_collection=True)
plt.title('Impulse response (rectangular window)')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot magnitude responses
plt.figure(figsize=(10, 3))
plt.plot([0, Omc, Omc], [0, 0, -300], 'r--', label='desired')
plt.plot(Om, 20 * np.log10(abs(H1)), label='rectangular window')
plt.plot(Om, 20 * np.log10(abs(H2)), label='Blackmann window')
plt.title('Magnitude response')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -120, 3])
plt.legend(loc=3)
plt.grid()
# plot phase responses
plt.figure(figsize=(10, 3))
plt.plot(Om, np.unwrap(np.angle(H1)), label='rectangular window')
plt.plot(Om, np.unwrap(np.angle(H2)), label='Blackmann window')
plt.title('Phase')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi (\Omega)$ in rad')
plt.legend(loc=3)
plt.grid()
```

**Exercises**

* Does the impulse response fulfill the required symmetries for a type 1 filter?
* Can you explain the differences between the magnitude responses $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ for the different window functions?
* What happens if you increase the length `N` of the filter?

Solution: Inspection of the impulse response reveals that it shows the symmetry $h[k] = h[N-1-k]$ of a type 1 filter for odd `N`. The rectangular window features a narrow main lobe at the cost of a high level of the side lobes, the main lobe of the Blackmann window is wider but the level of the side lobes is lower compared to the rectangular window. This explains the behavior of the magnitude responses $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ in the stop-band of the realized low-passes. The distance between the local minima in the magnitude responses $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ decreases with increasing length `N` and the attenuation for frequencies towards the Nyquist frequency increases.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# Übungsblatt 3: sWeights

 * [Aufgabe 1](#Aufgabe-1)
 * [Aufgabe 2](#Aufgabe-2)

Eine experimentelle Verteilung in den Variablen $(x, m)$ habe eine Signalkomponente $s(x, m)$ = $s(x)s(m)$ und eine Untergrundkomponente $b(x,m)$ = $b(x)b(m)$. Der erlaubte Bereich ist $0 < x < 1$ und $0 < m < 1$. Es sei $s(m)$ eine Gaussverteilung mit Mittelwert $\mu = 0.5$ und Standardabweichung $\sigma = 0.05$. Die Verteilungen der anderen Komponenten werden aus gleichverteilten Zufallzahlen $z$ gewonnen. Für $s(x)$ verwende man $x = −0.2\ln{z}$, für $b(m)$ verwende man $m = \sqrt{z}$ und für $b(x)$ die Transformation $x = 1 − \sqrt{z}$.

Erzeugen Sie für zwei angenommene Effizienzfunktionen

 * $\varepsilon(x, m) = 1$
 * $\varepsilon(x, m) = (x + m) / 2$

Datensätze von Paaren $(x, m)$ die 20000 akzeptierte Signalereignisse und 100000 akzeptierte Untergrundereignisse umfassen.
Betrachten Sie nun die gemeinsame $m$-Verteilung und parametrisieren Sie diese durch

\begin{equation}
    f(m) = s(m) + b(m)
\end{equation}

mit

\begin{equation}
    s(m) = p_0 \exp\left(-\frac{(m - p_1)^2}{2p_p^2}\right)
\end{equation}

und

\begin{equation}
    b(m) = p_3 + p_4m + p_5m^2 + p_6\sqrt{m} \,.
\end{equation}

 Für den Fall $\varepsilon(x, m) = (x + m)/2$ benutzen Sie die obige Parametrisierung auch zur Beschreibung der $m_c$ und $m_{cc}$-Verteilungen, für die jeder $m$-Wert mit $1/\varepsilon(x, m)$, bzw. $1/\varepsilon^2(x, m)$ gewichtet wird, und die für die korrekte Behandlung von nicht-konstanten Effizienzen benötigt werden.

---

```python

```

---
## Aufgabe 1
Bestimmen Sie für beide Effizienzfunktion die sWeights $w(m)$ aus den beobachteten $m$-Verteilungen, und verwenden Sie $w(m)/\varepsilon(x, m)$ um die Verteilung $N_{s}s(x)$ aus den Daten heraus zu projizieren. Vergleichen Sie für beide Effizienzfunktionen das Resultat mit der Erwartung.

---

```python

```

---
## Aufgabe 2
Bestimmen Sie für $\varepsilon(x, m) = (x + m)/2$ unter Berücksichtigung der Funktion $\varepsilon(x, m)$ in der Bestimmung von $w(m)$ die korrekten sWeights aus den mit $1/\varepsilon(x, m)$ gewichteten Daten. Verwenden Sie die korrekten sWeights um mit $w(m)/\varepsilon(x, m)$ um die Verteilung $N_{s}s(x)$ zu extrahieren.

```python

```


# 14 Linear Algebra: String Problem
## Motivating problem: Two masses on three strings
Two masses $M_1$ and $M_2$ are hung from a horizontal rod with length $L$ in such a way that a rope of length $L_1$ connects the left end of the rod to $M_1$, a rope of length $L_2$ connects $M_1$ and $M_2$, and a rope of length $L_3$ connects $M_2$ to the right end of the rod. The system is at rest (in equilibrium under gravity).

Find the angles that the ropes make with the rod and the tension forces in the ropes.

## Theoretical background
Treat $\sin\theta_i$ and $\cos\theta_i$ together with $T_i$, $1\leq i \leq 3$, as unknowns that have to simultaneously fulfill the nine equations
\begin{align}
-T_1 \cos\theta_1 + T_2\cos\theta_2 &= 0\\
 T_1 \sin\theta_1 - T_2\sin\theta_2 - W_1 &= 0\\
                   -T_2\cos\theta_2 + T_3\cos\theta_3 &= 0\\
                    T_2\sin\theta_2 + T_3\sin\theta_3 - W_2 &= 0\\
 L_1\cos\theta_1 + L_2\cos\theta_2 + L_3\cos\theta_3 - L &= 0\\
-L_1\sin\theta_1 -L_2\sin\theta_2 + L_3\sin\theta_3  &= 0\\
\sin^2\theta_1 + \cos^2\theta_1 - 1 &= 0\\
\sin^2\theta_2 + \cos^2\theta_2 - 1 &= 0\\
\sin^2\theta_3 + \cos^2\theta_3 - 1 &= 0
\end{align}

Consider the nine equations a vector function $\mathbf{f}$ that takes a 9-vector $\mathbf{x}$ of the unknowns as argument:
\begin{align}
\mathbf{f}(\mathbf{x}) &= 0\\
\mathbf{x} &= \left(\begin{array}{c}
x_0 \\ x_1 \\ x_2 \\
x_3 \\ x_4 \\ x_5 \\
x_6 \\ x_7 \\ x_8
\end{array}\right)
 =
\left(\begin{array}{c}
\sin\theta_1 \\ \sin\theta_2 \\ \sin\theta_3 \\
\cos\theta_1 \\ \cos\theta_2 \\ \cos\theta_3 \\
T_1 \\ T_2 \\ T_3
\end{array}\right) \\
\mathbf{L} &= \left(\begin{array}{c}
L \\ L_1 \\ L_2 \\ L_3
\end{array}\right), \quad
\mathbf{W} = \left(\begin{array}{c}
W_1 \\ W_2
\end{array}\right)
\end{align}

Solve with generalized Newton-Raphson:
$$
\mathsf{J}(\mathbf{x}) \Delta\mathbf{x} = -\mathbf{f}(\mathbf{x})
$$
and
$$
\mathbf{x} \leftarrow \mathbf{x} + \Delta\mathbf{x}.
$$

## Problem setup
Set the problem parameters and the objective function $\mathbf{f}(\mathbf{x})$

```python
import numpy as np

# problem parameters
W = np.array([10, 20])
L = np.array([8, 3, 4, 4])

def f_2masses(x, L, W):
    return np.array([
            -x[6]*x[3] + x[7]*x[4],
             x[6]*x[0] - x[7]*x[1] - W[0],
            -x[7]*x[4] + x[8]*x[5],
             x[7]*x[1] + x[8]*x[2] - W[1],
             L[1]*x[3] + L[2]*x[4] + L[3]*x[5] - L[0],
            -L[1]*x[0] - L[2]*x[1] + L[3]*x[2],
            x[0]**2 + x[3]**2 - 1,
            x[1]**2 + x[4]**2 - 1,
            x[2]**2 + x[5]**2 - 1,
        ])

def fLW(x):
    return f_2masses(x, L, W)
```

### Initial values
Guess some initial values (they don't have to fullfil the equations!):

```python
# initial parameters
theta0 = np.deg2rad([45, 45, 90])
T0 = np.array([1, 1, 2])
x0 = np.concatenate([np.sin(theta0), np.cos(theta0), T0])
```

```python
x0
```

    array([  7.07106781e-01,   7.07106781e-01,   1.00000000e+00,
             7.07106781e-01,   7.07106781e-01,   6.12323400e-17,
             1.00000000e+00,   1.00000000e+00,   2.00000000e+00])

```python
f_2masses(x0, L, W)
```

    array([  0.        , -10.        ,  -0.70710678, -17.29289322,
            -3.05025253,  -0.94974747,   0.        ,   0.        ,   0.        ])

### Visualization
Plot the positions of the 2 masses and the 3 strings for any solution vector $\mathbf{x}$:

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def plot_2masses(x, L, W, **kwargs):
    """Plot 2 mass/3 string problem for parameter vector x and parameters L and W"""

    kwargs.setdefault('linestyle', '-')
    kwargs.setdefault('marker', 'o')
    kwargs.setdefault('linewidth', 1)

    r0 = np.array([0, 0])
    r1 = r0 + np.array([L[0], 0])
    rod = np.transpose([r0, r1])

    L1 = r0 + np.array([L[1]*x[3], -L[1]*x[0]])
    L2 = L1 + np.array([L[2]*x[4], -L[2]*x[1]])
    L3 = L2 + np.array([L[3]*x[5], L[3]*x[2]])
    strings = np.transpose([r0, L1, L2, L3])

    ax = plt.subplot(111)
    ax.plot(rod[0], rod[1], color="black", marker="d", linewidth=4)
    ax.plot(strings[0], strings[1], **kwargs)
    ax.set_aspect(1)
    return ax
```

What does the initial guess look like?

```python
plot_2masses(x0, L, W)
```

## Jacobian
Write a function `Jacobian(f, x, h=1e-5)` that computes the Jacobian matrix numerically (use the central difference algorithm).

```python
def Jacobian(f, x, h=1e-5):
    """df_i/dx_j with central difference (f(x+h/2)-f(x-h/2))/h"""
    J = np.zeros((len(f(x)), len(x)), dtype=np.float64)
    hvec = np.zeros_like(x, dtype=np.float64)
    for j in range(len(x)):
        hvec *= 0
        hvec[j] = 0.5*h
        J[:, j] = (f(x + hvec) - f(x - hvec))/h
    return J
```

Test Jacobian on
$$
\mathbf{f}(\mathbf{x}) = \left( \begin{array}{c}
  x_0^2 - x_1 \\ x_0
  \end{array}\right)
$$
with analytical result
$$
\mathsf{J} = \frac{\partial f_i}{\partial x_j} =
\left( \begin{array}{cc}
  2 x_0 & -1\\
  1 & 0
\end{array}\right)
$$

```python
def ftest(x):
    return np.array([
            x[0]**2 - x[1],
            x[0]
        ])
x0test = np.array([1, 0])
J = Jacobian(ftest, x0test)
print(J)
```

    [[ 2. -1.]
     [ 1.  0.]]

Test that it also works for our starting vector:

```python
Jacobian(fLW, x0)
```

    array([[ 0.        ,  0.        ,  0.        , -1.        ,  1.        ,
             0.        , -0.70710678,  0.70710678,  0.        ],
           [ 1.        , -1.        ,  0.        ,  0.        ,  0.        ,
             0.        ,  0.70710678, -0.70710678,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        , -1.        ,
             2.        ,  0.        , -0.70710678,  0.        ],
           [ 0.        ,  1.        ,  2.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.70710678,  1.        ],
           [ 0.        ,  0.        ,  0.        ,  3.        ,  4.        ,
             4.        ,  0.        ,  0.        ,  0.        ],
           [-3.        , -4.        ,  4.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ],
           [ 1.41421356,  0.        ,  0.        ,  1.41421356,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  1.41421356,  0.        ,  0.        ,  1.41421356,
             0.        ,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  2.        ,  0.        ,  0.        ,
             0.        ,  0.        ,  0.        ,  0.        ]])

## n-D Newton-Raphson Root Finding
Write a function `newton_raphson(f, x, Nmax=100, tol=1e-8, h=1e-5)` to find a root for a vector function `f(x)=0`. (See also [13 Root-finding by trial-and-error](http://asu-compmethodsphysics-phy494.github.io/ASU-PHY494//2017/03/16/13_Root_finding/) and the _1D Newton-Raphson algorithm_ in [13-Root-finding.ipynb](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/13_root_finding/13-Root-finding.ipynb).) As a convergence criterion we demand that the length of the vector `f(x)` (the norm --- see `np.linalg.norm`) be less than the tolerance.

```python
def newton_raphson(f, x, Nmax=100, tol=1e-8, h=1e-5):
    """n-D Newton-Raphson: solves f(x) = 0.

    Iterate until |f(x)| < tol or nmax steps.
    """
    x = x.copy()
    for istep in range(Nmax):
        fx = f(x)
        if np.linalg.norm(fx) < tol:
            break
        J = Jacobian(f, x, h=h)
        Delta_x = np.linalg.solve(J, -fx)
        x += Delta_x
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
            "best guess is {2} with error {3}".format(Nmax, tol, x, fx))
    return x
```

### Solve 2 masses/3 strings string problem

#### Solution

```python
x = newton_raphson(fLW, x0)
print(x0)
print(x)
```

    [  7.07106781e-01   7.07106781e-01   1.00000000e+00   7.07106781e-01
       7.07106781e-01   6.12323400e-17   1.00000000e+00   1.00000000e+00
       2.00000000e+00]
    [  0.76100269   0.26495381   0.83570583   0.64874872   0.9642611
       0.54917735  17.16020978  11.54527968  20.27152804]

Plot the starting configuration and the solution:

```python
plot_2masses(x0, L, W)
plot_2masses(x, L, W)
```

Pretty-print the solution (angles in degrees):

```python
def pretty_print(x):
    theta = np.rad2deg(np.arcsin(x[0:3]))
    tensions = x[6:]
    print("theta1 = {0[0]:.1f} \t theta2 = {0[1]:.1f} \t theta3 = {0[2]:.1f}".format(theta))
    print("T1     = {0[0]:.1f} \t T2     = {0[1]:.1f} \t T3     = {0[2]:.1f}".format(tensions))
```

```python
print("Starting values")
pretty_print(x0)
print()
print("Solution")
pretty_print(x)
```

    Starting values
    theta1 = 45.0 	 theta2 = 45.0 	 theta3 = 90.0
    T1     = 1.0 	 T2     = 1.0 	 T3     = 2.0

    Solution
    theta1 = 49.6 	 theta2 = 15.4 	 theta3 = 56.7
    T1     = 17.2 	 T2     = 11.5 	 T3     = 20.3

#### Show intermediate steps
Create a new function `newton_raphson_intermediates()` based on `newton_raphson()` that returns *all* trial `x` values including the last one.

```python
def newton_raphson_intermediates(f, x, Nmax=100, tol=1e-8, h=1e-5):
    """n-D Newton-Raphson: solves f(x) = 0.

    Iterate until |f(x)| < tol or nmax steps.

    Returns all intermediates.
    """
    intermediates = []
    x = x.copy()
    for istep in range(Nmax):
        fx = f(x)
        if np.linalg.norm(fx) < tol:
            break
        J = Jacobian(f, x, h=h)
        Delta_x = np.linalg.solve(J, -fx)
        intermediates.append(x.copy())
        x += Delta_x
    else:
        print("Newton-Raphson: no root found after {0} iterations (eps={1}); "
            "best guess is {2} with error {3}".format(Nmax, tol, x, fx))
    return np.array(intermediates)
```

Visualize the intermediate configurations:

```python
x_series = newton_raphson_intermediates(fLW, x0)
```

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis_r(i) for i in np.linspace(0, 1, len(x_series))])
for x in x_series:
    plot_2masses(x, L, W)
```

It's convenient to turn the above plotting code into a function that we can reuse:

```python
def plot_series(x_series, L, W):
    """Plot all N masses/strings solution vectors in x_series (N, 9) array"""
    ax = plt.subplot(111)
    ax.set_prop_cycle("color", [plt.cm.viridis_r(i) for i in np.linspace(0, 1, len(x_series))])
    for x in x_series:
        plot_2masses(x, L, W)
    return ax
```

## Additional work
Try different masses, e.g. M1 = M2 = 10, or M1= 0 , M2 = 10.

### M1 = M2 = 10

```python
W_2 = np.array([10, 10])
def fLW_2(x):
    return f_2masses(x, L, W_2)
```

```python
x_series_2 = newton_raphson_intermediates(fLW_2, x0)
pretty_print(x_series_2[-1])
```

    theta1 = 57.9 	 theta2 = 8.8 	 theta3 = 52.1
    T1     = 13.1 	 T2     = 7.0 	 T3     = 11.3

```python
plot_series(x_series_2, L, W_2)
```

### M1 = 0, M2 = 10

```python
W_3 = np.array([0, 10])
def fLW_3(x):
    return f_2masses(x, L, W_3)
```

```python
x_series_3 = newton_raphson_intermediates(fLW_3, x0)
pretty_print(x_series_3[-1])
```

    theta1 = 30.0 	 theta2 = 30.0 	 theta3 = 61.0
    T1     = 4.8 	 T2     = 4.8 	 T3     = 8.7

```python
plot_series(x_series_3, L, W_3)
```

```python

```


# Vibrações Livres e Amortecidas - 1 Grau de Liberdade
 _Prof. Tito Dias Júnior_

## Exemplo 1: Sistema Amortecido

Uma massa, como mostrada na Figura, está sujeita ao movimento vertical sob a influência de uma mola de constante de elasticidade $k$ e um amortecedor viscoso de constante $c$. Resolva a equação governante do movimento harmônico resultante. Mostre graficamente o movimento da massa em oscilação.

A equação governante é:

$m\ddot{x} + c \dot{x} +kx = 0$

Considere os seguintes valores conhecidos:

$m = 450.0, k = 26519.2, c = 1000.0, x_0 = 0.539657, \dot{x}_0 = 1.0$

### **Solução**

Para a solução do problema temos que utilizar um método para integração de equações diferenciais. O módulo _scipy_ possui a classe _scipy.integrate.ode_ para a solução de equações diferenciais ordinárias de primeira ordem. O nosso problema envolve uma equação diferencial de segunda ordem, portanto para utilizar o _scipy_ precisamos manipular matematicamente o nosso problema para transformá-lo de uma equação segunda ordem para um sistema de equações de primeir ordem. Para isso, basta definirmos mais uma variável
 $ y = \dot{x}  $, que é equivalente a $\dot{x} = y$ representando uma equação diferencial de primeira ordem. Derivando-a obtemos $ \ddot{x} = \dot{y} $ e substitundo na equação governante finalmente obtemos a segunda equação diferencial ordinária de primeira ordem:

 $ m \dot{y} + cy + kx = 0$

 E reescrevendo as equações na forma de um sistema de equações diferenciais ordinárias, podemos obter:

 $ \dot{x} = y $

 $ \dot{y}= \frac{-cy - kx}{m} $

Com as condições iniciais $\dot{x}(0) = x_0 $ e $ \dot{y}(0) = \dot{x}_0$

Para o programa, vamos precisar do módulo  _scipy_ para a integração e o _pylab_ para plotar os resultados.

Inicialmente, vamos definir, como uma função, o sistema de equações que será chamado posteriormente:

```python
def sistema(w, t, p):
    """
    Define o sistema de equações diferenciais para o sistema.

    Argumentos:
        w :  vetor das variáveis de estado
                  w = [x,y]
        t :  tempo
        p :  vetor de parâmetros conhecidos
                  p = [m,k,c]
    """
    x, y = w
    m, k, c = p

    # Cria a função f = (x',y'):
    f = [y,
         (-c * y - k * x) / m]
    return f
```

Agora podemos escrever a parte do programa para efetuar a integração do sistema de EDO:

```python
import numpy as np       # Importa o módulo numpy (métodos numéricos)
import matplotlib.pyplot as plt
from scipy.integrate import odeint

# variáveis dos enunciado
m = 450.                # Massa
k = 26519.2             # Coeficiente de mola
c = 1000.0              # Coeficiente de amortecimento viscoso

x0 = 0.539657          # Condição inicial para x0
y0 = 1.0               # Condição inicial para y0

# Parâmetros e critérios de parada do algoritmo de integração ODE
abserr = 1.0e-8        # Erro absoluto
relerr = 1.0e-6        # Erro relativo
stoptime = 4.0        # Tempo final de integração
numpoints = 550        # Número de pontos de integração

# Cria o vetor com os valores dos tempos de integração para utiliza no plot
t = [stoptime * float(i) / (numpoints - 1) for i in range(numpoints)]

# Agrupando os parâmetros e condições iniciais par ao algoritmo:
p = [m, k, c]
w0 = [x0, y0]

# Chamada ao algoritmo de integração
wsol = odeint(sistema, w0, t, args=(p,),atol=abserr, rtol=relerr)

# Separa a solução nos vetores x e y
x = wsol[:,0]
y = wsol[:,1]

# Plotando o resultado
plt.figure()         # Define uma figura para plotar
                     # Figura é formada por dois subplots

plt.subplot(211)     # Primeira forma de solução
plt.plot(t,x, label="x(t)")
plt.plot(t,y, label="x'(t)")
plt.legend()
plt.xlabel('t [s]')
plt.ylabel('x(t), y(t)')

plt.subplot(212)     # Segunda forma de solução
plt.plot(x,y)
plt.xlabel('x(t) [m]')
plt.ylabel('y(t) [m/s]')

plt.show()

```

### Exemplo 2: Amortecedor Subamortecido

Um amortecedor subamortecido deve ser projetado para uma motocicleta de 200 kg de massa (Fig. 2.30 (a)).
Quando o amortecedor é submetido a uma velocidade vertical inicial devido a um solavanco na estrada, o resultado a curva de deslocamento-tempo deve ser conforme indicado na Fig. 2.30 (b).

Encontre as constantes de rigidez e de amortecimento necessárias para o amortecedor, se o período amortecido de vibração for de 2s e a amplitude $𝑥_1$ deve ser reduzida a um quarto em meio ciclo ($𝑥_{1,5}=𝑥_1/4$).

```python
# Cálculo das propriedades do sistema
from sympy import *
m = 200   # Massa em kg
t = 2.0     # Período em s
x1_x2 = 16
#Decremento logarítmico (delta)
d = log(x1_x2)

# Cálculo do Zeta
x = Symbol('x')
z = solve(2*np.pi*x/(sqrt(1-x**2))-d,x)
z = z[0]

print('zeta = ',z)

# Cálculo da vibração amortecida
wn = 2*np.pi/(t*sqrt(1-z**2))
print('wn = ',wn)

# Constante Crítica de amortecimento
cc = 2*m*wn
print('cc = ', cc)

# Constante de amortecimento
c = z*cc
print('c = ', c)

# Constante de Mola
k = m*wn**2
print('k = ', k)

```

    zeta =  0.403712751943420
    wn =  3.43386319715305
    cc =  1373.54527886122
    c =  554.517744447956
    k =  2358.28329135243

```python
# Resolve novamente o sistema considerando os parâmetros calculados

x0 = 0.539657          # Condição inicial para x0
y0 = 1.0               # Condição inicial para y0

# Parâmetros e critérios de parada do algoritmo de integração ODE
abserr = 1.0e-8        # Erro absoluto
relerr = 1.0e-6        # Erro relativo
stoptime = 4.0        # Tempo final de integração
numpoints = 550        # Número de pontos de integração

# Cria o vetor com os valores dos tempos de integração para utiliza no plot
t = [stoptime * float(i) / (numpoints - 1) for i in range(numpoints)]

# Agrupando os parâmetros e condições iniciais par ao algoritmo:
p = [m, k, c]
w0 = [x0, y0]

# Chamada ao algoritmo de integração
wsol = odeint(sistema, w0, t, args=(p,),atol=abserr, rtol=relerr)

# Separa a solução nos vetores x e y
x = wsol[:,0]
y = wsol[:,1]

# Plotando o resultado
plt.figure()         # Define uma figura para plotar
                     # Figura é formada por dois subplots

plt.subplot(211)     # Primeira forma de solução
plt.plot(t,x, label="x(t)")
plt.plot(t,y, label="x'(t)")
plt.legend()
plt.xlabel('t [s]')
plt.ylabel('x(t), y(t)')

plt.subplot(212)     # Segunda forma de solução
plt.plot(x,y)
plt.xlabel('x(t) [m]')
plt.ylabel('y(t) [m/s]')

plt.show()

```

## Referências

Kiusalaas, J. (2013). **Numerical Methods in Engineering With Python 3**. Cambridge: Cambridge.

S. Rao. (2012) **Vibrações Mecânicas**. Prentice Hall, 4.ª ed.

```python

```

```python

```


### 분포
- 데이터 전체의 특성
- 히스토그램 등 시각화, 분포 함수 등 수식을 사용하여 특성을 살핌

# 대표값 (Representative Value)
(정의)
- 분포를 하나의 숫자로 대표할 수 있는 값
- 몇 개의 수치를 통해 분포의 특성을 파악 가능
- 평균, 중앙값, 최빈값

## 샘플 평균 (Sample Mean)

1) **샘플 평균**
$$ m = \bar{x} = \dfrac{1}{N}\sum_{i=1}^{N} x_i $$

N : 자료의 갯수
i = 1 ... N : 자료의 순서, 인덱스
$x_i$ : 자료의 값

2) 어떤 확률 분포를 따르는 **확률 변수 X의 샘플 평균**(매번 다르기 때문에 달라질 수 있다)
$$ \bar{X} = \dfrac{1}{N}\sum_{i=1}^{N} X_i $$

## 확률 분포의 기대값 (Expectation)
(정의)
- 확률모형, 확률 밀도 함수를 알고 있을 경우, 이론적인 평균
- 확률모형이 존재한다는 것이 확실한 경우 단순히 평균(mean)이라고도 함

>연속형 : f(x) * dx 은 곧 면적(확률)
>$$ \mu_X = \mu = \operatorname{E}[X] = \int_{-\infty}^{\infty} x f(x) dx $$

>이산형 : P(x) 확률
>$$ \mu = \operatorname{E}[X] = \sum xP(x) $$

- 기대값은 확률 P(x) 또는 확률밀도 f(x)를 가중치로 하는 x의 '가중합'
- (연속형) 확률밀도함수 f(x)를 가중치로 x를 적분해서 구한다
- (이산형) x 값에 P(x) 가중치를 곱해서 구한다 (이산형)
- 기대값은 가장 확률 밀도가 높은 x값 근처의 값이 된다. 즉 확률 밀도가 모여있는 근처의 위치를 나타냄

### 기대값의 성질

고정된 값 c에 대해서 ...
$$\operatorname{E}[c] = c$$

$$\begin{align}\operatorname{E}[cX]&= c \operatorname{E}[X]\end{align}$$

$$\begin{align}\operatorname{E}[X + Y] &= \operatorname{E}[X] + \operatorname{E}[Y] \end{align}$$

독립인 두 확률 변수의 기대값

$$ \text{E}\left[ (X-\mu_X)(Y-\mu_Y) \right] = 0 $$

```python
# 기대값 구하기
np.random.seed(0)
x = np.random.normal(size=1000)
```

```python
np.mean(x)
```

    -0.045256707490195384

### 리얼공간+샘플평균 $\leftarrow$ _관계_ $\rightarrow$ 확률모형+기대값

- 샘플 평균의 기대값 : 매번 뽑을 때마다 다를 수 있음
- 확률 모형의 기대값 : 모델, 함수의 특성을 이용하기 때문에 변하지 않음

**하지만 기대값은 일치한다**$\cdot\cdot\cdot \operatorname{E}[\bar{X}] = \operatorname{E}[X] $

(증명)

$$
\begin{eqnarray}
\operatorname{E}[\bar{X}]
&=& \operatorname{E}\left[\dfrac{1}{N}\sum_{1}^{N}X_i \right]  \\\\
&=& \dfrac{1}{N}\sum_{1}^{N}\operatorname{E}[X_i]  \\\\
&=& \dfrac{1}{N}\sum_{1}^{N}\operatorname{E}[X] \;\; (X_1, X_2, ... 모두 X를 복제한 것이므로, X와 같다)\\\\
&=& \dfrac{1}{N} N \operatorname{E}[X]\;\; (N번째를 모두 더한 다는 건, 결국 N번 곱한다는 것)\\ \\
&=& \operatorname{E}[X]  \\\\
\end{eqnarray}
$$

## 중앙값 (Median)

- 전체 자료를 크기 별로 정렬 했을 때, 가장 중앙에 위치한 값
- 샘플 중앙값, 이론적 중앙값(확률분포)

(정의)
- N 홀수 : (n+1)/2번 째 샘플의 값
- N 짝수 : (n/2)번째, (n/2 + 1)번째 값의 평균

- 이론적 중앙값 :
$$ \text{median} = F^{-1}(0.5) $$
$$ 0.5 = F(\text{median}) $$

```python
# 중앙값 구하기
x = np.random.normal(size=1000)
np.median(x)
```

    -0.052497821772340664

## 최빈값 (Mode)
(정의)
- 가장 빈번하게 나올 수 있는 값
- $\text{mode} = \arg \max_x f(x) $

- 연속 확률 분포일 때는, 확률 밀도의 함수값이 가장 큰 확률 변수의 값
- 연속 자료는 구간을 분할하여 구간의 대표값으로 카테고리화 하여 구함

```python
x = np.random.normal(size=1000)
ns, bins = np.histogram(x, bins=np.linspace(-10,10,20))
M = np.argmax(ns)
bins[M], bins[M+1]
```

    (-0.52631578947368496, 0.52631578947368318)

# 기대값, 중앙값, 최빈값의 비교
- 확률 밀도 함수가 대칭인 경우에는 기대값, 중앙값, 최빈값 모두 같음
- 한 쪽으로 찌그러진 (skewed) 형태일 때는 모두 값이 달라짐

        E[X]
        Median
        Mode

        계산
        쉬움 : 다 더하면 그만
        E[X]보다 어려움 : sorting 작업
        제일 어려움 : optimization 작업

        outlier 영향
        가장 많이 받음
        영향 적게 받음
        영향 아예 안 받음


# Introduction to the Harmonic Oscillator

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week week we are going to begin studying molecular dynamics, which uses classical mechanics to study molecular systems. Our "hydrogen atom" in this section will be the 1D harmomic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2}  \omega_0^2 m {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

In classical mechanics, our goal is to determine the equations of motion, $x(t),y(t)$, that describe our system.

In this notebook we will use sympy to solve an second order, ordinary differential equation.

## 1. Solving differential equations with sympy

Soliving differential equations can be tough, and there is not always a set plan on how to proceed. Luckily for us, the harmonic osscillator is the classic second order diffferential eqations.

Consider the following second order differential equation

$$ay(t)''+by(t)'=c$$

where $y(t)'' = {{{d^2}y} \over {dt^2}}$, and $y(t)' = {{{d}y} \over {dt}}$

We can rewrite this as a homogeneous linear differential equations

$$ay(t)''+by(t)'-c=0$$

The goal here is to find $y(t)$, similar to our classical mechanics problems. Lets use sympy to solve this equation

### Second order ordinary differential equation

First we import the sympy library

```python
import sympy as sym
```

Next we initialize pretty printing

```python
sym.init_printing()
```

Next we will set our symbols

```python
t,a,b,c=sym.symbols("t,a,b,c")
```

Now for somehting new. We can define functions using `sym.Function("f")`

```python
y=sym.Function("y")
y(t)
```

Now, If I want to define a first or second derivative, I can use `sym.diff`

```python
sym.diff(y(t),(t,1)),sym.diff(y(t),(t,2))
```

My differential equation can be written as follows

```python
dfeq=a*sym.diff(y(t),(t,2))+b*sym.diff(y(t),(t,1))-c
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

The two constants $C_1$ and $C_2$ can be determined by setting boundry conditions.
First, we can set the condition $y(t=0)=y_0$

The next intial condition we will set is $y'(t=0)=v_0$

To setup the equality we want to solve, we are using `sym.Eq`. This function sets up an equaility between a lhs aand rhs of an equation

```python
# sym.Eq example
alpha,beta=sym.symbols("alpha,beta")
sym.Eq(alpha+2,beta)
```

Back to the actual problem

```python
y0,v0=sym.symbols("y_0,v_0")
ics=[sym.Eq(sol.args[1].subs(t, 0), y0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

We can use this result to first solve for $C_2$ and then solve for $C_1$.
Or we can use sympy to solve this for us.

```python
solved_ics=sym.solve(ics)
solved_ics
```

Substitute the result back into $y(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

We can plot this result too. Assume that $a,b,c=1$ and that the starting conditions are $y_0=0,v_0=0$

We will use two sample problems:

* case 1 : initial position is nonzero and initial velocity is zero
* case 2 : initial position is zero and initialvelocity is nonzero

```python
# Print plots
%matplotlib inline
```

#### Initial velocity set to zero

```python
case1 = sym.simplify(full_sol.subs({y0:0, v0:0, a:1, b:1, c:1}))
case1
```

```python
sym.plot(case1.rhs)
sym.plot(case1.rhs,(t,-2,2))
```

#### Initial velocity set to one

```python
case2 = sym.simplify(full_sol.subs({y0:0, v0:1, a:1, b:1, c:1}))
case2
```

```python
sym.plot(case2.lhs,(t,-2,2))
```

## Calculate the phase space

As we will see in lecture, the state of our classical systems are defined as points in phase space, a hyperspace defined by ${{\bf{r}}^N},{{\bf{p}}^N}$. We will convert our sympy expression into a numerical function so that we can plot the path of $y(t)$ in phase space $y,y'$.

```python
case1
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
yfunc=sym.lambdify(t,case1.rhs,'numpy')
vfunc=sym.lambdify(t,case1.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(yfunc(tlst),vfunc(tlst))
plt.xlabel('$y$')
plt.ylabel("$y'$")
plt.show()
```

### Exercise 1.1

Change the initial starting conditions and see how that changes the plots. Make three different plots with different starting conditions

```python
casea = sym.simplify(full_sol.subs({y0:0, v0:5, a:2, b:2, c:3}))
casea
```

```python
sym.plot(casea.rhs)
sym.plot(casea.rhs,(t,-5,10))
```

```python
caseb = sym.simplify(full_sol.subs({y0:10, v0:5, a:9, b:1, c:2}))
caseb

```

```python

sym.plot(caseb.rhs)
sym.plot(caseb.rhs,(t,-5,10))

```

```python
casec = sym.simplify(full_sol.subs({y0:7, v0:1, a:0, b:1, c:2}))
casec
```

```python
sym.plot(casec.rhs)
sym.plot(casec.rhs,(t,-5,10))
```

## 2. Harmonic oscillator

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= - \omega_0^2 m x $$

$$ a = - \omega_0^2 x $$

$$ x(t)'' = - \omega_0^2 x $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

Your goal is determine and plot the equations of motion of a 1D harmomnic oscillator

### Exercise 2.1

1. Use the methodology above to determine the equations of motion $x(t), v(t)$ for a harmonic ocillator
1. Solve for any constants by using the following initial conditions: $x(0)=x_0, v(0)=v_0$
1. Show expressions for and plot the equations of motion for the following cases:
    1. $x(0)=0, v(0)=0$
    1. $x(0)=0, v(0)>0$
    1. $x(0)>0, v(0)=0$
    1. $x(0)<0, v(0)=0$
1. Plot the phasespace diagram for the harmonic oscillator

```python
# 1
m,omega_0=sym.symbols('m,omega_0')

```

```python
x=sym.Function("x")
x(t)

```

```python
sym.diff(x(t),(t,1)),sym.diff(x(t),(t,2))
```

```python
dfeq1=sym.diff(x(t),(t,2))+omega_0**2*x(t)
dfeq1
```

```python
sol1=sym.dsolve(dfeq1)
sol1

```

```python
x0,v0=sym.symbols("x_0,v_0")
ics1=[sym.Eq(sol1.args[1].subs(t, 0), x0),
     sym.Eq(sol1.args[1].diff(t).subs(t, 0), v0)]
ics1
```

```python
solved_ics1=sym.solve(ics1)
solved_ics1
```

```python
full_sol1 = sol1.subs(solved_ics1[0])
full_sol1
```

```python
# Print plots
%matplotlib inline
```

```python
case_n = sym.simplify(full_sol1.subs({x0:0, v0:0, omega_0:1}))
case_n
```

```python
sym.plot(case_n.rhs)
sym.plot(case_n.rhs,(t,-2,2))
```

```python
case_j = sym.simplify(full_sol1.subs({x0:0, v0:2, omega_0:1}))
case_j
```

```python
sym.plot(case_j.rhs)
sym.plot(case_j.rhs,(t,-2,2))
```

```python
case_k = sym.simplify(full_sol1.subs({x0:2, v0:0, omega_0:1}))
case_k
```

```python
sym.plot(case_k.rhs)
sym.plot(case_k.rhs,(t,-2,2))
```

```python
case_v = sym.simplify(full_sol1.subs({x0:-2, v0:0, omega_0:1}))
case_v
```

```python
sym.plot(case_v.rhs)
sym.plot(case_v.rhs,(t,-2,2))
```

```python
case_n
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,case_n.rhs,'numpy')
vfunc=sym.lambdify(t,case_n.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,case_j.rhs,'numpy')
vfunc=sym.lambdify(t,case_j.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,case_j.rhs,'numpy')
vfunc=sym.lambdify(t,case_j.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-5,5,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,case_v.rhs,'numpy')
vfunc=sym.lambdify(t,case_v.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python

```