# $k$ Nearest Neighbors

## Illustration of Nearest Neighbors

Arguably one of the simplest classification algorithm is $k$ nearest neighbors (KNN). Below figure portrays the algorithm's simplicity. We see a training set of five red and five blue dots, representing some label 1 and 0, respectively. The two axes represent two features, e.g. income and credit card balance. If we now add a new test data point $x_0$ (green dot), KNN will label this test data according to its $k$ closest neighbors. In below figure we set $k=3$. The three closest neighbors are: one blue dot and two red dots, resulting in estimated probabilities of 2/3 for the red class and 1/3 for the blue class. Hence KNN will predict that the new data point will belong to the red class. More so, based on the training set the algorithm is able to draw a decision boundary. (Note: In a classification settings with $j$ classes a *decision boundary* is a hyperplane that partitions the underlying vector space into $j$ sets, one for each class.). This is shown with the jagged line that separates background colors cyan (blue label) and light blue (red label). Given any possible pair of feature values, KNN labels the response along the drawn decision boundary. With $k=1$ the boundary line is very jagged. Increasing the number of $k$ will smoothen the decision boundary. This tells us that small values of $k$ will produce large variance but low bias, meaning that each new added training point might change the decision boundary line significantly but the decision boundary separates the training set (almost) correctly. As $k$ increases, variance decreases but bias increases. This is a manifestation of the Bias-Variance Trade-Off as discussed in the script ([Fortmann-Roe (2012)](http://scott.fortmann-roe.com/docs/BiasVariance.html)) and highlights the importance of selecting an adequate value of $k$ - a topic we will pick up in a future chapter.

## Mathematical Description of KNN

Having introduced KNN illustratively, let us now define this in mathematical terms. Let our data set be $\{(x_1, y_1), (x_2, y_2), \ldots (x_n, y_n)\}$ with $x_i \in \mathbb{R}^p$ and $y_i \in \{0, 1\}\; \forall i \in \{1, 2, \ldots, n\}$. Based on the $k$ neighbors, KNN estimates the conditional probability for class $j$ as the fraction of points in $\mathcal{N}(k, x_0)$ (the set of the $k$ closest neighbors of $x_0$) whose response values equals $j$ (Russell and Norvig (2009)):

$$\begin{equation}
\Pr(Y = j | X = x_0) = \frac{1}{k} \sum_{i \in \mathcal{N}(k, x_0)} \mathbb{I}(y_i = j).
\end{equation}$$

Once the probability for each class $j$ is calculated, the  KNN classifier predicts a class label $\hat{y}_0$ for the new data point $x_0$ by maximizing the conditional probability (Batista and Silva (2009)).

$$\begin{equation}
\hat{y}_0 = \arg \max_{j} \frac{1}{k} \sum_{i \in \mathcal{N}(k,x_0)} \mathbb{I}(y_i = j)
\end{equation}$$

Selecting a new data point $x_0$'s nearest neighbors requires some notion of **distance measure**. Most researchers chose Minkowski's distance, which is often referred to as $L^m$ norm (Guggenbuehler (2015)). The distance between points $x_a$ and $x_b$ in $\mathbb{R}^p$ is then defined as follows (Russell and Norvig (2009)):

$$\begin{equation}
L^m (x_a, x_b) = \left(\sum_{i=1}^p |x_{a, i} - x_{b, i}|^m \right)^{1/m}
\end{equation}$$

Using $m=2$, above equation simplifies to the well known Euclidean distance and $m=1$ yields the Manhattten distance. Python's `sklearn` package, short for scikit-learn, offers [several other options](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html) that we will not discuss.

Beyond the pure distance measure, it is also possible to weight training data points relative to their distance from a certain point. In above figure distance is weighted uniformly. Alternatively one could weight points by the inverse of their distance (closer neighbors of a query point will have a greater influence than neighbors which are further away) or any other user-defined weighting function. For [further details check `sklearn`'s documentation](http://scikit-learn.org/stable/modules/neighbors.html\#classification} for details).

## Application: Predicting Share Price Movement

### Loading Data

The application of KNN is shown using simple stock market data. The idea is to predict a stock's movement based on simple features such as:
* `Lag1, Lag2`: log returns of the two previous trading days
* `SMI`: SMI log return of the previous day

The response is a binary variable: if a stock closed above the previous day closing price it equals 1, and 0 if it fell. We start by loading the necessary packages and stock data from a csv - a procedure we are well acquainted with by now. Thus comments are held short.

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
plt.rcParams['font.size'] = 14
```

```python
# Import daily shares prices and select window
shsPr = pd.read_csv('Data/SMIDataDaily.csv', sep=',',
                 parse_dates=['Date'], dayfirst=True,
                 index_col=['Date'])
shsPr = shsPr['2017-06-30':'2012-06-01']
shsPr.head()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      ...
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK
      SMI

      Date

      2017-06-30
      23.68
      72.90
      50.45
      79.00
      13.86
      447.2
      1918.0
      54.90
      207.3
      83.45
      ...
      244.2
      462.7
      2322.0
      323.6
      87.65
      16.24
      354.1
      279.1
      6160.0
      8906.89

      2017-06-29
      23.63
      72.85
      51.30
      78.60
      14.10
      446.0
      1913.0
      55.00
      204.7
      83.65
      ...
      246.9
      463.9
      2310.0
      323.3
      87.80
      16.36
      353.4
      276.3
      6145.0
      8944.04

      2017-06-28
      24.07
      73.55
      51.65
      80.00
      13.82
      452.5
      1927.0
      56.35
      207.5
      85.40
      ...
      251.2
      464.9
      2342.0
      325.3
      88.55
      16.29
      360.0
      278.0
      6195.0
      9076.73

      2017-06-27
      24.40
      73.70
      51.65
      80.20
      13.74
      453.8
      1931.0
      56.40
      207.6
      84.30
      ...
      252.0
      465.0
      2355.0
      322.7
      88.80
      16.00
      362.0
      281.7
      6280.0
      9072.92

      2017-06-26
      24.63
      73.85
      51.35
      80.45
      13.49
      457.0
      1947.0
      56.25
      208.2
      85.65
      ...
      251.7
      469.3
      2375.0
      323.9
      89.40
      15.71
      367.5
      284.4
      6310.0
      9121.22

5 rows × 21 columns

Having the data in a proper dataframe, we are now in a position to create the features and response values.

```python
# Calculate log-returns and label responses:
# 'direction' equals 1 if stock closed above
# previous day and 0 if it fell.
today = np.log(shsPr / shsPr.shift(-1))
direction = np.where(today >= 0, 1, 0)

# Convert 'direction' to dataframe
direction = pd.DataFrame(direction, index=today.index, columns=today.columns)

# Lag1, 2: t-1 and t-2 returns; excl. smi (in last column)
Lag1 = np.log(shsPr.iloc[:, :-1].shift(-1) / shsPr.iloc[:, :-1].shift(-2))
Lag2 = np.log(shsPr.iloc[:, :-1].shift(-2) / shsPr.iloc[:, :-1].shift(-3))

# Previous day return for SMI index
smi  = np.log(shsPr.iloc[:, -1].shift(-1) / shsPr.iloc[:, -1].shift(-2))
```

### KNN Algorithm Applied

Now comes the difficult part. What we want to achieve is to run the KNN algorithm for every stock and for different hyperparameter $k$ and see how it performs. For this we do the following steps:

1. Create a feature matrix `X` containing `Lag1`, `Lag2` and `SMI` data for share i
2. Create a response vector `y` with binary direction values
3. Split data to training (before 2016-06-30) and test set (after 2016-06-30)
4. Run KNN for different values of $k$ (loop)
5. Write test score for given $k$ to matrix `scr`
6. Once we've run through all $k$'s we proceed with step 1. with share i+1

This means we need two loops. The first corresponds to the share (e.g. ABB, Adecco, etc.), the second runs the KNN algorithm for different values of $k$.

The reason for this approach is that we are interested in finding any pattern/structure that would provide a successful trading strategy. There is obviously no free lunch. Predicting share price direction is by no means an easy task and we must be well aware that we are in for a difficult job here. If it were simple, neither one of us would be sitting here but run his own fund. But nonetheless, let us see how KNN performs and how homogeneous (or heterogeneous) the results are.

Our first step is as usual to prepare the ground by loading the necessary package and defining some auxiliary variables. The KNN function we will be using is available through the `sklearn` (short for Scikit-learn) package. We only load the `neighbor` sublibrary which contains the needed KNN function called `KNeigborsClassifier()`. KNN is applied with the default distance metric: Euclidean distance (Minkowski's distance with $m=2$). If we would prefer another distance metric we would have to specify it ([see documentation](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)).

```python
# Import relevant functions
from sklearn import neighbors

# k = {1, 3, ..., 200}
k = np.arange(1, 200, 2)

# Array to store results in. Dimension is [k x m]
# with m=20 for the 20 companies (excl. SMI)
scr = np.empty(shape=(len(k), len(shsPr.columns)-1))
```

```python
for i in range(len(shsPr.columns)-1):

    # 1) Create matrix with feature values of stock i
    X = pd.concat([Lag1.iloc[:, i], Lag2.iloc[:, i], smi], axis=1)
    X = X[:-3]  # Drop last three rows with NaN (due to lag)

    # 2) Remove last three rows of response dataframe
    #    to have equal no. of rows for features and response
    y = direction.iloc[:, i]
    y = y[:-3]

    # 3) Split data into training set...
    X_train = X['2016-06-30':]
    y_train = y['2016-06-30':]
    # ...and test set.
    X_test = X[:'2016-07-01']
    y_test = y[:'2016-07-01']

    # Convert responses to 1xN array
    y_train = y_train.values.ravel()
    y_test  = y_test.values.ravel()

    for j in range(len(k)):

        # 4) Run KNN
        # Instantiate KNN class
        knn = neighbors.KNeighborsClassifier(n_neighbors=k[j])
        # Fit KNN classifier using training set
        knn = knn.fit(X_train, y_train)

        # 5) Extract test score for k[j]
        scr[j, i] = knn.score(X_test, y_test)
```

```python
# Convert data to pandas dataframe
tickers = shsPr.columns
scr = pd.DataFrame(scr, index=k, columns=tickers[:-1])
```

```python
scr.head()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      1
      0.462451
      0.533597
      0.521739
      0.513834
      0.509881
      0.533597
      0.498024
      0.498024
      0.505929
      0.411067
      0.549407
      0.553360
      0.513834
      0.521739
      0.509881
      0.470356
      0.490119
      0.549407
      0.498024
      0.529644

      3
      0.478261
      0.470356
      0.529644
      0.486166
      0.577075
      0.533597
      0.525692
      0.482213
      0.525692
      0.486166
      0.517787
      0.573123
      0.486166
      0.517787
      0.541502
      0.450593
      0.490119
      0.557312
      0.517787
      0.513834

      5
      0.498024
      0.494071
      0.537549
      0.494071
      0.525692
      0.573123
      0.537549
      0.474308
      0.501976
      0.486166
      0.525692
      0.513834
      0.466403
      0.501976
      0.521739
      0.482213
      0.494071
      0.553360
      0.533597
      0.509881

      7
      0.517787
      0.486166
      0.505929
      0.498024
      0.505929
      0.553360
      0.525692
      0.517787
      0.474308
      0.509881
      0.513834
      0.482213
      0.470356
      0.509881
      0.525692
      0.466403
      0.490119
      0.561265
      0.533597
      0.509881

      9
      0.553360
      0.478261
      0.505929
      0.470356
      0.486166
      0.533597
      0.505929
      0.513834
      0.458498
      0.462451
      0.545455
      0.486166
      0.462451
      0.501976
      0.501976
      0.470356
      0.490119
      0.517787
      0.482213
      0.470356

### Results & Analysis

Now let's see the results in an overview.

```python
scr.describe()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      count
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000

      mean
      0.553202
      0.506798
      0.515968
      0.508577
      0.517431
      0.499763
      0.533320
      0.492016
      0.537510
      0.519960
      0.546877
      0.513043
      0.479051
      0.516996
      0.490514
      0.520791
      0.489289
      0.507352
      0.553557
      0.530040

      std
      0.031183
      0.010579
      0.023187
      0.024694
      0.016666
      0.020266
      0.013498
      0.019546
      0.027272
      0.020842
      0.018939
      0.016882
      0.019902
      0.020634
      0.020150
      0.017837
      0.014962
      0.027736
      0.020282
      0.028188

      min
      0.462451
      0.470356
      0.462451
      0.454545
      0.446640
      0.458498
      0.498024
      0.450593
      0.458498
      0.411067
      0.505929
      0.474308
      0.446640
      0.474308
      0.450593
      0.450593
      0.438735
      0.454545
      0.482213
      0.466403

      25%
      0.529644
      0.500988
      0.501976
      0.493083
      0.509881
      0.486166
      0.525692
      0.477273
      0.529644
      0.509881
      0.533597
      0.501976
      0.462451
      0.501976
      0.474308
      0.516798
      0.482213
      0.486166
      0.545455
      0.512846

      50%
      0.557312
      0.505929
      0.517787
      0.509881
      0.517787
      0.498024
      0.533597
      0.490119
      0.547431
      0.525692
      0.545455
      0.513834
      0.476285
      0.521739
      0.494071
      0.525692
      0.490119
      0.501976
      0.557312
      0.525692

      75%
      0.581028
      0.513834
      0.533597
      0.525692
      0.525692
      0.513834
      0.537549
      0.505929
      0.553360
      0.533597
      0.561265
      0.525692
      0.490119
      0.533597
      0.501976
      0.533597
      0.498024
      0.525692
      0.569170
      0.546443

      max
      0.600791
      0.533597
      0.569170
      0.557312
      0.577075
      0.573123
      0.573123
      0.541502
      0.577075
      0.549407
      0.592885
      0.573123
      0.541502
      0.561265
      0.541502
      0.545455
      0.521739
      0.573123
      0.592885
      0.581028

Following finance theory, returns should be distributed symmetrically. Thus the simplest guess would be to expect a share price to increase on 50% of the days and to decrease on the remaining 50%. Similar to guessing a coin flip, if we would guess an 'up' movement for every day, we obviously would - in the long run - be correct 50% of the times. This would make for a score of 50%.

Looking in that light at the above summary, we see some very interesting results. For 13 out of 20 stocks KNN produces test scores of > 50% for even the 0.25th percentile. Let's plot the ones with the highest test-scores (ABBN, NOVN, SIK and ZURN) to see at what value of $k$ the best test-score is achieved.

```python
nms = ['ABBN', 'NOVN', 'SIK', 'ZURN']

plt.figure(figsize=(12, 8))
for col in nms:
    scr[col].plot(legend=True)
plt.axhline(0.50, c='k', ls='--');
```

For ABB and Novartis the max. score is around $k=100$ while for Sika and Zurich it is between 177 - 185. Furthermore, it seems interesting that for ABB, Novartis and Zurich the test score was barely below 50%. If this is indeed a pattern we would have found a trading strategy, wouldn't we?

To further assess our results we look into KNN's prediction of ABB stock movements. For this we rerun our KNN classifier algorithm for ABB as before.

```python
# 1) Create matrix with feature values of stock i
X = pd.concat([Lag1['ABBN'], Lag2['ABBN'], smi], axis=1)
X = X[:-3]  # Drop last three rows with NaN (due to lag)
```

```python
# 2) Remove last three rows of response dataframe
#    to have equal no. of rows for features and response
y = direction['ABBN']
y = y[:-3]
```

```python
# 3) Split data into training set...
X_train = X['2016-06-30':]
y_train = y['2016-06-30':]
# ...and test set.
X_test = X[:'2016-07-01']
y_test = y[:'2016-07-01']

# Covert responses to 1xN array
y_train = y_train.values.ravel()
y_test  = y_test.values.ravel()
```

For ABB the maximum score is reached where $k=119$. You can check this with the `scr['ABBN'].idxmax()` command, which provides the index of the maximum value of the selected column. In our case, the index is equivalent to the value of $k$. Thus we run KNN with $k=119$.

```python
# 4) Run KNN
# Instantiate KNN class for ABB with k=119
knn = neighbors.KNeighborsClassifier(n_neighbors=119)
# Fit KNN classifier using training set
knn = knn.fit(X_train, y_train)

# 5) Extract test score for ABB
scr_ABB = knn.score(X_test, y_test)
scr_ABB
```

    0.60079051383399207

The score of 60.08% is the very same as above. Nothing new so far. (Recall that the score is the total of correctly predicted outcomes.)

However, the alert reader should by now raise some questions regarding our assumption that 50% of the returns should have been positive. In the long run, this might be true. But our training sample contained only 1'018 records and of these 535 were positive.

```python
# Percentage of 'up' days in training set
y_train.sum() / y_train.size
```

    0.52554027504911593

Therefore, if we would guess 'up' for every day of our test set and **given the distribution of classes in the test set is exactly as in our training set**, then we would predict the correct movement in 52.55% of the cases. So in that light, the predictive power of our KNN algorithm has to be put in perspective to the 52.55%.

In summary, our KNN algorithm has a score of 60.08%. Our best guess (based on the training set) would yield a score of 52.55%. This still displays that overall our KNN algorithm outperforms our best guess. Nonetheless, the margin is smaller than initially thought.

### Confusion Matrix

There are more tools to assess the accuracy of an algorithm. We postpone the discussion of these tools to a later chapter and at this stage restrict ourselves to the discussion of a tool called "confusion matrix".

A confusion matrix is a convenient way of displaying how our classifier performs. In binary classification (with e.g. response $y \in \{0, 1\}$) there are four prediction categories possible (Ting (2011)):

* **True positive**: True response value is 1, predicted value is 1 ("hit")
* **True negative**: True response value is 0, predicted value is 0 ("correct rejection")
* **False positive**: True response value is 0, predicted value is 1 ("False alarm", Type 1 error)
* **False negative**: True response value is 1, predicted value is 0 ("Miss", Type 2 error)

These information help us to understand how our (KNN) algorithm performed. There are different two ways of arranging confusion matrix. James et al. (2013) follow the convention that column labels indicate the true class label and rows the predicted response class. Others have it transposed, such that column labels indicate predicted classes and row labels show true values. We will use the latter approach.

To run this in Python, we first predict the response value for each data entry in our test matrix `X_test`. Then we arrange the data in a suitable manner.

```python
# Predict 'up' (=1) or 'down' (=0) for test set
pred = knn.predict(X_test)
```

```python
# Store data in DataFrame
cfm = pd.DataFrame({'True direction': y_test,
                    'Predicted direction': pred})
cfm.replace(to_replace={0:'Down', 1:'Up'}, inplace=True)

# Arrange data to confusion matrix
print(cfm.groupby(['Predicted direction','True direction']) \
      .size().unstack('Predicted direction'))
```

    Predicted direction  Down   Up
    True direction
    Down                   31   83
    Up                     18  121

As mentioned before, rows represent the true outcome and columns show what class KNN predicted. In 31 cases, the test set's true response was 'down' (in our case represented by 0) and KNN correctly predicted 'down'. 121 times KNN was correct in predicting an 'up' (=1) movement. 18 returns in the test set were positive but KNN predicted a negative return. And in 83 out of 253 cases KNN predicted a 'up' movement whereas in reality the stock price decreased. The KNN score of 60.08% for ABB is the sum of true positive and negative (31 + 121) in relation to the total number of predictions (253 = 31 + 18 + 83 + 121). The error rate is 1 - score or (18 + 83)/253.

Class-specific performance is also helpful to better understand results. The related terms are **sensitivity** and **specifity**. In the above case, sensitivity is the percentage of true 'up' movements that are identified. A good 88.1% (= 121 / (18 + 121)). The specifity is the percentage of 'down' movements that are correctly identified, here a poor 27.2% (= 31 / (31 + 83)). More on this in the next chapter.

Because confusion matrices are important to analyze results, `Scikit-learn` has its own [command to generate it](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html). It is part of the `metrics` sublibrary. The difficulty is that in contrast to above (manually generated) table, the function's output provides no labels. Therefore one must be sure to know which value are where. Here's the code to generate the confusion matrix.

```python
from sklearn.metrics import confusion_matrix

# Confusion matrix
confusion_matrix(y_test, pred)
```

    array([[ 31,  83],
           [ 18, 121]], dtype=int64)

Often it is helpful to visualize results. `Sklearn` unfortunately doesn't have a specific plotting function for the confusion matrix. However, on the package's website a [function code is provided](http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) that does exactly this. Below their code is applied to our KNN results on ABB's stock price movement.

```python
import itertools
plt.style.use('default')

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
```

```python
#plt.style.use('default')
# Compute confusion matrix
cfm_matrix = confusion_matrix(y_test, pred)
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cfm_matrix, classes=['Down', 'Up'],
                      title='Confusion matrix, without normalization');
```

```python
# Plot normalized confusion matrix
plt.figure()
plot_confusion_matrix(cfm_matrix, classes=['Down', 'Up'], normalize=True,
                      title='Normalized confusion matrix');
```

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Batista, Gustavo, and Diego Furtado Silva, 2009, How k-nearest neighbor parameters affect its performance, in *Argentine Symposium on Artificial Intelligence*, 1–12, sn.
* Fortmann-Roe, Scott, 2012, Understanding the Bias-Variance Tradeoff from website, http://scott.fortmann-roe.com/docs/BiasVariance.html, 08/15/17.
* Guggenbuehler, Jan P., 2015, Predicting net new money using machine learning algorithms and newspaper articles, Technical report, University of Zurich, Zurich.
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Russell, Stuart, and Peter Norvig, 2009, *Artificial Intelligence: A Modern Approach* (Prentice Hall Press, Upper Saddle River, NJ).
* Ting, Kai Ming, 2011, Confusion matrix, in Claude Sammut, and Geoffrey I. Webb, eds., *Encyclopedia of Machine Learning* (Springer Science & Business Media, New York, NY).


```python
import numpy as np
from sympy import *
init_printing(use_latex='mathjax')
```

```python
x = symbols('x')
f = x ** 6 / 6 - 3 * x ** 4 - 2 * x ** 3 / 3 + 27 * x ** 2 / 2 + 18 * x - 30
f
```

$$\frac{x^{6}}{6} - 3 x^{4} - \frac{2 x^{3}}{3} + \frac{27 x^{2}}{2} + 18 x - 30$$

```python
df = diff(f, x)
df
```

$$x^{5} - 12 x^{3} - 2 x^{2} + 27 x + 18$$

```python
- f.evalf(subs={x:1}) / df.evalf(subs={x:1})
```

$$0.0625$$

```python

```

Certification 2/Week5.1 - Newton-Raphson method.ipynb


## Fourier methods

The Fourier transform (FT) for a well-behaved functions $f$ is defined as:

$$f(k) = \int e^{-ikx} f(x) ~dx$$

The inverse FT is then

$$f(x) = \frac{1}{2\pi} \int e^{ikx} f(k) ~dk$$

## Discrete Fourier transforms (DFTs)

If the function is periodic in real space, $f(x+L) = f(x)$, then the Fourier space is discrete with spacing $\frac{2\pi}{L}$. Moreover, if the real space is periodic as well as discrete with the spacing $h$, then the Fourier space is discrete as well as bounded.

$$f(x) = \sum e^{ikx} f(k) ~dk~~~~~~      \text{where } k = \Bigg[ -\frac{\pi}{h}, \frac{\pi}{h}\Bigg];~~ \text{with interval} \frac{2\pi}{L} $$

This is very much in line with crystallography with $ [ -\frac{\pi}{h}, \frac{\pi}{h} ]$ being the first Brillouin zone. So we see that there is a concept of the maximum wavenumber $ k_{max}=\frac{\pi}{h} $, we will get back to this later in the notes. Usually in computations we need to find FT of discrete function rather than of a well defined analytic function. Since the real space is discrete and periodic, the Fourier space is also discrete and periodic or bounded. Also, the Fourier space is continuous if the real space is unbounded. If the function is defined at $N$ points in real space and one wants to calculate the function at $N$ points in Fourier space, then **DFT** is defined as

$$f_k = \sum_{n=0}^{N-1} f_n ~ e^{-i\frac{2\pi~n~k}{N}}$$

while the inverse transform of this is

$$f_n = \frac1N \sum_{n=0}^{N-1} f_k ~ e^{~i\frac{2\pi~n~k}{N}}$$

To calculate each $f_n$ one needs $N$ computations and it has to be done $N$ times, i.e, the algorithm is simply $\mathcal{O}(N^2)$. This can be implemented numerically as a matrix multiplication, $f_k = M\cdot f_n$, where $M$ is a $N\times N$ matrix.

## Fast fourier tranforms (FFTs)

The discussion here is based on the Cooley-Tukey algorithm. FFTs improves on DFTs by exploiting their symmetries.

$$ \begin{align}
f_k &= \sum_{n=0}^{N-1} f_n  e^{-i~\frac{2\pi~k~n}{N}} \\
&= \sum_{n=0}^{N/2-1} f_{2n} e^{-i~\frac{2\pi~k~2n}{N}} &+ \sum_{n=0}^{N/2-1} f_{2n + 1} e^{-i~\frac{2\pi~k~(n+1)}{N}}\\
&= \sum_{n=0}^{N/2 - 1} f_{2n} e^{-i~\frac{2\pi k~n}{N/2}} &+ e^{-i\frac{2\pi k}{N}} \sum_{n=0}^{N/2 - 1} f_{2n + 1} e^{-i~\frac{2\pi~k~n~}{N/2}}\\
&=\vdots &\vdots
\end{align}$$

We can use the symmetry property, from the definition, $f_{N+k} = f_k$. Notice that, because of the tree structure, there are $\ln_2 N$ stages of the calculation. By applying the method of splitting the computation in two halves recursively, the complexity of the problem becomes  $\mathcal{O}(N \ln N)$  while the naive algorithm is $\mathcal{O}(N^2)$. This is available in standard python packages like numpy and scipy.

We will now use PyGL to explore its usage to solve physics problems.

```python
import pygl
import numpy as np
import matplotlib.pyplot as plt

dim, Nx, Ny = 2, 128, 128
grid = {"dim":dim, "Nx":Nx, "Ny":Ny}

# now construct the spectral solver
ss = pygl.dms.FourierSpectral(grid)
```

#### We first demonstrate the translation of blob using momentum operator

```python
#  The momentum operator, e^{-ikr}, generates translation!

f = plt.figure(figsize=(20, 5), dpi=80);
L, N = 128, 128
x, y = np.meshgrid(np.linspace(0, L, N), np.linspace(0, L, N))
rr = np.sqrt( ((x-L/2)*(x-L/2)+(y-L/2)*(y-L/2))*.5 )
sig = np.fft.fft2(np.exp(-0.1*rr))

def plotFirst(x, y, sig, n_):
    sp =  f.add_subplot(1, 3, n_ )
    plt.pcolormesh(x, y, sig, cmap=plt.cm.Blues)
    plt.axis('off');

xx = ([0, -L/4, -L/2,])
yy = ([0, -L/3,  L/2])
for i in range(3):
    kdotr = ss.kx*xx[i] + ss.ky*yy[i]
    sig = sig*np.exp(-1j*kdotr)
    plotFirst(x, y, np.real(np.fft.ifftn(sig)), i+1)
```

### Sampling: Aliasing error

We saw that because of the smallest length scale, $h$, in the real space there is a corresponding largest wave-vector, $k_{max}$ in the Fourier space. The error is because of this $k_{max}$ and a signal which has $k>k_{max}$ can not be distinguished on this grid. In the given example, below, we see that if the real space has 10 points that one can not distinguish between $sin(2\pi x/L)$ and $sin(34 \pi x/L)$. In general, $sin(k_1 x)$ and $sin(k_2 x)$ can not be distinguished if $k_1 -k_2$ is a multiple of $\frac{2\pi}{h}$. This is a manifestation of the gem called the sampling theorem which is defined, as on wikipedia:

If a function x(t) contains no frequencies higher than B hertz, it is completely determined by giving its ordinates at a series of points spaced 1/(2B) seconds apart.

```python
L, N = 1, 16
x = np.arange(0, L, L/512)
xx = np.arange(0, L, L/N)

def ff(k, x):
    return np.sin(k*x)

f = plt.figure(figsize=(17, 6), dpi=80);

plt.plot(x, ff(x, 2*np.pi), color="#A60628", linewidth=2);
plt.plot(x, ff(x, 34*np.pi), color="#348ABD", linewidth=2);
plt.plot(xx, ff(xx, 2*np.pi), 'o', color="#020e3e", markersize=8)
plt.xlabel('x', fontsize=15); plt.ylabel('y(x)', fontsize=15);
plt.title('Aliasing in sampling of $sin(2\pi x/L)$ and $sin(34 \pi x/L)$', fontsize=24);
plt.axis('off');
```

To avoid this error, we truncate higher mode in PyGL using `ss.dealias'

### Differentiation

We now show the usage of PyGL to compute differentiation matrices. We first compute the second derivative of $\cos x$.

```python
import pygl
import numpy as np
import matplotlib.pyplot as plt

dim, Nx, Ny = 1, 32, 32
grid = {"dim":dim, "Nx":Nx, "Ny":Ny}

ss = pygl.dms.FourierSpectral(grid)
```

```python
def f1(kk, x):
    return np.cos(kk*x)

f = plt.figure(figsize=(10, 5), dpi=80);

L, N = Nx, Nx;   x=np.arange(0, N);  fac=2*np.pi/L
k = ss.kx

fk = np.fft.fft(f1(fac, x))
f1_kk = -k*k*fk
f1_xx = np.real(np.fft.ifft(f1_kk))

plt.plot(x, -f1(fac, x)*fac*fac, color="#348ABD", label = 'analytical', linewidth=2)
plt.plot(x, f1_xx, 'o', color="#A60628", label = 'numerical', markersize=6)
plt.legend(loc = 'best'); plt.xlabel('x', fontsize=15);
```


## 3.2 Decentralized Gradient Descent (Adapt-then-combine)

### 3.2.1 Problem and distributed gradient descent

Consider $n$ computing nodes collaborate to solve the problem:

$$\min_{x\in \mathbb{R}^d} \quad \frac{1}{n}\sum_{i=1}^n f_i(x)$$

where $f_i(x)$ is a local and private function held by node $i$. Each node $i$ can access its own variable $x$ or gradient $\nabla f_i(x)$, but it has to communicate to access information from other nodes.

If each $f_i(x)$ is assumed to be smooth, the leading algorithm to solve the above problem is gradient descent:

$$\begin{align}
x^{(k+1)} = \frac{1}{n}\sum_{i=1}^n \Big(x^{(k)} - \alpha \nabla f_i(x^{(k)}) \Big) \quad \mbox{(distributed gradient descent)}
\end{align}$$

### 3.2.2 Decentralized gradient descent

In this section we disscuss a new gradient descent that can solve the optimization problem in a decentralized manner. To this end, we first organize all computing nodes with a connected network topology. Next we generate a doubly stochastic combination matrix $W$. For more details on the network topology and combination matrix, please check Sections [2.2](../Sec-2.2-Network-topology.ipynb)-[2.4](../Sec-2.4-Combination-matrix-over-directed-network.ipynb).

With combination matrix $W$, the decentralized gradient descent has recursions as follows:

\begin{align}
y_i^{(k)} &= x_i^{(k)} - \alpha \nabla f_i(x_i^{(k)}) \hspace{5.5cm} \mbox{(local update)} \\
x_i^{(k+1)} &= \sum_{j=1}^n w_{ij} y_j^{(k)} = w_{ii} y_i^{(k)} + \sum_{j\in\mathcal{N}(i)}w_{ij} y_j^{(k)} \hspace{2.1cm} \mbox{(combination)}
\end{align}

where $\mathcal{N}(i)$ is the set of incoming neighbors of node $i$. It is observed that the combination step incurs partial averaging within neighborhood, which is different from the global averaging used in distributed gradient descent. The partical averaging typically triggers $O(1)$ bandwidth cost and $O(1)$ latency, which is independent of the number of all computing nodes $n$. On a sparse network such as the ring, or the one-peer exponential-two graph \[Refs\], decentralized gradient descent can save significant communications than distributed gradient descdent.

Decentralized gradient descent has two major variants. One variant is as the above recursions. Since the recursion has the "adapt-then-combine" order, we will refer to it as the ATC-DGD \[Refs\]. The other variant will mix the adaptation (i.e., the gradient descent) and combination in the same update, which will be referred to as "adapt-with-combination (AWC)" decentralized gradient descent. There are subtle differences between these two variants. We will leave the discussion of AWC-DGD algorithm in the next section.

### 3.2.3 Convergence properties

We give a brief descrption on the convergence property of the ATC-DGD algorithm. For $L$-smooth and $\mu$-strongly convex problems, if the step-size $\alpha$ is sufficiently small, the ATC-DGD algorithm will converge as follows.

\begin{align}
\frac{1}{n}\sum_{i=1}^n \|x_i^{(k)} - x^\star \|^2 = O\Big( (1-\alpha \mu)^{k} + \frac{\alpha^2 \rho^2 b^2}{(1-\rho)^2}\Big) \hspace{1cm} \mbox{(DGD-Convergence)}
\end{align}

where $x^\star$ is the glboal solution to the optimization problem, $\rho = \max\{|\lambda_2(W), \lambda_n(W)|\}$ and $b^2 = \frac{1}{n}\sum_{i=1}^n \|\nabla f_i(x^\star)\|^2$ denotes the data heterogeneity between nodes. Quantity $1-\rho$ measures the connectivity of the network topology. It is observed that ATC-DGD cannot converge exactly to the solution $x^\star$, but to a neighborhood around it. The limiting error is on the order of $O(\frac{\alpha^2 b^2}{(1-\rho)^2})$. When step-size $\alpha$ is small, or the data heterogeneity $b^2$ is small, or the network is well-connected, i.e., $\rho \to 0$, the limiting error can be negligible.

### 3.2.4 An example: least-square problem

In this section, we will show a demo on how to solve a least-square problem with ATC-DGD using BlueFog. Suppose $n$ computing nodes collaborate to solve the following problem:

$$\min_x \quad \frac{1}{n}\sum_{i=1}^n \|A_i x - b_i\|^2$$

where $\{A_i, b_i\}$ are local data held in node $i$.

#### 3.2.4.1 Set up BlueFog

In the following code, you should be able to see the id of your CPUs. We use 8 CPUs to conduct the following experiment.

```
import ipyparallel as ipp

rc = ipp.Client(profile="bluefog")
dview = rc[:]  # A DirectView of all engines
dview.block = True
rc.ids
```

    [0, 1, 2, 3, 4, 5, 6, 7]

```
%%px
import numpy as np
import bluefog.torch as bf
import torch
from bluefog.common import topology_util
import networkx as nx
import matplotlib.pyplot as plt

%matplotlib inline

bf.init()
print(f"Hello, I am {bf.rank()} among {bf.size()} processes")
```

    [stdout:0] Hello, I am 2 among 8 processes
    [stdout:1] Hello, I am 4 among 8 processes
    [stdout:2] Hello, I am 7 among 8 processes
    [stdout:3] Hello, I am 6 among 8 processes
    [stdout:4] Hello, I am 3 among 8 processes
    [stdout:5] Hello, I am 0 among 8 processes
    [stdout:6] Hello, I am 5 among 8 processes
    [stdout:7] Hello, I am 1 among 8 processes

#### 3.2.4.2 Generate local data $A_i$ and $b_i$

```
%%px

def generate_data(m, n):

    A = torch.randn(m, n).to(torch.double)
    x_o = torch.randn(n, 1).to(torch.double)
    ns = 0.1 * torch.randn(m, 1).to(torch.double)
    b = A.mm(x_o) + ns

    return A, b
```

#### 3.2.4.3 Distributed gradient descent method

```
%%px

def distributed_grad_descent(A, b, maxite=5000, alpha=1e-1):

    x_opt = torch.zeros(n, 1, dtype=torch.double)

    for _ in range(maxite):
        # calculate local gradient
        grad_local = A.t().mm(A.mm(x_opt) - b)

        # global gradient
        grad = bf.allreduce(grad_local, name="gradient")

        # distributed gradient descent
        x_opt = x_opt - alpha * grad

    grad_local = A.t().mm(A.mm(x_opt) - b)
    grad = bf.allreduce(grad_local, name="gradient")  # global gradient

    # evaluate the convergence of distributed gradient descent
    # the norm of global gradient is expected to 0 (optimality condition)
    global_grad_norm = torch.norm(grad, p=2)
    if bf.rank() == 0:
        print(
            "[Distributed Grad Descent] Rank {}: global gradient norm: {}".format(
                bf.rank(), global_grad_norm
            )
        )

    return x_opt
```

In the following code we run distributed gradient descent to achieve the global solution $x^\star$ to the optimization problem. To validate whether $x^\star$ is optimal, it is enough to examine $\frac{1}{n}\sum_{i=1}^n \nabla f_i(x^\star) = 0$.

```
%%px

m, n = 20, 5
A, b = generate_data(m, n)
x_opt = distributed_grad_descent(A, b, maxite=200, alpha=1e-2)
```

    [stdout:5] [Distributed Grad Descent] Rank 0: global gradient norm: 7.236065407667669e-15

#### 3.2.4.3 Decentralized gradient descent method

In this section, we depict the convergence curve of the decentralied gradient descent (the ATC version). We will utilize the $x^\star$ achieved by distributed gradient descent as the optimal solution. First, we define one step of the ATC-DGD method.

```
%%px

def ATC_DGD_one_step(x, x_opt, A, b, alpha=1e-2):

    # one-step ATC-DGD.
    # The combination weights have been determined by the associated combination matrix.

    grad_local = A.t().mm(A.mm(x) - b)  # compute local grad
    y = x - alpha * grad_local  # adapte
    x_new = bf.neighbor_allreduce(y)  # combination

    # the relative error: |x^k-x_gloval_average|/|x_gloval_average|
    rel_error = torch.norm(x_new - x_opt, p=2) / torch.norm(x_opt, p=2)

    return x_new, rel_error
```

Next we run ATC-DGD algorithm.

```
%%px

# Set topology as exponential-two topology.
G = topology_util.ExponentialTwoGraph(bf.size())
bf.set_topology(G)

maxite = 200
x = torch.zeros(n, 1, dtype=torch.double)  # Initialize x
rel_error = torch.zeros((maxite, 1))
for ite in range(maxite):

    if bf.rank() == 0:
        if ite % 10 == 0:
            print("Progress {}/{}".format(ite, maxite))

    x, rel_error[ite] = ATC_DGD_one_step(
        x, x_opt, A, b, alpha=3e-3
    )  # you can adjust alpha to different values
```

    [stdout:5]
    Progress 0/200
    Progress 10/200
    Progress 20/200
    Progress 30/200
    Progress 40/200
    Progress 50/200
    Progress 60/200
    Progress 70/200
    Progress 80/200
    Progress 90/200
    Progress 100/200
    Progress 110/200
    Progress 120/200
    Progress 130/200
    Progress 140/200
    Progress 150/200
    Progress 160/200
    Progress 170/200
    Progress 180/200
    Progress 190/200

In the following, we adjust step-size to differnt values to examine its influence on the convergence rate and limiting bias.

```
# collect relative error from node 0 for step-size 1e-2
rel_error_exp2_alpha1em2 = dview.pull("rel_error", block=True, targets=0)
```

```
# collect relative error from node 0 for step-size 5e-3
rel_error_exp2_alpha5em3 = dview.pull("rel_error", block=True, targets=0)
```

```
# collect relative error from node 0 for step-size 3e-3
rel_error_exp2_alpha3em3 = dview.pull("rel_error", block=True, targets=0)
```

```
import matplotlib.pyplot as plt

%matplotlib inline

plt.semilogy(rel_error_exp2_alpha1em2)
plt.semilogy(rel_error_exp2_alpha5em3)
plt.semilogy(rel_error_exp2_alpha3em3)

plt.legend(["1e-2", "5e-3", "3e-3"], fontsize=16)

plt.xlabel("Iteration", fontsize=16)
plt.ylabel("Relative error", fontsize=16)
```

It is observed from the above figures that smaller alpha can lead to more accuate solution, but the convergence rate will get slower. This observation is consistent with Eq. (DGD-Convergence).


# Automatic Differentiation
### [Dr David Race](dr.david.race@gmail.com)

This notebook is specifically designed to provide a quick demonstration of "autograd" capabilities.  This is designed to be the first in a series on convex function minimization within Machine Learning (ML) environments, so it starts with the basics of differentiation.  This notebook uses the "tensor" concepts to demonstrate some of the nice methods available with both the HIPS and PyTorch packages.  These are both foundations for Deep Learning (DL) environments, but are equally adapt with some standard mathematics.

There are two main sections:
1.  HIPS
2.  PyTorch

The examples grow in sophistication as the notebook progresses, so be sure to follow the instructions carefully.

NOTE:  This is designed to run in Colaboratory, but is likely to run in most other Jupyter envrionments also.  In particular this does not connect to a GPU, so it will run on minimal hardware.

##1.   Differentiation with HIPS Autograd

sympy is a great package for symbolic differentiation, but sympy will just be used for simple comarison of results so you can better understand the results.  autograd is much more appropriate for larger problems so is more important for Differential Equations and Linear Algebra applications.  The reference for autograd is found at [HIPS - Autograd](https://github.com/HIPS/autograd).

### 1.1 - Set Up Environment

This section installs Autograd into the Colaboratory environment and imports the standard python numeric packages.

```
!pip install autograd
#python imports
import os, sys
#numpy
import numpy as np
from numpy import linspace
import scipy as sp
#sympy
import sympy as smp
from sympy import *
from sympy import Function, Symbol
from sympy import Derivative
#
import autograd
```

Thse section sets up the graphics

```
#The general plot capabilities
import matplotlib
import matplotlib.pyplot as plt
#Since these are images, turn off the grids
matplotlib.rc('axes',**{'grid':False})
#  sympy plotting
from sympy.plotting import plot
#seaborn
import seaborn as sns
```

### 1.2 -Example 1 - $f(x) = x^2 + 4$

We start with the basic problem and progress along the knowledge path.

#### 1.2.1  Sympy Implementation

This section uses the symbolic package to derive the known quantities for our function.  This could be done by hand, but is intended to show how the results mesh together.

```
#Define the function
x = Symbol('x')
f = Function('f')(x)
f = x**2 + 4
#Show the function definition
print("The function f")
smp.pprint(f)
#take the derivative
f_prime = f.diff(x)
print('The derivative of f')
smp.pprint(f_prime)
#  Plot the function and derivative
p1 = plot(f,xlim=(-3.0,3.0),ylim=(0.0,12.0))
#  Compute the values of f between -3 and 3
f_n = lambdify(x, f, "numpy")
f_prime_n = lambdify(x,f_prime,"numpy")
x_vals = linspace(-3.0, 3.0)
y_vals = f_n(x_vals)
y_prime_vals = f_prime_n(x_vals)

sns.set_style('dark')
fig, ax = plt.subplots()
plt.ylim(0.0,12.0)
plt.yticks(np.arange(1,13))
ax.axvline(0.0, color='k')
ax.axhline(0.0, color='k')
fn, = ax.plot(x_vals,y_vals, label='$f$')
fprimen, = ax.plot(x_vals,y_prime_vals, label='$\\frac{\\partial f}{\\partial x}$')
plt.legend(handles=[fn, fprimen])
plt.show()
```

This is a standard an easily understood problem, so not much effort is put into the plot.  The main point is generation of the x and y values.

#### 1.2.2  Autograd Implementation

Autograd understands the same type operations, but rather than a focus on symbolic computation the focus is on numeric computation using a similar underlying framework.  The main difference is that the gradient (yes, these are the partial derivatives) are taken relative to a scalar "loss" value.  Therefore when working with tensors of numbers, we need to define the function that will be differentiated in terms of a loss value (NOTE:  The use of the loss value stems from Machine Learning.)  The following code generates the same example data.

It may not be obvious, but this provides a way to automatically compute the derivative of a function at many point concurrently.  Here is the process:

1.  Define your function, $f$, so it inputs a tensor (vector, matrix, etc).
2.  Define your loss function, $loss_f$ to be $np.sum(f(x)) $
3.  Define the gradient of $f$ to be $grad(loss_f)$
4.  Then for clarity, define a function g, that outputs the $f(x)$ and $f^\prime(x)$

These steps are shown in the next example:

```
import autograd.numpy as np #This is so the gradient understands the numpy operations
from autograd import grad
#Follow the steps

def f(x):
  y = x*x + 4.0
  return y
def loss_f(x):
  loss = np.sum(f(x))
  return loss
f_p = grad(loss_f)
def g(x):
  return f(x), f_p(x)
#Compute points
y, y_p = g(x_vals)
#plot
sns.set_style('dark')
fig, ax = plt.subplots()
plt.ylim(0.0,12.0)
plt.yticks(np.arange(1,13))
ax.axvline(0.0, color='k')
ax.axhline(0.0, color='k')
fn, = ax.plot(x_vals,y, label='$f$')
fprimen, = ax.plot(x_vals,y_p, label='$\\frac{\\partial f}{\\partial x}$')
plt.legend(handles=[fn,fprimen])
plt.show()
#
#  check the results
#
max_der_diff = np.max(np.abs(y_p - y_prime_vals))
print("The max difference in the derivative computation: {:.8f}".format(max_der_diff))
```

As you can see, ther results are exactly the same as expected and performs correctly.  Lets, see why:

Recall from above, we defined the loss function as $np.sum(f(x))$, thus $loss_f = \sum_{i=0}^{N-1} f(x_i)$; therefore,

$\frac{\partial f(x_i)}{\partial x_i} = f^{\prime}(x_i)$

since the value of $f(x_i)$ only appears once in the summation.

Consequently, using the $np.sum$ function provides a quick way to compute the derivatives of $f$ for the input $x$ values.

Obviously we now want to consider a second derivative.  This is computed using the following code cell.

```
import autograd.numpy as np #This is so the gradient understands the numpy operations
from autograd import grad
#Follow the steps

def f(x):
  y = x*x + 4.0
  return y
def loss_f(x):
  loss = np.sum(f(x))
  return loss
f_p = grad(loss_f)
def loss_fp(x):
  loss = np.sum(f_p(x))
  return loss
f_pp = grad(loss_fp)
def g(x):
  return f(x), f_p(x)
def h(x):
  return f(x), f_p(x), f_pp(x)

#Compute points
y, y_p,y_pp = h(x_vals)
pprint(y_pp)
```

### Example 1.3 - $f(x,y) = x^2 + y^2 + 4$

In this example, we will compute the gradient of f, namely  of $grad(f) = \nabla(f) = \begin{bmatrix} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{bmatrix} = \begin{bmatrix}2x \\ 2y\end{bmatrix}$ for a set of random points with $x,y \in [0,1)$.

This works much the same as the previous example by leveraging the grad function and providing the appropriate loss function that can operate on multiple inputs concurrently.

```
import autograd.numpy as np #This is so the gradient understands the numpy operations
from autograd import grad
#Follow the steps

def f(xy):
  z = xy[0]*xy[0] + xy[1]*xy[1] + 4.0
  return z
def loss_f(z):
  loss = np.sum(f(z))
  return loss
f_p = grad(loss_f)
def g(xy):
  return np.array(f(xy)),np.array(f_p(xy))
#Define the x and y
x_vals = np.random.uniform(-1,1,50)
y_vals = np.random.uniform(-1,1,50)
xy = [x_vals,y_vals]
#
#Compute points
z, z_p = g(xy)
#Compute the formula values
z_p_compute = np.array([2*x_vals,2*y_vals])

max_err = np.max(np.abs(z_p - z_p_compute))
pprint("Max Error: {:8f}".format(max_err))

```

Once again, the use of grad allows for easy computation of exactly the values we need for computation.

### 1.4  Conclusion

With autograd, the computation of gradients is automatic.  Even though autograd has its primary use in Machine Learning, this tool can be very powerful for mathematics operations since it supports both GPUs and targets numpy compatibility.

## 2. Differentiation with PyTorch.autograd

The Autograd is a very nice package, but at this point PyTorch probably has a larger user community and it is also very pythonic.  PyTorch has GPU support, but it doesn't overload the numpy packages.  Given its sponsors (including Facebook), the implementation for Machine Learning is very robust and it has several pre-trained models that are ready for use in solving problems.  This series of studies on using gradients generally focuses on PyTorch; however, most of the work can be done within Autograd.

The documentation for Pytorch can be found at [Docs](https://pytorch.org/docs/stable/index.html).

###2.1  Set up Environment

```
#
!pip3 install -U torch
#
import torch as torch
import torch.tensor as T
import torch.autograd as t_autograd  #normally I use autograd, but I want to distinguish between autograd and torch
#
#  Output Environment Information
#
has_cuda = torch.cuda.is_available()
current_device = torch.cuda.current_device() if has_cuda else -1
gpu_count = torch.cuda.device_count() if has_cuda else -1
gpu_name = torch.cuda.get_device_name(current_device) if has_cuda else "NA"
print("Current device {}".format(current_device))
print("Number of devices: {:d}".format(gpu_count))
print("Current GPU Number: {:d}".format(current_device))
print("GPU Name: {:s}".format(gpu_name))
#Set the accelerator variable
accelerator = 'cuda' if has_cuda else 'cpu'
print("Accelerator: {:s}".format(accelerator))
```

### 2.2 -Example 1 - $f(x) = x^2 + 4$

This section solves the same problem as the previous section, but is written to accomodate a GPU so it includes some of the details to use a GPU.

```
#define setup
#
N = 50
device = torch.device(accelerator)
#
#Define the function and loss
#
def f(x):
  y = x * x + 2.0
  return y
def loss_f(x):
  z = f(x).sum()
  return z
def f_p(x):
  z = loss_f(x)
  z.backward()
  return x.grad
x_val = np.linspace(-3., 3.0, N)
x = T(x_val, requires_grad = True).to(device)
#Get the data
x_vals = x.data.numpy()
y = f(x).data.numpy()
y_p = f_p(x).data.numpy()
#Graph
sns.set_style('dark')
fig, ax = plt.subplots()
plt.ylim(0.0,12.0)
plt.yticks(np.arange(1,13))
ax.axvline(0.0, color='k')
ax.axhline(0.0, color='k')
fn, = ax.plot(x_vals,y, label='$f$')
fprimen, = ax.plot(x_vals,y_p, label='$\\frac{\\partial f}{\\partial x}$')
plt.legend(handles=[fn,fprimen])
plt.show()
#
#  check the results
#
max_der_diff = np.max(np.abs(y_p - y_prime_vals))
print("The max difference in the derivative computation: {:.8f}".format(max_der_diff))

```

As you can see, the computations are similar to using Autograd, but instead of using a grad function this uses a backward function (backward is the word in Machine Learning that computes the derivative relative to the loss) and then grad is a property of the variable that us used for the computation that required the gradient.

### Example 2.3 - $f(x,y) = x^2 + y^2 + 4$

```

#Follow the steps

def f(xy):
  z = xy[0]*xy[0] + xy[1]*xy[1] + 4.0
  return z
def loss_f(z):
  loss = f(z).sum()
  return loss
def f_p(xy):
  z = loss_f(xy)
  z.backward()
  return xy.grad
def g(xy):
  return f(xy).data.numpy(),f_p(xy).data.numpy()
#Define the x and y
x_vals = np.random.uniform(-1,1,50)
y_vals = np.random.uniform(-1,1,50)
xy = T([x_vals,y_vals], requires_grad = True).to(device)
#
#Compute points
z, z_p = g(xy)
#Compute the formula values
z_p_compute = np.array([2*x_vals,2*y_vals])

max_err = np.max(np.abs(z_p - z_p_compute))
pprint("Max Error: {:8f}".format(max_err))
```

### 2.4 Conclusion

PyTorch provide both a numpy compatible interface for the numpy functions, so starting with a minimal set of code using numpy, it is easy to scale up to use GPUs and PyTorch.autograd.  The interworking of PyTorch.autograd are exactly as we expect.

## Overall Conclusion

Both Autograd and PyTorch are environments for Machine Learning, but they provide many benefits to numerical computations and modeling.  These free tools coupled with Colaboratory greatly expands the types of mathematic modeling and computations that are available to developers.  Autograd appears to have a smaller footprint, but PyTorch appears to have a larger following (especially when considering ML).  Using both isn't a bad option depending on the resources available for processing.


## Taylor expansions

We first import some prerequisite modules, most importantly Sympy and Matplotlib.

```python
def uniq(seq):
    # order preserving
    seen = {}
    result = []
    for item in seq:
        if item in seen:
            continue
        seen[item] = 1
        result.append(item)
    return result
```

```python
import sympy as sm
from sympy.abc import x
from matplotlib.colors import cnames
```

```python
import matplotlib
from matplotlib.colors import rgb2hex

from numpy import arange
```

```python
from sympy import Symbol,plot
import matplotlib.pyplot as plt

def move_sympyplot_to_axes(p, ax):
    backend = p.backend(p)
    backend.ax = ax
    backend.process_series()
    backend.ax.spines['right'].set_color('none')
    backend.ax.spines['bottom'].set_position('zero')
    backend.ax.spines['top'].set_color('none')
    plt.close(backend.fig)
```

The following routine will plot the function `f` together with its Taylor polynomials up to the degree `degree`, in the square box with sides `[-A,A]`. We will use it several times for different functions `f`.

```python
def taylorplot(f, degree, A):
    L = [f.series(n=i).removeO() for i in range (1, degree+2)] # +1 for the offset of `range`, another +1 for the offset of `.series`
    L = uniq(L)
    L.insert(0,f)
    # Pick colors from the Blues colorscheme
    a = arange(0,1,1/len(L))
    c = matplotlib.cm.Blues(a)[:,:3]

    # Plot the original function in red, then add the graphs of Taylor polynomials in different shades of blue
    p1 = sm.plot(L[0], xlim=[-A,A], ylim=[-A,A], line_color = 'r',  show=False)
    for i in range(1,len(L)):
        p1.extend(sm.plot(L[i], xlim=[-A,A], ylim=[-A,A], line_color = rgb2hex(c[i,:]), show=False))
    p1.legend = False
    # p1.show()

    # Plot the original function alone
    p2 = sm.plot(L[0], xlim=[-A,A], ylim=[-A,A], line_color = 'r', show=False)
    p2.legend = False
    p2.grid = True
    # p2.show()

    # Generate the legend by LaTeXing the expressions for the functions involved.
    l =[]
    for i in range(len(L)):
        l.append(sm.latex(L[i], mode='inline'))

    # %matplotlib gtk3 # uncomment to use the external plot window
    # Combine the two resulting plots, add legend
    fig, (ax1,ax2) = plt.subplots(ncols=2,figsize=(20,10))
    move_sympyplot_to_axes(p1, ax1)
    move_sympyplot_to_axes(p2, ax2)
    fig.legend(l,loc='lower center', bbox_to_anchor=(0.5, 0.0),
              ncol=3, fancybox=True, shadow=True)
    plt.show()
```

----

Let us first plot the partial sums of the Maclaurin series of the Bessel function $J_0$, up to the degree 14. We will restrict the graphs to the box `[-10,10]` by `[-10,10]`:

```python
B = sm.besselj(0,x)
taylorplot(B, 14, 10)
```

In these graphs we show the Taylor polynomials (centered at $a=0$ here), using the darker shade to denote to the higher degree polynomial $T_n(x)$. The plot on the right shows $J_0(x)$ itself.

----

Next, let us do the same with $\sin(x)$. We plot its Taylor polynomials at 0 up to degree 13 in the box `[-10,10]` by `[-10,10]`.

```python
S = sm.sin(x)
taylorplot(S, 13,10)
```

---

Finally, let us look at two examples in which the Maclaurin series of a function does not converge for all `x`. The first example is $ 1/(1-x) $; we will plot its Taylor polynomials up to degree 10 in the square box `[-6,6]` by `[-6,6]`.

```python
f1 = 1/(1-x)
taylorplot(f1,10,6)
```

Observe how the polynomials $ T_n $ diverge away from the graph of the function `f1` when $x=-1$, even though the function does not have a singularity there. This is of course because the radius of convergence of the geometric series $$ \sum_{n=0}^\infty x^n $$ is $1$. Since the interval of convergence of a series must be symmetric, divergence of the series at $ x = 1 $, where the function `f1` has a singularity, means that the series must also diverge at $x = -1$.

---

The following example is even more subtle. The function we look at here, `f2`$= 1/(1+x^2) $, does not exhibit any singularities at all! On the other hand, its Maclaurin series  $$ \sum_{n=0}^\infty\, (-1)^n x^{2n} $$ diverges as soon as $|x|\geq 1 $. Let us plot the Taylor polynomials at $ a = 0 $ up to the degree  $ n = 20$.

```python
f2 = 1/(1+x**2)
taylorplot(f2,20,6)
```

**Outside of Calc2:** It turns out, to understand the reason for such behavior of this series, one has to extend the function `f2`$=1/(1+x^2)$ to *complex numbers*, where for $x = i$, our function `f2` has a singularity (the complex number $i$ is such that $i^2+1 = 0$, turning the denominator of `f2` into $0$). In the complex plane, series converge on *discs* instead of intervals, so because the function `f2` has a singularity at the point $0+i\,1$, its series centered at $a=0$ will have the radius of convergence equal to $1$. We show this relation in the following graph. In it, a point $(x,y)$ in the plane stands for the complex number $z = x+ iy$.

```python
    fig, ax = plt.subplots(figsize=(10,10))
    circle =  matplotlib.patches.Circle ((0, 0), 1, edgecolor='none',alpha=0.5)
    p = matplotlib.collections.PatchCollection([circle], alpha=0.4)
    ax.add_collection(p);
    ax.scatter([0.0],[1.0],marker='o',c='r',s=40)
    ax.annotate('Singularity of f2 at $0+i$', xy=(0.1, 1.05), xytext=(1, 1.5),
            arrowprops=dict(arrowstyle="->")
            )
    ax.annotate('For complex $z= x + i\,y$\nthat lie inside this disc,\nthe series converges.', xy=(0.5, -0.5), xytext=(0.5, -1.5),
        arrowprops=dict(arrowstyle="->")
        )
    ax.annotate('The series will diverge at the\n points of the disc boundary', xy=(-1.02, -0.13), xytext=(-2, -1),
            arrowprops=dict(arrowstyle="->")
            )
    # This adjusts the position of the axes
    ax.spines['left'].set_position('center')
    ax.spines['bottom'].set_position('center')
    ax.spines['right'].set_color('none')
    ax.spines['top'].set_color('none')
    ax.xaxis.set_ticks_position('bottom')
    ax.yaxis.set_ticks_position('left')

    plt.ylim(-2,2)
    plt.xlim(-2,2)
    plt.show()
```


# Solutions to 1st-order ODEs

## 1. Solution by direct integration

When equations are of this form, we can directly integrate:

\begin{align}
\frac{dy}{dx} &= y^{\prime} = f(x) \\
\int dy &= \int f(x) dx \\
y(x) &= \int f(x) dx + C
\end{align}

For example:
\begin{align}
\frac{dy}{dx} &= x^2 \\
y(x) &= \frac{1}{3} x^3 + C
\end{align}

While these problems look simple, there may not be an obvious closed-form solution to all:

\begin{align}
\frac{dy}{dx} &= e^{-x^2} \\
y(x) &= \int e^{-x^2} dx + C
\end{align}

(You may recognize this as leading to the error function, $\text{erf}$:
$\frac{1}{2} \sqrt{\pi} \text{erf}(x) + C$,
so the exact solution to the integral over the range $[0,1]$ is 0.7468.)

## 2. Solution by separation of variables

If the given derivative is a separate function of $x$ and $y$, then we can solve via separation of variables:
\begin{align}
\frac{dy}{dx} &= f(x) g(y) = \frac{h(x)}{j(y)} \\
\int \frac{1}{g(y)} dy &= \int f(x) dx
\end{align}

For example, consider this problem:
\begin{equation}
y^{\prime} = \frac{dy}{dx} = 1 + y^2 \\
\end{equation}
We can separate this into a problem that looks like $f(y) dy = g(x) dx$, where $dy = \frac{1}{1+y^2}$ and $g(x) = 1$.
\begin{align}
\int \frac{dy}{1 + y^2} &= \int dx \\
\arctan y &= x + c \\
y(x) &= \tan(x+c)
\end{align}

Unfortunately, not every separable ODE can be integrated:
\begin{align}
\frac{dy}{dx} &= \frac{e^x / 2 + 5}{y^2 + \cos y} \\
(y^2 + \cos y) dy &= (e^x / 2 + 5) dx
\end{align}

## 3. General solution to linear 1st-order ODEs

Given a general linear 1st-order ODE of the form
\begin{equation}
\frac{dy}{dx} + p(x) y = q(x)
\end{equation}
we can solve by integration factor:
\begin{equation}
y(x) = e^{-\int p(x) dx} \left[ \int e^{\int p(x) dx} q(x) dx + C \right]
\end{equation}

For example, in this equation
\begin{equation}
y^{\prime} + xy - 5 e^x = 0
\end{equation}
after rearranging to the standard form
\begin{equation}
y^{\prime} + xy = 5 e^x
\end{equation}
we see that $p(x) = x$ and $q(x) = 5e^x$.

## 4. Solution to nonlinear 1st-order ODEs

Given a general nonlinear 1st-order ODE
\begin{equation}
\frac{dy}{dx} + p(x) y = q(x) y^a
\end{equation}
where $a \neq 1$ and $a$ is a constant. This is known as the Bernoulli equation.

We can solve by transforming to a linear equation, by changing the dependent variable from $y$ to $z$:
\begin{align}
\text{let} \quad z &= y^{1-a} \\
\frac{dz}{dx} &= (1-a) y^{-a} \frac{dy}{dx}
\end{align}
Multiply the original equation by $(1-a) y^{-a}$:
\begin{align}
(1-a) y^{-a} \frac{dy}{dx} + (1-a) y^{-a} p(x) y &= (1-a) y^{-a} q(x) y^a \\
\frac{dz}{dx} + p(x) (1-a) z &= q(x) (1-a) \;,
\end{align}
which is now a *linear* first-order ODE, that looks like
\begin{equation}
\frac{dz}{dx} + p(x)^{\prime} z = q(x)^{\prime}
\end{equation}
where $p(x)^{\prime} = (1-a) p(x)$ and $q(x)^{\prime} = (1-a)q(x)$.

We can solve this using the integrating-factor approach discussed above. Then, once we have $z(x)$, we can find $y(x)$:
\begin{align}
z &= y^{1-a} \\
y &= z^{\frac{1}{1-a}}
\end{align}


# Pyomo - Getting started

Pyomo installation: see http://www.pyomo.org/installation

```
pip install pyomo
```

See also these excellent tutorials that details how to install Pyomo and several solvers (GLPK, COIN-OR CBC, COIN-OR Ipopt, ...):
- https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.01-Installing-Pyomo.ipynb
- https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.02-Running-Pyomo-on-Google-Colab.ipynb
- https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.04-Cross-Platform-Installation-of-Pyomo-and-Solvers.ipynb

```python
from pyomo.environ import *
```

## Example 1

```python
model = ConcreteModel(name="Getting started")

model.x = Var(bounds=(-10, 10))

model.obj = Objective(expr=model.x)

model.const_1 = Constraint(expr=model.x >= 5)

# @tail:
opt = SolverFactory('glpk')  # "glpk" or "cbc"

res = opt.solve(model)    # solves and updates instance

model.display()

print()
print("Optimal solution: ", value(model.x))
print("Cost of the optimal solution: ", value(model.obj))
# @:tail
```

## Example 2

$$
\begin{align}
    \max_{x_1,x_2} & \quad 4 x_1 + 3 x_2 \\
    \text{s.t.}    & \quad   x_1 +   x_2 \leq 100 \\
                   & \quad 2 x_1 +   x_2 \leq 150 \\
                   & \quad 3 x_1 + 4 x_2 \leq 360 \\
                   & \quad x_1, x_2 \geq 0
\end{align}
$$

```
Optimal total cost is:  350.0

x_1 = 50.
x_2 = 50.
```

```python
model = ConcreteModel(name="Getting started")

model.x1 = Var(within=NonNegativeReals)
model.x2 = Var(within=NonNegativeReals)

model.obj = Objective(expr=4. * model.x1 + 3. * model.x2, sense=maximize)

model.ineq_const_1 = Constraint(expr=model.x1 + model.x2 <= 100)
model.ineq_const_2 = Constraint(expr=2. * model.x1 + model.x2 <= 150)
model.ineq_const_3 = Constraint(expr=3. * model.x1 + 4. * model.x2 <= 360)

# @tail:
opt = SolverFactory('glpk')  # "glpk" or "cbc"

results = opt.solve(model)    # solves and updates instance

model.display()

print()
print("Optimal solution: ({}, {})".format(value(model.x1), value(model.x2)))
print("Gain of the optimal solution: ", value(model.obj))
# @:tail
```


# Straight Forward Expansion

```python
import sympy as sp
from sympy.simplify.fu import TR7, TR8
```

```python
N = 3

# Define the symbolic parameters
# epsilon = sp.symbols('epsilon_(1:' + str(N+1) + ')')
epsilon = sp.symbols('epsilon')
omega_0 = sp.symbols('omega_0')
alpha = sp.symbols('alpha_(2:' + str(N+1) + ')')

# Define time variable
t = sp.symbols('t', real=True)

# Define function variable
x = sp.Function('x')(t)
xdot = sp.Derivative(x, t)  # first time derivative
xddot = sp.Derivative(xdot, t)  # second time derivative

# EOM
EOM = xddot + omega_0**2 * x + sum([alpha[i-2] * x**i for i in range(2,N+1)])
EOM
```

$\displaystyle \alpha_{2} x^{2}{\left(t \right)} + \alpha_{3} x^{3}{\left(t \right)} + \omega_{0}^{2} x{\left(t \right)} + \frac{d^{2}}{d t^{2}} x{\left(t \right)}$

```python
# Exponential expansion of the variable x
x1 = sp.Function('x_1', real=True)(t)
x2 = sp.Function('x_2', real=True)(t)
x3 = sp.Function('x_3', real=True)(t)
```

```python
x_i = (x1, x2, x3)
x_e = sum([epsilon**i * x_i[i-1] for i in range(1,N+1)])
x_e
```

$\displaystyle \epsilon^{3} \operatorname{x_{3}}{\left(t \right)} + \epsilon^{2} \operatorname{x_{2}}{\left(t \right)} + \epsilon \operatorname{x_{1}}{\left(t \right)}$

```python
# Substitute this into the EOM
EOM = EOM.subs(x, x_e)
EOM
```

$\displaystyle \alpha_{2} \left(\epsilon^{3} \operatorname{x_{3}}{\left(t \right)} + \epsilon^{2} \operatorname{x_{2}}{\left(t \right)} + \epsilon \operatorname{x_{1}}{\left(t \right)}\right)^{2} + \alpha_{3} \left(\epsilon^{3} \operatorname{x_{3}}{\left(t \right)} + \epsilon^{2} \operatorname{x_{2}}{\left(t \right)} + \epsilon \operatorname{x_{1}}{\left(t \right)}\right)^{3} + \omega_{0}^{2} \left(\epsilon^{3} \operatorname{x_{3}}{\left(t \right)} + \epsilon^{2} \operatorname{x_{2}}{\left(t \right)} + \epsilon \operatorname{x_{1}}{\left(t \right)}\right) + \frac{\partial^{2}}{\partial t^{2}} \left(\epsilon^{3} \operatorname{x_{3}}{\left(t \right)} + \epsilon^{2} \operatorname{x_{2}}{\left(t \right)} + \epsilon \operatorname{x_{1}}{\left(t \right)}\right)$

```python
EOM = sp.expand(sp.expand(EOM).doit())
```

```python
# Collect the coefficients for the epsilons
epsilon_Eq = sp.collect(EOM, epsilon, evaluate=False)
epsilon_1_Eq = sp.Eq(epsilon_Eq[epsilon], 0)
epsilon_1_Eq
```

$\displaystyle \omega_{0}^{2} \operatorname{x_{1}}{\left(t \right)} + \frac{d^{2}}{d t^{2}} \operatorname{x_{1}}{\left(t \right)} = 0$

```python
epsilon_2_Eq = sp.Eq(epsilon_Eq[epsilon**2], 0)
epsilon_2_Eq
```

$\displaystyle \alpha_{2} \operatorname{x_{1}}^{2}{\left(t \right)} + \omega_{0}^{2} \operatorname{x_{2}}{\left(t \right)} + \frac{d^{2}}{d t^{2}} \operatorname{x_{2}}{\left(t \right)} = 0$

```python
epsilon_3_Eq = sp.Eq(epsilon_Eq[epsilon**3], 0)
epsilon_3_Eq
```

$\displaystyle 2 \alpha_{2} \operatorname{x_{1}}{\left(t \right)} \operatorname{x_{2}}{\left(t \right)} + \alpha_{3} \operatorname{x_{1}}^{3}{\left(t \right)} + \omega_{0}^{2} \operatorname{x_{3}}{\left(t \right)} + \frac{d^{2}}{d t^{2}} \operatorname{x_{3}}{\left(t \right)} = 0$

```python
sp.dsolve(epsilon_1_Eq, x1)
```

$\displaystyle \operatorname{x_{1}}{\left(t \right)} = C_{1} e^{- i \omega_{0} t} + C_{2} e^{i \omega_{0} t}$

Not as convenient as working in the polar form

```python
a = sp.symbols('a')
beta = sp.symbols('beta')
x1_polar = a * sp.cos(omega_0 * t + beta)
x1_polar
```

$\displaystyle a \cos{\left(\beta + \omega_{0} t \right)}$

Updated $\epsilon$-2 equation

```python
epsilon_2_Eq = sp.expand(TR7(epsilon_2_Eq.subs(x1, x1_polar)))
epsilon_2_Eq
```

$\displaystyle \frac{a^{2} \alpha_{2} \cos{\left(2 \beta + 2 \omega_{0} t \right)}}{2} + \frac{a^{2} \alpha_{2}}{2} + \omega_{0}^{2} \operatorname{x_{2}}{\left(t \right)} + \frac{d^{2}}{d t^{2}} \operatorname{x_{2}}{\left(t \right)} = 0$

```python
sp.dsolve(epsilon_2_Eq, x2)
```

$\displaystyle \operatorname{x_{2}}{\left(t \right)} = C_{1} e^{- i \omega_{0} t} + C_{2} e^{i \omega_{0} t} + \frac{a^{2} \alpha_{2} \cos{\left(2 \beta + 2 \omega_{0} t \right)}}{6 \omega_{0}^{2}} - \frac{a^{2} \alpha_{2}}{2 \omega_{0}^{2}}$

We only want to keep the particular solution in the above

```python
x2_p = alpha[0] * a**2 * (sp.cos(2 * omega_0 * t + 2*beta) - 3)/6/omega_0**2
x2_p
```

$\displaystyle \frac{a^{2} \alpha_{2} \left(\cos{\left(2 \beta + 2 \omega_{0} t \right)} - 3\right)}{6 \omega_{0}^{2}}$

Updated $\epsilon$-3 equation

```python
epsilon_3_Eq = TR7(epsilon_3_Eq.subs([
    (x1, x1_polar), (x2, x2_p)
]))
epsilon_3_Eq = TR8(epsilon_3_Eq)
epsilon_3_Eq
```

$\displaystyle \frac{a^{3} \alpha_{2}^{2} \left(- 5 \cos{\left(\beta + \omega_{0} t \right)} + \cos{\left(3 \beta + 3 \omega_{0} t \right)}\right)}{6 \omega_{0}^{2}} + a^{3} \alpha_{3} \left(\frac{3 \cos{\left(\beta + \omega_{0} t \right)}}{4} + \frac{\cos{\left(3 \beta + 3 \omega_{0} t \right)}}{4}\right) + \omega_{0}^{2} \operatorname{x_{3}}{\left(t \right)} + \frac{d^{2}}{d t^{2}} \operatorname{x_{3}}{\left(t \right)} = 0$

```python
sp.dsolve(epsilon_3_Eq)
```

$\displaystyle \operatorname{x_{3}}{\left(t \right)} = C_{1} e^{- i \omega_{0} t} + C_{2} e^{i \omega_{0} t} + \frac{5 a^{3} \alpha_{2}^{2} t \sin{\left(\beta + \omega_{0} t \right)}}{12 \omega_{0}^{3}} + \frac{a^{3} \alpha_{2}^{2} \cos{\left(3 \beta + 3 \omega_{0} t \right)}}{48 \omega_{0}^{4}} - \frac{3 a^{3} \alpha_{3} t \sin{\left(\beta + \omega_{0} t \right)}}{8 \omega_{0}} + \frac{a^{3} \alpha_{3} \cos{\left(3 \beta + 3 \omega_{0} t \right)}}{32 \omega_{0}^{2}}$


## Problem 2

```python
import numpy as np
import sympy as sp
import scipy as sc
import matplotlib.pyplot as plt
from math import e
%matplotlib inline
```

The given equation is $x(t)=A_1e^{s_1t} + A_2e^{s_2t}$ thus we can find the values of $A_1$ and $A_2$ by
 substituting $A_2=1-A_1$ and $A_1=\frac{-s_2}{s_1-s_2}$ using the given intital conditions.
Then get the values of $s_1$ and $s_2$ from the provided equation for parts A to D using the different values of $a$ and $b$.

#### Part A

```python
a=1
b=0.25
s1=-a/2 + np.sqrt((a/2)**2 - b**2)
s2=-a/2 - np.sqrt((a/2)**2 - b**2)
print("s1=",s1,"s2=",s2)
A1=-s2/(s1-s2)
A2=1-A1
# A1=A2=1
print("A1=",A1,"A2=",A2)
t=np.arange(0,2*np.pi,0.1)
x=A1*(e**(s1*t)) + A2*(e**(s2*t))
# # plt.ylim(-10,10,0.5)
# plt.xlim(-6,6,0.5)
plt.xlabel("Period")
plt.ylabel("x(t)")
plt.plot(t,x)
plt.legend(['for a=1 & b=0.25'])
plt.show()
```

#### Part B

```python
a=-1
b=0.25
s1=-a/2 + np.sqrt((a/2)**2 - b**2)
s2=-a/2 - np.sqrt((a/2)**2 - b**2)
print("s1=",s1,"s2=",s2)
A1=-s2/(s1-s2)
A2=1-A1
print("A1=",A1,"A2=",A2)
t=np.arange(0,2*np.pi,0.1)
x=A1*(e**(s1*t)) + A2*(e**(s2*t))
plt.xlabel("Period")
plt.ylabel("x(t)")
plt.plot(t,x)
plt.legend(['for a=-1 & b=0.25'])
plt.show()
```

#### Part C

```python
a=1
b=1
s1=complex(-0.5,0.5)
s2=complex(-0.5,-0.5)
print("s1=",s1,"s2=",s2)
A1=-s1/(s1-s2)
A2=1-A1
print("A1=",A1,"A2=",A2)
t=np.arange(0,2*np.pi,0.1)
x=A1.real*(e**(s1.real*t)) + A2.real*(e**(s2.real*t)) #ignoring imaginary parts
plt.xlabel("Period")
plt.ylabel("x(t)")
plt.plot(t,x)
plt.legend(['for a=1 & b=1'])
plt.show()
```

#### Part D

```python
a=-1
b=1
s1=complex(0.5,0.5)
s2=complex(0.5,-0.5)
print("s1=",s1,"s2=",s2)
A1=-s2/(s1-s2)
A2=1-A1
print("A1=",A1,"A2=",A2)
t=np.arange(0,2*np.pi,0.1)
x=A1.real*(e**(s1.real*t)) + A2.real*(e**(s2.real*t)) #ignoring imaginary parts
plt.xlabel("Period")
plt.ylabel("x(t)")
plt.plot(t,x)
plt.legend(['for a=-1 & b=1'])
plt.show()
```

#### Part E

```python
#Undamped oscillation which is pretty much a straight line.
a=0
b=1
s1=complex(0,1)
s2=complex(0,-1)
print("s1=",s1,"s2=",s2)
A1=-s2/(s1-s2)
A2=1-A1
print("A1=",A1,"A2=",A2)
t=np.arange(0,2*np.pi,0.1)
x=A1.real*(e**(s1.real*t)) + A2.real*(e**(s2.real*t)) #ignoring imaginary parts
plt.xlabel("Period")
plt.ylabel("x(t)")
plt.plot(t,x)
plt.legend(['for a=0 & b=1'])
plt.show()
```

1. Single Degree of Freedom Oscillation/1. SDOF Code.ipynb


# Essential Math for Machine Learning: Python Edition

Course source: [LinkedIn Learning](https://www.linkedin.com/learning/essential-math-for-machine-learning-python-edition)

## 1. Equations, Graphs, and Functions

### 1.1 Linear Equations

#### 1.1.1 Solving a Linear Equation
Consider the following equation:

\begin{equation}2y + 3 = 3x - 1 \end{equation}

After simplification: \begin{equation}y = \frac{3x - 4}{2} \end{equation}

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the solved equation to x
df['y'] = (3*df['x'] - 4) / 2

#Display the dataframe
df
```

      x
      y

      0
      -10
      -17.0

      1
      -9
      -15.5

      2
      -8
      -14.0

      3
      -7
      -12.5

      4
      -6
      -11.0

      5
      -5
      -9.5

      6
      -4
      -8.0

      7
      -3
      -6.5

      8
      -2
      -5.0

      9
      -1
      -3.5

      10
      0
      -2.0

      11
      1
      -0.5

      12
      2
      1.0

      13
      3
      2.5

      14
      4
      4.0

      15
      5
      5.5

      16
      6
      7.0

      17
      7
      8.5

      18
      8
      10.0

      19
      9
      11.5

      20
      10
      13.0

#### 1.1.2 Visualization

```python
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey", marker = "o")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.show()
```

#### 1.1.3 Intercepts

Let's take a look at the line from our linear equation with the X and Y axis shown through the origin (0,0).

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.show()
```

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
## After calculation equation x-intercept: 4/3, y-intercept: -2
plt.annotate('x-intercept',(1.333, 0))
plt.annotate('y-intercept',(0,-2))
plt.show()
```

#### 1.1.4 Slope

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# set the slope
m = 1.5

# get the y-intercept
yInt = -2

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

##### 1.1.4.1 Slope-Intercept Form

Now that we know the slope and y-intercept for the line that this equation defines, we can rewrite the equation as:

\begin{equation}y = 1\frac{1}{2}x + -2 \end{equation}

```python
# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Define slope and y-intercept
m = 1.5
yInt = -2

# Add a y column by applying the slope-intercept equation to x
df['y'] = m*df['x'] + yInt

# Plot the line
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# label the y-intercept
plt.annotate('y-intercept',(0,yInt))

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

### 1.2 Systems of Equations

Here are the equations

\begin{equation}x + y = 16 \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

Taken together, these equations form a *system of equations* that will enable us to determine how many of each chip denomination we have.

#### 1.2.1 Graphing Lines to Find the Intersection Point

One approach is to determine all possible values for x and y in each equation and plot them.

Let's plot each of these ranges of values as lines on a graph:

```python
# Get the extremes for number
x1 = [16, 0]
y1 = [0, 16]

# Get the extremes for values
x2 = [25,0]
y2 = [0,10]

# Plot the lines
plt.plot(x1, y1, color='blue')
plt.plot(x2, y2, color="orange")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

plt.show()
```

### 1.3 Exponentials, Radicals, and Logs

#### 1.3.1 Exponentials

\begin{equation}2^{2} = 2 \cdot 2 = 4\end{equation}

```python
x = 2**2
print(x)
```

    4

#### 1.3.2 Radicals (Roots)

\begin{equation}\sqrt[3]{64} = 4 \end{equation}

```python
import math

# Calculate square root of 25
x = math.sqrt(25)
print (x)

# Calculate cube root of 64
cr = round(64 ** (1. / 3))
print(cr)
```

    5.0
    4

#### 1.3.3 Logarithms

\begin{equation}log_{4}(16) = 2 \end{equation}
\begin{equation}log(1000) = 3 \end{equation}
\begin{equation}log_{e}(64) = ln(64) = 4.1589 \end{equation}

```python
# log of 16 base 4
print(math.log(16, 4))

# Common log of 1000
print(math.log10(1000))

# Natural log of 64
print (math.log(64))
```

    2.0
    3.0
    4.1588830833596715

#### 1.3.4 Solving Equations with Exponentials

\begin{equation}2y = 2x^{4} ( \frac{x^{2} + 2x^{2}}{x^{3}} ) \end{equation}

\begin{equation}y = 3x^{3} \end{equation}

Now we have a solution that defines y in terms of x. We can use Python to plot the line created by this equation for a set of arbitrary *x* and *y* values:

```python
# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 3*df['x']**3

#Display the dataframe
print(df)

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

For Equation:
\begin{equation}
y = 2^{x}
\end{equation}

```python
# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 2.0**df['x']

#Display the dataframe
print(df)

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

### 1.4 Quadratic Equations

Consider the following equation:

\begin{equation}y = 2(x - 1)(x + 2)\end{equation}

If you multiply out the factored ***x*** expressions, this equates to:

\begin{equation}y = 2x^{2} + 2x - 4\end{equation}

Note that the highest ordered term includes a squared variable (x2).

Let's graph this equation for a range of ***x*** values:

```python
# Create a dataframe with an x column containing values to plot
df = pd.DataFrame ({'x': range(-9, 9)})

# Add a y column by applying the quadratic equation to x
df['y'] = 2*df['x']**2 + 2 *df['x'] - 4

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

For following **Parabola**

\begin{equation}y = -2x^{2} + 6x + 7\end{equation}

```python
# Create dataFrame for x column
df = pd.DataFrame({'x': range(-9,13)})

# Add y column
df['y'] = -2 * df['x'] ** 2 + 6 * df['x'] - 4

plt.plot(df.x, df.y, color="blue")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Again, the graph shows a parabola, but this time instead of being open at the top, the parabola is open at the bottom.

Equations that assign a value to ***y*** based on an expression that includes a squared value for ***x*** create parabolas. If the relationship between ***y*** and ***x*** is such that ***y*** is a *positive* multiple of the ***x2*** term, the parabola will be open at the top; when ***y*** is a *negative* multiple of the ***x2*** term, then the parabola will be open at the bottom.

These kinds of equations are known as *quadratic* equations, and they have some interesting characteristics. There are several ways quadratic equations can be written, but the *standard form* for quadratic equation is:

\begin{equation}y = ax^{2} + bx + c\end{equation}

Where ***a***, ***b***, and ***c*** are numeric coefficients or constants.

#### 1.4.1 Parabola Vertex and Line of Symmetry

Parabolas are symmetrical, with x and y values converging exponentially towards the highest point (in the case of a downward opening parabola) or lowest point (in the case of an upward opening parabola). The point where the parabola meets the line of symmetry is known as the *vertex*.

```python
import numpy as np

def plot_parabola(a, b, c):
    # get the x value for the line of symmetry
    vx = (-1*b)/(2*a)

    # get the y value when x is at the line of symmetry
    vy = a*vx**2 + b*vx + c

    # Create a dataframe with an x column containing values from x-10 to x+10
    minx = int(vx - 10)
    maxx = int(vx + 11)
    df = pd.DataFrame ({'x': range(minx, maxx)})

    # Add a y column by applying the quadratic equation to x
    df['y'] = a*df['x']**2 + b *df['x'] + c

    # get min and max y values
    miny = df.y.min()
    maxy = df.y.max()

    # Plot the line
    plt.plot(df.x, df.y, color="grey")
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid()
    plt.axhline()
    plt.axvline()

    # plot the line of symmetry
    sx = [vx, vx]
    sy = [miny, maxy]
    plt.plot(sx,sy, color='magenta')

    # Annotate the vertex
    plt.scatter(vx,vy, color="red")
    plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy + 5)* np.sign(a)))

    plt.show()
```

```python
plot_parabola(2, 2, -4)
plot_parabola(-2, 3, 5)
```

#### 1.4.2 Parabola Intercepts

When y is 0, x is -2 or 1. Let's plot these points on our parabola:

```python
# Assign the calculated x values
x1 = -2
x2 = 1

# Create a dataframe with an x column containing some values to plot
df = pd.DataFrame ({'x': range(x1-5, x2+6)})

# Add a y column by applying the quadratic equation to x
df['y'] = 2*(df['x'] - 1) * (df['x'] + 2)

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = 2*(vx -1)*(vx + 2)

# get min and max y values
miny = df.y.min()
maxy = df.y.max()

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# Plot calculated x values for y = 0
plt.scatter([x1,x2],[0,0], color="green")
plt.annotate('x1',(x1, 0))
plt.annotate('x2',(x2, 0))

# plot the line of symmetry
sx = [vx, vx]
sy = [miny, maxy]
plt.plot(sx,sy, color='magenta')

# Annotate the vertex
plt.scatter(vx,vy, color="red")
plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy - 5)))

plt.show()
```

#### 1.4.3 Solving Quadratics Using the Square Root Method

Let's consider this equation:

\begin{equation}y = 3x^{2} - 12\end{equation}

Let's restate it so we're solving for ***x*** when ***y*** is 0:

\begin{equation}3x^{2} - 12 = 0\end{equation}

\begin{equation}x = \pm\sqrt{4}\end{equation}

Let's see this in Python, and use the results to calculate and plot the parabola with its line of symmetry and vertex:

```python
y = 0
x1 = int(- math.sqrt(y + 12 / 3))
x2 = int(math.sqrt(y + 12 / 3))

# Create a dataframe with an x column containing some values to plot
df = pd.DataFrame ({'x': range(x1-10, x2+11)})

# Add a y column by applying the quadratic equation to x
df['y'] = 3*df['x']**2 - 12

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = 3*vx**2 - 12

# get min and max y values
miny = df.y.min()
maxy = df.y.max()

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# Plot calculated x values for y = 0
plt.scatter([x1,x2],[0,0], color="green")
plt.annotate('x1',(x1, 0))
plt.annotate('x2',(x2, 0))

# plot the line of symmetry
sx = [vx, vx]
sy = [miny, maxy]
plt.plot(sx,sy, color='magenta')

# Annotate the vertex
plt.scatter(vx,vy, color="red")
plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy - 20)))

plt.show()
```

#### 1.4.4 Solving Quadratics Using the Completing the Square Method

Let's look at an example:

\begin{equation}y = x^{2} + 6x - 7\end{equation}

Let's start as we've always done so far by restating the equation to solve ***x*** for a ***y*** value of 0:

\begin{equation}x^{2} + 6x - 7 = 0\end{equation}

\begin{equation}x^{2} + 6x = 7\end{equation}

\begin{equation}x^{2} + 6x + 9 = 16\end{equation}

\begin{equation}(x + 3)^{2} = 16\end{equation}

\begin{equation}x + 3 =\pm\sqrt{16}\end{equation}

\begin{equation}x = -7, 1\end{equation}

Let's see what the parabola for this equation looks like in Python:

```python
x1 = int(- math.sqrt(16) - 3)
x2 = int(math.sqrt(16) - 3)

# Create a dataframe with an x column containing some values to plot
df = pd.DataFrame ({'x': range(x1-10, x2+11)})

# Add a y column by applying the quadratic equation to x
df['y'] = ((df['x'] + 3)**2) - 16

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = ((vx + 3)**2) - 16

# get min and max y values
miny = df.y.min()
maxy = df.y.max()

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# Plot calculated x values for y = 0
plt.scatter([x1,x2],[0,0], color="green")
plt.annotate('x1',(x1, 0))
plt.annotate('x2',(x2, 0))

# plot the line of symmetry
sx = [vx, vx]
sy = [miny, maxy]
plt.plot(sx,sy, color='magenta')

# Annotate the vertex
plt.scatter(vx,vy, color="red")
plt.annotate('vertex',(vx, vy), xytext=(vx - 1, (vy - 10)))

plt.show()
```

#### 1.4.5 Vertex Form

Th *vertex form* of a quadratic equation which is generically described as:

\begin{equation}y = a(x - h)^{2} + k\end{equation}

The neat thing about this form of the equation is that it tells us the coordinates of the vertex - it's at ***h,k***.

```python
def plot_parabola_from_vertex_form(a, h, k):
    # Create a dataframe with an x column a range of x values to plot
    df = pd.DataFrame ({'x': range(h-10, h+11)})

    # Add a y column by applying the quadratic equation to x
    df['y'] = (a*(df['x'] - h)**2) + k

    # get min and max y values
    miny = df.y.min()
    maxy = df.y.max()

    # calculate y when x is 0 (h+-h)
    y = a*(0 - h)**2 + k

    plt.plot(df.x, df.y, color="grey")
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid()
    plt.axhline()
    plt.axvline()

    # Plot calculated y values for x = 0 (h-h and h+h)
    plt.scatter([h-h, h+h],[y,y], color="green")
    plt.annotate(str(h-h) + ',' + str(y),(h-h, y))
    plt.annotate(str(h+h) + ',' + str(y),(h+h, y))

    # plot the line of symmetry (x = h)
    sx = [h, h]
    sy = [miny, maxy]
    plt.plot(sx,sy, color='magenta')

    # Annotate the vertex (h,k)
    plt.scatter(h,k, color="red")
    plt.annotate('v=' + str(h) + ',' + str(k),(h, k), xytext=(h - 1, (k - 10)))

    plt.show()
```

```python
# Call the function for the example discussed above
plot_parabola_from_vertex_form(2, 4, -30)
```

```python
plot_parabola_from_vertex_form(3, -1, -1)
```

#### 1.4.6 Shortcuts for Solving Quadratic Equations

##### 1.4.6.1 The Quadratic Formula
Another useful formula to remember is the *quadratic formula*, which makes it easy to calculate values for ***x*** when ***y*** is **0**; or in other words:

\begin{equation}ax^{2} + bx + c = 0\end{equation}

Here's the formula:

\begin{equation}x = \frac{-b \pm \sqrt{b^{2} - 4ac}}{2a}\end{equation}

```python
def plot_parabola_from_formula (a, b, c):
    # Get vertex
    print('CALCULATING THE VERTEX')
    print('vx = -b / 2a')

    nb = -b
    a2 = 2*a
    print('vx = ' + str(nb) + ' / ' + str(a2))

    vx = -b/(2*a)
    print('vx = ' + str(vx))

    print('\nvy = ax^2 + bx + c')
    print('vy = ' + str(a) + '(' + str(vx) + '^2) + ' + str(b) + '(' + str(vx) + ') + ' + str(c))

    avx2 = a*vx**2
    bvx = b*vx
    print('vy = ' + str(avx2) + ' + ' + str(bvx) + ' + ' + str(c))

    vy = avx2 + bvx + c
    print('vy = ' + str(vy))

    print ('\nv = ' + str(vx) + ',' + str(vy))

    # Get +x and -x (showing intermediate calculations)
    print('\nCALCULATING -x AND +x FOR y=0')
    print('x = -b +- sqrt(b^2 - 4ac) / 2a')

    b2 = b**2
    ac4 = 4*a*c
    print('x = ' + str(nb) + ' +- sqrt(' + str(b2) + ' - ' + str(ac4) + ') / ' + str(a2))

    sr = math.sqrt(b2 - ac4)
    print('x = ' + str(nb) + ' +- ' + str(sr) + ' / ' + str(a2))
    print('-x = ' + str(nb) + ' - ' + str(sr) + ' / ' + str(a2))
    print('+x = ' + str(nb) + ' + ' + str(sr) + ' / ' + str(a2))

    posx = (nb + sr) / a2
    negx = (nb - sr) / a2
    print('-x = ' + str(negx))
    print('+x = ' + str(posx))

    print('\nPLOTTING THE PARABOLA')

    # Create a dataframe with an x column a range of x values to plot
    df = pd.DataFrame ({'x': range(round(vx)-10, round(vx)+11)})

    # Add a y column by applying the quadratic equation to x
    df['y'] = a*df['x']**2 + b*df['x'] + c

    # get min and max y values
    miny = df.y.min()
    maxy = df.y.max()

    plt.plot(df.x, df.y, color="grey")
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid()
    plt.axhline()
    plt.axvline()

    # Plot calculated x values for y = 0
    plt.scatter([negx, posx],[0,0], color="green")
    plt.annotate('-x=' + str(negx) + ',' + str(0),(negx, 0), xytext=(negx - 3, 5))
    plt.annotate('+x=' + str(posx) + ',' + str(0),(posx, 0), xytext=(posx - 3, -10))

    # plot the line of symmetry
    sx = [vx, vx]
    sy = [miny, maxy]
    plt.plot(sx,sy, color='magenta')

    # Annotate the vertex
    plt.scatter(vx,vy, color="red")
    plt.annotate('v=' + str(vx) + ',' + str(vy),(vx, vy), xytext=(vx - 1, vy - 10))

    plt.show()
```

```python
plot_parabola_from_formula (2, -16, 2)
```

### 1.5 Functions

For example, consider the following equation:

\begin{equation}f(x) = x^{2} + 2\end{equation}

```python
def f(x):
    return x**2 + 2
```

```python
# Create an array of x values from -100 to 100
x = np.array(range(-100, 101))

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot x against f(x)
plt.plot(x,f(x), color='purple')

plt.show()
```

#### 1.5.1 Bounds of a Function

For example, consider the function ***u*** defined here:

\begin{equation}u(x) = x + 1\end{equation}
\begin{equation}\{x \in \rm I\!R\}\end{equation}

##### 1.5.1.1 Domain of a Function

Now consider the following function ***g***:

\begin{equation}g(x) = (\frac{12}{2x})^{2}\end{equation}

This is interpreted as *Any value for x where x is in the set of real numbers such that x is not equal to 0*, and we can incorporate this into the function's definition like this:

\begin{equation}g(x) = (\frac{12}{2x})^{2}, \{x \in \rm I\!R\;\;|\;\; x \ne 0 \}\end{equation}

Or more simply:

\begin{equation}g(x) = (\frac{12}{2x})^{2},\;\; x \ne 0\end{equation}

```python
# Define function g
def g(x):
    if x != 0:
        return (12/(2*x))**2
```

```python
# Create an array of x values from -100 to 100
x = range(-100, 101)

# Get the corresponding y values from the function
y = [g(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='purple')

# plot an empty circle to show the undefined point
plt.plot(0,g(0.0000001), color='purple', marker='o', markerfacecolor='w', markersize=8)

plt.show()
```

We can indicate the domain of this function in its definition like this:

\begin{equation}h(x) = 2\sqrt{x}, \{x \in \rm I\!R\;\;|\;\; x \ge 0 \}\end{equation}

```python
def h(x):
    if x >= 0:
        return 2 * np.sqrt(x)
```

```python
# Create an array of x values from -100 to 100
x = range(-100, 101)

# Get the corresponding y values from the function
y = [h(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('h(x)')
plt.grid()

# Plot x against h(x)
plt.plot(x,y, color='purple')

# plot a filled circle at the end to indicate a closed interval
plt.plot(0, h(0), color='purple', marker='o', markerfacecolor='purple', markersize=8)

plt.show()
```

##### 1.5.1.2 Range of a Function

For example, consider the following function:

\begin{equation}p(x) = x^{2} + 1\end{equation}

It's range is:

\begin{equation}\{p(x) \in \rm I\!R\;\;|\;\; p(x) \ge 1 \}\end{equation}

Let's create and plot the function for a range of ***x*** values in Python:

```python
# define a function to return x^2 + 1
def p(x):
    return x**2 + 1
```

```python
# Create an array of x values from -100 to 100
x = np.array(range(-100, 101))

# Set up the graph
plt.xlabel('x')
plt.ylabel('p(x)')
plt.grid()

# Plot x against f(x)
plt.plot(x,p(x), color='purple')

plt.show()
```

Codes/Essential Math for Machine Learning/1. Equations, Graphs, and Functions.ipynb