# Vertical Line Test

```
# imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import math
```

## 1.1 Create two graphs, one that passes the vertical line test and one that does not.

```
x = np.arange(-9, 10)
y = x**3 + 2*x
plt.plot(x, y)
plt.axvline(x=0, color='purple');
```

## 1.2 Why are graphs that don't pass the vertical line test not considered "functions?"

The vertical line test can be used to determine whether a graph represents a function. If we can draw any vertical line that intersects a graph more than once, then the graph does not define a function because a function has only one output value for each input value.

# Functions as Relations

## 2.1 Which of the following relations are functions? Why?

\begin{align}
\text{Relation 1: } \{(1, 2), (3, 2), (1, 3)\}
\\
\text{Relation 2: } \{(1, 3), (2, 3), (6, 7)\}
\\
\text{Relation 3: } \{(9, 4), (2, 1), (9, 6)\}
\\
\text{Relation 4: } \{(6, 2), (8, 3), (6, 4)\}
\\
\text{Relation 5: } \{(2, 6), (2, 7), (2, 4)\}
\end{align}

**Answer: only relation 2 is the function, because a function cannot have the same input and the other functions intersects the vertical line more than once**

```
# Relation 1
x = np.array([1, 3, 1])
y = np.array([2, 2, 3])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=2, color='green');
```

```
# Relation 2
x = np.array([1, 2, 6])
y = np.array([3, 3, 7])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=2, color='green');
```

```
# Relation 3
x = np.array([9, 2, 9])
y = np.array([4, 1, 6])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=3, color='green');
```

```
# Relation 4
x = np.array([6, 8, 6])
y = np.array([2, 3, 4])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=7, color='green');
```

```
# Relation 5
x = np.array([2, 2, 2])
y = np.array([6, 7, 4])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=2, color='green');
```

# Functions as a mapping between dimensions

## 3.1 for the following functions what is the dimensionality of the domain (input) and codomain (range/output)?

\begin{align}
m(𝑥_1,𝑥_2,𝑥_3)=(x_1+x_2, x_1+x_3, x_2+x_3)
\\
n(𝑥_1,𝑥_2,𝑥_3,𝑥_4)=(x_2^2 + x_3, x_2x_4)
\end{align}

```

```

## 3.2 Do you think it's possible to create a function that maps from a lower dimensional space to a higher dimensional space? If so, provide an example.

# Vector Transformations

## 4.1 Plug the corresponding unit vectors into each function. Use the output vectors to create a transformation matrix.

\begin{align}
p(\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}
\\
\\
q(\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}
\end{align}

## 4.2 Verify that your transformation matrices are correct by choosing an input matrix and calculating the result both via the traditional functions above and also via vector-matrix multiplication.

```

```

# Eigenvalues and Eigenvectors

## 5.1 In your own words, give an explanation for the intuition behind eigenvalues and eigenvectors.

# The Curse of Dimensionality

## 6.1 What are some of the challenges of working with high dimensional spaces?

## 6.2 What is the rule of thumb for how many observations you should have compared to parameters in your model?

# Principal Component Analysis

## 7.1 Code for loading and cleaning the 2013 national dataset from the [Housing Affordability Data System (HADS)](https://www.huduser.gov/portal/datasets/hads/hads.html) --housing data, can be found below.

## Perform PCA on the processed dataset `national_processed` (Make sure you standardize your data!) and then make a scatterplot of PC1 against PC2. Some of our discussion and work around PCA with this dataset will continue during tomorrow's lecture and assignment.

Not only does this dataset have decent amount columns to begin with (99), but in preparing the data for PCA we have also [one-hot-encoded](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f#targetText=One%20hot%20encoding%20is%20a,the%20entry%20in%20the%20dataset.) all of the categorical variables. This has the effect of creating a new column for each individual category of each categorical variable. After processing this dataset has 64738 columns. --Das a lot of columns.

Don't worry too much about the mechanics of one-hot encoding right now, you will learn and experiment with a whole bunch of categorical encoding approaches in unit 2.

The code below will read in the dataset and perform the one-hot encoding of the categorical variables. Start adding your PCA code at the bottom of the provided code.

```
from urllib.request import urlopen
from zipfile import ZipFile
from io import BytesIO
import os.path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Read Natinal Data
national_url = 'https://www.huduser.gov/portal/datasets/hads/hads2013n_ASCII.zip'
national_file = 'thads2013n.txt'

if os.path.exists(national_file):
    national = pd.read_csv(national_file)
else:
    z_national = urlopen(national_url)
    zip_national = ZipFile(BytesIO(z_national.read())).extract(national_file)
    national = pd.read_csv(zip_national)

print(national.shape)
national.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      '100003130103'
      82
      '3'
      '1'
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      '1'
      1
      40000
      -6
      '1'
      1
      6
      3117.394239
      1
      18021
      '1'
      533
      1
      '1'
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      '1 Adequate'
      '-5'
      '2000-2009'
      '1 Single Family'
      '2 2BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '1 LTE 50% FMR'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '.'
      '2 30% to 50%'
      '-5'
      '-5'

      1
      '100006110249'
      50
      '5'
      '3'
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      '1'
      1
      130000
      -6
      '1'
      1
      6
      2150.725544
      4
      122961
      '1'
      487
      1
      '1'
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '3 GT FMR'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '7 120% AMI +'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      2
      '100006370140'
      53
      '5'
      '3'
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      '1'
      1
      150000
      -6
      '1'
      1
      7
      2213.789404
      2
      27974
      '1'
      1405
      1
      '1'
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '4 60 - 80% AMI'
      '5 80 - 100% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '.'
      '3 50% or More'
      '-5'
      '-5'

      3
      '100006520140'
      67
      '5'
      '3'
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      '1'
      1
      200000
      -6
      '1'
      1
      6
      2364.585097
      2
      32220
      '1'
      279
      1
      '1'
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '3 3BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '6 100 - 120% AMI'
      '7 120% AMI +'
      '7 120% AMI +'
      '5 80 - 100% AMI'
      '4 60 - 80% AMI'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      4
      '100007130148'
      26
      '1'
      '3'
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      '1'
      1
      -6
      -6
      '2'
      100
      4
      2314.524902
      2
      96874
      '1'
      759
      5
      '2'
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      '1 Adequate'
      'Central City'
      '1980-1989'
      '5 50+ units'
      '2 2BR'
      '2 Renter'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '7 120% AMI +'
      '0 Not Assisted'
      '1 Less than 30%'
      '-5'
      '-5'

5 rows × 99 columns

```
# Look at datatypes
# a lot of object datatypes even though they seem to be strings of numbers.
national.dtypes
```

    CONTROL            object
    AGE1                int64
    METRO3             object
    REGION             object
    LMED                int64
                        ...
    FMTINCRELAMICAT    object
    FMTASSISTED        object
    FMTBURDEN          object
    FMTREGION          object
    FMTSTATUS          object
    Length: 99, dtype: object

```
# check for null values
national.isnull().sum().any()
```

    False

```
# check for number of categorical vs numeric columns
cat_cols = national.columns[national.dtypes=='object']
num_cols = national.columns[national.dtypes!='object']

print(f'{len(cat_cols)} categorical columns')
print(f'{len(num_cols)} numerical columns')
```

    32 categorical columns
    67 numerical columns

```
# We're making a copy of our data in case we mess something up.
national_processed = national.copy()

# Categorically Encode our Variables:
# They need to all be numeric before we do PCA.
# https://pbpython.com/categorical-encoding.html

# Cast categorical columns to "category" data type
national_processed[cat_cols] = national_processed[cat_cols].astype('category')

national_processed.dtypes
```

    CONTROL            category
    AGE1                  int64
    METRO3             category
    REGION             category
    LMED                  int64
                         ...
    FMTINCRELAMICAT    category
    FMTASSISTED        category
    FMTBURDEN          category
    FMTREGION          category
    FMTSTATUS          category
    Length: 99, dtype: object

```
# Replace all category cell values with their numeric category codes
for col in cat_cols:
  national_processed[col] = national_processed[col].cat.codes

print(national_processed.shape)
national_processed.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      0
      82
      2
      0
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      0
      1
      40000
      -6
      1
      1
      6
      3117.394239
      1
      18021
      1
      533
      1
      0
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      1
      0
      5
      1
      2
      0
      4
      4
      4
      4
      3
      1
      1
      1
      1
      1
      1
      1
      1
      1
      2
      0
      2
      0
      0

      1
      1
      50
      4
      2
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      0
      1
      130000
      -6
      1
      1
      6
      2150.725544
      4
      122961
      1
      487
      1
      0
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      1
      0
      3
      1
      4
      0
      3
      4
      4
      3
      4
      2
      2
      2
      1
      3
      3
      3
      5
      3
      7
      0
      1
      0
      0

      2
      2
      53
      4
      2
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      0
      1
      150000
      -6
      1
      1
      7
      2213.789404
      2
      27974
      1
      1405
      1
      0
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      1
      0
      3
      1
      4
      0
      4
      4
      4
      4
      3
      2
      2
      2
      1
      2
      3
      4
      5
      3
      4
      0
      3
      0
      0

      3
      3
      67
      4
      2
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      0
      1
      200000
      -6
      1
      1
      6
      2364.585097
      2
      32220
      1
      279
      1
      0
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      1
      0
      3
      1
      3
      0
      4
      4
      4
      4
      4
      2
      2
      2
      2
      2
      5
      6
      6
      4
      4
      0
      1
      0
      0

      4
      4
      26
      0
      2
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      0
      1
      -6
      -6
      2
      100
      4
      2314.524902
      2
      96874
      1
      759
      5
      1
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      1
      1
      3
      5
      2
      1
      3
      3
      3
      3
      4
      2
      2
      2
      2
      3
      2
      2
      2
      2
      7
      1
      1
      0
      0

5 rows × 99 columns

```
# Now we only ahve numeric columns (ints and floats)
national_processed.dtypes
```

    CONTROL            int32
    AGE1               int64
    METRO3              int8
    REGION              int8
    LMED               int64
                       ...
    FMTINCRELAMICAT     int8
    FMTASSISTED         int8
    FMTBURDEN           int8
    FMTREGION           int8
    FMTSTATUS           int8
    Length: 99, dtype: object

```
### Your Code Here
from sklearn.preprocessing import StandardScaler
myscaler = StandardScaler()
```

```
# Standardize the dataset (default is normalization)
national_processed = myscaler.fit_transform(national_processed)
pd.DataFrame(national_processed).shape
```

    (64535, 99)

```
from sklearn.decomposition import PCA
mypca = PCA(n_components = 10)
pca_features = mypca.fit_transform(national_processed)
pd.DataFrame(pca_features).shape
```

    (64535, 10)

```
#  how much variation did each principal component explain?
mypca.explained_variance_ratio_
```

    array([0.34050542, 0.14442164, 0.10287616, 0.07245401, 0.05166581,
           0.04006691, 0.02785838, 0.02220382, 0.02016894, 0.01770104])

```
print(sum(mypca.explained_variance_ratio_)) # variance explained
1 - sum(mypca.explained_variance_ratio_)
```

    0.8399221336739114

    0.16007786632608856

```
plt.scatter(pca_features[:,0], pca_features[:,1], s=10, alpha=.2)
plt.show()
```

```

```

```

```

```
# Standardize the dataset (default is normalization)
national_processed = myscaler.fit_transform(national_processed)
pd.DataFrame(national_processed).shape
```

    (64535, 99)

# Stretch Goals

## 1) Perform further data exploration on the HADS national dataset (the version before we one-hot encoded it) Make scatterplots and see if you can see any resemblance between the original scatterplots and the plot of the principal components that you made in 7.1.

(You may or may not not see very much resemblance depending on the variables you choose, and that's ok!)

## 2) Study "Scree Plots" and then try and make one for your PCA dataset. How many principal conponents do you need to retain in order for your PCs to contain 90% of the explained variance?

We will present this topic formally at the beginning of tomorrow's lecture, so if you figure this stretch goal out, you're ahead of the game.

## 3) Explore further the intuition behind eigenvalues and eigenvectors by creating your very own eigenfaces:

Prioritize self-study over this stretch goal if you are not semi-comfortable with the topics of PCA, Eigenvalues, and Eigenvectors.

You don't necessarily have to use this resource, but this will get you started:
[Eigenface Tutorial](https://sandipanweb.wordpress.com/2018/01/06/eigenfaces-and-a-simple-face-detector-with-pca-svd-in-python/)


### Usha Joshi

# Projectile using class

- **Maximum Height**
\begin{equation}
H=\frac{{u^2}{\sin^2\theta}}{2g}
\end{equation}

- **Horizontal Range**
\begin{equation}
R=\frac{{u^2}{\sin2\theta}}{g}
\end{equation}

- **Time of Flight**
\begin{equation}
T=\frac{{2u}{\sin\theta}}{g}
\end{equation}

```python
import numpy as np                     # from numpy import* not required to write np
import pandas as pd                    # from pandas import*
import json as json
import matplotlib.pyplot as plt        # from matplotlib import*
%matplotlib inline
```

```python
### (a)
class projectile(object):

    def __init__(self, g, u, angle):
        self.pi = np.pi
        self.g = g    # acceleration due to gravity
        self.u = u    # projecting velocity
        self.angle = angle  # projecting angle

    def find_range(self):
        RA = self.u**2*(np.sin(2*self.pi*self.angle/180))/self.g     # horizontal range
        return RA

    def find_height(self):
        MH = self.u**2*(np.sin(self.pi*self.angle/180)**2)/(2*self.g)# maximum height
        return MH

    def find_time(self):
        TF = 2*self.u*np.sin(self.pi*self.angle/180)/self.g          # Time of flight
        return TF
```

```python
P1 =projectile(g=9.8, u=100, angle = 45)
```

```python
P1.g ,P1.u
```

    (9.8, 100)

```python
 P1.angle,P1.find_range(),P1.find_height(),P1.find_time()
```

    (45, 1020.408163265306, 255.10204081632656, 14.430750636460154)

```python
### (b)
class projectile(object):

    def __init__(self, g, u):
        self.pi = np.pi
        self.g = g    # acceleration due to gravity
        self.u = u    # projecting velocity
        #self.angle = angle  # projecting angle

    def find_range(self,angle):
        RA = self.u**2*(np.sin(2*self.pi*angle/180))/self.g     # horizontal range
        return RA

    def find_height(self,angle):
        MH = self.u**2*(np.sin(self.pi*angle/180)**2)/(2*self.g)# maximum height
        return MH

    def find_time(self,angle):
        TF = 2*self.u*np.sin(self.pi*angle/180)/self.g          # Time of flight
        return TF
```

```python
P2 =projectile(g=9.8, u=100)
```

```python
P2.g ,P2.u,P2.find_range(60),P2.find_height(60),P2.find_time(60)
```

    (9.8, 100, 883.699391616774, 382.6530612244897, 17.67398783233548)

```python
g=9.8        # m/s^2
u = 100
angle = []   # list of angle
RA = []      # list of  range
MH = []      # list of max height
TF = []      #list of time of flight
for i  in range(0,90+1,5):

    ra = P2.find_range(i)
    mh = P2.find_height(i)
    tf = P2.find_time(i)
    angle.append(i)
    RA.append(ra)   #add element in list RA
    MH.append(mh)    #add element in list MH
    TF.append(tf)    #add element in list TF
```

```python
#Multiplots/Subplots
plt.figure(figsize=[10,4])
plt.subplot(1,2,1)
plt.plot(angle,RA,'rs--' ,label='Range')
plt.plot(angle,MH ,'g-o', label='Max height')
plt.xlabel('Angle(deg)')
plt.ylabel('Distance(m)')
plt.title('Projectile Motion')
plt.legend()
plt.subplot(1,2,2)
plt.plot(angle,TF,'k^:' ,label='Time of flight')
plt.xlabel('Angle(deg)')
plt.ylabel('Time(sec)')
plt.title('Projectile Motion')
plt.legend()
plt.savefig('plot/projective.png')
#plt.savefig('projective.eps')
#plt.show()
```

```python
data={}    # to save data in dictionary
data.update({"Angle":angle,"Range":RA ,"Max.Height": MH,"Time of flight":TF})
#print(data)
```

```python
df = pd.DataFrame(data)
df.head()
```

      Angle
      Range
      Max.Height
      Time of flight

      0
      0
      0.000000
      0.000000
      0.000000

      1
      5
      177.192018
      3.875573
      1.778689

      2
      10
      349.000146
      15.384536
      3.543840

      3
      15
      510.204082
      34.177193
      5.282021

      4
      20
      655.905724
      59.682540
      6.980003

```python
with open("data/projectile.json", 'w')as f:   #to load data in json file
    json.dump(data,f)
```

```python
# to upload json file
with open("data/projectile.json", 'r')as f:
    uploaded_data = json.load(f)
```

```python
df.to_csv("data/projectile.csv")  #save data in csv format in excel
```

```python
uploaded_data = pd.read_csv("data/projectile.csv")
uploaded_data.head(3)
```

      Unnamed: 0
      Angle
      Range
      Max.Height
      Time of flight

      0
      0
      0
      0.000000
      0.000000
      0.000000

      1
      1
      5
      177.192018
      3.875573
      1.778689

      2
      2
      10
      349.000146
      15.384536
      3.543840

```python

```


---
# How does the covariance matrix $\Sigma$ determine the shape of the gaussian probability distribution function?
---

In this script, we show several examples of 2-D gaussian probability distribution functions (PDF).

In its matrix-form, the equation for the 2-D gausian PDF reads like this:

<blockquote>  $P(\bf{x}) = \frac{1}{2\pi |\Sigma|^{0.5}} \exp{[-\frac{1}{2}(\bf{x}-\bf{\mu})^\top \Sigma^{-1} (\bf{x}-\bf{\mu})]}$  </blockquote>

where

<blockquote>
$
\begin{align}
\bf{x} &= [x_{1} x_{2}]^\top \\
\bf{\mu} &= [\mu_{1} \mu_{2}]^\top  \\
\Sigma &= \begin{pmatrix} \sigma_{x_{1}}^2 & \rho\sigma_{x_{1}}\sigma_{x_{2}} \\ \rho\sigma_{x_{1}}\sigma_{x_{2}} & \sigma_{x_{2}}^2 \end{pmatrix}
\end{align}
$
</blockquote>

and where $\rho$ is the correlation factor between the $x_{1}$ and $x_{2}$ data.

As shown below, the shape of the PDF is entirely determined by the values of the
standard deviations $\sigma_{x_{1}}$, $\sigma_{x_{2}}$ and the correlation factor $\rho$.

```python
print(__doc__)

# Author: Pierre Gravel <pierre.gravel@iid.ulaval.ca>
# License: BSD

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.stats import multivariate_normal

import seaborn as sns
sns.set(color_codes=True)

# Used for reproductibility of the results
np.random.seed(43)
```

    Automatically created module for IPython interactive environment

Let us first define the printing function that we use to show the PDF.

```python
def display_pdf(Sigma, text, ax):
    # The PDF is centered at the origin
    Mu = [0.,0.]

    # Generate a spatial grid where the PDF will be evaluated locally
    x_min, x_max = -2., 2.
    y_min, y_max = -2., 2.
    h = 0.01
    x, y = np.mgrid[x_min:x_max:h, y_min:y_max:h]
    pos = np.dstack((x, y))

    # Generator for the 2-D gaussian PDF
    rv = multivariate_normal(Mu, Sigma)

    # Display 2-D filled contours
    cm = plt.get_cmap('viridis')
    ax.contourf(x, y, rv.pdf(pos),cmap=cm)

    # Display the origin of the PDF
    ax.text(0., 0., '$\mu$', horizontalalignment='center', verticalalignment='center', fontsize=20, color='red')

    # Display the PDF specifications
    props = dict(boxstyle='round', facecolor='wheat', alpha=0.99)
    ax.text(0.05, 0.95, text, transform=ax.transAxes, fontsize=14, verticalalignment='top', bbox=props)

    ax.set_xlabel('$x_{1}$',fontsize=18)
    ax.set_ylabel('$x_{2}$',rotation=0,fontsize=18)

    ax.yaxis.set_label_coords(-0.1,0.5)
    ax.xaxis.set_label_coords(0.5, -0.05)

```

Here are some examples of PDF with isotropy, anisotropy and correlation:

```python
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2,2,figsize=(10,10))

# ISOTROPIC distribution
Sigma = [(1., 0.), (0., 1.)]
text = r'$\sigma_{1} =  \sigma_{2}, \rho = 0$'
display_pdf(Sigma, text, ax1)
ax1.yaxis.set_label_coords(-0.15,0.5)
ax1.xaxis.set_label_coords(0.5, -0.1)

# ANISOTROPIC and UNCORRELATED distribution
Sigma = [(2., 0.), (0., 1.)]
text = r'$\sigma_{1} >  \sigma_{2}, \rho = 0$'
display_pdf(Sigma, text, ax2)
ax2.yaxis.set_label_coords(-0.15,0.5)
ax2.xaxis.set_label_coords(0.5, -0.1)

# ANISOTROPIC and positively CORRELATED distribution
Sigma = [(2., 0.7), (0.7, 1.)]
text = r'$\sigma_{1} >  \sigma_{2}, \rho > 0$'
display_pdf(Sigma, text, ax3)
ax3.yaxis.set_label_coords(-0.15,0.5)
ax3.xaxis.set_label_coords(0.5, -0.1)

# ANISOTROPIC and negatively CORRELATED distribution
Sigma = [(2., -0.7), (-0.7, 1.)]
text = r'$\sigma_{1}  > \sigma_{2}, \rho < 0$'
display_pdf(Sigma, text, ax4)
ax4.yaxis.set_label_coords(-0.15,0.5)
ax4.xaxis.set_label_coords(0.5, -0.1)
fig.tight_layout()

plt.savefig('Examples_of_PDF_isotropy_and_correlation.png')
plt.savefig('Examples_of_PDF_isotropy_and_correlation.pdf')
plt.show()
```

```python

```


```julia
using JuMP, Cbc, Plots
```

# MS-E2121 - Linear optimization
## Exercise session 4

### Demo exercise: Simplex

\begin{align}
    \max \ & 40x_1 + 60x_2\\
    \text{s.t. } & 2x_1 + x_2 && \le 7 \\
    & x_1 + x_2 && \le 4 \\
    & x_1 + 3x_2 && \le 9 \\
    & x_i \geq 0, \ \forall i \in \{1,2\}
\end{align}

We first plot the problem so we can connect our results with the plot. Based on this, the point (0,3) is not optimal, but we will confirm this using reduced costs.

```julia
function plot_model(A, b, c; sol=nothing, xlim=5, ylim=5)

    # Check problem dimensions and throw an error if not 2D
    m,n = size(A)
    if n != 2
        throw(ArgumentError("Model is not 2-dimensional"))
    end

    # Some ranges to help with plotting, these are similar to what you may have used in Matlab
    x1_range = collect(0:0.1:6)
    x2_range = collect(0:0.1:6)

    # Create empty plot
    plot(xlim=(-1,6), ylim=(-1,6), xlabel="x1", ylabel="x2")

    # Add lines Ax=b to the plot
    for con in 1:m
        a_x1 = A[con,1]
        a_x2 = A[con,2]
        if a_x1!=0 && a_x2!=0
            plot!(x1_range, (-a_x1*x1_range.+b[con])/a_x2, label="$(a_x1==1 ? "" : a_x1)x1+$(a_x2==1 ? "" : a_x2)x2=$(b[con])")
        elseif a_x1!=0 && a_x2==0
            vline!([b[con]/a_x1], label="$(a_x1==1 ? "" : a_x1)x1=$(b[con])")
        elseif a_x1==0 && a_x2!=0
            hline!([b[con]/a_x2], label="$(a_x2==1 ? "" : a_x2)x2=$(b[con])")
        end
    end
    vline!([0], label="x1=0")
    hline!([0], label="x2=0")

    # Collect the points in the feasible region using a grid
    feasible_region = []
    for x1 in x1_range
        for x2 in x2_range
            if all(A*[x1;x2].<=b)
                push!(feasible_region, [x1, x2])
            end
        end
    end

    # Turn the array of arrays into a 2D-array and plot the points
    feasible_region = hcat(feasible_region...)
    scatter!(feasible_region[1,:], feasible_region[2,:], ms=1, label=false)

    # If objective function value was given as an argument, plot the objective function with that value
    if !isnothing(sol)
        plot!(x1_range, (-c[1]*x1_range.+sol)/c[2], linestyle=:dash, linewidth=2, color=:black, label="Objective function")
    end

    # This is how you would add a title to your plot
    title!("")
end

A_1 = [2 1; 1 1; 1 3]
b_1 = [7; 4; 9]
c_1 = [-40; -60]
plot_model(A_1,b_1,c_1; sol=-210)
```

## Standard form

We transform the problem to standard form (minimization with equality constraints) in order to use Simplex method.

\begin{align}
    \min \ & -40x_1 - 60x_2\\
    \text{s.t. } & 2x_1 + x_2 + x_3 && = 7 \\
    & x_1 + x_2 + x_4 && = 4 \\
    & x_1 + 3x_2 + x_5 && = 9 \\
    & x_i \geq 0, \ \forall i \in \{1,...,5\}
\end{align}

```julia
# Problem data
A = [2 1 1 0 0;
     1 1 0 1 0;
     1 3 0 0 1]
b = [7; 4; 9]
c = [-40; -60; 0; 0; 0]

x = [0; 3; 4; 1; 0] # Initial solution
B_ind = [2; 3; 4]   # Indices of basic variables
B = A[:,B_ind]      # Basis matrix
c_B = c[B_ind]      # Basic variable costs

p = c_B'*inv(B)     # Dual vector for reduced cost calculation
c_r = c' - p*A      # Reduced costs
println("Reduced costs: $(round.(c_r, digits = 2))")
```

    Reduced costs: [-20.0 0.0 0.0 0.0 20.0]

The reduced cost for $x_1$ is negative => solution is not optimal. A negative reduced cost means that there is a *basic direction* that leads to a feasible point with a smaller (and thus better) objective value. Let's make $x_1$ a basic variable and calculate the corresponding *feasible direction* $d^1 = (d_B^1, d_N^1)$.

Originally, the base and non-base is selected as per $I_B = (2,3,4)$, $I_N = (1,5)$, where $x_1$ is a non-basic variable. $x_1$ enters as a basic variable by being excluded from $I_N$, while in the meantime $I_B$ keeps intact.

The *basic direction* are calculated as $d_B^1 = -B^{-1}A_1$, the component corresponding to the "entering" variable $x_1$ is 1 and the components corresponding to other non-basic variables (here only remains $I_N = (5,)$) are 0. The basic direction thus describes the changes in variables when the entering variable value is increased by 1.

```julia
# Basic direction corresponding to x1
d_B1 = -inv(B)*A[:,1]
d_1 = [1;d_B1;0]
println("Basic direction corresponding to x_1: $(round.(d_1, digits = 2))")
```

    Basic direction corresponding to x_1: [1.0, -0.33, -1.67, -0.67, 0.0]

We can now connect this to the constraints and the figure above. In order to make $x_1$ positive, we move along the constraint $x_1 + 3x_2 \le 9$. For each unit of increase in $x_1$, the other variable $x_2$ must decrease by $\frac{1}{3}$.

Next, we need to determine the step length $\theta$, that is, how far along this direction $d_1$ to move. We want to reach the next vertex, but not go further (the points after the first vertex are infeasible). The corresponding formula is:

\begin{align}
\theta = \min_{i \in I_B, d_i < 0} \{ - \frac{ x_{B(i)} }{ d_{B(i)} } \}
\end{align}

This step will make one basic variable 0, from $I_B = (2,3,4)$, and thus non-basic. That's why we only consider the variables with negative components in the basic direction, the others can't become zero by moving along this direction.

```julia
# Obtain the indices of negative basic direction
I = findall(d_B1 .< -1E-9) # -1e-9 instead of zero because of possible numerical inaccuracies

θ = minimum(-x[B_ind[I]]./d_B1[I])

println("Ratios: $(round.(-x[B_ind[I]]./d_B1[I], digits = 2))")
println("Minimum ratio θ: $(round(θ, digits = 2))")
```

    Ratios: [9.0, 2.4, 1.5]
    Minimum ratio θ: 1.5

The ratios represent the amount by which $x_1$ changes that makes the corresponding basic variables. Here as initialised, $x_B = (x_2, x_3, x_4) = (3,4,1)$. Looking at the ratios, we can see that increasing $x_1$ by 9 would make the first basic variable $x_2$ zero. That also seems to be true, based on the earlier plot.

Among the basic variables $x_i \in x_B$, $x_3$ and $x_4$ are slack variables as initially deviced. Making a slack variable zero = making the corresponding constraint active.

$x_1 = 1.5$ when $x_4=0$, makeing the second constraint of the original problem $x_1 + x_2 \le 4$ active, and that is the minimum ratio we obtained.

```julia
x_new = x + θ*d_1
println("New point x: $(round.(x_new, digits = 2))")
```

    New point x: [1.5, 2.5, 1.5, 0.0, 0.0]

Now that we have a new basic feasible solution, we can check whether it is optimal or not by calculating the reduced costs

```julia
B_ind_new = [2; 3; 1]
B_new = A[:,B_ind_new]
c_B_new = c[B_ind_new]

p_new = c_B_new'*inv(B_new)
c_r_new = c' - p_new*A
println("Reduced costs: $(round.(c_r_new, digits = 2))")
```

    Reduced costs: [0.0 0.0 0.0 30.0 10.0]

All nonnegative, the point $x_{new}$ is optimal
#'
### Student exercise:
Solve the following problem using two-phase simplex

\begin{align*}
    \max \ & 5x_1 + &&x_2 \\
    \text{s.t. } & 2x_1 + &&x_2 &&& \ge 5 \\
    & &&x_2 &&& \ge 1 \\
    & 2x_1 + &&3x_2 &&& \le 12 \\
\end{align*}
\begin{align*}
x_1, x_2 \ge 0, \ \forall i \in \{1,2\}
\end{align*}

In standard form:

\begin{align}
    \min \ & -5x_1 - x_2\\
    \text{s.t. } & 2x_1 + x_2 - x_3 && = 5 \\
    & x_2 - x_4 && = 1 \\
    & 2x_1 + 3x_2 + x_5 && = 12 \\
    & x_i \geq 0, \ \forall i \in \{1,...,5\}
\end{align}

```julia
function plot_model(A, b, c; sol=nothing, xlim=5, ylim=5)

    # Check problem dimensions and throw an error if not 2D
    m,n = size(A)
    if n != 2
        throw(ArgumentError("Model is not 2-dimensional"))
    end

    # Some ranges to help with plotting, these are similar to what you may have used in Matlab
    x1_range = collect(0:0.1:6)
    x2_range = collect(0:0.1:6)

    # Create empty plot
    plot(xlim=(-1,6), ylim=(-1,6), xlabel="x1", ylabel="x2")

    # Add lines Ax=b to the plot
    for con in 1:m
        a_x1 = A[con,1]
        a_x2 = A[con,2]
        if a_x1!=0 && a_x2!=0
            plot!(x1_range, (-a_x1*x1_range.+b[con])/a_x2, label="$(a_x1==1 ? "" : a_x1)x1+$(a_x2==1 ? "" : a_x2)x2=$(b[con])")
        elseif a_x1!=0 && a_x2==0
            vline!([b[con]/a_x1], label="$(a_x1==1 ? "" : a_x1)x1=$(b[con])")
        elseif a_x1==0 && a_x2!=0
            hline!([b[con]/a_x2], label="$(a_x2==1 ? "" : a_x2)x2=$(b[con])")
        end
    end
    vline!([0], label="x1=0")
    hline!([0], label="x2=0")

    # Collect the points in the feasible region using a grid
    feasible_region = []
    for x1 in x1_range
        for x2 in x2_range
            if all(A*[x1;x2].<=b)
                push!(feasible_region, [x1, x2])
            end
        end
    end

    # Turn the array of arrays into a 2D-array and plot the points
    feasible_region = hcat(feasible_region...)
    scatter!(feasible_region[1,:], feasible_region[2,:], ms=1, label=false)

    # If objective function value was given as an argument, plot the objective function with that value
    if !isnothing(sol)
        plot!(x1_range, (-c[1]*x1_range.+sol)/c[2], linestyle=:dash, linewidth=2, color=:black, label="Objective function")
    end

    # This is how you would add a title to your plot
    title!("")
end

A = [-2 -1; 0 -1; 2 3]
b = [-5; -1; 12]
c = [-5; -1]
plot_model(A,b,c; sol=-23.5)
```

### Phase 1
The goal is to obtain a feasible starting point for the second phase.

Because the origin point $(x_1, x_2) = (0, 0)$ lies outside the feasible region of the problem (standard form), a base formed by the three slack variables, being $(0,0,x_3,x_4,x_5)$, is infeasible. In addition, $x_3$ and $x_4$ would need all be negative at the beginning, violating the non-negative constraints.

The above situation leads to solving *auxiliary problem* in phase 1, where simplex method is also employed.

#### The AUX problem:
\begin{align}
    \min \ & x_6 + x_7\\
    \text{s.t. } & 2x_1 + x_2 - x_3 + x_6 && = 5 \\
    & x_2 - x_4 + x_7 && = 1 \\
    & 2x_1 + 3x_2 + x_5 && = 12 \\
    & x_i \geq 0, \ \forall i \in \{1,...,7\}
\end{align}

```julia
# Problem data
A = [2 1 -1 0 0 1 0; 0 1 0 -1 0 0 1; 2 3 0 0 1 0 0]
b = [5; 1; 12]
c = [0; 0; 0; 0; 0; 1; 1]

x = [0; 0; 0; 0; 12; 5; 1] # Initial solution: slack variables equal to RHS
B_ind = [5; 6; 7]   # Indices of basic variables
B = A[:,B_ind]      # Basis matrix
c_B = c[B_ind]      # Basic variable costs

p = c_B'*inv(B)     # Dual vector for reduced cost calculation
c_r = c' - p*A      # Reduced costs
println("Reduced costs: $(round.(c_r, digits = 2))")
```

    Reduced costs: [-2.0 -2.0 1.0 1.0 0.0 0.0 0.0]

```julia
# Round 1

## Rebuild the base
i = 2                   # select x_2 to enter as a basic variable
d_B = -inv(B)*A[:,i]    # calculate the basic direction of changing x_2
d = zeros(7)            # initialisation for a feasible direction
d[B_ind] = d_B          # set the new value of original basic variables resulted from introducing x_2 into the base
d[i] = 1                # set the basic direction of the entering variable x_2 to be 1
println("Basic direction corresponding to introducing x_$(i): $(round.(d, digits = 2))")

## Calculate the moving step length theta
I = findall(d_B .< -1E-9)   # Obtain the indices of negative basic direction
θ = minimum(- x[B_ind[I]] ./ d_B[I])

println("Ratios: $(round.(-x[B_ind[I]]./d_B[I], digits = 2))")
println("Minimum ratio θ: $(round(θ, digits = 2))")

## Set new solution x
x = x + θ*d
println("New point x: $(round.(x, digits = 2))")

## Calculate the new reduced cost
B_ind = findall(x .!= 0.0)
@show B_ind
B = A[:, B_ind]
c_B = c[B_ind]

p = c_B'*inv(B)
c_r = c' - p*A
println("Reduced cost: $(round.(c_r, digits = 2))")

# Round 1 finished

neg_reduced_cost = findall(c_r' .< -1E-9)
if neg_reduced_cost == []
    println("no negative reduced cost, solution x = $(x) is optimal.")
else
    println("indices of x having negative reduced cost: $neg_reduced_cost")
    println("solution not optimal")
end

```

    Basic direction corresponding to introducing x_2: [0.0, 1.0, 0.0, 0.0, -3.0, -1.0, -1.0]
    Ratios: [4.0, 5.0, 1.0]
    Minimum ratio θ: 1.0
    New point x: [0.0, 1.0, 0.0, 0.0, 9.0, 4.0, 0.0]
    B_ind = [2, 5, 6]
    Reduced cost: [-2.0 0.0 1.0 -1.0 0.0 0.0 2.0]
    indices of x having negative reduced cost: [1, 4]
    solution not optimal

```julia
# Round 2

## Rebuild the base
i = 1                   # select x_2 to enter as a basic variable
d_B = -inv(B)*A[:,i]    # calculate the basic direction of changing x_2
d = zeros(7)            # initialisation for a feasible direction
d[B_ind] = d_B          # set the new value of original basic variables resulted from introducing x_2 into the base
d[i] = 1                # set the basic direction of the entering variable x_2 to be 1
println("Basic direction corresponding to introducing x_$(i): $(round.(d, digits = 2))")

## Calculate the moving step length theta
I = findall(d_B .< -1E-9)   # Obtain the indices of negative basic direction
θ = minimum(- x[B_ind[I]] ./ d_B[I])

println("Ratios: $(round.(-x[B_ind[I]]./d_B[I], digits = 2))")
println("Minimum ratio θ: $(round(θ, digits = 2))")

## Set new solution x
x = x + θ*d
println("New point x: $(round.(x, digits = 2))")

## Calculate the new reduced cost
B_ind = findall(x .!= 0.0)
@show B_ind
B = A[:, B_ind]
c_B = c[B_ind]

p = c_B'*inv(B)
c_r = c' - p*A
println("Reduced cost: $(round.(c_r, digits = 2))")

# Round 2 finished

neg_reduced_cost = findall(c_r' .< -1E-9)
if neg_reduced_cost == []
    println("no negative reduced cost, solution x = $(x) is optimal.")
else
    println("indices of x having negative reduced cost: $neg_reduced_cost")
    println("solution not optimal")
end
```

    Basic direction corresponding to introducing x_1: [1.0, 0.0, 0.0, 0.0, -2.0, -2.0, 0.0]
    Ratios: [4.5, 2.0]
    Minimum ratio θ: 2.0
    New point x: [2.5, 0.0, 0.0, 0.0, 7.0, 0.0, 1.0]
    B_ind = [1, 2, 5]
    Reduced cost: [0.0 0.0 0.0 0.0 0.0 1.0 1.0]
    no negative reduced cost, solution x = [2.0, 1.0, 0.0, 0.0, 5.0, 0.0, 0.0] is optimal.

### Phase 2
Starting from the feasible solution obtained from phase 1, find the optimal solution to the original problem (in standard form).

Revisit the original problem in standard form:
\begin{align}
    \min \ & -5x_1 - x_2\\
    \text{s.t. } & 2x_1 + x_2 - x_3 && = 5 \\
    & x_2 - x_4 && = 1 \\
    & 2x_1 + 3x_2 + x_5 && = 12 \\
    & x_i \geq 0, \ \forall i \in \{1,...,5\}
\end{align}

```julia
# Problem data
A = [2 1 -1 0 0; 0 1 0 -1 0; 2 3 0 0 1]
b = [5; 1; 12]
c = [-5; -1; 0; 0; 0]

x = x[1:5]                   # Initial solution: optimal solution of the corresponding AUX problem, excluding artificial variables
B_ind = findall(x .!= 0.0)   # Indices of basic variables
B = A[:,B_ind]               # Basis matrix
c_B = c[B_ind]               # Basic variable costs

p = c_B'*inv(B)     # Dual vector for reduced cost calculation
c_r = c' - p*A      # Reduced costs
println("Reduced costs: $(round.(c_r, digits = 2))")
```

    Reduced costs: [0.0 0.0 0.0 6.5 2.5]

```julia
# Round 1

## Rebuild the base
i = 3                   # select x_2 to enter as a basic variable
d_B = -inv(B)*A[:,i]    # calculate the basic direction of changing x_2
d = zeros(5)            # initialisation for a feasible direction
d[B_ind] = d_B          # set the new value of original basic variables resulted from introducing x_2 into the base
d[i] = 1                # set the basic direction of the entering variable x_2 to be 1
# println("Basic direction corresponding to introducing x_$(i): $(round.(d, digits = 2))")

## Calculate the moving step length theta
I = findall(d_B .< -1E-9)   # Obtain the indices of negative basic direction
θ = minimum(- x[B_ind[I]] ./ d_B[I])

# println("Ratios: $(round.(-x[B_ind[I]]./d_B[I], digits = 2))")
# println("Minimum ratio θ: $(round(θ, digits = 2))")

## Set new solution x
x = x + θ*d
println("New point x: $(round.(x, digits = 2))")

## Calculate the new reduced cost
B_ind = findall(x .!= 0.0)
# @show B_ind
B = A[:, B_ind]
c_B = c[B_ind]

p = c_B'*inv(B)
c_r = c' - p*A
println("Reduced cost: $(round.(c_r, digits = 2))")

# Round 1 finished

neg_reduced_cost = findall(c_r' .< -1E-9)
if neg_reduced_cost == []
    println("No negative reduced cost!")
    println("Optimal solution $(sum(-c .* x)) at x = $(x[1:2]).")
else
    println("indices of x having negative reduced cost: $neg_reduced_cost")
    println("solution not optimal")
end
```

    New point x: [4.5, 1.0, 20.0, 0.0, 0.0]
    Reduced cost: [0.0 0.0 0.0 6.5 2.5]
    No negative reduced cost!
    Optimal solution 23.5 at x = [4.5, 1.0].


# Differentiation example
This demonstrates solving the problem I found [here](http://math.stackexchange.com/questions/221197/a-tough-differential-calculus-problem), using the *SymEngine* gem.

---

The function is described as
$$f_p(x) = \dfrac{9\sqrt{x^2+p}}{x^2+2}$$
We need to differentiate w.r.t. $x$

One solution is:
$$f_p^2(x) (x^2 + 2)^2 = 81(x^2 + p) \quad\Longrightarrow$$

$$2 f_p(x) f_p'(x)(x^2 + 2)^2 + 4x f_p^2(x^2 +2) = 162x \quad\Longrightarrow$$

\begin{align}f_p'(x) &= \frac{162x - 4x f_p^2(x)(x^2+2)}{2f_p(x)(x^2+2)^2}\\ \\ &= \frac{162x - 324x\frac{x^2 + p}{x^2 +2}}{18(x^2 +2)\sqrt{x^2+p}}\\ \\ &= \frac{9 x (x^2+2) - 18x(x^2+p)}{(x^2 + 2)^2 \sqrt{x^2 + p}}\\ \\ &= \frac{9x(2-2p-x^2)}{(x^2 + 2)^2\sqrt{x^2 + p}}\end{align}

We will attempt to solve this using features in *SymEngine*

---

```ruby
require 'symengine'
```

    true

Declare the symbols

```ruby
x = SymEngine::Symbol.new('x')
p = SymEngine::Symbol.new('p')
half = 1.quo(2)
```

    (1/2)

Create the expression

```ruby
fp = 9*((x**2 + p)**half)/((x**2)+2)
fp.to_s
```

    "9*(p + x**2)**(1/2)/(2 + x**2)"

Differentiate wrt $x$

```ruby
answer = fp.diff(x)
answer.to_s
```

    "-18*x*(p + x**2)**(1/2)/(2 + x**2)**2 + 9*x/((2 + x**2)*(p + x**2)**(1/2))"

Which is indeed correct! If you simplify the second last answer that we got from our solution. You will find the exact same answer. Using *SymEngine* for solving problems is as simple as that.


# 附录 D 变分法

函数 $y(x)$ 可以看成一种操作符，即对于任意 $x$，返回一个输出 $y$。在这种情况下，我们可以定义泛函 $F[y]$：接受一个函数 $y(x)$，返回输出值 $F$。

机器学习中一个常见的例子熵 $H[x]$，因为对于任何一个概率分布 $p(x)$，它返回了 $x$ 在该分布下的熵，因此 $p(x)$ 的熵也可以写作 $H[p]$。

微积分中一个常见的问题是找到一个 $x$ 最大（小）化一个函数 $y(x)$，而变分法要解决的问题是找到一个 $y(x)$ 最大（小）化一个泛函 $F[y]$。

例如变分法可以解释为什么两点之间直线最短，或者最大熵分布是高斯分布。

传统的微积分给出：

$$
y(x+\epsilon) = y(x) + \frac{dy}{dx} \epsilon + O(\epsilon^2)
$$

以及

$$
y(x_1+\epsilon_1, \dots, x_D+\epsilon_D) = y(x_1, \dots, x_D) + \sum_{i=1}^D \frac{\partial y}{\partial x} \epsilon_i + O(\epsilon^2)
$$

类似的，泛函求导时，我们考虑一个 $y(x)$ 的小变化 $\epsilon\eta(x)$，（$\eta(x)$ 是关于 $x$ 的任意函数），记泛函 $F[y]$ 对函数 $y(x)$ 的导数为 $\frac{\delta F[y]}{\delta y(x)}$，定义：

$$
F[y(x)+\epsilon\eta(x)] = F[y(x)] + \epsilon \int \frac{\delta F[y]}{\delta y(x)} \eta(x) dx + O(\epsilon^2)
$$

泛函达到驻点的条件为，对任意 $\eta(x)$，有：

$$
\int \frac{\delta F[y]}{\delta y(x)} \eta(x) dx = 0
$$

要对任意 $\eta(x)$ 成立，$\frac{\delta F[y]}{\delta y(x)}$ 必须为 0。

考虑这样一个泛函：

$$
F[y] = \int G(y(x), y'(x), x) dx
$$

其中，$y'(x)$ 是 $y(x)$ 关于 $x$ 的导数，**$y(x)$ 在积分区间的边界的值是固定的**。考虑 $y(x)$ 的变分，展开之后有：

$$
\begin{align}
F[y(x)+\epsilon\eta(x)]
& = \int G(y+\epsilon\eta, y'+\epsilon\eta', x) dx \\
& = \int G(y, y', x) dx + \epsilon \int
\left\{ \frac{\partial G}{\partial y} \eta(x) +
\frac{\partial G}{\partial y'} \eta'(x)\right\} dx + O(\epsilon^2) \\
& = F[y(x)] + \epsilon \int
\left\{ \frac{\partial G}{\partial y} \eta(x) +
\frac{\partial G}{\partial y'} \eta'(x)\right\} dx + O(\epsilon^2) \\
\end{align}
$$

为了得到定义的形式，对积分的第二部分进行分部积分：

$$
\int \frac{\partial G}{\partial y'} \eta'(x) dx = \left.\frac{\partial G}{\partial y'} \eta(x) \right|_x - \int \frac{d}{dx} \left(\frac{\partial G}{\partial y'}\right) \eta(x) dx
$$

考虑到 $y(x)$ 在边界上的值是固定的，因此在边界上 $\eta(x)$ 必须为 0，于是我们有

$$
F[y(x)+\epsilon\eta(x)]
= F[y(x)] + \epsilon \int
\left\{ \frac{\partial G}{\partial y} -
\frac{d}{dx} \left(\frac{\partial G}{\partial y'}\right) \right\}\eta(x) dx + O(\epsilon^2) \\
$$

令变分为 0，我们有

$$
\frac{\partial G}{\partial y} -
\frac{d}{dx} \left(\frac{\partial G}{\partial y'}\right) = 0
$$

这就是著名的欧拉-拉格朗日方程（`Euler-Lagrange equation`）。

例如 $G=y(x)^2+\left(y'(x)\right)^2$，欧拉-拉格朗日方程为：

$$
y(x) - \frac{d^2y}{dx^2} = 0
$$

利用边界条件，我们可以解出这个方程。

在很多问题中，如果我们的被积函数为 $G(y,x)$，那么驻点的要求为

$$
\frac{\partial G}{\partial y(x)}= 0
$$


# Neural Networks Summer 2019

This notebook seeks to summarize and document the content learned during the Summer 2019 S-STEM summer reasearch program.

| Week 1 | [Week 2](#week2) | [Week 3](#week3) |
|---| --- | --- |
| [Linear Algebra](#linear-algebra) | [Multilayer Nets](#multilayer-networks)| [Standardization of data](#standardization) |
| [PCA](#PCA) | [Reinforcement Learning](#reinforcement)| [Inductive Bias](#inductive) |
| [Single Layer Neural Nets](#single-layer-neural-networks) | | |

---

## Week 1
During the first week, we covered introductory topics needed to get started in the area of neural networks. We began by learning to use Jupyter notebooks, followed by an intro to LaTeX and Markdown. These tools will be used throughout the program to help document our code and processes.

In addition we covered some python libraries that will be useful, such as Matplotlib and numpy. Towards the end of the week, we began learning about the basics of neural networks, starting with tools like Keras and TensorFlow. We used these to practice with a single layer neural network and started learning about multilayer networks.

#### Linear Algebra
##### Distance and Similarity
When speaking in terms of neural networks, it's often useful to describe something as similar or disimilar to a class of things. For this, it is helpful to have some mathmatical methods of difining this similarity or difference. A common distance metric is the **Euclidean distance**: $\sqrt{\sum_{i=1}^{n}{(\boldsymbol{x}_i - \boldsymbol{y}_i)^2}}$ . Let's take a peak at that using numpy. Notice how easy numpy arrays make it.

```python
import numpy as np

X = np.array([5.0, 10.0])
Y = np.array([1.0, 8.0])

np.sqrt(np.sum(pow(X-Y, 2.0)))
```

    4.47213595499958

```python
# This can also be done using scipy's euclidean method
import scipy.spatial.distance as ssd

ssd.euclidean(X, Y)
```

    4.47213595499958

A common metric used for similarity is the **cosine similarity** function: $\cos {\theta} = \frac{\boldsymbol{x} \cdot \boldsymbol{y}}
                      {\lVert \boldsymbol{x} \rVert_2
                       \lVert \boldsymbol{y} \rVert_2}$
We can calculate that as well using numpy:

```python
# Calculate the cosine similarity
np.sum(X*Y) / (np.sqrt(np.dot(X,X)) * np.sqrt(np.dot(Y,Y)))
```

    0.9429903335828895

There is also a **cosine dissimilarity** function derived from the similarity function. It looks like so: $\cos {\theta} = 1 - \frac{\boldsymbol{x} \cdot \boldsymbol{y}}
                      {\lVert \boldsymbol{x} \rVert_2
                       \lVert \boldsymbol{y} \rVert_2}$
This can be calculated by using numpy, or easily with the scipy method:

```python
ssd.cosine(X, Y)
```

    0.05700966641711047

##### Matrices
It is also important to become familiar with matrix operations, as they are an integral piece of neural networks. Luckily, numpy also makes this quite easy.

```python
# For displaying matrices
from sympy import *
init_printing(use_latex=True)

X = np.array([5.0, 10.0])
Y = np.array([1.0, 8.0])
Z = np.array([2.0, 4.5])
# Create the matrix
data = np.array([X, Y, Z])
Matrix(data)
```

$$\left[\begin{matrix}5.0 & 10.0\\1.0 & 8.0\\2.0 & 4.5\end{matrix}\right]$$

An important concept with matrices is **pairwise distance**. This involves calculating the distance between each of the vectors in a given matrices. Let's take a look at how this is done using `pdist()`:

```python
ssd.squareform(ssd.pdist(data, metric='euclidean'))
```

    array([[0.        , 4.47213595, 6.26498204],
           [4.47213595, 0.        , 3.64005494],
           [6.26498204, 3.64005494, 0.        ]])

Yet another important skill with matrices is decomposition, where $\boldsymbol{A} = \boldsymbol{U} \boldsymbol{\Sigma} \boldsymbol{V}^\intercal$. Numpy makes this easy as well. Using `np.linalg.svd()` we will decompose a matrix into singular, left-singular, and right-singular values.

```python
U, S, V = np.linalg.svd(data, full_matrices=True)
Mul(Matrix(U), Matrix(np.diag(S)), Matrix(V), evaluate=False)
```

$$\left[\begin{matrix}-0.768489058813465 & 0.531109622575895 & -0.35685730382225\\-0.542199584547175 & -0.836660812335674 & -0.0775776747439673\\-0.339770771257426 & 0.13387028762612 & 0.930932096927608\end{matrix}\right] \left[\begin{matrix}14.46678736452 & 0.0\\0.0 & 2.22756893266284\end{matrix}\right] \left[\begin{matrix}-0.350056048625512 & -0.936728756268694\\0.936728756268694 & -0.350056048625512\end{matrix}\right]$$

#### PCA (Principal Component Analysis)
PCA is an extremely useful tool for neural networks, and is often one of the first things done when starting to analyze the data. It's a great way to get rid of noise in the data and prep it for being used by the neural net. The basic principal is to eliminate uneeded dimensions in the data (compression and noise reduction) and to project the data into a manner it can be vizualized in.

Let's take a look at a typical PCA workflow:

```python
import pandas
import numpy as np
from sympy import *
init_printing(use_latex=True)
from sklearn.decomposition import PCA
import keras
import matplotlib.pyplot as plt
%matplotlib inline

iris_data = np.array(pandas.read_table("https://www.cs.mtsu.edu/~jphillips/courses/CSCI4850-5850/public/iris-data.txt",
                                       delim_whitespace=True,
                                       header=None))
# Separate into the data and class labels
X = iris_data[:,0:4] # 0,1,2,3
Y = iris_data[:,4]   # 4

# Mean center the data
def mean_center(x):
    return x - np.mean(x)
Xcentered = np.apply_along_axis(mean_center, 0, X)

# Decomp
U, S, V = np.linalg.svd(Xcentered, full_matrices=True)

# How much varience do the first two principal components account for?
print((100 * (S[0] + S[1]))/np.sum(S))
```

    85.4490160873562

We can see that just the first two components account for over 85 percent of the varience. This is an excellent case for PCA. The singular values will tell us which dimensions captures the most amount of variance. This dimension will then be put along the x-axis. The second principal component will be placed along the y-axis.

```python
# Rotate and remove uneeded dimensions
D = np.zeros([X.shape[0], X.shape[1]])
np.fill_diagonal(D, S)
Xrotated = np.dot(U, D)
PCs = Xrotated[:,0:2]

plt.scatter(PCs[:,0],PCs[:,1],
            color=[['red','green','blue'][i] for i in Y.astype(int)])
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.show()
```

#### Single Layer Neural Networks
Creating the neural network requires four steps:
1. Declare which model you'd like to use. In this case _`keras.Sequential()`_
2. Add the output layer
3. Compile the network
4. Train the network (using _`fit()`_)

Example below:

```python
import pandas
import numpy as np
import keras

data = np.array(pandas.read_table("https://www.cs.mtsu.edu/~jphillips/courses/CSCI4850-5850/public/iris-data.txt",
                                  delim_whitespace=True,
                                  header=None))
X = data[:,0:4]

labels = data[:,4]
Y = keras.utils.to_categorical(labels,
                               len(np.unique(labels)))

model = keras.Sequential()
# Input size - 4
input_size = X.shape[1]
# Output size - 3
output_size = Y.shape[1]

model.add(keras.layers.Dense(output_size,
                             activation='sigmoid',
                             input_shape=[input_size]))
model.compile(loss=keras.losses.mse,
              optimizer=keras.optimizers.SGD(lr=0.7),
              metrics=['accuracy'])

batch_size = 16
epochs = 500
validation_split = 0.5

history = model.fit(X, Y,
          batch_size = batch_size,
          epochs = epochs,
          verbose = 0,
          validation_split = validation_split)

# Plot Results
import matplotlib.pyplot as plt
%matplotlib inline

plt.figure(1)

# summarize history for accuracy

plt.subplot(211)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')

# summarize history for loss

plt.subplot(212)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')

plt.tight_layout()
plt.show()
```

```python
score = model.evaluate(X, Y, verbose=1)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

    150/150 [==============================] - 0s 43us/step
    Test loss: 0.06692401697238286
    Test accuracy: 0.9533333309491475

---

## Week 2

During this week, we began to dive _deeper_ into mulitlayer neural networks...

#### Multilayer Networks
##### Activation Function and Loss Function Pairings
There is often a pairing of what activation and loss functions one should be using for a given neural network. They can be broken down into two categories. Regression and classification, where regression involves a singular output of continuous values, and classification entails categorizing something using one or more discrete categories. The classification category can be further broken down into binary classification or multiclass classification. There is often an activation function and loss function that lend themselves particularly well to a specific one of these categories.

| Category | Activation Function | Loss Function |
| --- | --- | ---|
| **Regression**| Linear $g(net_i) = net_i$ | Mean Sum Error _(Usually)_|
| **Binary Classification** | Logistic Sigmoid | Binary Cross-Entropy |
| **Multiclass Classification** | Softmax | Categorical Cross-Entropy |

#### Reinforcement Learning
The networks we have worked on up until this point involved supervised learning. Upon each pass through the neural net, the network was not only shown whether it was right or wrong, but also what the answer should have been. _Reinforcement Learning_ works in a different manner. There is an __agent__ which is placed into an environment, trying to reach a goal. Only upon reaching this goal does it receive feedback whether it was right or wrong. Therefore, there is a _difference in time_ of when the agent receives feedback. The agent must then figure out which actions it made were correct. This is known as __TD Learning__. See the example program below:

The following program demonstrates _Temporal Difference Learning_. We begin with a 1D maze which contains a single goal. Each section of the maze can be thought of as a __state__. We set the reward for the goal to one, and the reward for all other states to zero. We then drop the __agent__ into a random state, and let it search for the goal. Each time the goal is found the __value__ of the states used to get there will update to show the most efficient path.

Epoch is set to the amount of times the agent will be dropped into a state to find the goal. Goal is set to which state in the maze represents the goal.

Try out adjusting the length of the 1D Maze:

```python
import random
from fractions import Fraction
import numpy as np

LENGTH = 14  # Try me out
GAMMA = .5
GOAL = 0
EPOCH = 100

class Maze:

    def __init__(self):
        self._reward = np.zeros(LENGTH)  # Reward Vector
        self._value = np.zeros(LENGTH) #np.ones(LENGTH, dtype='int')         # Value Vector
        self._reward[GOAL] = 1
        self.final_value = np.zeros(LENGTH) # Vector NOT used in learning.
                                            # Soley used to see how many episodes required to train

        # fill self._finalvalue
        for i in range(LENGTH):
            distance = np.abs(GOAL-i)
            # account for wraping
            if ((np.abs(GOAL + LENGTH) - i) < distance):
                distance = np.abs(GOAL + LENGTH) - i
            self.final_value[i] = GAMMA ** distance

    # Disply the maze with current values inside
    def display(self):
        print(" ------" * LENGTH)

        print("|", end='')
        for i in range(LENGTH):
            print(" %4s |" %Fraction(self._value[i]), end='')
        print()
        print(" ------" * LENGTH)
        for i in range(LENGTH):
            print("  ", i, "  ", end='')
        print()
        return

    # Drop agent into maze and search for goal
    def episode(self, s):
        # Search and update value until reaches goal
        while(s != GOAL):
            self._value[s] += self.delta(s)
            # Move to next state
            s = self._nextS(s)
        self._value[s] += self.delta(s)

        return

    # delta(s) = (r(s) + gamma v(s + 1)) - v(s)
    def delta(self, s):
        if (s == GOAL):
            future_val = 0.0
        else:
            future_val = GAMMA * self.v(self._nextS(s))
        term = self._reward[s] + future_val
        return term - self._value[s]

    # V(s) = v(s) + gamma V(s+1)
    def v(self, s):
        if (s == GOAL):
            return self._value[GOAL]
        else:
            return self._reward[s] + (GAMMA * self._value[self._nextS(s)])

    # Obtain the next state to  be moved to
    def _nextS(self, s):
        # determine left and right values (accounting for wrap around)
        if (s == 0):
            left = (LENGTH-1)
            right = s + 1
        elif (s == (LENGTH-1)):
            left = s - 1
            right = 0
        else:
            left = s - 1
            right = s + 1

        # determine whether to go left or right
        if (self._value[left] >= self._value[right]):
            nextS = left
        else:
            nextS = right

        return nextS

    # Obtain the matrix of state values
    def get_values(self):
        return self._value
```

```python
import numpy as np
import time
import matplotlib.pyplot as plt
%matplotlib inline

maze = Maze()
print("The goal is", GOAL)
maze.display()

done = False
i = 0
while (not done):
    i += 1
    s = random.randint(0, LENGTH-1)
    maze.episode(s)
    if (np.array_equal(maze.get_values(), maze.final_value)):
        done = True

print("Maze after completion:")
maze.display()

print("Elapsed episodes:", i)

# Plot the results
v = maze.get_values()
states = np.linspace(0, LENGTH-1, LENGTH)
plt.plot(states, v)
plt.xlabel("States")
plt.ylabel("V(s)")
plt.show()
```

# Week 3

##### Standardization of Data
An extra technique that may be helpful to training neural nets is standardization of data. This helps "level the playing field" so to speak to give each of the data points an equal chance of affecting the weights. The formula for standardization is like so:

$z = \frac{x - \mu}{\sigma}$ where $\mu$ is the mean and $\sigma$ is the standard diviation.

##### Inductive Bias
We continued to learn about neural networks and reinforcement learning. Specifically, we learned about the concept of *inductive bias*. Inductive bias is tuning a network to more specifically fit a problem. This often makes the network lend itself more readily to a problem domain. For instance, *working memory* is an inductive bias for the problem domain of reinforcement learning. Other inductive biases exist as well. Convolutional neural networks work better for working with images, such as image recognition and classification. They provide a special modification to the neural network, that aides them in this process. By only looking at a specific region of an image and then "sliding" across it, it is able to better recognize images.

# WEEK 4

This week, we became more specialized in learning, looking more specifically at reinforcement learning and working memory.

```python

```