# Vertical Line Test

```
import matplotlib.pyplot as plt
import numpy as np
```

## 1.1 Create two graphs, one that passes the vertical line test and one that does not.

```
plt.axhline(y=2)
plt.title("passes the vertical line test")
plt.show()
```

```
plt.axvline(x=2)
plt.title("fails the vertical line test")
plt.show()
```

## 1.2 Why are graphs that don't pass the vertical line test not considered "functions?"

A function cannot have the same input (x value/domain value) mapped to multiple outputs (y value/co-domain value).

# Functions as Relations

## 2.1 Which of the following relations are functions? Why?

\begin{align}
\text{Relation 1: } \{(1, 2), (3, 2), (1, 3)\}
\\
\text{Relation 2: } \{(1, 3), (2, 3), (6, 7)\}
\\
\text{Relation 3: } \{(9, 4), (2, 1), (9, 6)\}
\\
\text{Relation 4: } \{(6, 2), (8, 3), (6, 4)\}
\\
\text{Relation 5: } \{(2, 6), (2, 7), (2, 4)\}
\end{align}

Relations 2 is the only one here that is a function because there are not repeated x values. In other words, there are no two different y values that are mapped to by the same x value.

# Functions as a mapping between dimensions

## 3.1 for the following functions what is the dimensionality of the domain (input) and codomain (range/output)?

\begin{align}
m(𝑥_1,𝑥_2,𝑥_3)=(x_1+x_2, x_1+x_3, x_2+x_3)
\\
n(𝑥_1,𝑥_2,𝑥_3,𝑥_4)=(x_2^2 + x_3, x_2x_4)
\end{align}

function m: 3D -> 3D

function n: 4D -> 2D

## 3.2 Do you think it's possible to create a function that maps from a lower dimensional space to a higher dimensional space? If so, provide an example.

Yes this is possible. For more information on functions from lower dimensions of space that map to higher dimensions of space, google the terms injective and surjective in relation to Linear Transformations:

<https://youtu.be/xKNX8BUWR0g>

Example:

$f(x) = (x, x+1)$

# Vector Transformations

## 4.1 Plug the corresponding unit vectors into each function. Use the output vectors to create a transformation matrix.

\begin{align}
p(\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}
\\
\\
q(\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}
\end{align}

---

\begin{align}
p(\begin{bmatrix} 1 \\ 0 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\ -x_1+ 2x_2 \\  \end{bmatrix} = \begin{bmatrix} 1 \\ -1 \end{bmatrix}
\end{align}

\begin{align}
p(\begin{bmatrix} 0 \\ 1 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\ -x_1+ 2x_2 \\  \end{bmatrix} = \begin{bmatrix} 3 \\ 2 \end{bmatrix}
\end{align}

\begin{align}
T = \begin{bmatrix} 1 & 3 \\ -1 & 2 \end{bmatrix}
\end{align}

---

\begin{align}
q(\begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\ -x_1 + 2x_2  + 3x_3 \\ 5x_1 + x_2 - 2x_3  \end{bmatrix} = \begin{bmatrix} 4 \\ -1 \\ 5\end{bmatrix}
\end{align}

\begin{align}
q(\begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\ -x_1 + 2x_2  + 3x_3 \\ 5x_1 + x_2 - 2x_3  \end{bmatrix}= \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}
\end{align}

\begin{align}
q(\begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\ -x_1 + 2x_2  + 3x_3 \\ 5x_1 + x_2 - 2x_3  \end{bmatrix}= \begin{bmatrix} 2 \\ 3 \\ -2\end{bmatrix}
\end{align}

\begin{align}
T = \begin{bmatrix} 4 & 1 & 2 \\ -1 & 2 & 3 \\ 5 & 1 & -2  \end{bmatrix}
\end{align}

---

## 4.2 Verify that your transformation matrices are correct by choosing an input matrix and calculating the result both via the traditional functions above and also via vector-matrix multiplication.

```
# Transformation Matrix
T1 = np.array([[1,3],[-1,2]])
# Input Vector
v1 = np.array([2,3])

# Product found by hand-calculating
product_by_hand_1 = [11, 4]

# Product found with NumPy
np.matmul(T1, v1)
```

    array([11,  4])

```
# Transformation Matrix
T2 = np.array([[4,1,2],[-1,2,3],[5,1,-2]])
# Input Vector
v2 = np.array([1,2,3])

# Product found by hand-calculating
product_by_hand_2 = [12, 12, 1]

# Product found with NumPy
np.matmul(T2, v2)
```

    array([12, 12,  1])

# Eigenvalues and Eigenvectors

## 5.1 In your own words, give an explanation for the intuition behind eigenvalues and eigenvectors.

An eigenvector is a vector that does not change orientation (direction in which it is pointing) during a linear transformation. An eigenvector always comes paired with a corresponding eigenvalue. Every linear transformation has at least one eigenvector-eigenvalue pair. The eigenvalue describes how much the eigenvector gets scaled by (stretched or squished) during the transformation.

# The Curse of Dimensionality

## 6.1 What are some of the challenges of working with high dimensional spaces?

- Increased computational inefficiency (searching)
- Increased data redundancy with increasing number of columns.
- More difficult to visualize/explore the data.
- Measures of euclidean distance breaks down in high-dimensional spaces
- Increased sparcity of data.
- As the number of parameters in a model grows while number of available observations remains fixed, overfitting a predictive model becomes more certain.

## 6.2 What is the rule of thumb for how many observations you should have compared to parameters in your model?

You should have 5x as many observations (rows) as you do parameters (columns) in a machine learning model.

# Principal Component Analysis

## 7.1 Code for loading and cleaning the 2013 national dataset from the [Housing Affordability Data System (HADS)](https://www.huduser.gov/portal/datasets/hads/hads.html) --housing data, can be found below.

## Perform PCA on the processed dataset `national_processed` (Make sure you standardize your data!) and then make a scatterplot of PC1 against PC2. Some of our discussion and work around PCA with this dataset will continue during tomorrow's lecture and assignment.

The code below will read in the dataset and perform categorical encoding of the categorical variables to ensure that we're only working with numeric columns from our dataset. Start adding your PCA code at the bottom of the provided code.

```
from urllib.request import urlopen
from zipfile import ZipFile
from io import BytesIO
import os.path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Read Natinal Data
national_url = 'https://www.huduser.gov/portal/datasets/hads/hads2013n_ASCII.zip'
national_file = 'thads2013n.txt'

if os.path.exists(national_file):
    national = pd.read_csv(national_file)
else:
    z_national = urlopen(national_url)
    zip_national = ZipFile(BytesIO(z_national.read())).extract(national_file)
    national = pd.read_csv(zip_national)

print(national.shape)
national.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      '100003130103'
      82
      '3'
      '1'
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      '1'
      1
      40000
      -6
      '1'
      1
      6
      3117.394239
      1
      18021
      '1'
      533
      1
      '1'
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      '1 Adequate'
      '-5'
      '2000-2009'
      '1 Single Family'
      '2 2BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '1 LTE 50% FMR'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '.'
      '2 30% to 50%'
      '-5'
      '-5'

      1
      '100006110249'
      50
      '5'
      '3'
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      '1'
      1
      130000
      -6
      '1'
      1
      6
      2150.725544
      4
      122961
      '1'
      487
      1
      '1'
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '3 GT FMR'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '7 120% AMI +'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      2
      '100006370140'
      53
      '5'
      '3'
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      '1'
      1
      150000
      -6
      '1'
      1
      7
      2213.789404
      2
      27974
      '1'
      1405
      1
      '1'
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '4 60 - 80% AMI'
      '5 80 - 100% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '.'
      '3 50% or More'
      '-5'
      '-5'

      3
      '100006520140'
      67
      '5'
      '3'
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      '1'
      1
      200000
      -6
      '1'
      1
      6
      2364.585097
      2
      32220
      '1'
      279
      1
      '1'
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '3 3BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '6 100 - 120% AMI'
      '7 120% AMI +'
      '7 120% AMI +'
      '5 80 - 100% AMI'
      '4 60 - 80% AMI'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      4
      '100007130148'
      26
      '1'
      '3'
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      '1'
      1
      -6
      -6
      '2'
      100
      4
      2314.524902
      2
      96874
      '1'
      759
      5
      '2'
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      '1 Adequate'
      'Central City'
      '1980-1989'
      '5 50+ units'
      '2 2BR'
      '2 Renter'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '7 120% AMI +'
      '0 Not Assisted'
      '1 Less than 30%'
      '-5'
      '-5'

5 rows × 99 columns

```
# Look at datatypes
# a lot of object datatypes even though they seem to be strings of numbers.
national.dtypes
```

    CONTROL                 object
    AGE1                     int64
    METRO3                  object
    REGION                  object
    LMED                     int64
    FMR                      int64
    L30                      int64
    L50                      int64
    L80                      int64
    IPOV                     int64
    BEDRMS                   int64
    BUILT                    int64
    STATUS                  object
    TYPE                     int64
    VALUE                    int64
    VACANCY                  int64
    TENURE                  object
    NUNITS                   int64
    ROOMS                    int64
    WEIGHT                 float64
    PER                      int64
    ZINC2                    int64
    ZADEQ                   object
    ZSMHC                    int64
    STRUCTURETYPE            int64
    OWNRENT                 object
    UTILITY                float64
    OTHERCOST              float64
    COST06                 float64
    COST12                 float64
                            ...
    COSTMedRELAMICAT         int64
    COSTMedRELPOVPCT       float64
    COSTMedRELPOVCAT         int64
    COSTMedRELFMRPCT       float64
    COSTMedRELFMRCAT         int64
    FMTZADEQ                object
    FMTMETRO3               object
    FMTBUILT                object
    FMTSTRUCTURETYPE        object
    FMTBEDRMS               object
    FMTOWNRENT              object
    FMTCOST06RELPOVCAT      object
    FMTCOST08RELPOVCAT      object
    FMTCOST12RELPOVCAT      object
    FMTCOSTMEDRELPOVCAT     object
    FMTINCRELPOVCAT         object
    FMTCOST06RELFMRCAT      object
    FMTCOST08RELFMRCAT      object
    FMTCOST12RELFMRCAT      object
    FMTCOSTMEDRELFMRCAT     object
    FMTINCRELFMRCAT         object
    FMTCOST06RELAMICAT      object
    FMTCOST08RELAMICAT      object
    FMTCOST12RELAMICAT      object
    FMTCOSTMEDRELAMICAT     object
    FMTINCRELAMICAT         object
    FMTASSISTED             object
    FMTBURDEN               object
    FMTREGION               object
    FMTSTATUS               object
    Length: 99, dtype: object

```
# check for null values
national.isnull().sum().any()
```

    False

```
# check for number of categorical vs numeric columns
cat_cols = national.columns[national.dtypes=='object']
num_cols = national.columns[national.dtypes!='object']

print(f'{len(cat_cols)} categorical columns')
print(f'{len(num_cols)} numerical columns')
```

    32 categorical columns
    67 numerical columns

```
# We're making a copy of our data in case we mess something up.
national_processed = national.copy()

# Categorically Encode our Variables:
# They need to all be numeric before we do PCA.
# https://pbpython.com/categorical-encoding.html

# Cast categorical columns to "category" data type
national_processed[cat_cols] = national_processed[cat_cols].astype('category')

national_processed.dtypes
```

    CONTROL                category
    AGE1                      int64
    METRO3                 category
    REGION                 category
    LMED                      int64
    FMR                       int64
    L30                       int64
    L50                       int64
    L80                       int64
    IPOV                      int64
    BEDRMS                    int64
    BUILT                     int64
    STATUS                 category
    TYPE                      int64
    VALUE                     int64
    VACANCY                   int64
    TENURE                 category
    NUNITS                    int64
    ROOMS                     int64
    WEIGHT                  float64
    PER                       int64
    ZINC2                     int64
    ZADEQ                  category
    ZSMHC                     int64
    STRUCTURETYPE             int64
    OWNRENT                category
    UTILITY                 float64
    OTHERCOST               float64
    COST06                  float64
    COST12                  float64
                             ...
    COSTMedRELAMICAT          int64
    COSTMedRELPOVPCT        float64
    COSTMedRELPOVCAT          int64
    COSTMedRELFMRPCT        float64
    COSTMedRELFMRCAT          int64
    FMTZADEQ               category
    FMTMETRO3              category
    FMTBUILT               category
    FMTSTRUCTURETYPE       category
    FMTBEDRMS              category
    FMTOWNRENT             category
    FMTCOST06RELPOVCAT     category
    FMTCOST08RELPOVCAT     category
    FMTCOST12RELPOVCAT     category
    FMTCOSTMEDRELPOVCAT    category
    FMTINCRELPOVCAT        category
    FMTCOST06RELFMRCAT     category
    FMTCOST08RELFMRCAT     category
    FMTCOST12RELFMRCAT     category
    FMTCOSTMEDRELFMRCAT    category
    FMTINCRELFMRCAT        category
    FMTCOST06RELAMICAT     category
    FMTCOST08RELAMICAT     category
    FMTCOST12RELAMICAT     category
    FMTCOSTMEDRELAMICAT    category
    FMTINCRELAMICAT        category
    FMTASSISTED            category
    FMTBURDEN              category
    FMTREGION              category
    FMTSTATUS              category
    Length: 99, dtype: object

```
# Replace all category cell values with their numeric category codes
for col in cat_cols:
  national_processed[col] = national_processed[col].cat.codes

print(national_processed.shape)
national_processed.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      0
      82
      2
      0
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      0
      1
      40000
      -6
      1
      1
      6
      3117.394239
      1
      18021
      1
      533
      1
      0
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      1
      0
      5
      1
      2
      0
      4
      4
      4
      4
      3
      1
      1
      1
      1
      1
      1
      1
      1
      1
      2
      0
      2
      0
      0

      1
      1
      50
      4
      2
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      0
      1
      130000
      -6
      1
      1
      6
      2150.725544
      4
      122961
      1
      487
      1
      0
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      1
      0
      3
      1
      4
      0
      3
      4
      4
      3
      4
      2
      2
      2
      1
      3
      3
      3
      5
      3
      7
      0
      1
      0
      0

      2
      2
      53
      4
      2
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      0
      1
      150000
      -6
      1
      1
      7
      2213.789404
      2
      27974
      1
      1405
      1
      0
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      1
      0
      3
      1
      4
      0
      4
      4
      4
      4
      3
      2
      2
      2
      1
      2
      3
      4
      5
      3
      4
      0
      3
      0
      0

      3
      3
      67
      4
      2
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      0
      1
      200000
      -6
      1
      1
      6
      2364.585097
      2
      32220
      1
      279
      1
      0
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      1
      0
      3
      1
      3
      0
      4
      4
      4
      4
      4
      2
      2
      2
      2
      2
      5
      6
      6
      4
      4
      0
      1
      0
      0

      4
      4
      26
      0
      2
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      0
      1
      -6
      -6
      2
      100
      4
      2314.524902
      2
      96874
      1
      759
      5
      1
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      1
      1
      3
      5
      2
      1
      3
      3
      3
      3
      4
      2
      2
      2
      2
      3
      2
      2
      2
      2
      7
      1
      1
      0
      0

5 rows × 99 columns

```
# Now we only ahve numeric columns (ints and floats)
national_processed.dtypes
```

    CONTROL                  int32
    AGE1                     int64
    METRO3                    int8
    REGION                    int8
    LMED                     int64
    FMR                      int64
    L30                      int64
    L50                      int64
    L80                      int64
    IPOV                     int64
    BEDRMS                   int64
    BUILT                    int64
    STATUS                    int8
    TYPE                     int64
    VALUE                    int64
    VACANCY                  int64
    TENURE                    int8
    NUNITS                   int64
    ROOMS                    int64
    WEIGHT                 float64
    PER                      int64
    ZINC2                    int64
    ZADEQ                     int8
    ZSMHC                    int64
    STRUCTURETYPE            int64
    OWNRENT                   int8
    UTILITY                float64
    OTHERCOST              float64
    COST06                 float64
    COST12                 float64
                            ...
    COSTMedRELAMICAT         int64
    COSTMedRELPOVPCT       float64
    COSTMedRELPOVCAT         int64
    COSTMedRELFMRPCT       float64
    COSTMedRELFMRCAT         int64
    FMTZADEQ                  int8
    FMTMETRO3                 int8
    FMTBUILT                  int8
    FMTSTRUCTURETYPE          int8
    FMTBEDRMS                 int8
    FMTOWNRENT                int8
    FMTCOST06RELPOVCAT        int8
    FMTCOST08RELPOVCAT        int8
    FMTCOST12RELPOVCAT        int8
    FMTCOSTMEDRELPOVCAT       int8
    FMTINCRELPOVCAT           int8
    FMTCOST06RELFMRCAT        int8
    FMTCOST08RELFMRCAT        int8
    FMTCOST12RELFMRCAT        int8
    FMTCOSTMEDRELFMRCAT       int8
    FMTINCRELFMRCAT           int8
    FMTCOST06RELAMICAT        int8
    FMTCOST08RELAMICAT        int8
    FMTCOST12RELAMICAT        int8
    FMTCOSTMEDRELAMICAT       int8
    FMTINCRELAMICAT           int8
    FMTASSISTED               int8
    FMTBURDEN                 int8
    FMTREGION                 int8
    FMTSTATUS                 int8
    Length: 99, dtype: object

---

```
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
```

```
# There isn't a super clear Y varaible in this dataset, so we'll just
# pretend that our whole dataset is our X matrix for now.

# Make a copy to save our work at this checkpoint
df = national_processed.copy()
# Turn our dataframe into a numpy array
X = df.values
# instantiate our standard scaler object
scaler = StandardScaler()
# Standardize our data
Z = scaler.fit_transform(X)
```

```
# Instantiate our PCA object
pca = PCA(2)
transformed_data = pca.fit_transform(Z)
```

```
transformed_data
```

    array([[-2.57657018, -1.37612728],
           [ 2.04064284, -0.98806146],
           [ 1.21527025,  0.37016814],
           ...,
           [ 1.83162816, -2.95570216],
           [-5.02514474, -2.52857809],
           [-9.89816404, -0.80284263]])

```
plt.scatter(transformed_data[:,0], transformed_data[:,1])
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.title("Principal Component 1 vs Principal Component 2")
plt.show()
```

# Stretch Goals

## 1) Perform further data exploration on the HADS national dataset (the version before we one-hot encoded it) Make scatterplots and see if you can see any resemblance between the original scatterplots and the plot of the principal components that you made in 7.1.

(You may or may not not see very much resemblance depending on the variables you choose, and that's ok!)

## 2) Study "Scree Plots" and then try and make one for your PCA dataset. How many principal conponents do you need to retain in order for your PCs to contain 90% of the explained variance?

We will present this topic formally at the beginning of tomorrow's lecture, so if you figure this stretch goal out, you're ahead of the game.

## 3) Explore further the intuition behind eigenvalues and eigenvectors by creating your very own eigenfaces:

Prioritize self-study over this stretch goal if you are not semi-comfortable with the topics of PCA, Eigenvalues, and Eigenvectors.

You don't necessarily have to use this resource, but this will get you started:
[Eigenface Tutorial](https://sandipanweb.wordpress.com/2018/01/06/eigenfaces-and-a-simple-face-detector-with-pca-svd-in-python/)


```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd
import os
import data_helper as dh
```

```python
def computeCost(X, y, theta,m):
    costs = np.power(np.matmul(X,theta) - y,2)
    return np.sum(costs) / (2*m)

def gradient(X, y, theta,alpha, iterations):
    m = len(X)
    cost = np.zeros(iterations)
    for i in range(0, iterations):
        h = np.matmul(X, theta) - y
        sm = np.sum(np.multiply(h, X), axis=0)
        z = np.multiply(sm, alpha*1/m)
        theta = (theta.T - z).T
        cost[i] = computeCost(X, y, theta, m)
    return [cost, theta]

def plotData(x, y,h, xlabel, ylabel, xlim, ylim):
    plt.subplots()
    plt.plot(X, h)
    plt.scatter(x, y, c='red')
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.xlim(xlim)
    plt.ylim(ylim)
    plt.grid(color='gray', linestyle='--', linewidth=.6, axis='both', which='both', alpha=.4)
    plt.show()
```

```python
data = np.genfromtxt('../data/food-cart.csv', delimiter=',')
m = len(data)
X = np.hstack([np.ones([m, 1]), data[:,0].reshape(m,1)])
y = data[:,1].reshape(m,1)
iterations = 1500;
alpha = 0.01;
initial_theta = np.array([[0],[0]])

[costs, thetas] = gradient(X, y, initial_theta, alpha, iterations)
thetas
```

    array([[-3.63029144],
           [ 1.16636235]])

```python
predictions = np.matmul(X, thetas)
plotData(X[:, 1], y, predictions, 'City Population (10K)',
         'Profits ($10K)', [4, 24], [-5, 25])
```

\begin{align}
(a+b)^2 = a^2+2ab+b^2
\end{align}
\begin{equation*}
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\end{equation*}

\begin{equation*}
minimize \theta_{0},\theta_{1} \frac{1}{2m} \sum \limits _{i=0} ^{m} \left(h{_\theta}\left(x^\left(i\right)\right) - y^\left(i\right) \right)^2
\end{equation*}


# 插值

```python
from sympy import *
```

拉格朗日插值

```python
def lagrange_interpolate(points: list, simplify_result=True, verbose=False):
    """拉格朗日插值

    Args:
        points: list, [(x1, y1), (x2, y2), ..., (xn, yn)]
        simplify_result: bool, 化简最终结果, default True
        verbose: bool, 输出每一步的结果, default False

    Returns:
        L: sympy object of Symbol('x'), 插值多项式 $L(x)$
    """
    x = Symbol('x')
    L = 0  # 插值多项式
    for i, point in enumerate(points):
        xi, yi = point
        li = 1
        for j in range(len(points)):
            if j == i:
                continue
            xj, yj = points[j]
            li *= (x - xj) / (xi - xj)
        L += yi * li
        if verbose:
            print(f"l_{i}(x) = ", simplify(expand(yi * li)))

    if simplify_result:
        L = simplify(L)
    return L
```

```python
points = [(11, 0.190809), (12, 0.207912), (13, 0.224951)]

L = lagrange_interpolate(points)
print('插值多项式:', L)
print('Hypothesis:  ', L.subs(Symbol('x'), 11.5))
print('Actual vaule:', sin(rad(11.5)).evalf())
```

    插值多项式: -3.20000000000042e-5*x**2 + 0.0178390000000004*x - 0.00154799999999966
    Hypothesis:   0.199368500000004
    Actual vaule: 0.199367934417197

牛顿插值

```python
# 差商缓存
__difference_quotient_cache = {}

def difference_quotient(f, xs: list, verbose=False):
    """求差商 $f[xs...]$

    Args:
        f: function, 函数
        xs: list, 要计算的差商的 $f[x0, x1, ...]$ 的参数值：[x0, x1, ...]
        verbose: bool, 打印出每一步计算差商的值, default False

    Returns:
        dq: sympy object, 差商值
    """

    # 尝试从缓存读取
    __key = str([(x, f(x)) for x in xs])
    if __key in __difference_quotient_cache:
        if verbose:
            print(f"cached: f{xs}: {__difference_quotient_cache[__key]}")
        return __difference_quotient_cache[__key]

    if len(xs) == 1: # 0阶
        dq = sympify(f(xs[0]))
    else: # n 阶
        dq_h = difference_quotient(f, xs[:-1], verbose)
        dq_l = difference_quotient(f, xs[1: ], verbose)
        dq = (dq_l - dq_h) / (xs[-1] - xs[0])

    if verbose:
        print(f"f{xs}: {dq}")

    # 写入缓存
    __difference_quotient_cache[__key] = dq

    return dq
```

```python
def newton_interpolate(points: list, N_start=0, points_start=[], simplify_result=True, verbose=False):
    """牛顿插值

    Args:
        points: [(x1, y1), (x2, y2), ..., (xn, yn)]： 插值点

        N_start: a sympy object of Symbol('x'), 起始插值多项式。
        points_start: [(x1, y1), (x2, y2), ..., (xn, yn)]: 计算 N_start 用的插值点
                 该函数中做的插值会「承袭」这个用 points_start 计算的 N_start，在其基础上用新增插值点 points 去改进。
                 default 0 (从头开始构建).

        simplify_result: bool, 化简最终结果, default True
        verbose: 打印出每一步计算差商的值, default False

    Returns:
        N: a sympy object of Symbol('x'), 插值多项式 $N(x)$
    """
    x = Symbol('x')

    # sympify points
    # for i in range(len(points)):
    #     points[i] = sympify(points[i][0]), sympify(points[i][1])

    f = lambda x: dict(points + points_start)[x]

    xs = [p[0] for p in points_start] # 承袭的插值点
    N = N_start   # 承袭的插值多项式
    for point in points:
        xs.append(point[0])
        N += difference_quotient(f, xs, verbose) * prod([x - xi for xi in xs[:-1]])

    if simplify_result:
        N = simplify(N)
    return N
```

```python
points = [(11, 0.190809), (12, 0.207912), (13, 0.224951)]

N = newton_interpolate(points)
print('插值多项式:', N)
print('Hypothesis:  ', N.subs(Symbol('x'), 11.5))
print('Actual vaule:', sin(rad(11.5)).evalf())

# 承袭之前的 N:
print('承袭:')
new_points = [(11.2, sin(rad(11.2)).evalf()), (11.7, sin(rad(11.7)).evalf())]
N1 = newton_interpolate(new_points, N_start=N, points_start=points, verbose=True)
print('插值多项式:', N1)
print('Hypothesis:  ', N1.subs(Symbol('x'), 11.5).evalf())
print('Actual vaule:', sin(rad(11.5)).evalf())
```

    插值多项式: -3.20000000000042e-5*x**2 + 0.0178390000000001*x - 0.00154800000000063
    Hypothesis:   0.199368500000000
    Actual vaule: 0.199367934417197
    承袭:
    cached: f[11, 12, 13]: -0.0000320000000000042
    cached: f[12, 13]: 0.0170390000000000
    cached: f[13]: 0.224951000000000
    f[11.2]: 0.194234351219972
    f[13, 11.2]: 0.0170648048777933
    f[12, 13, 11.2]: -0.0000322560972416778
    f[11, 12, 13, 11.2]: -0.00000128048620836791
    cached: f[11, 12, 13, 11.2]: -0.00000128048620836791
    cached: f[12, 13, 11.2]: -0.0000322560972416778
    cached: f[13, 11.2]: 0.0170648048777933
    cached: f[11.2]: 0.194234351219972
    f[11.7]: 0.202787295356512
    f[11.2, 11.7]: 0.0171058882730810
    f[13, 11.2, 11.7]: -0.0000316026117597499
    f[12, 13, 11.2, 11.7]: -0.00000217828493975971
    f[11, 12, 13, 11.2, 11.7]: -0.00000128256961627400
    插值多项式: -1.282569616274e-6*x**4 + 5.9256799679765e-5*x**3 - 0.00105582207039453*x**2 + 0.0256792199573976*x - 0.0240006476355346
    Hypothesis:   0.199367875528590
    Actual vaule: 0.199367934417197

```python
__difference_quotient_cache
```

    {'[(11, 0.190809)]': 0.190809000000000,
     '[(12, 0.207912)]': 0.207912000000000,
     '[(11, 0.190809), (12, 0.207912)]': 0.0171030000000000,
     '[(13, 0.224951)]': 0.224951000000000,
     '[(12, 0.207912), (13, 0.224951)]': 0.0170390000000000,
     '[(11, 0.190809), (12, 0.207912), (13, 0.224951)]': -3.20000000000042e-5,
     '[(11.2, 0.194234351219972)]': 0.194234351219972,
     '[(13, 0.224951), (11.2, 0.194234351219972)]': 0.0170648048777933,
     '[(12, 0.207912), (13, 0.224951), (11.2, 0.194234351219972)]': -3.22560972416778e-5,
     '[(11, 0.190809), (12, 0.207912), (13, 0.224951), (11.2, 0.194234351219972)]': -1.28048620836791e-6,
     '[(11.7, 0.202787295356512)]': 0.202787295356512,
     '[(11.2, 0.194234351219972), (11.7, 0.202787295356512)]': 0.0171058882730810,
     '[(13, 0.224951), (11.2, 0.194234351219972), (11.7, 0.202787295356512)]': -3.16026117597499e-5,
     '[(12, 0.207912), (13, 0.224951), (11.2, 0.194234351219972), (11.7, 0.202787295356512)]': -2.17828493975971e-6,
     '[(11, 0.190809), (12, 0.207912), (13, 0.224951), (11.2, 0.194234351219972), (11.7, 0.202787295356512)]': -1.28256961627400e-6}

三弯矩法

```python
import numpy as np

def spline3_interpolate(points: list, simplify_result=True):
    """三弯矩法插值

    三弯矩法进行三次样条插值。使用自由边界条件。

    Args:
        points: list, [(x1, y1), (x2, y2), ..., (xn, yn)]
        simplify_result: bool, 化简最终结果, default True

    Returns:
        L: sympy Piecewise object of Symbol('x'), 插值多项式 $L(x)$
    """
    # 排序给定的点
    ps = sorted(points, key=lambda p: p[0])
    n = len(points)

    # points to a function
    _f_dict = dict(ps)
    def f(x):
        return _f_dict[x]

    # $h_k = x_{k+1} - x_k$
    def h(k):
        return ps[k+1][0] - ps[k][0]

    hks = [h(0)]

    # 用方程 D * M = d 解出 M

    D = np.zeros((n, n))
    d = np.zeros(n)

    for k in range(1, n-1):
        # $h_k$, $h_{k-1}$
        hks.append(h(k))
        hk, hks1 = hks[k], hks[k-1]

        # $\mu_k$ -> mu, $\lambda_k$ -> ld
        _fra = hks1 + hk
        mu = hks1 / _fra
        ld = hk / _fra

        # $\mu_kM_{k-1}+2M_k+\lambda_kM_{k+1} = d_k$
        D[k, k-1] = mu
        D[k, k]   = 2
        D[k, k+1] = ld
        d[k] = 6 * difference_quotient(f, [ps[k-1][0], ps[k][0], ps[k+1][0]])

    # 边界条件
    # Natural Boundary
    D[0, 0] = 1
    D[n-1, n-1] = 1

    d[0] = 0
    d[n-1] = 0

    # 解出 M
    M = np.linalg.solve(D, d)

    # 插值函数
    piecewises = []
    for k in range(n-1):
        s  = M[k] * (ps[k+1][0] - _x) ** 3 / (6 * hks[k])
        s += M[k+1] * (_x - ps[k][0]) ** 3 / (6 * hks[k])
        s += (ps[k][1] - M[k] * hks[k]**2 / 6) * (ps[k+1][0] - _x) / hks[k]
        s += (ps[k+1][1] - M[k+1] * hks[k]**2 / 6) * (_x - ps[k][0]) / hks[k]
        if simplify_result:
            s = simplify(s)
        piecewises.append((s, And(_x >= ps[k][0], _x <= ps[k+1][0])))

    return Piecewise(*piecewises)
```

题目
2．区间 $[-5, 5]$ 作等距划分：...

```python
def f(x):
    return 1 / (1 + x ** 2)

def get_points(s, n, f):
    """在闭区间 s 作 n 等距划分, 输出 n+1 个插值点 [(x0, f(x0)), (x1, f(x1)), ..., (xn, f(xn))]
    e.g.
        get_points([-5, 5], 2, lambda x: 1 / (1 + x ** 2))
    output: [(-5.0, 0.038461538461538464), (0.0, 1.0), (5.0, 0.038461538461538464)]
    """
    h = (s[1] - s[0]) / n
    xs = [s[0] + k * h for k in range(n+1)]
    points = [(x, f(x)) for x in xs]
    return points
```

```python
hps = {}  # 多项式插值
hss = {}  # 三次样条插值
for n in [5, 10, 20]:
    ps = get_points([-5, 5], n, f)
    # 多项式插值
    # h = lagrange_interpolate(ps)
    h = newton_interpolate(ps)
    hps[n] = h

    # 三次样条插值
    s = spline3_interpolate(ps)
    hss[n] = s

print('多项式插值：')
for k in hps:
    print(f'n = {k}:\t{hps[k]}')
print('\n三次样条插值：')
for k in hss:
    print(f'n = {k}:\t{hss[k]}')
```

    多项式插值：
    n = 5:	0.00192307692307692*x**4 - 0.0692307692307692*x**2 + 0.567307692307692
    n = 10:	-2.26244343891403e-5*x**10 + 0.00126696832579186*x**8 + 8.67361737988404e-19*x**7 - 0.0244117647058824*x**6 - 6.93889390390723e-18*x**5 + 0.19737556561086*x**4 + 5.55111512312578e-17*x**3 - 0.67420814479638*x**2 + 2.4980018054066e-16*x + 1.0
    n = 20:	2.72817068149796e-9*x**20 - 2.65314598775677e-7*x**18 + 2.11758236813575e-22*x**17 + 1.07425130797334e-5*x**16 - 0.000236412102809886*x**14 + 5.42101086242752e-20*x**13 + 0.00310184793200599*x**12 - 4.33680868994202e-19*x**11 - 0.0251135266738695*x**10 + 1.04083408558608e-17*x**9 + 0.126252909857124*x**8 - 4.16333634234434e-17*x**7 - 0.391630076762969*x**6 + 1.52655665885959e-16*x**5 + 0.753353962815113*x**4 - 8.32667268468867e-17*x**3 - 0.965739184991246*x**2 + 8.50014503228635e-17*x + 1.0

    三次样条插值：
    n = 5:	Piecewise((-0.0242914979757085*x + 1.72064777327935*(0.2*x + 1)**3 - 0.0829959514170041, (x >= -5.0) & (x <= -3.0)), (-0.0265182186234818*x**3 - 0.156072874493927*x**2 - 0.0795546558704453*x + 0.55, (x >= -3.0) & (x <= -1.0)), (0.576518218623482 - 0.0765182186234818*x**2, (x >= -1.0) & (x <= 1.0)), (0.0265182186234818*x**3 - 0.156072874493927*x**2 + 0.0795546558704453*x + 0.55, (x >= 1.0) & (x <= 3.0)), (0.0242914979757085*x - 1.72064777327935*(0.2*x - 1)**3 - 0.0829959514170041, (x >= 3.0) & (x <= 5.0)))
    n = 10:	Piecewise((0.0176283092922677*x + 0.341710207244819*(0.2*x + 1)**3 + 0.126603084922877, (x >= -5.0) & (x <= -4.0)), (0.00714607134821629*x**3 + 0.0939539011524711*x**2 + 0.434449138771531*x + 0.750706232344191, (x >= -4.0) & (x <= -3.0)), (0.00669108272293192*x**3 + 0.0898590035249119*x**2 + 0.422164445888853*x + 0.738421539461514, (x >= -3.0) & (x <= -2.0)), (0.107266068348291*x**3 + 0.693308917277068*x**2 + 1.62906427339317*x + 1.54302142446439, (x >= -2.0) & (x <= -1.0)), (-0.435755356116097*x**3 - 0.935755356116097*x**2 + 1.0, (x >= -1.0) & (x <= 0.0)), (0.435755356116097*x**3 - 0.935755356116097*x**2 + 1.0, (x >= 0.0) & (x <= 1.0)), (-0.107266068348291*x**3 + 0.693308917277068*x**2 - 1.62906427339317*x + 1.54302142446439, (x >= 1.0) & (x <= 2.0)), (-0.00669108272293192*x**3 + 0.0898590035249118*x**2 - 0.422164445888853*x + 0.738421539461513, (x >= 2.0) & (x <= 3.0)), (-0.00714607134821629*x**3 + 0.0939539011524711*x**2 - 0.434449138771531*x + 0.750706232344191, (x >= 3.0) & (x <= 4.0)), (-0.0176283092922677*x - 0.341710207244819*(0.2*x - 1)**3 + 0.126603084922877, (x >= 4.0) & (x <= 5.0)))
    n = 20:	Piecewise((0.0159894257773933*x + 0.602572179176649*(0.2*x + 1)**3 + 0.118408667348505, (x >= -5.0) & (x <= -4.5)), (0.00123647934877117*x**3 + 0.0239233373585306*x**2 + 0.15979877464138*x + 0.39437990856215, (x >= -4.5) & (x <= -4.0)), (0.00396184120838417*x**3 + 0.0566276796738866*x**2 + 0.290616143902804*x + 0.568803067577382, (x >= -4.0) & (x <= -3.5)), (0.00688951874776995*x**3 + 0.0873682938374373*x**2 + 0.398208293475231*x + 0.694327242078547, (x >= -3.5) & (x <= -3.0)), (0.0126608790854682*x**3 + 0.139310536876722*x**2 + 0.554035022593086*x + 0.850153971196401, (x >= -3.0) & (x <= -2.5)), (0.0283485524184901*x**3 + 0.256968086874386*x**2 + 0.848178897587244*x + 1.09527386702487, (x >= -2.5) & (x <= -2.0)), (0.0458282003652398*x**3 + 0.361845974554884*x**2 + 1.05793467294824*x + 1.23511105059886, (x >= -2.0) & (x <= -1.5)), (0.100274985643097*x**3 + 0.606856508305243*x**2 + 1.42545047357378*x + 1.41886895091163, (x >= -1.5) & (x <= -1.0)), (-0.262312758322244*x**3 - 0.48090672359078*x**2 + 0.337687241677756*x + 1.05628120694629, (x >= -1.0) & (x <= -0.5)), (-0.712562413892585*x**3 - 1.15628120694629*x**2 - 2.22044604925031e-16*x + 1.0, (x >= -0.5) & (x <= 0.0)), (0.712562413892585*x**3 - 1.15628120694629*x**2 + 2.22044604925031e-16*x + 1.0, (x >= 0.0) & (x <= 0.5)), (0.262312758322244*x**3 - 0.480906723590781*x**2 - 0.337687241677756*x + 1.05628120694629, (x >= 0.5) & (x <= 1.0)), (-0.100274985643097*x**3 + 0.606856508305243*x**2 - 1.42545047357378*x + 1.41886895091163, (x >= 1.0) & (x <= 1.5)), (-0.0458282003652397*x**3 + 0.361845974554884*x**2 - 1.05793467294824*x + 1.23511105059886, (x >= 1.5) & (x <= 2.0)), (-0.0283485524184901*x**3 + 0.256968086874386*x**2 - 0.848178897587245*x + 1.09527386702487, (x >= 2.0) & (x <= 2.5)), (-0.0126608790854682*x**3 + 0.139310536876722*x**2 - 0.554035022593086*x + 0.850153971196401, (x >= 2.5) & (x <= 3.0)), (-0.00688951874776995*x**3 + 0.0873682938374373*x**2 - 0.398208293475231*x + 0.694327242078547, (x >= 3.0) & (x <= 3.5)), (-0.00396184120838416*x**3 + 0.0566276796738865*x**2 - 0.290616143902804*x + 0.568803067577381, (x >= 3.5) & (x <= 4.0)), (-0.00123647934877118*x**3 + 0.0239233373585307*x**2 - 0.15979877464138*x + 0.39437990856215, (x >= 4.0) & (x <= 4.5)), (-0.0159894257773933*x - 0.602572179176649*(0.2*x - 1)**3 + 0.118408667348505, (x >= 4.5) & (x <= 5.0)))

```python
# 画图
def show_plot_range(r, hs, title):
    """画图比较插值结果
    r：区间
    hs: 插值结果组 {n: h(x)}
    title: 图标题
    """
    x = Symbol('x')
    colors = (c for c in [0, 'b', 'g', 'r'])

    plt = plot(f(x), (x, *r), show=False, line_color=next(colors), label='f(x)', legend=True, title=title)
    for n in hs:
        p = plot(hs[n], (x, *r), show=False, line_color=next(colors), label=f'n={n}', legend=True)
        plt.extend(p)

    plt.show()
```

```python
# 画图：多项式插值
show_plot_range((-5, 5), hps, 'Polynomial Interpolation')
show_plot_range((-5, -3), hps, 'Polynomial Interpolation')
show_plot_range((-1, 1), hps, 'Polynomial Interpolation')
```

```python
# 画图：三次样条插值
show_plot_range((-5, 5), hss, 'Cubic Spline Interpolation')
show_plot_range((-5, -3), hss, 'Cubic Spline Interpolation')
show_plot_range((-1, 1), hss, 'Cubic Spline Interpolation')
```

# 拟合

```python
def normalEquationFit(X, y):
    """最小二乘拟合
    Compute the normal equations:

    $\theta = (X^TX)^{-1}X^Ty$

    Args:
        X: np.array, $$
            X = \left[\begin{array}{c}
            - (x^{(1)})^T - \\
            - (x^{(2)})^T - \\
            \vdots \\
            - (x^{(m)})^T - \\
            \end{array}\right]
        $$
        y: np.array, $y = [y^{(1)},y^{(2)},\dots,y^{(m)}]^T$

    return theta.
    """
    return np.dot(np.dot(np.linalg.pinv(np.dot(X.T, X)), X.T), y)
```

题目：3.4.2 数据拟合 1.

```python
x = np.array([1, 2, 3, 4, 5])
y = np.array([4, 4.5, 6, 8, 8.5])

X = np.c_[np.ones((len(x), 1)), x]

theta = normalEquationFit(X, y)
print(theta)

pred_f = lambda t: theta[0] + theta[1] * t
```

    [2.45 1.25]

```python
# 用梯度下降检查

a, b = 0, 0

num_epoch = 10000
learning_rate = 5e-4

for e in range(num_epoch):
    y_pred = a * x + b

    grad_a = 2 * (y_pred - y).dot(x)
    grad_b = 2 * (y_pred - y).sum()

    a = a - learning_rate * grad_a
    b = b - learning_rate * grad_b

print(a, b)
```

    1.2501150707283306 2.449584558158982

```python
# 画图
import matplotlib.pyplot as plt

pred_x = [k/10 for k in range(50)]
pred_y = [pred_f(x) for x in pred_x]

plt.plot(x, y, 'bo', label='date')
plt.plot(pred_x, pred_y, 'r-', label='pred')
plt.legend()
plt.show()
```

### 题目：3.4.2 数据拟合 2.

```python
x = np.array([2, 3, 4, 7, 8, 10, 11, 14, 16, 18, 19])
y = np.array([106.42, 108.2, 109.5, 110, 109.93, 110.49, 110.59, 110.6, 110.76, 111, 111.2])
```

(1) $y(x) = c_0 + c_1x + c_2x^2$

数据预处理：

```python
# X: [1, x, x**2]
X = np.c_[np.ones((len(x), 1)), x, x ** 2]
print(X)
```

    [[  1.   2.   4.]
     [  1.   3.   9.]
     [  1.   4.  16.]
     [  1.   7.  49.]
     [  1.   8.  64.]
     [  1.  10. 100.]
     [  1.  11. 121.]
     [  1.  14. 196.]
     [  1.  16. 256.]
     [  1.  18. 324.]
     [  1.  19. 361.]]

定义预测函数、代价函数：

```python
def hypothesis(x, theta):
    return np.dot(theta, np.array([1, x, x**2]))

def cost(x, y, hypothesis, theta):
    return np.sum( (hypothesis(x, theta) - y) ** 2 )
```

最小二乘正规方程拟合：

```python
# theta: [c0, c1, c2]
theta = normalEquationFit(X, y)
theta
```

    array([ 1.06292705e+02,  6.26350752e-01, -2.04540298e-02])

计算代价：

```python
cost(x, y, hypothesis, theta)
```

    2.7796090121179953

可视化拟合：

```python
x_test = np.linspace(0, 25, num=100, endpoint=True)
y_pred = hypothesis(x_test, theta)

plt.plot(x, y, 'bo', label='date')
plt.plot(x_test, y_pred, 'r-', label='pred')
plt.legend()
plt.show()
```

(2) $y(x) = ae^{\frac{b}{x}}$

取对数：$\ln(y) = \ln(a) + \frac{b}{x}$

数据预处理：

```python
# X: [1, 1/x]
X = np.c_[np.ones((len(x), 1)), 1 / x]
print(X)
```

    [[1.         0.5       ]
     [1.         0.33333333]
     [1.         0.25      ]
     [1.         0.14285714]
     [1.         0.125     ]
     [1.         0.1       ]
     [1.         0.09090909]
     [1.         0.07142857]
     [1.         0.0625    ]
     [1.         0.05555556]
     [1.         0.05263158]]

新的预测函数：

```python
def hypothesis(x, theta):
    a, b = theta
    h = np.exp(a) * np.exp(b / x)
    return h
```

拟合：

```python
# theta: [ln(a), b]
theta = normalEquationFit(X, np.log(y))
theta
```

    array([ 4.7139705 , -0.09032026])

代价计算：

```python
cost(x, y, hypothesis, theta)
```

    0.47193209161233546

```python
x_test = np.linspace(0, 25, num=100, endpoint=True)
y_pred = hypothesis(x_test, theta)

plt.plot(x, y, 'bo', label='date')
plt.plot(x_test, y_pred, 'r-', label='pred')
plt.ylim(90, 120)
plt.legend()
plt.show()
```

```python

```


# A simple demonstration of coregionalization

This notebook shows how to construct a multi-output GP model using GPflow. We will consider a regression problem for functions $f: \mathbb{R}^D \rightarrow \mathbb{R}^P$. We assume that the dataset is of the form $(X_1, f_1), \dots, (X_P, f_P)$, that is, we do not necessarily observe all the outputs for a particular input location (for cases where there are fully observed outputs for each input, see [Multi-output Gaussian processes in GPflow](./multioutput.ipynb) for a more efficient implementation). We allow each $f_i$ to have a different noise distribution by assigning a different likelihood to each.

For this problem, we model $f$ as a *coregionalized* Gaussian process, which assumes a kernel of the form:

\begin{equation}
\textrm{cov}(f_i(X), f_j(X^\prime)) = k(X, X^\prime) \cdot B[i, j].
\end{equation}

The covariance of the $i$th function at $X$ and the $j$th function at $X^\prime$ is a kernel applied at $X$ and $X^\prime$, times the $(i, j)$th entry of a positive definite $P \times P$ matrix $B$. This is known as the **intrinsic model of coregionalization (ICM)** (Bonilla and Williams, 2008).

To make sure that B is positive-definite, we parameterize it as:

\begin{equation}
B = W W^\top + \textrm{diag}(\kappa).
\end{equation}

To build such a model in GPflow, we need to perform the following two steps:

 * Create the kernel function defined previously, using the `Coregion` kernel class.
 * Augment the training data X with an extra column that contains an integer index to indicate which output an observation is associated with. This is essential to make the data work with the `Coregion` kernel.
 * Create a likelihood for each output using the `SwitchedLikelihood` class, which is a container for other likelihoods.
 * Augment the training data Y with an extra column that contains an integer index to indicate which likelihood an observation is associated with.

```python
import gpflow
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

from gpflow.ci_utils import ci_niter

plt.rcParams["figure.figsize"] = (12, 6)
np.random.seed(123)
```

## Data preparation
We start by generating some training data to fit the model with. For this example, we choose the following two correlated functions for our outputs:

\begin{align}
y_1 &= \sin(6x) + \epsilon_1, \qquad \epsilon_1 \sim \mathcal{N}(0, 0.009) \\
y_2 &= \sin(6x + 0.7) + \epsilon_2, \qquad \epsilon_2 \sim \mathcal{N}(0, 0.01)
\end{align}

```python
# make a dataset with two outputs, correlated, heavy-tail noise. One has more noise than the other.
X1 = np.random.rand(100, 1)  # Observed locations for first output
X2 = np.random.rand(50, 1) * 0.5  # Observed locations for second output

Y1 = np.sin(6 * X1) + np.random.randn(*X1.shape) * 0.03
Y2 = np.sin(6 * X2 + 0.7) + np.random.randn(*X2.shape) * 0.1

plt.figure(figsize=(8, 4))
plt.plot(X1, Y1, "x", mew=2)
_ = plt.plot(X2, Y2, "x", mew=2)
```

## Data formatting for the coregionalized model
We add an extra column to our training dataset that contains an index that specifies which output is observed.

```python
# Augment the input with ones or zeros to indicate the required output dimension
X_augmented = np.vstack((np.hstack((X1, np.zeros_like(X1))), np.hstack((X2, np.ones_like(X2)))))

# Augment the Y data with ones or zeros that specify a likelihood from the list of likelihoods
Y_augmented = np.vstack((np.hstack((Y1, np.zeros_like(Y1))), np.hstack((Y2, np.ones_like(Y2)))))
```

## Building the coregionalization kernel
We build a coregionalization kernel with the Matern 3/2 kernel as the base kernel. This acts on the leading ([0]) data dimension of the augmented X values. The `Coregion` kernel indexes the outputs, and acts on the last ([1]) data dimension (indices) of the augmented X values. To specify these dimensions, we use the built-in `active_dims` argument in the kernel constructor. To construct the full multi-output kernel, we take the product of the two kernels (for a more in-depth tutorial on kernel combination, see [Manipulating kernels](./kernels.ipynb)).

```python
output_dim = 2  # Number of outputs
rank = 1  # Rank of W

# Base kernel
k = gpflow.kernels.Matern32(active_dims=[0])

# Coregion kernel
coreg = gpflow.kernels.Coregion(output_dim=output_dim, rank=rank, active_dims=[1])

kern = k * coreg
```

**Note:** W = 0 is a saddle point in the objective, which would result in the value of `W` not being optimized to fit the data.
Hence, by default, the `W` matrix is initialized with 0.1. Alternatively, you could re-initialize the matrix to random entries.

## Constructing the model
The final element in building the model is to specify the likelihood for each output dimension. To do this, use the `SwitchedLikelihood` object in GPflow.

```python
# This likelihood switches between Gaussian noise with different variances for each f_i:
lik = gpflow.likelihoods.SwitchedLikelihood(
    [gpflow.likelihoods.Gaussian(), gpflow.likelihoods.Gaussian()]
)

# now build the GP model as normal
m = gpflow.models.VGP((X_augmented, Y_augmented), kernel=kern, likelihood=lik)

# fit the covariance function parameters
maxiter = ci_niter(10000)
gpflow.optimizers.Scipy().minimize(
    m.training_loss, m.trainable_variables, options=dict(maxiter=maxiter), method="L-BFGS-B",
)
```

          fun: -225.66186977081497
     hess_inv: <11483x11483 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 1.23457887e-01, -5.01540314e-03, -1.60446231e-02, ...,
            1.21856960e-10, -5.10839847e-12,  8.28116661e-03])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 1417
          nit: 1302
         njev: 1417
       status: 0
      success: True
            x: array([ 7.97358981e-02, -6.13157171e-01, -1.28684745e+00, ...,
           -7.47875321e-12,  4.16495748e-13,  4.49475580e-01])

That's it: the model is trained. Let's plot the model fit to see what's happened.

```python
def plot_gp(x, mu, var, color, label):
    plt.plot(x, mu, color=color, lw=2, label=label)
    plt.fill_between(
        x[:, 0],
        (mu - 2 * np.sqrt(var))[:, 0],
        (mu + 2 * np.sqrt(var))[:, 0],
        color=color,
        alpha=0.4,
    )

def plot(m):
    plt.figure(figsize=(8, 4))
    Xtest = np.linspace(0, 1, 100)[:, None]
    (line,) = plt.plot(X1, Y1, "x", mew=2)
    mu, var = m.predict_f(np.hstack((Xtest, np.zeros_like(Xtest))))
    plot_gp(Xtest, mu, var, line.get_color(), "Y1")

    (line,) = plt.plot(X2, Y2, "x", mew=2)
    mu, var = m.predict_f(np.hstack((Xtest, np.ones_like(Xtest))))
    plot_gp(Xtest, mu, var, line.get_color(), "Y2")

    plt.legend()

plot(m)
```

From the plots, we can see:

 - The first function (blue) has low posterior variance everywhere because there are so many observations, and the noise variance is small.
 - The second function (orange) has higher posterior variance near the data, because the data are more noisy, and very high posterior variance where there are no observations (x > 0.5).
 - The model has done a reasonable job of estimating the noise variance and lengthscale.
 - The model recognises the correlation between the two functions and is able to suggest (with uncertainty) that because x > 0.5 the orange curve should follow the blue curve (which we know to be the truth from the data-generating procedure).

The covariance matrix between outputs is as follows:

```python
B = coreg.output_covariance().numpy()
print("B =", B)
_ = plt.imshow(B)
```

## References

Bonilla, Edwin V., Kian M. Chai, and Christopher Williams. "Multi-task Gaussian process prediction." _Advances in neural information processing systems_. 2008.


```python
import numpy as np
import matplotlib.pyplot as plt
```

# O método de Newton com aproximação da secante

Podemos modifiar o método de Newton para que,
em vez de usar a reta tangente, e com isso necessitar da derivada (que podemos não saber calcular de forma simples!)
ele use uma secante, e desta forma apenas uma aproximação da derivada.

Como vimos na parte anterior, a equação da reta secante é dada por
$$ S_{f,x1,x2}(t) = f(x_1) + \frac{f(x_2) - f(x_1)}{x_2 - x_1}(t - x_1), $$
daonde vemos que $\frac{f(x_2) - f(x_1)}{x_2 - x_1}$ é, naturalmente, uma aproximação da derivada.

Substituindo diretamente no método de Newton, temos assim uma fórmula que usa dois valores:
$$S_f(x_1,x_2) = x_1 - \frac{f(x_1)}{\frac{f(x_2) - f(x_1)}{x_2 - x_1}}.$$

## Simetria

É claro que a reta secante que passa em $x_1$ e $x_2$ não depende da ordem,
e as fórmulas que obtivemos acima parecem privilegiar $x_1$ face a $x_2$.
Isso é apenas porque não nos demos ao trabalho de explicitar a simetria:

Para a equação da reta secante:
$$\begin{align}
S_{f,x1,x2}(t)
  & = f(x_1) + \frac{f(x_2) - f(x_1)}{x_2 - x_1}(t - x_1) \\
  & = \frac{f(x_1)(x_2 - x_1) + \big[f(x_2) - f(x_1)\big]t - \big[f(x_2) - f(x_1)\big]x_1}{x_2 - x_1} \\
  & = \frac{x_2 f(x_1) - x_1 f(x_2) + t\big[f(x_2) - f(x_1)\big]}{x_2 - x_1}.
\end{align}$$
Para encontrar o zero da secante, podemos agora tanto fazer diretamente da equação acima,
igualando o numerador a zero,
ou então simetrizando $S_f(x_1,x2)$,
o que dá a mesma fórmula (ainda bem!):
$$S_f(x_1,x_2)
  = x_1 - \frac{f(x_1)}{\frac{f(x_2) - f(x_1)}{x_2 - x_1}}
  = x_1 - \frac{f(x_1)\big( x_2 - x_1 \big)}{f(x_2) - f(x_1)}
  = \frac{x_1 f(x_2) - x_2 f(x_1)}{f(x_2) - f(x_1)}.$$

No caso de um método iterativo, podemos usar outros pontos de $f$ já calculados,
para evitar chamar $f$ várias vezes!
Assim, temos a seguinte recorrência:
- Os valores $x_1$ e $x_2$ são tomados ("ao acaso", ou talvez "espertamente", dependendo se podemos ser espertos)
- Iteramos $x_{n+2} = S_f(x_{n+1},x_n)$
Neste caso, como temos uma _ordem_ entre os pontos utilizados,
haverá uma certa assimetria no algoritmo.

```python
def newton_sec(f,x1,x2, tol=1e-10, maxiter=100):
    """Método de Newton com aproximação da secante, para a função $f$, começando em x1 e x2."""
    prox = (x1*f(x2) - x2*f(x1))/(f(x2) - f(x1))
    if abs(prox - x2) < tol: # Erro pequeno: convergiu
        return prox
    if maxiter == 0: # Não convergiu
        return prox

    return newton_sec(f,x2,prox,tol, maxiter-1)
```

```python
import numpy as np
```

```python
ans = newton_sec(np.cos, 0,3)
ans - np.pi/2
```

    0.0

## Performance

O "método da secante", como é conhecido, converge mais lentamente do que o método de Newton:
podemos mostrar que os erros $e_i$ satisfazem uma "recorrência de desigualdade":
$$ e_{n+1} < C (e_n \cdot e_{n-1}). $$
Mas, como este calcula menos vezes a função, ele converge mais rápido por número de chamadas a $f$.

### Exercício: Fazer uma comparação entre as velocidades de Newton e da Secante.

Comece fazendo um gráfico simples, com a precisão da resposta em função do número de passos.
Em seguida, faça o gráfico da precisão em função do número de vezes que a função $f$ é calculada.

Se tiver tempo, faça desenhos indicando os pontos das iterações.
Enfim, se você tiver mais tempo ainda, anime o desenho "ao longo do tempo":
faça aparecer os pontos de Newton e da secante "na ordem em que foram obtidos"
(em função do número de vezes que $f$ foi calculada).

```python
def newton_list(f,df,x, prec=1e-8, maxiter=100, iterr = 0, lst = [], reti = False):
    if maxiter == iterr:
        return None
    dx = f(x)/df(x)
    newx = x - dx
    xs = lst[:]
    xs.append(x)
    if abs(dx) < prec:
        if reti:
            return xs, iterr
        else:
            return xs
    else:
        return newton_list(f,df,newx, prec,iterr = iterr+1, lst = xs)
```

```python
def newton_sec_list(f,x1,x2, tol=1e-10, maxiter=100, lst = []):
    """Método de Newton com aproximação da secante, para a função $f$, começando em x1 e x2."""
    prox = (x1*f(x2) - x2*f(x1))/(f(x2) - f(x1))
    xs = lst[:]
    xs.append(x1)
    if abs(prox - x2) < tol: # Erro pequeno: convergiu
        return xs
    if maxiter == 0: # Não convergiu
        return xs

    return newton_sec_list(f,x2,prox,tol, maxiter-1,xs)

```

```python
def f(x): return x**3 -5
def df(x): return 3*x**2

lnewton = newton_list(f,df,100)
lsecante = newton_sec_list(f,100,200)
ans = 5**(1/3)
plt.semilogy(np.abs(np.array(lnewton) - ans), label='Newton')
plt.semilogy(np.abs(np.array(lsecante) - ans), label='Secante')
plt.legend()
plt.show()
```

```python
a = newton_list(np.sin,np.cos,3,prec=1e-14)
a
```

    [3, 3.142546543074278, 3.141592653300477, 3.141592653589793]

Gráfico em função do andamento da função:

```python
### Resposta aqui
nwt_lst = newton_list(f,df,100)
nwt_sec_lst = newton_sec_list(f,100,200)
v1 = np.arange(0,len(nwt_lst),1)
v2 = np.arange(0,len(nwt_sec_lst),1)

_, [ax1, ax2] = plt.subplots(ncols=2, figsize=(15,5))
ax1.set_title('Newton')
ax1.semilogy(np.abs(np.array(nwt_lst) - np.pi))
#ax1.plot(v2, nwt_sec_lst)

ax2.set_title('Newton Secante')
ax2.plot(v2, np.abs(np.array(nwt_sec_lst) - np.pi))
plt.show()
```

```python

```


```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

## Exercise 2.17 Expected value of the minimum
Suppose $X$, $Y$ are two points sampled independently and uniformly at random from the interval [0,1]. What is the expected location of the left most point?

### Introduction

In this question, we have two random variables $X$, $Y \sim\mathcal{U}(0, 1)$. We want to know the expected value of the leftmost point (i.e. the expected value of the minimum). Since $E[X] = E[Y] = \frac{1}{2}$, it is reasonable to assume that if we repeat this experiment several times, then $\max(X, Y)\gt\frac{1}{2}$, and $\min(X, Y)\lt\frac{1}{2}$. Let's check if this is true.

### Solution
Random variable:

$X\sim\mathcal{U}(0, 1)$, $Y\sim\mathcal{U}(0, 1)$

The minimum between the two random variables can be computed by the $\min$ function:

\begin{equation}
\min(x, y) = \frac{x + y - |x-y|}{2}
\end{equation}
Now, we just have to compute the expected value of this function.

\begin{equation}
\mathbb{E}[\min(x, y)] = \iint_0^1\min(x, y)p(x, y) dx dy
\end{equation}

Since $X$, $Y$ are independent, we can easily decompose the joint distribution of (2):

\begin{aligned}
\mathbb{E}[\min(x, y)] & = \iint_0^1\min(x, y)p(x)p(y) dxdy \\
& = \iint_0^1\min(x,y)dxdy\\
& = \iint_0^1\frac{x+y-|x-y|}{2}dxdy \\
& = \frac{1}{2}\iint_0^1x + y - |x-y|dxdy
\end{aligned}

For the final step, we have to decompose the double integral, in order to strip of the absolute value of the integrand.

\begin{aligned}
& \frac{1}{2}\int_0^1\left(\int_0^yx+y-(-x+y)dx + \int_y^1x+y-(x-y)dx\right)dy \\
= & \frac{1}{2}\int_0^1\left(x^2\left\vert_0^y + 2xy\right\vert_y^1\right)dy \\
= & \frac{1}{2}\int_0^1(y^2 + 2y - 2y^2) dy \\
= & \frac{1}{2}\left[-\frac{y^3}{3}\left\vert_0^1 + y^2\right\vert_0^1\right] \\
= & \frac{1}{3}
\end{aligned}

Therefore, the expectation of the leftmost variable is 1/3.


```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import sympy as sy
sy.init_printing()
```

```python
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

```python
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all" # display multiple results
```

```python
def round_expr(expr, num_digits):
    return expr.xreplace({n : round(n, num_digits) for n in expr.atoms(sy.Number)})
```

#  Matrix Operations

Matrix _addition_ operations are straightforward:
1. $A+ B= B+ A$
2. $(A+B)+ C=A+(B+C)$
3. $c(A+B)=cA+cB$
4. $(c+d)A=cA+c{D}$
5. $c(dA)=(cd)A$
6. $A+{0}=A$, where ${0}$ is the zero matrix
7. For any $A$, there exists an $- A$, such that $ A+(- A)=0$.

They are as obvious as it looks, so no proofs are provided. And the matrix _multiplication_ properties are:
1. $ A({BC})=({AB}) C$
2. $c({AB})=(cA)B=A(cB)$
3. $A(B+ C)={AB}+{AC}$
4. $(B+C)A={BA}+{CA}$

Note that we need to differentiate two kinds of multiplication, **Hadamard multiplication** (element-wise multiplication) and **matrix multiplication**:

```python
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
```

```python
A*B # this is Hadamard elementwise product
```

    array([[ 5, 12],
           [21, 32]])

```python
A@B # this is matrix product
```

    array([[19, 22],
           [43, 50]])

We show explicitly the matrix multiplication rule:

```python
np.sum(A[0,:]*B[:,0]) # (1, 1)
np.sum(A[1,:]*B[:,0]) # (2, 1)
np.sum(A[0,:]*B[:,1]) # (1, 2)
np.sum(A[1,:]*B[:,1]) # (2, 2)
```

    19

    43

    22

    50

##  SymPy Demonstration: Addition

Let's define all the letters as symbols in case we might use them repetitively.

```python
a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z = sy.symbols('a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z', real = True)
```

```python
A = sy.Matrix([[a, b, c], [d, e, f]])
A + A
A - A
```

$\displaystyle \left[\begin{matrix}2 a & 2 b & 2 c\\2 d & 2 e & 2 f\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

```python
B = sy.Matrix([[g, h, i], [j, k, l]])
A + B
A - B
```

$\displaystyle \left[\begin{matrix}a + g & b + h & c + i\\d + j & e + k & f + l\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}a - g & b - h & c - i\\d - j & e - k & f - l\end{matrix}\right]$

##  SymPy Demonstration: Multiplication

The matrix multiplication rules can be clearly understood by using symbols.

```python
A = sy.Matrix([[a, b, c], [d, e, f]])
B = sy.Matrix([[g, h, i], [j, k, l], [m, n, o]])
A
B
```

$\displaystyle \left[\begin{matrix}a & b & c\\d & e & f\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}g & h & i\\j & k & l\\m & n & o\end{matrix}\right]$

```python
AB = A*B; AB
```

$\displaystyle \left[\begin{matrix}a g + b j + c m & a h + b k + c n & a i + b l + c o\\d g + e j + f m & d h + e k + f n & d i + e l + f o\end{matrix}\right]$

##  Commutability

The matrix multiplication usually do not commute, such that ${AB} \neq {BA}$. For instance, consider $ A$ and $ B$:

```python
A = sy.Matrix([[3, 4], [7, 8]])
B = sy.Matrix([[5, 3], [2, 1]])
A*B
B*A
```

$\displaystyle \left[\begin{matrix}23 & 13\\51 & 29\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}36 & 44\\13 & 16\end{matrix}\right]$

How do we find a commutable matrix?

```python
A = sy.Matrix([[a, b], [c, d]])
B = sy.Matrix([[e, f], [g, h]])
A*B
B*A
```

$\displaystyle \left[\begin{matrix}a e + b g & a f + b h\\c e + d g & c f + d h\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}a e + c f & b e + d f\\a g + c h & b g + d h\end{matrix}\right]$

To show ${AB} = {BA}$, we need to prove ${AB} - {BA} = 0$

```python
M = A*B - B*A; M
```

$\displaystyle \left[\begin{matrix}b g - c f & a f - b e + b h - d f\\- a g + c e - c h + d g & - b g + c f\end{matrix}\right]$

That means
\begin{align}
b g - c f&=0 \\
 a f - b e + b h - d f&=0\\
- a g + c e - c h + d g&=0 \\
- b g + c f&=0
\end{align}

If we treat $a, b, c, d$ as coefficients of the system, we and extract an augmented matrix

```python
A_aug = sy.Matrix([[0, -c, b, 0], [-b, a-d, 0, b], [c, 0, d -a, -c], [0, c, -b, 0]]); A_aug
```

$\displaystyle \left[\begin{matrix}0 & - c & b & 0\\- b & a - d & 0 & b\\c & 0 & - a + d & - c\\0 & c & - b & 0\end{matrix}\right]$

Perform **Gaussian-Jordon** elimination till row reduced formed.

```python
A_aug.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & - \frac{a}{c} + \frac{d}{c} & -1\\0 & 1 & - \frac{b}{c} & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1\right)\right)$

The general solution is
\begin{align}
e - \frac{a-d}cg - h &=0\\
f - \frac{b}{c} & =0\\
g &= free\\
h & =free
\end{align}

if we set coefficients $a = 10, b = 12, c = 20, d = 8$, or $ A = \left[\begin{matrix}10 & 12\\20 & 8\end{matrix}\right]$ then general solution becomes

\begin{align}
e - .1g - h &=0\\
f - .6 & =0\\
g &= free\\
h & =free
\end{align}
Then try a special solution when $g = h = 1$
\begin{align}
e  &=1.1\\
f & =.6\\
g &=1 \\
h & =1
\end{align}
And this is a **commutable matrix of $A$**, we denote $ C$.

```python
C = sy.Matrix([[1.1, .6], [1, 1]]);C
```

$\displaystyle \left[\begin{matrix}1.1 & 0.6\\1 & 1\end{matrix}\right]$

Now we can see that ${AC}={CA}$.

```python
A = sy.Matrix([[10, 12], [20, 8]])
A*C
C*A
```

$\displaystyle \left[\begin{matrix}23.0 & 18.0\\30.0 & 20.0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}23.0 & 18.0\\30 & 20\end{matrix}\right]$

#  Transpose of Matrices

Matrix $A_{n\times m}$ and its transpose is

```python
A = np.array([[1, 2, 3], [4, 5, 6]]); A
A.T # transpose
```

    array([[1, 2, 3],
           [4, 5, 6]])

    array([[1, 4],
           [2, 5],
           [3, 6]])

```python
A = sy.Matrix([[1, 2, 3], [4, 5, 6]]); A
A.transpose()
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & 4\\2 & 5\\3 & 6\end{matrix}\right]$

The properties of transpose are
1. $(A^T)^T$
2. $(A+B)^T=A^T+B^T$
3. $(cA)^T=cA^T$
4. $(AB)^T=B^TA^T$

We can show why the last property holds with SymPy, define $A$ and $B$, multiply them, then transpose, that means $(AB)^T$

```python
A = sy.Matrix([[a, b], [c, d], [e, f]])
B = sy.Matrix([[g, h, i], [j, k, l]])
AB = A*B
AB_tr = AB.transpose(); AB_tr
```

$\displaystyle \left[\begin{matrix}a g + b j & c g + d j & e g + f j\\a h + b k & c h + d k & e h + f k\\a i + b l & c i + d l & e i + f l\end{matrix}\right]$

Transpose each of them, then multiply, that means $B^TA^T$

```python
B_tr_A_tr = B.transpose()*A.transpose()
B_tr_A_tr
```

$\displaystyle \left[\begin{matrix}a g + b j & c g + d j & e g + f j\\a h + b k & c h + d k & e h + f k\\a i + b l & c i + d l & e i + f l\end{matrix}\right]$

To see if they are equal

```python
AB_tr - B_tr_A_tr
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

#  Identity Matrices

This is an identity matrix $I_5$

```python
sy.eye(5)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 1 & 0\\0 & 0 & 0 & 0 & 1\end{matrix}\right]$

Identity matrix properties:

$$
AI=IA = A
$$

Let's generate $ I$ and $ A$ and show if it holds

```python
I = np.eye(5); I
```

    array([[1., 0., 0., 0., 0.],
           [0., 1., 0., 0., 0.],
           [0., 0., 1., 0., 0.],
           [0., 0., 0., 1., 0.],
           [0., 0., 0., 0., 1.]])

```python
A = np.around(np.random.rand(5, 5)*100); A # generate a random matrix
```

    array([[59., 75., 69., 73., 34.],
           [94.,  8., 73.,  6., 10.],
           [73., 23., 55., 27., 67.],
           [ 1., 36., 25., 68., 75.],
           [35., 10., 33., 31., 79.]])

Apparently it holds

```python
A@I - I@A
```

    array([[0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.]])

#  Elementary Matrix

An **elementary matrix** is a matrix that can be obtained from a single elementary row operation on an identity matrix. Such as:

$$
\left[\begin{matrix}1 & 0 & 0\cr 0 & 1 & 0\cr 0 & 0 & 1\end{matrix}\right]\ \matrix{R_1\leftrightarrow R_2\cr ~\cr ~}\qquad\Longrightarrow\qquad \left[\begin{matrix}0 & 1 & 0\cr 1 & 0 & 0\cr 0 & 0 & 1\end{matrix}\right]
$$

where $R_1\leftrightarrow R_2$ means exchanging row $1$ and row $2$, and we denote the transformed matrix as ${E}$, then left multiply $  E$ onto a matrix $  A$. Generate $ A$ first

```python
A = sy.randMatrix(3, percent = 80); A # generate a random matrix with 80% of entries being nonzero
```

$\displaystyle \left[\begin{matrix}34 & 5 & 0\\34 & 67 & 4\\14 & 32 & 44\end{matrix}\right]$

Create an elementary matrix with $R_1\leftrightarrow R_2$

```python
E = sy.Matrix([[0, 1, 0], [1, 0, 0], [0, 0, 1]]);E
```

$\displaystyle \left[\begin{matrix}0 & 1 & 0\\1 & 0 & 0\\0 & 0 & 1\end{matrix}\right]$

It turns out that by left-multiplying $ E$ onto $ A$, $ A$ also switches the row $1$ and $2$.

```python
E*A
```

$\displaystyle \left[\begin{matrix}34 & 67 & 4\\34 & 5 & 0\\14 & 32 & 44\end{matrix}\right]$

Adding a multiple of a row onto another row in the identity matrix also gives us an elementary matrix.

$$
\left[\begin{matrix}1 & 0 & 0\cr 0 & 1 & 0\cr 0 & 0 & 1\end{matrix}\right]\ \matrix{~\cr ~\cr R_3-7R_1}\qquad\longrightarrow\left[\begin{matrix}1 & 0 & 0\cr 0 & 1 & 0\cr -7 & 0 & 1\end{matrix}\right]
$$

Let's verify with SymPy.

```python
A = sy.randMatrix(3, percent = 80); A
E = sy.Matrix([[1, 0, 0], [0, 1, 0], [-7, 0, 1]]); E
```

$\displaystyle \left[\begin{matrix}72 & 31 & 34\\25 & 12 & 7\\54 & 8 & 0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\-7 & 0 & 1\end{matrix}\right]$

```python
E*A
```

$\displaystyle \left[\begin{matrix}72 & 31 & 34\\25 & 12 & 7\\-450 & -209 & -238\end{matrix}\right]$

We can also show this by explicit row operation on $ A$.

```python
EA = sy.matrices.MatrixBase.copy(A)
EA[2,:]=-7*EA[0,:]+EA[2,:]
EA
```

$\displaystyle \left[\begin{matrix}72 & 31 & 34\\25 & 12 & 7\\-450 & -209 & -238\end{matrix}\right]$

In next section, we will see an important conclusion of elementary matrices multiplication is that an **invertible matrix** is a product of a series of elementary matrices.

#  Inverse Matrices

If ${AB}={BA}=\mathbf{I}$, $ B$ is called the inverse of matrix $  A$, denoted as $ B=  A^{-1}$.

NumPy has convenient function ```np.linalg.inv()``` for computing inverse matrices. Generate $ A$

```python
A = np.round(10*np.random.randn(5,5)); A
```

    array([[ -1.,  -8.,  -6.,  -2., -16.],
           [ -1.,  -2.,   8., -13.,   4.],
           [  1.,  11., -13.,   2.,   3.],
           [ -1., -13.,   6.,   8.,   9.],
           [-12.,   9.,  -4.,  -1.,   7.]])

```python
Ainv = np.linalg.inv(A); Ainv
```

    array([[-0.026,  0.013,  0.042, -0.005, -0.078],
           [-0.048, -0.038, -0.034, -0.064,  0.01 ],
           [-0.05 , -0.034, -0.096, -0.043,  0.003],
           [-0.026, -0.08 , -0.04 , -0.002,  0.006],
           [-0.015,  0.041,  0.055,  0.049, -0.002]])

Verify if they are truly inverse of each other

```python
A@Ainv
```

    array([[ 1.,  0., -0.,  0., -0.],
           [-0.,  1.,  0., -0.,  0.],
           [-0.,  0.,  1.,  0.,  0.],
           [ 0.,  0.,  0.,  1., -0.],
           [ 0., -0., -0.,  0.,  1.]])

The ```-0.``` means there are more digits after point, but omitted here.

##  $[A\,|\,I]\sim [I\,|\,A^{-1}]$ Algorithm

A convenient way of calculating inverse is that we can construct an augmented matrix $[ A\,|\,\mathbf{I}]$, then multiply a series of $ E$'s which are elementary row operations till matrix $A$ becomes row reduced form, if full rank it means $ A \rightarrow \mathbf{I}$. Then $\mathbf{I}$ on the RHS of augmented matrix will be converted into $ A^{-1}$ automatically.

We can show with SymPy's ```.rref()``` function on the augmented matrix $[A\,|\,I]$.

```python
AI = np.hstack((A, I)) # stack the matrix A and I horizontally
AI = sy.Matrix(AI); AI
```

$\displaystyle \left[\begin{matrix}-1.0 & -8.0 & -6.0 & -2.0 & -16.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0\\-1.0 & -2.0 & 8.0 & -13.0 & 4.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0\\1.0 & 11.0 & -13.0 & 2.0 & 3.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0\\-1.0 & -13.0 & 6.0 & 8.0 & 9.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0\\-12.0 & 9.0 & -4.0 & -1.0 & 7.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0\end{matrix}\right]$

```python
AI_rref = AI.rref(); AI_rref
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & 0 & 0 & -0.0257937689879073 & 0.0134627985941502 & 0.0420563531303985 & -0.00488473223327575 & -0.0783939953535474\\0 & 1 & 0 & 0 & 0 & -0.0475017432957576 & -0.0379022282648688 & -0.0338111774166985 & -0.0638116329512683 & 0.00961703559126635\\0 & 0 & 1 & 0 & 0 & -0.0496953174550335 & -0.0339723665721724 & -0.0957148213428458 & -0.0433738992707943 & 0.00261056349626286\\0 & 0 & 0 & 1 & 0 & -0.0259829910399854 & -0.0804474050298025 & -0.0399608941092372 & -0.00152428875285147 & 0.00566614922611685\\0 & 0 & 0 & 0 & 1 & -0.0152533998647413 & 0.04090525231359 & 0.0551652364049212 & 0.0486668605608682 & -0.00159612305039964\end{matrix}\right], \  \left( 0, \  1, \  2, \  3, \  4\right)\right)$

Extract the RHS block, this is the $A^{-1}$.

```python
Ainv = AI_rref[0][:,5:];Ainv # extract the RHS block
```

$\displaystyle \left[\begin{matrix}-0.0257937689879073 & 0.0134627985941502 & 0.0420563531303985 & -0.00488473223327575 & -0.0783939953535474\\-0.0475017432957576 & -0.0379022282648688 & -0.0338111774166985 & -0.0638116329512683 & 0.00961703559126635\\-0.0496953174550335 & -0.0339723665721724 & -0.0957148213428458 & -0.0433738992707943 & 0.00261056349626286\\-0.0259829910399854 & -0.0804474050298025 & -0.0399608941092372 & -0.00152428875285147 & 0.00566614922611685\\-0.0152533998647413 & 0.04090525231359 & 0.0551652364049212 & 0.0486668605608682 & -0.00159612305039964\end{matrix}\right]$

I wrote a function to round the float numbers to the $4$th digits, on the top of this file, but this is not absolutely necessary.

```python
round_expr(Ainv, 4)
```

$\displaystyle \left[\begin{matrix}-0.0258 & 0.0135 & 0.0421 & -0.0049 & -0.0784\\-0.0475 & -0.0379 & -0.0338 & -0.0638 & 0.0096\\-0.0497 & -0.034 & -0.0957 & -0.0434 & 0.0026\\-0.026 & -0.0804 & -0.04 & -0.0015 & 0.0057\\-0.0153 & 0.0409 & 0.0552 & 0.0487 & -0.0016\end{matrix}\right]$

We can verify if $AA^{-1}=\mathbf{I}$

```python
A = sy.Matrix(A)
round_expr(A*Ainv, 4)
```

$\displaystyle \left[\begin{matrix}1.0 & 0 & 0 & 0 & 0.0\\0.0 & 1.0 & 0.0 & 0 & 0.0\\0.0 & 0.0 & 1.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 1.0 & 0.0\\0.0 & 0 & 0 & 0.0 & 1.0\end{matrix}\right]$

We got $\mathbf{I}$, which means the RHS block is indeed $A^{-1}$.

##  An Example of Existence of Inverse

Determine the values of $\lambda$ such that the matrix
$$A=\left[ \begin{matrix}3 &\lambda &1\cr 2 & -1 & 6\cr 1 & 9 & 4\end{matrix}\right]$$
is not invertible.

Still,we are using SymPy to solve the problem.

```python
lamb = sy.symbols('lamda') # SymPy will automatically render into LaTeX greek letters
A = np.array([[3, lamb, 1], [2, -1, 6], [1, 9, 4]])
I = np.eye(3); A
```

    array([[3, lamda, 1],
           [2, -1, 6],
           [1, 9, 4]], dtype=object)

Form the augmented matrix.

```python
AI = np.hstack((A, I))
AI = sy.Matrix(AI); AI
```

$\displaystyle \left[\begin{matrix}3 & \lambda & 1 & 1.0 & 0.0 & 0.0\\2 & -1 & 6 & 0.0 & 1.0 & 0.0\\1 & 9 & 4 & 0.0 & 0.0 & 1.0\end{matrix}\right]$

```python
AI_rref = AI.rref()
AI_rref
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & \frac{116.0}{4.0 \lambda + 310.0} & \frac{8.0 \lambda - 18.0}{4.0 \lambda + 310.0} & \frac{- 12.0 \lambda - 2.0}{4.0 \lambda + 310.0}\\0 & 1 & 0 & \frac{4.0}{4.0 \lambda + 310.0} & - \frac{22.0}{4.0 \lambda + 310.0} & \frac{32.0}{4.0 \lambda + 310.0}\\0 & 0 & 1 & \frac{57.0}{- 6 \lambda - 465} & \frac{27.0 - 1.0 \lambda}{2.0 \lambda + 155.0} & \frac{2.0 \lambda + 3.0}{2.0 \lambda + 155.0}\end{matrix}\right], \  \left( 0, \  1, \  2\right)\right)$

To make the matrix $A$ invertible we notice that is one conditions to be satisfied (in the denominators):
\begin{align}
-6\lambda -465 &\neq0\\
\end{align}

Solve for $\lambda$'s.

```python
sy.solvers.solve(-6*lamb-465, lamb)
```

So this is one of $\lambda$ that cause the matrix invertible. Let's test with **determinant**. If $| A|=0$, then the matrix is not invertible. Don't worry about determinants, we will get back to this.

```python
A = np.array([[3, -155/2, 1], [2, -1, 6], [1, 9, 4]])
np.linalg.det(A)
```

The $| A|$ is $0$.

So we found that one condition, as long as $\lambda \neq -\frac{155}{2}$, the matrix $A$ is invertible.

##  Properties of Inverse Matrices

1. If $A$ and $B$ are both invertible, then $(AB)^{-1}=B^{-1}A^{-1}$.
2. If $A$ is invertible, then $(A^T)^{-1}=(A^{-1})^T$.
3. If $A$ and $B$ are both invertible and symmetric such that $AB=BA$, then $A^{-1}B$ is symmetric.

The _first property_ is straightforward
\begin{align}
ABB^{-1}A^{-1}=AIA^{-1}=I=AB(AB)^{-1}
\end{align}

The trick of _second property_ is to show that
$$
A^T(A^{-1})^T = I
$$
We can use the property of transpose
$$
A^T(A^{-1})^T=(A^{-1}A)^T = I^T = I
$$

The _third property_ is to show
$$
A^{-1}B = (A^{-1}B)^T
$$
Again use the property of transpose
$$
(A^{-1}B)^{T}=B^T(A^{-1})^T=B(A^T)^{-1}=BA^{-1}
$$
We use the $AB = BA$ condition to proceed
\begin{align}
AB&=BA\\
A^{-1}ABA^{-1}&=A^{-1}BAA^{-1}\\
BA^{-1}&=A^{-1}B
\end{align}
The plug in the previous equation, we have
$$
(A^{-1}B)^{T}=BA^{-1}=A^{-1}B
$$