# K-Means Clustering

# 1) Use the "Breast Cancer Wisconsin (Diagnostic) Data Set" from Kaggle to try and cluster types of cancer cells.

Here's the original dataset for your reference:

<https://www.kaggle.com/uciml/breast-cancer-wisconsin-data>

## This is a supervised learning dataset

(Because it has **labels** - The "diagnosis" column.)

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA # You don't necessarily have to use this
from sklearn.cluster import KMeans # You don't necessarily have to use this
from sklearn.preprocessing import StandardScaler # You don't necessarily have to use this

df = pd.read_csv("https://raw.githubusercontent.com/ryanleeallred/datasets/master/Cancer_Cells.csv")
print(df.shape)
df.head()
```

    (569, 33)

      id
      diagnosis
      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst
      fractal_dimension_worst
      Unnamed: 32

      0
      842302
      M
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890
      NaN

      1
      842517
      M
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902
      NaN

      2
      84300903
      M
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758
      NaN

      3
      84348301
      M
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300
      NaN

      4
      84358402
      M
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678
      NaN

```
# This is cheating
# in a truly unsupervised learning situation
# I would not be able to create the graph below.
fig, ax = plt.subplots()
colors = {'M':'red', 'B':'blue'}
grouped = df.groupby('diagnosis')
for key, group in grouped:
  group.plot(ax=ax, kind='scatter', x='radius_mean', y='texture_mean', label=key, color=colors[key])
plt.show()
```

```
df.diagnosis.value_counts()
```

    B    357
    M    212
    Name: diagnosis, dtype: int64

```
# Sanity check for baseline accuracy
# If we had classified everything as Benign
357 / (212 + 357)
```

    0.6274165202108963

```
df.isnull().sum()
```

    id                           0
    diagnosis                    0
    radius_mean                  0
    texture_mean                 0
    perimeter_mean               0
    area_mean                    0
    smoothness_mean              0
    compactness_mean             0
    concavity_mean               0
    concave points_mean          0
    symmetry_mean                0
    fractal_dimension_mean       0
    radius_se                    0
    texture_se                   0
    perimeter_se                 0
    area_se                      0
    smoothness_se                0
    compactness_se               0
    concavity_se                 0
    concave points_se            0
    symmetry_se                  0
    fractal_dimension_se         0
    radius_worst                 0
    texture_worst                0
    perimeter_worst              0
    area_worst                   0
    smoothness_worst             0
    compactness_worst            0
    concavity_worst              0
    concave points_worst         0
    symmetry_worst               0
    fractal_dimension_worst      0
    Unnamed: 32                569
    dtype: int64

## Now it's an unsupervised learning dataset

(Because we've removed the diagnosis label) - Use this version.

```
df_copy = df.drop(['diagnosis', 'Unnamed: 32', 'id'], axis=1)
df_copy.head()
```

      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst
      fractal_dimension_worst

      0
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890

      1
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902

      2
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758

      3
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300

      4
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678

```
plt.scatter(x=df_copy['radius_mean'], y=df['texture_mean'])
plt.show()
```

## Let's do it!

- You might want to do some data exploration to see if you can find specific columns that will help you find distinct clusters of cells
- You might want to use the elbow method to decide on the number of clusters to use.

```
from sklearn.cluster import KMeans
kmeans = KMeans(2)
X = df_copy.values
kmeans.fit(X)

print(kmeans.labels_)
```

    [0 0 0 1 0 1 0 1 1 1 1 0 0 1 1 1 1 0 0 1 1 1 1 0 0 0 1 0 0 0 0 1 0 0 0 0 1
     1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1
     1 0 1 0 0 1 1 1 0 0 1 0 1 0 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1
     1 1 1 1 1 1 1 0 0 1 0 0 1 1 1 1 0 1 0 1 1 1 1 0 1 1 1 1 1 1 0 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 1 0 1 1 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 1 1
     1 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 0 0 1 1 1 1 0 1 1 0 1 0 1 1 1 1 1 0 0 1 1
     1 1 1 1 1 1 1 1 0 1 1 0 1 1 0 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 0 0 0 1 0 1 0
     1 0 0 0 1 0 0 1 1 1 1 1 1 0 1 0 1 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 0 1 1 1 1 1 1 1 1 1
     1 1 0 1 0 1 0 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 0
     1 1 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1
     1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 0 1 1
     0 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 0 1 1 1 0 0 1 1 1 1 1 0 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0
     1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 0 0 0 1 0 1]

```
df_copy['cluster'] = kmeans.labels_
df_copy.head()
```

      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst
      fractal_dimension_worst
      cluster

      0
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.3001
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890
      0

      1
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.0869
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902
      0

      2
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.1974
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758
      0

      3
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.2414
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300
      1

      4
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.1980
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678
      0

```
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
colors = {0:'red', 1:'blue'}
grouped = df_copy.groupby('cluster')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='radius_mean', y='texture_mean', label=key, color=colors[key])
plt.show()
```

## Check you work:

This is something that in a truly unsupervised learning situation **WOULD NOT BE POSSIBLE**. But for educational purposes go back and grab the true diagnosis column (label) from the original dataset. Take your cluster labels and compare them to the original diagnosis column. You can make scatterplots for each to see how they compare or you can calculate a percent accuracy score like:
\begin{align}
\frac{\text{Num Correct Labels}}{\text{Num Total Observations}}
\end{align}

```
# Make diagnosis values numeric:
df['diagnosis'] = df['diagnosis'].astype('category')
df['diagnosis'] = df['diagnosis'].cat.codes
df.head(10)
```

      id
      diagnosis
      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst
      fractal_dimension_worst
      Unnamed: 32

      0
      842302
      1
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.30010
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890
      NaN

      1
      842517
      1
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.08690
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902
      NaN

      2
      84300903
      1
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.19740
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758
      NaN

      3
      84348301
      1
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.24140
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300
      NaN

      4
      84358402
      1
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.19800
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678
      NaN

      5
      843786
      1
      12.45
      15.70
      82.57
      477.1
      0.12780
      0.17000
      0.15780
      0.08089
      0.2087
      0.07613
      0.3345
      0.8902
      2.217
      27.19
      0.007510
      0.03345
      0.03672
      0.01137
      0.02165
      0.005082
      15.47
      23.75
      103.40
      741.6
      0.1791
      0.5249
      0.5355
      0.1741
      0.3985
      0.12440
      NaN

      6
      844359
      1
      18.25
      19.98
      119.60
      1040.0
      0.09463
      0.10900
      0.11270
      0.07400
      0.1794
      0.05742
      0.4467
      0.7732
      3.180
      53.91
      0.004314
      0.01382
      0.02254
      0.01039
      0.01369
      0.002179
      22.88
      27.66
      153.20
      1606.0
      0.1442
      0.2576
      0.3784
      0.1932
      0.3063
      0.08368
      NaN

      7
      84458202
      1
      13.71
      20.83
      90.20
      577.9
      0.11890
      0.16450
      0.09366
      0.05985
      0.2196
      0.07451
      0.5835
      1.3770
      3.856
      50.96
      0.008805
      0.03029
      0.02488
      0.01448
      0.01486
      0.005412
      17.06
      28.14
      110.60
      897.0
      0.1654
      0.3682
      0.2678
      0.1556
      0.3196
      0.11510
      NaN

      8
      844981
      1
      13.00
      21.82
      87.50
      519.8
      0.12730
      0.19320
      0.18590
      0.09353
      0.2350
      0.07389
      0.3063
      1.0020
      2.406
      24.32
      0.005731
      0.03502
      0.03553
      0.01226
      0.02143
      0.003749
      15.49
      30.73
      106.20
      739.3
      0.1703
      0.5401
      0.5390
      0.2060
      0.4378
      0.10720
      NaN

      9
      84501001
      1
      12.46
      24.04
      83.97
      475.9
      0.11860
      0.23960
      0.22730
      0.08543
      0.2030
      0.08243
      0.2976
      1.5990
      2.039
      23.94
      0.007149
      0.07217
      0.07743
      0.01432
      0.01789
      0.010080
      15.09
      40.68
      97.65
      711.4
      0.1853
      1.0580
      1.1050
      0.2210
      0.4366
      0.20750
      NaN

```
df_copy.head(10)
```

      radius_mean
      texture_mean
      perimeter_mean
      area_mean
      smoothness_mean
      compactness_mean
      concavity_mean
      concave points_mean
      symmetry_mean
      fractal_dimension_mean
      radius_se
      texture_se
      perimeter_se
      area_se
      smoothness_se
      compactness_se
      concavity_se
      concave points_se
      symmetry_se
      fractal_dimension_se
      radius_worst
      texture_worst
      perimeter_worst
      area_worst
      smoothness_worst
      compactness_worst
      concavity_worst
      concave points_worst
      symmetry_worst
      fractal_dimension_worst
      cluster

      0
      17.99
      10.38
      122.80
      1001.0
      0.11840
      0.27760
      0.30010
      0.14710
      0.2419
      0.07871
      1.0950
      0.9053
      8.589
      153.40
      0.006399
      0.04904
      0.05373
      0.01587
      0.03003
      0.006193
      25.38
      17.33
      184.60
      2019.0
      0.1622
      0.6656
      0.7119
      0.2654
      0.4601
      0.11890
      0

      1
      20.57
      17.77
      132.90
      1326.0
      0.08474
      0.07864
      0.08690
      0.07017
      0.1812
      0.05667
      0.5435
      0.7339
      3.398
      74.08
      0.005225
      0.01308
      0.01860
      0.01340
      0.01389
      0.003532
      24.99
      23.41
      158.80
      1956.0
      0.1238
      0.1866
      0.2416
      0.1860
      0.2750
      0.08902
      0

      2
      19.69
      21.25
      130.00
      1203.0
      0.10960
      0.15990
      0.19740
      0.12790
      0.2069
      0.05999
      0.7456
      0.7869
      4.585
      94.03
      0.006150
      0.04006
      0.03832
      0.02058
      0.02250
      0.004571
      23.57
      25.53
      152.50
      1709.0
      0.1444
      0.4245
      0.4504
      0.2430
      0.3613
      0.08758
      0

      3
      11.42
      20.38
      77.58
      386.1
      0.14250
      0.28390
      0.24140
      0.10520
      0.2597
      0.09744
      0.4956
      1.1560
      3.445
      27.23
      0.009110
      0.07458
      0.05661
      0.01867
      0.05963
      0.009208
      14.91
      26.50
      98.87
      567.7
      0.2098
      0.8663
      0.6869
      0.2575
      0.6638
      0.17300
      1

      4
      20.29
      14.34
      135.10
      1297.0
      0.10030
      0.13280
      0.19800
      0.10430
      0.1809
      0.05883
      0.7572
      0.7813
      5.438
      94.44
      0.011490
      0.02461
      0.05688
      0.01885
      0.01756
      0.005115
      22.54
      16.67
      152.20
      1575.0
      0.1374
      0.2050
      0.4000
      0.1625
      0.2364
      0.07678
      0

      5
      12.45
      15.70
      82.57
      477.1
      0.12780
      0.17000
      0.15780
      0.08089
      0.2087
      0.07613
      0.3345
      0.8902
      2.217
      27.19
      0.007510
      0.03345
      0.03672
      0.01137
      0.02165
      0.005082
      15.47
      23.75
      103.40
      741.6
      0.1791
      0.5249
      0.5355
      0.1741
      0.3985
      0.12440
      1

      6
      18.25
      19.98
      119.60
      1040.0
      0.09463
      0.10900
      0.11270
      0.07400
      0.1794
      0.05742
      0.4467
      0.7732
      3.180
      53.91
      0.004314
      0.01382
      0.02254
      0.01039
      0.01369
      0.002179
      22.88
      27.66
      153.20
      1606.0
      0.1442
      0.2576
      0.3784
      0.1932
      0.3063
      0.08368
      0

      7
      13.71
      20.83
      90.20
      577.9
      0.11890
      0.16450
      0.09366
      0.05985
      0.2196
      0.07451
      0.5835
      1.3770
      3.856
      50.96
      0.008805
      0.03029
      0.02488
      0.01448
      0.01486
      0.005412
      17.06
      28.14
      110.60
      897.0
      0.1654
      0.3682
      0.2678
      0.1556
      0.3196
      0.11510
      1

      8
      13.00
      21.82
      87.50
      519.8
      0.12730
      0.19320
      0.18590
      0.09353
      0.2350
      0.07389
      0.3063
      1.0020
      2.406
      24.32
      0.005731
      0.03502
      0.03553
      0.01226
      0.02143
      0.003749
      15.49
      30.73
      106.20
      739.3
      0.1703
      0.5401
      0.5390
      0.2060
      0.4378
      0.10720
      1

      9
      12.46
      24.04
      83.97
      475.9
      0.11860
      0.23960
      0.22730
      0.08543
      0.2030
      0.08243
      0.2976
      1.5990
      2.039
      23.94
      0.007149
      0.07217
      0.07743
      0.01432
      0.01789
      0.010080
      15.09
      40.68
      97.65
      711.4
      0.1853
      1.0580
      1.1050
      0.2210
      0.4366
      0.20750
      1

```
(df['diagnosis'] == df_copy['cluster'])
```

    0      False
    1      False
    2      False
    3       True
    4      False
    5       True
    6      False
    7       True
    8       True
    9       True
    10      True
    11     False
    12     False
    13      True
    14      True
    15      True
    16      True
    17     False
    18     False
    19     False
    20     False
    21     False
    22      True
    23     False
    24     False
    25     False
    26      True
    27     False
    28     False
    29     False
           ...
    539    False
    540    False
    541    False
    542    False
    543    False
    544    False
    545    False
    546    False
    547    False
    548    False
    549    False
    550    False
    551    False
    552    False
    553    False
    554    False
    555    False
    556    False
    557    False
    558    False
    559    False
    560    False
    561    False
    562     True
    563    False
    564    False
    565    False
    566     True
    567    False
    568    False
    Length: 569, dtype: bool

```
((df['diagnosis'] == df_copy['cluster'])).value_counts()
```

    False    486
    True      83
    dtype: int64

```
# Numerator: number of matching labels between the diagnosis and our clusters
((df['diagnosis'] == df_copy['cluster'])).value_counts()[1]
```

    83

```
# Compare numeric diagnosis to clusters and get accuracy percentage
(~(df['diagnosis'] == df_copy['cluster'])).value_counts()[1]/df.shape[0]
```

    0.8541300527240774

# 2) Perform PCA on your dataset first and *then* use k-means clustering.

- You need to standardize your data before PCA.
- First try clustering just on PC1 and PC2 so that you can make a scatterplot of your clustering.
- Then use use a scree plot to decide how many principal components to include in your clustering, and use however many principal components you need in order to retain 90% of the variation of the original dataset

```
# New Dataframe for PCA data
pca_df = df.drop(['diagnosis', 'Unnamed: 32', 'id'], axis=1)
```

```
# Standardize Data to prep for PCA
scaler = StandardScaler()
Z = scaler.fit_transform(pca_df.values)
```

```
# Perform PCA
pca = PCA(20)
pca.fit(Z)
transformed_data = pca.transform(Z)
transformed_data

```

    array([[ 9.19283683,  1.94858307, -1.12316616, ..., -0.54956269,
            -0.1337675 ,  0.3455649 ],
           [ 2.3878018 , -3.76817174, -0.52929269, ...,  0.31829738,
             0.24756463, -0.11413308],
           [ 5.73389628, -1.0751738 , -0.55174759, ..., -0.08797493,
             0.39262636, -0.20453223],
           ...,
           [ 1.25617928, -1.90229671,  0.56273053, ...,  0.02110806,
             0.04201985, -0.08141992],
           [10.37479406,  1.67201011, -1.87702933, ..., -0.30687352,
            -0.31056879,  0.17321637],
           [-5.4752433 , -0.67063679,  1.49044308, ..., -0.18097241,
            -0.18910358,  0.16325407]])

```
plt.scatter(transformed_data[:,0], transformed_data[:,1])
plt.show()
```

## Scree Plot

```
pca_20 = PCA(20)
pca_20.fit(Z)
transformed_data = pca_20.transform(Z)
```

```
# Define scree plot function
def scree_plot(pca):
    '''
    Creates a scree plot associated with the principal components

    INPUT: pca - the result of instantian of PCA in scikit learn

    OUTPUT:
            None
    '''
    num_components=len(pca.explained_variance_ratio_)
    ind = np.arange(num_components)
    vals = pca.explained_variance_ratio_

    plt.figure(figsize=(18, 6))
    ax = plt.subplot(111)
    cumvals = np.cumsum(vals)
    ax.bar(ind, vals)
    ax.plot(ind, cumvals)
    for i in range(num_components):
        ax.annotate(r"%s" % ((str(vals[i]*100)[:3])), (ind[i], vals[i]), va="bottom", ha="center", fontsize=4.5)

    ax.xaxis.set_tick_params(width=0)
    ax.yaxis.set_tick_params(width=2, length=12)

    ax.set_xlabel("Principal Component")
    ax.set_ylabel("Variance Explained (%)")
    plt.title('Explained Variance Per Principal Component')
```

```
scree_plot(pca_20)
```

## K-Means on PCA'd data.

```
# K-Means on PCA data
kmeans = KMeans(2)
kmeans.fit(transformed_data)
print(kmeans.labels_)
```

    [0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     1 1 1 1 1 0 0 1 0 1 0 1 1 1 1 1 0 1 1 0 0 1 1 1 1 0 1 0 0 1 1 0 1 0 1 0 1
     1 0 1 0 0 1 1 0 0 0 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 0 1 1 0 1 1
     1 0 1 1 1 1 0 0 1 1 0 0 1 1 1 1 0 0 0 1 0 0 1 0 1 1 1 0 1 1 0 1 1 1 1 0 1
     1 1 1 1 0 1 1 1 0 1 1 1 1 0 0 1 0 1 1 0 0 1 1 1 0 1 1 1 0 0 1 1 0 0 1 1 1
     1 1 1 1 1 0 1 1 0 0 1 0 0 0 0 1 0 0 0 1 1 1 1 1 1 0 1 0 0 0 0 1 1 0 0 1 1
     1 0 1 1 1 1 1 0 0 1 1 0 1 1 0 0 1 0 1 1 0 1 0 1 1 0 1 1 0 1 0 0 0 1 0 0 0
     0 0 1 0 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1
     1 1 1 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 1 1
     1 1 0 1 0 1 0 1 1 1 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 0
     0 1 0 0 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1
     1 0 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 1 1 1 1 0 1 0 0 1 1 1 1 1 1 1 0 1 1
     1 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 0 0 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 0 1
     1 1 1 1 0 1 0 1 1 1 1 0 1 1 1 1 1 0 0 1 0 1 0 0 0 1 1 1 0 1 1 0 1 1 1 0 0
     1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 0 0 0 0 0 0 1]

```
pca_df['cluster'] = kmeans.labels_
```

## Check your work:

- Compare your PC1, PC2 clustering scatterplot to the clustering scatterplots you made on the raw data
- Calculate accuracy scores for both the PC1,PC2 Principal component clustering and the 90% of explained variance clustering.

How do your accuracy scores -when preprocessing the data with PCA- compare to the accuracy when simply clustering on the raw data?

```
# Compare numeric diagnosis to clusters and get accuracy percentage
(~(df['diagnosis'] == pca_df['cluster'])).value_counts()[1]/df.shape[0]
```

    0.9050966608084359

```
fig, ax = plt.subplots()
colors = {1:'red', 0:'blue'}
grouped = pca_df.groupby('cluster')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='radius_mean', y='texture_mean', label=key, color=colors[key])
plt.show()
```

```

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.

# Riding the wave

We've reached the third lesson of the module _Riding the wave: Convection problems_, where we explore the numerical solution of conservation laws. We learned in the [first lesson](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_01_conservationLaw.ipynb) how to think about conservation laws, starting with the most intuitive case: that of conservation of mass. But there are many physical quantities that can be conserved: energy, for example. Or cars on a road.

Developing a conservation law for traffic in a one-lane roadway is fun, because we can relate to the insights it gives. We've all experienced traffic slowing down to a crawl when the density of cars gets very high, and stepping on the accelerator pedal with glee when there are no cars on the road!

In the previous two lessons, we developed a traffic-flow model and explored [different choices of numerical scheme](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_02_convectionSchemes.ipynb).  Not everything worked as you expected, and by now you might be realizing how some restrictions come about from the numerical methods themselves, while others still are imposed by the models we use. This lesson will develop a better traffic model, and also show you some impressive SymPy kung fu.

## Traffic flow, revisited

### A better flux model

Like you saw in the first lesson, cars obey a typical conservation law,

\begin{equation}
 \frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial x} = 0, \end{equation}

where $F$ is the flux, $F=\rho u$—flux equals density times velocity. From our experience on the road, we know that the traffic speed is a function of traffic density, and we proposed as a first approximation a linear relation between the two. Thus,

\begin{equation}
F(\rho) = \rho \, u_{max}\left(1 - \frac{\rho}{\rho_{max}} \right)
\end{equation}

This flux model meets the two requirements, based on a qualitative view of traffic flow, that:
1.  $u \rightarrow u_{max}$ and $F\rightarrow 0$ when $\rho \rightarrow 0$.
2.  $u \rightarrow 0$ as $\rho \rightarrow \rho_{max}$

However, it leads to some unrealistic or at least improbable results.  For example, note that if the traffic speed is a linear function of density, the flux function will be quadratic (see Figure 1). In this case, the maximum flux will occur when $\rho^{\star} = \rho_{max}/2$, corresponding to a traffic speed $u_{max}/2$.

#### Figure 1. Traffic speed (left) and flux (right) vs. density.

A good question to ask here is:  should the maximum flux on a given stretch of road have a strict dependence on the maximum speed that the roadway allows, be it by physical restrictions or posted speed limits? In other words, do we really expect the maximum flux to increase if we allow arbitrarily high speeds?

Probably not.  But there *should* be some ideal traffic speed, $u^{\star}$, corresponding to an ideal traffic density, $\rho^{\star}$, resulting in the maximum traffic flux:

\begin{equation}F_{\rm max} = \rho^{\star}u^{\star}\end{equation}

Let us improve the initial flux model by taking this observation into account. One thing that we can try is to introduce a flux model that is cubic in $\rho$ instead of quadratic:

\begin{equation}F(\rho) = u_{\rm max}\rho (1 - A\rho - B \rho^2)\end{equation}

This new model still meets the first criterion listed above: $F\rightarrow 0$ when $\rho \rightarrow 0$. Moreover, we impose the following conditions:

*  When $\rho = \rho_{\rm max}$ traffic flux goes to zero:

\begin{equation}F(\rho_{\rm max}) = 0 = u_{\rm max}\, \rho_{\rm max}(1 - A \rho_{\rm max} - B \rho_{\rm max}^2)\end{equation}

*  Based on eq. (3), maximum flux occurs when $\rho = \rho^{\star}$ and $F'(\rho^{\star}) = 0$:

\begin{equation}
F'(\rho^{\star}) = 0  = u_{\rm max}(1 - 2A\rho^{\star} - 3B(\rho^{\star})^2)\end{equation}

*  $u^{\star}$ is obtained when $\rho = \rho^{\star}$:

\begin{equation}
u^{\star} = u_{\rm max}(1 - A \rho^{\star} - B(\rho^{\star})^2)\end{equation}

We have three equations and four unknowns $A,B,\rho^{\star}, u^{\star}$. However, in practice, the ideal traffic speed could be obtained for a given road by observations. Similarly to $u_{max}$ and $\rho_{max}$ it will therefore be taken as a parameter.

### Solving the new flux equation

Equations $(5)$, $(6)$, and $(7)$ are not incredibly difficult to solve with pen and paper. Instead of following that route, we can use [SymPy](http://sympy.org/en/index.html), the symbolic mathematics library of Python.  You used SymPy already in the [Burgers' equation lesson](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_04_1DBurgers.ipynb) of Module 2, _"Space & Time"_, and you will learn some new functionalities here.

We begin by importing SymPy, initializing $\LaTeX$ printing and defining a set of symbolic variables that we'll use in the calculations. Remember: variables are not defined automatically, you have to tell SymPy that a name will correspond to a symbolic variable by using the keyword `symbols`. This behavior is different from many other symbolic math systems that implicitly construct symbols for you, so you may be perplexed if you've used these other sytems before. The reason for this behavior in SymPy is that the system is fully built on Python, a general-purpose language that needs you to define all objects before using them.

```python
import sympy
sympy.init_printing()

u_max, u_star, rho_max, rho_star, A, B = sympy.symbols('u_max u_star rho_max rho_star A B')
```

Notice that we used the special character `_` (the under-dash) to create a symbol with a subscript. Here, for example, we've created the symbols $u_{max}$ and $u_{star}$, representing $u_{\rm max}$ and $u^{\star}$ ($u$-star) in the equations, and assigned them to the variable names `u_max` and `u_star`. SymPy also allows you to create symbols with a superscript by means of the special character `^`. Be careful, though: SymPy is built on Python, so you denote an exponent in an expression by `**` (this may also be different from other symbolic math systems that you have used in the past).

Next, use `sympy.Eq()` to define the three equations—corresponding to Equations $(5)$, $(6)$ and $(7)$, above—in terms of symbolic variables. The function `sympy.Eq()` creates an equality between two SymPy objects, passed as arguments separated by a comma. We need to remember that the equal sign in Python is used for variable assignment, and for that reason it cannot be used to build a symbolic equation with SymPy. That is why we need `sympy.Eq()` to create symbolic equalities. But the equal sign _is_ used to _assign_ an equation to a name: here, `eq1`, `eq2`, `eq3` are names for the symbolic equalities we create with `sympy.Eq()`.

```python
eq1 = sympy.Eq( 0, u_max*rho_max*(1 - A*rho_max-B*rho_max**2) )
eq2 = sympy.Eq( 0, u_max*(1 - 2*A*rho_star-3*B*rho_star**2) )
eq3 = sympy.Eq( u_star, u_max*(1 - A*rho_star - B*rho_star**2) )
```

Check this out: you can display these equations in pretty typeset mathematics just by executing their name in a code cell:

```python
eq1
```

```python
eq2
```

```python
eq3
```

As stressed above, we have three equations with three unknowns (assuming we have some observed value of $u_{star}$, corresponding to the ideal traffic speed)—there must be a solution!

To eliminate the term with $B$ in `eq2`, leaving it only in terms of $A$ and $\rho^{\star}$, we might subtract `3*eq3`.  But this will not work in SymPy if you attempt equation subtraction using the equation names. Just like we couldn't use the equal sign (which in Python means assignment) to create a symbolic equality (needing `sympy.Eq` instead), we can't use mathematical operators directly on the SymPy equations, because they are really _equalities_. What does it mean to subtract two _equalities_? It doesn't make sense. Try it ...

```python
eq2 - 3*eq3
```

See?  SymPy just printed out what you suggested but it did not manipulate algebraically the left and right sides of `eq2` like we were aiming for. What we _can_ do is create a _new_ equation, perform the left-hand side (LHS) and right-hand side (RHS) operations separately and then recombine them into our desired result. For this, it is helpful to know that there are built-in properties of a SymPy equality for the left-hand side and the right-hand side of the equality. Check it out:

```python
eq4 = sympy.Eq(eq2.lhs - 3*eq3.lhs, eq2.rhs - 3*eq3.rhs)
eq4
```

That still needs a little work. SymPy offers several methods to help reduce expressions. You can use [`simplify()`](http://docs.sympy.org/latest/modules/simplify/simplify.html?highlight=simplify#sympy.simplify.simplify.simplify) to attempt to make the expression "simpler," but you can imagine that the quality of an expression being simple is not well defined. One may expect the simpler expression to be shorter, maybe; but not always. The SymPy `simplify()` function applies several strategies, heuristically, to give you an equivalent reduced expression. But some expressions are uncooperative and `simplify()` gives up, returning the argument unchanged. Let's see what it can do with our expression `eq4`.

```python
eq4.simplify()
```

That is actually a useful result. We see that `eq4` allows solving for $\rho^{\star}$ in terms of $A$, for example (remember that $u_{max}$ is known: the road's maximum velocity). Then we could try to manipulate `eq1` to solve for $B$ in terms of $A$, as well, so that we can substitute back in `eq2` leaving everything in terms of $A$. We can do all that without the help of `simplify()`, but using it interactively to reason about the long expression helped us to decide on a course of action.

Another SymPy function that can help us examine complicated expressions is [`expand()`](http://docs.sympy.org/latest/modules/core.html?highlight=expand#sympy.core.function.expand). Its purpose is to expand bracketed factors in expressions and group powers of symbols. Let's see what that does here. Notice first that `eq4` hasn't changed; we just printed the result of applying `simplify()` to it.

```python
eq4
```

And now we print the result of applying `expand()` to it.

```python
eq4.expand()
```

That's very similar to our previous result, except without the bracketed factor. Simplifying an expression can be accomplished by expanding or factoring terms, or a combination of the two. Whether to `simplify()` or `expand()` depends on the situation and you just have to experiment.

We now have an idea of what to do with our three equations. We'll get expressions in terms of $A$ from `eq4` and `eq1` and substitute them back into `eq2`. For that, we can use the SymPy functions [`solve()`](http://docs.sympy.org/dev/modules/solvers/solvers.html#sympy.solvers.solvers.solve) and `subs()`, respectively. The arguments to `solve()` are the equality that needs to be solved, and the symbol to solve for.

**Note**: `sympy.solve()` always returns results in a *list*, since you often end up with multiple solutions for a given variable.  These linear equations will only return one solution, so we can skip a step and ask right away for the `[0]`-th element of the list.

```python
rho_sol = sympy.solve(eq4,rho_star)[0]
rho_sol
```

```python
B_sol = sympy.solve(eq1,B)[0]
B_sol
```

The arguments to the SymPy function [`subs()`](http://docs.sympy.org/dev/modules/core.html#sympy.core.basic.Basic.subs) are given as (old, new) pairs, where the "new" expression substitutes the "old" one. So here, in `eq2`, we substitute `rho_sol` in place of `rho_s` and we substitute `B_sol` in place of `B`.

```python
quadA = eq2.subs([(rho_star, rho_sol), (B,B_sol)])
quadA
```

```python
quadA.simplify()
```

It's a little bit ugly, but that is quadratic in $A$, and that means we can solve for the roots of the equation. SymPy's `solve()` function in this case will return a list with the two roots. They are long expressions, so let's print each root separately.

```python
A_sol = sympy.solve(quadA, A)
```

```python
A_sol[0]
```

```python
A_sol[1]
```

### Evaluating the new flux equation

Quadratic equations, of course, have two solutions. Here we have to select the positive root, otherwise our model would have an inconsistency. Are you able to see why?  To determine its value, we need to fill in some actual numbers into the model.

Let's start with the same numerical values that we used in [lesson 1](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_01_conservationLaw.ipynb) for $\rho_{\rm max}$ and $u_{\rm max}$. But we also have to supply a value for $u^{\star}$, a quantity that should be experimentally observed in a given road. We propose $u^{\star} = 0.7\, u_{\rm max}$ for this exercise.  This would correspond to 84 km/h for a highway with a 120-km/h speed limit, for example.

Let's numerically evaluate the solutions for $A$ using the following values:

\begin{align}
\rho_{\rm max} &=10.0 \nonumber\\ u_{\rm max} &=1.0 \nonumber\\ u^{\star} &=0.7 \nonumber
\end{align}

Now evaluate the numeric result for each root of $A$ using the `evalf()` function, where you pass the numeric substitution as an argument. It's very cool.  Let's try the `[0]`-th solution for $A$, first:

```python
aval = A_sol[0].evalf(subs={u_star: 0.7, u_max:1.0, rho_max:10.0} )
aval
```

That's the positive root.  Let's evaluate the negative root (but we're not going to save this result, because we don't need it!):

```python
A_sol[1].evalf(subs={u_star: 0.7, u_max:1.0, rho_max:10.0} )
```

Now, numerically evaluate $B$ in the same way:

```python
bval = B_sol.evalf(subs={rho_max:10.0, A:aval} )
bval
```

### Turn off $\LaTeX$

$\LaTeX$ output is great when we're looking at algebraic expressions, but it's a little too much for simple numeric output.  Let's turn it off for the rest of the exercise:

```python
sympy.init_printing(use_latex=False)
```

## Green light: take 2

Let's re-examine the green-light problem from [lesson 1](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_01_conservationLaw.ipynb) but using our newly computed traffic-flux equation.  We shouldn't have to change much—in fact, we can simply create a new `computeF` function and leave the rest of the code unchanged.

There's one last bit of housekeeping to do so we don't run into trouble -- we used the variables `rho_max` and `u_max` in our SymPy code and we defined them as SymPy `symbols`.  You can check on the status of a variable using `type()`.

```python
print(type(rho_max), type(u_max))
```

    <class 'sympy.core.symbol.Symbol'> <class 'sympy.core.symbol.Symbol'>

If we try to use SymPy variables with NumPy arrays, Python is going to be very unhappy.  We can re-define them as floats and avoid that messy situation.

```python
rho_max = 10.
u_max = 1.
```

```python
def computeF(u_max, rho, aval, bval):
    return u_max*rho*(1 - aval*rho-bval*rho**2)
```

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

```python
def rho_green_light(nx, rho_light):
    """Computes "green light" initial condition with shock, and linear distribution behind

    Parameters
    ----------
    nx        : int
        Number of grid points in x
    rho_light : float
        Density of cars at stoplight

    Returns
    -------
    rho_initial: array of floats
        Array with initial values of density
    """
    rho_initial = numpy.arange(nx)*2./nx*rho_light  # Before stoplight
    rho_initial[int((nx-1)/2):] = 0

    return rho_initial
```

```python
#Basic initial condition parameters
#defining grid size, time steps
nx = 81
nt = 30
dx = 4.0/(nx-1)

x = numpy.linspace(0,4,nx)

rho_light = 5.5

```

```python
rho_initial = rho_green_light(nx, rho_light)
```

```python
pyplot.plot(x, rho_initial, color='#003366', ls='-', lw=3)
pyplot.ylim(-0.5,11.);
```

```python
def ftbs(rho, nt, dt, dx, rho_max, u_max):
    """ Computes the solution with forward in time, backward in space

    Parameters
    ----------
    rho    : array of floats
            Density at current time-step
    nt     : int
            Number of time steps
    dt     : float
            Time-step size
    dx     : float
            Mesh spacing
    rho_max: float
            Maximum allowed car density
    u_max  : float
            Speed limit

    Returns
    -------
    rho_n : array of floats
            Density after nt time steps at every point x
    """

    #initialize our results array with dimensions nt by nx
    rho_n = numpy.zeros((nt,len(rho)))

    #copy the initial u array into each row of our new array
    rho_n[0,:] = rho.copy()

    for t in range(1,nt):
        F = computeF(u_max, rho, aval, bval)
        rho_n[t,1:] = rho[1:] - dt/dx*(F[1:]-F[:-1])
        rho_n[t,0] = rho[0]
        rho_n[t,-1] = rho[-1]
        rho = rho_n[t].copy()

    return rho_n
```

```python
sigma = 1.
dt = sigma*dx/u_max

rho_n = ftbs(rho_initial, nt, dt, dx, rho_max, u_max)
```

```python
from matplotlib import animation
from JSAnimation.IPython_display import display_animation
```

```python
fig = pyplot.figure();
ax = pyplot.axes(xlim=(0,4),ylim=(-1,8),xlabel=('Distance'),ylabel=('Traffic density'));
line, = ax.plot([],[],color='#003366', lw=2);

def animate(data):
    x = numpy.linspace(0,4,nx)
    y = data
    line.set_data(x,y)
    return line,

anim = animation.FuncAnimation(fig, animate, frames=rho_n, interval=50)
display_animation(anim, default_mode='once')
```

    <input id="_anim_sliderUPLORRVQRWHARTBF" type="range" style="width:350px" name="points" min="0" max="1" step="1" value="0" onchange="animUPLORRVQRWHARTBF.set_frame(parseInt(this.value));"></input>

    <button onclick="animUPLORRVQRWHARTBF.slower()">–</button>
    <button onclick="animUPLORRVQRWHARTBF.first_frame()"></button>
    <button onclick="animUPLORRVQRWHARTBF.previous_frame()"></button>
    <button onclick="animUPLORRVQRWHARTBF.reverse_animation()"></button>
    <button onclick="animUPLORRVQRWHARTBF.pause_animation()"></button>
    <button onclick="animUPLORRVQRWHARTBF.play_animation()"></button>
    <button onclick="animUPLORRVQRWHARTBF.next_frame()"></button>
    <button onclick="animUPLORRVQRWHARTBF.last_frame()"></button>
    <button onclick="animUPLORRVQRWHARTBF.faster()">+</button>
  <form action="#n" name="_anim_loop_selectUPLORRVQRWHARTBF" class="anim_control">
    <input type="radio" name="state" value="once" checked> Once </input>
    <input type="radio" name="state" value="loop" > Loop </input>
    <input type="radio" name="state" value="reflect" > Reflect </input>
  </form>

That definitely looks different!  Do you think this is more or less accurate than our previous model?

##### Dig Deeper

The new traffic-flux model that we developed here changes the way that traffic patterns evolve. In this lesson, we only experimented with the most basic scheme: forward-time, backward-space.  Try to implement the green-light problem using one of the second-order schemes from [Lesson 2](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_02_convectionSchemes.ipynb).

## References

* Neville D. Fowkes and John J. Mahony, *"An Introduction to Mathematical Modelling,"* Wiley & Sons, 1994. Chapter 14: Traffic Flow.

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>


Notes

Slide 30, no factor of 2 for Ld

Slide 28 clarify which region is n-type or p-type. Bullet points 3 and 4 are contradictory

Finite difference matrix incorrect, line 3 should be 0 1 -4 0 0 1 0 0 0

Slide 31 boundary conditions should not be multiplied by h^2

```python
# define constants
import time
start_time = time.time()
import math

q = 1.602e-19
eps = 11.68*8.85e-12
boltzmann = 1.38e-23
ni_cm = 1e10
ni = ni_cm*1e6
T = 298
Vt = boltzmann*T/q
Ld = math.sqrt(eps*Vt/q/ni)
```

```python
# from sympy import symbols, init_printing
import symengine as se
from symengine import symbols
from symengine.lib.symengine_wrapper import DenseMatrix, Symbol
import numpy as np
from matplotlib import pylab as plt
%matplotlib notebook
#sympy.init_printing(use_latex=True)

box_length = 10e-6*3/Ld
dimension = 18
elements = dimension*dimension
h_val = box_length/dimension
Nd_val = -1e15/ni_cm
Na_val = 1e17/ni_cm
#bc_val = 0.0
bc_val = 1/Vt
max_bc = 1/Vt
#v_sym = np.full((dimension, dimension), None, dtype=sympy.symbol.Symbol)
v_sym = DenseMatrix(dimension, dimension, [0]*elements)
index = 0
for x in range(dimension):
    for y in range(dimension):
        v_sym[x, y] = Symbol('psi_'+str(index))
        index += 1

def get_xy(index):
    x = math.floor(index/dimension)
    y = index - x*dimension
    return (x, y)
def to_index(x, y):
    return y + x*dimension

def to_val(sym):
    return sym.subs(h, h_val).subs(Na, Na_val).subs(Nd, Nd_val).subs(bc, bc_val)
```

```python
def get_doping(x, y):
    cutoff = math.floor(dimension/2.0)
    if (x+y <= cutoff):
        return Nd
    else:
        return Na
```

```python
def get_bc(x, y, pos):
    # left is 0, top is 1, right is 2, bottom is 3
    grid_scale = dimension/3
    if pos == 0 and x == 0:
        if y < grid_scale:
            return bc
        else:
            return 0
    elif pos == 1 and y == 0:
        if x < grid_scale:
            return bc
        else:
            return 0
    else:
        return 0

def get_neighbor(x, y, pos, matrix=v_sym):
    # left is 0, top is 1, right is 2, bottom is 4
    offsets = [(-1, 0), (0, -1), (1, 0), (0, 1)]
    x += offsets[pos][0]
    y += offsets[pos][1]
    if x < 0 or x >= dimension or y < 0 or y >= dimension:
        return 0
    else:
        return matrix[x, y]

def get_top(x, y):
    return get_bc(x, y, 1) + get_neighbor

```

```python
# full formulation
h = Symbol('h')
Nd = Symbol('Nd')
Na = Symbol('Na')
bc = Symbol('bc')
#v_func_sym = np.full((dimension, dimension), None, dtype=sympy.symbol.Symbol)
v_func_sym = DenseMatrix(dimension, dimension, [0]*elements)
for x in range(dimension):
    for y in range(dimension):
        func = -4*v_sym[x, y]
        for pos in range(4):
            func += get_neighbor(x, y, pos)
            func += get_bc(x, y, pos)
        func += h**2*(se.exp(-v_sym[x, y]) - se.exp(v_sym[x, y]) + get_doping(x, y))
        v_func_sym[x, y] = func
#v_func_sym = np.array(list(v_func_sym.flat))
v_func_sym = v_func_sym.reshape(elements, 1)
```

```python
# compute the jacobian
v_sym_flat = v_sym.reshape(elements, 1)
jacobian_sym = v_func_sym.jacobian(v_sym_flat)

```

```python
# verify boundary conditions

bcs = np.full((4), None, dtype=DenseMatrix)
for i in range(4):
    bcs[i] = DenseMatrix(dimension, dimension, [0]*elements)
    for x in range(dimension):
        for y in range(dimension):
            bcs[i][x, y] = get_bc(x, y, i)

bcs_sub = np.full((4), None, dtype=np.ndarray)
for i in range(4):
    bcs_sub[i] = np.array(bcs[i].subs(bc, bc_val).tolist())

for i in range(4):
    plt.subplot(2, 2, i+1)
    cont = plt.contourf(bcs_sub[i], 1)
    plt.colorbar(cont, ticks=np.unique(bcs))
plt.show(cont)
```

```python
# verify doping
doping = DenseMatrix(dimension, dimension, [0]*elements)
for x in range(dimension):
    for y in range(dimension):
        doping[x, y] = get_doping(x, y)
doping = doping.subs(Na, Na_val).subs(Nd, Nd_val)
cont = plt.contourf(doping.tolist(), 0)
plt.show(cont)
```

```python
# homogeneous formulation
v_func_homog = DenseMatrix(dimension, dimension, [0]*elements)
for x in range(dimension):
    for y in range(dimension):
        v_func_homog[x, y] = -4*v_sym[x, y]
        for pos in range(4):
            v_func_homog[x, y] = v_func_homog[x, y] + get_neighbor(x, y, pos)

v_coeffs = v_func_homog.reshape(elements, 1).jacobian(v_sym_flat)
v_coeffs = np.array(v_coeffs.tolist(), dtype=np.double)
#v_coeffs
```

```python
#homogeneous solution
rhs = DenseMatrix(elements, 1, [0]*elements)
for i in range(elements):
    (x, y) = get_xy(i)
    rhs[i] = h*h*-get_doping(x, y)
    for j in range(4):
        rhs[i] = rhs[i] - get_bc(x, y, j)

rhs = np.array(to_val(rhs).tolist(), dtype=np.double)
homog_soln = np.linalg.solve(v_coeffs, rhs)
homog_soln*= Vt
#print homog_soln.reshape((dimension, dimension))

```

```python
# Test solution
sol = homog_soln.reshape((dimension, dimension))

def verify_soln(x, y):
    neighbors = 0
    for i in range(4):
        neighbors += Vt*get_bc(x, y, i) + get_neighbor(x, y, i, sol)
    del2 = to_val((neighbors - 4*sol[x][y])/(h*Ld)**2)
    rhs = to_val((-q*get_doping(x, y)*ni/eps))
    return del2, rhs, del2/rhs

print verify_soln(2, 2)[2]
print verify_soln(0, 0)[2]
```

    1.00000000000015
    1.00000000000004

```python
cont = plt.contourf(sol)
plt.colorbar(cont)
plt.show(cont)
```

```python
# evaluate f(v)
import re
pattern = 'psi_([0-9]+)'
prog = re.compile(pattern)

vfunc_indices = np.empty((elements), dtype=np.ndarray)
for i in range(elements):
    vfunc_indices[i] = np.unique(prog.findall(str(v_func_sym[i]))).astype(int)

def evaluate_f(v_func, soln):
    temp_eval = np.copy(v_func)
    for i in range(elements):
        for j in vfunc_indices[i]:
            temp_eval[i] = temp_eval[i].subs(v_sym_flat[j], soln[j])
    return map(lambda x: x.to_double(), temp_eval)

```

```python
# evaluate jacobian
# jacobian is independent of bc
jacobian = to_val(jacobian_sym)
jacobian_eval = np.copy(jacobian)
np.fill_diagonal(jacobian_eval, 0)
jacobian_eval = jacobian_eval.astype(np.double)

# to make indexing faster
jacobian_diag = np.diag(jacobian.tolist())

range_index = range(elements)
def evaluate_jacobian(soln):
    for i in range_index:
        jacobian_eval[i][i] = jacobian_diag[i].subs(v_sym_flat[i], soln[i])
    return jacobian_eval
```

```python
#initial_soln = np.zeros(elements)
#%timeit evaluate_f(initial_soln)
#%timeit evaluate_jacobian(initial_soln)
```

```python
# solve
source_steps = 18

no_plots = 6
plot_step = int(source_steps/no_plots)
plot_index = 0

v_data = np.empty((no_plots, dimension, dimension), dtype=np.double)
f_data = np.empty((no_plots), dtype=np.ndarray)

max_iterations = 30
tol = 1e-8
initial_sol = np.zeros(elements)

curr_sol = initial_sol
source_step = 0

bc_vals = np.linspace(0, -1/Vt, source_steps)

for source_step in range(source_steps):

    bc_val = bc_vals[source_step]
    v_func = np.squeeze(to_val(v_func_sym).tolist())
    f_sol = evaluate_f(v_func, curr_sol)
    f_sum = np.absolute(f_sol).sum()
    min_t = -20
    print'bc = {} :'.format(bc_val*Vt),
    for i in range(max_iterations):
        print '{}..'.format(i),
        f_sol = evaluate_f(v_func, curr_sol)
        jacobian_sol = evaluate_jacobian(curr_sol)
        del_v = -np.linalg.solve(jacobian_sol, f_sol)
        t = 2
        f_sum = np.absolute(f_sol).sum()
        # line search
        while t > min_t:
            scale = 2**t
            temp_sol = curr_sol + scale*del_v
            temp_f_sol = evaluate_f(v_func, temp_sol)
            temp_sum = np.absolute(temp_f_sol).sum()
            if temp_sum < f_sum:
                curr_sol = temp_sol
                f_sol = temp_f_sol
                break
            #print 't = ', t
            t -= 1
        if t <= min_t:
            print("No t reduces F(V)")
            break
        if temp_sum < tol:
            break
    if source_step % plot_step == 0:
        v_data[plot_index] = Vt*curr_sol.reshape((dimension, dimension))
        f_data[plot_index] = np.array(f_sol).reshape((dimension, dimension))
        plot_index += 1
    print '\n'

```

    bc = 0.0 : 0.. 1.. 2.. 3.. 4.. 5.. 6.. 7.. 8.. 9.. 10.. 11.. 12.. 13.. 14.. 15.. 16..

    bc = -0.0588235294118 : 0.. 1.. 2.. 3..

    bc = -0.117647058824 : 0.. 1.. 2.. 3..

    bc = -0.176470588235 : 0.. 1.. 2.. 3..

    bc = -0.235294117647 : 0.. 1.. 2.. 3..

    bc = -0.294117647059 : 0.. 1.. 2.. 3..

    bc = -0.352941176471 : 0.. 1.. 2.. 3..

    bc = -0.411764705882 : 0.. 1.. 2.. 3..

    bc = -0.470588235294 : 0.. 1.. 2.. 3..

    bc = -0.529411764706 : 0.. 1.. 2.. 3..

    bc = -0.588235294118 : 0.. 1.. 2.. 3..

    bc = -0.647058823529 : 0.. 1.. 2.. 3..

    bc = -0.705882352941 : 0.. 1.. 2.. 3..

    bc = -0.764705882353 : 0.. 1.. 2.. 3..

    bc = -0.823529411765 : 0.. 1.. 2.. 3..

    bc = -0.882352941176 : 0.. 1.. 2.. 3..

    bc = -0.941176470588 : 0.. 1.. 2.. 3..

    bc = -1.0 : 0.. 1.. 2.. 3..

```python
default_plot_size = plt.rcParams['figure.figsize']
#plt.rcParams['figure.figsize'] = [6, 6]

fig, axes = plt.subplots(nrows=2, ncols=3)
axes = axes.flat

for i in range(no_plots):
    ax = axes[i]
    cont = ax.contourf(v_data[i])
    ticks = np.linspace(v_data[i].min(), v_data[i].max(), 5)
    plt.colorbar(cont, ax=ax, ticks=ticks, format='%.2f')
    ax.axis('off')
    ax.set_aspect('equal')
    plot_index += 1

plt.subplots_adjust(wspace=0.3)
plt.show()
plt.rcParams['figure.figsize'] = default_plot_size
```

```python
v_data.max()
```

    0.41375815577403896

```python
%matplotlib notebook
cont = plt.contourf(v_data[-1])
plt.colorbar(cont, ax=cont.ax, format='%.2f')
cont.ax.set_aspect('equal')
plt.show()
```

    <IPython.core.display.Javascript object>

```python
%matplotlib notebook
#cont = plt.contourf(v_data[0], vmax=0.5, vmin=-0.5)
cont = plt.contourf(v_data[0])
plt.colorbar(cont, ax=cont.ax, format='%.2f')
cont.ax.set_aspect('equal')
plt.show()
```

    <IPython.core.display.Javascript object>

```python
v_data[0][0][0]
```

    -0.29182775725095933

```python

```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Crossover Range
***

For an escort type jammer, the crossover range may be expressed as (Equation 11.18)

\begin{equation}\label{eq:crossover_escort}
    r_c = \left[ \frac{P_t\, G_t^2 \,\sigma\, r_j^2 \,B_j}{{ERP}\, \, G_r \,4 \,\pi \,B_r \,L_r}\right]^{1/4} \hspace{0.5in} \text{(m)},
\end{equation}

and for a self-screening type jammer (Equation 11.19)

\begin{equation}\label{eq:crossover_self}
    r_c = \sqrt{\frac{P_t\, G_t^2 \,\sigma\, B_j}{{ERP}\, \, G_r \,4 \,\pi \,B_r \,L_r}} \hspace{0.5in} \text{(m)}
\end{equation}
***

Begin by getting the library path

```python
import lib_path
```

Set up the jammer effective radiated power (dBW)

```python
jammer_erp = [10, 40]
```

Import the `linspace` routine from `scipy` to create the jammer ERP array

```python
from numpy import linspace

jammer_erp_vector = linspace(float(jammer_erp[0]), float(jammer_erp[1]), 1000)
```

Set the peak power (W), antenna gain (dB), target RCS (dBsm), jammer bandwidth (Hz), jammer ERP (dBW), radar bandwidth (Hz), and losses (dB) for a self-screening type jammer

```python
peak_power = 50e3

antenna_gain = 40

target_rcs = 3.0

jammer_bandwidth = 20e6

radar_bandwidth = 15e6

losses = 9.0
```

Set up the keyword args

```python
kwargs = {'peak_power': peak_power,

          'antenna_gain': 10 ** (antenna_gain / 10.0),

          'target_rcs': 10 ** (target_rcs / 10.0),

          'jammer_bandwidth': jammer_bandwidth,

          'effective_radiated_power': 10 ** (jammer_erp_vector / 10.0),

          'radar_bandwidth': radar_bandwidth,

          'losses': 10 ** (losses / 10.0)}
```

Import the `crossover_range_selfscreen` routine from countermeasures

```python
from Libs.ecm.countermeasures import crossover_range_selfscreen
```

Calculate the crossover range for a self-screening type jammer (m)

```python
crossover_range = crossover_range_selfscreen(**kwargs)
```

Import the `matplotlib` routines for plotting the crossover range for a self-screening type jammer

```python
from matplotlib import pyplot as plt

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(jammer_erp_vector, crossover_range, '')

# Set the plot title and labels

plt.title('Crossover Range', size=14)

plt.xlabel('Jammer ERP (dBW)', size=12)

plt.ylabel('Crossover Range (m)', size=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the tick label size

plt.tick_params(labelsize=12)
```

For an escort type jammer, set the jammer range (m) and the antenna gain in the direction of the jammer (dB)

```python
jammer_range = 10e3

antenna_gain_jammer_direction = 5.0
```

Set up the keyword args

```python
kwargs = {'peak_power': peak_power,

          'antenna_gain': 10 ** (antenna_gain / 10.0),

          'target_rcs': 10 ** (target_rcs / 10.0),

          'jammer_range': jammer_range,

          'jammer_bandwidth': jammer_bandwidth,

          'effective_radiated_power': 10 ** (jammer_erp_vector / 10.0),

          'radar_bandwidth': radar_bandwidth,

          'losses': 10 ** (losses / 10.0),

          'antenna_gain_jammer_direction': 10 ** (antenna_gain_jammer_direction / 10.0)}
```

Import the `crossover_range_escort` routine from countermeasures

```python
from Libs.ecm.countermeasures import crossover_range_escort
```

Calculate the crossover range for an escort type jammer (m)

```python
crossover_range = crossover_range_escort(**kwargs)
```

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(jammer_erp_vector, crossover_range, '')

# Set the plot title and labels

plt.title('Crossover Range', size=14)

plt.xlabel('Jammer ERP (dBW)', size=12)

plt.ylabel('Crossover Range (m)', size=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the tick label size

plt.tick_params(labelsize=12)
```


# Regressão linear 1: equações normais

Dado o dataset $(\mathbf{x}_{1}, y_{1}), \dots ,(\mathbf{x}_{N}, y_{N})$ onde $\mathbf{x}_i \in \mathbb{R}^{d}$ e $y_i \in \mathbb{R}$, podemos aproximar a função desconhecida $f:\mathbb{R}^{d} \rightarrow \mathbb{R}$ (lembrando que $y_i =f(\mathbf{x}_i)$) por meio de um modelo linear $h$:
$$
h(\mathbf{x}_{i}; \mathbf{w}, b) = \mathbf{w}^\top  \mathbf{x}_{i} + b
$$

Note que $h(\mathbf{x}_{i}; \mathbf{w}, b)$ é na verdade uma [transformação afim](https://en.wikipedia.org/wiki/Affine_transformation) de $\mathbf{x}_{i}$. Como em outros lugares, vamos usar o termo "linear" também para caracterizar uma transformação afim.

A saída de $h$ é uma transformação linear de $\mathbf{x}_{i}$. Usamos a notação $h(\mathbf{x}_{i}; \mathbf{w}, b)$ para deixar claro que $h$ é um modelo parametrizado, i.e., a transformação $h$ é definida pelos parâmetros $\mathbf{w}$ e $b$. Podemos pensar no vetor $\mathbf{w}$ como um vetor de *pesos* controlando o efeito de cada *feature* na predição.

Adicionando uma feature a mais na obsevação $\mathbf{x}_{i}$ (com o valor 1) -- coordenada artificial -- podemos simplificar a notação do modelo:

$$
h(\mathbf{x}_{i}; \mathbf{w}) = \hat{y}_{i} = \mathbf{w}^\top  \mathbf{x}_{i}
$$

Procuramos os melhores parâmetros $\mathbf{w}$ de modo que a predição $\hat{y}_{i}$ seja a mais próxima de $y_{i}$ de acordo com alguma métrica de erro. Usando o *erro quadrárico médio* como tal métrica podemos obter a seguinte função de custo:

\begin{equation}
J(\mathbf{w}) = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_{i} - y_{i})^{2}
\end{equation}

Desse modo, a tarefa de achar a função $h$ mais próxima de $f$ se torna a tarefa de encontrar os valores de $\mathbf{w}$ para minimizar $J(\mathbf{w})$.

**Aqui vamos começar a explorar esse modelo olhando para um dataset bem simples**

```python
# all imports
import numpy as np
import time
from util.util import get_housing_prices_data, r_squared
from util.plots import plot_points_regression

%matplotlib inline
```

### O dataset

Os dados que vamos trabalhar vão ser dados artificiais. Vamos pegar 100 observações com apenas uma *feature* (metros quadrados de um imóvel) e com isso vamos associar um valor (o preço desse imóvel em $). Nossa tarefa é construir um modelo que consiga predizer o valor dos imóveis.

```python
X, y = get_housing_prices_data(N=100)
```

### Plotando os dados

Acima temos algumas informações sobre os dados, podemos também visualizar cada ponto.

```python
plot_points_regression(X,
                       y,
                       title='Real estate prices prediction',
                       xlabel="m\u00b2",
                       ylabel='$')
```

### Equações normais

Dados $f:\mathbb{R}^{n\times m} \rightarrow \mathbb{R}$ e $\mathbf{A} \in \mathbb{R}^{n\times m}$, definimos o gradiente de $f$ com respeito a $\mathbf{A}$ como:

\begin{equation*}
\nabla_{\mathbf{A}}f = \frac{\partial f}{\partial \mathbf{A}} =  \begin{bmatrix}
\frac{\partial f}{\partial \mathbf{A}_{1,1}} & \dots & \frac{\partial f}{\partial \mathbf{A}_{1,m}} \\
\vdots &  \ddots & \vdots \\
\frac{\partial f}{\partial \mathbf{A}_{n,1}} &  \dots & \frac{\partial f}{\partial \mathbf{A}_{n,m}}
\end{bmatrix}
\end{equation*}

Seja $\mathbf{X} \in \mathbb{R}^{N\times m}$ a matriz cujas linhas são as observações do dataset (também chamada de *design matrix*) e seja $\mathbf{y} \in \mathbb{R}^{N}$ o vetor contendo todos os valores de $y_{i}$ (i.e., $\mathbf{X}_{i,:} = \mathbf{x}_{i}$ e $\mathbf{y}_{i} =y_{i}$). É fácil checar que:

\begin{equation}
J(\mathbf{w}) = \frac{1}{N}(\mathbf{X}\mathbf{w} - \mathbf{y})^{T}(\mathbf{X}\mathbf{w} - \mathbf{y})
\end{equation}

Usando certos conceitos básicos de derivada com matrizes podemos chegar no gradiente de $J(\mathbf{w})$ com respeito a $\mathbf{w}$:

\begin{equation}
\nabla_{\mathbf{w}}J(\mathbf{w}) = \frac{2}{N} (\mathbf{X}^{T}\mathbf{X}\mathbf{w} -\mathbf{X}^{T}\mathbf{y})
\end{equation}

Assim, quando $\nabla_{\mathbf{w}}J(\mathbf{w}) = 0$ temos que

\begin{equation}
\mathbf{X}^{T}\mathbf{X}\mathbf{w} = \mathbf{X}^{T}\mathbf{y}
\end{equation}

Desse modo,

\begin{equation}
\mathbf{w} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}
\end{equation}

A solução dada por essas equações são conhecidas como **equações normais**. Note que esse tipo de solução tem um custo, pois conforme cresce o número de variáveis, o tempo da inversão da matriz fica proibitivo. Vale a pena ler [esse material](http://cs229.stanford.edu/notes/cs229-notes1.pdf) para ver o argumento com mais detalhes.

### Exercício
Implemente a predição usando o método de equações normais. Usando apenas a biblioteca **NumPy** você deve completar a função abaixo (uma introdução rápida a certas funcionalidades dessa biblioteca pode ser encontrada [aqui](http://cs231n.github.io/python-numpy-tutorial/)). Lembre que $\mathbf{X} \in \mathbb{R}^{N\times d}$, você precisa adicionar uma componente com apenas 1s a cada observação em $\mathbf{X}$ e realizar a computação descrita acima.

```python
def normal_equation_prediction(X, y):
    """
    Calculates the prediction using the normal equation method.
    You should add a new row with 1s.

    :param X: design matrix
    :type X: np.ndarray(shape=(N, d))
    :param y: regression targets
    :type y: np.ndarray(shape=(N, 1))
    :return: prediction
    :rtype: np.ndarray(shape=(N, 1))
    """
    # YOUR CODE HERE:
    raise NotImplementedError("falta completar a função normal_equation_prediction")
    # END YOUR CODE
    return prediction

```

```python
prediction = normal_equation_prediction(X, y)
r_2 = r_squared(y, prediction)
plot_points_regression(X,
                       y,
                       title='Real estate prices prediction',
                       xlabel="m\u00b2",
                       ylabel='$',
                       prediction=prediction,
                       legend=True,
                       r_squared=r_2)
```

Você pode usar a métrica [$R^2$](https://pt.wikipedia.org/wiki/R%C2%B2) para ver o quão bem o modelo linear está se ajustando aos dados.

**Nesse caso $R^2$ tem que estar próximo de $0.5$.**

```python
X, y = get_housing_prices_data(N=1000000, verbose=False)
init = time.time()
prediction = normal_equation_prediction(X, y)
init = time.time() - init
print("Tempo de execução da função de predição = {:.8f}(s)".format(init))
print("Tem que ser em menos de 1 segundo ")
```


# Inference Overview and Variable Elimination Algorithm

> So long, we have talked about probabilistic graphical model representation (PGM).
> - We've seen that there are two main types: Bayesian networks (directed) and Markov networks (undirected).
> - We've alse seen that they encode different **independence assumptions**.
>
> In this module, we will operationalize the PGMs and study how to use these representations (models) to answer actual queries or questions.

> **Objetives:**
> - To learn the different types of queries one can perform to a PGM.
> - To describe the Variable Elimination algorithm.
> - To analyze the computational complexity of the Variable Elimination algorithm.
> - To learn how to use the Variable Elimination algorithm to answer a query to an actual network.

> **References:**
> - Probabilistic Graphical Models: Principles and Techniques, By Daphne Koller and Nir Friedman. Ch. 9.
> - Mastering Probabilistic Graphical Models Using Python, By Ankur Ankan and Abinash Panda. Ch. 3.
> - Probabilistic Graphical Models Specialization, offered through Coursera. Prof. Daphne Koller.

 Imagen recuperada de: https://static.thenounproject.com/png/542457-200.png.

___

## 1. Inference - Overview

In the previous module, we looked at the different types of representations and how to use them to create models for some problems.

We also saw how the probabilities of variables change when we incorporate evidence, in an intuitive way.

In this module, we will describe several algorithms to compute these probabilities and how they may change. Similarly, we will see how to use inference algorithms to predict the values of variables based on our model.

### 1.1. Conditional probability queries

These are the most common queries. Given:

- A joint probability distribution

  $$P(X_1,\dots, X_n),$$

  (which is modeled through a BN or a MN);

- a set of query variables $\bar{Y}\subseteq \left\{X_1,\dots, X_n\right\}$;

- and a set of observed variables (evidence) $\bar{E} = \bar{e}$, con $\bar{E}\subseteq \left\{X_1,\dots, X_n\right\}$,

we define $\bar{W} = \left\{X_1,\dots, X_n\right\} \setminus \left\{\bar{Y} \cup \bar{E}\right\}$ as the rest of the variables.

The task is to compute the *conditional probability query* $P(\bar{Y} | \bar{E}=\bar{e})$.

**Applications:**

- In the credit-worthiness model of the exam, we saw that the bank was able to observe some variables. Then we would be interested in computing the probability of the unobserved variables given the evidence that the bank observes.

- In a medical diagnosis system, we observe certain symptoms and some test results and we are interested in computing the probability of different diseases.

By definition of conditional probability:

$$P(\bar{Y} | \bar{E}=\bar{e}) = \frac{P(\bar{Y}, \bar{e})}{P(\bar{e})}.$$

In this expression:

- $P(\bar{Y}, \bar{e}) = \sum_{\bar{W}} P(\bar{Y}, \bar{e}, \bar{W})$.

  Recall that $\left\{X_1,\dots, X_n\right\} = \bar{Y} \cup \bar{E} \cup \bar{W}$. Then, the terms in the right-side sumation are joint probabilities of all the variables.

- $P(\bar{e}) = \sum_{\bar{Y}} P(\bar{Y}, \bar{e})$.

  This is simply a normalizing constant for converting $P(\bar{Y}, \bar{e})$ into $P(\bar{Y} | \bar{e})$.

Hence, in principle, we could:

- Take a PGM;
- multiply all of its factors to obtain the joint distribution;
- sum out (marginalize) the unwanted variables in the joint distribution;
- and, that's it!

**Examples:**

1. Bayesian networks: restaurant example.

   Random variables:
   - Location $L$ (Bad: $l^0$, Good: $l^1$).
   - Quality $Q$ (Bad: $q^0$, Normal: $q^1$, Good: $q^2$).
   - Cost $C$ (Low: $c^0$, High: $c^1$).
   - Number of people $N$ (Low: $n^0$, High: $n^1$).

```python
from IPython.display import Image
```

```python
Image("figures/restaurant.png")
```

We will forget about if factors represent CPDs or general affinity functions.

In this case:

$$P(L,Q,C,N)= \phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C).$$

We'd like, for example:

$$P(N)=\sum_{L,Q,C}\phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C).$$

> Sum-product algorithm.

```python
# Import pgmpy.factors.discrete.DiscreteFactor

```

```python
# Define factors

```

```python
# Joint probability

```

```python
# P(N)

```

```python
# Show

```

2. Markov networks: pairwise.

```python
Image("figures/pairwiseMN.png")
```

For example

$$P(D) = \frac{1}{Z}\sum_{A,B,C}\phi_1(A,B)\phi_2(B,C)\phi_3(C,D)\phi_4(A,D).$$

> Sum-product algorithm.

**So, what is the problem?**

*There is an exponential blow-up of the joint distribution that the graphical model representation was precisely designed to avoid*.

In fact, the problem of inference in PGMs (just like most interesting problems) is $\mathcal{NP}-$hard. *What does this mean?*

- If a problem is $\mathcal{NP}-$hard it is very unlikely to come up with an efficient solution (in the general case).

- This means that **all the algorithms** that people have constructed to solve this problem require a time (or number of operations) which is at leat exponential in the size of the representation of the problem.

However, we will be seeing a variety of algorithms for both, exact and approximate inference that can do considerably better that this worst case result.

### 1.3. Maximum A-Posteriori (MAP) Inference

Given:

- A joint probability distribution

  $$P(X_1,\dots, X_n),$$

  (which is modeled through a BN or a MN);

- a set of query variables $\bar{Y}\subseteq \left\{X_1,\dots, X_n\right\}$;

- and a set of observed variables (evidence) $\bar{E} = \bar{e}$, con $\bar{E}\subseteq \left\{X_1,\dots, X_n\right\}$.

In this case, we assume for simplicity that $\bar{Y}\cup \bar{E} = \left\{X_1,\dots, X_n\right\}$.

The task is to compute

$$MAP(\bar{Y} | \bar{E}=\bar{e}) = \arg\max_{\bar{y}\in\mathrm{Val}(\bar{Y})} P(\bar{Y} | \bar{E}=\bar{e}).$$

> There may be multiple solutions!

**Application:**

- Classifier: Once we learn a model from data, we would like to compute the most likely assignment for a set of variables given some observed variables (evidence).

**MAP $\neq$ Max over marginals:**

```python
Image("figures/simple.png")
```

Again, this problem is $\mathcal{NP}-$hard.

___
## 2. Variable Elimination

This is the simplest and most fundamental algorithm.

### 2.1. Initial examples

We will introduce the **variable elimination (VE) algorithm** through a number of simple examples.

**Elimination in a chain**

Consider the following pairwise chain MN:

```python
Image("figures/MNchain.png")
```

We have that:

$$P(A,B,C,D,E) \propto \phi_1(A,B)\phi_2(B,C)\phi_3(C,D)\phi_4(D,E)$$

Our objective is to calculate $P(E)$ (in the whiteboard, then show):

$$\begin{align}P(E) \propto & %\sum_{D,C,B,A} \phi_1(A,B)\phi_2(B,C)\phi_3(C,D)\phi_4(D,E) \\
                          %= & \sum_{D}\sum_{C}\sum_{B}\sum_{A} \phi_1(A,B)\phi_2(B,C)\phi_3(C,D)\phi_4(D,E) \\
                          %= & \sum_{D}\sum_{C}\sum_{B} \phi_2(B,C)\phi_3(C,D)\phi_4(D,E) \underbrace{\sum_{A} \phi_1(A,B)}_{\tau_1(B)} \\
                          %= & \sum_{D}\sum_{C}\sum_{B} \phi_2(B,C)\phi_3(C,D)\phi_4(D,E) \tau_1(B) \\
                          %= & \sum_{D}\sum_{C} \phi_3(C,D)\phi_4(D,E) \underbrace{\sum_{B} \phi_2(B,C)\tau_1(B)}_{\tau_2(C)} \\
                          %= & \sum_{D}\phi_4(D,E) \underbrace{\sum_{C} \phi_3(C,D)\tau_2(C)}_{\tau_3(D)} \\
                          %= & \sum_{D}\phi_4(D,E)\tau_3(D) = \tau_4(E)
\end{align}$$

**Elimination in a BN**

```python
Image("figures/restaurant.png")
```

The joint probability is:

$$P(L,Q,C,N)= \phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C).$$

If we want to calculate $P(N)$ (in the whiteboard, then show):

$$\begin{align} P(N) = & %\sum_{L,C,Q}\phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C) \\
                    %= & \sum_{L}\sum_{C}\sum_{Q} \phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C) \\
                    %= & \sum_{L}\sum_{C} \phi_L(L)\phi_N(N,L,C) \underbrace{\sum_{Q} \phi_Q(Q) \phi_C(C,L,Q)}_{\tau_1(C,L)} \\
                    %= & \sum_{L} \phi_L(L) \underbrace{\sum_{C}\phi_N(N,L,C)\tau_1(C,L)}_{\tau_2(N,L)} \\
                    %= & \sum_{L} \phi_L(L)\tau_2(N,L) = \tau_3(N)
\end{align}$$

```python
# Import pgmpy.models.BayesianModel

```

```python
# Import pgmpy.factors.discrete.TabularCPD

```

```python
# Define model skeleton

```

```python
# Define CPDs

```

```python
# Attach CPDs to the model

```

```python
# Check if the model is correctly defined

```

```python
# Import pgmpy.inference.VariableElimination

```

```python
# Create inference object

```

```python
# Perform query

```

```python
# Show

```

```python
# Compare

```

**What happens if we have evidence?**

If we now want to calculate $P(N, L=l^1)$:

- First, we should reduce our factors according to the given evidence: $\phi_L(l^1)=\phi_L'()$, $\phi_C(C,l^1,Q)=\phi_C'(C,Q)$, $\phi_N(N,l^1,C)=\phi_N'(N,C)$.
- Then, we run the VE algorithm as usual:

$$\begin{align} P(N, l^1) = & \sum_{C,Q}\phi_L'()\phi_Q(Q)\phi_C'(C,Q)\phi_N'(N,C) \\
                         = & \sum_{C}\sum_{Q} \phi_L'()\phi_Q(Q)\phi_C'(C,Q)\phi_N'(N,C) \\
                         = & \sum_{C} \phi_L'()\phi_N'(N,C) \underbrace{\sum_{Q} \phi_Q(Q) \phi_C'(C,Q)}_{\tau_1(C)} \\
                         = & \sum_{C} \phi_L'()\phi_N'(N,C) \tau_1(C)
\end{align}$$

**And, if we want the conditional probability?**

You only have to take the above and renormalize:

$$P(N|l^1)=\frac{P(N,l^1)}{P(l^1)}.$$

```python
# Perform query

```

```python
# Show

```

### 2.2. Algorithm description and computational complexity

#### Algorithm description

For the algorithm description, we have a set of factors $\bar{\Phi}$.

In the examples above:

1. MN: $\bar{\Phi}=\{\phi_1(A,B), \phi_2(B,C), \phi_3(C,D), \phi_4(D,E)\}$.
2. BN: $\bar{\Phi}=\{\phi_L(L), \phi_Q(Q), \phi_C(C,L,Q), \phi_N(N,L,C)\}$.

We will describe the steps for eliminating some variable $Z$ from $\bar{\Phi}$: *Eliminate Var-$Z$ from $\bar{\Phi}$*.

1. Determine the set of factors that involve $Z$:

   $$\Phi' = \left\{\phi_i \in \Phi : Z  \in \mathrm{scope}[\phi_i]\right\}$$

2. Compute:

   $$\psi = \prod_{\phi_i \in \Phi'} \phi_i$$

3. Compute:

   $$\tau = \sum_Z \psi$$

4. Overwrite:

   $$\bar{\Phi} := \left(\bar{\Phi}\setminus \Phi'\right) \cup \{\tau\}$$

Thus, the complete algorithm can be described as:

1. The first step is to reduce all factors in $\bar{\Phi}$ acording to the given evidence, if any.

2. For each non-query variable $Z$:
   - Eliminate Var-$Z$ from $\bar{\Phi}$.

3. Multiply all the remaining factors.

#### Computational complexity

We analyze the computational complexity of Eliminate Var-$Z$ from $\bar{\Phi}$.

1. Determine the set of factors that involve $Z$:

   $$\Phi' = \left\{\phi_i \in \Phi : Z  \in \mathrm{scope}[\phi_i]\right\}$$

2. Compute:

   $$\psi = \prod_{\phi_i \in \Phi'} \phi_i$$

3. Compute:

   $$\tau = \sum_Z \psi$$

4. Overwrite:

   $$\Phi := \left(\Phi\setminus \Phi'\right) \cup \{\tau\}$$

**Complexity of the second step:** Assume that $|\Phi'| = m_k$ (the number of factors that involve $Z$ is $m_k$). Thus,

   $$\psi(\bar{X}_k) = \prod_{i=1}^{m_k} \phi_i.$$

   Now, assuming that $N_k = |\mathrm{Val}(\bar{X}_k)|$ (the number of rows in the factor resulting from the multiplication), the computational cost of this step is:

   $$N_k (m_k - 1)$$

   (each row in the resulting factor is produced by $m_k - 1$ multiplications). See this in the whiteboard

**Complexity of the third step:**

   $$\tau_k (\bar{X}_k \setminus \{Z\}) = \sum_Z \phi(\bar{X}_k)$$

   Assuming that $|\mathrm{Val}(Z)|=z$ (the number of possible values of the variable $Z$). The computational cost of this step is (Each row of the resulting factor involves $z-1$ sumations):

   $$N_k \frac{(z-1)}{z} \leq N_k.$$

   See this in the whiteboard

**The whole algorithm:**

1. At most, $n$ elimination steps ($n$ is the number of variables).

2. Start with $m = |\bar{\Phi}|$ factors.

3. At each elimination step, generate $1$ factor ($\psi$).

4. Total number of factors is $m^{\ast}\leq m+n$.

5. $N:=\max_k N_k$: size of the biggest factor.

6. Number of product operations (each factor is multiplied only once):

   $$\sum_k N_k(m_k-1)\leq N m^{\ast}.$$

7. Operaciones marginalización (cada factor se multiplica a lo sumo una vez):

   $$\sum_k N_k\leq N n.$$

> Until here, the computational cost is linear in $N$ and in $m^\ast$.

8. However, $N_k=|\mathrm{Val}(\bar{X}_k)|=\mathcal{O}(d^{r_k})$, where
  - $d:=\max |\mathrm{Val}(X_i)|$;
  - $r_k=|\bar{X}_k|$.

> Exponential blowup!

**In the BN example**

- Operation: $\tau_1(C,L)=\sum_{Q}\phi_Q(Q)\phi_C(C,L,Q)$ - $r_1=3$.

- Operation: $\tau_2(N,L)=\sum_{C}\phi_N(N,L,C)\tau_1(C,L)$ - $r_2=3$.

- Operation: $\tau_3(N)=\sum_{L}\phi_L(L)\tau_2(N,L)$ - $r_3=2$.

On the other hand $d=3$. Total computational complexity $\leq (n+m^\ast) d^{r_k}=(4+7) 3^3 = 297$.

```python
# Perform query with given elimination order

```

```python
# Show

```

**A different elimination order**

$$\begin{align} P(N) = & \sum_{L,C,Q}\phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C) \\
                     = & \sum_{L}\sum_{C}\sum_{Q} \phi_L(L)\phi_Q(Q)\phi_C(C,L,Q)\phi_N(N,L,C) \\
                     = & \sum_{Q}\sum_{C} \phi_Q(Q) \underbrace{\sum_{L} \phi_L(L) \phi_C(C,L,Q) \phi_N(N,L,C)}_{\tau_1(Q,C,N)} \\
                     = & \sum_{Q} \phi_Q(Q) \underbrace{\sum_{C} \tau_1(Q,C,N)}_{\tau_2(Q,N)} \\
                     = & \sum_{Q} \phi_Q(Q)\tau_2(Q,N)
\end{align}$$

- Operation: $\tau_1(Q,C,N) = \sum_{L} \phi_L(L) \phi_C(C,L,Q) \phi_N(N,L,C)$ - $r_1=4$

- Operation: $\tau_2(Q,N) = \sum_{C} \tau_1(Q,C,N)$ - $r_2=3$

- Operation: $\tau_3(Q) = \sum_{Q} \phi_Q(Q)\tau_2(Q,N)$ - $r_3=2$

On the other hand $d=3$. Total computational complexity $\leq (n+m^\ast) d^{r_k}=(4+7) 3^4 = 891$.

```python
# Perform query with given elimination order

```

```python
# Show

```

```python
# Compare execution times using timeit

```

```python

```

The computational complexity strongly depends on the elimination order!

The result is the same for any elimination order though.

### 2.3. Finding elimination orderings

The VE algorithm works well no matter the selected ordering. However, we have shown that the ordering significantly affects the computation complexity of the algorithm.

**How do we find a good elimination ordering?**

- Good ideas can be obtained from the graphical representation. We won't cover that here, but you can take a look at Section 9.4, pages 305-315.

In practice, one often performs **greedy search using heuristic cost functions** (at each iteration calculate the variable to eliminate to obtain the smallest cost).

> These algorithms are not optimal, but perform sufficiently well.

Possible cost functions:

- min-weight: # of values of factor formed ($N_k$).
- min-neighbors: # of resulting neighbor nodes after variable elimination.
- min-fill: # of new fill edges.

```python
# min-weight

```

```python
# min-neighbors

```

```python
# min-fill (default)

```

```python
# Compare execution times using timeit

```

```python

```

```python

```

# Announcements

## 1. Quiz.

Created with Jupyter by Esteban Jiménez Rodríguez.