# Vertical Line Test

## 1.1 Create two graphs, one that passes the vertical line test and one that does not.

```
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

```

```
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

fig = plt.figure(figsize=(4,4))
ax = plt.axes()

ax.plot(x, y);
```

```
x = np.linspace(-1.0, 1.0, 100)
y = np.linspace(-1.0, 1.0, 100)

X, Y = np.meshgrid(x,y)
F = X**2 + Y**2 - 0.6

plt.contour(X,Y,F, [0])
plt.show()
```

## 1.2 Why are graphs that don't pass the vertical line test not considered "functions?"

A valid function may have only one output(y) for every input(x)

# Functions as Relations

## 2.1 Which of the following relations are functions? Why?

\begin{align}
\text{Relation 1: } \{(1, 2), (3, 2), (1, 3)\}
\\
\text{Relation 2: } \{(1, 3), (2, 3), (6, 7)\}
\\
\text{Relation 3: } \{(9, 4), (2, 1), (9, 6)\}
\\
\text{Relation 4: } \{(6, 2), (8, 3), (6, 4)\}
\\
\text{Relation 5: } \{(2, 6), (2, 7), (2, 4)\}
\end{align}

Only relation two is a function as the other relations have domain values which map to the same range value more than once

# Functions as a mapping between dimensions

## 3.1 for the following functions what is the dimensionality of the domain (input) and codomain (range/output)?

\begin{align}
m(𝑥_1,𝑥_2,𝑥_3)=(x_1+x_2, x_1+x_3, x_2+x_3)
\\
n(𝑥_1,𝑥_2,𝑥_3,𝑥_4)=(x_2^2 + x_3, x_2x_4)
\end{align}

```
1) 3d to 3d
2) 4d to 2d
```

## 3.2 Do you think it's possible to create a function that maps from a lower dimensional space to a higher dimensional space? If so, provide an example.

```
Of course, x = x,0
```

# Vector Transformations

## 4.1 Plug the corresponding unit vectors into each function. Use the output vectors to create a transformation matrix.

\begin{align}
p(\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}
\\
\\
q(\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}
\end{align}

\begin{align}
\\
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = x_1\begin{bmatrix} 1 \\ 0 \end{bmatrix} + x_2\begin{bmatrix} 0 \\ 1 \end{bmatrix}
\\
\end{align}

\begin{align}
p(\begin{bmatrix} 1 \\ 0 \end{bmatrix}) = \begin{bmatrix} 1 \\ -1 \end{bmatrix}
\\
p(\begin{bmatrix} 0 \\ 1 \end{bmatrix}) = \begin{bmatrix} 3 \\ 2 \end{bmatrix}
\end{align}

\begin{align}
\begin{bmatrix} 1 & 3\\ -1 &2\end{bmatrix}
\end{align}

\begin{align}
\\
\begin{bmatrix} x_1 \\ x_2 \\x3 \end{bmatrix} = x_1\begin{bmatrix} 1 \\ 0 \\ 0\end{bmatrix} + x_2\begin{bmatrix} 0 \\ 1 \\0 \end{bmatrix}
+ x_3\begin{bmatrix} 0 \\ 0 \\ 1\end{bmatrix}
\end{align}
\begin{align}
q(\begin{bmatrix} 1 \\ 0 \\0 \end{bmatrix}) = \begin{bmatrix} 4 \\ -1 \\5 \end{bmatrix}
\\
q(\begin{bmatrix} 0 \\ 1 \\0 \end{bmatrix}) = \begin{bmatrix} 1 \\ 2 \\1 \end{bmatrix}
\\
q(\begin{bmatrix} 0 \\ 0 \\1 \end{bmatrix}) = \begin{bmatrix} 2 \\ 3 \\-2 \end{bmatrix}
\end{align}

\begin{align}
\begin{bmatrix} 4 & 1 & 2\\ -1 & 2 & 3 \\ 5 & 1 & -2 \end{bmatrix}
\end{align}

## 4.2 Verify that your transformation matrices are correct by choosing an input matrix and calculating the result both via the traditional functions above and also via vector-matrix multiplication.

\begin{align}
p(\begin{bmatrix}3 \\ 6\end{bmatrix}) = \begin{bmatrix} 1(3) + 3(6)\\2(6) - 3 \\\end{bmatrix} = \begin{bmatrix} 21 \\ 9 \end{bmatrix}
\\
\\
q(\begin{bmatrix}2 \\ 4 \\ 6\end{bmatrix}) = \begin{bmatrix} 4(2) + 4 + 2(6) \\2 (4) - 2  + 3(6) \\ 5(2) - 2(6) + 4  \end{bmatrix} = \begin{bmatrix} 24 \\ 24 \\ 2 \end{bmatrix}
\end{align}

```
pT = np.array([[1, 3],
              [-1, 2]])
p_input = np.array ([[3],
                   [6]])
print(np.matmul(pT, p_input))

qT = np.array([[4, 1, 2],
              [-1, 2, 3],
              [5, 1, -2]])
q_input = np.array([[2],
                   [4],
                   [6]])
print("\n\n", np.matmul(qT, q_input))

```

    [[21]
     [ 9]]

     [[24]
     [24]
     [ 2]]

# Eigenvalues and Eigenvectors

## 5.1 In your own words, give an explanation for the intuition behind eigenvalues and eigenvectors.

An eigenvector is any vector which is not reoriented by a function, it may however be scaled and the scalar which represents the change in the eigenvector during the transformation is referred to as the eigenvalue.

# The Curse of Dimensionality

## 6.1 What are some of the challenges of working with high dimensional spaces?

With any set of data, if the amount of data remains constant while the features(dimensions) increase then the distance between the points of data will increase relative to the increase in the amount of dimensions. This makes data computation exponentially expensive as the distance between points in space become further seperated. In addition to this many of the features may prove redundant and adding them to the calculations may provide negligible benefit at best. Also, with the increased data we run the risk of overfitting the data.

## 6.2 What is the rule of thumb for how many observations you should have compared to parameters in your model?

n = 5

# Principal Component Analysis

## 7.1 Code for loading and cleaning the 2013 national dataset from the [Housing Affordability Data System (HADS)](https://www.huduser.gov/portal/datasets/hads/hads.html) --housing data, can be found below.

## Perform PCA on the processed dataset `national_processed` (Make sure you standardize your data!) and then make a scatterplot of PC1 against PC2. Some of our discussion and work around PCA with this dataset will continue during tomorrow's lecture and assignment.

Not only does this dataset have decent amount columns to begin with (99), but in preparing the data for PCA we have also [one-hot-encoded](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f#targetText=One%20hot%20encoding%20is%20a,the%20entry%20in%20the%20dataset.) all of the categorical variables. This has the effect of creating a new column for each individual category of each categorical variable. After processing this dataset has 64738 columns. --Das a lot of columns.

Don't worry too much about the mechanics of one-hot encoding right now, you will learn and experiment with a whole bunch of categorical encoding approaches in unit 2.

The code below will read in the dataset and perform the one-hot encoding of the categorical variables. Start adding your PCA code at the bottom of the provided code.

```
from urllib.request import urlopen
from zipfile import ZipFile
from io import BytesIO
import os.path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Read Natinal Data
national_url = 'https://www.huduser.gov/portal/datasets/hads/hads2013n_ASCII.zip'
national_file = 'thads2013n.txt'

if os.path.exists(national_file):
    national = pd.read_csv(national_file)
else:
    z_national = urlopen(national_url)
    zip_national = ZipFile(BytesIO(z_national.read())).extract(national_file)
    national = pd.read_csv(zip_national)

print(national.shape)
national.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      '100003130103'
      82
      '3'
      '1'
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      '1'
      1
      40000
      -6
      '1'
      1
      6
      3117.394239
      1
      18021
      '1'
      533
      1
      '1'
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      '1 Adequate'
      '-5'
      '2000-2009'
      '1 Single Family'
      '2 2BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '1 LTE 50% FMR'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '.'
      '2 30% to 50%'
      '-5'
      '-5'

      1
      '100006110249'
      50
      '5'
      '3'
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      '1'
      1
      130000
      -6
      '1'
      1
      6
      2150.725544
      4
      122961
      '1'
      487
      1
      '1'
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '3 GT FMR'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '7 120% AMI +'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      2
      '100006370140'
      53
      '5'
      '3'
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      '1'
      1
      150000
      -6
      '1'
      1
      7
      2213.789404
      2
      27974
      '1'
      1405
      1
      '1'
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '4 60 - 80% AMI'
      '5 80 - 100% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '.'
      '3 50% or More'
      '-5'
      '-5'

      3
      '100006520140'
      67
      '5'
      '3'
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      '1'
      1
      200000
      -6
      '1'
      1
      6
      2364.585097
      2
      32220
      '1'
      279
      1
      '1'
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '3 3BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '6 100 - 120% AMI'
      '7 120% AMI +'
      '7 120% AMI +'
      '5 80 - 100% AMI'
      '4 60 - 80% AMI'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      4
      '100007130148'
      26
      '1'
      '3'
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      '1'
      1
      -6
      -6
      '2'
      100
      4
      2314.524902
      2
      96874
      '1'
      759
      5
      '2'
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      '1 Adequate'
      'Central City'
      '1980-1989'
      '5 50+ units'
      '2 2BR'
      '2 Renter'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '7 120% AMI +'
      '0 Not Assisted'
      '1 Less than 30%'
      '-5'
      '-5'

5 rows × 99 columns

```
# Look at datatypes
# a lot of object datatypes even though they seem to be strings of numbers.
national.dtypes

```

    CONTROL                 object
    AGE1                     int64
    METRO3                  object
    REGION                  object
    LMED                     int64
    FMR                      int64
    L30                      int64
    L50                      int64
    L80                      int64
    IPOV                     int64
    BEDRMS                   int64
    BUILT                    int64
    STATUS                  object
    TYPE                     int64
    VALUE                    int64
    VACANCY                  int64
    TENURE                  object
    NUNITS                   int64
    ROOMS                    int64
    WEIGHT                 float64
    PER                      int64
    ZINC2                    int64
    ZADEQ                   object
    ZSMHC                    int64
    STRUCTURETYPE            int64
    OWNRENT                 object
    UTILITY                float64
    OTHERCOST              float64
    COST06                 float64
    COST12                 float64
                            ...
    COSTMedRELAMICAT         int64
    COSTMedRELPOVPCT       float64
    COSTMedRELPOVCAT         int64
    COSTMedRELFMRPCT       float64
    COSTMedRELFMRCAT         int64
    FMTZADEQ                object
    FMTMETRO3               object
    FMTBUILT                object
    FMTSTRUCTURETYPE        object
    FMTBEDRMS               object
    FMTOWNRENT              object
    FMTCOST06RELPOVCAT      object
    FMTCOST08RELPOVCAT      object
    FMTCOST12RELPOVCAT      object
    FMTCOSTMEDRELPOVCAT     object
    FMTINCRELPOVCAT         object
    FMTCOST06RELFMRCAT      object
    FMTCOST08RELFMRCAT      object
    FMTCOST12RELFMRCAT      object
    FMTCOSTMEDRELFMRCAT     object
    FMTINCRELFMRCAT         object
    FMTCOST06RELAMICAT      object
    FMTCOST08RELAMICAT      object
    FMTCOST12RELAMICAT      object
    FMTCOSTMEDRELAMICAT     object
    FMTINCRELAMICAT         object
    FMTASSISTED             object
    FMTBURDEN               object
    FMTREGION               object
    FMTSTATUS               object
    Length: 99, dtype: object

```
# check for null values
national.isnull().sum().any()
```

    False

```
# check for number of categorical vs numeric columns
cat_cols = national.columns[national.dtypes=='object']
num_cols = national.columns[national.dtypes!='object']

print(f'{len(cat_cols)} categorical columns')
print(f'{len(num_cols)} numerical columns')
```

    32 categorical columns
    67 numerical columns

```
# We're making a copy of our data in case we mess something up.
national_processed = national.copy()

# Categorically Encode our Variables:
# They need to all be numeric before we do PCA.
# https://pbpython.com/categorical-encoding.html

# Cast categorical columns to "category" data type
national_processed[cat_cols] = national_processed[cat_cols].astype('category')

national_processed.dtypes
```

    CONTROL                category
    AGE1                      int64
    METRO3                 category
    REGION                 category
    LMED                      int64
    FMR                       int64
    L30                       int64
    L50                       int64
    L80                       int64
    IPOV                      int64
    BEDRMS                    int64
    BUILT                     int64
    STATUS                 category
    TYPE                      int64
    VALUE                     int64
    VACANCY                   int64
    TENURE                 category
    NUNITS                    int64
    ROOMS                     int64
    WEIGHT                  float64
    PER                       int64
    ZINC2                     int64
    ZADEQ                  category
    ZSMHC                     int64
    STRUCTURETYPE             int64
    OWNRENT                category
    UTILITY                 float64
    OTHERCOST               float64
    COST06                  float64
    COST12                  float64
                             ...
    COSTMedRELAMICAT          int64
    COSTMedRELPOVPCT        float64
    COSTMedRELPOVCAT          int64
    COSTMedRELFMRPCT        float64
    COSTMedRELFMRCAT          int64
    FMTZADEQ               category
    FMTMETRO3              category
    FMTBUILT               category
    FMTSTRUCTURETYPE       category
    FMTBEDRMS              category
    FMTOWNRENT             category
    FMTCOST06RELPOVCAT     category
    FMTCOST08RELPOVCAT     category
    FMTCOST12RELPOVCAT     category
    FMTCOSTMEDRELPOVCAT    category
    FMTINCRELPOVCAT        category
    FMTCOST06RELFMRCAT     category
    FMTCOST08RELFMRCAT     category
    FMTCOST12RELFMRCAT     category
    FMTCOSTMEDRELFMRCAT    category
    FMTINCRELFMRCAT        category
    FMTCOST06RELAMICAT     category
    FMTCOST08RELAMICAT     category
    FMTCOST12RELAMICAT     category
    FMTCOSTMEDRELAMICAT    category
    FMTINCRELAMICAT        category
    FMTASSISTED            category
    FMTBURDEN              category
    FMTREGION              category
    FMTSTATUS              category
    Length: 99, dtype: object

```
# Replace all category cell values with their numeric category codes
for col in cat_cols:
  national_processed[col] = national_processed[col].cat.codes

print(national_processed.shape)
national_processed.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      0
      82
      2
      0
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      0
      1
      40000
      -6
      1
      1
      6
      3117.394239
      1
      18021
      1
      533
      1
      0
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      1
      0
      5
      1
      2
      0
      4
      4
      4
      4
      3
      1
      1
      1
      1
      1
      1
      1
      1
      1
      2
      0
      2
      0
      0

      1
      1
      50
      4
      2
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      0
      1
      130000
      -6
      1
      1
      6
      2150.725544
      4
      122961
      1
      487
      1
      0
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      1
      0
      3
      1
      4
      0
      3
      4
      4
      3
      4
      2
      2
      2
      1
      3
      3
      3
      5
      3
      7
      0
      1
      0
      0

      2
      2
      53
      4
      2
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      0
      1
      150000
      -6
      1
      1
      7
      2213.789404
      2
      27974
      1
      1405
      1
      0
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      1
      0
      3
      1
      4
      0
      4
      4
      4
      4
      3
      2
      2
      2
      1
      2
      3
      4
      5
      3
      4
      0
      3
      0
      0

      3
      3
      67
      4
      2
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      0
      1
      200000
      -6
      1
      1
      6
      2364.585097
      2
      32220
      1
      279
      1
      0
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      1
      0
      3
      1
      3
      0
      4
      4
      4
      4
      4
      2
      2
      2
      2
      2
      5
      6
      6
      4
      4
      0
      1
      0
      0

      4
      4
      26
      0
      2
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      0
      1
      -6
      -6
      2
      100
      4
      2314.524902
      2
      96874
      1
      759
      5
      1
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      1
      1
      3
      5
      2
      1
      3
      3
      3
      3
      4
      2
      2
      2
      2
      3
      2
      2
      2
      2
      7
      1
      1
      0
      0

5 rows × 99 columns

```
# Now we only ahve numeric columns (ints and floats)
national_processed.dtypes
```

    CONTROL                  int32
    AGE1                     int64
    METRO3                    int8
    REGION                    int8
    LMED                     int64
    FMR                      int64
    L30                      int64
    L50                      int64
    L80                      int64
    IPOV                     int64
    BEDRMS                   int64
    BUILT                    int64
    STATUS                    int8
    TYPE                     int64
    VALUE                    int64
    VACANCY                  int64
    TENURE                    int8
    NUNITS                   int64
    ROOMS                    int64
    WEIGHT                 float64
    PER                      int64
    ZINC2                    int64
    ZADEQ                     int8
    ZSMHC                    int64
    STRUCTURETYPE            int64
    OWNRENT                   int8
    UTILITY                float64
    OTHERCOST              float64
    COST06                 float64
    COST12                 float64
                            ...
    COSTMedRELAMICAT         int64
    COSTMedRELPOVPCT       float64
    COSTMedRELPOVCAT         int64
    COSTMedRELFMRPCT       float64
    COSTMedRELFMRCAT         int64
    FMTZADEQ                  int8
    FMTMETRO3                 int8
    FMTBUILT                  int8
    FMTSTRUCTURETYPE          int8
    FMTBEDRMS                 int8
    FMTOWNRENT                int8
    FMTCOST06RELPOVCAT        int8
    FMTCOST08RELPOVCAT        int8
    FMTCOST12RELPOVCAT        int8
    FMTCOSTMEDRELPOVCAT       int8
    FMTINCRELPOVCAT           int8
    FMTCOST06RELFMRCAT        int8
    FMTCOST08RELFMRCAT        int8
    FMTCOST12RELFMRCAT        int8
    FMTCOSTMEDRELFMRCAT       int8
    FMTINCRELFMRCAT           int8
    FMTCOST06RELAMICAT        int8
    FMTCOST08RELAMICAT        int8
    FMTCOST12RELAMICAT        int8
    FMTCOSTMEDRELAMICAT       int8
    FMTINCRELAMICAT           int8
    FMTASSISTED               int8
    FMTBURDEN                 int8
    FMTREGION                 int8
    FMTSTATUS                 int8
    Length: 99, dtype: object

```
from numpy import array
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
```

```
d = np.array(national_processed) # Define
```

```
scaler = StandardScaler()
Z = scaler.fit_transform(d) # Standardization

```

```
pca = PCA(2) # select # of PCs
pca.fit(Z)
print("Eigenvectors \n\n", "**"*40+"\n\n", pca.components_ , "\n\n","**"*40)
```

    Eigenvectors

     ********************************************************************************

     [[-0.01762294  0.04564172  0.00224448 -0.0056805   0.04142715  0.08969161
       0.06498292  0.0649871   0.06422065  0.06147403  0.08585244  0.01801879
      -0.05896537 -0.01528452  0.13234213 -0.05711476 -0.03969967 -0.02559715
       0.09915137  0.02420272  0.06848524  0.10804133  0.02549375  0.12591589
      -0.0686994  -0.09686185  0.09495694  0.06829523  0.13890808  0.13843111
       0.13897851  0.138511    0.0905027  -0.07741872  0.04142715  0.06498292
       0.0649871   0.06422065  0.07923954  0.0921104   0.09210336  0.09490276
       0.08739559  0.00091387  0.10149966  0.10199758  0.1018149   0.08553625
       0.09217413  0.0781451   0.13215061  0.14670343  0.1335704   0.09089986
       0.12974258  0.12772056  0.13365417  0.14824806  0.13368685  0.0902927
       0.13209618  0.12802126  0.13470573  0.1470892   0.13320525  0.08919073
       0.13411974  0.12560459  0.13025193  0.14427022  0.13314257  0.09127039
       0.1269716   0.12604755  0.02549375 -0.02868194  0.02198791 -0.06878637
       0.08502306 -0.09686185  0.12430766  0.1236005   0.12184815  0.12455057
       0.10357345  0.12772056  0.12802126  0.12560459  0.12604755  0.09767683
       0.14670343  0.14824806  0.1470892   0.14427022  0.11025228 -0.08541172
       0.00320622  0.01421383 -0.        ]
     [-0.00232269 -0.12881974  0.02787248  0.02355993 -0.05103525 -0.03833234
      -0.10241577 -0.10242769 -0.10406663 -0.16868994 -0.01158043  0.02224373
       0.22379537  0.00406118  0.07554282  0.22202935 -0.17613668  0.01102132
      -0.00219946 -0.00774407 -0.20946021 -0.03104596 -0.16686104 -0.0276374
       0.00239876 -0.02138781 -0.01017961  0.03905815  0.08279444  0.08127423
       0.08232351  0.08302786 -0.03769061 -0.1026406  -0.05103525 -0.10241577
      -0.10242769 -0.10406663 -0.20229415 -0.04714243 -0.0472246  -0.04437526
      -0.04544984 -0.00410633 -0.0243843  -0.17315565 -0.02241456 -0.20463355
      -0.03479403 -0.21272755  0.102165    0.0761984   0.04409669 -0.19809081
       0.10852792  0.07185395  0.10098787  0.07256343  0.04730334 -0.19897927
       0.10684339  0.06898156  0.09847618  0.06586526  0.05166391 -0.20021042
       0.10334089  0.06413585  0.10281403  0.07875107  0.04107963 -0.19724869
       0.10950206  0.07374035 -0.16686104 -0.00998768  0.02400899  0.00247039
      -0.01286988 -0.02138781 -0.10647349 -0.10987843 -0.11449743 -0.10326949
      -0.11457314  0.07185395  0.06898156  0.06413585  0.07374035 -0.12050048
       0.0761984   0.07256343  0.06586526  0.07875107 -0.07724314 -0.09534592
      -0.09249567 -0.00290366 -0.        ]]

     ********************************************************************************

```
print("Eigenvalues: \n\n", "**"*40, "\n", pca.explained_variance_, "\n", "**"*40)
```

    Eigenvalues:

     ********************************************************************************
     [33.37004789 14.1535396 ]
     ********************************************************************************

```
T = pca.transform(Z)
print('Projected Data: \n\n\n', T)
```

    Projected Data:

     [[-2.57657018 -1.37612733]
     [ 2.04064284 -0.98806143]
     [ 1.21527025  0.37016818]
     ...
     [ 1.83162816 -2.95570212]
     [-5.02514474 -2.52857818]
     [-9.89816404 -0.80284269]]

```
plt.scatter(T[:,0], T[:,1])
plt.show()
```

# Stretch Goals

## 1) Perform further data exploration on the HADS national dataset (the version before we one-hot encoded it) Make scatterplots and see if you can see any resemblance between the original scatterplots and the plot of the principal components that you made in 7.1.

(You may or may not not see very much resemblance depending on the variables you choose, and that's ok!)

## 2) Study "Scree Plots" and then try and make one for your PCA dataset. How many principal conponents do you need to retain in order for your PCs to contain 90% of the explained variance?

We will present this topic formally at the beginning of tomorrow's lecture, so if you figure this stretch goal out, you're ahead of the game.

## 3) Explore further the intuition behind eigenvalues and eigenvectors by creating your very own eigenfaces:

Prioritize self-study over this stretch goal if you are not semi-comfortable with the topics of PCA, Eigenvalues, and Eigenvectors.

You don't necessarily have to use this resource, but this will get you started:
[Eigenface Tutorial](https://sandipanweb.wordpress.com/2018/01/06/eigenfaces-and-a-simple-face-detector-with-pca-svd-in-python/)


```python
%matplotlib inline

import matplotlib.pylab as plt
import numpy as np
from sympy import *
import pandas as pd
from scipy import stats
from scipy.integrate import quad
```

```python
init_printing()
```

```python
t=np.arange(0,np.pi,0.1)
f= lambda x: np.sin(x)

plt.plot(t,f(t))
```

```python
quad(f,0,np.pi)#integral de la funcion f de cero a pi
```

# CAMNATAS ALEATORIAS

```python
% matplotlib inline

```


# Lab 3.1: Radar Transmissions and Receptions

```python
%matplotlib widget
import rad.example as ex
import rad.quiz as qz
from rad.const import c, k
from rad.radar import to_db, from_db, deg2rad, rad2deg
from math import sqrt, sin, asin, cos, acos, tan, atan2, pi, log, log10
```

**Reminders**:

- Hit the *Run All* button  button above before continuing
- Useful formulae and definitions are available in [Reference](Reference.ipynb)

In this lab, we will dive into two of the boxes in the basic radar system diagram: *transmitters* and *receivers*. The transmitter decides how and where to emit energy, and the receiver decides how and where to listen for echoes.

Following the order of the diagram, we will first focus on transmissions and how they are designed, then move on to receptions.

## Radar Transmissions

There are three main design options (often called *degrees-of-freedom*) when choosing how to emanate energy out of a radar:

- **Waveform**: Shape of the transmitted wave
- **Space**: Region to concentrate transmitted energy
- **Polarization**: Orientation of the transmitted wave relative to direction of travel

We will go over each of these in detail.

### Waveform

Let us first study the *waveform* of a transmission, or the shape of the wave being emitted, dictated by the electrical signal driving the transmit antenna. In [Lab 1.2: Introduction to Radar](1_2_Introduction_to_Radar.ipynb), we introduced the basics of ranging via a hypothetical sensor that could transmit an impulse, i.e., a very short burst of energy (see figure below). The impulse would propagate outward, impinge on a target, and then an echo of the impulse propagates back to the sensor.

An impulse is an idealized shape for a wave as it allows for arbitrarily high precision while estimating the time at which a target echo arrives; however, compressing electromagnetic energy into a very short duration requires a transmitter with an extremely large amount of power (i.e., power is the rate that energy is transmitted), which is impractical for most applications.

To avoid requiring extremely high power, the transmitted energy is typically spread out over time. To see how this is done, we will look at the basic radar wave shape: the pulsed sine wave.

In the above figure, we see a time-limited sine wave that is characterized by its frequency, $f$ , amplitude, $a$ , and **pulsewidth**, $t_p$ , where the pulsewidth is the duration of the pulse. Note that the energy contained in a sine wave is $\mathcal{E}_t = a^2 t_p / 2$; thus, increasing the amplitude or pulsewidth will increase the amount of transmit energy.

In the following interactive example, we will look at designing a single sine wave pulse to impart a desired amount of transmit energy. The adjustable parameters of the pulsed sine wave are *Transmit Energy*, *Transmit Frequency*, and *Pulsewidth*. The maximum instantaneous power, called the **peak power**, is shown as a **black** dashed line.

```python
ex.ex_3_1_1()
```

We can see as the pulsewidth increases, the peak power of the wave decreases; this is because the energy is being spread out over a longer time span. The peak power is a major design aspect of a transmitter (more specifically, a component called an *amplifier* which creates high power sinusoidal signals to be emitted). The higher the peak power required of a transmitter, the more sophisticated and expensive it will be. Therefore, if we can tolerate long pulsewidths, we can build a cheaper system because less peak power is required to achieve a desired amount of transmit energy.

***

### Question 1

What pulsewidth, $t_p$ , is needed to get peak power below $7~\mathrm{kW}$ when using a transmit energy of $\mathcal{E}_t = 50~\mathrm{mJ}$ at transmit frequency $f = 4~\mathrm{MHz}$?

```python
qz.quiz_3_1_1()
```

```python
# Scratch space
```

***

Radar systems traditionally transmit pulses at a chosen rate, called the **pulse repetition frequency** (PRF), $f_{rep}$ , and listen for echoes in between pulses. The inverse of this is the **pulse repetition interval** (PRI), $t_{rep}$ , which is the amount of time total for one transmission and one listening period. Mathematically, we have

$$
f_{rep} = \frac{1}{t_{rep}}
$$

For instance, if a radar transmits using a PRF of $f_{rep} = 2~\mathrm{Hz}$ (i.e., a pulse repetition interval of $t_{rep} = 0.5~\mathrm{s}$), it sends a pulse once every half-second and uses the duration in between to search for target responses. The ratio of the time transmitting versus total operational time is called the **duty cycle**, $d$ , and it gives a notion of how much of the radar's operation is dedicated to actively transmitting. The duty cycle can be calculated as the ratio of the pulsewidth, $t_p$ , to the pulse repetition interval, $t_{rep}$ :

$$
d = \frac{t_p}{t_{rep}}
$$

The structure of multiple radar transmissions can be seen in the following figure:

To look at the transmissions of a radar over multiple pulses, we will study the following interactive example. This example is very similar to the previous pulsed sine wave example, except now we are transmitting multiple pulses. To the right side, we can also see the duty cycle of the radar system.

```python
ex.ex_3_1_2()
```

We can notice that the pulsewidth, $t_p$ , can only get as large as the PRI, $t_{rep}$ , before it no longer becomes a pulsed signal.

***

### Question 2

**(a)** What is the pulse repetition interval, $t_{rep}$ , for a system with pulse repetition frequency $f_{rep} = 10~\mathrm{kHz}$?

```python
qz.quiz_3_1_2a()
```

```python
# Scratch space
```

**(b)** What is the duty cycle, $d$ , for a radar that uses a pulsewidth of $t_p = 150~\mu\mathrm{s}$ and a pulse repetition frequency of $f_{rep} = 1~\mathrm{kHz}$?

```python
qz.quiz_3_1_2b()
```

```python
# Scratch space
```

***

The pulsed sine wave was a very common waveform for early radar systems due to its simplicity; however, modern radar systems use variations on sine waves created by **modulation**. *Modulation* is achieved by using a base sine wave and altering it with another signal (typically via multiplication). Some common forms of modulation used in radar are:

- **Linear frequency modulation** (LFM), sweeps from one frequency to another over the course of the pulse (also known as a *chirp* waveform)
- **Coded waveforms**, creates a sequence of base sine wave pulses but with varying phases and frequencies

In the interactive example below, we will look at the most common radar waveform used in modern systems: the LFM waveform. Try changing the *Start Frequency* and *Stop Frequency* to see how the sine wave sweeps from one to another. In the next section on radar receptions, we will discuss why sweeping over multiple frequencies is important.

```python
ex.ex_3_1_3()
```

### Space

From [Lab 1.2: Introduction to Radar](1_2_Introduction_to_Radar.ipynb), we know that radars use directed ranging to be able to simultaneously estimate the range and angle of a target. Directed ranging is performed by focusing energy into a beam and then steering the beam in a desired direction. Therefore, choosing the spatial degree-of-freedom for a radar transmission is the same as focusing the energy and steering it to a desired region.

Focusing in radar systems follows the same logic as the generic ranging systems in [Lab 1.2: Introduction to Radar](1_2_Introduction_to_Radar.ipynb):

- An aperture is used to concentrate energy in a swath of angle
- As aperture extent/area increases relative to the wavelength, the beam narrows and the gain increases

In reality, a radar system will not be able to focus its energy perfectly into a beam, it will transmit some energy outside of the main beamwidth. This can be seen by investigating a radar's **beam pattern**. A beam pattern shows the strength of transmission as a function of angle. Below is an interactive plot showing the beam pattern for a dish radar with a circular aperture; try changing the *Dish Radius* and *Transmit Frequency*.

```python
ex.ex_3_1_4()
```

Beam patterns are primarily dictated by the radar aperture shape; however, it can be further modified by changing how emitted energy is spread over the aperture on transmission and reception. Tailoring the radar transmission to create a desired beam pattern and direction of transmission is commonly referred to as **beamforming**.

In [Lab 1.2: Introduction to Radar](1_2_Introduction_to_Radar.ipynb), we also learned about the idea of *steering*, or choosing the direction to transmit and listen to electromagnetic energy. There were two types of steering discussed:

- *Mechanical*, movement of aperture to change direction of transmission/reception
- *Electronic*, coordinated variation of elements in an array to direct transmitted energy

As the concept of mechanical steering is somewhat straightforward, we will focus on going into more depth on electronic steering for radar systems. The first type of steering used in radar systems is **delay steering**, which changes the transmission time of each element so each transmission will arrive at the target at the same time.

To recall the general view of delay steering, we can look back at a hypothetical array of elements, each emitting an impulse at a different delay where propagation velocity is $v_p = 1000~\mathrm{m/s}$. In the interactive example below, we can see how the delay of each element changes to have all impulse reach the target at the same time.

```python
ex.ex_3_1_5()
```

As discussed above, practical radar systems employ sine waves (and variations thereof) for transmission to be able to spread out energy over a duration and lower necessary power. This means *delay steering* for radar systems is equivalent to designing the element delays so that sine waves transmitted from each element arrive at the target at the same time.

To see how this works, let us look at an interactive example in which we transmit four cycles of a sine wave from each element of a five element array (see figure below).

There are two plots in the following example:

- The upper plot shows all of the different waves from the elements at the moment they arrive at the target; the center is aligned to the wave emitted by the middle element, *Element \#3* (shown as a **black** dashed line)
- The lower plot shows the sum of all of the incident waves; this will be the actual wave that hits the target

We can control the *Transmit Frequency* of the radar, the *Azimuth* and *Range* of the target, and the *Delay* of each element (delay is relative to the wave from the middle element, *Element \#3*). We can also choose to hide the wave from each element to be able to focus on a single wave. Additionally, an *Energy Loss* is displayed showing the difference between the achieved energy delivered versus the maximum possible energy delivered, i.e., an *Energy Loss* of $0~\mathrm{dB}$ means that the maximum energy possible is hitting the target.

```python
ex.ex_3_1_6()
```

***

### Question 3

**(a)** What is the delay needed for the *Element \#1* wave to match the reference wave (in **black** dashed line) when the target is at a range of $r = 3~\mathrm{km}$ and azimuth $\theta = 45~\mathrm{deg}$ and the radar transmits at a frequency of $f = 800~\mathrm{MHz}$?

```python
qz.quiz_3_1_3a()
```

```python
# Scratch space
```

**(b)** What is the delay needed for the *Element \#1* wave from **(a)** if the frequency is now $f = 1~\mathrm{GHz}$?

```python
qz.quiz_3_1_3b()
```

```python
# Scratch space
```

***

Try aligning the waves at different target azimuths and ranges. There are a few important things to notice about delay steering:

- Changing the *Transmit Frequency* does not affect the delays. Once you have the waves aligned in a certain direction, that configuration will work for any frequency. Thus, delay steering can work for waves that consist of a large span of frequencies and not only single-frequency sine waves.

- For most geometries and radar parameters, range has no impact on the delays required to align the element waves. However, you may notice that the wave alignment changes slightly and *Transmit Frequency* varies as the range gets very small ($\approx 0.1~\mathrm{km}$). This is called being in the **near field** of the radar, where the optimal relative delays not only depend on azimuth but also range. Typically, targets are well beyond this region, in what is called the **far field**[1](#foot_far).

We can also notice that the delays used are on the order of a few nanoseconds—delaying signals at this timescale requires very precise timing and delay electronics. Another way to achieve a similar effect for sine wave transmissions is *phase steering*. The main concept of phase steering is to transmit from all elements at the same time but change the phase of each sine wave so they match when they reach the target.

In the interactive example below, we use the same five element array we used for delay steering. This time, however, all elements will transmit at the same time, and we can change the phase of each sine wave to try and match the reference wave. Try match the reference wave for different azimuths, ranges, and transmit frequencies.

```python
ex.ex_3_1_7()
```

***

### Question 4

**(a)** What is the phase needed for the *Element \#4* wave to match the reference wave (in **black** dashed line) when the target is at a range of $r = 3~\mathrm{km}$ and azimuth $\theta = 30~\mathrm{deg}$ and the radar transmits at a frequency of $f = 700~\mathrm{MHz}$?

```python
qz.quiz_3_1_4a()
```

```python
# Scratch space
```

**(b)** What is the phase needed for the *Element \#4* wave from **(a)** if the frequency is now $f = 500~\mathrm{MHz}$?

```python
qz.quiz_3_1_4b()
```

```python
# Scratch space
```

***

Here, we notice a few interesting differences compared to delay steering:

- The phase needed to match an element's transmission to the reference wave *does depend on frequency* (i.e., try aligning and then changing the frequency). This is a major consideration when using phase steering: if using phase steering and a transmit waveform that utilizes a broad range of frequencies, there may be appreciable energy loss due to the fact that the phases are only optimal for a single frequency.

- Since the waves arrive at different times, the edges of the waves are no longer aligned. This causes a small amount of energy loss but is typically negligible as the pulse is often much longer than the mismatched portion.

### Polarization

The last degree-of-freedom for transmission is the orientation of the wave with respect to propagation. We can think of this as the same decision that one can make when making waves using a taut rope: we can go up-and-down, left-and-right, corkscrew, etc. The big difference being that we are making waves using electromagnetics and not tension in a rope.

There are a few common polarizations used in radar systems; the first we will look at is **linear** polarization, where the wave stays in the same plane as it travels outward. This is analogous to the up-and-down and left-and-right waves created in the rope example. Below is an animation of a wave with **vertical** polarization, which varies up and down as it moves outward:

The other most common type of linear polarization is **horizontal** polarization, where the wave moves in the horizontal plane relative to the radar aperture. Another polarization typically used in radar is **circular** polarization, which moves in a corkscrew shape as it propagates; below is an animation of a circularly polarized wave.

The polarization of the transmitted wave is completely controlled the choice of transmit antenna; different antenna shapes and configurations leads to differently polarized waves.

A way to be able to create any polarization is to jointly control a vertically polarized antenna and a horizontally polarized antenna. If we change how energy is distributed between the two and allow for a phase difference, we can generate whatever polarization is desired. To see this, let us look at the following interactive example with some interesting examples to try:

| Polarization        | Orientation Angle (deg) | Phase Difference (deg) |
|---------------------|-------------------------|------------------------|
| Linear, Horizontal  | 0                       | 0                      |
| Linear, Vertical    | 90                      | 0                      |
| Circular, Right-hand| 45                      | 90                     |
| Circular, Left-hand | 45                      | 270                    |

```python
ex.ex_3_1_8()
```

Differently polarized waves will create different strength echoes from targets based on a target's shape and material properties. For instance, a horizontally polarized wave hitting a long horizontal edge, e.g., an aircraft wing, will create a very strong echo; however, a vertically polarized wave hitting the same horizontal edge will create a very weak echo. Transmit polarization is generally chosen to be able to create strong echoes over the expected targets and geometries.

## Radar Receptions

Just like transmission, there are the same design options when choosing how to listen to incoming signals:

* **Waveform**: Shape of the incoming wave of interest
* **Space**: Region to listen for incoming waves
* **Polarization**: Orientation of the incoming wave of interest relative to direction of travel

We will go over each of these in detail.

### Waveform

For reception, we almost always want to find the waveform that was transmitted. This is akin to using our voice for acoustic ranging and only listening for echoes of the word that we shouted. Finding the presence of the transmitted waveform is called **matched filtering**, because we are filtering out all other signals except those matched to the transmitted wave. (Note that matched filtering is also often called **pulse compression**.)

The good news is that we know the waveform we transmitted because we chose it; the bad news is that we do not know at what time the echo signal will be arriving. This means we have to search over all possible delays for the presence of echoes. Each possible arrival time we will call a **delay hypothesis**. To test for the presence of an echo, we will take an *inner product*[2](#foot_inner) of the received signal with the transmitted waveform. An inner product tells us how much one signal is like another; if it is a perfect match, the value will be large, otherwise, the value will be small.

In the following interactive example, we can see how a matched filter works. Try changing the *Delay Hypothesis* and watch how the inner product value (shown as a **red** dot in the lower plot) changes; the true echo is in **blue** and the hypothesized echo is in **red**. When the *Delay Hypothesis* is at zero, i.e., a perfect match, the value is at its maximum; otherwise, it trails off. Also, take note on how the *Pulsewidth* affects the shape of the output—we will return to this.

```python
ex.ex_3_1_9()
```

In the **black** dashed line, we see the curve that the matched filter output takes over all possible delay hypotheses; this is called a **delay profile**. The *delay profile* is key for measuring how well a radar system can resolve different targets. This is because the total received signal will be a superposition of a collection of delay profiles, one for each target at its according range. In particular, we will look at **delay resolution**, $\delta \tau$, which is the minimum distance in time required to confidently discern two echoes.

To see how the delay profile relates to delay resolution, we can look at the following interactive example. In this example, we will look at the filter output over all delay hypotheses when receiving two echoes from variably spaced targets. We can now change the *Range Separation* between targets and the *Pulsewidth*; the true target ranges are illustrated as **black** dashed lines. Note how the two copies of the delay profile interact as the targets spread apart with increasing *Range Separation*.

```python
ex.ex_3_1_10()
```

We can note that as the pulsewidth increases, it becomes more difficult to confidently separate the two target echoes. However, if we shorten the pulsewidth, we will also be lowering the transmit energy, potentially making the echoes too weak to observe. Luckily, there is a way to improve the resolution of targets in delay (i.e., range) without shortening the pulsewidth. To see how this is done, let us revisit the matched filter output interactive example, but now we are using a LFM waveform with an adjustable *Start Frequency* and *Stop Frequency*. Watch what happens to the delay profile shape when adjusting the span of frequencies used by the LFM waveform.

```python
ex.ex_3_1_11()
```

As the spread of frequencies increase, the delay profile narrows, which will improve the ability to resolve targets in range. The spread of frequency content of a waveform (i.e., the distance between the start and stop frequencies) is called its **bandwidth**, $B$. It can be shown[[1]](#ref_skolnik) that the delay resolution, $\delta \tau$ , is:

$$
\delta \tau = \frac{1}{B}
$$

Likewise, we can use this result and the relationship between delay and range $r = c\tau/2$ to derive the **range resolution** (i.e., distance required in range to confidently resolve two target echoes), $\delta r$ , as:

$$
\delta r = \frac{c}{2B}
$$

For example, if we use an LFM waveform that spans a bandwidth of $B = 300~\mathrm{MHz}$, the minimum distance needed to resolve to targets in range is:

$$
\begin{align}
\delta r &= \frac{c}{2B}\\
&= \frac{3\times 10^8}{2\cdot 3\times 10^8}\\
&= 0.5~\mathrm{m}
\end{align}
$$

Let us revisit the delay resolution example but now with an LFM waveform. Note how we can improve resolution of the targets by increasing the bandwidth of the signal. Another important property to notice is that at bandwidths  greater than $1~\mathrm{MHz}$, the pulsewidth has very little effect on the delay resolution. This is a powerful result as it means we can have long pulsewidths to impart large amounts of transmit energy while also controlling our delay and range resolution using bandwidth.

```python
ex.ex_3_1_12()
```

One thing to note is that increasing the bandwidth of the waveform does come with a cost implication for the radar. Higher bandwidth transmit waveforms require more sophisticated equipment for generation and amplification of the signal to be transmitted along with more advanced receiver hardware to appropriately process the echo signals.

***

### Question 5

What transmit waveform bandwidth, $B$ , is required to be able to discern targets with a range separation of at least $\delta r = 10~\mathrm{m}$?

```python
qz.quiz_3_1_5()
```

```python
# Scratch space
```

***

Thus, the bandwidth of the transmit waveform is the primary driver for how well we can resolve to targets in range; this is why *modulation* of the transmit waveform is performed: to spread the waveform to a span of frequencies and, therefore, designing the range resolution capability of the radar.

### Space

Just as we did with the waveform used for reception, we will typically match the spatial degree-of-freedom on reception to the spatial degree-of-freedom on transmit. This means that we will only listen for echoes in the region of space in which we transmitted energy.

Focusing and steering are performed in the exact same way as transmit:

- Focusing is performed by using a receive aperture to collect signals from a swath of angle in space
- Steering is performed by either moving the aperture (i.e., mechanical) or appropriately combining the outputs of elements of a receive array (i.e., electronic); phase and delay steering calculations are done in an identical manner to transmission.

### Polarization

Similar to transmit polarization, receive polarization is dictated by the antenna hardware used for reception. Further, the polarization characteristics of the radar are chosen, if possible, to match the polarization of target echoes to strengthen their presence in the received signals. Note that there may be other emissions present that polarization is chosen to suppress on reception, e.g., echoes from buildings or a sea surface, emissions from other sensors.

## Integration

One last important topic relating to transmission and receptions are their *combination*. Let us say we have reached the limit of our budget for building a radar but still need to observe targets beyond our designed sensitivity. One way to achieve this is through combining multiple received signals in a process called **integration**.

Integration relies on the fact that, for each target observation, the shape and characteristics of target echo will be remain largely the same; the noise, however, is random from observation to observation. We can exploit this fact and average out the effect of the noise by adding together received pulses (see figure below). Note that if the target is moving quickly, we will have to account for the Doppler effect during integration—this will be covered in [Lab 4.1: Target Parameter Estimation](4_2_Target_Parameter_Estimation.ipynb).

There are two types of integration:

- **Coherent integration**: Uses phase and amplitude information from the incoming echoes when averaging over multiple observations
- **Noncoherent integration**: Uses only amplitude information from the incoming echoes when averaging over multiple observations

For coherent integration using $N$ observations, the resulting signal-to-noise ratio after integration, $\mathrm{SNR}_{c}$ , compared to a single pulse SNR, $\mathrm{SNR}_0$ , will be:

$$
\mathrm{SNR}_{c} = N \cdot \mathrm{SNR}_0
$$

Thus, we can achieve an $N$-fold improvement in SNR via coherent integration with $N$ pulses.

Coherent integration requires certain stability guarantees in radar hardware that may increase cost prohibitively. In that case, noncoherent integration can be used; noncoherent integration is more robust to radar system instabilities, but the gain in SNR will be appreciably smaller. A rough estimate of the resulting SNR for noncoherent integration, $\mathrm{SNR}_{n}$ , is:

$$
\mathrm{SNR}_{n} \approx \sqrt{N} \cdot \mathrm{SNR}_0
$$

## Robby Revisited

To see how some of the topics discussed above, e.g., waveform bandwidth, integration, affect the output of a radar, we will again look at Robby the test radar. We can now define Robby in terminology we have recently learned:

- **Waveform**: Robby uses a LFM waveform for transmission and performs matched filtering on receive; the display shows the output of the matched filter
- **Space**: Robby focuses energy using a dish to create a circular aperture and then performs mechanical steering to rotate
- **Polarization**: Robby uses vertical polarization on transmit and vertical polarization on receive; we will not see enough target detail in this course to exercise other polarizations

As compared to [Lab 2.2: Basic Radar Design](2_2_Basic_Radar_Design.ipynb), there are a few new knobs that can be adjusted when designing the transmission and processing receptions:

- *Bandwidth*: Bandwidth of the transmitted waveform; dictates the range resolution of the radar system
- *Number of Integrated Pulses*: Number of pulses used in integration; a value of one means no integration is performed
- *Number of Integrated Pulses*: Toggles whether coherent or noncoherent integration is used

Again there is a test target at azimuth $\theta = 45~\mathrm{deg}$ and range $r = 6~\mathrm{km}$. First, adjust *Dish Radius*, *Transmit Energy*, and *Transmit Energy* to be able to see the target. Then, adjust the new parameters to see how the radar output changes. Note that the display of Robby is actually showing the matched filter output discussed above.

```python
ex.ex_3_1_13()
```

## Summary

In this lab, we covered the decisions made by the transmitter, which decides how and where to transmit energy, and the receiver, which decides how and where to listen for echoes. For both transmission and reception, the degrees-of-freedom are:

- Waveform
- Space
- Polarization

Typically, all the receive degrees-of-freedom are chosen to match the transmit degrees-of-freedom. For the waveform, this comes in the form of a matched filter, which looks for the presence of echoes of the transmit waveform. By analyzing the performance of a matched filter, we saw that the ability to resolve two targets in range is dictated by the bandwidth of the transmit waveform, i.e., its span of frequencies.

Additionally, multiple observations can be combined via integration, or averaging, to improve the signal-to-noise ratio. This can be done both using phase information, i.e., coherent, and without, i.e., noncoherent.

## Footnotes

1The range at which the far field starts for an aperture, $r_f$, (called the **Fraunhofer distance**) is defined as:

$$
r_f = \frac{2D^2}{\lambda}
$$

where $D$ is the largest extent of the aperture (in $\mathrm{m}$) and $\lambda$ is the transmit wavelength (in $\mathrm{m}$). We can see that $r_f$ increases with frequency and aperture size.

2The inner product is the same notion as the *dot product* in linear algebra. For vectors (and discrete signals), this is simply multiplying entries and summing over the products. For continuous signals, this is done via multiplication and integration. Luckily, calculating the inner product over a range of possible delay hypotheses can be made very efficient using a fast Fourier transform.

## References

[1] M. I. Skolnik, “Theoretical accuracy of radar measurements,” *IRE Transactions on Aeronautical and Navigational Electronics*, vol. 7, no. 4, pp. 123–129, Dec. 1960.


```python
from __future__ import print_function
import sisl
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

In this example you will familiarize your-self to the concept of *buffer atoms*. A buffer atom is an atom that is completely neglected in the self-consistent calculation, but is used as an initialization for the bulk electrode regions.

Here a pristine graphene flake will be constructed and subsequently a Carbon chain will act as an STM-like tip to simulate STM experiments.

As the Carbon chain is terminated to vacuum, the dangling bonds will create spurious effects very different from a pristine, bulk chain. To understand why it is necessary to add buffer atoms it is useful to understand the TranSiesta method. **Any** TranSiesta calculation starts with calculating an initial guess for the Hamiltonian as input for the Green function method:
\begin{equation}
   \mathbf G^{-1}(E) = \mathbf S (E+i\eta) - \mathbf H - \sum_i\boldsymbol\Sigma_i
\end{equation}
If the initial $\mathbf H'$ represents a Hamiltonian close to the open-boundary problem $\mathbf H$; it will converge with a higher probability, and in much less time. Improving the initial guess Hamiltonian is time well-worth spent as TranSiesta is, typically, more difficult to converge.

As an example consider the Hamiltonian for the chain:

     <vacuum>     C -- C -- C -- C -- C -- C ...

It is clear that the atom closest to the vacuum region resides in a *very* different chemical and potential landscape than an atom in the middle of the chain. If TranSiesta uses the initial Hamiltonian for the chain electrode as the atom closest to the vacuum region it will be very far from the potential landscape of a bulk electrode. So to mitigate this one can specify:

    %block TBT.Atoms.Buffer
      atom [ 1 -- 4 ]
    %endblock

to remove the first 4 atoms from the calculation (note that negative indices counts from the *end*). Then the electrode will begin from the 5th atom which is relatively far from the dangling bond. This will be a *much* better initial guess for the Hamiltonian.

```python
graphene = sisl.geom.graphene(1.44)
elec = graphene.tile(2, axis=0)
elec.write('ELEC_GRAPHENE.fdf')
elec.write('ELEC_GRAPHENE.xyz')
```

```python
C1d = sisl.Geometry([[0,0,0]], graphene.atom[0], [10, 10, 1.4])
elec_chain = C1d.tile(4, axis=2)
elec_chain.write('ELEC_CHAIN.fdf')
elec_chain.write('ELEC_CHAIN.xyz')
chain = elec_chain.tile(4, axis=2)
```

```python
device = elec.repeat(5, axis=1).tile(4, axis=0)

# Attach the chain on-top of an atom
# First find an atom in the middle of the device
idx = device.close(device.center(what='xyz'), R=1.45)[1]
# Attach the chain at a distance of 2.25 along the third lattice vector
device = device.attach(idx, chain, 0, dist=2.25, axis=2)
# Add vacuum along chain, we really no not care how much vacuum, but it
# is costly on memory, not so much on performance.
device = device.add_vacuum(15, axis=2)
device.write('DEVICE.fdf')
device.write('DEVICE.xyz')
```

## Exercises

- Perform all required TranSiesta calculations, first the electrodes, then the device region.
- Create a new directory for a different range of buffer atoms, from 0 to 4, start by using 4 buffer atoms.
  How does convergence behave for different number of buffer atoms?

  **REMARK** there are 2 places in the fdf file you should change when changing the number of atoms (the electrode atom specification *and* the buffer atoms).
- Calculate transport properties for all (converged) TranSiesta calculations
- Plot the transmission and DOS for all TBtrans calculations, do they differ?

```python
tbt = sisl.get_sile('siesta.TBT.nc')
```

```python

```


```python
from IPython.display import Image
Image('../../../python_for_probability_statistics_and_machine_learning.jpg')
```

[Python for Probability, Statistics, and Machine Learning](https://www.springer.com/fr/book/9783319307152)

```python
from __future__ import division
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

In a previous coin-flipping discussion, we discussed estimation of the
underlying probability of getting a heads. There, we derived the
estimator as

$$
\hat{p}_n = \frac{1}{n}\sum_{i=1}^n X_i
$$

 where $X_i\in \lbrace 0,1 \rbrace$. Confidence intervals allow us to
estimate how close we can get to the true value that we are estimating.
Logically, that seems strange, doesn't it? We really don't know the exact value
of what we are estimating (otherwise, why estimate it?), and yet, somehow we
know how close we can get to something we admit we don't know? Ultimately, we
want to make statements like the *probability of the value in a certain
interval is 90\%*. Unfortunately, that is something we will not be able to say
using our methods. Note that Bayesian estimation gets closer to this statement
by using *credible intervals*, but that is a story for another day. In our
situation, the best we can do is say roughly the following: *if we ran the
experiment multiple times, then the confidence interval would trap the true
parameter 90\% of the time*.

Let's return to our coin-flipping example and see this in action. One
way to get at a confidence interval is to use Hoeffding's inequality
from the section ref{ch:prob:sec:ineq}
specialized to our Bernoulli variables as

$$
\mathbb{P}(\mid \hat{p}_n-p\mid >\epsilon) \leq 2 \exp(-2n \epsilon^2)
$$

 Now, we can form the interval
$\mathbb{I}=[\hat{p}_n-\epsilon_n,\hat{p}_n+\epsilon_n]$, where $\epsilon_n$ is
carefully constructed as

$$
\epsilon_n = \sqrt{ \frac{1}{2 n}\log\frac{2}{\alpha}}
$$

 which makes the right-side of the Hoeffding inequality equal to
$\alpha$. Thus, we finally have

$$
\mathbb{P}(p \notin \mathbb{I}) = \mathbb{P}\left(\mid \hat{p}_n-p\mid >\epsilon_n\right) \leq \alpha
$$

Thus, $ \mathbb{P}(p \in \mathbb{I})
\geq 1-\alpha$. As a numerical example, let's take $n=100$, $\alpha=0.05$, then plugging into everything we have gives $\epsilon_n=0.136$.
So, the 95\% confidence interval here is therefore

$$
\mathbb{I}=[\hat{p}_n-\epsilon_n,\hat{p}_n+\epsilon_n] = [\hat{p}_n-0.136,\hat{p}_n+0.136]
$$

The following code sample is a simulation to see if we can really trap
the underlying parameter in our confidence interval.

```python
from scipy import stats
import numpy as np

b= stats.bernoulli(.5) # fair coin distribution
nsamples = 100
# flip it nsamples times for 200 estimates
xs = b.rvs(nsamples*200).reshape(nsamples,-1)
phat = np.mean(xs,axis=0) # estimated p
# edge of 95% confidence interval
epsilon_n=np.sqrt(np.log(2/0.05)/2/nsamples)
pct=np.logical_and(phat-epsilon_n<=0.5,
                   0.5 <= (epsilon_n +phat)
                   ).mean()*100
print 'Interval trapped correct value ', pct,'% of the time'
```

    Interval trapped correct value  99.0 % of the time

 The result shows that the estimator and the corresponding
interval was able to trap the true value at least 95\% of the time. This is how
to interpret the action of confidence intervals.

However, the usual practice is to not use Hoeffding's inequality and
instead use arguments around asymptotic normality.
The definition of the standard error is the following:

$$
\texttt{se} = \sqrt{\mathbb{V}(\hat{\theta}_n)}
$$

 where $\hat{\theta}_n$ is the point-estimator for the parameter
$\theta$, given $n$ samples of data $X_n$, and $\mathbb{V}(\hat{\theta}_n)$ is
the variance of $\hat{\theta}_n$. Likewise, the estimated standard error is
$\widehat{\texttt{se}}$. For example, in our coin-flipping example, the estimator
was $\hat{p}=\sum X_i/n$ with corresponding variance
$\mathbb{V}(\hat{p}_n)=p(1-p)/n$.  Plugging in the point estimate gives us the
estimated standard error: $\widehat{\texttt{se}}=\sqrt{\hat{p}(1-\hat{p})/n}$.
Because maximum likelihood estimators are asymptotically normal [^mle], we know
that $\hat{p}_n \sim \mathcal{N}(p,\widehat{\texttt{se}}^2)$. Thus, if we want
a $1-\alpha$ confidence interval, we can compute

[^mle]: Certain technical regularity conditions must hold for this property
of maximum likelihood estimator to work. See
[[wasserman2004all]](#wasserman2004all) for more details.

$$
\mathbb{P}(\mid \hat{p}_n-p\mid  < \xi)> 1-\alpha
$$

 but since we know that $ (\hat{p}_n-p)$ is asymptotically normal,
$\mathcal{N}(0,\widehat{\texttt{se}}^2)$, we can instead compute

$$
\int_{-\xi}^{\xi} \mathcal{N}(0,\widehat{\texttt{se}}^2) dx > 1-\alpha
$$

 This looks ugly to compute because we need to find $\xi$, but
Scipy has everything we need for this.

```python
# compute estimated se for all trials
se=np.sqrt(phat*(1-phat)/xs.shape[0])
# generate random variable for trial 0
rv=stats.norm(0, se[0])
# compute 95% confidence interval for that trial 0
np.array(rv.interval(0.95))+phat[0]
def compute_CI(i):
    return stats.norm.interval(0.95,loc=i,
                              scale=np.sqrt(i*(1-i)/xs.shape[0]))

lower,upper = compute_CI(phat)
```

The gray circles are the point estimates that are bounded above and below by both asymptotic confidence intervals and Hoeffding intervals. The asymptotic intervals are tighter because the underpinning asymptotic assumptions are valid for these estimates.

[Figure](#fig:Confidence_Intervals_001) shows the asymptotic
confidence intervals and the Hoeffding-derived confidence intervals.
As shown, the Hoeffding intervals are a bit more generous than the
asymptotic estimates.  However, this is only true so long as the
asympotic approximation is valid. In other words, there exists some
number of $n$ samples for which the asymptotic intervals may not work.
So, even though they may be a bit more generous, the Hoeffding
intervals do not require arguments about asymptotic convergence. In
practice, nonetheless, asymptotic convergence is always in play (even
if not explicitly stated).

### Confidence Intervals and Hypothesis testing

It turns out that there is a close dual relationship between hypothesis testing
and confidence intervals. To see this in action, consider the following
hypothesis test for a normal distribution, $H_0 :\mu=\mu_0$ versus $H_1: \mu
\neq \mu_0$. A reasonable test has the following rejection region:

$$
\left\{ x: \mid \bar{x}-\mu_0\mid  > z_{\alpha/2}\frac{\sigma}{\sqrt n} \right\}
$$

 where $\mathbb{P}(Z > z_{\alpha/2}) = \alpha/2$ and
$\mathbb{P}(-z_{\alpha/2}< Z < z_{\alpha/2}) = 1-\alpha$ and where $Z \sim
\mathcal{N}(0,1)$. This is the same thing as saying that the region
corresponding to acceptance of $H_0$ is then,

$$
\begin{equation}
\bar{x} -z_{\alpha/2}\frac{\sigma}{\sqrt n}  \leq \mu_0 \leq \bar{x} +z_{\alpha/2}\frac{\sigma}{\sqrt n}
\end{equation}
\label{eq:ci} \tag{1}
$$

 Because the test has size $\alpha$, the false alarm probability, $\mathbb{P}(H_0
\texttt{  rejected}\mid \mu=\mu_0)=\alpha$.  Likewise, the $\mathbb{P}(H_0
\texttt{  accepted}\mid \mu=\mu_0)=1-\alpha$. Putting this all together with
interval defined above means that

$$
\mathbb{P}\left(\bar{x} -z_{\alpha/2}\frac{\sigma}{\sqrt n} \leq \mu_0 \leq \bar{x}+z_{\alpha/2}\frac{\sigma}{\sqrt n} \Big\vert H_0\right)=1-\alpha
$$

 Because this is valid for any $\mu_0$, we can drop the $H_0$ condition
and say the following:

$$
\mathbb{P}\left(\bar{x} -z_{\alpha/2}\frac{\sigma}{\sqrt n}  \leq \mu_0 \leq \bar{x} +z_{\alpha/2}\frac{\sigma}{\sqrt n} \right) =1-\alpha
$$

As may be obvious by now, the interval in Equation ref{eq:ci} above *is* the
$1-\alpha$ confidence interval! Thus, we have just obtained the confidence
interval by inverting the acceptance region of the level $\alpha$ test. The
hypothesis test fixes the *parameter* and then asks what sample values
(i.e., the acceptance region) are consistent with that fixed value.
Alternatively, the confidence interval fixes the sample value and then asks
what parameter values (i.e., the confidence interval) make this sample value
most plausible. Note that sometimes this inversion method results in disjoint
intervals (known as *confidence sets*).


```python
import numpy as np    # Numeric python
import sympy as sp    # Symbolic python
import pandas as pd   # Data processing python

# By convention, import all modules at the top of the notebook so they don't get in the way later
# Import only the modules you actually use!

# Plotting library
import matplotlib.pyplot as plt

# Needed only to render matplotlib plots inside the notebook properly
%matplotlib inline
```

# Example Jupyter Notebook

Your name here

Current date

## Introduction

A Jupyter notebook is organized into "cells" that execute independently.  Double-clicking in a cell places it in "edit" mode.  Shift-enter executes the cell.  Other key comamnd help is available in the Help menu at the top of the notebook.

Each cell can be marked as a Markdown cell or a Code cell.  Code cells execute python code directly by default, though there are number of back-ends for different languages that are becoming available.  One should organize the cells in a logical way, so that each cell is a self-contained logical chunk of either text or code.

Markdown is a simplified markup language for html, which focuses on content-oriented readability.  The main header of the notebook should be prepended with "#" to indicate an h1 header environment.  Sections should be prepended with "##" to indicate an h2 header environment.  Subsections should be prepended with "###" etc.  Text with no prepended symbol renders as a p (paragraph) environment.  Make sure you put blank lines between different Markdown environments so the parser renders it properly.

Markdown cells can also render $\LaTeX$ math environments by enclosing the code in dollar-signs: \$code here\$.  For example, $\sum_{n=1}^\infty \frac{x^n}{n!} = \exp(x)$. If you desire a separated equation that is centered on the page, this is accomplished by doubling the dollar-signs: \$\$code here\$\$. For example, $$\int_a^b \! \frac{\mathrm{d}f(x)}{\mathrm{d}x}\,\mathrm{d}x \equiv f(b) - f(a)$$

### Creating Reports

When preparing a Jupyter notebook, treat it like a professional report. It should be a well-formatted, aesthetic-looking document that describes exactly what you are doing and why. Large blocks of code should be imported as modules, rather than copy-pasted whole-sale into the notebook.  Any code that appears in the notebook should be short, sweet, and to the point.  As an example, here is a demonstration of the plotting package Matplotlib, which is designed to be as similar as possible to the way MATLAB plots:

```python
# Plot a sum of sines using Matplotlib

x = np.linspace(0,10,100)
f = 2 * np.sin(2*x) + 3 * np.cos(x)

fig = plt.figure(1)
plt.plot(x, f)
plt.xlabel("x")
plt.ylabel("f(x)")
plt.show(fig)
```

Code block in notebooks should not be much longer than the above. Ideally, they should be single function calls. Longer code blocks should be put into modules with extension .py and then imported at the beginning of the notebook.  The notebook itself should focus on results, not on reading long blocks of code.

To include web images, the format is as follows (note the lack of quotes around the image filename):

# Exporting your notebook

In the File menu, you can Download your notebook as a number of different formats if desired, including pdf via $\LaTeX$ if you have it installed and configured properly.  A notebook that is properly formatted with Markdown headings and paragraphs should generally produce a reasonably good looking pdf document.

## Conclusion

Happy notebooking.


```python
%matplotlib inline
```

# Stepsize adaption in the CQE method

This example demonstrates the ability of the conservation quantity error (CQE)
method [H2009]_ to locally decrease the stepsize when higher accuracy is
needed. As test case, the interaction dynamics of two colliding fundamental
soliton governed by the standard nonlinear Schrödinger equation is considered.

In contrast to the LEM, which uses a local error derived by means of
step-doubling and extrapolation, the CQE uses a conservation quantity of the
underlying propagation equation to drive stepsize adaption.

.. codeauthor:: Oliver Melchert <melchert@iqo.uni-hannover.de>

We first import the functionality needed to perform the sequence of numerical
experiments:

```python
import sys
import numpy as np
from fmas.models import ModelBaseClass
from fmas.config import FTFREQ, FT, IFT, C0
from fmas.grid import Grid
from fmas.solver import CQE
from fmas.data_io import save_h5
from fmas.tools import plot_evolution
```

Next, we implement a model for the nonlinear Schrödinger equation.  In
particular, we here consider the standard nonlinear Schrödinger equation,
given by

\begin{align}\partial_z u = -i \frac{\beta_2}{2}\partial_t^2 u + i\gamma |u|^2 u,\end{align}

wherein $u = u(z, t)$ represents the slowly varying pulse envelope,
$\beta_2=-1$ is the second order dispersion parameter, and
$\gamma=1$ is the nonlinear parameter:

```python
class NSE(ModelBaseClass):

    def __init__(self, w, beta, gamma):
        super().__init__(w, beta_w=beta)
        self.gamma = gamma

    @property
    def Lw(self):
        return 1j*self.beta_w

    def N(self, uw):
        ut = IFT(uw)
        return 1j*self.gamma*FT(np.abs(ut)**2*ut)
```

Next, we set up the computational domain, the model, and the LEM solver and
prepare an initial condition with two fundamental solitons. The velocity of
the solitons is adjusted so that they collide after approximately half a
soliton period.

To construct the initial condition, we use the exact single-soliton solution
of the nonlinar Schrödinger equation, given by

\begin{align}u_{\rm{exact}}(z,t) = \sqrt{P_0} {\rm{sech}}(t/t_0)\,e^{-i\gamma P_0 z/2},\end{align}

with $P_0=|\beta_2|/(\gamma t_0^2)$. We here consider two fundamental
solitons of duration $t_0=1$ and frequency detunings
$\omega_0=25$ to construct the initial condition

.. math:
   u_0(t) = u{\rm{exact}}(0, t+t_{\rm{off}})\,e^{i \omega_0 t}+
            u{\rm{exact}}(0, t-t_{\rm{off}})\,e^{- i \omega_0 t}.

The propagation is performed up to $z_{\rm{max}}=\pi/2$, i.e.  for one
soliton period.

```python
# -- SET MODEL PARAMETERS
t_max = -50.
Nt = 2**12
# ... PROPAGATION CONSTANT (POLYNOMIAL MODEL)
beta = np.poly1d([-0.5, 0.0, 0.0])
beta1 = np.polyder(beta, m=1)
beta2 = np.polyder(beta, m=2)
# ... NONLINEAR PARAMETER
gamma = 1.
# ... SOLITON PARAMTERS
t0 = 1.                             # duration
t_off = 20.                         # temporal offset
w0 = 25.                            # detuning
P0 = np.abs(beta2(0))/t0/t0/gamma   # peak-intensity
LD = t0*t0/np.abs(beta2(0))         # dispersion length
# ... EXACT SOLUTION
u_exact = lambda z, t: np.sqrt(P0)*np.exp(0.5j*gamma*P0*z)/np.cosh(t/t0)

# -- INITIALIZATION STAGE
# ... COMPUTATIONAL DOMAIN
grid = Grid(t_max=t_max, t_num=Nt)
t, w = grid.t, grid.w
# ... NONLINEAR SCHROEDINGER EQUATION
model = NSE(w, beta(w), gamma)
# ... PROPAGATION ALGORITHM
```

In order to drive stepsize adaption, the CQE method monitors a conservation
law exhibited by the underlying propagation equation. Here, for the standard
nonlinear Schrödinger equation, we opt to use the conserved quantity

.. math:
   C_E = \int |u(z,t)|^2 ~ {\rm{d}}t,

implemented by the function:

```python
def my_CQE_fun(i, zi, w, uw):
    return np.sum(np.abs(uw)**2)
```

The function can be set when an instance of the solver is initialized:

```python
solver = CQE(model.Lw, model.N, del_G = 1e-7, user_action = my_CQE_fun)
# ... INITIAL CONDITION
u0_t  = u_exact(0.0, t+t_off)*np.exp(1j*w0*t)
u0_t += u_exact(0.0, t-t_off)*np.exp(-1j*w0*t)
solver.set_initial_condition(w, FT(u0_t))

# -- RUN SOLVER
solver.propagate(
    z_range = 0.5*np.pi*LD,     # propagation range
    n_steps = 512,
    n_skip = 2
)
```

The figure below shows the propagation dynamics of the above initial
condition:

```python
plot_evolution( solver.z, grid.t, solver.utz, t_lim = (-30,30), w_lim = (-50.,50.))
```

Below we prepare a figure showing the variation of the local relative error
upon propagation (top figure), and the  decrease of the local stepsize in the
vicinity of the soliton-soliton collision (bottom subfigure).
In the top figure, the shaded region indicates the local goal error range.
Aim of the CQE method is to keep the conservation quantity error within that
range.

```python
# sphinx_gallery_thumbnail_number = 2

import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.colors as col

f, (ax1,ax2) = plt.subplots(2, 1, figsize=(8,6))

ax1.plot(range(len(solver._del_rle)), solver._del_rle)
ax1.axhspan(0.1e-7,1e-7,color='lightgray')
ax1.set_ylabel(r"$\delta_{\rm{CQE}}$")

ax2.plot(range(len(solver._dz_a)), solver._dz_a)
ax2.set_ylabel(r"$h~{(\mathrm{\mu m})}$")
ax2.set_xlabel(r"$z$-slice number $n$")

plt.show()
```

**References:**

.. [H2009] A. M. Heidt,
       Efficient Adaptive Step Size Method for the Simulation of
       Supercontinuum Generation in Optical Fibers,
       IEEE J. Lightwave Tech. 27 (2009) 3984,
       https://doi.org/10.1109/JLT.2009.2021538