$\Huge{\text{Linear Regression with Regularizers}}$

In this session, we shall learn about linear regression when we impose regularizers on the parameters.

Let us first consider a sample data, which will be useful for our study of linear regression with regularizers.

```python
#First, we import the required packages
import pandas as pd #the pandas library is useful for data processing
import matplotlib.pyplot as plt #the matplotlib library is useful for plotting purposes
import numpy as np
# The following python directive helps to plot the graph in the notebook directly
%matplotlib inline
```

Now let us consider some open source data sets available in the internet.
The data set we will consider represents airline costs for different airline companies as a response variable dependent on multiple attributes.

```python
#Let us explore the data and the description

#Get the data from the web
airline_costs_data = pd.read_csv('https://raw.githubusercontent.com/balamurugan-palaniappan-CEP/AIML_CEP_2021/main/data/airline_costs.txt', delim_whitespace=True, index_col=False, header=None)

#print the first few rows of the data
airline_costs_data.head()

```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0
      AllAmerican
      57
      133
      6.10
      20200
      116.3
      0.96
      0.400
      2.400
      21.13
      3.21
      17.92

      1
      American
      270
      216
      6.93
      56928
      43.0
      3.98
      0.689
      5.776
      1436.53
      165.22
      1271.31

      2
      Bonanza
      100
      140
      4.45
      183
      141.5
      0.79
      0.358
      2.207
      6.65
      0.01
      6.64

      3
      Braniff
      176
      182
      6.60
      11869
      50.6
      2.57
      0.557
      4.614
      160.30
      5.81
      154.49

      4
      Capital
      142
      167
      7.47
      41097
      51.0
      2.68
      0.510
      5.255
      195.02
      6.06
      188.96

```python
#if you wish to print the entire data, then use the following
airline_costs_data
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0
      AllAmerican
      57
      133
      6.10
      20200
      116.3
      0.96
      0.400
      2.400
      21.13
      3.21
      17.92

      1
      American
      270
      216
      6.93
      56928
      43.0
      3.98
      0.689
      5.776
      1436.53
      165.22
      1271.31

      2
      Bonanza
      100
      140
      4.45
      183
      141.5
      0.79
      0.358
      2.207
      6.65
      0.01
      6.64

      3
      Braniff
      176
      182
      6.60
      11869
      50.6
      2.57
      0.557
      4.614
      160.30
      5.81
      154.49

      4
      Capital
      142
      167
      7.47
      41097
      51.0
      2.68
      0.510
      5.255
      195.02
      6.06
      188.96

      5
      Central
      51
      134
      4.67
      1757
      318.5
      0.35
      0.167
      2.096
      14.02
      0.01
      14.01

      6
      C&S
      175
      175
      8.60
      18000
      59.2
      2.17
      0.558
      3.889
      114.16
      3.11
      111.05

      7
      Colonial
      112
      150
      6.87
      13500
      77.0
      1.68
      0.505
      3.327
      35.34
      0.79
      34.55

      8
      Continental
      131
      179
      6.50
      3831
      62.3
      1.70
      0.537
      3.166
      49.74
      3.49
      46.25

      9
      Delta
      174
      191
      7.60
      13119
      45.3
      2.74
      0.598
      4.582
      174.32
      3.87
      170.45

      10
      Eastern
      182
      187
      9.50
      44000
      42.6
      3.07
      0.528
      5.814
      1042.58
      187.84
      854.74

      11
      Empire
      59
      143
      4.83
      451
      112.4
      0.69
      0.313
      2.204
      4.71
      0.02
      4.69

      12
      Frontier
      81
      141
      7.47
      2500
      125.2
      0.72
      0.398
      1.809
      17.66
      0.12
      17.54

      13
      LakeCentral
      73
      142
      3.85
      5405
      169.3
      0.48
      0.212
      2.264
      6.32
      0.01
      6.31

      14
      Mid-Continent
      144
      167
      6.20
      6725
      64.8
      1.53
      0.565
      2.708
      76.12
      2.44
      73.68

      15
      Mohawk
      79
      137
      5.88
      9250
      100.5
      1.02
      0.476
      2.143
      14.54
      0.09
      14.45

      16
      National
      199
      207
      8.28
      23431
      42.9
      3.12
      0.567
      5.503
      181.27
      16.14
      165.13

      17
      Northeast
      94
      150
      6.77
      16000
      81.1
      1.19
      0.597
      1.988
      49.38
      2.11
      47.27

      18
      Northwest
      271
      202
      6.88
      27000
      56.7
      4.30
      0.570
      7.544
      471.50
      7.98
      463.52

      19
      Piedmont
      90
      153
      8.13
      3362
      75.4
      1.08
      0.449
      2.405
      21.60
      3.10
      18.50

      20
      Pioneer
      89
      150
      7.05
      2050
      71.3
      1.16
      0.483
      2.402
      19.11
      0.01
      19.10

      21
      Southern
      67
      147
      6.15
      2337
      150.1
      0.60
      0.317
      1.893
      12.44
      0.01
      12.43

      22
      Southwest
      54
      127
      5.85
      5143
      78.9
      1.18
      0.491
      2.403
      16.06
      0.08
      15.98

      23
      Trans-Texas
      78
      147
      6.08
      2600
      130.3
      0.65
      0.372
      1.747
      11.10
      0.42
      10.68

      24
      TWA
      293
      193
      6.40
      51500
      46.2
      3.62
      0.670
      5.403
      1217.98
      120.07
      1097.91

      25
      United
      270
      207
      6.63
      48913
      42.3
      3.75
      0.630
      5.952
      1127.25
      79.98
      1047.27

      26
      WestCoast
      58
      135
      4.50
      1850
      103.3
      0.82
      0.421
      1.948
      10.98
      0.01
      10.97

      27
      Western
      172
      181
      6.67
      6250
      44.2
      2.68
      0.656
      4.085
      137.39
      17.02
      120.37

      28
      Inland
      147
      158
      9.38
      1000
      64.7
      1.54
      0.575
      2.678
      16.61
      0.03
      16.58

      29
      Wiggins
      45
      116
      2.35
      2500
      820.9
      0.07
      0.166
      0.422
      2.03
      0.62
      1.41

      30
      Wisconsin
      69
      142
      5.72
      6500
      130.9
      0.81
      0.430
      1.884
      10.80
      0.01
      10.79

What do the numbers in the above data mean? Let us understand the data by seeing its description.

```python
import urllib.request  #this package is useful for accessing text files over internet
drug_vs_mathscore_data_description_file = urllib.request.urlopen('http://users.stat.ufl.edu/~winner/data/airline_costs.txt')

for line in drug_vs_mathscore_data_description_file:
  decoded_line = line.decode("utf-8")
  print(decoded_line)
```

    Dataset:  airline_costs.dat

    Source: J.W. Proctor and J.S. Duncan (1954). "A Regression Analysis

    of Airline Costs," Journal of Air Law and Commerce, Vol.21, #3, pp.282-292.

    Description: Regression relating Operating Costs per revenue ton-mile

    to 7 factors: length of flight, speed of plane, daily flight time per aircraft,

    population served, ton-mile load factor, available tons per aircraft mile,

    and firms net assets. Regression based on natural logarithms of all

    factors, except load factor. Load factor and available tons (capacity)

    for Northeast Airlines was imputed from summary calculations.

    Variables/columns

    Airline   1-20

    Length of flight (miles)  22-28

    Speed of Plane (miles per hour)  30-36

    Daily Flight Time per plane (hours)  38-44

    Population served (1000s)   46-52

    Total Operating Cost (cents per revenue ton-mile)  54-60

    Revenue Tons per Aircraft mile   62-68

    Ton-Mile load factor (proportion)  70-76

    Available Capacity (Tons per mile)  78-84

    Total Assets  ($100,000s)   86-92

    Investments and Special Funds  ($100,000s)  94-100

    Adjusted Assets  ($100,000s)   102-108

Having known the data description, let us insert the descriptions into the data now.

```python
airline_costs_data.columns = ['Airline', 'Flight Length', 'Plane Speed','Daily flight time per plane', 'Population Served','TOC','Revenue','Load Factor','Capacity','Total Assets','Funds','Adjusted Assets']

#check by printing the data again
airline_costs_data.head()
```

      Airline
      Flight Length
      Plane Speed
      Daily flight time per plane
      Population Served
      TOC
      Revenue
      Load Factor
      Capacity
      Total Assets
      Funds
      Adjusted Assets

      0
      AllAmerican
      57
      133
      6.10
      20200
      116.3
      0.96
      0.400
      2.400
      21.13
      3.21
      17.92

      1
      American
      270
      216
      6.93
      56928
      43.0
      3.98
      0.689
      5.776
      1436.53
      165.22
      1271.31

      2
      Bonanza
      100
      140
      4.45
      183
      141.5
      0.79
      0.358
      2.207
      6.65
      0.01
      6.64

      3
      Braniff
      176
      182
      6.60
      11869
      50.6
      2.57
      0.557
      4.614
      160.30
      5.81
      154.49

      4
      Capital
      142
      167
      7.47
      41097
      51.0
      2.68
      0.510
      5.255
      195.02
      6.06
      188.96

Let us move the TOC column as the second column, just after the Airline company name.

```python
# shift column 'TOC' to second position
TOC_column = airline_costs_data.pop('TOC') #collect the contents of TOC  column into a temporary object

# insert column into the dataframe using insert(position,column_name,
# column_contents) function
airline_costs_data.insert(1, 'TOC', TOC_column) #Insert as second column, so position is 1

print("After shifting TOC column to second position")
airline_costs_data.head()
```

    After shifting TOC column to second position

      Airline
      TOC
      Flight Length
      Plane Speed
      Daily flight time per plane
      Population Served
      Revenue
      Load Factor
      Capacity
      Total Assets
      Funds
      Adjusted Assets

      0
      AllAmerican
      116.3
      57
      133
      6.10
      20200
      0.96
      0.400
      2.400
      21.13
      3.21
      17.92

      1
      American
      43.0
      270
      216
      6.93
      56928
      3.98
      0.689
      5.776
      1436.53
      165.22
      1271.31

      2
      Bonanza
      141.5
      100
      140
      4.45
      183
      0.79
      0.358
      2.207
      6.65
      0.01
      6.64

      3
      Braniff
      50.6
      176
      182
      6.60
      11869
      2.57
      0.557
      4.614
      160.30
      5.81
      154.49

      4
      Capital
      51.0
      142
      167
      7.47
      41097
      2.68
      0.510
      5.255
      195.02
      6.06
      188.96

Also note that Adjusted Assets column is sufficient for our analysis, since if is obtained as difference of Total Assets and Funds. However for this exercise, we shall $\textbf{not}$ remove Total Assets and Funds columns. Instead, we will keep them.

$\large{\text{How to estimate the parameters } \beta_j, j=0,1,\ldots,d?}$

One way to estimate the values of $\beta_j, j=0,1,\ldots,d$ is by minimizing the sum of squared errors given by:
$
\begin{align}
\min \sum_{i=1}^{n} (e^i)^2
\end{align}
$
which can be equivalently written as the following optimization problem:
$
\begin{align}
\min_{\beta_0,\beta_1,\ldots,\beta_d} \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2.
\end{align}
$

$\textbf{Note}$ that in the high dimensional case as well, the optimization problem is called the ordinary least squares (OLS) problem, and the term $\sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2$ is called the OLS objective function and we will denote the OLS objective by:

$
\begin{align}
L(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2.
\end{align}
$

$\Large{\text{Solving the OLS optimization problem}}:$

Assuming the responses $y^1,y^2,\ldots,y^n$ are present in a $n \times 1$ matrix represented as:

$
\mathbf{y}=\begin{bmatrix}
y^1 \\ y^2 \\ \vdots \\ y^n
\end{bmatrix}
$

and the predictor variables $\mathbf{x}^1,\mathbf{x}^2,\ldots,\mathbf{x}^d$ are placed in a $n\times (d+1)$ matrix represented as:

$
\mathbf{X}=\begin{bmatrix}
\leftarrow & \mathbf{x}^1 & \rightarrow & 1 \\
\leftarrow & \mathbf{x}^2 & \rightarrow & 1 \\
\leftarrow & \ldots & \rightarrow & 1  \\
\leftarrow & \mathbf{x}^n & \rightarrow & 1 \\
\end{bmatrix}
=
\begin{bmatrix}
{x}_1^1 & {x}_2^1 &\ldots & {x}_d^1 & 1 \\
{x}_1^2 & {x}_2^2 &\ldots & {x}_d^2 & 1 \\
\vdots & \vdots & \ldots & \vdots \\
{x}_1^n & {x}_2^n &\ldots & {x}_d^n & 1
\end{bmatrix}
$
where note that the last column of matrix $\mathbf{X}$ contains a column of all ones. This column is useful to incorporate the effect of $\beta_0$ parameter in $\mathbf{X}$ matrix.

Similarly assume that
 we can write the coefficients $\beta_0,\beta_1,\ldots,\beta_d$ as a $(d+1)\times 1$ matrix represented as:

$
\mathbf{\beta} = \begin{bmatrix}
\beta_1 \\ \beta_2 \\ \vdots \\ \beta_d \\ \beta_0
\end{bmatrix}.
$

Now we can write the objective function as:

$
L(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2.
$

To solve

$
\min_\beta L(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2,
$
we find the gradient with respect to $\beta$ and equate to zero.

Thus we get:

$
\begin{align}
\nabla_\beta L(\beta) &= \mathbf{0} \\
\implies -\mathbf{X}^\top \mathbf{y} + \mathbf{X}^\top\mathbf{X} \beta &= \mathbf{0} \\
\implies \beta &= (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}.
\end{align}
$

Note that the closed form expression for $\beta$ is valid only when $(\mathbf{X}^\top\mathbf{X})^{-1}$ is invertible. Otherwise we need to solve the system given by:

$
\mathbf{X}^\top\mathbf{X} \beta = \mathbf{X}^\top \mathbf{y}
$
using a solver.

$\Large{\text{Computing } \beta \text{ for the airline costs data set}}:$

```python
#Let us now compute beta
n = len(airline_costs_data.index) #number of data points in the data set
print('number of data points in the data set:',n)
```

    number of data points in the data set: 31

```python
#extract the predictor variable columns
cols = [2,3,4,5,6,7,8,9, 10, 11]
X_data = airline_costs_data[airline_costs_data.columns[cols]]

X_data
```

      Flight Length
      Plane Speed
      Daily flight time per plane
      Population Served
      Revenue
      Load Factor
      Capacity
      Total Assets
      Funds
      Adjusted Assets

      0
      57
      133
      6.10
      20200
      0.96
      0.400
      2.400
      21.13
      3.21
      17.92

      1
      270
      216
      6.93
      56928
      3.98
      0.689
      5.776
      1436.53
      165.22
      1271.31

      2
      100
      140
      4.45
      183
      0.79
      0.358
      2.207
      6.65
      0.01
      6.64

      3
      176
      182
      6.60
      11869
      2.57
      0.557
      4.614
      160.30
      5.81
      154.49

      4
      142
      167
      7.47
      41097
      2.68
      0.510
      5.255
      195.02
      6.06
      188.96

      5
      51
      134
      4.67
      1757
      0.35
      0.167
      2.096
      14.02
      0.01
      14.01

      6
      175
      175
      8.60
      18000
      2.17
      0.558
      3.889
      114.16
      3.11
      111.05

      7
      112
      150
      6.87
      13500
      1.68
      0.505
      3.327
      35.34
      0.79
      34.55

      8
      131
      179
      6.50
      3831
      1.70
      0.537
      3.166
      49.74
      3.49
      46.25

      9
      174
      191
      7.60
      13119
      2.74
      0.598
      4.582
      174.32
      3.87
      170.45

      10
      182
      187
      9.50
      44000
      3.07
      0.528
      5.814
      1042.58
      187.84
      854.74

      11
      59
      143
      4.83
      451
      0.69
      0.313
      2.204
      4.71
      0.02
      4.69

      12
      81
      141
      7.47
      2500
      0.72
      0.398
      1.809
      17.66
      0.12
      17.54

      13
      73
      142
      3.85
      5405
      0.48
      0.212
      2.264
      6.32
      0.01
      6.31

      14
      144
      167
      6.20
      6725
      1.53
      0.565
      2.708
      76.12
      2.44
      73.68

      15
      79
      137
      5.88
      9250
      1.02
      0.476
      2.143
      14.54
      0.09
      14.45

      16
      199
      207
      8.28
      23431
      3.12
      0.567
      5.503
      181.27
      16.14
      165.13

      17
      94
      150
      6.77
      16000
      1.19
      0.597
      1.988
      49.38
      2.11
      47.27

      18
      271
      202
      6.88
      27000
      4.30
      0.570
      7.544
      471.50
      7.98
      463.52

      19
      90
      153
      8.13
      3362
      1.08
      0.449
      2.405
      21.60
      3.10
      18.50

      20
      89
      150
      7.05
      2050
      1.16
      0.483
      2.402
      19.11
      0.01
      19.10

      21
      67
      147
      6.15
      2337
      0.60
      0.317
      1.893
      12.44
      0.01
      12.43

      22
      54
      127
      5.85
      5143
      1.18
      0.491
      2.403
      16.06
      0.08
      15.98

      23
      78
      147
      6.08
      2600
      0.65
      0.372
      1.747
      11.10
      0.42
      10.68

      24
      293
      193
      6.40
      51500
      3.62
      0.670
      5.403
      1217.98
      120.07
      1097.91

      25
      270
      207
      6.63
      48913
      3.75
      0.630
      5.952
      1127.25
      79.98
      1047.27

      26
      58
      135
      4.50
      1850
      0.82
      0.421
      1.948
      10.98
      0.01
      10.97

      27
      172
      181
      6.67
      6250
      2.68
      0.656
      4.085
      137.39
      17.02
      120.37

      28
      147
      158
      9.38
      1000
      1.54
      0.575
      2.678
      16.61
      0.03
      16.58

      29
      45
      116
      2.35
      2500
      0.07
      0.166
      0.422
      2.03
      0.62
      1.41

      30
      69
      142
      5.72
      6500
      0.81
      0.430
      1.884
      10.80
      0.01
      10.79

```python
#convert predictor variable columns into a numpy array

X_array = airline_costs_data[airline_costs_data.columns[cols]].to_numpy()
X_array
```

    array([[5.70000e+01, 1.33000e+02, 6.10000e+00, 2.02000e+04, 9.60000e-01,
            4.00000e-01, 2.40000e+00, 2.11300e+01, 3.21000e+00, 1.79200e+01],
           [2.70000e+02, 2.16000e+02, 6.93000e+00, 5.69280e+04, 3.98000e+00,
            6.89000e-01, 5.77600e+00, 1.43653e+03, 1.65220e+02, 1.27131e+03],
           [1.00000e+02, 1.40000e+02, 4.45000e+00, 1.83000e+02, 7.90000e-01,
            3.58000e-01, 2.20700e+00, 6.65000e+00, 1.00000e-02, 6.64000e+00],
           [1.76000e+02, 1.82000e+02, 6.60000e+00, 1.18690e+04, 2.57000e+00,
            5.57000e-01, 4.61400e+00, 1.60300e+02, 5.81000e+00, 1.54490e+02],
           [1.42000e+02, 1.67000e+02, 7.47000e+00, 4.10970e+04, 2.68000e+00,
            5.10000e-01, 5.25500e+00, 1.95020e+02, 6.06000e+00, 1.88960e+02],
           [5.10000e+01, 1.34000e+02, 4.67000e+00, 1.75700e+03, 3.50000e-01,
            1.67000e-01, 2.09600e+00, 1.40200e+01, 1.00000e-02, 1.40100e+01],
           [1.75000e+02, 1.75000e+02, 8.60000e+00, 1.80000e+04, 2.17000e+00,
            5.58000e-01, 3.88900e+00, 1.14160e+02, 3.11000e+00, 1.11050e+02],
           [1.12000e+02, 1.50000e+02, 6.87000e+00, 1.35000e+04, 1.68000e+00,
            5.05000e-01, 3.32700e+00, 3.53400e+01, 7.90000e-01, 3.45500e+01],
           [1.31000e+02, 1.79000e+02, 6.50000e+00, 3.83100e+03, 1.70000e+00,
            5.37000e-01, 3.16600e+00, 4.97400e+01, 3.49000e+00, 4.62500e+01],
           [1.74000e+02, 1.91000e+02, 7.60000e+00, 1.31190e+04, 2.74000e+00,
            5.98000e-01, 4.58200e+00, 1.74320e+02, 3.87000e+00, 1.70450e+02],
           [1.82000e+02, 1.87000e+02, 9.50000e+00, 4.40000e+04, 3.07000e+00,
            5.28000e-01, 5.81400e+00, 1.04258e+03, 1.87840e+02, 8.54740e+02],
           [5.90000e+01, 1.43000e+02, 4.83000e+00, 4.51000e+02, 6.90000e-01,
            3.13000e-01, 2.20400e+00, 4.71000e+00, 2.00000e-02, 4.69000e+00],
           [8.10000e+01, 1.41000e+02, 7.47000e+00, 2.50000e+03, 7.20000e-01,
            3.98000e-01, 1.80900e+00, 1.76600e+01, 1.20000e-01, 1.75400e+01],
           [7.30000e+01, 1.42000e+02, 3.85000e+00, 5.40500e+03, 4.80000e-01,
            2.12000e-01, 2.26400e+00, 6.32000e+00, 1.00000e-02, 6.31000e+00],
           [1.44000e+02, 1.67000e+02, 6.20000e+00, 6.72500e+03, 1.53000e+00,
            5.65000e-01, 2.70800e+00, 7.61200e+01, 2.44000e+00, 7.36800e+01],
           [7.90000e+01, 1.37000e+02, 5.88000e+00, 9.25000e+03, 1.02000e+00,
            4.76000e-01, 2.14300e+00, 1.45400e+01, 9.00000e-02, 1.44500e+01],
           [1.99000e+02, 2.07000e+02, 8.28000e+00, 2.34310e+04, 3.12000e+00,
            5.67000e-01, 5.50300e+00, 1.81270e+02, 1.61400e+01, 1.65130e+02],
           [9.40000e+01, 1.50000e+02, 6.77000e+00, 1.60000e+04, 1.19000e+00,
            5.97000e-01, 1.98800e+00, 4.93800e+01, 2.11000e+00, 4.72700e+01],
           [2.71000e+02, 2.02000e+02, 6.88000e+00, 2.70000e+04, 4.30000e+00,
            5.70000e-01, 7.54400e+00, 4.71500e+02, 7.98000e+00, 4.63520e+02],
           [9.00000e+01, 1.53000e+02, 8.13000e+00, 3.36200e+03, 1.08000e+00,
            4.49000e-01, 2.40500e+00, 2.16000e+01, 3.10000e+00, 1.85000e+01],
           [8.90000e+01, 1.50000e+02, 7.05000e+00, 2.05000e+03, 1.16000e+00,
            4.83000e-01, 2.40200e+00, 1.91100e+01, 1.00000e-02, 1.91000e+01],
           [6.70000e+01, 1.47000e+02, 6.15000e+00, 2.33700e+03, 6.00000e-01,
            3.17000e-01, 1.89300e+00, 1.24400e+01, 1.00000e-02, 1.24300e+01],
           [5.40000e+01, 1.27000e+02, 5.85000e+00, 5.14300e+03, 1.18000e+00,
            4.91000e-01, 2.40300e+00, 1.60600e+01, 8.00000e-02, 1.59800e+01],
           [7.80000e+01, 1.47000e+02, 6.08000e+00, 2.60000e+03, 6.50000e-01,
            3.72000e-01, 1.74700e+00, 1.11000e+01, 4.20000e-01, 1.06800e+01],
           [2.93000e+02, 1.93000e+02, 6.40000e+00, 5.15000e+04, 3.62000e+00,
            6.70000e-01, 5.40300e+00, 1.21798e+03, 1.20070e+02, 1.09791e+03],
           [2.70000e+02, 2.07000e+02, 6.63000e+00, 4.89130e+04, 3.75000e+00,
            6.30000e-01, 5.95200e+00, 1.12725e+03, 7.99800e+01, 1.04727e+03],
           [5.80000e+01, 1.35000e+02, 4.50000e+00, 1.85000e+03, 8.20000e-01,
            4.21000e-01, 1.94800e+00, 1.09800e+01, 1.00000e-02, 1.09700e+01],
           [1.72000e+02, 1.81000e+02, 6.67000e+00, 6.25000e+03, 2.68000e+00,
            6.56000e-01, 4.08500e+00, 1.37390e+02, 1.70200e+01, 1.20370e+02],
           [1.47000e+02, 1.58000e+02, 9.38000e+00, 1.00000e+03, 1.54000e+00,
            5.75000e-01, 2.67800e+00, 1.66100e+01, 3.00000e-02, 1.65800e+01],
           [4.50000e+01, 1.16000e+02, 2.35000e+00, 2.50000e+03, 7.00000e-02,
            1.66000e-01, 4.22000e-01, 2.03000e+00, 6.20000e-01, 1.41000e+00],
           [6.90000e+01, 1.42000e+02, 5.72000e+00, 6.50000e+03, 8.10000e-01,
            4.30000e-01, 1.88400e+00, 1.08000e+01, 1.00000e-02, 1.07900e+01]])

```python
import numpy as np
#add a column of ones to X_array
X = np.hstack((X_array, np.ones((X_array.shape[0], 1), dtype=X_array.dtype)))

X
```

    array([[5.70000e+01, 1.33000e+02, 6.10000e+00, 2.02000e+04, 9.60000e-01,
            4.00000e-01, 2.40000e+00, 2.11300e+01, 3.21000e+00, 1.79200e+01,
            1.00000e+00],
           [2.70000e+02, 2.16000e+02, 6.93000e+00, 5.69280e+04, 3.98000e+00,
            6.89000e-01, 5.77600e+00, 1.43653e+03, 1.65220e+02, 1.27131e+03,
            1.00000e+00],
           [1.00000e+02, 1.40000e+02, 4.45000e+00, 1.83000e+02, 7.90000e-01,
            3.58000e-01, 2.20700e+00, 6.65000e+00, 1.00000e-02, 6.64000e+00,
            1.00000e+00],
           [1.76000e+02, 1.82000e+02, 6.60000e+00, 1.18690e+04, 2.57000e+00,
            5.57000e-01, 4.61400e+00, 1.60300e+02, 5.81000e+00, 1.54490e+02,
            1.00000e+00],
           [1.42000e+02, 1.67000e+02, 7.47000e+00, 4.10970e+04, 2.68000e+00,
            5.10000e-01, 5.25500e+00, 1.95020e+02, 6.06000e+00, 1.88960e+02,
            1.00000e+00],
           [5.10000e+01, 1.34000e+02, 4.67000e+00, 1.75700e+03, 3.50000e-01,
            1.67000e-01, 2.09600e+00, 1.40200e+01, 1.00000e-02, 1.40100e+01,
            1.00000e+00],
           [1.75000e+02, 1.75000e+02, 8.60000e+00, 1.80000e+04, 2.17000e+00,
            5.58000e-01, 3.88900e+00, 1.14160e+02, 3.11000e+00, 1.11050e+02,
            1.00000e+00],
           [1.12000e+02, 1.50000e+02, 6.87000e+00, 1.35000e+04, 1.68000e+00,
            5.05000e-01, 3.32700e+00, 3.53400e+01, 7.90000e-01, 3.45500e+01,
            1.00000e+00],
           [1.31000e+02, 1.79000e+02, 6.50000e+00, 3.83100e+03, 1.70000e+00,
            5.37000e-01, 3.16600e+00, 4.97400e+01, 3.49000e+00, 4.62500e+01,
            1.00000e+00],
           [1.74000e+02, 1.91000e+02, 7.60000e+00, 1.31190e+04, 2.74000e+00,
            5.98000e-01, 4.58200e+00, 1.74320e+02, 3.87000e+00, 1.70450e+02,
            1.00000e+00],
           [1.82000e+02, 1.87000e+02, 9.50000e+00, 4.40000e+04, 3.07000e+00,
            5.28000e-01, 5.81400e+00, 1.04258e+03, 1.87840e+02, 8.54740e+02,
            1.00000e+00],
           [5.90000e+01, 1.43000e+02, 4.83000e+00, 4.51000e+02, 6.90000e-01,
            3.13000e-01, 2.20400e+00, 4.71000e+00, 2.00000e-02, 4.69000e+00,
            1.00000e+00],
           [8.10000e+01, 1.41000e+02, 7.47000e+00, 2.50000e+03, 7.20000e-01,
            3.98000e-01, 1.80900e+00, 1.76600e+01, 1.20000e-01, 1.75400e+01,
            1.00000e+00],
           [7.30000e+01, 1.42000e+02, 3.85000e+00, 5.40500e+03, 4.80000e-01,
            2.12000e-01, 2.26400e+00, 6.32000e+00, 1.00000e-02, 6.31000e+00,
            1.00000e+00],
           [1.44000e+02, 1.67000e+02, 6.20000e+00, 6.72500e+03, 1.53000e+00,
            5.65000e-01, 2.70800e+00, 7.61200e+01, 2.44000e+00, 7.36800e+01,
            1.00000e+00],
           [7.90000e+01, 1.37000e+02, 5.88000e+00, 9.25000e+03, 1.02000e+00,
            4.76000e-01, 2.14300e+00, 1.45400e+01, 9.00000e-02, 1.44500e+01,
            1.00000e+00],
           [1.99000e+02, 2.07000e+02, 8.28000e+00, 2.34310e+04, 3.12000e+00,
            5.67000e-01, 5.50300e+00, 1.81270e+02, 1.61400e+01, 1.65130e+02,
            1.00000e+00],
           [9.40000e+01, 1.50000e+02, 6.77000e+00, 1.60000e+04, 1.19000e+00,
            5.97000e-01, 1.98800e+00, 4.93800e+01, 2.11000e+00, 4.72700e+01,
            1.00000e+00],
           [2.71000e+02, 2.02000e+02, 6.88000e+00, 2.70000e+04, 4.30000e+00,
            5.70000e-01, 7.54400e+00, 4.71500e+02, 7.98000e+00, 4.63520e+02,
            1.00000e+00],
           [9.00000e+01, 1.53000e+02, 8.13000e+00, 3.36200e+03, 1.08000e+00,
            4.49000e-01, 2.40500e+00, 2.16000e+01, 3.10000e+00, 1.85000e+01,
            1.00000e+00],
           [8.90000e+01, 1.50000e+02, 7.05000e+00, 2.05000e+03, 1.16000e+00,
            4.83000e-01, 2.40200e+00, 1.91100e+01, 1.00000e-02, 1.91000e+01,
            1.00000e+00],
           [6.70000e+01, 1.47000e+02, 6.15000e+00, 2.33700e+03, 6.00000e-01,
            3.17000e-01, 1.89300e+00, 1.24400e+01, 1.00000e-02, 1.24300e+01,
            1.00000e+00],
           [5.40000e+01, 1.27000e+02, 5.85000e+00, 5.14300e+03, 1.18000e+00,
            4.91000e-01, 2.40300e+00, 1.60600e+01, 8.00000e-02, 1.59800e+01,
            1.00000e+00],
           [7.80000e+01, 1.47000e+02, 6.08000e+00, 2.60000e+03, 6.50000e-01,
            3.72000e-01, 1.74700e+00, 1.11000e+01, 4.20000e-01, 1.06800e+01,
            1.00000e+00],
           [2.93000e+02, 1.93000e+02, 6.40000e+00, 5.15000e+04, 3.62000e+00,
            6.70000e-01, 5.40300e+00, 1.21798e+03, 1.20070e+02, 1.09791e+03,
            1.00000e+00],
           [2.70000e+02, 2.07000e+02, 6.63000e+00, 4.89130e+04, 3.75000e+00,
            6.30000e-01, 5.95200e+00, 1.12725e+03, 7.99800e+01, 1.04727e+03,
            1.00000e+00],
           [5.80000e+01, 1.35000e+02, 4.50000e+00, 1.85000e+03, 8.20000e-01,
            4.21000e-01, 1.94800e+00, 1.09800e+01, 1.00000e-02, 1.09700e+01,
            1.00000e+00],
           [1.72000e+02, 1.81000e+02, 6.67000e+00, 6.25000e+03, 2.68000e+00,
            6.56000e-01, 4.08500e+00, 1.37390e+02, 1.70200e+01, 1.20370e+02,
            1.00000e+00],
           [1.47000e+02, 1.58000e+02, 9.38000e+00, 1.00000e+03, 1.54000e+00,
            5.75000e-01, 2.67800e+00, 1.66100e+01, 3.00000e-02, 1.65800e+01,
            1.00000e+00],
           [4.50000e+01, 1.16000e+02, 2.35000e+00, 2.50000e+03, 7.00000e-02,
            1.66000e-01, 4.22000e-01, 2.03000e+00, 6.20000e-01, 1.41000e+00,
            1.00000e+00],
           [6.90000e+01, 1.42000e+02, 5.72000e+00, 6.50000e+03, 8.10000e-01,
            4.30000e-01, 1.88400e+00, 1.08000e+01, 1.00000e-02, 1.07900e+01,
            1.00000e+00]])

```python
#convert predictor variable columns into a numpy array

response_cols = [1]
y = airline_costs_data[airline_costs_data.columns[response_cols]].to_numpy()
y

```

    array([[116.3],
           [ 43. ],
           [141.5],
           [ 50.6],
           [ 51. ],
           [318.5],
           [ 59.2],
           [ 77. ],
           [ 62.3],
           [ 45.3],
           [ 42.6],
           [112.4],
           [125.2],
           [169.3],
           [ 64.8],
           [100.5],
           [ 42.9],
           [ 81.1],
           [ 56.7],
           [ 75.4],
           [ 71.3],
           [150.1],
           [ 78.9],
           [130.3],
           [ 46.2],
           [ 42.3],
           [103.3],
           [ 44.2],
           [ 64.7],
           [820.9],
           [130.9]])

```python
#Now we are ready for finding beta
#print('X shape:',X.shape,'y shape:',y.shape)

XTX = np.matmul(np.transpose(X),X)
print('XTX shape:',XTX.shape)

Xy = np.matmul(np.transpose(X),y)
#print('Xy shape:',Xy.shape)

#solving a system of linear equations Ax=b, can be done using np.linalg.solve(A,b)

beta =np.linalg.solve(XTX,Xy)
#beta = np.matmul(np.linalg.inv(XTX),Xy)

print('beta',beta)

```

    XTX shape: (11, 11)
    beta [[ 7.52395660e-01]
     [-2.25317845e+00]
     [ 3.31746733e+00]
     [ 3.10628655e-03]
     [ 7.23197186e+02]
     [-2.17522853e+03]
     [-3.82138951e+02]
     [ 1.74533242e+01]
     [-1.60157778e+01]
     [-1.79765303e+01]
     [ 1.43301982e+03]]

```python
print(np.linalg.matrix_rank(XTX))
XTX_inv = np.linalg.inv(XTX)

#print(np.matmul(XTX,XTX_inv))
is_identity = np.all(np.matmul(XTX,XTX_inv)==np.identity(XTX.shape[0]))
print(is_identity)
```

    10
    False

$\textbf{Note:}$ Though the matrix $X^\top X$ is not full-rank, that is, $10:= \text{rank}(X^\top X)<11$, we see that $\texttt{numpy}$ does not raise any error while computing the inverse of $X^\top X$. Hence we will use $\texttt{scipy}$ to solve for $\beta$ and check if we get any error.

```python
import scipy.linalg
#Now we are ready for finding beta
#print('X shape:',X.shape,'y shape:',y.shape)

XTX = np.matmul(np.transpose(X),X)
#print('XTX shape:',XTX.shape)

Xy = np.matmul(np.transpose(X),y)
#print('Xy shape:',Xy.shape)

beta =scipy.linalg.solve(XTX,Xy)
#beta = np.matmul(scipy.linalg.inv(XTX),Xy)
```

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:11: LinAlgWarning: Ill-conditioned matrix (rcond=7.39613e-21): result may not be accurate.
      # This is added back by InteractiveShellApp.init_path()

$\textbf{Note:}$ $\texttt{scipy}$ gives a warning that the matrix $X^\top X$ is ill-conditioned.

We can use the condition number of a matrix $A$ to identify if the matrix is ill-conditioned or not.

Suppose we solve a system $Ax=b$, then the condition number of $A$ denotes the amount of error that can be incurred in the decision variable $x$ when there is a small error in $b$.

For a symmetric matrix $A$, we can compute the condition number of $A$ denoted by $\kappa(A)$ as:

$\begin{align}
\kappa(A) = \frac{|\lambda_{\max}(A)|}{|\lambda_{\min}(A)|}
\end{align}
$

where $\lambda_{\max}(A)$ and $\lambda_{\min}(A)$ denote respectively the maximum and minimum eigen values of $A$ and $|z|$ denotes the absolute value of a real number $z$.

If the condition number $\kappa(A)$ is a $\textbf{large number}$ then the error incurred in $x$ would be large and hence the matrix $A$ is called $\textbf{ill-conditioned}$.

Let us compute the condition number of $X^\top X$.

```python
#condition number of XTX
print(np.linalg.cond(XTX))
```

    2.4610368283570592e+20

$\textbf{Adding $\ell_2$ regularizer to improve the conditioning of the matrix}$:

We shall motivate the use of $\ell_2$ regularizer to improve the conditioning of the matrix.

Instead of minimizing the original OLS objective
$
\begin{align}
L_{OLS}(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2.
\end{align}
$

we shall now minimize the OLS objective added with a scaled $\ell_2$ regularizer.

The $\ell_2$ regularizer in $\beta \in {\mathbb{R}}^{d+1}$ is defined as the squared $\ell_2$ norm of $\beta$:

$
\begin{align}
\|\beta\|_2^2 = \sum_{i=1}^{d+1} \beta_i^2.
\end{align}
$

Hence we shall now solve:
$
\begin{align}
L_{\text{ridge}}(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2 + \lambda \sum_{i=1}^{d+1} \beta_i^2.
\end{align}
$

The above objective function is called $\textbf{ridge}$ regression objective. $\lambda>0$ is a regularization hyperparameter.

Now we can write the objective function as:

$
L_{\text{ridge}}(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2 + \lambda \|\beta\|_2^2.
$

To solve

$
\min_\beta L_{\text{ridge}}(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2 + \lambda \|\beta\|_2^2,
$
we find the gradient with respect to $\beta$ and equate to zero.

Thus we get:

$
\begin{align}
\nabla_\beta L_{\text{ridge}}(\beta) &= \mathbf{0} \\
\implies -\mathbf{X}^\top \mathbf{y} + \mathbf{X}^\top\mathbf{X} \beta + \lambda I\beta &= \mathbf{0} \\
\implies \beta &= (\mathbf{X}^\top\mathbf{X}+\lambda I)^{-1} \mathbf{X}^\top \mathbf{y}.
\end{align}
$

Note that the closed form expression for $\beta$ is always valid since $(\mathbf{X}^\top\mathbf{X}+\lambda I)$ is invertible.

With $\lambda \in \{10^{-5},10^{-4}, 10^{-3},10^{-2},0.1, 1, 10, 100\}$ let us check the condition number of $(X^\top X+\lambda I)$.

```python
lambdas = np.array([1e-5,1e-4,1e-3,1e-2,1e-1,1,10,100])
I = np.identity(XTX.shape[0])

for lambda_ in lambdas:
  print('lambda:',lambda_,'cond(XTX+lambda I):', np.linalg.cond(XTX+lambda_*I))
```

    lambda: 1e-05 cond(XTX+lambda I): 1501310292340303.5
    lambda: 0.0001 cond(XTX+lambda I): 150126912538287.34
    lambda: 0.001 cond(XTX+lambda I): 15012672255523.154
    lambda: 0.01 cond(XTX+lambda I): 1501267250531.3953
    lambda: 0.1 cond(XTX+lambda I): 150126721553.5753
    lambda: 1.0 cond(XTX+lambda I): 15012672102.428282
    lambda: 10.0 cond(XTX+lambda I): 1501267211.0623977
    lambda: 100.0 cond(XTX+lambda I): 150126722.00526762

With $\lambda \in \{10^{-5},10^{-4}, 10^{-3},10^{-2},0.1, 1, 10, 100\}$ , let us now solve for $\beta$ and check the solutions.

```python
import scipy.linalg
#Now we are ready for finding beta
#print('X shape:',X.shape,'y shape:',y.shape)

betas = []
for lambda_ in lambdas:
  beta =scipy.linalg.solve(XTX+lambda_*I,Xy)
  print('lambda:',lambda_)
  print('beta:',beta)
  print('***************')
  betas.append(beta)
```

    lambda: 1e-05
    beta: [[ 7.52538381e-01]
     [-2.25222818e+00]
     [ 3.30238546e+00]
     [ 3.10642807e-03]
     [ 7.22976204e+02]
     [-2.17465207e+03]
     [-3.82040729e+02]
     [ 3.04818986e-01]
     [ 1.13266371e+00]
     [-8.27966389e-01]
     [ 1.43271639e+03]]
    ***************
    lambda: 0.0001
    beta: [[ 7.53818742e-01]
     [-2.24369420e+00]
     [ 3.16700342e+00]
     [ 3.10769932e-03]
     [ 7.20992498e+02]
     [-2.16947737e+03]
     [-3.81159028e+02]
     [ 3.04758743e-01]
     [ 1.13215169e+00]
     [-8.27378531e-01]
     [ 1.42999211e+03]]
    ***************
    lambda: 0.001
    beta: [[ 7.66223755e-01]
     [-2.16014295e+00]
     [ 1.84741577e+00]
     [ 3.12018008e-03]
     [ 7.01649939e+02]
     [-2.11902946e+03]
     [-3.72563812e+02]
     [ 3.04604230e-01]
     [ 1.12668192e+00]
     [-8.22074185e-01]
     [ 1.40339039e+03]]
    ***************
    lambda: 0.01
    beta: [[ 8.58860080e-01]
     [-1.46856675e+00]
     [-8.62792126e+00]
     [ 3.22604166e-03]
     [ 5.47573013e+02]
     [-1.71783994e+03]
     [-3.04249073e+02]
     [ 3.02345843e-01]
     [ 1.08073317e+00]
     [-7.78387362e-01]
     [ 1.18855672e+03]]
    ***************
    lambda: 0.1
    beta: [[ 9.69096121e-01]
     [ 1.06214566e+00]
     [-3.77107029e+01]
     [ 3.64746664e-03]
     [ 1.11307749e+02]
     [-5.89083650e+02]
     [-1.13855402e+02]
     [ 2.74163395e-01]
     [ 8.96850101e-01]
     [-6.22686686e-01]
     [ 5.14767044e+02]]
    ***************
    lambda: 1.0
    beta: [[ 3.70477677e-01]
     [ 3.35668167e+00]
     [-4.94049677e+01]
     [ 3.68878106e-03]
     [-4.96526290e+01]
     [-8.75003893e+01]
     [-4.47139202e+01]
     [ 2.06809466e-01]
     [ 6.61417794e-01]
     [-4.54608325e-01]
     [ 9.34613495e+01]]
    ***************
    lambda: 10.0
    beta: [[-8.27448708e-01]
     [ 3.65810207e+00]
     [-4.06632787e+01]
     [ 1.65815556e-03]
     [-2.20470430e+01]
     [-1.09706276e+01]
     [-2.99020124e+01]
     [ 6.93082632e-02]
     [ 1.87803313e-01]
     [-1.18495049e-01]
     [ 1.20070190e+01]]
    ***************
    lambda: 100.0
    beta: [[-1.92987054e+00]
     [ 2.78279481e+00]
     [-1.30766014e+01]
     [-1.28146618e-03]
     [-3.78653268e+00]
     [-1.37030952e+00]
     [-5.87413916e+00]
     [-1.90761657e-01]
     [-6.64761608e-01]
     [ 4.73999950e-01]
     [ 1.36538093e+00]]
    ***************

$\Large{\text{Residual Plot}}$

Sometimes it would be useful to plot the error (or) $\textbf{residual}$ $e^i$ versus the fitted values $\hat{y}^i=\beta_0 + \sum_{j=1}^{d} \beta_j x^i_j$.

```python
beta_idx = 0
for lambda_ in lambdas:
  beta = betas[beta_idx]
  beta_idx += 1
  xlabel = 'TOC lambda='+str(lambda_)
  e_i_residuals_list = []

  for i in range(n):
    x_i = X[i,:] #access i-th row of X
    y_i = y[i] #access i-th row of y

    y_pred_i = np.dot(x_i,beta) #compute the prediction obtained using the regression coefficients
    e_i = y_i - y_pred_i #compute the difference between the actual observation y_i and prediction y_pred_i

    e_i_residuals_list.append(e_i) #append the value of e_i to the list

  #plot the residuals e_i against the actual observations y_i
  plt.scatter(airline_costs_data['TOC'], e_i_residuals_list,color='r')
  plt.title("Residual plot")
  plt.xlabel(xlabel)
  plt.ylabel("Residuals")
  plt.grid()
  plt.show()
```

Note that the residual plot helps to check the variance in the errors $e^i$.

$\Large{\text{Coefficient of multiple determination for high dimensions}}$

Note that the description of coefficient of multiple determination for high dimensions is essentially the same as that for single dimension case.

Consider the term

$
\begin{align}
\Sigma_{YY} = \sum_{i=1}^{n} (y^i - \bar{y})^2.
\end{align}
$
This term $\Sigma_{YY}$ measures the $\textbf{observed total variation of the response}$, without considering the effect of predictor variables.

When we have access to a predictor variable, then the $\textbf{unexplained variation of response variable given the predictor variable}$ is given by the sum of squared residuals:

$
\begin{align}
L = \sum_{i=1}^{n} (e^i)^2.
\end{align}
$

Define the term:

$
\begin{align}
\text{SSD} = \Sigma_{YY}- L
\end{align}
$

as the difference between observed total variation of the response and the unexplained variation given the predictor. Hence this can be considered as $\textbf{variability explained by the regression on the predictor variable}$.

Assuming $\Sigma_{YY}\neq 0$, consider:

$
\begin{align}
\frac{\text{SSD}}{\Sigma_{YY}} = 1- \frac{L}{\Sigma_{YY}}.
\end{align}
$
Here the term $\frac{\text{SSD}}{\Sigma_{YY}}$ denotes the proportion of observed variability in the response variable which is explained by the regression on the predictor variable. From the equality, this proportion of observed variability given the predictor is simply $1$ minus the proportion of unexplained variability.

This term $\frac{\text{SSD}}{\Sigma_{YY}}$ denoting the proportion of observed variability which is explained by the predictor variable has a special name called $\textbf{coefficient of multiple determination}$ and is denoted by $R^2$.

It is known that $0\leq R^2 \leq 1$.

Let us compute $R^2$.

```python
#We shall now compute R^2 for airline costs data

#first we shall compute y_bar
y_bar = 0
for i in range(n): #access each row from the data set
  y_i = float(airline_costs_data.iat[i,1]) #access element at TOC column
  y_bar += y_i

y_bar /= n
print('y_bar',y_bar)

#Next we shall compute sigma_yy, which is the variance of y
sigma_yy = 0
for i in range(n): #access each row from the data set
  y_i = float(airline_costs_data.iat[i,1]) #access element at TOC column
  sigma_yy += (y_i-y_bar)**2

print('sigma_yy:', sigma_yy)

#now we can compute sum of squared residuals
beta_idx = 0
for lambda_ in lambdas:
  beta = betas[beta_idx]
  beta_idx+=1
  sum_sq_residuals = 0
  for i in range(n): #access each row from the data set
    x_i = X[i,:] #access i-th row of X
    y_i = y[i] #access i-th row of y

    y_pred_i = np.dot(x_i,beta) #compute the prediction obtained using the regression coefficients
    e_i = y_i - y_pred_i #compute the difference between the actual observation y_i and prediction y_pred_i

    sum_sq_residuals += (e_i)**2

  print('lambda:',lambda_)
  print('sum of squared residuals:', sum_sq_residuals)

  #Then we will compute the R^2 quantity
  R_sq = 1-sum_sq_residuals/sigma_yy
  print('R^2:',R_sq)
  print('*********************')
```

    y_bar 113.50645161290323
    sigma_yy: 610938.3987096774
    lambda: 1e-05
    sum of squared residuals: [48091.04515484]
    R^2: [0.92128332]
    *********************
    lambda: 0.0001
    sum of squared residuals: [48092.90331628]
    R^2: [0.92128027]
    *********************
    lambda: 0.001
    sum of squared residuals: [48270.38435013]
    R^2: [0.92098977]
    *********************
    lambda: 0.01
    sum of squared residuals: [60051.44363136]
    R^2: [0.90170622]
    *********************
    lambda: 0.1
    sum of squared residuals: [198507.15296848]
    R^2: [0.67507828]
    *********************
    lambda: 1.0
    sum of squared residuals: [330055.62735503]
    R^2: [0.45975629]
    *********************
    lambda: 10.0
    sum of squared residuals: [379056.04870797]
    R^2: [0.37955111]
    *********************
    lambda: 100.0
    sum of squared residuals: [463653.0991366]
    R^2: [0.24108044]
    *********************

$\textbf{Adding $\ell_1$ regularizer to find sparse solutions}$:

Instead of $\ell_2$ regularizer, we can use $\ell_1$ regularizer of the parameters $\beta$. $\ell_1$ regularizer is useful to obtain sparse solutions, where most of the components of $\beta$ are zero and only a few weights are non-zero.

The $\ell_1$ regularizer with $\beta$ is given as the $\ell_1$ norm of $\beta$:

$
\begin{align}
\|\beta\|_1 = \sum_{i=1}^{d+1} |\beta_i|.
\end{align}
$

Recall the original OLS objective:
$
\begin{align}
L_{OLS}(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2.
\end{align}
$

and the objective for ridge regression where we minimize the OLS objective added with a scaled $\ell_2$ regularizer:

$
\begin{align}
L_{\text{ridge}}(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2 + \lambda \sum_{i=1}^{d+1} \beta_i^2.
\end{align}
$

Similar to ridge regression, we can use the scaled $\ell_1$ regularizer along with the OLS objective to get the following objective:

$
\begin{align}
L_{\text{lasso}}(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2 + \alpha \sum_{i=1}^{d+1} |\beta_i|.
\end{align}
$

The above objective function is called $\textbf{lasso}$ regression objective. $\alpha>0$ is a regularization hyperparameter.

Unlike ridge regression, lasso cannot be solved by a simple differentiation type idea.

Instead we need to use an iterative algorithm to solve the lasso optimization problem.

```python
from sklearn import linear_model
from sklearn.metrics import r2_score
alphas = np.array([1e-5, 1e-4, 1e-3,1e-2,0.1,1,10,100])

for alpha_ in alphas:
  clf = linear_model.Lasso(alpha=alpha_,max_iter=100000)
  clf.fit(X, y)
  print('alpha:',alpha_)
  print(clf.coef_)
  print(clf.intercept_)
  y_pred = clf.predict(X)
  R2=r2_score(y,y_pred)
  print('R2 score:',R2)
  print('************************')
```

    /usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 229.85867555970617, tolerance: 61.09383987096774
      positive)
    /usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 230.7560244394408, tolerance: 61.09383987096774
      positive)

    alpha: 1e-05
    [ 7.52401775e-01 -2.25317646e+00  3.31721242e+00  3.10628549e-03
      7.23193567e+02 -2.17521925e+03 -3.82137254e+02 -2.78886609e-01
      1.71643369e+00 -2.44318836e-01  0.00000000e+00]
    [1433.01645838]
    R2 score: 0.92128334639926
    ************************
    alpha: 0.0001
    [ 7.52456817e-01 -2.25315853e+00  3.31491825e+00  3.10627593e-03
      7.23160998e+02 -2.17513575e+03 -3.82121985e+02 -2.78816556e-01
      1.71636988e+00 -2.44382341e-01  0.00000000e+00]
    [1432.9861773]
    R2 score: 0.9212833456535767
    ************************

    /usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 239.72510696386598, tolerance: 61.09383987096774
      positive)
    /usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 329.0326550937971, tolerance: 61.09383987096774
      positive)

    alpha: 0.001
    [ 7.53007233e-01 -2.25297921e+00  3.29197652e+00  3.10618035e-03
      7.22835308e+02 -2.17430073e+03 -3.81969290e+02 -2.78116027e-01
      1.71573174e+00 -2.45017392e-01  0.00000000e+00]
    [1432.68336649]
    R2 score: 0.9212832710852403
    ************************
    alpha: 0.01
    [ 7.58511392e-01 -2.25118599e+00  3.06255921e+00  3.10522453e-03
      7.19578406e+02 -2.16595052e+03 -3.80442344e+02 -2.71110737e-01
      1.70935036e+00 -2.51367904e-01  0.00000000e+00]
    [1429.65525834]
    R2 score: 0.9212758142516358
    ************************
    alpha: 0.1
    [ 8.13552981e-01 -2.23325387e+00  7.68386149e-01  3.09566627e-03
      6.87009388e+02 -2.08244846e+03 -3.65172884e+02 -2.01057840e-01
      1.64553651e+00 -3.14873020e-01  0.00000000e+00]
    [1399.37417686]
    R2 score: 0.9205301308911755
    ************************
    alpha: 1.0
    [ 1.33203160e+00 -2.10277243e+00 -1.93345383e+01  2.92881337e-03
      3.71126241e+02 -1.28114129e+03 -2.17165267e+02 -0.00000000e+00
      1.43815928e+00 -4.36373247e-01  0.00000000e+00]
    [1104.46533157]
    R2 score: 0.850846970213033
    ************************
    alpha: 10.0
    [ 8.82426626e-01 -3.03164456e+00 -4.68105718e+01  1.72330050e-04
     -0.00000000e+00 -0.00000000e+00 -2.23593400e+00 -0.00000000e+00
      1.11505802e+00 -1.67225015e-01  0.00000000e+00]
    [805.86775277]
    R2 score: 0.502977069366046
    ************************
    alpha: 100.0
    [-0.00000000e+00 -3.49897491e+00 -0.00000000e+00 -1.99036650e-03
     -0.00000000e+00 -0.00000000e+00 -0.00000000e+00  0.00000000e+00
     -2.04655434e-01  1.99564915e-01  0.00000000e+00]
    [671.83094337]
    R2 score: 0.3271408138037739
    ************************

    /usr/local/lib/python3.7/dist-packages/sklearn/linear_model/_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 1183.7711423726214, tolerance: 61.09383987096774
      positive)

$\Large{\text{Exercise:}}$

Perform ridge regression and lasso with the following data sets:
1. Data: http://users.stat.ufl.edu/~winner/data/bridge_risk.dat
   Description: http://users.stat.ufl.edu/~winner/data/bridge_risk.txt
2. Data: http://users.stat.ufl.edu/~winner/data/lpga2009.dat
   Description: http://users.stat.ufl.edu/~winner/data/lpga2009.txt
Choose $\lambda$ for ridge regression and $\alpha$ for lasso from the set $\{10^{-5},10^{-4},10^{-3},10^{-2},10^{-1},1,10,100\}$.

Choose the best $\lambda$ and $\alpha$ based on the train set $R^2$ value.

Report the residual plots and $R^2$ value for each data set for the best hyperparameters. Explain your observations.

$\Large{\text{References:}}$

1.   Olive J. David, $\textit{Linear Regression}$. Springer International Publishing,  2017.
2.   Sanford Weisberg. $\textit{Applied Linear Regression}$. John Wiley $\&$ Sons, Inc., Hoboken, New Jersey, 4-th Edition, 2014.

```python

```

Linear Regression/Linear Regress with Regularization.ipynb


[@ggruszczynski](https://github.com/ggruszczynski)

You should have completed previous tutorial, [diffusion_part1_and_convolution](./diffusion_part1_and_convolution.ipynb) before continuing.

Diffusion part 3: with spatially variable diffusivity in 1-D
-----
***

# Balance of Enthalpy

The simplified balance of Enthalpy $H=\rho c_p T$ is known as:
\begin{align}
\int \frac{\partial }{\partial t} (\rho c_p T ) dV +  \oint  ( \rho c_p T \boldsymbol{u} ) \cdot \boldsymbol{n} dS &= \oint \boldsymbol{n} \cdot \boldsymbol{q} dS + \int \dot{q} dV \\
\boldsymbol{q} &= k \nabla T
\end{align}
The heat flux $ \boldsymbol{q} $ is related to the thermal conductivity coefficient $k [ W/mK] $.

Applying the GGO theorem, we obtain:

$$
\frac{\partial }{\partial t} (\rho c_p T ) + \nabla \cdot (\boldsymbol{u} \rho c_p T ) = \nabla \cdot (k \nabla T)  + \dot{q}
$$

## Conservative and non-conservative scheme

Consider a 1D, unsteady heat transfer equation with variable conduction coefficient.
For simplicity, assume that the heat capacity $ \rho c_p = 1 $.

The equation be expressed in two mathematically equivalent forms:
$$
\frac{\partial u}{\partial t}=\frac{\partial}{\partial x}\left(k(x) \frac{\partial u}{\partial x}\right) \\
$$
or
$$
\frac{\partial u}{\partial t}=\frac{\partial u}{\partial x} \frac{\partial k}{\partial x}+k(x) \frac{\partial^{2} u}{\partial x^{2}}
$$

Both of these continous forms can be discretized.

First form leads to the *conservative scheme*

$$
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\frac{1}{\Delta x^{2}}\left(k_{i+\frac{1}{2}}\left(u_{i+1}^{n}-u_{i}^{n}\right)-k_{i-\frac{1}{2}}\left(u_{i}^{n}-u_{i-1}^{n}\right)\right)
$$

while the second form results in a *non-conservative scheme*

$$
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\frac{1}{\Delta x^{2}}\left(\frac{\left(k_{i+1}-k_{i-1}\right)}{2} \frac{\left(u_{i+1}^{n}-u_{i-1}^{n}\right)}{2}+k_{i}\left(u_{i+1}^{n}+u_{i-1}^{n}-2 u_{i}^{n}\right)\right)
$$

## Task

* Implement both the *conservative* and the *non-conservative* schemes.

* Impose the following BC:

```
u[int(nx/4)] = 10.
u[-int(nx/4)] = 1.

u[0] = 5.
u[-1] = 5.
```

* Run the simulation for different $k$ and compare results. Which physical quantity may become *not conserved* ?

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
# variable conductivity

nx = 128
domain_length = 64
dx = domain_length / (nx-1)
xspace = np.linspace(0, domain_length, nx)

u_IC = 5.*np.ones(nx)                    # numpy function ones()
u_IC[int((nx-1)/4):int(nx/2 + 1)] = 10   # setting u = 2 between 0.5 and 1 as per our I.C.s

k = 1.*np.ones(nx)                      # be aware that dt = sigma * dx**2 / nu
k[:int(nx/2)] = 0.1                     # try with 0.1

sigma = .2              # sigma is a parameter, we'll learn more about it later
dt = sigma * dx**2 / max(k)              #dt is defined using sigma
nt = 10000  # the number of timesteps we want to calculate

def calc_diffusion_variable_conductivity(IC,nx,nt,k,dt):
    u = IC.copy()
    un = IC.copy() #our placeholder array, un, to advance the solution in time

    for n in range(nt):  #iterate through time
        #impose BC
        u[int(nx/4)] = 10.
        u[-int(nx/4)] = 1.

        u[0] = 5.
        u[-1] = 5.

        un = u.copy() ##copy the existing values of u into un
        for i in range(1, nx - 1):
            # this is obviously bad : du/dt = k (d^2)u/(dx)^2
            # u[i] = un[i] + k[i] * dt / dx**2 * (un[i+1] - 2 * un[i] + un[i-1])

            # non conservative scheme
            # u[i] = un[i] +  dt / dx**2 * ((k[i+1]-k[i-1])*(un[i+1]-un[i-1])/4.  + k[i]*(un[i+1] - 2 * un[i] + un[i-1]))

            # conservative scheme
            kf = (k[i+1]+k[i])/2
            kb = (k[i]+k[i-1])/2
            u[i] = un[i] +  dt / dx**2 * (kf*(un[i+1]-un[i]) - kb*(un[i] - un[i-1]))

    return u

u_FD = calc_diffusion_variable_conductivity(u_IC,nx,nt,k,dt)
plt.plot(xspace, u_FD)
```


```python
import numpy as np
import matplotlib.pyplot as plt

import sys
sys.path.append('../../pyutils')
import metrics
import utils
```

# Projection Pursuit Regression

The PPR model has the form:

$$f(x) = \sum_{m=1}^M g_m(w_m^Tx)$$

with $g_m: \mathbb{R} \to \mathbb{R}$ unspecified functions, $w_m$ and $x \in \mathbb{R}^p$.

For $M$ large enough, and special kinds of $g_m$, the model can approximate any continuous functions, it's called a universal approximator.

We fit the model by finding the $g_m$ and $w_m$ that minimizes the following criterion:
$$\sum_{i=1}^N \left( y_i - \sum_{m-1}^M g_m(w_m^Tx_i) \right) ^2$$

If the $g_m$ are fixed, we can solve it one $g$ at a time, using a quasi-newton method:

$$\sum_{i=1}^N (y_i - g(w^Tx_i))^2 \approx \sum_{i=1}^N g'(w^T_\text{old}x_i)^2 \left( (w_\text{old}^Tx_i + \frac{y_i - g(w^T_{old}x_i)}{g'(w^T_\text{old}x_i)}) - w^T x_i \right)^2$$

We can minimize the right-hand side using a weighted least squares to find $w$, the new weights estimates.
After several iterations, $w$ converges to the weights that minimizes the criterion.
We can then go to the next $g$ and fit $w$ with the residual.

PPR Algorithm:

1. Set $f_0(x) = 0$
2. For $m=1$ to $M$:

    - Initialize $w_m$ randomly
    - Let the residual $r_i = y_i - f_{m-1}(x_i)$
    - Let the target $\hat{y}_i = w^T_\text{old}x_i \frac{y_i - g(w^T_\text{old} x_i)}{g'(w^T_\text{old} x_i)}$
    - Let the weights $v_i = g'(w^T_\text{old} x_i)^2$
    - Solve the WLS of $x_i$ onto $\hat{y}_i$ with weights $v_i$ iteratively until convergence
    - Set $f_m(x) = f_{m-1}(x) + g_m(w_m^Tx)$

3. Output $f(x) = f_M(x)$

# Neural Networks

We focused on vanilla net, with only one hidden layer. The network has $p$ inputs, $K$ outputs, and the hidden layer size is $M$.
$$Z_m = \sigma (\alpha_{0m} + \alpha^T_m x), \space m=1,\text{...},M$$
$$T_k = \beta_{0k} + \beta^T_Z x), \space k=1,\text{...},K$$
$$f_k(X) = g_k(T), \space k=1,\text{...},K$$

Usually, the activation function $\sigma$ is the sigmoid: $\sigma(z) = \frac{1}{1 + \exp(-z)}$

$g_k$ is a final transformation of output vectors. For regression, it's usually the identity. For classification, it's usually the softmax function:
$$g_k(T) = \frac{e^{T_k}}{\sum_{l=1}^K e^{T_l}}$$

We can think of $Z_m$ as the basis expansion (non-linear) of the original inputs $X$. The network is then a simple linear or logistic model with this transformation as input.

If $\sigma$ is a linear function, the model reduces to a simple linear or logistic model. A non-linear $\sigma$ greaty enlarges the class of linear models.

# Fitting Neural Networks

We define $\theta$ as the whole set of weights of our network, containing $\alpha_{0m}, \alpha_m, \beta_{0k}, \beta_k$, so $M(p+1) + K(M + 1)$ weights.

For regression we usually use the sum-of-squared errors:
$$R(\theta) = \sum_{k=1}^K\sum_{i=1}^N (y_{ik} - f_k(x_i))^2$$

For $K$-classes classification we usually use the cross-entropy (deviance):

$$R(\theta) = -\sum_{k=1}^K\sum_{i=1}^N y_{ik} \log f_k(x_i)$$

The model is fit using gradient descent. Backprogation makes use of the chain rule to compute the gradients of every weights:

For sum of squares, let's define:
$$R(\theta) = \sum_{i=1}^N R_i$$
$$\text{with } R_i = \sum_{k=1}^K (y_{ik} - f_k(x_i))^2$$

$$\frac{\partial R_i}{\partial \beta_{km}} = -2(y_{ik} - f_k(x_i)) g'_k(\beta^T_kz_i) z_{mi}$$
$$\frac{\partial R_i}{\partial \alpha_{ml}} = -2(y_{ik} - f_k(x_i)) g'_k(\beta^T_kz_i) \beta_{km} \sigma'(\alpha_m^Tx_i) x_{il}$$

A gradient descent update has the form:
$$\beta_{km} \leftarrow \beta_{km} - \gamma \sum_{i=1}^N \frac{\partial R_i}{\partial \beta_{km}}$$
$$\alpha_{ml} \leftarrow \alpha_{ml} - \gamma \sum_{i=1}^N \frac{\partial R_i}{\partial \alpha_{ml}}$$

with $\gamma$ the learning rate.

The backprogagation is a 2-pass algorithm: A forward pass that computes the output from the input, and the backward pass, that goes backward to compute the weights gradients.

These updates are in batch learning, getting all datasets at once. But it can also be done with a few or only one example between each update.
An epoch is a sweep through the entiere training set.
Usuaully, the learning rate $\gamma$ should decrease over time.

```python
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_prime(x):
    return sigmoid(x) * (1 - sigmoid(x))

class VanillaNetReg:

    def __init__(self, ninputs, nhidden, noutputs):
        self.p = ninputs
        self.M = nhidden
        self.K = noutputs

        self.w0 = np.random.randn(ninputs, nhidden)
        self.b0 = np.random.randn(nhidden)
        self.w1 = np.random.randn(nhidden, noutputs)
        self.b1 = np.random.randn(noutputs)

    def fit(self, X, y, nepochs, batch_size, lr):

        for epoch in range(nepochs):

            p = np.random.permutation(len(X))
            X, y = X[p], y[p]
            for ki in range(0, len(X_train), batch_size):
                Xb = X[ki:ki+batch_size]
                yb = y[ki:ki+batch_size]
                self.backprop(Xb, yb, lr)

            if epoch % 10 == 0:
                print('Epoch {}: loss = {}'.format(epoch + 1,
                                                  self.loss(X, y)/len(X)))

    def backprop(self, X, y, lr):
        z0 = X @ self.w0 + self.b0
        y0 = sigmoid(z0)
        z1 = y0 @ self.w1 + self.b1
        y1 = z1

        dy1 = 2 * (y1 - y)
        dz1 = dy1
        dw1 = y0.T @ dz1
        db1 = np.sum(dz1, axis=0)
        dy0 = dz1 @ self.w1.T
        dz0 = z0 * sigmoid_prime(dy0)
        dw0 = X.T @ dz0
        db0 = np.sum(dz0, axis=0)
        dx = dz0 @ self.w0.T

        self.w0 -= lr * dw0
        self.b0 -= lr * db0
        self.w1 -= lr * dw1
        self.b1 -= lr * db1

    def forward(self, X):
        z0 = X @ self.w0 + self.b0
        y0 = sigmoid(z0)
        z1 = y0 @ self.w1 + self.b1
        y1 = z1
        return y1

    def loss(self, X, y):
        preds = self.forward(X)
        return np.sum((y - preds)**2)

X, y = load_boston().data, load_boston().target
X = X.astype(np.float32)
X = X / np.std(X, axis=0)
y = y.reshape(-1, 1).astype(np.float32)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                   test_size=0.2,
                                                   random_state=15)

model = VanillaNetReg(ninputs=X.shape[1], nhidden=40, noutputs=1)
model.fit(X_train, y_train, nepochs=200, batch_size=32, lr=1e-3)
```

    Epoch 1: loss = 61.053075987293404
    Epoch 11: loss = 29.972021464020678
    Epoch 21: loss = 27.859139470811176
    Epoch 31: loss = 27.167323663051786
    Epoch 41: loss = 28.227063298712316
    Epoch 51: loss = 25.35478667463649
    Epoch 61: loss = 26.29497722419557
    Epoch 71: loss = 25.936603803175114
    Epoch 81: loss = 26.741483662628262
    Epoch 91: loss = 26.560964940265357
    Epoch 101: loss = 26.0333711289981
    Epoch 111: loss = 27.719647569882234

    /home/aiw/rep/ml-notebooks/env/lib/python3.6/site-packages/ipykernel_launcher.py:5: RuntimeWarning: overflow encountered in exp
      """

    Epoch 121: loss = 26.600198187313705
    Epoch 131: loss = 25.675793773929158
    Epoch 141: loss = 26.385492964196903
    Epoch 151: loss = 25.92974859137518
    Epoch 161: loss = 27.144747910748084
    Epoch 171: loss = 25.843922850252664
    Epoch 181: loss = 26.78885721575312
    Epoch 191: loss = 27.985616514137263

# Some Issues in training neural nets

## Starting values

If weights are near $0$, the sigmoid is roughly linear, and the model is roughly linear.
Exact $0$ leads to $0$ derivative and the network doesn't learn.
To high weights leads to poor solutions.
Usually, it's better to start with near $0$ weights.

## Overfitting

Neural networks often overfit because they have too many weights.
Early stopping is a technique to reduce overfitting. Whe stop training before reaching a minimum, when the error on a validation set starts increasing.

Another solution is weight decay, we add a penatly $\lambda J(\theta)$ to $R(\theta)$:
$$J(\theta) = \sum_{k,m} \beta_{km}^2 + \sum_{m,l} \alpha_{ml}^2$$
with $\lambda \geq 0$ hyperparameter. It shrink the weights towards $0$.

Another penaltty is weight elimination, that shrink smaller weights more:
$$J(\theta) = \sum_{k,m} \frac{\beta_{km}^2}{1 + \beta_{km}^2} + \sum_{m,l} \frac{\alpha_{ml}^2}{1+\alpha_{ml}^2}$$

## Scaling of the inputs

The scaling of the inputs also determines the effective scaling of the weights. It's best to standardize all inputs to mean $0$ and standard deviation $1$.

## Number of hidden units and layers

It's better to have too many hidden units than too few. With too few, the model might not be complex enought to learn the correct representation.
With too many, extra weights can be shrunk towards $0$ with the proper regularization.

The number of layers is chosen by background knowledge and experimentation. Multple hidden layers allows construction of hierarchical features.

## Multiple Minima

$R(\theta)$ as many local minima, and the final solution is quite dependant of the starting weights.
One can try several starting configurations and choose the one with the lowest error, or average over several networks, or use another approach such as bagging.

# Bayesian Neural Net

[Bayesian Methods for Neural Networks](https://www.microsoft.com/en-us/research/wp-content/uploads/1995/01/NCRG_95_009.pdf)

Let consider a feedforward neural network that maps an input vector $x \in \mathbb{R}^p$ to an output value $y \in \mathbb{R}$, using a weights vector $w \in \mathbb{R}^W$.
The observed dataset $D$ consist of $N$ input vector $x_i$ and corresponding target $t_i$.

## Prediction

We can find the posterior distribution of the weights using Bayes theorem:
$$p(w|D) = \frac{p(D|w)p(w)}{p(D)}$$

The conditional distribution $p(D|w)$ is called the likelihood. The conventional approach is to find $w^*$ maximizing the likelihood function.
$p(w)$ is a prior distribution over the weights.

In order to evaluate the posterior $p(w|D)$, we need expressions for both the prior distribution $p(w)$ and the likelihood function $p(D|w)$.

One simple choice or a prior is a multivariate normal distribution with mean $\vec{0}$ and fixed variance $\alpha^{-1}$:

$$p(w) = \frac{1}{Z_W(\alpha)} \exp (-\frac{\alpha}{2}||w||^2)$$
$$\text{with } Z_W(\alpha) = \left( \frac{2 \pi}{\alpha} \right) ^{W/2}$$

For the likelihhod function, let's suppose our model follows a Gaussian distribution, with mean the output of the network, and fixed variance $\beta^{-1}$:
$$p(t|x,w) = \left( \frac{\beta}{2 \pi} \right) ^{1/2} \exp \left( \frac{\beta}{2} (y(x;w) - t)^2 \right)$$

The likelihood function is then:
$$
\begin{equation}
\begin{split}
p(D|w)  & = \prod_{n=1}^N p(t^n|x^n,w) \\
& = \frac{1}{Z_D(\beta)} \exp \left( -\frac{\beta}{2} \sum_{i=1}^N (y(x_i;w) - t_i)^2 \right)
\end{split}
\end{equation}
$$

$$\text{with } Z_D(\beta) = \left( \frac{2\pi}{\beta} \right) ^{N/2}$$

$\alpha$ and $\beta$ are hyperpameters, let's suppose for now that they are know, fixed values.

We can make prediction for a new $x$ by integrating over the weights:
$$p(t|x,D) = \int p(t|x,w)p(w|D)dw$$

If the posterior $p(w|D)$ is sharply peaked around $w_{MP}$ (center, maximum value of the distribution), then we can approximate is using:

$$p(t|x,D) \approx p(t|x,w_{MP})$$

Predictions are made using the neural network, with weights $w_{MP}$. We need to find the $w$ that maximizes the posterior $p(w|D)$.
Instead of maximizing the posterior probability, let's minimize the negative logarithm of the posterior probability. For this particular prior distribution and likelihood function, we have to minize the following criterion:
$$E(w) = \frac{\beta}{2} \sum_{i=1}^N (y(x_i:w) - t_i))^2 + \frac{\alpha}{2} ||w||^2$$
Up to a constant factor, this is the same as minimizing the usual sum-of-squares error with $L2$ regularization.

## Confidence interval

We can use the Bayesian Neural Network to get a confidence interval around the predictions. We can make a Gaussian approximation of the posterior $p(w|D)$.
Let's estimate $E(w)$ using a second-order taylor expansion:
$$E(w) = E(w_{MP}) + \frac{1}{2} (w - w_{MP})^T A (w - w_{MP})$$

$A$ is the hessian matrix of $E$ with respect to the weigthts, calculated at $w=w_{MP}$.

Let's approximate the network $y(x;w)$ by a linar expansion:
$$y(x;w) = y(x;w_{MP}) + g^T\Delta w$$
with $\Delta w = w - w_{MP}$ and $g = \nabla_w y(x; w_{MP})$

With these approximations, the prediction becames Gaussian and can be evaluated:
$$p(t|x,D) = \frac{1}{(2 \pi \sigma_t^2)^{1/2}} \exp \left( - \frac{(t - y_{MP})^2}{2 \sigma_t^2} \right)$$

The distribution has mean $y_{MP} = y(x;w_{MP})$ and variance $\sigma_t^2$ given by:
$$\sigma_t^2 = \frac{1}{\beta} + g^TA^{-1}g$$

We can use this distribution to estimate confidence intervals of new predictions.

## Hyperparameters

We need to chose the correct values for $\alpha$ and $\beta$.
The posterior is given by:
$$p(w|D) = \int \int p(w|\alpha,\beta,D)p(\alpha,\beta|D)d\alpha d\beta$$

Let's suppose the posterior $p(\alpha,\beta|D)$ is sharply peaked around their maximum $\alpha_{MP}$ and $\beta_{MP}$. Then:
$$p(w|D) \approx p(w|\alpha_{MP}, \beta_{MP},D)$$

We need to find the hyperparemets values that maximize the probability of the posterior $p(\alpha, \beta|D)$:

$$p(\alpha, \beta|D) = \frac{p(D|\alpha, \beta) p(\alpha,\beta)}{p(D)}$$

We need to choose a prior $p(\alpha,\beta)$. Such a prior on hyperparameters is called a hyperprior.
We chose an non-informative prior, because whe have no idea of what values they could be. It gives equal weights to all possible values.
The maximum posterior value is found my maximizing the likelihood term, $p(D|\alpha,\beta)$, also called the evidence. It can be rewritten as:
$$p(D|\alpha,\beta) = \int p(D|w,\beta) p(w|\alpha)dw$$

It simplifies to:
$$p(D|\alpha,\beta) = \frac{1}{Z_D(\beta)} \frac{1}{Z_W(\alpha)} \int \exp (-E(w)) dw$$

Using the taylor expension over $E(w)$ as above, this distribution becames tractable, and the resulting expression can be maximized with respect to $\alpha$ and $\beta$.
We finally get expressions for $\alpha_{MP}$ and $\beta_{MP}$ that can be used in the rest of the calculations.


Computational Seismology

Finite Differences Method - Acoustic Waves in 3D

---

This notebook is part of the supplementary material
to [Computational Seismology: A Practical Introduction](https://global.oup.com/academic/product/computational-seismology-9780198717416?cc=de&lang=en&#),
Oxford University Press, 2016.

##### Authors:
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))
* Lion Krischer ([@krischer](https://github.com/krischer))
* Taufiqurrahman ([@git-taufiqurrahman](https://github.com/git-taufiqurrahman))

---

This notebook covers the following aspects:

* implementation of the 3D acoustic wave equation
* understanding the input parameters for the simulation and the plots that are generated
* understanding the concepts of stability (Courant criterion)
* modifying source and receiver locations and observing the effects on the seismograms
* allowing you to explore the finite-difference method

---

### Getting started
Before you start, make a copy of the original notebook (e.g., orig.ipynb).
Now, run all of the code. Understand the input parameters for the simulation and the plots that are generated. We may modify source and receiver locations and observe the effects on the seismograms. Finally, we can relate the time extrapolation loop with the numerical algorithm we developed.

### Stability (Courant criterion)
Now, we introduce parameter epsilon $\epsilon$ and we want to calculate the Courant criterion.
Determine numerically the stability limit of the code as accurately as possible by increasing the time step.

The Courant criterion is defined as

$$
\epsilon = c \frac{ dt}{dx} \leq 1
$$

With this information we can calculate the maximum possible and stable time step.

### Analytical Solution

In the code below we present the analytical solution for the acoustic wave equation

$$
\partial_t^2 p(x,t) \ - \ c^2 \Delta p(x,t) \ = s(x,t)
$$

assuming constant velocity c and infinite space. Note that in 1D and 2D this equation is mathematically equivalent to the problem of SH wave propagation (i.e., shear waves polarised perpendicular to the plane through source and receiver). In 3D it is (only) descriptive of pressure (sound) waves.

Analytical solution for inhomogeneous partial differential equations (i.e., with non-zero source terms) are usually developed using the concept of Green's functions $G(x, t; x_0, t_0)$. Green's functions are the solutions to the specific partial differential equations for $\delta$-function as source terms evaluated at $(x, t)$ and activated at $(x_0, t_0)$. Thus, we seek solutions to

$$
\partial_t^2 G(x,t;x_0, t_0) \ - \ c^2 \Delta G(x,t;x_0, t_0) \ = \delta (x-x_0) \delta (t-t_0)
$$

where $\Delta$ is the Laplace operator. We recall the definition of the  $\delta-$function as a generalised function with

$$
\delta(x) = \left\{
\begin{array}{ll}
	\infty &x=0 \\
	0 &x\neq 0
\end{array}
\right.
$$

and

$$
\int_{-\infty}^{\infty}\delta(x)dx\ = \ 1 \ , \   \int_{-\infty}^{\infty}f(x)\delta(x)dx\ = \ f(0) \
$$

When comparing numerical with analytical solutions the functions that - in the limit - lead to the $\delta-$function will become very important. An example is the boxcar function

$$
\delta_{bc}(x) = \left\{
\begin{array}{ll}
	1/dx &|x|\leq dx/2 \\
	0 &\text{elsewhere}
\end{array}
\right.
$$

fulfilling these properties as $dx\rightarrow0$. These functions are used to properly scale the source terms to obtain correct absolute amplitudes.

To describe analytical solutions for the acoustic wave equation we also make use of the unit step function, also known as the Heaviside function, defined as

$$
H(x) = \left\{
\begin{array}{ll}
    0 &x<0 \\
	1 &x \geq 0
\end{array}
\right.
$$

The Heaviside function is the integral of the $\delta-$function (and vice-versa the $\delta$-function is defined as the derivative of the Heaviside function). in 3D the Green's function is proportional to a $\delta-$function scaled by the distance from the source (geometrical spreading). Beware of the practical significance of this result for sound wave propagation

$$
G = \frac{1}{4 \pi c^2 r} \delta(t-r/c)
$$

$$
r = \sqrt{x^2+y^2+z^2}
$$

### Numerical Solution (Finite Differences Method)

In 3D the constant-density acoustic wave equation is given by

$$
\ddot{p}(x,y,z,t) \ = \ c(x,y,z)^2 (\partial_x^2 p(x,y,z,t) + \partial_y^2 p(x,y,z,t) + \partial_z^2 p(x,y,z,t)) \ + s(x,y,z,t)
$$

In accordance with the above developments we discretise space-time using

$$
p(x,y,z,t) \ \rightarrow \ p^n_{i,j,k} \ = \ p(n dt, i dx, j dy, k dz) \ .
$$

Using the 3-point operator for the 2nd derivatives in time leads us to the extrapolation scheme

$$
 \frac{p_{i,j,k}^{n+1} - 2 p_{i,j,k}^n + p_{i,j,k}^{n-1}}{dt^2} \ = \ c^2 ( \partial_x^2 p + \partial_y^2 p + \partial_z^2 p) \ + s_{i,j,k}^n
$$

where on the r.h.s. the space and time dependencies are implicitly assumed and the partial derivatives are approximated by

\begin{equation}
\begin{split}
\partial_x^2 p \ &= \ \frac{p_{i+1,j,k}^{n} - 2 p_{i,j,k}^n + p_{i-1,j,k}^{n}}{dx^2} \\
\partial_y^2 p \ &= \ \frac{p_{i,j+1,k}^{n} - 2 p_{i,j,k}^n + p_{i,j-1,k}^{n}}{dy^2} \\
\partial_z^2 p \ &= \ \frac{p_{i,j,k+1}^{n} - 2 p_{i,j,k}^n + p_{i,j,k-1}^{n}}{dz^2}  \ .
\end{split}
\end{equation}
Note that for a regular 3D grid $dz=dy=dx$

### Analytical and Numerical Comparisons
The code below is given with a 3-point difference operator. Compare the results from the numerical simulation with the 3-point operator with the analytical solution.

### High-order operators
Extend the code to higher order by adding the option to use a 5-point difference operator. The 5-pt weights are:
$
[-1/12, 4/3, -5/2, 4/3, -1/12] / dx^2
$.
Compare simulations of the 3-point and 5-point operators.

---

```
# Import Libraries (PLEASE RUN THIS CODE FIRST!)
# ----------------------------------------------
import numpy as np
import matplotlib
# Show Plot in The Notebook
matplotlib.use("nbagg")
import matplotlib.pyplot as plt

# Sub-plot Configuration
# ----------------------
from matplotlib import gridspec
from mpl_toolkits.axes_grid1 import make_axes_locatable

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")
```

```
# Parameter Configuration
# -----------------------

nx   = 100           # number of grid points in x-direction
ny   = nx            # number of grid points in y-direction
nz   = nx            # number of grid points in z-direction
# Note that for a regular 3D grid dz = dy = dx
dx   = 0.5           # grid point distance in x-direction
dy   = dx            # grid point distance in y-direction
dz   = dx            # grid point distance in z-direction
c0   = 250.          # wave velocity in medium (m/s)
isx  = 50            # source location in grid in x-direction
isy  = isx           # source location in grid in y-direction
isz  = isx           # source location in grid in z-direction
irx  = 57            # receiver location in grid in x-direction
iry  = irx           # receiver location in grid in y-direction
irz  = irx           # receiver location in grid in z-direction
nt   = 251           # maximum number of time steps
dt   = 0.0005        # time step

# CFL Stability Criterion
# -----------------------
eps  = c0 * dt / dx  # epsilon value

print('Stability criterion =', eps)
```

```
# Plot Source Time Function
# -------------------------

f0   = 100. # dominant frequency of the source (Hz)
t0   = 4. / f0 # source time shift

print('Source frequency =', f0, 'Hz')

# Source time function (Gaussian)
# -------------------------------
src  = np.zeros(nt + 1)
time = np.linspace(0 * dt, nt * dt, nt)
# 1st derivative of a Gaussian
src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(-1.0 * (f0 ** 2) * (time - t0) ** 2))

# Plot Source Time Function
# -------------------------
plt.ion()
fig1  = plt.figure(figsize=(12, 6))
gs1   = gridspec.GridSpec(1, 2, width_ratios=[1, 1], hspace=0.3, wspace=0.3)

# Plot Source Time Function
# -------------------------
ax1  = plt.subplot(gs1[0])
ax1.plot(time, src) # plot source time function
ax1.set_title('Source Time Function')
ax1.set_xlim(time[0], time[-1])
ax1.set_xlabel('Time (s)')
ax1.set_ylabel('Amplitude')

# Plot Source Spectrum
# --------------------
ax2  = plt.subplot(gs1[1])
spec = np.fft.fft(src) # amplitude of the source in frequency domain
freq = np.fft.fftfreq(spec.size, d = dt / 4.) # change time domain to frequency domain of the source
ax2.plot(np.abs(freq), np.abs(spec)) # plot frequency and spectrum of the source
ax2.set_xlim(0, 500) # only display frequency from 0 to 500 Hz
ax2.set_title('Source Spectrum')
ax2.set_xlabel('Frequency (Hz)')
ax2.set_ylabel('Amplitude')

ax2.yaxis.tick_right()
ax2.yaxis.set_label_position("right")

plt.show()
```

```
# Plot 2D Slices & Seismogram (PLEASE RERUN THIS CODE AGAIN AFTER SIMULATION!)
# ---------------------------------------------------------------------------

# Initialize Empty Pressure
# -------------------------
p    = np.zeros((nx, ny, nz)) # p at time n (now)
pold = np.zeros((nx, ny, nz)) # p at time n-1 (past)
pnew = np.zeros((nx, ny, nz)) # p at time n+1 (present)
d2px = np.zeros((nx, ny, nz)) # 2nd space derivative of p in x-direction
d2py = np.zeros((nx, ny, nz)) # 2nd space derivative of p in y-direction
d2pz = np.zeros((nx, ny, nz)) # 2nd space derivative of p in z-direction

# Initialize Velocity Model (assume homogeneous model)
# ----------------------------------------------------
c    = np.zeros((nx, ny, nz))
c    = c + c0                 # initialize wave velocity in model

# Initialize Grid
# ---------------
x    = np.arange(nx)
x    = x * dx                 # coordinate in x-direction
y    = np.arange(ny)
y    = y * dy                 # coordinate in y-direction
z    = np.arange(nz)
z    = z * dz                 # coordinate in z-direction

# Initialize Empty Seismogram
# ---------------------------
seis = np.zeros(nt)

# Analytical Solution
# -------------------
G    = time * 0.0
r    = np.sqrt((x[isx] - x[irx]) ** 2 + (y[isy] - y[iry]) ** 2 + (z[isz] - z[irz]) ** 2)
t_ar = time[0] + r / c0
i_ar = int(t_ar / dt)
G[i_ar] = 1.0 / (4 * np.pi * c0 ** 2 * r) # Calculate Green's function
Gc   = np.convolve(G, src)
Gc   = Gc[0:nt]
lim  = Gc.max() # get limit value from maximum amplitude of analytical solution

# Plot Position Configuration
# ---------------------------
plt.ion()
fig2 = plt.figure(figsize=(12, 12))
gs2  = gridspec.GridSpec(2,2,width_ratios=[1,1],height_ratios=[1,1],hspace=0.15, wspace=0.3)

# Plot 3D Wave Propagation
# ------------------------
# Note: comma is needed to update the variable

# 2D Slice at nz
# --------------
ax3  = plt.subplot(gs2[0])
leg1,= ax3.plot(isx, isz, 'r*', markersize=11) # plot source location
leg2,= ax3.plot(irx, irz, 'k^', markersize=8) # plot receiver location
slc1,= plt.plot(np.array([0,nx]),np.array([irx,irx]), linewidth=1, linestyle='--') # plot 2D slice at nx
slc2,= plt.plot(np.array([irx,irx]),np.array([0,ny]), linewidth=1, linestyle='--') # plot 2D slice at ny
im3  = ax3.imshow(p[:,:,isx],vmin=-lim, vmax=+lim, interpolation='nearest', cmap=plt.cm.RdBu)
div3 = make_axes_locatable(ax3)
cax3 = div3.append_axes("right", size="5%", pad=0.05) # size & position of colorbar
fig2.colorbar(im3, cax=cax3) # plot colorbar
ax3.set_title('Slice at nz = %d' % isz)
ax3.set_xlim(0, nz)
ax3.set_ylim(0, nz)
ax3.set_xlabel('nx')
ax3.set_ylabel('ny')
ax3.legend((leg1, leg2), ('Source', 'Receiver'), loc='upper right', fontsize=10, numpoints=1)

# 2D Slice at nx
# --------------
ax4  = plt.subplot(gs2[1])
im4  = ax4.imshow(p[irx,:,:],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
div4 = make_axes_locatable(ax4)
cax4 = div4.append_axes("right", size="5%", pad=0.05) # size & position of colorbar
fig2.colorbar(im4, cax=cax4) # plot colorbar
ax4.set_title('Slice at nx = %d' % irx)
ax4.set_xlim(0, nz)
ax4.set_ylim(0, ny)
ax4.set_xlabel('nz')
ax4.set_ylabel('ny')

# 2D Slice at ny
# --------------
ax5  = plt.subplot(gs2[2])
im5  = ax5.imshow(p[:,irx,:],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
div5 = make_axes_locatable(ax5)
cax5 = div5.append_axes("right", size="5%", pad=0.05) # size & position of colorbar
fig2.colorbar(im5, cax=cax5) # plot colorbar
ax5.set_title('Slice at ny = %d' % iry)
ax5.set_xlim(0, nz)
ax5.set_ylim(0, ny)
ax5.set_xlabel('nx')
ax5.set_ylabel('nz')

# Plot Seismogram
# ---------------
# Note: comma is needed to update the variable
ax6  = plt.subplot(gs2[3])
up61,= ax6.plot(time, seis) # update seismogram each time step
up62,= ax6.plot([0], [0], 'r|', markersize=15) # update time step position
ax6.set_ylim(-lim, lim)
ax6.set_xlim(time[0], time[-1])
ax6.set_title('Seismogram - Time Step (nt) = 0')
ax6.set_xlabel('Time [s]')
ax6.set_ylabel('Amplitude')
leg3,= ax6.plot(0,0,'r--',markersize=1)
leg4,= ax6.plot(0,0,'b-',markersize=1)
ax6.legend((leg3, leg4), ('Analytical', 'FD'), loc='upper right', fontsize=10, numpoints=1)

ax6.yaxis.tick_right()
ax6.yaxis.set_label_position("right")

plt.plot(time,Gc,'r--')
plt.show()
```

```
# 3D Wave Propagation (Finite Difference Solution)
# ------------------------------------------------

# Point Operator (choose 3 or 5 point operator)
# ---------------------------------------------
op   = 3
print(op, '- point operator')

# Calculate Partial Derivatives
# -----------------------------
for it in range(nt):
    if op == 3: # use 3 point operator FD scheme
        for i in range(1, nz - 1):
                d2pz[:, :, i] = (p[:, :, i + 1] - 2 * p[:, :, i] + p[:, :, i - 1]) / dz ** 2
        for j in range(1, ny - 1):
                d2py[:, j, :] = (p[:, j + 1, :] - 2 * p[:, j, :] + p[:, j - 1, :]) / dy ** 2
        for k in range(1, nx - 1):
                d2px[k, :, :] = (p[k + 1, :, :] - 2 * p[k, :, :] + p[k - 1, :, :]) / dx ** 2

    if op == 5: # use 5 point operator FD scheme
        #-----------------------------------------------#
        #     IMPLEMENT 5 POINT OPERATOR CODE HERE!     #
        #-----------------------------------------------#
        pass

    # Time Extrapolation
    # ------------------
    pnew = 2 * p - pold + (c ** 2) * (dt ** 2) * (d2px + d2py + d2pz)

    # Add Source Term at isx, isy and isz
    # -----------------------------------
    pnew[isx, isy, isz] = pnew[isx, isy, isz] + src[it] / (dx * dy * dz) * (dt ** 2)

    # Remap Time Levels
    # -----------------
    pold, p = p, pnew

    # Output Seismogram
    # -----------------
    seis[it] = p[irx, iry, irz]

    # Update data for Wave Propagation Plot
    # -------------------------------------
    idisp = 5 # display frequency
    if (it % idisp) == 0:
        ax3.imshow(p[:, :, isx],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
        ax4.imshow(p[irx, :, :],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
        ax5.imshow(p[:, irx, :],vmin=-lim, vmax=+lim, interpolation="nearest", cmap=plt.cm.RdBu)
        ax6.set_title('Seismogram - Time Step (nt) = %d' % it)
        up61.set_ydata(seis)
        up62.set_data(time[it], seis[it])
        fig2.canvas.draw()
```

```

```

```

```

notebooks/Computational Seismology/The Finite-Difference Method/fd_ac3d_homogeneous.ipynb