#Logistic Regression in Practice

In this session, we shall apply logistic regression and look at predictions using it.

```python
#First, we import the required packages
import pandas as pd #the pandas library is useful for data processing
import numpy as np #numpy package will be useful for numerical computations
import matplotlib.pyplot as plt #the matplotlib library is useful for plotting purposes

# The following python directive helps to plot the graph in the notebook directly
%matplotlib inline
```

Now let us consider some open source data sets available in the internet.

```python
from sklearn.datasets import load_iris  #importing the load_iris class
iris_data = load_iris() #loading the iris dataset in iris_data

print(iris_data['DESCR']) #checking out the description of the dataset
```

    .. _iris_dataset:

    Iris plants dataset
    --------------------

    **Data Set Characteristics:**

        :Number of Instances: 150 (50 in each of three classes)
        :Number of Attributes: 4 numeric, predictive attributes and the class
        :Attribute Information:
            - sepal length in cm
            - sepal width in cm
            - petal length in cm
            - petal width in cm
            - class:
                    - Iris-Setosa
                    - Iris-Versicolour
                    - Iris-Virginica

        :Summary Statistics:

        ============== ==== ==== ======= ===== ====================
                        Min  Max   Mean    SD   Class Correlation
        ============== ==== ==== ======= ===== ====================
        sepal length:   4.3  7.9   5.84   0.83    0.7826
        sepal width:    2.0  4.4   3.05   0.43   -0.4194
        petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
        petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
        ============== ==== ==== ======= ===== ====================

        :Missing Attribute Values: None
        :Class Distribution: 33.3% for each of 3 classes.
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
    from Fisher's paper. Note that it's the same as in R, but not as in the UCI
    Machine Learning Repository, which has two wrong data points.

    This is perhaps the best known database to be found in the
    pattern recognition literature.  Fisher's paper is a classic in the field and
    is referenced frequently to this day.  (See Duda & Hart, for example.)  The
    data set contains 3 classes of 50 instances each, where each class refers to a
    type of iris plant.  One class is linearly separable from the other 2; the
    latter are NOT linearly separable from each other.

    .. topic:: References

       - Fisher, R.A. "The use of multiple measurements in taxonomic problems"
         Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
         Mathematical Statistics" (John Wiley, NY, 1950).
       - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.
         (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
       - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
         Structure and Classification Rule for Recognition in Partially Exposed
         Environments".  IEEE Transactions on Pattern Analysis and Machine
         Intelligence, Vol. PAMI-2, No. 1, 67-71.
       - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
         on Information Theory, May 1972, 431-433.
       - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
         conceptual clustering system finds 3 classes in the data.
       - Many, many more ...

```python
iris_data
```

    {'DESCR': '.. _iris_dataset:\n\nIris plants dataset\n--------------------\n\n**Data Set Characteristics:**\n\n    :Number of Instances: 150 (50 in each of three classes)\n    :Number of Attributes: 4 numeric, predictive attributes and the class\n    :Attribute Information:\n        - sepal length in cm\n        - sepal width in cm\n        - petal length in cm\n        - petal width in cm\n        - class:\n                - Iris-Setosa\n                - Iris-Versicolour\n                - Iris-Virginica\n                \n    :Summary Statistics:\n\n    ============== ==== ==== ======= ===== ====================\n                    Min  Max   Mean    SD   Class Correlation\n    ============== ==== ==== ======= ===== ====================\n    sepal length:   4.3  7.9   5.84   0.83    0.7826\n    sepal width:    2.0  4.4   3.05   0.43   -0.4194\n    petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)\n    petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)\n    ============== ==== ==== ======= ===== ====================\n\n    :Missing Attribute Values: None\n    :Class Distribution: 33.3% for each of 3 classes.\n    :Creator: R.A. Fisher\n    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n    :Date: July, 1988\n\nThe famous Iris database, first used by Sir R.A. Fisher. The dataset is taken\nfrom Fisher\'s paper. Note that it\'s the same as in R, but not as in the UCI\nMachine Learning Repository, which has two wrong data points.\n\nThis is perhaps the best known database to be found in the\npattern recognition literature.  Fisher\'s paper is a classic in the field and\nis referenced frequently to this day.  (See Duda & Hart, for example.)  The\ndata set contains 3 classes of 50 instances each, where each class refers to a\ntype of iris plant.  One class is linearly separable from the other 2; the\nlatter are NOT linearly separable from each other.\n\n.. topic:: References\n\n   - Fisher, R.A. "The use of multiple measurements in taxonomic problems"\n     Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to\n     Mathematical Statistics" (John Wiley, NY, 1950).\n   - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.\n     (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.\n   - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System\n     Structure and Classification Rule for Recognition in Partially Exposed\n     Environments".  IEEE Transactions on Pattern Analysis and Machine\n     Intelligence, Vol. PAMI-2, No. 1, 67-71.\n   - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions\n     on Information Theory, May 1972, 431-433.\n   - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II\n     conceptual clustering system finds 3 classes in the data.\n   - Many, many more ...',
     'data': array([[5.1, 3.5, 1.4, 0.2],
            [4.9, 3. , 1.4, 0.2],
            [4.7, 3.2, 1.3, 0.2],
            [4.6, 3.1, 1.5, 0.2],
            [5. , 3.6, 1.4, 0.2],
            [5.4, 3.9, 1.7, 0.4],
            [4.6, 3.4, 1.4, 0.3],
            [5. , 3.4, 1.5, 0.2],
            [4.4, 2.9, 1.4, 0.2],
            [4.9, 3.1, 1.5, 0.1],
            [5.4, 3.7, 1.5, 0.2],
            [4.8, 3.4, 1.6, 0.2],
            [4.8, 3. , 1.4, 0.1],
            [4.3, 3. , 1.1, 0.1],
            [5.8, 4. , 1.2, 0.2],
            [5.7, 4.4, 1.5, 0.4],
            [5.4, 3.9, 1.3, 0.4],
            [5.1, 3.5, 1.4, 0.3],
            [5.7, 3.8, 1.7, 0.3],
            [5.1, 3.8, 1.5, 0.3],
            [5.4, 3.4, 1.7, 0.2],
            [5.1, 3.7, 1.5, 0.4],
            [4.6, 3.6, 1. , 0.2],
            [5.1, 3.3, 1.7, 0.5],
            [4.8, 3.4, 1.9, 0.2],
            [5. , 3. , 1.6, 0.2],
            [5. , 3.4, 1.6, 0.4],
            [5.2, 3.5, 1.5, 0.2],
            [5.2, 3.4, 1.4, 0.2],
            [4.7, 3.2, 1.6, 0.2],
            [4.8, 3.1, 1.6, 0.2],
            [5.4, 3.4, 1.5, 0.4],
            [5.2, 4.1, 1.5, 0.1],
            [5.5, 4.2, 1.4, 0.2],
            [4.9, 3.1, 1.5, 0.2],
            [5. , 3.2, 1.2, 0.2],
            [5.5, 3.5, 1.3, 0.2],
            [4.9, 3.6, 1.4, 0.1],
            [4.4, 3. , 1.3, 0.2],
            [5.1, 3.4, 1.5, 0.2],
            [5. , 3.5, 1.3, 0.3],
            [4.5, 2.3, 1.3, 0.3],
            [4.4, 3.2, 1.3, 0.2],
            [5. , 3.5, 1.6, 0.6],
            [5.1, 3.8, 1.9, 0.4],
            [4.8, 3. , 1.4, 0.3],
            [5.1, 3.8, 1.6, 0.2],
            [4.6, 3.2, 1.4, 0.2],
            [5.3, 3.7, 1.5, 0.2],
            [5. , 3.3, 1.4, 0.2],
            [7. , 3.2, 4.7, 1.4],
            [6.4, 3.2, 4.5, 1.5],
            [6.9, 3.1, 4.9, 1.5],
            [5.5, 2.3, 4. , 1.3],
            [6.5, 2.8, 4.6, 1.5],
            [5.7, 2.8, 4.5, 1.3],
            [6.3, 3.3, 4.7, 1.6],
            [4.9, 2.4, 3.3, 1. ],
            [6.6, 2.9, 4.6, 1.3],
            [5.2, 2.7, 3.9, 1.4],
            [5. , 2. , 3.5, 1. ],
            [5.9, 3. , 4.2, 1.5],
            [6. , 2.2, 4. , 1. ],
            [6.1, 2.9, 4.7, 1.4],
            [5.6, 2.9, 3.6, 1.3],
            [6.7, 3.1, 4.4, 1.4],
            [5.6, 3. , 4.5, 1.5],
            [5.8, 2.7, 4.1, 1. ],
            [6.2, 2.2, 4.5, 1.5],
            [5.6, 2.5, 3.9, 1.1],
            [5.9, 3.2, 4.8, 1.8],
            [6.1, 2.8, 4. , 1.3],
            [6.3, 2.5, 4.9, 1.5],
            [6.1, 2.8, 4.7, 1.2],
            [6.4, 2.9, 4.3, 1.3],
            [6.6, 3. , 4.4, 1.4],
            [6.8, 2.8, 4.8, 1.4],
            [6.7, 3. , 5. , 1.7],
            [6. , 2.9, 4.5, 1.5],
            [5.7, 2.6, 3.5, 1. ],
            [5.5, 2.4, 3.8, 1.1],
            [5.5, 2.4, 3.7, 1. ],
            [5.8, 2.7, 3.9, 1.2],
            [6. , 2.7, 5.1, 1.6],
            [5.4, 3. , 4.5, 1.5],
            [6. , 3.4, 4.5, 1.6],
            [6.7, 3.1, 4.7, 1.5],
            [6.3, 2.3, 4.4, 1.3],
            [5.6, 3. , 4.1, 1.3],
            [5.5, 2.5, 4. , 1.3],
            [5.5, 2.6, 4.4, 1.2],
            [6.1, 3. , 4.6, 1.4],
            [5.8, 2.6, 4. , 1.2],
            [5. , 2.3, 3.3, 1. ],
            [5.6, 2.7, 4.2, 1.3],
            [5.7, 3. , 4.2, 1.2],
            [5.7, 2.9, 4.2, 1.3],
            [6.2, 2.9, 4.3, 1.3],
            [5.1, 2.5, 3. , 1.1],
            [5.7, 2.8, 4.1, 1.3],
            [6.3, 3.3, 6. , 2.5],
            [5.8, 2.7, 5.1, 1.9],
            [7.1, 3. , 5.9, 2.1],
            [6.3, 2.9, 5.6, 1.8],
            [6.5, 3. , 5.8, 2.2],
            [7.6, 3. , 6.6, 2.1],
            [4.9, 2.5, 4.5, 1.7],
            [7.3, 2.9, 6.3, 1.8],
            [6.7, 2.5, 5.8, 1.8],
            [7.2, 3.6, 6.1, 2.5],
            [6.5, 3.2, 5.1, 2. ],
            [6.4, 2.7, 5.3, 1.9],
            [6.8, 3. , 5.5, 2.1],
            [5.7, 2.5, 5. , 2. ],
            [5.8, 2.8, 5.1, 2.4],
            [6.4, 3.2, 5.3, 2.3],
            [6.5, 3. , 5.5, 1.8],
            [7.7, 3.8, 6.7, 2.2],
            [7.7, 2.6, 6.9, 2.3],
            [6. , 2.2, 5. , 1.5],
            [6.9, 3.2, 5.7, 2.3],
            [5.6, 2.8, 4.9, 2. ],
            [7.7, 2.8, 6.7, 2. ],
            [6.3, 2.7, 4.9, 1.8],
            [6.7, 3.3, 5.7, 2.1],
            [7.2, 3.2, 6. , 1.8],
            [6.2, 2.8, 4.8, 1.8],
            [6.1, 3. , 4.9, 1.8],
            [6.4, 2.8, 5.6, 2.1],
            [7.2, 3. , 5.8, 1.6],
            [7.4, 2.8, 6.1, 1.9],
            [7.9, 3.8, 6.4, 2. ],
            [6.4, 2.8, 5.6, 2.2],
            [6.3, 2.8, 5.1, 1.5],
            [6.1, 2.6, 5.6, 1.4],
            [7.7, 3. , 6.1, 2.3],
            [6.3, 3.4, 5.6, 2.4],
            [6.4, 3.1, 5.5, 1.8],
            [6. , 3. , 4.8, 1.8],
            [6.9, 3.1, 5.4, 2.1],
            [6.7, 3.1, 5.6, 2.4],
            [6.9, 3.1, 5.1, 2.3],
            [5.8, 2.7, 5.1, 1.9],
            [6.8, 3.2, 5.9, 2.3],
            [6.7, 3.3, 5.7, 2.5],
            [6.7, 3. , 5.2, 2.3],
            [6.3, 2.5, 5. , 1.9],
            [6.5, 3. , 5.2, 2. ],
            [6.2, 3.4, 5.4, 2.3],
            [5.9, 3. , 5.1, 1.8]]),
     'feature_names': ['sepal length (cm)',
      'sepal width (cm)',
      'petal length (cm)',
      'petal width (cm)'],
     'filename': '/usr/local/lib/python3.7/dist-packages/sklearn/datasets/data/iris.csv',
     'target': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
            1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
            1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
            2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
            2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]),
     'target_names': array(['setosa', 'versicolor', 'virginica'], dtype='<U10')}

```python
X = iris_data['data'][:,:2]
print(X)
```

    [[5.1 3.5]
     [4.9 3. ]
     [4.7 3.2]
     [4.6 3.1]
     [5.  3.6]
     [5.4 3.9]
     [4.6 3.4]
     [5.  3.4]
     [4.4 2.9]
     [4.9 3.1]
     [5.4 3.7]
     [4.8 3.4]
     [4.8 3. ]
     [4.3 3. ]
     [5.8 4. ]
     [5.7 4.4]
     [5.4 3.9]
     [5.1 3.5]
     [5.7 3.8]
     [5.1 3.8]
     [5.4 3.4]
     [5.1 3.7]
     [4.6 3.6]
     [5.1 3.3]
     [4.8 3.4]
     [5.  3. ]
     [5.  3.4]
     [5.2 3.5]
     [5.2 3.4]
     [4.7 3.2]
     [4.8 3.1]
     [5.4 3.4]
     [5.2 4.1]
     [5.5 4.2]
     [4.9 3.1]
     [5.  3.2]
     [5.5 3.5]
     [4.9 3.6]
     [4.4 3. ]
     [5.1 3.4]
     [5.  3.5]
     [4.5 2.3]
     [4.4 3.2]
     [5.  3.5]
     [5.1 3.8]
     [4.8 3. ]
     [5.1 3.8]
     [4.6 3.2]
     [5.3 3.7]
     [5.  3.3]
     [7.  3.2]
     [6.4 3.2]
     [6.9 3.1]
     [5.5 2.3]
     [6.5 2.8]
     [5.7 2.8]
     [6.3 3.3]
     [4.9 2.4]
     [6.6 2.9]
     [5.2 2.7]
     [5.  2. ]
     [5.9 3. ]
     [6.  2.2]
     [6.1 2.9]
     [5.6 2.9]
     [6.7 3.1]
     [5.6 3. ]
     [5.8 2.7]
     [6.2 2.2]
     [5.6 2.5]
     [5.9 3.2]
     [6.1 2.8]
     [6.3 2.5]
     [6.1 2.8]
     [6.4 2.9]
     [6.6 3. ]
     [6.8 2.8]
     [6.7 3. ]
     [6.  2.9]
     [5.7 2.6]
     [5.5 2.4]
     [5.5 2.4]
     [5.8 2.7]
     [6.  2.7]
     [5.4 3. ]
     [6.  3.4]
     [6.7 3.1]
     [6.3 2.3]
     [5.6 3. ]
     [5.5 2.5]
     [5.5 2.6]
     [6.1 3. ]
     [5.8 2.6]
     [5.  2.3]
     [5.6 2.7]
     [5.7 3. ]
     [5.7 2.9]
     [6.2 2.9]
     [5.1 2.5]
     [5.7 2.8]
     [6.3 3.3]
     [5.8 2.7]
     [7.1 3. ]
     [6.3 2.9]
     [6.5 3. ]
     [7.6 3. ]
     [4.9 2.5]
     [7.3 2.9]
     [6.7 2.5]
     [7.2 3.6]
     [6.5 3.2]
     [6.4 2.7]
     [6.8 3. ]
     [5.7 2.5]
     [5.8 2.8]
     [6.4 3.2]
     [6.5 3. ]
     [7.7 3.8]
     [7.7 2.6]
     [6.  2.2]
     [6.9 3.2]
     [5.6 2.8]
     [7.7 2.8]
     [6.3 2.7]
     [6.7 3.3]
     [7.2 3.2]
     [6.2 2.8]
     [6.1 3. ]
     [6.4 2.8]
     [7.2 3. ]
     [7.4 2.8]
     [7.9 3.8]
     [6.4 2.8]
     [6.3 2.8]
     [6.1 2.6]
     [7.7 3. ]
     [6.3 3.4]
     [6.4 3.1]
     [6.  3. ]
     [6.9 3.1]
     [6.7 3.1]
     [6.9 3.1]
     [5.8 2.7]
     [6.8 3.2]
     [6.7 3.3]
     [6.7 3. ]
     [6.3 2.5]
     [6.5 3. ]
     [6.2 3.4]
     [5.9 3. ]]

From the description, it is clear that *iris* dataset consists of essentially 3 classes: *Iris-Setosa*, *Iris-Versicolour*, *Iris-Virginica*. As logistic regression has been introduced as a binary classifier, we can alter the data into a binary classification problem based on finding whether a flower belongs to *Iris-Virginica* or not. In order to do that, we change the labels $\{0,1\}$ to $0$ and 2 to 1.

```python
y = np.where(iris_data['target'] == 2, 1, 0) #shorthand notation to change all labels other than 2 as 0 and 2 as 1
print(y)
```

    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1]

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 1)
print('Size of training set:', len(X_train))
n = len(X_train)
print('Size of testing set:', len(X_test))
print(y_test)
```

    Size of training set: 105
    Size of testing set: 45
    [0 0 0 0 1 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 1 1 0 0
     0 0 1 1 0 1 1 0]

We can further write $p(x)$ equivalently as:

$
p(x)=p(x;\beta_0,\beta_1) = \frac{1}{1+e^{-(\beta_0 + \beta_1^T x)}}, \text{where }\beta_1,x\in\mathbb{R}^2.
$

If we denote $p(x)$ simply as $p$ and if we have the notations $\mathbf{x}=\begin{bmatrix}
x \\ 1
\end{bmatrix}, \beta=\begin{bmatrix}
\beta_1 \\ \beta_0
\end{bmatrix}$
then we can write:

$
\begin{align}
p = \frac{1}{1+e^{-\beta^\top \mathbf{x}}}.
\end{align}
$

Then note the following that the following algebraic manipulations hold:

$
\begin{align}
p &= \frac{1}{1+e^{-\beta^\top \mathbf{x}}} \\
\implies 1 + e^{-\beta^\top \mathbf{x}} &= \frac{1}{p} \\
\implies  e^{-\beta^\top \mathbf{x}} &= \frac{1}{p} - 1 \\
\implies  e^{-\beta^\top \mathbf{x}} &= \frac{1-p}{p} \\
\implies \ln e^{-\beta^\top \mathbf{x}} &= \ln \frac{1-p}{p} \\
\implies -\beta^\top \mathbf{x} &= \ln \frac{1-p}{p} \\
\implies \beta^\top \mathbf{x} &= \ln \frac{p}{1-p} \\
\implies \beta_0 + \beta_1^T x &= \ln \frac{p}{1-p}.
\end{align}
$

Thus, even if we did not have a straightforward dependence of $Y$ on an observation $x$ of $X$ as a linear relation, we see that the linear relation $\beta_0 + \beta_1 x$ is related to the probability $p$ using:

$
\ln\frac{p}{1-p}=\beta^\top \mathbf{x}.
$

Note that the ratio $\frac{p}{1-p}$ is called $\textbf{odds}$ that the event $Y=1$ occurs, and hence $\ln \frac{p}{1-p}$ denotes the $\textbf{log odds}$.

More popularly, the log odds $\ln \frac{p}{1-p}$ is called the $\textbf{logit}$ function.

$\Large{\text{Likelihood function}}$

We now define a quantity which will be useful in the estimation of the parameters $\beta_0, \beta_1$ used to model $p$.

Given an observation $X=x$ we define the $\textbf{likelihood function}$ as:

$
L(y;p) = p^y(1-p)^{(1-y)}
$

where recall that $p=p(x)=p(x;\beta_0,\beta_1)=\frac{1}{1+e^{-{(\beta_0 + \beta_1^T x)}}}$. Note that likelihood function is simply an equivalent way to represent $P[Y=y]$, when $Y$ is assumed to be Bernoulli random variable.

Then observe that the natural goal is to maximize the likelihood function with respect to parameters $\beta_0, \beta_1$.

Now given a data set $D$ containing $n$ observations of the form $\{({x}^1,y^1), ({x}^2,y^2), \ldots, ({x}^n,y^n)\}$, and assuming that the pairs $({x}^i,y^i)$ are independent observations, then it is possible to extend the likelihood function as:

$
\begin{align}
L(y^1,\ldots,y^n;p^1,\ldots,p^n) = \Pi_{i=1}^{n} {(p^i)}^{y^i}(1-p^i)^{(1-y^i)}.
\end{align}
$

We can now write the $\textbf{log likelihood}$ function as:

$
\begin{align}
\ln L(y^1,\ldots,y^n;p^1,\ldots,p^n) = \sum_{i=1}^{n} y^i \ln {(p^i)} + (1-y^i) \ln (1-p^i).
\end{align}
$

Since log likelihood function is monotonic, maximizing the likelihood function is equivalent to maximizing the log likelihood function.

Hence the concerned optimization problem is:

$
\max_{\beta_0, \beta_1} \ln L(y^1,\ldots,y^n;p^1;\ldots,p^n)= \sum_{i=1}^{n} y^i \ln {(p^i)} + (1-y^i) \ln (1-p^i).
$

Note that $p^i = p(x^i) = p(x^i; \beta_0, \beta_1) = \frac{1}{1+e^{-{(\beta_0 + \beta_1^T x^i)}}}, \forall i = 1,\ldots,n$.

$\Large{\text{Solving the likelihood maximization problem}}:$

To solve

$
\max_{\beta_0, \beta_1} \ln L(y^1,\ldots,y^n;p^1;\ldots,p^n)= \sum_{i=1}^{n} y^i \ln {(p^i)} + (1-y^i) \ln (1-p^i).
$

we can find the derivatives of the objective function with respect to $\beta_0$ and $\beta_1$ as:

$
\begin{align}
\frac{\partial{ \ln L }} {\partial \beta_0} &= \sum_{i=1}^{n} y^i (1-p^i) - (1-y^i) p^i \\
\frac{\partial{ \ln L }} {\partial \beta_1} &= \sum_{i=1}^{n} x^i(y^i (1-p^i) - (1-y^i) p^i)
\end{align}.
$

Equating these derivatives to zero will lead to system of non-linear equations which is generally difficult to solve to obtain closed form expression.

Hence we generally adopt an iterative procedure as follows to find the optimal $\beta_0, \beta_1$.

$\large{\text{Gradient ascent for solving the likelihood maximization problem}}:$

$
\begin{align}
&\textbf{Step 0:}  \text{Input data set $D$, tolerances $\epsilon_1, \epsilon_2$.} \\
&\textbf{Step 1:}  \text{Start with arbitrary $\beta_0, \beta_1$.} \\
&\textbf{Step 2:}  \text{For $k=1,2,\ldots$} \\
&\quad \quad \textbf{Step 2.1:} \text{Compute differentials  $\frac{\partial{ \ln L }} {\partial \beta_0}$ and $\frac{\partial{ \ln L }} {\partial \beta_1}$} \\
&\quad \quad \textbf{Step 2.2:}  \text{Compute step length $\eta$ using line search procedure} \\
&\quad \quad \textbf{Step 2.3:}  \beta_0 = \beta_0 + \eta \frac{\partial{ \ln L }} {\partial \beta_0} \\
&\quad \quad \textbf{Step 2.4:}  \beta_1 = \beta_1 + \eta \frac{\partial{ \ln L }} {\partial \beta_1} \\
&\quad \quad \textbf{Step 2.5:}  \text{if $\|\nabla_{\beta} \ln L\|_2 \leq \epsilon_1$ break from loop} \\
&\quad \quad \textbf{Step 2.6:}  \text{if relative change in function value is $\leq \epsilon_2$ break from loop} \\
&\textbf{Step 3:}  \text{ Output $\beta_0, \beta_1$}
\end{align}
$

Note that $\nabla_{\beta} \ln L$ denotes the gradient of $\ln L$ with respect to $\beta$ given by:

$
\nabla_{\beta} \ln L = \begin{bmatrix}
\frac{\partial{ \ln L }} {\partial \beta_0}\\
\frac{\partial{ \ln L }} {\partial \beta_1}
\end{bmatrix}
$

and $\eta$ denotes the learning rate.

$\Large{\text{Computing } \beta \text{ for the training data set}}:$

```python
import numpy as np
#computing log likelihood function
def log_likelihood(beta_0,beta_1):
  log_likelihood = 0.0
  for i in range(n):
    x_i = X_train[i] #access element in training data
    y_i = float(y_train[i]) #access label in trainning data
    p_i = 1.0/(1.0+np.exp(-beta_0 - np.dot(beta_1,x_i))) #probability with the current beta_0, beta_1
    log_likelihood += y_i*np.log(p_i) + (1.0-y_i)*np.log(1-p_i)
  return log_likelihood
```

```python
#gradient ascent for likelihood maximization
beta_0 = 0.0
beta_1 = np.zeros(2)
max_iter = 1000000
eps_1 = 0.00001 #tolerance for gradient norm
eps_2 = 1e-9 #tolerance for relative function value difference

f_val = log_likelihood(beta_0,beta_1)

#store the objective function values for plotting purposes
f_val_list = []
f_val_list.append(f_val)

print('Initial values: beta_0:',beta_0, ' beta_1:',beta_1, ' log likeihood:', f_val)
#the loop
for k in range(max_iter):

  grad_beta_0 = 0.0
  grad_beta_1= np.zeros(2)
  #pass through the data to compute gradients
  for i in range(n):
    x_i = X_train[i] #access element in training set
    y_i = float(y_train[i]) #access the label
    p_i = 1.0/(1.0+np.exp(-beta_0 - np.dot(beta_1,x_i))) #probability with the current beta_0, beta_1
    grad_beta_0 +=  (y_i * (1.0-p_i)) - ((1.0-y_i) * p_i)
    grad_beta_1 +=  x_i * (y_i * (1.0-p_i) - (1.0-y_i) * p_i)

  #lr = linesearch(beta_0, beta_1, grad_beta_0, grad_beta_1, float(f_val_list[-1]))
  lr = 0.0001
  beta_0 += lr * grad_beta_0
  beta_1 += lr * grad_beta_1
  #print('k: ', k, ' grad beta_0: ', grad_beta_0, 'grad beta_1: ',grad_beta_1, 'beta_0:', beta_0, 'beta_1:',beta_1)
  grad_norm = np.sqrt(grad_beta_0**2 + grad_beta_1[0]**2+ grad_beta_1[1]**2)
  f_val = log_likelihood(beta_0,beta_1)
  f_val_list.append(f_val)

  rel_change_in_fval = np.abs((f_val - f_val_list[-2])/f_val_list[-2])

  if k%10000+1 == 1:
    print('k: ', k,  ' beta_0:', beta_0, ' beta_1:',beta_1, ' grad norm:', grad_norm, 'log likeihood:', f_val)
  if grad_norm <= eps_1 or rel_change_in_fval <= eps_2:
    break
print('Final: k: ', k,  ' beta_0:', beta_0, ' beta_1:',beta_1, ' grad norm:', grad_norm, 'log likeihood:', f_val)

#plot the function values during optimization
plt.plot( f_val_list, '-r', label='Log Likelihood values')
plt.title("Log likelihood function values vs iterations")
plt.xlabel("Iteration")
plt.ylabel("Log Likelihood")
plt.legend(loc='lower right')
#plt.grid()
plt.show()
```

As discussed earlier, in logistic regression we aim to solve the following optimization problem:
$$
\max_{\beta_0, \beta_1} \ln L(y^1,\ldots,y^n;p^1;\ldots,p^n)= \sum_{i=1}^{n} y^i \ln {(p^i)} + (1-y^i) \ln (1-p^i),
$$
where $p^i=p(x^i)= p(x^i; \beta_0, \beta_1) = \frac{1}{1+e^{-{(\beta_0 + \beta_1^T x^i)}}}$,
$x^i,\beta_1\in\mathbb{R}^2,\beta_0\in\mathbb{R}\text{ and } y^i\in\{0,1\},\forall i = 1,\ldots,n$.

Provided the lables are $\{-1,1\}$ instead of $\{0,1\}$, using the probabilistic interpetation, we can write

$p(x)=p(x;\beta_0,\beta_1)=\frac{e^{(\beta_0+\beta_1^Tx)}}{1+e^{(\beta_0+\beta_1^Tx)}}$. Then we can rewrite this as
$$P[y=1|x]=p(x)=\frac{e^{(\beta_0+\beta_1^Tx)}}{1+e^{(\beta_0+\beta_1^Tx)}},$$
$$P[y=-1|x]=1-p(x)=\frac{1}{1+e^{(\beta_0+\beta_1^Tx)}}.$$
Therefore,
$$P[y=-1\text{ or }1|x]=\frac{1}{1+e^{-y(\beta_0+\beta_1^Tx)}}.$$
Thus, the maximum log likelihood problem becomes
\begin{align*}
\max_{\beta_0,\beta_1}\ln L &= \sum_{i=1}^n\ln\left(\frac{1}{1+e^{-y^i(\beta_0+\beta_1^Tx^i)}}\right)\\
&=-\sum_{i=1}^n{\ln\left(1+e^{-y^i(\beta_0+\beta_1^Tx^i)}\right)}.
\end{align*}
The maximization problem can be converted to a minimization problem owing to the - sign ahead of the summation.
Scikit-learn uses a regularized version of the minimization problem given by:
$$\min_{\beta_0,\beta_1}\frac{1}{2}\beta_1^T\beta_1+C\sum_{i=1}^n\ln\left({1+e^{-y^i(\beta_0+\beta_1^Tx^i)}}\right),$$
and considers the labels as -1 and 1 instead of 0 and 1.

#Using sklearn to find the Logistic regression model

```python
from sklearn.linear_model import LogisticRegression
# creating an object of LogisticRegression class
logit = LogisticRegression(C=1e6) # C is set to be large number in order to remove the inbuilt regularization
# fitting the training data
logit.fit(X_train,y_train)
```

    LogisticRegression(C=1000000.0, class_weight=None, dual=False,
                       fit_intercept=True, intercept_scaling=1, l1_ratio=None,
                       max_iter=100, multi_class='auto', n_jobs=None, penalty='l2',
                       random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                       warm_start=False)

```python
print(logit.intercept_)
print(logit.coef_)
```

    [-14.57054803]
    [[ 2.45422089 -0.22889307]]

```python
print('Train Accuracy:', logit.score(X_train,y_train)) #Average accuracy on training data
```

    Train Accuracy: 0.819047619047619

```python
from sklearn.metrics import accuracy_score
y_test_predicted = logit.predict(X_test)
print('Test Accuracy:', accuracy_score(y_test, y_test_predicted)) #Average accuracy on test data
sum(y_test_predicted)
```

    Test Accuracy: 0.8222222222222222

    17

**Visualization of the Model**

```python
# point in the mesh [x_min, x_max]x[y_min, y_max].
x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
h = .01  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)) #creating x-y mesh for plot
Z = logit.predict(np.c_[xx.ravel(), yy.ravel()]) #using the learned model to predict all points on the mesh
# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure
plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)

# Plot also the training points
plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', cmap=plt.cm.Paired) #Complete data set with the actual labels
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
#plt.xticks(())
#plt.yticks(())

plt.show()
```

Though the test accuracy is coming out to be better than training accuracy, we need to be careful as essentially we are dealing with imbalanced data. The number of samples with 0 label is significantly larger than the number of samples with 1 label. A better look may be obtained by looking at the corresponding **Confusion Matrix**.

```python
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_test_predicted)
print(cm)
```

    [[26  6]
     [ 2 11]]

```python
#Plotting the confusion matrix for better interpretation
import seaborn as sns
ax = plt.subplot()
sns.heatmap(cm, annot=True, fmt='g', ax=ax)
# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
ax.xaxis.set_ticklabels(['0', '1'])
ax.yaxis.set_ticklabels(['0', '1'])
```

In order to deal with *imbalanced data*, we can look at the following:

*   True Positive  (TP)
*   True Negative  (TN)
*   False Positive (FP)
*   False Negative (FN).

Based on the above classes, the following can be defined:
$$Sensitivity = \frac{TP}{TP + FN}$$
$$Specificity = \frac{TN}{FP + TN}$$
$$Precision = \frac{TP}{TP + FP}$$
$$Recall = \frac{TP}{TP + FN}$$
$$F-Measure = 2\frac{Precision\times Recall}{Precision + Recall}$$

Note: All the expressions are based on the existence of a positive class and a negative class.

```python
from sklearn.metrics import precision_recall_fscore_support
precision_recall_fscore_support(y_test, y_test_predicted) #class-wise evaluation of the metrics.
```

    (array([0.92857143, 0.64705882]),
     array([0.8125    , 0.84615385]),
     array([0.86666667, 0.73333333]),
     array([32, 13]))

$\Huge{\text{Linear Regression with Regularizers}}$

Now we shall learn about linear regression when we impose regularizers on the parameters.

Let us first consider a sample data, which will be useful for our study of linear regression with regularizers. Now let us consider some open source data sets available in the internet. The data set we will consider represents airline maintanence costs for different airline companies as a response variable dependent on multiple attributes.

```python
#Let us explore the data and the description

#Get the data from the web
const_maint_data = pd.read_csv('http://users.stat.ufl.edu/~winner/data/const_maint.dat', delim_whitespace=True, index_col=False, header=None)

#print the first few rows of the data
const_maint_data.head()

```

      0
      1
      2
      3
      4
      5

      0
      1
      6.068
      1
      0
      0
      16.60

      1
      2
      4.602
      1
      0
      0
      20.37

      2
      3
      3.282
      1
      0
      0
      20.37

      3
      4
      2.192
      1
      1
      0
      20.37

      4
      5
      2.572
      1
      1
      0
      20.37

```python
#if you wish to print the entire data, then use the following
const_maint_data
```

      0
      1
      2
      3
      4
      5

      0
      1
      6.068
      1
      0
      0
      16.60

      1
      2
      4.602
      1
      0
      0
      20.37

      2
      3
      3.282
      1
      0
      0
      20.37

      3
      4
      2.192
      1
      1
      0
      20.37

      4
      5
      2.572
      1
      1
      0
      20.37

      5
      6
      4.142
      1
      0
      0
      20.37

      6
      7
      5.321
      1
      1
      1
      21.00

      7
      8
      4.421
      1
      1
      1
      21.00

      8
      9
      5.301
      1
      1
      1
      21.00

      9
      10
      9.679
      1
      0
      0
      27.01

      10
      11
      11.997
      1
      0
      1
      31.00

      11
      12
      7.757
      1
      0
      0
      31.00

      12
      13
      12.597
      1
      0
      0
      31.00

      13
      14
      13.045
      1
      0
      1
      33.20

      14
      15
      2.471
      0
      0
      0
      45.00

      15
      16
      6.689
      0
      0
      0
      46.27

      16
      17
      12.139
      1
      0
      0
      46.27

      17
      18
      10.859
      1
      0
      0
      46.27

      18
      19
      7.119
      0
      1
      0
      68.42

      19
      20
      3.728
      0
      1
      0
      83.70

      20
      21
      46.057
      1
      1
      0
      218.00

      21
      22
      49.847
      1
      1
      0
      218.00

      22
      23
      61.671
      1
      1
      0
      335.00

      23
      24
      56.167
      1
      0
      1
      228.00

      24
      25
      9.721
      1
      0
      1
      22.60

      25
      26
      8.535
      1
      0
      1
      21.70

      26
      27
      66.910
      1
      1
      1
      229.00

      27
      28
      11.101
      1
      0
      1
      22.60

      28
      29
      12.511
      1
      0
      1
      22.60

      29
      30
      11.397
      1
      0
      1
      31.00

      30
      31
      4.069
      0
      0
      0
      46.27

      31
      32
      3.622
      1
      0
      0
      20.37

      32
      33
      40.727
      1
      1
      1
      229.00

```python

```

```python
#if you wish to print the entire data, then use the following
const_maint_data
```

      0
      1
      2
      3
      4
      5

      0
      1
      6.068
      1
      0
      0
      16.60

      1
      2
      4.602
      1
      0
      0
      20.37

      2
      3
      3.282
      1
      0
      0
      20.37

      3
      4
      2.192
      1
      1
      0
      20.37

      4
      5
      2.572
      1
      1
      0
      20.37

      5
      6
      4.142
      1
      0
      0
      20.37

      6
      7
      5.321
      1
      1
      1
      21.00

      7
      8
      4.421
      1
      1
      1
      21.00

      8
      9
      5.301
      1
      1
      1
      21.00

      9
      10
      9.679
      1
      0
      0
      27.01

      10
      11
      11.997
      1
      0
      1
      31.00

      11
      12
      7.757
      1
      0
      0
      31.00

      12
      13
      12.597
      1
      0
      0
      31.00

      13
      14
      13.045
      1
      0
      1
      33.20

      14
      15
      2.471
      0
      0
      0
      45.00

      15
      16
      6.689
      0
      0
      0
      46.27

      16
      17
      12.139
      1
      0
      0
      46.27

      17
      18
      10.859
      1
      0
      0
      46.27

      18
      19
      7.119
      0
      1
      0
      68.42

      19
      20
      3.728
      0
      1
      0
      83.70

      20
      21
      46.057
      1
      1
      0
      218.00

      21
      22
      49.847
      1
      1
      0
      218.00

      22
      23
      61.671
      1
      1
      0
      335.00

      23
      24
      56.167
      1
      0
      1
      228.00

      24
      25
      9.721
      1
      0
      1
      22.60

      25
      26
      8.535
      1
      0
      1
      21.70

      26
      27
      66.910
      1
      1
      1
      229.00

      27
      28
      11.101
      1
      0
      1
      22.60

      28
      29
      12.511
      1
      0
      1
      22.60

      29
      30
      11.397
      1
      0
      1
      31.00

      30
      31
      4.069
      0
      0
      0
      46.27

      31
      32
      3.622
      1
      0
      0
      20.37

      32
      33
      40.727
      1
      1
      1
      229.00

What do the numbers in the above data mean? Let us understand the data by seeing its description.

```python
import urllib.request  #this package is useful for accessing text files over internet
const_maint_data_description_file = urllib.request.urlopen('http://users.stat.ufl.edu/~winner/data/const_maint.txt')

for line in const_maint_data_description_file:
  decoded_line = line.decode("utf-8")
  print(decoded_line)
```

    Dataset:  const_maint.dat

    Source: D.J. Edwards, G.D. Holt, F.C. Harris (2000). "A Comparative

    Analysis Between the Multilayer Perceptron "Neural Network" and Multiple

    Regression Analysis for Predicting Construction Plant Maintenance Costs,"

    Journal of Quality in Maintenance Engineering, Vol. 6, #1, pp. 45-61

    Description: Regression Analysis relating construction plant maintenance costs

    to: machine weight, and indicators attitude to used oil analysis,

    type of industry, and type of machine.

    Variables/Columns

    Machine ID   7-8

    Cost (pounds)  10-16

    Type of Industry    24    /* 1=opencast coal   0=opencast slate */

    Type of machine   32     /* 1=front shovel   0=backacter  */

    Company attitude to used oil analysis   40  /* 1=regular us   0=not  */

    Machine weight  (tonnes)  42-48

Having known the data description, let us insert the descriptions into the data now.

```python
const_maint_data.columns = ['Machine ID', 'Cost (pounds)', 'Type of Industry','Type of machine','Company attitude to used oil analysis','Machine weight (tonnes)']

#check by printing the data again
const_maint_data.head()
```

      Machine ID
      Cost (pounds)
      Type of Industry
      Type of machine
      Company attitude to used oil analysis
      Machine weight (tonnes)

      0
      1
      6.068
      1
      0
      0
      16.60

      1
      2
      4.602
      1
      0
      0
      20.37

      2
      3
      3.282
      1
      0
      0
      20.37

      3
      4
      2.192
      1
      1
      0
      20.37

      4
      5
      2.572
      1
      1
      0
      20.37

$\Large{\text{Let's now split the data into train and test set}}$.

```python
from sklearn.model_selection import train_test_split
const_maint_data_train,const_maint_data_test = train_test_split(const_maint_data,test_size=0.2, random_state = 1)
print ('size of train set:',len(const_maint_data_train.index))
print ('Training data:')
const_maint_data_train.head()
```

    size of train set: 26
    Training data:

      Machine ID
      Cost (pounds)
      Type of Industry
      Type of machine
      Company attitude to used oil analysis
      Machine weight (tonnes)

      25
      26
      8.535
      1
      0
      1
      21.70

      24
      25
      9.721
      1
      0
      1
      22.60

      30
      31
      4.069
      0
      0
      0
      46.27

      17
      18
      10.859
      1
      0
      0
      46.27

      22
      23
      61.671
      1
      1
      0
      335.00

$\large{\text{How to estimate the parameters } \beta_j, j=0,1,\ldots,d?}$

One way to estimate the values of $\beta_j, j=0,1,\ldots,d$ is by minimizing the sum of squared errors given by:
$
\begin{align}
\min \sum_{i=1}^{n} (e^i)^2
\end{align}
$
which can be equivalently written as the following optimization problem:
$
\begin{align}
\min_{\beta_0,\beta_1,\ldots,\beta_d} \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2.
\end{align}
$

$\textbf{Note}$ that in the high dimensional case as well, the optimization problem is called the ordinary least squares (OLS) problem, and the term $\sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2$ is called the OLS objective function and we will denote the OLS objective by:

$
\begin{align}
L(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2.
\end{align}
$

$\Large{\text{Solving the OLS optimization problem}}:$

Assuming the responses $y^1,y^2,\ldots,y^n$ are present in a $n \times 1$ matrix represented as:

$
\mathbf{y}=\begin{bmatrix}
y^1 \\ y^2 \\ \vdots \\ y^n
\end{bmatrix}
$

and the predictor variables $\mathbf{x}^1,\mathbf{x}^2,\ldots,\mathbf{x}^d$ are placed in a $n\times (d+1)$ matrix represented as:

$
\mathbf{X}=\begin{bmatrix}
\leftarrow & \mathbf{x}^1 & \rightarrow & 1 \\
\leftarrow & \mathbf{x}^2 & \rightarrow & 1 \\
\leftarrow & \ldots & \rightarrow & 1  \\
\leftarrow & \mathbf{x}^n & \rightarrow & 1 \\
\end{bmatrix}
=
\begin{bmatrix}
{x}_1^1 & {x}_2^1 &\ldots & {x}_d^1 & 1 \\
{x}_1^2 & {x}_2^2 &\ldots & {x}_d^2 & 1 \\
\vdots & \vdots & \ldots & \vdots \\
{x}_1^n & {x}_2^n &\ldots & {x}_d^n & 1
\end{bmatrix}
$
where note that the last column of matrix $\mathbf{X}$ contains a column of all ones. This column is useful to incorporate the effect of $\beta_0$ parameter in $\mathbf{X}$ matrix.

Similarly assume that
 we can write the coefficients $\beta_0,\beta_1,\ldots,\beta_d$ as a $(d+1)\times 1$ matrix represented as:

$
\mathbf{\beta} = \begin{bmatrix}
\beta_1 \\ \beta_2 \\ \vdots \\ \beta_d \\ \beta_0
\end{bmatrix}.
$

Now we can write the objective function as:

$
L(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2.
$

To solve

$
\min_\beta L(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2,
$
we find the gradient with respect to $\beta$ and equate to zero.

Thus we get:

$
\begin{align}
\nabla_\beta L(\beta) &= \mathbf{0} \\
\implies -\mathbf{X}^\top \mathbf{y} + \mathbf{X}^\top\mathbf{X} \beta &= \mathbf{0} \\
\implies \beta &= (\mathbf{X}^\top\mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}.
\end{align}
$

Note that the closed form expression for $\beta$ is valid only when $(\mathbf{X}^\top\mathbf{X})^{-1}$ is invertible. Otherwise we need to solve the system given by:

$
\mathbf{X}^\top\mathbf{X} \beta = \mathbf{X}^\top \mathbf{y}
$
using a solver.

$\Large{\text{Computing } \beta \text{ for the airline costs data set}}:$

```python
#Let us now compute beta
n = len(const_maint_data_train.index) #number of data points in the data set
print('number of points in the training data set:',n)
```

    number of points in the training data set: 26

```python
#extract the predictor variable columns
cols = [2,3,4,5]
X_train_data = const_maint_data_train[const_maint_data_train.columns[cols]]

X_train_data
```

      Type of Industry
      Type of machine
      Company attitude to used oil analysis
      Machine weight (tonnes)

      25
      1
      0
      1
      21.70

      24
      1
      0
      1
      22.60

      30
      0
      0
      0
      46.27

      17
      1
      0
      0
      46.27

      22
      1
      1
      0
      335.00

      4
      1
      1
      0
      20.37

      2
      1
      0
      0
      20.37

      21
      1
      1
      0
      218.00

      23
      1
      0
      1
      228.00

      10
      1
      0
      1
      31.00

      29
      1
      0
      1
      31.00

      28
      1
      0
      1
      22.60

      18
      0
      1
      0
      68.42

      6
      1
      1
      1
      21.00

      13
      1
      0
      1
      33.20

      7
      1
      1
      1
      21.00

      32
      1
      1
      1
      229.00

      1
      1
      0
      0
      20.37

      16
      1
      0
      0
      46.27

      0
      1
      0
      0
      16.60

      15
      0
      0
      0
      46.27

      5
      1
      0
      0
      20.37

      11
      1
      0
      0
      31.00

      9
      1
      0
      0
      27.01

      8
      1
      1
      1
      21.00

      12
      1
      0
      0
      31.00

```python
#convert predictor variable columns into a numpy array

X_train_array = const_maint_data_train[const_maint_data_train.columns[cols]].to_numpy()
X_train_array
```

    array([[  1.  ,   0.  ,   1.  ,  21.7 ],
           [  1.  ,   0.  ,   1.  ,  22.6 ],
           [  0.  ,   0.  ,   0.  ,  46.27],
           [  1.  ,   0.  ,   0.  ,  46.27],
           [  1.  ,   1.  ,   0.  , 335.  ],
           [  1.  ,   1.  ,   0.  ,  20.37],
           [  1.  ,   0.  ,   0.  ,  20.37],
           [  1.  ,   1.  ,   0.  , 218.  ],
           [  1.  ,   0.  ,   1.  , 228.  ],
           [  1.  ,   0.  ,   1.  ,  31.  ],
           [  1.  ,   0.  ,   1.  ,  31.  ],
           [  1.  ,   0.  ,   1.  ,  22.6 ],
           [  0.  ,   1.  ,   0.  ,  68.42],
           [  1.  ,   1.  ,   1.  ,  21.  ],
           [  1.  ,   0.  ,   1.  ,  33.2 ],
           [  1.  ,   1.  ,   1.  ,  21.  ],
           [  1.  ,   1.  ,   1.  , 229.  ],
           [  1.  ,   0.  ,   0.  ,  20.37],
           [  1.  ,   0.  ,   0.  ,  46.27],
           [  1.  ,   0.  ,   0.  ,  16.6 ],
           [  0.  ,   0.  ,   0.  ,  46.27],
           [  1.  ,   0.  ,   0.  ,  20.37],
           [  1.  ,   0.  ,   0.  ,  31.  ],
           [  1.  ,   0.  ,   0.  ,  27.01],
           [  1.  ,   1.  ,   1.  ,  21.  ],
           [  1.  ,   0.  ,   0.  ,  31.  ]])

```python
import numpy as np
#add a column of ones to X_array
X_train = np.hstack((X_train_array, np.ones((X_train_array.shape[0], 1), dtype=X_train_array.dtype)))

X_train
```

    array([[  1.  ,   0.  ,   1.  ,  21.7 ,   1.  ],
           [  1.  ,   0.  ,   1.  ,  22.6 ,   1.  ],
           [  0.  ,   0.  ,   0.  ,  46.27,   1.  ],
           [  1.  ,   0.  ,   0.  ,  46.27,   1.  ],
           [  1.  ,   1.  ,   0.  , 335.  ,   1.  ],
           [  1.  ,   1.  ,   0.  ,  20.37,   1.  ],
           [  1.  ,   0.  ,   0.  ,  20.37,   1.  ],
           [  1.  ,   1.  ,   0.  , 218.  ,   1.  ],
           [  1.  ,   0.  ,   1.  , 228.  ,   1.  ],
           [  1.  ,   0.  ,   1.  ,  31.  ,   1.  ],
           [  1.  ,   0.  ,   1.  ,  31.  ,   1.  ],
           [  1.  ,   0.  ,   1.  ,  22.6 ,   1.  ],
           [  0.  ,   1.  ,   0.  ,  68.42,   1.  ],
           [  1.  ,   1.  ,   1.  ,  21.  ,   1.  ],
           [  1.  ,   0.  ,   1.  ,  33.2 ,   1.  ],
           [  1.  ,   1.  ,   1.  ,  21.  ,   1.  ],
           [  1.  ,   1.  ,   1.  , 229.  ,   1.  ],
           [  1.  ,   0.  ,   0.  ,  20.37,   1.  ],
           [  1.  ,   0.  ,   0.  ,  46.27,   1.  ],
           [  1.  ,   0.  ,   0.  ,  16.6 ,   1.  ],
           [  0.  ,   0.  ,   0.  ,  46.27,   1.  ],
           [  1.  ,   0.  ,   0.  ,  20.37,   1.  ],
           [  1.  ,   0.  ,   0.  ,  31.  ,   1.  ],
           [  1.  ,   0.  ,   0.  ,  27.01,   1.  ],
           [  1.  ,   1.  ,   1.  ,  21.  ,   1.  ],
           [  1.  ,   0.  ,   0.  ,  31.  ,   1.  ]])

```python
#convert predictor variable columns into a numpy array

response_cols = [1]
y_train = const_maint_data_train[const_maint_data_train.columns[response_cols]].to_numpy()
y_train

```

    array([[ 8.535],
           [ 9.721],
           [ 4.069],
           [10.859],
           [61.671],
           [ 2.572],
           [ 3.282],
           [49.847],
           [56.167],
           [11.997],
           [11.397],
           [12.511],
           [ 7.119],
           [ 5.321],
           [13.045],
           [ 4.421],
           [40.727],
           [ 4.602],
           [12.139],
           [ 6.068],
           [ 6.689],
           [ 4.142],
           [ 7.757],
           [ 9.679],
           [ 5.301],
           [12.597]])

```python
#Now we are ready for finding beta
#print('X shape:',X.shape,'y shape:',y.shape)

XTX = np.matmul(np.transpose(X_train),X_train)
print('XTX shape:',XTX.shape)

Xy = np.matmul(np.transpose(X_train),y_train)
#print('Xy shape:',Xy.shape)

#solving a system of linear equations Ax=b, can be done using np.linalg.solve(A,b)

beta =np.linalg.solve(XTX,Xy)
#beta = np.matmul(np.linalg.inv(XTX),Xy)

print('beta',beta)

```

    XTX shape: (5, 5)
    beta [[ 5.888155  ]
     [-4.81296515]
     [ 2.89210439]
     [ 0.20253284]
     [-3.30324027]]

```python
print(np.linalg.matrix_rank(XTX))
XTX_inv = np.linalg.inv(XTX)

#print(np.matmul(XTX,XTX_inv))
print ('XTX*XTX_inv =')
print (np.matmul(XTX,XTX_inv))
is_identity = np.all(np.matmul(XTX,XTX_inv)==np.identity(XTX.shape[0]))
print(is_identity)
```

    5
    XTX*XTX_inv =
    [[ 1.00000000e+00 -5.55111512e-17  4.44956572e-16  2.46655994e-18
      -3.33066907e-16]
     [ 0.00000000e+00  1.00000000e+00 -3.81639165e-17  2.60208521e-18
      -8.88178420e-16]
     [-6.66133815e-16  5.55111512e-17  1.00000000e+00  1.70761842e-18
      -7.77156117e-16]
     [ 4.48349785e-14 -8.57011096e-16 -1.19145303e-14  1.00000000e+00
      -6.86421727e-14]
     [-3.99680289e-15 -3.33066907e-16  2.96637714e-16  2.22261445e-18
       1.00000000e+00]]
    False

$\textbf{Note:}$ The matrix $X^\top X$ is full-rank

Let us compute the condition number of $X^\top X$.

```python
#condition number of XTX
print(np.linalg.cond(XTX))
```

    212939.41237197566

$\textbf{Adding $\ell_2$ regularizer to improve the conditioning of the matrix}$:

We shall motivate the use of $\ell_2$ regularizer to improve the conditioning of the matrix.

Instead of minimizing the original OLS objective
$
\begin{align}
L_{OLS}(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2.
\end{align}
$

we shall now minimize the OLS objective added with a scaled $\ell_2$ regularizer.

The $\ell_2$ regularizer in $\beta \in {\mathbb{R}}^{d+1}$ is defined as the squared $\ell_2$ norm of $\beta$:

$
\begin{align}
\|\beta\|_2^2 = \sum_{i=1}^{d+1} \beta_i^2.
\end{align}
$

Hence we shall now solve:
$
\begin{align}
L_{\text{ridge}}(\beta_0, \beta_1,\ldots,\beta_d) = \sum_{i=1}^{n} [y^i - ( \beta_0 + \sum_{j=1}^{d} \beta_j x_j^i) ]^2 + \lambda \sum_{i=1}^{d+1} \beta_i^2.
\end{align}
$

The above objective function is called $\textbf{ridge}$ regression objective. $\lambda>0$ is a regularization hyperparameter.

Now we can write the objective function as:

$
L_{\text{ridge}}(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2 + \lambda \|\beta\|_2^2.
$

To solve

$
\min_\beta L_{\text{ridge}}(\beta) = \|\mathbf{y} - \mathbf{X}\mathbf{\beta} \|_F^2 + \lambda \|\beta\|_2^2,
$
we find the gradient with respect to $\beta$ and equate to zero.

Thus we get:

$
\begin{align}
\nabla_\beta L_{\text{ridge}}(\beta) &= \mathbf{0} \\
\implies -\mathbf{X}^\top \mathbf{y} + \mathbf{X}^\top\mathbf{X} \beta + \lambda I\beta &= \mathbf{0} \\
\implies \beta &= (\mathbf{X}^\top\mathbf{X}+\lambda I)^{-1} \mathbf{X}^\top \mathbf{y}.
\end{align}
$

Note that the closed form expression for $\beta$ is always valid since $(\mathbf{X}^\top\mathbf{X}+\lambda I)$ is invertible.

With $\lambda \in \{10^{-5},10^{-4}, 10^{-3},10^{-2},0.1, 1, 10, 100\}$ let us check the condition number of $(X^\top X+\lambda I)$.

```python
lambdas = np.array([1e-5,1e-4,1e-3,1e-2,1e-1,1,10,100])
I = np.identity(XTX.shape[0])

for lambda_ in lambdas:
  print('lambda:',lambda_,'cond(XTX+lambda I):', np.linalg.cond(XTX+lambda_*I))
```

    lambda: 1e-05 cond(XTX+lambda I): 212937.83725164508
    lambda: 0.0001 cond(XTX+lambda I): 212923.66221737218
    lambda: 0.001 cond(XTX+lambda I): 212782.01560408747
    lambda: 0.01 cond(XTX+lambda I): 211375.84630231213
    lambda: 0.1 cond(XTX+lambda I): 198272.98653555237
    lambda: 1.0 cond(XTX+lambda I): 122399.62483265983
    lambda: 10.0 cond(XTX+lambda I): 25359.49153277391
    lambda: 100.0 cond(XTX+lambda I): 2841.267762821886

With $\lambda \in \{10^{-5},10^{-4}, 10^{-3},10^{-2},0.1, 1, 10, 100\}$ , let us now solve for $\beta$ and check the solutions.

```python
import scipy.linalg
#Now we are ready for finding beta
#print('X shape:',X.shape,'y shape:',y.shape)

betas = []
for lambda_ in lambdas:
  beta =scipy.linalg.solve(XTX+lambda_*I,Xy)
  print('lambda:',lambda_)
  print('beta:',beta)
  print('***************')
  betas.append(beta)
```

    lambda: 1e-05
    beta: [[ 5.88812298]
     [-4.81295663]
     [ 2.8921031 ]
     [ 0.20253281]
     [-3.30321107]]
    ***************
    lambda: 0.0001
    beta: [[ 5.88783488]
     [-4.8128799 ]
     [ 2.89209145]
     [ 0.20253258]
     [-3.30294832]]
    ***************
    lambda: 0.001
    beta: [[ 5.88495591]
     [-4.81211266]
     [ 2.89197478]
     [ 0.20253023]
     [-3.30032267]]
    ***************
    lambda: 0.01
    beta: [[ 5.85636863]
     [-4.80443798]
     [ 2.8907866 ]
     [ 0.2025068 ]
     [-3.27424986]]
    ***************
    lambda: 0.1
    beta: [[ 5.58935359]
     [-4.72760773]
     [ 2.87696783]
     [ 0.2022772 ]
     [-3.03064336]]
    ***************
    lambda: 1.0
    beta: [[ 4.00917372]
     [-4.01600582]
     [ 2.65702969]
     [ 0.20041244]
     [-1.58905163]]
    ***************
    lambda: 10.0
    beta: [[ 1.64475018]
     [-1.44856647]
     [ 1.39842655]
     [ 0.19638749]
     [ 0.32019518]]
    ***************
    lambda: 100.0
    beta: [[ 0.41253984]
     [-0.16163301]
     [ 0.31448326]
     [ 0.20154865]
     [ 0.26069268]]
    ***************

$\Large{\text{Residual Plot}}$

Sometimes it would be useful to plot the error (or) $\textbf{residual}$ $e^i$ versus the fitted values $\hat{y}^i=\beta_0 + \sum_{j=1}^{d} \beta_j x^i_j$.

```python
beta_idx = 0
for lambda_ in lambdas:
  beta = betas[beta_idx]
  beta_idx += 1
  xlabel = 'Cost lambda='+str(lambda_)
  e_i_residuals_list = []

  for i in range(n):
    x_i = X_train[i,:] #access i-th row of X
    y_i = y_train[i] #access i-th row of y

    y_pred_i = np.dot(x_i,beta) #compute the prediction obtained using the regression coefficients
    e_i = y_i - y_pred_i #compute the difference between the actual observation y_i and prediction y_pred_i

    e_i_residuals_list.append(e_i) #append the value of e_i to the list

  #plot the residuals e_i against the actual observations y_i
  plt.scatter(const_maint_data_train['Cost (pounds)'], e_i_residuals_list,color='r')
  plt.title("Residual plot")
  plt.xlabel(xlabel)
  plt.ylabel("Residuals")
  plt.grid()
  plt.show()
```

Note that the residual plot helps to check the variance in the errors $e^i$.

$\Large{\text{Coefficient of multiple determination for high dimensions}}$

Note that the description of coefficient of multiple determination for high dimensions is essentially the same as that for single dimension case.

Consider the term

$
\begin{align}
\Sigma_{YY} = \sum_{i=1}^{n} (y^i - \bar{y})^2.
\end{align}
$
This term $\Sigma_{YY}$ measures the $\textbf{observed total variation of the response}$, without considering the effect of predictor variables.

When we have access to a predictor variable, then the $\textbf{unexplained variation of response variable given the predictor variable}$ is given by the sum of squared residuals:

$
\begin{align}
L = \sum_{i=1}^{n} (e^i)^2.
\end{align}
$

Define the term:

$
\begin{align}
\text{SSD} = \Sigma_{YY}- L
\end{align}
$

as the difference between observed total variation of the response and the unexplained variation given the predictor. Hence this can be considered as $\textbf{variability explained by the regression on the predictor variable}$.

Assuming $\Sigma_{YY}\neq 0$, consider:

$
\begin{align}
\frac{\text{SSD}}{\Sigma_{YY}} = 1- \frac{L}{\Sigma_{YY}}.
\end{align}
$
Here the term $\frac{\text{SSD}}{\Sigma_{YY}}$ denotes the proportion of observed variability in the response variable which is explained by the regression on the predictor variable. From the equality, this proportion of observed variability given the predictor is simply $1$ minus the proportion of unexplained variability.

This term $\frac{\text{SSD}}{\Sigma_{YY}}$ denoting the proportion of observed variability which is explained by the predictor variable has a special name called $\textbf{coefficient of multiple determination}$ and is denoted by $R^2$.

It is known that $0\leq R^2 \leq 1$.

Let us compute $R^2$.

```python
#We shall now compute R^2 for airline costs data

#first we shall compute y_bar
y_bar = 0
for i in range(n): #access each row from the training data set
  y_i = float(const_maint_data_train.iat[i,1]) #access element at TOC column
  y_bar += y_i

y_bar /= n
print('y_bar',y_bar)

#Next we shall compute sigma_yy, which is the variance of y
sigma_yy = 0
for i in range(n): #access each row from the data set
  y_i = float(const_maint_data_train.iat[i,1]) #access element at TOC column
  sigma_yy += (y_i-y_bar)**2

print('sigma_yy:', sigma_yy)

#now we can compute sum of squared residuals
beta_idx = 0
for lambda_ in lambdas:
  beta = betas[beta_idx]
  beta_idx+=1
  sum_sq_residuals = 0
  for i in range(n): #access each row from the data set
    x_i = X_train[i,:] #access i-th row of X
    y_i = y_train[i] #access i-th row of y

    y_pred_i = np.dot(x_i,beta) #compute the prediction obtained using the regression coefficients
    e_i = y_i - y_pred_i #compute the difference between the actual observation y_i and prediction y_pred_i

    sum_sq_residuals += (e_i)**2

  print('lambda:',lambda_)
  print('sum of squared residuals:', sum_sq_residuals)

  #Then we will compute the R^2 quantity
  R_sq = 1-sum_sq_residuals/sigma_yy
  print('R^2:',R_sq)
  print('*********************')
```

    y_bar 14.70134615384615
    sigma_yy: 7108.239977884616
    lambda: 1e-05
    sum of squared residuals: [196.61263052]
    R^2: [0.97234018]
    *********************
    lambda: 0.0001
    sum of squared residuals: [196.61263085]
    R^2: [0.97234018]
    *********************
    lambda: 0.001
    sum of squared residuals: [196.61266345]
    R^2: [0.97234018]
    *********************
    lambda: 0.01
    sum of squared residuals: [196.61588909]
    R^2: [0.97233972]
    *********************
    lambda: 0.1
    sum of squared residuals: [196.90696914]
    R^2: [0.97229877]
    *********************
    lambda: 1.0
    sum of squared residuals: [210.69534587]
    R^2: [0.970359]
    *********************
    lambda: 10.0
    sum of squared residuals: [324.43958074]
    R^2: [0.95435726]
    *********************
    lambda: 100.0
    sum of squared residuals: [485.85703305]
    R^2: [0.93164876]
    *********************

Here we observe that $R^2$ value does not change at all from $\lambda = 10^{-5}$ to $\lambda = 0.001$. There is a transition (decrease) in $R^2$ value from $\lambda = 0.001$ to $\lambda = 0.01$. Therefore, we choose the $\beta$ values corresponding to $\lambda = 0.001$ for predicting test accuracy.

```python
## extracting beta value associated with lambdaa = 0.001
beta_train = betas[2] ## index 2 corresponds to lambda = 0.001
print ('best beta')
print (beta_train)
```

    best beta
    [[ 5.88495591]
     [-4.81211266]
     [ 2.89197478]
     [ 0.20253023]
     [-3.30032267]]

$\Large{\text{Checking model performance on test data}}$

Let's first prepare the test data into numpy array

```python
#extract the predictor variable columns
cols = [2,3,4,5]
X_test_data = const_maint_data_test[const_maint_data_test.columns[cols]]

X_test_data
```

      Type of Industry
      Type of machine
      Company attitude to used oil analysis
      Machine weight (tonnes)

      14
      0
      0
      0
      45.00

      19
      0
      1
      0
      83.70

      3
      1
      1
      0
      20.37

      27
      1
      0
      1
      22.60

      31
      1
      0
      0
      20.37

      26
      1
      1
      1
      229.00

      20
      1
      1
      0
      218.00

```python
#convert predictor variable columns into a numpy array

X_test_array = const_maint_data_test[const_maint_data_test.columns[cols]].to_numpy()
X_test_array
```

    array([[  0.  ,   0.  ,   0.  ,  45.  ],
           [  0.  ,   1.  ,   0.  ,  83.7 ],
           [  1.  ,   1.  ,   0.  ,  20.37],
           [  1.  ,   0.  ,   1.  ,  22.6 ],
           [  1.  ,   0.  ,   0.  ,  20.37],
           [  1.  ,   1.  ,   1.  , 229.  ],
           [  1.  ,   1.  ,   0.  , 218.  ]])

```python
#add a column of ones to X_train_array
X_test = np.hstack((X_test_array, np.ones((X_test_array.shape[0], 1), dtype=X_test_array.dtype)))

X_test
```

    array([[  0.  ,   0.  ,   0.  ,  45.  ,   1.  ],
           [  0.  ,   1.  ,   0.  ,  83.7 ,   1.  ],
           [  1.  ,   1.  ,   0.  ,  20.37,   1.  ],
           [  1.  ,   0.  ,   1.  ,  22.6 ,   1.  ],
           [  1.  ,   0.  ,   0.  ,  20.37,   1.  ],
           [  1.  ,   1.  ,   1.  , 229.  ,   1.  ],
           [  1.  ,   1.  ,   0.  , 218.  ,   1.  ]])

```python
## extracting response variables
response_cols = [1]
y_test_data = const_maint_data_test[const_maint_data_test.columns[response_cols]]
y_test_data
```

      Cost (pounds)

      14
      2.471

      19
      3.728

      3
      2.192

      27
      11.101

      31
      3.622

      26
      66.910

      20
      46.057

```python
response_cols = [1]
y_test = const_maint_data_test[const_maint_data_test.columns[response_cols]].to_numpy()
y_test
```

    array([[ 2.471],
           [ 3.728],
           [ 2.192],
           [11.101],
           [ 3.622],
           [66.91 ],
           [46.057]])

```python
## we first compute size of test data
n_test = len(const_maint_data_test.index) #number of data points in the test data
# print('number of data points in the test data:',n_test)

e_i_residuals_list_test = [] ## stores residuals on test data
for i in range(n_test):
  x_i = X_test[i,:] #access i-th row of X_test
  y_i = y_test[i] #access i-th row of y_test

  y_pred_i = np.dot(x_i,beta_train) # compute the prediction obtained using the regression coefficients
  e_i_test = y_i - y_pred_i #compute the difference between the actual observation y_i and prediction y_pred_i

  e_i_residuals_list_test.append(e_i_test) #append the value of e_i_test to the list

plt.scatter(const_maint_data_test['Cost (pounds)'], e_i_residuals_list_test,color='g')
plt.title("Residual plot of test data")
plt.xlabel("Cost")
plt.ylabel("Residuals")
plt.grid()
plt.show()
```

```python
#We shall now compute R^2 for training airline costs data

#first we shall compute y_bar
y_bar_test = 0
for i in range(n_test): #access each row from the test data set
  y_test_i = float(const_maint_data_test.iat[i,1]) #access element at TOC column
  y_bar_test += y_test_i

y_bar_test /= n_test
print('y_bar_test',y_bar_test)

#Next we shall compute sigma_yy, which is the variance of y_test
sigma_yy_test = 0
for i in range(n_test): #access each row from the training data set
  y_i = float(const_maint_data_test.iat[i,1]) #access element at TOC column
  sigma_yy_test += (y_i-y_bar_test)**2

print('sigma_yy_test:', sigma_yy_test)

#now we can compute sum of squared residuals
sum_sq_residuals_test = 0
for i in range(n_test): #access each row from the training data set
  x_i = X_test[i,:] #access i-th row of X_test
  y_i = y_test[i] #access i-th row of y_test

  y_pred_i = np.dot(x_i,beta_train) #compute the prediction obtained using the regression coefficients
  e_i = y_i - y_pred_i #compute the difference between the actual observation y_i and prediction y_pred_i

  sum_sq_residuals_test += (e_i)**2

print('sum of squared residuals:', sum_sq_residuals_test)
rmse = np.sqrt(sum_sq_residuals_test/n_test)
print('Root Mean Squarred Error (RMSE):', rmse)

#Then we will compute the R^2 quantity
R_sq_test = 1 - sum_sq_residuals_test/sigma_yy_test
print('R^2 score:',R_sq_test)
```

    y_bar_test 19.440142857142856
    sigma_yy_test: 4113.921042857142
    sum of squared residuals: [459.76024863]
    Root Mean Squarred Error (RMSE): [8.10432203]
    R^2 score: [0.88824281]

$\Large{\text{References:}}$

1.   Olive J. David, $\textit{Linear Regression}$. Springer International Publishing,  2017.
2.   Sanford Weisberg. $\textit{Applied Linear Regression}$. John Wiley $\&$ Sons, Inc., Hoboken, New Jersey, 4-th Edition, 2014.

```python

```


## Машинное обучение
### Семинар 13. ЕМ-алгоритм

9 декабря 2021

Будем решать задачу восставновления картинки лица по набору зашумленных картинок (взято с курса deep bayes 2018 https://github.com/bayesgroup/deepbayes-2018).

У вас есть $K$ фотографий, поврежденных электромагнитным шумом. Известно, что на каждом фото есть лицо в неизвестно где начинающейся прямоугольной области ширины $w$ и фон, одинаковый для всех фотографий.

```python
from matplotlib import pyplot as plt
import numpy as np
```

```python
import zipfile
with zipfile.ZipFile('data_em.zip', 'r') as zip_ref:
    zip_ref.extractall('.')
```

```python
DATA_FILE = "data_em"
w = 73 # face_width
```

```python
X = np.load(DATA_FILE)
```

```python
X.shape # H, W, K
```

    (100, 200, 1000)

```python
plt.imshow(X[:, :, 7], cmap="Greys_r")
plt.axis("off")
```

```python
tH, tW, tw, tK = 2, 3, 1, 2
tX = np.arange(tH*tW*tK).reshape(tH, tW, tK)
tF = np.arange(tH*tw).reshape(tH, tw)
tB = np.arange(tH*tW).reshape(tH, tW)
ts = 0.1
ta = np.arange(1, (tW-tw+1)+1)
ta = ta / ta.sum()
tq = np.arange(1, (tW-tw+1)*tK+1).reshape(tW-tw+1, tK)
tq = tq / tq.sum(axis=0)[np.newaxis, :]
```

1. **Реализуйте calculate_log_probability**

Для $k$-й картини $X_k$ и некоторой позиции $d_k$:
    $$ p(X_k  \mid d_k,\,F,\,B,\, std) = \prod\limits_{ij}\begin{cases}
        \mathcal{N}(X_k[i,j]\mid F[i,\,j-d_k],\, std^2),
        & \text{if}\, (i,j)\in faceArea(d_k)\\
        \mathcal{N}(X_k[i,j]\mid B[i,j],\, std^2), & \text{else}
    \end{cases}
    $$
Замечания:
* $faceArea(d_k) = \{[i, j]| d_k \leq j \leq d_k + w - 1 \}$
* Априорное распределение задаётся обучаемым вектором $a \in \mathbb{R}^{W-w+1}$: $$p(d_k \mid a) = a[d_k],\ \sum\limits_j a[j] = 1$$
* Итоговая вероятностная модель: $$ p(X, d  \mid F,\,B,\,std,\,a) = \prod\limits_k p(X_k \mid d_k,\,F,\,B,\,std) p(d_k \mid a)$$
* Не забудьте про логарифм!
* `scipy.stats.norm` может вам пригодиться

```python
import scipy.stats
```

```python
def calculate_log_probability(X, F, B, s):
    """
    Calculates log p(X_k|d_k, F, B, s) for all images X_k in X and
    all possible face position d_k.

    Parameters
    ----------
    X : array, shape (H, W, K)
        K images of size H x W.
    F : array, shape (H, w)
        Estimate of prankster's face.
    B : array, shape (H, W)
        Estimate of background.
    s : float
        Estimate of standard deviation of Gaussian noise.

    Returns
    -------
    ll : array, shape(W-w+1, K)
        ll[dw, k] - log-likelihood of observing image X_k given
        that the prankster's face F is located at position dw
    """
    H, W, K = X.shape
    _, w = F.shape
    # your code here
    ll = np.zeros((W-w+1, K))
    for dw in range(W-w+1):
        combined = np.copy(B)
        combined[:, dw:dw+w] = F
        d_combined = X - np.expand_dims(combined, 2)
        ll[dw] = scipy.stats.norm(0, s).logpdf(d_combined).sum(axis=(0,1))
    return ll
```

```python
# run this cell to test your implementation
expected = np.array([[-3541.69812064, -5541.69812064],
       [-4541.69812064, -6741.69812064],
       [-6141.69812064, -8541.69812064]])
actual = calculate_log_probability(tX, tF, tB, ts)
assert np.allclose(actual, expected)
print("OK")
```

    OK

2. **Реализуйте calculate_lower_bound**
\begin{equation}\mathscr{L}(q, \,F, \,B,\, s,\, a) = \sum_k \biggl (\mathbb{E} _ {q( d_k)}\bigl ( \log p(  X_{k}  \mid {d}_{k} , \,F,\,B,\,s) +
    \log p( d_k  \mid a)\bigr) - \mathbb{E} _ {q( d_k)} \log q( d_k)\biggr) \end{equation}

Замечания
  * Используйте calculate_log_probability!
  * Обратите внимание, что $q( d_k)$ и $p( d_k  \mid a)$ дискретны. Например, $P(d_k=i \mid a) = a[i]$.

```python
def calculate_lower_bound(X, F, B, s, a, q):
    """
    Calculates the lower bound L(q, F, B, s, a) for
    the marginal log likelihood.

    Parameters
    ----------
    X : array, shape (H, W, K)
        K images of size H x W.
    F : array, shape (H, w)
        Estimate of prankster's face.
    B : array, shape (H, W)
        Estimate of background.
    s : float
        Estimate of standard deviation of Gaussian noise.
    a : array, shape (W-w+1)
        Estimate of prior on position of face in any image.
    q : array
        q[dw, k] - estimate of posterior
                   of position dw
                   of prankster's face given image Xk

    Returns
    -------
    L : float
        The lower bound L(q, F, B, s, a)
        for the marginal log likelihood.
    """
    # your code here
    return (q * (calculate_log_probability(X,F,B,s) + np.expand_dims(np.log(a), 1) -  np.log(q))).sum()
```

```python
calculate_lower_bound(tX, tF, tB, ts, ta, tq)
```

    -12761.187501001436

```python
# run this cell to test your implementation
expected = -12761.1875
actual = calculate_lower_bound(tX, tF, tB, ts, ta, tq)
assert np.allclose(actual, expected)
print("OK")
```

    OK

3. **Реализуем E шаг**
$$q(d_k) = p(d_k \mid X_k, \,F, \,B, \,s,\, a) =
\frac {p(  X_{k}  \mid {d}_{k} , \,F,\,B,\,s)\, p(d_k \mid a)}
{\sum_{d'_k} p(  X_{k}  \mid d'_k , \,F,\,B,\,s) \,p(d'_k \mid a)}$$

Замечания
 * Используйте calculate_log_probability!
 * Считайте в логарифмах, возведите в экспоненту в конце.
 * Рекомендется использовать следующее утверждение для выч. стабильности: $$\beta_i = \log{p_i(\dots)} \quad\rightarrow \quad
  \frac{e^{\beta_i}}{\sum\limits_k e^{\beta_k}} =
  \frac{e^{(\beta_i - \max_j \beta_j)}}{\sum\limits_k e^{(\beta_k- \max_j \beta_j)}}$$

```python
def run_e_step(X, F, B, s, a):
    """
    Given the current esitmate of the parameters, for each image Xk
    esitmates the probability p(d_k|X_k, F, B, s, a).

    Parameters
    ----------
    X : array, shape(H, W, K)
        K images of size H x W.
    F  : array_like, shape(H, w)
        Estimate of prankster's face.
    B : array shape(H, W)
        Estimate of background.
    s : float
        Estimate of standard deviation of Gaussian noise.
    a : array, shape(W-w+1)
        Estimate of prior on face position in any image.

    Returns
    -------
    q : array
        shape (W-w+1, K)
        q[dw, k] - estimate of posterior of position dw
        of prankster's face given image Xk
    """
    # your code here
    log_nom = calculate_log_probability(X,F,B,s) + np.expand_dims(np.log(a), 1)
    mx = log_nom.max(axis=0)
    nom = np.exp(log_nom - mx)
    return nom / nom.sum(axis=0)
```

```python
run_e_step(tX, tF, tB, ts, ta)
```

    array([[1., 1.],
           [0., 0.],
           [0., 0.]])

```python
# run this cell to test your implementation
expected = np.array([[ 1.,  1.],
                   [ 0.,  0.],
                   [ 0.,  0.]])
actual = run_e_step(tX, tF, tB, ts, ta)
assert np.allclose(actual, expected)
print("OK")
```

    OK

4. **Реализуйте M шаг**

Надо
\begin{equation}\mathscr{L}(q, \,F, \,B,\, s,\, a) = \sum_k \biggl (\mathbb{E} _ {q( d_k)}\bigl ( \log p(  X_{k}  \mid {d}_{k} , \,F,\,B,\,s) +
    \log p( d_k  \mid a)\bigr) - \mathbb{E} _ {q( d_k)} \log q( d_k)\biggr)\rightarrow \max\limits_{\theta, a} \end{equation}
После долгих вычислений получим:
$$a[j] = \frac{\sum_k q( d_k = j )}{\sum_{j'}  \sum_{k'} q( d_{k'} = j')}$$$$F[i, m] = \frac 1 K  \sum_k \sum_{d_k} q(d_k)\, X^k[i,\, m+d_k]$$\begin{equation}B[i, j] = \frac {\sum_k \sum_{ d_k:\, (i, \,j) \,\not\in faceArea(d_k)} q(d_k)\, X^k[i, j]}
        {\sum_k \sum_{d_k: \,(i, \,j)\, \not\in faceArea(d_k)} q(d_k)}\end{equation}\begin{equation}s^2 = \frac 1 {HWK}   \sum_k \sum_{d_k} q(d_k)
        \sum_{i,\, j}  (X^k[i, \,j] - Model^{d_k}[i, \,j])^2\end{equation}
где $Model^{d_k}[i, j]$ картинка из фона и лица, сдвинутого на $d_k$.

Замечания
* Обновляйте параметры в порядке: $a$, $F$, $B$, $s$.
* Используйте обновленный параметр для оценки следующего параметра.

```python
def run_m_step(X, q, w):
    """
    Estimates F, B, s, a given esitmate of posteriors defined by q.

    Parameters
    ----------
    X : array, shape (H, W, K)
        K images of size H x W.
    q  :
        q[dw, k] - estimate of posterior of position dw
                   of prankster's face given image Xk
    w : int
        Face mask width.

    Returns
    -------
    F : array, shape (H, w)
        Estimate of prankster's face.
    B : array, shape (H, W)
        Estimate of background.
    s : float
        Estimate of standard deviation of Gaussian noise.
    a : array, shape (W-w+1)
        Estimate of prior on position of face in any image.
    """
# your code here
    H, W, K = X.shape
    dw, _ = q.shape
    w = W - dw + 1

    a = q.sum(axis=1)/q.sum()

    F = np.zeros((H, w))
    for dk in range(dw):
        F += (q[dk] * X[:, dk:dk+w]).sum(axis=2) / K

    B = np.zeros((H, W))
    denom = np.zeros((H, W))
    for dk in range(dw):
        if dk > 0:
            denom[:, :dk] += q[dk].sum()
            B[:, :dk] += (q[dk] * X[:, :dk]).sum(axis=2)
        if dk + w < W:
            B[:, dk+w:] += (q[dk] * X[:, dk+w:]).sum(axis=2)
            denom[:, dk + w:] += q[dk].sum()
    B /= denom

    s2 = 0
    for dk in range(dw):
        model = np.copy(B)
        model[:, dk:dk+w] = F
        s2 += (q[dk] * ((X - np.expand_dims(model,2)) ** 2)).sum()
    s2 /= H * W * K

    return F, B, np.sqrt(s2), a

```

```python
run_m_step(tX, tq, tw)
```

    (array([[3.27777778],
            [9.27777778]]),
     array([[ 0.48387097,  2.5       ,  4.52941176],
            [ 6.48387097,  8.5       , 10.52941176]]),
     0.9486806229147358,
     array([0.13888889, 0.33333333, 0.52777778]))

```python
# run this cell to test your implementation
expected = [np.array([[ 3.27777778],
                      [ 9.27777778]]),
 np.array([[  0.48387097,   2.5       ,   4.52941176],
           [  6.48387097,   8.5       ,  10.52941176]]),
  0.94868,
 np.array([ 0.13888889,  0.33333333,  0.52777778])]
actual = run_m_step(tX, tq, tw)
for a, e in zip(actual, expected):
    assert np.allclose(a, e)
print("OK")
```

    OK

5. **Реализуйте EM алгоритм**

```python
def run_EM(X, w, F=None, B=None, s=None, a=None, tolerance=0.001,
           max_iter=50):
    """
    Runs EM loop until the likelihood of observing X given current
    estimate of parameters is idempotent as defined by a fixed
    tolerance.

    Parameters
    ----------
    X : array, shape (H, W, K)
        K images of size H x W.
    w : int
        Face mask width.
    F : array, shape (H, w), optional
        Initial estimate of prankster's face.
    B : array, shape (H, W), optional
        Initial estimate of background.
    s : float, optional
        Initial estimate of standard deviation of Gaussian noise.
    a : array, shape (W-w+1), optional
        Initial estimate of prior on position of face in any image.
    tolerance : float, optional
        Parameter for stopping criterion.
    max_iter  : int, optional
        Maximum number of iterations.

    Returns
    -------
    F, B, s, a : trained parameters.
    """
    # your code here
    H, W, N = X.shape
    if F is None:
        F = np.random.randint(0, 255, (H, w))
    if B is None:
        B = np.random.randint(0, 255, (H, W))
    if a is None:
        a = np.ones(W - w + 1)
        a /= np.sum(a)
    if s is None:
        s = np.random.rand()*64*64
    l_prev = -np.inf
    for it in range(max_iter):
        print(f"iteration = {it}")
        q = run_e_step(X, F, B, s, a)
        print("e")
        F, B, s, a = run_m_step(X, q, w)
        print("m")
        print(s)
        if it == max_iter - 1:
            print("no convergence")
            break
        l_cur = calculate_lower_bound(X, F, B, s, a, q)
        if l_cur - l_prev < tolerance:
            print(f"converged in {it} iterations {l_cur - l_prev}")
            break
        else:
            l_prev = l_cur

    return F, B, s, a
```

Расшифровываем картинку:

```python
def show(F, i=1, n=1):
    """
    shows face F at subplot i out of n
    """
    plt.subplot(1, n, i)
    plt.imshow(F, cmap="Greys_r")
    plt.axis("off")
```

```python
%%time
F, B, s, a = [None] * 4
lens = [50, 100, 300, 500, 1000]
iters = [5, 1, 1, 1, 1]
plt.figure(figsize=(20, 5))
for i, (l, it) in enumerate(zip(lens, iters)):
    F, B, s, a = run_EM(X[:, :, :l], w, F, B, s, a, max_iter=it)
    print(s)
    show(F, i+1, 5)
```

И фон:

```python
show(B)
```


```python
import matplotlib.pyplot as plt
import scipy.stats as st
import seaborn as sns
import pandas as pd
from scipy.stats import norm, uniform, expon, t
from scipy.integrate import quad
from sympy.solvers import solve
from sympy import Symbol
import numpy as np
from pandas import Series, DataFrame
```

```python
fuellungen = Series([71, 69, 67, 68, 73, 72, 71, 71, 68, 72, 69, 72])
```

```python
fuellungen.mean()
```

    70.25

```python
fuellungen_standardisiert = fuellungen
```