# Decision Tree Induction with scikit-learn

**CS5483 Data Warehousing and Data Mining**
___

```python
%reset -f
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import datasets, tree
# produce vector inline graphics
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')
```

## Decision Tree Induction

We first import the [*iris dataset*](https://en.wikipedia.org/wiki/Iris_flower_data_set) from [`sklearn.datasets` package](https://scikit-learn.org/stable/datasets/index.html)

```python
from sklearn import datasets
import pandas as pd

iris = datasets.load_iris()
```

Recall that the classification task is to train a model that can classify the spieces (*target*) automatically based on the lengths and widths of the petals and sepals (*input features*).

To build a decision tree, we simply create a tree using `DecisionTreeClassifier` from `sklearn.tree` and apply its method `fit` on the training set.

```python
clf_gini = tree.DecisionTreeClassifier(random_state=0).fit(iris.data, iris.target)
```

To display the decision tree, we can use the function `plot_tree` from `sklearn.tree`:

```python
# to make the tree look better
options = {'feature_names': iris.feature_names,
           'class_names': iris.target_names,
           'filled': True,
           'node_ids': True,
           'rounded': True,
           'fontsize': 6}

plt.figure(figsize=(9,6))
tree.plot_tree(clf_gini, **options)
plt.show()
```

For each node:
- `___ <= ___` is the splitting criterion for internal nodes, satisfied only by samples going left.
- `gini = ...` shows the impurity index. By default, the algorithm uses Gini impurity index to find the best binary split. Observe that the index decreases down the tree towards the leafs.
- `value = [_, _, _]` shows the number of examples for each of the three classes, and `class = ...` indicates a majority class, which may be used as the decision for a leaf node. The majority classes are also color coded. Observe that the color gets lighter towards the root, as the class distribution is more impure.

In particular, check that iris setosa is distinguished immediately after checking the petal width/length.

All the information of the decision is stored in the `tree_` attribute of the classifer. For more details:

```python
help(clf_gini.tree_)
```

**Exercise** Assign to `clf_entropy` the decision tree classifier created using *entropy* as the impurity measure. You can do so with the keyword argument `criterion='entropy'` in `DecisionTreeClassifier`. Furthermore, Use `random_state=0` and fit the classifier on the entire iris dataset. Check whether the resulting decision tree same as the one created using the Gini impurity index.

```python
# YOUR CODE HERE
raise NotImplementedError()

plt.figure(figsize=(9, 6))
tree.plot_tree(clf_entropy, **options)
plt.show()
```

YOUR ANSWER HERE

It is important to note that, although one can specify whether to use Gini impurity or entropy, `sklearn` implements neither C4.5 nor CART algorithms. In particular, it supports only binary splits on numeric input attributes, unlike C4.5 which supports multi-way splits using information gain ratio.
(See some [workarounds][categorical].)

[categorical]: https://stackoverflow.com/questions/38108832/passing-categorical-data-to-sklearn-decision-tree

## Compute Splitting Criterion

To understand how the decision tree is generated, we will implements the computation of the splitting criterion.

### Basic data analysis using `pandas`

To have an idea of qualities of the features, create a [`pandas`](https://pandas.pydata.org/docs/user_guide/index.html) [`DataFrame`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html?highlight=dataframe#pandas.DataFrame)
to operate on the dataset.

```python
# write the input features first
iris_df = pd.DataFrame(data=iris.data, columns=iris.feature_names)

# append the target values to the last column
iris_df['target'] = iris.target
iris_df.target = iris_df.target.astype('category')
iris_df.target.cat.categories = iris.target_names
iris_df
```

To display some statistics of the input features for different classes:

```python
iris_df.groupby('target').boxplot(rot=90, layout=(1,3))
iris_df.groupby('target').agg(['mean','std']).round(2)
```

**Exercise** Identify good feature(s) based on the above statistics. Does you choice agree with the decision tree generated by `DecisionTreeClassifier`?

YOUR ANSWER HERE

### Compute impurity

Given a distribution $\boldsymbol{p}=(p_1,p_2,\dots)$ where $p_k\in [0,1]$ and $\sum_k p_k =1$, the Gini impurity index is defined as:

\begin{align}
\operatorname{Gini}(\boldsymbol{p}) = \operatorname{Gini}(p_1,p_2,\dots) &:= \sum_k p_k(1-p_k)\\
&= 1- \sum_k p_k^2.
\end{align}

We can represent a distribution simply as a `numpy` array. To return the empirical class distributions of the iris dataset:

```python
def dist(values):
    '''Returns the empirical distribution of the given 1D array of values as a
    1D array of probabilites.'''
    counts = np.unique(values, return_counts=True)[-1]
    return counts / counts.sum()

print(f"Distribution of target: {dist(iris.target).round(4)}")
```

The Gini impurity index can be computed as follows:

```python
def gini(p):
    '''Returns the Gini impurity of distribution p.'''
    return 1 - (p**2).sum()

print(f"Gini impurity of target: {gini(dist(iris.target)):.4g}")
```

**Exercise** Complete the following function to compute the entropy of a distribution:

\begin{align}
h(\boldsymbol{p}) = h(p_1,p_2,\dots) &= \sum_k - p_k \log_2 p_k\\
&= \sum_{k:p_k>0} - p_k \log_2 p_k.
\end{align}

You may use the function `log2` from `numpy` to calculate the logarithm base 2. Note that logarithm of $0$ is undefined, we use the last expression of the entropy to avoid taking the limit $\lim_{p\to 0} p\log p=0$.

You solution should look like:
```Python
def entropy(p):
    ...
    return (p * ___ * ___).sum()
```

```python
def entropy(p):
    '''Returns the entropy of distribution p.'''
    p = np.array(p)
    p = p[(p > 0) & (p < 1)]  # 0 log 0 = 1 log 1 = 0
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# tests
assert np.isclose(entropy([1/2, 1/2]), 1)
assert np.isclose(entropy([1, 0]), 0)
```

### Compute drop in impurity and best split

Now, to compute the drop in impurity for given splitting criterion:

\begin{align}
\Delta \operatorname{Gini}_{X\leq s}(Y) = \operatorname{Gini}(P_Y) - \left[\Pr\{X\leq s\} \operatorname{Gini}(P_{Y|X\leq s}) + \Pr\{X> s\}\operatorname{Gini}(P_{Y|X> s})\right]
\end{align}

```python
def drop_in_gini(X, Y, split_pt):
    '''Returns the drop in Gini impurity of Y for the split X <= split_pt.

    Parameters
    ----------
    X: 1D array the values of a feature for different instances
    Y: 1D array of the corresponding target values
    split_pt: the value of the split point
    '''
    S = X <= split_pt
    q = S.mean()
    return gini(dist(Y)) - q * gini(dist(Y[S])) - (1 - q) * gini(dist(Y[~S]))

X, Y = iris_df['petal width (cm)'], iris_df.target
print(f"Drop in Gini: {drop_in_gini(X, Y, 0.8):.4g}")
```

To compute the best split point for a feature, we check every consecutive mid-points of the possible feature values:

```python
def find_best_split_pt(X, Y, gain_function):
    '''Return the best split point s and the maximum gain evaluated using
    gain_function for the split X <= s and target Y.

    Parameters
    ----------
    X: 1D array the values of a feature for different instances
    Y: 1D array of the corresponding target values
    gain_function: a function such as drop_in_gini for evaluating a split

    Returns
    -------
    A tuple (s, g) where s is the best split point and g is the maximum gain.

    See also
    --------
    drop_in_gini
    '''
    values = np.sort(np.unique(X))
    split_pts = (values[1:] + values[:-1]) / 2
    gain = np.array([gain_function(X, Y, s) for s in split_pts])
    max_index = np.argmax(gain)
    return split_pts[max_index], gain[max_index]

print('''Best split point: {0:.4g}
Maximum gain: {1:.4g}'''.format(*find_best_split_pt(X, Y, drop_in_gini)))
```

The following ranks the features according to the gains of their best binary splits:

```python
rank_by_gini = pd.DataFrame({
    'feature': feature,
    **(lambda s, g: {
        'split point': s,
        'gain': g
    })(*find_best_split_pt(iris_df[feature], iris_df.target, drop_in_gini))
} for feature in iris.feature_names).sort_values(by='gain', ascending=False)
rank_by_gini
```

**Exercise** Complete the following function to calculate the *information gain* for a binary split $X\leq s$ and target $Y$:

\begin{align}
\operatorname{Gain}_{X\leq s}(Y) := h(P_Y) - \left[\Pr(X\leq s) h(P_{Y|X\leq s}) + \Pr(X> s) h(P_{Y|X> s})\right].
\end{align}

You may use `dist` and `entropy` defined previously.

```python
def info_gain(X, Y, split_pt):
    '''Returns the information Gain of Y for the split X <= split_pt.

    Parameters
    ----------
    X: 1D array the values of a feature for different instances
    Y: 1D array of the corresponding target values
    split_pt: the value of the split point
    '''
    S = X <= split_pt
    q = S.mean()
    # YOUR CODE HERE
    raise NotImplementedError()

print(f"Information gain: {info_gain(X, Y, 0.8):.4g}")
```

```python
# tests
rank_by_entropy = pd.DataFrame({
    'feature':
    feature,
    **(lambda s, g: {
        'split point': s,
        'gain': g
    })(*find_best_split_pt(iris_df[feature], iris_df.target, info_gain))
} for feature in iris.feature_names).sort_values(by='gain', ascending=False)
rank_by_entropy
```

**Exercise** Complete the following function to calculate the *information gain ratio* for a binary split $X\leq s$ and target $Y$:

\begin{align}
\operatorname{GainRatio}_{X\leq s}(Y) &:= \frac{\operatorname{Gain}_{X\leq s}(Y)}{\operatorname{SplitInfo}(X\leq s)} \qquad\text{where}\\
\operatorname{SplitInfo}(X\leq s) &:= h(\Pr(X\leq s),\Pr(X> s)).
\end{align}

You may use `entropy` and `info_gain` defined previously.

```python
def info_gain_ratio(X, Y, split_pt):
    S = X <= split_pt
    q = S.mean()
    # YOUR CODE HERE
    raise NotImplementedError()
```

```python
# tests
rank_by_info_gain_ratio = pd.DataFrame({
    'feature':
    feature,
    **(lambda s, g: {
        'split point': s,
        'gain': g
    })(*find_best_split_pt(iris_df[feature], iris_df.target, info_gain_ratio))
} for feature in iris.feature_names).sort_values(by='gain', ascending=False)
rank_by_info_gain_ratio
```

**Exercise** Does the information gain ratio give a different ranking of the features? Why?

Information gain ratio gives the same ranking as information gain in this case. This is because the split is restricted to be binary and so the normalization by split information has less effect on the ranking.

Tutorial3/Decision Tree Induction with scikit-learn.ipynb


```python
from scipy.misc import derivative
import scipy.integrate
from math import *
import sympy as sp
import matplotlib.pyplot as plt
import pandas as pd
import random
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import seaborn as sns
```

```python
def f1(x):
    return sin(x) * cos(x) + exp(2*x) + 2*x**4 - 10
```

```python
f1_result = derivative(f1, 2, dx = 1e-10)
f1_result
```

    172.5426557186438

```python
def f2(x):
    return (5*(x**2))/sin(x)
```

```python
f2_result = derivative(f2, 2, dx = 1e-10)
f2_result
```

    32.06119814080921

## Partial Derivative

```python
x = sp.Symbol('x')
y = sp.Symbol('y')
z = sp.Symbol('z')
```

```python
w = (x**2)*y - 10*(y**2)*(z**3) + 43*x - 7*sp.tan(4*y)
```

```python
w_p = sp.diff(w, x)
w_p
```

    2*x*y + 43

```python
w_p_v = w_p.subs({x:3})
w_p_v
```

    6*y + 43

## Derivarive Rule

```python
def f(x):
    return 4*(x**3) + 3*(x**2)
def g(x):
    return 5*(x**6) + 4*x
```

```python
x = 2
fg_prime_of_x = derivative(f, x, dx=1e-10)*g(x) + f(x)*derivative(g, x, dx=1e-10)
fg_prime_of_x
```

    62096.008264234115

```python
f_prime_g_prime_of_x = derivative(f, x, dx=1e-10) * derivative(g, x, dx=1e-10)
f_prime_g_prime_of_x
```

    57840.01383466328

## Definite Integral

```python
f = lambda x,y: 1 + 8*x*y
g = lambda x:0
h = lambda x:3
```

```python
i, e = scipy.integrate.dblquad(f, 1,2, g, h)
i
```

    56.99999999999999

## Gradient Descent

```python
h = [74, 74, 72, 72, 73, 69, 69, 71, 76, 71, 73, 73, 74, 74, 69, 70, 73, 75, 78, 79, 76, 74, 76, 72, 71, 75, 77, 74, 73, 74, 78, 73, 75, 73, 75, 75, 74, 69, 71, 74, 73, 73, 76, 74, 74, 70, 72, 77, 74, 70, 73, 75, 76, 76, 78, 74, 74, 76, 77, 81, 78, 75, 77, 75, 76, 74, 72, 72, 75, 73,  73, 73, 70, 70, 70, 76, 68, 71, 72, 75, 75, 75, 75, 68, 74, 78, 71, 73, 76, 74, 74, 79, 75, 73, 76, 74, 74, 73, 72, 74, 73, 74, 72, 73, 69, 72, 73, 75, 75, 73, 72, 72, 76, 74, 72, 77, 74, 77, 75, 76, 80, 74, 74, 75, 78, 73, 73, 74, 75, 76, 71, 73, 74, 76, 76, 74, 73, 74, 70, 72,  73, 73, 73, 73, 71, 74, 74, 72, 74, 71, 74, 73, 75, 75, 79, 73, 75, 76, 74, 76, 78, 74, 76, 72, 74, 76, 74, 75, 78, 75, 72, 74, 72, 74, 70, 71, 70, 75, 71, 71, 73, 72, 71, 73, 72, 75, 74, 74, 75, 73, 77, 73, 76, 75, 74, 76, 75, 73, 71, 76, 75, 72, 71, 77, 73, 74, 71, 72, 74, 75,    73, 72, 75, 75, 74, 72, 74, 71, 70, 74, 77, 77, 75, 75, 78, 75, 76, 73, 75, 75, 79, 77, 76, 71, 75, 74, 69, 71, 76, 72, 72, 70, 72, 73, 71, 72, 71, 73, 72, 73, 74, 74, 72, 75, 74, 74, 77, 75, 73, 72, 71, 74, 77, 75, 75, 75, 78, 78, 74, 76, 78, 76, 70, 72, 80, 74, 74, 71, 70, 72,    71, 74, 71, 72, 71, 74, 69, 76, 75, 75, 76, 73, 76, 73, 77, 73, 72, 72, 77, 77, 71, 74, 74, 73, 78, 75, 73, 70, 74, 72, 73, 73, 75, 75, 74, 76, 73, 74, 75, 75, 72, 73, 73, 72, 74, 78, 76, 73, 74, 75, 70, 75, 71, 72, 78, 75, 73, 73, 71, 75, 77, 72, 69, 73, 74, 72, 70, 75, 70, 72,    72, 74, 73, 74, 76, 75, 80, 72, 75, 73, 74, 74, 73, 75, 75, 71, 73, 75, 74, 74, 72, 74, 74, 74, 73, 76, 75, 72, 73, 73, 73, 72, 72, 72, 72, 71, 75, 75, 74, 73, 75, 79, 74, 76, 73, 74, 74, 72, 74, 74, 75, 78, 74, 74, 74, 77, 70, 73, 74, 73, 71, 75, 71, 72, 77, 74, 70, 77, 73, 72, 76, 71, 76, 78, 75, 73, 78, 74, 79, 75, 76, 72, 75, 75, 70, 72, 70, 74, 71, 76, 73, 76, 71, 69, 72, 72, 69, 73, 69, 73, 74, 74, 72, 71, 72, 72, 76, 76, 76, 74, 76, 75, 71, 72, 71, 73, 75, 76, 75, 71, 75, 74, 72, 73, 73, 73, 73, 76, 72, 76, 73, 73, 73, 75, 75, 77, 73, 72, 75, 70, 74, 72, 80, 71, 71, 74, 74, 73, 75, 76, 73, 77, 72, 73, 77, 76, 71, 75, 73, 74, 77, 71, 72, 73, 69, 73, 70, 74, 76, 73, 73, 75, 73, 79, 74, 73, 74, 77, 75, 74, 73, 77, 73, 77, 74, 74, 73, 77, 74, 77, 75, 77, 75, 71, 74, 70, 79, 72, 72, 70, 74, 74, 72, 73, 72, 74, 74, 76, 82, 74, 74, 70, 73, 73, 74, 77, 72, 76, 73, 73, 72, 74, 74, 71, 72, 75, 74, 74, 77, 70, 71, 73, 76, 71, 75, 74, 72, 76, 79, 76, 73, 76, 78, 75, 76, 72, 72, 73, 73, 75, 71, 76, 70, 75, 74, 75, 73, 71, 71, 72, 73, 73, 72, 69, 73, 78, 71, 73, 75, 76, 70, 74, 77, 75, 79, 72, 77, 73, 75, 75, 75, 73, 73, 76, 77, 75, 70, 71, 71, 75, 74, 69, 70, 75, 72, 75, 73, 72, 72, 72, 76, 75, 74, 69, 73, 72, 72, 75, 77, 76, 80, 77, 76, 79, 71, 75, 73, 76, 77, 73, 76, 70, 75, 73, 75, 70, 69, 71, 72, 72, 73, 70, 70, 73, 76, 75, 72, 73, 79, 71, 72, 74, 74, 74, 72, 76, 76, 72, 72, 71, 72, 72, 70, 77, 74, 72, 76, 71, 76, 71, 73, 70, 73, 73, 72, 71, 71, 71, 72, 72, 74, 74, 74, 71, 72, 75, 72, 71, 72, 72, 72, 72, 74, 74, 77, 75, 73, 75, 73, 76, 72, 77, 75, 72, 71, 71, 75, 72, 73, 73, 71, 70, 75, 71, 76, 73, 68, 71, 72, 74, 77, 72, 76, 78, 81, 72, 73, 76, 72, 72, 74, 76, 73, 76, 75, 70, 71, 74, 72, 73, 76, 76, 73, 71, 68, 71, 71, 74, 77, 69, 72, 76, 75, 76, 75, 76, 72, 74, 76, 74, 72, 75, 78, 77, 70, 72, 79, 74, 71, 68, 77, 75, 71, 72, 70, 72, 72, 73, 72, 74, 72, 72, 75, 72, 73, 74, 72, 78, 75, 72, 74, 75, 75, 76, 74, 74, 73, 74, 71, 74, 75, 76, 74, 76, 76, 73, 75, 75, 74, 68, 72, 75, 71, 70, 72, 73, 72, 75, 74, 70, 76, 71, 82, 72, 73, 74, 71, 75, 77, 72, 74, 72, 73, 78, 77, 73, 73, 73, 73, 73, 76, 75, 70, 73, 72, 73, 75, 74, 73, 73, 76, 73, 75, 70, 77, 72, 77, 74, 75, 75, 75, 75, 72, 74, 71, 76, 71, 75, 76, 83, 75, 74, 76, 72, 72, 75, 75, 72, 77, 73, 72, 70, 74, 72, 74, 72, 71, 70, 71, 76, 74, 76, 74, 74, 74, 75, 75, 71, 71, 74, 77, 71, 74, 75, 77, 76, 74, 76, 72, 71, 72, 75, 73, 68, 72, 69, 73, 73, 75, 70, 70, 74, 75, 74, 74, 73, 74, 75, 77, 73, 74, 76, 74, 75, 73, 76, 78, 75, 73, 77, 74, 72, 74, 72, 71, 73, 75, 73, 67, 67, 76, 74, 73, 70, 75, 70, 72, 77, 79, 78, 74, 75, 75, 78, 76, 75, 69, 75, 72, 75, 73, 74, 75, 75, 73]

```

```python
w = [180, 215, 210, 210, 188, 176, 209, 200, 231, 180, 188, 180, 185, 160, 180, 185, 189, 185, 219, 230, 205, 230, 195, 180, 192, 225, 203, 195, 182, 188, 200, 180, 200, 200, 245, 240, 215, 185, 175, 199, 200, 215, 200, 205, 206, 186, 188, 220, 210, 195, 200, 200, 212, 224, 210, 205, 220, 195, 200, 260, 228, 270, 200, 210, 190, 220, 180, 205, 210, 220, 211, 200, 180, 190, 170, 230, 155, 185, 185, 200, 225, 225, 220, 160, 205, 235, 250, 210, 190, 160, 200, 205, 222, 195, 205, 220, 220, 170, 185, 195, 220, 230, 180, 220, 180, 180, 170, 210, 215, 200, 213, 180, 192, 235, 185, 235, 210, 222, 210, 230, 220, 180, 190, 200, 210, 194, 180, 190, 240, 200, 198, 200, 195, 210, 220, 190, 210, 225, 180, 185, 170, 185, 185, 180, 178, 175, 200, 204, 211, 190, 210, 190, 190, 185, 290, 175, 185, 200, 220, 170, 220, 190, 220, 205, 200, 250, 225, 215, 210, 215, 195, 200, 194, 220, 180, 180, 170, 195, 180, 170, 206, 205, 200, 225, 201, 225, 233, 180, 225, 180, 220, 180, 237, 215, 190, 235, 190, 180, 165, 195, 200, 190, 190, 185, 185, 205, 190, 205, 206, 220, 208, 170, 195, 210, 190, 211, 230, 170, 185, 185, 241, 225, 210, 175, 230, 200, 215, 198, 226, 278, 215, 230, 240, 184, 219, 170, 218, 190, 225, 220, 176, 190, 197, 204, 167, 180, 195, 220, 215, 185, 190, 205, 205, 200, 210, 215, 200, 205, 211, 190, 208, 200, 210, 232, 230, 210, 220, 210, 202, 212, 225, 170, 190, 200, 237, 220, 170, 193, 190, 150, 220, 200, 190, 185, 185, 200, 172, 220, 225, 190, 195, 219, 190, 197, 200, 195, 210, 177, 220, 235, 180, 195, 195, 190, 230, 190, 200, 190, 190, 200, 200, 184, 200, 180, 219, 187, 200, 220, 205, 190, 170, 160, 215, 175, 205, 200, 214, 200, 190, 180, 205, 220, 190, 215, 235, 191, 200, 181, 200, 210, 240, 185, 165, 190, 185, 175, 155, 210, 170, 175, 220, 210, 205, 200, 205, 195, 240, 150, 200, 215, 202, 200, 190, 205, 190, 160, 215, 185, 200, 190, 210, 185, 220, 190, 202, 205, 220, 175, 160, 190, 200, 229, 206, 220, 180, 195, 175, 188, 230, 190, 200, 190, 219, 235, 180, 180, 180, 200, 234, 185, 220, 223, 200, 210, 200, 210, 190, 177, 227, 180, 195, 199, 175, 185, 240, 210, 180, 194, 225, 180, 205, 193, 230, 230, 220, 200, 249, 190, 208, 245, 250, 160, 192, 220, 170, 197, 155, 190, 200, 220, 210, 228, 190, 160, 184, 180, 180, 200, 176, 160, 222, 211, 195, 200, 175, 206, 240, 185, 260, 185, 221, 205, 200, 170, 201, 205, 185, 205, 245, 220, 210, 220, 185, 175, 170, 180, 200, 210, 175, 220, 206, 180, 210, 195, 200, 200, 164, 180, 220, 195, 205, 170, 240, 210, 195, 200, 205, 192, 190, 170, 240, 200, 205, 175, 250, 220, 224, 210, 195, 180, 245, 175, 180, 215, 175, 180, 195, 230, 230, 205, 215, 195, 180, 205, 180, 190, 180, 190, 190, 220, 210, 255, 190, 230, 200, 205, 210, 225, 215, 220, 205, 200, 220, 197, 225, 187, 245, 185, 185, 175, 200, 180, 188, 225, 200, 210, 245, 213, 231, 165, 228, 210, 250, 191, 190, 200, 215, 254, 232, 180, 215, 220, 180, 200, 170, 195, 210, 200, 220, 165, 180, 200, 200, 170, 224, 220, 180, 198, 240, 239, 185, 210, 220, 200, 195, 220, 230, 170, 220, 230, 165, 205, 192, 210, 205, 200, 210, 185, 195, 202, 205, 195, 180, 200, 185, 240, 185, 220, 205, 205, 180, 201, 190, 208, 240, 180, 230, 195, 215, 190, 195, 215, 215, 220, 220, 230, 195, 190, 195, 209, 204, 170, 185, 205, 175, 210, 190, 180, 180, 160, 235, 200, 210, 180, 190, 197, 203, 205, 170, 200, 250, 200, 220, 200, 190, 170, 190, 220, 215, 206, 215, 185, 235, 188, 230, 195, 168, 190, 160, 200, 200, 189, 180, 190, 200, 220, 187, 240, 190, 180, 185, 210, 220, 219, 190, 193, 175, 180, 215, 210, 200, 190, 185, 220, 170, 195, 205, 195, 210, 190, 190, 180, 220, 190, 186, 185, 190, 180, 190, 170, 210, 240, 220, 180, 210, 210, 195, 160, 180, 205, 200, 185, 245, 190, 210, 200, 200, 222, 215, 240, 170, 220, 156, 190, 202, 221, 200, 190, 210, 190, 200, 165, 190, 185, 230, 208, 209, 175, 180, 200, 205, 200, 250, 210, 230, 244, 202, 240, 200, 215, 177, 210, 170, 215, 217, 198, 200, 220, 170, 200, 230, 231, 183, 192, 167, 190, 180, 180, 215, 160, 205, 223, 175, 170, 190, 240, 175, 230, 223, 196, 167, 195, 190, 250, 190, 190, 190, 170, 160, 150, 225, 220, 209, 210, 176, 260, 195, 190, 184, 180, 195, 195, 219, 225, 212, 202, 185, 200, 209, 200, 195, 228, 210, 190, 212, 190, 218, 220, 190, 235, 210, 200, 188, 210, 235, 188, 215, 216, 220, 180, 185, 200, 210, 220, 185, 231, 210, 195, 200, 205, 200, 190, 250, 185, 180, 170, 180, 208, 235, 215, 244, 220, 185, 230, 190, 200, 180, 190, 196, 180, 230, 224, 160, 178, 205, 185, 210, 180, 190, 200, 257, 190, 220, 165, 205, 200, 208, 185, 215, 170, 235, 210, 170, 180, 170, 190, 150, 230, 203, 260, 246, 186, 210, 198, 210, 215, 180, 200, 245, 200, 192, 192, 200, 192, 205, 190, 186, 170, 197, 219, 200, 220, 207, 225, 207, 212, 225, 170, 190, 210, 230, 210, 200, 238, 234, 222, 200, 190, 170, 220, 223, 210, 215, 196, 175, 175, 189, 205, 210, 180, 180, 197, 220, 228, 190, 204, 165, 216, 220, 208, 210, 215, 195, 200, 215, 229, 240, 207, 205, 208, 185, 190, 170, 208, 225, 190, 225, 185, 180, 165, 240, 220, 212, 163, 215, 175, 205, 210, 205, 208, 215, 180, 200, 230, 211, 230, 190, 220, 180, 205, 190, 180, 205, 190, 195]
```

```python
heights = np.array(h)
weights = np.array(w)
heights = heights*0.0254
weights = weights * 0.453592
```

```python
plt.figure(figsize=(12,8))
sns.jointplot(x=heights, y = weights)
plt.show()
```

```python
# weights = m * heights + b
```

```python
print(heights.shape)
print(weights.shape)
```

    (1015, 1)
    (1015, 1)

```python
weights = weights.reshape(weights.size, 1)
heights = heights.reshape(weights.size, 1)
```

```python
import sys
sys.path.insert(
    0,
    r"C:\Users\DELL\Desktop\Sentifi\machine_learning\linear_regression_from_scratch\gradient_descent",
)
```

```python
import GD_linear_regression as glr

builder = glr.GDLinearRegression(iterations=200)
reg = builder.fit(heights, weights)
predictions = reg.predict(heights)
costs = reg.costs
thetas = reg.theta
```

```python
print(len(costs))
print(reg.theta[-1])
print(costs[-1])
```

    200
    [[48.79217707]]
    34.286653301755706

```python
plt.plot(costs)
plt.show()
```

```python

```


### Linear Equations

An equation is an statement that equates to expressions. In mathematics, a linear equation is an equation that may be put in the form of $ax + by + c = z$. In this form the values a is called the *coefficients*, that are often real numbers.*x,y,z* are called the variables. The *c* is often used for constants.

If the equation has only one undefined variabe then we call the it Linear equation in one variabe.These equations take the form $a x + b = 0$.

Similarly, equations with two undefined variabes are called Linear equation in two variabe. These equations take generally take the form $a x + by + c = 0$.

We can also have equations with multiple variable called polynomials. What is a polynomial? the word "polynomial" just means many terms. And a polynomial is an algebraic expression, that contains constants, variables with coefficients, and exponentials with a positive exponent.

#### Slope-Intercept Form

A line can be defined by its *m* slope, and its y-intercept y0 (the y coordinate where is intersects the y-axis).
The most common form is the slope-intercept equation of a straight line:

\begin{align}
y & = mx+b \\
\end{align}

Lets look at the video to get more understanding of linear equations.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('CWByMDHyTos', width=860, height=460)
```

#### Plotting slope intercept form

```python
import numpy as np
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.add_subplot(111)

# https://numpy.org/doc/stable/reference/generated/numpy.linspace.html
x = np.linspace(-10, 10, 1000)

# https://matplotlib.org/3.1.1/api/spines_api.html
ax.spines['left'].set_position('center')
ax.spines['bottom'].set_position('center')
ax.spines['right'].set_position('center')
ax.spines['top'].set_position('center')

plt.plot(x, 2*x+1, '-r', label = 'y = 2x+1')
plt.plot(x, 2*x-3, '-b', label = 'y = 2x+1')

plt.legend(loc = 'upper left')
plt.show()
```

### Functions

Sometimes a equation is written as a function, with f(x) instead of y. The above slope intecept equation can be written as $f(x) = mx + b$. We see functions everywhere in our lives, anything that has a input, process and output is a function. There are other functions like $x2$ is a squaring function, $y3+1$ is also a function and in geometry, Sine, Cosine and Tangent are functions.
Lets understand functions more clearly using the video below.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('P3LsdcjqS8s', width=860, height=460)
```

One important thing to keep in mind is that function is single valued. one value will have only one output value. This can be tested by [verticle line test](https://en.wikipedia.org/wiki/Vertical_line_test), meaning no vertical line ever crosses more than one value

```python
def function(x, y):
    return (x**2+y)

print(function(5,3))
```

#### Plotting Functions

```python
import matplotlib.pyplot as plt
import numpy as np

# create 1000 equally spaced points between -10 and 10
# https://numpy.org/doc/stable/reference/generated/numpy.linspace.html
x = np.linspace(-100, 100, 1000)

# calculate the y value for each element of the x vector
y = x**2 - x + 2

fig, ax = plt.subplots()

#https://matplotlib.org/api/_as_gen/matplotlib.axes.Axes.plot.html
ax.plot(x, y)
```

### Vectors

Any object that has length and direction is called a vector. We can use vectors to define the movement of objects from one point to another. Two vectors can be same if they have same magnitude and direction. Some exmaples of vectors can be Velocity, Acceleration, Force and increase/decrease in anything. Vectors are built from ordinary numbers, which form the components
of the vector. As described in the figure below, a vector has a head, a tail direction and magnitude.

Vector algebra is very useful in solving problems in Digital Technologies like Machine Learning, computer graphics, computer vision and data science.

#### Plotting a vectors in 2D

```python
import numpy as np
import matplotlib.pyplot as plt

v = np.array([2,4])

fig, ax = plt.subplots()

ax.plot([0, v[0]], [0, v[1]])

```

#### Plotting a vectors in 3D

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

v = np.array([2,4,6])

fig = plt.figure()

ax = Axes3D(fig)

# x,y,z

ax.plot([0,0], [0,0], [-10, 10], 'k--')
ax.plot([0,0], [-10, 10], [0,0], 'k--')
ax.plot( [-10, 10], [0,0], [0,0],  'k--')

ax.plot([0, v[0]],[0,v[1]],[0,v[2]])

```

#### Vector Addition

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

v = np.array([2,4,6])

u = np.array([-5, -3, -2])

z = np.add(v,u)

fig = plt.figure()

ax = Axes3D(fig)

# x,y,z

ax.plot([0,0], [0,0], [-10, 10], 'k--')
ax.plot([0,0], [-10, 10], [0,0], 'k--')
ax.plot( [-10, 10], [0,0], [0,0],  'k--')

print (z)
ax.plot([0, v[0]],[0,v[1]],[0,v[2]])
ax.plot([0, u[0]],[0,u[1]],[0,u[2]])
ax.plot([0, z[0]],[0,z[1]],[0,z[2]])
```

#### Vector Multiplication

Vector Multiplication is of three types "Scalar product", “Cross product” and “Dot product.”

In an scalar product we multiply the number directly by a scalar quantity and our vector scales by that much value.

A dot product can be thought as direction growth on a vector when multiplied with another vector, the result gives us the effect on the original vector(positive, negative, or zero). Dot product is a scalar quantity. We calculate the dot product using this formula $a · b = |a| × |b| × cos(θ)$

Cross Product is defined as a vector that is at right angles to the other two vectors . We calculate cross product using this formula $a × b = |a| |b| sin(θ) n$. We leverage the "Right Hand Rule" to better understand the cross product.

#### Try it out

If you point your index finger along vector a, and point your middle finger along vector b: the cross product goes in the direction of your thumb. ,

```python
u = np.array([1,2,3])
v = np.array([5,2,4])

# Dot product

dotp = np.dot(u,v)

# Cross Product

crossp = np.cross(u,v)

print("Dot Product is : {}".format(dotp))

print("Cross Product is : {}".format(crossp))
```

    Dot Product is : 21
    Cross Product is : [ 2 11 -8]

### Exercices

#### Pen and Paper

1. Find a vector that is orthogonal to the 3D vectors,  P = ( 3 , 2 , 1 ) , Q = ( 4 , − 1 , 1 )

2. Solve the given systems of linear equations:-
$x + 2y = 10$, $2x + y = 6$

####  Code

3. Using the above cross product approach, plot two 3D vectors and their cross product

```python
# Plot 3D vectors and their cross product in 3D Space.

def plot_cross_product(vector_a, vector_b):
    pass

plot_cross_product(u,v)
```


# R(2,2) playground
Ted Corcovilos, 2021-01-08

Playing around with the 2d "mother algebra" $R(2,2)$, as described in C. Doran, et al., "Lie Groups as Spin Groups," *Journal of Mathematical Physics 34*, 3642 (1993). doi:[10.1063/1.530050](http://doi.org/10.1063/1.530050)

I'll name the basis vectors $p_1, p_2, m_1, m_2$ with the diagonal metric $[1,1,-1,-1]$.

Position vectors in this basis are represented by null vectors of the form $x (p_1+m_1) + y (p_2+m_2)$.

```python
from sympy import *
from galgebra.ga import Ga
from galgebra.printer import latex
from IPython.display import Math

init_printing(latex_printer=latex, use_latex='mathjax')
```

```python
xy = (x,y) = symbols("x y", real=True)
```

```python
xyxy = (xp, yp, xm, ym) = symbols("xp yp xm ym", real=True)
```

```python
R22 = Ga('p1 p2 m1 m2', g=[1,1,-1,-1], coords=xyxy)
```

```python
p1, p2, m1, m2 = R22.mv() # break out basis vectors
```

```python
# a real position vector has the form...
r = x*(p1+m1)+y*(p2+m2)
```

```python
r
```

\begin{equation*} x \boldsymbol{p}_{1} + y \boldsymbol{p}_{2} + x \boldsymbol{m}_{1} + y \boldsymbol{m}_{2} \end{equation*}

```python
a, b, c, d = symbols("a b c d", real=True)
f, g, h, j = symbols("f g h j", real=True)
```

```python
# big pseudoscalar
I=p1^p2^m1^m2
# special bivectors (ref. Doran)
K = (p1^m1)+(p2^m2)
E = (p1^p2)+(m1^m2)
F = (p1^m2)+(p2^m1)
```

From the paper, only bivectors that commute with $K$ will preserve null vectors. $E$ corresponds to the pseudo-scalar in the 2d position space. $F$ is the leftover bit needed to complete the bivector space.  Geometrically, $K$ describes scaling, $E$ rotations, and $F$ shears.

```python
B1 = a*(p1^m1)+b*(p2^m2)+c*(p1^m2)+d*(p2^m1)+f*(p1^p2)+g*(m1^m2)
```

```python
# check if commutator is zero
B1 >> K
```

\begin{equation*} \left ( c - d\right ) \boldsymbol{p}_{1}\wedge \boldsymbol{p}_{2} + \left ( f + g\right ) \boldsymbol{p}_{1}\wedge \boldsymbol{m}_{2} + \left ( - f - g\right ) \boldsymbol{p}_{2}\wedge \boldsymbol{m}_{1} + \left ( c - d\right ) \boldsymbol{m}_{1}\wedge \boldsymbol{m}_{2} \end{equation*}

```python
# So, this will be zero iff c=d and f=-g
# Redefine, change up the names a bit:
B1 = (a+b)/2*(p1^m1)+(a-b)/2*(p2^m2)+c/2*(p1^m2)+c/2*(p2^m1)+d/2*(p1^p2)-d/2*(m1^m2)
```

```python
# check commutator again
B1 >> K
```

\begin{equation*}  0  \end{equation*}

```python
# also need to normalize B1
# the norm squared is
Bnorm2=(B1*(B1.rev())).scalar()
```

```python
A, B, C, D = symbols("A B C D", real=True)
```

Let's break `B1` down term by term to see how it transforms position.

```python
# Define a small number as a placeholder to go from members of the Lie algebra to the Lie group
ϵ = symbols("ϵ", real=True)
```

Look at each bivector in the Lie algebra and see how the corresponding Lie group member transforms a position vector.

```python
# for the "a" term
((-ϵ*p1^m1/2).exp())*r*((ϵ*p1^m1/2).exp())
```

\begin{equation*} x e^{ϵ} \boldsymbol{p}_{1} + y \boldsymbol{p}_{2} + x e^{ϵ} \boldsymbol{m}_{1} + y \boldsymbol{m}_{2} \end{equation*}

So, the $p_1 \wedge m_1$ term looks like a scaling of $x$.

```python
# for the "b" term
((-ϵ*p2^m2/2).exp())*r*((ϵ*p2^m2/2).exp())
```

\begin{equation*} x \boldsymbol{p}_{1} + y e^{ϵ} \boldsymbol{p}_{2} + x \boldsymbol{m}_{1} + y e^{ϵ} \boldsymbol{m}_{2} \end{equation*}

Scaling of $y$, as expected.

```python
# confirm that the a and b pieces commute:
(p1^m1) >> (p2^m2)
```

\begin{equation*}  0  \end{equation*}

```python
# commuting Lie algebra elements => we can apply the exponentials individually
# overall scaling
(-ϵ*p1^m1/2).exp()*((-ϵ*p2^m2/2).exp())*r*((ϵ*p2^m2/2).exp())*(ϵ*p1^m1/2).exp()
```

\begin{equation*} x e^{ϵ} \boldsymbol{p}_{1} + y e^{ϵ} \boldsymbol{p}_{2} + x e^{ϵ} \boldsymbol{m}_{1} + y e^{ϵ} \boldsymbol{m}_{2} \end{equation*}

```python
# inverse scaling (is there a better name?)
(-ϵ*p1^m1/2).exp()*((ϵ*p2^m2/2).exp())*r*((-ϵ*p2^m2/2).exp())*(ϵ*p1^m1/2).exp()
```

\begin{equation*} x e^{ϵ} \boldsymbol{p}_{1} + y e^{- ϵ} \boldsymbol{p}_{2} + x e^{ϵ} \boldsymbol{m}_{1} + y e^{- ϵ} \boldsymbol{m}_{2} \end{equation*}

```python
# check the pieces of the c term for commuting
(p1^m2) >> (p2^m1)
```

\begin{equation*}  0  \end{equation*}

```python
# the c term
(-ϵ*p1^m2/2).exp()*((-ϵ*p2^m1/2).exp())*r*((ϵ*p2^m1/2).exp())*(ϵ*p1^m2/2).exp()
```

\begin{equation*} \left ( x \cosh{\left (ϵ \right )} + y \sinh{\left (ϵ \right )}\right ) \boldsymbol{p}_{1} + \left ( x \sinh{\left (ϵ \right )} + y \cosh{\left (ϵ \right )}\right ) \boldsymbol{p}_{2} + \left ( x \cosh{\left (ϵ \right )} + y \sinh{\left (ϵ \right )}\right ) \boldsymbol{m}_{1} + \left ( x \sinh{\left (ϵ \right )} + y \cosh{\left (ϵ \right )}\right ) \boldsymbol{m}_{2} \end{equation*}

Scissor shear? (Not really a boost because this isn't Minkowski space...)

```python
#check the d term for commuting:
(p1^p2) >> (m1^m2)
```

\begin{equation*}  0  \end{equation*}

```python
# the d term
(-ϵ*p1^p2/2).exp()*(ϵ*m1^m2/2).exp()*r*(-ϵ*m1^m2/2).exp()*(ϵ*p1^p2/2).exp()
```

\begin{equation*} \left ( x \cos{\left (ϵ \right )} - y \sin{\left (ϵ \right )}\right ) \boldsymbol{p}_{1} + \left ( x \sin{\left (ϵ \right )} + y \cos{\left (ϵ \right )}\right ) \boldsymbol{p}_{2} + \left ( x \cos{\left (ϵ \right )} - y \sin{\left (ϵ \right )}\right ) \boldsymbol{m}_{1} + \left ( x \sin{\left (ϵ \right )} + y \cos{\left (ϵ \right )}\right ) \boldsymbol{m}_{2} \end{equation*}

Rotation.

So, the matrix version of these terms looks something like
$$
\begin{array}{cc}
a \rightarrow \begin{pmatrix} e^A & 0 \\ 0 & e^A \end{pmatrix}
&
b \rightarrow \begin{pmatrix} e^B & 0 \\ 0 & e^{-B} \end{pmatrix}
\\
c \rightarrow \begin{pmatrix} \cosh C & \sinh C \\ \sinh C & \cosh C \end{pmatrix}
&
d \rightarrow \begin{pmatrix} \cos D & -\sin D \\ \sin D & \cos D \end{pmatrix}
\end{array}
$$

Note that these do not mutually commute, so it's hard to decouple a generic matrix into these pieces.

Also, these are all positive-definite matrices, so we're not covering the full GL group.  Need one more piece:
$$
m \rightarrow \begin{pmatrix} 1 & 0 \\ 0 & \pm 1 \end{pmatrix}
$$
to cover reflections.

Stepping back, we can identify the 4 Lie algebra generators as the identity and the (real) Pauli matrices, demonstrating the isomorphism between the bivectors and spinors.

```python
# for example σ_x:
simplify(exp(C*Matrix([[0,1],[1,0]])))
```

$\displaystyle \left[\begin{array}{cc}\cosh{\left (C \right )} & \sinh{\left (C \right )}\\\sinh{\left (C \right )} & \cosh{\left (C \right )}\end{array}\right]$

```python
# σ_y
simplify(exp(D*Matrix([[0,-1],[1,0]])))
```

$\displaystyle \left[\begin{array}{cc}\cos{\left (D \right )} & - \sin{\left (D \right )}\\\sin{\left (D \right )} & \cos{\left (D \right )}\end{array}\right]$

```python
# σ_z
simplify(exp(B*Matrix([[1,0],[0,-1]])))
```

$\displaystyle \left[\begin{array}{cc}e^{B} & 0\\0 & e^{- B}\end{array}\right]$

```python
# I
simplify(exp(A*Matrix([[1,0],[0,1]])))
```

$\displaystyle \left[\begin{array}{cc}e^{A} & 0\\0 & e^{A}\end{array}\right]$

We can combine the last three terms and simplify using the axis-angle formula for SU(2).  (Need to use complex values?)

This still leaves the open problem of decomposing a generic 2x2 matrix into a product of the operators above.   The linear algebra solution is straight-forward but tedious.  Does GA give us any shortcuts?

```python

```