# Lambda School Data Science - Ridge Regression

Regularize your way to a better tomorrow.

# Lecture

Data science depends on math, and math is generally focused on situations where:

1. a solution exists,
2. the solution is unique,
3. the solution's behavior changes continuously with the initial conditions.

These are known as [well-posed problems](https://en.wikipedia.org/wiki/Well-posed_problem), and are the sorts of assumptions so core in traditional techniques that it is easy to forget about them. But they do matter, as there can be exceptions:

1. no solution - e.g. no $x$ such that $Ax = b$
2. multiple solutions - e.g. several $x_1, x_2, ...$ such that $Ax = b$
3. "chaotic" systems - situations where small changes in initial conditions interact and reverberate in essentially unpredictable ways - for instance, the difficulty in longterm predictions of weather (N.B. not the same thing as longterm predictions of *climate*) - you can think of this as models that fail to generalize well, because they overfit on the training data (the initial conditions)

Problems suffering from the above are called ill-posed problems. Relating to linear algebra and systems of equations, the only truly well-posed problems are those with a single unique solution.

Think for a moment - what would the above plot look like if there was no solution? If there were multiple solutions? And how would that generalize to higher dimensions?

# Well-Posed problems in Linear Algebra

A lot of what you covered with linear regression was about getting matrices into the right shape for them to be solvable in this sense. But some matrices just won't submit to this, and other problems may technically "fit" linear regression but still be violating the above assumptions in subtle ways.

[Overfitting](https://en.wikipedia.org/wiki/Overfitting) is in some ways a special case of this - an overfit model uses more features/parameters than is "justified" by the data (essentially by the *dimensionality* of the data, as measured by $n$ the number of observations). As the number of features approaches the number of observations, linear regression still "works", but it starts giving fairly perverse results. In particular, it results in a model that fails to *generalize* - and so the core goal of prediction and explanatory power is undermined.

How is this related to well and ill-posed problems? It's not clearly a no solution or multiple solution case, but it does fall in the third category - overfitting results in fitting to the "noise" in the data, which means the particulars of one random sample or another (different initial conditions) will result in dramatically different models.

## Two Equations with Two Unknowns (well-posed)

\begin{align}
x-y = -1
\end{align}
\begin{align}
3x+y = 9
\end{align}
\begin{align}
\begin{bmatrix}
1 & -1 \\
3 & 1
\end{bmatrix}
\begin{bmatrix}
x \\
y
\end{bmatrix}
=
\begin{bmatrix}
-1 \\
9
\end{bmatrix}
\end{align}

```
import numpy as np
A = np.array([[1, -1], [3, 1]])
b = [[-1],[9]]

solution = np.linalg.solve(A, b)
print(solution)

x = solution[0][0]
y = solution[1][0]

print('\n')
print("x:", x)
print("y:", y)
```

    [[2.]
     [3.]]

    x: 2.0
    y: 3.0

## Two Equations with Three Unknowns (not well-posed)
\begin{align}
x-y+z = -1
\end{align}
\begin{align}
3x+y-2z = 9
\end{align}
\begin{align}
\begin{bmatrix}
1 & -1 & 1 \\
3 & 1 & -2
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z
\end{bmatrix}
=
\begin{bmatrix}
-1 \\
9
\end{bmatrix}
\end{align}

```
import numpy as np
A = np.array([[1, -1, 1], [3, 1, -2]])
b = [[-1],[9]]

solution = np.linalg.solve(A, b)
print(solution)
```

You can reduce these formulas, but there is no single solution, there are infinitely many solutions where the solution to at least one of these variables must be a function of the other variables.

Example:

<https://www.youtube.com/watch?v=tGPSEXVYw_o>

# Generalization in Machine Learning

The goal of machine learning is to end up with a model that can predict well on new data that it has never seen before. This is sometimes called "out of sample accuracy". This is what we are simulating when we do a train-test-split. We allow or model to fit to the training dataset and then we test its ability to generalize to new data by evaluating its accuracy on a test dataset. We want models that will be usable on new data indefinitely that way we can train them once and then reap the rewards of accurate predictions for a long time to come.

## Underfitting
An underfit model will not perform well on the test data and will also not generalize to new data. Because of this, we can usually detect it easily (it just performs poorly in all situations). Because it's easy to identify we either remedy it quickly or move onto new methods.

```
X = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20])
y = np.array([10,9,8,7,6,5,4,3,2,1,1,2,3,4,5,6,7,8,9,10])

import matplotlib.pyplot as plt
plt.scatter(X,y)
plt.show()
```

```
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)
```

```
plt.scatter(X_train, y_train)
plt.show()
```

```
plt.scatter(X_test, y_test)
plt.show()
```

```
from sklearn.linear_model import LinearRegression

X_train = X_train.reshape(-1, 1)

model = LinearRegression().fit(X_train, y_train)
model.score(X_train, y_train)

beta_0 = model.intercept_
beta_1 = model.coef_[0]

print("Slope Coefficient: ", beta_1)
print("\nIntercept Value: ", beta_0)

plt.scatter(X_train, y_train)
y_hat = [beta_1*x + beta_0 for x in X]
plt.plot(x, y_hat)
plt.show()
```

```
plt.scatter(X_test, y_test)
y_hat = [beta_1*x + beta_0 for x in X]
plt.plot(x, y_hat)
plt.show()
```

## Overfitting

Lets explore the problem of overfitting (and possible remedy - Ridge Regression) in the context of some housing data.

```
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.preprocessing import scale

# Load and Scale the Data
boston = load_boston()
boston.data = scale(boston.data)  # Very helpful for regularization!

# Put it in a dataframe
df = pd.DataFrame(boston.data, columns=boston.feature_names)
df['Price'] = boston.target
df.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      Price

      0
      -0.419782
      0.284830
      -1.287909
      -0.272599
      -0.144217
      0.413672
      -0.120013
      0.140214
      -0.982843
      -0.666608
      -1.459000
      0.441052
      -1.075562
      24.0

      1
      -0.417339
      -0.487722
      -0.593381
      -0.272599
      -0.740262
      0.194274
      0.367166
      0.557160
      -0.867883
      -0.987329
      -0.303094
      0.441052
      -0.492439
      21.6

      2
      -0.417342
      -0.487722
      -0.593381
      -0.272599
      -0.740262
      1.282714
      -0.265812
      0.557160
      -0.867883
      -0.987329
      -0.303094
      0.396427
      -1.208727
      34.7

      3
      -0.416750
      -0.487722
      -1.306878
      -0.272599
      -0.835284
      1.016303
      -0.809889
      1.077737
      -0.752922
      -1.106115
      0.113032
      0.416163
      -1.361517
      33.4

      4
      -0.412482
      -0.487722
      -1.306878
      -0.272599
      -0.835284
      1.228577
      -0.511180
      1.077737
      -0.752922
      -1.106115
      0.113032
      0.441052
      -1.026501
      36.2

## preprocessing.scale(x) does the same thing as preprocessing.StandardScaler()

The difference is that `.scale(x)` is a function (lowercase naming convention)

`StandardScaler()` is a class (uppercase naming convention) with some extra functionality, they will both scale our data equally well.

```
?scale
```

```
from sklearn.preprocessing import StandardScaler

# Load and scale the data
boston = load_boston()
scaler = StandardScaler()
boston.data = scaler.fit_transform(boston.data)

# Put it in a dataframe
df = pd.DataFrame(boston.data, columns=boston.feature_names)
df['Price'] = boston.target
df.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      Price

      0
      -0.419782
      0.284830
      -1.287909
      -0.272599
      -0.144217
      0.413672
      -0.120013
      0.140214
      -0.982843
      -0.666608
      -1.459000
      0.441052
      -1.075562
      24.0

      1
      -0.417339
      -0.487722
      -0.593381
      -0.272599
      -0.740262
      0.194274
      0.367166
      0.557160
      -0.867883
      -0.987329
      -0.303094
      0.441052
      -0.492439
      21.6

      2
      -0.417342
      -0.487722
      -0.593381
      -0.272599
      -0.740262
      1.282714
      -0.265812
      0.557160
      -0.867883
      -0.987329
      -0.303094
      0.396427
      -1.208727
      34.7

      3
      -0.416750
      -0.487722
      -1.306878
      -0.272599
      -0.835284
      1.016303
      -0.809889
      1.077737
      -0.752922
      -1.106115
      0.113032
      0.416163
      -1.361517
      33.4

      4
      -0.412482
      -0.487722
      -1.306878
      -0.272599
      -0.835284
      1.228577
      -0.511180
      1.077737
      -0.752922
      -1.106115
      0.113032
      0.441052
      -1.026501
      36.2

```
df.shape
```

    (506, 14)

## OLS Baseline Model

```
# Let's try good old least squares!
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

X = df.drop('Price', axis='columns')
y = df.Price

lin_reg = LinearRegression().fit(X, y)
mean_squared_error(y, lin_reg.predict(X))
```

    21.894831181729206

That seems like a pretty good score, but...

Chances are this doesn't generalize very well. You can verify this by splitting the data to properly test model validity.

```
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=43)

lin_reg_split = LinearRegression().fit(X_train, y_train)

print(mean_squared_error(y, lin_reg_split.predict(X)))
print(mean_squared_error(y_test, lin_reg_split.predict(X_test)))
```

    22.347018673376052
    26.273991426429014

Oops! 💥 - You have overfitting if you are fitting well to training data, but not generalizing well to test data.

### What can we do?

- Use fewer features - sure, but it can be a lot of work to figure out *which* features, and (in cases like this) there may not be any good reason to really favor some features over another.
- Get more data! This is actually a pretty good approach in tech, since apps generate lots of data all the time (and we made this situation by artificially constraining our data). But for case studies, existing data, etc. it won't work.
- **Regularize!**

## Regularization just means "add bias"

OK, there's a bit more to it than that. But that's the core intuition - the problem is the model working "too well", so fix it by making it harder for the model!

It may sound strange - a technique that is purposefully "worse" - but in certain situations, it can really get results.

What's bias? In the context of statistics and machine learning, bias is when a predictive model fails to identify relationships between features and the output. In a word, bias is *underfitting*.

We want to add bias to the model because of the [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) - variance is the sensitivity of a model to the random noise in its training data (i.e. *overfitting*), and bias and variance are naturally (inversely) related. Increasing one will always decrease the other, with regards to the overall generalization error (predictive accuracy on unseen data).

Visually, the result looks like this:

The blue line is overfit, using more dimensions than are needed to explain the data and so much of the movement is based on noise and won't generalize well. The green line still fits the data, but is less susceptible to the noise - depending on how exactly we parameterize "noise" we may throw out actual correlation, but if we balance it right we keep that signal and greatly improve generalizability.

### Look carefully at the above plot and think of ways you can quantify the difference between the blue and green lines...

```
# Now with regularization via ridge regression
from sklearn.linear_model import Ridge

ridge_reg = Ridge().fit(X, y)
mean_squared_error(y, ridge_reg.predict(X))
```

    21.895862166800143

```
# The score is a bit worse than OLS - but that's expected (we're adding bias)
# Let's try split

ridge_reg_split = Ridge(alpha=0).fit(X_train, y_train)
mean_squared_error(y_test, ridge_reg_split.predict(X_test))
```

    26.273991426429053

```
# A little better (to same test split w/OLS) - can we improve it further?
# We just went with defaults, but as always there's plenty of parameters
help(Ridge)
```

    Help on class Ridge in module sklearn.linear_model.ridge:

    class Ridge(_BaseRidge, sklearn.base.RegressorMixin)
     |  Linear least squares with l2 regularization.
     |
     |  Minimizes the objective function::
     |
     |  ||y - Xw||^2_2 + alpha * ||w||^2_2
     |
     |  This model solves a regression model where the loss function is
     |  the linear least squares function and regularization is given by
     |  the l2-norm. Also known as Ridge Regression or Tikhonov regularization.
     |  This estimator has built-in support for multi-variate regression
     |  (i.e., when y is a 2d-array of shape [n_samples, n_targets]).
     |
     |  Read more in the :ref:`User Guide <ridge_regression>`.
     |
     |  Parameters
     |  ----------
     |  alpha : {float, array-like}, shape (n_targets)
     |      Regularization strength; must be a positive float. Regularization
     |      improves the conditioning of the problem and reduces the variance of
     |      the estimates. Larger values specify stronger regularization.
     |      Alpha corresponds to ``C^-1`` in other linear models such as
     |      LogisticRegression or LinearSVC. If an array is passed, penalties are
     |      assumed to be specific to the targets. Hence they must correspond in
     |      number.
     |
     |  fit_intercept : boolean
     |      Whether to calculate the intercept for this model. If set
     |      to false, no intercept will be used in calculations
     |      (e.g. data is expected to be already centered).
     |
     |  normalize : boolean, optional, default False
     |      This parameter is ignored when ``fit_intercept`` is set to False.
     |      If True, the regressors X will be normalized before regression by
     |      subtracting the mean and dividing by the l2-norm.
     |      If you wish to standardize, please use
     |      :class:`sklearn.preprocessing.StandardScaler` before calling ``fit``
     |      on an estimator with ``normalize=False``.
     |
     |  copy_X : boolean, optional, default True
     |      If True, X will be copied; else, it may be overwritten.
     |
     |  max_iter : int, optional
     |      Maximum number of iterations for conjugate gradient solver.
     |      For 'sparse_cg' and 'lsqr' solvers, the default value is determined
     |      by scipy.sparse.linalg. For 'sag' solver, the default value is 1000.
     |
     |  tol : float
     |      Precision of the solution.
     |
     |  solver : {'auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga'}
     |      Solver to use in the computational routines:
     |
     |      - 'auto' chooses the solver automatically based on the type of data.
     |
     |      - 'svd' uses a Singular Value Decomposition of X to compute the Ridge
     |        coefficients. More stable for singular matrices than
     |        'cholesky'.
     |
     |      - 'cholesky' uses the standard scipy.linalg.solve function to
     |        obtain a closed-form solution.
     |
     |      - 'sparse_cg' uses the conjugate gradient solver as found in
     |        scipy.sparse.linalg.cg. As an iterative algorithm, this solver is
     |        more appropriate than 'cholesky' for large-scale data
     |        (possibility to set `tol` and `max_iter`).
     |
     |      - 'lsqr' uses the dedicated regularized least-squares routine
     |        scipy.sparse.linalg.lsqr. It is the fastest and uses an iterative
     |        procedure.
     |
     |      - 'sag' uses a Stochastic Average Gradient descent, and 'saga' uses
     |        its improved, unbiased version named SAGA. Both methods also use an
     |        iterative procedure, and are often faster than other solvers when
     |        both n_samples and n_features are large. Note that 'sag' and
     |        'saga' fast convergence is only guaranteed on features with
     |        approximately the same scale. You can preprocess the data with a
     |        scaler from sklearn.preprocessing.
     |
     |      All last five solvers support both dense and sparse data. However,
     |      only 'sag' and 'saga' supports sparse input when `fit_intercept` is
     |      True.
     |
     |      .. versionadded:: 0.17
     |         Stochastic Average Gradient descent solver.
     |      .. versionadded:: 0.19
     |         SAGA solver.
     |
     |  random_state : int, RandomState instance or None, optional, default None
     |      The seed of the pseudo random number generator to use when shuffling
     |      the data.  If int, random_state is the seed used by the random number
     |      generator; If RandomState instance, random_state is the random number
     |      generator; If None, the random number generator is the RandomState
     |      instance used by `np.random`. Used when ``solver`` == 'sag'.
     |
     |      .. versionadded:: 0.17
     |         *random_state* to support Stochastic Average Gradient.
     |
     |  Attributes
     |  ----------
     |  coef_ : array, shape (n_features,) or (n_targets, n_features)
     |      Weight vector(s).
     |
     |  intercept_ : float | array, shape = (n_targets,)
     |      Independent term in decision function. Set to 0.0 if
     |      ``fit_intercept = False``.
     |
     |  n_iter_ : array or None, shape (n_targets,)
     |      Actual number of iterations for each target. Available only for
     |      sag and lsqr solvers. Other solvers will return None.
     |
     |      .. versionadded:: 0.17
     |
     |  See also
     |  --------
     |  RidgeClassifier : Ridge classifier
     |  RidgeCV : Ridge regression with built-in cross validation
     |  :class:`sklearn.kernel_ridge.KernelRidge` : Kernel ridge regression
     |      combines ridge regression with the kernel trick
     |
     |  Examples
     |  --------
     |  >>> from sklearn.linear_model import Ridge
     |  >>> import numpy as np
     |  >>> n_samples, n_features = 10, 5
     |  >>> np.random.seed(0)
     |  >>> y = np.random.randn(n_samples)
     |  >>> X = np.random.randn(n_samples, n_features)
     |  >>> clf = Ridge(alpha=1.0)
     |  >>> clf.fit(X, y) # doctest: +NORMALIZE_WHITESPACE
     |  Ridge(alpha=1.0, copy_X=True, fit_intercept=True, max_iter=None,
     |        normalize=False, random_state=None, solver='auto', tol=0.001)
     |
     |  Method resolution order:
     |      Ridge
     |      _BaseRidge
     |      abc.NewBase
     |      sklearn.linear_model.base.LinearModel
     |      abc.NewBase
     |      sklearn.base.BaseEstimator
     |      sklearn.base.RegressorMixin
     |      builtins.object
     |
     |  Methods defined here:
     |
     |  __init__(self, alpha=1.0, fit_intercept=True, normalize=False, copy_X=True, max_iter=None, tol=0.001, solver='auto', random_state=None)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  fit(self, X, y, sample_weight=None)
     |      Fit Ridge regression model
     |
     |      Parameters
     |      ----------
     |      X : {array-like, sparse matrix}, shape = [n_samples, n_features]
     |          Training data
     |
     |      y : array-like, shape = [n_samples] or [n_samples, n_targets]
     |          Target values
     |
     |      sample_weight : float or numpy array of shape [n_samples]
     |          Individual weights for each sample
     |
     |      Returns
     |      -------
     |      self : returns an instance of self.
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes defined here:
     |
     |  __abstractmethods__ = frozenset()
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from sklearn.linear_model.base.LinearModel:
     |
     |  predict(self, X)
     |      Predict using the linear model
     |
     |      Parameters
     |      ----------
     |      X : array_like or sparse matrix, shape (n_samples, n_features)
     |          Samples.
     |
     |      Returns
     |      -------
     |      C : array, shape (n_samples,)
     |          Returns predicted values.
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from sklearn.base.BaseEstimator:
     |
     |  __getstate__(self)
     |
     |  __repr__(self)
     |      Return repr(self).
     |
     |  __setstate__(self, state)
     |
     |  get_params(self, deep=True)
     |      Get parameters for this estimator.
     |
     |      Parameters
     |      ----------
     |      deep : boolean, optional
     |          If True, will return the parameters for this estimator and
     |          contained subobjects that are estimators.
     |
     |      Returns
     |      -------
     |      params : mapping of string to any
     |          Parameter names mapped to their values.
     |
     |  set_params(self, **params)
     |      Set the parameters of this estimator.
     |
     |      The method works on simple estimators as well as on nested objects
     |      (such as pipelines). The latter have parameters of the form
     |      ``<component>__<parameter>`` so that it's possible to update each
     |      component of a nested object.
     |
     |      Returns
     |      -------
     |      self
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors inherited from sklearn.base.BaseEstimator:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from sklearn.base.RegressorMixin:
     |
     |  score(self, X, y, sample_weight=None)
     |      Returns the coefficient of determination R^2 of the prediction.
     |
     |      The coefficient R^2 is defined as (1 - u/v), where u is the residual
     |      sum of squares ((y_true - y_pred) ** 2).sum() and v is the total
     |      sum of squares ((y_true - y_true.mean()) ** 2).sum().
     |      The best possible score is 1.0 and it can be negative (because the
     |      model can be arbitrarily worse). A constant model that always
     |      predicts the expected value of y, disregarding the input features,
     |      would get a R^2 score of 0.0.
     |
     |      Parameters
     |      ----------
     |      X : array-like, shape = (n_samples, n_features)
     |          Test samples. For some estimators this may be a
     |          precomputed kernel matrix instead, shape = (n_samples,
     |          n_samples_fitted], where n_samples_fitted is the number of
     |          samples used in the fitting for the estimator.
     |
     |      y : array-like, shape = (n_samples) or (n_samples, n_outputs)
     |          True values for X.
     |
     |      sample_weight : array-like, shape = [n_samples], optional
     |          Sample weights.
     |
     |      Returns
     |      -------
     |      score : float
     |          R^2 of self.predict(X) wrt. y.

How to tune alpha? For now, let's loop and try values.

(For longterm/stretch/next week, check out [cross-validation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html#sklearn.linear_model.RidgeCV).)

```
alphas = []
mses = []

for alpha in range(0, 200, 1):
  ridge_reg_split = Ridge(alpha=alpha).fit(X_train, y_train)
  mse = mean_squared_error(y_test, ridge_reg_split.predict(X_test))
  print(alpha, mse)
  alphas.append(alpha)
  mses.append(mse)
```

    0 26.273991426429053
    1 26.192201358877668
    2 26.118328007697226
    3 26.051117952293595
    4 25.989569283205444
    5 25.93287356811407
    6 25.880372753122625
    7 25.831526788692837
    8 25.785889053385123
    9 25.743087513207442
    10 25.702810145277628
    11 25.66479356379559
    12 25.628814073392263
    13 25.59468057863174
    14 25.56222892458644
    15 25.53131734932431
    16 25.50182280665663
    17 25.473637974726547
    18 25.44666880864133
    19 25.420832527348388
    20 25.396055949160893
    21 25.372274108781504
    22 25.349429102822004
    23 25.327469121742823
    24 25.30634763462536
    25 25.286022699825878
    26 25.266456379775086
    27 25.24761424230921
    28 25.229464934192976
    29 25.211979815108453
    30 25.19513264248022
    31 25.178899299197408
    32 25.163257557659424
    33 25.14818687468406
    34 25.13366821272317
    35 25.11968388357409
    36 25.106217411385522
    37 25.093253412260974
    38 25.080777488180427
    39 25.068776133307583
    40 25.057236651039766
    41 25.046147080399017
    42 25.035496130566347
    43 25.02527312253186
    44 25.015467936977434
    45 25.006070967630855
    46 24.997073079433843
    47 24.98846557095439
    48 24.980240140548986
    49 24.972388855844812
    50 24.96490412616673
    51 24.95777867758141
    52 24.951005530271846
    53 24.944577977990345
    54 24.9384895693689
    55 24.9327340908919
    56 24.92730555135946
    57 24.92219816768905
    58 24.917406351921144
    59 24.912924699309215
    60 24.9087479773882
    61 24.904871115926827
    62 24.901289197679855
    63 24.897997449864803
    64 24.8949912362963
    65 24.892266050117623
    66 24.889817507075655
    67 24.88764133929069
    68 24.885733389477625
    69 24.88408960557926
    70 24.882706035776263
    71 24.88157882384208
    72 24.880704204813682
    73 24.880078500952195
    74 24.879698117969724
    75 24.879559541500758
    76 24.87965933379892
    77 24.879994130641087
    78 24.8805606384229
    79 24.881355631430836
    80 24.882375949277577
    81 24.88361849448833
    82 24.88508023022692
    83 24.886758178151386
    84 24.888649416389928
    85 24.89075107762813
    86 24.89306034730016
    87 24.895574461876226
    88 24.898290707239912
    89 24.90120641714914
    90 24.90431897177517
    91 24.907625796314402
    92 24.911124359668285
    93 24.914812173186736
    94 24.918686789471128
    95 24.92274580123304
    96 24.92698684020521
    97 24.93140757610152
    98 24.93600571562298
    99 24.94077900150688
    100 24.945725211616683
    101 24.950842158070053
    102 24.95612768640294
    103 24.96157967476758
    104 24.96719603316249
    105 24.972974702692664
    106 24.97891365485829
    107 24.985010890870456
    108 24.99126444099231
    109 24.99767236390434
    110 25.004232746092597
    111 25.010943701258537
    112 25.017803369749362
    113 25.02480991800798
    114 25.0319615380414
    115 25.03925644690685
    116 25.04669288621445
    117 25.05426912164612
    118 25.061983442489353
    119 25.069834161185728
    120 25.077819612893087
    121 25.085938155060855
    122 25.09418816701804
    123 25.102568049573183
    124 25.111076224625787
    125 25.119711134788766
    126 25.128471243021377
    127 25.137355032272303
    128 25.14636100513223
    129 25.155487683495902
    130 25.164733608232837
    131 25.174097338866744
    132 25.183577453263027
    133 25.193172547324206
    134 25.2028812346929
    135 25.212702146462046
    136 25.222633930892243
    137 25.232675253135735
    138 25.24282479496694
    139 25.25308125451928
    140 25.26344334602802
    141 25.273909799578966
    142 25.284479360862818
    143 25.29515079093497
    144 25.305922865980495
    145 25.31679437708437
    146 25.32776413000649
    147 25.338830944961526
    148 25.349993656403374
    149 25.361251112814
    150 25.37260217649681
    151 25.384045723373994
    152 25.39558064278813
    153 25.407205837307533
    154 25.418920222535693
    155 25.430722726924202
    156 25.442612291589445
    157 25.45458787013271
    158 25.466648428463827
    159 25.478792944627976
    160 25.491020408635883
    161 25.50332982229701
    162 25.515720199055906
    163 25.528190563831558
    164 25.540739952859465
    165 25.5533674135368
    166 25.56607200427009
    167 25.578852794325684
    168 25.591708863682904
    169 25.604639302889613
    170 25.617643212920317
    171 25.63071970503678
    172 25.643867900650903
    173 25.657086931189966
    174 25.670375937964163
    175 25.68373407203625
    176 25.697160494093474
    177 25.71065437432154
    178 25.724214892280617
    179 25.737841236783435
    180 25.751532605775324
    181 25.765288206216116
    182 25.779107253964067
    183 25.792988973661497
    184 25.8069325986223
    185 25.820937370721257
    186 25.83500254028498
    187 25.849127365984653
    188 25.863311114730404
    189 25.87755306156723
    190 25.891852489572667
    191 25.906208689755893
    192 25.920620960958406
    193 25.93508860975623
    194 25.949610950363557
    195 25.964187304537848
    196 25.97881700148633
    197 25.99349937777395
    198 26.008233777232597
    199 26.023019550871716

```
from matplotlib.pyplot import scatter
scatter(alphas, mses);
```

## What's the intuition? What are we doing?

The `alpha` parameter corresponds to the weight being given to the extra penalty being calculated by [Tikhonov regularization](https://en.wikipedia.org/wiki/Tikhonov_regularization) (this parameter is sometimes referred to as $\lambda$ in the context of ridge regression).

Normal linear regression (OLS) minimizes the **sum of square error of the residuals**.

Ridge regression minimizes the **sum of square error of the residuals** *AND* **the squared slope of the fit model, times the alpha parameter**.

This is why the MSE for the first model in the for loop (`alpha=0`) is the same as the MSE for linear regression - it's the same model!

As `alpha` is increased, we give more and more penalty to a steep slope. In two or three dimensions this is fairly easy to visualize - beyond, think of it as penalizing coefficient size. Each coefficient represents the slope of an individual dimension (feature) of the model, so ridge regression is just squaring and summing those.

So while `alpha=0` reduces to OLS, as `alpha` approaches infinity eventually the penalty gets so extreme that the model will always output every coefficient as 0 (any non-zero coefficient resulting in a penalty that outweighs whatever improvement in the residuals), and just fit a flat model with intercept at the mean of the dependent variable.

Of course, what we want is somewhere in-between these extremes. Intuitively, what we want to do is apply an appropriate "cost" or penalty to the model for fitting parameters, much like adjusted $R^2$ takes into account the cost of adding complexity to a model. What exactly is an appropriate penalty will vary, so you'll have to put on your model comparison hat and give it a go!

PS - scaling the data helps, as that way this cost is consistent and can be added uniformly across features, and it is simpler to search for the `alpha` parameter.

### Bonus - magic! ✨

Ridge regression doesn't just reduce overfitting and help with the third aspect of well-posed problems (poor generalizability). It can also fix the first two (no unique solution)!

```
df_tiny = df.sample(10, random_state=27)
print(df_tiny.shape)
X = df_tiny.drop('Price', axis='columns')
y = df_tiny.Price

lin_reg = LinearRegression().fit(X, y)
lin_reg.score(X, y)  # Perfect multi-collinearity!
# NOTE - True OLS would 💥 here
# scikit protects us from actual error, but still gives a poor model
```

    (10, 14)

    1.0

```
ridge_reg = Ridge().fit(X, y)
ridge_reg.score(X, y)  # More plausible (not "perfect")
```

    0.9760119331942763

```
# Using our earlier test split
mean_squared_error(y_test, lin_reg.predict(X_test))
```

    103.04429449784261

```
# Ridge generalizes *way* better (and we've not even tuned alpha)
mean_squared_error(y_test, ridge_reg.predict(X_test))
```

    41.79869373639458

```
# e.g. (x1^2 + x2^2 + ...) * alpha is the extra penalty from Ridge
```

```
from sklearn.linear_model import RidgeCV
ridgecv = RidgeCV(alphas=[1e-3, 1e-2, 1e-1, 1, 10, 100, 1000]).fit(X, y)
ridgecv.score(X, y)
mean_squared_error(y_test, ridgecv.predict(X_test))
```

    35.425045824319774

## And a bit more math

The regularization used by Ridge Regression is also known as **$L^2$ regularization**, due to the squaring of the slopes being summed. This corresponds to [$L^2$ space](https://en.wikipedia.org/wiki/Square-integrable_function), a metric space of square-integrable functions that generally measure what we intuitively think of as "distance" (at least, on a plane) - what is referred to as Euclidean distance.

The other famous norm is $L^1$, also known as [taxicab geometry](https://en.wikipedia.org/wiki/Taxicab_geometry), because it follows the "grid" to measure distance like a car driving around city blocks (rather than going directly like $L^2$). When referred to as a distance this is called "Manhattan distance", and can be used for regularization (see [LASSO](https://en.wikipedia.org/wiki/Lasso_(statistics%29), which [uses the $L^1$ norm](https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization-How-does-it-solve-the-problem-of-overfitting-Which-regularizer-to-use-and-when)).

All this comes down to - regularization means increasing model bias by "watering down" coefficients with a penalty typically based on some sort of distance metric, and thus reducing variance (overfitting the model to the noise in the data). It gives us another lever to try and another tool for our toolchest!

## Putting it all together - one last example

The official scikit-learn documentation has many excellent examples - [this one](https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols_ridge_variance.html#sphx-glr-auto-examples-linear-model-plot-ols-ridge-variance-py) illustrates how ridge regression effectively reduces the variance, again by increasing the bias, penalizing coefficients to reduce the effectiveness of features (but also the impact of noise).

```
Due to the few points in each dimension and the straight line that linear regression uses to follow these points as well as it can, noise on the observations will cause great variance as shown in the first plot. Every line’s slope can vary quite a bit for each prediction due to the noise induced in the observations.

Ridge regression is basically minimizing a penalised version of the least-squared function. The penalising shrinks the value of the regression coefficients. Despite the few data points in each dimension, the slope of the prediction is much more stable and the variance in the line itself is greatly reduced, in comparison to that of the standard linear regression
```

```
# Code source: Gaël Varoquaux
# Modified for documentation by Jaques Grobler
# License: BSD 3 clause

import numpy as np
import matplotlib.pyplot as plt

from sklearn import linear_model

X_train = np.c_[.5, 1].T
y_train = [.5, 1]
X_test = np.c_[0, 2].T

np.random.seed(0)

classifiers = dict(ols=linear_model.LinearRegression(),
                   ridge=linear_model.Ridge(alpha=.1))

for name, clf in classifiers.items():
    fig, ax = plt.subplots(figsize=(4, 3))

    for _ in range(6):
        this_X = .1 * np.random.normal(size=(2, 1)) + X_train
        clf.fit(this_X, y_train)

        ax.plot(X_test, clf.predict(X_test), color='gray')
        ax.scatter(this_X, y_train, s=3, c='gray', marker='o', zorder=10)

    clf.fit(X_train, y_train)
    ax.plot(X_test, clf.predict(X_test), linewidth=2, color='blue')
    ax.scatter(X_train, y_train, s=30, c='red', marker='+', zorder=10)

    ax.set_title(name)
    ax.set_xlim(0, 2)
    ax.set_ylim((0, 1.6))
    ax.set_xlabel('X')
    ax.set_ylabel('y')

    fig.tight_layout()

plt.show()
```

Between the first and the second graph, we have decreased the slope (penalized or watered-down coefficients), but we have less variance between our lines)

# Assignment

Following is data describing characteristics of blog posts, with a target feature of how many comments will be posted in the following 24 hours.

https://archive.ics.uci.edu/ml/datasets/BlogFeedback

Investigate - you can try both linear and ridge. You can also sample to smaller data size and see if that makes ridge more important. Don't forget to scale!

Focus on the training data, but if you want to load and compare to any of the test data files you can also do that.

Note - Ridge may not be that fundamentally superior in this case. That's OK! It's still good to practice both, and see if you can find parameters or sample sizes where ridge does generalize and perform better.

When you've fit models to your satisfaction, answer the following question:

```
Did you find cases where Ridge performed better? If so, describe (alpha parameter, sample size, any other relevant info/processing). If not, what do you think that tells you about the data?
```

You can create whatever plots, tables, or other results support your argument. In this case, your target audience is a fellow data scientist, *not* a layperson, so feel free to dig in!

```
# TODO - write some code!
from google.colab import drive
drive.mount('/content/gdrive')
```

    Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code

    Enter your authorization code:
    ··········
    Mounted at /content/gdrive

```
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import RidgeCV
from sklearn.linear_model import Ridge
from sklearn.preprocessing import scale
from sklearn.metrics import mean_squared_error
```

```
import pandas as pd

df = pd.read_csv('/content/gdrive/My Drive/blogData_train.csv', header=None)
```

```
df.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      271
      272
      273
      274
      275
      276
      277
      278
      279
      280

      0
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0

      1
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      2
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      3
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0

      4
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      27.0

5 rows × 281 columns

```
df.isnull().sum()
```

```
df = scale(df)
```

```
df
```

```
df = pd.DataFrame(df)
```

```
df
```

```
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=45)

```

```

```

```
y = df[280]
```

```
X = df.drop(280, axis='columns')
```

```
X.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      270
      271
      272
      273
      274
      275
      276
      277
      278
      279

      0
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      1
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      2
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      3
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      4
      40.30467
      53.845657
      0.0
      401.0
      15.0
      15.52416
      32.44188
      0.0
      377.0
      3.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

5 rows × 280 columns

```
lin_reg_split = LinearRegression().fit(X_train, y_train)

print(mean_squared_error(y, lin_reg_split.predict(X)))
print(mean_squared_error(y_test, lin_reg_split.predict(X_test)))
```

    0.6367428134255578
    0.602491653804951

```
ridge = Ridge(300).fit(X_train, y_train)

print(mean_squared_error(y_train, ridge.predict(X_train)))
print(mean_squared_error(y_test, ridge.predict(X_test)))
```

    0.6502917208779181
    0.6001996018140513

```
ridge_cv = RidgeCV([.001,.01,1,10,100,10000]).fit(X_train, y_train)

print(mean_squared_error(y_train, ridge_cv.predict(X_train)))
print(mean_squared_error(y_test, ridge_cv.predict(X_test)))
```

    0.6535553660154282
    0.5988194186410885

# Resources and stretch goals

Resources:
- https://www.quora.com/What-is-regularization-in-machine-learning
- https://blogs.sas.com/content/subconsciousmusings/2017/07/06/how-to-use-regularization-to-prevent-model-overfitting/
- https://machinelearningmastery.com/introduction-to-regularization-to-reduce-overfitting-and-improve-generalization-error/
- https://towardsdatascience.com/ridge-and-lasso-regression-a-complete-guide-with-python-scikit-learn-e20e34bcbf0b
- https://stats.stackexchange.com/questions/111017/question-about-standardizing-in-ridge-regression#111022

Stretch goals:
- Revisit past data you've fit OLS models to, and see if there's an `alpha` such that ridge regression results in a model with lower MSE on a train/test split
- Yes, Ridge can be applied to classification! Check out [sklearn.linear_model.RidgeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeClassifier.html#sklearn.linear_model.RidgeClassifier), and try it on a problem you previous approached with a different classifier (note - scikit LogisticRegression also automatically penalizes based on the $L^2$ norm, so the difference won't be as dramatic)
- Implement your own function to calculate the full cost that ridge regression is optimizing (the sum of squared residuals + `alpha` times the sum of squared coefficients) - this alone won't fit a model, but you can use it to verify cost of trained models and that the coefficients from the equivalent OLS (without regularization) may have a higher cost


ECMM426 - Computer Vision / ECMM441 - Machine Vision (Professional)

Workshop 1

Image Processing

Simple examples of image processing concepts on OpenCV.

## Imports

```python
import urllib
import matplotlib, cv2
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (20.0, 10.0)

print('OpenCV version: {}'.format(cv2.__version__))
```

## Download Images
Download some images and prepare for reading

```python
import os
if not os.path.exists('images.zip'):
  !wget --no-check-certificate https://empslocal.ex.ac.uk/people/staff/ad735/ECMM426/images.zip
  !unzip -q images.zip
```

## Image Data Structures in OpenCV
Color images have three channels: red, green and blue

```python
# read an image
img = cv2.imread('images/lena.png')

# show image format (basically a 3-d array of pixel color info, in BGR format)
print('Image shape: {}'.format(img.shape))
print('Image: {}'.format(img))
```

## Color Conversions
By default OpenCV loads images in BGR format.

```python
# show image with matplotlib
plt.imshow(img)
```

```python
# convert image to RGB color space
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

# show image with matplotlib
plt.imshow(img)
```

```python
# convert image to grayscale
gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

print('Image shape: {}'.format(gray_img.shape))
# grayscale image represented as a 2-d array
print(gray_img)
```

Gray images have single channel

```python
# plot the gray image, note the cmap parameter
plt.imshow(gray_img, cmap='gray')
```

## Average color of an image

```python
# find average per row
# np.average() takes in an axis argument which finds the average across that axis.
average_color_per_row = np.average(img, axis=0)

# find average across average per row
average_color = np.average(average_color_per_row, axis=0)

# convert back to uint8
average_color = np.uint8(average_color)
print(average_color)

```

```python
# create 100 x 100 pixel array with average color value
average_color_img = np.array([[average_color]*100]*100, np.uint8)

plt.imshow(average_color_img)
```

## Box Filtering

```python
img = cv2.cvtColor(cv2.imread('images/books.jpg'), cv2.COLOR_BGR2RGB)
plt.imshow(img)
```

```python
gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
blur_img = cv2.blur(gray_img, (10, 10))
plt.subplot(1, 2, 1); plt.imshow(gray_img, cmap='gray')
plt.subplot(1, 2, 2); plt.imshow(blur_img, cmap='gray')
```

### Ringing Artifact

```python
# box filtering with 20x20 kernel
blur = cv2.blur(img, (20, 20))
plt.imshow(blur)
```

```python
gray_img = cv2.imread('images/grass.png')
blur_img = cv2.blur(gray_img, (25, 25))
plt.figure(figsize=(16, 8))
plt.subplot(1, 2, 1); plt.imshow(gray_img, cmap='gray')
plt.subplot(1, 2, 2); plt.imshow(blur_img, cmap='gray')
```

## Gaussian Filtering

```python
img = cv2.cvtColor(cv2.imread('images/oy.jpg'), cv2.COLOR_BGR2RGB)
plt.imshow(img)
```

```python
# preproccess with blurring, with 5x5 kernel (note kernel size should be odd)
img_blur_small = cv2.GaussianBlur(img, (5, 5), 0)
plt.imshow(img_blur_small)
```

```python
img_blur_small = cv2.GaussianBlur(img, (5,5), 25)
plt.imshow(img_blur_small)
```

```python
img_blur_large = cv2.GaussianBlur(img, (15,15), 0)
plt.imshow(img_blur_large)
```

## Unsharp Masking or Sharpening
\begin{align}
I_\text{sharp} &= I_\text{original} + \alpha I_\text{detail}\\
               &= I_\text{original} + \alpha I_\text{original} - \alpha I_\text{blurred}\\
               &=(1+\alpha)I_\text{original} - \alpha I_\text{blurred}
\end{align}

```python
alpha = 0.5
img = cv2.cvtColor(cv2.imread('images/oy.jpg'), cv2.COLOR_BGR2RGB)
img_sharp = cv2.addWeighted(img, 1.0 + alpha, img_blur_large, -alpha, 0)
plt.subplot(1, 3, 1); plt.imshow(img); plt.title('Original');
plt.subplot(1, 3, 2); plt.imshow(img_blur_large); plt.title('Gaussian Blurred');
plt.subplot(1, 3, 3); plt.imshow(img_sharp); plt.title('Sharp');
```

## Median Filtering
First create the function for creating noisy images

```python
def add_sp_noise(image, amount=0.1):
    row, col, ch = image.shape
    s_vs_p = 0.5
    out = np.copy(image)
    # Salt mode
    num_salt = np.ceil(amount * image.size * s_vs_p)
    coords = [np.random.randint(0, i - 1, int(num_salt))
            for i in image.shape]
    out[coords[0], coords[1], coords[2]] = 1

    # Pepper mode
    num_pepper = np.ceil(amount* image.size * (1. - s_vs_p))
    coords = [np.random.randint(0, i - 1, int(num_pepper))
            for i in image.shape]
    out[coords[0], coords[1], coords[2]] = 0
    return out
```

Load an image and apply salt and pepper noise and then try to smooth it with Gaussian and Median filter

```python
img = cv2.cvtColor(cv2.imread('images/coins.jpg'), cv2.COLOR_BGR2RGB)
noisy_img = add_sp_noise(img, amount=0.1)
img_gaus = cv2.GaussianBlur(noisy_img, (5, 5), 3)
img_med = cv2.medianBlur(noisy_img, 5)
plt.subplot(1, 4, 1); plt.imshow(img); plt.title('Original');
plt.subplot(1, 4, 2); plt.imshow(noisy_img); plt.title('Salt & Pepper Noise');
plt.subplot(1, 4, 3); plt.imshow(img_gaus); plt.title('Gaussian Filtered');
plt.subplot(1, 4, 4); plt.imshow(img_med); plt.title('Median Filtered');
```

## Image Gradient
Experiment with negative values. Note `cv2.CV_8U` cannot contain negative values.

```python
import cv2
import numpy as np
from matplotlib import pyplot as plt

img = cv2.imread('images/books.jpg', 0)

laplacian = cv2.Laplacian(img, cv2.CV_8U)
sobelx = cv2.Sobel(img, cv2.CV_8U, 1, 0, ksize=5)
sobely = cv2.Sobel(img, cv2.CV_8U, 0, 1, ksize=5)

plt.subplot(2,2,1),plt.imshow(img, cmap = 'gray')
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,2),plt.imshow(laplacian, cmap = 'gray')
plt.title('Laplacian'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,3),plt.imshow(sobelx, cmap = 'gray')
plt.title('Sobel X'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,4),plt.imshow(sobely, cmap = 'gray')
plt.title('Sobel Y'), plt.xticks([]), plt.yticks([])

plt.show()
```

Note `cv2.CV_64F` can contain negative values.

```python
laplacian = cv2.Laplacian(img, cv2.CV_64F)
sobelx = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=5)
sobely = cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=5)

plt.subplot(2,2,1),plt.imshow(img, cmap = 'gray')
plt.title('Original'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,2),plt.imshow(laplacian, cmap = 'gray')
plt.title('Laplacian'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,3),plt.imshow(sobelx, cmap = 'gray')
plt.title('Sobel X'), plt.xticks([]), plt.yticks([])
plt.subplot(2,2,4),plt.imshow(sobely, cmap = 'gray')
plt.title('Sobel Y'), plt.xticks([]), plt.yticks([])

plt.show()
```

## Edge Detection
Canny edge detector on OpenCV. Usage of edge detection versus thresholding to obtain binary image.

```python
cups = cv2.cvtColor(cv2.imread('images/cups.jpg'), cv2.COLOR_BGR2RGB)
plt.imshow(cups)
```

```python
# preprocess by blurring and grayscale
cups_preprocessed  = cv2.cvtColor(cv2.GaussianBlur(cups, (7,7), 0), cv2.COLOR_RGB2GRAY)
```

```python
# find binary image with thresholding
low_thresh = 120
high_thresh = 200
_, cups_thresh = cv2.threshold(cups_preprocessed, low_thresh, 255, cv2.THRESH_BINARY)
plt.imshow(cv2.cvtColor(cups_thresh, cv2.COLOR_GRAY2RGB))

_, cups_thresh_hi = cv2.threshold(cups_preprocessed, high_thresh, 255, cv2.THRESH_BINARY)
```

```python
# find binary image with edges
cups_edges = cv2.Canny(cups_preprocessed, threshold1=90, threshold2=110)
plt.imshow(cv2.cvtColor(cups_edges, cv2.COLOR_GRAY2RGB))
```

## Binary Thresholding

Binarization converts an image to a two tone (0,255 or 0,1) image. Examples using thresholding on brightness/darkness of grayscale image and on color ranges

### On Grayscale Image

### Global Thresholding

```python
# threshold for grayscale image
img = cv2.imread('images/oy.jpg')
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
_, threshold_img = cv2.threshold(gray_img, 127, 255, cv2.THRESH_BINARY)
plt.figure(figsize=(20,10))
plt.subplot(1, 2, 1); plt.imshow(gray_img, cmap='gray')
plt.subplot(1, 2, 2); plt.imshow(threshold_img, cmap='gray')
```

```python
#threshold on blurred image
gray_blur_img = cv2.cvtColor(img_blur_small, cv2.COLOR_BGR2GRAY)
_, threshold_img_blur = cv2.threshold(gray_blur_img, 100, 255, cv2.THRESH_BINARY)
plt.imshow(cv2.cvtColor(threshold_img_blur, cv2.COLOR_GRAY2RGB))
```

### Adaptive Thresholding
It is local thresholding where threshold is decided in local windows.

```python
# using adaptive threshold instead of global
adaptive_thresh = cv2.adaptiveThreshold(gray_img,255,\
                                         cv2.ADAPTIVE_THRESH_GAUSSIAN_C,\
                                         cv2.THRESH_BINARY,11,2)
plt.imshow(adaptive_thresh, cmap='gray')
```

### On Color Image

```python
# open new Mondrian Piet painting photo
piet = cv2.cvtColor(cv2.imread('images/piet.png'), cv2.COLOR_BGR2RGB)
piet_hsv = cv2.cvtColor(piet, cv2.COLOR_RGB2HSV)
plt.imshow(piet)
```

### Range thresholding
The HSV color space is quite similar to the way in which humans perceive color. Most of the other models define color in relation to the primary colors. The colors used in HSV can be clearly defined by human perception.

```python
# threshold for hue channel in blue range
blue_min = np.array([85, 60, 60], np.uint8)
blue_max = np.array([150, 255, 255], np.uint8)
threshold_blue_img = cv2.inRange(piet_hsv, blue_min, blue_max)

# show threshold bits
plt.imshow(threshold_blue_img, cmap='gray')
```

### Binary Thesholding and Image Masking

```python
upstate = cv2.cvtColor(cv2.imread('images/upstate-ny.jpg'), cv2.COLOR_BGR2RGB)
upstate_hsv = cv2.cvtColor(upstate, cv2.COLOR_RGB2HSV)
plt.imshow(upstate)
```

Note `bitwise_not` to filter out the blue sky

```python
mask_inverse = cv2.inRange(upstate_hsv, blue_min, blue_max)
mask = cv2.bitwise_not(mask_inverse)
plt.imshow(mask, cmap='gray')
```

Use the above mask to select the non sky part

```python
# convert single channel mask back into 3 channels
mask_rgb = cv2.cvtColor(mask, cv2.COLOR_GRAY2RGB)

# perform bitwise and on mask to obtain cut-out image that is not blue
masked_upstate = cv2.bitwise_and(upstate, mask_rgb)

# replace the cut-out parts with white
masked_replace_white = cv2.addWeighted(masked_upstate, 1, \
                                       cv2.cvtColor(mask_inverse, cv2.COLOR_GRAY2RGB), 1, 0)

plt.imshow(masked_replace_white)
```


## Isentropic Vortex

> WORK IN PROGRESS !!!

In this example we are going to solve the Euler equations for an isentropic two-dimensional vortex in a full-periodic square domain. Since the problem is not diffusive, the expected behavior is for the vortex to be convected unchanged forever. This is a useful example for testing the diffusive properties of our methods, as well as its numerical stability.

\begin{equation}
    \begin{array}{c}
        \rho_t + \nabla \cdot (\rho u) = 0 \\
        (\rho \mathbf{u})_t + (\mathbf{u} \cdot \nabla)(\rho \mathbf{u}) + \nabla p = 0 \\
        (\rho e)_t + \nabla \cdot(\mathbf{u} (  \rho e + p )) = 0
    \end{array}
\end{equation}

The inputs to the network will be the independent variables $x$, $y$ and $t$ and the outputs will be the conserved variables $\rho$, $\rho \mathbf{u}$ and $\rho e$ where $\rho$ is the density, $\mathbf{u} = (u, v)$ is the velocity and $e$ is the specific energy.

```
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```
#imports
import math
import numpy as np
import matplotlib.pyplot as plt
import torch
```

First we define our PDE and set the values for training.

```
from nangs.pde import PDE
from nangs.bocos import PeriodicBoco, DirichletBoco, NeumannBoco
from nangs.solutions import MLP

class MyPDE(PDE):
    def __init__(self, inputs, outputs, params):
        super().__init__(inputs, outputs, params)
    def computePDELoss(self, grads, inputs, outputs, params):
        drdt = grads['r']['t']
        r, u, v, p = outputs['r'], outputs['u'], outputs['v'], outputs['p']
        ru = r*u
        drudx = self.computeGrad(ru, 'x')
        drudt = self.computeGrad(ru, 't')
        rv = r*v
        drvdy = self.computeGrad(rv, 'y')
        drvdt = self.computeGrad(rv, 't')
        ruup = r*u*u + p
        druupdx =  self.computeGrad(ruup, 'x')
        ruv = r*u*v
        druvdy =  self.computeGrad(ruv, 'y')
        rvvp = r*v*v + p
        drvvpdy =  self.computeGrad(rvvp, 'y')
        rvu = r*v*u
        drvudx =  self.computeGrad(rvu, 'x')
        re = p/(params['g']-1) + 0.5*r*(u*u + v*v)
        dredt = self.computeGrad(re, 't')
        repu = (re+p)*u
        drepudx = self.computeGrad(repu, 'x')
        repv = (re+p)*v
        drepvdy = self.computeGrad(repv, 'y')

        return [
            drdt + drudx + drvdy,
            drudt + druupdx + druvdy,
            drvdt + drvudx + drvvpdy,
            dredt + drepudx + drepvdy
        ]

# instanciate pde
pde = MyPDE(inputs=['x', 'y', 't'], outputs=['r', 'u', 'v', 'p'], params=['g'])

# define input values
x = np.linspace(-5,5,30)
y = np.linspace(-5,5,30)
t = np.linspace(0,6,40)

pde.setValues({'x': x , 'y': y[1:-1], 't': t[:-1], 'g': np.array([1.4])})
#pde.setValues({'x': x, 'y': y, 't': t}, train=False)
```

Boundary conditions.

```
# periodic b.c for the space dimension

boco = PeriodicBoco('boco_x', {'x': x[:1], 'y': y, 't': t[:-1]}, {'x': x[-1:], 'y': y, 't': t[:-1]})
pde.addBoco(boco)

boco = PeriodicBoco('boco_y', {'x': x, 'y': y[:1], 't': t[:-1]}, {'x': x, 'y': y[-1:], 't': t[:-1]})
pde.addBoco(boco)

# initial condition (dirichlet for temporal dimension)
gamma = 1.4
r0 = np.zeros((len(y)*len(x)))
u0 = np.zeros((len(y)*len(x)))
v0 = np.zeros((len(y)*len(x)))
p0 = np.zeros((len(y)*len(x)))
b = 10
for i, _y in enumerate(y):
    for j, _x in enumerate(x):
        r2 = _x*_x + _y*_y
        ro = (1-(b*(gamma-1))/(8*np.pi*np.pi*gamma)*math.exp(1-r2))**(1/(gamma-1))
        r0[i*len(x) + j] = ro
        u0[i*len(x) + j] = 1 - b/(2*np.pi)*math.exp(0.5*(1-r2))*_y
        v0[i*len(x) + j] = b/(2*np.pi)*math.exp(0.5*(1-r2))*_x
        p0[i*len(x) + j] = ro**gamma

boco = DirichletBoco('initial_condition', {'x': x, 'y': y, 't': t[:1]}, {'r': r0, 'u': u0, 'v': v0, 'p': p0})
pde.addBoco(boco)
```

```
# visualize initial condition

fig, ax = plt.subplots()
u, v = u0.reshape(len(y),len(x)), v0.reshape(len(y),len(x))
q = ax.quiver(x, y, u, v)
ax.quiverkey(q, X=0.3, Y=1.1, U=10, label='Quiver key, length = 10', labelpos='E')
plt.show()
```

```
plt.imshow(r0.reshape(len(y),len(x)))
plt.colorbar()
plt.show()
```

Now we define a topology for our solution and set the training parameters. Then we can find a solution for our PDE.

```
# define solution topology
mlp = MLP(pde.n_inputs, pde.n_outputs, 3, 100)
optimizer = torch.optim.Adam(mlp.parameters(), lr=3e-4)
pde.compile(mlp, optimizer)
```

```
# find the solution
hist = pde.solve()
```

```
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))
ax1.plot(hist['train_loss'], label="train_loss")
#ax1.plot(hist['val_loss'], label="val_loss")
ax1.grid(True)
ax1.legend()
ax1.set_yscale("log")
for boco in pde.bocos:
    ax2.plot(hist['bocos'][boco.name], label=boco.name)
ax2.legend()
ax2.grid(True)
ax2.set_yscale("log")
plt.show()
```

Finally, we can evaluate our solution.

```
# evaluate the solution
x = np.linspace(-5,5,30)
y = np.linspace(-5,5,30)
t = np.array([0])
pde.evaluate({'x': x, 'y': y, 't': t})
r = pde.outputs['r']
plt.imshow(r.reshape(len(y),len(x)))
plt.colorbar()
plt.show()
```

```
x = np.linspace(-5,5,30)
y = np.linspace(-5,5,30)
t = np.linspace(0,4,30)

from matplotlib import animation, rc
rc('animation', html='html5')

def plot(t):
    ax.clear()
    #tit = ax.set_title(f"t = {t:.2f}, l2 = {l2:.5f}", fontsize=14)
    tit = ax.set_title(f"t = {t:.2f}", fontsize=14)
    pde.evaluate({'x': x, 'y': y, 't': np.array([t])})
    ax.imshow(pde.outputs['r'].reshape(len(y),len(x)))
    ax.set_xlabel("x", fontsize=14)
    ax.set_ylabel("y", fontsize=14, rotation=np.pi/2)
    #ax.colorbar()
    return [tit]

def get_anim(fig, ax, t):
    def anim(i):
        return plot(t[i])
    return anim

fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(111, autoscale_on=False)
animate = get_anim(fig, ax, t)
anim = animation.FuncAnimation(fig, animate, frames=len(t), interval=100, blit=True)
```

```
anim
```

```

```


```python
%matplotlib inline
```

Writing mathematical expressions
================================

An introduction to writing mathematical expressions in Matplotlib.

You can use a subset TeX markup in any matplotlib text string by placing it
inside a pair of dollar signs ($).

Note that you do not need to have TeX installed, since Matplotlib ships
its own TeX expression parser, layout engine, and fonts.  The layout engine
is a fairly direct adaptation of the layout algorithms in Donald Knuth's
TeX, so the quality is quite good (matplotlib also provides a ``usetex``
option for those who do want to call out to TeX to generate their text (see
:doc:`/tutorials/text/usetex`).

Any text element can use math text.  You should use raw strings (precede the
quotes with an ``'r'``), and surround the math text with dollar signs ($), as
in TeX. Regular text and mathtext can be interleaved within the same string.
Mathtext can use DejaVu Sans (default), DejaVu Serif, the Computer Modern fonts
(from (La)TeX), `STIX <http://www.stixfonts.org/>`_ fonts (with are designed
to blend well with Times), or a Unicode font that you provide.  The mathtext
font can be selected with the customization variable ``mathtext.fontset`` (see
:doc:`/tutorials/introductory/customizing`)

Here is a simple example::

    # plain text
    plt.title('alpha > beta')

produces "alpha > beta".

Whereas this::

    # math text
    plt.title(r'$\alpha > \beta$')

produces ":mathmpl:`\alpha > \beta`".

Note

Mathtext should be placed between a pair of dollar signs ($). To make it
   easy to display monetary values, e.g., "$100.00", if a single dollar sign
   is present in the entire string, it will be displayed verbatim as a dollar
   sign.  This is a small change from regular TeX, where the dollar sign in
   non-math text would have to be escaped ('\\\$').

Note

While the syntax inside the pair of dollar signs ($) aims to be TeX-like,
   the text outside does not.  In particular, characters such as::

     # $ % & ~ _ ^ \ { } \( \) \[ \]

   have special meaning outside of math mode in TeX.  Therefore, these
   characters will behave differently depending on the rcParam ``text.usetex``
   flag.  See the :doc:`usetex tutorial </tutorials/text/usetex>` for more
   information.

Subscripts and superscripts
---------------------------

To make subscripts and superscripts, use the ``'_'`` and ``'^'`` symbols::

    r'$\alpha_i > \beta_i$'

\begin{align}\alpha_i > \beta_i\end{align}

Some symbols automatically put their sub/superscripts under and over the
operator.  For example, to write the sum of :mathmpl:`x_i` from :mathmpl:`0` to
:mathmpl:`\infty`, you could do::

    r'$\sum_{i=0}^\infty x_i$'

\begin{align}\sum_{i=0}^\infty x_i\end{align}

Fractions, binomials, and stacked numbers
-----------------------------------------

Fractions, binomials, and stacked numbers can be created with the
``\frac{}{}``, ``\binom{}{}`` and ``\genfrac{}{}{}{}{}{}`` commands,
respectively::

    r'$\frac{3}{4} \binom{3}{4} \genfrac{}{}{0}{}{3}{4}$'

produces

\begin{align}\frac{3}{4} \binom{3}{4} \stackrel{}{}{0}{}{3}{4}\end{align}

Fractions can be arbitrarily nested::

    r'$\frac{5 - \frac{1}{x}}{4}$'

produces

\begin{align}\frac{5 - \frac{1}{x}}{4}\end{align}

Note that special care needs to be taken to place parentheses and brackets
around fractions.  Doing things the obvious way produces brackets that are too
small::

    r'$(\frac{5 - \frac{1}{x}}{4})$'

.. math ::

    (\frac{5 - \frac{1}{x}}{4})

The solution is to precede the bracket with ``\left`` and ``\right`` to inform
the parser that those brackets encompass the entire object.::

    r'$\left(\frac{5 - \frac{1}{x}}{4}\right)$'

.. math ::

    \left(\frac{5 - \frac{1}{x}}{4}\right)

Radicals
--------

Radicals can be produced with the ``\sqrt[]{}`` command.  For example::

    r'$\sqrt{2}$'

.. math ::

    \sqrt{2}

Any base can (optionally) be provided inside square brackets.  Note that the
base must be a simple expression, and can not contain layout commands such as
fractions or sub/superscripts::

    r'$\sqrt[3]{x}$'

.. math ::

    \sqrt[3]{x}

Fonts
-----

The default font is *italics* for mathematical symbols.

Note

This default can be changed using the ``mathtext.default`` rcParam.  This is
   useful, for example, to use the same font as regular non-math text for math
   text, by setting it to ``regular``.

To change fonts, e.g., to write "sin" in a Roman font, enclose the text in a
font command::

    r'$s(t) = \mathcal{A}\mathrm{sin}(2 \omega t)$'

\begin{align}s(t) = \mathcal{A}\mathrm{sin}(2 \omega t)\end{align}

More conveniently, many commonly used function names that are typeset in
a Roman font have shortcuts.  So the expression above could be written as
follows::

    r'$s(t) = \mathcal{A}\sin(2 \omega t)$'

\begin{align}s(t) = \mathcal{A}\sin(2 \omega t)\end{align}

Here "s" and "t" are variable in italics font (default), "sin" is in Roman
font, and the amplitude "A" is in calligraphy font.  Note in the example above
the calligraphy ``A`` is squished into the ``sin``.  You can use a spacing
command to add a little whitespace between them::

    r's(t) = \mathcal{A}\/\sin(2 \omega t)'

.. Here we cheat a bit: for HTML math rendering, Sphinx relies on MathJax which
   doesn't actually support the italic correction (\/); instead, use a thin
   space (\,) which is supported.

\begin{align}s(t) = \mathcal{A}\,\sin(2 \omega t)\end{align}

The choices available with all fonts are:

    ========================= ================================
    Command                   Result
    ========================= ================================
    ``\mathrm{Roman}``        :mathmpl:`\mathrm{Roman}`
    ``\mathit{Italic}``       :mathmpl:`\mathit{Italic}`
    ``\mathtt{Typewriter}``   :mathmpl:`\mathtt{Typewriter}`
    ``\mathcal{CALLIGRAPHY}`` :mathmpl:`\mathcal{CALLIGRAPHY}`
    ========================= ================================

.. role:: math-stix(mathmpl)
   :fontset: stix

When using the `STIX <http://www.stixfonts.org/>`_ fonts, you also have the
choice of:

    ================================ =========================================
    Command                          Result
    ================================ =========================================
    ``\mathbb{blackboard}``          :math-stix:`\mathbb{blackboard}`
    ``\mathrm{\mathbb{blackboard}}`` :math-stix:`\mathrm{\mathbb{blackboard}}`
    ``\mathfrak{Fraktur}``           :math-stix:`\mathfrak{Fraktur}`
    ``\mathsf{sansserif}``           :math-stix:`\mathsf{sansserif}`
    ``\mathrm{\mathsf{sansserif}}``  :math-stix:`\mathrm{\mathsf{sansserif}}`
    ================================ =========================================

There are also three global "font sets" to choose from, which are
selected using the ``mathtext.fontset`` parameter in `matplotlibrc
<matplotlibrc-sample>`.

``cm``: **Computer Modern (TeX)**

``stix``: **STIX** (designed to blend well with Times)

``stixsans``: **STIX sans-serif**

Additionally, you can use ``\mathdefault{...}`` or its alias
``\mathregular{...}`` to use the font used for regular text outside of
mathtext.  There are a number of limitations to this approach, most notably
that far fewer symbols will be available, but it can be useful to make math
expressions blend well with other text in the plot.

Custom fonts
~~~~~~~~~~~~

mathtext also provides a way to use custom fonts for math.  This method is
fairly tricky to use, and should be considered an experimental feature for
patient users only.  By setting the rcParam ``mathtext.fontset`` to ``custom``,
you can then set the following parameters, which control which font file to use
for a particular set of math characters.

    ============================== =================================
    Parameter                      Corresponds to
    ============================== =================================
    ``mathtext.it``                ``\mathit{}`` or default italic
    ``mathtext.rm``                ``\mathrm{}`` Roman (upright)
    ``mathtext.tt``                ``\mathtt{}`` Typewriter (monospace)
    ``mathtext.bf``                ``\mathbf{}`` bold italic
    ``mathtext.cal``               ``\mathcal{}`` calligraphic
    ``mathtext.sf``                ``\mathsf{}`` sans-serif
    ============================== =================================

Each parameter should be set to a fontconfig font descriptor (as defined in the
yet-to-be-written font chapter).

.. TODO: Link to font chapter

The fonts used should have a Unicode mapping in order to find any
non-Latin characters, such as Greek.  If you want to use a math symbol
that is not contained in your custom fonts, you can set the rcParam
``mathtext.fallback_to_cm`` to ``True`` which will cause the mathtext system
to use characters from the default Computer Modern fonts whenever a particular
character can not be found in the custom font.

Note that the math glyphs specified in Unicode have evolved over time, and many
fonts may not have glyphs in the correct place for mathtext.

Accents
-------

An accent command may precede any symbol to add an accent above it.  There are
long and short forms for some of them.

    ============================== =================================
    Command                        Result
    ============================== =================================
    ``\acute a`` or ``\'a``        :mathmpl:`\acute a`
    ``\bar a``                     :mathmpl:`\bar a`
    ``\breve a``                   :mathmpl:`\breve a`
    ``\ddot a`` or ``\''a``        :mathmpl:`\ddot a`
    ``\dot a`` or ``\.a``          :mathmpl:`\dot a`
    ``\grave a`` or ``\`a``        :mathmpl:`\grave a`
    ``\hat a`` or ``\^a``          :mathmpl:`\hat a`
    ``\tilde a`` or ``\~a``        :mathmpl:`\tilde a`
    ``\vec a``                     :mathmpl:`\vec a`
    ``\overline{abc}``             :mathmpl:`\overline{abc}`
    ============================== =================================

In addition, there are two special accents that automatically adjust to the
width of the symbols below:

    ============================== =================================
    Command                        Result
    ============================== =================================
    ``\widehat{xyz}``              :mathmpl:`\widehat{xyz}`
    ``\widetilde{xyz}``            :mathmpl:`\widetilde{xyz}`
    ============================== =================================

Care should be taken when putting accents on lower-case i's and j's.  Note that
in the following ``\imath`` is used to avoid the extra dot over the i::

    r"$\hat i\ \ \hat \imath$"

\begin{align}\hat i\ \ \hat \imath\end{align}

Symbols
-------

You can also use a large number of the TeX symbols, as in ``\infty``,
``\leftarrow``, ``\sum``, ``\int``.

.. math_symbol_table::

If a particular symbol does not have a name (as is true of many of the more
obscure symbols in the STIX fonts), Unicode characters can also be used::

   ur'$\u23ce$'

Example
-------

Here is an example illustrating many of these features in context.

.. figure:: ../../gallery/pyplots/images/sphx_glr_pyplot_mathtext_001.png
   :target: ../../gallery/pyplots/pyplot_mathtext.html
   :align: center
   :scale: 50

   Pyplot Mathtext

```python

```


# Deep Learning 101

This notebook presents the basics concepts that involve the concept of Deep Learning.

1. Linear Regression
* Logistic Regression
* Artificial Neural Networks
* **Deep Neural Networks**
* Convolutional Neural Networks

## 4. Deep Neural Networks

Deep artificial neural networks are represented by the composition of **several hidden layers**.

Researchers agree that deep learning involves more than two transformations from input to output **CAP> 2** [[Schmidhuber, 2014]](https://arxiv.org/pdf/1404.7828.pdf).

$$\begin{align}
        \hat{y} &= f(x) \\
                &= f^{(n)}(\dots f^{(2)}(f^{(1)}(x))) \\
                &= \sigma_n( W_n \dots \sigma_2(W_2 \sigma_1(W_1 x + b_1) + b_2) \dots + b_n)
        \end{align}
          $$

---

## Deep Neural Networks with Keras and TensorFlow

## 1. Load data

#### Load libraries

```python
%matplotlib inline
import matplotlib.pyplot as plt

import numpy as np
```

```python
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import plot_model, to_categorical
```

#### Getting the data

```python
# load dataset
(X_train, y_train), (X_test, y_test) = mnist.load_data()
```

#### Explore visual data

```python
fig = plt.figure()
for i in range(10):
    plt.subplot(2, 5, i+1)
    x_y = X_train[y_train == i]
    plt.imshow(x_y[0], cmap='gray', interpolation='none')
    plt.title("Class %d" % (i))
    plt.xticks([])
    plt.yticks([])

plt.tight_layout()
```

```python
print('X_train.shape', X_train.shape)
print('y_train.shape', y_train.shape)
print('X_test.shape', X_test.shape)
print('y_train.shape', y_test.shape)
```

    X_train.shape (60000, 28, 28)
    y_train.shape (60000,)
    X_test.shape (10000, 28, 28)
    y_train.shape (10000,)

#### Reshaping and normalizing the inputs

```python
# reshaping the inputs
X_train = X_train.reshape(60000, 28*28)
X_test = X_test.reshape(10000, 28*28)

# normalizing the inputs
X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0
print('X_train reshape:', X_train.shape)
print('X_test reshape:', X_test.shape)
```

    X_train reshape: (60000, 784)
    X_test reshape: (10000, 784)

#### Convert class vectors to binary class matrices

```python
# 10 classes
y_train_cat = to_categorical(y_train, 10)
y_test_cat = to_categorical(y_test, 10)
print('y_train_cat shape:', y_train_cat.shape)
print('y_test_cat shape:', y_test_cat.shape)
```

    y_train_cat shape: (60000, 10)
    y_test_cat shape: (10000, 10)

## 2. Define model

#### Add the input-, hidden- and output-layers

```python
# building a linear stack of layers with the sequential model
model = Sequential()

# Add the input layer and hidden layer 1
model.add(Dense(128, input_shape=(784,), activation='relu'))

# Add the hidden layer 2
model.add(Dense(64, activation='relu'))

# Add the output layer
model.add(Dense(10, activation='softmax'))
```

    2021-08-08 20:27:36.214271: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

#### Model visualization

```python
# plot a Keras model
plot_model(model, to_file='img/model04_dnn.png',
           show_shapes=True, show_layer_names=True)

# prints a summary representation of your model
model.summary()
```

    ('Failed to import pydot. You must `pip install pydot` and install graphviz (https://graphviz.gitlab.io/download/), ', 'for `pydotprint` to work.')
    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense (Dense)                (None, 128)               100480
    _________________________________________________________________
    dense_1 (Dense)              (None, 64)                8256
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                650
    =================================================================
    Total params: 109,386
    Trainable params: 109,386
    Non-trainable params: 0
    _________________________________________________________________

## 3. Compile model

```python
# compiling the sequential model
model.compile('rmsprop', loss='categorical_crossentropy',
              metrics=['categorical_accuracy'])
```

## 4. Fit model

```python
# training the model and saving metrics in history
history = model.fit(X_train, y_train_cat,
                    batch_size=256, epochs=50,
                    verbose=2,
                    validation_data=(X_test, y_test_cat))
```

    Epoch 1/50

    2021-08-08 20:27:36.359242: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
    2021-08-08 20:27:36.377824: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 1800000000 Hz

    235/235 - 2s - loss: 0.3847 - categorical_accuracy: 0.8928 - val_loss: 0.1974 - val_categorical_accuracy: 0.9418
    Epoch 2/50
    235/235 - 1s - loss: 0.1675 - categorical_accuracy: 0.9509 - val_loss: 0.1785 - val_categorical_accuracy: 0.9458
    Epoch 3/50
    235/235 - 1s - loss: 0.1180 - categorical_accuracy: 0.9646 - val_loss: 0.1237 - val_categorical_accuracy: 0.9613
    Epoch 4/50
    235/235 - 1s - loss: 0.0889 - categorical_accuracy: 0.9735 - val_loss: 0.1133 - val_categorical_accuracy: 0.9652
    Epoch 5/50
    235/235 - 1s - loss: 0.0715 - categorical_accuracy: 0.9783 - val_loss: 0.0818 - val_categorical_accuracy: 0.9742
    Epoch 6/50
    235/235 - 1s - loss: 0.0587 - categorical_accuracy: 0.9822 - val_loss: 0.0857 - val_categorical_accuracy: 0.9737
    Epoch 7/50
    235/235 - 1s - loss: 0.0492 - categorical_accuracy: 0.9850 - val_loss: 0.0804 - val_categorical_accuracy: 0.9740
    Epoch 8/50
    235/235 - 1s - loss: 0.0421 - categorical_accuracy: 0.9871 - val_loss: 0.0781 - val_categorical_accuracy: 0.9775
    Epoch 9/50
    235/235 - 1s - loss: 0.0350 - categorical_accuracy: 0.9895 - val_loss: 0.0959 - val_categorical_accuracy: 0.9713
    Epoch 10/50
    235/235 - 1s - loss: 0.0306 - categorical_accuracy: 0.9906 - val_loss: 0.0822 - val_categorical_accuracy: 0.9756
    Epoch 11/50
    235/235 - 1s - loss: 0.0256 - categorical_accuracy: 0.9922 - val_loss: 0.0732 - val_categorical_accuracy: 0.9783
    Epoch 12/50
    235/235 - 1s - loss: 0.0215 - categorical_accuracy: 0.9937 - val_loss: 0.0852 - val_categorical_accuracy: 0.9761
    Epoch 13/50
    235/235 - 1s - loss: 0.0185 - categorical_accuracy: 0.9943 - val_loss: 0.0853 - val_categorical_accuracy: 0.9778
    Epoch 14/50
    235/235 - 1s - loss: 0.0154 - categorical_accuracy: 0.9956 - val_loss: 0.0770 - val_categorical_accuracy: 0.9802
    Epoch 15/50
    235/235 - 1s - loss: 0.0136 - categorical_accuracy: 0.9960 - val_loss: 0.0888 - val_categorical_accuracy: 0.9767
    Epoch 16/50
    235/235 - 1s - loss: 0.0116 - categorical_accuracy: 0.9967 - val_loss: 0.0804 - val_categorical_accuracy: 0.9799
    Epoch 17/50
    235/235 - 1s - loss: 0.0099 - categorical_accuracy: 0.9974 - val_loss: 0.0831 - val_categorical_accuracy: 0.9792
    Epoch 18/50
    235/235 - 1s - loss: 0.0077 - categorical_accuracy: 0.9980 - val_loss: 0.0849 - val_categorical_accuracy: 0.9793
    Epoch 19/50
    235/235 - 1s - loss: 0.0070 - categorical_accuracy: 0.9979 - val_loss: 0.0884 - val_categorical_accuracy: 0.9810
    Epoch 20/50
    235/235 - 1s - loss: 0.0063 - categorical_accuracy: 0.9982 - val_loss: 0.1064 - val_categorical_accuracy: 0.9769
    Epoch 21/50
    235/235 - 1s - loss: 0.0049 - categorical_accuracy: 0.9987 - val_loss: 0.1253 - val_categorical_accuracy: 0.9729
    Epoch 22/50
    235/235 - 1s - loss: 0.0045 - categorical_accuracy: 0.9988 - val_loss: 0.0998 - val_categorical_accuracy: 0.9794
    Epoch 23/50
    235/235 - 1s - loss: 0.0036 - categorical_accuracy: 0.9991 - val_loss: 0.1119 - val_categorical_accuracy: 0.9771
    Epoch 24/50
    235/235 - 1s - loss: 0.0037 - categorical_accuracy: 0.9989 - val_loss: 0.0994 - val_categorical_accuracy: 0.9802
    Epoch 25/50
    235/235 - 1s - loss: 0.0030 - categorical_accuracy: 0.9992 - val_loss: 0.1041 - val_categorical_accuracy: 0.9777
    Epoch 26/50
    235/235 - 1s - loss: 0.0031 - categorical_accuracy: 0.9989 - val_loss: 0.1026 - val_categorical_accuracy: 0.9806
    Epoch 27/50
    235/235 - 1s - loss: 0.0022 - categorical_accuracy: 0.9995 - val_loss: 0.1086 - val_categorical_accuracy: 0.9805
    Epoch 28/50
    235/235 - 1s - loss: 0.0024 - categorical_accuracy: 0.9994 - val_loss: 0.1065 - val_categorical_accuracy: 0.9801
    Epoch 29/50
    235/235 - 1s - loss: 0.0023 - categorical_accuracy: 0.9992 - val_loss: 0.1102 - val_categorical_accuracy: 0.9816
    Epoch 30/50
    235/235 - 1s - loss: 0.0020 - categorical_accuracy: 0.9994 - val_loss: 0.1216 - val_categorical_accuracy: 0.9787
    Epoch 31/50
    235/235 - 1s - loss: 0.0016 - categorical_accuracy: 0.9996 - val_loss: 0.1223 - val_categorical_accuracy: 0.9800
    Epoch 32/50
    235/235 - 1s - loss: 0.0012 - categorical_accuracy: 0.9997 - val_loss: 0.1177 - val_categorical_accuracy: 0.9795
    Epoch 33/50
    235/235 - 1s - loss: 0.0012 - categorical_accuracy: 0.9997 - val_loss: 0.1144 - val_categorical_accuracy: 0.9818
    Epoch 34/50
    235/235 - 1s - loss: 0.0014 - categorical_accuracy: 0.9997 - val_loss: 0.1303 - val_categorical_accuracy: 0.9794
    Epoch 35/50
    235/235 - 1s - loss: 0.0016 - categorical_accuracy: 0.9995 - val_loss: 0.1215 - val_categorical_accuracy: 0.9818
    Epoch 36/50
    235/235 - 1s - loss: 0.0011 - categorical_accuracy: 0.9996 - val_loss: 0.1261 - val_categorical_accuracy: 0.9811
    Epoch 37/50
    235/235 - 1s - loss: 0.0010 - categorical_accuracy: 0.9997 - val_loss: 0.1284 - val_categorical_accuracy: 0.9799
    Epoch 38/50
    235/235 - 1s - loss: 0.0013 - categorical_accuracy: 0.9996 - val_loss: 0.1305 - val_categorical_accuracy: 0.9805
    Epoch 39/50
    235/235 - 1s - loss: 9.8454e-04 - categorical_accuracy: 0.9997 - val_loss: 0.1557 - val_categorical_accuracy: 0.9781
    Epoch 40/50
    235/235 - 1s - loss: 7.7010e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1311 - val_categorical_accuracy: 0.9798
    Epoch 41/50
    235/235 - 1s - loss: 7.4921e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1379 - val_categorical_accuracy: 0.9797
    Epoch 42/50
    235/235 - 1s - loss: 0.0011 - categorical_accuracy: 0.9997 - val_loss: 0.1450 - val_categorical_accuracy: 0.9799
    Epoch 43/50
    235/235 - 1s - loss: 9.3898e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1386 - val_categorical_accuracy: 0.9805
    Epoch 44/50
    235/235 - 1s - loss: 9.4533e-04 - categorical_accuracy: 0.9996 - val_loss: 0.1450 - val_categorical_accuracy: 0.9804
    Epoch 45/50
    235/235 - 1s - loss: 8.0124e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1457 - val_categorical_accuracy: 0.9813
    Epoch 46/50
    235/235 - 1s - loss: 7.7941e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1463 - val_categorical_accuracy: 0.9814
    Epoch 47/50
    235/235 - 1s - loss: 7.2025e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1449 - val_categorical_accuracy: 0.9810
    Epoch 48/50
    235/235 - 1s - loss: 5.5038e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1664 - val_categorical_accuracy: 0.9784
    Epoch 49/50
    235/235 - 1s - loss: 8.9560e-04 - categorical_accuracy: 0.9997 - val_loss: 0.1527 - val_categorical_accuracy: 0.9811
    Epoch 50/50
    235/235 - 1s - loss: 9.9526e-04 - categorical_accuracy: 0.9998 - val_loss: 0.1526 - val_categorical_accuracy: 0.9804

## 5. Evaluate model

```python
# plotting the metrics
fig = plt.figure()
plt.subplot(2,1,1)
plt.plot(history.history['categorical_accuracy'])
plt.plot(history.history['val_categorical_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='lower right')

plt.subplot(2,1,2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')

plt.tight_layout()
```

```python
# evaluate model on test data
[test_loss, test_acc] = model.evaluate(X_test, y_test_cat)
print("Evaluation result on Test Data:\nLoss = {}\nAccuracy = {}".format(test_loss, test_acc))
```

    313/313 [==============================] - 0s 554us/step - loss: 0.1526 - categorical_accuracy: 0.9804
    Evaluation result on Test Data:
    Loss = 0.1526349037885666
    Accuracy = 0.980400025844574

## References

* [Deep Learning Book](http://www.deeplearningbook.org)

* [Zero to Deep Learning™ Udemy Video Course](https://github.com/dataweekends/zero_to_deep_learning_udemy)

* [THE MNIST DATABASE](http://yann.lecun.com/exdb/mnist/)


# Gafica

```python
import matplotlib.pyplot as plt
from numpy import *

x = arange(0.0, 10, 0.01)
y = sin(e**x) - cos(x)
plt.plot(x,y)
plt.grid(axis = 'both')
```

# Identificador

```python
Entrada_1 = "678.32 1234 hola mundo234 34 + - "
Arreglo = []
Lista_1 = []
Contador = 0;
Conteo = 0

def Separador(Arreglo, Contador):
    for x in range(len(Entrada_1)):
        Lista_1.insert(x, Entrada_1[x])

    Space = 0
    for x in range(Contador,len(Entrada_1)):
        if Entrada_1[x] != ' ':
            Arreglo.insert(Space, Entrada_1[x])
            Space += 1
        else:
            y = x + 1
            break;
    return y

def Identificador(Arreglo):

    if '1' <= Arreglo[0] <= '9':
        for x in range(0, len(Arreglo)):
            if Arreglo[x] == '.':
                a = 1;
                break;
            else:
                a = 0

        if a == 1:
            Reales = "".join(Arreglo)
            dic_Real = dict([('Real', Reales)])
            print(dic_Real)
            a = 0
        else:
            Entero = "".join(Arreglo)
            dic_Entero = dict([('Entero', Entero)])
            print(dic_Entero)
            a = 0

    if 'a' <= Arreglo[0] <= 'z':
        for x in range(0, len(Arreglo)):
            ID = "".join(Arreglo)
            dic_ID = dict([('Id', ID)])
        print(dic_ID)

    if Arreglo[0] == '+' or Arreglo[0] == '-':
        Operador = "".join(Arreglo)
        dic_Operador = dict([('Op', Operador)])
        print(dic_Operador)

while Conteo < len(Entrada_1):
    Conteo = Separador(Arreglo, Conteo)
    Identificador(Arreglo)
    del Arreglo[:]
```

    {'Real': '678.32'}
    {'Entero': '1234'}
    {'Id': 'hola'}
    {'Id': 'mundo234'}
    {'Entero': '34'}
    {'Op': '+'}
    {'Op': '-'}

# Mensaje

```python
abc = "abcdefghijklmnopqrstuvwxyz"
text = "g fmnc wms bgblr rpylqjyrc gr zw fylb. rfyrq ufyr amknsrcpq ypc dmp. bmgle gr gl zw fylb gq glcddgagclr ylb rfyr'q ufw rfgq rcvr gq qm jmle. sqgle qrpgle.kyicrpylq() gq pcamkkclbcb."
list_resp = []
t_abc = list(abc)
t_oracion = list(text)
s = 0
p = 0
x = 0
y = 0

for x in text:
    s = s + 1
    for y in abc:
        p = p + 1
        if x == y:
            list_resp.insert(s,t_abc[p+1])
            p = 0
            break;
        if x == " ":
            list_resp.insert(s, " ")
            p = 0
            break;
        if x == "y":
            list_resp.insert(s, t_abc[0])
            p = 0
            break;
        if x == "z":
            list_resp.insert(s, t_abc[1])
            p = 0
            break;
        if x == "." or x == "(" or x == ")" or x == "'":
            list_resp.insert(s, x)
            p = 0
            break;

res = "".join(list_resp)
print(res)
```

    i hope you didnt translate it by hand. thats what computers are for. doing it in by hand is inefficient and that's why this text is so long. using string.maketrans() is recommended.

# Sympy

```python
from sympy import *

x,y = symbols('x y')
solve([x + y - 25, x*2 - 14],[x , y])
```

    {x: 7, y: 18}

# Numpy

```python
import numpy as np
a = np.array([[3,0,0],
             [1,2,0],
             [0,1,-1]])

b = np.array([30,18,2])
x = np.linalg.solve(a,b)
print(x)

answer = sum(x)
print(answer)
```

    [10.  4.  2.]
    16.0

```python

```