# A simple demonstration of coregionalization

This notebook shows how to construct a multi-output GP model using GPflow. We will consider a regression problem for functions $f: \mathbb{R}^D \rightarrow \mathbb{R}^P$. We assume that the dataset is of the form $(X_1, f_1), \dots, (X_P, f_P)$, that is, we do not necessarily observe all the outputs for a particular input location (for cases where there are fully observed outputs for each input, see [Multi-output Gaussian processes in GPflow](./multioutput.ipynb) for a more efficient implementation). We allow each $f_i$ to have a different noise distribution by assigning a different likelihood to each.

For this problem, we model $f$ as a *coregionalized* Gaussian process, which assumes a kernel of the form:

\begin{equation}
\textrm{cov}(f_i(X), f_j(X^\prime)) = k(X, X^\prime) \cdot B[i, j].
\end{equation}

The covariance of the $i$th function at $X$ and the $j$th function at $X^\prime$ is a kernel applied at $X$ and $X^\prime$, times the $(i, j)$th entry of a positive definite $P \times P$ matrix $B$. This is known as the **intrinsic model of coregionalization (ICM)** (Bonilla and Williams, 2008).

To make sure that B is positive-definite, we parameterize it as:

\begin{equation}
B = W W^\top + \textrm{diag}(\kappa).
\end{equation}

To build such a model in GPflow, we need to perform the following two steps:

 * Create the kernel function defined previously, using the `Coregion` kernel class.
 * Augment the training data X with an extra column that contains an integer index to indicate which output an observation is associated with. This is essential to make the data work with the `Coregion` kernel.
 * Create a likelihood for each output using the `SwitchedLikelihood` class, which is a container for other likelihoods.
 * Augment the training data Y with an extra column that contains an integer index to indicate which likelihood an observation is associated with.

```python
import gpflow
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

from gpflow.ci_utils import ci_niter

plt.rcParams["figure.figsize"] = (12, 6)
np.random.seed(123)
```

## Data preparation
We start by generating some training data to fit the model with. For this example, we choose the following two correlated functions for our outputs:

\begin{align}
y_1 &= \sin(6x) + \epsilon_1, \qquad \epsilon_1 \sim \mathcal{N}(0, 0.009) \\
y_2 &= \sin(6x + 0.7) + \epsilon_2, \qquad \epsilon_2 \sim \mathcal{N}(0, 0.01)
\end{align}

```python
# make a dataset with two outputs, correlated, heavy-tail noise. One has more noise than the other.
X1 = np.random.rand(100, 1)  # Observed locations for first output
X2 = np.random.rand(50, 1) * 0.5  # Observed locations for second output

Y1 = np.sin(6 * X1) + np.random.randn(*X1.shape) * 0.03
Y2 = np.sin(6 * X2 + 0.7) + np.random.randn(*X2.shape) * 0.1

plt.figure(figsize=(8, 4))
plt.plot(X1, Y1, "x", mew=2)
_ = plt.plot(X2, Y2, "x", mew=2)
```

## Data formatting for the coregionalized model
We add an extra column to our training dataset that contains an index that specifies which output is observed.

```python
# Augment the input with ones or zeros to indicate the required output dimension
X_augmented = np.vstack((np.hstack((X1, np.zeros_like(X1))), np.hstack((X2, np.ones_like(X2)))))

# Augment the Y data with ones or zeros that specify a likelihood from the list of likelihoods
Y_augmented = np.vstack((np.hstack((Y1, np.zeros_like(Y1))), np.hstack((Y2, np.ones_like(Y2)))))
```

## Building the coregionalization kernel
We build a coregionalization kernel with the Matern 3/2 kernel as the base kernel. This acts on the leading ([0]) data dimension of the augmented X values. The `Coregion` kernel indexes the outputs, and acts on the last ([1]) data dimension (indices) of the augmented X values. To specify these dimensions, we use the built-in `active_dims` argument in the kernel constructor. To construct the full multi-output kernel, we take the product of the two kernels (for a more in-depth tutorial on kernel combination, see [Manipulating kernels](./kernels.ipynb)).

```python
output_dim = 2  # Number of outputs
rank = 1  # Rank of W

# Base kernel
k = gpflow.kernels.Matern32(active_dims=[0])

# Coregion kernel
coreg = gpflow.kernels.Coregion(output_dim=output_dim, rank=rank, active_dims=[1])

kern = k * coreg
```

**Note:** W = 0 is a saddle point in the objective, which would result in the value of `W` not being optimized to fit the data.
Hence, by default, the `W` matrix is initialized with 0.1. Alternatively, you could re-initialize the matrix to random entries.

## Constructing the model
The final element in building the model is to specify the likelihood for each output dimension. To do this, use the `SwitchedLikelihood` object in GPflow.

```python
# This likelihood switches between Gaussian noise with different variances for each f_i:
lik = gpflow.likelihoods.SwitchedLikelihood(
    [gpflow.likelihoods.Gaussian(), gpflow.likelihoods.Gaussian()]
)

# now build the GP model as normal
m = gpflow.models.VGP((X_augmented, Y_augmented), kernel=kern, likelihood=lik)

# fit the covariance function parameters
maxiter = ci_niter(10000)
gpflow.optimizers.Scipy().minimize(
    m.training_loss,
    m.trainable_variables,
    options=dict(maxiter=maxiter),
    method="L-BFGS-B",
)
```

          fun: -225.66159909739758
     hess_inv: <11483x11483 LbfgsInvHessProduct with dtype=float64>
          jac: array([ 0.29142903,  0.39240235,  0.04690075, ..., -0.00111807,
            0.02794148,  0.00111627])
      message: b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
         nfev: 1354
          nit: 1230
         njev: 1354
       status: 0
      success: True
            x: array([-0.79611578,  2.21703729, -1.3562529 , ..., -1.7786879 ,
           -7.38844622, -4.83461769])

That's it: the model is trained. Let's plot the model fit to see what's happened.

```python
def plot_gp(x, mu, var, color, label):
    plt.plot(x, mu, color=color, lw=2, label=label)
    plt.fill_between(
        x[:, 0],
        (mu - 2 * np.sqrt(var))[:, 0],
        (mu + 2 * np.sqrt(var))[:, 0],
        color=color,
        alpha=0.4,
    )

def plot(m):
    plt.figure(figsize=(8, 4))
    Xtest = np.linspace(0, 1, 100)[:, None]
    (line,) = plt.plot(X1, Y1, "x", mew=2)
    mu, var = m.predict_f(np.hstack((Xtest, np.zeros_like(Xtest))))
    plot_gp(Xtest, mu, var, line.get_color(), "Y1")

    (line,) = plt.plot(X2, Y2, "x", mew=2)
    mu, var = m.predict_f(np.hstack((Xtest, np.ones_like(Xtest))))
    plot_gp(Xtest, mu, var, line.get_color(), "Y2")

    plt.legend()

plot(m)
```

From the plots, we can see:

 - The first function (blue) has low posterior variance everywhere because there are so many observations, and the noise variance is small.
 - The second function (orange) has higher posterior variance near the data, because the data are more noisy, and very high posterior variance where there are no observations (x > 0.5).
 - The model has done a reasonable job of estimating the noise variance and lengthscale.
 - The model recognises the correlation between the two functions and is able to suggest (with uncertainty) that because x > 0.5 the orange curve should follow the blue curve (which we know to be the truth from the data-generating procedure).

The covariance matrix between outputs is as follows:

```python
B = coreg.output_covariance().numpy()
print("B =", B)
_ = plt.imshow(B)
```

## References

Bonilla, Edwin V., Kian M. Chai, and Christopher Williams. "Multi-task Gaussian process prediction." _Advances in neural information processing systems_. 2008.


```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from approx1D import regression, comparison_plot
from Lagrange import Lagrange_polynomials
import sympy as sym
import numpy as np

x = sym.Symbol('x')
Omega = [0,1]
N_values = 3, 7, 11, 15

for s in 5, 20:
    f = -sym.tanh(s*(x-0.5))  # sympy expression
    for distribution in 'uniform', 'Chebyshev':
        for N in N_values:
            # Compute the points from a 2*N Lagrange polynomial
            dummy, points = Lagrange_polynomials(
                x, 2*N, Omega,
                point_distribution=distribution)
            # Compute phi from N points Lagrange polynomial
            phi, dummy = Lagrange_polynomials(
                x, N, Omega,
                point_distribution=distribution)
            points = np.array(points, dtype=float)
            point_values = -np.tanh(s*(points-0.5))

            u, c = regression(f, phi, points)
            filename = 'tmp_tanh_%d_%d_%s' % (N, s, distribution)
            comparison_plot(f, u, Omega, filename,
                            plot_title='s=%g, N=%d, %s points' %
                            (s, N, distribution),
                            points=points, point_values=point_values,
                            points_legend='%s points' % (2*N))
        # Combine plot files (2x2)
        for ext in 'png', 'pdf':
            cmd = 'doconce combine_images ' + ext + ' '
            cmd += ' '.join([
                'tmp_tanh_%d_%d_%s' % (N, s, distribution)
                for N in N_values])
            cmd += ' tanh_Lagrange_regr_%s_s%s' % (distribution, s)
            os.system(cmd)

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/23_TANH_LAGRANGE_REGRESSION.ipynb


# Customer Assignment Problem

## Objective and Prerequisites

In this notebook you will learn how:

- To select facility locations, based on customer proximity.
- Machine learning can be leveraged to cope with big datasets. To this end, the reader should be familiar with clustering techniques, such as the k-means algorithm.

This modeling example is at the intermediate level, where we assume that you know Python and are familiar with the Gurobi Python API. In addition, you should have some knowledge about building mathematical optimization models.

**Note:** You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip). In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=CommercialDataScience) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=Github&utm_medium=website_JupyterME&utm_campaign=AcademicDataScience) as an *academic user*.

---
## Motivation

Many companies in various industries must, at some point, make strategic decisions about where to build facilities to support their operations. For example:

- Producers of goods need to decide how to design their supply chains – which encompass factories, distribution centers, warehouses, and retail stores.
- Healthcare providers need to determine where to build hospitals to maximize their population coverage.

These are strategic decisions that are difficult to implement and costly to change because they entail long-term investments. Furthermore, these decisions have a significant impact, both in terms of customer satisfaction and cost management. One of the critical factors to consider in this process is the location of the customers that the company is planning to serve.

---
## Problem Description

The Customer Assignment Problem is closely related to the Facility Location Problem, which is concerned with the optimal placement of facilities (from a set of candidate locations) in order to minimize the distance between the company's facilities and the customers. When the facilities have unlimited capacity, customers are assumed to be served by the closest facility.

In cases where the number of customers considered is too big, the customers can be grouped into clusters. Then, the cluster centers can be used in lieu of the individual customer locations. This pre-processing makes the assumption that all customers belonging to a given cluster will be served by the facility assigned to that cluster. The k-means algorithm can be used for this task, which aims to partition $n$ objects into $k$ distinct and non-overlapping clusters.

---
## Solution Approach

Mathematical optimization is a declarative approach where the modeler formulates a mathematical optimization model that captures the key aspects of a complex decision problem. The Gurobi Optimizer solves such models using state-of-the-art mathematics and computer science.

A mathematical optimization model has five components, namely:

- Sets and indices.
- Parameters.
- Decision variables.
- Objective function(s).
- Constraints.

We now present a Binary Integer Programming (BIP) formulation:

### Sets and Indices
$i \in I$: Set of customer clusters.

$j \in J$: Set of potential facility locations.

$\text{Pairings}= \{(i,j) \in I \times J: \text{dist}_{i,j} \leq \text{threshold}\}$: Set of allowed pairings

### Parameters
$\text{threshold} \in \mathbb{R}^+$: Maximum distance for a cluster-facility pairing to be considered.

$\text{max_facilities} \in \mathbb{N}$: Maximum number of facilities to be opened.

$\text{weight}_i \in \mathbb{N}$: Number of customers in cluster $i$.

$\text{dist}_{i,j} \in \mathbb{R}^+$: Distance from cluster $i$ to facility location $j$.

### Decision Variables
$\text{select}_j \in \{0,1\}$: 1 if facility location $j$ is selected; 0 otherwise.

$\text{assign}_{i,j} \in \{0,1\}$: 1 if cluster $i$ is assigned to facility location $j$; 0 otherwise.

### Objective Function

- **Total distance**: Minimize the total distance from clusters to their assigned facility:

\begin{equation}
\text{Min} \quad Z = \sum_{(i,j) \in \text{Pairings}}\text{weight}_i \cdot \text{dist}_{i,j} \cdot \text{assign}_{i,j}
\tag{0}
\end{equation}

### Constraints

- **Facility limit**: The number of facilities built cannot exceed the limit:

\begin{equation}
\sum_{j}\text{select}_j \leq \text{max_facilities}
\tag{1}
\end{equation}

- **Open to assign**: Cluster $i$ can only be assigned to facility $j$ if we decide to build that facility:

\begin{equation}
\text{assign}_{i,j} \leq \text{select}_{j} \quad \forall (i,j) \in \text{Pairings}
\tag{2}
\end{equation}

- **Closest store**: Cluster $i$ must be assigned to exactly one facility:

\begin{equation}
\sum_{j:(i,j) \in \text{Pairings}}\text{assign}_{i,j} = 1 \quad \forall i \in I
\tag{3}
\end{equation}

---
## Python Implementation

### Dataset Generation

In this simple example, we choose random locations for customers and facility candidates. Customers are distributed using Gaussian distributions around a few randomly chosen population centers, whereas facility locations are uniformly distributed.

```python
%matplotlib inline
import random

import gurobipy as gp
from gurobipy import GRB

import matplotlib.pyplot as plt
import numpy as np
from sklearn.cluster import MiniBatchKMeans

# Tested with Gurobi v9.0.0 and Python 3.7.0

seed = 10101
num_customers = 50000
num_candidates = 50
max_facilities = 8
num_clusters = 1000
num_gaussians = 10
threshold = 0.99

random.seed(seed)
customers_per_gaussian = np.random.multinomial(num_customers,
                                               [1/num_gaussians]*num_gaussians)
customer_locs = []
for i in range(num_gaussians):
    # each center coordinate in [-0.5, 0.5]
    center = (random.random()-0.5, random.random()-0.5)
    customer_locs += [(random.gauss(0,.1) + center[0], random.gauss(0,.1) + center[1])
                  for i in range(customers_per_gaussian[i])]
# each candidate coordinate in [-0.5, 0.5]
facility_locs = [(random.random()-0.5, random.random()-0.5)
              for i in range(num_candidates)]
print('First customer location:', customer_locs[0])
```

    First customer location: (0.33164437091949245, -0.2809884943538464)

### Preprocessing
**Clustering**

To limit the size of the optimization model, we group individual customers into clusters and optimize on these clusters. Clusters are computed using the K-means algorithm, as implemented in the scikit-learn package.

```python
kmeans = MiniBatchKMeans(n_clusters=num_clusters, init_size=3*num_clusters,
                         random_state=seed).fit(customer_locs)
memberships = list(kmeans.labels_)
centroids = list(kmeans.cluster_centers_) # Center point for each cluster
weights = list(np.histogram(memberships, bins=num_clusters)[0]) # Number of customers in each cluster
print('First cluster center:', centroids[0])
print('Weights for first 10 clusters:', weights[:10])
```

    First cluster center: [ 0.42402163 -0.02717844]
    Weights for first 10 clusters: [42, 60, 55, 50, 56, 71, 49, 49, 75, 39]

***Viable Customer-Store Pairings***

Some facilities are just too far away from a cluster center to be relevant, so let's heuristically filter all distances that exceed a given `threshold`:

```python
def dist(loc1, loc2):
    return np.linalg.norm(loc1-loc2, ord=2) # Euclidean distance

pairings = {(facility, cluster): dist(facility_locs[facility], centroids[cluster])
            for facility in range(num_candidates)
            for cluster in range(num_clusters)
            if  dist(facility_locs[facility], centroids[cluster]) < threshold}
print("Number of viable pairings: {0}".format(len(pairings.keys())))
```

    Number of viable pairings: 49175

###  Model Deployment

Build facilities from among candidate locations to minimize total distance to cluster centers:

```python
m = gp.Model("Facility location")

# Decision variables: select facility locations
select = m.addVars(range(num_candidates), vtype=GRB.BINARY, name='select')
# Decision variables: assign customer clusters to a facility location
assign = m.addVars(pairings.keys(), vtype=GRB.BINARY, name='assign')

# Deploy Objective Function
# 0. Total distance
obj = gp.quicksum(weights[cluster]
               *pairings[facility, cluster]
               *assign[facility, cluster]
               for facility, cluster in pairings.keys())
m.setObjective(obj, GRB.MINIMIZE)

# 1. Facility limit
m.addConstr(select.sum() <= max_facilities, name="Facility_limit")

# 2. Open to assign
m.addConstrs((assign[facility, cluster] <= select[facility]
             for facility, cluster in pairings.keys()),
            name="Open2assign")

# 3. Closest store
m.addConstrs((assign.sum('*', cluster) == 1
             for cluster in range(num_clusters)),
            name="Closest_store")

# Find the optimal solution
m.optimize()
```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-
    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 50176 rows, 49225 columns and 147575 nonzeros
    Model fingerprint: 0xde8baaab
    Variable types: 0 continuous, 49225 integer (49225 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [2e-02, 1e+02]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 8e+00]
    Presolve time: 0.22s
    Presolved: 50176 rows, 49225 columns, 147575 nonzeros
    Variable types: 0 continuous, 49225 integer (49225 binary)
    Found heuristic solution: objective 9758.1354528

    Deterministic concurrent LP optimizer: primal and dual simplex
    Showing first log only...

    Concurrent spin time: 0.00s

    Solved with dual simplex

    Root relaxation: objective 5.625486e+03, 5174 iterations, 0.94 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0 5625.48604    0  498 9758.13545 5625.48604  42.4%     -    1s
    H    0     0                    5641.7009130 5625.48604  0.29%     -    1s
    H    0     0                    5635.4418602 5625.48604  0.18%     -    2s

    Cutting planes:
      Gomory: 1
      Zero half: 1

    Explored 1 nodes (5199 simplex iterations) in 2.97 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 3: 5635.44 5641.7 9758.14

    Optimal solution found (tolerance 1.00e-04)
    Best objective 5.635441860165e+03, best bound 5.635441860165e+03, gap 0.0000%

## Analysis

Let's plot a map with:

- Customer locations represented as small pink dots.
- Customer cluster centroids represented as large red dots.
- Facility location candidates represented as green dots. Notice that selected locations have black lines emanating from them towards each cluster that is likely to be served by that facility.

```python
plt.figure(figsize=(8,8), dpi=150)
plt.scatter(*zip(*customer_locs), c='Pink', s=0.5)
plt.scatter(*zip(*centroids), c='Red', s=10)
plt.scatter(*zip(*facility_locs), c='Green', s=10)
assignments = [p for p in pairings if assign[p].x > 0.5]
for p in assignments:
    pts = [facility_locs[p[0]], centroids[p[1]]]
    plt.plot(*zip(*pts), c='Black', linewidth=0.1)
```

---
## Conclusions

We learned how mathematical optimization can be used to solve the Customer Assignment Problem. Moreover, it has been shown how machine learning can be used in the pre-processing so as to reduce the computational burden of big datasets. Of course, this comes at a cost, as using fewer clusters will result in coarser approximations to the global optimal solution.

---
## References

1. Drezner, Z., & Hamacher, H. W. (Eds.). (2001). Facility location: applications and theory. Springer Science & Business Media.
2. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. New York: springer.
3. Klose, A., & Drexl, A. (2005). Facility location models for distribution system design. European journal of operational research, 162(1), 4-29.

Copyright © 2020 Gurobi Optimization, LLC


# Modelo matemático del sistema

El péndulo invertido es un ejemplo clásico de control de sistema inestable. En su forma más básica, se puede modelizar el vehículo como un mecanismo con movimiento plano, en el que una barra se conecta a una rueda a través de un motor, como se muestra en la figura:

La rueda tiene radio $r$, masa $m$, y momento de inercia $I$. La barra, cuyo centro de gravedad se encuentra a una distancia $l$ del eje de la rueda, tiene masa $M$ y momento de inercia $J$. Entre la rueda y la barra se introduce un par $\tau_m$ por medio de un motor.

A continuación se desarrollarán las ecuaciones que modelizan la dinámica del sistema, que serán necesarias para poder diseñar el controlador. En este proyecto vamos a emplear un controlador LQR (*Linear Quadratic Regulator*), que requiere que las ecuaciones del sistema estén expresadas de la siguiente forma, denominada representación *state-space*:

$$
\begin{align}
    \dot{\mathbf x} &= \mathbf A\mathbf x + \mathbf B\mathbf u \\
    \mathbf y &= \mathbf C\mathbf x + \mathbf D\mathbf u
\end{align}
$$

La primera ecuación describe la dinámica del sistema, que se modeliza a través de una serie de *estados* contenidos en un vector $\mathbf x$, que son controlados por medio de unas entradas o *inputs* $\mathbf u$. Las matrices $\mathbf A$ y $\mathbf B$ son constantes, de modo que se trata de un sistema de ecuaciones diferenciales lineales con coeficientes constantes.

La segunda ecuación relaciona las salidas o *outputs* $\mathbf y$, es decir, las magnitudes del sistema que queremos conocer y/o controlar, con el estado del sistema y los *inputs*, a través de dos matrices $\mathbf C$ y $\mathbf D$, también constantes.

Como se verá más adelante, nuestro sistema no es lineal, así que habrá que *linealizarlo*, es decir, aproximar sus ecuaciones a un sistema lineal en el formato estándar, que será válido alrededor de la posición de equilibrio.

## Ecuaciones del movimiento

Las ecuaciones del movimiento del sistema las podemos obtener utilizando el método de Lagrange. Para ello, lo primero es establecer un conjunto mínimo de variables que definan el estado del sistema. Si asumimos rodadura sin deslizamiento, el sistema tiene dos grados de libertad, así que se puede describir su estado a partir de las dos coordenadas indicadas en la figura: la posición del eje, $x$, y el ángulo de la barra respecto a la vertical, $\theta$.

En lugar de desarrollar las ecuaciones de Lagrange a mano, se utilizará SymPy, un CAS (*Computer Algebra System*) para Python. Lo primero es cargar los módulos correspondientes e inicializar las variables del sistema. Las variables dependientes del tiempo se definen utilizando el submódulo de Mecánica, para poder utilizar la notación de Newton (puntos sobre las variables para indicar derivación respecto al tiempo):

```python
from sympy import symbols, Matrix, sin, cos, solve, linsolve, lambdify
from sympy.physics.mechanics import dynamicsymbols, init_vprinting

# Activamos notación de Newton
init_vprinting()

# Definimos las constantes del sistema y el tiempo
r, l, M, m, I, Im, J, L, R, g, Kt, Kv, b, t = symbols('r l M m I Im J L R g K_t K_v b t')

# Definimos variables dinámicas (dependientes del tiempo)
x, th, V, i, tm, wm = dynamicsymbols('x theta V i tau_m omega_m')

# Guardamos derivadas respecto al tiempo en variables auxiliares
thp = th.diff(t)
ths = th.diff(t, 2)
xp = x.diff(t)
xs = x.diff(t, 2)
```

Lo primero que hay que hacer es expresar la energía cinética y potencial del sistema en función de las coordenadas generalizadas $x$ y $\theta$:

```python
# Velocidad del CDG de la barra (derivada de la posición)
vb = Matrix([x + l*sin(th), l*cos(th)]).diff(t)

# Energía cinética de la barra
Tb = (M*vb.dot(vb) + J*thp**2)/2

# Energía cinética de la rueda
Tr = (m*xp**2 + I*(xp/r)**2)/2

# Energía potencial de la barra
Vb = M*g*l*cos(th)
```

Después, se obtienen las fuerzas generalizadas $Q_\theta$ y $Q_x$ correspondientes al par $\tau_m$:

```python
# Fuerzas generalizadas asociadas al par del motor
Qt = -tm  # Efecto sobre el ángulo 'theta'
Qx = tm/r # efecto sobre la posición 'x'
```

Con todos estos términos, ya se puede montar la función Lagrangiana y obtener las ecuaciones del movimiento:

```python
# Función Lagrangiana L = T - V
L = Tb + Tr - Vb

# Ecuaciones del movimiento (vector de ecuaciones iguales a cero)
eqs = Matrix([L.diff(thp).diff(t) - L.diff(th) - Qt, L.diff(xp).diff(t) - L.diff(x) - Qx])
```

Simplificando y reordenando términos, se obtiene el siguiente sistema no lineal de ecuaciones diferenciales de segundo orden en $\theta$ y $x$:

$$
\begin{align}
    \left(J + Ml^2\right)\ddot\theta + Ml\cos\theta\ddot x - Mgl\sin\theta &= -\tau_m \\
    Ml\cos\theta\ddot\theta + \left(m + M + \frac{I}{r^2}\right)\ddot{x} - Ml\sin\theta\dot{\theta}^2 &= \frac{\tau_m}{r}
    \label{eq:motion}
\end{align}
$$

Con este sistema de ecuaciones se podría simular la dinámica del sistema para un par motor $\tau_m$ variable en el tiempo, partiendo de unas condiciones iniciales cualesquiera $x_0$, $\dot x_0$, $\theta_0$, $\dot\theta_0$.

## Introducción del modelo de motor

Cuando el par $\tau_m$ (el *input* del sistema) se produce en un motor DC, no se puede establecer su valor directamente. Normalmente, lo que podemos utilizar en su lugar como variable de control $u$ es el voltaje $V$ aplicado a los bornes, o el voltaje promedio si se usa un amplificador con salidas PWM. Pero el par no depende sólo del voltaje, también varía con la velocidad de giro $\omega_m$. Las ecuaciones que modelizan el comportamiento del motor se explican en detalle en el [documento correspondiente](./Motor.ipynb), y son las siguientes:

$$
\begin{align}
    \tau_m &= K_ti - I_m\dot\omega_m - b\omega_m \\
    V &= Ri + K_v\omega_m
\end{align}
$$

donde $i$ es la corriente, $R$ es la resistencia del bobinado, $K_t$ es la constante de par, $K_v$ es la constante de velocidad, $I_m$ es el momento de inercia equivalente del rotor, y $b$ es el coeficiente de rozamiento viscoso.

Lo primero que vamos a hacer es eliminar la corriente de las ecuaciones, despejándola en la segunda ecuación y sustituyéndola en la primera, utilizando la función `subs` de SymPy:

```python
# Ecuación del par del motor (expresión igual a cero)
eqm = Kt*i - Im*wm.diff(t) - b*wm - tm

# Sustituimos la corriente por su valor despejado en la segunda ecuación
eqm = eqm.subs(i, (V - Kv*wm)/R)
```

La ecuación resultante relaciona el par motor con el voltaje y la velocidad de giro. Esta última, que es la que nos proporcionará el *encoder*, se puede calcular a partir de las variables del sistema de la siguiente manera, teniendo en cuenta que se trata de la velocidad angular relativa entre la barra y la rueda:

$$
    \omega_m=\frac{\dot x}{r} - \dot\theta
$$

Si se sustituyen $\omega_m$ y su derivada en la ecuación del motor, se puede expresar el par motor como función del voltaje y las variables del sistema. Una vez despejado $\tau_m$ utilizando la función `solve`, lo sustituimos en las ecuaciones del sistema:

```python
# Sustituimos 'wm' y su derivada en la ecuación del motor
eqm = eqm.subs(wm, xp/r - thp)

# Despejamos el par y lo sustituimos en las ecuaciones del movimiento
eqs = eqs.subs(tm, solve(eqm, tm)[0]).doit()

# Hay que ejecutar la función 'doit()' para que SymPy evalúe completamente todas las derivadas
# Si no se hace, algunas segundas derivadas tipo 'd2x/dt2' quedarán en la forma 'd/dt(dx/dt)'
# En ese caso, al linealizar fallará al calcular el Jacobiano, porque sólo reconoce 'd2x/dt2' como 'xs'
```

## Linealización del modelo alrededor de la posición de equilibrio

Ahora procedemos a linealizar las ecuaciones alrededor de la posición de equilibrio, utilizando los dos primeros términos de la serie de Taylor. La linealizacion de una función vectorial $\mathbf f\left(\mathbf x\right)$ alrededor de una posición de equilbrio $\mathbf x_0$ se obtiene como:

$$
    \mathbf f\left(\mathbf x\right) \approx \mathbf f\left(\mathbf x_0\right) +
    \mathbf F\left(\mathbf x_0\right)\left(\mathbf x - \mathbf x_0\right)
$$

donde la matriz Jacobiana $\mathbf F\left(\mathbf x\right)$ es tal que:

$$
    \mathbf F_{ij}\left(\mathbf x\right) = \frac{\partial \mathbf f_i\left(\mathbf x\right)}{\partial x_j}
$$

Utilizaremos la función `jacobian` de SymPy para calcular automáticamente la matriz Jacobiana. Luego, sólo hay que evaluar $\mathbf f$ y $\mathbf F$ en la posición de equilibrio (cero en nuestro caso) y construir la serie de Taylor:

```python
# Vector de variables
var = Matrix([th, x, thp, xp, ths, xs])

# Matriz Jacobiana del sistema
jac = eqs.jacobian(var)

# Evaluación de ecuaciones y matriz Jacobiana en posición de equilibrio
e0 = eqs.subs({v: 0 for v in var})
j0 = jac.subs({v: 0 for v in var})

# Sistema linealizado: dos primeros términos de la serie de Taylor
lsys = e0 + j0*var
```

## Representación *state-space*

Ya tenemos las ecuaciones del movimiento linealizadas alrededor de la posición de equilibrio, en forma de sistema de ecuaciones diferenciales lineales con coeficientes constantes. Para poder diseñar el controlador, queremos reescribir este sistema usando la estructura estándar $\dot{\mathbf x}=\mathbf{Ax}+\mathbf{B}u$. A partir de aquí consideraremos el *input* $u$ como un escalar, ya que nuestro sistema sólo tiene una entrada: el voltaje del motor $V$. Las ecuaciones del movimiento son de segundo orden, así que tendremos que establecer el vector de estados de forma que se puedan expresar como un sistema de primer orden. En el sistema linealizado aparecen $\theta$, $\dot\theta$, $\ddot\theta$, $\dot x$ y $\ddot x$, de modo que podemos elegir los estados de la siguiente manera:

$$
    \mathbf{x}=
    \begin{bmatrix}
        \theta \\ \dot\theta \\ \dot x
    \end{bmatrix}
    \implies \dot{\mathbf{x}}=
    \begin{bmatrix}
        \dot\theta \\ \ddot\theta \\ \ddot x
    \end{bmatrix}
$$

El primer elemento de $\dot{\mathbf x}$ ya es directamente el segundo de $\mathbf x$. Por lo tanto, sólo nos falta expresar los otros dos elementos $\ddot\theta$ y $\ddot x$ en función de $\mathbf x$ y $u$ . Para eso, hay que resolver un sistema lineal de dos ecuaciones con dos incógnitas, usando la función `linsolve`:

```python
thsxs = Matrix(linsolve(lsys, [xs, ths]).args[0][::-1])
```

Una vez conocemos $\ddot\theta$ y $\ddot x$ en función de $\mathbf x$ y $u$, podemos determinar las matrices constantes $\mathbf A$ y $\mathbf B$. Para calcular $\mathbf A$, ponemos la solución para $\ddot\theta$ y $\ddot x$ en forma matricial, y añadimos una fila encima para $\dot\theta$. Como sabemos que la ecuación es lineal, podemos usar `jacobian` para expresarla en forma matricial:

```python
A = Matrix([[0, 1, 0]]).row_insert(1, thsxs.jacobian([th, thp, xp]))
```

Para calcular $\mathbf B$, hacemos lo mismo pero derivando respecto al *input*, y añadimos un cero en la primera fila:

```python
B = Matrix([0]).row_insert(1, thsxs.jacobian([V]))
```

Con esto ya tenemos las expresiones de las matrices $\mathbf A$ y $\mathbf B$, que dependerán únicamente de los parámetros del sistema. El controlador necesita también las ecuaciones de salida, que dan los *outputs* del sistema en la forma $\mathbf y = \mathbf C\mathbf x + \mathbf Du$. En nuestro caso, los *outputs* (variables que queremos controlar) son directamente los estados $\theta$ y $\dot x$, así que las matrices $\mathbf C$ y $\mathbf D$ son muy fáciles de obtener:

```python
# Matriz de salida: ángulo 'th' y velocidad 'xp' (2x3)
Cn = [[1, 0, 0], [0, 0, 1]]

# Matriz de transferencia directa (2x1)
Dn = [[0], [0]]
```

Si queremos diseñar un controlador para un sistema real, ahora hay que darles valores numéricos. En este ejemplo se muestran unos valores orientativos de los parámetros, los reales habrá que identificarlos lo mejor posible con los medios a nuestro alcance (báscula, medición de oscilaciones, hojas de características, etc.):

```python
import numpy as np

par = {m: 4e-2, M: 3e-1, I: 4e-5, Im: 2e-4, J: 4e-4, l: 2e-2, r: 4e-2, g: 9.81}
par.update({Kt: 2.0e-1, Kv: 1.5e-1, R: 4.0, b: 3e-4})

# Convertimos expresiones simbólicas en funciones de los parámetros y las evaluamos
An = lambdify([par.keys()], A)(par.values())
Bn = lambdify([par.keys()], B)(par.values())

# Guardamos matrices numéricas en fichero, para el controlador
np.savez('./Data/Model.npz', A=An, B=Bn, C=Cn, D=Dn, r=par[r])
```

Una vez generadas las matrices del sistema, se puede proceder al [diseño del controlador](./Control.ipynb).


# Multivariate Linear Regression from scratch
> Multivariate linear regression and PCA from scratch using Pytorch for car price prediction.
- author: "Axel Mendoza"
- categories: [linear-regression, car-price, pca, pytorch, from-scratch]
- toc: false
- comments: true
- badges: true
- image: images/cadillac.jpg

The goal of this experiment is to train a linear model to predict the selling price of a car.
We will use the framework Pytorch for the tensor calculus and the CarDekho dataset that contains information about used cars listed on the website of the same name.
This dataset has 301 unique entities with the following features:
- car name
- year of release
- selling price
- present price
- kilometers driven
- fuel: such as petrol or diesel
- transmission: such as manual or automatic
- owner: how many times the car changed owner

### Read data from csv using Pandas

```python
import os
import sys
import torch
import pandas as pd
import numpy as np
import seaborn as sns
import statsmodels.api as sm
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
```

```python
df = pd.read_csv(os.path.join('data', 'car_data.csv'))
df.head()
```

      Car_Name
      Year
      Selling_Price
      Present_Price
      Kms_Driven
      Fuel_Type
      Seller_Type
      Transmission
      Owner

      0
      ritz
      2014
      3.35
      5.59
      27000
      Petrol
      Dealer
      Manual
      0

      1
      sx4
      2013
      4.75
      9.54
      43000
      Diesel
      Dealer
      Manual
      0

      2
      ciaz
      2017
      7.25
      9.85
      6900
      Petrol
      Dealer
      Manual
      0

      3
      wagon r
      2011
      2.85
      4.15
      5200
      Petrol
      Dealer
      Manual
      0

      4
      swift
      2014
      4.60
      6.87
      42450
      Diesel
      Dealer
      Manual
      0

### Convert categorical variable into indicator variables

```python
f_continuous = df[['Year', 'Selling_Price', 'Present_Price', 'Kms_Driven', 'Owner']]
f_categorical = pd.get_dummies(df[['Fuel_Type', 'Seller_Type', 'Transmission']])
df = pd.concat([f_continuous, f_categorical], axis=1)

# Drop refundant features
df.drop(['Transmission_Automatic', 'Seller_Type_Dealer', 'Fuel_Type_CNG'], axis=1, inplace=True)
df.head()
```

      Year
      Selling_Price
      Present_Price
      Kms_Driven
      Owner
      Fuel_Type_Diesel
      Fuel_Type_Petrol
      Seller_Type_Individual
      Transmission_Manual

      0
      2014
      3.35
      5.59
      27000
      0
      0
      1
      0
      1

      1
      2013
      4.75
      9.54
      43000
      0
      1
      0
      0
      1

      2
      2017
      7.25
      9.85
      6900
      0
      0
      1
      0
      1

      3
      2011
      2.85
      4.15
      5200
      0
      0
      1
      0
      1

      4
      2014
      4.60
      6.87
      42450
      0
      1
      0
      0
      1

### Visualize histogram of all features

```python
df.hist(bins=14, color='steelblue', edgecolor='black', linewidth=1.0, xlabelsize=8, ylabelsize=8, grid=False)
plt.tight_layout(rect=(0, 0, 1.2, 1.2))
```

Most cars on sales are consuming petrol instead of diesel, have had only one owner, are from 2012-present, are manual and most have a selling price between 1000 and 10000 dollars.

### Heatmap correlation

```python
plt.figure(figsize=(16, 8))
sns.heatmap(df.corr(), square= True, annot=True, fmt='.2f')
```

Most of the variables are highly correlated.
As expected, the present price variable is the most correlated with the target selling price.

### Pairwise Plots

```python
cols_viz = ['Kms_Driven', 'Year', 'Selling_Price', 'Present_Price']
pp = sns.pairplot(df[cols_viz], height=1.8, aspect=1.8,
                  plot_kws=dict(edgecolor="k", linewidth=0.5),
                  diag_kind="kde", diag_kws=dict(shade=True))

fig = pp.fig
fig.subplots_adjust(top=0.93, wspace=0.3)
t = fig.suptitle('Wine Attributes Pairwise Plots', fontsize=14)
```

Most of the features are highly correlated to each other.
Some outliers are present but as there is very few, we will keep them in the training set.
The year feature have a polynomial correlation with the selling price so a polynomial regression will most likely outperform a standard linear regression.

### Make train test split

```python

# Separate the target from the dataFrame
Y = df['Selling_Price']
X = df.drop('Selling_Price', axis=1)

# Convert data to Pytorch tensor
X_t = torch.from_numpy(X.to_numpy()).float()
Y_t = torch.from_numpy(Y.to_numpy()).float().unsqueeze(1)
X_train, X_test, Y_train, Y_test = train_test_split(X_t, Y_t, test_size=0.33, random_state=42)
```

### Train a multivariate linear regression
Training a linear model using least square regression is equivalent to minimize the mean squared error:

$$
\begin{align}
    \text{Mse}(\boldsymbol{\hat{y}}, \boldsymbol{y}) &= \frac{1}{n}\sum_{i=1}^{n}{||\hat{y}_i - y_i ||_{2}^{2}} \\
    &= \frac{1}{n}||\boldsymbol{X}\boldsymbol{w} - \boldsymbol{y} ||_2^2
\end{align}
$$

where $n$ is the number of samples, $\hat{y}$ is the predicted value of the model and $y$ is the true target.
The prediction $\hat{y}$ is obtained by matrix multiplication between the input $\boldsymbol{X}$ and the weights of the model $\boldsymbol{w}$.

Minimizing the $\text{Mse}$ can be achieved by solving the gradient of this equation equals to zero in regards to the weights $\boldsymbol{w}$:

$$
\begin{align}
    \nabla_{\boldsymbol{w}}\text{Mse} &= 0 \\
    (\boldsymbol{X}^\top \boldsymbol{X})^{-1}\boldsymbol{X}^\top \boldsymbol{y} &= \boldsymbol{w}
\end{align}
$$

For more information on how to find $\boldsymbol{w}$ please visit the following [link](https://en.wikipedia.org/wiki/Least_squares#Linear_least_squares:~:text=A%20regression%20model%20is%20a%20linear%20one%20when%20the%20model%20comprises%20a%20linear%20combination%20of%20the%20parameters).

```python
def add_ones_col(X):
    """Add a column a one to the input torch tensor"""
    x_0 = torch.ones((X.shape[0],), dtype=torch.float32).unsqueeze(1)
    X = torch.cat([x_0, X], dim=1)
    return X

def multi_linear_reg(X, y):
    """Multivariate linear regression function

    Args:
        X: A torch tensor for the data.
        y: A torch tensor for the labels.
    """
    X = add_ones_col(X)  # Add a column of ones to X to agregate the bias to the input matrices
    Xt_X = X.T.mm(X)
    Xt_y = X.T.mm(y)

    Xt_X_inv = Xt_X.inverse()
    w = Xt_X_inv.mm(Xt_y)
    return w

def prediction(X, w):
    """Predicts a selling price for each input

    Args:
        X: A torch tensor for the data.
        w: A torch tensor for the weights of the linear regression mode.
    """
    X = add_ones_col(X)
    return X.mm(w)
```

```python

# Fit the training set into the model to get the weights
w = multi_linear_reg(X_train, Y_train)

# Predict using matrix multiplication with the weights
Y_pred_train = prediction(X_train, w)
Y_pred_test = prediction(X_test, w)
```

### Compute prediction error

```python
def mse(Y_true, Y_pred):
    error = Y_pred - Y_true
    return (error.T.mm(error) / Y_pred.shape[0]).item()

def mae(Y_true, Y_pred):
    error = Y_pred - Y_true
    return error.abs().mean().item()
```

```python
mse_train = mse(Y_train, Y_pred_train)
mae_train = mae(Y_train, Y_pred_train)
print('MSE Train:\t', mse_train)
print('MAE Train:\t', mae_train, end='\n\n')

mse_test = mse(Y_test, Y_pred_test)
mae_test = mae(Y_test, Y_pred_test)
print('MSE Test:\t', mse_test)
print('MAE Test:\t', mae_test, end='\n\n')
```

    MSE Train:	 2.808985471725464
    MAE Train:	 1.1321566104888916

    MSE Test:	 3.7205495834350586
    MAE Test:	 1.2941011190414429

The model has an error of 1.29 on average on the training test.
Not bad for a linear model, taking into consideration that the mean of the present price is 7.62.

### Principal component analysis visualization
In this section, we will use PCA to reduce the number of feature to two, in order to visualize the plane of the linear regressor.

Suppose a collection of $m$ points $\{\boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)}\}$ in $\mathbb{R}^n$.
The principal components analysis aims to reduce the dimensionality of the points while losing the least precision as possible.
For each point $\boldsymbol{x}^{(i)} \in \mathbb{R}^n$ we will find a corresponding code vector $\boldsymbol{c}^{(i)} \in \mathbb{R}^l$ where $l$ is smaller than $n$.
Let $f$ be the encoding function and $g$ be the decoding function and $\boldsymbol{D} \in \mathbb{R}^{n,l}$ is the decoding matrix whose columns are orthonormal:
$$
\begin{align}
    f(\boldsymbol{x}) &= \boldsymbol{D}^\top \boldsymbol{x} \\
    g(f(\boldsymbol{x})) &= \boldsymbol{D}\boldsymbol{D}^\top \boldsymbol{x}
\end{align}
$$

```python
def cov(X):
    """Computes the covariance of the input

        The covariance  matrix gives some sense of how much two values are
        linearly related to each other, as well as the scale of these variables.
        It is computed by (1 / (N - 1)) * (X - E[X]).T (X - E[X]).

    Args:
        X: A torch tensor as input.
    """
    X -= X.mean(dim=0, keepdim=True)
    fact = 1.0 / (X.shape[0] - 1)
    cov = fact * X.T.mm(X)
    return cov

def pca(X, target_dim=2):
    """Computes the n^th first principal components of the input

    PCA can be implemented using the n^th principal components of the covariance matrix.
    We could have been using an eigen decomposition because the covariance matrix is always squared
    but singular value decomposition does also the trick if we take the right singular vectors
    and perform a matrix multiplication to the right.

    Args:
        X: A torch tensor as the input.
        target_dim: An integer for selecting the n^th first components.
    """
    cov_x = cov(X)

    U, S, V = torch.svd(cov_x)
    transform_mat = V[:, :target_dim]
    X_reduced = X.mm(transform_mat)
    return X_reduced, transform_mat
```

```python
X_test_pca, _ = pca(X_test, target_dim=2)
X_train_pca, _ = pca(X_train, target_dim=2)

points = torch.cat([X_test_pca[:3], Y_pred_test[:3]], axis=1)
v1 = points[2, :] - points[0, :]
v2 = points[1, :] - points[0, :]
cp = torch.cross(v1, v2)
a, b, c = cp
d = cp.dot(points[2, :])

min_mesh_x = min(X_test_pca[:, 0].min(), X_train_pca[:, 0].min())
max_mesh_x = max(X_test_pca[:, 0].max(), X_train_pca[:, 0].max())
min_mesh_y = min(X_test_pca[:, 1].min(), X_train_pca[:, 1].min())
max_mesh_y = max(X_test_pca[:, 1].max(), X_train_pca[:, 1].max())

mesh_x = np.linspace(min_mesh_x, max_mesh_x, 25)
mesh_y = np.linspace(min_mesh_y, max_mesh_y, 25)
mesh_xx, mesh_yy = np.meshgrid(mesh_x, mesh_y)

mesh_zz = (d - a * mesh_xx - b * mesh_yy) / c
```

Here we recreate the prediction plane using three points of the prediction.
More information at this [link](http://kitchingroup.cheme.cmu.edu/blog/2015/01/18/Equation-of-a-plane-through-three-points).

```python
fig = plt.figure(figsize=(25,7))
ax1 = fig.add_subplot(131, projection='3d')
ax2 = fig.add_subplot(132, projection='3d')
ax3 = fig.add_subplot(133, projection='3d')
axes = [ax1, ax2, ax3]

for ax in axes:
    ax.scatter(X_test_pca[:, 0], X_test_pca[:, 1], Y_test, color='red', edgecolor='black')
    ax.scatter(X_train_pca[:, 0], X_train_pca[:, 1], Y_train, color='green', edgecolor='black')
    ax.scatter(mesh_xx.flatten(), mesh_yy.flatten(), mesh_zz.flatten(), facecolor=(0, 0, 0, 0), s=20, edgecolor='#70b3f0')

    ax.set_xlabel('1st component', fontsize=12)
    ax.set_ylabel('2nd component', fontsize=12)
    ax.set_zlabel('Selling Price', fontsize=12)
    ax.ticklabel_format(axis="x", style="sci", scilimits=(0,0))
    ax.ticklabel_format(axis="y", style="sci", scilimits=(0,0))
    ax.ticklabel_format(axis="z", style="sci", scilimits=(0,0))
ax1.view_init(elev=60, azim=50)
ax2.view_init(elev=10, azim=0)
ax3.view_init(elev=-15, azim=140)
```

The plane is fitting pretty well the data !

### Exploratory Data Analysis
I made an attempt to discard some features based on p-value but it didn't improve the results.
More on p-value [here](https://www.statsdirect.com/help/basics/p_values.htm#:~:text=The%20P%20value%2C%20or%20calculated,the%20hypothesis%20is%20being%20tested).

```python
columns = X.columns
while len(columns) > 0:
    pvalues = []
    X_1 = X[columns]
    X_1 = sm.add_constant(X)  # add a columns of ones for the bias
    model = sm.OLS(Y, X_1).fit()  # fit a linear regression
    pvalues = pd.Series(model.pvalues[1:], index=columns)
    max_idx = np.argmax(pvalues)
    max_pval = pvalues[max_idx]
    if max_pval > 0.05:
        # if the p_values is greater than 0.05, the feature has not enough
        # informational value for the training
        columns = columns.drop(columns[max_idx])
        print('Dropping column ' + columns[max_idx] + ', pvalue is: ' + str(max_pval))
    else:
        break
```

    Dropping column Seller_Type_Individual, pvalue is: 0.6296373292654155
    Dropping column Fuel_Type_Diesel, pvalue is: 0.11176717429491591
    Dropping column Seller_Type_Individual, pvalue is: 0.05428653381413104

```python

# Keeping only the columns with very low p-value
X = df[columns]

X_t = torch.from_numpy(X.to_numpy()).float()
X_train, X_test, Y_train, Y_test = train_test_split(X_t, Y_t, test_size=0.33, random_state=42)

w = multi_linear_reg(X_train, Y_train)

Y_pred_train = prediction(X_train, w)
Y_pred_test = prediction(X_test, w)

mse_train = mse(Y_train, Y_pred_train)
mse_test = mse(Y_test, Y_pred_test)
print('MSE Train:\t', mse_train)
print('MSE Test:\t', mse_test)
```

    MSE Train:	 3.4574925899505615
    MSE Test:	 3.8332533836364746

### Conclusion

A linear regression can perform pretty well if the data is highly correlated.
It is an appropriate method as a baseline in the majority of the regression tasks.
The pvalue is not always a good indicator for feature selection.


# 2021-09-20 Nonlinear solvers

## Last time

* Upwinding for robustness to (cell) Péclet number and oscillations
* Effective use of sparse/structured matrices

## Today
* Intro to rootfinding and Newton's method
* Bratu and p-Laplacian problems

```julia
using Plots
using LinearAlgebra
using SparseArrays

function vander(x, k=nothing)
    if k === nothing
        k = length(x)
    end
    V = ones(length(x), k)
    for j = 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function fdstencil(source, target, k)
    "kth derivative stencil from source to target"
    x = source .- target
    V = vander(x)
    rhs = zero(x)'
    rhs[k+1] = factorial(k)
    rhs / V
end

function advdiff_sparse(n, kappa, wind, forcing)
    x = LinRange(-1, 1, n)
    xstag = (x[1:end-1] + x[2:end]) / 2
    rhs = forcing.(x)
    kappa_stag = kappa.(xstag)
    rows = [1, n]
    cols = [1, n]
    vals = [1., 1.] # diagonals entries (float)
    rhs[[1,n]] .= 0 # boundary condition
    for i in 2:n-1
        flux_L = -kappa_stag[i-1] * fdstencil(x[i-1:i], xstag[i-1], 1) +
            wind * (wind > 0 ? [1 0] : [0 1])
        flux_R = -kappa_stag[i] * fdstencil(x[i:i+1], xstag[i], 1) +
            wind * (wind > 0 ? [1 0] : [0 1])
        weights = fdstencil(xstag[i-1:i], x[i], 1)
        append!(rows, [i,i,i])
        append!(cols, i-1:i+1)
        append!(vals, weights[1] *  [flux_L..., 0] + weights[2] * [0, flux_R...])
    end
    L = sparse(rows, cols, vals)
    x, L, rhs
end
```

    advdiff_sparse (generic function with 1 method)

# Nonlinear problems and rootfinding

We want to solve problems like
$$ -(|u_x|^{p-2} u_x)_x + (u^2)_x = 0 $$

For linear problems, we found $u$ such that $L u = b$ using linear algebra.

For nonlinear problems, we'll need to find $u$ such that $F(u) = 0$.

# An algorithm: Bisection

Bisection is a rootfinding technique that starts with an interval $[a,b]$ containing a root and does not require derivatives. Suppose $f$ is continuous. What is a **sufficient** condition for $f$ to have a root on $[a,b]$?

```julia
hasroot(f, a, b) = f(a) * f(b) < 0

f(x) =  cos(x) - x
plot([f, x->0], xlims=(-4,4))
```

```julia
hasroot(f, 0, 1)
```

    true

# Bisection

```julia
function bisect(f, a, b, tol)
    mid = (a + b) / 2
    if abs(b - a) < tol
        return mid
    elseif hasroot(f, a, mid)
        return bisect(f, a, mid, tol)
    else
        return bisect(f, mid, b, tol)
    end
end

x0 = bisect(f, -1, 3, 1e-5)
x0, f(x0)
```

    (0.7390861511230469, -1.7035832658995886e-6)

# How fast does it converge?

```julia
function bisect_hist(f, a, b, tol)
    mid = (a + b) / 2
    if abs(b - a) < tol
        return [mid]
    elseif hasroot(f, a, mid)
        return prepend!(bisect_hist(f, a, mid, tol), [mid])
    else
        return prepend!(bisect_hist(f, mid, b, tol), [mid])
    end
end
```

    bisect_hist (generic function with 1 method)

```julia
bisect_hist(f, -1, 3, 1e-4)
```

    17-element Vector{Float64}:
     1.0
     0.0
     0.5
     0.75
     0.625
     0.6875
     0.71875
     0.734375
     0.7421875
     0.73828125
     0.740234375
     0.7392578125
     0.73876953125
     0.739013671875
     0.7391357421875
     0.73907470703125
     0.739105224609375

# Let's plot the error

$$ \lvert \texttt{bisect}^k(f, a, b) - r \rvert, \quad k = 1, 2, \dotsc $$

where $r$ is the true root, $f(r) = 0$.

```julia
r = bisect(f, -1, 3, 1e-14) # what are we trusting?
hist = bisect_hist(f, -1, 3, 1e-10)
scatter( abs.(hist .- r), yscale=:log10)
ks = 1:length(hist)
plot!(ks, 10 * 0.5 .^ ks)
```

Evidently the error $e_k = x_k - x_*$ after $k$ bisections satisfies the bound
$$ |e^k| \le c 2^{-k} . $$

# Convergence classes

A convergent rootfinding algorithm produces a sequence of approximations $x_k$ such that $$\lim_{k \to \infty} x_k \to x_*$$ where $f(x_*) = 0$.  For analysis, it is convenient to define the errors $e_k = x_k - x_*$. We say that an iterative algorithm is **$q$-linearly convergent** if $$\lim_{k \to \infty} |e_{k+1}| / |e_k| = \rho < 1.$$  (The $q$ is for "quotient".)  A smaller convergence factor $\rho$ represents faster convergence.  A slightly weaker condition ($r$-linear convergence or just **linear convergence**) is that
$$ |e_k| \le \epsilon_k $$
for all sufficiently large $k$ when the sequence $\{\epsilon_k\}$ converges $q$-linearly to 0.

### Bisection: A = q-linearly convergent, B = r-linearly convergent, C = neither

# Remarks on bisection

* Specifying an interval is often inconvenient
* An interval in which the function changes sign guarantees convergence (robustness)
* No derivative information is required
* Roots of even degree are problematic
* A bound on the solution error is directly available
* The convergence rate is modest -- one iteration per bit of accuracy
* **No good generalization to higher dimensions**

## Newton-Raphson Method

Much of numerical analysis reduces to [Taylor series](https://en.wikipedia.org/wiki/Taylor_series), the approximation
$$ f(x) = f(x_0) + f'(x_0) (x-x_0) + f''(x_0) (x - x_0)^2 / 2 + \underbrace{\dotsb}_{O((x-x_0)^3)} $$
centered on some reference point $x_0$.

In numerical computation, it is exceedingly rare to look beyond the first-order approximation
$$ \tilde f_{x_0}(x) = f(x_0) + f'(x_0)(x - x_0) . $$
Since $\tilde f_{x_0}(x)$ is a linear function, we can explicitly compute the unique solution of $\tilde f_{x_0}(x) = 0$ as
$$ x = x_0 - f(x_0) / f'(x_0) . $$
This is Newton's Method (aka Newton-Raphson or Newton-Raphson-Simpson) for finding the roots of differentiable functions.

# An implementation

```julia
function newton(f, fp, x0; tol=1e-8, verbose=false)
    x = x0
    for k in 1:100 # max number of iterations
        fx = f(x)
        fpx = fp(x)
        if verbose
            println("[$k] x=$x  f(x)=$fx  f'(x)=$fpx")
        end
        if abs(fx) < tol
            return x, fx, k
        end
        x = x - fx / fpx
    end
end
eps = 1e-1
newton(x -> eps*(cos(x) - x), x -> eps*(-sin(x) - 1), 1; tol=1e-15, verbose=true)
```

    [1] x=1  f(x)=-0.04596976941318603  f'(x)=-0.18414709848078967
    [2] x=0.7503638678402439  f(x)=-0.0018923073822117442  f'(x)=-0.16819049529414878
    [3] x=0.7391128909113617  f(x)=-4.645589899077152e-6  f'(x)=-0.16736325442243014
    [4] x=0.739085133385284  f(x)=-2.8472058044570758e-11  f'(x)=-0.16736120293089507
    [5] x=0.7390851332151607  f(x)=0.0  f'(x)=-0.1673612029183215

    (0.7390851332151607, 0.0, 5)

# That's really fast!

* 10 digits of accuracy in 4 iterations.
* How is this convergence test different from the one we used for bisection?
* How can this break down?

$$ x_{k+1} = x_k - f(x_k)/f'(x_k) $$

```julia
newton(x -> cos(x) - x, x -> -sin(x) - 1, .2*pi/2; verbose=true)
```

    [1] x=0.3141592653589793  f(x)=0.6368972509361742  f'(x)=-1.3090169943749475
    [2] x=0.8007054703914749  f(x)=-0.10450500785604744  f'(x)=-1.7178474183038182
    [3] x=0.7398706100060081  f(x)=-0.0013148113494647617  f'(x)=-1.6741923555416478
    [4] x=0.7390852693340639  f(x)=-2.2781024078266654e-7  f'(x)=-1.6736121297866664
    [5] x=0.7390851332151648  f(x)=-6.8833827526759706e-15  f'(x)=-1.673612029183218

    (0.7390851332151648, -6.8833827526759706e-15, 5)

# Convergence of fixed-point iteration

Consider the iteration
$$x_{k+1} = g(x_k)$$
where $g$ is a continuously differentiable function.
Suppose that there exists a fixed point $x_* = g(x_*)$.  By the [mean value theorem](https://en.wikipedia.org/wiki/Mean_value_theorem), we have that
$$ x_{k+1} - x_* = g(x_k) - g(x_*) = g'(c_k) (x_k - x_*) $$
for some $c_i$ between $x_k$ and $x_*$.

Taking absolute values, $$|e_{k+1}| = |g'(c_k)| |e_k|,$$ which converges to zero if $|g'(c_k)| < 1$.

## Exercise

* Write Newton's method for $f(x) = 0$ from initial guess $x_0$ as a fixed point method.
* Suppose the Newton iterates $x_k$ converge to a simple root $x_*$, $x_k \to x_*$. What is $\lvert g'(x_*) \rvert$ for Newton's method?

# Newton for systems of equations

Let $\mathbf u \in \mathbb R^n$ and consider the function $\mathbf f(\mathbf u) \in \mathbb R^n$. Then
$$ \mathbf f(\mathbf u) = \mathbf f(\mathbf u_0) + \underbrace{\mathbf f'(\mathbf u_0)}_{n\times n} \underbrace{\mathbf \delta u}_{n\times 1} + \frac 1 2 \underbrace{\mathbf f''(\mathbf u_0)}_{n\times n \times n} {(\delta \mathbf u)^2}_{n\times n} + O(|\delta\mathbf u|^3).$$

We drop all but the first two terms and name the Jacobian matrix $J(\mathbf u) = \mathbf f'(\mathbf u)$,
$$ \mathbf f(\mathbf u) \approx \mathbf f(\mathbf u_0) + J \delta \mathbf u .$$
Solving the right hand side equal to zero yields
\begin{align}
J \delta \mathbf u &= -\mathbf f(\mathbf u_k) \\
\mathbf u_{k+1} &= \mathbf u_k + \delta \mathbf u
\end{align}

# Newton in code

```julia
function newton(residual, jacobian, u0)
    u = u0
    uhist = [copy(u)]
    normhist = []
    for k in 1:20
        f = residual(u)
        push!(normhist, norm(f))
        J = jacobian(u)
        delta_u = - J \ f
        u += delta_u
        push!(uhist, copy(u))
    end
    uhist, normhist
end
```

    newton (generic function with 1 method)

```julia
function residual(u)
    x, y = u
    [x^2 + y^2 - 1, x^2 - y]
end
function jacobian(u)
    x, y = u
    [2x 2y; 2x -1]
end

uhist, normhist = newton(residual, jacobian, [0.1, 2])
plot(normhist, marker=:auto, yscale=:log10)
```

# Plotting the trajectory

```julia
xy = hcat(uhist...)
plot(xy[1,:], xy[2,:], marker=:auto)
circle = exp.(1im*LinRange(0, 2*pi, 50))
plot!(real(circle), imag(circle))
plot!(x -> x^2, xlims=(-2, 3), ylims=(-2, 2), axes=:equal, legend=:bottomright)
```

# Bratu problem

$$-(u_x)_x - \lambda e^{u}= 0$$

```julia
function bratu_f(u; lambda=.5)
    n = length(u)
    h = 2 / (n - 1)
    weights = -fdstencil([-h, 0, h], 0, 2)
    u = copy(u)
    f = copy(u)
    u[1] = 0
    u[n] = 1
    f[n] -= 1
    for i in 2:n-1
        f[i] = weights * u[i-1:i+1] - lambda * exp(u[i])
    end
    f
end
```

    bratu_f (generic function with 1 method)

```julia
function bratu_J(u; lambda=.5)
    n = length(u)
    h = 2 / (n - 1)
    weights = -fdstencil([-h, 0, h], 0, 2)
    rows = [1, n]
    cols = [1, n]
    vals = [1., 1.] # diagonals entries (float)
    for i in 2:n-1
        append!(rows, [i,i,i])
        append!(cols, i-1:i+1)
        append!(vals, weights + [0 -lambda*exp(u[i]) 0])
    end
    sparse(rows, cols, vals)
end
```

    bratu_J (generic function with 1 method)

# Solving Bratu

```julia
n = 20
x = collect(LinRange(-1., 1, n))
u0 = (1 .+ x) ./ 2
uhist, normhist = newton(bratu_f, bratu_J, u0);

plot(normhist, marker=:auto, yscale=:log10)
```

```julia
plot(x, uhist[end], marker=:auto)
```

```julia

```


# Ritz method for a beam

**November, 2018**

We want to find a Ritz approximation of the deflection $w$ of a beam under applied
transverse uniform load of intensity $f$ per unit lenght and an end moment $M$.
This is described by the following boundary value problem.

$$
\frac{\mathrm{d}^2}{\mathrm{d}x^2}\left(EI \frac{\mathrm{d}^2w}{\mathrm{d}x^2}\right) = f\, ,\quad
0 < x < L,\quad EI>0\, ,
$$

with

$$
w(0) = w'(0) = 0,\quad
\left(EI \frac{\mathrm{d}^2w}{\mathrm{d}x^2}\right)_{x=L} = M,\quad
\left[\frac{\mathrm{d}}{\mathrm{d}x}\left(EI \frac{\mathrm{d}^2w}{\mathrm{d}x^2}\right)\right]_{x=L} = 0\, .
$$

```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
```

```python
%matplotlib notebook
init_printing()

# Graphics setup
gray = '#757575'
plt.rcParams["mathtext.fontset"] = "cm"
plt.rcParams["text.color"] = gray
plt.rcParams["font.size"] = 12
plt.rcParams["xtick.color"] = gray
plt.rcParams["ytick.color"] = gray
plt.rcParams["axes.labelcolor"] = gray
plt.rcParams["axes.edgecolor"] = gray
plt.rcParams["axes.spines.right"] = False
plt.rcParams["axes.spines.top"] = False
plt.rcParams["figure.figsize"] = 4, 3
```

The exact solution for this problem is

$$w(x) = \left(\frac{2M + fL^2}{4EI}\right)x^2 - \frac{fL}{6EI}x^3 + \frac{f}{24EI}x^4\, .$$

```python
x = symbols('x')
M, EI, f, L, Mb = symbols("M EI f L Mb")
w_exact = (2*M + f*L**2)/(4*EI)*x**2 - f*L/(6*EI)*x**3 + f/(24*EI)*x**4
psi_exact = -(2*M + f*L**2)/(2*EI)*x + f*L*x**2/(2*EI) - f*x**3/(6*EI)
M_exact = f/2*(x - L)**2 + Mb
lamda_exact = f*(L - x)
```

```python
def plot_expr(expr, x, rango=(0, 1), ax=None, linestyle="solid"):
    """Plot SymPy expressions of a single variable"""
    expr_num = lambdify(x, expr, "numpy")
    x0 = rango[0]
    x1 = rango[1]
    x_num = np.linspace(0, 1, 101)
    if ax is None:
        plt.figure()
        ax = plt.gca()
    ax.plot(x_num, expr_num(x_num), linestyle=linestyle)
```

## Conventional formulation

We can transform the boundary value problem to

$$\frac{\mathrm{d}^2}{\mathrm{d}x^2}\left(EI\frac{\mathrm{d}^2 u}{\mathrm{d}x^2}\right) = \hat{f}\, ,\quad 0 < x< L$$

with

$$u(0) = u'(0) = EI u''(L) = \left[\frac{\mathrm{d}}{\mathrm{d}x^2}(EI w'')\right]_{x=L} = 0\, ,$$

and

$$u = w - w_0, \hat{f} = f - \frac{\mathrm{d}}{\mathrm{d}x^2}(EI w_0'')$$

where $w_0$ satisfies the boundary conditions. For this case we can chose

$$w_0 = \frac{M x^2}{2EI}\, ,$$

that satisfies the boundary conditions. For this choice, we have $\hat{f} = f$.

The quadratic functional for this problem is

$$J[u] = \int\limits_0^L \left[EI\left(\frac{\mathrm{d}^2 u}{\mathrm{d}x^2}\right)^2 - fu\right]\mathrm{d}x\, ,$$

and the weak problem $B(v, u) = l(v)$, with

$$
B(v, u) = \int\limits_0^L EI\frac{\mathrm{d}^2 v}{\mathrm{d}x^2}\frac{\mathrm{d}^2 u}{\mathrm{d}x^2}\mathrm{d}x\, ,\quad
l(v) = \int\limits_0^L v f\mathrm{d}x\, .
$$

```python
def quad_fun(x, u, M, EI, f, L):
    F = EI/2*diff(u, x, 2)**2 - f*u
    L = integrate(F, (x, 0, L))
    return L

```

```python
def ritz_conventional(x, M, EI, f, L, nterms):
    a = symbols("a0:%i"%(nterms))
    u = sum(a[k]*x**(k + 2) for k in range(nterms))
    M, EI, f, L = symbols("M EI f L")
    L = quad_fun(x, u, M, EI, f, L)
    eqs = [L.diff(C) for C in a]
    sol = solve(eqs, a)
    return u.subs(sol)
```

```python
w0 = M*x**2/(2*EI)
subs = {L: 1, EI:1, M:1, f: 1}
errors_conv = []
for nterms in range(1, 4):
    u = ritz_conventional(x, M, EI, f, L, nterms)
    w = u + w0
    err = integrate((w - w_exact)**2, (x, 0, L))
    norm = integrate(w_exact**2, (x, 0, L))
    errors_conv.append(N(sqrt((err/norm).subs(subs))))
    plt.figure(figsize=(8, 3))
    ax = plt.subplot(121)
    plot_expr(w_exact.subs(subs), x, ax=ax)
    plot_expr(w.subs(subs), x, ax=ax, linestyle="dashed")
    ax = plt.subplot(122)
    plot_expr(psi_exact.subs(subs), x, ax=ax)
    plot_expr(-w.diff(x).subs(subs), x, ax=ax, linestyle="dashed")
    plt.legend(["Exact", "Ritz"]);
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

## Lagrange multiplier formulation

We can write the problem as minimizing the functional

$$J(\psi, w) = \int\limits_0^L\left[\frac{EI}{2}\left(\frac{\mathrm{d} \psi}{\mathrm{d}x}\right)^2 -
f w\right]\mathrm{d}x + M\psi(L)\, ,$$

subject to

$$G(\psi, w) \equiv \psi + \frac{\mathrm{d}w}{\mathrm{d}x} = 0\, .$$

The Lagrangian is given by

$$L(\psi, w, \lambda) = \int\limits_0^L\left[\frac{EI}{2}\left(\frac{\mathrm{d} \psi}{\mathrm{d}x}\right)^2 -
f w\right]\mathrm{d}x + \int\limits_0^L \lambda\left(\psi + \frac{\mathrm{d}x}{\mathrm{d}x}\right)\mathrm{d}x + M\psi(L)\, , $$

where $\lambda$ is the Lagrange multiplier, which in this case represents the shear force.

```python
errors_conv
```

```python
def lagran(x, psi, w, lamda, M, EI, f, L):
    F = EI/2*diff(psi, x)**2 - f*w
    G = lamda*(psi + diff(w, x))
    L = integrate(F, (x, 0, L)) + integrate(G, (x, 0, L)) + M*psi.subs(x, L)
    return L

```

```python
def ritz_multiplier(x, M, EI, f, L, nterms):
    a = symbols("a0:%i"%(nterms))
    b = symbols("b0:%i"%(nterms))
    c = symbols("c0:%i"%(nterms))
    var = a + b + c
    psi = sum(a[k]*x**(k + 1) for k in range(nterms))
    w = sum(b[k]*x**(k + 1) for k in range(nterms))
    lamda = sum(c[k]*x**k for k in range(nterms))
    M, EI, f, L = symbols("M EI f L")
    L = lagran(x, psi, w, lamda, M, EI, f, L)
    eqs = [L.diff(C) for C in var]
    sol = solve(eqs, var)
    return w.subs(sol), psi.subs(sol), lamda.subs(sol)
```

```python
subs = {L: 1, EI:1, M:1, f: 1}
errors_mult = []
for nterms in range(1, 4):
    w, psi, lamda = ritz_multiplier(x, M, EI, f, L, nterms)
    err = (integrate((w - w_exact)**2, (x, 0, L)) +
           integrate((psi - psi_exact)**2, (x, 0, L)) +
           integrate((lamda - lamda_exact)**2, (x, 0, L)))
    norm = (integrate(w_exact**2, (x, 0, L)) +
           integrate(psi_exact**2, (x, 0, L)) +
           integrate(lamda_exact**2, (x, 0, L)))
    errors_mult.append(N(sqrt((err/norm).subs(subs))))
    plt.figure(figsize=(8, 3))
    ax = plt.subplot(121)
    plot_expr(w_exact.subs(subs), x, ax=ax)
    plot_expr(w.subs(subs), x, ax=ax, linestyle="dashed")
    ax = plt.subplot(122)
    plot_expr(psi_exact.subs(subs), x, ax=ax)
    plot_expr(psi.subs(subs), x, ax=ax, linestyle="dashed")
    plt.legend(["Exact", "Ritz with multipliers"]);

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```python
errors_mult
```

# The penalty function formulation

The augmented functional for this formulation is given by

$$P_K (\psi, w) = J(\psi, w) + \frac{K}{2}\int\limits_0^L \left(\psi + \frac{\mathrm{d}w}{\mathrm{d}x}\right)^2\mathrm{d}x\, ,$$

where $K$ is the penalty parameter.

```python
def augmented(x, psi, w, K, M, EI, f, L):
    F = EI/2*diff(psi, x)**2 - f*w
    G = (psi + diff(w, x))
    P = integrate(F, (x, 0, L)) + K/2*integrate(G**2, (x, 0, L)) + M*psi.subs(x, L)
    return P
```

```python
def ritz_penalty(x, K, M, EI, f, L, nterms):
    a = symbols("a0:%i"%(nterms))
    b = symbols("b0:%i"%(nterms))
    var = a + b
    w = sum(a[k]*x**(k + 1) for k in range(nterms))
    psi = sum(b[k]*x**(k + 1) for k in range(nterms))
    M, EI, f, L = symbols("M EI f L")
    P = augmented(x, psi, w, K, M, EI, f, L)
    eqs = [P.diff(C) for C in var]
    sol = solve(eqs, var)
    return w.subs(sol), psi.subs(sol)
```

```python
K = symbols("K")
errors_penalty = []
for K_val in [1, 10, 100]:
    subs = {L: 1, EI:1, M:1, f: 1, K: K_val}
    w, psi = ritz_penalty(x, K, M, EI, f, L, 2)
    err = (integrate((w - w_exact)**2, (x, 0, L)) +
           integrate((psi - psi_exact)**2, (x, 0, L)) +
           integrate((lamda - lamda_exact)**2, (x, 0, L)))
    norm = (integrate(w_exact**2, (x, 0, L)) +
           integrate(psi_exact**2, (x, 0, L)) +
           integrate(lamda_exact**2, (x, 0, L)))
    errors_penalty.append(N(sqrt((err/norm).subs(subs))))
    plt.figure(figsize=(8, 3))
    ax = plt.subplot(121)
    plot_expr(w_exact.subs(subs), x, ax=ax)
    plot_expr(w.subs(subs), x, ax=ax, linestyle="dashed")
    ax = plt.subplot(122)
    plot_expr(psi_exact.subs(subs), x, ax=ax)
    plot_expr(psi.subs(subs), x, ax=ax, linestyle="dashed")
    plt.legend(["Exact", "Ritz with penalty"]);
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```python
errors_penalty
```

## Mixed formulation

The mixed formulation involves rewriting a given higher order equation as a pair of llower
order equations by introducing secondary dependent variables. The original equation can be
decomposed into

$$
\frac{M(x)}{EI} = \frac{\mathrm{d}^2 w}{\mathrm{d}x^2}\, ,\quad
\frac{\mathrm{d}^2M(x)}{\mathrm{d}x^2} = f\, ,\quad 0<x<L\, .
$$

The functional in this case is

$$
I(w, M) = \int\limits_0^L\left(\frac{\mathrm{d}w}{\mathrm{d}x}\frac{\mathrm{d}M}{\mathrm{d}x}
+ \frac{M^2}{2EI}+ fw\right)\mathrm{d}x
$$

```python
def mixed_fun(x, w, M, EI, f, L):
    F = diff(w, x)*diff(M, x) + M**2/(2*EI) + f*w
    L = integrate(F, (x, 0, L))
    return L

```

```python
def ritz_mixed(x, Mb, EI, f, L, nterms):
    a = symbols("a0:%i"%(nterms))
    b = symbols("b0:%i"%(nterms))
    var = a + b
    w = sum(a[k]*x**(k + 1) for k in range(nterms))
    M = Mb + sum(b[k]*(x - L)**(k + 1) for k in range(nterms))
    EI, f, L = symbols("EI f L")
    L = mixed_fun(x, w, M, EI, f, L)
    eqs = [L.diff(C) for C in var]
    sol = solve(eqs, var)
    return w.subs(sol), M.subs(sol)
```

```python
subs = {L: 1, EI:1, f: 1, M:1, Mb:1}
Mb = 1
errors_mix = []
for nterms in range(1, 5):
    w, Ms = ritz_mixed(x, Mb, EI, f, L, nterms)
    err = integrate((w - w_exact)**2, (x, 0, L))
    norm = integrate(w_exact**2, (x, 0, L))
    errors_mix.append(N(sqrt((err/norm).subs(subs))))
    plt.figure(figsize=(8, 3))
    ax = plt.subplot(121)
    plot_expr(w_exact.subs(subs), x, ax=ax)
    plot_expr(w.subs(subs), x, ax=ax, linestyle="dashed")
    ax = plt.subplot(122)
    plot_expr(M_exact.subs(subs), x, ax=ax)
    plot_expr(Ms.subs(subs), x, ax=ax, linestyle="dashed")
    plt.legend(["Exact", "Ritz mixed"]);
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

```python

```


```python
from collections import defaultdict
import numpy as np
import scipy
import scipy.sparse as sps
import math
import matplotlib.pyplot as plt
import time
from sklearn.datasets import load_svmlight_file
import random
%matplotlib inline
```

# Support Vector Machines
## Classification Using SVM
Load dataset. We will use w1a dataset from LibSVM datasets https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/

The original optimization problem for the Support Vector Machine (SVM) is given by
\begin{equation}\label{eq:primal}
  \min_{w \in R^d} \  \sum_{i=1}^n \ell(y_i A_i^\top w) + \frac\lambda2 \|w\|^2
\end{equation}
where $\ell : R\rightarrow R$, $\ell(z) := \max\{0,1-z\}$ is the hinge loss function.
Here for any $i$, $1\le i\le n$, the vector $A_i\in R^d$ is the $i$-th data example, and $y_i\in\{\pm1\}$ is the corresponding label.

The dual optimization problem for the SVM is given by
\begin{equation}\label{eq:dual}
 \max_{\boldsymbol{\alpha} \in R^n } \  \alpha^\top\boldsymbol{1} - \tfrac1{2\lambda} \alpha^\top Y A A^\top Y\alpha
 \text{    such that    $0\le \alpha_i \le 1  \ \forall i$}
\end{equation}
where $Y := \mathop{diag}(y)$, and $A\in R^{n \times d}$ again collects all $n$ data examples as its columns.

Note that $w$ can be derived from $\alpha$ as
\begin{equation}
    w(\alpha) = \frac{1}{\lambda} A^\top Y \alpha.
\end{equation}

```python
DATA_TRAIN_PATH = 'data/w1a'

A, y = load_svmlight_file(DATA_TRAIN_PATH)
A = A.toarray()
print(y.shape, A.shape)
```

    (2477,) (2477, 300)

## Prepare cost and prediction functions

```python
def calculate_primal_objective(y, A, w, lambda_):
    """
    Compute the full cost (the primal objective), that is loss plus regularizer.
    y: +1 or -1 labels, shape = (num_examples)
    A: Dataset matrix, shape = (num_examples, num_features)
    w: Model weights, shape = (num_features)
    return: scalar value
    """
    # ***************************************************
    s = 0
    for i in range(len(y)):
        loss = max(0, 1 - y[i] * (A[i].T @ w))
        s += loss
    w2 = np.sum(w ** 2)

    obj = s + lambda_ / 2 * w2
    # ***************************************************
    return obj
```

```python
def calculate_accuracy(y, A, w):
    """
    Compute the training accuracy on the training set (can be called for test set as well).
    y: +1 or -1 labels, shape = (num_examples)
    A: Dataset matrix, shape = (num_examples, num_features)
    w: Model weights, shape = (num_features)
    return: scalar value
    """
    # ***************************************************
    # compute predictions
    pred = A @ w
    # correct predictions to be -1 or 1
    for i in range(len(pred)):
        if pred[i] > 0:
            pred[i] = 1
        else:
            pred[i] = -1

    # compute accuracy
    acc = np.mean(pred == y)
    # ***************************************************
    return acc
```

## Coordinate Descent (Ascent) for SVM

Compute the closed-form update for the i-th variable alpha, in the dual optimization problem, given alpha and the current corresponding w.

Hints:
- Differentiate the dual objective with respect to one `alpha[i]`.
- Set the derivative to zero to compute a new `alpha[i]`.
- Make sure the values of alpha stay inside a `[0, 1]` box.
- You can formulate the update as `alpha[i] = projection(alpha[i] + lambda_ * (some update))`.
- You can test the correctness of your implementation by checking if the difference between the dual objective and primal objective goes to zero. This difference, the duality gap, should get smaller than 10 in 700000 iterations.

```python
def calculate_coordinate_update(y, A, lambda_, alpha, w, i):
    """
    Compute a coordinate update (closed form) for coordinate i.
    y: +1 or -1 labels, shape = (num_examples)
    A: Dataset matrix, shape = (num_examples, num_features)
    lambda_: Regularization parameter, scalar
    alpha: Dual variables, shape = (num_examples)
    w: Model weights, shape = (num_examples)
    i: Index of the entry of the dual variable 'alpha' that is to be updated
    return: New weights w (shape (num_features)), New dual variables alpha (shape (num_examples))
    """
    # calculate the update of coordinate at index=n.
    a_i, y_i = A[i], y[i]
    old_alpha_i = np.copy(alpha[i])

    # ***************************************************
    C = old_alpha_i + lambda_ * ((1 - y_i * a_i.T @ w)/(a_i.T @ a_i))
    if C < 0: C = 0
    if C > 1: C = 1
    alpha[i] = C

    # update w
    w += (alpha[i] - old_alpha_i) * y_i * a_i * (1/lambda_)
    # ***************************************************

    return w, alpha
```

```python
def calculate_dual_objective(y, A, w, alpha, lambda_):
    """
    Calculate the objective for the dual problem.
    Follow the formula given above.
    y: +1 or -1 labels, shape = (num_examples)
    A: Dataset matrix, shape = (num_examples, num_features)
    alpha: Dual variables, shape = (num_examples)
    lambda_: Regularization parameter, scalar
    return: Scalar value
    """
    # ***************************************************
    w2 = np.sum(w ** 2)
    obj = np.sum(alpha) - lambda_/2 * w2
    # ***************************************************
    return obj
```

```python
def coordinate_descent_for_svm_demo(y, A, trace=False):
    max_iter = 1000000
    lambda_ = 0.01
    history = defaultdict(list) if trace else None

    num_examples, num_features = A.shape
    w = np.zeros(num_features)
    alpha = np.zeros(num_examples)

    for it in range(max_iter):
        # i = sample one data point uniformly at random from the columns of A
        i = random.randint(0,num_examples-1)

        w, alpha = calculate_coordinate_update(y, A, lambda_, alpha, w, i)

        if it % 100000 == 0:
            # primal objective
            primal_value = calculate_primal_objective(y, A, w, lambda_)
            # dual objective
            dual_value = calculate_dual_objective(y, A, w, alpha, lambda_)
            # primal dual gap
            duality_gap = primal_value - dual_value

            print('iteration=%i, primal:%.5f, dual:%.5f, gap:%.5f'%(
                    it, primal_value, dual_value, duality_gap))
        if it % 1000 == 0:
            primal_value = calculate_primal_objective(y, A, w, lambda_)
            if trace:
                history["objective_function"] += [primal_value]
                history['iter'].append(it)

    print("training accuracy = {l}".format(l=calculate_accuracy(y, A, w)))
    return history

history_cd = coordinate_descent_for_svm_demo(y, A, trace=True)
```

    iteration=0, primal:2198.07161, dual:0.00018, gap:2198.07143

    c:\users\melanija\appdata\local\programs\python\python37\lib\site-packages\ipykernel_launcher.py:17: RuntimeWarning: divide by zero encountered in double_scalars

    iteration=100000, primal:279.61403, dual:216.25006, gap:63.36397
    iteration=200000, primal:263.39900, dual:222.35389, gap:41.04512
    iteration=300000, primal:244.70078, dual:223.61763, gap:21.08315
    iteration=400000, primal:237.81812, dual:224.60328, gap:13.21483
    iteration=500000, primal:245.41629, dual:225.51759, gap:19.89869
    iteration=600000, primal:241.74883, dual:225.98354, gap:15.76530
    iteration=700000, primal:235.64178, dual:226.36782, gap:9.27396
    iteration=800000, primal:235.74450, dual:226.68425, gap:9.06025
    iteration=900000, primal:237.35067, dual:226.99495, gap:10.35572
    training accuracy = 0.994751715785224

```python
# plot training curve
plt.plot(history_cd["iter"], history_cd["objective_function"], label="Coordinate Descent")
plt.yscale('log')
plt.legend()
```

# Stochastic gradient descent for SVM

Let's now compare it with SGD on original problem for the SVM. In this part, you will implement stochastic gradient descent on the primal SVM objective. The stochasticity comes from sampling data points.

```python
def compute_stoch_gradient_svm(A_sample, b_sample, lambda_, w_t, num_data_points):
    """
    Calculate stochastic gradient over A_batch, b_batch.
    A_sample: A data sample, shape=(num_features)
    b_sample: Corresponding +1 or -1 label, scalar
    w_t: Model weights, shape=(num_features)
    num_data_points: Total size of the dataset, scalar integer
    """
    # ***************************************************
    z = A_sample.dot(w_t) * b_sample
    if z < 1:
        gradient = (lambda_ * w_t) - (num_data_points * b_sample * A_sample)
    else:
        gradient = (lambda_ * w_t)
    # ***************************************************
    return gradient.reshape(-1)
```

```python
def stochastic_gradient_descent_svm_demo(A, b, gamma, batch_size=1, trace=False):
    history = defaultdict(list) if trace else None
    num_data_points, num_features = np.shape(A)
    max_iter = 1000000
    lambda_ = 0.01

    w_t = np.zeros(num_features)

    current_iter = 0
    while (current_iter < max_iter):
        i = random.randint(0,num_data_points - 1)
        b_batch, A_batch = b[i], A[i]
        gradient = compute_stoch_gradient_svm(A_batch, b_batch, lambda_, w_t, num_data_points)
        w_t = w_t - gamma * gradient
        if current_iter % 100000 == 0:
            primal_value = calculate_primal_objective(y, A, w_t, lambda_)
            print('iteration=%i, primal:%.5f'%(
                    current_iter, primal_value))
        if current_iter % 1000 == 0:
            primal_value = calculate_primal_objective(y, A, w_t, lambda_)
            if trace:
                history['objective_function'].append(primal_value)
                history['iter'].append(current_iter)
        current_iter += 1
    print("training accuracy = {l}".format(l=calculate_accuracy(y, A, w_t)))
    return history

```

Try different stepsizes and find the best one

```python
lrs = [0.1, 0.01, 0.05, 0.001, 0.0005]
results = []
for lr in lrs:
    print('Running SGD with lr=', lr)
    sgd = stochastic_gradient_descent_svm_demo(A, y, lr, trace=True)
    results.append(sgd)
    print('---------------------------------------------')
```

    Running SGD with lr= 0.1
    iteration=0, primal:2477.00000
    iteration=100000, primal:110097.59283
    iteration=200000, primal:66169.28576
    iteration=300000, primal:60418.10744
    iteration=400000, primal:66759.96304
    iteration=500000, primal:51609.27102
    iteration=600000, primal:56694.14760
    iteration=700000, primal:54628.00356
    iteration=800000, primal:59701.83805
    iteration=900000, primal:62832.66969
    training accuracy = 0.9604360113039968
    ---------------------------------------------
    Running SGD with lr= 0.01
    iteration=0, primal:2704.13212
    iteration=100000, primal:2997.47830
    iteration=200000, primal:2902.55030
    iteration=300000, primal:2608.37200
    iteration=400000, primal:2990.86377
    iteration=500000, primal:3821.56364
    iteration=600000, primal:3999.55021
    iteration=700000, primal:2648.27864
    iteration=800000, primal:2887.71066
    iteration=900000, primal:2295.59859
    training accuracy = 0.9802180056519983
    ---------------------------------------------
    Running SGD with lr= 0.05
    iteration=0, primal:7996.44112
    iteration=100000, primal:19427.61448
    iteration=200000, primal:14790.81258
    iteration=300000, primal:23330.29655
    iteration=400000, primal:38011.09413
    iteration=500000, primal:34758.62805
    iteration=600000, primal:26086.74112
    iteration=700000, primal:17469.29459
    iteration=800000, primal:22557.04987
    iteration=900000, primal:25115.32774
    training accuracy = 0.9729511505853855
    ---------------------------------------------
    Running SGD with lr= 0.001
    iteration=0, primal:2477.00000
    iteration=100000, primal:390.66261
    iteration=200000, primal:368.84870
    iteration=300000, primal:361.60669
    iteration=400000, primal:364.48207
    iteration=500000, primal:409.86861
    iteration=600000, primal:393.81841
    iteration=700000, primal:391.60344
    iteration=800000, primal:384.84073
    iteration=900000, primal:351.65206
    training accuracy = 0.9899071457408155
    ---------------------------------------------
    Running SGD with lr= 0.0005
    iteration=0, primal:1117.83303
    iteration=100000, primal:284.79567
    iteration=200000, primal:303.49552
    iteration=300000, primal:288.66791
    iteration=400000, primal:275.83349
    iteration=500000, primal:297.25770
    iteration=600000, primal:291.40584
    iteration=700000, primal:280.49232
    iteration=800000, primal:286.82717
    iteration=900000, primal:277.28330
    training accuracy = 0.9919257165926524
    ---------------------------------------------

Plot learning curves

```python
# plotting
for i in range(len(results)):
    sgd = results[i]
    plt.plot(sgd["iter"], sgd["objective_function"], label="SGD lr = {:1.5f}".format(lrs[i]))
    plt.yscale('log')
plt.legend()
```

It seems that the first 3 learning rates are too big.
Repeat experiment with even smaller learning rates (lr <= 0.001):

```python
lrs = [0.001, 0.0005, 0.0001, 0.00005, 0.00001]
results = []
for lr in lrs:
    print('Running SGD with lr=', lr)
    sgd = stochastic_gradient_descent_svm_demo(A, y, lr, trace=True)
    results.append(sgd)
    print('---------------------------------------------')
```

    Running SGD with lr= 0.001
    iteration=0, primal:988.62016
    iteration=100000, primal:349.17183
    iteration=200000, primal:364.29257
    iteration=300000, primal:369.87797
    iteration=400000, primal:366.53268
    iteration=500000, primal:377.20163
    iteration=600000, primal:398.52138
    iteration=700000, primal:357.86760
    iteration=800000, primal:335.80020
    iteration=900000, primal:411.15856
    training accuracy = 0.9862737182075091
    ---------------------------------------------
    Running SGD with lr= 0.0005
    iteration=0, primal:2234.06902
    iteration=100000, primal:283.66106
    iteration=200000, primal:385.30826
    iteration=300000, primal:284.98281
    iteration=400000, primal:295.02302
    iteration=500000, primal:290.33378
    iteration=600000, primal:287.14758
    iteration=700000, primal:287.17260
    iteration=800000, primal:277.65706
    iteration=900000, primal:304.45808
    training accuracy = 0.9903108599111828
    ---------------------------------------------
    Running SGD with lr= 0.0001
    iteration=0, primal:2477.00000
    iteration=100000, primal:252.70261
    iteration=200000, primal:243.05037
    iteration=300000, primal:240.04018
    iteration=400000, primal:238.99765
    iteration=500000, primal:238.66440
    iteration=600000, primal:244.46554
    iteration=700000, primal:238.59503
    iteration=800000, primal:242.71816
    iteration=900000, primal:235.88044
    training accuracy = 0.9951554299555915
    ---------------------------------------------
    Running SGD with lr= 5e-05
    iteration=0, primal:2330.37550
    iteration=100000, primal:254.48615
    iteration=200000, primal:243.52702
    iteration=300000, primal:237.63993
    iteration=400000, primal:237.11102
    iteration=500000, primal:235.35983
    iteration=600000, primal:234.34964
    iteration=700000, primal:234.18111
    iteration=800000, primal:234.55288
    iteration=900000, primal:233.14598
    training accuracy = 0.994751715785224
    ---------------------------------------------
    Running SGD with lr= 1e-05
    iteration=0, primal:2377.12739
    iteration=100000, primal:276.42534
    iteration=200000, primal:259.53120
    iteration=300000, primal:252.75191
    iteration=400000, primal:249.99986
    iteration=500000, primal:247.15174
    iteration=600000, primal:243.67913
    iteration=700000, primal:241.48185
    iteration=800000, primal:239.45611
    iteration=900000, primal:237.73139
    training accuracy = 0.9939442874444893
    ---------------------------------------------

```python
# plotting
for i in range(len(results)):
    sgd = results[i]
    plt.plot(sgd["iter"], sgd["objective_function"], label="SGD lr = {:f}".format(lrs[i]))
    plt.yscale('log')
plt.legend()
```

The best learning rates are 0.0001 with a training accuracy of 0.9951.

## Compare SGD with Coordinate Descent

Compare two algorithms in terms of convergence, time complexities per iteration. Which one is easier to use?

```python
# plot CD and best SGD for comparison

#index of best lr
i = 2
plt.plot(results[i]["iter"], results[i]["objective_function"], label="SGD")
plt.plot(history_cd["iter"], history_cd["objective_function"], label="CD")
plt.yscale('log')
plt.legend()
```

We can observe that the convergence of CD is a bit not stable at the beginning. However, it manages to converge at nearly the same iteration as the SGD, even though for SGD we have tunned the learning rate. Both methods have the same time complexity per iteration.


# Convexity Problems in Machine Learning

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy.random import seed, randn
from sklearn.linear_model import LinearRegression

plt.rcParams["font.size"] = 15
plt.rcParams["figure.figsize"] = (10,7.5)
```

```python
def sigmoid(x): return 1 / (1 + np.exp(-x))
def lreg(x, theta): return theta.T @ x
def nnet(x, W1, W2): return W2 @ sigmoid(W1 @ x)

def loss_surface(model_parameters, target_weights, target_ix, model, y):
    """
    Function to return the cost surface between two chosen weights
    in a model and its L2 loss surface.

    Parameters
    ----------
    model_parameters: tuple containing the parameters to compute the model
    target_weights: tuple of lenght 2 containing the parameters compute the loss
    target_ix: typle of lenght 2 containing the indices for each parameter to
               compute the loss
    model: function with parameters "model_parameters" used to compute the output of
           the model
    y: vector with m model outputs

    Returns
    -------
    tuppple with grid containing the surface along the chosen parameters
    """
    # target weights
    tweight1, tweight2 = target_weights
    # target indices
    tix1, tix2 = target_ix

    # Grid surface for target weights to plot
    tw1, tw2 = np.mgrid[-5:5:0.1, -5:5:0.1]
    # Initialize cost surface
    J = np.zeros_like(tw1.ravel())

    for ix, (tw1i, tw2i) in enumerate(zip(tw1.ravel(), tw2.ravel())):
        tweight1[tix1] = tw1i
        tweight2[tix2] = tw2i
        yhat = model(*model_parameters)
        # Updating cost function
        try:
            J[ix] =  np.sum((yhat - y) ** 2) / 2
        except Exception:
            print(ix)

    J = J.reshape(*tw1.shape)
    return tw1, tw2, J

def plot_2dconvex_test(func, alpha, xline, xlims, ax=None):
    """
    Parameters
    ----------
    func: function
        The f:R->R function to test convexity
    alpha: alpha
        value between 0 and 1 inclusive to compare heights
    xline: tuple with len = 2
        Lower and upper limits to draw the line that goes from
        f(x1) to f(x2)
    xlims: tuple with len = 2
        Lower and upper limits of the neighborhood to show the function
    """
    x = np.linspace(*xlims, 100)
    x1, x2 = xline
    val_point = alpha * x1 + (1 - alpha) * x2
    # Lower bound for convex function
    ylow = func(alpha * x1 + (1 - alpha) * x2)
    # Upper bound for convex function
    yupp = alpha * func(x1) + (1 - alpha) * func(x2)

    if ax is None:
        ax = plt

    ax.plot([x1, x2], [func(x1), func(x2)], color="tab:red")
    ax.scatter(val_point, ylow, color="tab:red")
    ax.scatter(val_point, yupp, color="tab:red")
    ax.plot(x, func(x))

def grad_desc_r1(f, fprime, x0, alpha=0.1, iterations=10):
    """
    Compute the gradient descent path for a function f:R->R
    Parameters
    ----------
    f: function
        The f:R->R function to apply gradient descent on
    fprime: function
        The first derivative of f
    x0: int
        Initial value to iterate

    Returns
    -------
    tuple with changes in x and f(x)
    """
    x0 = 1
    gradx, grady = [x0], [f(x0)]
    for i in range(iterations):
        x = gradx[-1]
        x -= alpha * fprime(x)
        gradx.append(x)
        grady.append(f(x))

    return gradx, grady
```

## Definition of a convex function

A function $f$ is said to be convex if $\forall x_1, x_2 \in D_f$ and $\alpha \in [0, 1]$
$$
    f(\alpha x_1 + (1 - \alpha) x_2) \leq \alpha f(x_1) + (1 - \alpha)f(x_2)
$$

```python
from math import pi
fig = plt.figure(figsize=(10,4))
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)
plot_2dconvex_test(lambda x: x**2, 0.5, (-1.5, 1), (-2, 2), ax1)
plot_2dconvex_test(lambda x: np.sin(x**2), 0.2, (-pi/2, 10 * pi / 16), (-2, 2), ax2)
ax1.set_title("Convex Function")
ax2.set_title("Non-Convex Function");
plt.savefig("conv_nonconvR2.pdf")
```

```python
alpha = 0.2
x1, x2 = -pi/2, 10 * pi / 16

print("Convex evaluation of f: {:0.4f}".format(np.sin((alpha * x1 + (1 - alpha) * x2)**2)))
print("Evaluation of the linear between f(x1) and f(x2): {:0.4f}".format(alpha * np.sin(x1 ** 2) + (1 - alpha) * np.sin(x2 ** 2)))
```

    Convex evaluation of f: 1.0000
    Evaluation of the linear between f(x1) and f(x2): -0.3989

For this experiment, we want to model model the relationship $g(X_i) \approx y_i$, where $X_i$ is a random vector whose entries are i.i.d. $N(0, 1)$, and $y_i \sim N(0,1)$ is the output variable.

We consider two models: a linear regression and a single-hiden-layer neural network with sigmoid activation function and linear output. The goal is to analize the loss surface for these two models.

## Loss Surface Under a Linear Regression Model
Considering $n$ training examples, we assume the linear relationship

$$
    \hat{y} = \theta^TX \approx y
$$

With $X\in\mathbb{R}^{m\times n}$, $\theta\in\mathbb{R}^{m}$

Where we want to learn $\theta$ such that
$$
\begin{align}
    J(\theta) &= \frac{1}{2}\sum_{i}^n\left(\hat{y}_i - y_i\right) ^ 2 \\
            &= \frac{1}{2}\sum_{i}^n\left(\theta^TX - y_i\right) ^ 2
\end{align}
$$

is as small as possible.

The surface for $J$ for linear model considering $\theta$ and $b$ is *convex*.

```python
seed(31415)
nobs = 1000
x = randn(10, nobs)
y = randn(1, nobs)

# Initializing the weights
w1 = randn(10, 1)

ax1, ax2 = 2, 3
p1, p2, J = loss_surface((x, w1), (w1, w1), ((ax1, 0), (ax2, 0)), lreg, y)
plt.contourf(p1, p2, J, 40)
plt.colorbar()
plt.xlabel(r"$\theta_{}$".format(ax1), fontsize=20)
plt.ylabel(r"$\theta_{}$".format(ax2), fontsize=20);
#plt.savefig("lreg_loss.pdf")
```

## Loss surface under a single hiden-layer neural network

With $n$ training examples, assume the the relationship
$$
    \hat{y} = W^{[2]}\sigma (W^{[1]} X) \approx y
$$

With
$X\in\mathbb{R}^{m\times n}$, $W^{[1]} \in \mathbb{R}^{k, m}$, $W^{[2]} \in \mathbb{R}^{1, k}$ and $\sigma(x) = 1/{(1 + e^{-x})}$ is the sigmoid activation function

```python
seed(31415)

# Creating the samples
nobs = 1000
x = randn(10, nobs)
y = randn(1, nobs)

# Initializing the weights
w1 = randn(4, 10)
w2 = randn(1, 4)

cuts = [((2,0), (3,0)),
        ((2,0), (3,1)),
        ((1,1), (2,1)),
        ((1,0), (2,0))]

fig = plt.figure()
for ix, cut in enumerate(cuts):
    ax = fig.add_subplot(2, 2, ix + 1)
    p1, p2, J = loss_surface((x, w1, w2), (w1, w1), cut, nnet, y)
    plt.contourf(p1, p2, J, 40)
    plt.colorbar()
    (cut1_ix, cut1_layer), (cut2_ix, cut2_layer) = cut
    xlab = "$W^{[" + f"{cut1_layer + 1}" + "]}" + f"_{cut1_ix + 1}$"
    ylab = "$W^{[" + f"{cut2_layer + 1}" + "]}" + f"_{cut2_ix + 1}$"
    plt.xlabel(xlab, fontsize=20)
    plt.ylabel(ylab, fontsize=20)
plt.tight_layout()

plt.savefig("nnet_loss.pdf")
```

## The problem of simple Gradient Descent

```python
def f(x): return x**2
def fprime(x): return 2 * x
dom = np.linspace(-2, 2, 100)

fig = plt.figure(figsize=(10,4))
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)

alpha1, alpha2 = 1.03, 0.03
grad1x, grad1y = grad_desc_r1(f, fprime, 1.5, alpha=alpha1)
grad2x, grad2y = grad_desc_r1(f, fprime, 1.5, alpha=alpha2)

ax1.set_title(r"$Overshoot, \alpha={}$".format(alpha1))
ax1.plot(dom, f(dom))
ax1.plot(grad1x, grad1y, color="tab:orange", linestyle="--")
ax1.scatter(grad1x[0], grad1y[0], color="tab:red", marker="x", s=110)
ax1.scatter(grad1x[1:], grad1y[1:], color="tab:red");

ax2.set_title(r"$Undershoot, \alpha={}$".format(alpha2))
ax2.plot(dom, f(dom))
ax2.plot(grad2x, grad2y, color="tab:orange", linestyle="--")
ax2.scatter(grad2x[0], grad2y[0], color="tab:red", marker="x", s=110)
ax2.scatter(grad2x[1:], grad2y[1:], color="tab:red");
plt.savefig("graddesc.pdf")
```

Convexity Problems in machine learning.ipynb


## Problem Sheet 3 Question 7b

The general form of the population growth differential equation
\begin{equation} y^{'}-y+x=0, \ \ (0 \leq x \leq 1) \end{equation}
with the initial condition
\begin{equation}y(0)=0\end{equation}
For h=0.2.
# Midpoint method Solution
\begin{equation}
\frac{w_{i+1}-w_i}{h}=f(x_i+\frac{h}{2},w_i+\frac{h}{2}f(x_i,w_i))
\end{equation}
Rearranging
\begin{equation}
w_{i+1}=w_i+hf(x_i+\frac{h}{2},w_i+\frac{h}{2}f(x_i,w_i))
\end{equation}
\begin{equation}
w_{i+1}=w_i+h(k_2)
\end{equation}
\begin{equation}
k_1=w_i-x_i+2
\end{equation}
\begin{equation}
k_2=w_i+\frac{h}{2}k_1-(x_i+\frac{h}{2})+2)
\end{equation}

```python
import numpy as np
import math
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
warnings.filterwarnings("ignore")
```

```python
def myfun_xy(x,y):
    return y-x+2

#PLOTS
def Midpoint_Question2(N,IC):

    x_start=0
    INTITIAL_CONDITION=IC
    h=0.2
    N=N+1
    x=np.zeros(N)
    w=np.zeros(N)
    k_mat=np.zeros((2,N))
    k=0
    w[0]=INTITIAL_CONDITION
    x[0]=x_start

    for k in range (0,N-1):
        k_mat[0,k]=myfun_xy(x[k],w[k])
        k_mat[1,k]=myfun_xy(x[k]+h/2,w[k]+h/2*k_mat[0,k])
        w[k+1]=w[k]+h*(k_mat[1,k])
        x[k+1]=x[k]+h

    fig = plt.figure(figsize=(10,4))
    plt.plot(x,w,'-.o',color='blue')
    plt.title('Numerical Solution h=%s'%(h))

    # --- title, explanatory text and save
    fig.suptitle(r"$y'=y-x+2$", fontsize=20)
    plt.tight_layout()
    plt.subplots_adjust(top=0.85)
    print('x')
    print(x)
    print('k1')
    print(k_mat[0,:])
    print('k2')
    print(k_mat[1,:])
    print('w')
    print(w)
```

```python
# Midpoint_Question2(N,IC)
Midpoint_Question2(5,1)
```

```python

```

```python

```

Chapter 03 - Runge Kutta/3_Problem_Sheet/307b_Problem_Sheet.ipynb