# Linear Regression

At a fundamental level Neural Network architectures are based on the principles of linear and logistic regression. We therefore begin with a review of these topics with some worked examples.

These notes provide a relatively detailed review of Linear Regression. The notes are based on a number of courses and in particular Andrew Ng's Machine Learning course on Youtube. The first video in that series can be found here:

https://www.youtube.com/watch?v=ylWBbxQHwFM

This material will often be covered in a Machine Learning course and as a result there is overlap here with modules you may have taken or may be taking currently. This is a necessary evil; this stuff is important, so it is worth understanding it from a couple of different directions!

## Introduction

### Definition: Target
We define the Target as the output feature in our model. The Target feature is referred to as the *Dependent Variable* in statistics derived literature. When modeling we typically denote the target feature with the symbol $Y$. When programming we may sometimes instantiate the variable with a more descriptive name.

### Example
In a typical linear regression example we might be interested in modeling the relationship between height and age. In this case our Target feature or dependent variable is *height*.

Assuming measurements in meters we can instantiate a vector of height measurements in numpy using the 'array' function where the resultant array is equivalent to a vector. Remember that the numpy array is actually initialized from a native python array.

```python
import numpy as np
Y = np.array([60,90,100.5,130,128,140,160,140,165,162,155])
print("Our height vector is ", Y)
```

    Our height vector is  [ 60.   90.  100.5 130.  128.  140.  160.  140.  165.  162.  155. ]

### Definition: Predictor Features
The Predictor Features are the input features to our model and are referred to as *Independent Variables* in the statistics literature. A nice way of remembering this is that we ideally want *independent features*. The predictor features are usually denoted with the symbol $X$. A single $X$ is used in the case where there is only one predictor feature (usually only in toy examples) while $X$ is indexed where there is more than one predictor feature.

In our height modeling example we might start out with the case of a single Predictor Feature *age*:

```python
X = np.array([1,3,4,7,10,11,14,13.5,16,18,22])
print("Our age vector is ", X)
```

    Our age vector is  [ 1.   3.   4.   7.  10.  11.  14.  13.5 16.  18.  22. ]

### Note - Terminology
By convention we usually refer to the Target feature as the **Target** and the Predictor Features as simply the **Features**.

### Example: Height as a Function of Age
We can visualize our Height and Age data as follows:

```python
%matplotlib inline

import matplotlib.pyplot as plt

plt.plot(X,Y, "o")
plt.title('Height versus Age')
plt.xlabel('Age (years)')
plt.ylabel('Height (cm)')
plt.show()
```

Examining our data we can clearly see a relationship between Age and Height but how do we model the relationship between the two variables? We could start with correlation:

```python
from scipy.stats import pearsonr
r,p = pearsonr(X,Y)
print('r value is ',r)
print('p value is',p)
```

    r value is  0.9028496585155188
    p value is 0.00014105270724641005

Correlation confirm that there is a strong positive relationship between the two variables, but this is useless if we want to be able to predict a value for our Target (*Height*) based on a new value for our Feature (*Age*). For example lets say that we want to know the most likely value for *Height* based on an *Age* of 9. We can do this intuitively by looking at our plot and estimating what value on the Y axis corresponds to 9 on the X axis.

```python
plt.axvline(9, color='red')
plt.plot(X,Y, "o")
plt.title('Height versus Age')
plt.xlabel('Age (years)')
plt.ylabel('Height (cm)')
plt.show()
```

Rather than having to do this manually, linear regression gives us a way to estimate this by fitting a function to our data that allows us to predict a likely value for our Target based on a choice of input features.

### Definition: Linear Regression
Linear Regression is an approach to modeling the relationship between a Target and Features with a parameterised function.

The most basic function we can fit to our example is a straight line of the form:

\begin{equation}
 Y = mX + c
\end{equation}

where $m$ defines the slope of the line and $c$ defines the line's offset from the origin.

For this toy example we can look at the data and guess that a reasonable fit to the data might be

\begin{equation}
 Y = 4X + 80
\end{equation}

Generally though for linear regression we rewrite the equation above such that the intercept is placed before the slope term as follows:

\begin{equation}
 Y = 80 + 4X
\end{equation}

We can plot a graph of this line / function against our data as follows:

```python
plt.plot(X,Y, "o")
plt.title('Height versus Age')
plt.xlabel('Age (years)')
plt.ylabel('Height (cm)')
x2 = np.array([i for i in range(25)])
y2 = 80 + 4*x2
plt.plot(x2,y2)
plt.show()
```

Alternatively we might think that the linear function $Y= 60 + 6X$ might give a better fit.

```python
plt.plot(X,Y, "o")
plt.title('Height versus Age')
plt.xlabel('Age (years)')
plt.ylabel('Height (cm)')
x2 = np.array([i for i in range(25)])
y2 = 60 + 6*x2
plt.plot(x2,y2)
plt.show()
```

We might even think that a polynomial such as
\begin{equation}
Y = 50 + 15X - 0.68 X^{2} + 0.011 X^{3}
\end{equation}

would provide a better fit again.

```python
plt.plot(X,Y, "o")
plt.title('Height versus Age')
plt.xlabel('Age (years)')
plt.ylabel('Height (cm)')
x2 = np.array([i for i in range(25)])
y2 = 50 + 15*x2 - 0.68*(x2**2)  + 0.011*(x2**3)
plt.plot(x2,y2)
plt.show()
```

This is clearly the case, but for the moment we will stay focused on purely linear functions.

## Hypothesis Function

In linear regression we assume the function which best fits the data is a linear functions of x the form:

\begin{equation}
h_{\theta}(x) = \theta_{0} + \theta_{1}x
\end{equation}

In the (textbook) literature this is referred to as our **hypothesis function** as it is the thing that we are trying to test out or evaluate. The hypothesis function is also often referred to as simply **the function** or **model**. In the notes below we will refer to it simply as our model.

$\theta$ are the parameters of the model. In the examples above the parameters were selected by manually playing with the parameters until a reasonable fit was found. In linear regression we automate the selection of these parameters by minimizing a Cost Function.

## Cost Functions

**Definition** The cost function, $J$, is a function which gives us a numeric estimate of the error between the predictions of our model and the test data values.

**Formal Definition**

\begin{equation}
J(\theta_{0},\theta_{1}) = \frac{1}{2m}     \sum_{i=0}^{m} ( h_{\theta}(x^{(i)}) - y^{(i)})^{2}
\end{equation}

where
 * $\theta_{0}$ and $\theta_{1}$ are the parameters of our model
 * $m$ is the number of training examples
 * $y^{i}$ is the ith target value
 * $x^{i}$ is the ith feature vector, and
 * $h_{\theta}$ is our model (hypothesis function of $x$) that we wish to evaluate.

**Intuition**

The intuition for the cost function is that it is measuring the difference between the ground truth values for our data and the predictions of our model. However rather than us adding up the differences we add up the square of the differences. This has two positive effects. First positive and negative difference values for different training examples won't cancel each other out. Secondly our model will heavily penalize cases where the predicted value is far away from the truth, but will only lightly penalize cases where there is a small difference between truth and prediction. This allows the search for the best fit to focus broadly on fixing up the big problems.

The factor $\frac{1}{2m}$ is not technically needed for the cost function, but we introduce it as it allows us to use a very simple derivative of the cost function later when we use numerical optimization to search for the best set of parameters for our model.

This cost function is often referred to as the **Sum of Squared Error** cost function.

**Example**
We can use this function to directly calculate the cost associated with our linear equation based estimates of the height and age data as shown below.

```python
def hyp(theta0,theta1,x):
    """This is our actual model function that combines parameters with an input feature (x)
        to return a value (which would be y)"""
    return theta0 + theta1*x

def J(theta0,theta1,x,y,h):
    """This is the function that calculates our cost for a hypothesised function and a set
        of training data. """
    s = 0
    m = len(x)
    for i in range(0,m):
        delta = h(theta0,theta1,x[i]) - y[i]
        delta = delta**2
        s+= delta
    return s / (2*m)

print("Cost for y = 80 + 4*x : " + str(J(80,4,X,Y,hyp)))

print("Cost for y = 60 + 6*x : " + str(J(60,6,X,Y,hyp)))
```

    Cost for y = 80 + 4*x : 123.92045454545455
    Cost for y = 60 + 6*x : 147.23863636363637

### Minimizing Cost Functions

**Intuition**
What we would like to do is select values for $\theta_{0}$ and $\theta_{1}$ that minimize the cost function $J$.

The cost function for a linear equation with two parameters such as that above will be a surface in a 3D space. We can produce a visualization of this Cost surface by scanning over a range of values for $\theta_{0}$ and $\theta_{1}$ as follows:

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.mlab as mlab
from matplotlib import cm
import numpy as np

theta0_range = range(60,90)
theta1_range = np.arange(3,8,0.1)

theta0s, theta1s = np.meshgrid(theta0_range, theta1_range)
costs = J(theta0s,theta1s,X,Y,hyp)

fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(theta0s, theta1s, costs, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
ax.set_zlim(100, 1400)

plt.show()
```

We can alternatively visualize our cost function as a countour plat.

```python
plt.figure()
CS = plt.contour(theta0s, theta1s, costs)
plt.clabel(CS, inline=1, fontsize=10)
plt.xlabel('Theta0')
plt.ylabel("Theta1")
plt.title('Contour Plot of Cost Function')
plt.show()
```

To find the best fit for our linear equation we want to minimize the cost function to locate

\begin{equation}
min_{\theta_{0},\theta_{1}} J(\theta_{0},\theta_{1})
\end{equation}

We can use gradient descent algorithms and other optimization methods to minimize any continuous function such as J.

**Intuition**
We start with some random set of values for $\theta_{0}$ and $\theta_{1}$ and vary those values such to decrease the values for these parameters. Rather than randomly perturbing the values for $\theta_{0}$ and $\theta_{1}$, we instead examine the derivatives of the Cost function and select a change in $\theta_{0}$ and $\theta_{1}$ such to minimize the values of these parameters as quickly as possible.

### Gradient Descent Algorithm

The kernel of the Gradient Descent Algorithm is given by the following iterative function:

Repeat Until Convergence {
\begin{equation}
\theta_{j} = \theta_{j} - \alpha \frac{\partial J(\theta_{0},\theta_{1})}{\partial \theta_{j}} \;\; for \; j=0:1
\end{equation}
}

where $\alpha$ = learning rate. Large values for $\alpha$ result in big step sizes in the minimization process

In this algorithm it is implied that in each loop we first make an update for $\theta_{0}$ and then $\theta_{1}$. In practice we need to make the updates to both parameters $\theta_{0}$ and $\theta_{1}$ in parallel. To perform simultaneous updates we will introduce temporary variables - more on that later.

The Gradient Descent Algorithm is dependent on calculating the partial derivative of the Cost Function. In practice this partial derivative is expressed easily. This expression is derived below.

**Derivation**

\begin{equation}
\frac{\partial}{\partial \theta_{j}}J(\theta_{0},\theta_{1})
\end{equation}

Substituting in the equation for $J(\theta_{0},\theta_{1})$ we get

\begin{equation}
\frac{\partial}{\partial \theta_{j}}   \frac{1}{2m}     \sum_{i=0}^{m} ( (h_{\theta}(x^{(i)}) - y^{(i)})^{2} )
\end{equation}

Substituting the definition $h_{\theta}(x^{(i)}) = \theta_{0} + \theta_{1}x^{i}$ we get:

\begin{equation}
\frac{\partial}{\partial \theta_{j}}   \frac{1}{2m}     \sum_{i=0}^{m} ( (\theta_{0} + \theta_{1}x^{i} - y^{(i)})^{2} )
\end{equation}

We now perform the partial derivations for each of the parameters:

\begin{equation}
j=0 \; : \; \frac{\partial}{\partial \theta_{0}}J(\theta_{0},\theta_{1}) = \frac{1}{m} \sum_{i=0}^{m} (h_{\theta}(x^{(i)}) - y^{(i)})
\end{equation}

\begin{equation}
j=1 \; : \; \frac{\partial}{\partial \theta_{1}}J(\theta_{0},\theta_{1}) = \frac{1}{m} \sum_{i=0}^{m} (h_{\theta}(x^{(i)}) - y^{(i)}).x^{(i)}
\end{equation}

We can then substitute these partial derivative formula back into the definition of gradient descent algorithm:

Repeat Until Convergence {

  > perform simultaneously {
\begin{equation}
\theta_{0} = \theta_{0} - \alpha \frac{1}{m} \sum_{i=0}^{m} (h_{\theta}(x^{(i)}) - y^{(i)})
\end{equation}

> \begin{equation}
\theta_{1} = \theta_{1} - \alpha \frac{1}{m} \sum_{i=0}^{m} (h_{\theta}(x^{(i)}) - y^{(i)}).x^{i}
\end{equation}
  }

}

Based on the above we can provide an implementation of the Gradient Descent algorithm as shown below.

```python
def gd(t0_in,t1_in,min_delta_in,a_in,X_in,Y_in,hyp_in):
    """Implementation of a simple gradient descent algorithm. Inputs are:
        t0_in --- initial guess at parameter t0
        t1_in --- initial guess at parameter t1
        min_delta_in --- a minimum change. If in an iteraction our change is less than this, we are finished.
        a_in --- learning rate
        X_in --- vector of input values
        Y_in --- vector of output values
        hyp_in --- our hypotheis function
        """
    m = len(X_in)
    c = []
    t0s = []
    t1s = []

    t0 = t0_in
    t1 = t1_in
    t0s.append(t0_in)
    t1s.append(t1_in)

    # note - there is no do while construct in python.
    while True:
        # calculate the cost based on the current parameters
        c.append(J(t0,t1,X_in,Y_in,hyp_in))

        temp0 = t0
        temp1 = t1
        s = 0
        # iterate over each of the inputs and calculate elements for t_0 update
        for i in range(0,m):
            s+=  (hyp_in(temp0,temp1,X_in[i]) - Y_in[i])
        t0 = temp0 - a_in * (1/m) * s

        s = 0
        # iterate over each of the inputs and calculate elements for t_1 update
        for i in range(0,m):
            s+=  (hyp_in(temp0,temp1,X_in[i]) - Y_in[i]) * X_in[i]
        t1 = temp1 - a_in * (1/m) * s

        t0s.append(t0)
        t1s.append(t1)

        # check if our parameters are still seeing large updates -- if not, break
        if( (abs(temp0 - t0) < min_delta_in) and (abs(temp1 - t1) < min_delta_in) ):
            break
    return t0s, t1s, c

t0s, t1s, c = gd(62,4,0.5,0.01,X,Y,hyp)

plt.plot(c, 'ro')
plt.title('Costs for each iteration')
plt.show()

plt.figure()
CS = plt.contour(theta0s, theta1s, costs)
plt.plot(t0s,t1s)
plt.xlabel('Theta0')
plt.ylabel('Theta1')
plt.clabel(CS, inline=1, fontsize=10)
plt.title('Cost space with Gradient Descent track shown')
plt.show()

plt.plot(X,Y, "o")
plt.title('Height versus Age')
plt.xlabel('Age (years)')
plt.ylabel('Height (cm)')
x2 = np.array([i for i in range(25)])
y2 = t0s[-1] + t1s[-1]*x2
plt.plot(x2,y2)
plt.show()
```

## Normalization

Our result is reasonable but not ideal. We can see that while good progress was made in finding an appropriate value for $\theta_{1}$, very poor progress was made with $\theta_{0}$. This was due to the very small learning rate chosen. However if we had selected a much larger learning rate, than we would have had problems finding a good value for $\theta_{1}$.

This problem is due to the very different scales used in our data for our two parameters. We can see just how bad the scale problem is if we use similar scales for visualization on both axes.

```python
theta0_range = range(70,80)
theta1_range = range(-5,5)

theta0s, theta1s = np.meshgrid(theta0_range, theta1_range)
costs = J(theta0s,theta1s,X,Y,hyp)

plt.figure()
plt.contour(theta0s, theta1s, costs)
plt.show()
```

To correct for this we need to make sure that each of our input features have a similar scale. This process is referred to as **normalization** and is necessary because the gradient descent algorithms does not possess the property of scale invariance.

Within the Machine Learning community two alternate forms of normalization are often applied: Feature Scaling and Z-Score Normalization. Feature scaling, also known as min-max scaling, adjusts the range of the data such that all data $\in [0,1]$. The adjustment is made as follows:

\begin{equation}
x' = \frac{x-min(x)}{max(x) - min(x)}
\end{equation}

Z-score normalization, sometimes referred to simply as **standardization** instead expresses individual points of data as being a number of standard deviations away from the mean of the data. The adjustment is made as follows:

\begin{equation}
x' = \frac{x-mean(x)}{std(x)}
\end{equation}

where $mean(x)$ is the arithmetic mean over all items in x and $std(x)$ is the standard deviation of x.

Feature Scaling is a sensible method to use if either of the following conditions are true:
 * The original data has features with fixed maximum and minimum in the original feature space. For example a set of sensors might produce values fixed in the range 0..255.
 * The algorithm which we will be subsequently using requires our data to be in the range 0...1. For example some activation functions in Neural Networks as we will see later need data to be in this specific range.

However in general z-score standardization tends to be a more favored method of normalization within machine learning. There are a number of reasons for this, but the most important of these is that z-score normalization naturally copes even when a new test data example was out of the original range of the training data that was used to train the algorithm. Since z-score normalization has no fixed maximum or minimum we can in practice use new data with features that are out of range of our original data set.

In practice though the choice of normalization algorithm is recognized to be very much application dependent. In the following examples we make use of z-score normalization.

#### Scaling and the Target

In the example above and for illustration purposes we scaled our target variable and searched for the best fit for $\theta_{0}$ as well as $\theta_{1}$. In turns out that in practice we do not need to search for $\theta_{0}$.

Instead we can set $\theta_{0}$ based on the average at 0 for the other scaled features. $\theta_{0}$ is thus truly a constant and this is why we will not usually see p-value information regarding $\theta_{0}$.

From here on out this is the approach we will take to setting $\theta_{0}$.

#### Example: Z-Score Normalization

```python
X_new = (X-np.mean(X))/np.std(X)

plt.plot(X_new,Y, "o")
plt.title('Height versus Age - Following Z-score standardization')
plt.xlabel('Age')
plt.ylabel('Height')

t0 = np.mean(Y)
print("Y intercept is at " + str(t0))

x2 = np.arange(-2,2,0.1)

# lets use the calculated value for t0 but guess the value for t1
y2 = t0 + 25*x2
plt.plot(x2,y2)
plt.show()
```

We can now re-structure the above to use Gradient Descret to search only for $\theta_{1}$.

```python
def gd1(t0_in,t1_in,min_delta_in,a_in,X_in,Y_in,hyp_in):
    m = len(X_in)
    c = []
    t1s = []

    t1 = t1_in
    t1s.append(t1_in)

    # note - there is no do while construct in python.
    while True:
        c.append(J(t0_in,t1,X_in,Y_in,hyp_in))

        temp1 = t1

        s = 0
        for i in range(0,m):
            s+=  (hyp_in(t0_in,temp1,X_in[i]) - Y_in[i]) * X_in[i]
        t1 = temp1 - a_in * (1/m) * s

        t1s.append(t1)

        if(abs(temp1 - t1) < min_delta_in):
            break
    return t1s, c

md = 0.01
a2 = 0.75
t1s, c = gd1(t0,4,md,a2,X_new,Y,hyp)

plt.plot(c, 'ro')
plt.xlabel('Iterations')
plt.ylabel('Cost')
plt.title("Cost Values at each iteration")
plt.show()

plt.figure()

theta0_range = np.arange(110,160,0.1)
theta1_range = np.arange(0,60,0.1)
theta0s_new, theta1s_new = np.meshgrid(theta0_range, theta1_range)

costs_new = J(theta0s_new,theta1s_new,X_new,Y,hyp)

CS = plt.contour(theta0s_new, theta1s_new, costs_new)
plt.plot([t0]*len(t1s),t1s)
plt.title("Cost contour map")
plt.xlabel('t1')
plt.ylabel('t0')
plt.show()

x2 = np.arange(-2,2.5,0.1)
t1 = t1s[-1]
print(t1)
y2 = t0 + t1*x2
plt.plot(X_new,Y, "o")
plt.plot(x2,y2)
plt.show()
```

Note that we don't try to convert our thetas in the new feature space back to the original feature space. Instead when we have new test data to calculate on we transform the test input features into the new data space.

For example given a new value for an individuals age $x_{i} = 20$ we can predict the height of the person as follows:

```python
x1 = 20
# transform the feature into the normalized feature space
x1_prime = (x1-np.mean(X))/np.std(X)

# we can the calculate the estimated value of y as follows
y1 = t0 + t1*x1_prime

print(y1)
```

    171.93264254506227

## Multivariate Regression
So far our explanations and examples have centered around a simple case with only one feature. Linear Regression straightforwardly scales up to deal with multiple features.

### Multivariate Hypothesis Function
Our hypothesis function must be extended to handle multiple features. We can extend the basic form of the function as follows:

\begin{equation}
h_{\theta}(x) = \theta_{0} + \theta_{1}x_{i} + \theta_{2}x_{2} + \dots + \theta_{n}x_{n}
\end{equation}

where we have n different features.

We can simplify this function as follows:

\begin{equation}
h_{\theta}(x) = \theta_{0} + \sum_{i=1}^{n} \theta_{i}x_{i}
\end{equation}

One limitation with this equation is that we still have $\theta_{0}$ handled in a separate way to all true features. It is convenient if we note that generally:

\begin{equation}
\theta \times 1 = \theta
\end{equation}

Thus we can instead write our hypothesis function as:

\begin{equation}
h_{\theta}(x) = \theta_{0} 1 + \sum_{i=1}^{n} \theta_{i}x_{i}
\end{equation}

Or if we introduce a variable $x_{0}$ and set it to 1, then:

\begin{equation}
h_{\theta}(x) = \theta_{0}x_{0} + \sum_{i=1}^{n} \theta_{i}x_{i}
\end{equation}

Which of course is just:

\begin{equation}
h_{\theta}(x) = \sum_{i=0}^{n} \theta_{i}x_{i}
\end{equation}

where $x_{0} = 1$ and all other x's correspond to our features. Note that this calculation looks like the scalar product of two vectors.

### Vectorized Multivariate Hypothesis Function
The above approach is particularly useful as it allows us to recast our calculations in a vectorized notation. Generally:

\begin{equation}
h_{\theta}(x) = \theta^{T}x
\end{equation}

This vectorized approach takes advantage of the fact that our hypothesis function is a summation of products of input features and parameters. To illustrate we can begin with our basic case of the single feature function defined above along with one data point for that function based on values we recently defined.

```python
# let's define our two parameters into a single vector
theta = np.array([t0,t1])
# now lets define our input date. Remember that we need to create a dummy value for x_{0}
# which we set equal 1.
x = np.array([1,x1_prime])
# now multiply the matrices together
result = np.matmul(theta.T,x)
print(result)
```

    171.93264254506227

#### Parallelism through Vectorization
This vectorized approach is useful not only because of the simple notation, but it allows us to easily write out and make calculations for multiple sets of parameters in a single computational step. To illustrate this, we can extend the case above to handle multiple data points by substituting the matrix multiplication operator for the simple inner product operator.

```python
# define the raw input data pints (different values for age)
ages = np.array([12.5,17.5,20,21,25])
# normalize these based on previously established paramters
ages = (ages-np.mean(X))/np.std(X)
# construct the data matrix x that includes dummy values for x_{0} as well as the real
# age values for x_{1}
x = np.matrix([[1,1,1,1,1],ages])
# now we can matrix multiply
results = np.matmul(theta.T,x)
print(results)
```

    [[137.54763747 160.47097419 171.93264255 176.51730989 194.85597926]]

Furthermore we can extend the approach to allow us to calculate the resultant values for each data point for multiple different sets of hypotheses. This can be useful if we ever want to test multiple sets of parameters in parallel.

```python
# construct the matrix of paraeters where each row is a single set of parameters i.e., theta
thetas = np.matrix([[t0,t1],[120,30]])
results = np.matmul(thetas,x)
print(results)
```

    [[137.54763747 160.47097419 171.93264255 176.51730989 194.85597926]
     [127.76155336 151.47741085 163.3353396  168.07851109 187.05119709]]

### Multivariate Cost Function
By introducing $x_{0}=1$ we can straightforwardly rework our cost function to treat all parameters equally. The generalized form of our Cost Function becomes:

\begin{equation}
J(\theta) = \frac{1}{2m}     \sum_{i=0}^{m} ( h_{\theta}(x^{(i)}) - y^{(i)})^{2}
\end{equation}

or in code:

```python
def J(theta,x,y):
    sse = 0
    m = len(x)
    for i in range(0,m):
        h = np.matmul(theta,x[i,:].T)
        error = h - y[i]
        sqerror = error**2
        sse+= sqerror
    return sse / (2*m)
```

While the general form of our gradient descent algorithm becomes:

Repeat Until Convergence {

  > perform simultaneously {

> \begin{equation}
\theta_{j} = \theta_{j} - \alpha \frac{1}{m} \sum_{i=0}^{m} (h_{\theta}(x^{(i)}) - y^{(i)}).x_{j}^{i}
\end{equation}
  }

}

We can rewrite our gradient descent implementation to work with this generic design. The inputs to our algorithm are now:
 * theta_in - a vector of initial parameterizations
 * min_delta - the minimum change in delta permissible before gradient descent stops
 * a_in - the learning rate
 * X_in - a matrix of input feature vectors
 * Y_in - a vector of target variables of length m

Our function returns
 * thetas - a matrix of vectors where each vector is a set of parameters at a particular epoch

Note that in this simplified form we have:
 * removed the input parameter hyp_in since we now assume the hypothesis function is always of the form $\theta^{T}x$
 * stopped calculating the cost for each iteration. Instead this can be calculated afterward for visualization if needed

```python
def gd(theta_in,min_delta_in,a_in,X_in,Y_in):
    m = len(Y_in) # number of training examples
    _ , n = np.shape(X_in) # number of features

    theta = theta_in
    thetas = np.array(theta_in,ndmin=2)

    while True:
        temp = np.copy(theta)
        for j in range(0,n):
            # making calculations for jth parameter
            s = 0
            for i in range(0,m):
                s+=  (np.matmul(temp.T,np.ravel(X_in[i,:])) - Y_in[i])  * X_in[i,j]
            theta[j] = temp[j] - a_in * (1/m) * s
        thetas = np.vstack((thetas,theta))

        progressing = False
        for j in range(0,n):
            if(abs(temp[j] - theta[j]) > min_delta_in):
                progressing = progressing | True
        if not progressing: break
    return thetas
```

To demonstrate this new implementation we can firstly test it against our original single feature height and age example as follows:

```python
# define initial values for theta
theta = np.array([t0,10])
# set min_delta and the learning rate (a)
md = 0.01
a = 0.75
# define the new matrix X based around our normalized vector for X previously obtained
x_0 = [1]*len(X_new)
X = np.matrix([x_0,X_new]).T

thetas = gd(theta,md,a,X,Y)
print("Theta0 and Theta1")
print(thetas[-1:])
```

    Theta0 and Theta1
    [[130.04545455  28.99784614]]

We note that since we initialized $\theta_{0}$ to the calculated value obtained when we normalized our input feature, that the value of $\theta_{0}$ never changed. We also note that the final value for $\theta_{1}$ is almost identical to the calculation we made for it earlier.

### A Simple Multivariate Example

We can try a proper, though simple, multivariate case by defining a new multivariate data set. For this example we will look at the relationship between a target variable Growth and two input features Average Temperature and Average Rainfall.

```python
# define our new data
Y = np.array([100,110,120,120,148,130,170,140,195])
X = np.array([[22,107],[28,110],[35,100],[21,201],[29,210],[33,201],[22,330],[25,330],[33,340]])

# Redefine our standardization operation to work over 2D arrays
def standardize(x):
    col_means = np.mean(x,axis=0)
    col_std = np.std(x,axis=0)
    return (x - col_means) / col_std , col_means, col_std

# standardize our input features
X, means, stds = standardize(X)

# define initial values for theta
rows, cols = X.shape
theta = np.ones(1+cols)*1
theta[0] = np.mean(Y) # remember - we directly calculate t0

# set min_delta and learning rate (a)
md = 0.01
a = 0.05

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*rows)
X = np.hstack((x_0[:, np.newaxis],X))

# Run gradient descent
thetas = gd(theta,md,a,X,Y)

# plot a surface based on the returned function parameters

# initialize values for features x1 and x2
x_range = np.arange(-2,2,0.1)

x1s, x2s = np.meshgrid(x_range, x_range) # remember - plt.surface likes this meshgrid output
mesh_rows, mesh_cols = x1s.shape

# construct 1D arrays for x1, x2 and x0
x1 = np.ravel(x1s)
x2 = np.ravel(x2s)
x0 = np.array([1]*len(x2))

# merge 1D arrays into a single x array for calculations \n",
x = np.matrix([x0,x1,x2])

# perform calculations \n",
z = np.matmul(thetas[-1,:],x)

# reshape result to have the same organization as result of meshgrid \n",
zs = np.reshape(z,(mesh_rows,mesh_cols))

# plot the surface along with our training data points
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(x1s, x2s, zs, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
surf = ax.scatter(X[:,2], X[:,1], Y, c='r', marker='o')
ax.set_xlabel('Average Rainfall')
ax.set_ylabel('Temperature')
ax.set_zlabel('Growth')
ax.set_zlim(50, 200)
plt.show()
```

## Non-Linear Linear Regression
In our vectorized calculations above we are assuming that our hypothesis function is always of the form:

\begin{equation}
h_{\theta}(x) = \sum_{i=0}^{n} \theta_{i}x_{i}
\end{equation}

However, if we remember back to our early investigations of the Age v Height example we noted that a more complex hypothesis function of the form

\begin{equation}
Y = 50 + 15X - 0.68 X^{2} + 0.011 X^{3}
\end{equation}

seemed to give us a better fit.

We can directly calculate such cases by pre-computing the additional features $X^{2}$ and $X^{3}$ based on our original features. Redefining our Age and Height example to work with the new gradient descent algorithm we get:

```python
Y = np.array([60,90,100.5,130,128,140,160,140,165,162,155])
x1 = np.array([1,3,4,7,10,11,14,13.5,16,18,22])
x2 = x1**2
x3 = x1**3
X = np.vstack((x1,x2,x3)).T

# standardize our input features
X, means, stds = standardize(X)

# define initial values for theta
rows, cols = X.shape
theta = np.ones(1+cols)*1
theta[0] = np.mean(Y) # remember - we directly calculate t0

# set min_delta and a
md = 0.01
a = 0.05

# define the new matrix X based around our standardized vector for X previously obtained
x_0 = np.array([1]*rows)
X = np.hstack((x_0[:, np.newaxis],X))

# Run gradient descent
thetas = gd(theta,md,a,X,Y)

# Generate data for visualizing our optimized function
x1s = X[:,1]
x2s = X[:,2]
x3s = X[:,3]
x0s = np.ones(len(x1s))
xs = np.vstack((x0s,x1s,x2s,x3s)).T

# calculate y values based on this generated data and results of optimization
ys = np.matmul(thetas[-1,:],xs.T)

# plot training points and function over generated data
plt.scatter(X[:,1],Y)
plt.plot(x1s,ys)
plt.show()

print('Parameters obtained for this normalized data')
print(thetas[-1,:])
```

# Assignments and Questions

For this week there are no specific assignments. Instead everyone should focus on:

 * Make sure you are comfortable using Google Collaborator to create new projects.
 * Improve your coding practice for working with Python. There are lots and lots of tutorials online. Here is a pretty detailed set of tutorial videos. I don't know if this guy is good -- i haven't wantched the videos myself -- but his ratio of likes to dislikes is good.
 https://www.youtube.com/watch?v=rfscVS0vtbw
 * Non essential for now, but making sure you have a runtime environment setup that supports python, jupyter, numpy, scipy, and optionally for this week, tensorflow.
 * Reading through both these notes and the notes for week 2 on Logistic Regression

**Optional Question**

If you are already well set up in this area, formulate a small study in python to investigate the effects of correlated predictors on linear regression. Compare and contrast the results with expected effects if regression was estimated within a Bayesian framework.

Complete Modules/Deep Learning/week 2/Linear Regression.ipynb


```python

```

# Diophantine Equation

Goal - finding the integer solutions

$a \in \mathbb{Z}$ and $b \in \mathbb{Z}$

Then the following diophantine equation has a solution.

$a.x + b.y = gcd(a,b)$

----

Example:
47x + 30y = 1
Use Euclidean Algorithm
a / b = q + r  =>   a = b.q + r

47/30 = 1, r 17

$\begin{equation}
\begin{split}
47 &= 30(1) &+ 17  &\implies 17 &= 47(1) &+ 30(-1) \\
30 &= 17(1) &+ 13  &\implies 13 &= 30(1) &+ 17(-1) \\
17 &= 13(1) &+ 4  &\implies 4 &= 17(1) &+ 13(-1) \\
13 &= 4(3) &+ 1  &\implies 1 &= 13(1) &+ 4(-3) \\
\end{split}
\end{equation}$

$\begin{equation}
\begin{split}
1 &= 13(1) &+ 4(-3) &&\text{  substitute 4} \\
&= 13(1)   &+ \Big(17(1) &+ 13(-1)\Big)&(-3) \\
&= 13(1)   &+ 17(-3) &+ 13(3) \\
&= 17(-3)  &+ 13(4) &&\text{  substitute 13} \\
&= 17(-3)  &+ \Big( 30(1) &+ 17(-1) \Big)&(4) \\
&= 17(-3)  &+ 30(4) &+ 17(-4) \\
&= 30(4)   &+ 17(-7) &&\text{  substitute 17}\\
&= 30(4)   &+ \Big( 47(1) &+ 30(-1) \Big)&(-7) \\
&= 30(4)   &+  47(-7) &+ 30(7) \\
&= 47(-7)  &+ 30(11) \\
\end{split}
\end{equation}$

47x + 30y = 1 has a solution, x=-7, y=11

```python

```

```python

```


# Jet transport for the simple pendulum

In this notebook we will use `TaylorSeries.jl` and `TaylorIntegration.jl` in order to integrate the the simple pendulum, defined by the Hamiltonian:

$$
H(x, p) = \frac{1}{2}p^2-\cos x
$$

The corresponding equations of motion are:

$$
    \begin{align}
        \dot x&=p \\
        \dot p&=-\sin x
    \end{align}
$$

We will integrate this problem for the initial condition $x(t_0)=x_0$, $p(t_0)=p_0$, as well as a neighbourhood $U_0$ around this initial condition, using the jet transport technique. For simplicity, we will take $p_0=0$, and $t_0=0$. Furthermore, we will choose $x_0$ such that the pendulum librates; that is, we will choose the numerical value of the energy, $E=H(x_0,p_0)=-\cos x_0$, such that the pendulum's motion in phase space is "below" (inside) the region bounded by the separatrix. Then, for the initial conditions given above, the librational period $T$ of the pendulum is

$$
T=\frac{4}{\sqrt{2}}\int_0^{x_0}\frac{dx}{\sqrt{\cos x_0-\cos x}}
$$

Or else, in terms of the complete elliptic integral of the first kind, $K$:

$$
T=4K(\sin(x_0/2))
$$

The jet transport technique consists in considering the neighbourhood $U_0$ as being parametrized by the sum $q_0+\xi$, where $q_0=(x_0,p_0)$ represents the coordinates of the initial condition in phase space, and $\xi=(\xi_1,\xi_2)$ represents an small variation with respect to this initial condition. We re-interpret each component of the sum $q_0+\xi$ as a multivariate polynomial in the variables $\xi_1$ and $\xi_2$, and then we integrate these multivariate polynomials in time using Taylor's method.

The packages we need to load are:

```julia
using TaylorIntegration, Elliptic
```

The Hamiltonian for the simple pendulum is:

```julia
H(x) = 0.5x[2].^2-cos(x[1])
```

The LHS of the equations of motion is:

```julia
function pendulum!(t, x, dx)
    dx[1] = x[2]
    dx[2] = -sin(x[1])
end
```

We must setup the the `TaylorN` variables necessary to perform the jet transport.; `varorder` represents the order of expansion in the variations $\xi$.

```julia
const varorder = 8
p = set_variables("ξ", numvars=2, order=varorder)
```

The nominal initial condition is:

```julia
q0 = [1.3, 0.0]
```

The initial value of the energy is:

```julia
H0 = H(q0)
```

The parametrization of the neighbourhood $U_0$ is represented by the following array of `TaylorN`'s:

```julia
q0TN = q0 + p
```

To get a feel of how the jet transport technique works, we will evaluate the
Hamiltonian at $q_0+\xi$ in order to obtain the 8-th order Taylor expansion of
the Hamiltonian wrt the variations $\xi$, around the initial condition $q_0$:

```julia
H(q0TN)
```

Below, we setup some parameters for the Taylor integration. In particular, we will use a method of `taylorinteg` which returns the solution only at `t0`, `t0+integstep`, `t0+2integstep`,...,`tmax`, where `t0` and `tmax` are the initial and final times of integration, whereas `integstep` is the time interval between successive evaluations of the solution vector, chosen by the user. In this case, we are choosing `integstep` to be $\frac{1}{8}$th of the period of the pendulum.

```julia
const order = 28 #the order of the Taylor expansion wrt time
const abstol = 1e-20 #the absolute tolerance of the integration
const T = 4*Elliptic.K(sin(q0[1]/2)^2) #the librational period
const t0 = 0.0 #the initial time
const tmax = 6T #the final time
const integstep = T/8 #the time interval between successive evaluations of the solution vector
```

Then, we perform a Taylor integration over the initial condition `x0TN`:

```julia
#this is a "warmup lap"
@time xv = taylorinteg(pendulum!, q0TN, t0:integstep:tmax, order, abstol, maxsteps=1);
```

```julia
tv = t0:integstep:tmax;
@time xv = taylorinteg(pendulum!, q0TN, t0:integstep:tmax, order, abstol);
```

Note that we the integration above "works" for any initial neighborhood $U_0$ around the nominal initial condition $x_0$, of sufficiently small radius.

We will consider the particular case where $U_0$ is a disk of radius $r$, centered at $q_0$; that is $U_0=\{ q_0+\xi:\xi=(r\cos\phi,r\sin\phi); \phi\in[0,2\pi) \}$ for a given radius $r>0$. We will denote by $U_t$ the propagation of the initial neighbourhood $U_0$ evaluated at time $t$. Also, we denote by $q(t)$ the coordinates of the nominal solution at time $t$: $q(t)=(x(t),p(t))$. Likewise, we will denote the propagation at time $t$ of a given initial variation $\xi_0$ by $\xi(t)$. Then, we can compute the propagation of the boundary $\partial U_t$ of the neighbourhood $U_t$.

```julia
polar2cart(r, ϕ) = [r*cos(ϕ), r*sin(ϕ)]
r = 0.05 #the radius of the neighbourhood
ϕ = 0.0:0.1:(2π+0.1) #the values of the angle
ξv = polar2cart.(r, ϕ);
```

Evaluate the jet at $\partial U_{x(t)}$ (the boundary of $U_{x(t)}$) at each value of the solution vector `xv`. We organize these values such that we can plot them later:

```julia
@time xjet_plot = map(λ->λ.(ξv), xv[:,1])
@time pjet_plot = map(λ->λ.(ξv), xv[:,2]);
```

Evaluate the jet at the nominal solution, which corresponds to $\xi=(0,0)$, at each value of the solution vector `xv`:

```julia
x_nom = xv[:,1]()
p_nom = xv[:,2]();
```

Now, we plot the nominal solution (black dots), as well as the evolution of the neighbourhood $U_0$ (in colors), each $\frac{1}{8}$th of a period. The initial condition corresponds to the black dot situated at $q_0=(1.3,0)$

```julia
using Plots, LaTeXStrings
gr()
```

```julia
plot(
    xjet_plot,
    pjet_plot,
    xaxis=(L"$x$",),
    yaxis=(L"$p$",),
    # title="Simple pendulum phase space",
    leg=false,
    aspect_ratio=1.0
)
scatter!(
    x_nom,
    p_nom,
    color=:black,
    m=(1,2.8,stroke(0))
)
#savefig("pendulum.pdf")
```

```julia

```


```python
import symbolic_modern_robotics as smr
import sympy as s
s.init_printing()
```

```python
# unit rotation axis
omega_hat = s.Matrix([[0],[0],[1]])
theta = s.pi/4
# 3-vector form  of angular velocity
omega = omega_hat*theta
omega
```

$\displaystyle \left[\begin{matrix}0\\0\\\frac{\pi}{4}\end{matrix}\right]$

```python
# get Unit axis of Rotation and theta from 3-vector of exponential coordinates
smr.AxisAng3(omega)
```

$\displaystyle \left( \left[\begin{matrix}0\\0\\1\end{matrix}\right], \  \frac{\pi}{4}\right)$

```python
# get so3(skew-symmetric matrix) from 3-vector of exponential coordinates
so3 = smr.VecToso3(omega)
so3
```

$\displaystyle \left[\begin{matrix}0 & - \frac{\pi}{4} & 0\\\frac{\pi}{4} & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

```python
# get 3-vector of exponential coordinates from so3(skew-symmetric matrix)
omega = smr.so3ToVec(so3)
omega
```

$\displaystyle \left[\begin{matrix}0\\0\\\frac{\pi}{4}\end{matrix}\right]$

```python
# get SO3(Rotation matrix) from so3(skew-symmetric matrix)
R = smr.MatrixExp3(so3)
R
```

$\displaystyle \left[\begin{matrix}\frac{\sqrt{2}}{2} & - \frac{\sqrt{2}}{2} & 0\\\frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2} & 0\\0 & 0 & 1\end{matrix}\right]$

```python
# get so3(skew-symmetric matrix) from SO3(Rotation matrix)
so3 = smr.MatrixLog3(R)
so3
```

$\displaystyle \left[\begin{matrix}0 & - \frac{\pi}{4} & 0\\\frac{\pi}{4} & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

```python
R = s.Matrix([[1, 0, 0],
              [0, 0, 1],
              [0, -1, 0]])
smr.so3ToVec(smr.MatrixLog3(R))
#smr.AxisAng3(smr.so3ToVec(smr.MatrixLog3(R)))

```

$\displaystyle \left[\begin{matrix}- \frac{\pi}{2}\\0\\0\end{matrix}\right]$

```python
R1 = s.Matrix([[1, 0, 0],
              [0, 0, -1],
              [0, 1, 0]])

R2 = s.Matrix([[-1, 0, 0],
              [0, -1, 0],
              [0, 0, 1]])

R1*R2

#smr.so3ToVec(smr.MatrixLog3(R1*R2))

```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0\\0 & 0 & -1\\0 & -1 & 0\end{matrix}\right]$

```python
# 6-vector form twist, V
V = s.Matrix([0,0,0,1,2,3])
V
```

$\displaystyle \left[\begin{matrix}0\\0\\0\\1\\2\\3\end{matrix}\right]$

```python
# get se3(twist 4x4 matrix) from 6-vector spatial velocity, V
se3 = smr.VecTose3(V)
se3
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 1\\0 & 0 & 0 & 2\\0 & 0 & 0 & 3\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
V = smr.se3ToVec(se3)
```

```python
S, theta_dot = smr.AxisAng6(V)
S, theta_dot
```

$\displaystyle \left( \left[\begin{matrix}0\\0\\0\\\frac{\sqrt{14}}{14}\\\frac{\sqrt{14}}{7}\\\frac{3 \sqrt{14}}{14}\end{matrix}\right], \  \sqrt{14}\right)$

```python
S.evalf(), theta_dot.evalf()
```

$\displaystyle \left( \left[\begin{matrix}0\\0\\0\\0.267261241912424\\0.534522483824849\\0.801783725737273\end{matrix}\right], \  3.74165738677394\right)$

```python
# get 6-vector spatial velocity, V from se3(twist 4x4 matrix)
V = smr.se3ToVec(se3)
V
```

$\displaystyle \left[\begin{matrix}0\\0\\0\\1\\2\\3\end{matrix}\right]$

```python
# get Homogeneous Transformation matrix T from se3(twist 4x4 matrix)
T = smr.MatrixExp6(se3)
T
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 1\\0 & 1 & 0 & 2\\0 & 0 & 1 & 3\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
# get se3(twist 4x4 matrix) from Homogeneous Transformation matrix, T
se3 = smr.MatrixLog6(T)
se3
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 1\\0 & 0 & 0 & 2\\0 & 0 & 0 & 3\\0 & 0 & 0 & 0\end{matrix}\right]$

```python

```

```python

```

```python

```


# CSE 330 Numerical Analysis Lab

### Lab 7: Root Finding using Newton's Mehtod

### Instructions
---

This week we will have two assignments.
1. Finding root of a polynomial using Newton's method.
2. Finding root of system of implicit functions using Newton's method.

### Root Finding (Newton's Method):
Newton's method for root finding is basically a special case of *Fixed-point* method which converges superlinearly. In this method we define $g(x)$ as this,

$$g(x) = x - \frac{f(x)}{f^{\prime}(x)}$$

so, if after $k$-th iteration we find $x_k$, then $x_{k+1}$ is,

$$x_{k+1} = x_k - \frac{f(x_k)}{f^{\prime}(x_k)} $$

Let's write a code for this.

```
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from numpy.polynomial import Polynomial

# Define a polynomial named 'f'
f = Polynomial([2.0, 1.0, -6.0, -2.0, 2.5, 1.0])

# Define the first derivative of polynomial 'f' named 'f_prime'
f_prime = f.deriv(1)

# Draw the X-axis
plt.axhline(y=0, color='k')

# Generate 100 values of x.
x = np.linspace(-2.5, 1.6, 100)

# Calculate y-values of corresponding x-values of f(x).
y = f(x)

# Plot the graph of f(x) using x-values and y-values
plt.plot(x, y)

# Plot the roots of the graph
plt.plot(f.roots(), f(f.roots()), 'ro')

# Print the roots of the function f(x)
print(f.roots())

# Calcuate y-values for corresponding x-values of f'(x).
y = f_prime(x)

# Plot the graph of f'(x)
plt.plot(x, y)

# Plot the roots of f'(x). Notice that, where f'(x) is zero the slop of f(x) is zero.
plt.plot(f_prime.roots(), f_prime(f_prime.roots()), 'bo')

# Print the roots of f'(x).
print(f_prime.roots())

# Guli mari tor template a
# Ami nije code likhum
x_zero = 6435
list_x = [x_zero]
list_y = []

k = 0
while True:
    list_y.append(f(list_x[k]))

    if abs(list_y[k]) < 1e-7 or abs(f_prime(list_x[k])) < 1e-7:
        break
    else:
        list_x.append(list_x[k] - (list_y[k] / f_prime(list_x[k])))

    k += 1

df = pd.DataFrame({"x": list_x, "f(x)": list_y})
print(df)
plt.show()
```

### Solving System of equations

We can use Newton's method to solve system of implicit functions. For example. Let's take assume $y=f_1(x)$ and $y=f_2(x)$ are two functions such that,

\begin{align}
xy - y^3 - 1 &= 0\\
x^2y + y -5 &= 0
\end{align}

Let's take a look at their graphs.

```
import matplotlib.pyplot as plt
from numpy import arange
from numpy import meshgrid

# Define a small value
delta = 0.01

# Create a set of x-values in the specified range with delta difference
# for example in this case the x values will be, -4.00, -3.99, -3.98, ... 4.99, 5.00 etc.
xrange = arange(-4, 5, delta)
yrange = arange(-3, 6, delta)

# Create a meshgrid (what is a meshgrid? it is basically co-ordinate matrix. you don't need to worry about this too much for this assignment)
# If you are interested take a look here: https://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html
X, Y = meshgrid(xrange,yrange)

# Let's define two functions as F1 and F2
F1 = X*Y - Y**3 - 1
F2 = X**2*Y + Y - 5

# Draw the plots for F1 (red) and F2 (blue).
plt.contour(X, Y, (F1), [0], colors=['red'])
plt.contour(X, Y, (F2), [0], colors=['blue'])
plt.show()
```

Now how do we solve these pair of equations? We will use matrix representations for these equations. Let, $\boldsymbol{s} = (x, y)^T$ and $\boldsymbol{f} = \left[f_1(\boldsymbol{s}), f_2(\boldsymbol{s}) \right] ^T$. Then, we can write,

\begin{align}
0 \approx f_1(\boldsymbol{s}_{k+1}) \approx f_1(\boldsymbol{s}_k) + \frac{\partial f_1}{\partial x} \biggr\rvert_{\boldsymbol{s}_k} (x_{k+1} - x_k) + \frac{\partial f_1}{\partial y} \biggr\rvert_{\boldsymbol{s}_k} (y_{k+1} - y_k)\\
0 \approx f_2(\boldsymbol{s}_{k+1}) \approx f_2(\boldsymbol{s}_k) + \frac{\partial f_2}{\partial x} \biggr\rvert_{\boldsymbol{s}_k} (x_{k+1} - x_k) + \frac{\partial f_2}{\partial y} \biggr\rvert_{\boldsymbol{s}_k} (y_{k+1} - y_k)
\end{align}

We can rewrite this in matrix form, like this,
\begin{align}
\bigg({\begin{array}{c} 0 \\0 \end{array} }\bigg) = \bigg({\begin{array}{c} f_1(\boldsymbol{s}_k) \\f_2(\boldsymbol{s}_k) \end{array} }\bigg) + \Bigg({\begin{array}{cc} \frac{\partial f_1}{\partial x}\boldsymbol{s}_k & \frac{\partial f_1}{\partial y}\boldsymbol{s}_k \\\frac{\partial f_2}{\partial x}\boldsymbol{s}_k & \frac{\partial f_2}{\partial y}\boldsymbol{s}_k \end{array} }\Bigg)\bigg({\begin{array}{c} x_{k+1} - x_k \\y_{k+1} - y_k \end{array} }\bigg)
\end{align}

The matrix part here is called $\boldsymbol{J}\text{acobian Matrix}, J(\boldsymbol{s}_k)$. From there we can derive,

\begin{equation}
\boldsymbol{s}_{k+1} = \boldsymbol{s}_k - \boldsymbol{J}^{-1}(\boldsymbol{s}_k)\boldsymbol{f}(\boldsymbol{s}_k)
\end{equation}

*Note:* Please refer to the class note for detailed proof.

For this purpose we will need to use `sympy` library. Let's take a look into it.

### `sympy` Library
Using `sympy` we can express mathematical expressions nicely.

```
# These lines are included for beautification
# from google.colab.output._publish import javascript
#url = "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=default"
#javascript(url=url)
# Don't worry about these lines

# import sympy
import sympy as sp
import math

# Defing x and y as mathematical symbols
vars = sp.symbols('x y')
x, y = vars

# Create an expression
expr1 = sp.sqrt(5 * x**3 + 1/x)
print("Expression 1:")
display(expr1)
print()
# We can substitute values like this
print(expr1.subs({x: 4.0}))

# Create another expression
expr2 = x**y - sp.exp(x**2)
print("\nExpression 2:")
display(expr2)
print()
print(expr2.subs({x: 0.1, y:0.2}))

# We can also differentiate
expr3 = 4*x**3 - 3*x**2 + sp.sqrt(1+x**2)
print("\nExpression 3:")
display(expr3)
print("\nd/d(x) of expr3:")
display(sp.diff(expr3))
print()

# Similarly we can also calculate partial derivative
expr4 = 4*x**3*y**2 - 3*x**2*y**3 + (sp.sqrt(1+x**2+y**2))
print("\nExpression 4:")
display(expr4)
print("\nd/d(x) of expr4:")
display(sp.diff(expr4, x))
print()
print("\nd/d(y) of expr4:")
display(sp.diff(expr4, y))
print()
```

    Expression 1:

$\displaystyle \sqrt{5 x^{3} + \frac{1}{x}}$

    17.8955301681733

    Expression 2:

$\displaystyle x^{y} - e^{x^{2}}$

    -0.379092822603975

    Expression 3:

$\displaystyle 4 x^{3} - 3 x^{2} + \sqrt{x^{2} + 1}$

    d/d(x) of expr3:

$\displaystyle 12 x^{2} - 6 x + \frac{x}{\sqrt{x^{2} + 1}}$

    Expression 4:

$\displaystyle 4 x^{3} y^{2} - 3 x^{2} y^{3} + \sqrt{x^{2} + y^{2} + 1}$

    d/d(x) of expr4:

$\displaystyle 12 x^{2} y^{2} - 6 x y^{3} + \frac{x}{\sqrt{x^{2} + y^{2} + 1}}$

    d/d(y) of expr4:

$\displaystyle 8 x^{3} y - 9 x^{2} y^{2} + \frac{y}{\sqrt{x^{2} + y^{2} + 1}}$

Now let's try this assignment.

```
import numpy as np
import sympy as sp

# Define x and y as mathematical symbols
vars = sp.symbols('x y')
x, y = vars

# Define the functions
f = ([x*y - y**3 - 1, x**2 * y + y - 5])

# Initialise Jacobian matrix
J = sp.zeros(len(f),len(vars))

# Fill Jacobian matrix with entries
for i, fi in enumerate(f):
    for j, s in enumerate(vars):
        J[i,j] = sp.diff(fi, s)

# Find the inverse of Jacobian Matrix
J_inv = sp.Matrix.inv(J)

# Initialize solution s with starting value x_0 = 2.0 and y_0 = 3.0
s = sp.Matrix([
    2.0,
    3.0
])

# Make a dictionary using the initial values
dictionary = {
    x: s[0],
    y: s[1]
}

# calculate f(s_k) using initial values
f_sk = sp.Matrix([
    f[0].subs(dictionary),
    f[1].subs(dictionary)
])

# Start iterating using while loop
i = 0
while i<10:
    # Calculate value of inverse jacobian, j^-1(sk), j_val
    j_val = J_inv.subs(dictionary)
    #print(j_val)

    # Calculate the new value of s using iterative formula
    s -= j_val * f_sk
    #print(s)

    # Update the dictionary using newer values
    dictionary = {
            x: s[0],
            y: s[1]
    }

    # Update f(s_k) using newer values of s_k
    f_sk = sp.Matrix([
        f[0].subs(dictionary),
        f[1].subs(dictionary)
    ])

    print(s)
    i += 1

print()
```

    Matrix([[1.55555555555556], [2.06666666666667]])
    Matrix([[1.54720541388114], [1.47779333496041]])
    Matrix([[1.78053502926115], [1.15886481100565]])
    Matrix([[1.95284300049945], [1.02844268807581]])
    Matrix([[1.99776297310451], [1.00124040908490]])
    Matrix([[1.99999523621189], [1.00000259961750]])
    Matrix([[1.99999999997887], [1.00000000001153]])
    Matrix([[2.00000000000000], [1.00000000000000]])
    Matrix([[2.00000000000000], [1.00000000000000]])
    Matrix([[2.00000000000000], [1.00000000000000]])


# Notes on Chapter 4 of *Hands-On Machine Learning with Scikit-Learn, Keras, & TensorFlow* by Aurélien Géron

Basic SVC example with Iris dataset

```python
import numpy as np
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
from matplotlib import pyplot as plt
```

```python
iris = datasets.load_iris()
X = iris['data'][:, (2,3)] # petal length and width
y = (iris['target'] == 2).astype(np.float64) # Iris virginica
```

```python
svm_clf = Pipeline([
    ('scaler', StandardScaler()),
    ('linear_svc', LinearSVC(C=1, loss='hinge'))
])

svm_clf.fit(X, y)
svm_clf.predict([[5.5, 1.7]])
```

    array([1.])

Let's make a simple plotting function to see the decision boundary (similar to the plots the author shows)

```python
def plot_2D_classifier(m, X, y):
    axis_mins = np.min(X, axis=0)
    axis_maxs = np.max(X, axis=0)
    axis_range = axis_maxs - axis_mins

    axis_padding = 0.2 * axis_range
    axis_mins = -axis_padding + axis_mins
    axis_maxs = axis_padding + axis_maxs
    axis_range = 2*axis_padding + axis_range

    plt.scatter(X[:,0], X[:,1], c=y)
    plt.xlim(axis_mins[0], axis_maxs[0])
    plt.ylim(axis_mins[1], axis_maxs[1])

    uv = np.linspace(axis_mins, axis_maxs, 300)
    grid_u, grid_v = np.meshgrid(uv[:,0], uv[:,1])
    grid_X = np.hstack(
        (grid_u.flatten()[:,np.newaxis],
         grid_v.flatten()[:,np.newaxis]))
    grid_decision = m.decision_function(grid_X).reshape(grid_u.shape)
    grid_predict = m.predict(grid_X).reshape(grid_u.shape)
    plt.contour(grid_u, grid_v, grid_decision, alpha=0.1)
    plt.contour(grid_u, grid_v, grid_predict, 1)
```

```python
plot_2D_classifier(svm_clf, X, y)
```

Non-linear example.  We start by loading a synthetic non-linear dataset.

```python
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
```

This obviously will not work well with a simple linear model:

```python
svm_clf.fit(X, y)
plot_2D_classifier(svm_clf, X, y)
```

Next, we fit by adding polynomial features.

```python
from sklearn.preprocessing import PolynomialFeatures

poly_svm = Pipeline([
    ('poly_features', PolynomialFeatures(degree=3)),
    ('scaler', StandardScaler()),
    ('svm_clf', LinearSVC(C=10, loss='hinge', max_iter=5000, tol=1e-2))
])

poly_svm.fit(X, y)
plot_2D_classifier(poly_svm, X, y)
```

We potentially can have better performance with polynomial kernels:

```python
from sklearn.svm import SVC
polyk_svm = Pipeline([
    ('scaler', StandardScaler()),
    ('svm_clf', SVC(kernel='poly', degree=3, coef0=1, C=5, max_iter=5000))
])

polyk_svm.fit(X, y)
plot_2D_classifier(polyk_svm, X, y)
```

We can also try radial basis functions:

```python
rbf_svm = Pipeline([
    ('scaler', StandardScaler()),
    ('svm_clf', SVC(kernel='rbf', gamma=5, C=0.001, max_iter=5000))
])

rbf_svm.fit(X, y)
plot_2D_classifier(rbf_svm, X, y)
```

SVMs also work for regression, of course.

```python
np.random.seed(42)
X = np.random.uniform(0., 2., (100,1))
y = (2*X + 3 + np.random.normal(0., 1., X.shape))[:,0]

def plot_1D_regressor(model, X, y):
    X_hat = np.linspace(np.min(X), np.max(X), 500)[:, np.newaxis]
    y_hat = model.predict(X_hat)
    plt.scatter(X, y)
    plt.plot(X_hat, y_hat)
```

```python
from sklearn.svm import LinearSVR

svr = LinearSVR(epsilon=1.)
svr.fit(X, y)
plot_1D_regressor(svr, X, y)
```

Polynomial features are also supported.

```python
X = np.random.uniform(-1., 1., (100,1))
y = (0.5*X**2 + 3 + np.random.normal(0., 0.1, X.shape))[:,0]
```

```python
np.random.seed(42)
from sklearn.svm import SVR

svr = SVR(kernel="poly", degree=2, C=100, epsilon=0.1)
svr.fit(X, y)
plot_1D_regressor(svr, X, y)
```

## Exercises

### 5.1

Maximizing the width of the decision boundary

### 5.2

The points limiting further expansion of the decision boundary

### 5.3

So that the width in the decision boundary is sensible for each dimension.

### 5.4

Not directly, but the distance from the center of the street and the width of the street can be used to construct a second model for confidence/probability.

### 5.5

Only if you need to use the kernel trick; otherwise it is likely more efficient to use the primal form.

### 5.6

Increase both gamma and/or C.

### 5.7

$$
\begin{align}
H_{i,j} &= \cases{1,&if $i = j$ and $i \ne 0$;\cr0,&otherwise.\cr} \\
f_i &= 0 \\
A_{i,j} &= \cases{1,&when $j = n + 1$ where $n$ is the number of features;\cr X_{i,j},&otherwise.\cr}
\end{align}
$$

### 5.8

First we generate some test data:

```python
X,y = datasets.make_classification(n_features=2, n_redundant=0, n_repeated=0, class_sep=0.3, random_state=42, n_clusters_per_class=1)
plt.scatter(X[:,0], X[:,1], c=y)
```

First the linear SVC:

```python
linear_svc = Pipeline([
    ('scaler', StandardScaler()),
    ('linear_svc', LinearSVC(C=1, loss='hinge'))
])

linear_svc.fit(X, y)
plot_2D_classifier(linear_svc, X, y)
```

Next, the standard SVC forced to use the linear model:

```python
svc = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC(kernel="poly", degree=1))
])

svc.fit(X, y)
plot_2D_classifier(svc, X, y)
```

Finally the SGDClassifier

```python
from sklearn.linear_model import SGDClassifier
sgd = Pipeline([
    ('scaler', StandardScaler()),
    ('sgd', SGDClassifier())
])

sgd.fit(X, y)
plot_2D_classifier(sgd, X, y)
```

### 5.9

First, load the MNIST dataset

```python
X, y = datasets.fetch_openml('mnist_784', version=1, return_X_y=True, as_frame=False)
```

```python
plt.imshow(X[1].reshape((28,28)), cmap="binary")
```

Split the data into test and training data

```python
X_train = X[:60000]
y_train = y[:60000]

X_test = X[60000:]
y_test = y[60000:]
```

Use a subset of the training data for faster prototyping

```python
X_minitrain = X_train[:1500]
y_minitrain = y_train[:1500]
```

```python
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

def minst_svc(kernel, **params):
    return Pipeline([
        ('scaler', StandardScaler()),
        ('svc', SVC(kernel=kernel, **params))
    ])
```

```python
cross_val_score(minst_svc("poly", degree=8), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.168, 0.166, 0.164])

```python
cross_val_score(minst_svc("poly", degree=12), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.144, 0.166, 0.148])

```python
cross_val_score(minst_svc("poly", degree=6), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.198, 0.188, 0.18 ])

```python
cross_val_score(minst_svc("poly", degree=4), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.298, 0.262, 0.294])

```python
cross_val_score(minst_svc("poly", degree=2), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.836, 0.796, 0.802])

```python
cross_val_score(minst_svc("poly", degree=1), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.876, 0.862, 0.852])

```python
cross_val_score(minst_svc("poly", degree=1, C=10), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.852, 0.846, 0.846])

```python
cross_val_score(minst_svc("poly", degree=1, C=0.1), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.792, 0.762, 0.752])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=1, C=1, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.118, 0.118, 0.118])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.001, C=1, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.862, 0.846, 0.818])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.0001, C=1, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.794, 0.756, 0.746])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.001, C=10, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.85 , 0.844, 0.834])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.001, C=100, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.85 , 0.844, 0.834])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.001, C=0.1, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.662, 0.634, 0.66 ])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.001, C=0.5, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.842, 0.83 , 0.792])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.001, C=2, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.858, 0.854, 0.828])

```python
cross_val_score(minst_svc(kernel='rbf', gamma=0.001, C=0.8, max_iter=5000), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    array([0.86 , 0.838, 0.816])

```python
def minst_lsvc(**params):
    return Pipeline([
        ('scaler', StandardScaler()),
        ('linear_svc', LinearSVC(C=1, loss='hinge'))
    ])
```

```python
cross_val_score(minst_lsvc(C=1, loss='hinge'), X_minitrain, y_minitrain, cv=3, scoring="accuracy")
```

    /home/kms15/.local/lib/python3.6/site-packages/sklearn/svm/_base.py:977: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      "the number of iterations.", ConvergenceWarning)
    /home/kms15/.local/lib/python3.6/site-packages/sklearn/svm/_base.py:977: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      "the number of iterations.", ConvergenceWarning)

    array([0.752, 0.77 , 0.76 ])

```python
svc = minst_svc("poly", degree=1)
svc.fit(X_train, y_train)
```

    Pipeline(steps=[('scaler', StandardScaler()),
                    ('svc', SVC(degree=1, kernel='poly'))])

```python
from sklearn.metrics import accuracy_score
y_test_predict = svc.predict(X_test)
accuracy_score(y_test, y_test_predict)
```

    0.9452

```python

```


```python
from system_soln_funcs import solution_handler, write_soln_func
import sympy as sp
import pprint
pp = pprint.PrettyPrinter()

var_names = [
             'wheel_diameter',
             'wheel_radius',
             'crank_arm_length',
             'bb_height',
             'pedal_clearance',
             'bb_drop'
            ]

constants = [
            ]

for v in var_names + constants:
    sp.var(v)

input_names = var_names

equations = [
             sp.Eq(wheel_diameter, wheel_radius * 2),
             sp.Eq(bb_height, wheel_radius - bb_drop),
             sp.Eq(pedal_clearance, bb_height - crank_arm_length),
            ]

soln = sp.solve(equations, ['wheel_diameter', 'bb_height', 'crank_arm_length'], dict=True)

print soln
```

    {wheel_diameter: 2*wheel_radius, crank_arm_length: -bb_drop - pedal_clearance + wheel_radius, bb_height: -bb_drop + wheel_radius}


```python
from sympy import *
init_printing()
```

```python
th1, th2, th3 = symbols('theta_1, theta_2, theta_3')
dcm_321 = Matrix([[cos(th1)*cos(th2),
         cos(th1)*sin(th2)*sin(th3) - sin(th1)*cos(th3),
         cos(th1)*sin(th2)*cos(th3) + sin(th1)*sin(th3)],
        [sin(th1)*cos(th2),
         sin(th1)*sin(th2)*sin(th3) + cos(th1)*cos(th3),
         sin(th1)*sin(th2)*cos(th3) - cos(th1)*sin(th3)],
        [-sin(th2),
        cos(th2)*sin(th3),
        cos(th2)*cos(th3)]
    ])
dcm_321
```

```python
phi, theta, psi = symbols('phi, theta, psi')
sub_eulers = {
    th1: psi, th2: theta, th3: phi
}
dcm_321_euler = dcm_321.subs(sub_eulers)
dcm_321_euler
```

```python
dcm_pitch_up = dcm_321_euler.subs({theta:pi/2, phi:0})
dcm_pitch_up
```

```python
atan2(dcm_pitch_up[1,2], dcm_pitch_up[0,2])
```

```python
dcm_pitch_down = dcm_321_euler.subs({theta:pi/2, phi:0})
dcm_pitch_down
```

```python
dcm_pitch_down = dcm_321_euler.subs({theta:-pi/2, phi:0})
atan2(-dcm_pitch_down[1,2], -dcm_pitch_down[0,2])
```


# Ecuación de Poisson

INTRODUCCION

## La ecuación diferencial parcial más sencilla

Solucionaremos la ecuación de Poisson, el problema "Hola Mundo" de la computación científica:
    Encontrar la incógnita $u$, conociendo el término fuente $f$, tal que:
    \begin{align}
      -\triangle u &= f, \ && \text{en} \: \Omega,\\
      u &= u_0, && \text{sobre} \: \partial\Omega.
    \end{align}

La ecuación modela distintos fenómenos difusivos:

- Conducción de calor, electrostática, flujos no viscosos, ondas.
- Como parte de métodos de solución de EDPs para configuraciones no-clásicas.

## La receta numérica para aproximar su solución

Aproximar numéricamente el problema por métodos numéricos variacionales consta de los siguiente pasos:
- Plantear la formulación variacional del problema.
- Plantear el espacio funcional de aproximación numérica: Galerkin Continuo, Galerkin Discontinuo, Colocación, etc.
- Ensamblar el sistema discreto de ecuaciones, incluyendo las condiciones de contorno.
FEniCS nos reduce este proceso al primer paso!

## Formulación variacional

La formulación variacional la definimos como la integral sobre el dominio $\Omega$ del producto interno entre la ecuación de Poisson y una función vectorial de test $\boldsymbol{v}\in \hat{V}$, donde $\hat{V}$ es el espacio de la funcion de test.

\begin{align*}
-\int_\Omega (\triangle u) v \text{d} x &= \int_\Omega fv \text{d} x, && \text{en} \: \Omega.
\end{align*}

Como $\triangle u$ contiene derivadas de segundo orden sobre $u$, debilitamos la ecuación integrando por partes:

\begin{align*}
-\int_\Omega (\triangle u) v \text{d} x &= \int_\Omega \nabla u \cdot \nabla v \text{d} x - \int_{\partial \Omega}\frac{\partial u}{\partial n} v \text{d} s, && \text{en} \: \Omega.
\end{align*}
donde $\boldsymbol{n}$ es vector normal en el contorno.

$\frac{\partial u}{\partial n}$ es conocido como el flujo en el contorno $\partial \Omega$ y puede ser usado para prescribir una condicion de contorno de tipo Neumann o Robin de la forma $\frac{\partial u}{\partial n} = t$.

Suponiendo que no hay flujos en los contornos del dominio, la formulación variacional está dada por
    \begin{align*}
         \int_\Omega \nabla u \cdot \nabla v \text{d} x&= \int_\Omega fv \text{d} x, && \text{en} \: \Omega.\\
    \end{align*}

### Definición de los espacios discretos de interpolación

Definición de un espacio discreto de aproximación contenido dentro del espacio continuo. Uno para el espacio de la función de test:
    \begin{align*}
        \widetilde{V}_h\subset \widetilde{V}:=\left\{v\in H(\Omega): v=0 \:\text{sobre} \: \partial\Omega\right\},
    \end{align*}
    y otra para la función de la incógnita:
    \begin{align*}
        V_h\subset V:=\left\{v\in H(\Omega): v=u_0 \:\text{sobre} \: \partial\Omega\right\}.
    \end{align*}
Tal que, el problema discreto es encontrar $u_h\in V_h\subset V$, tal que:
    \begin{align*}
     \int_\Omega \nabla u_h \cdot \nabla v_h \text{d} x&= \int_\Omega fv_h \text{d} x, && \text{en} \: \Omega.\\
    \end{align*}
    para todo $v_h\in \widetilde{V}_h\subset \widetilde{V}$.

### Notación variacional en FEniCS
Este tipo de ecuaciones variacionales discretas se puede escribir usando la siguiente notación: Encontrar $u_h\in V_h\subset V$, tal que:
    \begin{align*}
     a(u_h,v_h)&=L(v_h),
    \end{align*}
 para todo $v_h\in \widetilde{V}_h\subset \widetilde{V}$.

 Donde $a(u,v)$ es una forma variacional bilineal (lineal en cada argumento) y $L(v)$ es una forma lineal:
    \begin{align*}
     a(u_h,v_h)&=\int_\Omega \nabla u_h \cdot \nabla v_h \text{d} x,\\
     L(v_h)&= \int_\Omega fv_h \text{d} x.\\
     \end{align*}

  ## Sistema discreto de ecuaciones lineales

Introduciendo las funciones de test y de interpolación de la incógnita como polinomios a trozos:
\begin{align*}
        v_h=&\sum_i^n \phi_i \subset \widetilde{V}_h\subset \widetilde{V}, &&
        u_h=&\sum_j^n U_j\phi_j \subset V_h\subset V.
    \end{align*}
La ecuación discreta resulta en:
\begin{align*}
    \int_\Omega \partial_k \sum_j^n U_j\phi_j \partial_k \sum_i^n \phi_i \text{d} x =& \int_\Omega f\sum_i \phi_i \text{d} x.
\end{align*}
O mejor:
\begin{align*}
    \sum_i^n\sum_j^n\int_{\Omega^e} U_j \partial_k\phi_j \partial_k \phi_i \text{d} x =& \sum_i^n \int_{\Omega^e} f \phi_i \text{d} x.
\end{align*}

Deshaciendo las sumatorias y el álgebra lineal:
\begin{align*}
        \begin{bmatrix}
a_{11} & a_{12} & a_{13} & \dots & a_{1n} \\
a_{21} & a_{22} & a_{23} & \dots & a_{2n} \\
\dots  & \dots  & \dots  & a_{ij} & \dots  \\
a_{n1} & a_{n2} & a_{n3} & \dots & a_{nn}
\end{bmatrix}
\begin{bmatrix}
U_1 \\ U_2 \\ \dots \\ U_n
\end{bmatrix}
=&
\begin{bmatrix}
L_{1} \\ L_{2} \\ \dots \\ L_{3}
\end{bmatrix},
\end{align*}
donde $U_j$ son las incógnitas nodales, $a_{ij}$ viene dada por una integración local sobre cada elemento $a_{ij} = \int_{\Omega^e} U_j \partial_k\phi_j \partial_k \phi_i \text{d} x$ y $L_i$ está dada por $L_i = \int_{\Omega^e} f \phi_i \text{d} x$.

Esto es lo mismo que un sistema lineal del tipo $\mathbb{A}\mathbb{U}=\mathbb{L}$, cuya resolución está dada por:

\begin{align*}
    \mathbb{U}=&\mathbb{A}^{-1}\mathbb{L}
\end{align*}

## Implementación en FEniCS

Modelamos un dominio bidimensional cuadrador con condiciones de Dirichlet en sus contornos y un término fuente constante.

### El programa completo

```python
from dolfin import *
import matplotlib.pyplot as plt

# Malla
mesh = UnitSquareMesh(8,8)
# Espacio de funciones para la incognita y test
V= FunctionSpace(mesh,"Lagrange",1)
# Expresion para las condiciones de contorno
u0=Expression("1+ x[0]*x[1] + 2*x[1]*x[1]",degree=2)
# Definicion de las condiciones de contorno
bc = DirichletBC ( V , u0 , "on_boundary")
# Fuente
f=Constant(-6.0)

# Funcion de trial
u= TrialFunction(V)
# Funcion de test
v= TestFunction(V)
# Forma bilineal del lado izquierdo
a = inner(grad(u), grad(v))*dx
# Forma lineal del lado derecho
L=f*v*dx
# Funcion donde se almacena el resultado
u = Function(V)
# Solucion del sistema algebraico
solve(a==L, u, bc)

#Impresion usando paraview
vtkfile = File('poisson_solution.pvd')
vtkfile << u
```

    Calling FFC just-in-time (JIT) compiler, this may take some time.
    Calling FFC just-in-time (JIT) compiler, this may take some time.

### Importar paquetes

Importamos los paquetes de solucion (dolfin), de mallado (mshr), numpy para realizar operaciones numericas y matplotlib para ver los resultados

```python
from dolfin import *
import matplotlib.pyplot as plt
```

### Dominio geometrico y mallado

Definimos el dominio $\Omega$ y generamos la malla directamente usando FEniCS:

```python
mesh = UnitSquareMesh(8,8)
```

Las posibilidades que existen de generación de dominios con la librería **mshr** son inmensas.

###  Definición de los espacios funcionales

FEniCS tienen programada internamente una gran cantidad de familias de elementos finitos, incluyendo: Lagrangeanos, Curl, Div, Discontinuos, y la posibilidad de realizar distintas combinaciones de espacios funcionales entre ellos.

Una vez creada la malla, podemos crear un espacio funcional de elementos finitos V sobre dicho dominio geométrico discreto:

```python
V= FunctionSpace(mesh,"Lagrange",1)
```

El segundo argumento especifica el tipo de elemento. El tipo de elemento aquí es "Lagrange", lo que implica la familia de elementos estándar de Lagrange. También puede utilizar 'P' para especificar este tipo de elemento. FEniCS soporta todas las familias de elementos simplex y la notación definida en la Tabla Periódica de Elementos Finitos.

El tercer argumento 1 especifica el grado del elemento finito. En este caso, el estándar 𝖯1 elemento lineal de Lagrange, que es un triángulo con nodos en los tres vértices. Algunos se refieren a este elemento como el "triángulo lineal". La solución calculada $u$ será continua en los elementos y variará linealmente en $(x,y)$ dentro de cada elemento. Las aproximaciones polinomiales de mayor grado sobre cada celda se obtienen trivialmente aumentando el tercer parámetro. Cambiar el segundo parámetro a 'DP' crea un espacio funcional para métodos de Galerkin discontinuos.

### Definición de las funciones de trial y test

En matemáticas, distinguimos entre los espacios de interpolación de la incógnita $V$ y el de prueba o test $\tilde{V}$. La única diferencia en el problema actual son las condiciones de contorno. En FEniCS no especificamos las condiciones de contorno como parte del espacio funcional, por lo que es suficiente trabajar con un espacio común $V$ para las distintas funciones:

```python
u = TrialFunction(V)
v = TestFunction(V)
```

### Definición de condiciones de contorno tipo Dirichlet

El siguiente paso es especificar la condición de contorno $u= u_D$ sobre $\partial\Omega$. Esto se hace muy fácilmente con la siguiente rutina:

```python
bc = DirichletBC(V, u0, "on_boundary")
```

donde $u_D$ es una expresión que define los valores de la solución en el contorno del dominio computacional. Dicho valor se puede determinar por una función (u objeto) definida sobre todo el dominio espacial, pero evaluada únicamente en el contorno.

La clase **DirichletBC** sirve para imponer condiciones de contorno en cada espacio funcional (incógnita).

Define que la función de la incógnita $u$ definida en el espacio $V$ debe igualarse a la expresión $u_D$ evaluada en la región "on_boundary".

Las condiciones de contorno del tipo $u = uD$ se conocen como condiciones de Dirichlet. Para el presente método de elementos finitos para el problema de Poisson, también se denominan condiciones de contorno esenciales, ya que deben imponerse explícitamente como parte del espacio funcional de la incógnita (en contraste con las que pueden definirse implícitamente como parte de la formulación variacional). Naturalmente, la clase FEniCS utilizada para definir las condiciones de contorno de Dirichlet se denomina DirichletBC. Esa clase instancia un objeto bc que se usa más adelante para ensamblar el sistema de ecuaciones lineales.

Ahora bien, la variable $u_D$ se refiere a un objeto **Expression**, que se utiliza para representar una función matemática. La construcción típica de ese tipo de objeto se da con la siguiente rutina:

```python
u0 = Expression(formula, degree=1)
```

donde **formula** es una cadena que contiene una expresión matemática. La fórmula debe escribirse con sintaxis C ++ y se convierte automáticamente en una función C ++ compilada y eficiente.

La expresión puede depender de las variables x[0] y x[1] correspondientes a las coordenadas $(x,y)$. En 3D, la expresión también puede depender de la variable x[2] correspondiente a la coordenada $z$. Este problema introductorio supone una expresión de la forma $u_D(x,y) = 1 + x2 + 2y^2$. Entonces, la expresión se puede escribir como 1 + x [0] * x [0] + 2 * x [1] * x [1]:

```python
u0=Expression("1+ x[0]*x[1] + 2*x[1]*x[1]", degree=2)
```

Establecemos el grado **degree** de la fórmula como 2 para que $u_D$ pueda representar la solución cuadrática exacta de nuestro problema de prueba.

La clase **Expression** es muy flexible y sirve para crear expresiones complicadas definidas por el usuario. Funciones espaciales, temporales u otras.

Por ejemplo, la fuente $f=-6$ puede ser definida como:

```python
f = Expression("-6",degree=0)
```

O más eficientemente con la clase Función Constante (en espacio y tiempo):

```python
f = Constant(-6.0)
```

Lo único que nos falta es definir la región sobre la cual se evalúa la expresión para $u_D$. Esto se realiza con el argumento "on_boundary". Otras opciones para determinar fácilmente la región de imposición de las condiciones de Dirichlet son:

- "on\_boundary"    Todo el contorno del dominio
- "near(x[0],0.0)"      En la región del contorno x=0.
- "near(x[0],0.0) || near(x[1],1.0)"    En la región del contorno x=0 o y=1.

### Definición del problema variacional

Ahora tenemos todos los ingredientes que necesitamos para definir el problema variacional:

```python
a = inner(grad(u), grad(v))*dx
L = f*v*dx
```

En esencia, estas dos líneas especifican el problema EDP a resolver. Se entiende la estrecha correspondencia entre la sintaxis de Python y las fórmulas matemáticas $\nabla u \cdot \nabla v dx$ y $fvdx$. Esta es una fortaleza clave de FEniCS: las fórmulas en la formulación variacional se traducen directamente a un código Python muy similar, una característica que facilita la formulación y resolución de problemas EDP complicados.

### Ensamblaje y solución del problema variacional

FEniCS realiza el ensamblaje del sistema lineal (impone las condiciones de contorno) y llama a uno de los solucionadores externos (según el problema).

```python
# Compute solution
u = Function(V)
# Solucion del sistema algebraico
solve(a==L, u, bc)
```

Se usa la función $u$ tanto como función trial como para guardar la solución que entrega el solucionador.

## Post-procesamiento

### Visualización de la solución

Posproceso, visualización y salida de datos puede realizarse:
- Interno: Numpy, Matplotlib, etc.
- Externo: .pvd, .xml, .xdmf (reestablecer casos).

Por ejemplo, para escribir un archivo que pueda ser visualizado en Paraview, se escriben las dos líneas siguientes:

```python
vtkfile = File("poisson_solution.pvd")
vtkfile << u
```

Una para crear el archivo que guarda los datos.
La otra línea para salvar la información dentro del archivo creado.


# Reconstructing an *off-axis* hologram by Convolution approach

Reference: Digital holography and wavefront sensing by Ulf Schnars, Claas Falldorf, John Watson, and Werner Jüptner, Springer-verlag Berlin an, 2016. (Section 3.2)

## Info about the digital hologram:
'ulf7.BMP' is a digital hologram created by recording an object at about 1 meter distance with HeNe laser (632.8 nm) and an image sensor with 6.8 µm pixel size.

```python
#Import libraries realted to matplotlib and mathematical operations
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
```

```python
# Read the hologram image file
hologram = mpimg.imread('ulf7.BMP')
hologram = hologram.astype(float) #Convert into float type. Crucial for non integer based mathematical operations
# plot/view the hologram
imgplot = plt.imshow(hologram, cmap="viridis")
```

```python
# DC term suppression. It involves subtraction of the mean value of hologram from itself
hologram2 = hologram - np.mean(hologram)
# plot/view the hologram
imgplot = plt.imshow(hologram2, cmap="viridis")
```

## Some equations from the book!
The *Fresnel-Kirchhoff* integral describing diffraction field beyond an aperture is given by the coherent superposition of the secondary waves (section 2.4)
\begin{equation}
\Gamma\left(\xi^{\prime}, \eta^{\prime}\right)=\frac{i}{\lambda} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} A(x, y) \frac{\exp \left(-i \frac{2 \pi}{\lambda} \rho^{\prime}\right)}{\rho^{\prime}} Q d x d y
\end{equation}
where, $A(x, y)$ is the complex amplitude in the plane of the diffracting aperture, $\rho^{\prime}$ is the distance between a point in the aperture plane and a point in the observation plane, and $Q$ is the inclination factor defined to take care of no backward propagation of the diffracted optical field. For holograms, $Q$ is approximately equal to 1.

A hologram $h(x,y)$ recorded by a reference light wave $E_{R}(x, y)$ can be reconstructed by a conjugate reference wave $E_{R}^{*}(x, y)$ as described by the following *Fresnel-Kirchhoff* integral
\begin{equation}
\Gamma(\xi, \eta)=\frac{i}{\lambda} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} h(x, y) E_{R}^{*}(x, y) \frac{\exp \left(-i \frac{2 \pi}{\lambda} \rho\right)}{\rho} d x d y
\end{equation}

The above equation can be interpreted as a superposition integral
\begin{equation}
\Gamma(\xi, \eta)= \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} h(x, y) E_{R}^{*}(x, y) g(\xi, \eta, x, y) d x d y
\end{equation}
where $g(\xi, \eta, x, y)$ is the impulse response function given by
\begin{equation}
g(\xi, \eta, x, y)=\frac{i}{\lambda} \frac{\exp \left[-i \frac{2 \pi}{\lambda} \sqrt{d^{2}+(x-\xi)^{2}+(y-\eta)^{2}}\right]}{\sqrt{d^{2}+(x-\xi)^{2}+(y-\eta)^{2}}}.
\end{equation}
These equations ensure that $\Gamma$ is a convolution of $h E_{R}^{*}$ and $g$. Hence,
\begin{equation}
\Gamma(\xi, \eta)=\Im^{-1}\left\{\Im\left(h \cdot E_{R}^{*}\right) \cdot \Im(g)\right\}
\end{equation}
where for the digital case impulse response function is given by
\begin{equation}
g(k, l)=\frac{i}{\lambda} \frac{\exp \left[-i \frac{2 \pi}{\lambda} \sqrt{d^{2}+k^{2} \Delta x^{2}+l^{2} \Delta y^{2}}\right]}{\sqrt{d^{2}+k^{2} \Delta x^{2}+l^{2} \Delta y^{2}}}
\end{equation}
and $E_{R}^{*}$ includes the field response due to the numerical reconstruction lens whose focal length depends on the object distance $d$ and magnification factor in reconstruction $m$.
\begin{equation}
E_{R}^{*} = L(k,l) = \exp \left[i \frac{\pi}{\lambda f}\left(k^{2} \Delta x^{2} + l^{2} \Delta y^{2} \right)\right] =\exp \left[i \frac{\pi}{\lambda}\left(\frac{1}{d}+\frac{1}{md}\right)\left(k^{2} \Delta x^{2} + l^{2} \Delta y^{2} \right)\right]
\end{equation}

```python
# prepare the Convolution operand for the hologram
Nr,Nc = np.shape(hologram) #number of rows and columns in the hologram
wavelength = 632.8e-9 #HeNe laser wavelength in SI units i.e. meters
dx = 6.8e-6 #sensor pixel size in meters
d = 1.054 #object distance in meters
m = 1/14 #magnification factor
d2 = d*m #reconstruction distance in meters
f = 1/(1/d+1/d2) #lens responsible for imaging an object kept at d distance to an image at d2 distance
Nr = np.linspace(0, Nr-1, Nr)-Nr/2
Nc = np.linspace(0, Nc-1, Nc)-Nc/2
k, l = np.meshgrid(Nc,Nr)

L = np.exp(1j*np.pi/(f*wavelength)*(np.multiply(k, k)*dx**2 + np.multiply(l, l)*dx**2))
rho = np.sqrt(d2**2 + np.multiply(k, k)*dx**2 + np.multiply(l, l)*dx**2)
g = 1j/wavelength*np.exp(-1j*2*np.pi/wavelength*rho)/(rho)
factor = np.multiply(hologram, L)
factor2 = np.multiply(hologram2, L) #DC term suppressed
reconstructed_field = np.fft.fftshift( np.fft.ifft2(np.multiply(np.fft.fft2(factor), np.fft.fft2(g))) )
reconstructed_field2 = np.fft.fftshift( np.fft.ifft2(np.multiply(np.fft.fft2(factor2), np.fft.fft2(g))) ) # DC term suppressed
```

```python
# save and plot
I = np.abs(reconstructed_field)/np.max(np.abs(reconstructed_field)) #normalized intensity profile
plt.imshow(I, cmap="hot", clim=(0.0, 0.4))
plt.colorbar()
mpimg.imsave('Convolution_reconstruction.png', I, cmap="hot", vmin=0.0, vmax=0.3) #save reconstruction matrix as image
```

```python
# save and plot DC suppressed hologram
I2 = np.abs(reconstructed_field2)/np.max(np.abs(reconstructed_field2)) #normalized intensity profile
plt.imshow(I2, cmap="hot", clim=(0.0, 0.4))
plt.colorbar()
mpimg.imsave('Convolution_reconstruction_DCsuppressed.png', I2, cmap="hot", vmin=0.0, vmax=0.6) #save reconstruction matrix as image
```