# Black-Scholes Option Pricing Model

Black–Scholes–Merton model is a mathematical model for the dynamics of a financial market containing derivative investment instruments. From the partial differential equation in the model, known as the Black–Scholes equation, one can deduce the Black–Scholes formula, which gives a theoretical estimate of the price of European-style options and shows that the option has a unique price regardless of the risk of the security and its expected return (instead replacing the security's expected return with the risk-neutral rate).

The key idea behind the model is to hedge the option by buying and selling the underlying asset in just the right way and, as a consequence, to eliminate risk. This type of hedging is called "continuously revised delta hedging" and is the basis of more complicated hedging strategies such as those engaged in by investment banks and hedge funds.

The Black–Scholes model assumes that the market consists of at least one risky asset, usually called the stock, and one riskless asset, usually called the money market, cash, or bond.

Now we make assumptions on the assets (which explain their names):

- (riskless rate) The rate of return on the riskless asset is constant and thus called the risk-free interest rate.
- (random walk) The instantaneous log return of stock price is an infinitesimal random walk with drift; more precisely, it is a geometric Brownian motion, and we will assume its drift and volatility are constant (if they are time-varying, we can deduce a suitably modified Black–Scholes formula quite simply, as long as the volatility is not random).
- The stock does not pay a dividend.

Assumptions on the market:

- There is no arbitrage opportunity (i.e., there is no way to make a riskless profit).
- It is possible to borrow and lend any amount, even fractional, of cash at the riskless rate.
- It is possible to buy and sell any amount, even fractional, of the stock (this includes short selling).
- The above transactions do not incur any fees or costs (i.e., frictionless market).

The Black-Scholes formula:

\begin{equation}
\frac{\partial V}{ \partial t} + \frac{1}{2}\sigma^{2} S^{2} \frac{\partial^{2} V}{\partial S^2} + r S \frac{\partial V}{\partial S} - r V = 0
\end{equation}

"The Greeks" measure the sensitivity of the value of a derivative or a portfolio to changes in parameter value(s) while holding the other parameters fixed. They are partial derivatives of the price with respect to the parameter values. One Greek, "gamma" is a partial derivative of another Greek, "delta" in this case.

The Greeks are important not only in the mathematical theory of finance, but also for those actively trading. Financial institutions will typically set (risk) limit values for each of the Greeks that their traders must not exceed. Delta is the most important Greek since this usually confers the largest risk. Many traders will zero their delta at the end of the day if they are speculating and following a delta-neutral hedging approach as defined by Black–Scholes.
Typically the following Greeks are of interest: Delta, Vega and Gamma.

Black-Scholes Call and Put Option Price Formulas:

$C=S_{0}*N(d_{1})-Ke^{-rt}*N(d_{2})$

$P=Ke^{-rt}*N(-d_{2})-S_{0}*N(-d_{1})$

where $N(x)$ is the standard normal cumulative distribution function

$d_{1}=\frac{\ln{\frac{S_{o}}{K}}+t(r+\frac{\sigma^{2}}{2})}{\sigma\sqrt{t}}$

$d_{2}=d_{1}-\sigma\sqrt{t}$

```python
import math
from scipy import stats
```

```python
# Option parameters

K = 110 # strike price
r = .04 # interest rate
sigma = .3 # volatility - standard deviation of the stock
T = 1 # time maturity
S0 = 100 # starting price of the stock
```

```python
def get_black_scholes_simulation_results(K, r, sigma, T, S0, is_call=True):
    """
    Calculates option price using Black-Scholes model,
    returns the option value and the dict of calculated greeks in the form of a list
    """
    d1 = (math.log(S0 / K) +
          (r + (0.5 * sigma**2)) * T) / (sigma * math.sqrt(T))
    d2 = d1 - (sigma * math.sqrt(T))

    if is_call:
        call_value = S0 * stats.norm.cdf(d1, 0, 1) - K * math.exp(
            -r * T) * stats.norm.cdf(d2, 0, 1)
        delta_call = stats.norm.cdf(d1, 0, 1)
        gamma_call = stats.norm.pdf(d1, 0, 1) / (S0 * sigma * math.sqrt(T))
        theta_call = -(r * K * math.exp(-r * T) * stats.norm.cdf(d2, 0, 1)) - (
            sigma * S0 * stats.norm.pdf(d1, 0, 1) / (2 * math.sqrt(T)))
        rho_call = T * K * math.exp(-r * T) * stats.norm.cdf(d2, 0, 1)
        vega_call = math.sqrt(T) * S0 * stats.norm.pdf(d1, 0, 1)

        return [
            call_value, {
                'delta': delta_call,
                'gamma': gamma_call,
                'theta': theta_call,
                'rho': rho_call,
                'vega': vega_call
            }
        ]

    else:
        put_value = K * math.exp(-r * T) * stats.norm.cdf(-d2, 0, 1) - (
            S0 * stats.norm.cdf(-d1, 0, 1))
        delta_put = -stats.norm.cdf(-d1, 0, 1)
        gamma_put = stats.norm.pdf(d1, 0, 1) / (S0 * sigma * math.sqrt(T))
        theta_put = (r * K * math.exp(-r * T) * stats.norm.cdf(-d2, 0, 1)) - (
            sigma * S0 * stats.norm.pdf(d1, 0, 1) / (2 * math.sqrt(T)))
        rho_put = -T * K * math.exp(-r * T) * stats.norm.cdf(-d2, 0, 1)
        vega_put = math.sqrt(T) * S0 * stats.norm.pdf(d1, 0, 1)

        return [
            put_value, {
                'delta': delta_put,
                'gamma': gamma_put,
                'theta': theta_put,
                'rho': rho_put,
                'vega': vega_put
            }
        ]
```

```python
bs_call_results = get_black_scholes_simulation_results(K, r, sigma, T, S0)
bs_put_results = get_black_scholes_simulation_results(K, r, sigma, T, S0, is_call=False)

print('\nVanila European Call:\n' + '\nValue: ' + str(bs_call_results[0]) + '\nDelta: ' +
      str(bs_call_results[1]['delta']))
print('\nVanila European Put:\n' + '\nValue: ' + str(bs_put_results[0]) + '\nDelta: ' +
      str(bs_put_results[1]['delta']))
```

    Vanila European Call:

    Value: 9.625357828843697
    Delta: 0.48629214299030143

    Vanila European Put:

    Value: 15.312196135599244
    Delta: -0.5137078570096986

Black-Scholes Option Pricing Model.ipynb


# Distributed clustering

We have many types of distributed clustering, where most are an modification of k-means. In this section we show three types: hard k-means (hcm), fuzzy k-means (fcm) and possibilistic k-means (pcm).

### Libraries

We need four libraries. Numpy is used for the matrices calculation. The math library is used to calcualte the square root when we calculate the Euclidean distance. Matplotlib is used for the plots. Finally, pandas is used here only for displaying the assignation matrix in a easy to ready form in Jupyter.

```python
import numpy as np
from math import sqrt
import matplotlib.pyplot as plt
import pandas as pd
```

## K-means

The most known method is called k-means and assign each case to one cluster strictly. It is also known as hard c-means where k is the same as c and are the number of clusters that we are willing to divide the data set to. The steps of hcm are like following:
1. choose the entrance cluster centroids,
2. item calculate the assignation matrix $U$,
3. item calculate new centroids matrix $V$,
4. calculate the difference between previously assignation matrix $U$ and the new one calculated in current iteration.

```python
%store -r data_set
```

Before we start, we should setup a few variables like the assignation matrix, number of clusters, the error margin and feature space:

```python
groups = 2
error_margin = 0.01
m=2
assignation=np.zeros((len(data_set),groups))
```

The error margin is a value of error that below ends the clustering loop.

The assignation matrix if filled with zeros as we don't have any guess for assignation yet. We can also fill it randomly with 1 and 0 for each group. The assignation matrix looks like following:

\begin{equation*}
U=\begin{bmatrix}
0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\
\end{bmatrix}.
\end{equation*}

It's time to generate centroid array randomly:
\begin{equation}
 V=[v_{1},v_{2},\ldots,v_{c}].
\end{equation}

We go through each group and add a random array of the feature space centroid positions:

```python
def select_centers():
    return np.random.rand(groups,len(data_set[0]))

centers = select_centers()
```

Let's take a look what centroids do we have:

```python
pd.DataFrame(centers, columns=['x1','x2'])
```

      x1
      x2

      0
      0.149994
      0.092524

      1
      0.607688
      0.589990

We can also set fixed centers. It is important that the values are normalized.

```python
#centers = [[0.2,0.2],  [0.8,0.8]]
#pd.DataFrame(centers)
#print(centers)
```

To check what is the distance between the centroids and the elements of data set we use the Euclidean distance:

\begin{equation}
 \rho_{Min}(x_{i},v_{j})=\sqrt{\sum_{i=1}^{d}(x_{i}-v_{j})^{2}}.
\end{equation}

```python
def calculate_distance(x,v):
    return sqrt((x[0]-v[0])**2+(x[1]-v[1])**2)
```

The next step is to calculate the new assignation matrix:

\begin{equation}
 \mu_{ik}^{(t)}=
 \begin{cases}
 1 & \text{if } d(x_{k},v_{i})<d(x_{k},v_{j}),  \text{for each } j\neq i\\
 0 & \text{in other case} \\
 \end{cases}.
\end{equation}

The code below relizes the equation above for two groups.

```python
def calculate_u(x, centers):
    if calculate_distance(x, centers[0]) < calculate_distance(x, centers[1]):
        return [1,0]
    else:
        return [0,1]
```

The third step is to calculate new centroids based on the new assignation matrix $U$:

\begin{equation}
 v_{i}=\frac{\sum_{k=1}^{M}\mu_{ik}^{(t)}x_{k}}{\sum_{k=1}^{M}\mu_{ik}^{(t)}}.
\end{equation}

The calculation is done in two steps: ```u_x_vector``` and ```u_scalar```:

```python
def calculate_new_centers(u):
    new_centers=[]
    for c in range(groups):
        u_x_vector=np.zeros(2)
        u_scalar=0.0
        for i in range(len(data_set)):
            u_scalar = u_scalar+(u[i][c]**m)
            u_x_vector=np.add(u_x_vector,np.multiply(u[i][c]**m,data_set[i]))
        new_centers.append(np.divide(u_x_vector,u_scalar))
    return new_centers
```

We are almost done here. The last step before we cluster is to set the rule that allow us to stop the loop.

```python
def calculate_differences(new_assignation, assignation):
    return np.sum(np.abs(np.subtract(assignation,new_assignation)))
```

It's time to combine all together:

```python
def cluster_hcm(assignation,centers):
    difference_limit_not_achieved=True
    new_centers = centers
    iter=0
    while difference_limit_not_achieved:
        new_assignation=[]
        for i in range(len(data_set)):
            new_assignation.append(calculate_u(data_set[i], new_centers))
        new_centers = calculate_new_centers(new_assignation)
        if iter>0:
            if calculate_differences(new_assignation, assignation) < error_margin:
                difference_limit_not_achieved=False
        assignation=new_assignation
        iter=iter+1
    return new_assignation, new_centers
```

Ready to build some new clusters:

```python
new_assignation_hcm, new_centers_hcm = cluster_hcm(assignation, centers)
%store new_assignation_hcm
%store new_centers_hcm
```

    Stored 'new_assignation_hcm' (list)
    Stored 'new_centers_hcm' (list)

The centers are like following:

```python
pd.DataFrame(new_centers_hcm, columns=['x1','x2'])
```

      x1
      x2

      0
      0.127701
      0.207853

      1
      0.829077
      0.970594

And the assignation matrix looks like:

```python
pd.DataFrame(new_assignation_hcm, columns = ['Cluster 1','Cluster 2'])
```

      Cluster 1
      Cluster 2

      0
      1
      0

      1
      1
      0

      2
      1
      0

      3
      1
      0

      4
      1
      0

      5
      1
      0

      6
      0
      1

      7
      0
      1

      8
      0
      1

      9
      1
      0

To plot it, we need to develop a short function that adds some colors to our plot:

```python
red = data_set[np.where(np.array(new_assignation_hcm)[:,0]==1)]
blue = data_set[np.where(np.array(new_assignation_hcm)[:,1]==1)]
```

And finally plot the results:

```python
fig, ax = plt.subplots()

ax.scatter(blue[:,0],blue[:,1],c='blue')
ax.scatter(red[:,0],red[:,1],c='red')
ax.scatter(np.array(new_centers_hcm)[:,0],np.array(new_centers_hcm)[:,1],c='black')
ax.set(xlabel='Seats count', ylabel='Distance range (km)',
       title='Aircrafts (clusters)')
ax.grid()
plt.show()
```

## Fuzzy k-means

We reset the assignation matrix and set the m parameter. The m paramtere is also known as fuzzifier. The higher value it is the values are more fuzzy. A lower value gives as results that are closer to the one that we got with the hard version of k-means.

```python
assignation=np.zeros((len(data_set),groups))

m = 2.0
```

The fuzzy implementation of k-means is a bit more complex and we need to modify the calculate_u function to be complient with the equation:

\begin{equation}
 \mu_{ik}=(\sum_{j=1}^{c}(\frac{d(x_{k},v_{i})}{d(x_{k},v_{j})})^{\frac{2}{m-1}})^{-1}
\end{equation}

The implementation is given as below.

```python
def calculate_u_fcm(x, centers, group_id):
    distance_centers = 0
    for group in range(groups):
        if group != group_id:
            distance_centers+= calculate_distance(x, centers[group])
    distance_sum=1.0+(calculate_distance(x, centers[group_id])/distance_centers)**m
    return distance_sum**-1
```

That's the only difference between HCM and FCM. The rest is almost the same in both cases.

```python
def cluster_fcm(assignation, centers):
    difference_limit_not_achieved=True
    new_centers = centers
    iter=0
    while difference_limit_not_achieved:
        new_assignation=[]
        for i in range(len(data_set)):
            new_assignation_vector=[]
            for k in range(groups):
                new_assignation_vector.append(calculate_u_fcm(data_set[i],new_centers,k))
            new_assignation.append(new_assignation_vector)
        new_centers = calculate_new_centers(new_assignation)

        if iter>0:
            if calculate_differences(new_assignation, assignation) < error_margin:
                difference_limit_not_achieved=False
        assignation=new_assignation
        iter=iter+1
    return new_assignation, new_centers
```

Calculation of the clusters is done the same way as in the previous example:

```python
new_assignation_fcm, new_centers_fcm = cluster_fcm(assignation, centers)
```

The cluster centers are similar to the previous example:

```python
pd.DataFrame(new_centers_hcm, columns=['x1','x2'])
```

      x1
      x2

      0
      0.127701
      0.207853

      1
      0.829077
      0.970594

The assignation matrix is different even we assign same objects to the same clusters. Values in each row sums to 1.

```python
pd.DataFrame(new_assignation_fcm, columns = ['Cluster 1','Cluster 2'])
```

      Cluster 1
      Cluster 2

      0
      0.988155
      0.011845

      1
      0.992877
      0.007123

      2
      0.983858
      0.016142

      3
      0.990163
      0.009837

      4
      0.992096
      0.007904

      5
      0.998543
      0.001457

      6
      0.006109
      0.993891

      7
      0.019840
      0.980160

      8
      0.076920
      0.923080

      9
      0.792276
      0.207724

To plot the objects in a fuzzy k-means we need to group them by values higher than 0.5 as both values sums to 1.

```python
red = data_set[np.where(np.array(new_assignation_fcm)[:,0]>0.5)]
blue = data_set[np.where(np.array(new_assignation_fcm)[:,1]>0.5)]

fig, ax = plt.subplots()

ax.scatter(blue[:,0],blue[:,1],c='blue')
ax.scatter(red[:,0],red[:,1],c='red')
ax.scatter(np.array(new_centers_fcm)[:,0],np.array(new_centers_fcm)[:,1],c='black')
ax.set(xlabel='Seats count', ylabel='Distance range (km)',
       title='Aircrafts (clusters)')
ax.grid()
plt.show()
```

### Possibilistic k-means (PCM)

In the fuzzy version, each row sums to 1. In real-world cases, it doesn't need to be like this. The possibilistic k-means returns the distance to the center rather than dividing the assignation between clusters.

As suggested by the authors, the initial assignation matrix should be created using the FCM method. We do a fixed number of FCM method loops. The number of loops is set by the variable ``F``. The ``error_margin`` variable is the error threshold were below of it we stop the loop.

```python
F = 2
error_margin = 0.08
assignation=np.zeros((len(data_set),groups))
```

The assignation function is more complex compared to the two previous one. In PCM we use the Mahalanobis distance instead of the Euclidean one, and the assignation function is set as:
\begin{equation}
 \mu_{ik}=(1+(\frac{\rho_{A}(x_{i},v_{j})}{\eta_{i}})^{\frac{2}{m-1}})^{-1},
\end{equation}
where
\begin{equation}
\eta_{i}=\frac{\sum_{k=1}^{M}(\mu_{ik})^{m}\rho_{A}(x_{i},v_{j})}{\sum_{k=1}^{M}(\mu_{ik}
)^{m}}.
\end{equation}
$\rho_{A}(x_{i},v_{j})$ is the Mahalanobis distance:
\begin{equation}
\rho_{A}(x_{i},v_{j})=(x_{i}-v_{j})^{T}A(x_{i}-v_{j}).
\end{equation}
It use ``A`` diagnoal matrix to measure the distance. The figure below show how the euclidean distance is measured:

The difference between two distances is that in Mahalanobis distance we use the diagonal matrix ``A``, which is also known as Mahalanobis norm, that allow us to measure the distance between objects as it's shown in figure below.

The Mahalanobis norm can be implemented as below.

```python
def calculate_A():
    mean=np.mean(data_set,axis=0)
    sumof = np.zeros((data_set[0].shape))
    for i in range(len(data_set)):
        subtracted = np.subtract(data_set[i],mean)
        sumof = sumof + np.multiply(subtracted, subtracted)
    variance = np.divide(sumof,len(data_set))
    ABcov = np.cov(data_set[:,0]*data_set[:,1])
    R = np.array([[variance[0], ABcov], [ABcov, variance[1]]])
    return R**-1
```

The matrix can be saved as global variable ``A``. It is the size of the feature number by feature number. In our case it will be a matrix of size $2\times2$.

```python
A = calculate_A()
print(A)
```

    [[7.89464944 6.69665317]
     [6.69665317 7.75894855]]

After getting the ``A`` matrix, we are able to calcualte the Mahalanobis distance. The ``A`` matrix is calculated once, because it depends on the whole data set, not the method steps.

```python
def calculate_mah_distance(group, centers):
    dmc = data_set - centers[group]
    dmca = np.dot(data_set - centers[group], A)

    distances = lambda dmc, dmca: [np.dot(dmca[i], dmc[i]) for i in range(dmc.shape[0])]
    return distances(dmc,dmca)
```

The $\eta$ can be implemented as below:

```python
def calculate_eta(assignation, group, mah_distances):
    ud = np.sum((assignation[:, group] ** m) * mah_distances, axis=0)
    uq = np.sum(assignation[:, group] ** m, axis=0)
    return ud/uq
```

Finally, we can calculate the $\nu$:

```python
def calculate_u_pcm(assignation, centers):
    new_assignation = np.zeros((len(data_set), groups))
    for group in range(groups):
        mah_distances = calculate_mah_distance(group, centers)
        group_eta = calculate_eta(assignation, group, mah_distances)
        new_assignation[:,group] = (1.0+(mah_distances/group_eta))**-1
    return new_assignation
```

A stop function in PCM is defined as the difference between old and newly calculated centers.

```python
def get_centers_difference(old_centers, new_centers):
    return np.sum(np.abs(np.subtract(old_centers,new_centers)))
```

The ``cluster_pcm`` function has two parts. The first one is a FCM method that returns the input assignation matrix for the PCM method.

```python
def cluster_pcm(assignation, centers):
    new_centers = centers
    new_assignation = assignation
    for f in range(F):
        assignation = []
        for i in range(len(data_set)):
            assignation_vector = []
            for k in range(groups):
                assignation_vector.append(calculate_u_fcm(data_set[i], new_centers, k))
            assignation.append(assignation_vector)
        new_centers = calculate_new_centers(assignation)
        new_assignation = np.array(assignation)

    difference_limit_not_achieved = True
    while difference_limit_not_achieved:
        new_assignation = calculate_u_pcm(new_assignation, new_centers)
        old_centers = new_centers
        new_centers = calculate_new_centers(new_assignation)

        if get_centers_difference(old_centers, new_centers) < error_margin:
            difference_limit_not_achieved = False
    return new_assignation, new_centers
```

Now, we can cluster the data set with PCM:

```python
new_assignation_pcm, new_centers_pcm = cluster_pcm(assignation, centers)
```

The assignation values does not sum to 1 as in fuzzy k-means. The matrix give a better understanding of where the object is placed in the feature space.

```python
pd.DataFrame(new_assignation_pcm, columns = ['Cluster 1','Cluster 2'])
```

      Cluster 1
      Cluster 2

      0
      0.505268
      0.022455

      1
      0.755340
      0.024592

      2
      0.910555
      0.028171

      3
      0.878186
      0.026017

      4
      0.951871
      0.027998

      5
      0.958231
      0.029036

      6
      0.011891
      0.867600

      7
      0.010423
      0.594965

      8
      0.017823
      0.565899

      9
      0.101788
      0.062391

In case of PCM we decided to extend the plot for many groups, up to 6. The colors are defined below.

```python
assigned_groups = []
colors = ['red','blue','green','orange','black','yellow']

for el in range(len(data_set)):
    group_id = np.argmax(new_assignation_pcm[el])
    assigned_groups.append(group_id)
```

We need a function that assign a color to each cluster.

```python
def get_colours(color_id):
    return data_set[np.where(np.array(assigned_groups)[:]==color_id)]
```

Finally, we go through groups we have and assign objects to colors and plot it. What is important to mention is that some assignation values for an object can be very low, means that this object is far from all centers. We can implement here a threshold where if all assignation values are below some threshold we treat such objects as noise. In the figure below, we see the last object that is closer to the red centroid, but was assigned to the blue cluster. In this case both values are very low, but the blue one is just a bit higher. In a hard k-means method it wouldn't be so easy to find the noise.

```python
fig, ax = plt.subplots()

for group in range(groups):
    small_set = get_colours(group)
    ax.scatter(small_set[:,0],small_set[:,1],c=colors.pop(0))
ax.scatter(np.array(new_centers_pcm)[:,0],np.array(new_centers_pcm)[:,1],marker='x',c='black')
ax.set(xlabel='Seats count', ylabel='Distance range (km)',
       title='Aircrafts (clusters)')
ax.grid()
plt.show()
```

```python

```


# Fitting Logistic Regression Models$
\newcommand{\cond}{{\mkern+2mu} \vert {\mkern+2mu}}
\newcommand{\SetDiff}{\mathrel{\backslash}}
\DeclareMathOperator{\BetaFunc}{Β}
\DeclareMathOperator{\GammaFunc}{Γ}
\DeclareMathOperator{\prob}{p}
\DeclareMathOperator{\cost}{J}
\DeclareMathOperator{\score}{V}
\DeclareMathOperator{\dcategorical}{Categorical}
\DeclareMathOperator{\dcategorical}{Categorical}
\DeclareMathOperator{\ddirichlet}{Dirichlet}
$

"The logistic regression model arises from the desire to model the posterior probabilities of the $K$ classes via linear functions in $x$, while at the same time ensuring that they sum to one and remain in $[0, 1]$.", Hastie et al., 2009 (p. 119).

$$
\begin{align}
\log \frac{\prob(Y = k \cond X = x)}{\prob(Y = K \cond X = x)} = \beta_k^{\text{T}} x && \text{for } k = 1, \dotsc, K-1.
\end{align}
$$

The probability for the case $Y = K$ is held out, as the probabilities must sum to one, so there are only $K-1$ free variables.
Thus if there are two categories, there is just a single linear function.

This gives us that
$$
\begin{align}
\prob(Y = k \cond X = x) &= \frac{\exp(\beta_k^{\text{T}}x)}{1 + \sum_{i=1}^{K-1} \exp(\beta_i^{\text{T}}x)} & \text{for } k = 1, \dotsc, K-1 \\[3pt]
\prob(Y = K \cond X = x) &= \frac{1}{1 + \sum_{i=1}^{K-1} \exp(\beta_i^{\text{T}}x)}.
\end{align}
$$

Note that if we fix $\beta_K = 0$, we have the form
$$
\begin{align}
\prob(Y = k \cond X = x) &= \frac{\exp(\beta_k^{\text{T}}x)}{\sum_{i=1}^K \exp(\beta_i^{\text{T}}x)} & \text{for } k = 1, \dotsc, K.
\end{align}
$$

Then, writing $\beta = \{\beta_1^{\text{T}}, \dotsc, \beta_{K}^{\text{T}}\}$, we have that $\prob(Y = k \cond X = x) = \prob_k(x; \beta)$.

The log likelihood for $N$ observations is
$$
\ell(\beta) = \sum_{n=1}^N \log \prob_{y_n}(x_n; \beta).
$$

We can use the cost function
$$
\begin{align}
\cost(\beta)
    &= -\frac{1}{N} \left[ \sum_{n=1}^N \sum_{k=1}^K 1[y_n = k] \log \prob_k(x_n; \beta) \right] \\
    &= -\frac{1}{N} \left[ \sum_{n=1}^N \sum_{k=1}^K 1[y_n = k] \log \frac{\exp(\beta_k^{\text{T}}x_n)}{\sum_{i=1}^K \exp(\beta_i^{\text{T}}x_n)} \right] \\
    &= -\frac{1}{N} \left[ \sum_{n=1}^N \sum_{k=1}^K 1[y_n = k] \left( \beta_k^{\text{T}}x_n - \log \sum_{i=1}^K \exp(\beta_i^{\text{T}}x_n) \right) \right].
\end{align}
$$

Thus the score function is
$$
\score_k(\beta) = \nabla_{\beta_k} \cost(\beta) = -\frac{1}{N} \left[ \sum_{n=1}^N x_n \big( 1[y_n = k] - \prob_k(x_n; \beta) \big) \right].
$$

GLM/Fitting Logistic Regression Models.ipynb


## TNK

Tanaka suggested the following two-variable problem:

**Definition**

\begin{equation}
\newcommand{\boldx}{\mathbf{x}}
\begin{array}
\mbox{Minimize} & f_1(\boldx) = x_1, \\
\mbox{Minimize} & f_2(\boldx) = x_2, \\
\mbox{subject to} & C_1(\boldx) \equiv x_1^2 + x_2^2 - 1 -
0.1\cos \left(16\arctan \frac{x_1}{x_2}\right) \geq 0, \\
& C_2(\boldx) \equiv (x_1-0.5)^2 + (x_2-0.5)^2 \leq 0.5,\\
& 0 \leq x_1 \leq \pi, \\
& 0 \leq x_2 \leq \pi.
\end{array}
\end{equation}

**Optimum**

Since $f_1=x_1$ and $f_2=x_2$, the feasible objective space is also
the same as the feasible decision variable space. The unconstrained
decision variable space consists of all solutions in the square
$0\leq (x_1,x_2)\leq \pi$. Thus, the only unconstrained Pareto-optimal
solution is $x_1^{\ast}=x_2^{\ast}=0$.
However, the inclusion of the first constraint makes this solution
infeasible. The constrained Pareto-optimal solutions lie on the boundary
of the first constraint. Since the constraint function is periodic and
the second constraint function must also be satisfied,
not all solutions on the boundary of the first constraint are Pareto-optimal. The
Pareto-optimal set is disconnected.
Since the Pareto-optimal
solutions lie on a nonlinear constraint surface, an optimization
algorithm may have difficulty in finding a good spread of solutions across
all of the discontinuous Pareto-optimal sets.

**Plot**

```python
from pymoo.factory import get_problem
from pymoo.util.plotting import plot

problem = get_problem("tnk")
plot(problem.pareto_front(), no_fill=True)
```


# Polynomial Regression from scratch
> Polynomial regression from scratch with Pytorch using normal equation and gradient descent.
- author: "Axel Mendoza"
- categories: [polynomial-regression, pytorch, gradient-descent, from-scratch]
- toc: false
- comments: true
- badges: true
- image: images/polynomial.jpg

During this experiment, we will see how to implement polynomial regression from scratch using Pytorch.

```python
import torch
import numpy as np
import matplotlib.pyplot as plt
```

Generate a polynomial distribution with random noise

```python

# Function for creating a vector with value between [r1, r2]
def randvec(r1, r2, shape):
    return (r1 - r2) * torch.rand(shape) + r2

# Defining the range of our distribution
X = torch.tensor([i for i in range(-30, 30)]).float()
# Creating random points from a gaussian with random noise
y = randvec(-1e4, 1e4, X.shape) - (1/2) * X + 3 * X.pow(2) - (6/4) * X.pow(3)

plt.scatter(X, y)
```

### Create the polynomial features
The formula of linear regression is as follow:
$$
    \boldsymbol{\hat{y}} = \boldsymbol{X}\boldsymbol{w}
$$
where $\boldsymbol{\hat{y}}$ is the target, $\boldsymbol{w}$ are the weights learned by the model and $\boldsymbol{X}$ is training data.
Polynomial regression is still considered as a linear regression because there is only linear learning parameters:
$$
    \boldsymbol{y} = \boldsymbol{w}_0 + \boldsymbol{X}\boldsymbol{w}_1 + \boldsymbol{X}^2\boldsymbol{w}_2 + \dots + \boldsymbol{X}^n\boldsymbol{w}_n
$$
As you have probably guessed, this equation is not linear. We use a trick to make it linear:
- We gather all the $\boldsymbol{X}^2$ to $\boldsymbol{X}^n$ as new features that we created and we concatenate them to $\boldsymbol{X}$.
- All the $\boldsymbol{w}_1$ to $\boldsymbol{w}_n$ are concatenated to $\boldsymbol{w}_0$.

At the end, the polynomial regression has the same formula as the linear regression but with the aggregated arrays.

```python
def create_features(X, degree=2, standardize=True):
    """Creates the polynomial features

    Args:
        X: A torch tensor for the data.
        degree: A intege for the degree of the generated polynomial function.
        standardize: A boolean for scaling the data or not.
    """
    if len(X.shape) == 1:
        X = X.unsqueeze(1)
    # Concatenate a column of ones to has the bias in X
    ones_col = torch.ones((X.shape[0], 1), dtype=torch.float32)
    X_d = torch.cat([ones_col, X], axis=1)
    for i in range(1, degree):
        X_pow = X.pow(i + 1)
        # If we use the gradient descent method, we need to
        # standardize the features to avoid exploding gradients
        if standardize:
            X_pow -= X_pow.mean()
            std = X_pow.std()
            if std != 0:
                X_pow /= std
        X_d = torch.cat([X_d, X_pow], axis=1)
    return X_d

def predict(features, weights):
    return features.mm(weights)

features = create_features(X, degree=3, standardize=False)
y_true = y.unsqueeze(1)
```

### Method 1: Normal equation
The first method is analytical and uses the normal equation.
Training a linear model using least square regression is equivalent to minimize the mean squared error:

$$
\begin{align}
    \text{Mse}(\hat{y}, y) &= \frac{1}{n}\sum_{i=1}^{n}{||\hat{y}_i - y_i ||_{2}^{2}} \\
    \text{Mse}(\hat{y}, y) &= \frac{1}{n}||\boldsymbol{X}\boldsymbol{w} - \boldsymbol{y} ||_2^2
\end{align}
$$

where $n$ is the number of samples, $\hat{y}$ is the predicted value of the model and $y$ is the true target.
The prediction $\hat{y}$ is obtained by matrix multiplication between the input $\boldsymbol{X}$ and the weights of the model $\boldsymbol{w}$.

Minimizing the $\text{Mse}$ can be achieved by solving the gradient of this equation equals to zero in regards to the weights $\boldsymbol{w}$:

$$
\begin{align}
\nabla_{\boldsymbol{w}}\text{Mse}(\hat{y}, y) &= 0 \\
(\boldsymbol{X}^\top \boldsymbol{X})^{-1}\boldsymbol{X}^\top \boldsymbol{y} &= \boldsymbol{w}
\end{align}
$$

For more information on how to find $\boldsymbol{w}$ please visit the section "Linear Least Squares" of this [link](https://en.wikipedia.org/wiki/Least_squares#:~:text=The%20linear%20least%2Dsquares%20problem,is%20similar%20in%20both%20cases).

```python
def normal_equation(y_true, X):
    """Computes the normal equation

    Args:
        y_true: A torch tensor for the labels.
        X: A torch tensor for the data.
    """
    XTX_inv = (X.T.mm(X)).inverse()
    XTy = X.T.mm(y_true)
    weights = XTX_inv.mm(XTy)
    return weights

weights = normal_equation(y_true, features)
y_pred = predict(features, weights)

plt.scatter(X, y)
plt.plot(X, y_pred, c='red')
```

With the normal equation method, the polynomial regressor fits well the synthetic data.

### Method 2: Gradient Descent
 The Gradient descent method takes steps proportional to the negative of the gradient of a function at a given point, in order to iteratively minimize the objective function. The gradient generalizes the notion of derivative to the case where the derivative is with respect to a vector: the gradient of $f$ is the vector containing all of the partial derivatives, denoted $\nabla_{\boldsymbol{x}}f(\boldsymbol{x})$.

 The directional derivative in direction $\boldsymbol{u}$ (a unit vector) is the slope of the function $f$ in direction $\boldsymbol{u}$. In other words, the directional derivative is the derivative of the function $f(\boldsymbol{x} + \sigma \boldsymbol{u})$ with respect to $\sigma$ close to 0. To minimize $f$, we would like to find the direction in which $f$ decreases the fastest. We can do this using the directional derivative:
$$
\begin{align}
    &\min_{\boldsymbol{u}, \boldsymbol{u}^\top \boldsymbol{u} = 1}{\boldsymbol{u}^\top \nabla_{\boldsymbol{x}} f(\boldsymbol{x})} \\
    = &\min_{\boldsymbol{u}, \boldsymbol{u}^\top \boldsymbol{u} = 1}{||\boldsymbol{u}||_2 ||\nabla_{\boldsymbol{x}}f(\boldsymbol{x})||_2 \cos \theta}
\end{align}
$$
ignoring factors that do not depend on $\boldsymbol{u}$, this simplifies to $\min_{u}{\cos \theta}$. This is minimized when $\boldsymbol{u}$ points in the opposite direction as the gradient. Each step of the gradient descent method proposes a new points:
$$
\boldsymbol{x'} = \boldsymbol{x} - \epsilon \nabla_{\boldsymbol{x}}f(\boldsymbol{x})
$$
where $\epsilon$ is the learning rate.
In the context of polynomial regression, the gradient descent is as follow:
$$
\boldsymbol{w} = \boldsymbol{w} - \epsilon \nabla_{\boldsymbol{w}}\text{MSE}
$$
where:
$$
\begin{align}
\nabla_{\boldsymbol{w}}\text{MSE} &= \nabla_{\boldsymbol{w}}\left(\frac{1}{n}{||\boldsymbol{X}\boldsymbol{w} - \boldsymbol{y} ||_2^2}\right) \\
&= \frac{2}{N}\boldsymbol{X}^\top(\boldsymbol{X}\boldsymbol{w} - \boldsymbol{y})
\end{align}
$$

```python
def gradient_descent(X, y_true, lr=0.001, it=30000):
    """Computes the gradient descent

    Args:
        X: A torch tensor for the data.
        y_true: A torch tensor for the labels.
        lr: A scalar for the learning rate.
        it: A scalar for the number of iteration
            or number of gradient descent steps.
    """
    weights_gd = torch.ones((X.shape[1], 1))
    n = X.shape[0]
    fact = 2 / n
    for _ in range(it):
        y_pred = predict(X, weights_gd)
        grad = fact * X.T.mm(y_pred - y_true)
        weights_gd -= lr * grad
    return weights_gd

features = create_features(X, degree=3, standardize=True)
weights_gd = gradient_descent(features, y_true)

pred_gd = predict(features, weights_gd)
```

The mean squared error is even lower when using gradient descent.

```python
plt.scatter(X, y)
plt.plot(X, pred_gd, c='red')
```

### Conclusion
The polynomial regression is an appropriate example to learn more about the concept of normal equation and gradient descent.
This method work well with data that has polynomial shapes but we need to choose the right polynomial degree for a good bias/variance trade-off. However, the polynomial regression method has an important drawback. In fact, it is necessary to transform the data to a higher dimensional space which can be unfeasable if the data is very large.


# Linear least squares

## Fitting data by polynomials

Here are 5-year averages of the worldwide temperature anomaly as compared to the 1951-1980 average (source: NASA).

```julia
year = collect(1955:5:2000);
anomaly = [ -0.0480; -0.0180; -0.0360; -0.0120; -0.0040; 0.1180; 0.2100; 0.3320; 0.3340; 0.4560 ];
using Plots;
plot(year,anomaly,m=:o,l=nothing)
```

The numbers work better if we measure time in years since 1955. (We can quantify that statement later in the course.)

```julia
t = year-1955;
@show m = length(t);
```

    m = length(t) = 10

A polynomial through all of these points has to have degree at least 9 in general (i.e., unless the points are special). We can solve for the coefficients of this polynomial by solving a $10\times 10$ Vandermonde system.

## Polynomial Data-Fitting

Suppose we are given $m$ distinct points $x_1,\dots, x_m\in\mathbb{C}$, and data $y_1,\dots, y_m \in \mathbb{C}$ at these points.  Then there is a unique _polynomial interpolant_ of degree $\leq m-1$:

$$
p(x) = c_0 + c_1x + \cdots + c_{m-1} x^{m-1},
$$

satisfying $p(x_i) = y_i$.

$$\begin{bmatrix} 1 & x_1 & x_1^2 & \cdots & x_1^{m-1} \\ 1 & x_2 & x_2^2 & \cdots & x_2^{m-1} \\  & \vdots & & & \vdots \\ 1 & x_m & x_m^2 & \cdots & x_m^{m-1}\end{bmatrix} \begin{bmatrix} c_0\\ c_1 \\ \vdots \\ c_{m-1} \end{bmatrix} = \begin{bmatrix} y_1 \\ y_2\\ \vdots \\ y_m\end{bmatrix}$$

The Vandermonde matrix is nonsingular if and only if $x_i\ne x_j$.

```julia
A = t.^0;
for j = 1:m-1
    A = [A t.^j];
end
b = anomaly;

c = A\b;   # coefficients, from degree 0 to m-1

using Polynomials
p = Poly(c)
```

    Poly(-0.048 + 0.3655471428571423x - 0.1821111666666659x^2 + 0.03600809999999972x^3 - 0.003737914999999952x^4 + 0.00022573177777777346x^5 - 8.205466666666444e-6x^6 + 1.7689396825396164e-7x^7 - 2.0826666666665615e-9x^8 + 1.0311111111110426e-11x^9)

```julia
plot(t->p(t-1955),1955,2000)
plot!(year,anomaly,m=:o,l=nothing);
title!("World temperature anomaly");
xlabel!("year"); ylabel!("anomaly (deg C)")
```

As intended, the polynomial interpolates the data, but it obviously gives a lot of unphysical nonsense. This phenomenon, which is common with interpolating polynomials over equally spaced times, is an example of _overfitting_.

We actually get a better approximation by reducing the degree to 3.

```julia
A = A[:,1:4];
@show size(A);
```

    size(A) = (10,4)

We now have an overdetermined system of linear equations. The easiest way to define a solution is to minimize the 2-norm of the residual--i.e., least squares. In MATLAB we use the same backslash operator as for square systems, even though the algorithms are very different.

```julia
c = A\b;   # coefficients, from degree 0 to 3
p = Poly(c);  # fitting polynomial
plot(t->p(t-1955),1955,2000)
plot!(year,anomaly,m=:o,l=nothing);
title!("World temperature anomaly");
xlabel!("year"); ylabel!("anomaly (deg C)")
```

This polynomial certainly describes the data better.

## Orthogonal Projection and the Normal Equations

> ** THEOREM. ** Let $A\in\mathbb{C}^{m\times n}$, $m\geq n$, and $\mathbf{b}\in\mathbb{C}^m$.   A vector $\mathbf{x}\in\mathbb{C}^m$ minimizes the residual norm $\|\mathbf{r}\|_2 = \|\mathbf{b} -A\mathbf{x}\|_2$, hence solves the linear squares problem if and only if $\mathbf{r}\perp C(A)$, i.e. $$A^*\mathbf{r} = \mathbf{0} \quad \Longleftrightarrow \quad A^*A \mathbf{x} = A^* \mathbf{b}.$$
Further, $A^*A$ is nonsingular if and only if $A$ has full rank (i.e. linearly independent columns).

** Proof. ** To show $\mathbf{y} = P \mathbf{b}$ is the unique point in $C(A)$ that minimizes $\|\mathbf{b}-\mathbf{y}\|_2$, suppose $\mathbf{z}\ne\mathbf{y}$ is another point.   Since $\mathbf{z}-\mathbf{y}$ is orthogonal to $\mathbf{b}-\mathbf{y}$:

$$\|\mathbf{b}-\mathbf{z} \|_2^2 = \|\mathbf{b}-\mathbf{y}\|_2^2 + \|\mathbf{y}-\mathbf{z}\|_2^2 > \|\mathbf{b}-\mathbf{y}\|_2^2.$$

## Three least squares algortithms

### Linear least squares

There are three distinct ways to solve the general dense linear least squares problem.

```julia
using Plots
pyplot() #choose plotting backend
input = [
    1 6
    2 5
    3 7
    4 10]
X = hcat(ones(size(input)[1]),input[:,1])
y = input[:,2]
betaHat = (X' * X ) \ X' * y #backslash computes LS-solution as in Matlab
print(betaHat)
plot(x->betaHat[2]*x + betaHat[1],0,5,label="curve fit")
scatter!(input[:,1],input[:,2],label="data")
```

    [3.4999999999999964,1.4000000000000012][Plots.jl] Initializing backend: pyplot

    INFO: Precompiling module PyPlot.
    WARNING: No working GUI backend found for matplotlib.

#### Normal equations

First, we can pose and solve the normal equations:

\begin{align}
A\mathbf{x} &= \mathbf{b}\\
A^* A \mathbf{x} &= A^* \mathbf{b}\\
\mathbf{x} &= (A^*A)^{-1}A^*\mathbf{b} = A^+ \mathbf{b}.
\end{align}

$A^+$ is called the _pseudoinverse_ of $A$.

##### Least squares via Normal equations:

1. Form the matrix $A^* A$ and the vector $A^*\mathbf{b}$.
- Compute the 'Cholesky factorization': $A^*A = R^*R$.
- Solve the lower-triangular system $R^*\mathbf{w} = A^*\mathbf{b}$ for $\mathbf{w}$.
- Solve the upper-triangular system $R\mathbf{x} = \mathbf{w}$ for $\mathbf{x}$.

Symmetry implies the computation of $A^*A$ requires only $mn^2$ flops (rather than $2mn^2$).   Step 2 requires $n^3/3$ flops.

> ** THEOREM. **  Least squares via normal equations has operation count $\sim mn^2+ \frac{n^3}{3}$ flops.

```julia
B = A'*A;  z = A'*b;
@show size(B);
```

    size(B) = (4,4)

```julia
[c B\z]
```

    4x2 Array{Float64,2}:
     -0.0261566    -0.0261566
     -0.00908228   -0.00908228
      0.000785734   0.000785734
     -7.74825e-6   -7.74825e-6

#### QR Factorization

Second, we can use a thin QR factorization to express the range of $A$ orthonormally, and reduce to a triangular square system.

Using either Gram-Schmidt or Householder triangularization, one constructs $A = \hat{Q}\hat{R}$.  The orthogonal projector $P$ can be written as $\hat{Q}\hat{Q}^*$.  Therefore:

\begin{align}
A\mathbf{x} &= \mathbf{b}\\
\hat{Q}\hat{R} \mathbf{x} & = \mathbf{b} \\
\hat{R} \mathbf{x} &= \hat{Q}^* \mathbf{b}\\
\mathbf{x} &= \hat{R}^{-1} \hat{Q}^* \mathbf{b} = A^+ \mathbf{b}
\end{align}

##### Least squares via QR factorization

1. Compute the reduced QR factorization $A = \hat{Q}\hat{R}$.
- Compute the vector $\hat{Q}^*\mathbf{b}$.
- Solve the upper-triangular system $\hat{R}\mathbf{x} = \hat{Q}^*\mathbf{b}$ for $\mathbf{x}$.

The work for this algorithm is dominated by the cost of the $QR$ factorization.  If Householder reflectors are used for this step, we have:

> ** THEOREM. ** Work for solving least squares via QR factorization is  $\sim 2mn^2 - \frac{2}{3}n^3$ flops.

```julia
(Q,R) = qr(A);
z = Q'*b;
[c R\z]
```

    4x2 Array{Float64,2}:
     -0.0261566    -0.0261566
     -0.00908228   -0.00908228
      0.000785734   0.000785734
     -7.74825e-6   -7.74825e-6

#### SVD

And third, we can use the SVD to orthgonalize both the range and the domain, ultimately getting a diagonal square system.

Suppose $A=\hat{U}\hat{\Sigma}V^*$ is an SVD of $A$:

\begin{align}
A \mathbf{x} &= \mathbf{b} \\
\hat{U}\hat{\Sigma}V^* \mathbf{x} &= \mathbf{b} \\
\hat{U}\hat{\Sigma} V^* \mathbf{x} & = \hat{U}\hat{U}^* \mathbf{b}\\
\hat{\Sigma} V^* \mathbf{x} &= \hat{U}^* \mathbf{b}\\
\mathbf{x} &= V\hat{\Sigma}^{-1} \hat{U}^*\mathbf{b} = A^+ \mathbf{b}.
\end{align}

##### Least squares via SVD

1. Compute the reduced SVD $A = \hat{U}\hat{\Sigma}V^*$.
- Compute the vector $\hat{U}^*\mathbf{b}$.
- Solve the diagonal system $\hat{\Sigma}\mathbf{w} = \hat{U}^*\mathbf{b}$ for $\mathbf{w}$.
- Set $\mathbf{x} = V\mathbf{w}$.

The operation count for this algorithm is dominated by the computation of the SVD.   Which for $m>> n$ is approximately the same as the QR factorization, however for $m\approx n$ it is typically:

> ** THEOREM. ** The work for the least squares via SVD is $\sim 2mn^2 + 11n^3$ flops.

```julia
(U,s,V) = svd(A);
z = U'*b;
[c V*(z./s)]
```

    4x2 Array{Float64,2}:
     -0.0261566    -0.0261566
     -0.00908228   -0.00908228
      0.000785734   0.000785734
     -7.74825e-6   -7.74825e-6

## Conditioning and stability

Given multiple algorithms to solve a given problem, how do we determine which to use in a given situation?

There are several factors: algorithm _complexity_ or runtime, _parallelizability_, mathematical _conditioning_ and numerical  _stability_.


```python
from sympy import *
from sympy.abc import *
Matrix([x,y,z])
```

$\displaystyle \left[\begin{matrix}x\\y\\z\end{matrix}\right]$

```python
e1 = 5*x+3*y+4*z
e2 = 3*x+4*y+5*z
e3 = 4*x+5*y+3*z
```

```python
A = Matrix([e1,e2,e3])
A
```

$\displaystyle \left[\begin{matrix}5 x + 3 y + 4 z\\3 x + 4 y + 5 z\\4 x + 5 y + 3 z\end{matrix}\right]$

```python
B = A*2
B
```

$\displaystyle \left[\begin{matrix}10 x + 6 y + 8 z\\6 x + 8 y + 10 z\\8 x + 10 y + 6 z\end{matrix}\right]$

```python
Integral(A,x)
```

$\displaystyle \int \left[\begin{matrix}5 x + 3 y + 4 z\\3 x + 4 y + 5 z\\4 x + 5 y + 3 z\end{matrix}\right]\, dx$

```python
Integral(A,x).doit()
```

$\displaystyle \left[\begin{matrix}\frac{5 x^{2}}{2} + x \left(3 y + 4 z\right)\\\frac{3 x^{2}}{2} + x \left(4 y + 5 z\right)\\2 x^{2} + x \left(5 y + 3 z\right)\end{matrix}\right]$

```python
Derivative(A,x)
```

$\displaystyle \frac{\partial}{\partial x} \left[\begin{matrix}5 x + 3 y + 4 z\\3 x + 4 y + 5 z\\4 x + 5 y + 3 z\end{matrix}\right]$

```python
Derivative(A,x).doit()
```

$\displaystyle \left[\begin{matrix}5\\3\\4\end{matrix}\right]$


(nm_gaussian_elimination)=
# Gaussian elimination
```{index} Gaussian elimination
```
## Method
The [Gaussian elimination](https://en.wikipedia.org/wiki/Gaussian_elimination) algorithm is simply a systematic implementation of the method of equation substitution we used in the {ref}`introduction section <nm_linear_algebra_intro>` to solve the \\(2\times 2\\) system (i.e. where we "multiply the second equation by 2 and subtract the first equation from the resulting equation to eliminate \\(x\\) and hence allowing us to find \\(y\\), we can then compute \\(x\\) from the first equation").

Gaussian elimination as the method is atributed to the mathematician Gauss (although it was certainly known before his time) and elimination as we seek to eliminate unknowns. To perform this method for arbitrarily large systems (on paper) we form the so-called augmented matrix

\\[
[A|\pmb{b}] =
\left[
  \begin{array}{rr|r}
    2 & 3 & 7 \\\\\\
    1 & -4 & 3  \\\\\\
  \end{array}
\right].
\\]

Note that this encodes the equations (including the RHS values). We can perform so-called row operations and as long as we are consistent with what we do for the LHS and RHS components then what this system is describing will not change - a solution to the updated system will be the same as the solution to the original system.

Our task is to update the system so we can easily read off the solution - of course this is exactly what we do via the substitution approach. First we multiplied the second equation by 2, this yield the updated augmented matrix:

\\[
\left[
  \begin{array}{rr|r}
    2 & 3 & 7 \\\\\\
    2 & -8 & 6 \\\\\\
  \end{array}
\right].
\\]
We can use the following notation to describe this operation:

\\[ \text{Eq. } (2) \leftarrow 2\times \text{Eq. } (2). \\]

Note importantly that this does not change anything about what these pair of equations are telling us about the unknown solution vector \\(\pmb{x}\\) which although it doesn't appear, is implicilty defined by this augmented equation.

The next step was to subtract the first equation from the updated second (\\( \text{Eq. } (2) \leftarrow \text{Eq. } (2) - \text{Eq. } (1) \\)):

\\[
\left[
  \begin{array}{rr|r}
    2 & 3 & 7 \\\\\\
    0 & -11 & -1 \\\\\\
  \end{array}
\right].
\\]

The square matrix that is now in the \\(A\\) position of this augmented system is an example of an upper-triangular matrix - all entries below the diagonal are zero.

For such a matrix we can perform back substitution - starting at the bottom to solve trivially for the final unknown (\\(y\\) here which clearly takes the value \\(-1/-11\\)), and then using this knowledge working our way up to solve for each remaining unknown in turn, here just \\(x\\) (solving \\(2x + 3\times (1/11) = 7\\)).

We can perform the similar substitution if we had a lower triangular matrix, first finding the first unknown and then working our way forward through the remaining unknowns - hence in this case forward substitution.

If we wished we could continue working on the augmented matrix to make the \\(A\\) component diagonal - divide the second equation by 11 and multiply by 3 (\\( \text{Eq. } (2) \leftarrow (3/11)\times \text{Eq. } (2) \\)) and add it to the first (\\( \text{Eq. } (1) \leftarrow \text{Eq. } (1) +  \text{Eq. } (2) \\)):

\\[
\left[
  \begin{array}{rr|r}
    2 & 0 & 7-3/11\\\\\\
    0 & -3 & -3/11 \\\\\\
  \end{array}
\right]
\\]

and we can further make it the identity by dividing the rows by 2 and -3 respectively (\\( \text{Eq. } (1) \leftarrow (1/2)\times \text{Eq. } (1) \\), \\( \text{Eq. } (2) \leftarrow (-1/3)\times \text{Eq. } (2) \\)) :

\\[
\left[
  \begin{array}{rr|r}
    1 & 0 & (7-3/11)/2 \\\\\\
    0 & 1 & 1/11 \\\\\\
  \end{array}
\right].
\\]

Each of these augmented matrices encodes exactly the same information as the original matrix system in terms of the unknown vector \\(\pmb{x}\\), and hence this is telling us that

\\[ \pmb{x} = I \pmb{x} = \left[
  \begin{array}{c}
    (7-3/11)/2 \\\\\\
    1/11 \\\\\\
  \end{array}
\right]
\\]

i.e. exactly the solution we found when we performed back substitution from the upper-triangular form of the augmented system.

### Gaussian elimination example

Consider a system of linear equations

\\[\begin{align*}
  2x + 3y - 4z &= 5 \\\\\\
  6x + 8y + 2z &= 3 \\\\\\
  4x + 8y - 6z &= 19
\end{align*}\\]

write this in matrix form, form the corresponding augmented system and perform row operations until you get to upper-triangular form, find the solution using back substitution (do this all with pen and paper).

We should find that \\(x=-6\\), \\(y=5\\), \\(z=-1/2\\).

```{admonition} Answer
:class: dropdown

We begin by taking first two equations and try eliminating \\(x\\). To do that we need to multiply first equation by 3:

\\[
\begin{align*}
  6x + 9y - 12z &= 15 \\\\\\
  6x + 8y + 2z &= 3 \\\\\\
\end{align*}
\\]

Then we will subtract first equation from the second to get:
\\[-1y+14z=-12.\\]

Our system now looks:

\\[
\begin{align*}
  2x + 3y - 4z &= 5 \\\\\\
  0x - 1y + 14z &= -12 \\\\\\
  4x + 8y - 6z &= 19
\end{align*}
\\]

Now let's take first and third equation and eliminate \\(x\\) by multiplying first equation by 2:

\\[
\begin{align*}
  4x + 6y - 8z &= 10 \\\\\\
  4x + 8y - 6z &= 19
\end{align*}
\\]

We subtract the equations to obtain:
\\[2y+2z=9.\\]

Our updated system is then:

\\[
\begin{align*}
  2x + 3y - 4z &= 5 \\\\\\
  0x - 1y  + 14z &= -12 \\\\\\
  0x + 2y + 2z &= 9
\end{align*}
\\]

Unknown \\(x\\) was successfully eliminated from two equations. Now comparing two last equations, we see that we need to multiply first one by -2 to eliminate \\(y\\):

\\[
\begin{align*}
  0x + 2y  - 28z &= 24 \\\\\\
  0x + 2y + 2z &= 9
\end{align*}
\\]

Subtracting the equations we get:
\\[ 0y - 30z = 15.\\]

Our updated system is then:
\\[
\begin{align*}
  2x + 3y - 4z &= 5 \\\\\\
  0x + y  - 14z &= 12 \\\\\\
  0x + 0y - 30z &= 15
\end{align*}
\\]

We have successfully eliminated the unknown \\(y\\). Now, our equations look like they have a triangle where the coefficients are \\(0\\) and another triangle where the coefficients are \\(\neq 0\\). Because the triangle where coefficients are \\(\neq 0\\) is above the triangle where the coefficients are \\(0\\), this form is called the **upper triangle form**.

Now we will use back substitution to solve the remaining system. From third equation we get that \\(z=-\frac{1}{2}\\).

We will proceed to equation 2 knowing that \\(z=-\frac{1}{2}\\), therefore \\(y=5\\).

Finally, we can solve first equation to find that \\(x=6\\).

```
(nm_gaussian_elimination_code)=
## Implementation
Notice that we are free to perform the following operations on the augmented system without changing the corresponding solution:

* exchanging two rows

* multiplying a row by a non-zero constant (\\(\text{Eq. } (i)\leftarrow \lambda \times \text{Eq. } (i)\\))

* subtracting a (non-zero) multiple of one row with another (\\(\text{Eq. } (i)\leftarrow \text{Eq. } (i) - \lambda \times \text{Eq. }(j)\\))

Let's consider the algorithm mid-way working on an arbitrary matrix system, i.e. assume that the first \\(k\\) rows (i.e. above the horizontal dashed line in the matrix below) have already been transformed into upper-triangular form, while the equations/rows below are not yet in this form. The augmented equation in this case can be assumed to look like

```{margin} Note
Remember that here as we are mid-way through the algorithm the \\(A\\)'s and \\(b\\)'s in the above are not the same as in the original system!
```

\\[
\left[
  \begin{array}{rrrrrrrrr|r}
    A_{11} & A_{12} & A_{13} & \cdots & A_{1k} & \cdots & A_{1j} & \cdots & A_{1n} & b_1 \\\\\\
    0      & A_{22} & A_{23} & \cdots & A_{2k} & \cdots & A_{2j} & \cdots & A_{2n} & b_2 \\\\\\
    0      & 0      & A_{33} & \cdots & A_{3k} & \cdots & A_{3j} & \cdots & A_{3n} & b_3 \\\\\\
    \vdots & \vdots & \vdots & \ddots & \vdots & \ddots & \vdots & \ddots & \vdots & \vdots \\\\\\
    0      & 0      & 0      & \cdots & A_{kk} & \cdots & A_{kj} & \cdots & A_{kn} & b_k \\\\\\
\hdashline
    \vdots & \vdots & \vdots & \ddots & \vdots & \ddots & \vdots & \ddots & \vdots & \vdots \\\\\\
    0      & 0      & 0      & \cdots & A_{ik} & \cdots & A_{ij} & \cdots & A_{in} & b_i \\\\\\
    \vdots & \vdots & \vdots & \ddots & \vdots & \ddots & \vdots & \ddots & \vdots & \vdots \\\\\\
    0      & 0      & 0      & \cdots & A_{nk} & \cdots & A_{nj} & \cdots & A_{nn} & b_n \\\\\\
\end{array}
\right].
\\]

Our aim as a next step in the algorithm is to use row \\(k\\) (the "pivot row") to eliminate \\(A_{ik}\\), and we need to do this for all of the rows \\(i\\) below the pivot, i.e. for all \\(i>k\\).

The zeros to the left of the leading term in the pivot row means that these operations will not mess up the fact that we have all the zeros we are looking for in the lower left part of the matrix.

To eliminate \\(A_{ik}\\) for a single row \\(i\\) we need to perform the operation
\\[ \text{Eq. } (i)\leftarrow \text{Eq. } (i) - \frac{A_{ik}}{A_{kk}} \times \text{Eq. }(k) \\]

or equivalently

\\[
\begin{align}
A_{ij} &\leftarrow A_{ij} - \frac{A_{ik}}{A_{kk}} A_{kj}, \quad j=k,k+1,\ldots,n\\\\\\
b_i &\leftarrow b_i - \frac{A_{ik}}{A_{kk}} b_{k}
\end{align}
\\]

\\(j\\) only needs to run from \\(k\\) upwards as we can assume that the earlier entries in column \\(i\\) have already been set to zero, and also that the corresponding terms from the pivot row are also zero (we don't need to perform operations that we know involve the addition of zeros!).

To eliminate these entries for all rows below the pivot we need to repeat for all \\(i>k\\).

### Upper triangular form code

We will write some code that takes a matrix \\(A\\) and a vector \\(\pmb{b}\\) and converts it into upper-triangular form using the above algorithm. For the \\(2 \times 2\\) and \\(3\times 3\\) examples from above we will compare the resulting \\(A\\) and \\(\pmb{b}\\) you obtain following elimination.

At first we will write a function to obtain the upper triangular matrix form:

```python
import numpy as np

def upper_triangle(A, b):
    """ A function to covert A into upper triangluar form through row operations.
    The same row operations are performed on the vector b.

    Note that this implementation does not use partial pivoting which is introduced below.

    Also note that A and b are overwritten, and hence we do not need to return anything
    from the function.
    """
    n = np.size(b)
    rows, cols = np.shape(A)
    # Check A is square
    assert(rows == cols)
    # Check A has the same numner of rows as the size of the vector b
    assert(rows == n)

    # Loop over each pivot row - all but the last row which we will never need to use as a pivot
    for k in range(n-1):
        # Loop over each row below the pivot row, including the last row which we do need to update
        for i in range(k+1, n):
            # Define the scaling factor for this row outside the innermost loop otherwise
            # its value gets changed as you over-write A!!
            # There's also a performance saving from not recomputing things when not strictly necessary
            s = (A[i, k] / A[k, k])
            # Update the current row of A by looping over the column j
            # start the loop from k as we can assume the entries before this are already zero
            for j in range(k, n):
                A[i, j] = A[i, j] - s*A[k, j]
            # and update the corresponding entry of b
            b[i] = b[i] - s*b[k]

    return A, b
```

Now we can test our code on the examples from above. First \\(2\times 2\\) example:

```python
A = np.array([[2., 3.],
              [1., -4.]])
b = np.array([7., 3.])

A, b = upper_triangle(A, b)

print('Our A matrix following row operations to transform it into upper-triangular form:')
print(A)
print('The correspondingly updated b vector:')
print(b)
```

    Our A matrix following row operations to transform it into upper-triangular form:
    [[ 2.   3. ]
     [ 0.  -5.5]]
    The correspondingly updated b vector:
    [ 7.  -0.5]

\\(3\times 3\\) example:

```python
A = np.array([[2., 3., -4.],
              [6., 8., 2.],
              [4., 8., -6.]])
b = np.array([5., 3., 19.])

A, b = upper_triangle(A, b)

print('\nOur A matrix following row operations to transform it into upper-triangular form:')
print(A)
print('The correspondingly updated b vector:')
print(b)
```

    Our A matrix following row operations to transform it into upper-triangular form:
    [[ 2.  3. -4.]
     [ 0. -1. 14.]
     [ 0.  0. 30.]]
    The correspondingly updated b vector:
    [  5. -12. -15.]

### Back substitution code

Now that we have an augmented system in the upper-triangular form

\\[
\left[
  \begin{array}{rrrrr|r}
    A_{11} & A_{12} & A_{13} & \cdots & A_{1n} &  b_1 \\\\\\
    0      & A_{22} & A_{23} & \cdots & A_{2n} &  b_2 \\\\\\
    0      & 0      & A_{33} & \cdots & A_{3n} &  b_3 \\\\\\
    \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\\\\\
    0      & 0      & 0      & \cdots & A_{nn} &  b_n \\\\\\
\end{array}
\right].
\\]

where the solution \\(\pmb{x}\\) of the original system also satisfies \\(A\pmb{x}=\pmb{b}\\) for the \\(A\\) and \\(\pmb{b}\\) in the above upper-triangular form (rather than the original \\(A\\) and \\(\pmb{b}\\)).

We can solve the final equation row to yield

\\[x_n = \frac{b_n}{A_{nn}}.\\]

The second to last equation then yields

\\[
\begin{align}
A_{n-1,n-1}x_{n-1} + A_{n-1,n}x_n &= b_{n-1}\\\\\\
\implies x_{n-1} = \frac{b_{n-1} - A_{n-1,n}x_n}{A_{n-1,n-1}}\\\\\\
\implies x_{n-1} = \frac{b_{n-1} - A_{n-1,n}\frac{b_n}{A_{nn}}}{A_{n-1,n-1}}
\end{align}
\\]

and so on to row \\(k\\) which yields

\\[
\begin{align}
A_{k,k}x_{k} + A_{k,k+1}x_{k+1} +\cdots +  A_{k,n}x_n &= b_{k}\\\\\\
\iff A_{k,k}x_{k} + \sum_{j=k+1}^{n}A_{kj}x_j &= b_{k}\\\\\\
\implies x_{k} &= \left( b_k - \sum_{j=k+1}^{n}A_{kj}x_j\right)\frac{1}{A_{kk}}
\end{align}
\\]

We will extend the code to perform back substitution and hence to obtain the final solution \\(\pmb{x}\\).

```python
# This function assumes that A is already an upper triangular matrix,
# e.g. we have already run our upper_triangular function if needed.

def back_substitution(A, b):
    """ Function to perform back subsitution on the system Ax=b.

    Returns the solution x.

    Assumes that A is on upper triangular form.
    """
    n = np.size(b)
    # Check A is square and its number of rows and columns same as size of the vector b
    rows, cols = np.shape(A)
    assert(rows == cols)
    assert(rows == n)
    # We can/should check that A is upper triangular using np.triu which is the
    # upper triangular part of a matrix - if A is already upper triangular, then
    # it should of course match the upper-triangular component of A!!
    assert(np.allclose(A, np.triu(A)))

    x = np.zeros(n)
    # Start at the end (row n-1) and work backwards
    for k in range(n-1, -1, -1):
        # Note that we could do this update in a single vectorised line
        # using np.dot or @ - this could also speed things up
        s = 0.
        for j in range(k+1, n):
            s = s + A[k, j]*x[j]
        x[k] = (b[k] - s)/A[k, k]

    return x
```

Let's test it:

```python
A = np.array([[2., 3., -4.],
              [6., 8., 2.],
              [4., 8., -6.]])
b = np.array([5., 3., 19.])

A_upp, b_upp = upper_triangle(A, b)

# Print the solution using our codes
x = back_substitution(A_upp, b_upp)
print('Our solution: ',x)

# Check our answer against what SciPy gives us by multiplying b by A inverse
import scipy.linalg as sl

print('SciPy solution: ',sl.inv(A) @ b)
print('Success: ', np.allclose(x, sl.inv(A) @ b))
```

    Our solution:  [-6.   5.  -0.5]
    SciPy solution:  [-6.   5.  -0.5]
    Success:  True

(nm_gauss_jordan_elimination)=
## Gauss-Jordan elimination
```{index} Gauss-Jordan elimination
```
Recall that for the augmented matrix example above we continued past the upper-triangular form so that the augmented matrix had the identity matrix in the \\(A\\) location. This algorithm has the name Gauss-Jordan elimination but note that it requires more operations than the conversion to upper-triangular form followed by back subsitution and so is only of academic interest.

### Matrix inversion

Note that if we were to form the augmented equation with the full identity matrix in the place of the vector \\(\pmb{b}\\), i.e. \\([A|I]\\) and performed row operations exactly as above until \\(A\\) is transformed into the identity matrix \\(I\\), then we would be left with the inverse of \\(A\\) in the original \\(I\\) location, i.e.

\\[ [A|I] \rightarrow [I|A^{-1}].\\]

We will write the code to construct inverse matrix.

```{admonition} Hints
:class: dropdown

We have written a a bunch of matrices below. Seee if you can find what the sequence of matrices mean. You should feel that the operations are oddly familiar.

\\[
A=
\left(
  \begin{array}{rrr}
    2 & -1 & 0 \\\\\\
    -1 & 2 & -1 \\\\\\
    0  & -1 & 2  \\\\\\
  \end{array}
\right)
\quad\quad I=
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    0 & 1 & 0 \\\\\\
    0 & 0 & 1  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    2 & -1 & 0 \\\\\\
    2 & -4 & +2 \\\\\\
    0  & -1 & 2  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    0 & -2 & 0 \\\\\\
    0  & 0 & 1  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    2 & -1 & 0 \\\\\\
    0 & -3 & +2 \\\\\\
    0  & -1 & 2  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    -1 & -2 & 0 \\\\\\
    0  & 0 & 1  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    2 & -1 & 0 \\\\\\
    0 & -3 & +2 \\\\\\
    0  & -3 & 6  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    -1 & -2 & 0 \\\\\\
    0  & 0 & 3  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    2 & -1 & 0 \\\\\\
    0 & -3 & +2 \\\\\\
    0  & 0 & 4  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    -1 & -2 & 0 \\\\\\
    1  & 2 & 3  \\\\\\
  \end{array}
\right)
\\]
\\[
\left(
  \begin{array}{rrr}
    2 & -1 & 0 \\\\\\
    0 & -6 & +4 \\\\\\
    0  & 0 & 4  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    -2 & -4 & 0 \\\\\\
    1  & 2 & 3  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    2 & -1 & 0 \\\\\\
    0 & -6 & 0 \\\\\\
    0  & 0 & 4  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    -3 & -6 & -3 \\\\\\
    1  & 2 & 3  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    12 & -6 & 0 \\\\\\
    0 & -6 & 0 \\\\\\
    0  & 0 & 4  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    6 & 0 & 0 \\\\\\
    -3 & -6 & -3 \\\\\\
    1  & 2 & 3  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    -12 & 0 & 0 \\\\\\
    0 & -6 & 0 \\\\\\
    0  & 0 & 4  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    -9 & -6 & -3 \\\\\\
    -3 & -6 & -3 \\\\\\
    1  & 2 & 3  \\\\\\
  \end{array}
\right)
\\]

\\[
\left(
  \begin{array}{rrr}
    1 & 0 & 0 \\\\\\
    0 & 1 & 0 \\\\\\
    0  & 0 & 1  \\\\\\
  \end{array}
\right)
\quad\quad
\left(
  \begin{array}{rrr}
    3/4 & 1/2 & 1/4 \\\\\\
    1/2 & 1 & 1/2 \\\\\\
    1/4 & 2/4 & 3/4  \\\\\\
  \end{array}
\right)
\\]

\\[
A^{-1}=
\left(
  \begin{array}{rrr}
    3/4 & 1/2 & 1/4 \\\\\\
    1/2 & 1 & 1/2 \\\\\\
    1/4 & 2/4 & 3/4  \\\\\\
  \end{array}
\right)
\\]

```

```python
# This updated version of the upper_triangular function now
# assumes that a matrix, B, is in the old vector location (what was b)
# in the augmented system, and applies the same operations to
# B as to A - only a minor difference

def upper_triangle2(A, B):
    n, m = np.shape(A)
    assert(n == m)  # This is designed to work for a square matrix

    # Loop over each pivot row.
    for k in range(n-1):
        # Loop over each equation below the pivot row.
        for i in range(k+1, n):
            # Define the scaling factor outside the innermost
            # loop otherwise its value gets changed as you are
            # over-writing A
            s = (A[i, k]/A[k, k])
            for j in range(n):
                A[i, j] = A[i, j] - s*A[k, j]
                # Replace the old b update with the same update as A
                B[i, j] = B[i, j] - s*B[k, j]

# This is a version which transforms the matrix into lower
# triangular form - the point here is that if you give it a
# matrix that is already in upper triangular form, then the
# result will be a diagonal matrix
def lower_triangle2(A, B):
    n, m = np.shape(A)
    assert(n == m)  # This is designed to work for a square matrix

    # Now it's basically just the upper triangular algorithm
    # applied backwards
    for k in range(n-1, -1, -1):
        for i in range(k-1, -1, -1):
            s = (A[i, k]/A[k, k])
            for j in range(n):
                A[i, j] = A[i, j] - s*A[k, j]
                B[i, j] = B[i, j] - s*B[k, j]
```

```python
# Let's redefine A as our matrix above
A = np.array([[2., 3., -4.],
              [3., -1., 2.],
              [4., 2., 2.]])

# and B is the identity of the corresponding size
B = np.eye(np.shape(A)[0])

# transform A into upper triangular form
# (and perform the same operations on B)
upper_triangle2(A, B)
print('Upper triangular transformed A = ')
print(A)

# now make this updated A lower triangular as well
# (the result should be diagonal)
lower_triangle2(A, B)
print('\nand following application of our lower triangular function = ')
print(A)

# The final step to achieve the identity is just to divide each row through by the value
# of the diagonal to end up with 1's on the main diagonal and 0 everywhere else.
for i in range(np.shape(A)[0]):
    B[i, :] = B[i, :]/A[i, i]
    A[i, :] = A[i, :]/A[i, i]

# the final A should be the identity
print('\nOur final transformed A = ')
print(A)

# the final B should therefore be the inverse of the original B
print('\nand the correspondingly transformed B = ')
print(B)

# let's compute the inverse using built-in functions and check
# we get the same answer (we need to reinitialise A)
A = np.array([[2., 3., -4.], [3., -1., 2.], [4., 2., 2.]])
print('\nSciPy computes the inverse as:')
print(sl.inv(A))

# B should now store the inverse of the original A - let's check
print('\nSuccess: ', np.allclose(B, sl.inv(A)))
```

    Upper triangular transformed A =
    [[ 2.          3.         -4.        ]
     [ 0.         -5.5         8.        ]
     [ 0.          0.          4.18181818]]

    and following application of our lower triangular function =
    [[ 2.          0.          0.        ]
     [ 0.         -5.5         0.        ]
     [ 0.          0.          4.18181818]]

    Our final transformed A =
    [[ 1.  0.  0.]
     [-0.  1. -0.]
     [ 0.  0.  1.]]

    and the correspondingly transformed B =
    [[ 0.13043478  0.30434783 -0.04347826]
     [-0.04347826 -0.43478261  0.34782609]
     [-0.2173913  -0.17391304  0.23913043]]

    SciPy computes the inverse as:
    [[ 0.13043478  0.30434783 -0.04347826]
     [-0.04347826 -0.43478261  0.34782609]
     [-0.2173913  -0.17391304  0.23913043]]

    Success:  True

You may have noticed above that we have no way of guaranteeing that the \\(A_{kk}\\) we divide through by in the Guassian elimination or back substitution algorithms is non-zero (or not very small which will also lead to computational problems).

We commented that we are free to exchange two rows in our augmented system - how could you use this fact to build robustness into our algorithms in order to deal with matrices for which our algorithms do lead to very small or zero \\(A_{kk}\\) values?

```python
# This function swaps rows in matrix A
# (and remember that we need to do likewise for the vector b
# we are performing the same operations on)

def swap_row(A, b, i, j):
    """ Swap rows i and j of the matrix A and the vector b.
    """
    if i == j:
        return
    print('swapping rows', i,'and', j)
    # If we are swapping two values, we need to take a copy of one of them first otherwise
    # we will lose it when we make the first swap and will not be able to use it for the second.
    # We need to make sure it is a real copy - not just a copy of a reference to the data!
    # use np.copy to do this.
    iA = np.copy(A[i, :])
    ib = np.copy(b[i])

    A[i, :] = A[j, :]
    b[i] = b[j]

    A[j, :] = iA
    b[j] = ib

# This is a new version of the upper_triangular function
# with the added step of swapping rows so the largest
# magnitude number is always our pivot/
# pp stands for partial pivoting which will be explained
# in more detail below.

def upper_triangle_pp(A, b):
    """ A function to covert A into upper triangluar form through row operations.
    The same row operations are performed on the vector b.

    This version uses partial pivoting.

    Note that A and b are overwritten, and hence we do not need to return anything
    from the function.
    """
    n = np.size(b)
    # check A is square and its number of rows and columns same as size of the vector b
    rows, cols = np.shape(A)
    assert(rows == cols)
    assert(rows == n)

    # Loop over each pivot row - all but the last row
    for k in range(n-1):
        # Swap rows so we are always dividing through by the largest number.
        # initiatise kmax with the current pivot row (k)
        kmax = k
        # loop over all entries below the pivot and select the k with the largest abs value
        for i in range(k+1, n):
            if abs(A[kmax, k]) < abs(A[i, k]):
                kmax = i
        # and swap the current pivot row (k) with the row with the largest abs value below the pivot
        swap_row(A, b, kmax, k)

        for i in range(k+1, n):
            s = (A[i, k]/A[k, k])
            for j in range(k, n):
                A[i, j] = A[i, j] - s*A[k, j]
            b[i] = b[i] - s*b[k]

# Apply the new code with row swaps to our matrix problem from above
A = np.array([[2., 3., -4.],
              [3., -1., 2.],
              [4., 2., 2.]])
b = np.array([10., 3., 8.])

upper_triangle_pp(A, b)

print('\nA and b with row swaps: ')
print(A)
print(b)
# compute the solution from these using our back substitution code
# could also have used SciPy of course
x1 = back_substitution(A, b)

# compare with our first function with no row swaps
A = np.array([[2., 3., -4.],
              [3., -1., 2.],
              [4., 2., 2.]])
b = np.array([10., 3., 8.])

upper_triangle(A, b)

print('\nA and b without any row swaps: ')
print(A)
print(b)
x2 = back_substitution(A, b)

# check these two systems are equivalent
print('\nThese two upper triangular systems are equivalent (i.e. have the same solution): ',np.allclose(x1, x2))
```

    swapping rows 2 and 0

    A and b with row swaps:
    [[ 4.   2.   2. ]
     [ 0.  -2.5  0.5]
     [ 0.   0.  -4.6]]
    [ 8.  -3.   3.6]

    A and b without any row swaps:
    [[ 2.          3.         -4.        ]
     [ 0.         -5.5         8.        ]
     [ 0.          0.          4.18181818]]
    [ 10.         -12.          -3.27272727]

    These two upper triangular systems are equivalent (i.e. have the same solution):  True


```python
import sympy
t, K ,r, P0, C1 = sympy.symbols('t, K, r, P_0, C_1')
P = sympy.Function('P')
edo = P(t).diff(t) - r * P(t) * (1 - P(t)/K)
edo
```

$\displaystyle - r \left(1 - \frac{P{\left(t \right)}}{K}\right) P{\left(t \right)} + \frac{d}{d t} P{\left(t \right)}$

```python
edo_sol = sympy.dsolve(edo, P(t))
edo_sol
```

$\displaystyle P{\left(t \right)} = \frac{K e^{C_{1} K + r t}}{e^{C_{1} K + r t} - 1}$

```python
ini_cond = {P(0): P0}
ini_cond
```

    {P(0): P_0}

```python
C_eq = edo_sol.subs(t,0).subs(ini_cond)
C_eq
```

$\displaystyle P_{0} = \frac{K e^{C_{1} K}}{e^{C_{1} K} - 1}$

```python
import matplotlib.pyplot as plt
import numpy as np
def logistica(t, P0=100, K=1000, r=0.25):
    A = P0 / (P0 - K)
    return K / (1 - np.exp(-r*t) / A)
```

```python
t = np.linspace(0, 40, 100)  # Intervalo de tiempo en que se obtiene la solución
p1 = plt.plot(t, logistica(t, P0 = 100), label=r'$P_0 = 100$')
p1 = plt.plot(t, logistica(t, P0 = 2000), label=r'$P_0 = 1500$')
plt.legend()
plt.show()
```


Universidade Tecnológica Federal do Paraná
Professor: Wellington José Corrêa
Orientando: Enzo Dornelles Italiano
Cálculo Numérico

# Integração Numérica

Como de costume, devemos executar os códigos abaixo antes de executar.

## Códigos

```
# !pip install mpld3
# !pip install "git+https://github.com/javadba/mpld3@display_fix"
import copy
import math
import mpld3
import numpy as np
from sympy import *
from mpld3 import plugins
import matplotlib.pyplot as plt
x,t = symbols('x t')

def  trapezio(f, a, b):
    h = b - a
    print("O valor de h é ", h)
    deriv = h/2*(f.subs(x,a)+f.subs(x,b))
    exact = integrate(f, (x, a, b))
    print("Pela Regra do Trapézio, temos que a integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é aproximadamente", deriv)

    print("\nComo comparação, o valor exato da integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é", exact)

    f = diff(f, x, 2)
    maior = abs(f.subs(x,a))
    if abs(f.subs(x,b)) > maior:
        maior = abs(f.subs(x,b))
    E = (-h**3/12)*maior
    print("\nLimitante")
    print("|E| <=", abs(E))

def trapezio_gen(f, a, b, n):
    h = (b - a)/n
    print("O valor de h é", h)
    xk = np.linspace(a,b,n+1)
    fx = 0
    for i in range(len(xk)):
        if i == 0 or i == len(xk)-1:
            fx += f.subs(x, xk[i])
        else:
            fx += 2*f.subs(x, xk[i])
    deriv = (h/2)*fx
    exact = integrate(f, (x, a, b))
    print("Pela Regra do Trapézio, temos que a integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é aproximadamente", deriv)

    print("\nComo comparação, o valor exato da integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é", exact)

    f = diff(f, x, 2)
    maior = abs(f.subs(x,a))
    if abs(f.subs(x,b)) > maior:
        maior = abs(f.subs(x,b))
    E = (h**2/12)*maior*(b-a)
    print("\nLimitante")
    print("|E| <=", abs(E))

def simpson13(f, a, b, n):
    h = (b - a)/n
    xk = np.linspace(a,b,n+1)
    fx = 0
    for i in range(len(xk)):
        if i == 0 or i == len(xk)-1:
            fx += f.subs(x, xk[i])
        elif i % 2 == 0:
            fx += 2*f.subs(x, xk[i])
        else:
            fx += 4*f.subs(x, xk[i])
    deriv = (h/3)*fx
    exact = integrate(f, (x, a, b)).evalf()

    print("Pela Regra 1/3 de Simpson, temos que a integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é aproximadamente", deriv)

    print("\nComo comparação, o valor exato da integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é", exact)

    f = diff(f, x, 4)
    maior = abs(f.subs(x,a))
    if abs(f.subs(x,b)) > maior:
        maior = abs(f.subs(x,b))
    E = (h**4/180)*maior*(b-a)
    print("\nLimitante")
    print("|E| <=", abs(E.evalf()))

def calc_parabola_vertex(x1, y1, x2, y2, x3, y3):
    denom = (x1-x2) * (x1-x3) * (x2-x3);
    A     = (x3 * (y2-y1) + x2 * (y1-y3) + x1 * (y3-y2)) / denom;
    B     = (x3*x3 * (y1-y2) + x2*x2 * (y3-y1) + x1*x1 * (y2-y3)) / denom;
    C     = (x2 * x3 * (x2-x3) * y1+x3 * x1 * (x3-x1) * y2+x1 * x2 * (x1-x2) * y3) / denom;
    return A,B,C

def simpson_tabela13(a,b,n,y):
    h = (b - a)/n
    xk = np.linspace(a,b,n+1)
    fx = 0
    for i in range(len(xk)):
        if i == 0 or i == len(xk)-1:
            fx += y[i]
        elif i % 2 == 0:
            fx += 2*y[i]
        else:
            fx += 4*y[i]
    deriv = (h/3)*fx
    print("O valor da integral é aproximadamente", deriv)

def simpson38(f,a,b,n):
    h = (b-a)/n
    xk = np.linspace(a,b,n+1)
    fx = 0
    for i in range(len(xk)):
        if i == 0 or i == len(xk)-1:
            fx += f.subs(x,xk[i])
        elif i % 3 == 0:
            fx += 2*f.subs(x,xk[i])
        else:
            fx += 3*f.subs(x,xk[i])
    deriv = (3/8)*h*fx
    exact = integrate(f, (x, a, b)).evalf()

    print("Pela Regra 1/3 de Simpson, temos que a integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é aproximadamente", deriv)

    print("\nComo comparação, o valor exato da integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é", exact)

    f = diff(f, x, 4)
    maior = abs(f.subs(x,a))
    if abs(f.subs(x,b)) > maior:
        maior = abs(f.subs(x,b))
    E = (h**4/80)*maior*(b-a)
    print("\nLimitante")
    print("|E| <=", abs(E.evalf()))

def simpson_tabela38(a,b,n,y):
    h = (b - a)/n
    xk = np.linspace(a,b,n+1)
    fx = 0
    for i in range(len(xk)):
        if i == 0 or i == len(xk)-1:
            fx += y[i]
        elif i % 3 == 0:
            fx += 2*y[i]
        else:
            fx += 3*y[i]
    deriv = (3/8)*h*fx
    print("O valor da integral é aproximadamente", deriv)

def quadGauss(f, a, b, n):
    table = [[[0.5773502692, 1],[-0.5773502692, 1]],
             [[0.7745966692, 0.5555555556],[0, 0.8888888889],[-0.7745966692, 0.555555556]],
             [[0.8611363116, 0.3478548451],[0.3399810436, 0.6521451549],[-0.3399810436, 0.6521451549],[0.8611363116, 0.3478548451]],
             [[0.9061798459, 0.2369268850],[0.5384693101, 0.4786286705],[0, 0.5688888889],[-0.5384693101, 0.4786286705],[-0.9061798459, 0.2369268850]]]

    g = f.subs(x, ((1/2)*((b-a)*t+a+b)))
    expr = lambdify(t, g, "numpy")
    if n > 2:
        soma = 0
        for i in range(n):
            soma += table[n-2][i][1]*expr(table[n-2][i][0])
        result = ((b-a)/2)*soma
    else:
        soma = 0
        for i in range(n):
            soma += expr(table[n-2][i][0])
        result = ((b-a)/2)*soma
    print("O valor aproximado da integral")
    pprint(Integral(f, (x, a, b)), use_unicode=True)
    print("é", result)
```

## 1. Regra dos Trapézios

### 1.1 Regra dos trapézios simples

O procedimento que usaremos para calcular a integral definida via Regra dos trapézios é trapezio(f,a,b)

Exemplo:Calcule o valor aproximado da integral definida da função $f(x)=ln(x)+x$ entre 0,5 e
1, usando a regra do trapézios e determine um limitante superior.

Solução: inicialmente, definamos $f$ e os pontos a e b:

```
def f(x): return log(x)+x
a = 0.5
b = 1
```

Usando o procedimento citado anteriormente, temos que

```
trapezio(f(x), a, b)
```

Para plotar o gráfico de $f(x)$ e o trapézio construído, usemos o comando:

```
fig, ax = plt.subplots()
z = np.arange(a,b+0.001,0.001)

y = lambdify(x, f(x), "numpy")

pontos = [[a,b],[y(a), y(b)]]

ax.fill_between([a,b],pontos[1], color="red")
ax.plot(z,y(z), "black")
ax.plot(pontos[0],pontos[1])

ax.grid()
plugins.connect(fig, plugins.MousePosition(fontsize=14))

mpld3.display()
```

### 1.2 Regra do trapézio generalizada

Neste momento, o procedimento é para calcular a integral definida pela regra dos trapézios generalizada é trapezio_gen(f,a,b,n)

Exemplo: Calcule o valor aproximado da integral definida de $x^{(1/2)}$ entre 1 e 4 usando a regra dos trapézios generalizada para 6 subintervalos e determine um limitante para o erro:

Solução: De fato, de antemão, definamos $f$, a, b e n:

```
def f(x): return sqrt(x)
a = 1
b = 4
n = 6
```

Fazendo uso do procedimento trapezio_gen, temos:

```
trapezio_gen(f(x), a, b, n)
```

Por fim, o gráfico de $f(x)$ com os trapézios é dado por:

```
fig, ax = plt.subplots()
z = np.arange(a,b+0.001,0.001)

y = lambdify(x, f(x), "numpy")

pontos = [[a,b],[y(a), y(b)]]

ax.fill_between([a,b],pontos[1], color="red")
ax.plot(z,y(z), "black")
ax.plot(pontos[0],pontos[1])

ax.grid()
plugins.connect(fig, plugins.MousePosition(fontsize=14))

mpld3.display()
```

## 2. Regra de Simpson

###  2.1 Regra $\frac{1}{3}$ de Simpson

Estudaremos duas situações:

(a) Quando f é dada analiticamente.

Neste caso, o procedimento é simpson13(f,a,b,n)

Exemplo: Usando a regra $\frac{1}{3}$ de Simpson para 4 subintervalos, calcule o valor aproximado da integral definida de $x*e^x+1$ no intervalo
$[0,3]$ e determine um limitante superior para o erro.

Solução: Definamos $f(x)$, a, b e n:

```
def f(x): return x*exp(x)+1
a = 0
b = 3
n = 4
```

Assim, a integral definida pela regra $\frac{1}{3}$ de Simpson é

```
simpson13(f(x), a, b, n)
```

Podemos exibir as parábolas interpoladoras juntamente com $f(x)$:

```
xk = np.linspace(a,b,n+1)
fx = []
y = []
for j in range(n-1):
    for i in range(len(xk)):
        fx.append(f(x).subs(x,xk[i]))
    A,B,C = calc_parabola_vertex(xk[j],fx[j],xk[j+1],fx[j+1],xk[j+2],fx[j+2])
    y.append(A*x**2 + B*x + C)

fig, ax = plt.subplots()
z = []
w = []
for i in range(n-1):
    z.append(np.arange(xk[i],xk[i+2]+0.001,0.001))

    w.append(lambdify(x, y[i], "numpy"))
    ax.plot(z[i],w[i](z[i]), label="Parabola "+str(i+1))

c = np.arange(a,b+0.001,0.001)
m = []
for i in range(len(c)):
    m.append(f(x).subs(x,c[i]))
ax.plot(c,m, label="f(x)")
ax.legend()
ax.grid()
plugins.connect(fig, plugins.MousePosition(fontsize=14))

mpld3.display()
```

(b) Quando f é dada por um conjunto de pontos discretos (tabela):

Neste caso, usaremos: simpson_tabela13(a,b,n,y)

Exemplo: Considere $f(x)$ dada pela tabela:

| x    | 0    | 1    | 2    | 3    | 4    | 5    | 6    |
|------|------|------|------|------|------|------|------|
| f(x) | 0.21 | 0.32 | 0.42 | 0.51 | 0.82 | 0.91 | 1.12 |

Do exposto, calcule a integral de $f(x)$ entre 0 e 6.

Solução: Primeiramente, declaramos os valores de a, b, n e a tabela citada:

```
a = 0
b = 6
n = 6
y = [0.21,0.32,0.42,0.51,0.82,0.91,1.12]
```

Deste modo, temos que o valor da integral é aproxidamente:

```
simpson_tabela13(a,b,n,y)
```

    O valor da integral é aproximadamente 3.59

### 2.2 Regra $\frac{3}{8}$ de Simpson

Como na regra $\frac{1}{3}$ de Simpson, estudaremos duas situações:

(a) Quando f é dada analiticamente.

Neste caso, o procedimento é para calcular a integral definida é simpson38(f,a,b,n)

Exemplo: Calcule o valor aproximado da integral de $ln(x+9)$ entre 1 e 7 usando a regra $\frac{3}{8} de Simpson e determine um limitante superior
para o erro para 6 subintervalos:

Solução: Declarando os valores de $f(x)$, a, b e n, temos:

```
def f(x): return log(x+9)
a = 1
b = 7
n = 6
```

Assim, empregando comando para o cálculo da integral, resulta:

```
simpson38(f(x), a,b,n)
```

(b) Quando f é dada por um conjunto de pontos discretos (tabela):

Neste caso, usaremos: simpson_tabela38(a,b,n,y)

Exemplo: Considere $f(x)$ dada pela tabela:

| x    | 0    | 1    | 2    | 3    | 4    | 5    | 6    |
|------|------|------|------|------|------|------|------|
| f(x) | 0.21 | 0.32 | 0.42 | 0.51 | 0.82 | 0.91 | 1.12 |

Do exposto, calcule a integral de $f(x)$ entre 0 e 6.

Solução: Primeiramente, declaramos os valores de a, b, n e a tabela citada:

```
a = 0
b = 6
n = 6
y = [0.21,0.32,0.42,0.51,0.82,0.91,1.12]
```

Deste modo, temos que o valor da integral é aproxidamente:

```
simpson_tabela38(a,b,n,y)
```

## 3. Quadratura de Gauss

O procedimento aqui é quadGauss(f,a,b,n)

onde $[a,b]$ é um intervalo arbitrário e n = 2,3,4,5 (veja páginas 3-5 e exercício 7 da lista
5).

Exemplo: Obtenha uma aproximação para a integral da função $f(x)=e^{-x^2}$ de 1 a 1.5,
utilizando a quadratura de Gauss com n = 5.

Solução: Primeiramente, definamos $f(x)$:

```
def f(x): return exp(-x**2)
a = 1
b = 1.5
n = 5
```

Logo, a quadratura de Gauss nos fornece:

```
quadGauss(f(x), a, b, n)
```


# Examples for use of `Eq`

```python
from symbolic_equation import Eq
```

## Solving a simple system of equations

https://pythonforundergradengineers.com/sympy-expressions-and-equations.html#Defining-Equations-in-Sympy

```python
from sympy import symbols
```

```python
x, y = symbols('x y')
```

```python
eq1 = Eq(2*x - y - 1, tag='I')
eq1
```

\begin{equation}
  2 x - y - 1 = 0
\tag{I}\end{equation}

```python
eq2 = Eq(x + y - 5, tag='II')
eq2
```

\begin{equation}
  x + y - 5 = 0
\tag{II}\end{equation}

```python
eq_y = (
    (eq1 - 2 * eq2).tag("I - 2 II")
    .transform(lambda eq: eq - 9)
    .transform(lambda eq: eq / (-3)).tag('y')
)
eq_y
```

\begin{align}
  9 - 3 y &= 0\tag{I - 2 II}\\
  - 3 y &= -9\\
  y &= 3
\tag{y}\end{align}

```python
eq_x = (
    eq1.apply_to_lhs('subs', eq_y.as_dict).reset().tag(r'$y$ in I')
    .transform(lambda eq: eq / 2)
    .transform(lambda eq: eq + 2).tag('x')
)
eq_x
```

\begin{align}
  2 x - 4 &= 0\tag{$y$ in I}\\
  x - 2 &= 0\\
  x &= 2
\tag{x}\end{align}

Alternatively, we could let `sympy` solve the equation directly:

```python
from sympy import solve
```

```python
sol = solve((eq1, eq2),(x, y))
```

```python
sol
```

    {x: 2, y: 3}

## Proof of Euler's equation (to 6th order)

https://austinrochford.com/posts/2014-02-05-eulers-formula-sympy.html

```python
from sympy import exp, sin, cos, I
```

```python
θ = symbols('theta', real=True)
```

```python
n = 6
```

```python
eq_euler = (
    Eq(exp(I * θ), cos(θ) + I * sin(θ))
    .apply('subs', {cos(θ): cos(θ).series(n=n)})
    .apply('subs', {sin(θ): sin(θ).series(n=n)})
    .apply_to_rhs('expand').amend(previous_lines=2)
    .apply_to_lhs('series', n=n)
)
eq_euler
```

\begin{align}
  e^{i \theta} &= i \sin{\left(\theta \right)} + \cos{\left(\theta \right)}\\
   &= 1 + i \theta - \frac{\theta^{2}}{2} - \frac{i \theta^{3}}{6} + \frac{\theta^{4}}{24} + \frac{i \theta^{5}}{120} + O\left(\theta^{6}\right)\\
  1 + i \theta - \frac{\theta^{2}}{2} - \frac{i \theta^{3}}{6} + \frac{\theta^{4}}{24} + \frac{i \theta^{5}}{120} + O\left(\theta^{6}\right) &= 1 + i \theta - \frac{\theta^{2}}{2} - \frac{i \theta^{3}}{6} + \frac{\theta^{4}}{24} + \frac{i \theta^{5}}{120} + O\left(\theta^{6}\right)
\end{align}

```python
eq_euler.lhs - eq_euler.rhs
```

$\displaystyle O\left(\theta^{6}\right)$