### Gradient Descent Algorithm

\begin{align}
w(0) & = \text{some initial value for} \; \tau = 1, 2,\dots \; w(\tau) = w(\tau-1) - \eta \nabla E(w(\tau-1))
\end{align}

| Symbol           | Meaning                                                        |
|:-----------------|:---------------------------------------------------------------|
| $$\tau$$         | Iteration number                                               |
| $$w$$            | Coefficients vector                                            |
| $$w(\tau)$$      | Coefficients vector at **$\tau$** iteration                    |
| $$\eta$$         | Learning rate                                                  |
| $$\nabla$$       | Gradient function                                              |
| $$E(w(\tau-1))$$ | Error function with **w** coefficients at **$\tau$** iteration |

---

\begin{align}
\theta = \theta - \eta \nabla_{\theta} J(\theta)
\end{align}

```python
%matplotlib inline
import matplotlib.pylab as plt

import time
from IPython import display
import numpy as np

#y = np.array([7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73])
y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
#y = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74])
x = np.array([10., 8., 13., 9., 11., 14., 6., 4., 12., 7., 5.])
N = len(x)

# Design matrix
A = np.vstack((np.ones(N), x)).T
# A = np.vstack((np.ones(N), x, x**2, x**3)).T

# Learning rate
eta = 0.01

# Initial parameters
w = np.array([2., 1.,])
# w = np.array([2., 1., 0.1, 0.001])

# Chart
f = A.dot(w)

# loop
for epoch in range(10):
    # Error
    err = y - A.dot(w)

    # Total error
    E = np.sum(err ** 2) / N

    # Gradient
    dE = -2. * A.T.dot(err) / N

    if epoch % 1 == 0:
        print(epoch, ':', E)

    # Perfom one descent step
    w = w - eta * dE

```

    0 : 15.9938818182
    1 : 11.6346546383
    2 : 8.57455118197
    3 : 6.4263659935
    4 : 4.91830088281
    5 : 3.85956758638
    6 : 3.11624261595
    7 : 2.5943186121
    8 : 2.2278072717
    9 : 1.97038821883


# Matrix as a Function & Plotting Vectors
** October 2017 **

** Andrew Riberio @ [AndrewRib.com](http://www.andrewrib.com) **

One of the fundamental concepts that is key to understanding and working with linear algebra is the notion that a matrix represents a linear transformation aka a linear function. This notebook will show you why that is.

**Questions explored by this notebook**
* Why is linear algebra called 'linear' algebra.
* How does a matrix represent a linear function?
* What is the geometric interpretation of linear funcitons?
* How do we plot vectors using matplotlib?

** Resources **
* https://matplotlib.org/devdocs/api/_as_gen/matplotlib.axes.Axes.quiver.html

## Libraries

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
```

## Matrices as Linear Functions

A matrix defines a linear function. A matrix multiplied by a vector is equivalent of applying a linear function. A matrix multiplied by another matrix is equivalent to function composition. Below we have a matrix *a* and it's equivalent function defintion *fn*. We show here how they are equivalent. See the notebook *Linear Vs. Non-Linear Functions* for why we can define a linear function as a matrix.

```python
sp.init_printing()
def fn(x,y):
    return sp.Matrix( [2*x + 1*y, (1*x+2*y)] )

a = sp.Matrix([[2,1],[1,2]]) # 2 * 2
b = sp.Matrix([2,2]) # 2 * 1
a
```

```python
a * b
```

```python
fn(2,2)
```

## Plotting Vectors

Due to linear algebra having such a strong geometric interpretation, it's helpful to have the tools to visualize what's going on. Here we will discuss how to plot vectors using matplotlib.

```python
# [start_loc_x, start_loc_y, x_magintute, y_magintude]
vects = np.array([[0, 2, 2, 0],
                  [0, 2, 0, 2],
                  [0, 4, 2, 0],
                  [2, 2, 0, 2]])

# Unpack vects into their respective tupules.
X_LOC, Y_LOC, X_MAG, Y_MAG = zip(*vects)

plt.figure()

# GCA = Get current axes.
ax = plt.gca()
ax.quiver(X_LOC, Y_LOC, X_MAG, Y_MAG, angles='xy', scale_units='xy', scale=1)

# Set the scale of the tics on the x and y axes.
ax.set_xlim([-1, 10])
ax.set_ylim([-1, 10])

plt.show()
```

## Linear Transformations

In this section we will define a rectagle by a series of points, then we will define a linear transformation, a matrix, that transforms these points, thereby transforming the defined rectangle. First we will start by defining a function, *plotRect*, which plots a rectangle.

```python
def plotRect(lb,lt,rt,rb):

    plt.figure(figsize=(8, 8))
    ax = plt.gca()

    # Setup grid and axes.
    ax.axhline(linewidth=2.0, color="black")
    ax.axvline(linewidth=2.0, color="black")
    ax.grid(color='b', linestyle='-', linewidth=1)

    ax.set_yticks(range(-10,11))
    ax.set_xticks(range(-10,11))

    # Set the scale of the tics on the x and y axes.
    ax.set_xlim([-10, 10])
    ax.set_ylim([-10, 10])

    # Plot square based on points.
    # [x1,x2],[y1,y2]
    plt.plot([lb[0],lt[0]], [lb[1],lt[1]], 'k-', lw=2)
    plt.plot([lt[0],rt[0]], [lt[1],rt[1]], 'k-', lw=2)
    plt.plot([rt[0],rb[0]], [rt[1],rb[1]], 'k-', lw=2)
    plt.plot([rb[0],lb[0]], [rb[1],lb[1]], 'k-', lw=2)

    plt.show()
```

Now we will define a rectangle by a series of points, (lb,lt,rt,br), and plot it using our above function.

```python
# Left bottom point
lb = sp.Matrix([0,0])
# Left top point
lt = sp.Matrix([0,4])
# Right top point
rt = sp.Matrix([4,4])
# Bottom right point

rb = sp.Matrix([4,0])

plotRect(lb,lt,rt,rb)
```

Let's definine a funciton that applies a 2 * 2 matrix to four points that we will use to transform the rectangle defined by the points above.

```python
def transform(m,lb,lt,rt,rb):
    lb_o = m * lb
    lt_o = m * lt
    rt_o = m * rt
    rb_o = m * rb

    return [lb_o,lt_o,rt_o,rb_o]
```

We will now define a matrix which we we will use to transform the rectangle.

```python
# M is an identity matrix.
m = sp.Matrix([[1,0],[0,1]])
m
```

```python
lb_t,lt_t,rt_t,rb_t = transform(m,lb,lt,rt,rb)
plotRect(lb_t,lt_t,rt_t,rb_t)
```

As we see, the identity matrix preserves the original size of the square. Let's now play around with different matricies to see how they spatially transform the rectangle.

```python
m = sp.Matrix([[2,0],[0,2]])
m
```

```python
lb_t,lt_t,rt_t,rb_t = transform(m,lb,lt,rt,rb)
plotRect(lb_t,lt_t,rt_t,rb_t)
```

As we see, changing the values on the diagonal scales the rectangle.

```python
m = sp.Matrix([[1,0],[0,-1]])
m
```

```python
lb_t,lt_t,rt_t,rb_t = transform(m,lb,lt,rt,rb)
plotRect(lb_t,lt_t,rt_t,rb_t)
```

As we see, changing the signs on the diagonal translates the rectangle over an axis.

```python
m = sp.Matrix([[1,0],[1,1]])
m
```

```python
lb_t,lt_t,rt_t,rb_t = transform(m,lb,lt,rt,rb)
plotRect(lb_t,lt_t,rt_t,rb_t)
```

When we start changing the values on the counterdiagonal, we start skewing the rectangle. By changing the values on the diagonal and counterdiagonal, we can perform any linear transformation to our square. If we apply successive linear transformations, affine transformations, we can do things like rotation. Play around with the values for the matrix to see how different values result in different linear transformations of the rectangle.

Notebooks/Matrix as a Function & Plotting Vectors.ipynb


```python
import numpy as np
import scipy.linalg as la
from sklearn.linear_model import Lasso, LinearRegression
from time import time
import matplotlib.pyplot as plt
from tqdm import tqdm
```

## Plain Algorithm

```python
def sign(x):
    """
    If x is nonzero, return x/|x|, else return 0
    """
    if x>0:
        return 1
    if x<0:
        return -1
    if x==0:
        return 0

def pos(x):
    """
    Return maximum between x and 0
    """
    return max(x, 0)

def BIC(x, Y, lambd, var_ols):
    """
    Compute BIC of certain penalty paramter choices
    """
    N = Y.shape[0]
    LASSO = Lasso(alpha=lambd, fit_intercept=False, normalize=False).fit(x, Y)
    Y_hat = LASSO.predict(x)
    df = sum(LASSO.coef_ != 0)
    err = sum((Y_hat - Y) ** 2)
    return err/(var_ols*N) + np.log(N)*df/N

def find_lambda(X, u, w_2):
    """
    Find the lambda with minimal BIC
    """
    n = X.shape[0]
    d = X.shape[1]
    Y = X.ravel("F")
    x = np.kron(np.eye(d), u.reshape(-1, 1))/w_2
    Y_ols = LinearRegression(fit_intercept=False).fit(x, Y).predict(x)
    var_ols = ((Y-Y_ols)**2).mean()
    # find approximate range of minimum
    lambdas = np.exp(np.linspace(3, 3.5, 50))
    BICS = list(map(lambda lambd: BIC(x, Y, lambd, var_ols), lambdas))
    index = np.argmin(BICS)
    # precise search
    s_lambdas = np.exp(np.linspace(np.log(lambdas[index-1]), np.log(lambdas[index+1]), 100))
    s_BICS = list(map(lambda lambd: BIC(x, Y, lambd, var_ols), s_lambdas))
    return s_lambdas[np.argmin(s_BICS)]

def ssvd(X, gamma_1 = 2, gamma_2 = 2, tol=1e-10, iters=10000):
    """
    Sparse SVD algorithm
    """
    # shapes of X
    n = X.shape[0]
    d = X.shape[1]
    # find starting vectors
    U, S, V = la.svd(X)
    u_old = U[:, 0]
    v_old = V[0, :]
    for i in range(iters):
        # update v
        v_sim = np.zeros(d) ## v_sim initialization
        v_sh = X.T @ u_old ## v with sim and hat
        w_2 = np.vectorize(lambda x: 0 if x==0 else x**(-gamma_2))(abs(v_sh)) ## find w_2
        lambda_v = find_lambda(X, u_old, w_2)
        for j in range(d): ## LASSO
            v_sim[j] = sign(v_sh[j])*pos(abs(v_sh[j])-lambda_v*w_2[j]/2)
        v_new = v_sim / la.norm(v_sim) ## normalization
        # update u
        u_sim = np.zeros(n) ## v_sim initialization
        u_sh = X @ v_new ## v with sim and hat
        w_1 = np.vectorize(lambda x: 0 if x==0 else x**(-gamma_1))(abs(u_sh)) ## find w_1
        lambda_u = find_lambda(X.T, v_new, w_1)
        for j in range(n): ## LASSO
            u_sim[j] = sign(u_sh[j])*pos(abs(u_sh[j])-lambda_u*w_1[j]/2)
        u_new = u_sim / la.norm(u_sim) ## normalization
        # check convergence
        if la.norm(u_new - u_old)<tol and la.norm(v_new - v_old)<tol:
            break
        else:
            u_old = u_new
            v_old = v_new
    return u_new, v_new, u_new.T @ X @ v_new
```

## Optimized Algorithm

Starting from the plain sparse SVD algorithm, we finished two types of modification:

1. The original algorithm updates each element of $v_{new}$ and $u_{new}$, the modified version vectorized this process.

2. When finding the penalty paramter $\lambda_u$ and $\lambda_v$, the plain version conducted a rough search followed by a precise search. However, when implementing this algorithm, we found that the rough search is enough for it to find the appropriate $\lambda$. So we deleted the precise search procedure, which improved the speed a lot but did not change the accuracy.

```python
def sign_new(x):
    """
    vectorized version of finding signals
    """
    y = x.copy()
    y[y!=0] = y[y!=0]/abs(y[y!=0])
    return y

def pos_new(x):
    """
    vectorized version of finding max(x, 0)
    """
    y = x.copy()
    y[y<0] = 0
    return y

def BIC_new(x, Y, lambd, var_ols):
    """
    vectorized version of computing BIC
    """
    N = Y.shape[0]
    beta_ols = la.inv(x.T@x) @ x.T @ Y
    beta_lasso = beta_ols * pos_new(1-lambd/beta_ols)
    Y_hat = x @ beta_lasso
    df = sum(beta_lasso != 0)
    err = sum((Y_hat - Y) ** 2)
    return err/(var_ols*N) + np.log(N)*df/N

def find_lambda_new(X, u, w_2):
    """
    vectorized version of finding lambda without precise search
    """
    n = X.shape[0]
    d = X.shape[1]
    Y = X.ravel("F")
    x = np.kron(np.eye(d), u.reshape(-1, 1))/w_2
    Y_ols = x @ la.inv(x.T @ x) @ x.T @ Y
    var_ols = ((Y-Y_ols)**2).mean()

    lambdas = np.exp(np.linspace(3, 3.5, 50))
    BICS = list(map(lambda lambd: BIC(x, Y, lambd, var_ols), lambdas))
    return lambdas[np.argmin(BICS)]

def ssvd_new(X, gamma_1 = 2, gamma_2 = 2, tol=1e-10, iters=10000):
    """
    vectorized version of sparse SVD algorithm
    """
    # shapes of X
    n = X.shape[0]
    d = X.shape[1]
    # find starting vectors
    U, S, V = la.svd(X)
    u_old = U[:, 0]
    v_old = V[0, :]
    for i in range(iters):
        # update v
        v_sim = np.zeros(d) ## v_sim initialization
        v_sh = X.T @ u_old ## v with sim and hat
        w_2 = np.vectorize(lambda x: 0 if x==0 else x**(-gamma_2))(abs(v_sh)) ## find w_2
        lambda_v = find_lambda_new(X, u_old, w_2)
        v_sim = sign_new(v_sh)*pos_new(abs(v_sh)-lambda_v*w_2/2)
        v_new = v_sim / la.norm(v_sim) ## normalization
        # update u
        u_sim = np.zeros(n) ## v_sim initialization
        u_sh = X @ v_new ## v with sim and hat
        w_1 = np.vectorize(lambda x: 0 if x==0 else x**(-gamma_1))(abs(u_sh)) ## find w_1
        lambda_u = find_lambda_new(X.T, v_new, w_1)
        u_sim = sign_new(u_sh)*pos_new(abs(u_sh)-lambda_u*w_1/2)
        u_new = u_sim / la.norm(u_sim) ## normalization
        # check convergence
        if la.norm(u_new - u_old)<tol and la.norm(v_new - v_old)<tol:
            break
        else:
            u_old = u_new
            v_old = v_new
    return u_new, v_new, u_new.T @ X @ v_new
```

### Speed Comparison

```python
u = np.array([10, 9, 8, 7, 6, 5, 4, 3]+[2]*17+[0]*75).reshape(-1, 1)
u = u / la.norm(u)
v = np.array([10, -10, 8, -8, 5, -5]+[3]*5+[-3]*5+[0]*34).reshape(1, -1)
v = v / la.norm(v)
X1_ = 50 * u @ v
```

```python
%timeit -n3 -r3 X1 = X1_ + np.random.normal(size=(100, 50)); ssvd(X1)
```

    10.7 s ± 376 ms per loop (mean ± std. dev. of 3 runs, 3 loops each)

```python
%timeit -n3 -r3 X1 = X1_ + np.random.normal(size=(100, 50)); ssvd_new(X1)
```

    5.84 s ± 180 ms per loop (mean ± std. dev. of 3 runs, 3 loops each)

## Applications to simulated datasets

In this section, we report two simulated study of sparse SVD. For both part, we set $\gamma_1=\gamma_2=2$ and update $\lambda_u$ and $\lambda_v$ in each step with respect to BIC.

### Case 1: rank-1 matrix approximation

Consider a one-rank true signal matrix $X^*=suv^T$ with

$\tilde{u}=[10, 9, 8, 7, 6, 5, 4, 3, r(2, 17), r(0, 75)]^T, u=\tilde{u}/||\tilde{u}||$,

$\tilde{v}=[10, −10, 8, −8, 5, −5, r(3, 5), r(−3, 5), r(0, 34)]^T, v=\tilde{v}/||\tilde{v}||$,

$s=50$

A data matrix $X$ is randomly created by adding $X^*$ and a noise matrix $\epsilon$, where each element is independently generated from standard normal distribution. This process is repeated 100 times. We mainly assessed the model's accuracy of distinguishing zero and nonzero entity in $u$ and $v$.

```python
num_zeros_u = []
cor_zeros_u = []
cor_non_zeros_u = []
mis_rate_u = []
num_zeros_v = []
cor_zeros_v = []
cor_non_zeros_v = []
mis_rate_v = []

np.random.seed(seed=0)
start = time()
iters = 100
with tqdm(total=iters) as pbar:
    for i in range(iters):
        X1 = X1_ + np.random.normal(size=(100, 50))
        u_new, v_new, s_new = ssvd_new(X1)

        num_zeros_u.append((u_new==0).sum())
        cor_zeros_u.append(np.logical_and(u_new.reshape(-1, 1)==0, u==0).sum()/75)
        cor_non_zeros_u.append(np.logical_and(u_new.reshape(-1, 1)!=0, u!=0).sum()/25)
        mis_rate_u.append(np.logical_xor(u_new.reshape(-1, 1)==0, u==0).sum()/100)

        num_zeros_v.append((v_new==0).sum())
        cor_zeros_v.append(np.logical_and(v_new.reshape(1, -1)==0, v==0).sum()/34)
        cor_non_zeros_v.append(np.logical_and(v_new.reshape(1, -1)!=0, v!=0).sum()/16)
        mis_rate_v.append(np.logical_xor(v_new.reshape(1, -1)==0, v==0).sum()/50)

        pbar.update(1)

print("Cost time: {:.2f}s per loop\n".format((time()-start)/iters))

avg = lambda x: sum(x)/iters

print("Number of zeros in u:", avg(num_zeros_u))
print("Correctly Labeled zero in u:", avg(cor_zeros_u))
print("Correctly Labeled non-zero in u:", avg(cor_non_zeros_u))
print("Misspecification in u:", avg(mis_rate_u), "\n")

print("Number of zeros in v:", avg(num_zeros_v))
print("Correctly Labeled zero in v:", avg(cor_zeros_v))
print("Correctly Labeled non-zero in v:", avg(cor_non_zeros_v))
print("Misspecification in v:", avg(mis_rate_v), "\n")
```

    100%|██████████| 100/100 [12:06<00:00,  8.12s/it]

    Cost time: 7.26s per loop

    Number of zeros in u: 72.87
    Correctly Labeled zero in u: 0.9705333333333331
    Correctly Labeled non-zero in u: 0.9967999999999999
    Misspecification in u: 0.0229

    Number of zeros in v: 32.8
    Correctly Labeled zero in v: 0.9647058823529397
    Correctly Labeled non-zero in v: 1.0
    Misspecification in v: 0.024000000000000018

SSVD performs pretty well in this rank-1 case. For example, in terms of correctly identifying the true zero and nonzero entries, on average it only misclassifies 1.5% and 1.0% of the entries in u and v, respectively.

### Case 2: rank-2 matrix approximation

Consider a true signal matrix $X^*$ with shape $50\times100$, where each element $X^*_{ij}=T_{ij}I_{(|T_{ij}|>1)}$ with

$$
T_{ij}=\begin{equation}
\left\{
\begin{array}{lr}
[26^2-(i-25)^2-(j-50)^2]/100 &\text{if } 26\leq j\leq 75\\
0 &\text{otherwise}
\end{array}
\right.
\end{equation}
$$

$X^*$ is almost rank-2 in that its eigenvalues are almost zero except the first two. The figure below shows the structure of $X^*$, where the positive entries are red, the negative ones are blue, and the zeros are white.

A data matrix $X$ is randomly created by adding $X^*$ and a noise matrix $\epsilon$, where each element is independently generated from standard normal distribution. This process is repeated 100 times. We mainly assessed the model's accuracy of distinguishing zero and nonzero entity in $u$ and $v$ as well as the distance of fitted matrix to the original matrix as the number of layers increases.

```python
X2_ = np.zeros(shape=(50, 100))
for i in range(50):
    for j in range(25, 75):
        T = (24**2-(i-24)**2-(j-49)**2)/100
        if abs(T) > 1:
            X2_[i, j] = T
```

```python
plt.figure(figsize=(10, 6))
plt.imshow(X2_, cmap='RdBu_r')
plt.colorbar()
plt.savefig('rank-2-true-signal.jpg')
```

```python
def ssvd_approx(X, layer=1):
    residual = X
    X_fit = np.zeros(X.shape)
    acc = []
    f_dist = []
    for i in range(layer):
        u_new, v_new, s = ssvd_new(residual)
        X_fit += s * u_new.reshape(-1, 1) @ v_new.reshape(1, -1)
        residual = X - X_fit
        acc.append(1 - np.logical_xor(X_fit == 0, X2_ == 0).sum()/(X.shape[0]*X.shape[1]))
        f_dist.append(np.sqrt(((X2_ - X_fit)**2).sum()))
    return X_fit, acc, f_dist
```

```python
np.random.seed(1)
X2 = X2_ + np.random.normal(size=(50, 100))
X_fit, acc, f_dist = ssvd_approx(X2, layer=10)
```

```python
plt.plot(range(1, 11), acc, color='red', label='SSVD')
plt.legend()
plt.title('Correctly classified entries')
plt.xlabel('Number of Layers')
plt.xticks(ticks=range(1, 11), labels=range(1, 11))
plt.savefig('acc-ssvd.jpg')
plt.show()
```

```python
plt.plot(range(1, 11), f_dist, color='red', label='SSVD')
plt.legend()
plt.title('Frobenius distance')
plt.xlabel('Number of Layers')
plt.xticks(ticks=range(1, 11), labels=range(1, 11))
plt.savefig('f-dist-ssvd.jpg')
plt.show()
```

In this case, the Sparse SVD algorithm gives the best approximation when the number of layers is 2. After the minimun point, this model starts to overfit the data.

```python

```

## Applications to real datasets

The real datasets "Lung Cancer data" used in the paper could not be found. So we provide another example of the real-world data that's not in the paper.

This dataset contains credit card transactions made by European cardholders in September 2013. Some of them are fraud transactions. For privacy reason, this dataset only has numerical inputs and we don't know exactly what each variable represents.

For convenience, we subset only first 100 rows of records to make it as the test matrix $X$. The shape of $X$ is 100 by 28. Each row represents each record of transaction, while each column represents a feature.

```python
import pandas as pd
```

```python
df_real = pd.read_csv("real_data.csv")
df = df_real.drop(['Time', 'Amount', 'Class'], axis = 1)[0:100]
X = np.array(df)
X.shape
```

    (100, 28)

```python
def svd(X):
    U, S, V = la.svd(X)
    return U[:, 0], V[0, :]
```

```python
u = svd(X)[0].reshape(-1, 1)
u = np.round(u / la.norm(u), 1)
v = svd(X)[1].reshape(1, -1)
v = np.round(v / la.norm(v), 1)
X1_ = 28 * u @ v
```

```python
np.count_nonzero(u)
```

    67

```python
np.count_nonzero(v)
```

    15

```python
num_zeros_u = []
cor_zeros_u = []
cor_non_zeros_u = []
mis_rate_u = []
num_zeros_v = []
cor_zeros_v = []
cor_non_zeros_v = []
mis_rate_v = []

np.random.seed(seed=0)
start = time()
iters = 100
with tqdm(total=iters) as pbar:
    for i in range(iters):
        X1 = X1_ + np.random.normal(size=(100, 28))
        u_new, v_new, s_new = ssvd_new(X1)

        num_zeros_u.append((u_new==0).sum())
        cor_zeros_u.append(np.logical_and(u_new.reshape(-1, 1)==0, u==0).sum()/33)
        cor_non_zeros_u.append(np.logical_and(u_new.reshape(-1, 1)!=0, u!=0).sum()/67)
        mis_rate_u.append(np.logical_xor(u_new.reshape(-1, 1)==0, u==0).sum()/100)

        num_zeros_v.append((v_new==0).sum())
        cor_zeros_v.append(np.logical_and(v_new.reshape(1, -1)==0, v==0).sum()/13)
        cor_non_zeros_v.append(np.logical_and(v_new.reshape(1, -1)!=0, v!=0).sum()/15)
        mis_rate_v.append(np.logical_xor(v_new.reshape(1, -1)==0, v==0).sum()/28)

        pbar.update(1)

print("Cost time: {:.2f}s per loop\n".format((time()-start)/iters))

avg = lambda x: sum(x)/iters

print("Number of zeros in u:", avg(num_zeros_u))
print("Correctly Labeled zero in u:", avg(cor_zeros_u))
print("Correctly Labeled non-zero in u:", avg(cor_non_zeros_u))
print("Misspecification in u:", avg(mis_rate_u), "\n")

print("Number of zeros in v:", avg(num_zeros_v))
print("Correctly Labeled zero in v:", avg(cor_zeros_v))
print("Correctly Labeled non-zero in v:", avg(cor_non_zeros_v))
print("Misspecification in v:", avg(mis_rate_v), "\n")
```

    100%|██████████| 100/100 [09:14<00:00,  5.03s/it]

    Cost time: 5.54s per loop

    Number of zeros in u: 45.07
    Correctly Labeled zero in u: 0.9739393939393933
    Correctly Labeled non-zero in u: 0.8070149253731347
    Misspecification in u: 0.13790000000000008

    Number of zeros in v: 15.55
    Correctly Labeled zero in v: 0.9676923076923072
    Correctly Labeled non-zero in v: 0.8019999999999993
    Misspecification in v: 0.12107142857142858

Following the same testing procedure as we did in rank-1 approximationn in the simulated datasets, we found that the performance is good, but not as good as what we have in the simulated datasets. The misspecification rates in u and v are about 0.14 and 0.12. We could see that this algorithm has a high accuracy of identifying 0 in both vectors.

When there're more 0 in both u and v, then it looks that this algorithm has a better performance in terms of labeling non-zero elements.

## Comparative analysis with competing algorithms

In this section, We compared Sparse SVD algorithm's performance in simulated datasets with two competing methods, which are SVD and Sparse PCA algorithm.

### Case 1

### SPCA

```python
def h(y, lambd):
    return y if abs(y) > lambd else 0

def spca(X, lambd=1, tol=1e-10, iters=10000):
    # shapes of X
    n = X.shape[0]
    d = X.shape[1]
    # find starting vectors
    U, S, V = la.svd(X)
    u_old = U[:, 0]
    v_old = V[0, :] * S[0]
    for i in range(iters):
        # update v
        v_new = np.zeros(d)
        for j in range(d):
            v_new[j] = h((X.T @ u_old)[j], lambd)
        u_new = (X @ v_new) / la.norm(X @ v_new)
        # check convergence
        if la.norm(u_new - u_old)<tol:
            break
        else:
            u_old = u_new
            v_old = v_new
    return u_new, v_new
```

```python
u = np.array([10, 9, 8, 7, 6, 5, 4, 3]+[2]*17+[0]*75).reshape(-1, 1)
u = u / la.norm(u)
v = np.array([10, -10, 8, -8, 5, -5]+[3]*5+[-3]*5+[0]*34).reshape(1, -1)
v = v / la.norm(v)
X1_ = 50 * u @ v
```

```python
num_zeros_u_spcav = []
cor_zeros_u_spcav = []
cor_non_zeros_u_spcav = []
mis_rate_u_spcav = []
num_zeros_v_spcav = []
cor_zeros_v_spcav = []
cor_non_zeros_v_spcav = []
mis_rate_v_spcav = []

iters=100
np.random.seed(2)

with tqdm(total=iters) as pbar:
    for i in range(iters):
        X1 = X1_ + np.random.normal(size=(100, 50))
        u_new, v_new = spca(X1, 5)

        num_zeros_u_spcav.append((u_new==0).sum())
        cor_zeros_u_spcav.append(np.logical_and(u_new.reshape(-1, 1)==0, u==0).sum()/75)
        cor_non_zeros_u_spcav.append(np.logical_and(u_new.reshape(-1, 1)!=0, u!=0).sum()/25)
        mis_rate_u_spcav.append(np.logical_xor(u_new.reshape(-1, 1)==0, u==0).sum()/100)

        num_zeros_v_spcav.append((v_new==0).sum())
        cor_zeros_v_spcav.append(np.logical_and(v_new.reshape(1, -1)==0, v==0).sum()/34)
        cor_non_zeros_v_spcav.append(np.logical_and(v_new.reshape(1, -1)!=0, v!=0).sum()/16)
        mis_rate_v_spcav.append(np.logical_xor(v_new.reshape(1, -1)==0, v==0).sum()/50)

        pbar.update(1)

print("Result of SPCA(v):")

avg = lambda x: sum(x)/iters

print("Number of zeros in u:", avg(num_zeros_u_spcav))
print("Correctly Labeled zero in u:", avg(cor_zeros_u_spcav))
print("Correctly Labeled non-zero in u:", avg(cor_non_zeros_u_spcav))
print("Misspecification in u:", avg(mis_rate_u_spcav), "\n")

print("Number of zeros in v:", avg(num_zeros_v_spcav))
print("Correctly Labeled zero in v:", avg(cor_zeros_v_spcav))
print("Correctly Labeled non-zero in v:", avg(cor_non_zeros_v_spcav))
print("Misspecification in v:", avg(mis_rate_v_spcav), "\n")
```

    100%|███████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 223.43it/s]

    Result of SPCA(v):
    Number of zeros in u: 0.0
    Correctly Labeled zero in u: 0.0
    Correctly Labeled non-zero in u: 1.0
    Misspecification in u: 0.75

    Number of zeros in v: 34.28
    Correctly Labeled zero in v: 1.0
    Correctly Labeled non-zero in v: 0.9825
    Misspecification in v: 0.005600000000000002

```python
num_zeros_u_spcau = []
cor_zeros_u_spcau = []
cor_non_zeros_u_spcau = []
mis_rate_u_spcau = []
num_zeros_v_spcau = []
cor_zeros_v_spcau = []
cor_non_zeros_v_spcau = []
mis_rate_v_spcau = []

np.random.seed(3)
iters=100

with tqdm(total=iters) as pbar:
    for i in range(iters):
        X1 = X1_ + np.random.normal(size=(100, 50))
        v_new, u_new = spca(X1.T, 3)

        num_zeros_u_spcau.append((u_new==0).sum())
        cor_zeros_u_spcau.append(np.logical_and(u_new.reshape(-1, 1)==0, u==0).sum()/75)
        cor_non_zeros_u_spcau.append(np.logical_and(u_new.reshape(-1, 1)!=0, u!=0).sum()/25)
        mis_rate_u_spcau.append(np.logical_xor(u_new.reshape(-1, 1)==0, u==0).sum()/100)

        num_zeros_v_spcau.append((v_new==0).sum())
        cor_zeros_v_spcau.append(np.logical_and(v_new.reshape(1, -1)==0, v==0).sum()/34)
        cor_non_zeros_v_spcau.append(np.logical_and(v_new.reshape(1, -1)!=0, v!=0).sum()/16)
        mis_rate_v_spcau.append(np.logical_xor(v_new.reshape(1, -1)==0, v==0).sum()/50)

        pbar.update(1)

print("Result of SPCA(u):")

avg = lambda x: sum(x)/iters

print("Number of zeros in u:", avg(num_zeros_u_spcau))
print("Correctly Labeled zero in u:", avg(cor_zeros_u_spcau))
print("Correctly Labeled non-zero in u:", avg(cor_non_zeros_u_spcau))
print("Misspecification in u:", avg(mis_rate_u_spcau), "\n")

print("Number of zeros in v:", avg(num_zeros_v_spcau))
print("Correctly Labeled zero in v:", avg(cor_zeros_v_spcau))
print("Correctly Labeled non-zero in v:", avg(cor_non_zeros_v_spcau))
print("Misspecification in v:", avg(mis_rate_v_spcau), "\n")
```

    100%|███████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 164.70it/s]

    Result of SPCA(u):
    Number of zeros in u: 75.58
    Correctly Labeled zero in u: 0.996933333333333
    Correctly Labeled non-zero in u: 0.9675999999999996
    Misspecification in u: 0.010400000000000006

    Number of zeros in v: 0.0
    Correctly Labeled zero in v: 0.0
    Correctly Labeled non-zero in v: 1.0
    Misspecification in v: 0.6800000000000002

### SVD

```python
def svd(X):
    U, S, V = la.svd(X)
    return U[:, 0], V[0, :]
```

```python
num_zeros_u_svd = []
cor_zeros_u_svd = []
cor_non_zeros_u_svd = []
mis_rate_u_svd = []
num_zeros_v_svd = []
cor_zeros_v_svd = []
cor_non_zeros_v_svd = []
mis_rate_v_svd = []

iters=100

with tqdm(total=iters) as pbar:
    for i in range(iters):
        X1 = X1_ + np.random.normal(size=(100, 50))
        u_new, v_new = svd(X1)

        num_zeros_u_svd.append((u_new==0).sum())
        cor_zeros_u_svd.append(np.logical_and(u_new.reshape(-1, 1)==0, u==0).sum()/75)
        cor_non_zeros_u_svd.append(np.logical_and(u_new.reshape(-1, 1)!=0, u!=0).sum()/25)
        mis_rate_u_svd.append(np.logical_xor(u_new.reshape(-1, 1)==0, u==0).sum()/100)

        num_zeros_v_svd.append((v_new==0).sum())
        cor_zeros_v_svd.append(np.logical_and(v_new.reshape(1, -1)==0, v==0).sum()/34)
        cor_non_zeros_v_svd.append(np.logical_and(v_new.reshape(1, -1)!=0, v!=0).sum()/16)
        mis_rate_v_svd.append(np.logical_xor(v_new.reshape(1, -1)==0, v==0).sum()/50)

        pbar.update(1)

print("Result of SVD:")

avg = lambda x: sum(x)/iters

print("Number of zeros in u:", avg(num_zeros_u_svd))
print("Correctly Labeled zero in u:", avg(cor_zeros_u_svd))
print("Correctly Labeled non-zero in u:", avg(cor_non_zeros_u_svd))
print("Misspecification in u:", avg(mis_rate_u_svd), "\n")

print("Number of zeros in v:", avg(num_zeros_v_svd))
print("Correctly Labeled zero in v:", avg(cor_zeros_v_svd))
print("Correctly Labeled non-zero in v:", avg(cor_non_zeros_v_svd))
print("Misspecification in v:", avg(mis_rate_v_svd), "\n")
```

    100%|███████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 363.27it/s]

    Result of SVD:
    Number of zeros in u: 0.0
    Correctly Labeled zero in u: 0.0
    Correctly Labeled non-zero in u: 1.0
    Misspecification in u: 0.75

    Number of zeros in v: 0.0
    Correctly Labeled zero in v: 0.0
    Correctly Labeled non-zero in v: 1.0
    Misspecification in v: 0.6800000000000002

| | |Avg.# of zeros(true)|Prop. of correctly identified zero|Prop. of correctly identified non-zero|misspecification rate|
|---|---|---|---|---|---|
|SSVD|u|72.87(75)|97.1%|96.5%|2.30%|
| |v|32.80(35)|99.7%|100.0%|2.40%|
|SPCA(u)|u|75.58(75)|99.7%|96.8%|1.04%|
| |v|0(35)|0.0%|100.0%|68%|
|SPCA(v)|u|0(75)|0.0%|100.0%|75%|
| |v|34.28(35)|100.0%|98.3%|0.56%|
|SVD|u|0(75)|0.0%|100.0%|75%|
| |v|0(35)|0.0%|100.0%|68%|

As one can see, SSVD performs much better than the competitors in all four metrics. The classic SVD can only identify non-zero entity without the ability to identify zero entities. For both SPCA(u) and SPCA(v), they can only demonstrate a compelling result in one dimension, either $u$ or $v$. And they perform poorly in the other dimension.

### Case 2

```python
def spca_v_approx(X, lambd=1, layer=1):
    residual = X
    X_fit = np.zeros(X.shape)
    acc = []
    f_dist = []
    for i in range(layer):
        u_new, v_new = spca(residual, lambd)
        X_fit += u_new.reshape(-1, 1) @ v_new.reshape(1, -1)
        residual = X - X_fit
        acc.append(1 - np.logical_xor(X_fit == 0, X2_ == 0).sum()/(X.shape[0]*X.shape[1]))
        f_dist.append(np.sqrt(((X2_ - X_fit)**2).sum()))
        lambd = lambd/1.1
    return X_fit, acc, f_dist

def spca_u_approx(X, lambd=1, layer=1):
    residual = X
    X_fit = np.zeros(X.shape)
    acc = []
    f_dist = []
    for i in range(layer):
        v_new, u_new = spca(residual.T, lambd)
        X_fit += u_new.reshape(-1, 1) @ v_new.reshape(1, -1)
        residual = X - X_fit
        acc.append(1 - np.logical_xor(X_fit == 0, X2_ == 0).sum()/(X.shape[0]*X.shape[1]))
        f_dist.append(np.sqrt(((X2_ - X_fit)**2).sum()))
        lambd = lambd/1.1
    return X_fit, acc, f_dist

def svd_approx(X, layer=1):
    U, S, V = la.svd(X)
    acc = []
    f_dist = []
    for i in range(layer):
        if i == 0:
            X_fit = S[0] * U[:, 0].reshape(-1, 1) @ V[0, :].reshape(1, -1)
        else:
            X_fit = U[:, :(i+1)] @ np.diag(S[:(i+1)]) @ V[:(i+1), :]
        acc.append(1 - np.logical_xor(X_fit == 0, X2_ == 0).sum()/(X.shape[0]*X.shape[1]))
        f_dist.append(np.sqrt(((X2_ - X_fit)**2).sum()))
    return X_fit, acc, f_dist
```

```python
np.random.seed(1)
X2 = X2_ + np.random.normal(size=(50, 100))
X_fit_v, acc_v, f_dist_v = spca_v_approx(X2, layer=10, lambd=5)
X_fit_u, acc_u, f_dist_u = spca_u_approx(X2, layer=10, lambd=3)
X_fit_svd, acc_svd, f_dist_svd = svd_approx(X2, layer=10)
```

```python
plt.plot(range(1, 11), acc, color='red', label='SSVD')
plt.plot(range(1, 11), acc_v, color='blue', label='SPCA(v)')
plt.plot(range(1, 11), acc_u, color='green', label='SPCA(u)')
plt.plot(range(1, 11), acc_svd, color='black', label='SVD')
plt.legend()
plt.title('Correctly classified entries')
plt.xlabel('Number of Layers')
plt.xticks(ticks=range(1, 11), labels=range(1, 11))
plt.savefig('acc-all.jpg')
plt.show()
```

```python
plt.plot(range(1, 11), f_dist, color='red', label='SSVD')
plt.plot(range(1, 11), f_dist_v, color='blue', label='SPCA(v)')
plt.plot(range(1, 11), f_dist_u, color='green', label='SPCA(u)')
plt.plot(range(1, 11), f_dist_svd, color='black', label='SVD')
plt.legend()
plt.title('Frobenius distance')
plt.xlabel('Number of Layers')
plt.xticks(ticks=range(1, 11), labels=range(1, 11))
plt.savefig('f-dist-all.jpg')
plt.show()
```

In rank-2 case, SSVD and SPCA(u) performs well when classifying entities while SVD and SPCA(v) has trouble in detecting the sparse structure. Both SSVD and SPCA(u) arrive at maximum accuracy with 2 layers and start to overfit the data with more layers. The accuracy of SPCA(u) decreses rapidly after it exceeds 7 layers. With respect to Frobenius distance, all four methods perform well in approximating the true signal with minimum distance arrived at 2 layers. SSVD has the smallest distance with true signal. In conclusion, SSVD is the best option among these competing algorithms.

```python

```


#Defining the Problem

Here we will derive the equations of motion for the classic mass-spring-damper system under the influence of gravity. The following figure gives a pictorial description of the problem.

```
from IPython.display import SVG
SVG(filename='mass_spring_damper.svg')
```

Start by loading in the core functionality of both SymPy and Mechanics.

```
import sympy as sym
import sympy.physics.mechanics as me
```

We can make use of the pretty printing of our results by loading SymPy's printing extension.

```
from sympy.physics.vector import init_vprinting
init_vprinting()
```

We'll start by defining the variables we will need for this problem:
- $x$: distance of the particle from the ceiling
- $v$: speed of the particle
- $m$: mass of the particle
- $c$: damping coefficient of the damper
- $k$: stiffness of the spring
- $g$: acceleration due to gravity
- $t$: time

```
x, v = me.dynamicsymbols('x v')
```

```
m, c, k, g, t = sym.symbols('m c k g t')
```

Now, we define a Newtonian reference frame that represents the ceiling which the particle is attached to, $C$.

```
ceiling = me.ReferenceFrame('C')
```

We will need two points, one to represent the original position of the particle which stays fixed in the ceiling frame, $O$, and the second one, $P$ which is aligned with the particle as it moves.

```
O = me.Point('O')
P = me.Point('P')
```

The velocity of point $O$ in the ceiling is zero.

```
O.set_vel(ceiling, 0)
```

Point $P$ can move downward in the $y$ direction and its velocity is $v$ in the downward direction.

```
P.set_pos(O, x * ceiling.x)
P.set_vel(ceiling, v * ceiling.x)
P.vel(ceiling)
```

There are three forces acting on the particle. Those due to the acceleration of gravity, the damper, and the spring.

```
damping = -c * P.vel(ceiling)
stiffness = -k * P.pos_from(O)
gravity = m * g * ceiling.x
forces = damping + stiffness + gravity
forces
```

Now we can use Newton's second law, $0=F-ma$, to form the equation of motion of the system.

```
zero = me.dot(forces - m * P.acc(ceiling), ceiling.x)
zero
```

We can then form the first order equations of motion by solving for $\frac{dv}{dt}$ and introducing the kinematical differential equation, $v=\frac{dx}{dt}$.

```
dv_by_dt = sym.solve(zero, v.diff(t))[0]
dx_by_dt = v
dv_by_dt, dx_by_dt
```

Forming the equations of motion can also be done with the automated methods available in the Mechanics package: `LagrangesMethod` and `KanesMethod`. Here we will make use of Kane's method to find the same equations of motion that we found manually above. First, define a particle that represents the mass attached to the damper and spring.

```
mass = me.Particle('mass', P, m)
```

Now we can construct a `KanesMethod` object by passing in the generalized coordinate, $x$, the generalized speed, $v$, and the kinematical differential equation which relates the two, $0=v-\frac{dx}{dt}$.

```
kane = me.KanesMethod(ceiling, q_ind=[x], u_ind=[v], kd_eqs=[v - x.diff(t)])
```

Now Kane's equations can be computed, and we can obtain $F_r$ and $F_r^*$.

```
fr, frstar = kane.kanes_equations([(P, forces)], [mass])
print fr
print frstar
```

The equations are also available in the form $M\frac{d}{dt}[q,u]^T=f(\dot{u}, u, q)$ and we can extract the mass matrix, $M$, and the forcing equations, $f$.

```
M = kane.mass_matrix_full
f = kane.forcing_full
M, f
```

Finally, we can form the first order differential equations of motion $\frac{d}{dt}[q,u]^T=M^{-1}f(\dot{u}, u, q)$, which is the same as previously found.

```
M.inv() * f
```

#Simulating the system#

Now that we have defined the mass-spring-damper system, we are going to simulate it.

PyDy's `System` is a wrapper that holds the Kanes object to integrate the equations of motion using numerical values of constants.

```
from pydy.system import System
```

```
sys = System(kane)    #Initializing System with the KanesMethod's object
```

Now, we specify the numerical values of the constants and the initial values of states in the form of a dict.

```
constants = {m:10, g:9.8, c:5, k:10}
init_conditions = {x:0, v:0}

sys.constants = constants
sys.initial_conditions = init_conditions
```

We must generate a time vector over which the integration will be carried out.

```
import numpy as np
sys.times = np.linspace(0, 8, 100)
```

```
x0 = sys.integrate()
```

#Visualizing the System

PyDy has a native module `pydy.viz` which is used to visualize a System in a GUI.

```
from pydy.viz import *
from numpy import pi
```

For visualizing the system, we need to create shapes for the objects we wish to visualize, and map each of them
to a VisualizationFrame, which holds the position and orientation of the object.

```
bob = Sphere(2, color="red", material="metal")
bob_vframe = VisualizationFrame(ceiling, P, bob)
```

```
ceil = Circle(radius=10, color="white", material="metal")
#Creating a new reference frame, to give correct orientation to ceiling.
ceil_reference_frame = ceiling.orientnew("ceil_new", 'Axis', [pi/2, ceiling.z])
ceil_vframe = VisualizationFrame(ceil_reference_frame, O, ceil)
```

Now we initialize a Scene. A Scene contains all the information required to visualize a System onto a canvas.
It takes a ReferenceFrame and Point as arguments.

```
scene = Scene(ceiling, O)
```

We provide the VisualizationFrames, which we want to visualize as a list to scene.

```
scene.visualization_frames = [bob_vframe, ceil_vframe]

```

The default camera of Scene has the z axis of the base frame pointing out of the screen, and the y axis pointing up. We want the x axis to point downwards, so we supply a new camera that will achieve this.

```
primary_camera = PerspectiveCamera(ceiling.orientnew('k','Axis', [pi/2, ceiling.z] ), O.locatenew('a',100*ceil_reference_frame.z))
scene.cameras = [primary_camera]
```

The `generate_visualization_json_system` method generates the required data for the animations, in the form of JSON files. These JSON files are needed before calling the display methods.

```
scene.generate_visualization_json_system(sys)
```

Now, we call the display method.

```
scene.display_ipython()
```

```

```


# Tutorial 1: resolution of a Poisson problem

### The problem definition

This tutorial presents how to solve with Physics-Informed Neural Networks a 2-D Poisson problem with Dirichlet boundary conditions.

The problem is written as:
\begin{equation}
\begin{cases}
\Delta u = \sin{(\pi x)} \sin{(\pi y)} \text{ in } D, \\
u = 0 \text{ on } \Gamma_1 \cup \Gamma_2 \cup \Gamma_3 \cup \Gamma_4,
\end{cases}
\end{equation}
where $D$ is a square domain $[0,1]^2$, and $\Gamma_i$, with $i=1,...,4$, are the boundaries of the square.

First of all, some useful imports.

```python
import os
import numpy as np
import argparse
import sys
import torch
from torch.nn import ReLU, Tanh, Softplus
from pina.problem import SpatialProblem
from pina.operators import nabla
from pina.model import FeedForward
from pina.adaptive_functions import AdaptiveSin, AdaptiveCos, AdaptiveTanh
from pina import Condition, Span, PINN, LabelTensor, Plotter
```

Now, the Poisson problem is written in PINA code as a class. The equations are written as *conditions* that should be satisfied in the corresponding domains. *truth_solution*
is the exact solution which will be compared with the predicted one.

```python
class Poisson(SpatialProblem):
    spatial_variables = ['x', 'y']
    bounds_x = [0, 1]
    bounds_y = [0, 1]
    output_variables = ['u']
    domain = Span({'x': bounds_x, 'y': bounds_y})

    def laplace_equation(input_, output_):
        force_term = (torch.sin(input_['x']*torch.pi) *
                      torch.sin(input_['y']*torch.pi))
        return nabla(output_['u'], input_).flatten() - force_term

    def nil_dirichlet(input_, output_):
        value = 0.0
        return output_['u'] - value

    conditions = {
        'gamma1': Condition(Span({'x': bounds_x, 'y':  bounds_y[-1]}), nil_dirichlet),
        'gamma2': Condition(Span({'x': bounds_x, 'y': bounds_y[0]}), nil_dirichlet),
        'gamma3': Condition(Span({'x':  bounds_x[-1], 'y': bounds_y}), nil_dirichlet),
        'gamma4': Condition(Span({'x': bounds_x[0], 'y': bounds_y}), nil_dirichlet),
    'D': Condition(Span({'x': bounds_x, 'y': bounds_y}), laplace_equation),
    }
    def poisson_sol(self, x, y):
        return -(np.sin(x*np.pi)*np.sin(y*np.pi))/(2*np.pi**2)

    truth_solution = poisson_sol
```

### The problem solution

Then, a feed-forward neural network is defined, through the class *FeedForward*. A 2-D grid is instantiated inside the square domain and on the boundaries. This neural network takes as input the coordinates of the points which compose the grid and gives as output the solution of the Poisson problem. The residual of the equations are evaluated at each point of the grid and the loss minimized by the neural network is the sum of the residuals.
In this tutorial, the neural network is composed by two hidden layers of 10 neurons each, and it is trained for 5000 epochs with a learning rate of 0.003. These parameters can be modified as desired.
The output of the cell below is the final loss of the training phase of the PINN.

```python
poisson_problem = Poisson()

model = FeedForward(layers=[10, 10],
                    output_variables=poisson_problem.output_variables,
                    input_variables=poisson_problem.input_variables)

pinn = PINN(poisson_problem, model, lr=0.003, regularizer=1e-8)
pinn.span_pts(20, 'grid', ['D'])
pinn.span_pts(20, 'grid', ['gamma1', 'gamma2', 'gamma3', 'gamma4'])
pinn.train(5000, 100)
```

    2.384537034558816e-05

The loss trend is saved in a dedicated txt file located in *tutorial1_files*.

```python
os.mkdir('tutorial1_files')
with open('tutorial1_files/poisson_history.txt', 'w') as file_:
    for i, losses in enumerate(pinn.history):
        file_.write('{} {}\n'.format(i, sum(losses)))
pinn.save_state('tutorial1_files/pina.poisson')
```

Now the *Plotter* class is used to plot the results.
The solution predicted by the neural network is plotted on the left, the exact one is represented at the center and on the right the error between the exact and the predicted solutions is showed.

```python
plotter = Plotter()
plotter.plot(pinn)
```

### The problem solution with extra-features

Now, the same problem is solved in a different way.
A new neural network is now defined, with an additional input variable, named extra-feature, which coincides with the forcing term in the Laplace equation.
The set of input variables to the neural network is:

\begin{equation}
[\mathbf{x}, \mathbf{y}, \mathbf{k}(\mathbf{x}, \mathbf{y})], \text{ with } \mathbf{k}(\mathbf{x}, \mathbf{y})=\sin{(\pi \mathbf{x})}\sin{(\pi \mathbf{y})},
\end{equation}

where $\mathbf{x}$ and $\mathbf{y}$ are the coordinates of the points of the grid and $\mathbf{k}(\mathbf{x}, \mathbf{y})$ is the forcing term evaluated at the grid points.

This forcing term is initialized in the class *myFeature*, the output of the cell below is also in this case the final loss of PINN.

```python
poisson_problem = Poisson()

class myFeature(torch.nn.Module):
    """
    """
    def __init__(self):
        super(myFeature, self).__init__()

    def forward(self, x):
        return (torch.sin(x['x']*torch.pi) *
                torch.sin(x['y']*torch.pi))

feat = [myFeature()]
model_feat = FeedForward(layers=[10, 10],
                    output_variables=poisson_problem.output_variables,
                    input_variables=poisson_problem.input_variables,
                    extra_features=feat)

pinn_feat = PINN(poisson_problem, model_feat, lr=0.003, regularizer=1e-8)
pinn_feat.span_pts(20, 'grid', ['D'])
pinn_feat.span_pts(20, 'grid', ['gamma1', 'gamma2', 'gamma3', 'gamma4'])
pinn_feat.train(5000, 100)
```

    7.93498870023341e-07

The losses are saved in a txt file as for the basic Poisson case.

```python
with open('tutorial1_files/poisson_history_feat.txt', 'w') as file_:
        for i, losses in enumerate(pinn_feat.history):
            file_.write('{} {}\n'.format(i, sum(losses)))
pinn_feat.save_state('tutorial1_files/pina.poisson_feat')
```

The predicted and exact solutions and the error between them are represented below.

```python
plotter_feat = Plotter()
plotter_feat.plot(pinn_feat)
```

### The problem solution with learnable extra-features

Another way to predict the solution is to add a parametric forcing term of the Laplace equation as an extra-feature. The parameters added in the expression of the extra-feature are learned during the training phase of the neural network. For example, considering two parameters, the parameteric extra-feature is written as:

\begin{equation}
\mathbf{k}(\mathbf{x}, \mathbf{y}) = \beta \sin{(\alpha \mathbf{x})} \sin{(\alpha \mathbf{y})}
\end{equation}

The new Poisson problem is defined in the dedicated class *ParametricPoisson*, where the domain is no more only spatial, but includes the parameters' space. In our case, the parameters' bounds are 0 and 30.

```python
from pina.problem import ParametricProblem

class ParametricPoisson(SpatialProblem, ParametricProblem):
    bounds_x = [0, 1]
    bounds_y = [0, 1]
    bounds_alpha = [0, 30]
    bounds_beta = [0, 30]
    spatial_variables = ['x', 'y']
    parameters = ['alpha', 'beta']
    output_variables = ['u']
    domain = Span({'x': bounds_x, 'y': bounds_y})

    def laplace_equation(input_, output_):
        force_term = (torch.sin(input_['x']*torch.pi) *
                      torch.sin(input_['y']*torch.pi))
        return nabla(output_['u'], input_).flatten() - force_term

    def nil_dirichlet(input_, output_):
        value = 0.0
        return output_['u'] - value

    conditions = {
        'gamma1': Condition(
            Span({'x': bounds_x, 'y': bounds_y[1], 'alpha': bounds_alpha, 'beta': bounds_beta}),
            nil_dirichlet),
        'gamma2': Condition(
            Span({'x': bounds_x, 'y': bounds_y[0], 'alpha': bounds_alpha, 'beta': bounds_beta}),
            nil_dirichlet),
        'gamma3': Condition(
            Span({'x': bounds_x[1], 'y': bounds_y, 'alpha': bounds_alpha, 'beta': bounds_beta}),
            nil_dirichlet),
        'gamma4': Condition(
            Span({'x': bounds_x[0], 'y': bounds_y, 'alpha': bounds_alpha, 'beta': bounds_beta}),
            nil_dirichlet),
        'D': Condition(
            Span({'x': bounds_x, 'y': bounds_y, 'alpha': bounds_alpha, 'beta': bounds_beta}),
            laplace_equation),
    }

    def poisson_sol(self, x, y):
        return -(np.sin(x*np.pi)*np.sin(y*np.pi))/(2*np.pi**2)

```

Here, as done for the other cases, the new parametric feature is defined and the neural network is re-initialized and trained, considering as two additional parameters $\alpha$ and $\beta$.

```python
param_poisson_problem = ParametricPoisson()

class myFeature(torch.nn.Module):
    """
    """
    def __init__(self):
        super(myFeature, self).__init__()

    def forward(self, x):
        return (x['beta']*torch.sin(x['alpha']*x['x']*torch.pi)*
               torch.sin(x['alpha']*x['y']*torch.pi))

feat = [myFeature()]
model_learn = FeedForward(layers=[10, 10],
                    output_variables=param_poisson_problem.output_variables,
                    input_variables=param_poisson_problem.input_variables,
                    extra_features=feat)

pinn_learn = PINN(poisson_problem, model_feat, lr=0.003, regularizer=1e-8)
pinn_learn.span_pts(20, 'grid', ['D'])
pinn_learn.span_pts(20, 'grid', ['gamma1', 'gamma2', 'gamma3', 'gamma4'])
pinn_learn.train(5000, 100)
```

    3.265163986679126e-06

The losses are saved as for the other two cases trained above.

```python
with open('tutorial1_files/poisson_history_learn_feat.txt', 'w') as file_:
    for i, losses in enumerate(pinn_learn.history):
        file_.write('{} {}\n'.format(i, sum(losses)))
pinn_learn.save_state('tutorial1_files/pina.poisson_learn_feat')
```

Here the plots for the prediction error (below on the right) shows that the prediction coming from the **parametric PINN** is more accurate than the one of the basic version of PINN.

```python
plotter_learn = Plotter()
plotter_learn.plot(pinn_learn)
```

Now the files containing the loss trends for the three cases are read. The loss histories are compared; we can see that the loss decreases faster in the cases of PINN with extra-feature.

```python
import pandas as pd

df = pd.read_csv("tutorial1_files/poisson_history.txt", sep=" ", header=None)
epochs = df[0]
poisson_data = epochs.to_numpy()*100
basic = df[1].to_numpy()

df_feat = pd.read_csv("tutorial1_files/poisson_history_feat.txt", sep=" ", header=None)
feat = df_feat[1].to_numpy()

df_learn = pd.read_csv("tutorial1_files/poisson_history_learn_feat.txt", sep=" ", header=None)
learn_feat = df_learn[1].to_numpy()

import matplotlib.pyplot as plt
plt.semilogy(epochs, basic, label='Basic PINN')
plt.semilogy(epochs, feat, label='PINN with extra-feature')
plt.semilogy(epochs, learn_feat, label='PINN with learnable extra-feature')
plt.legend()
plt.grid()
plt.show()
```


# CLASE 7: SymPy

_ __SymPy es una biblioteca de Python para matemática simbólica__. Apunta a convertirse en un sistema de algebra computacional (__CAS__) con todas sus prestaciones manteniendo el código tan simple como sea posible para manterlo comprensible y fácilmente extensible. SymPy está __escrito totalmente en Python y no requiere bibliotecas adicionales__. _Este proyecto comenzó en 2005, fue lanzado al público en 2007 y a él han contribuido durante estos años cientos de personas._

_ Otros CAS conocidos son Mathematica y Maple, sin embargo ambos son software privativo y de pago. [Aquí](https://github.com/sympy/sympy/wiki/SymPy-vs.-Maple) puedes encontrar una comparativa de SymPy con Maple. _

Hoy veremos cómo:

* Crear símbolos y expresiones.
* Manipular expresiones (simplificación, expansión...)
* Calcular derivadas e integrales.
* Límites y desarrollos en serie.
* Resolución de ecuaciones.
* Resolción de EDOs.
* Matrices

Sin embargo, SymPy no acaba aquí ni mucho menos...

## Documentación & SymPy Live Shell

[Documentacion](http://docs.sympy.org/latest/index.html)

[SympyGamma](http://www.sympygamma.com/input/?i=integrate%281+%2F+%281+%2B+x^2%29%29)

## Creación de símbolos

Lo primero, como siempre, es importar aquello que vayamos a necesitar:

```python
!pip install sympy
```

```python
#import sympy as sp
from sympy import *
```

```python
init_printing(use_latex=True)
```

* Gracias a `use_latex=True` obtenemos la salida en $\LaTeX$.
* __Ha creado una serie de variables__ para que podamos ponernos a trabajar en el momento.

Nota:
En Python, no se declaran las variables, sin embargo, no puedes usar una hasta que no le hayas asignado un valor. Si ahora intentamos crear una variable `a` que sea `a = 2 * b`, veamos qué ocurre:

```python
# Intentamos usar un símbolo que no hemos creado
a = 2 * b
```

Como en `b` no había sido creada, Python no sabe qué es `b`.

Esto mismo nos ocurre con los símbolos de SymPy. __Antes de usar una variable, debo decir que es un símbolo y asignárselo:__

```python
# Creamos el símbolo a
a = symbols("a")
a
```

```python

```

```python
# Número pi
(a + pi)**2
```

```python
# Unidad imaginaria
1 + 2*I
```

```python
# Número e
E ** 2
```

```python
# Vemos qué tipo de variable es a
type(a)
```

    sympy.core.symbol.Symbol

Ahora ya podría crear `b = 2 * a`:

```python
b = 2 * a
b
```

```python

```

¿Qué está ocurriendo? Python detecta que a es una variable de tipo `Symbol` y al multiplicarla por `2` devuelve una variable de Sympy.

Como Python permite que el tipo de una variable cambie, __si ahora le asigno a `a` un valor float deja de ser un símbolo.__

```python
b = 2.0
```

```python
type(b)
```

    float

---
__Las conclusiones son:__

* __Si quiero usar una variable como símbolo debo crearla previamente.__
* Las operaciones con símbolos devuelven símbolos.
* Si una varibale que almacenaba un símbolo recibe otra asignación, cambia de tipo.

---

__Las variables de tipo `Symbol` actúan como contenedores en los que no sabemos qué hay (un real, un complejo, una lista...)__. Hay que tener en cuenta que: __una cosa es el nombre de la variable y otra el símbolo con el que se representa__.

```python
#creación de símbolos
alpha, beta = symbols("alpha beta")
alpha, beta
```

```python
c_T = symbols("c_T")
c_T
```

Incluso puedo hacer cosas raras como:

```python
# Diferencia entre variable y símbolo
b = symbols("d")
b
```

Además, se pueden crear varos símbolos a la vez:

```python
x, y, z, t = symbols("x, y, z, t")
x, y, z, t
```

y símbolos griegos:

```python

```

__Por defecto, SymPy entiende que los símbolos son números complejos__. Esto puede producir resultados inesperados ante determinadas operaciones como, por ejemplo, lo logaritmos. __Podemos indicar que la variable es real, entera... en el momento de la creación__:

```python
# Creamos símbolos reales
x, y, z, t = symbols("x, y, z, t")
```

```python
# restricciones sobre el simbolo
x.assumptions0
```

    {'commutative': True}

```python
# Podemos ver las asunciones de un símbolo
x = symbols("x", real=True, positive=True)
```

```python
x.assumptions0

```

    {'real': True,
     'commutative': True,
     'imaginary': False,
     'finite': True,
     'hermitian': True,
     'complex': True,
     'extended_real': True,
     'infinite': False,
     'positive': True,
     'nonpositive': False,
     'negative': False,
     'extended_positive': True,
     'extended_nonnegative': True,
     'nonnegative': True,
     'extended_negative': False,
     'extended_nonpositive': False,
     'extended_nonzero': True,
     'nonzero': True,
     'zero': False}

## Expresiones

Comencemos por crear una expresión como: $\cos(x)^2+\sin(x)^2$

```python
cos(z) ** 2 + sin(z) ** 2
```

### `simplify()`

Podemos pedirle que simplifique la expresión anterior:

```python
simplify(_25)
```

```python
sqrt(z**2)
```

```python
simplify(_27)
```

```python
y = symbols("y", real=True)
```

```python
simplify(sqrt(y**2))
```

```python
simplify(sqrt(x**2))
```

En este caso parece estar claro lo que quiere decir más simple, pero como en cualquier _CAS_ el comando `simplify` puede no devolvernos la expresión que nosotros queremos. Cuando esto ocurra necesitaremos usar otras instrucciones.

### `.subs()`

En algunas ocasiones necesitaremos sustituir una variable por otra, por otra expresión o por un valor.

```python
_25.subs(z, 1+x**2)
```

```python
(sin(x)**2 + cos(y)**2).subs({x: 1+ z, y: 1-z})
```

```python

```

```python

```

Cambia el `sin(x)` por `exp(x)`

```python
sin(x).replace(sin, exp)
```

Particulariza la expresión $sin(x) + 3 x $ en $x = \pi$

```python
(sin(x)+3*x).subs(x, pi)
```

__Aunque si lo que queremos es obtener el valor numérico lo mejor es `.evalf()`__

```python
_36.evalf()
```

```python
_36.evalf(n=27)
```

```python
#ver pi con 25 decimales

```

```python
#el mismo resultado se obtiene ocn la función N()
N(_36)
```

# Simplificación

SymPy ofrece numerosas funciones para __simplificar y manipular expresiones__. Entre otras, destacan:

* `expand()`
* `factor()`
* `collect()`
* `apart()`
* `cancel()`

Puedes consultar en la documentación de SymPy lo que hace cada una y algunos ejemplos. __Existen también funciones específicas de simplificación para funciones trigonométricas, potencias y logaritmos.__ Abre [esta documentación](http://docs.sympy.org/latest/tutorial/simplification.html) si lo necesitas.

##### ¡Te toca!

Pasaremos rápidamente por esta parte, para hacer cosas "más interesantes". Te proponemos algunos ejemplos para que te familiarices con el manejor de expresiones:

__Crea las expresiones de la izquierda y averigua qué función te hace obtener la de la derecha:__

expresión 1| expresión 2
:------:|:------:
$\left(x^{3} + 3 y + 2\right)^{2}$    |    $x^{6} + 6 x^{3} y + 4 x^{3} + 9 y^{2} + 12 y + 4$
$\frac{\left(3 x^{2} - 2 x + 1\right)}{\left(x - 1\right)^{2}} $ | $3 + \frac{4}{x - 1} + \frac{2}{\left(x - 1\right)^{2}}$
$x^{3} + 9 x^{2} + 27 x + 27$         |    $\left(x + 3\right)^{3}$
$\sin(x+2y)$                          |    $\left(2 \cos^{2}{\left (y \right )} - 1\right) \sin{\left (x \right )} + 2 \sin{\left (y \right )} \cos{\left (x \right )} \cos{\left (y \right )}$

```python
x, y = symbols("x y")
```

```python
#1
(x**3 + 3*y+2)**2
```

```python
expand(_43)
```

```python

```

```python
#2
(3*x**2-2*x+1)/(x-1)**2
```

```python
apart(_46)
```

```python

```

```python
#3

```

```python
(x**3 + 9*x**2 + 27*x + 27).factor()
```

```python

```

```python
#4
sin(x+2*y)
```

```python
expand(_, trig=True)
```

```python
expand_trig(_50)
```

```python

```

```python

```

# Derivadas e integrales

Puedes derivar una expresion usando el método `.diff()` y la función `diff()`

```python
#creamos una expresión
expresion = cos(x)

#obtenemos la derivada primera con funcion
expresion.diff(x)
```

```python
#utilizando método
diff(expresion)
```

__¿derivada tercera?__

```python
expresion.diff(x, x, x)
```

```python
expresion.diff(x, 3)
```

__¿varias variables?__

```python
expr = y * exp(2*x) + x / (x**2 + y**3)
expr
```

```python
expr.diff(x, 2, y, 3)
```

__Queremos que la deje indicada__, usamos `Derivative()`

```python
Derivative(expr, y)
```

__¿Será capaz SymPy de aplicar la regla de la cadena?__

```python
# Creamos una función F
F = Function("F")
F(x)
```

```python
# Creamos una función G
G = Function("G")
G(x)
```

$$\frac{d}{d x} F{\left (G(x) \right )} $$

```python
# Derivamos la función compuesta F(G(x))
F(G(x)).diff(x)
```

En un caso en el que conocemos las funciones:

```python
# definimos una f
f = 3 * x + cos(y)
```

```python
# definimos una g(f)
g = 1 / f
g
```

```python
#la derivamos
g.diff(x)
```

##### Te toca integrar

__Si te digo que se integra usando el método `.integrate()` o la función `integrate()`__. ¿Te atreves a integrar estas casi inmediatas...?:

$$\int{\cos(x)^2}dx$$
$$\int{\frac{dx}{\sin(x)}}$$
$$\int{\frac{dx}{(x^2+a^2)^2}}$$

```python
#integrate?
```

```python
cos(x)**2
```

```python
integrate(cos(x)**2)
```

```python
integrate(cos(x)**2, (x, 0, 1))
```

```python
(1/sin(x)).integrate()
```

```python
integrate(1/(x**2+a**2)**2, x)
```

```python

```

# Límites

Calculemos este límite sacado del libro _Cálculo: definiciones, teoremas y resultados_, de Juan de Burgos:

$$\lim_{x \to 0} \left(\frac{x}{\tan{\left (x \right )}}\right)^{\frac{1}{x^{2}}}$$

Primero creamos la expresión:

```python
expr = (x/tan(x))**(1/x**2)
expr
```

Obtenemos el límite con la función `limit()` y si queremos dejarlo indicado, podemos usar `Limit()`:

```python
limit(expr, x, 0)
```

```python
oo
```

```python
limit(expr, x, oo)
```

```python
# para dejar indicado
Limit(expr, x, 0)
```

# Series

Los desarrollos en serie se pueden llevar a cabo con el método `.series()` o la función `series()`

```python
#creamos la expresión
expr = cos(x)
```

```python
#la desarrollamos en serie
expr.series(x)
```

Se puede especificar el número de términos pasándole un argumento `n=...`. El número que le pasemos será el primer término que desprecie.

```python
# Indicando el número de términos
expr.series(x, n=10)
```

Si nos molesta el $\mathcal{O}(x^{10})$ lo podemos quitar con `removeO()`:

```python
_.removeO()
```

```python
expr.series(x, n=10, x0=pi/2)
```

---

## Resolución de ecuaciones

Como se ha mencionado anteriormente las ecuaciones no se pueden crear con el `=`

```python
#creamos la ecuación
a + x = cos(x)
```

```python
Eq(1+x, x**2)

```

```python
Eq(1+x - x**2, 0)
```

```python
# También la podemos crear como
ecuacion = Eq(1+x-sin(x), 0)
ecuacion
```

```python
ecuacion2 = Eq(x**2+1, 0)
```

```python
#la resolvemos
sol = solve(ecuacion2, x)
sol
```

```python

```

Pero la gracia es resolver con símbolos, ¿no?
$$a e^{\frac{x}{t}} = C$$

```python
# Creamos los símbolos y la ecuación
C = symbols("C")
C
```

```python
# La resolvemos
ecuacion = Eq(a*exp(x/t), C)
ecuacion
```

```python
solve(ecuacion, x)
```

Si consultamos la ayuda, vemos que las posibilidades y el número de parámetros son muchos, no vamos a entrar ahora en ellos, pero ¿se ve la potencia?

## Ecuaciones diferenciales

Tratemos de resolver, por ejemplo:

$$y{\left (x \right )} + \frac{d}{d x} y{\left (x \right )} + \frac{d^{2}}{d x^{2}}  y{\left (x \right )} = \cos{\left (x \right )}$$

```python
# solo sirve para ejemplos acádemicos sencillos

y = Function('y')
y
```

    y

```python
edo = Eq(y(x) + y(x).diff(x) + y(x).diff(x, 2), cos(x))
edo
```

```python
#resolvemos
dsolve(edo,y(x))
```

# Matrices

```python
#creamos una matriz llena de símbolos
a, b, c, d = symbols('a, b, c, d')
a, b, c, d

A = Matrix([
            [a, b],
            [c, d]
])
A
```

```python
#sacamos autovalores
A.eigenvals()

```

```python
A.eigenvects()
```

```python
#inversa
A.inv()
```

```python
#elevamos al cuadrado la matriz
A ** 2
```


```python
# Imports
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from matplotlib.colors import Normalize
import sympy as smp # symbolic python, for equations
import pprint # pretty printer

# Ask jupyter to display plots inline
%matplotlib inline

# OSX specific (for a nicer display on "retina" screen)
%config InlineBackend.figure_format = 'retina'

# Vector plotting function -- feel free to reuse this in the linear algebra exercises
def plot_vec(origin,x,y,color='black'):
    ax = plt.gca()
    ax.quiver(*origin, x, y, color=color,angles='xy', scale_units='xy', scale=1)
    ax.set_xlabel('x-axis')
    ax.set_ylabel('y-axis')
    ax.set_xlim([-np.amax(np.abs(x)),np.amax(np.abs(x))])
    ax.set_ylim([-np.amax(np.abs(y)),np.amax(np.abs(y))])
    ax.grid()
    return ax
```

---

1. Represent the following vectors of ${\rm I\!R}^{2}$ on a plane with two orthogonal axes

```python
V = np.zeros((10,2))
V[0] = np.array([0,2])
V[1] = np.array([3, -1])
V[2] = 2*np.array([1, -3])
V[3] = np.array([2,4]) - np.array([0,-2])
V[4] = np.array([1,4]) + 3*np.array([2,-1])
V[5] = 5*np.array([1,1]) + 2*np.array([-2,1])
V[6] = (np.array([[1,3],[0,2]]) @ np.array([[-1],[2]])).T
V[7] = (np.array([[-2,1],[1,2]]) @ np.array([[1],[3]])).T
V[8] = (np.array([[0,2],[-1,1]]) @ np.array([[3],[1]])).T
V[9] = ((np.array([[1,3],[0,2]]) + np.array([[-2,1],[1,2]])) @ np.array([[1],[3]])).T

print(V)
```

    [[ 0.  2.]
     [ 3. -1.]
     [ 2. -6.]
     [ 2.  6.]
     [ 7.  1.]
     [ 1.  7.]
     [ 5.  4.]
     [ 1.  7.]
     [ 2. -2.]
     [11. 13.]]

```python
origin = np.zeros((2,10)) # origin point

# Color each vector separately
colors = np.arange(np.max(V.shape))
labels = ['V'+str(i) for i in colors]

# Select a colormap
colormap = cm.inferno

# Normalize the colormap
norm = Normalize()
norm.autoscale(colors)

fig, ax = plt.subplots(1,figsize=(12,6))
plt.quiver(*origin,
           V[:,0],
           V[:,1],
           scale=1,
           scale_units='xy',
           angles = 'xy',
           label=labels,
           color=colormap(norm(colors)))
plt.xlim((V[:,0].min()-1, V[:,0].max()+1))
plt.ylim(( V[:,1].min()-1, V[:,1].max()+1))
plt.grid()
plt.legend()

plt.show()
```

---

2. Find the scalar $k$ for which the vector $k \begin{bmatrix}1 & 2 & 1 & 3 & 1\end{bmatrix}$ is a unit vector of $R^5$.

The norm of the vector when $k=1$ is 4. This is given by the formula $\sqrt{1^2 + 2^2 + 1^2 + 3^2 + 1^2}$. In order for the vector to have a magnitude of 1, we need to solve the following for k:

$\sqrt{k^2 + (2k)^2 + k^2 + (3k)^2 + k^2} = 1$

The above formula is equal to $k \lVert \vec{v} \rVert_2$. Given that we already calculated that $\lVert \vec{v} \rVert_2  = 4$, $k=0.25$.

```python
# The vector has to have a length (norm2) of 1
v = np.array([[1,2,1,3,1]])
print('Vector magnitude: ', np.linalg.norm(v), '(a.u.)')

k = 1/np.linalg.norm(v)

print('Scaling factor k is: ', k)
print('Vector magnitude w/ new k: ', np.linalg.norm(k*v), '(a.u)')

```

    Vector magnitude:  4.0 (a.u.)
    Scaling factor k is:  0.25
    Vector magnitude w/ new k:  1.0 (a.u)

---

3. Equation systems in the form $M X = P$, where $X =\begin{bmatrix} x & y & z \end{bmatrix}^T \in {\rm I\!R}^{3}$. Since $M$ is multiplied with $\vec{x}$, a column vector, $M$ is going to be a matrix of size $(3x3)$ and each row will contain the scalars in front of the vecto elements $x,y,z$ per equation. To solve the equation systems, we will calculate the inverse, $M^{-1}$, and solve for $\vec{v}$. The solution is shown below, in code.

```python
# First system of equations
M1 = np.array([[1,1,2],[1,2,1],[2,1,1]])
P1 = np.array([[3],[1],[0]])

x,y,z = smp.symbols('x,y,z')
X = np.array([[x],[y],[z]])
print(M1@X)

print(np.linalg.det(M1))

X_sol = np.round(np.linalg.inv(M1)@P1, 4)

print()

print('Solution X= \n', X_sol)
print()
# Evaluation
print('Evaluation: \n', M1@X_sol == P1)
```

    [[x + y + 2*z]
     [x + 2*y + z]
     [2*x + y + z]]
    -4.0

    Solution X=
     [[-1.]
     [ 0.]
     [ 2.]]

    Evaluation:
     [[ True]
     [ True]
     [ True]]

```python
# Second system of equations
M2 = np.array([[1,0,2],[0,-1,1],[1,-2,0]])
P2 = np.array([[1],[2],[1]])

X_sol = np.round(np.linalg.inv(M2)@P2, 4)

print('Solution X= \n', X_sol)

# Evaluation
print('Evaluation: \n', M2@X_sol == P2)
```

    Solution X=
     [[-1.]
     [-1.]
     [ 1.]]
    Evaluation:
     [[ True]
     [ True]
     [ True]]

---

4. Same as before. The equations correspond to lines in 2D space. The solution to the system is the crossing point of the lines and can be calculated analytically by computing $M^{-1}$ and solving the system as before. The crossing point is given as:

$$
\begin{matrix}
P = \begin{bmatrix} \frac{6}{m-2} \\ \frac{-3}{m-2} \end{bmatrix},& m^2\neq4
\end{matrix}
$$

The code below calculates the determinant and the inverse of $M$. Use these to evaluate your solution.

**hint**: $4 - m^2 = 2^2 - m^2 = (2+m)(2-m)$

```python
x,y = smp.symbols('x,y')
m = smp.Symbol('m')

M = smp.Matrix(np.array([[1,m],[m,4]]))
P = np.array([[-3],[6]])

print('Matrix determinant: ', M.det()) # This must be != 0 for the matrix to be invertible. Therefore m!=+-2
print('Inverse of matrix M: ', M.inv()) # Matrix inverse
print('Solve: ', M.inv()@P)
```

    Matrix determinant:  4 - m**2
    Inverse of matrix M:  Matrix([[4/(4 - m**2), -m/(4 - m**2)], [-m/(4 - m**2), 1/(4 - m**2)]])
    Solve:  Matrix([[-6*m/(4 - m**2) - 12/(4 - m**2)], [3*m/(4 - m**2) + 6/(4 - m**2)]])

```python
# example plot for m = 1
#M1 = M.subs('m',1)
#b = smp.Matrix([[x],[y]])

Eq1 = smp.Eq(x + m*y, -3)
Eq2 = smp.Eq(m*x + 4*y, 6)

mval = 1

fig4 = plt.figure(figsize=(5,5))
line1 = smp.plot_implicit(Eq1.subs('m',mval))
line2 = smp.plot_implicit(Eq2.subs('m',mval))

line1.extend(line2)
line1.show()
```

---

5. Calculation of products done by hand. You can use sympy for the evaluation, as in the example below

```python
x = np.array([[1],[-1]])
y = np.array([[smp.symbols('y1')],[smp.symbols('y2')]])
print('1.\n',x.T @ y)
print('Dimensionality: ',(x.T @ y).shape)

W = np.array([[smp.symbols('w11'), smp.symbols('w12')],
              [smp.symbols('w21'), smp.symbols('w22')]])

x_sym = np.array([[smp.symbols('x1')],
                  [smp.symbols('x2')]])

print()
print('2.\n', W@x_sym)
print('Dimensionality: ', (W@x_sym).shape)

x11,x12,x13,x21,x22,x23 = smp.symbols('x11,x12,x13,x21,x22,x23')
X = np.array([[x11,x12,x13],[x21,x22,x23]])

print()
print('3.\n', X@X.T)
print('Dimensionality: ', (X@X.T).shape)
```

    1.
     [[y1 - y2]]
    Dimensionality:  (1, 1)

    2.
     [[w11*x1 + w12*x2]
     [w21*x1 + w22*x2]]
    Dimensionality:  (2, 1)

    3.
     [[x11**2 + x12**2 + x13**2 x11*x21 + x12*x22 + x13*x23]
     [x11*x21 + x12*x22 + x13*x23 x21**2 + x22**2 + x23**2]]
    Dimensionality:  (2, 2)

---

6. Find all possible products among the matrices.

These are AC (1x2), AE (1x3), DB (2x1), EC (3x2), DD (2x2), EE (3x3)

```python
A = np.array([[1,2,3]])
B = np.array([[1],[-2]])
C = np.array([[2,1],[-3,0],[1,2]])
D = np.array([[-2,5],[5,0]])
E = np.array([[-1,1,3],[-1,-4,0],[0,2,5]])

print('Dimensionality:')
print('A: ', A.shape)
print('B: ', B.shape)
print('C: ', C.shape)
print('D: ', D.shape)
print('E: ', E.shape)

# list example
matlist = [A,B,C,D,E]
matlab = ['A','B','C','D','E']

# dictionary example
matdic = {'A':A,
         'B':B,
         'C':C,
         'D':D,
         'E':E}

print('-----------------')
print('Products:')
for ii in matlab:
    for jj in matlab:
        print(ii+jj,':')
        try:
            print(matdic[ii] * matdic[jj])
        except:
            print('element-wise product not defined')

        try:
            print(matdic[ii] @ matdic[jj])
        except:
            print('not defined')
        print()
```

    Dimensionality:
    A:  (1, 3)
    B:  (2, 1)
    C:  (3, 2)
    D:  (2, 2)
    E:  (3, 3)
    -----------------
    Products:
    AA :
    not defined

    AB :
    not defined

    AC :
    [[-1  7]]

    AD :
    not defined

    AE :
    [[-3 -1 18]]

    BA :
    [[ 1  2  3]
     [-2 -4 -6]]

    BB :
    not defined

    BC :
    not defined

    BD :
    not defined

    BE :
    not defined

    CA :
    not defined

    CB :
    [[ 0]
     [-3]
     [-3]]

    CC :
    not defined

    CD :
    [[  1  10]
     [  6 -15]
     [  8   5]]

    CE :
    not defined

    DA :
    not defined

    DB :
    [[-12]
     [  5]]

    DC :
    not defined

    DD :
    [[ 29 -10]
     [-10  25]]

    DE :
    not defined

    EA :
    not defined

    EB :
    not defined

    EC :
    [[-2  5]
     [10 -1]
     [-1 10]]

    ED :
    not defined

    EE :
    [[ 0  1 12]
     [ 5 15 -3]
     [-2  2 25]]

---

7.  This exercise correlates with the video [Nonsquare matrices as transformations between dimensions | Chapter 8, Essence of linear algebra](https://www.youtube.com/watch?v=v8VSDg_WQlA&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab&index=8). To solve the exercise, we transform a 3D vector to a 4D one. So our matrix $U$ will have dimensionality of $(3,4)$.

```python
x,y,z = smp.symbols('x,y,z')

v = smp.Matrix([x,y,z])
print(v)

U = np.array([[-1,1,0],[1,-1,0],[-1,0,1],[0,-1,1]])

print(U@v)
```

    Matrix([[x], [y], [z]])
    Matrix([[-x + y], [x - y], [-x + z], [-y + z]])

To determine if the vectors $\vec(u_x), \vec(u_y), \vec(u_z)$ are orthogonal, we will calculate their inner product using numpy's `inner`:

```python
ux = np.array([[1,0,0]])
uy = np.array([[0,1,0]])
uz = np.array([[0,0,1]])

print(np.inner(ux,uy))
print(np.inner(ux,uz))
print(np.inner(uy,uz))
print('The vectors are bases of the 3D space, therefore they are orthogonal! They have inner products of 0!')
```

    [[0]]
    [[0]]
    [[0]]
    The vectors are bases of the 3D space, therefore they are orthogonal! They have inner products of 0!

---

8. Calculate, when defined, the products of AB and BA
    1. Product is zero in both cases
    2. $A_{3x3}$, $B_{2,3}$. AB is not defined; BA is defined and has dimensionality $(BA)_{2x3}$.
    3. $A_{3x2}$, $B_{2,4}$. AB is defined and has dimensionality $(AB)_{3x4}$. BA is not defined

```python
# (a)
A = np.array([[1,0],[0,0]])
B = np.array([[0,0],[0,1]])
print('AB: \n', A@B, '\nBA: \n', B@A)

# (b)
A = np.array([[0,2,1],[1,1,0],[-1,-2,-1]])
B = np.array([[2,0,1],[-1,1,2]])

print('--------------------')
print('(b)')
print('AB: ')
try:
    print(A@B)
except:
    print('not defined')

print('BA: ')
try:
    print(B@A)
except:
    print('not defined')

# (c)
A = np.array([[1,2],[1,1],[0,3]])
B = np.array([[-1,1,0,1],[2,1,0,0]])

print('--------------------')
print('(c)')
print('AB: ')
try:
    print(A@B)
except:
    print('not defined')

print('BA: ')
try:
    print(B@A)
except:
    print('not defined')
```

    AB:
     [[0 0]
     [0 0]]
    BA:
     [[0 0]
     [0 0]]
    --------------------
    (b)
    AB:
    not defined
    BA:
    [[-1  2  1]
     [-1 -5 -3]]
    --------------------
    (c)
    AB:
    [[3 3 0 1]
     [1 2 0 1]
     [6 3 0 0]]
    BA:
    not defined

---

9. Calculate $AB$ and $AC$. May $A$ be invertible? Find all 3x3 matrices $M$ such that $AM=0$.

```python
A = np.array([[1,0,0], [0,1,1], [3,1,1]])
B = np.array([[1,1,1], [0,1,0], [1,0,0]])
C = np.array([[1,1,1], [1,2,1], [0,-1,-1]])
M = np.array([smp.symbols('m11:14'),
            smp.symbols('m21:24'),
            smp.symbols('m31:34')])

print('AB:\n',A@B)
print('---------------')
print('AC:\n',A@C)

print('Determinant of A: ', np.linalg.det(A))
#np.linalg.inv(A) # this produces an error, matrix is singular

print('---------------')
print('M:\n',M)
print('---------------')
print('AM:\n',A@M)
```

    AB:
     [[1 1 1]
     [1 1 0]
     [4 4 3]]
    ---------------
    AC:
     [[1 1 1]
     [1 1 0]
     [4 4 3]]
    Determinant of A:  0.0
    ---------------
    M:
     [[m11 m12 m13]
     [m21 m22 m23]
     [m31 m32 m33]]
    ---------------
    AM:
     [[m11 m12 m13]
     [m21 + m31 m22 + m32 m23 + m33]
     [3*m11 + m21 + m31 3*m12 + m22 + m32 3*m13 + m23 + m33]]

Given the above, we have:

1. First row:
    * $m_{11} = m_{12} = m_{13} = 0$;
2. Second row:
    * $m_{21} = -m_{31}$
    * $m_{22} = -m_{32}$
    * $m_{23} = -m_{33}$
3. Third row:
    * $3m_{11}+m_{21}+m_{31} = 0$
    * $3m_{12} + m_{22} + m_{32} = 0$
    * $3m_{13} + m_{23} + m_{33} = 0$

---

10. We must solve the equation $AB = BA$. We are given A, therefore we can calculate both $AB$ and $BA$ and equate their elements. This will produce 4 equations. Solving them yields

$$ B = \begin{bmatrix} x & y \\ 0 & x \end{bmatrix}, x,y\in{ $$

```python
a,b = smp.symbols('a,b')

b11,b12,b21,b22 = smp.symbols('b11,b12,b21,b22')

A = smp.Matrix([[a,b],[0,a]])
B = smp.Matrix([[b11,b12],[b21,b22]])

print('AB:')
pprint.pprint(A@B)
print()
print('BA:')
pprint.pprint(B@A)

Bt = np.array([[2,1],[0,2]])
print()
print(A@Bt==Bt@A)
```

    AB:
    Matrix([
    [a*b11 + b*b21, a*b12 + b*b22],
    [        a*b21,         a*b22]])

    BA:
    Matrix([
    [a*b11, a*b12 + b*b11],
    [a*b21, a*b22 + b*b21]])

    False

---

11. A and B are 2x2 matrices and we must solve $AB=0$, $BA\neq0$.

```python
a11,a12,a21,a22 = smp.symbols('a11,a12,a21,a22')
b11,b12,b21,b22 = smp.symbols('b11,b12,b21,b22')

A = smp.Matrix([[a11,a12],[a21,a22]])
B = smp.Matrix([[b11,b12],[b21,b22]])

print('Product of AB:')
pprint.pprint(A@B)
print()
print('Product of BA:')
pprint.pprint(B@A)
```

    Product of AB:
    Matrix([
    [a11*b11 + a12*b21, a11*b12 + a12*b22],
    [a21*b11 + a22*b21, a21*b12 + a22*b22]])

    Product of BA:
    Matrix([
    [a11*b11 + a21*b12, a12*b11 + a22*b12],
    [a11*b21 + a21*b22, a12*b21 + a22*b22]])

---

12. Let's calculate the matrices for $n\in[2,5]$. Do you notice anything?

```python
# Define the matrices
A = np.array([[1.,-1.],[-1.,1.]])
B = np.array([[1.,1.],[0.,2.]])

# max power
N = 6

for n in np.arange(2,N):
    print('A^',n,':\n',A**n)
    print('B^',n,':\n',B**n)
    print('---------------------')
```

    A^ 2 :
     [[1. 1.]
     [1. 1.]]
    B^ 2 :
     [[1. 1.]
     [0. 4.]]
    ---------------------
    A^ 3 :
     [[ 1. -1.]
     [-1.  1.]]
    B^ 3 :
     [[1. 1.]
     [0. 8.]]
    ---------------------
    A^ 4 :
     [[1. 1.]
     [1. 1.]]
    B^ 4 :
     [[ 1.  1.]
     [ 0. 16.]]
    ---------------------
    A^ 5 :
     [[ 1. -1.]
     [-1.  1.]]
    B^ 5 :
     [[ 1.  1.]
     [ 0. 32.]]
    ---------------------

---

13. The matrix $A$ is given above. Let's calculate the two sides of the equation and then proceed to find the inverse of the given matrix.

```python
I = np.eye(2) # 2x2 identity matrix
#print(I)

print('A^2:\n',A**2)
print()
print('2*I-A:\n', 2*I-A)
print('------------------')

# Inverse of 3x3 A matrix
A33 = np.array([[-1.,1.,1.],[1.,-1.,1.],[1.,1.,-1.]])
print('A_3x3:\n',A33)
print('------------------')
print('Inverse:\n',np.linalg.inv(A33))
```

    A^2:
     [[1. 1.]
     [1. 1.]]

    2*I-A:
     [[1. 1.]
     [1. 1.]]
    ------------------
    A_3x3:
     [[-1.  1.  1.]
     [ 1. -1.  1.]
     [ 1.  1. -1.]]
    ------------------
    Inverse:
     [[-0.   0.5  0.5]
     [ 0.5  0.   0.5]
     [ 0.5  0.5  0. ]]

---

14. Let's calculate the inverse of the matrix for N=3,5,10. Do you see a pattern?

```python
for N in (3,5,10):
    A = np.ones((1,N))

    for ii in np.arange(1,N):
        tmp = np.ones((1,N))
        tmp[0,:ii] = 0
        A = np.vstack((A,tmp))

    print('N=',N)
    print('A:\n')
    print(A)
    print()
    print('rank(A)=', np.linalg.matrix_rank(A))
    print('det(A)=',np.linalg.det(A))

    print()
    print('inv(A):\n')
    print(np.linalg.inv(A))
    print('------------------')
```

    N= 3
    A:

    [[1. 1. 1.]
     [0. 1. 1.]
     [0. 0. 1.]]

    rank(A)= 3
    det(A)= 1.0

    inv(A):

    [[ 1. -1.  0.]
     [ 0.  1. -1.]
     [ 0.  0.  1.]]
    ------------------
    N= 5
    A:

    [[1. 1. 1. 1. 1.]
     [0. 1. 1. 1. 1.]
     [0. 0. 1. 1. 1.]
     [0. 0. 0. 1. 1.]
     [0. 0. 0. 0. 1.]]

    rank(A)= 5
    det(A)= 1.0

    inv(A):

    [[ 1. -1.  0.  0.  0.]
     [ 0.  1. -1.  0.  0.]
     [ 0.  0.  1. -1.  0.]
     [ 0.  0.  0.  1. -1.]
     [ 0.  0.  0.  0.  1.]]
    ------------------
    N= 10
    A:

    [[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
     [0. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
     [0. 0. 1. 1. 1. 1. 1. 1. 1. 1.]
     [0. 0. 0. 1. 1. 1. 1. 1. 1. 1.]
     [0. 0. 0. 0. 1. 1. 1. 1. 1. 1.]
     [0. 0. 0. 0. 0. 1. 1. 1. 1. 1.]
     [0. 0. 0. 0. 0. 0. 1. 1. 1. 1.]
     [0. 0. 0. 0. 0. 0. 0. 1. 1. 1.]
     [0. 0. 0. 0. 0. 0. 0. 0. 1. 1.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]

    rank(A)= 10
    det(A)= 1.0

    inv(A):

    [[ 1. -1.  0.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  1. -1.  0.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  1. -1.  0.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  1. -1.  0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  1. -1.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  1. -1.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  1. -1.  0.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  1. -1.  0.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  1. -1.]
     [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  1.]]
    ------------------

```python

```

lessons/Jupyter Notebooks/Solutions/Linear Algebra Exercises Solutions.ipynb


# 1 Curve Fitting (NumPy)

## 1.1 Data generation
Let the following polynomial be the "true" model.

\begin{equation}
t = w_2x^2 + w_1x +w_0
\end{equation}

First we sample $n$ equidistant values for $x$, in a range of $[0,1]$; Then compute $n$ corresponding target values for $t$ using the above polynomial equation; To "simulate" the real-world scenario where the observed data usually come with noise, add random Gaussian noise to our target values.

```python
import random

# get ground-truth data from the "true" model
n = 20  # number of data samples
x = [(idx-round(n/2))/(n/2) for idx in range(n)]
print(x)

w = [1, 2, 3]
t = [xn**2*w[2] + xn*w[1] + w[0] for xn in x]
print(t)

# "simulated" observed target values
std_noise = 0.2
t_observed = [t[idx]+random.gauss(0,std_noise) for idx in range(n)]
```

    [-1.0, -0.9, -0.8, -0.7, -0.6, -0.5, -0.4, -0.3, -0.2, -0.1, 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
    [2.0, 1.6300000000000001, 1.3200000000000003, 1.0699999999999998, 0.8800000000000001, 0.75, 0.68, 0.67, 0.72, 0.83, 1.0, 1.23, 1.52, 1.87, 2.2800000000000002, 2.75, 3.2800000000000002, 3.8699999999999997, 4.5200000000000005, 5.23]

Python plotting library [matplotlib](https://matplotlib.org/) is used to plot the observed target values *t_observed* with respect at each sampling location $x$ in blue points [$x$,*t_observed*]. We can also plot noise-free "true" target values $t$ versus the same $x$. This represents the underlying model as a curve, hence *curve fitting*.

```python
# this line is useful for jupyter notebook only
%matplotlib inline

import matplotlib.pyplot as plt

# plot the curve and the noise-corrupted data
plt.plot(x,t,'r')
plt.plot(x,t_observed,'bo')
```

## 1.2 Model Fitting
For numerical computations such as curve fitting, we can use the powerful [numpy](http://www.numpy.org/). In this tutorial, we use [*numpy.linalg.lstsq*](https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.linalg.lstsq.html#numpy.linalg.lstsq) to solve [a system of linear equations](https://en.wikipedia.org/wiki/System_of_linear_equations) $\textbf{a}\textbf{x}=\textbf{b}$ for a [least sqaures solution](https://en.wikipedia.org/wiki/Least_squares).

Re-arrange the observed data to a linear system in matrix form "$\textbf{a}\textbf{x}=\textbf{b}$" (N.B. the $\textbf{x}$ is the unknown here, not the learning input $x$):

\begin{equation}
\textbf{t} = \textbf{X}\textbf{w}
\end{equation}

that is:

\begin{equation}
\begin{vmatrix}
t_{(1)} \\ t_{(2)} \\ \cdots \\ t_{(n)}
\end{vmatrix} = \begin{vmatrix}
x^2_{(1)} & x_{(1)} & 1 \\
x^2_{(2)} & x_{(2)} & 1 \\
& \cdots \\
x^2_{(n)} & x_{(n)} & 1
\end{vmatrix} \cdot
\begin{vmatrix}
w_2 \\ w_1 \\ w_0
\end{vmatrix}
\end{equation}

```python
import numpy as np

# then use nunmpy for a least-square solution to the linear system "Xw=t"
t_observed = np.reshape(t_observed, [-1, 1])
x_1 = np.reshape(x, [-1, 1])
x_2 = np.square(x_1)
x_0 = np.ones_like(x_1)
X = np.concatenate([x_2,x_1,x_0],1)
# print to check the inputs
print(X)
print(X.shape)
print(t_observed.shape)
w_estimate = np.linalg.lstsq(X, t_observed, rcond=None)
print(w_estimate[0])  # print the output

# plot to see the estimated curve, i.e.
# t_estimate = [xn**2*w_estimate[2]+xn*w_estimate[1]+w_estimate[0] for xn in x]
# but matrix multiplication is more compact:
t_estimate = np.matmul(X,w_estimate[0])
```

    [[ 1.   -1.    1.  ]
     [ 0.81 -0.9   1.  ]
     [ 0.64 -0.8   1.  ]
     [ 0.49 -0.7   1.  ]
     [ 0.36 -0.6   1.  ]
     [ 0.25 -0.5   1.  ]
     [ 0.16 -0.4   1.  ]
     [ 0.09 -0.3   1.  ]
     [ 0.04 -0.2   1.  ]
     [ 0.01 -0.1   1.  ]
     [ 0.    0.    1.  ]
     [ 0.01  0.1   1.  ]
     [ 0.04  0.2   1.  ]
     [ 0.09  0.3   1.  ]
     [ 0.16  0.4   1.  ]
     [ 0.25  0.5   1.  ]
     [ 0.36  0.6   1.  ]
     [ 0.49  0.7   1.  ]
     [ 0.64  0.8   1.  ]
     [ 0.81  0.9   1.  ]]
    (20, 3)
    (20, 1)
    [[3.04570644]
     [1.85739825]
     [0.97774253]]

*w_estimate* are the best estimates of the weights $w$ - in a least squares sense. Finally, we can add a green curve (formed by points [$x$, *t_estimate*]) to the previous plot to visualise the estimated polynomial.

```python
plt.plot(x,t,'r')
plt.plot(x,t_observed,'bo')
plt.plot(x,t_estimate,'g')
```

## 1.3 Model Fitting Error
From the above plot, we can see that the green curve (i.e. the fitted polynomial model) does not go through all the observed points (in blue) which were used in fitting the model. This is known as redidual error, also known as training error. There is also discrepancy between the "true" curve (in red) and the estimated one (in green), but this difference is often impossible to obtain without access to the true underlying model.

```python
# residuals:
Residuals = t_estimate-t_observed
SR = np.sum(np.square(Residuals))  # sums of residuals: b - a*x
# root-mean-square error
RMSE = np.sqrt(np.mean(np.square(Residuals)))
print(SR)
print(RMSE)

# plot the error distribution
plt.hist(Residuals)
```

## Questions
### Sample size
- Effect on changing the sample size on model fitting and its errors.
- How many samples are needed?

### Model fitting
- What is the objective (loss) function in this curve fitting problem?
- How does change of the noise level (i.e. *std_noise*) affect the model fitting?
- Are residualds a good estimate of the difference to the "true" target values?
- How to measure how good the estimates are?


## Kepler problem

```python
import sympy
sympy.init_printing()

import numpy

import matplotlib.pyplot as plt

import sys
sys.path.insert(0, './code')

from gauss_legendre import gauss_legendre
from evaluate_functional import evaluate_functional
```

```python
# state vector (symbolic)
q_x = sympy.Symbol('q_x')
q_y = sympy.Symbol('q_y')
p_x = sympy.Symbol('p_x')
p_y = sympy.Symbol('p_y')
x = [q_x, q_y, p_x, p_y]
```

```python
# physical parameters of Hamiltonian function
from sympy.abc import m, G
params = {
    # mass of the planet (kg)
    m: 4.0,

    # gravitational constant (m**3/s**2)
    G: 10.0
}
```

```python
# Hamiltonian function
# (kinetic energy T + potential energy V)
T = sympy.Rational(1,2) * (p_x**2 / m + p_y**2 / m)
V = - G * m / sympy.sqrt(q_x**2 + q_y**2)
H = T + V
H
```

```python
# efforts
e = sympy.Matrix([H.diff(co) for co in x])
e
```

```python
# structure matrix
J = sympy.Matrix([[0, 0, 1, 0],
                  [0, 0, 0, 1],
                  [-1, 0, 0, 0],
                  [0, -1, 0, 0]])
```

```python
# time evolution
xdot = J @ e
xdot
```

### compute stationary orbit

$F_G \overset{!}{=} F_{\omega}$

$\frac{G \, m}{r^2} \overset{!}{=} m \, \omega^2 \, r$

$\omega = \frac{G}{r^3}$

```python
r = 5.0
ω = numpy.sqrt(params[G]/r**3)
```

```python
# initial condition
x_0 = numpy.array([0.0, r, params[m] * r * ω, 0.0])

# duration of simulation
t_f = 260.0

# time step
dt = 0.1

# order of collocation polynomial
s = 2
```

```python
%time time, solution = gauss_legendre(x, xdot, x_0, t_f, dt, s, params=params)
```

    CPU times: user 1.03 s, sys: 5.76 ms, total: 1.04 s
    Wall time: 1.04 s

```python
# plot trajectory (x-y-plane)
fig, ax = plt.subplots(dpi=200)
ax.set_aspect('equal')
ax.plot(solution[:, 0], solution[:, 1]);
```

```python
# relative error of energy conservation
energy = evaluate_functional(x, H, solution, params=params)
abs(energy[0] - energy[-1]) / energy[0]
```


```python
from sympy import *
```

```python
y1, y2, y3, y4, z = symbols('y1 y2 y3 y4 z')
```

```python
# Problem equationsal formulation
eqs = [y1 + 2*y2 + 3*y3 + 3*y4 - z , y1 + y2 + 2*y3 + y4 - 3]
eqs = eqs + [x*(x-1) for x in [y1,y2,y3,y4]]
print(eqs)
```

    [y1 + 2*y2 + 3*y3 + 3*y4 - z, y1 + y2 + 2*y3 + y4 - 3, y1*(y1 - 1), y2*(y2 - 1), y3*(y3 - 1), y4*(y4 - 1)]

```python
#Computing Groebner basis
result = groebner(eqs, y1, y2, y3, y4, z, order='lex')
list(result)
```

    [2*y1 + 2*y3 - z**2 + 11*z - 32,
     y2 + y3 + z**2 - 10*z + 23,
     y3**2 - y3,
     y3*z - 6*y3 - z + 6,
     2*y4 - z**2 + 9*z - 20,
     z**3 - 15*z**2 + 74*z - 120]

```python
# Solving the last equation of the Groebner basis
solve(list(result)[-1], z)
```

    [4, 5, 6]

```python

```

notebooks/unused/Groebner basis simple example.ipynb


```python
import loader
from sympy import *
init_printing()
from root.solver import *
```

#### Solve $t^2 y' + 2ty - y^3 = 0$
First convert to standard form: $y' + \frac{2}{t} = \frac{1}{t^2} y^3$

```python
_, p = first_order_bernoulli(2 / t, 1 / (t ** 2), 2)
p.display()
```

$\displaystyle \text{Use the substitution }v = y^{1-n}\text{, }y = v y^n$

$\displaystyle \frac{d}{d t} v{\left (t \right )} = - \frac{\frac{d}{d t} y{\left (t \right )}}{y^{2}{\left (t \right )}}\Rightarrow\frac{d}{d t} y{\left (t \right )} = - y^{2}{\left (t \right )} \frac{d}{d t} v{\left (t \right )}$

$\displaystyle \text{Substitute }y'(t)\text{ into the original equation}$

$\displaystyle - y^{2}{\left (t \right )} \frac{d}{d t} v{\left (t \right )} + \frac{2 y{\left (t \right )}}{t} = \frac{y^{2}{\left (t \right )}}{t^{2}}$

$\displaystyle \text{Divide both sides by }\frac{y^{n}}{n - 1}$

$\displaystyle \frac{d}{d t} v{\left (t \right )} - \frac{2}{t y{\left (t \right )}} = - \frac{1}{t^{2}}$

$\displaystyle \text{Note that }v = y^{1-n}$

$\displaystyle \frac{d}{d t} v{\left (t \right )} - \frac{2 v{\left (t \right )}}{t} = - \frac{1}{t^{2}}$

$\displaystyle \text{Now we need to solve the linear ODE in }v$

$\displaystyle \text{Calculate the integrating factor }\mu$

$\displaystyle \mu = e^{\int \left(- \frac{2}{t}\right)\, dt} = \frac{1}{t^{2}}$

$\displaystyle \text{Multiply both sides of the equation by }\mu$

$\displaystyle \frac{1}{t^{2}} \frac{d}{d t} v{\left (t \right )} - \frac{2 v{\left (t \right )}}{t^{3}} = - \frac{1}{t^{4}}$

$\displaystyle \frac{d}{d t} \frac{v{\left (t \right )}}{t^{2}} = - \frac{1}{t^{4}}$

$\displaystyle \frac{v{\left (t \right )}}{t^{2}} = C + \frac{1}{3 t^{3}}$

$\displaystyle v{\left (t \right )} = C t^{2} + \frac{1}{3 t}$

$\displaystyle \text{Use }y = v^{\frac{1}{1 - n}}\text{ to solve for }y$

$\displaystyle y{\left (t \right )} = \frac{3 t}{3 C t^{3} + 1}$

```python

```


```python
import sympy as sym
from sympy.physics.units.quantities import Quantity
from IPython.display import display, Math, Latex

# Specially for linear models
class HasforLind:

    def __init__(self, function, *args):
        self.variables = []
        self.c = []
        self.L = None
        self.alpha = []
        self.y_star = []
        self.beta = Quantity("β")
        self.lamb = Quantity("λ")
        for variable in args:
            self.variables.append(variable)
        self.function = function
        self.partial_derivative()
        self.calc_length()
        self.calc_alpha()
        self.calc_y_star()
        self.set_equation()

    def partial_derivative(self):
        i = 0
        display(Math(r"c_i = \frac{\partial G}{\partial y_i}"))
        for variable in self.variables:
            self.c.append(self.lamb*sym.diff(self.function, variable))
            display(Math(r"c_" + str(i+1) + " = " + str(self.c[i])))
            i += 1

    def calc_length(self):
        summation = 0
        for c in self.c:
            summation += c**2
        self.L = summation**0.5
        display(Math(r"L = " + str(self.L)))

    def calc_alpha(self):
        display(Math(r"\alpha_i = \frac{c_i}{L}"))
        i = 0
        for c in self.c:
            self.alpha.append(c/self.L)
            display(Math(r"\alpha_" + str(i+1) + " = " + str(self.alpha[i])))
            i += 1

    def calc_y_star(self):
        i = 0
        display(Math(r"y^*_i = -\beta \times \alpha_i"))
        for alpha in self.alpha:
            self.y_star.append(-1*alpha*self.beta)
            display(Math(r"y^*_" + str(i+1) + " = " + str(self.y_star[i])))
            i += 1

    def set_equation(self):
        arguments = {}
        for index in range(len(self.y_star)):
            arguments[self.variables[index]] = self.y_star[index]
        self.final_equation = self.function.subs(arguments)
        display(Math(str(self.final_equation) + str(" = 0")))

    def get_equation(self):
        return self.final_equation

    def get_beta(self):

        solve = sym.solvers.solve(self.get_equation(), self.beta)
        display(Math(str(self.beta) + " = " + str(solve[0])))
        return solve

x_1 = sym.Symbol("x_1")
x_2 = sym.Symbol("x_2")
f = 10*x_1 - 4*x_2 + 12

hasfor_lind = HasforLind(f, x_1, x_2)
beta = hasfor_lind.get_beta()
```

$\displaystyle c_i = \frac{\partial G}{\partial y_i}$

$\displaystyle c_1 = 10*λ$

$\displaystyle c_2 = -4*λ$

$\displaystyle L = 10.770329614269*λ**1.0$

$\displaystyle \alpha_i = \frac{c_i}{L}$

$\displaystyle \alpha_1 = 0.928476690885259$

$\displaystyle \alpha_2 = -0.371390676354104$

$\displaystyle y^*_i = -\beta \times \alpha_i$

$\displaystyle y^*_1 = -0.928476690885259*β$

$\displaystyle y^*_2 = 0.371390676354104*β$

$\displaystyle 12 - 10.770329614269*β = 0$

$\displaystyle β = 1.11417202906231$


```python
import loader
from sympy import *
init_printing()
from root.solver import *
```

#### Given that $y_1 = \frac{4}{t}$ is a solution of the differential equation $t^2 y'' + 3t y' + y = 0$, find the other solution using reduction of order.

```python
t = Symbol('t')
y, p = red_order(4 / t, *to_std(t**2, 3*t, 1), 0, t)
p.display_ipython()
```

$\displaystyle \text{Solution }y_2\text{ takes the form }y_2 = v(t)y_1$

$\displaystyle y_{2} = \frac{4 v{\left (t \right )}}{t}$

$\displaystyle \text{Calculate the derivatives}$

$\displaystyle \begin{align*}&\frac{d}{d t} \frac{4 v{\left (t \right )}}{t} = \frac{4}{t} \frac{d}{d t} v{\left (t \right )} - \frac{4 v{\left (t \right )}}{t^{2}}\\&\frac{d^{2}}{d t^{2}} \frac{4 v{\left (t \right )}}{t} = \frac{4}{t} \frac{d^{2}}{d t^{2}} v{\left (t \right )} - \frac{8}{t^{2}} \frac{d}{d t} v{\left (t \right )} + \frac{8 v{\left (t \right )}}{t^{3}}\\\end{align*}$

$\displaystyle \text{Plug in the derivatives and simplify LHS of }y'' + p(t)y' + q(t)y = g(t)$

$\displaystyle \begin{align*}&\frac{3}{t} \left(\frac{4}{t} \frac{d}{d t} v{\left (t \right )} - \frac{4 v{\left (t \right )}}{t^{2}}\right) + \frac{4}{t} \frac{d^{2}}{d t^{2}} v{\left (t \right )} - \frac{8}{t^{2}} \frac{d}{d t} v{\left (t \right )} + \frac{12 v{\left (t \right )}}{t^{3}} = 0\\&\frac{4}{t} \frac{d^{2}}{d t^{2}} v{\left (t \right )} + \frac{4}{t^{2}} \frac{d}{d t} v{\left (t \right )} = 0\\\end{align*}$

$\displaystyle \text{Given that }y_1(t)\text{ satisfies the homogeneous equation } y_1''(t) + p(t)y_1' + q(t)y_1 = 0$

$\displaystyle \frac{4}{t} \frac{d^{2}}{d t^{2}} v{\left (t \right )} + \frac{4}{t^{2}} \frac{d}{d t} v{\left (t \right )} = 0$

$\displaystyle \text{Let }w(t) = v'(t)\text{ and convert to standard form}$

$\displaystyle \begin{align*}&\frac{4}{t} \frac{d}{d t} w{\left (t \right )} + \frac{4 w{\left (t \right )}}{t^{2}} = 0\\&\frac{d}{d t} w{\left (t \right )} + \frac{w{\left (t \right )}}{t} = 0\\\end{align*}$

$\displaystyle \text{Solve the first order linear differential equation in }w(t)$

$\displaystyle w{\left (t \right )} = \frac{C_{1}}{t}$

$\displaystyle v(t) = \int \frac{C_{1}}{t}\, dt = C_{1} \ln{\left (t \right )} + C_{2}$

$\displaystyle y_{2} = \frac{4 C_{1}}{t} \ln{\left (t \right )} + \frac{4 C_{2}}{t} = \frac{1}{t} \left(C_{1} \ln{\left (t \right )} + C_{2}\right)$

#### Given that $y_1 = e^{-\frac{b}{2a}}$ is a solution of the differential equation $a y'' + b y' + c y = 0$, find the other solution using reduction of order.

```python
a, b, c, t = symbols('a b c t', real=True)
y1 = exp(-b / (2*a) * t)
y, p = red_order(y1, *to_std(a, b, c), 0, t)
p.display_ipython()
```

$\displaystyle \text{Solution }y_2\text{ takes the form }y_2 = v(t)y_1$

$\displaystyle y_{2} = v{\left (t \right )} e^{- \frac{b t}{2 a}}$

$\displaystyle \text{Calculate the derivatives}$

$\displaystyle \begin{align*}&\frac{\partial}{\partial t} v{\left (t \right )} e^{- \frac{b t}{2 a}} = e^{- \frac{b t}{2 a}} \frac{d}{d t} v{\left (t \right )} - \frac{b v{\left (t \right )} e^{- \frac{b t}{2 a}}}{2 a}\\&\frac{\partial^{2}}{\partial t^{2}} v{\left (t \right )} e^{- \frac{b t}{2 a}} = e^{- \frac{b t}{2 a}} \frac{d^{2}}{d t^{2}} v{\left (t \right )} - \frac{b e^{- \frac{b t}{2 a}}}{a} \frac{d}{d t} v{\left (t \right )} + \frac{b^{2} v{\left (t \right )} e^{- \frac{b t}{2 a}}}{4 a^{2}}\\\end{align*}$

$\displaystyle \text{Plug in the derivatives and simplify LHS of }y'' + p(t)y' + q(t)y = g(t)$

$\displaystyle \begin{align*}&e^{- \frac{b t}{2 a}} \frac{d^{2}}{d t^{2}} v{\left (t \right )} + \frac{b}{a} \left(e^{- \frac{b t}{2 a}} \frac{d}{d t} v{\left (t \right )} - \frac{b v{\left (t \right )} e^{- \frac{b t}{2 a}}}{2 a}\right) - \frac{b e^{- \frac{b t}{2 a}}}{a} \frac{d}{d t} v{\left (t \right )} + \frac{c e^{- \frac{b t}{2 a}}}{a} v{\left (t \right )} + \frac{b^{2} v{\left (t \right )} e^{- \frac{b t}{2 a}}}{4 a^{2}} = 0\\&\left(\frac{c e^{- \frac{b t}{2 a}}}{a} - \frac{b^{2} e^{- \frac{b t}{2 a}}}{4 a^{2}}\right) v{\left (t \right )} + e^{- \frac{b t}{2 a}} \frac{d^{2}}{d t^{2}} v{\left (t \right )} = 0\\\end{align*}$

$\displaystyle \text{Given that }y_1(t)\text{ satisfies the homogeneous equation } y_1''(t) + p(t)y_1' + q(t)y_1 = 0$

$\displaystyle e^{- \frac{b t}{2 a}} \frac{d^{2}}{d t^{2}} v{\left (t \right )} = 0$

$\displaystyle \text{Let }w(t) = v'(t)\text{ and convert to standard form}$

$\displaystyle \begin{align*}&e^{- \frac{b t}{2 a}} \frac{d}{d t} w{\left (t \right )} = 0\\&\frac{d}{d t} w{\left (t \right )} = 0\\\end{align*}$

$\displaystyle \text{Solve the first order linear differential equation in }w(t)$

$\displaystyle w{\left (t \right )} = C_{1}$

$\displaystyle v(t) = \int C_{1}\, dt = C_{1} t + C_{2}$

$\displaystyle y_{2} = C_{1} t e^{- \frac{b t}{2 a}} + C_{2} e^{- \frac{b t}{2 a}} = \left(C_{1} t + C_{2}\right) e^{- \frac{b t}{2 a}}$