# Newton's Method for Finding Equation Roots

Newton's method, also known as Newton-Raphson, is an approach for finding the roots of nonlinear equations and is one of the most common root-finding algorithms due to its relative simplicity and speed. The [root of a function](http://mathworld.wolfram.com/Root.html) is the point at which $f(x) = 0$. Many equations have more than one root. Every real polynomial of odd degree has an odd number of real roots ("Zero of a function," 2016). Newton-Raphson is an iterative method that begins with an initial guess of the root. The method uses the derivative of the function $f'(x)$ as well as the original function $f(x)$, and thus only works when the derivative can be determined.

## Newton-Raphson Iteration

The initial guess of the root is typically denoted $x_0$ with the true root represented by $r$. The true root can thus be expressed as $r = x_0 + h$, and therefore $h = r - x_0$, where $h$ measures how far the guess is from the true value of the root. As $h$ will be small, a linear tangent line is used to approximate the location of the root and can be written as:

$$ 0 = f(r) = f(x_0 + h) \approx f(x_0) + hf'(x_0) $$

where $h$ is approximately:

$$ h \approx -\frac{f(x_0)}{f'(x_0)} $$

Unless the derivative $f'(x_o)$ is close to 0. Combining this approximation with the value of the true root yields:

$$ r = x_0 + h \approx x_0 -\frac{f(x_0)}{f'(x_0)} $$

Therefore the new estimate of $r$, $x_1$, becomes:

$$ x_1 = x_0 - \frac{f(x_0)}{f'(x_0)} $$

Then the new estimate $x_2$ is obtained from $x_1$ in the same manner:

$$ x_2 = x_1 - \frac{f(x_1)}{f'(x_1)} $$

The iteration of Newton-Raphson can thus be generalized:

$$ x_{n + 1} = x_n - \frac{f(x_n)}{f'(x_n)} $$

## Newton-Raphson Convergence

Newton-Raphson is not a foolproof method in that it can fail to converge to a solution. In fact, there are no 'perfect' numerical methods that will always converge on a solution. Particularly, the assumption $f''(x)$ exists and is continuous near $r$ must be made.

The method can also fail to converge when $f'(x)$ is close to 0. For example, 5.02 is close to 5 as it is only `r (5.02 - 5) / 5 * 100`% different, however, $5.02/10^{-7}$, is quite different than $5/10^{-7}$.

## Examples of Newton-Raphson

An example will help illuminate further on the definitions and equations above. The NR method can be used to approximate square roots such as $\sqrt{10}$. The square root of 10 is about three, so we can use that as a good starting value.

It often helps to plot the function to see where the roots occur. The function is first rearranged to be an expression of $f(x)$ before plotting.

$$ x = \sqrt{10} $$

$$ x - \sqrt{10} = 0 $$

$$ f(x) = x^2 - 10 $$

```python
from sympy import symbols, limit, diff, sin, cos, log, tan, sqrt, init_printing, plot, oo
from mpmath import ln, e, pi

init_printing()
x = symbols('x')
y = symbols('y')
```

```python
plot(x ** 2 - 10)
```

It can be seen from the graph the function crosses the x-axis at $\sqrt{10}$ on an interval [3, 4]. The derivative of the function is $2x$.

Solving for the function root using the Newton-Raphson method proceeds as follows using three as an initial guess.

$$ x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)} = x_n - \frac{x_n^2 - 10}{2x_n} $$

$$ x_{1} = 3 - \frac{(3)^2 - 10}{2(3)} = 3.166667 $$

$$ x_{2} = 3.166667 - \frac{(3.166667)^2 - 10}{2(3.166667)} = 3.162281 $$

$$ x_{3} = 3.162281 - \frac{(3.162281)^2 - 10}{2(3.162281)} = 3.162278 $$

And so on until the ${x_n}$ estimates are within a particular level of tolerance. This example converges in three iterations. Thus $3.162278$ is the estimated root of the function. This result can be verified by simply taking the square root of 10.

```python
10 ** (1/2) # math.sqrt(10)
```

## Newton-Raphson Method in Python

As an example, suppose the equation we are interested in locating the roots is:

$$ f(x) = x^3 - 2x - 5 $$

```python
def f(x):
    return x ** 3 - 2 * x - 5
```

Plot the function to visualize how the equation behaves and where any roots may be located.

```python
plot(x ** 3 - 2 * x - 5, xlim=(-5,5), ylim=(-10,5))
```

```python
from mathpy.numerical.roots import newtonraph
from scipy.optimize import newton
```

It looks like the function equals 0 when $y$ is about 2. To find the root of the equation, use the `uniroot` function with a starting value of 2 and upper bound of 3.

```python
newtonraph(f, 2)[0]
```

```python
newton(f,2)
```

As suspected, the root of the function is very close to 2 at $2.09455$. The iterations can be written as follows:

$$ x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)} = x_n - \frac{x_n^3 - 2x - 5}{3x_n^2 - 2} $$

$$ x_{1} = 2 - \frac{(2)^3 - 2(2) - 5}{3(2)^2 - 2} = 1.281239 $$

$$ x_{2} = 1.281239 - \frac{(1.281239)^3 - 2(1.281239) - 5}{3(1.281239)^2 - 2} = 3.147821 $$

$$ x_{3} = 3.147821 - \frac{(3.147821)^3 - 2(3.147821) - 5}{3(3.147821)^2 - 2} = 2.430257 $$

$$ x_{4} = 2.430257 - \frac{(2.430257)^3 - 2(2.430257) - 5}{3(2.430257)^2 - 2} = 2.144418 $$

$$ x_{5} = 2.144418 - \frac{(2.144418)^3 - 2(2.144418) - 5}{3(2.144418)^2 - 2} = 2.095897 $$

$$ x_{6} = 2.095897 - \frac{(2.095897)^3 - 2(2.095897) - 5}{3(2.095897)^2 - 2} = 2.094552 $$

## References

Agresti, A. (2002). Categorical data analysis (2nd ed.). New York, NY: Wiley-Interscience.

Kiusalaas, J. (2013). Numerical methods in engineering with python (2nd ed.). New York: Cambridge University Press.

The Newton-Raphson method. Retrieved from https://www.math.ubc.ca/~anstee/math104/104newtonmethod.pdf

Stewart, J. (2007). Essential calculus: Early transcendentals. Belmont, CA: Thomson Higher Education.

Zero of a function (2016). In Wikipedia. Retrieved from https://en.wikipedia.org/wiki/Zero_of_a_function#Polynomial_roots


# SymPy
**SymPy** is a Computer Algebra System (CAS) for Python. It does symbolic computation instead of numeric computation. This means that mathematical objects are represented exactly, not approximately as in the case of numerical representation.

Take the example of $\sqrt{8}$. When calculated numerically, we get the approximate answer 2.82842712475. But in SymPy it is represented as $2 \sqrt{2}$. Further, performing operations on such representations will continue to retain accuracy. Note that $\frac{\sqrt{8}}{\sqrt{3}}$ is simplified to $\frac{2 \sqrt{6}}{3}$, retaining full accuracy. You can numerically evaluate any expression with the **`N()`** function in SymPy.

```python
from sympy import *
import math

x, y, z, t = symbols('x y z t')          # Symbols representing real numbers
k, m, n = symbols('k m n', integer=True) # Symbols representing integers
f, g, h = symbols('f g h', cls=Function) # Symbols repesenting function names
init_printing()

print math.sqrt(8)
print sqrt(8)
print math.sqrt(8) / math.sqrt(3)
print sqrt(8) / sqrt(3)
print N(sqrt(8) / sqrt(3))  # Numerical evaluation
print N(sqrt(8. / 3.))
```

    2.82842712475
    2*sqrt(2)
    1.63299316186
    2*sqrt(6)/3
    1.63299316185545
    1.63299316185545

## Numerical Simplification

```python
print nsimplify(0.1)
print nsimplify(6.28, [pi], tolerance=0.01)
print nsimplify(pi, tolerance=0.1)
print nsimplify(pi, tolerance=0.001)
```

    1/10
    2*pi
    22/7
    355/113

## Algebra
SymPy can handle algebraic expressions, simplify them and evaluate them.

```python
eq = ((x+y)**2 * (x+1))
eq
```

```python
expand(eq)
```

You can substitute a numerical value for any of the symbols and simplify the expression. The method to do this is **`subs()`**. It takes two arguments, the symbol and the numerical value it is to assume. If an expression has more than one symbol, substitution must be done one symbol at a time.

```python
eq.subs(x, 1).subs(y,1)
```

```python
a = 1/x + (x*sin(x) - 1)/x
a
```

```python
N(a.subs(x, 1))
```

```python

```

## Integral Calculus

SymPy performs integration symbolically, like you would if you were doing so by hand rather than numerically.

```python
a = Integral(cos(x), x)
Eq(a, a.doit())
```

```python
b = Integral(sqrt(1/x), x)
Eq(b, b.doit())
```

Here is how we can evaluate the definite integral $\int_0^{\infty} e^{-x} dx$

```python
b = Integral(x**2+2*x+3, x)
Eq(b, b.doit())
integrate(b, (x, 0, 1))
```

```python
integrate(exp(-x), (x, 0, oo))
```

Here is the definite integral $\int_{-\infty}^{\infty} -x^2 - y^2 dx$

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

```python
c = Integral(exp(-x**2), (x, 0, 1))
Eq(c, c.doit())
```

```python
N(c.doit())
```

## Differential Calculus

SymPy can also perform differentiation symbolically. Derivative of $y = x^2 + 3x - \frac{1}{2}$ is $y' = 2x + 3$. Substituting $x=2$ in the derivative results in the numerical value $y'(2) = 7$.

```python
s = "x**2 + 3*x - 1/2"
c = sympify(s)
print c
d = diff(c)
print d
d.subs(x, 2)
```

It is possible to differentiate a function multiple times and obtain the second or higher derivatives.

```python
print diff(x**4)
print diff(x**4, x, x) # Differentiate w.r.t. x two times
print diff(x**4, x, 2) # Differentiate w.r.t. x two times
```

    4*x**3
    12*x**2
    12*x**2

A function of two or more variables can be differentiated with respect to any of the variables any number of times.

```python
expr = exp(x*y*z)
deriv = diff(expr, x, y, z)
print deriv
```

    (x**2*y**2*z**2 + 3*x*y*z + 1)*exp(x*y*z)

## Limits

SymPy can evaluate limits of functions.

```python
print limit(sin(x)/x, x, 0)
print limit(tan(x)/x, x, 0)
```

    1
    1

```python
expr = x**2 / exp(x)
print expr.subs(x, oo)
print limit(expr, x, oo)
```

    nan
    0

```python
expr = Limit((cos(x) -1) / x, x, 0)
expr
```

```python
expr.doit()
```

```python
c = Limit((-x + sin(x)) / (x * cos(x) - sin(x)), x, 0)
c
```

```python
r = Limit(x * (sin(x) - x * cos(x)) / (2*(1-cos(x)) - x * sin(x)), x, 0)
r
```

```python
print c.doit()
print r.doit()
```

    1/2
    4

## Solution of Equations

To solve the equation $x^2 = 1$, first form the equation with the **`Eq()`** SymPy function by defining the left and right hand sides. Then solve the equation by calling the SymPy function **`solve()`**.

```python
solve(Eq(x**2, 1), x)
```

The same equation could also be expressed as $x^2 - 1 = 0$ and solved as show below:

```python
solve(Eq(x**2 - 1, 0), x)
```

Since it is a common form to have zero on the right hand side, SymPy allows you to dispense with the **`Eq()`** function call to form the equation and solve the equation directly as follows:

```python
solve(x**2 - 1, x)
```

Let us now solve the polynomial equation $x^2 - x = 0$

```python
print solve(x**2 - x, x)
```

    [0, 1]

For polynomial equations, **`solve`** prints repeated roots, if any, only once. The function **`roots()`** prints the roots and their frequency.

```python
print solve(x**3 - 6*x**2 + 9*x, x)
print roots(x**3 - 6*x**2 + 9*x, x)
```

    [0, 3]
    {0: 1, 3: 2}

Differential equations can be solved using the SymPy function **`dsolve()`**. Let us first represent the differential equation $f''(x) - 2 f'(x) + f(x) = \sin(x)$ as follows using **`Eq()`**, and then solve it using **`dsolve()`**:

```python
diffeq = Eq(f(x).diff(x, x) - 2 * f(x).diff(x) + f(x), sin(x))
diffeq
```

```python
dsolve(diffeq, f(x))
```

In the above solution $C_1$ and $C_2$ are arbitrary constants of integration which will have to be determined by applying known conditions.

```python

```


# Predicting the energy in the one-dimensional Ising model

We will in this notebook use linear (ordinary least squares), ridge and LASSO regression to predict the energy in the nearest neighbor one-dimensional Ising model on a ring, i.e., the endpoints wrap around. We will use the linear regression models to fit a value for the coupling constant to achieve this.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable
import seaborn as sns
import scipy.linalg as scl
from sklearn.model_selection import train_test_split
import sklearn.linear_model as skl
import tqdm

%matplotlib inline

sns.set(color_codes=True)
cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
```

## The one-dimensional Ising model

The one-dimensional Ising model with nearest neighbor interaction, no external field and a constant coupling constant $J$ is given by

\begin{align}
    H = -J \sum_{k}^L s_k s_{k + 1},
\end{align}

where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins in the system is determined by $L$. For the low temperature limit there is no phase transition.

We will look at a system of $L = 40$ spins with a coupling constant of $J = 1$. To get enough training data we will generate 10000 states with their respective energies.

```python
L = 40
n = int(1e4)

spins = np.random.choice([-1, 1], size=(n, L))
J = 1.0

energies = np.zeros(n)

for i in range(n):
    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
```

## Reformulating the problem to suit regression

A more general form for the one-dimensional Ising model is

\begin{align}
    H = - \sum_j^L \sum_k^L s_j s_k J_{jk}.
\end{align}

Here we allow for interactions beyond the nearest neighbors and a more adaptive coupling matrix. This latter expression can be formulated as a matrix-product on the form

\begin{align}
    H = X J,
\end{align}

where $X_{jk} = s_j s_k$ and $J$ is the matrix consisting of the elements $-J_{jk}$. This form of writing the energy fits perfectly with the form utilized in linear regression, viz.

\begin{align}
    y = X\omega + \epsilon,
\end{align}

where $\omega$ are the weights we wish to fit and $\epsilon$ is noise with zero-mean. In the case of the Ising model we have $\sigma = 0$.

```python
X = np.zeros((n, L ** 2))
for i in range(n):
    X[i] = np.outer(spins[i], spins[i]).ravel()
```

```python
y = energies
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96)

X_train_own = np.concatenate(
    (np.ones(len(X_train))[:, np.newaxis], X_train),
    axis=1
)

X_test_own = np.concatenate(
    (np.ones(len(X_test))[:, np.newaxis], X_test),
    axis=1
)
```

## Linear regression

The problem at hand is to try to fit the equation

\begin{align}
    y = f(x) + \epsilon,
\end{align}

where $f(x)$ is some unknown function of the data $x$ and $\epsilon$ is normally distributed with mean zero noise with standard deviation $\sigma_{\epsilon}$. Our job is to try to find a predictor which estimates the function $f(x)$. In linear regression we assume that we can formulate the problem as

\begin{align}
    y = X\omega + \epsilon,
\end{align}

where $X$ and $\omega$ are now matrices. Our job at hand is now to find a _cost function_ $C$, which we wish to minimize in order to find the best estimate of $\omega$.

### Ordinary least squares

In the ordinary least squares method we choose the cost function

\begin{align}
    C(X, \omega) = ||X\omega - y||^2
    = (X\omega - y)^T(X\omega - y)
\end{align}

We then find the extremal point of $C$ by taking the derivative with respect to $\omega$ and setting it to zero, i.e.,

\begin{align}
    \dfrac{\mathrm{d}C}{\mathrm{d}\omega}
    = 0.
\end{align}

This yields the expression for $\omega$ to be

\begin{align}
    \omega = \frac{X^T y}{X^T X},
\end{align}

which immediately imposes some requirements on $X$ as there must exist an inverse of $X^T X$. If the expression we are modelling contains an intercept, i.e., a constant expression we must make sure that the first column of $X$ consists of $1$.

```python
def get_ols_weights_naive(x: np.ndarray, y: np.ndarray) -> np.ndarray:
    return scl.inv(x.T @ x) @ (x.T @ y)
```

```python
omega = get_ols_weights_naive(X_train_own, y_train)
```

Hmmm, doing the inversion directly turns out to be a bad idea as the matrix $X^TX$ is singular. An alternative approach is to use the _singular value decomposition_. Using the definition of the Moore-Penrose pseudoinverse we can write the equation for $\omega$ as

\begin{align}
    \omega = X^{+}y,
\end{align}

where the pseudoinverse of $X$ is given by

\begin{align}
    X^{+} = \frac{X^T}{X^T X}.
\end{align}

Using singular value decomposition we have that $X = U\Sigma V^T$, where $X^{+} = V\Sigma^{+} U^T$. This reduces the equation for $\omega$ to

\begin{align}
    \omega = V\Sigma^{+} U^T y.
\end{align}

Note that solving this equation by actually doing the pseudoinverse (which is what we will do) is not a good idea as this operation scales as $\mathcal{O}(n^3)$, where $n$ is the number of elements in a general matrix. Instead, doing $QR$-factorization and solving the linear system as an equation would reduce this down to $\mathcal{O}(n^2)$ operations.

```python
def get_ols_weights(x: np.ndarray, y: np.ndarray) -> np.ndarray:
    u, s, v = scl.svd(x)
    return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y
```

Before passing in the data to the function we append a column with ones to the training data.

```python
omega = get_ols_weights(X_train_own,y_train)
```

Next we fit a `LinearRegression`-model from Scikit-learn for comparison.

```python
clf = skl.LinearRegression().fit(X_train, y_train)
```

Extracting the $J$-matrix from both our own method and the Scikit-learn model where we make sure to remove the intercept.

```python
J_own = omega[1:].reshape(L, L)
J_sk = clf.coef_.reshape(L, L)
```

A way of looking at the coefficients in $J$ is to plot the matrices as images.

```python
fig = plt.figure(figsize=(20, 14))
im = plt.imshow(J_own, **cmap_args)
plt.title("Home-made OLS", fontsize=18)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
cb = fig.colorbar(im)
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)

fig = plt.figure(figsize=(20, 14))
im = plt.imshow(J_sk, **cmap_args)
plt.title("LinearRegression from Scikit-learn", fontsize=18)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
cb = fig.colorbar(im)
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)

plt.show()
```

We can see that our model for the least squares method performes close to the benchmark from Scikit-learn. It is interesting to note that OLS considers both $J_{j, j + 1} = -0.5$ and $J_{j, j - 1} = -0.5$ as valid matrix elements for $J$.

### Ridge regression

Having explored the ordinary least squares we move on to ridge regression. In ridge regression we include a _regularizer_. This involves a new cost function which leads to a new estimate for the weights $\omega$. This results in a penalized regression problem. The cost function is given by

\begin{align}
    C(X, \omega; \lambda) = ||X\omega - y||^2 + \lambda ||\omega||^2
    = (X\omega - y)^T(X\omega - y) + \lambda \omega^T\omega.
\end{align}

Finding the extremum of this function yields the weights

\begin{align}
    \omega(\lambda) = \frac{X^Ty}{X^TX + \lambda} \to \frac{\omega_{\text{LS}}}{1 + \lambda},
\end{align}

where $\omega_{\text{LS}}$ is the weights from ordinary least squares. The last assumption assumes that $X$ is orthogonal, which it is not. We will therefore resort to solving the equation as it stands on left hand side.

```python
def get_ridge_weights(x: np.ndarray, y: np.ndarray, _lambda: float) -> np.ndarray:
    return x.T @ y @ scl.inv(
        x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda
    )
```

```python
_lambda = 0.1
```

```python
omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda]))
```

```python
clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
```

```python
J_ridge_own = omega_ridge[1:].reshape(L, L)
J_ridge_sk = clf_ridge.coef_.reshape(L, L)
```

```python
fig = plt.figure(figsize=(20, 14))
im = plt.imshow(J_ridge_own, **cmap_args)
plt.title("Home-made ridge regression", fontsize=18)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
cb = fig.colorbar(im)
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)

fig = plt.figure(figsize=(20, 14))
im = plt.imshow(J_ridge_sk, **cmap_args)
plt.title("Ridge from Scikit-learn", fontsize=18)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
cb = fig.colorbar(im)
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)

plt.show()
```

In other words, our implementation of ridge regression seems to match well with the benchmark from Scikit-learn. We can also see the same symmetry pattern as for OLS.

### LASSO regression

In the _Least Absolute Shrinkage and Selection Operator_ (LASSO)-method we get a third cost function.

\begin{align}
    C(X, \omega; \lambda) =
    ||X\omega - y||^2 + \lambda ||\omega||
    = (X\omega - y)^T(X\omega - y) + \lambda \sqrt{\omega^T\omega}.
\end{align}

Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from Scikit-learn.

```python
clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train)
J_lasso_sk = clf_lasso.coef_.reshape(L, L)
```

```python
fig = plt.figure(figsize=(20, 14))
im = plt.imshow(J_lasso_sk, **cmap_args)
plt.title("Lasso from Scikit-learn", fontsize=18)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
cb = fig.colorbar(im)
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)

plt.show()
```

It is quite striking how LASSO breaks the symmetry of the coupling constant as opposed to ridge and OLS. We get a sparse solution with $J_{j, j + 1} = -1$.

## Performance of the different models

In order to judge which model performs best at varying values of $\lambda$ (for ridge and LASSO) we compute $R^2$ which is given by

\begin{align}
    R^2 = 1 - \frac{(y - \hat{y})^2}{(y - \bar{y})^2},
\end{align}

where $y$ is a vector with the true values of the energy, $\hat{y}$ is the predicted values of $y$ from the models and $\bar{y}$ is the mean of $\hat{y}$.

```python
def r_squared(y, y_hat):
    return 1 - np.sum((y - y_hat) ** 2) / np.sum((y - np.mean(y_hat)) ** 2)
```

This is the same metric used by Scikit-learn for their regression models when scoring.

```python
y_hat = clf.predict(X_test)
r_test = r_squared(y_test, y_hat)
sk_r_test = clf.score(X_test, y_test)

assert abs(r_test - sk_r_test) < 1e-2
```

## Performance as  function of the regularization parameter

We see how the different models perform for a different set of values for $\lambda$.

```python
lambdas = np.logspace(-4, 5, 10)

train_errors = {
    "ols_own": np.zeros(lambdas.size),
    "ols_sk": np.zeros(lambdas.size),
    "ridge_own": np.zeros(lambdas.size),
    "ridge_sk": np.zeros(lambdas.size),
    "lasso_sk": np.zeros(lambdas.size)
}

test_errors = {
    "ols_own": np.zeros(lambdas.size),
    "ols_sk": np.zeros(lambdas.size),
    "ridge_own": np.zeros(lambdas.size),
    "ridge_sk": np.zeros(lambdas.size),
    "lasso_sk": np.zeros(lambdas.size)
}

plot_counter = 1

fig = plt.figure(figsize=(32, 54))

for i, _lambda in enumerate(tqdm.tqdm(lambdas)):
    omega = get_ols_weights(X_train_own, y_train)
    y_hat_train = X_train_own @ omega
    y_hat_test = X_test_own @ omega

    train_errors["ols_own"][i] = r_squared(y_train, y_hat_train)
    test_errors["ols_own"][i] = r_squared(y_test, y_hat_test)

    plt.subplot(10, 5, plot_counter)
    plt.imshow(omega[1:].reshape(L, L), **cmap_args)
    plt.title("Home made OLS")
    plot_counter += 1

    omega = get_ridge_weights(X_train_own, y_train, _lambda)
    y_hat_train = X_train_own @ omega
    y_hat_test = X_test_own @ omega

    train_errors["ridge_own"][i] = r_squared(y_train, y_hat_train)
    test_errors["ridge_own"][i] = r_squared(y_test, y_hat_test)

    plt.subplot(10, 5, plot_counter)
    plt.imshow(omega[1:].reshape(L, L), **cmap_args)
    plt.title(r"Home made ridge, $\lambda = %.4f$" % _lambda)
    plot_counter += 1

    for key, method in zip(
        ["ols_sk", "ridge_sk", "lasso_sk"],
        [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)]
    ):
        method = method.fit(X_train, y_train)

        train_errors[key][i] = method.score(X_train, y_train)
        test_errors[key][i] = method.score(X_test, y_test)

        omega = method.coef_.reshape(L, L)

        plt.subplot(10, 5, plot_counter)
        plt.imshow(omega, **cmap_args)
        plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda))
        plot_counter += 1

plt.show()
```

We can see that LASSO quite fast reaches a good solution for low values of $\lambda$, but will "wither" when we increase $\lambda$ too much. Ridge is more stable over a larger range of values for $\lambda$, but eventually also fades away.

## Finding the optimal value of $\lambda$

To determine which value of $\lambda$ is best we plot the accuracy of the models when predicting the training and the testing set. We expect the accuracy of the training set to be quite good, but if the accuracy of the testing set is much lower this tells us that we might be subject to an overfit model. The ideal scenario is an accuracy on the testing set that is close to the accuracy of the training set.

```python
fig = plt.figure(figsize=(20, 14))

colors = {
    "ols_own": "b",
    "ridge_own": "g",
    "ols_sk": "r",
    "ridge_sk": "y",
    "lasso_sk": "c"
}

for key in train_errors:
    plt.semilogx(
        lambdas,
        train_errors[key],
        colors[key],
        label="Train {0}".format(key),
        linewidth=4.0
    )

for key in test_errors:
    plt.semilogx(
        lambdas,
        test_errors[key],
        colors[key] + "--",
        label="Test {0}".format(key),
        linewidth=4.0
    )
#plt.semilogx(lambdas, train_errors["ols_own"], label="Train (OLS own)")
#plt.semilogx(lambdas, test_errors["ols_own"], label="Test (OLS own)")

plt.legend(loc="best", fontsize=18)
plt.xlabel(r"$\lambda$", fontsize=18)
plt.ylabel(r"$R^2$", fontsize=18)
plt.tick_params(labelsize=18)
plt.show()
```

From the above figure we can see that LASSO with $\lambda = 10^{-2}$ achieve a very good accuracy on the test set. This by far surpases the other models for all values of $\lambda$.


# Elliptic PDEs

The classic example of an elliptic PDE is **Laplace's equation** (yep, the same Laplace that gave us the Laplace transform), which in two dimensions for a variable $u(x,y)$ is
\begin{equation}
\frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2} = \nabla^2 u = 0 \;,
\end{equation}
where $\nabla$ is del, or nabla, and represents the gradient operator: $\nabla = \frac{\partial}{\partial x} + \frac{\partial}{\partial y}$.

Laplace's equation shows up in a number of physical problems, including heat transfer, fluid dynamics, and electrostatics. For example, the heat equation for conduction in two dimensions is
\begin{equation}
\frac{\partial u}{\partial t} = \alpha \left( \frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2} \right) \;,
\end{equation}
where $u(x,y,t)$ is temperature and $\alpha$ is thermal diffusivity. Steady-state heat transfer (meaning after any initial transient period) is then described by Laplace's equation.

A related elliptic PDE is **Poisson's equation**:
\begin{equation}
\nabla^2 u = f(x,y) \;,
\end{equation}
which also appears in multiple physical problems—most notably, when solving for pressure in the Navier–Stokes equations.

To numerically solve these equations, and any elliptic PDE, we can use finite differences, where we replace the continuous $x,y$ domain with a discrete grid of points. This is similar to what we did with boundary-value problems in one dimension—but now we have two dimensions.

To approximate the second derivatives in Laplace's equation, we can use central differences in both the $x$ and $y$ directions, applied around the $u_{i,j}$ point:
\begin{align}
\frac{\partial^2 u}{\partial x^2} &\approx \frac{u_{i-1,j} - 2u_{i,j} + u_{i+1,j}}{\Delta x^2} \\
\frac{\partial^2 u}{\partial y^2} &\approx \frac{u_{i,j-1} - 2u_{i,j} + u_{i,j+1}}{\Delta y^2}
\end{align}
where $i$ is the index used in the $x$ direction, $j$ is the index in the $y$ direction, and $\Delta x$ and $\Delta y$ are the step sizes in the $x$ and $y$ directions.
In other words, $x_i = (i-1) \Delta x$ and $y_j = (j-1) \Delta y$.

The following figure shows the points necessary to approximate the partial derivatives in the PDE at a location $(x_i, y_j)$, for a general 2D region. This is known as a **five-point stencil**:

:::{figure-md} fig-stencil

Five-point finite difference stencil
:::

Applying these finite differences gives us an approximation for Laplace's equation:
\begin{equation}
\frac{u_{i-1,j} - 2u_{i,j} + u_{i+1,j}}{\Delta x^2} + \frac{u_{i,j-1} - 2u_{i,j} + u_{i,j+1}}{\Delta y^2} = 0 \;.
\end{equation}
If we use a uniform grid where $\Delta x = \Delta y = h$, then we can simplify to
\begin{equation}
u_{i+1,j} + u_{i,j+1} + u_{i-1,j} + u_{i,j-1} - 4 u_{i,j} = 0 \;.
\end{equation}

## Example: heat transfer in a square plate

As an example, let's consider the problem of steady-state heat transfer in a square solid object. If $u(x,y)$ is temperature, then this is described by Laplace's equation:
\begin{equation}
\frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2} = \nabla^2 u = 0 \;,
\end{equation}
and we can solve this using finite differences. Using a uniform grid where $\Delta x = \Delta y = h$, Laplace's equation gives us a recursion formula that relates the values at neighboring points:
\begin{equation}
u_{i+1,j} + u_{i,j+1} + u_{i-1,j} + u_{i,j-1} - 4 u_{i,j} = 0 \;.
\end{equation}

Consider a case where the square has sides of length $L$, and the boundary conditions are that the temperature is fixed at 100 on the left, right, and bottom sides, and fixed at 0 on the top.
For now, we'll use two segments to discretize the domain in each directions, giving us nine total points in the grid.
The following figures show the example problem, and the grid of points we'll use.

:::{figure-md} fig-heat-transfer-square

Heat transfer in a square object
:::

:::{figure-md} fig-grid-three

Simple 3x3 grid of points
:::

Using the above recursion formula, we can write an equation for each of the nine unknown points (in the interior, not the boundary points):
\begin{align}
u_{1,1} &= 100 \\
u_{2,1} &= 100 \\
u_{3,1} &= 100 \\
u_{1,2} &= 100 \\
\text{for } u_{2,2}: \quad u_{3,2} + u_{2,3} + u_{1,2} + u_{2,1} - 4u_{2,2} &= 0 \\
u_{3,2} &= 100 \\
u_{1,3} &= 100 \\
u_{2,3} &= 0 \\
u_{3,3} &= 100
\end{align}
where $u_{i,j}$ are the unknowns. Note that in this we used the side boundary condition values for the corner points $u_{1,3}$ and $u_{3,3}$, rather than the top value. (In reality this would represent a discontinuity in temperature, so these aren't very realistic boundary conditions.)

This is a system of linear equations, that we can represent as a matrix-vector product:
\begin{align}
\begin{bmatrix}
1 & 0 & 0 & 0 &  0 & 0 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 &  0 & 0 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 &  0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 1 &  0 & 0 & 0 & 0 & 0 \\
0 & 1 & 0 & 1 & -4 & 1 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 &  0 & 1 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 &  0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 0 &  0 & 0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 &  0 & 0 & 0 & 0 & 1 \\\end{bmatrix}
\begin{bmatrix} u_{1,1} \\ u_{2,1} \\ u_{3,1} \\ u_{1,2} \\ u_{2,2} \\ u_{3,2} \\ u_{1,3} \\ u_{2,3} \\ u_{3,3} \end{bmatrix} &=
\begin{bmatrix} 100 \\ 100 \\ 100 \\
100 \\ 0 \\ 100 \\
100 \\ 0 \\ 100 \end{bmatrix} \\
\text{or} \quad A \mathbf{u} &= \mathbf{b}
\end{align}
where $A$ is a $9\times 9$ coefficient matrix, $\mathbf{u}$ is a nine-element vector of unknown variables, and $\mathbf{b}$ is a nine-element right-hand side vector.
For $\mathbf{u}$, we had to take variables that physically represent points in a two-dimensional space and combine them in some order to form a one-dimensional column vector. Here, we used a **row-major** mapping, where we started with the point in the first row and first column, then added the remaining points in that row, before moving to the next row and repeating. We'll discuss this a bit more later.

If we set this up in Matlab, we can solve with `u = A \ b`:

```matlab
clear all; clc

A = [
1 0 0 0  0 0 0 0 0;
0 1 0 0  0 0 0 0 0;
0 0 1 0  0 0 0 0 0;
0 0 0 1  0 0 0 0 0;
0 1 0 1 -4 1 0 1 0;
0 0 0 0  0 1 0 0 0;
0 0 0 0  0 0 1 0 0;
0 0 0 0  0 0 0 1 0;
0 0 0 0  0 0 0 0 1];
b = [100; 100; 100; 100; 0; 100; 100; 0; 100];

% solve system of linear equations
u = A \ b;

disp(u)
```

       100
       100
       100
       100
        75
       100
       100
         0
       100

This gives us the values for temperature at each of the nine points. In this example, we really only have one unknown temperature: $u_{2,2}$, located in the middle. Does the value given make sense? We can check by rearranging the recursion formula for Laplace's equation:
\begin{equation}
u_{i,j} = \frac{u_{i+1,j} + u_{i,j+1} + u_{i-1,j} + u_{i,j-1}}{4} \;,
\end{equation}
which shows that in such problems the value of the middle point should be the average of the four surrounding points. This matches the value of 75 found above.

We can use a contour plot to visualize the results, though we'll need to convert the one-dimensional solution array into a two-dimensional matrix to plot. The Matlab `reshape()` function can help us here: it reshapes an array into a matrix, by specifying the target number of desired columns and rows:

```matlab
% Example of using the reshape function, with a simple array going from 1 to 10

% We want to convert it into a matrix with 5 columns and 2 rows.
% The expected output is:
% [1 2 3 4 5;
%  6 7 8 9 10]

b = (1 : 10)';
A = reshape(b, [5, 2]);
disp('b array:')
disp(b)
disp('A matrix:')
disp(A)
```

    b array:
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10

    A matrix:
         1     6
         2     7
         3     8
         4     9
         5    10

This behavior may be a bit unexpected, because `reshape()` uses a column-major mapping. We can fix this by taking the transpose of the resulting matrix:

```matlab
disp('transpose of output matrix:')
disp(A')
```

    transpose of output matrix:
         1     2     3     4     5
         6     7     8     9    10

```matlab
% We can use the reshape function to convert the calculated temperatures
% into a 3x3 matrix:

n = 3; m = 3;
u_square = reshape(u, [n, m]);

contourf(u_square')
colorbar
```

Overall that looks correct: the boundary conditions are right, and we see that the center is the average of the boundaries.

But, clearly only using nine points (with eight of those being boundary conditions) doesn't give us a very good solution. To make this more accurate, we'll need to use more points, which also means we need to automate the construction of the system of equations.

## Row-major mapping

For a two-dimensional elliptic PDE like Laplace's equation, we can generate a general recursion formula, but we need a way to take a grid of points where location is defined by row and column index and map these into a one-dimensional column vector, which has its own index.

The following figure shows a general 2D grid of points, with $n$ number of columns in the $x$ direction (using index $i$) and $m$ number of rows in the $y$ direction (using index $j$):

:::{figure-md} fig-twodim-grid

2D grid of points with *n* columns and *m* columns.
:::

We want to convert the rows and columns of $u_{i,j}$ points defined by column and row index into a single column array using a different index, $k$ (this choice is arbitrary):
\begin{equation}
\begin{bmatrix} u_{1,1} \\ u_{2,1} \\ u_{3,1} \\ \vdots \\ u_{n,1} \\
u_{1,2} \\ u_{2,2} \\ u_{3,2} \\ \vdots \\ u_{n, 2} \\ u_{1,3} \\ \vdots \\
u_{1,m} \\ u_{2,m} \\ \vdots \\ u_{n,m}
\end{bmatrix}
\end{equation}
where $k$ refers to the index used in that array.

To do this mapping, we can use this formula:
\begin{equation}
k_{i,j} = (j-1)n + i
\end{equation}
where $k_{i,j}$ refers to the 1D index $k$ mapped from the 2D indices $i$ and $j$.

:::{figure-md} fig-grid-three

Simple 3x3 grid of points
:::

For example, in this $3\times 3$ grid, where $n=3$ and $m=3$, consider the point where $i=2$ and $j=2$ (the point right in the center). Using our formula,
\begin{equation}
k_{2,2} = (2-1)3 + 2 = 5
\end{equation}
which matches what we can visually confirm.

Using that mapping, we can also identify the 1D indices associated with the points surrounding location $(i,j)$:
\begin{align}
k_{i-1,j} &= (j-1)n + i - 1 \\
k_{i+1,j} &= (j-1)n + i + 1 \\
k_{i,j-1} &= (j-2)n + i \\
k_{i,j+1} &= j n + i
\end{align}
which we can use to determine the appropriate locations to place values in the coefficient and right-hand side matrices.

## Example: heat transfer in a square plate (redux)

Let's return to the example of steady-state heat transfer in a square plate—but this time we'll set the solution up more generally so we can vary the step size $h = \Delta x = \Delta y$.

```matlab
clear; clc; close all

h = 0.1;
x = [0 : h : 1]; n = length(x);
y = [0 : h : 1]; m = length(y);

% The coefficient matrix A is now m*n by m*n, since that is the total number of points.
% The right-hand side vector b is m*n by 1.
A = zeros(m*n, m*n);
b = zeros(m*n, 1);

u_left = 100;
u_right = 100;
u_bottom = 100;
u_top = 0;

for j = 1 : m
    for i = 1 : n
        % for convenience we calculate all the indices once
        kij = (j-1)*n + i;
        kim1j = (j-1)*n + i - 1;
        kip1j = (j-1)*n + i + 1;
        kijm1 = (j-2)*n + i;
        kijp1 = j*n + i;

        if i == 1
            % this is the left boundary
            A(kij, kij) = 1;
            b(kij) = u_left;
        elseif i == n
            % right boundary
            A(kij, kij) = 1;
            b(kij) = u_right;
        elseif j == 1
            % bottom boundary
            A(kij, kij) = 1;
            b(kij) = u_bottom;
        elseif j == m
            % top boundary
            A(kij, kij) = 1;
            b(kij) = u_top;
        else
            % these are the coefficients for the interior points,
            % based on the recursion formula
            A(kij, kim1j) = 1;
            A(kij, kip1j) = 1;
            A(kij, kijm1) = 1;
            A(kij, kijp1) = 1;
            A(kij, kij) = -4;
        end
    end
end
u = A \ b;

u_square = reshape(u, [n, m]);
contourf(x, y, u_square')
c = colorbar;
c.Label.String = 'Temperature';
```

## Neumann (derivative) boundary conditions

So far, we have only discussed cases where we have Dirichlet boundary conditions; in other words, when we have all fixed values at the boundary. Frequently we also encounter Neumann-style boundary conditions, where we have the *derivative* specified at the boundary.

We can handle this in the same way we do for one-dimensional boundary value problems: either with a forward or backward difference (both of which are first-order accurate), or with a central difference using an imaginary point/ghost node (which is second-order accurate). Let's focus on using the central difference, since it is more accurate.

:::{figure-md} fig-ghost-node

Ghost/imaginary node beyond an upper boundary
:::

For example, let's say that at the upper boundary, the derivative of temperature is zero:
\begin{equation}
\left. \frac{\partial u}{\partial y} \right|_{\text{boundary}} = 0
\end{equation}

Let's consider this boundary condition applied at the point shown, $u_{2,3}$.
We can approximate this derivative using a central difference:
\begin{align}
\frac{u_{2,3}}{\partial y} \approx \frac{u_{2,4} - u_{2,2}}{\Delta x} &= 0 \\
u_{2,4} &= u_{2,2}
\end{align}
This tells us the value of the point above the boundary, $u_{2,4}$; however, this point is a "ghost" or imaginary point located outside the boundary, so we don't really care about its value. Instead, we can use this relationship to give us a usable equation for the boundary point, by incorporating it into the normal recursion formula for Laplace's equation:
\begin{align}
u_{1,3} + u_{3,3} + u_{2,4} + u_{2,2} - 4u_{2,3} &= 0 \\
u_{1,3} + u_{3,3} + u_{2,2} + u_{2,2} - 4u_{2,3} &= 0 \\
\rightarrow u_{1,3} + u_{3,3} + 2 u_{2,2} - 4u_{2,3} &= 0
\end{align}

The recursion formula for points along the upper boundary would then become
\begin{equation}
u_{i+1,j} + u_{i-1,j} + 2 u_{i,j-1} - 4 u_{i,j} = 0 \;.
\end{equation}

Now let's try solving the above example, but with $\frac{\partial u}{\partial y} = 0$ at the top boundary and $u = 0$ at the bottom boundary:

```matlab
clear; clc; close all

h = 0.1;
x = [0 : h : 1]; n = length(x);
y = [0 : h : 1]; m = length(y);

% The coefficient matrix A is now m*n by m*n, since that is the total number of points.
% The right-hand side vector b is m*n by 1.
A = zeros(m*n, m*n);
b = zeros(m*n, 1);

u_left = 100;
u_right = 100;
u_bottom = 0;

for j = 1 : m
    for i = 1 : n
        % for convenience we calculate all the indices once
        kij = (j-1)*n + i;
        kim1j = (j-1)*n + i - 1;
        kip1j = (j-1)*n + i + 1;
        kijm1 = (j-2)*n + i;
        kijp1 = j*n + i;

        if i == 1
            % this is the left boundary
            A(kij, kij) = 1;
            b(kij) = u_left;
        elseif i == n
            % right boundary
            A(kij, kij) = 1;
            b(kij) = u_right;
        elseif j == 1
            % bottom boundary
            A(kij, kij) = 1;
            b(kij) = u_bottom;
        elseif j == m
            % top boundary, using the ghost node + recursion formula
            A(kij, kim1j) = 1;
            A(kij, kip1j) = 1;
            A(kij, kijm1) = 2;
            A(kij, kij) = -4;
        else
            % these are the coefficients for the interior points,
            % based on the recursion formula
            A(kij, kim1j) = 1;
            A(kij, kip1j) = 1;
            A(kij, kijm1) = 1;
            A(kij, kijp1) = 1;
            A(kij, kij) = -4;
        end
    end
end
u = A \ b;

u_square = reshape(u, [n, m]);
% the "20" indicates the number of levels for the contour plot
contourf(x, y, u_square', 20);
c = colorbar;
c.Label.String = 'Temperature';
```

## Iterative solutions for (very) large problems

So far, we've been able to solve our systems of linear equations in Matlab by using `y = A \ b`, which directly finds the solution to the equation $A \mathbf{y} = \mathbf{b}$.

However, this approach will become very slow as the grid resolution ($h = \Delta x = \Delta y$) becomes smaller, and eventually unfeasable due to the associated computational requirements. First, let's create a function that takes as input the segment size $h$, then returns the time ittakes to solve the problem for different sizes.

```matlab
%%file heat_equation.m
function [time, num] = heat_equation(h)

x = [0 : h : 1]; n = length(x);
y = [0 : h : 1]; m = length(y);

% The coefficient matrix A is now m*n by m*n, since that is the total number of points.
% The right-hand side vector b is m*n by 1.
A = zeros(m*n, m*n);
b = zeros(m*n, 1);

num = m*n; % number of points

tic;

u_left = 100;
u_right = 100;
u_bottom = 100;
u_top = 0;

for j = 1 : m
    for i = 1 : n
        % for convenience we calculate all the indices once
        kij = (j-1)*n + i;
        kim1j = (j-1)*n + i - 1;
        kip1j = (j-1)*n + i + 1;
        kijm1 = (j-2)*n + i;
        kijp1 = j*n + i;

        if i == 1
            % this is the left boundary
            A(kij, kij) = 1;
            b(kij) = u_left;
        elseif i == n
            % right boundary
            A(kij, kij) = 1;
            b(kij) = u_right;
        elseif j == 1
            % bottom boundary
            A(kij, kij) = 1;
            b(kij) = u_bottom;
        elseif j == m
            % top boundary
            A(kij, kij) = 1;
            b(kij) = u_top;
        else
            % these are the coefficients for the interior points,
            % based on the recursion formula
            A(kij, kim1j) = 1;
            A(kij, kip1j) = 1;
            A(kij, kijm1) = 1;
            A(kij, kijp1) = 1;
            A(kij, kij) = -4;
        end
    end
end
u = A \ b;

u_square = reshape(u, [n, m]);
% the "20" indicates the number of levels for the contour plot
%contourf(x, y, u_square', 20);
%c = colorbar;
%c.Label.String = 'Temperature';

time = toc;
```

    Created file '/Users/kyle/projects/ME373-book/content/pdes/heat_equation.m'.

Now, we can see how long it takes to solve as we increase the resolution, and get an idea about the relationship between time-to-solution and number of unknowns.

```matlab
clear all; clc;

step_sizes = [0.1, 0.05, 0.025, 0.02, 0.0125, 0.01];

n = length(step_sizes);
nums = zeros(n,1); times = zeros(n,1);

for i = 1 : n
    [times(i), nums(i)] = heat_equation(step_sizes(i));
end

loglog(nums, times, '-o')
xlabel('Number of unknowns');
ylabel('Time for direct solution (sec)')
hold on

x = nums(3:end);
n2 = x.^2 * (times(3) / x(1)^2);
n3 = x.^3 * (times(3) / x(1)^3);
plot(x, n2, '-x');
plot(x, n3, '-s');
legend('Actual cost', 'Quadratic cost', 'Cubic cost', 'Location', 'northwest')
```

Interestingly, we see that the slope in this log-log plot that after about 400 unknowns (so a coefficient matrix of  about 160,000), the cost begins to increase exponentially, somewhere between quadratic ($\mathcal{O}(n^2)$) and cubic ($\mathcal{O}(n^3)$).

If we try to reduce the step size further, for example to 0.005, we'll see that we cannot get a solution in a reasonable amount of time. But, clearly we want to get solutions for large numbers of unknowns, so what can we do?

We can solve larger systems of linear equations using *iterative* methods. There are a number of these, and we'll focus on two:

- Jacobi method
- Gauss-Seidel method

### Jacobi method

The Jacobi method essentially works by starting with an initial guess to the solution, then using the recursion formula to solve for values at each point, then repeating this until the values converge (i.e., stop changing).

An algorithm we can use to solve Laplace's equation:

1. Set some initial guess for all unknowns: $u_{i,j}^{\text{old}}$
2. Set the boundary values
3. For each point in the interior, use the recursion formula to solve for new values based on old values at the surrounding points: $u_{i,j} = \left( u_{i+1,j}^{\text{old}} + u_{i-1,j}^{\text{old}} + u_{i,j+1}^{\text{old}} + u_{i,j-1}^{\text{old}} \right)/4$.
4. Check for convergence: is $\epsilon$ less than some tolerance, such as $10^{-6}$? Where $\epsilon = \max \left| u_{i,j} - u_{i,j}^{\text{old}} \right|$. If no, then return to step 2 and repeat.

More formally, if we have a system $A \mathbf{x} = \mathbf{b}$, where
\begin{equation}
A = \begin{bmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix}
\quad \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}
\quad \mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix}
\end{equation}
then we can solve iterative for $\mathbf{x}$ using
\begin{equation}
x_i^{(k+1)} = \frac{1}{a_{ii}} \left( b_i - \sum_{j \neq i} a_{ij} x_j^{(k)} \right) , \quad i = 1,2,\ldots, n
\end{equation}
where $x_i^{(k)}$ is a value of the solution at iteration $k$ and $x_i^{(k+1)}$ is at the next iteration.

```matlab
%%file heat_equation_jacobi.m
function [time, num_point, num_iter] = heat_equation_jacobi(h)

x = [0 : h : 1]; n = length(x);
y = [0 : h : 1]; m = length(y);

% The coefficient matrix A is now m*n by m*n, since that is the total number of points.
% The right-hand side vector b is m*n by 1.
A = zeros(m*n, m*n);
b = zeros(m*n, 1);
num_point = m*n;

tic;

u_left = 100;
u_right = 100;
u_bottom = 100;
u_top = 0;

% initial guess
u = 100*ones(m*n, 1);

% dummy value for residual variable
epsilon = 1.0;

num_iter = 0;
while epsilon > 1e-6
    u_old = u;

    epsilon = 0;
    for j = 1 : m
        for i = 1 : n
            kij = (j-1)*n + i;
            kim1j = (j-1)*n + i - 1;
            kip1j = (j-1)*n + i + 1;
            kijm1 = (j-2)*n + i;
            kijp1 = j*n + i;

            if i == 1
                % this is the left boundary
                u(kij) = u_left;
            elseif i == n
                % right boundary
                u(kij) = u_right;
            elseif j == 1
                % bottom boundary
                u(kij) = u_bottom;
            elseif j == m
                % top boundary
                u(kij) = u_top;
            else
                % interior points
                u(kij) = (u_old(kip1j) + u_old(kim1j) + u_old(kijm1) + u_old(kijp1))/4.0;
            end
        end
    end

    epsilon = max(abs(u - u_old));
    num_iter = num_iter + 1;
end

u_square = reshape(u, [n, m]);
% the "20" indicates the number of levels for the contour plot
contourf(x, y, u_square', 20);
c = colorbar;
c.Label.String = 'Temperature';

time = toc;
fprintf('Number of iterations: %d\n', num_iter)
```

    Created file '/Users/kyle/projects/ME373-book/content/pdes/heat_equation_jacobi.m'.

```matlab
step_sizes = [0.1, 0.05, 0.025, 0.02, 0.0125, 0.01, 0.005];
n = length(step_sizes);

nums_jac = zeros(n,1); times_jac = zeros(n,1); num_iter_jac = zeros(n,1);

for i = 1 : n
    [times_jac(i), nums_jac(i), num_iter_jac(i)] = heat_equation_jacobi(step_sizes(i));
end
```

### Gauss-Seidel method

The Gauss-Seidel method is very similar to the Jacobi method, but with one important difference: rather than using all old values to calculate the new values, incorporate updated values as they are available. Because the method incorporates newer information more quickly, it tends to converge faster (meaning, with fewer iterations) than the Jacobi method.

Formally, if we have a system $A \mathbf{x} = \mathbf{b}$, where
\begin{equation}
A = \begin{bmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix}
\quad \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}
\quad \mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix}
\end{equation}
then we can solve iterative for $\mathbf{x}$ using
\begin{equation}
x_i^{(k+1)} = \frac{1}{a_{ii}} \left( b_i - \sum_{j=1}^{i-1} a_{ij} x_j^{(k+1)} - \sum_{j =i+1}^n a_{ij} x_j^{(k)} \right) , \quad i = 1,2,\ldots, n
\end{equation}
where $x_i^{(k)}$ is a value of the solution at iteration $k$ and $x_i^{(k+1)}$ is at the next iteration.

```matlab
%%file heat_equation_gaussseidel.m
function [time, num_point, num_iter] = heat_equation_gaussseidel(h)

x = [0 : h : 1]; n = length(x);
y = [0 : h : 1]; m = length(y);

% The coefficient matrix A is now m*n by m*n, since that is the total number of points.
% The right-hand side vector b is m*n by 1.
A = zeros(m*n, m*n);
b = zeros(m*n, 1);
num_point = m*n;

tic;

u_left = 100;
u_right = 100;
u_bottom = 100;
u_top = 0;

% initial guess
u = 100*ones(m*n, 1);

% dummy value for residual variable
epsilon = 1.0;

num_iter = 0;
while epsilon > 1e-6
    u_old = u;

    epsilon = 0;
    for j = 1 : m
        for i = 1 : n
            kij = (j-1)*n + i;
            kim1j = (j-1)*n + i - 1;
            kip1j = (j-1)*n + i + 1;
            kijm1 = (j-2)*n + i;
            kijp1 = j*n + i;

            if i == 1
                % this is the left boundary
                u(kij) = u_left;
            elseif i == n
                % right boundary
                u(kij) = u_right;
            elseif j == 1
                % bottom boundary
                u(kij) = u_bottom;
            elseif j == m
                % top boundary
                u(kij) = u_top;
            else
                % interior points
                u(kij) = (u(kip1j) + u(kim1j) + u(kijm1) + u(kijp1))/4.0;
            end
        end
    end

    epsilon = max(abs(u - u_old));
    num_iter = num_iter + 1;
end

u_square = reshape(u, [n, m]);
%% the "20" indicates the number of levels for the contour plot
contourf(x, y, u_square', 20);
c = colorbar;
c.Label.String = 'Temperature';

time = toc;
fprintf('Number of iterations: %d\n', num_iter)
```

    Created file '/Users/kyle/projects/ME373-book/content/pdes/heat_equation_gaussseidel.m'.

```matlab
step_sizes = [0.1, 0.05, 0.025, 0.02, 0.0125, 0.01, 0.005];
n = length(step_sizes);

nums_gs = zeros(n,1); times_gs = zeros(n,1); num_iter_gs = zeros(n,1);

for i = 1 : n
    [times_gs(i), nums_gs(i), num_iter_gs(i)] = heat_equation_gaussseidel(step_sizes(i));
end

loglog(nums, times, '-o'); hold on
loglog(nums_jac, times_jac, '-^')
loglog(nums_gs, times_gs, '-x')
xlabel('Number of unknowns');
ylabel('Time for solution (sec)')
legend('Direct solution', 'Jacobi solution', 'Gauss-Seidel solution', 'Location', 'northwest')
```

```matlab
loglog(nums_jac, num_iter_jac, '-^')
hold on;
loglog(nums_gs, num_iter_gs, '-x')
xlabel('Number of unknowns')
ylabel('Number of iterations required')
legend('Jacobi method', 'Gauss-Seidel method', 'Location', 'northwest')
```

These results show us a few things:

- For very small problems, the direct solution method is faster.
- For the heat equation, once we get to around 1000 unknowns, the methods perform similarly. Beyond this, the direct solution method becomes unreasonably slow, and even fails to solve in a reasonable time for a step size of 0.005.
- The Gauss-Seidel method converges with around half the number of iterations than the Jacobi method.

For larger, more-realistic problems, iterative solution methods like Jacobi and Gauss-Seidel are essential.


# SciPy
## Solve Linear Equation

```python
#import
from scipy import linalg as sla
import numpy as np
import sympy as sy
sy.init_printing()

from matplotlib import pyplot as plt
from pprint import pprint
```

```python
A=sy.Matrix([[3,2],[-3,5]])
b=sy.Matrix([8,-1])
```

```python
A, b
```

## try %timeit

```python
%timeit x=A.inv() * b
```

    998 µs ± 60.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
%timeit x=A.solve(b)
```

    1.23 ms ± 196 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
%timeit x=A.LUsolve(b)
```

    1.06 ms ± 522 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
A=np.array(A).astype(float)
b=np.array(b).astype(float)
print(A)
print(b)
```

    [[ 3.  2.]
     [-3.  5.]]
    [[ 8.]
     [-1.]]

```python
%timeit x=sla.solve(A,b)
```

    199 µs ± 53.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
# Handle Exception
```

```python
A=sy.Matrix([[3,2],[6,4 ]])
b=sy.Matrix([8,-1])
```

```python
A,b
```

```python
x=A.LUsolve(b)
x
```

```python
A=np.array(A).astype(float)
b=np.array(b).astype(float)
print(A)
print(b)
```

    [[ 3.  2.]
     [ 6.  4.]]
    [[ 8.]
     [-1.]]

```python
x=sla.solve(A,b)
```


# Matemática: Funções e Seus Usos

## Equação de segundo Grau com Python

```
from math import sqrt
from sympy import *
import math
init_printing()

a = 1
b = 8
c = 6
```

```
# Resolvendo equação de segundo grau
def raizes(a, b, c):
    x = Symbol('x')
    # ax² + bx + c
    return solve( a*x**2 + b*x + c , x)

# Plotando equação de segundo grau
def grafico_2grau(x1,x2):
  eixo_x = []
  eixo_y = []
  zero = []

  variacao = abs(x1 - x2)

  if variacao < 3:
      variacao = 3

  for x in np.arange(x1 - variacao, x2 + variacao, variacao / 100):
      y = a * (x ** 2 ) + b * (x) + c
      eixo_x.append(x)
      eixo_y.append(y)
      zero.append(0.0)

  # Desenha linha
  plt.plot(eixo_x,eixo_y,color="blue")

  # Desenha pontos
  plt.plot((x1,x2),(0,0), marker='o',color='red')
  plt.plot(c, marker='o',color='red')

  # Desenha eixos
  plt.plot(eixo_x,zero,color="black")
  plt.plot(zero,eixo_y,color='black')

  plt.show()

r = raizes(a,b,c)
```

```
import matplotlib.pyplot as plt
import numpy as np

try:
  x1 = float(r[0])
  x2 = float(r[1])
  grafico_2grau(x1,x2)
  print('x1 = %0.3f' % x1, 'x2= %0.3f' % x2)
except:
  try:
    x1 = complex(r[0])
    x2 = complex(r[1])
    print('O resultado das raízes é um número complexo')
    print('x1 =', x1, '\nx2 =', x2)
  except:
    print('O valor de a é 0, resultado indefinido.')
```

## Utilização do sympy para criar variáveis simbólicas

```
# define x, y e z como variáveis simbólicas
var('x y z')
```

# Funções

## Exemplo 01: Custo de operação de uma máquina

*   A empresa deseja saber o custo de energia elátrica de uma máquina.
*   A potência da máquina é de 500 kW
*   O custo de cada kW*h é de R$ 0,10

Qual a função matemática que descreve o consumo da máquina em função da hora?

```
# Custo da máquina por hora
def f(x): return 500*0.1*x
x = 24 * 30
print('O custo é de:', f(x), 'por %2.0f hora(s)' % x)

# Custo da máquina por dia
def C(x): return (500*0.1*x)*24
x = 30
print('O custo é de:', C(x), 'por %2.0f dia(s)' % x)
```

    O custo é de: 36000.0 por 720 hora(s)
    O custo é de: 36000.0 por 30 dia(s)

### Plotando Gráficos

```
# Calculando horas em um mês
dias = 30
horas = dias*24

lista_preco = list()
for hora in range(0,horas):
  lista_preco.append(f(hora))

# Gráfico

fig, ax = plt.subplots()

# Título do gráfico
plt.title('Preço kW*h em um mês')

# Eixos
ax.set_xlabel('horas')
ax.set_ylabel('Preço em (R$)')

plt.plot(lista_preco)
```

```
# Podemos redefinir o preço em função dos dias

dias = 30

lista_preco2 = list()

for dia in range(0,dias):
  lista_preco2.append(C(dia))

# Gráfico

fig, ax = plt.subplots()

# Título do gráfico
plt.title('Preço kW*dia em um mês')

# Eixos
ax.set_xlabel('dias')
ax.set_ylabel('Preço em (R$)')

plt.plot(lista_preco2)
```

## Exemplo 02: Realizando Previsões com funções

*   A situação problema da empresa é: temos a função custo C(x):

 C(x) =  0.02x² + 80/x

*   Qual a dimensão x da caixa, a partir da qual o custo é maior ou igual a R$ 22,00?

### Definindo Funções para o exemplo

```
def C(x): return (0.02*(x**2)) + (80/x)

def c(x): return round(C(x) - 22,2)

def find_raizes(y,x1,x2,p=0.005):
  lista_raizes = list()
  lista_index = list()
  raizes_= list()

  for i in np.arange(x1,x2,step=0.001):
    if (C(i)<= y and C(i)>= y-p):
      # Preço
      lista_raizes.append(round(C(i),4))
      # m²
      lista_index.append(round(i,2))

  return lista_index, lista_raizes
```

### Plotando Gráficos

```
resultado = find_raizes(22,3,5)

import itertools

for i, j in itertools.product(resultado[0], resultado[1]):
  valor_x = float(i)
  valor_y = float(j)

print('O valor de x é:', valor_x)
print('O valor de y é:', valor_y)

resultado2 = find_raizes(22,30,35)

print()

for i, j in itertools.product(resultado2[0], resultado2[1]):
  valor_x2 = float(i)
  valor_y2 = float(j)

print('O valor de x2 é:', valor_x2)
print('O valor de y2 é:', valor_y2)

```

    O valor de x é: 3.68
    O valor de y é: 21.9985

    O valor de x2 é: 31.17
    O valor de y2 é: 21.9991

```
lista_ex02 = list()

x = 35

zero_ex02 = []
eixo_x_ex02 = []
raiz = []

for i in np.arange(0.5,x, step=0.01):
  lista_ex02.append(C(i))
  zero_ex02.append(22)
  eixo_x_ex02.append(i)

# Gráfico
fig, ax = plt.subplots()

# Título do gráfico
plt.title('Custo de produção da caixa')

# Eixos
ax.set_xlabel('cm')
ax.set_ylabel('Preço em (R$)')

plt.plot(eixo_x_ex02,lista_ex02)

plt.plot(eixo_x_ex02,zero_ex02,color='red')

plt.plot(valor_x,valor_y, marker='o',color='black')
plt.plot(valor_x2,valor_y2, marker='o',color='black')

print('Logo, para tamanhos menores que: %0.2fcm e maiores que %0.2fcm, o custo de R$22,00 é inviável.' % (valor_x,valor_x2))
```

## Variação Media de Função: Parte 1

A variação média de uma função f(x), dentro do intervalo [a,b] é definida por:

$
\dfrac{\Delta f(x)}{\Delta x} = \dfrac{f(b)-f(a)}{b-a}
$

```
var ('a b')

def f(a): return 2*a**2-1
f(a)

eixo_x_exvm = []
eixo_y_exvm = []
raiz = []

for i in np.arange(0,5,step=0.1):
  eixo_y_exvm .append(f(i))
  eixo_x_exvm.append(i)

# Gráfico
fig, ax = plt.subplots()

# Título do gráfico
plt.title('Gráfico da Função')

plt.plot(eixo_x_exvm,eixo_y_exvm)
```

```
def variacao(a,b): return (f(b)-f(a))/(b-a)

# Alguns exemplos de variação média da função
print(variacao(4,2))
print(variacao(5,2))
print(variacao(1,2))
print(variacao(3,5))
```

    12.0
    14.0
    6.0
    16.0

## Variação Media de Função: Parte 2

Uma descrição mais flexível para variação média de uma função é:

A variação média da função f(x), no intevalo delta(x) é dada por:

$
\dfrac{\Delta f(x)}{\Delta x} = \dfrac{(f(x) + \Delta x) - f(x)}{\Delta x}
$

```
var ('x delta_x p')
def C(p): return 0.02*p**2+80/p
print(C(p))
def h(x,delta_x): return (C(x + delta_x) - C(x) )/ delta_x

# A função h() pode ser usada para calcular a variação da média
print(h(10,0.5))
```

    0.02*p**2 + 80/p
    -0.3519047619047626

## Dominio e Imagem de uma Função

O domínio da função f(x) corresponde ao conjunto de todos os valores x onde f(x) pode ser calculada.

A imagem da função f(x) corresponde ao conjunto de todos os valores que a função assume.

```
var ('x')
def f(x): return sqrt(x-1)

eixo_y_ex_dominio = list()
eixo_x_ex_dominio = list()

for i in np.arange(3,5, step=0.01):
  eixo_y_ex_dominio.append(f(i))
  eixo_x_ex_dominio.append(i)

plt.plot(eixo_x_ex_dominio,eixo_y_ex_dominio)

```

## Inversa de uma Função

### Função Injetora
 Uma função é injetora, se e somente se, para quaisquer x1 e x2 pertencentes ao domínio, temos que:

 $
 x_1 \neq x_2 \rightarrow f(x_1) \neq f(x_2)
 $

**Exemplo :** Note que: $ f: \mathbb{R} \rightarrow \mathbb{R}  $ **tal que:** $ f(x) = 5 $ é uma função, mas não é injetora!

```
var ('x')
def f(x): return 5

eixo_x = list()
eixo_y = list()

for i in range(-5,6):
  eixo_x.append(i)
  eixo_y.append(f(i))

plt.plot(eixo_x, eixo_y)
```

**Exemplo02 :** Note que: $ f: \mathbb{R} \rightarrow \mathbb{R}  $ **tal que:** $ f(x) = 5 * x $ é uma função injetora!

```
var ('x')
def f(x): return 5*x

eixo_x = list()
eixo_y = list()

for i in range(-5,6):
  eixo_x.append(i)
  eixo_y.append(f(i))

plt.plot(eixo_x, eixo_y)
```

### Função Sobrejetora
Uma função $ f: A \rightarrow B $  é sobrejetora, se cada elemento do conjunto imagem de B é imagem de pelo menos um elemento de A.

Em outras palavras, a imagem B de f é a totalidade do conjunto B.

$ f: \mathbb{R} \rightarrow \mathbb{R} | f(x) = 3*x  $

```
var ('x')
def f(x): return 3*x

eixo_x = list()
eixo_y = list()

for i in range(-5,6):
  eixo_x.append(i)
  eixo_y.append(f(i))

plt.plot(eixo_x, eixo_y)
```

### Função Bijetora

Definição: Uma função que é injetora e subjetora.

$ f(x) $ bijetora é reversível $ f^{-1}(x)$

$ f^{-1}(f(x)) = x$

$ f^{-1}(x) = \sqrt{x} $

## Polinômios e raizes

Uma função polinomia de grau n, é toda função que pode ser escrita da seguinte forma:
$ f(x) = a_0 + a_1x^1 + a_2x^2 + ... + a_nx^n  $

**exemplo:** $ f(x) = -2x + 5x^3 + 3  $ é uma função de grau 3.

### Exemplo
Vamos mostrar alguns exemplos de como modelamos alguns problemas reais, usando funções. Dessa vez, vamos descrever a queda livre de um objeto, o modelo matemático parte da segunda lei de Newton:

A queda livre de um objeto é solução do seguinte problema:

1.   Solta-se um objeto, de uma altura $ h0 $, deixando-o cair em direção ao solo.
2.   Considera-se desprezível a resistência do ar, durante o movimento da queda.
3.   A única força que age sobre o objeto é o peso do mesmo ( que descreve a atração do planeta sobre esse corpo)
4.   Usa-se a segunda lei de Newton:

      $   \overrightarrow{P} = m\overrightarrow{g}=m\overrightarrow{a}_(t)$

A função que descreve o modelo do objeto em queda, é solução da lei de Newton e é dada pela função polinomial de segundo grau:

$h(t) = h_0 + v_{0}t - \dfrac {g}{2}t^2 $

```
var ('t h0 v0 g')
# A aceleração da gravidade na Terra ao nível do mar e à latitude de 45°,
# possui o valor aproximado de 9,80665 m/s²
def h(t,h0=h0,v0=v0,g=9.80665): return h0 + (v0 * t) - ((g/2)*t**2)
h(t,h0,v0,g)
```

### Problema

Podemos realizar um gráfico para o seguinte exemplo de queda livre:
Um objeto é solto de uma altura inicial $ h_0 = 200m, v_0 = 0ms^{-1}.$

Pede-se o gráfico da altura $ h(t) $ para $ t > 0 $ e quanto tempo esse objeto leva para alcançar o solo.

```
h(t,200,0)

eixo_x = list()
eixo_y = list()

zero = list()

for i in np.arange(-20,20,step=0.1):
  eixo_y.append(h(i,200,0))
  eixo_x.append(i)
  zero.append(0)

plt.plot(eixo_x,eixo_y)

# Desenha eixos
plt.plot(eixo_x,zero,color="black")
plt.plot(zero,eixo_y,color='black')

# Raíz do polinômio é quando f(x) = 0
round(h(6.382,200,0))
```

## Juros

Os juros são divididos entre simples ou compostos.

### Juros Simples

$ J(C,i,n) = Cin$

**Valor futuro de uma aplicação financeira:**

$ C + J(C,i,n) = C(1+in)  $

**Onde:**

*   C --> Capital inicial
*   i --> Taxa de Juros
*   n --> Período

Capital inicial também é conhecido como Valor presente:

$ VF = VP(1+in)  $

**Exemplo:** Considere um capital inicial $ C= 10000,00$ capitalizado  com uma taxa de juros $ i = 0,01 $ ao mês.

*   Qual é o valor montante VF depois de 10 meses?
*   Faça um gráfico de VF pelo tempo.

```
var ('i n vp')
def vf(i,n,vp): return vp*(1+i*n)

VF = vf(0.01,10,10000)
print('O montante final foi de: R$', VF)
```

    O montante final foi de: R$ 11000.0

```
eixo_x = list()
eixo_y = list()

for t in np.arange(0,10, step=0.1):
  eixo_x.append(t)
  eixo_y.append(vf(0.01,t,10000))

plt.plot(eixo_x,eixo_y)
```

### Juros Compostos

Fórmula dos juros compostos e uma comparação:

**Juros compostos:**

$ VF(VP,i,n,) = VP(1+i)^2  $

**Juros simples:**

$ VF = C + J(C,i,n) = C(1+in)  $

```
# Definindo função de juros compostos
def vf_c(i,n,vp): return vp*(1+i)**n
```

```
eixo_x = list()
eixo_y = list()

eixo_y2 = list()

for t in np.arange(0,30, step=0.1):
  eixo_x.append(t)
  eixo_y.append(vf(0.03,t,100))

  eixo_x2.append(t)
  eixo_y2.append(vf_c(0.03,t,100))

plt.plot(eixo_x,eixo_y)
plt.plot(eixo_x,eixo_y2)
```

## Função exponencial

**Definição:**
É uma função contínua da variável “x” que depende de um parâmetro “C”, que é chamado de base.

 Esse “C” é um número real/positivo e todo esse “C” está elevado a uma função contínua e bem-comportada da variável “x”. E “g(x)” pode ter sinal positivo ou negativo.

 $ f(x) = C ^{g(x)}$

**Usos mais comuns:**

$ C=2, C=10, C=e = 2,71828183... $

Estimativas numéricas para o $ e $:

$ e = \lim_{x \to oo} ( 1 + \dfrac{1}{x})^{x}$

**Usos mais comuns:**

$ f(x) = e^{x}$

$h(x) = e^{-x}$

**Exemplo:**  Em uma reportagem de março de 2020 do Jornal Nacional, foi mostrado o dado de que, a cada pessoa que contrai o virus coronavirus, ela passa para duas pessoas.

 Pensando em estimar o contágio na população, crie uma função que represente esse dado de contaminação.

```
var('k')
def N(k): return 2**k
N(10)
```

# Limites

Podemos usar o SymPy para calcular o limite de uma função quando  x  tende a um ponto.

## Exemplo 03: Calculando Limites

```
# define x e y como variáveis simbólicas
var('x y')

# Definindo função
f = Lambda(x, (x**2 -1))

# Limite da função f(x) quando x tende a 2
r1 = limit(f(x),x,2)

# Limite da função f(x) quando x tende a -1
r2 = limit(f(x),x,-1)

# Limite da função f(x) quando x tende a 5
r3 = limit(f(x),x,5)

print(r1,r2,r3)
```

    3 0 24

### Gráfico da Função

```
def f(x): return (x**2 -1)

lista_ex03 = list()

x = 5

eixo_x_ex03 = []
raiz = []

for i in np.arange(-1,x,step=0.1):
  lista_ex03.append(f(i))
  eixo_x_ex03.append(i)

# Gráfico
fig, ax = plt.subplots()

# Título do gráfico
plt.title('Gráfico da Função')

plt.plot(eixo_x_ex03,lista_ex03)

# Limites
plt.plot(2,r1, marker='o',color='black')
plt.plot(-1,r2, marker='o',color='black')
plt.plot(5,r3, marker='o',color='black')
```

## Calculando Limites Laterais

Como padrão, a função ```limit()```
calcula o limite lateral pela direita.

```
limit(1/y, y, 0)
```

para calcularmos o limite lateral pela esquerda, basta digitar:

```
limit(1/y, y, 0, '-')
```

### Exemplo 04

Devido aos limites laterais não coincidirem, o limite da função não existe.

```
lista_ex04 = list()
eixo_x_ex04 = list()

for x in np.arange(0,6, step=0.1):
  if (x<3):
    lista_ex04.append(1+x**2)
  else:
    lista_ex04.append(20)
  eixo_x_ex04.append(x)

plt.plot(eixo_x_ex04,lista_ex04)
```

```
# Limite de x quando x tende a 3
limite01 = limit(1+x**2, x, 3, '-')

# Limite de x quando x tende a 3
limite02 = limit(20, x, 3)

print('O limite pela direita é de:', limite01 )
print('enquanto o limite pela esquerda é de:' , limite02 )
```

    O limite pela direita é de: 35.8100000000000
    enquanto o limite pela esquerda é de: 20

**Conclusão:** Os limites laterais refletem o fato de que podemos nos aproximar de um dado valor, quer seja pela esquerda ou pela direita.

Em alguns casos, o valor resultante do limite pode ser igual pela equerda e pela direita, mas em outros pode não ser.

## Propriedades Limites: Parte 1

1 - O limite da soma de duas funções quando x se aproxima de x0, é igual à soma dos limites das funções.

$
\lim_{x \to x0} (f(x) + g(x))= \lim_{x \to x0} f(x) + \lim_{x \to x0} g(x)
$

2 - Propriedade da multiplicação por escalar do Limite:

$
\lim_{x \to x0} C( f(x))= C \lim_{x \to x0} f(x)
$

### Soma de **Funções**

```
def f(z): return z + 1
def g(z): return z

soma_funcoes = f(z) + g(z)
print('O limite da soma das funções é:',limit(soma_funcoes,z, 1))

soma_limites = limit(g(z),z,1) + limit(f(z),z,1)

print('A soma dos limites das funções é:',soma_limites)

```

    O limite da soma das funções é: 3
    A soma dos limites das funções é: 3

### Constante vezes função

```
soma_funcoes = 4 * (f(z) + g(z))
print('O limite da soma das funções é:',limit(soma_funcoes,z, 1))

soma_limites = 4 * (limit(g(z),z,1) + limit(f(z),z,1))

print('A soma dos limites das funções é:',soma_limites)
```

    O limite da soma das funções é: 12
    A soma dos limites das funções é: 12

## Propriedades Limites: Parte 2

1 - O limite do produto das funções é equivalente ao produto dos limites.

$
\lim_{x \to x_0} f(x)  *g(x) = \lim_{x \to x_0} f(x) * \lim_{x \to x_0} g(x)
$

2 - O limite da divisão das funções é equivalente à divisão dos limites.

$
\lim_{x \to x_0} \dfrac{f(x)}{g(x)} =  \dfrac{\lim_{x \to x_0}f(x)}{\lim_{x \to x_0} g(x)}
$

**Desde que:**

$
\lim_{x \to x_0} g(x) \neq 0
$

```
# Reutilizando as funções já criadas na parte 01
print(f(z),g(z),'\n')

produto_funcoes = f(z) * g(z)
print('O limite do produto das funções é:',limit(produto_funcoes, z, 2))

produto_limites = limit(f(z),z, 2) * limit(g(z), z, 2)
print('O limite do produto dos limites é:',limit(produto_limites, z, 2))

print()
try:
  divisao_funcoes = f(z) / g(z)
  print('O limite da divisão das funções é:',limit(divisao_funcoes, z, 2))

  divisao_limites = limit(f(z),z, 2) / limit(g(z), z, 2)
  print('O limite da divisão dos limites é:',limit(divisao_limites, z, 2))
except:
  print('g(x) == 0')
```

    z + 1 z

    O limite do produto das funções é: 6
    O limite do produto dos limites é: 6

    O limite da divisão das funções é: 3/2
    O limite da divisão dos limites é: 3/2

## Existência do Limite

```
var('x')
def f(x): return 2*x**2

eixo_x_ex05 = list()
eixo_y_ex05 = list()

for i in np.arange(0,5, step= 0.1):
  eixo_x_ex05.append(i)
  eixo_y_ex05.append(f(i))

plt.plot(eixo_x_ex05,eixo_y_ex05)
```

```
print('Limite Lateral pela Direita:',limit(f(x),x,2))
print('Limite Lateral pela Esquerda:',limit(f(x),x,2,'-'))
```

    Limite Lateral pela Direita: 8
    Limite Lateral pela Esquerda: 8


# Curse of Dimensionality

```julia
using Plots, SpecialFunctions
```

## 2D bionomial coefficients

This problem can be simplified as choosing M pieces out of 2M total pieces of the path. The selected pieces will form the right directions and others will be upward directions.

$$
\begin{equation}
\binom{2M}{M} = \frac{2M!}{M! \times M!}
\end{equation}
$$

## 3D problem

This problem can be simplified as choosing $k_1=M$ pieces and $k_2=M$ out of 3M total pieces of the path. The selected pieces will form the directions in each dimension

$$
\begin{equation}
\binom{3M}{M,M} = \frac{3M!}{M! \times M! \times M!}
\end{equation}
$$

## ND problem

This problem can be simplified as choosing $k_1=M$ pieces, $k_2=M$, and $\dots$, etc. out of NM total pieces of the path. The selected pieces will form the directions in each dimension

$$
\begin{equation}
\binom{NM}{k_1=M,k_2=M,\dots,k_{N-1}=M} = \frac{NM!}{M!^N}
\end{equation}
$$

# Visualization

```julia
function npaths(N)
    nom = gamma(N*5+1)
    den = 1
    for n=1:N
        den *= gamma(6)
    end
    nom/den
end

N = 1:10
scatter(N, npaths.(N), yaxis=:log, label=:false, title="Path through lattice of side 5",
    xlabel="Numer of dimention", ylabel="Path count")
```

## Adjourn

```julia
using Dates
println("mahdiar")
Dates.format(now(), "Y/U/d HH:MM")
```

    mahdiar

    "2021/April/17 17:05"


# Solving Linear Systems

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as la
%matplotlib inline
```

## Linear Systems

A [linear system of equations](https://en.wikipedia.org/wiki/System_of_linear_equations) is a collection of linear equations

\begin{align}
a_{0,0}x_0 + a_{0,1}x_2 + \cdots + a_{0,n}x_n & = b_0 \\\
a_{1,0}x_0 + a_{1,1}x_2 + \cdots + a_{1,n}x_n & = b_1 \\\
& \vdots \\\
a_{m,0}x_0 + a_{m,1}x_2 + \cdots + a_{m,n}x_n & = b_m \\\
\end{align}

In matrix notation, a linear system is $A \mathbf{x}= \mathbf{b}$ where

$$
A = \begin{bmatrix}
a_{0,0} & a_{0,1} & \cdots & a_{0,n} \\\
a_{1,0} & a_{1,1} & \cdots & a_{1,n} \\\
\vdots & & & \vdots \\\
a_{m,0} & a_{m,1} & \cdots & a_{m,n} \\\
\end{bmatrix}
 \ \ , \ \
\mathbf{x} = \begin{bmatrix}
x_0 \\\ x_1 \\\ \vdots \\\ x_n
\end{bmatrix}
 \ \ , \ \
\mathbf{b} = \begin{bmatrix}
b_0 \\\ b_1 \\\ \vdots \\\ b_m
\end{bmatrix}
$$

## Gaussian elimination

The general procedure to solve a linear system of equation is called [Gaussian elimination](https://en.wikipedia.org/wiki/Gaussian_elimination). The idea is to perform elementary row operations to reduce the system to its row echelon form and then solve.

### Elementary Row Operations

[Elementary row operations](https://en.wikipedia.org/wiki/Elementary_matrix#Elementary_row_operations) include:

1. Add $k$ times row $j$ to row $i$.
2. Multiply row $i$ by scalar $k$.
3. Switch rows $i$ and $j$.

Each of the elementary row operations is the result of matrix multiplication by an elementary matrix (on the left).
To add $k$ times row $i$ to row $j$ in a matrix $A$, we multiply $A$ by the matrix $E$ where $E$ is equal to the identity matrix except the $i,j$ entry is $E_{i,j} = k$. For example, if $A$ is 3 by 3 and we want to add 3 times row 2 to row 0 (using 0 indexing) then

$$
E_1 = \begin{bmatrix}
1 & 0 & 3 \\\
0 & 1 & 0 \\\
0 & 0 & 1
\end{bmatrix}
$$

Let's verify the calculation:

```python
A = np.array([[1,1,2],[-1,3,1],[0,5,2]])
print(A)
```

    [[ 1  1  2]
     [-1  3  1]
     [ 0  5  2]]

```python
E1 = np.array([[1,0,3],[0,1,0],[0,0,1]])
print(E1)
```

    [[1 0 3]
     [0 1 0]
     [0 0 1]]

```python
E1 @ A
```

    array([[ 1, 16,  8],
           [-1,  3,  1],
           [ 0,  5,  2]])

To multiply $k$ times row $i$ in a matrix $A$, we multiply $A$ by the matrix $E$ where $E$ is equal to the identity matrix except the $,i,j$ entry is $E_{i,i} = k$. For example, if $A$ is 3 by 3 and we want to multiply row 1 by -2 then

$$
E_2 = \begin{bmatrix}
1 & 0 & 0 \\\
0 & -2 & 0 \\\
0 & 0 & 1
\end{bmatrix}
$$

Let's verify the calculation:

```python
E2 = np.array([[1,0,0],[0,-2,0],[0,0,1]])
print(E2)
```

    [[ 1  0  0]
     [ 0 -2  0]
     [ 0  0  1]]

```python
E2 @ A
```

    array([[ 1,  1,  2],
           [ 2, -6, -2],
           [ 0,  5,  2]])

Finally, to switch row $i$ and row $j$ in a matrix $A$, we multiply $A$ by the matrix $E$ where $E$ is equal to the identity matrix except $E_{i,i} = 0$, $E_{j,j} = 0$, $E_{i,j} = 1$ and $E_{j,i} = 1$. For example, if $A$ is 3 by 3 and we want to switch row 1 and row 2 then

$$
E^3 = \begin{bmatrix}
1 & 0 & 0 \\\
0 & 0 & 1 \\\
0 & 1 & 0
\end{bmatrix}
$$

Let's verify the calculation:

```python
E3 = np.array([[1,0,0],[0,0,1],[0,1,0]])
print(E3)
```

    [[1 0 0]
     [0 0 1]
     [0 1 0]]

```python
E3 @ A
```

    array([[ 1,  1,  2],
           [ 0,  5,  2],
           [-1,  3,  1]])

### Implementation

Let's write function to implement the elementary row operations. First of all, let's write a function called `add_rows` which takes input parameters $A$, $k$, $i$ and $j$ and returns the NumPy array resulting from adding $k$ times row $j$ to row $i$ in the matrix $A$. If $i=j$, then let's say that the function scales row $i$ by $k+1$ since this would be the result of $k$ times row $i$ added to row $i$.

```python
def add_row(A,k,i,j):
    "Add k times row j to row i in matrix A."
    n = A.shape[0]
    E = np.eye(n)
    if i == j:
        E[i,i] = k + 1
    else:
        E[i,j] = k
    return E @ A
```

Let's test our function:

```python
M = np.array([[1,1],[3,2]])
print(M)
```

    [[1 1]
     [3 2]]

```python
add_row(M,2,0,1)
```

    array([[7., 5.],
           [3., 2.]])

```python
add_row(M,3,1,1)
```

    array([[ 1.,  1.],
           [12.,  8.]])

Let's write a function called `scale_row` which takes 3 input parameters $A$, $k$, and $i$ and returns the matrix that results from $k$ times row $i$ in the matrix $A$.

```python
def scale_row(A,k,i):
    "Multiply row i by k in matrix A."
    n = A.shape[0]
    E = np.eye(n)
    E[i,i] = k
    return E @ A
```

```python
M = np.array([[3,1],[-2,7]])
print(M)
```

    [[ 3  1]
     [-2  7]]

```python
scale_row(M,3,1)
```

    array([[ 3.,  1.],
           [-6., 21.]])

```python
A = np.array([[1,1,1],[1,-1,0]])
print(A)
```

    [[ 1  1  1]
     [ 1 -1  0]]

```python
scale_row(A,5,1)
```

    array([[ 1.,  1.,  1.],
           [ 5., -5.,  0.]])

Let's write a function called `switch_rows` which takes 3 input parameters $A$, $i$ and $j$ and returns the matrix that results from switching rows $i$ and $j$ in the matrix $A$.

```python
def switch_rows(A,i,j):
    "Switch rows i and j in matrix A."
    n = A.shape[0]
    E = np.eye(n)
    E[i,i] = 0
    E[j,j] = 0
    E[i,j] = 1
    E[j,i] = 1
    return E @ A
```

```python
A = np.array([[1,1,1],[1,-1,0]])
print(A)
```

    [[ 1  1  1]
     [ 1 -1  0]]

```python
switch_rows(A,0,1)
```

    array([[ 1., -1.,  0.],
           [ 1.,  1.,  1.]])

## Examples

### Find the Inverse

Let's apply our functions to the augmented matrix $[M \ | \ I]$ to find the inverse of the matrix $M$:

```python
M = np.array([[5,4,2],[-1,2,1],[1,1,1]])
print(M)
```

    [[ 5  4  2]
     [-1  2  1]
     [ 1  1  1]]

```python
A = np.hstack([M,np.eye(3)])
print(A)
```

    [[ 5.  4.  2.  1.  0.  0.]
     [-1.  2.  1.  0.  1.  0.]
     [ 1.  1.  1.  0.  0.  1.]]

```python
A1 = switch_rows(A,0,2)
print(A1)
```

    [[ 1.  1.  1.  0.  0.  1.]
     [-1.  2.  1.  0.  1.  0.]
     [ 5.  4.  2.  1.  0.  0.]]

```python
A2 = add_row(A1,1,1,0)
print(A2)
```

    [[1. 1. 1. 0. 0. 1.]
     [0. 3. 2. 0. 1. 1.]
     [5. 4. 2. 1. 0. 0.]]

```python
A3 = add_row(A2,-5,2,0)
print(A3)
```

    [[ 1.  1.  1.  0.  0.  1.]
     [ 0.  3.  2.  0.  1.  1.]
     [ 0. -1. -3.  1.  0. -5.]]

```python
A4 = switch_rows(A3,1,2)
print(A4)
```

    [[ 1.  1.  1.  0.  0.  1.]
     [ 0. -1. -3.  1.  0. -5.]
     [ 0.  3.  2.  0.  1.  1.]]

```python
A5 = scale_row(A4,-1,1)
print(A5)
```

    [[ 1.  1.  1.  0.  0.  1.]
     [ 0.  1.  3. -1.  0.  5.]
     [ 0.  3.  2.  0.  1.  1.]]

```python
A6 = add_row(A5,-3,2,1)
print(A6)
```

    [[  1.   1.   1.   0.   0.   1.]
     [  0.   1.   3.  -1.   0.   5.]
     [  0.   0.  -7.   3.   1. -14.]]

```python
A7 = scale_row(A6,-1/7,2)
print(A7)
```

    [[ 1.          1.          1.          0.          0.          1.        ]
     [ 0.          1.          3.         -1.          0.          5.        ]
     [ 0.          0.          1.         -0.42857143 -0.14285714  2.        ]]

```python
A8 = add_row(A7,-3,1,2)
print(A8)
```

    [[ 1.          1.          1.          0.          0.          1.        ]
     [ 0.          1.          0.          0.28571429  0.42857143 -1.        ]
     [ 0.          0.          1.         -0.42857143 -0.14285714  2.        ]]

```python
A9 = add_row(A8,-1,0,2)
print(A9)
```

    [[ 1.          1.          0.          0.42857143  0.14285714 -1.        ]
     [ 0.          1.          0.          0.28571429  0.42857143 -1.        ]
     [ 0.          0.          1.         -0.42857143 -0.14285714  2.        ]]

```python
A10 = add_row(A9,-1,0,1)
print(A10)
```

    [[ 1.          0.          0.          0.14285714 -0.28571429  0.        ]
     [ 0.          1.          0.          0.28571429  0.42857143 -1.        ]
     [ 0.          0.          1.         -0.42857143 -0.14285714  2.        ]]

Let's verify that we found the inverse $M^{-1}$ correctly:

```python
Minv = A10[:,3:]
print(Minv)
```

    [[ 0.14285714 -0.28571429  0.        ]
     [ 0.28571429  0.42857143 -1.        ]
     [-0.42857143 -0.14285714  2.        ]]

```python
result = Minv @ M
print(result)
```

    [[ 1.00000000e+00  4.44089210e-16  2.22044605e-16]
     [-6.66133815e-16  1.00000000e+00 -2.22044605e-16]
     [ 0.00000000e+00  0.00000000e+00  1.00000000e+00]]

Success! We can see the result more clearly if we round to 15 decimal places:

```python
np.round(result,15)
```

    array([[ 1.e+00,  0.e+00,  0.e+00],
           [-1.e-15,  1.e+00, -0.e+00],
           [ 0.e+00,  0.e+00,  1.e+00]])

### Solve a System

Let's use our functions to perform Gaussian elimination and solve a linear system of equations $A \mathbf{x} = \mathbf{b}$.

```python
A = np.array([[6,15,1],[8,7,12],[2,7,8]])
print(A)
```

    [[ 6 15  1]
     [ 8  7 12]
     [ 2  7  8]]

```python
b = np.array([[2],[14],[10]])
print(b)
```

    [[ 2]
     [14]
     [10]]

Form the augemented matrix $M$:

```python
M = np.hstack([A,b])
print(M)
```

    [[ 6 15  1  2]
     [ 8  7 12 14]
     [ 2  7  8 10]]

Perform row operations:

```python
M1 = scale_row(M,1/6,0)
print(M1)
```

    [[ 1.          2.5         0.16666667  0.33333333]
     [ 8.          7.         12.         14.        ]
     [ 2.          7.          8.         10.        ]]

```python
M2 = add_row(M1,-8,1,0)
print(M2)
```

    [[  1.           2.5          0.16666667   0.33333333]
     [  0.         -13.          10.66666667  11.33333333]
     [  2.           7.           8.          10.        ]]

```python
M3 = add_row(M2,-2,2,0)
print(M3)
```

    [[  1.           2.5          0.16666667   0.33333333]
     [  0.         -13.          10.66666667  11.33333333]
     [  0.           2.           7.66666667   9.33333333]]

```python
M4 = scale_row(M3,-1/13,1)
print(M4)
```

    [[ 1.          2.5         0.16666667  0.33333333]
     [ 0.          1.         -0.82051282 -0.87179487]
     [ 0.          2.          7.66666667  9.33333333]]

```python
M5 = add_row(M4,-2,2,1)
print(M5)
```

    [[ 1.          2.5         0.16666667  0.33333333]
     [ 0.          1.         -0.82051282 -0.87179487]
     [ 0.          0.          9.30769231 11.07692308]]

```python
M6 = scale_row(M5,1/M5[2,2],2)
print(M6)
```

    [[ 1.          2.5         0.16666667  0.33333333]
     [ 0.          1.         -0.82051282 -0.87179487]
     [ 0.          0.          1.          1.19008264]]

```python
M7 = add_row(M6,-M6[1,2],1,2)
print(M7)
```

    [[1.         2.5        0.16666667 0.33333333]
     [0.         1.         0.         0.1046832 ]
     [0.         0.         1.         1.19008264]]

```python
M8 = add_row(M7,-M7[0,2],0,2)
print(M8)
```

    [[1.         2.5        0.         0.13498623]
     [0.         1.         0.         0.1046832 ]
     [0.         0.         1.         1.19008264]]

```python
M9 = add_row(M8,-M8[0,1],0,1)
print(M9)
```

    [[ 1.          0.          0.         -0.12672176]
     [ 0.          1.          0.          0.1046832 ]
     [ 0.          0.          1.          1.19008264]]

Success! The solution of $Ax=b$ is

```python
x = M9[:,3].reshape(3,1)
print(x)
```

    [[-0.12672176]
     [ 0.1046832 ]
     [ 1.19008264]]

Or, we can do it the easy way...

```python
x = la.solve(A,b)
print(x)
```

    [[-0.12672176]
     [ 0.1046832 ]
     [ 1.19008264]]

## `scipy.linalg.solve`

We are mostly interested in linear systems $A \mathbf{x} = \mathbf{b}$ where there is a unique solution $\mathbf{x}$. This is the case when $A$ is a square matrix ($m=n$) and $\mathrm{det}(A) \not= 0$. To solve such a system, we can use the function [`scipy.linalg.solve`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.solve.html).

The function returns a solution of the system of equations $A \mathbf{x} = \mathbf{b}$. For example:

```python
A = np.array([[1,1],[1,-1]])
print(A)
```

    [[ 1  1]
     [ 1 -1]]

```python
b1 = np.array([2,0])
print(b1)
```

    [2 0]

And solve:

```python
x1 = la.solve(A,b1)
print(x1)
```

    [1. 1.]

Note that the output $\mathbf{x}$ is returned as a 1D NumPy array when the vector $\mathbf{b}$ (the right hand side) is entered as a 1D NumPy array. If we input $\mathbf{b}$ as a 2D NumPy array, then the output is a 2D NumPy array. For example:

```python
A = np.array([[1,1],[1,-1]])
b2 = np.array([2,0]).reshape(2,1)
x2 = la.solve(A,b2)
print(x2)
```

    [[1.]
     [1.]]

Finally, if the right hand side $\mathbf{b}$ is a matrix, then the output is a matrix of the same size. It is the solution of $A \mathbf{x} = \mathbf{b}$ when $\mathbf{b}$ is a matrix. For example:

```python
A = np.array([[1,1],[1,-1]])
b3 = np.array([[2,2],[0,1]])
x3 = la.solve(A,b3)
print(x3)
```

    [[1.  1.5]
     [1.  0.5]]

### Simple Example

Let's compute the solution of the system of equations

\begin{align}
2x + y &= 1 \\\
x + y &= 1
\end{align}

Create the matrix of coefficients:

```python
A = np.array([[2,1],[1,1]])
print(A)
```

    [[2 1]
     [1 1]]

And the vector $\mathbf{b}$:

```python
b = np.array([1,-1]).reshape(2,1)
print(b)
```

    [[ 1]
     [-1]]

And solve:

```python
x = la.solve(A,b)
print(x)
```

    [[ 2.]
     [-3.]]

We can verify the solution by computing the inverse of $A$:

```python
Ainv = la.inv(A)
print(Ainv)
```

    [[ 1. -1.]
     [-1.  2.]]

And multiply $A^{-1} \mathbf{b}$ to solve for $\mathbf{x}$:

```python
x = Ainv @ b
print(x)
```

    [[ 2.]
     [-3.]]

We get the same result. Success!

### Inverse or Solve

It's a bad idea to use the inverse $A^{-1}$ to solve $A \mathbf{x} = \mathbf{b}$ if $A$ is large. It's too computationally expensive. Let's create a large random matrix $A$ and vector $\mathbf{b}$ and compute the solution $\mathbf{x}$ in 2 ways:

```python
N = 1000
A = np.random.rand(N,N)
b = np.random.rand(N,1)
```

Check the first entries $A$:

```python
A[:3,:3]
```

    array([[0.35754719, 0.63135432, 0.6572258 ],
           [0.18450506, 0.14639832, 0.23528745],
           [0.27576474, 0.46264005, 0.26589724]])

And for $\mathbf{b}$:

```python
b[:4,:]
```

    array([[0.82726751],
           [0.96946096],
           [0.31351176],
           [0.63757837]])

Now we compare the speed of `scipy.linalg.solve` with `scipy.linalg.inv`:

```python
%%timeit
x = la.solve(A,b)
```

    2.77 s ± 509 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%%timeit
x = la.inv(A) @ b
```

    4.46 s ± 2.04 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

Solving with `scipy.linalg.solve` is about twice as fast!

Python/3. Computational Sciences and Mathematics/Linear Algebra/Solving Systems of Linear Equations.ipynb


# Neural Network

## neuron

a neuron takes input $x \in \mathbb{R}^{d}$, multiply $x$ by weights $w$ and add bias term $b$, finally use a activation function $g$.

that is:

$$f(x) = g(w^{T}x + b)$$

it is analogous to the functionality of biological neuron.

some useful activation function:

$$
\begin{equation}
\begin{split}
\text{sigmoid:}\quad &g(z) = \frac{1}{1 + e^{-z}} \\
\text{tanh:}\quad &g(z) = \frac{e^{z}-e^{-z}}{e^{z} + e^{-z}} \\
\text{relu:}\quad &g(z) = max(z,0) \\
\text{leaky relu:}\quad &g(z) = max(z, \epsilon{z})\ ,\ \epsilon\text{ is a small positive number}\\
\text{identity:}\quad &g(z) = z
\end{split}
\end{equation}
$$

linear regression's forward process is a neuron with identity activation function.

logistic regression's forward process is a neuron with sigmoid activation function.

## neural network

building neural network is analogous to lego bricks: you take individual bricks and stack them together to build complex structures.

we use bracket to denote layer, we take the above as example

$[0]$ denote input layer, $[1]$ denote hidden layer, $[2]$ denote output layer

$a^{[l]}$ denote the output of layer $l$, set $a^{[0]} := x$

$z^{[l]}$ denote the affine result of layer $l$

we have:

$$z^{[l]} = W^{[l]}a^{[l-1]} + b^{[l]}$$

$$a^{[l]} = g^{[l]}(z^{[l]})$$

where $W^{[l]} \in \mathbb{R}^{d[l] \times d[l-1]}$, $b^{[l]} \in \mathbb{R}^{d[l]}$.

## weight decay

recall that to mitigate overfitting, we use $l_{2}$ and $l_{1}$ regularization in linear and logistic regression.

weight decay is a alias of $l_{2}$ regularization, can be generalize to neural network, we concatenate $W^{[l]}$ and flatten it to get $w$ in this setting.

first adding $l_{2}$ norm penalty:

$$J(w,b) = \sum_{i=1}^{n}l(w, b, x^{(i)}, y^{(i)}) + \frac{\lambda}{2}\left \|  w \right \|^{2} $$

then by gradient descent, we have:

$$
\begin{equation}
\begin{split}
w:=& w-\eta\frac{\partial}{\partial w}J(w, b) \\
=& w-\eta\frac{\partial}{\partial w}\left(\sum_{i=1}^{n}l(w, b, x^{(i)}, y^{(i)}) + \frac{\lambda}{2}\left \|  w \right \|^{2}\right) \\
=& (1 - \eta\lambda)w - \eta\frac{\partial}{\partial w}\sum_{i=1}^{n}l(w, b, x^{(i)}, y^{(i)})
\end{split}
\end{equation}
$$

multiply by $(1 - \eta\lambda)$ is weight decay.

often we do not calculate bias term in regularization, so does weight decay.

## dropout

to strength robustness through perturbation, we can deliberately add perturbation in traning, dropout is one of that skill.

we actually do the following in hidden neuron:

$$
a_{dropout} =
\begin{cases}
0 &\text{with probability }p \\
\frac{a}{1-p} &\text{otherwise}
\end{cases}
$$

this operation randomly dropout neuron with probability $p$ and keep the expectation unchanged:

$$E(a_{dropout}) = E(a)$$

depict this process below:

one more thing: we do not use dropout in predicting.

## prerequesities for back-propagation

suppose in forward-propagation $x \to y \to l$, where $x \in \mathbb{R}^{n}$, $y \in \mathbb{R} ^{m}$, loss $l \in \mathbb{R}$.

then:

$$
\frac{\partial l}{\partial y} = \begin{bmatrix}
 \frac{\partial l}{\partial y_{1}} \\
 ...\\
 \frac{\partial l}{\partial y_{m}}
\end{bmatrix}
\quad
\frac{\partial l}{\partial x} = \begin{bmatrix}
 \frac{\partial l}{\partial x_{1}} \\
 ...\\
 \frac{\partial l}{\partial x_{n}}
\end{bmatrix}
$$

by total differential equation:

$$
\frac{\partial l}{\partial x_{k}} = \sum_{j=1}^{m}\frac{\partial l}{\partial y_{j}}\frac{\partial y_{j}}{\partial x_{k}}
$$

then we can connect $\frac{\partial l}{\partial x}$ and $\frac{\partial l}{\partial y}$ by:

$$
\frac{\partial l}{\partial x} = \begin{bmatrix}
 \frac{\partial l}{\partial x_{1}} \\
 ...\\
 \frac{\partial l}{\partial x_{n}}
\end{bmatrix}
=
\begin{bmatrix}
 \frac{\partial y_{1}}{\partial x_{1}} & ... & \frac{\partial y_{m}}{\partial x_{1}}\\
 \vdots  & \ddots  & \vdots \\
  \frac{\partial y_{1}}{\partial x_{n}}& .... & \frac{\partial y_{m}}{\partial x_{n}}
\end{bmatrix}
\begin{bmatrix}
 \frac{\partial l}{\partial y_{1}} \\
 ...\\
 \frac{\partial l}{\partial y_{m}}
\end{bmatrix}
=
(\frac{\partial y}{\partial x})^{T}\frac{\partial l}{\partial y}
$$

here $\frac{\partial y}{\partial x}$ is the jacobian matrix.

unlike other activation functions, calculate softmax depend on other neurons, so jacobian of softmax.

$$
\frac{\partial a_{i}}{\partial z_{j}} = \frac{\partial}{\partial z_{j}}\left(\frac{exp(z_{i})}{\sum_{s=1}^{k}exp(z_{s})}\right)
$$

it is easy to check the jacobian of matrix-multiplication:

$$\frac{\partial Mx}{\partial x} = M$$

## back-propagation

gradient descent update rule:

$$W^{[l]} = W^{[l]} - \alpha\frac{\partial{L}}{\partial{W^{[l]}}}$$

$$b^{[l]} = b^{[l]} - \alpha\frac{\partial{L}}{\partial{b^{[l]}}}$$

to proceed, we must compute the gradient with respect to the parameters.

we can define a three-step recipe for computing the gradients as follows:

1.for output layer, we have:

$$
\frac{\partial L(\hat{y}, y)}{\partial z^{[N]}} = (\frac{\partial \hat{y}}{\partial z^{[N]}})^{T}\frac{\partial L(\hat{y}, y)}{\partial \hat{y}}
$$

if $g^{[N]}$ is softmax.

$$
\frac{\partial L(\hat{y}, y)}{\partial z^{[N]}} = \frac{\partial L(\hat{y}, y)}{\partial \hat{y}} \odot {g^{[N]}}'(z^{[N]})
$$

if not softmax.

the above computations are all straight forward.

2.for $l=N-1,...,1$, we have:

$$z^{[l + 1]} = W^{[l + 1]}a^{[l]} + b^{[l + 1]}$$

so by our prerequesities:

$$
\frac{\partial L}{\partial a^{[l]}} = (\frac{\partial z^{[l+1]}}{\partial a^{[l]}})^{T}\frac{\partial L}{\partial z^{[l+1]}} = (W^{[l+1]})^{T}\frac{\partial L}{\partial z^{[l+1]}}
$$

we also have:

$$a^{[l]} = g^{[l]}z^{[l]}$$

we do not use softmax activation in hidden layers, so the dependent is direct:

$$\frac{\partial L}{\partial z^{[l]}} = \frac{\partial L}{\partial a^{[l]}} \odot {g^{[l]}}'(z^{[l]})$$

combine two equations:

$$\frac{\partial L}{\partial z^{[l]}} = (W^{[l+1]})^{T}\frac{\partial L}{\partial z^{[l+1]}} \odot {g^{[l]}}'(z^{[l]})$$

3.final step, because:

$$z^{[l]} = W^{[l]}a^{[l - 1]} + b^{[l]}$$

so:

$$\frac{\partial L}{\partial W^{[l]}} = \frac{\partial L}{\partial z^{[l]}}(a^{[l - 1]})^{T}$$

$$\frac{\partial L}{\partial b^{[l]}}=\frac{\partial L}{\partial z^{[l]}}$$

## xavier initialization

to mitigate vanishing and exploding gradient, to insure breaking symmtry, we should carefully initialize weights.

consider a fully connected layer without bias term and activation function:

$$o_{i} = \sum_{j=1}^{n_{in}}w_{ij}x_{j}$$

suppose $w_{ij}$ draw from a distribution of 0 mean and $\sigma^{2}$ variance, not necessarily guassian.

suppose $x_{j}$ draw from a distribution of 0 mean and $\gamma^{2}$ variance, all $w_{ij}, x_{j}$ are independent.

then mean of $o_{i}$ is of course 0, variance:

$$
\begin{equation}
\begin{split}
Var[o_{i}] =& E[o_{i}^{2}] - (E[o_{i}])^{2}\\
=&\sum_{j=1}^{n_{in}}E[w_{ij}^{2}x_{j}^{2}] \\
=&\sum_{j=1}^{n_{in}}E[w_{ij}^{2}]E[x_{j}^{2}] \\
=&n_{in}\sigma^{2}\gamma^{2}
\end{split}
\end{equation}
$$

to keep variance fixed, we need to set $n_{in}\sigma^{2}=1$.

consider back-propagation, we have:

$$\frac{\partial L}{\partial x_{j}} = \sum_{i=1}^{n_{out}}w_{ij}\frac{\partial L}{\partial o_{i}}$$

so by the same inference, we need to set $$n_{out}\sigma^{2} = 1$$.

we cannot satisfy both conditions simutaneously, we simply try to satisfy:

$$\frac{1}{2}(n_{in} + n_{out})\sigma^{2} = 1 \ \text{ or }\ \sigma = \sqrt{\frac{n_{in} + n_{out}}{2}}$$

this is the reasoning under xavier initialization.

```python

```