## In-Class Demonstration and Visualization

```python
%matplotlib inline
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
```

```python
# You can change the default figure size to be a bit larger if you want,
# uncomment the next line for that:
plt.rc('figure', figsize=(10, 6))
```

The function `plot_taylor_approximations` included here was written by Fernando Perez and was part of work on the original `IPython` project.  Although attribution seems to have been lost over time, we gratefully acknowledge FP and thank him for this code!

```python
def plot_taylor_approximations(func, x0=None, orders=(2, 4), xrange=(0,1), yrange=None, npts=200):
    """Plot the Taylor series approximations to a function at various orders.

    Parameters
    ----------
    func : a sympy function
    x0 : float
      Origin of the Taylor series expansion.  If not given, x0=xrange[0].
    orders : list
      List of integers with the orders of Taylor series to show.  Default is (2, 4).
    xrange : 2-tuple or array.
      Either an (xmin, xmax) tuple indicating the x range for the plot (default is (0, 1)),
      or the actual array of values to use.
    yrange : 2-tuple
      (ymin, ymax) tuple indicating the y range for the plot.  If not given,
      the full range of values will be automatically used.
    npts : int
      Number of points to sample the x range with.  Default is 200.
    """
    if not callable(func):
        raise ValueError('func must be callable')
    if isinstance(xrange, (list, tuple)):
        x = np.linspace(float(xrange[0]), float(xrange[1]), npts)
    else:
        x = xrange
    if x0 is None: x0 = x[0]
    xs = sp.Symbol('x')
    # Make a numpy-callable form of the original function for plotting
    fx = func(xs)
    f = sp.lambdify(xs, fx, modules=['numpy'])
    # We could use latex(fx) instead of str(), but matploblib gets confused
    # with some of the (valid) latex constructs sympy emits.  So we play it safe.
    plt.plot(x, f(x), label=str(fx), lw=2)
    # Build the Taylor approximations, plotting as we go
    apps = {}
    for order in orders:
        app = fx.series(xs, x0, n=order).removeO()
        apps[order] = app
        # Must be careful here: if the approximation is a constant, we can't
        # blindly use lambdify as it won't do the right thing.  In that case,
        # evaluate the number as a float and fill the y array with that value.
        if isinstance(app, sp.numbers.Number):
            y = np.zeros_like(x)
            y.fill(app.evalf())
        else:
            fa = sp.lambdify(xs, app, modules=['numpy'])
            y = fa(x)
        tex = sp.latex(app).replace('$', '')
        plt.plot(x, y, label=r'$n=%s:\, %s$' % (order, tex) )

    # Plot refinements
    if yrange is not None:
        plt.ylim(*yrange)
    plt.grid()
    plt.legend(loc='best').get_frame().set_alpha(0.8)

# For an expression made from elementary functions, we must first make it into
# a callable function, the simplest way is to use the Python lambda construct.
# plot_taylor_approximations(lambda x: 1/sp.cos(x), 0, [2,4,6,8], (0, 2*sp.pi), (-5,5))
```

```python
plot_taylor_approximations(sp.sin, 0, [2, 4, 6, 8], (0, 2*sp.pi), (-2,2))
```

# Lecture 10:  Taylor's Series and Discrete Calculus

## Background

It is common in physics and engineering to represent transcendental functions and other nonlinear expressions using a few terms from a Taylor series.  This series provides a fast and efficient way to compute quantities such as $\mathrm{sin}(x)$ or $e^x$ to a prescribed error.  Learning how to calculate the series representation of these functions will provide practical experience with the Taylor series and help the student understand the results of Python methods designed to accelerate and simplify computations.  The series can be written generally as:

$$f(x) = f{\left (0 \right )} + x \left. \frac{d}{d x} f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{2}}{2} \left. \frac{d^{2}}{d x^{2}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{3}}{6} \left. \frac{d^{3}}{d x^{3}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{4}}{24} \left. \frac{d^{4}}{d x^{4}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \frac{x^{5}}{120} \left. \frac{d^{5}}{d x^{5}}  f{\left (x \right )} \right|_{\substack{ x=0 }} + \mathcal{O}\left(x^{6}\right)$$

Of equal importance the Taylor series permits discrete representation of derivatives and is a common way to perform numerical integration of partial and ordinary differential equations.  Expansion of a general function $f(x)$ about a point, coupled with algebraic manipulation, will produce expressions that can be used to approximate derivative quantities.  Although any order of derivative can be computed, this lesson will focus on first and second derivatives that will be encountered in the diffusion equation.

## What Skills Will I Learn?

You will practice the following skills:

* Defining and determining the limits of infinite sequences, series and power series.
* Define the Taylor series and write the general form about any point and to any order.
* Derive the central and forward difference formulae for numerical derivatives using the Taylor's series.

## What Steps Should I Take?

1.  Learn to use Sympy to define and find the limits of sequences are series.
1.  Learn how to approximate transcendental functions about center points of your choosing.
1.  Differentiate an explicit series representation of a function to see that the coefficients of such a series can be determined algebraically.
1.  Use Sympy to compute a power series symbolically
1.  Derive the finite difference expressions for the first and second derivatives.
1.  Read the relevant pages from Hornbeck's text on numerical methods.
1.  Generate a list of values that approximate the function $f(x)=x^8$ on the domain $\{x | 0 \leq x \leq 1\}$.  Using these values, numerically compute the derivative at your selected grid points and compare it to the analytical solution.  Using this technique, examine how the observed error changes as the number of grid points is varied.  Visualize and explain the results.
1.  Prepare a new notebook (not just modifications to this one) that describes your approach.

Optional challenge:  A _list_ is one of the fundamental data structures within Python.  Numpy (a Python library) and other parts of Python libraries use _vectorized_ computations.  From Wikipedia, vectorization is "a style of computer programming where operations are applied to whole arrays instead of individual elements."

With this in mind, we certainly _can_ iterate over our list of points and apply the function that you will soon write in an element by element fashion, however, it is a more common practice in Python and other modern languages to write vectorized code.  If this is your first exposure to vectorized computation, I recommend two initial strategies:  write out your algorithms and use "classic" flow control and iteration to compute the results.  From that point you will more easily see the strategy you should use to write vectorized code.

Using the discrete forms of the first and second derivatives (based on central differences) can you devise a vectorized operation that computes the derivative without looping in Python?

## A Sucessful Jupyter Notebook Will

* Present a description of the essential elements of Taylor's series and how to compute numerical derivatives;
* Identify the audience for which the work is intended;
* Run the code necessary to compute and visualize the error associated with the second order approximation and the changes in grid point spacing;
* Provide a narrative and equations to explain why your approach is relevant to solving the problem;
* Provide references and citations to any others' work you use to complete the assignment;
* Be checked into your GitHub repository by the due date (one week from assignment).

A high quality communication provides an organized, logically progressing, blend of narrative, equations, and code that teaches the reader a particular topic or idea.  You will be assessed on:
* The functionality of the code (i.e. it should perform the task assigned).
* The narrative you present.  I should be able to read and learn from it.  Choose your audience wisely.
* The supporting equations and figures you choose to include.

If your notebook is just computer code your assignment will be marked incomplete.

## Reading and Reference

* Essential Mathematical Methods for Physicists, H. Weber and G. Arfken, Academic Press, 2003
* Advanced engineering Mathematics, E. Kreyszig, John wiley and Sons, 2010
* Numerical Recipes, W. Press, Cambridge University Press, 1986
* Numerical Methods, R. Hornbeck, Quantum Publishers, 1975

### Infinite Sequences
----

Ideas relating to sequences, series, and power series are used in the formulation of integral calculus and in the construction of polynomial representations of functions.  The limit of functions will also be investigated as boundary conditions for differential equations.  For this reason understanding concepts related to sequences and series are important to review.

A sequence is an ordered list of numbers.  A list such as the following represents a sequence:

$$a_1, a_2, a_3, a_4, \dots, a_n, \dots $$

The sequence maps one value $a_n$ for every integer $n$.  It is typical to provide a formula for construction of the nth term in the sequence.  While _ad-hoc_ strategies could be used to develop sequences using SymPy and lists in Python, SymPy has a sequence class that can be used.  A short demonstration is provided next:

```python
import sympy as sp
x, y, z, t, a = sp.symbols('x y z t a')
k, m, n = sp.symbols('k m n', integer=True)
f, g, h = sp.symbols('f g h', cls=sp.Function)
sp.var('a1:6')
sp.init_printing()
```

It is important to read about `SymPy` symbols at this time.

We can generate a sequence using `SeqFormula`.

```python
a1 = sp.SeqFormula(n**2, (n,0,5))
list(a1)
```

if we want the limit of the sequence at infinity:

$$[0, 1, 4, 9, \ldots ]$$

we can use `limit_seq`:

```python
sp.limit_seq(a1.formula, n)
```

### DIY:  Determine if the following sequences are convergent or divergent.  If convergent, what is the limit?

$$
\begin{aligned}
a_n = & \frac{1}{n} \\
a_n = & 1 - (0.2)^n \\
a_n = & \frac{1}{2n+1}
\end{aligned}
$$

```python
# Your code here.
```

### Infinite Series
----

A series is the sum of a sequence.  An infinite series will converge if the partial sums of the series has a finite limit.  For example, examine the partial sums of the series:

$$
\sum^{\infty}_{n=1} \frac{1}{2^n}
$$

```python
a2 = sp.Sum(1/2**n, (n,0,1))
a2
```

```python
a2.doit()
```

```python
a4 = sp.Sum(n**2, (n,0,5))
a4
```

```python
a5 = sp.Sum(k**2, (k, 1, m))
a5
```

```python
a4.doit()
```

```python
a5.doit()
```

A power series is of the form:

$$
\sum_{n=0}^{\infty} M_{n} x^{n} = M_0 + M_1 x + M_2 x^2 + \cdots
$$

```python
M = sp.IndexedBase('M')
```

```python
sp.Sum(M[n]*x**n, (n,0,m))
```

We can define the series about the point $a$ as follows:

```python
sp.Sum(M[n]*(x-a)**n, (n,0,m))
```

SymPy has a function that can take SymPy expressions and represent them as power series:

```python
sp.series(f(x), x, x0=0)
```

### DIY:  Use SymPy to determine series approximations to $e^x$, $sin(x)$, and $cos(x)$ about the point $x=0$.

```python
# Your code here.
```

### Taylor's Series
----

Below we present a derivation of Taylor's series and small algebraic argument for series representations of functions.  In contrast to the ability to use `sympy` functions without any deeper understanding, these presentations are intended to give you insight into the origin of the series representation and the factors present within each term.  While the algebraic presentation isn't a general case, the essential elements of a general polynomial representation are visible.

The function $f(x)$ can be expanded into an infinite series or a finite series plus an error term.  Assume that the function has a continuous nth derivative over the interval $a \le x \le b$.  Integrate the nth derivative n times:

$$\int_a^x f^n(x) dx = f^{(n-1)}(x) - f^{(n-1)}(a)$$

The power on the function $f$ in the equation above indicates the order of the derivative.  Do this n times and then solve for f(x) to recover Taylor's series.  One of the key features in this derivation is that the integral is definite.  This derivation is outlined on [Wolfram’s Mathworld](http://mathworld.wolfram.com/TaylorSeries.html).

As a second exercise, assume that we wish to expand sin(x) about x=0.  First, assume that the series exists and can be written as a power series with unknown coefficients.  As a first step, differentiate the series and the function we are expanding.  Next, let the value of x go to the value of the expansion point and it will be possible to evaluate the coefficients in turn:

$$
\sin x = A+Bx+Cx^2+Dx^3+Ex^4
$$

We can choose an expansion point (e.g. $x = 0$) and differentiate to get a set of simultaneous equations permitting determination of the coefficients.  The computer algebra system can help us with this activity:

```python
import sympy as sp
sp.init_printing()
```

```python
x, A, B, C, D, E = sp.symbols('x, A, B, C, D, E')
```

To help us get our work done we can use `sympy`'s `diff` function.  Testing this function with a known result, we can write:

```python
sp.diff(sp.sin(x),x)
```

A list comprehension is used to organize the results.  In each iteration the exact function and the power series are differentiated and stored as an element of a list.  The list can be inspected and a set of simultaneous equations can be written down and solved to determine the values of the coefficients.  Casting the list as a `sympy` `Matrix` object clarifies the correspondance between entries in the list.

```python
orderOfDifferentiation = 1
powerSeries = A+B*x+C*x**2+D*x**3+E*x**4
# Differentiate, element by element, the list [sp.sin(x),powerSeries]
[sp.diff(a,x,orderOfDifferentiation) for a in [sp.sin(x),powerSeries]]
```

A list comprehension can be used to organize and extend the results further.  We can wrap the list above into another list that changes the order of differentiation each iteration.

```python
maximumOrder = 5
funcAndSeries = [[sp.diff(a,x,order) for a in [sp.sin(x),powerSeries]] for order in range(maximumOrder)]
funcAndSeries
```

Casting the results as a `sympy` `Matrix` object the list is more easily viewed in the Jupyter notebook:

```python
sp.Matrix(funcAndSeries)
```

### DIY:  Determine the coefficients in the above power series.
----

You don't necessarily need to write code to complete this DIY problem.

```python
# Your code here if you feel you need it.
```

`Your markdown here.`

### Computing a Taylor's Series Symbolically
---

Using `sympy` the Taylor's series can be computed symbolically.

```python
from sympy import init_printing, symbols, Function
init_printing()

x, c = symbols("x,c")
f = Function("f")

f(x).series(x, x0=c, n=3)
```

One of the major uses of Taylor's series in computation is for the evaluation of derivatives.  Take note of the fact that the derivatives of a function appear in the evaluation of the series.

### Computing Derivatives of Discrete Data
---

It may be straightforward to compute the derivative of some functions.  For example:

$$f(x) = x^2$$
$$f'(x) = 2x$$

In numerical computing situations there is no analytical solution to the problem being solved and therefore no function to integrate or differentiate.  The approximate solution is available as a list of discrete points in the domain of the problem's independent variables (e.g. space, time).  The values could be represented as a list of numbers:

$$\{f(x_0), f(x_1), f(x_2), ...\}$$

The neighboring points $f(x_0)$ and $f(x_1)$ are seperated by a distance $\Delta x = x_1 - x_0$ in the independent variable.  Although this will not be apparent from the values, it is implicit in the structure of the data.  Taylor's series can be used to compute approximate derivatives of the unknown function directly from the list of points in this situation.  We are going to compute a series expansion for an unknown function $f(x)$ in the vicinity of the point $c$ and then examine the relationship between that function and it's derivative quantities at a point $c\pm h$.  The goal of the activity is to see if we can find expressions for the derivatives using the data point of interest ($c$) and its neighbors ($c \pm h$).  We are going to use the idea of _forward_ and _backward_ differences.

1. Forward differences are computed by expanding an unknown function in a Taylor series about a point “x=c” and then letting x go to c+h.
1. Then, for backward differences, let x go to c-h.

### Symbolically Computing Forward and Backward Differences
---

In the figure below we indicate the following:

* the unknown function $f(x)$ as a dashed line
* the point about which the unknown function is expanded at $x=c$
* the distance between successive points is shown as $h$
* the approximate values of the function given at the filled squares

Imagine that we take the above series expansion and use it to compute the value of the function near the point $c$.  Let us evaluate this series by adding and subtracting to the independent varable the quantity $h$.  To accomplish this we write down the series expansion for our function about the point $c$, then we let the independent variable $x \rightarrow c+h$ and $c-h$.

```python
x, h, c = sp.symbols("x,h,c")
f = sp.Function("f")

# the .subs() method replaces occurences of 'x' with something else
taylorExpansionPlus = f(x).series(x, x0=c, n=3).removeO().subs(x,c+h)
taylorExpansionMinus = f(x).series(x, x0=c, n=3).removeO().subs(x,c-h)
```

```python
taylorExpansionPlus
```

Meaning that:

$$
f(c+h) = \frac{h^{2}}{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + h \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + f{\left (c \right )}
$$

```python
taylorExpansionMinus
```

Meaning that:

$$
f(c-h) = \frac{h^{2}}{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} - h \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + f{\left (c \right )}
$$

### Solving for First and Second Derivatives
----

Inspection of the results shows that the signs on the terms containing the first derivative are different between the two expressions.  We can use this to our advantage in solving for the derivative terms explicitly.  Note that each grouped expression is equal to zero as is the default in `sympy`.

_Find the first derivative in this expression:_

```python
(taylorExpansionMinus-f(c-h))-(taylorExpansionPlus-f(c+h))
```

Remember that `sympy` expressions are zero by default.  So this is true:

$$
- 2 h \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} - f{\left (c - h \right )} + f{\left (c + h \right )} = 0
$$

_Find the second derivative in this expression:_

```python
(taylorExpansionMinus-f(c-h))+(taylorExpansionPlus-f(c+h))
```

Similarly:

$$
h^{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=c }} + 2 f{\left (c \right )} - f{\left (c - h \right )} - f{\left (c + h \right )} = 0
$$

Solve each of the equations above for the derivative quantites and you will have recovered the formula for discrete derivatives.

Use these expressions to complete your assignment as outlined above.

```python

```


```python
from sympy import *; x,h,t,y,z,n   = symbols("x h t y z n", real=True)
f, g, h = symbols('f g h', cls=Function)

for n in range(1,21,5):
    f = E**(x/n)
    g = E**(x/(n+1))
    h = E**(x/(n+2))
    p0 = plot(f,g,h,show = False,xlim = (1,10.5),size = (13,4),legend = True)
    p0[0].line_color = 'r'
    p0[1].line_color = 'g'
    p0[2].line_color = 'b'
    p0.show()

```

```python
n = symbols('n')
Eq(Limit(E**(x/n),n,oo),Limit(E**(x/n),n,oo).doit())
```

$\displaystyle \lim_{n \to \infty} e^{\frac{x}{n}} = 1$

Personal_Projects/Limit of The Denominator When Euler's Number Is Raised To A Fraction.ipynb


# HW 1

## Imports

```python
import nbtools
nbtools.setup_nb()
```

```python
import itertools
from astropy import units
import sympy
import pandas
from scipy import integrate
import numpy
import plotly.express as px

from sympy.diffgeom import Manifold, Patch
from pystein import coords, metric, curvature, geodesic
from pystein.utilities import tensor_pow as tpow, full_simplify, boundary_filter
```

```python
show_plots = True
```

## Exercises

### B1 - Polar Coords for $\mathbb{R}^2$

#### Setup Metric

```python
M = Manifold('M', dim=2)
P = Patch('origin', M)
```

##### Cartesian

```python
x, y = sympy.symbols('x, y', nonnegative=False)
cs = coords.CoordSystem('cartesian', P, [x, y])
dx, dy = cs.base_oneforms()
ds2 = tpow(dx, 2) + tpow(dy, 2)
g_cart = metric.Metric(twoform=ds2)
g_cart
```

##### Polar

```python
r, theta = sympy.symbols('r theta', nonnegative=True)
cs = coords.CoordSystem('polar', P, [r, theta])
dr, dtheta = cs.base_oneforms()
ds2 = tpow(dr, 2) + r ** 2 * tpow(dtheta, 2)
g1 = metric.Metric(twoform=ds2)
g1
```

#### Compute curvature components

```python
crs, rmns, rcs = curvature.compute_components(g1)
```

```python
curvature.display_components(crs)
```

```python
curvature.display_components(rmns)
```

```python
curvature.display_components(rcs)
```

#### Parallel Transport

```python
# Unit circle parameterized such that x(0) == x(1)
param = sympy.symbols('lambda')
r_0 = sympy.symbols('r_0', real=True)
curve = [
    r_0,
    2 * sympy.pi * param,
]
```

```python
soln = geodesic.parallel_transport_soln(param, curve, g1)
```

```python
soln.soln[0]
```

```python
g1.angle(soln.vec(0), soln.vec(1))
```

#### Compute Geodesics

```python
full_simplify(geodesic.geodesic_equation(0, sympy.symbols('lambda'), g1))
```

```python
full_simplify(geodesic.geodesic_equation(1, sympy.symbols('lambda'), g1))
```

##### Visualize Cartesian

```python
ls = numpy.arange(0, 2, 0.01)
init = (0.0, 0.0, 0.1, 0.1)
df1cart = geodesic.numerical_geodesic(g_cart, init, ls)
```

```python
ls = numpy.arange(0, 2, 0.001)
df1scart = geodesic.numerical_sampler(g_cart, ls, (0, 0), num_angles=6)
```

```python
if show_plots:
    fig = px.scatter(df1scart, x="x", y='y',
                     color='theta_0',
                     title="B1 Cart Geodesic",
                     height=600, width=600)

    fig.show()
```

##### Visualize Polar

```python
ls = numpy.arange(0, 2, 0.001)
r_0 = 1.0
dfs = [geodesic.numerical_geodesic(g1, (r_0, theta_0, 0.0, numpy.pi/2), ls).assign(theta_0=theta_0)
       for theta_0 in numpy.arange(0.0, 2 * numpy.pi, 0.2)]
df = pandas.concat(dfs, axis=0)
```

```python
df['theta'] = numpy.mod(df['theta'], 2*numpy.pi)
# df['lam'] = ls
```

```python
if show_plots:
    fig = px.scatter(df, x="theta", y='r',
                     color='theta_0',
                     title="Sample Geodesic")

    fig.show()
```

### B2 2-Sphere

#### Setup Metric

```python
M = Manifold('M', dim=2)
P = Patch('origin', M)

theta, phi, a = sympy.symbols('theta phi a', nonnegative=True)
cs = coords.CoordSystem('spherical', P, [theta, phi])
dtheta, dphi = cs.base_oneforms()
ds2 = a**2 * (tpow(dtheta, 2) + sympy.sin(theta)**2 * tpow(dphi, 2))
g2 = metric.Metric(twoform=ds2)
g2
```

#### Compute Components

```python
g2.matrix
```

```python
g2.inverse.matrix
```

```python
crs, rmn, rcs = curvature.compute_components(g2)
```

```python
curvature.display_components(crs)
```

```python
curvature.display_components(rmn)
```

```python
curvature.display_components(rcs)
```

```python
full_simplify(curvature.ricci_scalar(metric=g2))
```

#### Parallel Transport

```python
param, theta_0 = sympy.symbols('lambda theta_0')

curve = [
    theta_0,
    param,
]
```

```python
geodesic.parallel_transport_equation(0, curve, param, g2)
```

```python
full_simplify(geodesic.parallel_transport_equation(1, curve, param, g2))
```

#### Geodesics

##### Geodesic Equations

```python
full_simplify(geodesic.geodesic_equation(0, sympy.symbols('lambda'), g2))
```

```python
full_simplify(geodesic.geodesic_equation(1, sympy.symbols('lambda'), g2))
```

##### Single Geodesic

```python
ls = numpy.arange(0, 8, 0.0001)
init = (0.01, 0.001, 3.14/4, 0.0)
st = 2
sp = 2
df2 = geodesic.numerical_geodesic(g2, (numpy.pi/2, 0.0, numpy.pi/st, numpy.pi/sp), ls)
#        for theta_0 in numpy.arange(0.01, 3.14, 0.5)]
```

```python
df2['theta'] = numpy.mod(df2['theta'], 2*numpy.pi)
df2['phi'] = numpy.mod(df2['phi'], numpy.pi)
df2 = df2.reset_index().rename(columns={'index': 'order'})
df2['lam'] = ls
```

```python
if show_plots:
    fig = px.scatter(df2, x="theta", y='phi',
                     color='order',
                     title="B2 Geodesic")

    fig.show()
```

##### Multiple Initial Conditions

```python
ls = numpy.arange(0, .04, 0.00001)
st = 2
sp = .01
dfs2 = [geodesic.numerical_geodesic(g2, (theta_0, 0.0, numpy.pi/st, numpy.pi/sp), ls).assign(theta_0=theta_0)
       for theta_0 in list(numpy.arange(numpy.pi/6, 5*numpy.pi/6, 0.1)) +
       list(numpy.arange(numpy.pi/6 + numpy.pi, 5*numpy.pi/6 + numpy.pi, 0.1))]
#        for theta_0 in numpy.arange(0.01, 3.14, 0.5)]
df2b = pandas.concat(dfs2, axis=0)
```

```python
df2b['theta'] = numpy.mod(df2b['theta'], 2*numpy.pi)
df2b = df2b.reset_index().rename(columns={'index': 'order'})
```

```python
lim_df2b = boundary_filter(df2b, theta=(0, 2*3.14), phi=(0, 3.14))
```

```python
if show_plots:
    fig = px.scatter(lim_df2b, x="theta", y='phi',
                     color='theta_0',
                     title="B2 Geodesics")

    fig.show()
```

### B3 2-Sphere Sch

#### Setup Metric

```python
M = Manifold('M', dim=2)
P = Patch('origin', M)

rho, phi, a = sympy.symbols('rho phi a', nonnegative=True)
cs = coords.CoordSystem('schw', P, [rho, phi])
drho, dphi = cs.base_oneforms()
ds2 = a**2 * ( (1 / (1 - rho**2)) * tpow(drho, 2) + rho ** 2 * tpow(dphi, 2))
g3 = metric.Metric(twoform=ds2)
g3
```

#### Geodesics

```python
full_simplify(geodesic.geodesic_equation(0, sympy.symbols('lambda'), g3))
```

```python
full_simplify(geodesic.geodesic_equation(1, sympy.symbols('lambda'), g3))
```

```python
# init = (numpy.sin(numpy.pi/4), 0.0, numpy.cos(numpy.pi/4), numpy.pi/4)
init = (numpy.sin(0), 0.0, numpy.cos(numpy.pi/4), numpy.pi/4)
lambdas = numpy.arange(0, 2.1, 0.001)
df3 = geodesic.numerical_geodesic(g3, init, lambdas)
```

```python
df3['phi'] = numpy.mod(df3['phi'], numpy.pi)
df3 = df3.reset_index().rename(columns={'index': 'order'})
df3['lam'] = lambdas
```

```python
if show_plots:
    fig = px.scatter(df3, x="rho", y='phi',
    #                  color='init',
                         title="B3 Geodesic")

    fig.show()
```

### B4 - Embedded Surface - General Case

#### Setup Metric

```python
M = Manifold('M', dim=2)
P = Patch('origin', M)

x, y = sympy.symbols('x y', nonnegative=False)
_coords = [x, y]
cs = coords.CoordSystem('Cartesian', P, [x, y])
dx, dy = cs.base_oneforms()
_1forms = [dx, dy]

f = sympy.Function('f')(x, y)

ds2 = []
for i in range(2):
    for j in range(2):
        ds2.append(((1 if i == j else 0) + sympy.diff(f, _coords[i]) * sympy.diff(f, _coords[j]))
                   * sympy.diffgeom.TensorProduct(_1forms[i],  _1forms[j]))
ds2 = sum(ds2)
g4 = metric.Metric(twoform=ds2)
g4
```

#### Compute Components

```python
g4.matrix
```

```python
sympy.det(g4.matrix)
```

```python
g4.inverse.matrix
```

```python
crs, rms, rcs = curvature.compute_components(g4)
```

```python
# Simplify notation
subscript_notation = [
    (sympy.diff(sympy.diff(f, x), x), sympy.symbols('f_xx')),
    (sympy.diff(sympy.diff(f, y), y), sympy.symbols('f_yy')),
    (sympy.diff(sympy.diff(f, x), y), sympy.symbols('f_xy')),
    (sympy.diff(f, x), sympy.symbols('f_x')),
    (sympy.diff(f, y), sympy.symbols('f_y')),
]
```

```python
crs = [(c[0], c[1].subs(subscript_notation)) for c in crs]
rms = [(c[0], c[1].subs(subscript_notation)) for c in rms]
rcs = [(c[0], c[1].subs(subscript_notation)) for c in rcs]
```

```python
curvature.display_components(crs)
```

```python
curvature.display_components(rms)
```

```python
curvature.display_components(rcs)
```

Clean Expr

```python
simplified = full_simplify(curvature.ricci_scalar(g4)).subs(subscript_notation)
simplified
```

#### Fun Examples

Paraboloid

```python
a, b = sympy.symbols('a b')
subs = {f: x**2/a + y**2/b}
```

```python
full_simplify(full_simplify(curvature.ricci_scalar(g4)).subs(subs))
```

### B5 - Embedded Surface - Sphere

#### Substitution

```python
a_0 = sympy.symbols('a_0')
subs = {f: sympy.sqrt(a_0**2 - x**2 - y**2)}
g5 = metric.Metric(twoform=g4.twoform.subs(subs))
g5
```

#### Curvature

```python
full_simplify(curvature.ricci_scalar(g5, simplify_intermediate=True))
```

#### Parallel Transport

```python
param = sympy.symbols('\\lambda')

curve = [
    sympy.cos(2 * sympy.pi * param),
    sympy.sin(2 * sympy.pi * param),
]
```

```python
full_simplify(geodesic.parallel_transport_equation(0, curve, param, g5))
```

```python
full_simplify(geodesic.parallel_transport_equation(1, curve, param, g5))
```

#### Geodesic

##### Geodesic Equations

```python
full_simplify(geodesic.geodesic_equation(0, sympy.symbols('lambda'), g5))
```

```python
full_simplify(geodesic.geodesic_equation(1, sympy.symbols('lambda'), g5))
```

##### Visual

```python
a_0_val = 1.0
g5_num = metric.Metric(twoform=g5.twoform.subs({a_0: a_0_val}))
```

```python
ls = numpy.arange(0.0, 8.0, 0.001)
df5 = geodesic.numerical_sampler(g5_num, ls, (0.5, 0.5), tangent_scale=0.1)
```

```python
lim_df5 = boundary_filter(df5, x=(-1, 1), y=(-1, 1))
```

```python
if show_plots:
    fig = px.scatter(lim_df5, x="x", y='y',
                     color='theta_0',
                     title="B5 Geodesic",
                     height=600, width=600)
    fig.show()
```

### B6 - Embedded Surface - Hyperboloid

#### Substitution

```python
a_0 = sympy.symbols('a_0')
subs = {f: sympy.sqrt(a_0**2 - x**2 + y**2)}
g6 = metric.Metric(twoform=g4.twoform.subs(subs))
g6
```

```python
g6.matrix.doit()
```

#### Curvature

```python
full_simplify(curvature.ricci_scalar(g6, simplify_intermediate=True))
```

#### Parallel Transport

```python
full_simplify(geodesic.parallel_transport_equation(0, curve, param, g6))
```

```python
full_simplify(geodesic.parallel_transport_equation(0, curve, param, g6))
```

#### Geodesic

```python
a_0_val = 1.0
g6_num = metric.Metric(twoform=g6.twoform.subs({a_0: a_0_val}))
```

```python
ls = numpy.arange(0.0, 8.0, 0.001)
df6 = geodesic.numerical_sampler(g6_num, ls, (0.5, 0.5), tangent_scale=0.1)
```

```python
lim_df6 = boundary_filter(df6, x=(-1, 1), y=(-1, 1))
```

```python
if show_plots:
    fig = px.scatter(lim_df6, x="x", y='y',
                     color='theta_0',
                     title="B6 Geodesic",
                     height=600, width=600)

    fig.show()
```

### B7 - Embedded Surface - Cylinder

#### Substitution

```python
a_0 = sympy.symbols('a_0')
subs = {f: sympy.sqrt(a_0**2 - y**2)}
g7 = metric.Metric(twoform=g4.twoform.subs(subs))
g7
```

```python
g7.matrix.doit()
```

#### Curvature

```python
crs, rmn, rcs = curvature.compute_components(g7)
```

```python
curvature.display_components(crs)
```

```python
full_simplify(curvature.ricci_scalar(g7))
```

#### Parallel Transport

```python
full_simplify(geodesic.parallel_transport_equation(0, curve, param, g7))
```

```python
full_simplify(geodesic.parallel_transport_equation(1, curve, param, g7))
```

#### Geodesic

```python
a_0_val = 1.0
g7_num = metric.Metric(twoform=g7.twoform.subs({a_0: a_0_val}))
```

```python
ls = numpy.arange(0.0, 8.0, 0.001)
df7 = geodesic.numerical_sampler(g7_num, ls, (0.5, 0.5), tangent_scale=0.1)
```

```python
lim_df7 = boundary_filter(df7, x=(-1, 1), y=(-1, 1))
```

```python
if show_plots:
    fig = px.scatter(lim_df7, x="x", y='y',
                     color='theta_0',
                     title="B7 Geodesic",
                     height=600, width=600)
    fig.show()
```

### B8 - Embedded Surface - Cone

#### Substitution

```python
a_0 = sympy.symbols('a_0')
subs = {f: sympy.sqrt(a_0 * (x**2 + y**2))}
g8 = metric.Metric(twoform=g4.twoform.subs(subs).doit())
g8
```

```python
full_simplify(g8.matrix)
```

#### Curvature

```python
crs, rmn, rcs = curvature.compute_components(g8)
```

```python
curvature.display_components(crs)
```

```python

```

```python
full_simplify(curvature.ricci_scalar(g8))
```

#### Parallel Transport

```python
full_simplify(geodesic.parallel_transport_equation(0, curve, param, g8))
```

```python
full_simplify(geodesic.parallel_transport_equation(1, curve, param, g8))
```

#### Geodesic

```python
a_0_val = 1.0
g8_num = metric.Metric(twoform=g8.twoform.subs({a_0: a_0_val}))
```

```python
ls = numpy.arange(0.0, 8.0, 0.001)
df8 = geodesic.numerical_sampler(g8_num, ls, (0.5, 0.5), tangent_scale=0.1)
```

```python
lim_df8 = boundary_filter(df8, x=(-1, 1), y=(-1, 1))
```

```python
if show_plots:
    fig = px.scatter(lim_df8, x="x", y='y',
                     color='theta_0',
                     title="B8 Geodesic",
                     height=600, width=600)

    fig.show()
```

## Applications

### C1 - Lower bound on size of universe

```python
M = Manifold('M', dim=3)
P = Patch('origin', M)

r, theta, phi = sympy.symbols('r theta phi', nonnegative=True)
a_0 = sympy.symbols('a_0', real=True, nonnegative=True)

cs = coords.CoordSystem('spherical', P, [r, theta, phi])
dr, dtheta, dphi = cs.base_oneforms()

ds2 = a_0 ** 2 * ((1 / (1 - r ** 2)) * tpow(dr, 2) + r ** 2 * (tpow(dtheta, 2) +
                                                               sympy.sin(theta) ** 2 * tpow(dphi, 2)))

gc1 = metric.Metric(twoform=ds2)
gc1
```

```python
gc1.matrix
```

```python
R_0 = 1.7e-54 * (1 / units.meter) ** 2
```

```python
a0_val = numpy.sqrt(6 / R_0)
```

```python
V_bound = 2 * numpy.pi ** 2 * (a0_val) ** (3)
```

```python
V_bound
```

### C2 - Radial Geodesic

$$ L = \int_{\gamma} \sqrt{g_{\mu\nu}\frac{d x^{\mu}}{d\lambda}\frac{d x^{\nu}}{d\lambda}} d \lambda$$

Assume that radial geodesic implies that $$\frac{d}{d\lambda}\theta = \frac{d}{d\lambda}\phi = 0$$

$$ L_0 = \int_{\gamma} \sqrt{g_{rr}\left(\frac{d r}{d\lambda}\right)^2} d \lambda = \int_{0}^{r_0} \sqrt{g_{rr}}d r $$

```python
r_0 = sympy.symbols('r_0', real=True, nonnegative=True)
```

```python
soln = sympy.integrate(sympy.sqrt(gc1.matrix[0,0]), (r, 0, r_0))
soln
```

```python
soln.args[0][0]
```

$$ r_0 = \sin\left(\frac{L_0}{a_0}\right)$$

```python
L_0 = (46e9 * units.lightyear).to(units.meter)
L_0
```

```python
a0_val
```

```python
L_0 / a0_val
```

```python
r0_val = numpy.sin((L_0/a0_val).value)
r0_val
```

```python
v_obs = 4 / 3 * numpy.pi * r0_val ** 3
v_obs_scaled = v_obs * (a0_val) ** 3
v_obs_scaled
```

```python
v_obs_scaled / V_bound
```


```python
# import libraries and modules
import sympy as sm

from sympy.stats import Normal, density, E
```

```python
# define symbols used to create distributions
x, mean1, std1, mean2, std2 = sm.symbols('x mean1 std1 mean2 std2')
```

```python
# create univariate gaussian p(x)
p = Normal('p', mean1, std1)
p
```

$\displaystyle p$

```python
# find probability density function of p(x)
dp = density(p)(x)
dp
```

$\displaystyle \frac{\sqrt{2} e^{- \frac{\left(- mean_{1} + x\right)^{2}}{2 std_{1}^{2}}}}{2 \sqrt{\pi} std_{1}}$

```python
# create univariate gaussian q(x)
q = Normal('q', mean2, std2)
q
```

$\displaystyle q$

```python
# find probability density function of q(x)
dq = density(q)(x)
dq
```

$\displaystyle \frac{\sqrt{2} e^{- \frac{\left(- mean_{2} + x\right)^{2}}{2 std_{2}^{2}}}}{2 \sqrt{\pi} std_{2}}$

```python
# calculate KL Divergence between p(x) and q(x)
kl = E(sm.log(dp/dq))
kl
```

$\displaystyle \log{\left(\frac{std_{2} e^{- \frac{\left(- mean_{1} + x\right)^{2}}{2 std_{1}^{2}}} e^{\frac{\left(- mean_{2} + x\right)^{2}}{2 std_{2}^{2}}}}{std_{1}} \right)}$

```python
# make the formula more presentable
sm.expand(kl, force=True).subs({x:mean1, x**2:mean1**2+std1**2}).expand().collect(std2)
```

$\displaystyle - \log{\left(std_{1} \right)} + \log{\left(std_{2} \right)} - \frac{1}{2} + \frac{\frac{mean_{1}^{2}}{2} - mean_{1} mean_{2} + \frac{mean_{2}^{2}}{2} + \frac{std_{1}^{2}}{2}}{std_{2}^{2}}$


## Street Fight Mathematics

Street fight Mathematics is a collection of techniques to make 'guesstimations' of results, solving problems without doing a mathematical proof or finding analytical or exact solutions.

---

Question: What is the next number in the series?

    1, 5, 12, 22, 35, 51, 70, 92, ?

Well, there might be many answers, but the best place to quickly find solutions is The On-Line Encyclopedia of Integer Sequences ([OEIS](https://oeis.org/)).

The site will return sequence [A000326](https://oeis.org/A000326): Pentagonal Numbers $a(n) = n*(3n-1)/2$.

```
def A000326(n):
   return [int(i*(3*i-1)/2) for i in range(1,n)]

A000326(10)
```

---

Question: What are Stirling numbers of second kind?

That's easy: wikipedia,

```
from IPython.display import IFrame

IFrame(src='https://en.wikipedia.org/wiki/Stirling_numbers_of_the_second_kind', width=800, height=400)
```

or Wolfram World,

```
from IPython.display import IFrame

IFrame(src='https://mathworld.wolfram.com/StirlingNumberoftheSecondKind.html', width=1000, height=400)
```

---

Question: What is $0.601907230197$? This is a typical problem after a simulation outputs a strange number like this.

To answer use the [inverse symbolic calculator](http://wayback.cecm.sfu.ca/projects/ISC/ISCmain.html).

In this example, the number is the value BesselK(1,1).

Of course, that leads to another question: what is the BesselK function? Well, check Wikipedia or Wolfram World!

---

Question: What is the Taylor series for $\arcsin(x)$?

There are powerful mathematical software applications, like Mathematica and Maple that are able to perform symbolic computations. There is also Matlab for numerical computations. [Wolfram Alpha](https://www.wolframalpha.com/) is an online tool based on Mathematica.

Python also has very good libraries to perform math and scientific computations. You should learn to explore the APIs of `numpy`, `scipy` and `sympy`.

```
from sympy import series, asin
from sympy.abc import x

series(asin(x), x0=0, n=10)
```

Searching the internet, we find an exact answer

$$\arcsin x = \displaystyle x + \frac 1 2 \frac {x^3} 3 + \frac {1 \times 3} {2 \times 4} \frac {x^5} 5 + \frac {1 \times 3 \times 5} {2 \times 4 \times 6} \frac {x^7} 7 + \cdots$$

---

Another useful website is [StackExchange](https://stackexchange.com/). It includes many expert [communities](https://stackexchange.com/sites#), some related to CompSci, Stats and Math:

+ [Stack Overflow](https://stackoverflow.com/) - computer science and programming related questions

+ [Mathematics](https://math.stackexchange.com/) - math related questions

+ [Cross Validated](https://stats.stackexchange.com/) - statistics related questions

---

Let's check a more complex example.

Suppose $p(x)$ is a polynomial of degree $\leq n$ bounded in $[-1,+1]$ for $x=[-1,+1]$. What is the largest $p'(0)$ possible?

Without knowing enough Math and not knowing how to google for the answer, can we apply Street Fight Mathematics?

Let's start by trying to find a solution with a small $n$, say $n=3$.

So, $p(x) = a + bx + cx^2 + dx^3$

If we select a valid value for $x$, say $x=0.2$, we have a constraint to satisfy:

$$-1 \leq a + 0.2b + 0.04c + 0.008d \leq +1$$

Since the domain of $x$ is less than $1$, we would like to maximize $b$ (the other variables contribute much less).

So what? Well, if we collect 10000 random values for $x$, we will have 20000 constraints. The result of maximizing $b$ for all those contraints will, surely, get us very close to its optimal value.

```
import numpy as np

np.random.seed(123)

size = 10000
xs = np.random.uniform(-1,1,size)  # generate random values for x in [-1,+1]
xs = xs.reshape(size,1)            # make it a vector

def polyCoeff(x):
  return np.array([np.array([1]), x, x**2, x**3])

# A is a matrix of dimension 10000x4 defining the constraints' values
A = np.apply_along_axis(polyCoeff, 1, xs).reshape(size,4)
b = np.ones(size)
```

This is a linear programming problem, so let's CVX it!

```
from cvxpy import *

coeffs = Variable(4) # for n=3 there are 4 coefficients

objective   = Maximize( coeffs[1] )
constraints = [A @ coeffs >= -b, A @ coeffs <= b]
problem     = Problem(objective, constraints)
```

And the answer is:

```
problem.solve()

print("Maximal b value:", problem.value)
print("Coefficients values:", np.round(coeffs.value,2))
```

    Maximal b value: 3.0005335738238035
    Coefficients values: [ 0.  3. -0. -4.]

So, the answer for $b$ is almost $3$. Probably it is $3$, remember that we are just estimating the true value.

The proposed cubic polynomial is $p(x) = 3x - 4x^3$

```
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(8,4))

axes = fig.add_axes([0.1, 0.1, 0.9, 0.9])
axes.set_xlabel('$x$')
axes.set_ylabel('$p(x)$')
axes.set_title('Optimal Polynomial $p(x) = 3x - 4x^3$');

xs = np.linspace(-1,+1,100)
p  = np.poly1d([-4,0,3,0])
_ = axes.plot(xs, p(xs))
```

Ok, we are on! Let's solve the problem for $n=1,2,3,\ldots$

```
## put all previous code in a function
def findBestPoly(n):
  size = 10000
  xs = np.random.uniform(-1,1,size)  # generate random values for x in [-1,+1]
  xs = xs.reshape(size,1)            # make it a vector

  def polyCoeff(x):
    one = np.array([[1]])
    cs  = np.array([x**i for i in range(1,n+1)])
    return np.concatenate((one, cs))

  A = np.apply_along_axis(polyCoeff, 1, xs).reshape(size,n+1)
  b = np.ones(size)

  coeffs = Variable(n+1) # for n=3 there are 4 coefficients

  objective   = Maximize( coeffs[1] )
  constraints = [A @ coeffs >= -b, A @ coeffs <= b]
  problem     = Problem(objective, constraints)

  problem.solve()
  return(coeffs.value)

## iterate
for n in range(1,10):
   print("n =", n, " --> ", np.round(findBestPoly(n)))
```

The polynomial for $2k$ is the same as $2k-1$ (with an exception for $n=2$), so let's skip repetitions.

The best polynomials are

$$\begin{array}{lcl}
  p_1(x) &=& 1x \\
  p_3(x) &=& 3x - 4x^3 \\
  p_5(x) &=& 5x - 20x^3 + 16x^3 \\
  p_7(x) &=& 7x - 56x^3 + 112x^5 - 64x^7 \\
  p_9(x) &=& 9x -120x^3 + 432x^5 - 576x^7 + 256x^9
\end{array}$$

The largest $b$ for $p'(0)$ seems equal to the polynomial degree... But what about the other coefficients?

We can try inserting all coefficientes into OEIS...

```
from IPython.display import IFrame

IFrame(src='https://oeis.org/search?q=1%2C3%2C-4%2C5%2C-20%2C16%2C7%2C-56&sort=&language=english&go=Search', width=1000, height=400)
```

Chebyshev polynomials?

```
from IPython.display import IFrame

IFrame(src='https://en.wikipedia.org/wiki/Chebyshev_polynomials', width=1000, height=400)
```

In the wikipage there is this sentence:

> The Chebyshev polynomials Tn are polynomials with the largest possible leading coefficient, whose absolute value on the interval [−1, 1] is bounded by 1. They are also the "extremal" polynomials for many other properties

with a reference: Rivlin, Theodore J. (1974). "Chapter  2, Extremal properties". The Chebyshev Polynomials. Pure and Applied Mathematics, pp. 56–123

The mathematical connection was made. We could now advance in solving the initial problem analytically.

---

## Dimensional Analysis

In Computer Science it is usual to use types to check for operations compatibility. This technique was already known in Science for a long time:

> dimensional analysis is the analysis of the relationships between different physical quantities by identifying their base quantities (such as length, mass, time, and electric charge) and units of measure (such as miles vs. kilometres, or pounds vs. kilograms) and tracking these dimensions as calculations or comparisons are performed [...] Physical quantities that are of the same kind (also called commensurable) (e.g., length or time or mass) have the same dimension and can be directly compared to other physical quantities of the same kind [...] Any physically meaningful equation (and any inequality) will have the same dimensions on its left and right sides.

This is a useful technique to check comparisons like

National Debt > 120% of GDP

Do this equation makes physical sense?

No, since national debt is in Euros, and GDP is in Euros/Year.

Sometimes dimensional analysis can greatly ease our work by removing incompatible options.

Consider that we already know Newton's law of universal gravitation,

$$\hbox{force} = {Gm_1m_2\over r^2}$$

and would wish to find the equation that relates a planet's year $T$, given its orbit radius $r$ and mass of the Sun $M$ (assume here that the planet's mass is irrelevant).

The equation would have this abstract relationship,

$$T(r) \sim G^x r^y M^z$$

We know that the units of $G$ are $m^3 kg^{−1} s^{−2}$ (since force comes in $m~kg~s^{-2}$) and $T$ is given in seconds.

+ $x$ must be $-1/2$ so that the seconds are compatible, ie, $s^{{-2}^{1/2}} = s$

+ that makes the units of $G^{-1/2}$, $m^{-3/2}kg^{1/2} s$

+ since $r$ comes in meters, $y$ must be $3/2$ so that both exponents for distance cancel each other.

+ since $M$ comes in kilograms, $z$ must be $-1/2$ so that both exponents for mass cancel each other.

That is

$$T \sim \frac{r^{3/2}}{G^{1/2}M^{1/2}} = \sqrt{ \frac{r^3}{GM}}$$

We don't know exactly the equation because we don't know the constant of proportionality. But this is a sanity check for the formal proof.

Btw, the [real formula](https://en.wikipedia.org/wiki/Kepler%27s_laws_of_planetary_motion#Third_law) is

$$T \approx \sqrt{ \frac{4\pi^2 r^3}{GM}}$$

and is known as the third of Kepler's laws of planetary motion.

## References

+ [Street Fighting Mathematics](https://www.youtube.com/watch?v=qP4XEZ54eSc), Ryan O'Donnell (most of the notebook is the Python implementation of this lecture)

+ [Street Fighting Mathematics MOOC](https://www.edx.org/course/street-fighting-math)


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.6 (Python 3.7.3-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.6/

```python
a, b, g, d, e, o, k, l0 = symbols('a b g d e o k l0', real=True, positive=True)
```

```python
ksq=k**2
A, B, G, D, E, O = a**2, b**2, g**2, d**2, e**2, o**2
TA = 2*A
A2B = A*A + B
C = TA + (G+E)*A2B
f1 = ksq + TA*k + A2B
f2 = 1 + G*k + D*ksq
f3 = 1 + E*k
den = A2B + C*k + O*ksq
f = l0*f1*f2*f3/den
f
```

```python
diff(f,l0)
```

```python
da=diff(f,a)
da.factor()
```

```python
db=diff(f,b)
db.factor()
```

```python
dg=diff(f,g)
dg.factor()
```

```python
dd=diff(f,d)
dd.factor()
```

```python
de=diff(f,e)
de.factor()
```

```python
do=diff(f,o)
dd.factor()
```


# Solving Linear Systems: Iterative Methods

This notebook by Xiaozhou Li is licensed under a Creative Commons Attribution 4.0 International License.
All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

## General Form
For solving the linear system
$$
    Ax = b,
$$
with the exact solution $x^{*}$.  The general form based on the fixed point interation:
\begin{equation}
     \begin{split}
          x^{(0)} & = \text{initial guess} \\
          x^{(k+1)} & = g(x^{(k)}) \quad k = 0,1,2,\ldots,
     \end{split}
\end{equation}
where
$$
    g(x) = x - C(Ax - b).
$$
Difficult: find a matrix $C$ such that
$$
    \lim\limits_{k\rightarrow\infty}x^{(k)} = x^{*}
$$
and the algorithm needs to be converge fast and economy.

**Example 1**
\begin{equation*}
    A = \left[\begin{array}{ccc} 9& -1 & -1 \\ -1 & 10 & -1 \\ -1 & -1& 15\end{array}\right],\quad b = \left[\begin{array}{c} 7 \\ 8 \\ 13\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[1, 1, 1]}^T$

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt
plt.rcParams['axes.unicode_minus']=False
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from IPython.display import clear_output, display
```

```python
def IterC(A, b, C, x0, x_star, iters):
    x = np.copy(x0)
    print ('Iteration No.     Numerical Solution     Max norm error ')
    print (0, x, np.linalg.norm(x_star-x, np.inf))
    for i in range(iters):
        x = x + np.dot(C, b - np.dot(A,x))
        print (i+1, x, np.linalg.norm(x_star-x,np.inf))
```

```python
A = np.array([[9., -1., -1.],[-1.,10.,-1.],[-1.,-1.,15.]])
b = np.array([7.,8.,13.])
```

**Naive Choice**

Choosing $C = I$, then
$$g(x) = x - (Ax - b),$$
and the fixed-point iteration
$$x^{(k+1)}  = (I - A)x^{(k)} + b \quad k = 0,1,2,\ldots. $$
Let the intial guess $x_0 = [0, 0, 0]^T$.

```python
C = np.eye(3)
x0 = np.zeros(3)
x_star = np.array([1.,1.,1.])

w = interactive(IterC, A=fixed(A), b=fixed(b), C=fixed(C), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Best Choice (theoretically)**

Choosing $C = A^{-1}$, then
$$g(x) = x - A^{-1}(Ax - b),$$
and the fixed-point iteration
$$x^{(k+1)}  = A^{-1}b \quad k = 0,1,2,\ldots. $$

* It equals to solve $Ax = b$ directly.
* However, it gives a hint that $C$ should be close to $A^{-1}$

## Some Classic Iterative Methods

Let $D$ denote the main diagonal of $A$, $L$ denote the lower triangle of $A$ (entries below the main diagonal), and $U$ denote the upper triangle (entries above the main diagonal). Then, we have $A = L + D + U$
\begin{equation}
A = \begin{pmatrix}{a_{11}} & {a_{12}} & {\cdots} & {a_{1 n}} \\ {a_{21}} & {a_{22}} & {\cdots} & {a_{2 n}} \\ {\vdots} & {\vdots} & {} & {\vdots} \\ {a_{n 1}} & {a_{n 2}} & {\cdots} & {a_{n n}}\end{pmatrix},\,\,D = \begin{pmatrix}{a_{11}} & {} & {} & {} \\ {} & {a_{22}} & {} & {} \\ {} & {} & {\ddots} & {} \\ {} & {} & {} & {a_{n n}}\end{pmatrix},\,\, L= \begin{pmatrix}{0} & {} & {} & {} \\ {a_{21}} & {\ddots} & {} & {} \\ {\vdots} & {\ddots} & {0} & {} \\ {a_{n 1}} & {\cdots} & {a_{n, n-1}} & {0}\end{pmatrix},\,\, U= \begin{pmatrix}{0} & {a_{12}} & {\cdots} & {a_{1n}} \\ {} & {0} & {\ddots} & {\vdots} \\ {} & {} & {\ddots} & {a_{n-1,n}} \\ {} & {} & {} & {0}\end{pmatrix}
\end{equation}

**First Approach**

Choosing $C = \text{diag}(A)^{-1} = D^{-1}$, then
$$g(x) = x - D^{-1}(Ax - b),$$
and the fixed-point iteration
$$Dx^{(k+1)}  = -(L + U)x^{(k)} + b \quad k = 0,1,2,\ldots. $$

```python
C = np.diag(1./np.diag(A))
x0 = np.zeros(np.size(b))
x_star = np.array([1.,1.,1.])
```

```python
w = interactive(IterC, A=fixed(A), b=fixed(b), C=fixed(C), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

## Jacobi Method
### Matrix Form:
$$
    x^{(k+1)} = x^{(k)} - D^{-1}(Ax^{(k)} - b)
$$
or
$$
    Dx^{(k+1)} = b - (L+U)x^{(k)}
$$
### Algorithm
$$
    x^{(k+1)}_i = \frac{b_i - \sum\limits_{j  i}a_{ij}x^{(k)}_j}{a_{ii}}
$$

```python
def Jacobi(A, b, x0, x_star, iters):
    x_old = np.copy(x0)
    x_new = np.zeros(np.size(x0))
    print (0, x_old, np.linalg.norm(x_star-x_old,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x_new[i] = (b[i] - np.dot(A[i,:i],x_old[:i]) - np.dot(A[i,i+1:],x_old[i+1:]))/A[i,i]
        print (k+1, x_new, np.linalg.norm(x_star-x_new,np.inf))
        x_old = np.copy(x_new)
```

```python
w = interactive(Jacobi, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Second Approach**

Let $D$ denote the main diagonal of $A$, $L$ denote the lower triangle of $A$ (entries below the main diagonal), and $U$ denote the upper triangle (entries above the main diagonal). Then $A = L + D + U$

Choosing $C = (L + D)^{-1}$, then
$$g(x) = x - (L + D)^{-1}(Ax - b),$$
and the fixed-point iteration
$$(L + D)x^{(k+1)}  = Ux^{(k)} + b \quad k = 0,1,2,\ldots. $$

```python
def GS(A, b, x0, x_star, iters):
    x = np.copy(x0)
    print (0, x, np.linalg.norm(x_star-x,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = (b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i+1:],x[i+1:]))/A[i,i]
        print (k+1, x, np.linalg.norm(x_star-x,np.inf))
```

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

## Gauss-Seidel Method
### Algorithm
$$
    x^{(k+1)}_i = \frac{b_i - \sum\limits_{j  i}a_{ij}x^{(k)}_j}{a_{ii}}
$$
### Matrix Form:
$$
    x^{(k+1)} = x^{(k)} - (L+D)^{-1}(Ax^{(k)} - b)
$$
or
$$
    (L+D)x^{(k+1)} = b - Ux^{(k)}
$$

**Example 2**
\begin{equation*}
    A = \left[\begin{array}{ccc} 3& 1 & -1 \\ 2 & 4 & 1 \\ -1 & 2& 5\end{array}\right],\quad b = \left[\begin{array}{c} 4 \\ 1 \\ 1\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[2, -1, 1]}^T$

```python
A = np.array([[3, 1, -1],[2,4,1],[-1,2,5]])
b = np.array([4,1,1])

x0 = np.zeros(np.size(b))
x_star = np.array([2.,-1.,1.])
```

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=40,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=40), Output()), _dom_classes=('widget-intera…

**Example 3**
\begin{equation*}
    A = \left[\begin{array}{ccc} 1& 2 & -2 \\ 1 & 1 & 1 \\ 2 & 2& 1\end{array}\right],\quad b = \left[\begin{array}{c} 7 \\ 8 \\ 13\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[-3, 8, 3]}^T$

```python
A = np.array([[1, 2, -2],[1,1,1],[2,2,1]])
b = np.array([7,8,13])

#x0 = np.zeros(np.size(b))
x0 = np.array([-1,1,1])
x_star = np.array([-3.,8.,3.])

w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
B = np.eye(3) - np.dot(np.diag(1./np.diag(A)),A)
print(B)
print (np.linalg.eig(B))
```

    [[ 0. -2.  2.]
     [-1.  0. -1.]
     [-2. -2.  0.]]
    (array([-6.16610918e-06+1.06799587e-05j, -6.16610918e-06-1.06799587e-05j,
            1.23322184e-05+0.00000000e+00j]), array([[-0.57734849-6.16605800e-06j, -0.57734849+6.16605800e-06j,
             0.57735383+0.00000000e+00j],
           [ 0.57735027+3.08300999e-06j,  0.57735027-3.08300999e-06j,
            -0.57735027+0.00000000e+00j],
           [ 0.57735205+0.00000000e+00j,  0.57735205-0.00000000e+00j,
            -0.57734671+0.00000000e+00j]]))

**Example 4**
\begin{equation*}
    A = \left[\begin{array}{cc} 1& 2 \\ 3 & 1 \end{array}\right],\quad b = \left[\begin{array}{c} 5 \\ 5\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[1, 2]}^T$

or
\begin{equation*}
    A = \left[\begin{array}{cc} 3& 1 \\ 1 & 2 \end{array}\right],\quad b = \left[\begin{array}{c} 5 \\ 5\end{array}\right],
\end{equation*}

```python
#A = np.array([[1, 2],[3,1]])
A = np.array([[3, 1],[1,2]])
b = np.array([5,5])

#x0 = np.zeros(np.size(b))
x0 = np.array([0,0])
x_star = np.array([1.,2.,])

w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Example 5**
Are Jacobi iteration and Gauss-Seidel iteration convergent for the following equations?
\begin{equation*}
    A_1 = \left[\begin{array}{ccc} 3& 0 & 4 \\ 7 & 4 & 2 \\ -1 & 1 & 2\end{array}\right],\quad A_2 = \left[\begin{array}{ccc} -3& 3 & -6 \\ -4 & 7 & -8 \\ 5 & 7 & -9\end{array}\right],
\end{equation*}

* Consider the **spectral radius** of the iterative matrix
* $B_J = -D^{-1}(L+U)$ and $B_{GS} = -(L+D)^{-1}U$

```python
def Is_Jacobi_Gauss(A):
    L = np.tril(A,-1)
    U = np.triu(A,1)
    D = np.diag(np.diag(A))

    B_J = np.dot(np.diag(1./np.diag(A)), L+U)
    B_GS = np.dot(np.linalg.inv(L+D),U)

    rho_J = np.linalg.norm(np.linalg.eigvals(B_J), np.inf)
    rho_GS = np.linalg.norm(np.linalg.eigvals(B_GS), np.inf)

    print ("Spectral Radius")
    print ("Jacobi: ", rho_J)
    print ("Gauss Sediel: ", rho_GS)

A1 = np.array([[3, 0, 4],[7, 4, 2], [-1,1,2]])
A2 = np.array([[-3, 3, -6], [-4, 7, -8], [5, 7, -9]])
Is_Jacobi_Gauss(A2)
```

    Spectral Radius
    Jacobi:  0.8133091054692768
    Gauss Sediel:  1.1111111111111105

## Successive Over-Relaxation (SOR)
### Algorithm
$$
    x^{(k+1)}_i = x^{(k)} + \omega \frac{b_i - \sum\limits_{j < i}a_{ij}x^{(k+1)}_j - \sum\limits_{j \geq i}a_{ij}x^{(k)}_j}{a_{ii}}
$$
### Matrix Form:
$$
    x^{(k+1)} = x^{(k)} - \omega(\omega L+D)^{-1}(Ax^{(k)} - b)
$$
or
$$
    (\omega L+D)x^{(k+1)} = ((1-\omega)D - \omega U)x^{(k)} + \omega b
$$

```python
def SOR(A, b, x0, x_star, omega, iters):
    x = np.copy(x0)
    print (0, x, np.linalg.norm(x_star-x,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
        print (k+1, x, np.linalg.norm(x_star-x,np.inf))
```

```python
def SOR2(A, b, x0, x_star, omega, iters):
    x = np.copy(x0)
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
    return (np.linalg.norm(x_star-x,np.inf))
```

```python
def SOR3(A, b, x0, x_star, omega, iters):
    x = np.copy(x0)
    print (0, np.linalg.norm(x_star-x,np.inf))
    for k in range(iters):
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
        print (k+1, np.linalg.norm(x_star-x,np.inf))
```

```python
A = np.array([[9., -1., -1.],[-1.,10.,-1.],[-1.,-1.,15.]])
b = np.array([7.,8.,13.])
```

```python
x0 = np.array([0.,0.,0.])
x_star = np.array([1.,1.,1.])
omega = 1.01

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

**Example 6**
\begin{equation*}
    A = \left[\begin{array}{ccc} 2& -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1& 2\end{array}\right],\quad b = \left[\begin{array}{c} 1 \\ 0 \\ 1.8\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[1.2, 1.4, 1.6]}^T$

```python
A = np.array([[2, -1, 0],[-1, 2, -1], [0, -1, 2]])
b = np.array([1., 0, 1.8])

x0 = np.array([1.,1.,1.])
x_star = np.array([1.2,1.4,1.6])
omega = 1.2

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
num = 21
omega = np.linspace(0.8, 1.8, num)
err1 = np.zeros(num)
for i in range(num):
    err1[i] = SOR2(A, b, x0, x_star, omega[i], 10)
print (err1)
plt.plot(omega, np.log10(err1), 'o')
```

**Example 7**
\begin{equation*}
    A = \left[\begin{array}{cccc} -4& 1 & 1 & 1 \\ 1 & -4 & 1 & 1 \\ 1 & 1& -4 &1 \\ 1 & 1 &1 & -4\end{array}\right],\quad b = \left[\begin{array}{c} 1 \\ 1 \\ 1 \\ 1\end{array}\right],
\end{equation*}
has the exact solution $x^{*} = {[-1, -1, -1, -1]}^T$

```python
A = np.array([[-4, 1, 1, 1],[1, -4, 1, 1], [1, 1, -4, 1], [1, 1, 1, -4]])
b = np.array([1, 1, 1, 1])

x0 = np.zeros(np.size(b))
x_star = np.array([-1,-1,-1,-1])
omega = 1.25

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
w = interactive(GS, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=100,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters'), Output()), _dom_classes=('widget-interact',))

```python
num = 21
omega = np.linspace(0.8, 1.8, num)
err1 = np.zeros(num)
for i in range(num):
    err1[i] = SOR2(A, b, x0, x_star, omega[i], 10)
print (err1)
plt.plot(omega, np.log10(err1), 'o')
```

**Example 8**
\begin{equation*}
    A=\begin{pmatrix}{3} & {-1} & {0} & 0 & 0 & \frac{1}{2}  \\ {-1} & {3} & {-1} & {0} & \frac{1}{2} & 0\\ {0} & {-1} & {3} & {-1} & {0} & 0 \\ 0& {0} & {-1} & {3} & {-1} & {0} \\ {0} & \frac{1}{2} & {0} & {-1} & {3} & {-1} \\ \frac{1}{2} & {0} & 0 & 0 & {-1} & {3}\end{pmatrix},\,\,b=\begin{pmatrix}\frac{5}{2} \\ \frac{3}{2} \\ 1 \\ 1 \\ \frac{3}{2}    \\ \frac{5}{2}  \end{pmatrix}
\end{equation*}
has the exact solution $x^{*} = {[1, 1, 1, 1, 1, 1]}^T$

```python
n0 = 6
A = 3*np.eye(n0) - np.diag(np.ones(n0-1),-1) - np.diag(np.ones(n0-1),+1)
for i in range(n0):
    if (abs(n0-1 - 2*i) > 1):
        A[i, n0-1-i] = - 1/2
print (A)
x_star = np.ones(n0)
b = np.dot(A, x_star)

x0 = np.zeros(np.size(b))
omega = 1.25

w = interactive(SOR, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=20,value=0))
display(w)
```

    [[ 3.  -1.   0.   0.   0.  -0.5]
     [-1.   3.  -1.   0.  -0.5  0. ]
     [ 0.  -1.   3.  -1.   0.   0. ]
     [ 0.   0.  -1.   3.  -1.   0. ]
     [ 0.  -0.5  0.  -1.   3.  -1. ]
     [-0.5  0.   0.   0.  -1.   3. ]]

    interactive(children=(IntSlider(value=0, description='iters', max=20), Output()), _dom_classes=('widget-intera…

```python
num = 21
omega = np.linspace(0.8, 1.8, num)
err1 = np.zeros(num)
for i in range(num):
    err1[i] = SOR2(A, b, x0, x_star, omega[i], 10)
print (err1)
plt.plot(omega, np.log10(err1), 'o')
```

```python
w = interactive(Jacobi, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), iters=widgets.IntSlider(min=0,max=100,value=0))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters'), Output()), _dom_classes=('widget-interact',))

## Sparse Matrix Computations
A coefficient matrix is called sparse if many of the matrix entries are known to be zero. Often, of the $n^2$ eligible entries in a sparse matrix, only $\mathcal{O}(n)$ of them are nonzero. A full matrix is the opposite, where few entries may be assumed to be zero.

**Example 9**
Consider the $n$-equation version of
\begin{equation*}
    A=\begin{pmatrix}{3} & {-1} & {0} & 0 & 0 & \frac{1}{2}  \\ {-1} & {3} & {-1} & {0} & \frac{1}{2} & 0\\ {0} & {-1} & {3} & {-1} & {0} & 0 \\ 0& {0} & {-1} & {3} & {-1} & {0} \\ {0} & \frac{1}{2} & {0} & {-1} & {3} & {-1} \\ \frac{1}{2} & {0} & 0 & 0 & {-1} & {3}\end{pmatrix},
\end{equation*}
has the exact solution $x^{*} = {[1, 1,\ldots, 1]}^T$ and $b = A x^{*}$

* First, let us have a look about the matrix $A$

```python
n0 = 10
A = 3*np.eye(n0) - np.diag(np.ones(n0-1),-1) - np.diag(np.ones(n0-1),+1)
for i in range(n0):
    if (abs(n0-1 - 2*i) > 1):
        A[i, n0-1-i] = - 1/2
plt.spy(A)
#plt.show()
```

* How about the $PA = LU$ for the above matrix $A$?
* Are the $L$ and $U$ matrices still sparse?

```python
import scipy.linalg
#P, L, U = scipy.linalg.lu(A)

#plt.spy(L)
#plt.show()
```

Gaussian elimination applied to a sparse matrix usually causes **fill-in**, where the coefficient matrix changes from sparse to full due to the necessary row operations. For this reason, the efficiency of Gaussian elimination and its $PA = LU$ implementation become questionable for sparse matrices, leaving iterative methods as a feasible alternative.

* Let us solve it with SOR method

```python
x_star = np.ones(n0)
b = np.dot(A, x_star)

x0 = np.zeros(np.size(b))
omega = 1.25
w = interactive(SOR3, A=fixed(A), b=fixed(b), x0=fixed(x0), x_star=fixed(x_star), omega=fixed(omega), iters=widgets.IntSlider(min=0,max=200,value=0, step=10))
display(w)
```

    interactive(children=(IntSlider(value=0, description='iters', max=200, step=10), Output()), _dom_classes=('wid…

## Application for Solving Laplace's Equation

### Laplace's equation
Consider the Laplace's equation given as
$$
    \nabla^2 u = 0,\quad\quad (x,y) \in D,
$$
where $\nabla^2 = \frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}$, and the boundary conditions are given as

### Finite Difference Approximation
Here, we use a rectangular grid $(x_i,y_j)$, where
$$
    x_i = i\Delta x, \,\,\text{for }\, i = 0,1,\ldots,N+1;\quad y_j = j\Delta y,\,\,\text{for }\, j = 0,1,\ldots,M+1.
$$
Five-points scheme:
$$
    -\lambda^2 u_{i+1,j} + 2(1+\lambda^2)u_{i,j} - \lambda^2u_{i-1,j} - u_{i,j+1} - u_{i,j-1} = 0,\quad\text{for}\,\, i = 1,\ldots,N,\,\, j = 1,\ldots,M，
$$
where $\lambda = \frac{\Delta y}{\Delta x}$. The boundary conditions are
- $x = 0: u_{0,j} = g_L(y_j), \quad\text{for }\, j = 1,\ldots,M$,
- $x = a: u_{N+1,j} = g_R(y_j), \quad\text{for }\, j = 1,\ldots,M$,
- $y = 0: u_{i,0} = g_B(x_i), \quad\text{for }\, i = 1,\ldots,N$,
- $y = b: u_{i,M+1} = g_T(x_i), \quad\text{for }\, i = 1,\ldots,N$.

```python
def generate_TD(N, dx, dy):
    T = np.zeros([N,N])
    a = - (dy/dx)**2
    b = 2*(1 - a)
    for i in range(N):
        T[i,i] += b
        if (i < N-1):
            T[i,i+1] += a
        if (i > 0):
            T[i,i-1] += a
    D = -np.identity(N)
    return T, D

def assemble_matrix_A(dx, dy, N, M):
    T, D = generate_TD(N, dx, dy)
    A = np.zeros([N*M, N*M])
    for j in range(M):
        A[j*N:(j+1)*N,j*N:(j+1)*N] += T
        if (j < M-1):
            A[j*N:(j+1)*N,(j+1)*N:(j+2)*N] += D
        if (j > 0):
            A[j*N:(j+1)*N,(j-1)*N:j*N] += D
    return A
```

```python
N = 4
M = 4
dx = 1./(N+1)
dy = 1./(M+1)
T, D = generate_TD(N, dx, dy)
#print (T)
A = assemble_matrix_A(dx, dy, N, M)
#print (A)
plt.spy(A)
plt.show()
```

```python
# Set boundary conditions
def gL(y):
    return 0.

def gR(y):
    return 0.

def gB(x):
    return 0.

def gT(x):
    return 1.
    #return x*(1-x)*(4./5-x)*np.exp(6*x)

def assemble_vector_b(x, y, dx, dy, N, M, gL, gR, gB, gT):
    b = np.zeros(N*M)
    # Left BCs
    for j in range(M):
        b[(j-1)*N] += (dy/dx)**2*gL(y[j+1])

    # Right BCs
    # b +=

    # Bottom BCs
    # b +=

    # Top BCs:
    for i in range(N):
        b[(M-1)*N+i] += gT(x[i+1])
    return b
```

```python
from mpl_toolkits import mplot3d
from mpl_toolkits.mplot3d import axes3d
```

```python
def Laplace_solver(a, b, N, M, gL, gR, gB, gT):
    dx = b/(M+1)
    dy = a/(N+1)
    x = np.linspace(0, a, N+2)
    y = np.linspace(0, b, M+2)

    A = assemble_matrix_A(dx, dy, N, M)
    b = assemble_vector_b(x, y, dx, dy, N, M, gL, gR, gB, gT)

    v = np.linalg.solve(A,b)

    # add boundary points + plotting
    u = np.zeros([(N+2),(M+2)])
    #u[1:(N+1),1:(M+1)] = np.reshape(v, (N, M))
    # Top BCs
    for i in range(N+2):
        u[i,M+1] = gT(x[i])
    u = np.transpose(u)
    u[1:(M+1),1:(N+1)] = np.reshape(v, (M, N))

    X, Y = np.meshgrid(x, y)
    #Z = np.sin(2*np.pi*X)*np.sin(2*np.pi*Y)

    fig = plt.figure()
    #ax = plt.axes(projection='3d')
    ax = fig.add_subplot(1, 1, 1, projection='3d')

    ax.plot_surface(X, Y, u, rstride=1, cstride=1,
                    cmap='viridis', edgecolor='none')
    ax.set_title('surface')
    plt.show()

Laplace_solver(1, 1, 40, 40, gL, gR, gB, gT)
```

```python
def Jacobi_tol(A, b, x0, tol):
    x_old = np.copy(x0)
    x_new = np.zeros(np.size(x0))
    for i in range(np.size(x0)):
        x_new[i] = (b[i] - np.dot(A[i,:i],x_old[:i]) - np.dot(A[i,i+1:],x_old[i+1:]))/A[i,i]
    iters = 1
    while ((np.linalg.norm(x_new-x_old,np.inf)) > tol):
        x_old = np.copy(x_new)
        for i in range(np.size(x0)):
            x_new[i] = (b[i] - np.dot(A[i,:i],x_old[:i]) - np.dot(A[i,i+1:],x_old[i+1:]))/A[i,i]
        iters += 1
    return x_new, iters
```

```python
def GS_tol(A, b, x0, tol):
    x_old = np.copy(x0)
    x = np.copy(x0)
    for i in range(np.size(x0)):
        x[i] = (b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i+1:],x[i+1:]))/A[i,i]
    iters = 1
    while ((np.linalg.norm(x-x_old,np.inf)) > tol):
        x_old = np.copy(x)
        for i in range(np.size(x0)):
            x[i] = (b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i+1:],x[i+1:]))/A[i,i]
        iters += 1
    return x, iters
```

```python
def SOR_tol(A, b, x0, omega, tol):
    x_old = np.copy(x0)
    x = np.copy(x0)
    for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
    iters = 1
    while ((np.linalg.norm(x-x_old,np.inf)) > tol):
        x_old = np.copy(x)
        for i in range(np.size(x0)):
            x[i] = x[i] + omega*(b[i] - np.dot(A[i,:i],x[:i]) - np.dot(A[i,i:],x[i:]))/A[i,i]
        iters += 1
    return x, iters
```

```python
def CG_tol(A, b, x0, x_star, tol):
    r_new = b - np.dot(A, x0)
    r_old = np.copy(np.size(x0))
    d_old = np.zeros(np.size(x0))
    x = np.copy(x0)
    iters = 0
    while ((np.linalg.norm(x-x_star,np.inf)) > tol):
        if (iters == 0):
            d_new = np.copy(r_new)
        else:
            beta = np.dot(r_new,r_new)/np.dot(r_old,r_old)
            d_new = r_new + beta*d_old
        Ad = np.dot(A, d_new)
        alpha = np.dot(r_new,r_new)/np.dot(d_new,Ad)
        x += alpha*d_new
        d_old = d_new
        r_old = r_new
        r_new = r_old - alpha*Ad
        iters += 1
    return x, iters
```

```python
def Iterative_solver(a, b, N, M, gL, gR, gB, gT, tol):
    dx = b/(M+1)
    dy = a/(N+1)
    x = np.linspace(0, a, N+2)
    y = np.linspace(0, b, M+2)

    A = assemble_matrix_A(dx, dy, N, M)
    b = assemble_vector_b(x, y, dx, dy, N, M, gL, gR, gB, gT)

    v = np.linalg.solve(A,b)
    #tol = 1.e-8
    v0 = np.zeros(np.size(b))
    #v_J, iters = Jacobi_tol(A, b, v0, tol)
    #print ("Jacobi Method:  %4d    %7.2e" %(iters, np.linalg.norm(v - v_J, np.inf)))

    #v_GS, iters = GS_tol(A, b, v0, tol)
    #print ("Gauss Seidel :  %4d    %7.2e" %(iters, np.linalg.norm(v - v_GS, np.inf)))

    omega = 2./(1 + np.sin(np.pi*dx))
    print ("omega = ", omega)
    v_SOR, iters = SOR_tol(A, b, v0, omega, tol)
    print ("SOR Method   :  %4d    %7.2e" %(iters, np.linalg.norm(v - v_SOR, np.inf)))

    v_CG, iters = CG_tol(A, b, v0, v, tol)
    print ("CG Method    :  %4d    %7.2e" %(iters, np.linalg.norm(v - v_CG, np.inf)))
```

```python
Iterative_solver(1, 1, 20, 20, gL, gR, gB, gT, 1.e-4)
```

    omega =  1.7405800107385732
    SOR Method   :    44    1.03e-04
    CG Method    :    37    8.38e-05

```python

```


# Probability

A probability distribution is a list of all of the possible outcomes of a random variable along with their corresponding probability values.

## Random Variable

A random variable is any variable whose outcome is a random event. It can be discrete (e.g., outcome of a coin toss or the roll of a die) as well as continuous (e.g., the heights of students in the class or the time needed by athletes to complete a 100 m race).

```python
from scipy.stats import binom
from scipy.stats import norm
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
np.random.seed(10)

plt.style.use('seaborn')

fig, axes = plt.subplots(1, 2)
fig.suptitle('Examples of discrete and continouns probability distributions')

data = binom.rvs(n=6, p=0.4, size=1000)
sns.histplot(data, ax=axes[0]);
axes[0].set_title('Discrete probability distribution')

x = np.linspace(-10, 10, 200)
y = norm.pdf(x, 0, 4)
sns.lineplot(x=x, y=y, ax=axes[1]);
axes[1].set_title('Continuous probability distribution');
```

## Discrete Random Variables

When a probability function is used to describe a discrete random variable (denoted by $X$), we call this function as probability mass function (PMF). Formally, a PMF is written as:

$$
\begin{equation}
f(x) = p(X=x)
\end{equation}
$$

$p(x)$ is called as a probability mass function if it satisfies the following conditions:
1. $p(x) > 0$
1. $p(x) <= 1$
1. $\sum_{x} p(x) = 1$

In the example of an unbiased dice roll, the probability of landing the number 2 (denoted by $x$) is represented as $p(X=x) = \frac{1}{6}$. The PMF just returns the probability of the outcome at a given point $x$.

## Continuous Random Variables

When a probability function is used to describe a continuous random variable (denoted by $X$), we call this function as probability distribution function (PDF). It is meaningless to represent a probability with any discrete value. To get the probability from a probability density function we need to find the area under the curve. We can then find the probability that the ourcome lies within an interval by calculating the area under the curve. Formally, an interval is used to represent the probability within that interval $X \in (x, x + \delta x)$. $p(x)\delta x$ is the probability that $X \in (x, x+\delta x)$ as $\delta x \rightarrow 0$. The probability density function satisfies the following criteria:

1. $p(x) \ge 0$
1. $\int_{-\infty}^{+\infty}p(x)dx = 1$
1. $\int_a^b p(x)dx = P(a \lt X \lt b)$

```python
from scipy import stats

fig, axes = plt.subplots(1, 2)
fig.suptitle('Examples of discrete and continouns probability distributions')

xk = np.arange(0, 7)
pk = (0.1, 0.2, 0.3, 0.1, 0.1, 0.0, 0.2)
custm = stats.rv_discrete(name='custm', values=(xk, pk))
data = custm.pmf(xk)
plt.subplot(1, 2, 1)
plt.title('Discrete probability distribution')
plt.stem(data)

x = np.linspace(-10, 10, 200)
y = norm.pdf(x, 0, 4)
sns.lineplot(x=x, y=y, ax=axes[1]);
axes[1].fill_between(x[150:170], y[150:170], color='cornflowerblue')
axes[1].set_title('Continuous probability distribution');
```

## Notation

1. $p(.)$ can mean different things depending on the context.
    1. $p(X)$ denotes the distribution (PMF/PDF) of a random variable $X$.
    1. $p(X=x)$ or $p(x)$ denotes the proability or probability density at point $x$. The actual meaning needs to be understood from context.
1. The following means **drawing from a random sample** from the distribution $p(X)$:

$$
\begin{equation}
x \sim p(X)
\end{equation}
$$

## Joint Distributions

Often in real life, we are interested in multiple random variables that are related to each other. For example, let us choose a random set of students in a classroom. We would like to study the number of students in the classroom, their ages, salaries in their first job, etc. Each of these is a random variable and they are dependent on one another. The ideas are similar to what we have seen so far. The only difference is that instead of one random variable, we will study two or more random variables.

## Joint Probability Mass Function (Joint PMF)

For a discrete random variable $X$, we define the PMF as $p_X(x) = p(X=x)$. Now, if there are two random variables $X$ and $Y$, their joint probability mass function is defined as follows:

$$
\begin{equation}
p_{XY}(x,y)=p(X=x, Y=y).
\end{equation}
$$

Since the comman means "and", so we can write

$$
\begin{align}
p_{XY}(x,y)&=p(X=x, Y=y) \\
&= p\big((X=x)\textrm{ and }(Y=y)\big).
\end{align}
$$

For discrete random variables, the joint PMF $p(X, Y)$ is like a table that sums to 1.

$$
\begin{equation}
\sum_x \sum_y p(X=x, Y=y) = 1
\end{equation}
$$

## Joint Cumulative Distributive Function (Joint CDF)

For a random variable $X$, we define the CDF as $p_X(x)=p(X \leq x)$. Now, if we have two random variables $X$ and $Y$ and we would like to study them jointly, we can define the joint cumulative function as follows:

$$
\begin{align}%\label{}
\nonumber p_{XY}(x,y)=p(X \leq x, Y \leq y).
\end{align}
$$

As usual, comma means "and," so we can write

$$
\begin{align}
\nonumber p_{XY}(x,y)&=p(X \leq x, Y \leq y) \\
\nonumber &= p\big((X \leq x)\textrm{ and }(Y\leq y)\big)=p\big((X \leq x)\cap(Y\leq y)\big).
\end{align}
$$

For continuous random variables, the joint PDF $p(X, Y)$ is given by

$$
\begin{equation}
\int_x \int_y p(X=x, Y=y)dxdy = 1
\end{equation}
$$

## Marginal Probability Mass Function (Marginal PMF)

## References

1. https://www.cse.iitk.ac.in/users/piyush/courses/pml_fall17/material/probabilty_tutorial.pdf
1. https://towardsdatascience.com/probability-concepts-explained-probability-distributions-introduction-part-3-4a5db81858dc
1. http://www.mit.edu/~6.s085/notes/lecture3.pdf
1. http://www.ams.sunysb.edu/~zhu/ams315/ch11.pdf
1. https://bookdown.org/sbikienga/Intro_to_stat_book/foundations-for-inference.html#confidence-intervals
1. https://www.cse.iitk.ac.in/users/piyush/courses/pml_fall17/material/probabilty_tutorial.pdf

```python

```


# SymPy Tutorial

*Arthur Ryman* <br/>
*Last Updated: 2020-04-16*

This notebook contains the examples from the
[SymPy Tutorial](https://docs.sympy.org/latest/tutorial/index.html).

## Solvers

```python
from sympy import *
x, y, z = symbols('x y z')
init_printing(use_unicode=True)
```

### A Note about Equations

```python
Eq(x, y)
```

```python
solveset(Eq(x**2, 1), x)
```

```python
solveset(Eq(x**2 - 1, 0), x)
```

```python
solveset(x**2 - 1, x)
```

### Solving Equations Algebraically

```python
solveset(x**2 - x, x)
```

```python
solveset(x - x, x, domain=S.Reals)
```

```python
solveset(sin(x) - 1, x, domain=S.Reals)
```

```python
solveset(exp(x), x)
```

```python
solveset(cos(x) - x, x)
```

```python
linsolve([x + y + z - 1, x + y + 2*z - 3], (x, y, z))
```

```python
linsolve(Matrix(([1, 1, 1, 1], [1, 1, 2, 3])), (x,y,z))
```

```python
M = Matrix(((1, 1, 1, 1), (1, 1, 2, 3)))
system = A, b = M[:, :-1], M[:, -1]
linsolve(system, x, y, z)
```

```python
a, b, c, d = symbols('a, b, c, d', real=True)
nonlinsolve([a**2 + a, a - b], [a, b])
```

```python
nonlinsolve([x*y - 1, x - 2], x, y)
```

```python
nonlinsolve([x**2 + 1, y**2 + 1], [x, y])
```

```python
from sympy import sqrt
system = [x**2 - 2*y**2 - 2, x*y - 2]
vars = [x, y]
nonlinsolve(system, vars)
```

```python
system = [exp(x) - sin(y), 1/y - 3]
nonlinsolve(system, vars)
```

```python
nonlinsolve([x*y, x*y - x], [x, y])
```

```python
system = [a**2 + a*c, a - b]
nonlinsolve(system, [a, b])
```

```python
solve([x**2 - y**2/exp(x)], [x, y], dict=True)
```

```python
solve([sin(x + y), cos(x - y)], [x, y])
```

```python
solveset(x**3 - 6*x**2  + 9*x, x)
```

```python
roots(x**3 - 6*x**2 + 9*x, x)
```

```python
solve(x*exp(x) - 1, x)
```

### Solving Differential Equations

```python
f, g = symbols('f g', cls=Function)
```

```python
f(x)
```

```python
f(x).diff(x)
```

```python
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x))
diffeq
```

```python
dsolve(diffeq, f(x))
```

```python
dsolve(f(x).diff(x)*(1 - sin(f(x))) - 1, f(x))
```


# Non-orthogonal curvilinear coordinates

In this notebook we solve Poisson's equation on a 2D wavy domain, using non-orthogonal basis vectors.

```python
import sympy as sp
import matplotlib.pyplot as plt
from IPython.display import Math
from shenfun import *
from shenfun.la import Solver2D
```

Define the non-orthogonal curvilinear coordinates. Here `psi=(u, v)` are the new coordinates and `rv` is the position vector

$$
\vec{r} = u \mathbf{i} + v\left(1- \frac{\sin 2u}{10} \right) \mathbf{j}
$$

where $\mathbf{i}$ and $\mathbf{j}$ are the Cartesian unit vectors in $x$- and $y$-directions, respectively.

```python
u = sp.Symbol('x', real=True, positive=True)
v = sp.Symbol('y', real=True)
psi = (u, v)
rv = (u, v*(1-sp.sin(2*u)/10))
```

Now choose basis functions and create tensor product space. Notice that one has to use complex Fourier space and not the real, because the integral measure is a function of u.

```python
N = 20
#B0 = FunctionSpace(N, 'C', bc=(0, 0), domain=(0, 2*np.pi))
B0 = FunctionSpace(N, 'F', dtype='D', domain=(0, 2*np.pi))
B1 = FunctionSpace(N, 'L', bc=(0, 0), domain=(-1, 1))

T = TensorProductSpace(comm, (B0, B1), dtype='D', coordinates=(psi, rv, sp.Q.negative(sp.sin(2*u)-10) & sp.Q.negative(sp.sin(2*u)/10-1)))
p = TrialFunction(T)
q = TestFunction(T)
b = T.coors.get_covariant_basis()
T.coors.sg
```

```python
sp.Matrix(T.coors.get_covariant_metric_tensor())
```

```python
Math(T.coors.latex_basis_vectors(symbol_names={u: 'u', v: 'v'}))
```

Plot the mesh to see the domain.

```python
mesh = T.local_cartesian_mesh()
x, y = mesh
plt.figure(figsize=(10, 4))
for i, (xi, yi) in enumerate(zip(x, y)):
    plt.plot(xi, yi, 'b')
    plt.plot(x[:, i], y[:, i], 'r')
```

Print the Laplace operator in curvilinear coordinates. We use `replace` to simplify the expression.

```python
dp = div(grad(p))
g = sp.Symbol('g', real=True, positive=True)
replace = [(1-sp.sin(2*u)/10, sp.sqrt(g)), (sp.sin(2*u)-10, -10*sp.sqrt(g)), (5*sp.sin(2*u)-50, -50*sp.sqrt(g))]
Math((dp*T.coors.sg**2).tolatex(funcname='p', symbol_names={u: 'u', v: 'v'}, replace=replace))
```

Solve Poisson's equation. First define a manufactured solution and assemble the right hand side

```python
ue = sp.sin(2*u)*(1-v**2)
f = (div(grad(p))).tosympy(basis=ue, psi=psi)
fj = Array(T, buffer=f*T.coors.sg)
f_hat = Function(T)
f_hat = inner(q, fj, output_array=f_hat)
```

Then assemble the left hand side and solve using a generic 2D solver

```python
M = inner(q, div(grad(p))*T.coors.sg)
#M = inner(grad(q*T.coors.sg), -grad(p))
u_hat = Function(T)
Sol1 = Solver2D(M)
u_hat = Sol1(f_hat, u_hat)
uj = u_hat.backward()
uq = Array(T, buffer=ue)
print('Error =', np.linalg.norm(uj-uq))
```

```python
for i in range(len(M)):
    print(len(M[i].mats[0].keys()), len(M[i].mats[1].keys()), M[i].mats[0].measure, M[i].mats[1].measure)
```

Plot the solution in the wavy Cartesian domain

```python
xx, yy = T.local_cartesian_mesh()
plt.figure(figsize=(12, 4))
plt.contourf(xx, yy, uj.real)
plt.colorbar()
```

Inspect the sparsity pattern of the generated matrix on the left hand side

```python
from matplotlib.pyplot import spy
plt.figure()
spy(Sol1.mat, markersize=0.1)
```

```python
from scipy.sparse.linalg import eigs
mats = inner(q*T.coors.sg, -div(grad(p)))
Sol1 = Solver2D(mats)
BB = inner(p, q*T.coors.sg)
Sol2 = Solver2D([BB])
f = eigs(Sol1.mat, k=20, M=Sol2.mat, which='LM', sigma=0)
```

```python
mats
```

```python
l = 10
u_hat = Function(T)
u_hat[:, :-2] = f[1][:, l].reshape(T.dims())
plt.contourf(xx, yy, u_hat.backward().real, 100)
plt.colorbar()
```