# Exercises

## Exercise 1: Stabilizing the Crank-Nicolson method by Rannacher time stepping

mathcal{I}_t is well known that the Crank-Nicolson method may give rise to
non-physical oscillations in the solution of diffusion equations
if the initial data exhibit jumps (see the section [diffu:pde1:analysis:CN](#diffu:pde1:analysis:CN)).
Rannacher [[Rannacher_1984]](#Rannacher_1984) suggested a stabilizing technique
consisting of using the Backward Euler scheme for the first two
time steps with step length $\frac{1}{2}\Delta t$. One can generalize
this idea to taking $2m$ time steps of size $\frac{1}{2}\Delta t$ with
the Backward Euler method and then continuing with the
Crank-Nicolson method, which is of second-order in time.
The idea is that the high frequencies of the initial solution are
quickly damped out, and the Backward Euler scheme treats these
high frequencies correctly. Thereafter, the high frequency content of
the solution is gone and the Crank-Nicolson method will do well.

Test this idea for $m=1,2,3$ on a diffusion problem with a
discontinuous initial condition. Measure the convergence rate using
the solution ([diffu:analysis:pde1:step:erf:sol](#diffu:analysis:pde1:step:erf:sol)) with the boundary
conditions
([diffu:analysis:pde1:p1:erf:uL](#diffu:analysis:pde1:p1:erf:uL))-([diffu:analysis:pde1:p1:erf:uR](#diffu:analysis:pde1:p1:erf:uR))
for $t$ values such that the conditions are in the vicinity of $\pm 1$.
For example, $t< 5a 1.6\cdot 10^{-2}$ makes the solution diffusion from
a step to almost a straight line. The
program `diffu_erf_sol.py` shows how to compute the analytical
solution.

## Project 2: Energy estimates for diffusion problems

This project concerns so-called *energy estimates* for diffusion problems
that can be used for qualitative analytical insight and for
verification of implementations.

**a)**
We start with a 1D homogeneous diffusion equation with zero Dirichlet
conditions:

$$
\begin{equation}
u_t = \alpha u_xx,  x\in \Omega =(0,L),\ t\in (0,T],
\label{diffu:exer:estimates:p1:pdf} \tag{1}
\end{equation}
$$

$$
\begin{equation}
u(0,t) = u(L,t) = 0,  t\in (0,T],
\label{diffu:exer:estimates:p1:bc} \tag{2}
\end{equation}
$$

$$
\begin{equation}
u(x,0) = I(x),  x\in [0,L]
\label{diffu:exer:estimates:p1:ic} \tag{3}
\thinspace .
\end{equation}
$$

The energy estimate for this problem reads

$$
\begin{equation}
||u||_{L^2} \leq ||I||_{L^2},
\label{diffu:exer:estimates:p1:result} \tag{4}
\end{equation}
$$

where the $||\cdot ||_{L^2}$ norm is defined by

$$
\begin{equation}
||g||_{L^2} = \sqrt{\int_0^L g^2dx}\thinspace .
\label{diffu:exer:estimates:L2} \tag{5}
\end{equation}
$$

The quantify  $||u||_{L^2}$ or $\frac{1}{2} ||u||_{L^2}$ is known
as the *energy* of the solution, although it is not the physical
energy of the system. A mathematical tradition has introduced the
notion *energy* in this context.

The estimate ([4](#diffu:exer:estimates:p1:result)) says that the
"size of $u$" never exceeds that of the initial condition,
or more precisely, it says that the area under the $u$ curve decreases
with time.

To show ([4](#diffu:exer:estimates:p1:result)), multiply the PDE
by $u$ and integrate from $0$ to $L$. Use that $uu_t$ can be
expressed as the time derivative of $u^2$ and that $u_xxu$ can
integrated by parts to form an integrand $u_x^2$. Show that
the time derivative of $||u||_{L^2}^2$ must be less than or equal
to zero. Integrate this expression and derive
([4](#diffu:exer:estimates:p1:result)).

**b)**
Now we address a slightly different problem,

$$
\begin{equation}
u_t = \alpha u_xx + f(x,t),  x\in \Omega =(0,L),\ t\in (0,T],
\label{diffu:exer:estimates:p2:pdf} \tag{6}
\end{equation}
$$

$$
\begin{equation}
u(0,t) = u(L,t) = 0,  t\in (0,T],
\label{diffu:exer:estimates:p2:bc} \tag{7}
\end{equation}
$$

$$
\begin{equation}
u(x,0) = 0,  x\in [0,L]
\label{diffu:exer:estimates:p2:ic} \tag{8}
\thinspace .
\end{equation}
$$

The associated energy estimate is

$$
\begin{equation}
||u||_{L^2} \leq ||f||_{L^2}\thinspace .
\label{diffu:exer:estimates:p2:result} \tag{9}
\end{equation}
$$

(This result is more difficult to derive.)

Now consider the compound problem with an initial condition $I(x)$ and
a right-hand side $f(x,t)$:

$$
\begin{equation}
u_t = \alpha u_xx + f(x,t),  x\in \Omega =(0,L),\ t\in (0,T],
\label{diffu:exer:estimates:p3:pdf} \tag{10}
\end{equation}
$$

$$
\begin{equation}
u(0,t) = u(L,t) = 0,  t\in (0,T],
\label{diffu:exer:estimates:p3:bc} \tag{11}
\end{equation}
$$

$$
\begin{equation}
u(x,0) = I(x),  x\in [0,L]
\label{diffu:exer:estimates:p3:ic} \tag{12}
\thinspace .
\end{equation}
$$

Show that if $w_1$ fulfills
([1](#diffu:exer:estimates:p1:pdf))-([3](#diffu:exer:estimates:p1:ic))
and $w_2$ fulfills
([6](#diffu:exer:estimates:p2:pdf))-([8](#diffu:exer:estimates:p2:ic)),
then $u=w_1 + w_2$ is the solution of
([10](#diffu:exer:estimates:p3:pdf))-([12](#diffu:exer:estimates:p3:ic)).
Using the triangle inequality for norms,

$$
||a + b|| \leq ||a|| + ||b||,
$$

show that the energy estimate for
([10](#diffu:exer:estimates:p3:pdf))-([12](#diffu:exer:estimates:p3:ic))
becomes

$$
\begin{equation}
||u||_{L^2} \leq ||I||_{L^2} + ||f||_{L^2}\thinspace .
\label{diffu:exer:estimates:p3:result} \tag{13}
\end{equation}
$$

**c)**
One application of ([13](#diffu:exer:estimates:p3:result)) is to prove uniqueness
of the solution.
Suppose $u_1$ and $u_2$ both fulfill
([10](#diffu:exer:estimates:p3:pdf))-([12](#diffu:exer:estimates:p3:ic)).
Show that $u=u_1 - u_2$ then fulfills
([10](#diffu:exer:estimates:p3:pdf))-([12](#diffu:exer:estimates:p3:ic))
with $f=0$ and $I=0$. Use ([13](#diffu:exer:estimates:p3:result))
to deduce that the energy must be zero for all times and therefore
that $u_1=u_2$, which proves that the solution is unique.

**d)**
Generalize ([13](#diffu:exer:estimates:p3:result)) to a 2D/3D
diffusion equation $u_t = \nabla\cdot (\alpha \nabla u)$ for $x\in\Omega$.

**Hint.**
Use integration by parts in multi dimensions:

$$
\int_\Omega u \nabla\cdot (\alpha\nabla u)\dx =
- \int_\Omega \alpha \nabla u\cdot\nabla u\dx
+ \int_{\partial\Omega} u \alpha\frac{\partial u}{\partial n},
$$

where $\frac{\partial u}{\partial n} = \boldsymbol{n}\cdot\nabla u$,
$\boldsymbol{n}$ being the outward unit normal to the boundary $\partial\Omega$
of the domain $\Omega$.

**e)**
Now we also consider the multi-dimensional PDE $u_t =
\nabla\cdot (\alpha \nabla u)$. Integrate both sides over $\Omega$
and use Gauss' divergence theorem, $\int_\Omega \nabla\cdot\boldsymbol{q}\dx
= \int_{\partial\Omega}\boldsymbol{q}\cdot\boldsymbol{n}\ds$ for a vector field
$\boldsymbol{q}$. Show that if we have homogeneous Neumann conditions
on the boundary, $\partial u/\partial n=0$, area under the
$u$ surface remains constant in time and

$$
\begin{equation}
\int_{\Omega} u\dx = \int_{\Omega} I\dx
\thinspace .
\label{diffu:exer:estimates:p4:result} \tag{14}
\end{equation}
$$

**f)**
Establish a code in 1D, 2D, or 3D that can solve a diffusion equation with a
source term $f$, initial condition $I$, and zero Dirichlet or
Neumann conditions on the whole boundary.

We can use ([13](#diffu:exer:estimates:p3:result))
and ([14](#diffu:exer:estimates:p4:result)) as a partial verification
of the code. Choose some functions $f$ and $I$ and
check that ([13](#diffu:exer:estimates:p3:result)) is obeyed at any
time when zero Dirichlet conditions are used.
mathcal{I}_terate over the same $I$ functions and check that
([14](#diffu:exer:estimates:p4:result)) is fulfilled
when using zero Neumann conditions.

**g)**
Make a list of some possible bugs in the code, such as indexing errors
in arrays, failure to set the correct boundary conditions,
evaluation of a term at a wrong time level, and similar.
For each of the bugs, see if the verification tests from the previous
subexercise pass or fail. This investigation shows how strong
the energy estimates and the estimate ([14](#diffu:exer:estimates:p4:result))
are for pointing out errors in the implementation.

Filename: `diffu_energy`.

## Exercise 3: Splitting methods and preconditioning

In the section [diffu:2D:direct_vs_iter](#diffu:2D:direct_vs_iter), we outlined a class of
iterative methods for $Au=b$ based on splitting $A$ into $A=M-N$
and introducing the iteration

$$
Mu^{k} = Nu^k + b\thinspace .
$$

The very simplest splitting is $M=I$, where $I$ is the identity
matrix. Show that this choice corresponds to the iteration

$$
\begin{equation}
u^k = u^{k-1} + r^{k-1},\quad r^{k-1} = b - Au^{k-1},
\label{diffu:exer:splitting_prec:simplest} \tag{15}
\end{equation}
$$

where $r^{k-1}$ is the residual in the linear system in iteration
$k-1$. The formula ([15](#diffu:exer:splitting_prec:simplest)) is known
as Richardson's iteration.
Show that if we apply the simple iteration method
([15](#diffu:exer:splitting_prec:simplest)) to the *preconditioned*
system $M^{-1}Au=M^{-1}b$, we arrive at the Jacobi method by choosing
$M=D$ (the diagonal of $A$) as preconditioner and the SOR method by
choosing $M=\omega^{-1}D + L$ ($L$ being the lower triangular part of
$A$).  This equivalence shows that we can apply one iteration of the
Jacobi or SOR method as preconditioner.

**Solution.**
Inserting $M=I$ and $N=I-A$ in the iterative method leads to

$$
u^{k} = (I-A)u^{k-1} + b = u^{k-1} + (b  - Au^{k-1}),
$$

which is ([15](#diffu:exer:splitting_prec:simplest)).
Replacing $A$ by $M^{-1}A$ and $b$ by $M^{-1}b$ in this equation
gives

$$
u^k = u^{k-1} + M^{-1}r^{k-1},\quad r^{k-1}=b-Au^{k-1},
$$

which we after multiplication by $M$ and reordering can write
as

$$
Mu^k = (M-A)u^{k-1} + b = Nu^{k-1} + b,
$$

which is the standard form for the Jacobi and SOR methods. Choosing $M=D$
gives Jacobi and $M=\omega^{-1}D+L$ gives SOR. We have shown that we may
view $M$ as a preconditioner of a simplest possible iteration method.

## Problem 4: Oscillating surface temperature of the earth

Consider a day-and-night or seasonal variation in temperature at
the surface of the earth. How deep down in the ground will the
surface oscillations reach? For simplicity, we model only the
vertical variation along a coordinate $x$, where $x=0$ at the
surface, and $x$ increases as we go down in the ground.
The temperature is governed by the heat equation

$$
\varrho c_v\frac{\partial T}{\partial t} = \nabla\cdot(k\nabla T),
$$

in some spatial domain $x\in [0,L]$, where $L$ is chosen large enough such
that we can assume that $T$ is approximately constant, independent of the surface
oscillations, for $x>L$. The parameters $\varrho$, $c_v$, and $k$ are the
density, the specific heat capacity at constant volume, and the
heat conduction coefficient, respectively.

**a)**
Derive the mathematical model for computing $T(x,t)$.
Assume the surface oscillations to be sinusoidal around some mean
temperature $T_m$. Let $T=T_m$ initially. At $x=L$, assume $T\approx T_m$.

**Solution.**
The surface temperature is set as

$$
T(0,t) = T_m + A\sin(\omega t)\thinspace .
$$

With only one "active" spatial coordinate we get the initial-boundary
value problem

$$
\begin{alignat*}{2}
\varrho c_v \frac{\partial T}{\partial t} &= \frac{\partial}{\partial x}
\left(k(x)\frac{\partial T}{\partial x}\right), & x\in (0,L),\ t\in (0,T],\\
T(x,0)&= T_m, & x\in [0,L],\\
T(0,t)&= T_m + A\sin(\omega t), & t\in (0,T],\\
T(L,t) &= T_m, & t\in (0,T].
\end{alignat*}
$$

**b)**
Scale the model in a) assuming $k$ is constant. Use a time scale
$t_c = \omega^{-1}$ and a length scale $x_c = \sqrt{2\dfc/\omega}$,
where $\dfc = k/(\varrho c_v)$. The primary unknown can be scaled
as $\frac{T-T_m}{2A}$.

Show that the scaled PDE is

$$
\frac{\partial u}{\partial \bar t} =
\frac{1}{2}\frac{\partial^2 u}{\partial x^2},
$$

with initial condition $u(\bar x,0) = 0$,
left boundary condition
$u(0,\bar t)  = \sin(\bar t)$,
and right boundary condition
$u(\bar L,\bar t)  = 0$. The bar indicates a dimensionless quantity.

Show that $u(\bar x, \bar t)=e^{-\bar x}\sin (\bar x - \bar t)$ is a
solution that fulfills the PDE and the boundary condition at $\bar x
=0$ (this is the solution we will experience as $\bar
t\rightarrow\infty$ and $L\rightarrow\infty$).  Conclude that an
appropriate domain for $x$ is $[0,4]$ if a damping $e^{-4}\approx
0.18$ is appropriate for implementing $\bar u\approx\hbox{const}$;
increasing to $[0,6]$ damps $\bar u$ to 0.0025.

**Solution.**
Chapter 3.2.4 in the book [[Langtangen_scaling]](#Langtangen_scaling) describes the
scaling of this problem in detail.
Inserting dimensionless variables $\bar t = \omega t$, $\bar x =
\sqrt{\omega/(2\dfc)} x$, and

$$
u = \frac{T-T_m}{2A},
$$

leads to

$$
\begin{alignat*}{2}
\frac{\partial u}{\partial \bar t} &=
\frac{1}{2}\frac{\partial^2 u}{\partial x^2},
\quad & \bar x\in (0,\bar L),\ \bar t\in (0,\bar T],
\\
u(\bar x,0) &= 0,
\quad &\bar x\in [0,1],
\\
u(0,\bar t) & = \sin(\bar t),
\quad  &\bar t\in (0,\bar T],
\\
u(\bar L,\bar t) & = 0,
\quad &\bar t\in (0,\bar T].
\end{alignat*}
$$

The domain lengths $\bar L$ and $\bar T$ follows from straightforward
scaling of $L$ and $T$.

Inserting $u(\bar x, \bar t)=e^{-\bar x}\sin (\bar t - \bar x)$ in the
PDE shows that this is a solution. mathcal{I}_t also obeys
the boundary condition $\bar u(0,\bar t)=sin(\bar t)$. As
$\bar t\rightarrow\infty$, the initial condition has no longer impact
on the solution and is "forgotten" and of no interest.
The boundary condition at $\bar x=\bar L$ is never compatible with the
given solution unless $\bar u$ is damped to zero, which happens
mathematically as $\bar L\rightarrow\infty$. For a numerical solution,
however, we may use a small finite value such as $\bar L=4$.

**c)**
Compute the scaled temperature and make animations comparing two solutions
with $\bar L=4$ and $\bar L=8$, respectively (keep $\Delta x$ the same).

**Solution.**
We can use the `viz` function in `diff1D_vc.py` to do the number
crunching. Appropriate calls and visualization go here:

```python
%matplotlib inline

import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src-diffu'))
from diffu1D_vc import viz

sol = []  # store solutions
for Nx, L in [[20, 4], [40, 8]]:
    dt = 0.1
    dx = float(L)/Nx
    D = dt/dx**2
    from math import pi, sin
    T = 2*pi*6
    from numpy import zeros
    a = zeros(Nx+1) + 0.5
    cpu, u_ = viz(
        I=lambda x: 0, a=a, L=L, Nx=Nx, D=D, T=T,
        umin=-1.1, umax=1.1, theta=0.5,
        u_L=lambda t: sin(t),
        u_R=0,
        animate=False, store_u=True)
    sol.append(u_)
    print('computed solution for Nx=%d in [0,%g]' % (Nx, L))

print sol[0].shape
print sol[1].shape
import scitools.std as plt
counter = 0
for u0, u1 in zip(sol[0][2:], sol[1][2:]):
    x0 = sol[0][0]
    x1 = sol[1][0]
    plt.plot(x0, u0, 'r-', x1, u1, 'b-',
             legend=['short', 'long'],
             savefig='tmp_%04d.png' % counter,
             axis=[x1[0], x1[-1], -1.1, 1.1])
    counter += 1
```

```python
from IPython.display import HTML
_s = """
<div>

</div>
<p><em></em></p>

<!-- Issue warning if in a Safari browser -->

"""
HTML(_s)
```

## Problem 5: Oscillating and pulsating flow in tubes

We consider flow in a straight tube with radius $R$ and straight walls.
The flow is driven by a pressure gradient $\beta(t)$. The effect of
gravity can be neglected. The mathematical problem reads

$$
\begin{equation}
\varrho\frac{\partial u}{\partial t} =
\mu\frac{1}{r}\frac{\partial}{\partial r}\left(
r\frac{\partial u}{\partial r}\right) + \beta(t),\quad
 r\in [0,R],\ t\in (0,T],
\label{_auto1} \tag{16}
\end{equation}
$$

$$
\begin{equation}
u(r,0) = I(r),\quad  r\in [0,R],
\label{_auto2} \tag{17}
\end{equation}
$$

$$
\begin{equation}
u(R,t) = 0,\quad  t\in (0,T],
\label{_auto3} \tag{18}
\end{equation}
$$

$$
\begin{equation}
\frac{\partial u}{\partial r}(0,t) = 0,\quad  t\in (0,T].
\label{_auto4} \tag{19}
\end{equation}
$$

We consider two models for $\beta(t)$. One plain, sinusoidal oscillation:

$$
\begin{equation}
\beta = A\sin(\omega t),
\label{_auto5} \tag{20}
\end{equation}
$$

and one with periodic pulses,

$$
\begin{equation}
\beta = A\sin^{16}(\omega t),
\label{_auto6} \tag{21}
\end{equation}
$$

Note that both models can be written as $\beta = A\sin^m(\omega t)$, with
$m=1$ and $m=16$, respectively.

**a)**
Scale the mathematical model, using the viscous time scale $\varrho R^2/\mu$.

**Solution.**
We can introduce

$$
\bar r = \frac{r}{R}, \quad \bar t = \frac{t}{\varrho R^2/\mu},\quad u = \frac{u}{u_c}\thinspace .
$$

Inserted in the PDE, we get

$$
\frac{\partial\bar u}{\partial\bar t} =
\frac{1}{\bar r}\frac{\partial}{\partial\bar r}\left(
\bar r\frac{\partial\bar u}{\partial\bar r}\right) +
\frac{R^2 A}{u_c \mu}\sin^m (\alpha\bar t)
$$

where $\alpha$ is a dimensionless number

$$
\alpha = \frac{\omega\varrho R^2}{\mu} = \frac{\varrho R^2/\mu}{1/\omega},
$$

reflecting the ratio of the viscous diffusion time scale and the
time scale of the oscillating pressure gradient.
We may choose $u_c$ such that the coefficient in the pressure gradient
term equals unity:

$$
u_c = \frac{R^2 A}{\mu}\thinspace .
$$

The governing PDE, dropping the bars, then reads

$$
\frac{\partial u}{\partial t} =
\frac{1}{r}\frac{\partial}{\partial r}\left(
r\frac{\partial u}{\partial r}\right) +
\sin^m (\alpha\bar t),\quad r\in (0,1),\ t\in (0,T]\thinspace .
$$

**b)**
Implement the scaled model from a), using the unifying $\theta$ scheme
in time and centered differences in space.

**Solution.**
We need to take into account extensions below: a coefficient in front of
the viscous term, and an extra source term.

A preliminary and unfinished code:

```python
"""
Solve the diffusion equation for axi-symmetric case:

    u_t = 1/r * (r*a(r)*u_r)_r + f(r,t)

on (0,R) with boundary conditions u(0,t)_r = 0 and u(R,t) = 0,
for t in (0,T]. Initial condition: u(r,0) = I(r).
Pressure gradient f.

The following naming convention of variables are used.

===== ==========================================================
Name  Description
===== ==========================================================
Nx    The total number of mesh cells; mesh points are numbered
      from 0 to Nx.
T     The stop time for the simulation.
I     Initial condition (Python function of x).
a     Variable coefficient (constant).
R     Length of the domain ([0,R]).
r     Mesh points in space.
t     Mesh points in time.
n     Index counter in time.
u     Unknown at current/new time level.
u_1   u at the previous time level.
dr    Constant mesh spacing in r.
dt    Constant mesh spacing in t.
===== ==========================================================

``user_action`` is a function of ``(u, r, t, n)``, ``u[i]`` is the
solution at spatial mesh point ``r[i]`` at time ``t[n]``, where the
calling code can add visualization, error computations, data analysis,
store solutions, etc.
"""

import scipy.sparse
import scipy.sparse.linalg
from numpy import linspace, zeros, random, array, ones, sum, log, sqrt
import time, sys
import sympy as sym

def solver_theta(I, a, R, Nr, D, T, theta=0.5, u_L=None, u_R=0,
                 user_action=None, f=0):
    """
    The array a has length Nr+1 and holds the values of
    a(x) at the mesh points.

    Method: (implicit) theta-rule in time.

    Nr is the total number of mesh cells; mesh points are numbered
    from 0 to Nr.
    D = dt/dr**2 and implicitly specifies the time step.
    T is the stop time for the simulation.
    I is a function of r.
    u_L = None implies du/dr = 0, i.e. a symmetry condition
    f(r,t) is pressure gradient with radius.

    user_action is a function of (u, x, t, n) where the calling code
    can add visualization, error computations, data analysis,
    store solutions, etc.

    r*alpha is needed midway between spatial mesh points, - use
    arithmetic mean of successive mesh values (i.e. of r_i*alpha_i)
    """
    import time
    t0 = time.perf_counter()

    r = linspace(0, R, Nr+1)   # mesh points in space
    dr = r[1] - r[0]
    dt = D*dr**2
    Nt = int(round(T/float(dt)))
    t = linspace(0, T, Nt+1)   # mesh points in time

    if isinstance(u_L, (float,int)):
        u_L_ = float(u_L)  # must take copy of u_L number
        u_L = lambda t: u_L_
    if isinstance(u_R, (float,int)):
        u_R_ = float(u_R)  # must take copy of u_R number
        u_R = lambda t: u_R_
    if isinstance(f, (float,int)):
        f_ = float(f)  # must take copy of f number
        f = lambda r, t: f_

    ra = r*a    # help array in scheme

    inv_r = zeros(len(r)-2)    # needed for inner mesh points
    inv_r = 1.0/r[1:-1]

    u   = zeros(Nr+1)   # solution array at t[n+1]
    u_1 = zeros(Nr+1)   # solution at t[n]

    Dl = 0.5*D*theta
    Dr = 0.5*D*(1-theta)

    # Representation of sparse matrix and right-hand side
    diagonal = zeros(Nr+1)
    lower    = zeros(Nr)
    upper    = zeros(Nr)
    b        = zeros(Nr+1)

    # Precompute sparse matrix (scipy format)
    diagonal[1:-1] = 1 + Dl*(ra[2:] + 2*ra[1:-1] + ra[:-2])*inv_r
    lower[:-1] = -Dl*(ra[1:-1] + ra[:-2])*inv_r
    upper[1:]  = -Dl*(ra[2:] + ra[1:-1])*inv_r
    # Insert boundary conditions
    if u_L == None:     # symmetry axis, du/dr = 0
        diagonal[0] = 1 + 8*a[0]*Dl
        upper[0] = -8*a[0]*Dl
    else:
        diagonal[0] = 1
        upper[0] = 0
    diagonal[Nr] = 1
    lower[-1] = 0

    A = scipy.sparse.diags(
        diagonals=[diagonal, lower, upper],
        offsets=[0, -1, 1],
        shape=(Nr+1, Nr+1),
        format='csr')
    #print A.todense()

    # Set initial condition
    for i in range(0,Nr+1):
        u_1[i] = I(r[i])

    if user_action is not None:
        user_action(u_1, r, t, 0)

    # Time loop
    for n in range(0, Nt):
        b[1:-1] = u_1[1:-1] + Dr*(
            (ra[2:] + ra[1:-1])*(u_1[2:] - u_1[1:-1]) -
            (ra[1:-1] + ra[0:-2])*(u_1[1:-1] - u_1[:-2]))*inv_r + \
            dt*theta*f(r[1:-1], t[n+1]) + \
            dt*(1-theta)*f(r[1:-1], t[n])

        # Boundary conditions
        if u_L == None:     # symmetry axis, du/dr = 0
            b[0] = u_1[0] + 8*a[0]*Dr*(u_1[1] - u_1[0]) + \
                   dt*theta*f(0, (n+1)*dt) + \
                   dt*(1 - theta)*f(0, n*dt)
        else:
            b[0]  = u_L(t[n+1])
        b[-1] = u_R(t[n+1])
        #print b

        # Solve
        u[:] = scipy.sparse.linalg.spsolve(A, b)

        if user_action is not None:
            user_action(u, r, t, n+1)

        # Switch variables before next step
        u_1, u = u, u_1

    t1 = time.perf_counter()
    # return u_1, since u and u_1 are switched
    return u_1, t, t1-t0

def compute_rates(h_values, E_values):
    m = len(h_values)
    q = [log(E_values[i+1]/E_values[i])/
         log(h_values[i+1]/h_values[i])
         for i in range(0, m-1, 1)]
    q = [round(q_, 2) for q_ in q]
    return q

def make_a(alpha, r):
    """
    alpha is a func, generally of r, - but may be constant.
    Note: when solution is to be axi-symmetric, alpha
    must be so too.
    """
    a = alpha(r)*ones(len(r))
    return a

def tests_with_alpha_and_u_exact():
    '''
    Test solver performance when alpha is either const or
    a fu of r, combined with a manufactured sol u_exact
    that is either a fu of r only, or a fu of both r and t.
    Note: alpha and u_e are defined as symb expr here, since
    test_solver_symmetric needs to automatically generate
    the source term f. After that, test_solver_symmetric
    redefines alpha, u_e and f as num functions.
    '''
    R, r, t = sym.symbols('R r t')

    # alpha const ...

    # ue = const
    print('Testing with alpha = 1.5 and u_e = R**2 - r**2...')
    test_solver_symmetric(alpha=1.5, u_exact=R**2 - r**2)

    # ue = ue(t)
    print('Testing with alpha = 1.5 and u_e = 5*t*(R**2 - r**2)...')
    test_solver_symmetric(alpha=1.5, u_exact=5*t*(R**2 - r**2))

    # alpha function of r ...

    # ue = const
    print('Testing with alpha = 1 + r**2 and u_e = R**2 - r**2...')
    test_solver_symmetric(alpha=1+r**2, u_exact=R**2 - r**2)

    # ue = ue(t)
    print('Testing with alpha = 1+r**2 and u_e = 5*t*(R**2 - r**2)...')
    test_solver_symmetric(alpha=1+r**2, u_exact=5*t*(R**2 - r**2))

def test_solver_symmetric(alpha, u_exact):
    '''
    Test solver performance for manufactured solution
    given in the function u_exact. Parameter alpha is
    either a const or a function of r. In the latter
    case, an "exact" sol can not be achieved, so then
    testing switches to conv. rates.
    R is tube radius and T is duration of simulation.
    alpha constant:
        Compares the manufactured solution with the
        solution from the solver at each time step.
    alpha function of r:
        convergence rates are tested (using the sol
        at the final point in time only).
    '''

    def compare(u, r, t, n):      # user_action function
        """Compare exact and computed solution."""
        u_e = u_exact(r, t[n])
        diff = abs(u_e - u).max()
        #print diff
        tol = 1E-12
        assert diff < tol, 'max diff: %g' % diff

    def pde_source_term(a, u):
        '''Return the terms in the PDE that the source term
        must balance, here du/dt - (1/r) * d/dr(r*a*du/dr).
        a, i.e. alpha, is either const or a fu of r.
        u is a symbolic Python function of r and t.'''

        return sym.diff(u, t) - \
               (1.0/r)*sym.diff(r*a*sym.diff(u, r), r)

    R, r, t = sym.symbols('R r t')

    # fit source term
    f = sym.simplify(pde_source_term(alpha, u_exact))

    R = 1.0     # radius of tube
    T = 2.0     # duration of simulation

    if sym.diff(alpha, r) == 0:
        alpha_is_const = True
    else:
        alpha_is_const = False

    # make alpha, f and u_exact numerical functions
    alpha = sym.lambdify([r], alpha, modules='numpy')
    f = sym.lambdify([r, t], f.subs('R', R), modules='numpy')
    u_exact = sym.lambdify(
        [r, t], u_exact.subs('R', R), modules='numpy')

    I = lambda r: u_exact(r, 0)

    # some help variables
    FE = 0      # Forward Euler method
    BE = 1      # Backward Euler method
    CN = 0.5    # Crank-Nicolson method

    # test all three schemes
    for theta in (FE, BE, CN):
        print('theta: ', theta)
        E_values = []
        dt_values = []
        for Nr in (2, 4, 8, 16, 32, 64):
            print('Nr:', Nr)
            r = linspace(0, R, Nr+1)   # mesh points in space
            dr = r[1] - r[0]
            a_values = make_a(alpha, r)
            if theta == CN:
                dt = dr
            else:   # either FE or BE
                # use most conservative dt as decided by FE
                K = 1.0/(4*a_values.max())
                dt = K*dr**2
            D = dt/dr**2

            if alpha_is_const:
                u, t, cpu = solver_theta(
                        I, a_values, R, Nr, D, T,
                        theta, u_L=None, u_R=0,
                        user_action=compare, f=f)
            else:   # alpha depends on r
                u, t, cpu = solver_theta(
                        I, a_values, R, Nr, D, T,
                        theta, u_L=None, u_R=0,
                        user_action=None, f=f)

                # compute L2 error at t = T
                u_e = u_exact(r, t[-1])
                e = u_e - u
                E = sqrt(dr*sum(e**2))
                E_values.append(E)
                dt_values.append(dt)

        if alpha_is_const is False:
            q = compute_rates(dt_values, E_values)
            print('theta=%g, q: %s' % (theta, q))
            expected_rate = 2 if theta == CN else 1
            tol = 0.1
            diff = abs(expected_rate - q[-1])
            print('diff:', diff)
            assert diff < tol

if __name__ == '__main__':
    tests_with_alpha_and_u_exact()
    print('This is just a start. More remaining for this Exerc.')
```

**c)**
Verify the implementation in b) using a manufactured solution that is
quadratic in $r$ and linear in $t$. Make a corresponding test function.

**Hint.**
You need to include an extra source term
in the equation to allow for such tests. Let the spatial variation be
$1-r^2$ such that the boundary condition is fulfilled.

**d)**
Make animations for $m=1,16$ and $\alpha=1,0.1$. Choose $T$ such that
the motion has reached a steady state (non-visible changes from period to
period in $u$).

**e)**
For $\alpha\gg 1$, the scaling in a) is not good, because the
characteristic time for changes (due to the pressure) is much smaller
than the viscous diffusion time scale ($\alpha$ becomes large).
We should in this case base
the short time scale on $1/\omega$. Scale the model again, and
make an animation for $m=1,16$ and $\alpha = 10$.

**Solution.**
Now the governing PDE becomes

$$
\frac{\partial u}{\partial t} =
\alpha^{-1}\frac{1}{r}\frac{\partial}{\partial r}\left(
r\frac{\partial u}{\partial r}\right) +
\sin^m t,\quad r\in (0,1),\ t\in (0,T]\thinspace .
$$

In this case,

$$
u_c = \frac{A}{\varrho\omega}\thinspace .
$$

We see that for $\alpha\gg 1$, we can neglect the viscous term, and we
basically have a balance between the acceleration and the driving pressure
gradient:

$$
\frac{\partial u}{\partial t} = \sin^m t\thinspace .
$$

[hpl 1: This may be a great challenge numerically, since we have a plug
independent of r that oscillates back and forth. CN is probably very
unstable. Can make a point out of this. Try $\alpha=1$ and increase
gently.]

Filename: `axisymm_flow`.

## Problem 6: Scaling a welding problem

Welding equipment makes a very localized heat source that moves in
time. We shall investigate the heating due to welding and choose, for
maximum simplicity, a one-dimensional heat equation with a fixed
temperature at the ends, and we neglect melting.  We shall scale the
problem, and besides solving such a problem numerically, the aim is to
investigate the appropriateness of alternative scalings.

The governing PDE problem reads

$$
\begin{alignat*}{2}
\varrho c\frac{\partial u}{\partial t} &= k\frac{\partial^2 u}{\partial x^2}
+ f, & x\in (0,L),\ t\in (0,T),\\
u(x,0) &= U_s, & x\in [0,L],\\
u(0,t) = u(L,t) &= 0, & t\in (0,T].
\end{alignat*}
$$

Here, $u$ is the temperature, $\varrho$ the density of the material,
$c$ a heat capacity, $k$ the heat conduction coefficient, $f$ is
the heat source from the welding equipment, and $U_s$ is the
initial constant (room) temperature in the material.

A possible model for the heat source is a moving Gaussian function:

$$
f = A\exp{\left(-\frac{1}{2}\left(\frac{x-vt}{\sigma}\right)^2\right)},
$$

where $A$ is the strength, $\sigma$ is a parameter governing how
peak-shaped (or localized in space) the heat source is, and
$v$ is the velocity (in positive $x$ direction) of the source.

**a)**
Let $x_c$, $t_c$, $u_c$, and $f_c$ be scales, i.e., characteristic
sizes, of $x$, $t$, $u$, and $f$, respectively. The natural choice of
$x_c$ and $f_c$ is $L$ and $A$, since these make the scaled $x$ and
$f$ in the interval $[0,1]$.  If each of the three terms in the PDE
are equally important, we can find $t_c$ and $u_c$ by demanding that
the coefficients in the scaled PDE are all equal to unity.  Perform
this scaling. Use scaled quantities in the arguments for the
exponential function in $f$ too and show that

$$
\bar f= e^{-\frac{1}{2}\beta^2(\bar x -\gamma \bar t)^2},
$$

where $\beta$ and $\gamma$ are dimensionless numbers. Give an
interpretation of $\beta$ and $\gamma$.

**Solution.**
We introduce

$$
\bar x=\frac{x}{L},\quad \bar t = \frac{t}{t_c},\quad \bar u = \frac{u-U_s}{u_c},
\quad \bar f=\frac{f}{A}\thinspace .
$$

Inserted in the PDE and dividing by $\varrho c u_c/t_c$ such that the
coefficient in front of $\partial\bar u/\partial\bar t$ becomes unity,
and thereby all terms become dimensionless, we get

$$
\frac{\partial\bar u}{\partial\bar t} =
\frac{k t_c}{\varrho c L^2}\frac{\partial^2\bar u}{\partial\bar x^2}
+ \frac{A t_c}{\varrho c u_c}\bar f\thinspace .
$$

Demanding that all three terms are equally important, it follows that

$$
\frac{k t_c}{\varrho c L^2} = 1,\quad \frac{A t_c}{\varrho c u_c}=1\thinspace .
$$

These constraints imply the *diffusion time scale*

$$
t_c = \frac{\varrho cL^2}{k},
$$

and a scale for $u_c$,

$$
u_c = \frac{AL^2}{k}\thinspace .
$$

The scaled PDE reads

$$
\frac{\partial\bar u}{\partial\bar t} =
\frac{\partial^2\bar u}{\partial\bar x^2}
+ \bar f\thinspace .
$$

Scaling $f$ results in

$$
\begin{align*}
\bar f &= \exp{\left(-\frac{1}{2}\left(\frac{x-vt}{\sigma}\right)^2\right)}\\
&= \exp{\left(-\frac{1}{2}\frac{L^2}{\sigma^2}
\left(\bar x- \frac{vt_c}{L}t\right)^2\right)}\\
&= \exp{\left(-\frac{1}{2}\beta^2\left(\bar x-\gamma \bar t\right)^2\right)},
\end{align*}
$$

where $\beta$ and $\gamma$ are dimensionless numbers:

$$
\beta = \frac{L}{\sigma},\quad
\gamma = \frac{vt_c}{L} = \frac{v\varrho cL}{k}\thinspace .
$$

The $\sigma$ parameter measures the width of the Gaussian peak, so
$\beta$ is the ratio of the domain and the width of the heat source (large
$\beta$ implies a very peak-formed heat source).  The $\gamma$
parameter arises from $t_c/(L/v)$, which is the ratio of the diffusion
time scale and the time it takes for the heat source to travel through
the domain. Equivalently, we can multiply by $t_c/t_c$ to get $\gamma
= v/(t_cL)$ as the ratio between the velocity of the heat source and
the diffusion velocity.

**b)**
Argue that for large $\gamma$ we should base the time scale on the
movement of the heat source. Show that this gives rise to the scaled
PDE

$$
\frac{\partial\bar u}{\partial\bar t} =
\gamma^{-1}\frac{\partial^2\bar u}{\partial\bar x^2}
+ \bar f,
$$

and

$$
\bar f = \exp{(-\frac{1}{2}\beta^2(\bar x - \bar t)^2)}\thinspace .
$$

Discuss when the scalings in a) and b) are appropriate.

**Solution.**
We perform the scaling as in a), but this time we determine $t_c$ such
that the heat source moves with unit velocity. This means that

$$
\frac{vt_c}{L} = 1\quad\Rightarrow\quad t_c = \frac{L}{v}\thinspace .
$$

Scaling of the PDE gives, as before,

$$
\frac{\partial\bar u}{\partial\bar t} =
\frac{k t_c}{\varrho c L^2}\frac{\partial^2\bar u}{\partial\bar x^2}
+ \frac{A t_c}{\varrho c u_c}\bar f\thinspace .
$$

Inserting the expression for $t_c$, we have

$$
\frac{\partial\bar u}{\partial\bar t} =
\frac{k L}{\varrho c L^2v}\frac{\partial^2\bar u}{\partial\bar x^2}
+ \frac{A L}{v\varrho c u_c}\bar f\thinspace .
$$

We recognize the first coefficient as $\gamma^{-1}$, while $u_c$ can
be determined from demanding the second coefficient to be unity:

$$
u_c = \frac{AL}{v\varrho c}\thinspace .
$$

The scaled PDE is therefore

$$
\frac{\partial\bar u}{\partial\bar t} =
\gamma^{-1}\frac{\partial^2\bar u}{\partial\bar x^2}
+ \bar f\thinspace .
$$

If the heat source moves very fast, there is little time for the
diffusion to transport the heat away from the source, and the heat
conduction term becomes insignificant. This is reflected in the
coefficient $\gamma^{-1}$, which is small when $\gamma$, the ratio of
the heat source velocity and the diffusion velocity, is large.

The scaling in a) is therefore appropriate if diffusion is a
significant process, i.e., the welding equipment moves at a slow speed
so heat can efficiently spread out by diffusion. For large $\gamma$,
the scaling in b) is appropriate, and $t=1$ corresponds to having the
heat source traveled through the domain (with the scaling in a), the
heat source will leave the domain in short time).

**c)**
One aim with scaling is to get a solution that lies in the interval
$[-1,1]$. This is not always the case when $u_c$ is based on a scale
involving a source term, as we do in a) and b).  However, from the
scaled PDE we realize that if we replace $\bar f$ with $\delta\bar f$,
where $\delta$ is a dimensionless factor, this corresponds to
replacing $u_c$ by $u_c/\delta$. So, if we observe that $\bar
u\sim1/\delta$ in simulations, we can just replace $\bar f$ by $\delta
\bar f$ in the scaled PDE.

Use this trick and implement the two scaled models. Reuse software for
the diffusion equation (e.g., the `solver` function in
`diffu1D_vc.py`).  Make a function `run(gamma, beta=10, delta=40,
scaling=1, animate=False)` that runs the model with the given
$\gamma$, $\beta$, and $\delta$ parameters as well as an indicator
`scaling` that is 1 for the scaling in a) and 2 for the scaling in
b). The last argument can be used to turn screen animations on or off.

Experiments show that with $\gamma=1$ and $\beta=10$, $\delta =20$
is appropriate. Then $\max |\bar u|$ will be larger than 4 for $\gamma
=40$, but that is acceptable.

Equip the `run` function with visualization, both animation of $\bar u$
and $\bar f$, and plots with $\bar u$ and $\bar f$ for $t=0.2$ and $t=0.5$.

**Hint.**
Since the amplitudes of $\bar u$ and $\bar f$ differs by a factor $\delta$,
it is attractive to plot $\bar f/\delta$ together with $\bar u$.

**Solution.**
Here is a possible `run` function:

```python
# from .diffu1D_vc import solver
import numpy as np

def run(gamma, beta=10, delta=40, scaling=1, animate=False):
    """Run the scaled model for welding."""
    if scaling == 1:
        v = gamma
        a = 1
    elif scaling == 2:
        v = 1
        a = 1.0/gamma

    b = 0.5*beta**2
    L = 1.0
    ymin = 0
    # Need gloal to be able change ymax in closure process_u
    global ymax
    ymax = 1.2

    I = lambda x: 0
    f = lambda x, t: delta*np.exp(-b*(x - v*t)**2)

    import time
    import scitools.std as plt
    plot_arrays = []

    def process_u(u, x, t, n):
        global ymax
        if animate:
            plt.plot(x, u, 'r-',
                     x, f(x, t[n])/delta, 'b-',
                     axis=[0, L, ymin, ymax], title='t=%f' % t[n],
                     xlabel='x', ylabel='u and f/%g' % delta)
        if t[n] == 0:
            time.sleep(1)
            plot_arrays.append(x)
        dt = t[1] - t[0]
        tol = dt/10.0
        if abs(t[n] - 0.2) < tol or abs(t[n] - 0.5) < tol:
            plot_arrays.append((u.copy(), f(x, t[n])/delta))
            if u.max() > ymax:
                ymax = u.max()

    Nx = 100
    D = 10
    T = 0.5
    u_L = u_R = 0
    theta = 1.0
    cpu = solver(
        I, a, f, L, Nx, D, T, theta, u_L, u_R, user_action=process_u)
    x = plot_arrays[0]
    plt.figure()
    for u, f in plot_arrays[1:]:
        plt.plot(x, u, 'r-', x, f, 'b--', axis=[x[0], x[-1], 0, ymax],
                 xlabel='$x$', ylabel=r'$u, \ f/%g$' % delta)
        plt.hold('on')
    plt.legend(['$u,\\ t=0.2$', '$f/%g,\\ t=0.2$' % delta,
                '$u,\\ t=0.5$', '$f/%g,\\ t=0.5$' % delta])
    filename = 'tmp1_gamma%g_s%d' % (gamma, scaling)
    s = 'diffusion' if scaling == 1 else 'source'
    plt.title(r'$\beta = %g,\ \gamma = %g,\ $' % (beta, gamma)
              + 'scaling=%s' % s)
    plt.savefig(filename + '.pdf');  plt.savefig(filename + '.png')
    return cpu
```

Note that we have dropped the bar notation in the plots. mathcal{I}_t is common
to drop the bars as soon as the scaled problem is established.

**d)**
Use the software in c) to investigate $\gamma=0.2,1,5,40$ for the
two scalings. Discuss the results.

**Solution.**
For these investigations, we compare the two scalings for each of
the different $\gamma$ values. An appropriate function for automating
the tasks is

```python
def investigate():
    """Do scienfic experiments with the run function above."""
    # Clean up old files
    import glob
    for filename in glob.glob('tmp1_gamma*') + \
            glob.glob('welding_gamma*'):
        os.remove(filename)

    gamma_values = 1, 40, 5, 0.2, 0.025
    for gamma in gamma_values:
        for scaling in 1, 2:
            run(gamma=gamma, beta=10, delta=20, scaling=scaling)

    # Combine images
    for gamma in gamma_values:
        for ext in 'pdf', 'png':
            cmd = 'doconce combine_images -2 '\
                  'tmp1_gamma%(gamma)g_s1.%(ext)s '\
                  'tmp1_gamma%(gamma)g_s2.%(ext)s '\
                  'welding_gamma%(gamma)g.%(ext)s' % vars()
            os.system(cmd)
            # pdflatex doesn't like 0.2 in filenames...
            if '.' in str(gamma):
                os.rename(
                    'welding_gamma%(gamma)g.%(ext)s' % vars(),
                    ('welding_gamma%(gamma)g' % vars()).replace('.', '_')
                    + '.' + ext)
```

We run here a Backward Euler scheme with $N_x=100$ and quite long
time steps.

Running the `investigate` function, we get the following plots:

For $\gamma\ll 1$ as in $\gamma = 0.025$, the heat source moves very
slowly on the diffusion time scale and has hardly entered the medium,
while the scaling in b) is not inappropriate, but a larger $\delta$ is
needed to bring $\bar u$ around unity.  We see that for $\gamma=0.2$,
each of the scalings work, but with the diffusion time scale, the heat
source has not moved much into the domain. For $\gamma=1$, the
mathematical problems are identical and hence the plots too. For
$\gamma=5$, the time scale based on the source is clearly the best
choice, and for $\gamma=40$, only this scale is appropriate.

A conclusion is that the scaling in b) works well for a range of $\gamma$
values, also in the case $\gamma\ll 1$.

Filename: `welding`.

## Exercise 7: Implement a Forward Euler scheme for axi-symmetric diffusion

Based on the discussion in the section [diffu:fd2:radial](#diffu:fd2:radial), derive in detail
the discrete equations for a Forward Euler in time, centered in space,
finite difference method for axi-symmetric diffusion. The
diffusion coefficient may be a function of the radial coordinate.
At the outer boundary $r=R$, we may have either a Dirichlet or Robin
condition.
Implement this scheme. Construct appropriate test problems.

**Solution.**
We start with the equation at $r=0$. According to the section [diffu:fd2:radial](#diffu:fd2:radial),
we get

$$
\frac{u^{n+1}_0-u^n_0}{\Delta t} = 4\dfc(0)\frac{u_1^n - u^n_0}{\Delta r^2}
+ f_0^n\thinspace .
$$

For $i>0$, we have

$$
\begin{align*}
\frac{u^{n+1}_i-u^n_i}{\Delta t} &= \frac{1}{r_i\Delta r^2}(
\frac{1}{2}(r_i + r_{i+1})\frac{1}{2}(\dfc_i + \dfc_{i+1})(u^n_{i+1} - u^n_i) -\\
&\qquad\frac{1}{2}(r_{i-1} + r_{i})\frac{1}{2}(\dfc_{i-1} + \dfc_{i})(u^n_{i} - u^n_{i-1}))
+ f_i^n
\end{align*}
$$

Solving with respect to $u^{n+1}_i$ and introducing $D=\Delta t/\Delta r^2$
results in

$$
\begin{align*}
u^{n+1}_0 &= u^n_0 + 4D\dfc(0)(u_1^n - u^n_0)
+ f_0^n,\\
u^{n+1}_i &= u^n_i + D\frac{1}{r_i}(
\frac{1}{2}(r_i + r_{i+1})\frac{1}{2}(\dfc_i + \dfc_{i+1})(u^n_{i+1} - u^n_i) -\\
&\qquad\frac{1}{2}(r_{i-1} + r_{i})\frac{1}{2}(\dfc_{i-1} + \dfc_{i})(u^n_{i} - u^n_{i-1}))
+ \Delta t f_i^n,\\
&\qquad i = 1,\ldots,N_r-1,
\end{align*}
$$

and $u^{n+1}_i$ at the end point $i=N_r$ is assumed known in case of
a Dirichlet condition. A Robin condition

$$
-\dfc\frac{\partial u}{\partial n} = h_T(u-U_s),
$$

can be discretized at $i=N_r$ by

$$
-\alpha_i\frac{u_{i+1}^n-u_{i-1}^n}{2\Delta r} = h_T(u_i^n - U_s)\thinspace .
$$

Solving with respect to the value at the fictitious point $i+1$ gives

$$
u_{i+1}^n = u_{i-1}^n - 2\Delta r \frac{h_T}{\alpha_i}(u_i^n - U_s)\thinspace .
$$

This value is then inserted for $u_{i+1}^n$ in the discrete PDE at $i=N_r$.

Filename: `FE_axisym`.


```python
%matplotlib inline
```

# Tutorial 04: Block-sparse reduction

In many cases, the interaction radius $R$ is much smaller than the size of the domain. Consequently, the sums in the local averages (see `tuto_averages`) contain only a small fraction of non zero terms. To gain in efficiency, we can follow the classical strategy:

* Subdivide the domain into a fixed number of cells of size at least $R$.
* For a particle in a given cell, only look at the contiguous cells to compute the local averages. In dimension $d$, there are $3^d$ contiguous cells (including the cell itself).

A practical implementation is called the *Verlet list method*. However, the implementation below is different than the classical one. It is adapted from the `block-sparse reduction method <https://www.kernel-operations.io/keops/_auto_examples/pytorch/plot_grid_cluster_pytorch.html>`_ implemented in the `KeOps <https://www.kernel-operations.io/keops/index.html>`_ library.

We illustrate the gain in efficency for the Vicsek model.

Note

The method is sub-optimal for moderate numbers of particles. As a rule of thumb, the block-sparse reduction method becomes useful for systems with at least $10^4$ particles.

## Set up and benchmarks

First, some standard imports...

```python
import copy
import time
import torch
from matplotlib import pyplot as plt

use_cuda = torch.cuda.is_available()
dtype = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor
```

Let the $N$ particles be uniformly scattered in a box of size $L$ with interaction radius  $R$ and uniformly sampled velocities.

```python
from sisyphe.models import Vicsek

N = 100000
L = 100.
R = 1.

pos = L*torch.rand((N,2)).type(dtype)
vel = torch.randn(N,2).type(dtype)
vel = vel/torch.norm(vel,dim=1).reshape((N,1))

simu=Vicsek(pos=pos,vel=vel,
            v=1.,
            sigma=1.,nu=3.,
            interaction_radius=R,
            box_size=L)

simu.__next__() #GPU warmup...
```

Without block-sparse reduction, let us compute the simulation time of 100 iterations.

```python
simu_copy = copy.deepcopy(simu) # Make a new deepcopy
s = time.time()
for k in range(100):
    simu_copy.__next__()
e = time.time()

simulation_time = e-s

print("Average simulation time without block-sparse reduction: " + str(simulation_time) + " seconds.")
```

Then with block-sparse reduction... First, turn on the attribute :attr:`blocksparse <sispyphe.particles.Particles.blocksparse>`.

```python
simu.blocksparse = True
```

Then, we need to define the maximum number of cells. This can be set by the keyword argument ``number_of_cells`` when an instance of the class :class:`sisyphe.particles.Particles` is created. The number of cells has a strong influence on the efficiency of the method and should be chosen wisely.  When the optimal value is not known a priori, it is recommanded to use the  method :meth:`best_blocksparse_parameters() <sisyphe.particles.Particles.best_blocksparse_parameters>` which will time 100 iterations of the simulation for various numbers of cells and automatically choose the best one. Below, we test all the numbers of cells which are powers of the dimension (here $d=2$) between $10^2$ and $70^2$.

```python
ncell_min = 10
ncell_max = 70
fastest, nb_cells, average_simu_time, simulation_time = simu.best_blocksparse_parameters(ncell_min, ncell_max, step=1, nb_calls=100)
```

We plot the average simulation time as a function of the square root of the number of cells and print the best.

```python
plt.plot(nb_cells,average_simu_time)
plt.xlabel("Square root of the number of cells")
plt.ylabel("Simulation time")

print("Average simulation time with block-sparse reduction: " + str(average_simu_time.min()) + " seconds.")
```

Same experiment with one million particles.

```python
N = 1000000
L = 100.
R = 1.

pos = L*torch.rand((N,2)).type(dtype)
vel = torch.randn(N,2).type(dtype)
vel = vel/torch.norm(vel,dim=1).reshape((N,1))

simu=Vicsek(pos=pos,vel=vel,
            v=1.,
            sigma=1.,nu=3.,
            interaction_radius=R,
            box_size=L,
            block_sparse_reduction=False)

simu_copy = copy.deepcopy(simu) # Make a new deepcopy
s = time.time()
for k in range(100):
    simu_copy.__next__()
e = time.time()

simulation_time = e-s

print("Average simulation time without block-sparse reduction: " + str(simulation_time) + " seconds.")
```

With block-sparse reduction...

```python
simu.blocksparse = True

fastest, nb_cells, average_simu_time, simulation_time = simu.best_blocksparse_parameters(30, 100, nb_calls=100)
```

We plot the average simulation time as a function of the square root of the number of cells and print the best.

```python
plt.plot(nb_cells,average_simu_time)
plt.xlabel("Square root of the number of cells")
plt.ylabel("Simulation time")

print("Average simulation time with block-sparse reduction: " + str(average_simu_time.min()) + " seconds.")
```

Note

The optimal parameters chosen initially may not stay optimal in the course of the simulation. This may be the case in particular if there is a strong concentration of particles.

## How does it work

### Cell size and number of cells

The cells have a rectangular shape. The length of the cells along each dimension cannot be smaller than the interaction radius $R$. The maximum number of cells is thus equal to:

\begin{align}n_\mathrm{max} = \prod_{k=1}^d \left\lfloor \frac{L_k}{R} \right\rfloor,\end{align}

where $L_k$ is the length of the (rectangular) domain along dimension $k$. This corresponds to rectangular cells with a length along dimension $k$ equal to:

\begin{align}\varepsilon_k = \frac{L_k}{\left\lfloor \frac{L_k}{R} \right\rfloor}.\end{align}

If the number of cells demanded $n_0$ exceeds $n_\mathrm{max}$, this will be the chosen value. Otherwise, we first compute the typical length:

\begin{align}\varepsilon_0 = \left(\frac{\prod_{k=1}^d L_k}{n_0}\right)^{1/d}\end{align}

Then the length of the cells along dimension $k$ is set to

\begin{align}\varepsilon_k = \frac{L_k}{\left\lfloor\frac{L_k}{\varepsilon_0}\right\rfloor}.\end{align}

In particular, in a square domain $L_k=L$ for all $k$ and when $n_0$ is a power of $d$, then there are exactly $n_0$ square cells with length $L/n_0^{1/d}$.

### The block-sparse parameters

The initialisation or the method :meth:`best_blocksparse_parameters() <sisyphe.particles.Particles.best_blocksparse_parameters>` define three attributes which are used to speed up the computations. Given a number of cells, they are computed by the method :meth:`compute_blocksparse_parameters() <sisyphe.particles.Particles.compute_blocksparse_parameters>`.

* :attr:`centroids <sisyphe.Particles.particles.centroids>` : the coordinates of the centers of the cells.
* :attr:`keep <sisyphe.Particles.particles.keep>` : a square BoolTensor which indicates whether two cells are contiguous.
* :attr:`eps <sisyphe.Particles.particles.keep>` : the length of the cells along each dimension.

The particles are clustered into the cells using the method :meth:`uniform_grid_separation() <sisyphe.toolbox.uniform_grid_separation>`.

Note

A drawback of the method is the high memory cost needed to store the boolean mask :attr:`keep <sisyphe.Particles.particles.keep>`. As a consequence, unlike the classical Verlet list method, the optimal number of cells is often **not** the maximum one. In the examples presented in this documentation, the optimal number of cells is always smaller than $10^4$.


[](https://colab.research.google.com/github/MridulaMaddukuri/Pytorch_Tutorial/blob/master/Pytorch_Basics.ipynb)

## Pytorch Basics

#### 1. What are tensors?
#### 2. Creating Tensors
#### 3. Tensor Data types
#### 4. Indexing Tensors
#### 5. Tensor Manipulation
#### 6. Operations on tensors
#### 7. Matrix/Vector Operations
#### 8. Conversion from and to Numpy
#### 9. Performance : Numpy arrays vs tensors on CPU vs tensors on GPU
#### 10. What are Variables?
#### 11. Activation Funtions

---

## Pytorch

A Deep Learning framework that provides

- Tensors (a replacement for numpy arrays) with strong GPU acceleration
- Dynamic Neural Networks (with GPU acceleration)

Both features useful in Deep Learning projects

#### Comparing with the other popular DL framework: TensorFlow

> - Relatively new when compared to ***_TensorFlow_*** but quickly gaining momentum.
Easier to adapt to Pytorch due to similarities with numpy.

> - More pythonic

> - Offers flexibility to change the NN as you go : change and execute nodes as you go

More Pros and Cons can be found in this blog:

https://towardsdatascience.com/pytorch-vs-tensorflow-spotting-the-difference-25c75777377b

```
# install pytorch
!pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl
```

    Collecting torch==0.3.0.post4 from http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl
      Downloading http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl (592.3MB)
        100% |████████████████████████████████| 592.3MB 45.9MB/s
    Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from torch==0.3.0.post4) (3.13)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch==0.3.0.post4) (1.14.6)
    Installing collected packages: torch
    Successfully installed torch-0.3.0.post4

```
# import libraries
import torch
import numpy as np
```

### 1. What are tensors?

A replacement for NumPy to use the power of GPUs

### 2. Creating Tensors

```
x = torch.Tensor(2,2)
print("Shape of tensor x:" + str(x.shape))
print("Datatype of x:" + str(type(x)))
print(x) # uninitialized tensor holding garbage values
```

    Shape of tensor x:torch.Size([2, 2])
    Datatype of x:<class 'torch.FloatTensor'>

    1.00000e-19 *
      0.0000  0.0000
      1.3563  1.3994
    [torch.FloatTensor of size 2x2]

```
torch.manual_seed(2) # to make results reproducible

x = torch.rand(2,3)  # returns tensor of shape (2,3) filled with random numbers from a uniform distribution on interval [0,1)
#torch.rand? # uncomment to know more

y = torch.randn(2,3) # returns a tensor of shape (2,3) filled with random numbers from standard normal distribution : mean =0 , variance = 1

#torch.randn? # uncomment to know more

print(x,y)
# if you want these tensors on GPU
x,y = x.cuda(),y.cuda()
print(x,y)
```

     0.6147  0.3810  0.6371
     0.4745  0.7136  0.6190
    [torch.FloatTensor of size 2x3]

    -2.1409 -0.5534 -0.5000
    -0.0815 -0.1633  1.5277
    [torch.FloatTensor of size 2x3]

     0.6147  0.3810  0.6371
     0.4745  0.7136  0.6190
    [torch.cuda.FloatTensor of size 2x3 (GPU 0)]

    -2.1409 -0.5534 -0.5000
    -0.0815 -0.1633  1.5277
    [torch.cuda.FloatTensor of size 2x3 (GPU 0)]

```
# Initialize with a range of values

x = torch.arange(5)
print("arange starting with 0: ")
print(x)

x = torch.arange(2,20, step = 4)
print("arange starting with a specific value: ")
print(x)

# Initialize on a scale
x = torch.linspace(0,20, steps = 6) # includes 0 and 20
print("linspace between a and b with n steps: ")
print(x)

```

    arange starting with 0:

     0
     1
     2
     3
     4
    [torch.FloatTensor of size 5]

    arange starting with a specific value:

      2
      6
     10
     14
     18
    [torch.FloatTensor of size 5]

    linspace between a and b with n steps:

      0
      4
      8
     12
     16
     20
    [torch.FloatTensor of size 6]

### 3. Tensor Data Types

(https://pytorch.org/docs/stable/tensor_attributes.html)

Common ones we work with:

**FloatTensor** : Equivalent to numpy.float32

**LongTensor** : Equivalent to numpy.int64

```
x = torch.randn([2,3])
print("Original x: ")
print(x)
# convert floattensor to long tensor

x = x.type(torch.LongTensor)
print("Datatype modified x: ")
print(x)

# converting data type of a tensor based on another tensor
y = torch.rand([2,2])
print("Original y: ")
print(y)

y = y.type_as(x) # tensor x used as reference to change dtype
print("Datatype of y modified based on x: ")
print(y)

```

    Original x:

    -0.4023  0.0972 -0.5682
    -1.2692  0.5789 -1.5181
    [torch.FloatTensor of size 2x3]

    Datatype modified x:

     0  0  0
    -1  0 -1
    [torch.LongTensor of size 2x3]

    Original y:

     0.0458  0.1755
     0.6177  0.8291
    [torch.FloatTensor of size 2x2]

    Datatype of y modified based on x:

     0  0
     0  0
    [torch.LongTensor of size 2x2]

**_Why is this important?_**

-- For operations between tensors, they should strictly have the same data type.

-- Certain functions are also very specific about the datatypes

### 4. Indexing Tensors

Very similar to numpy indexing

```
## 2D tensor

x = torch.randn(5,3).type(torch.FloatTensor)
print(x)

print("Indexing the second column")
print(x[:,1])

print("Indexing the 2nd element of the 4th row")
print(x[3,1])

```

     1.1125 -0.7474  1.4220
    -0.5941  0.7546  0.5748
    -0.8063  0.6410  0.7828
     1.6251  0.1756 -0.8472
    -0.8448 -0.8347 -0.7278
    [torch.FloatTensor of size 5x3]

    Indexing the second column

    -0.7474
     0.7546
     0.6410
     0.1756
    -0.8347
    [torch.FloatTensor of size 5]

    Indexing the 2nd element of the 4th row
    0.17558613419532776

```
## 3D tensor

y = torch.randn(1,5,3).type(torch.FloatTensor)
print(y)

print("Indexing the 5X3 tensor from y: ")
print(y[0,:,:])

print("Indexing the 1X3 tensor from y:")
print(y[:,0,:])

print("Indexing the size 3 tensor from y:")
print(y[0,0,:])
```

    (0 ,.,.) =
     -0.0602 -0.8961  0.8293
     -0.5403  0.5956 -0.7712
      0.0709 -0.8890 -1.6091
      0.5055  0.9108  0.8569
     -1.0442  0.1758  0.6451
    [torch.FloatTensor of size 1x5x3]

    Indexing the 5X3 tensor from y:

    -0.0602 -0.8961  0.8293
    -0.5403  0.5956 -0.7712
     0.0709 -0.8890 -1.6091
     0.5055  0.9108  0.8569
    -1.0442  0.1758  0.6451
    [torch.FloatTensor of size 5x3]

    Indexing the 1X3 tensor from y:

    -0.0602 -0.8961  0.8293
    [torch.FloatTensor of size 1x3]

    Indexing the 3 tensor from y:

    -0.0602
    -0.8961
     0.8293
    [torch.FloatTensor of size 3]

### 5. Tensor Manipulation

```

x = torch.arange(9)
print(x)
print(x.shape)

# Squeeze and unsqueeze : to add or remove a dimension
print("Unsqueeze on the first axis to get a row vector: ")
x = torch.unsqueeze(x,0)
print(x)

print("Squeeze to get rid of the additional dimension: ")
x= torch.squeeze(x,0) # specify squeeze dimension: the dimension we want to get rid of
print(x)

print("Unsqueeze on the second axis to get a column vector: ")
x = torch.unsqueeze(x,1)
print(x)

# View : similar to numpy.reshape
print("Reshape the tensor: ")
x = x.view(3,3)
print(x)

```

     0
     1
     2
     3
     4
     5
     6
     7
     8
    [torch.FloatTensor of size 9]

    torch.Size([9])
    Unsqueeze on the first axis to get a row vector:

        0     1     2     3     4     5     6     7     8
    [torch.FloatTensor of size 1x9]

    Squeeze to get rid of the additional dimension:

     0
     1
     2
     3
     4
     5
     6
     7
     8
    [torch.FloatTensor of size 9]

    Unsqueeze on the second axis to get a column vector:

        0
        1
        2
        3
        4
        5
        6
        7
        8
    [torch.FloatTensor of size 9x1]

    Reshape the tensor:

     0  1  2
     3  4  5
     6  7  8
    [torch.FloatTensor of size 3x3]

```
# Concatenation
print("Concatenate in the 0th dimension:")
print(torch.cat((x,x),0))

print("Concatenate in the 1st dimension:")
print(torch.cat((x,x),1))
```

    Concatenate in the 0th dimension:

     0  1  2
     3  4  5
     6  7  8
     0  1  2
     3  4  5
     6  7  8
    [torch.FloatTensor of size 6x3]

    Concatenate in the 1st dimension:

     0  1  2  0  1  2
     3  4  5  3  4  5
     6  7  8  6  7  8
    [torch.FloatTensor of size 3x6]

```
# Stacking
print("Stacking on 0th dimension: ")
print(torch.stack((x,x),0))

print("Stacking on first dimension: ")
print(torch.stack((x,x),1))
```

    Stacking on 0th dimension:

    (0 ,.,.) =
      0  1  2
      3  4  5
      6  7  8

    (1 ,.,.) =
      0  1  2
      3  4  5
      6  7  8
    [torch.FloatTensor of size 2x3x3]

    Stacking on first dimension:

    (0 ,.,.) =
      0  1  2
      0  1  2

    (1 ,.,.) =
      3  4  5
      3  4  5

    (2 ,.,.) =
      6  7  8
      6  7  8
    [torch.FloatTensor of size 3x2x3]

### 6. Operations on Tensors

```
x= torch.randn(2,3)
print(x)

print("Get absolute values")
print(torch.abs(x))

print("Get the over all mean")
print(torch.mean(x))

print("Get the means in the first dimension: ")
print(torch.mean(x,0))

print("Get element wise square: ")
print(x.pow(2))
```

    -1.7490  1.3883 -0.1098
     1.2384  0.5407  0.5787
    [torch.FloatTensor of size 2x3]

    Get absolute values

     1.7490  1.3883  0.1098
     1.2384  0.5407  0.5787
    [torch.FloatTensor of size 2x3]

    Get the over all mean
    0.3145650302370389
    Get the means in the first dimension:

    -0.2553
     0.9645
     0.2345
    [torch.FloatTensor of size 3]

    Get element wise square:

     3.0590  1.9274  0.0121
     1.5336  0.2924  0.3349
    [torch.FloatTensor of size 2x3]

### 7. Matrix/Vector Operations

```
## MATRICES
x = torch.ones(2,2)

y = torch.rand(2,2)

print(x,y)

# addition
print("Sum of x and y: ")
Sum = x+y
print(Sum)

# element wise multiplication
print("Element wise multiplication of x and y:")
mul = torch.mul(x,y)
print(mul)

# Matrix multiplication
print("Matrix multiplication of x and y: ")
MatMul = x.mm(y)
print(MatMul)

```

     1  1
     1  1
    [torch.FloatTensor of size 2x2]

     0.8177  0.8756
     0.0064  0.5755
    [torch.FloatTensor of size 2x2]

    Sum of x and y:

     1.8177  1.8756
     1.0064  1.5755
    [torch.FloatTensor of size 2x2]

    Element wise multiplication of x and y:

     0.8177  0.8756
     0.0064  0.5755
    [torch.FloatTensor of size 2x2]

    Matrix multiplication of x and y:

     0.8241  1.4511
     0.8241  1.4511
    [torch.FloatTensor of size 2x2]

```
## VECTORS:

x = torch.arange(5)
#x= torch.unsqueeze(x,1)
print(x)

y = torch.arange(1,6)
print(y)

# inner product
print("Inner Product: ")
inner = torch.dot(x,y) # will produce a scalar. Make sure x and y are of the same size
print(inner)

# outer product
print("\nOuter Product: ")
outer = torch.ger(x,y) # will produce a matrix of size size(x) X size(y)
print(outer)

```

     0
     1
     2
     3
     4
    [torch.FloatTensor of size 5]

     1
     2
     3
     4
     5
    [torch.FloatTensor of size 5]

    Inner Product:
    40.0

    Outer Product:

      0   0   0   0   0
      1   2   3   4   5
      2   4   6   8  10
      3   6   9  12  15
      4   8  12  16  20
    [torch.FloatTensor of size 5x5]

### 8. Conversion from and to Numpy

```
x = np.array([[1,2],[3,4]])
print(x)
print(type(x))

# From numpy ndarray to Tensor
print("\nConverted to : ")
x_tensor = torch.from_numpy(x)
print(x_tensor)

# From Tensor to numpy ndarray
print("Converted back to : ")
print(x_tensor.numpy())
print(type(x_tensor.numpy()))
```

    [[1 2]
     [3 4]]
    <class 'numpy.ndarray'>

    Converted to :

     1  2
     3  4
    [torch.LongTensor of size 2x2]

    Converted back to :
    [[1 2]
     [3 4]]
    <class 'numpy.ndarray'>

### 9. Performance:  Numpy arrays vs Tensors on CPU vs Tensors on GPU

```

# numpy ndarray on CPU
print("Numpy array on CPU")
x = np.random.random((1,64))
y = np.random.random((1000, 64))
%timeit z = (x*y).sum(axis=1)

# torch Tensor on CPU
print("Tensor on CPU")
x = torch.from_numpy(x)
y = torch.from_numpy(y)
%timeit z=(x*y).sum(dim=1)

# torch Tensor on GPU
print("Tensor on GPU")
x, y = x.cuda(), y.cuda()
%timeit z = (x*y).sum(dim=1)

```

    Numpy array on CPU
    1000 loops, best of 3: 185 µs per loop
    Tensor on CPU
    1000 loops, best of 3: 235 µs per loop
    Tensor on GPU
    10000 loops, best of 3: 75.8 µs per loop

### 10. What are Variables?

Autograd package:

```
from torch.autograd import Variable

x = torch.Tensor([4.0]) #[1,2,3],[3,4,5]
print("When the shape of x is: "+str(x.shape))
var = Variable(x,requires_grad = True) # can differentiate wrt this # FOCUS MORE
print(var)

# when the output is scalar
y = torch.mean(2*var)

print(y)

y.backward() # This will throw an error

print(var.grad)

```

    When the shape of x is: torch.Size([1])
    Variable containing:
     4
    [torch.FloatTensor of size 1]

    Variable containing:
     8
    [torch.FloatTensor of size 1]

    Variable containing:
     2
    [torch.FloatTensor of size 1]

#### When x is an m X n Tensor

\begin{equation}
x =
\begin{bmatrix}
  x_{11} & x_{12} \\
  x_{21} & x_{22}
\end{bmatrix}
\end{equation}

\begin{equation}
y = mean(2*x)
\end{equation}

\begin{equation}
 y = 2 \times 1/4 \times (x_{11} +x_{12} +x_{21}+x_{22}  )
\end{equation}

\begin{equation}
\frac{dy}{dx} =
\begin{bmatrix}
  \frac{\partial y}{\partial x_{11}} & \frac{\partial y}{\partial x_{12}} \\
  \frac{\partial y}{\partial x_{21}} & \frac{\partial y}{\partial x_{22}}
\end{bmatrix}
\end{equation}

```
x = torch.Tensor([[1,2],[3,4]]) #[1,2,3],[3,4,5]
print("When the shape of x is: "+str(x.shape))
print("Wrapping Tensor x into Variable x:")
var = Variable(x,requires_grad = True) # can differentiate wrt this # FOCUS MORE
print(var)
print("Variables essentially tensors but with an ability to \
  utilize autograd package to compute gradients ")

# when the output is scalar
y = torch.mean(2*var.pow(2))

print(y)

y.backward() # This will throw an error

print(var.grad)
```

    When the shape of x is: torch.Size([2, 2])
    Wrapping Tensor x into Variable x:
    Variable containing:
     1  2
     3  4
    [torch.FloatTensor of size 2x2]

    Variables essentially tensors but with an ability to   utilize autograd package to compute gradients
    Variable containing:
     15
    [torch.FloatTensor of size 1]

    Variable containing:
     1  2
     3  4
    [torch.FloatTensor of size 2x2]

#### 11. Activation Functions

Activation functions can be found here :** Torch.nn.functional**

```
import torch.nn.functional as F
import matplotlib.pyplot as plt
x = torch.linspace(-5,5,1000)
x = Variable(x)

# using activation function
y_relu = F.relu(x)
y_sigmoid = F.sigmoid(x)
y_tanh = F.tanh(x)

# plt to visualize these activation function
plt.plot(x.data.numpy(), y_relu.data.numpy(), c='red', label='relu')
plt.plot(x.data.numpy(), y_sigmoid.data.numpy(), c='blue', label='sigmoid')
plt.plot(x.data.numpy(), y_tanh.data.numpy(), c='green', label='tanh')
plt.plot()
plt.ylim((-1.5, 1.5))
plt.title("Activation Functions")
plt.legend(loc='best');

```

```
"""# When the output is non scalar

x = Variable(torch.randn(10), requires_grad=True)
print(x)
y = x ** 2
print(y)

#grad = torch.randn(10)
#print(grad)

torch.autograd.backward([y], [x])

"""
```


```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Introduction to Control Systems v1a

Hello everyone, in this occasion I would like to share my notebook that I used to create the poster for this year EuroPython.

Don't forget to follow me on github then :)

Install the require library first, if you already installed it then skip to the next section.

```
!pip install control
!pip install slycot
!pip install scipy
!pip install numpy
!pip install matplotlib
```

    Collecting control
      Downloading https://files.pythonhosted.org/packages/e8/b0/32a903138505dd4ea523f8a3fc156c4272aa58b10100ef24ff74ced2fae8/control-0.8.3.tar.gz (249kB)
         |████████████████████████████████| 256kB 3.3MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from control) (1.18.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from control) (1.4.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from control) (3.2.2)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->control) (1.2.0)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->control) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->control) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->control) (2.8.1)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib->control) (1.12.0)
    Building wheels for collected packages: control
      Building wheel for control (setup.py) ... done
      Created wheel for control: filename=control-0.8.3-py2.py3-none-any.whl size=260982 sha256=0a6c52a9e379f850f1dbd04c38b1d1cea598de1914909142665e138bafb36f0f
      Stored in directory: /root/.cache/pip/wheels/c2/d9/cc/90b28cb139a6320a3af2285428b6da87eee8d8920c78bb0223
    Successfully built control
    Installing collected packages: control
    Successfully installed control-0.8.3
    Collecting slycot
      Downloading https://files.pythonhosted.org/packages/85/21/4e7110462f3529b2fbcff8a519b61bf64e0604b8fcbe9a07649c9bed9d7a/slycot-0.4.0.0.tar.gz (1.5MB)
         |████████████████████████████████| 1.6MB 3.4MB/s
      Installing build dependencies ... done
      Getting requirements to build wheel ... done
        Preparing wheel metadata ... done
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from slycot) (1.18.5)
    Building wheels for collected packages: slycot
      Building wheel for slycot (PEP 517) ... done
      Created wheel for slycot: filename=slycot-0.4.0-cp36-cp36m-linux_x86_64.whl size=1413148 sha256=056b26702cf834f59b6978482606dd82235755a9573fa98f837e577056f6b59f
      Stored in directory: /root/.cache/pip/wheels/a2/46/56/f82cbb2fd06556f4f3952a2eb2396e8fd29264fffecbaad3cf
    Successfully built slycot
    Installing collected packages: slycot
    Successfully installed slycot-0.4.0
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (1.4.1)
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from scipy) (1.18.5)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (1.18.5)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (3.2.2)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.8.1)
    Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.18.5)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.2.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.4.7)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib) (1.12.0)

Caveat:
*   Sorry for the array mess because the python control systems library always print that kind of array and I still try to figure out the way to get rid of it.
*   In case the equation got broken or not displayed properly try to visit the button below to view the notebook immediately from the Google Colab.

# Tutorial on Control Systems Design with Python

The LTI (Linear Time-Invariant) were assumed to be used here because nonlinear or other complex systems are difficult to design and need a more advanced understanding of the control systems field.

## Library Importing

First of all, we need to import several essential libraries for designing the control systems, as listed below

```
import control # This is python control library (https://python-control.readthedocs.io/en/latest/intro.html)
import matplotlib.pyplot as plt
import numpy as np
import scipy
from control.matlab import * # To import matlab like function in designing control systems
```

## Defining Transfer Function

Let's assume we have an arbitrary transfer function equal to

Continuous Transfer Function

\begin{align}
\frac{s}{s^2 + 2s + 6}
\end{align}

Discrete Transfer Function

\begin{align}
\frac{z}{z^2 + 2z + 6}
\end{align}

Import the Function from Python Control

```
from control import TransferFunction, pole, zero # Transfer Function function import
```

### Python Control Function

```
# Continuous Time Systems
s = TransferFunction.s
sysc = s / (s**2 + 2*s + 6)
# Discrete Time Systems
z = TransferFunction.z
sysd = z / (z**2 + 2*z + 6)
```

### MATLAB like Function

```
# Continuous Time Systems
s = tf('s')
sysc = s/(s**2 + 2*s + 6)
# Discrete Time Systems
z = tf('z')
sysd = z / (z**2 + 2*z + 6)
```

### Stability Check

In order to get the specified output, the various parameters of the system must be controlled. Along with this, the system must be stable enough so that the output must not get affected by the undesirable variations in the parameter of the system or disturbances.

Thus we can say that a stable system is designed so as to get the desired response of the system without any intolerable variation with the changes in the system parameters.

Source: https://electronicscoach.com/stability-of-control-system.html

Source: https://flylib.com/books/en/2.729.1/the_z_transform.html

In the continuous systems, it called unstable if

*   The system poles were located on right half plane of s-plane

In the discrete systems, it caled unstable if

*   The systems poles were located outside the unitary circle

```
# The function of python control libraries are the same as MATLAB
# From this we can analyze the systems stability by find out the poles

# Continuous Systems

print('Continuous Systems')
# The poles
pc = pole(sysc)
zc = zero(sysc)

print('Pole of The Systems'),
print(pc),

print()

# Discrete Systems

print('Discrete systems')

# The poles
pd = pole(sysd)
zd = zero(sysd)

print('Pole of The Systems'),
print(pd)

```

    Continuous Systems
    Pole of The Systems
    [-1.+2.23606798j -1.-2.23606798j]

    Discrete systems
    Pole of The Systems
    [-1.+2.23606798j -1.-2.23606798j]

## Defining State Space Matrix of System

Importing The Main Function

```
from control import StateSpace
```

### Convert Transfer Function to State Space Form and Vice Versa

```
# In this case Python Control Function as same as MATLAB Like Function
from control import tf2ss
sysc = tf2ss(sysc)
sysd = tf2ss(sysd)
sysc
```

    A = [[-2. -6.]
     [ 1.  0.]]

    B = [[-1.]
     [ 0.]]

    C = [[-1.  0.]]

    D = [[0.]]

```
# Assume we have systems as below
A = np.array([[-2, -6], [1, 0]])
B = np.array([[-1], [0]])
C = np.array([[-1, 0]])
D = np.array([[0]])
```

### Python Control Function

```
# Continuous Systems
sysc = StateSpace(A,B,C,D)
sysc
# Discrete Systems
ts = 0.01 # The Sampling Time
sysd = StateSpace(A,B,C,D,ts)
sysd
```

    A = [[-2. -6.]
     [ 1.  0.]]

    B = [[-1.]
     [ 0.]]

    C = [[-1.  0.]]

    D = [[0.]]

    dt = 0.01

### MATLAB like Function

```
# Continuous Systems
sysc = ss(A,B,C,D)
sysc
# Discrete Systems
ts = 0.01 # The Sampling Time
sysd = ss(A,B,C,D,ts)
sysd
```

    A = [[-2. -6.]
     [ 1.  0.]]

    B = [[-1.]
     [ 0.]]

    C = [[-1.  0.]]

    D = [[0.]]

    dt = 0.01

### Stability Check

The aim of stability check in state space form is same as in transfer function, it's to make the system is can reach the desired point or reference without any intolerable variation or change.

The way to check it also the same but in this case instead of using pole, the stability is checked using eigenvalue of the state space matrix.

```
# Check the systems stability by viewing the eigenvalue
# Continuous Systems
eigs, eigvs = np.linalg.eig(sysc.A)
print('Continuous Systems Eigenvalues'),
print(eigs)
# Discrete Systems
eigd, eigvd = np.linalg.eig(sysd.A)
print('Discrete Systems Eigenvalues'),
print(eigd)
```

    Continuous Systems Eigenvalues
    [-1.+2.23606798j -1.-2.23606798j]
    Discrete Systems Eigenvalues
    [-1.+2.23606798j -1.-2.23606798j]

### Controllability and Observability Check

The intuition according to the controllability and observability checking
*   Controllability:In order to be able to do whatever we want with the given dynamic system under control input,the system must be controllable.
* Observability:In order to see what is going on inside the system under observation,the system must be observable.

Source: https://www.ece.rutgers.edu/~gajic/psfiles/chap5.pdf

```
from control import obsv, ctrb
# In this case the function for control libraries and MATLAB are same
# Continuous Systems
# Controllability Check
cc = ctrb(sysc.A, sysc.B)
rankcc = np.linalg.matrix_rank(cc)
print('Continuous Systems', '\n')
print('The Controllability Matrix'),
print(cc),
print('Rank of Controllability Matrix'),
print(rankcc),
# Observability Check
oc = obsv(sysc.A, sysc.C)
rankoc = np.linalg.matrix_rank(oc)
print('The Observability Matrix'),
print(oc),
print('Rank of Observability Matrix'),
print(rankoc),
print()
# Discrete Systems
# Controllability Check
cd = ctrb(sysd.A, sysc.B)
rankcd = np.linalg.matrix_rank(cd)
print('Discrete Systems', '\n')
print('The Controllability Matrix'),
print(cd),
print('Rank of Controllability Matrix'),
print(rankcd),
# Observability Check
od = obsv(sysd.A, sysc.C)
rankod = np.linalg.matrix_rank(od)
print('The Observability Matrix'),
print(od),
print('Rank of Observability Matrix'),
print(rankod)
```

    Continuous Systems

    The Controllability Matrix
    [[-1.  2.]
     [ 0. -1.]]
    Rank of Controllability Matrix
    2
    The Observability Matrix
    [[-1.  0.]
     [ 2.  6.]]
    Rank of Observability Matrix
    2

    Discrete Systems

    The Controllability Matrix
    [[-1.  2.]
     [ 0. -1.]]
    Rank of Controllability Matrix
    2
    The Observability Matrix
    [[-1.  0.]
     [ 2.  6.]]
    Rank of Observability Matrix
    2

## Analyze the Control Systems

```
from control import bode_plot, nyquist_plot, root_locus, pzmap
```

##### Pole Zero Map

```
# Continuous
pzmap(sysc)
```

```
# Discrete
pzmap(sysd)
```

##### Root Locus

```
# Continuous
root_locus(sysc)
```

```
# Discrete
root_locus(sysd)
```

##### Bode Plot

```
# continuous
bode_plot(sysc)
```

```
# Discrete
bode_plot(sysd)
```

##### Nyquist Plot

```
# Continuous
nyquist_plot(sysc)
```

```
# Discrete
nyquist_plot(sysd)
```

## Test the Systems Response

Actually, we can analyze the response of systems from two approach, first time domain approach, then frequency domain approach

#### Time Domain Approach

##### Step Response

```
from control import step_response # Step Response function import
```

###### Python Control Function

```
# Continuous Time Systems
tc, yc = step_response(sysc)
plt.subplot(2,1,1)
plt.plot(tc,yc)
plt.title('Continuous Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
# Discrete Time Systems
td, yd = step_response(sysd)
plt.subplot(2,1,2)
plt.plot(td,yd)
plt.title('Discrete Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.tight_layout(1)
```

###### MATLAB like function

```
# Continuous Time Systems
yc, tc = step(sysc)
plt.subplot(2,1,1)
plt.plot(tc,yc)
plt.title('Continuous Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
# Discrete Time Systems
yd, td = step(sysd)
plt.subplot(2,1,2)
plt.plot(td,yd)
plt.title('Discrete Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.tight_layout(1)
```

##### Impulse Response

```
from control import impulse_response # Impulse Response function import
```

###### Python Control Function

```
# Continuous Time Systems
tc, yc = impulse_response(sysc)
plt.subplot(2,1,1)
plt.plot(tc,yc)
plt.title('Continuous Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
# Discrete Time Systems
td, yd = impulse_response(sysd)
plt.subplot(2,1,2)
plt.plot(td,yd)
plt.title('Discrete Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.tight_layout(1)
```

###### MATLAB like Function

```
# Continuous Time Systems
yc, tc = impulse(sysc)
plt.subplot(2,1,1)
plt.plot(tc,yc)
plt.title('Continuous Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
# Discrete Time Systems
yd, td = impulse(sysd)
plt.subplot(2,1,2)
plt.plot(td,yd)
plt.title('Discrete Step Response')
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.tight_layout(1)
```

#### Frequency Domain Approach

##### Frequency Response

```
from control import freqresp
```

```
# Continuous
freqresp(sysc, [10])
```

    (array([[[0.10405382]]]), array([[[-1.36115648]]]), array([10]))

```
# Discrete
freqresp(sysd, [12])
```

    (array([[[0.11148702]]]), array([[[0.06678141]]]), array([12]))

## Okay Then, Let's Simulate The Real World Systems!

In this occasion, the design of the control algorithm was using MATLAB like function, so the code also compatible with MATLAB. In case you want to try the simulation then it will be also worked on MATLAB but you must revise it with slight several changes on the syntax.

### DC Motor Speed Control

Source : http://ctms.engin.umich.edu/CTMS/index.php?example=MotorSpeed&section=SystemModeling (CTMS, University of Michigan)

**The controlling aim for this system is to drive the DC motor speed into the desired speed by regulating a voltage to the motor.**

The system characteristics are continuous and linear in nature. This is because in this tutorial I only limit the system to Linear System.

As well, in this case the name for the controlling goal is tracking. Since, the goal of our control problem is to drive the DC motor speed to the desired speed.

```
J = 0.08; # The motor moment of inertia
b = 0.05;
K = 0.01;
R = 0.5;
L = 0.5;
s = tf('s');
sys_tf = K/((J*s+b)*(L*s+R)+K**2)
# Also change the form of transfer function into state space model for controlling through the modern control algorithm
sys_ss = tf2ss(sys_tf)
```

```
## stability Checking
pzmap(sys_ss)
pole(sys_tf)
```

```
# Stability Checking
eig = np.linalg.eig(sys_ss.A)
print('System Eigenvalue'),
print(eig[0], "\n"),

# Controllability Check
ctrM = ctrb(sys_ss.A, sys_ss.B)
rankCtr = np.linalg.matrix_rank(ctrM)
print('Controllability Rank'),
print(rankCtr, "\n"),

# Observabiity Check
obsM = obsv(sys_ss.A, sys_ss.C)
rankObs = np.linalg.matrix_rank(obsM)
print('Observability Rank'),
print(rankObs)
```

    System Eigenvalue
    [-0.9932104 -0.6317896]

    Controllability Rank
    2

    Observability Rank
    2

It's stable due to pole or the eigenvalue located in left half plane. The system were controllable and observable because both of controllability and observability matrix have a full rank.

#### PID Controller

The equation of PID controller is denoted below,

\begin{align}
u(t) = K_p*e(t) + K_d*\frac{de(t)}{dt} + K_i * \int{e(t)}
\end{align}

Where Kp, Ki, Kd, e(t) are proportional gain, integral gain, derivative gain, and system error, respectively.

If we want to write it as transfer function form then,
\begin{align}
U(s)=K_p*E(s) + K_d * sE(s) + K_i * \frac{1}{s} * E(s)
\end{align}

If we assume
\begin{align}
\tau_i = \frac{K_p}{K_i} ;
\tau_d = \frac{K_d}{K_p}
\end{align}

Thus, the final equation for the controller is,
\begin{align}
U(s) = K_p * E(s)(1 + s * \tau_d + \frac{1}{s*\tau_i})
\end{align}

In this test I select the gain arbitrary as follows, Kp = 160, Ki = 4, and Kd = 80. As well try to experiment with the other gain combination to see the effect of gain changing.

```
Kp = 160; # The proportional gain
Ki = 4; # The integral gain
Kd = 80; # The derivative gain
Ti = Kp / Ki
Td = Kd / Kp
pid = Kp * (1 + 1 / (Ti * s) + Td * s)
isys = feedback(sys_tf * pid, 1)
t = np.linspace(0, 10)
y, t = step(isys, t)
plt.plot(t, y)
plt.title('PID Controller Response')
plt.xlabel('Time (s)')
plt.ylabel('Motor Speed (RPM)')
```

#### State Feedback Controller (Pole Placement Method)

In this design, I would like to place all of the two pole to the left half plane coordinate and place it to specific pole location as this (-3, -4). You can also try the other combination to test how the system response change

```
desiredPole = np.array([-3,-4])
ppGain = place(sys_ss.A, sys_ss.B, desiredPole)
feedbackMech = sys_ss.A - sys_ss.B * ppGain
newSys = ss(feedbackMech, sys_ss.B, sys_ss.C, 0)
t = np.linspace(0,10)
scaleSF = 1 / 0.02083333 # Because there are large steady state error we should a precompensator for scaling the reference signal
yref = 20*(np.sin(2*np.pi*t*0.1))
y, t = step(newSys * scaleSF, t)
plt.plot(t,y)
plt.title('State-Feedback Controller Response')
plt.xlabel('Time (s)')
plt.ylabel('Motor Speed (RPM)')
```

#### LQR Controller

LQR (Linear Quadratic Regulator) is the one of Optimal Control Algorithm variant, it uses calculus variation such as Algebraic Riccati Equation (ARE) to determine the optimal gain for the controller.

```
# Defining the LQR Parameter
Q = np.array([[0,0],
              [0,100]])
R = 10
gainLQR, X, E =  lqr(sys_ss.A, sys_ss.B, Q, R)
feedbackMech = sys_ss.A - sys_ss.B * gainLQR
newSysqr = ss(feedbackMech, sys_ss.B, sys_ss.C, 0)
t = np.linspace(0,10)
scaleLQR = 1 / 0.07754505 # Because there are large steady state error we should a precompensator for scaling the reference signal
y,t = step(newSysqr * scaleLQR, t)
plt.plot(t,y)
plt.title('LQR Controller Response')
plt.xlabel('Time (s)')
plt.ylabel('Motor Speed (RPM)')
```

#### Compare it Together

Before we can use the variations of reference signal such as square wave, sinusoidal signal, etc. We ought to import the forced_response function first from the Python Control Systems Library and the signal function from SciPy library

```
from control import forced_response
from scipy import signal
```

##### Step Speed Reference of 1200 RPM

```
# The Step Signal with 1500 RPM amplitude
maxSim = 10 # The simulation time is 10 second
t = np.linspace(0, maxSim)
amp = 1200 # Because the reference signal is on 1500 RPM
ref = amp *  np.ones(np.shape(t))
isys # The transfer function
y1 = forced_response(isys, T = t, U = ref)
res1 = y1[1]
y2 = forced_response(newSys * scaleSF, T = t, U = ref)
res2 = y2[1]
y3 = forced_response(newSysqr * scaleLQR, T = t, U = ref)
res3 = y3[1]
plt.plot(t,ref)
plt.plot(t,res1)
plt.plot(t,res2)
plt.plot(t,res3)
plt.title('1200 RPM Speed Reference')
plt.xlabel('Time (s)')
plt.ylabel('Motor Speed (RPM)')
plt.legend(['Reference','PID', 'State Feedback','LQR'])
from google.colab import files
plt.savefig('step.png',type = 'png',DPI = 1200)
files.download('step.png')
```

##### Sinusoidal Speed Reference

```
# The Sinusoidal Signal with 1000 RPM amplitude
maxSim = 10 # The simulation time is 10 second
t = np.linspace(0, maxSim)
amp = 1000 # Because the reference signal is on 1500 RPM
f = 0.1 # The sinusoidal signal frequency
ref = amp *  np.sin(2 * np.pi * f * t)
isys # The transfer function
y1 = forced_response(isys, T = t, U = ref)
res1 = y1[1]
y2 = forced_response(newSys * scaleSF, T = t, U = ref)
res2 = y2[1]
y3 = forced_response(newSysqr * scaleLQR, T = t, U = ref)
res3 = y3[1]
plt.plot(t,ref)
plt.plot(t,res1)
plt.plot(t,res2)
plt.plot(t,res3)
plt.title('Sinusoidal Speed Reference')
plt.xlabel('Time (s)')
plt.ylabel('Motor Speed (RPM)')
plt.legend(['Reference','PID', 'State Feedback','LQR'])
plt.savefig('')
```

##### Square Speed Reference

```
# The Square Signal with 500 RPM amplitude
maxSim = 10 # The simulation time is 10 second
t = np.linspace(0, maxSim, endpoint = True)
amp = 500 # Because the reference signal is on 500 RPM
f = 0.1 # The square signal frequency in Hz
ref = amp *  signal.square(2 * np.pi * f * t)
isys # The transfer function
y1 = forced_response(isys, T = t, U = ref)
res1 = y1[1]
y2 = forced_response(newSys * scaleSF, T = t, U = ref)
res2 = y2[1]
y3 = forced_response(newSysqr * scaleLQR, T = t, U = ref)
res3 = y3[1]
plt.plot(t,ref)
plt.plot(t,res1)
plt.plot(t,res2)
plt.plot(t,res3)
plt.title('Square Wave Speed Reference')
plt.xlabel('Time (s)')
plt.ylabel('Motor Speed (RPM)')
plt.legend(['Reference','PID', 'State Feedback','LQR'])
plt.savefig('square.png',type = 'png',DPI = 1200)
files.download('square.png')
```

##### Sawtooth Speed Reference

```
# The Sawtooth Signal with 800 RPM amplitude
maxSim = 20 # The simulation time is 20 second
t = np.linspace(0, maxSim)
amp = 800 # Because the reference signal is on 800 RPM
f = 0.2 # The sawtooth signal frequency
ref = amp *  signal.sawtooth(2 * np.pi * f * t)
y1 = forced_response(isys, T = t, U = ref)
res1 = y1[1]
y2 = forced_response(newSys * scaleSF, T = t, U = ref)
res2 = y2[1]
y3 = forced_response(newSysqr * scaleLQR, T = t, U = ref)
res3 = y3[1]
plt.plot(t,ref)
plt.plot(t,res1)
plt.plot(t,res2)
plt.plot(t,res3)
plt.title('Sawtooth Signal Speed Reference')
plt.xlabel('Time (s)')
plt.ylabel('Motor Speed (RPM)')
plt.legend(['Reference','PID', 'State Feedback','LQR'])
```

# More Advanced Control Systems Design (Optional)

Coming Soon on V2

## Robust Control Design

```

```

Copyright ©  2020 - brilianputraa


```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Philosopher', sans-serif;
    font-weight: 400;
    font-size: 2.2em;
    line-height: 100%;
    color: rgb(0, 80, 120);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Philosopher', serif;
    font-weight: 400;
    font-size: 1.9em;
    line-height: 100%;
    color: rgb(245,179,64);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Philosopher', serif;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: italic;
    color: rgb(94,127,192);
}

.text_cell_render h4 {
    font-family: 'Philosopher', serif;
}

.text_cell_render h5 {
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 16pt;
    color: grey;
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h6 {
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 10pt;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 100%;
}

</style>

```python
from sympy import init_printing, Matrix, symbols, eye, Rational
init_printing()
```

# Columnspace and nullspace of a matrix

## Columnspaces of matrices

We saw in the previous lecture that columns of a matrix can be regarded as vectors.

Consider the following example matrix.

```python
A = Matrix([[1, 1, 2], [2, 1, 3], [3, 1, 4], [4, 1, 5]])
A
```

```python
b = Matrix([1, 2, 3, 4])
b
```

Each of the column spaces are vectors (column space) in $\mathbb{R}^4$.  The linear combinations of all the column vectors form a subspace.  Is it the whole $V= \mathbb{R}^4$, though?  The reason why we ask, is because we want to bring it back to a system of linear equations and ask the question: _Is there (always) a solution to_ (1)_?_

$${A}\underline{x}=\underline{b}\tag{1}$$

In other words, which right-hand sides $\underline{b}$ are allowed?  In our example above we are in $\mathbb{R}^4$ and we ask if linear combination of all of them fill $\mathbb{R}^4$.

From our example above some right-hand sides will be allowed (they form a subspace).  Let's look at an example for $\underline{b}$.

```python
x1, x2, x3 = symbols('x1, x2, x3')
vec_x = Matrix([x1, x2, x3])
b = Matrix([1, 2, 3, 4])
A, vec_x, b
```

```python
A * vec_x
```

While we can view this as a system of linear equation, we will prefer the column view.  By viewing the matrix of coefficients, with the columns as vectors, we are asking: _How many, $x_1$, of column $1$ plus how many, $x_2$, of columns $2$, plus how many, $x_3$ of column $3$ equals $\underline{b}$_?  The question is easier to visualize when written as in (2).

$$x_1\begin{bmatrix}1\\2\\3\\4\end{bmatrix}+x_2\begin{bmatrix}1\\1\\1\\1\end{bmatrix}+x_3\begin{bmatrix}2\\3\\4\\5\end{bmatrix}=\underline{b}\tag{2}$$

Well, since $\underline{b}$ is the same as the first column, $\underline{x}$ would be as shown in (3).

$$\begin{bmatrix}1\\0\\0\end{bmatrix}\tag{3}$$

```python
A * Matrix([1, 0, 0]) == b
```

    True

### Linear independence

We really need to know if the columns above are linearly independent (not constant multiples or additions of each other).  We note that column three above is a linear combination of the first two, so adds nothing _new_.  Actually, we could also throw away the first one because it is column $3$ plus $-1$ times column $2$.  Same for column$2$.  We thus have two columns left and we say that the column space is of dimension $2$ (a 2-dimensional subspace of $\mathbb{R}^4$).

## The nullspace

The null space contains all solutions of $\underline{x}$ for $A\underline{x}=\underline{0}$.  This solution(s) is (are) in $\mathbb{R}^3$.

```python
zero_b = Matrix([0, 0, 0, 0])
A, vec_x, zero_b
```

$$\begin{pmatrix}\left[\begin{matrix}1 & 1 & 2\\2 & 1 & 3\\3 & 1 & 4\\4 & 1 & 5\end{matrix}\right], & \left[\begin{matrix}x_{1}\\x_{2}\\x_{3}\end{matrix}\right], & \left[\begin{matrix}0\\0\\0\\0\end{matrix}\right]\end{pmatrix}$$

Some possible solutions are shown in (4).

$$\begin{bmatrix}0\\0\\0\end{bmatrix},\quad\begin{bmatrix}1\\1\\-2\end{bmatrix},\quad\begin{bmatrix}2\\2\\-2\end{bmatrix}\tag{4}$$

In fact, fro any constant, $c$, these can all be written as in (5).

$$ {c}\begin{bmatrix}1\\1\\-1\end{bmatrix}\tag{5}$$

__Always__ remember, for any space the rules of addition and scalar multiplication must hold for vectors to remain in that space.

```python

```

Mathematics/Linear Algebra/0.0 MIT-18.06 - Jupyter/Lecture_6_Column_and_null_spaces.ipynb