# Robust linear programming in portfolio optimization using the NAG Library

# Correct Rendering of this notebook

This notebook makes use of the `latex_envs` Jupyter extension for equations and references.  If the LaTeX is not rendering properly in your local installation of Jupyter , it may be because you have not installed this extension.  Details at https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/latex_envs/README.html

# Installing the NAG library and running this notebook

This notebook depends on the NAG library for Python to run. Please read the instructions in the [Readme.md](https://github.com/numericalalgorithmsgroup/NAGPythonExamples/blob/master/local_optimization/Readme.md#install) file to download, install and obtain a licence for the library.

Instruction on how to run the notebook can be found [here](https://github.com/numericalalgorithmsgroup/NAGPythonExamples/blob/master/local_optimization/Readme.md#jupyter).

## Introduction

We consider a classic portfolio problem with loss risk constraints:
\vspace{0.1cn}
\begin{equation}\label{prob}
\begin{array}{ll}
\underset{x\in\Re^n}{\mbox{maximize}} & \bar{r}^Tx\\[0.6ex]
\mbox{subject to} & \mbox{Prob}(r^Tx\leq\alpha)\leq\beta,\\[0.6ex]
     & \sum_{i=1}^n x_i=1,\\[0.6ex]
     & x\geq0.
\end{array}
\end{equation}
\vspace{0.1cn}
Here we suppose the return of aseets $r\in\Re^n$ is Gaussian with mean $\bar{r}\in\Re^n$ and covariance $V\in\Re^{n\times n}$, $\alpha$  is a given unwanted return level (e.g. an excessive lose) and $\beta$ is a given maximum probability.

We will demonstrate that problem (\ref{prob}) can be transformed into a second-order cone programming (SOCP) of the following form and solved easily using the SOCP solver introduced at Mark $27$ in the NAG Library:
\vspace{0.1cn}
\begin{equation}\label{SOCP}
\begin{array}{ll}
\underset{x\in\Re^n}{\mbox{minimize}} & c^Tx\\[0.6ex]
\mbox{subject to} & l_A\leq Ax\leq u_A,\\[0.6ex]
     & l_x\leq x\leq u_x,\\[0.6ex]
     & x\in{\cal K},
\end{array}
\end{equation}
\vspace{0.1cn}
where $A\in\Re^{m\times n}$, $l_A, u_A\in\Re^m$, $c, l_x, u_x\in\Re^n$ are the problem data, and $\cal K={\cal K}^{n_1}\times\cdots\times{\cal K}^{n_r}\times\Re^{n_l}$ where ${\cal K}^{n_i}$ is either a quadratic cone or a rotated quadratic cone defined as follows:
\begin{itemize}
\item Quadratic cone:
\begin{equation}\label{SOC}
{\cal K}_q^{n_i}:=\left\lbrace x=\left(x_1,\ldots,x_{n_i}\right)\in\Re^{n_i}~:~x_1^2\geq\sum_{j=2}^{n_i} x_j^2, ~x_1\geq0\right\rbrace.
\end{equation}
\item Rotated quadratic cone:
\begin{equation}\label{RSOC}
{\cal K}_r^{n_i}=\left\lbrace x=\left(x_1,x_2,\ldots,x_{n_i}\right)\in\Re^{n_i}~:~2x_1x_2\geq\sum_{j=3}^{n_i} x_j^2,\quad x_1\geq0,\quad x_2\geq0\right\rbrace.
\end{equation}
\end{itemize}

## A closer look at the probability constraint

Let $u=r^Tx$ with $\sigma=x^TVx$ denoting its variance, the probability constraint in problem (\ref{prob}) can be written as
\vspace{0.1cn}
$$
\mbox{Prob}\left(\frac{u-\bar{u}}{\sqrt{\sigma}}\leq\frac{\alpha-\bar{u}}{\sqrt{\sigma}}\right)\leq\beta.
$$
\vspace{0.1cn}
Note that $(u-\bar{u})/\sqrt{\sigma}$ is a standard Gaussian random variable, the probability above is simply $\Phi((\alpha-\bar{u})/\sqrt{\sigma})$, where
\vspace{0.1cn}
$$
\Phi(z) = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^z e^{-t^2/2}dt
$$
\vspace{0.1cn}
is the CDF of the standard Gaussian random variable. Thus the probability constraint in problem (\ref{prob}) can be written as
\vspace{0.1cn}
$$
\frac{\alpha-\bar{u}}{\sqrt{\sigma}}\leq\Phi^{-1}(\beta)
$$
\vspace{0.1cn}
or, equivalently,
\vspace{0.1cn}
\begin{equation}\label{label_1}
\bar{u} + \Phi^{-1}(\beta)\sqrt{\sigma}\geq\alpha.
\end{equation}
\vspace{0.1cn}
From $\bar{u}=\bar{r}^Tx$ and $\sigma=x^TVx=\|Fx\|^2$ by factorizing $V=F^TF$, (\ref{label_1}) is equivalent to
\vspace{0.1cn}
\begin{equation}\label{label_2}
\bar{r}^Tx + \Phi^{-1}(\beta)\|Fx\|\geq\alpha.
\end{equation}
\vspace{0.1cn}
Depending on the value of $\beta$, (\ref{label_2}) could be convex or concave. By setting $\beta\leq0.5$ (which is reasonable for risk control), (\ref{label_2}) is convex and can be written as
\vspace{0.1cn}
\begin{equation}\label{label_3}
\bar{r}^Tx + \Phi^{-1}(\beta)t=\alpha ~ \mbox{and} ~ \|Fx\|\leq t.
\end{equation}
\vspace{0.1cm}
Note that by letting $Fx=y$, $\|Fx\|=\|y\|\leq t$ in (\ref{label_3}) fits exactly the quadratic cone constraint (\ref{SOC}), thus can be solved by second-order cone programming. The final equivalent SOCP model to problem (\ref{prob}) is
\vspace{0.1cn}
\begin{equation}\label{final}
\begin{array}{ll}
\underset{x\in\Re^n}{\mbox{maximize}} & \bar{r}^Tx\\[0.6ex]
\mbox{subject to} & \sum_{i=1}^n x_i=1,\\[0.6ex]
     & Fx = y,\\[0.6ex]
     & \bar{r}^Tx + \Phi^{-1}(\beta)t=\alpha, \\[0.6ex]
     & (t;y)\in{\cal K}^{n+1}_q,\\[0.6ex]
     & x\geq0.
\end{array}
\end{equation}
\vspace{0.1cn}

## Using the NAG Library

We demonstrate how to use the NAG SOCP solver to model and solve problem (\ref{prob}) by solving the equivalent SOCP problem (\ref{final}).

```python
# Import utility libraries and the NAG Library
import numpy as np
import math as mt
from scipy.stats import norm
from naginterfaces.library import opt
from naginterfaces.library import lapackeig
```

As an example, we set $\alpha=0.0001$, $\beta=0.05$ and randomly generate $\bar{r}$ and $V$ for $8$ assets as follows.

```python
# Set alpha and beta
alpha = 0.0001
beta = 0.05

# Fix random seed
np.random.seed(9)

# Number of assets
n_assets = 8

# Vector of expected returns
r = np.ones(n_assets)*.02 + np.random.rand(n_assets)*.15

# Covariance matrix
V = np.matrix(np.random.randn(n_assets, n_assets))
V = V.T * V
V = V / np.max(np.abs(np.diag(V))) * .2
```

Now factorize $V=F^TF$ using eigenvalue decomposition from the NAG Library.

```python
# Note one could use sparse factorization if V is input as sparse matrix
U, lamda = lapackeig.dsyevd('V', 'L', V)

# Find positive eigenvalues and corresponding eigenvectors
i = 0
k = 0
F = []

while i<len(lamda):
    if lamda[i] > 0:
        F = np.append(F, mt.sqrt(lamda[i])*U[:,i])
        k += 1
    i += 1

F = F.reshape((k, n_assets))
```

For modelling, NAG SOCP solver requires several input arguments for objective and constraints. Now we initialize the data that will be used to feed NAG SOCP solver.

```python
# Number of variables
n = n_assets
# Number of constraints
m = 0

# Objective coefficient c
c = np.full(n, 0.0, float)

# Bounds on variables
blx = np.full(n, -1.e20, float)
bux = np.full(n, 1.e20, float)

# Linear constraint bu <= Ax <= bu
# A in coordinate list format (COO)
irowa = np.empty(0, int)
icola = np.empty(0, int)
a = np.empty(0, float)
# Bounds on Ax
bl = np.empty(0, float)
bu = np.empty(0, float)

# Cone constraints
ctype = []
group = []
```

Because we will add auxiliary variables and constraints during the process, it is necessary we keep tracking the number of variables and constraints in the model by maintaining the up-to-date problem size.

```python
# Initialize the up-to-date problem size
n_up = n
m_up = m
```

Now we keep modifying the above data during adding objective and constraints one by one. First is the objective coefficient $c$.

```python
# Add objective function min -r'x
c = -r
```

Then we add the long-only constraint.

```python
# Number of linear constraints will increase by 1
m += 1

# Set lower bound on x to 0
blx[0:n] = np.zeros(n)

# Add sum(x) = 1
irowa = np.append(irowa, np.full(n, m_up+1, dtype=int))
icola = np.append(icola, np.arange(1, n+1))
a = np.append(a, np.full(n, 1.0, dtype=float))
bl = np.append(bl, np.full(1, 1.0, dtype=float))
bu = np.append(bu, np.full(1, 1.0, dtype=float))
```

Now add the probability constraint by adding
$$
Fx = y,~ \bar{r}^Tx + \Phi^{-1}(\beta)t=\alpha ~and~(t;y)\in{\cal K}^{n+1}_q.
$$

```python
# Get quantile function of beta
quantile = norm.ppf(beta)

# Up-to-date problem size
m_up = m
n_up = n

# Then k + 1 more variables need to be added together with
# k + 1 linear constraints and a second-order cone contraint
# Enlarge the model
n = n + k + 1
m = m + k + 1

# All the added auxiliary variables do not take part in obj
c = np.append(c, np.zeros(k+1))

# Enlarge bounds on x, add inf bounds on the new added k+1 variables
blx = np.append(blx, np.full(k+1, -1.e20, dtype=float))
bux = np.append(bux, np.full(k+1, 1.e20, dtype=float))

# Enlarge linear constraints
# Sparsity pattern of F (COO)
row, col = np.nonzero(F)
val = F[row, col]

# Convert to 1-based and move row down by m
# Add Fx = y and Phi^-1(beta)*t + r_bar'x - alpha = 0
# [x,t,y]
row = row + 1 + m_up
col = col + 1

row = np.append(row, np.arange(m_up+1, m_up+k+1+1))
col = np.append(np.append(col, np.arange(n_up+2, n_up+k+1+1)), n_up+1)
val = np.append(val, np.append(np.full(k, -1.0, dtype=float), quantile))

irowa = np.append(irowa, row)
icola = np.append(icola, col)
a = np.append(a, val)
bl = np.append(bl, np.append(np.zeros(k), alpha))
bu = np.append(bu, np.append(np.zeros(k), alpha))

# coeffient of x in Phi^-1(beta)*t + r'x - alpha = 0
irowa = np.append(irowa, np.full(n_assets, m_up+k+1, dtype=int))
icola = np.append(icola, np.arange(1, n_assets+1))
a = np.append(a, r)

# Enlarge cone constraints
ctype.extend('Q')
group_temp = np.arange(n_up+1, n_up+k+1+1)
group.append(group_temp)
```

By now, all the data we need is ready. Feed them to the NAG SOCP solver and solve.

```python
# Create problem handle
handle = opt.handle_init(n)

# Set objective function
opt.handle_set_linobj(handle, c)

# Set box constraints
opt.handle_set_simplebounds(handle, blx, bux)

# Set linear constraints
opt.handle_set_linconstr(handle, bl, bu, irowa, icola, a)

# Set cone constraints
i = 0
while i<len(ctype):
    opt.handle_set_group(handle, ctype[i], 0, group[i])
    i += 1

# Set options
for option in [
        'Print Options = NO',
        'Print File = 1',
        'SOCP Scaling = A'
]:
    opt.handle_opt_set(handle, option)

# Call socp interior point solver
slt = opt.handle_solve_socp_ipm(handle)
```

    naginterfaces.base.opt.handle_solve_socp_ipm:
    naginterfaces.base.opt.handle_solve_socp_ipm:  ------------------------------------------------
    naginterfaces.base.opt.handle_solve_socp_ipm:   E04PT, Interior point method for SOCP problems
    naginterfaces.base.opt.handle_solve_socp_ipm:  ------------------------------------------------
    naginterfaces.base.opt.handle_solve_socp_ipm:
    naginterfaces.base.opt.handle_solve_socp_ipm:  Problem Statistics
    naginterfaces.base.opt.handle_solve_socp_ipm:    No of variables                 17
    naginterfaces.base.opt.handle_solve_socp_ipm:      free (unconstrained)           9
    naginterfaces.base.opt.handle_solve_socp_ipm:      bounded                        8
    naginterfaces.base.opt.handle_solve_socp_ipm:    No of lin. constraints          10
    naginterfaces.base.opt.handle_solve_socp_ipm:      nonzeroes                     89
    naginterfaces.base.opt.handle_solve_socp_ipm:    No of quad.constraints           0
    naginterfaces.base.opt.handle_solve_socp_ipm:    No of cones                      1
    naginterfaces.base.opt.handle_solve_socp_ipm:      biggest cone size              9
    naginterfaces.base.opt.handle_solve_socp_ipm:    Objective function          Linear
    naginterfaces.base.opt.handle_solve_socp_ipm:
    naginterfaces.base.opt.handle_solve_socp_ipm:  Presolved Problem Measures
    naginterfaces.base.opt.handle_solve_socp_ipm:    No of variables                 17
    naginterfaces.base.opt.handle_solve_socp_ipm:    No of lin. constraints          10
    naginterfaces.base.opt.handle_solve_socp_ipm:      nonzeroes                     89
    naginterfaces.base.opt.handle_solve_socp_ipm:    No of cones                      1
    naginterfaces.base.opt.handle_solve_socp_ipm:
    naginterfaces.base.opt.handle_solve_socp_ipm:
    naginterfaces.base.opt.handle_solve_socp_ipm:  ------------------------------------------------------------------------
    naginterfaces.base.opt.handle_solve_socp_ipm:   it|    pobj    |    dobj    |  p.inf  |  d.inf  |  d.gap  |   tau  | I
    naginterfaces.base.opt.handle_solve_socp_ipm:  ------------------------------------------------------------------------
    naginterfaces.base.opt.handle_solve_socp_ipm:    0 -1.13221E+00  0.00000E+00  8.43E-01  2.14E-01  4.45E+00  1.0E+00
    naginterfaces.base.opt.handle_solve_socp_ipm:    1 -2.82490E-01 -8.98481E-02  1.91E-01  4.87E-02  1.01E+00  1.3E+00
    naginterfaces.base.opt.handle_solve_socp_ipm:    2 -1.32896E-01 -9.68963E-02  3.73E-02  9.50E-03  1.97E-01  1.4E+00
    naginterfaces.base.opt.handle_solve_socp_ipm:    3 -1.10308E-01 -9.31071E-02  1.83E-02  4.66E-03  9.65E-02  1.4E+00
    naginterfaces.base.opt.handle_solve_socp_ipm:    4 -1.02701E-01 -8.92455E-02  1.10E-02  2.81E-03  5.83E-02  1.1E+00
    naginterfaces.base.opt.handle_solve_socp_ipm:    5 -8.78903E-02 -8.59020E-02  1.62E-03  4.12E-04  8.55E-03  1.1E+00
    naginterfaces.base.opt.handle_solve_socp_ipm:    6 -8.58224E-02 -8.53194E-02  3.89E-04  9.90E-05  2.05E-03  1.0E+00
    naginterfaces.base.opt.handle_solve_socp_ipm:    7 -8.54027E-02 -8.51613E-02  1.71E-04  4.34E-05  9.01E-04  9.6E-01
    naginterfaces.base.opt.handle_solve_socp_ipm:    8 -8.50652E-02 -8.50604E-02  3.51E-06  8.93E-07  1.85E-05  9.7E-01
    naginterfaces.base.opt.handle_solve_socp_ipm:    9 -8.50588E-02 -8.50583E-02  3.86E-07  9.84E-08  2.04E-06  9.7E-01
    naginterfaces.base.opt.handle_solve_socp_ipm:   10 -8.50580E-02 -8.50580E-02  9.88E-09  2.51E-09  5.21E-08  9.7E-01
    naginterfaces.base.opt.handle_solve_socp_ipm:   11 -8.50580E-02 -8.50580E-02  3.19E-10  1.47E-10  8.04E-10  9.7E-01
    naginterfaces.base.opt.handle_solve_socp_ipm:  ------------------------------------------------------------------------------
    naginterfaces.base.opt.handle_solve_socp_ipm:  Status: converged, an optimal solution found
    naginterfaces.base.opt.handle_solve_socp_ipm:  ------------------------------------------------------------------------------
    naginterfaces.base.opt.handle_solve_socp_ipm:  Final primal objective value        -8.505797E-02
    naginterfaces.base.opt.handle_solve_socp_ipm:  Final dual objective value          -8.505797E-02
    naginterfaces.base.opt.handle_solve_socp_ipm:  Absolute primal infeasibility        2.869531E-09
    naginterfaces.base.opt.handle_solve_socp_ipm:  Relative primal infeasibility        3.186829E-10
    naginterfaces.base.opt.handle_solve_socp_ipm:  Absolute dual infeasibility          1.765929E-09
    naginterfaces.base.opt.handle_solve_socp_ipm:  Relative dual infeasibility          1.471607E-10
    naginterfaces.base.opt.handle_solve_socp_ipm:  Absolute complementarity gap         1.266105E-09
    naginterfaces.base.opt.handle_solve_socp_ipm:  Relative complementarity gap         8.044616E-10
    naginterfaces.base.opt.handle_solve_socp_ipm:  Iterations                                     11

Now we can print the optimal portfolio and the corresponding return.

```python
# Optimal portfolio
slt.x[0:n_assets]
```

    array([1.93322779e-10, 3.97165177e-01, 3.35524516e-01, 7.15459782e-10,
           1.16762817e-01, 1.15110232e-10, 8.27684160e-02, 6.77790766e-02])

```python
# Optimal expected return
r.dot(slt.x[0:n_assets])
```

    0.08505797359531912

# Conclusion

In this notebook, we demonstrated how to use the NAG Library to model and solve portfolio optimization with probability constraint via SOCP. It is worth noting that SOCP is widely used in portfolio optimization due to its flexibility and versatility to handle a large variety of problems with different kinds of constraints, not only the probability constraint mentioned above, e.g. leverage constraint, turnover constraint, max position constraint and tracking-error constraint. We refer the readers to \cite{AG03, LVBL98, NAGDOC} for more details.

# References

[1] Alizadeh Farid and Goldfarb Donald, ``_Second-order cone programming_'', Mathematical programming, vol. 95, number 1, pp. 3--51,  2003.

[2] Lobo Miguel Sousa, Vandenberghe Lieven, Boyd Stephen et al., ``_Applications of second-order cone programming_'', Linear algebra and its applications, vol. 284, number 1-3, pp. 193--228,  1998.

[3] Numerical Algorithms Group, ``_NAG documentation_'',  2019.  [online](https://www.nag.com/numeric/fl/nagdoc_latest/html/frontmatter/manconts.html)


# 2 Linear Equations and Computer Basics

```python
#%pylab inline
%pylab notebook
# pylab Populating the interactive namespace from numpy and matplotlib
# numpy for numerical computation
# matplotlib for ploting
```

    Populating the interactive namespace from numpy and matplotlib

    /opt/conda/lib/python3.5/site-packages/IPython/core/magics/pylab.py:161: UserWarning: pylab import has clobbered these variables: ['cos', 'invert', 'diff', 'trace', 'tan', 'sign', 'plotting', 'product', 'diag', 'roots', 'sinc', 'eye', 'zeros', 'sinh', 'poly', 'test', 'flatten', 'sqrt', 'cbrt', 'beta', 'source', 'cosh', 'tanh', 'binomial', 'ones', 'multinomial', 'vectorize', 'trunc', 'exp', 'interactive', 'pi', 'seterr', 'var', 'add', 'floor', 'reshape', 'gamma', 'take', 'conjugate', 'prod', 'sin', 'det', 'Circle', 'mod', 'nan', 'Polygon', 'solve', 'plot', 'transpose', 'log', 'power']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

```python
import numpy as np
from numpy import append, array, diagonal, tril, triu
from numpy.linalg import inv
from numpy.linalg import norm
from numpy.linalg import solve
from scipy.linalg import lu
#from scipy.linalg import solve
from pprint import pprint
from numpy import array, zeros, diag, diagflat, dot

from sympy import *
import sympy as sym
```

```python
init_printing()
```

## 2.1 Linear Equations and the L-U Factorization

The linear equation is the most elementary problem that arises in computational
economic analysis. In a linear equation, an $n \times n$ matrix A and an n-vector b are
given, and one must compute the n-vector x that satisfies

$$A x = b$$

Most linear equations encountered in practice, however, do not have a triangular A matrix. In such cases, the linear equation is often best solved using the L-U factorization algorithm. The L-U algorithm is designed to decompose the A matrix into the product of lower and upper triangular matrices, allowing the linear equation to be solved using a combination of backward and forward substitution.

The L-U factorization algorithm is faster than other linear equation solution methods that are typically presented in elementary linear algebra courses.

```python
# example problem

A = np.array([[-3.0, 2.0, 3.0], [-3.0, 2.0, 1.0], [3.0, 0, 0]])

b = [10, 8, -3]

x0 = [1.0, 1.0, 1.0]

n = 25

print( "A:")
pprint(A)

#rint( "diag(A):")
#print((diag(A)))

#rint("pprint(diagflat(diag(A)))")
#print(diagflat(diag(A)))

print( "b:")
pprint(b)

print( "x0:")
pprint(x0)

print( "n:")
pprint(n)

```

    A:
     [[-3.  2.  3.]
      [-3.  2.  1.]
     [ 3.  0.  0.]]
    b:
    [10, 8, -3]
    x0:
    [1.0, 1.0, 1.0]
    n:
    25

### Using Sympy

```python
M = Matrix(A)
v = Matrix(b)
```

```python
M
```

$$\left[\begin{matrix}-3.0 & 2.0 & 3.0\\-3.0 & 2.0 & 1.0\\3.0 & 0.0 & 0.0\end{matrix}\right]$$

```python
v
```

$$\left[\begin{matrix}10\\8\\-3\end{matrix}\right]$$

http://docs.sympy.org/latest/modules/matrices/matrices.html

Returns (L, U, perm) where L is a lower triangular matrix with unit diagonal, U is an upper triangular matrix, and perm is a list of row swap index pairs. If A is the original matrix, then A = (L*U).permuteBkwd(perm), and the row permutation matrix P such that P*A = L*U can be computed by P=eye(A.row).permuteFwd(perm).

See documentation for LUCombined for details about the keyword argument rankcheck, iszerofunc, and simpfunc.

```python
M.LUdecomposition()
```

$$\left ( \left[\begin{matrix}1 & 0 & 0\\-1.0 & 1 & 0\\1.0 & 0 & 1\end{matrix}\right], \quad \left[\begin{matrix}-3.0 & 2.0 & 3.0\\0 & 2.0 & 3.0\\0 & 0 & -2.0\end{matrix}\right], \quad \left [ \left [ 2, \quad 1\right ]\right ]\right )$$

```python
L,U,perm = M.LUdecomposition()

```

```python
L
```

$$\left[\begin{matrix}1 & 0 & 0\\-1.0 & 1 & 0\\1.0 & 0 & 1\end{matrix}\right]$$

```python
U
```

$$\left[\begin{matrix}-3.0 & 2.0 & 3.0\\0 & 2.0 & 3.0\\0 & 0 & -2.0\end{matrix}\right]$$

```python
perm
```

```python
L*U
```

$$\left[\begin{matrix}-3.0 & 2.0 & 3.0\\3.0 & 0 & 0\\-3.0 & 2.0 & 1.0\end{matrix}\right]$$

```python
(L*U).permuteBkwd(perm)
```

$$\left[\begin{matrix}-3.0 & 2.0 & 3.0\\-3.0 & 2.0 & 1.0\\3.0 & 0 & 0\end{matrix}\right]$$

```python
v = v.permuteBkwd(perm)
v
```

$$\left[\begin{matrix}10\\-3\\8\end{matrix}\right]$$

the elements of x can be computed recursively using forward-substitution

```python
L.LUsolve(v)
```

$$\left[\begin{matrix}10\\7.0\\-2.0\end{matrix}\right]$$

```python

```

If A is an upper triangular matrix, then the elements of x can be computed recursively using backward-substitution.

```python
U.LUsolve(L.LUsolve(v))
```

$$\left[\begin{matrix}-1.0\\2.0\\1.0\end{matrix}\right]$$

```python

```

```python
y= b
b = [10, -3, 8]
y[0] = b[0]/L[0,0]
for i in range(len(b)-1):
    y[i+1] = (b[i+1] - L[i+1,:i+1].dot(y[:i+1]))/L[i+1,i+1]

y
```

```python

```

```python
L.dot([10, 7.0 , -2.0])
```

### Using Numpy

```python

```

```python
"""
Solve a linear equation by LU-decomposition
Comes from LU decomposition of a matrix A s.t. A = LU
Then
LUx = b => Ux = y => Ly = b
"""
p, l, u = lu(A)
```

```python
p

```

    array([[ 1.,  0.,  0.],
           [ 0.,  0.,  1.],
           [ 0.,  1.,  0.]])

```python
l@u
```

    array([[-3.,  2.,  3.],
           [ 3.,  0.,  0.],
           [-3.,  2.,  1.]])

```python
p@l@u
```

    array([[-3.,  2.,  3.],
           [-3.,  2.,  1.],
           [ 3.,  0.,  0.]])

Lower triangle matrix

```python
l
```

    array([[ 1.,  0.,  0.],
           [-1.,  1.,  0.],
           [ 1.,  0.,  1.]])

```python
p@l
```

    array([[ 1.,  0.,  0.],
           [ 1.,  0.,  1.],
           [-1.,  1.,  0.]])

```python
b=p@l.dot([10, 7, -2])
b
```

    array([ 10.,   8.,  -3.])

Forward solving

```python
y = np.linalg.solve(p@l, b)
y
```

    array([ 10.,   7.,  -2.])

Upper triangle matrix

Backward solving

```python
x = np.linalg.solve(u, y)
x
```

    array([-1.,  2.,  1.])

```python
## Other example

A = np.array([[54, 14, -11, 2],
               [14, 50, -4, 29],
               [-11, -4, 55, 22],
               [2, 29, 22, 95]])
print("Left hand side of linear system, matrix A:\n", A)
b = np.array([1, 1, 1, 1])
x0 = [1, 1, 1,1]

n = 20

p, l, u = lu(A)

print("perm matrix:\n",p)

print("Lower matrix:\n",l)

print("Upper matrix\n",u)

print("Right hand side of linear equation system:\n",b)

y = np.linalg.solve(l, b)

print("Intermedia output matrix y:\n",y)

x = np.linalg.solve(u, y)
print("Solution x:\n",x)
```

    Left hand side of linear system, matrix A:
     [[ 54  14 -11   2]
     [ 14  50  -4  29]
     [-11  -4  55  22]
     [  2  29  22  95]]
    perm matrix:
     [[ 1.  0.  0.  0.]
     [ 0.  1.  0.  0.]
     [ 0.  0.  1.  0.]
     [ 0.  0.  0.  1.]]
    Lower matrix:
     [[ 1.          0.          0.          0.        ]
     [ 0.25925926  1.          0.          0.        ]
     [-0.2037037  -0.02476038  1.          0.        ]
     [ 0.03703704  0.61421725  0.43831321  1.        ]]
    Upper matrix
     [[ 54.          14.         -11.           2.        ]
     [  0.          46.37037037  -1.14814815  28.48148148]
     [  0.           0.          52.73083067  23.11261981]
     [  0.           0.           0.          67.30154198]]
    Right hand side of linear equation system:
     [1 1 1 1]
    Intermedia output matrix y:
     [ 1.          0.74074074  1.22204473 -0.02765113]
    Solution x:
     [ 0.01893441  0.01680508  0.02335523 -0.00041085]

```python

```

```python

```

## 2.2 Gaussian Elimination

The L-U factors of a matrix A are computed using Gaussian elimination. Gaussian
elimination is based on two elementary row operations: subtracting a constant multiple
of one row of a linear equation from another row, and interchanging two rows of
a linear equation. Either operation may be performed on a linear equation without
altering its solution.

The Gaussian elimination algorithm begins with matrices L and U initialized as
L = I and U = A, where I is the identity matrix. The algorithm then uses elementary
row operations to transform U into an upper triangular matrix, while preserving the
permuted lower diagonality of L and the factorization A = LU:

```python

```

## 2.3 Rounding Error

In practice, Gaussian elimination performed on a computer can sometimes render
inaccurate solutions due to rounding errors. The eects of rounding errors, however,
can often be controlled by pivoting.

```python

```

## 2.4 Ill Conditioning

Pivoting cannot cure all the problems caused by rounding error. Some linear equations
are inherently diÆcult to solve accurately on a computer, despite pivoting. This
occurs when the A matrix is structured in such a way that a small perturbation Æb
in the data vector b induces a large change Æx in the solution vector x. In such cases
the linear equation or, more generally, the A matrix are said to be **ill-conditioned**.

```python

```

## 2.5 Special Linear Equations

Gaussian elimination can be accelerated for matrices possessing certain special structures.
Two such classes arising frequently in computational economic analysis are
**symmetric positive deffinite matrices and sparse matrices**.

```python

```

### Cholesky factorization algorithm

A special form
of Gaussian elimination, the Cholesky factorization algorithm, may be applied to
such linear equations. Cholesky factorization requires only half as many operations
as general Gaussian elimination and has the added advantage that it is less vulnerable
to rounding error and does not require pivoting.
The essential idea underlying Cholesky factorization is that any symmetric positive
deffinite matrix A can be uniquely expressed as the product

$$\mathbf {A=U^T U}$$

$${\displaystyle {\begin{aligned}\left({\begin{array}{*{3}{r}}4&12&-16\\12&37&-43\\-16&-43&98\\\end{array}}\right)=\left({\begin{array}{*{3}{r}}2&0&0\\6&1&0\\-8&5&3\\\end{array}}\right)\left({\begin{array}{*{3}{r}}2&6&-8\\0&1&5\\0&0&3\\\end{array}}\right)\end{aligned}}}$$

$$\mathbf {A s=U^T U x = U^T (U x)     = b}$$

may be solved eÆciently by using forward substitution to solve

$$\mathbf { U^T y = b}$$

and then using backward substitution to solve

$$\mathbf { U x = y}$$

```python

```

https://en.wikipedia.org/wiki/Cholesky_decomposition
In linear algebra, the Cholesky decomposition or Cholesky factorization is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose, which is useful e.g. for efficient numerical solutions and Monte Carlo simulations. It was discovered by André-Louis Cholesky for real matrices. When it is applicable, the Cholesky decomposition is roughly twice as efficient as the LU decomposition for solving systems of linear equations.

## 2.6 Iterative Methods

Algorithms based on Gaussian elimination are called **exact** or, more properly, **direct**
methods because they would generate exact solutions for the linear equation $Ax = b$
after a finite number of operations, if not for rounding error. Such methods are ideal
for moderately-sized linear equations, but may be impractical for large ones. Other
methods, called **iterative methods** can often be used to solve large linear equations
more efficiently if the A matrix is sparse, that is, if $A$ is composed mostly of zero
entries. Iterative methods are designed to generate a sequence of increasingly accurate
approximations to the solution of a linear equation, but generally do not yield an exact
solution after a prescribed number of steps, even in theory.

The most widely-used iterative methods for solving a linear equation $Ax = b$ are
developed by choosing an easily invertible matrix $Q$ and writing the linear equation
in the equivalent form

$$Q x = b + (Q - A)x$$

or

$$x = Q^{-1}b + (I - Q^{-1}A)x.$$

This form of the linear equation suggests the iteration rule

$$x^{(k+1)} \leftarrow   Q^{-1}b + (I - Q^{-1}A)x^{(k)}$$

Ideally, the so-called splitting matrix Q will satisfy two criteria. First, $Q^{-1}b$ and
$Q^{-1}A$ should be relatively easy to compute. This is true if Q is either diagonal or
triangular. Second, the iterates should converge quickly to the true solution of the
linear equation. If

$$||I - Q^{-1}A|| < 1$$

in any matrix norm, then the iteration rule is a contraction mapping and is guaranteed
to converge to the solution of the linear equation from any initial value.

```python

```

```python

```

The two most popular iterative methods are the **Gauss-Jacobi** and **Gauss-Seidel**
methods.

 - The Gauss-Jacobi method sets $Q$ equal to the diagonal matrix formed from the diagonal entries of A.

 - The Gauss-Seidel method sets $Q$ equal to the upper triangular matrix formed from the upper triagonal elements of A.

### Jacobi-Method

Here, the user specifies the data A and b and an initial guess $x$ for the solution of
the linear equation, typically the zero vector or $b$. Iteration continues until the norm
of the change dx in the iterate falls below the specified convergence tolerance tol or
until a specified maximum number of allowable iterations maxit are performed.

```python
#p17

maxit = 1000
tol = 1/10000
x0 = np.array([1,1,1,1])
x0= b

d = np.diag(np.diag(A))
x = x0
for it in np.arange(maxit):
    dx =  inv(d).dot(b - A.dot(x))
    x = x + dx
    if np.linalg.norm(dx)<tol:
        break

print("x:\n", x)
```

    x:
     [ 0.01892514  0.01678369  0.02335003 -0.00042463]

```python
def gjacobi(A, b, maxit=1000,tol = 1/1000, x0=None):
    """
    Solve a linear equation by the gauss jacobi iteration outlined in the book.
    Follows the eq:
        x = inv(D)(b - Rx)
    Where D is the diagonal matrix of A and R is the remainder s.t D + R = A
    """
    # If we have not provided an initial array for x make a new one
    if x0==None:
        x = array([1 for _ in range(A.shape[1])])
    else:
        x = x0
    d = np.diag(np.diag(A))

    for it in np.arange(maxit):
        dx =  inv(d).dot(b - A.dot(x))
        x = x + dx
        if np.linalg.norm(dx)<tol:
            break

    print("x:\n", x)
```

```python
gjacobi(A, b, maxit,tol, x0=None)
```

    x:
     [ 0.01892514  0.01678369  0.02335003 -0.00042463]

### Jacobi-Method-1

The Jacobi method is a matrix iterative method used to solve the equation $Ax=b$ for a known square matrix A of size n×n and known vector b or length n.

$$\begin{eqnarray*}
Ax = b
\end{eqnarray*}$$

A is split into the sum of two separate matrices, $D$ and $R$, such that $A=D+R$. $D_{ii}=A_{ii}$, but $D_{ij}=0$, for $i≠j$. $R$ is essentially the opposite. $R_{ii}=0$, but $R-{ij}=A_{ij}$ for $i≠j$. The solution to the equation, i.e. the value of $x$

, is given by the following iterative equation:

$$\begin{eqnarray*}
x^{(k+1)} = D^{-1}(b-Rx^{(k)}).
\end{eqnarray*}$$

https://www.quantstart.com/articles/Jacobi-Method-in-Python-and-NumPy

```python
def gauss_jacobi(A, b, maxit, x0=None):
    """
    Solve a linear equation by the gauss jacobi iteration outlined in the book.
    Follows the eq:
        x = inv(D)(b - Rx)
    Where D is the diagonal matrix of A and R is the remainder s.t D + R = A
    """
    D = np.diag(np.diag(A))
    # Calculate the remainder matrix
    R = A - D
    # If we have not provided an initial array for x make a new one
    if x0==None:
        x = array([1 for _ in range(A.shape[1])])
    else:
        x = x0
    for _ in range(maxit):
        x = inv(d).dot(b - R.dot(x))
    return x

```

```python
gauss_jacobi(A, b, maxit, x0)
```

    /opt/conda/lib/python3.5/site-packages/ipykernel/__main__.py:12: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.

    array([ 0.01893441,  0.01680508,  0.02335523, -0.00041085])

Check result with direct method

```python
np.linalg.solve(A,b)
```

    array([ 0.01893441,  0.01680508,  0.02335523, -0.00041085])

### gauss seidel method

https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html

```python
A = np.array([[3, 1],
               [2, 5]])
b = np.array([7 , 9])
```

```python

```

```python
Q = np.tril(A)

x=b
lamb = 1.5

for it in np.arange(maxit):
    dx = np.linalg.inv(Q).dot(b -A.dot(x))
    x = x + lamb*dx
    if np.linalg.norm(dx)<tol:
        break
print("x:\n", x)
```

    x:
     [ 1.99995232  1.        ]

```python

def gseidel(A, b, maxit=1000, tol=1/1000,lamb =1, x0=None):
    """
    Solve a linear equation by the gauss seidel iteration outlined in the book
    Follows the eq:

    """
    Q = np.tril(A)
    # If we have not provided an initial array for x make a new one
    if x0 == None:
        x = array([1 for _ in range(A.shape[1])])
    else:
        x = x0
    for it in np.arange(maxit):
        dx = np.linalg.inv(Q).dot(b -A.dot(x))
        x = x + lamb*dx
        if np.linalg.norm(dx)<tol:
            break
    print("x:\n", x)
    return x

gseidel(A,b)
```

    x:
     [ 2.  1.]

    array([ 2.,  1.])

Here, we have incorporated a so-called over-relaxation parameter, lambda. Instead of using $x + dx$, we use $x + lamb*x$ to compute the next iterate.

a value of lambda between 1 and 2 will accelerate convergence of the Gauss-Seidel algorithm.

```python

```

### gauss seidel method

http://austingwalters.com/gauss-seidel-method/

Using the Gauss-Seidel Method

The method is fairly straight forward, given a standard system of linear equations, Ax = b. Where, A is a matrix (often representing a series of equations), x is a vector of x variables (Gauss-Seidel method is used to solve this vector) and b is the solution vector. In Gauss-Seidel method, we then split the A matrix into Upper (U) and Lower (L) matrices (the lower matrix in this case also contains the diagonal), then iterate using the following method:

```python
"""
author:  hahnicity
https://github.com/hahnicity/ace/blob/master/ace/solvers.py
Solve a simple linear equation Ax = b in a particular way
"""

def gauss_seidel(A, b, maxit=1000, x0=None):
    """
    Solve a linear equation by the gauss seidel iteration outlined in the book
    Follows the eq:
        x = inv(L)*(b - U*x)
    """
    l = tril(A)
    upper_plus_diagonal = triu(A)
    u = upper_plus_diagonal - np.diag(np.diag(A))
    # If we have not provided an initial array for x make a new one
    if x0 == None:
        x = array([1 for _ in range(A.shape[1])])
    else:
        x = x0

    for _ in np.arange(maxit):
        x = inv(l).dot(b - u.dot(x))
    return x

gauss_seidel(A, b)

```

    array([ 2.,  1.])

The following script solves a linear equation using Gauss-Seidel
iteration with default value of 1 for the over-relaxation parameter:

```python
A = np.array([[3, 1],
               [2, 5]])
b = np.array([7 , 9])
x = gauss_seidel(A,b,maxit)
print("x:\n",x)
```

    x:
     [ 2.  1.]

When A=[3 2; 4 1], however,
the algorithm diverges.

```python
A = np.array([[3, 2],
               [4, 1]])
b = np.array([7 , 9])
x = gauss_seidel(A,b,maxit)
print("x:\n",x)
```

    x:
     [ nan  nan]

A general rule of thumb is that if A is large and sparse, then the linear equation is
a good candidate for iterative methods, provided that sparse matrix storage functions
are used to reduce storage requirements and computational effort. Iterative methods,
however, have some drawbacks. First, iterative methods, in contrast to direct
methods, can fail to converge. Furthermore, it is often difficult or computationally
costly to check whether a specific problem falls into a class of problems known to
be convergent. It is therefore always a good idea to monitor whether the iterations
seem to be diverging and try something else if they are. Second, satisfaction of the
termination criteria do not necessarily guarantee a similar level of accuracy in the
solution, as measured as the deviation of the approximate solution from the true (but
unknown) solution.

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## 2.1 Computer Arithmetic

Some knowledge of how computers perform numerical computations and how programming
languages work is useful in applied numerical work, especially if one is to
write efficient programs and avoid errors. It often comes as an unpleasant surprise to
many people to learn that exact arithmetic and computer arithmetic do not always
give the same answers, even in programs without programming errors.

Typically, computer languages such as Fortran and C allow several ways of representing
a number.

The exact details of the representation depends on the hardware but it will
suffice for our purposes to suppose that floating point numbers are stored in the form
$m2^e$, where m and e are integers with $-2^b <= m <2^b$ and $-2^d <= e < 2^d$.

The obvious way of computing this term will result in loss of precision.

These arise not only from overflow but from division by 0.

In addition,floating point numbers may get set to $NaN$, which stands for not-anumber.

Roundoff error is only one of the pitfalls in evaluating mathematical expressions.
In numerical computations, error is also introduced by the computer's inherent inability
to evaluate certain mathematical expressions exactly. For all its power, a computer
can only perform a limited set of operations in evaluating expressions. Essentially this
list includes the four arithmetic operations of addition, subtraction, multiplication
and division, as well as logical operations of comparison.

Other common functions,such as exponential, logarithmic, and trigonometric functions cannot be evaluated directly using computer arithmetic. They can only be evaluated approximately using algorithms based on the four basic arithmetic operations.

For the common functions very efficient algorithms typically exist and these are
sometimes "hardwired" into the computer's processor or coprocessor. An important
area of numerical analysis involves determining efficient approximations that can be
computed using basic arithmetic operations.

$$exp(x) = \sum^{inf}_{i=0} x^{n}/n!$$

Obviously one cannot compute the infinite sum, but one could compute a finite number
of these terms, with the hope that one will obtain sufficient accuracy for the
purpose at hand. The result, however, will always be inexact.

```python

```

##  Bibliographic Notes

```python

```

```python

```

```python

```

```python

```

https://github.com/mmcky/nyu-econ-370/blob/master/notebooks/notes-linear-algebra.ipynb

https://www3.nd.edu/~zxu2/acms40390F12/Lec-7.3.pdf

http://austingwalters.com/gauss-seidel-method/

https://stackoverflow.com/questions/17580666/improving-numpy-speed-for-gauss-seidel-jacobi-solver


## First Assignment

#### 1) Apply the appropriate string methods to the **x** variable (as '.upper') to change it exactly to: "$Dichlorodiphenyltrichloroethane$".

```python
x = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"
```

```python
xx= x.replace("DiClOrod  IFeNi  lTRicLOr oETaNo","")
title= xx.replace(" ","")
title.title()

```

    'Dichlorodiphenyltrichloroethane'

#### 2) Assign respectively the values: 'word', 15, 3.14 and 'list' to variables A, B, C and D in a single line of code. Then, print them in that same order on a single line separated by a space, using only one print statement.

```python
exercise2 = {'A': 'word', 'B': 15, 'C':3.14, 'D': 'list'}
print(exercise2['A']," ", exercise2['B']," ", exercise2['C']," ",exercise2['D'])

```

    word   15   3.14   list

#### 3) Use the **input()** function to receive an input in the form **'68.4 1.71'**, that is, two floating point numbers in a line separated by space. Then, assign these numbers to the variables **w** and **h** respectively, which represent an individual's weight and height (hint: take a look at the '.split()' method). With this data, calculate the individual's Body Mass Index (BMI) from the following relationship:

\begin{equation}
BMI = \dfrac{weight}{height^2}
\end{equation}

```python
wh = input()
wh= wh.split()
h = float(wh[1])
w = float(wh[0])
BMI = w / h ** 2
print(BMI)

```

    68.4 1.71
    23.39181286549708

#### This value can also be classified according to ranges of values, following to the table below. Use conditional structures to classify and print the classification assigned to the individual.

<\center>

(source: https://healthtravelguide.com/bmi-calculator/)

```python
BMI= round(BMI,2)
if BMI < 18.5:
    print(BMI,'underweight')
if BMI >= 18.5 and BMI < 24.9:
    print(BMI,'Normal Weight')
if BMI >= 24.9 and BMI < 30.0:
    print(BMI,'Pre-Obseity')
if BMI >= 30.0 and BMI < 34.9:
    print(BMI,'Obesity Class I')
if BMI >= 34.9 and BMI <= 39.9:
    print(BMI,'Obesity Class II')
if BMI >= 40.00:
    print(BMI,'Obesity Class III')

```

    23.39 Normal Weight

#### 4) Receive an integer as an input and, using a loop, calculate the factorial of this number, that is, the product of all the integers from one to the number provided.

```python
num = int(input("enter a number: "))
fac = 1
for x in range(1, num +1 ):
    fac = fac * x

print("factorial of ", num, " is ", fac)

```

    enter a number: 8
    factorial of  8  is  40320

#### 5) Using a while loop and the input function, read an indefinite number of integers until the number read is -1. Present the sum of all these numbers in the form of a print, excluding the -1 read at the end.

```python
x = -5
while  x < -1:
    print(x)
    x += 1
```

    -5
    -4
    -3
    -2

#### 6) Read the **first name** of an employee, his **amount of hours worked** and the **salary per hour** in a single line separated by commas. Next, calculate the **total salary** for this employee and show it to two decimal places.

```python
employee = ['John Doe', 37.7, 15.3]
salary = employee[1] * employee[2]
round(salary,2)

```

    576.81

#### 7) Read three floating point values **A**, **B** and **C** respectively. Then calculate itens a, b, c, d and e:

```python
A = float(input())
B = float(input())
C = float(input())
```

    -9.5
    -8.3
    7.2

    a) the area of the triangle rectangle with A as the base and C as the height.

```python
area = A * B /2
area
```

    39.425000000000004

    b) the area of the circle of radius C. (pi = 3.14159)

```python
area_circle = 3.14159 * C**2
area_circle
```

    162.8600256

    c) the area of the trapezoid that has A and B for bases and C for height.

```python
trapezoid = A * C / 2  + B *C / 2
trapezoid
```

    -64.08000000000001

    d) the area of the square that has side B.

```python
 area_square = B*2
 area_square
```

    -16.6

    e) the area of the rectangle that has sides A and B.

```python
rectangle = A*B
rectangle
```

    78.85000000000001

#### 8) Read **the values a, b and c** and calculate the **roots of the second degree equation** $ax^{2}+bx+c=0$ using [this formula](https://en.wikipedia.org/wiki/Quadratic_equation). If it is not possible to calculate the roots, display the message **“There are no real roots”**.

```python

if ((B**2)-4*A*C) < 0:
  print("There are no real roots")
else:
  x1 = (-B+math.sqrt(B**2-(4*A*C)))/(2*A)
  x2 = (-B-math.sqrt(B**2-(4*A*C)))/(2*A)
  print("Solutions:")
  print(x1)
  print(x2)
```

    Solutions:
    -1.410867649986115
    0.5371834394597992

#### 9) Read four floating point numerical values corresponding to the coordinates of two geographical coordinates in the cartesian plane. Each point will come in a line with its coordinates separated by space. Then calculate and show the distance between these two points.

(obs: $d=\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2}$)

```python
x  = input()
y = input()
x = x.split()
y = y.split()

distance = ((float(x[0]) - float(x[1]))**2 + (float(y[ 0]) - float(y[1]))**2)**0.5
print('The distance between the points is:',distance)
```

    15.5 13.8
    8.9 17.7
    The distance between the points is: 8.96270048590267

#### 10) Read **two floating point numbers** on a line that represent **coordinates of a cartesian point**. With this, use **conditional structures** to determine if you are at the origin, printing the message **'origin'**; in one of the axes, printing **'x axis'** or **'y axis'**; or in one of the four quadrants, printing **'q1'**, **'q2**', **'q3'** or **'q4'**.

```python
Cor  = input()
Cor = Cor.split()

if float(Cor[0]) > 0 and float(Cor[1]) > 0:
    print('X and Y are at First quadrant')

elif float(Cor[0])  < 0 and float(Cor[1]) > 0:
    print('X and Y are at Second quadrant')

elif float(Cor[0])  < 0 and float(Cor[1]) < 0:
    print('X and Y are at Third quadrant')

elif float(Cor[0])  > 0 and float(Cor[1]) < 0:
    print('X and Y are at Fourth quadrant')
else:
    print('X and Y are at Origin')

```

    -1.4 -8.6
    X and Y are at Third quadrant

#### 11) Read an integer that represents a phone code for international dialing.
#### Then, inform to which country the code belongs to, considering the generated table below:
(You just need to consider the first 10 entries)

```python
import pandas as pd
df = pd.read_html('https://en.wikipedia.org/wiki/Telephone_numbers_in_Europe')[1]
df = df.iloc[:,:2]
df.head(20)

```

    C:\Users\Leox\Anaconda3\lib\site-packages\pandas\compat\_optional.py:106: UserWarning: Pandas requires version '1.2.1' or newer of 'bottleneck' (version '1.2.0' currently installed).
      warnings.warn(msg, UserWarning)

      Country
      Country calling code

      0
      Austria
      43

      1
      Belgium
      32

      2
      Bulgaria
      359

      3
      Croatia
      385

      4
      Cyprus
      357

      5
      Czech Republic
      420

      6
      Denmark
      45

      7
      Estonia
      372

      8
      Finland
      358

      9
      France
      33

      10
      Germany
      49

      11
      Greece
      30

      12
      Hungary
      36

      13
      Iceland
      354

      14
      Ireland
      353

      15
      Italy
      39

      16
      Latvia
      371

      17
      Liechtenstein
      423

      18
      Lithuania
      370

      19
      Luxembourg
      352

```python
code = input()
countries= {'43':'Austria', '32':'Belgium', '359':'Bulgaria', '358':'Croatia',
            '357':'Cyprus', '420':'Czech Republic', '45':'Denmark',
            '372':'Estonia', '358':'Finland', '33':'France'}
if code in countries:
    print( 'The countrie is:', countries[code])
else:
    print('The code is not in the list')
```

    357
    The countrie is: Cyprus

#### 12) Write a piece of code that reads 6 numbers in a row. Next, show the number of positive values entered. On the next line, print the average of the values to one decimal place.

```python
numbers = [12, 22, 55, 48,31,62]
def average(num):
    sum_num = 0
    for t in num:
        sum_num = sum_num + t

    avg = sum_num / len(num)
    return avg

print("The average is", round(average(numbers), 1))

```

    The average is 38.3

#### 13) Read an integer **N**. Then print the **square of each of the even values**, from 1 to N, including N, if applicable, arranged one per line.

```python
for N in range(1,9):
    if N % 2 == 0:
        N = N**3
        print(N)

```

    8
    64
    216
    512

 14) Using **input()**, read an integer and print its classification as **'even / odd'** and **'positive / negative'** . The two classes for the number must be printed on the same line separated by a space. In the case of zero, print only **'null'**.

```python

my_input = int(input())
def even_odd(num):
    if num % 2 == 0 and num != 0:
        return ('even')
    if num % 2 != 0:
        return ('odd')
    if num % 2 == 0:
        return ('0')

def negative_positive(num):
    if num < 0 and num != 0:
        return ('negative')
    if num > 0 and num != 0:
        return ('positive')
    if num == 0:
        return('')

print(even_odd(my_input), negative_positive (my_input))

```

    8
    even positive

## Challenge
#### 15) Ordering problems are recurrent in the history of programming. Over time, several algorithms have been developed to fulfill this function. The simplest of these algorithms is the [**Bubble Sort**](https://en.wikipedia.org/wiki/Bubble_sort), which is based on comparisons of elements two by two in a loop of passes through the elements. Your mission, if you decide to accept it, will be to input six whole numbers ramdonly ordered. Then implement the **Bubble Sort** principle to order these six numbers **using only loops and conditionals**.
#### At the end, print the six numbers in ascending order on a single line separated by spaces.

```python

mylist = [ input(), input(),input(),input(),input(),input()]
separator = " "

stop= False
while stop == False:
    stop = True
    for n in range(len(mylist) -1):
        if mylist[n] > mylist[n+1]:
            iterated = mylist[n]
            mylist[n] = mylist[n+1]
            mylist[n+1] = iterated
            stop = False

print(separator.join(map(str, mylist)))
```

    6
    8
    0
    9
    -5
    2
    -5 0 2 6 8 9

```python

```

```python

```

```python

```

```python

```


# 14 Partial Differential Equations — 3 (Movie Making)
This notebook is based on `14_PDEs-2.ipynb` and includes code to make movies of the relaxation of the solutions. It is included in case students are interested in how to accomplish simple movie making.

## Solving Laplace's or Poisson's equation

Still solving **Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

General solution by iteration:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

```python
def Laplace_Jacobi_slow(Phi):
    # Don't use, very slow AND inefficient
    Phi_new = Phi.copy()
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi_new[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                  + Phi[xi, yj+1] + Phi[xi, yj-1])
    Phi[:, :] = Phi_new
    return Phi
```

Fast implementation using numpy array operations (vectorized, run at C speed, not Python speed) (see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/16_PDEs/16_PDEs-2-LectureNotes.pdf) or [local PDF](16_PDEs-2-LectureNotes.pdf) for an explanation for how to set up the numpy array operations that give you a 100-fold speed up over `Laplace_Jacobi_slow()`):

```python
def Laplace_Jacobi(Phi):
    """One update in the Jacobi algorithm"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2])
    return Phi
```

```python
def Laplace_Gauss_Seidel(Phi):
    """One update in the Gauss-Seidel algorithm"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
    return Phi
```

We only use the fastest methods at our disposal in this notebook, namely the "checker-board" fast Gauss-Seidel one:

#### Fast Gauss-Seidel-like

Divide the lattice into a checkerboard of black and white cells. Update the odd cells first (like Jacobi) but then use the odd cells to update the even ones (Gauss-Seidel-like; see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/16_PDEs/16_PDEs-2-LectureNotes.pdf) or [local PDF](16_PDEs-2-LectureNotes.pdf) ). This leads to faster convergence *and* can be done with numpy array operations.

```python
def Laplace_Gauss_Seidel_odd_even(Phi):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2] + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2])
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2] + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2])

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2] + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2])
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2] + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2])
    return Phi
```

### *Converged* solution of the wire-in-a-box problem

Solve the box-wire problem and **make sure that the solution is converged to  `tol = 1e-3`.**

Check convergence with the [_Frobenius Norm_](http://mathworld.wolfram.com/FrobeniusNorm.html):

$$
||\mathsf{\Phi}|| := \sqrt{\sum_{i,j}|\Phi_{ij}|^2}
$$

which is implemented as [numpy.linalg.norm()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html) when the argument is a matrix.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

For movie making we do not need interactive plots (and in fact we will disable plot output) but for convenience we enable the standard in-line plotting:

```python
%matplotlib inline
```

#### Convience plotting functions.
If you provide a filename then output is only written to a file and figures are close to conserve memory. This allows you to plot files in loops and later assemble them into movies using other programs such as ffmpeg, ImageMagick, mencoder, QuickTime 7, ...

```python
def plot_contour(Phi, filename=None, zlabel=r"potential $\Phi$ (V)",
                 cmap=plt.cm.coolwarm):
    """Plot Phi as a contour plot.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    cmap : colormap
          pick one from matplotlib.cm
    """
    fig = plt.figure(figsize=(5,4))
    ax = fig.add_subplot(111)

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]
    cset = ax.contourf(X, Y, Z, 20, cmap=cmap)
    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_aspect(1)

    cb = fig.colorbar(cset, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax

def plot_surf(Phi, filename=None, offset=-20, zlabel=r'potential $\Phi$ (V)',
             elevation=40, azimuth=-65, cmap=plt.cm.coolwarm):
    """Plot Phi as a 3D plot with contour plot underneath.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    cmap : colormap
          pick one from matplotlib.cm
    """

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
    surf = ax.plot_surface(X, Y, Z, cmap=cmap, alpha=0.6)
    cset = ax.contourf(X, Y, Z, 20, zdir='z', offset=offset+Z.min(), cmap=cmap)

    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Z.min(), Z.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

#### Wire problem

Use the fastest Poisson solver that we have at the moment: `Laplace_Gauss_Seidel_odd_even()`

```python
Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

Plot the result and visualy compare to what we had before:

```python
plot_contour(Phi)
```

```python
plot_surf(Phi)
```

#### Make a movie showing convergence

The following solution can also write out intermediate solution steps as PNG graphics (commented out for speed). Filenames and directory names are hard-coded.(Important: `plt.ioff()` to disable interactive plotting in the notebook frontend, otherwise it is very slow.)

Put all images into a directory `wire`:

```python
import os, shutil

def mk_new_dir(dirname):
    """Create new directory (remove existing one)"""
    if os.path.exists(dirname):
        shutil.rmtree(dirname)
        print("Removed directory {}".format(dirname))
    os.mkdir(dirname)
    print("Created empty directory {}".format(dirname))

mk_new_dir("./wire")
```

    Created empty directory ./wire

Make a movie by saving an image of the potential every 100 steps.

```python
plt.ioff()  # suppress interactive plotting

nsave = 100

Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
        break
    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

    <matplotlib.pyplot._IonContext at 0x7fc90e580550>

```python
ax = plot_surf(Phi, elevation=40, azimuth=-65)
```

## Successive Over Relaxation (SOR)
Accelerate convergence with the scheme

\begin{align}
r_{i, j} &= \Phi_{i,j}^\text{new} - \Phi_{i, j}^\text{old}\\
\Phi_{i,j}^\text{new} &= \Phi_{i,j}^\text{old} + \omega r_{i,j}
\end{align}

where the new solution is computed with the Gauss-Seidel scheme.

Values of $1 \leq \omega \leq 2$ may work well, $\omega > 2$ can lead to numerical instabilities. Experiment!

### Solve Wire-in-a-box with SOR

Run to convergence (tol = 1e-03). Record the number of steps required (and visually check the solution).
* Start with $\omega = 1$. What should you get?
* Change $\omega$ and try to get faster convergence? Who can get it to converge in the fewest number of steps?

```python
Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual  # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3797 iterations to 0.0009987117944452034

```python
plot_surf(Phi)
```

### Movie of SOR

Create images in directory `./wire_SOR`

```python
mk_new_dir("./wire_SOR")
mk_new_dir("./wire_SOR_surf")
```

    Created empty directory ./wire_SOR
    Created empty directory ./wire_SOR_surf

```python
# %%timeit
plt.ioff()

nsave = 10

Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, "wire_SOR/phi_{0:08d}.png".format(n_iter))
        plot_surf(Phi, "./wire_SOR_surf/phi_{0:08d}.png".format(n_iter),
                  elevation=40, azimuth=-65)
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, "./wire_SOR/phi_{0:08d}.png".format(n_iter))
        plot_surf(Phi, "./wire_SOR_surf/phi_{0:08d}.png".format(n_iter),
                  elevation=40, azimuth=-65)

else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    SOR converged in 3797 iterations to 0.0009987117944452034

    <matplotlib.pyplot._IonContext at 0x7fc910150f10>

## Simple example of Poisson's equation: Wire and charge
Add a positive charge and a negative charge in the box.

Now we need to solve **Poisson's equation**.

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

Finite difference solution:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

### Poisson Solver
Modify the Laplace solvers to now solve Poisson's equation — we are only using our fastest solver in this notebook:

```python
import numpy as np

def Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    a = np.pi * Delta**2
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2]
                                + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2]) + a * rho[1:-2:2, 1:-2:2]
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2]
                                + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2]) + a * rho[2:-1:2, 2:-1:2]

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2]
                                + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2]) + a * rho[1:-2:2, 2:-1:2]
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2]
                                + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2]) + a * rho[2:-1:2, 1:-2:2]
    return Phi
```

### Solve wire with dipole

Solve the Poisson problem with SOR (as above), but now employing the Poisson solvers (we are using the fastest one that we have):

```python
Nmax = 100
Max_iter = 10000
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3823 iterations to 0.0009991605868815044

```python
plot_surf(Phi, elevation=40, azimuth=20)
```

#### Make a movie

The following code can write partial solutions as images. To make it a bit more comfortable, we generate a directory for the images from Python:

```python
import os

def makedir(dirname):
    try:
        os.mkdir(dirname)
    except OSError:
        pass
    return dirname
```

```python
plt.ioff()

nsave = 100

Nmax = 100
Max_iter = 10000
omega = 1.99

dirname = makedir("dipole_wire")
filename = os.path.join(dirname, "phi_{0:08d}.png")

dirname3 = makedir("dipole_wire_3d")
filename3 = os.path.join(dirname3, "phi_{0:08d}.png")

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename=filename.format(n_iter))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename=filename.format(n_iter))
        plot_surf(Phi, filename=filename3.format(n_iter),
                  elevation=40, azimuth=20)
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

print("Images in {0} and {1}".format(dirname, dirname3))
plt.ion()

```

    SOR converged in 3823 iterations to 0.0009991605868815044
    Images in dipole_wire and dipole_wire_3d