# Приготовление EDTA Fe

## Подготовка

Для приготовления микро нам нужны следующие компоненты

1. Хелаторы, один из них:
   - Трилон Б
      * C10H14N2Na2O8*2H2O (Na2 - `ди`натриевая соль), ЭДТУК, EDTA

        Лучше использовать её.

      * C10H12N2O8Na4*4H2O (Na4 - `тетра`натриевая соль), ЭДТУК, EDTA
2. KOH

3. Fe - Сульфат железа (II) семиводный `FeSO4*7H2O` (Fe 20,1%)

4. Основное микро
   * Mn - Марганец (II) сернокислый 5-водный, `MnSO4*5H2O` (Mn 21.95%)
   * B - Борная кислота `H3BO3`, (B 17,5%)
   * Zn - Цинк сернокислый 7-водный `ZnSO4*7H2O`, (Zn 22,7%)
   * Cu - Медь (II) сернокислая 5-водная `CuSO4*5H2O`, (Cu 25,5%)
   * Mo - Аммоний молибденовокислый 4х-водный `(NH4)6Mo7O24*4H2O`, (Mo 54,3%)

5. Опционально, обычно не требуется
   * Co - Сульфат кобальта семиводный `CoSO4*7H2O` (Co 21%)
   * Si - Кремнекислый натрий пятиводный `Na2SiO3*5H2O` (Si 13,2%)

6. Стакан термостойкий на 250-400мл. Не колба, т.к. нужно измерять pH
7. Мешалка или стеклянная палочка

## Расчет

Сначала определяемся с профилем микро.

Допустим у наc профиль, в микрограммах:

`Fe=4500; Mn=600; B=600; Zn=400; Cu=100; Mo=100`

Хотим приготовить концентрат микро 1:1000 на 2 литра

На 1 моль железа приходится 1 моль Трилона-Б

Находим молярную массу Трилона Б: - `372.2369`

* [Можно воспользоваться этим калькулятором](https://apkawa.github.io/react-fertilizer-calculator/#/formula/C10H14N2Na2O8*2H2O/)
* [Или этим](https://ru.webqc.org/molecular-weight-of-C10H14N2Na2O8%2A2H2O.html)

Находим молярную массу сульфата железа: - `278.0146`

Делим моли трилона на моли железа, получаем коэфициент:

$$ K_{ТрилонБ} = \frac{M_{ТрилонБ}}{M_{FeSO4*7H2O}} = \frac{372.2369}{278.0146} = 1.3389
$$

Далее нужно вычислить навеску железа

Есть 2 пути.

1) Рассчитать самому

   * Вычисляем массовую долю железа в сульфате железа, с учетом паспорта (ЧДА 100,6%):\

$$Fe = \frac{М_{Fe}}{M_{FeSO4*7H2O}} \cdot \frac{Чистота_{FeSO4*7H2O}}{100} $$
   Где $M_{Fe}$ и $M_{FeSO4*7H2O}$ - молярные массы

   Или воспользоваться [калькулятором](https://apkawa.github.io/react-fertilizer-calculator/#/formula/FeSO4*7H2O/100.6) и взяв железо из NPK

   * Вычисляем необходимое количество сульфата железа в г/л
   ```
   m = ((Профиль / Fe% ) / мкг в г) * V * C
   m = ((4500 / 0.2021) / 10^6) * 2 * 1000 = 44.5г
   ```
   V - объем
   C - концентрация

2) Воспользоваться калькулятором
   * Забиваем формулу сульфата железа с учетом паспорта в удобрения
   * Выбираем в калькуляторе сульфат железа, профиль, желаемый объем и концентрацию
   * Calculate

Получаем `44.5г` сульфата железа.
Аналогично поступаем с остальными элементами:
   * `5.47г` - Марганец (II) сернокислый 5-водный
   * `0.79г` - Медь сернокислая (II) 5-водная
   * `3.58г` - Цинк сернокислый
   * `6.87г` - Борная кислота
   * `0.37г` - Аммоний молибденовокислый 4-водный

Необходимое количество Трилона Б, с учетом паспорта (ЧДА 99,6%)
$$
m_{ТрилонБ} = m{Fe} \cdot K_{ТрилонБ} \cdot \left( \frac{100+(100-Чистота_{ТрилонБ})}{100}\right)
$$
```
(44.5 * 1.3389) * ((100 + (100 - 99.6)) / 100) =  59.82г
```

Необходимое количество KOH, с учетом паспорта (ЧДА 90%)

[2 KOH + FeSO4 → K2SO4 + Fe(OH)2](https://chemequations.com/ru/?s=KOH%20%2B%20FeSO4%20%3D%20K2SO4%20%2B%20Fe(OH)2&ref=search)
$$
m_{KOH} = m{Fe} \cdot \frac{2M_{KOH}}{M_{FeSO4*7H2O}} \cdot \left( \frac{100+(100-Чистота_{KOH})}{100}\right)
$$

```
56.10564 * 2 / 278.0146 = 0.403
(44.5 * 0.403) * ((100 + (100 - 90)) / 100)  = 19.72г
```

Массу остальных солей не учитываем, т.к. нет необходимости хелатировать остальные соли.
Хелатирование солей нужно в случае аквариумистики, чтобы рыбам было хорошо.

```python
# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install sympy

import sympy as sym
from sympy import Eq
sym.init_printing()
from IPython.display import Math, display
import ipywidgets as widgets
```

    Requirement already satisfied: sympy in /home/apkawa/.local/lib/python3.9/site-packages (1.7.1)
    Requirement already satisfied: mpmath>=0.19 in /home/apkawa/.local/lib/python3.9/site-packages (from sympy) (1.1.0)

```python
Fe_profile = widgets.BoundedIntText(
    value=4500,
    min=1,
    max=10000,
    step=0.1,
    description='Fe профиль, в мкг/л',
    disabled=False
)
Volume = widgets.BoundedFloatText(
    value=2,
    min=0.1,
    max=100,
    step=0.01,
    description='Объем, в л',
    disabled=False
)
Concentration = widgets.BoundedIntText(
    value=1000,
    min=1,
    max=5000,
    step=1,
    description='Концентрация, разы',
    disabled=False
)
Fe_clean = widgets.BoundedFloatText(
    value=100.6,
    min=1,
    max=200,
    step=0.01,
    description='Чистота FeSO4',
    disabled=False
)
ChelateB_clean = widgets.BoundedFloatText(
    value=99.6,
    min=1,
    max=200,
    step=0.01,
    description='Чистота Трилона Б',
    disabled=False
)
KOH_clean = widgets.BoundedFloatText(
    value=90,
    min=1,
    max=200,
    step=0.01,
    description='Чистота KOH',
    disabled=False
)

out = widgets.Output(layout={'border': '1px solid black'})
out

M_ChelateB = 372.2369
M_Fe = 55.8452
M_FeSO4_7H2O = 278.0146
M_KOH = 56.10564

# Функция расчета
def calculate(*args):
    Fe_p = (M_Fe / M_FeSO4_7H2O) * (Fe_clean.value / 100)
    m_FeSO4 = round(((Fe_profile.value / Fe_p) / 10**6) * Volume.value * Concentration.value,2)
    m_ChelateB = round((m_FeSO4 * (M_ChelateB / M_FeSO4_7H2O)) * ((100 + (100 - ChelateB_clean.value)) / 100), 2)
    m_KOH = round((m_FeSO4 * ((M_KOH * 2) / M_FeSO4_7H2O)) * ((100 + (100 - KOH_clean.value)) / 100), 2)
    out.clear_output()
    with out:
        print(f'Навеска FeSO4 = {m_FeSO4}г')
        print(f'Навеска ТрилонаБ = {m_ChelateB}г')
        print(f'Навеска KOH = {m_KOH}г')

calculate()

Inputs = [Fe_profile, Volume, Concentration, Fe_clean, ChelateB_clean, KOH_clean]
for i in Inputs:
    i.observe(calculate, names='value')

display(*(Inputs + [out]))

```

    BoundedIntText(value=4500, description='Fe профиль, в мкг/л', max=10000, min=1, step=0)

    BoundedFloatText(value=2.0, description='Объем, в л', min=0.1, step=0.01)

    BoundedIntText(value=1000, description='Концентрация, разы', max=5000, min=1)

    BoundedFloatText(value=100.6, description='Чистота FeSO4', max=200.0, min=1.0, step=0.01)

    BoundedFloatText(value=99.6, description='Чистота Трилона Б', max=200.0, min=1.0, step=0.01)

    BoundedFloatText(value=90.0, description='Чистота KOH', max=200.0, min=1.0, step=0.01)

    Output(layout=Layout(border='1px solid black'))

## Изготовление

### Способ Сергея

1. Налить в стакан `100мл` кипятка
2. Добавить рассчитанную выше навеску сульфата железа - `44.5г`
3. Тщательно перемешивать. Цвет бурый

4. В зависимости от вида трилона, ди или тетра
   * Динатриевая соль

      Добавлять KOH до тех пор пока не станет pH=7

      Цвет становится черным, как кока-кола

   * Тетранатриевая соль

      Ничего не делать, раствор дб кислым. (TODO уточнить)

5. Высыпать трилон, перемешивая. Не всю навеску сразу, а понемногу.

   Раствор начнет светлеть но еще будет мутным.

   Как только он станет прозрачным - все прореагировало.

   Раствор может еще покраснеть через некоторое время

6. Добавить оставшееся микро
7. Разбавить, в нашем случае до 2л

### Способ Вильяма

1. Налить в стакан `100мл` кипятка
2. Добавить рассчитанную выше навеску сульфата железа
3. Тщательно перемешивать. Цвет бурый

4. Высыпать трилон, перемешивая. Всю рассчитанную навеску.
   (TODO картиночки)

5. В зависимости от вида трилона, ди или тетра

   * Динатриевая соль

     Добавлять понемногу KOH

     И когда допустим только посветлел, я немного мути оставляю, оно само доходит

     после молочного цвета

   * Тетранатриевая соль

     Ничего не делать, раствор дб кислым. (TODO уточнить)

6. Добавить оставшееся микро
7. Разбавить, в нашем случае до 2л

## Способ `Alex Simonec` (технолога по удобрениям)

1. Делаем навески:

   * 1 л обычной воды.
   * 90 грамм дистиллированной воды,
   * расчетное количество Сульфата Железа
   * расчетное количество Трилона Б
   * расчетное количество KOH

2. Берется 2 л бутылка и наливается в нее 1 л обычной воды.

   Добавить сульфат железа, закрыть крышкой и расколачивать до полного растворения.

   Открыть крышку и засыпать KOH.

   Закрыть крышку и перемешиваешь до полного растворения.

3. начнется реакция: вода становиться прозрачной, выпадает темный осадок.
4. Поставить на 12-24 часов отстояться.
5. После отстоя максимально слить воду, оставляя только осадок.
6. После к осадку долить 90 грамм дистиллированной воды и добавить Трилон Б, размешать, раствор зеленеет.
7. рН должен быть в диапазоне 4-6, если не доходить - догони до 5 щелочью, порциями по 0,05 грамм.

8. Добавить оставшееся микро
9. Разбавить, в нашем случае до 2л


# NRPy+'s Reference Metric Interface

## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

### NRPy+ Source Code for this module: [reference_metric.py](../edit/reference_metric.py)

## Introduction:
### Why use a reference metric? Benefits of choosing the best coordinate system for the problem

When solving a partial differential equation on the computer, it is useful to first pick a coordinate system well-suited to the geometry of the problem. For example, if we are modeling a spherically-symmetric star, it would be hugely wasteful to model the star in 3-dimensional Cartesian coordinates ($x$,$y$,$z$). This is because in Cartesian coordinates, we would need to choose high sampling in all three Cartesian directions. If instead we chose to model the star in spherical coordinates  ($r$,$\theta$,$\phi$), so long as the star is centered at $r=0$, we would not need to model the star with more than one point in the $\theta$ and $\phi$ directions!

A similar argument holds for stars that are *nearly* spherically symmetric. Such stars may exhibit density distributions that vary slowly in $\theta$ and $\phi$ directions (e.g., isolated neutron stars or black holes). In these cases the number of points needed to sample the angular directions will still be much smaller than in the radial direction.

Thus choice of an appropriate reference metric may directly mitigate the [Curse of Dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality).

# Table of Contents
$$\label{toc}$$

This module is organized as follow

1. [Step 1](#define_ref_metric): Defining a reference metric, `reference\_metric.py`
1. [Step 2](#define_geometric): Defining geometric quantities, **`ref_metric__hatted_quantities()`**
1. [Step 3](#prescribed_ref_metric): Prescribed reference metrics in `reference_metric.py`
    1. [Step 3.a](#sphericallike): Spherical-like coordinate systems
    1. [Step 3.b](#cylindricallike): Cylindrical-like coordinate systems
    1. [Step 3.c](#cartesianlike): Cartesian-like coordinate systems
    1. [Step 3.d](#prolatespheroidal): Prolate spheroidal coordinates
1. [Step 4](#latex_pdf_output): Output this module to $\LaTeX$-formatted PDF file

# Step 1: Defining a reference metric, `reference_metric.py` \[Back to [top](#toc)\]
$$\label{define_ref_metric}$$

***Note that currently only orthogonal reference metrics of dimension 3 or fewer are supported. This can be extended if desired.***

NRPy+ assumes all curvilinear coordinate systems map directly from a uniform, Cartesian numerical grid with coordinates $(x,y,z)$=(`xx[0]`,`xx[1]`,`xx[2]`). Thus when defining reference metrics, all defined coordinate quantities must be in terms of the `xx[]` array. As we will see, this adds a great deal of flexibility

For example,  [**reference_metric.py**](../edit/reference_metric.py) requires that the *orthogonal coordinate scale factors* be defined. As described [here](https://en.wikipedia.org/wiki/Curvilinear_coordinates), the $i$th scale factor is the positive root of the metric element $g_{ii}$. In ordinary spherical coordinates $(r,\theta,\phi)$, with line element $ds^2 = g_{ij} dx^i dx^j = dr^2+ r^2 d \theta^2 + r^2 \sin^2\theta \ d\phi^2$, we would first define
* $r = xx_0$
* $\theta = xx_1$
* $\phi = xx_2$,

so that the scale factors are defined as
* `scalefactor_orthog[0]` = $1$
* `scalefactor_orthog[1]` = $r$
* `scalefactor_orthog[2]` = $r \sin \theta$

Here is the corresponding code:

```python
import sympy as sp
import NRPy_param_funcs as par
import reference_metric as rfm

r = rfm.xx[0]
th = rfm.xx[1]
ph = rfm.xx[2]

rfm.scalefactor_orthog[0] = 1
rfm.scalefactor_orthog[1] = r
rfm.scalefactor_orthog[2] = r*sp.sin(th)

# Notice that the scale factor will be given
#    in terms of the fundamental Cartesian
#    grid variables, and not {r,th,ph}:
print("r*sin(th) = "+str(rfm.scalefactor_orthog[2]))
```

    r*sin(th) = xx0*sin(xx1)

Next suppose we wish to modify our radial coordinate $r(xx_0)$ to be an exponentially increasing function, so that our numerical grid $(xx_0,xx_1,xx_2)$ will map to a spherical grid with radial grid spacing ($\Delta r$) that *increases* with $r$. Generally we will find it useful to define $r(xx_0)$ to be an odd function, so let's choose

$$r(xx_0) = a \sinh(xx_0/s),$$

where $a$ is an overall radial scaling factor, and $s$ denotes the scale (in units of $xx_0$) over which exponential growth will take place. In our implementation below, note that we use the relation

$$\sinh(x) = \frac{e^x - e^{-x}}{2},$$

as SymPy finds it easier to evaluate exponentials than hyperbolic trigonometric functions.

```python
a,s = sp.symbols('a s',positive=True)
xx0_rescaled = rfm.xx[0] / s
r = a*(sp.exp(xx0_rescaled) - sp.exp(-xx0_rescaled))/2

# Must redefine the scalefactors since 'r' has been updated!
rfm.scalefactor_orthog[0] = 1
rfm.scalefactor_orthog[1] = r
rfm.scalefactor_orthog[2] = r*sp.sin(th)

print(rfm.scalefactor_orthog[2])
```

    a*(exp(xx0/s) - exp(-xx0/s))*sin(xx1)/2

Often we will find it useful to also define the appropriate mappings from (`xx[0]`,`xx[1]`,`xx[2]`) to Cartesian coordinates (for plotting purposes) and ordinary spherical coordinates (e.g., in case initial data when solving a PDE are naturally written in spherical coordinates). For this purpose, reference_metric.py also declares lists **`xxCart[]`** and **`xxSph[]`**, which in this case are defined as

```python
rfm.xxSph[0] = r
rfm.xxSph[1] = th
rfm.xxSph[2] = ph

rfm.xxCart[0] = r*sp.sin(th)*sp.cos(ph)
rfm.xxCart[1] = r*sp.sin(th)*sp.sin(ph)
rfm.xxCart[2] = r*sp.cos(th)

# Here we show off SymPy's pretty_print()
#   and simplify() functions. Nice, no?
sp.pretty_print(sp.simplify(rfm.xxCart[0]))
```

                            ⎛xx₀⎞
    a⋅sin(xx₁)⋅cos(xx₂)⋅sinh⎜───⎟
                            ⎝ s ⎠

# Step 2: Define geometric quantities, `ref_metric__hatted_quantities()` \[Back to [top](#toc)\]
$$\label{define_geometric}$$

Once `scalefactor_orthog[]` has been defined, the function **`ref_metric__hatted_quantities()`** within [reference_metric.py](../edit/reference_metric.py) can be called to define a number of geometric quantities useful for solving PDEs in curvilinear coordinate systems.

Adopting the notation of [Baumgarte, Montero, Cordero-Carrión, and Müller, PRD 87, 044026 (2012)](https://arxiv.org/abs/1211.6632), geometric quantities related to the reference metric are named "hatted" quantities, . For example, the reference metric is defined as $\hat{g}_{ij}$=`ghatDD[i][j]`:

```python
rfm.ref_metric__hatted_quantities()

sp.pretty_print(sp.Matrix(sp.simplify(rfm.ghatDD)))
```

    ⎡1           0                         0              ⎤
    ⎢                                                     ⎥
    ⎢                     2                               ⎥
    ⎢      ⎛ xx₀    -xx₀ ⎞                                ⎥
    ⎢      ⎜ ───    ─────⎟                                ⎥
    ⎢    2 ⎜  s       s  ⎟                                ⎥
    ⎢   a ⋅⎝ℯ    - ℯ     ⎠                                ⎥
    ⎢0  ───────────────────                0              ⎥
    ⎢            4                                        ⎥
    ⎢                                                     ⎥
    ⎢                                          2          ⎥
    ⎢                           ⎛ xx₀    -xx₀ ⎞           ⎥
    ⎢                           ⎜ ───    ─────⎟           ⎥
    ⎢                         2 ⎜  s       s  ⎟     2     ⎥
    ⎢                        a ⋅⎝ℯ    - ℯ     ⎠ ⋅sin (xx₁)⎥
    ⎢0           0           ─────────────────────────────⎥
    ⎣                                      4              ⎦

In addition to $\hat{g}_{ij}$, **`ref_metric__hatted_quantities()`** also provides:
* The rescaling "matrix" `ReDD[i][j]`, used for separating singular (due to chosen coordinate system) pieces of smooth rank-2 tensor components from the smooth parts, so that the smooth parts can be used within temporal and spatial differential operators.
* Inverse reference metric: $\hat{g}^{ij}$=`ghatUU[i][j]`.
* Reference metric determinant: $\det\left(\hat{g}_{ij}\right)$=`detgammahat`.
* First and second derivatives of the reference metric: $\hat{g}_{ij,k}$=`ghatDD_dD[i][j][k]`; $\hat{g}_{ij,kl}$=`ghatDD_dDD[i][j][k][l]`
* Christoffel symbols associated with the reference metric, $\hat{\Gamma}^i_{jk}$ = `GammahatUDD[i][j][k]` and their first derivatives $\hat{\Gamma}^i_{jk,l}$ = `GammahatUDD_dD[i][j][k][l]`

For example, the Christoffel symbol $\hat{\Gamma}^{xx_1}_{xx_2 xx_2}=\hat{\Gamma}^1_{22}$ is given by `GammahatUDD[1][2][2]`:

```python
sp.pretty_print(sp.simplify(rfm.GammahatUDD[1][2][2]))
```

    -sin(2⋅xx₁)
    ────────────
         2

Given the trigonometric identity $2\sin(x)\cos(x) = \sin(2x)$, notice that the above expression is equivalent to Eq. 18 of [Baumgarte, Montero, Cordero-Carrión, and Müller, PRD 87, 044026 (2012)](https://arxiv.org/abs/1211.6632). This is expected since the sinh-radial spherical coordinate system is equivalent to ordinary spherical coordinates in the angular components.

# Step 3: Prescribed reference metrics in `reference_metric.py` \[Back to [top](#toc)\]
$$\label{prescribed_ref_metric}$$

One need not manually define scale factors or other quantities for reference metrics, as a number of prescribed reference metrics are already defined in [reference_metric.py](../edit/reference_metric.py). These can be accessed by first setting the parameter **reference_metric::CoordSystem** to one of the following, and then calling the function **`rfm.reference_metric()`**:

## Step 3.a: Spherical-like coordinate systems \[Back to [top](#toc)\]
$$\label{sphericallike}$$

* **reference_metric::CoordSystem = "Spherical"**: Standard spherical coordinates, with $(r,\theta,\phi)=(xx_0,xx_1,xx_2)$
* **reference_metric::CoordSystem = "SinhSpherical"**: Spherical coordinates, but with $$r(xx_0) = \text{AMPL} \frac{\sinh\left(\frac{xx_0}{\text{SINHW}}\right)}{\sinh\left(\frac{1}{\text{SINHW}}\right)}$$
* **reference_metric::CoordSystem = "SinhSphericalv2"**: Spherical coordinates, but with
    $$r(xx_0) = \text{AMPL} \left[\text{const_dr}\ xx_0 + \frac{\sinh\left(\frac{xx_0}{\text{SINHW}}\right)}{\sinh\left(\frac{1}{\text{SINHW}}\right)}\right]$$

Next let's plot **SinhSpherical** coordinates with `SINHW=0.4` and `AMPL=1.0`.

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

plt.clf()

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r * 0
fig = plt.figure()

ax = plt.subplot(111, projection='polar')

# Plot individual points:
#ax.plot(0.5,1.0,color="green",marker='s')

#ax.plot(theta, r)
ax.set_rmax(2)
# ax.set_rticks([0.1, 0.3, 1.0, 2])  # less radial ticks
ax.set_rlabel_position(-22.5)  # get radial labels away from plotted line

Nr = 20
xx0s = np.linspace(0,2,Nr, endpoint=True) + 1.0/(2.0*Nr)
# print(xx0s)

rs = []
# print(const_dr)
AMPL = 1.0
const_dr = 0.1
const_dr = 1.0/(xx0s[len(xx0s)-1] * AMPL)
SINHW = 0.4
for i in range(Nr):
    rs.append(AMPL * (const_dr*xx0s[i] + (np.exp(xx0s[i] / SINHW) - np.exp(-xx0s[i] / SINHW)) / \
                        (np.exp(1.0 / SINHW) - np.exp(-1.0 / SINHW)) ) )

ax.set_rgrids(rs,labels=[])

thetas = np.linspace(0,360,25, endpoint=True)
# print(thetas)
ax.set_thetagrids(thetas,labels=[])

# ax.grid(True)
ax.grid(True,linewidth='1.0')

# ax.set_title("A line plot on a polar axis", va='bottom')
# plt.savefig("sphgrid.png",dpi=1000)
plt.show()
# plt.close(fig)
```

## Step 3.b: Cylindrical-like coordinate systems \[Back to [top](#toc)\]
$$\label{cylindricallike}$$

* **reference_metric::CoordSystem = "Cylindrical"**: Standard cylindrical coordinates, with $(\rho,\phi,z)=(xx_0,xx_1,xx_2)$
    * **reference_metric::CoordSystem = "SinhCylindrical"**: Cylindrical coordinates, but with
    $$\rho(xx_0) = \text{AMPLRHO} \frac{\sinh\left(\frac{xx_0}{\text{SINHWRHO}}\right)}{\sinh\left(\frac{1}{\text{SINHWRHO}}\right)}$$
    and
    $$z(xx_2) = \text{AMPLZ} \frac{\sinh\left(\frac{xx_2}{\text{SINHWZ}}\right)}{\sinh\left(\frac{1}{\text{SINHWZ}}\right)}$$

* **reference_metric::CoordSystem = "SinhCylindricalv2"**: Cylindrical coordinates, but with
$$\rho(xx_0) = \text{AMPLRHO} \left[\text{const_drho}\ xx_0 + \frac{\sinh\left(\frac{xx_0}{\text{SINHWRHO}}\right)}{\sinh\left(\frac{1}{\text{SINHWRHO}}\right)}\right]$$
and
$$z(xx_2) = \text{AMPLZ} \left[\text{const_dz}\ xx_2 + \frac{\sinh\left(\frac{xx_2}{\text{SINHWZ}}\right)}{\sinh\left(\frac{1}{\text{SINHWZ}}\right)}\right]$$

For example, let's set up SinhCylindricalv2 coordinates and output the Christoffel symbol $\hat{\Gamma}^{xx_2}_{xx_2 xx_2}$, or more simply $\hat{\Gamma}^2_{22}$:

```python
par.set_parval_from_str("reference_metric::CoordSystem","SinhCylindricalv2")

rfm.reference_metric()

sp.pretty_print(sp.simplify(rfm.GammahatUDD[2][2][2]))
```

                             ⎛ 2⋅xx₂     ⎞    1
                             ⎜ ──────    ⎟  ──────
                             ⎜ SINHWZ    ⎟  SINHWZ
                            -⎝ℯ       - 1⎠⋅ℯ
    ────────────────────────────────────────────────────────────────────────
           ⎛                  ⎛   2       ⎞   xx₂     ⎛ 2⋅xx₂     ⎞    1   ⎞
           ⎜                  ⎜ ──────    ⎟  ──────   ⎜ ──────    ⎟  ──────⎟
           ⎜                  ⎜ SINHWZ    ⎟  SINHWZ   ⎜ SINHWZ    ⎟  SINHWZ⎟
    SINHWZ⋅⎝- SINHWZ⋅const_dz⋅⎝ℯ       - 1⎠⋅ℯ       - ⎝ℯ       + 1⎠⋅ℯ      ⎠

As we will soon see, defining these "hatted" quantities will be quite useful when expressing hyperbolic ([wave-equation](https://en.wikipedia.org/wiki/Wave_equation)-like) PDEs in non-Cartesian coordinate systems.

## Step 3.c: Cartesian coordinate systems \[Back to [top](#toc)\]
$$\label{cartesianlike}$$

* **reference_metric::CoordSystem = "Cartesian"**: Standard Cartesian coordinates, with $(x,y,z)=(xx_0,xx_1,xx_2)$

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

plt.clf()

fig = plt.figure()
ax = fig.gca()
Nx = 16
ax.set_xticks(np.arange(0, 1., 1./Nx))
ax.set_yticks(np.arange(0, 1., 1./Nx))
# plt.scatter(x, y)
ax.set_aspect('equal')

plt.grid()

# plt.savefig("Cartgrid.png",dpi=300)
plt.show()
# plt.close(fig)
```

## Step 3.d: [Prolate spheroidal](https://en.wikipedia.org/wiki/Prolate_spheroidal_coordinates)-like coordinate systems \[Back to [top](#toc)\]
$$\label{prolatespheroidal}$$

* **reference_metric::CoordSystem = "SymTP"**: Symmetric TwoPuncture coordinates, with $(\rho,\phi,z)=(xx_0\sin(xx_1), xx_2, \sqrt{xx_0^2 + \text{bScale}^2}\cos(xx_1))$
* **reference_metric::CoordSystem = "SinhSymTP"**: Symmetric TwoPuncture coordinates, but with $$xx_0 \to \sinh(xx_0/\text{AW})$$

# Step 4: Output this module to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Reference_Metric.pdf](Tutorial-Reference_Metric.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx Tutorial-Reference_Metric.ipynb
!pdflatex -interaction=batchmode Tutorial-Reference_Metric.tex
!pdflatex -interaction=batchmode Tutorial-Reference_Metric.tex
!pdflatex -interaction=batchmode Tutorial-Reference_Metric.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    [NbConvertApp] Converting notebook Tutorial-Reference_Metric.ipynb to latex
    [NbConvertApp] Support files will be in Tutorial-Reference_Metric_files/
    [NbConvertApp] Making directory Tutorial-Reference_Metric_files
    [NbConvertApp] Making directory Tutorial-Reference_Metric_files
    [NbConvertApp] Writing 50091 bytes to Tutorial-Reference_Metric.tex
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


# Inferring parameters of SDEs using a Euler-Maruyama scheme

_This notebook is derived from a presentation prepared for the Theoretical Neuroscience Group, Institute of Systems Neuroscience at Aix-Marseile University._

```python
%pylab inline
import arviz as az
import pymc3 as pm
import scipy
import theano.tensor as tt

from pymc3.distributions.timeseries import EulerMaruyama
```

    Populating the interactive namespace from numpy and matplotlib

```python
%config InlineBackend.figure_format = 'retina'
az.style.use("arviz-darkgrid")
```

## Toy model 1

Here's a scalar linear SDE in symbolic form

$ dX_t = \lambda X_t + \sigma^2 dW_t $

discretized with the Euler-Maruyama scheme

```python
# parameters
λ = -0.78
σ2 = 5e-3
N = 200
dt = 1e-1

# time series
x = 0.1
x_t = []

# simulate
for i in range(N):
    x += dt * λ * x + sqrt(dt) * σ2 * randn()
    x_t.append(x)

x_t = array(x_t)

# z_t noisy observation
z_t = x_t + randn(x_t.size) * 5e-3
```

```python
figure(figsize=(10, 3))
subplot(121)
plot(x_t[:30], "k", label="$x(t)$", alpha=0.5), plot(z_t[:30], "r", label="$z(t)$", alpha=0.5)
title("Transient"), legend()
subplot(122)
plot(x_t[30:], "k", label="$x(t)$", alpha=0.5), plot(z_t[30:], "r", label="$z(t)$", alpha=0.5)
title("All time")
tight_layout()
```

What is the inference we want to make? Since we've made a noisy observation of the generated time series, we need to estimate both $x(t)$ and $\lambda$.

First, we rewrite our SDE as a function returning a tuple of the drift and diffusion coefficients

```python
def lin_sde(x, lam):
    return lam * x, σ2
```

Next, we describe the probability model as a set of three stochastic variables, `lam`, `xh`, and `zh`:

```python
with pm.Model() as model:

    # uniform prior, but we know it must be negative
    lam = pm.Flat("lam")

    # "hidden states" following a linear SDE distribution
    # parametrized by time step (det. variable) and lam (random variable)
    xh = EulerMaruyama("xh", dt, lin_sde, (lam,), shape=N, testval=x_t)

    # predicted observation
    zh = pm.Normal("zh", mu=xh, sigma=5e-3, observed=z_t)
```

Once the model is constructed, we perform inference, i.e. sample from the posterior distribution, in the following steps:

```python
with model:
    trace = pm.sample(2000, tune=1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [xh, lam]

  <progress value='12000' class='' max='12000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [12000/12000 00:56<00:00 Sampling 4 chains, 2,000 divergences]

    Sampling 4 chains for 1_000 tune and 2_000 draw iterations (4_000 + 8_000 draws total) took 58 seconds.
    The acceptance probability does not match the target. It is 0.9255362275622311, but should be close to 0.8. Try to increase the number of tuning steps.
    The chain contains only diverging samples. The model is probably misspecified.
    The acceptance probability does not match the target. It is 0.2038179163556457, but should be close to 0.8. Try to increase the number of tuning steps.
    The rhat statistic is larger than 1.4 for some parameters. The sampler did not converge.
    The estimated number of effective samples is smaller than 200 for some parameters.

Next, we plot some basic statistics on the samples from the posterior,

```python
figure(figsize=(10, 3))
subplot(121)
plot(percentile(trace[xh], [2.5, 97.5], axis=0).T, "k", label=r"$\hat{x}_{95\%}(t)$")
plot(x_t, "r", label="$x(t)$")
legend()

subplot(122)
hist(trace[lam], 30, label=r"$\hat{\lambda}$", alpha=0.5)
axvline(λ, color="r", label=r"$\lambda$", alpha=0.5)
legend();
```

A model can fit the data precisely and still be wrong; we need to use _posterior predictive checks_ to assess if, under our fit model, the data our likely.

In other words, we
- assume the model is correct
- simulate new observations
- check that the new observations fit with the original data

```python
# generate trace from posterior
ppc_trace = pm.sample_posterior_predictive(trace, model=model)

# plot with data
figure(figsize=(10, 3))
plot(percentile(ppc_trace["zh"], [2.5, 97.5], axis=0).T, "k", label=r"$z_{95\% PP}(t)$")
plot(z_t, "r", label="$z(t)$")
legend()
```

Note that

- inference also estimates the initial conditions
- the observed data $z(t)$ lies fully within the 95% interval of the PPC.
- there are many other ways of evaluating fit

### Toy model 2

As the next model, let's use a 2D deterministic oscillator,
\begin{align}
\dot{x} &= \tau (x - x^3/3 + y) \\
\dot{y} &= \frac{1}{\tau} (a - x)
\end{align}

with noisy observation $z(t) = m x + (1 - m) y + N(0, 0.05)$.

```python
N, τ, a, m, σ2 = 200, 3.0, 1.05, 0.2, 1e-1
xs, ys = [0.0], [1.0]
for i in range(N):
    x, y = xs[-1], ys[-1]
    dx = τ * (x - x**3.0 / 3.0 + y)
    dy = (1.0 / τ) * (a - x)
    xs.append(x + dt * dx + sqrt(dt) * σ2 * randn())
    ys.append(y + dt * dy + sqrt(dt) * σ2 * randn())
xs, ys = array(xs), array(ys)
zs = m * xs + (1 - m) * ys + randn(xs.size) * 0.1

figure(figsize=(10, 2))
plot(xs, label="$x(t)$")
plot(ys, label="$y(t)$")
plot(zs, label="$z(t)$")
legend()
```

Now, estimate the hidden states $x(t)$ and $y(t)$, as well as parameters $\tau$, $a$ and $m$.

As before, we rewrite our SDE as a function returned drift & diffusion coefficients:

```python
def osc_sde(xy, τ, a):
    x, y = xy[:, 0], xy[:, 1]
    dx = τ * (x - x**3.0 / 3.0 + y)
    dy = (1.0 / τ) * (a - x)
    dxy = tt.stack([dx, dy], axis=0).T
    return dxy, σ2
```

As before, the Euler-Maruyama discretization of the SDE is written as a prediction of the state at step $i+1$ based on the state at step $i$.

We can now write our statistical model as before, with uninformative priors on $\tau$, $a$ and $m$:

```python
xys = c_[xs, ys]

with pm.Model() as model:
    τh = pm.Uniform("τh", lower=0.1, upper=5.0)
    ah = pm.Uniform("ah", lower=0.5, upper=1.5)
    mh = pm.Uniform("mh", lower=0.0, upper=1.0)
    xyh = EulerMaruyama("xyh", dt, osc_sde, (τh, ah), shape=xys.shape, testval=xys)
    zh = pm.Normal("zh", mu=mh * xyh[:, 0] + (1 - mh) * xyh[:, 1], sigma=0.1, observed=zs)
```

```python
with model:
    trace = pm.sample(2000, tune=1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [xyh, mh, ah, τh]

  <progress value='12000' class='' max='12000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [12000/12000 02:02<00:00 Sampling 4 chains, 0 divergences]

    Sampling 4 chains for 1_000 tune and 2_000 draw iterations (4_000 + 8_000 draws total) took 125 seconds.
    The number of effective samples is smaller than 10% for some parameters.

Again, the result is a set of samples from the posterior, including our parameters of interest but also the hidden states

```python
figure(figsize=(10, 6))
subplot(211)
plot(percentile(trace[xyh][..., 0], [2.5, 97.5], axis=0).T, "k", label=r"$\hat{x}_{95\%}(t)$")
plot(xs, "r", label="$x(t)$")
legend(loc=0)
subplot(234), hist(trace["τh"]), axvline(τ), xlim([1.0, 4.0]), title("τ")
subplot(235), hist(trace["ah"]), axvline(a), xlim([0, 2.0]), title("a")
subplot(236), hist(trace["mh"]), axvline(m), xlim([0, 1]), title("m")
tight_layout()
```

Again, we can perform a posterior predictive check, that our data are likely given the fit model

```python
# generate trace from posterior
ppc_trace = pm.sample_posterior_predictive(trace, model=model)

# plot with data
figure(figsize=(10, 3))
plot(percentile(ppc_trace["zh"], [2.5, 97.5], axis=0).T, "k", label=r"$z_{95\% PP}(t)$")
plot(zs, "r", label="$z(t)$")
legend()
```

```python
%load_ext watermark
%watermark -n -u -v -iv -w
```

    scipy            1.4.1
    logging          0.5.1.2
    matplotlib.pylab 1.18.5
    re               2.2.1
    pymc3            3.9.0
    matplotlib       3.2.1
    numpy            1.18.5
    arviz            0.8.3
    last updated: Mon Jun 15 2020

    CPython 3.7.7
    IPython 7.15.0
    watermark 2.0.2


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  This material largely follows the text "Numerical Linear Algebra" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there.

```python
%matplotlib inline
%precision 3
import numpy
import matplotlib.pyplot as plt
```

# Numerical Linear Algebra

Numerical methods for linear algebra problems lies at the heart of many numerical approaches and is something we will spend some time on.  Roughly we can break down problems that we would like to solve into three general problems, solving a system of equations

$$
A \mathbf{x} = \mathbf{b}
$$

Projection problems or Linear Least Squares

$$ A^TA\mathbf{x} = A^T\mathbf{b}$$

and solving the eigenvalue problem

$$A \mathbf{v} = \lambda \mathbf{v}.$$

We examine each of these problems separately and will evaluate some of the fundamental properties and methods for solving these problems. We will be careful in deciding how to evaluate the results of our calculations and try to gain some understanding of when and how they fail.

## Factorizations

For each fundamental problem, there are a range of algorithms to solve them, but all of these algorithms can often be succinctly described in terms of a matrix "factorization", the ability to write an arbitrary matrix $A$ as a product of matrices with special properties (e.g. triangular, diagonal, orthogonal) that make solving the general problem straightforward.  For example

Problem	 Algorithms	 "Factorizations"

$$A \mathbf{x} = \mathbf{b}$$	 Gaussian-Elimination, Gauss-Jordan Elimination	 $PA=LU$, $A=ER$

$$ A^TA\mathbf{x} = A^T\mathbf{b}$$	 Orthogonalization algorithms (Gram-Schmidt, Modified GS, Householder, Givens)	 $A=QR$

$$A \mathbf{v} = \lambda \mathbf{v}$$	 Various Iterative methods (Power, inverse power, $RQ$ with shifts$\ldots$	 $A=S\Lambda S^{-1}$, $A=Q\Lambda Q^T$, $A=MJM^{-1}$

All of the above	 Singular Value Decomposition (SVD)	 $A = U\Sigma V^T$

To develop all of these algorithms in depth is a course in itself, however, here we will highlight some of the key factorizations and algorithms used in modern computational linear algebra.  In particular we will highlight issues of accuracy, stability and computational cost particularly in the limit of large systems.

## Some Example Problems

The number and power of the different tools made available from the study of linear algebra makes it an invaluable field of study. Before we dive in to numerical approximations we first consider some of the pivotal problems that numerical methods for linear algebra are used to address.

For this discussion we will be using the common notation $m \times n$ to denote the dimensions of a matrix $A$.  The $m$ refers to the number of rows and $n$ the number of columns.  If a matrix is square, i.e. $m = n$, then we will use the notation that $A$ is $m \times m$.

### Systems of Equations

The first type of problem is to find the solution to a linear system of equations.  If we have $m$ equations for $m$ unknowns it can be written in matrix/vector form,

$$A \mathbf{x} = \mathbf{b}.$$

For this example $A$ is an $m \times m$ matrix, denoted as being in $\mathbb{R}^{m\times m}$, and $\mathbf{x}$ and $\mathbf{b}$ are column vectors with $m$ entries, denoted as $\mathbb{R}^m$.

#### Example 1:  Polynomial Fitting
In our previous work on interpolation we found that the unique interpolating polynomial of order $n$ through the $n+1$ points $(x_i, y_i)$ can be found by solving a Linear system of equations

$$
V[\phi_j(\mathbf{x})]\mathbf{w} = \mathbf{y}
$$

where $V$ is a VanderMonde matrix whose columns are the basis functions $\phi_j(\mathbf{x})$ (e.g. the monomials) and $\mathbf{w}\in\mathbb{R}^{n+1}$ are the coefficients (weights) such that the interpolating polynomial is given uniquely by
$$
    \cal{P}_n(x) = \sum_{j=0}^n w_j\phi_j(x)
$$

#### Examples 2: Solution of Non-linear equations by Newton's Method

Given a non-linear system of equations

$$\mathbf{F}(\mathbf{x})=\mathbf{0}$$

Newton's method provides an iterative method to find roots where at every step of the iteration a linear system of equations

$$
    J(\mathbf{x_k})\boldsymbol{\delta}_k = -\mathbf{F}(\mathbf{x_k})
$$

### Linear least squares

In a similar case as above, say we want to fit a particular function (could be a polynomial) to a given number of data points except in this case we have more data points than free parameters.  In the case of polynomials this could be the same as saying we have $m$ data points but only want to fit a $n - 1$ order polynomial through the data where $n - 1 \leq m$.  One of the common approaches to this problem is to minimize the "least-squares" error between the data and the resulting function:
$$
    E = \left( \sum^m_{i=1} |y_i - f(x_i)|^2 \right )^{1/2}.
$$

E.g. Consider fitting the function
$$
    f(x) = w_1 + w_2 x + w_3 e^x
$$

data that has random noise added to it.

If our function $f(x)$ can be written as a linear combination of basis functions

$$
    f(x) = \sum_{j=1}^n w_j\phi_j(x)
$$

then the linear system through $m$ points becomes $A\mathbf{w} = \mathbf{y}$ where $A$ is a generalized $m\times n$ vandermonde matrix

$$
    A = \begin{bmatrix}
        \phi_1(x_1) & \phi_2(x_1) & \cdots & \phi_n(x_1) \\
        \phi_1(x_2) & \phi_2(x_2) & \cdots &  \phi_n(x_2) \\
        \vdots & \vdots & &\vdots   \\
        \phi_1(x_m) & \phi_2(x_m) & \cdots &  \phi_n(x_m) \\
    \end{bmatrix}
$$

Turns out if we solve the system

$$A^T A \mathbf{x} = A^T \mathbf{b}$$

we can guarantee that the error is minimized in the least-squares sense[1](#footnoteRegression). (Although we will also show that this is not the most numerically stable way to solve this problem)

#### Define the data

```python
N = 20
x = numpy.linspace(-1.0, 1.0, N)
y = x + numpy.random.random((N))
```

#### Define basis functions and Vandermonde matrix

```python
# define the basis functions
phi_1 = lambda x: numpy.ones(x.shape)
phi_2 = lambda x: x
phi_3 = lambda x: numpy.exp(x)
# Define various Vandermonde matrix based on our x-values
A = numpy.array([ phi_1(x), phi_2(x), phi_3(x)]).T
A.shape
```

```python
# Determine the weights of our linear function
# result in the smallest sum of the squares of the residual.
w = numpy.linalg.solve(numpy.dot(A.T, A), numpy.dot(A.T, y))
error = y - A.dot(w)
print('w = {}, ||e|| = {}'.format(w,numpy.linalg.norm(error)))
```

```python
# Do the same using numpy's lstsq routine (which solves a more numerically stable problem)
w = numpy.linalg.lstsq(A,y, rcond=None)[0]
error = y - A.dot(w)
print('w = {}, ||e|| = {}'.format(w,numpy.linalg.norm(error)))
```

```python
# Plot it out, cuz pictures are fun!
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
f = A.dot(w)
axes.plot(x, y, 'ko')
axes.plot(x, f, 'r')
axes.set_title("Least Squares Fit to Data")
axes.set_xlabel("$x$")
axes.set_ylabel("$f(x)$ and $y_i$")
axes.grid()

plt.show()
```

### Eigenproblems

Eigenproblems come up in a variety of contexts and often are integral to many problem of scientific and engineering interest. It is such a powerful idea that it is not uncommon for us to take a problem and convert it into an eigenproblem.

One of my favorite examples [The Tacoma Narrows Bridge Collapse](https://www.youtube.com/watch?v=XggxeuFDaDU)

Or the original Google [Page-Rank](https://en.wikipedia.org/wiki/PageRank) algorithm which essentially finds the dominant eigenvector of an enormous sparse Markov Matrix.

Here we introduce the idea and give some examples.

As a review, if $A \in \mathbb{C}^{m\times m}$ (a square matrix with complex values), a non-zero vector $\mathbf{v}\in\mathbb{C}^m$ is an **eigenvector** of $A$ with a corresponding **eigenvalue** $\lambda \in \mathbb{C}$ if

$$A \mathbf{v} = \lambda \mathbf{v}.$$

One way to interpret the eigenproblem is that we are attempting to ascertain the "action" of the matrix $A$ on some subspace of $\mathbb{C}^m$ where this action acts like scalar multiplication.  This subspace is called an **eigenspace**.

#### General idea of EigenProblems

Rewriting the standard Eigen problem $A\mathbf{v}=\lambda\mathbf{v}$ for $A \in \mathbb{C}^{m\times m}$, $\mathbf{v}\in\mathbb{C}^m$ as

$$
    (A - \lambda I)\mathbf{v} = \mathbf{0}
$$

it becomes clear that for $\mathbf{v}$ to be non-trivial (i.e. $\neq \mathbf{0}$), requires that the matrix $(A-\lambda I)$ be singular,

This is equivalent to finding all values of $\lambda$ such that $|A-\lambda I| = 0$ (the determinant of singular matrices is always zero).  However, it can also be shown that

$$
   | A-\lambda I| = P_m(\lambda)
$$

which is a $m$th order polynomial in $\lambda$.  Thus $P_m(\lambda)=0$ implies the eigenvalues are the $m$ roots of $P$, and the **eigenspace** corresponding to $\lambda_i$ is just $N(A-\lambda_i I)$

#### Solving EigenProblems

The temptation (and what we usually teach in introductory linear algebra classes) is to find the roots of $P_m(\lambda)$  to get the eigenvalues, then find the null-space of $A-\lambda I$.

However that would be **wrong**.  The best algorithms for finding Eigenvalues are completely unrelated to rootfinding as we shall see (and in fact, the way you find the roots of polynomials is to find the eigenvalues of a "companion matrix")

### The Singular Value Decomposition

One of the most beautiful, and useful, factorizations is the Singular Value Decomposition (or SVD),

$$
    A = U\Sigma V^T
$$

where $A$ is a general $m\times n$ matrix, $U$ and $V$ are unitary (orthogonal) matrices such that $U^TU=I^{m\times m}$, $V^TV = I^{n\times n}$ and $\Sigma$ is a diagonal matrix with real, positive  diagonal entries (the singular values)

$$
    \sigma_1 \geq \sigma_2\geq \ldots \sigma_{r} > 0
$$
where $r$ is the rank of $A$

#### Applications of the SVD

The SVD combines all the aspects of basic linear algebra and is used in a large number of applications including

* diagnosing ill-conditioned matrices
* providing orthonormal bases for the 4-subspaces of $A$
* Solving linear systems and linear least-squares problems
* Solving ill-conditioned and singular linear systems (Pseudo-inverse)
* Dimensional reduction in Data analysis (PCA, EOF, POD analysis)
* and more...

Because of its ubiquity in computational linear algebra and data science. we will spend a bit of time to understand the SVD and its applications (but not the specific algorithms).  But first

## Fundamentals

**Objectives**
* Understand basic linear-algebraic operations and their **computational costs**
* Understand Numpy implementation (and performance) for Linear Algebra
* Understand Singular vs Invertible matrices
* Understand Orthonormal vectors and $Q$ matrices
* Understand vector and Matrix norms
* Undertand the **Condition Number**

### Matrix-Vector Multiplication

One of the most basic operations we can perform with matrices is to multiply them by a vector which maps a vector to a vector.  There are multiple ways to interpret and compute the matrix-vector product $A \mathbf{x}$.

#### index notation
$$
    b_i = \sum^n_{j=1} a_{ij} x_j \quad \text{where}\quad i = 1, \ldots, m
$$

#### row picture (dot products)
We can also consider matrix-vector multiplication as a sequence of inner products (dot-products between the rows of $A$ and the vector $\mathbf{x}$.
\begin{align}
    \mathbf{b} &= A \mathbf{x}, \\
     &=
    \begin{bmatrix}  \mathbf{a}_1^T \mathbf{x} \\ \mathbf{a}_2^T \mathbf{x} \\ \vdots \\ \mathbf{a}_m^T \mathbf{x}\end{bmatrix}
\end{align}
where $\mathbf{a}_i^T$ is the $i$th **row** of $A$

#### column picture

An alternative (and entirely equivalent way) to write the matrix-vector product is as a linear combination of the columns of $A$ where each column's weighting is $x_j$.

$$
\begin{align}
    \mathbf{b} &= A \mathbf{x}, \\
    &=
    \begin{bmatrix}  &  &  &  \\  &  &  &  \\ \mathbf{a}_1 & \mathbf{a}_2 & \cdots & \mathbf{a}_n \\  &  &  &   \\  &  &  &  \end{bmatrix}
    \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}, \\
  &= x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 + \cdots + x_n \mathbf{a}_n.
\end{align}
$$

This view will be useful later when we are trying to interpret various types of matrices.

### Operation Counts
No matter how you compute $A\mathbf{x}$,  the total number of operations is the same (for a dense matrix $A$).
The row view however is convenient for calculating the **Operation counts** required for $A\mathbf{x}$.

If $A\in\mathbb{C}^{m\times n}$ and $\mathbf{x}\in\mathbb{C}^n$.  Then just counting the number of multiplications involved to compute $A\mathbf{x}$ is $O(??)$

One important property of the matrix-vector product is that is a **linear** operation, also known as a **linear operator**.  This means that the for any $\mathbf{x}, \mathbf{y} \in \mathbb{C}^n$ and any $c \in \mathbb{C}$ we know that

1. $A (\mathbf{x} + \mathbf{y}) = A\mathbf{x} + A\mathbf{y}$
1. $A\cdot (c\mathbf{x}) = c A \mathbf{x}$

#### Example:  Numerical matrix-vector multiply

Write a matrix-vector multiply function and check it with the appropriate `numpy` routine.  Also verify the linearity of the matrix-vector multiply.

```python
# Mat-vec in index notation (the long way...don't do this)
def matrix_vector_product_index(A, x):
    m, n = A.shape
    b = numpy.zeros(m)
    for i in range(m):
        for j in range(n):
            b[i] += A[i, j] * x[j]
    return b

# Mat-vec by row picture (still don't do this)
def matrix_vector_product_row(A, x):
    m, n = A.shape
    b = numpy.zeros(m)
    # loop on rows
    for i in range(m):
        b[i] = A[i, :].dot(x)
    return b

# Mat-vec by column picture (still don't do this)
def matrix_vector_product_col(A, x):
    m, n = A.shape
    b = numpy.zeros(m)
    # loop over columns
    for j in range(n):
            b += A[:, j] * x[j]
    return b

```

#### Check equivalence

first set up some large random matrices and vectors and compare to numpy built in .dot function

```python
m = 1000
n = 1000
A = numpy.random.uniform(size=(m,n))
x = numpy.random.uniform(size=(n))
y = numpy.random.uniform(size=(n))
c = numpy.random.uniform()
```

```python
funcs = [ matrix_vector_product_index,  matrix_vector_product_row,  matrix_vector_product_col ]
for f in funcs:
    b = f(A, x)
    print('{}(A,x) = A.dot(x)? {}'.format(f.__name__, numpy.allclose(b, A.dot(x))))
```

#### Check Linearity

```python
for f in funcs:
    print('{}(A,x+y) = Ax + Ay is {}'.format(f.__name__,
                                         numpy.allclose(f(A, (x + y)), f(A, x) + f(A, y))))
    print('{}(A,cx) = cAx  is {}\n'.format(f.__name__,
                                         numpy.allclose(f(A, c * x), c*f(A, x))))

```

#### Check Timing/performance

```python
for f in funcs:
    print(f.__name__,end='\t')
    %timeit matrix_vector_product_index(A,x)

print('numpy.dot(A,x)',end='\t\t\t')
%timeit A.dot(x)
```

### Matrix-Matrix Multiplication

The matrix product with another matrix $ C=AB$ is defined as
$$
    c_{ij} = \sum^m_{k=1} a_{ik} b_{kj} = \mathbf{a}_i^T\mathbf{b}_j
$$

i.e. each component of $C$ is a dot-product between the $i$th row of $A$ and the $j$th column of $B$

As with matrix-vector multiplication, Matrix-matrix multiplication can be thought of multiple ways

* $m\times p$ dot products (each with $n$ flops)
* $A$ multiplying the columns of $B$
$$
    C = AB = \begin{bmatrix}
                A\mathbf{b}_1 & A\mathbf{b}_2 & \ldots & A\mathbf{b}_p\\
             \end{bmatrix}
$$
* Linear combinations of the rows of $B$
$$
C = AB = \begin{bmatrix}
                \mathbf{a}_1^T B \\ \mathbf{a}_2^T B \\ \vdots \\ \mathbf{a}_m^T B\\
             \end{bmatrix}
$$

### Questions
* What are the dimensions of $A$ and $B$ so that the multiplication works?
* What are the Operations Counts for Matrix-Matrix Multiplication?
* Comment on the product $\mathbf{c}=(AB)\mathbf{x}$ vs. $\mathbf{d} = A(B\mathbf{x})$

#### Example:  Outer Product

The product of two vectors $\mathbf{u} \in \mathbb{C}^m$ and $\mathbf{v} \in \mathbb{C}^n$ is a $m \times n$ matrix where the columns are the vector $u$ multiplied by the corresponding value of $v$:
$$
\begin{align}
    \mathbf{u} \mathbf{v}^T &=
    \begin{bmatrix}  u_1 \\ u_2 \\ \vdots \\ u_m  \end{bmatrix}
    \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}, \\
    & = \begin{bmatrix} v_1u_1 & \cdots & v_n u_1 \\ \vdots &  & \vdots \\ v_1 u_m & \cdots & v_n u_m \end{bmatrix}.
\end{align}
$$

It is useful to think of these as operations on the column vectors, and an equivalent way to express this relationship is
$$
\begin{align}
    \mathbf{u} \mathbf{v}^T &=
    \begin{bmatrix}  \\ \mathbf{u} \\ \\  \end{bmatrix}
    \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}, \\
    &=
    \begin{bmatrix}  &  &  &  \\  &  &  &  \\ \mathbf{u}v_1  & \mathbf{u} v_2  & \cdots & \mathbf{u} v_n  \\  &  &  & \\  &  &  &  \end{bmatrix}, \\
    & = \begin{bmatrix} v_1u_1 & \cdots & v_n u_1 \\ \vdots &  & \vdots \\ v_1 u_m & \cdots & v_n u_m \end{bmatrix}.
\end{align}
$$

Or each column is just a scalar multiple of $\mathbf{u}$

Alternatively you can think of this as the rows are all just scalar multiples of $\mathbf{v}$

### rank 1 updates

We call any matrix of the form $\mathbf{u}\mathbf{v}^T$ a "rank one matrix"  ( because its rank r=?).  These sort of matrix operations are very common in numerical algorithms for orthogonalization, eigenvalues and the original page-rank algorithm for google.  Again, the order of operations is critical.

Comment on the difference in values and operation counts between

$$
    \mathbf{y} = (\mathbf{u}\mathbf{v}^T)\mathbf{x}
$$

and
$$
    \tilde{\mathbf{y}} = \mathbf{u}(\mathbf{v}^T\mathbf{x})
$$
for $\mathbf{u}$, $\mathbf{v}$, $\mathbf{x}$, $\mathbf{y}$, $\tilde{\mathbf{y}}\in\mathbb{R}^n$,

### Check

Set up some random vectors

```python
m = 10000
n = 10000
u = numpy.random.rand(m)
v = numpy.random.rand(n)
x = numpy.random.rand(n)
```

#### Check Equivalence of numbers

show
$$
    (\mathbf{u}\mathbf{v}^T)\mathbf{x} = \mathbf{u}(\mathbf{v}^T\mathbf{x})
$$

```python
b = numpy.outer(u,v).dot(x)
c = u.dot(v.dot(x))
d = u*v.dot(x)
print('Max Difference = {}'.format(numpy.max(numpy.abs(b-c))))
print('Max Difference = {}'.format(numpy.max(numpy.abs(b-d))))
print('Max Difference = {}'.format(numpy.max(numpy.abs(c-d))))
```

#### Check Timing/performance

```python
# timing
print('(uv^T).dot(x)',end='\t\t')
%timeit numpy.outer(u,v).dot(x)
print('\n(u.dot(v.dot(x))',end='\t')
%timeit u.dot(v.dot(x))
```

#### Example:  Upper Triangular Multiplication

Consider the multiplication of a matrix $A \in \mathbb{C}^{m\times n}$ and the **upper-triangular** matrix $R$ defined as the $n \times n$ matrix with entries $r_{ij} = 1$ for $i \leq j$ and $r_{ij} = 0$ for $i > j$.  The product can be written as
$$
    \begin{bmatrix}  \\  \\ \mathbf{b}_1 & \cdots & \mathbf{b}_n \\ \\ \\ \end{bmatrix} = \begin{bmatrix} \\ \\  \mathbf{a}_1 & \cdots & \mathbf{a}_n \\ \\ \\ \end{bmatrix} \begin{bmatrix} 1 & \cdots & 1 \\  & \ddots & \vdots \\  &  & 1 \end{bmatrix}.
$$

The columns of $B$ are then
$$
    \mathbf{b}_j = A \mathbf{r}_j = \sum^j_{k=1} \mathbf{a}_k
$$
so that $\mathbf{b}_j$ is the sum of the first $j$ columns of $A$.

#### Example: Write Matrix-Matrix Multiplication

Write a function that computes matrix-matrix multiplication and demonstrate the following properties:
1. $A (B + C) = AB + AC$ (for square matrices))
1. $A (cB) = c AB$ where $c \in \mathbb{C}$
1. $AB \neq BA$ in general

```python
def matrix_matrix_product(A, B):
    C = numpy.zeros((A.shape[0], B.shape[1]))
    for i in range(A.shape[0]):
        for j in range(B.shape[1]):
            for k in range(A.shape[1]):
                C[i, j] += A[i, k] * B[k, j]
    return C

m = 4
n = 4
p = 4
A = numpy.random.uniform(size=(m, n))
B = numpy.random.uniform(size=(n, p))
C = numpy.random.uniform(size=(m, p))
c = numpy.random.uniform()
print(numpy.allclose(matrix_matrix_product(A, B), numpy.dot(A, B)))
print(numpy.allclose(matrix_matrix_product(A, (B + C)), matrix_matrix_product(A, B) + matrix_matrix_product(A, C)))
print(numpy.allclose(matrix_matrix_product(A, c * B), c*matrix_matrix_product(A, B)))
print(numpy.allclose(matrix_matrix_product(A, B), matrix_matrix_product(B, A)))
```

#### NumPy Products

NumPy and SciPy contain routines that are optimized to perform matrix-vector and matrix-matrix multiplication.  Given two `ndarray`s you can take their product by using the `dot` function.

```python
n = 10
# Matrix vector with identity
I = numpy.identity(n)
x = numpy.random.random(n)
print(x)
```

```python
print('x = Ix is {}\n'.format(numpy.allclose(x, numpy.dot(I, x))))
print('x - I.dot(x) = {}\n'.format(x-I.dot(x)))
print('I*x = \n{}'.format(I*x))
print(I.dot(x))
```

```python
# Matrix vector product
m = 5
A = numpy.random.random((m, n))
print(numpy.dot(A, x))
```

```python
# Matrix matrix product
B = numpy.random.random((n, m))
print(numpy.dot(A, B))
print()
print(A.dot(B))
```

#### Check non-commutative property of matrix-matrix multiplication

It's easy to demonstrate the non-commutative nature of Matrix-Matrix multiplication with *almost* any two matrices $A$ and $B$.  For a clear demonstration,  consider the two matrices $D$ a diagonal matrix and $A$ a matrix of all 1s

```python
N=5
D = numpy.diag(numpy.array(range(1,N+1)))
A = numpy.ones((N,N))
print(D,'\n')
print(A)
```

```python
print(D.dot(A))
```

```python
print(A.dot(D))
```

### Range and Null-Space

#### Range
- The **range** of a matrix $A \in \mathbb R^{m \times n}$ (similar to any function), denoted as $\text{range}(A)$, is the set of vectors that can be expressed as $A x$ for $x \in \mathbb R^n$.
- We can also then say that that $\text{range}(A)$ is the space **spanned** by the columns of $A$.  In other words the columns of $A$ provide a basis for $\text{range}(A)$, also called the **column space** of the matrix $A$.
- $C(A)$ controls the **existence** of solutions to $A\mathbf{x}=\mathbf{b}$

#### Null-Space
- Similarly the **null-space** of a matrix $A$, denoted $\text{null}(A)$ is the set of vectors $\mathbf{x}$ that satisfy $A \mathbf{x} = \mathbf{0}$.
- $N(A)$ controls the **uniqueness** of solutions to $A\mathbf{x}=\mathbf{b}$
- A similar concept is the **rank** of the matrix $A$, denoted as $\text{rank}(A)$, is the dimension of the column space.  A matrix $A$ is said to have **full-rank** if $\text{rank}(A) = \min(m, n)$.  This property also implies that the matrix mapping is **one-to-one**.

### Inverse

A **non-singular** or **invertible** matrix is characterized as a square matrix with full-rank.  This is related to why we know that the matrix is one-to-one, we can use it to transform a vector $x$ and using the inverse, denoted $A^{-1}$, we can map it back to the original matrix.  The familiar definition of this is
\begin{align*}
    A \mathbf{x} &= \mathbf{b}, \\
    A^{-1} A \mathbf{x} & = A^{-1} \mathbf{b}, \\
    x &=A^{-1} \mathbf{b}.
\end{align*}
Since $A$ has full rank, its columns form a basis for $\mathbb{R}^m$ and the vector $\mathbf{b}$ must be in the column space of $A$.

There are a number of important properties of an invertible matrix $A$.  Here we list them as the following equivalent statements
1. $A$ has a *unique* inverse $A^{-1}$ such that  $A^{-1}A=AA^{-1}=I$
1. $\text{rank}(A) = m$
1. $\text{range}(A) = \mathbb{C}^m$
1. $\text{null}(A) = \mathbf{0}$
1. 0 is not an eigenvalue of $A$
1. $\text{det}(A) \neq 0$

#### Example:  Properties of invertible matrices

Show that given an invertible matrix that the rest of the properties hold.  Make sure to search the `numpy` packages for relevant functions.

```python
m = 5
for n in range(100):
    A = numpy.random.uniform(size=(m, m))
    if numpy.linalg.det(A) != 0:
        break

print('A^{{-1}}*A = \n\n{}\n'.format(numpy.dot(numpy.linalg.inv(A), A)))
print('rank(A) = {}\n'.format(numpy.linalg.matrix_rank(A)))
print("N(A)= {}\n".format(numpy.linalg.solve(A, numpy.zeros(m))))
print("Eigenvalues = {}".format(numpy.linalg.eigvals(A)))
```

### Orthogonal Vectors and Matrices

Orthogonality is a very important concept in linear algebra that forms the basis of many of the modern methods used in numerical computations.

Two vectors are said to be *orthogonal* if their **inner-product** or **dot-product** defined as
$$
    < \mathbf{x}, \mathbf{y} > \equiv (\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^T\mathbf{y} \equiv \mathbf{x} \cdot \mathbf{y} = \sum^m_{i=1} x_i y_i = 0
$$
Here we have shown the various notations you may run into (the inner-product is in-fact a general term for a similar operation for mathematical objects such as functions).

If $\langle \mathbf{x},\mathbf{y} \rangle = 0$ then we say $\mathbf{x}$ and $\mathbf{y}$ are orthogonal.  The reason we use this terminology is that the inner-product of two vectors can also be written in terms of the angle between them where

$$
    \cos \theta = \frac{\langle \mathbf{x}, \mathbf{y} \rangle}{||\mathbf{x}||_2~||\mathbf{y}||_2}
$$

and $||\mathbf{x}||_2$ is the Euclidean ($\ell^2$) norm of the vector $\mathbf{x}$, which we can interpret as the *length* of a vector.

We can write the 2-norm or length of a vector  in terms of the inner-product as well as
$$
    ||\mathbf{x}||_2^2 = \langle \mathbf{x}, \mathbf{x} \rangle = \mathbf{x}^T\mathbf{x} = \sum^m_{i=1} |x_i|^2.
$$

$$
    ||\mathbf{x}||_2 = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle}
$$

The generalization of the inner-product to complex spaces is defined as
$$
    \langle x, y \rangle = \sum^m_{i=1} x_i^* y_i
$$
where $x_i^*$ is the complex-conjugate of the value $x_i$.

#### Orthonormality

Taking this idea one step further we can say a set of vectors $\mathbf{x} \in X$ are orthogonal to $\mathbf{y} \in Y$ if $\forall \mathbf{x},\mathbf{y}$ $< \mathbf{x}, \mathbf{y} > = 0$.  If $\forall \mathbf{x},\mathbf{y}$ $||\mathbf{x}|| = 1$ and $||\mathbf{y}|| = 1$ then they are also called orthonormal.

Note that we dropped the 2 as a subscript to the notation for the norm of a vector. Later we will explore other ways to define a norm of a vector other than the Euclidean norm defined above.

Another concept that is related to orthogonality is linear-independence.  A set of vectors $\mathbf{x} \in X$ are **linearly independent** if $\forall \mathbf{x} \in X$ that each $\mathbf{x}$ cannot be written as a linear combination of the other vectors in the set $X$.

An equivalent statement is that given a set of $n$ vectors $\mathbf{x}_i$,  the only  set of scalars $c_i$ that satisfies
$$
    \sum_{i=1}^n c_i\mathbf{x}_i = \mathbf{0}
$$
is if $c_i=0$ for all $i\in[1,n]$

Show that if a set of vectors are orthonormal,  they must be linearly independent.

This can be related directly through the idea of projection.  If we have a set of vectors $\mathbf{x} \in X$ we can project another vector $\mathbf{v}$ onto the vectors in $X$ by using the inner-product.  This is especially powerful if we have a set of **orthogonal** vectors $X$, which are said to **span** a space (or provide a **basis** for a space), s.t. any vector in the space spanned by $X$ can be expressed as a linear combination of the basis vectors $X$
$$
    \mathbf{v} = \sum^n_{i=1} \, \langle \mathbf{v}, \mathbf{x}_i \rangle \, \mathbf{x}_i.
$$
Note if $\mathbf{v} \in X$ that
$$
    \langle \mathbf{v}, \mathbf{x}_i \rangle = 0 \quad \forall \mathbf{x}_i \in X \setminus \mathbf{v}.
$$

Looping back to matrices, the column space of a matrix is spanned by its linearly independent columns.  Any vector $v$ in the column space can therefore be expressed via the equation above.  A special class of matrices are called **unitary** matrices when complex-valued and **orthogonal** when purely real-valued if the columns of the matrix are orthonormal to each other.  Importantly this implies that for a unitary matrix $Q$ we know the following

1. $Q^* = Q^{-1}$
1. $Q^*Q = I$

where $Q^*$ is called the **adjoint** (or Hermitian) of $Q$.  The adjoint is defined as the transpose of the original matrix with the entries being the complex conjugate of each entry as the notation implies.  Note, if $Q\in\mathbb{R}^{n\times n}$ then $Q^*=Q^T$

As an example if we have the matrix
$$
\begin{aligned}
    Q &= \begin{bmatrix} q_{11} & q_{12} \\ q_{21} & q_{22} \\ q_{31} & q_{32} \end{bmatrix} \quad \text{then} \\
    Q^* &= \begin{bmatrix} q^*_{11} & q^*_{21} & q^*_{31} \\ q^*_{12} & q^*_{22} & q^*_{32} \end{bmatrix}
\end{aligned}
$$
The important part of being an unitary matrix is that the projection onto the column space of the matrix $Q$ preserves geometry in an Euclidean sense, i.e. preserves the Cartesian distance.

# Vector and Matrix Norms and the condition number

The following sections will lay out the definitions and computation required to calculate a range of vector and matrix norms which provide a measure of the "size" of an object or the distance in some vector space.

In the context of Numerical Linear algebra,  norms are essential for defining a key property of a matrix,  the "condition number".  In infinite precision, a square matrix is either invertible or singular,  however, in finite precision, even an invertible matrix can behave poorly if it is 'ill-conditioned', i.e. it is almost singular.

We need a quantitiative measure of how "near-singular" a matrix is

### Why not the Determinant $|A|$?

We know that if a matrix $A$ is singular, $|A|=0$.  But what if $|A|$ is small?

Turns out even a perfectly invertible, well-conditioned matrix can have arbitrarily small determinant.

Consider the two diagonal matrices

$$
    I = \begin{bmatrix} 1 & & \\ & 1 & \\ & & 1\\ \end{bmatrix}\quad\text{and}\quad
    \epsilon I = \begin{bmatrix} \epsilon & & \\ & \epsilon & \\ & & \epsilon \\ \end{bmatrix}
$$

by definition, $|I|=1$,  but $|\epsilon I| = ??$.

More generally if $I\in\mathbb{R}^{n\times n}$, $|\epsilon I| = ??$.  Yet all these matrices are diagonal and easily inverted (i.e.  $(\epsilon I )^{-1} = (1/\epsilon) I$).  Thus we need something better (the condition number). But to get there requires developing important ideas about vector and matrix norms.

### Vector Norms

Norms (and also measures) provide a means for measure the "size" or distance in a space.  In general a norm is a function, denoted by $||\cdot||$, that maps $\mathbb{C}^m \rightarrow \mathbb{R}$.  In other words we stick in a multi-valued object and get a single, real-valued number out the other end.

All norms satisfy the properties:

1. $||\mathbf{x}|| \geq 0$
2. $||\mathbf{x}|| = 0$ only if $\mathbf{x} = \mathbf{0}$
3. $||\mathbf{x} + \mathbf{y}||\leq ||\mathbf{x}|| + ||\mathbf{y}||$ (triangle inequality)
4. $||c \mathbf{x}|| = |c|||\mathbf{x}||$ where $c \in \mathbb{C}$

There are a number of relevant norms that we can define beyond the Euclidean norm, also know as the 2-norm or $\ell_2$ norm:

* $\ell_1$ norm:
$$
    ||\mathbf{x}||_1 = \sum^m_{i=1} |x_i|,
$$

* $\ell_2$ norm:
$$
    ||\mathbf{x}||_2 = \left( \sum^m_{i=1} |x_i|^2 \right)^{1/2},
$$

* $\ell_p$ norm:
$$
    ||\mathbf{x}||_p = \left( \sum^m_{i=1} |x_i|^p \right)^{1/p}, \quad \quad 1 \leq p < \infty,
$$

* $\ell_\infty$ norm:
$$
    ||\mathbf{x}||_\infty = \max_{1\leq i \leq m} |x_i|,
$$

1. weighted $\ell_p$ norm:
$$
    ||\mathbf{x}||_{W_p} = \left( \sum^m_{i=1} |w_i x_i|^p \right)^{1/p}, \quad \quad 1 \leq p < \infty,
$$

These are also related to other norms denoted by capital letters ($L_2$ for instance).  In this case we use the lower-case notation to denote finite or discrete versions of the infinite dimensional counterparts.

#### Example:  Comparisons Between Norms

Compute the norms given some vector $\mathbf{x}$ and compare their values.  Verify the properties of the norm for one of the norms.

```python
def pnorm(x, p):
    """ return the vector p norm of a vector
    parameters:
    -----------
    x: numpy array
        vector
    p: float or numpy.inf
        value of p norm such that ||x||_p = (sum(|x_i|^p))^{1/p} for p< inf
        for infinity norm return max(abs(x))
    returns:
    --------
    pnorm: float
        pnorm of x
    """
    if p == numpy.inf:
        norm = numpy.max(numpy.abs(x))
    else:
        norm = numpy.sum(numpy.abs(x)**p)**(1./p)
    return norm
```

```python
m = 10
p = 4
x = numpy.random.uniform(size=m)
ell_1 = pnorm(x, 1)
ell_2 = pnorm(x, 2)
ell_p = pnorm(x, p)
ell_infty = pnorm(x, numpy.inf)

print('x = {}'.format(x))
print()
print("L_1   = {}\nL_2   = {}\nL_{}   = {}\nL_inf = {}".format(ell_1, ell_2, p, ell_p, ell_infty))
```

```python
y = numpy.random.uniform(size=m)
print()
print("Properties of norms:")

print('x = {}'.format(x))
print('y = {}\n'.format(y))
p = 2
print('||x+y||_{p}         = {nxy}\n||x||_{p} + ||y||_{p} = {nxny}'.format(
    p=p,nxy=pnorm(x+y, p), nxny=pnorm(x, p) + pnorm(y, p)))
c = -0.1
print('||c x||_{} = {}'.format(p,pnorm(c * x, p)))
print('|c||x||_{} = {}'.format(p,abs(c) * pnorm(x, p)))
```

#### Geometric Interpretation

For every $p-$norm we can define a set of 'unit vectors' that is the set of all vectors in $\mathbb{R}^n$ with $||\mathbf{x}||_p = 1$.

For example in $\mathbb{R}^2$, the unit spheres in the 1-,2- and $\infty$-norm look like

```python
import matplotlib.patches as patches

# Note: that this code is a bit fragile to angles that go beyond pi
# due to the use of arccos.

head_width = 0.1
head_length = 1.5 * head_width
def draw_unit_vectors(axes, A, head_width=0.1):
    head_length = 1.5 * head_width
    image_e = numpy.empty(A.shape)
    angle = numpy.empty(A.shape[0])
    image_e[:, 0] = numpy.dot(A, numpy.array((1.0, 0.0)))
    image_e[:, 1] = numpy.dot(A, numpy.array((0.0, 1.0)))
    for i in range(A.shape[0]):
        angle[i] = numpy.arccos(image_e[0, i] / numpy.linalg.norm(image_e[:, i], ord=2))
        axes.arrow(0.0, 0.0, image_e[0, i] - head_length * numpy.cos(angle[i]),
                             image_e[1, i] - head_length * numpy.sin(angle[i]),
                             head_width=head_width, color='b', alpha=0.5)

# comparison of norms
# ============
# 1-norm
# Unit-ball
fig = plt.figure(figsize=(8,6))
#fig.suptitle("1-Norm: $||A||_1 = {}$".format(numpy.linalg.norm(A,ord=1)), fontsize=16)

theta=numpy.linspace(0., 2.*numpy.pi,100)
axes = fig.add_subplot(1, 1, 1, aspect='equal')
axes.plot((1.0, 0.0, -1.0, 0.0, 1.0), (0.0, 1.0, 0.0, -1.0, 0.0), 'r', label='$||\mathbf{x}||_1=1$')
axes.plot(numpy.cos(theta),numpy.sin(theta), 'g', label='$||\mathbf{x}||_2=1$')
axes.plot((1.0, -1.0, -1.0, 1.0, 1.0), (1.0, 1.0, -1.0, -1.0, 1.0), 'b', label='$||\mathbf{x}||_\infty=1$')
draw_unit_vectors(axes, numpy.eye(2))
axes.arrow(0.0, 0.0, 1.0 - head_length * numpy.cos(numpy.pi/4.),
                     1.0 - head_length * numpy.sin(numpy.pi/4.),
                             head_width=head_width, color='k', linestyle='--', alpha=0.5)
axes.set_title("Unit Ball in 1-,2-,and $\infty$ norm")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)
#axes.legend([ '$||\mathbf{x}||_1=1$', '$||\mathbf{x}||_2=1$', '$||\mathbf{x}||_\infty=1$'], loc='best')
axes.legend(loc='upper center', bbox_to_anchor = (0.5,0.5))
plt.show()
```

And if $\mathbf{x} = \begin{bmatrix} 1 \\ 1 \\ \end{bmatrix}$   then $||\mathbf{x}||_1 = ??,~~$
  $||\mathbf{x}||_2 = ??,~~$  and  $||\mathbf{x}||_\infty = ??$

### Induced Matrix Norms

The most direct way to consider a matrix norm is one induced by a vector-norm.  Given a vector norm, we can define a matrix p-norm as the smallest number $C$ that satisfies the inequality
$$
    ||A \mathbf{x}||_{p} \leq C ||\mathbf{x}||_{p} \quad\forall\quad \mathbf{x}\in\mathbb{C}^n
$$
or as the supremum of the ratios so that
$$
    C = ||A||_p = \sup_{\mathbf{x}\in\mathbb{C}^n ~ \mathbf{x}\neq\mathbf{0}} \frac{||A \mathbf{x}||_{p}}{||\mathbf{x}||_p}.
$$

with no loss of generality,  we can restrict $\mathbf{x}$ to the set of all unit vectors $||\mathbf{x}||_p=1$ and interpret the matrix p-norm as the maximum distortion of the "unit sphere"

## Properties of all Matrix Norms (induced and  non-induced)

In general matrix-norms have the following properties whether they are induced from a vector-norm or not:
1. $||A|| \geq 0$ and $||A|| = 0$ only if $A = 0$
1. $||A + B|| \leq ||A|| + ||B||$ (Triangle Inequality)
1. $||c A|| = |c| ||A||$

In addition,  all induced p-norms satisfy the product rules
1. $||AB|| \leq ||A||\,||B||$
1. $||A\mathbf{x}|| \leq ||A||\,||\mathbf{x}||$

## Computation of induced matrix p-norms

With a little work it can be shown that

1. $||A||_1$ = `numpy.linalg.norm(A, ord=1)` is the maximum 1-norm of the Columns of $A$
1. $||A||_2$ = `numpy.linalg.norm(A, ord=2) = max(numpy.linalg.svd(A)[1]` is the maximum Singular Value of $A$
1. $||A||_\infty$ = `numpy.linalg.norm(A, ord=numpy.inf) ` is the maximum 1-norm of the Rows of $A$ (or $||A^T||_1$)

### Examples

#### The Identity Matrix $||I||$
$$
 I = \begin{bmatrix} 1 & & \\ & 1 & \\ & & 1\\ \end{bmatrix}
$$

$||I||_1=||I||_2=||I||_\infty = ??$

```python
for ord in [1,2,numpy.inf]:
    print('||I||_{} = {}'.format(ord,numpy.linalg.norm(I,ord=ord)))
```

#### Diagonal Matrices $||D||$
$$
 D = \begin{bmatrix} d_1 & & \\ & d_2 & \\ & & d_3\\ \end{bmatrix} \quad\text{e.g.}\quad
  \begin{bmatrix} 2 & & \\ & 1 & \\ & & -3\\  \end{bmatrix}
$$

$||D||_1=|D||_2=||D||_\infty = ??$

```python
D = numpy.diag([2,1,-3])
for ord in [1,2,numpy.inf]:
    print('||D||_{} = {}'.format(ord,numpy.linalg.norm(D,ord=ord)))
```

#### Orthogonal Matrices $||Q||_2$ where $Q^*Q=I$

A fundamental property of $Q$ matrices is that they do not change the length of vectors i.e.
$$
 ||Q\mathbf{x}||_2^2 = \mathbf{x}^*Q^*Q\mathbf{x} = \mathbf{x}^*\mathbf{x} = ||\mathbf{x}||_2^2
$$

Therefore
$$
    ||Q||_2 = \sup_{\mathbf{x}\in\mathbb{C}^n, ||\mathbf{x}||_2=1}  ||Q\mathbf{x}||_2 = ??
$$

#### Example: Induced Matrix Norms

Consider the matrix
$$
    A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix}.
$$
Compute the induced-matrix norm of $A$ for the vector norms $\ell_2$ and $\ell_\infty$.

$\ell^2$: For both of the requested norms the unit-length vectors $[1, 0]$ and $[0, 1]$ can be used to give an idea of what the norm might be and provide a lower bound.

$$
    ||A||_2 = \sup_{x \in \mathbb{R}^n} \left( ||A \cdot [1, 0]^T||_2, ||A \cdot [0, 1]^T||_2 \right )
$$

computing each of the norms we have

$$\begin{aligned}
    \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} &= \begin{bmatrix} 1 \\ 0 \end{bmatrix} \\
    \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \end{bmatrix} &= \begin{bmatrix} 2 \\ 2 \end{bmatrix}
\end{aligned}$$

which translates into the norms $||A \cdot [1, 0]^T||_2 = 1$ and $||A \cdot [0, 1]^T||_2 = 2 \sqrt{2}$.  This implies that the $\ell_2$ induced matrix norm of $A$ is at least $||A||_{2} = 2 \sqrt{2} \approx 2.828427125$.

The exact value of $||A||_2$ can be computed using the spectral radius defined as
$$
    \rho(A) = \max_{i} |\lambda_i|,
$$
where $\lambda_i$ are the eigenvalues of $A$.  With this we can compute the $\ell_2$ norm of $A$ as
$$
    ||A||_2 = \sqrt{\rho(A^\ast A)}
$$

Computing the norm again here we find
$$
    A^\ast A = \begin{bmatrix} 1 & 0 \\ 2 & 2 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} = \begin{bmatrix} 1 & 2 \\ 2 & 8 \end{bmatrix}
$$
which has eigenvalues
$$
    \lambda = \frac{1}{2}\left(9 \pm \sqrt{65}\right )
$$
so $||A||_2 \approx 2.9208096$.

The actual  induced 2-norm of a matrix can be derived using the Singular Value Decomposition (SVD) and is simply the largest singular value $\sigma_1$.

**Proof**:
Given that every Matrix $A\in\mathbb{C}^{m\times n}$ can be factored into its SVD (see notebook 10.1):

$$
    A = U\Sigma V^*
$$

where $U\in\mathbb{C}^{m\times m}$ and $V\in\mathbb{C}^{n\times n}$ are unitary matrices with the property $U^*U=I$ and $V^*V=I$ (of their respective sizes) and $\Sigma$ is a real diagonal matrix of singular values $\sigma_1 \geq\sigma_2\geq...\sigma_n\geq 0$.

Then the 2-norm squared of a  square matrix is
$$
    ||A||^2_2 = \sup_{\mathbf{x} \in \mathbb{C}^n ~ ||\mathbf{x}||_2 = 1} ||A \mathbf{x}||_2^2 = \mathbf{x}^TA^*A\mathbf{x}
$$
but $A^*A = V\Sigma^2V^*$ so

\begin{align}
    ||A \mathbf{x}||_2^2 &= \mathbf{x}^*V\Sigma^2V^*\mathbf{x} \\
                         &= \mathbf{y}^*\Sigma^2\mathbf{y} \quad\mathrm{where}\quad \mathbf{y}=V^*\mathbf{x}\\
                         &= \sum_{i=1}^n \sigma_i^2|y_i|^2\\
                         &\leq \sigma_1^2\sum_{i=1}^n |y_i|^2 = \sigma_i^2||\mathbf{y}||_2\\
\end{align}

but if $||\mathbf{x}||_2 = 1$ (i.e. is a unit vector), then so is $\mathbf{y}$ because unitary matrices don't change the length of vectors.  So it follows that
$$
    ||A||_2 = \sigma_1
$$

```python
A = numpy.array([[1, 2], [0, 2]])

#calculate the SVD(A)
U, S, Vt = numpy.linalg.svd(A)

print('Singular_values = {}'.format(S))
print('||A||_2 = {}'.format(S.max()))
print('||A||_2 = {}'.format(numpy.linalg.norm(A, ord=2)))

# more fun facts about the SVD
#print(U.T.dot(U))
#print(Vt.T.dot(Vt))
#print(A - numpy.dot(U,numpy.dot(numpy.diag(S),Vt)))

```

###  Some Quick proofs

* The induced 1-norm is the max of the 1-norm of the **columns** of $A$

Given
$$
    A\mathbf{x} = x_1\mathbf{a}_1 + x_2\mathbf{a}_2 + \ldots + x_n\mathbf{a}_n
$$

where $||\mathbf{x}||_1 = 1$. Then
$$
\begin{align}
||A \mathbf{x}||_1 &= ||  x_1\mathbf{a}_1 + x_2\mathbf{a}_2 + \ldots + x_n\mathbf{a}_n || \\
    &\leq |x_1|\,||\mathbf{a}_1||_1 + |x_2|\,||\mathbf{a}_2||_1 + \ldots + |x_n|\,||\mathbf{a}_n||_1 \quad\text{(triangle rule)}\\
    &\leq \max_{1\leq j\leq n} ||\mathbf{a}_j||_1\sum_{j=1}^n |x_j| =  \max_{1\leq j\leq n} ||\mathbf{a}_j||_1 ||\mathbf{x}||_1\\
    &= \max_{1\leq j\leq n} ||\mathbf{a}_j||_1\\
\end{align}
$$

* The induced 2-norm is the maximum singular value (short version...slightly scrappy)
Given
$$
    A = U\Sigma V^T
$$

where $U^TU = V^TV = I$ and $\Sigma$ is a diagonal matrix with diagonal entries $\sigma_1\geq\sigma_2\geq\ldots\geq\sigma_r>0$

then
$$
\begin{align}
    ||AV||_2 &= ||U\Sigma||_2\\
    ||A||_2||V||_2 &= ||U||_2||\Sigma||_2\\
    ||A||_2 &\leq ||\Sigma||_2 = \sigma_1 \\
\end{align}
$$

* The induce $\infty$-norm is  the max of the 1-norm of **rows** of $A$

$$
||A \mathbf{x}||_\infty = \max_{1 \leq i \leq m} | \mathbf{a}^*_i \mathbf{x} | \leq \max_{1 \leq i \leq m} ||\mathbf{a}^*_i||_1
$$
because the largest unit vector on the unit sphere in the $\infty$ norm is a vector of 1's.

####  Example:

$$
    A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix}.
$$

$$ ||A||_1 = 4, \quad ||A||_\infty = 3$$

```python
# Calculate the 1-norm of A
A = numpy.array([ [ 1, 2], [ 0, 2]])
normA_1 = numpy.max(numpy.sum(numpy.abs(A), axis=0))
print('||A||_1 = {}'.format(normA_1))
print('||A||_1 = {}'.format(numpy.linalg.norm(A, ord=1)))
```

```python
# calculate the 2  norm of A
normA_2 = numpy.max(numpy.linalg.svd(A, compute_uv=False))
print('||A||_2 = {}'.format(normA_2))
print('||A||_2 = {}'.format(numpy.linalg.norm(A, ord=2)))
U,S, V = numpy.linalg.svd(A)
print(U.dot(numpy.diag(S).dot(V)))
```

```python
# calculate the infinity norm of A
normA_inf = numpy.max(numpy.sum(numpy.abs(A), axis=1))
print('||A||_inf = {}'.format(normA_inf))
print('||A||_inf = {}'.format(numpy.linalg.norm(A, ord=numpy.inf)))
```

### The Geometric Picture

One of the most useful ways to think about matrix norms is as a transformation of a unit-ball to an ellipse.  Depending on the norm in question, the norm will be some combination of the resulting ellipse.

#### 1-Norm: $A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \\ \end{bmatrix}$

```python

A = numpy.array([[1, 2], [0, 2]])
#============
# 1-norm
# Unit-ball
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
fig.suptitle("1-Norm:  $||A||_1 = {}$".format(numpy.linalg.norm(A,ord=1)), fontsize=16)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.plot((1.0, 0.0, -1.0, 0.0, 1.0), (0.0, 1.0, 0.0, -1.0, 0.0), 'r')
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.plot((1.0, 2.0, -1.0, -2.0, 1.0), (0.0, 2.0, 0.0, -2.0, 0.0), 'r')
draw_unit_vectors(axes, A, head_width=0.2)

axes.set_title("Images Under A")
axes.grid(True)

plt.show()
```

#### 2-Norm: $A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \\ \end{bmatrix}$

```python

# ============
# 2-norm

# Unit-ball
fig = plt.figure()
fig.suptitle("2-Norm: $||A||_2 = ${:3.4f}".format(numpy.linalg.norm(A,ord=2)),fontsize=16)
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.add_artist(plt.Circle((0.0, 0.0), 1.0, edgecolor='r', facecolor='none'))
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
# Compute some geometry
u, s, v = numpy.linalg.svd(A)
theta = numpy.empty(A.shape[0])
ellipse_axes = numpy.empty(A.shape)
theta[0] = numpy.arccos(u[0][0]) / numpy.linalg.norm(u[0], ord=2)
theta[1] = theta[0] - numpy.pi / 2.0
for i in range(theta.shape[0]):
    ellipse_axes[0, i] = s[i] * numpy.cos(theta[i])
    ellipse_axes[1, i] = s[i] * numpy.sin(theta[i])

axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.add_artist(patches.Ellipse((0.0, 0.0), 2 * s[0], 2 * s[1], theta[0] * 180.0 / numpy.pi,
                                edgecolor='r', facecolor='none'))
for i in range(A.shape[0]):
    axes.arrow(0.0, 0.0, ellipse_axes[0, i] - head_length * numpy.cos(theta[i]),
                         ellipse_axes[1, i] - head_length * numpy.sin(theta[i]),
                         head_width=head_width, color='k')
draw_unit_vectors(axes, A, head_width=0.2)
axes.set_title("Images Under A")
axes.set_xlim((-s[0] + 0.1, s[0] + 0.1))
axes.set_ylim((-s[0] + 0.1, s[0] + 0.1))
axes.grid(True)

plt.show()
```

#### $\infty$-Norm: $A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \\ \end{bmatrix}$

```python
# ============
# infty-norm
# Unit-ball
fig = plt.figure()
fig.suptitle("$\infty$-Norm: $||A||_\infty = {}$".format(numpy.linalg.norm(A,ord=numpy.inf)),fontsize=16)
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.plot((1.0, -1.0, -1.0, 1.0, 1.0), (1.0, 1.0, -1.0, -1.0, 1.0), 'r')
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
# Geometry - Corners are A * ((1, 1), (1, -1), (-1, 1), (-1, -1))
# Symmetry implies we only need two.  Here we just plot two
u = numpy.empty(A.shape)
u[:, 0] = numpy.dot(A, numpy.array((1.0, 1.0)))
u[:, 1] = numpy.dot(A, numpy.array((-1.0, 1.0)))
theta[0] = numpy.arccos(u[0, 0] / numpy.linalg.norm(u[:, 0], ord=2))
theta[1] = numpy.arccos(u[0, 1] / numpy.linalg.norm(u[:, 1], ord=2))

axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.plot((3, 1, -3, -1, 3), (2, 2, -2, -2, 2), 'r')
for i in range(A.shape[0]):
    axes.arrow(0.0, 0.0, u[0, i] - head_length * numpy.cos(theta[i]),
                         u[1, i] - head_length * numpy.sin(theta[i]),
                         head_width=head_width, color='k')

draw_unit_vectors(axes, A, head_width=0.2)
axes.set_title("Images Under A")
axes.set_xlim((-4.1, 4.1))
axes.set_ylim((-3.1, 3.1))
axes.grid(True)

plt.show()
```

#### Cauchy-Schwarz and Hölder Inequalities

Computing matrix norms where $p \neq 1$ or $\infty$ is more difficult unfortunately.  We have a couple of tools that can be useful however.

 - **Cauchy-Schwarz Inequality**:  For the special case where $p=q=2$, for any vectors $\mathbf{x}$ and $\mathbf{y}$
$$
    |\mathbf{x}^*\mathbf{y}| \leq ||\mathbf{x}||_2 ||\mathbf{y}||_2
$$
 - **Hölder's Inequality**:  Turns out this holds in general if given a $p$ and $q$ that satisfy $1/p + 1/q = 1$ with $1 \leq p, q \leq \infty$

$$
    |\mathbf{x}^*\mathbf{y}| \leq ||\mathbf{x}||_p ||\mathbf{y}||_q.
$$

**Note**: this is essentially what we used in the proof of the $\infty-$norm with $p=1$ and $q=\infty$

The most widely used matrix norm not induced by a vector norm is the **Frobenius norm** defined by
$$
    ||A||_F = \left( \sum^m_{i=1} \sum^n_{j=1} |A_{ij}|^2 \right)^{1/2}.
$$

#### Invariance under unitary multiplication

One important property of the matrix 2-norm (and Frobenius norm) is that multiplication by a unitary matrix does not change the product (kind of like multiplication by 1).  In general for any $A \in \mathbb{C}^{m\times n}$ and unitary matrix $Q \in \mathbb{C}^{m \times m}$ we have
\begin{align*}
    ||Q A||_2 &= ||A||_2 \\ ||Q A||_F &= ||A||_F.
\end{align*}

1 http://www.utstat.toronto.edu/~brunner/books/LinearModelsInStatistics.pdf

# The condition number

Finally we have enough machinery to define the condition number $\kappa(A)$ (or often $\mathrm{cond}(A)$) which is simply

$$
    \kappa(A) = ||A||\,||A^{-1}|| \in [1,\infty)
$$

### Examples for simple matrices

* Identity Matrix
$$\kappa(I) = ||I||\,||I^{-1}|| = ??$$

* Diagonal Matrix
$$\kappa(D) = ||D||\,||D^{-1}|| = ??$$

* 2-norm condition number of a matrix
$$\kappa_2(A) = ||A||_2\,||A^{-1}||_2 = \frac{\sigma_1}{\sigma_n}$$

if $A$ is singular, $\kappa_2(A) = ??$

### Back to our initial example

Let
$$
    A = \epsilon I \in \mathbb{R}^{n\times n}
$$

* The Determinant $|A| =\epsilon^n$ (which can be arbitrarily small)
* But
$$
\begin{align}
\kappa(A) &= ||A||\,||A^{-1}|| \\
&= ||\epsilon I||\,||(1/\epsilon)I|| \\
& \leq |\epsilon||1/\epsilon|||I||^2 = 1\\
\end{align}
$$

More generally, it's easy to show that scaling of a matrix does not change its condition number

$$
    \kappa(\alpha A) = ||\alpha A||\,||(1/\alpha)A^{-1}|| =\kappa(A)
$$

### Example

$$
    A = \begin{bmatrix} 1 & 2 \\ 1 +\epsilon & 2\\ \end{bmatrix}\quad \epsilon \geq \epsilon_{mach}
$$

then
$$
    A^{-1} = \frac{-1}{2\epsilon} \begin{bmatrix} 2 & -2 \\ -(1 +\epsilon) & 1\\ \end{bmatrix}
$$

So
$$
    \kappa_1(A) = ||A||_1||A^{-1}||_1 = \frac{4}{2\epsilon}(3 + \epsilon)\sim \frac{6}{\epsilon}
$$

which is very ill-conditioned.

## The condition number and error analysis of $A\mathbf{x}=\mathbf{b}$

The condition number is important in many parts of analysis of numerical linear algebra, but is easily illustrated in understanding the behavior of solutions  of linear systems

Assume that $\mathbf{x}$ is a solution to $A\mathbf{x}=\mathbf{b}$ and we want to understand how a small change in the RHS $\mathbf{b}$  propagates to errors in $\mathbf{x}$

Consider the perturbed problem
$$
    A(\mathbf{x} +\Delta\mathbf{x}) = \mathbf{b} + \Delta\mathbf{b}
$$
which by linearity of Matrix vector multiplication, and that $A\mathbf{x}=\mathbf{b}$ implies that

$$
    A\Delta\mathbf{x} = \Delta\mathbf{b}
$$

or
$$
    \Delta\mathbf{x} = A^{-1}\Delta\mathbf{b}
$$

Given
$$
    \Delta\mathbf{x} = A^{-1}\Delta\mathbf{b}
$$
Taking the norm of both sides implies

$$
    ||\Delta\mathbf{x}|| \leq  ||A^{-1}||\,||\Delta\mathbf{b}||
$$

since $||A\mathbf{x}|| = ||\mathbf{b}||$, it follows that

$$
    \frac{||\Delta\mathbf{x}||}{||A\mathbf{x}||} \leq  ||A^{-1}||\,\frac{||\Delta\mathbf{b}||}{||\mathbf{b}||}
$$

or since $||A\mathbf{x}||\leq||A||\,||\mathbf{x}||$, it follows that

$$
    \frac{||\Delta\mathbf{x}||}{||A||\,||\mathbf{x}||} \leq  ||A^{-1}||\,\frac{||\Delta\mathbf{b}||}{||\mathbf{b}||}
$$

or

$$
    \frac{||\Delta\mathbf{x}||}{||\mathbf{x}||} \leq  \kappa(A)\,\frac{||\Delta\mathbf{b}||}{||\mathbf{b}||}
$$

or the relative error in the solution $\mathbf{x}$ depends on the relative error in the RHS $\mathbf{b}$ times the condition number.

### Example

Consider the problem

$$
    \begin{bmatrix} 1 & 2 \\ 1+\alpha & 2\\ \end{bmatrix}\mathbf{x} = \begin{bmatrix} 1 \\ 1 \\ \end{bmatrix}
$$
for $\alpha = O(\epsilon_{mach})$

```python
alpha = numpy.finfo(float).eps
A = numpy.array([ [ 1, 2], [1 + alpha, 2] ])

# first version
b = numpy.array( [ 1., 1.])
x = numpy.linalg.solve(A,b)
print('b = {}\nx = {}'.format(b,x))
```

```python
# now perturb b by epsilon
bp = numpy.array( [ 1  , 1 - alpha])
xp = numpy.linalg.solve(A,bp)
print("b'={}\nx' = {}".format(bp,xp))
```

```python
# and calculate relative error and condition number
err_b = numpy.linalg.norm(bp-b)/numpy.linalg.norm(b)
err_x = numpy.linalg.norm(xp-x)/numpy.linalg.norm(x)
condA = numpy.linalg.cond(A)
print('k(A) = {}'.format(condA))
print('err_b = {}, err_x = {}, err_x/err_b ={}'.format(err_b, err_x, err_x/err_b))
```


---

# Gauss-Legendre Quadrature implementation in Python

By: Pramod Kumar Yadav (@iampramodyadav)

---

# Legendre Polynomial (symbolic)

[SymPy Documentation](https://docs.sympy.org/latest/tutorial/basic_operations.html)

[Gaussian quadrature](https://en.wikipedia.org/wiki/Gaussian_quadrature)

```python
from sympy import *
import numpy as np
```

```python
x=symbols('x')
```

```python
def Legendre(n,x):
  """
  n: Order of polynomial
  x: Variable
  This function print Legendre polynomial of order n
  """

  x=symbols('x')
  if (n==0):
    return x*0+1.0
  elif (n==1):
    return x
  else:
    return ((2.0*n-1.0)*x*Legendre(n-1,x)-(n-1)*Legendre(n-2,x))/n
```

```python
print(Legendre.__doc__)
```

      n: Order of polynomial
      x: Variable
      This function print Legendre polynomial of order n

```python
Legendre(2,x)
```

$\displaystyle 1.5 x^{2} - 0.5$

# Derivative of the Legendre polynomials

```python
def DLegendre(n,x):
  """
  n: Order of polynomial
  x: Variable
  This function print Derivative of Legendre polynomial of order n
  """
  x=symbols('x')
  if (n==0):
    return x*0
  elif (n==1):
    return x*0+1.0
  else:
    return (n/(x**2-1.0))*(x*Legendre(n,x)-Legendre(n-1,x))
```

```python
print(DLegendre.__doc__)
```

      n: Order of polynomial
      x: Variable
      This function print Derivative of Legendre polynomial of order n

```python
simplify(DLegendre(2,x))
```

$\displaystyle 3.0 x$

# Plot Legendre Polynomial

```python
f=Legendre(3,x)
```

```python
from matplotlib import style
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 14, 8
style.use('ggplot')
plot(f,(x,-1,1),legend=True)
```

```python
f1=Legendre(1,x)
f2=simplify(Legendre(3,x))
```

```python
pl=plot(f1,f2,(x,-1,1),legend=True,show=False)
pl[0].line_color = 'red'
pl[1].line_color = 'green'
pl.show()
```

```python
plot(Legendre(1,x),Legendre(2,x),Legendre(3,x),Legendre(4,x),Legendre(5,x),(x,-1,1))
```

```python
import seaborn as sns
sns.set()
sns.set_style("whitegrid", {'grid.linestyle': '--'})
p=plot(Legendre(1,x),Legendre(2,x),Legendre(3,x),Legendre(4,x),Legendre(5,x),(x,-1,1),title='SymPy plot example',show=False);
p[0].line_color = 'red'
p[1].line_color = 'green'
p[2].line_color = 'yellow'
p[3].line_color = ['b','r']
p.figsize=(14, 8)
p.show()
```

# Legedre Roots

```python
def LegendreRoots(polyorder,tolerance=1e-20):
  """
  polyorder: Order of polynomial
  tolerance: tolerance of error (default: 1e-20)
  This function return Roots of Legendre polynomial
  """
  if polyorder<2:
    err=1 # bad polyorder no roots can be found
  else:
    roots=[]
		# The polynomials are alternately even and odd functions. So we evaluate only half the number of roots.
    for i in range(1,int((polyorder)/2) +1):
      x1=cos(pi*(i-0.25)/(polyorder+0.5))
      error=10*tolerance
      iters=0
      while (error>tolerance) and (iters<1000):
        dx=-Legendre(polyorder,x)/DLegendre(polyorder,x)
        dx=N(dx.subs(x,x1))
        x1=N(x1+dx)
        iters=iters+1
        error=abs(dx)
      #print(roots)
      roots.append(x1)
      #print(roots)
		# Use symmetry to get the other roots
    roots=np.array(roots)
    if polyorder%2==0:
      roots=np.concatenate( (-1.0*roots, roots[::-1]) )
    else:
      roots=np.concatenate( (-1.0*roots, [0.0], roots[::-1]) )
    err=0 # successfully determined roots
  return [roots, err]
```

```python
print(LegendreRoots.__doc__)
```

      polyorder: Order of polynomial
      tolerance: tolerance of error (default: 1e-20)
      This function return Roots of Legendre polynomial

```python
LegendreRoots(5)
```

    [array([-0.906179845938664, -0.538469310105683, 0.0, 0.538469310105683,
            0.906179845938664], dtype=object), 0]

# Weight Coefficient

```python
def GaussLegendreWeights(polyorder):
  """
  polyorder: Order of polynomial
  This function return Weights of Legendre polynomial
  """
  W=[]
  [xis,err]=LegendreRoots(polyorder)
  xis=list(xis)
  if err==0:
    for x1 in xis:
      w=2.0/( (1.0-x**2)*(DLegendre(polyorder,x)**2) )
      w=w.subs(x,x1)
      W.append(w)
      err=0
  else:
    err=1 # could not determine roots - so no weights
  return [W, xis, err]
```

```python
print(GaussLegendreWeights.__doc__)
```

      polyorder: Order of polynomial
      This function return Weights of Legendre polynomial

# Final Result

```python
for Polyorder in range(2,7):
  plt.rcParams['figure.figsize'] = 6.4, 4.8
  plot(Legendre(Polyorder,x),(x,-1,1))

  [W, xis, err]=GaussLegendreWeights(Polyorder)
  print("order:", Polyorder)
  print("Weights:", W)
  print ("Roots    : ",xis)

  print('='*110)
  print('\n\n')
  print('='*110)
```

---


## The stochastic SIkR model

```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
```

```python
%matplotlib inline
import numpy as np
import pyross
import matplotlib.pyplot as plt
#from matplotlib import rc; rc('text', usetex=True)
```

### Simulation of a single stochastic trajectory

```python
M = 1                  # the SIR model has no age structure
Ni = 1000*np.ones(M)   # so there is only one age group
N = np.sum(Ni)         # and the total population is the size of this age group

beta  = 0.2            # infection rate
gI   = 0.05            # recovery rate of asymptomatic infectives
kI   = 5           # recovery rate of symptomatic infectives
fsa   = 1              # the self-isolation parameter

I0 = np.zeros(kI,dtype=int)
I0[0] = 5                     # initially, we have 5 infected in "stage 0"
R0  = np.array([0])     # and assume there are no recovered individuals initially
S0  = N-(np.sum(I0)+R0)    # so that the initial susceptibles are obtained from S + Ia + Is + R = N

# there is no contact structure
def contactMatrix(t):
    return np.identity(M)

# duration of simulation and data file
Tf = 100;  Nt=Tf+1;

# instantiate model
parameters = {'alpha':0., 'beta':beta, 'gI':gI, 'kI':kI,'fsa':fsa}
model = pyross.stochastic.SIkR(parameters, M, Ni)

# simulate model
data = model.simulate(S0, I0, contactMatrix, Tf, Nt)
```

```python
# Plot of the k stages
fig, ax = plt.subplots(1,1,figsize=(10,7))

X = data['X'].T
t_arr = data['t']
for i,e in enumerate(X):
    if i == 0:
        continue
    ax.plot(t_arr,e,label='Stage {0}'.format(i),lw=2)
ax.legend()
ax.set_xlim(np.min(t_arr),np.max(t_arr))
ax.set_xlabel('Time',fontsize=25)
ax.set_ylabel(r'Number of infectives',fontsize=25)
plt.show(fig)
plt.close(fig)
```

```python
# Plot of the sum of all stages, i.e. the total number of infectives

X = data['X'].T

I = np.sum( X[1:,:], axis = 0)
S  = X[0].flatten()
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
plt.plot(t, S/N, '-', color="#348ABD", label='$S$', lw=4)

plt.fill_between(t, 0, I/N, color='#A60628', alpha=0.3)
plt.plot(t, I/N, '-', color='#A60628', label='$I$', lw=4)

R=N-S-I; plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
plt.plot(t, R/N, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.xlabel('Time')
plt.ylabel('Population fractions')
plt.autoscale(enable=True, axis='x', tight=True)
```

Note that, since here sample trajectories for the stochastic model are generated, a slightly different plot will result every time the two cells above are run.

### Averaging over several simulations

If the simulation is run many times, mean and standard deviation of the resulting trajectories can be calculated.

```python
# run 100 simulations, calculate mean and standard deviation

N_runs = 100
trajectories_s = np.zeros([N_runs,Nt,kI+1],dtype=float)
for i in range(N_runs):
    print('Running simulation {0} of {1}'.format(i+1,N_runs),end='\r')
    cur_result = model.simulate(S0, I0, contactMatrix, Tf, Nt)
    trajectories_s[i] = cur_result['X']
t_s = cur_result['t']
```

    Running simulation 100 of 100

```python
# plot means and standard deviations

I = np.sum( trajectories_s[:,:,1:], axis = -1)
mean_s = np.mean(trajectories_s,axis=0)
std_s = np.std(trajectories_s,axis=0)
S_s  = mean_s[:,0].flatten()
I_s = np.mean(I,axis=0).flatten()
dS_s = std_s[:,0].flatten()
dI_s =  np.std(I,axis=0).flatten()
# for the recovered, we still need to calculate mean and variance
R_s_trajectories = N - np.sum( trajectories_s, axis =-1)
R_s = np.mean(R_s_trajectories,axis=0)
dR_s = np.std(R_s_trajectories,axis=0)

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

# stochastic
plt.fill_between(t_s,(S_s - dS_s)/N,(S_s + dS_s)/N,color="#348ABD", alpha=0.2)
plt.plot(t_s, S_s/N, '-', color="#348ABD", label=r'$\langle S\rangle$', lw=4)
plt.fill_between(t_s,(I_s - dI_s)/N,(I_s + dI_s)/N,color="#A60628", alpha=0.2)
plt.plot(t_s, I_s/N, '-', color="#A60628", label=r'$\langle I\rangle$', lw=4)
plt.fill_between(t_s,(R_s - dR_s)/N,(R_s + dR_s)/N,color="dimgrey", alpha=0.2)
plt.plot(t_s, R_s/N, '-', color="dimgrey", label=r'$\langle R\rangle $', lw=4)

plt.legend(fontsize=26,loc='upper right',bbox_to_anchor=(1.3,1)); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
```

While the solid lines denote the instantaneous means, the colored shaded regions denote the standard deviation of the samples from the instantaneous mean.

## Distributions for time-of-infection

By repeatedly simulating a single infective person, we can sample the time it takes until an infective recovers.

The total time until recovery $\tau$ is the sum of the times to get through all stages $i$, i.e.

\begin{equation}
    \tau = \sum_{i=1}^{k} \tau_i,
\end{equation}

where $\tau_i$ is the time spent in stage $i$.

Since the $\tau_i$ are all exponentially distributed, with rate parameter $\lambda = k \gamma_I$, the sum is a Gamma distribution with shape parameter $k$ and scale parameter $1/(k\gamma_I)$.

### $k=1$:

For $k=1$, i.e. a single infective stage, we expect an exponential waiting time until recovery.

```python
beta  = 0.2            # infection rate
gI   = 0.05            # recovery rate of asymptomatic infectives
kI   = 1           # number of stages

I0 = np.zeros(kI,dtype=int)
I0[0] = 1                     # initially, we have 5 infected in "stage 0"
S0  = np.array([0])
Ni = np.array([1],dtype=float)

# instantiate model
parameters = {'alpha':0., 'beta':beta, 'gI':gI, 'kI':kI,'fsa':fsa}
model = pyross.stochastic.SIkR(parameters, M, Ni)

N_runs = 10000
waiting_times = np.zeros(N_runs,dtype=float)
for i in range(N_runs):
    data = model.simulate(S0, I0, contactMatrix, Tf, -1)
    waiting_times[i] = data['t'][-1]
```

```python
bin_edges = np.arange(np.max(waiting_times)+1)-0.5

hist, bin_edges = np.histogram(waiting_times, bins= bin_edges,density=True)
bin_centers = bin_edges[1:] + bin_edges[:-1]
bin_centers /= 2

fig,ax = plt.subplots(1,1,figsize=(10,6))
ax.bar(bin_centers,hist,width=1,
       color='dodgerblue',label='Simulations',
      alpha=0.7)
ax.plot(bin_centers,np.exp(-bin_centers*gI)*gI,
        lw=3,label='Exponential distribution',
       ls='--',color='crimson')
ax.set_ylim(1e-4,1e-1)
ax.set_xlim(-0.5,150)
ax.set_yscale('log')
ax.set_xlabel(r'Waiting time $\tau$')
ax.set_ylabel(r'$P(\tau)$')
ax.legend(loc='best')
plt.show(fig)
plt.close(fig)
```

### $k=100$:

```python
kI   = 100           # number of stages
gI   = 0.05            # recovery rate of asymptomatic infectives

I0 = np.zeros(kI,dtype=int)
I0[0] = 1
S0  = np.array([0])
Ni = np.array([1],dtype=float)

# instantiate model
parameters = {'alpha':0., 'beta':beta, 'gI':gI, 'kI':kI,'fsa':fsa}
model = pyross.stochastic.SIkR(parameters, M, Ni)

N_runs = 10000
waiting_times = np.zeros(N_runs,dtype=float)
for i in range(N_runs):
    data = model.simulate(S0, I0, contactMatrix, Tf, -1)
    waiting_times[i] = data['t'][-1]
```

```python
bin_edges = np.arange(np.floor(np.min(waiting_times)),np.ceil(np.max(waiting_times))+1)-0.5

hist, bin_edges = np.histogram(waiting_times, bins= bin_edges,density=True)
bin_centers = bin_edges[1:] + bin_edges[:-1]
bin_centers /= 2

import scipy.special
shape = kI
scale = 1/(kI*gI)
gamma_dist = bin_centers**(shape-1)*np.exp(-bin_centers/scale)/scale**shape/scipy.special.factorial(shape-1)

fig,ax = plt.subplots(1,1,figsize=(10,6))
ax.bar(bin_centers,hist,width=1,
       color='dodgerblue',label='Simulations',
      alpha=0.7)
ax.plot(bin_centers,gamma_dist,
        label=r'$\Gamma$ distribution',
        marker='o',color='crimson')
#ax.set_yscale('log')
ax.legend() #loc=8)
ax.set_xlabel(r'Waiting time $\tau$')
ax.set_ylabel(r'$P(\tau)$')
plt.show(fig)
plt.close(fig)
```

```python

```