$$Exercise: 1$$

$In\; this\; lab,\; we \;will\; consider \;non-linear \;problems \;which\; can \;be \;converted\; to\; linear\; programs.\; One \;famous\; example\; is\; the \;absolute \;value \;function \;given\; as: $ \\
\begin{align}
|x| = \begin{cases}
        x  & \text{ if } x\geq 0 \\
        -x & \text{ otherwise }.
      \end{cases}
\end{align}

$We\; will \;now \;consider \;the \;absolute\; value\; function \;appearing \;in \;the\; objective\; function.\; Consider \;the\; optimization \;problem: $ \\
$\textbf{(OP)}$:
\begin{align}
\min \ |x_1| + |x_2| + |x_3| + |x_4| + |x_5| &  \nonumber \\
{\rm{s.t.}} \ 85 x_1 + 92 x_2 + 45 x_3 + 27 x_4 + 31 x_5 & \geq 1 \nonumber \\
92 x_1 + 54 x_2 + 22 x_3 + 20 x_4 + 7 x_5 & \geq 1 \nonumber \\
96 x_1 + 67 x_2 + 29 x_3 + 20 x_4 + 11 x_5 & \geq 1 \nonumber \\
-91 x_1 - 57 x_2 - 33 x_3 - 23 x_4 - 12 x_5 & \geq 1 \nonumber \\
-99 x_1 - 75 x_2 - 26 x_3 - 24 x_4 - 41 x_5 & \geq 1 \nonumber \\
-98 x_1 - 99 x_2 - 57 x_3 - 45 x_4 - 65 x_5 & \geq 1 \nonumber
\end{align}

$In\; this\; optimization\; problem \\
\textbf{(OP)}, we \;wish \;to \;minimize\; the \;sum \;of\; absolute \;values\; of\; the \;decision \;variables.\; Such \;problems\; are \;useful \;when \;the \;decision\; variables \;might\; take\; both\; positive\; and\; negative \;values, \;and \;we \;want\; to \;optimize \;the \;magnitudes\; of \;the\; decision \;variables. \\
We \;will\; now \;consider\; two\; possible\; ways\; to\; solve\; this \;problem.$

$\textbf{Approach 1:}$

$We \;can\; use\; a \;substitution \;of\; the\; absolute\; function \;values\; of \;the \;variables \;to \;new \;variables\; as\; u_i = |x_i|, \; \forall i \in \{1,\ldots,5\}. \;With \;this \;substitution, \;note\; that\; we\; have\; u_i\geq 0, \ \forall i \in \{1,\ldots,5\}.\; Also,\; the\; following\; inequalities\; are \;satisfied:\; x_i \leq u_i, -x_i \leq u_i, \ \forall i \in \{1,\ldots,5\}. \;Thus\; we \;can\; transform \;the \;problem \; \textbf{(OP)} \; as\; the \;following \;optimization\; problem\; \textbf{(OP1)}, \;where \;the \;absolute\; values\; of\; x_i \; are\; replaced \;with\; u_i \; and \;constraints \;related\; to \;the\; new \;variables \; u_i \;are\; introduced.\; Thus\; we \;have\; the \;problem \; \textbf{(OP1)}:$

\begin{align}
\min \ u_1 + u_2 + u_3 + u_4 + u_5 &  \nonumber \\
{\rm{s.t.}} \ 85 x_1 + 92 x_2 + 45 x_3 + 27 x_4 + 31 x_5 & \geq 1 \nonumber \\
92 x_1 + 54 x_2 + 22 x_3 + 20 x_4 + 7 x_5 & \geq 1 \nonumber \\
96 x_1 + 67 x_2 + 29 x_3 + 20 x_4 + 11 x_5 & \geq 1 \nonumber \\
-91 x_1 - 57 x_2 - 33 x_3 - 23 x_4 - 12 x_5 & \geq 1 \nonumber \\
-99 x_1 - 75 x_2 - 26 x_3 - 24 x_4 - 41 x_5 & \geq 1 \nonumber \\
-98 x_1 - 99 x_2 - 57 x_3 - 45 x_4 - 65 x_5 & \geq 1 \nonumber \\
u_i \geq x_i, & \  \forall i \in \{1,\ldots,5\} \nonumber \\
u_i \geq -x_i, & \  \forall i \in \{1,\ldots,5\} \nonumber \\
u_i \geq 0, & \ \forall i \in \{1,\ldots,5\}.
\end{align}

$Note\; that \;the\; number\; of \;decision \;variables\; in \; \textbf{(OP1)} is\; twice\; the\; number\; of \;decision\; variables\; in \; \textbf{(OP)}. \;However\; the\; objective\; function\; and\; the \;constraints\; in\;  \textbf{(OP1)} \;are\; now\; linear, \;and\; thus\;  \textbf{(OP1)} \;is\; a\; linear\; program. $

```python
! pip install -q pyomo
```

```python
from pyomo.environ import *
```

```python
import numpy as np
```

```python
coef = np.loadtxt('file31.txt', delimiter=',')
```

```python
model_lab6_ex1 = ConcreteModel()
```

```python
N = coef.shape[1]-1
M = coef.shape[0]-1
obj_coef = coef[0,:-1]
constr_coef = coef[1:,:-1]
constr_rhs = coef[1:,-1]
```

```python
col_indices_x = np.arange(int(N/2))
model_lab6_ex1.x = Var(col_indices_x)
```

```python
col_indices_u = np.arange(int(N/2))
model_lab6_ex1.u = Var(col_indices_u, domain=NonNegativeReals)
```

```python
row_indices = np.arange(M)
```

```python
model_lab6_ex1.constraints = ConstraintList()
```

```python
model_lab6_ex1.objective = Objective(expr = summation(obj_coef[0:int(N/2)],model_lab6_ex1.x) + summation(obj_coef[int(N/2):N],model_lab6_ex1.u), sense=minimize)
```

```python
for i in row_indices:
  model_lab6_ex1.constraints.add(summation(constr_coef[i][0:int(N/2)],model_lab6_ex1.x) + summation(constr_coef[i][int(N/2):N],model_lab6_ex1.u)  >= constr_rhs[i])
```

```python
model_lab6_ex1.pprint()
```

    3 Set Declarations
        constraints_index : Size=1, Index=None, Ordered=Insertion
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :   16 : {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}
        u_index : Size=1, Index=None, Ordered=False
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :    5 : {0, 1, 2, 3, 4}
        x_index : Size=1, Index=None, Ordered=False
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :    5 : {0, 1, 2, 3, 4}

    2 Var Declarations
        u : Size=5, Index=u_index
            Key : Lower : Value : Upper : Fixed : Stale : Domain
              0 :     0 :  None :  None : False :  True : NonNegativeReals
              1 :     0 :  None :  None : False :  True : NonNegativeReals
              2 :     0 :  None :  None : False :  True : NonNegativeReals
              3 :     0 :  None :  None : False :  True : NonNegativeReals
              4 :     0 :  None :  None : False :  True : NonNegativeReals
        x : Size=5, Index=x_index
            Key : Lower : Value : Upper : Fixed : Stale : Domain
              0 :  None :  None :  None : False :  True :  Reals
              1 :  None :  None :  None : False :  True :  Reals
              2 :  None :  None :  None : False :  True :  Reals
              3 :  None :  None :  None : False :  True :  Reals
              4 :  None :  None :  None : False :  True :  Reals

    1 Objective Declarations
        objective : Size=1, Index=None, Active=True
            Key  : Active : Sense    : Expression
            None :   True : minimize :  + (u[0] + u[1] + u[2] + u[3] + u[4])

    1 Constraint Declarations
        constraints : Size=16, Index=constraints_index, Active=True
            Key : Lower : Body                                                             : Upper : Active
              1 :   1.0 :   85.0*x[0] + 92.0*x[1] + 45.0*x[2] + 27.0*x[3] + 31.0*x[4] + () :  +Inf :   True
              2 :   1.0 :    92.0*x[0] + 54.0*x[1] + 22.0*x[2] + 20.0*x[3] + 7.0*x[4] + () :  +Inf :   True
              3 :   1.0 :   96.0*x[0] + 67.0*x[1] + 29.0*x[2] + 20.0*x[3] + 11.0*x[4] + () :  +Inf :   True
              4 :   1.0 : - 91.0*x[0] - 57.0*x[1] - 33.0*x[2] - 23.0*x[3] - 12.0*x[4] + () :  +Inf :   True
              5 :   1.0 : - 99.0*x[0] - 75.0*x[1] - 26.0*x[2] - 24.0*x[3] - 41.0*x[4] + () :  +Inf :   True
              6 :   1.0 : - 98.0*x[0] - 99.0*x[1] - 57.0*x[2] - 45.0*x[3] - 65.0*x[4] + () :  +Inf :   True
              7 :   0.0 :                                                  - x[0] + (u[0]) :  +Inf :   True
              8 :   0.0 :                                                  - x[1] + (u[1]) :  +Inf :   True
              9 :   0.0 :                                                  - x[2] + (u[2]) :  +Inf :   True
             10 :   0.0 :                                                  - x[3] + (u[3]) :  +Inf :   True
             11 :   0.0 :                                                  - x[4] + (u[4]) :  +Inf :   True
             12 :   0.0 :                                                    x[0] + (u[0]) :  +Inf :   True
             13 :   0.0 :                                                    x[1] + (u[1]) :  +Inf :   True
             14 :   0.0 :                                                    x[2] + (u[2]) :  +Inf :   True
             15 :   0.0 :                                                    x[3] + (u[3]) :  +Inf :   True
             16 :   0.0 :                                                    x[4] + (u[4]) :  +Inf :   True

    7 Declarations: x_index x u_index u constraints_index constraints objective

```python
!apt-get install -y -qq coinor-cbc
```

$Q:1\; Using \;cbc \;solver \;to \;solve \;optimization \;problem$

```python
opt_cbc = SolverFactory('cbc')
result = opt_cbc.solve(model_lab6_ex1)
```

$Q:2\;Reporting \;the\; solver \;status\; and \;solver\; termination \;condition$

```python
print('Solver status:', result.solver.status)
print('Solver termination condition:',result.solver.termination_condition)
```

    Solver status: ok
    Solver termination condition: optimal

$Yes, \; the\; solver\; yield\; an\; optimal\;solution $

$Q:3\;Reporting\; the \;optimal \;objective \;function \;value,\;
values\; of \;the \;decision\; variables \;x_i \;and \;the\; new \;variables\; u_i\; and \;the \;constraint\; activities \;at\;the\; optimal\; solution.$

```python
print('\nObjective = ', model_lab6_ex1.objective())

print('\nDecision Variables')
for j in col_indices_x:
    print('x[',j+1,']:', model_lab6_ex1.x[j].value)

for j in col_indices_u:
    print('u[',j+1,']:', model_lab6_ex1.u[j].value)

print('\nConstraints')
model_lab6_ex1.constraints.display()
```

    Objective =  0.547955725

    Decision Variables
    x[ 1 ]: -0.046448575
    x[ 2 ]: 0.18612441
    x[ 3 ]: -0.17131802
    x[ 4 ]: 0.0
    x[ 5 ]: -0.14406472
    u[ 1 ]: 0.046448575
    u[ 2 ]: 0.18612441
    u[ 3 ]: 0.17131802
    u[ 4 ]: 0.0
    u[ 5 ]: 0.14406472

    Constraints
    constraints : Size=16
        Key : Lower : Body               : Upper
          1 :   1.0 : 0.9999996250000001 :  None
          2 :   1.0 : 0.9999997599999997 :  None
          3 :   1.0 :  1.458337769999999 :  None
          4 :   1.0 : 1.0000002550000002 :  None
          5 :   1.0 :  1.000000215000001 :  None
          6 :   1.0 : 5.2549776999999995 :  None
          7 :   0.0 :         0.09289715 :  None
          8 :   0.0 :                0.0 :  None
          9 :   0.0 :         0.34263604 :  None
         10 :   0.0 :                0.0 :  None
         11 :   0.0 :         0.28812944 :  None
         12 :   0.0 :                0.0 :  None
         13 :   0.0 :         0.37224882 :  None
         14 :   0.0 :                0.0 :  None
         15 :   0.0 :                0.0 :  None
         16 :   0.0 :                0.0 :  None

$Observations:\; Here \;constraint \;1 , \;constraint \;2 , \;constraint\; 4 , \;constraint \;5 ,\; constraint\; 8 , \;constraint \;10 ,\; constraint \;12 , \;constraint \;14 , \;constraint \;15 ,\;constraint \;16 \;are\; active\; and\; rest \;of\; the \;constraints \;are\; inactive.$

Recall our optimization problem $\textbf{(OP)}$:
\begin{align}
\min \ |x_1| + |x_2| + |x_3| + |x_4| + |x_5| &  \nonumber \\
{\rm{s.t.}} \ 85 x_1 + 92 x_2 + 45 x_3 + 27 x_4 + 31 x_5 & \geq 1 \nonumber \\
92 x_1 + 54 x_2 + 22 x_3 + 20 x_4 + 7 x_5 & \geq 1 \nonumber \\
96 x_1 + 67 x_2 + 29 x_3 + 20 x_4 + 11 x_5 & \geq 1 \nonumber \\
-91 x_1 - 57 x_2 - 33 x_3 - 23 x_4 - 12 x_5 & \geq 1 \nonumber \\
-99 x_1 - 75 x_2 - 26 x_3 - 24 x_4 - 41 x_5 & \geq 1 \nonumber \\
-98 x_1 - 99 x_2 - 57 x_3 - 45 x_4 - 65 x_5 & \geq 1 \nonumber
\end{align}

Now we will consider a different approach to make the problem $\textbf{(OP)}$ linear.

$\textbf{Approach 2:}$

In this approach we will use some fundamental properties of real numbers. Indeed, for any real number $x$, we can write $x=a-b$ where $a\geq 0, b\geq 0$.

Hence we can substitute $|x| = |a-b|, \ a\geq 0, b\geq 0$.

Now we will consider another interesting property of real numbers:
$|u-v| = u+v \iff u\geq 0, v\geq 0, uv = 0$.

Using this property we can write $|x| = a+b, \ a\geq 0, \ b\geq 0$. Note that this replacement will imply that $ab=0$.

Thus we can transform the optimization problem $\textbf{(OP)}$ into a new optimization problem $\textbf{(OP2)}$ where $x_i = a_i - b_i, \text{ and } |x_i| = a_i + b_i, \ \forall i \in \{1,\dots,5\}$ where $a_i \geq 0, b_i \geq 0, \forall i \in \{1,\dots,5\}$. Thus we have the new optimization problem $\textbf{(OP2)}$:

\begin{align}
\min \ (a_1 + b_1) +  (a_2 + b_2) + (a_3 + b_3) + (a_4 + b_4) + (a_5 + b_5) &  \nonumber \\
{\rm{s.t.}} \ 85 (a_1-b_1) + 92 (a_2-b_2) + 45 (a_3-b_3) + 27 (a_4-b_4) + 31 (a_5-b_5) & \geq 1 \nonumber \\
92 (a_1-b_1) + 54 (a_2-b_2) + 22 (a_3-b_3) + 20 (a_4-b_4) + 7 (a_5-b_5) & \geq 1 \nonumber \\
96 (a_1-b_1) + 67 (a_2-b_2) + 29 (a_3-b_3) + 20 (a_4-b_4) + 11 (a_5-b_5) & \geq 1 \nonumber \\
-91 (a_1-b_1) - 57 (a_2-b_2) - 33 (a_3-b_3) - 23 (a_4-b_4) - 12 (a_5-b_5) & \geq 1 \nonumber \\
-99 (a_1-b_1) - 75 (a_2-b_2) - 26 (a_3-b_3) - 24 (a_4-b_4) - 41 (a_5-b_5) & \geq 1 \nonumber \\
-98 (a_1-b_1) - 99 (a_2-b_2) - 57 (a_3-b_3) - 45 (a_4-b_4) - 65 (a_5-b_5) & \geq 1 \nonumber \\
a_i \geq 0, \ b_i \geq 0, & \ \forall i \in \{1,\ldots,5\}. \nonumber
\end{align}

$Q: 4 - Done$

$Q: 5 - Done$

$Q: 6 - Using\; numpy\;  to\;  load \; the \; .txt\;  file\; contents$

```python
coef1 = np.loadtxt('file310P2.txt',delimiter=',')
```

$Q:7 - Writing\; a \;pyomo\; model\; to\; solve \;problem$

```python
N = coef1.shape[1]-1
M=coef1.shape[0]-1
obj_coef1 = coef1[0,:-1]
constr_coef1 = coef1[1:,:-1]
rhs_coef1 = coef1[1:,-1]
model_lab6_ex1_op2 = ConcreteModel()
col_indices_a = np.arange(int(N/2))
model_lab6_ex1_op2.a = Var(col_indices_a,domain=NonNegativeReals)
col_indices_b = np.arange(int(N/2))
model_lab6_ex1_op2.b = Var(col_indices_b,domain=NonNegativeReals)
row_indices = np.arange(M)
model_lab6_ex1_op2.constraints = ConstraintList()
model_lab6_ex1_op2.objective = Objective(expr = summation(obj_coef1[0:int(N/2)],model_lab6_ex1_op2.a) + summation(obj_coef1[int(N/2):N],model_lab6_ex1_op2.b), sense=minimize)
for i in row_indices:
  model_lab6_ex1_op2.constraints.add(summation(constr_coef1[i][0:int(N/2)],model_lab6_ex1_op2.a) + summation(constr_coef1[i][int(N/2):N],model_lab6_ex1_op2.b)  >= rhs_coef1[i])
```

```python
model_lab6_ex1_op2.pprint()
```

    3 Set Declarations
        a_index : Size=1, Index=None, Ordered=False
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :    5 : {0, 1, 2, 3, 4}
        b_index : Size=1, Index=None, Ordered=False
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :    5 : {0, 1, 2, 3, 4}
        constraints_index : Size=1, Index=None, Ordered=Insertion
            Key  : Dimen : Domain : Size : Members
            None :     1 :    Any :    6 : {1, 2, 3, 4, 5, 6}

    2 Var Declarations
        a : Size=5, Index=a_index
            Key : Lower : Value : Upper : Fixed : Stale : Domain
              0 :     0 :  None :  None : False :  True : NonNegativeReals
              1 :     0 :  None :  None : False :  True : NonNegativeReals
              2 :     0 :  None :  None : False :  True : NonNegativeReals
              3 :     0 :  None :  None : False :  True : NonNegativeReals
              4 :     0 :  None :  None : False :  True : NonNegativeReals
        b : Size=5, Index=b_index
            Key : Lower : Value : Upper : Fixed : Stale : Domain
              0 :     0 :  None :  None : False :  True : NonNegativeReals
              1 :     0 :  None :  None : False :  True : NonNegativeReals
              2 :     0 :  None :  None : False :  True : NonNegativeReals
              3 :     0 :  None :  None : False :  True : NonNegativeReals
              4 :     0 :  None :  None : False :  True : NonNegativeReals

    1 Objective Declarations
        objective : Size=1, Index=None, Active=True
            Key  : Active : Sense    : Expression
            None :   True : minimize : a[0] + a[1] + a[2] + a[3] + a[4] + (b[0] + b[1] + b[2] + b[3] + b[4])

    1 Constraint Declarations
        constraints : Size=6, Index=constraints_index, Active=True
            Key : Lower : Body                                                                                : Upper : Active
              1 :   1.0 :     85.0*a[0] + 92.0*a[1] + 45.0*a[2] + 27.0*a[3] + 31.0*a[4] - 85.0*b[0] - 92.0*b[1] - 45.0*b[2] - 27.0*b[3] - 31.0*b[4] :  +Inf :   True
              2 :   1.0 :       92.0*a[0] + 54.0*a[1] + 22.0*a[2] + 20.0*a[3] + 7.0*a[4] - 92.0*b[0] - 54.0*b[1] - 22.0*b[2] - 20.0*b[3] - 7.0*b[4] :  +Inf :   True
              3 :   1.0 :     96.0*a[0] + 67.0*a[1] + 29.0*a[2] + 20.0*a[3] + 11.0*a[4] - 96.0*b[0] - 67.0*b[1] - 29.0*b[2] - 20.0*b[3] - 11.0*b[4] :  +Inf :   True
              4 :   1.0 : - 91.0*a[0] - 57.0*a[1] - 33.0*a[2] - 23.0*a[3] - 12.0*a[4] + (91.0*b[0] + 57.0*b[1] + 33.0*b[2] + 23.0*b[3] + 12.0*b[4]) :  +Inf :   True
              5 :   1.0 : - 99.0*a[0] - 75.0*a[1] - 26.0*a[2] - 24.0*a[3] - 41.0*a[4] + (99.0*b[0] + 75.0*b[1] + 26.0*b[2] + 24.0*b[3] + 41.0*b[4]) :  +Inf :   True
              6 :   1.0 : - 98.0*a[0] - 99.0*a[1] - 57.0*a[2] - 45.0*a[3] - 65.0*a[4] + (98.0*b[0] + 99.0*b[1] + 57.0*b[2] + 45.0*b[3] + 65.0*b[4]) :  +Inf :   True

    7 Declarations: a_index a b_index b constraints_index constraints objective

$Q:8 - Using \;cbc\; solver\; to\; solve\; the\; model \;for\; optimization \;problem$

```python
result = opt_cbc.solve(model_lab6_ex1_op2)
```

$Q: 9 - Reporting\; the \;solver \;status\; and \;solver \;termination \;condition.$

```python
print('Solver status:', result.solver.status)
print('Solver termination condition:',result.solver.termination_condition)
```

    Solver status: ok
    Solver termination condition: optimal

$Yes, \; the\; solver\; yield\; an\; optimal\;solution $

$Q:10 -  Reporting \;the \;optimal \;objective \;function \;value,\; values\;
of\; the\; decision\; variables \;ai \;and \;bi, \; and \;the \;constraint \;activities\; at\; the \;optimal \;solution.$

```python
print('\nObjective = ', model_lab6_ex1_op2.objective())

print('\nDecision Variables')
for j in col_indices_a:
    print('a[',j+1,']:', model_lab6_ex1_op2.a[j].value)

for j in col_indices_b:
    print('b[',j+1,']:', model_lab6_ex1_op2.b[j].value)

print('\nConstraints')
model_lab6_ex1_op2.constraints.display()
```

    Objective =  0.547955725

    Decision Variables
    a[ 1 ]: 0.0
    a[ 2 ]: 0.18612441
    a[ 3 ]: 0.0
    a[ 4 ]: 0.0
    a[ 5 ]: 0.0
    b[ 1 ]: 0.046448575
    b[ 2 ]: 0.0
    b[ 3 ]: 0.17131802
    b[ 4 ]: 0.0
    b[ 5 ]: 0.14406472

    Constraints
    constraints : Size=6
        Key : Lower : Body               : Upper
          1 :   1.0 :  0.999999625000001 :  None
          2 :   1.0 : 0.9999997599999997 :  None
          3 :   1.0 :         1.45833777 :  None
          4 :   1.0 : 1.0000002549999998 :  None
          5 :   1.0 :        1.000000215 :  None
          6 :   1.0 :  5.254977699999998 :  None

$Here \;constraint\; 1 , \;constraint\; 2 ,\; constraint \;4 ,\; constraint \;5\; are\; active \; and \;rest \;of\; the\; constraints\; are \;inactive.$

$Q: 11-Writing \;code \;to \;print \;the \;original\; variables\; x_i \;from \;a_i\; and\; b_i\; and\; print\; the\; values\; of\; x_i$

```python
model_lab6_ex1_op2.x = Var(col_indices_a, domain=Reals)
```

```python
 for i in col_indices_b:
    model_lab6_ex1_op2.x[i].value=model_lab6_ex1_op2.a[i].value-model_lab6_ex1_op2.b[i].value
```

```python
print('\nObjective = ', model_lab6_ex1_op2.objective())

print('\nDecision Variables')
for i in col_indices_a:
     print('x[',i+1,']:', model_lab6_ex1_op2.x[i].value)

print('\nConstraints')
model_lab6_ex1_op2.constraints.display()
```

    Objective =  0.547955725

    Decision Variables
    x[ 1 ]: -0.046448575
    x[ 2 ]: 0.18612441
    x[ 3 ]: -0.17131802
    x[ 4 ]: 0.0
    x[ 5 ]: -0.14406472

    Constraints
    constraints : Size=6
        Key : Lower : Body               : Upper
          1 :   1.0 :  0.999999625000001 :  None
          2 :   1.0 : 0.9999997599999997 :  None
          3 :   1.0 :         1.45833777 :  None
          4 :   1.0 : 1.0000002549999998 :  None
          5 :   1.0 :        1.000000215 :  None
          6 :   1.0 :  5.254977699999998 :  None

$Q:12-The \;values \;of \;x_i \;obtained \;from\; solving \;OP1 \;are\; same\; as\; the \;values \;of \;x_i \;obtained\; from\; solving \;OP2$

Objective function with absolute value function/Objective function with absolute value function, provided file31 and file310P2.ipynb


# 14 Linear Algebra: Singular Value Decomposition

One can always decompose a matrix $\mathsf{A}$

\begin{gather}
\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}\\
\mathsf{U}^T \mathsf{U} = \mathsf{U} \mathsf{U}^T = 1\\
\mathsf{V}^T \mathsf{V} = \mathsf{V} \mathsf{V}^T = 1
\end{gather}

where $\mathsf{U}$ and $\mathsf{V}$ are orthogonal matrices and the $w_j$ are the _singular values_ that are assembled into a diagonal matrix $\mathsf{W}$.

$$
\mathsf{W} = \text{diag}(w_j)
$$

The inverse (if it exists) can be directly calculated from the SVD:

$$
\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T
$$

## Solving ill-conditioned coupled linear equations

```python
import numpy as np
```

### Non-singular matrix
Solve the linear system of equations

$$
\mathsf{A}\mathbf{x} = \mathbf{b}
$$

Using the standard linear solver in numpy:

```python
A = np.array([
        [1, 2, 3],
        [3, 2, 1],
        [-1, -2, -6],
    ])
b = np.array([0, 1, -1])
```

```python
np.linalg.solve(A, b)
```

    array([ 0.83333333, -0.91666667,  0.33333333])

Using the inverse from SVD:

$$
\mathbf{x} = \mathsf{A}^{-1} \mathbf{b}
$$

```python
U, w, VT = np.linalg.svd(A)
print(w)
```

    [ 7.74140616  2.96605874  0.52261473]

First check that the SVD really factors $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python
U.dot(np.diag(w).dot(VT))
```

    array([[ 1.,  2.,  3.],
           [ 3.,  2.,  1.],
           [-1., -2., -6.]])

```python
np.allclose(A, U.dot(np.diag(w).dot(VT)))
```

    True

Now calculate the matrix inverse $\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T$:

```python
inv_w = 1/w
print(inv_w)
```

    [ 0.1291755   0.33714774  1.91345545]

```python
A_inv = VT.T.dot(np.diag(inv_w)).dot(U.T)
print(A_inv)
```

    [[ -8.33333333e-01   5.00000000e-01  -3.33333333e-01]
     [  1.41666667e+00  -2.50000000e-01   6.66666667e-01]
     [ -3.33333333e-01  -1.38777878e-17  -3.33333333e-01]]

Check that this is the same that we get from `numpy.linalg.inv()`:

```python
np.allclose(A_inv, np.linalg.inv(A))
```

    True

Now, *finally* solve (and check against `numpy.linalg.solve()`):

```python
x = A_inv.dot(b)
print(x)
np.allclose(x, np.linalg.solve(A, b))
```

    [ 0.83333333 -0.91666667  0.33333333]

    True

```python
A.dot(x)
```

    array([ -8.88178420e-16,   1.00000000e+00,  -1.00000000e+00])

```python
np.allclose(A.dot(x), b)
```

    True

### Singular matrix

If the matrix $\mathsf{A}$ is *singular* (i.e., its rank (linearly independent rows or columns) is less than its dimension and hence the linear system of equation does not have a unique solution):

For example, the following matrix has the same row twice:

```python
C = np.array([
     [ 0.87119148,  0.9330127,  -0.9330127],
     [ 1.1160254,   0.04736717, -0.04736717],
     [ 1.1160254,   0.04736717, -0.04736717],
    ])
b1 = np.array([ 2.3674474,  -0.24813392, -0.24813392])
b2 = np.array([0, 1, 1])
```

```python
np.linalg.solve(C, b1)
```

NOTE: failure is not always that obvious: numerically, a matrix can be *almost* singular

```python
D = C.copy()
D[2, :] = C[0] - 3*C[1]
D
```

    array([[ 0.87119148,  0.9330127 , -0.9330127 ],
           [ 1.1160254 ,  0.04736717, -0.04736717],
           [-2.47688472,  0.79091119, -0.79091119]])

```python
np.linalg.solve(D, b1)
```

    array([ -1.70189831e+00,   2.34823174e+16,   2.34823174e+16])

Note that some of the values are huge, and suspiciously like the inverse of machine precision? Sign of a nearly singular matrix.

Now back to the example with $\mathsf{C}$:

#### SVD for singular matrices
If a matrix is *singular* or *near singular* then one can *still* apply SVD.

One can then compute the *pseudo inverse*

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

i.e., any singular $w_j = 0$ is being "augmented" by setting

$$
\frac{1}{w_j} \rightarrow 0 \quad\text{if}\quad w_j = 0
$$

in $\text{diag}(1/w_j)$.

Perform the SVD for the singular matrix $\mathsf{C}$:

```python
U, w, VT = np.linalg.svd(C)
print(w)
```

    [  1.99999999e+00   1.00000000e+00   1.06263691e-33]

Note the third value $w_2 \approx 0$: sign of a singular matrix.

Test that the SVD really decomposes $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python
U.dot(np.diag(w).dot(VT))
```

    array([[ 0.87119148,  0.9330127 , -0.9330127 ],
           [ 1.1160254 ,  0.04736717, -0.04736717],
           [ 1.1160254 ,  0.04736717, -0.04736717]])

```python
np.allclose(C, U.dot(np.diag(w).dot(VT)))
```

    True

There are the **singular values**:

```python
singular_values = np.abs(w) < 1e-12
print(singular_values)
```

    [False False  True]

#### Pseudo-inverse

Calculate the **pseudo-inverse** from the SVD

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

Augment:

```python
inv_w = 1/w
inv_w[singular_values] = 0
print(inv_w)
```

    [ 0.5  1.   0. ]

```python
C_inv = VT.T.dot(np.diag(inv_w)).dot(U.T)
print(C_inv)
```

    [[-0.04736717  0.46650635  0.46650635]
     [ 0.5580127  -0.21779787 -0.21779787]
     [-0.5580127   0.21779787  0.21779787]]

Now solve the linear problem with SVD:

```python
x1 = C_inv.dot(b1)
print(x1)
```

    [-0.34365138  1.4291518  -1.4291518 ]

```python
C.dot(x1)
```

    array([ 2.3674474 , -0.24813392, -0.24813392])

```python
C.dot(x1) - b1
```

    array([  8.88178420e-16,  -1.11022302e-16,  -1.11022302e-16])

Thus, using the pseudo-inverse $\mathsf{C}^{-1}$ we can obtain solutions to the equation

$$
\mathsf{C} \mathbf{x}_1 = \mathbf{b}_1
$$

However, $\mathbf{x}_1$ is not the only solution: there's a whole line of solutions that are formed the special solution and a combination of the basis vectors in the *null space* of the matrix:

The (right) *kernel* or *null space* contains all vectors $\mathbf{x^0}$ for which

$$
\mathsf{C} \mathbf{x^0} = 0
$$

(The dimension of the null space corresponds to the number of singular values.) You can find a basis that spans the null space. Any linear combination of null space basis vectors will also end up in the null space when $\mathbf{A}$ is applied to it.

Specifically, if $\mathbf{x}_1$ is a special solution and $\lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots$ is a vector in the null space then

$$
\mathbf{x} = \mathbf{x}_1 + ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots )
$$

is **also a solution** because

$$
\mathsf{C} \mathbf{x} = \mathsf{C} \mathbf{x^0} + \mathsf{C} ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots ) = \mathsf{C} \mathbf{x^0} + 0 = \mathbf{b}_1 + 0 = \mathbf{b}_1
$$

The $\lambda_i$ are arbitrary real numbers and hence there is an infinite number of solutions.

In SVD:

* The columns $U_{\cdot, i}$ of $\mathsf{U}$ (i.e. `U.T[i]` or `U[:, i]`) corresponding to non-zero $w_i$, i.e. $\{i : w_i \neq 0\}$, form the basis for the _range_ of the matrix $\mathsf{A}$.
* The columns $V_{\cdot, i}$ of $\mathsf{V}$ (i.e. `V.T[i]` or `V[:, i]`) corresponding to zero $w_i$, i.e. $\{i : w_i = 0\}$, form the basis for the _null space_ of the matrix $\mathsf{A}$.

Note that `x1` can be written as a linear combination of `U.T[0]` and `U.T[1]`:

```python
x1
```

    array([-0.34365138,  1.4291518 , -1.4291518 ])

```python
U.T
```

    array([[ -7.07106782e-01,  -4.99999999e-01,  -4.99999999e-01],
           [  7.07106780e-01,  -5.00000001e-01,  -5.00000001e-01],
           [ -8.23369199e-17,  -7.07106781e-01,   7.07106781e-01]])

```python
VT
```

    array([[-0.8660254 , -0.35355339,  0.35355339],
           [-0.5       ,  0.61237244, -0.61237244],
           [-0.        , -0.70710678, -0.70710678]])

```python
U.T[0].dot(x1), U.T[1].dot(x1)
```

    (0.24299822382783764, -0.24299822305983237)

```python
VT[2].dot(x1)
```

    0.0

```python
U.T[0].dot(x1) * U.T[0] + U.T[1].dot(x1) * U.T[1] + 2 * VT[2]
```

    array([-0.34365138, -1.41421356, -1.41421356])

Thus, **all** solutions are
```
x1 + lambda * VT[2]
```

The solution vector $x_2$ is in the null space:

```python
x2 = C_inv.dot(b2)
print(x2)
print(C.dot(x2))
print(C.dot(x2) - b2)
```

    [ 0.9330127  -0.43559574  0.43559574]
    [ -5.55111512e-16   1.00000000e+00   1.00000000e+00]
    [ -5.55111512e-16   2.22044605e-16   2.22044605e-16]

```python
C.dot(10*x2)
```

    array([ -4.44089210e-15,   1.00000000e+01,   1.00000000e+01])

```python
C.dot(VT[2])
```

    array([  0.00000000e+00,  -6.93889390e-18,  -6.93889390e-18])

```python
VT[2]
```

    array([-0.        , -0.70710678, -0.70710678])

```python
null_basis = VT[singular_values]
```

```python
C.dot(null_basis.T)
```

    array([[  0.00000000e+00],
           [ -6.93889390e-18],
           [ -6.93889390e-18]])

## SVD for fewer equations than unknowns
$N$ equations for $M$ unknowns with $N < M$:

* no unique solutions (underdetermined)
* $M-N$ dimensional family of solutions
* SVD: at least $M-N$ zero or negligible $w_j$: columns of $\mathsf{V}$ corresponding to singular $w_j$ span the solution space when added to a particular solution.

Same as the above **Solving ill-conditioned coupled linear equations**.

## SVD for more equations than unknowns
$N$ equations for $M$ unknowns with $N > M$:

* no exact solutions in general (overdetermined)
* but: SVD can provide best solution in the least-square sense
  $$
  \mathbf{x} = \mathsf{V}\, \text{diag}(1/w_j)\, \mathsf{U}^{T}\, \mathbf{b}
  $$
  where

  * $\mathbf{x}$ is a $M$-dimensional vector of the unknowns,
  * $\mathsf{V}$ is a $M \times N$ matrix
  * the $w_j$ form a square $M \times M$ matrix,
  * $\mathsf{U}$ is a $M \times N$ matrix (and $\mathsf{U}^T$ is a $N \times M$ matrix), and
  * $\mathbf{b}$ is the $N$-dimensional vector of the given values

It can be shown that $\mathbf{x}$ minimizes the residual

$$
\mathbf{r} := |\mathsf{A}\mathbf{x} - \mathbf{b}|.
$$

(For a $N \le M$, one can find $\mathbf{x}$ so that $\mathbf{r} = 0$ – see above.)

(In the following, $\mathbf{x}$ will correspond to the $N$ parameter values of the model and $M$ is the number of observations.)

### Linear least-squares fitting

This is the *liner least-squares fitting problem*: Given $N$ data points $(x_i, y_i)$ (where $1 \le i \le N$), fit to a linear model $y(x)$, which can be any linear combination of $M$ functions of $x$.

For example, if we have $N$ functions $x^k$ with parameters $a_k$
$$
y(x) = a_1 + a_2 x + a_3 x^2 + \dots + a_M x^{M-1}
$$
or in general
$$
y(x) = \sum_{k=1}^M a_k X_k(x)
$$

The goal is to determine the $M$ coefficients $a_k$.

Define the **merit function**
$$
\chi^2 = \sum_{i=1}^N \left[ \frac{y_i - \sum_{k=1}^M a_k X_k(x_i)}{\sigma_i}\right]^2
$$
(sum of squared deviations, weighted with standard deviations $\sigma_i$ on the $y_i$).

Best parameters $a_k$ are the ones that *minimize $\chi^2$*.

*Design matrix* $\mathsf{A}$ ($N \times M$, $N \geq M$), vector of measurements $\mathbf{b}$ ($N$-dim) and parameter vector $\mathbf{a}$ ($M$-dim):

\begin{align}
A_{ij} &= \frac{X_j(x_i)}{\sigma_i}\\
b_i &= \frac{y_i}{\sigma_i}\\
\mathbf{a} &= (a_1, a_2, \dots, a_M)
\end{align}

Minimum occurs when the derivative vanishes:
$$
0 = \frac{\partial\chi^2}{\partial a_k} = \sum_{i=1}^N {\sigma_i}^{-2} \left[ y_i - \sum_{k=1}^M a_k X_k(x_i) \right] X_k(x_i), \quad 1 \leq k \leq M
$$
($M$ coupled equations)
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \mathsf{\beta}
\end{align}
with the $M \times M$ matrix
\begin{align}
\alpha_{kj} &= \sum_{i=1}^N \frac{X_j(x_i) X_k(x_i)}{\sigma_i^2}\\
\mathsf{\alpha} &= \mathsf{A}^T \mathsf{A}
\end{align}
and the vector of length $M$
\begin{align}
\beta_{k} &= \sum_{i=1}^N \frac{y_i X_k(x_i)}{\sigma_i^2}\\
\mathsf{\beta} &= \mathsf{A}^T \mathbf{b}
\end{align}

The inverse of $\mathsf{\alpha}$ is related to the uncertainties in the parameters:
$$
\mathsf{C} := \mathsf{\alpha}^{-1}
$$
in particular
$$
\sigma(a_i) = C_ii
$$
(and the $C_{ij}$ are the co-variances).

#### Solution of the linear least-squares fitting problem with SVD
We need to solve the overdetermined system of $M$ coupled equations
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \mathsf{\beta}
\end{align}

SVD finds $\mathbf{a}$ that minimizes
$$
\chi^2 = |\mathsf{A}\mathbf{a} - \mathbf{b}|
$$

The errors are
$$
\sigma^2(a_j) = \sum_{i=1}^{M} \left(\frac{V_{ji}}{w_i}\right)^2
$$

#### Example
Synthetic data

$$
y(x) = 3\sin x - 2\sin 3x + \sin 4x
$$

with noise $r$ added (uniform in range $-5 < r < 5$).

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import numpy as np
```

```python
def signal(x, noise=0):
    r = np.random.uniform(-noise, noise, len(x))
    return 3*np.sin(x) - 2*np.sin(3*x) + np.sin(4*x) + r
```

```python
X = np.linspace(-10, 10, 500)
Y = signal(X, noise=5)
```

```python
plt.plot(X, Y, 'r-', X, signal(X, noise=0), 'k--')
```

```python
def fitfunc(x, a):
    return a[0]*np.cos(x) + a[1]*np.sin(x) + \
           a[2]*np.cos(2*x) + a[3]*np.sin(2*x) + \
           a[4]*np.cos(3*x) + a[5]*np.sin(3*x) + \
           a[6]*np.cos(4*x) + a[7]*np.sin(4*x)

def basisfuncs(x):
    return np.array([np.cos(x), np.sin(x),
                     np.cos(2*x), np.sin(2*x),
                     np.cos(3*x), np.sin(3*x),
                     np.cos(4*x), np.sin(4*x)])
```

```python
M = 8
sigma = 1.
alpha = np.zeros((M, M))
beta = np.zeros(M)
for x in X:
    Xk = basisfuncs(x)
    for k in range(M):
        for j in range(M):
            alpha[k, j] += Xk[k]*Xk[j]
for x, y in zip(X, Y):
    beta += y * basisfuncs(x)/sigma
```

```python
U, w, VT = np.linalg.svd(alpha)
V = VT.T
```

In this case, the singular values do not immediately show if any basis functions are superfluous (this would be the case for values close to 0).

```python
w
```

    array([ 296.92809624,  282.94804954,  243.7895787 ,  235.7300808 ,
            235.15938555,  235.14838812,  235.14821093,  235.14821013])

... nevertheless, remember to routinely mask any singular values or close to singular values:

```python
w_inv = 1/w
w_inv[np.abs(w) < 1e-12] = 0
alpha_inv = V.dot(np.diag(w_inv)).dot(U.T)
```

Compare the fitted values to the original parameters $a_j = 0, +3, 0, 0, 0, -2, 0, +1$.

```python
a_values = alpha_inv.dot(beta)
print(a_values)
```

    [-0.05602761  2.76553973  0.25531225 -0.03780974 -0.05668003 -1.76371356
      0.28272354  0.68902357]

```python
plt.plot(X, fitfunc(X, a_values), 'b-', label="fit")
plt.plot(X, signal(X, noise=0), 'k--', label="signal")
plt.legend(loc="best", fontsize="small")
```

```python

```


```python
import numpy as np
import math
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore")
import sympy as sp
```

```python
x, x_m, h, s=sp.symbols('x x_m h s')
sp.integrate(h*(1-s)*(s),(s,0,1))
```

    h/6

```python
sp.integrate(h*(1-s)*(1-s),(s,0,1))
```

    h/3

```python
x=x_m+h/2*X
b_0=sp.integrate()
```

```python
sp.integrate(h/2*(1+X)*(1-X),(X,0,1))
```

    h/3

```python
sp.integrate(h/8*(1-X)**2,(X,-1,1))
```

Chapter 11 - Finite Element Method/.ipynb_checkpoints/Matrix-checkpoint.ipynb


Let's implement a logistic regression for the lobster problem!  Let's begin with a few imports.

```python
from __future__ import division    # Eliminates annoying default integer division behavior
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

Logistic regression is relatively simple to implement.  To begin with, we need the logistic function $\sigma(\mathbf{w}^T \mathbf{x})$:

```python
# The sigmoid function
def _sigmoid(w,X):
    z = np.dot(X,w)
    return 1./(1+np.exp(-z))
```

Next we'll need an objective function, given by the log-likelihood of the Bernoulli distribution:
\begin{equation}
\mathcal{J}(X,\mathbf{y},\mathbf{w}) = -\sum_{i=1}^m y_i \ln \sigma(\mathbf{w}^T X_i) + (1-y_i) \ln (1-\sigma(\mathbf{w}^T X_i)).
\end{equation}

```python
# The objective function
def _J_fun(w,X):
    return -sum(y*np.log(_sigmoid(w,X)) + (1-y)*np.log(1-_sigmoid(w,X)))
```

Finally, we'll need the gradient of the objective function with respect to the parameter values:
\begin{equation}
\frac{\partial \mathcal{J}}{\partial \mathbf{w}_j} = \sum_{i=1}^m (\sigma(\mathbf{w}^T X_i) - \mathbf{y}_i) X_{ij}.
\end{equation}

```python
# The gradient of the objective function
def _gradient_fun(w,X):
    return np.dot(_sigmoid(w,X)-y,X)
```

With these functions in hand, we just need to import some data:

```python
data = pd.read_csv('lobster_survive.dat',header=0,sep=r"\s{2,}",engine='python')
x = data['Len'].as_matrix().astype(float)
y = data['Survive'].as_matrix().astype(float)
N = len(y)
```

generate a design matrix $X$ (the Vandermonde matrix):

```python
X = np.vander(x,2,increasing=True)
```

To implement gradient descent, we need a learning rate $\eta$.  A bit of experimentation has shown that using a tensor-valued $\eta$ works well

```python
eta = np.array([[0.000001,0],[0,0.000000001]])
```

We need to specify a stopping criterion:  in this case let's just run the thing for $2\times 10^4$ and hope that we've found the optimal value afterwards.

```python
N_iterations = 200000
```

Since gradient descent is an iterative algorithm, we need to specify some values of $\mathbf{w}$ as an initial guess.

```python
w = np.array([-1.,0.5])
```

Finally, to do gradient descent, we just repeatedly evaluate the gradient, scale it with the learning rate, and update the parameters

```python
# Do N_iterations rounds of gradient descent
for i in range(N_iterations):
    grad_w = _gradient_fun(w,X)    # Compute the gradient of the objective function
    w -= np.dot(eta,grad_w)
```

```python
w
```

    array([-8.63293781,  0.21401289])

We can easily plot our modelled class probabilities over the data

```python
plt.plot(x,y+np.random.randn(len(y))*0.02,'b.')  # Adding Gaussian noise so that we can see the density of points
plt.plot(x,_sigmoid(w,X),'k')
plt.xlabel('Lobster Size (cm)')
plt.ylabel('Probability of survival')
plt.show()
```

We can also compute the error in our classification by noting that if $\sigma(\mathbf{w}^T\mathbf{x})>0.5$, then we'll predict survival.

```python
classification_error = sum((_sigmoid(w,X)>0.5)==y)/len(y)
print classification_error
```

    0.710691823899371

71%.  Better than guessing, but not great.  This isn't surprising because these data are obviously not seperable!

```python

```


# Case study 1: Diffusion of fluid pressure and seismicity below Mt. Hood

We will apply our new transient model to study the relation between fluid pressure and seismicity in the crust below an active volcano, Mt. Hood in Oregon, USA. We will follow a publication by Saar and Manga (2003). The central claim of this paper is that there appears to be a correlation between seasonal recharge and watertable changes and seismicity in the upper crust. Saar and Manga suggest that this may be due to the effect of fluid pressure on seismicity. Groundwater recharge increases the watertable, which in turn increases pore fluid pressure in the crust. High fluid pressure means a lower effective normal stress on fault planes, which makes faults more likely to fail and generate seismic activity. We will use our 1D model to model the seasonal change in fluid pressure in the crust, and will try to quantify the magnitude of changes at a depth of 4500 m, where seismic activity is thought to originate.

## Workflow

1. Experiment with the timestep size to get a model that is stable
2. Model the effect of an instantaneous increase in recharge and record how long it takes before it reaches the depth where seismic activity occurs (4500 m below the surface)
3. Adjust hydraulic conductivity and storativity to find the optimal values that correspond to the phase shift between the recharge events and the spike in seimsic activity (151 days)
4. Implement a periodic boundary condition to simulate seasonal recharge and calculate the pore-pressure change at depth due to seismic activity

## 1. Keeping the model stable

The solution that we are using here is a so-called explicit finite difference solution. This is, next to the steady-state solution, the easiest and most intuitive approach to solving partial differential equations and simulating physical processes. However, explicit solutions have one drawback, the solution can become numerically unstable at large timestep sizes ($\Delta t$). We can test this experimentally. Increase the value of the timestep `dt` and run your model again. Keep on increasing the timestep until you get weird results. Record the timestep at which this occurs. Bonus points for the most artistic figure of unstable model results.

The timestep value at which these solutions become numerically unstable is predictable and has been quantified by three mathematicians from Göttingen, Courant, Friedrichs and Lewy in 1928. The stability condition is therefore also called the CFL or Courant number, and follows:

\begin{equation}
    CFL = \dfrac{q \Delta t}{\Delta x}
\end{equation}

The numerical solution becomes unstable for values of CFL that exceed 1.

**Assignment 1: Run the model several times and adjust the timestep size until the model becomes unstable. Choose a timestep size that is still stable, but large enough for the model to finish relatively fast. What timestep did you choose? Make a figure of an unstable model result.**

## 2. Model the effects of an instantaneous recharge event

We will start by setting up our numerical model and simulating the effect of an instantaneous increase in hydraulic head at the surface. The seasonal recharge at Mt. Hood is approximately equal to a layer of water of 1.5 m. The volcanic rocks at the surface have a specific yield of approximately 0.15. This means that a recharge of 1.5 m will increase the watertable by 1.5/0.15 = 10 m.

We will try to implement an instantaneous change in hydraulic head at the surface by adjusting the top boundary condition *after* running the steady-state solution first. Add a line where you change the value of the hydraulic head at the surface to 10 m. Note that the variable controlling hydraulic head is an array instead of a single number like in exercise 2. This is because we want to make a boundary condition that varies over time later on.

**Assignment 2 Model the effects of an instantaneous increase in hydraulic head. How long does it take for the change to reach depths of 4500 m, where seismic activity occurs?**

## 3. Adjusting storativity and hydraulic conductivity

Next we will try to make our model more realistic. Instead of a single number for storativity ``S`` calculate the storativity using the specific storage equation given in your lecture handout (lecture 7). Note that we do not directly model the change in watertable. All the nodes in our model are assumed to be located below the watertable, and therefore we can ignore specific yield and assume that storativity is equal to the specific storage.

**Assignment 3: Implement the full equation for storativity in the model code and replace the line that currently assigns a value to ``S``. Use a reasonable value for fluid density, fluid compressibility and the compressibility of the rock matrix. Describe which parameter values you use and which value of storativity this results in. (see equation 16 and the parameter values mentioned below this equation. Use 1000 kg/m3 as density).**

The observed peak in seismicity underneath Mt. Hood shows a delay of approximately 151 days when compared to streamflow peaks at this location. Streamflow can be regarded as an indicator of seasonal groundwater recharge by snowmelt. The offset of peak seismicity may be an indicator of the time it takes for the fluid pressure increase at the surface to affect fluid pressures at 4500 m depth.

**Assignment 4: Calibrate your model by adjusting ``K`` until the observed delay in increase in hydraulic head at 4500 m compared to the surface equals the observed delay in peak seismicity of ~151 days. Make figures of the model results. Which parameter value for K did you use for the final calibrated model?**

Note that changing K and/or S may make the model unstable again in some cases. This can be fixed by decreasing the timestep size.

## 4. Adding periodic boundary conditions

So far we have modeled the effect of an instantaneous groundwater recharge event. A more realistic way to model the effects of seasonal recharge is to include a periodic change in hydraulic head at the top boundary. Saar and Manga (2003) approximate this by using a cosine function (see equation 8). Uncomment the line where a periodic boundary condition is assigned in this notebook (in the transient parameters box below) and complete the line. Add a new box below, copy-paste the line to calculate the boundary condition and inspect the variable by typing ``print(h0)`` in the next line. Or ``print(h0[:100])`` if you want to inspect the first 100 values for instance. Make sure that you are modelling an increase from 0 to 10 m by checking the min and max values of ``h0`` by typing ``h0.min()`` and ``h0.max()``.

**Assignment 5 Run the model with the new boundary condition and make a figure of the result. What is the seasonal change in hydraulic head at a depth of 4500 m? What is the corresponding change in pore pressure (Pa)? Is this difference high or low compared to the difference between hdyraostatic and lithostatic pressure at these depths? (note, see lecture handouts for lecture on compaction, or ask the dutch guy in front of the classroom)**

# The actual model code:

## import python modules

```python
import matplotlib

%matplotlib inline

import numpy as np
import matplotlib.pyplot as pl
```

## Function to calculate steady-state hydraulic head

```python
def run_steady_state_model(x, dx, K, u0, W, n_iter=20000):

    C = (W * dx**2) / K

    u_new = np.ones_like(x) * u0
    u_old = np.ones_like(x) * u0

    # iterative solution steady-state h

    for t in range(n_iter):
    # make sure you indent anything below a for loop

        # set bnd conditions
        # left bnd:
        u_new[0] = u0
        # right bnd:
        u_new[-1] = 0.5 * (C[-1] + u_old[-1] + u_old[-2])

        # middle nodes:
        u_new[1:-1] = 0.5 * (C[1:-1] + u_old[2:] + u_old[:-2])

        u_old = u_new.copy()

    return u_new
```

## Model parameters

```python
day = 24.0 * 60.0 * 60.0
year = 365 * 24 * 60 * 60.0

L = 10000.0
dx = 100.0

# top boundary condition for the initial steady-state model
h0 = 0

# hydraulic conductivity
K = 1e-6
```

## Set up arrays

```python
# calculate depth of each node
z = np.arange(0, L + dx, dx)

# source term, we keep this at 0 for this exercise:
W = np.zeros_like(z)
```

## Parameters for transient model runs

```python
# specify storage coefficient
# for unconfined groundwater flow this is the specific yield
# for confined this is the specific storage
porosity = 0.1
density = 1000.0
g = 9.81
S = 1e-4

# formulate storativity using the compressibilites of water and the rock matrix
# uncomment and complete the following lines
# compressibility_water = ....
# compressibility_rocks = ....
# fluid_density = 1000.0
# g = 9.81
# S = ... a function of porosity, compressibility_water, compressibility_rocks, g, density

# timestep size
dt = 5.0 * day

#total duration
duration = 2.0 * year

# calculate total number of timesteps
n_timesteps = int(duration / dt)
```

## run steady-state model

We will first run the steady state model. The steady state value of h will be used as an initial value for the transient model runs

```python
h_steady = run_steady_state_model(z, dx, K, h0, W)
```

## Set up parameters for the transient model

```python
# set up array that records timesteps:
time = np.arange(n_timesteps) * dt

# define array to store flux and the variable over time:
n_nodes = len(z)
h = np.zeros((n_timesteps, n_nodes))

# set the steady-state value of u as value for first timestep
h[0] = h_steady

# increase the hydraulic head in the upper node:
h0 = np.zeros(n_timesteps)

# uncomment the following line to change the hydraulic head at the top
# boundary for the transient model runs:
h0[:] = 10

# uncomment the next lines to add a periodic boundary condition
#period = 1 * year
#amplitude = 10.0
#h0 = ... add a variation of equation 8 in the Saar and Manga (2003) paper here, in python language.
# use the variables time, period, amplitude
# use np.pi for the number pi, np.cos(x) for the cosine of a variable x (replace x with your own desired function or number),
# and use np.sin() for sine.
```

## run the transient model:

```python
for j in range(1, n_timesteps):

    # calculate the flux between nodes
    q = -K * (h[j-1, 1:] - h[j-1, :-1]) / dx

    # set specified variable value at the left-hand node:
    h[j, 0] = h0[j]

    # implement no-flow boundary condition at right-hand side:
    q_right = 0.0
    h[j, -1] = h[j-1, -1] + (dt/S) * (-(q_right - q[-1])/dx) + (dt / S) * W[-1]

    # update nodes in the middle:
    h[j, 1:-1] = h[j-1, 1:-1] + (dt/S)*(-(q[1:] - q[:-1])/dx) + (dt/S) * W[1:-1]

    # print results to screen each 100 timesteps
    if j / 1000 == j / 1000.0:
        print('time = ', ((j * dt) / year), ', min, max value of h = ', h[j].min(), h[j].max())

```

    time =  0.0136986301369863 , min, max value of h =  0.0 10.0
    time =  0.0273972602739726 , min, max value of h =  0.0 10.0
    time =  0.0410958904109589 , min, max value of h =  0.0 10.0
    time =  0.0547945205479452 , min, max value of h =  0.0 10.0
    time =  0.0684931506849315 , min, max value of h =  0.0 10.0
    time =  0.0821917808219178 , min, max value of h =  0.0 10.0
    time =  0.0958904109589041 , min, max value of h =  0.0 10.0
    time =  0.1095890410958904 , min, max value of h =  0.0 10.0
    time =  0.1232876712328767 , min, max value of h =  0.0 10.0
    time =  0.136986301369863 , min, max value of h =  0.0 10.0
    time =  0.1506849315068493 , min, max value of h =  0.0 10.0
    time =  0.1643835616438356 , min, max value of h =  0.0 10.0
    time =  0.1780821917808219 , min, max value of h =  0.0 10.0
    time =  0.1917808219178082 , min, max value of h =  0.0 10.0
    time =  0.2054794520547945 , min, max value of h =  0.0 10.0
    time =  0.2191780821917808 , min, max value of h =  0.0 10.0
    time =  0.2328767123287671 , min, max value of h =  0.0 10.0
    time =  0.2465753424657534 , min, max value of h =  0.0 10.0
    time =  0.2602739726027397 , min, max value of h =  0.0 10.0
    time =  0.273972602739726 , min, max value of h =  0.0 10.0
    time =  0.2876712328767123 , min, max value of h =  0.0 10.0
    time =  0.3013698630136986 , min, max value of h =  0.0 10.0
    time =  0.3150684931506849 , min, max value of h =  0.0 10.0
    time =  0.3287671232876712 , min, max value of h =  0.0 10.0
    time =  0.3424657534246575 , min, max value of h =  0.0 10.0
    time =  0.3561643835616438 , min, max value of h =  0.0 10.0
    time =  0.3698630136986301 , min, max value of h =  0.0 10.0
    time =  0.3835616438356164 , min, max value of h =  0.0 10.0
    time =  0.3972602739726027 , min, max value of h =  0.0 10.0
    time =  0.410958904109589 , min, max value of h =  0.0 10.0
    time =  0.4246575342465753 , min, max value of h =  0.0 10.0
    time =  0.4383561643835616 , min, max value of h =  0.0 10.0
    time =  0.4520547945205479 , min, max value of h =  0.0 10.0
    time =  0.4657534246575342 , min, max value of h =  0.0 10.0
    time =  0.4794520547945205 , min, max value of h =  0.0 10.0
    time =  0.4931506849315068 , min, max value of h =  0.0 10.0
    time =  0.5068493150684932 , min, max value of h =  0.0 10.0
    time =  0.5205479452054794 , min, max value of h =  0.0 10.0
    time =  0.5342465753424658 , min, max value of h =  0.0 10.0
    time =  0.547945205479452 , min, max value of h =  0.0 10.0
    time =  0.5616438356164384 , min, max value of h =  0.0 10.0
    time =  0.5753424657534246 , min, max value of h =  0.0 10.0
    time =  0.589041095890411 , min, max value of h =  0.0 10.0
    time =  0.6027397260273972 , min, max value of h =  0.0 10.0
    time =  0.6164383561643836 , min, max value of h =  0.0 10.0
    time =  0.6301369863013698 , min, max value of h =  0.0 10.0
    time =  0.6438356164383562 , min, max value of h =  0.0 10.0
    time =  0.6575342465753424 , min, max value of h =  0.0 10.0
    time =  0.6712328767123288 , min, max value of h =  0.0 10.0
    time =  0.684931506849315 , min, max value of h =  0.0 10.0
    time =  0.6986301369863014 , min, max value of h =  0.0 10.0
    time =  0.7123287671232876 , min, max value of h =  0.0 10.0
    time =  0.726027397260274 , min, max value of h =  0.0 10.0
    time =  0.7397260273972602 , min, max value of h =  0.0 10.0
    time =  0.7534246575342466 , min, max value of h =  0.0 10.0
    time =  0.7671232876712328 , min, max value of h =  0.0 10.0
    time =  0.7808219178082192 , min, max value of h =  0.0 10.0
    time =  0.7945205479452054 , min, max value of h =  0.0 10.0
    time =  0.8082191780821918 , min, max value of h =  0.0 10.0
    time =  0.821917808219178 , min, max value of h =  0.0 10.0
    time =  0.8356164383561644 , min, max value of h =  0.0 10.0
    time =  0.8493150684931506 , min, max value of h =  0.0 10.0
    time =  0.863013698630137 , min, max value of h =  0.0 10.0
    time =  0.8767123287671232 , min, max value of h =  0.0 10.0
    time =  0.8904109589041096 , min, max value of h =  0.0 10.0
    time =  0.9041095890410958 , min, max value of h =  0.0 10.0
    time =  0.9178082191780822 , min, max value of h =  0.0 10.0
    time =  0.9315068493150684 , min, max value of h =  0.0 10.0
    time =  0.9452054794520548 , min, max value of h =  0.0 10.0
    time =  0.958904109589041 , min, max value of h =  0.0 10.0
    time =  0.9726027397260274 , min, max value of h =  0.0 10.0
    time =  0.9863013698630136 , min, max value of h =  0.0 10.0
    time =  1.0 , min, max value of h =  0.0 10.0
    time =  1.0136986301369864 , min, max value of h =  0.0 10.0
    time =  1.0273972602739727 , min, max value of h =  0.0 10.0
    time =  1.0410958904109588 , min, max value of h =  0.0 10.0
    time =  1.0547945205479452 , min, max value of h =  0.0 10.0
    time =  1.0684931506849316 , min, max value of h =  0.0 10.0
    time =  1.082191780821918 , min, max value of h =  0.0 10.0
    time =  1.095890410958904 , min, max value of h =  0.0 10.0
    time =  1.1095890410958904 , min, max value of h =  0.0 10.0
    time =  1.1232876712328768 , min, max value of h =  0.0 10.0
    time =  1.1369863013698631 , min, max value of h =  0.0 10.0
    time =  1.1506849315068493 , min, max value of h =  0.0 10.0
    time =  1.1643835616438356 , min, max value of h =  0.0 10.0
    time =  1.178082191780822 , min, max value of h =  0.0 10.0
    time =  1.1917808219178083 , min, max value of h =  0.0 10.0
    time =  1.2054794520547945 , min, max value of h =  0.0 10.0
    time =  1.2191780821917808 , min, max value of h =  0.0 10.0
    time =  1.2328767123287672 , min, max value of h =  0.0 10.0
    time =  1.2465753424657535 , min, max value of h =  0.0 10.0
    time =  1.2602739726027397 , min, max value of h =  0.0 10.0
    time =  1.273972602739726 , min, max value of h =  0.0 10.0
    time =  1.2876712328767124 , min, max value of h =  0.0 10.0
    time =  1.3013698630136987 , min, max value of h =  0.0 10.0
    time =  1.3150684931506849 , min, max value of h =  0.0 10.0
    time =  1.3287671232876712 , min, max value of h =  0.0 10.0
    time =  1.3424657534246576 , min, max value of h =  0.0 10.0
    time =  1.356164383561644 , min, max value of h =  0.0 10.0
    time =  1.36986301369863 , min, max value of h =  0.0 10.0
    time =  1.3835616438356164 , min, max value of h =  3.5348800511017475e-36 10.0
    time =  1.3972602739726028 , min, max value of h =  5.313631692816152e-35 10.0
    time =  1.4109589041095891 , min, max value of h =  4.702562351399871e-34 10.0
    time =  1.4246575342465753 , min, max value of h =  3.0863830579969544e-33 10.0
    time =  1.4383561643835616 , min, max value of h =  1.6559108715242797e-32 10.0
    time =  1.452054794520548 , min, max value of h =  7.640322532973994e-32 10.0
    time =  1.4657534246575343 , min, max value of h =  3.1291528655245264e-31 10.0
    time =  1.4794520547945205 , min, max value of h =  1.162072933417117e-30 10.0
    time =  1.4931506849315068 , min, max value of h =  3.973726770181997e-30 10.0
    time =  1.5068493150684932 , min, max value of h =  1.2656787237550293e-29 10.0
    time =  1.5205479452054795 , min, max value of h =  3.788787813944579e-29 10.0
    time =  1.5342465753424657 , min, max value of h =  1.0735736265308414e-28 10.0
    time =  1.547945205479452 , min, max value of h =  2.896332787766082e-28 10.0
    time =  1.5616438356164384 , min, max value of h =  7.475654216806541e-28 10.0
    time =  1.5753424657534247 , min, max value of h =  1.8535377639074024e-27 10.0
    time =  1.5890410958904109 , min, max value of h =  4.4300831529537575e-27 10.0
    time =  1.6027397260273972 , min, max value of h =  1.0237148300631569e-26 10.0
    time =  1.6164383561643836 , min, max value of h =  2.2931490763115765e-26 10.0
    time =  1.63013698630137 , min, max value of h =  4.990710683381966e-26 10.0
    time =  1.643835616438356 , min, max value of h =  1.057412582053079e-25 10.0
    time =  1.6575342465753424 , min, max value of h =  2.185026456811923e-25 10.0
    time =  1.6712328767123288 , min, max value of h =  4.41057306624575e-25 10.0
    time =  1.6849315068493151 , min, max value of h =  8.709337287331064e-25 10.0
    time =  1.6986301369863013 , min, max value of h =  1.684581880519881e-24 10.0
    time =  1.7123287671232876 , min, max value of h =  3.1954260163651656e-24 10.0
    time =  1.726027397260274 , min, max value of h =  5.9506167157784476e-24 10.0
    time =  1.7397260273972603 , min, max value of h =  1.0889766431303269e-23 10.0
    time =  1.7534246575342465 , min, max value of h =  1.9601528822971494e-23 10.0
    time =  1.7671232876712328 , min, max value of h =  3.4732511286285386e-23 10.0
    time =  1.7808219178082192 , min, max value of h =  6.063026304830206e-23 10.0
    time =  1.7945205479452055 , min, max value of h =  1.0434147870082653e-22 10.0
    time =  1.8082191780821917 , min, max value of h =  1.7714286877602342e-22 10.0
    time =  1.821917808219178 , min, max value of h =  2.9686177050970946e-22 10.0
    time =  1.8356164383561644 , min, max value of h =  4.913549945320053e-22 10.0
    time =  1.8493150684931507 , min, max value of h =  8.036687946572415e-22 10.0
    time =  1.8630136986301369 , min, max value of h =  1.299614763804055e-21 10.0
    time =  1.8767123287671232 , min, max value of h =  2.0787847067469636e-21 10.0
    time =  1.8904109589041096 , min, max value of h =  3.2904162194893586e-21 10.0
    time =  1.904109589041096 , min, max value of h =  5.156040450766675e-21 10.0
    time =  1.917808219178082 , min, max value of h =  8.00153297845471e-21 10.0
    time =  1.9315068493150684 , min, max value of h =  1.2302096292176574e-20 10.0
    time =  1.9452054794520548 , min, max value of h =  1.8744888194840886e-20 10.0
    time =  1.9589041095890412 , min, max value of h =  2.8315548806160036e-20 10.0
    time =  1.9726027397260273 , min, max value of h =  4.2416860891387943e-20 10.0
    time =  1.9863013698630136 , min, max value of h =  6.303031500820405e-20 10.0

## Some figures:

### A figure of h vs depth:

```python
fig, panel = pl.subplots(1, 1)

#  show change in hydraulic head over time
for j in range(0, n_timesteps, int(n_timesteps/20)):
    if j == 0:
        label = 't=0'
    else:
        label = None
    panel.plot(h[j], z, label=label)

label = 't=%0.1f yr' % (duration/year)
panel.plot(h[-1], z, color='black', lw=1.0, label=label)
panel.legend(loc='upper left', fontsize='small')

panel.set_ylabel('Depth (m)')
panel.set_xlabel('Hydraulic head (m)')
panel.set_ylim(L, 0)

fig.savefig('simulated_h_vs_depth.png')
```

### A figure of h over time for a specific depth

```python
# add the depths that you would like to show in the figures here (m)
target_depths = np.array([4500])

# add colors for the different depths
target_colors = ['blue', 'orange']

# look up the node for the different target depths
target_nodes = (target_depths / dx).astype(int)

fig, panel = pl.subplots(1, 1)

for target_depth, target_node, color in zip(target_depths, target_nodes, target_colors):
    label = '-%0.0f m' % target_depth
    panel.plot(time/year, h[:, target_node], color=color, label=label)

panel.legend()
panel.set_xlabel('Time (years)')
panel.set_ylabel('Hydraulic head (m)')

fig.savefig('simulated_h_over_time.png')
```

# References

Courant, R, K Friedrichs, and H Lewy. 1928. “Über Die Partiellen Differenzengleichungen Der Mathematischen Physik.” Mathematische Annalen 100 (1): 32–74.

Saar, M. O. & Manga, M. Seismicity induced by seasonal groundwater recharge at Mt. Hood, Oregon. Earth Planet. Sci. Lett. 214, 605–618 (2003).

Note, you can find these publications using google scholar: https://scholar.google.com

```python

```


Neural Networks

In this last chapter, we will take a look at the mathematical tools used to model and understand biological neural networks.
We're going to see how to use Python to do **operations on matrices** and then we will take a look at some of the fundamental concepts in **Graph Theory** which will be the starting point for network modeling.

What we're going to do in this Notebook, is that we're going to see how to use SymPy and NumPy to implement some matrix properties and operations, and after that we're going to use another Python Package "NetworkX" which is used to study and implement graphs and networks.

Table of contents

1  Matrices

2  Introduction to Graph Theory

The necessary libraries and functions are :

```python
import numpy as np
import sympy as sp
import networkx as nx
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
from itertools import combinations
from collections import Counter
```

# Matrices

**Matrix addition and substraction**

- If we have the following matrices :

$$ A = \begin{bmatrix} 2 & 5 & 7 \\ 4 & 8 & 3 \\ 9 & 1 & 4 \\\end{bmatrix} \ et \  B = \begin{bmatrix} 1 & 5 & 3 \\ 3 & 6 & 1 \\ 7 & 5 & 9 \\\end{bmatrix}$$
> And we want to find $A+B$ or $A-B$

- We can proceed with NumPy with the following manner :

```python
# Creating the arrays
A = np.array([2,5,7,4,8,3,9,1,4])
B = np.array([1,5,3,3,6,1,7,5,9])
# Reshaping them to a 3x3 array
for mat in (A,B) :
    mat.shape = (3,3) #

print("A + B \n" , A + B, "\nA - B \n", A - B) # Matrix addition and substraction
```

    A + B
     [[ 3 10 10]
     [ 7 14  4]
     [16  6 13]]
    A - B
     [[ 1  0  4]
     [ 1  2  2]
     [ 2 -4 -5]]

- Or we can choose SymPy :

```python
A = sp.Matrix(3, 3,[2, 5, 7, 4, 8, 3, 9, 1, 4])  # First argument is number of row and second is number of columns
B = sp.Matrix(3, 3, [1, 5, 3, 3, 6, 1, 7, 5, 9])
```

>  And the addition/substration are as simple as :

```python
A + B
```

$\displaystyle \left[\begin{matrix}3 & 10 & 10\\7 & 14 & 4\\16 & 6 & 13\end{matrix}\right]$

```python
A - B
```

$\displaystyle \left[\begin{matrix}1 & 0 & 4\\1 & 2 & 2\\2 & -4 & -5\end{matrix}\right]$

**Matrix multiplication or dot product**

- If we take the following vectors and we want to calculate their dot product :

$$ A = \begin{bmatrix} 2 & 3 & 4 \end{bmatrix} \ et \  B = \begin{bmatrix} 4 \\ 1 \\  5 \end{bmatrix}$$

> With NumPy, this can be done using the function np.dot() :

```python
A = np.array([[2,3,4]])
B = np.array([4,1,5])
B.shape = (3,1)
print('A.B \n', np.dot(A,B))
```

    A.B
     [[31]]

> And with SymPy, the multiplication symbol will do the trick :

```python
A = sp.Matrix(1,3, [2,3,4]) # 1x3
B = sp.Matrix(3,1, [4,1,5]) # 3x1
A * B
```

$\displaystyle \left[\begin{matrix}31\end{matrix}\right]$

- Let's take another example

$$ A = \begin{bmatrix} 2 & 8 \\ 5 & -6 \end{bmatrix} \ et \  B = \begin{bmatrix} 7 & -4 \\ 3 & 12 \end{bmatrix}$$

- This time let's calculate $A.B$ and $B.A$

> In SymPy :

```python
A = sp.Matrix(2,2,[2,8,5,-6])
B = sp.Matrix(2,2,[7,-4,3,12])
A*B
```

$\displaystyle \left[\begin{matrix}38 & 88\\17 & -92\end{matrix}\right]$

```python
B*A
```

$\displaystyle \left[\begin{matrix}-6 & 80\\66 & -48\end{matrix}\right]$

> And with NumPy :

```python
A = np.array([2,8,5,-6])
B = np.array([7,-4,3,12])
for mat in (A,B) :
    mat.shape = (2,2)
print('AB \n', np.dot(A,B), '\n BA \n', np.dot(B,A))
```

    AB
     [[ 38  88]
     [ 17 -92]]
     BA
     [[ -6  80]
     [ 66 -48]]

**Solving equations with matrices**

> Let's imagine that we have the following equation :

$$ 2 \begin{bmatrix} x+2 & y+3 \\ 3 & 0 \end{bmatrix} = \begin{bmatrix} 3 & 6 \\ y & z \end{bmatrix}^\top$$

> Solving it at hand would look like this :
    $$\Rightarrow  \begin{bmatrix} 2x+4 & 2y+6 \\ 6 & 0 \end{bmatrix} = \begin{bmatrix} 3 & y \\ 6 & z \end{bmatrix}$$
    $$\Rightarrow \begin{cases}
    z = 0 \\
    y = -6 \\
    x = \frac{-1}{2}
    \end{cases}$$

- But we can use SymPy to solve it in the following manner :

```python
x, y, z = sp.symbols('x,y,z') # Initialize our variable symbols
lhs = 2 * sp.Matrix([[x+2, y+3],[3,0]]) # Create the matrix on the left hand side
lhs
```

$\displaystyle \left[\begin{matrix}2 x + 4 & 2 y + 6\\6 & 0\end{matrix}\right]$

```python
rhs = sp.transpose(sp.Matrix([[3,6],[y,z]])) # Create the matrix on the right hand side
rhs
```

$\displaystyle \left[\begin{matrix}3 & y\\6 & z\end{matrix}\right]$

```python
sp.solve(sp.Eq(lhs,rhs)) # And solve the equality between the two
```

    {x: -1/2, y: -6, z: 0}

**Identity matrix**

$$ A = \begin{bmatrix} 2 & 5 & 7 \\ 4 & 8 & 3 \\ 9 & 1 & 4 \\\end{bmatrix} \ et \  I = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\\end{bmatrix}$$

Let's verify that $AI = A$

> With NumPy :

```python
A = np.array([2,5,7,4,8,3,9,1,4])
A.shape = (3,3)
I = np.eye(3) # Identity matrix 3x3
print('A.I \n', np.dot(A,I))
```

    A.I
     [[2. 5. 7.]
     [4. 8. 3.]
     [9. 1. 4.]]

> And with SymPy :

```python
A = sp.Matrix(3,3,[2,5,7,4,8,3,9,1,4])
I = sp.eye(3)
A * I
```

$\displaystyle \left[\begin{matrix}2 & 5 & 7\\4 & 8 & 3\\9 & 1 & 4\end{matrix}\right]$

```python
sp.Eq(A*I, A) # Verifying that A.I is A
```

$\displaystyle \text{True}$

**Matrix transposition**

With the following matrices, let's verify an important property of matrices and their transpose : $(AB)^\top = B^\top A^\top$
$$ A = \begin{bmatrix} 3 & 2 & 3 \\ 2 & 2 & 1 \\ 2 & 3 & 1 \\\end{bmatrix} \ et \  B = \begin{bmatrix} 2 & -2 \\ -3 & 4 \\ 1 & 1\\\end{bmatrix}$$

> SymPy :

```python
A = sp.Matrix(3,3,[3,2,3,2,2,1,2,3,1])
B = sp.Matrix(3,2,[2,-2,-3,4,1,1])
(A*B).T # We can use sp.transpose(A*B) too
```

$\displaystyle \left[\begin{matrix}3 & -1 & -4\\5 & 5 & 9\end{matrix}\right]$

```python
B.T * A.T
```

$\displaystyle \left[\begin{matrix}3 & -1 & -4\\5 & 5 & 9\end{matrix}\right]$

```python
(A*B).T == B.T * A.T # Let's verify the property
```

    True

> NumPy :

```python
A = np.array([3,2,3,2,2,1,2,3,1])
A.shape = (3,3)
B = np.array([2,-2,-3,4,1,1])
B.shape = (3,2)
print('(AB)T \n', np.dot(A,B).T, '\n BT.AT \n', np.dot(B.T,A.T)) # .T method will give the array transpose
```

    (AB)T
     [[ 3 -1 -4]
     [ 5  5  9]]
     BT.AT
     [[ 3 -1 -4]
     [ 5  5  9]]

```python
np.array_equal(np.dot(A,B).T, np.dot(B.T,A.T)) # Verifying the equality
```

    True

**Matrix Determinant**

- If we want to calculate the determinant for the follwing matrix :
$$ A = \begin{bmatrix} 2 & 3 \\ 4 & -2 \end{bmatrix}$$

> With SymPy :

```python
A = sp.Matrix(2,2,[2,3,4,-2])
A.det()
```

$\displaystyle -16$

> And with NumPy :

```python
A = np.array([[2,3],[4,-2]])
np.linalg.det(A)
```

    -15.999999999999998

- Another example :
$$ I = \begin{bmatrix} 2 & -2 & 4 \\ 1 & -3 & 2 \\ -2 & 5 & 1 \end{bmatrix}$$

The determinant is :

$$det =  +2 \begin{pmatrix} -3 & 2 \\ 5 & 1 \end{pmatrix} - (-2)\begin{pmatrix} 1 & 2 \\ -2 & 1 \end{pmatrix} +4\begin{pmatrix} 1 & -3 \\ -2 & 5 \end{pmatrix}$$
$$\Rightarrow det = +2(-13) +2(5) +4(-1) = -20$$

> Easily with SymPy :

```python
I = sp.Matrix(3,3,[2,-2,4,1,-3,2,-2,5,1])
I.det()
```

$\displaystyle -20$

> And with NumPy :

```python
I = np.array([2,-2,4,1,-3,2,-2,5,1])
I.shape = (3,3)
np.linalg.det(I)
```

    -19.999999999999996

**Matrix inverse**
- If we have the following matrix :

$$ A = \begin{bmatrix} 1 & 1 & 1 \\ 1 & -1 & -1 \\ 1 & 1 & -1 \end{bmatrix}$$

the matrix inverse has a property which is that $AA^{−1} = A^{−1}A = I$.

> We can verify it with SymPy :

```python
A = sp.Matrix(3,3,[1,1,1,1,-1,-1,1,1,-1])
A_1 = A.inv() # Calculating the inverse
A_1
```

$\displaystyle \left[\begin{matrix}\frac{1}{2} & \frac{1}{2} & 0\\0 & - \frac{1}{2} & \frac{1}{2}\\\frac{1}{2} & 0 & - \frac{1}{2}\end{matrix}\right]$

```python
A*A_1 == A_1*A == sp.eye(3) # Verifying the property
```

    True

> And with NumPy, it goes as follows :

```python
A = np.array([1,1,1,1,-1,-1,1,1,-1])
A.shape = (3,3)
A_1 = np.linalg.inv(A)
print('A^-1 \n', A_1)
```

    A^-1
     [[ 0.5  0.5  0. ]
     [-0.  -0.5  0.5]
     [ 0.5 -0.  -0.5]]

```python
if np.array_equal(np.dot(A,A_1), np.dot(A_1,A)) and np.array_equal(np.dot(A,A_1), np.eye(3)) :
    print("The property has been verified")
```

    The property has been verified

**Solving linear systems of equations**

Let's take the following system of equations :

$$x + y + z = 8$$
$$ x−y−z = −4$$
$$ x + y−z = 6$$

> We can solve them in the following manner with NumPy :

```python
a = np.array([[1,1,1],[1,-1,-1],[1,1,-1]])
b = np.array([[8],[-4],[6]])
sol = np.linalg.solve(a,b)
print('The solution is the vector column [x,y,z] \n',sol)
```

    The solution is the vector column [x,y,z]
     [[2.]
     [5.]
     [1.]]

```python
np.array_equal(np.dot(a,sol),b) # Verifying the solution is true
```

    True

**Eigenvalues and Eigenvectors**

- If we want to calculate the eigenvalues and the eignvectors for the following matrix :

$$ B = \begin{bmatrix} 2 & 16 & 8 \\ 4 & 14 & 8 \\ -8 & -32 & -18 \end{bmatrix}$$

```python
B = np.array([2,16,8,4,14,8,-8,-32,-18])
B.shape = (3,3) # A 3x3 matrix
eig_vals, eig_vects = np.linalg.eig(B)
print("The Eigenvalues are \n", eig_vals, "\nAnd the Eigenvectors are \n", eig_vects)
```

    The Eigenvalues are
     [ 2. -2. -2.]
    And the Eigenvectors are
     [[ 0.40824829 -0.84689344  0.29640505]
     [ 0.40824829 -0.05284196  0.36680613]
     [-0.81649658  0.52913064 -0.88181478]]

# Introduction to Graph Theory

**Graph creation and structure description**

- Let's write a function that creates a NetworkX graph and gives the degree distribution as well as the adjacency matrix

```python
def graph(nodes, edges, type="simple"):
    """This function creates a graph and sends back the degree distribution as well
    as the sparse matrix. graph can be simple or oriented by using the parameter type.
    nodes : a list of graph vertices.
    edges : a list of tuples, each tuple contains two vertices to represent an edge.
    """

    if type == "simple":
        g = nx.Graph()
    elif type == "oriented":
        g = nx.DiGraph()

    g.add_nodes_from(nodes)  # Add nodes from the list of nodes
    g.add_edges_from(edges)  # Add edges

    a = nx.adjacency_matrix(g)  # The corresponding adjacency matrix

    degree, freq = zip( \
        *Counter(sorted([deg for node, deg in g.degree()])).items()) # Sorting the degrees and counting them

    fig, ax = plt.subplots(1, 3, figsize=(15, 5), dpi=150)
    plt.subplot(131)  # Drawing the graph
    nx.draw(g, with_labels=True)
    plt.title("Graph")
    plt.subplot(132)  # The degree distribution
    plt.bar(degree, freq, width=0.2)
    plt.xlabel("Degrees")
    plt.ylabel("Frequency")
    plt.xticks(nodes)
    plt.title("Degree distribution of the graph")
    plt.subplot(133)  # The sparse matrix
    plt.spy(a, marker='x')
    plt.xticks([i for i in range(len(nodes))], nodes)
    plt.yticks([i for i in range(len(nodes))], nodes)
    plt.title("Sparse Matrix")

    return g, a  # Graph and adjacency matrix
```

- Now Let's create a graph that has 4 vertices (1,2,3,4) where each vertex is connnected to the other 3.

```python
vertices = [n for n in range(1, 5)]
g1, a1 = graph(vertices, list(combinations(
    vertices, 2)))  # combinations function creates the list of edges needed
```

>  From the degree distribution diagram, we can see that our graph has 4 nodes of degree 3.

- We can also create an oriented graph :

```python
g2, a2 = graph(vertices, [(1,2),(2,1),(2,3),(2,1),(4,1),(4,2),(4,3)], type = "oriented")
```

**Centrality measures**

- Centrality measures in graph theory are very important because they can indicate to some very important aspects of the vertices/nodes in a graph.

- Some of the indicators of centrality are : *Degree centrality*, *Eigenvector centrality*, *Closeness centrality* and *betweenness centrality*.

> Now let's write a function that can represent those types of indicators for a certain graph.

```python
def centrality(graph, centr_type, title):
    """This function creates a represntation for a given centrality type and for a given graph.
    graph : networkx graph object
    centr_type : type of centrality on a given graph, for example : nx.degree_centrality(graph)
    title : string for the title of the representation.
    """

    plt.figure(dpi=150)
    pos = nx.spring_layout(graph)
    nodes = nx.draw_networkx_nodes(
        graph,
        pos,
        node_size=250,
        cmap=plt.cm.plasma,
        node_color=list(centr_type.values()
                        ),  # color nodes according to their centrality values
        nodelist=list(centr_type.keys()))
    nodes.set_norm(mcolors.SymLogNorm(linthresh=0.01, linscale=1, base=10))

    labels = nx.draw_networkx_labels(graph, pos)
    edges = nx.draw_networkx_edges(graph, pos)

    plt.title(title)
    plt.colorbar(nodes)
    plt.axis('off')
```

> Now let's initialize a random graph so we can try our function on.

```python
ver = [n for n in range(1, 15)] # nodes from 1 to 14
edges = [(l1,l2) for l1,l2 in map(lambda n : (n,8), [n for n in range(11,15)])] + \
    [(l1,l2) for l1,l2 in map(lambda n : (n,6), [n for n in range(1,6)])] + \
    [(7,6), (8,7),(9,7), (10,7)] # list of edges
g3, a3 = graph(ver, edges)
```

**Degree centrality**

Let's check the degree centrality for our 14 vertices.

```python
for node in g3.nodes():
    print(node, nx.degree_centrality(g3)[node])
```

    1 0.07692307692307693
    2 0.07692307692307693
    3 0.07692307692307693
    4 0.07692307692307693
    5 0.07692307692307693
    6 0.46153846153846156
    7 0.3076923076923077
    8 0.38461538461538464
    9 0.07692307692307693
    10 0.07692307692307693
    11 0.07692307692307693
    12 0.07692307692307693
    13 0.07692307692307693
    14 0.07692307692307693

```python
centrality(g3, nx.degree_centrality(g3), 'Degree Centrality')
```

**Eigenvector centrality**

```python
for node in g3.nodes():
    print(node, nx.eigenvector_centrality(g3)[node])
```

    1 0.2005897407903768
    2 0.2005897407903768
    3 0.2005897407903768
    4 0.2005897407903768
    5 0.2005897407903768
    6 0.543357022804172
    7 0.4688940252630566
    8 0.3805718175986731
    9 0.17310146940310608
    10 0.17310146940310608
    11 0.1404965947281777
    12 0.1404965947281777
    13 0.1404965947281777
    14 0.1404965947281777

```python
centrality(g3, nx.eigenvector_centrality(g3), 'Eigenvector centrality')
```

**Closeness centrality**

```python
for node in g3.nodes():
    print(node, nx.closeness_centrality(g3)[node])
```

    1 0.3611111111111111
    2 0.3611111111111111
    3 0.3611111111111111
    4 0.3611111111111111
    5 0.3611111111111111
    6 0.5416666666666666
    7 0.5909090909090909
    8 0.5
    9 0.38235294117647056
    10 0.38235294117647056
    11 0.34210526315789475
    12 0.34210526315789475
    13 0.34210526315789475
    14 0.34210526315789475

```python
centrality(g3, nx.closeness_centrality(g3), 'Closeness centrality')
```

**Betweennes centrality**

```python
for node in g3.nodes():
    print(node, nx.betweenness_centrality(g3)[node])
```

    1 0.0
    2 0.0
    3 0.0
    4 0.0
    5 0.0
    6 0.641025641025641
    7 0.6794871794871795
    8 0.5384615384615384
    9 0.0
    10 0.0
    11 0.0
    12 0.0
    13 0.0
    14 0.0

```python
centrality(g3, nx.betweenness_centrality(g3),'Betweenness centrality')
```