# Tetrads for Evaluating the Outgoing Gravitational Wave Weyl scalar $\psi_4$

## Authors: Patrick Nelson & Zach Etienne

[comment]: <> (Abstract: TODO)

**Notebook Status:**  Validated

**Validation Notes:** This module has been validated to agree at roundoff error with the WeylScal4 ETK thorn in Cartesian coordinates (as it agrees to roundoff error with Patrick Nelson's [Cartesian Weyl Scalars & Invariants NRPy+ tutorial notebook](Tutorial-WeylScalarsInvariants-Cartesian.ipynb), which itself was validated against WeylScal4). In addition, in SinhSpherical coordinates it yields results for a ringing Brill-Lindquist black hole remnant that agree with black hole perturbation theory to more than 7 decades in amplitude, surpassing the agreement seen in Fig. 6 of [Ruchlin, Etienne, & Baumgarte](https://arxiv.org/pdf/1712.07658.pdf).

### NRPy+ Source Code for this module: [BSSN/Psi4_tetrads.py](../edit/BSSN/Psi4_tetrads.py)

## Introduction:
This module constructs tetrad vectors $l^\mu$, $m^\mu$, and $n^\mu$ for the $\psi_4$ Weyl scalar, a quantity that is immensely useful when extracting gravitational wave content from a numerical relativity simulation. $\psi_4$ is related to the gravitational wave strain via

$$
\psi_4 = \ddot{h}_+ - i \ddot{h}_\times.
$$

We construct $\psi_4$ from the standard ADM spatial metric $\gamma_{ij}$ and extrinsic curvature $K_{ij}$, and their derivatives. The full expression is given by Eq. 5.1 in [Baker, Campanelli, Lousto (2001)](https://arxiv.org/pdf/gr-qc/0104063.pdf):

\begin{align}
\psi_4 &= \left[ {R}_{ijkl}+2K_{i[k}K_{l]j}\right]
{n}^i\bar{m}^j{n}^k\bar{m}^l  \\
& -8\left[ K_{j[k,l]}+{\Gamma }_{j[k}^pK_{l]p}\right]
{n}^{[0}\bar{m}^{j]}{n}^k\bar{m}^l \\
& +4\left[ {R}_{jl}-K_{jp}K_l^p+KK_{jl}\right]
{n}^{[0}\bar{m}^{j]}{n}^{[0}\bar{m}^{l]},
\end{align}

Note that $\psi_4$ is complex, with the imaginary components originating from the tetrad vector $m^\mu$. This module does not specify a tetrad; instead it only constructs the above expression leaving $m^\mu$ and $n^\mu$ unspecified. This module defines these tetrad quantities, implementing the quasi-Kinnersley tetrad of [Baker, Campanelli, Lousto (2001)](https://arxiv.org/pdf/gr-qc/0104063.pdf), also referred to as "***the BCL paper***".

### A Note on Notation:

As is standard in NRPy+,

* Greek indices range from 0 to 3, inclusive, with the zeroth component denoting the temporal (time) component.
* Latin indices range from 0 to 2, inclusive, with the zeroth component denoting the first spatial component.

As a corollary, any expressions involving mixed Greek and Latin indices will need to offset one set of indices by one: A Latin index in a four-vector will be incremented and a Greek index in a three-vector will be decremented (however, the latter case does not occur in this tutorial notebook).

# Table of Contents
$$\label{toc}$$

This tutorial notebook is organized as follows

1. [Step 1](#initializenrpy): Initialize needed NRPy+ modules
1. [Step 2](#quasikinnersley): The quasi-Kinnersley tetrad
1. [Step 3](#code_validation): Code Validation against `BSSN.Psi4_tetrads` NRPy+ module
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize core NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

```python
# Step 1.a: import all needed modules from NRPy+:
import sympy as sp                # SymPy: The Python computer algebra package upon which NRPy+ depends
import NRPy_param_funcs as par    # NRPy+: Parameter interface
import indexedexp as ixp          # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm    # NRPy+: Reference metric support
import sys                        # Standard Python modules for multiplatform OS-level functions

# Step 1.b: Set the coordinate system for the numerical grid
par.set_parval_from_str("reference_metric::CoordSystem","Spherical")

# Step 1.c: Given the chosen coordinate system, set up
#           corresponding reference metric and needed
#           reference metric quantities
# The following function call sets up the reference metric
#    and related quantities, including rescaling matrices ReDD,
#    ReU, and hatted quantities.
rfm.reference_metric()

# Step 1.d: Set spatial dimension (must be 3 for BSSN, as BSSN is
#           a 3+1-dimensional decomposition of the general
#           relativistic field equations)
DIM = 3

# Step 1.e: Import all ADM quantities as written in terms of BSSN quantities
import BSSN.ADM_in_terms_of_BSSN as AB
AB.ADM_in_terms_of_BSSN()

# Step 1.f: Initialize TetradChoice parameter
thismodule = __name__
# Current option: QuasiKinnersley = choice made in Baker, Campanelli, and Lousto. PRD 65, 044001 (2002)
par.initialize_param(par.glb_param("char", thismodule, "TetradChoice", "QuasiKinnersley"))
```

# Step 2: The quasi-Kinnersley tetrad of [Baker, Campanelli, Lousto (2001)](https://arxiv.org/pdf/gr-qc/0104063.pdf) \[Back to [top](#toc)\]
$$\label{quasikinnersley}$$

To define the Weyl scalars, first a tetrad must be chosen. Below, for compatibility with the [WeylScal4 diagnostic module](https://bitbucket.org/einsteintoolkit/einsteinanalysis/src/master/WeylScal4/), we implement the quasi-Kinnersley tetrad of [Baker, Campanelli, Lousto (2001)](https://arxiv.org/pdf/gr-qc/0104063.pdf).

We begin with the vectors given in eqs. 5.6 and 5.7 of the BCL paper, which are orthogonal to each other in flat spacetime; one is in the $\phi$ direction, one is in $r$, and the third is the cross product of the first two:
\begin{align}
    v_1^a &= [-y,x,0] \\
    v_2^a &= [x,y,z] \\
    v_3^a &= {\rm det}(\gamma)^{1/2} \gamma^{ad} \epsilon_{dbc} v_1^b v_2^c,
\end{align}

Notice that $v_1^a$ and $v_2^a$ assume the Cartesian basis, but $\gamma^{ad}$ will be in the $xx^i$ basis given by the chosen `reference_metric::CoordSystem`. Thus to construct $v_3^a$, we must first perform a change of basis on $v_1^a$ and $v_2^a$:

$$
v_{1,{\rm xx}}^a = \frac{\partial xx^a}{\partial x_{\rm Cart}^b} v_{1,{\rm Cart}}^b.
$$
This equation is problematic because we generally do not have a closed-form expression for components of the $xx^a$ vector as functions of the Cartesian coordinate vector components $x_{\rm Cart}^a$. However we do have closed-form expressions for components of $x_{\rm Cart}^a$ as functions of $xx^a$. Thus we can construct the needed Jacobian matrix $\frac{\partial xx^a}{\partial x_{\rm Cart}^b}$ by evaluating the derivative $\frac{\partial x_{\rm Cart}^b}{\partial xx^a}$ and performing a simple matrix inversion:
$$
\frac{\partial xx^a}{\partial x_{\rm Cart}^b} = \left(\frac{\partial x_{\rm Cart}^b}{\partial xx^a} \right)^{-1}.
$$

```python
# Step 2.a: Declare the Cartesian x,y,z in terms of
#           xx0,xx1,xx2.
x = rfm.xxCart[0]
y = rfm.xxCart[1]
z = rfm.xxCart[2]

# Step 2.b: Declare v_1^a, v_2^a, and v_3^a tetrads,
#           as well as detgamma and gammaUU from
#           BSSN.ADM_in_terms_of_BSSN
v1UCart = ixp.zerorank1()
v2UCart = ixp.zerorank1()

detgamma = AB.detgamma
gammaUU = AB.gammaUU

# Step 2.c: Define v1U and v2U
v1UCart = [-y, x, sp.sympify(0)]
v2UCart = [x, y, z]

# Step 2.d: Construct the Jacobian d x_Cart^i / d xx^j
Jac_dUCart_dDrfmUD = ixp.zerorank2()
for i in range(DIM):
    for j in range(DIM):
        Jac_dUCart_dDrfmUD[i][j] = sp.simplify(sp.diff(rfm.xxCart[i], rfm.xx[j]))

# Step 2.e: Invert above Jacobian to get needed d xx^j / d x_Cart^i
Jac_dUrfm_dDCartUD, dummyDET = ixp.generic_matrix_inverter3x3(Jac_dUCart_dDrfmUD)

# Step 2.e.i: Simplify expressions for d xx^j / d x_Cart^i:
for i in range(DIM):
    for j in range(DIM):
        Jac_dUrfm_dDCartUD[i][j] = sp.simplify(Jac_dUrfm_dDCartUD[i][j])

# Step 2.f: Transform v1U and v2U from the Cartesian to the xx^i basis
v1U = ixp.zerorank1()
v2U = ixp.zerorank1()
for i in range(DIM):
    for j in range(DIM):
        v1U[i] += Jac_dUrfm_dDCartUD[i][j] * v1UCart[j]
        v2U[i] += Jac_dUrfm_dDCartUD[i][j] * v2UCart[j]
```

... next we construct the third tetrad vector $v_3^a={\rm det}(\gamma)^{1/2} \gamma^{ad} \epsilon_{dbc} v_1^b v_2^c$:

Let's start by defining the Levi-Civita symbol $\epsilon_{dbc}$:

```python
# Step 2.g: Define the rank-3 version of the Levi-Civita symbol. Amongst
#         other uses, this is needed for the construction of the approximate
#         quasi-Kinnersley tetrad.
def define_LeviCivitaSymbol_rank3(DIM=-1):
    if DIM == -1:
        DIM = par.parval_from_str("DIM")

    LeviCivitaSymbol = ixp.zerorank3()

    for i in range(DIM):
        for j in range(DIM):
            for k in range(DIM):
                # From https://codegolf.stackexchange.com/questions/160359/levi-civita-symbol :
                LeviCivitaSymbol[i][j][k] = (i - j) * (j - k) * (k - i) / 2
    return LeviCivitaSymbol

# Step 2.h: Define v3U
v3U = ixp.zerorank1()
LeviCivitaSymbolDDD = define_LeviCivitaSymbol_rank3(DIM=3)
for a in range(DIM):
    for b in range(DIM):
        for c in range(DIM):
            for d in range(DIM):
                v3U[a] += sp.sqrt(detgamma)*gammaUU[a][d]*LeviCivitaSymbolDDD[d][b][c]*v1U[b]*v2U[c]

# Step 2.h.i: Simplify expressions for v1U,v2U,v3U. This greatly expedites the C code generation (~10x faster)
for a in range(DIM):
    v1U[a] = sp.simplify(v1U[a])
    v2U[a] = sp.simplify(v2U[a])
    v3U[a] = sp.simplify(v3U[a])
```

As our next step, we carry out the Gram-Schmidt orthonormalization process. The vectors $v_i^a$ are placeholders in the code; the final product of the orthonormalization is the vectors $e_i^a$. So,
\begin{align}
e_1^a &= \frac{v_1^a}{\sqrt{\omega_{11}}} \\
e_2^a &= \frac{v_2^a - \omega_{12} e_1^a}{\sqrt{\omega_{22}}} \\
e_3^a &= \frac{v_3^a - \omega_{13} e_1^a - \omega_{23} e_2^a}{\sqrt{\omega_{33}}}, \text{ where}\\
\omega_{ij} &= v_i^a v_j^b \gamma_{ab}
\end{align}

Note that the above expressions must be evaluated with the numerators first, so that the denominators generate the proper normalization.

```python
# Step 2.i: Define omega_{ij}
omegaDD = ixp.zerorank2()
gammaDD = AB.gammaDD
def v_vectorDU(v1U,v2U,v3U,  i,a):
    if i==0:
        return v1U[a]
    elif i==1:
        return v2U[a]
    elif i==2:
        return v3U[a]
    else:
        print("ERROR: unknown vector!")
        sys.exit(1)

def update_omega(omegaDD, i,j, v1U,v2U,v3U,gammaDD):
    omegaDD[i][j] = sp.sympify(0)
    for a in range(DIM):
        for b in range(DIM):
            omegaDD[i][j] += v_vectorDU(v1U,v2U,v3U, i,a)*v_vectorDU(v1U,v2U,v3U, j,b)*gammaDD[a][b]

# Step 2.j: Define e^a_i. Note that:
#           omegaDD[0][0] = \omega_{11} above;
#           omegaDD[1][1] = \omega_{22} above, etc.
e1U = ixp.zerorank1()
e2U = ixp.zerorank1()
e3U = ixp.zerorank1()
# First e_1^a: Orthogonalize & normalize:
update_omega(omegaDD, 0,0, v1U,v2U,v3U,gammaDD)
for a in range(DIM):
    e1U[a] = v1U[a]/sp.sqrt(omegaDD[0][0])

# Next e_2^a: First orthogonalize:
update_omega(omegaDD, 0,1, e1U,v2U,v3U,gammaDD)
for a in range(DIM):
    e2U[a] = (v2U[a] - omegaDD[0][1]*e1U[a])
# Then normalize:
update_omega(omegaDD, 1,1, e1U,e2U,v3U,gammaDD)
for a in range(DIM):
    e2U[a] /= sp.sqrt(omegaDD[1][1])

# Next e_3^a: First orthogonalize:
update_omega(omegaDD, 0,2, e1U,e2U,v3U,gammaDD)
update_omega(omegaDD, 1,2, e1U,e2U,v3U,gammaDD)
for a in range(DIM):
    e3U[a] = (v3U[a] - omegaDD[0][2]*e1U[a] - omegaDD[1][2]*e2U[a])
# Then normalize:
update_omega(omegaDD, 2,2, e1U,e2U,e3U,gammaDD)
for a in range(DIM):
    e3U[a] /= sp.sqrt(omegaDD[2][2])
```

Once we have orthogonal, normalized vectors, we can construct the tetrad itself, again drawing on eqs. 5.6. We can draw on SymPy's built-in tools for complex numbers to build the complex vector $m^a$:
\begin{align}
    l^\mu &= \frac{1}{\sqrt{2}} \left(u^\mu + r^\mu\right) \\
    n^\mu &= \frac{1}{\sqrt{2}} \left(u^\mu - r^\mu\right) \\
    \Re(m^\mu) &= \frac{1}{\sqrt{2}} \theta^\mu \\
    \Im(m^\mu) &= \frac{1}{\sqrt{2}} \phi^\mu,
\end{align}
where $r^\mu=\{0,e_2^i\}$, $\theta^\mu=\{0,e_3^i\}$, $\phi^\mu=\{0,e_1^i\}$, and $u^\mu$ is the time-like unit normal to the hypersurface.

```python
# Step 2.k: Construct l^mu, n^mu, and m^mu, based on r^mu, theta^mu, phi^mu, and u^mu:
r4U     = ixp.zerorank1(DIM=4)
u4U     = ixp.zerorank1(DIM=4)
theta4U = ixp.zerorank1(DIM=4)
phi4U   = ixp.zerorank1(DIM=4)

for a in range(DIM):
    r4U[    a+1] = e2U[a]
    theta4U[a+1] = e3U[a]
    phi4U[  a+1] = e1U[a]

# FIXME? assumes alpha=1, beta^i = 0
u4U[0] = 1

l4U = ixp.zerorank1(DIM=4)
n4U = ixp.zerorank1(DIM=4)
mre4U  = ixp.zerorank1(DIM=4)
mim4U  = ixp.zerorank1(DIM=4)

# M_SQRT1_2 = 1 / sqrt(2) (defined in math.h on Linux)
M_SQRT1_2 = par.Cparameters("#define",thismodule,"M_SQRT1_2","")
isqrt2 = M_SQRT1_2 #1/sp.sqrt(2) <- SymPy drops precision to 15 sig. digits in unit tests
for mu in range(4):
    l4U[mu]   = isqrt2*(u4U[mu] + r4U[mu])
    n4U[mu]   = isqrt2*(u4U[mu] - r4U[mu])
    mre4U[mu] = isqrt2*theta4U[mu]
    mim4U[mu] = isqrt2*  phi4U[mu]

# ltetU,ntetU,remtetU,immtetU,e1U,e2U,e3U
for mu in range(4):
    l4U[mu]   = isqrt2*(u4U[mu] + r4U[mu])
    n4U[mu]   = isqrt2*(u4U[mu] - r4U[mu])
    mre4U[mu] = isqrt2*theta4U[mu]
    mim4U[mu] = isqrt2*  phi4U[mu]
```

# Step 3: Code validation against `BSSN.Psi4_tetrads` NRPy+ module \[Back to [top](#toc)\]
$$\label{code_validation}$$

As a code validation check, we verify agreement in the SymPy expressions for the RHSs of the BSSN equations between
1. this tutorial and
2. the NRPy+ [BSSN.Psi4_tetrads](../edit/BSSN/Psi4_tetrads.py) module.

By default, we compare all quantities in Spherical coordinates, though other coordinate systems may be chosen.

```python
all_passed=True
def comp_func(expr1,expr2,basename,prefixname2="BP4T."):
    if str(expr1-expr2)!="0":
        print(basename+" - "+prefixname2+basename+" = "+ str(expr1-expr2))
        all_passed=False

def gfnm(basename,idx1,idx2=None,idx3=None):
    if idx2==None:
        return basename+"["+str(idx1)+"]"
    if idx3==None:
        return basename+"["+str(idx1)+"]["+str(idx2)+"]"
    return basename+"["+str(idx1)+"]["+str(idx2)+"]["+str(idx3)+"]"

expr_list = []
exprcheck_list = []
namecheck_list = []

import BSSN.Psi4_tetrads as BP4T
BP4T.Psi4_tetrads()

for mu in range(4):
    namecheck_list.extend([gfnm("l4U",mu),gfnm("n4U",mu),gfnm("mre4U",mu),gfnm("mim4U",mu)])
    exprcheck_list.extend([BP4T.l4U[mu],BP4T.n4U[mu],BP4T.mre4U[mu],BP4T.mim4U[mu]])
    expr_list.extend([l4U[mu],n4U[mu],mre4U[mu],mim4U[mu]])

for i in range(len(expr_list)):
    comp_func(expr_list[i],exprcheck_list[i],namecheck_list[i])

if all_passed:
    print("ALL TESTS PASSED!")
```

    ALL TESTS PASSED!

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Psi4_tetrads.pdf](Tutorial-Psi4_tetrads.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-Psi4_tetrads.ipynb
!pdflatex -interaction=batchmode Tutorial-Psi4_tetrads.tex
!pdflatex -interaction=batchmode Tutorial-Psi4_tetrads.tex
!pdflatex -interaction=batchmode Tutorial-Psi4_tetrads.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    [NbConvertApp] Converting notebook Tutorial-Psi4_tetrads.ipynb to latex
    [NbConvertApp] Writing 56186 bytes to Tutorial-Psi4_tetrads.tex
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


<div align='center'>

# Resolução da Lista 4
### complemento, calculos e solução das equações
---
<small>
    William Henrique - 119111128

    22 de fevereiro de 2022
</small>

---
</div>

```python
from sympy import Eq, symbols, solve, E, limit, oo, linsolve, diff, integrate
```

```python
# Matrícula
R1 = 1 + 1
R2 = 1 + 2
R3 = 1 + 9
```

# Questão 1

## Letra C

```python
ia, ib, ic = symbols("i_a, i_b, i_c")

eq1 = Eq(ia, 0.25)
eq2 = Eq(-2 + R1*ic + R3*(ic-ib), 0)
eq3 = Eq(-R3*(ic-ib) - R2*(0.25-ib) + R1*(ic-ib), 0)

soluc = solve((eq1, eq2, eq3)) # resolve o sistema
ia, ib, ic = [soluc[sol] for sol in soluc] # obtêm as variáveis

for sol in soluc:
    print(sol, '=', soluc[sol], 'A')

Vth = 1 + 2*ic # calcula Vth
print("Vth = ", Vth, "V")
```

    i_a = 0.250000000000000 A
    i_b = 0.480769230769231 A
    i_c = 0.567307692307692 A
    Vth =  2.13461538461538 V

```python
ia, ib, ic, id = symbols("i_a, i_b, i_c, i_d")

eq1 = Eq(ia, 0.25)
eq2 = Eq(-2 + R1*(ic-id) + R3*(ic-ib), 0)
eq3 = Eq(-R3*(ic-ib) - R2*(0.25-ib) + R1*(ic-ib), 0)
eq4 = Eq(-R1*(ic-id), 1)

soluc = solve((eq1, eq2, eq3, eq4))
for sol in soluc:
    print(sol, '=', soluc[sol], 'A')

_, _, _, icc  = [soluc[sol] for sol in soluc] # obtem Icc
Rth = Vth / icc # calcula Rth
print("icc", "=", icc, "A")
print("Rth", "=", Rth, "R")

```

    i_a = 0.250000000000000 A
    i_b = 1.05000000000000 A
    i_c = 1.35000000000000 A
    i_d = 1.85000000000000 A
    icc = 1.85000000000000 A
    Rth = 1.15384615384615 R

```python
t = symbols("t")
tau = 35e-3 / Rth

# A corrente que atravessa os indutores é a mesma Il1 = Il2 = Ileq
il = Vth/Rth + (0.5 - Vth/Rth)*E**(-t/tau)
print("il(t) = ",il)
```

    il(t) =  1.85 - 1.35*exp(-32.967032967033*t)

```python
vl1 = (15*1e-3)*diff(il)
vl2 = (20*1e-3)*diff(il)
print("tau = ", tau)
print("Vl1(t) = ", vl1)
print("Vl2(t) = ", vl2)
```

    tau =  0.0303333333333333
    Vl1(t) =  0.667582417582418*exp(-32.967032967033*t)
    Vl2(t) =  0.89010989010989*exp(-32.967032967033*t)

## Letra D

Resolvendo o sistemas de equações, considerando $i_d(t) = i_{Leq}(t)$

```python
ia, ib, ic, id = symbols("i_a, i_b, i_c, i_d")

eq1 = Eq(ia, 0.25)
eq2 = Eq(-2 + R1*(ic-id) + R3*(ic-ib), 0)
eq3 = Eq(-R3*(ic-ib) - R2*(0.25-ib) + R1*(ic-ib), 0)

soluc = solve((eq1, eq2, eq3))
for sol in soluc:
    print(sol, '=', soluc[sol], 'A')

ia, ib, ic = [soluc[sol] for sol in soluc]

ix = (ic - ib)
print("ix(t) = ", ix)
print("ix(t) = ", ix.subs({id: il}))
```

    i_a = 0.250000000000000 A
    i_b = 0.307692307692308*i_d + 0.480769230769231 A
    i_c = 0.423076923076923*i_d + 0.567307692307692 A
    ix(t) =  0.115384615384615*i_d + 0.0865384615384615
    ix(t) =  0.3 - 0.155769230769231*exp(-32.967032967033*t)

```python
eq3
```

$\displaystyle 11 i_{b} - 8 i_{c} - 0.75 = 0$

## Letra E

```python
il_final = limit(il, t, oo)
print("i(oo) = ", il_final)
print("w(oo) = ", 0.5*35e-3*il_final**2 *1e3, "mJ")
```

    i(oo) =  1.85000000000000
    w(oo) =  59.8937500000000 mJ

# Questão 2

## Letra A

```python
t = symbols('t')
R = 1 / (1/R1 + 1/R2)
L = 2
C = 1e-6
tau = L/R
Io = -1 # A
il = Io*E**(-t/tau)
il_t1 = il.subs({t: 1}) # t = 1+
print('il(t) = ', il, 'A')
print('il(1+) = ', il_t1, 'A')
```

    il(t) =  -exp(-0.6*t) A
    il(1+) =  -0.548811636094026 A

## Letra B

```python
alpha = 0.5/R/C
omega = (L*C)**-.5
S1 = -alpha + (alpha**2 - omega**2)**.5 # rever a equação
S2 = -alpha - (alpha**2 - omega**2)**.5

print('alpha = ', alpha, 'rad/s')
print('omega = ', omega, 'rad/s')
print('S1 = ', S1, 'rad/s')
print('S2 = ', S2, 'rad/s')

```

    alpha =  416666.6666666666 rad/s
    omega =  707.1067811865476 rad/s
    S1 =  0.5999995680176653 rad/s
    S2 =  -0.6000004319939762 rad/s

```python
a1, a2 = symbols("a_1, a_2")

Vo = 5              # Vo = 5 V
Io = il_t1          # Io = I(t = 1) A
print('Io = ', Io, 'A')
print('Vo = ', Vo, 'V')

eq1 = Eq(a1 + a2, Vo)
eq2 = Eq(S1*a1 + S2*a2, 1/C*(-Vo/R - Io))

soluc = solve((eq1, eq2))
for x in soluc:
    print(x, soluc[x])
a1, a2 = [soluc[x] for x in soluc]
```

    Io =  -0.548811636094026 A
    Vo =  5 V
    a_1 -3014876.69211282
    a_2 3014881.69211282

```python
Vt = a1*E**(S1*t) + a2*E**(S2*t) # tensão do RLC paralelo resposta natural
print('V(t) = ', Vt, 'V')
print("IL(t) = ", 1/L*integrate(Vt, (t, 1, t)) + Io, 'A')
```

    V(t) =  -3014876.69211282*exp(0.599999568017665*t) + 3014881.69211282*exp(-0.600000431993976*t) V
    IL(t) =  -2512399.0522807*exp(0.599999568017665*t) + 5956720.56086932 - 2512399.60119153*exp(-0.600000431993976*t) A

## Letra D - Parte 1

```python
# parte 1 - degrau - 0 <= t < 1
R = 3
L = 2
C = 1e-6
Vo = 5              # Vo = 5 V
Io = -1          # Io = Il = -1A
Is = -1          # Il(oo) = -1A

alpha = 0.5/R/C
omega = (L*C)**-.5
print('alpha = ', alpha, 'rad/s')
print('omega = ', omega, 'rad/s')

S1 = -alpha + (alpha**2 - omega**2)**.5
S2 = -alpha - (alpha**2 - omega**2)**.5
print('S1 = ', S1, 'rad/s')
print('S2 = ', S2, 'rad/s')

print('Io = ', Io, 'A')
print('Vo = ', Vo, 'V')

### RESPOSTA AO DEGRAU - RLC PARALELO
a1, a2 = symbols('a1, a2')
eq1 = Eq(a1 + a2, Io - Is)
eq2 = Eq(S1*a1 + S2*a2, Vo/L)

soluc = solve((eq1, eq2))

for x in soluc:
    print(x, soluc[x])
a1, a2 = [soluc[x] for x in soluc]

# corrente e tensão no indutor
Il = Is + a1*E**(S1*t) + a2*E**(S2*t)
Vl = L*diff(Il, t)

print('Il(t) = ', Il, 'A')
print('Vl(t) = ', Vl, 'V') # Vl = Vc = Vr = Vs
```

    alpha =  166666.66666666666 rad/s
    omega =  707.1067811865476 rad/s
    S1 =  1.4999932500650175 rad/s
    S2 =  -1.5000067500513978 rad/s
    Io =  -1 A
    Vo =  5 V
    a1 0.333331083303265
    a2 -1.33333108330326
    Il(t) =  0.333331083303265*exp(1.49999325006502*t) - 1 - 1.33333108330326*exp(-1.5000067500514*t) A
    Vl(t) =  0.999988749983514*exp(1.49999325006502*t) + 4.00001125001648*exp(-1.5000067500514*t) V

## Letra D - Parte 2
* para t >= 1

```python
# t >= 1, resposta natural
t, a1, a2 = symbols('t, A_1, A_2')
R = 1 / (1/R1 + 1/R2) # 1.2 ohm
L = 2
C = 1e-6

## calcula os parâmetros de depende apenas de R, L e C
alpha = 0.5/R/C
omega = (L*C)**-.5
S1 = -alpha + (alpha**2 - omega**2)**.5
S2 = -alpha - (alpha**2 - omega**2)**.5
print('alpha = ', alpha, 'rad/s')
print('omega = ', omega, 'rad/s')
print('S1 = ', S1, 'rad/s')
print('S2 = ', S2, 'rad/s')

## calcula as condições inicias para t = 1
Io = Il.subs({t: 1})
Vo = Vl.subs({t: 1})
print('Il(1) = ', Io, 'A')
print('Vl(1) = ', Vo, 'V')

## encontra A1 e A2
eq1 = Eq(a1 + a2, Vo)
eq2 = Eq(S1*a1 + S2*a2, 1/C*(-Vo/R - Io))
soluc = solve((eq1, eq2))
for x in soluc:
    print(x, soluc[x], 'V')
a1, a2 = [soluc[x] for x in soluc]

Vl = a1*E**(S1*(t-1)) + a2*E**(S2*(t-1))
print("VL(t) = ", Vl, 'V')
print("IL(t) = ", 1/L*integrate(Vl, (t, 1, t)) + Io, 'A')

```

    alpha =  416666.6666666666 rad/s
    omega =  707.1067811865476 rad/s
    S1 =  0.5999995680176653 rad/s
    S2 =  -0.6000004319939762 rad/s
    Il(1) =  5.37412552628666 A
    Vl(1) =  5.37412552684961 V
    A_1 -8210466.86729565 V
    A_2 8210472.24142118 V
    VL(t) =  -4506001.70104904*exp(0.599999568017665*t) + 14960462.2940061*exp(-0.600000431993976*t) V
    IL(t) =  -3755004.12103328*exp(0.599999568017665*t) + 13684121.2979273 - 12467042.935526*exp(-0.600000431993976*t) A

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
```

```python
t = 0
x = np.linspace(-4,4,1000)
```

```python
y = np.exp(-np.power(x-3*t,2))*np.sin(3*np.pi*(x-t))
```

```python
plt.figure(figsize=(10,5),dpi=250)
plt.plot(x,y)
```

```python
x, y, t = symbols('x y t')
y = exp(-(x-3*t)**2)*sin(3*pi*(x-t))
simplify(y)
```

$\displaystyle - e^{- \left(3 t - x\right)^{2}} \sin{\left(\pi \left(3 t - 3 x\right) \right)}$

```python

```


```python
#load watermark
%load_ext watermark
%watermark -a 'Gopala KR' -u -d -v -p watermark,numpy,pandas,matplotlib,nltk,sklearn,tensorflow,theano,mxnet,chainer,seaborn,keras,tflearn,bokeh,gensim
```

    /srv/conda/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters
    WARNING (theano.configdefaults): install mkl with `conda install mkl-service`: No module named 'mkl'
    Using TensorFlow backend.
    /srv/conda/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
      if d.decorator_argspec is not None), _inspect.getargspec(target))

    Gopala KR
    last updated: 2018-02-15

    CPython 3.6.1
    IPython 6.2.1

    watermark 1.6.0
    numpy 1.14.0
    pandas 0.21.1
    matplotlib 2.1.1
    nltk 3.2.5
    sklearn 0.19.1
    tensorflow 1.5.0
    theano 1.0.1
    mxnet 1.0.0
    chainer 3.3.0
    seaborn 0.8.1
    keras 2.1.4
    tflearn n
    bokeh 0.12.14
    gensim 3.3.0

    /srv/conda/lib/python3.6/site-packages/scipy/sparse/sparsetools.py:20: DeprecationWarning: `scipy.sparse.sparsetools` is deprecated!
    scipy.sparse.sparsetools is a private module for scipy.sparse, and should not be used.
      _deprecated()

```python
from preamble import *
%matplotlib inline
```

## Model Evaluation and Improvement

```python
from sklearn.datasets import make_blobs
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# create a synthetic dataset
X, y = make_blobs(random_state=0)
# split data and labels into a training and a test set
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
# instantiate a model and fit it to the training set
logreg = LogisticRegression().fit(X_train, y_train)
# evaluate the model on the test set
print("Test set score: {:.2f}".format(logreg.score(X_test, y_test)))
```

    Test set score: 0.88

### Cross-Validation

```python
mglearn.plots.plot_cross_validation()
```

#### Cross-Validation in scikit-learn

```python
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

iris = load_iris()
logreg = LogisticRegression()

scores = cross_val_score(logreg, iris.data, iris.target)
print("Cross-validation scores: {}".format(scores))
```

    Cross-validation scores: [0.961 0.922 0.958]

```python
scores = cross_val_score(logreg, iris.data, iris.target, cv=5)
print("Cross-validation scores: {}".format(scores))
```

    Cross-validation scores: [1.    0.967 0.933 0.9   1.   ]

```python
print("Average cross-validation score: {:.2f}".format(scores.mean()))
```

    Average cross-validation score: 0.96

### Benefits of Cross-Validation

### Stratified K-Fold cross-validation and other strategies

```python
from sklearn.datasets import load_iris
iris = load_iris()
print("Iris labels:\n{}".format(iris.target))
```

    Iris labels:
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2]

```python
mglearn.plots.plot_stratified_cross_validation()
```

### More control over cross-validation

```python
from sklearn.model_selection import KFold
kfold = KFold(n_splits=5)
```

```python
print("Cross-validation scores:\n{}".format(
      cross_val_score(logreg, iris.data, iris.target, cv=kfold)))
```

    Cross-validation scores:
    [1.    0.933 0.433 0.967 0.433]

```python
kfold = KFold(n_splits=3)
print("Cross-validation scores:\n{}".format(
    cross_val_score(logreg, iris.data, iris.target, cv=kfold)))
```

    Cross-validation scores:
    [0. 0. 0.]

```python
kfold = KFold(n_splits=3, shuffle=True, random_state=0)
print("Cross-validation scores:\n{}".format(
    cross_val_score(logreg, iris.data, iris.target, cv=kfold)))
```

    Cross-validation scores:
    [0.9  0.96 0.96]

#### Leave-one-out cross-validation

```python
from sklearn.model_selection import LeaveOneOut
loo = LeaveOneOut()
scores = cross_val_score(logreg, iris.data, iris.target, cv=loo)
print("Number of cv iterations: ", len(scores))
print("Mean accuracy: {:.2f}".format(scores.mean()))
```

    Number of cv iterations:  150
    Mean accuracy: 0.95

#### Shuffle-split cross-validation

```python
mglearn.plots.plot_shuffle_split()
```

```python
from sklearn.model_selection import ShuffleSplit
shuffle_split = ShuffleSplit(test_size=.5, train_size=.5, n_splits=10)
scores = cross_val_score(logreg, iris.data, iris.target, cv=shuffle_split)
print("Cross-validation scores:\n{}".format(scores))
```

    Cross-validation scores:
    [0.92  0.933 0.96  0.96  0.947 0.867 0.88  0.947 0.92  0.92 ]

##### Cross-validation with groups

```python
mglearn.plots.plot_group_kfold()
```

```python
from sklearn.model_selection import GroupKFold
# create synthetic dataset
X, y = make_blobs(n_samples=12, random_state=0)
# assume the first three samples belong to the same group,
# then the next four, etc
groups = [0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 3]
scores = cross_val_score(logreg, X, y, groups, cv=GroupKFold(n_splits=3))
print("Cross-validation scores:\n{}".format(scores))
```

    Cross-validation scores:
    [0.75  0.8   0.667]

### Grid Search

#### Simple Grid-Search

```python
# naive grid search implementation
from sklearn.svm import SVC
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target,
                                                    random_state=0)
print("Size of training set: {}   size of test set: {}".format(
      X_train.shape[0], X_test.shape[0]))

best_score = 0

for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    for C in [0.001, 0.01, 0.1, 1, 10, 100]:
        # for each combination of parameters, train an SVC
        svm = SVC(gamma=gamma, C=C)
        svm.fit(X_train, y_train)
        # evaluate the SVC on the test set
        score = svm.score(X_test, y_test)
        # if we got a better score, store the score and parameters
        if score > best_score:
            best_score = score
            best_parameters = {'C': C, 'gamma': gamma}

print("Best score: {:.2f}".format(best_score))
print("Best parameters: {}".format(best_parameters))
```

    Size of training set: 112   size of test set: 38
    Best score: 0.97
    Best parameters: {'C': 100, 'gamma': 0.001}

#### The danger of overfitting the parameters and the validation set

```python
mglearn.plots.plot_threefold_split()
```

```python
from sklearn.svm import SVC
# split data into train+validation set and test set
X_trainval, X_test, y_trainval, y_test = train_test_split(
    iris.data, iris.target, random_state=0)
# split train+validation set into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(
    X_trainval, y_trainval, random_state=1)
print("Size of training set: {}   size of validation set: {}   size of test set:"
      " {}\n".format(X_train.shape[0], X_valid.shape[0], X_test.shape[0]))

best_score = 0

for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    for C in [0.001, 0.01, 0.1, 1, 10, 100]:
        # for each combination of parameters train an SVC
        svm = SVC(gamma=gamma, C=C)
        svm.fit(X_train, y_train)
        # evaluate the SVC on the validation set
        score = svm.score(X_valid, y_valid)
        # if we got a better score, store the score and parameters
        if score > best_score:
            best_score = score
            best_parameters = {'C': C, 'gamma': gamma}

# rebuild a model on the combined training and validation set,
# and evaluate it on the test set
svm = SVC(**best_parameters)
svm.fit(X_trainval, y_trainval)
test_score = svm.score(X_test, y_test)
print("Best score on validation set: {:.2f}".format(best_score))
print("Best parameters: ", best_parameters)
print("Test set score with best parameters: {:.2f}".format(test_score))
```

    Size of training set: 84   size of validation set: 28   size of test set: 38

    Best score on validation set: 0.96
    Best parameters:  {'C': 10, 'gamma': 0.001}
    Test set score with best parameters: 0.92

#### Grid-search with cross-validation

```python
# reference: manual_grid_search_cv
for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    for C in [0.001, 0.01, 0.1, 1, 10, 100]:
        # for each combination of parameters,
        # train an SVC
        svm = SVC(gamma=gamma, C=C)
        # perform cross-validation
        scores = cross_val_score(svm, X_trainval, y_trainval, cv=5)
        # compute mean cross-validation accuracy
        score = np.mean(scores)
        # if we got a better score, store the score and parameters
        if score > best_score:
            best_score = score
            best_parameters = {'C': C, 'gamma': gamma}
# rebuild a model on the combined training and validation set
svm = SVC(**best_parameters)
svm.fit(X_trainval, y_trainval)
```

    SVC(C=100, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

```python
mglearn.plots.plot_cross_val_selection()
```

```python
mglearn.plots.plot_grid_search_overview()
```

```python
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100],
              'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}
print("Parameter grid:\n{}".format(param_grid))
```

    Parameter grid:
    {'C': [0.001, 0.01, 0.1, 1, 10, 100], 'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}

```python
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
grid_search = GridSearchCV(SVC(), param_grid, cv=5,
                          return_train_score=True)
```

```python
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, random_state=0)
```

```python
grid_search.fit(X_train, y_train)
```

    GridSearchCV(cv=5, error_score='raise',
           estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False),
           fit_params=None, iid=True, n_jobs=1,
           param_grid={'C': [0.001, 0.01, 0.1, 1, 10, 100], 'gamma': [0.001, 0.01, 0.1, 1, 10, 100]},
           pre_dispatch='2*n_jobs', refit=True, return_train_score=True,
           scoring=None, verbose=0)

```python
print("Test set score: {:.2f}".format(grid_search.score(X_test, y_test)))
```

    Test set score: 0.97

```python
print("Best parameters: {}".format(grid_search.best_params_))
print("Best cross-validation score: {:.2f}".format(grid_search.best_score_))
```

    Best parameters: {'C': 100, 'gamma': 0.01}
    Best cross-validation score: 0.97

```python
print("Best estimator:\n{}".format(grid_search.best_estimator_))
```

    Best estimator:
    SVC(C=100, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

##### Analyzing the result of cross-validation

```python
import pandas as pd
# convert to Dataframe
results = pd.DataFrame(grid_search.cv_results_)
# show the first 5 rows
display(results.head())
```

      mean_fit_time
      mean_score_time
      mean_test_score
      mean_train_score
      ...
      std_fit_time
      std_score_time
      std_test_score
      std_train_score

      0
      9.13e-04
      3.18e-04
      0.37
      0.37
      ...
      1.08e-04
      3.64e-05
      0.01
      2.85e-03

      1
      8.09e-04
      3.00e-04
      0.37
      0.37
      ...
      1.13e-05
      7.58e-06
      0.01
      2.85e-03

      2
      8.31e-04
      3.15e-04
      0.37
      0.37
      ...
      1.45e-05
      3.49e-05
      0.01
      2.85e-03

      3
      8.23e-04
      3.04e-04
      0.37
      0.37
      ...
      3.22e-05
      1.45e-05
      0.01
      2.85e-03

      4
      8.07e-04
      3.00e-04
      0.37
      0.37
      ...
      1.83e-05
      1.03e-05
      0.01
      2.85e-03

5 rows × 22 columns

```python
scores = np.array(results.mean_test_score).reshape(6, 6)

# plot the mean cross-validation scores
mglearn.tools.heatmap(scores, xlabel='gamma', xticklabels=param_grid['gamma'],
                      ylabel='C', yticklabels=param_grid['C'], cmap="viridis")
```

```python
fig, axes = plt.subplots(1, 3, figsize=(13, 5))

param_grid_linear = {'C': np.linspace(1, 2, 6),
                     'gamma':  np.linspace(1, 2, 6)}

param_grid_one_log = {'C': np.linspace(1, 2, 6),
                      'gamma':  np.logspace(-3, 2, 6)}

param_grid_range = {'C': np.logspace(-3, 2, 6),
                    'gamma':  np.logspace(-7, -2, 6)}

for param_grid, ax in zip([param_grid_linear, param_grid_one_log,
                           param_grid_range], axes):
    grid_search = GridSearchCV(SVC(), param_grid, cv=5)
    grid_search.fit(X_train, y_train)
    scores = grid_search.cv_results_['mean_test_score'].reshape(6, 6)

    # plot the mean cross-validation scores
    scores_image = mglearn.tools.heatmap(
        scores, xlabel='gamma', ylabel='C', xticklabels=param_grid['gamma'],
        yticklabels=param_grid['C'], cmap="viridis", ax=ax)

plt.colorbar(scores_image, ax=axes.tolist())
```

```python
param_grid = [{'kernel': ['rbf'],
               'C': [0.001, 0.01, 0.1, 1, 10, 100],
               'gamma': [0.001, 0.01, 0.1, 1, 10, 100]},
              {'kernel': ['linear'],
               'C': [0.001, 0.01, 0.1, 1, 10, 100]}]
print("List of grids:\n{}".format(param_grid))
```

    List of grids:
    [{'kernel': ['rbf'], 'C': [0.001, 0.01, 0.1, 1, 10, 100], 'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}, {'kernel': ['linear'], 'C': [0.001, 0.01, 0.1, 1, 10, 100]}]

```python
grid_search = GridSearchCV(SVC(), param_grid, cv=5,
                          return_train_score=True)
grid_search.fit(X_train, y_train)
print("Best parameters: {}".format(grid_search.best_params_))
print("Best cross-validation score: {:.2f}".format(grid_search.best_score_))
```

    Best parameters: {'C': 100, 'gamma': 0.01, 'kernel': 'rbf'}
    Best cross-validation score: 0.97

```python
results = pd.DataFrame(grid_search.cv_results_)
# we display the transposed table so that it better fits on the page:
display(results.T)
```

      0
      1
      2
      3
      ...
      38
      39
      40
      41

      mean_fit_time
      0.001
      0.00082
      0.00084
      0.00084
      ...
      0.00043
      0.0004
      0.00041
      0.00041

      mean_score_time
      0.00034
      0.00031
      0.00031
      0.0003
      ...
      0.00024
      0.00023
      0.00023
      0.00021

      mean_test_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.95
      0.97
      0.96
      0.96

      mean_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.97
      0.98
      0.99
      0.99

      param_C
      0.001
      0.001
      0.001
      0.001
      ...
      0.1
      1
      10
      100

      param_gamma
      0.001
      0.01
      0.1
      1
      ...
      NaN
      NaN
      NaN
      NaN

      param_kernel
      rbf
      rbf
      rbf
      rbf
      ...
      linear
      linear
      linear
      linear

      params
      {'C': 0.001, 'gamma': 0.001, 'kernel': 'rbf'}
      {'C': 0.001, 'gamma': 0.01, 'kernel': 'rbf'}
      {'C': 0.001, 'gamma': 0.1, 'kernel': 'rbf'}
      {'C': 0.001, 'gamma': 1, 'kernel': 'rbf'}
      ...
      {'C': 0.1, 'kernel': 'linear'}
      {'C': 1, 'kernel': 'linear'}
      {'C': 10, 'kernel': 'linear'}
      {'C': 100, 'kernel': 'linear'}

      rank_test_score
      27
      27
      27
      27
      ...
      11
      1
      3
      3

      split0_test_score
      0.38
      0.38
      0.38
      0.38
      ...
      0.96
      1
      0.96
      0.96

      split0_train_score
      0.36
      0.36
      0.36
      0.36
      ...
      0.97
      0.99
      0.99
      0.99

      split1_test_score
      0.35
      0.35
      0.35
      0.35
      ...
      0.91
      0.96
      1
      1

      split1_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.98
      0.98
      0.99
      0.99

      split2_test_score
      0.36
      0.36
      0.36
      0.36
      ...
      1
      1
      1
      1

      split2_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.94
      0.98
      0.98
      0.99

      split3_test_score
      0.36
      0.36
      0.36
      0.36
      ...
      0.91
      0.95
      0.91
      0.91

      split3_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.98
      0.99
      0.99
      1

      split4_test_score
      0.38
      0.38
      0.38
      0.38
      ...
      0.95
      0.95
      0.95
      0.95

      split4_train_score
      0.36
      0.36
      0.36
      0.36
      ...
      0.97
      0.99
      1
      1

      std_fit_time
      0.00023
      1.1e-05
      3.4e-05
      3.1e-05
      ...
      1.3e-05
      1.2e-05
      2.5e-05
      4.5e-05

      std_score_time
      4.2e-05
      1.5e-05
      9.9e-06
      1.2e-05
      ...
      2.7e-05
      2.8e-05
      2.3e-05
      1.3e-05

      std_test_score
      0.011
      0.011
      0.011
      0.011
      ...
      0.033
      0.022
      0.034
      0.034

      std_train_score
      0.0029
      0.0029
      0.0029
      0.0029
      ...
      0.012
      0.0055
      0.007
      0.0055

23 rows × 42 columns

#### Using different cross-validation strategies with grid-search

#### Nested cross-validation

```python
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100],
              'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}
scores = cross_val_score(GridSearchCV(SVC(), param_grid, cv=5),
                         iris.data, iris.target, cv=5)
print("Cross-validation scores: ", scores)
print("Mean cross-validation score: ", scores.mean())
```

    Cross-validation scores:  [0.967 1.    0.967 0.967 1.   ]
    Mean cross-validation score:  0.9800000000000001

```python
def nested_cv(X, y, inner_cv, outer_cv, Classifier, parameter_grid):
    outer_scores = []
    # for each split of the data in the outer cross-validation
    # (split method returns indices of training and test part)
    for training_samples, test_samples in outer_cv.split(X, y):
        # find best parameter using inner cross-validation
        best_parms = {}
        best_score = -np.inf
        # iterate over parameters
        for parameters in parameter_grid:
            # accumulate score over inner splits
            cv_scores = []
            # iterate over inner cross-validation
            for inner_train, inner_test in inner_cv.split(
                    X[training_samples], y[training_samples]):
                # build classifier given parameters and training data
                clf = Classifier(**parameters)
                clf.fit(X[inner_train], y[inner_train])
                # evaluate on inner test set
                score = clf.score(X[inner_test], y[inner_test])
                cv_scores.append(score)
            # compute mean score over inner folds
            mean_score = np.mean(cv_scores)
            if mean_score > best_score:
                # if better than so far, remember parameters
                best_score = mean_score
                best_params = parameters
        # build classifier on best parameters using outer training set
        clf = Classifier(**best_params)
        clf.fit(X[training_samples], y[training_samples])
        # evaluate
        outer_scores.append(clf.score(X[test_samples], y[test_samples]))
    return np.array(outer_scores)
```

```python
from sklearn.model_selection import ParameterGrid, StratifiedKFold
scores = nested_cv(iris.data, iris.target, StratifiedKFold(5),
                   StratifiedKFold(5), SVC, ParameterGrid(param_grid))
print("Cross-validation scores: {}".format(scores))
```

    Cross-validation scores: [0.967 1.    0.967 0.967 1.   ]

##### Parallelizing cross-validation and grid search

### Evaluation Metrics and Scoring
#### Keep the End Goal in Mind

#### Metrics for Binary Classification
##### Kinds of errors
##### Imbalanced datasets

```python
from sklearn.datasets import load_digits

digits = load_digits()
y = digits.target == 9

X_train, X_test, y_train, y_test = train_test_split(
    digits.data, y, random_state=0)
```

```python
from sklearn.dummy import DummyClassifier
dummy_majority = DummyClassifier(strategy='most_frequent').fit(X_train, y_train)
pred_most_frequent = dummy_majority.predict(X_test)
print("Unique predicted labels: {}".format(np.unique(pred_most_frequent)))
print("Test score: {:.2f}".format(dummy_majority.score(X_test, y_test)))
```

    Unique predicted labels: [False]
    Test score: 0.90

```python
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=2).fit(X_train, y_train)
pred_tree = tree.predict(X_test)
print("Test score: {:.2f}".format(tree.score(X_test, y_test)))
```

    Test score: 0.92

```python
from sklearn.linear_model import LogisticRegression

dummy = DummyClassifier().fit(X_train, y_train)
pred_dummy = dummy.predict(X_test)
print("dummy score: {:.2f}".format(dummy.score(X_test, y_test)))

logreg = LogisticRegression(C=0.1).fit(X_train, y_train)
pred_logreg = logreg.predict(X_test)
print("logreg score: {:.2f}".format(logreg.score(X_test, y_test)))
```

    dummy score: 0.82
    logreg score: 0.98

##### Confusion matrices

```python
from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, pred_logreg)
print("Confusion matrix:\n{}".format(confusion))
```

    Confusion matrix:
    [[401   2]
     [  8  39]]

```python
mglearn.plots.plot_confusion_matrix_illustration()
```

```python
mglearn.plots.plot_binary_confusion_matrix()
```

```python
print("Most frequent class:")
print(confusion_matrix(y_test, pred_most_frequent))
print("\nDummy model:")
print(confusion_matrix(y_test, pred_dummy))
print("\nDecision tree:")
print(confusion_matrix(y_test, pred_tree))
print("\nLogistic Regression")
print(confusion_matrix(y_test, pred_logreg))
```

    Most frequent class:
    [[403   0]
     [ 47   0]]

    Dummy model:
    [[353  50]
     [ 43   4]]

    Decision tree:
    [[390  13]
     [ 24  23]]

    Logistic Regression
    [[401   2]
     [  8  39]]

###### Relation to accuracy
\begin{equation}
\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}
\end{equation}

##### Precision, recall and f-score
\begin{equation}
\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}
\end{equation}

\begin{equation}
\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}
\end{equation}
\begin{equation}
\text{F} = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}
\end{equation}

```python
from sklearn.metrics import f1_score
print("f1 score most frequent: {:.2f}".format(
    f1_score(y_test, pred_most_frequent)))
print("f1 score dummy: {:.2f}".format(f1_score(y_test, pred_dummy)))
print("f1 score tree: {:.2f}".format(f1_score(y_test, pred_tree)))
print("f1 score logistic regression: {:.2f}".format(
    f1_score(y_test, pred_logreg)))
```

    f1 score most frequent: 0.00
    f1 score dummy: 0.08
    f1 score tree: 0.55
    f1 score logistic regression: 0.89

    /srv/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.
      'precision', 'predicted', average, warn_for)

```python
from sklearn.metrics import classification_report
print(classification_report(y_test, pred_most_frequent,
                            target_names=["not nine", "nine"]))
```

                 precision    recall  f1-score   support

       not nine       0.90      1.00      0.94       403
           nine       0.00      0.00      0.00        47

    avg / total       0.80      0.90      0.85       450

    /srv/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
      'precision', 'predicted', average, warn_for)

```python
print(classification_report(y_test, pred_dummy,
                            target_names=["not nine", "nine"]))
```

                 precision    recall  f1-score   support

       not nine       0.89      0.88      0.88       403
           nine       0.07      0.09      0.08        47

    avg / total       0.81      0.79      0.80       450

```python
print(classification_report(y_test, pred_logreg,
                            target_names=["not nine", "nine"]))
```

                 precision    recall  f1-score   support

       not nine       0.98      1.00      0.99       403
           nine       0.95      0.83      0.89        47

    avg / total       0.98      0.98      0.98       450

##### Taking uncertainty into account

```python
from mglearn.datasets import make_blobs
X, y = make_blobs(n_samples=(400, 50), centers=2, cluster_std=[7.0, 2],
                  random_state=22)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
svc = SVC(gamma=.05).fit(X_train, y_train)
```

```python
mglearn.plots.plot_decision_threshold()
```

```python
print(classification_report(y_test, svc.predict(X_test)))
```

                 precision    recall  f1-score   support

              0       0.97      0.89      0.93       104
              1       0.35      0.67      0.46         9

    avg / total       0.92      0.88      0.89       113

```python
y_pred_lower_threshold = svc.decision_function(X_test) > -.8
```

```python
print(classification_report(y_test, y_pred_lower_threshold))
```

                 precision    recall  f1-score   support

              0       1.00      0.82      0.90       104
              1       0.32      1.00      0.49         9

    avg / total       0.95      0.83      0.87       113

##### Precision-Recall curves and ROC curves

```python
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(
    y_test, svc.decision_function(X_test))
```

```python
# create a similar dataset as before, but with more samples
# to get a smoother curve
X, y = make_blobs(n_samples=(4000, 500), centers=2, cluster_std=[7.0, 2],
                  random_state=22)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
svc = SVC(gamma=.05).fit(X_train, y_train)
precision, recall, thresholds = precision_recall_curve(
    y_test, svc.decision_function(X_test))
# find threshold closest to zero
close_zero = np.argmin(np.abs(thresholds))
plt.plot(precision[close_zero], recall[close_zero], 'o', markersize=10,
         label="threshold zero", fillstyle="none", c='k', mew=2)

plt.plot(precision, recall, label="precision recall curve")
plt.xlabel("Precision")
plt.ylabel("Recall")
plt.legend(loc="best")
```

```python
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, random_state=0, max_features=2)
rf.fit(X_train, y_train)

# RandomForestClassifier has predict_proba, but not decision_function
precision_rf, recall_rf, thresholds_rf = precision_recall_curve(
    y_test, rf.predict_proba(X_test)[:, 1])

plt.plot(precision, recall, label="svc")

plt.plot(precision[close_zero], recall[close_zero], 'o', markersize=10,
         label="threshold zero svc", fillstyle="none", c='k', mew=2)

plt.plot(precision_rf, recall_rf, label="rf")

close_default_rf = np.argmin(np.abs(thresholds_rf - 0.5))
plt.plot(precision_rf[close_default_rf], recall_rf[close_default_rf], '^', c='k',
         markersize=10, label="threshold 0.5 rf", fillstyle="none", mew=2)
plt.xlabel("Precision")
plt.ylabel("Recall")
plt.legend(loc="best")
```

```python
print("f1_score of random forest: {:.3f}".format(
    f1_score(y_test, rf.predict(X_test))))
print("f1_score of svc: {:.3f}".format(f1_score(y_test, svc.predict(X_test))))
```

    f1_score of random forest: 0.610
    f1_score of svc: 0.656

```python
from sklearn.metrics import average_precision_score
ap_rf = average_precision_score(y_test, rf.predict_proba(X_test)[:, 1])
ap_svc = average_precision_score(y_test, svc.decision_function(X_test))
print("Average precision of random forest: {:.3f}".format(ap_rf))
print("Average precision of svc: {:.3f}".format(ap_svc))
```

    Average precision of random forest: 0.660
    Average precision of svc: 0.666

##### Receiver Operating Characteristics (ROC) and AUC
\begin{equation}
\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}
\end{equation}

```python
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, svc.decision_function(X_test))

plt.plot(fpr, tpr, label="ROC Curve")
plt.xlabel("FPR")
plt.ylabel("TPR (recall)")
# find threshold closest to zero
close_zero = np.argmin(np.abs(thresholds))
plt.plot(fpr[close_zero], tpr[close_zero], 'o', markersize=10,
         label="threshold zero", fillstyle="none", c='k', mew=2)
plt.legend(loc=4)
```

```python
fpr_rf, tpr_rf, thresholds_rf = roc_curve(y_test, rf.predict_proba(X_test)[:, 1])

plt.plot(fpr, tpr, label="ROC Curve SVC")
plt.plot(fpr_rf, tpr_rf, label="ROC Curve RF")

plt.xlabel("FPR")
plt.ylabel("TPR (recall)")
plt.plot(fpr[close_zero], tpr[close_zero], 'o', markersize=10,
         label="threshold zero SVC", fillstyle="none", c='k', mew=2)
close_default_rf = np.argmin(np.abs(thresholds_rf - 0.5))
plt.plot(fpr_rf[close_default_rf], tpr[close_default_rf], '^', markersize=10,
         label="threshold 0.5 RF", fillstyle="none", c='k', mew=2)

plt.legend(loc=4)
```

```python
from sklearn.metrics import roc_auc_score
rf_auc = roc_auc_score(y_test, rf.predict_proba(X_test)[:, 1])
svc_auc = roc_auc_score(y_test, svc.decision_function(X_test))
print("AUC for Random Forest: {:.3f}".format(rf_auc))
print("AUC for SVC: {:.3f}".format(svc_auc))
```

    AUC for Random Forest: 0.937
    AUC for SVC: 0.916

```python
y = digits.target == 9

X_train, X_test, y_train, y_test = train_test_split(
    digits.data, y, random_state=0)

plt.figure()

for gamma in [1, 0.05, 0.01]:
    svc = SVC(gamma=gamma).fit(X_train, y_train)
    accuracy = svc.score(X_test, y_test)
    auc = roc_auc_score(y_test, svc.decision_function(X_test))
    fpr, tpr, _ = roc_curve(y_test , svc.decision_function(X_test))
    print("gamma = {:.2f}  accuracy = {:.2f}  AUC = {:.2f}".format(
        gamma, accuracy, auc))
    plt.plot(fpr, tpr, label="gamma={:.3f}".format(gamma))
plt.xlabel("FPR")
plt.ylabel("TPR")
plt.xlim(-0.01, 1)
plt.ylim(0, 1.02)
plt.legend(loc="best")
```

#### Multi-class classification

```python
from sklearn.metrics import accuracy_score
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target, random_state=0)
lr = LogisticRegression().fit(X_train, y_train)
pred = lr.predict(X_test)
print("Accuracy: {:.3f}".format(accuracy_score(y_test, pred)))
print("Confusion matrix:\n{}".format(confusion_matrix(y_test, pred)))
```

    Accuracy: 0.953
    Confusion matrix:
    [[37  0  0  0  0  0  0  0  0  0]
     [ 0 39  0  0  0  0  2  0  2  0]
     [ 0  0 41  3  0  0  0  0  0  0]
     [ 0  0  1 43  0  0  0  0  0  1]
     [ 0  0  0  0 38  0  0  0  0  0]
     [ 0  1  0  0  0 47  0  0  0  0]
     [ 0  0  0  0  0  0 52  0  0  0]
     [ 0  1  0  1  1  0  0 45  0  0]
     [ 0  3  1  0  0  0  0  0 43  1]
     [ 0  0  0  1  0  1  0  0  1 44]]

```python
scores_image = mglearn.tools.heatmap(
    confusion_matrix(y_test, pred), xlabel='Predicted label',
    ylabel='True label', xticklabels=digits.target_names,
    yticklabels=digits.target_names, cmap=plt.cm.gray_r, fmt="%d")
plt.title("Confusion matrix")
plt.gca().invert_yaxis()
```

```python
print(classification_report(y_test, pred))
```

                 precision    recall  f1-score   support

              0       1.00      1.00      1.00        37
              1       0.89      0.91      0.90        43
              2       0.95      0.93      0.94        44
              3       0.90      0.96      0.92        45
              4       0.97      1.00      0.99        38
              5       0.98      0.98      0.98        48
              6       0.96      1.00      0.98        52
              7       1.00      0.94      0.97        48
              8       0.93      0.90      0.91        48
              9       0.96      0.94      0.95        47

    avg / total       0.95      0.95      0.95       450

```python
print("Micro average f1 score: {:.3f}".format(
    f1_score(y_test, pred, average="micro")))
print("Macro average f1 score: {:.3f}".format(
    f1_score(y_test, pred, average="macro")))
```

    Micro average f1 score: 0.953
    Macro average f1 score: 0.954

#### Regression metrics

### Using evaluation metrics in model selection

```python
# default scoring for classification is accuracy
print("Default scoring: {}".format(
    cross_val_score(SVC(), digits.data, digits.target == 9)))
# providing scoring="accuracy" doesn't change the results
explicit_accuracy =  cross_val_score(SVC(), digits.data, digits.target == 9,
                                     scoring="accuracy")
print("Explicit accuracy scoring: {}".format(explicit_accuracy))
roc_auc =  cross_val_score(SVC(), digits.data, digits.target == 9,
                           scoring="roc_auc")
print("AUC scoring: {}".format(roc_auc))
```

    Default scoring: [0.9 0.9 0.9]
    Explicit accuracy scoring: [0.9 0.9 0.9]
    AUC scoring: [0.994 0.99  0.996]

```python
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target == 9, random_state=0)

# we provide a somewhat bad grid to illustrate the point:
param_grid = {'gamma': [0.0001, 0.01, 0.1, 1, 10]}
# using the default scoring of accuracy:
grid = GridSearchCV(SVC(), param_grid=param_grid)
grid.fit(X_train, y_train)
print("Grid-Search with accuracy")
print("Best parameters:", grid.best_params_)
print("Best cross-validation score (accuracy)): {:.3f}".format(grid.best_score_))
print("Test set AUC: {:.3f}".format(
    roc_auc_score(y_test, grid.decision_function(X_test))))
print("Test set accuracy: {:.3f}".format(grid.score(X_test, y_test)))

# using AUC scoring instead:
grid = GridSearchCV(SVC(), param_grid=param_grid, scoring="roc_auc")
grid.fit(X_train, y_train)
print("\nGrid-Search with AUC")
print("Best parameters:", grid.best_params_)
print("Best cross-validation score (AUC): {:.3f}".format(grid.best_score_))
print("Test set AUC: {:.3f}".format(
    roc_auc_score(y_test, grid.decision_function(X_test))))
print("Test set accuracy: {:.3f}".format(grid.score(X_test, y_test)))
```

    Grid-Search with accuracy
    Best parameters: {'gamma': 0.0001}
    Best cross-validation score (accuracy)): 0.970
    Test set AUC: 0.992
    Test set accuracy: 0.973

    Grid-Search with AUC
    Best parameters: {'gamma': 0.01}
    Best cross-validation score (AUC): 0.997
    Test set AUC: 1.000
    Test set accuracy: 1.000

```python
from sklearn.metrics.scorer import SCORERS
print("Available scorers:\n{}".format(sorted(SCORERS.keys())))
```

    Available scorers:
    ['accuracy', 'adjusted_mutual_info_score', 'adjusted_rand_score', 'average_precision', 'completeness_score', 'explained_variance', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'fowlkes_mallows_score', 'homogeneity_score', 'log_loss', 'mean_absolute_error', 'mean_squared_error', 'median_absolute_error', 'mutual_info_score', 'neg_log_loss', 'neg_mean_absolute_error', 'neg_mean_squared_error', 'neg_mean_squared_log_error', 'neg_median_absolute_error', 'normalized_mutual_info_score', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc', 'v_measure_score']

### Summary and Outlook

```python

```

```python
tested; Gopal
```


**Objective:**

In this tutorial we will create a simple magnetic problem from scratch using the SimPEG framework.

We are using the integral form of the magnetostatic problem. In the absence of free-currents or changing magnetic field, magnetic material can give rise to a secondary magnetic field according to:

$$\vec b = \frac{\mu_0}{4\pi}  \int_{V}   \vec M \cdot \nabla \nabla \left(\frac{1}{r}\right) \; dV $$

Where $\mu_0$ is the magnetic permealitity of free-space, $\vec M$ is the magnetization per unit volume and $r$  defines the distance between the observed field $\vec b$ and the magnetized object. Assuming a purely induced response, the strenght of magnetization can be written as:

$$ \vec M = \mu_0 \kappa \vec H_0 $$

where $\vec H$ is an external inducing magnetic field, and $\kappa$ the magnetic susceptibility of matter.
As derived by Sharma 1966, the integral can be evaluated for rectangular prisms such that:

$$ \vec b(P) =  \mathbf{T} \cdot \vec H_0 \; \kappa $$

Where the tensor matrix $\bf{T}$ relates the three components of magnetization $\vec M$ to the components of the field $\vec b$:

$$\mathbf{T} =
	 \begin{pmatrix}
       		T_{xx} & T_{xy} & T_{xz}    \\
		T_{yx} & T_{yy} & T_{yz}    \\
		T_{zx} & T_{zy} & T_{zz}
	\end{pmatrix} $$

In general, we discretize the earth into a collection of cells, each contributing to the magnetic data such that:

$$\vec b(P) = \sum_{j=1}^{nc} \mathbf{T}_j \cdot \vec H_0 \; \kappa_j$$

giving rise to a linear problem.

```python
## Need to be on \pf\dev branch !!!
%pylab inline
import SimPEG.PF as PF
from SimPEG import *
from SimPEG.Utils import io_utils
```

    Populating the interactive namespace from numpy and matplotlib
    Efficiency Warning: Interpolation will be slow, use setup.py!

                python setup.py build_ext --inplace

```python
import matplotlib
matplotlib.rcParams['font.size'] = 14
```

```python
# We first need to create a susceptibility model.
# Based on a set of parametric surfaces representing TKC,
# we use VTK to discretize the 3-D space.

mshfile = 'MEsh_TEst.msh'

model_dir = '../../Geological_model/'

# Load mesh file
mesh = Mesh.TensorMesh.readUBC(model_dir+mshfile)

# Create our own mesh!
csx, csy, csz = 10., 10., 10.
ncx, ncy, ncz = 55, 55, 30
npad = 10
hx = [(csx,npad, -1.3),(csx,ncx),(csx,npad, 1.3)]
hy = [(csy,npad, -1.3),(csy,ncy),(csy,npad, 1.3)]
hz = [(csz,npad, -1.3),(csz,ncz), (10,20)]
mesh = Mesh.TensorMesh([hx, hy, hz],x0="CCN")
xc = 300+5.57e5
yc = 600+7.133e6
zc = 450.
x0_new = np.r_[mesh.x0[0]+xc, mesh.x0[1]+yc, mesh.x0[2]+zc]
mesh._x0 = x0_new

#Mesh.TensorMesh.writeUBC(mesh,model_dir+"Mesh_mag.msh")
# Define no-data-value
ndv = -100

# Define survey flight height
Z_bird = 20.

# Read in topo surface
topofile = model_dir+'TKCtopo.dat'
geosurf = [
    [model_dir+'Till.ts',True,True,0],
    [model_dir+'XVK.ts',True,True,1],
    [model_dir+'PK1.ts',True,True,2],
    [model_dir+'PK2.ts',True,True,3],
    [model_dir+'PK3.ts',True,True,4],
    [model_dir+'HK1.ts',True,True,5],
    [model_dir+'VK.ts',True,True,6]
]

```

```python
import time as tm
import mpl_toolkits.mplot3d as a3
import matplotlib.colors as colors
import scipy as sp
# from mayavi import mlab
# mlab.close(all=True)
# fig = plt.figure(figsize=(8,6))

# mlab.figure(bgcolor=(1.,1.,1.))
modelInd = np.ones(mesh.nC)*ndv
for ii in range(len(geosurf)):
    tin = tm.time()
    print "Computing indices with VTK: " + geosurf[ii][0]
    T, S = io_utils.read_GOCAD_ts(geosurf[ii][0])
    indx = io_utils.surface2inds(T,S,mesh, boundaries=geosurf[ii][1], internal=geosurf[ii][2])
    print "VTK operation completed in " + str(tm.time() - tin) + " sec"
    modelInd[indx] = geosurf[ii][3]

#     S = S-1
#     mlab.triangular_mesh(T[:,0], T[:,1], T[:,2], S)
#     mlab.view(azimuth=-45.,distance=1000,elevation=75.)

# arr = mlab.screenshot()

# plt.imshow(arr)
```

    Computing indices with VTK: ../../Geological_model/Till.ts
    Extracting indices from grid...
    VTK operation completed in 30.1270000935 sec
    Computing indices with VTK: ../../Geological_model/XVK.ts
    Extracting indices from grid...
    VTK operation completed in 5.49399995804 sec
    Computing indices with VTK: ../../Geological_model/PK1.ts
    Extracting indices from grid...
    VTK operation completed in 12.4779999256 sec
    Computing indices with VTK: ../../Geological_model/PK2.ts
    Extracting indices from grid...
    VTK operation completed in 5.28500008583 sec
    Computing indices with VTK: ../../Geological_model/PK3.ts
    Extracting indices from grid...
    VTK operation completed in 7.125 sec
    Computing indices with VTK: ../../Geological_model/HK1.ts
    Extracting indices from grid...
    VTK operation completed in 12.9039998055 sec
    Computing indices with VTK: ../../Geological_model/VK.ts
    Extracting indices from grid...
    VTK operation completed in 13.7690000534 sec

```python
# Load topography file in UBC format and find the active cells
#T, S = io_utils.read_GOCAD_ts(topsurf)
#indx = io_utils.surface2inds(T,S, mesh, boundaries=True, internal=True)
#actv = np.zeros(mesh.nC)
#actv[indx] = 1

topo = np.genfromtxt(topofile,skip_header=1)
# Find the active cells
actv = Utils.surface2ind_topo(mesh, topo, gridLoc='N')
# actv = PF.Magnetics.getActiveTopo(mesh,topo,'N')

# Create active map to go from reduce set to full
actvMap = Maps.InjectActiveCells(mesh, actv, ndv)

print "Active cells created from topography!"
```

    Active cells created from topography!

```python
# Build model

def getModel(Till=0.0, XVK=2e-3, PK1=5e-3, PK2=2e-3, PK3=1e-3, HK1=2e-2, VK=1e-2, bkgr=0.):
    vals = [Till, XVK, PK1, PK2, PK3, HK1, VK]
    model= np.ones(mesh.nC) * bkgr

    for ii, sus in zip(range(7),vals):
        model[modelInd == ii] = sus
    return model
model = getModel()
sus = getModel()
model = model[actv]
```

```python
from ipywidgets.widgets import interact, IntSlider
```

```python
# Here you can visualize the current model
m_true = actvMap*model
Mesh.TensorMesh.writeModelUBC(mesh,model_dir+"Synthetic_mag.sus",m_true)
airc = m_true==ndv
m_true[airc] = np.nan
def slide(s,normal):
    colorbar(mesh.plotSlice(m_true, normal=normal, ind=s, clim=np.r_[model.min(), model.max()])[0])
    plt.gca().set_aspect('equal')
interact(slide, s=(0,60), normal=['X','Y','Z'])
```

**Forward system:**

Now that we have all our spatial components, we can create our linear system. For a single location and single component of the data, the system would looks like this:

$$ b_x =
	\begin{bmatrix}
	T_{xx}^1 &... &T_{xx}^{nc} & T_{xy}^1 & ... & T_{xy}^{nc} & T_{xz}^1 & ... & T_{xz}^{nc}\\
	 \end{bmatrix}
	 \begin{bmatrix}
		\mathbf{M}_x \\ \mathbf{M}_y \\ \mathbf{M}_z
	\end{bmatrix} \\ $$

where each of $T_{xx},\;T_{xy},\;T_{xz}$ are [nc x 1] long. For the $y$ and $z$ component, we need the two other rows of the tensor $\mathbf{T}$.
In our simple induced case, the magnetization direction $\mathbf{M_x,\;M_y\;,Mz}$ are known and assumed to be constant everywhere, so we can reduce the size of the system such that:

$$ \vec{\mathbf{d}}_{\text{pred}} = (\mathbf{T\cdot M})\; \kappa$$

In most geophysical surveys, we are not collecting all three components, but rather the magnitude of the field, or $Total\;Magnetic\;Intensity$ (TMI) data.
Because the inducing field is really large, we will assume that the anomalous fields are parallel to $H_0$:

$$ d^{TMI}  = \hat H_0 \cdot \vec d$$

We then end up with a much smaller system:

$$ d^{TMI} = \mathbf{F\; \kappa}$$

where $\mathbf{F} \in \mathbb{R}^{nd \times nc}$ is our $forward$ operator.

```python
from scipy.interpolate import NearestNDInterpolator
# We need to define the direction of the inducing field
# From old convention, field orientation is given as an azimuth from North
# (positive clockwise) and dip from the horizontal (positive downward).
# The field parameters at TKC are [H:60,308 nT, I:83.8 d D:25.4 d ]
H0 = (60308.,83.8,25.4)

# We create a synthetic survey with observations in cell center.
X, Y = np.meshgrid(mesh.vectorCCx[npad:-npad:2], mesh.vectorCCy[npad:-npad:2])

# Using our topography, we trape the survey and shift it up by the flight height
Ftopo = NearestNDInterpolator(topo[:,:2], topo[:,2])
Z = Ftopo(Utils.mkvc(X.T),Utils.mkvc(Y.T)) + Z_bird

rxLoc = np.c_[Utils.mkvc(X.T), Utils.mkvc(Y.T), Utils.mkvc(Z.T)]
rxLoc = PF.BaseMag.RxObs(rxLoc)

srcField = PF.BaseMag.SrcField([rxLoc])
srcField.param = H0
survey = PF.BaseMag.LinearSurvey(srcField)
```

```python
# Now that we have a model and a survey we can build the linear system ...
nactv = np.int(np.sum(actv))

# Creat reduced identity map
idenMap = Maps.IdentityMap(nP=nactv)

# Create the forward model operator
prob = PF.Magnetics.Problem3D_Integral(mesh, mapping=idenMap, actInd=actv, forwardOnly=False, rtype = 'tmi')

# Pair the survey and problem
survey.pair(prob)

```

```python
# Fist time that we ask for predicted data,
# the dense matrix T is calculated.
# This is generally the bottleneck of the integral formulation,
# in terms of time and memory.
d = prob.fields(model)

# Add noise to the data and assign uncertainties
data = d + randn(len(d)) # We add some random Gaussian noise (1nT)
wd = np.ones(len(data))*1. # Assign flat uncertainties

survey.dobs = data
survey.std = wd

PF.Magnetics.writeUBCobs('MAG_Synthetic_data.obs',survey,data)

d2D = data.reshape(X.shape)
dat = plt.contourf(X-xc,Y-yc, d2D,40)
plt.gca().set_aspect('equal')
plt.plot(X.flatten()-xc,Y.flatten()-yc,'k.', ms=2)
plt.colorbar(dat)
plt.xlabel("Easting (m)")
plt.ylabel("Northing (m)")
plt.title("Total Magnetic Intensity (nT)")
xlim(-500, 500)
ylim(-500, 500)
```

```python
# d2Dx, d2Dy, d2Dz = d[0:survey.nRx].reshape(X.shape), d[survey.nRx:2*survey.nRx].reshape(X.shape),  d[2*survey.nRx:].reshape(X.shape)

# plt.figure(figsize=[5,5])
# xc, yc = (X.min()+X.max())*0.5, (Y.min()+Y.max())*0.5
# #     colorbar(imshow(d2D,extent=[X.min()-xc, X.max()-xc, Y.min()-yc, Y.max()-yc],origin = 'lower'))
# #     plt.contour(X-xc,Y-yc, d2D,20,colors='k')
# # dat = plt.contourf(X-xc,Y-yc, d2Dz,40)

# # plt.gca().set_aspect('equal')

# mlab.figure(bgcolor=(1.,1.,1.))
# mlab.triangular_mesh(T[:,0], T[:,1], T[:,2], S)
# mlab.quiver3d(X,Y,Z,d2Dx, d2Dy, d2Dz, color)
# mlab.view(azimuth=-45.,distance=1500,elevation=75.)
# arr = mlab.screenshot()

# plt.imshow(arr)
```

**Inverse problem**

We have generated synthetic data, we now what to see if we can solve the inverse. Using the usual formulation, we seek a model that can reproduce the data, let’s say a least-squares measure of the form:

\begin{equation}
\phi_d =   \|\mathbf{W}_d \left( \mathbb{F}[\mathbf{m}] - \mathbf{d}^{obs} \right)\|_2^2
\end{equation}

The inverse problem is hard because we don’t have great data coverage, and the Earth is big, and there is usually noise in the data. So we need to add something to regularize it.
The simplest way to do it is to penalize solutions that won’t make sense geologically, for example to assume that the model is small.
The usual smooth inversion function use an l2-norm measure:

\begin{equation}
\phi_d =   \|\mathbf{W}_d \left( \mathbb{F}[\mathbf{m}] - \mathbf{d}^{obs} \right)\|_2^2 \\
\phi_m = \beta \Big [ {\| \mathbf{W}_s \;( \mathbf{m - m^{ref}})\|}^2_2  + \sum_{i = x,y,z}  {\|   \mathbf{W}_i  \; \mathbf{G}_i \; \mathbf{m}\|}^2_2  \Big ]\;,
\end{equation}

The full objective function to be minimized can be written as:
\begin{equation}
\phi(m) =  \phi_d + \beta \phi_m\;,
\end{equation}
which will yield our usual *small* and *smooth* models.

We propose a fancier regularization function that can allow to recover *sparse* and *blocky* solutions.
Starting with the well known Ekblom norm:
\begin{equation}
\phi_m =  \sum_{i=1}^{nc} {(x_i^2 + \epsilon^2)}^{p/2} \;,
\end{equation}
where $x_i$ denotes some function of the model parameter, and $\epsilon$ is a small value to avoid singularity as $m\rightarrow0$.
For p=2, we get the usual least-squares measure and we recover the regularization presented above. For $p \leq 1$, the function becomes non-linear which requires some tweaking.

We can linearize the function by updating the penality function iteratively, commonly known as an Iterative Re-weighted Least-Squares (IRLS) method:
\begin{equation}
\phi_m^{(k)} =  \frac{1}{2}\sum_{i=1}^{nc} r_i \; x_i^2
\end{equation}
where we added the superscript $\square^{(k)}$ to denote the IRLS iterations. The weights $r(x)$ are computed from model values obtained at a previous iteration such that:
\begin{equation}
	{r}_i  ={\Big( {({x_i}^{(k-1)})}^{2} + \epsilon^2 \Big)}^{p/2 - 1} \;,
\end{equation}
where ${r}(x) \in \mathbb{R}^{nc}$.

In matrix form, our objective function simply becomes:
\begin{equation}
\phi(m) =   \|\mathbf{W}_d \left( \mathbb{F}[\mathbf{m}] - \mathbf{d}^{obs} \right)\|_2^2 + \beta \Big [ {\| \mathbf{W}_s \;\mathbf{R}_s\;( \mathbf{m - m^{ref}})\|}^2_2  + \sum_{i = x,y,z}  {\|   \mathbf{W}_i\; \mathbf{R}_i  \; \mathbf{G}_i \; \mathbf{m}\|}^2_2  \Big ]\;,
\end{equation}
where the IRLS weights $\mathbf{R}_s$ and $\mathbf{R}_i$ are diagonal matrices defined as:
\begin{equation}
\begin{split}
	{R}_{s_{jj}}  &=  \sqrt{\eta_p}{\Big[ {({m_j}^{(k-1)})}^{2} + \epsilon_p^2 \Big]}^{(p/2 - 1)/2} \\
	{R}_{i_{jj}}  &=  \sqrt{\eta_q}{\Big[ {\left ({{(G_i\;m^{(k-1)})}_j }\right)}^{2} + \epsilon_q^2 \Big]}^{(q/2 - 1)/2} \\
\eta_p &=  {\epsilon_p}^{(1-p/2)} \\
\eta_q &=   {\epsilon_q}^{(1-q/2)}  \;,
\end{split}
\end{equation}

we added two scaling parameters $\eta_p$ and $\eta_q$ for reasons that we won't dicuss here, but turn out to be important to get stable solves.

In order to initialize the IRLS and get an estimate for the stabilizing parameters $\epsilon_p$ and $\epsilon_q$, we first invert with the smooth $l_2$-norm.
The whole IRLS process is implemented with a directive added to the inversion workflow (see below).

```python
# It is potential fields, so we will need to push the inverison down
# Create distance weights from our linera forward operator
wr = np.sum(prob.G**2.,axis=0)**0.5
wr = ( wr/np.max(wr) )
#wr = PF.Magnetics.get_dist_wgt(mesh, survey.srcField.rxList[0].locs, actv, 3., np.min(mesh.hx)/4.)
#wr = wr**2. # Need to square to be included as weights

reg = Regularization.Sparse(mesh, indActive = actv, mapping = idenMap)
reg.cell_weights = wr

dmis = DataMisfit.l2_DataMisfit(survey)
dmis.Wd = 1/wd

# Add directives to the inversion
opt = Optimization.ProjectedGNCG(maxIter=100 ,lower=0.,upper=1., maxIterLS = 20, maxIterCG= 10, tolCG = 1e-3)
invProb = InvProblem.BaseInvProblem(dmis, reg, opt)
betaest = Directives.BetaEstimate_ByEig()

# Here is where the norms are applied
eps = [3e-4,3e-4] # Threshold parameters to penalize different model parameters
norms = [0,1,1,1] # Norms applies on the model and 3 gradients [p, qx, qy, qz]

IRLS = Directives.Update_IRLS( norms=norms,  eps=eps, f_min_change = 1e-1, minGNiter=6)
update_Jacobi = Directives.Update_lin_PreCond()
inv = Inversion.BaseInversion(invProb, directiveList=[IRLS,betaest,update_Jacobi])

m0 = np.ones(idenMap.nP)*1e-4

```

```python
# Run inversion...
mrec = inv.run(m0)
```

```python
# Get the final model back to full space
m_lp = actvMap*mrec
m_lp[airc] = np.nan

# Get the smooth model aslo
m_l2 = actvMap*reg.l2model
m_l2[airc] = np.nan

# Save both models to file
Mesh.TensorMesh.writeModelUBC(mesh,'SimPEG_MAG_l2l2.sus',m_l2)
Mesh.TensorMesh.writeModelUBC(mesh,'SimPEG_MAG_lplq.sus',m_lp)
```

```python
# Plot the recoverd models
vmin, vmax = 0., 0.015

mesh = Mesh.TensorMesh([mesh.hx, mesh.hy, mesh.hz],x0="CCN")

def slide(s,normal):

    if normal == "Z":
        fig = plt.figure(figsize(10*1.2, 8))
    else:
        fig = plt.figure(figsize(10*1.2, 4))

    ax1 = plt.subplot(2,2,3)
    dat = mesh.plotSlice(m_lp, ax = ax1, normal=normal, ind=s, clim=np.r_[vmin, vmax],pcolorOpts={'cmap':'viridis'})
#     plt.colorbar(dat[0])
    plt.gca().set_aspect('equal')
    plt.title('Compact model')

    if normal == "Z":
        xlim(-600, 600)
        ylim(-600, 600.)
    else:
        xlim(-600, 600)
        ylim(-500, 0.)

    ax2 = plt.subplot(2,2,1)
    dat = mesh.plotSlice(m_l2, ax = ax2, normal=normal, ind=s, clim=np.r_[vmin, vmax],pcolorOpts={'cmap':'viridis'})
#     plt.colorbar(dat[0])
    plt.gca().set_aspect('equal')
    plt.title('Smooth model')

    if normal == "Z":
        xlim(-600, 600)
        ylim(-600, 600.)
    else:
        xlim(-600, 600)
        ylim(-500, 0.)

    ax2.set_xticklabels([])

    ax2 = plt.subplot(1,2,2)
    dat = mesh.plotSlice(m_true, ax = ax2, normal=normal, ind=s, clim=np.r_[vmin, vmax],pcolorOpts={'cmap':'viridis'})
#     plt.colorbar(dat[0])
    plt.gca().set_aspect('equal')
    plt.title('True model')

    pos =  ax2.get_position()

    ax2.yaxis.set_visible(False)
    if normal == "Z":
        xlim(-600, 600)
        ylim(-600, 600.)
        ax2.set_position([pos.x0 -0.04 , pos.y0,  pos.width, pos.height])
    else:
        xlim(-600, 600)
        ylim(-500, 0.)

    pos =  ax2.get_position()
    cbarax = fig.add_axes([pos.x0 + 0.375 , pos.y0 + 0.05,  pos.width*0.1, pos.height*0.75])  ## the parameters are the specified position you set
    cb = fig.colorbar(dat[0],cax=cbarax, orientation="vertical", ax = ax2, ticks=np.linspace(vmin,vmax, 4))
    cb.set_label("Susceptibility (SI)",size=12)
    fig.savefig('PF_Compact.png',dpi = 150)

interact(slide, s=(0,60), normal=['X','Y','Z'])

# interact(lambda ind: viz(m_l2, ind, normal="Z"), ind=IntSlider(min=0, max=32,step=1, value=28))

```

```python
# Lets compare the distribution of model parameters and model gradients
plt.figure(figsize=[15,5])
ax = plt.subplot(121)
plt.hist(reg.l2model,100)
plt.plot((eps[0],eps[0]),(0,mesh.nC),'r--')
plt.yscale('log', nonposy='clip')
plt.title('Hist model - l2-norm')
ax = plt.subplot(122)
plt.hist(reg.regmesh.cellDiffxStencil*reg.l2model,100)
plt.plot((eps[1],eps[1]),(0,mesh.nC),'r--')
plt.yscale('log', nonposy='clip')
plt.title('Hist model gradient - l2-norm')

# Lets look at the distribution of model parameters and model gradients
plt.figure(figsize=[15,5])
ax = plt.subplot(121)
plt.hist(mrec,100)
plt.plot((eps[0],eps[0]),(0,mesh.nC),'r--')
plt.yscale('log', nonposy='clip')
plt.title('Hist model - lp-norm')
ax = plt.subplot(122)
plt.hist(reg.regmesh.cellDiffxStencil*mrec,100)
plt.plot((eps[1],eps[1]),(0,mesh.nC),'r--')
plt.yscale('log', nonposy='clip')
plt.title('Hist model gradient - lp-norm')
```

```python
Mesh.MeshIO.TensorMeshIO.writeUBC(mesh,'MAG_mesh.msh')

# import pickle
# Results = {"mesh":mesh, "model_true":sus, "model_pred":m_IRLS, "Obs":data, "XYZ": xyz}
# outputs = open("Magresults", 'wb')
# pickle.dump(Results, outputs)
# outputs.close()
```

```python

```


```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
from hottbox.core import Tensor
```

[Return to Table of Contents](./0_Table_of_contents.ipynb)

# Tensors and basic operations

A tensor is a multi-dimenaional array of data where each dimension is conventionally referred to as the **mode**. Tensor order is defined by the number of its modes which is equivivalent to the number of indices required to identify a particular entry of a multi-dimensional array. For ease of visualisation and compact notation, we consider a third order tensor $\mathbf{\underline{X}} \in \mathbb{R}^{I \times J \times K}$. Thus, an element of such a tensor, $\mathbf{\underline{X}} \in \mathbb{R}^{I \times J \times K}$, can be written in a general form as:

$$ x_{ijk} = \mathbf{\underline{X}}[i, j, k]$$

> **Note:** In order to be consistent with Python indexing, count of modes and elements within starts from zeros.

Creation of a tensor starts with a formation of a multidimensional array of data.

```python
# Create 3-d array of data
array_3d = np.arange(24).reshape((2, 3, 4))
# Create tensor
tensor = Tensor(array_3d)
# Result preview
print(tensor)
tensor.data

```

    This tensor is of order 3 and consists of 24 elements.
    Sizes and names of its modes are (2, 3, 4) and ['mode-0', 'mode-1', 'mode-2'] respectively.

    array([[[ 0,  1,  2,  3],
            [ 4,  5,  6,  7],
            [ 8,  9, 10, 11]],

           [[12, 13, 14, 15],
            [16, 17, 18, 19],
            [20, 21, 22, 23]]])

## **Assigment 1**

1. Create a tensor of order 5 (of any shape) and with all its values equal to 5. Then, calculate its Frobenious norm.
2. Construct several tensors of orders varying from 3 to 20, with the size of each dimension equal to 2. Plot the relation between an order of a tensor and the number of its elements.
3. Construct several tensors of order 3 with the size of each dimension varying from 3 to 15. Plot the relation between the size of dimensions of a tensor and the number of its elements.
4. Analyse and compare the so obtained graphs.

### Solution: Part 1

```python
# Create a tensor
array_5d=5*np.ones(48).reshape((2, 2 , 3, 2 ,2))
tensor1 = Tensor(array_5d)
print(tensor1)
```

    This tensor is of order 5 and consists of 48 elements.
    Sizes and names of its modes are (2, 2, 3, 2, 2) and ['mode-0', 'mode-1', 'mode-2', 'mode-3', 'mode-4'] respectively.

```python
# Calculate Frobenious norm
tensor1.frob_norm
```

    34.64101615137755

### Solution: Part 2

```python
# Create tensors and collect information
num=[]
for order in range(3,21):
    shape=[]
    for i in range(order):
            shape.append(2)
    array=np.ones(2**order).reshape(shape)
    tensor2=Tensor(array)
    num.append(tensor2.size)
```

```python
# Plot results
plt.plot(range(3,21),num)
plt.title('Elements of varying Order')
plt.xlabel('order')
plt.ylabel('Number')
```

### Solution: Part 3

```python
# Create tensors and collect information
num=[]
for dim in range(3,16):
    array=np.zeros(dim**3).reshape(dim,dim,dim)
    tensor3=Tensor(array)
    num.append(tensor3.size)
```

```python
# Plot results
plt.plot(range(3,16),num)
plt.title('Elements of varying dimensions')
plt.xlabel('dimension')
plt.ylabel('Number')
```

### Solution: Part 4

When analysing of large size data, using tensor is one way to reduce the computatinal cost. In addition, tre number of elements are related with the order and dimension of each order. Part 2 verifies that the relationship between order and number of elements presents in exponential fuction with base 2, in mathematical expression $y=2^x$. As shown in figure, the number of elements is approximately over $10^6$ when order is 20 even if the dimension is two. As to the effect of dimension, the elements' number increases in cube function $y=x^3$ due to the fixed three dimension. Compared these two plots, the order of tensor will have a decisive role of the number of elements.

# Most common tensor substructures

1. A **fiber** is a vector obtained by fixing all but one of the indices, e.g.  $\mathbf{\underline{X}}[i,:,k]$ is the mode-2 fiber $\mathbf{\underline{X}}$ (usually refered to as row fiber).

1. Fixing all but two of the indices yields a matrix called a **slice** of a tensor, e.g. $\mathbf{\underline{X}}[:,:,k]$ is the mode-[1,2] slice (usually refered to as the frontal slice).

**Note:** The same principles and definitions can be applied to a tensor of an arbitrarily large order. In addition, you can obtain a **subtensor** by fixing at least three indices and letting the remaining ones vary.

```python
# Mode-2 fiber
tensor.data[0,1,:]
```

    array([4, 5, 6, 7])

```python
# Frontal slice
tensor.data[0,:,:]
```

    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])

## **Assigment 2**

1. Create a tensor of order 4 which consists of 81 elements and display all its horizontal slices and any five row fibers.
2. Create a tensor of order 3 and shape `(4,2,5)`, where the values of each consequent frontal slice are doubled.

### Solution: Part 1

```python
# Create a tensor of order 4
array_4d=np.arange(81).reshape(3,3,3,3)
tensor4=Tensor(array_4d)
tensor4.data
```

    array([[[[ 0,  1,  2],
             [ 3,  4,  5],
             [ 6,  7,  8]],

            [[ 9, 10, 11],
             [12, 13, 14],
             [15, 16, 17]],

            [[18, 19, 20],
             [21, 22, 23],
             [24, 25, 26]]],

           [[[27, 28, 29],
             [30, 31, 32],
             [33, 34, 35]],

            [[36, 37, 38],
             [39, 40, 41],
             [42, 43, 44]],

            [[45, 46, 47],
             [48, 49, 50],
             [51, 52, 53]]],

           [[[54, 55, 56],
             [57, 58, 59],
             [60, 61, 62]],

            [[63, 64, 65],
             [66, 67, 68],
             [69, 70, 71]],

            [[72, 73, 74],
             [75, 76, 77],
             [78, 79, 80]]]])

```python
# Display slices
for i in range(3):
    for j in range(3):
        print(tensor4.data[i,:,j,:])
```

    [[ 0  1  2]
     [ 9 10 11]
     [18 19 20]]
    [[ 3  4  5]
     [12 13 14]
     [21 22 23]]
    [[ 6  7  8]
     [15 16 17]
     [24 25 26]]
    [[27 28 29]
     [36 37 38]
     [45 46 47]]
    [[30 31 32]
     [39 40 41]
     [48 49 50]]
    [[33 34 35]
     [42 43 44]
     [51 52 53]]
    [[54 55 56]
     [63 64 65]
     [72 73 74]]
    [[57 58 59]
     [66 67 68]
     [75 76 77]]
    [[60 61 62]
     [69 70 71]
     [78 79 80]]

```python
# Display fibers
for i in range (3):
    print(tensor4.data[0,0,i,:])
for i in range (3):
    print(tensor4.data[1,i,1,:])
```

    [0 1 2]
    [3 4 5]
    [6 7 8]
    [30 31 32]
    [39 40 41]
    [48 49 50]

### Solution: Part 2

```python
# Create a tensor of order 3
array=np.arange(40).reshape(4,2,5)
for i in range(1,4):
    array[i,:,:]=2*array[i-1,:,:]
tens=Tensor(array)
for i in range(4):
    print(tens.data[i,:,:])
```

    [[0 1 2 3 4]
     [5 6 7 8 9]]
    [[ 0  2  4  6  8]
     [10 12 14 16 18]]
    [[ 0  4  8 12 16]
     [20 24 28 32 36]]
    [[ 0  8 16 24 32]
     [40 48 56 64 72]]

# Unfolding

Conventionally, unfolding is considered to be a process of element mapping from a tensor to a matrix. In other words, it arranges the mode-$n$ fibers of a tensor to be the matrix columns. Thus, the mode-$n$ unfolding is denoted as:

$$\mathbf{\underline{A}} \xrightarrow{n} \mathbf{A}_{(n)}$$

Thus, this operations requires to specify a mode along which a tensor will be unfolded. For a third order tensor, a visual representation of such operation is as following

In `hottbox` this functionality is available through the corresponding methods of the `Tensor` object, for instance:

```python
# Create 3-d array of data
array_3d = np.arange(24).reshape((2, 3, 4))

# Create tensor
tensor = Tensor(array_3d)

# Unfolding of a tensor
tensor.unfold(mode=0)

# Result preview
print(tensor)
tensor.data
```

    This tensor is of order 2 and consists of 24 elements.
    Sizes and names of its modes are (2, 12) and ['mode-0', 'mode-1_mode-2'] respectively.

    array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11],
           [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]])

## **Assigment 3**

1. Create a tensor, $\mathbf{\underline{A}}$, of order 4, in such a way that its version unfolded along the second mode would have sizes `(5,72)`, that is $\mathbf{A}_{(n)} \in  \mathbb{R}^{5 \times 72}$. Validate your results by displaying the unfolded tensor.

### Solution : Part 1

```python
# Create a tensor
array=np.arange(5*72).reshape(4,2,9,5)
A=Tensor(array)
```

```python
# Display unfolded version
A.unfold(mode=3)
print(A)
A.data
```

    This tensor is of order 2 and consists of 360 elements.
    Sizes and names of its modes are (5, 72) and ['mode-3', 'mode-0_mode-1_mode-2'] respectively.

    array([[  0,   5,  10,  15,  20,  25,  30,  35,  40,  45,  50,  55,  60,
             65,  70,  75,  80,  85,  90,  95, 100, 105, 110, 115, 120, 125,
            130, 135, 140, 145, 150, 155, 160, 165, 170, 175, 180, 185, 190,
            195, 200, 205, 210, 215, 220, 225, 230, 235, 240, 245, 250, 255,
            260, 265, 270, 275, 280, 285, 290, 295, 300, 305, 310, 315, 320,
            325, 330, 335, 340, 345, 350, 355],
           [  1,   6,  11,  16,  21,  26,  31,  36,  41,  46,  51,  56,  61,
             66,  71,  76,  81,  86,  91,  96, 101, 106, 111, 116, 121, 126,
            131, 136, 141, 146, 151, 156, 161, 166, 171, 176, 181, 186, 191,
            196, 201, 206, 211, 216, 221, 226, 231, 236, 241, 246, 251, 256,
            261, 266, 271, 276, 281, 286, 291, 296, 301, 306, 311, 316, 321,
            326, 331, 336, 341, 346, 351, 356],
           [  2,   7,  12,  17,  22,  27,  32,  37,  42,  47,  52,  57,  62,
             67,  72,  77,  82,  87,  92,  97, 102, 107, 112, 117, 122, 127,
            132, 137, 142, 147, 152, 157, 162, 167, 172, 177, 182, 187, 192,
            197, 202, 207, 212, 217, 222, 227, 232, 237, 242, 247, 252, 257,
            262, 267, 272, 277, 282, 287, 292, 297, 302, 307, 312, 317, 322,
            327, 332, 337, 342, 347, 352, 357],
           [  3,   8,  13,  18,  23,  28,  33,  38,  43,  48,  53,  58,  63,
             68,  73,  78,  83,  88,  93,  98, 103, 108, 113, 118, 123, 128,
            133, 138, 143, 148, 153, 158, 163, 168, 173, 178, 183, 188, 193,
            198, 203, 208, 213, 218, 223, 228, 233, 238, 243, 248, 253, 258,
            263, 268, 273, 278, 283, 288, 293, 298, 303, 308, 313, 318, 323,
            328, 333, 338, 343, 348, 353, 358],
           [  4,   9,  14,  19,  24,  29,  34,  39,  44,  49,  54,  59,  64,
             69,  74,  79,  84,  89,  94,  99, 104, 109, 114, 119, 124, 129,
            134, 139, 144, 149, 154, 159, 164, 169, 174, 179, 184, 189, 194,
            199, 204, 209, 214, 219, 224, 229, 234, 239, 244, 249, 254, 259,
            264, 269, 274, 279, 284, 289, 294, 299, 304, 309, 314, 319, 324,
            329, 334, 339, 344, 349, 354, 359]])

# Mode-n product

The mode-$n$ product is a multiplication of a tensor  by a matrix along the $n^{th}$ mode of a tensor. This essentially means that each mode-$n$ fiber should be multiplied by this matrix. Mathematically, this can be expressed as:

$$\mathbf{\underline{X}} \times_n \mathbf{A} = \mathbf{\underline{Y}} \quad \Leftrightarrow  \quad \mathbf{Y}_{(n)} = \mathbf{A} \mathbf{X}_{(n)}  $$

```python
I, J, K = 2, 3, 4
J_new = 5

# Create tensor
array_3d = np.arange(I * J * K).reshape(I, J ,K)
X = Tensor(array_3d)

# Create matrix
A = np.arange(J_new * J).reshape(J_new, J)

# Performs Mode-n product
Y = X.mode_n_product(A, mode=1, inplace=False)

# Preview of resulting tensor
print(Y)

# Preview of its unfolded version
Y.unfold(mode=1, inplace=False).data
```

    This tensor is of order 3 and consists of 40 elements.
    Sizes and names of its modes are (2, 5, 4) and ['mode-0', 'mode-1', 'mode-2'] respectively.

    array([[ 20,  23,  26,  29,  56,  59,  62,  65],
           [ 56,  68,  80,  92, 200, 212, 224, 236],
           [ 92, 113, 134, 155, 344, 365, 386, 407],
           [128, 158, 188, 218, 488, 518, 548, 578],
           [164, 203, 242, 281, 632, 671, 710, 749]])

## **Assignment 4**

1. Perform the mode-$n$ product of a 3-rd order tensor, $\mathbf{\underline{X}}$, with a matrix, $\mathbf{A} \in  \mathbb{R}^{5 \times 4}$, provided that the resulting tensor, $\mathbf{\underline{Y}}$, consists of `100` elements. Validate your results.

> **Note:** You are encouraged to experiment with the optional parameter `inplace` and to investigate its effect on the result.

### Solution: Part 1

```python
# Create original tensor
I,J,K=6,4,5
array = np.ones(I * J * K).reshape(I, J ,K)
X = Tensor(array)
X.data
```

    array([[[1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.]],

           [[1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.]],

           [[1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.]],

           [[1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.]],

           [[1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.]],

           [[1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.],
            [1., 1., 1., 1., 1.]]])

```python
# Create matrix
I_new=5
A = np.arange(I_new * I).reshape(I_new, I)
print(A)
```

    [[ 0  1  2  3  4  5]
     [ 6  7  8  9 10 11]
     [12 13 14 15 16 17]
     [18 19 20 21 22 23]
     [24 25 26 27 28 29]]

```python
# Perform mode-n product
Y = X.mode_n_product(A, mode=0, inplace=False)
```

```python
# Validate equivalence between the left and right hand side
print('Tensor Y')
print(Y)
# use inplace=True will moidfy itself and size (4,4,5) to (5,4,5)
print('Tensor X without inplace')
print(X)
X.mode_n_product(A, mode=0, inplace=True)
print('Tensor X with inplace')
print(X)
Y.data
```

    Tensor Y
    This tensor is of order 3 and consists of 100 elements.
    Sizes and names of its modes are (5, 4, 5) and ['mode-0', 'mode-1', 'mode-2'] respectively.
    Tensor X without inplace
    This tensor is of order 3 and consists of 120 elements.
    Sizes and names of its modes are (6, 4, 5) and ['mode-0', 'mode-1', 'mode-2'] respectively.
    Tensor X with inplace
    This tensor is of order 3 and consists of 100 elements.
    Sizes and names of its modes are (5, 4, 5) and ['mode-0', 'mode-1', 'mode-2'] respectively.

    array([[[ 15.,  15.,  15.,  15.,  15.],
            [ 15.,  15.,  15.,  15.,  15.],
            [ 15.,  15.,  15.,  15.,  15.],
            [ 15.,  15.,  15.,  15.,  15.]],

           [[ 51.,  51.,  51.,  51.,  51.],
            [ 51.,  51.,  51.,  51.,  51.],
            [ 51.,  51.,  51.,  51.,  51.],
            [ 51.,  51.,  51.,  51.,  51.]],

           [[ 87.,  87.,  87.,  87.,  87.],
            [ 87.,  87.,  87.,  87.,  87.],
            [ 87.,  87.,  87.,  87.,  87.],
            [ 87.,  87.,  87.,  87.,  87.]],

           [[123., 123., 123., 123., 123.],
            [123., 123., 123., 123., 123.],
            [123., 123., 123., 123., 123.],
            [123., 123., 123., 123., 123.]],

           [[159., 159., 159., 159., 159.],
            [159., 159., 159., 159., 159.],
            [159., 159., 159., 159., 159.],
            [159., 159., 159., 159., 159.]]])

# Properties of mode-n product

1. For distinct modes in a series of multiplications, the order of the multiplication is :

    $$\mathbf{\underline{X}} \times_n \mathbf{A} \times_m \mathbf{B} = \mathbf{\underline{X}} \times_m \mathbf{B} \times_n \mathbf{A} \quad (m \neq n)$$

1. However, this does not hold if the modes are the same :

    $$\mathbf{\underline{X}} \times_n \mathbf{A} \times_n \mathbf{B} = \mathbf{\underline{X}} \times_n (\mathbf{B}\mathbf{A})$$

## **Assignment 5**

1. Validate both of the above properties empirically
2. Prove the second property analytically

> **Note:** Be aware of consequences of the optional parameter `inplace` (its default value is set to `True`).

### Solution: Part 1

```python
# Create tensor
array=np.ones(2*4*6).reshape(2,4,6)
X=Tensor(array)
```

```python
# Create matrices
A1=np.arange(3*2).reshape(3,2)
B1=np.arange(2*4).reshape(2,4)
B2=np.arange(4*3).reshape(4,3)
```

```python
# Validate property 1
left=X.mode_n_product(A1, mode=0, inplace=False)
left.mode_n_product(B1, mode=1)
print(left)
print(left.data)
right=X.mode_n_product(B1, mode=1, inplace=False)
right.mode_n_product(A1, mode=0)
print(right)
print(right.data)
if left.data.all()==right.data.all():
    print('Left is equal to Right')
```

    This tensor is of order 3 and consists of 36 elements.
    Sizes and names of its modes are (3, 2, 6) and ['mode-0', 'mode-1', 'mode-2'] respectively.
    [[[  6.   6.   6.   6.   6.   6.]
      [ 22.  22.  22.  22.  22.  22.]]

     [[ 30.  30.  30.  30.  30.  30.]
      [110. 110. 110. 110. 110. 110.]]

     [[ 54.  54.  54.  54.  54.  54.]
      [198. 198. 198. 198. 198. 198.]]]
    This tensor is of order 3 and consists of 36 elements.
    Sizes and names of its modes are (3, 2, 6) and ['mode-0', 'mode-1', 'mode-2'] respectively.
    [[[  6.   6.   6.   6.   6.   6.]
      [ 22.  22.  22.  22.  22.  22.]]

     [[ 30.  30.  30.  30.  30.  30.]
      [110. 110. 110. 110. 110. 110.]]

     [[ 54.  54.  54.  54.  54.  54.]
      [198. 198. 198. 198. 198. 198.]]]
    Left is equal to Right

```python
# Validate property 2
left=X.mode_n_product(A1, mode=0, inplace=False)
left.mode_n_product(B2, mode=0)
print(left)
print(left.data)
right=X.mode_n_product(B2@A1, mode=0, inplace=False)
print(right)
print(right.data)
if left.data.all()==right.data.all():
    print('Left is equal to Right')
```

    This tensor is of order 3 and consists of 96 elements.
    Sizes and names of its modes are (4, 4, 6) and ['mode-0', 'mode-1', 'mode-2'] respectively.
    [[[ 23.  23.  23.  23.  23.  23.]
      [ 23.  23.  23.  23.  23.  23.]
      [ 23.  23.  23.  23.  23.  23.]
      [ 23.  23.  23.  23.  23.  23.]]

     [[ 68.  68.  68.  68.  68.  68.]
      [ 68.  68.  68.  68.  68.  68.]
      [ 68.  68.  68.  68.  68.  68.]
      [ 68.  68.  68.  68.  68.  68.]]

     [[113. 113. 113. 113. 113. 113.]
      [113. 113. 113. 113. 113. 113.]
      [113. 113. 113. 113. 113. 113.]
      [113. 113. 113. 113. 113. 113.]]

     [[158. 158. 158. 158. 158. 158.]
      [158. 158. 158. 158. 158. 158.]
      [158. 158. 158. 158. 158. 158.]
      [158. 158. 158. 158. 158. 158.]]]
    This tensor is of order 3 and consists of 96 elements.
    Sizes and names of its modes are (4, 4, 6) and ['mode-0', 'mode-1', 'mode-2'] respectively.
    [[[ 23.  23.  23.  23.  23.  23.]
      [ 23.  23.  23.  23.  23.  23.]
      [ 23.  23.  23.  23.  23.  23.]
      [ 23.  23.  23.  23.  23.  23.]]

     [[ 68.  68.  68.  68.  68.  68.]
      [ 68.  68.  68.  68.  68.  68.]
      [ 68.  68.  68.  68.  68.  68.]
      [ 68.  68.  68.  68.  68.  68.]]

     [[113. 113. 113. 113. 113. 113.]
      [113. 113. 113. 113. 113. 113.]
      [113. 113. 113. 113. 113. 113.]
      [113. 113. 113. 113. 113. 113.]]

     [[158. 158. 158. 158. 158. 158.]
      [158. 158. 158. 158. 158. 158.]
      [158. 158. 158. 158. 158. 158.]
      [158. 158. 158. 158. 158. 158.]]]
    Left is equal to Right

### Solution: Part 2

Use [LaTeX syntax](https://stackoverflow.com/questions/13208286/how-to-write-latex-in-ipython-notebook) for analytical deriviations for the second property below.

Based on the relationship in Assignment 4:
$$\mathbf{\underline{X}} \times_n \mathbf{A} = \mathbf{\underline{Y}} \quad \Leftrightarrow  \quad \mathbf{Y}_{(n)} = \mathbf{A} \mathbf{X}_{(n)}  $$
The first two term on the left side can be expressed as:
$$
\mathbf{\underline{X}} \times_n \mathbf{A}= \mathbf{A}\times \mathbf{X}_{(n)}=\mathbf Y_{(n)}\Rightarrow \mathbf{\underline{Y}}
$$
Then, make product with $_n\mathbf {B}$:
$$
\begin{align}
\mathbf{\underline{Y}}\times_n\mathbf{B}
&=\mathbf{B}\times \mathbf{Y}_{(n)}\\
&=\mathbf{B}\times\mathbf{A}\times \mathbf{X}_{(n)}\\
&=(\mathbf{B}\mathbf{A})\times \mathbf{X}_{(n)}\\
\end{align}
$$
Thus, for the second property, the left side can be expressed as:
$$
(\mathbf{B}\mathbf{A})\times \mathbf{X}_{(n)}=\mathbf{\underline X}\times _n(\mathbf{BA})\\
\Rightarrow \mathbf{\underline{X}} \times_n \mathbf{A} \times_n \mathbf{B} = \mathbf{\underline{X}} \times_n (\mathbf{B}\mathbf{A})
$$