# 4. Gyakorlat - Vasúti ütközőbak
2021.03.01

## Feladat:

```python
from IPython.display import Image
Image(filename='gyak4_1.png',width=900)
```

A mellékelt ábrán látható módon egy $m$ tömegű vasúti szerlvény egy ütközőbakba csapódik $v_0$ kezdősebességgel. Feltételezzük, hogy a folyamat során a bak mozdulatlan marad. Modellezvén a bak elaszticitását és energia disszipációját, a fenti ábrán látható mechanikai modellt használjuk. Miközben a szerelvény ütközője érintezik a bak ütközőjével, kettejük rugómerevségét és csillapítási tényezőjét kombinálhatjuk egy $k$ eredő rugómerevséggé, valamint egy $2c$ nagyságú eredő csillapítási tényezővé.

### Adatok:

|                       |                       |
|:-----------------------|-----------------------|
| $m$ = 5$\cdot$10$^4$ kg | $v_0$ = 1 m/s         |
| $k$ = 10$^6$ N/m | $c$ = 10$^5$ Ns/m |

### Részfeladatok:
1. Számítsa ki az ütközés során a rugóban ébredő maximális erőt!
2. Határozza meg az ütközés során elnyelt energiát!

## Megoldás:

### 1. Feladat

További részletezés nélkül a mozgásegyenlet:

$$m\ddot{x} + 2c\dot{x} + kx = 0.$$

```python
import sympy as sp
from IPython.display import Math # hogy tudjunk LaTeX szöveget kiírni

sp.init_printing()

m,c,k,ζ,ω_n,ω_d,v_0,t = sp.symbols('m,c,k,ζ,ω_n,ω_d,v_0,t')
x = sp.Function('x')(t)

# Készítsünk behelyettesítési listát az adatok alapján, SI-ben
adatok = [(m, 5*10**4), (v_0, 1), (k, 10**6), (c, 10**5)]

mozgasegy = m*sp.diff(x,t,2) + 2*c*sp.diff(x,t,1) + k*x

# Osszunk le a főegyütthatóval:
foegyutthato = mozgasegy.coeff(sp.diff(x,t,2))

mozgasegy = (mozgasegy/foegyutthato).expand()

mozgasegy
```

```python
mozgasegy = mozgasegy.subs([(2*c/m,2*ζ*ω_n),(k/m,ω_n**2)])
mozgasegy
```

```python
ω_n_num = sp.sqrt(k/m).subs(adatok)

display(Math('\omega_n = {}'.format(sp.latex(ω_n_num))))

# rad/s
```

$\displaystyle \omega_n = 2 \sqrt{5}$

```python
ζ_num = (c/m/ω_n).subs(adatok).subs(ω_n,ω_n_num)

display(Math('\zeta = {}'.format(sp.latex(ζ_num))))

# 1
```

$\displaystyle \zeta = \frac{\sqrt{5}}{5}$

```python
ω_d_num = (ω_n*sp.sqrt(1-ζ**2)).subs(adatok).subs(ω_n,ω_n_num).subs(ζ,ζ_num)

display(Math('\omega_d = {}'.format(sp.latex(ω_d_num))))

# rad/s
```

$\displaystyle \omega_d = 4$

```python
T_d_num = ((2*sp.pi)/ω_d).subs(adatok).subs(ω_d,ω_d_num)

display(Math('T_d = {}'.format(sp.latex(T_d_num))))

# s
```

$\displaystyle T_d = \frac{\pi}{2}$

```python
mozgasegy_num = mozgasegy.subs([(ζ,ζ_num),(ω_n,ω_n_num)])
mozgasegy_num # természetesen ez a bal oldal, amennyiben a jobb oldal 0
```

```python
# Oldjuk meg a mozgásegyenletet előszőr
# az integrálási konstansok kézi meghatározásával.

# általános megoldás:
alt_meg = sp.dsolve(mozgasegy_num,x).rhs  # right hand side
alt_meg  # figyelem, itt az eredeti kidolgozásban pont fordítva van C1 és C2!
```

```python
# Általános megoldás deriváltja:
d_alt_meg = sp.diff(alt_meg,t)
d_alt_meg
```

```python
# Kezedit értékek: x(0) = 0, v(0) = v0.
# C1, C2 konstansok kifejezése:
v0, C1, C2 = sp.symbols("v0, C1, C2")

# Oldjuk meg az egyenletrendszert C1 és C2-re.
konst = sp.solve([alt_meg.subs(t,0),d_alt_meg.subs(t,0)-v0],C1, C2)
C1_num = konst[C1]
C2_num = konst[C2]

display(Math('C_1 = {},'.format(sp.latex(konst[C1]))))
display(Math('C_2 = {}.'.format(sp.latex(konst[C2]))))
```

$\displaystyle C_1 = \frac{v_{0}}{4},$

$\displaystyle C_2 = 0.$

```python
alt_meg.subs([(C1,C1_num),(C2,C2_num)])
```

```python
# Most pedig értékeljük, hogy a sympy a fentieket
# automatikusan is képes elvégezni:

kezdeti_ert = {x.subs(t,0): 0, x.diff(t).subs(t,0): v_0}
display(kezdeti_ert)

mozg_torv = sp.dsolve(mozgasegy_num,x,ics=kezdeti_ert)
mozg_torv
```

Maximális rugóerő a maximális kitérésnél: keressük a maximális kitéréshez tartozó $t^*$ időt:

```python
# A maximumban a derivált értéke zérus.
# Itt már felhasználhatjuk azt, hogy C2 = 0
d_alt_meg.subs(C2,0)
```

```python
# Megoldva az egyenletet t-re:
meg = sp.solve(d_alt_meg.subs(C2,0),t)
t_csillag = meg[0]

display(Math('t^* = {}'.format(t_csillag.evalf(4))))

# s
```

$\displaystyle t^* = 0.2768$

```python
# Visszahelyettesítve a mozgástörvénybe:
x_max = mozg_torv.rhs.subs(t,t_csillag).subs(v_0,1)

display(Math('x_{{max}} = {}'.format(x_max.evalf(4))))

# m
```

$\displaystyle x_{max} = 0.1285$

```python
# Amiből a maximális rugóerő:
F_max = (k*x_max).subs(adatok)

display(Math('F_{{r,max}} = {}'.format(F_max.evalf(5)/1000)))

# kN
```

$\displaystyle F_{r,max} = 128.55$

## Második feladat

```python
# Newton II-ből a mozdony által az ütközőre ható erő
kontakt_egy = sp.Eq(k*alt_meg+2*c*d_alt_meg,0)
kontakt_egy = kontakt_egy.subs(C2,0).simplify()
kontakt_egy
```

```python
# Oldjuk meg t-re, felhasználva, hogy C2 = 0 és C1 ≠ 0:
kontakt_megold = sp.solve(kontakt_egy.subs(adatok),t)
display(kontakt_megold)

# a 2 megoldás közül a legkisebb kell, ami még pozitív.
# Módszer: `ternary operator`, ami könnyen olvasható és értelmezhető.
t_cscs = kontakt_megold[0] if 0 < kontakt_megold[0] < kontakt_megold[1] else kontakt_megold[1]

display(Math('t^{{**}} = {}'.format(t_cscs.evalf(4))))

# s
```

```python
# A mozdony sebessége az elválás pillanatában (t** időpontban)

v_t = mozg_torv.rhs.diff(t) # mozgástörvény deriváltja -> sebesség
v_tcscs = v_t.subs(t,t_cscs).subs(adatok).evalf(4)

display(Math('v_{{t^{{**}}}} = {}'.format(v_tcscs.evalf(4))))

# m/s
```

$\displaystyle v_{t^{**}} = -0.3305$

```python
# A kinetikus energia megváltozása a munkavégzéssel egyenlő:
W_diss = (1/2*m*v_tcscs**2-1/2*m*v_0**2).subs(adatok)

# A disszipált energia ennek mínusz egyszerese
display(Math('E^\mathrm{{diss}} = {}'.format(-W_diss.evalf(4)/1000)))

# kJ
```

$\displaystyle E^\mathrm{diss} = 22.27$

Készítette:

            Juhos-Kiss Álmos (Alkalmazott Mechanika Szakosztály) és
            Csuzdi Domonkos (Alkalmazott Mechanika Szakosztály)
            Takács Dénes (BME MM) ábrái és Vörös Illés (BME MM) kidolgozása alapján.

            Hibák, javaslatok:
            amsz.bme@gmail.com
            csuzdi02@gmail.com
            almosjuhoskiss@gmail.com

            2021.03.01


```python
import numpy as np
import sympy as sy
import matplotlib.pyplot as plt
import control.matlab as cm
%matplotlib inline
```

```python
np.log(2)/2.0
```

    0.34657359027997264

```python
Hc = cm.tf([1],[1, 0], np.log(2)/2.0)
Hc
```

    1
    -
    z

    dt = 0.34657359028

```python
Td = np.log(2)/2*np.arange(20)
(yd,td) = cm.step(Hc, Td)
plt.plot(td,yd[0], '*')
plt.xlim([0,8])
plt.ylim([0,1.4])
plt.savefig("step-dead-beat.pdf")
```

```python
b = (2-np.sqrt(2))/2.0
a1 = -(3*np.sqrt(2)-2)/2.0
a2 = (2*np.sqrt(2)-2)/2.0
Hc2 = cm.tf([2*b, -b], [1, a1, a2], np.log(2)/4.0)
Td2 = np.log(2)/4*np.arange(40)
(yd2,td2) = cm.step(Hc2, Td2)
plt.plot(td,yd[0], '*')
plt.plot(td2,yd2[0], 'r*')
plt.xlim([-0.2,8])
plt.ylim([-0.2,1.4])
plt.savefig("step-half-sampling-time.pdf")
```

```python
cm.feedback?
```

```python
H = cm.tf([0.5],[1, -0.5], np.log(2)/2.0)
F = 1.05*cm.tf([2, -1],[1, -1], np.log(2)/2.0)
FH = F*H
Hcl = cm.feedback(FH)
print Hcl
Hcu = cm.feedback(F, H)
print Hcu
```

       1.05 z - 0.525
    --------------------
    z^2 - 0.45 z - 0.025

    dt = 0.34657359028

    2.1 z^2 - 2.1 z + 0.525
    -----------------------
      z^2 - 0.45 z - 0.025

    dt = 0.34657359028

```python
Td = np.log(2)/2*np.arange(20)
(yd,td) = cm.step(Hcl, Td)
(ud,td) = cm.step(Hcu, Td)
plt.plot(td,yd[0], '*')
plt.plot(td,ud[0], 'r*')
plt.xlim([-0.2,8])
plt.ylim([-0.2,3.1])
plt.savefig("step-dead-beat.pdf")
```

```python
ud
```

    array([[ 2.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,
             1.,  1.,  1.,  1.,  1.,  1.,  1.]])

```python

```


```
import numpy as np
import pandas as pd
import scipy as sp
import matplotlib.pyplot as plt

import plotly
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, plot, iplot

# for colab notebooks:
def enable_plotly_in_cell():
    import IPython
    from plotly.offline import init_notebook_mode
    display(IPython.core.display.HTML(''''''))
    init_notebook_mode(connected=False)

# import the PCA module from sklearn
from sklearn.decomposition import PCA
```

# Vertical Line Test

## 1.1 Create two graphs, one that passes the vertical line test and one that does not.

```
# Graph that doesn't pass the vertical line test:
theta = np.arange(0,8*np.pi,1.4)
x = np.cos(theta)*np.exp(0.4*theta)
y = np.sin(theta)*np.exp(0.4*theta)
plt.plot(x,y)
plt.axvline(x=0,color='red')
plt.show()
```

```
# Graph that passes the vertical line test:
x = np.arange(-10,11)
y = x**25+2*x
plt.plot(x,y)
plt.axvline(x=0,color='r');
```

## 1.2 Why are graphs that don't pass the vertical line test not considered "functions?"

**Answer:**
>The graphs that don't pass the vertical line test are not considered 'functions', because it a function should only have one `y` for its given `x`.

# Functions as Relations

## 2.1 Which of the following relations are functions? Why?

\begin{align}
\text{Relation 1: } \{(1, 2), (3, 2), (1, 3)\}
\\
\text{Relation 2: } \{(1, 3), (2, 3), (6, 7)\}
\\
\text{Relation 3: } \{(9, 4), (2, 1), (9, 6)\}
\\
\text{Relation 4: } \{(6, 2), (8, 3), (6, 4)\}
\\
\text{Relation 5: } \{(2, 6), (2, 7), (2, 4)\}
\end{align}

**Answer:**
> Only *Relation 2* is a function, because it is the only one mapped to a single output.

# Functions as a mapping between dimensions

## 3.1 for the following functions what is the dimensionality of the domain (input) and codomain (range/output)?

\begin{align}
m(𝑥_1,𝑥_2,𝑥_3)=(x_1+x_2, x_1+x_3, x_2+x_3)
\\
n(𝑥_1,𝑥_2,𝑥_3,𝑥_4)=(x_2^2 + x_3, x_2x_4)
\end{align}

**Answer:**
> for function `m` the domain has a dimensionality of 3 and the codomain has a dimensionality of 3.

> for function `n` the domain has a dimensionality of 4 and the codomain has a dimensionality of 2.

## 3.2 Do you think it's possible to create a function that maps from a lower dimensional space to a higher dimensional space? If so, provide an example.

You can have a linear map from a low-dimensional space to a high-dimensional space. However, such a map will "miss" most of the target space.
>

# Vector Transformations

## 4.1 Plug the corresponding unit vectors into each function. Use the output vectors to create a transformation matrix.

\begin{align}
p(\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}
\\
\\
q(\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}
\end{align}

```
def p(vect):
    x1 = vect[0][0]
    x2 = vect[1][0]
    row1 = np.array([x1+3*x2])
    row2 = np.array([2*x2+x1])
    output_vector = np.array([row1,row2])
    return output_vector

i1 = np.array([[1],[0]])
j1 = np.array([[0],[1]])
i1 = p(i1)
j1 = p(j1)
T = np.column_stack([i,j])

def q(vect):
    x1 = vect[0][0]
    x2 = vect[1][0]
    x3 = vect[2][0]
    row1 = np.array([4*x1+x2+2*x3])
    row2 = np.array([2*x2-x1+3*x3])
    row3 = np.array([5*x1-2*x3+x2])
    output_vector = np.array([row1,row2,row3])
    return output_vector

i2 = np.array([[1],[0],[0]])
j2 = np.array([[0],[1],[0]])
k2 = np.array([[0],[0],[1]])
T2 = np.column_stack([q(i2),q(j2),q(k2)])
print(T,'\n')
print(T2)
```

    [[1 3]
     [1 2]]

    [[ 4  1  2]
     [-1  2  3]
     [ 5  1 -2]]

## 4.2 Verify that your transformation matrices are correct by choosing an input matrix and calculating the result both via the traditional functions above and also via vector-matrix multiplication.

```
input_vect = np.array([[3],[-2],[5]])
trans = q(input_vect)
ouput_vect = np.matmul(T2, input_vect)

print(trans)
print(ouput_vect)
```

    [[20]
     [ 8]
     [ 3]]
    [[20]
     [ 8]
     [ 3]]

# Eigenvalues and Eigenvectors

## 5.1 In your own words, give an explanation for the intuition behind eigenvalues and eigenvectors.

**Answer:**
> Eigenvectors are the vectors that when scaled stay on the same line and don't move even when our basis vectors move.

# The Curse of Dimensionality

## 6.1 What are some of the challenges of working with high dimensional spaces?

**Answer:**
> Can't visualize anythig past 3 dimensions, so it forces us to only work with higher dimensions strictly through numbers.

## 6.2 What is the rule of thumb for how many observations you should have compared to parameters in your model?

**Answer:**
> The general rule of thumb is that you have 5 times the number of observations as you have parameters in your model.

> If you have more parameters than observations you are in danger of overfitting the model.

# Principal Component Analysis

## 7.1 Code for loading and cleaning the 2013 national dataset from the [Housing Affordability Data System (HADS)](https://www.huduser.gov/portal/datasets/hads/hads.html) --housing data, can be found below.

## Perform PCA on the processed dataset `national_processed` (Make sure you standardize your data!) and then make a scatterplot of PC1 against PC2. Some of our discussion and work around PCA with this dataset will continue during tomorrow's lecture and assignment.

Not only does this dataset have decent amount columns to begin with (99), but in preparing the data for PCA we have also [one-hot-encoded](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f#targetText=One%20hot%20encoding%20is%20a,the%20entry%20in%20the%20dataset.) all of the categorical variables. This has the effect of creating a new column for each individual category of each categorical variable. After processing this dataset has 64738 columns. --Das a lot of columns.

Don't worry too much about the mechanics of one-hot encoding right now, you will learn and experiment with a whole bunch of categorical encoding approaches in unit 2.

The code below will read in the dataset and perform the one-hot encoding of the categorical variables. Start adding your PCA code at the bottom of the provided code.

```
from urllib.request import urlopen
from zipfile import ZipFile
from io import BytesIO
import os.path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Read Natinal Data
national_url = 'https://www.huduser.gov/portal/datasets/hads/hads2013n_ASCII.zip'
national_file = 'thads2013n.txt'

if os.path.exists(national_file):
    national = pd.read_csv(national_file)
else:
    z_national = urlopen(national_url)
    zip_national = ZipFile(BytesIO(z_national.read())).extract(national_file)
    national = pd.read_csv(zip_national)

print(national.shape)
national.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      '100003130103'
      82
      '3'
      '1'
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      '1'
      1
      40000
      -6
      '1'
      1
      6
      3117.394239
      1
      18021
      '1'
      533
      1
      '1'
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      '1 Adequate'
      '-5'
      '2000-2009'
      '1 Single Family'
      '2 2BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '1 LTE 50% FMR'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '2 30 - 50% AMI'
      '.'
      '2 30% to 50%'
      '-5'
      '-5'

      1
      '100006110249'
      50
      '5'
      '3'
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      '1'
      1
      130000
      -6
      '1'
      1
      6
      2150.725544
      4
      122961
      '1'
      487
      1
      '1'
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '3 GT FMR'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '7 120% AMI +'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      2
      '100006370140'
      53
      '5'
      '3'
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      '1'
      1
      150000
      -6
      '1'
      1
      7
      2213.789404
      2
      27974
      '1'
      1405
      1
      '1'
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '4 4BR+'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 150-200% Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '2 50.1 - 100% FMR'
      '4 60 - 80% AMI'
      '5 80 - 100% AMI'
      '6 100 - 120% AMI'
      '4 60 - 80% AMI'
      '4 60 - 80% AMI'
      '.'
      '3 50% or More'
      '-5'
      '-5'

      3
      '100006520140'
      67
      '5'
      '3'
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      '1'
      1
      200000
      -6
      '1'
      1
      6
      2364.585097
      2
      32220
      '1'
      279
      1
      '1'
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      '1 Adequate'
      '-5'
      '1980-1989'
      '1 Single Family'
      '3 3BR'
      '1 Owner'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '2 50.1 - 100% FMR'
      '6 100 - 120% AMI'
      '7 120% AMI +'
      '7 120% AMI +'
      '5 80 - 100% AMI'
      '4 60 - 80% AMI'
      '.'
      '1 Less than 30%'
      '-5'
      '-5'

      4
      '100007130148'
      26
      '1'
      '3'
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      '1'
      1
      -6
      -6
      '2'
      100
      4
      2314.524902
      2
      96874
      '1'
      759
      5
      '2'
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      '1 Adequate'
      'Central City'
      '1980-1989'
      '5 50+ units'
      '2 2BR'
      '2 Renter'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '3 150-200% Poverty'
      '4 200%+ Poverty'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 GT FMR'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '3 50 - 60% AMI'
      '7 120% AMI +'
      '0 Not Assisted'
      '1 Less than 30%'
      '-5'
      '-5'

5 rows × 99 columns

```
# Look at datatypes
# a lot of object datatypes even though they seem to be strings of numbers.
national.dtypes
```

    CONTROL            object
    AGE1                int64
    METRO3             object
    REGION             object
    LMED                int64
                        ...
    FMTINCRELAMICAT    object
    FMTASSISTED        object
    FMTBURDEN          object
    FMTREGION          object
    FMTSTATUS          object
    Length: 99, dtype: object

```
# check for null values
national.isnull().sum().any()
```

    False

```
# check for number of categorical vs numeric columns
cat_cols = national.columns[national.dtypes=='object']
num_cols = national.columns[national.dtypes!='object']

print(f'{len(cat_cols)} categorical columns')
print(f'{len(num_cols)} numerical columns')
```

    32 categorical columns
    67 numerical columns

```
# We're making a copy of our data in case we mess something up.
national_processed = national.copy()

# Categorically Encode our Variables:
# They need to all be numeric before we do PCA.
# https://pbpython.com/categorical-encoding.html

# Cast categorical columns to "category" data type
national_processed[cat_cols] = national_processed[cat_cols].astype('category')

national_processed.dtypes
```

    CONTROL            category
    AGE1                  int64
    METRO3             category
    REGION             category
    LMED                  int64
                         ...
    FMTINCRELAMICAT    category
    FMTASSISTED        category
    FMTBURDEN          category
    FMTREGION          category
    FMTSTATUS          category
    Length: 99, dtype: object

```
# Replace all category cell values with their numeric category codes
for col in cat_cols:
  national_processed[col] = national_processed[col].cat.codes

print(national_processed.shape)
national_processed.head()
```

    (64535, 99)

      CONTROL
      AGE1
      METRO3
      REGION
      LMED
      FMR
      L30
      L50
      L80
      IPOV
      BEDRMS
      BUILT
      STATUS
      TYPE
      VALUE
      VACANCY
      TENURE
      NUNITS
      ROOMS
      WEIGHT
      PER
      ZINC2
      ZADEQ
      ZSMHC
      STRUCTURETYPE
      OWNRENT
      UTILITY
      OTHERCOST
      COST06
      COST12
      COST08
      COSTMED
      TOTSAL
      ASSISTED
      GLMED
      GL30
      GL50
      GL80
      APLMED
      ABL30
      ...
      COST08RELPOVCAT
      COST08RELFMRPCT
      COST08RELFMRCAT
      COST12RELAMIPCT
      COST12RELAMICAT
      COST12RELPOVPCT
      COST12RELPOVCAT
      COST12RELFMRPCT
      COST12RELFMRCAT
      COSTMedRELAMIPCT
      COSTMedRELAMICAT
      COSTMedRELPOVPCT
      COSTMedRELPOVCAT
      COSTMedRELFMRPCT
      COSTMedRELFMRCAT
      FMTZADEQ
      FMTMETRO3
      FMTBUILT
      FMTSTRUCTURETYPE
      FMTBEDRMS
      FMTOWNRENT
      FMTCOST06RELPOVCAT
      FMTCOST08RELPOVCAT
      FMTCOST12RELPOVCAT
      FMTCOSTMEDRELPOVCAT
      FMTINCRELPOVCAT
      FMTCOST06RELFMRCAT
      FMTCOST08RELFMRCAT
      FMTCOST12RELFMRCAT
      FMTCOSTMEDRELFMRCAT
      FMTINCRELFMRCAT
      FMTCOST06RELAMICAT
      FMTCOST08RELAMICAT
      FMTCOST12RELAMICAT
      FMTCOSTMEDRELAMICAT
      FMTINCRELAMICAT
      FMTASSISTED
      FMTBURDEN
      FMTREGION
      FMTSTATUS

      0
      0
      82
      2
      0
      73738
      956
      15738
      26213
      40322
      11067
      2
      2006
      0
      1
      40000
      -6
      1
      1
      6
      3117.394239
      1
      18021
      1
      533
      1
      0
      169.000000
      213.750000
      648.588189
      803.050535
      696.905247
      615.156712
      0
      -9
      73738
      15738
      26213
      40322
      51616.6
      20234.571429
      ...
      4
      72.898038
      2
      48.402635
      2
      290.250487
      4
      84.001102
      2
      37.077624
      2
      222.339102
      4
      64.346936
      2
      1
      0
      5
      1
      2
      0
      4
      4
      4
      4
      3
      1
      1
      1
      1
      1
      1
      1
      1
      1
      2
      0
      2
      0
      0

      1
      1
      50
      4
      2
      55846
      1100
      17165
      28604
      45744
      24218
      4
      1980
      0
      1
      130000
      -6
      1
      1
      6
      2150.725544
      4
      122961
      1
      487
      1
      0
      245.333333
      58.333333
      1167.640781
      1669.643405
      1324.671218
      1058.988479
      123000
      -9
      55846
      17165
      28604
      45744
      55846.0
      19911.400000
      ...
      4
      120.424656
      3
      103.094063
      6
      275.768999
      4
      151.785764
      3
      65.388468
      4
      174.909320
      3
      96.271680
      2
      1
      0
      3
      1
      4
      0
      3
      4
      4
      3
      4
      2
      2
      2
      1
      3
      3
      3
      5
      3
      7
      0
      1
      0
      0

      2
      2
      53
      4
      2
      55846
      1100
      13750
      22897
      36614
      15470
      4
      1985
      0
      1
      150000
      -6
      1
      1
      7
      2213.789404
      2
      27974
      1
      1405
      1
      0
      159.000000
      37.500000
      1193.393209
      1772.627006
      1374.582175
      1068.025168
      28000
      -9
      55846
      13750
      22897
      36614
      44676.8
      19937.500000
      ...
      4
      124.962016
      3
      109.452905
      6
      458.339239
      4
      161.147910
      3
      65.946449
      4
      276.153890
      4
      97.093197
      2
      1
      0
      3
      1
      4
      0
      4
      4
      4
      4
      3
      2
      2
      2
      1
      2
      3
      4
      5
      3
      4
      0
      3
      0
      0

      3
      3
      67
      4
      2
      55846
      949
      13750
      22897
      36614
      13964
      3
      1985
      0
      1
      200000
      -6
      1
      1
      6
      2364.585097
      2
      32220
      1
      279
      1
      0
      179.000000
      70.666667
      1578.857612
      2351.169341
      1820.442900
      1411.700224
      0
      -9
      55846
      13750
      22897
      36614
      44676.8
      17875.000000
      ...
      4
      191.827492
      3
      161.926709
      7
      673.494512
      4
      247.752301
      3
      97.224801
      5
      404.382763
      4
      148.756610
      3
      1
      0
      3
      1
      3
      0
      4
      4
      4
      4
      4
      2
      2
      2
      2
      2
      5
      6
      6
      4
      4
      0
      1
      0
      0

      4
      4
      26
      0
      2
      60991
      737
      14801
      24628
      39421
      15492
      2
      1980
      0
      1
      -6
      -6
      2
      100
      4
      2314.524902
      2
      96874
      1
      759
      5
      1
      146.000000
      12.500000
      759.000000
      759.000000
      759.000000
      759.000000
      96900
      0
      60991
      14801
      24628
      39421
      48792.8
      16651.125000
      ...
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      55.308707
      3
      195.972115
      3
      102.985075
      3
      1
      1
      3
      5
      2
      1
      3
      3
      3
      3
      4
      2
      2
      2
      2
      3
      2
      2
      2
      2
      7
      1
      1
      0
      0

5 rows × 99 columns

```
# Now we only ahve numeric columns (ints and floats)
national_processed.dtypes
```

    CONTROL            int32
    AGE1               int64
    METRO3              int8
    REGION              int8
    LMED               int64
                       ...
    FMTINCRELAMICAT     int8
    FMTASSISTED         int8
    FMTBURDEN           int8
    FMTREGION           int8
    FMTSTATUS           int8
    Length: 99, dtype: object

```
from sklearn.preprocessing import StandardScaler
feature_cols = national_processed.columns
x = national_processed.loc[:,feature_cols].values
x = StandardScaler().fit_transform(x)
normal_nat = pd.DataFrame(x,columns=feature_cols)
normal_nat
```

```
from sklearn.decomposition import PCA
pca_nat = PCA(n_components=2)
pca_data = pca_nat.fit_transform(x)
df_final = pd.DataFrame(data=pca_data,columns=['comp_1','comp_2'])
df_final
```

    (64535, 2)

```
pca_nat.explained_variance_ratio_
```

    array([0.34050542, 0.14442164])

```
sum(pca_nat.explained_variance_ratio_)
```

    0.4849270519748006

```
1-sum(pca_nat.explained_variance_ratio_)
```

    0.5150729480251994

```
df_final['COST06']=national_processed['COST06']
df_final.sample(5)
```

      comp_1
      comp_2
      CONTROL
      BEDRMS
      COST06

      31485
      0.819190
      -1.139287
      31485
      4
      1050.807448

      11919
      9.676782
      -0.440610
      11919
      3
      3636.977363

      12062
      8.673044
      0.121435
      12062
      5
      3172.177199

      28480
      -1.310413
      0.380853
      28480
      3
      924.095473

      39897
      -0.067866
      -1.178848
      39897
      2
      1143.000000

```
plt.figure(figsize=(10,10))
plt.xlabel('comp_1')
plt.ylabel('comp_2')
for k, c in [(0,'indigo'),(1,'gold')]:
    subset=df_final[df_final['COST06']==k]
    plt.scatter(x=subset['comp_1'],
                y=subset['comp_2'],
                s=200,
                alpha=.5,
                color=c,
                label=k);
plt.legend
```

```

```

```

```

# Stretch Goals

## 1) Perform further data exploration on the HADS national dataset (the version before we one-hot encoded it) Make scatterplots and see if you can see any resemblance between the original scatterplots and the plot of the principal components that you made in 7.1.

(You may or may not not see very much resemblance depending on the variables you choose, and that's ok!)

## 2) Study "Scree Plots" and then try and make one for your PCA dataset. How many principal conponents do you need to retain in order for your PCs to contain 90% of the explained variance?

We will present this topic formally at the beginning of tomorrow's lecture, so if you figure this stretch goal out, you're ahead of the game.

## 3) Explore further the intuition behind eigenvalues and eigenvectors by creating your very own eigenfaces:

Prioritize self-study over this stretch goal if you are not semi-comfortable with the topics of PCA, Eigenvalues, and Eigenvectors.

You don't necessarily have to use this resource, but this will get you started:
[Eigenface Tutorial](https://sandipanweb.wordpress.com/2018/01/06/eigenfaces-and-a-simple-face-detector-with-pca-svd-in-python/)


# Домашняя лабораторная работа №1 по вычислительной математике

### Державин Андрей, Б01-909 группа

```python
import numpy as np
from matplotlib import pyplot as plt
import sympy as sp
```

## Объявляем некоторые вспомогательные константы
 - символ х - для работы sympy
 - машинное $\varepsilon$
 - массив $h$

```python
# Some constants
x_s = sp.symbols('x')
eps = np.finfo(float).eps

def get_h(n):
  return 2 / 2 ** (n)

n_arr = np.arange(1, 21)
h_arr = get_h(n_arr)
```

## Выводим формулы для погрешностей методов

### Метод 1
$$
\frac{f(x + h) - f(x)}{h} \approx f'(x) + f''(x)\frac{h}{2} \Rightarrow
\Delta_m = f''(x)\frac{h}{2}
$$
Полная погрешность:
$$
\Delta = \boxed{\frac{2\varepsilon}{h} + f''(x)\frac{h}{2}}
$$

### Метод 2
$$
\frac{f(x + h) - f(x)}{h} \approx f'(x) + f''(x)\frac{h}{2} \Rightarrow
\Delta_m = f''(x)\frac{h}{2}
$$
Полная погрешность:
$$
\Delta = \boxed{\frac{2\varepsilon}{h} + f''(x)\frac{h}{2}}
$$

### Метод 3
$$
\frac{f(x + h) - f(x-h)}{2h} \approx f'(x) + f'''(x)\frac{h^2}{6} \Rightarrow
\Delta_m = f'''(x)\frac{h^2}{6}
$$
Полная погрешность:
$$
\Delta = \boxed{\frac{\varepsilon}{h} + f'''(x)\frac{h^2}{6}}
$$

### Метод 4
$$
  \frac{4}{3} \frac{f(x + h) - f(x  - h)}{2h}
  - \frac{1}{3} \frac{f(x + 2h) - f(x  - 2h)}{4h} \approx
  \frac{4}{3} \left( f'(x) + f^{(5)}(x) \cdot h^4 / 120 \right) -
  \frac{1}{3} \left( f'(x) + f^{(5)}(x) \cdot 16h^4 / 120 \right) \Rightarrow
  $$$$
  \Delta_{m} = f^{(5)}(x) \cdot \frac{h^4}{3\cdot 120} \left(4-16 \right) =
  -f^{(5)}(x) \cdot \frac{h^4}{30}
$$
Полная погрешность:
$$
\Delta = \frac{4\varepsilon}{3h} - \frac{1\varepsilon}{6h} -f^{(5)}(x) \cdot \frac{h^4}{30}
 = \boxed{\frac{3}{2}\frac{\varepsilon}{h} -f^{(5)}(x) \cdot \frac{h^4}{30}}
$$

### Метод 5
$$
  \frac{3}{2} \frac{f(x + h) - f(x  - h)}{2h}
  - \frac{3}{5} \frac{f(x + 2h) - f(x  - 2h)}{4h}+
   \frac{1}{10} \frac{f(x + 3h) - f(x  - 3h)}{6h} \approx
  \frac{3}{2} \left( f'(x) + f^{(7)}(x) \cdot \frac{h^6}{5040} \right) -
  \frac{3}{5} \left( f'(x) + f^{(7)}(x) \cdot 64\frac{h^6}{5040} \right) +
  \frac{1}{10} \left( f'(x) + f^{(7)}(x) \cdot 729\frac{h^6}{5040} \right)
   \Rightarrow
  $$$$
  \Delta_{m} = f^{(7)}(x) \cdot \frac{h^6}{10 \cdot 5040}
  \left(15 -  384 +729\right) =
  f^{(7)}(x) \cdot \frac{h^6}{140}
$$
Полная погрешность:
$$
\Delta = \frac{3\varepsilon}{2h} + \frac{3\varepsilon}{10h} +
\frac{\varepsilon}{30h}
+f^{(7)}(x) \cdot \frac{h^6}{140}
 = \boxed{\frac{11}{6}\frac{\varepsilon}{h} + f^{(7)}(x) \cdot \frac{h^6}{140}}
$$

## Реализуем полученные погрешности в функциях
- Функции являются функциями из `sympy`, каждой функции погрешности передается такая функция, точка, а также шаг.

```python
def err_1(f, x, h):
  return sp.diff(f, x_s, 2).subs(x_s, x).doit() * h  / 2 + 2 * eps / h

def err_2(f, x, h):
  return 2 * eps / h - sp.diff(f, x_s, 2).subs(x_s, x).doit() * h / 2

def err_3(f, x, h):
  return eps / h + sp.diff(f, x_s, 3).subs(x_s, x) * h * h / 6

def err_4(f, x, h):
  delta_round =  4 / 3 * eps / h + 1 / 3 * eps / (2 * h)
  meth_delta = - sp.diff(f, x_s, 5).subs(x_s, x) * h * h * h * h / 30
  return delta_round + meth_delta

def err_5(f, x, h):
  delta_round =  1.5 * eps / h + 0.6 * eps / (2 * h) + 0.1 * eps / (3 * h)
  meth_delta = sp.diff(f, x_s, 7).subs(x_s, x) * h * h * h * h * h * h / 140
  return delta_round + meth_delta

err_func_list = [err_1, err_2, err_3, err_4, err_5]
```

## Массив для исследуемых функций

```python
funcs_list = [
  sp.sin(x_s ** 2),
  sp.cos(sp.sin(x_s)),
  sp.exp(sp.sin(sp.cos(x_s))),
  sp.log(x_s + 3),
  (x_s + 3) ** 0.5
]
```

## Реализуем класс, представляющий метод
- Класс хранит в себе функцию ошибки, получаемую в конструкторе, имеет один метод:
  1. `plot` - строит требуемый график для функции `f` в точке `x`, в качестве функции ошибки используется функция, полученная в конструкторе

```python
class Method:
  # func : func(x)
  # err : err(f, x, h)
  def __init__(self, err):
    self.__err = err

  def plot(self, f, x):
    apply_x = lambda h: self.__err(f, x, h)

    delta_arr = abs(apply_x(h_arr))
    plt.loglog(h_arr, delta_arr, label = f'Method #{self.__err.__name__[-1]}')
```

## Реализуем функцию, для более удобного построения графиков
Данная функция принимает на вход символьную (`sympy`) функцию `func` и точку `x` (в которой считать производную), после чего строит на одном рисунке графики зависимости шага от погрешности для каждой функции ошибки.

```python
def run_method(func, x = np.pi / 4):
  for err in err_func_list:
    meth = Method(err)
    meth.plot(func, x)
  plt.legend()
  plt.title(f"f(x) = {str(func)}")
  plt.show()
```

## Запустим написанную функцию для каждой из требуемых функций, и получим графики

```python
run_method(funcs_list[0])
```

```python
run_method(funcs_list[1])
```

```python
run_method(funcs_list[2])
```

```python
run_method(funcs_list[3])
```

```python
run_method(funcs_list[4])
```


# Bayesian Linear Regression

```python
import MVG as MVG
import SuperMatExpr as SME
from sympy import *
import utils as utils
from IPython.display import display, Math, Latex
```

```python
# Define some symbols
D, N, Ns = symbols('D N Ns')
sig_y = symbols('\u03c3_y')
```

## 1. Prior

```python
# Prior
w = SME.SuperMatSymbol(D,1,name='w',mat_type='var')
mu_w = SME.SuperMatSymbol(D,1,mat_type='mean',dep_vars=[w],expanded=ZeroMatrix(D,1))
S_ww = SME.SuperMatSymbol(D,D,mat_type='covar',dep_vars=[w],expanded=Identity(D))
p_w = MVG.MVG([w],moments=[mu_w,S_ww,utils.getZ(S_ww)])

print("p_w:")
display(Latex(utils.matLatex(p_w)))
```

    p_w:

\begin{align*}
p\left(\mathbf{w}\right)&= \mathcal{N}\left(\mathbf{m}_{\mathbf{w}},\mathbf{\Sigma}_{\mathbf{w}}\right)\\
\mathbf{m}_{\mathbf{w}} &= \mathbf{0}\\
\mathbf{\Sigma}_{\mathbf{w}} &= \mathbf{I}\\
\end{align*}

# 2. Likelihood

```python
# Likelihood of w given X
X = SME.SuperMatSymbol(D,N,name='X',mat_type='var')
y = SME.SuperMatSymbol(N,1,name='y',mat_type='var')
mu_y = SME.SuperMatSymbol(N,1,mat_type='mean',dep_vars=[y],cond_vars=[w,X],expanded=X.T*w)
S_yy = SME.SuperMatSymbol(N,N,mat_type='covar',dep_vars=[y],cond_vars=[w,X],expanded=sig_y**2*Identity(N))
p_y = MVG.MVG([y],moments=[mu_y,S_yy,utils.getZ(S_yy)],conditioned_vars=[w,X])

print("p_y:")
display(Latex(utils.matLatex(p_y)))
```

    p_y:

\begin{align*}
p\left(\mathbf{y}|\mathbf{w},\mathbf{X}\right)&= \mathcal{N}\left(\mathbf{m}_{\mathbf{y}|\mathbf{w},\mathbf{X}},\mathbf{\Sigma}_{\mathbf{y}|\mathbf{w},\mathbf{X}}\right)\\
\mathbf{m}_{\mathbf{y}|\mathbf{w},\mathbf{X}} &= \mathbf{X}^T \mathbf{w}\\
\mathbf{\Sigma}_{\mathbf{y}|\mathbf{w},\mathbf{X}} &= \sigma_y^{2} \mathbf{I}\\
\end{align*}

# 3. Posterior

```python
# Joint of w and y
p_w_y = p_w*p_y

print("p_w_y:")
display(Latex(utils.matLatex(p_w_y)))
```

    cond_vars:  [w]
    X'*w
    coeffs:  [X']
    rem:  0
    Lambda:  [0]
    Omega:  [[X']]
    marginal_mean:  [0]
    utils.matmul(Omega,marginal_mean):  [0]
    p_w_y:

\begin{align*}
p\left(\mathbf{y},\mathbf{w}|\mathbf{X}\right)&= \mathcal{N}\left(\mathbf{m}_{\mathbf{y},\mathbf{w}|\mathbf{X}},\mathbf{\Sigma}_{\mathbf{y},\mathbf{w}|\mathbf{X}}\right)\\
\mathbf{m}_{\mathbf{y},\mathbf{w}|\mathbf{X}} &= \left[\begin{smallmatrix}\mathbf{0}\\\mathbf{0}\end{smallmatrix}\right]\\
\mathbf{\Sigma}_{\mathbf{y},\mathbf{w}|\mathbf{X}} &= \left[\begin{smallmatrix}\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}&\mathbf{X}^T\\\mathbf{X}&\mathbf{I}\end{smallmatrix}\right]\\
\end{align*}

```python
# Inference: posterior over w
p_w_post = p_w_y.condition([y])

print("p_w_post:")
display(Latex(utils.matLatex(p_w_post)))
```

    S.blockform:  [[S_(y,y)]]
    S.variables_dim1:  {y: 0}
    S.variables_dim2:  {y: 0}
    old_mean:  [0]
    P: [[S_(w,w)]], Q: [[S_(w,y)]], R: [[S_(y,w)]], S: [[S_(y,y)]]
    v_marg_vars:  [y]
    marg_mean:  [0]
    p_w_post:

\begin{align*}
p\left(\mathbf{w}|\mathbf{X},\mathbf{y}\right)&= \mathcal{N}\left(\mathbf{m}_{\mathbf{w}|\mathbf{X},\mathbf{y}},\mathbf{\Sigma}_{\mathbf{w}|\mathbf{X},\mathbf{y}}\right)\\
\mathbf{m}_{\mathbf{w}|\mathbf{X},\mathbf{y}} &= \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\\
\mathbf{\Sigma}_{\mathbf{w}|\mathbf{X},\mathbf{y}} &= \mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\\
\end{align*}

# 4. Prediction

```python
#Prediction

# Likelihood of w given Xs
Xs = SME.SuperMatSymbol(D,Ns,name='X_s',mat_type='var')
ys = SME.SuperMatSymbol(Ns,1,name='y_s',mat_type='var')
mu_ys = SME.SuperMatSymbol(Ns,1,mat_type='mean',dep_vars=[ys],cond_vars=[w,Xs],expanded=Xs.T*w)
S_ysys = SME.SuperMatSymbol(Ns,Ns,mat_type='covar',dep_vars=[ys],cond_vars=[w,Xs],expanded=sig_y**2*Identity(Ns))
p_ys = MVG.MVG([ys],moments=[mu_ys,S_ysys,utils.getZ(S_ysys)],conditioned_vars=[w,Xs])

print("p_ys:")
display(Latex(utils.matLatex(p_ys)))
```

    p_ys:

\begin{align*}
p\left(\mathbf{y}_{*}|\mathbf{w},\mathbf{X}_{*}\right)&= \mathcal{N}\left(\mathbf{m}_{\mathbf{y}_{*}|\mathbf{w},\mathbf{X}_{*}},\mathbf{\Sigma}_{\mathbf{y}_{*}|\mathbf{w},\mathbf{X}_{*}}\right)\\
\mathbf{m}_{\mathbf{y}_{*}|\mathbf{w},\mathbf{X}_{*}} &= \mathbf{X}_{*}^T \mathbf{w}\\
\mathbf{\Sigma}_{\mathbf{y}_{*}|\mathbf{w},\mathbf{X}_{*}} &= \sigma_y^{2} \mathbf{I}\\
\end{align*}

```python
# Joint of w and ys
p_w_ys = p_w_post*p_ys

print("p_w_ys:")
display(Latex(utils.matLatex(p_w_ys)))
```

    cond_vars:  [w]
    X_s'*w
    coeffs:  [X_s']
    rem:  0
    Lambda:  [0]
    Omega:  [[X_s']]
    marginal_mean:  [S_(w,y)*S_(y,y)^-1*y]
    utils.matmul(Omega,marginal_mean):  [X_s'*S_(w,y)*S_(y,y)^-1*y]
    p_w_ys:

\begin{align*}
p\left(\mathbf{y}_{*},\mathbf{w}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}\right)&= \mathcal{N}\left(\mathbf{m}_{\mathbf{y}_{*},\mathbf{w}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}},\mathbf{\Sigma}_{\mathbf{y}_{*},\mathbf{w}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}}\right)\\
\mathbf{m}_{\mathbf{y}_{*},\mathbf{w}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}} &= \left[\begin{smallmatrix}\mathbf{X}_{*}^T \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\\\mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\end{smallmatrix}\right]\\
\mathbf{\Sigma}_{\mathbf{y}_{*},\mathbf{w}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}} &= \left[\begin{smallmatrix}\sigma_y^{2} \mathbf{I} + \mathbf{X}_{*}^T \left(\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right) \mathbf{X}_{*}&\mathbf{X}_{*}^T \left(\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right)\\\left(\mathbf{I} + \left(-1 \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right)^T\right) \mathbf{X}_{*}&\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\end{smallmatrix}\right]\\
\end{align*}

```python
# Predictive distribution of ys
p_ys_post = p_w_ys.marginalize([w])

print("p_ys_post:")
display(Latex(utils.matLatex(p_ys_post)))
```

    new_blockfom:  [[S_(y_s,y_s), S_(y_s,w)], [S_(w,y_s), S_(w,w)]]
    self.covar.blockform:  [[S_(y_s,y_s), S_(y_s,w)], [S_(w,y_s), S_(w,w)]]
    new_blockfom:  [[S_(y_s,y_s)]]
    self.covar.blockform:  [[S_(y_s,y_s), S_(y_s,w)], [S_(w,y_s), S_(w,w)]]
    p_ys_post:

\begin{align*}
p\left(\mathbf{y}_{*}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}\right)&= \mathcal{N}\left(\mathbf{m}_{\mathbf{y}_{*}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}},\mathbf{\Sigma}_{\mathbf{y}_{*}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}}\right)\\
\mathbf{m}_{\mathbf{y}_{*}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}} &= \mathbf{X}_{*}^T \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\\
\mathbf{\Sigma}_{\mathbf{y}_{*}|\mathbf{y},\mathbf{X},\mathbf{X}_{*}} &= \sigma_y^{2} \mathbf{I} + \mathbf{X}_{*}^T \left(\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right) \mathbf{X}_{*}\\
\end{align*}

```python

```


# piston example with Gauss-Legendre collocation

```python
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.animation as anim

import sympy
sympy.init_printing()

from IPython.display import display

import numpy

import sys
sys.path.insert(0, './code')

from gauss_legendre import gauss_legendre
from symbolic import eval_expr
from evaluate_functional import evaluate_functional
import ideal_gas_lumped
```

### macroscopic state

```python
q = sympy.Symbol('q')
p = sympy.Symbol('p')
s_1 = sympy.Symbol('s_1')
s_2 = sympy.Symbol('s_2')

x = [q, p, s_1, s_2]
x
```

### parameters

```python
l = sympy.Symbol('l')
r = sympy.Symbol('r')
w = sympy.Symbol('w')
A = sympy.Symbol('A')

m_Al = sympy.Symbol('m_Al')
m_Cu = sympy.Symbol('m_Cu')
m = sympy.Symbol('m')

κ_Al = sympy.Symbol('κ_Al')
κ_Cu = sympy.Symbol('κ_Cu')
α = sympy.Symbol('α')

d = sympy.Symbol('d')

params = {
    # length of cylinder (m)
    l: 0.1,

    # radius of cylinder / piston (m)
    r: 0.05,

    # length of piston (m)
    w: 0.006,

    # cross-sectional area of piston (m**2)
    A: r**2 * sympy.pi,

    # density of piston material (kg/m**3)
    m_Al: 2700.0,
    m_Cu: 8960.0,

    # mass of piston (kg)
    m: m_Cu * A * w,

    # thermal conductivity piston material (W/(m*K))
    κ_Al: 237.0,
    κ_Cu: 401.0,

    # thermal conduction coefficient through piston (W/K)
    α: κ_Cu * A / w,

    # friction coefficient between piston and cylinder (N*s/m)
    d: 1.0,
}

params = {**params, **ideal_gas_lumped.params}
```

### functionals

```python
functionals = {}

v_1 = sympy.Symbol('v_1') # volume (m**3)
functionals[v_1] = A * (q - (w/2))

U_1 = sympy.Symbol('U_1') # internal energy
m_1 = sympy.Symbol('m_1') # fixed mass (kg)
θ_1 = sympy.Symbol('θ_1') # temperature (K)
π_1 = sympy.Symbol('π_1') # pressure (Pa)
ideal_gas_lumped.add_functionals(functionals, U=U_1, s=s_1, v=v_1, m=m_1, m_a=ideal_gas_lumped.m_Ar, θ=θ_1, π=π_1)

v_2 = sympy.Symbol('v_2')
functionals[v_2] = A * (l - (q + w/2))

U_2 = sympy.Symbol('U_2')
m_2 = sympy.Symbol('m_2')
θ_2 = sympy.Symbol('θ_2')
π_2 = sympy.Symbol('π_2')
ideal_gas_lumped.add_functionals(functionals, U=U_2, s=s_2, v=v_2, m=m_2, m_a=ideal_gas_lumped.m_Ar, θ=θ_2, π=π_2)

υ = sympy.Symbol('υ')
functionals[υ] = p / m

E = sympy.Symbol('E') # total energy (J)
functionals[E] = sympy.Rational(1,2) * p**2 / m + U_1 + U_2
```

```python
eval_expr(E, functionals)
```

### initial conditions

determine $m_1$, $m_2$ and $s_1(0)$, $s_2(0)$

```python
# wanted conditions
q0 = params[l]/2
v_10 = float(eval_expr(v_1, functionals, params, {q: q0}))
v_20 = float(eval_expr(v_2, functionals, params, {q: q0}))

θ_10 = 273.15 + 25.0
π_10 = 1.5 * 1e5

θ_20 = 273.15 + 20.0
π_20 = 1.0 * 1e5
```

```python
import ideal_gas
from scipy.optimize import fsolve
```

```python
n_10 = fsolve(lambda n : ideal_gas.S_π(ideal_gas.U2(θ_10, n), v_10, n) - π_10, x0=2e22)[0]
s_10 = ideal_gas.S(ideal_gas.U2(θ_10, n_10), v_10, n_10)
print(f"n = {n_10}")
print(f"s = {s_10}")
print(f"θ = {ideal_gas.U_θ(s_10, v_10, n_10) - 273.15} °C")
print(f"π = {ideal_gas.U_π(s_10, v_10, n_10) * 1e-5} bar")
print(f"u = {ideal_gas.U(s_10, v_10, n_10)}")
```

    n = 1.345119603317771e+22
    s = 3.3828942208847232
    θ = 25.00000000000159 °C
    π = 1.500000000000008 bar
    u = 83.05585577928

```python
n_20 = fsolve(lambda n : ideal_gas.S_π(ideal_gas.U2(θ_20, n), v_20, n) - π_20, x0=2e22)[0]
s_20 = ideal_gas.S(ideal_gas.U2(θ_20, n_20), v_20, n_20)
print(f"n = {n_20}")
print(f"s = {s_20}")
print(f"θ = {ideal_gas.U_θ(s_20, v_20, n_20) - 273.15} °C")
print(f"π = {ideal_gas.U_π(s_20, v_20, n_20) * 1e-5} bar")
print(f"u = {ideal_gas.U(s_20, v_20, n_20)}")
```

    n = 9.12041411630436e+21
    s = 2.3394613409617664
    θ = 20.000000000004206 °C
    π = 1.0000000000000147 bar
    u = 55.37057051952051

```python
x_0 = [q0, 0, s_10, s_20]
params[m_1] = n_10 * ideal_gas_lumped.m_Ar
params[m_2] = n_20 * ideal_gas_lumped.m_Ar
```

### dynamics

```python
x
```

```python
F = [υ, A*(π_1 - π_2) - d*υ, (α*(θ_2 - θ_1) + d*υ**2/2)/θ_1, (α*(θ_1 - θ_2) + d*υ**2/2)/θ_2]
F
```

```python
t_f = 0.8
dt = 1e-3
s = 2
print(f"K = {int(t_f // dt)}")
```

    K = 800

```python
%time time, solution = gauss_legendre(x, F, x_0, t_f, dt, s, functionals, params)
```

    CPU times: user 2.27 s, sys: 11.1 ms, total: 2.28 s
    Wall time: 2.28 s

```python
sol = solution.copy()
```

```python
fig, ax = plt.subplots(dpi=200)
ax.plot(time,solution[:, 0]);
ax.plot(time,sol[:, 0], '--');
```

```python
energy = evaluate_functional(x, E, solution, functionals, params)
S = s_1 + s_2
entropy = evaluate_functional(x, S, solution, functionals, params)

fig, axes = plt.subplots(nrows=3, sharex=True, dpi=200)
fig.tight_layout(pad=1.5)

#axes[0].set_title(f"time step = {dt}")

axes[0].plot(time, solution[:, 0])
axes[0].set_ylabel("$q \: (\mathrm{m})$")
axes[0].ticklabel_format(style='sci', axis='y', scilimits=(0,0))
axes[0].xaxis.major.formatter._useMathText = True

axes[1].plot(time, energy)
axes[1].set_ylabel("$E \: (\mathrm{J})$")
axes[1].ticklabel_format(style='sci', axis='y', scilimits=(0,0))

axes[2].plot(time, entropy)
axes[2].set_xlabel("$t \: (\mathrm{s})$")
axes[2].set_ylabel("$S \: (\mathrm{J}/\mathrm{K})$");
axes[2].ticklabel_format(style='sci', axis='y', scilimits=(0,0))

font = {'family' : 'Calibri',
        'weight' : 'normal',
        'size'   : 15}
matplotlib.rc('font', **font)
matplotlib.rc('text', usetex=True)

#fig.savefig("simulation.pdf")
```

```python
import matplotlib
import matplotlib.animation as anim

def animate(solution, functionals, params, file):
    plt.ioff()

    r2 = 2 * params[r]
    w2 = params[w] / 2
    l2 = params[l] - w2

    cmap = matplotlib.cm.get_cmap('YlOrRd')

    fig, ax = plt.subplots(dpi=200)
    ax.set_xlim(left=0, right=params[l])
    ax.set_ylim(bottom=-params[r], top=params[r])

    q0 = params[l]/2
    vol1 = plt.Rectangle((0, -params[r]), q0-w2, r2, fc='b')
    ax.add_patch(vol1)
    pist = plt.Rectangle((q0-w2, -params[r]), params[w], r2, fc='#C84843')
    ax.add_patch(pist)
    vol2 = plt.Rectangle((q0+w2, -params[r]), l2-q0, r2, fc='g')
    ax.add_patch(vol2)

    θ_1sol = evaluate_functional(x, θ_1, solution, functionals, params)
    θ_2sol = evaluate_functional(x, θ_2, solution, functionals, params)
    θ_min = numpy.min([numpy.min(θ_1sol), numpy.min(θ_2sol)])
    θ_max = numpy.max([numpy.max(θ_1sol), numpy.max(θ_2sol)])
    θ_swing = θ_max - θ_min

    data = numpy.block([solution, θ_1sol.reshape(-1,1), θ_2sol.reshape(-1,1)])

    def animate(datum):
        qd = datum[0]
        θ_1d = datum[4]
        θ_2d = datum[5]

        l_1 = qd - w2
        l_2 = l2 - qd

        vol1.set_width(l_1)
        pist.set_x(l_1)
        vol2.set_x(l_1 + params[w])
        vol2.set_width(l_2)

        vol1.set_fc(cmap(0.8 * ((θ_1d-θ_min) / θ_swing)))
        vol2.set_fc(cmap(0.8 * ((θ_2d-θ_min) / θ_swing)))

        return (vol1, pist, vol2)

    animation = anim.FuncAnimation(fig, animate, frames=data, blit=True, repeat=False)
    animation.save(file, fps=20, extra_args=['-vcodec', 'libx264'])
    plt.close(fig)
```

```python
%time animate(solution[:4000], functionals, params, 'piston.mp4')
```

    CPU times: user 26.9 s, sys: 1.39 s, total: 28.3 s
    Wall time: 28.9 s


### Exercise H3.2: MLP Regression

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy
import random
%matplotlib inline
```

```python
rdata = np.loadtxt(r"C:\Users\lcube\Desktop\jupyter\BCCN\MI\given\RegressionData.txt") #load the given regression data
#print(rdata)
```

```python
## A) Initialization:

N1 = 3 #number of hidden nodes

x = rdata[:,0]
y = rdata[:,1]

w10 = random.uniform(-0.5,0.5) #input weight
b10 = random.uniform(-0.5,0.5) #input bias

def hidden(x): #hidden layer

    w10 = [random.uniform(-0.5,0.5) for _ in range(N1)] #input weight

    return (np.tanh((np.dot(w10,x)) - b10))

w21 = random.uniform(-0.5,0.5) #output weight
b21 = random.uniform(-0.5,0.5) #output bias

def output(hid): #output layer

    w21 = random.uniform(-0.5,0.5) #output weight

    return((np.dot(w21,hid)) - b21)

```

```python
## B) Iterative learning:

# 1. Forward & Back Propagation

# a) Forward Propagation

z = np.linspace(1,len(x),len(x))

hid = [hidden(x[i]) for i in range(len(x))] #hidden layer for each input x
hid = np.array(hid)
print(hid)

#plot the activity of the hidden layer
plt.plot(z,hid)
plt.xlabel('input # (x_i)')
plt.ylabel('hidden layer activity')
plt.show()

out = np.array(output(hid)) #connect each hidden layer neuron to the output
print(out)

#plot the activity of the output layer
plt.plot(z,out)
plt.xlabel('input # (x_i)')
plt.ylabel('output layer activity')
plt.show()

yout = [(out[i][0] + out[i][1] + out[i][2]) for i in range(len(x))] #sum each of the output's activity
print(yout)

#plot the activity of the total output y
plt.plot(z,yout)
plt.xlabel('input # (x_i)')
plt.ylabel('output y')
plt.show()

```

```python
# b) Output error

def output_error(yout):

    qe = 0.5 * (y - yout) ** 2 #quadratic error cost function
    qe = np.sum(qe) #sum over each input to return a single value
    return qe

qe = output_error(yout)
print('output error:', qe)

```

    output error: 2.7240786364193976

```python
# c) Backpropagation

def derhidden(x): #hidden layer derivative

    return (1 / (np.cosh(np.dot(w10,x) - b10) ** 2))

def deroutput(hid): #output layer derivative

    return(0.5 * (np.dot(w21,hid) - b21) ** 2)

def le_hidden(): #local error hidden

    derhid = [derhidden(x[i]) for i in range(len(x))]
    derhid = np.array(hid)

    return derhid

le_hid = np.array(le_hidden())

def le_output(): #local error output

    derout = np.array(deroutput(hid))
    deryout = [(derout[i][0] + derout[i][1] + derout[i][2]) for i in range(len(x))]
    deryout = np.array(deryout)

    return deryout

le_out = np.array(le_output())

#plot the local errors of the hidden layer
plt.plot(z,le_hid)
plt.xlabel('input # (x_i)')
plt.ylabel('hidden layer local errors')
plt.show()

#plot the local error of the output
plt.plot(z,le_out)
plt.xlabel('input # (x_i)')
plt.ylabel('output local error')
plt.show()

```

```python
# d) Gradient

#to find the gradient, it will be the 'factor depending on the cost function' -> (y_calculated - y_actual)
#multiplied by the 'factor depending on the model class' -> (local error at neuron * activity of neuron)
#summed over and then divided by 'p'

p = 10 #size of given training set

def grad():

    grad_hid = [((yout - y) * (le_hid[:,i] * hid[:,i]) / p) for i in range(N1)] #gradient of hidden, w10
    grad_out = (yout - y) * (le_out * yout) / p #gradient of output, w21

    return grad_hid,grad_out

grad = grad()
grad_hid = np.array(grad[0])
grad_out = np.array(grad[1])

print('hidden layer gradients:', grad_hid)
print('output gradients:', grad_out)

```

    hidden layer gradients: [[ 1.75412019e-03  4.10324816e-03  3.66291943e-05  3.54705999e-03
      -4.34699120e-03  6.25035392e-04 -1.58082876e-02 -2.69021000e-04
      -1.72619741e-02 -1.97376046e-02]
     [ 5.54701527e-03  6.84732102e-04  8.96649994e-09  5.37751093e-05
      -4.48407825e-04  2.75067494e-03 -5.01165308e-03 -1.08967390e-06
      -1.39248532e-04 -1.02441810e-03]
     [ 1.09608779e-02  4.11846661e-03  3.22437867e-05  1.06828840e-02
      -6.30875298e-03  3.89574568e-02 -1.94278308e-02 -1.75406825e-04
      -1.94055267e-02 -2.10028452e-02]]
    output gradients: [-3.79416392e-04 -2.04707287e-04 -1.79144354e-06  2.39965435e-04
      2.21018507e-04  3.99713512e-03  1.41568304e-03  1.15654316e-05
     -2.74524100e-06  1.07612532e-03]

```python
# 2. Batch Gradient

def bgrad():

    bgrad_hid = [(-1) * np.sum(grad_hid[i]) for i in range(N1)] #batch gradient of hidden, w10
    bgrad_out = (-1) * np.sum(grad_out) #batch gradient of output, w21

    return bgrad_hid, bgrad_out

bgrad = bgrad()
bgrad_hid = bgrad[0]
bgrad_out = bgrad[1]

print('hidden layer batch gradient:', bgrad_hid)
print('output batch gradient:', bgrad_out)

```

    hidden layer batch gradient: [0.04735778551696786, -0.002411389175229092, 0.0015684332969992491]
    output batch gradient: -0.0063728324867093865

```python
# 3. Weight Update

#update the weights according to -> w(t+1) = w(t) + eta * bgrad

eta = 0.5 #learning rate

#training error ->

def training_error(qe):

    traine = (qe / p) #output error divided by p

    return traine

#print(training_error(qe))

def weight_update(w10,w21):

    w10 = w10 + (eta * bgrad_hid)
    w21 = w21 + (eta * bgrad_out)

    return w10, w21

wup = weight_update(w10,w21)

w10 = wup[0]
w21 = wup[1]

print('example hidden weights:', w10)
print('example output weights:', w21)

```

    example hidden weights: -0.10634993315935562
    example output weights: -0.02192414036231627

```python
# C) Stopping Criterion

ssv = 10 ** -5 #some small value

tmax = 3000

w10 = [random.uniform(-0.5,0.5) for _ in range(N1)] #input weights
b10 = [random.uniform(-0.5,0.5) for _ in range(N1)] #input bias
w21 = [random.uniform(-0.5,0.5) for _ in range(N1)] #input weights
b21 = [random.uniform(-0.5,0.5) for _ in range(N1)] #input bias

print('original w10:', w10)
print('original w21:', w21)

def mlp_reg(): #mlp regression

    count = 0
    old_te = 0

    hid = np.array([hidden(x[i]) for i in range(len(x))])
    out = np.array(output(hid))
    yout = [(out[i][0] + out[i][1] + out[i][2]) for i in range(len(x))]

    qe = output_error(yout)

    dte = (training_error(qe) - old_te) #change in training error

    while (abs(dte) / training_error(qe)) > ssv:

        count += 1

        le_hid = np.array(le_hidden())
        le_out = np.array(le_output())

        grad = grad()
        grad_hid = grad[0]
        grad_out = grad[1]

        bgrad = bgrad()
        bgrad_hid = bgrad[0]
        bgrad_out = bgrad[1]

        weight_update(w10,w21)

        if count == tmax:

            print('maximum iteration number of', tmax, 'has been reached')
            break

mlp_reg()

print('final w10:', w10)
print('final w21:', w21)

```

```python

```


\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\   \nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

```python
\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}
```

Display
\begin{equation*}
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\end{equation*}
display
import numpy as np
import matplotlib.pyplot as plt

X = np.linspace(-3, 3, 50, endpoint=True)
F = p(X)
plt.plot(X,F)

plt.show()

```python

```


```python
import torch
from torch.autograd import Variable
import torch.nn.functional as F
import torch.nn as nn
import collections
import time
import numpy as np

import sys

import os
```

# Learning Graph Structures
Consider the following Structure Learning Convolution (Zhang et al. 2020):

\begin{equation}
  y_i=f\left(\sum_{e_{i,j}\in\varepsilon}S_{ij}w_jx_j\right) \quad (1)
\end{equation}
We introduce a learnable graph Structure $S$ which encodes correlations $S_{ij}$ between node $n_i$ and $n_j$ where $S$ is a learnable variable. In $(1)$ $x_j$ is the j-th nodes' embedded feature vector and $y_i$ is the convolved output signal of node $i$.

Given a Graph $G=(V,E)$ with signal $X\in\mathbb{R}^{N\times D}$ and adjacency matrix $A$ we can use $(1)$ to define a graph convolutional layer as:
\begin{equation}
  \tilde{A}=D^{-1/2}AD^{-1/2} \quad (2)
\end{equation}
\begin{equation}
  H^{(l+1)}=\sigma(\tilde{A}\circ SH^{(l)}W^{(l)}) \quad (3)
\end{equation}
where $W^{(l)}\in\mathbb{R}^{C_{in}\times C_{out}}$ is a learnable matrix, $C_{in}$ is the number of input chanels, $C_{out}$ is the number of output chanels and $\circ$ denotes the Hadamard product. Remark that $H^{(0)}=X\in\mathbb{R}^{N\times D}$ is the graph signal of all nodes of the Graph.

## Target of the structure learning
Consider a signal $X_t$ at time step $t$ and a convolutional neural network as defined by equation $(2)$ with $l$ layers $H^{(1)},...,H^{(l)}$ and associated filters $W^{(1)},...,W^{(l)}$ we try to predict the graphs signal at timestep $t+1$.

```python
# Load data into a dataset (for time-series preferrably)

# Generate a data loader
```

```python
import math

from torch.nn.modules.module import Module
from torch.nn.parameter import Parameter
import torch
import torch.nn.functional as F

# define SLC layer
class SLConv(Module):
    def __init__(self, in_chanels, out_chanels):
        super(SLConv,self).__init__()
        self.in_chanels = in_chanels
        self.out_chanels = out_chanels
        self.weight = Parameter(torch.FloatTensor(in_chanels, out_chanels))
        self.reset_parameters()

    def reset_parameters(self):
        stdv = 1. / math.sqrt(self.weight.size(1))
        self.weight.data.uniform_(-stdv, stdv)

    def forward(self, x, adj, S):
        x = torch.mm(x, self.weight) # (N,out_chanels)
        weighting = torch.mul(S, adj) # (N,N)
        output = torch.mm(weighting,x)
        return output

    def __repr__(self):
        return self.__class__.__name__ + ' (' \
               + str(self.in_chanels) + ' -> ' \
               + str(self.out_chanels) + ')'

# model defitinition
class GCN(nn.Module):
    def __init__(self, nfeat, nhid, nclass, N):
        super(GCN, self).__init__()

        self.gc1 = SLConv(nfeat, nhid)
        self.gc2 = SLConv(nhid, nclass)

        self.S = Parameter(torch.FloatTensor(N,N)).data.uniform_(-1.,1.)

    def forward(self, x, adj):
        x = F.relu(self.gc1(x, adj, self.S))
        x = self.gc2(x, adj, self.S)
        return x
```

```python
# 3 x 3 Matrix
adj = torch.tensor([[1,0,0], [1,1,0], [0,1,1]])
S = torch.rand(3,3)
in_chanels = 1
out_chanels = 1
X = torch.rand(3,1)

model = GCN(1, 1, 1, 3)
model.forward(X,adj)
```

```python

```

```python
# 3 x 3 Matrix
adj = torch.tensor([[1,0,0], [1,1,0], [0,1,1]])
S = torch.rand(3,3)
in_chanels = 1
out_chanels = 1
X = torch.rand(3,1)

model = GCN(1, 1, 1, 3)
model.forward(X,adj)
```

    tensor([[-0.0081],
            [-0.0740],
            [-0.0450]], grad_fn=<MmBackward>)

```python

```


# Software profesional en Acústica 2020-21 (M2i)

*This notebook contains a modification of the notebook [FEM_Helmholtz_equation](https://github.com/spatialaudio/computational_acoustics/blob/master/FEM_Helmholtz_equation.ipynb), created by Sascha Spors, Frank Schultz, Computational Acoustics Examples, 2018. The text/images are licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/). The code is released under the [MIT license](https://opensource.org/licenses/MIT).*

# Revisiting the numerical solution of the Helmholtz equation: FEM and planewaves

This notebook illustrates the numerical solution of the wave equation for harmonic excitation using the so called [Finite Element Method](https://en.wikipedia.org/wiki/Finite_element_method) (FEM). The method aims at an approximate solution by subdividing the area of interest into smaller parts with simpler geometry, linking these parts together and applying methods from the calculus of variations to solve the problem numerically. The FEM is a well established method for the numerical approximation of the solution of partial differential equations (PDEs). The solutions of PDEs are often known analytically only for rather simple geometries. FEM based simulations allow to gain insights into other more complex cases.

## Problem Statement

The inhomogeneous linear [wave equation](https://en.wikipedia.org/wiki/Wave_equation) is given as

\begin{equation}
\Delta p(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2}{\partial t^2} p(\mathbf{x}, t) = - q(\mathbf{x}, t) ,
\end{equation}

where $p(\mathbf{x}, t)$ denotes the sound pressure at position $\mathbf{x}$, $c$ the speed of sound and $q(\mathbf{x}, t)$ the inhomogeneity.
For an harmonic excitation $q(\mathbf{x}, t) = \Re \{ Q(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} \}$ with frequency $\omega = 2 \pi f$ we choose the Ansatz $p(\mathbf{x}, t) = \Re \{ P(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} \}$ for the sound pressure.
Introduction of the complex quantities into the wave equation yields

\begin{equation}
\Delta P(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} + \frac{\omega^2}{c^2} P(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} = - Q(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} ,
\end{equation}

and canceling out the $\mathrm{e}^{\mathrm{j} \omega t}$ terms yields the [Helmholtz equation](https://en.wikipedia.org/wiki/Helmholtz_equation)

\begin{equation}
\Delta P(\mathbf{x}, \omega) + \frac{\omega^2}{c^2} P(\mathbf{x}, \omega) = - Q(\mathbf{x}, \omega) .
\end{equation}

We aim for a numerical solution of the Helmholtz equation on the domain $V$ with respect to the homogeneous Dirichlet boundary condition

\begin{equation}
P(\mathbf{x}, \omega) = P_{D} \qquad \text{for } x \in \partial V
\end{equation}

where $P_{D}$ is the Dirichlet data.

## Variational Formulation

The FEM is based on expressing the partial differential equation (PDE) to be solved in its [variational](https://en.wikipedia.org/wiki/Calculus_of_variations) or weak form.
The first step towards this formulation is to multiply the Helmholtz equation by the test function $V(\mathbf{x}, \omega)$

\begin{equation}
\Delta P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega)  + \frac{\omega^2}{c^2} P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) = - Q(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) ,
\end{equation}

followed by integration over the domain $V$

\begin{equation}
\int_V \Delta P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) \mathrm{d}x  + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) \mathrm{d}x = - \int_V Q(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) \mathrm{d}x ,
\end{equation}

where $\mathrm{d}x$ denotes a suitably chosen differential element for integration.
Application of [Green's first identity](https://en.wikipedia.org/wiki/Green%27s_identities) yields

\begin{equation}
{-} \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x  + \int_{\partial V} V(\mathbf{x}, \omega) \frac{\partial}{\partial n}  P(\mathbf{x}, \omega) \mathrm{d}s + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x = - \int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x .
\end{equation}

This way the differential order of the first integral is lowered which is advisable for application of the FEM.
The second integral vanishes as

* the variation formulation requires $V(\mathbf{x}, \omega) = 0$ on $\partial V$ where $P(\mathbf{x}, \omega)$ is known - here by the pure Dirichlet boundary condition

This results in the variational/weak formulation of the Helmholtz equation

\begin{equation}
{-} \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x  + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x = - \int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x .
\end{equation}

It is common to express the integral equation above in terms of the bilinear $a(P, V)$ and linear $L(V)$ forms

\begin{equation}
a(P, V) = \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x - \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x ,
\end{equation}

\begin{equation}
L(V) = - \int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x ,
\end{equation}

where

\begin{equation}
a(P, V) = L(V) .
\end{equation}

## Numerical Solution

The numerical solution of the variational problem is based on [FEniCS](https://fenicsproject.org/), an open-source framework for numerical solution of PDEs.
Its high-level Python interface `dolfin` is used in the following to define the problem and compute its solution.
The implementation is based on the variational formulation derived above.
It is common in the FEM to denote the solution of the problem by $u$ and the test function by $v$.
The definition of the problem in FEniCS is very close to the mathematical formulation of the problem.
We limit ourselves to real-valued $P(\mathbf{x}, \omega)$ due to the assumption of Dirichlet real data.

A function is defined for this purpose, accompanied by a plotting routine for the resulting sound field.

```python
import numpy as np
import dolfin
import mshr
import matplotlib.pyplot as plt
%matplotlib inline

# Compute the Finite Element approximation
def FEM_Helmholtz(mesh, frequency, dirichlet_data, c=343):
    '''numerical solution of the Helmholtz equation using the FEM'''

    # squared wavenumber
    k2 = dolfin.Constant(2*dolfin.pi*frequency/c)**2

    # define function space
    V = dolfin.FunctionSpace(mesh, "CG", 2)

    # define boundary conditions
    bcs = dolfin.DirichletBC(V, dirichlet_data, "on_boundary")

    # define variational problem
    u = dolfin.TrialFunction(V)
    v = dolfin.TestFunction(V)

    a = k2 * dolfin.inner(u, v) * dolfin.dx - dolfin.inner(dolfin.nabla_grad(u), dolfin.nabla_grad(v)) * dolfin.dx
    L = dolfin.Constant(0) * v * dolfin.dx

    # Assemble the linear system of equations
    A, b = dolfin.assemble_system(a, L, bcs)

    # compute solution
    u = dolfin.Function(V)
    dolfin.solve(A, u.vector(), b)

    return u, V

# Compute the Finite Element error
def compute_error(u, uex, V):
    uex_interp = dolfin.interpolate(uex, V)
    return dolfin.errornorm(uex_interp, u)/dolfin.norm(uex_interp)

# Plot the Finite Element approximation
def plot_soundfield(u):
    '''plot solution of FEM-based simulation'''
    fig = plt.figure(figsize=(10,10))
    fig = dolfin.plot(u)
    plt.title(r'$P(\mathbf{x}, \omega)$')
    plt.xlabel(r'$x$ / m')
    plt.ylabel(r'$y$ / m')
    plt.colorbar(fig, fraction=0.038, pad=0.04);
```

### Sound Field in a Rectangular Room with Sound-soft Boundaries

The two-dimensional sound field in a rectangular room whose height is very small compared to the wavelength and with soft boundaries (Dirichlet boundary condition) is computed for the frequency $f=120$ Hz. The Dirichlet data is settled in such a manner that the exact solution is the real part of a plane wave
$$
P_{ex}(\mathbf{x}, \omega)=\cos(k_{0}x_0+k_{1}x_{1}),\qquad (k_0,k_1)=\frac{\omega}{c}(\cos\theta,\sin\theta),\qquad \theta=\frac{\pi}{4}.
$$

```python
# define geometry and mesh
mesh = dolfin.RectangleMesh(dolfin.Point(0,0), dolfin.Point(5,4), 200, 200, "right/left")

# Physical data
freq = 120.
c = 343.

# Exact solution: plane wave with wavenumber vector (k0,k1)
theta = np.pi/4
k0 = 2*np.pi*freq/c*np.cos(theta)
k1 = 2*np.pi*freq/c*np.sin(theta)
uex = dolfin.Expression("cos(k0*x[0]+k1*x[1])", k0=k0, k1=k1, degree=5)

# compute solution
u, V = FEM_Helmholtz(mesh, freq, uex, c)

# compute error
print('L2-relative error:', compute_error(u, uex, V))

# plot sound field
plot_soundfield(u)
plt.title(r'$|P(\mathbf{x}, \omega)|$');
```

### Study of convergence

Let's check the order of convergence of this finite element discretization computing the error for sucesive refinements of the mesh

```python
# Mesh 50,50
mesh = dolfin.RectangleMesh(dolfin.Point(0,0), dolfin.Point(5,4), 50, 50, "right/left")
u, V = FEM_Helmholtz(mesh, freq, uex, c)
print('Mesh size', mesh.hmax())
print('L2-relative error:', compute_error(u, uex, V))
# Mesh 50,50
mesh = dolfin.RectangleMesh(dolfin.Point(0,0), dolfin.Point(5,4), 100, 100, "right/left")
u, V = FEM_Helmholtz(mesh, freq, uex, c)
print('Mesh size', mesh.hmax())
print('L2-relative error:', compute_error(u, uex, V))
# Mesh 50,50
mesh = dolfin.RectangleMesh(dolfin.Point(0,0), dolfin.Point(5,4), 200, 200, "right/left")
u, V = FEM_Helmholtz(mesh, freq, uex, c)
print('Mesh size', mesh.hmax())
print('L2-relative error:', compute_error(u, uex, V))
```

    Mesh size 0.12806248474865742
    L2-relative error: 2.3773320886652655e-05
    Mesh size 0.06403124237432907
    L2-relative error: 2.606655683008537e-06
    Mesh size 0.032015621187164535
    L2-relative error: 3.133589561437329e-07

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT).