| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section10.4, chauffage par effet Brinkman: source de chauffage causé par la viscosité

>
>> Ici on traitera le problème de façon légèrement différente de Transport Phenomena, en effet, pour traiter ce problème on peut aller chercher les équations générales de l'annexe B.9.

>>Regardons donc l'équation B.9-1 qui est utilisée puisqu'on suppose que le système a une courbure négligeable et peut être traité en coordonnées cartésiennes.
Tous les termes de gauche sont nuls car:

>>> On veut le résultat en état de régime, donc la dérivée en fonction du temps est nulle.

>>> Il n'y a pas de vitesse dans la direction x ni dans la direction y, et il n'y a pas de gradient de température dans la direction z.

>>> Le seul terme dans les dérivées de la température qui reste à droite est celui dans la direction x, et le terme de source $\Phi$, donc:

>>>> ###  $k \frac {\partial^2 T}{\partial x^2}+\mu \Phi_v = 0$

>> Le terme de source visqueuse est obtenu par l'annexe B7

>

qui peut sembler très complexe (en effet) mais tous les termes s'annulent sauf celui du gradient de $v_z$ en fonction de x donc:

>>>> ### $\Phi_v =\big(\frac {\partial v_z}{\partial x} \big)^2 $
>> L'équation à résoudre sera donc directement:

>>>> ### $k \frac {\partial^2 T}{\partial x^2}+ \mu \big(\frac {\partial v_z}{\partial x} \big)^2= 0$

> Et puisque le profil de vitesse vous a déjà été donné sur la figure 10.4-2, on pourra résoudre

```python
#
# Pierre Proulx
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
%matplotlib inline
```

```python
# Paramètres, variables et fonctions
#
mu,k,b,Tb,T0,vb,x=sp.symbols('mu k b T_b T_0 v_b x')
T=sp.Function('T')(x)
v=sp.Function('v')(x)
```

```python
v=vb*x/b
eq=k*T.diff(x,x)+mu*v.diff(x)**2
display(eq)
```

```python
T=sp.dsolve(eq)
display('Le profil de température, solution générale avec les constantes inconnues',T)
```

```python
condition_1=sp.Eq(T.rhs.subs(x,0),T0)
condition_2=sp.Eq(T.rhs.subs(x,b),Tb)
display('Les conditions aux limites',condition_1, condition_2)
constantes=sp.solve([condition_1,condition_2],sp.symbols('C1,C2'))
display('Les constantes après solution',constantes)
T=T.subs(constantes)
display('Le profil de température final',T)
```

```python
### Tracage
T=T.rhs
dico={'T_0':0,'T_b':10,'k':0.1,'mu':0.01,'b':0.01,'v_b':10}
T1=T.subs(dico)
dico={'T_0':0,'T_b':10,'k':0.1,'mu':0.05,'b':0.01,'v_b':10}
T2=T.subs(dico)
dico={'T_0':0,'T_b':10,'k':0.1,'mu':0.1,'b':0.01,'v_b':10}
T3=T.subs(dico)
dico={'T_0':0,'T_b':10,'k':0.1,'mu':0.15,'b':0.01,'v_b':10}
T4=T.subs(dico)
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize']=10,8
p=sp.plot((T1,(x,0,0.01)),(T2,(x,0,0.01)),(T3,(x,0,0.01)),(T4,(x,0,0.01)),
       legend=True,ylabel='T',xlabel='x',show=False)
p[0].label='mu=0.01'
p[1].label='.05'
p[2].label='.1'
p[3].label='.15'
p[0].line_color='blue'
p[1].line_color='red'
p[2].line_color='green'
p[3].line_color='black'
p.show()
```


# Labeling: Excess Return Over Median

_*Fig. 1:*_ Distribution of excess over median return for 22 stock tickers from period between Jan 2019 and May 2020.

## Abstract

In this notebook, we demonstrate labeling financial returns data according to excess over median. Using cross-sectional data on returns of many different stocks, each observation is labeled according to whether (or how much) its return exceeds the median return. Correlations can then be found between features and the likelihood that a stock will outperform the market.

## Introduction
This technique is used in the following paper:
["The benefits of tree-based models for stock selection"](https://link.springer.com/article/10.1057/jam.2012.17) by _Zhu et al._ (2012).

In that paper, independent composite features are constructed as weighted averages of various parameters in fundamental and quantitative analysis, such as PE ratio, corporate cash flows, debt etc. The composite features are applied as parameters in linear regression or a decision tree to predict whether a stock will outperform the market median return.

## How it works

A dataframe containing forward total stock returns is calculated from close prices. The median return of all companies at time $t$ in the dataframe is used to represent the market return, and excess returns are calculated by subtracting the median return from each stock's return over the time period $t$ \[Zhu et al. 2012\]. The numerical returns over median can then be used as is (for regression analysis), or can be relabeled simply to its sign (for classification analysis).

At time $t$:

$$P_t = \{p_{t,0}, p_{t,1}, ..., p_{t,n}\}$$
$$m_t = median(P_t)$$
$$L(P_t) = \{p_{t,0} - m_t, p_{t,1} - m_t, ...,p_{t,n} - m_t\}$$

If categorical rather than numerical labels are desired:

$$
     \begin{equation}
     \begin{split}
       L(p_{t,n}) = \begin{cases}
       -1 &\ \text{if} \ \ p_{t,n} - m_t < 0\\
       0 &\ \text{if} \ \ p_{t,n} - m_t = 0\\
       1 &\ \text{if} \ \ p_{t,n} - m_t > 0\\
       \end{cases}
     \end{split}
     \end{equation}
$$

---
## Examples of use

```python
import numpy as np
import pandas as pd
import yfinance as yf

from mlfinlab.labeling import excess_over_median

import matplotlib.pyplot as plt
```

```python
# Load price data for 22 stocks
tickers = "AAPL MSFT COST PFE SYY F GE BABA AMD CCL ZM FB WFC JPM NVDA CVX AAL UBER C UA VZ NOK"

data = yf.download(tickers, start="2019-01-20", end="2020-05-25",
                   group_by="ticker")
data = data.loc[:, (slice(None), 'Adj Close')]
data.columns = data.columns.droplevel(1)
data.head()
```

    [*********************100%***********************]  22 of 22 completed

      AMD
      FB
      BABA
      NVDA
      MSFT
      AAL
      C
      JPM
      COST
      F
      ...
      UA
      ZM
      UBER
      NOK
      VZ
      AAPL
      PFE
      CVX
      SYY
      CCL

      Date

      2019-01-22
      19.760000
      147.570007
      152.149994
      148.102676
      103.568062
      32.219025
      59.116608
      98.963676
      209.413116
      7.837517
      ...
      18.590000
      NaN
      NaN
      5.846427
      54.096535
      150.266403
      39.946537
      105.162872
      60.661575
      51.750248

      2019-01-23
      19.799999
      144.300003
      152.029999
      148.620316
      104.577469
      31.146364
      59.384232
      98.713730
      209.107468
      7.689988
      ...
      18.400000
      NaN
      NaN
      5.924772
      54.827435
      150.874130
      39.842587
      104.273567
      60.884239
      51.512947

      2019-01-24
      20.850000
      145.830002
      155.860001
      157.131958
      104.077667
      33.124386
      59.938599
      98.771400
      207.352509
      7.929724
      ...
      18.860001
      NaN
      NaN
      6.032495
      54.172474
      149.678253
      38.699097
      106.258125
      60.535717
      52.224846

      2019-01-25
      21.930000
      149.009995
      159.210007
      159.431610
      105.028282
      34.423378
      61.190701
      99.396294
      206.129929
      8.169458
      ...
      19.490000
      NaN
      NaN
      6.463387
      53.536488
      154.638153
      38.406132
      105.986641
      60.041988
      52.689953

      2019-01-28
      20.180000
      147.470001
      158.919998
      137.390930
      102.980049
      35.988075
      61.028221
      99.867378
      207.806046
      7.985046
      ...
      19.370001
      NaN
      NaN
      6.355664
      52.274014
      153.207047
      37.357147
      105.003731
      60.284012
      53.534740

5 rows × 22 columns

We find the excess return over median for all tickers in the time period, calculate the mean and standard deviation of returns, and plot the distribution.

```python
excess1 = excess_over_median(data)
excess1.head()
```

      AMD
      FB
      BABA
      NVDA
      MSFT
      AAL
      C
      JPM
      COST
      F
      ...
      UA
      ZM
      UBER
      NOK
      VZ
      AAPL
      PFE
      CVX
      SYY
      CCL

      Date

      2019-01-22
      0.001406
      -0.022777
      -0.001406
      0.002877
      0.009129
      -0.033911
      0.003909
      -0.003143
      -0.002077
      -0.019441
      ...
      -0.010838
      NaN
      NaN
      0.012783
      0.012893
      0.003427
      -0.003220
      -0.009074
      0.003053
      -0.005203

      2019-01-23
      0.043061
      0.000634
      0.015223
      0.047302
      -0.014748
      0.053538
      -0.000634
      -0.009385
      -0.018362
      0.021206
      ...
      0.015031
      NaN
      NaN
      0.008213
      -0.021915
      -0.017895
      -0.038669
      0.009063
      -0.015693
      0.003851

      2019-01-24
      0.034036
      0.004044
      0.003731
      -0.003127
      -0.008629
      0.021453
      0.003127
      -0.011436
      -0.023659
      0.012470
      ...
      0.015642
      NaN
      NaN
      0.053666
      -0.029502
      0.015375
      -0.025333
      -0.020317
      -0.025918
      -0.008857

      2019-01-25
      -0.070535
      -0.001071
      0.007443
      -0.128981
      -0.010238
      0.054719
      0.006609
      0.014004
      0.017396
      -0.013309
      ...
      0.003107
      NaN
      NaN
      -0.007402
      -0.014317
      0.000010
      -0.018049
      -0.000010
      0.013295
      0.025297

      2019-01-28
      -0.040490
      -0.016647
      -0.007242
      -0.040851
      -0.014771
      -0.002062
      -0.004429
      0.008386
      0.003412
      0.017142
      ...
      -0.017121
      NaN
      NaN
      0.014840
      -0.026909
      -0.004770
      0.036963
      0.002564
      0.002062
      0.003112

5 rows × 22 columns

We can visualize the distribution as a histogram.

```python
s2 = pd.Series(excess1.iloc[:-1, :].values.flatten())
ax2 = s2.plot.hist(bins=50)
ax2.set_xlim(-0.2,0.2)
ax2.set_xlabel('Return Over Median')
ax2.set_title('Distribution of Return Over Median for 22 Stocks')
plt.savefig('distribution_over_median.png')
```

Instead of returning the numerical value of excess return over median, we can also simply return the sign. Using categorical rather than numerical labels alleviates problems that can arise due to extreme outlier returns [Zhu et al. 2012].

```python
excess2 = excess_over_median(data, binary=True)
excess2.head()
```

    c:\users\ruifan\mlfinlab\mlfinlab\labeling\excess_over_median.py:41: RuntimeWarning: invalid value encountered in sign
      return np.sign(returns_over_median)

      AMD
      FB
      BABA
      NVDA
      MSFT
      AAL
      C
      JPM
      COST
      F
      ...
      UA
      ZM
      UBER
      NOK
      VZ
      AAPL
      PFE
      CVX
      SYY
      CCL

      Date

      2019-01-22
      1.0
      -1.0
      -1.0
      1.0
      1.0
      -1.0
      1.0
      -1.0
      -1.0
      -1.0
      ...
      -1.0
      NaN
      NaN
      1.0
      1.0
      1.0
      -1.0
      -1.0
      1.0
      -1.0

      2019-01-23
      1.0
      1.0
      1.0
      1.0
      -1.0
      1.0
      -1.0
      -1.0
      -1.0
      1.0
      ...
      1.0
      NaN
      NaN
      1.0
      -1.0
      -1.0
      -1.0
      1.0
      -1.0
      1.0

      2019-01-24
      1.0
      1.0
      1.0
      -1.0
      -1.0
      1.0
      1.0
      -1.0
      -1.0
      1.0
      ...
      1.0
      NaN
      NaN
      1.0
      -1.0
      1.0
      -1.0
      -1.0
      -1.0
      -1.0

      2019-01-25
      -1.0
      -1.0
      1.0
      -1.0
      -1.0
      1.0
      1.0
      1.0
      1.0
      -1.0
      ...
      1.0
      NaN
      NaN
      -1.0
      -1.0
      1.0
      -1.0
      -1.0
      1.0
      1.0

      2019-01-28
      -1.0
      -1.0
      -1.0
      -1.0
      -1.0
      -1.0
      -1.0
      1.0
      1.0
      1.0
      ...
      -1.0
      NaN
      NaN
      1.0
      -1.0
      -1.0
      1.0
      1.0
      1.0
      1.0

5 rows × 22 columns

We can verify that the number of positive labels matches the number of negative labels. Note: for larger data sets, there is increased chance that some tickers will have the exact same return for a given time index. If that return is also equal to the the median, the number of positive labels may not match exactly with the number of negatives, but should be very close.

```python
excess2.stack().value_counts()
```

    -1.0    3629
     1.0    3629
     0.0      19
    dtype: int64

---
## Conclusion

This notebook presents the method to label data according to excess return over median. This method can return either numerical or categorical labels for observations. Zhu et al. utilize these labels to predict monthly stock returns using linear regression and decision trees based on composite features as independent variables. In this process:
 - Forward rates of return for assets are calculated for the entire selection of stocks indexed by time bars.
 - At each time index, the median rate of return for all stocks is calculated. The median is subtracted from each stock's return to find the excess return over median.
 - If categorical labels are desired, the excess returns are converted to their signs.

This method is useful for labelling data used for training regression models and decision trees. Zhu et al. found that decision trees were slightly better at predicting outperformers than linear regression.

## References

1. Zhu, M., Philpotts, D. and Stevenson, M., 2012. The benefits of tree-based models for stock selection. Journal of Asset Management, [online] 13(6), pp.437-448. Available at: <https://link.springer.com/article/10.1057/jam.2012.17>.

jupyter-notebooks/hudson-and-thames-quant/Labelling/Labels Excess Over Median/excess_over_median.ipynb


```python
import control
from control.matlab import *
import numpy as np
import scipy as sc
import sympy as sy
import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
```

```python
def multiply_poly(a):
    p1 = np.array([1])
    for i in range(len(a)):
        polynom = np.array(a[i])
        p1 = sc.signal.convolve(p1, polynom)
    return p1
```

```python
multiply_poly([[1,0], [1, 5.6], [1, 10]])
```

    array([ 1. , 15.6, 56. ,  0. ])

```python
poles = []
zeros = []
n = 190
for k in range(1, n):
    p1 = np.array([98*k])
    p2 = np.array([1,12+1.4*k,98*k])
    p, z = control.pzmap(tf(p1,p2), Plot=False, grid=False)
    poles.append(p)
    zeros.append(z)

plt.figure(figsize=(8,6))

p_x = []
p_y = []
for i in range(0, n-1):
    p_x.append(np.real(poles[i][0]))
    p_y.append(np.imag(poles[i][0]))
plt.plot(p_x, p_y, '-b')

p_x = []
p_y = []
for i in range(0, n-1):
    p_x.append(np.real(poles[i][1]))
    p_y.append(np.imag(poles[i][1]))
plt.plot(p_x, p_y, '-b')
```


# Representations of the Wave Equation and its Solutions

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the masters course Selected Topics in Audio Signal Processing, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Time-Domain

The inhomogeneous wave equation in the time-domain is given as

\begin{equation}
\Delta p(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2}{\partial t^2} p(\mathbf{x}, t) = - q(\mathbf{x}, t) .
\end{equation}

The wave equation is a linear partial differential equation (PDE) with constant coefficients.

## Monochromatic

Let's assume a monochromatic real-valued excitation $q(\mathbf{x}, t) = Q(\mathbf{x}, \omega_0) \cos(\omega_0 t)$ with angular frequency $\omega_0 = 2 \pi f_0$. Due to the linearity of the wave equation, the following Ansatz is chosen for the sound pressure $p(\mathbf{x}, t) = P(\mathbf{x}, \omega_0) \cos(\omega_0 t)$. It is common to use complex calculus for PDEs, hence

\begin{eqnarray}
q(\mathbf{x}, t) &= \Re \{ Q(\mathbf{x}, \omega_0) e^{j \omega_0 t} \} \\
p(\mathbf{x}, t) &= \Re \{ P(\mathbf{x}, \omega_0) e^{j \omega_0 t} \} .
\end{eqnarray}

Introducing the complex quantities into the wave equation yields

\begin{equation}
\Delta P(\mathbf{x}, \omega_0) e^{j \omega_0 t} + \left( \frac{\omega_0}{c} \right)^2 P(\mathbf{x}, \omega_0) e^{j \omega_0 t} = - Q(\mathbf{x}, \omega_0) e^{j \omega_0 t} .
\end{equation}

The complex exponential function can be canceled resulting in

\begin{equation}
\Delta P(\mathbf{x}, \omega_0) + \left( \frac{\omega_0}{c} \right)^2 P(\mathbf{x}, \omega_0) = - Q(\mathbf{x}, \omega_0) .
\end{equation}

For $t=0$ the time-domain solution of the wave equation for $\omega_0$ is given as $p(\mathbf{x}, 0) = \Re \{ P(\mathbf{x}, \omega_0) \}$. It is common to discard the index 0 in the angular frequency. The index was however introduced for the sake of comparison with the temporal Fourier transform of the wave equation discussed in the following.

## Temporal Fourier Domain

Temporal Fourier transform of the inhomogeneity and the pressure yields the Helmholtz equation

\begin{equation}
\Delta P(\mathbf{x}, \omega) + \left( \frac{\omega}{c} \right)^2 P(\mathbf{x}, \omega) = - Q(\mathbf{x}, \omega) ,
\end{equation}

where the differentiation theorem of the Fourier transform has been applied. The link to the monochromatic case can be found by considering a monochromatic excitation

\begin{equation}
Q(\mathbf{x}, \omega_0) = Q(\mathbf{x}, \omega) \delta(\omega - \omega_0) .
\end{equation}

Due to the linearity of the wave equation the following Ansatz is chosen

\begin{equation}
P(\mathbf{x}, \omega_0) = P(\mathbf{x}, \omega) \delta(\omega - \omega_0) .
\end{equation}

Introducing the right-hand-sides of the latter two equations into the Helmholtz equation yields the same result as for the monochromatic case

\begin{equation}
\Delta P(\mathbf{x}, \omega_0) + \left( \frac{\omega_0}{c} \right)^2 P(\mathbf{x}, \omega_0) = - Q(\mathbf{x}, \omega_0)
\end{equation}

Hence, the time-domain solution for a monochromatic excitation for $t=0$ may also be derived from the temporal Fourier transform of the problem by evaluation of $p(\mathbf{x}, \omega) = \Re \{ P(\mathbf{x}, \omega) \}$ for one particular frequency $\omega_0$.

## Temporal Excitation

Let's consider the solution $P_\delta(\mathbf{x}, \omega)$ of the Helmholtz equation for the excitation with a temporal Dirac impulse $q(\mathbf{x}, t) = q(\mathbf{x}) \delta(t)$

\begin{equation}
\Delta P_\delta(\mathbf{x}, \omega) + \left( \frac{\omega}{c} \right)^2 P_\delta(\mathbf{x}, \omega) = - Q(\mathbf{x})
\end{equation}

Due to the linearity of the wave equation the pressure field for a generic time-domain excitation $q(\mathbf{x}, t) = q(\mathbf{x}) \hat{q}(t)$ is given as

\begin{equation}
P(\mathbf{x}, \omega) = P_\delta(\mathbf{x}, \omega) \hat{Q}(\omega)
\end{equation}

or

\begin{equation}
p(\mathbf{x},t) = p_\delta(\mathbf{x}, t) * \hat{q}(t)
\end{equation}

**Copyright**

This notebook is provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text/images/data are licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Selected Topics in Audio Signal Processing - Supplementary Material*.


---
### 적분

 - 정적분
 - 부정적분 : 미분과 반대되는 개념

---
##### 부정적분으로 찾은 원래 함수의 표기

 - $\frac{dF(x)}{dx} = f(x) \leftrightarrow F(x) = \int f(x)dx+C$

----
#### 편미분의 부정적분

 - $\int f(x,y)dx+C(y)$
  * $dx$에서 $x$는 $x$로 미분했다는 것을 알려주는 표시

 - $\int (1+xy)dx \rightarrow x + \frac{x^2y}{2}+C(y)$

---
#### 다차 도함수와 다중적분

 - $\int\int f(x,y)dxdy$
  * 순서는 안쪽부터 바깥쪽으로 진행

---
### SymPy를 이용한 부정적분

 - intergrate()

```python
import sympy

sympy.init_printing(use_unicode='mathjax')
```

```python
x, e = sympy.symbols('x e')

f = x * sympy.exp(x) + sympy.exp(x)
f
```

```python
sympy.integrate(f)
```

```python
x, y = sympy.symbols('x y')
f = 2*x+y
f
```

```python
sympy.integrate(f, x)
```

---
#### 연습문제

```python
x, d = sympy.symbols('x d')
```

```python
f = 3*x**2
f
```

```python
sympy.integrate(f)
```

```python
f = (3*x**2 - 6*x + 1)
f
```

```python
sympy.integrate(f)
```

```python
f = (2+6*x+4*sympy.exp(x)+5/x)
f
```

```python
sympy.integrate(f)
```

```python
f = 2*x / (x**2-1)
f
```

```python
sympy.integrate(f)
```

```python
x, y, e = sympy.symbols('x y e')
```

```python
f = (1+x*y)
sympy.integrate(f, x)
```

```python
f = (x*y)*sympy.exp(x**2+y**2)
sympy.integrate(f, x)
```


# Creates the following SBML models needed for the Sub-SBML notebook

1. Transcription SBML model
1. Translation SBML model
1. Signal in mixture model
1. Inducer molecule in abundance in mixture (external)
1. Simple membrane model - One step diffusion
1. Advanced membrane model - Two step diffusion

## Transcription Model:

Consider the following simple transcription-only model where $G$ is a gene, $T$ is a transcript, and $S$ is the signaling molecule.

We can write the following reduced order dynamics:
1. $G \xrightarrow[]{\rho_{tx}(G, S)} G + T$;
\begin{align}
\rho_{tx}(G, S) = G K_{X}\frac{S^{2}}{K_{S}^{2}+S^{2}}
\\
\end{align}
Here, $S$ is the inducer signal that cooperatively activates the transcription of the gene $G$. Since, this is a positive activation of the gene by the inducer, we have a positive proportional Hill function.

1. $T \xrightarrow[]{\delta} \varnothing$; massaction kinetics at rate $\delta$.

```python
from bioscrape.types import Model
import numpy as np
import pylab as plt
#Create a list of species names (strings)
species = ["G", "mRNA_T", "S"]

#create a list of parameters in the form (param_name[string], param_val[number])
params = [("KX", 2), ("KI", 10), ("n", 2.0), ("delta", .1)]

#create reaction tuples in the form:
#(Inputs[string list], Outputs[string list], propensity_type[string], propensity_dict {propensity_param:model_param})
rxn1 = (["G"], ["G", "mRNA_T"], "proportionalhillpositive", {"d":"G", "s1":"S", "k":"KX", "K":"KI", "n":"n"})

# Or alternatively, use general propensity (simulation will be slower):
# rxn2 = (["T"], ["T", "X"], "general", {"rate":"ktl * T/(KR + T)"})

#Notice that parameters can also take numerical values instead of being named directly
rxn2 = (["mRNA_T"], [], "massaction", {"k":"delta"})

#Create a list of all reactions
rxns = [rxn1, rxn2]

#create an initial condition dictionary species not included in the dictionary will default to 0
x0 = {"G":1, "S":0, "mRNA_T":0}

#Instaniate the Model object
M = Model(species = species, parameters = params, reactions = rxns, initial_condition_dict = x0)
_ = M.write_sbml_model('transcription_SBML_model.xml')
print('Model written to a SBML file successfully')
```

    Model written to a SBML file successfully

## Translation model:

1. $T \xrightarrow[]{\rho_{tl}(T)} T+X$;
\begin{align}
\rho_{tl}(T) = K_{TR} \frac{T}{K_{R} + T}
\\
\end{align}
Here $X$ is the protein species.
The lumped parameters $K_{TR}$ and $K_R$ model effects due to ribosome saturation. This is the similar Hill function as derived in the enzymatic reaction example.

1. $X \xrightarrow[]{\delta} \varnothing$; massaction kinetics at rate $\delta$.

```python
from bioscrape.types import Model
import numpy as np
import pylab as plt
#Create a list of species names (strings)
species = ["T", "X"]

#create a list of parameters in the form (param_name[string], param_val[number])
params = [("KTR", 5), ("n", 2.0), ("KR", 20), ("delta", .1)]

#create reaction tuples in the form:
#(Inputs[string list], Outputs[string list], propensity_type[string], propensity_dict {propensity_param:model_param})
rxn1 = (["T"], ["T", "X"], "hillpositive", {"s1":"T", "k":"KTR", "K":"KR", "n":1})

#Notice that parameters can also take numerical values instead of being named directly
rxn2 = (["X"], [], "massaction", {"k":"delta"})

#Create a list of all reactions
rxns = [rxn1, rxn2]
x0 = {"X":0,"T":0}
#Instaniate the Model object
M = Model(species = species, parameters = params, reactions = rxns, initial_condition_dict = x0)
_ = M.write_sbml_model('translation_SBML_model.xml')
print('Model written to a SBML file successfully')
```

    Model written to a SBML file successfully

## Signal induction model:

1. $\varnothing \xrightarrow[]{\rho(I)} S$;
\begin{align}
\rho(S) = K_{0} \frac{I^2}{K_{I} + I^2}
\\
\end{align}
Here $S$ is the signal produced on induction by an inducer $I$.
The lumped parameters $K_{0}$ and $K_S$ model effects of cooperative production of the signal by the inducer. This is the similar Hill function as derived in the enzymatic reaction example.

```python
from bioscrape.types import Model
import numpy as np
import pylab as plt
#Create a list of species names (strings)
species = ["I", "S"]

#create a list of parameters in the form (param_name[string], param_val[number])
params = [("KI", 25), ("n", 2.0), ("K0", 20)]

#create reaction tuples in the form:
#(Inputs[string list], Outputs[string list], propensity_type[string], propensity_dict {propensity_param:model_param})
rxn1 = ([], ["S"], "hillpositive", {"s1":"I", "k":"K0", "K":"KI", "n":"n"})

#Create a list of all reactions
rxns = [rxn1]
x0 = {"I":100, "S":0}
#Instaniate the Model object
M = Model(species = species, parameters = params, reactions = rxns,initial_condition_dict = x0)
_ = M.write_sbml_model('signal_in_mixture.xml')
print('Model written to a SBML file successfully')
```

    Model written to a SBML file successfully

## Inducer reservoir in mixture:

A simple model with only one species - "IPTG" - the inducer. This species is present in abundance in the "external" compartment in this model. Hence, this model represents the exterior of a cell system.

```python
##### Creating a IPTG reservoir model####

from subsbml import createBasicSubsystem, SimpleModel
import libsbml

IPTG_ss, IPTG_model = createBasicSubsystem('reservoir')

per_second = IPTG_model.createNewUnitDefinition(uId = 'per_second',uKind = libsbml.UNIT_KIND_SECOND,
                                                uExponent = -1)
substance = IPTG_model.createNewUnitDefinition('substance',libsbml.UNIT_KIND_DIMENSIONLESS, uExponent = 1)

IPTG_model.createNewSpecies('IPTG',sComp = 'reservoir', ListOfAmounts = 1e4,
                             sConstant = False, sSubstance = 'substance')

# Write to SBML file
_ = IPTG_ss.writeSBML('IPTG_reservoir.xml')
print('Model written to a SBML file successfully')
```

    Model written to a SBML file successfully

## IPTG simple membrane model

IPTG_internal $\leftrightarrow$ IPTG_external

```python
##### Creating a basic membrane model for IPTG transport####

from subsbml import createNewSubsystem, SimpleModel, SimpleReaction

IPTG_mb_ss = createNewSubsystem()
model = IPTG_mb_ss.createNewModel('IPTG_membrane','second','mole','substance')

IPTG_mb = SimpleModel(model)

per_second = IPTG_mb.createNewUnitDefinition('per_second',libsbml.UNIT_KIND_SECOND,-1,0,1)
substance = IPTG_mb.createNewUnitDefinition('substance',libsbml.UNIT_KIND_DIMENSIONLESS, 1, 0, 1)

################# Two important lines of code when creating membrane #########################
IPTG_mb.createNewCompartment('internal','internal',2e-6,'litre',True)
IPTG_mb.createNewCompartment('external','external',1e-6,'litre',True)
##############################################################################################

IPTG_mb.createNewSpecies( 'IPTG','external',0,False,'substance')
IPTG_mb.createNewSpecies( 'IPTG','internal',0,False,'substance')

IPTG_mb.createNewParameter( 'kb',0.01,False,'per_second')
IPTG_mb.createNewParameter( 'kd',0.2,False,'per_second')

# model = IPTG_mb_ss.getSBMLDocument().getModel()
r1_sbml = model.createReaction()
r1 = SimpleReaction(r1_sbml)
r1.setId('r1')
r1.setReversible(True)
r1.createNewReactant('IPTG',False,1)
r1.createNewProduct('IPTG_1', False, 1)
r1.createRate('kb * IPTG - kd * IPTG_1')

# Write to SBML file
_ = IPTG_mb_ss.writeSBML('membrane_IPTG.xml')
print('Model written to a SBML file successfully')
```

    Model written to a SBML file successfully

## IPTG advanced membrane model

1. IPTG_external + transport_protein $\leftrightarrow$ IPTG_transport_protein
1. IPTG_transport_protein $\leftrightarrow$ IPTG_internal

```python
##### Creating a advanced membrane model for IPTG transport####

from subsbml import createNewSubsystem, SimpleModel, SimpleReaction

IPTG_mb_ss = createNewSubsystem()
model = IPTG_mb_ss.createNewModel('IPTG_membrane','second','mole','substance')

IPTG_mb = SimpleModel(model)

per_second = IPTG_mb.createNewUnitDefinition('per_second',libsbml.UNIT_KIND_SECOND,-1,0,1)
substance = IPTG_mb.createNewUnitDefinition('substance',libsbml.UNIT_KIND_DIMENSIONLESS, 1, 0, 1)

################# Two important lines of code when creating membrane #########################
IPTG_mb.createNewCompartment('internal','internal',1e-6,'litre',True)
IPTG_mb.createNewCompartment('external','external',1e-6,'litre',True)
##############################################################################################

IPTG_mb.createNewSpecies( 'IPTG','external',0,False,'substance')
IPTG_mb.createNewSpecies( 'IPTG','internal',0,False,'substance')
IPTG_mb.createNewSpecies( 'transport_protein','internal',1,False,'substance')
IPTG_mb.createNewSpecies( 'IPTG_transport_protein','internal',0,False,'substance')

IPTG_mb.createNewParameter( 'kb',0.01,False,'per_second')
IPTG_mb.createNewParameter( 'kd',0.2,False,'per_second')

model = IPTG_mb_ss.getSBMLDocument().getModel()

r1 = SimpleReaction(model.createReaction())
r1.setId('r1')
r1.setReversible(True)
r1.createNewReactant('IPTG',False,1)
r1.createNewReactant('transport_protein', False, 1)
r1.createNewProduct('IPTG_transport_protein', False, 1)
r1.createRate('kb * IPTG * transport_protein - kd * IPTG_transport_protein')

r2 = SimpleReaction(model.createReaction())
r2.setId('r2')
r2.setReversible(True)
r2.createNewReactant('IPTG_transport_protein',False,1)
r2.createNewProduct('IPTG_1',False,1)
r2.createNewProduct('transport_protein',False,1)
r2.createRate('kd * IPTG_transport_protein - kb * IPTG_1 * transport_protein')

# Write to SBML file
_ = IPTG_mb_ss.writeSBML('membrane_IPTG_detailed.xml')
print('Model written to a SBML file successfully')
```

    Model written to a SBML file successfully

```python

```

examples/Generate models (Lecture 4 BE240).ipynb


# Bistatic RCS generated by dielectric spheres

In this notebook, we consider the bistatic radar cross section generated from plane wave scattering by an array of dielectric spheres $\Omega_j$, each with its individual permittivity $\epsilon_j$ and permeability $\mu_j$.

We denote by $\epsilon_{0}$ and $\mu_0$ the electric permittivity and magnetic permeability in vacuum, and by $\epsilon_{r, j} = \frac{\epsilon_j}{\epsilon_0}$ and $\mu_{r, j} = \frac{\mu_j}{\mu_0}$ the relative permittivity and relative permeability for each dielectric object. Denote by $\mathbf{E}^\text{s}$, $\mathbf{H}^\text{s}$ the scattered electric and magnetic field in the exterior of the scatterers. Moreover, let $\mathbf{E}$ and $\mathbf{H}$ be the total exterior fields. Correspondingly, we denote the interior fields in the $j$th obstacle by $\mathbf{E}_j$ and $\mathbf{H}_j$.

For a given medium, the Maxwell equations are given by
\begin{align}
\nabla\times \mathbf{E} &= \mathrm{i}\omega\mu \mathbf{H},\nonumber\\
\nabla\times \mathbf{H} &= -\mathrm{i}\omega\epsilon \mathbf{E}.\nonumber
\end{align}
We define the wavenumber $k$ by $k=\omega\sqrt{\mu\epsilon}$.

The electric field equations for the dielectric scattering problem now take the form
\begin{align}
\nabla\times\nabla\times \mathbf{E}^{+}(x) - k_0^2\mathbf{E}^{+}(x) &= 0,~x\in\Omega^{+}\nonumber\\
\nabla\times\nabla \times \mathbf{E}_{j}(x) - k_j^2\mathbf{E}_{j}(x) &= 0,~x\in\Omega_j\nonumber\\
\end{align}
with $k_0 = \omega\sqrt{\epsilon_0\mu_0}$ and $k_j = k_0\sqrt{\epsilon_{r,j}\mu_{r,j}}$. We define the normal trace  $\gamma_t\mathbf{E} = \mathbf{E}\times n$ and the Neumann trace $\gamma_N = \frac{1}{ik}\gamma_t\left[\nabla\times \mathbf{E}\right]$. The exterior field $\mathbf{E}^{s}$ is the sum of the incident wave $\mathbf{E}^{inc}$ and the scattered wave $\mathbf{E}^{s}$.

Now let $ V := \begin{bmatrix}\gamma_t \mathbf{E}\\ \rho\gamma_N\mathbf{E}\end{bmatrix}$, where $\rho := \sqrt{\epsilon_r}/\sqrt{\mu_r}$. With $V_j^{+}$ being the exterior trace data (using $\epsilon_0$ and $\mu_0$) on the boundary the $j$th scatterer, and $V_j^{-}$ the corresponding interior trace data (using $\epsilon_{r, j}$ and $\mu_{r, j}$) the correct boundary condition for this scattering problem is
$$
V_j^{-} = V_j^{+}.
$$
It is immediately visible that this condition satisfies $\gamma_t\mathbf{E}_j^{-} = \gamma_t\mathbf{E}_j^{+}$. Furthermore, this condition implies that $\rho^{-}\gamma_N \mathbf{E}_j^{-} = \rho^{+}\gamma_N \mathbf{E}_j^{+}$, which is equivalent to $\mathbf{H}_j^{-} = \mathbf{H}_j^{+}$. We now define the scaled multitrace operator $A = \begin{bmatrix} M & \rho^{-1}T\\ -\rho T & M\end{bmatrix}$ with $M$ the magnetic field operator and $T$ the electric field operator. For this scaled multitrace operator it holds that
$$
\begin{bmatrix}\frac{1}{2}I - A^{+}_{j}\end{bmatrix}V_j^{+} - \sum_{i\neq j}A_{j, i}V_i^{+} = V_j^{s}.
$$
Here, $A_j^{+}$ is the multitrace operator in vacuum ($\rho = 1$) and $A_{j, i}$ is the multitrace operator with domain on the boundary of the $i$th scatterer and range on the boundary of the $j$th scatterer. This accounts for the influence of the other scatterers in a multi-scattering configuration. The data $V_j^{s}$ is the trace data of the scattered wave on the $j$th scatterer. Further, on the boundary of $\Omega_j$ we have for the interior trace data that
$$
\begin{bmatrix}\frac{1}{2}I + A_j^{-}\end{bmatrix}V_j^{-} = V_j^{-}
$$
with $A_j^{-}$ being the scaled multitrace operator for the interior problem and $V_j^{-}$ the corresponding interior trace data. Using the boundary conditions $V_j^{-} = V_j^{+}$ and the fact that $V_j^{inc} = V_j^{+} + V_j^{s}$ for the trace data $V_j^{inc}$ of the incident wave the interior and exterior equations combined result in
$$
\begin{bmatrix}A_j^{-} + A_j^{+}\end{bmatrix}V_j^{+} + \sum_{i\neq j}A_{j, i}V_i^{+} = V_j^{inc}.
$$
Taking all corresponding equations over the $N$ scatterers we arrive at a complete system of equations for the unknown data $V_j$ on each scatterer.

We start with the usual imports and enable console logging.

```python
import bempp.api
import numpy as np

bempp.api.enable_console_logging()
# bempp.api.pool.create_device_pool("AMD")
```

    <StreamHandler stderr (INFO)>

For this notebook we will use two spheres of radius 0.4, centered at $-1$ and $1$ on the x-axis.

```python
sphere0 = bempp.api.shapes.sphere(r=.4, origin=(-1., 0, 0), h=.2)
sphere1 = bempp.api.shapes.sphere(r=.4, origin=(1., 0, 0), h=.2)

```

    bempp:HOST:INFO: Created grid with id 6adb0780-2f24-4df1-a879-c36c71913dbd. Elements: 224. Edges: 336. Vertices: 114
    bempp:HOST:INFO: Created grid with id c002fa49-b687-4702-bfc8-c9c3b7afc936. Elements: 224. Edges: 336. Vertices: 114

We now define the associated constants and material parameters. We have chosen the material parameters for Teflon. The overall simulation will be running at a frequency of 300Mhz.

```python
frequency = 300E6 # 300Mhz

vacuum_permittivity = 8.854187817E-12
vacuum_permeability = 4 * np.pi * 1E-7

eps_r = 2.1
mu_r = 1.0

k_ext = 2 * np.pi * frequency * np.sqrt(vacuum_permittivity * vacuum_permeability)
k_int = k_ext * np.sqrt(eps_r * mu_r)
```

We now assemble the incident wave field. To that affect we choose a z-polarized plane wave travelling at an inciden angle theta in the (x,y) plane.

```python
theta = np.pi / 4 # Incident wave travelling at a 45 degree angle
direction = np.array([np.cos(theta), np.sin(theta), 0])
polarization = np.array([0, 0, 1.0])

def plane_wave(point):
    return polarization * np.exp(1j * k_ext * np.dot(point, direction))

@bempp.api.complex_callable
def tangential_trace(point, n, domain_index, result):
    value = polarization * np.exp(1j * k_ext * np.dot(point, direction))
    result[:] =  np.cross(value, n)

@bempp.api.complex_callable
def neumann_trace(point, n, domain_index, result):
    value = np.cross(direction, polarization) * 1j * k_ext * np.exp(1j * k_ext * np.dot(point, direction))
    result[:] =  1./ (1j * k_ext) * np.cross(value, n)

```

We now create all the interior and exterior multitrace operators and setup the left-hand side blocked operator.

```python
from bempp.api.operators.boundary.maxwell import multitrace_operator
from bempp.api.operators.boundary.sparse import multitrace_identity

A0_int = multitrace_operator(
    sphere0, k_int, epsilon_r=eps_r, mu_r=mu_r, space_type='all_rwg')
A1_int = multitrace_operator(
    sphere1, k_int, epsilon_r=eps_r, mu_r=mu_r, space_type='all_rwg')
A0_ext = multitrace_operator(
    sphere0, k_ext, space_type='all_rwg')
A1_ext = multitrace_operator(
    sphere1, k_ext, space_type='all_rwg')
A01 = multitrace_operator(sphere1, k_ext, target=sphere0, space_type='all_rwg')
A10 = multitrace_operator(sphere0, k_ext, target=sphere1, space_type='all_rwg')

A = bempp.api.GeneralizedBlockedOperator([[A0_int + A0_ext, A01],
                                          [A10, A1_int + A1_ext]])
```

    bempp:HOST:INFO: Created grid with id 082f5630-7e93-4771-8705-fa48cc9c56ca. Elements: 1344. Edges: 2016. Vertices: 674
    bempp:HOST:INFO: Created grid with id 854d8621-4818-4fe1-a162-dd4fc6cb67e4. Elements: 1344. Edges: 2016. Vertices: 674

The following code assembles the right-hand sides.

```python
rhs = [bempp.api.GridFunction(space=A.range_spaces[0], dual_space=A.dual_to_range_spaces[0], fun=tangential_trace),
       bempp.api.GridFunction(space=A.range_spaces[1], dual_space=A.dual_to_range_spaces[1], fun=neumann_trace),
       bempp.api.GridFunction(space=A.range_spaces[2], dual_space=A.dual_to_range_spaces[2], fun=tangential_trace),
       bempp.api.GridFunction(space=A.range_spaces[3], dual_space=A.dual_to_range_spaces[3], fun=neumann_trace)]
```

We now have everything in place to solve the resulting scattering problem. Here, we solve directly with LU decomposition. For larger problems iterative solvers with preconditioning is appropriate.

```python
bempp.api.enable_console_logging()
sol = bempp.api.linalg.lu(A, rhs)
```

    bempp:HOST:INFO: OpenCL Device set to: pthread-Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

In the following we visualize the near-field. For this we evaluate the interior and exterior field using the Stratton-Chu representation formula.

```python
# Number of points in the x-direction
nx = 200

# Number of points in the y-direction
ny = 200

# Generate the evaluation points with numpy
x, y, z = np.mgrid[-2:2:nx * 1j, -2:2:ny * 1j, 0:0:1j]
points = np.vstack((x.ravel(), y.ravel(), z.ravel()))

# Compute interior and exterior indices
all_indices = np.ones(points.shape[1], dtype='uint32')

interior_indices0 = np.sum((points - np.array([[-1, 0, 0]]).T)**2, axis=0) < .4**2
interior_indices1 = np.sum((points - np.array([[1, 0, 0]]).T)**2, axis=0) < .4**2
exterior_indices = ~interior_indices0 & ~interior_indices1

ext_points = points[:, exterior_indices]
int_points0 = points[:, interior_indices0]
int_points1 = points[:, interior_indices1]

mpot0_int = bempp.api.operators.potential.maxwell.magnetic_field(sol[0].space, int_points0, k_int)
epot0_int = bempp.api.operators.potential.maxwell.electric_field(sol[1].space, int_points0, k_int)
mpot0_ext = bempp.api.operators.potential.maxwell.magnetic_field(sol[0].space, ext_points, k_ext)
epot0_ext = bempp.api.operators.potential.maxwell.electric_field(sol[1].space, ext_points, k_ext)

mpot1_int = bempp.api.operators.potential.maxwell.magnetic_field(sol[2].space, int_points1, k_int)
epot1_int = bempp.api.operators.potential.maxwell.electric_field(sol[3].space, int_points1, k_int)
mpot1_ext = bempp.api.operators.potential.maxwell.magnetic_field(sol[2].space, ext_points, k_ext)
epot1_ext = bempp.api.operators.potential.maxwell.electric_field(sol[3].space, ext_points, k_ext)

exterior_values = -epot0_ext * sol[1] - mpot0_ext * sol[0]
exterior_values += -epot1_ext * sol[3] - mpot1_ext * sol[2]
interior_values0 = (np.sqrt(mu_r) / np.sqrt(eps_r) * epot0_int * sol[1] + mpot0_int * sol[0])
interior_values1 = (np.sqrt(mu_r) / np.sqrt(eps_r) * epot1_int * sol[3] + mpot1_int * sol[2])
```

Now that we have computed the interior and exterior values we can plot them.

```python
from matplotlib import pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable
from matplotlib.patches import Circle
plt.rcParams['figure.figsize'] = (20, 16) # Increase the figure size in the notebook

# First compute the scattered field
scattered_field = np.empty((3, points.shape[1]), dtype='complex128')
scattered_field[:, :] = np.nan
scattered_field[:, exterior_indices] = exterior_values

# Now compute the total field
total_field = np.empty((3, points.shape[1]), dtype='complex128')

for ext_ind in np.arange(points.shape[1])[exterior_indices]:
    total_field[:, ext_ind] = scattered_field[:, ext_ind] + plane_wave(points[:, ext_ind])
total_field[:, interior_indices0] = interior_values0
total_field[:, interior_indices1] = interior_values1

# Compute the squared field density
squared_scattered_field = np.sum(np.abs(scattered_field)**2, axis=0)
squared_total_field = np.sum(np.abs(total_field)**2, axis=0)

# Show the resulting images
scattered_image = squared_scattered_field.reshape(nx, ny).T
total_image = squared_total_field.reshape(nx, ny).T
fig, axes = plt.subplots(1, 2, sharex=True, sharey=True)

f0 = axes[0].imshow(scattered_image, origin='lower', cmap='magma',
                    extent=[-2, 2, -2, 2], vmin=0, vmax=5)
axes[0].add_patch(
    Circle((-1, 0), .4, facecolor='None', edgecolor='k', lw=5)
)
axes[0].add_patch(
    Circle((1, 0), .4, facecolor='None', edgecolor='k', lw=5)
)

axes[0].set_title("Squared scattered field strength")
divider = make_axes_locatable(axes[0])
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(f0, cax=cax)

f1 = axes[1].imshow(total_image, origin='lower', cmap='magma',
                    extent=[-2, 2, -2, 2], vmin=0, vmax=5)
axes[1].add_patch(
    Circle((-1, 0), .4, facecolor='None', edgecolor='k', lw=5)
)
axes[1].add_patch(
    Circle((1, 0), .4, facecolor='None', edgecolor='k', lw=5)
)

axes[1].set_title("Squared total field strength")
divider = make_axes_locatable(axes[1])
cax = divider.append_axes("right", size="5%", pad=0.05)
fig.colorbar(f1, cax=cax)

plt.show()
```

Plotting the bistatic RCS is also easy. The code below demonstrates this.

```python
number_of_angles = 400
angles = np.pi * np.linspace(0, 1, number_of_angles)
unit_points = np.array([-np.cos(angles), -np.sin(angles), np.zeros(number_of_angles)])

far_field = np.zeros((3, number_of_angles), dtype='complex128')

for i in range(2):
    electric_far = bempp.api.operators.far_field.maxwell.electric_field(sol[2 * i + 1].space, unit_points, k_ext)
    magnetic_far = bempp.api.operators.far_field.maxwell.magnetic_field(sol[2 * i].space, unit_points, k_ext)
    far_field += -electric_far * sol[2 * i + 1] - magnetic_far * sol[2 * i]

plt.rcParams['figure.figsize'] = (10, 8) # Resize the figure

cross_section = 10 * np.log10(4 * np.pi * np.sum(np.abs(far_field)**2, axis=0))
plt.plot(angles * 180 / np.pi, cross_section)
plt.title("Scattering Cross Section [dB]")
_ = plt.xlabel('Angle (Degrees)')
```


## Problem Set 13

First the exercises:
* The exercise did not specify a loss function, but let us assume the standard average cross entropy function. To simplify things let us not use biases. Then the logistic regression model involves the following functions:
\begin{align}
f(x,W_1) &= xW_1 \\
g(f) & = e^f \textrm{(pointwise)} \\
p(g) &= \sum \pi_i g \textrm{ (sum across all vectors in sample)} \\
h(g, p) &= g/p \textrm{ (applied sample wise)} \\
logp(p) &= log(p+epsilon) \textrm{ (epsilon is some small constant to avoid underflow)}\\
sub(f,logp) &= f-logp \textrm{ (broadcast) } \\
pl(y,sub) &= y \cdot sub \textrm{ (Hadamard product)} \\
l(p1) &= mean(pl) \textrm{ (samplewise)}\\
al(pl) &=mean(pl)
\end{align}

Here $h$ yields the predictions, $l$ yields the samplewise cross entropy, $al$ yields the average cross entropy.

Now we have to figure out the weight update rule. For this we need to calculate $\frac{dal}{dW_1}$ by the chain rule.

Here is a simplified form
\begin{align}
 \frac{dal}{dW_1} &= \frac{dal}{dl}(\frac{dl}{dsub} \left(\frac{dsub}{df}\frac{df}{dW_1}+\frac{dsub}{dlogp}\frac{dlogp}{dW_1}\right)\\
 \frac{df}{dW_1}_{(s,a,b,c,d)} &= x_{(s,a,c)}\delta(b==d) \\
 \frac{dlogp}{dW_1} &=\frac{dlogp}{dp}\frac{dp}{dg}\frac{dg}{df}\frac{df}{dW_1}\\
 \frac{dg}{df}_{(s,a,b,c)} &= e^f \delta(s==b)\delta(a==c) \\
 \frac{dp}{dg}_{(s,a,b)} &= \delta(s==b)
\end{align}

### Python Lab

Now let us load our standard libraries.

```python
import numpy as np
import pandas as pd
import keras
from keras.datasets import mnist
```

    Using TensorFlow backend.

Let us load up a sample dataset.

```python
(x_train, y_train), (x_test, y_test) = mnist.load_data()
```

```python
x_train.shape
```

    (60000, 28, 28)

```python
import matplotlib.pyplot as plt
```

```python
%matplotlib inline
```

```python
randix = np.random.randint(0,60000)
plt.imshow(x_train[randix])
print("Label is {}.".format(y_train[randix]))
```

```python
x_train_f = x_train.reshape(60000,-1)
```

```python
x_train_f.shape
```

    (60000, 784)

```python
x_test_f = x_test.reshape(-1, 28**2)
```

```python
x_test_f.shape
```

    (10000, 784)

```python
from sklearn.preprocessing import StandardScaler
```

```python
scaler = StandardScaler()
```

```python
scaler.fit(x_train_f)
```

    /Users/justin/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype uint8 was converted to float64 by StandardScaler.
      warnings.warn(msg, _DataConversionWarning)

    StandardScaler(copy=True, with_mean=True, with_std=True)

```python
train_x_tr = scaler.transform(x_train_f)
```

    /Users/justin/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype uint8 was converted to float64 by StandardScaler.
      warnings.warn(msg, _DataConversionWarning)

```python
test_x_tr = scaler.transform(x_test_f)
```

    /Users/justin/anaconda/envs/py36/lib/python3.6/site-packages/sklearn/utils/validation.py:429: DataConversionWarning: Data with input dtype uint8 was converted to float64 by StandardScaler.
      warnings.warn(msg, _DataConversionWarning)

```python
from sklearn.preprocessing import OneHotEncoder as OHE
```

```python
ohe = OHE(sparse = False)
y_train_ohe = ohe.fit_transform(y_train.reshape(-1,1))
y_test_ohe = ohe.fit_transform(y_test.reshape(-1,1))
```

```python
np.argmax(y_train_ohe[randix]) == y_train[randix]
```

    True

```python
from keras.models import Sequential
from keras.layers import Dense, Dropout, BatchNormalization
```

```python
logreg = Sequential()
```

```python
logreg.add(Dense(10,activation='softmax',input_dim = (28**2)))
```

```python
from keras.optimizers import Adam, SGD
```

```python
logreg.compile(optimizer=Adam(),
             loss='categorical_crossentropy',
             metrics=['accuracy'])
```

```python
history=logreg.fit(train_x_tr, y_train_ohe, validation_split=0.05,
                 epochs=25, batch_size=64)
```

    Train on 57000 samples, validate on 3000 samples
    Epoch 1/25
    57000/57000 [==============================] - 1s - loss: 0.4622 - acc: 0.8704 - val_loss: 0.2408 - val_acc: 0.9353
    Epoch 2/25
    57000/57000 [==============================] - 1s - loss: 0.3149 - acc: 0.9145 - val_loss: 0.2305 - val_acc: 0.9420
    Epoch 3/25
    57000/57000 [==============================] - 1s - loss: 0.2950 - acc: 0.9206 - val_loss: 0.2324 - val_acc: 0.9420
    Epoch 4/25
    57000/57000 [==============================] - 1s - loss: 0.2844 - acc: 0.9229 - val_loss: 0.2295 - val_acc: 0.9430
    Epoch 5/25
    57000/57000 [==============================] - 1s - loss: 0.2779 - acc: 0.9256 - val_loss: 0.2262 - val_acc: 0.9433
    Epoch 6/25
    57000/57000 [==============================] - 1s - loss: 0.2723 - acc: 0.9274 - val_loss: 0.2285 - val_acc: 0.9450
    Epoch 7/25
    57000/57000 [==============================] - 1s - loss: 0.2681 - acc: 0.9283 - val_loss: 0.2306 - val_acc: 0.9447
    Epoch 8/25
    57000/57000 [==============================] - 1s - loss: 0.2664 - acc: 0.9278 - val_loss: 0.2360 - val_acc: 0.9460
    Epoch 9/25
    57000/57000 [==============================] - 1s - loss: 0.2631 - acc: 0.9295 - val_loss: 0.2377 - val_acc: 0.9410
    Epoch 10/25
    57000/57000 [==============================] - 1s - loss: 0.2615 - acc: 0.9296 - val_loss: 0.2355 - val_acc: 0.9437
    Epoch 11/25
    57000/57000 [==============================] - 1s - loss: 0.2597 - acc: 0.9300 - val_loss: 0.2457 - val_acc: 0.9430
    Epoch 12/25
    57000/57000 [==============================] - 1s - loss: 0.2585 - acc: 0.9306 - val_loss: 0.2405 - val_acc: 0.9447
    Epoch 13/25
    57000/57000 [==============================] - 1s - loss: 0.2570 - acc: 0.9306 - val_loss: 0.2445 - val_acc: 0.9390
    Epoch 14/25
    57000/57000 [==============================] - 1s - loss: 0.2554 - acc: 0.9315 - val_loss: 0.2376 - val_acc: 0.9453
    Epoch 15/25
    57000/57000 [==============================] - 1s - loss: 0.2544 - acc: 0.9308 - val_loss: 0.2465 - val_acc: 0.9440
    Epoch 16/25
    57000/57000 [==============================] - 1s - loss: 0.2534 - acc: 0.9323 - val_loss: 0.2393 - val_acc: 0.9410
    Epoch 17/25
    57000/57000 [==============================] - 1s - loss: 0.2527 - acc: 0.9317 - val_loss: 0.2495 - val_acc: 0.9393
    Epoch 18/25
    57000/57000 [==============================] - 1s - loss: 0.2524 - acc: 0.9324 - val_loss: 0.2471 - val_acc: 0.9403
    Epoch 19/25
    57000/57000 [==============================] - 1s - loss: 0.2512 - acc: 0.9321 - val_loss: 0.2505 - val_acc: 0.9407
    Epoch 20/25
    57000/57000 [==============================] - 1s - loss: 0.2505 - acc: 0.9318 - val_loss: 0.2475 - val_acc: 0.9413
    Epoch 21/25
    57000/57000 [==============================] - 1s - loss: 0.2497 - acc: 0.9334 - val_loss: 0.2487 - val_acc: 0.9433
    Epoch 22/25
    57000/57000 [==============================] - 1s - loss: 0.2489 - acc: 0.9325 - val_loss: 0.2483 - val_acc: 0.9433
    Epoch 23/25
    57000/57000 [==============================] - 1s - loss: 0.2487 - acc: 0.9328 - val_loss: 0.2518 - val_acc: 0.9420
    Epoch 24/25
    57000/57000 [==============================] - 1s - loss: 0.2482 - acc: 0.9325 - val_loss: 0.2574 - val_acc: 0.9400
    Epoch 25/25
    57000/57000 [==============================] - 1s - loss: 0.2483 - acc: 0.9329 - val_loss: 0.2508 - val_acc: 0.9433

```python
plt.plot(history.history['acc'])
```

```python
plt.plot(history.history['val_acc'])
```

```python
deep = Sequential()
```

```python
deep.add(Dense(256, activation = 'relu', input_dim = (28**2)))
```

```python
deep.add(Dropout(0.2))
```

```python
deep.add(Dense(128,activation = 'relu'))
```

```python
deep.add(Dropout(0.2))
```

```python
deep.add(Dense(64,activation = 'relu'))
```

```python
deep.add(Dropout(0.2))
```

```python
deep.add(Dense(10,activation = 'softmax'))
```

```python
deep.compile(optimizer=Adam(),
             loss='categorical_crossentropy',
             metrics=['accuracy'])
```

```python
history2=deep.fit(train_x_tr, y_train_ohe, validation_split=0.05,
                 epochs=25, batch_size=64)
```

    Train on 57000 samples, validate on 3000 samples
    Epoch 1/25
    57000/57000 [==============================] - 4s - loss: 0.3828 - acc: 0.8863 - val_loss: 0.1097 - val_acc: 0.9683
    Epoch 2/25
    57000/57000 [==============================] - 4s - loss: 0.1952 - acc: 0.9437 - val_loss: 0.0968 - val_acc: 0.9753
    Epoch 3/25
    57000/57000 [==============================] - 4s - loss: 0.1520 - acc: 0.9555 - val_loss: 0.0861 - val_acc: 0.9757
    Epoch 4/25
    57000/57000 [==============================] - 4s - loss: 0.1281 - acc: 0.9632 - val_loss: 0.0875 - val_acc: 0.9773
    Epoch 5/25
    57000/57000 [==============================] - 4s - loss: 0.1121 - acc: 0.9671 - val_loss: 0.0897 - val_acc: 0.9777
    Epoch 6/25
    57000/57000 [==============================] - 4s - loss: 0.0995 - acc: 0.9714 - val_loss: 0.0950 - val_acc: 0.9783
    Epoch 7/25
    57000/57000 [==============================] - 4s - loss: 0.0922 - acc: 0.9731 - val_loss: 0.0810 - val_acc: 0.9800
    Epoch 8/25
    57000/57000 [==============================] - 4s - loss: 0.0840 - acc: 0.9749 - val_loss: 0.0852 - val_acc: 0.9820
    Epoch 9/25
    57000/57000 [==============================] - 4s - loss: 0.0765 - acc: 0.9774 - val_loss: 0.0766 - val_acc: 0.9840
    Epoch 10/25
    57000/57000 [==============================] - 4s - loss: 0.0769 - acc: 0.9783 - val_loss: 0.0831 - val_acc: 0.9810
    Epoch 11/25
    57000/57000 [==============================] - 4s - loss: 0.0690 - acc: 0.9798 - val_loss: 0.0845 - val_acc: 0.9810
    Epoch 12/25
    57000/57000 [==============================] - 4s - loss: 0.0604 - acc: 0.9822 - val_loss: 0.0982 - val_acc: 0.9817
    Epoch 13/25
    57000/57000 [==============================] - 4s - loss: 0.0595 - acc: 0.9822 - val_loss: 0.0977 - val_acc: 0.9807
    Epoch 14/25
    57000/57000 [==============================] - 4s - loss: 0.0553 - acc: 0.9835 - val_loss: 0.0810 - val_acc: 0.9823
    Epoch 15/25
    57000/57000 [==============================] - 4s - loss: 0.0551 - acc: 0.9845 - val_loss: 0.0899 - val_acc: 0.9807
    Epoch 16/25
    57000/57000 [==============================] - 3s - loss: 0.0532 - acc: 0.9840 - val_loss: 0.0794 - val_acc: 0.9817
    Epoch 17/25
    57000/57000 [==============================] - 3s - loss: 0.0500 - acc: 0.9856 - val_loss: 0.0840 - val_acc: 0.9787
    Epoch 18/25
    57000/57000 [==============================] - 3s - loss: 0.0480 - acc: 0.9860 - val_loss: 0.0849 - val_acc: 0.9827
    Epoch 19/25
    57000/57000 [==============================] - 3s - loss: 0.0456 - acc: 0.9863 - val_loss: 0.0875 - val_acc: 0.9817
    Epoch 20/25
    57000/57000 [==============================] - 3s - loss: 0.0442 - acc: 0.9873 - val_loss: 0.1048 - val_acc: 0.9803
    Epoch 21/25
    57000/57000 [==============================] - 3s - loss: 0.0446 - acc: 0.9875 - val_loss: 0.0860 - val_acc: 0.9837
    Epoch 22/25
    57000/57000 [==============================] - 3s - loss: 0.0422 - acc: 0.9878 - val_loss: 0.0890 - val_acc: 0.9830
    Epoch 23/25
    57000/57000 [==============================] - 3s - loss: 0.0409 - acc: 0.9877 - val_loss: 0.0892 - val_acc: 0.9823
    Epoch 24/25
    57000/57000 [==============================] - 4s - loss: 0.0394 - acc: 0.9880 - val_loss: 0.0957 - val_acc: 0.9843
    Epoch 25/25
    57000/57000 [==============================] - 4s - loss: 0.0381 - acc: 0.9890 - val_loss: 0.0872 - val_acc: 0.9837

```python
from keras.layers import Conv2D, GlobalAveragePooling2D
```

```python
train_x_re = train_x_tr.reshape(-1,28,28,1)
```

```python
CNN = Sequential()
```

```python
CNN.add(Conv2D(filters = 16, kernel_size = (3,3), padding = 'SAME',input_shape = (28,28,1), activation = 'relu'))
```

```python
CNN.add(Conv2D(filters = 32, kernel_size = (3,3), padding = 'SAME', activation = 'relu'))
```

```python
CNN.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'SAME', activation = 'relu'))
```

```python
CNN.add(GlobalAveragePooling2D())
```

```python
CNN.add(Dense(10,activation='softmax'))
```

```python
CNN.compile(optimizer=Adam(),
             loss='categorical_crossentropy',
             metrics=['accuracy'])
```

```python
history3=CNN.fit(train_x_re, y_train_ohe, validation_split=0.05,
                 epochs=25, batch_size=64)
```

    Train on 57000 samples, validate on 3000 samples
    Epoch 1/25
    57000/57000 [==============================] - 88s - loss: 1.4696 - acc: 0.4872 - val_loss: 0.8768 - val_acc: 0.7490
    Epoch 2/25
    57000/57000 [==============================] - 86s - loss: 0.8037 - acc: 0.7588 - val_loss: 0.5660 - val_acc: 0.8610
    Epoch 3/25
    57000/57000 [==============================] - 86s - loss: 0.5973 - acc: 0.8276 - val_loss: 0.5143 - val_acc: 0.8413
    Epoch 4/25
    57000/57000 [==============================] - 86s - loss: 0.4874 - acc: 0.8607 - val_loss: 0.3775 - val_acc: 0.8937
    Epoch 5/25
    57000/57000 [==============================] - 86s - loss: 0.4188 - acc: 0.8808 - val_loss: 0.3267 - val_acc: 0.9067
    Epoch 6/25
    57000/57000 [==============================] - 86s - loss: 0.3590 - acc: 0.8967 - val_loss: 0.3364 - val_acc: 0.8970
    Epoch 7/25
    57000/57000 [==============================] - 86s - loss: 0.3227 - acc: 0.9066 - val_loss: 0.2335 - val_acc: 0.9323
    Epoch 8/25
    57000/57000 [==============================] - 86s - loss: 0.2887 - acc: 0.9172 - val_loss: 0.2236 - val_acc: 0.9377
    Epoch 9/25
    57000/57000 [==============================] - 87s - loss: 0.2629 - acc: 0.9255 - val_loss: 0.2114 - val_acc: 0.9413
    Epoch 10/25
    57000/57000 [==============================] - 85s - loss: 0.2467 - acc: 0.9295 - val_loss: 0.1958 - val_acc: 0.9460
    Epoch 11/25
    57000/57000 [==============================] - 84s - loss: 0.2231 - acc: 0.9365 - val_loss: 0.1812 - val_acc: 0.9527
    Epoch 12/25
    57000/57000 [==============================] - 84s - loss: 0.2068 - acc: 0.9406 - val_loss: 0.1624 - val_acc: 0.9553
    Epoch 13/25
    57000/57000 [==============================] - 83s - loss: 0.1974 - acc: 0.9442 - val_loss: 0.1693 - val_acc: 0.9533
    Epoch 14/25
    57000/57000 [==============================] - 83s - loss: 0.1861 - acc: 0.9469 - val_loss: 0.1706 - val_acc: 0.9530
    Epoch 15/25
    57000/57000 [==============================] - 83s - loss: 0.1769 - acc: 0.9495 - val_loss: 0.1247 - val_acc: 0.9667
    Epoch 16/25
    57000/57000 [==============================] - 84s - loss: 0.1663 - acc: 0.9520 - val_loss: 0.1521 - val_acc: 0.9610
    Epoch 17/25
    57000/57000 [==============================] - 84s - loss: 0.1575 - acc: 0.9546 - val_loss: 0.1330 - val_acc: 0.9627
    Epoch 18/25
    57000/57000 [==============================] - 84s - loss: 0.1514 - acc: 0.9560 - val_loss: 0.1134 - val_acc: 0.9690
    Epoch 19/25
    57000/57000 [==============================] - 84s - loss: 0.1432 - acc: 0.9586 - val_loss: 0.1221 - val_acc: 0.9677
    Epoch 20/25
    57000/57000 [==============================] - 84s - loss: 0.1390 - acc: 0.9590 - val_loss: 0.1339 - val_acc: 0.9640
    Epoch 21/25
    57000/57000 [==============================] - 84s - loss: 0.1314 - acc: 0.9613 - val_loss: 0.1105 - val_acc: 0.9683
    Epoch 22/25
    57000/57000 [==============================] - 84s - loss: 0.1270 - acc: 0.9624 - val_loss: 0.0966 - val_acc: 0.9740
    Epoch 23/25
    57000/57000 [==============================] - 84s - loss: 0.1258 - acc: 0.9636 - val_loss: 0.1046 - val_acc: 0.9677
    Epoch 24/25
    57000/57000 [==============================] - 84s - loss: 0.1161 - acc: 0.9663 - val_loss: 0.0900 - val_acc: 0.9757
    Epoch 25/25
    57000/57000 [==============================] - 84s - loss: 0.1148 - acc: 0.9663 - val_loss: 0.0962 - val_acc: 0.9747


**Case 21-031**

```python
import numpy as np
import sympy as sp
sp.init_printing()
```

**Data and measurements**

```python
alpha_az = 107.056 # deg
alpha_el = 43.497 # deg
tau1_az = 109.238 # deg
tau1_el = 44.468 #deg
delta_pix = 288 # pixels between two stars
frame_rate = 29.97 # frames per second
t_traverse = 54 + 12/frame_rate - (36 + 25/frame_rate) # seconds
print("Time to traverse screen: ",t_traverse," seconds")
in_dist = 1536 # pixels from alpha tau
in_ang = 180- 44.26 # degrees
out_dist = 1093.9 # pixels
out_ang = 270+ 21.32 # degrees
```

    Time to traverse screen:  17.56623289956623  seconds

**Calculations**

```python
delta_az = (tau1_az - alpha_az)*np.pi/180
delta_el = (tau1_el - alpha_el)*np.pi/180
delta_ang = np.arccos(np.cos(delta_az)*np.cos(delta_el) +
                      (1-np.cos(delta_az))*np.sin(alpha_el*np.pi/180)*np.sin(tau1_el*np.pi/180)) # angle between two star
print("angle between stars: ",delta_ang*180/np.pi," deg")
obj_pix = 31 #pixels
obj_size = delta_ang/delta_pix*obj_pix # radians
print("Object Size in degrees: ",obj_size*180/np.pi)
#print("Object size at 500 meters: ",obj_size*500)
```

    angle between stars:  1.8459602308070742  deg
    Object Size in degrees:  0.19869710817715033

```python
in_ang_rad = in_ang*np.pi/180 # convert to radians
out_ang_rad = out_ang*np.pi/180
#print(np.array((np.cos(in_ang_rad),np.sin(in_ang_rad))))
twoD_in_vec = in_dist*np.array((np.cos(in_ang_rad),np.sin(in_ang_rad)))
twoD_out_vec = out_dist*np.array((np.cos(out_ang_rad),np.sin(out_ang_rad)))
diff_vec = twoD_out_vec - twoD_in_vec
#print(twoD_in_vec)
d_traverse = np.sqrt(np.dot(diff_vec,diff_vec)) # pixels
print("angle traversed = ",d_traverse*delta_ang/delta_pix*180/np.pi," degrees")
diam_per_sec = (d_traverse/obj_pix)/t_traverse # object speed in daimeters per second
print("time_to_traverse_frame: ",t_traverse," seconds")
print("average speed = ", diam_per_sec," diameters per second")
```

    angle traversed =  16.48614963530884  degrees
    time_to_traverse_frame:  17.56623289956623  seconds
    average speed =  4.7233383240896645  diameters per second

```python
print(twoD_in_vec)
```

    [-1100.05270288  1071.99815806]

```python
gnd_dist_meters = 7500

alt_estimate_meters = np.arctan(alpha_el*np.pi/180)*gnd_dist_meters
alt_estimate_feet = alt_estimate_meters/0.3048
range_meters = np.sqrt(gnd_dist_meters**2 + alt_estimate_meters**2)
print("total range to object: ",range_meters," meters")
print("approximate size ",obj_size*range_meters," meters")
print("approximate altitude: ",alt_estimate_feet," feet")
projected_speed_mps = (d_traverse*(delta_ang/delta_pix)/t_traverse)*range_meters
print("projected_speed: ",projected_speed_mps," meters per second")
print("projected_speed: ",(projected_speed_mps/0.3048)*5280/3600," miles per hour")
```

    total range to object:  8942.454729141951  meters
    approximate size  31.0117064430528  meters
    approximate altitude:  15977.891241505402  feet
    projected_speed:  146.47878153788966  meters per second
    projected_speed:  704.8410310222162  miles per hour

```python
print(d_traverse*(delta_ang/delta_pix)/t_traverse)
print(delta_ang/delta_pix)
```

    0.016380153545596382
    0.00011186834683492328


# Point Lander Model with Drag

```python
from sympy import *
init_printing()
```

```python
# State
x, y, vx, vy, m = symbols('x y vx vy m')
r = Matrix([x, y])
v = Matrix([vx, vy])
s = Matrix([r, v, [m]])
# Control
u, theta = symbols('u theta')
# Parameters
c1, c2, c3, g, a = symbols('c1 c2 c3 g a')
```

```python
# State Dynamics
vnorm = v.norm()
vhat  = v/vnorm
print vhat
dx = vx
dy = vy
dvx = c1*u/m*cos(theta) - c3*(vnorm**2)*vhat[0]/m
dvy = c1*u/m*sin(theta) - g - c3*(vnorm**2)*vhat[1]/m
dm = -c1*u/c2
ds = Matrix([dx, dy, dvx, dvy, dm])
```

    Matrix([[vx/sqrt(Abs(vx)**2 + Abs(vy)**2)], [vy/sqrt(Abs(vx)**2 + Abs(vy)**2)]])

```python
# Homotopic Cost Lagrangian
L = a*c1 / c2 * u + (1-a) * c1**2 /c2 * u**2
```

```python
# Hamiltonian
H = l.dot(ds) + L
```

```python
# Costate Dynamics
dl = -Matrix([diff(H, i) for i in s])
dl
```

```python

```


# Joint Representation Learning

In many problems, our network might be more than just the information contained in its adjacency matrix (called its topology, or its collection of nodes and edges). If we were investigating a social network, we might have access to extra information about each person -- their gender, for instance, or their age. If we were investigating a brain network, we might have information about the physical location of neurons, or the volume of a brain region. When we we embed a network, it seems like we should be able to use these extra bits of information - called the "features" or "covariates" of a network - to somehow improve our analysis. The techniques and tools that we'll explore in this section use both the covariates and the topology of a network to create and learn from new representations of the network. Because they jointly use both the topology of the network and its extra covariate information, these techniques and tools are called joint representation learning.

There are two primary reasons that we might want to explore using node covariates in addition to topological structure. First, they might improve our standard embedding algorithms, like Laplacian and Adjacency Spectral Embedding. For example, if the latent structure of the covariates of a network lines up with the latent structure of its topology, then we might be able to reduce noise when we embed, even if the communities in our network don't overlap perfectly with the communities in our covariates. Second, figuring out what the clusters of an embedding actually mean can sometimes be difficult and covariates create a natural structure in our network that we can explore. Covariate information in brain networks telling us where in the brain each node is, for instance, might let us better understand the types of characteristics that distinguish between different brain regions.

In this section, we'll explore different ways to learn from our data when we have access to the covariates of a network in addition to its topological structure. We'll explore *Covariate-Assisted Spectral Embedding* (CASE), a variation on Spectral Embedding. In CASE, instead of embedding just the adjacency matrix or its regularized Laplacian, we'll combine the Laplacian and our covariates into a new matrix and embed that.

A good way to illustrate how using covariates might help us is to use a model in which some of our community information is in the covariates and some is in our topology. Using the Stochastic Block Model, we’ll create a simulation using three communities: the first and second community will be indistinguishable in the topological structure of a network, and the second and third community will be indistinguishable in its covariates. By combining the topology and the covariates, we'll get a nice embedding that lets us find three distinct community clusters.

### Stochastic Block Model

Suppose we have a network generated from a Stochastic Block Model (or, commonly, SBM) with three communities. In our adjacency matrix, which contains only our network's topological information, we'd like to create a situation where the first two communities are completely indistinguishable: Any random node in the first community will have exactly the same chance of being connected to another node in the first community or to a node in the second community. We'd like the third community to be distinct, with only a small probability that nodes in it will connect to nodes in either of the other two communities.

The Python code below generates a matrix that looks like this. There are 1500 nodes, with 500 nodes per community. Because the $3 \times 3$ block probability matrix that generated this SBM has the same probability values (.3) in its upper-left $2 \times 2$ square, a node in community 1 has a 30% chance of being connected to either another node in community 1 or a node in community 2. As a result, in our adjacency matrix, we'll see the nodes in communities one and two as a single giant block. On the other hand, nodes in community three only have a 15% chance to connect to nodes in the first community. So, the end result is that we've created a situation where we have three communities that we'd like to separate into distinct clusters, but the topological structure in the adjacency matrix can't separate the three groups by itself.

```python
import warnings
warnings.filterwarnings("ignore")

import numpy as np
np.random.seed(42)
```

```python
import numpy as np
from graspologic.simulations import sbm

# Start with some simple parameters
N = 1500  # Total number of nodes
n = N // 3  # Nodes per community
p, q = .3, .15
B = np.array([[.3, .3, .15],
              [.3, .3, .15],
              [.15, .15, .3]])  # Our block probability matrix

# Make our Stochastic Block Model
A, labels = sbm([n, n, n], B, return_labels = True)
```

Here you can see what our adjacency matrix looks like. Notice the giant block in the top-left: this block contains both nodes in both of the first two communities, and they're indistinguishable from each other.

```python
import matplotlib.pyplot as plt
from graphbook_code import heatmap
import matplotlib

# visualize
fig, ax = plt.subplots(figsize=(10,10))

def plot_heatmap(A, ax, title="A Stochastic Block Model With 3 Communities", show_cbar=True):
    cmap = matplotlib.colors.ListedColormap(["white", 'black'])
    ax = heatmap(A, cmap=cmap, ax=ax, inner_hier_labels=labels,
                 title=title, center=None, cbar=False)
    ax.set_frame_on(False)
    if show_cbar:
        fig = plt.gcf()
        cax = fig.add_axes([0.95, 0.4, 0.05, 0.2])
        colorbar = fig.colorbar(ax.imshow(A, cmap=cmap), cax=cax)
        colorbar.set_ticks([0.25, 0.75])
        colorbar.set_ticklabels(["Edge", "No Edge"])
        cax.set_frame_on(True)

plot_heatmap(A, ax)
```

If we wanted to embed this graph using our Laplacian or Adjacency Spectral Embedding methods, we'd find the first and second communities layered on top of each other (though we wouldn't be able to figure that out from our embedding if we didn't cheat by knowing in advance which community each node is supposed to belong to). The python code below embeds our latent positions all the way down to two dimensions. Below it, you can see a plot of the latent positions, with each node color-coded by its true community.

```python
from graspologic.embed import LaplacianSpectralEmbed as LSE
from graspologic.utils import to_laplacian

L = to_laplacian(A, form="R-DAD")
L_latents = LSE(n_components=2).fit_transform(L)
```

```python
import seaborn as sns
from graphbook_code import plot_latents

plot = plot_latents(L_latents, title="Latent positions when we\n only embed the Laplacian",
                    labels=labels)
```

As you can see, we'd have a tough time clustering this - the first and second community are completely indistinguishable. It would be nice if we could use extra information to more clearly distinguish between them. We don't have this information in our adjacency matrix: it needs to come from somewhere else.

### Covariates

But we're in luck - we have a set of covariates for each node! These covariates contain the extra information we need that allows us to separate our first and second community. However, with only these extra covariate features, we can no longer distinguish between the last two communities - they contain the same information.

Below is Python code that generates these covariates. Each node is associated with its own group of 30 covariates (thirty being chosen primarily to visualize what's happening more clearly). We'll organize this information into a matrix, where the $i_{th}$ row contains the covariates associated with node $i$. Remember that we have 1500 nodes in our network, so there will be 1500 rows. We'll draw all the covariates for each node from the same Beta distribution (with values ranging from 0 to 1), but the nodes in the first community will be drawn from a different Beta distribution than the nodes in the last two.

```python
from scipy.stats import beta

def make_community(a, b, n=500):
    return beta.rvs(a, b, size=(n, 30))

def gen_covariates():
    c1 = make_community(2, 5)
    c2 = make_community(2, 2)
    c3 = make_community(2, 2)

    covariates = np.vstack((c1, c2, c3))
    return covariates

# Generate a covariate matrix
Y = gen_covariates()
```

Here is a visualization of the covariates we just created. The first community is represented by the lighter-colored rows, and the last two are represented by the darker-colored rows.

```python
from graphbook_code import GraphColormap

# Plot and make the axis look nice
cmap = GraphColormap("sequential", discrete=False).palette
fig, ax = plt.subplots(figsize=(5, 8))
ax = sns.heatmap(Y, ax=ax, cmap=cmap, yticklabels=500)
ax.set(title="Visualization of the covariates", xticks=[],
       ylabel="Nodes (each row is a node)",
       xlabel="Covariates for each node (each column is a covariate)");
```

We can play almost the same game here as we did with the Laplacian. If we embed the information contained in this matrix of covariates into lower dimensions, we can see the reverse situation as before - the first community is separate, but the last two are overlayed on top of each other.

Below is Python code that embeds the covariates. We'll use custom embedding code rather than graspologic's LSE class, because it isn't technically right to act as if we're embedding a Laplacian (even though we're doing essentially the same thing under the hood). Underneath it is a plot of the resulting embedding.

```python
from sklearn.utils.extmath import randomized_svd

def embed(matrix, *, dimension):
    latents, _, _ = randomized_svd(matrix, n_components=dimension)
    return latents

Y_latents = embed(Y, dimension=2)
```

As you can see, we're in a similar situation as we were in with the adjacency matrix, but with different communities: instead of the first and second communities being indistinguishable, now the second and third are. We'd like to see full separation between all three communities, so we need some kind of representation of our network that allows us to use both the information in the topology and the information in the covariates. This is where CASE comes in.

## Covariate-Assisted Spectral Embedding

Covariate-Assisted Spectral Embedding, or CASE1, is a simple way of combining our network and our covariates into a single model. In the most straightforward version of CASE, we combine the network's regularized Laplacian matrix $L$ and a function of our covariate matrix $YY^\top$. Here, $Y$ is just our covariate matrix, in which row $i$ contains the covariates associated with node $i$. Notice the word "regularized" - This means (from the Laplacian section earlier) that our Laplacian looks like $L = L_{\tau} = D_{\tau}^{-1/2} A D_{\tau}^{-1/2}$ (remember, $D$ is a diagonal matrix with $D_{ii}$ telling us the degree of node $i$).

```{note}
Suppose that $Y$ only contains 0's and 1's. To interpret $YY^T$, notice we're effectively taking the the dot product of each row of $Y$ with each other row, because the transpose operation turns rows into columns. Now, look at what happens below when we take the dot product of two example vectors with only 0's and 1's in them:

\begin{align}
\begin{bmatrix}
1 & 1 & 1
\end{bmatrix} \cdot
\begin{bmatrix}
0 \\
1 \\
1 \\
\end{bmatrix} = 1\times 0 + 1\times 1 + 1\times 1 = 2
\end{align}

If there are two overlapping 1's in the same position of the left vector and the right vector, then there will be an additional 1 added to their weighted sum. So, in the case of a binary $YY^T$, when we matrix-multiply a row of $Y$ by a column of $Y^T$, the resulting value, $(YY^T)_{i, j}$, will be equal to the number of shared locations in which vectors $i$ and $j$ both have ones.
```

A particular value in $YY^\top$, $(YY^\top)_{i, j}$, can be interpreted as measuring the "agreement" or "similarity" between row $i$ and row $j$ of our covariate matrix. Since each row represents the covariates for a particular node, the higher the value of $(YY^\top)_{i, j}$, the more similar the covariates of the $i_{th}$ and $j_{th}$ nodes are. The overall result is a matrix that looks fairly similar to our Laplacian!

The following Python code generates our covariate similarity matrix $YY^\top$. We'll also normalize the rows of our covariate matrix to have unit length using scikit-learn - this is because we want the scale for our covariate matrix to be roughly the same as the scale for our adjacency matrix. Later, we'll weight $YY^\top$ to help with this as well.

```python
from sklearn.preprocessing import normalize

Y = normalize(Y, axis=0)
YYt = Y@Y.T
```

Below, you can see the Laplacian we generated earlier next to $YY^\top$. Remember, each matrix contains information about our communities that the other doesn't have - and our goal is to combine them in a way that lets us distinguish between all three communities.

```python
# plot
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(10,5), constrained_layout=True)
L_ax = plot_heatmap(L, title=r"Regularized Laplacian", ax=axs[0], show_cbar=False)
X_ax = heatmap(YYt, title=r"Covariate matrix times its transpose ($YY^\top$)", ax=axs[1]);
```

The way we'll combine the two matrices will simply be a weighted sum of these two matrices - this is what CASE is doing under the hood. The weight (here called $\alpha$) is multiplied by $YY^\top$ - that way, both matrices contribute an equal amount of useful information to the embedding.

$$
L + \alpha YY^\top
$$

### Exploring Possible Weights

An obvious question here is how to weight the covariates. If we simply summed the two matrices by themselves, we'd unfortunately be in a situation where whichever matrix contained larger numbers would dominate over the other. In our current setup, without a weight on $YY^\top$, the covariates of our network would dominate over its topology.

```python
fig, axs = plt.subplots(1, 2, figsize=(10, 5))
heatmap(L + Y@Y.T, title="Summing the two matrices \nwithout a weight", ax=axs[0], context='notebook');
plot_latents(embed(L + Y@Y.T, dimension=2), title="Embedding the summed \nmatrix without a weight",
             ax=axs[1], labels=labels);
```

What do different potential weights look like? Let's do a comparison. Below you can see the embeddings for 9 possible weights on $YY^\top$, ranging between $10^{-5}$ and 100.

```python
fig, axs = plt.subplots(nrows=3, ncols=3, figsize=(10, 10))
for a, ax in zip(np.geomspace(10e-5, 100, num=9), axs.flat):
    l_ = embed(L+a*YYt, dimension=2)
    plot_latents(l_, title=f"weight: {a:.3f}", labels=labels, ax=ax)
    ax.get_legend().remove()

fig.suptitle(r"Comparison of embeddings for different weights on $YY^\top$", y=1, fontsize=25);
plt.tight_layout()
```

It looks like we'd probably want a weight somewhere between 0.01 and 0.5  - then, we'll have three communities which are fairly distinct from each other, implying that we're pulling good information from both our network's topology and its covariates. We could just pick our weight manually, but it would be nice to have some kind of algorithm or equation which lets us pick a reasonable weight automatically.

### Finding a Reasonable Weight Automatically

In general, we'd like to embed in a way that lets us distinguish between communities. This means that if we knew which community each node belonged to, we'd like to be able to correctly retrieve the correct commmunities for each node as possible with a clustering algorithm after embedding. This also implies that the communities will be as distinct as possible.

We already found a range of possible weights, embedded our combined matrix for every value in this range, and then looked at which values produced the best clustering. But, how do we find a weight which lets us consistently use useful information from both the topology and the covariates?

When we embed symmetric matrices, keep in mind that the actual points we're plotting are the components of the eigenvectors with the biggest eigenvalues. When we embed into two-dimensional space, for instance, the X-axis values of our points are the components of the eigenvector with the biggest eigenvalue, and the Y-axis values are the components of the eigenvector with the second-biggest eigenvalue. This means that we should probably be thinking about how much information the Laplacian and $YY^\top$ contributes to the biggest eigenvalue/eigenvector pairs.

Thinking about this more, if we have a small weight, $YY^\top$ will contribute only a small amount to the biggest eigenvalue/vector pair. If we have a large weight, $YY^\top$ will contribute a large amount to the biggest eigenvalue/vector pair. The weight that causes the Laplacian and $YY^\top$ to contribute the same amount of information, then, is just the ratio of the biggest eigenvalue of $L$ and the biggest eigenvalue of $YY^\top$:

$$
weight = \frac{\lambda_1 (L)}{\lambda_1 (YY^\top)}
$$

Let's check what happens when we combine our Laplacian and covariates matrix using the weight described in the equation above. Our embedding works the same as it does in Laplacian Spectral Embedding: we decompose our combined matrix using Singular Value Decomposition, truncating the columns, and then we visualize the rows of the result. Remember, we'll be embedding $L + \alpha YY^\top$, where $\alpha$ is our weight. We'll embed all the way down to two dimensions, just to make visualization simpler.

```python
from scipy.sparse.linalg import eigsh

# Find the biggest eigenvalues in both of our matrices
leading_eigval_L, = eigsh(L, return_eigenvectors=False, k=1)
leading_eigval_YYt, = eigsh(YYt, return_eigenvectors=False, k=1)

# Per our equation above, we get the weight using
# the ratio of the two biggest eigenvalues.
weight = leading_eigval_L / leading_eigval_YYt

# Do our weighted sum, then embed
L_ = L + weight*YYt
latents_ = embed(L_, dimension=2)
```

```python
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(10, 5))
heatmap(L_, title=f"Our Combined Laplacian and \ncovariates matrix with a weight of {weight:.2f}", ax=axs[0])
plot_latents(latents_,
             title=f"Embedding of the combined matrix",
             labels=labels, ax=axs[1]);

plt.tight_layout()
```

Success! We've managed to achieve separation between all three communities. Below we can see (from left to right) a comparison of our network's latent position when we only use its topological information, when we only use the information contained in its covariates, and finally our embedding using the weight we found.

```python
from graspologic.embed import LaplacianSpectralEmbed as LSE

# Plot
fig, axs = plt.subplots(1, 3, figsize=(15,5))
plot_latents(L_latents, title="Latent positions when we only use the Laplacian",
             labels=labels, ax=axs[0])
plot_latents(Y_latents, title="Latent positions when we only use our covariates",
             labels=labels, ax=axs[1]);
plot_latents(latents_, title="Latent positions when we combine\n our network and its covariates",
             labels=labels, ax=axs[2])

plt.tight_layout()
```

## Using Graspologic

Graspologic's CovariateAssistedSpectralEmbedding class implements CASE directly. The following code applies CASE to reduce the dimensionality of $L + aYY^T$ down to two dimensions, and then plots the latent positions to show the clustering.

```{admonition} Non-Assortative Networks
We don't always necessarily want to embed $L + \alpha YY^\top$. Using the regularized Laplacian by itself, for instance, isn't always best. If your network is *non-assortative* - meaning, the between-block probabilities are greater than the within-block probabilities - it's better to square our Laplacian. This is because the adjacency matrices of non-assortative networks have a lot of negative eigenvalues; squaring the Laplacian gets rid of a lot of annoying negative eigenvalues, and we end up with a better embedding. In the non-assortative case, we end up embedding $LL + aYY^\top$. The `embedding_alg` parameter controls this: you can write `embedding_alg="non-assortative"` if you're in the non-assortative situation.
```

```python
from graspologic.embed import CovariateAssistedEmbed as CASE

casc = CASE(assortative=True, n_components=2)
latents = casc.fit_transform(A, covariates=Y)
plot_latents(latents, title="Embedding our model using graspologic", labels=labels);
```

## Omnibus Joint Embedding

If you've read the Multiple-Network Representation Learning section, you've seen the Omnibus Embedding (if you haven't read that section, you should go read it before reading this one!). To recap, the way the omnibus embedding works is:

1. Have a bunch of networks
2. Combine the adjacency matrices from all of those networks into a single, huge network
3. Embed that huge network

Remember that the Omnibus Embedding is a way to bring all of your networks into the same space (meaning, you don't run into any rotational nonidentifiability issues when you embed). Once you embed the Omnibus Matrix, it'll produce a huge latent position matrix, which you can break apart along the rows to recover latent positions for your individual networks.

You might be able to predict where this is going. What if we created an Omnibus embedding not with a set of networks, but with a network and covariates?

```python
from graspologic.embed import OmnibusEmbed

# embed with Omni
omni = OmnibusEmbed()

# Normalize the covariates first
YYt = Y@Y.T
YYt /= np.max(YYt)

# Create a joint embedding
joint_embedding = omni.fit_transform([A, YYt])
```

```python
from graphbook_code import plot_latents

fig, axs = plt.subplots(1, 2, figsize=(10, 5))

# network
network_plot = plot_latents(joint_embedding[0], ax=axs[0], legend=None,
                            labels=labels, title="Omni embedding for topology")

# covariates
covariates_plot = plot_latents(joint_embedding[1], ax=axs[1],
                               labels=labels, title="Omni embedding for covariates")
sns.move_legend(covariates_plot, loc="center right", bbox_to_anchor=(1.4, .5))
```

There's a few things going on here. First, we had to normalize the covariates by dividing $YY^\top$ by its maximum. This is because if we didn't, the covariates and the adjacency matrix would contribute vastly different amounts of information to the omnibus matrix. You can see that by looking at the average value of $YY^\top$ compared to the average value of $A$:

```python
print(r"Mean of Y@Y.T:", f"{np.mean(Y@Y.T):.2f}")
print(r"Mean of A:", f"{np.mean(A):.2f}")
```

```python
print(r"Mean of normalized Y@Y.T:", f"{np.mean(YYt):.2f}")
print(r"Mean of A:", f"{np.mean(A):.2f}")
```

Second, you can see that *both embeddings* are able to separate all three communities, even though we're using the same data that we used for the CASE section. This is because the Omnibus Embedding induces dependence on the latent positions it outputs. Remember that the off-diagonals of the Omnibus Matrix contain the averages of pairs of networks fed into it. These off-diagonal elements are responsible for some "information leakage" - the latent representation for the topology uses information from the covariates, and the latent representation for the covariates uses information from the topology.

## MASE Joint Embedding

Just like you can use the OMNI to do a joint embedding, you can also use MASE to do a joint embedding. This fundamentally comes down to the fact that both embeddings fundamentally just eat matrices as their input - whether those matrices are the adjacency matrix or $YY^\top$ doesn't really matter.

Just like OMNI, we'll quickly recap how MASE works here:
1. Have a bunch of networks
2. Embed them all separately with ASE or LSE
3. Concate those embeddings into a single latent position matrix with a lot more dimensions
4. Embed that new matrix

```python
from graspologic.embed import MultipleASE as MASE

# Remmeber that YY^T is still normalized!
mase = MASE(n_components=2)
joint_embedding = mase.fit_transform([A, YYt])
```

```python
# network
plot = plot_latents(joint_embedding, labels=labels,
                            title="MASE embedding")

sns.move_legend(plot, loc="center right", bbox_to_anchor=(1.3, .5))
```

As you can see, MASE lets us get fairly clear separation between communities. The covariates are still normalized, as with OMNI, so that they can contribute the same amount to the embedding as the adjacency matrix.

#### References

[1] N. Binkiewicz, J. T. Vogelstein, K. Rohe, Covariate-assisted spectral clustering, Biometrika, Volume 104, Issue 2, June 2017, Pages 361–377, https://doi.org/10.1093/biomet/asx008
[2] Lloyd, S. (1982). Least squares quantization in PCM. IEEE transactions on information theory, 28(2), 129-137.
[3] https://scikit-learn.org/stable/modules/clustering.html#k-means
[4] Hotelling, H. (1936). Relations between two sets of variates. Biometrika 28, 321–77.


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section 10.6, Conduction de la chaleur une conduite circulaire composite:

```python
#
# Pierre Proulx
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
%matplotlib inline
```

```python
# Paramètres, variables et fonctions
r,k01,k12,k23,h0,h3=sp.symbols('r k_1 k_2 k_3 h_0 h_3')
r0,r1,r2,r3,Ta,Tb=sp.symbols('r_0 r_1 r_2 r_3 T_a T_b')
q=sp.symbols('q')
T=sp.Function('T')(r)
```

```python
eq1=sp.Eq(k01/r*sp.Derivative(r*sp.Derivative(T,r)),0)
eq2=sp.Eq(k12/r*sp.Derivative(r*sp.Derivative(T,r)),0)
eq3=sp.Eq(k23/r*sp.Derivative(r*sp.Derivative(T,r)),0)
T1=sp.dsolve(eq1).rhs
T2=sp.dsolve(eq2)
T2=T2.subs(sp.symbols('C1'),sp.symbols('C3'))
T2=T2.subs(sp.symbols('C2'),sp.symbols('C4')).rhs
T3=sp.dsolve(eq3)
T3=T3.subs(sp.symbols('C1'),sp.symbols('C5'))
T3=T3.subs(sp.symbols('C2'),sp.symbols('C6')).rhs
display(T1)
display(T2)
display(T3)
```

```python
# Maintenant on pose les conditions aux limites pour trouver les 6 constantes
cl1=sp.Eq(T1.subs(r,r1)-T2.subs(r,r1))   # températures égales sur les points intérieurs
cl2=sp.Eq(T2.subs(r,r2)-T3.subs(r,r2))
#                           flux égaux sur les points intérieurs
cl3=sp.Eq(k01*T1.diff(r).subs(r,r1)-k12*T2.diff(r).subs(r,r1))
cl4=sp.Eq(k12*T2.diff(r).subs(r,r2)-k23*T3.diff(r).subs(r,r2))
#                           flux donnés par la loi de refroidissement de Newton sur les parois
cl5=sp.Eq(-k01*T1.diff(r).subs(r,r0)+h0*(T1.subs(r,r0)-Ta))
cl6=sp.Eq(-k23*T3.diff(r).subs(r,r3)+h3*(Tb-T3.subs(r,r3)))
```

```python
constantes=sp.solve((cl1,cl2,cl3,cl4,cl5,cl6),sp.symbols('C1 C2 C3 C4 C5 C6'))
T1=T1.subs(constantes)
T2=T2.subs(constantes)
T3=T3.subs(constantes)
dico={'k_1':4,'k_2':25,'k_3':1,
      'h_0':100,'h_3':20,'r_0':0.020,'r_1':0.023,'r_2':0.028,'r_3':0.030,'T_a':100,'T_b':20}
T1p=T1.subs(dico).evalf(5)
T2p=T2.subs(dico).evalf(5)
T3p=T3.subs(dico).evalf(5)
display(T1p,T2p,T3p)
# Calcule les taux de chaleur en 0 et en 3 (doivent être égaux) (watts / mètre de longueur)
#
taux3=(h3*(T3-Tb)*2*sp.pi*r3).subs(dico)      # pour mettre les valeurs numériques dans
taux0=(h0*(Ta-T1)*2*sp.pi*r0).subs(dico)      # l'expression symbolique, on subs(dico)
#
#
print(taux3.subs(r,r3.subs(dico)), taux0.subs(r,r0.subs(dico)))
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = 12, 10
#plt.figure(figsize=(12,10))
p=sp.plot((T1p,(r,r0.subs(dico),r1.subs(dico)))
         ,(T2p,(r,r1.subs(dico),r2.subs(dico)))
         ,(T3p,(r,r2.subs(dico),r3.subs(dico)))
         ,legend=True,ylabel='T(r)',xlabel='r',show=False) #affiche pas tout de suite
p[0].line_color = 'red'
p[0].label='de r = r_0 à r=r_1 '
p[1].line_color = 'black'
p[1].label='de r = r_1 à r=r_2 '
p[2].line_color = 'green'
p[2].label='de r = r_2 à r=r_3 '
p.show()                         # maintenant on est prêts à afficher
```