# Mackey-Glass Time Series Prediction Using Look Up Table

## Introduction

The purpose of this work is to create a fuzzy system for time series prediction of Mackey-Glass equation, which is delay differential equation:
\begin{equation}
    \frac{dx(t)}{dt} = \beta \frac{x(t-\tau)}{1+x^{n}(t-\tau)}-\gamma x(t)
\end{equation}
The goal is in time $t_1$ to predict $x(t_1+k)$, we choose\footnote{$\tau$ should be more than 17.} $\tau = 30$. We need to set $n \approx 10$, because that is the value where chaos appears. The remaining two parameters are set as follows: $\beta = 0.2$ and $\gamma = 0.1$.

```python
# imports
from math import pow
import numpy as np
import matplotlib.pyplot as plt
```

```python
# Mackey-Glass differential eq. parameters
Beta = 0.2
Gamma = 0.1
Tau = 30
n = 10
# initializing an array of random values for numerical simulation
y = np.random.rand(Tau+1)
# lenght of our simulation
samples = 600
```

## Implementation of Mackey-Glass

We can approximate function (1) like this:
\begin{equation}
    x(t+k)\approx x(t)+k\frac{dx(t)}{dt}
\end{equation}
We put equations (1) and (2) together and $k=1$, since we go step by step by samples, hence the following equation is our approximation:
\begin{equation}
    x(t+1)= x(t)+\beta \frac{x(t-\tau)}{1+x^{n}(t-\tau)}-\gamma x(t)
\end{equation}
Put our constants in equation (3):
\begin{equation}
    x(t+1)=x(t)+ \frac{0.2x(t-30)}{1+x^{10}(t-30)}-0.1 x(t)
\end{equation}
Which is the final form of our equation.

```python
# calculating Mackey-Glass eq. for samples
for _ in range(samples):
    value=y[-1]+Beta*y[-1-Tau]/(1+pow(y[-1-Tau],n))-Gamma*y[-1]
    y = np.r_[y,value]

# we dont need those first random values
y=y[-samples:]

# lets plot it
plt.figure()
plt.plot(y,label='Mackey-Glass')
plt.legend()
plt.xlabel('samples [-]')
plt.ylabel('x')
plt.grid(True)
plt.show()
```

## Design of a fuzzy system

The goal is to create a fuzzy system only from input-output data. We use only the first 300 values of x. The n represents number of statements in fuzzy IF-THEN rule. Thus we make a input-output pairs for creating fuzzy rules for look up table system only from input set.
\begin{align*}
    [x(0),x(1),...,x(n-1);x(n)] \\
    [x(1),x(2),...,x(n);x(n+1)] \\
            . \\
            . \\
            . \\
    [x(299-n),x(299-n+1),...,x(298);x(299)] \\
\end{align*}

We choose that our system is Look up table with product inference engine, triangular fuzzyfier and center average defuzzyfier:
\begin{equation}
    f(x)=\frac{\sum_{l=1}^M \overline{y}^l(\prod_{i=1}^n \mu_{A_i^l}(x))}{\sum_{l=1}^M (\prod_{i=1}^n \mu_{A_i^l}(x))}
\end{equation}

Where M is number of Fuzzy IF-THEN rules and n is number of fuzzy statements in one rule.
With fuzzy IF-THEN rules looking like this:
\begin{equation}
    IF \bigwedge_{i=0}^{n-1} ( x(k-n) \ is \ A_i^l )\ THEN \ x(k+1) \ is \ B^l
\end{equation}

And we use triangular membership function:
\begin{equation}
    \mu_{A_i^l}(x) = \left\{ \begin{array}{r@{\quad}c}
    1-\frac{|x-c_{A_i^l}|}{D}, & x \in (c_{A_i^l}-D,c_{A_i^l}+D) \\
    0, & else \\ \end{array} \right.
\end{equation}

```python
# Look Up Table parameters
TrainingData = y[0:300]    #Training data for fuzzy system, we use only first 300 samples
TestingData = y            #We useall of our data for testing
NumStatements = 5          #Number of fuzzy-statements in fuzzy IF-THEN rule
NumFuzzySets = 7           #Number of Fuzzy Sets in input and output area

# definine our membership function
def triangle(x,center,D):
    if x<=(center-D):
        return 0
    if x>=(center+D):
        return 0
    else:
        return -(1/D)*abs(x-center)+1
```

```python
#Creating an array of fuzzy-sets centers
Minimum = min(TrainingData)
Maximum = max(TrainingData)
D = (Maximum-Minimum)/(NumFuzzySets-1)
FuzzySetsCenters=[]
for i in range(NumFuzzySets-1):
    FuzzySetsCenters.append(Minimum+i*D)
FuzzySetsCenters.append(Maximum)
```

```python
# creating INPUT-OUTPUT pairs
RulesData = []
for i in range(len(TrainingData)-NumStatements):
    rule = []
    for j in range(NumStatements):
        rule.append(TrainingData[i+j])
    RulesData.append(rule)

# fuzzy IF-THEN rules
FuzzyRules = []
for i in range(len(RulesData)):
    rulefp = []
    for j in range(NumStatements):
        fp = []
        for k in range(NumFuzzySets):
            _ = triangle(RulesData[i][j],FuzzySetsCenters[k],D)
            fp.append(_)
        rulefp.append(FuzzySetsCenters[fp.index(max(fp))])
    FuzzyRules.append(rulefp)
```

Since the number of input-output is usually large, and with each pair generating a single rule, there is high chance, that there are some conflict ones. Which means, there are rules with same IF part but different THEN parts. Therefore, we assign degree of a rule:
\begin{equation}
D(Ru^{(l)})=\mu_{B^{l*}}(y_0^p)\prod_{i=1}^{n}\mu_{A_i^{l*}}(x_{0i}^p)
\end{equation}
Hence we choose non-conflict fuzzy rules, and from conflict rules we choose ones with the highes degree. Chosen Fuzzy Rules are our fuzzy system base.

```python
# function, which assign degree to a fuzzy IF-THEN rule
def DegreeRule(RuleData,FuzzyRule):
    value = 1
    for i in range(len(RuleData)):
        value=value*triangle(RuleData[i],FuzzyRule[i],D)
    return value

#Assign degrees to fuzzy IF-THEN rules and deleting conflict ones with lower degree
FRules = FuzzyRules.copy()
for i in range(len(FuzzyRules)):
    for j in range(i,len(FuzzyRules)):
        if FuzzyRules[i][:-1] == FuzzyRules[j][:-1] and not i==j:
            Degree1 = DegreeRule(RulesData[i],FuzzyRules[i])
            Degree2 = DegreeRule(RulesData[j],FuzzyRules[j])
            if Degree1 > Degree2:
                FRules[j] = None
            else:
                FRules[i] = None
        else:
            pass

#New, non-conflict fuzzy IF-THEN rules, our fuzzy system base
FuzzyRules=[]
for i in range(len(FRules)):
    if not FRules[i] == None:
        FuzzyRules.append(FRules[i])
```

## Look Up Table prediciton results

We have used first 300 samples to create our Look Up Rable, now we use our created system to compute whole interval including our testing data. Once again, here is equation of our fuzzy system:
\begin{equation}
    f(x)=\frac{\sum_{l=1}^M \overline{y}^l(\prod_{i=1}^n \mu_{A_i^l}(x))}{\sum_{l=1}^M (\prod_{i=1}^n \mu_{A_i^l}(x))}
\end{equation}
Which we will now write as a function in python:

```python
# function of our fuzzy system
def MamdaniValue(X,FuzzyRules,D):
    pom1 = 0
    pom2 = 0
    for FuzzyRule in FuzzyRules:
        pom3 = 1
        for i in range(len(X)):
            pom3 = pom3*triangle(X[i],FuzzyRule[i],D)
        pom1=pom1+FuzzyRule[-1]*pom3
        pom2=pom2+pom3
    try:
        return (pom1/pom2)
    except ZeroDivisionError:
        return 0
```

```python
# Calculating predicted value, real Mackey-Glass and absolute deviation
TestingInputs = []
for i in range(len(TestingData)-NumStatements):
    vector = []
    for j in range(NumStatements-1):
        vector.append(TestingData[i+j])
    TestingInputs.append(vector)

Predicted=[]
for i in range(NumStatements):
    Predicted.append(TestingData[i]) # Insert values, for same lenght of arrays
for j in range(len(TestingData)-NumStatements):
    Predicted.append(MamdaniValue(TestingData[j:j+NumStatements-1],FuzzyRules,D))

Deviation=[]
for i in range(len(TestingData)):
    Deviation.append(abs(TestingData[i]-Predicted[i]))
```

```python
# ploting predicted value, real Mackey-Glass and absolute deviation
plt.figure(dpi=80, figsize=(12,8))
plt.plot(TestingData,'b',label='Mackey-Glass')
plt.plot(Predicted,'k',label='Mackey-Glass - LUT')
plt.plot(Deviation,'r',label='Absolute Deviation')
plt.legend()
plt.xlabel('samples [-]')
plt.ylabel('x')
plt.grid(True)
plt.show()
```

```python

```


```python
# import packages
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sympy import *
from scipy.optimize import fsolve
%matplotlib inline
```

```python
# set up variables
length = 25 #(nm)
delta_x = length/332 #(nm)
delta_t = 0.016667 #(s)
```

```python
# set up constant for follwing calculation
a = 5.6737*10**(-6)
b = 4.295*10**(-5)
c = 6.3129*10**(-5)
d = 1.76386*10**(-6)
e = 1.5863*10**(-8)
f = 6.29973*10**(-10)
```

```python
# import two-phase data
df = pd.read_excel('D.xlsx', sheet_name='3')
C = df['Li Fraction'] # the concentration in two-phase region
B = df['Symbol']
```

```python
def myFunction(BB):

    for i in range(0,333):
        B[i] = BB[i]

    F = np.empty((333))

    F[0] = B[0] - 0
    F[332] = B[332] - C[332]

    for i in range(1,332):

        F[i] = (2*B[i]-2*C[i]) * (a*B[i]**2 + b*B[i] - a*B[i]*C[i] - b*C[i] + c*C[i]**3 - d*C[i]**2 + e*C[i] - f) - ( (2*C[i+1]-2*B[i+1]-2*C[i]+2*B[i])**2 / (2*C[i+1]-B[i+1]-2*C[i-1]+B[i-1]) *delta_x / delta_t - (B[i+1]-B[i])**2 / (B[i+1]-1.99*B[i]+B[i-1])* delta_x / delta_t )

    return F

BBGuess = np.linspace(0, 0.8, 333)
BB = fsolve(myFunction,BBGuess)
print(BB[0],BB[331],BB[332])
```

    C:\Users\user\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      after removing the cwd from sys.path.

    0.0 0.7975903614457832 0.8

    C:\Users\user\Anaconda3\lib\site-packages\scipy\optimize\minpack.py:163: RuntimeWarning: The iteration is not making good progress, as measured by the
      improvement from the last ten iterations.
      warnings.warn(msg, RuntimeWarning)

```python

```


# Homework 7 - Part 1: LASSO Regression
## BIOST 558 Spring 2020
### Juan Solorio

```python
# needed libraries

# vizualization
import matplotlib.pyplot as plt

# data wrangling and math
import pandas as pd
import numpy as np

# machine learing
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn import linear_model

%matplotlib inline
plt.style.use('ggplot')
```

```python
# Load the data
hitters = pd.read_csv('https://raw.githubusercontent.com/selva86/datasets/master/Hitters.csv',
                      sep=',', header=0)
hitters = hitters.dropna()
X = hitters.drop('Salary', axis=1)
X = pd.get_dummies(X, drop_first=True)
y = hitters.Salary

def standarize_data(X,y):
    """
    Function to train test split and standarize the data
    """
    x_train, x_test, y_train, y_test = train_test_split(X, y, random_state=0)

    xmean = np.mean(x_train, axis=0)
    xsdev = np.std(x_train, axis=0)
    ymean = np.mean(y_train, axis=0)
    ysdev = np.std(y_train, axis=0)
    x_train = (x_train - xmean)/xsdev[np.newaxis, :]
    x_test = (x_test - xmean)/xsdev[np.newaxis, :]
    y_train= (y_train - ymean)/ysdev
    y_test = (y_test - ymean)/ysdev

    return np.array(x_train), np.array(y_train), np.array(x_test), np.array(y_test)

x_train, y_train, x_test, y_test = standarize_data(X=X, y=y)
```

### LASSO
#### Theory

For LASSO regression, we have the $l_1$-regularized least-square regression defined as:

\begin{equation}
\min_{\beta}F(\beta) = \frac{1}{2n}\sum_{i=1}^n(y_i - x^T_i\beta)^2 + \lambda\|\beta\|_1
\end{equation}

assuming we have $n=d=1$, the sum then becomes

\begin{equation}
F(\beta) = \frac{1}{2}(y - x\beta)^2 + \lambda|\beta|
\end{equation}

We can then take the derivative of $F(\beta)$ with respect to $\beta$ and solve the equation:

\begin{equation}
\frac{dF}{d\beta} = -x(y - x\beta) + \lambda \\
\Rightarrow x^2\beta + \lambda - yx = 0 \\
\Rightarrow \beta = \frac{xy-\lambda}{x^2}
\end{equation}

From the derivative we get the piece wise function for $\beta$:

$$ \beta(c)=   \left\{
\begin{array}{ll}
      c+\lambda & if\ \ c < -\lambda \\
      c-\lambda & if\ \ c > +\lambda \\
      0 & if\ \ |c| \leq \lambda
\end{array}
\right.  $$

$$ \beta(c)=   \left\{
\begin{array}{ll}
      \frac{c+\lambda}{a} & if\ \ c < -\lambda \\
      \frac{c-\lambda}{a} & if\ \ c > +\lambda \\
      0 & if\ \ |c| \leq \lambda
\end{array}
\right.  $$

```python
def beta_stepwise(xy, lamda):
    if xy < -lamda:
        numerator = xy + lamda
    elif xy > lamda:
        numerator = xy - lamda
    else:
        numerator = 0
    return numerator
```

```python
xplot = np.linspace(-5,5,50)
lamda = 0.5
y_Fx = [beta_stepwise(xi,lamda) for xi in xplot ]
plt.plot(xplot, y_Fx)
plt.title(r'Basic Plot of $F(\beta)$')
plt.show()
```

We can then take this basic derivation for $n=d=1$ and implement it for $n>1\ ,\ d>1$. In Matrix Notation, we can take the gradient of $F(\beta)$ with respect to $\beta$ as follows:
\begin{equation}
F(\beta) = (Y - X\beta)^T(Y-X\beta) + \lambda\beta
\end{equation}
\begin{equation}
= Y^TY - \beta^TX^TY-Y^TX\beta+\beta^TX^TX\beta+\lambda\beta
\end{equation}
\begin{equation}
= Y^TY - 2\beta^TX^TY+\beta^TX^TX\beta+\lambda\beta \\
\end{equation}
\begin{equation}
\nabla_\beta F(\beta)= -X^TY + X^TX\beta + \lambda \\
\Rightarrow \beta = \frac{X^TY - \lambda}{X^TX}
\end{equation}

From here, we can take the previous piece wise function for $\beta$ and expand to define $c$ and $a$ as:
$$ \beta(c_j)=   \left\{
\begin{array}{ll}
      \frac{c_j+\lambda}{a_j} & if\ \ c_j < -\lambda \\
      \frac{c_j-\lambda}{a_j} & if\ \ c_j > +\lambda \\
      0 & if\ \ |c_j| \leq \lambda
\end{array}
\right.  $$

\begin{equation}
a_j = \sum_{i=1}^nx^2_{i,j} \ \  \ \ \ \ c_j=\sum_{i=1}^nx_{i,j}(y_i-x^T_{i,-j}\beta_j)
\end{equation}

```python
def Fx_min_beta(b, j, X, y, lamda):
    n, d = X.shape
    j_i = [i for i in range(d) if i != j ]

    c_j = X[:, j] @ (y[:, np.newaxis] - X[:, j_i] @ b[:, np.newaxis][j_i, :])
    a_j = np.linalg.norm(X[:,j])**2

    r = beta_stepwise(xy=c_j, lamda=lamda*n/2)

    return r/a_j
```

```python
def computeobj(b, lamda, X, y):

    n, d = X.shape
    penalty = lamda*np.linalg.norm(b)

    return (1/(2*n)) * np.sum((y - (X @ b))**2) - penalty

```

```python
def cycliccoorddescet(beta_init, X, y, lamda, max_iter):
    n, d = X.shape
    beta = beta_init
    beta_vals = beta
    for i in range(max_iter):
        for j in range(d):
            beta_j = Fx_min_beta(b=beta, j=j, X=X, y=y, lamda=lamda)
            beta[j] = beta_j
        beta_vals = np.vstack((beta_vals, beta))
    return beta_vals
```

```python
def pickcoord(d, j):
    num = np.random.randint(0,d)
    while num == j:
        num = np.random.randint(0,d)
    return num
```

```python
def randcoorddescent(beta_init, X, y, lamda, max_iter):
    n, d = X.shape
    beta = np.copy(beta_init)
    beta_vals = beta
    for i in range(max_iter):
        j = 1
        for k in range(d):
            j = pickcoord(d, j)
            beta_j = Fx_min_beta(b=beta, j=j, X=X, y=y, lamda=lamda)
            beta[j] = beta_j
        beta_vals = np.vstack((beta_vals, beta))
    return beta_vals
```

```python
def compute_correct_zero_nonzero(b_own, b_skl, non_zero=False):
    num_iter = len(b_skl)
    own = 0
    skl = 0
    for n in range(num_iter):
        if non_zero == False:
            if b_skl[n] == 0:
                skl += 1
                if b_own[n] == 0:
                    own += 1
        else:
            if b_skl[n] != 0:
                skl += 1
                if b_own[n] != 0:
                    own += 1

    return own/skl
```

```python
n, d = x_train.shape
betas_init = np.zeros(d)
lasso_cv = linear_model.LassoCV(cv=10, random_state=0,fit_intercept=False,max_iter=1000)
lasso_cv.fit(x_train, y_train)

optimal_lambda = 2*lasso_cv.alpha_
beta_skl = lasso_cv.coef_

betas_cyclic = cycliccoorddescet(beta_init=betas_init, X=x_train, y=y_train, lamda=optimal_lambda, max_iter=1000)
betas_random = randcoorddescent(beta_init=betas_init, X=x_train, y=y_train, lamda=optimal_lambda, max_iter=1000)

```

```python
betas_cyclic[-1], betas_random[-1], beta_skl
```

    (array([-0.40178734,  0.44817461,  0.09057008,  0.        ,  0.01092209,
             0.24231362, -0.06912608,  0.        ,  0.38853311,  0.        ,
             0.35260131,  0.        , -0.29704019,  0.13550563,  0.04299316,
            -0.05854589,  0.0261108 , -0.12125553,  0.        ]),
     array([-0.40178734,  0.44817461,  0.09057008,  0.        ,  0.01092209,
             0.24231362, -0.06912608,  0.        ,  0.38853311,  0.        ,
             0.35260131,  0.        , -0.29704019,  0.13550563,  0.04299316,
            -0.05854589,  0.0261108 , -0.12125553,  0.        ]),
     array([-0.40151056,  0.44792634,  0.09077503, -0.        ,  0.01070252,
             0.24238375, -0.06950319, -0.        ,  0.39076738, -0.        ,
             0.35045337,  0.        , -0.29676259,  0.13544564,  0.04297573,
            -0.05859965,  0.0260473 , -0.12133111,  0.        ]))

```python
n_b, d_b = betas_cyclic.shape

obj_cyclic = np.zeros(n_b)
obj_random = np.zeros(n_b)

for i in range(n_b):
    obj_cyclic[i] = computeobj(b=betas_cyclic[i], lamda=optimal_lambda,X=x_train, y=y_train)
    obj_random[i] = computeobj(b=betas_random[i], lamda=optimal_lambda, X=x_train, y=y_train)

fig, ax = plt.subplots()
ax.plot(range(1, n_b + 1), obj_cyclic, c='blue', label='Cyclic', linewidth=3.0, linestyle='dashed')
ax.plot(range(1, n_b + 1), obj_random, c='red', label='Random')
plt.xlabel('Iteration')
plt.ylabel('Objective')
plt.legend()
plt.show()
```

```python
cyclic_zero = np.zeros(n_b)
random_zero = np.zeros(n_b)

for i in range(n_b):
    cyclic_zero[i] = compute_correct_zero_nonzero(b_own=betas_cyclic[i],b_skl=beta_skl,non_zero=False)
    random_zero[i] = compute_correct_zero_nonzero(b_own=betas_random[i],b_skl=beta_skl,non_zero=False)

fig, ax = plt.subplots()
ax.plot(range(1, n_b + 1), cyclic_zero, c='blue', label='Cyclic', linewidth=3.0, linestyle='dashed')
ax.plot(range(1, n_b + 1), random_zero, c='red', label='Random')
plt.xlabel('Iteration')
plt.ylabel('Fraction of Correct Zero')
plt.legend()
plt.show()
```

```python
cyclic_nonzero = np.zeros(n_b)
random_nonzero = np.zeros(n_b)
for i in range(n_b):
    cyclic_nonzero[i] = compute_correct_zero_nonzero(b_own=betas_cyclic[i,:],b_skl=beta_skl,non_zero=True)
    random_nonzero[i] = compute_correct_zero_nonzero(b_own=betas_random[i,:],b_skl=beta_skl,non_zero=True)

fig, ax = plt.subplots()
ax.plot(range(1, n_b + 1), cyclic_nonzero, c='blue', label='Cyclic', linewidth=3.0, linestyle='dashed')
ax.plot(range(1, n_b + 1), random_nonzero, c='red', label='Random')
plt.xlabel('Iteration')
plt.ylabel('Fraction of Correct Non-Zero')
plt.legend()
plt.show()
```

1-Python/.ipynb_checkpoints/Homework 7 - Part 1-checkpoint.ipynb


# Non Linear Regression Analysis

Estimated time needed: **20** minutes

## Objectives

After completing this lab you will be able to:

-   Differentiate between Linear and non-linear regression
-   Use Non-linear regression model in Python

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \\ y = a x^3 + b x^2 + c x + d \\ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \\log(x)$$

Or even, more complicated such as :
$$ y = \\log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of **log()**: i.e. $$ y = \\log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\\begin{equation}
y = \\log(X)
\\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

import urllib.request
url = 'https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv'
filename = 'china_gdp.csv'
urllib.request.urlretrieve(url, filename)

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

The formula for the logistic function is the following:
\begin{equation}
\hat{Y} = \frac{1+e^{\beta_1(X-\beta_2)}}
\end{equation}

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, \*popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.451711, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice

Can you calculate what is the accuracy of our model?

```python
# write your code here
from sklearn.metrics import r2_score

r_squared = r2_score(ydata, y)
r_squared
```

    0.9726049983525928

Double-click **here** for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

### Thank you for completing this lab!

## Author

Saeed Aghabozorgi

### Other Contributors

Joseph Santarcangelo

## Change Log

| Date (YYYY-MM-DD) | Version | Changed By | Change Description                 |
| ----------------- | ------- | ---------- | ---------------------------------- |
| 2020-08-27        | 2.0     | Lavanya    | Moved lab to course repo in GitLab |
|                   |         |            |                                    |
|                   |         |            |                                    |

##
 © IBM Corporation 2020. All rights reserved.


Solution to: [Day 3: Conditional Probability](https://www.hackerrank.com/challenges/s10-mcq-4/problem)

Table of Contents

- Table of Contents
- Notes on Conditional Probability
    - Conditional Probability
    - Bayes' Theorem
- Sample Problems
    - Sample Problem 1
        - Approach 1: Use first 3 events
        - Approach 2: Use 4th event
    - Sample Problem 2
        - Bayes Theorem solution:
- Math solution to Conditional Probability Challenge
- Monte Carlo Solution
    - Imports
    - Constants
    - Create families
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Notes on Conditional Probability

## Conditional Probability
**Conditional probability** is defined as the probability of an event occurring, assuming that one or more other events have already occurred. Two events, A and B are considered to be independent if event A has no effect on the probability of event B (i.e. P(B | A) = P(A)).

If events A and B are not independent, then we must consider the probability that both events occur.
This can be referred to as the intersection of events A and B:

\begin{equation}
\large
P(A \cap B) = P(B | A) * P(B)
\end{equation}

We can then use this definition to find the conditional probability by dividing the probability of the intersection of the two events (A $\cap$ B) by the probability of the event that is assumed to have already occurred (event A):

\begin{equation}
\large
P(B | A) =
\frac
{P(A \cap B)}
{P(A)}
\end{equation}

## Bayes' Theorem
Let A and B be two events such that P(A | B) denotes the probability of the occurrence of A given that B has occurred and P(B | A) denotes the probability of the occurrence of B given that A has occurred, then:

\begin{equation}
\large
P(A | B) =
\frac{ P(B | A) * P(A)}
{P(B)}
\end{equation}

\begin{equation}
\large
= \frac
{P(A | B) * P(A)}
{P(B | A) * P(A) + P(B | A_{C}) * P(A_{C})}
\end{equation}

*c subscript indicates complement.*

# Sample Problems
## Sample Problem 1

Question 1
If the probability of student A passing an exam is 2/7 and the probability of student B *failing* the exam is 3/7,
then find the probability that at least 1 of the 2 students will pass the exam.

We are given P(A) and P(Bc).

There are 4 possible events in our sample space:

1. A passes the exam and B fails P(A ∩ Bc).
2. B passes the exam and A fails P(B ∩ $A_{C}$).
3. A and B both pass the exam P(A ∩ B).
4. A and B both fail the exam P(Ac ∩ Bc).

We are interested in 3 of these events: 1, 2, and 3.

### Approach 1: Use first 3 events
Calculate the probability of events 1 - 3:

\begin{equation}
\large
P(\text{1+ students pass}) = P(\text{S1 passes}) + P(\text{S2 passes}) - P(\text{S1 & S2 pass})
\end{equation}

\begin{equation}
\large
= \frac{2}{7} + \frac{4}{7} - (\frac{2}{7} * \frac{4}{7})
\end{equation}

\begin{equation}
\large
=\frac{42}{49} - \frac{8}{49}
\end{equation}

\begin{equation}
\large
\frac{34}{49}
\end{equation}

### Approach 2: Use 4th event
Calculate the probability that both will fail the exam, and subtract that from the problem space P(S) to find the complement:

\begin{equation}
\large
P(\text{1+ students pass}) = P(S) - P({both fail})
\end{equation}

\begin{equation}
\large
= 1 - (\frac{5}{7} * \frac{3}{7})
\end{equation}

\begin{equation}
\large
= 1 - \frac{15}{49}
\end{equation}

\begin{equation}
\large
= \frac{34}{49}
\end{equation}

## Sample Problem 2

Historical data shows that it has only rained 5 days per year in some desert region (assuming a 365 day year). A meteorologist predicts that it will rain today.

- When it actually rains, the meteorologist correctly predicts rain 90% of the time.
- When it doesn't rain, the meteorologist incorrectly predicts rain 10% of the time.

Find the probability that it will rain today.

In this question, the probability of rain today depends on whether or not the meteorolgist predicted it will rain today.
We define the following events:

- Event R: It rains today. P(R) =  5/365 = 1/73
- Event Rc: It doesn't rain today. P(R) = 360/365 = 72/73
- Event M: The meteorologist predicted it will rain today:
    - P(M | R) = 9/10
    - P(M | Rc) = 1/10

Now we want to find the value of P(R|M):

### Bayes Theorem solution:
Given that P(E) is the probability of evidence (a.k.a. P(S) = 1)

\begin{equation}
\large
P(R | M) = \frac
{P(M|R) * P(R)}
{P(E)}
\end{equation}

\begin{equation}
\large
= \frac
{P(M | R) * P(R)}
{P(M | R) * P(R) + P(M | R_{C}) * P(R_{C}}
\end{equation}

\begin{equation}
\large
= \frac
{ \frac{9}{10} * \frac{1}{73} }
{\frac{9}{10} * \frac{1}{73} + \frac{1}{10} * \frac{72}{73}  }
\end{equation}

\begin{equation}
\large
= \frac
{\frac{9}{730}}
{\frac{9}{730} + \frac{72}{730} }
\end{equation}

\begin{equation}
\large
= \frac
{9}
{81}
\end{equation}

\begin{equation}
\large
\frac{1}{9}
\end{equation}

# Math solution to Conditional Probability Challenge

Suppose a family has 2 children, one of which is a boy. What is the probability that both children are boys?

*Note*: This is a [famous question in probability theory](https://en.wikipedia.org/wiki/Boy_or_Girl_paradox)

The wording on this question is a little difficult, but the logic followed makes sense:

Possible families:
- 1	:	(B, B)
- 2	:	(B, G)
- 3	:	(G, B)
- 4	:	(G, G)

We select only the families with at least one boy, thus we have 3 options:
- 1	:	(B, B)
- 2	:	(B, G)
- 3	:	(G, B)

What is the probability that we select the family with both boys?
1/3!

# Monte Carlo Solution

## Imports

```python
import random
from typing import List, Set
```

## Constants

```python
CHILDREN = 2
TYPES = ['B', 'G']
NEEDED_GENDER = 'B'
DESIRED_GENDER = 'G'
```

## Create families

```python
def create_all_possible_families(num_children: int, child_types: List[str]) -> List[List[str]]:
	"""Returns list of list of all possible family combinations.

	Referenced: https://www.geeksforgeeks.org/print-all-possible-combinations-of-r-elements-in-a-given-array-of-size-n/
	"""
	def helper(combos, combo) -> list:
		"""Helper recursive function to return all possible family combinations."""
		if len(combo) == num_children:		## Base case
			combos.append(combo)
			return combos

		for child in child_types:
			combos = helper(
				combos,
				combo + [child])
		return combos

	return helper([], [])
```

```python
def remove_without_gender(families: List[List[str]], needed_gender: str) -> List[List[str]]:
	"""Returns families after removing the needed gender from the family.

	If the gender is not inside the family, remove the gender.
	"""
	removed = 0
	for i in range(len(families)):
		family = families[i - removed]
		flag_removed_gender = False

		for j in range(len(family)):
			if family[j] == needed_gender:
				del family[j]
				flag_removed_gender = True

			if flag_removed_gender:
				break

		if not flag_removed_gender:
			del families[i - removed]
			removed += 1

	return families
```

```python
def get_ratio(families: List[List[str]], gender: str, iterations: int) -> float:
	"""Returns probabily that a child picked from the families is the specified gender."""
	number_picked = 0
	for _ in range(iterations):
		family = random.choice(families)
		child = random.choice(family)

		if child == gender:
			number_picked += 1

	return number_picked / iterations
```

## Main

```python
def main():
	possible_families = create_all_possible_families(
		num_children= CHILDREN,
		child_types= TYPES
	)

	allowed_families = remove_without_gender(possible_families, NEEDED_GENDER)

	iterations = 100000
	ratio = get_ratio(allowed_families, NEEDED_GENDER, iterations)
	print(ratio)		# approximates 1/3
```

```python
if __name__ == "__main__":
	main()
```

    0.33297


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Bistatic Radar Range Equation
***

The power at the receiving radar for a bistatic configuration is given by (Equation 4.60)

 \begin{equation}
     P_{radar} = \frac{P_t\, G_t(\theta, \phi)\, G_r(\theta, \phi)\, \sigma(\theta, \phi)\, \lambda^2}{(4\pi)^3\, r_t^2\, r_r^2} \hspace{0.5in} \text{(W)}
 \end{equation}
 ***

Begin by getting the library path

```python
import lib_path
```

Set the minimum and maximum range product (m)

```python
range_product_min = 1e6

range_product_max = 1e7
```

Import the `linspace` routine from `scipy`

```python
from numpy import linspace
```

Set up the range product array

```python
range_product = linspace(range_product_min, range_product_max, 2000)
```

Set up the transmit antenna gain (dB), receive antenna gain (dB),  bistatic target RCS (dBsm), peak transmit power (W), and the operating frequency (Hz)

```python
transmit_antenna_gain = 25.0

receive_antenna_gain = 30.0

bistatic_target_rcs = -10.0

peak_power = 50e3

frequency = 1e9
```

Set up the input args

```python
kwargs = {'transmit_target_range': 1.0,

          'receive_target_range': range_product,

          'peak_power': peak_power,

          'transmit_antenna_gain': 10 ** (transmit_antenna_gain / 10.0),

          'receive_antenna_gain': 10 ** (receive_antenna_gain / 10.0),

          'frequency': frequency,

          'bistatic_target_rcs': 10 ** (bistatic_target_rcs / 10.0)}
```

Import the `power_at_radar` routine from `bistatic_radar_range`

```python
from Libs.radar_range.bistatic_radar_range import power_at_radar
```

Calculate the power at the receiving radar

```python
power_at_radar = power_at_radar(**kwargs)
```

Import the `matplotlib` routines and `log10` from `scipy` for displaying the results

```python
from matplotlib import pyplot as plt

from numpy import log10
```

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(range_product / 1.0e6, power_at_radar, '')

# Set the plot title and labels

plt.title('Bistatic Power at the Receiver', size=14)

plt.xlabel('Range Product (km$^2$)', size=12)

plt.ylabel('Power at Receiver (W)', size=12)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)
```


# AM 201 Final Project

In this notebook, we set up a recurrent neural network:

$\dot{h}_i = - h_i + \sum_{j \neq i} J_{ij}\phi(h_j)$

Here, $\phi$ is a nonlinear activation function, typically chosen to be $tanh$

```python
! nvidia-smi
```

    Mon Dec 13 20:44:22 2021
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 495.44       Driver Version: 460.32.03    CUDA Version: 11.2     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   36C    P8    25W / 149W |      0MiB / 11441MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

$$J_{ij}\sim \mathcal{N}(0,J^2/N)$$

```python
! pip install cupy
```

    Collecting cupy
      Downloading cupy-10.0.0.tar.gz (1.7 MB)
         |████████████████████████████████| 1.7 MB 4.0 MB/s
    Requirement already satisfied: numpy<1.24,>=1.18 in /usr/local/lib/python3.7/dist-packages (from cupy) (1.19.5)
    Requirement already satisfied: fastrlock>=0.5 in /usr/local/lib/python3.7/dist-packages (from cupy) (0.8)
    Building wheels for collected packages: cupy
      Building wheel for cupy (setup.py) ... done
      Created wheel for cupy: filename=cupy-10.0.0-cp37-cp37m-linux_x86_64.whl size=95176805 sha256=1ebd8b35ead46b3d607651269eeccfb98253ba4b0bd44defb02be125ad8a9e6c
      Stored in directory: /root/.cache/pip/wheels/67/44/70/78a82ee1a02ce156d6972b2dd3e802218777d83ebdcb8d5e66
    Successfully built cupy
    Installing collected packages: cupy
    Successfully installed cupy-10.0.0

```python
import os
import sys
import time
import numpy as np
import cupy as cp
import matplotlib as mpl
from matplotlib import pyplot as plt
import cmath
from sympy import *
from sympy.abc import *
init_printing(use_latex=True)

from scipy.optimize import brentq
from scipy.integrate import odeint
from scipy.integrate import solve_ivp
from scipy import integrate
```

```python

```

    0.00018358230590820312

```python
#J is the connectivity matrix
#h is the vector of neural activations
def deriv_func_tanh(t,h,J):
  return -h + np.matmul(J,np.tanh(np.transpose(h)))

def genConnMatrix(N, JScale):
  J = np.random.normal(loc = 0, scale =JScale/np.sqrt(N), size = [N,N])
  for i in range(N):
    J[i,i]=0
  return J
```

```python
N = 1000
JScale = 1.5
totTime= 1000
numPts = 5000
J = genConnMatrix(N, JScale)
h_init = np.random.normal(loc=0, scale = 1, size=N)
tempFunc = lambda t, x: deriv_func_tanh(t,x,J)
```

```python
start = time.time()

sol = solve_ivp(tempFunc, (0,totTime), h_init, t_eval=np.linspace(0,totTime, numPts))

print(time.time() - start)
```

    2.975574493408203

```python

```

```python
start = time.time()

# Plot the mean-square average of the neuronal values in the network over time.

fig, ax = plt.subplots()
MSA = np.mean(np.square(sol['y']), axis = 0)
ax.plot(sol['t'],MSA)

print(time.time() - start)
```

```python
np.logspace(2,4, 3).tolist()
```

```python
fig, ax = plt.subplots()
for N in np.logspace(2,4,3).tolist():
  J = genConnMatrix(int(N), JScale)
  h_init = np.random.normal(loc=0, scale = 1, size=int(N))
  tempFunc = lambda t, x: deriv_func_tanh(t,x,J)
  sol = solve_ivp(tempFunc, (0,100), h_init, t_eval=np.linspace(0,100, 1000))
  ax.plot(sol['t'], np.sqrt(np.mean(np.square(sol['y']), axis = 0)), label = int(N))

ax.legend()
```

```python
N=1000
fig, ax = plt.subplots()
for JScale in np.logspace(-2, 2, 5).tolist():
  J = genConnMatrix(int(N), JScale)
  h_init = np.random.normal(loc=0, scale = 1, size=int(N))
  tempFunc = lambda t, x: deriv_func_tanh(t,x,J)
  sol = solve_ivp(tempFunc, (0,100), h_init, t_eval=np.linspace(0,100, 1000))
  ax.plot(sol['t'], np.sqrt(np.mean(np.square(sol['y']), axis = 0)), label = JScale)

ax.legend(title = 'J')
```

# TO DO
- Implement $V(\Delta)$ function in order to reproduce Figure 1
- Vary size of the network
- Vary gain parameter gJ
- Alternative nonlinearity to tanh
- Modify assumption of zero average so that we have a preferred type of synaptic connection
- Track the distance between intial conditions over time
- Short range synaptic matrices (


```python
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
import random
import sympy
```

    Bad key "text.kerning_factor" on line 4 in
    /home/sc/anaconda3/envs/old_nx/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test_patch.mplstyle.
    You probably need to get an updated matplotlibrc file from
    http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
    or from the matplotlib source distribution

```python
max_B = 10000 # maximum budget
no_runs = 100 # no. of runs to average
K = 4 # controls number of restarts

np.random.seed(0)
random.seed(42)
```

```python
edges = np.loadtxt('facebook_combined.txt',dtype=int)

G = nx.Graph()
G.add_edges_from(edges)

G_no_edges=G.number_of_edges()
G_no_nodes=G.number_of_nodes()
print(G_no_nodes,G_no_edges)
```

    4039 88234

```python
from louvain import detect_communities, modularity
'''
def draw_communities(G, node_size=90, alpha=1, k=None, randomized=False):
    partition = detect_communities(G, randomized=randomized)
    print("Modularity for best partition:", modularity(G, partition))
    community_map = {}
    for community, nodes in enumerate(partition):
        for node in nodes:
            community_map[node] = community

    cmap = plt.get_cmap("jet")
    plt.figure(figsize=(15, 15))
    pos = nx.spring_layout(G, k=k)
    indexed = [community_map.get(node) for node in G]
    plt.axis("off")
    nx.draw_networkx_nodes(G, pos=pos, cmap=cmap, node_color=indexed, node_size=node_size, alpha=alpha)
    nx.draw_networkx_edges(G, pos=pos, alpha=0.2)
    for x in range(len(pos)):
        pos[x] = pos[x]+np.array([0.02,0])
    labels = dict([(n, n) for n in G.nodes()]) #add labels
    #_ = nx.draw_networkx_labels(G, pos, labels=labels, font_color='#000000', font_size=15) #draw labels
draw_communities(G)
'''
```

    '\ndef draw_communities(G, node_size=90, alpha=1, k=None, randomized=False):\n    partition = detect_communities(G, randomized=randomized)\n    print("Modularity for best partition:", modularity(G, partition))\n    community_map = {}\n    for community, nodes in enumerate(partition):\n        for node in nodes:\n            community_map[node] = community\n    \n    cmap = plt.get_cmap("jet")\n    plt.figure(figsize=(15, 15))\n    pos = nx.spring_layout(G, k=k)\n    indexed = [community_map.get(node) for node in G]\n    plt.axis("off")\n    nx.draw_networkx_nodes(G, pos=pos, cmap=cmap, node_color=indexed, node_size=node_size, alpha=alpha)\n    nx.draw_networkx_edges(G, pos=pos, alpha=0.2)\n    for x in range(len(pos)):\n        pos[x] = pos[x]+np.array([0.02,0])\n    labels = dict([(n, n) for n in G.nodes()]) #add labels\n    #_ = nx.draw_networkx_labels(G, pos, labels=labels, font_color=\'#000000\', font_size=15) #draw labels\ndraw_communities(G)\n'

```python
partition = detect_communities(G, randomized=False)
community_map = {}
node_map = {}

for community, nodes in enumerate(partition):
    for node in nodes:
        node_map[node] = community
    community_map[community] = nodes
```

```python
def RDSRR_sampling(G,B,U=None):
    restart_ind = [10]
    t=20
    while restart_ind[-1]<B:
        restart_ind.append(restart_ind[-1]+int(80*np.log(t)))
        t+=1

    est_RW = []
    est_RW_t1 = 0
    est_RW_t2 = 0
    sample = np.random.choice(G.nodes())
    deg_pr_sent = G.degree(sample)
    est_RW_t1 += node_fn(sample)/deg_pr_sent
    est_RW_t2 += 1/deg_pr_sent
    est_RW.append(est_RW_t1/est_RW_t2)
    for ii in range(2,B+1):
        if ii in restart_ind:
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                sample = np.random.choice(U)
                #z = [node_map[x] for x in U].index(node_map[sample])
                #sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample = np.random.choice(neighbors)

        deg_pr_sent = G.degree(sample)
        est_RW_t1 += node_fn(sample)/deg_pr_sent
        est_RW_t2 += 1/deg_pr_sent
        est_RW.append(est_RW_t1/est_RW_t2)
    return np.array(est_RW)

def MHRR_sampling(G,B,U=None):
    restart_ind = [10]
    t=20
    while restart_ind[-1]<B:
        restart_ind.append(restart_ind[-1]+int(K*np.log(t)))
        t+=1

    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        if ii in restart_ind:
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                sample = np.random.choice(U)
                #z = [node_map[x] for x in U].index(node_map[sample])
                #sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample_t = np.random.choice(neighbors)
            if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
                sample = sample_t

        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)
```

## Finding High Degree Nodes in Clusters

```python
U = []

for com in list(community_map.keys()):
    nodes = community_map[com]
    maxd = -1
    maxn = -1
    for node in nodes:
        if G.degree(node)>maxd:
            maxd = G.degree(node)
            maxn = node
    U.append(maxn)
```

## F(v) = int(node_map[node] == 1)

```python
def node_fn(node):
    return int(node_map[node]==1)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.10670958157959891

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B,U)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDSRR')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR-T')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=1)
plt.grid()
```

## F(v) = int(G.degree(node)>100)

```python
def node_fn(node):
    return int(G.degree(node)>100)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.11908888338697697

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B,U)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDSRR')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR-T')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=1)
plt.grid()
```

## F(v) = isprime(v)

```python
def node_fn(node):
    return int(sympy.isprime(G.degree(node)))
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.2904184204010894

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B,U)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDSRR')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR-T')
plt.ylim(0,0.3)

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.grid()
```

## F(v) = random

```python
fn_mapping = np.random.exponential(1,size=(G_no_nodes))
def node_fn(node):
    return fn_mapping[node]

F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.9856442941970124

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B,U)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5,label='MHRR')
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5,label='RDSRR')
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5,label='RDSRR-T')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(0,0.15)
plt.grid()
```


# Wave propagation in Time Domain

In this section we discuss how to solve the PDE for 2D sonic wave equation in time domain using `esys.escript`. It is assumed that you have worked through the [introduction section on `esys.escript`](escriptBasics.ipynb).
First we will provide the basic theory.

## Sonic Wave Equation in Time domain

The sonic wave equation in time domain can be written as a system of first order differential equations:
\begin{equation}\label{eqWAVEF1}
\dot{\mathbf{V}} = - \nabla p
\end{equation}
\begin{equation}\label{eqWAVEF2}
\frac{1}{c^2} \dot{p} + \nabla^t \mathbf{V} = \delta_{\mathbf{x}_s} \cdot w(t)
\end{equation}
where $p$ is the pressure (or volume change) and
$\mathbf{V}$ it gradient. Source at location $\mathbf{x}_s$. $c$ is the wave propagation speed.
The dot refers time derivative. $\mathbf{x}_s$ is the location of the source,
$\delta_{\mathbf{x}_s}$ is Dirac $\delta$-function and $w(t)$ the source wavelet as function of time $t$.
In more details these equations are given as
\begin{equation}\label{eqWAVEF3}
\dot{\mathbf{V}} = -\begin{bmatrix}
\frac{\partial p}{\partial x_0 }\\
 \frac{\partial p}{\partial x_1 }
\end{bmatrix}
\end{equation}
\begin{equation}\label{eqWAVEF4}
\frac{1}{c^2} \dot{p} + \frac{\partial V_0}{\partial x_0 }+
 \frac{\partial V_1}{\partial x_1 } = \delta_{\mathbf{x}_s}  \cdot w(t)
\end{equation}
For the wavelet we use the [Ricker wavelet](http://subsurfwiki.org/wiki/Ricker_wavelet)

```python
f = 15       # peak frequency of the Ricker wavelet
dt=0.001     # time resolution
tend=0.75    # end of wavelet
%matplotlib inline
```

We use the implementation in `esys.escript` which is a bit more suitable for us them the [scipy version](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.ricker.html):

```python
from esys.escript import *
from esys.downunder import Ricker
wavelet=Ricker(f)
```

For plotting we evaluate the Ricker over the interval 0 to `tend`:

```python
import numpy as np
times=np.arange(0,tend, dt)
signal=wavelet.getValue(times)
```

```python

import matplotlib.pyplot as plt

plt.figure()# figsize=(7,7))
plt.plot(times, signal)
plt.xlabel('time')
plt.ylabel('amplitude')
plt.title(f'Ricker peak frequency ={f} Hz')
```

We also want to take a look at the power spectrum. We get the Fourier coefficients and frequencies using `np.fft`.
Notice that we are using the real value fast fourier transformation (FFT) as the values of `signal` are real.
As a consequence only the spectrum for positive frequencies needs to be inspected:

```python
fourier_full = np.fft.rfft(signal)
freq_all = np.fft.rfftfreq(signal.size, d=dt)
```

And we plot the power spectrum over frequency:

```python
plt.figure()# figsize=(7,7))
plt.plot(freq_all, abs(fourier_full))
plt.xlabel('frequency [Hz]')
plt.ylabel('power')
plt.show()
```

To assess the requirement is terms of grid sizes for solving the wave equation we need to have an idea of the
largest frequency making contributions to the signal of the wavelet. To do this we first
need to collect a list of all frequencies for which the spectrum lays above a threshold. Here we
use $0.001$ times the maximum spectrum value to get an index of all frequencies that have
significant contribution to the signal:

```python
freq_index=np.where( abs(fourier_full) > max(abs(fourier_full))*0.001 )[0]
```

Then we grab the relevant frequencies from the array of frequencies and get the max:

```python
fmax=max(freq_all[freq_index])
print(f"maximum freqency = {fmax} Hz")
```

    maximum freqency = 46.666666666666664 Hz

## Domain set up

We consider a single reflector set up. The reflector is located at a depth of $500m$
where the top layer has propagation speed of $c_{top}=1500m/s$
and the bottom layer has propagation speed of $c_{top}=3000m/s$.
Use a frequency of $f=5 Hz$. The domain has depth of $1km$ and width $3km$
with $300 \times 100$ grid.
The source is located at the surface at an offset of $1500m$ from the boundary.

This time we use a `sys.speckley` domain which is more suitable for wave problems. It actual uses higher
order polynomial approximations:

```python
nx=100  # elements in horizontal direction
ny=60   # elements in vertical direction
dx=15   # element size
order=6 # element order
```

```python
Width=nx*dx
Depth=ny*dx
print(f"Domain dimension is {Width} x {Depth}")
```

    Domain dimension is 1500 x 900

Again we use a single source at the center of the top edge of the domain:

```python
from esys.speckley import Rectangle
sources=[(Width/2, Depth) ]
sourcetags=["source66" ]
domain = Rectangle(order, n0=nx, n1=ny, l0=Width, l1=Depth, diracPoints=sources, diracTags=sourcetags)
```

Velocity configuration is

```python
c_top=1800      # m/s
c_bottom=3000      # m/s
d0=Depth-200.    # m top layer is 200m thick
```

Set up of the velocity $c$:

```python
X=ReducedFunction(domain).getX()
m=wherePositive(X[1]-d0)
c=c_top*m+c_bottom*(1-m)
c=interpolate(c, Function(domain))
```

We solve the wave equation using the Heun scheme which is 2-nd order Runge-Kutta method with time step size: Starting from $U^{(0)}$:
\begin{equation}\label{eq:H1}
\hat{U}^{(n+1)} =  U^{(n)} + h \cdot F(U^{(n)}, t^{(n)})
\end{equation}
\begin{equation}\label{eq:H2}
\hat{U}^{(n+1)} = U^{(n)} + \frac{h}{2} (F(U^{(n)}, t^{(n)})+F(\hat{U}^{(n+1)}, t^{(n+1)}) )
\end{equation}
In our case:
\begin{equation}\label{eqWAVEF8}
U=\begin{bmatrix}
\mathbf{V}\\
 p
\end{bmatrix}
\end{equation}
and
\begin{equation}\label{eqWAVEF9}
F=\begin{bmatrix}
-\nabla p\\
 dp
\end{bmatrix}
\end{equation}
with
\begin{equation}\label{eqWAVEF10}
\frac{1}{c^2} dp = - \nabla^t \mathbf{V} + \delta_{\mathbf{x}_s} \cdot w(t)
\end{equation}
for which we use the `LinearSinglePDE` with $D=\frac{1}{c^2}$, $X=-\mathbf{V}$ and
$y_{dirac}=\delta_{\mathbf{x}_s} \cdot w(t)$

Define the PDE:

```python
from esys.escript.linearPDEs import LinearSinglePDE, SolverOptions
pde = LinearSinglePDE(domain)
pde.getSolverOptions().setSolverMethod(SolverOptions.LUMPING)
```

Setting $D$ is the PDE:

```python
pde.setValue(D=1./(c**2) )
```

An define the Dirac $\delta$ function for the source `source66` which we will later multiply with the wavelet value $(w(t)$ to define $y_{dirac}$:

```python
input_loc=Scalar(0., DiracDeltaFunctions(domain))
input_loc.setTaggedValue("source66", 1.)
```

The wave length $\lambda$ needs to smaller then the node spacing in order to be able to resolve
the wave and its gradient accurately (note $2\pi f \lambda=c$):

```python
dx_max=inf(c/(2*np.pi*fmax))
dx_used=dx/order
print(f"maximum grid step size = {dx_max}. found = {dx_used}")
```

    maximum grid step size = 6.13883351925882. found = 2.5

## Time integrations

Also should the changes between to time steps not be two dramatic in the sense that the wave from should not
travel more than then a node distance within a time step. This leads to the condition $h<h_max$:

```python
h_max=inf(dx/c/order)
print(f"maximum time step size = {h_max}.")
```

    maximum time step size = 0.0008333333333333334.

** Note **: if the time step is two large then solution can become unstable. The exact maximum size
    depends on additional factors such as the time integration scheme and the spatial discretization scheme.
    In some an exact correction factor is known but here we use $h_{max}$ as a guidance to find the appropriate step size.

This is a Heun step:

```python
def stepHeun(t, p, v, n, h):
    for k in range(n):

        dv1=grad(p)

        a=wavelet.getValue(t+k*h)
        #print(t+k*h, a)
        pde.setValue(X=-v, y_dirac=input_loc*a)
        dp1=pde.getSolution()

        vp=v+dv1*h
        pp=p+dp1*h

        dv2=grad(pp)
        a=wavelet.getValue(t+k*h+h)
        pde.setValue(X=-vp, y_dirac=input_loc*a)
        dp2=pde.getSolution()

        v=v+(dv1+dv2)*h/2
        p=p+(dp1+dp2)*h/2

    return t+n*h, p, v
```

We want to track the solution at three locations at the surface:

```python
from esys.escript.pdetools import Locator
loc0=Locator(Solution(domain), x=( Width/2, Depth) )
loc1=Locator(Solution(domain), x=( Width/2*0.9, Depth) )
loc2=Locator(Solution(domain), x=( Width/2*0.5, Depth) )
```

And also collect over a set of virtual geophones to create a synthetic seismic survey:

```python
loc=Locator(Solution(domain), x=[ ( Width/2+z, Depth) for z in np.linspace(-500, 500, 80) ] )
```

We write a function to progress the time integration from some time `t` to some
time `tend` and collect the values at the geophones over time is lists:

```python
Ts=[]
Trace0=[]
Trace1=[]
Trace2=[]
Traces=[]
h=0.00004
n=int(dt/h)
print("n= ",n)

def progress(p, v, t, tend):
    while t < tend:
        t, p, v = stepHeun(t, p, v, n, h)
        Ts.append(t)
        Trace0.append(loc0(p))
        Trace1.append(loc1(p))
        Trace2.append(loc2(p))
        Traces.append(loc(p))
        #print(Ts[-1], Trace0[-1], Trace1[-1], Trace2[-1])
    return t, p, v
```

    n=  25

We set the initial values and progress to $20\%$ of the total end time. This will take some time:

```python
# start time integration
v=Vector(0., Function(domain))
p=Scalar(0., Solution(domain))
t=0

t, p, v = progress(p, v, t, tend*0.2)
```

Lets plot the solution at that time:

```python
p_np=convertToNumpy(p)
x_np=convertToNumpy(p.getFunctionSpace().getX())
plt.figure(figsize=(15,5))
plt.tricontourf(x_np[0], x_np[1], p_np[0], 20)
plt.xlabel('x0 [m]')
plt.ylabel('x1 [m]')
plt.title(f"Pressure at time t={t*1000.} msec")
plt.colorbar()
plt.show()
```

And now progress to $40\%$ end time:

```python
t, p, v = progress(p, v, t, tend*0.4)
```

Lets plot the solution at that time:

```python
p_np=convertToNumpy(p)
x_np=convertToNumpy(p.getFunctionSpace().getX())
plt.figure(figsize=(15,5))
plt.tricontourf(x_np[0], x_np[1], p_np[0], 20)
plt.xlabel('x0 [m]')
plt.ylabel('x1 [m]')
plt.title(f"Pressure at time t={t*1000.} msec")
plt.colorbar()
plt.show()
```

And now progress to 60%
end time:

```python
t, p, v = progress(p, v, t, tend*0.6)
```

Lets plot the solution at that time:

```python
p_np=convertToNumpy(p)
x_np=convertToNumpy(p.getFunctionSpace().getX())
plt.figure(figsize=(15,5))
plt.tricontourf(x_np[0], x_np[1], p_np[0], 20)
plt.xlabel('x0 [m]')
plt.ylabel('x1 [m]')
plt.title(f"Pressure at time t={t*1000.} msec")
plt.colorbar()
plt.show()
```

And now progress to 80% end time:

```python
t, p, v = progress(p, v, t, tend*0.8)
```

```python
p_np=convertToNumpy(p)
x_np=convertToNumpy(p.getFunctionSpace().getX())
plt.figure(figsize=(15,5))
plt.tricontourf(x_np[0], x_np[1], p_np[0], 20)
plt.xlabel('x0 [m]')
plt.ylabel('x1 [m]')
plt.title(f"Pressure at time t={t*1000.} msec")
plt.colorbar()
plt.show()
```

## Seismograms

Plot the traces at the monitoring points:

```python
plt.figure(figsize=(10,5))
plt.plot(Ts, Trace0, label="%s"%(loc0.getX()[0]))
plt.plot(Ts, Trace1, label="%s"%(loc1.getX()[0]))
plt.plot(Ts, Trace2, label="%s"%(loc2.getX()[0]))

plt.xlabel('x0 [m]')
plt.ylabel('aplitude [m]')
plt.title("Amplitude vs offset")
plt.legend()
```
Finally we can create the synthetic recordings as `numpy` array and plot them:

```python
traces=np.array(Traces)
```

```python
a=abs(traces).max()
plt.figure(figsize=(10,5))
plt.imshow(traces, extent=(0, 80*10, len(Traces),0), cmap='seismic', vmin=-a, vmax=a )
plt.colorbar()
```


# Spectral Estimation of Random Signals

*This jupyter/Python notebook is part of a [collection of notebooks](../index.ipynb) in the masters module [Digital Signal Processing](http://www.int.uni-rostock.de/Digitale-Signalverarbeitung.48.0.html), Comunications Engineering, Universität Rostock. Please direct questions and suggestions to <mailto:Sascha.Spors@uni-rostock.de>.*

## Parametric Methods

### Motivation

Non-parametric methods for the estimation of the power spectral density (PSD), like the [periodogram](periodogram.ipynb) or [Welch's method](welch_method.ipynb), don't rely on a-priori information about the process generating the random signal. Often some a-priori information is available that can be used to formulate a parametric model of the random process. The goal is then to estimate these parameters in order to characterize the random signal. Such techniques are known as *[parametric methods](https://en.wikipedia.org/wiki/Spectral_density_estimation#Parametric_estimation)* or *model-based methods*. The incorporation of a-priori knowledge can improve the estimation of the PSD significantly, as long as the underlying model is a valid description of the random process. The parametric model of the random process can also be used to generate random signals with a desired PSD.

### Process Models

For the remainder we assume wide-sense stationary real-valued random processes. For many applications the process can be modeled by a linear-time invariant (LTI) system

where $n[k]$ is [white noise](../random_signals/white_noise.ipynb) and $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ denotes the transfer function of the system. In general, the random signal $x[k]$ will be correlated as a result of the processing of the uncorrelated input signal $n[k]$ by the system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. Due to the white noise assumption $\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0$, the PSD of the random process is given as

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0 \cdot | H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2
\end{equation}

Parametric methods model the system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ by a limited number of parameters. These parameters are then estimated from $x[k]$, providing an estimate $\hat{H}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the transfer function. This estimate is then used to calculate the desired estimate $\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the PSD.

#### Autoregressive model

The [autoregressive](https://en.wikipedia.org/wiki/Autoregressive_model) (AR) model assumes a recursive system with a direct path. Its output relation is given as

\begin{equation}
x[k] = \sum_{n=1}^{N} a_n \cdot x[k-n] + n[k]
\end{equation}

where $a_n$ denote the coefficients of the recursive path and $N$ the order of the model. Its system function $H(z)$ is derived by $z$-transformation of the output relation

\begin{equation}
H(z) = \frac{1}{1 - \sum_{n=1}^{N} a_n z^{-n}}
\end{equation}

Hence, the AR model is a pole-only model of the system.

#### Moving average model

The [moving average](https://en.wikipedia.org/wiki/Moving-average_model) (MA) model assumes a non-recursive system. The output relation is given as

\begin{equation}
x[k] = \sum_{m=0}^{M-1} b_m \cdot n[k-m] = h[k] * n[k]
\end{equation}

with the impulse response of the system $h[k] = [ b_0, b_1, \dots, b_{M-1} ]$. The MA model is a finite impulse response (FIR) model of the random process. Its system function is given as

\begin{equation}
H(z) = \mathcal{Z} \{ h[k] \} = \sum_{m=0}^{M-1} b_m \; z^{-m}
\end{equation}

#### Autoregressive moving average model

The [autoregressive moving average](https://en.wikipedia.org/wiki/Autoregressive%E2%80%93moving-average_model) (ARMA) model is a combination of the AR and MA model. It constitutes a general linear process model. Its output relation is given as

\begin{equation}
x[k] = \sum_{n=1}^{N} a_n \cdot x[k-n] + \sum_{m=0}^{M-1} b_m \cdot n[k-m]
\end{equation}

Its system function reads

\begin{equation}
H(z) = \frac{\sum_{m=0}^{M-1} b_m \; z^{-m}}{1 - \sum_{n=1}^{N} a_n z^{-n}}
\end{equation}

### Parametric Spectral Estimation

The models above describe the synthesis of the samples $x[k]$ from the white noise $n[k]$. For spectral estimation only the random signal $x[k]$ is known and we are aiming at estimating the parameters of the model. This can be achieved by determining an analyzing system $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ such to decorrelate the signal $x[k]$

where $e[k]$ should be white noise. Due to its desired operation, the filter $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is also denoted as *whitening filter*. The optimal filter $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given by the inverse system $\frac{1}{H(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$. However, $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is in general not known. But this nevertheless implies that our linear process model of $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ also applies to $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. Various techniques have been developed to estimate the parameters of the filter $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ such that $e[k]$ becomes decorrelated. For instance, by expressing the auto-correlation function (ACF) $\varphi_{xx}[\kappa]$ in terms of the model parameters and solving with respect to these. The underlying set of equations are known as [Yule-Walker equations](https://en.wikipedia.org/wiki/Autoregressive_model#Yule-Walker_equations).

Once the model parameters have been estimated, these can be used to calculate an estimate $\hat{G}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the analysis system. The desired estimate of the PSD is then given as

\begin{equation}
\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{|\hat{G}(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2}
\end{equation}

where if $e[k]$ is white noise, $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0$.

### Example

In the follwing example $n[k]$ is drawn from normal distributed white noise with $N_0 = 1$. The Yule-Walker equations are used to estimate the parameters of an AR model of $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The implementation provided by `statsmodels.api.regression.yule_walker` returns the estimated AR coefficients of the system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. These parameters are then used to numerically evaluate the estimated transfer function, resulting in $\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 1 \cdot | \hat{H}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2 $.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import scipy.signal as sig

K = 4096  # length of random signal
N = 3  # order of AR model
a = np.array((1, -1, .5))  # coefficients of AR model

# generate random signal n[k]
np.random.seed(2)
n = np.random.normal(size=K)

# AR model for random signal x[k]
x = np.zeros(K)
for k in np.arange(3, K):
    x[k] = a[0]*x[k-1] + a[1]*x[k-2] + a[2]*x[k-3] + n[k]

# estimate AR parameters by Yule-Walker method
rho, sigma = sm.regression.yule_walker(x, order=N, method='mle')

# compute true and estimated transfer function
Om, H = sig.freqz(1, np.insert(-a, 0, 1))
Om, He = sig.freqz(1, np.insert(-rho, 0, 1))
# compute PSD by Welch method
Om2, Pxx = sig.welch(x, return_onesided=True)

# plot PSDs
plt.figure(figsize=(10,5))
plt.plot(Om, np.abs(H)**2, label=r'$\Phi_{xx}(e^{j\Omega})$')
plt.plot(Om2*2*np.pi, .5*np.abs(Pxx), 'k-', alpha=.5 , label=r'$\hat{\Phi}_{xx}(e^{j\Omega})$ (Welch)')
plt.plot(Om, np.abs(He)**2, label=r'$\hat{\Phi}_{xx}(e^{j\Omega})$ (parametric)')

plt.xlabel(r'$\Omega$')
plt.axis([0, np.pi, 0, 20])
plt.legend()
plt.grid()
```

**Exercise**

* Change the order `N` of the AR model used for estimation by the Yule-Walker equations. What happens if the order is smaller or higher than the order of the true system? Why?

* Change the number of samples `K`. Is the estimator consistent?

Solution: Choosing the order of the estimated AR model differently from the true process results in a mismatch of the model with the consequence of potentially large deviations between the estimated PSD and true PSD. These deviations are typically larger when choosing the order smaller since the model does definitely not fit to the true process. However, choosing the order larger is typically not that problematic since some of the AR coefficients are estimated as approximately zero due to the lower order of the process generating the random signal. Increasing the number of samples seems to lower the bias and variance of the estimated PSD, the estimator can therefore be assumed to be consistent.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


```
# default_exp
```

```
#hide
#all_slow
```

# Distribution of maximum for 1-D posterior example
> Investigation into the distribution of the maximum for a 1-D toy example.

```
#hide
from dolfin import *
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import matplotlib.cm as cm
# plt.rcParams['figure.figsize'] = (10,6)
from scipy.stats import multivariate_normal, linregress
from scipy import integrate
from scipy.spatial.distance import cdist, pdist, squareform
from scipy.linalg import sqrtm
from scipy.interpolate import interp1d
import sympy; sympy.init_printing()
from tqdm.notebook import tqdm
# code for displaying matrices nicely
def display_matrix(m):
    display(sympy.Matrix(m))
# import required functions from oneDim
from statFEM.oneDim import mean_assembler, cov_assembler, kernMat, m_post, gen_sensor, MyExpression, m_post_fem_assembler, c_post, c_post_fem_assembler, sample_gp
```

Let's first import the `wass` function created in the sub-module `maxDist`.

```
from statFEM.maxDist import wass
```

# Sampling from true maxiumum distribution in 1D

We will be interested in the toy example introduced in oneDim, however we will use different values for the parameters in the distribution of $f$. To obtain a sample for the maximum from the true posterior we must first sample trajectories from the true posterior.

```
# set up mean and kernel functions
l_f = 0.01
σ_f = 0.1

def m_f(x):
    return 1.0

def c_f(x,y):
    return (σ_f**2)*np.exp(-(x-y)**2/(2*(l_f**2)))

# translation invariant form of c_f
def k_f(x):
    return (σ_f**2)*np.exp(-(x**2)/(2*(l_f**2)))

# mean of forcing for use in FEniCS
f_bar = Constant(1.0)

# true prior solution mean
μ_true = Expression('0.5*x[0]*(1-x[0])',degree=2)
```

```
# set up true prior cov function for solution
# compute inner integral over t
def η(w,y):
    I_1 = integrate.quad(lambda t: t*c_f(w,t),0.0,y)[0]
    I_2 = integrate.quad(lambda t: (1-t)*c_f(w,t),y,1.0)[0]
    return (1-y)*I_1 + y*I_2

# use this function eta and compute the outer integral over w
def c_u(x,y):
    I_1 = integrate.quad(lambda w: (1-w)*η(w,y),x,1.0)[0]
    I_2 = integrate.quad(lambda w: w*η(w,y),0.0,x)[0]
    return x*I_1 + (1-x)*I_2
```

We will also need a function `u_quad` to accurately compute the solution for a given realisation of $f$ in order to generate sensor data. This is set up below:

```
def u_quad(x,f,maxiter=50):
    I_1 = integrate.quadrature(lambda w: w*f(w), 0.0, x,maxiter=maxiter)[0]
    I_2 = integrate.quadrature(lambda w: (1-w)*f(w),x, 1.0,maxiter=maxiter)[0]
    return (1-x)*I_1 + x*I_2
```

We now set up a reference grid on which we will simulate trajectories. We take a grid of length $N = 41$.

```
N = 41
grid = np.linspace(0,1,N)
```

We now set up the sensor grid and the true prior covariance on this sensor grid which will be needed in all further computations. We also set up the function which gives the vector $\{c_u(x,y_i)\}_{i=1}^{s}$ needed for the posterior.

```
s = 10 # number of sensors
# create sensor grid
Y = np.linspace(0.01,0.99,s)[::-1]
# get true prior covariance on sensor grid
C_true_s = kernMat(c_u,Y.flatten())
# create function to compute vector mentioned above
def c_u_vect(x):
    return np.array([c_u(x,y_i) for y_i in Y])
```

We now set up a function to get the statFEM prior and posterior for a FE mesh of size $h$, using functions from oneDim.

```
# set up function to compute fem_prior
def fem_prior(h,f_bar,k_f,grid):
    J = int(np.round(1/h))
    μ = mean_assembler(h,f_bar)
    Σ = cov_assembler(J,k_f,grid,False,True)
    return μ,Σ
```

```
# set up function to compute statFEM posterior
def fem_posterior(h,f_bar,k_f,ϵ,Y,v_dat,grid):
    J = int(np.round(1/h))
    m_post_fem = m_post_fem_assembler(J,f_bar,k_f,ϵ,Y,v_dat)
    μ = MyExpression()
    μ.f = m_post_fem
    Σ = c_post_fem_assembler(J,k_f,grid,Y,ϵ,False,True)
    return μ,Σ
```

Let's see how different levels of sensor noise change the statFEM posterior from the statFEM prior.

```
ϵ_list = [0.0001,0.0005,0.001,0.01]
```

```
h = 0.05
μ_prior, Σ_prior = fem_prior(h,f_bar,k_f,grid)
posteriors = {}
sensor_dat = {}
np.random.seed(12345)
for ϵ in ϵ_list:
    v_dat = gen_sensor(ϵ,m_f,k_f,Y,u_quad,grid,maxiter=400)
    sensor_dat[ϵ] = v_dat
    μ_posterior, Σ_posterior = fem_posterior(h,f_bar,k_f,ϵ,Y,v_dat,grid)
    posteriors[ϵ] = (μ_posterior,Σ_posterior)
```

```
#hide_input
plt.figure(figsize=(20,15))
J_plot = 50
for (i,ϵ) in enumerate(ϵ_list):
    μ_posterior, Σ_posterior = posteriors[ϵ]
    v_dat = sensor_dat[ϵ]
    plt.subplot(2,2,i + 1)
    plot(μ_prior,mesh=UnitIntervalMesh(J_plot),label='prior mean')
    plot(μ_posterior,mesh=UnitIntervalMesh(J_plot),label='posterior mean')
    plt.scatter(Y,v_dat,c='red',linewidth=3,marker='x',label='sensor observations')
    plt.grid()
    plt.xlabel("$x$")
    plt.ylabel("$u$")
    plt.ylim(0,0.15)
    plt.title('Prior and Posterior Means, ϵ: %.4f' % ϵ)
    plt.legend()
plt.show()
```

Let's also plot the prior covariances next to the posterior covariances

```
#hide_input
fig, axs = plt.subplots(4,3,figsize=(24,28))
for (i,ϵ) in enumerate(ϵ_list):
    μ_posterior, Σ_posterior = posteriors[ϵ]
    sns.heatmap(Σ_prior,cbar=True,
              annot=False,
              xticklabels=False,
              yticklabels=False,
              cmap=cm.viridis,
              ax=axs[i,0])
    axs[i,0].set_title('Prior covariance matrix, ϵ: %.4f' % ϵ)\

    sns.heatmap(Σ_posterior,cbar=True,
                  annot=False,
                  xticklabels=False,
                  yticklabels=False,
                  cmap=cm.viridis,
                  ax=axs[i,1])
    axs[i,1].set_title('Posterior covariance matrix, ϵ: %.4f' % ϵ)

    axs[i,2].plot(grid,np.diag(Σ_prior),label='prior')
    axs[i,2].plot(grid,np.diag(Σ_posterior),label='posterior')
    axs[i,2].grid()
    axs[i,2].set_xlabel('$x$')
    axs[i,2].set_ylabel('$\operatorname{var}(u(x))$')
    axs[i,2].set_title('Variance at each grid point, ϵ: %.4f' % ϵ)
    axs[i,2].legend()
plt.show()
```

Let's now consider a particular example to see if the code works and also to calibrate the number of bins. We now generate sensor data and set up the true posterior mean and covariance functions.

```
# sensor noise level
ϵ = ϵ_list[2]
print('ϵ: ',ϵ)
np.random.seed(235)
v_dat = gen_sensor(ϵ,m_f,k_f,Y,u_quad,grid,maxiter=300)

# get true B mat required for posterior
B_true = (ϵ**2)*np.eye(s) + C_true_s

# set up true posterior mean
def true_mean(x):
    return m_post(x,μ_true,c_u_vect,v_dat,Y,B_true)

# set up true posterior covariance
def c_post_true(x,y):
    return c_post(x,y,c_u,Y,B_true)
```

    ϵ:  0.001

We now sample trajectories from the true posterior and plot these.

```
%%time
n_sim = 1000
np.random.seed(12345)
u_sim = sample_gp(n_sim, true_mean, c_post_true, grid,True,False)
```

    CPU times: user 11.5 s, sys: 0 ns, total: 11.5 s
    Wall time: 3min 58s

```
#hide_input
plt.rcParams['figure.figsize'] = (10,6)
plt.plot(grid,u_sim)
plt.xlabel('$x$')
plt.ylabel('$u$')
plt.title('Trajectories from the true posterior')
plt.grid()
plt.show()
```

Let's now get the maximum value for each simulated curve and store this in a variable `max_true` for future use. We plot the histogram for these maximum values.

```
max_true = u_sim.max(axis=0)
```

```
#hide_input
sns.displot(max_true,kde=True,stat="density")
plt.xlabel('$\max$')
plt.title('Histogram of maximum for true posterior')
plt.show()
```

# Sampling from statFEM posterior maxiumum distribution in 1D

We now create a function to utilise FEniCS to draw trajectories from the statFEM posterior.

```
def statFEM_posterior_sampler(n_sim, grid, h, f_bar, k_f, ϵ, Y, v_dat, par = False, trans = True, tol = 1e-9):
    # get length of grid
    d = len(grid)

    # get size of FE mesh
    J = int(np.round(1/h))

    # get statFEM posterior mean function
    m_post_fem = m_post_fem_assembler(J,f_bar,k_f,ϵ,Y,v_dat)
    μ_func = MyExpression()
    μ_func.f = m_post_fem

    # evaluate this on the grid
    μ = np.array([μ_func(x) for x in grid]).reshape(d,1)

    # get statFEM posterior cov mat on grid
    Σ = c_post_fem_assembler(J,k_f,grid,Y,ϵ,False,True)

    # construct the cholesky decomposition Σ = GG^T
    # we add a small diagonal perturbation to Σ to ensure it
    # strictly positive definite
    G = np.linalg.cholesky(Σ + tol * np.eye(d))

    # draw iid standard normal random vectors
    Z = np.random.normal(size=(d,n_sim))

    # construct samples from GP(m,k)
    Y = G@Z + np.tile(μ,n_sim)

    # return the sampled trajectories
    return Y
```

Let's test this function out by using it to obtain samples from the statFEM posterior for a particular value of $h$.

```
%%time
h = 0.1
np.random.seed(3542)
u_sim_statFEM = statFEM_posterior_sampler(n_sim, grid, h, f_bar, k_f, ϵ, Y, v_dat)
```

    CPU times: user 82 ms, sys: 9.38 ms, total: 91.4 ms
    Wall time: 70.4 ms

Let's plot the above trajectories.

```
#hide_input
plt.rcParams['figure.figsize'] = (10,6)
plt.plot(grid,u_sim_statFEM)
plt.grid()
plt.xlabel('$x$')
plt.ylabel('$u$')
plt.title('Trajectories from statFEM posterior with $h = $ %0.1f' % h)
plt.show()
```

Let's now get the maximum value for each simulated curve and store this in a variable `max_statFEM` for future use. We then plot the histograms for these maximum values together with those from the true posterior.

```
max_statFEM = u_sim_statFEM.max(axis=0)
```

```
#hide_input
max_data = pd.DataFrame(data={'True': max_true, 'statFEM': max_statFEM})
sns.displot(max_data,kde=True,stat="density")
plt.xlabel('$\max$')
plt.title('Histograms of maximums for true and statFEM posteriors')
plt.show()
```

Our `wass` function requires the number of bins to be specified. Let's investigate how varying this parameter affects the computed distance between the true maximums and our statFEM maximums.

```
n_bins = np.arange(10,200,10)
wass_bin_dat = [wass(max_true,max_statFEM,n) for n in n_bins]
```

```
#hide_input
plt.plot(n_bins,wass_bin_dat)
plt.grid()
plt.xlabel('number of bins')
plt.ylabel('2-Wasser distance')
plt.title('Plot showing how number of bins influences the computed distance')
plt.show()
```

From the plot above we see that the distance seems to stabilize after around $100$ bins. We choose to take `n_bins=150` for the rest of this example.

We now set up a range of $h$-values to use for the statFEM posterior.

```
# set up range of h values to use
h_range_tmp = np.linspace(0.25,0.02,100)
h_range = 1/np.unique(np.round(1/h_range_tmp))
np.round(h_range,2)
```

    array([0.25, 0.2 , 0.17, 0.14, 0.12, 0.11, 0.1 , 0.09, 0.08, 0.08, 0.07,
           0.07, 0.06, 0.06, 0.06, 0.05, 0.05, 0.05, 0.05, 0.04, 0.04, 0.04,
           0.04, 0.03, 0.03, 0.03, 0.03, 0.02, 0.02, 0.02])

We now loop over these $h$-values, and for each value we simulate maximums from the statFEM posterior and then compute the 2-Wasserstein distance between these maximums and those from the true posterior.

```
%%time
errors = []
###################
n_bins = 150
##################
np.random.seed(3252)
for h in tqdm(h_range):
    # sample trajectories from statFEM prior for current h value
    sim = statFEM_posterior_sampler(n_sim, grid, h, f_bar, k_f, ϵ, Y, v_dat)
    # get max
    max_sim = sim.max(axis=0)
    # compute error
    error = wass(max_true,max_sim,n_bins)
    # append to errors
    errors.append(error)
```

      0%|          | 0/30 [00:00<?, ?it/s]

    CPU times: user 4.97 s, sys: 11.7 s, total: 16.7 s
    Wall time: 2.29 s

```
#hide
errors = np.array(errors)
```

We see that as $h$ decreases so too does the distance/error. Let's investigate the rate by plotting this data in log-log space and then estimating the slope of the line of best fit.

```
#hide_input
plt.plot(h_range,errors)
plt.grid()
plt.xlabel('$h$')
plt.ylabel('error')
plt.title('Plot of errors vs $h$ for $\max$')
plt.show()
```

We see that as $h$ decreases so too does the distance/error. Let's investigate the rate by plotting this data in log-log space and then estimating the slope of the line of best fit.

```
#hide_input
log_h_range = np.log(h_range)
log_errors = np.log(errors)

# perform linear regression to get line of best fit for the log scales:
lm = linregress(log_h_range,log_errors)
print("slope: %f    intercept: %f    r value: %f     p value: %f" % (lm.slope, lm.intercept, lm.rvalue, lm.pvalue))

# plot line of best fit with the data:
x = np.linspace(np.min(log_h_range),np.max(log_h_range),100)
plt.scatter(log_h_range,log_errors,label='data')
plt.plot(x,lm.intercept + lm.slope * x, 'r', label='fitted line')
plt.grid()
plt.legend()
plt.xlabel('$\log(h)$')
plt.ylabel('$\log(W)$')
plt.title('Plot of $\log(W)$ vs $\log(h)$')
plt.savefig('1D_prior_results.png',dpi=300,bbox_inches='tight',facecolor="w")
plt.show()
```

We can see from the above plot that the slope is around $1.42$ - this differs from our theory showing the results for the $\max$ are out of the remit of our theory.

We now repeat the above for different levels of sensor noise.

```
results = {}
```

```
%%time
np.random.seed(345346)
tol = 1e-10
n_sim = 1000
for ϵ in tqdm(ϵ_list,desc='Eps loop'):
    # generate sensor data
    v_dat = gen_sensor(ϵ,m_f,k_f,Y,u_quad,grid,maxiter=400)
    # get true B mat required for posterior
    B_true = (ϵ**2)*np.eye(s) + C_true_s

    # set up true posterior mean
    def true_mean(x):
        return m_post(x,μ_true,c_u_vect,v_dat,Y,B_true)

    # set up true posterior covariance
    def c_post_true(x,y):
        return c_post(x,y,c_u,Y,B_true)

    u_sim = sample_gp(n_sim, true_mean, c_post_true, grid,True,False)

    max_true = u_sim.max(axis=0)

    errors = []

    for h in tqdm(h_range):
        # sample trajectories from statFEM prior for current h value
        sim = statFEM_posterior_sampler(n_sim, grid, h, f_bar, k_f, ϵ, Y, v_dat)
        # get max
        max_sim = sim.max(axis=0)
        # compute error
        error = wass(max_true,max_sim,n_bins)
        # append to errors
        errors.append(error)

    results[ϵ] = (errors,u_sim)
```

    Eps loop:   0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/30 [00:00<?, ?it/s]

      0%|          | 0/30 [00:00<?, ?it/s]

      0%|          | 0/30 [00:00<?, ?it/s]

      0%|          | 0/30 [00:00<?, ?it/s]

    CPU times: user 1min 39s, sys: 1min 10s, total: 2min 49s
    Wall time: 17min 48s

```
#hide_input
plt.plot()
plt.grid()
plt.xlabel('$h$')
plt.ylabel('error')
plt.title('Plot of errors vs $h$ for different levels of sensor noise')
for ϵ in ϵ_list:
    errors = results[ϵ][0]
    plt.plot(h_range,errors, label = 'ϵ = ' + str(ϵ))
plt.legend()
plt.show()
```

```
#hide_input
plt.plot()
plt.grid()
plt.xlabel('$\log(h)$')
plt.ylabel('$\log(W)$')
plt.title('Plot of $\log(W)$ vs $\log(h)$ for different levels of sensor noise')
log_h_range = np.log(h_range)
for ϵ in ϵ_list:
    errors = results[ϵ][0]
    log_errors = np.log(errors)
    lm = linregress(log_h_range,log_errors)
    print('ϵ: %.5f,    slope: %.4f,    intercept: %.4f' % (ϵ, lm.slope,lm.intercept))
    plt.scatter(log_h_range,log_errors)
    plt.plot(x,lm.intercept + lm.slope * x, label='ϵ = ' +str(ϵ))
plt.legend()
# plt.savefig('1D_max_posterior_results.png',dpi=300,bbox_inches='tight',facecolor="w")
plt.show()
```

We can see from these results that for we obtain different results for the rates. In particular, we observe the rate increasing from $1.3$ when $\epsilon=0.01$ up to around $2.2$ when $\epsilon$ decreases to $0.0001$. Thus, for small values of sensor noise we obtain rates similar to the case of linear quantities of interest. This is potentially do to the fact that for small levels of sensor noise the posterior trajectories concentrate and so the maximum is almost always located at the maximum of the true prior mean. Thus, we come to the regime where taking the maximum is the same as point evaluation, which is a linear functional.


```python
from sympy import *
import numpy as np
import matplotlib.pyplot as plt
```

```python
x = Symbol('x')# x sada postaje simbolicka prom
```

```python
y = (x-3)**2 #funkcija
```

```python
dy = y.diff(x) #prvi izvod fje y
```

```python
dy
```

    2*x - 6

```python
def plotF():
    space = np.linspace(-5,5,100)
    data = np.array([N(y.subs(x,value)) for value in space])
    plt.plot(space, data)
    plt.show()
```

t1 = t0 - a*dy(t0)

```python
t0 = 84 #pocetna tacka
t1 = 0
a = 0.01
```

```python
iteracija = 0
provera = 0
preciznost = 1/1000000
```

```python
plot = True
iteracijaMaks = 10000 #najveci broj iteracija posle kojih treba da se odustane
```

```python
divergencijaMaks = 50 #parametar za sprecavanje divergencije
```

```python
while True:
    t1 = t0 - a*N(dy.subs(x, t0)).evalf()
    #dy.subs direktno menja t0 da bismo izracunali dy(t0)
    iteracija+=1 #povecaj broj iteracija

    #ako ima previse iteracija to znaci da verovatno param nisu ok
    if iteracija>iteracijaMaks:
        print("Previse iteracija")
        break

    #sada ide provera da li t0 > t1 ako nije onda dozvoljavamo
    #najvise 50 divergiranja
    if t0<t1:
        print("t0 divergira")
        provera+=1
        if provera>divergencijaMaks:
            print("Previse iteracija (%s), t0 divergira"%divergencijaMaks)
            print("Manje a ili proveriti da li fja konvekna")
            plot = False
            break
    #sada ide uslov kojim mi zakljucujemo da t0 konvergira
    #to je zapravo ovo t0-t1< preciznosti i tako izlazimo iz petlje
    if abs(t0-t1)<preciznost:
        break

    #obnavlajmo vrednost za sledecu iteracijuw
    t0=t1

if plot:
    print("Broj iteracija",iteracija,"t1=",t1)
    plt.plot(t0,N(y.subs(x,t0)).evalf(),marker='o',color='r')
    plotF()
```

```python

```