```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym

import pypwt
import solowpy
```

# 2. Computing the steady state

Traditionally, most analysis of the Solow model focuses almost excusively on the long run steady state of the model. Recall that the steady state of the Solow model is the value of capital stock (per unit effective labor) that solves

$$ 0 = sf(k^*) - (g + n + \delta)k^*. \tag{2.0.1} $$

In words: in the long-run, capital stock (per unit effective labor) converges to the value that balances actual investment, $sf(k)$, with effective depreciation, $(g + n + \delta)$. Given the assumption made about the aggregate production technology, $F$, and its intensive form, $f$, there is always a unique value $k^* >0$ satisfying equation 2.0.1.

```python
# define model parameters
ces_params = {'A0': 1.0, 'L0': 1.0, 'g': 0.02, 'n': 0.03, 's': 0.15,
              'delta': 0.05, 'alpha': 0.33, 'sigma': 0.95}

# create an instance of the solow.Model class
ces_model = solowpy.CESModel(params=ces_params)
```

## 2.1 Analytic results
For many (all?) of the commonly used functional forms for $F$ one can easily derive a closed-form expression for the steady state value of capital stock (per unit effective labor). For example, assuming $F$ is constant elasticity of substitution (CES), the analytical solution for $k^*$ is

$$ k^* = \left[\frac{1-\alpha}{\bigg(\frac{g+n+\delta}{s}\bigg)^{\rho}-\alpha}\right]^{\frac{1}{\rho}} \tag{2.1.0} $$

This analytic solution is available via the `steady_state` attribute.

```python
# check the docstring...
ces_model.steady_state?
```

```python
ces_model.steady_state
```

    1.8258317310596595

## 2.2 Numerical methods
Although it is trivial to derive an analytic expression for the long-run equilibrium of the Solow model for most intensive production functions, the Solow model serves as a good illustrative case for various numerical methods for solving non-linear equations.

The `solow.Model.find_steady_state` method provides a simple interface to the various 1D root finding routines available in `scipy.optimize` and uses them to solve the non-linear equation 2.0.1. To see the list of currently supported methods, check out the docstring for the `Model.find_steady_state` method...

```python
solow.Model.find_steady_state?
```

### Example usage
Guessing that the true steady state value lies somewhere between `1e-6` and `1e6`, compute the steady state value of capital stock (per unit effective labor) using the bisection method.

```python
k_star, result = ces_model.find_steady_state(1e-6, 1e6, method='bisect', full_output=True)
```

We can display the value and confirm that the algorithm did indeed converge as follows.

```python
print("The steady-state value is {}".format(k_star))
print("Did the bisection algorithm coverge? {}".format(result.converged))
```

    The steady-state value is 1.82583173149
    Did the bisection algorithm coverge? True

### 2.1.1 Comparing the various methods
The various methods, for the most part, are equally accurate...

```python
valid_methods = ['brenth', 'brentq', 'ridder', 'bisect']

for method in valid_methods:
    actual_ss = ces_model.find_steady_state(1e-6, 1e6, method=method)
    expected_ss = ces_model.steady_state

    print("Steady state value computed using {} is {}".format(method, actual_ss))
    print("Absolute error in is {}\n".format(abs(actual_ss - expected_ss)))
```

    Steady state value computed using brenth is 1.82583173106
    Absolute error in is 1.08801856413e-14

    Steady state value computed using brentq is 1.82583173106
    Absolute error in is 2.84217094304e-14

    Steady state value computed using ridder is 1.82583173115
    Absolute error in is 9.1334051433e-11

    Steady state value computed using bisect is 1.82583173149
    Absolute error in is 4.32963442876e-10

...however the `brentq` and `brenth` routines are generally more efficient.

```python
valid_methods = ['brenth', 'brentq', 'ridder', 'bisect']

for method in valid_methods:
    print("Profiling results using {}:".format(method))
    %timeit -n 1 -r 3 ces_model.find_steady_state(1e-6, 1e6, method=method)
    print("")
```

    Profiling results using brenth:
    1 loops, best of 3: 112 µs per loop

    Profiling results using brentq:
    1 loops, best of 3: 154 µs per loop

    Profiling results using ridder:
    1 loops, best of 3: 323 µs per loop

    Profiling results using bisect:
    1 loops, best of 3: 400 µs per loop

While the bisection method tends to be slow (relative to other methods), so long as the user brackets the true steady state value, it is guranteed to converge. Using numerical methods to solve for the steady state of the Solow model illustrates a trade-off between robustness (i.e., guaranteed convergence) and computational efficiency (i.e., speed) that is commonly encountered in numerical work.

```python

```

examples/2 Finding the steady state.ipynb


This notebook can be useful to quickly make a latex representation of linear systems of 2 variables.

```python
pre_str = r"""
$
\left\{
\begin{align}
"""

post_str = r"""\end{align}
\right.
$
"""

def complete_matrix_to_latex(Ab):
    #6 x_1 + 7 x_2 &  = 11

    sys_str = ""

    for eq in Ab:

        eq_str = ""

        for coef_index, coef in enumerate(eq[:-1]):

            if coef != 0:
                if len(eq_str) > 0:
                    eq_str += "+ "
                if coef == 1:
                    eq_str += "x_{" + str(coef_index + 1) + "} "
                elif coef == -1:
                    eq_str += "-x_{" + str(coef_index + 1) + "} "
                else:
                    eq_str += str(coef) + " x_{" + str(coef_index + 1) + "} "

        eq_str = eq_str.replace("+ -", "- ")
        eq_str += "& = {} \\\\\n".format(eq[-1])
        sys_str += eq_str

    return pre_str + sys_str + post_str

```

```python
Ab = np.array([[0, -1, 1],
               [2, 0, 1],
               [1, -1, 2]])

print(complete_matrix_to_latex(Ab))
```


# Introduction

$\newcommand{\G}{\mathcal{G}}$
$\newcommand{\V}{\mathcal{V}}$
$\newcommand{\E}{\mathcal{E}}$
$\newcommand{\R}{\mathbb{R}}$

This notebook shows how to apply the graph ConvNet ([paper] & [code]), for classifying graph signals (EEG). For this example we have a set of EEG signals (epochs) belonging to two different classes (passive vs attentive task). The SG.npy file correndonds to the $W$ matrix of our paper and is used as the basis to construct the spatiotemporal graph.

[paper]: https://arxiv.org/abs/1606.09375
[code]:  https://github.com/mdeff/cnn_graph

We note that this just an exemplar usage scenario. The full code for the GCNN is provided at (and can be cloned) at [code].

```python
from lib import models, graph, coarsening, utils
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import block_diag
from sklearn import preprocessing
%matplotlib inline
```

# 1 Data

For the purpose of the demo, let's create a random data matrix $X \in \R^{n \times d_x}$ and somehow infer a label $y_i = f(x_i)$.

```python
X=np.load('total_graph_train_trials.npy')
d = X.shape[1]    # Dimensionality.
n = X.shape[0]  # Number of samples.
#c = 5      # Number of feature communities.
y=np.load('total_graph_train_labels.npy')
y=y[0,:]
print('Graph Signal Dimensionality: '+str(d))
print('Number of Graph Signals: '+str(n))
print('Class imbalance: ', np.unique(y, return_counts=True)[1])
print(y.shape)
print(len(X))
```

    Graph Signal Dimensionality: 24256
    Number of Graph Signals: 474
    Class imbalance:  [234 240]
    (474,)
    474

Then load training, validation and testing sets.

```python
#min_max_scaler = preprocessing.MinMaxScaler()
#X_train = min_max_scaler.fit_transform(X)
X_train=X
X_val   = np.load('graph_val_trials.npy')
#X_val = min_max_scaler.fit_transform(X_val)
X_test  = np.load('graph_test_trials.npy')
#X_test = min_max_scaler.fit_transform(X_test)

y_train = y
y_val   = np.load('val_labels.npy')
y_val = y_val[0,:]
y_test   = np.load('test_labels.npy')
y_test = y_test[0,:]
print('Class imbalance in Train Set: ', np.unique(y_train, return_counts=True)[1])
print('Class imbalance in Val Set: ', np.unique(y_val, return_counts=True)[1])
print('Class imbalance in Test Set: ', np.unique(y_test, return_counts=True)[1])
n = X_train.shape[0]
```

    Class imbalance in Train Set:  [234 240]
    Class imbalance in Val Set:  [5 5]
    Class imbalance in Test Set:  [5 5]

If you choose to run the following cell then the GCNN will operate on the initial train set without the augmentation data (set the condition to "True"). The first 5/6 of the X_train contain the artificially generated epochs using the GEMD code (provided as Matlab scripts) with the last 1/6 of X_train contains the original training set.

```python
if False:
    N=round(n/6)
    X_train=X[-N:]
    X_val   = np.load('graph_val_trials.npy')
    X_test  = np.load('graph_test_trials.npy')
    y_train = y[-N:]
    y_val   = np.load('val_labels.npy')
    y_val = y_val[0,:]
    y_test   = np.load('test_labels.npy')
    y_test = y_test[0,:]
    print('Class imbalance in Train Set: ', np.unique(y_train, return_counts=True)[1])
    print('Class imbalance in Val Set: ', np.unique(y_val, return_counts=True)[1])
    print('Class imbalance in Test Set: ', np.unique(y_test, return_counts=True)[1])

```

# 2 Spatiotemporal Graph Construction

Let $\mathcal{G}=(\mathcal{V}, \mathcal{E}, \textbf{W})$ be a graph that expresses the topology of the EEG recording sensor array with $W \in \{0,1\}^{E\times E}$ and $E$ being the number of sensors. The spatiotemporal graph that expresses jointly the relationship of an EEG signal with $T$ time samples is defined through its adjacency matrix as
\begin{equation}
\label{eq:multiplex_graph}
    \textbf{W}_{ST}=\textbf{W}\otimes \textbf{I}_E + \textbf{S}+\textbf{S}^\top
\end{equation}
where ($\otimes$) denotes the Kronecker product operator, $\textbf{I}_E$ the $E \times E$ identity matrix and $\textbf{S}$ an $ET \times ET$ matrix whose elements are calculated as $S_{i,j}=\delta_{i,i+E}$ with $\delta_{i,j}$ being the Kronecker delta.

```python
SG=np.load('SG.npy')
temporal_dimension=int(d/SG.shape[0])
spatial_dimension=int(SG.shape[0])
SG=SG-np.eye(spatial_dimension,dtype=int)
A=np.kron(np.eye(temporal_dimension,dtype=int),SG).astype(np.float32)

#assert A.shape == (d, d)
#print('d = |V| = {}, k|V| < |E| = {}'.format(d, A.nnz))
#plt.spy(A, markersize=2, color='black');
print(temporal_dimension)
```

    379

```python
for i in range(spatial_dimension,d):
    j=i-spatial_dimension;
    A[i,j]=1
    A[j,i]=1
```

```python
from scipy import sparse
A=sparse.coo_matrix(A)
assert A.shape == (d, d)
print('d = |V| = {}, k|V| < |E| = {}'.format(d, A.nnz))
plt.spy(A, markersize=2, color='black');
```

To be able to pool graph signals, we need first to coarsen the graph, i.e. to find which vertices to group together. At the end we'll have multiple graphs, like a pyramid, each at one level of resolution. The finest graph is where the input data lies, the coarsest graph is where the data at the output of the graph convolutional layers lie. That data, of reduced spatial dimensionality, can then be fed to a fully connected layer.

The parameter here is the number of times to coarsen the graph. Each coarsening approximately reduces the size of the graph by a factor two. Thus if you want a pooling of size 4 in the first layer followed by a pooling of size 2 in the second, you'll need to coarsen $\log_2(4+2) = 3$ times.

After coarsening we rearrange the vertices (and add fake vertices) such that pooling a graph signal is analog to pooling a 1D signal. See the [paper] for details.

[paper]: https://arxiv.org/abs/1606.09375

```python
graphs, perm = coarsening.coarsen(A, levels=4, self_connections=False)

X_train = coarsening.perm_data(X_train, perm)
X_val = coarsening.perm_data(X_val, perm)
X_test = coarsening.perm_data(X_test, perm)
```

    Layer 0: M_0 = |V| = 28272 nodes (4016 added),|E| = 100371 edges
    Layer 1: M_1 = |V| = 14136 nodes (1472 added),|E| = 64474 edges
    Layer 2: M_2 = |V| = 7068 nodes (489 added),|E| = 35301 edges
    Layer 3: M_3 = |V| = 3534 nodes (103 added),|E| = 18608 edges
    Layer 4: M_4 = |V| = 1767 nodes (0 added),|E| = 9404 edges

We finally need to compute the graph Laplacian $L$ for each of our graphs (the original and the coarsened versions), defined by their adjacency matrices $A$. The sole parameter here is the type of Laplacian, e.g. the combinatorial Laplacian, the normalized Laplacian or the random walk Laplacian.

```python
L = [graph.laplacian(A, normalized=True) for A in graphs]
#graph.plot_spectrum(L)
```

# 3 Graph ConvNet

Here we apply the graph convolutional neural network to signals lying on graphs. After designing the architecture and setting the hyper-parameters, the model takes as inputs the data matrix $X$, the target $y$ and a list of graph Laplacians $L$, one per coarsening level.

The data, architecture and hyper-parameters are absolutely *not engineered to showcase performance*. Its sole purpose is to illustrate usage and functionality.

```python
params = dict()
params['dir_name']       = 'demo'
params['num_epochs']     = 20
params['batch_size']     = round(n/6)
params['eval_frequency'] = 6

# Building blocks.
params['filter']         = 'chebyshev5'
params['brelu']          = 'b1relu'
params['pool']           = 'mpool1'

# Number of classes.
C = y.max() + 1
assert C == np.unique(y).size

# Architecture.
params['F']              = [20, 20]  # Number of graph convolutional filters.
params['K']              = [10, 10]  # Polynomial orders.
params['p']              = [1, 2]    # Pooling sizes.
params['M']              = [128, 64, C]  # Output dimensionality of fully connected layers.

# Optimization.
params['regularization'] = 5e-4
params['dropout']        = 0.95
params['learning_rate']  = 1e-3
params['decay_rate']     = 0.95
params['momentum']       = 0.9
params['decay_steps']    = n / params['batch_size']
```

```python
tf.reset_default_graph()
model = models.cgcnn(L, **params)
print(X_train.shape)
accuracy, loss, t_step = model.fit(X_train, y_train, X_val, y_val)
```

    NN architecture
      input: M_0 = 28272
      layer 1: cgconv1
        representation: M_0 * F_1 / p_1 = 28272 * 20 / 1 = 565440
        weights: F_0 * F_1 * K_1 = 1 * 20 * 10 = 200
        biases: F_1 = 20
      layer 2: cgconv2
        representation: M_1 * F_2 / p_2 = 28272 * 20 / 2 = 282720
        weights: F_1 * F_2 * K_2 = 20 * 20 * 10 = 4000
        biases: F_2 = 20
      layer 3: fc1
        representation: M_3 = 128
        weights: M_2 * M_3 = 282720 * 128 = 36188160
        biases: M_3 = 128
      layer 4: fc2
        representation: M_4 = 64
        weights: M_3 * M_4 = 128 * 64 = 8192
        biases: M_4 = 64
      layer 5: logits (softmax)
        representation: M_5 = 2.0
        weights: M_4 * M_5 = 64 * 2.0 = 128.0
        biases: M_5 = 2.0
    WARNING:tensorflow:From C:\Users\fkalaganis\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    WARNING:tensorflow:From C:\Users\fkalaganis\Desktop\cnn_graph\lib\models.py:969: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
    WARNING:tensorflow:From C:\Users\fkalaganis\Desktop\cnn_graph\lib\models.py:209: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.
    (474, 28272)
    step 6 / 120 (epoch 1.00 / 20):
      learning_rate = 1.00e-03, loss_average = 3.40e+01

    C:\Users\fkalaganis\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.
      'precision', 'predicted', average, warn_for)

      validation accuracy: 50.00 (5 / 10), f1 (weighted): 33.33, loss: 5.56e+02
      time: 11s (wall 10s)
    step 12 / 120 (epoch 2.00 / 20):
      learning_rate = 9.50e-04, loss_average = 5.13e+01
      validation accuracy: 50.00 (5 / 10), f1 (weighted): 33.33, loss: 5.66e+02
      time: 25s (wall 20s)
    step 18 / 120 (epoch 3.00 / 20):
      learning_rate = 9.02e-04, loss_average = 6.03e+01
      validation accuracy: 70.00 (7 / 10), f1 (weighted): 67.03, loss: 5.56e+02
      time: 37s (wall 30s)
    step 24 / 120 (epoch 4.00 / 20):
      learning_rate = 8.57e-04, loss_average = 6.50e+01
      validation accuracy: 70.00 (7 / 10), f1 (weighted): 67.03, loss: 5.57e+02
      time: 49s (wall 39s)
    step 30 / 120 (epoch 5.00 / 20):
      learning_rate = 8.15e-04, loss_average = 6.74e+01
      validation accuracy: 60.00 (6 / 10), f1 (weighted): 52.38, loss: 5.61e+02
      time: 62s (wall 48s)
    step 36 / 120 (epoch 6.00 / 20):
      learning_rate = 7.74e-04, loss_average = 6.87e+01
      validation accuracy: 70.00 (7 / 10), f1 (weighted): 67.03, loss: 5.61e+02
      time: 74s (wall 57s)
    WARNING:tensorflow:From C:\Users\fkalaganis\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use standard file APIs to delete files with this prefix.
    step 42 / 120 (epoch 7.00 / 20):
      learning_rate = 7.35e-04, loss_average = 6.93e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 80.00, loss: 5.59e+02
      time: 87s (wall 66s)
    step 48 / 120 (epoch 8.00 / 20):
      learning_rate = 6.98e-04, loss_average = 6.97e+01
      validation accuracy: 70.00 (7 / 10), f1 (weighted): 69.70, loss: 5.59e+02
      time: 99s (wall 75s)
    step 54 / 120 (epoch 9.00 / 20):
      learning_rate = 6.63e-04, loss_average = 6.98e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 112s (wall 84s)
    step 60 / 120 (epoch 10.00 / 20):
      learning_rate = 6.30e-04, loss_average = 6.99e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 124s (wall 93s)
    step 66 / 120 (epoch 11.00 / 20):
      learning_rate = 5.99e-04, loss_average = 7.00e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 136s (wall 102s)
    step 72 / 120 (epoch 12.00 / 20):
      learning_rate = 5.69e-04, loss_average = 7.00e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 149s (wall 111s)
    step 78 / 120 (epoch 13.00 / 20):
      learning_rate = 5.40e-04, loss_average = 7.00e+01
      validation accuracy: 90.00 (9 / 10), f1 (weighted): 89.90, loss: 5.59e+02
      time: 161s (wall 120s)
    step 84 / 120 (epoch 14.00 / 20):
      learning_rate = 5.13e-04, loss_average = 7.00e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 173s (wall 132s)
    step 90 / 120 (epoch 15.00 / 20):
      learning_rate = 4.88e-04, loss_average = 7.00e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 186s (wall 141s)
    step 96 / 120 (epoch 16.00 / 20):
      learning_rate = 4.63e-04, loss_average = 7.00e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 198s (wall 152s)
    step 102 / 120 (epoch 17.00 / 20):
      learning_rate = 4.40e-04, loss_average = 7.00e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 210s (wall 161s)
    step 108 / 120 (epoch 18.00 / 20):
      learning_rate = 4.18e-04, loss_average = 7.00e+01
      validation accuracy: 80.00 (8 / 10), f1 (weighted): 79.17, loss: 5.60e+02
      time: 223s (wall 171s)
    step 114 / 120 (epoch 19.00 / 20):
      learning_rate = 3.97e-04, loss_average = 7.00e+01
      validation accuracy: 90.00 (9 / 10), f1 (weighted): 89.90, loss: 5.59e+02
      time: 235s (wall 180s)
    step 120 / 120 (epoch 20.00 / 20):
      learning_rate = 3.77e-04, loss_average = 7.00e+01
      validation accuracy: 90.00 (9 / 10), f1 (weighted): 89.90, loss: 5.59e+02
      time: 247s (wall 191s)
    validation accuracy: peak = 90.00, mean = 83.00

# 4 Evaluation

We often want to monitor:
1. The convergence, i.e. the training loss and the classification accuracy on the validation set.
2. The performance, i.e. the classification accuracy on the testing set (to be compared with the training set accuracy to spot overfitting).

The `model_perf` class in [utils.py](utils.py) can be used to compactly evaluate multiple models.

```python
fig, ax1 = plt.subplots(figsize=(15, 5))
ax1.plot(accuracy, 'b.-')
ax1.set_ylabel('validation accuracy', color='b')
ax2 = ax1.twinx()
ax2.plot(loss, 'g.-')
ax2.set_ylabel('training loss', color='g')
plt.show()
```

```python
print('Time per step: {:.2f} ms'.format(t_step*1000))
```

    Time per step: 1626.03 ms

Please keep in mind that some incosistencies may occur due to weight initialization and small validation/testing sets

```python
res = model.evaluate(X_test, y_test)
print(res[0])
```

    WARNING:tensorflow:From C:\Users\fkalaganis\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use standard file APIs to check for files with this prefix.
    INFO:tensorflow:Restoring parameters from C:\Users\fkalaganis\Desktop\cnn_graph\lib\..\checkpoints\demo\model-120
    accuracy: 80.00 (8 / 10), f1 (weighted): 80.00, loss: 5.60e+02
    time: 3s (wall 1s)


# PyMC3 Introduction

Last revised 18-Jun-2019 by Dick Furnstahl (furnstahl.1@osu.edu)

A good starting point for notebooks with PyMC3 examples is the official documentation site: https://docs.pymc.io/.  We've adapted some examples from that site here and in other notebooks.

**Aside.** Here is a good quote from Rob Hicks on HMC and No U-Turn:

"The idea: rather than blindly stumbling around the posterior, use the posterior gradient to skate around the gradient contour. As you skate closer to a drop-off (gradient is steep and probability is lower), potential energy decreases and kinetic energy increases (since energy is always conserved). When this happens the skater is turned back uphill and pushed from the precipice and skates on along a posterior likelihood contour. The No U-Turn sampler keeps skating until the skater tries to turn back towards the original point."

## Imports

```python
%matplotlib inline
import numpy as np
import scipy.stats as stats

import pymc3 as pm
import theano as tt

import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('white')
sns.set_context('talk')

# Recommended: document what PyMC3 version we are using
print(f'Running on PyMC3 v{pm.__version__}')

```

## Basic setup of a model

First we need to create a *model*, which will be an instance of the Model class.  The model has references to all random variables (RVs) and computes the model log posterior (logp) and its gradients.  We typically instantiate it using a with context.  For example:

```python
with pm.Model() as my_model:
    mu = pm.Normal('mu', mu=0, sigma=1)
    obs = pm.Normal('obs', mu=mu, sigma=1, observed=np.random.randn(100))
```

So `my_model` is an instance of the PyMC3 Model class, and we have set up a prior for `mu` in the form of a standard normal distribution (i.e., mean = 0 and standard deviation = 1).  The last line sets up the likelihood, also distributed as a normal with observed data taken as 100 random draw from a standard normal distribution.  The standard deviation sd for the mu posterior is given.  The goal will be to sample the posterior for mu.

## Sampling

The main entry point to MCMC sampling algorithms is via the `pm.sample()` function. By default, this function tries to auto-assign the right sampler(s) and auto-initialize if you don’t pass anything.

As you can see, on a continuous model, `PyMC3` assigns the NUTS sampler, which is very efficient even for complex models. `PyMC3` also runs variational inference (i.e. `ADVI`) to find good starting parameters for the sampler. Here we draw 1000 samples from the posterior and allow the sampler to adjust its parameters in an additional 500 iterations. These 500 samples are discarded by default:

```python
with pm.Model() as my_NUTS_model:
    mu = pm.Normal('mu', mu=0, sigma=1)
    obs = pm.Normal('obs', mu=mu, sigma=1, observed=np.random.randn(100))

    trace_NUTS = pm.sample(1000, tune=500)

pm.traceplot(trace_NUTS);

```

### Available samplers

`PyMC3` offers a variety of samplers, found in pm.step_methods:

```python
list(filter(lambda x: x[0].isupper(), dir(pm.step_methods)))
```

Commonly used step-methods besides NUTS are Metropolis and Slice. The claim is that for almost all continuous models, ``NUTS`` should be preferred. There are hard-to-sample models for which NUTS will be very slow causing many users to use Metropolis instead. This practice, however, is rarely successful. NUTS is fast on simple models but can be slow if the model is very complex or it is badly initialized. In the case of a complex model that is hard for NUTS, Metropolis, while faster, will have a very low effective sample size or not converge properly at all. A better approach is to instead try to improve initialization of NUTS, or reparameterize the model.

For completeness, other sampling methods can be passed to sample.  Here is an example (Metropolis-Hastings):

```python
with pm.Model() as my_Metropolis_model:
    mu = pm.Normal('mu', mu=0, sigma=1)
    obs = pm.Normal('obs', mu=mu, sigma=1, observed=np.random.randn(100))

    step = pm.Metropolis()
    trace_MH = pm.sample(1000, step=step)

pm.traceplot(trace_MH);
```

### Analyze sampling results

The most common used plot to analyze sampling results is the so-called trace-plot:

```python
pm.traceplot(trace_NUTS);

```

```python
with pm.Model() as model:
    mu = pm.Normal('mu', mu=0, sd=1)
    sd = pm.HalfNormal('sd', sd=1)
    obs = pm.Normal('obs', mu=mu, sd=sd, observed=np.random.randn(100))

    step1 = pm.Metropolis(vars=[mu])
    step2 = pm.Slice(vars=[sd])
    trace_2_samplers = pm.sample(10000, step=[step1, step2], cores=4)

pm.traceplot(trace_2_samplers);

```

### Diagnostics

```python
pm.gelman_rubin(trace_MH)
```

```python
pm.plot_posterior(trace_MH);

```

## Examples from Rob Hicks

See https://rlhick.people.wm.edu/stories/bayesian_7.html.  We also have a notebook from his Bayesian 8 "story".

We start with a very simple one parameter model and then move to slightly more complicated settings:

```python
sigma = 3.  # standard deviation
mu = 10.    # mean
num_samples = 100  # 10**6

# sample from a normal distribution
data = stats.norm(mu, sigma).rvs(num_samples)

# plot a histogram of the sampled data
num_bins = 20
plt.hist(data, bins=num_bins)
plt.show()

```

Run the previous cell a few times to see the fluctuations.  Crank up the number of samples to `10**6` to see a smoother histogram.

### PyMC3 implementation

We instantiate a Model with a descriptions of priors and the likelihood. Here, `mu` is defined to be a random variable (we want to sample this variable by generating a Markov chain) and we provide a prior distribution with associated hyper-parameters for it. The likelihood function is chosen to be Normal, with one parameter to be estimated (mu), and we use known $\sigma$ (denoted as `sigma`). Our "dependent variable" is given by `observed=data`, where `data` is generated above and shown in the histogram. So we our implementing Bayes theorem in the form:
\begin{align}
   \newcommand{\Pr}{\textrm{pr}}
   \newcommand{\data}{\textbf{data}}
   \Pr(\mu | \sigma, \data) \propto \Pr(\data | \mu, \sigma) \times \Pr(\mu |\mu^0_\mu, \sigma^0_\mu)
\end{align}

```python
# parameters for the prior on mu
mu_prior = 8.
sigma_prior = 1.5  # Note this is our prior on the std of mu

# Could do this instead as:
#   basic_model = pm3.Model()
#   with basic_model:

with pm.Model() as basic_model:

    # Prior for unknown model parameters (mean and sd of the normal pdf)
    mu = pm.Normal('Mean of Data', mu_prior, sigma_prior)

    # Likelihood (sampling distribution) of observations
    data_in = pm.Normal('Y_obs', mu=mu, sigma=sigma, observed=data)

```

Next we define how the Markov chain will be constructed. The example we are following set `startvals` to be the MAP and used a Metropolis step method.  There always seems to be a complaint with the latest pyMC3 about using find_MAP to start the sampler.

```python
chain_length = 10000

with basic_model:
    # obtain starting values via MAP (maximum a posteriori)
    startvals = pm.find_MAP(model=basic_model)  # model here is optional
    print(startvals)

    # instantiate sampler
    step = pm.Metropolis()   # Metropolis-Hastings

    # draw 10000 posterior samples for each chain (4 chains by default?)
    trace = pm.sample(draws=chain_length, step=step, start=startvals)
    #trace = pm.sample(chain_length, step=step)
```

```python

# Plot the four chains
pm.traceplot(trace, figsize=(20,5));

# Summary information on the Markov chains
pm.summary(trace)

```

Remember that what we are generating is a posterior for the mean given the data and our (assumed) knowledge of the standard deviation.

So for the summary info we get the mean and standard deviation (sd) of the distribution, with an estimate of the Monte Carlo error. What does hpd stand for?  "Highest posterior density"  2.5 and 97.5 are percentages, so one talks of a 95% hpd interval in this case.

From an answer online: "You create the parameter trace plots to make sure that your a priori distribution is well calibrated which is indicated by your parameters having sufficient state changes as the MCMC algorithm runs."

"All the results are contained in the trace variable. This is a pymc3 results object. It contains some information that we might want to extract at times. `Varnames` tells us all the variable names setup in our model."

```python
trace.varnames
```

This was set up when we initiated our model (in specifying the prior for mu).  With the variable names, we can extract chain values for each variable:

```python
trace['Mean of Data']

```

Is this one chain or all four chains?  Check the length!  Looks like all four.

```python
print(len(trace['Mean of Data']))
print(trace['Mean of Data'].shape)
```

Now for diagnostics.

### Autocorrelation plots

```python
pm.plots.autocorrplot(trace,figsize=(17,5));
```

What do we see here?  An autocorrelation time around 10 or so.

### Acceptance rate

```python
accept = np.sum(trace['Mean of Data'][1:] != trace['Mean of Data'][:-1])
print("Acceptance Rate: ", accept/trace['Mean of Data'].shape[0])

```

That looks like we have to work harder than one might have expected.  It is taking the array of results and comparing each point to the previous one and including it in the sum if it is different.  So if there wasn't an acceptance, then the point remains the same.  The ratio to the full length is the acceptance rate.  Maybe we should define a function here instead.

```python
def acceptance_rate(trace_array):
    """Calculate how many times the entry in the trace array changed compared
       to the total length.
    """
    changed = np.sum(trace_array[1:] != trace_array[:-1])
    total_length = trace_array.shape[0]
    return changed / total_length
```

```python
acceptance_rate(trace['Mean of Data'])
```

### Geweke Score

From a different lecture:

We can explicitly think of this test as a test for the Ergodicity (stationarity) of your chain.

Take the first 10 and last 50% of your chain and do a z test comparing means (correcting for autocorrelation). Software packages, take this a step further: The geweke function in pymc3 by default chooses the first 10% of your chain, and the final 50%; divides the final 50% of the chain into 20 segments and performs a z-test for each segment. You want to fail to reject the null, since the hypothesis is:

H0:θ10%=$θs50%H1:θ10%≠$θs50%
for each segment s. If our means are the same (we fail to reject the null), then we have strong evidence of chain convergence.

```python
score=pm.geweke(trace, first=0.1, last=0.5, intervals=20)
score
```

```python
score=pm.geweke(trace, first=0.1, last=0.5, intervals=20)
plt.scatter(score[0]['Mean of Data'][:,0],score[0]['Mean of Data'][:,1], marker = 'o', s=100)
plt.axhline(-1.98, c='r')
plt.axhline(1.98, c='r')
plt.ylim(-2.5,2.5)
plt.xlim(0-10,.5*trace['Mean of Data'].shape[0]/2+10)
plt.title('Geweke Plot Comparing first 10% and Slices of the Last 50% of Chain\nDifference in Mean Z score')
plt.show()
```

```python
score=pm.geweke(trace, first=0.0005, last=0.5, intervals=20)
plt.scatter(score[0]['Mean of Data'][:,0],score[0]['Mean of Data'][:,1],
            marker = 'o', s=100)
plt.axhline(-1.98, c='r')
plt.axhline(1.98, c='r')
plt.ylim(-2.5,2.5)
plt.xlim(0-10,.5*trace['Mean of Data'].shape[0]/2+10)
my_title = 'Geweke Plot Comparing first 0.05% and Slices of the Last 50%' +\
           'of Chain\nDifference in Mean Z score'
plt.title(my_title)
plt.show()
```

From a different lecture: "Even without dropping any burn-in observations, we have convergence. We only start seeing issues when we restrict ourselves to the first 5 values in the chain. Suggests we should drop the first few dozen observations for burn-in."

### Gelman Rubin Diagnostic (quoted verbatim from the Hicks notebook)

If our MH MCMC Chain reaches a stationary distribution, and we repeat the excercise multiple times, then we can examine if the posterior for each chain converges to the same place in the distribution of the parameter space.

Steps:
1. Run $M>1$ Chains of length $2 \times N$.
2. Discard the first $N$ draws of each chain, leaving $N$ iterations in the chain.
3. Calculate the within and between chain variance.
    * Within chain variance:
    $$
    W = \frac{1}{M}\sum_{j=1}^M s_j^2
    $$
    where $s_j^2$ is the variance of each chain (after throwing out the first $N$ draws).
    * Between chain variance:
    $$
    B = \frac{N}{M-1} \sum_{j=1}^M (\bar{\theta_j} - \bar{\bar{\theta}})^2
    $$

    where $\bar{\bar{\theta}}$ is the mean of each of the M means.
4. Calculate the estimated variance of $\theta$ as the weighted sum of between and within chain variance.
$$
\hat{var}(\theta) = \left ( 1 - \frac{1}{N}\right ) W + \frac{1}{N}B
$$
5. Calculate the potential scale reduction factor.
$$
\hat{R} = \sqrt{\frac{\hat{var}(\theta)}{W}}
$$

We want this number to be close to 1.  Why?  This would indicate that the between chain variance is small.  This makes sense, if between chain variance is small, that means both chains are mixing around the stationary distribution.  Gelmen and Rubin show that when $\hat{R}$ is greater than 1.1 or 1.2, we need longer burn-in.

```python
chain_length = 100000

with basic_model:
    # obtain starting values via MAP
    startvals = pm.find_MAP(model=basic_model)
    print(startvals)
    # instantiate sampler
    step = pm.Metropolis()

    # draw 5000 posterior samples
    trace = pm.sample(chain_length, step=step, start=startvals)
```

```python
pm.traceplot(trace,figsize=(20,5));
```

```python
pm.gelman_rubin(trace)
```

"Based on Gelman-Rubin and Geweke, we can be confident we have a chain that has converged to the limiting distribution (although perhaps for Gelman-Rubin, we'd want to try different starting values rather than MAP)."

Trying without `find_MAP`, i.e., not specifying `start` in `pm.sample`.

```python
chain_length = 100000

with basic_model:
    # obtain starting values via MAP
    #startvals = pm.find_MAP(model=basic_model)
    #print(startvals)
    # instantiate sampler
    step = pm.Metropolis()

    # draw 5000 posterior samples
    trace = pm.sample(chain_length, step=step)
```

```python
pm.traceplot(trace,figsize=(20,5));
```

```python
pm.gelman_rubin(trace)
```

"The diagnostics we have discussed are all univariate (they work perfectly when there is only 1 parameter to estimate). Other diagnostics have been derived for the multivariate case, but these are useful only when using Gibbs Samplers or other specialized versions of Metropolis-Hastings.

So most people examine univariate diagnostics for each variable, examine autocorrelation plots, acceptance rates and try to argue chain convergence based on that- unless they are using Gibbs or other specialized samplers."

### In-class exercise

Let's try to modify the code below to estimate sigma as well as the mean:

```python
sigma = 3.  # standard deviation
mu = 10.    # mean
num_samples = 100  # 10**6

# sample from a normal distribution
data = stats.norm(mu, sigma).rvs(num_samples)

# plot a histogram of the sampled data
num_bins = 20
plt.hist(data, bins=num_bins)
plt.show()

```

```python
# parameters for the prior on mu
mu_mean_prior = 8.
mu_sd_prior = 1.5  # Note this is our prior on the std of mu

with pm.Model() as basic_model:

    # Priors for unknown model parameters
    mu = pm.Normal('Mean of Data', mu_mean_prior, mu_sd_prior)

    # Likelihood (sampling distribution) of observations
    data_in = pm.Normal('Y_obs', mu=mu, sd=sigma, observed=data)

```

```python
chain_length = 10000

with basic_model:
    # obtain starting values via MAP
    startvals = pm.find_MAP(model=basic_model)
    print(startvals)
    # instantiate sampler
    step = pm.Metropolis()

    # draw 10000 posterior samples
    trace = pm.sample(chain_length, step=step, start=startvals)

```

```python
score=pm.geweke(trace, first=0.1, last=0.5, intervals=20)
plt.scatter(score[0]['Mean of Data'][:,0],score[0]['Mean of Data'][:,1],
            marker = 'o', s=100)
plt.axhline(-1.98, c='r')
plt.axhline(1.98, c='r')
plt.ylim(-2.5,2.5)
plt.xlim(0-10,.5*trace['Mean of Data'].shape[0]/2+10)
my_title = 'Geweke Plot Comparing first 10% and Slices of the Last 50%' +\
           ' of Chain\nDifference in Mean Z score'
plt.title(my_title)
plt.show()
```

Ok, we're trying it!

```python
sigma = 3.  # standard deviation
mu = 10.    # mean
num_samples = 1000  # 100 # 10**6

# sample from a normal distribution
data = stats.norm(mu, sigma).rvs(num_samples)

# plot a histogram of the sampled data
num_bins = 20
plt.hist(data, bins=num_bins)
plt.show()

```

```python
# parameters for the prior on mu
mu_mean_prior = 8.
mu_sd_prior = 1.5  # Note this is our prior on the std of mu

sigma_mean_prior = 1.
sigma_sd_prior = 1.

with pm.Model() as two_param_model:

    # Priors for unknown model parameters
    mu = pm.Normal('Mean of Data', mu_mean_prior, mu_sd_prior)
    sigma = pm.Normal('SD of Data', sigma_mean_prior, sigma_sd_prior)

    # Likelihood (sampling distribution) of observations
    data_in = pm.Normal('Y_obs', mu=mu, sd=sigma, observed=data)

```

```python
chain_length = 10000

with two_param_model:
    # obtain starting values via MAP
    startvals = pm.find_MAP(model=two_param_model)
    print(startvals)
    # instantiate sampler
    step = pm.Metropolis()

    # draw 10000 posterior samples
    trace_two_param = pm.sample(chain_length, step=step, start=startvals)

```

```python
pm.traceplot(trace_two_param, figsize=(20,10));
```

```python
score=pm.geweke(trace_two_param, first=0.1, last=0.5, intervals=20)
plt.scatter(score[0]['Mean of Data'][:,0],score[0]['Mean of Data'][:,1],
            marker = 'o', s=100)
plt.axhline(-1.98, c='r')
plt.axhline(1.98, c='r')
plt.ylim(-2.5,2.5)
plt.xlim(0-10,.5*trace['Mean of Data'].shape[0]/2+10)
my_title = 'Geweke Plot Comparing first 10% and Slices of the Last 50%' +\
           ' of Chain\nDifference in Mean Z score'
plt.title(my_title)
plt.show()
```

```python
score=pm.geweke(trace_two_param, first=0.1, last=0.5, intervals=20)
plt.scatter(score[0]['SD of Data'][:,0],score[0]['SD of Data'][:,1], marker = 'o', s=100)
plt.axhline(-1.98, c='r')
plt.axhline(1.98, c='r')
plt.ylim(-2.5,2.5)
#plt.xlim(0-10,.5*trace['SD of Data'].shape[0]/2+10)
plt.title('Geweke Plot Comparing first 10% and Slices of the Last 50% of Chain\nDifference in SD Z score')
plt.show()
```

```python

```


# 2022-02-28 Low Rank

## Last time

* Solving least squares problems
* Geometry of the SVD

## Today

* Reflection on algorithm choices
* Low-rank structure
* Primer on interpolation

```julia
using LinearAlgebra
using Plots
default(linewidth=4, legendfontsize=12)

function vander(x, k=nothing)
    if isnothing(k)
        k = length(x)
    end
    m = length(x)
    V = ones(m, k)
    for j in 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function gram_schmidt_classical(A)
    m, n = size(A)
    Q = zeros(m, n)
    R = zeros(n, n)
    for j in 1:n
        v = A[:,j]
        R[1:j-1,j] = Q[:,1:j-1]' * v
        v -= Q[:,1:j-1] * R[1:j-1,j]
        R[j,j] = norm(v)
        Q[:,j] = v / R[j,j]
    end
    Q, R
end

function qr_householder(A)
    m, n = size(A)
    R = copy(A)
    V = [] # list of reflectors
    for j in 1:n
        v = copy(R[j:end, j])
        v[1] += sign(v[1]) * norm(v) # <---
        v = normalize(v)
        R[j:end,j:end] -= 2 * v * v' * R[j:end,j:end]
        push!(V, v)
    end
    V, R
end

function qr_chol(A)
    R = cholesky(A' * A).U
    Q = A / R
    Q, R
end

function qr_chol2(A)
    Q, R = qr_chol(A)
    Q, R1 = qr_chol(Q)
    Q, R1 * R
end

function peanut()
    theta = LinRange(0, 2*pi, 50)
    r = 1 .+ .4*sin.(3*theta) + .6*sin.(2*theta)
    r' .* [cos.(theta) sin.(theta)]'
end

function circle()
    theta = LinRange(0, 2*pi, 50)
    [cos.(theta) sin.(theta)]'
end

function Aplot(A)
    "Plot a transformation from X to Y"
    X = peanut()
    Y = A * X
    p = scatter(X[1,:], X[2,:], label="in")
    scatter!(p, Y[1,:], Y[2,:], label="out")
    X = circle()
    Y = A * X
    q = scatter(X[1,:], X[2,:], label="in")
    scatter!(q, Y[1,:], Y[2,:], label="out")
    plot(p, q, layout=2, aspect_ratio=:equal)
end
```

    Aplot (generic function with 1 method)

# Condition number via SVD

$$ U \overbrace{\begin{bmatrix} \sigma_{\max} && \\ & \ddots & \\ && \sigma_{\min} \end{bmatrix}}^{\Sigma} V^T = A $$

\begin{align}
\lVert A \rVert &= \sigma_{\max} &
\kappa(A) &= \frac{\sigma_{\max}}{\sigma_{\min}} = \texttt{cond}(A)
\end{align}

```julia
A = randn(2,2) # nonsymmetric
A = A + A'
```

    2×2 Matrix{Float64}:
      1.13519   -0.257307
     -0.257307   0.753558

```julia
@show svdvals(A)
U, S, V = svd(A)
@show norm(U - U')
Aplot(A)
```

    svdvals(A) = [1.2647127088905932, 0.624033667117622]
    norm(U - U') = 7.850462293418876e-17

# Example: autonomous vehicles

* Need to solve least squares problems in real time
* Weight/cost/size increase with compute
* What algorithm to choose?
* What precision to use?

```julia
A = rand(5000, 500)
A_32 = Float32.(A)
@show cond(A)
@time qr(A);       # Householder; backward stable
@time qr_chol(A);  # Unstable
@time qr(A_32);
```

    cond(A) = 56.62895367363473
      0.401766 seconds (7 allocations: 19.348 MiB)
      0.267054 seconds (10 allocations: 22.888 MiB)
      0.262063 seconds (7 allocations: 9.674 MiB)

```julia
V = vander(LinRange(-1, 1, 20))
Q, R = qr(Float32.(V))
@show norm(Q' * Q - I)
Q, R = qr_chol(V)
@show norm(Q' * Q - I)
```

    norm(Q' * Q - I) = 1.422574f-6
    norm(Q' * Q - I) = 0.7929497451834747

    0.7929497451834747

# Best low rank approximation

The SVD can be truncated to yield the best rank-$k$ approximation of a matrix.

```julia
n, k = 2, 1
A = randn(n, n)
Aplot(A)
```

```julia
U, S, V = svd(A)
@show S[1:k+1]
Uhat = U[:, 1:k]
Shat = S[1:k]
Vhat = V[:, 1:k]
Ahat = Uhat * diagm(Shat) * Vhat'
@show norm(Ahat)
Aplot(Ahat - A)
```

    S[1:k + 1] = [1.5595652883771995, 0.8263763919470137]
    norm(Ahat) = 1.559565288377199

# Example: Galaxies

Suppose we have two galaxies of size $n_1 = 100$ and $n_2 = 200$, each randomly distributed around their respective centers.

```julia
galaxy(center, sigma, n) = reshape(center, 1, 3) .+ sigma*randn(n, 3)
g1 = galaxy([0 0 0], 1, 100)
g2 = galaxy([10 0 0], 1, 100)

scatter(g1[:,1], g1[:,2], aspect_ratio=:equal)
scatter!(g2[:,1], g2[:,2])
```

## Forces between stars

Consider the gravitational force from a star at position $x_2$ acting on a star at position $x_1$,
$$ F_{1,2} = G \frac{m_1 m_2}{\lVert x_2 - x_1 \rVert^3} (x_2 - x_1) $$
where $m_1$ and $m_2$ are the masses of each star respectively.

```julia
function gravity(g1, g2)
    m = size(g1, 1)
    n = size(g2, 1)
    F = zeros(3*m, n)
    for i in 0:m-1
        for j in 1:n
            r = g2[j,:] - g1[1+i,:]
            F[1+3*i:3*(i+1),j] = r / norm(r)^3
        end
    end
    F
end
gravity(g1, g2)
```

    300×200 Matrix{Float64}:
      0.00774771    0.00742588    0.00885278   …   0.00827471    0.00612012
     -0.000555126  -1.41668e-5   -0.00128975      -0.000192375  -0.000424015
     -0.000815177  -0.000124244  -0.00141273      -0.000347141   0.000162658
      0.00877027    0.00786987    0.0106905        0.00894689    0.00660422
      0.000877896   0.00132149    0.00040886       0.00139203    0.00053646
     -4.48706e-6    0.000703741  -0.000513661  …   0.000618362   0.000826044
      0.00971415    0.0083071     0.012377         0.00955195    0.00707152
      0.00204436    0.00232325    0.00191547       0.00261379    0.0012409
      0.000790051   0.00145803    0.000472912      0.00151998    0.00145083
      0.00947463    0.00836521    0.0117358        0.00957543    0.0069844
      0.00108515    0.00154438    0.000601652  …   0.00165472    0.00065026
      0.000308721   0.00104587   -0.00016147       0.00101583    0.00110748
      0.00995128    0.00880498    0.0123169        0.0101064     0.00727068
      ⋮                                        ⋱
      0.00842946    0.00769163    0.00990472       0.00868048    0.00616175
     -0.00089865   -0.000249396  -0.00186198      -0.000491765  -0.000615135
      0.000907699   0.00148484    0.000705833  …   0.0015553     0.00134826
      0.00624127    0.00578017    0.00727109       0.00642333    0.00488913
      7.97508e-5    0.000418597  -0.000329543      0.000368067   2.90026e-5
      5.37733e-5    0.000486054  -0.000216821      0.000429595   0.000562206
      0.00950141    0.00888005    0.0112827        0.0100565     0.00743525
      0.000626147   0.00122891   -3.52537e-5   …   0.00124206    0.000376343
     -0.00137201   -0.000397072  -0.00236665      -0.000751152   4.19668e-5
      0.00780408    0.0072665     0.00921643       0.00814798    0.00620523
      0.000726216   0.00114782    0.000311379      0.00118684    0.000470496
     -0.000918796  -0.000204389  -0.0016015       -0.000441785   0.000102482

# Spectrum

```julia
g1 = galaxy([0 0 0], 1, 500)
g2 = galaxy([10 0 0], 1, 500)
F = gravity(g1, g2)
U, S, V = svd(F)
scatter(S, yscale=:log10, ylims=(1e-10, 10), xlims=(0, 200))
```

```julia
k = 20
Uhat = U[:,1:k]
Shat = S[1:k]
Vhat = V[:,1:k]
Fhat = Uhat * diagm(Shat) * Vhat'
@show norm(F)
@show norm(F - Fhat)
size(F)
```

    norm(F) = 2.5194440056980536
    norm(F - Fhat) = 6.79544216333678e-5

    (600, 300)

# What is interpolation?

Given data $(x_i, y_i)$, find a (smooth?) function $f(x)$ such that $f(x_i) = y_i$.

## Data in

* direct field observations/measurement of a physical or social system
* numerically processed observations, perhaps by applying physical principles
* output from an expensive "exact" numerical computation
* output from an approximate numerical computation

## Function out

* Polynomials
* Piecewise polynomials (includes nearest-neighbor)
* Powers and exponentials
* Trigonometric functions (sine and cosine)
* Neural networks

Interpolation fits the data exactly!

# Polynomial interpolation

We've seen how we can fit a polynomial using Vandermonde matrices, one column per basis function and one row per observation.

$$ \underbrace{\Bigg[ 1 \Bigg| x \Bigg| x^2 \Bigg| x^3 \Bigg]}_{A \in \mathbb R^{m\times n}} \Bigg[ \mathbf p \Bigg] = \Bigg[ \mathbf y \Bigg] $$

It's possible to find a unique polynomial $\mathbf p$ when which of the following are true?
1. $m \le n$
2. $m = n$
3. $m \ge n$

# Polynomial interpolation with a Vandermonde matrix

```julia
x = LinRange(-1.5, 2, 4)
y = sin.(x)
A = vander(x)
p = A \ y

scatter(x, y)
s = LinRange(-3, 3, 50)
plot!(s, [sin.(s) vander(s, length(p)) * p])
```

## Vandermonde matrices can be ill-conditioned

```julia
A = vander(LinRange(-1, 1, 30))
cond(A)
```

    1.838577055392608e13

1. It's because of the points $x$?
2. It's because of the basis functions $\{ 1, x, x^2, x^3, \dotsc \}$?


# Imposed Displacements

The dynamic system in figure is composed of two uniform beams of
negligible mass, their flexural stiffness being $EJ=\text{const}$,
supporting a lumped mass of negligible rotatory inertia (the
flexibility matrix is $\boldsymbol F=\frac{L^3}{6EJ}
\begin{bmatrix}3&2\\2&96\end{bmatrix}$).

The left support is subjected to an imposed horizontal displacement
that varies between $0$ and $\Delta$
\begin{equation*}
  u_\mathcal{A} = \Delta
  \begin{cases}
    \frac{20 \tau^3 - 15 \tau^4 + 3 \tau^5}{16} & \text{for }
    0\le\tau\le2,\\
    1&\text{for }\tau>2,
  \end{cases}
\end{equation*}
where $\tau=\omega_0t$ and $\omega_0^2=\frac{EJ}{mL^3}$.

Plot the _total_ vertical displacement of the mass in the time interval $0\le\tau\le10$.

## Solution

We are going to

 - determine the structural matrices for the 2 DoF dynamic system,
 - compute its eigenvalues and its eigenvectors,
 - determine the pseudo-static mass motion when the support is displaced
 - write the EoM for the inertial forces due to the support motion,
 - determine the particular integrals and the homogeneous solution first, and next the frre vibrations,
 - present our results.

### Structural Matrices

Because the text of the problem gives us $\boldsymbol F$, we can easily compute $\boldsymbol K$ by inversion of the flexibility; the mass matrix is super easy... $\boldsymbol M = m \boldsymbol I$

```python
F = array(((3., 2.), (2., 96.)))/6
K = array(((96., -2.), (-2., 3)))*6/(96*3-2*2)
M = array(((1., 0.), (0., 1.)))
```

### Eigenvalues, eigenvectors, frequencies

Next we compute the eigenvalues and the eigenvectors

```python
l2, Psi = eigh(K, M)
l1 = sqrt(l2)
L2 = diag(l2) ; L1 = sqrt(L2)
```

#### The structural matrices, the eigen* and the frequencies

```python
Psi[:,0] *= -1 # change sign to the first column
dL(r'\begin{align}',
   r'\boldsymbol M &= m\,', prmat(M), r',\\',
   r'\\',
   r'\boldsymbol F &= \frac{L^3}{6EJ}\,', prmat(6*F), ',&',
   r'\boldsymbol K &= \frac{3EJ}{142L^3}\,', prmat(142*K/3), r',\\',
   r'\\',
   r'\boldsymbol \Lambda^2 &=', prmat(L2), ',&',
   r'\boldsymbol \Lambda &=', prmat(L1), r',\\',
   r'\\',
   r'\boldsymbol \Psi &=', prmat(Psi), '.',
   r'\end{align}')
```

\begin{align} \boldsymbol M &= m\, \begin{bmatrix}+1&+0\\+0&+1\end{bmatrix} ,\\ \\ \boldsymbol F &= \frac{L^3}{6EJ}\, \begin{bmatrix}+3&+2\\+2&+96\end{bmatrix} ,& \boldsymbol K &= \frac{3EJ}{142L^3}\, \begin{bmatrix}+96&-2\\-2&+3\end{bmatrix} ,\\ \\ \boldsymbol \Lambda^2 &= \begin{bmatrix}+0.062472&+0\\+0&+2.02908\end{bmatrix} ,& \boldsymbol \Lambda &= \begin{bmatrix}+0.249944&+0\\+0&+1.42446\end{bmatrix} ,\\ \\ \boldsymbol \Psi &= \begin{bmatrix}+0.0214905&-0.999769\\+0.999769&+0.0214905\end{bmatrix} . \end{align}

### Kinematics of the Problem
Starting from the imposed displacement of the left node we compute also its velocity and its acceleration,
next we analyze the effects of a static displacement on the position of the supported mass.

#### The node displacement and its derivatives
We use a _polynomial class_ to represent the displacement of $\mathcal A$ (the coefficients must be given in descending order).  The class supports evaluation and derivation (and other things that we are not going to use).

```python
d = p(3, -15, 20, 0, 0, 0)/16
v = d.deriv()
a = v.deriv()
```

```python
dL('\\begin{align}',
       r"x_\mathcal{A} &= ", ltx_p(d), r',\\',
       r"\frac{dx_\mathcal{A}}{d\tau} = x'_\mathcal{A} &= ", ltx_p(v), r',\\',
       r"\frac{d^2x_\mathcal{A}}{d\tau^2} = x''_\mathcal{A} &= ", ltx_p(a),
       r'=\frac{15}{4}\,\left(', ltx_p(p(1,-3,2,0)), r'\right).'
   '\\end{align}')
```

\begin{align} x_\mathcal{A} &=  0.1875\,{\tau}^{5}-0.9375\,{\tau}^{4}+1.25\,{\tau}^{3} ,\\ \frac{dx_\mathcal{A}}{d\tau} = x'_\mathcal{A} &=  0.9375\,{\tau}^{4}-3.75\,{\tau}^{3}+3.75\,{\tau}^{2} ,\\ \frac{d^2x_\mathcal{A}}{d\tau^2} = x''_\mathcal{A} &=  3.75\,{\tau}^{3}-11.25\,{\tau}^{2}+7.5\,{\tau} =\frac{15}{4}\,\left( \,{\tau}^{3}-3\,{\tau}^{2}+2\,{\tau} \right).\end{align}

#### The influence matrix

The  displacement of the supported mass is analyzed degrading the $\mathcal{A}$ hinge to a roller: the structure is hence a rigid system with 1 DoF and the CIR of the left-top bar is on the vertical of the roller and on the line joining the two hinges.

For $x_\mathcal{A}=1$ we have $x_1=1/2$ and $x_2=2$, hence $E=\begin{Bmatrix}1/2&2\end{Bmatrix}^T$. The modal load is $\boldsymbol{p}^\star=-\boldsymbol\Psi^T\,\boldsymbol{M}\,\boldsymbol{E}\,\ddot{x}_\mathcal{A}
=-m\boldsymbol\Psi^T\,\boldsymbol{E}\,\ddot{x}_\mathcal{A}
=m\boldsymbol\Gamma\,\ddot{x}_\mathcal{A}$
where $\boldsymbol\Gamma = -\boldsymbol\Psi^T\,\boldsymbol{E}$.

```python
E = array((0.5, 2.0))
G = -Psi.T@E
```

```python
dL(r'$$\boldsymbol E=', prmat(E[:,None], type='B'), r',\qquad',
   r'\boldsymbol\Gamma=', prmat(G[:,None], type='B'),'.$$')
```

$$\boldsymbol E= \begin{Bmatrix}+0.5\\+2\end{Bmatrix} ,\qquad \boldsymbol\Gamma= \begin{Bmatrix}-2.01028\\+0.456904\end{Bmatrix} .$$

##### Plots of displacement, velocity and acceleration of the node
... against a dimensionless time that runs from `0` to `2` (note the normalization factors, especially the ones for velocity and acceleration).

```python
t02 = linspace(0,2,201)
plt.plot(t02, d(t02), label=r'd${}/\Delta$')
plt.plot(t02, v(t02), label=r'v${}/(\omega_0\Delta)$')
plt.plot(t02, a(t02), label=r'a${}/(\omega_0^2\Delta)$')
plt_labels(xl='$\\omega_0t$', c=3,
           t='Normalized displacement, velocity and acceleration')
```

### Dynamic Response

1. We write the EoM with reference to the dimensionless time $\tau = \omega_0t$,
2. we find the particular integral that satisfies the EoM when the mass is subjected to the _static acceleration_,
3. the forced modal responses are determined imposing the initial conditions for $\tau=0$ and
4. eventually we determine the free response, this time imposing that the displacement and the velocity of the forced and the free responses are equal for $\tau=2$ .

#### The modal EoM with respect to dimensionless time
The equation of motion is $\boldsymbol M \ddot{\boldsymbol x} + \boldsymbol K \boldsymbol x = - \boldsymbol M \boldsymbol E \ddot x_\mathcal{A}$.

If we denote with primes the derivative with respect to $\tau=\omega_0t$, it is $\dot f = \omega_0 f'$ and the equation of motion is
$$
  \omega_0^2\boldsymbol M \boldsymbol x'' + \boldsymbol K \boldsymbol x =
- \omega_0^2\boldsymbol M \boldsymbol E x''_\mathcal{A}.
$$

Applying the modal transformation, with $\boldsymbol M^\star=\boldsymbol M=m\boldsymbol I$ (that implies that
$\boldsymbol M\boldsymbol E = m\boldsymbol E$)
$$
    m\omega_0^2\boldsymbol q'' + m\omega_0^2\boldsymbol \Lambda^2 \boldsymbol q =
  - m\omega_0^2\boldsymbol\Psi^T \boldsymbol E x''_\mathcal{A}
$$
and, simplifying $m\omega_0^2$ and developing the right member
$$
  \boldsymbol q'' + \boldsymbol \Lambda^2 \boldsymbol q =
- \boldsymbol\Psi^T \boldsymbol E x''_\mathcal{A} =
  \frac{15}{4}\boldsymbol\Gamma(\tau^3-3\tau^2+2\tau),
  \qquad\text{where } \boldsymbol\Gamma = -\boldsymbol\Psi^T\boldsymbol{E}.
$$

#### Particular Integral
A particular integral (omitting the indices) is $\xi(\tau) = P\tau^3 + Q\tau^2+R\tau+S$. With the provision that we have to multiply our results by $15\boldsymbol\Gamma/4$ we can write
$$
  6P\tau + 2Q + \lambda^2\left(P\tau^3 + Q\tau^2+R\tau+S\right) = \tau^3 -3\tau^2+2\tau.
$$
Equating for each power of $\tau$ the coefficients on both sides, we have
$$
  \lambda^2P=1,\qquad \lambda^2Q=-3,\qquad 6P+\lambda^2R=2,\qquad 2Q+\lambda^2S=0
$$
a set of equations that can be readily translated to code (taking also into account the multiplication by $15\gamma_i/4$).

```python
P = 1/l2 ; Q = -3/l2 ; R = (2-6*P)/l2 ; S = (0-2*Q)/l2
P, Q, R, S = map(lambda coeff: 3.75*coeff*G, (P, Q, R, S))
xi = [p(P[i], Q[i], R[i], S[i]) for i in (0, 1)]
```

```python
dL(r'\begin{align}',
   r'\\'.join(r'\xi_{%d} &= %s'%(i,ltx_p(xi_i)) for i, xi_i in enumerate(xi, 1)),
   r'\end{align}')
```

\begin{align} \xi_{1} &= -120.671\,{\tau}^{3}+362.013\,{\tau}^{2}+11348.3\,{\tau}-11589.6\\\xi_{2} &= 0.844418\,{\tau}^{3}-2.53325\,{\tau}^{2}-0.808115\,{\tau}+2.49695 \end{align}

#### Modal Responses, Forced Phase
It is $q_i(\tau)=A_i\cos\lambda_i\tau + B_i\sin\lambda_i\tau + \xi_i(0)$; the $\xi$ have been determined, we have to determine the constants of integration $A$ and $B$ imposing that $\boldsymbol q(0) = \boldsymbol 0$ and $\boldsymbol q'(0) = \boldsymbol 0$: it is $A_i=-\xi_i(0)$ and $\lambda_iB_i=-\xi_i'(0)$.

```python
xi0 = array([Xi(0) for Xi in xi])          ; A = -xi0
dxi0 = array([Xi.deriv()(0) for Xi in xi]) ; B = -dxi0/l1
```

##### Plot of the forced modal responses
We compute first the 2D array `l1t1`, that has $\lambda_i\omega_0t$ in its two columns, next the modal response `q02` because it's the first phase of the response for $0\le\tau\le2$...

```python
l1t02 = outer(t02, l1)
q02 = A*cos(l1t02) + B*sin(l1t02) + array([Xi(t02) for Xi in xi]).T
dq02 = -A*l1*sin(l1t02) + B*l1*cos(l1t02) + array([Xi.deriv()(t02) for Xi in xi]).T
```

```python
plt.legend(plt.plot(t02, q02), (r'$q_1/\Delta$','$q_2/\Delta$'))
plt_labels(xl='$\\omega_0t$', l=0, t='Normalized modal responses, forced phase')
```

#### Modal Responses, free phase
The free response starts at `t22=2`, we can impose that the modal displacements and derivatives of the free response, computed at `t22`, are equal to the modal displacements and derivatives for the forced phase. Indicating with a ${}^1$ the quantities relative to the forced phase and with ${}^2$ the ones relative to free response,
$$\begin{bmatrix}\cos\lambda_i\tau_{22} &\sin\lambda_i\tau_{22}\\
-\sin\lambda_i\tau_{22} &\cos\lambda_i\tau_{22}\end{bmatrix}
\begin{Bmatrix}{}^2A_i\\{}^2B_i\end{Bmatrix} = \begin{Bmatrix}
{}^1q_i(\tau_{22})\\\frac{\displaystyle{{}^1\dot q_i(\tau_{22})}}{\lambda_i}\end{Bmatrix}.
$$
The coefficient matrix is an orthogonal matrix (i.e., $\boldsymbol A^{-1}\equiv\boldsymbol A^T$), so we can write our solution as follows

```python
t22 = 2
C22 = cos(l1*t22)
S22 = sin(l1*t22)
q22 = q1[-1]
dq22_over_l1 = dq1[-1]/l1
Afree = C22*q22-S22*dq22_over_l1
Bfree = S22*q22+C22*dq22_over_l1
```

It is now possible to compute the modal response in $2\le\tau\le10$, so that we can compute the total displacements of the mass in said interval, as requested by the problem.

```python
t210 = linspace(2.01, 10, 800)
l1t210 = outer(t210, l1)
q210 = Afree*cos(l1t210) + Bfree*sin(l1t210)
dq210 = -l1*Afree*sin(l1t210) + l1*Bfree*cos(l1t210)
```

### Summing Up and Showing  the Results

Finally we stick together the forced and the free responses, and compute the nodal deformations by post-multiplying the modal responses by the transpose of `Psi` and the static displacement by evaluating the polynomial `d` and taking the outer product with the influence matrix, finally we compute the total displacement of the mass

```python
t = hstack((t02, t210))
q = vstack((q02, q210))
dq = vstack((dq02, dq210))
x = q@Psi.T
xst = outer(where(t<t22,d(t), 1), E)
xtot = xst + x
```

#### The Plot of the Total Displacements

... normalized with respect to the final support of the displacement.

```python
plt.legend(plt.plot(t, xtot), [r'$x_{%d,\mathrm{tot,}}/\Delta$'%i for i in (1,2) ])
plt_labels(xl='$\\omega_0t$', t='Normalized total displacements', l=0)
```

#### The Total Displacements over a Longer Time Span

The requirements of the problem are satisfied, but I'm left with the curiosity of what happens to the vertical displacements in the following... let's plot the total displacements for $0\le\tau\le30$.

```python
t230 = linspace(2.01, 30, 2800)
l1t230 = outer(t230, l1)
q230 = Afree*cos(l1t230) + Bfree*sin(l1t230)
dq230 = -l1*Afree*sin(l1t230) + l1*Bfree*cos(l1t230)

t = hstack((t02, t230))
q = vstack((q02, q230))
dq = vstack((dq02, dq230))
x = q@Psi.T
xst = outer(where(t<t22,d(t), 1), E)
xtot = xst + x
```

```python
plt.legend(plt.plot(t, xtot),
           [r'$x_{%d,\mathrm{tot,}}/\Delta$'%i for i in (1,2)])
plt.xlim((-0.5, 30.5))
plt_labels(xl=r'$\omega_0t$', yl=r'$x_i^\mathrm{total}/\Delta$',
           t='Normalized total displacements', xt=range(0, 31,2), l=0)
```

Quite a sling effect...

## Post Scriptum

That's all, except that here I show you the code that is executed at the beginning of the notebook.

There is the initialization of the plotting machinery and the imports, a further initialization of the plotting _style_ and eventually the definition of a number of utility functions.

If you were interested (who knows...) in the gory details of the production of the mathematical displays and the plots,
you can examine [the notebook (code+text) that is the source of this paper](http://nbviewer.jupyter.org/url/boffi.github.io/dati_2018/hw03/2dof.ipynb).

### Initialization, Imports

```python
%matplotlib inline

import numpy as np
from numpy import array, cos, diag, hstack, linspace
from numpy import outer, sin, sqrt, vstack, where
import matplotlib.pyplot as plt
from scipy.linalg import eigh
```

```python
from jupyterthemes import jtplot
jtplot.style(context='paper', fscale=1.5, figsize=(15, 3))
```

### Function Definitions

#### An helper funtion for the decoration of plots
The Matplotlib API is a little verbose, this is an attempt to squeeze the typical decoration that is applied to a plot in a single statement

```python
def plt_labels(xl='', yl='', xt='', yt='', t='', l=1, c=1):
    if xl: plt.xlabel(xl)
    if yl: plt.ylabel(yl)
    if xt: plt.xticks(xt)
    if yt: plt.yticks(yt)
    if t:  plt.title(t)
    if l:  plt.legend(ncol=c)
```

#### Polynomials' stuff

A function to instantiate a library provided polynomial class and a formatter function to prepare the mathematical display of a polynomial.

```python
def p(*coefs):
    from numpy import poly1d
    return poly1d(coefs)

def ltx_p(p, leading_sign=False):
    order = p.order
    coeffs = p.coefficients
    def _fmt(i, c):
        coef = ('%+g' if (i or leading_sign) else '%g') % c
        if c == 1 : coef = '+' if (i or leading_sign) else ''
        if c == -1: coef = '-'
        var  = '{\\tau}' if i < order else ''
        expn = '^{%d}'%(order-i) if i < order-1 else ''
        return coef + ('\\,' if var else '') + var + expn
    return ''.join(_fmt(i, c) for i, c in enumerate(coeffs) if c)
```

#### Other math stuff

A formatter to prepare the math display of a matrix and the function that is actually used to render the math displays.

```python
def prmat(mat, fmt='%+.6g', type='b'):
    return (r'\begin{%smatrix}' +
            r'\\'.join('&'.join(fmt%num for num in row) for row in mat) +
            r'\end{%smatrix}')%(type, type)

def dL(*pieces):
    from IPython.display import Latex
    display(Latex(' '.join(pieces)))
```


```python
import cvxpy as cp
import numpy as np
import cmath
import hsbalance as hs
import matplotlib.pyplot as plt
%matplotlib widget
```

    Using the Community license in this session. If you have a full Xpress license, first set the XPAUTH_PATH environment variable to the full path to your license file, xpauth.xpr, and then restart Python. If you want to use the FICO Community license and no longer want to see this message, set the XPAUTH_PATH environment variable to: /home/maged/anaconda3/envs/balance/lib/python3.8/site-packages/xpress/license/community-xpauth.xpr
    NB: setting XPAUTH_PATH will also affect any other Xpress products installed on your system.

# Introduction

This notebook is aimed to give a quick presentation for `hsbalance` package.
`hsbalance` is Python based package that is meant to find the optimized solution of rotor balancing problem

# A. Independent Systems
# # Creating Model
for independent systems where number of balancing planes are equal to the number of measuring points we do not need optimization process as number of equations are equal to the number of unknowns

1. Enter the initial vibration column vector `A`:
- each row represents the vibration at certain measuring plane.
- vibration is to be represented in the form ('amplitude' @ 'phase(degrees)')
- Enter slow roll vibration column `A0`

```python
A_math = [['170@112'],
     ['53@78']]

A0_math = [['12@30'],
      ['12@30']]
```

2. Enter trial mass effect matrix `B`
B = [['B00', 'B01']
     ['B10', 'B11']]
where:
- B00: vibration at measuring point 1 when trial mass at balancing plane 1
- B01: vibration at measuring point 1 when trial mass at balancing plane 2
- B00: vibration at measuring point 2 when trial mass at balancing plane 1
- B00: vibration at measuring point 2 when trial mass at balancing plane 2

As a general rule in this notebook columns will be for balancing planes and rows are for measuring points

```python
B_math = [['235@94', '189@115'],
     ['58@68', '77@104']]
```

3. Enter the trial mass amounts in row vector `U`:

```python
U_math = ['1.15@0', '1.15@0']
```

4. Transform matrices to cartesian (complex number) form:

```python
A = hs.convert_matrix_to_cart(A_math)
A0 = hs.convert_matrix_to_cart(A0_math)
B = hs.convert_matrix_to_cart(B_math)
U = hs.convert_matrix_to_cart(U_math)
print('A=\n{}\n\nA0=\n{}\n\nB=\n{}\n\nU = {}'.format(A, A0, B, U))
```

    A=
    [[-63.68312088+157.62125528j]
     [ 11.01931961 +51.84182284j]]

    A0=
    [[10.39230485+6.j]
     [10.39230485+6.j]]

    B=
    [[-16.39277133+234.42755181j -79.87485147+171.29217175j]
     [ 21.72718242 +53.77666356j -18.62798596 +74.71277092j]]

    U = [1.15+0.j 1.15+0.j]

As in this example, this is an independent system where number of measuring points (M) are equal to the number of balancing planes (N).
we, thus, except an exact solution for balancing weights `W` that can be calculated first by find the Influence Coefficients matrix `ALPHA`:
\begin{align}
  \tag {1}
  \alpha = \frac{(B - A)}{U} \label{eq:test1}
\end{align}
\begin{align}
    \tag {2}
    W = - \alpha^{-1}(A - A_{0})
\end{align}

```python
Alpha_CI = (B - A)/U
W = -np.linalg.inv(Alpha_CI) @ (A - A0)
```

```python
hs.convert_matrix_to_math(Alpha_CI)
```

    array([['78.433@58.4', '18.427@139.8'],
           ['9.462@10.2', '32.56@142.4']], dtype='<U12')

```python
W
```

    array([[-0.99141015-1.68129487j],
           [-0.67544632+0.58516963j]])

Transform back to mathematical expression form

```python
ALPHA_math = hs.convert_matrix_to_math(Alpha_CI)
W_math = hs.convert_matrix_to_math(W)
print('ALPHA=\n{}\n\nW=\n{}'.format(ALPHA_math, W_math))
```

    ALPHA=
    [['78.433@58.4' '18.427@139.8']
     ['9.462@10.2' '32.56@142.4']]

    W=
    [['1.952@239.5']
     ['0.894@139.1']]

>This means we need to put 2 grams at angel 57.4 degrees on balancing plane 1, and 1.1 grams at 301.1 degrees on plane 2.

- Lets Try out the same independent system problem using our optimization modeling code:
- first we will create a model of the system parameters:
- we will be using least squares model to solve the problem using optimization technique to minimize the squares of errors:

```python
alpha = hs.Alpha()  # create an instance of alpha class
alpha.add(A=A, B=B, U=U)  # calculate alpha from parameters
```

```python
hs.convert_matrix_to_math(alpha.value)
```

    array([['78.433@58.4', '18.427@139.8'],
           ['9.462@10.2', '32.56@142.4']], dtype='<U12')

```python
my_model = hs.LeastSquares(A-A0, alpha)
```

```python
W = my_model.solve()
```

```python
hs.convert_matrix_to_math(W)
```

    array([['1.952@239.5'],
           ['0.894@139.1']], dtype='<U11')

```python
my_model.rmse()
```

    0.0

Which is exactly as the exact solution

# A. Dependent Systems
## Introduction
In dependent systems, number of measuring points are less than the number of balancing planes.
This will lead to a problem with infinite number of solutions as the number of unknowns are less than the number of equations.
We can use optimization technique here effectively to reduce the error and we can apply constraints to our model.

We will be example.[[1]](#1) which presents a 1150 MW nuclear power turbine-generator
balancing problem. The system consists of 11 measuring points and 5 balancing planes. (independent system)
In practical plane #4 was not accessible.

[1]
Gunter, E J, Allaire, P E, and Foiles, W C. Balancing a 1150 MW turbine-generator. United Kingdom: N. p., 2000. Web.

## Parameters

```python
ALPHA_math=[
                            ['9.8@117', '17@124', '7.2@114', '38.5@77'],
                            ['2.7@43', '14.3@317', '4.5@213', '14.3@270'],
                            ['12.5@323', '25@261', '15.2@158', '30@238'],
                            ['22.4@92', '32.6@45', '23.3@315', '27.8@210'],
                            ['26@94', '40.3@9', '25@330', '34@213'],
                            ['40.3@355', '43@144', '29.6@61', '65.4@322'],
                            ['20.6@339', '32.3@152', '36.7@41', '61.8@322'],
                            ['12.6@226', '37.6@52', '18.8@153', '26@176'],
                            ['13.4@209', '26.9@76', '47.5@98', '71.7@312'],
                            ['13.4@154', '22.4@307', '52@299', '102@165'],
                            ['5.4@24', '7.2@199', '22.4@2', '27.8@99']]

A_math=[
                            ['55@259'],
                            ['45@118'],
                            ['124@21'],
                            ['138@349'],
                            ['107@349'],
                            ['90@280'],
                            ['58@354'],
                            ['108@201'],
                            ['88@190'],
                            ['56@48'],
                            ['73@158']]
```

Convert to complex numbers (cartesian) form

```python
A = hs.convert_matrix_to_cart(A_math)
ALPHA = hs.convert_matrix_to_cart(ALPHA_math)
# A, ALPHA
```

Adding ALPHA

```python
alpha = hs.Alpha()
alpha.add(direct_matrix=ALPHA)
```

```python
alpha.check()
```

    Not a square matrix --> no exact solution

    No ill conditioned planes --> ok

## Solving with Least squares:

```python
model_LeastSquares = hs.LeastSquares(A, alpha, name='Least_squares') # Instantiate least square model
```

```python
W_LeastSquares = model_LeastSquares.solve() #solve
```

```python
hs.convert_matrix_to_math(W_LeastSquares)
```

    array([['3.827@90.7'],
           ['2.243@358.4'],
           ['1.747@299.3'],
           ['1.461@292.5']], dtype='<U11')

```python
residuals_LeastSquares = model_LeastSquares.expected_residual_vibration()
hs.convert_matrix_to_math(residuals_LeastSquares) # Expected residule vibrations
```

    array([['19.858@266.8'],
           ['37.648@147.4'],
           ['106.573@35.2'],
           ['59.988@5.1'],
           ['63.118@328.9'],
           ['39.649@125.6'],
           ['58.524@355.2'],
           ['67.018@112.2'],
           ['51.873@242.4'],
           ['33.458@132.4'],
           ['46.913@141.6']], dtype='<U12')

Root mean square error:

```python
rmse_LeastSquares = model_LeastSquares.rmse()
rmse_LeastSquares
```

    53.1472

### Discussion
Least square has iterated over to get the minimum squares of errors (hence, the least `RMSE`)
Doing so, it does it blindly so we can see that it add huge amount of weight at plane 1 (3.8 kg!), meanwhile vibration on bearing #3 is expected to be 106 $\mu$ which is probably an alarm value!!

## Solving with MinMax:

```python
model_MinMax = hs.Min_max(A, alpha, name='MinMax') # Instantiate MinMax model
```

```python
W_MinMax = model_MinMax.solve() #solve
```

```python
hs.convert_matrix_to_math(W_MinMax)
```

    array([['4.423@88.6'],
           ['2.892@352.5'],
           ['1.537@322.5'],
           ['1.91@305.5']], dtype='<U11')

```python
residuals_MinMax = model_MinMax.expected_residual_vibration()
hs.convert_matrix_to_math(residuals_MinMax) # Expected residule vibrations
```

    array([['10.271@95.7'],
           ['31.904@177.5'],
           ['69.941@30.1'],
           ['69.51@13.7'],
           ['69.941@330.6'],
           ['69.941@105.2'],
           ['69.941@14.9'],
           ['69.941@108.8'],
           ['69.941@249.7'],
           ['52.389@136.6'],
           ['69.941@116.7']], dtype='<U12')

Root mean square error:

```python
rmse_MinMax = model_MinMax.rmse()
rmse_MinMax
```

    59.4235

### Discussion
`MinMax` is a great optimization tool that tends to, instead of unbiased `Least Squares`, level up the residuals to minimize the maximum. here we see that we have a great improvement in the residual vibrations (max 70$\mu\$).
The downside is putting more and more weights in the correction (4.4 grams in plane 1 now!) and ends up with higher RMSE.

In order to constraint the weight to a certain limit we can perform a Constrained Minmax` model

```python
weight_const ={0 : 3.402, 1 : 3.402, 2 : 3.402, 3 : 3.402} # limit weight to 120 oz
model_MinMax_const = hs.Min_max(A, alpha, weight_const=weight_const, name='MinMax_const') # Instantiate MinMax model
```

```python
W_MinMax_const = model_MinMax_const.solve() #solve
```

```python
hs.convert_matrix_to_math(W_MinMax_const)
```

    array([['3.402@91.0'],
           ['2.322@354.6'],
           ['1.363@317.7'],
           ['1.778@309.7']], dtype='<U11')

```python
residuals_MinMax_const = model_MinMax_const.expected_residual_vibration()
hs.convert_matrix_to_math(residuals_MinMax_const) # Expected residule vibrations
```

    array([['7.447@43.0'],
           ['31.72@168.6'],
           ['72.931@27.9'],
           ['72.931@2.6'],
           ['72.931@329.7'],
           ['13.579@126.6'],
           ['72.931@352.2'],
           ['72.931@128.6'],
           ['72.931@249.9'],
           ['69.667@132.0'],
           ['67.667@121.9']], dtype='<U12')

Root mean square error:

```python
rmse_MinMax_const = model_MinMax_const.rmse()
rmse_MinMax_const
```

    57.0606

### Discussion
Constrained MinMAx` has done its job in minimizing the weights to 3.402 Kg (120 oz).
The downside is that we got more maximum vibration in residuals (73$\mu\$)

## Solving with Linear Matrix Inequality (LMI)

In certain situations, instead of being unbiased ---> `Least Squares` or leveled ---> `MinMax`, we actually want to be BIASED to certain planes. In other words we want the optimzer to do its best to decrease certain planes (`critical planes`) and keep the others under a `lazy constrains` just below certain amount of vibration level.

```python
weight_const ={0 : 3.402, 1 : 3.402, 2 : 3.402, 3 : 3.402} # limit weight to 120 oz
critical_planes = {1, 9} #  setting the critical planes to be 2, 10 (note python start counting at 0)
V_max = 76 # max vibration for non-critical planes
model_LMI = hs.LMI(A, alpha, weight_const=weight_const, critical_planes=critical_planes, V_max=V_max
                     , name='LMI') # Instantiate LMI model
```

```python
W_LMI = model_LMI.solve() #solve
```

```python
hs.convert_matrix_to_math(W_LMI)
```

    array([['3.402@98.0'],
           ['1.951@358.6'],
           ['1.298@337.6'],
           ['1.605@314.8']], dtype='<U11')

```python
residuals_LMI = model_LMI.expected_residual_vibration()
hs.convert_matrix_to_math(residuals_LMI) # Expected residule vibrations
```

    array([['4.731@224.2'],
           ['31.601@163.9'],
           ['76.0@29.0'],
           ['76.0@350.9'],
           ['76.0@325.9'],
           ['24.723@129.9'],
           ['76.0@7.5'],
           ['76.0@145.6'],
           ['67.023@237.1'],
           ['45.143@123.2'],
           ['74.507@120.0']], dtype='<U12')

Root mean square error:

```python
rmse_LMI = model_LMI.rmse()
rmse_LMI
```

    57.0662

### Discussion
LMI model has been biased to plane 2 with a slight enhancement and plane 10 which greatly decreased from 69.7$\mu$ to 45.1$\mu$ (35% decrease) but that was with the cost of increasing non critical planes to the limit we have assigned (76$\mu\$)

```python

```

## Plotting

```python
models = [model_LeastSquares, model_MinMax, model_MinMax_const, model_LMI]
```

```python
def plot_models(models):
    residule_vibration = {model.name:abs(model.expected_residual_vibration().ravel()) for model in models}
    rmse = {model.name:model.rmse() for model in models}
    fig, (ax0, ax1) = plt.subplots(2, 1)
    ax0.bar(rmse.keys(), rmse.values())
    plt.xlabel('Models')
    plt.ylabel('Vibration');
    models_number = len(residule_vibration.values())
    measuring_points = max((len(array) for array in residule_vibration.values()))

    jet= plt.get_cmap('jet')
    colors = iter(jet(np.linspace(0,1,models_number)))

    step = 0
    for array in residule_vibration.values():
        ax1.bar(np.arange(len(array)) + step, array, color = next(colors), width = 1/models_number)
        step += 1 / (models_number+1)
    ax1.legend([model.name for model in models])
    ax1.set_xticks(range(measuring_points), ['M.P '+ str(point) for point in range(1, 1+measuring_points)],
                  rotation =45);
```

```python
plot_models(models)
```

        Figure

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


Gate Decomposition

```
%load_ext sympyprinting
```

```
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

Example 1

Create a symbolic controlled-Y gate

```
CY10 = CGate(1, Y(0)); CY10

```

Decompose it into elementary gates and plot it

```
CY10.decompose()

```

```
circuit_plot(CY10.decompose(), nqubits=2)
```

Example 2

Create a controlled-Z gate

```
CZ01 = CGate(0, Z(1)); CZ01

```

Decompose and plot it

```
CZ01.decompose()

```

```
circuit_plot(CZ01.decompose(), nqubits=2)
```

Example 3

Create a SWAP gate

```
SWAP10 = SWAP(1, 0); SWAP10

```

Decompose and plot it

```
SWAP10.decompose()
```

```
circuit_plot(SWAP10.decompose(), nqubits=2)
```

All together now

```
gates = [CGate(1,Y(0)), CGate(0,Z(1)), SWAP(1, 0)]
```

```
for g in gates:
    dg = g.decompose()
    display(Eq(g, dg))
    circuit_plot(g, nqubits=2)
    circuit_plot(dg, nqubits=2)
```


# First steps toward functional testing, part 2

Author(s): Jukka Aho <jukka.aho@kapsi.fi>

Abstract: This is demonstration document describing how to combine functional testing, continous integration and documentation in to the single document.

These multi-purposal documents test several things at once. While they test that documentation is up-to-date, calculation results are what is expected and code base is working, they also serve as a tutorials and examples how to use JuliaFEM.

Let's start. **Hello world**. *Local* time is here

```julia
Libc.strftime(time())
```

Basic testing

```julia
f(x) = 2*x
f(1)
```

```julia
@assert f(1) == 2
```

\begin{equation}
A_{ij}=\int_{0}^{L}\phi_{i}\phi_{j}\,\mathrm{d}x
\end{equation}

## Use FactCheck to validate your results

We actually use FactCheck to test our results

```julia
using FactCheck
```

```julia
facts("test that f(1) equals to 2") do
    @fact f(1) => 2
end
```

```julia

```


# 06 Calculus with Sympy and Numpy
# Review Exercises - Solutions

## Review Exercise 1 : Differentiation

$$f(x) = cos(x) + bx + c$$

Use symbolic math to find the second derivative of $f(x)$

```python

```

```python
# Review Exercise 1
# Example Solution
import sympy as sp
from sympy import diff, symbols

b, c, x = symbols('b c x')

f = sp.cos(x) + b*x + c

print(diff( f, x, 2 ) )
```

    -cos(x)

## Review Exercise 2 : Integration
$$f(x) = cos(x) + bx + c$$

Find: $\int_{0}^{1} f(x) \; dx$:

1. Symbolically
2. Non-symbolically (where $b=2$, $c=5$)

```python

```

```python
# Review Exercise 2 : Symbolically
# Example Solution

from sympy import integrate
from sympy import pprint

b, c, x = symbols('b, c, x')
f = sp.cos(x) + b*x + c
pprint(integrate(f, (x, 0, 1)))
```

    b
    ─ + c + sin(1)
    2

```python

```

```python
# Review Exercise 2 : Non Symbolically
# Example Solution
import numpy as np
from scipy.integrate import quad

# first create a function to describe the integrand
def integrand(x):
    return np.cos(x) + 2*x + 5

# create two variables to store the integral and the error
ans, err = quad(integrand, 0, 1)

print(ans)
print(f'The estimated error is {err}')
```

    6.841470984807897
    The estimated error is 7.595558609638671e-14

```python

```

## Review Exercise 3 : Estimating the integral of exerimental data

Use the trapezoid method to integrate the array of raw experimental `y` data with respect to indpendent variable `x`.

```
x = np.array([0, 10, 15, 30, 35, 50, 75, 80])

y = np.array([5, 6, 1, 1.5, 2, 5, 6, 1])
```

```python
# Review Exercise 3
```

```python
# Review Exercise 3
# Example Solution

x = np.array([0, 10, 15, 30, 35, 50, 75, 80])

y = np.array([5, 6, 1, 1.5, 2, 5, 6, 1])

print(np.trapz(y, x))
```

    307.5

## Review Exercise 4: Solving ordinary differntial equations.

During braking, a car’s velocity is given by $v = v_{0} e^{−t/R}$.

$v = \frac{dD}{dt}$

Initial condition: distance, $D(0) = 0$

Solve the equtaion using symbolic mathematics to show that the distance travelled while braking as a function of time can be expressed as:

$$D(t) =  Rv_0 - R v_0 e^{−t/R} $$

a) Define variables as symbols sybolically. Define function `D` (distance).

b) Use `Eq` from the `sympy` package to define the differential equation.

c) Use `dsolve` to find the general solution to the equation by solving the equation symbolically.

d) Find the value of the constant of ingtegration $C1$.

e) Substitute $C1$ into the general solution to find the exact solution to the equation.

```python
# Review Exercise 4
# Example Solution
import sympy as sp
from sympy import symbols, Eq, Function, dsolve, exp, solve
from IPython.display import display

# a) define symbols
t, v0, R, C1 = symbols("t v0 R C1")
D = Function("D")

# b)define differential equation
eqn = Eq(D(t).diff(t), v0*exp(-t/(R)))
display(eqn)

# c) find general solution
gen_sol = dsolve(eqn, D(t))
display(gen_sol)

# d) find C1
cnts = gen_sol.subs([(t,    0),       # x = 0 --> subs 0 for x
                     (D(0), 0)])      # f(0) = 0 --> subs 0 for f(0)

display(cnts)

C_1 = solve([cnts], (C1))

display(C_1)

# e) Substitute C1 into the general solution to find exact solution
ode_sol = gen_sol.subs(C_1)
display(ode_sol)
```

$\displaystyle \frac{d}{d t} D{\left(t \right)} = v_{0} e^{- \frac{t}{R}}$

$\displaystyle D{\left(t \right)} = C_{1} - R v_{0} e^{- \frac{t}{R}}$

$\displaystyle 0 = C_{1} - R v_{0}$

    {C1: R*v0}

$\displaystyle D{\left(t \right)} = R v_{0} - R v_{0} e^{- \frac{t}{R}}$

```python
# Review Exercise 4
```

## Review Exercise 5 : Solving ordinary differntial equations.

$\frac{dy}{dx} + y = x$

Initial value: $y(0) = 1$

Find the numerical solution for $y$ using `odeint` for the range $x = [0, 5]$

```python
# Review Exercise 5
```

```python
# Review Exercise 5
# Example solution
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import odeint
%matplotlib inline

xs = np.linspace(0,5,100)
y0 = 1.0  # initial condition

# Define a function which calculates the derivative
def dy_dx(y, x):
    return x - y

y = odeint(dy_dx, y0, xs)
```

```python

```