```python
%pylab inline
import sk_dsp_comm.sigsys as ss
import sk_dsp_comm.fir_design_helper as fir_d
import sk_dsp_comm.iir_design_helper as iir_d
import sk_dsp_comm.multirate_helper as mrh
import scipy.signal as signal
from IPython.display import Audio, display
from IPython.display import Image, SVG
```

```python
%config InlineBackend.figure_formats=['svg'] # SVG inline viewing
```

# Filter Design Using the Helper Modules

The Scipy package *signal* assists with the design of many digital filter types. As an alternative, here we explore the use of the filter design modules found in `scikit-dsp-comm`
(https://github.com/mwickert/scikit-dsp-comm).

In this note we briefly explore the use of `sk_dsp_comm.fir_design_helper` and `sk_dsp_comm.iir_design_helper`. In the examples that follow we assume the import of these modules is made as follows:

```python
import sk_dsp_comm.fir_design_helper as fir_d
import sk_dsp_comm.iir_design_helper as iir_d
```

The functions in these modules provide an easier and more consistent interface for both finte impulse response (FIR) (linear phase) and infinite impulse response (IIR) classical designs. Functions inside these modules *wrap* `scipy.signal` functions and also incorporate new functionality.

# Design From Amplitude Response Requirements
With both `fir_design_helper` and `iir_design_helper` a design starts with amplitude response requirements, that is the filter passband critical frequencies, stopband critical frequencies, passband ripple, and stopband attenuation. The number of taps/coefficients (FIR case) or the filter order (IIR case) needed to meet these requirements is then determined and the filter coefficients are returned as an ndarray `b` for FIR, and for IIR both `b` and `a` arrays, and a second-order sections `sos` 2D array, with the rows containing the corresponding cascade of second-order sections toplogy for IIR filters.

For the FIR case we have in the $z$-domain
$$
    H_\text{FIR}(z) = \sum_{k=0}^N b_k z^{-k}
$$

with ndarray `b` = $[b_0, b_1, \ldots, b_N]$. For the IIR case we have in the $z$-domain

$$\begin{align}
    H_\text{IIR}(z) &= \frac{\sum_{k=0}^M b_k z^{-k}}{\sum_{k=1}^N a_k z^{-k}} \\
    &= \prod_{k=0}^{N_s-1} \frac{b_{k0} + b_{k1} z^{-1} + b_{k2} z^{-2}}{1 + a_{k1} z^{-1} + a_{k2} z^{-2}} = \prod_{k=0}^{N_s-1} H_k(z)
\end{align}$$

where $N_s = \lfloor(N+1)/2\rfloor$. For the `b/a` form the coefficients are arranged as
```python
b = [b0, b1, ..., bM-1], the numerator filter coefficients
a = [a0, a1, ..., aN-1], the denominator filter ceofficients
```
For the `sos` form each row of the 2D `sos` array corresponds to the coefficients of $H_k(z)$, as follows:
```python
SOS_mat = [[b00, b01, b02, 1, a01, a02], #biquad 0
           [b10, b11, b12, 1, a11, a12], #biquad 1
            .
            .
           [bNs-10, bNs-11, bNs-12, 1, aNs-11, aNs-12]] #biquad Ns-1
```

# Linear Phase FIR Filter Design
The primary focus of this module is adding the ability to design linear phase FIR filters from user friendly amplitude response requirements.

Most digital filter design is motivated by the desire to approach an ideal filter. Recall an ideal filter will pass signals of a certain of frequencies and block others. For both analog and digital filters the designer can choose from a variety of approximation techniques. For digital filters the approximation techniques fall into the categories of IIR or FIR. In the design of FIR filters two popular techniques are truncating the ideal filter impulse response and applying a window, and optimum equiripple approximations [Oppenheim2010](https://www.amazon.com/Discrete-Time-Signal-Processing-3rd-Prentice-Hall/dp/0131988425/ref=sr_1_1?ie=UTF8&qid=1519940790&sr=8-1&keywords=oppenheim+discrete+time+signal+processing&dpID=51v48p99JjL&preST=_SX218_BO1,204,203,200_QL40_&dpSrc=srch). Frequency sampling based approaches are also popular, but will not be considered here, even though `scipy.signal` supports all three. Filter design generally begins with a specification of the desired frequency response. The filter frequency response may be stated in several ways, but amplitude response is the most common, e.g., state how $H_c(j\Omega)$ or $H(e^{j\omega}) = H(e^{j2\pi f/f_s})$ should behave. A completed design consists of the number of coefficients (taps) required and the coefficients themselves (double precision float or `float64` in Numpy, and `float64_t` in C). Figure 1, below, shows amplitude response requirements in terms of filter gain and critical frequencies for lowpass, highpass, bandpass, and bandstop filters. The critical frequencies are given here in terms of analog requirements in Hz. The sampling frequency is assumed to be in Hz. The passband ripple and stopband attenuation values are in dB. Note in dB terms attenuation is the negative of gain, e.g., -60 of stopband gain is equivalent to 60 dB of stopband attenuation.

```python
Image('300ppi/FIR_Lowpass_Highpass_Bandpass_Bandstop@300ppi.png',width='90%')
```

There are 10 filter design functions and one plotting function available in `fir_design_helper.py`. Four functions for designing Kaiser window based FIR filters and four functions for designing equiripple based FIR filters. Of the eight just described, they all take in amplitude response requirements and return a coefficients array. Two of the 10 filter functions are simply wrappers around the `scipy.signal` function `signal.firwin()` for designing filters of a specific order when one (lowpass) or two (bandpass) critical frequencies are given. The wrapper functions fix the window type to the `firwin` default of hann (hanning). The remamining eight are described below in Table 1. The plotting function provides an easy means to compare the resulting frequency response of one or more designs on a single plot. Display modes allow gain in dB, phase in radians, group delay in samples, and group delay in seconds for a given sampling rate. This function, `freq_resp_list()`, works for both FIR and IIR designs. Table 1 provides the interface details to the eight design functions where d_stop and d_pass are positive dB values and the critical frequencies have the same unit as the sampling frequency $f_s$. These functions do not create perfect results so some tuning of of the design parameters may be needed, in addition to bumping the filter order up or down via `N_bump`.

```python
Image('300ppi/FIR_Kaiser_Equiripple_Table@300ppi.png',width='80%')
```

## Design Examples

### Example 1: Lowpass with $f_s = 1$ Hz
For this 31 tap filter we choose the cutoff frequency to be $F_c = F_s/8$, or in normalized form $f_c = 1/8$.

```python
b_k = fir_d.firwin_kaiser_lpf(1/8,1/6,50,1.0)
b_r = fir_d.fir_remez_lpf(1/8,1/6,0.2,50,1.0)
```

```python
fir_d.freqz_resp_list([b_k,b_r],[[1],[1]],'dB',fs=1)
ylim([-80,5])
title(r'Kaiser vs Equal Ripple Lowpass')
ylabel(r'Filter Gain (dB)')
xlabel(r'Frequency in kHz')
legend((r'Kaiser: %d taps' % len(b_k),r'Remez: %d taps' % len(b_r)),loc='best')
grid();
```

```python
b_k_hp = fir_d.firwin_kaiser_hpf(1/8,1/6,50,1.0)
b_r_hp = fir_d.fir_remez_hpf(1/8,1/6,0.2,50,1.0)
```

```python
fir_d.freqz_resp_list([b_k_hp,b_r_hp],[[1],[1]],'dB',fs=1)
ylim([-80,5])
title(r'Kaiser vs Equal Ripple Lowpass')
ylabel(r'Filter Gain (dB)')
xlabel(r'Frequency in kHz')
legend((r'Kaiser: %d taps' % len(b_k),r'Remez: %d taps' % len(b_r)),loc='best')
grid();
```

```python
b_k_bp = fir_d.firwin_kaiser_bpf(7000,8000,14000,15000,50,48000)
b_r_bp = fir_d.fir_remez_bpf(7000,8000,14000,15000,0.2,50,48000)
```

```python
fir_d.freqz_resp_list([b_k_bp,b_r_bp],[[1],[1]],'dB',fs=48)
ylim([-80,5])
title(r'Kaiser vs Equal Ripple Bandpass')
ylabel(r'Filter Gain (dB)')
xlabel(r'Frequency in kHz')
legend((r'Kaiser: %d taps' % len(b_k_bp),
        r'Remez: %d taps' % len(b_r_bp)),
       loc='lower right')
grid();
```

## A Design Example Useful for Interpolation or Decimation
Here we consider a lowpass design that needs to pass frequencies from [0, 4000] Hz with a sampling rate of 96000 Hz. This scenario arises when building an interpolator using the classes of the `scikit-dps-comm` module `multirate_helper.py` to increase the sampling rate from 8000 Hz to 96000 Hz, or an interpolation factor of $L = 12$. Note at the top of this notebook we have also have the import

```python
import sk_dsp_comm.multirate_helper as mrh
```

so that some of the functionality can be accessed. For more details on the use of `multirate_helper` [see](https://mwickert.github.io/scikit-dsp-comm/example_notebooks/multirate_helper/Multirate_Processing.html).

Start with an equalripple design having transition band centered on 4000 Hz with passband ripple of 0.5 dB and stopband attenuation of 60 dB.

```python
b_up = fir_d.fir_remez_lpf(3300,4300,0.5,60,96000)
```

```python
mr_up = mrh.multirate_FIR(b_up)
```

* Consider the pole-zero configuration for this high-order filter

```python
# Take a look at the pole-zero configuration of this very
# high-order (many taps) linear phase FIR
mr_up.zplane()
```

* Check out the passband and stopband gains

```python
# Verify the passband and stopband gains are as expected
mr_up.freq_resp('db',96000)
```

* See that the group delay is the expected value of $(N_\text{taps} - 1)/2 = 98$ samples

```python
(len(b_up-1))/2
```

```python
# Verify that the FIR design has constant group delay (N_taps - 1)/2 samples
mr_up.freq_resp('groupdelay_s',96000,[0,100])
```

The object `mr_up` can now be used for interpolation or decimation with a rate change factor of 12.

# Traditional IIR Filter Design using the Bilinear Transform
The scipy.signal package fully supports the design of IIR digital filters from analog prototypes.  IIR filters like FIR filters, are typically designed with amplitude response requirements in mind. A collection of design functions are available directly from `scipy.signal` for this purpose, in particular the function `scipy.signal.iirdesign()`. To make the design of lowpass, highpass, bandpass, and bandstop filters consistent with the module `fir_design_helper.py` the module `iir_design_helper.py` was written. Figure 2, below, details how the amplitude response parameters are defined graphically.

```python
Image('300ppi/IIR_Lowpass_Highpass_Bandpass_Bandstop@300ppi.png',width='90%')
```

Within `iir_design_helper.py` there are four filter design functions and a collection of supporting functions available. The four filter design functions are used for designing lowpass, highpass, bandpass, and bandstop filters, utilizing Butterworth, Chebshev type 1, Chebyshev type 2, and elliptical filter prototypes. See [Oppenheim2010](https://www.amazon.com/Discrete-Time-Signal-Processing-3rd-Prentice-Hall/dp/0131988425/ref=sr_1_1?ie=UTF8&qid=1519940790&sr=8-1&keywords=oppenheim+discrete+time+signal+processing&dpID=51v48p99JjL&preST=_SX218_BO1,204,203,200_QL40_&dpSrc=srch) and [ECE 5650 notes Chapter 9](http://www.eas.uccs.edu/~mwickert/ece5650/notes/N5650_9.pdf) for detailed design information. The function interfaces are described in Table 2.

```python
Image('300ppi/IIR_Table@300ppi.png',width='80%')
```

The filter functions return the filter coefficients in two formats:

1. Traditional transfer function form as numerator coefficients `b` and denominator `a` coefficients arrays, and
2. Cascade of biquadratic sections form using the previously introduced sos 2D array or matrix.

Both are provided to allow further analysis with either a direct form topology or the sos form. The underlying `signal.iirdesign()` function also provides a third option: a list of poles and zeros. The `sos` form desireable for high precision filters, as it is more robust to coefficient quantization, in spite using double precision coefficients in the `b` and `a` arrays.

Of the remaining support functions four are also described in Table 2, above. The most significant functions are `freqz_resp_cas_list`, available for graphically comparing the frequency response over several designs, and `sos_zplane` a function for plotting the pole-zero pattern. Both operate using the `sos` matrix. A transfer function form (`b/a`) for frequency response plotting, `freqz_resp_list`, is also present in the module. This function was first introduced in the FIR design section. The frequency response function plotting offers modes for gain in dB, phase in radians, group delay in samples, and group delay in seconds, all for a given sampling rate in Hz. The pole-zero plotting function locates pole and zeros more accurately than `sk_dsp_commsigsys.zplane`, as the numpy function `roots()` is only solving quadratic polynomials. Also, repeated roots can be displayed as theoretically expected, and also so noted in the graphical display by superscripts next to the pole and zero markers.

## IIR Design Based on the Bilinear Transformation
There are multiple ways of designing IIR filters based on amplitude response requirements. When the desire is to have the filter approximation follow an analog prototype such as Butterworth, Chebychev, etc., is using the bilinear transformation. The function `signal.iirdesign()` described above does exactly this.

In the example below we consider lowpass amplitude response requirements and see how the filter order changes when we choose different analog prototypes.

### Example: Lowpass Design Comparison
The lowpass amplitude response requirements given $f_s = 48$ kHz are:
1. $f_\text{pass} = 5$ kHz
2. $f_\text{stop} = 8$ kHz
3. Passband ripple of 0.5 dB
4. Stopband attenuation of 60 dB

Design four filters to meet the same requirements: `butter`, `cheby1`, ,`cheby2`, and `ellip`:

```python
fs = 48000
f_pass = 5000
f_stop = 8000
b_but,a_but,sos_but = iir_d.IIR_lpf(f_pass,f_stop,0.5,60,fs,'butter')
b_cheb1,a_cheb1,sos_cheb1 = iir_d.IIR_lpf(f_pass,f_stop,0.5,60,fs,'cheby1')
b_cheb2,a_cheb2,sos_cheb2 = iir_d.IIR_lpf(f_pass,f_stop,0.5,60,fs,'cheby2')
b_elli,a_elli,sos_elli = iir_d.IIR_lpf(f_pass,f_stop,0.5,60,fs,'ellip')
```

#### Frequency Response Comparison
Here we compare the magnitude response in dB using the `sos` form of each filter as the input. The elliptic is the most efficient, and actually over achieves by reaching the stopband requirement at less than 8 kHz.

```python
iir_d.freqz_resp_cas_list([sos_but,sos_cheb1,sos_cheb2,sos_elli],'dB',fs=48)
ylim([-80,5])
title(r'IIR Lowpass Compare')
ylabel(r'Filter Gain (dB)')
xlabel(r'Frequency in kHz')
legend((r'Butter order: %d' % (len(a_but)-1),
        r'Cheby1 order: %d' % (len(a_cheb1)-1),
        r'Cheby2 order: %d' % (len(a_cheb2)-1),
        r'Elliptic order: %d' % (len(a_elli)-1)),loc='best')
grid();
```

Next plot the pole-zero configuration of just the butterworth design. Here we use the a special version of `ss.zplane` that works with the `sos` 2D array.

```python
iir_d.sos_zplane(sos_but)
```

Note the two plots above can also be obtained using the transfer function form via `iir_d.freqz_resp_list([b],[a],'dB',fs=48)` and `ss.zplane(b,a)`, respectively. The `sos` form will yield more accurate results, as it is less sensitive to coefficient quantization. This is particularly true for the pole-zero plot, as rooting a 15th degree polynomial is far more subject to errors than rooting a simple quadratic.

For the 15th-order Butterworth the bilinear transformation maps the expected 15 s-domain zeros at infinity to $z=-1$. If you use `sk_dsp_comm.sigsys.zplane()` you will find that the 15 zeros at are in a tight circle around $z=-1$, indicating polynomial rooting errors. Likewise the frequency response will be more accurate.

Signal filtering of ndarray `x` is done using the filter designs is done using functions from `scipy.signal`:

1. For transfer function form `y = signal.lfilter(b,a,x)`
2. For sos form `y = signal.sosfilt(sos,x)`

## A Half-Band Filter Design to Pass up to $W/2$ when $f_s = 8$ kHz
Here we consider a lowpass design that needs to pass frequencies up to $f_s/4$. Specifically when $f_s = 8000$ Hz, the filter passband becomes [0, 2000] Hz. Once the coefficients are found a `mrh.multirate` object is created to allow further study of the filter, and ultimately implement filtering of a white noise signal.

Start with an elliptical design having transition band centered on 2000 Hz with passband ripple of 0.5 dB and stopband attenuation of 80 dB. The transition bandwidth is set to 100 Hz, with 50 Hz on either side of 2000 Hz.

```python
# Elliptic IIR Lowpass
b_lp,a_lp,sos_lp = iir_d.IIR_lpf(1950,2050,0.5,80,8000.,'ellip')
mr_lp = mrh.multirate_IIR(sos_lp)
```

```python
mr_lp.freq_resp('db',8000)
```

Pass Gaussian white noise of variance $\sigma_x^2 = 1$ through the filter. Use a lot of samples so the spectral estimate can accurately form $S_y(f) = \sigma_x^2\cdot |H(e^{j2\pi f/f_s})|^2 = |H(e^{j2\pi f/f_s})|^2$.

```python
x = randn(1000000)
y = mr_lp.filter(x)
psd(x,2**10,8000);
psd(y,2**10,8000);
title(r'Filtering White Noise Having $\sigma_x^2 = 1$')
legend(('Input PSD','Output PSD'),loc='best')
ylim([-130,-30])
```

```python
fs = 8000
print('Expected PSD of %2.3f dB/Hz' % (0-10*log10(fs),))
```

## Amplitude Response Bandpass Design
Here we consider FIR and IIR bandpass designs for use in an SSB demodulator to remove potential adjacent channel signals sitting either side of a frequency band running from 23 kHz to 24 kHz.

```python
b_rec_bpf1 = fir_d.fir_remez_bpf(23000,24000,28000,29000,0.5,70,96000,8)
fir_d.freqz_resp_list([b_rec_bpf1],[1],mode='dB',fs=96000)
ylim([-80, 5])
grid();
```

The group delay is flat (constant) by virture of the design having linear phase.

```python
b_rec_bpf1 = fir_d.fir_remez_bpf(23000,24000,28000,29000,0.5,70,96000,8)
fir_d.freqz_resp_list([b_rec_bpf1],[1],mode='groupdelay_s',fs=96000)
grid();
```

Compare the FIR design with an elliptical design:

```python
b_rec_bpf2,a_rec_bpf2,sos_rec_bpf2 = iir_d.IIR_bpf(23000,24000,28000,29000,
                                                   0.5,70,96000,'ellip')
with np.errstate(divide='ignore'):
    iir_d.freqz_resp_cas_list([sos_rec_bpf2],mode='dB',fs=96000)
ylim([-80, 5])
grid();
```

This high order elliptic has a nice tight amplitude response for minimal coefficients, but the group delay is terrible:

```python
with np.errstate(divide='ignore', invalid='ignore'): #manage singularity warnings
    iir_d.freqz_resp_cas_list([sos_rec_bpf2],mode='groupdelay_s',fs=96000)
#ylim([-80, 5])
grid();
```


# Bayesian Logistic Regression

Contact: mihaelacr@google.com

Aim of this colab:
Logistic regression with a twist! We will use variational inference to learn a distribution over the learned parameters.

Througout this work we will be focusing on a binary classification task, on
a small dataset (the UCD breast cancer dataset).

## Logistic regression

Learn the weights which best classify the $(x, y)$ pairs in the dataset.
The weights ($w$) and biases $b$, form the parameters which are learned via the following maximization problem:

\begin{equation}
\mathbb{E}_{p^*(x, y)} \log p(y|x, \theta) = \\
 \mathbb{E}_{p^*(x, y)} \left[y \log \sigma(w x + b) + (1 - y) \log(1- \sigma(w x + b)\right]
\end{equation}

Here $\sigma$ denotes the sigmoid function.

## Bayesian logistic regression - computing the evidence lower bound

Learn a distribution over the weights which best classify the $(x, y)$ pairs in the dataset.

We now want to learn the distributional parameters of $q(w)$ in order to maximize:

\begin{equation}
\mathbb{E}_{p^*(x, y)} \log p_w(y| x) = \\
\mathbb{E}_{p^*(x, y)} \log \int p(y|x, w) p(w) \delta w = \\
\mathbb{E}_{p^*(x, y)} \log \int p(y|x, w) p(w) \frac{q(w)}{q(w)} \delta w \ge \\
\mathbb{E}_{p^*(x, y)} \mathbb{E}_{q(w)} \log \left[p(y| x, w) \frac{p(w)}{q(w)} \right]= \\
\mathbb{E}_{q(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)  -  KL(q(w)||p(w)) = \\
 \mathbb{E}_{q(w)}\mathbb{E}_{p^*(x, y)} \left[y \log \sigma(w x + b) + (1 - y) \log(1- \sigma(w x + b)\right] -  KL(q(w)||p(w))
\end{equation}

In Bayesian Logistic Regression, we thus learn a distribution over parameters
$w$ which can explain the data, while staying close to a chosen prior $p(w)$.

Throughout this lab, we will use Gaussian Distributions for $p(w)$ and $q(w)$.

For more details, see
[this paper](https://pdfs.semanticscholar.org/e407/ea7fda6d152d2186f4b5e27aa04ec2d32dcd.pdf)
or this tutorial
[https://www.ece.rice.edu/~vc3/elec633/logistic.pdf].

### Gradient estimation

In Bayesian Logistic Regression, we aim to learn the parameters of the distribution $q(w)$. We will call these parameters $\theta$. In the case of a
Gaussian distribution, these will be the mean and covariance matrix of a
multivariate distribution. Since we will use stochastic gradient descent to learn $\theta$, we need to be able to compute the gradients with respect to
$\theta$ of our objective. Since our objective contains an expectation with respect to $q(w)$, this can be challenging. To illustrate this, from now on we will denote $q(w)$ as $q_{\theta}(w)$.

Specifically, we are interested in:
\begin{equation}
  \nabla_{\theta} \left[\mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)  -  KL(q_{\theta}(w)||p(w))\right]
\end{equation}

Since in we will be using Gaussian distributions for $q_{\theta}(w)$ and $p(w)$
$KL(q_{\theta}(w)||p(w))$ can be computed in closed form and we can rely on
TensorFlow's automatic differentiation to compute
$\nabla_{\theta} KL(q_{\theta}(w)||p(w))$.

Let's now turn attention to computing $\nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)$. Unlike the KL, the integral
$\mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y|x, w)$
is not tractable, and thus cannot be computed in closed form. Thus, we are interested in other approaches to compute the gradient.

### Reinforce gradient estimation

A first approach to compute an estimate of the gradient is to use the
REINFORCE gradient estimator:

\begin{equation}
  \nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  = \\
  \nabla_{\theta} \int q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \delta w = \\
  \int \nabla_{\theta} \left[q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right] \delta w = \\
 \int \nabla_{\theta} \left[q_{\theta}(w)\right] \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \delta w = \\
 \int q_{\theta}(w) \nabla_{\theta} \left[\log q_{\theta}(w)\right] \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  \delta w = \\
 \mathbb{E}_{q_{\theta}(w)} \left[ \nabla_{\theta} \log q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right]
\end{equation}

We can use samples from $q_{\theta}(w)$ to compute the Monte Carlo estimate of
the last integral, to get an unbiased estimator of the true gradient. The more
samples we use to estimate the integral, the more accurate the gradient estimate
will be.

### Pathwise gradient estimation (reparametrization)

In the REINFORCE estimator, we did not use any knowledge of the variational
distribution $q_{\theta}(w)$.
For a Gaussian distribution, we can use the reparametrization trick to obtain
an unbiased gradient estimator of  $\nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)$.

To do so, we will use the fact that
\begin{equation}
    z \sim N(\mu, \sigma), z = \mu + \epsilon \sigma, {\text {with }} \epsilon \sim N(0, 1)
\end{equation}

Namely:
\begin{equation}
  \nabla_{\theta} \mathbb{E}_{q_{\theta}(w)} \mathbb{E}_{p^*(x, y)} \log p(y| x, w)  = \\
  \nabla_{\theta} \mathbb{E}_{p(\epsilon)} \mathbb{E}_{p^*(x, y)} \log p(y| x, \mu + \epsilon \sigma)  = \\
  \mathbb{E}_{p(\epsilon)} \nabla_{\theta} \mathbb{E}_{p^*(x, y)} \log p(y| x, \mu + \epsilon \sigma)
\end{equation}

Note that we were able to move the gradient inside the integral since
$p(\epsilon)$ does not depend on $\theta$.
To estimate the last integral, we can use a Monte Carlo estimate using samples
of $p(\epsilon)$.

## Your tasks

*   Define the Gaussian posterior distribution.
*   Fill in the code for the ELBO
*   Fill in the reinforce function to create the surrogate loss
*   Fill in the loss function for reparametrization
*   Visualize the effects of batch size, learning rates and number of posterior samples on the learned model, as well as gradient variance.

```
import enum
import numpy as np

import math
from sklearn import datasets
from sklearn import linear_model
import collections

from matplotlib import pyplot as plt
import tensorflow as tf
import tensorflow_probability as tfp
import seaborn as sns

tfd = tfp.distributions

#ELBO je evidence lower bound.

```

```
sns.set(rc={"lines.linewidth": 2.8}, font_scale=2)
sns.set_style("whitegrid")
```

## Get the dataset and visualize it

```
def get_data(normalize = True):
    data = datasets.load_breast_cancer()

    print(data.target_names)
    print(data.feature_names)

    features = np.array(data.data, dtype=np.float32)
    targets = np.array(data.target, dtype=np.float32)

    if normalize:
        # Note: the data dimensions have very different scales.
        # We normalize by the mean and scale of the entire dataset.
        features = features - features.mean(axis=0)
        features = features / features.std(axis=0)

    # Add a dimension of 1 (bias).
    features = np.concatenate((features, np.ones((features.shape[0], 1))), axis=1)
    features = np.array(features, dtype=np.float32)

    return features, targets, data.feature_names
```

### Dataset

This datasets has meaningful features: we are trying to classify based on these features whether someone has breast cancer or not. We can look at the
features we are using to classify the data below.

```
features, targets, feature_names = get_data()
```

    ['malignant' 'benign']
    ['mean radius' 'mean texture' 'mean perimeter' 'mean area'
     'mean smoothness' 'mean compactness' 'mean concavity'
     'mean concave points' 'mean symmetry' 'mean fractal dimension'
     'radius error' 'texture error' 'perimeter error' 'area error'
     'smoothness error' 'compactness error' 'concavity error'
     'concave points error' 'symmetry error' 'fractal dimension error'
     'worst radius' 'worst texture' 'worst perimeter' 'worst area'
     'worst smoothness' 'worst compactness' 'worst concavity'
     'worst concave points' 'worst symmetry' 'worst fractal dimension']

### Trivial baseline

How well can we do with a constant classifier?

If we look at the average label, we can see how many 1s are in the data, so we know that our classifier has to get more than that accuracy

```
np.mean(targets)
```

    0.6274165

```
features.shape
```

    (569, 31)

```
dataset_size = features.shape[0]
data_dims = features.shape[-1]
```

## Logistic regression baseline - how well can we do on this dataset?

Let's look at a simple, non Bayesian classification approach to get an idea of how well we can do on the dataset.

```
lr = linear_model.LogisticRegression()
```

```
lr.fit(features, targets)
lr.score(features, targets)
```

    /usr/local/lib/python2.7/dist-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)

    0.9876977152899824

## Utilities

```
def get_shape_list(tensor):
  return tensor.shape.as_list()
```

```
def get_sklearn_data_as_tensors(
      features, targets, batch_size, dataset_name='breast_cancer'):
  """Read sklearn datasets as tf.Tensors.

  Args:
    batch_size: Integer or None. If None, the entire dataset is used.
    dataset_name: A string, the name of the dataset.

  Returns:
    A tuple of size two containing two tensors of rank 2 `[B, F]`, the data
    features and targets.
  """
  dataset = tf.data.Dataset.from_tensor_slices((features, targets))
  if batch_size:
    # Shuffle, repeat, and batch the examples.
    batched_dataset = dataset.shuffle(1000).repeat().batch(batch_size)
  else:
    batch_size = features.shape[0]
    batched_dataset = dataset.repeat().batch(batch_size)

  iterator = batched_dataset.make_one_shot_iterator()
  batch_features, batch_targets = iterator.get_next()

  data_dim = features.shape[1]
  batch_features.set_shape([batch_size, data_dim])
  batch_targets.set_shape([batch_size])

  return batch_features, batch_targets
```

## Bayesian Logistic Regression - define the prior and the posterior

We will learn a distribution over parameters, so we will first define prior distribution, and then a posterior distribution over parameters which we
will learn.

Both of these distibutions will be Gaussian.

## Task: implement the posterior distribution

You have to implement two functions

1.   multi_normal, which takes as arguments the mean and the log scale of the normal distribution, and returns a tensorflow distribution object. Have a look at the TensorFlow distibutions package.
2.   diagonal_gaussian_posterior which returns a tuple of two elements, one being the posterior distribution (a call to multi_normal) and the learned variables. Your task here is to define the variables we will learn. The argument `data_dims` tells you the size of the learned parameters, and thus the size of the variables of the distribution).

```
def multi_normal(loc, log_scale):
  scale = tf.exp(log_scale)
  return tfd.MultivariateNormalDiag(loc=loc, scale_diag=scale)

def diagonal_gaussian_posterior(data_dims, log_scale_init=0.0):
  mean = tf.Variable(tf.zeros(data_dims, dtype=tf.float32), name="mean")
  log_scale = tf.Variable(log_scale_init*tf.ones(data_dims, dtype=tf.float32), name="log_scale")
  learned_vars = [mean, log_scale]
  return multi_normal(loc=mean, log_scale=log_scale), learned_vars
```

## Bayesian Logistic Regression - the model

### Hypers

```
BATCH_SIZE = 500
NUM_POSTERIOR_SAMPLES = 100
```

```
prior = multi_normal(loc=tf.zeros(data_dims), log_scale=tf.zeros(data_dims))
posterior, learned_vars  = diagonal_gaussian_posterior(data_dims)
```

```
def _predictions(logits):
  return tf.to_float(logits >= 0)

def _accuracy(targets, predictions):
  # `predictions` have rank 2: batch_size, num_posterior samples.
  # We expand dims of the targets to compute the accuracy per sample.
  targets = tf.expand_dims(targets, axis=1)
  return tf.reduce_mean(tf.to_float(tf.equal(targets, predictions)))

def linear_model(data, targets, posterior_samples):
  num_posterior_samples = tf.shape(posterior_samples)[0]
  logits = tf.matmul(data, posterior_samples, transpose_b=True)
  # Make targets [B, 1] to use broadcasting.
  targets = tf.expand_dims(targets, axis=1)
  targets = targets * tf.ones([1, num_posterior_samples])

  log_probs = - tf.nn.sigmoid_cross_entropy_with_logits(
      labels=targets, logits=logits)
  return log_probs, logits
```

## Task: Run the model and return the components of the elbo

```
batch_features, batch_targets = \
    get_sklearn_data_as_tensors(features, targets, BATCH_SIZE)

def run_model(posterior, num_samples):
  # Sample from the posterior to obtain a set of parameters used for
  # prediction. Use `num_samples` for the number of samples.
  posterior_samples = posterior.sample(sample_shape=num_samples)

  # Compute the log probs of the data under all the models we have sampled.
  # These tensors are [num_samples, B]. Use the features and targets tensors.
  log_probs, logits = linear_model(batch_features, batch_targets, posterior_samples)

  # Compute the KL between the posterior and the prior.
  # Note: since we use Gaussian distributions, the closed form of the
  # KL can be computed. Remember that distributions are objects in TensorFlow!
  kl = tfd.kl_divergence(prior, posterior, name="KLdivergence")

  # Sum over data. Normalize by dataset size to ensure that the gradients
  # are not very different in magnitude when we change batch size.
  param_log_probs = tf.reduce_mean(log_probs, axis=0) * dataset_size
  #zakaj to zgoraj nardimo: ker delamo v batching approximatamo logprob celega dataseta z tistim od batcha: logP(y|x,w) = sum_{i=1-n} logP(y_i|x_i,w) =approx= N* 1/B * sum_{i=1-B}logP(y_i|x_i,w)
  #in to zato, ker bir adi delali stochastic grad descent.
  #sometimes grad on whole dataset might be very flat, and the noisy approx of it by smaller subsets might be useful.
  param_log_probs.shape.assert_is_compatible_with([num_samples])

  return param_log_probs, kl, posterior_samples, logits
```

```
param_log_probs, kl, posterior_samples, logits = run_model(
      posterior, NUM_POSTERIOR_SAMPLES)
predictions = _predictions(logits)
accuracy = _accuracy(targets=batch_targets, predictions=predictions)
accuracy = tf.reduce_mean(accuracy)

```

## Task: compute the elbo from the log probs and the KL

`per_sample_elbo` should have shape [num_samples], since we obtain a model prediction for each of the sampled parameters.

```
print(kl.shape, param_log_probs.shape)
```

    (TensorShape([]), TensorShape([Dimension(100)]))

```
per_sample_elbo = param_log_probs - kl
elbo = tf.reduce_mean(per_sample_elbo)
```

## Training loop

## Training the model via reparametrization

```
# Stochastic loss depends on the output of a sampling operation  (posterior samples)
# but tensorflow implements reparametrization by default.
per_sample_reparametrization_loss = - per_sample_elbo
reparametrization_loss = tf.reduce_mean(per_sample_reparametrization_loss)
```

```
optimizer = tf.train.GradientDescentOptimizer(0.0001)
reparam_min_op = optimizer.minimize(reparametrization_loss)
```

```
NUM_ITERATIONS = 1000
```

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

reparam_accuracies = []
reparam_kls = []
reparam_elbos = []

for i in xrange(NUM_ITERATIONS):
  sess.run(reparam_min_op)

  if i % 10 == 0:
    reparam_acc, reparam_kl, reparam_elbo = sess.run([accuracy, kl, elbo])
    reparam_accuracies += [reparam_acc]
    reparam_kls += [reparam_kl]
    reparam_elbos += [reparam_elbo]

    print('Iteration {}. Elbo {}. KL {}'.format(
        i, reparam_elbo, reparam_kl))

reparam_learned_mean, reparam_learned_log_scale = sess.run(learned_vars)
```

    W0706 15:00:38.218702 139815012120448 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py:193: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
    Instructions for updating:
    Use `tf.global_variables_initializer` instead.

    Iteration 0. Elbo -1043.81713867. KL 0.00327511876822
    Iteration 10. Elbo -636.395141602. KL 0.276360750198
    Iteration 20. Elbo -480.654663086. KL 0.756315946579
    Iteration 30. Elbo -378.839416504. KL 1.23966491222
    Iteration 40. Elbo -350.873504639. KL 1.70637965202
    Iteration 50. Elbo -315.716400146. KL 2.14785432816
    Iteration 60. Elbo -265.325714111. KL 2.54041051865
    Iteration 70. Elbo -275.458282471. KL 2.90840959549
    Iteration 80. Elbo -205.419281006. KL 3.23674511909
    Iteration 90. Elbo -227.61517334. KL 3.58280205727
    Iteration 100. Elbo -195.08934021. KL 3.8931210041
    Iteration 110. Elbo -176.498092651. KL 4.18404865265
    Iteration 120. Elbo -178.006072998. KL 4.45383262634
    Iteration 130. Elbo -192.494628906. KL 4.71894168854
    Iteration 140. Elbo -154.041351318. KL 4.96762561798
    Iteration 150. Elbo -165.845001221. KL 5.2126750946
    Iteration 160. Elbo -164.833831787. KL 5.4498500824
    Iteration 170. Elbo -162.906936646. KL 5.66854524612
    Iteration 180. Elbo -150.8931427. KL 5.88886022568
    Iteration 190. Elbo -146.579101562. KL 6.11492061615
    Iteration 200. Elbo -134.531723022. KL 6.32833957672
    Iteration 210. Elbo -142.4012146. KL 6.50668001175
    Iteration 220. Elbo -130.055465698. KL 6.69718456268
    Iteration 230. Elbo -132.092025757. KL 6.8787689209
    Iteration 240. Elbo -123.959190369. KL 7.05900526047
    Iteration 250. Elbo -130.256790161. KL 7.23445558548
    Iteration 260. Elbo -109.56186676. KL 7.4054145813
    Iteration 270. Elbo -133.149291992. KL 7.58307218552
    Iteration 280. Elbo -129.128753662. KL 7.75109529495
    Iteration 290. Elbo -124.381347656. KL 7.91038703918
    Iteration 300. Elbo -122.278816223. KL 8.06728363037
    Iteration 310. Elbo -145.384689331. KL 8.22702217102
    Iteration 320. Elbo -115.329093933. KL 8.37921142578
    Iteration 330. Elbo -121.612678528. KL 8.52447891235
    Iteration 340. Elbo -112.941345215. KL 8.6596364975
    Iteration 350. Elbo -121.284416199. KL 8.80675888062
    Iteration 360. Elbo -104.771392822. KL 8.94297981262
    Iteration 370. Elbo -114.515838623. KL 9.08836460114
    Iteration 380. Elbo -100.544509888. KL 9.2234287262
    Iteration 390. Elbo -127.147888184. KL 9.35147571564
    Iteration 400. Elbo -88.1568756104. KL 9.48739433289
    Iteration 410. Elbo -108.84072113. KL 9.6213684082
    Iteration 420. Elbo -95.7826919556. KL 9.75115585327
    Iteration 430. Elbo -80.1415634155. KL 9.87467765808
    Iteration 440. Elbo -99.7651138306. KL 10.0027179718
    Iteration 450. Elbo -107.753326416. KL 10.1212205887
    Iteration 460. Elbo -104.782920837. KL 10.244600296
    Iteration 470. Elbo -100.614685059. KL 10.3477230072
    Iteration 480. Elbo -93.695098877. KL 10.4596738815
    Iteration 490. Elbo -94.9154510498. KL 10.5703468323
    Iteration 500. Elbo -103.088905334. KL 10.6872062683
    Iteration 510. Elbo -84.6145935059. KL 10.7939090729
    Iteration 520. Elbo -118.238868713. KL 10.9046058655
    Iteration 530. Elbo -94.8312988281. KL 11.0128097534
    Iteration 540. Elbo -95.7698059082. KL 11.1152467728
    Iteration 550. Elbo -84.7660751343. KL 11.2125892639
    Iteration 560. Elbo -94.2164001465. KL 11.3166236877
    Iteration 570. Elbo -95.7602539062. KL 11.4207544327
    Iteration 580. Elbo -96.0360641479. KL 11.5252332687
    Iteration 590. Elbo -96.5693969727. KL 11.6238174438
    Iteration 600. Elbo -98.0314331055. KL 11.711101532
    Iteration 610. Elbo -103.387557983. KL 11.8092775345
    Iteration 620. Elbo -91.0787582397. KL 11.9060516357
    Iteration 630. Elbo -94.8880157471. KL 11.9881744385
    Iteration 640. Elbo -105.964889526. KL 12.0876579285
    Iteration 650. Elbo -94.4729232788. KL 12.1770954132
    Iteration 660. Elbo -94.2794113159. KL 12.2682323456
    Iteration 670. Elbo -93.1441497803. KL 12.3540916443
    Iteration 680. Elbo -82.5736541748. KL 12.4341335297
    Iteration 690. Elbo -77.8994979858. KL 12.5235757828
    Iteration 700. Elbo -72.5232925415. KL 12.5994262695
    Iteration 710. Elbo -81.4371795654. KL 12.6841011047
    Iteration 720. Elbo -84.8502426147. KL 12.7661657333
    Iteration 730. Elbo -67.3670578003. KL 12.8423204422
    Iteration 740. Elbo -86.6210479736. KL 12.9235944748
    Iteration 750. Elbo -72.2513504028. KL 13.0030422211
    Iteration 760. Elbo -90.5619506836. KL 13.0805072784
    Iteration 770. Elbo -91.4573898315. KL 13.1528501511
    Iteration 780. Elbo -81.9247436523. KL 13.2243204117
    Iteration 790. Elbo -93.2465820312. KL 13.298623085
    Iteration 800. Elbo -85.8871765137. KL 13.3715810776
    Iteration 810. Elbo -84.9741287231. KL 13.4495382309
    Iteration 820. Elbo -91.8676376343. KL 13.5227899551
    Iteration 830. Elbo -86.3598022461. KL 13.5878419876
    Iteration 840. Elbo -79.7975463867. KL 13.6533432007
    Iteration 850. Elbo -73.0026092529. KL 13.7221403122
    Iteration 860. Elbo -83.836807251. KL 13.7898302078
    Iteration 870. Elbo -79.073638916. KL 13.8596811295
    Iteration 880. Elbo -87.2807617188. KL 13.9240674973
    Iteration 890. Elbo -80.2492904663. KL 13.9910812378
    Iteration 900. Elbo -97.0606918335. KL 14.0603475571
    Iteration 910. Elbo -84.9369049072. KL 14.1285400391
    Iteration 920. Elbo -85.4461517334. KL 14.189781189
    Iteration 930. Elbo -89.4969406128. KL 14.2507276535
    Iteration 940. Elbo -89.9543380737. KL 14.3136911392
    Iteration 950. Elbo -96.215133667. KL 14.3790845871
    Iteration 960. Elbo -95.7035369873. KL 14.4458646774
    Iteration 970. Elbo -75.3545150757. KL 14.5069952011
    Iteration 980. Elbo -72.4596176147. KL 14.5681419373
    Iteration 990. Elbo -70.4646377563. KL 14.6268873215

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(reparam_elbos, label='ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].set_ylim((-500, -50))
axes[0].legend()

axes[1].plot(reparam_kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(reparam_accuracies, label='Accuracy')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

## Training the model via reinforce

## Task: define the surrogate reinforce loss for the log prob term.

Note: we do not need to use reinforce the KL term, since we can compute the KL analytically for the two Gaussians, and can use standard backprop.

For the reinforce implementation, instead of changing the gradients, we will change the loss such that TensorFlow's automatic differentiation does the right thing. This is called a surrogate loss.

The gradient we want to obtain is the following (since we want to minimize the loss, we add a minus):
\begin{equation}
- \mathbb{E}_{q_{\theta}(w)} \left[ \nabla_{\theta} \log q_{\theta}(w) \mathbb{E}_{p^*(x, y)} \log p(y| x, w) \right]
\end{equation}

so we need to construct a loss which has this gradient, by ensuring that the gradients will flow through the $\log q_{\theta}(w)$ term, but not the rest.

```
# Since TensorFlow does automatic differentation, we have to use
# tf.stop_gradient to ensure the gradient !_only_flows_! to
# the log_{q_\theta}(w) term.
per_sample_reinforce_loss = -tf.stop_gradient(param_log_probs)*posterior.log_prob(tf.stop_gradient(posterior_samples))  # Shape [num_samples, batch size].
# Reduce now over the number of samples.
reinforce_loss = tf.reduce_mean(per_sample_reinforce_loss) + kl #tale vrstica ti naredi tisti zunanji expectation. tanotranji expectation je pa narejen ze nekje v eni od zgornjih celic (tam kjer sumamo/averageamo po batchu).

```

```
optimizer = tf.train.GradientDescentOptimizer(0.0001)
reinforce_min_op = optimizer.minimize(reinforce_loss)
```

```
NUM_ITERATIONS = 1000
```

```
reinforce_accuracies = []
reinforce_kls = []
reinforce_elbos = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reinforce_min_op)

  if i % 10 == 0:
    reinforce_acc, reinforce_kl, reinforce_elbo = sess.run([accuracy, kl, elbo])
    reinforce_accuracies += [reinforce_acc]
    reinforce_kls += [reinforce_kl]
    reinforce_elbos += [reinforce_elbo]

    print('Iteration {}. Elbo {}. KL {}'.format(
        i, reinforce_elbo, reinforce_kl))

reinforce_learned_mean, reinforce_learned_log_scale = sess.run(learned_vars)
```

    Iteration 0. Elbo -1264.86010742. KL 0.0169694069773
    Iteration 10. Elbo -689.8828125. KL 0.406080245972
    Iteration 20. Elbo -497.553436279. KL 0.891268253326
    Iteration 30. Elbo -419.460662842. KL 1.3592966795
    Iteration 40. Elbo -399.536804199. KL 1.88225066662
    Iteration 50. Elbo -313.028900146. KL 2.30571317673
    Iteration 60. Elbo -288.394927979. KL 2.67438960075
    Iteration 70. Elbo -232.134628296. KL 3.00853157043
    Iteration 80. Elbo -217.006561279. KL 3.31562328339
    Iteration 90. Elbo -203.153686523. KL 3.64438796043
    Iteration 100. Elbo -190.448013306. KL 3.92471075058
    Iteration 110. Elbo -210.835876465. KL 4.15419006348
    Iteration 120. Elbo -203.819213867. KL 4.41300058365
    Iteration 130. Elbo -192.832855225. KL 4.6386961937
    Iteration 140. Elbo -175.646362305. KL 4.87356853485
    Iteration 150. Elbo -191.283538818. KL 5.10979557037
    Iteration 160. Elbo -180.681716919. KL 5.35692310333
    Iteration 170. Elbo -145.590332031. KL 5.62900781631
    Iteration 180. Elbo -165.358474731. KL 5.88225269318
    Iteration 190. Elbo -134.004852295. KL 6.08527183533
    Iteration 200. Elbo -147.557113647. KL 6.26247024536
    Iteration 210. Elbo -152.138473511. KL 6.40464687347
    Iteration 220. Elbo -146.272384644. KL 6.64486455917
    Iteration 230. Elbo -141.719284058. KL 6.89347982407
    Iteration 240. Elbo -134.607955933. KL 7.07207489014
    Iteration 250. Elbo -131.410079956. KL 7.26515483856
    Iteration 260. Elbo -127.269844055. KL 7.43008708954
    Iteration 270. Elbo -115.788856506. KL 7.64360189438
    Iteration 280. Elbo -122.976036072. KL 7.86154270172
    Iteration 290. Elbo -126.953010559. KL 8.0165348053
    Iteration 300. Elbo -112.014434814. KL 8.15097427368
    Iteration 310. Elbo -124.555198669. KL 8.36483860016
    Iteration 320. Elbo -134.718917847. KL 8.50629997253
    Iteration 330. Elbo -112.258644104. KL 8.66214179993
    Iteration 340. Elbo -112.644035339. KL 8.81447315216
    Iteration 350. Elbo -112.629013062. KL 8.97210788727
    Iteration 360. Elbo -111.54875946. KL 9.15925598145
    Iteration 370. Elbo -109.072189331. KL 9.2902545929
    Iteration 380. Elbo -111.574645996. KL 9.41396045685
    Iteration 390. Elbo -108.640769958. KL 9.57712364197
    Iteration 400. Elbo -113.159553528. KL 9.76281833649
    Iteration 410. Elbo -96.0975875854. KL 9.83867263794
    Iteration 420. Elbo -97.4447097778. KL 9.92580127716
    Iteration 430. Elbo -107.146316528. KL 10.0585975647
    Iteration 440. Elbo -111.243103027. KL 10.1193380356
    Iteration 450. Elbo -109.670402527. KL 10.2024822235
    Iteration 460. Elbo -111.255569458. KL 10.287563324
    Iteration 470. Elbo -83.2108078003. KL 10.4059066772
    Iteration 480. Elbo -108.126716614. KL 10.4983043671
    Iteration 490. Elbo -93.4847640991. KL 10.5798397064
    Iteration 500. Elbo -102.550849915. KL 10.6927242279
    Iteration 510. Elbo -88.9992599487. KL 10.8164844513
    Iteration 520. Elbo -100.02947998. KL 10.9715051651
    Iteration 530. Elbo -86.6336898804. KL 11.0539741516
    Iteration 540. Elbo -84.3364868164. KL 11.1408901215
    Iteration 550. Elbo -102.549072266. KL 11.2221794128
    Iteration 560. Elbo -97.2721862793. KL 11.3164968491
    Iteration 570. Elbo -99.7806091309. KL 11.3721914291
    Iteration 580. Elbo -103.348457336. KL 11.451256752
    Iteration 590. Elbo -102.566101074. KL 11.5567569733
    Iteration 600. Elbo -99.2347717285. KL 11.6708698273
    Iteration 610. Elbo -87.8675079346. KL 11.7250738144
    Iteration 620. Elbo -108.254707336. KL 11.8579998016
    Iteration 630. Elbo -100.640892029. KL 11.9559202194
    Iteration 640. Elbo -86.2919845581. KL 12.0304584503
    Iteration 650. Elbo -90.7585678101. KL 12.1136741638
    Iteration 660. Elbo -99.7998275757. KL 12.2165565491
    Iteration 670. Elbo -94.993270874. KL 12.3034648895
    Iteration 680. Elbo -91.2248840332. KL 12.3683404922
    Iteration 690. Elbo -84.0728530884. KL 12.4677639008
    Iteration 700. Elbo -107.90927124. KL 12.5038385391
    Iteration 710. Elbo -90.3980789185. KL 12.6159133911
    Iteration 720. Elbo -81.7424240112. KL 12.7185697556
    Iteration 730. Elbo -106.966110229. KL 12.7870378494
    Iteration 740. Elbo -96.5921859741. KL 12.8590450287
    Iteration 750. Elbo -87.1632995605. KL 12.9666252136
    Iteration 760. Elbo -83.6748275757. KL 13.0511817932
    Iteration 770. Elbo -90.7325668335. KL 13.1392688751
    Iteration 780. Elbo -77.8932495117. KL 13.2066020966
    Iteration 790. Elbo -82.2818908691. KL 13.2714490891
    Iteration 800. Elbo -84.981803894. KL 13.3532419205
    Iteration 810. Elbo -94.9546127319. KL 13.3892364502
    Iteration 820. Elbo -102.457023621. KL 13.4336261749
    Iteration 830. Elbo -85.7875061035. KL 13.528175354
    Iteration 840. Elbo -77.1722183228. KL 13.6191015244
    Iteration 850. Elbo -89.8138504028. KL 13.6761331558
    Iteration 860. Elbo -83.654548645. KL 13.6801214218
    Iteration 870. Elbo -85.8260345459. KL 13.7484617233
    Iteration 880. Elbo -69.8591766357. KL 13.8478555679
    Iteration 890. Elbo -91.5780944824. KL 13.8972568512
    Iteration 900. Elbo -77.7632293701. KL 13.910777092
    Iteration 910. Elbo -86.2669143677. KL 13.9568367004
    Iteration 920. Elbo -89.8471374512. KL 14.0222406387
    Iteration 930. Elbo -90.8879776001. KL 14.0767593384
    Iteration 940. Elbo -75.9265213013. KL 14.1284236908
    Iteration 950. Elbo -87.5227966309. KL 14.182513237
    Iteration 960. Elbo -86.584777832. KL 14.2346200943
    Iteration 970. Elbo -80.5763549805. KL 14.2867107391
    Iteration 980. Elbo -79.4926681519. KL 14.3929758072
    Iteration 990. Elbo -75.9349822998. KL 14.4287252426

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(reinforce_elbos, label='ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].set_ylim((-500, -50))
axes[0].legend()

axes[1].plot(reinforce_kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(reinforce_accuracies, label='Accuracy')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

## Task: Analysis

* How sensitive are the different estimators to the number of posterior samples used? Try 1, 10, 100.
* How sensitive are the different estimators to the learning rate used? Try 1e-4, 1e-3 and 1e-2 for both estimators.
* How sensitive are the different estimators to batch size? Try batch size 10 for both estimators.

# Visualize learned parameters

* Let's visualize the learned parameters, as well as check the different uncertanties around different parameters
* Can we now see which parameters are most important in the classification problem?

```
learned_mean = reparam_learned_mean
learned_scale = np.exp(reparam_learned_log_scale)
```

```
PARAM_INDEX = 10
```

```
def gaussian(x, mu, sig):
    return np.exp(-np.power(x - mu, 2.) / (2 * np.power(sig, 2.)))
```

```
mean = learned_mean[PARAM_INDEX]
scale = learned_scale[PARAM_INDEX]
x_values = np.linspace(-3, 3, num=1000)
y_values = [gaussian(x, mean, scale) for x in x_values]

plt.figure(figsize=(7,5))
plt.plot(x_values, y_values)
plt.vlines(
      learned_mean[PARAM_INDEX],
      ymin=0,
      ymax=gaussian(mean, mean, scale),
      colors='g',
      alpha=0.8)

plt.grid('off')
x_std_values = np.linspace(mean - scale, mean + scale, num=1000)
y_std_values = [gaussian(x, mean, scale) for x in x_std_values]
plt.fill_between(x_std_values, y_std_values, 0, alpha=0.3, color='b')

plt.title('Parameter name= {}'.format(feature_names[PARAM_INDEX]))
```

## Visualize gradient variance - which estimator has lower variance?

* Let's assess the gradient variance per of each estimator, by looking at how much gradients vary for each estimators.

```
def per_batch_grads(ys, xs):
  """Computes the gradients of ys[i] wrt xs."""
  grads = []

  for y in tf.unstack(ys, axis=0):
    grads.append(tf.squeeze(tf.gradients(y, xs)[0]))
  return tf.stack(grads, axis=0)
```

```
def compute_grad_var(grads, param_index):
  return np.mean(grads[:, param_index]), np.std(grads[:, param_index])**2
```

```
per_batch_reinforce_grads = per_batch_grads(per_sample_reinforce_loss, learned_vars[0])
per_batch_reinforce_grads.shape
```

    TensorShape([Dimension(100), Dimension(31)])

```
per_batch_reparam_grads = per_batch_grads(per_sample_reparametrization_loss, learned_vars[0])
per_batch_reparam_grads.shape
```

    TensorShape([Dimension(100), Dimension(31)])

```
reinforce_grads = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reinforce_min_op)

  if i % 10 == 0:
    reinforce_grads.append(sess.run(per_batch_reinforce_grads))
```

```
reparam_grads = []

sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())

for i in xrange(NUM_ITERATIONS):
  sess.run(reparam_min_op)

  if i % 10 == 0:
    reparam_grads.append(sess.run(per_batch_reparam_grads))
```

```
PARAM_INDEX = 19
```

```
reinforce_grad_vars = []
for timestep_grad in reinforce_grads:
  _, reinforce_grad_var = compute_grad_var(timestep_grad, PARAM_INDEX)
  reinforce_grad_vars.append(reinforce_grad_var)
```

```
reparam_grad_vars = []
for timestep_grad in reparam_grads:
  _, reparam_grad_var = compute_grad_var(timestep_grad, PARAM_INDEX)
  reparam_grad_vars.append(reparam_grad_var)
```

```
plt.figure(figsize=(7,5))
plt.plot(reinforce_grad_vars, label='reinforce')
plt.plot(reparam_grad_vars, label='reparametrization')
plt.yscale('log')
plt.xlabel('It')
plt.ylabel('Gradient variance')
plt.legend()
```

### Task: Analysis

* What do you notice about gradient variance as training progresses?
* What do you notice about the variance of the different estimators?
* How do batch size and number of posterior samples affect the variance of the different estimators?
* How does gradient variance affect convergence?
* Which gradient estimator would you use in practice for this problem?

## More

You can find similar experiments with more analysis and more gradient estimators in [this paper](https://arxiv.org/pdf/1906.10652.pdf). See Section 8.3.


# `sympy`

## 참고문헌
References

* Pedregosa, F., Sympy : Symbolic Mathematics in Python, Scipy Lecture Notes, 2019 March,[Online] Available : http://www.scipy-lectures.org/packages/sympy.html [Accessed 2019 10 28]

```python
import sympy as sy
sy.init_printing()

```

## 3.2.1.1 Calculator

```python
a = sy.Rational(1, 2)
a

```

```python
sy.pi ** 2

```

```python
sy.pi.evalf()

```

```python
(sy.pi + sy.exp(1))

```

```python
(sy.pi + sy.exp(1)).evalf()

```

```python
sy.oo

```

```python
(sy.oo > 99999)

```

```python
sy.sqrt(2).evalf(100)

```

```python
(sy.Rational(1, 2) + sy.Rational(1, 3))

```

인장하중 $F$ 에 항복하지 않을 환봉의 반지름 $r$

Radius of a circular bar $r$ that would not yield under tensile load F

```python
F, r, sigma_max, sf = sy.symbols('F r sigma_max safety_factor', real=True, nonnegative=True)
area = sy.pi * r * r
sigma = F / area
solution = sy.solve([sigma - sigma_max / sf], r, simplify=True)

```

```python
solution

```

## 3.2.1.2 Symbols

```python
x = sy.Symbol('x')
y = sy.Symbol('y')
(x + y + x - y)

```

```python
(x + y) ** 2

```

## 3.2.2.1 Expand

```python
sy.expand((x + y) ** 3)

```

```python
sy.expand(x + y, complex=True)

```

```python
sy.expand(sy.cos(x + y), trig=True)

```

## 3.2.2.2 Simplify

```python
sy.simplify((x + x * y) / x)

```

## 3.2.3.1 Limits

$$
\lim_{x \to 0} \frac{sin x}{x}
$$

```python
sy.limit(sy.sin(x) / x, x, 0)

```

$$
\lim_{x \to \infty} x
$$

```python
sy.limit(x, x, sy.oo)

```

$$
\lim_{x \to \infty} \frac{1}{x}
$$

```python
sy.limit(1 / x, x, sy.oo)

```

$$
\lim_{x \to 0} x^x
$$

```python
sy.limit(x ** x, x, 0)

```

## 3.2.3.2 Differentiation

```python
sy.diff(sy.sin(x), x)

```

```python
sy.diff(sy.sin(2 * x), x)

```

```python
sy.diff(sy.tan(x), x)

```

```python
sy.limit((sy.tan(x + y) - sy.tan(x)) / y, y, 0)

```

```python
sy.diff(sy.sin(2 * x), x, 1)

```

```python
sy.diff(sy.sin(2 * x), x, 2)

```

```python
sy.diff(sy.sin(2 * x), x, 3)

```

## 3.2.3.3 Series expansion

```python
sy.series(sy.cos(x), x)

```

```python
sy.series(1 / sy.cos(x), x)

```

```python
# Series 설명을 위하여 그래프를 그림
# 그래프 그리기 관련 기능 등을 담고 있는 pylab 모듈을 불러 들임
import pylab
# x 범위
x_deg = pylab.arange(-90, 90 + 1)
x_rad = pylab.deg2rad(x_deg)

# 참값
y_cos = pylab.cos(x_rad)

# 첫 항
y_series_1 = 1 * pylab.ones_like(x_rad)

# 둘째 항 까지의 합
y_series_2 = 1 - x_rad ** 2 / 2

# 셋째 항 까지의 합
y_series_3 = 1 - x_rad ** 2 / 2 + x_rad ** 4 / 24

# 그래프를 그림
pylab.plot(x_deg, y_cos, 'o', label='cos')
pylab.plot(x_deg, y_series_1, label='series 1')
pylab.plot(x_deg, y_series_2, label='series 1+2')
pylab.plot(x_deg, y_series_3, label='series 1+2+3')
pylab.grid()
pylab.legend(loc=0)
pylab.show()

```

## 3.2.3.4 Integration

$$
\int 6 x^5 dx
$$

```python
sy.integrate(6 * x ** 5, x)

```

$$
\int sin x dx
$$

```python
sy.integrate(sy.sin(x), x)

```

$$
\int log x dx
$$

```python
sy.integrate(sy.log(x), x)

```

$$
\int (2x + sinh x) dx
$$

```python
sy.integrate(2 * x + sy.sinh(x), x)

```

$$
\int \left(exp\left(-x^2\right)erf(x)\right) dx
$$

```python
sy.integrate(sy.exp(-x ** 2) * sy.erf(x), x)

```

정적분
Definite integral

$$
\int_{-1}^{1} x^3 dx
$$

```python
sy.integrate(x ** 3, (x, -1, 1))

```

$$
\int_{0}^{\frac{\pi}{2}} sinx dx
$$

```python
sy.integrate(sy.sin(x), (x, 0, sy.pi / 2))

```

$$
\int_{-\frac{\pi}{2}}^{\frac{\pi}{2}} cos x dx
$$

```python
sy.integrate(sy.cos(x), (x, -sy.pi / 2, sy.pi / 2))

```

Improper integrals

$$
\int_{0}^{\infty} e^{-x} dx
$$

```python
sy.integrate(sy.exp(-x), (x, 0, sy.oo))

```

$$
\int_{-\infty}^{\infty} e^{-x^2} dx
$$

```python
sy.integrate(sy.exp(-x ** 2), (x, -sy.oo, sy.oo))

```

## 3.2.4 Equation solving

$$
x^4=1
$$

```python
sy.solve(x ** 4 - 1, x)

```

```python
sy.solveset(x ** 4 - 1, x)

```

$$
e^x=-1
$$

```python
sy.solve(sy.exp(x) + 1, x)

```

$$
\begin {align}
      x + 5y &= 2 \\
    -3x + 6y &= 15
\end {align}
$$

```python
sy.solve([x + 5 * y - 2, -3 * x + 6 * y - 15], [x, y])

```

$$
x^4 - 3x^2 +1
$$

```python
f = x ** 4 - 3 * x ** 2 + 1
sy.factor(f)

```

```python
sy.factor(f, modulus=5)

```

Boolean equations

```python
sy.satisfiable(x & y)

```

```python
sy.satisfiable(x ^ y)

```

## 3.2.5.1 Matrices

```python
sy.Matrix([[1, 0], [0, 1]])

```

```python
A = sy.Matrix([[1, x], [y, 1]])
A

```

```python
A ** 2

```

## 3.2.5.2 Differential Equations

$$
\frac{d^2}{dx^2}f(x) + f(x)
$$

```python
f = sy.Function('f', real=True)
(f(x).diff(x, x) + f(x))

```

```python
sy.dsolve(f(x).diff(x, x) + f(x))

```

Mechanical Vibration

```python
m, c, k, t = sy.symbols('m c k t')
x = sy.Function('x', real=True)
vib_eq = m * x(t).diff(t, t) + c * x(t).diff(t) + k * x(t)
vib_eq

```

```python
result = sy.dsolve(vib_eq)
result

```

```python
sy.simplify(result)

```

Forced Vibration

```python
forced_vib_eq = m * x(t).diff(t, t) + c * x(t).diff(t) + k * x(t) - sy.sin(t)
forced_vib_eq

```

```python
result = sy.dsolve(forced_vib_eq)
result

```

```python
sy.simplify(result)

```

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```


# $Re$ corrections to static foil data

In this notebook we evaluate methods for correcting static foil data for Reynolds number effects.

We will use the Sheldahl and Klimas (1981) NACA 0021 datasets.

Yamauchi and Johnson (1983) proposed a simple correction derived from turbulent flat
plate boundary layer results, where the drag coefficient is corrected as

\begin{equation}
    C_d = C_{d_{\mathrm{table}}}/ K,
\end{equation}

where $K = f(Re_\mathrm{table})/f(Re)$. There are several choices for the
scaling function $f(Re)$:

\begin{equation}
    f(Re) = Re^{-0.5},
\end{equation}

\begin{equation}
    f(Re) = Re^{-0.2},
\end{equation}

\begin{equation}
    f(Re) = \frac{0.455}{\ln Re}2.58 - \frac{A}{Re},
\end{equation}

\begin{equation}
    f(Re) = (3.46 \ln Re - 5.6)^{-2},
\end{equation}

or

\begin{equation}
    f(Re) = (\ln Re - 0.407)^{-2.64}.
\end{equation}

Yamauchi and Johnson propose a correction for the lift coefficient that will
keep the lift slope constant, i.e.

\begin{equation}
    C_l = K C_{l_\mathrm{table}} (\alpha/K),
\end{equation}

where

\begin{equation}
    K = \alpha_{\max} / \alpha_{\max_\mathrm{table}} = C_{l_{\max}} /
    C_{l_{\max_\mathrm{table}}}.
\end{equation}

The scaling constant is chosen as

\begin{equation}
    K = \left( Re/Re_\mathrm{table} \right)^n,
\end{equation}

where $n$ is a small positive number between approximately 0.125 and 0.2.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from scipy.interpolate import interp1d
import pandas as pd
from pxl.styleplot import set_sns
set_sns()
```

    C:\Users\Pete\Anaconda3\lib\site-packages\matplotlib\__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
      warnings.warn(self.msg_depr % (key, alt_key))

```python
def load_data(Re, alpha_deg_max=25):
    """Load foil data."""
    fname = "data/NACA0021_{:.1e}.csv".format(Re).replace("+0", "")
    df = pd.read_csv(fname)
    return df[df.alpha_deg <= alpha_deg_max]

def lookup(df, alpha_deg, quantity="cl"):
    """Lookup specified quantity at given angle of attack using linear interpolation."""
    alpha_deg = np.asarray(alpha_deg)
    f = interp1d(df.alpha_deg, df[quantity])
    return f(alpha_deg)

def correct_cl(df, Re, Re_table, n=0.125):
    """Correct lift coefficient."""
    K = (Re/Re_table)**n
    df["cl_corr"] = K*lookup(df, df.alpha_deg/K, quantity="cl")
    return df

def f_Re_a(Re):
    return Re**(-0.5)

def f_Re_b(Re):
    return Re**(-0.2)

def f_Re_c(Re):
    A = None # ???
    return 0.455/np.log(Re) * 2.58 - A/Re

def f_Re_d(Re):
    return (3.46*np.log(Re) - 5.6)**(-2)

def f_Re_e(Re):
    return (np.log(Re) - 0.407)**(-2.64)

def correct_cd(df, Re, Re_table, f_Re=f_Re_a):
    """Correct drag coefficient."""
    K = f_Re(Re_table)/f_Re(Re)
    df["cd_corr"] = df.cd/K
    return df

Re_low = 1.6e5
Re_high = 3.6e5
df = load_data(Re_low)
df = correct_cl(df, Re_high, Re_low, n=0.23)
df = correct_cd(df, Re_high, Re_low, f_Re=f_Re_e)
df_high = load_data(Re_high)

fig, ax = plt.subplots(figsize=(7.5, 3.25), ncols=2)

ax[0].plot(df.alpha_deg, df.cl, label=r"Input ($Re_c = {:.1f} \times 10^5$)".format(Re_low/1e5))
ax[0].plot(df.alpha_deg, df.cl_corr, linestyle="--",
           label=r"Corr. ($Re_c = {:.1f} \times 10^5$)".format(Re_high/1e5))
ax[0].plot(df_high.alpha_deg, df_high.cl, label=r"Raw ($Re_c = {:.1f} \times 10^5$)".format(Re_high/1e5))
ax[0].set_ylabel("$C_l$")
ax[0].legend(loc="best")
ax[1].plot(df.alpha_deg, df.cd)
ax[1].plot(df.alpha_deg, df.cd_corr, linestyle="--")
ax[1].plot(df_high.alpha_deg, df_high.cd)
ax[1].set_xlim((0, 12))
ax[1].set_ylim((0, 0.04))
for a in ax:
    a.set_xlabel(r"$\alpha$ (deg.)")
fig.tight_layout()
plt.show()
```


# Amortized Neural Variational Inference for a toy probabilistic model

Consider a certain number of sensors  placed at known locations, $\mathbf{s}_1,\mathbf{s}_2,\ldots,\mathbf{s}_L$. There is a target at an unknown position $\mathbf{z}\in\mathbb{R}^2$ that is emitting a certain signal that is received at the $i$-th sensor with a signal strength distributed as follows:

\begin{align}
x_i \sim \mathcal{N}\Big(- A \log\left(||\mathbf{s}_i-\mathbf{z} ||^2\right), \sigma^2\Big),
\end{align}

where $A$ is a constant related to how fast signal strength degrades with distance. We assume a Gaussian prior for the unknown position $\mathcal{N}(\mathbf{0},\mathbf{I})$. Given a set of $N$ i.i.d. samples for each sensor, $\mathbf{X}\in\mathbb{R}^{L\times N}$, we will use a Amortized Neural Variational Inference to find a Gaussian approximation to

\begin{align}
p(\mathbf{z}|\mathbf{X}) \propto  p(\mathbf{X}|\mathbf{z}) p(\mathbf{z})
\end{align}

Our approximation to $p(\mathbf{z}|\mathbf{X})$ is of the form
\begin{align}
p(\mathbf{z}|\mathbf{X}) \approx q(\mathbf{z}|\mathbf{X})=\mathcal{N}\Big(\mu(\mathbf{X}),\Sigma(\mathbf{X})\Big),
\end{align}
where

- $\mu(\mathbf{X})$ --> Given by a Neural Network with parameter vector $\theta$ and input $\mathbf{X}$

- $\Sigma(\mathbf{X})$ --> Diagonal covariance matrix, where the log of the main diagonal is constructed by a Neural Network with parameter vector $\gamma$ and input $\mathbf{X}$

## ELBO lower-bound to $p(\mathbf{X})$

We will optimize $q(\mathbf{z}|\mathbf{X})$ w.r.t. $\theta,\gamma$ by optimizing the Evidence-Lower-Bound (ELBO):

\begin{align}
p(\mathbf{X}) &= \int p(\mathbf{X}|\mathbf{z}) p(\mathbf{z}) d\mathbf{z}\\
&\geq \int q(\mathbf{X}|\mathbf{z}) \log \left(\frac{p(\mathbf{X},\mathbf{z})}{q(\mathbf{X}|\mathbf{z})}\right)d\mathbf{z}\\
& = \mathbb{E}_{q}\left[\log  p(\mathbf{X}|\mathbf{z})\right] - D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})\triangleq \mathcal{L}(\mathbf{X},\theta,\gamma),
\end{align}
where $D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})$ is known in closed form since it is the KL divergence between two Gaussian pdfs:

\begin{align}
D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})) = \frac{1}{2} \left[\text{tr}\left(\Sigma(\mathbf{X})\right)+\left(\mu(\mathbf{X})^T\mu(\mathbf{X})\right)-2-\log\det \left(\Sigma(\mathbf{X})\right) \right]
\end{align}

## SGD optimization

- Sample $\mathbf{\epsilon}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$
- Sample from $q(\mathbf{z}|\mathbf{X})$:
\begin{align}
\mathbf{z}^0 = \mu(\mathbf{X}) + \sqrt{\text{diag}(\Sigma(\mathbf{X}))} \circ \mathbf{\epsilon}
\end{align}
- Compute gradients of
\begin{align}
\hat{\mathcal{L}}(\mathbf{X},\theta,\gamma) =\log  p(\mathbf{X}|\mathbf{z}^0) - D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})
\end{align}
w.r.t. $\theta,\gamma$

```python
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
%matplotlib inline

# use seaborn plotting defaults
import seaborn as sns; sns.set()
```

### Probabilistic model definition and generating samples

```python
############## Elements of the true probabilistic model ####################

loc_info = {}

loc_info['S'] = 3 # Number o sensors

loc_info['pos_s'] = np.array([[0.5,1], [3.5,1], [2,3]]) #Position of sensors

#loc_info['target'] = np.random.uniform(-3,3,[2,])  #(Unknown target position)

loc_info['target'] = np.array([-1,2])  #(Unknown target position)

loc_info['var_s'] = 5.*np.ones(loc_info['S']).reshape([loc_info['S'],1]) #Variance of sensors

loc_info['A'] = np.ones(loc_info['S'],dtype=np.float32) * 10.0 #Attenuation mean factor per sensor

loc_info['N'] = 5 # Number of measurements per sensor

def sample_X(S,M,z,pos_s,A,var_s):

    means = -1*A*np.log(np.sum((pos_s-z)**2,1))

    X = means.reshape([S,1]) + np.random.randn(S,M) * np.sqrt(var_s)

    return X

```

```python
# Sampling from model for the right target

X = sample_X(loc_info['S'],loc_info['N'], loc_info['target'],loc_info['pos_s'],loc_info['A'],loc_info['var_s'])
```

```python
plt.plot(loc_info['pos_s'][:,0],loc_info['pos_s'][:,1],'b>',label='Sensors',ms=15)
plt.plot(loc_info['target'][0],loc_info['target'][1],'ro',label='Target',ms=15)
plt.legend()
```

### TensorFlow Computation Graph and Loss Function

```python
z_dim = 2       #Latent Space

model_name = 'model1'   #In 'model1.py' we define the variational family

learning_rate = 1e-2
num_samples_avg = 1     #Number of samples to approximate the expectation in the ELBO
num_samples = 10        #Number of samples from the posterior (for testing)
num_it = int(1e4)       #SGD iterations
period_plot = int(1000) #Show resuts every period_plot iterations
dims = X.shape #X.shape

```

```python
sess_VAE = tf.Graph()

with sess_VAE.as_default():

    print('[*] Importing model: ' + model_name)
    model = __import__(model_name)

    print('[*] Defining placeholders')

    inputX = tf.placeholder(tf.float32, shape=dims, name='x-input')

    print('[*] Defining the encoder')
    log_var, mean, samples_z, KL = model.encoder(inputX,dims,z_dim,num_samples)

    print('[*] Defining the log_likelyhood')

    loglik = model.decoder(loc_info,inputX,samples_z,num_samples_avg)

    loss = -(loglik-KL)

    optim = tf.train.AdamOptimizer(learning_rate).minimize(loss)

    # Output dictionary -> Useful if computation graph is defined in a separate .py file

    tf_nodes = {}

    tf_nodes['X'] = inputX

    tf_nodes['mean'] = mean

    tf_nodes['logvar'] = log_var

    tf_nodes['KL'] = KL

    tf_nodes['loglik'] = loglik

    tf_nodes['optim'] = optim

    tf_nodes['samples'] = samples_z

```

    [*] Importing model: model1
    [*] Defining placeholders
    [*] Defining the encoder
    [*] Defining the log_likelyhood

## SGD optimization

```python

############ SGD Inference #####################################

mean_list = []

with tf.Session(graph=sess_VAE) as session:

    # Add ops to save and restore all the variables.
    saver = tf.train.Saver()

    tf.global_variables_initializer().run()

    print('Training the VAE ...')

    for it in range(num_it):

        feedDict = {tf_nodes['X'] : X}

        _= session.run([tf_nodes['optim'],tf_nodes['loglik'],tf_nodes['KL']],feedDict)

        if(it % period_plot ==0):

            mean, logvar,loglik,KL = session.run([tf_nodes['mean'],tf_nodes['logvar']],feedDict)

            print("It = %d, loglik = %.5f, KL = %.5f" %(it,loglik,KL))

            mean_list.append(mean)

    samples = session.run(tf_nodes['samples'],feedDict)

```

    Training the VAE ...
    It = 0, loglik = -6306.58984, KL = 0.34426
    It = 1000, loglik = -2361.39893, KL = 5.47620
    It = 2000, loglik = -180.33772, KL = 8.74085
    It = 3000, loglik = -93.22960, KL = 9.47736
    It = 4000, loglik = -93.06091, KL = 9.50548
    It = 5000, loglik = -92.69877, KL = 9.45156
    It = 6000, loglik = -94.65843, KL = 9.36135
    It = 7000, loglik = -92.64162, KL = 9.31016
    It = 8000, loglik = -92.65298, KL = 9.26902
    It = 9000, loglik = -102.03999, KL = 9.32650

```python
#Samples from q(z|x)
m_evol = np.vstack(mean_list)
nsamples = 50

samples = mean + np.sqrt(np.exp(logvar)) * np.random.randn(nsamples,2)

plt.plot(loc_info['pos_s'][:,0],loc_info['pos_s'][:,1],'b>',label='Sensors',ms=15)
plt.plot(loc_info['target'][0],loc_info['target'][1],'ro',label='Target',ms=15)
plt.plot(m_evol[:,0],m_evol[:,1],'g>',label='Post Mean')
plt.scatter(samples[:,0],samples[:,1],label='Post Samples')
plt.rcParams["figure.figsize"] = [8,8]
plt.legend()
```

```python

```

Notebooks/Part_7/Vartional Neural Inference Localization/.ipynb_checkpoints/Neural Variational Inference Localization Problem-checkpoint.ipynb


# Algorithms Exercise 2

## Imports

```python
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
```

## Peak finding

Write a function `find_peaks` that finds and returns the indices of the local maxima in a sequence. Your function should:

* Properly handle local maxima at the endpoints of the input array.
* Return a Numpy array of integer indices.
* Handle any Python iterable as input.

```python
a=[1,2,3,4,5,3]
for x in a:
    print(x)

```

```python
def find_peaks(a):
    """Find the indices of the local maxima in a sequence."""
    a=list(a)
    index=[]
    first=a[0]

    if first>a[1]:
        index.append(0)

    start=1
    for x in a[1:len(a)-1]:
        prev=a[a.index(x)-1]
        foll=a[a.index(x)+1]
        if x> prev and x>foll:
            index.append(a.index(x,start))
        start+=1
    last=a[len(a)-1]
    if last>a[len(a)-2]:
        index.append(len(a)-1)
    return(np.array(index))

find_peaks([3,2,1,0])

```

    array([0])

```python
p1 = find_peaks([2,0,1,0,2,0,1])
assert np.allclose(p1, np.array([0,2,4,6]))
p2 = find_peaks(np.array([0,1,2,3]))
assert np.allclose(p2, np.array([3]))
p3 = find_peaks([3,2,1,0])
assert np.allclose(p3, np.array([0]))
```

Here is a string with the first 10000 digits of $\pi$ (after the decimal). Write code to perform the following:

* Convert that string to a Numpy array of integers.
* Find the indices of the local maxima in the digits of $\pi$.
* Use `np.diff` to find the distances between consequtive local maxima.
* Visualize that distribution using an appropriately customized histogram.

```python
from sympy import pi, N
pi_digits_str = str(N(pi, 10001))[2:]
```

```python
pi_digits_list=np.array([int(x) for x in pi_digits_str])
pi_digits_list
x=np.diff(find_peaks(pi_digits_list))
plt.hist(x,20)
plt.title('Distance between local maxes in the digits of $\pi$')
plt.xlabel('Distance between Maxima')
plt.ylabel('Number of Maxima')
plt.xticks(range(0,30,2))
plt.xlim(right=22)
```

```python
assert True # use this for grading the pi digits histogram
```


# Exercise 1.1: Compute 1+1

```python
a = 1 + 1
print(a)
```

    2

# Exercise 1.2: Write a Hello World program

```python
print("Hello, World!")
```

    Hello, World!

# Exercise 1.3: Derive and compute a formula

```python
from sympy import Symbol

t = Symbol('t')    # Symbol: time [s]

t = t / 60 / 60 / 24 / 365.24225    # Unit conversion: t: [s] -> [a]

# Result
result = t.subs('t', 1e9)
print(f"10^9 seconds are {result:.3g} years!")
```

    10^9 seconds are 31.7 years!

# Exercise 1.4: Convert from meters to British length units

```python
x = 640    # length [m]

print(f"""
{x} m in British units are:
{x * 100 / 2.54 :.6g} inches or
{x * 100 / 2.54 / 12 :.6g} feet or
{x * 100 / 2.54 / 12 / 3 :.6g} yards or
{x * 100 / 2.54 / 12 / 3 / 1760 :.6g} miles
""")
```

    640 m in British units are:
    25196.9 inches or
    2099.74 feet or
    699.913 yards or
    0.397678 miles

# Exercise 1.5: Compute the mass of various substances

```python
from sympy import Symbol, solve

# Constants and symbols
V = 1                  # volume [dm^3]
rho = Symbol('rho')    # density [g / cm^3]
m = Symbol('m')        # mass [kg]

# Unit conversions:
V = V * 1000        # V: [dm^3 -> m^3]
rho = rho / 1000    # rho: [g / cm^3 -> kg / m^3]

e = m/V - rho
m = solve(e, m)[0]

print(f"One liter of air has a mass of {m.subs('rho', 0.0012):.3g} kg")
print(f"One liter of gasoline has a mass of {m.subs('rho', 0.67):.3g} kg")
print("...")
print(f"One liter of protons has a mass of {m.subs('rho', 2.3e14):.3g} kg")
```

    One liter of air has a mass of 0.00120 kg
    One liter of gasoline has a mass of 0.670 kg
    ...
    One liter of protons has a mass of 2.30e+14 kg

# Exercise 1.6: Compute the growth of money in a bank

```python
# Constants
A = 1000    # initial amount [€]
p = 5.0     # interest rate [% / a]
n = 3       # years

result = A * (1 + p/100)**n
print(f"The initial {A}€ will have turned into {result:.6g}€ after {n} years.")
```

    The initial 1000€ will have turned into 1157.63€ after 3 years.

# Exercise 1.7: Find error(s) in a program

```python
# Faulty Program:
"""
x=1; print("sin(%g)=%g" % (x, sin(x)))
"""

# Correct Program:
from math import sin
x=1; print("sin(%g)=%g" % (x, sin(x)))
```

    sin(1)=0.841471

# Exercise 1.8: Type in program text

```python
from math import pi

h = 5.0    # height
b = 2.0    # base
r = 1.5    # radius

area_parallelogram = h * b
print("The area of the parallelogram is %.3f" % area_parallelogram)

area_square = b ** 2
print("The area of the square is %g" % area_square)

area_circle = pi * r**2
print("The area of the circle is %.3f" % area_circle)

volume_cone = 1.0 / 3 * pi * r**2 * h
print("The volume of the cone is %.3f" % volume_cone)
```

    The area of the parallelogram is 10.000
    The area of the square is 4
    The area of the circle is 7.069
    The volume of the cone is 11.781

# Exercise 1.9: Type in programs and debug them

```python
from math import sin, cos, pi
x = pi/4
val = sin(x)**2 + cos(x)**2
print(val)
```

    1.0

```python
v0 = 3
t=1
a = 2
s = v0*t + (0.5 * a * t**2)
print(s)
```

    4.0

```python
a = 3.3; b = 5.3
a2 = a**2
b2 = b**2

eq1_sum = a2 + 2*a*b + b2
eq2_sum = a2 - 2*a*b + b2
eq1_pow = (a + b)**2
eq2_pow = (a - b)**2
print("First equation:  %g = %g" % (eq1_sum, eq1_pow))
print("Second equation: %g = %g" % (eq2_pow, eq2_pow))
```

    First equation:  73.96 = 73.96
    Second equation: 4 = 4

# Exercise 1.10: Evaluate a Gaussian function

```python
from math import sqrt, pi, exp

m = 0    # mean
s = 2    # standard deviation
x = 1    # independent variable

f = 1 / sqrt(2 * pi) / s * exp(-1 / 2 * ((x - m) / s)**2)
print(f)
```

    0.17603266338214976

# Exercise 1.11: Compute the air resistance on a football

```python
from math import pi

# Constants
CD = 0.4     # drag coefficient [1]
rho = 1.2    # air density [kg / m^3]
a = 11e-2    # radius of the ball [m]
m = 0.43     # mass of the ball [kg]
g = 9.81     # acceleration of gravity [m / s^2]
V1 = 120     # (1) velocity of the ball [km / h]
V2 = 30      # (2) velocity of the ball [km / h]

# Unit conversions
V1_ = V1 * 1000 / 60 / 60    # V1: [km / h] -> [m / s]
V2_ = V2 * 1000 / 60 / 60    # V2: [km / h] -> [m / s]

# Calculations
A = pi * a**2    # area of the ball [m^2]
Fg = m * g       # force of gravity [N]
Fd1 = 1 / 2 * CD * rho * A * V1_**2    # drag force on the ball (with V1)
Fd2 = 1 / 2 * CD * rho * A * V2_**2    # drag force on the ball (with V2)

print(f"""
Force of gravity on the ball: {Fg:.1f} N
Drag Force on the ball (V = {V1} km / h): {Fd1:.1f} N
Drag Force on the ball (V = {V2} km / h): {Fd2:.1f} N
""")
```

    Force of gravity on the ball: 4.2 N
    Drag Force on the ball (V = 120 km / h): 10.1 N
    Drag Force on the ball (V = 30 km / h): 0.6 N

# Exercise 1.12: How to cook the perfect egg

```python
from sympy import Symbol, log, lambdify
from math import pi
from numpy import *

# Constants and symbols
Ty = 70              # maximum temperature of the yolk [°C]
Tw = 100             # temperature of the boiling water [°C]
M = 67e-3            # mass of the egg [kg]
rho = 1.038e3        # density of the egg [kg / m^3]
c = 3.7e3            # specific heat capacity of the egg [J / kg / K]
K = 0.54             # thermal conductivity [W / K]
T0 = Symbol('T0')    # initial temperature of the egg [°C] (Symbol)
T01 = 4              # initial temperature of the egg when taken from the fridge [°C]
T02 = 20             # initial temperature of the egg at room temperature [°C]

t = M**(2 / 3) * c * rho**(1 / 3) / K / pi**2 / (4 * pi / 3)**(2 / 3) \
    * log(0.76 * (T0 - Tw) / (Ty - Tw))    # time to boil the egg (t(T0))
t = lambdify((T0), t, dummify = False)

t1 = t(T0 = T01)    # time to boil the egg taken from the fridge
t2 = t(T0 = T02)    # time to boil the egg at room temperature

print(f"""
Time to boil the egg from the fridge: {t1:.3g} s
Time to boil the egg at room temperature: {t2:.3g} s
""")
```

    Time to boil the egg from the fridge: 397 s
    Time to boil the egg at room temperature: 315 s

# Exercise 1.13: Derive the trajectory of a ball

Prerequisites:
$$
\begin{align*}
a_x &= \frac{d^2x}{dt^2}\\
a_y &= \frac{d^2y}{dt^2}\\
\\
\frac{d}{dt}x(0) &= v_0 \cos\theta\\
\frac{d}{dt}y(0) &= v_0 \sin\theta\\
x(0) &= 0\\
y(0) &= y_0\\~\\
\end{align*}
$$
Calculations for $x(t)$ and $y(t)$:
$$
\begin{align*}
\int\int a_x &= \frac{1}{2} a_x t^2 + v_0 \cos\theta \dot~ t + x_0\\
&= v_0 \cos\theta \dot~ t\\
&= x(t)\\
\int\int a_y &= \frac{1}{2} a_y t^2 + v_0 \sin\theta \dot~ t + y_0\\
&= \frac{-1}{2} g t^2 + v_0 \sin\theta \dot~ t + y_0\\
&= y(t)\\~\\
\end{align*}
$$
$\theta = \frac{\pi}{2}$:
$$
\begin{align*}
x(t) &= v_0 \cos\theta \dot~ t\\
&= 0\\
y(t) &= \frac{-1}{2} g t^2 + v_0 \sin\theta \dot~ t + y_0\\
&= \frac{-1}{2} g t^2 + v_0 t + y_0\\~\\
\end{align*}
$$
Elimination of $t$:
$$
\begin{align*}
&~ &x(t) &= v_0 \cos\theta \dot~ t\\
&\Leftrightarrow &t(x) &= \frac{x}{v_0 \cos\theta}\\
&\Rightarrow &y(x) &= \frac{-1}{2} g \frac{x^2}{v_0^2 \cos^2\theta} + v_0 \sin\theta \frac{x}{v_0 \cos\theta} + y_0\\
&&&= x \tan{\theta} - \frac{1}{2 v_0^2} \frac{g x^2}{\cos^2{\theta}} + y_0
\end{align*}
$$

# Exercise 1.14: Find errors in the coding of formulas

Correct formula:
$$
F = \frac{9}{5}C + 32
$$

```python
# Adapted for Python 3
C = 21;   F = 9/5*C + 32;      print(F)    # works
C = 21.0; F = (9/5)*C + 32;    print(F)    # works
C = 21.0; F = 9*C/5 + 32;      print(F)    # works
C = 21.0; F = 9.*(C/5.0) + 32; print(F)    # works
C = 21.0; F = 9.0*C/5.0 + 32;  print(F)    # works
C = 21;   F = 9*C/5 + 32;      print(F)    # works
C = 21;   F = (1/5)*9*C + 32;  print(F)    # works
C = 21;   F = (1./5)*9*C + 32; print(F)    # works

# yeah, Pyhon 3!
# Note for Python 2: watch out for integer divisions!
```

    69.80000000000001
    69.80000000000001
    69.8
    69.80000000000001
    69.8
    69.8
    69.80000000000001
    69.80000000000001

# Exercise 1.15: Explain why a program does not work

```python
# Code (illegal):
"""
C = A + B
A = 3
B = 2
print(C)
"""

# Code (correct):
A = 3
B = 2
C = A + B
print(C)

# Objects can only be used after they have been defined!
```

    5

# Exercise 1.16: Find errors in Python statements

```python
# 1a = 2
# Variable names cannot start with a number.
a1 = 2

# a1 = b
# b is not defined.
a1 = 3

x = 2

# y = X + 4
# python differentiates between upper- and lowercase
y = x + 4

# from Math import tan
# the module is named math (lowercase)
from math import tan

# print(tan(pi))
# pi also has to be imported.
from math import pi
print(tan(pi))

# pi = "3.14159’
# strings can be defined using single or double quotes, but not both for the same string.
pi = "3.14159"

# print(tan(pi))
# pi is now a string. firs, it has to be converted to float.
print(tan(float(pi)))

c = 4**3**2**3

# _ = ((c-78564)/c + 32))
# a number cannot be assigned anything. it only can be assigned to a variable. also, there is an extra parenthesis
d = ((c-78564)/c + 32)

# discount = 12%
# the percent sign is not defined as such in python.
discount = 12e-2

# AMOUNT = 120.-
# same goes for that .- thingy.
AMOUNT = 120.0

# amount = 120$
# again for the dollar sign
amount = 120    # [$]

# address = hpl@simula.no
# that is not a valid string.
address = "hpl@simula.no"

# and = duck
# much wrong here. a variable cannot be named "and", because that is a reserved name in python.
# also, duck is not defined / not a valid string.
and_ = "duck"

# class = ’INF1100, gr 2"
# wow. ok, class is again reserved, and you still can't use different quotes to define one string
class_ = "INF1100, gr 2"

continue_ = x > 0

rev = fox = True

Norwegian = ["a human language"]

true = fox is rev in Norwegian
```

    -1.2246467991473532e-16
    -2.653589793362073e-06

# Exercise 1.17: Find errors in the coding of a formula

```python
# Code (illegal)
"""
a = 2; b = 1; c = 2
from math import sqrt
q = b*b - 4*a*c
q_sr = sqrt(q)
x1 = (-b + q_sr)/2*a
x2 = (-b - q_sr)/2*a
print(x1, x2)
"""

# Code (correct)
a = 2; b = 1; c = 2
from cmath import sqrt
q = b*b - 4*a*c
q_sr = sqrt(q)
x1 = (-b + q_sr)/2*a
x2 = (-b - q_sr)/2*a
print(x1, x2)
```

    (-1+3.872983346207417j) (-1-3.872983346207417j)

# Exercise 1.18: Find errors in a program

```python
# Code (wrong)
"""
from math import pi, tan
tan = tan(pi/4)
tan2 = tan(pi/3)
print(tan, tan2)
"""

# Code (correct)
from math import pi, tan
tan1 = tan(pi/4)
tan2 = tan(pi/3)
print(tan1, tan2)
```

    0.9999999999999999 1.7320508075688767