## Automatic Gain Control with PYNQ

We present a digital Automatic Gain Control (AGC) circuit with interactive control of it's parameters. This is a purely digital loopback system, so no extra hardware is required. We'll generate various input signals, explore some interesting effects of the AGC algorithm, and practice tweaking our parameters for best performance.
The design of this AGC example will be explored, featuring various hardware arithmetic approximations for power estimation, logarithms and exponentiation.

## Aims
* Interactively explore the behaviour of an AGC circuit
* Learn to tune the AGC parameters for different scenarios
* Explore our open-source hardware design, with a focus on arithmetic approximations

## Table of Contents

* [1. Introduction](#1.-Introduction)
* [2. ...And we're off!](#go)
* [3. Exploring some scenarios](#3.-Exploring-some-scenarios)
* [4. A closer look at the hardware](#hw)
    * [4.1 Power estimation](#power-est)
    * [4.2 Logarithms and exponentials](#log-est)
* [5. Conclusion](#conclusion)

----

## 1. Introduction

This example is a purely digital loopback, so there is no requirement for extra hardware. If you've got a Pynq Z2 board, ZCU111, or RFSoC 2x2, you can run it and have a play with all of the interactive features. Let's take a look at how this demonstration is structured.

Figure 1: System overview

The datapath is a simple loopback and we'll generally be following only a few steps:

  * Generate a signal in software
  * Stream our input signal _to_ the AGC circuit, via the first DMA
  * Stream our output signal _from_ the AGC, via the second DMA
  * Inspect the output signal via this notebook

We should also note that there are a few parameters for the AGC circuit, which we can set from software as well. Just before we start interacting with our AGC circuit, let's take a moment to look at it's design and appreciate what its behaviour should be. Our circuit is loosely based on the algorithm presented in [this book](https://www.elsevier.com/books/rf-and-digital-signal-processing-for-software-defined-radio/rouphael/978-0-7506-8210-7)[1] (and there's a free excerpt from the AGC section [online](https://www.edn.com/wireless-101-automatic-gain-control-agc/)[2]).

### Why use AGC?

The main goal of the AGC circuit is to keep a signal's average power steady, at a known level. This is used to mitigate fluctuations in a signal's received power. In the analogue world, this is often to ensure that we present a full-scale signal to our ADC to minimise the impact of quantisation errors. We'll show the use of such an external analogue gain with RFSoC in a future notebook. However, there are quite a few uses in the digital world too --- generally when we want to normalise signals digitally at IF or baseband. Synchronisation for QAM receivers is one example. The synchronisation task is much simpler if we have a priori knowledge of the signal's average power, since we know what amplitude to expect for each point in the constellation. We can use an AGC, just like this one, to ensure our synchronisation circuit is given a signal with it's preferred average power. This can not only "correct" the received signal's power, but also adjust for any changes over time.

### Our Design

Our AGC circuit is a feedback loop that will amplify our signal, aiming to match a reference average power. There are 3 main parameters we will be tuning from software: the desired average power (ref), the number of samples used to calculate the average power (window), and how aggressively we want to vary our gain (alpha).

Figure 2: AGC block diagram

Our control is based on the _logarithm_ of the input power... and these $log_{10}(x)$ and $10^{x}$ blocks _really_ complicate the digital design. A much simpler AGC design could respond linearly (instead of logarithmically) but this would have slightly different behaviour. We opt for the more painful route for two reasons:

  * The log response lets us react to large abrupt changes to the input signal roughly as quickly as we can to small changes. This is a nice property to have when dealing with packets or bursts of data. It also simplifies any analysis if we can assume that the "recovery time" is independent of the input signal.

  * It will make it easier to interface to analogue Variable Gain Amplifiers (VGAs) in the future. Most  VGAs offer a control for their gain that is "linear in dB" (i.e. it's logarithmic!). By having our control loop already using logarithms, we could have direct linear control of a VGA without extra interfacing circuitry.

Let's now explore the behaviour of our system while getting our hands dirty

## 2. ...And we're off!

Run the following code cell to launch our GUI. This lets you set AGC parameters and generate your own input signals. The input signals will automatically be streamed through the AGC circuit and visualised below. You can control the envelope of the input signal by clicking and dragging the "envelope handles" (the blue circles). The window parameter (how many samples are averaged during our power calculation) is shown on the output plot as a grey area at the far left.

Check [this animation](https://github.com/strath-sdr/pynq_agc/blob/master/demonstration.gif) to see an example of setting the envelope.

```python
import pynq_agc

model = pynq_agc.AgcDashModel(fs=int(1e6), N=6000)
app = pynq_agc.AgcDashController(model)
app.show()
```

We'd recommend having a play with different envelopes, signal types, and AGC parameters to get an intuition for how the AGC behaves. Once you've done some exploring, we present a couple of (somewhat) realistic scenarios and discuss them below.

## 3. Exploring some scenarios

Here are a few examples of configurations which demonstrate real-world communications challenges related to AGC.
You can select each scenario the with "Preset Scenarios" input at the bottom of the GUI.

> ### Slow Fading
> Here we have an effect that emulates "slow fading" of a simple signal. We'd usually expect this to be caused by movement, often with buildings or terrain impeding the signal. If the receiver is sensitive to the amplitude of the received signal (e.g. AM or many QAM synchronisers), we'd need use an AGC to compensate for the changes in signal strength.
>
> Try different window and alpha parameters to see how constant you can keep the output power. Notice that increasing the window size will reduce the time resolution of our gain control, which leads to a noticeable saw-tooth envelope for slow fading effects. If alpha is too high, we can see the output ringing where we've overshot by adjusting the gain too aggressively. If alpha is too low, we might not be compensating quickly enough for the fading effects.

> ### AM Envelope Preservation
> AM demodulation is a classic AGC example. For AM radio receivers, we want to play baseband signals with a fixed average power so broadcasts from different locations are all audible (and the more powerful ones aren't deafening!). Our information is encoded in the envelope of the signal and we need to be very careful not to distort it. If our AGC reacts too quickly, it can start to smooth out the envelope, destroying our information. Try reducing the window until this starts to happen. The window length is shown as the grey rectangle on the output plot. Try to achieve the same effect by decreasing the modulated "Data" frequency as well.

> ### Packet Preambles with QPSK
> Here we present a bursty QPSK transmission, as if receiving a single packet.
> We can clearly see the recovery time the AGC needs in order to converge to a steady gain. Having this steady, known power can be important for synchronisation (especially for higher order QAMs) and we can change this level using the ref parameter. However the AGC's recovery time means that the first few symbols will likely not be received correctly. In this situation we need to tune our AGC parameters to ensure we'll reach equilibrium before the packet's preamble is finished (or vice versa). We can tune our alpha, ref, and window parameters and then deduce the minimum preamble length that would be appropriate for this packet format. Just like our AM scenario, QPSK and QAM signals will start to become distorted if our window is small relative to the symbol period.

The rest of the notebook will address our AGC's hardware design in a little more depth. If you only want to use the AGC and aren't interested in the FPGA design, feel free to stop here.

## 4. A closer look at the hardware

This section will give some insights into the FPGA design for our AGC and we'll focus on the fixed-point arithmetic techniques used to implement the more exotic functions such as power estimation, logarithms, exponentiation. The full source code for the circuit is available on [our GitHub](https://github.com/strath-sdr/pynq_agc) and is written in the excellent functional language, [Clash](https://clash-lang.org/). First of all, let's revisit our block diagram.

Figure 3: Revisiting the AGC block diagram

This model presents quite a concise mathematical system. Drawing from the loop analysis presented by [Tony Rouphael](https://www.elsevier.com/books/rf-and-digital-signal-processing-for-software-defined-radio/rouphael/978-0-7506-8210-7), we can formally represent this model with a few manageable equations.

Our averaged input power, $P_{in}$, is at the output of our integrate-and-dump block. Given inputs $I_b(n)$, $Q_b(n)$, and our window parameter $N$, we define $P_{in}(n)$ as:

$$ P_{in}(n) = \frac{1}{N} \sum_{k=0}^{N-1}\sqrt{ I_b^2(n-k) + Q_b^2(n-k) } $$

We also need to find an equation for the output of our accumulator, $v(n)$. Given our alpha and ref parameters, $\alpha$ and $P_{ref}$, we can write an expression for $v(n)$ in state-space representation:

$$ v(n+1) = v(n) + \alpha [P_{ref} - \log_{10}(10^{v(n)} P_{in}(n))] $$

And finally to close the loop, we can model the outputs of our gain stage, $I_b(n)$ and $Q_b(n)$, in terms of the input signal, $I_a(n)$ and $Q_a(n)$, and $v(n)$.

$$ I_b(n) = 10^{v(n)}I_a(n) \\ Q_b(n) = 10^{v(n)}Q_a(n) $$

Although the _analysis_ of this system can be quite complex, it is at least simple to describe using this state-space form. In fact, we could translate these equations to a software implementation with very little effort.

```python
import numpy as np

def python_agc(i_a, q_a, p_ref, alpha, N):

    v = 0
    i_b = np.zeros((len(i_a),))
    q_b = np.zeros((len(q_a),))

    for (n,(i,q)) in enumerate(zip(i_a,q_a)):
            i_b[n] = (10**v * i)
            q_b[n] = (10**v * q)

            # Protect against averaging empty lists           vvvvvvvvvvvvvv
            p_in = np.average( np.abs(i_b + 1j*q_b)[n-N:n]    if n>=N else 0 )

            # Protect against log10(0) for later              vvvvvvvvvvvvvvvvvvvvvv
            inner_term = 10**v * p_in                         if p_in > 0 else 1e-12

            v = v + alpha * (p_ref - np.log10(inner_term))

    return (i_b, q_b)
```

So, if a software implementation (albeit a _very_ slow one) is so easy to implement, what is the challenge in translating this to hardware? Of course the standard answers apply here, such as complexity introduced by implementing AXI-Stream / DataFlow control, fixed-point conversion, pipelining, etc. In this particular case, though, there are two interesting challenges: power estimation and logarithms/exponentials.

### 4.1. Power estimation

Our power estimation step is presented with a complex signal and needs to compute it's magnitude:

$$ P_{in}(n) = \sqrt{ I_b^2(n) + Q_b^2(n) } $$

First of all, we could implement the $x^2$ function as a single multiply but we might want to avoid this if the wordlengths exceed that of our DSP48 slice (27x18 for RFSoC or 25x18 for Pynq-Z2). For RFSoC devices our I/Q values will be 16 bit and we immediately multiply it by our gain. Even a small gain would take our signal beyond the 18 bit limit, so each $x^2$ operation would consume 4 DSP48 slices! Beyond this, we still face the challenge of implementing the $\sqrt{x}$ function. This is much harder to implement directly with our standard set of arithmetic features. We could do this using a technique such as CORDIC (presented in the [next section](#4.2.-Logarithms-and-exponentials)) but since the multiplications are also a little awkward to implement, let's consider an approximation for the whole function.

Stepping back from circuit design for a moment, one common mathematical approximation for our power estimation is:

$$ P_{in}(n) \approx \alpha \cdot \text{max} \{ I_b(n),Q_b(n) \} + \beta \cdot\text{min} \{ I_b(n),Q_b(n) \} $$

This somewhat surprising approximation reduces the power estimation to one comparison (to find the min and max of our inputs), one addition, and two multiplications. This is a very good starting point, but we can avoid even the multiplications if we intelligently select values for our constants, $\alpha$ and $\beta$. Some suggested values are listed by a lovely little bit of software by [Grant R. Griffin](https://dspguru.com/dsp/tricks/magnitude-estimator/)[3]. We opt for $\alpha = \frac{61}{64}$ and $\beta = \frac{13}{32}$.

Multiplications by powers of 2 can clearly be replaced by shifts (implemented for free in the routing). In order to implement our chosen $\alpha$ and $\beta$ constants, we can construct non-power of 2 multiplications by adding/subtracting powers of 2 or any other intermediate values. We can model these constant multiplications as a graph of shifts and additions/subtractions. We could probably construct this graph intuitively for simple examples but we can rely on software tools such as the [SPIRAL project](http://spiral.ece.cmu.edu/mcm/gen.html) to construct graphs for more awkward coefficients[4]. In our case, our constant multiplication circuits become:

Figure 4: Implementing constant multiplications

In summary, we can use an approximation to reduce our power estimation to the sum of two multiplications. We can select our coefficients wisely to further reduce our circuit to a graph of shifts and additions/subtractions --- in our case, each of our multiplications require only two additions. Of course this is just an _approximation_ of the actual power. Our coefficients produce an average error of less than $1\%$ and a worst case error of $4.7\%$; a decent balance between implementation cost and accuracy.

### 4.2. Logarithms and exponentials

We explored two methods of implementing logarithms and exponentials --- one single-cycle and one iterative implementation. A single-cycle approach was appealing in the context of this AGC because we'd like to avoid introducing extra latency into our feedback loop. The longer our latency, the greater the potential for our control signal to "ring" or oscillate around our desired value. Unfortunately this approach requires FPGA resources that grow exponentially with the desired precision. This worked very nicely for wordlengths around 16 bits, but our full 26 bit words would pose a serious challenge. While our final design uses the second approach, both are interesting to discuss.

#### Intuitive  single-cycle logarithm

Since we're working with digital logic, let's start by looking at base 2 only. We're working towards designing a circuit to perform $\log_2(x)$ and $2^x$. This approach uses one mathematical trick and then brute-forces the rest with a precomputed look-up table. It resembles the solution proposed by [this Beyond Circuits blog post](https://www.beyond-circuits.com/wordpress/2015/10/computing-the-logarithm-base-2/)[5].

There is one trick that applies to unsigned numbers greater than 0. Here the index of the most significant '1' bit in a binary word is equal to the integer part of our base 2 logarithm. We can demonstrate this just by considering the information we have about a number's range given the position of its most significant '1'.
For any binary number, $x$, whose most significant one is at bit index $b$:

$$2^b \leq x \lt 2^{b+1}$$

This inequality holds also for the logarithm of x:

$$
\begin{align}
\log_2(2^b) &\leq \log_2(x) \lt \log_2(2^{b+1}) \\
b &\leq \log_2(x) \lt b+1
\end{align}
$$

This tells us that, since $b$ is an integer, $\lfloor \log_2(x) \rfloor = b$. Equipped with this observation we can implement a full fixed-point logarithm circuit by adding a look-up table for the fractional bits. We'll feed the $N$ bits immediately following the most significant '1' into the address lines of this look-up table, returning our precomputed result for $log_2(1+2^{-N}\text{addr})$.

Figure 5: Example of LUT-based logarithm

The final step is to change the base of the logarithm to 10. We can use the following identity to do this with a multiplication by a constant.

$$
\begin{align}
\log_{10}(x) &= \frac{\log_2(x)}{\log_2(10)} \\~\\
  &\approx 0.301 \log_2(x)
\end{align}
$$

Just be mindful of your output wordlengths in order to fit this multiplication in a single DSP48 slice. The main issue for our design is the size of the LUT required --- it needs $2^N$ entries to perform the logarithm over $N+1$ input bits. Let's now take a look at our second method, which instead sacrifices some extra clock cycles to keep our circuit slim.

#### Hyperbolic CORDIC

Our second implementation is based on Jack Volder's 1959(!) Coordinate Rotation DIgital Computer, or [CORDIC](https://dl.acm.org/doi/10.1145/1457838.1457886)[6], and J.S. Walther's [insights from 1971](https://dl.acm.org/doi/10.1145/1478786.1478840) which unified many similar algorithms[7]. Let's begin by moving the goalposts and discussing how we can solve trigonometric functions in digital logic.

Although we are very used to seeing $\sin(\theta)$ plotted as a continuous wave against a time-axis, we can also think of it as a rotation of a unit vector by $\theta$ radians. This has been very nicely animated by a Desmos user (unfortunately anonymously), embedded below.

```python
%%html

```

The heart of Volder's observation is that we can very efficiently rotate a vector by any angle of the form $\pm\tan^{-1}(2^{-n})$, albeit with a slight change in magnitude. As with our multiplications by a constant in the previous section, shifts and adds are the key to keeping our circuit small. Volder noted that we can rotate a vector by simply incrementing the $x$ coordinate by a power-of-two fraction (a right shift) of the $y$ coordinate and vice versa. This means we only need two additions and a shift for each rotation. The main disadvantage of this simple rotation is that the magnitude of the vector is slightly distorted, but we can correct for this with a constant multiplication or wisely selecting our input values.

Our rotation of a vector, $(x_i,y_i)$, by an angle of $\tan^{-1}(2^{-n})$ radians is formally shown below.

$$
x_{i+1} = x_i + 2^{-n}y_i  \\
y_{i+1} = y_i - 2^{-n}x_i$$

To rotate in the other direction, we just need to the swap the signs of the second terms. Why this simple expression results in a rotation might not be immediately obvious, so let's explore this using our own Desmos animation. We show the original vector in purple and the rotated vector in orange. We can clearly see a change in angle and a modest change in length. We can directly calculate the length of the green vector (which is perpendicular to the original vector) and use a little bit of trigonometry to find the angle of rotation.

```python
%%html

```

Given that we can rotate in either direction, we can iterate this rotation step over progressively smaller and smaller angles until we converge on our target angle. We're essentially performing a binary search for our given angle --- the number of bits of precision is approximately equal to the number of iterations we perform. This point is key since our circuit size no longer scales with $2^N$ for $N$ bits; it now scales linearly.

We can chain multiple iterations together in order to rotate a vector by an arbitrary angle, and this gives us the sine/cosine functions. We could also choose to configure our iterations to minimise the $y$ coordinate instead of achieving a fixed rotation. This gives us a means of calculating the magnitude and angle of a vector, since we're left with only a real component and we know how much it was rotated. That's already a huge amount of utility for one circuit, but there's more! From here, it's a small step to tweak the algorithm to allow for hyperbolic trigonometric functions as well. This step was contributed by [J. S. Walther](https://dl.acm.org/doi/10.1145/1478786.1478840) when unifying some of Volder's ideas. This is important to us because we can express both $\ln(x)$ and $e^x$ in terms of hyperbolic trigonometric functions:

$$
\ln(x) = 2 \tanh^{-1}\left(\frac{x-1}{x+1}\right) \\
e^x = \sinh(x) + \cosh(x)
$$

We just need to include the same base-changing expression as in the previous section and we can move from base $e$ to base $10$ with a single constant multiplication. There we have it --- a circuit for logarithms and exponentials for large wordlengths! Again, all of the source code for this is available on [our GitHub](https://github.com/strath-sdr/pynq_agc). We'd also like to thank Adam Walker who has published [a suite of DSP implementations](https://github.com/adamwalker/clash-utils) in Clash, including a non-hyperbolic CORDIC that was a useful reference for this work[8].

## 5. Conclusion

We've presented a digital AGC design for use at baseband or IF, and provided an interactive GUI to help _gain_ an intuitive understanding of how the AGC behaves. We've discussed some example scenarios which highlight real-world applications for such an AGC design and how this influences our choice of reference power, alpha, and window length parameters. We've also dived into two interesting parts of the hardware design in detail, explaining how some of the more advanced mathematical functions can be implemented cheaply on the FPGA fabric. Hopefully this has encouraged a better understanding of how AGCs work, what they can be used for, and some memorable DSP tricks for your own hardware designs.

We will shortly release another version of this demonstration that uses an external Variable Gain Amplifier and some of the RFSoC's signal monitoring features, so watch this space if you have an RFSoC.

## References

[1] - [Tony Rouphael, "RF and Digital Signal Processing for Software-Defined Radio"](https://www.elsevier.com/books/rf-and-digital-signal-processing-for-software-defined-radio/rouphael/978-0-7506-8210-7)

[2] - [EDN, "Wireless 101: Automatic Gain Control (AGC)", Accessed: 09/02/2021](https://www.edn.com/wireless-101-automatic-gain-control-agc/)

[3] - [Grant Griffin, DSPGuru, "DSP Trick: Magnitude Estimator", Accessed: 09/02/2021](https://dspguru.com/dsp/tricks/magnitude-estimator/)

[4] - [Yevgen Voronenko, SPIRAL Project, "Multiplier Block Generator", Accessed: 09/02/2021](http://spiral.ece.cmu.edu/mcm/gen.html)

[5] - [Pete Johnson, Beyond Circuits, "Computing the Logarithm Base 2", Accessed: 22/02/2021](https://www.beyond-circuits.com/wordpress/2015/10/computing-the-logarithm-base-2/)

[6] - [Jack Volder, "The CORDIC computing technique", Accessed: 22/02/2021](https://dl.acm.org/doi/10.1145/1457838.1457886)

[7] - [J. S. Walther, "A unified algorithm for elementary functions", Accessed: 22/02/2021](https://dl.acm.org/doi/10.1145/1478786.1478840)

[8] - [Adam Walker, "clash-utils" GitHub Repository, Accessed: 22/02/2021](https://github.com/adamwalker/clash-utils)

## Revision History
* **v0.2** | *Craig Ramsay* | 22/09/2021 | Added scenarios and notebook discussion
* **v0.1** | *Craig Ramsay* | 09/02/2021 | First alpha demo

[⬅ Previous Notebook](previous_notebook.ipynb) | | [Next Notebook ➡](next_notebook.ipynb)

----
----


# The effect of tangential velocities

It was first noted by [Rezzolla, Zanotti and Pons](http://dx.doi.org/10.1017/S0022112002003506) that there is one effect that is specific to *relativistic* Riemann problems. Thanks to the Lorentz factor coupling the tangential velocities more tightly to the other terms, it is possible to change the wave pattern in the problem solely by changing the tangential velocity. This cannot happen in the Newtonian case.

The examples used in their paper use a *modified* relativistic Sod problem as a base. We set that problem up first:

```python
from r3d2 import eos_defns, State, RiemannProblem
from IPython.display import display, display_png
```

```python
eos = eos_defns.eos_gamma_law(5.0/3.0)
U_left_sod_modified = State(1.0, 0.5, 0.0, 1.5, eos)
U_right_sod_modified = State(0.125, 0.0, 0.0, 1.2, eos)
rp_sod_modified = RiemannProblem(U_left_sod_modified, U_right_sod_modified)
display(rp_sod_modified)
display_png(rp_sod_modified)
```

We see the wave pattern has a left going rarefaction and a right going shock.

Now, by modifying the tangential velocity of the right state, the wave pattern changes:

```python
U_right_vt_0_9 = State(0.125, 0.0, 0.9, 1.2, eos)
rp_vt_0_9 = RiemannProblem(U_left_sod_modified, U_right_vt_0_9)
display(rp_vt_0_9)
display_png(rp_vt_0_9)
```

The result is a left going *shock*, rather than a rarefaction.

A shock-rarefaction problem can be modified, via a tangential velocity, to a two rarefaction problem in a similar fashion:

```python
U_left_sod_modified2 = State(1.0, 0., 0.0, 1.5, eos)
U_right_sod_modified2 = State(0.125, 0.5, 0.0, 1.2, eos)
rp_sod_modified2 = RiemannProblem(U_left_sod_modified2, U_right_sod_modified2)
display(rp_sod_modified2)
display_png(rp_sod_modified2)
```

And now with tangential velocity:

```python
U_left_vt_0_9 = State(1.0, 0.0, 0.9, 1.5, eos)
rp2_vt_0_9 = RiemannProblem(U_left_vt_0_9, U_right_sod_modified2)
display(rp2_vt_0_9)
display_png(rp2_vt_0_9)
```

# Reactive case

An interesting question is on the potential impact of this relativistic effect on detonations and deflagrations. Let us set up a reference reactive problem and add tangential velocities:

```python
eos_reactive = eos_defns.eos_gamma_law_react(5.0/3.0, 0.25, 1.0, 1.0, eos)
U_reactive = State(1.0, -0.5, 0.0, 1.5, eos_reactive)
U_burnt = State(0.125, 0.0, 0., 1.2, eos)
rp_reactive_base = RiemannProblem(U_reactive, U_burnt)
display(rp_reactive_base)
display_png(rp_reactive_base)
```

We see that this is a deflagration-shock problem.

Now add tangential velocities to the unburnt material:

```python
U_reactive_vt_0_9 = State(1.0, -0.5, 0.8, 1.5, eos_reactive)
rp_reactive_vt_reactive = RiemannProblem(U_reactive_vt_0_9, U_burnt)
display(rp_reactive_vt_reactive)
display_png(rp_reactive_vt_reactive)
```

Again, by adding tangential velocities to the right state the shock has become a rarefaction.


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Impulse Response

The response $y(t)$ of a linear time-invariant (LTI) system $\mathcal{H}$ to an arbitrary input signal $x(t)$ is derived in the following. The input signal can be represented as an integral when applying the [sifting-property of the Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse)

\begin{equation}
x(t) = \int_{-\infty}^{\infty} x(\tau) \cdot \delta(t-\tau) \; d \tau
\end{equation}

Introducing above relation for the the input signal $x(t)$ into the output signal $y(t) = \mathcal{H} \{ x(t) \}$ of the system yields

\begin{equation}
y(t) = \mathcal{H} \left\{ \int_{-\infty}^{\infty} x(\tau) \cdot \delta(t-\tau) \; d \tau \right\}
\end{equation}

where $\mathcal{H} \{ \cdot \}$ denotes the system response operator. The integration and system response operator can be exchanged under the assumption that the system is linear

\begin{equation}
y(t) = \int_{-\infty}^{\infty}  x(\tau) \cdot \mathcal{H} \left\{ \delta(t-\tau) \right\} \; d \tau
\end{equation}

where $\mathcal{H} \{\cdot\}$ was only applied to the Dirac impulse, since $x(\tau)$ can be regarded as constant factor with respect to the time $t$. It becomes evident that the response of a system to a Dirac impulse plays an important role in the calculation of the output signal for arbitrary input signals.

The response of a system to a Dirac impulse as input signal is denoted as [*impulse response*](https://en.wikipedia.org/wiki/Impulse_response). It is defined as

\begin{equation}
h(t) = \mathcal{H} \left\{ \delta(t) \right\}
\end{equation}

If the system is time-invariant, the response to a shifted Dirac impulse is $\mathcal{H} \left\{ \delta(t-\tau) \right\} = h(t-\tau)$. Hence, for an LTI system we finally get

\begin{equation}
y(t) = \int_{-\infty}^{\infty}  x(\tau) \cdot h(t-\tau) \; d \tau
\end{equation}

Due to its relevance in the theory of LTI systems, this operation is explicitly termed as [*convolution*](https://en.wikipedia.org/wiki/Convolution). It is commonly abbreviated by $*$, hence for above integral we get $y(t) = x(t) * h(t)$. In some books the mathematically more precise nomenclature $y(t) = (x*h)(t)$ is used, since $*$ is the operator acting on the two signals $x$ and $h$ with regard to time $t$.

In can be concluded that the properties of an LTI system are entirely characterized by its impulse response. The response $y(t)$ of a system to an arbitrary input signal $x(t)$ is given by the convolution of the input signal $x(t)$ with its impulse response $h(t)$.

**Example**

The following example considers an LTI system whose relation between input $x(t)$ and output $y(t)$ is given by an ordinary differential equation (ODE) with constant coefficients

\begin{equation}
y(t) + \frac{d}{dt} y(t) = x(t)
\end{equation}

The system response is computed for the input signal $x(t) = e^{- 2 t} \cdot \epsilon(t)$ by

1. explicitly solving the ODE and by
2. computing the impulse response $h(t)$ and convolution with the input signal.

The solution should fulfill the initial conditions $y(t)\big\vert_{t = 0-} = 0$ and $\frac{d}{dt}y(t)\big\vert_{t = 0-} = 0$ due to causality.

First the ODE is defined in `SymPy`

```python
import sympy as sym
sym.init_printing()

t = sym.symbols('t', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(y + y.diff(t), x)
ode
```

The ODE is solved for the given input signal in order to calculate the output signal. The integration constant is calculated such that the solution fulfills the initial conditions

```python
solution = sym.dsolve(ode.subs(x, sym.exp(-2*t)*sym.Heaviside(t)))
integration_constants = sym.solve(
    (solution.rhs.limit(t, 0, '-'), solution.rhs.diff(t).limit(t, 0, '-')), 'C1')
y1 = solution.subs(integration_constants)
y1
```

Lets plot the output signal derived by explicit solution of the ODE

```python
sym.plot(y1.rhs, (t, -1, 10), ylabel=r'$y(t)$');
```

The impulse response $h(t)$ is computed by solving the ODE for a Dirac impulse as input signal, $x(t) = \delta(t)$

```python
h = sym.Function('h')(t)
solution2 = sym.dsolve(ode.subs(x, sym.DiracDelta(t)).subs(y, h))
integration_constants = sym.solve((solution2.rhs.limit(
    t, 0, '-'), solution2.rhs.diff(t).limit(t, 0, '-')), 'C1')
h = solution2.subs(integration_constants)
h
```

Lets plot the impulse response $h(t)$ of the LTI system

```python
sym.plot(h.rhs, (t, -1, 10), ylabel=r'$h(t)$');
```

As alternative to the explicit solution of the ODE, the system response is computed by evaluating the convolution $y(t) = x(t) * h(t)$. Since `SymPy` cannot handle the Heaviside function properly in integrands, the convolution integral is first simplified. Both the input signal $x(t)$ and the impulse response $h(t)$ are causal signals. Hence, the convolution integral degenerates to

\begin{equation}
y(t) = \int_{0}^{t} x(\tau) \cdot h(t - \tau) \; d\tau
\end{equation}

for $t \geq 0$. Note that $y(t) = 0$ for $t<0$.

```python
tau = sym.symbols('tau', real=True)

y2 = sym.integrate(sym.exp(-2*tau) * h.rhs.subs(t, t-tau), (tau, 0, t))
y2
```

Lets plot the output signal derived by evaluation of the convolution

```python
sym.plot(y2, (t, -1, 10), ylabel=r'$y(t)$');
```

**Exercise**

* Compare the output signal derived by explicit solution of the ODE with the signal derived by convolution. Are both equal?
* Check if the impulse response $h(t)$ is a solution of the ODE by manual calculation. Hint $\frac{d}{dt} \epsilon(t) = \delta(t)$.
* Check the solution of the convolution integral by manual calculation including the Heaviside functions.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


```python
from decodes.core import *
from decodes.io.jupyter_out import JupyterOut
out = JupyterOut.unit_square( )
```

    http://decod.es/	v0.2.3
    io loaded

# Vector Length and Direction
The **length** of a vector $\vec{v}$ is denoted mathematically as $\lvert \vec{v}\rvert$. Elementary geometry, in particular Pythagoras’ theorem, gives us the length of a vector in two dimensions $\vec{v} = (x,y)$ as the hypotenuse of a right triangle.

\begin{align}
\lvert \vec{v}\rvert = \sqrt{x^2+y^2}
\end{align}

In $\mathbb{R}^3$, the length of a vector $\vec{v} = (x,y,z)$ is similarly the result of two applications of Pythagoras and is given by:

\begin{align}
\lvert \vec{v}\rvert = \sqrt{x^2+y^2+z^2}
\end{align}

### Vector Length

```python
"""
Vector Length
Returns the length and the squared length of this vector.
"""
@property
def length(self):
    return math.sqrt(self.x * self.x + self.y * self.y + self.z * self.z)

@property
def length2(self):
    return self.x ** 2 + self.y ** 2 + self.z ** 2

```

```python
"""
Vector Length Setter
Sets the length of a vector, working backwards to determine
the scale factor to set the appropriate x,y, and z coordinates.
"""
@length.setter
def length(self,value):
    factor = value / self.length
    self.x *= factor
    self.y *= factor
    self.z *= factor
```

### Vector Normalization
In the event that the direction of a vector is the primary concern, it is useful to be able to set the length to some convenient value. Appropriately, a **unit vector** is defined as a vector with a length of exactly one unit, and the act of scaling a vector such that its length is exactly $1.0$ is called **normalization**. Mathematically, a unit vector in the direction of a vector $\vec{v}$ is given by dividing the vector by its length:

\begin{align}
\vec{u} = \frac{\vec{v} }{ \lvert \vec{v}\rvert }
\end{align}

```python
"""
Normalizing a Vector
Returns a vector in the same direction as this vector
but scaled to given length
"""
def normalized(self, length=1.0):
    factor = length / self.length
    return Vec(self.x * factor, self.y * factor, self.z * factor)

```

### Test for Perpendicularity

Normalization gives a simple test to check when two vectors are parallel, but what about a test to check whether two vectors are perpendicular? Two vectors $\vec{v_{1}}$ and $\vec{v_{2}}$ are perpendicular if and only if the adjacent diagram is satisfied which, invoking the ever-useful Pythagoras’ theorem, corresponds to the following identity.

\begin{align}
\lvert \vec{v_{1}}\rvert^2 + \lvert \vec{v_{2}}\rvert^2 = \lvert \vec{v_{1}} - \vec{v_{2}}\rvert^2
\end{align}

We can expand out this equation in coordinates to get:

\begin{align}
x_{1}^2 + y_{1}^2 + z_{1}^2 + x_{2}^2 + y_{2}^2 + z_{2}^2 = (x_{1}-x_{2})^2 + (y_{1}-y_{2})^2 + (z_{1}-z_{2})^2
\end{align}

The left hand side cancels with part of the right hand side, and what is left is the following identity.

\begin{align}
x_{1}x_{2} + y_{1}y_{2} + z_{1}z_{2} = 0
\end{align}

This combination of terms on the left hand side is so prevalent in the geometry of three dimensional space that it has a special name, the dot product. This operation is one kind of multiplication of two vectors and is denoted using the dot symbol. The check for perpendicularity can then be succinctly expressed as:

\begin{align}
\vec{v_{1}} \bullet \vec{v_{2}} = 0
\end{align}

```python

```

103 - Points, Vectors, and Coordinate Systems/085 - Vector Length and Direction.ipynb


# 线性回归

- 解决回归问题
- 思想简单，实现容易
- 许多强大的非线性模型的基础
- 结果具有很好的可解释性
- 蕴含机器学习中的很多重要思想

线性回归是利用数理统计中回归分析，来确定两种或两种以上变量间相互依赖的定量关系的一种统计分析方

样本特征只有一个的称为简单线性回归

假设我们找到了最佳拟合的直线方程 `y = ax + b`, 则对于每一个样本点 $x^{(i)}$, 根据直线方程，预测值为$\hat y^{(i)} = ax^{(i)} + b$, 真实值为：$y^{(i)}$

表达 $y^{(i)}$ 和 $\hat y^{(i)}$ 的差距：$ (y^{(i)} - \hat y^{(i)})^2 $

对于所有的样本得
$$\sum_{i=1}^m(y^{(i)} - \hat y^{(i)})^2$$
带入$\hat y^{(i)} = ax^{(i)} + b$，可得
$$\sum_{i=1}^m(y^{(i)} - ax^{(i)} - b)^2$$

目标转换为找到 a 和 b 使得上式(损失函数)尽可能的小

一类机器学习算法的基本思路：
通过分析问题，确定问题的损失函数或者效用函数;
通过最优化损失函数或者效用函数，获得机器学习的模型；
这类机器学习算法包括 线性回归、多项式回归、逻辑回归、SVM、神经网络等

## 最小二乘法问题

最小化误差的平方

假设$J(a, b) = \sum_{i=1}^m(y^{(i)} - ax^{(i)} - b)^2$, 要获得该函数的极值，即对该函数的各个未知分量求导，使其导数为0。即：

$\frac {\partial J(a, b)}{\partial a} = 0$，
$\frac {\partial J(a, b)}{\partial b} = 0$

$
\begin{align}
\frac {\partial J(a, b)}{\partial b} =
 & \sum_{i=1}^m2(y^{(i)} - ax^{(i)} - b)(-1) = 0 \\
 & \sum_{i=1}^m(y^{(i)} - ax^{(i)} - b) = 0 \\
 & \sum_{i=1}^my^{(i)} - a\sum_{i=1}^mx^{(i)} - \sum_{i=1}^mb = 0\\
 & \sum_{i=1}^my^{(i)} - a\sum_{i=1}^mx^{(i)} - mb = 0\\
 & \sum_{i=1}^my^{(i)} - a\sum_{i=1}^mx^{(i)} = mb \Rightarrow b = \bar y - a\bar x (\bar x和\bar y分别为x和y的均值)\\
\end{align}
$

$
\begin{align}
\frac {\partial J(a, b)}{\partial a} =
 & \sum_{i=1}^m2(y^{(i)} - ax^{(i)} - b)(-x^{(i)}) = 0 \\
 & \sum_{i=1}^m(y^{(i)} - ax^{(i)} - b)x^{(i)} = 0 \\
 & \sum_{i=1}^m(y^{(i)} - ax^{(i)} - \bar y + a\bar x)x^{(i)} = 0\\
 & \sum_{i=1}^m(x^{(i)}y^{(i)} - a(x^{(i)})^2 - x^{(i)}\bar y + a\bar xx^{(i)}) = 0\\
 & \sum_{i=1}^m(x^{(i)}y^{(i)} - x^{(i)}\bar y  - a(x^{(i)})^2 + a\bar xx^{(i)}) = 0\\
 & \sum_{i=1}^m(x^{(i)}y^{(i)} - x^{(i)}\bar y) - \sum_{i=1}^m(a(x^{(i)})^2 - a\bar xx^{(i)}) = 0\\
 & \sum_{i=1}^m(x^{(i)}y^{(i)} - x^{(i)}\bar y) - a\sum_{i=1}^m((x^{(i)})^2 - \bar xx^{(i)}) = 0\\
 & a = \frac{\sum_{i=1}^m(x^{(i)}y^{(i)} - x^{(i)}\bar y)}{\sum_{i=1}^m((x^{(i)})^2 - \bar xx^{(i)})}
\end{align}
$

对于公式$\sum_{i=1}^mx^{(i)}\bar y$ 可以进一步推导

$
\begin{align}
\sum_{i=1}^mx^{(i)}\bar y
 & = \bar y\sum_{i=1}^mx^{(i)} \\
 & = m\bar y\bar x = \sum_{i=1}^m\bar x\bar y\\
 & = \bar x \sum_{i=1}^my^{(i)}\\
 & = \sum_{i=1}^m\bar xy^{(i)} \\
\end{align}
$

带入a中可得到

$
\begin{align}
a = \frac{\sum_{i=1}^m(x^{(i)}y^{(i)} - x^{(i)}\bar y)- \bar xy^{(i)} + \bar x \bar y)}{\sum_{i=1}^m((x^{(i)})^2 - \bar xx^{(i)} - \bar xx^{(i)} + \bar x^2)}
 & = \frac{\sum_{i=1}^m(x^{(i)} - \bar x)(y^{(i)} - \bar y)}{\sum_{i=1}^m(x^{(i)} - \bar x)^2}
\end{align}
$

总结: 通过上面的推导可以得到a和b的值
$$a = \frac{\sum_{i=1}^m(x^{(i)} - \bar x)(y^{(i)} - \bar y)}{\sum_{i=1}^m(x^{(i)} - \bar x)^2}, b = \bar y - a\bar x $$

## 实现简单线性回归算法

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
x_train = np.array([1., 2., 3., 4., 5.])
y_train = np.array([1., 3., 2., 3., 5.])
```

```python
plt.scatter(x_train, y_train)
plt.axis([0, 6, 0, 6])
plt.show()
```

```python
x_mean = np.mean(x_train)
y_mean = np.mean(y_train)
```

```python
# 求a的值
num = 0.0 # 分母
d = 0.0 # 分子
for x_i, y_i in zip(x_train, y_train):
    num += (x_i - x_mean) * (y_i - y_mean)
    d += (x_i - x_mean) ** 2
```

```python
a = num / d
b = y_mean - a * x_mean
```

```python
a
```

    0.80000000000000004

```python
b
```

    0.39999999999999947

```python
# 使用求出的a和b的值，预测x_train
y_hat = a * x_train + b
```

```python
plt.scatter(x_train, y_train)
plt.plot(x_train, y_hat, color='r')
plt.axis([0, 6, 0, 6])
plt.show()
```

```python
x_predict = 6
y_predict = a * x_predict + b
y_predict
```

    5.2000000000000002

## 向量化实现简单线性回归

上述中求a的值时，对于分子和分母使用了for循环操作，效率低下。仔细观察后，可以将a的表达式转换为向量操作

这里简单介绍下两个向量之间的点乘：

向量的点乘,也叫向量的内积、数量积，对两个向量执行点乘运算，就是对这两个向量对应位一一相乘之后求和的操作，点乘的结果是一个标量

假设存在向量$\vec a = [a_1, a_2,...a_n]$, 存在向量$\vec b = [b_1, b_2,...b+n]$,则
$$
\vec a\cdot \vec b = a_1b_1 + a_2b_2+...+a_nb_n
$$

则对于
$$a = \frac{\sum_{i=1}^m(x^{(i)} - \bar x)(y^{(i)} - \bar y)}{\sum_{i=1}^m(x^{(i)} - \bar x)^2}$$

可以用下面的代码求解其值

```python
num = (x_train - x_mean).dot(y_train - y_mean)
d = (x_train - x_mean).dot(x_train - x_mean)
a = num / d
a
```

    0.80000000000000004

#### 测试向量化的性能

```python
m = 1000000
big_x = np.random.random(size=m)
big_y = big_x * 2.0 + 3.0 + np.random.normal(size=m)
```

```python
x_mean = np.mean(big_x)
y_mean = np.mean(big_y)
```

```python
%%time
num = 0.0
d = 0.0
for x_i, y_i in zip(big_x, big_y):
    num += (x_i - x_mean) * (y_i - y_mean)
    d += (x_i - x_mean) ** 2
print(num)
print(d)
```

    167019.255941
    83386.6447275
    CPU times: user 1.04 s, sys: 7 ms, total: 1.04 s
    Wall time: 1.06 s

```python
%%time
num = (big_x - x_mean).dot(big_y - y_mean)
d = (big_x - x_mean).dot(big_x - x_mean)
print(num)
print(d)
```

    167019.255941
    83386.6447275
    CPU times: user 11.4 ms, sys: 7.52 ms, total: 18.9 ms
    Wall time: 17.1 ms

```python
a = num / d
b = y_mean - a * x_mean
print(a)
print(b)
```

    2.00294971079
    2.99821712073


```python
from logicqubit.logic import *
from cmath import *
import numpy as np
import sympy as sp
from scipy.linalg import expm, sinm, cosm, logm, sqrtm
```

```python
def qft(qr):
    for i in range(len(qr)):
        for j in range(i):
            qr[i].CU1(qr[j], pi/float(2**(i-j)))
        qr[i].H()

def iqft(qr):
    for i in range(len(qr)):
        for j in range(i):
            qr[i].CU1(qr[j], -pi/float(2**(i-j)))
        qr[i].H()

def swap(s1, s2):
    s2.CX(s1)
    s1.CX(s2)
    s2.CX(s1)
```

```python
D = np.matrix('1.5707963267948966 0; 0 3.141592653589793')
P = np.matrix('1 0; 0 1')/sqrt(2)

A = np.matrix('3 1; 1 3')
#A = P*D*P.I
#A_ad = A.getH()
t0 = 2*pi

#U = np.matrix(expm(-1j*pi*A))
U_A_div2 = np.matrix(expm(1j*A*t0/2))
U_A_div4 = np.matrix(expm(1j*A*t0/4))
print(U_A_div2.tolist())
```

    [[(1.0000000000000004+4.440892098500627e-16j), 0j], [0j, (1.0000000000000004+4.440892098500627e-16j)]]

```python
b = np.array([1,0])
x = np.linalg.solve(A, b)
print(x)
```

    [ 0.375 -0.125]

```python
logicQuBit  = LogicQuBit(4)

a = Qubit()
c1 = Qubit()
c2 = Qubit()
b = Qubit()
r = 2

c1.H()
c2.H()

#b.X()
#b.H()

# phase estimation
b.CU(c2, U_A_div4.tolist())
b.CU(c1, U_A_div2.tolist())

# iqft
swap(c1,c2)
c2.H()
c2.CS(c1, True)
c1.H()
swap(c1,c2)

a.CRY(c1, pi/2**r)
a.CRY(c2, 2*pi/2**r)

swap(c1,c2)
# qft
c1.H()
c2.CS(c1)
c2.H()
swap(c1,c2)

# reverse phase estimation
b.CU(c1, U_A_div4.conj().T.tolist())
b.CU(c2, U_A_div2.conj().T.tolist())

c1.H()
c2.H()
c2.PrintState()
c2.getDictPsi()
```

$\displaystyle \begin{pmatrix}0.5-2.78426e-17j \\ 0.5-2.76685e-17j \\ -2.23706e-16+7.5546e-17j \\ -1.11856e-17+1.46499e-16j \\ -2.45038e-16-4.73355e-17j \\ -2.77556e-17+1.76998e-17j \\ 0.353553+2.78426e-17j \\ -0.353553-1.14097e-17j \\ -1.89288e-17+2.78426e-17j \\ 9.81308e-18-2.78426e-17j \\ -1.26364e-16-3.10767e-17j \\ 6.51042e-17+3.10767e-17j \\ -1.50275e-16+3.776e-18j \\ -1.14967e-17-1.53233e-16j \\ 0.353553-2.78426e-17j \\ -0.353553-6.70949e-17j \end{pmatrix}$

    {'0000': (0.5-2.7842613392665686e-17j),
     '0001': (0.4999999999999999-2.7668537838592144e-17j),
     '0010': (-2.23705968685014e-16+7.554599442488008e-17j),
     '0011': (-1.1185571585378691e-17+1.4649861050015117e-16j),
     '0100': (-2.4503807662796223e-16-4.7335537465165435e-17j),
     '0101': (-2.7755575615628914e-17+1.7699821591488393e-17j),
     '0110': (0.3535533905932736+2.78426133926657e-17j),
     '0111': (-0.35355339059327373-1.1409698074428693e-17j),
     '1000': (-1.8928761761890062e-17+2.7842613392665704e-17j),
     '1001': (9.813077866773592e-18-2.7842613392665717e-17j),
     '1010': (-1.2636372189240907e-16-3.107669770100199e-17j),
     '1011': (6.510420273541852e-17+3.1076697701002e-17j),
     '1100': (-1.5027461392961003e-16+3.7760034294587894e-18j),
     '1101': (-1.149673585146546e-17-1.5323324243891858e-16j),
     '1110': (0.3535533905932737-2.7842613392665674e-17j),
     '1111': (-0.3535533905932737-6.709492485976009e-17j)}

```python
res_a = logicQuBit.Measure_One(a)
print(res_a)
res = logicQuBit.Measure([b])
logicQuBit.Plot()
print(res)
```

```python
def Text2List(table):
    list = [int(i, base=2) for i in table]
    size = len(table[0])
    tmp = sorted(list, key=int, reverse=False)  # values in ascending order
    result = [[int(bin(j)[2:].zfill(size)[i]) for i in range(size)] for j in tmp]
    return result

def BinList(n):
    blist = []
    for i in iter(range(2 ** n)):
        b = bin(i)[2:].zfill(n)  # value in binary, ex: i=1, n=4 -> '0001'
        blist.append(b)
    return Text2List(blist)

#size=3
#blist = [value for i,value in enumerate(BinList(size)) if res[i]>0.1]
#results = [sum([value/2**(i+1) for i,value in enumerate(j)]) for j in blist]
#print(results)
```

```python
# todos possíveis valores
size = 2
blist = [[i >> bl & 0x1 for bl in range(size)][::-1] for i in range(2**size)]
results = [2*pi*sum([value/2**(i+1) for i,value in enumerate(j)]) for j in blist]
results
```

    [0.0, 1.5707963267948966, 3.141592653589793, 4.71238898038469]

```python

```


```python
from epipack import SymbolicEpiModel
from epipack.interactive import InteractiveIntegrator, Range, LogRange
import sympy
import numpy as np

%matplotlib widget

S, I, R, R0, tau, omega = sympy.symbols("S I R R_0 tau omega")

I0 = 0.01
model = SymbolicEpiModel([S,I,R])\
         .set_processes([
                (S, I, R0/tau, I, I),
                (I, 1/tau, R),
                (R, omega, S),
            ])\
         .set_initial_conditions({S:1-I0, I:I0})

parameters = {
    R0: LogRange(min=0.1,max=10,step_count=1000),
    tau: Range(min=0.1,max=10,value=8.0),
    omega: 1/14
}
```

```python
t = np.logspace(-3,2,1000)
InteractiveIntegrator(model, parameters, t, figsize=(4,4))
```

    InteractiveIntegrator(children=(VBox(children=(FloatLogSlider(value=1.0, continuous_update=False, description=…

```python

```


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Discussion: Week 9

In this Notebook, we will investigate how the degree to which the central bank responds to changes inflation affects how exogenous shocks to demand or inflation propagate.

## The New-Keynesian Model

The most basic version of the New-Keynesian Model can be expressed as:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
i_{t} & = \bar{r} + \pi^T + \phi_{\pi}\big(\pi_t - \pi^T\big) + \phi_{y}\big(y_t - \bar{y}\big)\\
\pi_t -\pi^T & = \beta \left( E_t\pi_{t+1} - \pi^T\right)  + \kappa (y_t -\bar{y})+ u_t,
\end{align}

where: $y_t$ is (log) output, $r_t$ is the real interest rate, $i_t$ is the nominal interest rate, $\pi_t$ is the rate of inflation between periods $t-1$ and $t$, $\bar{r}$ is the long-run average real interest rate or the *natural rate of interest*, $\beta$ is the household's subjective discount factor, and $\pi^T$ is the central bank's inflation target. The coefficients $\phi_{\pi}$ and $\phi_{y}$ reflect the degree of intensity to which the central bank *endogenously* adjusts the nominal interest rate in response to movements in inflation and output.

The variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}
\end{align}

I have removed the monetary policy from the model because today we are going to focus on demand and inflation shocks. We will use the following parameterization:

| $\bar{y}$ | $\beta$ | $\bar{r}$    | $\kappa$ | $\pi^T$ | $\phi_{\pi}$ | $\phi_y$ | $\rho_g$ | $\rho_u$ |
|-----------|---------|--------------|----------|---------|--------------|----------|----------|----------|
| 0         | 0.995   | $-\log\beta$ | 0.1      | 0.02/4  | —            | 0.5/4    | 0.5      | 0.5      |

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series. CELL PROVIDED
parameters = pd.Series()
parameters['y_bar'] = 0
parameters['beta'] = 0.995
parameters['r_bar'] = -np.log(parameters.beta)
parameters['kappa'] = 0.1
parameters['pi_T'] = 0.02/4
parameters['phi_pi'] = 1.5
parameters['phi_y'] = 0.5/4
parameters['rho_g'] = 0.5
parameters['rho_u'] = 0.5

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable.
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero.
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters
    p = parameters

    # Current variables
    cur = variables_current

    # Forward variables
    fwd = variables_forward

    # IS equation
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy
    monetary_policy = p.r_bar + p.pi_T + p.phi_pi*(cur.pi - p.pi_T) + p.phi_y*cur.y - cur.i

    # Phillips curve
    phillips_curve = p.beta*(fwd.pi- p.pi_T) + p.kappa*cur.y + cur.u - (cur.pi-p.pi_T)

    # Demand process
    demand_process = p.rho_g*cur.g - fwd.g

    # Inflation process
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array
    return np.array([
            is_equation,
            fisher_equation,
            monetary_policy,
            phillips_curve,
            demand_process,
            inflation_process
        ])

# Initialize the model into a variable named 'nk_model'
nk_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model
guess = [0,0,0,0.01,0.01,0.01]
nk_model.compute_ss(guess)
```

## Exercise: Demand Shocks and $\phi_{\pi}$

For $\phi_{\pi} = 1.5,2,10,100$, compute impulse responses for output and inflation to a 0.01 unit increase in $g_t$. Simulate a total of 21 periods with the arriving in period 5. Construct a figure with two axes side-by-side with output ploted on the left and inflation plotted on the right.

```python
# Create a variable called 'phi_pi_values' that stores the values for phi_pi provided above.
phi_pi_values = [1.5,2,10,100]

# Create a figure with dimensions 12x4. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create the left axis. PROVIDED
ax1 = fig.add_subplot(1,2,1)
# Create the right axis. PROVIDED
ax2 = fig.add_subplot(1,2,2)

# Set the title of the left axis. PROVIDED
ax1.set_title('Output')
# Set the title of the right axis. PROVIDED
ax2.set_title('Inflation')

# Set the y-axis label of the left axis. PROVIDED
ax1.set_ylabel('Percent')
# Set the y-axis label of the right axis. PROVIDED
ax2.set_ylabel('Percent')

# Construct a for loop that iterates over the values in phi_pi
for phi_pi in phi_pi_values:

    # Set the phi_pi value in nk_model.parameters to the current value of phi_pi
    nk_model.parameters['phi_pi'] = phi_pi

    # Approximate and solve the model. Set log_linear argument to False since the model is already linear
    nk_model.approximate_and_solve(log_linear=False)

    # Compute impulse responses to a one percent shock to demand
    nk_model.impulse(T=21,t0=5,shocks=[0.01,0])

    # Plot output
    (nk_model.irs['e_g']['y']*100).plot(ax=ax1,grid=True,lw=3)

    # Plot inflation impulse response
    (nk_model.irs['e_g']['pi']*400).plot(ax=ax2,grid=True,lw=3,label='$\\phi_{\\pi}='+str(phi_pi)+'$')

# Construct legend. PROVIDED
ax2.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

**Questions**

1. How do the computed impulse responses for output change as $\phi_{\pi}$ increases from 1.5 to 100?
2. How do the computed impulse responses for inflation change as $\phi_{\pi}$ increases from 1.5 to 100?
3. Does the central banker face a trade-off between stabilizing output fluctuations or stabilizing inflation fluctuations in the presence of a demand shock?

**Answers**

1. The magnitude of the output response declines as $\phi_{\pi}$ increases. <!-- answer -->
1. The magnitude of the inflation response declines as $\phi_{\pi}$ increases. <!-- answer -->
3. No. By increasing the nominal interest rate more aggressively in response to rising inflation, the central bank can stabilize *both* output and inflation against a demand shock. <!-- answer -->

## Exercise: Inflation Shocks and $\phi_{\pi}$

For $\phi_{\pi} = 1.5,2,10,100$, compute impulse responses for output and inflation to a 0.01/4 unit increase in $u_t$. Simulate a total of 21 periods with the arriving in period 5. Construct a figure with two axes side-by-side with output ploted on the left and inflation plotted on the right.

```python
# Create a variable called 'phi_pi_values' that stores the values for phi_pi provided above.
phi_pi_values = [1.5,2,10,100]

# Create a figure with dimensions 12x4. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create the left axis. PROVIDED
ax1 = fig.add_subplot(1,2,1)
# Create the right axis. PROVIDED
ax2 = fig.add_subplot(1,2,2)

# Set the title of the left axis. PROVIDED
ax1.set_title('Output')
# Set the title of the right axis. PROVIDED
ax2.set_title('Inflation')

# Set the y-axis label of the left axis. PROVIDED
ax1.set_ylabel('Percent')
# Set the y-axis label of the right axis. PROVIDED
ax2.set_ylabel('Percent')

# Construct a for loop that iterates over the values in phi_pi
for phi_pi in phi_pi_values:

    # Set the phi_pi value in nk_model.parameters to the current value of phi_pi
    nk_model.parameters['phi_pi'] = phi_pi

    # Approximate and solve the model. Set log_linear argument to False since the model is already linear
    nk_model.approximate_and_solve(log_linear=False)

    # Compute impulse responses to a one percent shock to demand
    nk_model.impulse(T=21,t0=5,shocks=[0.0,0.01/4])

    # Plot output
    (nk_model.irs['e_u']['y']*100).plot(ax=ax1,grid=True,lw=3)

    # Plot inflation impulse response
    (nk_model.irs['e_u']['pi']*400).plot(ax=ax2,grid=True,lw=3,label='$\\phi_{\\pi}='+str(phi_pi)+'$')

# Construct legend. PROVIDED
ax2.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

**Questions**

1. How do the computed impulse responses for output change as $\phi_{\pi}$ increases from 1.5 to 100?
2. How do the computed impulse responses for inflation change as $\phi_{\pi}$ increases from 1.5 to 100?
3. Does the central banker face a trade-off between stabilizing output fluctuations or stabilizing inflation fluctuations in the presence of an inflation shock?

**Answers**

1. The magnitude of the output response increases as $\phi_{\pi}$ increases. <!-- answer -->
1. The magnitude of the inflation response declines as $\phi_{\pi}$ increases. <!-- answer -->
3. Yes. By increasing the nominal interest rate more aggressively in response to rising inflation, the central bank improves inflation stability at the cost of making output more volatile against an inflation shock. <!-- answer -->


# The Harmonic Oscillator Strikes Back

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week we continue our adventures with the harmonic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2} m \omega_0^2 {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

If the equilibrium value of the harmonic oscillator is not zero, then

$$ F=- \omega_0^2 m (x-x_{eq}), \text{ }  V(x) = {1 \over 2} m \omega_0^2 (x-x_{eq})^2$$

## 1. Harmonic oscillator from last time (with some better defined conditions)

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= -m \omega_0^2 (x-x_{eq}) $$

$$ a = - \omega_0^2 (x-x_{eq}) $$

$$ x(t)'' = - \omega_0^2 (x-x_{eq}) $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

This is already solved to remind you how we found these values

```python
import sympy as sym
sym.init_printing()
```

**Note** that this time we define some of the properties of the symbols. Namely, that the frequency is always positive and real and that the positions are always real

```python
omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)
x(t),omega0
```

```python
dfeq=sym.Derivative(x(t),t,2)+omega0**2*(x(t)-xeq)
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

```python
sol,sol.args[0],sol.args[1]
```

**Note** this time we define the initial positions and velocities as real

```python
x0,v0=sym.symbols("x_0,v_0",real=True)
ics=[sym.Eq(sol.args[1].subs(t, 0), x0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

```python
solved_ics=sym.solve(ics)
solved_ics
```

### 1.1 Equation of motion for $x(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

### 1.2 Equation of motion for $p(t)$

```python
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
```

## 2. Time average values for a harmonic oscillator

If we want to understand the average value of a time dependent observable, we need to solve the following integral

$${\left\langle {A(t)} \right\rangle}_t  = \begin{array}{*{20}{c}}
{\lim }\\
{\tau  \to 0}
\end{array}\frac{1}{\tau }\int\limits_0^\tau  {A(t)dt} $$

### 2.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
xfunc=full_sol.args[1]
xavet=(xfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
xavet
```

The computer does not always make the best choices the first time. If you treat each sum individually this is not a hard limit to do by hand. The computer is not smart. We can help it by inseting an `expand()` function in the statement

```python
xavet=(xfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
xavet
```

### 2.2 Excercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

```python
# Your code here
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))

tau=sym.symbols("tau",nonnegative=True,real=True)
pfunc=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t)).args[1]
pavet=(pfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
pavet
```

### 2.3 Exercise: Calculate the average kinetic energy of a harmonic oscillator

```python
# Your code here
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
KE=sym.Function("KE")
sym.Eq(KE(t),(m*sol.args[1].subs(solved_ics[0]).diff(t))**2/(2*m))

```

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
KEfunc=sym.Eq(KE(t),(m*sol.args[1].subs(solved_ics[0]).diff(t))**2/(2*m)).args[1]
KEavet=(KEfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
KEavet
```

```python
KEavet=(KEfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
KEavet
```

## 3.  Ensemble (Thermodynamic) Average values for a harmonic oscillator

If we want to understand the thermodynamics ensemble average value of an observable, we need to solve the following integral.

$${\left\langle {A(t)} \right\rangle}_{T}  = \frac{\int{A e^{-\beta H}dqdp}}{\int{e^{-\beta H}dqdp} } $$

You can think of this as a Temperature average instead of a time average.

Here $\beta=\frac{1}{k_B T}$ and the classical Hamiltonian, $H$ is

$$ H = \frac{p^2}{2 m} + V(q)$$

**Note** that the factors of $1/h$ found in the classical partition function cancel out when calculating average values

### 3.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

For a harmonic oscillator with equilibrium value $x_{eq}$, the Hamiltonian is
$$ H = \frac{p^2}{2 m} + \frac{1}{2} m \omega_0 (x-x_{eq})^2 $$

First we will calculate the partition function $\int{e^{-\beta H}dqdp}$

```python
k,T=sym.symbols("k,T",positive=True,nonnegative=True,real=True)
xT,pT=sym.symbols("x_T,p_T",real=True)
ham=sym.Rational(1,2)*(pT)**2/m + sym.Rational(1,2)*m*omega0**2*(xT-xeq)**2
beta=1/(k*T)
bolz=sym.exp(-beta*ham)
z=sym.integrate(bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
z
```

Then we can calculate the numerator $\int{A e^{-\beta H}dqdp}$

```python
numx=sym.integrate(xT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
numx
```

And now the average value

```python
xaveT=numx/z
xaveT
```

### 3.2 Exercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

After calculating the value, explain why you think you got this number

```python
# your code here
nump=sym.integrate(pT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
nump
```

```python
paveT=nump/z
paveT
```

### 3.3 Exercise: Calculate the average kinetic energy

The answer you get here is a well known result related to the energy equipartition theorem

```python
# Your code here
numKE=sym.integrate((pT**2)/(2*m)*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
numKE
```

```python
KEaveT=numKE/z
KEaveT
```

# Back to the lecture

## 4. Exercise Verlet integrators

In this exercise we will write a routine to solve for the equations of motion for a hamonic oscillator.

Plot the positions and momenta (seprate plots) of the harmonic oscillator as a functions of time.

Calculate trajectories using the following methods:
1. Exact solution
2. Simple taylor series expansion
3. Predictor-corrector method
4. Verlet algorithm
5. Leapfrog algorithm
6. Velocity Verlet algorithm

```python
# Your code here
```


### Artificial Neural Net Demonstration and Exercise

#### Goal

Reservoir property estimation from seismic information with artificial neural nets.

#### Description

Here's a simple, documented workflow, demonstration of reservoir property estimation from seismic information with artificial neural nets for subsurface modeling workflows. This should help you get started with building subsurface models that integrate machine learning to integrate data sources.

#### Artificial Neural Networks

Machine learning method for supervised learning for classification and regression analysis.  Here are some key aspects of support vector machines.

Machine learning method for supervised learning for classification and regression analysis.  Here are some key aspects of support vector machines.

**Basic Design** *"...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs."* Caudill (1989).

**Nature-inspire Computing** based on the neuronal structure in the brain, including many interconnected simple, processing units, known as nodes that are capable of complicated emergent pattern detection due to a large number of nodes and interconnectivity.

**Training and Testing** just like and other predictive model (e.g. linear regression, decision trees and support vector machines) we perform training to fit parameters and testing to tune hyperparameters.  Here we observe the error with training and testing datasets, but do not demonstrate tuning of the hyperparameters.

**Parameters** are the weights applied to each connection and a bias term applied to each node.  For a single node in an artificial neural network, this includes the slope terms, $\beta_i$, and the bias term, $\beta_{0}$.

\begin{equation}
Y = \sum_{i=1}^m \beta_i X + \beta_0
\end{equation}

it can be seen that the number of parameters increases rapidly as we increase the number of nodes and the connectivity between the nodes.

**Layers** the typical artificial neural net is structured with an **input layer**, with one node for each $m$ predictor feature, $X_1,\ldots,X_m$. There is an **ouput layer**, with one node for each $r$ response feature, $Y_1,\ldots,Y_r$.  There may be one or more layers of nodes between the input and output layers, known as **hidden layer(s)**.

**Connections** are the linkages between the nodes in adjacent layers.  For example, in a fully connected artificial neural network, all the input nodes are connected to all of the nodes in the first layer, then all of the nodes in the first layer are connected to the next layer and so forth. Each connection includes a weight parameter as indicated above.

**Nodes** receive the weighted signal from the connected previous layer nodes, sum and then apply this result to the **activation** function in the node. Some example activation functions include:

* **Binary** the node fires or not.  This is represented by a Heaviside step function.

* **Identify** the input is passed to the output $f(x) = x$

* **Linear** the node passes a signal that increases linearly with the weighted input.

* **Logistic** also known as sigmoid or soft step $f(x) = \frac{1}{1+e^{-x}}$

the node output is the nonlinear activiation function applied to the linearly weighted inputs. This is fed to all nodes in the next layer.

**Training Cycles** - the presentation of a batch of data, forward application of the current prediction model to make estimates, calculation of error and then backpropagation of error to correct the artificial neural network parameters to reduce the error over all of the batches.

**Batch** is the set of training data for each training cycle of forward prediction and back propagation of error, drawn to train for each iteration. There is a trade-off, a larger batch results in more computational time per iteration, but a more accurate estimate of the error to adjust the weights.  Smaller batches result in a nosier estimate of the error, but faster epochs, this results in faster learning and even possibly more robust models.

**Epochs** - is a set of training cycles, batches covering all available training data.

**Local Minimums** - if one calculated the error hypersurface over the range of model parameters it would be hyparabolic, there is a global minimium error solution.  But this error hyper surface is rough and it is possible to be stuck in a local minimum. **Learning Rate** and **Mommentum** coefficients are introduced to avoid getting stuck in local minimums.

* **Mommentum** is a hyperparameter to control the use of information from the weight update over the last epoch for consideration in the current epoch.  This can be accomplished with an update vector, $v_i$, a mommentum parameter, $\alpha$, to calculate the current weight update, $v_{i+1}$ given the new update $\theta_{i+1}$.

\begin{equation}
v_{i+1} = \alpha v_i + \theta_{i+1}
\end{equation}

* **Learning Rate** is a hyperparameter that controls the adjustment of the weights in response to the gradient indicated by backpropagation of error

##### Applications to subsurface modeling

We demonstrate the estimation of normal score transformed porosity from depth.  This would be useful for building a vertical trend model.

* modeling the complicated relationship between porosity and depth.

#### Limitations of Neural Network Estimation

Since we demonstrate the use of an artificial neural network to estimate porosity from sparsely sampled data over depth, we should comment on limitations of our artificial neural networks for this estimation problem:

* does not honor the well data

* does not honor the histogram of the data

* does not honor spatial correlation

* does not honor the multivariate relationship

* generally low interpretability models

* requires a large number of data for effective training

* high model complexity with high model variance

#### Workflow Goals

Learn the basics of machine learning in python to predict subsurface features. This includes:

* Loading and visualizing sample data
* Trying out neural nets

#### Import Required Packages

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
import seaborn as sns                     # for plotting
import warnings                           # supress warnings from seaborn pairplot
from sklearn.model_selection import train_test_split # train / test DatFrame split

from intake import cat
```

    /Users/john/miniconda3/envs/daytum-eds/lib/python3.7/site-packages/intake/source/discovery.py:137: FutureWarning: The drivers ['sql_cat', 'sql', 'sql_auto', 'sql_manual'] do not specify entry_points and were only discovered via a package scan. This may break in a future release of intake. The packages should be updated.
      FutureWarning)

We will also need the following packages to train and test our artificial neural nets:

* Tensorflow - open source machine learning

* Keras - high level application programing interface (API) to build and train models

* This workflow was designed with tensorflow version 2.1.

To check your current version of tensorflow you could run the next block of code.

```python
import tensorflow as tf
tf.__version__                            # check the installed version of tensorflow
```

    '2.0.0'

Let's import all of the tensorflow and keras methods that we will need in our workflow.

```python
import tensorflow as tf
from keras.models import Sequential, load_model
from keras.layers.core import Dense, Dropout, Activation
from keras.utils import np_utils
import keras
from tensorflow.python.keras import backend as k
```

    Using TensorFlow backend.

#### Loading Data

* Depth ($m$)
* Normal Score Porosity

It is common to transform properties to standard normal for geostatistical workflows.

```python
df2 = cat.porosity_1D.read()                                 # display first 5 samples in the table as a preview
df2.head()                                                   # we could also use this command for a table preview
```

      Depth
      Nporosity

      0
      0.25
      -1.37

      1
      0.50
      -2.08

      2
      0.75
      -1.60

      3
      1.00
      -1.16

      4
      1.25
      -0.24

#### Data Normalization

We must normalize the features before we apply them in an artificial neural network model. These are the motivations for this normalization:

* remove impact of scale of different types of data (i.e., depth varies between $[0,10]$, but porosity only varies between $[-3,3.0]$).

* activation functions in artificial neural networks are designed to be more sensive to values of nodes closer to 0.0 (i.e., results in higher gradient and improves backpropagation in training)

Let's normalize each feature.

* We apply the min max normalization by-hand to force both the predictor and response features to be bound $[-1,1]$.

* It is easy to backtransform given we keep track of the original min and max values

```python
depth_min = df2['Depth'].values.min(); depth_max = df2['Depth'].values.max()
Npor_min = df2['Nporosity'].values.min(); Npor_max = df2['Nporosity'].values.max()
df2['norm_Depth'] = (df2['Depth'] - depth_min)/(depth_max - depth_min) * 2 - 1
df2['norm_Nporosity'] = (df2['Nporosity'] - Npor_min)/(Npor_max - Npor_min) * 2 - 1
df2.head()
```

      Depth
      Nporosity
      norm_Depth
      norm_Nporosity

      0
      0.25
      -1.37
      -1.000000
      -0.679458

      1
      0.50
      -2.08
      -0.948718
      -1.000000

      2
      0.75
      -1.60
      -0.897436
      -0.783296

      3
      1.00
      -1.16
      -0.846154
      -0.584650

      4
      1.25
      -0.24
      -0.794872
      -0.169300

It is also a good idea to check the summary statistics.

* All normalized features should now range from -1.0 to 1.0

```python
df2.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Depth
      40.0
      5.125000
      2.922613
      0.25
      2.687500
      5.125000e+00
      7.562500
      10.00

      Nporosity
      40.0
      0.024000
      0.989107
      -2.08
      -0.757500
      1.400000e-01
      0.742500
      2.35

      norm_Depth
      40.0
      0.000000
      0.599510
      -1.00
      -0.500000
      -5.551115e-17
      0.500000
      1.00

      norm_Nporosity
      40.0
      -0.050113
      0.446549
      -1.00
      -0.402935
      2.257336e-03
      0.274266
      1.00

#### Separation of Training and Testing Data

We also need to split our data into training / testing datasets so that we:

* can train our artificial neural networks using the training data

* while testing their performance with the withheld testing (validation) data.

```python
X2 = df2.iloc[:,[0,2]]                        # extract the predictor feature - acoustic impedance
y2 = df2.iloc[:,[1,3]]                        # extract the response feature - porosity
X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y2, test_size=0.2, random_state=73073)
```

#### Visualize the Dataset

Let's visualize the train and test data split on a single scatter plot.

* we want to make sure it is fair

* ensure that the test samples are not clustered or too far away from the training data.

We will look at the original data and normalized data, the input to the neural network.

```python
plt.subplot(211)
plt.plot(X2_train['Depth'].values,y2_train['Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=0.4, label = "Train")
plt.plot(X2_test['Depth'].values,y2_test['Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=0.4, label = "Test")
plt.title('Standard Normal Porosity vs. Depth')
plt.xlabel('Z (m)')
plt.ylabel('NPorosity')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)

plt.subplot(212)
plt.plot(X2_train['norm_Depth'].values,y2_train['norm_Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=0.4, label = "Train")
plt.plot(X2_test['norm_Depth'].values,y2_test['norm_Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=0.4, label = "Test")
plt.title('Normalized Standard Normal Porosity vs. Normalized Depth')
plt.xlabel('Normalized Z')
plt.ylabel('Normalized NPorosity')
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.5, wspace=0.2, hspace=0.3)
plt.show()
```

#### Specify the Prediction Locations

Given this training and testing data, let's specify the prediction locations over the range of the observed depths at regularly spaced $nbins$ locations.

```python
# Specify the prediction locations
nbins = 1000
depth_bins = np.linspace(depth_min, depth_max, nbins) # set the bins for prediction
norm_depth_bins = (depth_bins-depth_min)/(depth_max-depth_min)*2-1 # use normalized bins
```

#### Build and Train a Simple Neural Network

For our first model we will build a simple model with:

* 1 predictor feature - depth ($d$)

* 1 response feature - normal score porosity ($N\{\phi\}$)

we will build a model to predict normal score porosity from depth over all locations in our model $\bf{u} \in AOI$.

\begin{equation}
N\{\phi(\bf{u})\} = \hat{f} (d(\bf{u}))
\end{equation}

and use this model to support the prediction of porosity between the wells.

#### Design, Train and Test a Neural Network

In the following code we use keras / tensorflow to:

1. **Design the Network** - we use a fully connected, feed forward neural network with one node in the input and output layers, to receive the normalized depth and output the normalized (normal score) porosity.  We found by trial and error, given the complexity of the dataset, that we required a significant network width (about 500 nodes) and a network depth of atleast 4 hidden layers.

2. **Select the Optimizer** - we selected the adam optimizer (Kingma and Ba, 2015).  This optimizer is computationally efficient and is well suited to problems with noisy data and sparse gradients. It is an extension of stochastic gradient descent with the addition of adaptive gradient algorithm that calculates per-parameter learning rates to improve learning with sparse gradients, and root mean square propopagation that sets the learning rate based on the recent magnitudes of the gradients for each parameter to improve performance with noisy data.  We include stochastic gradient descent for experimentation.

    * we found a learning rate of 0.01 to 0.001 works well
    * we found the rate of decay parameters of $\beta_1=0.9$ and $\beta_2=0.999$ performed well

3. **Compile the Machine** - specify the optimizer, loss function and the metric for model training.

4. **Train the Network** - fit / train the model parameters over a specified number of batch size within a specified number of epochs.  We specify the train and test normalized predictor and response features.

Then we visualize the model in the original units.

```python
# Design the neural network
model_2 = Sequential([
    Dense(1, activation='linear', input_shape=(1,)),                  # input layer
    Dense(50, activation='relu'),
    Dense(50, activation='relu'),
    Dense(50, activation='relu'),
#    Dense(100, activation='relu'),                                    # uncomment these to add hidden layers
#    Dense(100, activation='relu'),
#    Dense(100, activation='relu'),
#    Dense(100, activation='relu'),
    Dense(1, activation='linear'),                                     # output layer
])

# Select the Optimizer
adam = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=0.1, decay=0.0, amsgrad=False) # adam optimizer
#sgd = keras.optimizers.SGD(lr=0.001, momentum=0.0, decay = 0.0, nesterov=False)  # stochastic gradient descent

# Compile the Machine
model_2.compile(optimizer=adam,loss='mse',metrics=['accuracy'])

# Train the Network
hist_2 = model_2.fit(X2_train['norm_Depth'], y2_train['norm_Nporosity'],
    batch_size=5, epochs=1000,
    validation_data=(X2_test['norm_Depth'], y2_test['norm_Nporosity']),verbose = 0)

# Predict with the Network
pred_norm_Nporsity = model_2.predict(np.array(norm_depth_bins)) # predict with our ANN
pred_Nporosity = ((pred_norm_Nporsity + 1)/2*(Npor_max - Npor_min)+Npor_min)

# Plot the Model Predictions
plt.subplot(1,1,1)
plt.plot(depth_bins,pred_Nporosity,'black',linewidth=2)
plt.plot(X2_train['Depth'].values,y2_train['Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=1.0, label = "Train")
plt.plot(X2_test['Depth'].values,y2_test['Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=1.0, label = "Test")
plt.xlabel('Depth (m)')
plt.ylabel('Porosity (faction)')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

#### Evaluation of the Model

For my specified artificial neural network design and optimization parameters I have a very flexible model to fit the data.

* artificial neural networks live up to their designation as **Universal Function Approximators**

Let's check the training curve, loss functions for our model over training and testing datasets.

* **square loss** ($L_2$ loss) is the:

\begin{equation}
L_2 = \sum_{\bf{u}_{\alpha} \in AOI} \left(y(\bf{u}_{\alpha}) - \hat{f}(x_1(\bf{u}_{\alpha}),\ldots,x_m(\bf{u}_{\alpha})\right)
\end{equation}

* this is a measure of the inaccuracy over the available data

We can see the progress of the model over epochs in reduction of training and testing error.

* we can observe that the model matches the training data after about 200 epochs, but continues to improve up the 1,000 epochs

```python
# Plot the Loss vs. Training Epoch
plt.subplot(1,1,1)
plt.plot(hist_2.history['loss'])
plt.plot(hist_2.history['val_loss'])
plt.title('Loss function of Artificial Neural Net Example #2')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')
plt.ylim(0,0.4)
plt.grid()
plt.tight_layout()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

#### Some Observations

We performed a set of experiments and made the following observations that may help you experiment with the design and training of this artificial neural network. Each machine was trained over 1000 epochs with a batch size of 5.

For a 1 $\times$ 100 $\times$ 100 $\times$ 1 neural network:
* Learning Rate of 0.001 still converging at 1000 epochs, missing some features
* Learning Rate of 0.1 - stuck in a local minum as a step function

For a 1 $\times$ 500 $\times$ 500 $\times$ 1 neural network:
* Learning Rate of 0.01 - 0.001 for a close fit to training data
* Learning Rate of 0.1 - stuck in a local minimum as a line
* Learning Rate of $\le$ 0.0001 still converging at 1000 epochs, missing some features

For a 1 $\times$ 500 $\times$ 500 $\times$ 500 $\times$ 1 neural network:
* Learning Rate of 0.01 - 0.001 for a close fit to training data
* Learning Rate of 0.1 - stuck as a line
* Learning Rate of $\le$ 0.0001 still converging at 1000 epochs, missing some features

#### Visualizing the Neural Net

There are some methods available to interogate the artificial neural net.

* neural net summary

* weights

Here's the summary from our neural net.  It lists by layers the number of nodes and number of parameters.

```python
model_2.summary()                                      # artificial neural network design and number of parameters
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_1 (Dense)              (None, 1)                 2
    _________________________________________________________________
    dense_2 (Dense)              (None, 50)                100
    _________________________________________________________________
    dense_3 (Dense)              (None, 50)                2550
    _________________________________________________________________
    dense_4 (Dense)              (None, 50)                2550
    _________________________________________________________________
    dense_5 (Dense)              (None, 1)                 51
    =================================================================
    Total params: 5,253
    Trainable params: 5,253
    Non-trainable params: 0
    _________________________________________________________________

We can also see the actual trained weights for each node in each layer.

```python
for layer in model_2.layers:                            # weights for the trained artificial neural network
    g = layer.get_config()
    h = layer.get_weights()
    print(g)
    print(h)
    print('\n')
```

    {'name': 'dense_1', 'trainable': True, 'batch_input_shape': (None, 1), 'dtype': 'float32', 'units': 1, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[-1.4147551]], dtype=float32), array([-0.01439498], dtype=float32)]

    {'name': 'dense_2', 'trainable': True, 'dtype': 'float32', 'units': 50, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[ 0.14932705, -0.10264604, -0.33969954,  0.31093252,  0.2145236 ,
             0.32180798, -0.10889235,  0.11232843,  0.19960962, -0.07476728,
             0.33294153, -0.16865699, -0.06445652, -0.09790467, -0.27024695,
            -0.13523069,  0.10387363, -0.04809465, -0.13572337, -0.06897725,
             0.01293801, -0.0818542 ,  0.3029526 ,  0.11031199, -0.05332224,
            -0.13779743, -0.1960506 ,  0.17890808,  0.03713334,  0.33075982,
            -0.3571539 ,  0.35861987, -0.4360629 ,  0.19757433,  0.34264985,
            -0.39899147, -0.40658435,  0.35157818,  0.3341597 , -0.17569157,
             0.06848703,  0.32751465,  0.11594795,  0.11036058,  0.21403904,
             0.12671849,  0.23514925,  0.35449335, -0.40580526, -0.1251979 ]],
          dtype=float32), array([ 0.15805052,  0.04639233, -0.14979026,  0.04190359, -0.05533486,
           -0.07855641,  0.07349111,  0.05456657,  0.10348058,  0.02873405,
            0.05281656,  0.06403538, -0.03816955,  0.06675835, -0.12390574,
            0.0499173 , -0.00900104, -0.02229033, -0.05759256, -0.03734602,
            0.1180668 ,  0.02488017, -0.05091116,  0.063106  , -0.02268437,
            0.05360034, -0.08751819, -0.01517691, -0.01356514, -0.05576697,
            0.1012039 ,  0.06980153, -0.00039215,  0.00675403,  0.07087351,
            0.01530368,  0.09749544, -0.0187586 , -0.103553  , -0.07356291,
            0.03181727, -0.07454017,  0.1214256 ,  0.11316214, -0.08233985,
           -0.0489201 , -0.02484316, -0.02451238,  0.02197754,  0.01190174],
          dtype=float32)]

    {'name': 'dense_3', 'trainable': True, 'dtype': 'float32', 'units': 50, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[ 0.0866008 , -0.21138714,  0.15490103, ...,  0.08791801,
            -0.20053284, -0.1396334 ],
           [ 0.20794626,  0.24296068, -0.05863846, ...,  0.06347975,
             0.13191554, -0.03819601],
           [-0.06086044,  0.09654687, -0.09340851, ...,  0.01657305,
            -0.20065042,  0.21152455],
           ...,
           [-0.04771208, -0.02832811, -0.12004983, ...,  0.06046649,
             0.09631606, -0.05782987],
           [-0.11109008, -0.06808387,  0.09883186, ...,  0.00329898,
             0.0576255 ,  0.06109186],
           [ 0.13807462,  0.18745   ,  0.05581758, ...,  0.2326659 ,
            -0.20272887, -0.16097307]], dtype=float32), array([ 0.05101582,  0.        ,  0.02542396, -0.00347636, -0.0223903 ,
            0.09835865,  0.0798394 , -0.02111746, -0.07176589, -0.06391353,
           -0.03750122,  0.        , -0.09393993, -0.0064932 , -0.02601191,
           -0.00417237,  0.0005519 ,  0.10209791, -0.06978188, -0.05172044,
           -0.01140783,  0.04800964, -0.00018656, -0.07705417,  0.        ,
            0.05335341, -0.00240031, -0.04992656,  0.        ,  0.07506768,
           -0.02420867, -0.13116798,  0.0325961 ,  0.07715552,  0.0288214 ,
            0.        ,  0.02394254, -0.10720296,  0.01065158,  0.14125985,
           -0.01599568, -0.00912616, -0.03190614, -0.01432877,  0.03649216,
            0.        , -0.1289962 ,  0.09527948,  0.03214816,  0.00883843],
          dtype=float32)]

    {'name': 'dense_4', 'trainable': True, 'dtype': 'float32', 'units': 50, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[ 0.2017359 ,  0.04852907, -0.09683076, ..., -0.16730429,
             0.15774786,  0.22081845],
           [ 0.07849912, -0.20400417,  0.11156546, ..., -0.0733072 ,
            -0.21984106,  0.14336787],
           [ 0.00401099,  0.02843739, -0.17725673, ..., -0.03047054,
            -0.21612985,  0.2441821 ],
           ...,
           [-0.07843167,  0.16164169, -0.1797968 , ...,  0.12517963,
            -0.08316862,  0.1434726 ],
           [ 0.10173218, -0.10393211, -0.06448039, ..., -0.22463293,
            -0.1503938 ,  0.02178239],
           [-0.11031582, -0.23266369,  0.07794039, ...,  0.05014483,
            -0.14935404, -0.23623961]], dtype=float32), array([ 9.3679875e-03,  1.2377323e-01,  0.0000000e+00,  0.0000000e+00,
           -3.2503634e-05,  2.8882851e-03, -3.2200597e-02, -1.0813296e-02,
           -5.4183293e-02, -3.1078339e-02,  1.8644951e-01,  8.5919648e-03,
            3.6846563e-02, -2.7950978e-02,  0.0000000e+00,  1.0588286e-02,
           -5.0117053e-02,  2.5520054e-01,  8.6884663e-02,  1.1265171e-02,
           -3.8141143e-02,  6.6869329e-03, -4.4384148e-02, -4.9200650e-02,
            5.8690425e-02,  4.1194636e-02, -1.5451083e-01, -2.6052179e-02,
           -2.3299415e-02,  0.0000000e+00, -7.5197183e-03,  5.6359409e-03,
            5.1748879e-02,  0.0000000e+00, -3.7939171e-03, -6.4558670e-02,
           -2.2134623e-03, -1.1186364e-01,  7.3970067e-03,  2.6078340e-02,
           -2.5315320e-03, -4.4786047e-02,  6.9038056e-02, -5.5410530e-02,
           -3.3260349e-02, -3.3550650e-02, -8.0370814e-02, -3.9808478e-02,
           -9.5356070e-03, -9.8134512e-03], dtype=float32)]

    {'name': 'dense_5', 'trainable': True, 'dtype': 'float32', 'units': 1, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[-0.16305435],
           [ 0.42568576],
           [ 0.18006864],
           [ 0.19075295],
           [-0.16755088],
           [ 0.09531375],
           [-0.24258316],
           [-0.23820902],
           [-0.19713452],
           [-0.35616592],
           [ 0.41602623],
           [-0.25280344],
           [-0.4190078 ],
           [ 0.4045327 ],
           [-0.20327619],
           [-0.08012178],
           [ 0.37359467],
           [ 0.62110245],
           [ 0.11764995],
           [-0.39577144],
           [ 0.41038996],
           [-0.03371648],
           [ 0.16390693],
           [-0.33833063],
           [-0.02636404],
           [-0.17616838],
           [ 0.13386253],
           [ 0.326801  ],
           [ 0.30463552],
           [ 0.12390026],
           [-0.15731595],
           [-0.460756  ],
           [-0.47303808],
           [-0.10336666],
           [ 0.21012333],
           [-0.43154874],
           [ 0.01475829],
           [ 0.29482684],
           [-0.19771074],
           [ 0.04589447],
           [-0.01176003],
           [ 0.1036011 ],
           [-0.3254732 ],
           [ 0.2954911 ],
           [ 0.11060128],
           [ 0.17618638],
           [ 0.3177019 ],
           [ 0.27403128],
           [ 0.19958013],
           [-0.23037782]], dtype=float32), array([0.08575688], dtype=float32)]

We can observe the weights and bias terms associate with our neural network.