\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\   \nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

```python
%matplotlib inline
from scipy import signal
import numpy as np
import matplotlib.pyplot as plt
import math

from matplotlib import animation, rc
from IPython.display import HTML

fig, ax = plt.subplots()

```

```python
ax.grid(True, linestyle = '--', color = 'black')
ax.set_xlim(( -6, 6))
ax.set_ylim((-2, 2))
ax.annotate('Oscillating at [1.5,1]', xy=(1.5, 1), xytext=(-1, 1.5),
            arrowprops=dict(facecolor='black', shrink=0.05),)

line, = ax.plot([], [], lw=1)
line1, = ax.plot([], [], lw=2)
line3, = ax.plot([], [], 'bx')
line4, = ax.plot([], [], 'g-.')
line5, = ax.plot([], [], 'y-')
line6, = ax.plot([], [], 'y-')

def gaussian(x,x0,sigma):
  return np.exp(-np.power((x - x0)/sigma, 2.)/2.)

def init():
    line.set_data([], [])
    line1.set_data([], [])
    line3.set_data([], [])
    line4.set_data([], [])
    line5.set_data([], [])
    line5.set_data([], [])
    return (line,line1,line3,line4,line5,line6,)

def animate(i):
    z = 1.5
    x = np.linspace(-6, 6, 250)
    y = np.sin(4 * np.pi * (x - 0.03 * i))
    # y1 = 0.5*np.sin(4 * np.pi * (x + 0.025 * i))
    y1 = gaussian(x, i*(.0003+.15)-8, .5) * np.sin(4 * np.pi * (x - 0.03 * i))
    #y1 = 0.75/2 * signal.square(2 * np.pi * (x + 0.01 * i)) + 0.75/2
    p = math.sin(4 * np.pi * (z - 0.03 * i))
    g = gaussian(x, i*(.0003+.15)-8, .5)
    # .square
    # 0.5*np.sin(4 * np.pi * (x + 0.025 * i))

    #line.set_data(x, y)
    line1.set_data(x, y1)
    line3.set_data([z,z], p)
    line4.set_data([z,z], [-1,1])
    line5.set_data(x, g)
    line6.set_data(x, -g)
    return (line1,line3,line4,line5,line6,)

anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=100, interval=100,
                               blit=True)

HTML(anim.to_html5_video())
```

```python
help(animation.FuncAnimation)
```

    Help on class FuncAnimation in module matplotlib.animation:

    class FuncAnimation(TimedAnimation)
     |  Makes an animation by repeatedly calling a function ``func``.
     |
     |  Parameters
     |  ----------
     |  fig : matplotlib.figure.Figure
     |     The figure object that is used to get draw, resize, and any
     |     other needed events.
     |
     |  func : callable
     |     The function to call at each frame.  The first argument will
     |     be the next value in ``frames``.   Any additional positional
     |     arguments can be supplied via the ``fargs`` parameter.
     |
     |     The required signature is::
     |
     |        def func(frame, *fargs) -> iterable_of_artists:
     |
     |  frames : iterable, int, generator function, or None, optional
     |      Source of data to pass ``func`` and each frame of the animation
     |
     |      If an iterable, then simply use the values provided.  If the
     |      iterable has a length, it will override the ``save_count`` kwarg.
     |
     |      If an integer, then equivalent to passing ``range(frames)``
     |
     |      If a generator function, then must have the signature::
     |
     |         def gen_function() -> obj:
     |
     |      If ``None``, then equivalent to passing ``itertools.count``.
     |
     |      In all of these cases, the values in *frames* is simply passed through
     |      to the user-supplied *func* and thus can be of any type.
     |
     |  init_func : callable, optional
     |     A function used to draw a clear frame. If not given, the
     |     results of drawing from the first item in the frames sequence
     |     will be used. This function will be called once before the
     |     first frame.
     |
     |     If ``blit == True``, ``init_func`` must return an iterable of artists
     |     to be re-drawn.
     |
     |     The required signature is::
     |
     |        def init_func() -> iterable_of_artists:
     |
     |  fargs : tuple or None, optional
     |     Additional arguments to pass to each call to *func*.
     |
     |  save_count : int, optional
     |     The number of values from *frames* to cache.
     |
     |  interval : number, optional
     |     Delay between frames in milliseconds.  Defaults to 200.
     |
     |  repeat_delay : number, optional
     |     If the animation in repeated, adds a delay in milliseconds
     |     before repeating the animation.  Defaults to ``None``.
     |
     |  repeat : bool, optional
     |     Controls whether the animation should repeat when the sequence
     |     of frames is completed.  Defaults to ``True``.
     |
     |  blit : bool, optional
     |     Controls whether blitting is used to optimize drawing. Note: when using
     |     blitting any animated artists will be drawn according to their zorder.
     |     However, they will be drawn on top of any previous artists, regardless
     |     of their zorder.  Defaults to ``False``.
     |
     |  Method resolution order:
     |      FuncAnimation
     |      TimedAnimation
     |      Animation
     |      builtins.object
     |
     |  Methods defined here:
     |
     |  __init__(self, fig, func, frames=None, init_func=None, fargs=None, save_count=None, **kwargs)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  new_frame_seq(self)
     |      Creates a new sequence of frame information.
     |
     |  new_saved_frame_seq(self)
     |      Creates a new sequence of saved/cached frame information.
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from Animation:
     |
     |  save(self, filename, writer=None, fps=None, dpi=None, codec=None, bitrate=None, extra_args=None, metadata=None, extra_anim=None, savefig_kwargs=None)
     |      Saves a movie file by drawing every frame.
     |
     |      Parameters
     |      ----------
     |
     |      filename : str
     |          The output filename, e.g., :file:`mymovie.mp4`.
     |
     |      writer : :class:`MovieWriter` or str, optional
     |          A `MovieWriter` instance to use or a key that identifies a
     |          class to use, such as 'ffmpeg'. If ``None``, defaults to
     |          :rc:`animation.writer`.
     |
     |      fps : number, optional
     |         Frames per second in the movie. Defaults to ``None``, which will use
     |         the animation's specified interval to set the frames per second.
     |
     |      dpi : number, optional
     |         Controls the dots per inch for the movie frames.  This combined with
     |         the figure's size in inches controls the size of the movie.  If
     |         ``None``, defaults to :rc:`savefig.dpi`.
     |
     |      codec : str, optional
     |         The video codec to be used. Not all codecs are supported
     |         by a given :class:`MovieWriter`. If ``None``, default to
     |         :rc:`animation.codec`.
     |
     |      bitrate : number, optional
     |         Specifies the number of bits used per second in the compressed
     |         movie, in kilobits per second. A higher number means a higher
     |         quality movie, but at the cost of increased file size. If ``None``,
     |         defaults to :rc:`animation.bitrate`.
     |
     |      extra_args : list, optional
     |         List of extra string arguments to be passed to the underlying movie
     |         utility. If ``None``, defaults to :rc:`animation.extra_args`.
     |
     |      metadata : Dict[str, str], optional
     |         Dictionary of keys and values for metadata to include in
     |         the output file. Some keys that may be of use include:
     |         title, artist, genre, subject, copyright, srcform, comment.
     |
     |      extra_anim : list, optional
     |         Additional `Animation` objects that should be included
     |         in the saved movie file. These need to be from the same
     |         `matplotlib.figure.Figure` instance. Also, animation frames will
     |         just be simply combined, so there should be a 1:1 correspondence
     |         between the frames from the different animations.
     |
     |      savefig_kwargs : dict, optional
     |         Is a dictionary containing keyword arguments to be passed
     |         on to the `savefig` command which is called repeatedly to
     |         save the individual frames.
     |
     |      Notes
     |      -----
     |      fps, codec, bitrate, extra_args, metadata are used to
     |      construct a :class:`MovieWriter` instance and can only be
     |      passed if `writer` is a string.  If they are passed as
     |      non-`None` and ``writer`` is a :class:`MovieWriter`, a
     |      `RuntimeError` will be raised.
     |
     |  to_html5_video(self, embed_limit=None)
     |      Returns animation as an HTML5 video tag.
     |
     |      This saves the animation as an h264 video, encoded in base64
     |      directly into the HTML5 video tag. This respects the rc parameters
     |      for the writer as well as the bitrate. This also makes use of the
     |      ``interval`` to control the speed, and uses the ``repeat``
     |      parameter to decide whether to loop.
     |
     |  to_jshtml(self, fps=None, embed_frames=True, default_mode=None)
     |      Generate HTML representation of the animation
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors inherited from Animation:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)

```python

```


# Zero delay the easy way

Raph Levien
25 Jan 2014

So-called "zero delay" filters have become popular in synthesizer and audio applications, due to their
accurate emulation of analog filter designs and efficient implementation. The techniques used
to create zero delay filters are considered standard in the circuit emulation field, but even so the
idea of creating a zero delay filter from scratch can be intimidating to those just wanting to design
a decent audio filter.

This notebook presents a very easy design methodology, using cookbook tools. There are references to
the literature for those wanting to dig deeper into the theory. None of these techniques are new, but
it is my hope that presenting them in this form will make them more accessible.

In broad outline, the approach recommended here is:

* Choose an analog filter "prototype".

* Write down the filter response in state space representation.

* Choose a discretization method. Bilinear transform is most popular.

* Discretize (convert from continuous time to discrete time).

* Implement the filter in discrete state space representation.

The resulting filter will be a reasonably (but not perfectly) accurate emulation of the analog filter.
It will share the same stability properties as the prototype, as well as similar behavior under
modulation.

It is presented here as an IPython notebook, so you can easily read it as a static document, but you
can also load it into your IPython environment, play with the tools, and adapt the code. Have fun!

```
%pylab inline
from scipy import signal
from scipy import linalg
```

    Populating the interactive namespace from numpy and matplotlib

## Choose a prototype filter

I'll assume you already have a good idea which analog filter you want to simulate. We're going to concentrate on the _linear_
response, but one of the great advantages of the state space approach is that it can be extended to nonlinearities without
too much trouble.

One of the most important factors is the _order_ of the filter, which corresponds directly to the dimensionality of the state
vector. This technique can handle arbitraty order filters, but direct state space evaluation is definitely most efficient for
smaller orders (2 or 4 especially). If you have a very high order filter, consider decomposing it into a series of smaller
filters. There are tools for doing this in transfer function space; it's particularly easy if you know where your poles and
zeros are. (Search for tf2sos for more information)

We'll use three filter designs as running examples, all of which are popular as filters in analog synthesizers: a simple
one-pole lowpass (also known as the "RC lowpass"), the classic two-pole "state variable filter", and the 4-pole Moog resonant lowpass
filter.

### One-pole lowpass

This is also known as the RC filter, as it can be trivially implemented with one resistor and one capacitor (generally in audio
applications you'd want an opamp to buffer the output so it's available at low impedance, but that's not central to the
response of the filter).

Being only one pole, this filter is super easy to analyze. Its impulse response is just exponential decay.

(TODO: images for circuit diagrams for these filters)

### State variable filter

See Figure 3 in https://ccrma.stanford.edu/~jos/svf/svf.pdf for a circuit diagram. This is given in highly idealized form,
with integrators drawn as integrators rather than expanded out as opamps. However, this is the same as the classic state
variable filter, for example, the one presented in Don Lancaster's Active Filter Cookbook, which has been the basis of many
synth builds.

### Moog 4-pole ladder filter

The Moog 4-pole filter is of course one of the classic designs. There's quite a bit of literature on it
(see http://www.timstinchcombe.co.uk/index.php?pge=papers).

A good simple description of the filter is given in https://ccrma.stanford.edu/~stilti/papers/moogvcf.pdf (Analyzing the
Moog VCF with Considerations for Digital Implementation, by Tom Stilson and Julious Smith). This paper presents a very
simple analog model of the linear response: it is simply 4 one pole lowpass filters in series, all tuned to the same corner
frequency, with a feedback path which controls resonance. Ignore the discretization in this paper, however; we'll do much
better.

## State space representation

Once you've chosen your filter, get it into state space representation. This will be four matrices $(A, B, C, D)$ that together
represent the differential equations of the filter. The general equations are:

$$ \begin{align}
\mathbf{\dot{x}}(t) = & A\mathbf{x}(t) + B\mathbf{u}(t) \\
\mathbf{y}(t) = & C\mathbf{x}(t) + D\mathbf{u}(t)
\end{align} $$

Here, $\mathbf{u}(t)$ represents the input, $\mathbf{x}(t)$ represents the state, and $\mathbf{y}(t)$
represents the output. Such systems are known as "single input" or "multiple input" depending on the
size of $\mathbf{u}$, and "single output" or "multiple output" depending on the size of $\mathbf{y}$.
In this presentation, we'll concentrate on the "single input, single output" case.

### One-pole lowpass

Because it's just one pole, the state can be a scalar rather than a vector, so we don't even need to write this in matrix form.
The differential equation is:

$ \dot{x}(t) = -\alpha x(t) + u(t) $

Variables are as in standard state variable representation. Lowpass is $y(t) = x(t)$, and highpass is $y(t) = u(t) - x(t)$.

Here, $\alpha$ controls the corner frequency. It's easiest just to normalize it to 1. The state space reprentation could hardly
be simpler:

$$ \begin{align}
A & = -1 \\
B & = 1 \\
C & = 1 \\
D & = 0
\end{align} $$

For highpass, $C = -1$ and $D = 1$ instead.

```
def rc_lp_system():
    return array([[-1]]), array([1]), array([1]), 0

def rc_hp_system():
    return array([[-1]]), array([1]), array([-1]), 1

```

### State variable filter

The jos paper gives the state variable representation (here we switch the order of the state variables so they
respresent left-to-right flow, and to emphasize the similarity to the other filters).

$$ \begin{align}
A & = \left[\begin{array}{cc} -k & -1 \\ 1 & 0 \\ \end{array}\right] \\
B & = \left[\begin{array}{c} 1 \\ 0 \\ \end{array}\right] \\
C & = [ \begin{array}{cc} 0 & 1 \end{array} ] \\
D & = 0
\end{align} $$

Here, $k$ controls resonance. It is equal to $1/Q$, and ranges from 2 for no resonance to 0 for undamped resonance,
i.e. self-oscillation. For a resonance control that varies from 0 to 1, simply set $k = 2 - 2 \cdot \mathit{resonance}$.

For bandpass, $C  = [  1 \: 0 ]$ instead, and for highpass,
$C  = [ {-k} \: {-1} ]$ and $D=1$. In fact, all "Audio EQ Cookbook" responses can easily be done with a state variable
filter, making it an appealing alternative to canonical form biquads. See [Second order sections in matrix form](Second%20order%20sections%20in%20matrix%20form.ipynb) notebook for all of these recipes. Also see [Solving the continuous SVF equations using trapezoidal
integration and equivalent currents](http://www.cytomic.com/files/dsp/SvfLinearTrapOptimised2.pdf) for a different
derivation.

```
def svf_lp_system(res):
    k = 2 - 2 * res
    A = array([[-k, -1], [1, 0]])
    B = array([1, 0])
    C = array([0, 1])
    D = 0
    return A, B, C, D

def svf_hp_system(res):
    k = 2 - 2 * res
    A = array([[-k, -1], [1, 0]])
    B = array([1, 0])
    C = array([-k, -1])
    D = 1
    return A, B, C, D
```

### Moog filter

The Moog filter also has a simple state-space representation. It is easy to build the total system by stitching
together four one-pole lowpass filters and adding a feedback path.

$$ \begin{align}
A & = \left[\begin{array}{cc} -1 & 0 & 0 & -k \\
1 & -1 & 0 & 0 \\
0 & 1 & -1 & 0 \\
0 & 0 & 1 & -1 \\
\end{array}\right] \\
B & = \left[\begin{array}{c} 1 \\ 0 \\ 0 \\ 0 \\ \end{array}\right] \\
C & = [ \begin{array}{cc} 0 & 0 & 0 & 1 \end{array} ] \\
D & = 0
\end{align} $$

Here, k varies from 0 (no resonance) to 4 (self oscillation).

```
def moog_system(res):
    k = 4 * res
    A = array([[-1, 0, 0, -k], [1, -1, 0, 0], [0, 1, -1, 0], [0, 0, 1, -1]])
    B = array([1, 0, 0, 0])
    C = array([0, 0, 0, 1])
    D = 0
    return A, B, C, D
```

## Response of continuous time filter in state space representation

It's possible to accurately compute the frequency and phase response of a state space filter, without for
example having to expand out an impulse response and compute a fourier transform.

The transfer function of a filter in (continuous) state space formulation is simple:

$$ H(s) = D + C(sI-A)^{-1}B $$

The Wikipedia article on State space representation isn't a bad source for this: http://en.wikipedia.org/wiki/State_space_representation

We can use it to plot out the frequency response of some of the filters we defined above. (Note: if you're interested in
phase response, just plot the angle(response) instead of abs(response)).

```
def transfer(sys, s):
    A, B, C, D = sys
    return D + dot(C, dot(inv(s * identity(len(A)) - A), B))
```

```
xs = logspace(-3, 0, num=200)
def db(x):
    return 20 * log10(abs(x))
def freqz_ss_continuous(sys, f):
    return array([transfer(sys, 1j * x / f) for x in xs])
```

The one-pole lowpass and highpass filters have the expected response, with both at -3dB at the corner frequency,
and 6 dB/octave rolloff.

```
f = sqrt(1e-3)
semilogx(xs, db(freqz_ss_continuous(rc_lp_system(), f)))
semilogx(xs, db(freqz_ss_continuous(rc_hp_system(), f)))
show()
```

Now we plot the response of the Moog filter model, and also see the expected response.

```
for i in range(5):
    sys = moog_system(0.2 * i)
    semilogx(xs, db(freqz_ss_continuous(sys, f)))
```

## Choose a discretization

By far, the most popular discretization method in audio applications is the bilinear transform. A second, valid choice is
step invariance. These two are the most popular discretization methods, though others exist, as they both preserve the
stability properties of the original filter.

A very simple discretization is forward Euler. You'll see this quite a bit in older papers (particularly if they seem to
be struggling with how to accommodate the delay-free feedback path), but it's basically rubbish. It works okay at very low
frequencies, but can easily transform a stable filter into an unstable one (as happens in the discretized state variable
filter), or totally lose the resonance properties (as in the Moog). It turns out to be totally fine for the one-pole lowpass,
so you'll see it there, and as a consequence works for the Moog filter in the zero resonance case, because that's just
four of those strung together.

Both bilinear transform and step invariant lose accuracy, but in different ways. The bilinear transform has exactly the same
_frequency_ response as the analog prototype, but with frequencies warped, so that infinite frequency in the analog domain
maps to the Nyquist frequency. By contrast, step invariance has exactly the same _impulse_ response assuming that the input
is a step exactly one sample period wide (also known as zero order hold), but since that signal is not band-limited, it
means that the frequency response is aliased. Note that it's just the response that's aliased; there are no frequencies in
the output that are not present in the input.

A good general criterion for choosing is: does the frequency response have a sharp lowpass characteristic? If so, then
with step invariance, the aliased components will be small, and primarily affect only the near-Nyquist frequencies. Otherwise,
for example for a high-pass response, bilinear transform is a better choice. We'll show this with graphs after implementing
the discretization techniques.

### Bilinear transform

First, we need to pre-warp the frequency response, so that the response of the digital filter matches that of the analog filter
at the cutoff frequency. Here, frequency $f$ is as a fraction of the sampling rate, so that 1/2 is the Nyquist frequency.

$ g = tan( \pi f ) $

Then, the discretization is as given by [Comparing digital implementation
via the bilinear and step-invariant transformations](http://myweb.polyu.edu.hk/~magzhang/Research/ZC04_auto.pdf), Guofeng Zhang and Tongwen Chen.

$$ \begin{align}
A_d & = (\mathbf{I} - gA)^{-1} (\mathbf{I} + gA) \\
B_d & = 2g(\mathbf{I} - gA)^{-1} B \\
C_d & = (\mathbf{I} - gA)^{-1} C \\
D_d & = g C (\mathbf{I} - gA)^{-1} B + D
\end{align} $$

Here, the $(\mathbf{I} - gA)^{-1}$ term represents solving the entire set of equations simultaneously, and includes the zero-delay
feedback term. To me, it's much easier just to throw the system into a matrix and solve it by matrix inverse in this way,
rather than solving all the individual equations by hand. Your mileage may vary.

```
def bilinear(sys, f):
    A, B, C, D = sys
    g = tan(pi * f)
    im = inv(identity(len(A)) - g * A)
    Ad = dot(im, identity(len(A)) + g * A)
    Bd = 2 * dot(im, g * B)
    Cd = dot(C, im)
    Dd = D + dot(C, dot(im, g * B))
    return Ad, Bd, Cd, Dd
```

Some historical notes. To my knowledge, the first publication of a proper bilinear transform of the analog Moog filter
was: http://quod.lib.umich.edu/cgi/p/pod/dod-idx/preserving-the-structure-of-the-moog-vcf-in-the-digital.pdf, Preserving the Structure
of the Moog VCF in the Digital Domain, Federico Fontana.

The earlier publications by Stilson and Smith, then Huovilainen, contained defective discretizations that attempted to deal with
the delay free loop some other technique than simply solving the equations.

These days, however, the proper bilinear transform is standard. In synthesizer circles, the technique goes by the name
"zero delay filters" and also "topology preserving transform". An important milestone in popularizing this technique was
the publication of http://www.native-instruments.com/fileadmin/ni_media/downloads/pdf/VAFilterDesign_1.0.3.pdf, The Art of
VA Filter Design, by Vadim Zavalishin.

Meanwhile, outside the published literature, there is an excellent discussion of this topic in the KVR DSP Forum thread,
http://www.kvraudio.com/forum/viewtopic.php?f=33&t=349859, Cheap non-linear zero-delay filters.

### Step-invariant transform

The step-invariant transform represents the exact response under the assumption that the input is a sequence of steps
each one sample period in width (also known as zero order hold).
There are a few ways to compute it; here is the one I find
easiest.

$ A' = \left[\begin{array}{cc} 0 & 0 \\ B & A \end{array}\right] $

Then,

$ \left[\begin{array}{cc} \_ & \_ \\ B_d & A_d \end{array}\right] = \exp(2 \pi f A') $

Here, $\exp(A)$ represents the matrix exponential, and the underscores respresent throwing away that part of the
resulting matrix.

And

$$ \begin{align}
C_d & = C \\
D_d & = D
\end{align} $$

```
def step_invariant(sys, f):
    A, B, C, D = sys
    A_ = concatenate([[zeros(len(A) + 1)], concatenate([[B], A.T]).T])
    eA = linalg.expm(2 * pi * f * A_)
    Ad = eA[1:, 1:]
    Bd = eA[1:, 0]
    return Ad, Bd, C, D
```

Matrix exponential is a cookbook technique (implemented as linalg.expm in NumPy). It is defined in terms of the straightforward extension of the Taylor series for
ordinary exp(x) generalized to matrices:

$$ exp(A) = \sum\limits_{k=0}^{\infty} \frac{A^k}{k!} $$

If you're implementing it by yourself, for small matrices, the "scaling and squaring"
technique works best. See http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.156.9316, The scaling and squaring method for the matrix exponential revisited, Nicholas Higham. Here's a simple implementation that computes the first n1 terms of the Taylor's series expansion
for exp, followed by n2 repeated squarings.

```
# should give the same results as linalg.expm for well-behaved systems
def my_expm(A, n1 = 8, n2 = 8):
    A1 = A / (1 << n2)
    A2 = np.identity(len(A))
    A3 = np.identity(len(A))
    for i in range(1, n1):
        A3 = dot(A1, A3) / i
        A2 = A2 + A3
    for i in range(n2):
        A2 = dot(A2, A2)
    return A2
```

## Implementing state space filters

Actually running a filter given in discrete time state space formulation is pretty straightforward. Mathematically, it is
given as:

$$ \begin{align}
\mathbf{x}(n + 1) = & A\mathbf{x}(n) + B\mathbf{u}(n) \\
\mathbf{y}(n) = & C\mathbf{x}(n) + D\mathbf{u}(n)
\end{align} $$

But it's probably just as easy to show the code:

```
def runss(sys, inp):
    A, B, C, D = sys
    x = zeros(len(A))
    result = []
    for u in inp:
        result.append(dot(C, x) + D * u)
        x = dot(A, x) + B * u
    return result
```

We can see that the impulse response for the bilinear and step-invariant transformations of the Moog filter are pretty similar,
but off in phase by a half sample or so.

```
sys = moog_system(0.8)
impulse = zeros(100)
impulse[10] = 1
plot(runss(bilinear(sys, 0.1), impulse))
plot(runss(step_invariant(sys, 0.1), impulse))
show()
```

Matrix-based implementations in state space can be very, very fast. The "raising" trick of [Implementation of recursive
digital filters into vector SIMD DSP architectures](https://mns.ifn.et.tu-dresden.de/Lists/nPublications/Attachments/360/Robelly_P_ICASSP_04.pdf) by Robelly et al for an especially useful technique for implementing second
order filters on SIMD hardware tuned for 4xfloat vectors.

See [Second order sections in matrix form](Second%20order%20sections%20in%20matrix%20form.ipynb) notebook for more details
on matrix implementation, including favorable comparisons of roundoff error compared to canonical direct form IIR implementations.

## Response of state space filters

It's possible to accurately compute the frequency and phase response of a state space filter, without for
example having to expand out an impulse response and compute a fourier transform. Of course, it doesn't
hurt to plot this Fourier transform of an impulse anyway, if for no other reason than a sanity check:

```
impulse = concatenate([[1], zeros(255)])
w, h = signal.freqz(runss(bilinear(sys, 0.1), impulse))
semilogx(w, db(h))
show()
```

The transfer function of a filter in (discrete time) state space formulation is simple, and very similar to
the continuous time version:

$$ H(z) = D + C(zI-A)^{-1}B $$

So similar, in fact, we can reuse the existing transfer() function for both continuous and discrete time.

See https://ccrma.stanford.edu/~jos/filters/Transfer_Function_State_Space.html for the derivation and more discussion.

Now we'll plot the bilinear and step-invariant transforms for the Moog filter, against the analog prototype,
for a single frequency. Both are quite close, although the frequency warping of the bilinear version causes
its rolloff to accelerate at high frequencies.

```
def freqz_ss(sys):
    return array([transfer(sys, exp(1j * pi * x)) for x in xs])

axis([1e-2, 1, -60, 10])
f = 0.05
sysd = bilinear(sys, f)
semilogx(xs, db(freqz_ss(sysd)))
sysd = step_invariant(sys, f)
semilogx(xs, db(freqz_ss(sysd)))

# Analog prototype
semilogx(xs, db(freqz_ss_continuous(sys, 2 * f)))
show()
```

The bilinear transform shows the characteristic warping of frequency response, including a zero at the
Nyquist frequency.

```
axis([1e-2, 1, -60, 10])
for i in range(5):
    f = 0.4 * .5 ** i
    sysd = bilinear(sys, f)
    semilogx(xs, db(freqz_ss(sysd)))
    semilogx(xs, db(freqz_ss_continuous(sys, 2 * f)))
show()
```

For this particular filter response, the step invariant transform is more accurate.

```
axis([1e-2, 1, -60, 10])
for i in range(5):
    f = 0.4 * .5 ** i
    sysd = step_invariant(sys, f)
    semilogx(xs, db(freqz_ss(sysd)))
    semilogx(xs, db(freqz_ss_continuous(sys, 2 * f)))
show()
```

For highpass filters, the bilinear transform performs fairly well, albeit with noticeable effects
from frequency warping at higher corner frequencies.

```
sys = svf_hp_system(0.5)
axis([1e-2, 1, -60, 10])
for i in range(5):
    f = 0.4 * .5 ** i
    sysd = bilinear(sys, f)
    semilogx(xs, db(freqz_ss(sysd)))
show()
```

However, step-invariant is not a good choice for high-pass filter responses, as expected.

```
axis([1e-2, 1, -60, 10])
for i in range(5):
    f = 0.4 * .5 ** i
    sysd = step_invariant(sys, f)
    semilogx(xs, db(freqz_ss(sysd)))
show()
```

### A bit of simplification

It's certainly possible to just use matrix primitives to design the filter. However, for modulating the filter parameters
at a high rate, it might make sense to simplify the math symbolically. For the state variable filter, the following
code goes straight to the bilinear transformed state space representation, using a by-hand symbolic solution to the matrix
inverse:

```
def svf_lp(f, res):
    g = tan(pi * f)
    k = 2 - 2 * res
    a1 = 1/(1 + g * (g + k))
    a2 = g * a1
    a3 = g * a2
    A = array([[2*a1 - 1, -2*a2],
               [2*a2, 1 - 2*a3]])
    B = array([2 * a2, 2 * a3])
    C = array([a2, 1 - a3])
    D = a3
    return A, B, C, D
```

Then we can check that this really does compute the same system.

```
f = 0.1
res = 0.2
print svf_lp(f, res)
print bilinear(svf_lp_system(res), f)
```

    (array([[ 0.23043279, -0.39979185],
           [ 0.39979185,  0.87009975]]), array([ 0.39979185,  0.12990025]), array([ 0.19989592,  0.93504988]), 0.064950123180475744)
    (array([[ 0.23043279, -0.39979185],
           [ 0.39979185,  0.87009975]]), array([ 0.39979185,  0.12990025]), array([ 0.19989592,  0.93504988]), 0.064950123180475744)

Simplified code for the rest of the [Audio EQ cookbook](http://www.musicdsp.org/files/Audio-EQ-Cookbook.txt) designs are also at the [Second order sections in matrix form](Second%20order%20sections%20in%20matrix%20form.ipynb) notebook.

Similarly, for the Moog filter, it's not too difficult to invert the matrix symbolically. Both Vadim's book and
the KVR thread on cheap zero delay filters contain these derivations.

TODO: adapt code and put here.

## A note on nonlinearity

Real analog filters have nonlinearities, and true "virtual analog" simulations attempt to model those as
well as the purely linear components.
Antti's paper is excellent at presenting differential equations representing
an only slightly idealized version of the Moog filter.

Really emulating the non-linearity is beyond the scope of this cookbook, but I will note that keeping the
state space variables matching physical quantities (voltages across capacitors, etc.) is a huge help. There
are a number of ways of simply adding the nonlinearity back in. These are covered in more detail in the
"cheap zero delay" KVR thread and Vadim's book.

## Conclusion

We have shown how to design digital IIR filters in a state space framework, starting from analog prototypes
and giving some choices regarding discretization. All these techniques are cookbook and don't require solving
systems of equations by hand, or doing symbolic manipulation.

These techniques work on a wide range of source filters and allow both easy, fast implementation and analysis, for example plotting the frequency response as a precise function of frequency, not requiring Fourier transforms
of impulse responses.

In addition, the literature is rich in techniques for state space systems. As an example, the poles of the resulting
filter are exactly the eigenvalues of the $A$ matrix. In analog form, state space matrices can be composed arbitrarily,
including with feedback paths, making the basic technique powerful for circuit design and analysis.

It is my hope that these matrix formulations will make the process of designing digital filters less mysterious,
and accessible to a broader audience, much in the way that Dan Lancaster's Active Filter Cookbook popularized analog
filter designs, and no doubt contributed to a great many home-built designs.

```

```


# Gradient estimation

For whatever reason, we want to know how functions inputs change their outputs. Sometimes we cannot use automatic differentiation to help us out. This might be because the function;

* has stochastic variables within it (can differentiate, but might not give sensible answers)
* has discrete variables in it discrete (and categorical) (ints, ...)
* is unknown to us.

Or maybe there are other complications, leading to _partial differentiability_;

* it's too big
* dont have access to make certain pertubations (may not be able to control some variables -- that is just back to stoachastic variables?!)
* structural/dynamic (closely related to discrete? this is more of a problem of propagation? can still estimate gradient, but cannot back prop them where we want. as no dependency exists?). various dtypes, trees, graphs, ...

So, how can AD systems be made more general to handle these cases? Ideally we would have a automated differentiation system that can handle all combinations of these issues. (We might need a few more hyper parameters to control the tradeoff/accuracy of some approximations)

Thus allowing us to handle something like this:

```python
# want to train the web application's interface to be
# - soo intuitive that no questions need to be asked,
# - or soo facinating that many questions are asked,
# or train the knowledge graph to answer corretly
# or ...

web_app.run()  # the site we are optimising (many discrete variables)
natural_langual_query = web_app.watch_for_queries()  # human interaction (unknown fn)
query = nlp.parse(natural_langual_query)  # neural parser (cts and deep)
answer = knowledge_graph.answer(query)  # a relational database (a BIG graph)
reply = nlp.explain(answer)  # neural translation (cts and deep)
feedback = web_app.send(reply)  # human feedback (unknown fn)
web_app.train(feedback)
```

and all the AD is just automated in the background. So what we want is: functions that take arbitrary;

- stochastic fns and return a low variance, zero bias, estimator. `f: stochastic -> cts`
- discrete fns and return a softened/cts estimator. `g: discrete -> cts`
- black boxes and return a cts approximation (dynamically approximates it with something like boosting? it allocates capacity depending on accuracy). `h: unknown -> cts`
- ?

So we can just wrap the troublesome ops with these functions, and we are done.
***

Cool. First, lets have a look at some of the existing problems and solutions.

# Contents

* [Stochastic functions](#stochastic)
* [Discrete functions](#discrete)
* [Unknown functions](#unknown)
* [Recent work](#recent)
* [Efficient estimation](#efficient)
* [Conditional differentiability]()

```python
import numpy as np
import numpy.random as rnd
import matplotlib.pyplot as plt
import networkx as nx
```

# Sotchastic

Why does stochasticity mean we cant take the derivative?
Let $\bar x$ be some random variable, and let $\bar y = f(\bar x)$. So, we want $\frac{d\bar y}{d\bar x}$. (no this doesnt work... the problem occurs when f the function, $f$, is a stochastic function. $\bar y = f(\bar x, \epsilon)$

If we take $\frac{d\bar y}{d\bar x} \mid_x$ for $ x\sim X, y \sim Y$ and

Dont know if $\Delta y$ is due to $\Delta \theta$ or noise produced by $\bar x$. _It is a credit assignment problem?_
If we have multiple RVs then what is the likelihood one of them produced $\Delta y$ versus the likelihood of our pertubation of $\theta$?

|      | Static noise           | Dynamic noise | Adversarial noise |
| ------------- |:-------------:| -----:| -----:|
| __Not-controllable__     | vanilla | ? | ? |
| __Partially-controllable__      | ?       |   ? | ? |

Why would we care about dynamic or adversarial (or static noise??)? It is less about noise and more about uncertainty.

There is noise in floats, rounding errors, etc, but this isnt generally an issue. Why? The noise is small compared to the signal of interest?! $\frac{\text{signal}}{\text{noise}}$ is large.

### Controlled variables

Stochastic variables can be reframed through the lens of controllable and uncontrollable variables.

```python
N = 1000
gaussian = lambda mean, stddev, N: rnd.standard_normal(N)*stddev + mean

p = np.linspace(-5, 5, N)
def fn(p):
    x = gaussian(0, 1, len(p))
    return x**2 + 2*p + x*p  # multiplicaive noise really fs things up...
y = fn(p)

# dydp = np.mean(2*x+p)   # take derivative and then marginalise
# must marginalise out the noise
# marginalisation works here because x is unbiased around 0
# print(dydp)

#####
# alternatively
E_y = np.mean(y)  # marginalise then take derivative
dydp = p
print(E_y)

plt.scatter(p, y, alpha=0.25, s=1.5)
plt.plot(p, 2*p, c='g')
_ = plt.title('y vs p')
```

```python
# ok, dydp seems clear, but what if we have fewer samples
n_samples = 5
p_val = 2
p = np.array([p_val]*n_samples)  # take n_samples at p=p_val
plt.scatter(p, fn(p))
plt.plot(p, 2*p, c='g', alpha=0.1)
plt.xlim(-5, 5)
plt.ylim(-10, 40)
# what is dydp?

# actually tbh, you still get a pretty decent estimate with
# few samples. what if we turn the variance up?
```

```python
# what if the noise is biased!?
```

## Variance reduction
What is necessary to estimate a graident?
__Why do we care about variance reduction? If the estimate is accurate, what does it matter?__

A motivating example??? Show that optimisation with high variance is slower!?

```python
def estimate(x):
    # cumlative mean
    return [np.mean(x[:i]) for i in range(1,N)]  # could be done recursively.

m_1 = estimate(gaussian(0, 1, N))
m_3 = estimate(gaussian(0, 4, N))
m_5 = estimate(gaussian(0, 8, N))
```

```python
plt.figure(figsize=(12,6))
plt.plot(np.zeros(len(m_1)))
plt.plot(np.clip(m_1, -1, 1), label='Var=1')
plt.plot(np.clip(m_3, -1, 1), label='Var=4')
plt.plot(np.clip(m_5, -1, 1), label='Var=8')
plt.legend()
plt.xlabel('Number of samples')
plt.ylabel('Estimated mean')
```

Ok, so the mean tends to wander around more if there is more variance. So there must be a bound on the accuracy of the mean, given the variance!?

$$
\epsilon = \mathbb E[x] - \frac{1}{N}\sum_{i=0}^N x \\
\epsilon = \mathcal O(\sigma)
$$

Still doesnt tell me why high variance matters... It matterns in the context of optimisation because !??!??

```python

```

Need some unbiased estimator? But now the framework wont be functional anymore? As we need states to remember things, so we can calculate a mean field? Or we just do a large amount of expensive MC simulations?

## Control variates

* https://en.wikipedia.org/wiki/Control_variates
* Control Variates By Christiane Lemieux

> In order to reduce the variance $\sigma^2/n$, two approaches are possible. One is to replace the function $\hat h$ by an alternative function $\hat h$ such that $E( h(X))= μ$ but with $Var( h(X))
< σ^2$. This is the idea behind variance reduction techniques. The second approach consists of replacing the iid sampling scheme at the basis of Monte Carlo simulation by quasi-random sampling (see Random Number Generation and Quasi-Monte Carlo), thereby changing the $1/n$ convergence rate of the variance by possibly something as fast as $\mathcal O((log n)^d/n^3 )$, where d is the dimension of the problem. The method of control variates that we are
about to describe falls in the first category of approaches.

?

> First, we must find a random variable W such that μ w = E(W ) can be computed analytically and also such that Y and W are dependent.

?

> To see why this is useful, consider the case where
Y and W are positively correlated. Then if the naive estimator μ ̂ w,n is lower than its expected value of μ w ,
it suggests that probably μ ̂ n is also lower than the unknown μ, and therefore, a positive correction should
be added to it to reduce its error.

?

> As the goal of this technique is to get an estimator with smaller
variance than the naive Monte Carlo estimator, it makes sense to try to choose β so as to minimize the
variance of μ ̂ cv,n .

__Q__; Is tha really true? Sure we dont want to pick b to minimise the variance of the target fn?

__Q__; What about memory limited settings. When you cannot keep thousands of past

```python
n_samples = 2000

def F(x):
    return 1/(1+x)

samples = rnd.random(n_samples)
f = F(samples)

print(np.mean(f), np.var(f))
```

    0.695897036654851 0.01939103346682

```python
mean_g = 3/2

def G(x):
    # why this fn!?
    # can different fns give better results?
    # why sorts of fns give better results? and why?
    # what if g is biased? biased to what alex...
    # the whole point was that we chose a G that wasnt...
    return x + 1

g = G(samples)

c = -np.var(f)/np.cov(f,g)[0,1]  # want to derive this myself
# print(c)

output = f + c*(g - mean_g)
# does this introduce any bias?
# seems like it should, why doesnt it?
# could try prove it? E[f] = E[f + c(g-mu_g)]

print('Mean: {}'.format(np.mean(output)),
      '\nVariance: {}'.format(np.var(output)),
      '\n{:.3f} times variance reduction'.format(np.var(f)/np.var(output))
     )
```

    Mean: 0.6927025624775074
    Variance: 0.0006591217049320103
    29.420 times variance reduction

```python
# ok, lets see f I am starting to understand.
# so if the control fn is more similar to
# the target fn then we should get more variance reduction?

mean_h = 0.4054  # found using samples
def H(x):
    return  1/(x + 2)
h = H(samples)

c = -np.var(f)/np.cov(f,h)[0,1]
output = f + c*(h - mean_h)

print('Mean: {}'.format(np.mean(output)),
      '\nVariance: {}'.format(np.var(output)),
      '\n{:.3f} times variance reduction'.format(np.var(f)/np.var(output))
     )
```

    Mean: 0.6927605874287974
    Variance: 0.00011262165997403508
    172.179 times variance reduction

Multiple control varaties.

In the example above, we are kinda saying, assume F is similar to G. If our sample is from G is a distance from its mean then the sample is also a similar distance from F's mean.
When F isnt like G, this will be wrong, but not a big deal as that is what we used their correlation for.

_What if we made $c$ a fn of the position of the sample? c(x)? Rather than a single linear approximation of the similarity between F and G we could tune it locally.

Generalising this to many controls. Constraining F to be within G_i. If sample is greater than G_1, and G_2 but less than G_3 then assume the sample was slightly too high and correct it. (but what if we had extra/partial info that G_3 was very similar to F? -- this is what c is trying to correct for?!)

https://gabrielhuang.gitbooks.io/machine-learning/reparametrization-trick.html

## The reparameterisation trick

In some cases (which ones!??), a random variable $x \sim p_{\theta}(X)$ can be rewritten as a deterministic function of the parameters $\theta$ and another random variable $z$.

$$
\begin{align}
x \sim p_{\theta}(X) &\implies x = f(\theta, z) \tag{reparameterise} \\
x &\sim \mathcal N(\mu, \sigma) \tag{normal distribution} \\
x &= f(\mu, \sigma, z) \\
x &= \mu + \sigma^2 \cdot z \\
\end{align}
$$

So which distribution should we pick z from? What if we tried to learn that? Rather than jsut setting it to be gaussian/uniform? Surely setting the distribution would make the reparameterisation biased!? <- need to explore.

__TODO__ what about the variance in the reparameterised gradient!?

## Score functions

$$
x \sim p(x|\theta) \\
\nabla_{\theta} \mathbb E_x[f(x)] = \mathbb E_x [f(x)\nabla_{\theta} \log p(x | \theta)]
$$

How is this anything special? It is just chain rule?!

Where does p(x|\theta) come from? How is it calculated/estimated?

$$

$$

http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/

#### Reinforce

> This estimator is unbiased, but in general has high variance. Intuitively, this estimator is limited by the fact that it doesn’t use any information about how $f$
depends on $b$, only on the final outcome $f(b)$.

> If instead x is a deterministic function of θ and another random variable z, the operators∇θandEz commute, yielding the
pathwise  derivative  estimator or reparameterisation trick.
??

# Discrete

Types of variable; continuious, discrete, categorical, ?
What we really care about; Ordered?

## Cts relaxations and soft-ness

What do we mean by soft?

* Sigmoids as a soft step fn.
* Gumbel-softmax as a soft argmax.
* ? as a soft look-up.
* ? as a

are these general enough to cover things like mcts, sat,

##
## Dice

* Why do we care about second order gradients?
* How did the old approaches fail?
*

$\dicei$

Generalising AD to stochastic graphs.

Why would we ever want to construct graphs with random variables in them? (not in them, but ?)
* Meta learning? Loss is a random variable (of the data)?
* RL?
*

Approaches

* Surrogate objective and AD (although objective is the wrong way to view it as it is independent of loss functions)
*

### Gumbel-max trick

Why does anyone care about the Gumbel max trick!?

$$
\begin{align}
\mathcal G &= -log(-log(\mathcal U)) \\
a_k &= P(x=k) \\
x \sim P(x)&\equiv \mathop{argmax}_k\log(a + g)  \;\;\;\; g \sim G \\
\end{align}
$$

```python
def gumbel_dist(u):
    return -np.log(-np.log(u))

U = np.linspace(1e-8, 1-1e-8, 1000)
G = gumbel_dist(U)
plt.plot(U, G)
plt.grid(b=True, which='major', color='k', alpha=0.25)
plt.title('The Gumbel distribution')
```

Need to add noise, else we just end up sampling the max everytime.

But why Gumbel distributed noise!?

```python
## want to explore an example.
# just for sanity.
# still skeptical and unsure about why it works

n = 10
P = rnd.random(n)
P /= np.sum(P)

def sample_gumbel_max_trick(a):
    """

    """
    u = gumbel_dist(rnd.random(len(a)))
    return np.argmax(np.log(a) + u)

def expected_gumbel(a, n_samples):
    samples = [sample_gumbel_max_trick(a) for i in range(n_samples)]
    h, _ = np.histogram(samples, bins=len(a))
    return h/np.sum(h)

width = 0.2
fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(np.arange(n), P, width, label='P(x)')
plt.bar(np.arange(n)+width, expected_gumbel(P, 10), width, alpha=0.25, label='10 samples')
plt.bar(np.arange(n)+2*width, expected_gumbel(P, 100), width, alpha=0.25, label='100 samples')
plt.bar(np.arange(n)+3*width, expected_gumbel(P, 1000), width, alpha=0.25, label='1000 samples')
plt.legend()
```

# (non)Smooth

What we really care about; Locality/smothness and order?

|      | Smooth           | Piecewise-smooth | Not smooth  |
| ------------- |:-------------:| -----:| -----:|
| __Ordered__     | cts | shattered |discrete |
| __Partially-Ordered__      | ?       |   ? | ? |
| __Not Ordered__      | reals without order!?       |   ? | categorical |

I mean smooth == local!?

## Piecewise-smooth

https://arxiv.org/abs/1702.08591

```python
import autograd.numpy as np
from autograd import grad
from autograd import elementwise_grad as egrad
```

```python
def relu(x):
    return np.maximum(x, 0)

def piecewise(x, n, fn):
    """
    Doesnt work if you only use scalars!?
    Needed to expand into a high dim.
    """
    x = np.expand_dims(x, 0)
    d = 10
    W = rnd.standard_normal((d, 1))
    x = np.dot(W, x)
    for i in range(n):
        a = rnd.standard_normal((d, d))
        b = rnd.standard_normal((d, 1))
        x = fn(np.dot(a, x) + b)

    W = rnd.standard_normal((1, d))
    x = np.dot(W, x)
    return np.squeeze(x)

def get_grad(n, fn):
    return egrad(lambda x: piecewise(x, n, fn))

x = np.linspace(-5, 5, 1000)

def plot(n):
    y = piecewise(x, n, relu)
    plt.plot(x, y/np.max(np.abs(y)), label='f(x) for {}'.format(n))
    d = get_grad(n, relu)(x)
    plt.plot(x, d/np.max(np.abs(d)), alpha=0.5, linestyle='--', label='df(x) for {}'.format(n))

plt.figure(figsize=(16, 8))

plt.subplot(3,1,1)
plot(2)
plt.ylabel('f(x)')
plt.title('Piecewise linear fns and their shattered grads')
plt.subplot(3,1,2)
plot(20)
plt.subplot(3,1,3)
plot(200)
plt.legend()
plt.xlabel('x')

# plots seem to be more shattered in the middle, is this because of the type of noise!?
```

```python
# TODO. show why this it bad. sample some points and plot their gradients
```

```python
def sigmoid(z, t=1):
    return 1/(1+np.exp(-z/t))

def plot(n):
    y = piecewise(x, n, sigmoid)
    plt.plot(x, y/np.max(np.abs(y)), label='f(x) for {}'.format(n))
    d = get_grad(n, sigmoid)(x)
    plt.plot(x, d/np.max(np.abs(d)), alpha=0.5, linestyle='--', label='df(x) for {}'.format(n))

plt.figure(figsize=(16, 8))
plt.subplot(3,1,1)
plt.ylabel('f(x)')
plt.title('Piecewise linear fns and their shattered grads')
plot(2)
plt.subplot(3,1,2)
plot(20)
plt.subplot(3,1,3)
plot(200)
plt.legend()
plt.xlabel('x')
```

Huh, I expected this to be more shattered. This cts fn doesnt do so badly. But what if I turn up the temperature/sharpness for the sigmoid?

Yep, that does it.

***

The problem is that for large n locality is lost.
For low n, locality can still remain as;
- you are operating within a cts domain
- partitioned the space into linear regions

__TODO__ (would like to make some of those partition plots!?)
What about tree search over the partitions!? They are each _weakly_ constrained by their neighbors.

### Local statistics

Mean, variance, line of best fit, SVD!?
what about the variance!? that could tell us how trust worthy the estimate is!?

Solution? Despite all the craziness in the gradient, just sample a bunch of times and average!?

```python
def plot_shattered_samples(x, n_samples, fn, use_subplot=False):
    def plotter(x, y):
        plt.plot(x, y/np.max(np.abs(y)), label='f(x) for {}'.format(n))
        d = get_grad(n, fn)(x)
        plt.plot(x, d/np.max(np.abs(d)), alpha=0.5,label='df(x) for {}'.format(n))

    def sample_avg_fit(loc, n_samples, width=25):
        r = int(loc*len(x))
        idx = rnd.randint(r-width, r+width, n_samples)
        best_fit = np.poly1d(np.polyfit(x[idx], y[idx]/np.max(np.abs(y)), 1))(x)
        return idx, best_fit

    plt.figure(figsize=(16, 8))
    y = piecewise(x, 30, fn)
    plt.subplot(3,1,1)
    plt.ylabel('y')
    plt.title('Local stats of shattered functions')

    plotter(x, y)
    idx, best_fit = sample_avg_fit(0.5, n_samples)
    plt.scatter(x[idx], y[idx]/np.max(np.abs(y)))
    plt.plot(x, np.clip(best_fit, -1, 1), alpha=0.5, linestyle='--')

    if use_subplot:
        plt.subplot(3,1,2)
        plotter(x, y)
    idx, best_fit = sample_avg_fit(0.35, n_samples)
    plt.scatter(x[idx], y[idx]/np.max(np.abs(y)))
    plt.plot(x, np.clip(best_fit, -1, 1), alpha=0.5, linestyle='--')

    if use_subplot:
        plt.subplot(3,1,3)
        plotter(x, y)
    idx, best_fit = sample_avg_fit(0.65, n_samples)
    plt.scatter(x[idx], y[idx]/np.max(np.abs(y)))
    plt.plot(x, np.clip(best_fit, -1, 1), alpha=0.5, linestyle='--')

    plt.legend()
    plt.xlabel('x')

plot_shattered_samples(x, 2, relu)
plot_shattered_samples(x, 5, relu)
```

```python

```

## Softened

What do we mean by soft?

* Cts relaxations: e.g. sigmoids as a cts step fn, gumbel-softmax as a cts argmax.
* Look up -- skip gradients and regularise.
* Soft attention: Weighted sum rather than indexing
* ?

## Cts relaxations

```python

```

### Binary choices

- Should I add a new node?
- Should I terminate a computation?

Add a new parameter, sigmoid and sample.

```python

```

```python
N = 10000
x = np.linspace(-1, 1, N)
y = (x > 0).astype(np.float32)
plt.plot(x, y, alpha=0.5, label='y')

# use finite differences to approximate as
# the gradident of greater than isnt defined.
dydx = (y[1:] - y[:-1])/(2/N)
plt.plot(x[1:], dydx/N, c='r', alpha=0.5, label='dydx')
plt.legend()
# scale the grad so we can make sense of it
```

```python
N = 10000
x = np.linspace(-1, 1, N)

def sigmoid(z, t=1):
    return 1/(1+np.exp(-z/t))
y = sigmoid(x, 0.1)
plt.plot(x, y, alpha=0.5, label='y')

# use finite differences to approximate as
# the gradident of greater than isnt defined.
dydx = (y[1:] - y[:-1])/(2/N)
plt.plot(x[1:], dydx, c='r', alpha=0.5, label='dydx')
plt.legend()
# scale the grad so we can make sense of it
```

```python
# what about many steps?
x = np.linspace(-1, 1, N)

def stepper(x, n):
    v = np.arange(n)
    m = len(x)//n
    return np.hstack([[v[i]]*m for i in range(n)])

n = 5
y = stepper(x, n)
plt.plot(x, y, label='y')

dydx = (y[1:] - y[:-1])/(2/N)
plt.plot(x[1:], dydx/N, c='r', alpha=0.5, label='dydx')
plt.plot(x, x*(n/2)+ n/2 - 0.5, label='my fn estimate')
plt.plot(x, np.ones_like(x)*(n/2), label='my grad estimate')
plt.legend()
```

```python
def soft_step(x, c, w):
    return 1/(1+np.exp(-w*(x+c)))

def soft_stepper(x, n):
    y = np.zeros_like(x)
    C = np.linspace(-1, 1, n)
    for i in range(n):
        y += soft_step(x, -C[i], 20)
    return y

y = soft_stepper(x, 5)
plt.plot(x, y, label='y')

dydx = (y[1:] - y[:-1])/(2/N)
plt.plot(x[1:], dydx, c='r', alpha=0.5, label='dydx')
plt.legend()

# not sure how helpful that is...
# hmm. all grad values are greater than zero,
# so if we were trying to do grad ascent then
# they should lead us in the right direction.

# fn is more accurate,
# but the gradient seems less helpful
```

### Softmax

Well that is just an extension of the sigmoids from before

What about the tradeoff between accurate representation and differentiability.
__TODO__ Want to quantify how inaccurate the gradients are!??!

```python
def softmax(z, t):
    return np.exp(z/t)/np.sum(np.exp(z/t))

x = np.linspace(-3, 3, 1000)
# just use sigmoid here to show visualisation.
# but the same applies to softmax
plt.plot(x, sigmoid(x, 10), label='temp=10')
plt.plot(x, sigmoid(x, 1), label='temp=1')
plt.plot(x, sigmoid(x, 1e-1), label='temp=1e-1')
plt.plot(x, sigmoid(x, 1e-2), label='temp=1e-2')
plt.legend()
```

```python

```

### Problems with 'soft' relaxations

Most use of 'soft' version of the function. To achieve this the gradients end up vanishing (sigmoids, ...)

## Weighted attention

```python
# instead of making a discrete choice, we take a weighted sum.
# The weights are optimised to be sparse.

```

```python

```

```python

```

## Partially-ordered

I meant in two ways
- Like a poset!
- Orderable

## Locality

You can have locality in graphs, as long as you have some __metric__ to order the nodes and construct the adjacency matrix.
So the metric is the key!?

So when we use the reals our metric is ???

```python
# connectedness using neighbors?!
# label propagation of loss?
# have some underlying topology of how fns are related.
# can query loss at some nodes and propagate loss the graph?!

# what if we did this for NNs?
# calculate their distance from each other in parameter space?
# MUST be better alternatives for measuing similarity/constructing the graph
# would be expensive to keep the actual parameter, but could easily store the graph

T = rnd.standard_normal((10,1))

def loss_fn(x):  # loss is distance from T
    return np.sum(np.power(x - T, 2))

n = 20
NNs = [rnd.standard_normal((10,1)) for _ in range(n)]
distances = np.array([[np.linalg.norm(a-b) for a in NNs] for b in NNs])
distances = distances / np.linalg.norm(distances, axis=0)  # is this the right axis?
losses = [loss_fn(nn) for nn in NNs[:n//2]] # only test the first half
losses = np.array(losses + [0]*(n//2))  # we dont know the loss of the other half
```

```python
# m = rnd.choice([0,1], (8,8,), p=[0.7, 0.3])
g = nx.from_numpy_matrix(distances)
g = nx.convert_node_labels_to_integers(g)
colors = range(len(g.edges))
nx.draw(g,pos=nx.spring_layout(G),
        node_color='#A0CBE2',
        edge_color=colors,
        width=4,
        edge_cmap=plt.cm.Blues,
        with_labels=False)
plt.show() # display
```

```python
losses
```

    array([20.2378646 , 21.95555553,  5.49127983, 13.96022894, 23.21754545,
           11.92296158, 10.38851321,  7.66327525, 17.79251064, 15.1077228 ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ])

```python
np.inner(distances, losses)
```

    array([29.621753  , 28.34841681, 32.16745621, 26.40204525, 27.44440327,
           28.02007125, 35.28426973, 28.58625361, 29.7459798 , 26.60329524,
           37.53892136, 34.57792728, 37.9942555 , 30.55105338, 33.86065416,
           33.57551757, 32.81755978, 35.64412152, 37.3795428 , 28.88069098])

```python
[loss_fn(nn) for nn in NNs[n//2:]] # only test the first half
```

    [15.664269938039999,
     16.604738263119025,
     17.569332354305367,
     8.057424451951501,
     13.049781370131504,
     18.407356390007173,
     15.68041532023816,
     22.842833849934504,
     17.322598039588833,
     11.243450623823959]

### Categorical

Embed and do VQ/k-means?
https://deepmind.com/research/publications/neural-discrete-representation-learning/

```python
# what about categorical variables?
def categorical(x, n):
    y = np.zeros_like(x)
    m = len(x)//n
    for i in range(n):
        # what if n is correlated with x
        y[m*i:m*(i+1)] = rnd.randint(0, n)
    return y

y = categorical(x, 10)
plt.plot(x, y, alpha=0.5, label='y')
dydx = (y[1:] - y[:-1])/(2/N)
plt.plot(x[1:], dydx/N, c='r', alpha=0.5, label='dydx')
plt.legend()
```

```python
# what about weak ordering of dimensions?
# DEFINE weak ordering!?
# somewhere in between discrete and categorical variables
# for example, below

def categorical(x, n):
    y = np.zeros_like(x)
    m = len(x)//n
    for i in range(n):
        # what if n is correlated with x
        y[m*i:m*(i+1)] = rnd.randint(0, n*i+1)
    return y

y = categorical(x, 30)
plt.plot(x, y, alpha=0.5, label='y')
dydx = (y[1:] - y[:-1])/(2/N)
plt.plot(x[1:], dydx/N, c='r', alpha=0.5, label='dydx')
plt.legend()

# what if indices have noise? cant trust the ordering!!!
# that is the main reason stochasticity is hard!?@?!#?
# what if we were allowed to perturb the indices and do a local sort/perturb?
# assuming the noise is ?? then this could work?
```

```python
def sort_1_step(x, y):
    # if your 2-neighbor is more similar to you
    # than your 1-neighbor then swap
    # we are assuming x is arbitrary so we can move it around.
    a = x.copy()
    b = y.copy()
    for i in range(len(x)-1):
        if b[i] > b[i+1]:
#             a[i:i] = list(reversed(a[i:i+2]))
            b[i:i+2] = list(reversed(b[i:i+2]))
    return a, b

x, y = sort_1_step(x, y)
plt.plot(x, y, alpha=0.5, label='y')
# dydx = (y[1:] - y[:-1])/(2/N)
# plt.plot(x[1:], dydx/N, c='r', alpha=0.5, label='dydx')
# plt.legend()
```

## Dice

* Why do we care about second order gradients?
* How did the old approaches fail?
*

$\dicei$

Generalising AD to stochastic graphs.

Why would we ever want to construct graphs with random variables in them? (not in them, but ?)
* Meta learning? Loss is a random variable (of the data)?
* RL?
*

Approaches

* Surrogate objective and AD (although objective is the wrong way to view it as it is independent of loss functions)
*

## Rebar
# Unknown

Two approaches (that I know of);

> model-free: _Use the unknown black box to estimate gradients using finite differences or ...?_

and

> model-based: _Construct a differentiable approximation of the black box._

|      | Unknown | Partial info |
| ------------- |:-------------:| -----:|
| __Efficient evaluation__ | model-free | ? |
| __Expensive evaluation__ | model-based|   ? |

* What information do we get from the black box? and what do we already know about it?
* do we want higher order gradients?

## Model-free

Can be expensive. Also, if noise is present in the blackbox, this can complicate things.
Alternate perspective, we are just fitting a cheap (linear) model.

***

Is there a more general formuation of differences that unifies; finite, temporal, counterfactual, ...?
What are these used for? Why do we care?

Quick review;

A derivative is defined as the limit of a finite difference.
$$
f'(x)=\lim _{h\to 0}{\frac {f(x+h)-f(x)}{h}}
$$

> The difference quotient is a measure of the average rate of change of the function over an interval

(that doesnt seem true, it just looks at the two ends, not how the fn changes inbetween? oh, maybe this is right?)

A counterfactual

* tbh, their experiments are abit dodgy? unclear if it works better than rebar.
* more accurate gradients means less noise... possibly less exploration..
* interesting problem. works ok for markov processes. but what about partial info settings. they are going to have biased estimators!?
* now we have a bunch of hyper param for $c_{\phi}$. What if it starts overfitting or diverges?
* one accurate estimators vs many less accurate ones!?
* i am a little confused as to what the contribution of this paper is... seems very incremental
* getting access to a_t created a longer dependency?
* relationship to Doubly Robust Estimation?

```python
$$
f(x) - f(x')
$$

A couple of notes.

* Even if the difference is not scaled properly (with h, or the run) the direction will still be correct.
* Estimating higher order derivatives with differences (does it get harder?).
* What if the function is noisy? A difference no longer gives a reliable estimate...
* Maybe I should look at target difference prop again?

## Higher order differences

https://en.wikipedia.org/wiki/Recurrence_relation#Relationship_to_difference_equations_narrowly_defined
More generally: the kth difference of the sequence an written as $$Δ k ( a_n ) {\displaystyle \Delta ^{k}(a_{n})} \Delta ^{k}(a_{n}) $$is defined recursively as

$$\Delta^k (a_n) = \Delta^{k−1}(a_{n+1}) − \Delta^{k−1}(a_n) = \sum^k_{t=0}  {k \choose t} (-1)^t a_{n+k-t}$$

What about for multidimensional variables!?
```

## Blackbox backpropagation with jacobian sensing

> Efficient coloring of G weak int for reducing the dimensionality of the problem, where each variable in the compressed problem corresponds to a subset of variables in the original problem.

#### Goal, recover the jacobian efficiently.

* If you happen to know the computation graph then just do AD.
* If you have partial information about the graph structure (sparsity structure of adjacency, then this is the solution!?)
    * What if the adjacency had probabilities of being connected, rather than abs values, 0, 1.
    * what if it has blocks?
    * symmetry?!? positive definiteness? rank? ...
* If you have partial information about the contents of the graph?
    * Edge types?
    * ?

Worst case. n function calls. Which for nns, might be ~1000000...

***

If we have a nested fn, such as $y = f_{\theta}(g_{\phi}(x))$, then what does this tell us about the structure of the jacobian (and/or hessian?)? Let $\psi = [\theta; \phi]$.
$$\frac{\partial y}{\partial \psi} =
\begin{bmatrix}
\frac{\partial y}{\partial \theta} \\
\frac{\partial y}{\partial z}\frac{\partial z}{\partial \phi} \\
\end{bmatrix}
$$

***

#### Goal, recover the graph efficiently.

In the non linear case, how do you represent the jacobian? Just use some arbitrary fn approximator? Can we find an approximator suited for this problem? What is unique to the problem?

* How can this be generalised to noisy settings?
* How can the sparsity structure be estimated? (what about noise...)
* Ok, so model-base vs model-free. In the model free version we must estimate the gradients every time. Finite difference ... So how well does this scale?
* What about non square matrices!?

```python
import autograd.numpy as np
from autograd import grad, jacobian
```

```python
def greedy_coloring(graph):
    """
    Args:
        graph (nx.graph): The graph to be colored

    Returns:
        (dict): mapping from colors (the keys) to nodes (the values)
    """
    nodes = list(graph.nodes())
    stable_sets = {0: [nodes[0]]}

    # for each node, put it in a stable set
    for v in nodes[1:]:
        done = False
        counter = 0
        while not done:
            # if we cannot add the node to any existing
            # stable sets, make a new one
            if counter not in list(stable_sets.keys()):
                stable_sets[counter] = [v]
                done = True
                break

            # add the curret node, v, to a stable set that
            # has none if its neighbors
            add = False
            for s in stable_sets[counter]:  # check each neighbor
                if (v, s) in graph.edges():
                    add = False
                    break
                else:
                    add = True
            if add:
                stable_sets[counter].append(v)
                break

            counter += 1
    return stable_sets
```

```python
m = np.random.randint(0, 2, (12,12))
g = nx.from_numpy_array(m)
coloring = greedy_coloring(g)
node_colors = list(sorted([(i, k) for k,v in coloring.items() for i in v]))
node_colors = [j for i, j in node_colors]
nx.draw(g, node_color=node_colors, with_labels=True)
```

```python
def rainbow(fn, m):
    # color the inputs
    g = nx.from_numpy_array(m)
    coloring = greedy_coloring(g)

    # peturb in the reduced dim space
    d_core = np.random.standard_normal(8)
    # ???
    ds = np.array([np.zeros(m.shape[0]) for k,v in coloring.items()])
    d = np.random.standard_normal((8,8))  # should be k by
    r = fn(d)

    # slove lp problem
    x = np.linalg.solve(r, d)
    return x
```

```python
def sigmoid(x):
    return 1/(1+np.exp(-x))

out = rainbow(sigmoid, np.random.randint(0,2,(8,8)))
out.shape
```

    (8, 8)

TODO. test the implementation.
Plot it's computational complexity.
?

## Estimating the interaction graph

Want to extend this work to estimating the interaction graph.
But, noise makes this nontrivial.

* How does a nonlinear functions interaction graph change over the space of its inputs (and parameters?).
* Want to recover a graph from data. Can use correlations to find direct connections, then higher order moments to find other connections?

```python
# if we take a vanilla neural network.
# its jacobian has low sparsity (wrt inputs, but what about params?)

def layer(shape):
    W = np.random.standard_normal(shape)
    return lambda x: sigmoid(np.dot(x, W))

def generate_rnd_fn(shape):

    def fn(x):
        l1 = layer((shape[0], 20))
        l2 = layer((20, shape[-1]))
        return l2(l1(x))

    return fn
```

```python
batch_size = 100
n_inputs = 28
n_outputs = 10
fn = generate_rnd_fn((n_inputs, n_outputs))
x = np.random.random((batch_size, n_inputs))
y = fn(x)
dfn = jacobian(fn)
j = dfn(x)
# j = np.mean(j, axis=(0,2)) # aggregate
```

```python
def expand(x, dims):
    for dim in dims:
        x = np.expand_dims(x, dim)
    return x
```

```python
def intersection_graph(mat, tol):
    # sparsify the jacobian by rounding small values to zero
    intersection = np.greater(mat, tol)
    intersection += np.greater(-mat, tol)
    return intersection.astype(np.float32)

intersection_graphs = [intersection_graph(j[i, :, i, :], 1e-1) for i in range(batch_size)]
```

```python
degrees = [np.max(np.sum(g, axis=0)) for g in intersection_graphs]
print(degrees)
```

    [6.0, 5.0, 6.0, 5.0, 5.0, 4.0, 6.0, 6.0, 7.0, 6.0, 4.0, 6.0, 5.0, 6.0, 5.0, 5.0, 7.0, 4.0, 5.0, 6.0, 5.0, 5.0, 6.0, 6.0, 5.0, 5.0, 5.0, 7.0, 6.0, 5.0, 7.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 4.0, 6.0, 6.0, 5.0, 7.0, 4.0, 7.0, 6.0, 6.0, 6.0, 5.0, 6.0, 5.0, 6.0, 6.0, 6.0, 7.0, 4.0, 7.0, 6.0, 4.0, 4.0, 5.0, 3.0, 5.0, 4.0, 5.0, 6.0, 7.0, 6.0, 6.0, 5.0, 4.0, 5.0, 6.0, 5.0, 6.0, 6.0, 5.0, 6.0, 5.0, 6.0, 6.0, 7.0, 6.0, 3.0, 6.0, 6.0, 5.0, 5.0, 5.0, 4.0, 6.0, 6.0, 7.0, 6.0, 6.0, 4.0, 6.0, 4.0, 6.0, 6.0]

```python
d = np.array([[np.sum(np.abs(a - b))
               for b in intersection_graphs]
              for a in intersection_graphs])
```

```python
plt.imshow(d)
```

```python

```

## ?

Wait a minute. If we have already compute a batch of activations, can we fit a linear model to these? How accurate is this to estimate the gradient?

(ps. why linear model. They are cheap. Only require 2 points per dimension!?)

```python
def batch_difference(y, x):
    # take a linear approximation over a batch
    dx = expand(x, [1]) - expand(x, [0])
    dy = expand(y, [1]) - expand(y, [0])
    # rise over run
    dydx = expand(dy.T, [-1])/(expand(dx, [0]) + 1e-8)
    return np.transpose(dydx, [1,0,2,3])

batch_size = 20
n_inputs = 784
n_outputs = 10
fn = generate_rnd_fn((n_inputs, n_outputs))
x = np.random.random((batch_size, n_inputs))
y = fn(x)
dfn = jacobian(fn)
j = dfn(x)
dydx = batch_difference(y, x)
np.sum(np.mean(np.power(dydx - j, 2), axis=(0,2)))
```

    355958565.4562501

# Efficient estimation

- What about approximate estimation?
- ?

## Model-based

- What about approximate estimation? What if our approximation is inaccurate? Relation to synthetic gradients?
- [World models](https://worldmodels.github.io/)

how can this be done efficiently? which priors are best for the types of models we will want to learn? how can we make a language for accepting arbitrary priors?

### Approximate with a neural network

?
- no guarantees for 'new' data
- what about when the neural network is first learning, the gradients will be nonsensible.
-

### Synthetic grads

?

```python

```

# Recent work

Putting the above tools together into an estimator for stochastic, discrete, and unknown functions.

## The gumbel-softmax trick

The reprarmeteriseation trick for stochastic dicrete variables.

$$
G=−log(−log(U)) \\
U \sim Unif[0,1] \\
P(X=k) = \alpha_k \\
X=\mathop{argmax}_k log(\alpha_k)+G_k\\
$$

https://cmaddis.github.io/

## Rebar

Our strategy will be to construct a control variate based on the difference between the REINFORCE gradient estimator for the relaxed model and the gradient estimator from the reparameterization trick.

how does this work with mini-batches!?

What is more similar to our function that another estimate of our fnction? Where can we get an accurate estimate of the mean of our fn?

$$
\begin{align*}
\mathbb E_{p(b)} [f(b, \theta)]& && \text{Want to maximise}\\
\frac{\partial f}{\partial \theta} \mathbb E_{p (b)} [f(b, \theta)]&= \sum \big[ \frac{\partial p(b)}{\partial \theta} f(b, \theta) +  p(b) \frac{\partial f(b, \theta)}{\partial \theta} \big] && \text{do GD on 1.}\\
x &=1   && x \text{ must be one}
\end{align*}
$$

https://github.com/tensorflow/models/tree/master/research/rebar

```python
def rebar():
    pass
```

## RELAX (Bprop through the void)

https://github.com/wgrathwohl/BackpropThroughTheVoidRL

When $c_{\phi} = f$, then LAX becomes the reparameterization estimator for $f$.

How to find $c_{\phi}$?

* $argmin_{\phi} (f(b) - c_{\phi}(b))^2$  (Relationship to actor-critic!? and qprop?!?)
* $argmin_{\phi} Var(\hat g)$

Seems very clear here? We are jointly learning a model of f(b)!
Are there pathogogical minima when trying to minimise variance? Seems dangerous?

***

Getting more info from stochastic gradients!
How can we measure the info in a set of feedback? Reliability/variance, first order vs 2nd order, ...?

***

* tbh, their experiments are abit dodgy? unclear if it works better than rebar.
* more accurate gradients means less noise... possibly less exploration..
* interesting problem. works ok for markov processes. but what about partial info settings. they are going to have biased estimators!?
* now we have a bunch of hyper param for $c_{\phi}$. What if it starts overfitting or diverges?
* one accurate estimators vs many less accurate ones!?
* i am a little confused as to what the contribution of this paper is... seems very incremental
* getting access to a_t created a longer dependency?
* relationship to Doubly Robust Estimation?

```python
def RELAX():
    pass
```

|      |  |            | Algol | Accuracy |
| ------------- |:-------------:| -----:| -----:|
| __Known__     | __Cts__ | __Not stochastic__ | vanilla | ? |
| __Unknown__   | __Cts__ | __Not stochastic__ | Approximate | ? |
| __Known__     | __Not Cts__ | __Not stochastic__ | ? | ? |
| __Unknown__   | __Not Cts__ | __Not stochastic__ | ? | ? |
| __Known__     | __Cts__ | __Stochastic__ | ? | ? |
| __Unknown__   | __Cts__ | __Stochastic__ | Approximate | ? |
| __Known__     | __Not Cts__ | __Stochastic__ | REBAR | ? |
| __Unknown__   | __Not Cts__ | __Stochastic__ | RELAX | ? |

__TODO__ Want to construct this from data!
- Evaluate the accuracies
- What about various methods;
    - model-free vs model-based.
    - reparam vs control variate vs score ...

# Partial

Problems
* Too big (a few different types of to big...)
* Local pertubations

Solutions
* Unroll
* Dynamic computation graphs
* Truncated backprop
* ?

# Resources

* http://timvieira.github.io/blog/
* https://casmls.github.io/general/2017/02/01/GumbelSoftmax.html

# Thoughts/questions
* Averaging over a set of samples: used it to help fix shattered grads, also used it to marginalise over noise. mini-batch SGD!
* Todo. Statistical manifold, uncertainty principle, local statistics, ...!?
* ?

```python

```