# Lecture 4: What are filters?
Audio Processing, MED4, Aalborg University, 2020

By Jesper Kjær Nielsen (jkn@create.aau.dk), Audio Analysis Lab, Aalborg University.

Last edited: 2020-02-17

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig
```

Central aspects of the course
- [x] What is sound?
- [x] How is sound generated?
- [x] How is sound turned into signals (i.e., data) on a computer?
- [ ] **How can we analyse these signals (i.e. extract information from them)?**
- [ ] How can we modify these signals?

Table of Contents

1  Lecture 4: What are filters?

1.1  Introduction to filtering

1.2  Feedforward filters and their impulse responses

1.3  Frequency response of feedforward filters

1.4  Introduction to the feedback filter

## Introduction to filtering
In the next 20 minutes, you will learn
- that you are already using filters in your everyday life
- how you can understand filtering in connection with audio

### Some filters from our everyday life:

#### What about filters for audio?

#### Example: measuring speed using filtering

- Assume that we wish to show the **instantaneous speed** of a car to its driver:
- Unfortunately, we cannot measure the speed directly, but only the driven distance $s(t)$ every $T_\text{s}$ seconds.
- After sampling, the discrete-time signal describing the the driven distance is $s_n=s(t_n)$.

To compute the **instantaneous speed** $v_n$ from $s_n$, we use that the speed of an object is **how much the object moves per unit time**. Thus, we could compute the **instantaneous speed** as
$$
    v_n = \frac{s_n-s_{n-1}}{T_\text{s}} = f_\text{s}(s_n-s_{n-1}) = f_\text{s}s_n-f_\text{s}s_{n-1}\ .
$$

The computation of the instantaneous speed can be interpreted as a **filter** given by
$$
    v_n = f_\text{s}s_n-f_\text{s}s_{n-1}
$$
and this is a so-called **difference equation**.
- Filter input: the discrete-time signal $s_n$ (position)
- Filter output: the discrete-time signal $v_n$ (speed)
- Filtering operation: a function which maps the input signal into the output signal

If we loook at the difference equation
$$
    v_n = f_\text{s}s_n-f_\text{s}s_{n-1}\ ,
$$
we have to do three things to do filtering:
1. **multiply** numbers (e.g., $ax_n$)
2. **sum** numbers (e.g., $x_n+s_n$)
3. **delay** samples (e.g., $x_{n-m}$)

We can also illustrate these three operations graphically.

The filter
$$
    v_n = f_\text{s}s_n-f_\text{s}s_{n-1}\ ,
$$
can then be illustrated as

### A digital filter
We have the following components in a digital filter:
- **Filter input**: a discrete-time signal $x_n$
- **Filter output**: a discrete-time signal $y_n$
- **Filter operation**: a function $f(y_{n-1}, y_{n-2}, \ldots, y_{n-p}, x_n, x_{n-1}, x_{n-2}, \ldots, x_{n-q})$ which computes the current output sample $y_n$ from
 - the current input sample $x_n$ and $q$ past input samples $x_{n-1}, x_{n-2}, \ldots, x_{n-q}$ and
 - the past $p$ output samples $y_{n-1}, y_{n-2}, \ldots, y_{n-p}$

---
Note that the filter output can, in principle, also depend on future values of the input. However, we will not cover this here.

Two main types of filters:
- **Feedforward filter**: the filter output only depends on the filter inputs, i.e.,
$$
    y_n = f(x_n, x_{n-1}, x_{n-2}, \ldots, x_{n-q})
$$
 - The number of past output samples $q$ is zero
 - The feedforward filter is also often referred to a finite impulse response (FIR) filter. More on why later.

- **Feedback filter**: the filter can depend on both the input and output signals, i.e.,
$$
    y_n = f(y_{n-1}, y_{n-2}, \ldots, y_{n-p}, x_n, x_{n-1}, x_{n-2}, \ldots, x_{n-q})
$$
 - The number of past output samples $p$ is at least 1
 - The feedback filter is also often referred to an infinite impulse response (IIR) filter. More on why later.

#### Example: measuring speed using filtering
We can say the following about the filter
$$
    v_n = f_\text{s}s_n-f_\text{s}s_{n-1}.
$$

- The filter input is the signal $s_n$
- The filter output is the signal $v_n$
- The filter is a feedforward filter since the output only depends on the input signal. Thus, $p=0$.
- The order $q$ is one, i.e., the filter is a first-order FIR filter

### Summary
1. Filtering can be based on
 - object size: e.g., coffee and fish
 - wavelength/**frequency**: e.g., light and **audio**
 - space: e.g., microphone array
2. Generally, you can think of a filter as something which extracts what you are interested in.

3. We usually draw a filter as in the figure below where
 - **Filter input**: a discrete-time signal $x_n$
 - **Filter output**: a discrete-time signal $y_n$
 - **Filter operation**: computes the current output sample as a combination of the **current and past input samples** and the **past output samples** (we have not seen the latter yet, though)

### Active break
1. Discuss with your neighbour what the three blocks illustrated below do. What eactly means $x_n$ (discussed in the last lecture) and does it relate to sampling?

Assume that we have a filter given by
$$
    y_n = x_n + b_1 x_{n-1}\ .
$$
2. Illustrate this filter graphically.
3. Let us assume that $x_n = 1$ for $n=0$ and zero otherwise. Compute $y_n$ by hand for $n = 0,1,\ldots,4$.

Now assume that we have the slightly more complicated filter
$$
    y_n = b_0 x_n+b_1 x_{n-1}+b_2x_{n-2}\ .
$$
4. Repeat 2. and 3. from above with this new filter.

## Feedforward filters and their impulse responses
In the next 20 minutes, you will learn
- what a feedforward filter is
- what the impulse response of a filter is

### Difference equation
The difference equation of a feedforward filter is
\begin{align}
    y_n &= b_0x_n + b_1 x_{n-1} + \cdots + b_q x_{n-q}\\
    &= \sum_{m=0}^{q} b_m x_{n-m}
\end{align}
where
- $x_n$ is the input signal
- $y_n$ is the output signal
- $b_0, b_1, \ldots, b_q$ are **filter coefficients**
- we can refer to the past values $x_{n-1}, x_{n-2}, \ldots, x_{n-q}$ as the **filter state**.

### Impulse response of a feedforward filter
#### Example: a simple filter
Let us look at the filter
$$
    y_n = x_n+b_1 x_{n-1}\ .
$$
Thus, we have
- $q=1$
- $b_0=1$
- the filter state is $x_{n-1}$

Let us try to send an impulse
$$
    \delta_n = \begin{cases}
        1 & n=0\\
        0 & \text{otherwise}
    \end{cases}
$$
through the filter.

When we set $x_n=\delta_n$, we get
$$
    y_n = \begin{cases}
        1 & n=0\\
        b_1 & n=1\\
        0 & \text{otherwise}
    \end{cases}\ .
$$

The filter's response to an impulse is referred to as the filter's **impulse response** and usually denoted by $h_n$, i.e., $y_n=h_n$ when $x_n=\delta_n$.

#### Example: a slightly more complicated filter
Let us look at the filter
$$
    y_n = b_0 x_n+b_1 x_{n-1}+b_3 x_{n-3}\ .
$$
Thus, we have
- $q=3$
- $b_2=0$
- the filter state is $x_{n-1}, x_{n-2}, x_{n-3}$

Let us try to send an impulse
$$
    \delta_n = \begin{cases}
        1 & n=0\\
        0 & \text{otherwise}
    \end{cases}
$$
through the filter.

When we set $x_n=\delta_n$, we obtain the impulse response $h_n=y_n$ to
$$
    h_n = \begin{cases}
        b_0 & n=0\\
        b_1 & n=1\\
        b_3 & n=3\\
        0 & \text{otherwise}
    \end{cases}\ .
$$

#### Impulse response of a general feedforward filter
If an impulse $\delta_n$ is send through a feedforward filter of the form
\begin{align}
    y_n &= b_0x_n + b_1 x_{n-1} + \cdots + b_p x_{n-q}\\
    &= \sum_{m=0}^{q} b_m x_{n-m}\ ,
\end{align}
we obtain the impulse response $h_n = y_n$ as
$$
    h_n = \begin{cases}
        b_n & 0\leq n \leq q\\
        0 & \text{otherwise}
    \end{cases}\ .
$$

Since the filter output goes to and stays at 0 for sampling indices $n>q$, the feedforward filter is usually referred to as the **finite impulse response (FIR)** filter.

```python
def firFiltering(filterCoefficients, inputSignal, filterState=None):
    nData = np.size(inputSignal)
    firFilterOrder = np.size(filterCoefficients)-1
    # set the filter state to zeros if not specified
    if filterState is None:
        filterState = np.zeros(firFilterOrder)
    # do the filtering
    outputSignal = np.zeros(nData)
    for n in np.arange(nData):
        outputSignal[n] = filterCoefficients[0]*inputSignal[n]
        for m in np.arange(firFilterOrder):
            outputSignal[n] = outputSignal[n] + \
                filterCoefficients[m+1]*filterState[m]
        filterState = np.r_[inputSignal[n],filterState[0:firFilterOrder-1]]
    return outputSignal, filterState
```

```python
# compute the impulse response
nData = 10
impulse = np.r_[1, np.zeros(nData-1)]
samplingIndices = np.arange(nData)
filterCoefficients = np.array([1.1, -0.5, 0, 0.5, 0.25])
impulseResponse, filterState = firFiltering(filterCoefficients, impulse)
```

```python
# plot the results
plt.figure(figsize=(10,6))
plt.stem(samplingIndices, impulse,'b', markerfmt='bo', label="$\delta_n$", use_line_collection=True)
plt.stem(samplingIndices, impulseResponse, 'r', markerfmt='rx', label="$h_n$", use_line_collection=True)
plt.legend()
plt.xlim((samplingIndices[0]-0.1,samplingIndices[nData-1])), plt.ylim((-0.5,1.1))
plt.xlabel('Sampling index [.]'), plt.ylabel('Signal value [.]');
```

### Summary
1. A feedforward filter has the **difference equation**
\begin{align}
    y_n &= b_0x_n + b_1 x_{n-1} + \cdots + b_p x_{n-q}= \sum_{m=0}^{q} b_m x_{n-m}\ .
\end{align}
2. If we send an impulse $\delta_n$ through a filter, we obtain the filter's **impulse response**.
3. For a general feedforward filter, the impulse response is **finite** and given by
$$
    h_n = \begin{cases}
        b_n & 0\leq n \leq q\\
        0 & \text{otherwise}
    \end{cases}\ .
$$

### Active break
1. Describe in words what an impulse response is and how you can compute it.
2. Sketch the impulse response of the following feedforward filter
$$
    y_n = \frac{1}{q+1}\sum_{m=0}^q x_{n-m}\ .
$$
What are the filter coefficients $b_0, b_1, \ldots, b_q$?
3. Simulate samples for $n=0,1,\ldots,47999$ from the following input signal
$$
    x_n = \cos(2\pi f_1 n/f_\text{s}) + \cos(2\pi f_2 n/f_\text{s})
$$
where $f_\text{s} = 48$ kHz, $f_1 = 500$ Hz, and $f_1 = 5$ kHz. (Tip: use the function `genSinusoid` from the last block of lecture 2). Plot the first 200 samples of the signal $x_n$.
4. Compute $y_n$ by filtering $x_n$ through the filter above (Tip: you can use [scipy's `lfilter` function](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.lfilter.html#scipy.signal.lfilter) instead of mine from above.
5. Plot the first 200 samples of $x_n$ and $y_n$ in the same plot. What does the filter seem to do? Can you also hear this if you play back the signals?

## Frequency response of feedforward filters
In the next 20 minutes, you will learn
- what the frequency response of a filter is
- what the frequency response of a feedforward filter is
- what the amplitude and phase response of a filter is

#### Example: a first order FIR filter
Let us again look at the simple filter
$$
    y_n = x_n+b_1 x_{n-1}
$$
which we also looked at before and now know is a **first order FIR filter** with impulse response
$$
    h_n = \begin{cases}
        1 & n=0\\
        b_1 & n=1\\
        0 & \text{otherwise}
    \end{cases}\ .
$$

Instead of an impulse, let us now try to send the phasor $\mathrm{e}^{j\omega n}$ through the filter.

We set $x_n=\mathrm{e}^{j\omega n}$ and get
\begin{align}
    y_n &= \mathrm{e}^{j\omega n} + b_1\mathrm{e}^{j\omega (n-1)} = \mathrm{e}^{j\omega n} + b_1\mathrm{e}^{-j\omega }\mathrm{e}^{j\omega n}\\
    &= \left[1+b_1\mathrm{e}^{-j\omega }\right] \mathrm{e}^{j\omega n}\ .
\end{align}
Note that
- we used the rule $a^n a^m = a^{n+m}$
- the filter output is simply the real value, i.e., $\text{Re}(y_n)$, if we set $x_n=\text{Re}(\mathrm{e}^{j\omega n})=\cos(\omega n)$.

Thus, the output signal is simply the phasor (input signal $x_n$) multiplied by the complex number
$$
    H(\omega) = 1+b_1\mathrm{e}^{-j\omega }
$$
which depends on the digital frequency $\omega$ and filter coefficients $1$ and $b_1$, but not on the sampling index $n$.

When viewed as a function of the digital frequency $\omega$, we call this complex number for the filter's **frequency response**.

Using Euler's formula
$$
    \mathrm{e}^{j\theta} = \cos(\theta)+j\sin(\theta)\ ,
$$
we can write the frequency response $H(\omega)$ in a rectangular form as
$$
    H(\omega) = 1+b_1\cos(\omega)-jb_1\sin(\omega)
$$
which has $1+b_1\cos(\omega)$ as the real part and $-b_1\sin(\omega)$ as the imaginary part.

More interestingly, the polar form of the frequency response can be written as
$$
    H(\omega) = |H(\omega)|\mathrm{e}^{j\angle H(\omega)}
$$
where
\begin{align}
    |H(\omega)| &= \sqrt{(1+b_1\cos(\omega))^2+(-b_1\sin(\omega))^2} = \sqrt{1+b_1^2+2b_1\cos(\omega)}\\
    \angle H(\omega) &= \mathrm{arctan}\left(\frac{-b_1\sin(\omega)}{1+b_1\cos(\omega)}\right)\ .
\end{align}
Note that $|H(\omega)|$ and $\angle H(\omega)$ are usually referred to as the **amplitude response** and **phase response**, respectively.

```python
def firFreqResp(filterCoefficients, freqGrid):
    nFreqs = np.size(freqGrid)
    firFilterOrder = np.size(filterCoefficients)-1
    # compute the frequency response
    freqResp = filterCoefficients[0]*np.ones(nFreqs)
    for m in np.arange(firFilterOrder)+1:
        freqResp = freqResp + filterCoefficients[m]*np.exp(-1j*m*freqGrid)
    return freqResp
```

```python
# compute the frequency response
nFreqs = 100
freqGrid = 2*np.pi*np.arange(nFreqs)/nFreqs
firFilterCoefficients = np.array([1, 0.9])
_, freqResp = sig.dfreqresp((firFilterCoefficients, 1, 1), freqGrid)
ampResp = np.abs(freqResp)
phaseResp = np.angle(freqResp)
```

```python
# plot the results
plt.figure(figsize=(10,6))
plt.subplot(2,1,1)
plt.plot(freqGrid, ampResp, linewidth=2)
plt.xlim((freqGrid[0],freqGrid[nFreqs-1])), plt.ylim((0,1.1*np.max(ampResp)))
plt.xlabel('Dig. freq. [rad/sample]'), plt.ylabel('Amplitude response [.]');
plt.subplot(2,1,2)
plt.plot(freqGrid, phaseResp, linewidth=2)
plt.xlim((freqGrid[0],freqGrid[nFreqs-1])), plt.ylim((-np.pi,np.pi))
plt.xlabel('Dig. freq. [rad/sample]'), plt.ylabel('Phase response [.]');
```

#### Example: filtering two phasors
Assume that we have the signal
$$
    x_n = \mathrm{e}^{j\omega_1 n} + \mathrm{e}^{j\omega_2 n}
$$
where $\omega_1=2\pi f_1/f_\text{s}$ and $\omega_2=2\pi f_2/f_\text{s}$ are **digital frequencies** (see last lecture).

The filter output is then
\begin{align}
    y_n &= x_n+b_1 x_{n-1} = H(\omega_1)\mathrm{e}^{j\omega_1 n}+H(\omega_2)\mathrm{e}^{j\omega_2 n}\\
    &= A_1 \mathrm{e}^{j\omega_1 n+\Phi_1}+ A_2 \mathrm{e}^{j\omega_2 n+\Phi_2}
\end{align}
where the amplitudes and phases of the sinusoids for $i=\{1,2\}$ are given by
\begin{align}
    A_i &= |H(\omega_i)| = \sqrt{1+b_1^2+2b_1\cos(\omega_i)}\\
    \Phi_i &= \angle H(\omega_i) = \mathrm{arctan}\left(\frac{-b_1\sin(\omega_i)}{1+b_1\cos(\omega_i)}\right)\ .
\end{align}

Thus, the filter changes the **amplitudes** and **phases** of the sinusoids, but not the **frequencies**!

### Frequency response of a general feedforward filter
If we send a phasor $\mathrm{e}^{j\omega n}$ through the feedforward filter
\begin{align}
    y_n &= b_0x_n + b_1 x_{n-1} + \cdots + b_q x_{n-q} = \sum_{m=0}^{q} b_m x_{n-m}\ ,
\end{align}
we obtain
$$
    y_n = \sum_{m=0}^{q} b_m \mathrm{e}^{j\omega (n-m)} = \mathrm{e}^{j\omega n} \sum_{m=0}^{q} b_m \mathrm{e}^{-j\omega m}\ .
$$

Thus, the **frequency response** of the feedforward filter is
$$
    H(\omega) = b_0 + b_1\mathrm{e}^{-j\omega} + \cdots + b_q \mathrm{e}^{-j\omega q} = \sum_{m=0}^{q} b_m \mathrm{e}^{-j\omega m}\ .
$$

### The transfer function
When we derived the frequency response, we used a phasor as the input signal, i.e., $x_n=\mathrm{e}^{j\omega n}$. If we set $x_n=z^n$ as the input signal, where $z$ in an arbitrary complex number, we get
$$
    y_n = \sum_{m=0}^{q} b_m z^{n-m} = z^n \sum_{m=0}^{q} b_m z^{-m}\ .
$$
Analogously to the frequency response, we see that the output signal is a multiplication between the input signal and a complex number $H(z)$ given by
$$
    H(z) = b_0 + b_1z^{-1} + \cdots + b_q z^{-q} = \sum_{m=0}^{q} b_m z^{-m}\ .
$$
This complex number is referred to as the filter's **transfer function**.

If we write the complex number $z$ in its polar form $z=r\mathrm{e}^{j\omega}$, we see that
$$
    z^n = (r\mathrm{e}^{j\omega})^n = r^n\mathrm{e}^{j\omega n}\ .
$$
Thus, the frequency response is a special case of the transfer function for $r=1$!

We often prefer working with the transfer function for a couple of reasons:
1. Simpler notation since we can write $z$ instead of $\mathrm{e}^{j\omega}$.
2. The frequency response is a special case of the transfer function (when we set $r=1$ in the latter).
3. The transfer function allow us to make a more thorough analysis of the filter.

We will say much more on the transfer function in lecture 6.

### Summary
1. If we send a phasor $\mathrm{e}^{j\omega n}$ through a filter, we obtain the filter's **frequency response** at the frequency $\omega$.
2. For a general feedforward filter, the frequency response is
$$
    H(\omega) = \sum_{m=0}^{q} b_m \mathrm{e}^{-j\omega m}
$$
which is just a complex number whose
 - magnitude $|H(\omega)|$ is known as the **amplitude response**
 - phase $\angle H(\omega)$ is known as the **phase response**.

### Active break
Consider the simple filter with difference equation
$$
    y_n = 0.5x_n+0.5x_{n-1}\ .
$$
1. What can you say about this filter (type, filter coefficients,  and order $q$)? Make a drawing of the filter.
2. Compute the filter's impulse response $h_n$.
3. Compute the filter's frequency response $H(\omega)$, amplitude response $|H(\omega)|$, and phase response $\angle H(\omega)$. Feel free to use either a computer or pen and paper.
4. Do exercise 1 in the *Introduction to Audio Processing* book (p. 60). You can use Python instead of Pure Data.

---
Tip: Send an impulse $\delta_n$ and a phasor $\mathrm{e}^{j\omega n}$ through the filter to get the impulse response and the frequency response, respectively.  Moreover, a feedforward filter has the difference equation
\begin{align}
    y_n &= b_0x_n + b_1 x_{n-1} + \cdots + b_q x_{n-q} = \sum_{m=0}^{q} b_m x_{n-m}\ .
\end{align}

## Introduction to the feedback filter
In the next 20 minutes, you will learn
- what a feedback filter is
- what the impulse response and frequency response of a feedback filter is
- what a resonance filter is

### Feedback filter
The difference equation of a feedback filter (with $q=1$) is
\begin{align}
    y_n &= b_0x_n + a_1y_{n-1} + \ldots + a_py_{n-p} = b_0x_n + \sum_{m=0}^{p} a_m y_{n-m}
\end{align}
where
- $x_n$ is the input signal
- $y_n$ is the output signal
- $b_0, a_1, a_2, \ldots, a_p$ are **filter coefficients**
- we often refer to the past values $y_{n-1}, y_{n-2}, \ldots, y_{n-p}$ as the **filter state**.

---
Note that general feedback filters can have $q>1$, but we wait with this general case until lecture 6.

### Impulse response of a feedback filter
#### Example: a simple filter
Let us look at the filter
$$
    y_n = x_n+a_1 y_{n-1}\ .
$$
Thus, we have
- $p=0$ and $q=1$
- $b_0=1$
- the filter state is $y_{n-1}$

Let us try to send an impulse
$$
    \delta_n = \begin{cases}
        1 & n=0\\
        0 & \text{otherwise}
    \end{cases}
$$
through the filter.

When we set $x_n=\delta_n$, we get the impulse response $h_n=y_n$ given by
$$
    h_n = \begin{cases}
        a_1^n & n\geq 0\\
        0 & \text{otherwise}
    \end{cases}\ .
$$
Note that $h_n$ never becones zero for $n\geq 0$. Therefore, feedback filters are referred to as **infinite impulse response** (IIR) filters.
- Unlike an FIR filter, the impulse response of a general IIR filter is not simply linked to the filter coefficients.

```python
def iirFiltering(filterCoefficients, inputSignal, filterState=None):
    # NOTE: this is only a special case of an IIR filter where the there is no feedforward part,
    # except for x_n which is unscaled (i.e., p=0 and b_0=1).
    nData = np.size(inputSignal)
    iirFilterOrder = np.size(filterCoefficients)
    # set the filter state to zeros if not specified
    if filterState is None:
        filterState = np.zeros(iirFilterOrder)
    # do the filtering
    outputSignal = np.zeros(nData)
    for n in np.arange(nData):
        outputSignal[n] = inputSignal[n]
        for m in np.arange(iirFilterOrder):
            outputSignal[n] = outputSignal[n] + \
                filterCoefficients[m]*filterState[m]
        filterState = np.r_[outputSignal[n],filterState[0:iirFilterOrder-1]]
    return outputSignal, filterState
```

```python
# compute the impulse response
nData = 25
impulse = np.r_[1, np.zeros(nData-1)]
samplingIndices = np.arange(nData)
filterCoefficients = np.array([0.7])
impulseResponse, filterState = iirFiltering(filterCoefficients, impulse)
# plot the results
plt.figure(figsize=(10,6))
plt.stem(samplingIndices, impulse,'b', markerfmt='bo', label="$\delta_n$")
plt.stem(samplingIndices, impulseResponse, 'r', markerfmt='rx', label="$h_n$")
plt.legend()
plt.xlim((samplingIndices[0]-0.1,samplingIndices[nData-1])),
plt.ylim((np.minimum(np.min(impulseResponse),0),np.max(impulseResponse)))
plt.xlabel('Sampling index [.]'), plt.ylabel('Signal value [.]');
```

### Frequency response of a feedback filter
#### Example: a resonance filter
Let us look at the filter
$$
    y_n = b_0x_n+a_1 y_{n-1}+a_2 y_{n-2}\ .
$$
Thus, we have
- $p=2$ and $q=0$
- the filter state is $y_{n-1}, y_{n-2}$

For particular values of the filter coefficients, we obtain a **resonance filter** which we will talk more about later.

Let us now try to send the phasor $\mathrm{e}^{j\omega n}$ through the filter.

We set $x_n=\mathrm{e}^{j\omega n}$ and get
\begin{align}
    y_n &= b_0\mathrm{e}^{j\omega n} +a_1 y_{n-1}+a_2 y_{n-2}
\end{align}
which we can rewrite as
$$
    b_0\mathrm{e}^{j\omega n} = y_n - a_1 y_{n-1} - a_2 y_{n-2}\ .
$$

If we guess that the output signal is a multipication of the input phasor and the frequency response, i.e., $y_n=H(\omega)\mathrm{e}^{j\omega n}$, we obtain
\begin{align}
    b_0\mathrm{e}^{j\omega n} &= H(\omega)\mathrm{e}^{j\omega n} - a_1 H(\omega)\mathrm{e}^{j\omega (n-1)} - a_2 H(\omega)\mathrm{e}^{j\omega (n-2)}\\
    &= (1-a_1\mathrm{e}^{-j\omega }-a_2\mathrm{e}^{-j2\omega})H(\omega)\mathrm{e}^{j\omega n}
\end{align}
from which we get that
$$
    H(\omega) = \frac{b_0}{1-a_1\mathrm{e}^{-j\omega }-a_2\mathrm{e}^{-j2\omega}}\ .
$$
The amplitude response $|H(\omega)|$ and the phase response $\angle H(\omega)$ are defined in exactly the same way as for the feedforward filter.

#### Frequency response of a general feedback filter
A filter with the difference equation
\begin{align}
    y_n &= b_0x_n + a_1y_{n-1} + \ldots + a_py_{n-q} = b_0x_0 + \sum_{m=0}^{p} a_m y_{n-m}
\end{align}
has the frequency response
$$
    H(\omega) = \frac{b_0}{1-\sum_{m=1}^p a_m\mathrm{e}^{-j\omega m}}\ .
$$

```python
# compute the frequency response
nFreqs = 100
freqGrid = 2*np.pi*np.arange(nFreqs)/nFreqs
iirFilterCoefficients = np.array([1, -0.8])
_, freqResp = sig.dfreqresp((1, iirFilterCoefficients, 1), freqGrid)
ampResp = np.abs(freqResp)
phaseResp = np.angle(freqResp)
```

```python
# plot the results
plt.figure(figsize=(10,6))
plt.subplot(2,1,1)
plt.plot(freqGrid, ampResp, linewidth=2)
plt.xlim((freqGrid[0],freqGrid[nFreqs-1])), plt.ylim((0,1.1*np.max(ampResp)))
plt.xlabel('Dig. freq. [rad/sample]'), plt.ylabel('Amplitude response [.]');
plt.subplot(2,1,2)
plt.plot(freqGrid, phaseResp, linewidth=2)
plt.xlim((freqGrid[0],freqGrid[nFreqs-1])), plt.ylim((-np.pi,np.pi))
plt.xlabel('Dig. freq. [rad/sample]'), plt.ylabel('Phase response [.]');
```

### Resonance filter
Assume that we can write the frequency response of a feedback filter as
$$
    H(\omega) = \frac{b_0}{(1-p_1\mathrm{e}^{-j\omega})(1-p_1^*\mathrm{e}^{-j\omega})}
$$
where $p_1$ is a so-called **pole** which we are going to talk more about in lecture 6.

A pole is a complex number so we can write it in a polar form as
\begin{align}
    p_1 &= R\mathrm{e}^{j\theta}\\
    p_1^* &= R\mathrm{e}^{-j\theta}
\end{align}
where $R$ is the magnitude and $\theta$ an angle.

If we insert these expression for $p_1$ and $p_1^*$ in the frequency response, we obtain
\begin{align}
    H(\omega) &= \frac{b_0}{(1-R\mathrm{e}^{j\theta}\mathrm{e}^{-j\omega})(1-R\mathrm{e}^{-j\theta}\mathrm{e}^{-j\omega})}\\
    &= \frac{b_0}{1-2R\cos(\theta)\mathrm{e}^{-j\omega}+R^2\mathrm{e}^{-j2\omega}}\ .
\end{align}

We now see that the resonance filter has filter coefficients
\begin{align}
    a_1 &= 2R\cos(\theta)\\
    a_2 &= -R^2\ .
\end{align}

We now wish to choose $b_0$, $R$, and $\theta$ so that we get
- the desired center frequency $\omega_\text{c}$
- the desired bandwidth $B_\text{w}$
- the desired gain at the center frequency

Unfortunately, it is a bit complicated to show, but it can be shown that
\begin{align}
    \omega_\text{c} &= \arccos\left(\frac{1+R^2}{2R}\cos(\theta)\right)\ ,\quad\text{if }\cos(\theta)\leq\frac{2R}{1+R^2}\\
    B_\text{w} &\approx 2-2R\\
    b_0 &= (1-R^2)\sin(\theta)\ .
\end{align}

From these equations, we can compute the values of $b_0$, $R$, and $\theta$ as
\begin{align}
    R &\approx \frac{2-B_\text{w}}{2}\\
    \theta &= \arccos\left(\frac{2R}{1+R^2}\cos(\omega_\text{c})\right)\\
    b_0 &= (1-R^2)\sin(\theta)\ .
\end{align}

```python
def resonanceFreqResp(centerFreq, bandwidth, freqGrid):
    nFreqs = np.size(freqGrid)
    poleRadius = (2-bandwidth)/2
    poleAngle = np.arccos(2*poleRadius*np.cos(centerFreq)/(1+poleRadius**2))
    gain = (1-poleRadius**2)*np.sin(poleAngle)
    iirCoefficients = np.array([2*poleRadius*np.cos(poleAngle), -poleRadius**2])
    freqResp = gain/(1-2*poleRadius*np.cos(poleAngle)*np.exp(-1j*freqGrid)+poleRadius**2*np.exp(-1j*2*freqGrid))
    return freqResp
```

```python
# compute the frequency response
nFreqs = 1000
samplingFreq = 1000;
digFreqGrid = 2*np.pi*np.arange(nFreqs)/nFreqs
freqGrid = digFreqGrid*samplingFreq/(2*np.pi)
centerFreq = 2*np.pi*100/samplingFreq
bandwidth = 2*np.pi*50/samplingFreq
freqResp = resonanceFreqResp(centerFreq, bandwidth, digFreqGrid)
ampResp = np.abs(freqResp)
phaseResp = np.angle(freqResp)
```

```python
# plot the results
plt.figure(figsize=(10,6))
plt.subplot(2,1,1)
plt.plot(freqGrid, ampResp, linewidth=2)
plt.xlim((freqGrid[0],freqGrid[nFreqs-1])), plt.ylim((0,1.1*np.max(ampResp)))
plt.xlabel('Frequency [Hz]'), plt.ylabel('Amplitude response [.]');
plt.subplot(2,1,2)
plt.plot(freqGrid, phaseResp, linewidth=2)
plt.xlim((freqGrid[0],freqGrid[nFreqs-1])), plt.ylim((-np.pi,np.pi))
plt.xlabel('Frequency [Hz]'), plt.ylabel('Phase response [.]');
```

### Summary
1. A feedback filter (with no feedforward part, i.e., $q=0$) has the difference equation
\begin{align}
    y_n &= b_0x_n + a_1y_{n-1} + \ldots + a_py_{n-p} = b_0x_n + \sum_{m=0}^{p} a_m y_{n-m}\ .
\end{align}
2. The feedback filter is often referred to as an **infinite impulse response** (IIR) filter since its output never becomes zero for a finite sampling index.
3. The frequency response of an IIR filter (with $q=0$) is
$$
    H(\omega) = \frac{b_0}{1-\sum_{m=1}^p a_m\mathrm{e}^{-j\omega m}}\ .
$$
4. A resonance filter is a special kind of filter with the difference equation
\begin{align}
    y_n &= b_0x_n + 2R\cos(\theta)y_{n-1} - R^2y_{n-2}\ .
\end{align}

### Active break
Do exercise 2-4 in the *Introduction to Audio Processing* book (p. 60). You can use Python instead of Pure Data.


```python
pip install mystic
```

    Collecting mystic
      Downloading mystic-0.3.8-py2.py3-none-any.whl (385 kB)
         |████████████████████████████████| 385 kB 16.2 MB/s eta 0:00:01
    Collecting klepto>=0.2.1
      Downloading klepto-0.2.1-py2.py3-none-any.whl (85 kB)
         |████████████████████████████████| 85 kB 5.3 MB/s  eta 0:00:01
    Requirement already satisfied: sympy>=0.6.7 in /home/vagrant/.pyenv/versions/3.7.2/envs/py3/lib/python3.7/site-packages (from mystic) (1.8)
    Collecting numpy<1.20.0,>=1.0
      Downloading numpy-1.19.5-cp37-cp37m-manylinux2010_x86_64.whl (14.8 MB)
         |████████████████████████████████| 14.8 MB 16.0 MB/s eta 0:00:01
    Requirement already satisfied: dill>=0.3.4 in /home/vagrant/.pyenv/versions/3.7.2/envs/py3/lib/python3.7/site-packages (from mystic) (0.3.4)
    Requirement already satisfied: pox>=0.3.0 in /home/vagrant/.pyenv/versions/3.7.2/envs/py3/lib/python3.7/site-packages (from klepto>=0.2.1->mystic) (0.3.0)
    Requirement already satisfied: mpmath>=0.19 in /home/vagrant/.pyenv/versions/3.7.2/envs/py3/lib/python3.7/site-packages (from sympy>=0.6.7->mystic) (1.2.1)
    Installing collected packages: numpy, klepto, mystic
      Attempting uninstall: numpy
        Found existing installation: numpy 1.20.3
        Uninstalling numpy-1.20.3:
          Successfully uninstalled numpy-1.20.3
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    oasys1-shadowoui-advanced-tools 1.0.85 requires oasys-srwpy>=1.0.5, which is not installed.
    oasys1-shadowoui-advanced-tools 1.0.85 requires oasys1>=1.2.89, which is not installed.
    tensorflow 2.3.1 requires h5py<2.11.0,>=2.10.0, but you have h5py 3.4.0 which is incompatible.
    tensorflow 2.3.1 requires numpy<1.19.0,>=1.16.0, but you have numpy 1.19.5 which is incompatible.
    sirepo-bluesky 0.3.0.post7+g9e00501 requires pyqt5<5.12,>=5.9, but you have pyqt5 5.15.4 which is incompatible.
    Successfully installed klepto-0.2.1 mystic-0.3.8 numpy-1.19.5
    WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
    You should consider upgrading via the '/home/vagrant/.pyenv/versions/py3/bin/python -m pip install --upgrade pip' command.
    Note: you may need to restart the kernel to use updated packages.

```python
import numpy as np
from mystic.tools import random_state
```

```python
# data=np.load('mesh_file.npy')
```

```python

def _random_samples(lb,ub,npts=1000):
  """
generate npts random samples between given lb & ub
Inputs:
    lower bounds  --  a list of the lower bounds
    upper bounds  --  a list of the upper bounds
    npts  --  number of sample points [default = 10000]
"""
  dim = len(lb)
  pts = random_state(module='numpy.random').rand(dim,npts)
  for i in range(dim):
    pts[i] = (pts[i] * abs(ub[i] - lb[i])) + lb[i]

  # np.savetxt("training.txt", pts.T)
  np.save( 'mesh_file.npy', pts)

  return pts

# Bounds of the parameter space: sheet_density and beam_charge
lb=[0.5e-9, 40e-6, 20e-6]
ub=[1.5e-9, 60e-6, 35e-6]

data=_random_samples(lb,ub,npts=500)
```

```python
# plot response surface
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm
import numpy as np
x_=data[0]
y_ =data[1]
z =data[2]
# plt.plot_wireframe(x_, y_, z_, rstride=d, cstride=d, color = 'green',linewidth = 1,
                          # antialiased = True, label='RBF')
```

```python
d=3

figure = plt.figure(figsize = (16,10))
kwds = {'projection':'3d'}
# fig, ax= plt.subplots(figsize = (12, 8))
LW=18
ax = figure.gca(**kwds)
ax.autoscale(tight=False)
ax.plot(x_*1.e9, y_*1.e6, z*1.e6, 'bo', linewidth=2, markersize=5)
ax.set_zlabel('$\\sigma_y$ [micron]',fontsize=20, labelpad=25)
ax.set_xlabel('Q [nC]',fontsize=20, labelpad=20)
ax.set_ylabel('$\\sigma_x$ [micron]',fontsize=20, labelpad=20)
ax.set_title('samples points N = '+str(len(x_)),fontsize=18)
# ax.grid(False)
ax.view_init(elev=40, azim=-50)
plt.savefig('training.png',dpi=199)
```

```python

```

```python
# import numpy
# data=np.load('mesh_file.npy')
```

```python
# len(data[0])
```

```python
# data[0][1], data[1][1], data[2][1]
```

```python

```


# A Simple Tree Example

```python
# Initialisations
from __future__ import unicode_literals
from __future__ import print_function

import sys
import os
from sympy import pprint
from copy import deepcopy

import gmpTools

from antares.core.settings import settings
from antares.core.unknown import Unknown, BHUnknown
from antares.particles.particles import Particles
```

```python
oTree = BHUnknown(helconf="pmpmpm", amppart="tree")
oUnknown = Unknown(oTree, load_partial_results=False, silent=True)
```

Scalings

```python
oUnknown.do_single_collinear_limits()
oUnknown.do_double_collinear_limits()
```

    Finished calculating single scalings. The partial result is:
    /⟨1|2⟩[1|2]⟨1|6⟩[1|6]⟨2|3⟩[2|3]⟨3|4⟩[3|4]⟨4|5⟩[4|5]⟨5|6⟩[5|6]s_123s_234s_345

    The mass dimension of the unknown is -2.
    The phase weights of the unknown are [-2, 2, -2, 2, -2, 2].
    The mass dimension of the new unknown is 16.
    The phase weights of the new unknown are [-2, 2, -2, 2, -2, 2].

    Finished calculating pair scalings. They are:
    [⟨1|2⟩, [1|2]]: 1.0, 2  → 2
    [⟨1|2⟩, ⟨1|6⟩]: 1.0, 30 → 5
    [⟨1|2⟩, [1|6]]: 1.0, 3  → 2
    [⟨1|2⟩, ⟨2|3⟩]: 1.0, 31 → 5
    [⟨1|2⟩, [2|3]]: 1.0, 3  → 2
    [⟨1|2⟩, ⟨3|4⟩]: 1.0, 2  → 2
    [⟨1|2⟩, [3|4]]: 2.0, 12 → 3
    [⟨1|2⟩, ⟨4|5⟩]: 1.0, 10 → 2
    [⟨1|2⟩, [4|5]]: 2.0, 3  → 2
    [⟨1|2⟩, ⟨5|6⟩]: 1.0, 2  → 2
    [⟨1|2⟩, [5|6]]: 2.0, 12 → 3
    [⟨1|2⟩, s_123]: 2.0, 10 → 3
    [⟨1|2⟩, s_234]: 1.0, 2  → 2
    [⟨1|2⟩, s_345]: 2.0, 10 → 3
    [[1|2], ⟨1|6⟩]: 1.0, 3  → 2
    [[1|2], [1|6]]: 1.0, 31 → 5
    [[1|2], ⟨2|3⟩]: 1.0, 3  → 2
    [[1|2], [2|3]]: 1.0, 30 → 5
    [[1|2], ⟨3|4⟩]: 2.0, 12 → 3
    [[1|2], [3|4]]: 1.0, 2  → 2
    [[1|2], ⟨4|5⟩]: 2.0, 3  → 2
    [[1|2], [4|5]]: 1.0, 10 → 2
    [[1|2], ⟨5|6⟩]: 2.0, 12 → 3
    [[1|2], [5|6]]: 1.0, 2  → 2
    [[1|2], s_123]: 2.0, 10 → 3
    [[1|2], s_234]: 1.0, 2  → 2
    [[1|2], s_345]: 2.0, 10 → 3
    [⟨1|6⟩, [1|6]]: 1.0, 2  → 2
    [⟨1|6⟩, ⟨2|3⟩]: 1.0, 2  → 2
    [⟨1|6⟩, [2|3]]: 2.0, 12 → 3
    [⟨1|6⟩, ⟨3|4⟩]: 1.0, 10 → 2
    [⟨1|6⟩, [3|4]]: 2.0, 4  → 2
    [⟨1|6⟩, ⟨4|5⟩]: 1.0, 2  → 2
    [⟨1|6⟩, [4|5]]: 2.0, 12 → 3
    [⟨1|6⟩, ⟨5|6⟩]: 1.0, 31 → 5
    [⟨1|6⟩, [5|6]]: 1.0, 3  → 2
    [⟨1|6⟩, s_123]: 1.0, 2  → 2
    [⟨1|6⟩, s_234]: 2.0, 10 → 3
    [⟨1|6⟩, s_345]: 2.0, 10 → 3
    [[1|6], ⟨2|3⟩]: 2.0, 12 → 3
    [[1|6], [2|3]]: 1.0, 2  → 2
    [[1|6], ⟨3|4⟩]: 2.0, 4  → 2
    [[1|6], [3|4]]: 1.0, 10 → 2
    [[1|6], ⟨4|5⟩]: 2.0, 12 → 3
    [[1|6], [4|5]]: 1.0, 2  → 2
    [[1|6], ⟨5|6⟩]: 1.0, 3  → 2
    [[1|6], [5|6]]: 1.0, 30 → 5
    [[1|6], s_123]: 1.0, 2  → 2
    [[1|6], s_234]: 2.0, 10 → 3
    [[1|6], s_345]: 2.0, 10 → 3
    [⟨2|3⟩, [2|3]]: 1.0, 2  → 2
    [⟨2|3⟩, ⟨3|4⟩]: 1.0, 30 → 5
    [⟨2|3⟩, [3|4]]: 1.0, 3  → 2
    [⟨2|3⟩, ⟨4|5⟩]: 1.0, 2  → 2
    [⟨2|3⟩, [4|5]]: 2.0, 12 → 3
    [⟨2|3⟩, ⟨5|6⟩]: 1.0, 10 → 2
    [⟨2|3⟩, [5|6]]: 2.0, 3  → 2
    [⟨2|3⟩, s_123]: 2.0, 10 → 3
    [⟨2|3⟩, s_234]: 2.0, 10 → 3
    [⟨2|3⟩, s_345]: 1.0, 2  → 2
    [[2|3], ⟨3|4⟩]: 1.0, 3  → 2
    [[2|3], [3|4]]: 1.0, 31 → 5
    [[2|3], ⟨4|5⟩]: 2.0, 12 → 3
    [[2|3], [4|5]]: 1.0, 2  → 2
    [[2|3], ⟨5|6⟩]: 2.0, 3  → 2
    [[2|3], [5|6]]: 1.0, 10 → 2
    [[2|3], s_123]: 2.0, 10 → 3
    [[2|3], s_234]: 2.0, 10 → 3
    [[2|3], s_345]: 1.0, 2  → 2
    [⟨3|4⟩, [3|4]]: 1.0, 2  → 2
    [⟨3|4⟩, ⟨4|5⟩]: 1.0, 31 → 5
    [⟨3|4⟩, [4|5]]: 1.0, 3  → 2
    [⟨3|4⟩, ⟨5|6⟩]: 1.0, 2  → 2
    [⟨3|4⟩, [5|6]]: 2.0, 12 → 3
    [⟨3|4⟩, s_123]: 1.0, 2  → 2
    [⟨3|4⟩, s_234]: 2.0, 10 → 3
    [⟨3|4⟩, s_345]: 2.0, 10 → 3
    [[3|4], ⟨4|5⟩]: 1.0, 3  → 2
    [[3|4], [4|5]]: 1.0, 30 → 5
    [[3|4], ⟨5|6⟩]: 2.0, 12 → 3
    [[3|4], [5|6]]: 1.0, 2  → 2
    [[3|4], s_123]: 1.0, 2  → 2
    [[3|4], s_234]: 2.0, 10 → 3
    [[3|4], s_345]: 2.0, 10 → 3
    [⟨4|5⟩, [4|5]]: 1.0, 2  → 2
    [⟨4|5⟩, ⟨5|6⟩]: 1.0, 30 → 5
    [⟨4|5⟩, [5|6]]: 1.0, 3  → 2
    [⟨4|5⟩, s_123]: 2.0, 10 → 3
    [⟨4|5⟩, s_234]: 1.0, 2  → 2
    [⟨4|5⟩, s_345]: 2.0, 10 → 3
    [[4|5], ⟨5|6⟩]: 1.0, 3  → 2
    [[4|5], [5|6]]: 1.0, 31 → 5
    [[4|5], s_123]: 2.0, 10 → 3
    [[4|5], s_234]: 1.0, 2  → 2
    [[4|5], s_345]: 2.0, 10 → 3
    [⟨5|6⟩, [5|6]]: 1.0, 2  → 2
    [⟨5|6⟩, s_123]: 2.0, 10 → 3
    [⟨5|6⟩, s_234]: 2.0, 10 → 3
    [⟨5|6⟩, s_345]: 1.0, 2  → 2
    [[5|6], s_123]: 2.0, 10 → 3
    [[5|6], s_234]: 2.0, 10 → 3
    [[5|6], s_345]: 1.0, 2  → 2
    [s_123, s_234]: 1.0, 2  → 2
    [s_123, s_345]: 1.0, 2  → 2
    [s_234, s_345]: 1.0, 2  → 2

Partial Fractioning

```python
invariants = oUnknown.poles_to_be_eliminated
print("Poles to be  eliminated:")
pprint(invariants)
print("")
lTerms = oUnknown.get_partial_fractioned_terms(invariants[0])
```

    [s_123, s_234, s_345, ⟨1|2⟩, [1|2], ⟨1|6⟩, [1|6], ⟨2|3⟩, [2|3], ⟨3|4⟩, [3|4], ⟨4|5⟩, [4|5], ⟨5|6⟩, [5|6]]
    Forced:
    []
    Forbidden:
    [⟨1|6⟩, [1|6], ⟨3|4⟩, [3|4], s_234, s_345]
    Optional:
    [(⟨5|6⟩, ⟨1|(2+3)|4]), ([4|5], ⟨6|(1+2)|3]), (⟨4|5⟩, ⟨3|(1+2)|6]), (⟨1|2⟩, ⟨6|(1+2)|3]), ([2|3], ⟨1|(2+3)|4]), (⟨2|3⟩, ⟨4|(2+3)|1]), ([5|6], ⟨4|(2+3)|1]), ([1|2], ⟨3|(1+2)|6])]

    There are 11 symmetries:
    [(165432, False), (216543, True), (234561, True), (321654, False), (345612, False), (432165, True), (456123, True), (543216, False), (561234, False), (612345, True), (654321, True)]

Numerator Fitting

```python
oTerms = lTerms[0]
oTerms.fit_numerators()
```

```python
print(oTerms)
```

```python
oParticles = Particles(6)
oParticles.fix_mom_cons()
assert(abs(oTerms(oParticles) - oTree(oParticles)) < 10 ** -280)
```

content/post/A Simple Tree Level Example/.ipynb_checkpoints/Tree Example-checkpoint.ipynb


```python
%load_ext autoreload
%autoreload 2
```

```python
import os
import logging

from kyle.sampling.fake_clf import DirichletFC, MultiDirichletFC
from kyle.evaluation import EvalStats, compute_accuracy, compute_ECE, compute_expected_max
from kyle.transformations import *
from kyle.calibration.calibration_methods import TemperatureScaling

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.init as init
import torchvision
import torchvision.datasets as datasets
import torchvision.transforms as transforms
import pytorch_lightning as pl

import scipy.stats
import scipy.optimize
import numpy as np
import matplotlib.pyplot as plt
```

# Realistic Fake Classifiers

It's good to have a model of what a realistic Fake Classifier should look like.

Probably the simplest model for the fake classifier confidence vector distribution is the **Dirichlet Fake Classifier**:

\begin{equation}
C \sim Dirichlet(\alpha_1, \alpha_2, \alpha_3, ...)
\end{equation}

This classifier has a total of 'num_classes' parameters

However, this model is possibly a bit too simple as it only has a single local maximum in the distribution. A realistic fake classifier might for example have multiple local maxima in each of the corners of the simplex, i.e. it generally is very confident in its prediction and only very rarely uncertain (center of simplex). Something similar can actually be achieved using the Dirichlet distribution by setting all the parameters $\alpha_n < 1$. This pushes the distribution out into the corners, BUT however also out onto the sides of the simplex, which is not quite what we want. The center of a side of the simplex corresponds to a confidence vector $\vec\alpha = (1/\text{num_classes}-1, 1/\text{num_classes}-1, ..., 1/\text{num_classes}-1, 0)$, i.e. very uncertain in all but one of the classes.

Therefore we also consider two other Fake Classifiers that can have multiple local maxima, one in each of the corners, and therefore possibly represent real neural networks better:

Firstly the **Multi-Dirichlet Fake Classifier**:

\begin{align}
K & \sim Catgeorical(p_1, p_2, p_3, ...) \\
C & \sim Dirichlet_k(\sigma_k\cdot[1, 1, ..., 1, \alpha_k, 1, ...])
\end{align}

This classifier has a total of '3 x num_classes' parameters

i.e. we first draw from a K-categorical distribution and based on the result we then draw from one of K Dirichlet distributions. Each of the K Dirichlet distributions has two parameters $\sigma$ and $\alpha_k$ which represent the width and position of the local maximum in the k-th corner of the simplex.

Note: The pdf of this mixture distribution will be a weighted sum of the individual dirichlet distributions

Secondly the **Multi-Gaussian Fake Classifier**:

K-Categorical followed by one of K Gaussians followed by softmax

(probably doesn't make too much sense as Multi-Gaussian pdf is analytically intractable due to the softmax transformation)

**Note**:

When talking about the simple dirichlet FC $\vec{\alpha}$ refers to the vector of $\alpha$ parameters of the dirichlet distribution.

When talking about the Multi-Dirichlet FC $\vec{\alpha_k}$ refers to the vector of $\alpha_k$ parameters, and should not be confused to be a parameter vector of a single dirichlet.

In order to get an actually realistic Fake Classifier we use these three Fake Classifier models and fit their distributions to the observed confidence vector distributions for a couple of different neural networks.

In this case we use:

**LeNet 5** on CIFAR 10

**ResNet 20** on CIFAR 10

**ResNet 110** on CIFAR 10

## Preparing Data and Neural Networks:

```python
#load cifar dataset
#normalizarion also from https://github.com/akamaster/pytorch_resnet_cifar10
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

cifar_train_set = datasets.CIFAR10(os.getcwd(), train=True, download=True,
        transform=transforms.Compose([transforms.ToTensor(), normalize]))
cifar_test_set = datasets.CIFAR10(os.getcwd(), train=False, download=True,
                transform=transforms.Compose([transforms.ToTensor(), normalize]))

cifar_train = torch.utils.data.DataLoader(cifar_train_set, batch_size=4, shuffle=True, num_workers=1)
cifar_test = torch.utils.data.DataLoader(cifar_test_set, batch_size=4, shuffle=False, num_workers=1)
```

```python
#Small simple LeNet5 for CIFAR 10 classification

class lenet5(pl.LightningModule):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), 2)
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def training_step(self, batch, batch_idx):
        x, target = batch
        output = self(x)
        loss = F.cross_entropy(output, target)
        return loss

    def validation_step(self, batch, batch_idx):
        x, target = batch
        output = self(x)
        loss = F.cross_entropy(output, target)
        self.log('val_loss', loss, on_step=False, on_epoch=True, prog_bar=True)

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.parameters(), lr=0.001, momentum=0.9)
        return optimizer

```

```python
#Proper implementation of ResNet20 for Cifar10. Pytorch only has ResNets for ImageNet which
#differ in number of parameters
#Code taken from: https://github.com/akamaster/pytorch_resnet_cifar10

def _weights_init(m):
    classname = m.__class__.__name__
    #print(classname)
    if isinstance(m, nn.Linear) or isinstance(m, nn.Conv2d):
        init.kaiming_normal_(m.weight)

class LambdaLayer(nn.Module):
    def __init__(self, lambd):
        super(LambdaLayer, self).__init__()
        self.lambd = lambd

    def forward(self, x):
        return self.lambd(x)

class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1, option='A'):
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != planes:
            if option == 'A':
                """
                For CIFAR10 ResNet paper uses option A.
                """
                self.shortcut = LambdaLayer(lambda x:
                                            F.pad(x[:, :, ::2, ::2], (0, 0, 0, 0, planes//4, planes//4), "constant", 0))
            elif option == 'B':
                self.shortcut = nn.Sequential(
                     nn.Conv2d(in_planes, self.expansion * planes, kernel_size=1, stride=stride, bias=False),
                     nn.BatchNorm2d(self.expansion * planes)
                )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        out = F.relu(out)
        return out

class ResNet(nn.Module):
    def __init__(self, block, num_blocks, num_classes=10):
        super(ResNet, self).__init__()
        self.in_planes = 16

        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(16)
        self.layer1 = self._make_layer(block, 16, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 32, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 64, num_blocks[2], stride=2)
        self.linear = nn.Linear(64, num_classes)

        self.apply(_weights_init)

    def _make_layer(self, block, planes, num_blocks, stride):
        strides = [stride] + [1]*(num_blocks-1)
        layers = []
        for stride in strides:
            layers.append(block(self.in_planes, planes, stride))
            self.in_planes = planes * block.expansion

        return nn.Sequential(*layers)

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = F.avg_pool2d(out, out.size()[3])
        out = out.view(out.size(0), -1)
        out = self.linear(out)
        return out

def resnet20():
    return ResNet(BasicBlock, [3, 3, 3])

def resnet32():
    return ResNet(BasicBlock, [5, 5, 5])

def resnet44():
    return ResNet(BasicBlock, [7, 7, 7])

def resnet56():
    return ResNet(BasicBlock, [9, 9, 9])

def resnet110():
    return ResNet(BasicBlock, [18, 18, 18])

def resnet1202():
    return ResNet(BasicBlock, [200, 200, 200])
```

```python
#Train a LeNet5
#Load selftrained LeNet 5 and pretrained Resnet20 and Resnet110 (don't have a dedicated GPU at hand D:)
#Pretrained nets taken from https://github.com/akamaster/pytorch_resnet_cifar10

#selftrained_lenet5 = lenet5()
#checkpoint_callback = pl.callbacks.ModelCheckpoint(monitor='val_loss', save_top_k=1, save_last=True)
#trainer = pl.Trainer(max_epochs=20, logger=False, checkpoint_callback=checkpoint_callback)
#trainer.fit(selftrained_lenet5, cifar_train, cifar_test)

selftrained_lenet5 = lenet5.load_from_checkpoint('./trained_models/lenet5.ckpt')

pretrained_resnet20 = resnet20()
pretrained_resnet110 = resnet110()

pretrained_resnet20_dict = torch.load('./trained_models/resnet20-12fca82f.th',
                               map_location=torch.device('cpu'))['state_dict']
pretrained_resnet20_dict = {key.replace("module.", ""): value for key, value in pretrained_resnet20_dict.items()}
pretrained_resnet20.load_state_dict(pretrained_resnet20_dict)

pretrained_resnet110_dict = torch.load('./trained_models/resnet110-1d1ed7c2.th',
                               map_location=torch.device('cpu'))['state_dict']
pretrained_resnet110_dict = {key.replace("module.", ""): value for key, value in pretrained_resnet110_dict.items()}
pretrained_resnet110.load_state_dict(pretrained_resnet110_dict)
```

## Set which neural net to use here

```python
neural_net = selftrained_lenet5
#neural_net = pretrained_resnet20
#neural_net = pretrained_resnet110
```

```python
#Get NN predictions on CIFAR10 test set
cifar_test_full = torch.utils.data.DataLoader(cifar_test_set, batch_size=len(cifar_test_set),
                                              shuffle=False, num_workers=2)
images, labels = next(iter(cifar_test_full))

neural_net.eval()
with torch.no_grad():
    logits = neural_net(images)
    prob = F.softmax(logits, dim=1)
    _, predicted = torch.max(prob, dim=1)
    print(f'NLL = {F.cross_entropy(logits, labels)}')
    print(f'accuracy = {(predicted == labels).sum().item() / labels.size(0)}')

gt_labels = labels.numpy()
confidences = prob.numpy()
```

```python
gt_labels_copy = gt_labels.copy()
confidences_copy = confidences.copy()
```

```python
gt_labels = gt_labels_copy.copy()
confidences = confidences_copy.copy()
```

```python
print(f'num non normalized confidence vectors = {np.sum((np.sum(confidences, axis=1) - 1) >= 1e-10)}')

#confidences are not perfectly normalized due to floating point error
#scipy.stats.dirichlet.pdf is very picky about normalization
#convert confidences to float64 first for better/more accurate normalization

confidences = np.array(confidences, dtype='float64')
confidences = confidences / np.sum(confidences, axis=1)[:,None]
print(f'num non normalized confidence vectors = {np.sum((np.sum(confidences, axis=1) - 1) >= 1e-10)}')

```

# Fitting Fake Classifiers using MLE

**All of this is relatively simple/rudimemtary. 'Proper' Dirichlet fitting would involve something closer to what is described in https://epub.wu.ac.at/4077/1/Report125.pdf.**

(Note: Any references to fitted results/distributions/graphs in the following sections used the LeNet5 as the 'real' neural network)

Having gotten the confidences of our neural net on the CIFAR 10 test set we can now try and fit an appropiate fake classifier to them. This can be done quite easily for the Dirichlet and Multi-Dirichlet FC's using MLE, as we have relatively simple expressions for the distribution pdf's. The Multi-Gaussian FC is not as easy, as the softmax function complicates the fake classifier's pdf. (It would be necessary to invert the softmax function, which is only possible up to an additive constant. As a result the Multi-Gaussian fake classifier's pdf will be an integral over a gaussian mixture model's pdf.)

## 'Normal' MLE Fitting

MLE is probably the easiest and simplest approach when the fake classifier's pdf is known exactly. We calculate the negative log likelihood of our neural net's confidence vector distribution under the assumption of either a Dirichlet or Multi-Dirichlet distribution. Using one of scipy's many minimization algorithms/functions we can then find the set of parameters of the fake classifier that maximize the log-likelihood/minimize the negative log-likelihood.

We have introduced this fitting as ``.fit()`` method for the FakeClassifier Classes.

Right away we run into a problem, divergences:

On the sides and corners of the simplex, when one or many components of the confidence vector are zero, the log-likelihood diverges. If $c_i = 0$ and the corrseponding dirichlet parameter $a_i < 1$ the pdf and the log-likelihood diverge $p(c_i = 0) \rightarrow + \infty$. If, however, $a_i > 1$ then $p(c_i)=0$ which will again lead to a divergence in the log-likelihood.

Unfortunately a lot of the neural networks are very confident and often predict class confidences near/at $c_i=0$. To mitigate this divergence problem somewhat we can rescale the confidences before fitting according to:

\begin{equation}
\vec{c} = \frac{\vec{c}(N-1) + 1/\text{num_classes}}{N}
\end{equation}

where N is the number of samples (see https://epub.wu.ac.at/4077/1/Report125.pd). All of the following fits use this rescaling.

As always with fitting the initial guesses and bounds are important:

As discussed at the start a somewhat alright fake classifier can possibly be achieved by using a simple Dirichlet Fake Classifier with alpha parameters $\alpha_n < 1$. For fitting the DirichletFC appropiate initial guesses and bounds might therefore be $\vec{\alpha}_\text{init} = (1,1,1,1,...)$  and $\alpha_\min, \alpha_\max = (0.0001, \text{None})$

As dicussed at the start the reasoning behind the Multi-Dirichlet FC is that each separate Dirichlet can be used to create a local maximum in one of the corners of the simplex. This only works if the full alpha vector of each dirichlet has all entries $>1$ (if any entry is $<1$ a local maximum does not exist), which means for each Dirichlet we need $\alpha_k >1$ and $\sigma_k>1$. We also expect the maxima to be very 'squished' into the corners. i.e. $\alpha_k$ to be large. For fitting the Multi-Dirichlet FC appropate initial guesses and bounds might therefore be $\vec{\alpha}_{k,\text{init}} = (10,10,10,10,...)$ $\vec{\sigma}_\text{init} = (2,2,2,2,...)$ and $\alpha_{k,\min}, \alpha_{k,\max} = (1, \text{None})$ $\sigma_\min, \sigma_\max = (1, \text{None})$

```python
#Save 'EvalStats' and NLL of all the fitted FC's for easier comparison later
save_fc_eval = []
save_NLL     = []
```

```python
#Fit Dirichlet FC to the test set confidence vector distributions using MLE fitting
num_classes = confidences.shape[1]

Dir_FC = DirichletFC(num_classes)
mle_results = Dir_FC.fit(confidences, initial_alpha=np.ones(num_classes), alpha_bounds=(0.0001, None))

save_NLL.append(mle_results.fun)
print(f'final NLL = {mle_results.fun}')
print(f'Fitted parameters = {Dir_FC.alpha}')
```

```python
#Fit Multi-Dirichlet FC to the test set confidence vector distributions using MLE fitting
num_classes = confidences.shape[1]

MultiDir_FC = MultiDirichletFC(num_classes)
mle_results = MultiDir_FC.fit(confidences,
                            initial_parameters=np.array([10, 2, 1/num_classes]),
                            parameter_bounds=[(1,None), (1,None), (0,1)],
                            simplified_fitting=False)

save_NLL.append(mle_results.fun)
print(f'final NLL = {mle_results.fun}')
print(f'Fitted parameters = {MultiDir_FC.get_parameters()}')
```

```python
#visual comparison of neural network and FC distributions
Dir_FC_eval = EvalStats(*Dir_FC.get_sample_arrays(50000))
MultiDir_FC_eval = EvalStats(*MultiDir_FC.get_sample_arrays(50000))
NNet_eval = EvalStats(gt_labels, confidences)

fig, axs = plt.subplots(3,3, figsize=(25,25))
plt.suptitle('Dirichlet (left) vs Multi-Dirichlet (middle) vs Real NN (right)', fontsize=25)
for i, class_ in enumerate(["top_class", 1, 2]):
    plt.sca(axs[i,0])
    Dir_FC_eval.plot_confidence_distributions([class_], new_fig=False)
    plt.sca(axs[i,1])
    MultiDir_FC_eval.plot_confidence_distributions([class_], new_fig=False)
    plt.sca(axs[i,2])
    NNet_eval.plot_confidence_distributions([class_], new_fig=False)
```

```python
save_fc_eval.append(Dir_FC_eval)
save_fc_eval.append(MultiDir_FC_eval)
```

Above we have plotted the top class confidence distribution as well as the marginal confidence distributions for the actual neural network and our fitted fake classifiers (the full 10D confidence vector distribution is unfortuanetly a bit difficult to visualise). This allows us to visually inspect how well our fake classifiers really reflect the true confidence distribution of our classifier. The left, middle and right columns correspond to the confidence distributions of the dirichlet fake clsssifier, the multi-dirichlet faske classifier and the real neural network respectively.

As expected the simple dirichlet FC does not work well. It doesn't capture the multimodal nature of the true marginal distribution nor the high frequency of p=1.0 confidences in the top class confidence distributions.

The Multi-dirichlet FC seems to be a bit better capturing both of these qualities. However the maxima of the distributions don't lie far enough at the extremes, which is likely due to the fact that the local maximum of a dirichlet only lies exactly on the corner in the limit of $\alpha_k \rightarrow \infty$

Also of note is that the Multi-Dirichlet FC always tries to fit $\sigma < 1$, always reaching the lower bound of $\sigma = 1$ which we have imposed.

## 'Sufficiently Confident/Class Split' MLE Fitting

Given the slight difficulties with fitting a 30 parameter Multi-Dirichlet model we try to simplify the fitting a bit. This can be done by assuming a scenario very similar to that of sufficiently confident fake classifiers. We assume that if we draw a confidence vector from the k'th dirichlet distribution, then that confidence vector always predicts the class k. That means:

\begin{equation}
P(\text{argmax}(\vec{c}) = k | \vec{c} \sim \text{Dir}_k) \simeq 1 \text{  or equivalently  } P(\text{argmax}(\vec{c}) = k) \simeq p_k
\end{equation}

where $p_k$ is the weight of the k'th dirichlet distribution. This is of course only a good approximation if $P(\text{argmax}(\vec{c}) \neq k | \vec{c} \sim \text{Dir}_k) \simeq 0$, i.e. if the individual Dirichlet distributions are sufficiently concentrated in their respective corner of the simplex with only negligible probability mass near the other corners of the simplex.

From the marginal class confidence distribution of the real neural network, we can see that this might be a very good approximation, as the class confidences always either lie near $c_k = 1.0$ or $c_k = 0.0$.

This approximation makes two simplifications possible:

1. We no longer have to fit the distribution weights $p_k$. Instead they can simply be read off from the real neural networks predicted class probabilities. This reduces the number of fitted parameters from 30 to 20

2. We can now fit each Dirichlet distribution from the Multi-Dirichlet Fake classifier individually to the subset of the data for which $\text{argmax}(\vec{c}) = k$. This reduces the problem further from a simultaneous fit of 20 parameters to 10 fits of 2 parameters each.

We proceed in the following way: First we split the real neural networks confidence distribution by predicted class into 10 subsets. Using the relative size of these subest we estimate the distribution weights of the dirichlet distributions in our Multi-Dirichlet FC. We then fit each dirichlet distribution of our Multi-Dirichlet FC separately to its corresponding data set, by minimizing the negative log likelihood as before.

This simplified fitting procedure can be called using ``.fit(..., simplified_fitting=True)``.

```python
#Fit Multi-Dirichlet FC to the test set confidence vector distributions using above simplification
num_classes = confidences.shape[1]

MultiDir_FC = MultiDirichletFC(num_classes)
mle_results = MultiDir_FC.fit(confidences,
                            initial_parameters=np.array([10, 2]),
                            parameter_bounds=[(1,None), (1,None)],
                            simplified_fitting=True)

save_NLL.append(np.sum([k_result.fun for k_result in mle_results]))
print(f'final NLL = {np.sum([k_result.fun for k_result in mle_results])}')
print('alpha = {}\n\nsigma = {}\n\ndistribution_weights = {}'.format(*MultiDir_FC.get_parameters()))
```

Similar to above we can now visiually compare the marginal and top class confidence distributions of our fake classifier and the real neural network:

```python
MultiDir_FC_eval = EvalStats(*MultiDir_FC.get_sample_arrays(50000))
NNet_eval = EvalStats(gt_labels, confidences)

fig, axs = plt.subplots(3,2, figsize=(10,15))
plt.suptitle('Simplified Multi-Dirichlet (left) vs Real NN (right)', fontsize=15)
for i, class_ in enumerate(["top_class", 1, 2]):
    plt.sca(axs[i,0])
    MultiDir_FC_eval.plot_confidence_distributions([class_], new_fig=False)
    plt.sca(axs[i,1])
    NNet_eval.plot_confidence_distributions([class_], new_fig=False)
```

```python
save_fc_eval.append(MultiDir_FC_eval)
```

This doesn't really seem to work much better than fitting all 30 parameters at the same time. It actually looks qualitativley worse. This is likely due to the fact that the approximation made to 'derive' this fitting procedure is simply not very good in the case of LeNet5. However, with ResNet20 and ResNet110 this approximation becomes much better and the 'class split' fitting produces almost exactly the same results.

Most importatntly though, it is much less computationally expensive.

## Sigma < 1

Interestingly enough the simplified fitting runs into the same problem of always trying to fit very small $\sigma_k < 1$, even though the sensible/intuitve choice would be $\sigma_k > 1$. This indicates that maybe our sensible/intuitve choice of wanting $\sigma_k > 1$ isn't actually that sensible. As explained earlier the reasoning behind the Multi-Dirichlet model is that each Dirichlet distribution is used to create a local maximum in one of the corners of the simplex, with $\alpha_k$ determining how squished into the corner and $\sigma_k$ how broad the local maximum is. The requirement of $\sigma_k > 1$ and $\alpha_k > 1$ arose, because if any parameter of the Dirichlet is $< 1$ a local maximum does not exist and our whole reasoning behind introducing the Multi-Dirichlet fails.

However, as it turns out a reasonable parameter choice might also be $\sigma_k < 1$, with $\alpha_k \cdot \sigma_k > 1$. This would mean that all parameters of the dirichlet, with exception of the k'th entry, are $ < 1$. While a local maximum no longer exists (we now have divergences at the sides and corners) this can still lead to an equivalent situation with most of the probability mass concentrated in one corner of the simplex. Below is a quick and dirty visualisation of this:

```python
fig, ax = plt.subplots(1, 2, figsize=(12,5))

vis_dir_FC = DirichletFC(3)

vis_dir_FC.set_alpha([0.2, 20, 0.2])
_, vis_confidences = vis_dir_FC.get_sample_arrays(10000)

ax[0].scatter(vis_confidences[:,0], vis_confidences[:,1], s=0.1)
ax[0].plot([0,1], [1,0], 'r--')
ax[0].plot([0,1], [0,0], 'r--')
ax[0].plot([0,0], [0,1], 'r--')
ax[0].set_xlim([-0.2,1.2])
ax[0].set_ylim([-0.2,1.2])

vis_dir_FC.set_alpha([2, 50, 2])
_, vis_confidences = vis_dir_FC.get_sample_arrays(10000)

ax[1].scatter(vis_confidences[:,0], vis_confidences[:,1], s=0.1)
ax[1].plot([0,1], [1,0], 'r--')
ax[1].plot([0,1], [0,0], 'r--')
ax[1].plot([0,0], [0,1], 'r--')
ax[1].set_xlim([-0.2,1.2])
ax[1].set_ylim([-0.2,1.2])
```

Below we have repeated both the direct Multi-Dirichlet FC fitting as well as the 'class split' fitting, but this time with a lower bound for sigma of $\sigma_\text{lower} = 0.001$. Allowing for small $\sigma$ seems to improve things quite a bit.

```python
#Fit Multi-Dirichlet FC allowing sigma < 1
num_classes = confidences.shape[1]

MultiDir_FC = MultiDirichletFC(num_classes)
mle_results = MultiDir_FC.fit(confidences,
                            initial_parameters=np.array([10, 2, 1/num_classes]),
                            parameter_bounds=[(1,None), (0.001,None), (0,1)],
                            simplified_fitting=False)

save_NLL.append(mle_results.fun)
print(f'final NLL = {mle_results.fun}')
print('alpha = {}\n\nsigma = {}\n\ndistribution_weights = {}'.format(*MultiDir_FC.get_parameters()))
```

```python
MultiDir_FC_eval = EvalStats(*MultiDir_FC.get_sample_arrays(50000))
NNet_eval = EvalStats(gt_labels, confidences)

fig, axs = plt.subplots(3,2, figsize=(10,15))
plt.suptitle('Multi-Dirichlet w/ sigma < 1 (left) vs Real NN (right)', fontsize=15)
for i, class_ in enumerate(["top_class", 1, 2]):
    plt.sca(axs[i,0])
    MultiDir_FC_eval.plot_confidence_distributions([class_], new_fig=False)
    plt.sca(axs[i,1])
    NNet_eval.plot_confidence_distributions([class_], new_fig=False)
```

```python
save_fc_eval.append(MultiDir_FC_eval)
```

```python
#Fit Multi-Dirichlet FC allowing sigma < 1 with simplified fitting
num_classes = confidences.shape[1]

MultiDir_FC = MultiDirichletFC(num_classes)
mle_results = MultiDir_FC.fit(confidences,
                            initial_parameters=np.array([10, 2]),
                            parameter_bounds=[(1,None), (0.001,None)],
                            simplified_fitting=True)

save_NLL.append(np.sum([k_result.fun for k_result in mle_results]))
print(f'final NLL = {np.sum([k_result.fun for k_result in mle_results])}')
print('alpha = {}\n\nsigma = {}\n\ndistribution_weights = {}'.format(*MultiDir_FC.get_parameters()))
```

```python
MultiDir_FC_eval = EvalStats(*MultiDir_FC.get_sample_arrays(50000))
NNet_eval = EvalStats(gt_labels, confidences)

fig, axs = plt.subplots(3,2, figsize=(10,15))
plt.suptitle('Simplified Multi-Dirichlet w/ sigma < 1 (left) vs Real NN (right)', fontsize=15)
for i, class_ in enumerate(["top_class", 1, 2]):
    plt.sca(axs[i,0])
    MultiDir_FC_eval.plot_confidence_distributions([class_], new_fig=False)
    plt.sca(axs[i,1])
    NNet_eval.plot_confidence_distributions([class_], new_fig=False)
```

```python
save_fc_eval.append(MultiDir_FC_eval)
```

## Conclusion:

The confidence distributions have been aggregated here again for easier comparison.

### Real NN's Confidence Distribution:

```python
fig, axs = plt.subplots(1,2, figsize=(12,5))
plt.axes(axs[0])
NNet_eval.plot_confidence_distributions(["top_class"], new_fig = False)
plt.axes(axs[1])
NNet_eval.plot_confidence_distributions([0], new_fig = False)

```

### Dirichlet FC Confidence Distribution:

```python
print(f'final NLL = {save_NLL[0]}')
fig, axs = plt.subplots(1,2, figsize=(12,5))
plt.axes(axs[0])
save_fc_eval[0].plot_confidence_distributions(["top_class"], new_fig = False)
plt.axes(axs[1])
save_fc_eval[0].plot_confidence_distributions([0], new_fig = False)
```

### Directly Fitted Multi Dirichlet FC Confidence Distribution:

```python
print(f'final NLL = {save_NLL[1]}')
fig, axs = plt.subplots(1,2, figsize=(12,5))
plt.axes(axs[0])
save_fc_eval[1].plot_confidence_distributions(["top_class"], new_fig = False)
plt.axes(axs[1])
save_fc_eval[1].plot_confidence_distributions([0], new_fig = False)
```

### Class Split Fitted Multi Dirichlet FC Confidence Distribution:

```python
print(f'final NLL = {save_NLL[2]}')
fig, axs = plt.subplots(1,2, figsize=(12,5))
plt.axes(axs[0])
save_fc_eval[2].plot_confidence_distributions(["top_class"], new_fig = False)
plt.axes(axs[1])
save_fc_eval[2].plot_confidence_distributions([0], new_fig = False)
```

### Directly Fitted Multi Dirichlet FC Confidence Distribution, sigma < 1:

```python
print(f'final NLL = {save_NLL[3]}')
fig, axs = plt.subplots(1,2, figsize=(12,5))
plt.axes(axs[0])
save_fc_eval[3].plot_confidence_distributions(["top_class"], new_fig = False)
plt.axes(axs[1])
save_fc_eval[3].plot_confidence_distributions([0], new_fig = False)
```

### Class Split Fitted Multi Dirichlet FC Confidence Distribution, sigma < 1:

```python
print(f'final NLL = {save_NLL[4]}')
fig, axs = plt.subplots(1,2, figsize=(12,5))
plt.axes(axs[0])
save_fc_eval[4].plot_confidence_distributions(["top_class"], new_fig = False)
plt.axes(axs[1])
save_fc_eval[4].plot_confidence_distributions([0], new_fig = False)
```

Comparing how well different models fit is always a bit of a tricky thing to do. The Likelihood ratio test only works for comparing nested models, i.e. models where one is a special/restricted case of the other model, which is unfortunately not the case here.

However, non-nested models can be compared (informally) using the Akaike Information Criterion (AIC), defined as:

\begin{equation}
AIC = 2k - 2\ln(\hat{L})
\end{equation}

where $k$ is the number of parameters of the model and $\hat{L}$ the maximum lilelihood.

Given that our models have parameters of either on the order of $\sim 10^1$ whereas $\ln(\hat{L}) \sim 10^6$ we can pretty much just compare likelihoods directly.

### For the LeNet 5 we conclude:

1. Suprisingly, the simple Dirichlet fits the real NN better than the Multi-Dirichlet with $\sigma > 1$, even though visually it looks like the Multi-Dirichlet captures/reproduces the marginal distributions of the real NN a lot better

2. Also suprisingly, fitting the Multi-Dirichlet directly gives worse results according to the AIC comared to fitting it via the approximation of 'class splitting' (even though, again, the directly fitted Multi-Dirichlet marginal distributions looks more 'real')

3. Allowing for $\sigma < 1$ improves the AIC a lot for both directly and class split fitted Multi-Dirichlet FC's

The 'most real' results seem to be achieved using the directly fitted Multi-Dirichlet FC and allowing sigma < 1. While it does actually result in a slightly worse AIC than the same class-split FC the marginal distributions look a lot more 'real'.

### For the ResNet 20 we conclude:

Similar results as for the LeNet 5 with one major exception:

The difference between the class split and directly fitted multi dirichlet FC is negligibly small. This not a suprise as we expect the more complex NN's to be more confident in general, i.e. for the confidences to be more concentrated in the corners. This makes the more complex NN's closer to a 'sufficiently confident like' behaviour that we based the class split fitting approximation on.

Again the most promising results seem to be achieved using the directly fitted Multi-Dirichlet FC and allowing sigma < 1.

### For the ResNet 110 we conclude:

Pretty much almost the same as for the ResNet 20

### In General:

The Multi-Dirichlet FC seems to be a lot better at reproducing the ResNet 20 and ResNet 110 confidence distributions. The LeNet 5 is more difficult for the Multi-Dirichlet FC to reproduce. This is likely due to the LeNet 5 not being 'overconfident enough', i.e. when looking at  the LeNet's 'top class' confidence distribution a fair amount of confidences lie in the middle of the distribution at $p \sim 0.5$. For ResNet 20 and ResNet 110 this is not the case, pretty much all confidences lie at the extreme value of $p \sim 1.0$.

Maybe TODO: Try other fitting methods: e.g. moment matching or fitting only to the marginal distributions

Maybe TODO: Try 'stochastic fitting' for the multi-gaussian FC where we sample the FC calculate the corrseponding marginal distributions and minimize e.g. the squared error loss between the neural network marginal distributions and the marginal distributions sampled from the FC. Could get around problem of multi-gaussian FC having analytically intractable pdf

TODO: Find a good simplex automorphism to also recreate the reliability diagrams of the neural networks.

```python

```

```python

```

```python

```

## This didn't really work but it's here for completeness

## Further Modifications to Fitting Procedure

There are definitely more ways to improve the fitting procedure (see https://epub.wu.ac.at/4077/1/Report125.pdf).

A relatively simple one is to transform the parameters before fitting. In general, introducing bounds into minimzation problems can complicate things. However instead of fitting the dirichlet parameters directly we can first rescale/transform them onto the interval $[-\infty, \infty]$ (Very similar to the idea of generalized linear models, just that we do not have predictor variables in our case).

For the simple dirichlet FC and appropiate rescaling would be $log(\cdot)$ mapping the allowed alpha interval $[0, \infty]$ onto $[-\infty, \infty]$

For the multi-dirichlet FC, with $\sigma > 0$ and $\alpha > 1$ an appropiate transform would be e.g. $log(\sigma)$ and $log(\alpha - 1)$

Below we fitted the Multi-Dirichlet FC again, using both the direct and 'clas split' fitting procedure, however this time with the above transformstions applied to the parameters.

**Note**:

As it turns out rescaling the parameters might not be that great of an idea. The problem is that the minimization algorithms end up taking stepsizes of up to $10$ in $\log(\alpha)$ which results in $\alpha \sim 10^{10}$. Even after rescaling the confidences to avoid 0 values, we still run into problems with such a large alpha. For a value of $c \sim 10^{-5}$, we get $c^\alpha \sim 10^{-50}$ which might as well be 0 and will lead to errors in the log-likelihood.

```python
#rescale the confidences to avoid 0 confidences and renormalize

num_classes = confidences.shape[1]

rescaled_confidences = (confidences * (confidences.shape[0] - 1) + 1/num_classes) / confidences.shape[0]
rescaled_confidences = rescaled_confidences / np.sum(rescaled_confidences, axis=1)[:,None]
```

### First the simultaneous fit of 30 parameters:

```python
MultiDir_FC = MultiDirichletFC(num_classes)

def MultiDir_NLL(log_alpha, log_sigma, logit_distribution_weights):
    alpha = np.exp(log_alpha)+1
    sigma = np.exp(log_sigma)
    distribution_weights = np.exp(logit_distribution_weights) / (1 + np.exp(logit_distribution_weights))
    return -np.sum(np.log( MultiDir_FC.pdf(rescaled_confidences, alpha, sigma, distribution_weights) ))

#initial guesses for fitting multi-dirichlet (no longer need bounds due to log transformations)
init_log_alpha = np.zeros(num_classes)
init_log_sigma = np.zeros(num_classes)
init_logit_distribution_weights = np.zeros(num_classes)

MultiDirichlet_bestfit = scipy.optimize.minimize(lambda parms: MultiDir_NLL(*np.split(parms,3)),
                                                 np.concatenate((init_log_alpha, init_log_sigma, init_logit_distribution_weights)),
                                                 options={'disp': True})
print(f'final NLL = {MultiDirichlet_bestfit.fun}')
```

```python
#Set FC parameters to those found from MLE fit
l_a, l_s, l_w = np.split(MultiDirichlet_bestfit.x,3)
MultiDir_FC.set_parameters(np.exp(l_a)+1, np.exp(l_s), np.exp(l_w) / (1 + np.exp(l_w)))
```

```python
print('alpha_k = {}\n\nsigma_k = {}\n\ndistribution_weights = {}'.format(*np.split(MultiDirichlet_bestfit.x,3)))
```

```python
MultiDir_FC_eval = EvalStats(*MultiDir_FC.get_sample_arrays(50000))
NNet_eval = EvalStats(gt_labels, confidences)

MultiDir_FC_eval.plot_confidence_distributions([0,"top_class"])
NNet_eval.plot_confidence_distributions([0,"top_class"])

for i in range(3):
    MultiDir_FC_eval.plot_confidence_distributions([i])
    NNet_eval.plot_confidence_distributions([i])
```

### And secondly the 10 fits of 2 parameters each:

```python
class_split_confidences = [rescaled_confidences[np.argmax(rescaled_confidences, axis=1)==i, :] for i in range(num_classes)]

estimated_distribution_weights = [k_class_confidences.shape[0] for k_class_confidences in class_split_confidences]
estimated_distribution_weights = estimated_distribution_weights / np.sum(estimated_distribution_weights)
```

```python
Dir_FC = DirichletFC(num_classes)
MultiDir_FC = MultiDirichletFC(num_classes)

alpha_k = []
sigma_k = []
NLL_k = []

for k, k_class_confidences in enumerate(class_split_confidences):

    def k_dir_NLL(log_alpha, log_sigma):
        alpha_vector = np.ones(num_classes)
        alpha_vector[k] = np.exp(log_alpha)+1
        alpha_vector *= np.exp(log_sigma)
        return -np.sum(np.log( Dir_FC.pdf(k_class_confidences, alpha_vector) ))

    #initial guesses fitting individual dirichlets (no longer need bounds due to log transformations)
    init_log_alpha = 0
    init_log_sigma = 0

    k_dir_bestfit = scipy.optimize.minimize(lambda parms: k_dir_NLL(*parms), np.array([init_log_alpha, init_log_sigma]),
                                            options={'disp': True})

    alpha_k.append(np.exp(k_dir_bestfit.x[0])+1)
    sigma_k.append(np.exp(k_dir_bestfit.x[1]))
    NLL_k.append(k_dir_bestfit.fun)

print(f'final NLL = {np.sum(NLL_k)}')
```

```python
MultiDir_FC.set_parameters(alpha_k, sigma_k, estimated_distribution_weights)
print('alpha_k = {}\n\nsigma_k = {}\n\ndistribution_weights = {}'.format(*MultiDir_FC.get_parameters()))
```

```python
MultiDir_FC_eval = EvalStats(*MultiDir_FC.get_sample_arrays(50000))
NNet_eval = EvalStats(gt_labels, confidences)

MultiDir_FC_eval.plot_confidence_distributions([0,"top_class"])
NNet_eval.plot_confidence_distributions([0,"top_class"])

for i in range(3):
    MultiDir_FC_eval.plot_confidence_distributions([i])
    NNet_eval.plot_confidence_distributions([i])

```