# `ConditionalDimension` tutorial

This tutorial explains how to create equations that only get executed under given conditions.

```python
from devito import Dimension, Function, Grid
import numpy as np

# We define a 10x10 grid, dimensions are x, y
shape = (10, 10)
grid = Grid(shape = shape)
x, y = grid.dimensions

# Define function 𝑓. We will initialize f's data with ones on its diagonal.

f = Function(name='f', grid=grid)
f.data[:] = np.eye(10)
f.data
```

    Data([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)

We begin by constructing an `Operator` that increases the values of `f`, a `Function` defined on a two-dimensional `Grid`, by one.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Eq, Operator
op0 = Operator(Eq(f, f + 1))
op0.apply()
f.data
```

    Operator `Kernel` ran in 0.01 s

    Data([[2., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
          [1., 2., 1., 1., 1., 1., 1., 1., 1., 1.],
          [1., 1., 2., 1., 1., 1., 1., 1., 1., 1.],
          [1., 1., 1., 2., 1., 1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 2., 1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1., 2., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1., 1., 2., 1., 1., 1.],
          [1., 1., 1., 1., 1., 1., 1., 2., 1., 1.],
          [1., 1., 1., 1., 1., 1., 1., 1., 2., 1.],
          [1., 1., 1., 1., 1., 1., 1., 1., 1., 2.]], dtype=float32)

Every value has been updated by one. You should be able to see twos on the diagonal and ones everywhere else.

```python
#print(op0.ccode) # Print the generated code
```

# Devito API for relationals

In order to construct a `ConditionalDimension`, one needs to get familiar with relationals. The Devito API for relationals currently supports the following classes of relation, which inherit from their [SymPy counterparts](https://docs.sympy.org/latest/modules/core.html#module-sympy.core.relational):

- Le (less-than or equal)
- Lt (less-than)
- Ge (greater-than or equal)
- Gt (greater-than)
- Ne (not equal)

Relationals are used to define a condition and are passed as an argument to `ConditionalDimension` at construction time.

# Devito API for ConditionalDimension

A `ConditionalDimension` defines a non-convex iteration sub-space derived from a ``parent`` Dimension, implemented by the compiler generating conditional "if-then" code within the parent Dimension's iteration space.

A `ConditionalDimension` is used in two typical cases:

- Use case A:
To constrain the execution of loop iterations to make sure certain conditions are honoured

- Use case B:
To subsample a `Function`

```python
from devito import ConditionalDimension
print(ConditionalDimension.__doc__)
```

        Symbol defining a non-convex iteration sub-space derived from a ``parent``
        Dimension, implemented by the compiler generating conditional "if-then" code
        within the parent Dimension's iteration space.

        Parameters
        ----------
        name : str
            Name of the dimension.
        parent : Dimension, optional
            The parent Dimension.
        factor : int, optional
            The number of iterations between two executions of the if-branch. If None
            (default), ``condition`` must be provided.
        condition : expr-like, optional
            An arbitrary SymPy expression, typically involving the ``parent``
            Dimension. When it evaluates to True, the if-branch is executed. If None
            (default), ``factor`` must be provided.
        indirect : bool, optional
            If True, use `self`, rather than the parent Dimension, to
            index into arrays. A typical use case is when arrays are accessed
            indirectly via the ``condition`` expression. Defaults to False.

        Examples
        --------
        Among the other things, ConditionalDimensions are indicated to implement
        Function subsampling. In the following example, an Operator evaluates the
        Function ``g`` and saves its content into ``f`` every ``factor=4`` iterations.

        >>> from devito import Dimension, ConditionalDimension, Function, Eq, Operator
        >>> size, factor = 16, 4
        >>> i = Dimension(name='i')
        >>> ci = ConditionalDimension(name='ci', parent=i, factor=factor)
        >>> g = Function(name='g', shape=(size,), dimensions=(i,))
        >>> f = Function(name='f', shape=(int(size/factor),), dimensions=(ci,))
        >>> op = Operator([Eq(g, 1), Eq(f, g)])

        The Operator generates the following for-loop (pseudocode)

        .. code-block:: C

            for (int i = i_m; i <= i_M; i += 1) {
              g[i] = 1;
              if (i%4 == 0) {
                f[i / 4] = g[i];
              }
            }

        Another typical use case is when one needs to constrain the execution of
        loop iterations so that certain conditions are honoured. The following
        artificial example uses ConditionalDimension to guard against out-of-bounds
        accesses in indirectly accessed arrays.

        >>> from sympy import And
        >>> ci = ConditionalDimension(name='ci', parent=i,
        ...                           condition=And(g[i] > 0, g[i] < 4, evaluate=False))
        >>> f = Function(name='f', shape=(int(size/factor),), dimensions=(ci,))
        >>> op = Operator(Eq(f[g[i]], f[g[i]] + 1))

        The Operator generates the following for-loop (pseudocode)

        .. code-block:: C

            for (int i = i_m; i <= i_M; i += 1) {
              if (g[i] > 0 && g[i] < 4) {
                f[g[i]] = f[g[i]] + 1;
              }
            }

# Use case A - Honour a condition

Now it's time to show a more descriptive example. We define a conditional that increments by one all values of a Function that are larger than zero. We name our `ConditionalDimension` `ci`.
In this example we want to update again by one all the elements in `f` that are larger than zero. Before updating the elements we reinitialize our data to the eye function.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Gt

f.data[:] = np.eye(10)

# Define the condition f(x, y) > 0
condition = Gt(f, 0)

# Construct a ConditionalDimension
ci = ConditionalDimension(name='ci', parent=y, condition=condition)

op1 = Operator(Eq(f, f + 1))
op1.apply()
f.data
# print(op1.ccode) # Uncomment to view code

```

    Operator `Kernel` ran in 0.01 s

    Data([[2., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
          [1., 2., 1., 1., 1., 1., 1., 1., 1., 1.],
          [1., 1., 2., 1., 1., 1., 1., 1., 1., 1.],
          [1., 1., 1., 2., 1., 1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 2., 1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1., 2., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1., 1., 2., 1., 1., 1.],
          [1., 1., 1., 1., 1., 1., 1., 2., 1., 1.],
          [1., 1., 1., 1., 1., 1., 1., 1., 2., 1.],
          [1., 1., 1., 1., 1., 1., 1., 1., 1., 2.]], dtype=float32)

We've constructed `ci`, but it's still unused, so `op1` is still identical to `op0`. We need to pass `ci` as an "implicit `Dimension`" to the equation that we want to be conditionally executed as shown in the next cell.

```python
#NBVAL_IGNORE_OUTPUT

f.data[:] = np.eye(10)

op2 = Operator(Eq(f, f + 1, implicit_dims=ci))
print(op2.body.body[-1])
op2.apply()

assert (np.count_nonzero(f.data - np.diag(np.diagonal(f.data)))==0)
assert (np.count_nonzero(f.data) == 10)

f.data
```

    Operator `Kernel` ran in 0.01 s

    /* Begin section0 */
    START_TIMER(section0)
    for (int x = x_m; x <= x_M; x += 1)
    {
      #pragma omp simd aligned(f:32)
      for (int y = y_m; y <= y_M; y += 1)
      {
        if (f[x + 1][y + 1] > 0)
        {
          f[x + 1][y + 1] = f[x + 1][y + 1] + 1;
        }
      }
    }
    STOP_TIMER(section0,timers)
    /* End section0 */

    Data([[2., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 2., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 2., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 2., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 2., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 2., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 2., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 2., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 2., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 2.]], dtype=float32)

The generated code is as expected and only the elements that were greater than zero were incremented by one.

Let's now create a new `Function` `g`, initialized with ones along its diagonal.
We want to add `f(x,y)` to `g(x,y)` only if (i) `g != 0` and (ii) `y < 5` (i.e., over the first five columns). To join the two conditions we can use `sympy.And`.

```python
#NBVAL_IGNORE_OUTPUT

from sympy import And
from devito import Ne, Lt

f.data[:] = np.eye(10)

g = Function(name='g', grid=grid)
g.data[:] = np.eye(10)

ci = ConditionalDimension(name='ci', parent=y, condition=And(Ne(g, 0), Lt(y, 5)))

op3 = Operator(Eq(f, f + g, implicit_dims=ci))
op3.apply()

print(op3.body.body[-1])

assert (np.count_nonzero(f.data - np.diag(np.diagonal(f.data)))==0)
assert (np.count_nonzero(f.data) == 10)
assert np.all(f.data[np.nonzero(f.data[:5,:5])] == 2)
assert np.all(f.data[5:,5:] == np.eye(5))

f.data
```

    Operator `Kernel` ran in 0.01 s

    /* Begin section0 */
    START_TIMER(section0)
    for (int x = x_m; x <= x_M; x += 1)
    {
      #pragma omp simd aligned(f,g:32)
      for (int y = y_m; y <= y_M; y += 1)
      {
        if (y < 5 && g[x + 1][y + 1] != 0)
        {
          f[x + 1][y + 1] = f[x + 1][y + 1] + g[x + 1][y + 1];
        }
      }
    }
    STOP_TIMER(section0,timers)
    /* End section0 */

    Data([[2., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 2., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 2., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 2., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 2., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)

You can see that `f` has been updated only for the first five columns with the `f+g` expression.

A `ConditionalDimension` can be also used at `Function` construction time. Let's use `ci` from the previous cell to explicitly construct a `Function` `h`. You will notice that in this case there is no need to pass `implicit_dims` to the `Eq` as the `ConditionalDimension` `ci` is now a dimension in `h`. `h` still has the size of the full domain, not the size of the domain that satisfies the condition.

```python
#NBVAL_IGNORE_OUTPUT

h = Function(name='h', shape=grid.shape, dimensions=(x, ci))

op4 = Operator(Eq(h, h + g))
op4.apply()

print(op4.body.body[-1])

assert (np.count_nonzero(h.data) == 5)

h.data
```

    Operator `Kernel` ran in 0.01 s

    /* Begin section0 */
    START_TIMER(section0)
    for (int x = x_m; x <= x_M; x += 1)
    {
      for (int y = y_m; y <= y_M; y += 1)
      {
        if (y < 5 && g[x + 1][y + 1] != 0)
        {
          h[x + 1][y + 1] = g[x + 1][y + 1] + h[x + 1][y + 1];
        }
      }
    }
    STOP_TIMER(section0,timers)
    /* End section0 */

    Data([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)

# Use case B - Function subsampling

`ConditionalDimension`s are additionally indicated to implement `Function` subsampling. In the following example, an `Operator` subsamples `Function` `g` and saves its content into `f` every `factor=4` iterations.

```python
#NBVAL_IGNORE_OUTPUT
size, factor = 16, 4
i = Dimension(name='i')
ci = ConditionalDimension(name='ci', parent=i, factor=factor)

g = Function(name='g', shape=(size,), dimensions=(i,))
# Intialize g
g.data[:,]= list(range(size))
f = Function(name='f', shape=(int(size/factor),), dimensions=(ci,))

op5 = Operator([Eq(f, g)])
print(op5.body.body[-1])

op5.apply()

assert np.all(f.data[:] == g.data[0:-1:4])

print("\n Data in g \n", g.data)
print("\n Subsampled data in f \n", f.data)
```

    Operator `Kernel` ran in 0.01 s

    /* Begin section0 */
    START_TIMER(section0)
    for (int i = i_m; i <= i_M; i += 1)
    {
      if ((i)%(4) == 0)
      {
        f[i / 4] = g[i];
      }
    }
    STOP_TIMER(section0,timers)
    /* End section0 */

     Data in g
     [ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15.]

     Subsampled data in f
     [ 0.  4.  8. 12.]

Finally, we can notice that `f` holds as expected the subsampled values of `g`.

`ConditionalDimension` can be used as an alternative to save snaps to disk as illustrated in [08_snapshotting](https://github.com/devitocodes/devito/blob/master/examples/seismic/tutorials/08_snapshotting.ipynb "08_snapshotting"). The example subsamples the `TimeDimension`, capturing snaps of the our wavefield at equally spaced snaps.

# Example A - Count nonzero elements

Here we present an example of using `ConditionalDimension` by counting the nonzero elements of an array.

```python
from devito.types import Scalar
from devito import Inc

grid = Grid(shape=shape)
# Define function 𝑓. We will initialize f's data with ones on its diagonal.
f = Function(name='f', shape=shape, dimensions=grid.dimensions, space_order=0)
f.data[:] = np.eye(shape[0])
f.data[:]
```

    Data([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)

```python
#NBVAL_IGNORE_OUTPUT
# ConditionalDimension with Ne(f, 0) condition
ci = ConditionalDimension(name='ci', parent=f.dimensions[-1],
                          condition=Ne(f, 0))

# g is our counter. g needs to be a Function to write in
g = Function(name='g', shape=(1,), dimensions=(ci,), dtype=np.int32)

# Extra Scalar used to avoid reduction in gcc-5
eq0 = Inc(g[0], 1, implicit_dims=(f.dimensions + (ci,)))

op0 = Operator([eq0])
op0.apply()

assert (np.count_nonzero(f.data) == g.data[0])

print(op0.body.body[-1])
g.data[0]
```

    Operator `Kernel` ran in 0.01 s

    /* Begin section0 */
    START_TIMER(section0)
    for (int x = x_m; x <= x_M; x += 1)
    {
      for (int y = y_m; y <= y_M; y += 1)
      {
        if (f[x][y] != 0)
        {
          int r0 = 1;
          g[1] += r0;
        }
      }
    }
    STOP_TIMER(section0,timers)
    /* End section0 */

    10

# Example B - Find nonzero positions

Here we present another example of using `ConditionalDimension`. An `Operator` saves the nonzero positions of an array. We know that we have `g.data[0]` nonzero elements from Example A.

```python
#NBVAL_IGNORE_OUTPUT
count = g.data[0] # Number of nonzeros

# Dimension used only to nest different size of Functions under the same dim
id_dim = Dimension(name='id_dim')

# Conditional for nonzero element
ci = ConditionalDimension(name='ci', parent=f.dimensions[-1],
                             condition=Ne(f, 0))
g = Function(name='g', shape=(count, len(f.dimensions)),
                dimensions=(ci, id_dim), dtype=np.int32, space_order=0)

eqs = []

# Extra Scalar used to avoid reduction in gcc-5
k = Scalar(name='k', dtype=np.int32)
eqi = Eq(k, -1)
eqs.append(eqi)
eqii = Inc(k, 1, implicit_dims=(f.dimensions + (ci,)))
eqs.append(eqii)

for n, i in enumerate(f.dimensions):
    eqs.append(Eq(g[k, n], f.dimensions[n], implicit_dims=(f.dimensions + (ci,))))

op0 = Operator(eqs)
op0.apply()
print(op0.body.body[-1])

g.data[:]

ids_np = np.nonzero(f.data[:])
for i in range(len(shape)-1):
    assert all(g.data[:, i] == ids_np[i])
```

    Operator `Kernel` ran in 0.01 s

    int k = -1;

    /* Begin section0 */
    START_TIMER(section0)
    for (int x = x_m; x <= x_M; x += 1)
    {
      for (int y = y_m; y <= y_M; y += 1)
      {
        if (f[x][y] != 0)
        {
          k += 1;
          g[k][0] = x;
          g[k][1] = y;
        }
      }
    }
    STOP_TIMER(section0,timers)
    /* End section0 */

# References

- [ConditionalDimension](https://github.com/devitocodes/devito/blob/e918e26757a4f0ea3c3c416b6e0f48f4e5023c37/devito/types/dimension.py#L635 "class ConditionalDimension(DerivedDimension):") documentation.


Boundary Data Immersion Method (BDIM)

---

Marin Lauber 2020

```python
import numpy as np
import matplotlib.pyplot as plt
import NSsolver as ns
from scipy import optimize, linalg
try:
    plt.style.use("jupyter")
except OSError:
    print("Delaut syle in use")
```

The idea of the Boudnary Data Immersion method is to analytically combine the field equation of the body and the fluid analytically, through a convolution. This results in the \emph{Meta}-equation, valid ofver the full domain

\begin{equation}
    \vec{u}_\varepsilon = \mu_0 \vec{f} + (1-\mu_0)\vec{b} + \mu_1\frac{\partial}{\partial n}(\vec{f}-\vec{b})
\end{equation}

where $\mu_0$ and $\mu_1$ are the zeroth- and first- kernel moments defined as

\begin{equation}
\mu_n = \int_{\Omega_{f}} (x-x_b)^n\phi(x-x_f, \varepsilon) \text{ d}x_f,
\end{equation}

where the convolution kernel is

\begin{equation}
\phi(d, \varepsilon) = \frac{1}{2\varepsilon}\left(1+\cos\left(\frac{d}{\varepsilon}\pi\right)\right)
\end{equation}

which means that the zeroth and first moments are, respectively

\begin{split}
\mu_0 & = \frac{1}{2}\left(1 + \frac{d}{\varepsilon} + \frac{1}{pi}\sin\left(\frac{d}{\varepsilon}\pi\right)\right)\\
\mu_1 & = \varepsilon\left(\frac{1}{4}- \left(\frac{d}{2\varepsilon}\right)^2  - \frac{1}{2\pi} \left[\frac{d}{\varepsilon}\sin\left(\frac{d}{\varepsilon}\pi\right) + \frac{1}{\pi}\left[1+\cos\left(\frac{d}{\varepsilon}\pi\right)\right]\right]\right)
\end{split}

```python
def kernel(d, e):
    return np.where(abs(d)<e, 0.5*(1+np.cos(d/e*np.pi)), 0)

def mu_0(d, e):
    return np.where(d<=-e, 0, np.where(abs(d)<e, 0.5*(1 + d/e + 1./np.pi*np.sin(d*np.pi/e)), 1.))

def mu_1(d, e):
    return np.where(abs(d)<e, e*(0.25 - (d/(2*e))**2 - 1./(2*np.pi)*((d/e)*np.sin(d*np.pi/e) +\
                                                                   (1./np.pi)*(1+np.cos(d*np.pi/e)))), 0)
```

The zeroth-kernel moment must be zero inside the body ($x/\varepsilon\le-1$), and 1 in the fluid ($x/\varepsilon\ge1$). Smothing occurs over a width of $2\varepsilon$ and inhthis tansition zone, the zeroth-moment gradually ramps from 0 to 1

```python
assert mu_0(-1, 1)==0
assert mu_0( 0, 1)==0.5
assert mu_0( 1, 1)==1
```

The first kernel-moment must also vanish at the edge of moothing regionaq

```python
assert mu_1( 1, 1)==0
assert mu_1(-1, 1)==0
```

We can now define our simple 1D piston body. This is defined with a given thickness $t$ and is located at $X$

```python
def Body(x, X, t=10, e=2):
    # distance normalised by grid size
    d = abs((X - x)/(x[1]-x[0])) - t/2
    return d, mu_0(d, e), mu_1(d, e)
```

```python
x, dx = np.linspace(-2, 2, 101, retstep=True)
d, mu0, mu1 = Body(x, 0, 10, 2)
plt.plot(x, mu0, "-o", x, mu1, "-o");
```

By changing $t$ we can ealisy change the thickness of our piston

```python
x = np.linspace(-6, 6, 301)
for t, eps in zip([4, 2],[1.,0.5]):
    d  = abs(x)-t/2.
    plt.plot(x, 1.-mu_0(d, eps), '-o', markevery=5);
```

## Projection Method

To solve the coupled momentum and continuity equation, we use Chorin's [projection method](https://en.wikipedia.org/wiki/Projection_method_(fluid_dynamics)). The intermediate velocity field is obtaned by explicitely adding viscous and convective contributions, as well as the influence of the body

$$
       u^* = \mu_0\left(u^n + r_{\Delta t}(u^n)\right) + (1-\mu_0)V_p + \mu_1\frac{\partial}{\partial n}\left(u^n + r_{\Delta t}(u^n) - V_b\right),
$$

to enforce the divergence-free constraint on this intermediate velocity field, we compute a pressure field that ensures that $\nabla\cdot u^{n+1}=0$ by solving the elliptical equation

$$
    \Delta t\nabla\cdot\left(\frac{\mu_0}{\rho}\nabla p^0\right) = \nabla\cdot u^*.
$$

The final velocity field is obtained by adding the resulting pressure field to the intermediate velocity field

$$
    u^{n+1} = u^* + \frac{\Delta t}{\rho}\mu_0\nabla p^0.
$$

```python
def update(x, u, V, X, dx, dt, t):

    # get body position
    d, mu0, mu1 = Body(x, X, t, e=2)

    # predict velocity
    u_star = mu0*(u + dt*ns.r(u, dx)) + (1-mu0)*V + mu1*ns.ddn(u + dt*ns.r(u, dx) - V, dx)

    # get pressure
    sigma = ns.div(u_star, dx)
    p = ns.solve_pressure(mu0, sigma, dx)

    # correct
    u_n = u_star - dt*mu0*ns.grad(p, dx)

    return d, mu0, mu1, u_star, sigma, p, u_n
```

We can wo test the method with our piston problem, We start with a thickness $t=10$ and 64 grid points

```python
N = 64
x, dx = np.linspace(-1, 1, N, retstep=True)
xs = x + 0.5*dx # scalar cell-centred values
X = 0.0 #np.random.rand()-0.5
V = 1
u0 = np.zeros_like(x)
dt = 1.

d, mu, mu1, us, sigma, p, u_n = update(x, u0, V, X, dx, dt, 10)
print("Interface at X: %.2f" % X)
print(r"L inf %.3e" % np.max(np.abs(u_n -  V)))
```

    Interface at X: 0.00
    L inf 2.804e-10

The results match the analytical solution very well, we can draw the piston case to see what each of the contribution are

```python
ns.draw_results(x, xs, X, us, u_n, p, sigma/10, V=1, fname='None')
```


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

## Marginal and Conditional Densities ##

Let random variables $X$ and $Y$ have the joint density defined by

$$
f(x, y) ~ = ~
\begin{cases}
30(y-x)^4, ~~~ 0 < x < y < 1 \\
0 ~~~~~~~~ \text{otherwise}
\end{cases}
$$

```python
def jt_dens(x,y):
    if y < x:
        return 0
    else:
        return 30 * (y-x)**4

Plot_3d(x_limits=(0,1), y_limits=(0,1), f=jt_dens, cstride=4, rstride=4)
```

Then the possible values of $(X, Y)$ are in the upper right hand triangle of the unit square.

```python
# NO CODE

plt.figure(figsize=(3,3))
plt.axes().set_aspect('equal')
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('Possible Values of $(X, Y)$');
```

Here is a quick check by `SymPy` to see that the function $f$ is indeed a joint density.

```python
x = Symbol('x', positive=True)
y = Symbol('y', positive=True)

joint_density = 30*(y-x)**4
```

```python
Integral(joint_density, (y, x, 1), (x, 0, 1)).doit()
```

### Marginal Density of $X$ ###
We can use the joint density $f$ to find the density of $X$. Call this density $f_X$. We know that

$$
\begin{align*}
f_X(x)dx &\sim P(X \in dx) \\
&= \int_y P(X \in dx, Y \in dy) \\
&= \int_y f(x, y)dxdy \\
&= \big{(} \int_y f(x, y)dy \big{)}dx
\end{align*}
$$

You can see the reasoning behind this calculation in the graph below. The blue strip shows the event $\{ X \in dx \}$ for a value of $x$ very near 0.25. To find the volume $P(X \in dx)$, we hold $x$ fixed and add over all $y$.

```python
# NO CODE

plt.figure(figsize=(3,3))
plt.axes().set_aspect('equal')
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
plt.plot([0.25, 0.25], [0.25, 1], color='blue', lw=3, alpha=0.3)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('$X \in dx$');
```

So the density of $X$ is given by

$$
f_X(x) ~ = ~ \int_y f(x, y)dy ~~~~~ \text{for all } x
$$

By analogy with the discrete case, $f_X$ is sometimes called the *marginal density* of $X$.

In our example, the possible values of $(X, Y)$ are the upper left hand triangle as shown above. So for each fixed $x$, the possible values of $Y$ go from $x$ to 1.

Therefore for $0 < x < 1$, the density of $X$ is given by

$$
\begin{align*}
f_X(x) &= \int_x^1 30(y-x)^4 dy \\
&= 30 \cdot \frac{1}{5} (y-x)^5 \Big{\rvert}_x^1 \\
&= 6(1-x)^5
\end{align*}
$$

Here is the joint density surface again. You can see that $X$ is much more likely to be near 0 than near 1.

```python
Plot_3d(x_limits=(0,1), y_limits=(0,1), f=jt_dens, cstride=4, rstride=4)
```

That can be seen in the shape of the density of $X$.

```python
# NO CODE
x_vals = np.arange(0, 1.01, 0.01)
f_X = 6*(1-x_vals)**5
plt.plot(x_vals, f_X, color='darkblue', lw=2)
plt.xlabel('$x$')
plt.ylabel('$f_X(x)$', rotation=0)
plt.title('$f_X$: Density of $X$');
```

### Density of $Y$ ###
Correspondingly, the density of $Y$ can be found by fixing $y$ and integrating over $x$ as follows:

$$
f_Y(y) = \int_x f(x, y)dx ~~~~ \text{for all } y
$$

In our example, the joint density surface indicates that $Y$ is more likely to be near 1 than near 0, which is confirmed by calculation. Remember that $y > x$ and therefore for each fixed $y$, the possible values of $x$ are 0 through $y$.

For $0 < y < 1$,

$$
f_Y(y) ~ = ~ \int_0^y 30(y-x)^4dx ~ = ~ 6y^5
$$

```python
# NO CODE
y_vals = np.arange(0, 1.01, 0.01)
f_Y = 6*y_vals**5
plt.plot(y_vals, f_Y, color='darkblue', lw=2)
plt.xlabel('$y$')
plt.ylabel('$f_Y(y)$', rotation=0)
plt.title('$f_Y$: Density of $Y$');
```

### Conditional Densities ###
Consider the conditional probability $P(Y \in dy \mid X \in dx)$. By the division rule,

$$
P(Y \in dy \mid X \in dx) ~ = ~
\frac{P(X \in dx, Y \in dy)}{P(X \in dx)} ~ = ~
\frac{f(x, y)dxdy}{f_X(x)dx} ~ = ~
\frac{f(x, y)}{f_X(x)} dy
$$

This gives us a division rule for densities. For a fixed value $x$, the *conditional density of $Y$ given $X=x$* is defined by

$$
f_{Y\mid X=x} (y) ~ = ~ \frac{f(x, y)}{f_X(x)} ~~~~ \text{for all } y
$$

Since $X$ has a density, we know that $P(X = x) = 0$ for all $x$. But the ratio above is of densities, not probabilities. It might help your intuition to think of "given $X=x$" to mean "given that $X$ is just around $x$".

Visually, the shape of this conditional density is the vertical cross section at $x$ of the joint density graph above. The numerator determines the shape, and the denominator is part of the constant that makes the density integrate to 1.

Note that $x$ is constant in this formula; it is the given value of $X$. So the denominator $f_X(x)$ is the same for all the possible values of $y$.

To see that the conditional density does integrate to 1, let's do the integral.

$$
\int_y f_{Y\mid X=x} (y)dy ~ = ~ \int_y \frac{f(x, y)}{f_X(x)} dy ~ = ~
\frac{1}{f_X(x)} \int_y f(x, y)dy ~ = ~ \frac{1}{f_X(x)} f_X(x) ~ = ~ 1
$$

In our example, let $x = 0.2$ and consider finding the conditional density of $Y$ given $X = 0.4$. Under that condition, the possible values of $Y$ are in the range 0.4 to 1, and therefore

$$
f_{Y \mid X=0.4} (y) ~ = ~ \frac{30(y - 0.4)^4}{6(1 - 0.4)^5} ~ = ~
\frac{5}{0.6^5} (y - 0.4)^4 ~~~~ y \in (0.4, 1)
$$

This is a density on $(0.4, 1)$:

```python
y = Symbol('y', positive=True)

conditional_density_Y_given_X_is_04 = (5/(0.6**5)) * (y - 0.4)**4

Integral(conditional_density_Y_given_X_is_04, (y, 0.4, 1)).doit()
```

The figure below shows the overlaid graphs of the density of $Y$ and the conditional density of $Y$ given $X = 0.4$. You can see that the conditional density is more concentrated on large values of $Y$, because under the condition $X = 0.4$ you know that $Y$ can't be small.

```python
# NO CODE
plt.plot(y_vals, f_Y, color='darkblue', lw=2, label='Density of $Y$')
new_y = np.arange(0.4, 1.01, 0.01)
dens = (5/(0.6**5)) * (new_y - 0.4)**4
plt.plot(new_y, dens, color='gold', lw=2, label='Density of $Y$ given $X=0.4$')
plt.legend()
plt.xlim(0, 1)
plt.xlabel('$y$');
```

### Using a Conditional Density ###
We can use conditional densities to find probabilities and expectations, just as we would use an ordinary density. Here are some examples of calculations. In each case we will set up the integrals and then use `SymPy`.

$$
P(Y > 0.9 \mid X = 0.4) = \int_{0.9}^1 \frac{5}{0.6^5} (y - 0.4)^4 dy
$$

The answer is about 60%.

```python
Integral(conditional_density_Y_given_X_is_04, (y, 0.9, 1)).doit()
```

Now we will use the conditional density to find a conditional expectation. Remember that in our example, given that $X = 0.4$ the possible values of $Y$ go from $0.4$ to 1.

$$
E(Y \mid X = 0.4) ~ = ~ \int_{0.4}^1 y \frac{5}{0.6^5} (y - 0.4)^4 dy ~ = ~ 0.9
$$

```python
Integral(y*conditional_density_Y_given_X_is_04, (y, 0.4, 1)).doit()
```

You can condition $X$ on $Y$ in the same way. By analogous arguments, for any fixed value of $y$ the conditional density of $X$ given $Y = y$ is

$$
f_{X \mid Y=y} (x) ~ = ~ \frac{f(x, y)}{f_Y(y)} ~~~~~ \text{for all } x
$$

All the examples in this section and the previous one have started with a joint density function that apparently emerged out of nowhere. In the next section, we will study a context in which they arise.

```python

```


## Natural language processing and non-negative matrix factorization (NMF) for topic modeling

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as stats

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.datasets import fetch_20newsgroups
from sklearn.decomposition import NMF, LatentDirichletAllocation
import TCD19_utils as TCD
TCD.set_plotting_style_2()

# Magic command to enable plotting inside notebook
%matplotlib inline
# Magic command to enable svg format in plots
%config InlineBackend.figure_format = 'svg'
#Set random seed
seed = np.random.seed(42)
```

## Natural language processing (NLP) vocabulary

Text is by itself an unstructured data type.

* corpus: set of documents
* stop words:
* tokenization: Tokenization breaks unstructured data, in this case text, into chunks of information which can be counted as discrete elements. These counts of token occurrences in a document can be used directly as a vector representing that document. This immediately turns an unstructured string (text document) into a structured, numerical data structure suitable for machine learning.
* n-gram :

```python
dataset = fetch_20newsgroups(shuffle=True, random_state=1,
                             remove=('headers', 'footers', 'quotes'))
documents = dataset.data
```

```python
type(documents)
```

    list

```python
documents[-1]
```

    '                                      ^^^^^^\nNo argument at all with Murphy.  He scared the hell out of me when he came in\nlast year.  On the other hand, the club though enough of Boever to put him into\nan awful lot of games (he may have led the league in appearances - he did at\nleast at some point).  He seemed to be a very viable setup guy - but I guess\nthat\'s not considered that crucial by the club.  I can just remember two years\nago so well, though...\n...\n\nI\'m not that concerned.  Those guys have been relatively consistent over the\nyears and they have no good reasons to decline (no injuries, not old, ...).\nI expect them to come through just fine.  It\'s those guys that have not\nbeen consistently good that are the worrisome part, even if they are coming\nthrough right now.\n\nThis sounds like their old road unis.  Pretty dull.  Buttons or pullovers?\nI\'ll check through my uniform book to see if they\'ve always had some orange.\n\n\nWell, we\'ll see.  I\'ve got a Astros pullover shirt with the "Astros stripes"\nacross the shoulders and I have trouble making myself wear it in public.  i\ncan see why they might want that to change.  Gee, if they eliminate the\norange, will they reupholster the seats in the Astros stripes section (what\nused to be the gold and yellow levels - I don\'t know those numbers they use\nnow).\n\nI saw a pinstripe version of an Astros cap and I actually thought it looked \ngood!'

We can see that each element of the list is a document containing a news article. We will now proceed to transform our document list to a term frequency-inverse document frequency (TF-IDF) matrix. This matrix has a  (`docs`, `words`) shape, where the words will be our features. The values inside the matrix will be TF-IDF that is defined by the following formula.

\begin{align}
\text{TF-IDF} = \text{TF} \times \log{ \frac{N}{DF}}
\end{align}

Where:
$\text{TF} = \text{word} \text{counts in} \, document_{i}$,  $\text{DF (document frequency) =  counts of the documents that contain} \, word_{i} $ , $\text{N = total number of documents}$.

Notice that if a word is ubiquitous in all documents (e.g. stop words), the term in the right turns to zero. In this sense TF-IDF makes a robust assesment of the word frequency in a document, eliminating the bias of highly repeated words. This makes sense as ubiquitous words throughout different documents might not contain any information about the documents themselves.

The sci-kit learn package has a great implementation using the `tfidf_vectorizer`. Some important arguments of this function are `max_df` threshold for the proportion of documents that have highly repeated words, (`min_df`is the low limit cutoff, if `int` it is in counts), `stop_words` let's you pick a language to eliminate stop words from, and `max_features` let's you consider the top words term frequency across the corpus.

Let's compute the TF-IDF vectorizer on our documents.

```python
no_features = 1000

# NMF is able to use tf-idf
tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2,
                                   max_features=no_features,
                                   stop_words='english')

tfidf = tfidf_vectorizer.fit_transform(documents)
tfidf_feature_names = tfidf_vectorizer.get_feature_names()

```

```python
type(tfidf)
```

We can see that our TF-IDF matrix is a scipy sparse matrix and that we cannot readily visualize it's components.

```python
tfidf_feature_names[-1]
```

We can see that our features are words.

```python
df_tfidf = pd.DataFrame(tfidf.toarray(), columns=tfidf_feature_names)

df_tfidf.head()
```

```python
df_tfidf.shape
```

Now we are ready to extract the topics from our documents.

## Non-negative matrix factorization (NMF)

NMF can be thought of as a clustering technique, as  it finds a decomposition of samples $\matrix{X}$ into two matrices $\matrix{W}$ and $\matrix{H}$ of non-negative elements, by minimizing the distance d between $\matrix{X}$ and the matrix product $\matrix{W}$$\matrix{H}$ . More specifically each matrix represents the following:

$\matrix{W}$ (clusters) = the topics (clusters) discovered from the documents.
$\matrix{H}$ (coefficient matrix) = the membership weights for the topics in each document.
$\matrix{X}$ (Document-word matrix) — input that contains which words appear in which documents.

\begin{align}
\matrix{X} = \matrix{W}\matrix{H}
\end{align}

We won't go to any mathematical detail but just know that the current implementation in scikit learn uses the minimization of the Frobenius norm, the matrix analog of the euclidean distance, and that you can use different divergence measurements like the Kullback-leibler divergence by modifying the `beta_loss` parameter.

```python
import sklearn
```

```python
sklearn.__version__
```

```python
n_topics = 10

# Run NMF on the TF-IDF matrix
nmf = NMF(n_components=n_topics).fit(tfidf)
```

```python
#Transform
nmf_W = nmf.transform(tfidf)
nmf_H = nmf.components_

```

```python
print('Topic (W) matrix has a', nmf_W.shape , 'shape')
print('Coeffficient (H) matrix has a ', nmf_H.shape, 'shape')
```

We can see that the $\matrix{W}$ matrix has a (`n_documents`, `n_topics`) shape.

We can readily see that the NMF $\matrix{H}$ matrix has a shape of (`n_topics`, `n_features`) .

Therefore if we want to get the topic associations from each document we must get the biggest argument from the topic matrix.

```python
nmf_topics = []
for i in range(nmf_W.shape[0]):
    nmf_topics.append(nmf_W[i].argmax())
```

```python
def display_topics(model, feature_names, no_top_words):

    for topic_idx, topic in enumerate(model.components_):

        #print topic index
        print ("Topic %d:" % (topic_idx))

        #print topic
        print (" ".join([feature_names[i] for i in topic.argsort()[: - no_top_words -1 :-1]]))

```

```python
no_top_words = 15
display_topics(nmf, tfidf_feature_names, no_top_words)
```

Voilà. We have our topics and the most important words associated with them. From this we can readily make a list that will serve for visualization purposes.

```python
classes = ['random','video','catholic church','gamers', 'bike / car selling',
           'email','windows','computer science','cybersecurity',
           'hardware']
```

Now, let's see if by using a PCA on the topic matrix, we can visualize the documents in its document space.

```python
palette = TCD.palette(cmap = True)
```

```python
from sklearn.decomposition import PCA

pca = PCA(random_state = 42)

doc_pca = pca.fit_transform(nmf_W)
```

```python
plt.figure(figsize = (8,6))

plt.scatter(doc_pca[:,0], doc_pca[:, 1], alpha = 0.8,
            c = nmf_topics, cmap = palette.reversed())

plt.xlabel('PCA 1')
plt.ylabel('PCA 2')
cbar = plt.colorbar(boundaries=np.arange(11)-0.5)
cbar.set_ticks(np.arange(10))
cbar.set_ticklabels(classes)

#plt.savefig('news_pca.png', dpi = 500, bbox_inches = 'tight')
plt.tight_layout();

```

```python
from umap import UMAP
```

```python
reducer = UMAP(random_state = 42)

doc_umap = reducer.fit_transform(nmf_W)
```

```python
plt.figure(figsize = (8,6))

plt.scatter(doc_umap[:,0], doc_umap[:, 1], alpha = 0.8,
            c = nmf_topics, cmap = palette.reversed())

plt.xlabel('UMAP 1')
plt.ylabel('UMAP 2')
cbar = plt.colorbar(boundaries=np.arange(11)-0.5)
cbar.set_ticks(np.arange(10))
cbar.set_ticklabels(classes)

#plt.savefig('news_UMAP.png', dpi = 500, bbox_inches = 'tight')
plt.tight_layout();
```

```python
reducer = UMAP(random_state = 42, y = nmf_topics)

doc_umap = reducer.fit_transform(nmf_W)
```

```python
plt.figure(figsize = (8,6))

plt.scatter(doc_umap[:,0], doc_umap[:, 1], alpha = 0.8,
            c = nmf_topics, cmap = palette.reversed())

plt.xlabel('UMAP 1')
plt.ylabel('UMAP 2')
cbar = plt.colorbar(boundaries=np.arange(11)-0.5)
cbar.set_ticks(np.arange(10))
cbar.set_ticklabels(classes)

#plt.savefig('news_UMAP_learn.png', dpi = 500, bbox_inches = 'tight')
plt.tight_layout();
```

We have seen an end to end clustering and visualization pipeline using NMF, PCA, and UMAP. We could also use another method for topic modelling in text called Latent Dirichlet Allocation (LDA). The implementation is very similar, but if you want to see how to do so follow this [great post](https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730).

Other than clustering, NMF can be applied for collaborative filtering and image analysis.


# Random Signals and LTI-Systems

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Auto-Correlation Function

The auto-correlation function (ACF) $\varphi_{yy}[\kappa]$ of the output signal of an LTI system $y[k] = \mathcal{H} \{ x[k] \}$ is derived. It is assumed that the input signal is a wide-sense stationary (WSS) real-valued random process and that the LTI system has a real-valued impulse repsonse $h[k] \in \mathbb{R}$.

Introducing the output relation $y[k] = h[k] * x[k]$ of an LTI system into the definition of the ACF and rearranging terms yields

\begin{align}
\varphi_{yy}[\kappa] &= E \{ y[k+\kappa] \cdot y[k] \} \\
&= E \left\{  \sum_{\mu = -\infty}^{\infty} h[\mu] \; x[k+\kappa-\mu] \cdot
\sum_{\nu = -\infty}^{\infty} h[\nu] \; x[k-\nu] \right\} \\
&= \underbrace{h[\kappa] * h[-\kappa]}_{\varphi_{hh}[\kappa]} * \varphi_{xx}[\kappa]
\end{align}

where the ACF $\varphi_{hh}[\kappa]$ of the deterministic impulse response $h[k]$ is commonly termed as *filter ACF*. This is related to the [link between ACF and convolution](../random_signals/correlation_functions.ipynb#Definition). The result above is known as the *Wiener-Lee theorem*. It states that the ACF of the output $\varphi_{yy}[\kappa]$ of an LTI system is given by the convolution of the input signal's ACF $\varphi_{xx}[\kappa]$ with the filter ACF $\varphi_{hh}[\kappa]$.

### Example - System Response to White Noise

Let's assume that the input signal $x[k]$ of an LTI system with impulse response $h[k] = \text{rect}_N[k]$ is normal distributed white noise. Hence, $\varphi_{xx}[\kappa] = N_0\, \delta[\kappa]$ which can be introduced into the Wiener-Lee theorem yielding

\begin{equation}
\varphi_{yy}[\kappa] = N_0 \cdot \varphi_{hh}[\kappa] = N_0 \cdot (\text{rect}_N[\kappa] * \text{rect}_N[-\kappa])
\end{equation}

The example is evaluated numerically for $N_0 = 1$ and $N=5$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

L = 10000  # number of samples
K = 30  # limit for lags in ACF

# generate input signal (white Gaussian noise)
np.random.seed(2)
x = np.random.normal(size=L)
# compute system response
y = np.convolve(x, [1, 1, 1, 1, 1], mode='full')

# compute and truncate ACF
acf = 1/len(y) * np.correlate(y, y, mode='full')
acf = acf[len(y)-K-1:len(y)+K-1]
kappa = np.arange(-K, K)

# plot ACF
plt.figure(figsize = (10, 6))
plt.stem(kappa, acf)
plt.title('Estimated ACF of output signal $y[k]$')
plt.ylabel(r'$\hat{\varphi}_{yy}[\kappa]$')
plt.xlabel(r'$\kappa$')
plt.axis([-K, K, 1.2*min(acf), 1.1*max(acf)]);
plt.grid()
```

**Exercise**

* Derive the theoretic result for $\varphi_{yy}[\kappa]$ by calculating the filter-ACF $\varphi_{hh}[\kappa]$.
* Why is the estimated ACF $\hat{\varphi}_{yy}[\kappa]$ of the output signal not exactly equal to its theoretic result $\varphi_{yy}[\kappa]$?
* Change the number of samples `L` and rerun the example. What changes?

The filter-ACF is given by $\varphi_{hh}[\kappa] = \text{rect}_N[\kappa] * \text{rect}_N[-\kappa]$. The convolution of two rectangular signals $\text{rect}_N[\kappa]$ results in a triangular signal. Taking the time reversal into account yields

\begin{equation}
\varphi_{hh}[\kappa] = \begin{cases}
N - |\kappa| & \text{for } -N < \kappa \leq N \\
0 & \text{otherwise}
\end{cases}
\end{equation}

for even $N$. The estimated ACF $\hat{\varphi}_{yy}[\kappa]$ differs from its theoretic value due to the statistical uncertainties when using random signals of finite length. Increasing its length `L` lowers the statistical uncertainties.

## Cross-Correlation Function

The cross-correlation functions (CCFs) $\varphi_{xy}[\kappa]$ and $\varphi_{yx}[\kappa]$ between the in- and output signal of an LTI system $y[k] = \mathcal{H} \{ x[k] \}$ are derived. As for the ACF it is assumed that the input signal originates from a wide-sense stationary real-valued random process and that the LTI system's impulse response is real-valued, i.e. $h[k] \in \mathbb{R}$.

Introducing the convolution into the definition of the CCF and rearranging the terms yields

\begin{align}
\varphi_{xy}[\kappa] &= E \{ x[k+\kappa] \cdot y[k] \} \\
&= E \left\{ x[k+\kappa] \cdot \sum_{\mu = -\infty}^{\infty} h[\mu] \; x[k-\mu] \right\} \\
&= \sum_{\mu = -\infty}^{\infty} h[\mu] \cdot E \{ x[k+\kappa] \cdot x[k-\mu] \} \\
&= h[-\kappa] * \varphi_{xx}[\kappa]
\end{align}

The CCF $\varphi_{xy}[\kappa]$ between in- and output is given as the time-reversed impulse response of the system convolved with the ACF of the input signal.

The CCF between out- and input is yielded by taking the symmetry relations of the CCF and ACF into account

\begin{equation}
\varphi_{yx}[\kappa] = \varphi_{xy}[-\kappa] = h[\kappa] * \varphi_{xx}[\kappa]
\end{equation}

The CCF $\varphi_{yx}[\kappa]$ between out- and input is given as the impulse response of the system convolved with the ACF of the input signal.

## System Identification by Cross-Correlation

The process of determining the impulse response or transfer function of a system is referred to as *system identification*. The CCFs of an LTI system play an important role in the estimation of the impulse response $h[k]$ of an unknown system. This is illustrated in the following.

The basic idea is to use a specific measurement signal as input signal to the system. Let's assume that the unknown LTI system is excited by [white noise](../random_signals/white_noise.ipynb). The ACF of the input signal $x[k]$ is then given as $\varphi_{xx}[\kappa] = N_0 \cdot \delta[\kappa]$. According to the relation derived above, the CCF between out- and input for this special choice of the input signal becomes

\begin{equation}
\varphi_{yx}[\kappa] = h[\kappa] * N_0 \cdot \delta[\kappa] = N_0 \cdot h[\kappa]
\end{equation}

For white noise as input signal $x[k]$, the impulse response of an LTI system can be estimated by estimating the CCF between its out- and input signals. Using noise as measurement signal instead of a Dirac impulse is beneficial since its [crest factor](https://en.wikipedia.org/wiki/Crest_factor) is limited.

### Example

The application of the CCF to the identification of a system is demonstrated. The system is excited by normally distributed white noise with $N_0 = 1$. The ACF of the in- and output, as well as the CCF between out- and input is estimated and plotted.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 10000  # number of samples for input signal
K = 50  # limit for lags in ACF

# generate input signal
np.random.seed(5)
x = np.random.normal(size=N)  # normally distributed (zero-mean, unit-variance) white noise
# impulse response of the system
h = np.concatenate((np.zeros(10), sig.triang(10), np.zeros(10)))
# output signal by convolution
y = np.convolve(h, x, mode='full')

# compute correlation functions
acfx = 1/len(x) * np.correlate(x, x, mode='full')
acfy = 1/len(y) * np.correlate(y, y, mode='full')
ccfyx = 1/len(y) * np.correlate(y, x, mode='full')

def plot_correlation_function(cf):
    cf = cf[N-K-1:N+K-1]
    kappa = np.arange(-len(cf)//2,len(cf)//2)
    plt.stem(kappa, cf)
    plt.xlabel(r'$\kappa$')
    plt.axis([-K, K, -0.2, 1.1*max(cf)])

# plot ACFs and CCF
plt.rc('figure', figsize=(10, 3))
plt.figure()
plot_correlation_function(acfx)
plt.title('Estimated ACF of input signal')
plt.ylabel(r'$\hat{\varphi}_{xx}[\kappa]$')

plt.figure()
plot_correlation_function(acfy)
plt.title('Estimated ACF of output signal')
plt.ylabel(r'$\hat{\varphi}_{yy}[\kappa]$')

plt.figure()
plt.hold(True)
plot_correlation_function(ccfyx)
plt.plot(np.arange(len(h)), h, 'g-')
plt.title('Estimated and true impulse response')
plt.ylabel(r'$\hat{h}[k]$, $h[k]$');
```

**Exercise**

* Why is the estimated CCF $\hat{\varphi}_{yx}[k]$ not exactly equal to the true impulse response $h[k]$ of the system?
* What changes if you change the number of samples `N` of the input signal?

The derived relations for system identification hold for the case of a wide-sense ergodic input signal of infinite duration. Since we can only numerically simulate signals of finite duration, the observed deviations are a result of the resulting statistical uncertainties. Increasing the length `N` of the input signal improves the estimate of the impulse response.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


```python
import torch
import numpy as np
from IPython.display import Image
from IPython.core.display import HTML
```

## What is PyTorch?
- A python library to operate on Tensors as a replacement over Numpy
- A Deep Learning library that provides easy of use and flexibility for researchers and practitioners

## 1. Create new tensors

### 1.1 Empty Tensor

```python
torch.empty(5, 4) #the values are close to zero
```

    tensor([[2.9159e+38, 4.5615e-41, 2.9159e+38, 4.5615e-41],
            [0.0000e+00, 0.0000e+00, 5.9819e+19, 4.5615e-41],
            [0.0000e+00, 0.0000e+00, 4.9615e+19, 4.5615e-41],
            [0.0000e+00, 0.0000e+00,        nan, 3.3460e-12],
            [1.4013e-45, 0.0000e+00, 4.9518e+28, 7.9463e+08]])

### 1.2 Random Tensor

```python
torch.randn(1, 4) # generates random values
```

    tensor([[ 0.2900, -1.8238,  0.0839, -0.5769]])

### 1.3 Zero Tensor

```python
torch.zeros(3, 3)
```

    tensor([[0., 0., 0.],
            [0., 0., 0.],
            [0., 0., 0.]])

### 1.4 From a list

```python
x = torch.tensor([[2.3, 4], [1.2, -0.3]])
```

```python
x
```

    tensor([[ 2.3000,  4.0000],
            [ 1.2000, -0.3000]])

### 1.5 From an existing tensor

```python
# rand - random positive
# randn - random positive and negative
torch.randn_like(x)
# torch.ones_like(x)
# torch.zeros_like(x)
```

    tensor([[-1.7953, -0.1728],
            [-0.2738, -0.2925]])

### 1.6 From numpy array

```python
n = np.array([[1, 2], [0.3, -1.2]])

torch.from_numpy(n)
```

    tensor([[ 1.0000,  2.0000],
            [ 0.3000, -1.2000]], dtype=torch.float64)

## 2. Slicing Tensors

If you are already familiar with numpy, you can skip this section

```python
x = torch.randn(3, 4, 5)
```

```python
x
```

    tensor([[[ 1.1930, -0.3808, -0.8907, -1.7130,  0.1500],
             [-0.3511, -1.1963,  0.3327, -1.1608, -0.4557],
             [ 1.9224, -0.9547,  0.5528,  2.0848,  0.9394],
             [-0.7937,  0.4034, -0.3482, -0.2343,  0.3509]],

            [[-0.6110, -0.4482, -1.2750, -0.4663,  0.2621],
             [ 0.2856, -1.0831,  0.9597,  0.5971, -1.4496],
             [-0.7251,  0.8108, -0.1436, -0.5671,  0.5719],
             [-0.8660, -1.1224,  0.2852,  0.2463, -0.9215]],

            [[-0.2689,  0.7704,  1.1415,  0.7557, -0.4813],
             [-0.2387,  2.9216, -0.1847,  0.9027,  1.1900],
             [ 1.5327,  0.3733, -0.9201,  0.6989,  0.0082],
             [ 0.1920,  1.0249,  0.6636,  1.1873,  0.1856]]])

### 2.1 Know the size

```python
x.size()
```

    torch.Size([3, 4, 5])

### 2.2 Access with numpy style slicing

```python
# identify the dimension you want to access [3, 4, 5]

# if you want all elements from first dimension, 2nd element from 2nd dimension, all elements from 3rd dimension
x[:, 1, :]
```

    tensor([[-0.3511, -1.1963,  0.3327, -1.1608, -0.4557],
            [ 0.2856, -1.0831,  0.9597,  0.5971, -1.4496],
            [-0.2387,  2.9216, -0.1847,  0.9027,  1.1900]])

```python
# if you want 1st element from first dimension, 2nd element from 2nd dimension, all elements from 3rd dimension
x[0, 1, :]
```

    tensor([-0.3511, -1.1963,  0.3327, -1.1608, -0.4557])

```python
# if you want first two elements from first dimension,
# first 3 elements from 2nd dimension and
# only 4th element from 3rd dimension
x[:2, :3, 3]
```

    tensor([[-1.7130, -1.1608,  2.0848],
            [-0.4663,  0.5971, -0.5671]])

## 3. Tensor Transform

### 3.1 Resize

```python
x.size()
```

    torch.Size([3, 4, 5])

```python
x.view(12, 5)
# x.view(3, 20)
# x.view(4, 15)
# x.view(60)
```

    tensor([[ 1.1930, -0.3808, -0.8907, -1.7130,  0.1500],
            [-0.3511, -1.1963,  0.3327, -1.1608, -0.4557],
            [ 1.9224, -0.9547,  0.5528,  2.0848,  0.9394],
            [-0.7937,  0.4034, -0.3482, -0.2343,  0.3509],
            [-0.6110, -0.4482, -1.2750, -0.4663,  0.2621],
            [ 0.2856, -1.0831,  0.9597,  0.5971, -1.4496],
            [-0.7251,  0.8108, -0.1436, -0.5671,  0.5719],
            [-0.8660, -1.1224,  0.2852,  0.2463, -0.9215],
            [-0.2689,  0.7704,  1.1415,  0.7557, -0.4813],
            [-0.2387,  2.9216, -0.1847,  0.9027,  1.1900],
            [ 1.5327,  0.3733, -0.9201,  0.6989,  0.0082],
            [ 0.1920,  1.0249,  0.6636,  1.1873,  0.1856]])

### 3.1 Add a new dimension

```python
# only one dimension can be added
x.view(3, 4, -1, 5).size()
# x.view(3, 4, 5, -1).size()
# x.view(-1, 3, 4, 5).size()
```

    torch.Size([3, 4, 1, 5])

## 4. Operations

### 4.1 Addition

```python
x = torch.rand(3, 4)
y = torch.rand(3, 4)
```

```python
x + y
```

    tensor([[1.3400, 0.6052, 0.7668, 1.6158],
            [0.7968, 0.2113, 1.1010, 0.6425],
            [1.0882, 0.6075, 0.8668, 1.0942]])

```python
x.add(y)
```

    tensor([[1.3400, 0.6052, 0.7668, 1.6158],
            [0.7968, 0.2113, 1.1010, 0.6425],
            [1.0882, 0.6075, 0.8668, 1.0942]])

```python
# inplace
x.add_(y)
```

    tensor([[1.3400, 0.6052, 0.7668, 1.6158],
            [0.7968, 0.2113, 1.1010, 0.6425],
            [1.0882, 0.6075, 0.8668, 1.0942]])

```python
x
```

    tensor([[1.3400, 0.6052, 0.7668, 1.6158],
            [0.7968, 0.2113, 1.1010, 0.6425],
            [1.0882, 0.6075, 0.8668, 1.0942]])

```python
torch.add(x, y)
```

    tensor([[1.9126, 0.8663, 0.9828, 2.3942],
            [1.0305, 0.3241, 1.7627, 1.1404],
            [1.9983, 0.9439, 1.1492, 1.5844]])

```python
# torch.mul(x, y)
# torch.div(x, y)
torch.sub(x, y)
```

    tensor([[0.7674, 0.3441, 0.5507, 0.8373],
            [0.5630, 0.0985, 0.4394, 0.1447],
            [0.1781, 0.2711, 0.5844, 0.6040]])

### More such operations can be found here https://pytorch.org/docs/stable/torch.html

## 5. Autograd

- Automatic differentiation on all tensor operations provided as part of the library
- Operates on tensor
- When creating a tensor set .requires_grad to True

\begin{align}
x & = \begin{bmatrix}{x}_1\\
{x}_2 \\
..
\\
{x}_n
\end{bmatrix} \\ \\
f(x) & = \sum_{n}  3 * (x+2)^2 \\ \\
\frac{d f}{dx} & = \begin{bmatrix}
6({x}_1 + 2) \\
6({x}_2 + 2) \\
.. \\
6({x}_n + 2)
\end{bmatrix}\\
\end{align}

### 5.1 Set requires_grad as True

```python
x = torch.ones(10, requires_grad=True)
x
```

    tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], requires_grad=True)

### 5.2 Perform Operations on the tensor

```python
y = x+2
z = 3 * y * y
out = sum(z)
```

```python
out
```

    tensor(270., grad_fn=<AddBackward0>)

### 5.3 Perform backward operation to calculate gradients

```python
out.backward()
```

### 5.4 Check gradients

```python
x.grad
```

    tensor([18., 18., 18., 18., 18., 18., 18., 18., 18., 18.])

### 5.5 Stop tracking operations

```python
x.detach()
# with torch.no_grad():
#      perform operations
```

### 5.6 Convert to Scalar

```python
y # needs a scalar to compute gradients
```

    tensor([3., 3., 3., 3., 3., 3., 3., 3., 3., 3.], grad_fn=<AddBackward0>)

```python
Image(url= "../Jacobian.jpg")
```

```python
Image(url = "../Jacobian-vector.jpg")
```

```python
y.backward(torch.ones(10))
```


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import numpy.polynomial.polynomial as poly
```

Constructing a function $f(x)$

```python
f = lambda x : x**2
print f(4)
```

    16

Construct the dual $f^*$ of $f$.

```python
x = lambda f: f(4.)
g = lambda x: x**2
print x(g)
```

    16.0

```python

```

Construct a set of this special points, that given a function return a value, we call this set $\hat N$ (the **set of nodal functions**, that are a basis for the dual space of $\hat{\mathcal{P}}^k([0,1])$).

```python
def construct_dual_basis():
    N = []
    N.append(lambda f: f(0.))
    #N.append(lambda f: f(1./3))
    #N.append(lambda f: f(.5))
    #N.append(lambda f: f(2./3))
    N.append(lambda f: f(1.))
    return N
```

```python
p = lambda p : p(0)
f = lambda x : x

Nk = construct_dual_basis()

print Nk
print Nk[0](f)
print Nk[1](f)
```

    [<function <lambda> at 0x1061b8de8>, <function <lambda> at 0x1061ce140>]
    0.0
    1.0

$\mathcal{P}^k([0,1]) = \textrm{span}\{e_i\}_{i=0}^k =: \textrm{span} E$
As we are free to choose our polinomials, we pick them in their simplest possible form, monomials:
$$
e_i = \mathrm{pow}(x,i)
$$

```python
def construct_poly(deg):
    E = []
    for i in range(deg+1):
        c = np.zeros((i+1,))
        c[-1] = 1
        E.append(poly.Polynomial(c))
    return E
```

```python
deg = 1
```

```python
Tk = np.array([0,1])
Nk = construct_dual_basis()
Pk = construct_poly(deg)
print Pk
```

    [Polynomial([ 1.], [-1,  1], [-1,  1]), Polynomial([ 0.,  1.], [-1,  1], [-1,  1])]

We construct basis functions $v_i$ asking them to be those particular polynomials orthogonal to the degrees of freedom $v^j$. In a compact way this means:

$$
v^j\, v_i = \delta^j_i
$$

$v^j$ are a special kind of points that given a function the do return a value. We can write this value for the $k$-th polynomial:

$v^j(e_k)$, and introduce the matrix $C_k^j$. This matrix clarifies the way canonical basis functions are related with the degrees of freedom:

$$
v^j\, v_i = \delta^j_i, \quad v^j(e_k)\, v^k_i = \delta^j_i, \quad C^j_k\, v^k_i = \delta^j_i
$$

Reading the last formula we get an easy way to construct the basis functions:

$$
v^k_i = (C^j_k)^{-1}\delta^j_i
$$

In practice these $v^k_i$ are the cofficents for the polynomials that are going to become the basis for our space.

```python
def construct_cij(N,P):
    C = np.zeros((0, len(P)))  # N are the degrees of freedom
    for dof in N:
        row = np.array([])
        for p in P:
            row = np.hstack((row,dof(p))) # horizontan append
        C = np.vstack((C,row))
    return C
```

```python
C = construct_cij(Nk,Pk)
print C
```

    [[ 1.  0.]
     [ 1.  1.]]

```python
def construct_canonical_basis(C):
    V = []
    for k in range(C.shape[0]):
        delta_ij = np.zeros((C.shape[0],))
        delta_ij[k] = 1.
        vk = np.linalg.solve(C, delta_ij)
        V.append(poly.Polynomial(vk))
    return V
```

```python
C = construct_cij(Nk,Pk)

V = construct_canonical_basis(C)

s = np.linspace(0,1,1025)
for v in V:
    plt.plot(s, v(s))
plt.show()
```

```python
from numpy.polynomial.legendre import leggauss

def apply_matrx_bc(A):
    ck = A[0,0]
    A[0] = 0
    A[0,0] = ck

    ck = A[-1,-1]
    A[-1] = 0
    A[-1,-1] = ck

def apply_rhs_bc(rhs):
    rhs[0] = 0
    rhs[-1] = 0

def assemble_system(x,V,myf):

    h = 1./(len(x)-1)

    degree = 2*len(V)-1
    q,w = leggauss(degree) # Gauss between -1 and 1
    q = (q+1)/2 # Go back to 0,1
    w = w/2

    Vq = np.zeros((len(V), len(q)))
    Vpq = np.zeros((len(V), len(q)))

    for i in range(len(V)):
        Vq[i] = V[i](q)  # evaluate at the quadrature points
        Vpq[i] = V[i].deriv()(q)

    n_dofs = (len(x)-1)*(len(V)-1)+1

    A = np.zeros((n_dofs, n_dofs))
    M = np.zeros((n_dofs, n_dofs))
    rhs = np.zeros((n_dofs,))

    # Construct the local matrix only once:
    local_A = np.einsum('jq, iq, q -> ij', Vpq/h, Vpq/h, w*h)
    local_M = np.einsum('jq, iq, q -> ij', Vq, Vq, w*h)

    for i in range(len(x)-1):
        start = i*(len(V)-1)
        end = start+len(V)

        index = range(start, end)
        A[np.ix_(index, index)] += local_A
        M[np.ix_(index, index)] += local_M

        # The transformation of the quadrature points
        xq = x[i]+h*q
        local_rhs = np.einsum('iq, q, q -> i', Vq, w*h, myf(xq))

        rhs[index] += local_rhs

    # Fix the boundary conditions
    apply_matrx_bc(A)
    apply_matrx_bc(M)
    apply_rhs_bc(rhs)
    #ck = A[0,0]
    #A[0] = 0
    #A[0,0] = ck
    #rhs[0] = 0

    #ck = A[-1,-1]
    #A[-1] = 0
    #A[-1,-1] = ck
    #rhs[-1] = 0

    return A,M,rhs
```

```python
def evaluate_l2_error(x,V,u,exact_sol):
    h = 1./(len(x)-1)

    degree = 2*len(V)-1
    q,w = leggauss(degree) # Gauss between -1 and 1
    q = (q+1)/2 # Go back to 0,1
    w = w/2

    Vq = np.zeros((len(V), len(q)))
    Vpq = np.zeros((len(V), len(q)))

    for i in range(len(V)):
        Vq[i] = V[i](q)  # evaluate at the quadrature points
        Vpq[i] = V[i].deriv()(q)

    error = 0.

    for i in range(len(x) - 1):
        start = i*(len(V) - 1)
        end = start + len(V)

        index = range(start, end)
        print index
        xq = x[i] + q*h

        fem_sol = np.einsum('j, jq ->q', u[index],Vq)
        an_sol = exact_sol[xq]

        error += np.sum((fem_sol - an_sol)**2.*w)

        print fem_sol
        print an_sol

    print np.sqrt(error)

```

```python
import sympy as sym

t = sym.var('t')
sol = t**2*(1-t)
force_a = -1*sym.diff(sol,t,2)
```

```python
n = 9.

for n in np.arange(2,4):

    n = n**2 +1

    x = np.linspace(0,1,n)

    force = sym.lambdify(t,force_a,"numpy")
    sol_n = sym.lambdify(t,sol,"numpy")

    A,M,rhs = assemble_system(x,V,force)

    ufem = np.linalg.solve(A,rhs)

    print evaluate_l2_error(x,V,ufem,sol_n)
```

```python
1.63733964
```

    1.63733964

```python
def interpolation_matrix(s, x, V):
    """
    Given the evaluation points s in the mesh x, construct the interpolation matrix
    according to the local basis V.
    """
    n_dofs = (len(x)-1)*(len(V)-1)+1

    M = np.zeros((n_dofs, len(s)))
    for i in range(len(x)-1):
        index = s>=x[i]
        index *= s<=x[i+1]

        # Transform the points to [0,1]
        xi = (s[index]-x[i])/(x[i+1]-x[i])

        for j in range(len(V)):
            k = i*(len(v)-1)+j
            M[k,index] = V[j](xi)
    return M
```

```python
s = np.linspace(0,1,1025)
us = np.sin(2*np.pi*s)
ux = np.sin(2*np.pi*x)

# Use this to plot high resolution versions
M = interpolation_matrix(s,x,V)

# Use this to compute errors at the mesh points
Mx = interpolation_matrix(x,x,V)

plt.plot(s, M.T.dot(ufem))
plt.plot(s, us)
plt.plot(x, ux,'ro')

print("Error at the s points:", (M.T.dot(ufem)-us).max())
print("Error at the x points:", (Mx.T.dot(ufem)-ux).max())
```

```python

```

```python

```

```python

```


$\newcommand{\xv}{\mathbf{x}}
 \newcommand{\tv}{\mathbf{t}}
 \newcommand{\wv}{\mathbf{w}}
 \newcommand{\Chi}{\mathcal{X}}
 \newcommand{\R}{\rm I\!R}
 \newcommand{\sign}{\text{sign}}
 \newcommand{\Tm}{\mathbf{T}}
 \newcommand{\Xm}{\mathbf{X}}
 \newcommand{\Im}{\mathbf{I}}
$
### ITCS6155

# Linear Model

**Supervised Learning**:

$$ f: \mathcal{X} \rightarrow y $$

Supervised learning can be formulated as above. When we want to predict tomorrow's temparature, for instance, we might look for the data that we can use as an input $\mathcal{X}$ such as humidity, history of temparature changes, air pressure, and vapor pressure along with the output (today's temparature) $y$. Once we recorded or found the data, we can build a table as follows.

humidity (%) | last year temparature (ºF) | yesterday's air pressure (inHG) | vapor pressure (inHG) | **Today's Temp** (ºF)
---|---|---|---|---
23 | 72 | 30.12 | 0.79 | 76
15 | 82 | 29.32 | 0.68 | 81
 | | ... | |

Here, we note that the output is *today's* not *tomorrow's* temparature.
As we discussed in the first lecture, machine learning model *learns* from data or experiences.
This learning is called as *"training"* and the data used for training is called *training samples*.
In this example, the table is training samples that we will feed in to our models.
To maintain the right relation between input data and output prediction, however, the humidity and pressure values from yesterday are paired with the output, today's temparature.

Once you have data to play with, now you can apply learning algorithms ($f$) to find parameters.
This model with the learned parameters is *hypothesis*, and will be your model for prediction.
From today's measurements, applying the hypothesis model can simply generate the prediction output.
When the training is successful, it is more likely to have a good estimation.

## Linear Model

Linear model can be defined as a Euclidean dot product between two vectors:

$$
\begin{align}
  f(\xv; \wv) &= \wv^\top \xv = \sum_0^D w_i x_i \\
     &= w_0 x_0 + w_1 x_1 + \cdots + w_D x_D
\end{align}
$$

where $\wv$ is a weight vector and $\xv$ is an input vector.
When it is a one dimensional vector, it represents a straight line, so it is called *linear*.

Assume that we have $N$ data observations, $\xv_i$ and target outputs, $t_i$, for $i = 1, \cdots, N$.
The simpliest model that we can think of is constant model, $f(\xv) = c$, where $c$ is any scalar.
In this case, we have the zero weights in the linear model.

The linearity in the parameter $\xv$ make the optimization based on deravatives to solvable analytically.
The model limits the complexity so its representation is also limited. However, it can prevent possible overfitting with the simple modeling, especially when you have sparsely sampled data.

### Dot Product

The dot product, also known as inner product or scalar product, computes the product of pair of elements in each vector and the summation of the products. It is geometrically interpreted as a cosine between two vectors as well.
Thus, it can be written as

$$
    \wv^\top \xv = \| \wv \| \| \xv \| \cos(\theta),
$$

where $\theta$ is the angle between two vectors.
Thus, when the vectors are unit vectors, the dot product is simply consine of the angle between two vectors.

<center>(from wikipedia)</center>

### Regression

The target output $t$ is s real number ($y, t \in \R$) as we discussed in last lecture. Thus, training $f(\xv; \wv)$ generates $y$ values close to the target outputs. We will discuss more about linear regression in next section.

### Classification

Classification has target outputs as discrete values. In case of binary classification, you have two values as target (ie. $t \in \{ -1, 1 \}$). As we know that $y$ can be any real value, we can cap the model to generate discrete values as below:

$$ y = \sign ( f(\xv; \wv) ).$$

### Advantages of Linear Model

- Simple
- Stable
- Avoid Overfitting
- Scalable

# Practice
**Finish this excercise and submit on Canvas.**

Q: Write a python code that create two vectors, $\wv$ and $\xv$ as follows:

$
\xv = \begin{bmatrix}
         4.0 \\
         2.3 \\
         1.2 \\
         5.8
        \end{bmatrix},
\wv = \begin{bmatrix}
         0.8 \\
         0.1 \\
         0.53 \\
         0.33
        \end{bmatrix}
$

```python
# TODO

```

Q: Write a function *linear_model(x, w)* that returns the result of dot product.

```python
def linear_model(x, w):
    # TODO: fill in here
    pass
```

Q: Pass the $\wv$ and $\xv$ and print the output of the linear model.

```python
# TODO

```

# Linear Regression

In this note, we solve regression problems using the linear model as follows.
For instance, we have example data as follows.
The goal we want to acheive in this problem is to find a best fit on all the data.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
X = np.linspace(0,10, 101)
T = 2 * X  + 4+ np.random.rand(101) * 5
```

```python
def data_scatter(k=101):
    plt.plot(T[:k], '.')
    plt.xticks(range(0, 101, 20)[:k], range(0, 11, 2)[:k])
```

```python
data_scatter()
```

When we use a linear model, there can be multiple options. One of them, or the simplest solution, can be the avergage value.

```python
mean = np.mean(T)

data_scatter()
plt.plot([0, 100],[mean, mean], 'r-')
```

When the data is linear or when we need simple solution, the linear model can suggest better solutions.
For instance, one dimensional affine model can be written as

$$
f(x; a, b) = a x + b.
$$

Unifying the weight symbol with $w$,
$$
f(x; \wv) =  w_1 x + w_0.
$$

Considering multiple inputs for $x$, we can extend the input $x$ to input vector $\xv$ with dummy input $x_0 = 1$:

$$
\begin{align}
f(\xv; \wv) &=  w_D x_D + \cdots + w_1 x_1 + w_0 \\
            &= \sum_{i=0}^{D} w_i x_i \quad\text{where } x_0 = 1\\
            &= \wv^\top \xv.
\end{align}
$$

#### Error (Cost) Function

When we define $\wv \in \R^D$, the choice of $D$ real numbers can result in the best result.
Here, the word "*best*" can be vague. Thus, we need to define what *best* means.
The sum of square error function defines as follows:

$$
E(\wv) = \sum_{i=1}^N \Big( f(\xv_i; \wv_i) - t_i \Big)^2
$$

This error function says that we want to minimize the sum of Euclidean distances between the target values and the model outputs. Here the square term leads the distances are not biased a few samples since sacrificing others with large errors will disturb the objective.

## Least Squares

The parameter that gives best fit will be

$$
\wv^* = \arg\min_\wv \sum_{i=1}^{N} \Big( f(\xv_i; \wv) - t_i \Big)^2
$$

Since the error funciton is quadratic, the problem can be analytically solved by simply setting derivative with respect to $\wv$ to zero.
For this, let us prepare data in matrix.
The target values are collected in matrix $\tv$, and the input samples are in matrix $\Xm$.

$$
\begin{align}
\tv &= [t_1, t_2, \cdots, t_N]^\top \\
\\
\wv &= [w_0, w_1, \cdots, w_D]^\top \\
\\
\Xm &= \begin{bmatrix}
    x_{10} & x_{11} & x_{12} & \dots  & x_{1D} \\
    x_{20} & x_{21} & x_{22} & \dots  & x_{2D} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    x_{N0} & x_{N1} & x_{N2} & \dots  & x_{ND}
\end{bmatrix}
\end{align}
$$
where the first column is one's, $\xv_{*0} = [1, 1, \dots, 1]^\top$.

With this matrix, $f(\xv; \wv)$ can be written in matrix form as:
$$
f(\xv; \wv) = \Xm \wv.
$$
Thus, the error function can be
$$
\begin{align}
E(\wv) &= \sum_{i=1}^N \Big(f(\xv_i; \wv_i) - t_i \Big)^2 \\
            \\
            &= (\Xm \wv - \tv)^\top (\Xm \wv - \tv) \\
            \\
            &= \wv^\top \Xm^\top \Xm \wv - 2 \tv^\top \Xm \wv + \tv^\top \tv
\end{align}
$$
because $\tv^\top \Xm \wv$ is a scalar, so symmetric.

Now, let us do the derivative.
$$
\begin{align}
\frac{\partial E(\wv)}{\partial \wv} &= \frac{\partial (\Xm \wv - \tv)^\top (\Xm \wv - \tv)}{\partial \wv} \\
        \\
        &= \frac{\partial (\wv^\top \Xm^\top \Xm \wv - 2 \tv^\top \Xm \wv + \tv^\top \tv )}{\partial \wv} \\
        \\
        &= \frac{\partial (\wv^\top \Xm^\top \Xm \wv)}{\partial \wv} - 2 \Xm^\top \tv \\
        \\
        &= \Xm^\top \Xm \wv + (\Xm^\top \Xm)^\top \wv - 2 \Xm^\top \tv  \\
        \\
        &= 2 \Xm^\top \Xm \wv - 2 \Xm^\top \tv
\end{align}
$$

Setting this to zero,
$$
\begin{align}
 2 \Xm^\top \Xm \wv - 2 \Xm^\top \tv  &= 0\\
\\
\Xm^\top \Xm \wv &= \Xm^\top \tv\\
\\
\wv &= \big(\Xm^\top \Xm\big)^{-1} \Xm^\top \tv
\end{align}
$$

# Practice

Implement the least squares model and apply to the simulated data X and T.
Consider using **np.linalg.inv**, **np.linalg.solve**, **np.linalg.lstsq**.
After getting the parameter w, plot the approximation line.

```python
import numpy as np

N = X.shape[0]
# TODO: code for finding w
# First create X1 by adding 1's column to X
X1 =

# Next, using inverse, solve, lstsq function to get w*
w =
```

    array([ 6.63917738,  1.99506176])

```python
# TODO: Write codes to generate the plot as below.
# call data_scatter function to present the training data
# then, write a plotting code showing the linear line

```

## Least Mean Squares (LMS)

Previously we observed that the least squares use all the available data for training or finding the best fit.
This can be often computationally costly, especially with large data sets. When data is sufficiently large, we can consider *sequential* or *online* learning.
During the online learning process, we introduce a data point one by one, and update the parameters. Using the updated parameters, it makes a new estimation and repeat these steps.

For this, we start with an initial guess $\wv$ and changes it as it reads more data until it converges.
When $k$ represents the steps for the repetition,

$$
\wv^{(k+1)} = \wv^{(k)} - \alpha \nabla E_k
$$

where $E_k$ is the error for the $k$'th sample and $\alpha$ is a learning rate.
This is called *stochastic gradient descent* or *sequential gradient descent*.

For the $k$'th sample $\xv_k$, the gradient for the sum-of-squares error is

$$
\begin{align}
\nabla E_k = \frac{\partial E}{\partial \wv^{(k)}} &= \frac{\partial }{\partial \wv^{(k)}}\Big( f(\xv_k; \wv^{(k)}) - t_k \Big)^2 \\
        &= 2 \Big( f(\xv_k; \wv^{(k)}) - t_k \Big) \frac{\partial }{\partial \wv^{(k)}} \Big( f(\xv_k; \wv^{(k)}) - t_k \Big) \\
        &= 2 \Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \frac{\partial }{\partial \wv^{(k)}} \Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \\
        &= 2\Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \xv_k.
\end{align}
$$
This gives the following update rule for each sample:

$$
\wv^{(k+1)} = \wv^{(k)} - \alpha \Big( {\wv^{(k)}}^\top \xv_k - t_k \Big) \xv_k.
$$

# Pratice

Implement the LMS for the simulated sample X, and show the plot

```python
import IPython.display as ipd  # for display and clear_output

# initial weights with random values
w = np.random.rand(X1.shape[1])

# learning rate
alpha = 0.01

fig = plt.figure()

# sequential learning
for k in range(N):

    # TODO: online update of weights

    plt.clf()
    data_scatter(k+1)
    # TODO: Plot the current model's estimation in a line

    ipd.clear_output(wait=True)
    ipd.display(fig)
ipd.clear_output(wait=True)
```


# Stochastic Processes:
Data Analysis and Computer Simulation

# Distribution function and random number

# 3. The central limit theorem

# 3.1. Binomial distribution $\rightarrow$ Gauss distribution

## From the previous lesson

- The binomial distribution becomes equivalent to the Gaussian distribution in the limit $n, M\gg 1$, as shown in the 1st plot of this week.

\begin{equation}
P(n)=\frac{M!}{n!(M-n)!}{{p}^{n}}{{(1-p)}^{M-n}}\tag{C6}
\end{equation}

\begin{equation}
\hspace{20mm}
\xrightarrow[n\to cont.]{n,M\gg 1}\
% P(n)=
\frac{1}{\sqrt{2\pi {{\sigma }^{2}}}}\exp \left[ -\frac{{{(n-{{\mu }_{1}})}^{2}}}{2{{\sigma }^{2}}} \right]\tag{C1}
\end{equation}

\begin{equation}
\hspace{20mm}
\mu_1=Mp,\ \ \ \
\sigma^2=Mp(1-p) \tag{C7, C8}
\end{equation}

## Numerical experiment 1

- While the proof for the equivalence has been given in the supplemental note, let us examine this by performing numerical experiments for various values of $M=1,2,4,10,100$ and $1000$.

### Include libraries

```python
% matplotlib inline
import numpy as np # import numpy library as np
import math # use mathematical functions defined by the C standard
import matplotlib.pyplot as plt # import pyplot library as plt
plt.style.use('ggplot') # use "ggplot" style for graphs
```

```python
p = 0.5    # set p, propability to obtain "head" from a coin toss
M = 50      # set M, number of tosses in one experiment
N = 100000 # number of experiments
ave = M*p
std = np.sqrt(M*p*(1-p))
print('p =',p,'M =',M)
np.random.seed(0) # initialize the random number generator with seed=0
X = np.random.binomial(M,p,N) # generate the number of head come up N times store it as X
nmin=np.int(ave-std*5)
nmax=np.int(ave+std*5)
nbin=nmax-nmin+1
plt.hist(X,range=[nmin,nmax],bins=nbin,normed=True) # plot normalized histogram of R
x = np.arange(nmin,nmax,0.01/std) # create array of x from nmin to nmax with increment 0.01/std
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # calculate the theoretical distribution
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$n$',fontsize=16) # set x-label
plt.ylabel(r'$P(n)$',fontsize=16) # set y-label
plt.legend([r'Gauss',r'histogram'], fontsize=16) # set legends
plt.show() # display plots
```

## What we can learn from the experiment

- Stochastic variable "$s$" is a result of single binary choice,

\begin{equation}
s=0\ {\rm or}\ 1
%\tag{D1}
\end{equation}

and Stochastic variable "$n^{M}$" is a sum of $M$ independent binary choices $s$, with the index $j$ representing the $j$-th choice.

\begin{equation}
%\color{red}
{
n^{M}=\sum^{M}_{j=1}s_j}
%\tag{D4}
\end{equation}

### For $M=1$

\begin{equation}
n^{M=1}=s_1=s=0\ {\rm or}\ 1
\tag{D1}
\end{equation}
- Distribution function $\rightarrow$ Binary choice,
$P^{M=1}(0)=1-p, \ P^{M=1}(1)=p, $ with

\begin{equation}
\hspace{15mm}\mu^{M=1}_1=p, \hspace{23mm} \sigma_{M=1}^2=p(1-p) \tag{D2, D3}
\end{equation}

### For $M\gg 1$

\begin{equation}
\color{red}{
n^{M}=\sum^{M}_{j=1}s_j=\sum^{M}_{j=1}n^{M=1}_j}  \tag{D4}
\end{equation}
- Distribution function $\rightarrow$ Gaussian with

\begin{equation}
%\hspace{20mm}
\color{red}{
\mu^{M\gg1}_1=M\mu^{M=1}_1,\hspace{20mm} \sigma_{M\gg1}^2=M\sigma_{M=1}^2} \tag{D5, D6}
\end{equation}

# 3.2. The central limiting theorem (CLT)

## Generalization of Eqs. (D4-D6) for $M\gg 1$

### CLT for sum of stochastic variables

- Stochastic variable "$n^M$" as a SUM of any $M$ independent stochastic variables $n^{M=1}$ with $\mu_1^{M=1}$ and $\sigma_{M=1}^2$,

\begin{equation}
n^{M}=\sum^{M}_{j=1}n^{M=1}_j \tag{D7}
\end{equation}
- Distribution function $\rightarrow$ Gauss with

\begin{equation}
%\hspace{20mm}
{
\mu^{M\gg1}_1=M\mu^{M=1}_1,\hspace{20mm} \sigma_{M\gg1}^2=M\sigma_{M=1}^2} \tag{D8, D9}
\end{equation}

### CLT for average of stochastic variables

- Stochastic variable "$n^M$" as an AVERAGE of any $M$ independent stochastic variables with $\mu^{M=1}$ and $\sigma_{M=1}^2$,

\begin{equation}
n^{M}=\frac{1}{M}\sum^{M}_{j=1}n^{M=1}_j \tag{D10}
\end{equation}
- Distribution function $\rightarrow$ Gauss with

\begin{equation}
%\hspace{20mm}
{
\mu^{M\gg1}_1=\mu^{M=1}_1,\hspace{20mm} \sigma_{M\gg1}^2=\frac{\sigma_{M=1}^2}{M}} \tag{D11, D12}
\end{equation}

- Eqs. (D7-D12) is called "the central limiting theorem".

# 3.3. Uniform distribution $\rightarrow$ Gauss distribution

## From CLT

### For $M=1$

- Stochastic variable "$x$" is uniformly distributed between 0 and 1,

\begin{equation}
x^{M=1}\in[0:1]
\tag{D13}
\end{equation}
- Distribution function:
$P^{M=1}(x)=1 \ ({\rm for\ } 0\le x < 1),\ P^{M=1}(x)=0\ {\rm (otherwise)}$

\begin{equation}
\hspace{15mm}\mu^{M=1}_1=\frac{1}{2}, \hspace{23mm} \sigma_{M=1}^2=\frac{1}{12} \tag{D14, D15}
\end{equation}

### For $M\gg 1$

- Stochastic variable "$x$" is a sum of $M$ independent uniform random numbers

\begin{equation}
{
x^{M}=\sum^{M}_{j=1}x^{M=1}_j} \tag{D16}
\end{equation}
- Distribution function $\rightarrow$ Gauss with

\begin{equation}
%\hspace{20mm}
{
\mu^{M\gg1}_1=M\mu^{M=1}_1=\frac{M}{2},\hspace{20mm} \sigma_{M\gg1}^2=M\sigma_{M=1}^2}=\frac{M}{12} \tag{D17, D18}
\end{equation}

## Numerical experiment 2

```python
M = 10    # set M, the number of random variables to add
N = 100000 # number samples to draw, for each of the random variables
ave = M/2
std = np.sqrt(M/12)
print('M =',M)
np.random.seed(0)        # initialize the random number generator with seed=0
X = np.zeros(N)
for i in range(N):
    X[i] += np.sum(np.random.rand(M)) # draw a random numbers for each of the x_i and add to get new value for x = \sum x_i
nmin=np.int(ave-std*5)
nmax=np.int(ave+std*5)
plt.hist(X,range=[nmin,nmax],bins=50,normed=True) # plot normalized histogram of R
x = np.arange(nmin,nmax,0.01/std) # create array of x from nmin to nmax with increment 0.01/std
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # calculate the theoretical distribution
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$n$',fontsize=16) # set x-label
plt.ylabel(r'$P(n)$',fontsize=16) # set y-label
plt.legend([r'Gauss',r'histogram'], fontsize=16) # set legends
plt.show() # display plots
```


# In this demo

This demo presents the design and functionality of `Signal`s, which are the core objects for representing coefficients in models. `Signal`s are structured to represent the mathematical formula:
\begin{equation}
    s(t) = Re[f(t)e^{i(2 \pi \nu t + \phi)}],
\end{equation}
where
- $f(t)$ is a complex-valued envelope,
- $\nu$ is the carrier frequency, and
- $\phi$ is the phase.

Using this formula as a basis for the representation of `Signal`s is motivated by:
- This is a very common representation when modelling control sequences in quantum systems; in particular, for representing signals whose Fourier spectrum is localized about a particular frequency.
- It is general; even when the previous point is not immediately applicable, the full generality of the envelope $f(t)$ allows for specification of arbitrary real-valued functions.

Taking the above as a given, the design for `Signal`s is further motivated to facilitate different modes of operation:
- Completely general envelopes: The user specifies $f(t)$ as python-callable function.
    - This allows for arbitrary signal definitions, including continuous functions.
- Piecewise constant, or *discrete* envelopes, in which $f(t)$ is a piecewise constant function of time.
    - This allows for sample-based construction and computation on `Signals`.
    - Signal processing and evaluation can be performed purely as collective array-based operations, which is more efficient than the function-based products required in the fully general case.

```python
import numpy as np
from qiskit_ode.signals import Signal, DiscreteSignal, SignalList, DiscreteSignalSum
from matplotlib import pyplot as plt

%matplotlib inline
```

# 1. `Signal` and its subclasses

## 1.1 `Signal`

The most general signal object takes the envelope as a python callable function.

```python
sig = Signal(envelope=lambda t: t**2, carrier_freq=2.)
```

We can evaluate the signal by directly calling the object, or we can evaluate its complex envelope or full complex value.

```python
print(sig(1.1))
print(sig.envelope(1.1))
print(sig.complex_value(1.1))
```

    0.373910563193685
    1.2100000000000002
    (0.373910563193685+1.1507783847171364j)

The evaluation functions are all setup for vectorized evaluation, and we assume that the user-specified envelope is vectorized. If not, it can be vectorized using `numpy.vectorize`.

```python
t_vals = np.array([0.1, 1.1])
print(sig(t_vals))
print(sig.envelope(t_vals))
print(sig.complex_value(t_vals))
```

    [0.00309017 0.37391056]
    [0.01 1.21]
    [0.00309017+0.00951057j 0.37391056+1.15077838j]

We can also plot the evaluation functions.

```python
# plot the signal
sig.draw(0, 5, 100)
```

```python
# plot the complex value
sig.draw(0, 5, 100, function='complex_value')
```

```python
# plot just the envelope
sig.draw(0, 5, 100, function='envelope')
```

## 1.2 Constant `Signal`

This is not a subclass, but an alternate mode of behaviour for `Signal`. Generally a user shouldn't need to do this, but internally, constant values are converted to a `Signal` with constant envelope and zero carrier frequency/phase.

This can be instantiated by passing a numeric value to the constructor of `Signal`.

```python
constant = Signal(2.)
constant.draw(0, 5, 100)
```

```python
constant.is_constant
```

    True

## 1.3 `DiscreteSignal(Signal)`

A `DiscreteSignal` represents a `Signal` whose envelope is piecewise constant, and is specified as an array of samples.

```python
discrete_signal = DiscreteSignal(dt=0.5, samples=[1., 2., 3.])
discrete_signal.draw(0, 1.5, 100)
```

A `DiscreteSignal` can still have an analog carrier.

```python
discrete_signal = DiscreteSignal(dt=0.5, samples=[1., 2., 3.], carrier_freq=1.)
discrete_signal.draw(0, 1.5, 100)
```

# 2. Algebraic operations on `Signal`s

It is natural to add and multiply signals, as these mathematical operations correspond to the behaviour of physical circuit elements. To facilitate this, we introduce the `SignalSum` object, which represents a sum of `Signal`s. The purpose of this class is to allow higher level algebraic operations on `Signal`s while preserving frequency information.

Algebraic operations have been implemented using python dunder methods, so that a user can add and multiply signals using standard python syntax.

## 2.1 Adding general `Signal`s

```python
sig1 = Signal(lambda t: t, carrier_freq=10.)
sig2 = Signal(lambda t: 10 * t, carrier_freq=20.)

# add using standard python syntax
sig_sum = sig1 + sig2
```

`sig_sum` is now a `SignalSum` object:

```python
print(type(sig_sum))
str(sig_sum)
```

    <class 'qiskit_ode.signals.signals.SignalSum'>

    'Signal(carrier_freq=10.0, phase=0.0) + Signal(carrier_freq=20.0, phase=0.0)'

This is a container class storing the original signals in the sum in the `components` attribute:

```python
sig_sum.components
```

    [<qiskit_ode.signals.signals.Signal at 0x7fb8d05b0370>,
     <qiskit_ode.signals.signals.Signal at 0x7fb8d05b0430>]

We can interact with this object in the same way as a regular `Signal`, e.g. we can draw it:

```python
sig_sum.draw(0, 1, 100)
```

We can produce the same plot by evaluating the individual signals in the sum separately:

```python
t_vals = np.linspace(0, 1, 100)
plt.plot(t_vals, sig1(t_vals) + sig2(t_vals))
```

## 2.2 Multiplying signals

Multiplication of signals is implemented via the formula:

\begin{equation}
    Re[f(t)e^{i(2 \pi \nu t + \phi)}] \times Re[g(t)e^{i(2 \pi \omega t + \psi)}]
         = Re[\frac{1}{2} f(t)g(t)e^{i(2\pi (\omega + \nu)t + (\phi + \psi))} ]
          + Re[\frac{1}{2} f(t)\overline{g(t)}e^{i(2\pi (\omega - \nu)t + (\phi - \psi))} ].
\end{equation}

When two signals are multiplied, a `SignalSum` representing the right hand side of the above equation is constructed.

```python
sig_prod = sig1 * sig2
str(sig_prod)
```

    'Signal(carrier_freq=30.0, phase=0.0) + Signal(carrier_freq=-10.0, phase=0.0)'

We see that the above product consists of a sum of signals centred at frequencies `4.0` and `0.0`.

```python
sig_prod(2.) - sig1(2.) * sig2(2.)
```

    0.0

## 2.3 `DiscreteSignalSum` and operations on `DiscreteSignal`s

The previous sections show how addition/multiplication work for general `Signal`s. In the general case, as the envelope is a user-specified function, we can only multiply envelopes as functions. For `DiscreteSignal`s with compatible sample strcuture, we can multiply envelopes as arrays.

```python
discrete_sig1 = DiscreteSignal(dt=1., samples=[1., 2., 3.], carrier_freq=2.)
discrete_sig2 = DiscreteSignal(dt=1., samples=[4j, 5j, 6j], carrier_freq=2.)
discrete_prod = discrete_sig1 * discrete_sig2
```

```python
print(type(discrete_prod))
str(discrete_prod)
```

    <class 'qiskit_ode.signals.signals.DiscreteSignalSum'>

    'DiscreteSignal(dt=1.0, carrier_freq=4.0, phase=0.0) + DiscreteSignal(dt=1.0, carrier_freq=0.0, phase=0.0)'

A `DiscreteSignalSum` stores the samples of its constituent components in a single array.

```python
discrete_prod.samples
```

    Array([[0.+2.j, 0.-2.j],
           [0.+5.j, 0.-5.j],
           [0.+9.j, 0.-9.j]], backend='numpy')

## 2.4 Frequency preservation

One of the benefits of `SignalSum` is that it "preserves frequency information", in the sense that, in applications where each `Signal` has a well-defined central frequency, the algebraic operations will result in `SignalSum`s whose components have well-defined central frequencies. This is useful for operations that require analyzing the frequency components of signals, e.g. the rotating wave approximation.

We can visualize this by examining the FFT of the product of signals above. The product of signals results in a component whose central frequency is at the sum of the frequencies in the product, as well as a component at the difference:

```python
N = 1000
T = 10.
t_vals = np.linspace(0., T, N)
fft_freqs = np.fft.fftfreq(N, T / N)
sig_prod_fft = np.fft.fft(sig_prod(t_vals))
plt.plot(fft_freqs, np.real(sig_prod_fft), fft_freqs, np.imag(sig_prod_fft))
```

We see components at `30.` and `10.`, as expected. Now, we can look at the FFT of each component of the product:

```python
N = 1000
T = 10.
t_vals = np.linspace(0., T, N)
fft_freqs = np.fft.fftfreq(N, T / N)
sig_prod0_fft = np.fft.fft(sig_prod[0](t_vals))
plt.plot(fft_freqs, np.real(sig_prod0_fft), fft_freqs, np.imag(sig_prod0_fft))
```

We see the first component in the sum is the term whose frequency is at the sum of the product term frequencies.

```python
N = 1000
T = 10.
t_vals = np.linspace(0., T, N)
fft_freqs = np.fft.fftfreq(N, T / N)
sig_prod1_fft = np.fft.fft(sig_prod[1](t_vals))
plt.plot(fft_freqs, np.real(sig_prod1_fft), fft_freqs, np.imag(sig_prod1_fft))
```

The second component in the sum is at the difference.

# 3. IQ mixer via algebraic operations

```python
# define some envelope function
def env(t):
    return t * (1 - t)

w_d = 2.
w_lo = 1.9
w_if = 0.1

I = Signal(env, carrier_freq=w_if, name='I') # envelope times cosine at w_if
Q = Signal(env, carrier_freq=w_if, phase=-np.pi/2, name='Q') # envelope times sine at w_if

slo_I = Signal(1., carrier_freq=w_lo, name='slo_I') # local oscillator for I
slo_Q = Signal(1., carrier_freq=w_lo, phase=-np.pi/2, name='slo_Q') # local oscillator for Q
```

The IQ output is a sum of two products. Hence, it produces a `SignalSum` with 4 outputs.

```python
IQ_output = (I * slo_I) - (Q * slo_Q)
str(IQ_output)
```

    'Signal(carrier_freq=2.0, phase=0.0) + Signal(carrier_freq=-1.7999999999999998, phase=0.0) + Signal(carrier_freq=2.0, phase=-3.141592653589793) + Signal(carrier_freq=-1.7999999999999998, phase=0.0)'

```python
IQ_output.draw(0, 1, 100, title='IQ output')
```

The $0^{th}$ and $2^{nd}$ terms in the sum are at frequency `2.0`, which we can plot:

```python
IQ_output[[0, 2]].draw(0, 1, 100, title='freq==2.0 terms')
```

The $1^{st}$ and $3^{rd}$ terms are at frequency `1.8`:

```python
IQ_output[[1, 3]].draw(0, 1, 100, title='freq==1.8 terms')
```

This demonstrates the cancellation at the lower sideband, with the remaining signal at the upper sideband. When modelling imperfect operation of an IQ mixer, this cancellation may not occur perfectly.

# 4. `SignalList`

A model of a Hamiltonian contains a list of `Signal`s representing coefficients of operators. Each coefficient may be any subclass of `Signal` (including `SignalSum` and `DiscreteSignalSum`). The `SignalList` object is similar to `SignalSum`, in that it stores a list of `Signal`s, but the evaluation functions return an array of each component evaluated separately.

```python
sig_list = SignalList([2. + Signal(lambda t: t, carrier_freq=1.),
                       1.,
                       DiscreteSignal(dt=1., samples=[1., 2., 3.])])
```

```python
sig_list(2.)
```

    array([4., 1., 3.])

`SignalList` also contains a helper `drift` property for evaluating the constant part:

```python
sig_list.drift
```

    Array([2., 1., 0.], backend='numpy')

# 5. Transfer functions

```python
from qiskit_ode.signals.transfer_functions import Convolution
```

```python
def gaus(t):
    sigma = 10
    dt = 0.1
    return 2.*dt/np.sqrt(2.*np.pi*sigma**2)*np.exp(-t**2/(2*sigma**2))
```

```python
ts = np.linspace(0, 100, 1000)
np.sum([gaus(_) for _ in ts])
```

    1.0029894228040142

```python
plt.plot(ts, gaus(ts))
```

```python
convolution = Convolution(gaus)
```

```python

samples = [0. if t<20. or t > 80. else 1 for t in ts]
piecewise_const = DiscreteSignal(dt=ts[1]-ts[0], samples=samples, carrier_freq=0.0, start_time=0)
```

```python
piecewise_const.draw(0, 100, 200)
piecewise_const.carrier_freq
```

```python
convolved = convolution(piecewise_const)
convolved.draw(0, 150, 200)
convolved.carrier_freq
```

```python
convolved.carrier_freq = 0.3
convolved.draw(0, 200, 500)
convolved.carrier_freq
```

# 6. Fancy indexing

`Signal` subclasses representing collections of signals can be subscripted in the same manner as a 1d numpy array.

```python
sig_sum = 1. + Signal(lambda t: t) + DiscreteSignal(dt=0.5, samples=[1, 2, 3, 4, 5])
str(sig_sum)
```

    'Signal(carrier_freq=0.0, phase=0.0) + Constant(1.0) + DiscreteSignal(dt=0.5, carrier_freq=0.0, phase=0.0)'

We can access the first two components of the sum by subscripting with the list `[0, 1]`.

```python
str(sig_sum[[0, 1]])
```

    'Signal(carrier_freq=0.0, phase=0.0) + Constant(1.0)'

This works for both `DiscreteSignalSum` and `SignalList`. Subscripting these results in an object of the appropriate type.

# 7. Sampling

We can generate `DiscreteSignal`s and `DiscreteSignalSum`s by sampling, respectively, general `Signal`s and `SignalSum`s. When performing this discretization, we can choose to sample the carrier as well, or keep it analog.

```python
sig = Signal(lambda t: t, carrier_freq=2.)

disc1 = DiscreteSignal.from_Signal(sig, dt=0.1, start_time=0., n_samples=10)
disc2 = DiscreteSignal.from_Signal(sig, dt=0.1, start_time=0., n_samples=10, sample_carrier=True)
```

```python
# plot the signal and the discretized version when keeping the carrier analog
sig.draw(0, 1, 100)
disc1.draw(0, 1, 100)
```

```python
# plot the signal and the discretized version when also sampling the carrier
sig.draw(0, 1, 100)
disc2.draw(0, 1, 100)
```

The same can be done with `DiscreteSignalSum`.

```python
sig1 = Signal(lambda t: t, carrier_freq=2.)
sig2 = Signal(lambda t: 1j * t**2, carrier_freq=3.)
sig_sum = sig1 + sig2

disc1 = DiscreteSignalSum.from_SignalSum(sig_sum, dt=0.1, start_time=0., n_samples=10)
disc2 = DiscreteSignalSum.from_SignalSum(sig_sum, dt=0.1, start_time=0., n_samples=10, sample_carrier=True)
```

```python
sig_sum.draw(0, 1, 100)
disc1.draw(0, 1, 100)
```

```python
sig_sum.draw(0, 1, 100)
disc2.draw(0, 1, 100)
```