```python
import loader
from sympy import *
init_printing()
from root.solver import *
```

#### Solve the nonhomogeneous differential equation $y'' + y = \sec t + \tan t + t$ using variation of parameters

```python
yc, p = nth_order_const_coeff(1, 0, 1)
p.display()
yp, p = variation_of_parameters(yc, sec(t) + tan(t) + t)
p.display()
```

$\displaystyle \text{Characteristic equation: }$

$\displaystyle r^{2} + 1 = 0$

$\displaystyle \text{Roots: }\left\{ \begin{array}{ll}r_{1} = i\\r_{2} = - i\\\end{array} \right.$

$\displaystyle \text{General Solution: }$

$\displaystyle y = C_{1} \sin{\left (t \right )} + C_{2} \cos{\left (t \right )}$

$\displaystyle \text{Compute the Wronskian determinant}$

$\displaystyle W = \left|{\left[\begin{matrix}\sin{\left (t \right )} & \cos{\left (t \right )}\\\cos{\left (t \right )} & - \sin{\left (t \right )}\end{matrix}\right]}\right| = -1$

$\displaystyle \text{Compute }W_i$

$\displaystyle \begin{align*}&W_{1} = \left|{\left[\begin{matrix}0 & \cos{\left (t \right )}\\1 & - \sin{\left (t \right )}\end{matrix}\right]}\right| = - \cos{\left (t \right )}\\&W_{2} = \left|{\left[\begin{matrix}\sin{\left (t \right )} & 0\\\cos{\left (t \right )} & 1\end{matrix}\right]}\right| = \sin{\left (t \right )}\\\end{align*}$

$\displaystyle \text{Calculate and simplify }\frac{g(t)}{W(t)}$

$\displaystyle \frac{1}{-1} \left(t + \tan{\left (t \right )} + \sec{\left (t \right )}\right) = - t - \tan{\left (t \right )} - \frac{1}{\cos{\left (t \right )}}$

$\displaystyle \text{Compute }\mu_i = \int \frac{g(t)W_i(t)}{W(t)} dt$

$\displaystyle \begin{align*}&\mu_{1} = \int \left(t \cos{\left (t \right )} + \cos{\left (t \right )} \tan{\left (t \right )} + 1\right)\, dt = t \sin{\left (t \right )} + t\\&\mu_{2} = \int \left(- t \sin{\left (t \right )} - \sin{\left (t \right )} \tan{\left (t \right )} - \frac{\sin{\left (t \right )}}{\cos{\left (t \right )}}\right)\, dt = t \cos{\left (t \right )} + \frac{1}{2} \ln{\left (\sin{\left (t \right )} - 1 \right )} - \frac{1}{2} \ln{\left (\sin{\left (t \right )} + 1 \right )} + \ln{\left (\cos{\left (t \right )} \right )}\\\end{align*}$

$\displaystyle \text{Compute the sum }\sum_{i=1}^{k} y_i \int \frac{g(t)W_i(t)}{W(t)} dt$

$\displaystyle \begin{align*}&y_{p} = \left(t \sin{\left (t \right )} + t\right) \sin{\left (t \right )} + \left(t \cos{\left (t \right )} + \frac{1}{2} \ln{\left (\sin{\left (t \right )} - 1 \right )} - \frac{1}{2} \ln{\left (\sin{\left (t \right )} + 1 \right )} + \ln{\left (\cos{\left (t \right )} \right )}\right) \cos{\left (t \right )}\\&y_{p} = t \sin{\left (t \right )} + t + \frac{\cos{\left (t \right )}}{2} \ln{\left (\sin{\left (t \right )} - 1 \right )} - \frac{\cos{\left (t \right )}}{2} \ln{\left (\sin{\left (t \right )} + 1 \right )} + \ln{\left (\cos{\left (t \right )} \right )} \cos{\left (t \right )}\\\end{align*}$

$\displaystyle \text{Complementray + particular = general}$

$\displaystyle y = t + \left(C_{1} + t\right) \sin{\left (t \right )} + \left(C_{2} + \frac{1}{2} \ln{\left (\sin{\left (t \right )} - 1 \right )} - \frac{1}{2} \ln{\left (\sin{\left (t \right )} + 1 \right )} + \ln{\left (\cos{\left (t \right )} \right )}\right) \cos{\left (t \right )}$

#### Solve the nonhomogeneous differential equation $y'' - 12 y' + 36y = t^{-2} e^{6t} + 14t^2$ using variation of parameters

```python
yc, p = nth_order_const_coeff(1, -12, 36)
p.display()
yp, p = variation_of_parameters(yc, t ** (-2) * exp(6 * t) + 14 * t ** 2)
p.display()
```

$\displaystyle \text{Characteristic equation: }$

$\displaystyle r^{2} - 12 r + 36 = 0$

$\displaystyle \text{Roots: }\left\{ \begin{array}{ll}r_{1,2} = 6\\\end{array} \right.$

$\displaystyle \text{General Solution: }$

$\displaystyle y = \left(C_{1} + C_{2} t\right) e^{6 t}$

$\displaystyle \text{Compute the Wronskian determinant}$

$\displaystyle W = \left|{\left[\begin{matrix}e^{6 t} & t e^{6 t}\\6 e^{6 t} & 6 t e^{6 t} + e^{6 t}\end{matrix}\right]}\right| = e^{12 t}$

$\displaystyle \text{Compute }W_i$

$\displaystyle \begin{align*}&W_{1} = \left|{\left[\begin{matrix}0 & t e^{6 t}\\1 & 6 t e^{6 t} + e^{6 t}\end{matrix}\right]}\right| = - t e^{6 t}\\&W_{2} = \left|{\left[\begin{matrix}e^{6 t} & 0\\6 e^{6 t} & 1\end{matrix}\right]}\right| = e^{6 t}\\\end{align*}$

$\displaystyle \text{Calculate and simplify }\frac{g(t)}{W(t)}$

$\displaystyle \frac{14 t^{2} + \frac{e^{6 t}}{t^{2}}}{e^{12 t}} = \frac{e^{- 12 t}}{t^{2}} \left(14 t^{4} + e^{6 t}\right)$

$\displaystyle \text{Compute }\mu_i = \int \frac{g(t)W_i(t)}{W(t)} dt$

$\displaystyle \begin{align*}&\mu_{1} = \int \left(- 14 t^{3} e^{- 6 t} - \frac{1}{t}\right)\, dt = \frac{e^{- 6 t}}{108} \left(252 t^{3} + 126 t^{2} + 42 t + 7\right) - \ln{\left (t \right )}\\&\mu_{2} = \int \left(14 t^{2} e^{- 6 t} + \frac{1}{t^{2}}\right)\, dt = \frac{e^{- 6 t}}{54} \left(- 126 t^{2} - 42 t - 7\right) - \frac{1}{t}\\\end{align*}$

$\displaystyle \text{Compute the sum }\sum_{i=1}^{k} y_i \int \frac{g(t)W_i(t)}{W(t)} dt$

$\displaystyle \begin{align*}&y_{p} = t \left(\frac{e^{- 6 t}}{54} \left(- 126 t^{2} - 42 t - 7\right) - \frac{1}{t}\right) e^{6 t} + \left(\frac{e^{- 6 t}}{108} \left(252 t^{3} + 126 t^{2} + 42 t + 7\right) - \ln{\left (t \right )}\right) e^{6 t}\\&y_{p} = \frac{7 t^{2}}{18} + \frac{7 t}{27} - e^{6 t} \ln{\left (t \right )} - e^{6 t} + \frac{7}{108}\\\end{align*}$

$\displaystyle \text{Complementray + particular = general}$

$\displaystyle y = C_{2} t e^{6 t} + \frac{7 t^{2}}{18} + \frac{7 t}{27} + \left(C_{1} - \ln{\left (t \right )}\right) e^{6 t} + \frac{7}{108}$

```python

```


## Brachistochrone Curve

For reference, c.f. [here](https://baike.baidu.com/item/%E6%9C%80%E9%80%9F%E9%99%8D%E7%BA%BF%E9%97%AE%E9%A2%98).

Another code for this problem (also in TensorFlow) can be found [herein](https://github.com/yimuw/Let-Tensor-Flow/blob/master/control/brachistochrone_curve.py).

For a trajectory $y(x)$ for $\forall x \in [0, 1]$, the action (loss), as the total time spent, is given by
\begin{equation}
  \mathcal{L}[y] = \int_{x_0}^{x_1} d x
                   \frac{ \sqrt{ 1 + {y^{\prime}}^2 (x) } } { \sqrt{ 2 g \left( y_0 - y(x) \right) } },
\end{equation}
with boundary condition
$$ y(x_0) = y_0 \;, y(x_1) = y_1 , $$
where $g$ is the gravitational constant and $y_0 > y_1$.

Employ the [Monte-Carlo approximation](https://en.wikipedia.org/wiki/Monte_Carlo_integration#Overview). Explicitly, let $\{ x_i: i = 1, \ldots, N \} \sim \text{Uniform}(x_0, x_1)$, the action with boundary condition and penalty becomes
\begin{equation}
  \mathcal{L}[y] = \frac{x_1 - x_0}{N} \sum_{i=1}^{N}
                   \frac{ \sqrt{ 1 + {y^{\prime}}^2 (x_i) } }{ \sqrt{ 2 g \left( y_0 - y(x_i) \right) } } .
\end{equation}

The $y(x)$ with the tricky boundary setting is:
\begin{equation}
  y(x) = (x - x_0) (x - x_1) N(x)
     + \frac{ (y_1 - y_0) ( x - x_0) }{ x_1 - x_0 }
      + y_0 ,
\end{equation}
where $N(x)$ is the neural network without any boundary restriction.

```python
%matplotlib notebook

import numpy as np
import tensorflow as tf
from matplotlib import pyplot as plt
from datetime import datetime
from collections import namedtuple
import random as rn
import os

# Global parameters
_SEED = 42
_EPSILON = 1e-8

# For reproducibility
os.environ['PYTHONHASHSEED'] = '0'
np.random.seed(_SEED)
rn.seed(_SEED)
tf.set_random_seed(_SEED)
```

    D:\ProgramData\Anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

```python
def make_neural_network(x, hidden_layers, output_shape, name=None):
  """Implement the generic neural network with the dense linear output
  layer.
  Args:
    x: Tensor-like, as the input of the neural network. It's shape is
        of `[batch_size] + x_shape`.
    hidden_layers: List of objects of the classes in `tf.layers`.
    output_shape: List of integers.
  Returns:
    The output tensor of the neural network.
  """
  with tf.name_scope(name, 'NeuralNetwork', [x]):
    x = tf.convert_to_tensor(x)

    # Hidden layers
    hidden = x  # initialize.
    for layer in hidden_layers:
      hidden = layer(hidden)

    # Output layer
    flatten_hidden = tf.layers.flatten(hidden)
    output_size = sum(flatten(output_shape))
    output = tf.layers.dense(flatten_hidden, output_size)
    output = tf.reshape(output, [-1]+output_shape)

  return output

def flatten(nested_list):
    """Helper. (Recursively) flatten an arbitrarily nested list."""
    if nested_list == []:
        return nested_list
    if isinstance(nested_list[0], list):
        return flatten(nested_list[0]) + flatten(nested_list[1:])
    return nested_list[:1] + flatten(nested_list[1:])

def make_action(y_bulk, grad_y_bulk, y_start, x_volumn,
                g=9.8, epsilon=_EPSILON, name=None):
    """Implements the action in Monte-Carlo integral.

    The action is a Monte-Carlo integral IN THE BULK.

    Args:
      y: Tensor-like, with shape `[batch_size] + y_shape`.
      grad_y: Tensor-like, with shape `[batch_size] + y_shape`.
      y_start: Tensor-like, with shape `y_shape`.
      x_volumn: Scalar-like.
      g: Scalar-like, as the gravitational constant, optional.
      epsilon: Scalar-like, for numerical stability, optional.
    Returns:
      Scalar.
    """
    with tf.name_scope(name, 'Action',
            [y_bulk, grad_y_bulk, y_start, x_volumn]):
      y_bulk = tf.convert_to_tensor(y_bulk)
      grad_y_bulk = tf.convert_to_tensor(grad_y_bulk)
      y_start = tf.convert_to_tensor(y_start)
      # Add `batch_size`-dimension
      y_start = tf.expand_dims(y_start, axis=0)
      x_volumn = tf.convert_to_tensor(x_volumn)

      with tf.name_scope('DeltaHeight'):
        delta_y = y_start - y_bulk
        # Clip `delta_y` so as to keep it positive
        delta_y = tf.where(delta_y > 0.0, delta_y, tf.zeros_like(delta_y))

      with tf.name_scope('Lagrangian'):
        lagrangians = tf.truediv(
            tf.sqrt(1.0 + grad_y_bulk**2),
            tf.sqrt(2.0 * g * delta_y) + epsilon)

      action = x_volumn * tf.reduce_mean(lagrangians)
      return action

ResultOps = namedtuple('ResultOps', 'loss, grad_y_bulk, y_bulk')

def make_loss(x_bulk, x_boundary, target_y_boundary, make_y,
              name=None):
  """Implements the loss.

  Args:
    x_bulk: Tensor-like, with shape `[batch_size] + x_shape`, as the
        non-boundary values of the input to the neural network.
    x_boundary: Tensor-like, with shape `[n_boundaries] + x_shape`, as
        the boundary values of the input to the neural network.
    target_y_boundary: Tensor-like, with the shape `[n_boundary] + y_shape`,
        as the target boundary to fit.
    make_y: Callable that maps `x` to the neural network output.
  Returns:
    Scalar.
  """
  with tf.name_scope(name, 'Loss',
        [x_bulk, x_boundary, target_y_boundary]):
    x_bulk = tf.convert_to_tensor(x_bulk)
    x_boundary = tf.convert_to_tensor(x_boundary)
    target_y_boundary = tf.convert_to_tensor(target_y_boundary)

    x = tf.concat([x_bulk, x_boundary], axis=0)
    y = make_y(x)

    batch_size = x_bulk.get_shape().as_list()[0]
    n_boundary = x_boundary.get_shape().as_list()[0]
    y_bulk, y_boundary = tf.split(y, [batch_size, n_boundary],
                                  axis=0)
    grad_y_bulk = tf.gradients(y_bulk, [x_bulk])[0]
    y_start = tf.unstack(target_y_boundary, axis=0)[0]

    x_start, x_end = tf.unstack(x_boundary, axis=0)
    x_volumn = tf.reduce_prod(x_end - x_start)

    action = make_action(y_bulk, grad_y_bulk, y_start, x_volumn)
    loss = action

  return ResultOps(loss, grad_y_bulk, y_bulk)
```

```python
def generate_analytic_solution(A, max_theta):
    """Generates the analytic solution of brachistochrone curve.
    with parameters to be fine-tuned so as to match the boundary
    condtions."""
    theta = np.linspace(0, max_theta, 100, dtype='float32')
    theta = np.expand_dims(theta, axis=1)
    X = A * (theta - np.sin(theta))
    y = -A * (1.0 - np.cos(theta))
    return X, y

def l1_norm(x):
    """Helper. L1-norm."""
    return np.mean(np.abs(x))

def plot_frame(figure, subplot, X, y, message,
               X_analytic, y_analytic):
    """Helper, for dynamic plot."""
    subplot.clear()
    subplot.plot(X, y, X_analytic, y_analytic)
    subplot.set_title(message)
    figure.canvas.draw()

def test(x_dim=1, y_dim=1, x_boundary=[[0],[3]], target_y_boundary=[[0],[-1]],
         n_samples=1000, hidden_units_list=[10]*3, activation=tf.nn.sigmoid,
         optimizer=tf.train.AdamOptimizer(0.001), n_iters=10**4, skip_step=100,
         analytic_A=0.63, analytic_max_theta=4.15):

    # -- Analytic solution for comparison
    X_analytic, y_analytic = generate_analytic_solution(
            analytic_A, analytic_max_theta)

    # -- Build graph

    tf.reset_default_graph()

    x_bulk = tf.placeholder(shape=[n_samples, x_dim], dtype='float32',
                            name='x_bulk')
    x_boundary = np.array(x_boundary, dtype='float32')
    target_y_boundary = np.array(target_y_boundary, dtype='float32')

    hidden_layers = []
    for units in hidden_units_list:
        hidden_layers += [
            lambda x: tf.layers.dense(
                x, units, activation=activation),
        ]

    def make_y(x):
        """The $y(x)$ with the tricky boundary setting is:
        \begin{equation}
          y(x) = (x - x_0) (x - x_1) N(x)
               + \frac{ (y_1 - y_0) ( x - x_0) }{ x_1 - x_0 }
               + y_0 .
        \end{equation}
        """
        nn = make_neural_network(x, hidden_layers, [y_dim])
        y_boundary_setting = (
            target_y_boundary[0,0] \
                + (target_y_boundary[1,0] - target_y_boundary[0,0]) \
                  *(x - x_boundary[0,0]) \
                  /(x_boundary[1,0] - x_boundary[0,0])
        )
        y = (x-x_boundary[0,0]) * (x-x_boundary[1,0]) * nn \
          + y_boundary_setting
        return y

    ops = make_loss(x_bulk, x_boundary, target_y_boundary, make_y)

    # -- Optimization

    train_op = optimizer.minimize(ops.loss)

    with tf.Session() as sess:

        # Initialize all `tf.Variable`s, explicit or implicit
        sess.run(tf.global_variables_initializer())

        # -- Optimizing

        def get_x_bulk_value():
            """For feed-dict."""
            # Notice that `np.random.uniform(a, b)` samples `x ~ [a, b)`
            x_bulk_value = -np.random.uniform(
                -x_boundary[1,0], -x_boundary[0,0], size=[n_samples, 1])
            x_bulk_value = x_bulk_value.astype('float32')
            x_bulk_value = np.sort(x_bulk_value, axis=0)
            return x_bulk_value

        # For visualization
        figure = plt.figure()
        subplot = figure.add_subplot(111)
        plt.ion()
        figure.show()
        figure.canvas.draw()

        # Iterations
        for i in range(n_iters):
            step = i + 1
            iter_ops = [train_op, ops.loss, ops.y_bulk, ops.grad_y_bulk]
            x_bulk_value = get_x_bulk_value()
            result = sess.run(iter_ops, feed_dict={x_bulk: x_bulk_value})
            _, loss, y_bulk, grad_y_bulk = result

            # Visualization
            if step % skip_step == 0:
                message = '{0} | {1:.3f}'\
                          .format(step, loss)
                plot_frame(figure, subplot, x_bulk_value, y_bulk, message,
                           X_analytic, y_analytic)

            # For debugging
            if np.isnan(loss):
                print(step, loss, grad_y_bulk, x_bulk_value, y_bulk)
                plt.plot(x_bulk_value, y_bulk)
                plt.show()
                break
```

```python
%%time
test(n_iters=3*10**4)
```

    <IPython.core.display.Javascript object>

    Wall time: 49.5 s


# Pulse stream recovery under additive noise (cont.)

We continue the discussion from the previous notebook in which we considered **_additive noise_** that corrupts our measurements of a finite pulse stream:

\begin{align}
y_{meas}[n] = y_{BL}[n] + w[n],   \nonumber
\end{align}

where the values $w[n]$ can be modeled as zero-mean Gaussian noise $N(0, \sigma^2)$ and $y_{BL}[n]$ are the lowpass filtered samples of a (periodic) finite pulse stream, e.g.:

\begin{align}
y(t) = \sum_{n\in\mathbb{Z}}\sum_{k=0}^{K-1}c_k \hspace{0.05cm} h(t-t_k-nT), \nonumber
\end{align}

where $T$ is the period (or finite duration) of the pulse stream, $h(t)$ is a known pulse shape, and $\{c_k, t_k\}_{k=0}^{K-1}$ are the desired pulse parameters to estimate from the measured samples.

In the previous notebook, we saw how to exploit oversampled measurements, namely more samples than the degrees of freedom for the pulse $N\geq2K+1$, in order to estimate the pulse parameters from noisy samples. The discussed method, **_total least-squares_** (TLS) [1], exploited the rank of the Toeptliz matrix of Fourier coefficients, given by $K$ for $K$ pulses.

Although the approach worked well for high SNRs (signal-to-noise ratio) ($50$ dB for $K=6$ pulses), it quickly broke down for moderate SNR values ($10$ dB). This motivates an approach for **_denoising_** the estimated Fourier coefficients. Note that with TLS, no modifications are made to the estimated Fourier coefficients!

There are multiple approaches for denoising; the one we will discuss in this notebook is **_Cadzow's iterative denoising_** [1]. In the previous notebook, we observed how it led to a successful recovery of the pulse parameters for an SNR of $10$ dB, which  proved difficult for TLS alone.

```python
import numpy as np
import plot_settings
import sys
sys.path.append('..')

from frius import create_pulse_param, sample_ideal_project, estimate_fourier_coeff, add_noise, compute_ann_filt, estimate_time_param, estimate_amplitudes, evaluate_recovered_param
```

# 1. Cadzow's iterative denoising

As TLS, Cadzow's iterative denoising exploits the rank property in order to denoise the estimated Fourier coefficients. Recall our Toeplitz matrix of Fourier coefficients:

\begin{align}
\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}}) = \begin{bmatrix}
\hat{x}[\mathrel{{-M}{+}{K}}]&\hat{x}[\mathrel{{-M}{+}{K}{-}{1}}] & \cdots & \hat{x}[-M]\\
\vdots&\ddots &  \ddots & \vdots\\
\hat{x}[1] & \hat{x}[0] &   \cdots & \hat{x}[\mathrel{{-K}{+}{1}}] \\
\hat{x}[2] & \hat{x}[1] &  \cdots & \hat{x}[\mathrel{{-K}{+}{2}}] \\
\vdots&\vdots& \ddots & \vdots\\
\hat{x}[M] & \hat{x}[\mathrel{{M}{-}{1}}]  & \cdots & \hat{x}[\mathrel{{M}{-}{K}}]
\end{bmatrix},  \nonumber
\end{align}

If we can safely assume that the number of pulses is $K$, we saw earlier that $\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})$ should have a rank of $K$ in the noiseless scenario. The rank of $\mathbf{T}(\mathbf{\hat{x}}, \mathrel{{K}{+}{1}})$ will most likely increase when we obtain noisy estimated of the Fourier coefficients.

As a "proxy" for measuring the rank, we can perform a singular value decomposition (SVD) and observe the raio between the $K^{th}$ and $(\mathrel{{K}{+}{1}})^{th}$ singular value. In the noiseless case, the $(\mathrel{{K}{+}{1}})^{th}$ singular value will be zero due to the rank property, resulting in a very large ratio between $K^{th}$ and $(\mathrel{{K}{+}{1}})^{th}$ singular value if we are close to the noiseless case. We can verify this below with "noiseless" Fourier coefficients as numerical errors are inevitable.

```python
from frius import Tmtx      # function to create Toeplitz convolution matrix, defined in `frius/fri_utils.py`
from scipy.linalg import svd

# signal parameters
K = 6
period = 1
ck, tk = create_pulse_param(K=K, period=period)

# sample
oversample_freq = 2
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, oversample_freq=oversample_freq)

# check singular value ratio
fs_coeff_hat = estimate_fourier_coeff(y_samp, t_samp)
T = Tmtx(fs_coeff_hat, K)
s = svd(T)[1]
print("K/(K+1) singular value = %f\n" % (s[K-1]/s[K]))
```

    K/(K+1) singular value = 61065944870001.031250

For decreasing SNRs, we can then observe how this ratio decreases decreases *exponentially*, see the figure below and note that the $y$-axis is in log scale.

```python
# signal parameters
K = 6
period = 1
oversample_freq = 2

# sweep parameters
snr_vals = np.arange(100, -1, -5)
n_trials = 20
ratios = np.zeros((len(snr_vals), n_trials))
for seed in range(n_trials):
    ck, tk = create_pulse_param(K=K, period=period, seed=seed)
    y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, oversample_freq=oversample_freq)
    for k, snr_db in enumerate(snr_vals):
        y_noisy = add_noise(y_samp, snr_db=snr_db)
        fs_coeff_hat = estimate_fourier_coeff(y_noisy, t_samp)
        T = Tmtx(fs_coeff_hat, K)
        s = svd(T)[1]
        ratios[k, seed] = s[K-1]/s[K]
avg_ratios = np.mean(ratios, axis=1)

plt.figure(figsize=(10,5))
plt.semilogy(snr_vals, avg_ratios)
plt.grid()
plt.xlabel("SNR [dB]")
plt.xlim([min(snr_vals), max(snr_vals)])
plt.ylabel("Singular value ratio");
```

How can we remedy this situation? Namely, how can we prevent such a decline in the singular value ratio? If we return to the SVD of our Toeplitz matrix:

\begin{align}
\mathbf{T}(\mathbf{\hat{x}_{noisy}}, L)  = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^H \text{ where } L \geq K+1, \nonumber
\end{align}

our desired result (for $K$ pulses) is that we have at most $K$ non-zero singular values in $ \mathbf{\Sigma} \in \mathbb{C}^{(2M+1-K)\times L} $. If this is not the case, we could think of **_projecting_** $\mathbf{\Sigma}$ to a matrix consisting of only the $K$ largest singular values, $\rightarrow \mathbf{\Sigma}'$.

As $ \mathbf{\Sigma}$ is a rectangular diagonal matrix with singular values in decreasing order along the diagonal, this projection consists of simply "zero-ing" those diagonal entries after the $K^{th}$ value. We could then construct a matrix of rank $K$ as such:

\begin{align}
\mathbf{F}_K = \mathbf{U} \mathbf{\Sigma}' \mathbf{V}^H. \nonumber
\end{align}

However, this new matrix of rank $K$ may no longer have a Toeplitz structure, i.e. constant along the diagonals. In order to preserve this property we could simply average along the diagonals. As the constants along the diagonals correspond to Fourier coefficients (see matrix above), we can extract the **_denoised_** coefficients as those values along  each diagonal.

Those two steps are the keys components of Cadzow's iterative denoising! Namely:
1. Projecting to low rank matrix with the SVD.
2. Averaging along diagonals to maintain Toeplitz structure.

The last step will most likely increase the rank of the matrix so the two steps are repeated for a fixed number of iterations or until the ratio between the $K^{th}$ and $(\mathrel{{K}{+}{1}})^{th}$ singular value reaches a desired threshold. Once the stopping criterion has been met, the denoised Fourier coefficients can be extracted along the diagonals of the new Toeplitz matrix.

# 2. Code and simulation

Below we implement a function that performs Cadzow's iterative denoising. An additional degree of freedom for the denoising approach is the number of columns $L$ in the Toeplitz matrix. In [1], they suggest creating a Toeplitz that is as square as possible.

```python
def cadzow_denoising(fs_coeff, K, L=None, n_iter=10, thresh=None, verbose=False):

    n_coeff = len(fs_coeff)
    if L is None:
        L = int(np.floor(0.5*n_coeff))

    # iterative denoising
    fs_coeff_denoised = fs_coeff.copy()
    diag_idx = L - np.arange(n_coeff)
    S_tild = np.zeros((n_coeff-L, L+1), dtype=np.complex)
    for k in range(n_iter):

        # form Toeplitz
        T = Tmtx(fs_coeff_denoised, L)

        # project to rank K
        U, s, Vh = svd(T)
        if verbose:
            print("ITERATION %d : K/(K+1) singular value = %f" % (k, s[K-1]/s[K]))
        if thresh is not None:
            if (s[K-1]/s[K]) >= thresh:
                break
        for d in range(K):
            S_tild[d,d] = s[d]
        T_tild = np.dot(np.dot(U, S_tild), Vh)

        # average along diagonal
        for i, m in enumerate(diag_idx):
            fs_coeff_denoised[i] = np.mean(np.diag(T_tild, m))

    return fs_coeff_denoised
```

The above function is also defined in `'frius/fri_utils.py'`.

In order to get a better idea of the algorithm performance, we suggest playing with the key denoising parameters below:
* oversampling factor: `oversample_fact`
* number of columns: `L`
* number of Cadzow iterations: `n_iter`

```python
# signal parameters
K = 6
period = 1
ck, tk = create_pulse_param(K=K, period=period)

# denoising parameters
oversample_freq = 7
L = None       # "None" for defaulting to square Toeplitz
n_iter = 10

# sample
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, oversample_freq=oversample_freq)

# add noise
snr_db = 10
y_noisy = add_noise(y_samp, snr_db=snr_db)

# apply cadzow denoising
fs_coeff_hat = estimate_fourier_coeff(y_noisy, t_samp)
fs_coeff_denoised = cadzow_denoising(fs_coeff_hat, K, L=L, n_iter=n_iter, verbose=True)
print()

# recovery
freqs = fs_ind_base/period
ann_filt = compute_ann_filt(fs_coeff_denoised, K, print_ratio=True)
tk_hat = estimate_time_param(ann_filt, period)
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

evaluate_recovered_param(ck, tk, ck_hat, tk_hat, viz=True, figsize=(10,5), t_max=period)
plt.legend(loc="lower left");
```

In the next notebook, we will use this motivation of SNR to deal with **_modulated_** pulses, i.e. pulses that are not centered at baseband. We can intuitively see that in order to have a better SNR in our measurements, we should not stick to using those Fourier coefficients around baseband _but rather_ those coefficients centered around the modulation frequency of our pulse.

# References

    [1] T. Blu, P. L. Dragotti, M. Vetterli, P. Marziliano and L. Coulot, "Sparse Sampling of Signal Innovations," in IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 31-40, March 2008.


## Creating a function from a symbolic expression
In SymPy there is a function to create a Python function which evaluates (usually numerically) an expression. SymPy allows the user to define the signature of this function (which is convenient when working with e.g. a numerical solver in ``scipy``).

```python
import sympy as sym
sym.init_printing()
```

We will look at an arbitrary expression $f(x, y)$:

$$
f(x, y) = 3 x^{2} + \log{\left (x^{2} + y^{2} + 1 \right )}
$$

```python
x, y = sym.symbols('x y')
expr = 3*x**2 + sym.log(x**2 + y**2 + 1)
expr
```

One way to evaluate above expression numerically is to invoke the ``subs`` method followed by the ``evalf`` method:

```python
expr.subs({x: 17, y: 42}).evalf()
```

However, if we need to do this repeatedly it can be quite slow:

```python
%timeit expr.subs({x: 17, y: 42}).evalf()
```

even compared to a simple lambda function:

```python
import math
f = lambda x, y: 3*x**2 + math.log(x**2 + y**2 + 1)
f(17, 42)
```

```python
%timeit f(17, 42)
```

SymPy can also create a function analogous to f above. The function for doing so is called ``lambdify``:

```python
g = sym.lambdify([x, y], expr, modules=['math'])
g(17, 42)
```

```python
%timeit g(17, 42)
```

Note how we specified ``modules`` above: it tells ``lambdify`` to use ``math.log``, if we don't specify modules SymPy will (since v1.1) use ``numpy`` by default. This can be useful when dealing with arrays in the input:

```python
import numpy as np
xarr = np.linspace(17, 18, 5)
h = sym.lambdify([x, y], expr)
out = h(xarr, 42)
out.shape
```

NumPy's [broadcasting](https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) (handling of different shapes) then works as expected:

```python
yarr = np.linspace(42, 43, 7).reshape((1, 7))
out2 = h(xarr.reshape((5, 1)), yarr)  # if we would try to use g() here, it would fail
out2.shape
```

Behind the scenes ``lambdify`` constructs a string representation of the Python code and uses Python's ``eval`` function to compile the function.

Let's now look at how we can get a specific function signature from ``lambdify``:

```python
z = z1, z2, z3 = sym.symbols('z:3')
expr2 = x*y*(z1 + z2 + z3)
func2 = sym.lambdify([x, y, z], expr2)
func2(1, 2, (3, 4, 5))
```

## Exercise: Create a function from a symbolic expression
Plot $z(x, y) = \frac{\partial^2 f(x,y)}{\partial x \partial y}$ from above ($f(x, y)$ is available as ``expr``) as a surface plot for $-5 < x < 5, -5 < y < 5$.

```python
xplot = np.outer(np.linspace(-5, 5, 100), np.ones(100))
yplot = xplot.T
```

```python
%load_ext scipy2017codegen.exercise
```

*Use either the *``%exercise``* or *``%load``* magic to get the exercise / solution respecitvely*

```python
%exercise exercise_lambdify_expr.py
```

Replace **???** with the correct expression above.

```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
fig = plt.figure(figsize=(15, 13))
ax = plt.axes(projection='3d')
ax.plot_surface(xplot, yplot, zplot, cmap=plt.cm.coolwarm)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('$%s$' % sym.latex(d2fdxdy))
```

```python

```


There are lots of statistics in statistical signal processing, but to use statistics effectively with signals, it helps to have a certain unifying perspective on both. To introduce these ideas, let's start with the  powerful and intimate connection between least mean-squared-error (MSE) problems and conditional expectation that is sadly not emphasized in most courses.

Let's start with an example: suppose we have two fair six-sided die ($X$ and $Y$) and I want to measure the sum of the two variables as $Z=X+Y$. Further, let's suppose that given $Z$, I want the best estimate of $X$ in the mean-squared-sense. Thus, I want to minimize the following:

$$ J(\alpha) = \sum ( x - \alpha z )^2 \mathbb{P}(x,z)   $$

Here $\mathbb{P}$ encapsulates the density (i.e. mass) function for this problem. The idea is that when we have solved this problem, we will have a function of $Z$ that is going to be the minimum MSE  estimate of $X$.

We can substitute in for $Z$ in $J$ and get:

$$ J(\alpha) = \sum ( x - \alpha (x+y) )^2 \mathbb{P}(x,y)   $$

Let's work out the steps in `sympy` in the following:

```
import sympy
from sympy import stats, simplify, Rational, Integer,Eq
from sympy.stats import density, E
from sympy.abc import a

x=stats.Die('D1',6) # 1st six sided die
y=stats.Die('D2',6) # 2nd six sides die
z = x+y             # sum of 1st and 2nd die

J = stats.E((x - a*(x+y))**2)         # expectation
sol=sympy.solve(sympy.diff(J,a),a)[0] # using calculus to minimize
print sol # solution is 1/2
```

    1/2

This says that $z/2$ is the MSE estimate of $X$ given $Z$ which means geometrically ( interpreting the MSE as a squared distance weighted by the probability mass function) that $z/2$ is as *close* to $x$ as we are going to get for a given $z$.

Let's look at the same problem using the conditional expectation operator $ \mathbb{E}(\cdot|z) $ and apply it to our definition of $Z$, then

$$ \mathbb{E}(z|z) = \mathbb{E}(x+y|z) = \mathbb{E}(x|z) + \mathbb{E}(y|z) =z  $$

where we've used the linearity of the expectation. Now, since by the symmetry of the problem, we have

$$ \mathbb{E}(x|z) = \mathbb{E}(y|z) $$

we can plug this in and solve

$$  2 \mathbb{E}(x|z)  =z $$

which gives

$$   \mathbb{E}(x|z)  =\frac{z}{2} $$

which is suspiciously equal to the MSE estimate we just found. This is not an accident! The proof of this is not hard,  but let's look at some pictures first

```
fig, ax = subplots()
v = arange(1,7) + arange(1,7)[:,None]
foo=lambda i: density(z)[Integer(i)].evalf() # some tweaks to get a float out
Zmass=array(map(foo,v.flat),dtype=float32).reshape(6,6)

ax.pcolor(arange(1,8),arange(1,8),Zmass,cmap=cm.gray)
ax.set_xticks([(i+0.5) for i in range(1,7)])
ax.set_xticklabels([str(i) for i in range(1,7)])
ax.set_yticks([(i+0.5) for i in range(1,7)])
ax.set_yticklabels([str(i) for i in range(1,7)])
for i in range(1,7):
    for j in range(1,7):
        ax.text(i+.5,j+.5,str(i+j),fontsize=18,color='y')
ax.set_title(r'Probability Mass for $Z$',fontsize=18)
ax.set_xlabel('$X$ values',fontsize=18)
ax.set_ylabel('$Y$ values',fontsize=18);
```

The figure shows the values of $Z$ in yellow with the corresponding values for $X$ and $Y$ on the axes. Suppose $z=2$, then the closest $X$ to this is $X=1$, which is what $\mathbb{E}(x|z)=z/2=1$ gives. What's more interesting is what happens when $Z=7$? In this case, this value is spread out along the $X$ axis so if $X=1$, then $Z$ is 6 units away, if $X=2$, then $Z$ is 5 units away and so on.

Now, back to the original question, if we had $Z=7$ and I wanted to get as close as I could to this using $X$, then why not choose $X=6$ which is only one unit away from $Z$? The problem with doing that is $X=6$ only occurs 1/6 of the time, so I'm not likely to get it right the other 5/6 of the time. So, 1/6 of the time I'm one unit away but 5/6 of the time I'm much more than one unit away. This means that the MSE score is going to be worse. Since each value of $X$ from 1 to 6 is equally likely, to play it safe, I'm going to choose $7/2$ as my estimate, which is what the conditional expectation suggests.

We can check this claim with samples using `sympy` below:

```
#generate samples conditioned on z=7
samples_z7 = lambda : stats.sample(x, sympy.Eq(z,7)) # Eq constrains Z
mn= mean([(6-samples_z7())**2 for i in range(100)]) #using 6 as an estimate
mn0= mean([(7/2.-samples_z7())**2 for i in range(100)]) #7/2 is the MSE estimate
print 'MSE=%3.2f using 6 vs MSE=%3.2f using 7/2 ' % (mn,mn0)
```

    MSE=9.99 using 6 vs MSE=2.97 using 7/2

Please run the above code repeatedly until you have convinced yourself  that the $\mathbb{E}(x|z)$ gives the lower MSE every time.

To push this reasoning,  let's consider the case where the die is so biased so that the outcome of *6* is ten times more probable than any of the other outcomes as in the following:

```
# here 6 is ten times more probable than any other outcome
x=stats.FiniteRV('D3',{1:Rational(1,15), 2:Rational(1,15), 3: Rational(1,15),
                       4:Rational(1,15), 5:Rational(1,15), 6: Rational(2,3)})
z = x + y

# now re-create the plot
fig, ax = subplots()
foo=lambda i: density(z)[Integer(i)].evalf() # some tweaks to get a float out
Zmass=array(map(foo,v.flat),dtype=float32).reshape(6,6)

ax.pcolor(arange(1,8),arange(1,8),Zmass,cmap=cm.gray)
ax.set_xticks([(i+0.5) for i in range(1,7)])
ax.set_xticklabels([str(i) for i in range(1,7)])
ax.set_yticks([(i+0.5) for i in range(1,7)])
ax.set_yticklabels([str(i) for i in range(1,7)])
for i in range(1,7):
    for j in range(1,7):
        ax.text(i+.5,j+.5,str(i+j),fontsize=18,color='y')
ax.set_title(r'Probability Mass for $Z$; Nonuniform case',fontsize=16)
ax.set_xlabel('$X$ values',fontsize=18)
ax.set_ylabel('$Y$ values',fontsize=18);
```

As compared with the first figure, the probability mass has been shifted away from the smaller numbers. Let's see what the conditional expectation says about how we can estimate $X$ from $Z$.

```
E(x, Eq(z,7)) # conditional expectation E(x|z=7)
```

    5

Now that we have $\mathbb{E}(x|z=7) = 5$, we can generate samples as before and see if this gives the minimum MSE.

```
#generate samples conditioned on z=7
samples_z7 = lambda : stats.sample(x, Eq(z,7)) # Eq constrains Z
mn= mean([(6-samples_z7())**2 for i in range(100)]) #using 6 as an estimate
mn0= mean([(5-samples_z7())**2 for i in range(100)]) #7/2 is the MSE estimate
print 'MSE=%3.2f using 6 vs MSE=%3.2f using 5 ' % (mn,mn0)
```

    MSE=3.27 using 6 vs MSE=2.92 using 5

### Summary

Using a simple example, we have emphasized the connection between minimum mean squared error problems and conditional expectation. Next, we'll continue revealing  the true power of the conditional expectation as we continue to develop a corresponding geometric intuition.

As usual, the corresponding ipython notebook for this post  is available for download [here](https://github.com/unpingco/Python-for-Signal-Processing/blob/master/Conditional_expectation_MSE.ipynb).

Comments and corrections welcome!


# Some fundamental concepts of wave propagation

## Introduction
This notebook describes some fundamental concepts regarding representation of functions as Fourier series and Fourier integrals and its application to wave propagation analysis. It uses some of the functions availabl in the set of subroutines stored in the Github repository Py_Series. The notebook is by no means intended to be a thorough treatment of Fourier analysis but its main purpose is to show some of the fundamental mathematical concepts implemented in easy-to-use python subroutines. The notebook starts with the description of periodic function and its representation in terms of a Fourier series. The possibility of representing a periodic function by a summation of independent harmonics in the form of a Fourier series is then extended to the case of Fourier integrals where a non-periodic function is now assumed periodic but with an infinite period. Such a representation of non-periodic functions opens up the possibility of describing functions in two alternative, but related domains, e.g., the time-frequency domain. The time-frequency representation of functions is then applied to the representation of plane waves where the Fourier representation is used to transform both space and time into wavenumber and frequency respectively.

```python
%matplotlib notebook
#from __future__ import division
#from __future__ import division
import matplotlib.pyplot as plt
import numpy as np
from os import sys
#sys.path.append('../pyseries/')
import fourier as fou
from sympy import *
from sympy import init_printing
init_printing()
from IPython.display import Image
from IPython.display import HTML
```

```python
def harmonic(n , x , sn , nfig):
    an = 8.0/(np.pi*(2*n-1))
    sn[:] = an*np.sin((2.0*n-1.0)*x[:])
    plt.figure(nfig)
    plt.plot(x , sn)

    return sn
```

## Periodic Functions

A function $f(x)$ is said to be periodic of period $T$ if there is a constant $T$ such:

$$f(x + T) = f(x).$$

A periodic function $f(x)$ of period $T = 2 \ell $ can be represented as a Fourier series, denoted here as $F{S_{f(x)}}$, and of the general form:

$$F{S_{f(x)}}(x) = {a_0} + \sum\limits_{n = 1}^\infty  {\left( {{a_n}{C_{\frac{{n\pi x}}{\ell }}} + {b_n}{S_{\frac{{n\pi x}}{\ell }}}} \right)} $$

where the coefficients $a_0$ , $a_n$ and $b_n$ are defined by:

$${a_0} = \frac{1}{{2\ell }}\int\limits_{ - \ell }^{ + \ell } {f(x)dx}$$

$${a_n} = \frac{1}{\ell }\int\limits_{ - \ell }^{ + \ell } {f(x){C_{\frac{{n\pi x}}{\ell }}}dx}$$

$${b_n} = \frac{1}{\ell }\int\limits_{ - \ell }^{ + \ell } {f(x){S_{\frac{{n\pi x}}{\ell }}}dx.} $$

In the expressions above and the rest of this document we use ${C_\theta } = \cos \theta$ and ${S_\theta } = \sin \theta. $ It is now evident that representing a periodic function by its Fourier series amounts to finding its conforming coefficients ${a_n}$ and ${b_n}.$

### A perfect harmonic

Accordingly, a periodic function $f(x)$ ends up being described by the sum of harmonics of the type ${a_n}{C_{\frac{{n\pi x}}{\ell }}}$ and ${b_n}{S_{\frac{{n\pi x}}{\ell }}}.$ For instance, consider the following harmonic:

$$f(t) = A{C_{\omega t}}$$

where $\omega  = 2\pi f$ and plotted below.

```python
N = 512
T_t = 10.0
dt = T_t/N
f = 3.0
A = 3.0
omega = f*2.0*np.pi
res = np.zeros(N, dtype = float )

lif = 0
lff = dt*N
time = np.linspace(lif, lff, N , dtype=float)
res[:] = A*np.cos(omega*time[:])
plt.figure(0)
plt.plot(time , res)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x11ca04fd0>]

Since each harmonic contributes with a different frequency and different amplitude it is convenient to describe the contribution from each independent term in the series. This amplitude versus frequency plot is known as the amplitude spectra. The next plot shows the frequency spectra for the perfect harmonic recently described:

```python
FS = 15.0
x , Samag , A , Nf = fou.Ftrans(res*dt , N , T_t/N , FS)
fou.grafFourier(Samag*dt , x , Nf , 1)
```

    <IPython.core.display.Javascript object>

### Square wave

To clarify consider the following square wave which is periodic with period $T = \pi$

and admitting the following Fourier series representation:

$$F{S_{f(x)}} = 2 + \frac{8}{{\pi (2n - 1)}}\sum\limits_{n = 1}^\infty  {{S_{(2n - 1)x}}}. $$

Each term in the function corresponds to a harmonic of a given frequency and amplitude. Let us consider the independent harmonics one step at a time. The next set of plots shows harmonics of increasing frequency.

```python
x = np.arange(-5.0 , 5.0 , 0.05)
n = len(x)
sum = np.zeros(n)
sn  = np.zeros(n)
```

The first term in the series correspond to the constant coefficient $a_0$

```python
# S0
sum[:] = 2.0
plt.figure(3)
plt.plot(x , sum)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x11ce984e0>]

Consider now the terms for $n=1$  and up to $ n = 4$

```python
for i in range(1, 5):
    sn = harmonic(i , x , sn , 2)
    sum[:] = sum[:] + sn[:]
    sn[:] = 0.0
```

The function resulting from the partial summation of independent harmonics is shown next. Each independent harmominc contributes with different amplitude and frequency.

```python
plt.figure(4)
plt.plot(x , sum)
N = len(sum)
FS = 15.0
T_t=10.0
```

The Fourier amplitude spectra of the resulting function is shown next:

```python
x , Samag , A , Nf = fou.Ftrans(sum , N , T_t/N , FS)
fou.grafFourier(Samag*dt , x , Nf , 5)
plt.show()
```

### Complex Valued Fourier Series
Substituting the sine and cosine terms in the Fourier series representation discussed above by its corresponding representations in terms of the well-known Euler formulas:

$${C_\theta } = \frac{{{e^{i\theta }} + {e^{ - i\theta }}}}{2}$$

$${S_\theta } = \frac{{{e^{i\theta }} - {e^{ - i\theta }}}}{2i}$$

we have the alternative representation of $f(x)$:

$$F{S_{f(x)}}(x) = \sum\limits_{n =  - \infty }^\infty  {{c_n}} {e^{\frac{{in\pi x}}{\ell }}}$$

$${c_n} = \frac{1}{{2\ell }}\int\limits_{ - \ell }^{ + \ell } {f(x){e^{ - \frac{{in\pi x}}{\ell }}}dx}$$

in which the $c_n$ terms and the exponentials are complex functions; although the resulting series  is real.

## Non periodic functions- The Fourier integral

A non-periodic function $f(x)$ can still be represented by a Fourier series if it is assumed periodic with an infinite period in which case the Fourier series representation becomes a continous integral representation:

$$f(x) = \frac{1}{{2\pi }}\int\limits_{ - \infty }^{ + \infty } {F(\omega ){e^{  i\omega x}}d\omega }$$

where:

$$F(\omega ) = \int\limits_{ - \infty }^{ + \infty } {f(x){e^{ - i\omega x}}dx}$$

The above pair of integrals allows one to express functions back and forth in the space (time) and frequency domain. The frequency domain representation $F(\omega)$ of the function $f(x)$ is its Fourier tansform, while the representation of the spatial function $f(x)$ in terms of frequency domain functions $F(\omega)$ is its inverse Fouier transform.

## Analytic transform

To show the alternative space-frequency domain representation of functions as described by the Fourier transform pair consider the following function:

$$f(t) = 4.0{e^{ - \left| t \right|}} - 5.0{e^{ - 3\left| {t + 2} \right|}}$$

```python
N   = 1024
T_t = 16.0
x = np.arange(-T_t/2.0 , T_t/2.0 ,T_t/N)
n = len(x)
FX = np.zeros((n), dtype = complex)

FX = 4.0*np.exp(-abs(x))-5.0*np.exp(-3.0*abs(x+2.0))
plt.figure(6)
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.plot(x , FX)
plt.show()
```

with transform given by:

$$F(\omega ) = \frac{8}{{{\omega ^2} + 1}} - \frac{{30{e^{2\hat i\omega }}}}{{{\omega ^2} + 9}}$$

and the following amplitude spectra

```python
omega = np.arange(0.0 , np.pi*N/T_t , 2.0*np.pi/T_t)
nw = len(omega)
Fw = np.zeros((nw), dtype = complex)
t1 = np.zeros(nw)
t2 = np.zeros(nw)

t1 = 8.0/(omega**2+1.0)
t2 = (30.0/(omega**2+9.0))*np.exp(2*omega*1j)
Fw = t1-t2
plt.figure(7)
plt.xlabel('Frequency (Hz)')
plt.ylabel('Amplitude')
plt.xlim(0.0 , 10.0)
plt.plot(omega/(2.0*np.pi) , np.abs(Fw))
plt.show()
# We can compute it numerically also
FS = 15.0
x , Samag , A , nfs = fou.Ftrans(FX*T_t/N , N , T_t/N , FS)
fou.grafFourier(Samag , x , nfs , 7)
```

## Fourier transform with the FFT() algorithm

In most of the problems of interest in engineering (and physics) the involved functions are not available in closed form and the Fourier transform must be computed numerically. This can be achieved efficiently via the Fast Fourier Transform algorithm, see Cooley and Tukey (1965). In this section we show some details about the computation of the Fourier transform using python and particularly the FFT algorithm. The transform itself is coded into subroutine Ftrans() availbale in this REPO. To show its use we introduce a Ricker pulse, which is a simple function widely used in wave propagation analysis.

Consider the discrete version of a function, or more general, a time series. Assume the function to be of maximum frequnency $f_{max}$. The sampling theorem mandates that the associated time series must be sampled at least at a rate

$$\delta = \frac{1}{2f_{max}}$$

in order to capture this maximum frequency.

Consider then the case of a signal of total duration $T$ sampled by $N$ points. In the FFT algorithm this time series is stored in memory as shown in the top part of the figure belows. Similarly, the corresponding frequency representation of the function is stored as shown in the bottom part of the figure.

After selecting the number of data points $N$, the resulting effective sampling rate turns out to be:

$$\Delta t = \frac{T}{N}$$

allowing to effectivly describe frquencies up to:

$$f = \frac{1}{{2\Delta t}}$$

sampled at:

$$df = \frac{1}{{N} \Delta t}.$$

In the following section we apply these ideas to the numerical treatment of a specific signal.

## Example: The Ricker Wavelet

The Ricker wavelet is a time signal defined by simple parameters thus allowing to produce time series with a perfectly known and therefore easy-to-manipulate spectra.

The time signal corresponding to the Ricker wavelet is defined by the following function:

$$f(t) = \left[ {1 - 2{\pi ^2}f_c^2{{(t - {t_c})}^2}} \right]{e^{ - {\pi ^2}f_c^2{{(t - {t_c})}^2}}}$$

where $f_c$ is the dominant peak frequency and $t_c$ defines the time location of the center peak in the tinme domain. Notice that the actual frequency content of the pulse is defined by its time-domain "size" or breadth denoted by $t_b$ and given by:

$${t_b} = \frac{{\sqrt 6 }}{{\pi {f_c}}}.$$

### Design of the Ricker pulse

To design a particular pulse we need to select values of the parameters defining its breadth $t_b$ (or frequency content $f_c$) and its location within a possible longer signal of duration $T_t$ required for analysis purposes.

The pulse perse has a duration (or breath) defined by its central frequency according to:

$${t_b} = \frac{{\sqrt 6 }}{{\pi {f_c}}}$$

To correctly compute the pulse numerically we need to define the total number of time samples $N$ making sure that we keep at last the minimum required sampling rate as specified by $f_c$.

Thus, in the design of the pulse we select:
(i) the total duration of the signal $T_t$
(ii) the total number of data points $N$ (usually a power of 2)
(iii) the time location of the central peak of the pulse $t_c$
(iv) the central frequency of the pulse $f_c$.

Consider for instance the set of parameters defined in the python code below where we use the function ricker() to compute the corresponding pulse:

```python
N = 1024
T_t = 8.0
tc = 1.0
fc = 4.0
t_b = np.sqrt(6)/np.pi/fc
print ("Breadth =") , t_b , ('s')
Rick , time = fou.ricker(N , T_t , tc, fc)
fou.grafsignalG(Rick , T_t/N , 9)
```

Notice the amplitude spectra can be reproduced up to:

```python
print ('Effctive maximum frquency =') , N/2.0/T_t
print ('Effective time step=') , T_t/N
```

    Effctive maximum frquency =
    Effective time step=

    (None, 0.0078125)

Now, conducting direct fourier transform of the above time signal we obtain the corresponding Fourier amplitude spectra spectra:

```python
FS = 10.0
x , Samag , A , nfs = fou.Ftrans(Rick*T_t/N , N , T_t/N , FS)
fou.grafFourier(Samag , x , nfs , 10)
```

We can also perform inverse Fourier transform to get the time signal once again as follows:

```python
ZZ = np.fft.ifft(A*N/T_t)
ZZ = np.real(ZZ)
fou.grafsignalG(ZZ , T_t/N , 11)
```

Notice that although we prescribed a central frequency of $f_c = 1.0 Hz$ the selected sampling rate produced an spectral description up to $64.0 Hz$. The user is invited to use a larger sampling rate and se what is the effect upon the frequency description of the pulse.

## Example: A Seismic pulse

Consider now the following pulse resembling a typical seismic signal:

$$f(t) = {C_{6\pi t}}{e^{ - \pi {t^2}}}$$

The pulse (which is shown in the figure) has been discretized according to the following parameters:

```python
N   = 2048
T_t = 4.0
x = np.arange(-2.0 , 2.0 ,T_t/N)
nn = len(x)
FX = np.zeros((nn), dtype = complex)
xm = x
pot = -np.pi*xm**2
FX = np.cos(6.0*np.pi*xm)*np.exp(pot)
fou.grafsignalG(FX , T_t/N , 12 )
```

The corresponding inverse Fourier transform produces the following amplitude spectra:

```python
FS = 10.0
x , Samag , A , nfs = fou.Ftrans(FX*T_t/N , N , T_t/N , FS)
fou.grafFourier(Samag , x , nfs , 13)
```

Notice that in the computation of the Fourier transform the signal had to be scaled by the paramter $\Delta t.$

An interesting description of the Fourier representation of a function is available in the Youtube site: https://www.youtube.com/watch?v=spUNpyF58BY.

## Popagation of a plane wave
Consider a plane front of constant frquency $\omega$ (i.e., a ray) propagating in the $x-y$ space with propagation direction $\hat n = {S_\gamma }\hat i - {C_\gamma }\hat j$ (see Figure below). The propagation media, corresponds to a full-space and is characterized by its propagation velocity $c = \beta$ and its mass density $\rho$. In the figure the space is assumed infinite and the system of reference is located at an arbitrary point. Notice that the angle $\gamma$ indicating the direction of propagation is defined with respect to the vertical.

The frequency domain representation (i.e., the Fourier transform) of the response at a point ${\vec x}$ due to the passage of the wave front is given by the (transfer) function:

$${F_i}(\hat i\omega ) = {d_i}{e^{\hat i\vec \kappa  \cdot \vec x}}$$

where $\vec \kappa  = \frac{\omega }{c}\hat n$ is the wave vector, while ${d_i}$ is a unit vector defining the polarization direction. In the case of $SH$ polarization $d$ is simply $1.0.$

If on the other hand, $R\left( {\hat i\omega } \right)$ is the Fourier spectra of a pulse, say a Ricker pulse, then the time domain response at ${\vec x}$ can be computed via:

$${u_i}(\vec x,t) = \frac{1}{{2\pi }}\int\limits_{ - \infty }^{ + \infty } {R\left( {\hat i\omega } \right)} F_i^*(\hat i\omega ){e^{ - \hat i\omega t}}d\omega.$$

In the above $F_i^*(\hat i\omega )$ is the complex conjugate of ${F_i}(\hat i\omega ).$

### Propagating pulse
Consider a Ricker pulse characterizd by the same parameters as defined previously and let us compute the response at a given point ${\vec x}.$ The location of the center of the pulse, defined by ${t_c}$ gives the response at the origin of the system of reference.

If we assume that the pulse starts to propagate from a distance $y = H$ it will arrive at the origin of the reference system at a time $t=H/c$. It means that the time signal must be assigned a duration long enough to allow the front to propagate along the whole domain.

For instance, assume that $\beta = 1.0$ and $H = 10.0$. It means that the front will appear at the origin approximately at $t = 8.0s$, so we will center it at $t = 9.0s$ and will also give   it a total duration of $T_t = 12.0$.

Let us assign values to the parameters and plot its time domain representation at the origin (i.e., after 8 seconds have passed).

```python
FS = 10.0
N  = 512
T_t = 12.0
tc  = 9.0
fc  = 1.0
fm  = 4.0*fc
dt  = T_t/N
Rick , time = fou.ricker(N , T_t , tc, fc)
x , Sas , Saf , nfs = fou.Ftrans(Rick , N , dt , FS)
fou.grafsignalG(Rick , dt , 14)
```

### Transfer function
We can now find the Fourier representation of the response (or also the transfer function of the system) ${F_i}(\hat i\omega )$. After defining th parameters required for the Fourier analysis we have:

```python
Nf =int((N - 1)/2)
df = 1.0/T_t
lif = df
lff = df*Nf
fre = np.linspace(lif, lff, Nf , dtype=float)
```

Assigning the angle of incidence $\gamma$, the propagation velocity $\beta$ and the coordinates of the response point ${\vec x}$

```python
desplaz = np.zeros(Nf, dtype=complex)
Gamma = 0.0
Gamma = (Gamma*np.pi/180.0)
nx1 = np.sin(Gamma)
ny1 =-np.cos(Gamma)
Beta = 1.0
xcor = 0.0
ycor = 8.0
```

```python
for l in range (0, Nf):
    kapa = 2.0*np.pi*fre[l]/Beta
    pha_ang_1  = -kapa*(nx1*xcor + ny1*ycor)
    desplaz[l] =  np.exp(1j*pha_ang_1)

TF = np.zeros(N , dtype=complex)
for i in range(Nf):
    TF[i+1 ] = desplaz[i]
    TF[-1-i] = np.conj(desplaz[i])
```

### Time domain response
Now we can conduct the convolution $R\left( {\hat i\omega } \right)F_i^*(\hat i\omega )$ and the final inverse Fourier transform to obtain the time domain response according to:

$${u_i}(\vec x,t) = \frac{1}{{2\pi }}\int\limits_{ - \infty }^{ + \infty } {R\left( {\hat i\omega } \right)} F_i^*(\hat i\omega ){e^{ - \hat i\omega t}}d\omega. $$

Notice the time-location of the pulse at the point $\vec x$ and compare it with its time location at the point $\vec x = 0.$

```python
for i in range(N):
    TF[i] = Saf[i] * TF[i]

ZZ = np.fft.ifft(TF)
ZZ = np.real(ZZ)
fou.grafsignalG(ZZ , dt , 15)
```

### Response at multiple points

Now we will compute the response at various points defined by the coordinate arrays x[] and y[] respectively:

```python
FS  = 10.0
N   = 512
T_t = 12.0
tc  = 9.0
fc  = 1.0
fm  = 4.0*fc
dt  = T_t/N
Rick , time = fou.ricker(N , T_t , tc, fc)
x , Sas , Saf , nfs = fou.Ftrans(Rick , N , dt , FS)
#
xx =[0.0 , 0.0 , 0.0 , 0.0 ]
yy =[0.0 , 3.0 , 2.0 , 1.0]
#
for j in range(4):
    desplaz = np.zeros(Nf, dtype=complex)
    xcor = xx[j]
    ycor = yy[j]
    for l in range(Nf):
        kapa = 2.0*np.pi*fre[l]/Beta
        pha_ang_1  = -kapa*(nx1*xcor + ny1*ycor)
        desplaz[l] =  np.exp(1j*pha_ang_1)

    for i in range(Nf):
        TF[i+1 ] = desplaz[i]
        TF[-1-i] = np.conj(desplaz[i])

    for i in range(N):
        TF[i] = Saf[i] * TF[i]
    ZZ = np.fft.ifft(TF)
    ZZ = np.real(ZZ)
    fou.grafsignalG(ZZ , dt, 16)
```

### Two propagating rays

Now, let us add a second ray of direction dfined by $\hat n =  - {S_\theta }\hat i - {C_\theta }\hat j$

```python
Theta = 0.0
Gamma = 15.0
```

```python
FS  = 10.0
N   = 512
T_t = 12.0
tc  = 9.0
fc  = 1.0
fm  = 4.0*fc
dt  = T_t/N
Rick , time = fou.ricker(N , T_t , tc, fc)
x , Sas , Saf , Nf = fou.Ftrans(Rick , N , dt , FS)
#
desplaz = np.zeros(Nf, dtype=complex)
Beta = 1.0
xcor = 3.0
ycor = 3.0
#
Theta = (Theta*np.pi/180.0)
Gamma = (Gamma*np.pi/180.0)
nx1 = np.sin(Theta)
ny1 =-np.cos(Theta)
nx2 =-np.sin(Gamma)
ny2 =-np.cos(Gamma)
```

```python
for l in range (0, Nf):
    kapa = 2.0*np.pi*fre[l]/Beta
    pha_ang_1  = -kapa*(nx1*xcor + ny1*ycor)
    pha_ang_2  = -kapa*(nx2*xcor + ny2*ycor)
    desplaz[l] = np.exp(1j*pha_ang_1) + np.exp(1j*pha_ang_2)

TF = np.zeros(N , dtype=complex)
for i in range(Nf):
    TF[i+1 ] = desplaz[i]
    TF[-1-i] = np.conj(desplaz[i])
```

```python
for i in range(N):
    TF[i] = Saf[i] * TF[i]

ZZ = np.zeros(N , dtype=complex)
ZZ = np.fft.ifft(TF)
ZZ = np.real(ZZ)
plt.clf()
fou.grafsignalG(ZZ , dt , 17)
```

and compute also the Fourier amplitude spectra of the response.

```python
x , Sas_out , Saf , nfs = fou.Ftrans(ZZ , N , dt , FS)
fou.grafFourier(Sas_out*dt , x , Nf , 18)
```

Notice that the time domain response is now a distroted pulse, while its Fourier spectra has frequency information different than in the single pulse.

### Ratio between 2 Fourier spectra
Sometimes it is required to obtain the output/input ratio:

```python
FT_out =Sas_out/Sas
fou.grafFourier(FT_out , x , Nf , 19)
```

## References

Advanced Engineering Mathematics. Michael D. Greenberg. Prentic Hall. Upper Saddle River, New Jersey. Second Edition, 1998.

Quantitative Seismology. Keiiti Aki and Paul G. Richards. University Scinc Books. Second Edition, 2009.

Wave Propagation in Elastic Solids. J.D. Achenbach. North-Holland series in Applied Mathematics and Mechanics, Volume 16. North-Holland Publishing Company, New York.

An algorithm for the machine calculation of complex Fourier series. Cooley, James W.; Tukey, John W. (1965). Mathematics of Computation. 19 (90): 297–301. ISSN 0025-5718.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```

examples/Fourier Series and Fourier transform.ipynb


```python
%pylab inline
import numpy as np
import numpy.polynomial.polynomial as n_poly
import numpy.polynomial.legendre as leg
from scipy import integrate
from scipy import optimize
from scipy import interpolate
import sympy as sym

```

```python
# Set the interval for L2 and the number of basis functions
A=-1.0
B=1.0
N=15
```

```python
# Define the scalar product in L2([A,B])
t = sym.symbols('t')

def scalar_prod(p0,p1):
    assert len(p0.free_symbols) <= 1, "I can only do this for single variable functions..."
    tt = p0.free_symbols.pop() if len(p0.free_symbols) == 1 else sym.symbols('t')
    foo1 = sym.lambdify(tt, p0, 'numpy')
    foo2 = sym.lambdify(tt, p1, 'numpy')
    foo3 = lambda x : foo1(x) * foo2(x)
    foo = integrate.quad(foo3,A,B)
    return foo[0]
```

```python

PLk = [1+0.*t] # Force it to be a sympy expression
PMk = [1+0.*t]

for k in range(1,N):
    s = 0
    for j in range(0,k):
        s+= scalar_prod(t**k,PLk[j])/scalar_prod(PLk[j],PLk[j])*PLk[j]
    pk = t**k-s
    # pk = pk/sym.sqrt(scalar_prod(pk,pk))
    pk = pk/pk.subs(t,1.)
    PLk.append(pk)
    PMk.append(t**k)

xx = np.linspace(A,B,100)
figure()
fs = sym.lambdify(t, PMk[0], 'numpy')
plot(xx, np.ones(xx.shape)*fs(xx))
for k in range(1,N):
    fs = sym.lambdify(t, PMk[k], 'numpy')
    plot(xx, fs(xx))
figure()
fs = sym.lambdify(t, PLk[0], 'numpy')
plot(xx, np.ones(xx.shape)*fs(xx))
for k in range(1,N):
    fs = sym.lambdify(t, PLk[k], 'numpy')
    plot(xx, fs(xx))

figure()
for k in range(N):
    c = np.zeros(k)
    if k == 0:
        c = 1
    else:
        c[-1] = 1
    plot(xx, leg.legval(xx,c))

```

```python
#Best Approximation
f = sym.cos(8.*np.pi/(B-A)*t)

M_LP = np.zeros([N,N])
M_MP = np.zeros([N,N])

f_M = np.zeros([N,1])

f_L = np.zeros([N,1])

for i in range(N):
    f_M[i] = scalar_prod(PMk[i],f)
    f_L[i] = scalar_prod(PLk[i],f)
    for j in range(N):
        M_LP[i,j] = scalar_prod(PLk[i],PLk[j])
        M_MP[i,j] = scalar_prod(PMk[i],PMk[j])

p_L = np.linalg.solve(M_LP, f_L)

p_M = np.linalg.solve(M_MP, f_M)
f_p_L = 0.
f_p_M = 0.
for i in range(N):
    f_p_L += p_L[i]*PLk[i]
    f_p_M += p_M[i]*PMk[i]

fpL = sym.lambdify(t, f_p_L[0], 'numpy')
fpM = sym.lambdify(t, f_p_M[0], 'numpy')
ff = sym.lambdify(t, f, 'numpy')

plot(xx, ff(xx), 'r*')
plot(xx, fpL(xx), 'b')
plot(xx, fpL(xx), 'g')

# print M_LP
print(np.linalg.cond(M_LP))
print(np.linalg.cond(M_MP))
```

```python
N_lagrange = N
x_interp = np.linspace(A,B,N_lagrange)
x_cheby = np.array([np.cos((2.*j-1)/(2.*N_lagrange)*np.pi) for j in range(1,N_lagrange+1)])
print x_cheby
lagrange_interp = interpolate.lagrange(x_interp, ff(x_interp))
cheb_interp = interpolate.lagrange(x_cheby, ff(x_cheby))
tck = interpolate.splrep(x_interp, ff(x_interp), s=0)
yspline = interpolate.splev(xx, tck, der=0)
figure()
plot(xx, lagrange_interp(xx))
figure()
plot(xx, cheb_interp(xx))
figure()
plot(xx, yspline)

print integrate.quad(lambda x: (ff(x) - lagrange_interp(x))**2,A,B)

print integrate.quad(lambda x: (ff(x) - cheb_interp(x))**2,A,B)

print integrate.quad(lambda x: (ff(x) - interpolate.splev(x, tck, der=0))**2,A,B)

print integrate.quad(lambda x: (ff(x) - fpL(x))**2,A,B)

print integrate.quad(lambda x: (ff(x) - fpM(x))**2,A,B)
```

```python
# Root of Legendre polynomial
fs = sym.lambdify(t, PLk[8], 'numpy')
x0 = linspace(A,B,8)
# roots = optimize.brentq(fs, A, B)
roots = optimize.fsolve(fs, x0)
# roots = optimize.root(fs, x0)
print roots
plot(xx,fs(xx))
x0[1] = -0.85
x0[-2] = 0.85
roots = optimize.fsolve(fs, x0)
print roots

c = np.zeros(8+1)
c[-1] = 1
print leg.legroots(c)

xg,wg = np.polynomial.legendre.leggauss(8)

print xg
```

```python
# Gauss Integration

f_gauss = lambda x : 8*np.sin(x)+3*np.cos(15.*x)

I = integrate.quad(f_gauss,A,B)

n_gauss = 10

xg,wg = np.polynomial.legendre.leggauss(n_gauss)

In = np.dot(wg,f_gauss(xg))

error = list()
Ngmax = 40
order = np.arange(1,Ngmax)
for i in order:
    xg,wg = np.polynomial.legendre.leggauss(i)
#     In = np.dot(wg,f_gauss(xg))
    In = np.einsum('j,j',wg,f_gauss(xg))
    error.append(np.abs(I[0]-In))

# print error
plot(order, error)
figure()
loglog(order,error)
loglog(order,np.ones(Ngmax-1)*I[1])
figure()
semilogy(order, error)
semilogy(order,np.ones(Ngmax-1)*I[1])

## REMARK QUADRATURE ORDER ON (f, vi) FOR BEST APPROXIMATION
```

```python

```


# Solutions

## Question 1

> `1`. Use the class created in {ref}`objects_tutorial` to find the roots of the
>  following quadratics:

> `1`. $f(x) = -4x ^ 2 + x + 6$

First we define the class:

```python
import math

class QuadraticExpression:
    """A class for a quadratic expression"""

    def __init__(self, a, b, c):
        self.a = a
        self.b = b
        self.c = c
        self.discriminant = self.b ** 2 - 4 * self.a * self.c

    def get_roots(self):
        """Return the real valued roots of the quadratic expression"""
        if self.discriminant >= 0:
            x1 = -(self.b + math.sqrt(self.discriminant)) / (2 * self.a)
            x2 = -(self.b - math.sqrt(self.discriminant)) / (2 * self.a)
            return x1, x2
        return ()

    def __add__(self, other):
        """A magic method: let's us have addition between expressions"""
        return QuadraticExpression(self.a + other.a, self.b + other.b, self.c + other.c)

    def __repr__(self):
        """A magic method: changes the default way an instance is displayed"""
        return f"Quadratic expression: {self.a} x ^ 2 + {self.b} x + {self.c}"

class QuadraticExpressionWithAllRoots(QuadraticExpression):
    """
    A class for a quadratic expression that can return imaginary roots

    The `get_roots` function returns two tuples of the form (re, im) where re is
    the real part and im is the imaginary part.
    """

    def get_roots(self):
        """Return the real valued roots of the quadratic expression"""
        if self.discriminant >= 0:
            x1 = -(self.b + math.sqrt(self.discriminant)) / (2 * self.a)
            x2 = -(self.b - math.sqrt(self.discriminant)) / (2 * self.a)
            return (x1, 0), (x2, 0)

        real_part = self.b / (2 * self.a)
        im1 = math.sqrt(-self.discriminant) / (2 * self.a)
        im2 = -math.sqrt(-self.discriminant) / (2 * self.a)
        return ((real_part, im1), (real_part, im2))

    def __add__(self, other):
        """A special method: let's us have addition between expressions"""
        return QuadraticExpressionWithAllRoots(
            self.a + other.a, self.b + other.b, self.c + other.c
        )
```

Now we use it:

```python
f = QuadraticExpressionWithAllRoots(a=-4, b=1, c=6)
f.get_roots()
```

    ((1.356107225224513, 0), (-1.106107225224513, 0))

> `2`. $g(x) = 3x^2 - 6$

```python
g = QuadraticExpressionWithAllRoots(a=3, b=0, c=-6)
g.get_roots()
```

    ((-1.414213562373095, 0), (1.414213562373095, 0))

> `3`. $h(x) = f(x) + g(x)$

```python
h = f + g
h.get_roots()
```

    ((1.0, 0), (0.0, 0))

## Question 2

> `2`. Write a class for a Linear expression and use it to find the roots of the
>  following expressions:

>  `1`. $f(x) = 2x + 6$

First we define the class:

```python
import math

class LinearExpression:
    """A class for a linear expression a x + b"""

    def __init__(self, a, b):
        self.a = a
        self.b = b

    def get_roots(self):
        """Return the roots of the linear expression"""
        if self.a != 0:
            return -self.b / self.a
        return None

    def __add__(self, other):
        """A magic method: let's us have addition between expressions"""
        return LinearExpression(self.a + other.a, self.b + other.b)

    def __repr__(self):
        """A magic method: changes the default way an instance is displayed"""
        return f"Linear expression: {self.a} x + {self.b}"
```

Now we use it:

```python
f = LinearExpression(a=2, b=6)
f.get_roots()
```

    -3.0

> `2`. $g(x) = 3x - 6$

```python
g = LinearExpression(a=3, b=-6)
g.get_roots()
```

    2.0

> `3`. $h(x) = f(x) + g(x)$

```python
h = f + g
h.get_roots()
```

    0.0

## Question 3

> `3`. If rain drops were to fall randomly on a square of side length $2r$ the
>  probability of the drops landing in an inscribed circle of radius $r$ would
>  be given by:

>  $$
      P = \frac{\text{Area of circle}}{\text{Area of square}}=\frac{\pi r ^2}{4r^2}=\frac{\pi}{4}
  $$

>  Thus, if we can approximate $P$ then we can approximate $\pi$ as $4P$. In this
>  question we will write code to approximate $P$ using the random library.

>  First create the following class:

>  ```
>  class Drop:
>      """
>      A class used to represent a random rain drop falling on a square of
>      length r.
>      """
>
>      def __init__(self, r=1):
>          self.x = (0.5 - random.random()) * 2 * r
>          self.y = (0.5 - random.random()) * 2 * r
>          self.in_circle = (self.y) ** 2 + (self.x) ** 2 <= r ** 2
>  ```

```python
import random

class Drop:
    """
    A class used to represent a random rain drop falling on a square of
    length r.
    """

    def __init__(self, r=1):
        self.x = (0.5 - random.random()) * 2 * r
        self.y = (0.5 - random.random()) * 2 * r
        self.in_circle = (self.y) ** 2 + (self.x) ** 2 <= r ** 2
```

>  To approximate $P$ create $N=1000$ instances of Drops and count the
>  number of those that are in the circle. Use this to approximate $\pi$.

We start by creating the required number of drops:

```python
number_of_instances = 10000
random.seed(0)
drops = [Drop() for number in range(number_of_instances)]
```

Now we count the number in the circle:

```python
number_in_circle = len([drop for drop in drops if drop.in_circle])
number_in_circle
```

    7809

The number in the circle leads to the probability $P$:

```python
P = number_in_circle / number_of_instances
```

And $\pi$ can be approximated:

```python
4 * P
```

    3.1236

## Question 4

> `4`. In a similar fashion to question 3, approximate the integral
>  $\int_{0}^11-x^2\;dx$. Recall that the integral corresponds to the area
>  under a curve.

We create a different drop class changing the `in_circle` attribute to
`under_curve` and simplifying where the `x` and `y` are sampled from.

```python
class Drop:
    """
    A class used to represent a random rain drop falling on a square of
    length 1.
    """

    def __init__(self):
        self.x = random.random()
        self.y = random.random()
        self.under_curve = self.y <= 1 - self.x ** 2
```

Now we repeat the steps of question 3:

```python
number_of_instances = 10000
random.seed(0)
drops = [Drop() for number in range(number_of_instances)]
```

Now we count the number in the circle:

```python
number_under_curve = len([drop for drop in drops if drop.under_curve])
number_under_curve
```

    6646

In this particular problem the area of the square is 1 so the probability of
being under the curve is equal to the 1: $P=\frac{\int_{0}^11-x^2\;dx}{1}$.

```python
number_under_curve / number_of_instances
```

    0.6646

We can confirm this:

```python
import sympy as sym

x = sym.Symbol("x")
sym.integrate(1 - x ** 2, (x, 0, 1))
```

$\displaystyle \frac{2}{3}$