# Compositions of Functions

## Compositions of Univariate Functions

Compositions of functions sound a lot worse than they are. The most basic way to describe them is as a "function of a function". As with most mathematical language, that description isn't particularily enlightening. So, let's start with an example, an example so simple it risks annoying a pedantic mathematician. Which is how we know that it is an excellent example. Suppose you have two functions $f(x)$ and $g(x)$ defined as follows

\begin{equation}
 f(x) = x
\end{equation}

and
\begin{equation}
g(x) = 1
\end{equation}

While keeping these simple functions in mind, let's work our way through a few definitions. A composition of functions defined as

\begin{equation}
 f \circ g = f[g(x)]
\end{equation}

where this is read as $f$ of $g$ of $x$. This notation is telling us to write $f(x)$ as a function of $g(x)$, so rather than a function of a variable, it's becoming a function of a function. With the functions above however, this works out to be exactly the same as putting a number into your function, which we see as follows

\begin{equation}
f[g(x)] = \underset{\substack{\downarrow \\ \text{Inserted g(x) at x}}}{g(x)} = 1
\end{equation}

Great, so a composition of functions reduces to the case of inserting a constant in for your variable. But what about if you had more complex functions? Well, consider the functions
\begin{equation}
 f(x) = x + 5
\end{equation}

and

\begin{equation}
g(x) = x^2 -4
\end{equation}

How will we write $f \circ g$ with these functions? It's actually just as simple as putting numbers in for the variables like we saw above, except this time, we'll put the definitions of functions in for each variable.

\begin{equation}
f[g(x)] =\underset{\substack{\downarrow \\ \text{Inserted g(x) at x}}}{g(x)} +
\overset{\substack{\text{Constants just hang out} \\ \uparrow} }{5} =
\underbrace{x^2 - 4}_\text{Insert definition of g(x) for x} + 5 = x^2 + 1
\end{equation}

or going the other direction

\begin{equation}
g[f(x)] = f^2(x) - 4 = (x+5)^2 -4
\end{equation}

where this applies to any composition of functions. To put it simply, if you have a composition of functions all you need to do is place the "inner" function everywhere you see the variable in the "outer" function exactly as you'd put a number into the formula.

Python provides us with some great tools in order to graph these functions easily. If you have some knowledge of Python you can easily skip this explanation and move right on to the next cell. If not, we're going to take a moment to understand what the code you're about to see does. First up, let's go over the first bit of code you're going to see.
## Digression: Basic Python

### Imports
At the beginning of most Python programs you're more than likely to see a few (or many) `import` statements. The purpose of these is to bring in other pieces of code, that either you or someone else have written, in order to keep your current program more manageable. For example, the set of import statements we're going to use to plot our functions look like this. Select the code cell below then press the `▶Run` button in the toolbar.

```python
#import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
print('Libraries succesfully imported.')
```

Starting from the top down, let's go over what each piece of this code does.

* **import matplotlib**:

* **import matplotlib.pyplot as plt**: This command tells Python to import the graphing subroutines from the `matplotlib` package and assign them the name `plt` so we don't have to type as much when we want to produce a graph.

* **%matplotlib inline:** This is specific to Jupyter notebooks. It tells matplotlib to output graphs in the cell directly below the one in which the code is executed. For more information about Jupyter "magic commands" feel free to read [this document](http://ipython.readthedocs.io/en/stable/interactive/magics.html).

* **import numpy as np**: This imports the Python package `numpy` or "numerical Python" and assigns the name to it within our code to `np` so we don't have to type out `numpy` every time we need a function. We'll be using this package for mathematical functions like sine and the square root.

### Plotting

After required modules have been imported into our script, we're ready to get started creating graphs. First up, we need to define a number of points which to plot. It is an unfortunate fact that our computer doesn't know that any variable $x$ is continuous, so we have to create some discrete data with which to plot our functions. We can do that in Python using numpy as follows (Click the next cell and press control and enter at the same time on your keyboard.

```python
x = np.linspace(-3,3,5)
print("x = ", x)
```

This creates a **list** of numbers called `x` which consists of 5 numbers evenly spaced in the range $[-3,3]$ that we'll use to plot our functions. The `print` function is a standard Python function that simply displays our variables to the screen as either numbers or characters. In order to plot our functions, we type the following. Note that we've increased the number of points in `x` to create a smoother plot. Feel free to change the number `500` to something smaller and observe how the plot changes.

```python
x = np.linspace(-3,3,500)
plt.plot(x, x**2-4, label='$g(x) = x^2 -4$')
```

Where there's actually a fair bit in the above line of code that we should take note of. First, by using `plt.plot` we're calling a function from `matplotlib.pyplot` that we called `plt` called `plot`. This function, unsurprisingly, is used to tell Python what to plot. We then pass this function a number of arguments. The first argument `x` is the list of numbers we generated earlier. The second argument `x**2 - 4` is the mathematical function we're going to plot. Here `x` is the variable we're going to plot, and also the list of points that we generated earlier. The `**` is the Python way of "the power of". So, what we're really saying here is $x^2-4$. Finally, the `label='$g(x) = x^2 -4$` is the label we're creating for the plot legend so we can distinguish our plots. The dollar signs are simply formatting so our plot looks nice.

Now, that graph is missing a lot of the important things like axis labels and a legend. We can add those like this

```python
plt.plot(x, x**2-4, label='$g(x) = x^2 -4$')
plt.xlabel("$x$", fontsize=16)
plt.ylabel("$f(x)$", fontsize=16)
plt.legend()
plt.margins(0)
plt.show()
```

where the `plt.` calls are still calling functions from `matplotlib.pyplot` however this time we're creating x-axis labels with `plt.xlabel` and y-axis labels with `plt.ylabel`. Finally `plt.legend()` creates the legend for our plot and `plt.show()` displays the plot of our functions. We can also do this all in one cell, rather than breaking it up, as shown below, as well as adding an additional function to plot.

```python
x = np.linspace(-3,3,500)
plt.plot(x, x**2 - 4, label = '$g(x) = x^2 -4$')
plt.plot(x, x+5, label= '$f(x) = x + 5$')
plt.xlabel("$x$", fontsize=16)
plt.ylabel("$f(x)$ and $g(x)$", fontsize=16)
plt.margins(0)
plt.legend()
plt.show()
```

We now have two functions: one linear and one parabolic. But what does the composition of these two functions look like? Well, let's use Python to graph their composition! Note that we're adding a few more functions to plot, but the basic plotting is identical. As well, we've already imported the packages we need, so we don't have to do it again!

### Question:
1. Do you understand what each line of code below is doing?

```python
# Anything in a line of code after a pound sign isn't run, and is known as a comment.

# First let's plot our original functions again
plt.plot(x, x**2 - 4, label = '$g(x) = x^2 -4$')
plt.plot(x, x+5, label= '$f(x) = x + 5$')
# And now the new functions
plt.plot(x, (x+5)**2 - 4, label = '$g[f(x)] = (x+5)^2 - 4$')
plt.plot(x, x**2 + 1, label = '$f[g(x)] = x^2 + 1$')
plt.xlabel("$x$", fontsize=16)
plt.ylabel("Value of Functions of $x$", fontsize=16)
plt.margins(0) # so there is no extra space on the plot
plt.legend()
plt.show()
```

Where we see from the plot above that depending on the order of the function of functions we get a very different plot! In one case, we have a parabola that shifts up the y-axis by four, and in the other case we get a much wider parabola!

**Do you understand why?**

*Hint: what is the affect of constants on polynomial functions? How about if the constant is inside parentheses?*

## More Complicated Examples

This can also be applied to more complicated compositions of functions. For example, can you have a function of function functions? Absolutely! Like anything in math, you can do whatever you want provided you have the patience to work through it. But just as an example, what if we had three functions, say $f(x)$, $g(x)$ and $h(x)$ which are defined as follows
\begin{equation}
 f(x) = 6\sqrt{x^2 + 1}
 \end{equation}

\begin{equation}
g(x) = 3\sin(\pi x - a)
\end{equation}

\begin{equation}
h(x) = x^3 + x^2
\end{equation}

where $a$ is an arbitrary numerical constant. How would we go about writing $f \circ g \circ h$ ($f$ of $g$ of $h$)? It certainly looks intimidating, but I promise you it looks a lot worse than it actually is. Let's follow our typical PEDMAS rules, in this case, we start with the only thing we have: parenthesis! And, because we have two sets, we start with the inner most set. In this case, let's solve the smaller problem of $g[h(x)]$ where we'll do the same as above, and simply shove $h(x)$ anywhere we see an $x$ in $g(x)$. This is outlined below, however we've dropped some of the explaination with the algebra, but, we're doing absolutely nothing different from what we saw in the previous examples.

\begin{equation}
g[h(x)] = \sin(\pi h(x) - a) = \sin(\pi(x^3 + x^2) - a)
\end{equation}

where we notice that the constant term $a$ simply comes along for the ride. We're half way done! Now that we have an expression for $g[h(x)]$, we can now easily find our complete expression of $f[g[h(x)]]$ in the _exact_ same way we have been doing.

\begin{equation}
f\circ g \circ\ h= f[g[h(x)]] = \sqrt{g^2[h(x)] + 1} = 6\sqrt{\sin^2[\pi (x^3 + x^2) - a] + 1}
\end{equation}

Where that's a pretty ugly function, but the process of coming to it was not nearly as horrible as that expression looks.

Where we just have to remember our one rule for compositions of univariate functions: shove the inner function anywhere you see the variable in the outer function, exactly as you would shove a number into a formula, and follow PEDMAS rules.

Let's take a look at how these functions and their compositions look graphically

```python
# Let's start by taking a look at our basic functions

# here np.sqrt is the square root function.
plt.plot(x,6*np.sqrt(x**2 + 1), label = '$f(x) = 6\sqrt{x^2+1}$')
plt.plot(x, 3 *np.sin(np.pi * x-1), label="$g(x,1) = \sin(\pi x - 1)$")
plt.plot(x, x**3 + x**2, label = "$h(x) = x^3 + x^2$")
plt.xlabel("$x$", fontsize=16)
plt.ylabel("Value of Functions of $x$", fontsize=16)
plt.margins(0)
plt.legend()
plt.show()
```

Where these functions are certainly a little more exciting than they were before, but in order to plot these compositions of functions, we're going to have to do a lot of typing! Worse than that, a lot of very redundant typing that is prone to error. Is there a better way to plot these functions, that is both more accurate and won't take us as long? The answer is absolutely! We just have to define a function within Python that represents our functions that we wish to plot and compose.

### Python Digression

In order to define a function in Python we use a key word known as `def`, which for us stands for "define a function". Let's work through the steps we would go through in order to define the function

\begin{equation}
f(x) = 6\sqrt{x^2 + 1}
\end{equation}

in Python. We can do this using the following snippet of code

```python
def f(x):
    return 6 * np.sqrt(x**2 + 1)
print("f(12) =", f(12))
```

where here we've used the Python keyword `def` and given the function we're defining a name "`f`". We've also told Python that our function named `f` takes one variable as input which we've called "`x`". We also have used keyword `return`, which for us can be though of as an equals sign: we pass a number in to `f` and the function will return what `f` is equal to at that number. Another important thing to notice is that `return` is indented relative to `def`. This is how Python knows that `return` and the statement after are part of the function. Every time you write a function, any code inside it will have to be consistently indented. The choice of function and variable names are completely arbitrary. A function defined like

```python
def A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(A_Really_Long_Variable_Name):
    return 6 * np.sqrt(A_Really_Long_Variable_Name**2 + 1)

print("A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(12) =" ,
      A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(12))
```

behaves exactly the same as our function `f`.

Another important fact about functions is that the value we pass this function need not be named `x`. We can pass it a number we've named `NickNackPattyWack` if we wanted to, it will behave the same. Shown explicitly below with a simple example where the functions take a value as input, and return that value divided by ten.

```python
# note: you must always define a function before you use it.

# Check out how this function is written differently but behaves the same.
# Also pay attention to the consistent indentation within the function
def f(x):
    Divisor = 10
    Result = x /10
    return Result

def A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(A_Really_Long_Variable_Name):
    return A_Really_Long_Variable_Name/10

number = 4
NickNackPattyWack = 4

print("The results are below")
print("f(",number,") =", f(number))
print("f(", NickNackPattyWack, ") =", f(NickNackPattyWack))
print("A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(", number, ") =",
      A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(number))
print("A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(", NickNackPattyWack, ") =",
      A_Really_Bad_Name_For_A_Function_That_Takes_Forever_To_Type(NickNackPattyWack))
```

Where I've rudely introduced Python's `print` function without telling you about it first. This function outputs numbers and text to the screen. But the point is, that these functions do the same thing even with different names and different numbers.

As an additional note, what I've shown here is very simple. Python functions can get considerably more involved, as you can have any amount of code between the `def` statement and the `return` statement. A function can do a lot of great things for you, and I'll hopefully touch on that later in an appendix.

Now that we've gone over defining functions, let's define the functions we've specified earlier in this notebook.

```python
def f(x):
    return 6*np.sqrt(x**2 + 1)

# Note here we've given our function an extra argument 'a'
# this is the arbitrary constant we discussed before, but we'll pass it any value we want!

# np.pi is simply the value of pi, or 3.1415962.... from the numpy package.
def g(x,a):
    return  3 *np.sin(np.pi * x-a)

def h(x):
    return x**3 + x**2
```

Great! Let's see if these still look the same if we plot them before moving on to seeing how their compositions look

```python
# Check out how convenient it is to not type our the expression every time!
plt.plot(x,f(x), label = '$f(x) = 6\sqrt{x^2+1}$')

# Also note that we're setting the value of 'a' equal to 1 for this plot.
plt.plot(x, g(x,a=1), label="$g(x,1) = \sin(\pi x - 1)$")

plt.plot(x,h(x), label = "$h(x) = x^3 + x^2$")
plt.xlabel("$x$", fontsize=16)
plt.ylabel("Value of Functions of $x$", fontsize=16)
plt.margins(0)
plt.legend()
plt.show()
```

Where the plot above is identical to the one we saw before! It works! Now we'll be able to see the power of both the composition of function notation $f[g(x)]$ and Python functions!

### Back to the Math

Now that we can make graphs in a more laz... efficient way, let's see what the compositions of these functions look like graphically. There's a handy feature about compositions of functions that behave the same way mathematically in Python as they would if you were to solve the composition by hand. For example, with the Python functions we have defined above, all we need to do to find $f[h(x)]$ where $x=1$ for example is

```python
number = 1
print("f[h(x)] =",f(h(number)))
```

Where this follows traditional PEDMAS like rules. Essentially the value of $h(x)$ gets passed in as the variable to $f(x)$ and then it evaluates that new expression at `Number = 1`. This evaluates to the same answer that you would get should you have done the algebra yourself. Let's take a look at how these compositions of functions look like. However we have decreased the range of the function to show from [-3,3] to [-1.5,1.5].

```python
# Look how much easier this is than typing the functions out every time!
plt.plot(x, g(h(x),a=1), label = "$g[h(x),1]= \sin(\pi(x^3 + x^2) - a)$")
plt.xlabel("$x$", fontsize=16)
plt.ylabel("Value of Functions of $x$", fontsize=16)
plt.xlim([-1.5,1.5])
plt.margins(0)
plt.legend()
plt.show()
```

That function is beginning to look pretty complicated, which is unsurprising considering the non-linearity of the functions we have chosen to combine. However, look how simple that was to plot in Python. Rather than having to solve the composition of functions ourselves, we told Python, we took advantage of Python functions and let the computer to do it for us! Convenient! Let's take a look at how these compositions of functions looking graphically.

```python
plt.plot(x, g(h(x),a=1), label='$g[h(x),1]$')

# We don't need to explicitly state 'a=1', we can put the value in the correct argument location

plt.plot(x, h(g(x,1)), label = '$h[g(x,1)]$')
plt.plot(x, f(g(x,1)), label= '$f[g(x,1)]$')
plt.plot(x, f(h(x)), label ='$f[h(x)]$')
plt.plot(x, g(f(x),a=1), label ='$g[f(x),1]$')
plt.xlabel("$x$", fontsize=16)
plt.ylabel("Value of functions", fontsize=16)

# This changes the x-axis range
plt.xlim([-1.5,1.5])

# This changes the y-axis range
plt.ylim([-25,50])

# This moves the legend from its default position to something more appropriate
plt.legend(bbox_to_anchor=(1.0, .5))

plt.show()
```

Where that's pretty cool how easily we can view the graphical behavior of compositions of these functions without defining them explicitly!
#### Exercise
1. Find (by hand) and plot the resulting function (by defining your own function) of $g[h(x),1]$, does it look like the plot above?

2. Find (by hand) and plot the resulting function (by defining your own function) of  $h[g(x,1)]$, does it look like the plot above?

3. Find (by hand) and plot the resulting function (by defining your own function) of  $f[g(x,1)]$, does it look like the plot above?

4. Find (by hand) and plot the resulting function (by defining your own function) of  $f[h(x)]$, does it look like the plot above?

5. Find (by hand) and plot the resulting function (by defining your own function) of  $g[f(x),1]$, does it look like the plot above?

## Operations of Functions

Operations of functions, for practical purposes, mostly a mathematical short hand. It's easier to see and avoid your mistakes by writing as little possible, so operations of functions are simply a small abstraction which prevents you from having to write as many things down, and reducing the chance of making a silly mistake.

Suppose you want to add two functions together, with as little writing as possible. Well, let's introduce the notation of function operations! Suppose we have two functions

\begin{equation}
f(x) = x^2
\end{equation}

and

\begin{equation}
g(x) = 2x + 10
\end{equation}

How would we add those together? Certainly we _could_ write

\begin{equation}
f(x) + g(x) = x^2 + 2x + 10
\end{equation}

but what if those functions were gigantic? In this case we have the following short-hand notation:

\begin{equation}
(f + g)(x) = f(x) + g(x) = x^2 + 2x + 10
\end{equation}

similarly for subtraction:
\begin{equation}
(f - g)(x) = f(x) - g(x) = x^2 -2x - 10
\end{equation}

I know what you're thinking: addition and subtraction are great, but I'm a bit of a mathematical sauvant, and I would also like to multiply and divide functions. We'll I've got news for you kiddo: we've got a short-hand for that too.

Multiplication:
\begin{equation}
(f \times g)(x) = f(x)g(x) = x^2(2x + 10)
\end{equation}

Division:
\begin{equation}
\left(\frac{f}{g}\right)(x) = \frac{f(x)}{g(x)} = \frac{x^2}{2x + 10}
\end{equation}

This shorthand behaves exactly like regular math, but you "map" your variable back onto the function afterwards, certainly you could even stop writing the variable completely. Of course, a natural question now arises: how is this useful? Suppose you have three functions

\begin{aligned}
f(x) = & x^2 \sin^2(x) + x^2 \\
g(x) = & \frac{x^3\log_{10}(x) + 10^x}{10} \\
h(x) = & x^2\cos^2 + x^2
\end{aligned}

First, let's define them in Python so we can observe how these functions behave

```python
def f(x):
    return x**2. * (np.sin(x)**2. + 1.)

def g(x):
    return (x**3. * np.log10(x) + 10.**x)/10.

def h(x):
    return x**2. * (np.cos(x)**2 + 1.)

x = np.linspace(0.01,2.5,1000)
plt.plot(x, f(x), label = "$f(x)$")
plt.plot(x, g(x), label = "$g(x)$")
plt.plot(x, h(x), label = "$h(x)$")
plt.legend()
plt.margins(0)
plt.xlabel("$x$")
plt.ylabel("Functions of $x$")
plt.show()
```

Where those functions seem to be pretty well behaved. However, what if we wanted to find an expression for

\begin{equation}
y=\frac{f(x)g(x) + h(x)g(x)}{g(x) \left([f(x) + 1]^2 + [h(x) +1]^2 - f^2(x) - h^2(x) - 2\right)^2}
\end{equation}

where that looks pretty intimidating, I certainly wouldn't want to put the definitions of my functions into the expression for $y$. But, let's use operations of functions to see if we can simplify that expression before we evaluate it.

\begin{aligned}
&\text{Factor the numerator} \\
y &=\frac{g \left(f + h \right)}{g \left([f + 1]^2 + [h +1]^2 - f^2 - h^2 - 2)\right)^2} \\
\\
&\text{Expand denominator} \\
y &=\frac{g \left(f + h \right)}{ g \left(f^2 +2f + 1 + h^2 + 2h + 1 -f^2 -h^2 - 2  \right)^2} \\ \\
&\text{Collect terms and simplify} \\
y & = \frac{f + h}{ (2f + 2h) ^2} = \frac{f+h}{4(f+h)^2} \\ \\
&\text{Cancel common terms}\\
y & = \frac{1}{4}\frac{1}{f + h} \\ \\
&\text{Insert Function definitions}\\
y &= \frac{1}{x^2 \sin^2(x) + x^2 + x^2 \cos^2(x) + x^2} \\ \\
&\text{Simplify} \\
y&=\frac{1}{4}\frac{1}{x^2( \sin^2(x) + \cos^2(x) + 2)} \\ \\
&\text{Pythagorean Theorem} \\
y&= \frac{1}{12x^2}
\end{aligned}

While we still had to do a little algebra, by using operations of functions we saved ourselves a considerable amount of time working through that simplification, and in fact found that our solution didn't depend on $g(x)$ at all! Although that looks all too convenient, let's test if this lines up using Python to double check our expression. First, let's define a helper function to handle that nasty composition of functions.

```python
def comp(x):
    numerator = g(x)*(f(x) + h(x))
    denominator = (f(x) + 1.)**2. + (h(x) + 1. )**2. - f(x)**2. - h(x)**2. - 2.
    denominator = g(x) * denominator**2.
    return numerator/denominator

def simplified(x):
    return 1. /(12.* x **2.)

x = np.linspace(0.5,10,1000)
plt.plot(x, comp(x), label = "comp")
plt.plot(x, simplified(x), label = "Simplified")
plt.legend()
plt.xlabel("$x$")
plt.ylabel("Function Value")
plt.show()
```

Where that looks pretty exact! But, if Python is happy to use our unsimplified functions, why bother with all that algebra in the first place? Well, let's take a look at how long it takes to make that calculation using each function. Run the cell below to time how long it takes each function to calculate a number. Note this cell takes some time to run.

```python
# import the time library
import time

# create empty lists we'll use for plotting
x_points = []
y_points = []

# This runs the code below it 100 times, from i=1 to i=99
for i in range(1, 100):
    # Record current time
    start1 = time.time()
    for b in range(i * 100):
        a = comp(i)
    # record end time
    end1 = time.time()

    # calculate total time
    total1 = end1 - start1

    # time our second function
    start = time.time()

    for b in range(i * 100):
        a = simplified(i)

    end = time.time()
    total2 = end - start
    x_points.append(total1/total2)
    y_points.append(i* 100)

plt.plot(x_points,y_points)
plt.xlabel("Ratio of time spent between the simplified and composite function")
plt.ylabel("Number of Points Calculated")
```

Where we see that the unsimplified function is about _fifty times_ slower than our simplified version. This is very important in many computations because you don't want to spend too long waiting for an answer. So even though your computer will calculate the value of a composition of functions, knowing how to simplify those functions by hand is still incredibly important.

## Conclusion

In summary, operations of functions come down to making a substitution: rather than manipulating the variables individually, we manipulate the entire function first to see if we can make things easier on ourselves. We can also use those compositions of functions to visualize them well in Python and observe their complex behaviour. While a lot of the time they look complicated, the process deriving the composition of functions is not very difficult.

## Real Life Examples

Compositions of functions are used everyday in physics and engineering. Have you ever made a substitution in your analysis? Well, then you've been using compositions of functions! Let's work through an example. For example, let's say we wanted to calculate the kinetic energy of an object undergoing uniform circular motion. I state without proof that the formula for kinetic energy is

$T(v) = \frac{1}{2} m v^2.$

Where $T$ is the kinetic energy, $v$ is the _velocity_ of the object, and $m$ is its mass. For circular motion the formula for velocity is

$v(\omega) = \omega R,$

where $\omega$ is the angular velocity, or how quickly the angle changes as a function of time, and $R$ is the radius of the circle. If you wanted to get the kinetic energy of a particle in uniform circular motion, we do what we've already been doing! Composition of functions, looking for $T(v(\omega))$, or the kinetic energy as a function of angular velocity (or kinetic energy of velocity of angular velocity).

$T(v(\omega)) = \frac{1}{2} m \left(\omega R \right)^2 = \frac{1}{2} m \omega^2 R^2.$

Essentially, if you ever hear the phrase "make a substitution" you should be thinking "composition of functions".

[](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)


# Introduction

## Karatsuba Multiplication(卡拉楚巴算法)

一种快速乘法算法，使两个 $n$ 位数字相乘所需的一位数乘法次数减少到至多 $3n^{log_23}\approx3n^{1.585}$ 次。

### 基本步骤

运用递归思想，将位数很多的两个大数 $x$ 和 $y$ 分成位数较少的数，每个数都是原来 $x$ 和 $y$ 位数的一半。这样处理后，简化为做三次乘法，并附带少量的加法操作和移位操作。

$$
\begin{align}
x \cdot y \, & = (10^{\frac n2} \cdot a + b) \times (10^{\frac n2} \cdot c + d) \\[2ex]
             & = 10^n \cdot ac + 10^{\frac n2} \cdot (ad + bc) + bd \tag {*} \\[2ex]
\end{align}
$$

> ${\it {e.g.}} \quad$ $x = 1234, \; y = 5678$，则此时 $a = 12, \; b = 34, \; c = 56, \; d = 78$

`*` 号算式可以进行进一步简化：

$$
\begin{align}
\because \qquad \; \; (a + b) \cdot (c + d) &= ac + ad + bc + bd \\[2ex]
\therefore \qquad \qquad \qquad ad + bc &= (a + b) \cdot (c + d)- ac - bd \\[2ex]
\end{align}
$$

所以我们只需递归计算出 $ac$、$bc$ 和 $(a + b) \cdot (c + d)$ 的值，即可计算出 $x \cdot y$ 的值。

```python
def karatsuba(x: int, y: int):
    if x < 10 or y < 10: return x * y # 递归边界

    x_str, y_str = str(x), str(y)
    # 处理负数
    if x_str[0] == '-': return -karatsuba(-x, y)
    if y_str[0] == '-': return -karatsuba(x, -y)
    # 补0
    max_length = max(len(x_str), len(y_str))
    x_str = f'{x_str:0>{max_length}}'
    y_str = f'{y_str:0>{max_length}}'

    split_positon = max_length // 2
    a = int(x_str[:-split_positon]) # x 高位
    b = int(x_str[-split_positon:]) # x 低位
    c = int(y_str[:-split_positon]) # y 高位
    d = int(y_str[-split_positon:]) # y 低位
    z0 = karatsuba(a + b, c + d) # 计算 (a+b)(c+d)
    z1 = karatsuba(a, c) # 计算 ac
    z2 = karatsuba(b, d) # 计算 bd

    return z1 * 10 ** (2 * split_positon) + \
           (z0 - z1 - z2) * 10 ** (split_positon) + \
           z2

karatsuba(1234, 5678)
```

    7006652

___

# Divide and Conquer(分治法)

## Merge Sort(归并排序)

### Motivation and Example

$$
\def\arraystretch{1.5}
\begin{array}{c} % 总表格
    \begin{array}{c} % 第一行一列
        \begin{array}{|c|c|c|c|c|c|c|c|}
            \hline
            5 & 4 & 1 & 8 & 7 & 2 & 6 & 3 \\
            \hline
        \end{array}
    \end{array}
    \\
    \begin{array}{cccc} % 第二行两列(两个箭头)
        \dArr & \quad & \quad & \dArr
    \end{array}
    \\
    \begin{array}{cc} % 第三行两列(分割出的两部分)
        \begin{array}{|c|c|c|c|}
            \hline
            5 & 4 & 1 & 8 \\
            \hline
        \end{array}
        &
        \begin{array}{|c|c|c|c|}
            \hline
            7 & 2 & 6 & 3 \\
            \hline
        \end{array}
    \end{array}
\end{array}
$$

Divide and Conquer, Sorting and Searching, and Randomized Algorithms/Week 1.ipynb


# OVERLAPPING GENERATIONS MODEL

Imports and set magics:

```python
import numpy as np
from scipy import optimize
import sympy as sm

# autoreload modules when code is run
%load_ext autoreload
%autoreload 2

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

# Model description

We consider the standard **Overlappen Generations Model** (OLG-model).

Individuals live for two periods, t = 1,2. $L_t$ individuals are born in period t and we assume that the population grows with a constant rate, n:

$$ L_t = L_{t-1}(1+n) $$

Agents derive utility while alive:

$$ U_t = u(c_{1t})+\frac{1}{1+\rho} u(c_{2t}) $$

where $c_{1t}$ is consumption when the indidual is young at time t, $c_{2t+1}$ is consumption when the individual is old at time t+1 and $\rho$ is the discount rate.

We assume that agents derive utility from consumption and that utility of consumption is logarithmic:

$$ u(c) = log  c $$

The agents' budget constraint in each period is given by:

$$ c_{1t} + s_t = w_t $$

$$ c_{2t+1} = (1+r_{t+1}) s_t $$

where it is assumed that all individuals work when young and consume from savings when old. $w_t$ is the wage receiving when young at time t and $s_t$ is savings at time t.

This implies that agents' lifetime budget constraint is given by:

$$ c_{1t} + \frac{c_{2t+1}}{1+r_{t+1}} = w_t $$

Moreover, firms produce output using two inputs; capital $K$ and labor $L$. We assume that the production function is Cobb-Doulgas:

$$ F(K,L) = K^\alpha L^{1-\alpha},      \alpha \in (0,1) $$

# Analytical solution

We use sympy to find the Euler, optimal savings, capital accumulation and capital in steady state.

First we define all **symbols**:

```python
c_1t = sm.symbols('c_1t')
c_2t = sm.symbols('c_2t+1')
rho = sm.symbols('rho')
u_1 = sm.symbols('u_1')
u_2 = sm.symbols('u_2')
sigma = sm.symbols('sigma')
w = sm.symbols('w_t')
r_t1 = sm.symbols('r_t+1')
lamb = sm.symbols('lambda')
s = sm.symbols('s_t')
k_2t = sm.symbols('k_t+1')
k_1t = sm.symbols('k_t')
alpha = sm.symbols('alpha')
n = sm.symbols('n')
k_star = sm.symbols('k^*')
Y = sm.symbols('Y_t')
K = sm.symbols('K_t')
L_t = sm.symbols('L_t')
r = sm.symbols('r_t')
```

## Household problem

Every agent born at time t maximize utility subject to their lifetime budget constraint.

We start out by defining lifetime utility U and lifetime budget constraint:

```python
# Lifetime utility
u_c1 = sm.ln(c_1t)
u_c2 = sm.ln(c_2t)
U = u_c1 + u_c2/(1+rho)
U
```

$\displaystyle \log{\left(c_{1t} \right)} + \frac{\log{\left(c_{2t+1} \right)}}{\rho + 1}$

```python
# Lifetime budget constraint
budget= c_1t + c_2t/(1+r_t1) - w
budget
```

$\displaystyle c_{1t} + \frac{c_{2t+1}}{r_{t+1} + 1} - w_{t}$

In order to characterize the household problem we set up the Lagrangian, L:

```python
# Lagrangian
L = U - lamb * budget
L
```

$\displaystyle - \lambda \left(c_{1t} + \frac{c_{2t+1}}{r_{t+1} + 1} - w_{t}\right) + \log{\left(c_{1t} \right)} + \frac{\log{\left(c_{2t+1} \right)}}{\rho + 1}$

We find the FOCs, substitute out the Lagrangian multiplier and find the OLG-version of the Euler equation:

```python
def Langrarian(x):
    L_1 = sm.diff(L, x)
    L_2 = sm.Eq(L_1,0)
    L_3 = sm.solve(L_2, lamb) [0]
    return L_3

# FOCs
L_c1 = Langrarian(c_1t)
L_c2 = Langrarian(c_2t)

# Set up the Euler
Eu = sm.Eq(L_c1, L_c2)
Eu = sm.solve(Eu, c_1t) [0]
Euler = sm.Eq(c_1t, Eu)
Euler
```

$\displaystyle c_{1t} = \frac{c_{2t+1} \left(\rho + 1\right)}{r_{t+1} + 1}$

In order to find optimal savings, we now substitute the budget constraint for each period into the Euler equation and solve for $s_t$:

```python
# Define budget constraints for each period
budget_1 = w-s
budget_2 = (1+r_t1)*s

# Substitution
savings_1 = Euler.subs(c_1t,budget_1)
savings_2 = savings_1.subs(c_2t, budget_2)

# Find optimal savings
savings = sm.solve(savings_2,s)[0]
opt_savings = sm.Eq(s,savings)
opt_savings
```

$\displaystyle s_{t} = \frac{w_{t}}{\rho + 2}$

We see that individuals save a constant fraction of income.

## Firms maximization problem

Firms compete and take rental rates and wages as given. Thus, the representative firm maximizes profits:

$$ max  F(K_t,L_t) - r_tK_t - w_tL_t $$

In order to find the wage rate and the rental rate, we find the FOCs with respect to $K_t$ and $L_t$:

```python
#Define production function
Y = K**alpha*L_t**(1-alpha)
```

```python
# Find rental rate
r_t = sm.diff(Y,K)
r_t = sm.simplify(r_t)
r_t
```

$\displaystyle K_{t}^{\alpha - 1} L_{t}^{1 - \alpha} \alpha$

```python
# Find wage
w_t = sm.diff(Y,L_t)
w_t = sm.simplify(w_t)
w_t
```

$\displaystyle K_{t}^{\alpha} L_{t}^{- \alpha} \left(1 - \alpha\right)$

We rewrite this in order to express the rental rate and the wage per worker, such that $\frac{K}{L} = k$:

```python
rr = sm.Eq(r,alpha*k_1t**(alpha-1))
rr
```

$\displaystyle r_{t} = \alpha k_{t}^{\alpha - 1}$

```python
wage = (1-alpha)*k_1t**alpha
ww = sm.Eq(w,(1-alpha)*k_1t**alpha)
ww
```

$\displaystyle w_{t} = k_{t}^{\alpha} \left(1 - \alpha\right)$

## Capital accumulation and steady state

We now substitute $w_t$ into optimal savings:

```python
savings_eq= opt_savings.subs(w,wage)
savings_eq
```

$\displaystyle s_{t} = \frac{k_{t}^{\alpha} \left(1 - \alpha\right)}{\rho + 2}$

Aggregate capital accumulation is given by:

$$k_{t+1}(1+n)=s_t$$

which we insert in optimal savings to find the capital accumulation equation:

```python
capital_acc = savings_eq.subs(s,(1+n)*k_2t)
capital_acc = sm.solve(capital_acc,k_2t) [0]
capital_acc = sm.Eq(k_2t,capital_acc)
capital_acc
```

$\displaystyle k_{t+1} = \frac{k_{t}^{\alpha} \left(1 - \alpha\right)}{n \rho + 2 n + \rho + 2}$

In steady state we have that $k_t=k_{t+1}=k^*$.

We insert this in the capital accumulation equation and solve for $k^*$:

```python
capital_eq=sm.Eq(k_star*(1+n), (((1-alpha)*k_star**alpha)/(rho+2)))

# Solve for k_star
SS = sm.solve(capital_eq,k_star)[0]
SS = sm.Eq(k_star,SS)
SS
```

$\displaystyle k^{*} = \left(\frac{1 - \alpha}{n \rho + 2 n + \rho + 2}\right)^{- \frac{1}{\alpha - 1}}$

# Dynamics

We now solve the OLG-model numerically and look at the dynamics of capital accumulation.

Define the model **parameters**:

```python
rho = 0.05
n = 0.02
alpha = 0.33
```

Define functions:

```python
def wage(k,alpha):
    return (1-alpha)*k**alpha

def rr(k,alpha):
     return alpha*k**(alpha-1)

def capital_2t(s,n):
    return s/(1+n)

def savings(w,rho):
    return w/(2+rho)

def transition(k,alpha,rho,n):
    w = wage(k,alpha)
    s = savings(w,rho)
    return capital_2t(s,n)
```

We define a function for the transition curve, which returns $k_t+1$ for values of $k_t$.

```python
def transition_curve(alpha,rho,n):

    # Grids
    capital_1 = np.linspace(1e-8,1,500)
    capital_2 = np.empty(500)

    # Solve for each k in grid
    for i,k in enumerate(capital_1):

        # Capital in t+1
        result = transition(k,alpha,rho,n)
        capital_2[i] = result

    return capital_1,capital_2
```

We call the function and plot the dynamics of the model

```python
# Call transition curve function
capital_1,capital_2 = transition_curve(alpha,rho,n)

# Plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(capital_1,capital_2)
ax.plot(capital_1,capital_1, '--', color='grey')
plt.text(0.26,0.19,'$k_t+1$')
plt.text(0.27,0.25,'$k_t=k_t+1$')
plt.text(0.18,0.19,'$k^*$')
ax.set_xlabel('$k_t$')
ax.set_ylabel('$k_t+1$')
ax.set_title('Transition curve')
ax.set_xlim([-0.001,0.3])
ax.set_ylim([0,0.3]);
```

The transition curve and the 45 degrees curve representing $k_t+1 = k_t$ intersect at the steady state level of capital $k^*$.

# Dynamics with CRRA utility

We now introduce CRRA utility, where $u(c)$ is given by the CRRA utility function:

\begin{equation}
  u(c)=\begin{cases}
    \frac{c^{1-\sigma}-1}{1-\sigma} , & \sigma ≠ 1 \\
    log (c), & \sigma = 1
  \end{cases}
\end{equation}

where $\sigma$ is the degree of relative risk aversion, which is constant.

This imply that the Euler equation is given by:

$$ c_{2t+1} = \left(\frac{1+r_{t+1}}{1+\rho}\right)^\frac{1}{\sigma} c_{1t} $$

Inserting the budget constraint and solve for s, optimal savings is given by:

$$ s_t = \frac{1}{(1+r_{t+1})^\frac{\sigma-1}{\sigma}(1+\rho)^\frac{1}{\sigma} +1} w_t $$

where the wage and the interest rate is given by:

$$ w_t = (1-\alpha)k_t^\alpha $$

$$ r_{t} = \alpha k_{t}^{\alpha-1} $$

We now want to see how the transitionscurve behaves for different values of $\sigma$.

First, we define the functions needed to derive the transition curve with CRRA utility:

```python
def savings2(w,r,rho,sigma):
    return (1/((1+r)**((sigma-1)/sigma)*(1+rho)**(1/sigma)+1))*w

def transition2(k,r,alpha,sigma,rho,n):
    w = (1-alpha)*k**alpha
    s = savings2(w,r,rho,sigma)
    return capital_2t(s,n)

def r2(alpha,k2):
    return alpha*k2**(alpha-1)
```

```python
def transition_curve2(alpha,rho,n):

    # a. grids
    capital_1 = np.linspace(1e-8,1,500)
    capital_2 = np.empty(500)

    #b. solve for each k in grid
    for i,k in enumerate(capital_1):

        # i. capital in t+1
        obj1 = lambda k2: k2 - transition2(k,r2(alpha,k2),alpha,sigma,rho,n)

        result = optimize.root_scalar(obj1,bracket=[1e-8,1],method='brentq')

        capital_2[i] = result.root

    return capital_1,capital_2
```

Now we derive transition curves for different values of $\sigma$ and plot these in a transition diagram:

```python
# Make a plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Transition curves
for sigma in [0.1,0.2,0.5,0.75,1]:
    capital_3,capital_4 = transition_curve2(alpha,rho,n)
    ax.plot(capital_3,capital_4,label=f'$\sigma = {sigma}$')

# Plot transition curves
ax.legend(loc='lower right',facecolor='white',frameon=True)
ax.plot(capital_3,capital_3, '--', color='grey')
ax.set_xlabel('$k_t$')
ax.set_ylabel('$k_t+1$')
ax.set_title('Transition curve')
ax.set_xlim([-0.001,0.5])
ax.set_ylim([0,0.5]);
```

We see that the higher $\sigma$ the lower capital accumulation. This is a consequence of individuals saving less when $\sigma$ is high. $1/\sigma$ is the elasticity of intertemporal substitution, thus the higher $\sigma$ the less are individuals willing to substitute consumption over time, and the less they save.

# PAYG System

We now introduce a social security system to the model. We assume that the government raises $d_t$ from the current young in period t and pays them out as benefits to current old:

$$(1+n)d_t = b_t$$

where $d_t$ is a fixed fraction of the wage income, such that: $d_t=\tau w_{t}$.

Thus we get a new budget constraint for the young in period t:

$$ c_{1t} + s_t + d_t = w_t$$

and a new budget constraint for them when old in period t+1 given by:
$$ c_{2t+1} = (1+r_{t+1})s_t + (1+n)d_{t+1} $$

We return to the case where utility $u(c)=log(c)$ and consider the transition curve for different values of $\tau$.

The savings equation will then be given by:

$$ s_t = \frac{1}{2+\rho} \left[(w_t-d_t) - \frac{1+\rho}{1+r_{t+1}} (1+n) d_{t+1}\right]$$

We define the functions for $d_t$, savings $s_t$ and the transition curve.

```python
tau=0.3

def savings(w,rho,d1,d2,r,n):
    return (1/(2+rho))*((w-d1)-((1+rho/(1+r))*(1+n)*d2))

def transition3(k,r,w2,alpha,rho,n,tau):
    w = (1-alpha)*k**alpha
    d1 = tau*w
    d2 = tau*w2
    s = savings(w,rho,d1,d2,r,n)
    return capital_2t(s,n)

def w2(alpha,k2):
    return (1-alpha)*k2**alpha
```

We define the new function for the transition curve

```python
def transition_curve3(alpha,rho,n,tau):

    # a. grids
    capital_1 = np.linspace(1e-8,1,500)
    capital_2 = np.empty(500)

    #b. solve for each k in grid
    for i,k in enumerate(capital_1):

        # i. capital in t+1
        obj1 = lambda k2: k2 - transition3(k,r2(alpha,k2),w2(alpha,k2),alpha,rho,n,tau)

        result = optimize.root_scalar(obj1,bracket=[1e-8,1],method='brentq')

        capital_2[i] = result.root

    return capital_1,capital_2
```

We construct a plot of the PAYG transition curve with a slider, which can be used to investigate the effect of different values of $\tau$.

```python
def fig(alpha, rho, n, tau):
    capital_5,capital_6 = transition_curve3(alpha,rho,n,tau)

    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(capital_5,capital_6, label=f'PAYG')
    ax.plot(capital_1,capital_2, label=f'Simple')
    ax.legend(loc='lower right',facecolor='white',frameon=True)
    ax.plot(capital_5,capital_5,'--',color='grey')
    ax.plot(capital_3,np.zeros(len(capital_3)), 'black')
    ax.set_xlabel('$k_t$')
    ax.set_ylabel('$k_t+1$')
    ax.set_title('Transition curve')
    ax.set_xlim([-0.001,0.3])
    ax.set_ylim([-0.05,0.3]);

    return

import ipywidgets as widgets
widgets.interact(fig,
    alpha = widgets.fixed(alpha),
    rho = widgets.fixed(rho),
    n = widgets.fixed(n),
    tau=widgets.FloatSlider(description='t', min=0, max=0.49, step=0.01, value=0.1),
);
```

    interactive(children=(FloatSlider(value=0.1, description='t', max=0.49, step=0.01), Output()), _dom_classes=('…

We see that the higher $\tau$ the lower capital accumulation, which is due to individuals saving less.

# Conclusion

In this project analyze the **Overlapping Generation Model**.
First we use a solver to find the analytical solution for capital accumulation and the steady state value of capital, $k^*$.

Next, we solve the model numerically and look at the dynamics of capital accumulation. Then we introduce CRRA utility, solve this numerically and analyze the dynamics of capital accumulation for different degrees of risk aversion. We find that less risk averse individuals, which means lower $\sigma$, will save more. Thus the lower $\sigma$, the higher capital accumulation is.

Finally we introduce a **pay-as-you-go system (PAYG)**. Here we find that individuals save less as they have less disposable income when young and they know that they receive benefits when they get old.


# Derivation of Bernoulli's Equations

## Learning outcomes

* Learn how Bernoulli's equations are derived.
* Work in curvilinear coordinates
* Learn about some applications of Bernoulli's equations.

## Recap of Vector fields

As we saw in the previous notebook, the velocity of a fluid flow at each point in an Eulerian sense can be represented by a vector field $\vec{V}(\vec{x},t)$ where the position vector $\vec{x}$ denotes the position of each velocity vector relative to the origin, $\vec{x}(x,y,z) = x\mathbf{\hat{i}}+y\mathbf{\hat{j}}+z\mathbf{\hat{k}}$ where $\mathbf{\hat{i}}$, $\mathbf{\hat{j}}$ and $\mathbf{\hat{k}}$ are unit basis vectors, and the current time is given by $t$. Any position vector can be described extending from the origin to some point $(x,y,z)$. Similarly $\vec{V}({\vec{x}},t) = u\mathbf{\hat{i}}+v\mathbf{\hat{j}}+w\mathbf{\hat{k}}$ where $u={u({\vec{x}}},t)$, $v={v({\vec{x}}},t)$ and $w={w({\vec{x}}},t)$ are the velocity magnitudes in each Cartesian direction for the current time, $t$. An example 2D flow field is shown below. Each vector's tail is positioned on the Cartesian grid according to its position vector $\vec{{}x}$ and the velocity vector field, which in this case is time invariant, is given by:

\begin{equation}
\vec{{}V}(x,y) = (\sin(y)+\pi) \mathbf{\hat{{}i}} + 1.2\cos(x) \mathbf{\hat{{}j}},
\end{equation}

which results in a flow from left to right with a slight sinuous meander in the $y$ direction.
We can plot this and also include some streamlines.

A simple 2D vector field created in MATLAB; can you recreate it?

Can you recreate this flow field with MATLAB or Python?

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

# Generate position vectors
x = np.arange(0, 6, 0.25)
y = np.arange(0, 3, 0.25)

X, Y = np.meshgrid(x, y)

# Velocity components
u = np.sin(Y) + np.pi
v = 1.2*np.cos(X)

# Create a new figure axes
fig, ax = plt.subplots(figsize=(10,5))

# Plot the vector field
ax.quiver(X, Y, u, v)

# Number of streamlines we want to draw
nstream = 18

# Points in (x,y) where the streamlines start (they are then integrated in both directions)
# In this case I take nstream points along from the lowest y value to the highest along
# the mid point of x. Rather than type actual numbers this method allows the code to work
# with any input should you change x and y
seed_points = np.array([np.linspace(np.median(x), np.median(x),nstream),\
                        np.linspace(np.min(y), np.max(y),nstream)])

# plot the streamlines
ax.streamplot(X, Y, u, v, color='g', start_points=seed_points.T)

plt.xlabel("x")
plt.ylabel("y")
plt.title("Vector Field")
plt.xlim([0, 6])
plt.ylim([0, 3])
ax.set_aspect('equal')
plt.show()
```

Streamlines, as we have already seen, are simply curves that are everywhere tangent to the vector field for any instance in time. This means that the flow is everywhere following the curvature of the streamline and hence never crosses them. **There is no mass flux normal to a streamline.** The sinusoidal nature of the flow field shown above is now readily apparent by employing streamlines to visualise the flow.

## Streamline orientated coordinate system

Let us now consider a new coordinate system, $(s,n)$, mapped to the curvature of the streamlines. The position of any particle and its velocity in a steady flow can described in terms of its position along a streamline $s$ and its position normal to that streamline $n$.
Unlike our simple Cartesian coordinate system this new $(s,n)$ system can be curvilinear but don't worry, it's much the same.

The figure below shows the green streamlines from previous figure as well as the curves normal to them, shown in red. As above the flow is still confined to the streamlines with no velocity normal along the red lines. Notice how as the green streamlines curve upwards the red normal curves fan outwards.

The s-n coordinate system for the velocity field defined above

Lets try to recreate this image with Python and `matplotlib`.

```python
# recreate our position vector grid
x = np.arange(0, 6, 0.05)
y = np.arange(0, 3, 0.05)

X, Y = np.meshgrid(x, y)

# components of velocity field
u = np.sin(Y) + np.pi
v = 1.5*np.cos(X)

# Create a new figure axes
fig, ax = plt.subplots(figsize=(16,8))

# Plot the vector field
# ax.quiver(X, Y, u, v)

# We can make this easy by plotting our green streamlines and
# then plotting red streamlines by flipping the velocity components
# This is a simple inplementation of a 'rotation matrix'

# Becasue we want to define starting points for both sets of streamlines
# we can define a function that we can call upon as needed
# Its good practise to avoid repeating code

def gen_seeds(x,y,nstream):
    seed_points = np.array([np.linspace(np.median(x), np.median(x),nstream),\
                  np.linspace(np.min(y), np.max(y),nstream)])
    return seed_points

# As above we want to seed our streamlines evenly from top to bottom from the midpoint in x
# our function gen_seeds is designed to do this
seed_points = gen_seeds(x,y,20)

# ax.plot(seed_points[0],seed_points[1],'go')
ax.streamplot(X, Y, u, v,color='green',start_points=seed_points.T, minlength = 1, arrowstyle='-')

# For the red streamlines we need to seed from left to right about the midpoint in y
# so we feed it (y,x) instead of (x,y) and then flip the result back to (x,y)
seed_points = np.flipud(gen_seeds(y,x,20))

# ax.plot(seed_points[0],seed_points[1],'ro')
ax.streamplot(X, Y, v, -u,color='red',start_points=seed_points.T, minlength = 0.5, arrowstyle='-')

plt.xlabel("x")
plt.ylabel("y")
plt.title("$(s,n)$ Coordinate System")
plt.xlim([0, 6])
plt.ylim([0, 3])
ax.set_aspect('equal')
plt.show()

```

As with the basis vectors $\mathbf{\hat{{}i}}$ and $\mathbf{\hat{{}j}}$ for the Cartesian system above (and $\mathbf{\hat{{}k}}$ for $\mathbb{R}^3$) we can define basis vectors $\mathbf{\hat{{}s}}$ and $\mathbf{\hat{{}n}}$ for this new coordinate system. A closeup view of the coordinate system is shown below. The local values of the $s$ and $n$ axes are shown as well as the $\mathbf{\hat{{}s}}$ and $\mathbf{\hat{{}n}}$ unit basis vectors for a particle traveling long the $n=0$ streamline between locations $s_1$ and $s_2$. Be careful not to confuse the basis vector $\mathbf{\hat{{}s}}$ with distance $s$.

**To reiterate, the velocity vector $\vec{{}V}$ is always tangent to the streamline.**

Detailed view of the s-n coordinate system

In streamline coordinates we can write the velocity vector in terms of these normal and tangent unit vectors:

\begin{align}
\vec{{}V} = V \mathbf{\hat{{}s}} + 0 \mathbf{\hat{{}n}} = V \mathbf{\hat{{}s}},
\end{align}

where $V = V(s,n)$ at position $\mathbf{\hat{{}s}} = {\hat{{}\textrm{s}}}(s,n) $. The velocity of the particle along streamline $s$ is $V$ while the velocity normal along $n$ is zero. For steady flow both flow speed $V$ and direction ${\hat{{}\textrm{s}}}$ are functions of location.

The motion of the fluid particles is governed by Newton's second law, $F=m\vec{{}a}$ and relates the influence of pressure, a force per unit area on fluid motion. The mass of our fluid particle is captured by its density $\rho$, the mass per unit volume. The components of the acceleration vector $\vec{{}a}$ can be written in terms of the $\mathbf{\hat{{}s}}$ and $\mathbf{\hat{{}n}}$ basis vectors using the **material derivative**:

\begin{align}
\vec{{}a} = \frac{D\vec{{}V}}{Dt} = a_s \mathbf{\hat{{}s}} + a_n \mathbf{\hat{{}n}}.
\end{align}

There is an additional Jupyter Notebook covering the Material Derivative in some detail.

The acceleration is then obtained by using the chain rule:

\begin{align}
\vec{{}a} =  \frac{D\left(V \mathbf{\hat{{}s}}\right)}{Dt} = \frac{DV}{Dt}\mathbf{\hat{{}s}} + V \frac{D \mathbf{\hat{{}s}}}{Dt}.
\end{align}

and can be expanded:
\begin{align}
\vec{{}a} = \left( \frac{\partial V}{\partial s} \frac{ds}{dt} + \frac{\partial V}{\partial n} \frac{dn}{dt} \right)\mathbf{\hat{{}s}} + V \left( \frac{\partial \mathbf{\hat{{}s}}}{\partial s} \frac{ds}{dt} + \frac{\partial \mathbf{\hat{{}s}}}{\partial n} \frac{dn}{dt} \right).
\end{align}

Since $\frac{ds}{dt} = V$ and $\frac{dn}{dt} = 0$ becasue the fluid particles always move along the streamline and not normal to it, the previous equation reduces to:

\begin{align}
\vec{{}a} = \left(V \frac{\partial V}{\partial s}  \right)\mathbf{\hat{{}s}} + V \left(V \frac{\partial \mathbf{\hat{{}s}}}{\partial s} \right).
\end{align}

Therefore:
\begin{align}
a_s = V  \frac{\partial V}{\partial s}.
\end{align}

Next we need to consider acceleration normal to the streamline, $a_n$. The effects of curvature on the flow will mean that $a_n$ may not be zero even though the velocity normal to the streamline is zero. This should make intuitive sense, you will experience lateral g-forces as you travel in a car going around a bend.

The figure below shows a fluid particle traveling distance $\delta s$ from point $A$ to $B$ along a curved streamline. The basis vectors $\mathbf{\hat{{}s}}$ and $\mathbf{\hat{{}n}}$ are shown positioned at point $A$ and again at point $B$ corresponding to a displacement $\mathbf{\hat{{}s}}(s+\delta s)$ along the streamline. The quantity $ \frac{\partial \mathbf{\hat{{}s}} }{\partial s}$ above represents the limit as $\delta s \rightarrow 0$ of the change in the unit vector orientation along the streamline, $\delta \mathbf{\hat{{}s}}$, per change along the streamline, $\delta s$. The basis vector $\mathbf{\hat{{}s}}$ is a unit vector so its magnitude is constant ($|\mathbf{\hat{{}s}}| = 1$) but its orientation along a curved streamline will change.

Analysis of normal acceleration on a curved streamline for a fluid particle moving a distance $\delta s$ from point $A$ to point $B$. Shown also is a velocity triangle showing change in orientation of the unit vector $\mathbf{\hat{{}s}}$ and the resultant vector $\mathbf{\hat{{}s}}(s+\delta s)$ due to the change $\delta\mathbf{\hat{{}s}}$.

We can analyse this motion by considering the velocity triangle shown in the previous figure. The orientation of the velocity vector of the particle as it moves from point $A$ to $B$ changes from $\mathbf{\hat{{}s}}(s)$ to $\mathbf{\hat{{}s}}(s+\delta s)$ therefore the vector $\delta\mathbf{\hat{{}s}}$ represents the change in the orientation of the velocity vector.

[//]: # "Your comment in here.
We can write an expression for $\delta\mathbf{\hat{{}s}}$ as:

\begin{align}
\delta\mathbf{\hat{{}s}} &= \mathbf{\hat{{}s}}(s+\delta s) - \mathbf{\hat{{}s}} \\
&= \mathbf{\hat{{}s}} + \frac{\partial\mathbf{\hat{{}s}}}{\partial s}\delta s - \mathbf{\hat{{}s}} \\
&= \frac{\partial\mathbf{\hat{{}s}}}{\partial s}\delta s.
\end{align}
"

Both triangles $AOB$ and $A'OB'$ shown in the last figure are similar triangles.
 As $\delta s \rightarrow 0 \Rightarrow \delta\theta \rightarrow 0$, and  $\delta\mathbf{\hat{{}s}}$ becomes increasingly normal to the streamline, i.e. along $n$. Small angle theory gives

 $\sin{\delta\theta} = \delta\theta$,

\begin{align}
\delta\mathbf{\hat{{}s}} &= |\mathbf{\hat{{s}}}|\sin\delta\theta \mathbf{\hat{{n}}} \\
&= |\mathbf{\hat{{s}}}|\delta\theta \mathbf{\hat{{n}}} \\
&= \delta\theta \mathbf{\hat{{}n}}
\end{align}

since $|\hat{{}s}|=1$

Therefore:

\begin{align}
\frac{\partial \mathbf{\hat{{}s}}}{\partial s} = \lim_{\delta s \rightarrow 0} \frac{\delta \mathbf{\hat{s}}}{\delta s} = \frac{\delta\theta}{\delta s}\mathbf{\hat{{}n}}.
\label{ds}
\end{align}

As $\delta s = \mathbf{R}_c \delta\theta$, this becomes

\begin{align}
\frac{\partial \mathbf{\hat{{}s}}}{\partial s}  = \frac{\mathbf{\hat{{}n}}}{\mathbf{R}_c}
\end{align}

the acceleration can now be written:

\begin{align}
\vec{a} = V\frac{\partial V}{\partial s}\mathbf{\hat{{}s}} + \frac{V^2}{{\mathbf{R}}_c}\mathbf{\hat{{}n}},
\end{align}

or in terms of streamwise and normal components:

\begin{align}
a_s = V\frac{\partial V}{\partial s},~a_n = \frac{V^2}{{\mathbf{R}}_c}
\end{align}

## Bernoulli's Equation

Thus far we have considered the motion of a fluid particle along a streamline and derived expressions for the acceleration of the particle along and normal to the flow direction. As previously noted Newton's second law, $F=m\vec{a}$, governs fluid motion. Having already considered acceleration we now turn our attention to the forces acting on the fluid element. These comprise the pressure forces and the gravitational forces and are illustrated below for a fluid particle of volume $\delta { \rlap{V}-} = \delta s\delta n \delta y$ and mass $\delta m$.

Analysis of forces acting on fluid particle. The terms $p$ and ${ \rlap{V}-}$ are the pressure of the particle and its volume. The top row shows the volume of the fluid particle, the areas where the streamwise and normal pressures are exerted and the decomposition of the gravitational forces into streamwise and normal components.

The sum of the forces in the streamwise direction is

\begin{equation}
\sum \delta F_s = \delta m~a_s = \delta mV \frac{\partial V}{\partial s} = \rho \delta { \rlap{V}-}~V\frac{\partial V}{\partial s}
\label{motion}\tag{1}
\end{equation}

The gravitational force (particle weight) acting in the streamwise direction:

\begin{equation*}
\delta W_s = -\delta W \sin{\theta} =-\gamma \delta { \rlap{V}-}\sin{\theta}
\end{equation*}

where $\gamma = \rho g$ is the specific weight of the fluid.

The pressure force acting in the streamwise direction:

\begin{equation*}
\delta F_{ps} = (p-\delta p_s)\delta n\delta y - (p+\delta p_s)\delta n\delta y = -2 \delta p_s \delta n\delta y.
\end{equation*}

Since:

\begin{equation*}
\delta p_s \approx \frac{\partial p}{\partial s}\frac{\delta s}{2}~\Rightarrow
\delta F_{ps} = -\frac{\partial p}{\partial s}\delta s\delta n\delta y
=-\frac{\partial p}{\partial s}{ \rlap{V}-}
\end{equation*}

Summing the pressure and gravitational forces:

\begin{equation}
\sum \delta F_s = \delta W_s + \delta F_{ps} = \left( -\gamma \sin{\theta}-\frac{\partial p}{\partial s} \right)\delta{ \rlap{V}-}
\label{F_s}\tag{2}
\end{equation}

Equating the RHS of equation \ref{F_s} with the RHS side of equation \ref{motion}, the equation of motion, we get:

\begin{equation*}
-\gamma \sin{\theta}-\frac{\partial p}{\partial s} = \rho V \frac{\partial V}{\partial s}
\end{equation*}

and rearranging:

\begin{equation*}
\frac{\partial p}{\partial s} + \rho V \frac{\partial V}{\partial s} + \gamma \sin{\theta} = 0
\end{equation*}

Using the chain rule, $dp = \frac{\partial p}{\partial s}ds + \frac{\partial p}{\partial n}dn $ and noting that along a streamline $dn = 0$

\begin{equation*}
\frac{d p}{d s} = \frac{\partial p}{\partial s}
\end{equation*}

The velocity derivative can be rewritten, using the power rule, as:
\begin{equation*}
\frac{d}{d s} \left( \frac{V^2}{2} \right) = V\frac{dV}{ds}
\end{equation*}

Additionally $\sin{\theta} = \frac{dz}{ds}$. Combining the above gives Euler's equation:

\begin{equation}
\frac{dp}{ds} + \rho \frac{d}{d s} \left( \frac{V^2}{2} \right) + \gamma \frac{dz}{ds} = 0
\end{equation}

Simplifying:
\begin{equation*}
{dp} + \frac{1}{2}\rho \left( {V^2} \right) + \gamma dz = 0
\end{equation*}

and integrating yields Bernoulli's first equation along a streamline:
\begin{equation}
p + \frac{1}{2}\rho {V^2} + \gamma z = \text{const}
\end{equation}

## Bernoulli's Second Equation

Repeating the process normal to the streamline in a similar manner and skipping several steps for brevity; the equation of motion is:

\begin{equation}
\sum \delta F_n = \delta{m}~{a_n} = \delta m \frac{V^2}{\mathbf{R}_c} = \rho \delta { \rlap{V}-} \frac{V^2}{\mathbf{R}_c}
\label{motion2}\tag{3}
\end{equation}

The pressure force in the normal direction is given by:

\begin{equation*}
\delta F_{pn} = (p-\delta p_n)\delta s\delta y - (p+\delta p_n)\delta s\delta y = -2 \delta p_n \delta s\delta y
\end{equation*}

Summing the pressure and gravitational forces normal to the streamline as above:

\begin{equation*}
\sum \delta F_n = \delta W_n + \delta F_{pn} =
\left( -\gamma \cos{\theta}-\frac{\partial p}{\partial n} \right)\delta{ \rlap{V}-}
\label{Fn}\tag{4}
\end{equation*}

As before, equating the RHS of equations \ref{motion2} and \ref{Fn}:

\begin{equation}
-\gamma \cos{\theta}-\frac{\partial p}{\partial n} = \frac{\rho V^2}{\mathbf{R}_c},
\end{equation}

and rearranging:

\begin{equation*}
\frac{\partial p}{\partial n} +  \frac{\rho V^2}{\mathbf{R}_c} + \gamma \cos{\theta} = 0
\end{equation*}

Using the chain rule once more, $dp = \frac{\partial p}{\partial s}ds + \frac{\partial p}{\partial n}dn $ and noting that along a streamline $dn = 0$ and noting that *normal* to a streamline $ds = 0$

\begin{equation*}
\frac{d p}{d n} = \frac{\partial p}{\partial n}
\end{equation*}

Noting that $\cos{\theta} = \frac{dz}{dn}$ and combining the above gives:
\begin{equation}
\frac{dp}{dn} + \frac{\rho V^2}{\mathbf{R}_c}+ \gamma \frac{dz}{dn} = 0
\end{equation}

Simplifying:
\begin{equation*}
{dp} + \frac{\rho {V^2} }{\mathbf{R}_c}dn + \gamma dz = 0
\end{equation*}

and integrating yields Bernoulli's second equation normal to a streamline:
\begin{equation}
p + \rho \int \frac{V^2}{\mathbf{R}_c} dn + \gamma z = \text{const}
\end{equation}

## Applications of Bernoulli's Equation

The two Bernoulli equations are valid when computed along and normal to a streamline respectively.
The assume the flow is steady, inviscid and and incompressible. Nevertheless these equations can let us solve a lot of fluids problems or obtain a first estimate of how a flow will behave before more complex analysis is undertaken.

Looking again at the first Bernoulli equation each term has the units $J/m^3$ or $N/m^2$ or $Pa$.

\begin{equation}
\underbrace{p}_{\text{static pressure}} +
\underbrace{\frac{1}{2}\rho {V^2}}_{\text{dynamic pressure}} +
\underbrace{\rho g z}_{\text{hydrostatic pressure}} =
\underbrace{p_{total}}_{\text{total pressure}}
\end{equation}

The *static pressure* is the pressure of the fluid due to its thermodynamic state. Consider the state equation for an ideal gas $p = \rho R T$. Here the pressure $p$ is the static pressure.

The *dynamic pressure* $\frac{1}{2}\rho {V^2}$ isn't really a pressure at all. Instead it is a measure of the kinetic energy of the flow measured as a force per unit area.

The sum of the *static pressure* and the *dynamic pressure* is referred to as the *stagnation pressure*. This is the pressure felt by when a flow is brought to rest at a point in a streamline. For example at the leading edge of a wing.

The *hydrostatic pressure* we have already covered, but to reiterate this the pressure due to gravitational forces and results from the weight of a column of fluid.

Dividing by $\rho g$ results in

\begin{equation}
\frac{p}{\rho g} + \frac{V^2}{2g} + z = \text{H}
\end{equation}

where $\text{H}$ is the total pressure head which has units of $J/N$ or simply $m$. Expressing pressure in the form of a pressure head has some advantages.

### Measuring Static Pressure

A common way to measure the static pressure is with a manometer. If we consider the second Bernoulli equation for the limit as $R_c \rightarrow \infty$, i.e. parallel flow,

\begin{align}
\lim_{\mathbf{R}_c \rightarrow \infty} \frac{\rho V^2}{\mathbf{R}_c} = 0
\end{align}

This means that for parallel flows any pressure change normal to the flow is due to hydrostatic changes only.
For many flows, such as in pipes where $z$ is small this means that $\frac{dp}{dn} = 0$. Therefore we can measure the static pressure in a flow by placing a *pressure tap* at the wall.

## Finishing up

The following video covers much of the same material covered within this notebook.
Many of the visualisations are excellent. Here is a link to the video or you can watch it embedded by running the cell below.

https://www.youtube.com/watch?v=LI9Mi1KhFTs&list=PL0EC6527BE871ABA3&index=6&feature=plpp_video

```python
from IPython.display import display, IFrame
from IPython.core.display import HTML

# Youtube
HTML('')

```

```python

```

3.2 Derivation of Bernoulli's Equation.ipynb


#### _Speech Processing Labs 2020: Signals: Module 2_

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

from dspMisc import *
```

# 2 Filtering the Source: Finite Impulse Response Filters

### Learning Outcomes
* Be able to describe what an FIR filter is
* Be able to explain what the impulse response of an FIR filter is
* See how an FIR filter can be used as a lowpass frequency filter.
* See how changing the coefficients of an FIR filter can change it's frequency response

### Need to know
* Topic Videos: Spectral Envelope, Filter, Impulse Train
* [Interpreting the DFT](./sp-m1-5-interpreting-the-dft.ipynb)
* [Building the source: impulse trains](sp-m2-1-impulse-as-source.ipynb)

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 2.0 Filters

We've seen in the past notebooks that sometimes our input signal isn't exactly what we want. There is a vast literature in signal processing about designing filters to transform one signal into another. In speech processing, our signals often includes some sort of noise that we'd like to get rid of.  However, we can also use filters to shape a simple input, like an impulse train, into something much more complicated, like a speech waveform.

In class you've seen two types of filters:

* Finite Impulse Response (FIR)
* Infinite Impulse Response (IIR)

Both perform a transform on an input sequence $x[n]$ to give us some desired output sequence $y[n]$.  The difference between the two types of filters is basically whether we only use the inputs to derive each output $y[n]$ (FIR), or whether we also use previous outputs (IIR).

In the following we'll illustrate some of the properties of FIR filters.

## 2.1  Finite Impulse Response Filters

Finite Impulse Response (FIR) filters have the following form:

$$
\begin{align}
y[n] &= b[0]x[n] + b[1]x[n-1] + \dots + b[K]x[n-K] \\
 &= \sum_{k=0}^K b(k) x[n-k]
\end{align}
$$

Here, we have:
* an input sequence $x[n]$ of length $N$
* a set of $K$ filter coefficients.

We can read the equation as saying that $n$th ouput of the filter, $y[n]$, is a weighted sum of the previous K inputs $x[n],...,x[n-K]$.

### Example
Let's plot the $b[k]\cdot x[n-k]$ terms where $x$ is sinusoid of 4 Hz and the filter is `b=[1.1, 1.2, 1.5, 1.2, 1.1]`

```python
## Plotting a filter input window

## Set the number of samples N, sampling rate f_s
N=64
f_s = 64
t_s = 1/f_s
print("sampling rate: f_s = %f\nsampling time: t_s: %f" % (f_s, t_s))

x, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

## Plot the sinusoid we've just created:
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.plot(time_steps, x, 'o', color='grey')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')

## Filter coefficients (you could try changing these)
b = [1.1, 1.2, 1.5, 1.2, 1.1]

## K is the number of filter coefficients
K=len(b)

## Let's see what happens when n=19 (or try changing this variable!)
n=19
print("filter using b at n=%d, K=%d\n" % (n, K))

## Plot the values that are input to the filter
## +1's because python indexing/slicing doesn't include the end point
timedom.plot(time_steps[n+1-K:n+1], x[n+1-K:n+1], 'o', color='red')

## Calculate the b[k]*x[n-k] terms and add them to a list
filtered_n = []
for k in range(len(b)):
    ## print out the variables here
    print("%d:, b[%d]=%f, x[%d-%d]=%f, b[%d]*x[%d-%d]=%f" % (n-k, k, b[k], n, k,  x[n-k], k, n, k, b[k]*x[n-k]))
    filtered_n.append(b[k]*x[n-k])

## reverse the list so that they're in time order
filtered_n.reverse()

## Plot the b[k]*x[n-k] terms
timedom.plot(time_steps[n+1-K:n+1], filtered_n, 'o', color='blue')

## Calculate the filter output (add up the product terms)
print("\ny[%d] = %f" % (n, sum(filtered_n)))
```

In the plot above, you should see:
* A cosine wave with frequency 4 Hz in grey.
* The inputs the the filter x[19],...,x[15] in red.
    * 5 input values for 5 filter coefficients
* The product $b[k]*x[n-k]$ for $n=19$, and $k=0,...,4$

So with `b = [1.1, 1.2, 1.5, 1.2, 1.1]` all the input values get scaled up, with the the middle of the filter window get biggest relative increase.

When we add all the product terms together, we get y[19] = 4.87 - a lot bigger than any of the input values!

## 2.2 An FIR moving average filter

A useful special case of an FIR filter is where each of the filter coefficients is just $1/K$. In this case our FIR equation looks like this:

 $$
 \begin{align}
     y[n] &= \sum_{k=0}^{K-1} \frac{1}{K} x[n-k] \\
          &= \frac{1}{K} \sum_{k=0}^{K-1}  x[n-k]
 \end{align}
 $$

This equation says that when we apply the filter, we step through the input. At each step, we output the average of the previous $K$ inputs. You might know this by another more intuitive name: a _moving average_.  You might also have seen this as method to 'smooth' an input.

Let's play around with this idea a bit and see how it relates to our notion of frequency response.

### Example:

Let's look at this 5-point moving average filter.  In this case all the filter coefficients $b[k] = 1/5$ for $k=0,..,K-1=4$

$$
\begin{align}
y[n] &= \frac{1}{5} \sum_{k=0}^4 x[n-k] \\
&= \frac{1}{5}x[n] + \frac{1}{5}x[n-1] + \frac{1}{5}x[n-2] + \frac{1}{5}x[n-3] + \frac{1}{5}x[n-3]
\end{align}
$$

Now, let's code this specific filter up and apply it to some sequences!

```python
# Apply a moving average filter of size K to input sequence x
def moving_average(x, K=5):
    ## There are quicker ways to do this in numpy, but let's just do it this way for now for transparency

    ## We know that we'll have as many outputs as inputs, so we can initialize y to all zeros
    N = len(x)
    y = np.zeros(N)

    ## Go through the input one step at a time
    for n in range(N):

        ## Add up the last K inputs (including the current one)
        for k in range(K):

            ## Exercise: why do we have to put this conditional in here?
            if n-k >= 0:
                y[n] = y[n] + x[n-k]

        ## Divide by the size of the input window to get an average
        y[n] = (1/K) * y[n]

    return y

```

### Generate a 'noisy' sinusoid

Let's generate a compound sinusoid with one low frequency and one high(er) frequency component.  We can take the higher frequency component as representing some sort of periodic noise in the signal.

```python
## Set the number of samples N, sampling rate f_s
N=64
f_s = 64
t_s = 1/f_s

print("sampling rate: f_s = %f\nsampling time: t_s: %f" % (f_s, t_s))
```

    sampling rate: f_s = 64.000000
    sampling time: t_s: 0.015625

```python
## make some sinusoids:
## Since the sample rate and sequence length is the same, the generated time steps will match for
## x1 and x2
x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=6, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

# add them up!
x_noisy = x1 + x2
```

```python
## Plot the compound sinusoid we've just created:
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.plot(time_steps, x_noisy, color='magenta')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
```

You should be able to see that the 4Hz cosine wave is perturbed by a 24 Hz cosine. So, this could represent a situation where there is high frequency noise in our signal that we'd like to get rid of.  Let's see if we can use our filter to smooth out this high frequency noise.

### Apply the moving average filter

```python
## Apply our moving average filter
K=5
y_mov = moving_average(x_noisy,K=K)
```

```python
## Plot the results
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')

## The input signal
timedom.plot(time_steps, x_noisy, color='magenta', label='Noisy input')

## The underlying 4Hz signal
timedom.plot(time_steps, x1, color='grey', label='low freq input component')

## After the moving average has been applied
timedom.plot(time_steps, y_mov, color='blue', label='filter output')

timedom.set_title('%d-point moving average filter applied to a noisy sinsuoid' % K)
timedom.legend()
```

You should see:
* the original signal in magenta
* the low frequency cosine component in grey
* the output of the filter in blue

### Exercise:

* From the plot, does it appear that the moving average recovered the original 4Hz signal in terms of:
    * frequency?
    * amplitude?
    * phase?
* Are there still high frequency components in the filter output?

Can you see some issues with applying this filter? What happens at the end points? Are they valid?

### Notes

```python
# recovery:
#   frequency: yes
#   amplitude: yes
#   phase: no
# yes but they're very smoothed out

# issue: the end points don't start at 1 or 0
```

### Get the DFT of the filtered signal

We can apply the DFT to our output to check our observations from the time-domain output.

```python
## DFT of the original input
mags_x, phases_x = get_dft_mag_phase(x_noisy, N)

## DFT of the filter output
mags_y, phases_y = get_dft_mag_phase(y_mov, N)
```

```python
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

fig, mags = plt.subplots(figsize=(16, 4))
mags.set(xlim=(-1, N/2), ylim=(-1, N))
mags.scatter(dft_freqs, mags_x, color='magenta', label='noisy input')
mags.scatter(dft_freqs, mags_y, color='blue', label='filtered output')
mags.set_xlabel("Frequency (Hz)")
mags.set_ylabel("Magnitude")
mags.set_title("DFT Magnitude before and after moving average filter")
mags.legend()

#Let's not worry about phase right now, but feel free to uncomment and have a look!
#fig, phases = plt.subplots(figsize=(16, 4))
#phases.set(xlim=(-1, N/2), ylim=(-10, 10))
#phases.scatter(dft_freqs, phases_x, color='magenta', label='noisy input')
#phases.scatter(dft_freqs, phases_y, color='blue', label='filtered output')
#phases.set_xlabel("Frequency (Hz)")
#phases.set_ylabel("Phase (rad)")
#phases.set_title("DFT Phase before and after moving average filter")
#phases.legend()

```

### Exercise

* Based on the magnitude spectrum:
    * Did the filter get rid of the 24Hz component?
    * Do you see any signs of leakage?

* What happens if you change the frequency of the second sinusoid to something lower (e.g. 6Hz)?

### Notes

```python
# Yes, pretty well, almost completely
# no

# yes, there's leakage
```

## 2.3 FIR as convolution

An FIR Filter that takes $K$ previous elements of $x[n]$ as input has the following general form:

$$ y[n] = \sum_{k=0}^{K-1} b[k] x[n-k] $$

You might recognize this as a **convolution** of the two sequences $b$ and $x$  (i.e. $b * x$).

So, we can theoretically set our filter to coefficients to whatever we want.  Here's a function that generalizes our moving average filter to allow for this:

```python
def fir_filter(x, filter_coeffs):
    N = len(x)
    K = len(filter_coeffs)

    y = np.zeros(N)
    for n in range(N):
        for k in range(K):
            if n-k >= 0:
                #print("y[%d]=%f, b[%d]=%f, x[%d]=%f" % (n, y[n], k, filter_coeffs[k], n-k, x[n-k]))
                y[n] = y[n] + (filter_coeffs[k]*x[n-k])
        #print("y[%d]=%f" % (n, y[n]))
    return y

```

### Changing the filter coefficients

Let's try it out with different coefficient values, comparing our unweighted average `h_avg` filter with a weighted average `h_wavg` filter.

```python

## The 5-point moving average from before
h_avg = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
y_avg = fir_filter(x_noisy, h_avg)

## A 5-point symmetrically weighted average
h_wavg = np.array([1/5, 1/3, 1, 1/3, 1/5])
y_wavg = fir_filter(x_noisy, h_wavg)
```

### Filter effects in the time domain

```python
## Plot the filter outputs
fig, timedom = plt.subplots(figsize=(16, 4))

## The original "noisy" input
timedom.plot(time_steps, x_noisy, color='magenta', label='input x_noisy')
timedom.scatter(time_steps, x_noisy, color='magenta')

## The 5-point moving average
timedom.plot(time_steps, y_avg, color='blue', label='unweighted average: y_avg')
timedom.scatter(time_steps, y_avg, color='blue')

## The 5-point weighted average
timedom.plot(time_steps, y_wavg, color='orange', label='weighted average: y_wavg')
timedom.scatter(time_steps, y_wavg, color='orange')
timedom.legend()
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
```

In this time vs amplitude graph, you should see:
* the 'noisy input' in magenta
* the output of the unweighted average filter in blue (`y_avg`)
* the output of the weighted average filter in orange (`y_wavg`)

### Exercise

Q: Why is the output of `y_wavg` more spikey than that of `y_avg`?

### Notes

```python
# ???
```

## 2.4 FIR Filters in the Frequency Domain

### The DFT of the filtered outputs

We can look at the effect of the two FIR filters defined above in the frequency domain by performing a DFT on the filter outputs.

```python
## DFT of the original input
mags_x, phases_x = get_dft_mag_phase(x_noisy, N)

## DFT after weighted average filter: h_wavg = np.array([1/5, 1/3, 1, 1/3, 1/5])
mags_wavg, phases_wavg = get_dft_mag_phase(y_wavg, N)

## DFT after unweighted average filter: h_avg = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
mags_avg, phases_avg = get_dft_mag_phase(y_avg, N)

dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot magnitude spectrums
fig, mags = plt.subplots(figsize=(16, 4))
mags.set(xlim=(-1, N/2), ylim=(-1, N))
mags.scatter(dft_freqs, mags_x, color='magenta', label='input')
mags.scatter(dft_freqs, mags_avg, color='blue', label='unweighted average')
mags.scatter(dft_freqs, mags_wavg, color='orange', label='weighted average')
mags.legend()

## Plot phase spectrums
fig, phases = plt.subplots(figsize=(16, 4))
phases.set(xlim=(-1, N/2), ylim=(-10, 10))
phases.scatter(dft_freqs, phases_x, color='magenta', label='input')
phases.scatter(dft_freqs, phases_avg, color='blue', label='unweighted average')
phases.scatter(dft_freqs, phases_wavg, color='orange', label='weighted average')
phases.legend()

```

### Exercise:

* Describe the difference between the different FIR filters based on the frequency magnitude and phases responses plotted above.

* Does the weighted average filter do as good a job at filtering out the higher frequency signals?

### Notes

```python
# the weighted filter increases the original frequencies but the unweighted one decreases them
# both filters phase shifts the input positively, but only the weighted average phase shifts the frequencies
#   with high amplitude negatively
```

## 2.5 Convolution in Time,  Multiplication in Frequency

Now we get to the really cool bit.  We know that the DFT allows us to go from the time domain to the frequency domain (and the Inverse DFT let's us go back).  But it also has this very important property:

$$ h[k] * x[n] \mapsto \text{ DFT} \mapsto H(m) \cdot X(m) $$

That is, convolving an input sequence $x$ with a set of filter coefficients, $h$ in the time domain  ($h*x$) is the same as (pointwise) multiplication of the DFT of $h$ with the DFT of $x$.  So, if we know what type of frequency response $h$ has, we can treat this as apply a sort of mask to the DFT of $x$.  This property is known as the **convolution theorem**.

Another way to think about it is that if the DFT outputs of $h$ has zero magnitude, then applying that filter to some input will also zero out those frequencies in the filter output!

We'll see some visualizations of this shortly, but first, we can also note that you can go back the other way using the Inverse DFT:

$$ H(m) \cdot X(m) \mapsto \text{ IDFT} \mapsto h[k] * x[n] $$

And also it works for multiplication in time domain too:

$$ h[k] \cdot x[n] \mapsto \text{DFT} \mapsto H(m) * X(m) $$

This is useful for understanding why the leakage graph around each DFT bin looks the way it does (though we won't go into it here!).

In fact, we can use this to not only show that the moving average acts as a low pass filter, but also to be able to calculate exactly the type of filter it will be.

###  Plotting different FIR filter frequency responses

In order to multiply together the filter and the input frequency responses, we need to make sure the filter and the input have the same number of samples.  We can do this by just padding out the filter with zeros (you can see why in the optional extra material at the end of this notebook).  We can then look how changing the 'shape' of a filter changes it's frequency response. The following exercise shows some examples.

### Exercise
* Run the function plot_filter_freq_responses (defined in the next cell) to plot the frequency responses of filters with different shapes (e.g. `h_plat`, `h_tri`, `h_rect` in the cell below the next).
    * What's the difference in frequency response of the triangular filter and the rectangular filter?
    * What's the difference between the moving average of size 5 and one of size 9 (h_rect9)?
    * Try some other FIR filters!

```python
# Given a list of filters, the sample rate and a specific sequence length
# plot the DFT frequency response of each filter.
# Each filter should be defined as a list of coeffients (b[k])
def plot_filter_freq_responses(filters, sample_rate, seq_length):

    ## Get the set of DFT output frequencies given the sample rate and desired sequence length
    dft_freqs_filter = (sample_rate/seq_length) * np.arange(seq_length)

    ## Calculate the time steps for each filter value given the sample rate and sequence length
    time_steps = (1/sample_rate) * np.arange(seq_length)

    ## Set up some plots:
    # the filter itself (time v amplitude)
    fig_time, sinusoid = plt.subplots(figsize=(16, 4))

    # the frequency response (freq v magnitude)
    fig_freq, fresponse = plt.subplots(figsize=(16, 4))

    x_filters = {}
    ## For each filter:
    for i, h in enumerate(filters):

        # pad the filter coefficients with zeros until we get the desired sequence length
        x_zeros = np.zeros(seq_length - len(h))
        x = np.concatenate([h, x_zeros])

        # Get the DFT outputs
        mags, phases = get_dft_mag_phase(x, seq_length)

        # Plot the filter
        sinusoid.scatter(time_steps, x)
        sinusoid.plot(time_steps, x, label=repr(h))

        # plot the magnitude response
        fresponse.scatter(dft_freqs_filter, mags)
        fresponse.plot(dft_freqs_filter, mags, label=repr(h))
        fresponse.set(xlim=(-1,seq_length/2))

        # return the filters and the DFT responses just in case
        x_filters = {'x':x, 'mags':mags, 'phases':phases, 'coeffs':h}

    sinusoid.set_xlabel('Time(s)')
    sinusoid.set_ylabel('Amplitude')
    sinusoid.set_title('Zero padded filters of different shapes')
    sinusoid.legend()

    fresponse.set_xlabel('Frequency (Hz)')
    fresponse.set_ylabel('Magnitude')
    fresponse.set_title('DFT magnitude response of ero padded filters of different shapes')
    fresponse.legend()

    return x_filters

```

```python
h_plat = np.array([0.1, 0.2, 0.2, 0.2, 0.1])
h_tri = np.array([0.04, 0.12, 0.15, 0.12, 0.01])
h_rect = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
h_rect9 = np.array([1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9])
## Try some others if you like!

N=64
f_s=64

filter_dict = plot_filter_freq_responses(filters=[h_tri, h_plat, h_rect, h_rect9], sample_rate=f_s, seq_length=N)
```

### Notes

```python
# triangular vs rectangular:
#   rectangular filters have bumps in their magnitude response
# moving average size:
#   the larger moving average filter filters out more higher-frequency waves than the smaller one
```

## 2.7 Applying an FIR filter to an impulse train

Now, finally we can look at the effect of applying an FIR filter to an impulse train.  Remember, we're using impulse trains to represent a sound source (i.e. vocal pulses at a specific frequency).  Eventually, we want to be able to define filters that capture the effects of the vocal tract.

First, here's a function that produces a sequence of pulses at a given frequency and sample rate. We'll then apply a triangular filter and look at what the frequency response is of the output is.

```python
## Let's make an impulse train
N=200
f_s = 8000
t_s = 1/f_s

frequency = 130

x_imp, time_steps = make_impulse_train(sample_rate=f_s, frequency=frequency, n_samples=N)

```

```python
## Plot the impulse train made in the cell before
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, x_imp)
td.plot(time_steps, x_imp)
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.set_title('Impulse train')
```

You should see an impulse train, with an impulse frequency of 200 Hz.  Let's see what happens when we apply the filter.

```python
## define our FIR filter coefficients
h_mov = np.array([0.1, 0.3, 0.5, 0.3, 0.1])

#try this later!
#h_mov = np.array([1])

## Apply the filter to our impulse train
y_mov = fir_filter(x_imp, h_mov)

## plot the impulse train and the filtered version of it
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, x_imp)
td.plot(time_steps, x_imp, label='impulse train')
td.plot(time_steps, y_mov, label='filtered impulse train')
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.legend()
td.set_title('Impulse train and filtered impulse train')
```

Here you should see our impulse train (red) and the filtered impulse train (blue).  The filtered version has smaller amplitude and a broader peak (more samples have non-zero value).  Also, the filtered peaks are slightly after the original impulses.

Now let's look at the DFT output:

```python
## Get the DFT the filter output
mags, phases = get_dft_mag_phase(y_mov, N)

## Plot the DFT frequencies rather than DFT output indices
dft_freqs = get_dft_freqs_all(f_s, N)

## Plot the magnitude spectrum
fig, ax = plt.subplots(figsize=(16, 4))
ax.scatter(dft_freqs[0:int(N/2)],mags[0:int(N/2)])
ax.plot(dft_freqs[0:int(N/2)], mags[0:int(N/2)])

```

### Exercises

* What do the spikes in the DFT outputs represent?
* What does the first spike after the 0 frequency one represent?
* What does this filter appear to do?
    * Remember applying the FIR filter in the time domain (via convolution) is the same as multiplying the DFT of the filter to the DFT of the input signal

### Notes

```python
# the spikes represent the harmonics
# the fundamental frequency
# lowpass filter
```

### Exercises

* What happens when the impulse train frequency doesn't fall on one of the bins? e.g. `frequency = 130`, for 200 samples, with sampling rate 8000 samples/second
* Does the magnitude spectrum have the harmonic structure you'd expect?

### Notes

```python
# less tidy magnitude spectrum
# somewhat, but there are lots of bleeding
```

## 2.6 (Extra Extension) The moving average filter as a rectangular function

This section (2.6) illustrates how we can use the convolution theorem understand why the moving average type filters act like low pass filters, and also the connection with leakage in the DFT magnitude response we saw previously.  This is optional extra material.

If you want to see an example of the convolution theorem working in the frequency domain, you can just run the code and have a look at the graph titled 'Frequency response after applying 5-point weighted average filter'

The convolution theorem tells us that, if we know the frequency response of an FIR filter, we know how it will affect the frequency response of it's input (we just multiply the individual frequency response together).

To understand what filter frequency response will look like, it's helpful to first observe that our unweighted moving average filter is pretty much a rectangular window function.  It's easy to see what this means when we plot it. The following function allows us to generate rectangular functions:

```python
def gen_rect_window(start_index, end_index, sample_rate=64, seq_length=64):
    nsteps = np.array(range(seq_length))
    t_s = 1/sample_rate
    time_steps = t_s * nsteps

    ## Let's make a rectangular window
    x_rect = np.zeros(seq_length)
    x_rect[start_index:end_index] = 1

    return x_rect, time_steps

```

###  Now, we make a rectangular window

```python
## Make rectangular window
N=64
K=16
f_s=64
start_index=24
end_index=start_index+K

x_rect, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N)

fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x_rect, color='magenta')
timedom.plot(time_steps, x_rect, color='magenta')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
timedom.set_title('a rectangular window')
```

You should see a sequence with 64 point where the mmiddle 16 points have value 1 and the rest have value 0  (i.e., it looks like a rectangle in the middle).

### Now, let's  look at the frequency response of the rectangular window

```python
## Now we do the DFT on the rectangular function:

## get the magnitudes and phases
mags_rect, phases_rect = get_dft_mag_phase(x_rect, N)

## the DFT output frequencies
dft_freqs_rect = get_dft_freqs_all(f_s, N)

## let's just look at the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2))
fdom.scatter(dft_freqs_rect, mags_rect)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel('Magnitude')
fdom.set_title('Frequency response of rectangular window')
## Looks leaky!
```

### Leaky windows?

The plot of the frequency magnitude response of our rectangular window has the hallmarks of leakiness. That is, the frequency response looks scalloped, with the biggest peak occuring around 0Hz. That is, it looks like a low pass filter!

With a bit of algebra we can derive the frequency for any $m$ (not just the DFT output bins indices) to be the following:

If $x[n]$ is a rectangular function of N samples with $K$ continugous samples of value 1 (starting at index $n_0$), we can figure out what the DFT output will be:

$$X[m] = e^{i(2\pi m/N)(n_0-(K-1)/2)} . \frac{\sin(2\pi mK/2N)}{\sin(2\pi m /2N)}$$

This is called the **Dirichlet kernel**. It has the **sinc** shape we saw when we looked at spectral leakage.

How is this useful?  Since we know what the frequency response of a rectangular window is, we know what convolving this with different input sequences will look like in the frequency domain.  We just multiply the frequency magnitude responses together.

On a more general note, this sort of convolution with a (short) window is how we do frequency analysis of speech: we taking windows of speech (aka frames) through time and and apply the DFT to get a frequency response.  A rectangular window is the simplest type of window we can take.  The equation above tells us that the sinc shaped response is an inherent part of using this sort of window.  In fact, we can use other window types (e.g. Hanning) to make the main lobes shaper and the sidelobes flatter, but we never really get away from this sinc shape in real world applications.  This is a key component of this soft of short term analysis.

Let's write this up in a function:

```python
def gen_rect_response(n_0, K, N, stepsize=0.01, polar=True, amplitude=1):
    ms = np.arange(0.01, N, stepsize)
    qs = 2*np.pi*ms/N

    ## Infact, we can work the frequency response to be the Dirichlet Kernel:
    response = (np.exp(-1j*qs*(n_0-(K-1)/2)) * np.sin(qs*K/2))/np.sin(qs/2)

    if polar:
        response_polar = [cmath.polar(z) for z in response]
        mags = np.array([m for m, _ in response_polar]) * amplitude
        phases = np.array([ph if round(mag) > 0 else 0 for mag, ph in response_polar])
        return (mags, phases, ms)

    return response, ms

```

Now we can plot the dirichlet kernel with the leaky looking DFT magnitudes we calculated earlier for our rectangular window.

```python
## Overlay the dirichlet kernel onto the DFT magnitudes we calculated earlier
## You should be able to see that the DFT magnitudes appear as discrete samples of the Dirichlet Kernel

mags_rect, phases_rect = get_dft_mag_phase(x_rect, N)
mags_rect_sinc , _ , ms = response = gen_rect_response(start_index, K, N)

fig, ax = plt.subplots(figsize=(16, 4))
ax.scatter(dft_freqs_rect, mags_rect, label='rectangular window')
ax.plot((f_s/N)*ms, mags_rect_sinc, color='C2', label='dirichlet')
ax.set(xlim=(-1,N/2))
ax.set_xlabel('Frequency (Hz)')
ax.set_ylabel('Magnitude')
ax.set_title('Frequency response of a rectangular sequence, %d samples with %d contiguous ones' % (N, K))
```

You should be able to see that the DFT magnitudes appear as discrete samples of the sinc shaped Dirichlet Kernel

### The unweighted average filter as a rectangular function

We can think of our 5-point unweighted average filter as a 5-point input sequence with all values set to 1/5.  We can then deduce that the frequency response of the filter will have the same shape as the frequency response of a rectangular window of all ones, but scaled down by 1/5.

Now let's check:

```python
N_h=5
f_s=64
start_index=0
end_index=N_h - start_index

## A 5 point rectangular window of all ones
h_avg, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N_h)
h_avg = h_avg/N_h

fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, h_avg, color='magenta')
td.plot(time_steps, h_avg, color='magenta')
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.set_title('5 point unweighted average as a rectangular function')

## Not very exciting looking!
print("h_avg:", h_avg)
```

You should just see 5 point in a row, all with value 1/5.  Now, we can plot the DFT magnitude response, as well as it's idealized continuous version:

```python
## Get the frequency magnitude response for our rectangular function
mags_h_avg, phases_h_avg = get_dft_mag_phase(h_avg, N_h)

## Get the continuous
rect_mags_h_avg, _ , ms = gen_rect_response(start_index, N_h, N_h, amplitude=np.max(h_avg))

## x-axis as frequencies rather than indices
ms_freqs_h_avg = (f_s/N_h) * ms
dft_freqs_h_avg = (f_s/N_h) * np.arange(N_h)

## Plot the frequency magnitude response
fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1, N/2))
fd.scatter(dft_freqs_h_avg, mags_h_avg)
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response of 5-point unweighter average filter')
#fd.scatter(dft_freqs_rect, mags_rect)

fd.plot(ms_freqs_h_avg, rect_mags_h_avg, color="C2")
```

You should see $floor(N/2) = 2$ points, with a main lobe peaking at 0 Hz, and side lobes peaking between each of the DFT output frequencies.
So,  DFT frequencies sit exactly at the zeros of this function when the windown size K is the same as the number of samples.

### Matching the filter and input size with zero padding

The theorem we saw above told us that we could calculate the frequency response of applying the FIR filter to an input sequence (via convolution), but multiply the DFT outputs of the filter and the input sequence.

Now, the x-axis range matches our that of our noisy input sequence because that is determined by the sampling rate. However, the filter frequency response we have above only has 5 outputs, while our input sample size was 64 because the number of DFT outputs is determined by the number of samples we put into the DFT.

To get things in the right form, we need to do some **zero padding** of the filter. We'll see that this basically gives us more samples of the Dirichlet Kernel corresponding to the filter frequency response.

```python
N=64
K=5
f_s=64
start_index=0
end_index=K

## Make a rectangular filter: K ones at the start
h_avg_pad, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N)

## Divide by K to make it an average
h_avg_pad = h_avg_pad/K

## Plot the filter
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, h_avg_pad, color='magenta')
td.plot(time_steps, h_avg_pad, color='magenta')
td.set_xlabel('Time (s)')
td.set_title('5 point unweighted average FIR filter padded with zeros')

#print("N=%d, K=%d, start=%d, end=%d" % (N, K, start_index, end_index))

```

```python
## Get the frequency magnitude response for our rectangular function
mags_havg, phases_havg = get_dft_mag_phase(h_avg_pad, N)

## Plot the frequency magnitude response
## x-axis as actual frequencies rather that DFT indices
dft_freqs_havg = (f_s/N) * np.arange(N)

fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1,N/2))
fd.scatter(dft_freqs_havg, mags_havg)
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Magnitude response of 5-point unweighter average filter zero padded to 64 samples')
```

You should be able to see more clearly in the frequency response graph that the zero padding doesn't change doesnt change the basic shape of the filter's frequency response, we just get a finer grained representation in terms of samples (red dots).

### Calculate the input and filter frequency responses

```python
## Now let's calculate frequency responses of the original input
mags, phases = get_dft_mag_phase(x_noisy, N)

## ...  the filter
mags_filter, phases_filter = get_dft_mag_phase(h_avg_pad, N)

## ... and the filtered output that we calculated above
mags_avg, phases_avg = get_dft_mag_phase(y_avg, N)

## Plot with actual frequencies on the x-axis
dft_freqs = get_dft_freqs_all(f_s, N)
```

```python
## plot frequency responses

fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1,N/2), ylim=(-1, N))

# DFT(input)
fd.scatter(dft_freqs, mags, color='magenta', label='DFT(input)')

# DFT(filter) * DFT(input)
fd.scatter(dft_freqs, mags_filter*mags, color='blue', label='DFT(filter).DFT(input)')

# DFT(filtered input)
fd.scatter(dft_freqs, mags_avg, color='red', label='DFT(filter*input)')
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response after applying 5-point weighted average filter')
fd.legend()
```

You should see that the results from multiplying the DFT magnitudes from the input and the filter (blue) is (more or less) the same as the DFT of applying the filter in th time domain via convolution (red)

* Notice that there are some differences between the results from the time domain application of the filter (red) and the frequency domain multiplication (blue).  In particular there appears to be some leakage in the time-domain convolution case, possibly due to floating point errors.

### Exercise

* Try changing the frequency of the second cosine component of our compound wave in the code below.
    * Does the amount of attenuation of the high frequency component change as suggested by the DFT of the filter?
    * e.g. try 26 Hz vs 19 Hz
* What does this tell you about how well this low pass filter get's rid of high frequency noise?

```python
## Change the frequency of x2

x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=19, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

# add them up!
x_noisy = x1 + x2

## Now let's calculate frequency responses of the original input
mags, phases = get_dft_mag_phase(x_noisy, N)

## ...  the filter
mags_filter, phases_filter = get_dft_mag_phase(h_avg_pad, N)

## Plot with actual frequencies on the x-axis
dft_freqs = get_dft_freqs_all(f_s, N)

## plot frequency responses

fig, fd = plt.subplots(figsize=(16, 4))

fd.set(xlim=(-1,N/2), ylim=(-1, N))

# DFT(input)
fd.scatter(dft_freqs, mags, color='magenta', label='DFT(input)')

# DFT(filter) * DFT(input)
fd.scatter(dft_freqs, mags_filter*mags, color='blue', label='DFT(filter)*DFT(input)')

fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response after applying 5-point weighted average filter')
fd.legend()
```

### Notes

```python

```

```python

```

```python

```