```python
Godfrey Beddard 'Applying Maths in the Chemical & Biomolecular Sciences an example-based approach' Chapter 9
```

```python
# import all python add-ons etc that will be needed later on
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
from scipy.integrate import quad
init_printing()                         # allows printing of SymPy results in typeset maths format
plt.rcParams.update({'font.size': 14})  # set font size for plots
```

### 1.6 Discontinuous functions

The square wave shown in Fig. 4, has a stepwise discontinuity. When $x \ge 0,\;  f(x) = 1$, and when $x \lt 0,  f (x) = -1$ and the wave repeats itself with a period of $2\pi$. This discontinuous function is sometimes called the _odd signum_ function, sgn($x$). It can be shown that, if the discontinuity occurs in the mid range $-\pi \to \pi$, each integral, $a_n$ and $b_n$, can be split into two, one being taken from $-\pi \to 0$ and the second from $0 \to \pi$. The Fourier series expansion now has coefficients

$$\displaystyle a_n=\frac{1}{\pi}\int_{\pi}^0 (-1)\cos(nx)dx+\frac{1}{\pi}\int_0^{\pi} (+1)\cos(nx)dx $$

and when $n = 0$,

$$\displaystyle a_0=-\frac{1}{\pi}\int_{\pi}^0 1dx+\frac{1}{\pi}\int_0^{\pi} 1dx =0$$

and because the function has odd symmetry $f(-x) = -f(x)$, the cosine integrals and all other $a$ coefficients are zero,

$$\displaystyle a_n=-\frac{1}{\pi}\int_{\pi}^0 \cos(nx)dx+\frac{1}{\pi}\int_0^{\pi}\cos(nx)dx = \frac{2}{\pi}\left. \sin(nx)\right|_{-\pi}^0 = 0$$

The $b$ coefficients are

$$\displaystyle b_n=\frac{1}{\pi}\int_{\pi}^0 (-1)\sin(nx)dx+\frac{1}{\pi}\int_0^{\pi} (+1)\sin(nx)dx = \frac{2}{\pi}\int_0^\pi sin(nx)dx  =  \frac{2}{n\pi}[1-\cos(n\pi)]$$

However, when $n$ is even, the coefficient $b_n$ is zero, because for integer multiples of $\pi$ the cosine is one, and when $n$ is odd the cosine is $-1$, consequently $b_n = 4/n\pi$. The Fourier series equation (1) reduces to

$$\displaystyle g(x) = \sum\limits_{n\,=\,odd}b_n\sin(nx)=\frac{4}{\pi}\sum\limits_{n\,=\,odd}\frac{\sin(nx)}{n} $$

Figure 4. A few cycles of a square wave, which is a discontinuous function. Left shows the function $f(x)$ and right its Fourier reconstruction $g(x)$ via a series of $n = 49$ terms.  The over- and undershoot in the Fourier series as shown on the right hand plot is a phenomenon called the Gibbs phenomenon and is discussed next.

____

### 1.8 The Gibbs phenomenon

The Fourier series of a square wave is shown in fig 4 (right) with a relatively large number of terms. Adding more will improve the fit to the function, but it will never be exact because an _infinite_ number of terms will be needed to follow the right-angled bends at the top and bottom of the wave. This angle effectively corresponds to an infinite sine frequency.

The size of the overshoot remains the same independent of the number of terms; this is called the Gibbs' phenomenon, after J. Willard Gibbs, of thermodynamics fame, who explained its cause as being due to the non-uniform convergence of the Fourier series in the vicinity of a discontinuity.

It is difficult to prove, but may be seen quite easily graphically; plot the Fourier series approximation to the square wave, say, from $-0.2 \to +0.2$ and with different numbers of terms in the series, and then observe the height of the overshoot. As more terms are added, this becomes closer to zero, but its amplitude above $1$ and below $-1$ remains the same, as shown in the next figure for $10$ and $100$ terms in the series.

The Gibbs phenomenon was first observed by Michelson who is more famous for the interferometric Michelson - Morley experiment that determined that the speed of light was independent of the position of the earth and thus disproved the hypothesis of the aether. By 1898 Michelson had constructed a machine, called the Harmonic Integrator, that could calculate up to eighty terms in a Fourier series and present the results graphically. He noticed the overshoot and wrote to Gibbs for an explanation thinking his machine was in error. You can see a photograph of this machine in 'A Student's Guide to Fourier Transforms' (James 1995). This book also contains a clear introduction to Fourier transforms.

Figure 5. The Gibbs phenomenon where the over- and undershoot remain at the same size as more and more terms are added to the Fourier series.

## 2 Integrating, differentiating and summing series

If the series for a function $f(x)$ has been calculated then that for $\int f(x)dx$ can be easily calculated, providing that the integration can be performed. Similarly if $df(x)/dx$ is available the series for this can also be obtained. As an example, the series for $x^4$ can be obtained from the Fourier series for $x^3$ by integrating term by term; in addition, integration can lead to a better representation of a function with the same number of terms in the summation. The algebraic result will be different from that of a direct series for $x^4$, but should be just as good a representation.

The series for $x^3$ is calculated first and as this is an odd function, all the even cosine terms are zero; therefore, all the $a$ coefficients are zero and the series only contains sine or $b$ terms of the form

$$\displaystyle b_n=\frac{1}{\pi}\int_{-\pi}^\pi x^3\sin(nx)dx$$

This and similar integrals can be integrated 'by parts' or the sine converted to an exponential and then integrated; in either case the sine or exponential part is integrated first, so that the power of $x$ is reduced in the second term of the 'by parts' integration. Using sympy the result is shown below.

```python
x = symbols('x' )
n = symbols('n',integer=True)
f01 = x**3*sin(n*x)/pi
bn = integrate(f01,(x,-pi,pi),conds='none')
simplify(bn)
```

As the sine terms are all zero and the cosines alternate 1 for even $n$ or -1 for odd $n$ in the limits $x=\pm n\pi$ and with $n \gt$ 0, the integral is $b_n=2(-1)^{n+1}(\pi^2/n - 6/n^3) $. The expansion of $x^3$ is therefore,

$$\displaystyle x^3 \approx g(x^3) = \sum\limits_{n=1} b_n \sin(nx) = 2(\pi^2-6)\sin(x)-\frac{2\pi^2-3}{2}\sin(2x)+\cdots$$

The series expansion is plotted in Fig 6; notice how the difference in the value of the series for $x^3$ is greatest at the limits $\pm \pi$, and how the series repeats itself. The Fourier series is zero at both -$\pi$ and $\pi$ because it assumes that the function $x^3$ is periodic; thus just before $\pi$ its value is $\pi^3$, and just after -$\pi^3$, and this leads to an overshoot.

Integrating term by term produces a new series, which is,

$$\displaystyle x^4 \approx g(x^4) = 2\int\limits_0^x \sum_{n=1}b_n\sin(nx)dx = 8\sum_{n=1}(-1)^{n+1}(\pi^2/n^2 - 6/n^4)(1-\cos(nx))$$

and the initial $8$ arises because the integration of $x^3$ is $x^4$/4 and $b_n$ was evaluated for $x^3$. The curve for $x^4$ and its Fourier series is shown in Fig.6. The fit is good up to the limit $\pm \pi$, the limit of the initial Fourier series. The $x^4$ series $g(x^4)$ is smoother than the $x^3$ and also a better fit to the curve, and this may be attributed to the integration, which measures the area under a curve and has the effect of smoothing the function.

Figure 6. The functions $x^3$ and $x^4$ are shown as dashed lines. The wiggly solid curve shows the result of having only 20 terms in the $x^3$ summation. The underlying oscillation of the trigonometric terms is obvious, even though the series clearly approximates $x^3$. The $g(x^4)$ series, after integrating the $x^3$ Fourier series, is a considerably better fit than is  the fit to $x^3$.

### 2.1 Summations

The Fourier series can also be used to evaluate summations; for example the expansion of $x^2$ in the range $\pm \pi$, equation (15), gives

$$\displaystyle  x^2= \frac{\pi^2}{3} - 4 \left ( \cos(x) - \cos(2x)/2^2 + \cos(3x)/3^2 -  \cdots \right ) $$

When $x=\pi/2$, the series produces the result

$$\displaystyle  \frac{\pi^2}{12}= 1-\frac{1}{4}+\frac{1}{9}-\frac{1}{16}+ \cdots \frac{(-1)^n}{n^2}+\cdots $$

where the series limit is taken to infinity. In fact, this series converges very slowly; the alternating sign in $\pm 1/n^2$ terms ensures this, and about 360 terms are needed to obtain a value for π accurate to five decimal places: not really a good way to calculate $\pi$. When $x$ = $\pi$ the series is

$$\displaystyle  \frac{\pi^2}{6}= 1+\frac{1}{4}+\frac{1}{9}+\frac{1}{16}+ \cdots \frac{1}{n^2}+\cdots $$

which converges more rapidly as all the terms have the same sign. Many other unusual summations can be achieved using different Fourier series; however, for us they are only curiosities.

## 3.1 Some formal points about the Fourier series

The sine and cosines making up the Fourier series have two important properties: orthogonality and  completeness.

In the language of vectors, the sine and cosine functions represent a _complete orthogonal basis set_ on which the target function, $f(x)$, is expanded as a sum of $N$ terms. This basis set is of infinite length and is $\sin(x), \sin(2x), \cdots \sin(nx) \cdots$, where $n = 1, 2, \cdots$, and there is a similar set for the cosines with $n$ $\ge$ 0. Any desired accuracy can be achieved, provided that $N$ is large enough. If the basis set functions are also normalized, the set is _orthonormal_ rather than just orthogonal.

Basis sets often seem rather abstract because we do not often need to use them explicitly. For example, although not normally described in this way, the exponential function $\displaystyle e^x = 1 + x + x^2/2! + x^3/3! \cdots $, is, by contrast, an expansion in the basis set of $x^n$ and the coefficients are $1/0!, 1/1!, 1/2!, 1/3!, \cdots 1/n! \cdots$. This basis set is not orthogonal because the condition for this to apply is that the product of any two elements is zero when taken over the whole range of the set, $\pm \pi$ for sine and cosine. The dot product of any two orthogonal vectors is zero, see Chapter on Vectors section 2.

Similarly for the sine and cosine basis set even though it is continuous, the condition is $\displaystyle \int_{-\pi}^\pi\sin(mx)\sin(nx)dx =\pi\delta_{n,m}$, which is zero if $m \ne n$. This is not true of the coefficient of
the $x^n$ basis set of the exponential expansion, thus we cannot form a Fourier series based on this.

The great importance and usefulness of the Fourier series is that it represents the best fit, in a least-squares way, to any function $f(x)$ because $\displaystyle \int_{-L}^{L}[f(x)-g(x)]^2dx$ is minimized when $g(x)$ is the series expansion of $f(x)$.

A given set of functions, $S_n(x)$ is said to be complete if some other arbitrary function $f(x)$ can be expanded in the set of the $S$ functions. If they have the same boundary conditions, this makes the functions _complete and orthogonal_ in a given range, and in the case of the sine and cosine Fourier series, this range is -$\pi$ to $\pi$. The general series describing $f(x)$, is

$$\displaystyle  g(x) = \sum\limits_{n=1}^\infty q_nS_n(x)  \tag{16}$$

$q_n$ being constants that are related to the target function $f(x)$. If the $S$ functions are orthonormal, then, when $n$ and $m$ are integers, $\displaystyle \int S_n^*(x)S_m(x) dx= \delta_{m,n}$.

In modern mathematics, the term 'Fourier series' does not refer just to the original sine and cosine series, or their complex exponential representation, but to a series formed by other functions that form a complete orthogonal basis set. Often the term generalized Fourier series is used to describe these, but this is not universal. The sine or cosine functions are not unique in forming series and many other functions could be used provided that they can form an orthogonal set. Other such functions include the Hermite polynomials, used to describe the harmonic oscillator wavefunctions, and the Legendre and Chebychev polynomials. In Section 4 it is shown how these can also be used to form series that describe arbitrary target functions $f(x)$.

### 3.2 A general method for numerically calculating a Fourier series

An algorithm is given below with which to calculate the Fourier series of a function and is fairly straightforward if equations (5) and (6) are used and care taken over the integration limits. The function is placed in the first line as and the range is from $\pm 7$, in this example. The target function is $\displaystyle f(x)= e^{-x/2}\sin^2(x)$. The equations (4 to 6) in section 1.2 are used. The series is calculated numerically.

The result of the calculation is shown in Fig. 7 and is compared with the original function, and for most of the curve the fit is tolerably good but not excellent. The main discrepancy is the overshoot, which is the Gibbs phenomenon.

```python
# using numpy first to do algebraic integration

f = lambda x: np.exp(-x/2.0)*np.sin(x)**2           # target function
L = 7.0                                             # limits +- L
m = 50                                              # number of terms in series
a0= quad(f,-L,L )[0]/L                              # integrate; the [0] returns just the result and not error as well

fa = lambda x, n : f(x)*np.cos( n*np.pi*x/L )/L     #  function to give a_n after integration
fb = lambda x, n : f(x)*np.sin( n*np.pi*x/L )/L     #  function to give b_n after integration

numx = 1000                                         # number of data points to plot
x = np.linspace(-L,L,numx)

FS = lambda x: a0/2.0   + sum( quad(fa,-L,L,args=n)[0]*np.cos(n*np.pi*x/L) for n in range(1,m)) \
                        + sum( quad(fb,-L,L,args=n)[0]*np.sin(n*np.pi*x/L) for n in range(1,m))

plt.plot(x,f(x), color ='lightgreen',linestyle='solid',linewidth=8)
plt.plot(x,FS(x),color ='red',linewidth=2)
plt.axhline(0,   color ='grey',linestyle='dashed')
plt.xlabel('x')
plt.title('Function & its Fourier series with '+str(m)+' terms')
plt.show()

```

**Fig 7**. The original function (thick green line) and its Fourier series (red). As can be seen the fit is not tremendously good although  the series does follow the general shape of the function except at the end points $\pm 7$.

______

## 4 Generalized Fourier series with orthogonal polynomials

The Fourier series expands a function in sine and cosines. In the language of vectors and matrices, the sine and cosines form a basis set in which the function f is expanded. The essential property that any basis set of functions must have is orthogonality. If $\varphi$ is such a function, then the basis set is the functions $\varphi_1, \varphi_2, \cdots$  and the condition for orthogonality is

$$\displaystyle \int_a^b\varphi_m(x)^*\varphi_n(x)dx=c_m\delta_{m,n}    \tag{17}$$

where $c$ is a constant and $\delta$ is the delta function and is zero if $n \ne m$ otherwise it is unity. The asterisk indicates the complex conjugate if the function $\varphi$ is complex. If the functions are normalized as well as being orthogonal, then $c_m  = 1$. The range of the integral, $a$ to $b$, depends on the type of function which might be $\pm 1$ or $\pm \infty$. Some orthogonal polynomials, such as the Hermite polynomials, are so defined that a weighting function $w$, is needed to make them orthogonal. In this case, equation (17) is redefined as

$$\displaystyle \int_a^b\varphi_m(x)^*\varphi_n(x)w(x)dx=c_m\delta_{m,n}    \tag{18}$$

Suppose $f(x)$ is the function we want to expand as a series in the $\varphi$'s. The generalized Fourier series is defined in a simpler way than the normal sine/cosine series, because only one function is involved; hence

$$ f(x)=\sum\limits_{n=0}^\infty a_n\varphi_n  \tag{19}$$

The next task is to find the coefficients $a_n$, and this is done by multiplying both sides of (19) by $\varphi$ and any weighting $w$, and integrating;

$$\displaystyle \int_a^b f(x)\varphi_m(x)w(x)dx=\int_a^b \sum\limits_{n=0}^\infty a_n\varphi_n(x)\varphi_m(x)w(x)dx=\sum\limits_{n=0}^\infty a_nc_n\delta_{n,m}=a_nc_n $$

The summation is taken outside the integral in the second step, because it depends on $n$ not $x$, and the orthogonality (18) is used to solve the integral. The summation disappears in the last step because of the property of the $\delta$ function. The coefficients $a_n$ are therefore,

$$\displaystyle  a_n=\frac{1}{c_n}\int_a^bf(x)\varphi_n(x)w(x)dx  \tag{20} $$

and this equation and (19) form the equations for the generalized Fourier transform. The only way that information about the function $f(x)$ enters the calculation, is _via_ the coefficients $a_n$.

The similarity of (19) to the expansion of a vector in its basis set is clear. In a vector equation in three dimensions, we might write $\displaystyle V=\sum\limits_{m=0}^3v_m \boldsymbol i$ where $\boldsymbol i$ is the unit vector along the $x$-axis, and then the $v_m$ are the projections of the vector $V$ onto this axis. If there were
$n$ dimensions, which clearly cannot be pictured graphically if $n \gt$ 3, then the $v_m$ would be projections of $V$ onto the $m^{th}$ axis. Similarly, in (19) the $a_n$ coefficients are the projections of the function $f(x)$ onto the $\varphi$, but in this case there are more than three 'axes'.

The Legendre polynomial $P_n(x)$ is used as an example of a basis set function with which to calculate the generalized Fourier series of $\displaystyle e^{-x}\cos^3(x)$ over the range $\pm $1. These polynomials appear in problems when an electric charge, perhaps on an ion, is close to other charges, such as from a dipole, or when an electron feels the effect of other electrons and nuclei. All the orthogonal polynomials have several different recursion formulae, and one of the most useful for the Legendre is

$$\displaystyle nP_n(x) = (2n - 1)xP_{n-1}(x) - (n - 1)P_{n-2}(x) $$

for $n = 2, 3, 4, \cdots$ with $P_0(x) = 1$ and $P_1(x) = x$, and the polynomials are defined only over the range $x = \pm $1. The weighting function is 1, and their normalization constant $c_n = 2/(2n + 1)$. Although it is not apparent from the formulae, these polynomials are oscillating functions with an increasing number of nodes as $n$ increases. The Fourier series using Legendre polynomials instead of sine/cosine is sometimes called the Fourier -Legendre series.

The Chebychev polynomials over the range $\pm$1 can be calculated directly as $T_n(x)=\cos(n\cdot\cos^{-1}(x))$. Some Legendre and Chebychev polynomials are shown in the next figure.

Figures 8 & 9.  Left: Legendre polynomials. Right: Chebychev polynomials. The numbers indicate the order, $n$, of the polynomial, $L(n, x)$ or $T(n, x)$. The even numbered ones are drawn with a solid line the odd, dashed. Notice the odd - even nature of the functions corresponds to the index number. Both polynomials are limited to the range $-1 \le x \le 1$.

______

An example of using Legendre polynomials is shown for the function $\cos^2(3x)e^{-x}$ in the next calculation. Equations 19 and 20 are used. The fit to he function is good with only 15 terms in the series. For an extended range then the equations need to be put into reduced units to make the range $\pm 1$.

```python
# using legendre polynomials to form a fourier series.
def P(x,n):         # recursion formula
    if n == 0 :
        return 1
    if n == 1:
        return x
    else:
        return ((2*n-1)*x*P(x,n-1)-(n-1)*P(x,n-2))/n  # P(n,x)

f = lambda x: np.cos(3*x)**2*np.exp(-x)    # target function
w = lambda x: 1.0                          # weighting
c = lambda n: 2.0/(2.0*n+1.0)              # normalisation
m = 10                                     # number of terms in series to sum, wiil be v. slow if this is large

numx = 200
x = np.linspace(-1,1,numx)

func  = lambda x,n: f(x)*P(x,n)*w(x)

LegFS = lambda x,m: sum( quad(func,-1,1,args=(n))[0]*P(x,n)/c(n) for n in range(0,m))    # Legendre Fourier

plt.axhline(0,linestyle='dashed',color='grey',linewidth=1)
plt.axvline(0,linestyle='dashed',color='grey',linewidth=1)
plt.plot(x,f(x),color='gray',linewidth=8,label='function')  # originla function
plt.plot(x,LegFS(x,m),color='yellow',label='series')

plt.title('Legendre Fourier series and function')
plt.legend()
plt.show()
```

The range of some functions, orthogonal polynomials and weighting factors.

$$\displaystyle \begin{array}{lcc}
\text{}\\
\hline
\text{Functions} & \text{symbol } f(x) & \text{Range } & \text{Weighting}^1 & \text{normalisation } c_n \\
\hline
\text{sine/cosine} &                           &\pm\pi\text{ or } \pm L & 1 & \pi \text{ or } L\\
\text{exponenial}  & \exp(i\pi nx/L)           &\pm\pi\text{ or } \pm L & 1 & \pi \text{ or } L\\
\text{Legendre}    &  P_n(x)                   &\pm 1                   & 1 & 2/(2n+1)         \\
\text{Hermite}     &  H_n(x)                   &\pm \infty              & \exp(-x^2)    & 2^nn!\sqrt{\pi}  \\
\text{Laguerre}    &  L_n(x)                   &0\rightarrow\infty      & \exp(-x)      & 1         \\
\text{Chebychev (1st kind) }   &  T_n(x)                   &\pm 1                  & 1/\sqrt{1-x^2} &\text{if } n=0\, \pi \text{ else } \pi/2      \\
\hline
\end{array}$$

$^1$ If the variable $x$ is actually an angle $\theta$ then the volume element for the integration has to be changed from $dx$ to $\sin(\theta)d\theta$ and in these cases $\sin(\theta)$ must multiply the weighting function.

### 4.1 Generating Functions for orthogonal polynomials

The named orthogonal polynomials are the solutions of specific differential equations. They occur widely in physics and chemistry; the Hermite polynomials describe the harmonic oscillator wavefunctions; the Legendre electric charge distributions; and the associated Laguerre the electron distribution in the H atom. The spherical harmonic polynomials are based on the Legendre polynomials, with $x = \cos(\theta)$, and are used to describe the angular momentum in atoms and molecules; they define the shapes of atomic orbitals, the rotational motion of molecules, and the heat flow around spheres.

The orthogonal polynomials, such as the Legendre polynomial used in the previous section, can be produced from a series expansion of a generating function. These functions have two variables $x$ and $u$; expansion is made in the powers of $u$ and the polynomials are the factors belonging to the terms in $u^n$ or $u^n/n!$. A typical series expansion producing polynomial $p_n(x)$ is

$$\displaystyle f(x,u)= p_0(x)+p_1(x)u + p_2(x)\frac{u^2}{2!}+\cdots = \sum\limits_{n=0}p)n(x)\frac{u^n}{n!}$$

This definition has to be checked, however, as some polynomials do not require $u^n$ to be divided by $1/n!$. As an example, the Hermite polynomials can be calculated as the coefficients of $u^n/n!$ in the expansion of $\displaystyle f(x, u)=\exp(-u^2+ 2xu)$, thus the Hermites found from the expansion  $\displaystyle \exp(-u^2+2xu) = \sum\limits_{n=0} H_n(x)u^n/n!$. The expansion is

$$\displaystyle \begin{align}
\exp(-u^2)\exp(2xu)& = \left(1-u^2+\frac{u^4}{2!} -\frac{u^6}{3!} +\cdots \right)\left(1+2xu+4xu^2\frac{u^2}{2!}+8x^3\frac{u^3}{3!} +\cdots \right)\\&= 1+2xu+(4x^2-2)\frac{u^2}{2!}-(8x^3-12x)\frac{u^3}{3!}+\cdots
\end{align}$$

Using SymPy to do this is simpler, the coefficients in $x$ of $u^n/n!$ are the Hermite polynomials

```python
x,u = symbols('x u')
f01 = exp(-u**2 + 2*x*u)          # expand terms to get series to u^10
s = series(f01,u,n = 15)
print('Hermite Polynomials H(n,x) from n =1 to 9')
for n in range(1,10):
    print('H(',n,',x) ', s.coeff(u**n)*factorial(n))  # extract coefficients. factorial is inbuilt in SymPy

```

    Hermite Polynomials H(n,x) from n =1 to 9
    H( 1 ,x)  2*x
    H( 2 ,x)  4*x**2 - 2
    H( 3 ,x)  8*x**3 - 12*x
    H( 4 ,x)  16*x**4 - 48*x**2 + 12
    H( 5 ,x)  32*x**5 - 160*x**3 + 120*x
    H( 6 ,x)  64*x**6 - 480*x**4 + 720*x**2 - 120
    H( 7 ,x)  128*x**7 - 1344*x**5 + 3360*x**3 - 1680*x
    H( 8 ,x)  256*x**8 - 3584*x**6 + 13440*x**4 - 13440*x**2 + 1680
    H( 9 ,x)  512*x**9 - 9216*x**7 + 48384*x**5 - 80640*x**3 + 30240*x

Other polynomials that are frequently met are the Legendre, associated Legendre, Laguerre, associated Laguerre, and Chebychev. The associated Legendre and associated Laguerre polynomials are obtained by differentiating their respective polynomials by $x$, $n$ times, but in these cases, and perhaps in others also, it is easier to use Rodrigues's derivative formula or one of the recursion equations to generate the polynomials. See Margenau & Murphy (1943) and Arkfen (1970) for a full discussion of these polynomials and generating functions.

The generating function definitions, and a derivative formula for integral $n$ and $k$, are shown in the next table. The associated Legendre generating function is omitted because of its complexity.

It is worth noting in passing that generating functions play a more general role in mathematics than is described here and are used to form various distribution functions and series of numbers. The generating function $1/(1 - x)$ produces a square wave -1, +1 repeating sequence; the function $1/(1 - ax)$ produces the sequence of increasing integer powers of $a$; and $x/(1 - x - x^2)$ generates the Fibonacci sequence. For example,

```python
x = symbols('x')
f01 = x/(1-x-x**2)                    # expand terms to get series to u^10
s = series(f01,x,n = 30)
print('Fibonacci from n = 1 to 20')
for n in range(1,21):
    print(' ', s.coeff(x**n),end='')  # extract coefficients.
```

    Fibonacci from n = 1 to 20
      1  1  2  3  5  8  13  21  34  55  89  144  233  377  610  987  1597  2584  4181  6765

Other generating functions can, for example, be used to work out the number of ways of selecting several items from a list, where the same item can be picked many times.


```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

%load_ext autoreload
%autoreload 2

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.pylab as pylab
from simplified_monorotor import Monorotor
import plotting
import testing
import trajectories

pylab.rcParams['figure.figsize'] = 10,10
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

# Feedforward control

The feedforward term captures the acceleration of the target path and adds to the $\bar{u}_1$ calculated by PD controller.

$$
\ddot{z} = \bar{u}_1= k_p(z_t-z)+k_d(z_t-z)+\ddot{z}
$$

#### TODO - Implement PD controller WITH feed forward acceleration

Modify `thrust_control` to incorporate the feedforward term into the PD Controller math.

$$
\begin{align}
e &= z_{\text{target}} - z_{\text{actual}} \\
\dot{e} &= \dot{z}_{\text{target}} - \dot{z}_{\text{actual}} \\
\bar{u}_1 &= k_p e + k_d \dot{e} + \ddot{z}_{\text{ff}} \\
u_1 &= m(g - \bar{u}_1)
\end{align}
$$

```python
class PDController:

    def __init__(self, k_p, k_d, m):
        self.k_p = k_p
        self.k_d = k_d
        self.vehicle_mass = m
        self.g = 9.81

    def tune(self, delta, omega_n):
        self.k_p = omega_n ** 2
        self.k_d = 2 * delta * omega_n

    def thrust_control(self,
                z_target,
                z_actual,
                z_dot_target,
                z_dot_actual,
                z_dot_dot_ff=0.0):

        err = z_target - z_actual
        err_dot = z_dot_target - z_dot_actual
        u_bar = self.k_p * err + self.k_d * err_dot + z_dot_dot_ff
        u = self.vehicle_mass * (self.g - u_bar)

        return u

testing.pd_controller_test(PDController, feed_forward=True)
```

    Tests pass

```python
MASS_ERROR = 1.0

drone = Monorotor()
perceived_mass = drone.m * MASS_ERROR
controller = PDController(5.0, 2.0, perceived_mass)

t, z_path, z_dot_path = trajectories.step()
dt = t[1] - t[0]

history = []
for z_target, z_dot_target in zip(z_path, z_dot_path):
    z_actual = drone.z
    z_dot_actual = drone.z_dot
    u = controller.thrust_control(z_target, z_actual,
                                  z_dot_target, z_dot_actual)
    drone.thrust = u
    drone.advance_state(dt)
    history.append(drone.X)

z_actual = [h[0] for h in history]
plotting.compare_planned_to_actual(z_actual, z_path, t)
```

[Solution](/notebooks/PD%20with%20FF%20Solution.ipynb)


Lambda School Data Science, Unit 2: Predictive Modeling

# Regression & Classification, Module 2

## Objectives
- Go from simple regression (1 feature) to multiple regression (2+ features)
- Use regression metrics: MAE (Mean Absolute Error), RMSE (Root Mean Squared Error), and $R^2$ Score
- Understand how ordinary least squares regression minimizes the sum of squared errors
- Get and plot coefficients
- Explain why overfitting is a problem. Do train/test split

## Setup

```python
# If you're in Colab...
import os, sys
in_colab = 'google.colab' in sys.modules

if in_colab:
    # Install required python packages:
    # pandas-profiling, version >= 2.0
    # plotly, version >= 4.0
    !pip install --upgrade pandas-profiling plotly

    # Pull files from Github repo
    os.chdir('/content')
    !git init .
    !git remote add origin https://github.com/LambdaSchool/DS-Unit-2-Regression-Classification.git
    !git pull origin master

    # Change into directory for module
    os.chdir('module2')
```

```python
# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
```

```python
# Use these functions later

%matplotlib inline
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import numpy as np
import plotly.express as px
import plotly.graph_objs as go
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

plt.style.use('fivethirtyeight')

def regression_3d(df, x, y, z, **kwargs):
    """
    Visualize linear regression in 3D: 2 features + 1 target

    df : Pandas DataFrame
    x : string, feature 1 column in df
    y : string, feature 2 column in df
    z : string, target column in df
    """

    # Plot data
    fig = px.scatter_3d(df, x, y, z, **kwargs)

    # Fit Linear Regression
    features = [x, y]
    target = z
    model = LinearRegression()
    model.fit(df[features], df[target])

    # Define grid of four points in the feature space
    xmin, xmax = df[x].min(), df[x].max()
    ymin, ymax = df[y].min(), df[y].max()
    coords = [[xmin, ymin],
              [xmin, ymax],
              [xmax, ymin],
              [xmax, ymax]]

    # Make predictions for the grid
    Z = model.predict(coords).reshape((2,2), order='F')

    # Plot predictions as a 3D surface (plane)
    fig.add_trace(go.Surface(x=[xmin,xmax], y=[ymin,ymax], z=Z))

    return fig

def regression_residuals(df, feature, target, m, b):
    """
    Visualize linear regression, with residual errors,
    in 2D: 1 feature + 1 target.

    Use the m & b parameters to "fit the model" manually.

    df : Pandas DataFrame
    feature : string, feature column in df
    target : string, target column in df
    m : numeric, slope for linear equation
    b : numeric, intercept for linear requation
    """

    # Plot data
    df.plot.scatter(feature, target)

    # Make predictions
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    # Plot predictions
    plt.plot(x, y_pred)

    # Plot residual errors
    for x, y1, y2 in zip(x, y, y_pred):
        plt.plot((x, x), (y1, y2), color='grey')

    # Print regression metrics
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

def regression_squared_errors(df, feature, target, m, b):
    """
    Visualize linear regression, with squared errors,
    in 2D: 1 feature + 1 target.

    Use the m & b parameters to "fit the model" manually.

    df : Pandas DataFrame
    feature : string, feature column in df
    target : string, target column in df
    m : numeric, slope for linear equation
    b : numeric, intercept for linear requation
    """

    # Plot data
    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    df.plot.scatter(feature, target, ax=ax)

    # Make predictions
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    # Plot predictions
    ax.plot(x, y_pred)

    # Plot squared errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    # Print regression metrics
    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

# Credit: Jake VanderPlas, Python Data Science Handbook, Chapter 5.3
# https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#Validation-curves-in-Scikit-Learn
def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree),
                         LinearRegression(**kwargs))
```

# Predict Elections! 🗳

```python
import pandas as pd
import plotly.express as px

df = pd.read_csv('../data/bread_peace_voting.csv')
px.scatter(
    df,
    x='Average Recent Growth in Personal Incomes',
    y='Incumbent Party Vote Share',
    text='Year',
    title='US Presidential Elections, 1952-2016',
    trendline='ols',
)
```

#### Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:

> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

#### Data sources
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

> Fatalities denotes the cumulative number of American military fatalities per millions of US population the in Korea, Vietnam, Iraq and Afghanistan wars during the presidential terms preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and 2012 elections. —[Hibbs](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 33

```python
# Look at the data
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Incumbent Party Vote Share

      0
      1952
      Stevenson
      Eisenhower
      2.40
      190
      44.60

      1
      1956
      Eisenhower
      Stevenson
      2.89
      0
      57.76

      2
      1960
      Nixon
      Kennedy
      0.85
      0
      49.91

      3
      1964
      Johnson
      Goldwater
      4.21
      1
      61.34

      4
      1968
      Humphrey
      Nixon
      3.02
      146
      49.60

      5
      1972
      Nixon
      McGovern
      3.62
      0
      61.79

      6
      1976
      Ford
      Carter
      1.08
      2
      48.95

      7
      1980
      Carter
      Reagan
      -0.39
      0
      44.70

      8
      1984
      Reagan
      Mondale
      3.86
      0
      59.17

      9
      1988
      Bush, Sr.
      Dukakis
      2.27
      0
      53.94

      10
      1992
      Bush, Sr.
      Clinton
      0.38
      0
      46.55

      11
      1996
      Clinton
      Dole
      1.04
      0
      54.74

      12
      2000
      Gore
      Bush, Jr.
      2.36
      0
      50.27

      13
      2004
      Bush, Jr.
      Kerry
      1.72
      4
      51.24

      14
      2008
      McCain
      Obama
      0.10
      14
      46.32

      15
      2012
      Obama
      Romney
      0.95
      5
      52.00

      16
      2016
      Clinton
      Trump
      0.10
      5
      48.20

```python
# What's the average incumbent vote share?
df['Incumbent Party Vote Share'].mean()
```

    51.82823529411765

```python
px.scatter_3d(
df,
x = 'Average Recent Growth in Personal Incomes',
y = 'US Military Fatalities per Million',
z = 'Incumbent Party Vote Share',
text = 'Year',
title = 'US Presidential Elections')
```

```python
regression_3d(
df,
x = 'Average Recent Growth in Personal Incomes',
y = 'US Military Fatalities per Million',
z = 'Incumbent Party Vote Share',
text = 'Year',
title = 'US Presidential Elections')
```

```python
# 1. Import the appropriate estimator class from Scikit-Learn
from sklearn.linear_model import LinearRegression

# 2. Instantiate this class
model = LinearRegression()

# 3. Arrange X features matrix & y target vector
features = ['Average Recent Growth in Personal Incomes','US Military Fatalities per Million']
target = 'Incumbent Party Vote Share'

X = df[features]
y = df[target]

# 4. Fit the model
model.fit(X,y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

```python
# Get and plot coefficients
model.intercept_
```

    46.88551694148949

```python
model.coef_
```

    array([ 3.40621407, -0.05375223])

```python
model.predict([[0,0]])
```

    array([46.88551694])

```python
model.predict([[1,0]])
```

    array([50.29173101])

```python
model.predict([[1,0]])-model.predict([[0,0]])
```

    array([3.40621407])

```python
model.predict([[2,0]]) - model.predict([[1,0]])
```

    array([3.40621407])

```python
# Use regression metrics - MAE and R^2

# Error for each prediction
y_pred = model.predict(X)

df['Predicted'] = y_pred
df['Error'] = y_pred - y

df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Incumbent Party Vote Share
      Predicted
      Error

      0
      1952
      Stevenson
      Eisenhower
      2.40
      190
      44.60
      44.847507
      0.247507

      1
      1956
      Eisenhower
      Stevenson
      2.89
      0
      57.76
      56.729476
      -1.030524

      2
      1960
      Nixon
      Kennedy
      0.85
      0
      49.91
      49.780799
      -0.129201

      3
      1964
      Johnson
      Goldwater
      4.21
      1
      61.34
      61.171926
      -0.168074

      4
      1968
      Humphrey
      Nixon
      3.02
      146
      49.60
      49.324458
      -0.275542

      5
      1972
      Nixon
      McGovern
      3.62
      0
      61.79
      59.216012
      -2.573988

      6
      1976
      Ford
      Carter
      1.08
      2
      48.95
      50.456724
      1.506724

      7
      1980
      Carter
      Reagan
      -0.39
      0
      44.70
      45.557093
      0.857093

      8
      1984
      Reagan
      Mondale
      3.86
      0
      59.17
      60.033503
      0.863503

      9
      1988
      Bush, Sr.
      Dukakis
      2.27
      0
      53.94
      54.617623
      0.677623

      10
      1992
      Bush, Sr.
      Clinton
      0.38
      0
      46.55
      48.179878
      1.629878

      11
      1996
      Clinton
      Dole
      1.04
      0
      54.74
      50.427980
      -4.312020

      12
      2000
      Gore
      Bush, Jr.
      2.36
      0
      50.27
      54.924182
      4.654182

      13
      2004
      Bush, Jr.
      Kerry
      1.72
      4
      51.24
      52.529196
      1.289196

      14
      2008
      McCain
      Obama
      0.10
      14
      46.32
      46.473607
      0.153607

      15
      2012
      Obama
      Romney
      0.95
      5
      52.00
      49.852659
      -2.147341

      16
      2016
      Clinton
      Trump
      0.10
      5
      48.20
      46.957377
      -1.242623

```python
df['Error'].mean()
```

    -2.507797890918001e-15

```python
df['Error'].abs().mean()
```

    1.3975663494016113

```python
#MEAN ABSOLUTE ERROR (MAE)
from sklearn.metrics import mean_absolute_error
mean_absolute_error(y, y_pred)
```

    1.3975663494016113

```python
#What was the average vote share?
y.mean()
```

    51.82823529411765

```python
#What if we guessed this number for every election?
# [42] * 10
guesses = [y.mean()] * len(y)
```

```python
#How far off would this be on average?
mean_absolute_error(y, guesses)
```

    4.463806228373702

```python
feature = 'Average Recent Growth in Personal Incomes'
regression_residuals(df, feature, target, m=0, b=y.mean())
```

```python
#OLS
#Guess and check

regression_squared_errors(df, feature, target, m=3, b=46)
```

```python
from ipywidgets import interact, fixed

interact(
regression_squared_errors,
    df=fixed(df),
    feature=fixed(feature),
    target = fixed(target),
    m=(-5,5,0.5),
    b=(40,60,0.5)
);
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=5.0, min=-5.0, step=0.5), FloatSlider(value=…

## Linear Algebra

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

\begin{align}
\hat{\beta} = (X^{T}X)^{-1}X^{T}y
\end{align}

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

### Lets calculate our $\beta$ coefficients with numpy!

```python
# This is NOT an objective you'll be tested on. It's just a demo.

# X is a matrix. Add constant for the intercept.
from statsmodels.api import add_constant
X = add_constant(df[feature].values)
print('X')
print(X)

# y is a column vector
y = df[target].values[:, np.newaxis]
print('y')
print(y)

# Least squares solution in code
X_transpose = X.T
X_transpose_X = X_transpose @ X
X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
X_transpose_y = X_transpose @ y
beta_hat = X_transpose_X_inverse @ X_transpose_y

print('Beta Hat')
print(beta_hat)
```

    X
    [[ 1.    2.4 ]
     [ 1.    2.89]
     [ 1.    0.85]
     [ 1.    4.21]
     [ 1.    3.02]
     [ 1.    3.62]
     [ 1.    1.08]
     [ 1.   -0.39]
     [ 1.    3.86]
     [ 1.    2.27]
     [ 1.    0.38]
     [ 1.    1.04]
     [ 1.    2.36]
     [ 1.    1.72]
     [ 1.    0.1 ]
     [ 1.    0.95]
     [ 1.    0.1 ]]
    y
    [[44.6 ]
     [57.76]
     [49.91]
     [61.34]
     [49.6 ]
     [61.79]
     [48.95]
     [44.7 ]
     [59.17]
     [53.94]
     [46.55]
     [54.74]
     [50.27]
     [51.24]
     [46.32]
     [52.  ]
     [48.2 ]]
    Beta Hat
    [[46.49920976]
     [ 2.97417709]]

```python
# Scikit-learn gives the exact same results
from sklearn.linear_model import LinearRegression
model = LinearRegression()
features = ['Average Recent Growth in Personal Incomes']
target = 'Incumbent Party Vote Share'
X = df[features]
y = df[target]
model.fit(X, y)
model.intercept_, model.coef_
```

    (46.499209757741625, array([2.97417709]))

## Overfitting Demo

```python
# Read New York City property sales data, from first 4 months of 2019.
# Dataset has 23040 rows, 21 columns.
df = pd.read_csv('../data/NYC_Citywide_Rolling_Calendar_Sales.csv')
assert df.shape == (23040, 21)

# Change column names. Replace spaces with underscores
df.columns = [col.replace(' ', '_') for col in df]

# Remove symbols from SALE_PRICE string, convert to integer
df['SALE_PRICE'] = (
    df['SALE_PRICE']
    .str.replace('$','')
    .str.replace('-','')
    .str.replace(',','')
    .astype(int)
)

# Keep subset of rows:
# Tribeca neighborhood, Condos - Elevator Apartments,
# 1 unit, sale price more than $1, less than $35 million
mask = (
    (df['NEIGHBORHOOD'].str.contains('TRIBECA')) &
    (df['BUILDING_CLASS_CATEGORY'] == '13 CONDOS - ELEVATOR APARTMENTS') &
    (df['TOTAL_UNITS'] == 1) &
    (df['SALE_PRICE'] > 0) &
    (df['SALE_PRICE'] < 35000000)
)
df = df[mask]

# Data now has 90 rows, 21 columns
assert df.shape == (90, 21)

# Convert SALE_DATE to datetime
df['SALE_DATE'] = pd.to_datetime(df['SALE_DATE'], infer_datetime_format=True)

from ipywidgets import interact
import pandas as pd
from sklearn.linear_model import LinearRegression

# Read New York City property sales data, from first 4 months of 2019.
# Dataset has 23040 rows, 21 columns.
df = pd.read_csv('../data/NYC_Citywide_Rolling_Calendar_Sales.csv')
assert df.shape == (23040, 21)

# Change column names. Replace spaces with underscores
df.columns = [col.replace(' ', '_') for col in df]

# Remove symbols from SALE_PRICE string, convert to integer
df['SALE_PRICE'] = (
    df['SALE_PRICE']
    .str.replace('$','')
    .str.replace('-','')
    .str.replace(',','')
    .astype(int)
)

# Keep subset of rows:
# Tribeca neighborhood, Condos - Elevator Apartments,
# 1 unit, sale price more than $1, less than $35 million
mask = (
    (df['NEIGHBORHOOD'].str.contains('TRIBECA')) &
    (df['BUILDING_CLASS_CATEGORY'] == '13 CONDOS - ELEVATOR APARTMENTS') &
    (df['TOTAL_UNITS'] == 1) &
    (df['SALE_PRICE'] > 0) &
    (df['SALE_PRICE'] < 35000000)
)
df = df[mask]

# Data now has 90 rows, 21 columns
assert df.shape == (90, 21)

# Convert SALE_DATE to datetime
df['SALE_DATE'] = pd.to_datetime(df['SALE_DATE'], infer_datetime_format=True)

# Arrange X features matrix & y target vector
features = ['GROSS_SQUARE_FEET']
target = 'SALE_PRICE'
X = df[features]
y = df[target]
```

```python
from IPython.display import display, HTML
from sklearn.model_selection import train_test_split

# Train/Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=11)

# Repeatedly fit increasingly complex models, and keep track of the scores
polynomial_degrees = range(1, 10, 2)
train_r2s = []
test_r2s = []

for degree in polynomial_degrees:
    model = PolynomialRegression(degree)
    display(HTML(f'Polynomial degree={degree}'))

    model.fit(X_train, y_train)
    train_r2 = model.score(X_train, y_train)
    test_r2 = model.score(X_test, y_test)
    display(HTML(f'<b style="color: blue">Train R2 {train_r2:.2f}</b>'))
    display(HTML(f'<b style="color: red">Test R2 {test_r2:.2f}</b>'))

    plt.scatter(X_train, y_train, color='blue', alpha=0.5)
    plt.scatter(X_test, y_test, color='red', alpha=0.5)
    plt.xlabel(features)
    plt.ylabel(target)

    x_domain = np.linspace(X.min(), X.max())
    curve = model.predict(x_domain)
    plt.plot(x_domain, curve, color='blue')
    plt.show()
    display(HTML('<hr/>'))

    train_r2s.append(train_r2)
    test_r2s.append(test_r2)

display(HTML('Validation Curve'))
plt.plot(polynomial_degrees, train_r2s, color='blue', label='Train')
plt.plot(polynomial_degrees, test_r2s, color='red', label='Test')
plt.xlabel('Model Complexity (Polynomial Degree)')
plt.ylabel('R^2 Score')
plt.legend()
plt.show()
```

#### Jake VanderPlas, [_Python Data Science Handbook,_ Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html#The-Bias-variance-trade-off)


# Loading and Plotting Data

For the first part, we'll be doing linear regression with one variable, and so we'll use only two fields from the daily data set: the normalized high temperature in C, and the total number of bike rentals.  The values for rentals are scaled by a factor of a thousand, given the  difference in magnitude between them and the normalized temperatures.

```python
import pandas as pd

data = pd.read_csv("./data.csv")
temps = data['atemp'].values
rentals = data['cnt'].values / 1000
print(temps)

```

    [0.363625  0.353739  0.189405  0.212122  0.22927   0.233209  0.208839
     0.162254  0.116175  0.150888  0.191464  0.160473  0.150883  0.188413
     0.248112  0.234217  0.176771  0.232333  0.298422  0.25505   0.157833
     0.0790696 0.0988391 0.11793   0.234526  0.2036    0.2197    0.223317
     0.212126  0.250322  0.18625   0.23453   0.254417  0.177878  0.228587
     0.243058  0.291671  0.303658  0.198246  0.144283  0.149548  0.213509
     0.232954  0.324113  0.39835   0.254274  0.3162    0.428658  0.511983
     0.391404  0.27733   0.284075  0.186033  0.245717  0.289191  0.350461
     0.282192  0.351109  0.400118  0.263879  0.320071  0.200133  0.255679
     0.378779  0.366252  0.238461  0.3024    0.286608  0.385668  0.305
     0.32575   0.380091  0.332     0.318178  0.36693   0.410333  0.527009
     0.466525  0.32575   0.409735  0.440642  0.337939  0.270833  0.256312
     0.257571  0.250339  0.257574  0.292908  0.29735   0.257575  0.283454
     0.315637  0.378767  0.542929  0.39835   0.387608  0.433696  0.324479
     0.341529  0.426737  0.565217  0.493054  0.417283  0.462742  0.441913
     0.425492  0.445696  0.503146  0.489258  0.564392  0.453892  0.321954
     0.450121  0.551763  0.5745    0.594083  0.575142  0.578929  0.497463
     0.464021  0.448204  0.532833  0.582079  0.40465   0.441917  0.474117
     0.512621  0.518933  0.525246  0.522721  0.5284    0.523363  0.4943
     0.500629  0.536     0.550512  0.538529  0.527158  0.510742  0.529042
     0.571975  0.5745    0.590296  0.604813  0.615542  0.654688  0.637008
     0.612379  0.61555   0.671092  0.725383  0.720967  0.643942  0.587133
     0.594696  0.616804  0.621858  0.65595   0.727279  0.757579  0.703292
     0.678038  0.643325  0.601654  0.591546  0.587754  0.595346  0.600383
     0.643954  0.645846  0.595346  0.637646  0.693829  0.693833  0.656583
     0.643313  0.637629  0.637004  0.692558  0.654688  0.637008  0.652162
     0.667308  0.668575  0.665417  0.696338  0.685633  0.686871  0.670483
     0.664158  0.690025  0.729804  0.739275  0.689404  0.635104  0.624371
     0.638263  0.669833  0.703925  0.747479  0.74685   0.826371  0.840896
     0.804287  0.794829  0.720958  0.696979  0.690667  0.7399    0.785967
     0.728537  0.729796  0.703292  0.707071  0.679937  0.664788  0.656567
     0.676154  0.715292  0.703283  0.724121  0.684983  0.651521  0.654042
     0.645858  0.624388  0.616167  0.645837  0.666671  0.662258  0.633221
     0.648996  0.675525  0.638254  0.606067  0.630692  0.645854  0.659733
     0.635556  0.647959  0.607958  0.594704  0.611121  0.614921  0.604808
     0.633213  0.665429  0.625646  0.5152    0.544229  0.555361  0.578946
     0.607962  0.609229  0.60213   0.603554  0.6269    0.553671  0.461475
     0.478512  0.490537  0.529675  0.532217  0.550533  0.554963  0.522125
     0.564412  0.572637  0.589042  0.574525  0.575158  0.574512  0.544829
     0.412863  0.345317  0.392046  0.472858  0.527138  0.480425  0.504404
     0.513242  0.523983  0.542925  0.546096  0.517717  0.551804  0.529675
     0.498725  0.503154  0.510725  0.522721  0.513848  0.466525  0.423596
     0.425492  0.422333  0.457067  0.463375  0.472846  0.457046  0.318812
     0.227913  0.321329  0.356063  0.397088  0.390133  0.405921  0.403392
     0.323854  0.362358  0.400871  0.412246  0.409079  0.373721  0.306817
     0.357942  0.43055   0.524612  0.507579  0.451988  0.323221  0.272721
     0.324483  0.457058  0.445062  0.421696  0.430537  0.372471  0.380671
     0.385087  0.4558    0.490122  0.451375  0.311221  0.305554  0.331433
     0.310604  0.3491    0.393925  0.4564    0.400246  0.256938  0.317542
     0.266412  0.253154  0.270196  0.301138  0.338362  0.412237  0.359825
     0.249371  0.245579  0.280933  0.396454  0.428017  0.426121  0.377513
     0.299242  0.279961  0.315535  0.327633  0.279974  0.263892  0.318812
     0.414121  0.375621  0.252304  0.126275  0.119337  0.278412  0.340267
     0.390779  0.340258  0.247479  0.318826  0.282821  0.381938  0.249362
     0.183087  0.161625  0.190663  0.364278  0.275254  0.190038  0.220958
     0.174875  0.16225   0.243058  0.349108  0.294821  0.35605   0.415383
     0.326379  0.272721  0.262625  0.381317  0.466538  0.398971  0.309346
     0.272725  0.264521  0.296426  0.361104  0.266421  0.261988  0.293558
     0.210867  0.101658  0.227913  0.333946  0.351629  0.330162  0.351629
     0.355425  0.265788  0.273391  0.295113  0.392667  0.444446  0.410971
     0.255675  0.268308  0.357954  0.353525  0.34847   0.475371  0.359842
     0.413492  0.303021  0.241171  0.255042  0.3851    0.524604  0.397083
     0.277767  0.35967   0.459592  0.542929  0.548617  0.532825  0.436229
     0.505046  0.464     0.532821  0.538533  0.513258  0.531567  0.570067
     0.486733  0.437488  0.43875   0.315654  0.47095   0.482304  0.375621
     0.421708  0.417287  0.427513  0.461483  0.53345   0.431163  0.390767
     0.426129  0.492425  0.476638  0.436233  0.337274  0.387604  0.431808
     0.487996  0.573875  0.614925  0.598487  0.457038  0.493046  0.515775
     0.542921  0.389504  0.301125  0.405283  0.470317  0.483583  0.452637
     0.377504  0.450121  0.457696  0.577021  0.537896  0.537242  0.590917
     0.584608  0.546737  0.527142  0.557471  0.553025  0.491783  0.520833
     0.544817  0.585238  0.5499    0.576404  0.595975  0.572613  0.551121
     0.566908  0.583967  0.565667  0.580825  0.584612  0.6067    0.627529
     0.642696  0.641425  0.6793    0.672992  0.611129  0.631329  0.607962
     0.566288  0.575133  0.578283  0.525892  0.542292  0.569442  0.597862
     0.648367  0.663517  0.659721  0.597875  0.611117  0.624383  0.599754
     0.594708  0.571975  0.544842  0.654692  0.720975  0.752542  0.724121
     0.652792  0.674254  0.654042  0.594704  0.640792  0.675512  0.786613
     0.687508  0.750629  0.702038  0.70265   0.732337  0.761367  0.752533
     0.804913  0.790396  0.654054  0.664796  0.650271  0.654683  0.667933
     0.666042  0.705196  0.724125  0.755683  0.745583  0.714642  0.613025
     0.549912  0.623125  0.690017  0.70645   0.654054  0.739263  0.734217
     0.697604  0.667933  0.684987  0.662896  0.667308  0.707088  0.722867
     0.751267  0.731079  0.710246  0.697621  0.707717  0.699508  0.667942
     0.638267  0.644579  0.662254  0.676779  0.654037  0.654688  0.2424
     0.618071  0.603554  0.595967  0.601025  0.621854  0.637008  0.6471
     0.618696  0.595996  0.654688  0.66605   0.635733  0.652779  0.6894
     0.702654  0.649     0.661629  0.686888  0.708983  0.655329  0.657204
     0.611121  0.578925  0.565654  0.554292  0.570075  0.579558  0.594083
     0.585867  0.563125  0.55305   0.565067  0.540404  0.532192  0.571971
     0.610488  0.518933  0.502513  0.544179  0.596613  0.607975  0.585863
     0.530296  0.517663  0.512     0.542333  0.599133  0.607975  0.580187
     0.538521  0.419813  0.387608  0.438112  0.503142  0.431167  0.433071
     0.391396  0.508204  0.53915   0.460846  0.450108  0.512625  0.537896
     0.472842  0.456429  0.482942  0.530304  0.558721  0.529688  0.52275
     0.515133  0.467771  0.4394    0.309909  0.3611    0.369942  0.356042
     0.323846  0.329538  0.308075  0.281567  0.274621  0.341891  0.355413
     0.393937  0.421713  0.475383  0.323225  0.281563  0.324492  0.347204
     0.326383  0.337746  0.375621  0.380667  0.364892  0.350371  0.378779
     0.248742  0.257583  0.339004  0.281558  0.289762  0.298422  0.323867
     0.316904  0.359208  0.455796  0.469054  0.428012  0.258204  0.321958
     0.389508  0.390146  0.435575  0.338363  0.297338  0.294188  0.294192
     0.338383  0.369938  0.4015    0.409708  0.342162  0.335217  0.301767
     0.236113  0.259471  0.2589    0.294465  0.220333  0.226642  0.255046
     0.2424    0.2317    0.223487 ]

The plot reveals some degree of correlation between temperature and bike rentals, as one might guess.

```python
import matplotlib.pyplot as plt
%matplotlib inline

plt.scatter(temps, rentals, marker='x', color='red')
plt.xlabel('Normalized Temperature in C')
plt.ylabel('Bike Rentals in 1000s')
```

# Simple Linear Regression

We'll start by implementing the [cost function](https://en.wikipedia.org/wiki/Loss_function) for linear regression, specifically [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error) (MSE).  Intuitively, MSE represents an aggregation of the distances between point's actual y value and what a hypothesis function $h_\theta(x)$ predicted it would be.  That hypothesis function and the cost function $J(\theta)$ are defined as

\begin{align}
h_\theta(x) & = \theta_0 + \theta_1x_1 \\
J(\theta) & = \frac{1}{2m}\sum\limits_{i = 1}^{m}(h_\theta(x^{(i)}) - y^{(i)})^2
\end{align}

where $\theta$ is a vector of feature weights, $x^{(i)}$ is the ith training example, $y^{(i)}$ is that example's y value, and $x_j$ is the value for its jth feature.

```python
import numpy as np

def compute_cost(X, y, theta):

    return np.sum(np.square(np.matmul(X, theta) - y)) / (2 * len(y))
```

Before computing the cost with an initial guess for $\theta$, a column of 1s is prepended onto the input data.  This allows us to vectorize the cost function, as well as make it usable for multiple linear regression later.  This first value $\theta_0$ now behaves as a constant in the cost function.

```python
# [T0 t1]
# X
# [T0 T1] X [1]
#           [X]
theta = np.zeros(2)
X = np.column_stack((np.ones(len(temps)), temps))
y = rentals
cost = compute_cost(X, y, theta)

print('theta:', theta)
print('cost:', cost)
print(X)
```

    theta: [0. 0.]
    cost: 12.018406441176468
    [[1.       0.363625]
     [1.       0.353739]
     [1.       0.189405]
     ...
     [1.       0.2424  ]
     [1.       0.2317  ]
     [1.       0.223487]]

We'll now minimize the cost using the [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) algorithm.  Intuitively, gradient descent takes small, linear hops down the slope of a function in each feature dimension, with the size of each hop determined by the partial derivative of the cost function with respect to that feature and a learning rate multiplier $\alpha$.  If tuned properly, the algorithm converges on a global minimum by iteratively adjusting feature weights $\theta$ of the cost function, as shown here for two feature dimensions:

\begin{align}
\theta_0 & := \theta_0 - \alpha\frac{\partial}{\partial\theta_0} J(\theta_0,\theta_1) \\
\theta_1 & := \theta_1 - \alpha\frac{\partial}{\partial\theta_1} J(\theta_0,\theta_1)
\end{align}

The update rule each iteration then becomes:

\begin{align}
\theta_0 & := \theta_0 - \alpha\frac{1}{m} \sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)}) \\
\theta_1 & := \theta_1 - \alpha\frac{1}{m} \sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})x_1^{(i)} \\
\end{align}

See [here](http://mccormickml.com/2014/03/04/gradient-descent-derivation/) for a more detailed explanation of how the update equations are derived.

```python
def gradient_descent(X, y, alpha, iterations):
    theta = np.zeros(2)
    m = len(y)

    for i in range(iterations):
        t0 = theta[0] - (alpha / m) * np.sum(np.dot(X, theta) - y)
        t1 = theta[1] - (alpha / m) * np.sum((np.dot(X, theta) - y) * X[:,1])
        theta = np.array([t0, t1])

    return theta

iterations = 5000
alpha = 0.1

theta = gradient_descent(X, y, alpha, iterations)
cost = compute_cost(X, y, theta)

print("theta:", theta)
print('cost:', compute_cost(X, y, theta))
```

    theta: [0.94588081 7.50171673]
    cost: 1.1275869258439812

We can examine the values of $\theta$ chosen by the algorithm using a few different visualizations, first by plotting $h_\theta(x)$ against the input data.  The results show the expected correlation between temperature and rentals.

```python
plt.scatter(temps, rentals, marker='x', color='red')
plt.xlabel('Normalized Temperature in C')
plt.ylabel('Bike Rentals in 1000s')
samples = np.linspace(min(temps), max(temps))
plt.plot(samples, theta[0] + theta[1] * samples)
```

A surface plot is a better illustration of how gradient descent approaches a global minimum, plotting the values for $\theta$ against their associated cost.  This requires a bit more code than an implementation in Octave / MATLAB, largely due to how the input data is generated and fed to the surface plot function.

```python
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm

Xs, Ys = np.meshgrid(np.linspace(-5, 5, 50), np.linspace(-40, 40, 50))
Zs = np.array([compute_cost(X, y, [t0, t1]) for t0, t1 in zip(np.ravel(Xs), np.ravel(Ys))])
Zs = np.reshape(Zs, Xs.shape)

fig = plt.figure(figsize=(7,7))
ax = fig.gca(projection="3d")
ax.set_xlabel(r't0')
ax.set_ylabel(r't1')
ax.set_zlabel(r'cost')
ax.view_init(elev=25, azim=40)    # rotate 3D image
ax.plot_surface(Xs, Ys, Zs, cmap=cm.rainbow)
```

Finally, a countour plot reveals slices of that surface plot in 2D space, and can show the resulting $\theta$ values sitting exactly at the global minimum.

```python
ax = plt.figure().gca()
ax.plot(theta[0], theta[1], 'r*')
plt.contour(Xs, Ys, Zs, np.logspace(-3, 3, 15))
#plt.contour(Xs, Ys, Zs)
```

# Multiple Linear Regression

First, we reload the data and add two more features, humidity and windspeed.

Before implementing gradient descent for multiple variables, we'll also apply [feature scaling](https://en.wikipedia.org/wiki/Feature_scaling) to normalize feature values, preventing any one of them from disproportionately influencing the results, as well as helping gradient descent converge more quickly.  In this case, each feature value is adjusted by subtracting the mean and dividing the result by the standard deviation of all values for that feature:

$$
z = \frac{x - \mu}{\sigma}
$$

More details on feature scaling and normalization can be found [here](http://sebastianraschka.com/Articles/2014_about_feature_scaling.html).

```python
data
```

      instant
      dteday
      season
      yr
      mnth
      holiday
      weekday
      workingday
      weathersit
      temp
      atemp
      hum
      windspeed
      casual
      registered
      cnt

      0
      1
      2011-01-01
      1
      0
      1
      0
      6
      0
      2
      0.344167
      0.363625
      0.805833
      0.160446
      331
      654
      985

      1
      2
      2011-01-02
      1
      0
      1
      0
      0
      0
      2
      0.363478
      0.353739
      0.696087
      0.248539
      131
      670
      801

      2
      3
      2011-01-03
      1
      0
      1
      0
      1
      1
      1
      0.196364
      0.189405
      0.437273
      0.248309
      120
      1229
      1349

      3
      4
      2011-01-04
      1
      0
      1
      0
      2
      1
      1
      0.200000
      0.212122
      0.590435
      0.160296
      108
      1454
      1562

      4
      5
      2011-01-05
      1
      0
      1
      0
      3
      1
      1
      0.226957
      0.229270
      0.436957
      0.186900
      82
      1518
      1600

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      726
      727
      2012-12-27
      1
      1
      12
      0
      4
      1
      2
      0.254167
      0.226642
      0.652917
      0.350133
      247
      1867
      2114

      727
      728
      2012-12-28
      1
      1
      12
      0
      5
      1
      2
      0.253333
      0.255046
      0.590000
      0.155471
      644
      2451
      3095

      728
      729
      2012-12-29
      1
      1
      12
      0
      6
      0
      2
      0.253333
      0.242400
      0.752917
      0.124383
      159
      1182
      1341

      729
      730
      2012-12-30
      1
      1
      12
      0
      0
      0
      1
      0.255833
      0.231700
      0.483333
      0.350754
      364
      1432
      1796

      730
      731
      2012-12-31
      1
      1
      12
      0
      1
      1
      2
      0.215833
      0.223487
      0.577500
      0.154846
      439
      2290
      2729

731 rows × 16 columns

```python
def feature_normalize(X):
    '''
    make sure features are on similar scale
    --> gradient decense will converge faster

    '''
    n_features = X.shape[1]
    means = np.array([np.mean(X[:,i]) for i in range(n_features)])
    stddevs = np.array([np.std(X[:,i]) for i in range(n_features)])
    normalized = (X - means) / stddevs

    return normalized
'''
X = data.values
X = np.matrix(X)
X = feature_normalize(X)
X = np.column_stack((np.ones(len(X)), X))
data_y = pd.read_csv("E:\DATAs\data.csv",usecols=['cnt'])
y = data_y['cnt'].values/1000
'''
data2 = pd.read_csv("./data.csv",usecols=['atemp', 'hum', 'windspeed'])
X = data2.values
#X = np.matrix(X)
#X = data.as_matrix(columns=['atemp', 'hum', 'windspeed'])
print(np.shape(X))
X = feature_normalize(X)
X = np.column_stack((np.ones(len(X)), X))

y = data['cnt'].values / 1000

```

    (731, 3)

```python
print(X)
print(X.shape)
```

    [[ 1.         -0.67994602  1.25017133 -0.38789169]
     [ 1.         -0.74065231  0.47911298  0.74960172]
     [ 1.         -1.749767   -1.33927398  0.74663186]
     ...
     [ 1.         -1.42434419  0.87839173 -0.85355213]
     [ 1.         -1.49004895 -1.01566357  2.06944426]
     [ 1.         -1.54048197 -0.35406086 -0.46020122]]
    (731, 4)

The next step is to implement gradient descent for any number of features.  Fortunately, the update step generalizes easily, and can be vectorized to avoid iterating through $\theta_j$ values as might be suggested by the single variable implementation above:

$$
\theta_j := \theta_j - \alpha\frac{1}{m} \sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})x_j^{(i)}
$$

```python
def gradient_descent_multi(X, y, theta, alpha, iterations):
    theta = np.zeros(X.shape[1])
    m = len(X)

    for i in range(iterations):
        gradient = (1/m) * np.matmul(X.T, np.matmul(X, theta) - y)
        theta = theta - alpha * gradient

    return theta

theta = gradient_descent_multi(X, y, theta, alpha, iterations)
cost = compute_cost(X, y, theta)

print('theta:', theta)
print('cost', cost)
```

    theta: [ 4.50434884  1.22203893 -0.45083331 -0.34166068]
    cost 1.0058709247119848

Unfortunately, it's now more difficult to evaluate the results visually, but we can check them a totally different method of calculating the answer, the [normal equation](http://eli.thegreenplace.net/2014/derivation-of-the-normal-equation-for-linear-regression/).  This solves directly for the solution without iteration specifying an $\alpha$ value, although it begins to perform worse than gradient descent with large (10,000+) numbers of features.

$$
\theta = (X^TX)^{-1}X^Ty
$$

```python
from numpy.linalg import inv

def normal_eq(X, y):
    return inv(X.T.dot(X)).dot(X.T).dot(y)

theta = normal_eq(X, y)
cost = compute_cost(X, y, theta)

print('theta:', theta)
print('cost:', cost)
```

    theta: [ 4.50434884  1.22203893 -0.45083331 -0.34166068]
    cost: 1.0058709247119846

The $\theta$ values and costs for each implementation are identical, so we can have a high degree of confidence they are correct.

## Linear Regression in Tensorflow

Tensorflow offers significantly higher-level abstractions to work with, representing the algorithm as a computational graph.  It has a built-in gradient descent optimizer that can minimize the cost function without us having to define the gradient manually.

We'll begin by reloading the data and adapting it to more Tensorflow-friendly data structures and terminology.  Features are still normalized as before, but the added column of 1s is absent: the constant is treated separately as a *bias* variable, the previous $\theta$ values are now *weights*.

```python
import tensorflow as tf

X = data.as_matrix(columns=['atemp', 'hum', 'windspeed'])
X = feature_normalize(X)
y = data['cnt'].values / 1000
y = y.reshape((-1, 1))

m = X.shape[0]
n = X.shape[1]

examples = tf.placeholder(tf.float32, [m,n])
labels = tf.placeholder(tf.float32, [m,1])
weights = tf.Variable(tf.zeros([n,1], dtype=np.float32), name='weight')
bias = tf.Variable(tf.zeros([1], dtype=np.float32), name='bias')
```

The entire gradient descent occurs below in only three lines of code.  All that's needed is to define the hypothesis and cost functions, and then a gradient descent optimizer to find the minimum.

```python
hypothesis = tf.add(tf.matmul(examples, weights), bias)
cost = tf.reduce_sum(tf.square(hypothesis - y)) / (2 * m)
optimizer = tf.train.GradientDescentOptimizer(alpha).minimize(cost)
```

The graph is now ready to use, and all the remains is to start up a session, run the optimizer iteratively, and check the results.

```python
with tf.Session() as sess:
    init = tf.global_variables_initializer()
    sess.run(init)

    for i in range(1, iterations):
        sess.run(optimizer, feed_dict={
            examples: X,
            labels: y
        })

    print('bias:', sess.run(bias))
    print('weights:', sess.run(weights))
```

The bias and weight values are identical to the $\theta$ values calculated in both implementations previously, so the Tensorflow implementation of the algorithm looks correct.


# Table of Contents

1  Determinant

1.1  Definition of determinant

1.2  Some interpretations of determinant

1.3  Some properties of determinant (important)

# Determinant

```julia
using Pkg
Pkg.activate(pwd())
Pkg.instantiate()
```

      Activating environment at `~/Documents/github.com/ucla-biostat216-2021fall.github.io/slides/09-det/Project.toml`

```julia
using LinearAlgebra, Plots, Symbolics
```

We review some basic facts about matrix determinant.

## Definition of determinant

- The **determinant** of a square matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$ is
$$
    \det (\mathbf{A}) = \sum (-1)^{\phi(j_1,\ldots,j_n)} \prod_{i=1}^n a_{ij_i},
$$
where the summation is over all permutation $(j_1, \ldots, j_n)$ of the set of integers $(1,\ldots,n)$ and $\phi(j_1,\ldots,j_n)$ is the number of transposition to change $(1,\ldots,n)$ to $(j_1,\ldots,j_n)$. $(-1)^{\phi(j_1,\ldots,j_n)}$ is also called the **sign of permutation**.

- Examples: $n = 2$ and 3.
$$
    \det \begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix} = (-1)^{\phi(1,2)} a_{11} a_{22} + (-1)^{\phi(2,1)} a_{12} a_{21} = a_{11} a_{22} - a_{12} a_{21}.
$$

```julia
# n = 2
@variables A[1:2, 1:2]
```

    1-element Vector{Symbolics.Arr{Num, 2}}:
     A[1:2,1:2]

```julia
det(A) |> Symbolics.scalarize
```

\begin{equation}
A{{_1}}ˏ{_1} A{{_2}}ˏ{_2} - A{{_1}}ˏ{_2} A{{_2}}ˏ{_1}
\end{equation}

```julia
# n = 3
@variables A[1:3, 1:3]
```

    1-element Vector{Symbolics.Arr{Num, 2}}:
     A[1:3,1:3]

```julia
det(A) |> Symbolics.scalarize |> expand
```

\begin{equation}
A{{_1}}ˏ{_2} A{{_2}}ˏ{_3} A{{_3}}ˏ{_1} + A{{_1}}ˏ{_1} A{{_2}}ˏ{_2} A{{_3}}ˏ{_3} + A{{_1}}ˏ{_3} A{{_2}}ˏ{_1} A{{_3}}ˏ{_2} - A{{_1}}ˏ{_3} A{{_2}}ˏ{_2} A{{_3}}ˏ{_1} - A{{_1}}ˏ{_1} A{{_2}}ˏ{_3} A{{_3}}ˏ{_2} - A{{_1}}ˏ{_2} A{{_2}}ˏ{_1} A{{_3}}ˏ{_3}
\end{equation}

```julia
# n = 4
@variables A[1:4, 1:4]
```

    1-element Vector{Symbolics.Arr{Num, 2}}:
     A[1:4,1:4]

```julia
det(A) |> Symbolics.scalarize |> expand
```

\begin{equation}
A{{_1}}ˏ{_2} A{{_2}}ˏ{_3} A{{_3}}ˏ{_1} A{{_4}}ˏ{_4} + A{{_1}}ˏ{_3} A{{_2}}ˏ{_4} A{{_3}}ˏ{_1} A{{_4}}ˏ{_2} + A{{_1}}ˏ{_1} A{{_2}}ˏ{_2} A{{_3}}ˏ{_3} A{{_4}}ˏ{_4} + A{{_1}}ˏ{_1} A{{_2}}ˏ{_4} A{{_3}}ˏ{_2} A{{_4}}ˏ{_3} + A{{_1}}ˏ{_4} A{{_2}}ˏ{_1} A{{_3}}ˏ{_3} A{{_4}}ˏ{_2} + A{{_1}}ˏ{_1} A{{_2}}ˏ{_3} A{{_3}}ˏ{_4} A{{_4}}ˏ{_2} + A{{_1}}ˏ{_4} A{{_2}}ˏ{_3} A{{_3}}ˏ{_2} A{{_4}}ˏ{_1} + A{{_1}}ˏ{_3} A{{_2}}ˏ{_1} A{{_3}}ˏ{_2} A{{_4}}ˏ{_4} + A{{_1}}ˏ{_3} A{{_2}}ˏ{_2} A{{_3}}ˏ{_4} A{{_4}}ˏ{_1} + A{{_1}}ˏ{_2} A{{_2}}ˏ{_4} A{{_3}}ˏ{_3} A{{_4}}ˏ{_1} + A{{_1}}ˏ{_2} A{{_2}}ˏ{_1} A{{_3}}ˏ{_4} A{{_4}}ˏ{_3} + A{{_1}}ˏ{_4} A{{_2}}ˏ{_2} A{{_3}}ˏ{_1} A{{_4}}ˏ{_3} - A{{_1}}ˏ{_3} A{{_2}}ˏ{_1} A{{_3}}ˏ{_4} A{{_4}}ˏ{_2} - A{{_1}}ˏ{_2} A{{_2}}ˏ{_4} A{{_3}}ˏ{_1} A{{_4}}ˏ{_3} - A{{_1}}ˏ{_4} A{{_2}}ˏ{_3} A{{_3}}ˏ{_1} A{{_4}}ˏ{_2} - A{{_1}}ˏ{_1} A{{_2}}ˏ{_2} A{{_3}}ˏ{_4} A{{_4}}ˏ{_3} - A{{_1}}ˏ{_3} A{{_2}}ˏ{_2} A{{_3}}ˏ{_1} A{{_4}}ˏ{_4} - A{{_1}}ˏ{_4} A{{_2}}ˏ{_2} A{{_3}}ˏ{_3} A{{_4}}ˏ{_1} - A{{_1}}ˏ{_2} A{{_2}}ˏ{_1} A{{_3}}ˏ{_3} A{{_4}}ˏ{_4} - A{{_1}}ˏ{_1} A{{_2}}ˏ{_4} A{{_3}}ˏ{_3} A{{_4}}ˏ{_2} - A{{_1}}ˏ{_3} A{{_2}}ˏ{_4} A{{_3}}ˏ{_2} A{{_4}}ˏ{_1} - A{{_1}}ˏ{_4} A{{_2}}ˏ{_1} A{{_3}}ˏ{_2} A{{_4}}ˏ{_3} - A{{_1}}ˏ{_1} A{{_2}}ˏ{_3} A{{_3}}ˏ{_2} A{{_4}}ˏ{_4} - A{{_1}}ˏ{_2} A{{_2}}ˏ{_3} A{{_3}}ˏ{_4} A{{_4}}ˏ{_1}
\end{equation}

## Some interpretations of determinant

- Interpretation of the (absolute value of) determinant as the **volume of the parallelotope** defined by the columns of the matrix. For example, if $\mathbf{X} \in \mathbb{R}^2$ has two columns $\mathbf{x}_1$ and $\mathbf{x}_2$, then
\begin{eqnarray*}
	\text{area} &=& bh = \|\mathbf{x}_1\|\|\mathbf{x}_2\| \sin(\theta) \\
	&=& \|\mathbf{x}_1\| \|\mathbf{x}_2\| \sqrt{1 - \left( \frac{\langle \mathbf{x}_1, \mathbf{x}_2 \rangle}{\|\mathbf{x}_1\| \|\mathbf{x}_2\|} \right)^2} \\
	&=& \sqrt{\|\mathbf{x}_1\|^2 \|\mathbf{x}\|^2 - (\langle \mathbf{x}_1, \mathbf{x}_2\rangle)^2} \\
	&=& \sqrt{(x_{11}^2 + x_{12}^2)(x_{21}^2+x_{22}^2) - (x_{11}x_{21} + x_{12}x_{22})^2} \\
	&=& |x_{11} x_{22} - x_{12} x_{21}| \\
	&=& |\det(\mathbf{X})|.
\end{eqnarray*}

- Another interpretation of the determinant is the volume changing factor when operating on a set in $\mathbb{R}^n$. $\text{vol}(f(S)) = |\det(\mathbf{A})| \text{vol}(S)$ where $f: \mathbb{R}^n \mapsto \mathbb{R}^n$ is the linear mapping defined by $\mathbf{A}$.

- Recall that for differentiable function $f: \mathbb{R}^n \mapsto \mathbb{R}^n$,  the **Jacobian matrix** $\operatorname{D} f(\mathbf{x}) \in \mathbb{R}^{n \times n}$ is
$$
\operatorname{D} f(\mathbf{x}) = \begin{pmatrix}
\frac{\partial f_1}{\partial x_1} (\mathbf{x}) & \frac{\partial f_1}{\partial x_2} (\mathbf{x}) & \cdots & \frac{\partial f_1}{\partial x_n} (\mathbf{x}) \\
\frac{\partial f_2}{\partial x_1} (\mathbf{x}) & \frac{\partial f_2}{\partial x_2} (\mathbf{x}) & \cdots & \frac{\partial f_2}{\partial x_n} (\mathbf{x}) \\
\vdots & \vdots & & \vdots \\
\frac{\partial f_n}{\partial x_1} (\mathbf{x}) & \frac{\partial f_n}{\partial x_2} (\mathbf{x}) & \cdots & \frac{\partial f_n}{\partial x_n} (\mathbf{x})
\end{pmatrix} = \begin{pmatrix}
\nabla f_1(\mathbf{x})' \\
\nabla f_2(\mathbf{x})' \\
\vdots \\
\nabla f_n(\mathbf{x})'
\end{pmatrix}.
$$
Its determinant, the **Jacobian determinant**, appears in the higher-dimensional version of **integration by substitution** or **change of variable**
$$
\int_{f(U)} \phi(\mathbf{v}) \, \operatorname{d} \mathbf{v} = \int_U \phi(f(\mathbf{u})) | \det \operatorname{D} f(\mathbf{u})| \, \operatorname{d} \mathbf{u}
$$
for function $\phi: \mathbb{R}^n \mapsto \mathbb{R}$. This result will be used in transformation of random variables in 202A.

    For an example of $n=1$, an indefinite integral can be transformed to a definite integral over box [-1,1] via change of variable $v = u / (1-u^2)$:
$$
\int_{-\infty}^\infty f(v) \, dv = \int_{-1}^1 f\left(\frac{u}{1-u^2}\right) \frac{1+u^2}{(1-u^2)^2} \, du.
$$

## Some properties of determinant (important)

- The determinant of a **lower or upper triangular matrix** $\mathbf{A}$ is the product of the diagonal elements $\prod_{i=1}^n a_{ii}$. (Why?)

- Any square matrix $\mathbf{A}$ is singular if and only if $\det(\mathbf{A}) = 0$.

- Product rule: $\det(\mathbf{A} \mathbf{B}) = \det(\mathbf{A}) \det(\mathbf{B})$.

    Product rule is extremely useful. For example, computer calculates the determinant of a square matrix $\mathbf{A}$ by first computing the LU decomposition $\mathbf{A} = \mathbf{L} \mathbf{U}$ and then $\det(\mathbf{A}) = \det(\mathbf{L}) \det(\mathbf{U})$.

- Determinant of an orthogonal matrix is 1 (**rotation**) or -1 (**reflection**).
    This classifies orthogonal matrices into two classes: rotations and reflections.

```julia
# a rotator
θ = π/4
A = [cos(θ) -sin(θ);
    sin(θ) cos(θ)]
```

    2×2 Matrix{Float64}:
     0.707107  -0.707107
     0.707107   0.707107

```julia
det(A)
```

    1.0

```julia
# a reflector
B = [cos(θ) sin(θ);
    sin(θ) -cos(θ)]
```

    2×2 Matrix{Float64}:
     0.707107   0.707107
     0.707107  -0.707107

```julia
B'B
```

    2×2 Matrix{Float64}:
     1.0  0.0
     0.0  1.0

```julia
det(B)
```

    -1.0

```julia
# 3 points for a triangle
X = [1 1 2 1; 1 3 1 1]
# rotation
Xrot = A * X
# reflection
Xref = B * X
```

    2×4 Matrix{Float64}:
      1.41421       2.82843  2.12132    1.41421
     -1.11022e-16  -1.41421  0.707107  -1.11022e-16

```julia
plt = plot(X[1, :], X[2, :], color = :blue,
    legend = :none, xlims = (-3, 3), ylims = (-3, 3),
    xticks = -3:1:3, yticks = -3:1:3,
    framestyle = :origin,
    aspect_ratio = :equal)
plot!(plt, Xrot[1, :], Xrot[2, :])
plot!(plt, Xref[1, :], Xref[2, :],
    annotations = [(-1.5, 2, "rotation"), (2, -1.75, "reflection")])
```

-  $\det(\mathbf{A}') = \det(\mathbf{A})$.

- $\det(\mathbf{A}^{-1}) = 1/\det(\mathbf{A})$.

- $\det(c\mathbf{A}) = c^n \det(\mathbf{A})$.

- Determinant of a permutation matrix is the sign of the permutation.

- For $\mathbf{A}$ and $\mathbf{D}$ square and nonsingular,
\begin{eqnarray*}
	\det \left( \begin{pmatrix}
	\mathbf{A} & \mathbf{B} \\
	\mathbf{C} & \mathbf{D}
	\end{pmatrix} \right) = \det (\mathbf{A}) \det (\mathbf{D} - \mathbf{C} \mathbf{A}^{-1} \mathbf{B}) = \det(\mathbf{D}) \det(\mathbf{A} - \mathbf{B} \mathbf{D}^{-1} \mathbf{C}).
\end{eqnarray*}

    Proof: Take determinant on the both sides of the matrix identity
\begin{eqnarray*}
	\begin{pmatrix}
	\mathbf{A} & \mathbf{0} \\
	\mathbf{0} & \mathbf{D} - \mathbf{C} \mathbf{A}^{-1} \mathbf{B}
	\end{pmatrix} = \begin{pmatrix}
	\mathbf{I} & \mathbf{0} \\ - \mathbf{C} \mathbf{A}^{-1} & \mathbf{I}
	\end{pmatrix} \begin{pmatrix}
	\mathbf{A} & \mathbf{B} \\
	\mathbf{C} & \mathbf{D}
	\end{pmatrix} \begin{pmatrix}
	\mathbf{I} & - \mathbf{A}^{-1} \mathbf{B} \\
	\mathbf{0} & \mathbf{I}
	\end{pmatrix}.
\end{eqnarray*}


```python
from sympy import sin
from sympy import cos
from math import pi
```

```python
#*****************************************************************************************************************************
# Felipe Coral 161003611
# Brayan Saldarriaga 161003640
#*****************************************************************************************************************************

Tabla =[[0,0,0,0,0,0],
        [0,0,0,0,0,0],
        [0,0,0,0,0,0],
        [0,0,0,0,0,0],
        [0,0,0,0,0,0],
        [0,0,0,0,0,0],
        [0,0,0,0,0,0],
        [0,0,0,0,0,0]]
x=0
for i in range(8):
    for j in range(6):
        if j==0:
            Tabla[i][j]=cos(x)
        if j==1:
            Tabla[i][j]=cos(0)-x*sin(0)
        if j==2:
            Tabla[i][j]=cos(0)-x*sin(0)-((x**2)/2)*cos(0)
        if j==3:
            Tabla[i][j]=cos(0)-x*sin(0)-((x**2)/2)*cos(0)+((x**3)/6)*sin(0)
        if j==4:
            Tabla[i][j]=cos(0)-x*sin(0)-((x**2)/2)*cos(0)+((x**3)/6)*sin(0)+((x**4)/24)*cos(0)
        if j==5:
            Tabla[i][j]=cos(0)-x*sin(0)-((x**2)/2)*cos(0)+((x**3)/6)*sin(0)+((x**4)/24)*cos(0)-((x**5)/120)*sin(0)
    if x==0:
        x=x+0.00001
    else:
        x=x*10
```

```python
print("x       Exacto           Aprx1     Aprox.2              Aprox.3           Aprox.4            Aprox.5")
print("0      ",Tabla[0][0],"               ",Tabla[0][1],"  ",Tabla[0][2],"                ",Tabla[0][3],"                ",Tabla[0][4],"               ",Tabla[0][5])
print("0.00001",Tabla[1][0],Tabla[1][1],"  ",Tabla[1][2],"",Tabla[1][3],"",Tabla[1][4],Tabla[1][5])
print("0.0001 ",Tabla[2][0],Tabla[2][1],"  ",Tabla[2][2],"",Tabla[2][3],"",Tabla[2][4],Tabla[2][5])
print("0.001  ",Tabla[3][0],Tabla[3][1],"  ",Tabla[3][2],"",Tabla[3][3],"",Tabla[3][4],Tabla[3][5])
print("0.01   ",Tabla[4][0],Tabla[4][1],"  ",Tabla[4][2],"",Tabla[4][3],"",Tabla[4][4],Tabla[4][5])
print("0.1    ",Tabla[5][0],Tabla[5][1],"  ",Tabla[5][2],"",Tabla[5][3],"",Tabla[5][4],Tabla[5][5])
print("1      ",Tabla[6][0],Tabla[6][1],"  ",Tabla[6][2],"",Tabla[6][3],"",Tabla[6][4],Tabla[6][5])
print("10    ",Tabla[7][0],Tabla[7][1],"  ",Tabla[7][1],"               ",Tabla[7][3]," ",Tabla[7][4],"",Tabla[7][5])
```

    x       Exacto           Aprx1     Aprox.2              Aprox.3           Aprox.4            Aprox.5
    0       1                 1    1                  1                  1                 1
    0.00001 0.999999999950000 1    0.999999999950000  0.999999999950000  0.999999999950000 0.999999999950000
    0.0001  0.999999995000000 1    0.999999995000000  0.999999995000000  0.999999995000000 0.999999995000000
    0.001   0.999999500000042 1    0.999999500000000  0.999999500000000  0.999999500000042 0.999999500000042
    0.01    0.999950000416665 1    0.999950000000000  0.999950000000000  0.999950000416667 0.999950000416667
    0.1     0.995004165278026 1    0.995000000000000  0.995000000000000  0.995004166666667 0.995004166666667
    1       0.540302305868140 1    0.500000000000000  0.500000000000000  0.541666666666667 0.541666666666667
    10     -0.839071529076452 1    1                 -49.0000000000000   367.666666666667  367.666666666667

```python

```

```python

```


# 附录A：关于布莱克-斯科尔斯-默顿模型的一些有用的推导式

BSM价格表达式如下:
$$C(S,K,\tau,\sigma,r)=SN(d_1)-Ke^{-r\tau}N(d_2)$$

$$d_1=\frac{\ln\left(\frac{S_F}{K}\right)+\frac{\sigma^2}{2}\tau}{\sigma\sqrt{\tau}}$$

$$d_2=\frac{\ln\left(\frac{S_F}{K}\right)-\frac{\sigma^2}{2}\tau}{\sigma\sqrt{\tau}}$$

$$S_F=e^{r\tau}S$$

$$N(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{x}{e^{-\frac{1}{2}y^2}\mathrm{d}y}$$

## 有用的推导式

$$N^{\prime}(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x^2}$$

$$\frac{\partial C}{\partial\sigma}=\frac{1}{\sqrt{2\pi}}Se^{-\frac{1}{2}d_1^2}\sqrt{\tau}$$

$$KN^{\prime}(d_2)=S_FN^{\prime}(d_1)$$

$$\frac{\partial C}{\partial S}=N(d_1)$$

$$\frac{\partial d_{1,2}}{\partial K}=\frac{-1}{K\sigma\sqrt{\tau}}$$

$$\frac{\partial C}{\partial K}=-e^{-r\tau}N(d_2)$$

$$\frac{\partial d_{1,2}}{\partial\sigma}=\frac{-1}{\sigma^2\sqrt{\tau}}\ln\left(\frac{S_F}{K}\right)\pm\frac{1}{2}\sqrt{\tau}$$

## 证明
###  准备

```python
import sympy as sy
import sympy.stats as systats
sy.init_printing()
```

```python
S, K, tau, sigma, r = sy.symbols("S K tau sigma r")
def S_F(tau, r, s):
    return sy.exp(tau*r)*s
def d1(S, K, tau, sigma, r):
    return (sy.ln(S_F(tau, r, S) / K) + 1/2 * sigma ** 2 * tau) / (sigma * sy.sqrt(tau))
def d2(S, K, tau, sigma, r):
    return (sy.ln(S_F(tau, r, S) / K) - 1/2 * sigma ** 2 * tau) / (sigma * sy.sqrt(tau))
def N(x):
    y = sy.Symbol('y')
    return 1/sy.sqrt(2*sy.pi)*sy.Integral(sy.exp(-0.5*y**2), (y, -sy.oo , x))
def C(S, K, tau, sigma, r):
    return S*N(d1(S, K, tau, sigma, r))-K*sy.exp(-r*tau)*N(d2(S, K, tau, sigma, r))
```

### 1. $$N(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{x}{e^{-\frac{1}{2}y^2}\mathrm{d}y}$$

```python
x = sy.Symbol("x")
N(x)
```

### 2. $$N^{\prime}(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x^2}$$

```python
sy.diff(N(x),x)
```

### 3.  $$KN^{\prime}(d_2)=S_FN^{\prime}(d_1)$$

```python
def Nprime(x):
    return 1/sy.sqrt(2*sy.pi)*sy.exp(-1/2*x**2)
Nprime(x)
```

```python
K*Nprime(d2(S, K, tau, sigma, r))
```

```python
S*sy.exp(r*tau)*Nprime(d1(S, K, tau, sigma, r))
```

$$KN^\prime(d_2)=\frac{K}{\sqrt{2\pi}}e^{-\frac{1}{2}d_2^2}=\frac{K}{\sqrt{2\pi}}e^{-\frac{1}{2}\frac{\left[\ln\left(\frac{S_F}{K}\right)-\frac{\sigma^2}{2}\tau\right]^2}{\sigma^2\tau}}$$

$$S_FN^\prime(d_1)=\frac{S_F}{\sqrt{2\pi}}e^{-\frac{1}{2}d_1^2}=\frac{S_F}{\sqrt{2\pi}}e^{-\frac{1}{2}\frac{\left[\ln\left(\frac{S_F}{K}\right)+\frac{\sigma^2}{2}\tau\right]^2}{\sigma^2\tau}}$$

```python
sy.simplify(K*Nprime(d2(S, K, tau, sigma, r))/(S*sy.exp(r*tau)*Nprime(d1(S, K, tau, sigma, r))))
```

$$\frac{KN^\prime(d_2)}{S_FN^\prime(d_1)}=\frac{K}{S_F}e^{\frac{1}{2}\frac{2\ln{\left(\frac{S_F}{K}\right)}\sigma^2\tau}{\sigma^2\tau}}=1$$

### 4. $$\frac{\partial d_{1,2}}{\partial\sigma}=\frac{-1}{\sigma^2\sqrt{\tau}}\ln\left(\frac{S_F}{K}\right)\pm\frac{1}{2}\sqrt{\tau}$$

```python
sy.diff(d1(S, K, tau, sigma, r), sigma)
```

```python
sy.diff(d2(S, K, tau, sigma, r), sigma)
```

### 5. $$\frac{\partial C}{\partial\sigma}=\frac{1}{\sqrt{2\pi}}Se^{-\frac{1}{2}d_1^2}\sqrt{\tau}$$

```python
sy.diff(C(S, K, tau, sigma, r), sigma)
```

$$d_1=\frac{\ln\left(\frac{S_F}{K}\right)+\frac{\sigma^2}{2}\tau}{\sigma\sqrt{\tau}}$$
$$d_2=\frac{\ln\left(\frac{S_F}{K}\right)-\frac{\sigma^2}{2}\tau}{\sigma\sqrt{\tau}}$$

$$\frac{Ke^{-r\tau}\sqrt{\tau}}{\sqrt{2\pi}}(1+\frac{d_2}{\sigma\sqrt{\tau}})e^{-\frac{1}{2}d_2^2}+\frac{S\sqrt{\tau}}{\sqrt{2\pi}}(1-\frac{d_1}{\sigma\sqrt{\tau}})e^{-\frac{1}{2}d_1^2}$$

$$C(S,K,\tau,\sigma,r)=SN(d_1)-Ke^{-r\tau}N(d_2)$$

$$\frac{\partial C(S,K,\tau,\sigma,r)}{\partial\sigma}=SN^\prime(d_1)\frac{\partial d_1}{\partial\sigma}-Ke^{-r\tau}N^\prime(d_2)\frac{\partial d_2}{\partial\sigma}=SN^\prime(d_1)\left(\frac{\partial d_1}{\partial\sigma}-\frac{\partial d_2}{\partial\sigma}\right)$$

$$\frac{\partial d_{1,2}}{\partial\sigma}=\frac{-1}{\sigma^2\sqrt{\tau}}\ln\left(\frac{S_F}{K}\right)\pm\frac{1}{2}\sqrt{\tau}$$

$$\frac{\partial C(S,K,\tau,\sigma,r)}{\partial\sigma}=SN^\prime(d_1)\sqrt{\tau}=\frac{1}{\sqrt{2\pi}}Se^{-\frac{1}{2}d_1^2}\sqrt{\tau}$$

### 6.  $$\frac{\partial d_{1,2}}{\partial K}=\frac{-1}{K\sigma\sqrt{\tau}}$$

```python
sy.diff(d1(S, K, tau, sigma, r), K)
```

```python
sy.diff(d2(S, K, tau, sigma, r), K)
```

### 7. $$\frac{\partial C}{\partial S}=N(d_1)$$

$$\frac{\partial C(S,K,\tau,\sigma,r)}{\partial S}=N(d_1)+SN^\prime(d_1)\frac{\partial d_1}{\partial S}-Ke^{-r\tau}N^\prime(d_2)\frac{\partial d_2}{\partial S}=N(d_1)$$

### 8. $$\frac{\partial C}{\partial K}=-e^{-r\tau}N(d_2)$$

$$\frac{\partial C(S,K,\tau,\sigma,r)}{\partial K}=SN^\prime(d_1)\frac{\partial d_1}{\partial K}-Ke^{-r\tau}N^\prime(d_2)\frac{\partial d_2}{\partial K}-e^{-r\tau}N(d_2)=-e^{-r\tau}N(d_2)$$

### 9. $$\frac{\partial C}{\partial\sigma^2}=\frac{1}{2\sigma\sqrt{2\pi}}Se^{-\frac{1}{2}d_1^2}\sqrt{\tau}$$

$$\frac{\partial C(S,K,\tau,\sigma,r)}{\partial\sigma^2}=SN^\prime(d_1)\frac{\partial d_1}{\partial\sigma^2}-Ke^{-r\tau}N^\prime(d_2)\frac{\partial d_2}{\partial\sigma^2}=SN^\prime(d_1)\left(\frac{\partial d_1}{\partial\sigma^2}-\frac{\partial d_2}{\partial\sigma^2}\right)$$

$$\frac{\partial d_1}{\partial\sigma^2}-\frac{\partial d_2}{\partial\sigma^2}=\frac{\partial\left(\sigma\sqrt{\tau}\right)}{\partial\sigma^2}=\sqrt{\tau}\frac{\partial\sigma}{\partial\sigma^2}=\frac{\sqrt{\tau}}{2\sigma}$$

$$\frac{\partial C(S,K,\tau,\sigma,r)}{\partial\sigma^2}=\frac{1}{2\sigma\sqrt{2\pi}}Se^{-\frac{1}{2}d_1^2}\sqrt{\tau}$$


```python
%matplotlib inline
```

# 592B, Class 2.2 (09/12). Fourier series, aliasing, Sampling theorem

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.io.wavfile as wavfile

from ipywidgets import interactive
from IPython.display import Audio, display
```

## Sine waves and cosine waves

Let's review the relation between sine and cosine waves. Let's plot a sine wave and a cosine wave:

$$y_{sin} = A \sin(440\pi t)$$
$$y_{cos} = A \cos(440\pi t)$$

```python
fs = 44100 # define the sampling rate, f_s = 144100 Hz

t_start = 0 # We start sampling at t = 0s
t_stop =  3 # We stop sampling at t = 3s

ns = (t_stop - t_start) * fs + 1
print(ns)

x = np.linspace(t_start, t_stop, ns)
print(len(x))

f = 440 # frequency of y_sin

y_sin = np.sin(2*np.pi*f*x) #do we need x1 and x2, or common x okay
y_cos = np.cos(2*np.pi*f*x)

print(x[0:10])

print(1/fs)
```

    132301
    132301
    [0.00000000e+00 2.26757370e-05 4.53514739e-05 6.80272109e-05
     9.07029478e-05 1.13378685e-04 1.36054422e-04 1.58730159e-04
     1.81405896e-04 2.04081633e-04]
    2.2675736961451248e-05

```python
plt.figure("Sine vs. cosine")
plt.title("Sine vs. cosine")
plt.xlim(0,1/f) # What is 1/f?

plt.plot(x,y_sin, 'g', label='$y_{sin}$')
plt.plot(x,y_cos, 'b', label='$y_{cos}$')

plt.legend(loc="upper right")
plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')

```

*** In-class exercise: how could I get the blue line by modifying the parameters of $y_{sin}(t)$?***

```python
y_cos_2 = np.sin(2*np.pi*f*x + np.pi/2)-1

plt.figure("Sine vs. cosine")
plt.title("Sine vs. cosine")
plt.xlim(0,1/f) # What is 1/f?

plt.plot(x,y_sin, 'g', label='$y_{sin}$')
plt.plot(x,y_cos_2, 'b', label='$y_{cos}$')

plt.legend(loc="upper right")
plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')

```

## Leftovers from Tuesday: converting between samples and timepoints

We read in some Hmong audio and plotted it, as in the cell below.

```python
# From https://stackoverflow.com/questions/18644166/how-to-manipulate-wav-file-data-in-python
fs, hmong_data = wavfile.read('rms-sample-sounds/hmong_m6_24_c.wav')

n = 4096 # number of samples to plot

# Set up a new figure
plt.figure("hmong")

# plot the first n samples
plt.plot(hmong_data[0:n])
# label the axes
plt.ylabel("Amplitude (dB?)")
plt.xlabel("Samples [n]")
# set the title
plt.title("Hmong wave file")

print(len(hmong_data))
print(fs)
```

Then you were working on this:

***In-class exercise: Plot the first second of the Hmong audio file, with a real time axis, i.e., time should be in seconds, not in samples. Hint: knowing the sampling rate will help you do this! I put a cell below for you to get started.***

***And if you finish that, can you also write a function that will plot the audio file from some start time `t_start` to some stop time `t_stop`?***

```python
# Recall that fs is 22050 Hz. So that means the first second is the first 22050 samples.

ns = fs # number of samples is 22050
# So we can use np.linspace to define a vector of sampled time points from 0 to 1 seconds.

x = np.linspace(0, 1, ns)

# Set up a new figure
plt.figure("hmong with time on x-axis")

# plot the first n samples
plt.plot(x, hmong_data[0:ns])
# label the axes
plt.ylabel("Amplitude (dB?)")
plt.xlabel("Time (s)")
# set the title
plt.title("Hmong wave file with time on x-axis")

# In general, suppose we want to plot the audio from t_start to t_stop.
# Then we could do: x = np.linspace(t_start, t_stop, ns)
# But what's ns? We did that in the Class 1.2 notebook.
# ns = (t_stop - t_start) / Ts + 1
# Or, if we are ignoring the +/- 1 sample, ns = (t_stop - t_start) * fs
# So, we can do this:

def plot_with_time_axis(t_start, t_stop, fs_data, DEBUG = True, data=hmong_data):
    ns = int((t_stop - t_start) * fs_data) # number of samples from t_start to t_stop
    if DEBUG: print("ns = ", ns)
    n_start = int(round(t_start * fs_data)) # sample number corresponding to t_start
    if DEBUG: print("n_start = ", n_start)
    n_stop = int(round(t_stop * fs_data)) # sample number corresponding to t_stop
    if DEBUG:
        print("n_stop = ", n_stop)
        print("len(data[n_start:n_stop]) = ", len(data[n_start:n_stop]))
    x = np.linspace(t_start, t_stop, ns)
    if DEBUG: print("len(x)= ", len(x))
    plt.figure()
    plt.plot(x, data[n_start:n_stop])
    # label the axes
    plt.ylabel("Amplitude (dB?)")
    plt.xlabel("Time (s)")

plot_with_time_axis(0.5, 0.8, fs, DEBUG = False)
plot_with_time_axis(0.5, 0.55, fs, DEBUG = False)  # oops, an error!  Why?

# how could we make this code more robust? -> HW

```

## Adding up sine waves: towards Fourier series

Let's take the two sinusoidal signals we worked on last week and in your homework and try adding them up.

```python
fs = 44100 # define the sampling rate, f_s = 44.1 kHz

t_start = 0 # We start sampling at t = 0s
t_stop =  1 # We stop sampling at t = 1s

ns = (t_stop - t_start) * fs + 1

x = np.linspace(t_start, t_stop, ns)

f1 = 440 # frequency of y_1(t)
f2 = 220 # frequency of y_2(t)

y1 = np.sin(2*np.pi*f1*x)
y2 = np.sin(2*np.pi*f2*x)

y1_plus_y2 = y1+y2

```

```python
plt.figure("Adding up sines") # Create a new figure

plt.xlim(0,0.01)
plt.plot(x , y1, "-g", label="y1") # plot (x,y1) as a green line
plt.plot(x , y2, "-b", label="y2") # plot (x,y2) as a blue line
plt.plot(x , y1_plus_y2, "-r", label="y1+y2") # plot (x,y2) as a blue line

#plt.stem(x,y1, 'r', )
plt.legend(loc="upper right")
plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')
plt.title("Adding up sines")

```

You should play with setting other parameters! Remember, you can listen to your generated sinusoids too, using `Audio`.

Below is some code that creates an interactive "widget" for both plotting and playing two sine waves and their superposition (sum).

You need to import these libraries, as we did at the beginning of the notebook:
```python
from ipywidgets import interactive
from IPython.display import Audio, display
```
The code is inspired by this [beat frequencies demo](https://ipywidgets.readthedocs.io/en/stable/examples/Beat%20Frequencies.html).

```python
def plot_play_summed_sines(f1 = 440, f2 = 880, t_start = 0, t_stop = 2, fs = 44100, xlim_max = 0.08):
    x = np.linspace(t_start, t_stop, fs * (t_stop - t_start))
    y1 = np.sin(2*np.pi*f1*x)
    y2 = np.sin(2*np.pi*f2*x)

    plt.xlim(t_start,xlim_max)
    plt.plot(x , y1, "-g", label="y1")
    plt.plot(x , y2, "-b", label="y2")
    plt.plot(x , y1 + y2, "-r", label="y1+y2")
    plt.legend(loc="upper right")
    plt.xlabel('Time (s)')
    plt.ylabel('Amplitude (dB)')
    plt.title("Adding up sines")

    display(Audio(data=y1, rate=fs))
    display(Audio(data=y2, rate=fs))
    display(Audio(data=y1+y2, rate=fs))
```

```python
v = interactive(plot_play_summed_sines, f1=(100,900), f2=(100,900), t_start = (0,0), t_stop = (0,5))
display(v)
```

    interactive(children=(IntSlider(value=440, description='f1', max=900, min=100), IntSlider(value=880, descripti…

## A first look at Fourier series

Recall from last week that a standard definition of a sinusoidal signal (i.e., a sine wave) is given as function of time $t$:

$$y(t) = A \sin (\omega t + \phi) $$

where:
- $A$ is the amplitude
- $\omega$ is the angular frequency, n.b., $\omega = 2\pi f$ (where $f$ is the frequency in cycles per second (Hertz or Hz))
- $\phi$ is the phase shift

One standard definition of a Fourier series is:

\begin{equation}
      f(t) = a_0 + \displaystyle\sum\limits_{n=1}^N \left(a_n\cos(2\pi nt) + b_n\sin(2\pi nt)\right)
\end{equation}

where

$n \in \mathbb{Z}$, and $a_0 \ldots a_N$ and $b_0 \ldots b_N$ are called **Fourier coefficients**. Note that these are amplitude values for each component sinusoidal function in the sum: they provide weights for the individual components in the sum. We'll see later that these determine the amplitude of peaks in a spectrum of $f(t)$.

***Discussion: Why is there no $b_0$ in the formula?***

Let's see what each of the individual $a_n\cos(2\pi nt) + b_n\sin(2\pi nt)$ terms look like, for $a_n = b_n = 1$ for all $n$, for $n <6$, otherwise 0. And let's include the $a_0$ term by starting at $N=0$.

```python
# Inspired by https://matplotlib.org/examples/pylab_examples/subplots_demo.html Three subplots sharing both x/y axes

t_start = 0; t_stop = 1; fs = 1000
x = np.linspace(t_start, t_stop, fs * (t_stop - t_start))
y = [np.sin(2*np.pi*n*x) + np.cos(2*np.pi*n*x) for n in np.arange(0,6)]

f_fourier_series, (ax1, ax2, ax3, ax4, ax5, ax6) = plt.subplots(6, sharex=True, sharey=True)
ax1.plot(x , y[0], "-", label="n = 0")
ax1.set_title('Fourier series up to N = 5, a_n = b_n = 1')
ax2.plot(x , y[1], "-", label="n = 1")
ax3.plot(x , y[2], "-", label="n = 2")
ax4.plot(x , y[3], "-", label="n = 3")
ax5.plot(x , y[4], "-", label="n = 4")
ax6.plot(x , y[5], "-", label="n = 5")

plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')

# Fine-tune figure; make subplots close to each other and hide x ticks for
# all but bottom plot.
f_fourier_series.subplots_adjust(hspace=0)
plt.setp([a.get_xticklabels() for a in f_fourier_series.axes[:-1]], visible=False)

```

These are the first 6 terms in a **set of basis functions**, in particular, the **Fourier basis functions**. Each term, for some given $N$, is a basis function. In the limit, as $N \rightarrow \infty$ we can approximate any function with this set of basis functions. And each of the basis functions are **orthogonal** to one another. For intuition, we'll review vector spaces and orthogonal vectors.

Below, we "multiply" one basis function by another to illustrate orthogonality: the integral of the product of any two of the basis functions is 0.

```python
t_start = 0; t_stop = 3; fs = 1000
x = np.linspace(t_start, t_stop, fs * (t_stop - t_start))
y = [np.sin(2*np.pi*f*x) + np.cos(2*np.pi*f*x) for f in np.arange(0,6)]

f_ortho, (ax1, ax2, ax3, ax4, ax5, ax6) = plt.subplots(6, sharex=True, sharey=True)
ax1.plot(x , y[1]*y[2], "-", label="n = 0")
ax1.plot((t_start, t_stop), (0, 0), 'r-')
ax1.set_title('Fourier series up to N = 5, a_n = b_n = 1')
ax2.plot(x , y[1]*y[3], "-", label="n = 1")
ax2.plot((t_start, t_stop), (0, 0), 'r-')
ax3.plot(x , y[2]*y[3], "-", label="n = 2")
ax3.plot((t_start, t_stop), (0, 0), 'r-')
ax4.plot(x , y[3]*y[4], "-", label="n = 3")
ax4.plot((t_start, t_stop), (0, 0), 'r-')
ax5.plot(x , y[4]*y[5], "-", label="n = 4")
ax5.plot((t_start, t_stop), (0, 0), 'r-')
ax6.plot(x , y[5]*y[1], "-", label="n = 5")
ax6.plot((t_start, t_stop), (0, 0), 'r-')

plt.xlabel('Time (s)')
plt.ylabel('Amplitude (dB)')

# Fine-tune figure; make subplots close to each other and hide x ticks for
# all but bottom plot.
f_ortho.subplots_adjust(hspace=0)
plt.setp([a.get_xticklabels() for a in f_ortho.axes[:-1]], visible=False)

```

In your homework, you'll explore building up a complex wave as a Fourier series.