# Path frame

> Renato Naville Watanabe, Marcos Duarte
> [Laboratory of Biomechanics and Motor Control](http://pesquisa.ufabc.edu.br/bmclab)
> Federal University of ABC, Brazil

Contents

1  Python setup

2  Cartesian coordinate system

3  Determination of a coordinate system

4  Time varying basis

4.1  Tangential versor

4.2  Normal versor

4.3  Binormal versor

5  Velocity and Acceleration in a time-varying frame

5.1  Velocity

5.2  Acceleration

6  Example

6.1  Solving numerically

6.2  Symbolic solution (extra reading)

7  Further Reading

8  Video lectures on the Internet

9  Problems

10  References

## Python setup

```python
import numpy as np
import sympy as sym
from sympy.vector import CoordSys3D
import matplotlib.pyplot as plt
sym.init_printing()
from sympy.plotting import plot_parametric
from sympy.physics.mechanics import ReferenceFrame, Vector, dot
from matplotlib.patches import FancyArrowPatch
plt.rcParams.update({'figure.figsize':(8, 5), 'lines.linewidth':2})
```

## Cartesian coordinate system

As we perceive the surrounding space as three-dimensional, a convenient coordinate system is the [Cartesian coordinate system](http://en.wikipedia.org/wiki/Cartesian_coordinate_system) in the [Euclidean space](http://en.wikipedia.org/wiki/Euclidean_space) with three orthogonal axes as shown below. The axes directions are commonly defined by the [right-hand rule](http://en.wikipedia.org/wiki/Right-hand_rule) and attributed the letters X, Y, Z. The orthogonality of the Cartesian coordinate system is convenient for its use in classical mechanics, most of the times the structure of space is assumed having the [Euclidean geometry](http://en.wikipedia.org/wiki/Euclidean_geometry) and as consequence, the motion in different directions are independent of each other.

Figure. Representation of a point and its position vector in a Cartesian coordinate system.

## Determination of a coordinate system

In Biomechanics, we may use different coordinate systems for convenience and refer to them as global, laboratory, local, anatomical, or technical reference frames or coordinate systems.

From [linear algebra](http://en.wikipedia.org/wiki/Linear_algebra), a set of unit linearly independent vectors (orthogonal in the Euclidean space and each with norm (length) equals to one) that can represent any vector via [linear combination](http://en.wikipedia.org/wiki/Linear_combination) is called a basis (or **orthonormal basis**). The figure below shows a point and its position vector in the Cartesian coordinate system and the corresponding versors (**unit vectors**) of the basis for this coordinate system. See the notebook [Scalar and vector](http://nbviewer.ipython.org/github/demotu/BMC/blob/master/notebooks/ScalarVector.ipynb) for a description on vectors.

Figure. Representation of a point $\mathbf{P}$ and its position vector $\overrightarrow{\mathbf{r}}$ in a Cartesian coordinate system. The versors $\hat{\mathbf{i}},\, \hat{\mathbf{j}},\, \hat{\mathbf{k}}\,$  form a basis for this coordinate system and are usually represented in the color sequence RGB (red, green, blue) for easier visualization.

One can see that the versors of the basis shown in the figure above have the following coordinates in the Cartesian coordinate system:

\begin{equation}
\hat{\mathbf{i}} = \begin{bmatrix}1\\0\\0 \end{bmatrix}, \quad \hat{\mathbf{j}} = \begin{bmatrix}0\\1\\0 \end{bmatrix}, \quad \hat{\mathbf{k}} = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}
\label{eq_1}
\end{equation}

Using the notation described in the figure above, the position vector $\overrightarrow{\mathbf{r}}$  can be expressed as:

\begin{equation}
\overrightarrow{\mathbf{r}} = x\hat{\mathbf{i}} + y\hat{\mathbf{j}} + z\hat{\mathbf{k}}
\label{eq_2}
\end{equation}

However, to use a fixed basis can lead to very complex expressions.

## Path basis

Consider that we have the position vector of a particle, moving in the path described by the parametric curve $s(t)$, described in a fixed reference frame as:

\begin{equation}
{\bf\hat{r}}(t) = {x}{\bf\hat{i}}+{y}{\bf\hat{j}} + {z}{\bf\hat{k}}
\label{eq_3}
\end{equation}

Figure. Position vector of a moving particle in relation to a coordinate system.

### Tangential versor

Often we describe all the kinematic variables in this fixed reference frame. However, it is often useful to define a path basis, attached to some point of interest. In this case, what is usually done is to choose as one of the basis vector a unitary vector in the direction of the velocity of the particle. Defining this vector as:

\begin{equation}
{\bf\hat{e}_t} = \frac{{\bf\vec{v}}}{\Vert{\bf\vec{v}}\Vert}
\label{eq_4}
\end{equation}

### Normal versor

For the second vector of the basis, we define first a vector of curvature of the path (the meaning of this curvature vector will be seeing in another notebook):

\begin{equation}
{\bf\vec{C}} = \frac{d{\bf\hat{e}_t}}{ds}
\label{eq_5}
\end{equation}

Note that $\bf\hat{e}_t$ is a function of the path $s(t)$. So, by the chain rule:

\begin{equation}
\frac{d{\bf\hat{e}_t}}{dt} = \frac{d{\bf\hat{e}_t}}{ds}\frac{ds}{dt} \longrightarrow \frac{d{\bf\hat{e}_t}}{ds} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\frac{ds}{dt}} \longrightarrow {\bf\vec{C}} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\frac{ds}{dt}}\longrightarrow {\bf\vec{C}} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\Vert{\bf\vec{v}}\Vert}
\label{eq_6}
\end{equation}

Now we can define the second vector of the basis, ${\bf\hat{e}_n}$:

\begin{equation}
{\bf\hat{e}_n} = \frac{{\bf\vec{C}}}{\Vert{\bf\vec{C}}\Vert}
\label{eq_7}
\end{equation}

Figure. A moving particle and a corresponding path basis.

### Binormal versor

The third vector of the basis is obtained by the cross product between ${\bf\hat{e}_n}$ and ${\bf\hat{e}_t}$:

\begin{equation}
{\bf\hat{e}_b} = {\bf\hat{e}_t} \times {\bf\hat{e}_n}
\label{eq_8}
\end{equation}

Note that the vectors  ${\bf\hat{e}_t}$ ,  ${\bf\hat{e}_n}$  and  ${\bf\hat{e}_b}$  vary together with the particle movement. This basis is also called as **Frenet-Serret frame**.

## Velocity and Acceleration in a path frame

### Velocity

Given the expression of $r(t)$ in a fixed frame we can write the velocity  ${\bf\vec{v}(t)}$ as a function of the fixed frame of reference  ${\bf\hat{i}}$,  ${\bf\hat{j}}$ and  ${\bf\hat{k}}$ (see http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/KinematicsParticle.ipynb)).

\begin{equation}
{\bf\vec{v}}(t) = \dot{x}{\bf\hat{i}}+\dot{y}{\bf\hat{j}}+\dot{z}{\bf\hat{k}}
\label{eq_9}
\end{equation}

However, this can lead to very complex functions. So it is useful to use the basis find previously ${\bf\hat{e}_t}$,  ${\bf\hat{e}_n}$ and ${\bf\hat{e}_b}$.

The velocity  ${\bf\vec{v}}$  of the particle is, by the definition of ${\bf\hat{e}_t}$, in the direction of ${\bf\hat{e}_t}$:

\begin{equation}
{\bf\vec{v}}={\Vert\bf\vec{v}\Vert}.{\bf\hat{e}_t}
\label{eq_10}
\end{equation}

### Acceleration

The acceleration can be written in the fixed frame of reference as:

\begin{equation}
{\bf\vec{a}}(t) = \ddot{x}{\bf\hat{i}}+\ddot{y}{\bf\hat{j}}+\ddot{z}{\bf\hat{k}}
\label{eq_11}
\end{equation}

But for the same reasons of the velocity vector, it is useful to describe the acceleration vector in the path basis. We know that the acceleration is the time derivative of the velocity:

\begin{align}
{\bf\vec{a}} =& \frac{{d\bf\vec{v}}}{dt}=\\
    =&\frac{{d({\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}})}{dt}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}\dot{{\bf\hat{e}_t}}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}\frac{d{\bf\hat{e}_t}}{ds}\frac{ds}{dt}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}^2\frac{d{\bf\hat{e}_t}}{ds}=\\
    =&\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}^2\Vert{\bf\vec{C}} \Vert {\bf\hat{e}_n}
\label{eq_12}
\end{align}

## Example
For example, consider that a particle follows the path described by the parametric curve below:

\begin{equation}
\vec{r}(t) = (10t+100){\bf{\hat{i}}} + \left(-\frac{9,81}{2}t^2+50t+100\right){\bf{\hat{j}}}
\label{eq_13}
\end{equation}

This curve could be, for example, from a projectile motion. See http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/ProjectileMotion.ipynb for an explanation on projectile motion.

### Solving numerically

Now we will obtain the path basis numerically. This method is useful when it is not available a mathematical expression of the path. This often happens when you have available data collected experimentally (most of the cases in Biomechanics).

First, data will be obtained from the expression of $r(t)$. This is done to replicate the example above. You could use data collected experimentally, for example.

```python
t = np.linspace(0, 10, 1000).reshape(-1,1)
r = np.hstack((10*t + 100, -9.81/2*t**2 + 50*t + 100))
```

Now, to obtain the  $\bf{\hat{e_t}}$  versor, we can use Equation (4).

```python
dt = t[1]
v = np.diff(r,axis=0)/dt
vNorm  = np.linalg.norm(v, axis=1, keepdims=True)

et = v/vNorm
```

And to obtain the versor $\bf{\hat{e_n}}$, we can use Equation (8).

```python
C = np.diff(et,axis=0)/dt
C = C/vNorm[1:]

CNorm = np.linalg.norm(C, axis=1, keepdims=True)
en = C/CNorm
```

```python
fig = plt.figure()
plt.plot(r[:, 0], r[:, 1], '.')
ax = fig.add_axes([0, 0, 1, 1])
time = np.linspace(0, 10, 10)
for i in np.arange(0, len(t)-2, 50):
    vec1 = FancyArrowPatch(r[i, :], r[i, :]+10*et[i, :], mutation_scale=20, color='r')
    vec2 = FancyArrowPatch(r[i, :], r[i, :]+10*en[i, :], mutation_scale=20, color='g')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((80, 250))
plt.ylim((80, 250))
plt.legend([vec1, vec2], [r'$\vec{e_t}$', r'$\vec{e_{n}}$'])
plt.show()
```

```python
v  = vNorm*et
vNormDot  = np.diff(vNorm, axis=0)/dt

a = vNormDot*et[1:,:] + CNorm*en*vNorm[1:]**2
```

```python
fig = plt.figure()
plt.plot(r[:, 0], r[:, 1], '.')
ax = fig.add_axes([0, 0, 1, 1])
for i in range(0, len(t)-2, 50):
    vec1 = FancyArrowPatch(r[i, :], r[i, :]+v[i, :], mutation_scale=10, color='r')
    vec2 = FancyArrowPatch(r[i, :], r[i, :]+a[i, :], mutation_scale=10, color='g')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((80, 250))
plt.ylim((80, 250))
plt.legend([vec1, vec2], [r'$\vec{v}$', r'$\vec{a}$'])
plt.show()
```

### Symbolic solution (extra reading)

The computation here will be performed symbolically, with the symbolic math package of Python, Sympy. Below,a reference frame, called O,  and a varible for time (t) are defined.

```python
O = sym.vector.CoordSys3D(' ')
t = sym.symbols('t')
```

Below the vector $r(t)$ is defined symbolically.

```python
r = (10*t+100)*O.i + (-9.81/2*t**2+50*t+100)*O.j+0*O.k
r
```

```python
p1 = plot_parametric(r.dot(O.i), r.dot(O.j), (t, 0, 10), line_width=4, aspect_ratio=(2, 1))
```

```python
v = sym.diff(r)
v
```

```python
et = v/sym.sqrt(v.dot(v))
et
```

```python
C = sym.diff(et)/sym.sqrt(v.dot(v))
C
```

```python
en = C/(sym.sqrt(C.dot(C)))
sym.simplify(en)
```

```python
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
time = np.linspace(0, 10, 30)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+10*float(et.dot(O.i).subs(t,instant)),
                          float(r.dot(O.j).subs(t, instant))+10*float(et.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${\hat{e_t}}$', lw=2)
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+10*float(en.dot(O.i).subs(t, instant)),
                          float(r.dot(O.j).subs(t, instant))+10*float(en.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${\hat{e_n}}$', lw=2)
    ax.add_artist(vn)
    ax.add_artist(vt)
plt.xlim((90, 250))
plt.ylim((90, 250))
plt.xlabel('x')
plt.legend(handles=[vt, vn], fontsize=20)
plt.show()
```

Now we can find the vectors ${\bf\vec{v}}$ and ${\bf\vec{a}}$ described in the path frame.

```python
v = sym.sqrt(v.dot(v))*et
```

```python
a = sym.diff(sym.sqrt(v.dot(v)))*et+v.dot(v)*sym.sqrt(C.dot(C))*en
sym.simplify(sym.simplify(a))
```

```python
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
time = np.linspace(0, 10, 10)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+float(v.dot(O.i).subs(t,instant)),
                          float(r.dot(O.j).subs(t, instant))+float(v.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${{v}}$', lw=2)
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+float(a.dot(O.i).subs(t, instant)),
                          float(r.dot(O.j).subs(t, instant))+float(a.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${{a}}$', lw=2)
    ax.add_artist(vn)
    ax.add_artist(vt)
plt.xlim((60, 250))
plt.ylim((60, 250))
plt.legend(handles=[vt, vn], fontsize=20)
plt.show()
```

## Further Reading

 - Read pages 932-971 of the 18th chapter of the [Ruina and Rudra's book](http://ruina.tam.cornell.edu/Book/index.html) about path basis vectors.

## Video lectures on the Internet

 - [Path Vectors](https://eaulas.usp.br/portal/video?idItem=7800)

## Problems

1. Obtain the vectors  $\hat{e_n}$  and  $\hat{e_t}$  for the problem 18.1.1 from Ruina and Rudra's book.
2. Solve the problem 18.1.9 from Ruina and Rudra's book.
3. Write a Python program to solve the problem 18.1.10 (only the part of  $\hat{e_n}$  and  $\hat{e_t}$).

## References

+ Ruina A, Rudra P (2019) [Introduction to Statics and Dynamics](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press.

```python
np.arctan2?
```

    Call signature:  np.arctan2(*args, **kwargs)
    Type:            ufunc
    String form:     <ufunc 'arctan2'>
    File:            ~/miniconda377/lib/python3.7/site-packages/numpy/__init__.py
    Docstring:
    arctan2(x1, x2, /, out=None, *, where=True, casting='same_kind', order='K', dtype=None, subok=True[, signature, extobj])

    Element-wise arc tangent of ``x1/x2`` choosing the quadrant correctly.

    The quadrant (i.e., branch) is chosen so that ``arctan2(x1, x2)`` is
    the signed angle in radians between the ray ending at the origin and
    passing through the point (1,0), and the ray ending at the origin and
    passing through the point (`x2`, `x1`).  (Note the role reversal: the
    "`y`-coordinate" is the first function parameter, the "`x`-coordinate"
    is the second.)  By IEEE convention, this function is defined for
    `x2` = +/-0 and for either or both of `x1` and `x2` = +/-inf (see
    Notes for specific values).

    This function is not defined for complex-valued arguments; for the
    so-called argument of complex values, use `angle`.

    Parameters
    ----------
    x1 : array_like, real-valued
        `y`-coordinates.
    x2 : array_like, real-valued
        `x`-coordinates.
        If ``x1.shape != x2.shape``, they must be broadcastable to a common
        shape (which becomes the shape of the output).
    out : ndarray, None, or tuple of ndarray and None, optional
        A location into which the result is stored. If provided, it must have
        a shape that the inputs broadcast to. If not provided or None,
        a freshly-allocated array is returned. A tuple (possible only as a
        keyword argument) must have length equal to the number of outputs.
    where : array_like, optional
        This condition is broadcast over the input. At locations where the
        condition is True, the `out` array will be set to the ufunc result.
        Elsewhere, the `out` array will retain its original value.
        Note that if an uninitialized `out` array is created via the default
        ``out=None``, locations within it where the condition is False will
        remain uninitialized.
    **kwargs
        For other keyword-only arguments, see the
        :ref:`ufunc docs <ufuncs.kwargs>`.

    Returns
    -------
    angle : ndarray
        Array of angles in radians, in the range ``[-pi, pi]``.
        This is a scalar if both `x1` and `x2` are scalars.

    See Also
    --------
    arctan, tan, angle

    Notes
    -----
    *arctan2* is identical to the `atan2` function of the underlying
    C library.  The following special values are defined in the C
    standard: [1]_

    ====== ====== ================
    `x1`   `x2`   `arctan2(x1,x2)`
    ====== ====== ================
    +/- 0  +0     +/- 0
    +/- 0  -0     +/- pi
     > 0   +/-inf +0 / +pi
     < 0   +/-inf -0 / -pi
    +/-inf +inf   +/- (pi/4)
    +/-inf -inf   +/- (3*pi/4)
    ====== ====== ================

    Note that +0 and -0 are distinct floating point numbers, as are +inf
    and -inf.

    References
    ----------
    .. [1] ISO/IEC standard 9899:1999, "Programming language C."

    Examples
    --------
    Consider four points in different quadrants:

    >>> x = np.array([-1, +1, +1, -1])
    >>> y = np.array([-1, -1, +1, +1])
    >>> np.arctan2(y, x) * 180 / np.pi
    array([-135.,  -45.,   45.,  135.])

    Note the order of the parameters. `arctan2` is defined also when `x2` = 0
    and at several other special points, obtaining values in
    the range ``[-pi, pi]``:

    >>> np.arctan2([1., -1.], [0., 0.])
    array([ 1.57079633, -1.57079633])
    >>> np.arctan2([0., 0., np.inf], [+0., -0., np.inf])
    array([ 0.        ,  3.14159265,  0.78539816])
    Class docstring:
    Functions that operate element by element on whole arrays.

    To see the documentation for a specific ufunc, use `info`.  For
    example, ``np.info(np.sin)``.  Because ufuncs are written in C
    (for speed) and linked into Python with NumPy's ufunc facility,
    Python's help() function finds this page whenever help() is called
    on a ufunc.

    A detailed explanation of ufuncs can be found in the docs for :ref:`ufuncs`.

    **Calling ufuncs:** ``op(*x[, out], where=True, **kwargs)``

    Apply `op` to the arguments `*x` elementwise, broadcasting the arguments.

    The broadcasting rules are:

    * Dimensions of length 1 may be prepended to either array.
    * Arrays may be repeated along dimensions of length 1.

    Parameters
    ----------
    *x : array_like
        Input arrays.
    out : ndarray, None, or tuple of ndarray and None, optional
        Alternate array object(s) in which to put the result; if provided, it
        must have a shape that the inputs broadcast to. A tuple of arrays
        (possible only as a keyword argument) must have length equal to the
        number of outputs; use None for uninitialized outputs to be
        allocated by the ufunc.
    where : array_like, optional
        This condition is broadcast over the input. At locations where the
        condition is True, the `out` array will be set to the ufunc result.
        Elsewhere, the `out` array will retain its original value.
        Note that if an uninitialized `out` array is created via the default
        ``out=None``, locations within it where the condition is False will
        remain uninitialized.
    **kwargs
        For other keyword-only arguments, see the :ref:`ufunc docs <ufuncs.kwargs>`.

    Returns
    -------
    r : ndarray or tuple of ndarray
        `r` will have the shape that the arrays in `x` broadcast to; if `out` is
        provided, it will be returned. If not, `r` will be allocated and
        may contain uninitialized values. If the function has more than one
        output, then the result will be a tuple of arrays.

```python

```


 Symbolic Computation: The Pitfalls

This collection of notebooks is mostly numerical, with not a lot of exact or symbolic computation.  Why not?  And, for that matter, why is numerical computing (even with all the unexpected behaviour of floating-point arithmetic) so much more popular than symbolic or exact computing?

This section explores symbolic computation and its pitfalls.  We do so from the point of view of experience and with some authority: we have used symbolic computation (usually in Maple, but also in other symbolic languages) for years (decades!) and know it and its benefits well. Caveat: We do not know SymPy so well, and so if we say that SymPy can't do something, we may well be wrong.

One of RMC's earliest co-authors, Honglin Ye, put it well when he suggested that not everyone needs numerical methods but that everyone could use symbolic computation.

Wait.  Isn't that contradictory?  If everyone could use it, why aren't they?

There are, we think, a few main obstacles.

 Symbolic computation systems are hard to learn how to use well, because there's a lot to learn (indeed, you kind of have to know the math first, too).  Look at the SymPy Tutorial for example. It has ten sections, one labeled "Gotchas". The SAGEMATH system, which also works with Python, is both more powerful and more complicated: See the SAGEMATH Tutorial  to get started there.

 Some mathematical problems are inherently too expensive to solve in human lifetimes, even with today's computers, and people unfairly blame symbolic computation systems for this.

 Even if you can solve a problem exactly, with extra effort, that effort might be wasted because the approximate answers are also the "exact" answers to similar problems, and those similar problems might be just as good a model of
        whatever system you were trying to understand.  This is especially true if the data is only known approximately.

 "Symbolic Computation" and "Computer Algebra" are related terms---about as close as "Numerical Analysis" and "Computational Science" if that comparison means anything---but the differences are remarkably important, because what gets implemented is usually a Computer Algebra system, whereas what people actually want to use is a symbolic computation system.  We'll show you what that means.

 Symbolic computation systems are hard to implement well.  The major systems (Maple, Mathematica, and Matlab) charge money for their products, and get what they ask for; this is because their systems are better than the free ones in many respects, because they have invested significant programmer time to address the inherent difficulties.  Free systems, such as SymPy, will do the easy things for you; and we will see that they can be useful.  But in reality there's no comparison (although we admit that the SAGEMATH people may well disagree with our opinion).

All that said, symbolic computation can be extremely useful (and interesting), and is sometimes worth all the bother. Let's look first at what Python and SymPy can do.  Later we'll look at what the difficulties are.

```python

n = 100
p = 1
for i in range(n):
    p = p*(i+1)
print( n, ' factorial is ', p)
print( 'The floating point value of p is ', 1.0*p )
```

    100  factorial is  93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000
    The floating point value of p is  9.332621544394415e+157

The first thing we see is that Python has, built-in, arbitrary precision integer arithmetic.  Yay?

```python
n = 720
p = 1
for i in range(n):
    p = p*(i+1)
print( n, ' factorial is ', p)
# print( 'The floating point value of p is ', 1.0*p )  # Causes OverflowError
```

    720  factorial is  2601218943565795100204903227081043611191521875016945785727541837850835631156947382240678577958130457082619920575892247259536641565162052015873791984587740832529105244690388811884123764341191951045505346658616243271940197113909845536727278537099345629855586719369774070003700430783758997420676784016967207846280629229032107161669867260548988445514257193985499448939594496064045132362140265986193073249369770477606067680670176491669403034819961881455625195592566918830825514942947596537274845624628824234526597789737740896466553992435928786212515967483220976029505696699927284670563747137533019248313587076125412683415860129447566011455420749589952563543068288634631084965650682771552996256790845235702552186222358130016700834523443236821935793184701956510729781804354173890560727428048583995919729021726612291298420516067579036232337699453964191475175567557695392233803056825308599977441675784352815913461340394604901269542028838347101363733824484506660093348484440711931292537694657354337375724772230181534032647177531984537341478674327048457983786618703257405938924215709695994630557521063203263493209220738320923356309923267504401701760572026010829288042335606643089888710297380797578013056049576342838683057190662205291174822510536697756603029574043387983471518552602805333866357139101046336419769097397432285994219837046979109956303389604675889865795711176566670039156748153115943980043625399399731203066490601325311304719028898491856203766669164468791125249193754425845895000311561682974304641142538074897281723375955380661719801404677935614793635266265683339509760000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Large integers cost more to manipulate---the above number is pretty long.  But SymPy will do it if you ask.  One thing you might want to do is factor those numbers.  Or one might just want to know the prime factors.

```python
from sympy import primefactors

primefactors_n = primefactors(n)
print("The prime factors of {} : {}".format(n, primefactors_n))

primefactors_p = primefactors(p)
print("The prime factors of {} : {}".format(p, primefactors_p))

```

    The prime factors of 720 : [2, 3, 5]
    The prime factors of 2601218943565795100204903227081043611191521875016945785727541837850835631156947382240678577958130457082619920575892247259536641565162052015873791984587740832529105244690388811884123764341191951045505346658616243271940197113909845536727278537099345629855586719369774070003700430783758997420676784016967207846280629229032107161669867260548988445514257193985499448939594496064045132362140265986193073249369770477606067680670176491669403034819961881455625195592566918830825514942947596537274845624628824234526597789737740896466553992435928786212515967483220976029505696699927284670563747137533019248313587076125412683415860129447566011455420749589952563543068288634631084965650682771552996256790845235702552186222358130016700834523443236821935793184701956510729781804354173890560727428048583995919729021726612291298420516067579036232337699453964191475175567557695392233803056825308599977441675784352815913461340394604901269542028838347101363733824484506660093348484440711931292537694657354337375724772230181534032647177531984537341478674327048457983786618703257405938924215709695994630557521063203263493209220738320923356309923267504401701760572026010829288042335606643089888710297380797578013056049576342838683057190662205291174822510536697756603029574043387983471518552602805333866357139101046336419769097397432285994219837046979109956303389604675889865795711176566670039156748153115943980043625399399731203066490601325311304719028898491856203766669164468791125249193754425845895000311561682974304641142538074897281723375955380661719801404677935614793635266265683339509760000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 : [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677, 683, 691, 701, 709, 719]

Factoring seems like such a simple problem, and it's so natural to have it implemented in a symbolic computation system.  The number 720! is 1747 digits long.  Maybe all 1700--odd digits long integers are so easy to factor?

Um, no.  See the discussion at https://en.wikipedia.org/wiki/Integer_factorization to get started.  Let's take a modest problem and time it here.

```python
funny = 3000000000238000000004719
#notfunny = 45000000000000000057990000000000000024761900000000000003506217
from sympy import factorint
import time
start_time = time.time()
factordict = factorint(funny)
print("The prime factors of {} : {}".format(funny, factordict))
print("--- %s seconds ---" % (time.time() - start_time))

```

    The prime factors of 3000000000238000000004719 : {3000000000121: 1, 1000000000039: 1}
    --- 11.154199123382568 seconds ---

That factoring of $3000000000238000000004719$ took between 8 and 11 seconds on this machine (different times if executed more than once); on this very same machine, Maple's "ifactor" command succeeded so quickly that it registered no time taken at all, possibly because it was using a very specialized method; factoring integers is an important feature of symbolic computation and Maple's procedures for it have been a subject of serious research for a long time.  Maple's help pages cite three important papers, and tell you that it uses an algorithm called the quadratic sieve.  Maple can factor $45000000000000000057990000000000000024761900000000000003506217$ into its three prime factors in about 7.5 seconds on this machine; in contrast, after fifty minutes running trying to factor that with factorint as above, RMC had to hard-restart to get Python's attention.

That SymPy takes so long to factor integers, in comparsion, suggests that it isn't using the best methods (the documentation says that it switches between three methods, trial division, Pollard rho, and Pollard p-1) ; and because factoring is such a basic algorithm (an even more basic one is GCD or Greatest Common Divisor) this will have important knock-on effects.

But factoring, as old an idea as it is, is complicated enough to be used as a basic idea in modern cryptography.  The slowness of SymPy is not completely its fault: the problem is hard.

Let's move on to computing with functions.  As previously stated, most supposedly "symbolic" systems are really "algebra" systems: this means that they work well with polynomials (even multivariate polynomials).  A polynomial considered as an algebraic object is isomorphic to a polynomial considered as a function---but the difference in viewpoint can alter the affordances.  An "affordance" is a word meaning "something can happen with it": for instance, you can pick out a lowest-degree term; or you can add it to another polynomial; or you can square it; and so on.  As a function, you can evaluate it at a particular value for the symbols (variables).

```python
from sympy import *
x = symbols('x')

solveset(Eq(x**2, 3), x)

```

    {-sqrt(3), sqrt(3)}

```python
solveset(Eq(x**3+x-1, 0), x)
```

    {-1/(3*(1/2 + sqrt(93)/18)**(1/3)) + (1/2 + sqrt(93)/18)**(1/3), -(1/2 + sqrt(93)/18)**(1/3)/2 + 1/(6*(1/2 + sqrt(93)/18)**(1/3)) + I*(sqrt(3)/(6*(1/2 + sqrt(93)/18)**(1/3)) + sqrt(3)*(1/2 + sqrt(93)/18)**(1/3)/2), -(1/2 + sqrt(93)/18)**(1/3)/2 + 1/(6*(1/2 + sqrt(93)/18)**(1/3)) + I*(-sqrt(3)*(1/2 + sqrt(93)/18)**(1/3)/2 - sqrt(3)/(6*(1/2 + sqrt(93)/18)**(1/3)))}

```python
start_time = time.time()
#solveset(Eq(x**4+x-1, 0), x) # Interrupted after about two hours: the code did not succeed
print("--- %s seconds ---" % (time.time() - start_time))
```

    --- 0.0 seconds ---

In those two hours, RMC went and had his dinner; then downloaded a paper by Dave Auckly from the American Mathematical Monthly 2007  which talks about solving the quartic with a pencil (an algebraic geometer's pencil!), read it, and solved the problem by hand, including solving the resolvent cubic by hand, which he already knew how to do.  And got it right, too. So there.

In contrast, Maple (nearly instantaneously) returns---if you force it to by saying you want the explicit solution---the answer
$$
\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}+\frac{\mathrm{I} \sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}-48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
,
\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}-\frac{\mathrm{I} \sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}-48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
,
-\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}+\frac{\sqrt{6}\, \sqrt{\frac{-\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}+48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
,
-\frac{\sqrt{6}\, \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{12}-\frac{\sqrt{6}\, \sqrt{\frac{-\left(108+12 \sqrt{849}\right)^{\frac{2}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}+12 \sqrt{6}\, \left(108+12 \sqrt{849}\right)^{\frac{1}{3}}+48 \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}} \sqrt{\frac{\left(108+12 \sqrt{849}\right)^{\frac{2}{3}}-48}{\left(108+12 \sqrt{849}\right)^{\frac{1}{3}}}}}}}{12}
$$

This is an example of what Velvel Kahan calls a "wallpaper expression."  He also famously said, "Have you ever asked a computer algebra system a question, and then, as the screensful of answer whizzed past your eyes, said "I wish I hadn't asked?""

The use of that exact answer (quickly obtained or not) is questionable.  Then, of course, the Abel-Ruffini theorem says that there is no general formula for solving polynomials of degree $5$ or higher in terms of radicals.  For degree $5$ polynomials, there is a solution in terms of elliptic functions; again, it's complicated enough that it's of questionable use.  Then there is Galois theory which describes the algebraic structures of polynomials.  See the interesting historical essay by Nick Trefethen on  What we learned from Galois .

The lesson here is that even when you can solve something exactly, maybe you shouldn't.

There are some interesting things you can do with univariate polynomials of high degree, including with the algebraic numbers that are their roots.  But computation with them isn't so easy.  SymPy actually has some quite advanced features for polynomials, including multivariate polynomials.

 Symbolic computation with functions

Let's try some calculus-like things.

```python
y = symbols('y')

solveset(Eq(exp(y), x), y)
```

    ImageSet(Lambda(_n, I*(2*_n*pi + arg(x)) + log(Abs(x))), S.Integers)

RMC really doesn't like that "solution"!  It has separated the real and imaginary parts without needing to.  A perfectly good answer would be $\ln_n(x)$, which looks a lot simpler.

$\ln_k(z)$, which might not look familiar to you, means $\ln(z) + 2\pi i k$.  Also, SymPy has chosen to emulate Maple and use I for the square root of $-1$, which made a kind of sense in the 1980's when Maple chose to do it that way, but given all the fonts we have nowadays that doesn't seem sensible at all.

Fine.  We will live with it.  The solution is not actually wrong.

```python
solveset(Eq( y*exp(y), x), y)
```

    ConditionSet(y, Eq(-x + y*exp(y), 0), Complexes(S.Reals x S.Reals, False))

Oh, that's disappointing.  See  the Wikipedia article on Lambert W to see what that should have been.

```python
solve( Eq( y*exp(y), x ), y )
```

    [LambertW(x)]

That's better, but---like the logarithm above---there should be multiple branches.

 Integrals, and the difference between Computer Algebra and Symbolic Computation

We'll start with a nasty example.  You can find nice examples of SymPy and integration in many places, so we will assume that you have seen instances of computer implementations of the fundamental theorem of calculus: to find areas under curves by using antiderivatives.  The nasty integral that we will use is

$$
f(x) = \frac{3}{5+4\cos x}
$$

and we will try to integrate this (infinitely smooth) function on various intervals in the real axis.  Since $\cos x$ is never larger than $1$ for real $x$, the denominator of that function is always positive, so the function is always positive.  Therefore the integral of $f(x)$ from any $a$ to any $b > a$ will also be positive.  Positive functions have positive area underneath them, end of story.

```python
from matplotlib import pyplot as plt
```

```python
# plt imported in a previous cell.
import numpy as np
n = 2021
xi = np.linspace(-2*np.pi, 2*np.pi, n)
yi = np.zeros(n)
for i in range(n):
    yi[i] = 3.0/(5.0+4.0*np.cos(xi[i]))
# Trapezoidal rule is spectrally accurate for periodic integrands
area = np.trapz(yi, x=xi)
print( 'Area under the curve from 0 to 2pi is approximately ', area/2/np.pi, ' times pi ')
plt.plot(xi,yi,'k,')
plt.axis([-2*np.pi, 2*np.pi, 0, 4])
plt.show()
```

```python
f = 3/(5+4*cos(x))
F = integrate(f, x )
print( 'The integral from SymPy is ', F )
FTOC = F.subs(x,2*pi) - F.subs(x,0)
print( 'The area under the curve from 0 to 2pi is positive, not ', FTOC )
defint = integrate( f, (x,0,2*pi))
print( 'No, the area is positive, not ', defint )
```

    The integral from SymPy is  -I*log(tan(x/2) - 3*I) + I*log(tan(x/2) + 3*I)
    The area under the curve from 0 to 2pi is positive, not  0
    No, the area is positive, not  0

As of this writing, many computer algebra systems (not just SymPy) are broken by this example.  Maple's indefinite integral returns something that looks a little nicer, namely
$$
\int f(x)\,dx = 2 \arctan \! \left(\frac{\tan \! \left(\frac{x}{2}\right)}{3}\right)
$$
but when you evaluate that at $x=2\pi$ you get the same value that you do at $x=0$, because the function is periodic.  So the difference is, as with SymPy's answer, wrongly zero.

There is something very important happening in that expression, though, which also happens in that more complex expression returned by SymPy: there is a jump discontinuity at $x=\pi$.  The computer algebra system has returned a discontinuous function as an antiderivative of a continuous function.

Anyone remember the tl;dr of the Fundamental Theorem of Calculus?  Likely not.  Even we have to look it up before we teach it, to be sure we have the fine details right.  The basic idea is, though, that integration smooths things out: integrate a continuous function, you get a continuously differentiable function, which is smoother.  Jump discontinuities spuriously introduced are right out.

So, what the computer algebra system is doing there is not integration.  Anyone get their knuckles rapped for not adding the "constant of integration" when you integrated on a test?  Lost a mark or two, maybe?

Turns out that's what's going on here.  By adding different constants in different intervals to this purported antiderivative, we can find a smooth antiderivative of $f(x)$.  The answer returned by SymPy (and Maple) is "correct" from the point of view of differential algebra, where constants are things that differentiate to zero and don't have "values" as such.

Many people who implement computer algebra systems (even some of our friends at Maple) argue that they are right and we are wrong, and don't see the point.  But they are wrong, and one day they'll get it.  Now, they do have a fix-up in their definite integration code: if you ask Maple int( f, x=0..2\*Pi) you will get the correct answer $2\pi$.  But you have to ask the right way, by using the syntax for definite integrals (as you see above, SymPy does not give the right answer that way).

Matlab gets it.  Matlab's Symbolic Toolbox works quite hard to get continuous antiderivatives.  So we have confidence that Maple will one day get it. But Wolfram Alpha gets a discontinuous antiderivative, too, at this time of writing, so there's only a little pressure from the competition.

These facts have been known for quite a while now.  See The Importance of Being Continuous by David Jeffrey, a paper that was written in 1994 (long predating Python or SymPy).

The disagreement amongst mathematicians---algebra versus analysis---has been going on for much longer, and goes at least back to Cauchy.  We are on Cauchy's side, here.  But there are lots of people who just want formulas, and don't care if they are discontinuous.

RMC retweeted something apropos this morning: Prof Michael Kinyon said "I used to think it was weird when people disagreed with me, but now I understand that some people just enjoy the feeling of being wrong."


# Chapter 0 なぜ`Sympy`か?

- SymPyは完全にフリーソフトウェア. MapleやMathematicaのようにお金をかけなくて良い.

- Pythonをベースにしているから. 完全にPython上で実行される. PythonのSyntaxも同様に使える.

- Sage等もPythnoを使っているが、ダウンロード容量が大変多い. それに対してSymPyは非常に軽量. Python以外に依存するソフトウェアがない.

- ライブラリとして使える. Pythonでimportするだけでいろいろな機能が利用可能になる.

# Chapter 1 Introduction

本章では、SymPyではどういうことができるのかを見ていく。詳しい内容は次章以降で見ていくので、ここではざっと読み通す程度で良い。

**用語について**:
チュートリアル原文では、SymPyで扱う変数を*SymPyシンボル*と呼んでいます。Pythonではシンボルに対応する概念がないのですが、チュートリアルに従っています。*SymPyオブジェクト*と呼んだときは、*SymPy*で定義されている関数・メソッドや、*SymPy*シンボル等を含めて総称として呼ぶことにします。

### 1.1 シンボリックな計算とは何か?

Pythonの標準ライブラリ`math`モジュールでは

```python
import math
```

```python
math.sqrt(9)
```

    3.0

という計算ができる。では8はどうか？この場合は

```python
math.sqrt(8)
```

    2.8284271247461903

となり、近似値を与える。 ただよく知っているように$$\sqrt{8}=2\sqrt{2}$$である。この結果は上の計算からは導けない。つまり、より式を簡単にして、見慣れた形にしてから近似値を出したい。-> シンボリックな計算の必要性. `Sympy`ならば

```python
import sympy
sympy.sqrt(3)
```

    sqrt(3)

と出してくれる。**近似値ではなく、数式として簡単なものがほしいとき**に、`Sympy`が使える。

### 1.2 より面白い例

`Sympy`は方程式も扱える:

```python
from sympy import symbols #一変数に対してもsymbolsが使える.
```

```python
x, y = symbols('x, y') #変数として扱いたいものにsymbols()関数を使う.
```

```python
expr = x + 2*y
```

```python
expr
```

    x + 2*y

`x`, `y`がPythonオブジェクトならば、これらに値が格納されていないと意味をなさない。(次章から詳しく見ていくように、エラーを出す。) 一方で`Sympy`シンボルとして(`symbols`を用いて)宣言すれば、それらは変数の格納有無にかかわらず、数学で私達が扱っているように、変数そのものとして扱うことができるようになる。

*注意* 上で扱った変数に対して、以下のように`Python`オブジェクトの代入操作を行うと、

```python
x = 4
y = 2
```

```python
expr
```

    x + 2*y

となって、`expr`に代入されている`Sympy`オブジェクトには何ら影響を与えない。(変数への代入は、`subs()`を用いる。(後述) )

--> この変数`expr`を使えば、変数`x`, `y`を用いて、

```python
expr + 1
```

    x + 2*y + 1

```python
expr - x
```

    x + 2*y - 4

といった計算もできる。ところで、文字同士の計算にもかかわらず、`x + 2*y - x`といった結果は出なかった。 **変数同士の演算も可能**

予想されるように、`expr`に`x`をかけてみると、

```python
x*expr
```

    4*x + 8*y

を得る。

展開した式がほしいときは、`expand`関数を用いて

```python
from sympy import expand
```

```python
expr_expand = expand(x*expr)
```

```python
expr_expand
```

    4*x + 8*y

とすればよい。

逆に因数をまとめて整理したいときは、`factor`関数を用いて

```python
from sympy import factor
```

```python
factor(expr_expand)
```

    4*(x + 2*y)

とすればよい。このように自由に表式を行き来できる。

### 1.4 強力なシンボリック計算

Sympyでは**行列計算、極限、微分、積分、方程式**を解く、その他もろもろが可能. LaTeXでグラフや式の出力も可能。幅広い数学の計算ができる！

```python
from sympy import * #すべての関数をimport
```

```python
x, t, z, nu = symbols('x, t, z, nu')
```

```python
init_printing(use_unicode=True) #unicode文字を出力するため.
```

☆関数$$\sin(x)e^{2}$$の微分

```python
diff( sin(x)*exp(x), x)
```

☆不定積分: $$\int \, (e^{x}\sin(x) + e^{x}\cos(x))\, dx$$

```python
integrate(exp(x)*sin(x) + exp(x)*cos(x), x)
```

☆定積分:$$\int_{-\infty}^{\infty}\sin(x^{2})dx$$

```python
integrate(sin(x**2), (x, -oo, oo)) #無限大はo(オー)２つ.
```

☆極限: $$\lim_{x\rightarrow 0}\frac{\sin(x)}{x}$$

```python
limit(sin(x)/x, x, 0)
```

☆方程式:$$x^2-2=0$$を解く

```python
solve(x**2 - 2, x)
```

☆微分方程式:$$\frac{d^2y}{dt^2}-y=e^{t}$$をyについて解く.

```python
y = Function('y')
```

```python
dsolve(Eq(y(t).diff(t,t) - y(t), exp(t)), y(t))
```

☆行列:$$\left[
    \begin{array}{cc}
    1 & 2 \\
    2 & 2 \\
    \end{array}\right]
    $$の固有値を見つける.

```python
Matrix([[1, 2],
        [2, 2]]).eigenvals() #行列はこう書いたほうがわかりやすい。
```

☆ベッセル関数$$J_{\nu}(z)$$を球ベッセル関数$$j_{\nu}(z)$$を用いて書く.

```python
besselj(nu, z).rewrite(jn)
```

☆式: $$\int_{0}^{\pi}\cos^2(x)dx$$を$$\LaTeX$$で出力.

```python
latex(Integral(cos(x)**2, (x, 0, pi)))
```

    '\\int_{0}^{\\pi} \\cos^{2}{\\left (x \\right )}\\, dx'

次は[Chapter2 Gotchas](https://hiroyuki827.github.io/SymPy_tutorial/Chapter2_Gotchas.html)へ!


```julia
using Distributions
using Plots
using WebIO

WebIO.install_jupyter_nbextension()

using Interact
```

    Unable to load WebIO. Please make sure WebIO works for your Jupyter client.
    For troubleshooting, please see
    the WebIO/IJulia documentation.

    ┌ Info: Installing Jupyter WebIO extension...
    │   cmd = `/home/dominik/python-venvs/default/bin/jupyter nbextension install --user /home/dominik/.julia/packages/WebIO/2nnB1/deps/bundles/webio-jupyter-notebook.js`
    └ @ WebIO /home/dominik/.julia/packages/WebIO/2nnB1/deps/jupyter.jl:237
    Up to date: /home/dominik/.local/share/jupyter/nbextensions/webio-jupyter-notebook.js

        To initialize this nbextension in the browser every time the notebook (or other app) loads:

              jupyter nbextension enable <the entry point> --user

    ┌ Info: Enabling Jupyter WebIO extension...
    │   cmd = `/home/dominik/python-venvs/default/bin/jupyter nbextension enable --user webio-jupyter-notebook`
    └ @ WebIO /home/dominik/.julia/packages/WebIO/2nnB1/deps/jupyter.jl:241
    Enabling notebook extension webio-jupyter-notebook...
          - Validating: OK

# Weber's Law
Weber's law (WL), initially reported in 1834 (Weber, 1834) and later formalized by Fechner (1860), states that the just-noticeable-difference (JND) between two stimuli grows linearly with the stimulus magnitude:

\begin{align}
    \Delta_{\theta} = w \cdot \theta,
\end{align}

where $w = \frac{ \Delta_{\theta}}{\theta}$ is also known as the Weber fraction (WF).

It is important to note that the JND is a statement about a discrimination task. The JND $ \Delta_{\theta}$ is the difference in two stimuli $\theta_1 = \theta + \frac{ \Delta_{\theta}}{2}$ and $\theta_2 = \theta - \frac{ \Delta_{\theta}}{2}$ around a value $\theta$ that is noticed in a fraction $p$ of repetitions.

```julia
θ = 0:0.01:10

@manipulate for w=0.1:0.1:1.0
    plot(θ, w*θ, title="Weber's law", xlabel="theta", ylabel="JND", ylim=(0,10), label="")
end
```

# Probabilistic Models of Perception
In probabilistic models of perception (Ma, Kording & Goldreich, unpublished), one typically assumes that observers have access to a noisy measurement $m$ of a presented stimulus $\theta$ with some probability distribution $p(m | \theta)$, which is often called the measurement (or noise) distribution. The observer's goal is then to compute an estimate of the presented stimulus $\hat \theta(m)$. Critically, while the observer's noisy measurement is a random variable, the estimate is a completely deterministic function of the measurement $m$. As an experimenter, we do not know the noisy internal measurement of an observer in any given trial. We can, however, compute the distribution of the estimates given a particular stimulus $p(\hat\theta | \theta)$.

JNDs are usually expressed in probabilistic model of perception in the following way. For a small $\Delta_{\theta}$, we can locally assume that the measurement distributions are Gaussian with equal variance $\sigma_\theta^2$. Then, in a 2-AFC discrimination task, the distribution of the difference of the measurements is Gaussian with variance $\frac{\sigma_\theta^2}{2}$ and thus the probability of estimating that $\hat\theta_1 > \hat\theta_2$ is given by a cumulative Gaussian with mean $\theta_1 - \theta_2$ and standard deviation $\frac{\sigma_\theta}{\sqrt{2}}$:

\begin{align}
    p(\hat\theta_1 - \hat\theta_2 | \theta_1, \theta_2) = \Phi(\theta_1 - \theta_2 | 0, \frac{\sigma_\theta}{\sqrt{2}}) = \Phi(\Delta_{\theta} | 0, \frac{\sigma_\theta}{\sqrt{2}})
\end{align}

Thus, for a given discrimination probability $p$, we have $\Delta_{\theta} = \Phi^{-1}(p | 0, \frac{\sigma_\theta}{\sqrt{2}})$, from which follows that

\begin{align}
    \Delta_{\theta} \propto \frac{\sigma_\theta}{\sqrt{2}},
\end{align}

where the proportionality constant depends on $p$. The $\sqrt{2}$ factor results from the 2-AFC task and would be different for tasks with more choice alternatives. This means that the standard deviation of the assumed measurement distribution is the same as the discriminability (up to a constant factor).

If we now assume that WL holds (or better, if we have measured that it holds), we can interpret this as a statement about the standard deviation of the measurement distribution. Particularly, it means that we get a linear scaling of the standard deviation with the stimulus magnitude

\begin{align}
    \sigma_\theta = w \cdot \theta.
\end{align}

This linear scaling of standard deviation is often used in combination with Gaussian measurement distributions in probabilistic observer models (e.g. Jazayri & Shadlen, 2010; Cicchini et al, 2012).

Keep in mind that we have not used the assumption of linearly scaled standard deviation in measuring the discrimination thresholds, but have assumed locally that the standard deviation is constant. For measurements of $\Delta_{\theta}$ at different reference stimuli, we can check whether WL holds.

An alternative to Gaussian noise with linearly scaled standard deviations is a logarithmic encoding of the stimuli. Here, the assumption is that $\psi = \ln{\theta}$ is Gaussian distributed with equal variance $\sigma^2_{\text{log}}$ everywhere. In this case, the measurements on the physical stimulus dimension follow a log-normal distribution:

\begin{align}
    p(m | \theta) = \text{Lognormal}(m | \ln{\theta}, \sigma_{\text{log}}).
\end{align}

Since the standard deviation of the log-normal distribution is $\sqrt{[e^{\sigma_{\text{log}}^2} - 1] \, e^{2 \ln \theta + \sigma_{\text{log}}^2}}$, it can be written as a linear function of the stimulus magnitude: $\sigma_\theta = c \cdot \theta$ with $c = \sqrt{[e^{\sigma_{\text{log}}^2} - 1]\,e^{\sigma_{\text{log}}^2}}$.

In probabilistic models of perception, the log-normal assumption is a popular way of dealing with magnitude variables, for which WL scaling of standard deviation is a good approximatition (e.g. Petzschner & Glasauer, 2011; Battaglia et al., 2011). This assumption makes computing estimates more convenient and avoids some problems of the Gaussian case, as we will see in the following sections.

```julia
measurement_norm(theta; wf=0.5) = Normal(theta, wf * theta)
measurement_logn(theta; wf=0.5) = LogNormal(log(theta), wf)
```

    measurement_logn (generic function with 1 method)

For the standard deviations scaled by a WF of 0.5 as demonstrated above, the log-normal distribution is very different from the normal distribution. For other parameters, approximating the log-normal by the normal works rather well. Let us now examine for which range of parameters this approximation holds.

```julia
m = 0:0.01:10

@manipulate for θ=1.0:0.5:5, w=0.1:0.05:1.0
    plot([θ, θ], [0., 0.5], linestyle=:dot, color=2, label="theta", xlabel="m", ylabel="p(m|theta)")
    plot!(m, pdf.(measurement_norm(θ, wf=w), m), color=0, label="norm", ylim=(0, 1.0))
    plot!(m, pdf.(measurement_logn(θ, wf=w), m), color=1, label="lognorm")
end
```

# Maximum-likelihood estimates
The observer does not have access to the actually presented stimulus $\theta_0$. If we assume that they still have knowledge about their own noise distribution, they can make use of the likelihood function. The likelihood function is the probability of the observed measurement $m$ as a function of the "true" stimulus $\theta$:

\begin{align}
    \lambda(\theta; m) = p(m | \theta).
\end{align}

Note that the likelihood function is different from the measurement distribution, since it is a function of $\theta$. For the Gaussian with scaled standard deviation, the shape of the likelihood function is no longer Gaussian but has a heavier tail to the right due to the increasing variance.

```julia
θ = 0:0.01:10

@manipulate for m=1:0.5:5, w=0.1:0.05:1.0
    plot([m, m], [0., 0.5], linestyle=:dot, color=2, label="m", xlabel="theta", ylabel="p(m|theta)")
    plot!(θ, pdf.(measurement_norm.(θ, wf=w), m), label="norm", ylim=(0, 0.5), color=0)
    plot!(θ, pdf.(measurement_logn.(θ, wf=w), m), label="lognorm", ylim=(0, 1.0), color=1)
end
```

An estimate $\hat\theta(m)$ is usually computed from this likelihood function (and possibly a prior distribution and a cost function). Here, we focus first on the simplest case, a maximum likelihood (ML) estimate $\hat\theta(m) = \text{argmax}_\theta \lambda(\theta; m)$ and compare the normal with scaled standard deviation and log-normal models.

## Normal distribution with scaled variance
For the normal distribution with scaled variance, the likelihood function is

\begin{align}
    \lambda(\theta; m) = p(m | \theta) = \frac{1}{w\theta \sqrt{2\pi}} \exp\left\{\frac{-(m - \theta)^2}{2(w\theta)^2}\right\}.
\end{align}

The ML estimate can then be computed (see Jazayeri & Shadlen, 2010) as

\begin{align}
    \hat\theta_{\text{MLE}} = \text{argmax}_\theta \lambda(\theta; m) = m \left[\frac{\sqrt{1 + 4w^2} - 1}{2 w^2} \right].
\end{align}

The proportionality constant is smaller than 1, resulting in a systematic underestimation of the true $\theta$, which becomes more severe the higher the WF.

## Log-normal distribution
Assuming that measurements are log-normally distributed is the same as assuming that the log of the measurements are normally distributed. The likelihood function in this case is

\begin{align}
    \lambda(\theta; m) =  p(m | \theta) = \frac{1}{m\sigma_{\text{log}}\sqrt{2\pi}} \exp\left\{ \frac{- (\ln m - \ln \theta)^2}{2 \sigma_{\text{log}}^2} \right\}.
\end{align}

Because the normalizing constant does not depend on $\theta$ anymore, the ML esimate is simply $\hat\theta(m) = m$.

```julia
using Optim

function maximum_likelihood(d, m; wf)
    f(x) = -logpdf(d(x, wf=wf), m)

    result = optimize(f, 0., 10)
    return Optim.minimizer(result)
end

linreg(x, y) = [ones(size(x)) x] \ y
```

    linreg (generic function with 1 method)

```julia
m = 0.01:0.01:5

@manipulate for w=0.1:0.05:1.0
    # TODO: compute maximum likelihood for different values of m
    theta_hat = [maximum_likelihood(measurement_norm, x, wf=w)[1] for x in m]
    theta_hat_log = [maximum_likelihood(measurement_logn, x, wf=w)[1] for x in m]

    plot(m, theta_hat, label="norm", color=0)
    plot!(m, theta_hat_log, label="lognorm", color=1)
    plot!(m, m, linestyle=:dash, label="theta^(m) = m", xlabel="m", ylabel="theta^")
end
```

# Posterior estimates

```julia

```


# Realization of Non-Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing.

## Introduction

Computing the output $y[k] = \mathcal{H} \{ x[k] \}$ of a [linear time-invariant](https://en.wikipedia.org/wiki/LTI_system_theory) (LTI) system is of central importance in digital signal processing. This is often referred to as [*filtering*](https://en.wikipedia.org/wiki/Digital_filter) of the input signal $x[k]$. The methods for this purpose are typically classified into

* non-recursive and
* recursive

techniques. This section focuses on the realization of non-recursive filters.

### Non-Recursive Filters

An LTI system can be characterized completely by its impulse response $h[k]$

The output signal $y[k]$ is given by (linear) convolution of the input signal $x[k]$ with the impulse response $h[k]$

\begin{equation}
y[k] = x[k] * h[k] = \sum_{\kappa = -\infty}^{\infty} x[\kappa] \; h[k-\kappa]
\end{equation}

Two aspects of this representation become evident when inspecting above equation:

1. The output signal $y[k]$ is a linear combination of the input signal $x[k]$. There is no feedback of the output signal of past time-instants. Therefore, such filters are termed as *non-recursive* filters.

2. In order to compute the output signal at one particular time-instant $k$, the input signal needs to be known for all past and future time-instants.

The second aspect prohibits a practical realization. In order to be able to realize a non-recursive filter by convolution, the output at time-instant $k$ should only depend on the input signal $x[k]$ up to time-index $k$

\begin{equation}
y[k] = \sum_{\kappa = -\infty}^{k} x[\kappa] \; h[k-\kappa]
\end{equation}

This is the case when the impulse response is causal, hence when $h[k] = 0$ for $k<0$. However, this still requires knowledge of the input signal for all past time-instants. If we further assume that the input signal is causal, $x[k] = 0$ for $k<0$, we get

\begin{equation}
y[k] = \sum_{\kappa = 0}^{k} x[\kappa] \; h[k-\kappa]
\end{equation}

### Finite Impulse Response

Many practical systems have an impulse response of finite length $N$ or can be approximated by an impulse response of finite length

\begin{equation}
h_N[k] = \begin{cases} h[k] & \text{ for } 0 \leq k < N \\ 0 & \text{ otherwise} \end{cases}
\end{equation}

Such an impulse response is denoted as [*finite impulse response*](https://en.wikipedia.org/wiki/Finite_impulse_response) (FIR). Introducing $h_N[k]$ into above sum and rearranging terms yields

\begin{equation}
y[k] = \sum_{\kappa = 0}^{k} x[\kappa] \; h_N[k-\kappa] = \sum_{\kappa = 0}^{N-1} h_N[\kappa] \; x[k-\kappa]
\end{equation}

Hence for a causal input signal $x[k]$ and a FIR the output of the system can be computed by a finite number of operations.

The evaluation of the convolution for a FIR of length $N$ requires $N$ multiplications and $N-1$ additions per time index $k$. For the real-time convolution of an audio signal with a sampling rate of $f_\text{S} = 48$ kHz with a FIR of length $N = 48000$ we have to compute around $2 \times 2.3 \cdot 10^9$ numerical operations per second. This is a considerable numerical complexity, especially on embedded or mobile platforms. Therefore, various techniques have been developed to lower the computational complexity.


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
import warnings
```

# Numerical Quadrature

**Goal:** Accurately Evaluate definite integrals

$$
\int^b_a f(x) dx
$$

using a finite number of samples

### Why?

1. Many integrals do not have closed form solutions

$$
   \int^b_a \sqrt{1 + \cos^2 x} dx
$$

Solution to ordinary differential equations

   $$\frac{\text{d}^2 u}{\text{d}t^2} = f\left(u, \frac{\text{d} u}{\text{d}t}, t \right)$$

   Defining $v = \frac{\text{d} u}{\text{d}t}$ then leads to

   $$\begin{bmatrix}
   \frac{\text{d} v}{\text{d}t} \\ \frac{\text{d} u}{\text{d}t} \end{bmatrix} = \begin{bmatrix} f(u, v, t) \\ v \end{bmatrix}$$

   which can be solved by integration

   $$\begin{bmatrix}
   v \\ u \end{bmatrix} = \begin{bmatrix} v(t_0) + \int^t_{t_0} f(u, v, \hat{t}) d\hat{t} \\ u(t_0) + \int^t_{t_0} v d\hat{t} \end{bmatrix}$$

Solving partial differential equations
$$
    u_t = \nabla^2 u
$$

### Basics of Quadrature

We want to approximate an integral $I$ with some approximation $I_N$ such that
$$
    I = \int^b_a f(x) dx \approx I_N = \sum^{N}_{i=1} w_i f(x_i)
$$
where the $x_i$ are the *quadrature points* or *nodes* and the $w_i$ are the *weights*.  Usually a particular quadrature rule specifies the points $x_i$ resulting in a particular set of weights $w_i$.

Convergence requires that
$$
    \lim_{N \rightarrow \infty} I_N = I.
$$

## Riemann Sums

Given $f(x)$ and a partition of the interval $[a,b]$ with $\{x_i\}^N_{i=0}$ and $a = x_0 < x_1 < \ldots < x_N = b$ and $x^*_i \in [x_i, x_{i+1}]$ we define the Riemann integral as

$$\int^b_a f(x) dx = \lim_{N\rightarrow \infty} \sum^{N-1}_{i=0} f(x_i^*) (x_{i+1} - x_i)$$

This is a general definition and leads to a number of quadrature approaches that depend on how we pick $x_i^* \in [x_i, x_{i+1}]$.

### Midpoint Rule

Choose $x_i^*$ such that

$$x_i^* = \frac{x_{i+1} + x_i}{2}$$

so that

$$I[f] = \int^b_a f(x) dx \approx \sum^{N-1}_{i=0} f\left(\frac{x_{i+1} + x_i}{2} \right ) (x_{i+1} - x_i) = I_N[f]$$

over $\Delta x_i = x_{i+1} - x_i$

#### Example:  Integrate using midpoint rule

Calculate and illustrate the midpoint rule.  Note that we are computing the cummulative integral here:

$$
    \int^x_0 \sin(\hat{x}) d\hat{x} = \left . -\cos \hat{x} \right|^x_0 = 1 - \cos x
$$

```python
# Note that this calculates the cummulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 40
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
x_star = 0.5 * (x_hat[1:] + x_hat[:-1])
delta_x = x_hat[1] - x_hat[0]
```

```python
fig = plt.figure(figsize=(8,6))
fig.subplots_adjust(hspace=.5)
axes = fig.add_subplot(2, 1, 1)

axes.plot(x, numpy.zeros(x.shape), 'k--')
axes.plot(x, f(x), 'b')

for i in range(num_partitions):
    axes.plot([x_hat[i], x_hat[i]], [0.0, f(x_star[i])], 'k--')
    axes.plot([x_hat[i + 1], x_hat[i + 1]], [0.0, f(x_star[i])], 'k--')
    axes.plot([x_hat[i], x_hat[i + 1]], [f(x_star[i]), f(x_star[i])], 'k--')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Partition and $f(x)$")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-1.1, 1.1))

I_hat = numpy.zeros(x_star.shape)
I_hat[0] = f(x_star[0]) * delta_x
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + f(x_star[i]) * delta_x

axes = fig.add_subplot(2, 1, 2)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_star + delta_x / 2.0, I_hat, 'ko')

err = numpy.abs(I(x_star) - I_hat)

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral, Max Err = {}".format(err.max()))
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))

plt.show()
```

### Newton-Cotes Quadrature

Using $N+1$ equally spaced points, evaluate $f(x)$ at these points and exactly integrate the interpolating polynomial:

$$I_N[f] = \int^b_a P_N(x) dx$$

#### Trapezoidal Rule

Use $N = 1$ polynomial to derive the trapezoidal rule.

Trapezoidal rule uses $N = 1$ order polynomials between each point (i.e. piece-wise defined linear polynomials).  Using the Linear Lagrange basis we can write the interpolating polynomial as

$$
    P_1(x) = f(x_i)\frac{x_{i+1}-x}{\Delta x} + f(x_{i+1})\frac{x - x_i}{\Delta x}
$$

where $\Delta x = x_{i+1}-x_i$ is the width of the interval.

Integrating this polynomial we have

\begin{aligned}
    I_N[f] &= \int^{x_{i+1}}_{x_i} f(x_i)\ell_0(x) + f(x_{i+1})\ell_1(x) dx  \\
    &=\frac{1}{\Delta x}  \left. \left[ f(x_i)\left(x_{i+1}x - \frac{x^2}{2}\right) + f(x_{i+1})\left(\frac{x^2}{2}-xx_i\right) \right] \right|_{x_i}^{x_{i+1}} \\
    &= \frac{1}{\Delta x}  \left[ f(x_i)\left(\frac{x_{i+1}^2}{2} -x_ix_{i+1} + x_i^2\right) + f(x_{i+1})\left(\frac{x_{i+1}^2}{2} -x_ix_{i+1} + x_i^2\right) \right]\\
    &= \frac{(x_{i+1} - x_i)^2}{2\Delta x}\left[ f(x_i) + f(x_{i+1})\right]\\
    & = \frac{\Delta x}{2}\left[ f(x_i) + f(x_{i+1}) \right] \\
\end{aligned}

We can also simplify the sum over all the intervals by noting that all but the end points will have total contribution of $\Delta x$ to the entire sum such that

$$
    I_N[f] = \frac{\Delta x}{2} (f(x_0) + f(x_N) ) + \sum^{N-1}_{j=1} \Delta x f(x_j)
$$

This is known as the composite or extended trapezoidal rule.

```python
# Note that this calculates the cummulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 20
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
delta_x = x_hat[1] - x_hat[0]
```

```python
fig = plt.figure()
fig.subplots_adjust(hspace=.5)
axes = fig.add_subplot(2, 1, 1)

axes.plot(x, numpy.zeros(x.shape), 'k--')
axes.plot(x, f(x), 'b')

for i in range(num_partitions):
    axes.plot([x_hat[i], x_hat[i]], [0.0, f(x_hat[i])], 'k--')
    axes.plot([x_hat[i + 1], x_hat[i + 1]], [0.0, f(x_hat[i+1])], 'k--')
    axes.plot([x_hat[i], x_hat[i + 1]], [f(x_hat[i]), f(x_hat[i+1])], 'k--')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Partition and $f(x)$")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-1.1, 1.1))

I_hat = numpy.zeros(x_hat.shape)
I_hat[0] = (f(x_hat[1]) + f(x_hat[0])) * delta_x / 2.0
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + (f(x_hat[i + 1]) + f(x_hat[i])) * delta_x / 2.0

axes = fig.add_subplot(2, 1, 2)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_hat + delta_x, I_hat, 'ko')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))

plt.show()
```

#### Simpson's Rule

Simpson's rule uses $N = 2$ order polynomials between each point (i.e. piece-wise defined quadratic polynomials).

The polynomial has the form

$$P_2(x) = \frac{2 f(x_i)}{\Delta x^2} \left (x - \frac{\Delta x}{2} \right ) (x - \Delta x) - \frac{4 f\left(x_i + \frac{\Delta x}{2}\right)}{\Delta x^2}  x (x - \Delta x) + \frac{2 f(x_{i+1})}{\Delta x^2} x \left (x - \frac{\Delta x}{2} \right )$$

Integrating this polynomial we have

$$
    I_N[f] = \int^{x_{i+1}}_{x_i} P_2(x) dx = \frac{\Delta x}{6} f(x_i) + \frac{2 \Delta x}{3} f\left(x_i + \frac{\Delta x}{2} \right ) + \frac{\Delta x}{6} f(x_{i+1})
$$

We can also show this by using the method of undetermined coefficients.

Use the general form of the quadrature rule and determine weights $w_j$ by using functions we know the solution to.  These functions can be any representation of polynomials up to order $N=2$ however the monomials $1$, $x$, $x^2$ are the easiest in this case.

$$
    I_{\Delta x}[f] = w_0 f(0) + w_1 f(\Delta x / 2) + w_2 f(\Delta x)
$$

$$\begin{aligned}
    &\text{if}~f = 1:  &I[f] =  \int^{\Delta x}_{0} 1 dx = \Delta x  \\
    &\text{if}~f = x:  &I[f] =  \int^{\Delta x}_{0} x dx = \frac{\Delta x^2}{2} \\
    &\text{if}~f = x^2:  &I[f] =  \int^{\Delta x}_{0} x^2 dx = \frac{\Delta x^3}{3}
\end{aligned}$$

What are the corresponding systems of equations?

$$
    I[f] = w_0 f(0) + w_1 f(\Delta x / 2) + w_2 f(\Delta x)
$$

$$\begin{aligned}
    &\text{if}~f = 1:  &I[f] =  \int^{\Delta x}_{0} 1 dx = \Delta x & & I_N[1] &= w_0 + w_1 + w_2 \\
    &\text{if}~f = x:  &I[f] =  \int^{\Delta x}_{0} x dx = \frac{\Delta x^2}{2} & & I_N[x] &= w_1 \frac{\Delta x}{2} + w_2\Delta x\\
    &\text{if}~f = x^2:  &I[f] =  \int^{\Delta x}_{0} x^2 dx = \frac{\Delta x^3}{3} & & I_N[x^2] &= \frac{\Delta x^2}{4} w_1 + w_2\Delta x^2\\
\end{aligned}$$

We then have the system of equations:
$$\begin{aligned}
    w_0 &+& w_1 &+& w_2 &=\Delta x \\
        &\quad& \frac{\Delta x}{2} w_1 &+& \Delta x w_2  &= \frac{\Delta x^2}{2} \\
        &\quad& \frac{\Delta x^2}{4} w_1 &+& \Delta x^2 w_2 &=\frac{\Delta x^3}{6} \\
\end{aligned}$$

or

$$
\begin{bmatrix}
    1 & 1 & 1 \\
    0 & \Delta x / 2 & \Delta x \\
    0 & \Delta x^2 / 4 & \Delta x^2 \\
\end{bmatrix} \begin{bmatrix}
    w_0 \\ w_1 \\ w_2
\end{bmatrix} = \begin{bmatrix}
    \Delta x \\ \Delta x^2 / 2 \\ \Delta x^3 / 3
\end{bmatrix} \Rightarrow \begin{bmatrix}
    1 & 1 & 1 \\
    0 & 1 / 2 & 1 \\
    0 & 1 / 4 & 1 \\
\end{bmatrix} \begin{bmatrix}
    w_0 \\ w_1 \\ w_2
\end{bmatrix} = \begin{bmatrix}
    \Delta x \\ \Delta x / 2 \\ \Delta x / 3
\end{bmatrix}\Rightarrow
\begin{bmatrix}
    1 & 1 & 1 \\
    0 & 1 / 2 & 1 \\
    0 & 0 & -1 \\
\end{bmatrix} \begin{bmatrix}
    w_0 \\ w_1 \\ w_2
\end{bmatrix} = \begin{bmatrix}
    \Delta x \\ \Delta x / 2 \\ -\Delta x / 6
\end{bmatrix}
$$

Leading to

$$ w_2 = \frac{\Delta x}{6} \quad w_1 = \frac{2}{3} \Delta x \quad w_0 = \frac{\Delta x}{6}$$

Another way to write Simpson's rule is to use intervals of three points (similar to one of the ways we did this last time).  The formulation here effectively has a $\Delta x$ half of what the intervals show but is easier to program.

```python
# Note that this calculates the cummulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 10
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
delta_x = x_hat[1] - x_hat[0]
```

```python
fig = plt.figure()
fig.subplots_adjust(hspace=.5)
axes = fig.add_subplot(2, 1, 1)

axes.plot(x, numpy.zeros(x.shape), 'k--')
axes.plot(x, f(x), 'b')

for i in range(num_partitions):
    axes.plot([x_hat[i], x_hat[i]], [0.0, f(x_hat[i])], 'k--')
    axes.plot([x_hat[i + 1], x_hat[i + 1]], [0.0, f(x_hat[i + 1])], 'k--')
    coeff = numpy.polyfit((x_hat[i], x_hat[i] + delta_x / 2.0, x_hat[i + 1]),
                          (f(x_hat[i]), f(x_hat[i] + delta_x / 2.0), f(x_hat[i+1])), 2)
    x_star = numpy.linspace(x_hat[i], x_hat[i+1], 10)
    axes.plot(x_star, numpy.polyval(coeff, x_star), 'k--')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Partition and $f(x)$")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-1.1, 1.1))

I_hat = numpy.zeros(x_hat.shape)
I_hat[0] = delta_x * (1.0 / 6.0 * (f(x_hat[0]) + f(x_hat[1])) + 2.0 / 3.0 * f(x_hat[0] + delta_x / 2.0))
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + delta_x * (1.0 / 6.0 * (f(x_hat[i]) + f(x_hat[i+1])) + 2.0 / 3.0 * f(x_hat[i] + delta_x / 2.0))

axes = fig.add_subplot(2, 1, 2)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_hat + delta_x, I_hat, 'ko')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))

plt.show()
```

## Error Analysis

From before we have a particular quadrature scheme $I_N$ which we can also write as

$$
    I_N[f] = \sum^{N-1}_{i=0} w_i f(x_i).
$$

Define the error $E[f]$ such that

$$I[f] = I_N[f] + E_N[f]$$

The degree of $I_N[f]$ is the integer $N$ such that $E_N[p_i] = 0 \quad \forall i \leq n$ and $\exists p_{n+1}$ such that $E[p_{n+1}] \neq 0$.

### Quadrature Accuracy

We can also use our polynomial analysis from before to analyze the errors made using both of the aforementioned methods.  From Lagrange's theorem we have the remainder term as before which we can use to look at the error

$$R_N(x) = (x - x_0)(x - x_1) \cdots (x- x_N) \frac{f^{(N+1)}(c)}{(N+1)!}$$

and integrate it to find the form and magnitude of the error on a single interval.

To find the total error we must sum the error over all the intervals:

$$I[f] = \sum_{i=0}^N \int^{x_{i+1}}_{x_i} P_N(x) dx + \sum_{i=0}^N \int^{x_{i+1}}_{x_i} R_N(x) dx = I_N[f] + E_N[f]$$

as we defined before.

#### Trapezoidal error

With $N=1$ we have

$$R_1(x) = (x - x_i) (x - x_{i+1}) \frac{f''(c)}{2}$$

Integrating this leads to

$$\int^{x_{i+1}}_{x_i} (x - x_i) (x - x_{i+1}) \frac{f''(c)}{2} dx = \frac{\Delta x^3}{12} f''(c)$$

giving us a form for the error.

If we sum up across all the intervals the total error is

$$E_N[f] = -\frac{\Delta x^3}{12} \sum_{i=0}^{N} f''(c_i)$$

or more illustrative

$$E_N[f] = -\frac{1}{2} \Delta x^2 (b - a) \left [ \frac{1}{N} \sum^{N-1}_{i=0} f''(c_i) \right ]$$

where the expression in the brackets is the mean value of the second derivative over the interval $[a,b]$.  This also shows that the trapezoidal rule converges quadratically as $\Delta x \rightarrow 0$.

#### Simpson's Rule Error

Similarly here we have $N = 2$ and

$$R_2(x) = (x - x_i) \left(x - x_i - \frac{\Delta x}{2} \right) (x - x_{i+1}) \frac{f'''(c)}{3!}$$

Integrating and summing the error contributions we find

$$E_N[f] = -\frac{1}{180} (b - a) \Delta x^4 f^{(4)}(c)$$

Interestingly we have gained two orders of accuracy by increasing the polynomial order by only 1!

### Review:  Newton-Cotes Formulas and errors for a single interval

given $x\in[x_i, x_{i+1}]$ and $\Delta x = x_{i+1} - x_i$, all of the   Newton-Cotes quadrature formula's can be derived by integrating the interpolating polynomial through $N$ equi-spaced points across the interval.  The error term is simply the integral of the Lagrange remainder term $R_N(x)$

* Mid-point: 1-point quadrature rule
$$
    I_M[f] = \Delta x\, f(x_i + \Delta x/2) + O(\Delta x^3f'')
$$
* Trapezoidal Rule: 2-point quadrature rule
$$
    I_T[f] = \Delta x\, \left[\frac{1}{2}f(x_i) +\frac{1}{2}f(x_{i+1})\right] + O(\Delta x^3f'')
$$
* Simpson's Rule: 3-point quadrature rule
$$
    I_S[f] = \Delta x\, \left[ \frac{1}{6}f(x_i) +  \frac{2}{3}f(x_i +\Delta x/2) +\frac{1}{6}f(x_{i+1})\right] + O(\Delta x^5f^{iv})
$$

```python
func = lambda x: 1 - x**2 + numpy.sin(5*x)
x0 = 0.
x1 = .5
x_buf = .2
x = numpy.linspace(x0 - x_buf, x1+ x_buf, 100)
x_interval = numpy.linspace(x0, x1, 100)

x_samp = numpy.array([ 0.5*(x0 + x1) ])
f_samp = func(x_samp)

fig = plt.figure(figsize=(15,5))
axes = fig.add_subplot(1,3,1)
axes.plot(x,func(x),linewidth=3)
axes.plot(x_samp, f_samp, 'ro')
axes.plot(x_interval, f_samp*numpy.ones(x_interval.shape), 'k--')
axes.plot(x0*numpy.ones(2), [0., f_samp],'k--')
axes.plot(x1*numpy.ones(2), [0., f_samp],'k--')
axes.plot(x0*numpy.ones(2), [0., func(x0)],'b')
axes.plot(x1*numpy.ones(2), [0., func(x1)],'b')
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$x_i$',fontsize=16)
axes.text(x1-.02,-.2,'$x_{i+1}$',fontsize=16)
axes.set_title('Midpoint Rule',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_ylim((-0.25, 2.))
axes.fill_between(x_interval,0.,func(x_interval))
axes.fill_between(x_interval, 0, f_samp*numpy.ones(x_interval.shape), hatch='/', alpha=0)
axes.grid()

axes = fig.add_subplot(1,3,2)
x_samp = numpy.array([x0, x1])
f_samp = func(x_samp)
axes.plot(x,func(x),linewidth=3)
axes.plot(x_samp, f_samp, 'ro')
axes.plot(x_samp, f_samp, 'k--')
axes.plot(x0*numpy.ones(2), [0., f_samp[0]],'k--')
axes.plot(x1*numpy.ones(2), [0., f_samp[1]],'k--')
axes.plot(x0*numpy.ones(2), [0., func(x0)],'b')
axes.plot(x1*numpy.ones(2), [0., func(x1)],'b')
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$x_i$',fontsize=16)
axes.text(x1-.02,-.2,'$x_{i+1}$',fontsize=16)
axes.set_title('Trapezoidal Rule',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_ylim((-0.25, 2.))
axes.fill_between(x_interval,0.,func(x_interval))
axes.fill_between(x_samp,0.,f_samp, hatch='/', alpha=0)

axes.grid()

axes = fig.add_subplot(1,3,3)
x_samp = numpy.array([x0, (x0 + x1)/2., x1])
f_samp = func(x_samp)
p = numpy.polyfit(x_samp, f_samp, 2)
f_p2 = numpy.polyval(p, x_interval)
axes.plot(x,func(x),linewidth=3)
axes.plot(x_samp, f_samp, 'ro')
axes.plot(x_interval, f_p2, 'k--')
axes.plot(x0*numpy.ones(2), [0., f_samp[0]],'k--')
axes.plot(x1*numpy.ones(2), [0., f_samp[-1]],'k--')
axes.plot(x0*numpy.ones(2), [0., func(x0)],'b')
axes.plot(x1*numpy.ones(2), [0., func(x1)],'b')
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$x_i$',fontsize=16)
axes.text(x1-.02,-.2,'$x_{i+1}$',fontsize=16)
axes.set_title('Simpsons Rule',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_ylim((-0.25, 2.))
axes.fill_between(x_interval,0.,func(x_interval))
axes.fill_between(x_interval,0.,f_p2, hatch='/', alpha=0)

axes.grid()
plt.show()
```

##### Example 1:

If $f(x) = \sin \pi x$ look at the relative accuracy of midpoint, trapezoidal and simpson's rules for a single interval $x\in[0,1]$.

$$\begin{aligned}
    \text{Exact:}  &I[f] = \int^1_0 \sin \pi x = \left . \frac{-\cos \pi x}{\pi} \right |^1_0 = \frac{2}{\pi} \approx 0.636619772 \\
    \text{Midpoint:}  &I_M[f] = \Delta x f(1/2) = \sin (\pi / 2) = 1 \\
    \text{Trapezoid:}  &I_T[f] = \frac{\Delta x}{2} (\sin(0) + \sin(\pi)) = 0 \\
    \text{Simpson's:}  &I_S[f] = \frac{\Delta x}{6} \sin(0) + \frac{2 \Delta x}{3} \sin(\pi / 2) + \frac{\Delta x}{6} \sin(\pi) = \frac{2 \Delta x}{3} = \frac{2}{3}
\end{aligned}$$

### Cumulative Error for extended Newton-Cotes formulas

For $N$ panels of any given N-C quadrature formula over the interval $x\in[a,b]$, the approximate integral and error formulas are

\begin{align}
I[f] &= \sum_{i=0}^N I_t + E_N\\
\end{align}
where
$$
    E_N = (b-a)O(\Delta x^p f^{m}(c))
$$

* Extended Mid-point:
$$
    E_{N_M} = (b-a) O(\Delta x^2 f''')
$$
* Extended Trapezoidal Rule:
$$
    E_{N_M} = (b-a) O(\Delta x^2 f''')
$$
* Extended Simpson:
$$
    E_{N_M} = (b-a) O(\Delta x^4 f^{(v)})
$$

Where $\Delta x = \frac{b -a }{N}$

```python
# Compute the error as a function of delta_x for each method
f = lambda x: numpy.sin(numpy.pi * x)

num_partitions = numpy.array([ 2**n for n in range(0, 16)])
delta_x = numpy.empty(len(num_partitions))
error_mid = numpy.empty(len(num_partitions))
error_trap = numpy.empty(len(num_partitions))
error_simpson = numpy.empty(len(num_partitions))

I_true = 2.0/numpy.pi

for (j, N) in enumerate(num_partitions):
    x_hat = numpy.linspace(0.0, 1.0, N + 1)
    delta_x[j] = x_hat[1] - x_hat[0]

    # Compute Midpoint
    x_star = 0.5 * (x_hat[1:] + x_hat[:-1])
    I_hat = 0.0
    for i in range(0, N):
        I_hat += f(x_star[i]) * delta_x[j]
    error_mid[j] = numpy.abs(I_hat - I_true)/I_true

    # Compute trapezoid
    I_hat = 0.0
    for i in range(1, N):
        I_hat += (f(x_hat[i + 1]) + f(x_hat[i])) * delta_x[j] / 2.0
    error_trap[j] = numpy.abs(I_hat - I_true)/I_true

    # Compute simpson's
    I_hat = 0.0
    for i in range(0, N):
        I_hat += delta_x[j] * (1.0 / 6.0 * (f(x_hat[i]) + f(x_hat[i+1])) + 2.0 / 3.0 * f(x_hat[i] + delta_x[j] / 2.0))
    error_simpson[j] = numpy.abs(I_hat - I_true)/I_true

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_x, error_mid, 'ro', label="Midpoint")
axes.loglog(delta_x, error_trap, 'bo', label="Trapezoid")
axes.loglog(delta_x, error_simpson, 'go', label="Simpson's")
axes.loglog(delta_x, order_C(delta_x[0], error_trap[0], 2.0) * delta_x**2.0, 'b--', label="2nd Order")
axes.loglog(delta_x, order_C(delta_x[0], error_simpson[0], 4.0) * delta_x**4.0, 'g--', label="4th Order")
axes.legend(loc='best')
axes.set_xlabel('$\Delta x$', fontsize=16)
axes.set_ylabel('Relative Error', fontsize=16)
axes.grid()

plt.show()
```

### Recursive Improvement of Accuracy

Say we ran the trapezoidal rule with step size $2 \Delta x$, we then will have

$$\begin{aligned}
			\int^{x_2}_{x_0} f(x) dx  &= \frac{2 \Delta x}{2} (f_0 + f_2) =  h (f_0 + f_2) \Rightarrow \\
			\int^b_a f(x)dx &\approx I_{2\Delta x}[f] = \sum^{N/2-1}_{j=0} \Delta x (f_{2j} + f_{2j+2}) \\
			&= \Delta x (f_{0} + f_{2})  + \Delta x (f_{2} + f_{4})  + \cdots + \Delta x (f_{N-2} + f_{N}) \\
			&= \Delta x\left ( f_0 + f_N +  2 \sum^{N/2-1}_{j=1} f_{2j} \right )
  \end{aligned}
$$

Now compare the two rules for $\Delta x$ and $2 \Delta x$:

$$\begin{align}I_{\Delta x}[f] &= \frac{\Delta x}{2} \left (f_0 + f_N + 2 \sum^{N-1}_{j=1} f_j \right)\\ I_{2 \Delta x}[f] &= \Delta x \left ( f_0 + f_N +  2 \sum^{N/2-1}_{j=1} f_{2j} \right )\end{align}$$

$$I_{\Delta x}[f] = \frac{1}{2} I_{2\Delta x} + \Delta x(f_1 + f_3 + \cdots + f_{N-1})$$

Here we see we can actually reuse the work we did to calculate $Q_{2 \Delta x}[f]$ to refine the integral.

### Arbitrary Intervals (Affine Transforms)

A lot of quadrature rules are defined on specific intervals, e.g $x\in[0,1]$ or $x\in[-1,1]$.  However any two intervals can always be mapped onto each other through an  *affine transform* or *affine map* which is a linear  transformation.

$$
    x = \alpha\xi + \beta
$$
where $\alpha$ is a scaling and $\beta$ is a shift.

Given $\alpha$ and $\beta$,  the inverse map is simply

$$
    \xi = \frac{x - \beta}{\alpha}
$$

### Example:

Map $ x \in [a,b]\rightarrow\xi \in [-1,1]$

```python
fig = plt.figure(figsize=(5,5))
axes = fig.add_subplot(1,1,1)
axes.plot(x,func(x),linewidth=3)
axes.fill_between(x_interval,0.,func(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$a$',fontsize=16)
axes.text((x0+x1)/2.-.02,-.2,'$x$',fontsize=16)
axes.text(x1-.02,-.2,'$b$',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.text(x0-.06,-.4,'$-1$',fontsize=16)
axes.text((x0+x1)/2.-.02,-.4,'$\\xi$',fontsize=16)
axes.text(x1-.02,-.4,'$1$',fontsize=16)
axes.set_ylim((-0.5, 2.))
axes.grid()

```

$$x =  \frac{b - a}{2}\xi + \frac{a+b}{2}$$

$$\xi = \left( x - \frac{a + b}{2}\right) \frac{2}{b-a}$$

Given an Affine mapping,  it is easy to transform any integral from one coordinate system to another

$$
    I[f] = \int^b_a f(x) dx = \int^1_{-1} f(x(\xi)) \frac{dx}{d\xi} d\xi = \frac{b - a}{2} \int^1_{-1} f(x(\xi)) d\xi\\
$$

and the $N$ point quadrature rule for a single interval is
$$
    I_N[f]_i = \frac{\Delta x_i}{2} \sum_{n=1}^{N} w_n f(x(\xi_n))
$$

and the cumulative integral over $M$ panels
$$
    I_N[f] = \sum_{i=1}^M I_N[f]_i
$$

#### Example:  Newton-Cotes Rules

We can rewrite our previous quadrature rules so that they are given on the interval $\xi \in [-1, 1]$ instead of $x \in [x_i, x_{i+1}]$.  Recall that a general quadrature rule can be written as
$$
    \sum^N_{i=1} w_i f(\xi_i)
$$
where $w_i$ are the weights and $\xi_i$ are the points specified to evaluate the function at.

 For Newton-Cotes rules we know that the points $\xi_i$ are uniformly distributed on $[-1, 1]$ but we still need to define the weights.  For a 2-Point trapezoid rule we can do this to find that
$$
    \int^1_{-1} f(x) dx \approx f(-1) + f(1)
$$
so the quadrature points are at  $\xi_0 = [-1, 1]$ with weights $w = [1, 1]$

Note that if we map this using our affine transform we would get back the original trapezoid rule:
$$
    I_2[f] = \frac{dx}{d\xi}\sum_{n=1}^{2} w_n f(x(\xi_n))  = \frac{\Delta x}{2}(f(-1) + f(1))
$$

Similarly for Simpson's rule we have
$$
    \xi = [-1, 0, 1] \quad \text{and} \quad w = \left[\frac{1}{3}, \frac{4}{3}, \frac{1}{3} \right].
$$

```python
def ncintegration(f, a, b, N):
    """ Approximate \int_a^b f(x)dx using  Newton-Cotes quadrature rules with N<=3 quadrature points"""

    assert (N > 0)
    assert (N < 4)
    # Build up a nested list of integration points and weights for the interval [-1, 1]:
    xNC = []
    wNC = []
    # 1 point Newton-Cotes (Mid-point)
    xNC.append(numpy.array([0.]))
    wNC.append(numpy.array([2.]))
    # 2 point Newton-Cotes quadrature (Trapezoidal)
    xNC.append(numpy.array([ -1., 1. ]))
    wNC.append(numpy.array([ 1. , 1.]))
    # 3 point Newton-Cotes quadrature (Simpson's rule)
    xNC.append(numpy.array([-1., 0., 1.]))
    wNC.append(numpy.array([1., 4., 1.])/3.)

    # make affine map between x and xi
    xi_map = lambda xi: (b - a)/2. * xi + (a + b) / 2.

    I = (b - a)/2. * numpy.sum(wNC[N-1]*f(xi_map(xNC[N-1])))
    return I

```

```python
# Note that this calculates the cumulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: - numpy.cos(x)

a = numpy.pi/4.
b = 3*numpy.pi/4.
b = numpy.pi/2
I_true = I(b) - I(a)
print(I_true)
```

    0.7071067811865475

```python
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)
x_interval = numpy.linspace(a, b, 100)
fig = plt.figure(figsize=(5,5))
axes = fig.add_subplot(1,1,1)
axes.plot(x,f(x),linewidth=3)
axes.fill_between(x_interval,0.,f(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')

axes.grid()
```

```python
# Calculate Newton-Cotes Quadrature and relative errors for 1,2, and 3 point NC quadrature schemes
for N in range(1,4):
    I = ncintegration(f,a,b,N)
    err = numpy.abs(I - I_true)/numpy.abs(I_true)
    print('N = {}, I_N = {}, err = {}'.format(N, I, err))

```

    N = 1, I_N = 0.7256132880348577, err = 0.026172152977030898
    N = 2, I_N = 0.670379265333622, err = 0.051940551031480015
    N = 3, I_N = 0.7072019471344457, err = 0.00013458497419382203

## Optimal Quadrature Methods

Newton-Cotes formulas assume a fixed, regularly spaced set of quadrature points and calculate quadrature rules based on integrating the interpolating polynomial.

Can we do better?

Actually, if we allow both the points and weights to be chosen in an optimal way, we can generate optimal quadrature rules with $N$ points that can exactly integrate polynomials up to order $2N-1$

### Generalized Gaussian Quadrature

Given $g(x) \in P_N(x)$ with roots $\{x_i\}^N_{i=1}$ with the property
$$
    \int^1_{-1} w(x) x^i g(x) dx = 0 \quad \forall i < N,
$$

implies that  $g(x)$ is *orthogonal* to all monomials $x^i$ ($i <N$) with respect to the weight function $w(x)$.

Recall something similar:
$$
    <x, y> = \sum^N_{i=1} x_i y_i = ||x|| \cdot ||y|| \cos \theta.
$$
If $<x, y> = 0$ then the vectors $x$ and $y$ are orthogonal.

Given the above $g(x)$ there then exists a set of weights $\{w_i\}$ such that

$$\int^1_{-1} w(x) P_j(x) dx = \sum^N_{i=1} w_i P_j(x_i) \quad \forall j \leq 2 N - 1$$

In other words, given a polynomial basis function and weight and orthogonality to all polynomials of order $i < N$ we can exactly integrate polynomials of order $2 N - 1$.  For a proof and considerably more detail see https://en.wikipedia.org/wiki/Gaussian_quadrature

Choosing the correct weighting function and basis leads to a number of useful quadrature approaches:

#### Gauss-Legendre Quadrature

General Gauss-Legendre quadrature uses $w(x) = 1$ and $g(x) = \ell_N(x)$ which are the Legendre Polynomials, which can be shown to have weights

$$w_i = \frac{2}{(1-x_i^2)(\ell'_n(x_i))^2}$$

and $x_i$ is the $i$th root of $\ell_N$.

The first few Legendre Polynomials are
\begin{align}
    \ell_1(x) &=x \\
    \ell_2(x) &= \frac{1}{2}(3x^2 -1)\\
    \ell_3(x) &= \frac{1}{2}x(5x^2 - 3)\\
       \vdots
\end{align}

```python
from scipy.special import legendre
x = numpy.linspace(-1,1,100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
for N in range(5):
    Pn = legendre(N)
    axes.plot(x, Pn(x), label='N={}'.format(N))

axes.grid()
axes.set_title('Legendre Polynomials', fontsize=18)
axes.set_xlabel('x',fontsize=16)
axes.legend(loc='best')
plt.show()

```

Which lead to the quadrature points and weights...

$$N$$	 $$x_i$$	  $$w_i$$

$$1$$	            $$0$$ 	  $$2$$

$$2$$	            $$\pm \sqrt{\frac{1}{3}}$$ 	  $$1$$

$$3$$	  $$0$$ 	  $$8/9$$

                      $$\pm \sqrt{\frac{3}{5}}$$ 	  $$5/9$$

$$4$$	  $$\pm \sqrt{\frac{3}{7} - \frac{2}{7} \sqrt{\frac{6}{5}}}$$	  $$\frac{18 + \sqrt{30}}{36}$$

                      $$\pm \sqrt{\frac{3}{7} + \frac{2}{7} \sqrt{\frac{6}{5}}}$$	 $$\frac{18 - \sqrt{30}}{36}$$

$$5$$	  $$0$$ 	  $$\frac{128}{225}$$

                      $$\pm \frac{1}{3} \sqrt{5 - 2 \sqrt{\frac{10}{7}}}$$	  $$\frac{322 + 13\sqrt{70}}{900}$$

                      $$\pm \frac{1}{3} \sqrt{5 + 2 \sqrt{\frac{10}{7}}}$$	  $$\frac{322 - 13\sqrt{70}}{900}$$

##### Example 2:  2-Point Gauss-Legendre Quadrature

Let $N=2$ on $x \in [-1,1]$

$$I_2[f] = w_0 f(x_0) + w_1 f(x_1)$$

From above table we are told that $x_0 = -\sqrt{\frac{1}{3}}$, $x_1 = \sqrt{\frac{1}{3}}$ and $w_0 = w_1 = 1$

We can actually solve for the 2 weights and 2 locations using the method of *undetermined coefficients*

I.e.  Assume
$$
    I_2[f] = w_0 f(x_0) + w_1 f(x_1)
$$

and solve for $w_0, w_1, x_0, x_1$ that exactly integrate the first 4 monomials, $1, x, x^2, x^3$.

$$\begin{aligned}
    &\text{if}~f = 1:  &I[f] =  \int^{1}_{-1} 1 dx = 2 & & I_2[1] &= w_0 + w_1\\
    &\text{if}~f = x:  &I[f] =  \int^{1}_{-1} x dx = 0 & & I_2[x] &= w_0 x_0 + w_1 x_1\\
    &\text{if}~f = x^2:  &I[f] =  \int^{1}_{-1} x^2 dx = \frac{2}{3} & & I_2[x^2] &= w_0 x_0^2 + w_1 x_1^2\\
    &\text{if}~f = x^3:  &I[f] =  \int^{1}_{-1} x^3 dx = 0 & & I_2[x^3] &= w_0 x_0^3 + w_1 x_1^3\\
\end{aligned}$$

$$\begin{aligned}
    &w_0 + w_1 = 2\\
    &w_0 x_0 + w_1 x_1 = 0\\
    &w_0 x_0^2 + w_1 x_1^2 = \frac{2}{3}\\
    &w_0 x_0^3 + w_1 x_1^3 = 0\\
\end{aligned}$$

Note that we need to solve for 4 unknowns $x_0$, $x_1$, $w_0$, and $w_1$.  Solving these non-linear equations leads to

$$x_0 = -\sqrt{\frac{1}{3}}, x_1 = \sqrt{\frac{1}{3}}  \quad \text{and} \quad w_0 = w_1 = 1 $$

##### Example: Gauss-Legendre Quadrature single interval

let $f(x) = \cos(x)$:

$$
    I[f] = \int_0^{\pi/2} f(x) dx = \left.\sin(x)\right|_0^{\pi/2} = 1.
$$

so the affine map from $x\in[0,\pi/2]\rightarrow\xi\in[-1,1]$ is

\begin{align}
x &= \frac{\pi}{4}\xi + \frac{\pi}{4} \\
&= \frac{\pi}{4}\left(\xi + 1\right)
\end{align}

```python
f = lambda x: numpy.cos(x)
I = 1.

x = numpy.linspace(-1.,2, 100)
x_interval = numpy.linspace(0., numpy.pi/2, 100)
fig = plt.figure(figsize=(5,5))
axes = fig.add_subplot(1,1,1)
axes.plot(x,f(x),linewidth=3)
axes.fill_between(x_interval,0.,f(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')

axes.grid()
```

```python
def glintegration(f, a, b, N):
    """ Approximate \int_a^b f(x)dx using  Gauss-Legendre quadrature rules with N<=4 quadrature points"""

    # Build up a nested list of gauss points and weights:
    xGl = []
    wGl = []
    # 1 point Gauss-legendre quadrature
    xGl.append(numpy.array([0.]))
    wGl.append(numpy.array([2.]))
    # 2 point Gauss-legendre quadrature
    xGl.append(numpy.array([ -1./numpy.sqrt(3.), 1./numpy.sqrt(3)]))
    wGl.append(numpy.array([ 1. , 1.]))
    # 3 point Gauss-legendre quadrature
    xGl.append(numpy.array([-numpy.sqrt(3. /5.), 0., numpy.sqrt(3. / 5.)]))
    wGl.append(numpy.array([5., 8., 5.])/9.)
    # 4 point Gauss-Legendre quadrature
    xGl.append(numpy.array([-numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
          -numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0))]))
    wGl.append(numpy.array([(18.0 + numpy.sqrt(30.0)) / 36.0, (18.0 + numpy.sqrt(30.0)) / 36.0,
         (18.0 - numpy.sqrt(30.0)) / 36.0, (18.0 - numpy.sqrt(30.0)) / 36.0]))

    # make affine map between x and xi
    xi_map = lambda xi: (b - a)/2. * xi + (a + b) / 2.

    I = (b - a)/2. * numpy.sum(wGl[N-1]*f(xi_map(xGl[N-1])))
    return I

```

```python
a = 0.
b = numpy.pi/2
I_true = 1.
err = numpy.zeros(4)
for N in range(1,5):
    I = glintegration(f, a, b, N)
    err[N-1] = numpy.abs(I_true - I)
    print('N = {}, I = {}, err = {}'.format(N,I, err[N-1]))
```

    N = 1, I = 1.1107207345395915, err = 0.11072073453959153
    N = 2, I = 0.9984726134041148, err = 0.0015273865958852495
    N = 3, I = 1.0000081215554983, err = 8.121555498341593e-06
    N = 4, I = 0.9999999771971154, err = 2.2802884580563898e-08

```python
errNC = numpy.zeros(4)
for N in range(1,4):
    I = ncintegration(f, a, b, N)
    errNC[N-1] = numpy.abs(I_true - I)
    print('N = {}, I = {}, err = {}'.format(N,I, errNC[N-1]))

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(range(1,5),errNC,'bs', markersize=10, label='Newton-Cotes')
axes.semilogy(range(1,5),err,'ro', markersize=10, label='Gauss-Legendre')
axes.set_ylabel('Error')
axes.set_xlabel('quadrature points (N)')
axes.legend(loc='best')
axes.grid()
plt.show()
```

```python
f = lambda x: - 4*x**3 - 3.*x**2 + 2.*x +300.
int_f = lambda x:  - x**4 - x**3. + x**2 + x*300.
#f = lambda x: x**5 - 4*x**3 - 3.*x**2 + 2.*x +300.
#int_f = lambda x: x**6/6. - x**4 - x**3. + x**2 + x*300.

a = 1.
b = 4.
I_true = int_f(b) - int_f(a)
print('I_true = {}'.format(I_true))
for N in range(1,5):
    I = glintegration(f, a, b, N)
    print('N = {}, I = {}'.format(N,I))
```

    I_true = 597.0
    N = 1, I = 671.25
    N = 2, I = 596.9999999999999
    N = 3, I = 597.0
    N = 4, I = 596.9999999999999

```python
x = numpy.linspace(a - 1, b + 1, 100)
x_interval = numpy.linspace(a, b, 100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(x,f(x),linewidth=3)
axes.fill_between(x_interval,0.,f(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')

axes.grid()
plt.show()
```

```python
# Compute Gauss-Legendre based quadrature and affine transforms
f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 10
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
delta_x = x_hat[1] - x_hat[0]

xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
xi_0 = -numpy.sqrt(1.0 / 3.0)
xi_1 =  numpy.sqrt(1.0 / 3.0)
I_hat = numpy.zeros(x_hat.shape)
I_hat[0] = (f(xi_map(x_hat[0], x_hat[1], xi_0)) + f(xi_map(x_hat[0], x_hat[1], xi_1))) * delta_x / 2.0
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + (f(xi_map(x_hat[i], x_hat[i+1], xi_0)) + f(xi_map(x_hat[i], x_hat[i+1], xi_1))) * delta_x / 2.0

error = numpy.abs(I_hat - I(x_hat + delta_x))
error[:-1]
```

    array([6.97744637e-06, 2.52446381e-05, 4.78241289e-05, 6.60913206e-05,
           7.30687670e-05, 6.60913206e-05, 4.78241289e-05, 2.52446381e-05,
           6.97744637e-06, 1.66533454e-16])

```python

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_hat + delta_x, I_hat, 'ko')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))
axes.grid()
plt.show()
```

##### Example 3:

If $f(x) = e^x$ look at the relative accuracy of midpoint, trapezoidal, Simpson's and 2-point Gauss-Legendre quadrature for a single interval $x \in [-1,1]$.

$$\begin{aligned}
    \text{Exact:} &I[f] = \int^1_{-1} e^x = \left . e^x \right |^1_{-1} = e - \frac{1}{e} \approx 2.350402387 \\
    \text{Midpoint:} &I_2[f] = 2 e^0 = 2 \\
    \text{Trapezoid:} &I_2[f] = \frac{2}{2} (e^{-1} + e^1) = e + \frac{1}{e} = 3.08616127 \\
    \text{Simpson's:} &I_2[f] = \frac{2}{6} e^{-1} + \frac{4}{3} e^0 + \frac{2}{6} e^1 = \frac{4}{3} + \frac{1}{3} (e^{-1} + e^1) \approx 2.362053757 \\
    \text{Gauss-Legendre:} &I_2[f] = e^{-\sqrt{\frac{1}{3}}} + e^{\sqrt{\frac{1}{3}}} \approx 2.342696088
\end{aligned}$$

```python
# Compute the error as a function of delta_x for each method
f = lambda x: numpy.sin(numpy.pi * x)
I = 2.0 / numpy.pi

# num_partitions = range(50, 1000, 50)
num_partitions = range(5, 50, 5)
delta_x = numpy.empty(len(num_partitions))
error_trap = numpy.empty(len(num_partitions))
error_simpson = numpy.empty(len(num_partitions))
error_2 = numpy.empty(len(num_partitions))
error_3 = numpy.empty(len(num_partitions))
error_4 = numpy.empty(len(num_partitions))

for (j, N) in enumerate(num_partitions):
    x_hat = numpy.linspace(0.0, 1.0, N)
    delta_x[j] = x_hat[1] - x_hat[0]

    # Compute trapezoid
    I_hat = 0.0
    for i in range(0, N - 1):
        I_hat += (f(x_hat[i + 1]) + f(x_hat[i])) * delta_x[j] / 2.0
    error_trap[j] = numpy.abs(I_hat - I)

    # Compute simpson's
    I_hat = 0.0
    for i in range(0, N - 1):
        I_hat += delta_x[j] * (1.0 / 6.0 * (f(x_hat[i]) + f(x_hat[i+1])) + 2.0 / 3.0 * f(x_hat[i] + delta_x[j] / 2.0))
    error_simpson[j] = numpy.abs(I_hat - I)

    # Compute Gauss-Legendre 2-point
    xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
    xi = [-numpy.sqrt(1.0 / 3.0), numpy.sqrt(1.0 / 3.0)]
    w = [1.0, 1.0]
    I_hat = 0.0
    for i in range(0, N - 1):
        for k in range(len(xi)):
            I_hat += f(xi_map(x_hat[i], x_hat[i+1], xi[k])) * w[k]
    I_hat *= delta_x[j] / 2.0
    error_2[j] = numpy.abs(I_hat - I)

    # Compute Gauss-Legendre 3-point
    xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
    xi = [-numpy.sqrt(3.0 / 5.0), 0.0, numpy.sqrt(3.0 / 5.0)]
    w = [5.0 / 9.0, 8.0 / 9.0, 5.0 / 9.0]
    I_hat = 0.0
    for i in range(0, N - 1):
        for k in range(len(xi)):
            I_hat += f(xi_map(x_hat[i], x_hat[i+1], xi[k])) * w[k]
    I_hat *= delta_x[j] / 2.0
    error_3[j] = numpy.abs(I_hat - I)

    # Compute Gauss-Legendre 4-point
    xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
    xi = [-numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
          -numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0))]
    w = [(18.0 + numpy.sqrt(30.0)) / 36.0, (18.0 + numpy.sqrt(30.0)) / 36.0,
         (18.0 - numpy.sqrt(30.0)) / 36.0, (18.0 - numpy.sqrt(30.0)) / 36.0]
    I_hat = 0.0
    for i in range(0, N - 1):
        for k in range(len(xi)):
            I_hat += f(xi_map(x_hat[i], x_hat[i+1], xi[k])) * w[k]
    I_hat *= delta_x[j] / 2.0
    error_4[j] = numpy.abs(I_hat - I)

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)

# axes.plot(delta_x, error)
axes.loglog(delta_x, error_trap, 'o', label="Trapezoid")
axes.loglog(delta_x, error_simpson, 'o', label="Simpson's")
axes.loglog(delta_x, error_2, 'o', label="G-L 2-point")
axes.loglog(delta_x, error_3, 'o', label="G-L 3-point")
axes.loglog(delta_x, error_4, 'o', label="G-L 4-point")

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_x, order_C(delta_x[0], error_trap[0], 2.0) * delta_x**2.0, 'r--', label="2nd Order")
axes.loglog(delta_x, order_C(delta_x[0], error_simpson[0], 4.0) * delta_x**4.0, 'g--', label="4th Order")
axes.loglog(delta_x, order_C(delta_x[1], error_3[1], 5) * delta_x**5, 'b--', label="5th Order")
axes.loglog(delta_x, order_C(delta_x[1], error_4[1], 7.0) * delta_x**7.0, 'g--', label="7th Order")

axes.legend(loc='best',fontsize=14)
axes.set_xlabel('$\Delta x$', fontsize=16)
axes.set_xlim((5e-3, delta_x[0]))
axes.grid()

plt.show()
```

#### Other Quadrature Families

 - Clenshaw-Curtis (Gauss-Chebyshev): If $w(x) = \frac{1}{\sqrt{1 - x^2}}$ and $g(x)$ are Chebyshev polynomials then we know the roots of the polynomials to be $x_i = \cos\left(\frac{2i-1}{2N} \pi \right)$ (the Chebyshev nodes) and we can derive that $w_i = \frac{\pi}{N}$.
 - Gauss-Hermite:  If $w(x) = e^{-x^2}$ and $g(x)$ are Hermite polynomials $H_i(x)$ then
   $$w_i = \frac{2^{N-1} N! \sqrt{\pi}}{N^2 (H_{N-1}(x_i))^2}$$

## SciPy Integration Routines

SciPy has a number of integration routines that we have derived here including general purpose integrators that can control error.

```python
import scipy.integrate as integrate
# integrate?
```