# Basis for grayscale images

## Introduction

Consider the set of real-valued matrices of size $M\times N$; we can turn this into a vector space by defining addition and scalar multiplication in the usual way:

\begin{align}
\mathbf{A} + \mathbf{B} &=
    \left[
        \begin{array}{ccc}
            a_{0,0} & \dots & a_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    +
    \left[
        \begin{array}{ccc}
            b_{0,0} & \dots & b_{0,N-1} \\
            \vdots & & \vdots \\
            b_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    \\
    &=
    \left[
        \begin{array}{ccc}
            a_{0,0}+b_{0,0} & \dots & a_{0,N-1}+b_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0}+b_{M-1,0} & \dots & a_{M-1,N-1}+b_{M-1,N-1}
        \end{array}
    \right]
    \\ \\ \\
\beta\mathbf{A} &=
    \left[
        \begin{array}{ccc}
            \beta a_{0,0} & \dots & \beta a_{0,N-1} \\
            \vdots & & \vdots \\
            \beta a_{M-1,0} & \dots & \beta a_{M-1,N-1}
        \end{array}
    \right]
\end{align}

As a matter of fact, the space of real-valued $M\times N$ matrices is completely equivalent to $\mathbb{R}^{MN}$ and we can always "unroll" a matrix into a vector. Assume we proceed column by column; then the matrix becomes

$$
    \mathbf{a} = \mathbf{A}[:] = [
        \begin{array}{ccccccc}
            a_{0,0} & \dots & a_{M-1,0} & a_{0,1} & \dots & a_{M-1,1} & \ldots & a_{0, N-1} & \dots & a_{M-1,N-1}
        \end{array}]^T
$$

Although the matrix and vector forms represent exactly the same data, the matrix form allows us to display the data in the form of an image. Assume each value in the matrix is a grayscale intensity, where zero is black and 255 is white; for example we can create a checkerboard pattern of any size with the following function:

```python
# usual python bookkeeping...
%pylab inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import IPython
from IPython.display import Image
import math
from __future__ import print_function

# ensure all images will be grayscale
gray();
```

    Populating the interactive namespace from numpy and matplotlib

    <Figure size 432x288 with 0 Axes>

#### (?1) `gray()  # ensure all images will be grayscale`? What?

#### (R1)

```python
gray.__module__
```

    'matplotlib.pyplot'

```python
help(gray)
```

    Help on function gray in module matplotlib.pyplot:

    gray()
        Set the colormap to "gray".

        This changes the default colormap as well as the colormap of the current
        image if there is one. See ``help(colormaps)`` for more information.

#### (!1) Worth digging deep: `colormaps`

```python
help(colormaps)
```

    Help on function colormaps in module matplotlib.pyplot:

    colormaps()
        Matplotlib provides a number of colormaps, and others can be added using
        :func:`~matplotlib.cm.register_cmap`.  This function documents the built-in
        colormaps, and will also return a list of all registered colormaps if
        called.

        You can set the colormap for an image, pcolor, scatter, etc,
        using a keyword argument::

          imshow(X, cmap=cm.hot)

        or using the :func:`set_cmap` function::

          imshow(X)
          pyplot.set_cmap('hot')
          pyplot.set_cmap('jet')

        In interactive mode, :func:`set_cmap` will update the colormap post-hoc,
        allowing you to see which one works best for your data.

        All built-in colormaps can be reversed by appending ``_r``: For instance,
        ``gray_r`` is the reverse of ``gray``.

        There are several common color schemes used in visualization:

        Sequential schemes
          for unipolar data that progresses from low to high
        Diverging schemes
          for bipolar data that emphasizes positive or negative deviations from a
          central value
        Cyclic schemes
          for plotting values that wrap around at the endpoints, such as phase
          angle, wind direction, or time of day
        Qualitative schemes
          for nominal data that has no inherent ordering, where color is used
          only to distinguish categories

        Matplotlib ships with 4 perceptually uniform color maps which are
        the recommended color maps for sequential data:

          =========   ===================================================
          Colormap    Description
          =========   ===================================================
          inferno     perceptually uniform shades of black-red-yellow
          magma       perceptually uniform shades of black-red-white
          plasma      perceptually uniform shades of blue-red-yellow
          viridis     perceptually uniform shades of blue-green-yellow
          =========   ===================================================

        The following colormaps are based on the `ColorBrewer
        <https://colorbrewer2.org>`_ color specifications and designs developed by
        Cynthia Brewer:

        ColorBrewer Diverging (luminance is highest at the midpoint, and
        decreases towards differently-colored endpoints):

          ========  ===================================
          Colormap  Description
          ========  ===================================
          BrBG      brown, white, blue-green
          PiYG      pink, white, yellow-green
          PRGn      purple, white, green
          PuOr      orange, white, purple
          RdBu      red, white, blue
          RdGy      red, white, gray
          RdYlBu    red, yellow, blue
          RdYlGn    red, yellow, green
          Spectral  red, orange, yellow, green, blue
          ========  ===================================

        ColorBrewer Sequential (luminance decreases monotonically):

          ========  ====================================
          Colormap  Description
          ========  ====================================
          Blues     white to dark blue
          BuGn      white, light blue, dark green
          BuPu      white, light blue, dark purple
          GnBu      white, light green, dark blue
          Greens    white to dark green
          Greys     white to black (not linear)
          Oranges   white, orange, dark brown
          OrRd      white, orange, dark red
          PuBu      white, light purple, dark blue
          PuBuGn    white, light purple, dark green
          PuRd      white, light purple, dark red
          Purples   white to dark purple
          RdPu      white, pink, dark purple
          Reds      white to dark red
          YlGn      light yellow, dark green
          YlGnBu    light yellow, light green, dark blue
          YlOrBr    light yellow, orange, dark brown
          YlOrRd    light yellow, orange, dark red
          ========  ====================================

        ColorBrewer Qualitative:

        (For plotting nominal data, `.ListedColormap` is used,
        not `.LinearSegmentedColormap`.  Different sets of colors are
        recommended for different numbers of categories.)

        * Accent
        * Dark2
        * Paired
        * Pastel1
        * Pastel2
        * Set1
        * Set2
        * Set3

        A set of colormaps derived from those of the same name provided
        with Matlab are also included:

          =========   =======================================================
          Colormap    Description
          =========   =======================================================
          autumn      sequential linearly-increasing shades of red-orange-yellow
          bone        sequential increasing black-white color map with
                      a tinge of blue, to emulate X-ray film
          cool        linearly-decreasing shades of cyan-magenta
          copper      sequential increasing shades of black-copper
          flag        repetitive red-white-blue-black pattern (not cyclic at
                      endpoints)
          gray        sequential linearly-increasing black-to-white
                      grayscale
          hot         sequential black-red-yellow-white, to emulate blackbody
                      radiation from an object at increasing temperatures
          jet         a spectral map with dark endpoints, blue-cyan-yellow-red;
                      based on a fluid-jet simulation by NCSA [#]_
          pink        sequential increasing pastel black-pink-white, meant
                      for sepia tone colorization of photographs
          prism       repetitive red-yellow-green-blue-purple-...-green pattern
                      (not cyclic at endpoints)
          spring      linearly-increasing shades of magenta-yellow
          summer      sequential linearly-increasing shades of green-yellow
          winter      linearly-increasing shades of blue-green
          =========   =======================================================

        A set of palettes from the `Yorick scientific visualisation
        package <https://dhmunro.github.io/yorick-doc/>`_, an evolution of
        the GIST package, both by David H. Munro are included:

          ============  =======================================================
          Colormap      Description
          ============  =======================================================
          gist_earth    mapmaker's colors from dark blue deep ocean to green
                        lowlands to brown highlands to white mountains
          gist_heat     sequential increasing black-red-orange-white, to emulate
                        blackbody radiation from an iron bar as it grows hotter
          gist_ncar     pseudo-spectral black-blue-green-yellow-red-purple-white
                        colormap from National Center for Atmospheric
                        Research [#]_
          gist_rainbow  runs through the colors in spectral order from red to
                        violet at full saturation (like *hsv* but not cyclic)
          gist_stern    "Stern special" color table from Interactive Data
                        Language software
          ============  =======================================================

        A set of cyclic color maps:

          ================  =================================================
          Colormap          Description
          ================  =================================================
          hsv               red-yellow-green-cyan-blue-magenta-red, formed by
                            changing the hue component in the HSV color space
          twilight          perceptually uniform shades of
                            white-blue-black-red-white
          twilight_shifted  perceptually uniform shades of
                            black-blue-white-red-black
          ================  =================================================

        Other miscellaneous schemes:

          ============= =======================================================
          Colormap      Description
          ============= =======================================================
          afmhot        sequential black-orange-yellow-white blackbody
                        spectrum, commonly used in atomic force microscopy
          brg           blue-red-green
          bwr           diverging blue-white-red
          coolwarm      diverging blue-gray-red, meant to avoid issues with 3D
                        shading, color blindness, and ordering of colors [#]_
          CMRmap        "Default colormaps on color images often reproduce to
                        confusing grayscale images. The proposed colormap
                        maintains an aesthetically pleasing color image that
                        automatically reproduces to a monotonic grayscale with
                        discrete, quantifiable saturation levels." [#]_
          cubehelix     Unlike most other color schemes cubehelix was designed
                        by D.A. Green to be monotonically increasing in terms
                        of perceived brightness. Also, when printed on a black
                        and white postscript printer, the scheme results in a
                        greyscale with monotonically increasing brightness.
                        This color scheme is named cubehelix because the (r, g, b)
                        values produced can be visualised as a squashed helix
                        around the diagonal in the (r, g, b) color cube.
          gnuplot       gnuplot's traditional pm3d scheme
                        (black-blue-red-yellow)
          gnuplot2      sequential color printable as gray
                        (black-blue-violet-yellow-white)
          ocean         green-blue-white
          rainbow       spectral purple-blue-green-yellow-orange-red colormap
                        with diverging luminance
          seismic       diverging blue-white-red
          nipy_spectral black-purple-blue-green-yellow-red-white spectrum,
                        originally from the Neuroimaging in Python project
          terrain       mapmaker's colors, blue-green-yellow-brown-white,
                        originally from IGOR Pro
          turbo         Spectral map (purple-blue-green-yellow-orange-red) with
                        a bright center and darker endpoints. A smoother
                        alternative to jet.
          ============= =======================================================

        The following colormaps are redundant and may be removed in future
        versions.  It's recommended to use the names in the descriptions
        instead, which produce identical output:

          =========  =======================================================
          Colormap   Description
          =========  =======================================================
          gist_gray  identical to *gray*
          gist_yarg  identical to *gray_r*
          binary     identical to *gray_r*
          =========  =======================================================

        .. rubric:: Footnotes

        .. [#] Rainbow colormaps, ``jet`` in particular, are considered a poor
          choice for scientific visualization by many researchers: `Rainbow Color
          Map (Still) Considered Harmful
          <https://ieeexplore.ieee.org/document/4118486/?arnumber=4118486>`_

        .. [#] Resembles "BkBlAqGrYeOrReViWh200" from NCAR Command
          Language. See `Color Table Gallery
          <https://www.ncl.ucar.edu/Document/Graphics/color_table_gallery.shtml>`_

        .. [#] See `Diverging Color Maps for Scientific Visualization
          <http://www.kennethmoreland.com/color-maps/>`_ by Kenneth Moreland.

        .. [#] See `A Color Map for Effective Black-and-White Rendering of
          Color-Scale Images
          <https://www.mathworks.com/matlabcentral/fileexchange/2662-cmrmap-m>`_
          by Carey Rappaport

```python
# let's create a checkerboard pattern
SIZE = 4
img = np.zeros((SIZE, SIZE))
for n in range(0, SIZE):
    for m in range(0, SIZE):
        if (n & 0x1) ^ (m & 0x1):
            ## recall that ^ is the exclusive or
            #img[n, m] = 255
            img[m, n] = 255

# now display the matrix as an image
plt.matshow(img);
```

```python
SIZE = 4
img = np.zeros((SIZE, SIZE))
img.dtype  # float64, better uint8 for images
```

    dtype('float64')

```python
np.uint8
```

    numpy.uint8

`m & 0x1` and `n & 0x1` tests the parity: if `m` is odd, then `m & 0x1` outputs `1` (or `True`); if `m` even,
output `0`. Using XOR after that guarantees that only when exactly one of `m` and `n` is odd will the case
be colored white. This implies that every time one crosses from one case to any of its neighboring cases,
the color changes

Note how the axes of `plt.matshow()` are arranged diff from that of `plt.imshow()`
help(plt.matshow)

```python
plt.imshow(img);
```

Given the equivalence between the space of $M\times N$ matrices and $\mathbb{R}^{MN}$ we can easily define the inner product between two matrices in the usual way:

$$
\langle \mathbf{A}, \mathbf{B} \rangle = \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} a_{m,n} b_{m, n}
$$

(where we have neglected the conjugation since we'll only deal with real-valued matrices); in other words, we can take the inner product between two matrices as the standard inner product of their unrolled versions. The inner product allows us to define orthogonality between images and this is rather useful since we're going to explore a couple of bases for this space.

## Actual images

Conveniently, using IPython, we can read images from disk in any given format and convert them to numpy arrays; let's load and display for instance a JPEG image:
img = np.array(plt.imread('cameraman.jpg'), dtype=int)
plt.matshow(img);---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-37d88f4789f6> in <module>()
----> 1 img = np.array(plt.imread('cameraman.jpg'), dtype=int)
      2 plt.matshow(img);

/home/phunc20/.virtualenvs/dsp-py2.7/lib/python2.7/site-packages/matplotlib/pyplot.pyc in imread(*args, **kwargs)
   2371 @docstring.copy_dedent(_imread)
   2372 def imread(*args, **kwargs):
-> 2373     return _imread(*args, **kwargs)
   2374
   2375

/home/phunc20/.virtualenvs/dsp-py2.7/lib/python2.7/site-packages/matplotlib/image.pyc in imread(fname, format)
   1351             raise ValueError('Only know how to handle extensions: %s; '
   1352                              'with Pillow installed matplotlib can handle '
-> 1353                              'more images' % list(handlers))
   1354         return im
   1355

ValueError: Only know how to handle extensions: [u'png']; with Pillow installed matplotlib can handle more images!pip install pillow

```python
img = np.array(plt.imread('cameraman.jpg'), dtype=int)
plt.matshow(img);
```

The image is a $64\times 64$ low-resolution version of the famous "cameraman" test picture. Out of curiosity, we can look at the first column of this image, which is is a $64×1$ vector:

```python
img[:,0]
```

    array([156, 157, 157, 152, 154, 155, 151, 157, 152, 155, 158, 159, 159,
           160, 160, 161, 155, 160, 161, 161, 164, 162, 160, 162, 158, 160,
           158, 157, 160, 160, 159, 158, 163, 162, 162, 157, 160, 114, 114,
           103,  88,  62, 109,  82, 108, 128, 138, 140, 136, 128, 122, 137,
           147, 114, 114, 144, 112, 115, 117, 131, 112, 141,  99,  97])

The values are integers between zero and 255, meaning that each pixel is encoded over 8 bits (or 256 gray levels).

## The canonical basis

The canonical basis for any matrix space $\mathbb{R}^{M\times N}$ is the set of "delta" matrices where only one element equals to one while all the others are 0. Let's call them $\mathbf{E}_n$ with $0 \leq n < MN$. Here is a function to create the canonical basis vector given its index:

```python
def canonical(n, M=5, N=10):
    e = np.zeros((M, N))
    #e[(n % M), int(n / M)] = 1
    e[n % M, n // M] = 1
    return e
```

```python
!python --version
```

    Python 3.8.2

```python
10 / 3, 10 // 3, int(10/3)
```

    (3.3333333333333335, 3, 3)

Here are some basis vectors: look for the position of white pixel, which differentiates them and note that we enumerate pixels column-wise:

```python
plt.matshow(canonical(0));
plt.matshow(canonical(1));
plt.matshow(canonical(49));
```

Note how diff `matshow()` is from `imshow()`:
> In a jupyter cell, successive `matshow()`'s can draw any number of images, while `imshow()` will only do it for the last one

```python
plt.imshow(canonical(0));
plt.imshow(canonical(1));
plt.imshow(canonical(49));
```

##### Stopped here (2020/11/19 12h25)

## Transmitting images

Suppose we want to transmit the "cameraman" image over a communication channel. The intuitive way to do so is to send the  pixel values one by one, which corresponds to sending the coefficients of the decomposition of the image over the canonical basis. So far, nothing complicated: to send the cameraman image, for instance, we will send $64\times 64 = 4096$ coefficients in a row.

Now suppose that a communication failure takes place after the first half of the pixels have been sent. The received data will allow us to display an approximation of the original image only. If we replace the missing data with zeros, here is what we would see, which is not very pretty:

```python
# unrolling of the image for transmission (we go column by column, hence "F")
tx_img = np.ravel(img, "F")

# oops, we lose half the data
tx_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.reshape(tx_img, (64, 64), "F")
plt.matshow(rx_img);
```
help(np.ravel)
Can we come up with a trasmission scheme that is more robust in the face of channel loss? Interestingly, the answer is yes, and it involves a different, more versatile basis for the space of images. What we will do is the following:

* describe the Haar basis, a new basis for the image space
* project the image in the new basis
* transmit the projection coefficients
* rebuild the image using the basis vectors

We know a few things: if we choose an orthonormal basis, the analysis and synthesis formulas will be super easy (a simple inner product and a scalar multiplication respectively). The trick is to find a basis that will be robust to the loss of some coefficients.

One such basis is the **Haar basis**. We cannot go into too many details in this notebook but, for the curious, a good starting point is [here](https://chengtsolin.wordpress.com/2015/04/15/real-time-2d-discrete-wavelet-transform-using-opengl-compute-shader/). (An even better starting point is [these two papers](http://grail.cs.washington.edu/projects/wavelets/article/).) Mathematical formulas aside, the Haar basis works by encoding the information in a *hierarchical* way: the first basis vectors encode the broad information and the higher coefficients encode the detail. Let's have a look.

First of all, to keep things simple, we will remain in the space of square matrices whose size is a power of two. The code to generate the Haar basis matrices is the following: first we generate a 1D Haar vector and then we obtain the basis matrices by taking the outer product of all possible 1D vectors (don't worry if it's not clear, the results are what's important):

```python
def haar1D(n, SIZE):
    # check power of two
    if math.floor(math.log(SIZE) / math.log(2)) != math.log(SIZE) / math.log(2):
        print("Haar defined only for lengths that are a power of two")
        return None
    if n >= SIZE or n < 0:
        print("invalid Haar index")
        return None

    # zero basis vector
    if n == 0:
        return np.ones(SIZE)

    # express n >= 1 as 2^p + q with p as large as possible;
    # then k = SIZE/2^p is the length of the support
    # and s = qk is the shift
    p = math.floor(math.log(n) / math.log(2))
    pp = int(pow(2, p))
    k = SIZE / pp
    s = (n - pp) * k

    h = np.zeros(SIZE)
    h[int(s):int(s+k/2)] = 1
    h[int(s+k/2):int(s+k)] = -1
    # these are not normalized
    return h

def haar2D(n, SIZE=8):
    # get horizontal and vertical indices
    hr = haar1D(n % SIZE, SIZE)
    hv = haar1D(int(n / SIZE), SIZE)
    # 2D Haar basis matrix is separable, so we can
    # just take the column-row product
    H = np.outer(hr, hv)
    # np.outer() is just column vector times row vector,
    # the 1st arg being the col vec, the 2nd the row vec.
    H = H / math.sqrt(np.sum(H * H))
    # the previous line just divides H by its Frobenius norm
    # so that the returned value of haar2D() has norm 1.
    return H
```
help(np.outer)
First of all, let's look at a few basis matrices; note that the matrices have **both positive and negative values**, so that the value of **zero** will be represented as **gray**:

```python
plt.matshow?
```

```python
plt.matshow(haar2D(0));
plt.matshow(haar2D(1));
plt.matshow(haar2D(10));
plt.matshow(haar2D(63));
```

```python
np.unique(haar2D(0))
```

    array([0.125])

```python
np.unique(haar2D(1))
```

    array([-0.125,  0.125])

```python
np.unique(haar2D(63))
```

    array([-0.5,  0. ,  0.5])

```python
white = np.unique(haar2D(0))[0]
black = -white
plt.matshow(haar2D(0), vmax=white, vmin=black);
```

We can notice two key properties

* each basis matrix has positive and negative values in some symmetric pattern: this means that the basis matrix will implicitly compute the difference between image areas
* low-index basis matrices take differences between large areas, while high-index ones take differences in smaller **localized** areas of the image

We can immediately verify that the Haar matrices are orthogonal:

```python
# let's use an 8x8 space; there will be 64 basis vectors
# compute all possible inner product and only print the nonzero results
for m in range(0,64):
    for n in range(0,64):
        r = np.sum(haar2D(m, 8) * haar2D(n, 8))
        if r != 0:
            print("[%dx%d -> %f] " % (m, n, r), end="")
```

    [0x0 -> 1.000000] [1x1 -> 1.000000] [2x2 -> 1.000000] [3x3 -> 1.000000] [4x4 -> 1.000000] [5x5 -> 1.000000] [6x6 -> 1.000000] [7x7 -> 1.000000] [8x8 -> 1.000000] [9x9 -> 1.000000] [10x10 -> 1.000000] [11x11 -> 1.000000] [12x12 -> 1.000000] [13x13 -> 1.000000] [14x14 -> 1.000000] [15x15 -> 1.000000] [16x16 -> 1.000000] [16x17 -> -0.000000] [17x16 -> -0.000000] [17x17 -> 1.000000] [18x18 -> 1.000000] [19x19 -> 1.000000] [20x20 -> 1.000000] [21x21 -> 1.000000] [22x22 -> 1.000000] [23x23 -> 1.000000] [24x24 -> 1.000000] [24x25 -> -0.000000] [25x24 -> -0.000000] [25x25 -> 1.000000] [26x26 -> 1.000000] [27x27 -> 1.000000] [28x28 -> 1.000000] [29x29 -> 1.000000] [30x30 -> 1.000000] [31x31 -> 1.000000] [32x32 -> 1.000000] [33x33 -> 1.000000] [34x34 -> 1.000000] [35x35 -> 1.000000] [36x36 -> 1.000000] [37x37 -> 1.000000] [38x38 -> 1.000000] [39x39 -> 1.000000] [40x40 -> 1.000000] [41x41 -> 1.000000] [42x42 -> 1.000000] [43x43 -> 1.000000] [44x44 -> 1.000000] [45x45 -> 1.000000] [46x46 -> 1.000000] [47x47 -> 1.000000] [48x48 -> 1.000000] [49x49 -> 1.000000] [50x50 -> 1.000000] [51x51 -> 1.000000] [52x52 -> 1.000000] [53x53 -> 1.000000] [54x54 -> 1.000000] [55x55 -> 1.000000] [56x56 -> 1.000000] [57x57 -> 1.000000] [58x58 -> 1.000000] [59x59 -> 1.000000] [60x60 -> 1.000000] [61x61 -> 1.000000] [62x62 -> 1.000000] [63x63 -> 1.000000]

OK! Everything's fine. Now let's transmit the "cameraman" image: first, let's verify that it works

```python
# project the image onto the Haar basis, obtaining a vector of 4096 coefficients
# this is simply the analysis formula for the vector space with an orthogonal basis
tx_img = np.zeros(64*64)
for k in range(0, (64*64)):
    tx_img[k] = np.sum(img * haar2D(k, 64))

# now rebuild the image with the synthesis formula; since the basis is orthonormal
# we just need to scale the basis matrices by the projection coefficients
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += tx_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

```python
np.linalg.norm(img - rx_img, inf)
```

    2.112088282046898e-12

help(np.linalg.norm)
Cool, it works! Now let's see what happens if we lose the second half of the coefficients:

```python
# oops, we lose half the data
lossy_img = np.copy(tx_img);
lossy_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

That's quite remarkable, no? We've lost the same amount of information as before but the image is still acceptable. This is because we lost the coefficients associated to the fine details of the image but we retained the "broad strokes" encoded by the first half.

Note that if we lose the first half of the coefficients, the result would look remarkably different:

```python
lossy_img = np.copy(tx_img);
lossy_img[0:int(len(tx_img)/2)] = 0

rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

In fact, schemes like this one are used in *progressive encoding*: send the most important information first and add details if the channel permits it. You may have experienced this while browsing the internet over a slow connection.

All in all, a great application of a change of basis!

## A few of my own questions

**(?)** About `# check power of two` of `haar1D()`

```python
help(math.log)
```

    Help on built-in function log in module math:

    log(...)
        log(x[, base])

        Return the logarithm of x to the given base.
        If the base not specified, returns the natural logarithm (base e) of x.

```python
math.log(2)
```

    0.6931471805599453

```python
math.log(2, 2)
```

    1.0

```python
math.log(2, 10)
```

    0.30102999566398114

The author is just checking whether
$
\log_{2} \texttt{SIZE} = \frac{\ln \texttt{SIZE}}{\ln 2}
$
is an integer.

**N.B.** Unlike `math.log`, numpy has
- `np.log`: Natural logarithm
- `np.log2`: base 2
- `np.log10`: base 10
- `np.log1p`: log(1+p)

**(?)** What is `n` in `haar2D()`?
Must `n` be bounded?

**(R)**
From reading the code, it seems that if `SIZE = k`, then `n = 0, 1, 2, ..., k**2 -1`, like the above examples
- when `SIZE=8`, `n = 0, 1, ..., 63`
- when `SIZE=64`, `n = 0, 1, ..., 64**2 - 1`

The `n` in `haar2D()` has to do with the `n` in `haar1D()`. To better understand how the function is written that way, readers would better have read [one of the papers](http://grail.cs.washington.edu/projects/wavelets/article/wavelet1.pdf) mentioned above. Briefly speaking, `Haar1D(n , SIZE)` will return a basis for the vector space $V^j$, where with `SIZE`$ = 2^j$. For example, with the box basis and the Haar wavelets described in the paper,

$$
\forall\, f \in V^3,\; \text{we have}\\
f =
c_{0}^{0} \phi_{0}^{0}
+ d_{0}^{0} \psi_{0}^{0}
+ \left(d_{0}^{1} \psi_{0}^{1} + d_{1}^{1} \psi_{1}^{1}\right)
+ \left(d_{0}^{2} \psi_{0}^{2} + d_{1}^{2} \psi_{1}^{2} + d_{2}^{2} \psi_{2}^{2} + d_{3}^{2} \psi_{3}^{2}\right).
$$
<br>
What `haar1D(n, SIZE)` does is that it returns those $\phi_{s}^{t}$ and $\psi_{s}^{t}$.<br> As you can see from the $V^3$ example, there should be $2^3$ such basis vectors/functions; in general, there will be `SIZE`$= 2^j$ basis vectors and that's why the subscript (here in the `haar1D` function `n`) runs from `0` to `SIZE-1`.

**(?)** Is it correct for `haar2D` to normalize by Frobenius norm?

**(R)**

## Ref.
- [http://grail.cs.washington.edu/projects/wavelets/](http://grail.cs.washington.edu/projects/wavelets/)
    - [http://grail.cs.washington.edu/Research/](http://grail.cs.washington.edu/Research/)
- **exercise3.4 p.57** of the textbook written by the same authors of the course also talks about Haar basis.

```python
type(pow(2,3)), type(pow(2,3.0))
```

    (int, float)

```python
pow(2,3), pow(2,3.0)
```

    (8, 8.0)

```python
2**3, 2**3.0
```

    (8, 8.0)

```python

```


# Lotka-Volterra Introduction

The Lotka-Volterra model is a basic dynamic model named after two biomathematicians, Alfred Lotka and Vito Volterra,
who developed the system of equations independent of each other in the first half of the twentieth century.
Lotka had developed the model to explain the dynamics of predator and prey populations, expanding on his previous model
of autocatalytic chemical reactions. Volterra had developed the system of equations to model the population of predator
fish in the Adriatic Sea.

The basic system consists of two linear ordinary differential equations (ode). One equation represents the change in population of
the prey over time and is dependent on the population of the predator. The other equation represents the same but
for the predator population.

As an example, we let the prey be sheep and the predators be wolves.
If sheep were to exist without a predator and without a limitation of resources, the population would grow
exponentially:

> $ \frac{dx}{dt}=\alpha*x , $

where $\alpha$ is a constant rate of growth and $x$ is the number of sheep.

If wolves were added to the environment, then the sheep population would depend on how
often they met with the wolves. If there are more wolves, then there is more likelihood that they would meet sheep and
eat one. If there are more sheep, then they would have more likelihood of meeting wolves. If a wolf eats a sheep anytime
it finds one, then the death of the sheep is proportional to how many wolves there are. This relationship is the same
as the law of mass action, the rate of the chemical reaction is proportional to the concentration of the reactants.
In this case, the death of the sheep is then represented by a constant multiplied by the population of the sheep and the
population of the wolves. The sheep population equation is represented as:

> $ \frac{dx}{dt}=\alpha*x-\beta*x*y $

The equation for the population of the wolves is given as:

> $ \frac{dy}{dt} = \delta x y - \gamma  y, $

where $y$ is the wolf population, $\delta$ is the growth rate of the wolves that is proportional to number of sheep and
wolves and $\gamma$ is the mortality rate of the wolves.

There are many assumptions with this model and some are stated here:
- The sheep population grows exponentially without the presence of wolves.
- The wolves will always eat a sheep when it meets one.
- The wolves only eat sheep.

To see the interaction between the wolf and sheep populations, let $\alpha= 1.1, \beta = 0.4, \delta = 0.1, and \gamma = 0.4$.
If we then solve this system, assuming time is in weeks, with initial population of 10 (thousands) sheep
and 1 (thousand) wolf, the following plot shows the solutions on the interval [0,100] (100 weeks or a little under 2 years).

```python
import matplotlib.pyplot as plt
import numpy as np
import random
from scipy.integrate import odeint as ode
import tabulate
```

```python
init_pop = [10,1] # initial population levels [prey, predator]

t_steps = 1000 # time steps
t_end = 100 # end of time interval
time = np.linspace(0, t_end, num=t_steps) # array to store time

alpha = 1.1 # prey birthrate
beta = 0.4 # prey death rate
delta = 0.1 # predator birthrate
gamma = 0.4 # predator death rate

k = 150 # prey carrying capacity
k_y = 100 # predator carrying capacity
# Array to track coefficients
coef = [alpha, beta, delta, gamma]
```

```python
# Define the simulation function
def run_simu(tmp_pop, tmp_time, tmp_coef):

    tmp_x = tmp_pop[0] # Prey population value
    tmp_y = tmp_pop[1] # Predator population value

    tmp_alpha = tmp_coef[0]
    tmp_beta = tmp_coef[1]
    tmp_delta = tmp_coef[2]
    tmp_gamma = tmp_coef[3]

    dxdt = tmp_alpha * tmp_x - tmp_beta * tmp_x * tmp_y
    dydt = tmp_delta * tmp_x * tmp_y - tmp_gamma * tmp_y

    return[dxdt, dydt]

# Call the ode solver function
output = ode(run_simu, init_pop, time, args = (coef,))

plt.plot(time, output[:,0], color = "green", label = "Prey Population") # Prey output
plt.plot(time, output[:,1], color = "red", label = "Predator Population") # Prey output

plt.xlabel("Time")
plt.ylabel("Population Level")

plt.title("Predator / Prey Dynamics")
plt.legend()
plt.grid()

plt.show()
```

It makes sense that we see oscillations in the populations as seen in the plot because we have negative
feedback. As the predator population approaches its maximum, the prey population decreases rapidly. Then
after the predator population has decreased, the prey population begins to increase and eventually the predator population
will begin to increase again.

The steady state of the model is when there is equilibrium between the populations.

To find the steady state we need to find the equilibrium points.
In other words, we want to know when the rate of change (derivative) is equal to zero.

The not-so-meaningful solution is when the prey and the predator populations are zero.

The more meaningful populations can be solved:

> $ \begin{align}
>0 &= \alpha  x - \beta x  y \\
>0 &= x ( \alpha - \beta  y ) \\
> y &= \frac{\alpha}{\beta} \text{ or }x = 0
> \end{align} $

> $ \begin{align}
>0 &= \delta x y - \gamma  y \\
>0 &= y ( \delta x - \gamma ) \\
> x &= \frac{\gamma}{\delta} \text{ or }y = 0
> \end{align} $

The non-zero equilibrium point for this system is $ (\frac{\gamma}{\delta}, \frac{\alpha}{\beta}) $

If we change our birth and death rate parameters to visualize this, we see that it is in fact an equilibrium state.

```python
alpha_eq = 1.1 # prey birthrate
beta_eq = 1.1 # prey death rate
delta_eq = 0.4 # predator birthrate
gamma_eq = 4 # predator death rate

# Array to track coefficients
coef_eq = [alpha_eq, beta_eq, delta_eq, gamma_eq]

# Call the ode solver function
output_eq = ode(run_simu, init_pop, time, args = (coef_eq,))

plt.plot(time, output_eq[:,0], color = "green", label = "Prey Population") # Prey output
plt.plot(time, output_eq[:,1], color = "red", label = "Predator Population") # Prey output

plt.xlabel("Time")
plt.ylabel("Population Level")

plt.title("Predator / Prey Dynamics")
plt.legend()
plt.grid()

plt.show()

```

It would be interesting to see how the change in the parameters will change the dynamics of the populations.
To visualize this, we can make a phase space plot with varying parameter values.

```python
def run_range_ode(lrange, urange, tmp_run_simu, tmp_init_pop, tmp_time, coef_base):

    output_range = [np.zeros((t_steps,2)), np.zeros((t_steps,2)),
                    np.zeros((t_steps,2)), np.zeros((t_steps,2)),
                    np.zeros((t_steps,2)), np.zeros((t_steps,2)),
                    np.zeros((t_steps,2)), np.zeros((t_steps,2))]

    idx = 0 # index to track output array storage

    for num in [0,1,2,3]:
    # for i in range(len(coef_base)):
        coef_new = coef_base[:]
        for j in [lrange,urange]:
            coef_new[num] = round(coef_base[num] + j, 2)
            output_temp = ode(tmp_run_simu, tmp_init_pop, tmp_time, args = (coef_new,))
            output_range[idx] = output_temp
            idx = idx + 1

    return output_range

output_vary = run_range_ode(-0.2, 0.2, run_simu, init_pop, time, coef)
```

```python
# original parameters
plt.plot(output[:,0], output[:,1], color = "black", label = "a: 1.1, b: 0.4, d:0.1, g: 0.4")

# alpha changes
plt.plot(output_vary[0][:,0], output_vary[0][:,1], color = "mediumvioletred", label = "alpha: 0.9")
plt.plot(output_vary[1][:,0], output_vary[1][:,1], color = "hotpink", label = "alpha: 1.3")

# beta changes
#plt.plot(output_vary[2][:,0], output_vary[2][:,1], color = "gold", label = "beta: 0.2")
#plt.plot(output_vary[3][:,0], output_vary[3][:,1], color = "goldenrod", label = "beta: 0.8")

# delta changes
#plt.plot(output_vary[4][:,0], output_vary[4][:,1], color = "springgreen", label = "delta: 0.05")
#plt.plot(output_vary[5][:,0], output_vary[5][:,1], color = "green", label = "delta: 0.2")

# gamma changes
plt.plot(output_vary[6][:,0], output_vary[6][:,1], color = "aqua", label = "gamma: 0.2")
plt.plot(output_vary[7][:,0], output_vary[7][:,1], color = "darkcyan", label = "gamma: 0.8")

#plt.plot(output)
plt.xlabel("Prey Population")
plt.ylabel("Predator Population")

plt.title("Predator / Prey Phase Space Plot")
plt.legend()
plt.grid()

plt.show()
```

The black line is the phase plot with the original parameters. It is a closed orbit that oscillates around the
fixed point $ (\frac{\gamma}{\delta}, \frac{\alpha}{\beta}) = (4, 2.75) $ (one of the equilibrium points).

When keeping all other parameters constant and changing only $\alpha$, the ellipse would stretch or shrink vertically.
Adjusting the values of only $\gamma$, the ellipse would stretch or shrink horizontally. This is because the closed
orbit oscillates around the fixed point and when we change $\gamma$, we change the x-value of the fixed point and when
we change $\alpha$ we change the y value.

These dynamics can be proven algebraically using concepts from linear algebra (calculating the eigenvalue and
eigenvector of the Jacobian matrix).

The assumption that the prey grow exponentially in the absence of predators is not realistic.
A more realistic approach is to assume there is a carrying capacity. When the sheep population is below the environmental
carrying capacity, the growth rate is large. When the sheep population is equal to it's carrying capacity, there is no
growth and when the sheep population is above the carrying capacity, there is negative growth.

Incorporating the carrying capacity into the model would alter the sheep population ode:

> $ \frac{dx}{dt} = \alpha x (1-\frac{x}{K}), $

where $K$ is carrying capacity.

Now we need to add the predators to the environment:

> $ \frac{dx}{dt} = \alpha x (1-\frac{x}{K}) - \beta x  y $

And the predator equation remains the same:

>$ \frac{dy}{dt} = \delta x y - \gamma  y $
>

Using these updated ODEs, we can rerun our system and visualize the dynamics.

```python
# Define the simulation function
def run_simu_carry_cap(tmp_pop, tmp_time, tmp_coef):

    tmp_x = tmp_pop[0] # Prey population value
    tmp_y = tmp_pop[1] # Predator population value

    tmp_alpha = tmp_coef[0]
    tmp_beta = tmp_coef[1]
    tmp_delta = tmp_coef[2]
    tmp_gamma = tmp_coef[3]

    dxdt = tmp_alpha * tmp_x * (1 - (tmp_x/k)) - tmp_beta * tmp_x * tmp_y
    dydt = tmp_delta * tmp_x * tmp_y - tmp_gamma * tmp_y
    #dxdt = tmp_alpha * x * (1 - ((x+tmp_beta*y)/k))
    #dydt = tmp_delta * x * y - tmp_gamma * y

    return[dxdt, dydt]

# Call the ode solver function
output_carry_cap = ode(run_simu_carry_cap, init_pop, time, args = (coef,))

plt.plot(time, output_carry_cap[:,0], color = "green", label = "Prey Population") # Prey output
plt.plot(time, output_carry_cap[:,1], color = "red", label = "Predator Population") # Prey output

plt.xlabel("Time")
plt.ylabel("Population Level")

plt.title("Predator / Prey Dynamics with Prey Carrying Capacity")
plt.legend()
plt.grid()

plt.show()
```

There is still an oscillation between the predator and prey populations, but the difference between the maximum and
minimum values is decreasing. The non-zero steady state of this system has changed. Previously, it was
$ (\frac{\gamma}{\delta}, \frac{\alpha}{\beta}) $, but now that has changed.

> $ \begin{align}
> 0 &= \alpha  x (1-\frac{x}{k}) - \beta x  y
>\end{align} $

substitute $x =\frac{\gamma}{\delta}$ and solve for $y$
> $ \begin{align}
> 0 &= \alpha  \frac{\gamma}{\delta} (1-\frac{\frac{\gamma}{\delta}}{k}) - \beta\frac{\gamma}{\delta}  y \\
> 0 &= \alpha (1-\frac{\gamma}{\delta k}) - \beta  y \\
> y &= \frac{\alpha}{\beta}(1-\frac{\gamma}{\delta k})  \\
> \end{align} $

The non-zero new steady state is $(\frac{\gamma}{\delta}, \frac{\alpha}{\beta}(1-\frac{\gamma}{\delta k}))$

With the narrowing of the prey and predator population, we know that our phase plot will spiral instead of have an
ellipse.

```python
# original parameters
plt.plot(output_carry_cap[:,0], output_carry_cap[:,1], color = "black", label = "original")

#plt.plot(output)
plt.xlabel("Prey Population")
plt.ylabel("Predator Population")

plt.title("Predator / Prey  with Carrying Capacity Phase Plot")
plt.legend()
plt.grid()

plt.show()

```

To add stochastic behavior to our system, we can use an algorithm to randomly choose events
and event times from probability distributions.
We can use the Gillespie algorithm to do this.

We keep track of the population of the predator and prey. An event occurs when birth or death
occurs in either the prey or predator population. The propensity for each event to occur at a given
time is shown in the table below.

```python
data = [['Prey + 1', 'alpha * x * (1 - x/k)'],
['Prey - 1', 'beta * x * y'],
['Predator + 1', 'delta * x * y'],
['Predator - 1','gamma * y']]

print(tabulate(data, headers=["Events", "Propensity"]))

```

```python
# Initial conditions
# Original init_pop = [10,1]. Too likely for prey/predator to die out with 10 and 1
x = [10] # track prey
y = [1] # track predator
t = [0] # to keep track of time
end = 100

x_all = [] # track results of prey from all model runs
y_all = [] # track results of predator from all model runs
t_all = []
# Keep same rates of coefficients as above
#alpha = 10 # prey birthrate
#beta = 0.1 # prey death rate
#delta = 0.3 # predator birthrate
#gamma = 30 # predator death rate

#k = 150
#k_y = 100
#r  = 0.05
#K = 100
for i in range(0,5): # simulate the model 100 times
    t = [0] # reset time to rerun
    x = [10] # reset time to rerun
    y = [1] # reset time to rerun
    i = i+1
    while t[-1] < end: # keep running until last item in t is after end
        #current_x = x[-1]
        props = [alpha * x[-1] * (1- x[-1]/k),
                      beta*x[-1]*y[-1],
                      delta*x[-1]*y[-1],
                      gamma * y[-1]]
        prop_sum = sum(props)
        #print("x: ", x[-1], "; y: ", y[-1])
        if prop_sum == 0: # can not divide by zero so break out of while loop
            break

        # choose next time increment randomly from exponential distribution with mean = 1/prop_sum
        #tau = np.random.exponential(scale=1/prop_sum)
        tau = np.random.uniform(0.5, 4.5)

        # add the randomly chosen tau to the current time
        t.append(t[-1]+tau)

        # randomly choose number to later weight with probability of events
        # to randomly choose which event that will occur at the next time point
        rand = random.uniform(0,1)

        if rand * prop_sum <= props[0]: # growth of prey
            x.append(x[-1] + 1)
            y.append(y[-1])
        elif rand * prop_sum > props[0] and rand * prop_sum <= props[0] + props[1]: # death of prey
            x.append(x[-1] - 1)
            y.append(y[-1])
        elif rand * prop_sum > props[0] and rand * prop_sum <= props[0] + props[1] + props[2]: # death of predator
            x.append(x[-1])
            y.append(y[-1] + 1)
        else:
            x.append(x[-1])
            y.append(y[-1] - 1)

    x_all.append(x)
    y_all.append(y)
    t_all.append(t)

# plot results
for i in range(len(x_all)):
    plt.plot(t_all[i],x_all[i], color = "green")
    plt.plot(t_all[i],y_all[i], color = "red")

plt.legend(['Prey', 'Predator'])
plt.xlabel("Time")
plt.ylabel("Population")
plt.show()
```

In the stochastic model, there are several simulations in which the predator and/or prey population becomes extinct.
This could be a more realistic situation, especially given that the original models showed the populations had fallen
to very low numbers and could likely become extinct.

If we change the predator-prey model dynamics such that the two species are competing for the same resource(s), then we
can use the competitive Lotka-Volterra Equations which are slightly different:

> $ \frac{dx}{dt} = \alpha x (1-\frac{x+\beta y}{K_x}) $
> $ \frac{dy}{dt} = \delta y (1-\frac{y+\gamma x}{K_y}) $
>
Same as before, $\alpha$ and $\delta$ represent the growth of x and y populations, respectively. $\beta$ and $\gamma$
are the effect that y has on x and x has on y, respectively. In this situation, both x and y populations have carrying
capacities.

This system (as with the others) is not restricted to two populations, it can be generalized to many more populations
interacting in the form:

> $ \frac{dx_i}{dt}=r_ix_i(1-\frac{\sum_{j=1}^{N} a_{ij}x_j}{K_i}) $,
>

where $r_i$ is the growth rate of $x_i$, N is the number of populations that interact with $x_i$ and $K_i$ is the
carrying capacity for $x_i$.


# Functions
We've the next function:
\begin{equation}f(x) = x^{3} - 6x^{2} - 15x + 40 \end{equation}

## Important Elements
- f' and f''
- \begin{equation}f'(x) = 3x^{2} - 12x - 15 \end{equation}
- \begin{equation}f''(x) = 6x - 12 \end{equation}
- Critical Points $(x_1=-1:max ; x_2=5:min)$
- increasing and decreasing $(↗↘)$
- Infection Points $x_1=2$
- $Concave ⋂ and ⋃$
- Principal Points ${(-1,48);(2,-6);(5,-60)}$

# Critical Points and Optimization
We've explored various techniques that we can use to calculate the derivative of a function at a specific *x* value; in other words, we can determine the *slope* of the line created by the function at any point on the line.

This ability to calculate the slope means that we can use derivatives to determine some interesting properties of the function.

## Function Direction at a Point
Consider the following function, which represents the trajectory of a ball that has been kicked on a football field:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Run the Python code below to graph this function and see the trajectory of the ball over a period of 10 seconds.

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

By looking at the graph of this function, you can see that it describes a parabola in which the ball rose in height before falling back to the ground. On the graph, it's fairly easy to see when the ball was rising and when it was falling.

Of course, we can also use  derivative to determine the slope of the function at any point. We can apply some of the rules we've discussed previously to determine the derivative function:

- We can add together the derivatives of the individual terms (***-10x2***, ***100x***, and ***3***) to find the derivative of the entire function.
- The *power* rule tells us that the derivative of ***-10x2*** is ***-10 • 2x***, which is ***-20x***.
- The *power* rule also tells us that the derivative of ***100x*** is ***100***.
- The derivative of any constant, such as ***3*** is ***0***.

So:

\begin{equation}k'(x) = -20x + 100 + 0 \end{equation}

Which of course simplifies to:

\begin{equation}k'(x) = -20x + 100 \end{equation}

Now we can use this derivative function to find the slope for any value of ***x***.

Run the cell below to see a graph of the function and its derivative function:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Look closely at the purple line representing the derivative function, and note that it is a constant  decreasing value - in other words, the slope of the function is reducing linearly as x increases. Even though the function value itself is increasing for the first half of the parabola (while the ball is rising), the slope is becoming less steep (the ball is not rising at such a high rate), until finally the ball reaches its apogee and the slope becomes negative (the ball begins falling).

Note also that the point where the derivative line crosses 0 on the y-axis is also the point where the function value stops increasing and starts decreasing. When the slope has a positive value, the function is increasing; and when the slope has a negative value, the function is decreasing.

The fact that the derivative line crosses 0 at the highest point of the function makes sense if you think about it logically. If you were to draw the tangent line representing the slope at each point, it would be rotating clockwise throughout the graph, initially pointing up and to the right as the ball rises, and turning until it is pointing down and right as the ball falls. At the highest point, the tangent line would be perfectly horizontal, representing a slope of 0.

Run the following code to visualize this:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot tangent slopes for x = 2, 5, and 8
x1 = 2
x2 = 5
x3 = 8
plt.plot([x1-1,x1+1],[k(x1)-(kd(x1)),k(x1)+(kd(x1))], color='red')
plt.plot([x2-1,x2+1],[k(x2)-(kd(x2)),k(x2)+(kd(x2))], color='red')
plt.plot([x3-1,x3+1],[k(x3)-(kd(x3)),k(x3)+(kd(x3))], color='red')

plt.show()
```

Now consider the following function, which represents the number of flowers growing in a flower bed before and after the spraying of a fertilizer:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the derivative values
yd = [wd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that the green line represents the function, showing the number of flowers for 10 days before and after the fertilizer treatment. Before treatment, the number of flowers was in decline, and after treatment the flower bed started to recover.

The derivative function is shown in purple, and once again shows a linear change in slope. This time, the slope is increasing at a constant rate; and once again, the derivative function line crosses 0 at the lowest point in the function line (in other words, the slope changed from negative to positive when the flowers started to recover).

## Critical Points
From what we've seen so far, it seems that there is a relationship between a function reaching an extreme value (a maximum or a minimum), and a derivative value of 0. This makes intuitive sense; the derivative represents the slope of the line, so when a function changes from a negative slope to a positive slope, or vice-versa, the derivative must pass through 0.

However, you need to be careful not to assume that just because the derivative is 0 at a given point, that this point represents the minimum or maximum of the function. For example, consider the following function:

\begin{equation}v(x) = x^{3} - 2x + 100 \end{equation}

Run the following Python code to visualize this function and its corresponding derivative function:

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (2*x) + 100

def vd(x):
    return 3*(x**2) - 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-1000, 2000, 100))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that in this case, the purple derivative function line passes through 0 as the green function line transitions from a *concave downwards* slope (a slope that is decreasing) to a *concave upwards* slope (a slope that is increasing). The slope flattens out to 0, forming a "saddle" before the it starts increasing.

What we can learn from this is that interesting things seem to happen to the function when the derivative is 0. We call points where the derivative crosses 0 *critical points*, because they indicate that the function is changing direction. When a function changes direction from positive to negative, it forms a peak (or a *local maximum*), when the function changes direction from negative to positive it forms a trough (or *local minimum*), and when it maintains the same overall direction but changes the concavity of the slope it creates an *inflexion point*.

## Finding Minima and Maxima
A common use of calculus is to find minimum and maximum points in a function. For example, we might want to find out how many seconds it took for the kicked football to reach its maximum height, or how long it took for our fertilizer to be effective in reversing the decline of flower growth.

We've seen that when a function changes direction to create a maximum peak or a minimum trough, the derivative of the function is 0, so a step towards finding these extreme points might be to simply find all of the points in the function where the derivative is 0. For example, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

From this, we've calculated the function for the derivative as:

\begin{equation}k'(x) = -20x + 100 \end{equation}

We can then solve the derivative equation for an f'(x) value of 0:

\begin{equation}-20x + 100 = 0 \end{equation}

We can remove the constant by subtracting 100 to both sides:

\begin{equation}-20x = -100 \end{equation}

Multiplying both sides by -1 gets rid of the negative values (this isn't strictly necessary, but makes the equation a little less confusing)

\begin{equation}20x = 100 \end{equation}

So:

\begin{equation}x = 5 \end{equation}

So we know that the derivative will be 0 when *x* is 5, but is this a minimum, a maximum, or neither? It could just be an inflexion point, or the entire function could be a constant value with a slope of 0) Without looking at the graph, it's difficult to tell.

## Second Order Derivatives
The solution to our problem is to find the derivative of the derivative! Until now, we've found the derivative of a function, and indicated it as ***f'(x)***. Technically, this is known as the *prime* derivative; and it describes the slope of the function. Since the derivative function is itself a function, we can find its derivative, which we call the *second order* (or sometimes just *second*) derivative. This is indicated like this: ***f''(x)***.

So, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Here's the function for the prime derivative:

\begin{equation}k'(x) = -20x + 100 \end{equation}

And using a combination of the power rule and the constant rule, here's the function for the second derivative:

\begin{equation}k''(x) = -20 \end{equation}

Now, without even drawing the graph, we can see that the second derivative has a constant value; so we know that the slope of the prime derivative is linear; and because it's a negative value, we know that it is decreasing. So when the prime derivative crosses 0, it we know that the slope of the function is decreasing linearly; so the point at *x=0* must be a maximum point.

Run the following code to plot the function, the prime derivative, and the second derivative for the kicked ball:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

def k2d(x):
    return -20

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the k'(x) values
yd = [kd(i) for i in x]

# Use the 2-derivative function to get the k''(x)
y2d = [k2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot  k'(x)
plt.plot(x,yd, color='purple')

# Plot k''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

Let's take the same approach for the flower bed problem. Here's the function:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

Using the power rule and constant rule, gives us the prime derivative function:

\begin{equation}w'(x) = 2x + 2 \end{equation}

Applying the power rule and constant rule to the prime derivative function gives us the second derivative function:

\begin{equation}w''(x) = 2 \end{equation}

Note that this time, the second derivative is a positive constant, so the prime derivative (which is the slope of the function) is increasing linearly. The point where the prime derivative crosses 0 must therefore be a minimum. Let's run the code below to check:

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

def w2d(x):
    return 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the w'(x) values
yd = [wd(i) for i in x]

# Use the 2-derivative function to get the w''(x) values
y2d = [w2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot w'(x)
plt.plot(x,yd, color='purple')

# Plot w''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

## Critical Points that are *Not* Maxima or Minima
Of course, it's possible for a function to form a "saddle" where the prime derivative is zero at a point that is not a minimum or maximum. Here's an example of a function like this:

\begin{equation}v(x) = x^{3} - 6x^{2} + 12x + 2 \end{equation}

And here's its prime derivative:

\begin{equation}v'(x) = 3x^{2} - 12x + 12 \end{equation}

Let's find a critical point where v'(x) = 0

\begin{equation}3x^{2} - 12x + 12 = 0 \end{equation}

Factor the x-terms

\begin{equation}3x(x - 4) = 12 \end{equation}

Divide both sides by 3:

\begin{equation}x(x - 4) = 4 \end{equation}

Factor the x terms back again

\begin{equation}x^{2} - 4x = 4 \end{equation}

Complete the square, step 1

\begin{equation}x^{2} - 4x + 4 = 0 \end{equation}

Complete the square, step 2

\begin{equation}(x - 2)^{2} = 0 \end{equation}

Find the square root:

\begin{equation}x - 2 = \pm\sqrt{0}\end{equation}

\begin{equation}x - 2 = +\sqrt{0} = 0, -\sqrt{0} = 0\end{equation}

v'(2) = 0 (only touches 0 once)

Is it a maximum or minimum? Let's find the second derivative:

\begin{equation}v''(x) = 6x - 12\end{equation}

So

\begin{equation}v''(2) = 0\end{equation}

So it's neither negative or positive, so it's not a maximum or minimum.

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (6*(x**2)) + (12*x) + 2

def vd(x):
    return (3*(x**2)) - (12*x) + 12

def v2d(x):
    return (3*(2*x)) - 12

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-5, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Use the derivative function to get the derivative values
y2d = [v2d(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-2000, 2000, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot the derivative
plt.plot(x,y2d, color='magenta')

plt.show()

print ("v(2) = " + str(v(2)))

print ("v'(2) = " + str(vd(2)))

print ("v''(2) = " + str(v2d(2)))

```

## Optimization
The ability to use derivatives to find minima and maxima of a function makes it a useful tool for scenarios where you need to optimize a function for a specific variable.

### Defining Functions to be Optimized
For example, suppose you have decided to build an online video service that is based on a subscription model. You plan to charge a monthly subscription fee, and you want to make the most revenue possible. The problem is that customers are price-sensitive, so if you set the monthly fee too high, you'll deter some customers from signing up. Conversely, if you set the fee too low, you may get more customers, but at the cost of reduced revenue.

What you need is some kind of function that will tell you how many subscriptions you might expect to get based on a given fee. So you've done some research, and found a formula to indicate that the expected subscription volume (in thousands) can be calculated as 5-times the monthly fee subtracted from 100; or expressed as a function:

\begin{equation}s(x) = -5x + 100\end{equation}

What you actually want to optimize is monthly revenue, which is simply the number of subscribers multiplied by the fee:

\begin{equation}r(x) = s(x) \cdot x\end{equation}

We can combine ***s(x)*** into ***r(x)*** like this:

\begin{equation}r(x) = -5x^{2} + 100x\end{equation}

### Finding the Prime Derivative
The function ***r(x)*** will return the expected monthly revenue (in thousands) for any proposed fee (*x*). What we need to do now is to find the fee that yields the maximum revenue. Fortunately, we can use a derivative to do that.

First, we need to determine the prime derivative of ***r(x)***, and we can do that easily using the power rule:

\begin{equation}r'(x) = 2 \cdot -5x + 100\end{equation}

Which is:

\begin{equation}r'(x) = -10x + 100\end{equation}

### Find Critical Points
Now we need to find any critical points where the derivative is 0, as this could indicate a maximum:

\begin{equation}-10x + 100 = 0\end{equation}

Let's isolate the *x* term:

\begin{equation}-10x = -100\end{equation}

Both sides are negative, so we can mulitply both by -1 to make them positive without affecting the equation:

\begin{equation}10x = 100\end{equation}

Now we can divide both sides by 10 to isolate *x*:

\begin{equation}x = \frac{100}{10}\end{equation}

So:

\begin{equation}x = 10\end{equation}

#### Check for a Maximum
We now know that with an *x* value of of **10**, the derivative is 0; or put another way, when the fee is 10, the slope indicating the change in subscription volume is flat. This could potentially be a point where the change in subscription volume has peaked (in other words, a maximum); but it could also be a minimum or just an inflexion point where the rate of change transitions from negative to positive.

To be sure, we can check the second order derivative. We can calculate this by applying the power rule to the prime derivative:

\begin{equation}r''(x) = -10\end{equation}

Note that the second derivative is a constant with a negative value. It will be the same for any point, including our critical point at *x=10*:

\begin{equation}r''(10) = -10\end{equation}

A negative value for the second derivative tells us that the derivative slope is moving in a negative direction at the point where it is 0, so the function value must be at a maximum.

In other words, the optimal monthly fee for our online video service is 10 - this will generate the maximum monthly revenue.

Run the code below to show the function ***r(x)*** as a graph, and verify that the maximum point is at x = 10.

```python
%matplotlib inline

# Create function s
def s(x):
    return (-5*x) + 100

# Create function r
def r(x):
    return s(x) * x

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 21))

# Use the function to get the y values
y = [r(i) for i in x]

# Set up the graph
plt.xlabel('x (monthly fee)')
plt.ylabel('r(x) (revenue in $,000)')
plt.xticks(range(0,22, 1))
plt.yticks(range(0, 600, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

```python

```

MathsToML/Module02-Derivatives and Optimization/02-04-Critical Points and Optimization.ipynb


```python
import numpy as np
import pandas as pd
import scipy as sp
import sklearn as sl
import seaborn as sns; sns.set()
import matplotlib as mpl
import sympy as spy
import matplotlib.pyplot as plt
import scipy.fftpack
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import axes3d
from matplotlib import cm
from scipy.fftpack import fft
x = spy.Symbol('x')
%matplotlib inline
```

# Tarea 4

Con base a los métodos vistos en clase resuelva las siguientes dos preguntas

## (A) Integrales

* $\int_{0}^{1}x^{-1/2}\,\text{d}x$ = 1.9936755351040816
* $\int_{0}^{\infty}e^{-x}\ln{x}\,\text{d}x$ = −𝛾 = -0.57721566490153286061
* $\int_{0}^{\infty}\frac{\sin{x}}{x}\,\text{d}x$ = 𝜋/2 = 1.5707963267948966

```python
def f1(x):
    if x >= 0:
        return x**(-1/2)
    return 0
f1_v = np.vectorize(f1)
```

```python
N = 1000000
X1 = np.linspace(0.000001,1,N)
Y1 = f1_v(X1)
#plt.scatter(X,Y)
```

```python
sp.integrate.simpson(y=Y1, x=X1)
```

    1.9980152377229565

```python
result2 = spy.integrate(spy.exp(-x)*spy.log(x),(x,0,np.inf))
print(result2," = ",float(result2))
```

    -EulerGamma  =  -0.5772156649015329

```python
result3 = spy.integrate(spy.sin(x)/x,(x,0,np.inf))
print(result3," = ",float(result3))
```

    pi/2  =  1.5707963267948966

## (B) Fourier

Calcule la transformada rápida de Fourier para la función de la **Tarea 3 (D)** en el intervalo $[0,4]$ ($k$ máximo $2\pi n/L$ para $n=25$). Ajuste la transformada de Fourier para los datos de la **Tarea 3** usando el método de regresión exacto de la **Tarea 3 (C)** y compare con el anterior resultado. Para ambos ejercicios haga una interpolación y grafique para comparar.

```python
df = pd.read_pickle('ex1.gz')
N = 25
T = 2*np.pi*N / 100
x = df.iloc[:,0].values
y = df.iloc[:,1].values
y_f = sp.fft.fft(y)
x_f = np.linspace(0.0, 1.0/(2.0*T), N//2)

plt.plot(x_f, 2.0/N * np.abs(y_f[:N//2]))
plt.show()
```

```python

```


# How to Draw Ellipse of Covariance Matrix
Given a 2x2 covariance matrix, how to draw the ellipse representing it. The following function explains the method to visualize multivariate normal distributions and correlation matrices. Formulae for radii & rotation are provided for covariance matrix shown below
\begin{align}
\Sigma = \begin{bmatrix} a & b \\ c & d \end{bmatrix}
\end{align}

## Radii and Rotation

\begin{align}
\lambda_{1,2} &= \frac{a+c}{2} \pm \sqrt{\left( \frac{a-c}{2} \right)^{2} + b^{2}} \\
\theta &= \begin{cases} 0 & \text{ if } b = 0 \text{ and } a \geq c \\
 \frac{\pi}{2} & \text{ if } b = 0 \text{ and } a < c \\
 \text{atan2}(\lambda_{1} - a, b) & \text{ if } b \neq 0
\end{cases}
\end{align}
Here, $\theta$ is the angle in radians from positive x-axis to the ellipse's major axis in the counterclockwise direction. $\sqrt{\lambda_{1}}$ is the radius of the major axis (the longer radius) and $\sqrt{\lambda_{2}}$ is the radius of the minor axis (shorter radius). In $atan2(\cdot, \cdot)$, the first parameter is $y$ and second is $x$.

```python
import random
import math
import numpy as np
import matplotlib.pyplot as plt
import time
from scipy.linalg import block_diag
from scipy.special import erfinv
from scipy.stats import t as tdist
from numpy.linalg import inv
from numpy import linalg as LA
from matplotlib.patches import Ellipse
```

```python
def GetAngleAndRadii(covar):
    """
    Given a covariance matrix, the function GetAngleAndRadii() calculates
    the major axis and minor axis radii and the orientation of the ellipse.

    Inputs:
    covar: 2x2 matrix

    Output:
    major_radius: Radius of the major axis of ellipse
    minor_radius: Radius of the minor axis of ellipse
    theta       : Orientation angle in radians from positive x-axis
                  to the ellipse's major axis in the counterclockwise direction
    """

    # Infer the a,b,c values
    a = covar[0,0]
    b = covar[0,1]
    c = covar[1,1]

    if b > a:
        raise Exception("Sorry, covariance matrix is invalid - Cov[0,1] should be < Cov[0,0] ")

    lambda_1 = (a+c)/2 + math.sqrt(((a-c)/2)**2 + b**2)
    lambda_2 = (a+c)/2 - math.sqrt(((a-c)/2)**2 + b**2)

    # Infer the radii
    major_radius = math.sqrt(lambda_1)
    minor_radius = math.sqrt(lambda_2)

    # Infer the rotation
    if b == 0:
        if a >= c:
            theta = 0
        else:
            theta = pi/2
    else:
        theta = math.atan2(lambda_1-a, b)

    return major_radius, minor_radius, theta

```

```python
# Check the above code
covar_check = np.array([[9,5],[5,4]])
major_radius_check, minor_radius_check, theta_check = GetAngleAndRadii(covar_check)
print('major axis radius = ', round(major_radius_check,2),
      'minor axis radius = ', round(minor_radius_check,2),
      'orientation = ', round(theta_check,2), 'rad')
```

    major axis radius =  3.48 minor axis radius =  0.95 orientation =  0.55 rad

```python
def plot_ellipse(center, cov = None):

    # Get the center of ellipse
    x_cent, y_cent = center

    print('center x at: ', x_cent)
    print('center y at: ', y_cent)

    # Get Ellipse Properties from cov matrix
    if cov is not None:
        major_radius, minor_radius, theta_orient = GetAngleAndRadii(cov)
        print('major axis radius = ', round(major_radius,2),
              'minor axis radius = ', round(minor_radius,2),
              'orientation = ', round(theta_orient,2), 'rad')
        eig_vec,eig_val,u = np.linalg.svd(cov)

    # Generate data for ellipse structure
    t    = np.linspace(0,2*np.pi,1000)
    x    = major_radius*np.cos(t)
    y    = minor_radius*np.sin(t)
    data = np.array([x,y])
    R    = np.array([[np.cos(theta_orient),-np.sin(theta_orient)],
                     [np.sin(theta_orient),np.cos(theta_orient)]])
    T    = np.dot(R,eig_vec)
    data = np.dot(T,data)

    # Center the ellipse at given center
    data[0] += x_cent
    data[1] += y_cent

    # Plot the ellipse
    fig,ax = plt.subplots()
    ax.plot(data[0],data[1],color='b',linestyle='-')
    ax.fill(data[0],data[1])
```

```python
covar_check = np.array([[9,4],[4,3]])
plot_ellipse(center = (1,2), cov=covar_check, plot_way = 2)
```

```python

```


```python
import sympy as sp
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
sp.init_printing()
%matplotlib inline
```

```python
def plot_RS(a, b, n, N, f, method):
    x = np.linspace(a, b, N + 1)
    y = f(x)
    X = np.linspace(a, b, n * N + 1)
    Y = f(X)
    plt.plot(X, Y, 'b')
    x_test = x
    y_test = y
    align = 'edge'
    width = (b - a) / N
    if method == "left":
        x_test = x[:-1]
        y_test = y[:-1]
    elif method == "mid":
        x_test = (x[:-1] + x[1:]) / 2
        y_test = f(x_test)
        align = 'center'
    elif method == "right":
        x_test = x[1:]
        y_test = y[1:]
        width = -width
    plt.plot(x_test, y_test, 'b.', markersize=10)
    plt.bar(x_test, y_test, width=width, alpha=0.3, align=align, edgecolor='b')
    return sum(y_test * (b - a) / N)

n = 10; N = 2
f = lambda x : x + 1
plt.figure(figsize=(35, 10))
plt.subplot(2, 3, 1)
s = plot_RS(0, 2, n, N, f, "left")
plt.title(f"Left Riemann Sum, N = {N}")
plt.subplot(2, 3, 2)
s = plot_RS(0, 2, n, N, f, "mid")
plt.title(f"Midpoint Riemann Sum, N = {N}")
plt.subplot(2, 3, 3)
s = plot_RS(0, 2, n, N, f, "right")
plt.title(f"Right Riemann Sum, N = {N}")
N = 10
plt.subplot(2, 3, 4)
s = plot_RS(0, 2, n, N, f, "left")
plt.title(f"Left Riemann Sum, N = {N}")
plt.subplot(2, 3, 5)
s = plot_RS(0, 2, n, N, f, "mid")
plt.title(f"Midpoint Riemann Sum, N = {N}")
plt.subplot(2, 3, 6)
s = plot_RS(0, 2, n, N, f, "right")
plt.title(f"Right Riemann Sum, N = {N}")

```

```python

```


Find the volume of the following "stadium". The inner wall is part of a sphere of radius 2 and the outer wall is a cylinder of radius 4.

$$ \int_{\pi/2}^{2\pi} \int_{\pi/4}^{\pi/2} \int_2^{4/\sin\phi} \rho^2 \sin\phi \,d\rho\,d\phi\,d\theta $$

```python
from sympy import *
```

```python
rh,ph,th = var("rho phi theta")
```

```python
vol = integrate(rh**2*sin(ph),(rh,2,4/sin(ph)),(ph,pi/4,pi/2),(th,pi/2,2*pi))
simplify(vol)
```

$\displaystyle 2 \pi \left(16 - \sqrt{2}\right)$

```python
vol.evalf()
```

$\displaystyle 91.6451990385567$

```python
import numpy as np
from scipy.integrate import tplquad
```

```python
tplquad(lambda r,p,t: r**2*np.sin(p),np.pi/2,2*np.pi,np.pi/4,np.pi/2,2,lambda t,p: 4/np.sin(p))
```

    (91.64519903855667, 1.0174661006896088e-12)


# Modeling and Simulation in Python

Chapter 7: Thermal systems

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

### Mixing liquids

We can figure out the final temperature of a mixture by setting the total heat flow to zero and then solving for $T$.

```python
from sympy import *

init_printing()
```

```python
C1, C2, T1, T2, T = symbols('C1 C2 T1 T2 T')

eq = Eq(C1 * (T - T1) + C2 * (T - T2), 0)
eq
```

```python
solve(eq, T)
```

### Analysis

We can use SymPy to solve the cooling differential equation.

```python
T_init, T_env, r, t = symbols('T_init T_env r t')
T = Function('T')

eqn = Eq(diff(T(t), t), -r * (T(t) - T_env))
eqn
```
Here's the general solution:

```python
solution_eq = dsolve(eqn)
solution_eq
```

```python
general = solution_eq.rhs
general
```

We can use the initial condition to solve for $C_1$.  First we evaluate the general solution at $t=0$

```python
at0 = general.subs(t, 0)
at0
```

Now we set $T(0) = T_{init}$ and solve for $C_1$

```python
solutions = solve(Eq(at0, T_init), C1)
value_of_C1 = solutions[0]
value_of_C1
```

Then we plug the result into the general solution to get the particular solution:

```python
particular = general.subs(C1, value_of_C1)
particular
```

We use a similar process to estimate $r$ based on the observation $T(t_{end}) = T_{end}$

```python
t_end, T_end = symbols('t_end T_end')
```

Here's the particular solution evaluated at $t_{end}$

```python
at_end = particular.subs(t, t_end)
at_end
```

Now we set $T(t_{end}) = T_{end}$ and solve for $r$

```python
solutions = solve(Eq(at_end, T_end), r)
value_of_r = solutions[0]
value_of_r
```

We can use `evalf` to plug in numbers for the symbols.  The result is a SymPy float, which we have to convert to a Python float.

```python
subs = dict(t_end=30, T_end=70, T_init=90, T_env=22)
r_coffee2 = value_of_r.evalf(subs=subs)
type(r_coffee2)
```

    sympy.core.numbers.Float

```python
r_coffee2 = float(r_coffee2)
r_coffee2
```


## z* visualization

Visualize the work of _Adcroft and Campin (2004)_. They incorporated a z* depth in the MITgcm which is defined as

$$z=\eta + s^* z^*$$
where $s^* = 1 + \eta/H$ and $z=-H$ at the bottom of the ocean and $\eta$ is at the top of the ocean.

```python
from numpy import linspace
```

```python
H=3000.0 # meters (m)
eta=30.0  # meters (m)
numPts=50
waterColumn=linspace(0,0,num=numPts)
z=linspace(eta, -H, num=numPts)
```

```python
sStar=1+eta/H
zStar= (z-eta) / sStar
```

#### Plotting z* next to z to show the difference

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.figure(figsize=(2,20))
plt.plot(waterColumn, z,
         'k_', ms=40, label='z')
plt.plot(waterColumn, zStar,
         'co', ms=10, alpha=0.6, label='z*')
plt.legend()
plt.show()

plt.figure(figsize=(10,2.5))
plt.plot(abs(z-zStar))
plt.show()
```

### Math proof

Let $z \in [-H, \eta]$, we want to show that $z^*$ contains only non-positive numbers. We do so by rearranging the relation
$$z=\eta + s^* z^*$$
to
$$z^*=\frac{z-\eta}{s^*}.$$

At $z=-H$

\begin{align}
    z^* &= \frac{-H-\eta}{s^*} \\
        &= \frac{-H-\eta}{1+\frac{\eta}{H}}\\
    (1+\frac{\eta}{H})z^* &= -H-\eta \\
    (H+\eta)z^* &= -H(H+\eta)\\
    z^* &= -H.
\end{align}

At $z=\eta$
\begin{align}
    z^* &= \frac{\eta-\eta}{s^*} \\
    z^* &= 0.
\end{align}

Therefore, $z^* \in [-H, 0]$, which is a non-positive range for all values of $H$ and $\eta$.

```python

```


Für $q \neq 1$ kann die n-te Partialsumme der geometrischen Reihe wie folgt berechnet werden:

$$
\sum_{k=0}^{n}{q^{k}} = \frac{1-q^{n+1}}{1-q}
$$

Zeigen Sie dies empirisch, indem Sie mithilfe von `sympy` die linke und rechte Seite in einer for-Schleife für $n = 1, \ldots, 5$ berechnen und deren faktorisierte Formen mit einem logischen Operator vergleichen. Geben Sie dies für jedes $n$ aus. Anschließend berechnen Sie die 5-te Partialsumme an der Stelle $q = 3$ (indem Sie mittels einer `sympy` Routine für $q$ den konkreten Wert übergeben) und geben diese aus.

```python
from sympy.abc import k, n, q
from sympy import Sum

for ni in range(1, 6):
    geom_series = Sum(q**k, (k, 0, ni))
    geom_formula = (1 - q**(ni + 1))/(1 - q)
    print(f'n = {ni}, Same factorized form: {geom_series.doit().factor() == geom_formula.factor()}')
```

    n = 1, Same factorized form: True
    n = 2, Same factorized form: True
    n = 3, Same factorized form: True
    n = 4, Same factorized form: True
    n = 5, Same factorized form: True

```python
geom_series = Sum(q**k, (k, 0, n))
geom_formula = (1 - q**(n + 1))/(1 - q)

subs_dict = {k: v for k, v in zip('nq', [5, 3])}
for function, label in zip(
    [geom_series, geom_formula],
    ['Series', 'Formula']
):
    print(f'{label}: {function.evalf(subs = subs_dict)}, n = 5, q = 3')
```

    Series: 364.000000000000, n = 5, q = 3
    Formula: 364.000000000000, n = 5, q = 3


# Coinflips

*Coinflips until you die.*

Here are some things that I like:

1. Reminding myself that probability theory is rooted in sets. (Blitzstein & Hwang's "Introduction to Probability" does a great job driving this point home. I claim that applied probability curricula normally fail to do so -- a couple of Venn Diagrams is not enough.)
2. Indicator variables that convert set theory expressions into random variables (c.f. Matching Problem).
3. Indicator variables that convert expected value calculations to probabilities.
4. Independent events.

I claim:

- Problems typically become confusing when random variables have complicated interdependencies, but often they are made up of "atoms" of independent events. For difficult problems, deconstructing a problem in terms of independent events can be half the battle.
- Recursion. Typically emerges most naturally from applying the law of total expectation.
- Expected values for complicated problems are much simpler creatures than full probability distributions, and can often be found using symmetry tricks. This provides fertile ground for the type of dumb probability puzzles that quant interviewers apparently *salivate* over.

## Expected Flips until $n$ Heads in a Row: Recursion and Law of Total Expectation
A coin is flipped until there is a sequence of $n$ subsequent heads. What is the exected number of flips $\mathbb{E}[N]$?

Condition on the outcome of the first flip:
\begin{equation}
\begin{array}{rl}
\mathbb{E}[N] &= \sum_{j=\{H,T\}}\mathbb{E}[N|f_1 = j]p(f_1=j)\\
&=\frac{1}{2}\underbrace{\mathbb{E}[N|f_1=T]}_{\mathbb{E}[N+1]=\mathbb{E}[N]+1} + \frac{1}{2}\mathbb{E}[N|f_1=H]
\end{array}
\end{equation}

If you the first flip is tails, you're back to where you started, except that you'e already done one flip. Hence $\mathbb{E}[N|f_1=T] = \mathbb{E}[N]+1$. If the first throw is heads, then we should find ourselves slightly closer to $k$ flips. Repeating the procedure by conditioning on the outcome of the second throw:

\begin{equation}
\begin{array}{rl}
\mathbb{E}[N|f_1=H] &= \sum_{j=\{H,T\}}\mathbb{E}[N|f_1 = H|f_2=j]p(f_2=j)\\
&=\frac{1}{2}\underbrace{\mathbb{E}[N|f_1=H,f_2=T]}_{\mathbb{E}[N+2]=\mathbb{E}[N]+2} + \frac{1}{2}\mathbb{E}[N|f_1=H,f_2=H]
\end{array}
\end{equation}

Unsurprisingly, this looks like a pattern where either you start over because you flipped tails, or you are dealing with the conditional probability of already having flipped some consecutive heads. Let $M_k = \mathbb{E}[N|f_1=H,f_2=H,...,f_k=H]$, so that $M_0 = \mathbb{E}[N]$ is the expected number of flips conditioned on no previous outcomes. Then:

\begin{equation}
M_k = \frac{1}{2}(k+1+M_0)+\frac{1}{2}M_{k+1}
\end{equation}

Cool. So far this is a linear system that goes on forever. Fortunately the agreement was that the game was going to be over if we throw $n$ heads in a row. That means that $\mathbb{E}[N|f_1=H,f_2=H,...,f_n=H] = M_n = n$ because here I am explicitly conditioning on throwing $n$ heads in a row.

Take $n=3$, for example. Then:
\begin{equation}
\begin{array}{rl}
M_0 &= \frac{1}{2}(0+1+M_0)+\frac{1}{2}\underbrace{\left[\frac{1}{2}(1+1+M_0)+\frac{1}{2}\underbrace{\left[\frac{1}{2}(2+1+M_0)+\frac{1}{2}\underbrace{\left[n\right]}_{M_3}\right]}_{M_2}\right]}_{M_1}\\
&=\sum^{n-1}_{k=0}\frac{1}{2^{k+1}}(1+k+M_0) + \frac{1}{2^n}n\\
M_0\left[1-\sum_{k=0}^{n-1} \frac{1}{2^{k+1}}\right]&=\sum^{n-1}_{k=0}\frac{1+k}{2^{k+1}} + \frac{1}{2^n}n\\
M_0 &= \left[\sum^{n-1}_{k=0}\frac{1+k}{2^{k+1}} + \frac{1}{2^n}n\right]\left[1-\sum_{k=0}^{n-1} \frac{1}{2^{k+1}}\right]^{-1}
\end{array}
\end{equation}

To the dismay of my friends and family, I will not simplify this further because dinner is almost ready. (Even though it is easily possible using the finite geometric series)

```python
import numpy as np
import matplotlib.pyplot as plt

def M_0(n):
    """
    Expected number of flips until there are n consecutive heads
    """
    numerator = np.sum([(1+k)/2**(k+1) for k in range(n)])+n/2**n
    denominator = 1-np.sum([1/2**(k+1) for k in range(n)])

    return numerator/denominator

def get_sequence(n):
    """
    generates a sequence of flips and stops the moment there are n consecutive heads, then returns the length of the sequence.
    might run for a long time.
    """
    sq = ''
    while sq[-n:] != n*'H':
        sq+=np.random.choice(['H','T'])
    return len(sq)

def M_0_simulated(n,n_trials=100):
    return np.mean([get_sequence(n) for i in range(n_trials)])

# look at distribution, because it's interesting
plt.figure(figsize=(12,8))
nn = [2,4,6,8]
for n in nn:
    data = [get_sequence(n) for i in range(5000)]
    _ = plt.hist(data,bins=list(range(200)),density=True)

plt.legend(['n=%i' % i for i in nn])
plt.xlabel('flips')
_ = plt.title('flips until n-consecutive heads')
```

```python
nn = list(range(10))

plt.figure(figsize=(12,8))
plt.plot(nn,[M_0(n) for n in nn],'d-',linewidth=2.5,markersize=10)
plt.plot(nn,[M_0_simulated(n,n_trials=1000) for n in nn],'o',linewidth=2.5,markersize=10)

plt.legend(['analytical','simulated'])
plt.title('Expected Numer of Flips until n Consecutive Heads')
plt.xlabel('n',fontsize=15)
_= plt.ylabel('Flips',fontsize=15)
```

## Why This Doesn't Generalize To Arbitrary Sequences

So what if instead of looking for a sequence of $n$ heads, I am looking for some arbitrary $n$-flip sequence, such as "HHTHHH" or whatever.

The formula does not work anymore, because it assumed that you start over the moment that you flip tails. For an arbirary sequence, flipping the wrong thing does not necessarily mean that you start over.

Let's say $S=\{s_1=H,s_2=H,s_3=T,s_4=H,s_5=H,s_6=H\}$.

*Previously*, for a sequence of all heads:

\begin{equation}
\mathbb{E}[N|f_1=H,...,f_k\neq H] = \mathbb{E}[N]
\end{equation}.

Now,

\begin{equation}
\mathbb{E}[N|f_1=s_1,f_2=s_2,f_3=s_3,f_4=s_4,f_5=s_5,f_6\neq s_6] = \mathbb{E}[N|f_1=s_1,f_2=s_2,f_3=s_3]
\end{equation}.

If you get only the last flip of the sequence wrong, then it puts you back to where you had the first three flips of the sequence right. Because of this, the expected number of flips until you see some sequence S is smaller.

Even though the result doesn't generalize, it is interesting because it provides an upper bound on the expected value. Any repeat of the first few items within the sequence at some point during the sequence will lower the expected number of flips.

An approach toward studying this might be by modeling the process as a Markov Chain. If $S$ has length $n$, then let the state space be all possible sequences of length $n$. Then the transition matrix acts in a way so the states are modified corresponding to dropping the first entry and tagging on a "T" or an "H" with equation probability. I.e., state "THH" --> "HHH" with p=0.5 and "THH" --> "HHT" with p=0.5. This is except for the state that corresponds to the sought after Sequence, $S$, which is absorbing. Denote the set of states where the last $r$ entries are equal to the first $k$ elements of $S$ as $\{x_r\}$ (these have size $2^{n-r}$ because you can assign $n-r$ arbitrarily and $r$ must match up with S). When $S$ consists of only heads, then the states $\{x_r\}$ only communiate with $\{x_0\}$ or $\{x_{r+1}\}$. For arbitrary sequences, you will see communication of the form $\{x_r\} \rightarrow \{x_{r-m}\}$. A setback, but not all the way to 0.

```python
def get_sequence_S(S):
    """
    generates a sequence of flips and stops the moment that sequence S is flipped.
    """
    sq = ''
    while sq[-len(S):] != S:
        sq+=np.random.choice(['H','T'])
    return len(sq)

def M_0_simulated_S(S,n_trials=100):
    return np.mean([get_sequence_S(S) for i in range(n_trials)])

SS = 'HHTHHH'
nn = list(range(len(SS)))

plt.figure(figsize=(12,8))
for n in nn:
    S = SS[:n]

    plt.plot(n,M_0(n),'d',color='blue',markersize=10)
    plt.plot(n,M_0_simulated_S(S,n_trials=1000),'o',color='orange',markersize=10)

plt.legend(np.hstack([['%i heads' % n,'sequence %s' % S[:n]] for n in nn]))
plt.title('Expected Numer of Flips until S')
plt.xlabel('n',fontsize=15)
_= plt.ylabel('Flips',fontsize=15)
```

Probability - Coinflips, Expected Flips until Sequence of n Heads.ipynb